From 1a24dd0b02879f9e2c9d27ba9962eb29c3654f0f Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Tue, 12 Jan 2016 16:34:47 -0800 Subject: [PATCH 0001/3084] Cretonne README and LICENSE. --- README.rst | 9 +++ cranelift/LICENSE | 202 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 211 insertions(+) create mode 100644 README.rst create mode 100644 cranelift/LICENSE diff --git a/README.rst b/README.rst new file mode 100644 index 0000000000..10db808462 --- /dev/null +++ b/README.rst @@ -0,0 +1,9 @@ +======================= +Cretonne Code Generator +======================= + +Cretonne is a low-level retargetable code generator. It translates a +target-independent intermediate language into executable machine code. Cretonne +aims to generate code that behaves identically on different target +architectures, and that can be executed safely in a sandbox. + diff --git a/cranelift/LICENSE b/cranelift/LICENSE new file mode 100644 index 0000000000..d645695673 --- /dev/null +++ b/cranelift/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. From 1803df091ef1bdb2762efa41473233c6e1e5b521 Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Tue, 12 Jan 2016 16:46:27 -0800 Subject: [PATCH 0002/3084] Initial Sphinx configuration. --- cranelift/docs/.gitignore | 1 + cranelift/docs/Makefile | 192 +++++++++++++++++++++++++ cranelift/docs/conf.py | 289 ++++++++++++++++++++++++++++++++++++++ cranelift/docs/index.rst | 16 +++ cranelift/docs/make.bat | 263 ++++++++++++++++++++++++++++++++++ 5 files changed, 761 insertions(+) create mode 100644 cranelift/docs/.gitignore create mode 100644 cranelift/docs/Makefile create mode 100644 cranelift/docs/conf.py create mode 100644 cranelift/docs/index.rst create mode 100644 cranelift/docs/make.bat diff --git a/cranelift/docs/.gitignore b/cranelift/docs/.gitignore new file mode 100644 index 0000000000..e35d8850c9 --- /dev/null +++ b/cranelift/docs/.gitignore @@ -0,0 +1 @@ +_build diff --git a/cranelift/docs/Makefile b/cranelift/docs/Makefile new file mode 100644 index 0000000000..619d2caf24 --- /dev/null +++ b/cranelift/docs/Makefile @@ -0,0 +1,192 @@ +# Makefile for Sphinx documentation +# + +# You can set these variables from the command line. +SPHINXOPTS = +SPHINXBUILD = sphinx-build +PAPER = +BUILDDIR = _build + +# User-friendly check for sphinx-build +ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) +$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) +endif + +# Internal variables. +PAPEROPT_a4 = -D latex_paper_size=a4 +PAPEROPT_letter = -D latex_paper_size=letter +ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . +# the i18n builder cannot share the environment and doctrees with the others +I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . + +.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext + +help: + @echo "Please use \`make ' where is one of" + @echo " html to make standalone HTML files" + @echo " dirhtml to make HTML files named index.html in directories" + @echo " singlehtml to make a single large HTML file" + @echo " pickle to make pickle files" + @echo " json to make JSON files" + @echo " htmlhelp to make HTML files and a HTML help project" + @echo " qthelp to make HTML files and a qthelp project" + @echo " applehelp to make an Apple Help Book" + @echo " devhelp to make HTML files and a Devhelp project" + @echo " epub to make an epub" + @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" + @echo " latexpdf to make LaTeX files and run them through pdflatex" + @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" + @echo " text to make text files" + @echo " man to make manual pages" + @echo " texinfo to make Texinfo files" + @echo " info to make Texinfo files and run them through makeinfo" + @echo " gettext to make PO message catalogs" + @echo " changes to make an overview of all changed/added/deprecated items" + @echo " xml to make Docutils-native XML files" + @echo " pseudoxml to make pseudoxml-XML files for display purposes" + @echo " linkcheck to check all external links for integrity" + @echo " doctest to run all doctests embedded in the documentation (if enabled)" + @echo " coverage to run coverage check of the documentation (if enabled)" + +clean: + rm -rf $(BUILDDIR)/* + +html: + $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html + @echo + @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." + +dirhtml: + $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml + @echo + @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." + +singlehtml: + $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml + @echo + @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." + +pickle: + $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle + @echo + @echo "Build finished; now you can process the pickle files." + +json: + $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json + @echo + @echo "Build finished; now you can process the JSON files." + +htmlhelp: + $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp + @echo + @echo "Build finished; now you can run HTML Help Workshop with the" \ + ".hhp project file in $(BUILDDIR)/htmlhelp." + +qthelp: + $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp + @echo + @echo "Build finished; now you can run "qcollectiongenerator" with the" \ + ".qhcp project file in $(BUILDDIR)/qthelp, like this:" + @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/cretonne.qhcp" + @echo "To view the help file:" + @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/cretonne.qhc" + +applehelp: + $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp + @echo + @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." + @echo "N.B. You won't be able to view it unless you put it in" \ + "~/Library/Documentation/Help or install it in your application" \ + "bundle." + +devhelp: + $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp + @echo + @echo "Build finished." + @echo "To view the help file:" + @echo "# mkdir -p $$HOME/.local/share/devhelp/cretonne" + @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/cretonne" + @echo "# devhelp" + +epub: + $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub + @echo + @echo "Build finished. The epub file is in $(BUILDDIR)/epub." + +latex: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo + @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." + @echo "Run \`make' in that directory to run these through (pdf)latex" \ + "(use \`make latexpdf' here to do that automatically)." + +latexpdf: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo "Running LaTeX files through pdflatex..." + $(MAKE) -C $(BUILDDIR)/latex all-pdf + @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." + +latexpdfja: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo "Running LaTeX files through platex and dvipdfmx..." + $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja + @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." + +text: + $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text + @echo + @echo "Build finished. The text files are in $(BUILDDIR)/text." + +man: + $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man + @echo + @echo "Build finished. The manual pages are in $(BUILDDIR)/man." + +texinfo: + $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo + @echo + @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." + @echo "Run \`make' in that directory to run these through makeinfo" \ + "(use \`make info' here to do that automatically)." + +info: + $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo + @echo "Running Texinfo files through makeinfo..." + make -C $(BUILDDIR)/texinfo info + @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." + +gettext: + $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale + @echo + @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." + +changes: + $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes + @echo + @echo "The overview file is in $(BUILDDIR)/changes." + +linkcheck: + $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck + @echo + @echo "Link check complete; look for any errors in the above output " \ + "or in $(BUILDDIR)/linkcheck/output.txt." + +doctest: + $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest + @echo "Testing of doctests in the sources finished, look at the " \ + "results in $(BUILDDIR)/doctest/output.txt." + +coverage: + $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage + @echo "Testing of coverage in the sources finished, look at the " \ + "results in $(BUILDDIR)/coverage/python.txt." + +xml: + $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml + @echo + @echo "Build finished. The XML files are in $(BUILDDIR)/xml." + +pseudoxml: + $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml + @echo + @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py new file mode 100644 index 0000000000..4f2a236b61 --- /dev/null +++ b/cranelift/docs/conf.py @@ -0,0 +1,289 @@ +# -*- coding: utf-8 -*- +# +# cretonne documentation build configuration file, created by +# sphinx-quickstart on Fri Jan 8 10:11:19 2016. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +import sys +import os +import shlex + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +#sys.path.insert(0, os.path.abspath('.')) + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +#needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ + 'sphinx.ext.autodoc', + 'sphinx.ext.todo', + 'sphinx.ext.mathjax', + 'sphinx.ext.ifconfig', +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The encoding of source files. +#source_encoding = 'utf-8-sig' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'cretonne' +copyright = u'2016, Jakob Stoklund Olesen' +author = u'Jakob Stoklund Olesen' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = u'0.0' +# The full version, including alpha/beta/rc tags. +release = u'0.0' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# There are two options for replacing |today|: either, you set today to some +# non-false value, then it is used: +#today = '' +# Else, today_fmt is used as the format for a strftime call. +#today_fmt = '%B %d, %Y' + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +exclude_patterns = ['_build'] + +# The reST default role (used for this markup: `text`) to use for all +# documents. +#default_role = None + +# If true, '()' will be appended to :func: etc. cross-reference text. +#add_function_parentheses = True + +# If true, the current module name will be prepended to all description +# unit titles (such as .. function::). +#add_module_names = True + +# If true, sectionauthor and moduleauthor directives will be shown in the +# output. They are ignored by default. +#show_authors = False + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# A list of ignored prefixes for module index sorting. +#modindex_common_prefix = [] + +# If true, keep warnings as "system message" paragraphs in the built documents. +#keep_warnings = False + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = True + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +html_theme = 'sphinx_rtd_theme' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +#html_theme_options = {} + +# Add any paths that contain custom themes here, relative to this directory. +#html_theme_path = [] + +# The name for this set of Sphinx documents. If None, it defaults to +# " v documentation". +#html_title = None + +# A shorter title for the navigation bar. Default is the same as html_title. +#html_short_title = None + +# The name of an image file (relative to this directory) to place at the top +# of the sidebar. +#html_logo = None + +# The name of an image file (within the static path) to use as favicon of the +# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 +# pixels large. +#html_favicon = None + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] + +# Add any extra paths that contain custom files (such as robots.txt or +# .htaccess) here, relative to this directory. These files are copied +# directly to the root of the documentation. +#html_extra_path = [] + +# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, +# using the given strftime format. +#html_last_updated_fmt = '%b %d, %Y' + +# If true, SmartyPants will be used to convert quotes and dashes to +# typographically correct entities. +#html_use_smartypants = True + +# Custom sidebar templates, maps document names to template names. +#html_sidebars = {} + +# Additional templates that should be rendered to pages, maps page names to +# template names. +#html_additional_pages = {} + +# If false, no module index is generated. +#html_domain_indices = True + +# If false, no index is generated. +#html_use_index = True + +# If true, the index is split into individual pages for each letter. +#html_split_index = False + +# If true, links to the reST sources are added to the pages. +#html_show_sourcelink = True + +# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. +#html_show_sphinx = True + +# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. +#html_show_copyright = True + +# If true, an OpenSearch description file will be output, and all pages will +# contain a tag referring to it. The value of this option must be the +# base URL from which the finished HTML is served. +#html_use_opensearch = '' + +# This is the file name suffix for HTML files (e.g. ".xhtml"). +#html_file_suffix = None + +# Language to be used for generating the HTML full-text search index. +# Sphinx supports the following languages: +# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' +# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' +#html_search_language = 'en' + +# A dictionary with options for the search language support, empty by default. +# Now only 'ja' uses this config value +#html_search_options = {'type': 'default'} + +# The name of a javascript file (relative to the configuration directory) that +# implements a search results scorer. If empty, the default will be used. +#html_search_scorer = 'scorer.js' + +# Output file base name for HTML help builder. +htmlhelp_basename = 'cretonnedoc' + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { +# The paper size ('letterpaper' or 'a4paper'). +#'papersize': 'letterpaper', + +# The font size ('10pt', '11pt' or '12pt'). +#'pointsize': '10pt', + +# Additional stuff for the LaTeX preamble. +#'preamble': '', + +# Latex figure (float) alignment +#'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, 'cretonne.tex', u'cretonne Documentation', + u'Jakob Stoklund Olesen', 'manual'), +] + +# The name of an image file (relative to this directory) to place at the top of +# the title page. +#latex_logo = None + +# For "manual" documents, if this is true, then toplevel headings are parts, +# not chapters. +#latex_use_parts = False + +# If true, show page references after internal links. +#latex_show_pagerefs = False + +# If true, show URL addresses after external links. +#latex_show_urls = False + +# Documents to append as an appendix to all manuals. +#latex_appendices = [] + +# If false, no module index is generated. +#latex_domain_indices = True + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, 'cretonne', u'cretonne Documentation', + [author], 1) +] + +# If true, show URL addresses after external links. +#man_show_urls = False + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, 'cretonne', u'cretonne Documentation', + author, 'cretonne', 'One line description of project.', + 'Miscellaneous'), +] + +# Documents to append as an appendix to all manuals. +#texinfo_appendices = [] + +# If false, no module index is generated. +#texinfo_domain_indices = True + +# How to display URL addresses: 'footnote', 'no', or 'inline'. +#texinfo_show_urls = 'footnote' + +# If true, do not generate a @detailmenu in the "Top" node's menu. +#texinfo_no_detailmenu = False diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst new file mode 100644 index 0000000000..5f8e7c2df1 --- /dev/null +++ b/cranelift/docs/index.rst @@ -0,0 +1,16 @@ +Cretonne Code Generator +======================= + +Contents: + +.. toctree:: + :maxdepth: 2 + + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` + diff --git a/cranelift/docs/make.bat b/cranelift/docs/make.bat new file mode 100644 index 0000000000..3f6fe2e48d --- /dev/null +++ b/cranelift/docs/make.bat @@ -0,0 +1,263 @@ +@ECHO OFF + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set BUILDDIR=_build +set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . +set I18NSPHINXOPTS=%SPHINXOPTS% . +if NOT "%PAPER%" == "" ( + set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% + set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS% +) + +if "%1" == "" goto help + +if "%1" == "help" ( + :help + echo.Please use `make ^` where ^ is one of + echo. html to make standalone HTML files + echo. dirhtml to make HTML files named index.html in directories + echo. singlehtml to make a single large HTML file + echo. pickle to make pickle files + echo. json to make JSON files + echo. htmlhelp to make HTML files and a HTML help project + echo. qthelp to make HTML files and a qthelp project + echo. devhelp to make HTML files and a Devhelp project + echo. epub to make an epub + echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter + echo. text to make text files + echo. man to make manual pages + echo. texinfo to make Texinfo files + echo. gettext to make PO message catalogs + echo. changes to make an overview over all changed/added/deprecated items + echo. xml to make Docutils-native XML files + echo. pseudoxml to make pseudoxml-XML files for display purposes + echo. linkcheck to check all external links for integrity + echo. doctest to run all doctests embedded in the documentation if enabled + echo. coverage to run coverage check of the documentation if enabled + goto end +) + +if "%1" == "clean" ( + for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i + del /q /s %BUILDDIR%\* + goto end +) + + +REM Check if sphinx-build is available and fallback to Python version if any +%SPHINXBUILD% 1>NUL 2>NUL +if errorlevel 9009 goto sphinx_python +goto sphinx_ok + +:sphinx_python + +set SPHINXBUILD=python -m sphinx.__init__ +%SPHINXBUILD% 2> nul +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.http://sphinx-doc.org/ + exit /b 1 +) + +:sphinx_ok + + +if "%1" == "html" ( + %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The HTML pages are in %BUILDDIR%/html. + goto end +) + +if "%1" == "dirhtml" ( + %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. + goto end +) + +if "%1" == "singlehtml" ( + %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. + goto end +) + +if "%1" == "pickle" ( + %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle + if errorlevel 1 exit /b 1 + echo. + echo.Build finished; now you can process the pickle files. + goto end +) + +if "%1" == "json" ( + %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json + if errorlevel 1 exit /b 1 + echo. + echo.Build finished; now you can process the JSON files. + goto end +) + +if "%1" == "htmlhelp" ( + %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp + if errorlevel 1 exit /b 1 + echo. + echo.Build finished; now you can run HTML Help Workshop with the ^ +.hhp project file in %BUILDDIR%/htmlhelp. + goto end +) + +if "%1" == "qthelp" ( + %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp + if errorlevel 1 exit /b 1 + echo. + echo.Build finished; now you can run "qcollectiongenerator" with the ^ +.qhcp project file in %BUILDDIR%/qthelp, like this: + echo.^> qcollectiongenerator %BUILDDIR%\qthelp\cretonne.qhcp + echo.To view the help file: + echo.^> assistant -collectionFile %BUILDDIR%\qthelp\cretonne.ghc + goto end +) + +if "%1" == "devhelp" ( + %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. + goto end +) + +if "%1" == "epub" ( + %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The epub file is in %BUILDDIR%/epub. + goto end +) + +if "%1" == "latex" ( + %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex + if errorlevel 1 exit /b 1 + echo. + echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. + goto end +) + +if "%1" == "latexpdf" ( + %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex + cd %BUILDDIR%/latex + make all-pdf + cd %~dp0 + echo. + echo.Build finished; the PDF files are in %BUILDDIR%/latex. + goto end +) + +if "%1" == "latexpdfja" ( + %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex + cd %BUILDDIR%/latex + make all-pdf-ja + cd %~dp0 + echo. + echo.Build finished; the PDF files are in %BUILDDIR%/latex. + goto end +) + +if "%1" == "text" ( + %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The text files are in %BUILDDIR%/text. + goto end +) + +if "%1" == "man" ( + %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The manual pages are in %BUILDDIR%/man. + goto end +) + +if "%1" == "texinfo" ( + %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo. + goto end +) + +if "%1" == "gettext" ( + %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The message catalogs are in %BUILDDIR%/locale. + goto end +) + +if "%1" == "changes" ( + %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes + if errorlevel 1 exit /b 1 + echo. + echo.The overview file is in %BUILDDIR%/changes. + goto end +) + +if "%1" == "linkcheck" ( + %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck + if errorlevel 1 exit /b 1 + echo. + echo.Link check complete; look for any errors in the above output ^ +or in %BUILDDIR%/linkcheck/output.txt. + goto end +) + +if "%1" == "doctest" ( + %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest + if errorlevel 1 exit /b 1 + echo. + echo.Testing of doctests in the sources finished, look at the ^ +results in %BUILDDIR%/doctest/output.txt. + goto end +) + +if "%1" == "coverage" ( + %SPHINXBUILD% -b coverage %ALLSPHINXOPTS% %BUILDDIR%/coverage + if errorlevel 1 exit /b 1 + echo. + echo.Testing of coverage in the sources finished, look at the ^ +results in %BUILDDIR%/coverage/python.txt. + goto end +) + +if "%1" == "xml" ( + %SPHINXBUILD% -b xml %ALLSPHINXOPTS% %BUILDDIR%/xml + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The XML files are in %BUILDDIR%/xml. + goto end +) + +if "%1" == "pseudoxml" ( + %SPHINXBUILD% -b pseudoxml %ALLSPHINXOPTS% %BUILDDIR%/pseudoxml + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The pseudo-XML files are in %BUILDDIR%/pseudoxml. + goto end +) + +:end From 5638bc464f65d9a07a9138ef36e025fe18beb24f Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Tue, 12 Jan 2016 16:53:43 -0800 Subject: [PATCH 0003/3084] Add a Cretonne domain for Sphinx. Include roles for documenting IL instructions and types, including index cross references. --- cranelift/.gitignore | 1 + cranelift/docs/conf.py | 3 +- cranelift/docs/cton_domain.py | 187 ++++++++++++++++++++++++++++++++++ 3 files changed, 190 insertions(+), 1 deletion(-) create mode 100644 cranelift/.gitignore create mode 100644 cranelift/docs/cton_domain.py diff --git a/cranelift/.gitignore b/cranelift/.gitignore new file mode 100644 index 0000000000..0d20b6487c --- /dev/null +++ b/cranelift/.gitignore @@ -0,0 +1 @@ +*.pyc diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py index 4f2a236b61..58cc740464 100644 --- a/cranelift/docs/conf.py +++ b/cranelift/docs/conf.py @@ -19,7 +19,7 @@ import shlex # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. -#sys.path.insert(0, os.path.abspath('.')) +sys.path.insert(0, os.path.abspath('.')) # -- General configuration ------------------------------------------------ @@ -34,6 +34,7 @@ extensions = [ 'sphinx.ext.todo', 'sphinx.ext.mathjax', 'sphinx.ext.ifconfig', + 'cton_domain' ] # Add any paths that contain templates here, relative to this directory. diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py new file mode 100644 index 0000000000..afa372921f --- /dev/null +++ b/cranelift/docs/cton_domain.py @@ -0,0 +1,187 @@ +# -*- coding: utf-8 -*- +# +# Sphinx domain for documenting compiler intermediate languages. +# +# This defines a 'cton' Sphinx domain with the following directives and roles: +# +# .. cton::type:: type +# Document an IR type. +# .. cton:inst:: v1, v2 = inst op1, op2 +# Document an IR instruction. +# + +import re + +from docutils import nodes + +from sphinx import addnodes +from sphinx.directives import ObjectDescription +from sphinx.domains import Domain, ObjType +from sphinx.locale import l_, _ +from sphinx.roles import XRefRole +from sphinx.util.docfields import Field, TypedField +from sphinx.util.nodes import make_refnode + +class CtonObject(ObjectDescription): + """ + Any kind of Cretonne IL object. + + This is a shared base class for the different kinds of indexable objects + in the Cretonne IL reference. + """ + + def add_target_and_index(self, name, sig, signode): + """ + Add ``name`` the the index. + + :param name: The object name returned by :func:`handle_signature`. + :param sig: The signature text. + :param signode: The output node. + """ + targetname = self.objtype + '-' + name + if targetname not in self.state.document.ids: + signode['names'].append(targetname) + signode['ids'].append(targetname) + signode['first'] = (not self.names) + self.state.document.note_explicit_target(signode) + inv = self.env.domaindata['cton']['objects'] + if name in inv: + self.state_machine.reporter.warning( + 'duplicate Cretonne object description of %s, ' % name + + 'other instance in ' + self.env.doc2path(inv[name][0]), + line=self.lineno) + inv[name] = (self.env.docname, self.objtype) + + indextext = self.get_index_text(name) + if indextext: + self.indexnode['entries'].append(('single', indextext, + targetname, '')) + +class CtonType(CtonObject): + """A Cretonne IL type description.""" + + def handle_signature(self, sig, signode): + """ + Parse type signature in ``sig`` and append description to signode. + + Return a global object name for ``add_target_and_index``. + """ + + name = sig.strip() + signode += addnodes.desc_name(name, name) + return name + + def get_index_text(self, name): + return name + ' (IL type)' + +sep_equal = re.compile('\s*=\s*') +sep_comma = re.compile('\s*,\s*') + +def parse_params(s, signode): + for i,p in enumerate(sep_comma.split(s)): + if i != 0: + signode += nodes.Text(', ') + signode += nodes.emphasis(p, p) + +class CtonInst(CtonObject): + """A Cretonne IL instruction.""" + + doc_field_types = [ + TypedField('argument', label=l_('Arguments'), + names=('in', 'arg'), + typerolename='type', typenames=('type',)), + TypedField('result', label=l_('Results'), + names=('out', 'result'), + typerolename='type', typenames=('type',)), + Field('resulttype', label=l_('Result type'), has_arg=False, + names=('rtype',)), + ] + + def handle_signature(self, sig, signode): + # Look for signatures like + # + # v1, v2 = foo op1, op2 + # v1 = foo + # foo op1 + + parts = re.split(sep_equal, sig, 1) + if len(parts) == 2: + # Outgoing parameters. + parse_params(parts[0], signode) + signode += nodes.Text(' = ') + name = parts[1] + else: + name = parts[0] + + # Parse 'name arg, arg' + parts = name.split(None, 1) + name = parts[0] + signode += addnodes.desc_name(name, name) + + if len(parts) == 2: + # Incoming parameters. + signode += nodes.Text(' ') + parse_params(parts[1], signode) + + return name + + def get_index_text(self, name): + return name + +class CretonneDomain(Domain): + """Cretonne domain for intermediate language objects.""" + name = 'cton' + label = 'Cretonne' + + object_types = { + 'type' : ObjType(l_('type'), 'type'), + 'inst' : ObjType(l_('instruction'), 'inst') + } + + directives = { + 'type' : CtonType, + 'inst' : CtonInst, + } + + roles = { + 'type' : XRefRole(), + 'inst' : XRefRole(), + } + + initial_data = { + 'objects': {}, # fullname -> docname, objtype + } + + def clear_doc(self, docname): + for fullname, (fn, _l) in list(self.data['objects'].items()): + if fn == docname: + del self.data['objects'][fullname] + + def merge_domaindata(self, docnames, otherdata): + for fullname, (fn, objtype) in otherdata['objects'].items(): + if fn in docnames: + self.data['objects'][fullname] = (fn, objtype) + + def resolve_xref(self, env, fromdocname, builder, typ, target, node, + contnode): + objects = self.data['objects'] + if target not in objects: + return None + obj = objects[target] + return make_refnode(builder, fromdocname, obj[0], + obj[1] + '-' + target, contnode, target) + + def resolve_any_xref(self, env, fromdocname, builder, target, + node, contnode): + objects = self.data['objects'] + if target not in objects: + return [] + obj = objects[target] + return [('cton:' + self.role_for_objtype(obj[1]), + make_refnode(builder, fromdocname, obj[0], + obj[1] + '-' + target, contnode, target))] + +def setup(app): + app.add_domain(CretonneDomain) + + return { 'version' : '0.1' } From 857e0f75a49bc92e6110c3fe24a99ef12493fef9 Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Tue, 12 Jan 2016 16:54:49 -0800 Subject: [PATCH 0004/3084] Begin the intermediate language reference. --- cranelift/docs/index.rst | 1 + cranelift/docs/langref.rst | 585 +++++++++++++++++++++++++++++++++++++ 2 files changed, 586 insertions(+) create mode 100644 cranelift/docs/langref.rst diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst index 5f8e7c2df1..137e7b1e84 100644 --- a/cranelift/docs/index.rst +++ b/cranelift/docs/index.rst @@ -6,6 +6,7 @@ Contents: .. toctree:: :maxdepth: 2 + langref Indices and tables ================== diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst new file mode 100644 index 0000000000..31f79df0c2 --- /dev/null +++ b/cranelift/docs/langref.rst @@ -0,0 +1,585 @@ +**************************************** +Cretonne Intermediate Language Reference +**************************************** + +Type system +=========== + +.. default-domain:: cton + +All SSA values have a type which determines the size and shape (for SIMD +vectors) of the value. Many instructions are polymorphic -- they can operate on +different types. + +.. type:: bool + + A boolean value that is either true or false. Booleans can't be stored in + memory. + +Integer types +------------- + +Integer values have a fixed size and can be interpreted as either signed or +unsigned. Some instructions will interpret an operand as a signed or unsigned +number, others don't care. + +.. type:: i8 + + A 8-bit integer value taking up 1 byte in memory. + +.. type:: i16 + + A 16-bit integer value taking up 2 bytes in memory. + +.. type:: i32 + + A 32-bit integer value taking up 4 bytes in memory. + +.. type:: i64 + + A 64-bit integer value taking up 8 bytes in memory. + +Floating point types +-------------------- + +The floating point types have the IEEE semantics that are supported by most +hardware. There is no support for higher-precision types like quads or +double-double formats. + +.. type:: f32 + + A 32-bit floating point type represented in the IEEE 754 *Single precision* + format. This corresponds to the :c:type:`float` type in most C + implementations. + +.. type:: f64 + + A 64-bit floating point type represented in the IEEE 754 *Double precision* + format. This corresponds to the :c:type:`double` type in most C + implementations. + +SIMD vector types +----------------- + +A SIMD vector type represents a vector of values from one of the scalar types +(:type:`bool`, integer, and floating point). Each scalar value in a SIMD type is +called a *lane*. The number of lanes must be a power of two in the range 2-256. + +.. type:: vNiB + + A SIMD vector of integers. The lane type :type:`iB` must be one of the + integer types :type:`i8` ... :type:`i64`. + + Some concrete integer vector types are :type:`v4i32`, :type:`v8i64`, and + :type:`v4i16`. + + The size of a SIMD integer vector in memory is :math:`N B\over 8` bytes. + +.. type:: vNf32 + + A SIMD vector of single precision floating point numbers. + + Some concrete :type:`f32` vector types are: :type:`v2f32`, :type:`v4f32`, + and :type:`v8f32`. + + The size of a :type:`f32` vector in memory is :math:`4N` bytes. + +.. type:: vNf64 + + A SIMD vector of double precision floating point numbers. + + Some concrete :type:`f64` vector types are: :type:`v2f64`, :type:`v4f64`, + and :type:`v8f64`. + + The size of a :type:`f64` vector in memory is :math:`8N` bytes. + +.. type:: vNbool + + A boolean SIMD vector. + + Like the :type:`bool` type, a boolean vector cannot be stored in memory. It + can only be used for ephemeral SSA values. + +Instructions +============ + +Control flow instructions +------------------------- + +.. inst:: br EBB(args...) + + Branch. + + Unconditionally branch to an extended basic block, passing the specified + EBB arguments. The number and types of arguments must match the destination + EBB. + +.. inst:: brz x, EBB(args...) + + Branch when zero. + + If ``x`` is a :type:`bool` value, take the branch when ``x`` is false. If + ``x`` is an integer value, take the branch when ``x = 0``. + + :param iN/bool x: Value to test. + :param EBB: Destination extended basic block. + +.. inst:: brnz x, EBB(args...) + + Branch when non-zero. + + If ``x`` is a :type:`bool` value, take the branch when ``x`` is true. If + ``x`` is an integer value, take the branch when ``x != 0``. + + :param iN/bool x: Value to test. + :param EBB: Destination extended basic block. + +Special operations +================== + +Most operations are easily classified as arithmetic or control flow. These +instructions are not so easily classified. + +.. inst:: a = iconst n + + Integer constant. + +.. inst:: a = fconst n + + Floating point constant. + +.. inst:: a = vconst n + + Vector constant (floating point or integer). + +.. inst:: a = select c, x, y + + Conditional select. + + :param c bool: Controlling flag. + :param x: Value to return when ``c`` is true. + :param y: Value to return when ``c`` is false. Must be same type as ``x``. + :rtype: Same type as ``x`` and ``y``. + + This instruction selects whole values. Use :inst:`vselect` for + lane-wise selection. + +Vector operations +================= + +.. inst:: a = vselect c, x, y + + Vector lane select. + + Select lanes from ``x`` or ``y`` controlled by the lanes of the boolean + vector ``c``. + + :arg vNbool c: Controlling flag vector. + :arg x: Vector with lanes selected by the true lanes of ``c``. + Must be a vector type with the same number of lanes as ``c``. + :arg y: Vector with lanes selected by the false lanes of ``c``. + Must be same type as ``x``. + :rtype: Same type as ``x`` and ``y``. + +.. inst:: a = vbuild x, y, z, ... + + Vector build. + + Build a vector value from the provided lanes. + +.. inst:: a = splat x + + Vector splat. + + Return a vector whose lanes are all ``x``. + +.. inst:: a = insertlane x, idx, y + + Insert ``y`` as lane ``idx`` in x. + + The lane index, ``idx``, is an immediate value, not an SSA value. It must + indicate a valid lane index for the type of ``x``. + +.. inst:: a = extractlane x, idx + + Extract lane ``idx`` from ``x``. + + The lane index, ``idx``, is an immediate value, not an SSA value. It must + indicate a valid lane index for the type of ``x``. + +Integer operations +================== + +.. inst:: a = icmp cond, x, y + + Integer comparison. + + :param cond: Condition code determining how ``x`` and ``y`` are compared. + :param x, y: Integer scalar or vector values of the same type. + :rtype: :type:`bool` or :type:`vNbool` with the same number of lanes as + ``x`` and ``y``. + + The condition code determines if the operands are interpreted as signed or + unsigned integers. + + ====== ======== ========= + Signed Unsigned Condition + ====== ======== ========= + eq eq Equal + ne ne Not equal + slt ult Less than + sge uge Greater than or equal + sgt ugt Greater than + sle ule Less than or equal + ====== ======== ========= + +.. inst:: a = iadd x, y + + Wrapping integer addition: :math:`a := x + y \pmod{2^B}`. This instruction + does not depend on the signed/unsigned interpretation of the operands. + +.. inst:: a = isub x, y + + Wrapping integer subtraction: :math:`a := x - y \pmod{2^B}`. This + instruction does not depend on the signed/unsigned interpretation of the + operands. + +.. todo:: Overflow arithmetic + + Add instructions for add with carry out / carry in and so on. Enough to + implement larger integer types efficiently. It should also be possible to + legalize :type:`i64` arithmetic to terms of :type:`i32` operations. + +.. inst:: a = ineg x + + Wrapping integer negation: :math:`a := -x \pmod{2^B}`. This instruction does + not depend on the signed/unsigned interpretation of the operand. + +.. inst:: a = imul x, y + + Wrapping integer multiplication: :math:`a := x y \pmod{2^B}`. This + instruction does not depend on the signed/unsigned interpretation of the + operands. + +.. todo:: Larger multiplication results. + + For example, ``smulx`` which multiplies :type:`i32` operands to produce a + :type:`i64` result. Alternatively, ``smulhi`` and ``smullo`` pairs. + +.. inst:: a = udiv x, y + + Unsigned integer division: :math:`a := \lfloor {x \over y} \rfloor`. This + operation traps if the divisor is zero. + + .. todo:: + Add a ``udiv_imm`` variant with an immediate divisor greater than 1. + This is useful for pattern-matching divide-by-constant, and this + instruction would be non-trapping. + +.. inst:: a = sdiv x, y + + Signed integer division rounded toward zero: :math:`a := sign(xy) \lfloor + {|x| \over |y|}\rfloor`. This operation traps if the divisor is zero, or if + the result is not representable in :math:`B` bits two's complement. This only + happens when :math:`x = -2^{B-1}, y = -1`. + + .. todo:: + Add a ``sdiv_imm`` variant with an immediate non-zero divisor. This is + useful for pattern-matching divide-by-constant, and this instruction + would be non-trapping. Don't allow divisors 0, 1, or -1. + +.. inst:: a = urem x, y + + Unsigned integer remainder. This operation traps if the divisor is zero. + + .. todo:: + Add a ``urem_imm`` non-trapping variant. + +.. inst:: a = srem x, y + + Signed integer remainder. This operation traps if the divisor is zero. + + .. todo:: + Clarify whether the result has the sign of the divisor or the dividend. + Should we add a ``smod`` instruction for the case where the result has + the same sign as the divisor? + +.. todo:: Minimum / maximum. + + NEON has ``smin``, ``smax``, ``umin``, and ``umax`` instructions. We should + replicate those for both scalar and vector integer types. Even if the + target ISA doesn't have scalar operations, these are good pattern mtching + targets. + +.. todo:: Saturating arithmetic. + + Mostly for SIMD use, but again these are good paterns to contract. + Something like ``usatadd``, ``usatsub``, ``ssatadd``, and ``ssatsub`` is a + good start. + +Bitwise operations +================== + +.. inst:: a = and x, y + + Bitwise and. + + :rtype: bool, iB, vNiB, vNfB? + +.. inst:: a = or x, y + + Bitwise or. + + :rtype: bool, iB, vNiB, vNfB? + +.. inst:: a = xor x, y + + Bitwise xor. + + :rtype: bool, iB, vNiB, vNfB? + +.. inst:: a = not x + + Bitwise not. + + :rtype: bool, iB, vNiB, vNfB? + +.. todo:: Redundant bitwise operators. + + ARM has instructions like ``bic(x,y) = x & ~y``, ``orn(x,y) = x | ~y``, and + ``eon(x,y) = x ^ ~y``. + +.. inst:: a = rotl x, y + + Rotate left. + + Rotate the bits in ``x`` by ``y`` places. + + :param x: Integer value to be rotated. + :param y: Number of bits to shift. Any integer type, not necessarily the + same type as ``x``. + :rtype: Same type as ``x``. + +.. inst:: a = rotr x, y + + Rotate right. + + Rotate the bits in ``x`` by ``y`` places. + + :param x: Integer value to be rotated. + :param y: Number of bits to shift. Any integer type, not necessarily the + same type as ``x``. + :rtype: Same type as ``x``. + +.. inst:: a = ishl x, y + + Integer shift left. Shift the bits in ``x`` towards the MSB by ``y`` + places. Shift in zero bits to the LSB. + + The shift amount is masked to the size of ``x``. + + :param x: Integer value to be shifted. + :param y: Number of bits to shift. Any integer type, not necessarily the + same type as ``x``. + :rtype: Same type as ``x``. + + When shifting a B-bits integer type, this instruction computes: + + .. math:: + s &:= y \pmod B, \\ + a &:= x \cdot 2^s \pmod{2^B}. + + .. todo:: Add ``ishl_imm`` variant with an immediate ``y``. + +.. inst:: a = ushr x, y + + Unsigned shift right. Shift bits in ``x`` towards the LSB by ``y`` places, + shifting in zero bits to the MSB. Also called a *logical shift*. + + The shift amount is masked to the size of the register. + + :param x: Integer value to be shifted. + :param y: Number of bits to shift. Can be any integer type, not necessarily + the same type as ``x``. + :rtype: Same type as ``x``. + + When shifting a B-bits integer type, this instruction computes: + + .. math:: + s &:= y \pmod B, \\ + a &:= \lfloor x \cdot 2^{-s} \rfloor. + + .. todo:: Add ``ushr_imm`` variant with an immediate ``y``. + +.. inst:: a = sshr x, y + + Signed shift right. Shift bits in ``x`` towards the LSB by ``y`` places, + shifting in sign bits to the MSB. Also called an *arithmetic shift*. + + The shift amount is masked to the size of the register. + + :param x: Integer value to be shifted. + :param y: Number of bits to shift. Can be any integer type, not necessarily + the same type as ``x``. + :rtype: Same type as ``x``. + + .. todo:: Add ``sshr_imm`` variant with an immediate ``y``. + +.. inst:: a = clz x + + Count leading zero bits. + + :param x: Integer value. + :rtype: :type:`i8` + + Starting from the MSB in ``x``, count the number of zero bits before + reaching the first one bit. When ``x`` is zero, returns the size of x in + bits. + +.. inst:: a = cls x + + Count leading sign bits. + + :param x: Integer value. + :rtype: :type:`i8` + + Starting from the MSB after the sign bit in ``x``, count the number of + consecutive bits identical to the sign bit. When ``x`` is 0 or -1, returns + one less than the size of x in bits. + +.. inst:: a = ctz x + + Count trailing zeros. + + :param x: Integer value. + :rtype: :type:`i8` + + Starting from the LSB in ``x``, count the number of zero bits before + reaching the first one bit. When ``x`` is zero, returns the size of x in + bits. + +.. inst:: a = popcnt x + + Population count + + :param x: Integer value. + :rtype: :type:`i8` + + Count the number of one bits in ``x``. + + +Floating point operations +========================= + +.. inst:: a = fcmp cond, x, y + + Floating point comparison. + + :param cond: Condition code determining how ``x`` and ``y`` are compared. + :param x, y: Floating point scalar or vector values of the same type. + :rtype: :type:`bool` or :type:`vNbool` with the same number of lanes as + ``x`` and ``y``. + + An 'ordered' condition code yields ``false`` if either operand is Nan. + + An 'unordered' condition code yields ``true`` if either operand is Nan. + + ======= ========= ========= + Ordered Unordered Condition + ======= ========= ========= + ord uno None (ord = no NaNs, uno = some NaNs) + oeq ueq Equal + one une Not equal + olt ult Less than + oge uge Greater than or equal + ogt ugt Greater than + ole ule Less than or equal + ======= ========= ========= + +.. inst:: fadd x,y + + Floating point addition. + +.. inst:: fsub x,y + + Floating point subtraction. + +.. inst:: fneg x + + Floating point negation. + + :returns: ``x`` with its sign bit inverted. + + Note that this is a pure bitwise operation. + +.. inst:: fabs x + + Floating point absolute value. + + :returns: ``x`` with its sign bit cleared. + + Note that this is a pure bitwise operation. + +.. inst:: a = fcopysign x, y + + Floating point copy sign. + + :returns: ``x`` with its sign changed to that of ``y``. + + Note that this is a pure bitwise operation. The sign bit from ``y`` is + copied to the sign bit of ``x``. + +.. inst:: fmul x, y +.. inst:: fdiv x, y +.. inst:: fmin x, y +.. inst:: fminnum x, y +.. inst:: fmax x, y +.. inst:: fmaxnum x, y +.. inst:: ceil x + + Round floating point round to integral, towards positive infinity. + +.. inst:: floor x + + Round floating point round to integral, towards negative infinity. + +.. inst:: trunc x + + Round floating point round to integral, towards zero. + +.. inst:: nearest x + + Round floating point round to integral, towards nearest with ties to even. + +.. inst:: sqrt x + + Floating point square root. + +.. inst:: a = fma x, y, z + + Floating point fused multiply-and-add. + + Computes :math:`a := xy+z` wihtout any intermediate rounding of the + product. + +Conversion operations +===================== + +.. inst:: a = bitcast x + + Reinterpret the bits in ``x`` as a different type. + + The input and output types must be storable to memory and of the same size. + A bitcast is equivalent to storing one type and loading the other type from + the same address. + +.. inst:: a = itrunc x +.. inst:: a = uext x +.. inst:: a = sext x +.. inst:: a = ftrunc x +.. inst:: a = fext x +.. inst:: a = cvt_ftou x +.. inst:: a = cvt_ftos x +.. inst:: a = cvt_utof x +.. inst:: a = cvt_stof x + From 152945f3d82fd70b8eafc916f3dc98c3c09da330 Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Tue, 19 Jan 2016 19:51:53 -0800 Subject: [PATCH 0005/3084] Emit list of todo items. --- cranelift/docs/index.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst index 137e7b1e84..668b055c97 100644 --- a/cranelift/docs/index.rst +++ b/cranelift/docs/index.rst @@ -15,3 +15,7 @@ Indices and tables * :ref:`modindex` * :ref:`search` +Todo list +========= + +.. todolist:: From fbb3174793f0738e140a97a7339394b6106f5994 Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Tue, 19 Jan 2016 20:28:33 -0800 Subject: [PATCH 0006/3084] Cretonne pygments lexer --- cranelift/docs/conf.py | 3 ++- cranelift/docs/cton_lexer.py | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 1 deletion(-) create mode 100644 cranelift/docs/cton_lexer.py diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py index 58cc740464..120d32011a 100644 --- a/cranelift/docs/conf.py +++ b/cranelift/docs/conf.py @@ -34,7 +34,8 @@ extensions = [ 'sphinx.ext.todo', 'sphinx.ext.mathjax', 'sphinx.ext.ifconfig', - 'cton_domain' + 'cton_domain', + 'cton_lexer', ] # Add any paths that contain templates here, relative to this directory. diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py new file mode 100644 index 0000000000..02e8945580 --- /dev/null +++ b/cranelift/docs/cton_lexer.py @@ -0,0 +1,36 @@ +# -*- coding: utf-8 -*- +# +# Pygments lexer for Cretonne. + +from pygments.lexer import RegexLexer, bygroups +from pygments.token import * + +class CretonneLexer(RegexLexer): + name = 'Cretonne' + aliases = ['cton'] + filenames = ['*.cton'] + + tokens = { + 'root': [ + (r';.*?$', Comment.Single), + (r'\b(function|entry)\b', Keyword), + (r'\b(align)\b', Name.Attribute), + (r'\b(v\d+)?(bool|i\d+|f32|f64)\b', Keyword.Type), + (r'\d+', Number.Integer), + (r'0[xX][0-9a-fA-F]+', Number.Hex), + (r'(v|ss|ebb)\d+', Name.Variable), + (r'(ebb)\d+', Name.Label), + (r'(=)( *)([a-z]\w*)', bygroups(Operator, Whitespace, Name.Function)), + (r'^( +)([a-z]\w*\b)(?! *[,=])', bygroups(Whitespace, Name.Function)), + (r'[a-z]\w*', Name), + (r'->|=|:', Operator), + (r'[{}(),.]', Punctuation), + (r'[ \t]+', Text), + ] + } + +def setup(app): + """Setup Sphinx extension.""" + app.add_lexer('cton', CretonneLexer()) + + return { 'version' : '0.1' } From a686c72ad3febe57486a9c29d5e6c94c0a3ff45a Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Thu, 21 Jan 2016 11:46:30 -0800 Subject: [PATCH 0007/3084] Add langref example --- cranelift/docs/langref.rst | 72 +++++++++++++++++++++++++++++++++++++- 1 file changed, 71 insertions(+), 1 deletion(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 31f79df0c2..364ea20f05 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -2,10 +2,80 @@ Cretonne Intermediate Language Reference **************************************** +.. default-domain:: cton +.. highlight:: cton + +The Cretonne intermediate language has two equivalent representations: an +*in-memory data structure* that the code generator library is using, and +a *text format* which is used for test cases and debug output. Files containing +Cretonne textual IL have the ``.cton`` filename extension. + +This reference uses the text format to describe IL semantics but glosses over +the details of the lexical and syntactic structure of the test format. + +Overall structure +================= + +Cretonne compiles functions independently. A ``.cton`` IL file may contain +multiple functions, and the programmatic API can create multiple function +handles at the same time, but the functions don't share any data or reference +each other directly. + +This is a C function that computes the average of an array of floats: + +.. code-block:: c + + float average(const float *array, size_t count) { + double sum = 0; + for (size_t i = 0; i < count; i++) + sum += array[i]; + return sum / count; + } + +Here it is compiled into Cretonne IL:: + + function average(i32, i32) -> f32 { + ; Preamble. + ss1 = local 8, align 4 + + entry ebb1(v1: i32, v2: i32): + v3 = fconst.f64 0.0 + stack_store v3, ss1 + brz v2, ebb3 ; Handle count == 0. + v4 = iconst.i32 0 + br ebb2(v4) + + ebb2(v5: i32): + ; Compute address of array element. + v6 = imul_imm v5, 4 + v7 = iadd v1, v6 + v8 = heap_load.f32 v7 ; array[i] + v9 = fext.f64 v8 + ; Add to accumulator in ss1. + v10 = stack_load.f64 ss1 + v11 = fadd v9, v10 + stack_store v11, ss1 + ; Increment loop counter. + v12 = iadd_imm v5, 1 + v13 = icmp ult v12, v2 + brnz v13, ebb2(v12) ; Loop backedge. + ; Compute average from sum. + v14 = stack_load.f64 ss1 + v15 = cvt_utof.f64 v2 + v16 = fdiv v14, v15 + v17 = ftrunc.f32 v16 + return v17 + + ebb3: + v100 = fconst.f32 0x7f800000 ; Inf + return v100 + } + + + Type system =========== -.. default-domain:: cton All SSA values have a type which determines the size and shape (for SIMD vectors) of the value. Many instructions are polymorphic -- they can operate on From 75544db19ab8dd024ec3833f69420f1f024a03cf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 21 Jan 2016 14:25:16 -0800 Subject: [PATCH 0008/3084] Update language reference. Add a glossary and explain the overall shape of a Cretonne function. --- cranelift/docs/example.c | 8 ++ cranelift/docs/example.cton | 31 ++++++++ cranelift/docs/langref.rst | 144 +++++++++++++++++++++++++----------- 3 files changed, 138 insertions(+), 45 deletions(-) create mode 100644 cranelift/docs/example.c create mode 100644 cranelift/docs/example.cton diff --git a/cranelift/docs/example.c b/cranelift/docs/example.c new file mode 100644 index 0000000000..0523123301 --- /dev/null +++ b/cranelift/docs/example.c @@ -0,0 +1,8 @@ +float +average(const float *array, size_t count) +{ + double sum = 0; + for (size_t i = 0; i < count; i++) + sum += array[i]; + return sum / count; +} diff --git a/cranelift/docs/example.cton b/cranelift/docs/example.cton new file mode 100644 index 0000000000..04c9e77fb4 --- /dev/null +++ b/cranelift/docs/example.cton @@ -0,0 +1,31 @@ +function average(i32, i32) -> f32 { + ss1 = local 8, align 4 ; Stack slot for ``sum``. + +entry ebb1(v1: i32, v2: i32): + v3 = fconst.f64 0.0 + stack_store v3, ss1 + brz v2, ebb3 ; Handle count == 0. + v4 = iconst.i32 0 + br ebb2(v4) + +ebb2(v5: i32): + v6 = imul_imm v5, 4 + v7 = iadd v1, v6 + v8 = heap_load.f32 v7 ; array[i] + v9 = fext.f64 v8 + v10 = stack_load.f64 ss1 + v11 = fadd v9, v10 + stack_store v11, ss1 + v12 = iadd_imm v5, 1 + v13 = icmp ult v12, v2 + brnz v13, ebb2(v12) ; Loop backedge. + v14 = stack_load.f64 ss1 + v15 = cvt_utof.f64 v2 + v16 = fdiv v14, v15 + v17 = ftrunc.f32 v16 + return v17 + +ebb3: + v100 = fconst.f32 0x7fc00000 ; 0/0 = NaN + return v100 +} diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 364ea20f05..aaff27a9f3 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -21,62 +21,53 @@ multiple functions, and the programmatic API can create multiple function handles at the same time, but the functions don't share any data or reference each other directly. -This is a C function that computes the average of an array of floats: +This is a simple C function that computes the average of an array of floats: -.. code-block:: c +.. literalinclude:: example.c + :language: c - float average(const float *array, size_t count) { - double sum = 0; - for (size_t i = 0; i < count; i++) - sum += array[i]; - return sum / count; - } +Here is the same function compiled into Cretonne IL: -Here it is compiled into Cretonne IL:: +.. literalinclude:: example.cton + :language: cton + :linenos: + :emphasize-lines: 2 - function average(i32, i32) -> f32 { - ; Preamble. - ss1 = local 8, align 4 +The first line of a function definition provides the function *name* and +the :term:`function signature` which declares the argument and return types. +Then follows the :term:`function preample` which declares a number of entities +that can be referenced inside the function. In the example above, the preample +declares a single local variable, ``ss1``. - entry ebb1(v1: i32, v2: i32): - v3 = fconst.f64 0.0 - stack_store v3, ss1 - brz v2, ebb3 ; Handle count == 0. - v4 = iconst.i32 0 - br ebb2(v4) +After the preample follows the :term:`function body` which consists of +:term:`extended basic block`\s, one of which is marked as the :term:`entry +block`. Every EBB ends with a :term:`terminator instruction`, and execution +can never fall through to the next EBB without an explicit branch. - ebb2(v5: i32): - ; Compute address of array element. - v6 = imul_imm v5, 4 - v7 = iadd v1, v6 - v8 = heap_load.f32 v7 ; array[i] - v9 = fext.f64 v8 - ; Add to accumulator in ss1. - v10 = stack_load.f64 ss1 - v11 = fadd v9, v10 - stack_store v11, ss1 - ; Increment loop counter. - v12 = iadd_imm v5, 1 - v13 = icmp ult v12, v2 - brnz v13, ebb2(v12) ; Loop backedge. - ; Compute average from sum. - v14 = stack_load.f64 ss1 - v15 = cvt_utof.f64 v2 - v16 = fdiv v14, v15 - v17 = ftrunc.f32 v16 - return v17 +Static single assignment form +----------------------------- - ebb3: - v100 = fconst.f32 0x7f800000 ; Inf - return v100 - } - +The instructions in the function body use and produce *values* in SSA form. This +means that every value is defined exactly once, and every use of a value must be +dominated by the definition. +Cretonne does not have phi instructions but uses *EBB arguments* instead. An EBB +can be defined with a list of typed arguments. Whenever control is transferred +to the EBB, values for the arguments must be provided. When entering a function, +the incoming function arguments are passed as arguments to the entry EBB. -Type system +Instructions define zero, one, or more result values. All SSA values are either +EBB arguments or instruction results. + +In the example above, the loop induction variable ``i`` is represented as three +SSA values: In the entry block, ``v4`` is the initial value. In the loop block +``ebb2``, the EBB argument ``v5`` represents the value of the induction +variable during each iteration. Finally, ``v12`` is computed as the induction +variable value for the next iteration. + +Value types =========== - All SSA values have a type which determines the size and shape (for SIMD vectors) of the value. Many instructions are polymorphic -- they can operate on different types. @@ -653,3 +644,66 @@ Conversion operations .. inst:: a = cvt_utof x .. inst:: a = cvt_stof x +Glossary +======== + +.. glossary:: + + function signature + A function signature describes how to call a function. It consists of: + + - The calling convention. + - The number of arguments and return values. (Functions can return + multiple values.) + - Type and flags of each argument. + - Type and flags of each return value. + + Not all function atributes are part of the signature. For example, a + function that never returns could be marked as ``noreturn``, but that + is not necessary to know when calling it, so it is just an attribute, + and not part of the signature. + + function preample + A list of declarations of entities that are used by the function body. + Some of the entities that can be declared in the preample are: + + - Local variables. + - Functions that are called directly. + - Function signatures for indirect function calls. + - Function flags and attributes that are not part of the signature. + + basic block + A maximal sequence of instructions that can only be entered from the + top, and that contains no branch or terminator instructions except for + the last instruction. + + extended basic block + EBB + A maximal sequence of instructions that can only be entered from the + top, and that contains no :term:`terminator instruction`s except for + the last one. An EBB can contain conditional branches that can fall + through to the following instructions in the block, but only the first + instruction in the EBB can be a branch target. + + The last instrution in an EBB must be a :term:`terminator instruction`, + so execion cannot flow through to the next EBB in the function. (But + there may be a branch to the next EBB.) + + Note that some textbooks define an EBB as a maximal *subtree* in the + control flow graph where only the root can be a join node. This + definition is not equivalent to Cretonne EBBs. + + terminator instruction + A control flow instruction that unconditionally directs the flow of + execution somewhere else. Execution never continues at the instruction + following a terminator instruction. + + The basic terminator instructions are :inst:`br`, :inst:`return`, and + :inst:`trap`. Conditional branches and instructions that trap + conditionally are not terminator instructions. + + entry block + The :term:`EBB` that is executed first in a function. Currently, a + Cretonne function must have exactly one entry block. The types of the + entry block arguments must match the types of arguments in the function + signature. From c228c0b3ae24c25c7e509070a7d9daa076715542 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 21 Jan 2016 16:39:45 -0800 Subject: [PATCH 0009/3084] Switch SIMD type spelling to i32x4. Add support for 'type variables' in type directives. --- cranelift/docs/cton_domain.py | 29 ++++++++++++- cranelift/docs/cton_lexer.py | 8 +++- cranelift/docs/langref.rst | 82 +++++++++++++++++++---------------- 3 files changed, 79 insertions(+), 40 deletions(-) diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index afa372921f..9f153f3a50 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -57,6 +57,33 @@ class CtonObject(ObjectDescription): self.indexnode['entries'].append(('single', indextext, targetname, '')) +# Type variables are indicated as %T. +typevar = re.compile('(\%[A-Z])') + +def parse_type(name, signode): + """ + Parse a type with embedded type vars and append to signode. + + Return a a string that can be compiled into a regular expression matching + the type. + """ + + re_str = '' + + for part in typevar.split(name): + if part == '': + continue + if len(part) == 2 and part[0] == '%': + # This is a type parameter. Don't display the %, use emphasis + # instead. + part = part[1] + signode += nodes.emphasis(part, part) + re_str += r'\w+' + else: + signode += addnodes.desc_name(part, part) + re_str += re.escape(part) + return re_str + class CtonType(CtonObject): """A Cretonne IL type description.""" @@ -68,7 +95,7 @@ class CtonType(CtonObject): """ name = sig.strip() - signode += addnodes.desc_name(name, name) + re_str = parse_type(name, signode) return name def get_index_text(self, name): diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py index 02e8945580..0567b748c7 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/cton_lexer.py @@ -15,10 +15,14 @@ class CretonneLexer(RegexLexer): (r';.*?$', Comment.Single), (r'\b(function|entry)\b', Keyword), (r'\b(align)\b', Name.Attribute), - (r'\b(v\d+)?(bool|i\d+|f32|f64)\b', Keyword.Type), + # Well known value types. + (r'\b(bool|i\d+|f32|f64)(x\d+)?\b', Keyword.Type), (r'\d+', Number.Integer), (r'0[xX][0-9a-fA-F]+', Number.Hex), - (r'(v|ss|ebb)\d+', Name.Variable), + # v = value + # ss = stack slot + (r'(v|ss)\d+', Name.Variable), + # ebb = extended basic block (r'(ebb)\d+', Name.Label), (r'(=)( *)([a-z]\w*)', bygroups(Operator, Whitespace, Name.Function)), (r'^( +)([a-z]\w*\b)(?! *[,=])', bygroups(Whitespace, Name.Function)), diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index aaff27a9f3..85f50ef32d 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -109,13 +109,13 @@ double-double formats. .. type:: f32 - A 32-bit floating point type represented in the IEEE 754 *Single precision* + A 32-bit floating point type represented in the IEEE 754 *single precision* format. This corresponds to the :c:type:`float` type in most C implementations. .. type:: f64 - A 64-bit floating point type represented in the IEEE 754 *Double precision* + A 64-bit floating point type represented in the IEEE 754 *double precision* format. This corresponds to the :c:type:`double` type in most C implementations. @@ -126,40 +126,42 @@ A SIMD vector type represents a vector of values from one of the scalar types (:type:`bool`, integer, and floating point). Each scalar value in a SIMD type is called a *lane*. The number of lanes must be a power of two in the range 2-256. -.. type:: vNiB +.. type:: i%Bx%N - A SIMD vector of integers. The lane type :type:`iB` must be one of the - integer types :type:`i8` ... :type:`i64`. + A SIMD vector of integers. The lane type :type:`iB` is one of the integer + types :type:`i8` ... :type:`i64`. - Some concrete integer vector types are :type:`v4i32`, :type:`v8i64`, and - :type:`v4i16`. + Some concrete integer vector types are :type:`i32x4`, :type:`i64x8`, and + :type:`i16x4`. The size of a SIMD integer vector in memory is :math:`N B\over 8` bytes. -.. type:: vNf32 +.. type:: f32x%N A SIMD vector of single precision floating point numbers. - Some concrete :type:`f32` vector types are: :type:`v2f32`, :type:`v4f32`, - and :type:`v8f32`. + Some concrete :type:`f32` vector types are: :type:`f32x2`, :type:`f32x4`, + and :type:`f32x8`. The size of a :type:`f32` vector in memory is :math:`4N` bytes. -.. type:: vNf64 +.. type:: f64x%N A SIMD vector of double precision floating point numbers. - Some concrete :type:`f64` vector types are: :type:`v2f64`, :type:`v4f64`, - and :type:`v8f64`. + Some concrete :type:`f64` vector types are: :type:`f64x2`, :type:`f64x4`, + and :type:`f64x8`. The size of a :type:`f64` vector in memory is :math:`8N` bytes. -.. type:: vNbool +.. type:: boolx%N A boolean SIMD vector. - Like the :type:`bool` type, a boolean vector cannot be stored in memory. It - can only be used for ephemeral SSA values. + Boolean vectors are used when comparing SIMD vectors. For example, + comparing two :type:`i32x4` values would produce a :type:`boolx4` result. + + Like the :type:`bool` type, a boolean vector cannot be stored in memory. Instructions ============ @@ -175,6 +177,9 @@ Control flow instructions EBB arguments. The number and types of arguments must match the destination EBB. + :arg EBB: Destination extended basic block. + :result: None. This is a terminator instruction. + .. inst:: brz x, EBB(args...) Branch when zero. @@ -182,8 +187,9 @@ Control flow instructions If ``x`` is a :type:`bool` value, take the branch when ``x`` is false. If ``x`` is an integer value, take the branch when ``x = 0``. - :param iN/bool x: Value to test. - :param EBB: Destination extended basic block. + :arg iN / bool x: Value to test. + :arg EBB: Destination extended basic block. + :result: None. .. inst:: brnz x, EBB(args...) @@ -192,15 +198,13 @@ Control flow instructions If ``x`` is a :type:`bool` value, take the branch when ``x`` is true. If ``x`` is an integer value, take the branch when ``x != 0``. - :param iN/bool x: Value to test. - :param EBB: Destination extended basic block. + :arg iN / bool x: Value to test. + :arg EBB: Destination extended basic block. + :result: None. Special operations ================== -Most operations are easily classified as arithmetic or control flow. These -instructions are not so easily classified. - .. inst:: a = iconst n Integer constant. @@ -217,13 +221,13 @@ instructions are not so easily classified. Conditional select. - :param c bool: Controlling flag. - :param x: Value to return when ``c`` is true. - :param y: Value to return when ``c`` is false. Must be same type as ``x``. - :rtype: Same type as ``x`` and ``y``. + :arg bool c: Controlling flag. + :arg T x: Value to return when ``c`` is true. + :arg T y: Value to return when ``c`` is false. Must be same type as ``x``. + :rtype: T. Same type as ``x`` and ``y``. - This instruction selects whole values. Use :inst:`vselect` for - lane-wise selection. + This instruction selects whole values. Use :inst:`vselect` for lane-wise + selection. Vector operations ================= @@ -235,7 +239,7 @@ Vector operations Select lanes from ``x`` or ``y`` controlled by the lanes of the boolean vector ``c``. - :arg vNbool c: Controlling flag vector. + :arg boolx%N c: Controlling flag vector. :arg x: Vector with lanes selected by the true lanes of ``c``. Must be a vector type with the same number of lanes as ``c``. :arg y: Vector with lanes selected by the false lanes of ``c``. @@ -277,7 +281,7 @@ Integer operations :param cond: Condition code determining how ``x`` and ``y`` are compared. :param x, y: Integer scalar or vector values of the same type. - :rtype: :type:`bool` or :type:`vNbool` with the same number of lanes as + :rtype: :type:`bool` or :type:`boolxN` with the same number of lanes as ``x`` and ``y``. The condition code determines if the operands are interpreted as signed or @@ -385,25 +389,25 @@ Bitwise operations Bitwise and. - :rtype: bool, iB, vNiB, vNfB? + :rtype: bool, iB, iBxN, fBxN? .. inst:: a = or x, y Bitwise or. - :rtype: bool, iB, vNiB, vNfB? + :rtype: bool, iB, iBxN, fBxN? .. inst:: a = xor x, y Bitwise xor. - :rtype: bool, iB, vNiB, vNfB? + :rtype: bool, iB, iBxN, fBxN? .. inst:: a = not x Bitwise not. - :rtype: bool, iB, vNiB, vNfB? + :rtype: bool, iB, iBxN, fBxN? .. todo:: Redundant bitwise operators. @@ -538,7 +542,7 @@ Floating point operations :param cond: Condition code determining how ``x`` and ``y`` are compared. :param x, y: Floating point scalar or vector values of the same type. - :rtype: :type:`bool` or :type:`vNbool` with the same number of lanes as + :rtype: :type:`bool` or :type:`boolxN` with the same number of lanes as ``x`` and ``y``. An 'ordered' condition code yields ``false`` if either operand is Nan. @@ -672,6 +676,10 @@ Glossary - Function signatures for indirect function calls. - Function flags and attributes that are not part of the signature. + function body + The extended basic blocks which contain all the executable code in a + function. The function body follows the function preample. + basic block A maximal sequence of instructions that can only be entered from the top, and that contains no branch or terminator instructions except for @@ -680,7 +688,7 @@ Glossary extended basic block EBB A maximal sequence of instructions that can only be entered from the - top, and that contains no :term:`terminator instruction`s except for + top, and that contains no :term:`terminator instruction`\s except for the last one. An EBB can contain conditional branches that can fall through to the following instructions in the block, but only the first instruction in the EBB can be a branch target. From ca02df9ce2d797f2912eb90cfaad611081b1464c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 21 Jan 2016 17:15:20 -0800 Subject: [PATCH 0010/3084] Document control flow instructions. --- cranelift/docs/langref.rst | 85 ++++++++++++++++++++++++++++++++------ 1 file changed, 73 insertions(+), 12 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 85f50ef32d..70602f232f 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -163,12 +163,9 @@ called a *lane*. The number of lanes must be a power of two in the range 2-256. Like the :type:`bool` type, a boolean vector cannot be stored in memory. -Instructions +Control flow ============ -Control flow instructions -------------------------- - .. inst:: br EBB(args...) Branch. @@ -187,7 +184,7 @@ Control flow instructions If ``x`` is a :type:`bool` value, take the branch when ``x`` is false. If ``x`` is an integer value, take the branch when ``x = 0``. - :arg iN / bool x: Value to test. + :arg iN/bool x: Value to test. :arg EBB: Destination extended basic block. :result: None. @@ -198,12 +195,76 @@ Control flow instructions If ``x`` is a :type:`bool` value, take the branch when ``x`` is true. If ``x`` is an integer value, take the branch when ``x != 0``. - :arg iN / bool x: Value to test. + :arg iN/bool x: Value to test. :arg EBB: Destination extended basic block. :result: None. +.. inst:: br_table x, JT + + Jump table branch. + + Use ``x`` as an index into the jump table ``JT``. If a jump table entry is + found, branch to the corresponding EBB. If no entry was found fall through + to the next instruction. + + Note that this branch instruction can't pass arguments to the targeted + blocks. Split critical edges as needed to work around this. + + :arg iN x: Integer index into jump table. + :arg JT: Jump table which was declared in the preample. + :result: None. + +.. inst:: return args... + + Return from function. + + Unconditionally transfer control to the calling function, passing the + provided return values. + + :arg args: Return values. The list of retur values must match the list if + return value types in the function signature. + :result: None. This is a terminator instruction. + +.. inst:: trap + + Terminate execution. + + :result: None. This is a terminator instruction. + +.. inst:: trapz x + + Trap when zero. + + if ``x`` is non-zero, execution continues at the following instruction. + + :arg iN/bool x: Value to test. + :result: None. + +.. inst:: trapnz x + + Trap when non-zero. + + if ``x`` is zero, execution continues at the following instruction. + + :arg iN/bool x: Value to test. + :result: None. + +Function calls +============== + +A function call needs a target function and a :term:`function signature`. The +target function may be determined dynamically at runtime, but the signature +must be known when the function call is compiled. + + + +Operations +========== + + + Special operations -================== +------------------ .. inst:: a = iconst n @@ -230,7 +291,7 @@ Special operations selection. Vector operations -================= +----------------- .. inst:: a = vselect c, x, y @@ -273,7 +334,7 @@ Vector operations indicate a valid lane index for the type of ``x``. Integer operations -================== +------------------ .. inst:: a = icmp cond, x, y @@ -383,7 +444,7 @@ Integer operations good start. Bitwise operations -================== +------------------ .. inst:: a = and x, y @@ -534,7 +595,7 @@ Bitwise operations Floating point operations -========================= +------------------------- .. inst:: a = fcmp cond, x, y @@ -628,7 +689,7 @@ Floating point operations product. Conversion operations -===================== +--------------------- .. inst:: a = bitcast x From e238df3e7c436f7fdbe3d3fd27122cf2025ad01a Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Fri, 22 Jan 2016 10:31:24 -0800 Subject: [PATCH 0011/3084] Add hexadecimal numbers to the lexer. Also decimal and hexadecimal exponential notation for float constants. --- cranelift/docs/cton_lexer.py | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py index 0567b748c7..0df2e6208a 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/cton_lexer.py @@ -2,9 +2,12 @@ # # Pygments lexer for Cretonne. -from pygments.lexer import RegexLexer, bygroups +from pygments.lexer import RegexLexer, bygroups, words from pygments.token import * +def keywords(*args): + return words(args, prefix=r'\b', suffix=r'\b') + class CretonneLexer(RegexLexer): name = 'Cretonne' aliases = ['cton'] @@ -13,19 +16,30 @@ class CretonneLexer(RegexLexer): tokens = { 'root': [ (r';.*?$', Comment.Single), - (r'\b(function|entry)\b', Keyword), - (r'\b(align)\b', Name.Attribute), + # Strings are in double quotes, support \xx escapes only. + (r'"([^"\\]+|\\[0-9a-fA-F]{2})*"', String), + # A naked function name following 'function' is also a string. + (r'\b(function)([ \t]+)(\w+)\b', bygroups(Keyword, Whitespace, String.Symbol)), + # Numbers. + (r'[-+]?0[xX][0-9a-fA-F]+', Number.Hex), + (r'[-+]?0[xX][0-9a-fA-F]*\.[0-9a-fA-F]*([pP]\d+)?', Number.Hex), + (r'[-+]?\d+\.\d+([eE]\d+)?', Number.Float), + (r'[-+]?\d+', Number.Integer), + # Reserved words. + (keywords('function', 'entry'), Keyword), + # Known attributes. + (keywords('align'), Name.Attribute), # Well known value types. (r'\b(bool|i\d+|f32|f64)(x\d+)?\b', Keyword.Type), - (r'\d+', Number.Integer), - (r'0[xX][0-9a-fA-F]+', Number.Hex), # v = value # ss = stack slot (r'(v|ss)\d+', Name.Variable), # ebb = extended basic block (r'(ebb)\d+', Name.Label), + # Match instruction names in context. (r'(=)( *)([a-z]\w*)', bygroups(Operator, Whitespace, Name.Function)), (r'^( +)([a-z]\w*\b)(?! *[,=])', bygroups(Whitespace, Name.Function)), + # Other names: results and arguments (r'[a-z]\w*', Name), (r'->|=|:', Operator), (r'[{}(),.]', Punctuation), From 05ecfc4149fcb6df74649f015c7544469714b98a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Jan 2016 12:13:38 -0800 Subject: [PATCH 0012/3084] Clarify local SSA form. Rename 'local' to 'stack_slot'. --- cranelift/docs/example.cton | 10 ++++----- cranelift/docs/langref.rst | 42 ++++++++++++++++++++++++++++++++----- 2 files changed, 42 insertions(+), 10 deletions(-) diff --git a/cranelift/docs/example.cton b/cranelift/docs/example.cton index 04c9e77fb4..c0001fe7c6 100644 --- a/cranelift/docs/example.cton +++ b/cranelift/docs/example.cton @@ -1,24 +1,24 @@ function average(i32, i32) -> f32 { - ss1 = local 8, align 4 ; Stack slot for ``sum``. + ss1 = stack_slot 8, align 4 ; Stack slot for ``sum``. entry ebb1(v1: i32, v2: i32): v3 = fconst.f64 0.0 stack_store v3, ss1 - brz v2, ebb3 ; Handle count == 0. + brz v2, ebb3 ; Handle count == 0. v4 = iconst.i32 0 br ebb2(v4) ebb2(v5: i32): v6 = imul_imm v5, 4 v7 = iadd v1, v6 - v8 = heap_load.f32 v7 ; array[i] + v8 = heap_load.f32 v7 ; array[i] v9 = fext.f64 v8 v10 = stack_load.f64 ss1 v11 = fadd v9, v10 stack_store v11, ss1 v12 = iadd_imm v5, 1 v13 = icmp ult v12, v2 - brnz v13, ebb2(v12) ; Loop backedge. + brnz v13, ebb2(v12) ; Loop backedge. v14 = stack_load.f64 ss1 v15 = cvt_utof.f64 v2 v16 = fdiv v14, v15 @@ -26,6 +26,6 @@ ebb2(v5: i32): return v17 ebb3: - v100 = fconst.f32 0x7fc00000 ; 0/0 = NaN + v100 = fconst.f32 0x7fc00000 ; 0/0 = NaN return v100 } diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 70602f232f..6a9bcc60eb 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -11,7 +11,8 @@ a *text format* which is used for test cases and debug output. Files containing Cretonne textual IL have the ``.cton`` filename extension. This reference uses the text format to describe IL semantics but glosses over -the details of the lexical and syntactic structure of the test format. +the finer details of the lexical and syntactic structure of the format. + Overall structure ================= @@ -41,8 +42,8 @@ declares a single local variable, ``ss1``. After the preample follows the :term:`function body` which consists of :term:`extended basic block`\s, one of which is marked as the :term:`entry -block`. Every EBB ends with a :term:`terminator instruction`, and execution -can never fall through to the next EBB without an explicit branch. +block`. Every EBB ends with a :term:`terminator instruction`, so execution can +never fall through to the next EBB without an explicit branch. Static single assignment form ----------------------------- @@ -65,6 +66,20 @@ SSA values: In the entry block, ``v4`` is the initial value. In the loop block variable during each iteration. Finally, ``v12`` is computed as the induction variable value for the next iteration. +It can be difficult to generate correct SSA form if the program being converted +into Cretonne IL contains multiple assignments to the same variables. Such +variables can be presented to Cretonne as :term:`stack slot`\s instead. Stack +slots are accessed with the :inst:`stack_store` and :inst:`stack_load` +instructions which behave more like variable accesses in a typical programming +language. Cretonne can perform the necessary dataflow analysis to convert stack +slots to SSA form. + +If all values are only used in the same EBB where they are defined, the +function is said to be in :term:`local SSA form`. It is much faster for +Cretonne to verify the correctness of a function in local SSA form since no +complicated control flow analysis is required. + + Value types =========== @@ -180,7 +195,7 @@ Control flow .. inst:: brz x, EBB(args...) Branch when zero. - + If ``x`` is a :type:`bool` value, take the branch when ``x`` is false. If ``x`` is an integer value, take the branch when ``x = 0``. @@ -191,7 +206,7 @@ Control flow .. inst:: brnz x, EBB(args...) Branch when non-zero. - + If ``x`` is a :type:`bool` value, take the branch when ``x`` is true. If ``x`` is an integer value, take the branch when ``x != 0``. @@ -776,3 +791,20 @@ Glossary Cretonne function must have exactly one entry block. The types of the entry block arguments must match the types of arguments in the function signature. + + stack slot + A fixed size memory allocation in the current function's activation + frame. Also called a local variable. + + local SSA form + A restricted version of SSA form where all values are defined and used + in the same EBB. A function is in local SSA form iff it is in SSA form + and: + + - No branches pass arguments to their target EBB. + - Only the entry EBB may have arguments. + + This also implies that there are no branches to the entry EBB. + + Local SSA form is easy to generate and fast to verify. It passes data + between EBBs by using stack slots. From cdc0047ee12e1f0e3f6412e3ecaaea45eb123959 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Jan 2016 13:08:18 -0800 Subject: [PATCH 0013/3084] Expand on control flow and direct function calls. Define the syntax for function signatures. --- cranelift/docs/cton_lexer.py | 2 +- cranelift/docs/langref.rst | 142 ++++++++++++++++++++++++++++++++--- 2 files changed, 131 insertions(+), 13 deletions(-) diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py index 0df2e6208a..4c309d779b 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/cton_lexer.py @@ -28,7 +28,7 @@ class CretonneLexer(RegexLexer): # Reserved words. (keywords('function', 'entry'), Keyword), # Known attributes. - (keywords('align'), Name.Attribute), + (keywords('align', 'uext', 'sext', 'inreg'), Name.Attribute), # Well known value types. (r'\b(bool|i\d+|f32|f64)(x\d+)?\b', Keyword.Type), # v = value diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 6a9bcc60eb..b3f8b14ea1 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -178,9 +178,27 @@ called a *lane*. The number of lanes must be a power of two in the range 2-256. Like the :type:`bool` type, a boolean vector cannot be stored in memory. +Pseudo-types +------------ + +These are not concrete types, but convenient names uses to refer to real types +in this reference. + +.. type:: iPtr + + A Pointer-sized integer. + + This is either :type:`i32`, or :type:`i64`, depending on whether the target + platform has 32-bit or 64-bit pointers. + Control flow ============ +Branches transfer control to a new EBB and provide values for the target EBB's +arguments, if it has any. Conditional branches only take the branch if their +condition is satisfied, otherwise execution continues at the following +instruction in the EBB. + .. inst:: br EBB(args...) Branch. @@ -190,6 +208,7 @@ Control flow EBB. :arg EBB: Destination extended basic block. + :arg args...: Zero or more arguments passed to EBB. :result: None. This is a terminator instruction. .. inst:: brz x, EBB(args...) @@ -201,6 +220,7 @@ Control flow :arg iN/bool x: Value to test. :arg EBB: Destination extended basic block. + :arg args...: Arguments passed to EBB. :result: None. .. inst:: brnz x, EBB(args...) @@ -212,15 +232,16 @@ Control flow :arg iN/bool x: Value to test. :arg EBB: Destination extended basic block. + :arg args...: Zero or more arguments passed to EBB. :result: None. .. inst:: br_table x, JT Jump table branch. - Use ``x`` as an index into the jump table ``JT``. If a jump table entry is - found, branch to the corresponding EBB. If no entry was found fall through - to the next instruction. + Use ``x`` as an unsigned index into the jump table ``JT``. If a jump table + entry is found, branch to the corresponding EBB. If no entry was found fall + through to the next instruction. Note that this branch instruction can't pass arguments to the targeted blocks. Split critical edges as needed to work around this. @@ -229,20 +250,31 @@ Control flow :arg JT: Jump table which was declared in the preample. :result: None. -.. inst:: return args... +.. inst:: JT = jump_table EBB0, EBB1, ..., EBBn - Return from function. + Declare a jump table in the :term:`function preample`. - Unconditionally transfer control to the calling function, passing the - provided return values. + This declares a jump table for use by the :inst:`br_table` indirect branch + instruction. Entries in the table are either EBB names, or ``0`` which + indicates an absent entry. - :arg args: Return values. The list of retur values must match the list if - return value types in the function signature. - :result: None. This is a terminator instruction. + The EBBs listed must belong to the current function, and they can't have + any arguments. + + :arg EBB0: Target EBB when ``x = 0``. + :arg EBB1: Target EBB when ``x = 1``. + :arg EBBn: Target EBB when ``x = n``. + :result: A jump table identifier. (Not an SSA value). + +Traps stop the program because something went wrong. The exact behavior depends +on the target instruction set architecture and operating system. There are +explicit trap instructions defined below, but some instructions may also cause +traps for certain input value. For example, :inst:`udiv` traps when the divisor +is zero. .. inst:: trap - Terminate execution. + Terminate execution unconditionally. :result: None. This is a terminator instruction. @@ -264,12 +296,98 @@ Control flow :arg iN/bool x: Value to test. :result: None. + Function calls ============== A function call needs a target function and a :term:`function signature`. The target function may be determined dynamically at runtime, but the signature -must be known when the function call is compiled. +must be known when the function call is compiled. The function signature +describes how to call the function, including arguments, return values, and the +calling convention: + +.. productionlist:: + signature : "(" [arglist] ")" ["->" retlist] [call_conv] + arglist : arg + : arglist "," arg + retlist : arglist + arg : type + : arg flag + flag : "uext" | "sext" | "inreg" + callconv : `string` + +Arguments and return values have flags whose meaning is mostly target +dependent. They make it possible to call native functions on the target +platform. When calling other Cretonne functions, the flags are not necessary. + +Functions that are called directly must be declared in the :term:`function +preample`: + +.. inst:: F = function NAME signature + + Declare a function so it can be called directly. + + :arg NAME: Name of the function, passed to the linker for resolution. + :arg signature: Function signature. See below. + :result F: A function identifier that can be used with :inst:`call`. + +.. inst:: a, b, ... = call F(args...) + + Direct function call. + + :arg F: Function identifier to call, declared by :inst:`function`. + :arg args...: Function arguments matching the signature of F. + :result a,b,...: Return values matching the signature of F. + +.. inst:: return args... + + Return from function. + + Unconditionally transfer control to the calling function, passing the + provided return values. + + :arg args: Return values. The list of return values must match the list of + return value types in the function signature. + :result: None. This is a terminator instruction. + +This simple example illustrates direct function calls and signatures:: + + function gcd(i32 uext, i32 uext) -> i32 uext "C" { + f1 = function divmod(i32 uext, i32 uext) -> i32 uext, i32 uext + + entry ebb1(v1: i32, v2: i32): + brz v2, ebb2 + v3, v4 = call f1(v1, v2) + br ebb1(v2, v4) + + ebb2: + return v1 + } + +Indirect function calls use a signature declared in the preample. + +.. inst:: SIG = signature signature + + Declare a function signature for use with indirect calls. + + :arg signature: Function signature. See :token:`signature`. + :result SIG: A signature identifier. + +.. inst:: a, b, ... = call_indirect SIG, x(args...) + + Indirect function call. + + :arg SIG: A function signature identifier declared with :inst:`signature`. + :arg iPtr x: The address of the function to call. + :arg args...: Function arguments matching SIG. + :result a,b,...: Return values matching SIG. + +.. todo:: Define safe indirect function calls. + + The :inst:`call_indirect` instruction is dangerous to use in a sandboxed + environment since it is not easy to verify the callee address. + We need a table-driven indirect call instruction, similar to + :inst:`br_table`. From 4bd4efaf67cd553f2f802cff035563f5bbacd46e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Jan 2016 15:20:10 -0800 Subject: [PATCH 0014/3084] Load, store, local variables. --- cranelift/docs/cton_domain.py | 3 +- cranelift/docs/cton_lexer.py | 4 +- cranelift/docs/langref.rst | 124 ++++++++++++++++++++++++++++++++++ 3 files changed, 128 insertions(+), 3 deletions(-) diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index 9f153f3a50..e1dc08a67b 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -19,7 +19,7 @@ from sphinx.directives import ObjectDescription from sphinx.domains import Domain, ObjType from sphinx.locale import l_, _ from sphinx.roles import XRefRole -from sphinx.util.docfields import Field, TypedField +from sphinx.util.docfields import Field, GroupedField, TypedField from sphinx.util.nodes import make_refnode class CtonObject(ObjectDescription): @@ -120,6 +120,7 @@ class CtonInst(CtonObject): TypedField('result', label=l_('Results'), names=('out', 'result'), typerolename='type', typenames=('type',)), + GroupedField('flag', names=('flag',), label=l_('Flags')), Field('resulttype', label=l_('Result type'), has_arg=False, names=('rtype',)), ] diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py index 4c309d779b..33f769dfc6 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/cton_lexer.py @@ -28,7 +28,7 @@ class CretonneLexer(RegexLexer): # Reserved words. (keywords('function', 'entry'), Keyword), # Known attributes. - (keywords('align', 'uext', 'sext', 'inreg'), Name.Attribute), + (keywords('align', 'aligntrap', 'uext', 'sext', 'inreg'), Name.Attribute), # Well known value types. (r'\b(bool|i\d+|f32|f64)(x\d+)?\b', Keyword.Type), # v = value @@ -38,7 +38,7 @@ class CretonneLexer(RegexLexer): (r'(ebb)\d+', Name.Label), # Match instruction names in context. (r'(=)( *)([a-z]\w*)', bygroups(Operator, Whitespace, Name.Function)), - (r'^( +)([a-z]\w*\b)(?! *[,=])', bygroups(Whitespace, Name.Function)), + (r'^( *)([a-z]\w*\b)(?! *[,=])', bygroups(Whitespace, Name.Function)), # Other names: results and arguments (r'[a-z]\w*', Name), (r'->|=|:', Operator), diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index b3f8b14ea1..6d64b42f03 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -390,6 +390,130 @@ Indirect function calls use a signature declared in the preample. :inst:`br_table`. +Memory +====== + +Cretonne provides fully general :inst:`load` and :inst:`store` instructions for +accessing memory. However, it can be very complicated to verify the safety of +general loads and stores when compiling code for a sandboxed environment, so +Cretonne also provides more restricted memory operations that are always safe. + +.. inst:: a = load p, Offset, Flags... + + Load from memory at ``p + Offset``. + + This is a polymorphic instruction that can load any value type which has a + memory representation (i.e., everything except :type:`bool` and boolean + vectors). + + :arg iPtr p: Base address. + :arg Offset: Immediate signed offset. + :flag align(N): Expected alignment of ``p + Offset``. Power of two. + :flag aligntrap: Always trap if the memory access is misaligned. + :result T a: Loaded value. + +.. inst:: store x, p, Offset, Flags... + + Store ``x`` to memory at ``p + Offset``. + + This is a polymorphic instruction that can store any value type with a + memory representation. + + :arg T x: Value to store. + :arg iPtr p: Base address. + :arg Offset: Immediate signed offset. + :flag align(N): Expected alignment of ``p + Offset``. Power of two. + :flag aligntrap: Always trap if the memory access is misaligned. + +Loads and stores are *misaligned* if the resultant address is not a multiple of +the expected alignment. Depending on the target architecture, misaligned memory +accesses may trap, or they may work. Sometimes, operating systems catch +alignment traps and emulate the misaligned memory access. + +On target architectures like x86 that don't check alignment, Cretonne expands +the aligntrap flag into a conditional trap instruction:: + + v5 = load.i32 v1, 4, align(4), aligntrap + ; Becomes: + v10 = and_imm v1, 3 + trapnz v10 + v5 = load.i32 v1, 4 + + +Local variables +--------------- + +One set of restricted memory operations access the current function's stack +frame. The stack frame is divided into fixed-size stack slots that are +allocated in the :term:`function preample`. Stack slots are not typed, they +simply represent a contiguous sequence of bytes in the stack frame. + +.. inst:: SS = stack_slot Bytes, Flags... + + Allocate a stack slot in the preample. + + If no alignment is specified, Cretonne will pick an appropriate alignment + for the stack slot based on its size and access patterns. + + :arg Bytes: Stack slot size on bytes. + :flag align(N): Request at least N bytes alignment. + :result SS: Stack slot index. + +.. inst:: a = stack_load SS, Offset + + Load a value from a stack slot at the constant offset. + + This is a polymorphic instruction that can load any value type which has a + memory representation. + + The offset is an immediate constant, not an SSA value. The memory access + cannot go out of bounds, i.e. ``sizeof(a) + Offset <= sizeof(SS)``. + + :arg SS: Stack slot declared with :inst:`stack_slot`. + :arg Offset: Immediate non-negative offset. + :result T a: Value loaded. + +.. inst:: stack_store x, SS, Offset + + Store a value to a stack slot at a constant offset. + + This is a polymorphic instruction that can store any value type with a + memory representation. + + The offset is an immediate constant, not an SSA value. The memory access + cannot go out of bounds, i.e. ``sizeof(a) + Offset <= sizeof(SS)``. + + :arg T x: Value to be stored. + :arg SS: Stack slot declared with :inst:`stack_slot`. + :arg Offset: Immediate non-negative offset. + +The dedicated stack access instructions are easy ofr the compiler to reason +about because stack slots and offsets are fixed at compile time. For example, +the alignment of these stack memory accesses can be inferred from the offsets +and stack slot alignments. + +It can be necessary to escape from the safety of the restricted instructions by +taking the address of a stack slot. + +.. inst:: a = stack_addr SS, Offset + + Get the address of a stack slot. + + Compute the absolute address of a byte in a stack slot. The offset must + refer to a byte inside the stack slot: ``0 <= Offset < sizeof(SS)``. + + :arg SS: Stack slot declared with :inst:`stack_slot`. + :arg Offset: Immediate non-negative offset. + :result iPtr a: Address. + +The :inst:`stack_addr` instruction can be used to macro-expand the stack access +instructions before instruction selection:: + + v1 = stack_load.f64 ss3, 16 + ; Expands to: + v9 = stack_addr ss3, 16 + v1 = load.f64 v9 + Operations ========== From 2aabcd5cffa61d7a970aa9a681bf1ef053d7f003 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Jan 2016 16:48:11 -0800 Subject: [PATCH 0015/3084] Document heaps. --- cranelift/docs/langref.rst | 87 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 6d64b42f03..d2896722c4 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -191,6 +191,10 @@ in this reference. This is either :type:`i32`, or :type:`i64`, depending on whether the target platform has 32-bit or 64-bit pointers. +.. type:: iN + + Any of the scalar integer types :type:`i8` -- :type:`i64`. + Control flow ============ @@ -514,6 +518,89 @@ instructions before instruction selection:: v9 = stack_addr ss3, 16 v1 = load.f64 v9 +Heaps +----- + +Code compiled from WebAssembly or asm.js runs in a sandbox where it can't access +all process memory. Instead, it is given a small set of memory areas to work +in, and all accesses are bounds checked. Cretonne models this through the +concept of *heaps*. + +A heap is declared in the function preample and can be accessed with restricted +instructions that trap on out-of-bounds accesses. Heap addresses can be smaller +than the native pointer size, for example unsigned :type:`i32` offsets on a +64-bit architecture. + +.. inst:: H = heap Name + + Declare a heap in the function preample. + + This doesn't allocate memory, it just retrieves a handle to a sandbox from + the runtime environment. + + :arg Name: String identifying the heap in the runtime environment. + :result H: Heap identifier. + +.. inst:: a = heap_load H, p, Offset + + Load a value at the address ``p + Offset`` in the heap H. + + Trap if the heap access would be out of bounds. + + :arg H: Heap identifier created by :inst:`heap`. + :arg iN p: Unsigned base address in heap. + :arg Offset: Immediate signed offset. + :flag align(N): Expected alignment of ``p + Offset``. Power of two. + :flag aligntrap: Always trap if the memory access is misaligned. + :result T a: Loaded value. + +.. inst:: a = heap_store H, x, p, Offset + + Store a value at the address ``p + Offset`` in the heap H. + + Trap if the heap access would be out of bounds. + + :arg H: Heap indetifier created by :inst:`heap`. + :arg T x: Value to be stored. + :arg iN p: Unsigned base address in heap. + :arg Offset: Immediate signed offset. + :flag align(N): Expected alignment of ``p + Offset``. Power of two. + :flag aligntrap: Always trap if the memory access is misaligned. + +When optimizing heap accesses, Cretonne may separate the heap bounds checking +and address computations from the memory accesses. + +.. inst:: a = heap_addr H, p, Size + + Bounds check and compute absolute address of heap memory. + + Verify that the address range ``p .. p + Size - 1`` is valid in the heap H, + and trap if not. + + Convert the heap-relative address in ``p`` to a real absolute address and + return it. + + :arg H: Heap identifier created by :inst:`heap`. + :arg iN p: Unsigned base address in heap. + :arg Size: Immediate unsigned byte count for range to verify. + :result iPtr a: Absolute address corresponding to ``p``. + +A small example using heaps:: + + function vdup(i32, i32) { + h1 = heap "main" + + entry ebb1(v1: i32, v2: i32): + v3 = heap_load.i32x4 h1, v1, 0 + v4 = heap_addr h1, v2, 32 ; Shared range check for two stores. + store v3, v4, 0 + store v3, v4, 16 + return + } + +The final expansion of the :inst:`heap_addr` range check and address conversion +depends on the runtime environment. + Operations ========== From b23b049140207d9b37438aa777a4e44427b5cca7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Jan 2016 17:30:30 -0800 Subject: [PATCH 0016/3084] Clean up the list of operations somewhat. --- cranelift/docs/langref.rst | 184 ++++++++++++++++++++++++------------- 1 file changed, 122 insertions(+), 62 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index d2896722c4..3aad746bfe 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -605,23 +605,32 @@ depends on the runtime environment. Operations ========== +The remaining instruction set is mostly arithmetic. +A few instructions have variants that take immediate operands (e.g., +:inst:`and` / :inst:`and_imm`), but in general an instruction is required to +load a constant into an SSA value. -Special operations ------------------- - -.. inst:: a = iconst n +.. inst:: a = iconst N Integer constant. -.. inst:: a = fconst n + Create a scalar integer SSA value with an immediate constant value, or an + integer vector where all the lanes have the same value. + +.. inst:: a = fconst N Floating point constant. -.. inst:: a = vconst n + Create a :type:`f32` or :type:`f64` SSA value with an immediate constant + value, or a floating point vector where all the lanes have the same value. + +.. inst:: a = vconst N Vector constant (floating point or integer). + Create a SIMD vector value where the lanes don't have to be identical. + .. inst:: a = select c, x, y Conditional select. @@ -680,12 +689,13 @@ Vector operations Integer operations ------------------ -.. inst:: a = icmp cond, x, y +.. inst:: a = icmp Cond, x, y Integer comparison. - :param cond: Condition code determining how ``x`` and ``y`` are compared. - :param x, y: Integer scalar or vector values of the same type. + :arg Cond: Condition code determining how ``x`` and ``y`` are compared. + :arg x: First value to compare. + :arg y: Second value to compare. :rtype: :type:`bool` or :type:`boolxN` with the same number of lanes as ``x`` and ``y``. @@ -708,29 +718,59 @@ Integer operations Wrapping integer addition: :math:`a := x + y \pmod{2^B}`. This instruction does not depend on the signed/unsigned interpretation of the operands. + Polymorphic over all integer types (vector and scalar). + +.. inst:: a = iadd_imm x, Imm + + Add immediate integer. + + Same as :inst:`iadd`, but one operand is an immediate constant. + + :arg iN x: Dynamic addend. + :arg Imm: Immediate addend. + + Polymorphic over all scalar integer types. + .. inst:: a = isub x, y Wrapping integer subtraction: :math:`a := x - y \pmod{2^B}`. This instruction does not depend on the signed/unsigned interpretation of the operands. -.. todo:: Overflow arithmetic + Polymorphic over all integer types (vector and scalar). + +.. inst:: a = isub_imm Imm, x + + Immediate subtraction. + + Also works as integer negation when :math:`Imm = 0`. Use :inst:`iadd_imm` with a + negative immediate operand for the reverse immediate subtraction. + + :arg Imm: Immediate minuend. + :arg iN x: Dynamic subtrahend. + + Polymorphic over all scalar integer types. + +.. todo:: Integer overflow arithmetic Add instructions for add with carry out / carry in and so on. Enough to implement larger integer types efficiently. It should also be possible to legalize :type:`i64` arithmetic to terms of :type:`i32` operations. -.. inst:: a = ineg x - - Wrapping integer negation: :math:`a := -x \pmod{2^B}`. This instruction does - not depend on the signed/unsigned interpretation of the operand. - .. inst:: a = imul x, y Wrapping integer multiplication: :math:`a := x y \pmod{2^B}`. This instruction does not depend on the signed/unsigned interpretation of the operands. + Polymorphic over all integer types (vector and scalar). + +.. inst:: a = imul_imm x, Imm + + Integer multiplication by immediate constant. + + Polymorphic over all scalar integer types. + .. todo:: Larger multiplication results. For example, ``smulx`` which multiplies :type:`i32` operands to produce a @@ -741,10 +781,11 @@ Integer operations Unsigned integer division: :math:`a := \lfloor {x \over y} \rfloor`. This operation traps if the divisor is zero. - .. todo:: - Add a ``udiv_imm`` variant with an immediate divisor greater than 1. - This is useful for pattern-matching divide-by-constant, and this - instruction would be non-trapping. +.. inst:: a = udiv_imm x, Imm + + Unsigned integer division by an immediate constant. + + This instruction never traps because a divisor of zero is not allowed. .. inst:: a = sdiv x, y @@ -753,37 +794,52 @@ Integer operations the result is not representable in :math:`B` bits two's complement. This only happens when :math:`x = -2^{B-1}, y = -1`. - .. todo:: - Add a ``sdiv_imm`` variant with an immediate non-zero divisor. This is - useful for pattern-matching divide-by-constant, and this instruction - would be non-trapping. Don't allow divisors 0, 1, or -1. +.. inst:: a = sdiv_imm x, Imm + + Signed integer division by an immediate constant. + + This instruction never traps because a divisor of -1 or 0 is not allowed. .. inst:: a = urem x, y - Unsigned integer remainder. This operation traps if the divisor is zero. + Unsigned integer remainder. - .. todo:: - Add a ``urem_imm`` non-trapping variant. + This operation traps if the divisor is zero. + +.. inst:: a = urem_imm x, Imm + + Unsigned integer remainder with immediate divisor. + + This instruction never traps because a divisor of zero is not allowed. .. inst:: a = srem x, y - Signed integer remainder. This operation traps if the divisor is zero. + Signed integer remainder. + + This operation traps if the divisor is zero. + + .. todo:: Integer remainder vs modulus. - .. todo:: Clarify whether the result has the sign of the divisor or the dividend. Should we add a ``smod`` instruction for the case where the result has the same sign as the divisor? +.. inst:: a = srem_imm x, Imm + + Signed integer remainder with immediate divisor. + + This instruction never traps because a divisor of 0 or -1 is not allowed. + .. todo:: Minimum / maximum. NEON has ``smin``, ``smax``, ``umin``, and ``umax`` instructions. We should replicate those for both scalar and vector integer types. Even if the - target ISA doesn't have scalar operations, these are good pattern mtching + target ISA doesn't have scalar operations, these are good pattern matching targets. .. todo:: Saturating arithmetic. - Mostly for SIMD use, but again these are good paterns to contract. + Mostly for SIMD use, but again these are good patterns for contraction. Something like ``usatadd``, ``usatsub``, ``ssatadd``, and ``ssatsub`` is a good start. @@ -825,9 +881,9 @@ Bitwise operations Rotate the bits in ``x`` by ``y`` places. - :param x: Integer value to be rotated. - :param y: Number of bits to shift. Any integer type, not necessarily the - same type as ``x``. + :arg T x: Integer value to be rotated. + :arg iN y: Number of bits to shift. Any scalar integer type, not necessarily + the same type as ``x``. :rtype: Same type as ``x``. .. inst:: a = rotr x, y @@ -836,9 +892,9 @@ Bitwise operations Rotate the bits in ``x`` by ``y`` places. - :param x: Integer value to be rotated. - :param y: Number of bits to shift. Any integer type, not necessarily the - same type as ``x``. + :arg T x: Integer value to be rotated. + :arg iN y: Number of bits to shift. Any scalar integer type, not necessarily + the same type as ``x``. :rtype: Same type as ``x``. .. inst:: a = ishl x, y @@ -848,9 +904,9 @@ Bitwise operations The shift amount is masked to the size of ``x``. - :param x: Integer value to be shifted. - :param y: Number of bits to shift. Any integer type, not necessarily the - same type as ``x``. + :arg T x: Integer value to be shifted. + :arg iN y: Number of bits to shift. Any scalar integer type, not necessarily + the same type as ``x``. :rtype: Same type as ``x``. When shifting a B-bits integer type, this instruction computes: @@ -868,9 +924,9 @@ Bitwise operations The shift amount is masked to the size of the register. - :param x: Integer value to be shifted. - :param y: Number of bits to shift. Can be any integer type, not necessarily - the same type as ``x``. + :arg T x: Integer value to be shifted. + :arg iN y: Number of bits to shift. Can be any scalar integer type, not + necessarily the same type as ``x``. :rtype: Same type as ``x``. When shifting a B-bits integer type, this instruction computes: @@ -888,9 +944,9 @@ Bitwise operations The shift amount is masked to the size of the register. - :param x: Integer value to be shifted. - :param y: Number of bits to shift. Can be any integer type, not necessarily - the same type as ``x``. + :arg T x: Integer value to be shifted. + :arg iN y: Number of bits to shift. Can be any scalar integer type, not + necessarily the same type as ``x``. :rtype: Same type as ``x``. .. todo:: Add ``sshr_imm`` variant with an immediate ``y``. @@ -899,7 +955,7 @@ Bitwise operations Count leading zero bits. - :param x: Integer value. + :arg x: Integer value. :rtype: :type:`i8` Starting from the MSB in ``x``, count the number of zero bits before @@ -910,7 +966,7 @@ Bitwise operations Count leading sign bits. - :param x: Integer value. + :arg x: Integer value. :rtype: :type:`i8` Starting from the MSB after the sign bit in ``x``, count the number of @@ -921,7 +977,7 @@ Bitwise operations Count trailing zeros. - :param x: Integer value. + :arg x: Integer value. :rtype: :type:`i8` Starting from the LSB in ``x``, count the number of zero bits before @@ -932,7 +988,7 @@ Bitwise operations Population count - :param x: Integer value. + :arg x: Integer value. :rtype: :type:`i8` Count the number of one bits in ``x``. @@ -941,12 +997,14 @@ Bitwise operations Floating point operations ------------------------- -.. inst:: a = fcmp cond, x, y +These operations generally follow IEEE 754-2008 semantics. + +.. inst:: a = fcmp Cond, x, y Floating point comparison. - :param cond: Condition code determining how ``x`` and ``y`` are compared. - :param x, y: Floating point scalar or vector values of the same type. + :arg Cond: Condition code determining how ``x`` and ``y`` are compared. + :arg x,y: Floating point scalar or vector values of the same type. :rtype: :type:`bool` or :type:`boolxN` with the same number of lanes as ``x`` and ``y``. @@ -978,7 +1036,7 @@ Floating point operations Floating point negation. - :returns: ``x`` with its sign bit inverted. + :result: ``x`` with its sign bit inverted. Note that this is a pure bitwise operation. @@ -986,7 +1044,7 @@ Floating point operations Floating point absolute value. - :returns: ``x`` with its sign bit cleared. + :result: ``x`` with its sign bit cleared. Note that this is a pure bitwise operation. @@ -994,18 +1052,19 @@ Floating point operations Floating point copy sign. - :returns: ``x`` with its sign changed to that of ``y``. + :result: ``x`` with its sign changed to that of ``y``. Note that this is a pure bitwise operation. The sign bit from ``y`` is copied to the sign bit of ``x``. -.. inst:: fmul x, y -.. inst:: fdiv x, y -.. inst:: fmin x, y -.. inst:: fminnum x, y -.. inst:: fmax x, y -.. inst:: fmaxnum x, y -.. inst:: ceil x +.. inst:: a = fmul x, y +.. inst:: a = fdiv x, y +.. inst:: a = fmin x, y +.. inst:: a = fminnum x, y +.. inst:: a = fmax x, y +.. inst:: a = fmaxnum x, y + +.. inst:: a = ceil x Round floating point round to integral, towards positive infinity. @@ -1053,6 +1112,7 @@ Conversion operations .. inst:: a = cvt_utof x .. inst:: a = cvt_stof x + Glossary ======== From 2929b6de848b7e62f76746e9be86b9c265bd5e6e Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Sat, 23 Jan 2016 18:03:23 -0800 Subject: [PATCH 0017/3084] Update README. --- README.rst | 38 +++++++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/README.rst b/README.rst index 10db808462..895247eb42 100644 --- a/README.rst +++ b/README.rst @@ -3,7 +3,39 @@ Cretonne Code Generator ======================= Cretonne is a low-level retargetable code generator. It translates a -target-independent intermediate language into executable machine code. Cretonne -aims to generate code that behaves identically on different target -architectures, and that can be executed safely in a sandbox. +target-independent intermediate language into executable machine code. + +Cretonne is designed to be a code generator for WebAssembly with these design +goals: + +No undefined behavior + Cretonne does not have a nasal demons clause, and it won't generate code + with unexpected behavior if invariants are broken. +Portable semantics + As far as possible, Cretonne's input language has well-defined semantics + that are the same on all target architectures. The semantics are usually + the same as WebAssembly's. +Fast sandbox verification + Cretonne's input language has a safe subset for sandboxed code. No advanced + analysis is required to verify memory safety as long as only the safe + instructions are used. The safe instruction set is expressive enough to + implement WebAssembly. +Scalable performance + Cretonne can be configured to generate code as quickly as possible, or it + can generate very good code at the cost of slower compile times. +Predictable performance + When optimizing, Cretonne focuses on adapting the target-independent IL to + the quirks of the target architecture. There are no advanced optimizations + that sometimes work, somtimes fail. + +Building the documentation +-------------------------- + +To build the Cretonne documentation, you need the `Sphinx documentation +generator `_:: + + $ pip install sphinx + $ cd cretonne/docs + $ make html + $ open _build/html/index.html From a82a4d27171d0a0cc1031d62d89fd8284e4bfc80 Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Thu, 4 Feb 2016 11:47:25 -0800 Subject: [PATCH 0018/3084] Add ReadTheDocs badge with link to documentation. --- README.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.rst b/README.rst index 895247eb42..8e8105f83b 100644 --- a/README.rst +++ b/README.rst @@ -5,6 +5,10 @@ Cretonne Code Generator Cretonne is a low-level retargetable code generator. It translates a target-independent intermediate language into executable machine code. +.. image:: https://readthedocs.org/projects/cretonne/badge/?version=latest + :target: http://cretonne.readthedocs.org/en/latest/?badge=latest + :alt: Documentation Status + Cretonne is designed to be a code generator for WebAssembly with these design goals: From e337d193738ae7ed02839ebf713e9225c75aab49 Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Thu, 4 Feb 2016 12:19:08 -0800 Subject: [PATCH 0019/3084] Add some more type classes. --- cranelift/docs/langref.rst | 89 ++++++++++++++++++++++++++++---------- 1 file changed, 67 insertions(+), 22 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 3aad746bfe..0f12a7489e 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -178,8 +178,8 @@ called a *lane*. The number of lanes must be a power of two in the range 2-256. Like the :type:`bool` type, a boolean vector cannot be stored in memory. -Pseudo-types ------------- +Pseudo-types and type classes +----------------------------- These are not concrete types, but convenient names uses to refer to real types in this reference. @@ -191,10 +191,38 @@ in this reference. This is either :type:`i32`, or :type:`i64`, depending on whether the target platform has 32-bit or 64-bit pointers. -.. type:: iN +.. type:: iB Any of the scalar integer types :type:`i8` -- :type:`i64`. +.. type:: Int + + Any scalar *or vector* integer type: :type:`iB` or :type:`iBxN`. + +.. type:: fB + + Either of the floating point scalar types: :type:`f32` or :type:`f64. + +.. type:: Float + + Any scalar *or vector* floating point type: :type:`fB` or :type:`fBxN`. + +.. type:: %Tx%N + + Any SIMD vector type. + +.. type:: Mem + + Any type that can be stored in memory: :type:`Int` or :type:`Float`. + +.. type:: Logic + + Either :type:`bool` or :type:`boolxN`. + +.. type:: Testable + + Either :type:`bool` or :type:`iN`. + Control flow ============ @@ -222,7 +250,7 @@ instruction in the EBB. If ``x`` is a :type:`bool` value, take the branch when ``x`` is false. If ``x`` is an integer value, take the branch when ``x = 0``. - :arg iN/bool x: Value to test. + :arg Testable x: Value to test. :arg EBB: Destination extended basic block. :arg args...: Arguments passed to EBB. :result: None. @@ -234,7 +262,7 @@ instruction in the EBB. If ``x`` is a :type:`bool` value, take the branch when ``x`` is true. If ``x`` is an integer value, take the branch when ``x != 0``. - :arg iN/bool x: Value to test. + :arg Testable x: Value to test. :arg EBB: Destination extended basic block. :arg args...: Zero or more arguments passed to EBB. :result: None. @@ -288,7 +316,7 @@ is zero. if ``x`` is non-zero, execution continues at the following instruction. - :arg iN/bool x: Value to test. + :arg Testable x: Value to test. :result: None. .. inst:: trapnz x @@ -297,7 +325,7 @@ is zero. if ``x`` is zero, execution continues at the following instruction. - :arg iN/bool x: Value to test. + :arg Testable x: Value to test. :result: None. @@ -618,6 +646,8 @@ load a constant into an SSA value. Create a scalar integer SSA value with an immediate constant value, or an integer vector where all the lanes have the same value. + :result Int a: Constant value. + .. inst:: a = fconst N Floating point constant. @@ -625,12 +655,16 @@ load a constant into an SSA value. Create a :type:`f32` or :type:`f64` SSA value with an immediate constant value, or a floating point vector where all the lanes have the same value. + :result Float a: Constant value. + .. inst:: a = vconst N Vector constant (floating point or integer). Create a SIMD vector value where the lanes don't have to be identical. + :result TxN a: Constant value. + .. inst:: a = select c, x, y Conditional select. @@ -638,7 +672,7 @@ load a constant into an SSA value. :arg bool c: Controlling flag. :arg T x: Value to return when ``c`` is true. :arg T y: Value to return when ``c`` is false. Must be same type as ``x``. - :rtype: T. Same type as ``x`` and ``y``. + :result T a: Same type as ``x`` and ``y``. This instruction selects whole values. Use :inst:`vselect` for lane-wise selection. @@ -653,12 +687,12 @@ Vector operations Select lanes from ``x`` or ``y`` controlled by the lanes of the boolean vector ``c``. - :arg boolx%N c: Controlling flag vector. - :arg x: Vector with lanes selected by the true lanes of ``c``. + :arg boolxN c: Controlling flag vector. + :arg TxN x: Vector with lanes selected by the true lanes of ``c``. Must be a vector type with the same number of lanes as ``c``. - :arg y: Vector with lanes selected by the false lanes of ``c``. + :arg TxN y: Vector with lanes selected by the false lanes of ``c``. Must be same type as ``x``. - :rtype: Same type as ``x`` and ``y``. + :result TxN a: Same type as ``x`` and ``y``. .. inst:: a = vbuild x, y, z, ... @@ -672,20 +706,32 @@ Vector operations Return a vector whose lanes are all ``x``. -.. inst:: a = insertlane x, idx, y + :arg T x: Scalar value to be replicated. + :result TxN a: Vector with identical lanes. - Insert ``y`` as lane ``idx`` in x. +.. inst:: a = insertlane x, Idx, y - The lane index, ``idx``, is an immediate value, not an SSA value. It must + Insert ``y`` as lane ``Idx`` in x. + + The lane index, ``Idx``, is an immediate value, not an SSA value. It must indicate a valid lane index for the type of ``x``. -.. inst:: a = extractlane x, idx + :arg TxN x: Vector to modify. + :arg Idx: Lane index smaller than N. + :arg T y: New lane value. + :result TxN y: Updated vector. - Extract lane ``idx`` from ``x``. +.. inst:: a = extractlane x, Idx - The lane index, ``idx``, is an immediate value, not an SSA value. It must + Extract lane ``Idx`` from ``x``. + + The lane index, ``Idx``, is an immediate value, not an SSA value. It must indicate a valid lane index for the type of ``x``. + :arg TxN x: Source vector + :arg Idx: Lane index + :result T a: Lane value. + Integer operations ------------------ @@ -694,10 +740,9 @@ Integer operations Integer comparison. :arg Cond: Condition code determining how ``x`` and ``y`` are compared. - :arg x: First value to compare. - :arg y: Second value to compare. - :rtype: :type:`bool` or :type:`boolxN` with the same number of lanes as - ``x`` and ``y``. + :arg Int x: First value to compare. + :arg Int y: Second value to compare. + :result Logic a: With the same number of lanes as ``x`` and ``y``. The condition code determines if the operands are interpreted as signed or unsigned integers. From c459c11a5a4b1606a6f35a033b401c8e94d6f445 Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Thu, 4 Feb 2016 17:25:32 -0800 Subject: [PATCH 0020/3084] Begin defining the meta language. The Cretonne meta language is used to describe Cretonne instructions, both the target independent ones in the base instruction set and real target instructions. Start by providing type definitions matching langref, and begin the meta language reference using autodoc to pull in the PYthon definitions. --- cranelift/docs/conf.py | 4 ++ cranelift/docs/index.rst | 1 + cranelift/docs/metaref.rst | 33 +++++++++++++++ meta/cretonne/__init__.py | 83 ++++++++++++++++++++++++++++++++++++++ meta/cretonne/types.py | 20 +++++++++ 5 files changed, 141 insertions(+) create mode 100644 cranelift/docs/metaref.rst create mode 100644 meta/cretonne/__init__.py create mode 100644 meta/cretonne/types.py diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py index 120d32011a..0b7f7667d8 100644 --- a/cranelift/docs/conf.py +++ b/cranelift/docs/conf.py @@ -21,6 +21,10 @@ import shlex # documentation root, use os.path.abspath to make it absolute, like shown here. sys.path.insert(0, os.path.abspath('.')) +# Also add the meta directory to sys.path so autodoc can find the Cretonne meta +# language definitions. +sys.path.insert(0, os.path.abspath('../meta')) + # -- General configuration ------------------------------------------------ # If your documentation needs a minimal Sphinx version, state it here. diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst index 668b055c97..88d9257213 100644 --- a/cranelift/docs/index.rst +++ b/cranelift/docs/index.rst @@ -7,6 +7,7 @@ Contents: :maxdepth: 2 langref + metaref Indices and tables ================== diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst new file mode 100644 index 0000000000..42a471cc93 --- /dev/null +++ b/cranelift/docs/metaref.rst @@ -0,0 +1,33 @@ +******************************** +Cretonne Meta Language Reference +******************************** + +.. default-domain:: py +.. highlight:: python + +The Cretonne meta language is used to define instructions for Cretonne. It is a +domain specific language embedded in Python. + +An instruction set is described by a Python module under the :file:`meta` +directory that has a global variable called ``instructions``. The basic +Cretonne instruction set described in :doc:`langref` is defined by the Python +module :mod:`cretonne.instrs`. + +Types +===== + +Concrete value types are represented as instances of :class:`cretonne.Type`. There are +subclasses to represent scalar and vector types. + +.. autoclass:: cretonne.Type +.. autoclass:: cretonne.ScalarType + :members: +.. autoclass:: cretonne.VectorType + :members: +.. autoclass:: cretonne.IntType + :members: +.. autoclass:: cretonne.FloatType + :members: +.. automodule:: cretonne.types + :members: + diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py new file mode 100644 index 0000000000..9133f8e062 --- /dev/null +++ b/meta/cretonne/__init__.py @@ -0,0 +1,83 @@ +""" +Cretonne meta language module. + +This module provides classes and functions used to describe Cretonne +instructions. +""" + +# Concrete types. +# +# Instances (i8, i32, ...) are provided in the cretonne.types module. + +class Type(object): + """A concrete value type.""" + + def __str__(self): + return self.name + +class ScalarType(Type): + """ + A concrete scalar (not vector) type. + + Also tracks a unique set of :class:`VectorType` instances with this type as + the lane type. + """ + + def __init__(self, name): + self.name = name + self._vectors = dict() + + def __repr__(self): + return 'ScalarType({})'.format(self.name) + + def by(self, lanes): + """ + Get a vector type with this type as the lane type. + + For example, ``i32.by(4)`` returns the :obj:`i32x4` type. + """ + if lanes in self._vectors: + return self._vectors[lanes] + else: + v = VectorType(self, lanes) + self._vectors[lanes] = v + return v + +class VectorType(Type): + """ + A concrete SIMD vector type. + + A vector type has a lane type which is an instance of :class:`ScalarType`, + and a positive number of lanes. + """ + + def __init__(self, base, lanes): + assert isinstance(base, ScalarType), 'SIMD lanes must be scalar types' + self.base = base + self.lanes = lanes + self.name = '{}x{}'.format(base.name, lanes) + + def __repr__(self): + return 'VectorType(base={}, lanes={})'.format(self.base.name, self.lanes) + +class IntType(ScalarType): + """A concrete scalar integer type.""" + + def __init__(self, bits): + assert bits > 0, 'IntType must have positive number of bits' + super(IntType, self).__init__('i{:d}'.format(bits)) + self.bits = bits + + def __repr__(self): + return 'IntType(bits={})'.format(self.bits) + +class FloatType(ScalarType): + """A concrete scalar floating point type.""" + + def __init__(self, bits): + assert bits > 0, 'FloatType must have positive number of bits' + super(FloatType, self).__init__('f{:d}'.format(bits)) + self.bits = bits + + def __repr__(self): + return 'FloatType(bits={})'.format(self.bits) diff --git a/meta/cretonne/types.py b/meta/cretonne/types.py new file mode 100644 index 0000000000..382fe0def2 --- /dev/null +++ b/meta/cretonne/types.py @@ -0,0 +1,20 @@ +"""Predefined types.""" + +from . import ScalarType, IntType, FloatType + +#: A boolean value. +bool = ScalarType('bool') + +i8 = IntType(8) #: 8-bit int. +i16 = IntType(16) #: 16-bit int. +i32 = IntType(32) #: 32-bit int. +i64 = IntType(64) #: 64-bit int. + +f32 = FloatType(32) #: IEEE 32-bit float. +f64 = FloatType(64) #: IEEE 64-bit float + +i8x16 = i8.by(16) #: Vector of 16 i8 lanes. + +f32x4 = f32.by(4) #: Vector of 4 f32 lanes. +f64x2 = f64.by(2) #: Vector of 2 f64 lanes. + From 19b4facbe0966502ceb8a35b7c4e8ef854390d0e Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Mon, 8 Feb 2016 18:21:58 -0800 Subject: [PATCH 0021/3084] Add a TypeDocumenter for Cretonne types. Use the autodoc Sphinx module to add a .. autoctontype:: directive which generates documentation for one of the types in the cretonne.types module. --- cranelift/docs/cton_domain.py | 34 ++++++++++++++++++++++++++++++ cranelift/docs/langref.rst | 39 +++++++---------------------------- meta/cretonne/__init__.py | 29 +++++++++++++++++++------- meta/cretonne/types.py | 35 +++++++++++++++++++++---------- 4 files changed, 87 insertions(+), 50 deletions(-) diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index e1dc08a67b..823ed9c8a5 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -13,6 +13,7 @@ import re from docutils import nodes +from docutils.parsers.rst import directives from sphinx import addnodes from sphinx.directives import ObjectDescription @@ -22,6 +23,8 @@ from sphinx.roles import XRefRole from sphinx.util.docfields import Field, GroupedField, TypedField from sphinx.util.nodes import make_refnode +import sphinx.ext.autodoc + class CtonObject(ObjectDescription): """ Any kind of Cretonne IL object. @@ -29,6 +32,11 @@ class CtonObject(ObjectDescription): This is a shared base class for the different kinds of indexable objects in the Cretonne IL reference. """ + option_spec = { + 'noindex': directives.flag, + 'module': directives.unchanged, + 'annotation': directives.unchanged, + } def add_target_and_index(self, name, sig, signode): """ @@ -209,7 +217,33 @@ class CretonneDomain(Domain): make_refnode(builder, fromdocname, obj[0], obj[1] + '-' + target, contnode, target))] + +class TypeDocumenter(sphinx.ext.autodoc.Documenter): + # Invoke with .. autoctontype:: + objtype = 'ctontype' + # Convert into cton:type directives + domain = 'cton' + directivetype = 'type' + + @classmethod + def can_document_member(cls, member, membername, isattr, parent): + return False + + def resolve_name(self, modname, parents, path, base): + return 'cretonne.types', [ base ] + + def add_content(self, more_content, no_docstring=False): + super(TypeDocumenter, self).add_content(more_content, no_docstring) + sourcename = self.get_sourcename() + membytes = self.object.membytes + if membytes: + self.add_line(u':bytes: {}'.format(membytes), sourcename) + else: + self.add_line(u':bytes: Can\'t be stored in memory', sourcename) + + def setup(app): app.add_domain(CretonneDomain) + app.add_autodocumenter(TypeDocumenter) return { 'version' : '0.1' } diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 0f12a7489e..ad2ccb4299 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -87,10 +87,7 @@ All SSA values have a type which determines the size and shape (for SIMD vectors) of the value. Many instructions are polymorphic -- they can operate on different types. -.. type:: bool - - A boolean value that is either true or false. Booleans can't be stored in - memory. +.. autoctontype:: bool Integer types ------------- @@ -99,21 +96,10 @@ Integer values have a fixed size and can be interpreted as either signed or unsigned. Some instructions will interpret an operand as a signed or unsigned number, others don't care. -.. type:: i8 - - A 8-bit integer value taking up 1 byte in memory. - -.. type:: i16 - - A 16-bit integer value taking up 2 bytes in memory. - -.. type:: i32 - - A 32-bit integer value taking up 4 bytes in memory. - -.. type:: i64 - - A 64-bit integer value taking up 8 bytes in memory. +.. autoctontype:: i8 +.. autoctontype:: i16 +.. autoctontype:: i32 +.. autoctontype:: i64 Floating point types -------------------- @@ -122,17 +108,8 @@ The floating point types have the IEEE semantics that are supported by most hardware. There is no support for higher-precision types like quads or double-double formats. -.. type:: f32 - - A 32-bit floating point type represented in the IEEE 754 *single precision* - format. This corresponds to the :c:type:`float` type in most C - implementations. - -.. type:: f64 - - A 64-bit floating point type represented in the IEEE 754 *double precision* - format. This corresponds to the :c:type:`double` type in most C - implementations. +.. autoctontype:: f32 +.. autoctontype:: f64 SIMD vector types ----------------- @@ -201,7 +178,7 @@ in this reference. .. type:: fB - Either of the floating point scalar types: :type:`f32` or :type:`f64. + Either of the floating point scalar types: :type:`f32` or :type:`f64`. .. type:: Float diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 9133f8e062..216607c9dc 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -12,6 +12,11 @@ instructions. class Type(object): """A concrete value type.""" + def __init__(self, name, membytes, doc): + self.name = name + self.membytes = membytes + self.__doc__ = doc + def __str__(self): return self.name @@ -19,12 +24,12 @@ class ScalarType(Type): """ A concrete scalar (not vector) type. - Also tracks a unique set of :class:`VectorType` instances with this type as - the lane type. + Also tracks a unique set of :py:class:`VectorType` instances with this type + as the lane type. """ - def __init__(self, name): - self.name = name + def __init__(self, name, membytes, doc): + super(ScalarType, self).__init__(name, membytes, doc) self._vectors = dict() def __repr__(self): @@ -53,9 +58,14 @@ class VectorType(Type): def __init__(self, base, lanes): assert isinstance(base, ScalarType), 'SIMD lanes must be scalar types' + super(VectorType, self).__init__( + name='{}x{}'.format(base.name, lanes), + membytes=lanes*base.membytes, + doc=""" + A SIMD vector with {} lanes containing a {} each. + """.format(lanes, base.name)) self.base = base self.lanes = lanes - self.name = '{}x{}'.format(base.name, lanes) def __repr__(self): return 'VectorType(base={}, lanes={})'.format(self.base.name, self.lanes) @@ -65,7 +75,10 @@ class IntType(ScalarType): def __init__(self, bits): assert bits > 0, 'IntType must have positive number of bits' - super(IntType, self).__init__('i{:d}'.format(bits)) + super(IntType, self).__init__( + name='i{:d}'.format(bits), + membytes=bits/8, + doc="An integer type with {} bits.".format(bits)) self.bits = bits def __repr__(self): @@ -74,9 +87,9 @@ class IntType(ScalarType): class FloatType(ScalarType): """A concrete scalar floating point type.""" - def __init__(self, bits): + def __init__(self, bits, doc): assert bits > 0, 'FloatType must have positive number of bits' - super(FloatType, self).__init__('f{:d}'.format(bits)) + super(FloatType, self).__init__( name='f{:d}'.format(bits), membytes=bits/8, doc=doc) self.bits = bits def __repr__(self): diff --git a/meta/cretonne/types.py b/meta/cretonne/types.py index 382fe0def2..27a74aec67 100644 --- a/meta/cretonne/types.py +++ b/meta/cretonne/types.py @@ -2,19 +2,32 @@ from . import ScalarType, IntType, FloatType -#: A boolean value. -bool = ScalarType('bool') +bool = ScalarType('bool', 0, + """ + A boolean value that is either true or false. + """) -i8 = IntType(8) #: 8-bit int. -i16 = IntType(16) #: 16-bit int. -i32 = IntType(32) #: 32-bit int. -i64 = IntType(64) #: 64-bit int. +i8 = IntType(8) +i16 = IntType(16) +i32 = IntType(32) +i64 = IntType(64) -f32 = FloatType(32) #: IEEE 32-bit float. -f64 = FloatType(64) #: IEEE 64-bit float +f32 = FloatType(32, + """ + A 32-bit floating point type represented in the IEEE 754-2008 *binary32* + interchange format. This corresponds to the :c:type:`float` type in most + C implementations. + """) -i8x16 = i8.by(16) #: Vector of 16 i8 lanes. +f64 = FloatType(64, + """ + A 64-bit floating point type represented in the IEEE 754-2008 *binary64* + interchange format. This corresponds to the :c:type:`double` type in + most C implementations. + """) -f32x4 = f32.by(4) #: Vector of 4 f32 lanes. -f64x2 = f64.by(2) #: Vector of 2 f64 lanes. +i8x16 = i8.by(16) + +f32x4 = f32.by(4) +f64x2 = f64.by(2) From a37005a1d4924f6301dac1f6a83c00772f7f30e6 Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Tue, 9 Feb 2016 08:11:20 -0800 Subject: [PATCH 0022/3084] Enable inheritance diagrams. --- cranelift/docs/conf.py | 9 +++++++++ cranelift/docs/metaref.rst | 17 ++++++++++++----- meta/cretonne/types.py | 21 ++++++++++----------- 3 files changed, 31 insertions(+), 16 deletions(-) diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py index 0b7f7667d8..c4ab93eff0 100644 --- a/cranelift/docs/conf.py +++ b/cranelift/docs/conf.py @@ -38,6 +38,8 @@ extensions = [ 'sphinx.ext.todo', 'sphinx.ext.mathjax', 'sphinx.ext.ifconfig', + 'sphinx.ext.graphviz', + 'sphinx.ext.inheritance_diagram', 'cton_domain', 'cton_lexer', ] @@ -293,3 +295,10 @@ texinfo_documents = [ # If true, do not generate a @detailmenu in the "Top" node's menu. #texinfo_no_detailmenu = False + + +# -- Options for Graphviz ------------------------------------------------- + +graphviz_output_format = 'svg' + +inheritance_graph_attrs = dict(rankdir='TD') diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 42a471cc93..8cdece065c 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -13,21 +13,28 @@ directory that has a global variable called ``instructions``. The basic Cretonne instruction set described in :doc:`langref` is defined by the Python module :mod:`cretonne.instrs`. +.. module:: cretonne + Types ===== Concrete value types are represented as instances of :class:`cretonne.Type`. There are subclasses to represent scalar and vector types. -.. autoclass:: cretonne.Type -.. autoclass:: cretonne.ScalarType +.. inheritance-diagram:: Type ScalarType VectorType IntType FloatType + :parts: 1 +.. autoclass:: Type +.. autoclass:: ScalarType :members: -.. autoclass:: cretonne.VectorType +.. autoclass:: VectorType :members: -.. autoclass:: cretonne.IntType +.. autoclass:: IntType :members: -.. autoclass:: cretonne.FloatType +.. autoclass:: FloatType :members: + +Predefined types +---------------- .. automodule:: cretonne.types :members: diff --git a/meta/cretonne/types.py b/meta/cretonne/types.py index 27a74aec67..3d369a60e4 100644 --- a/meta/cretonne/types.py +++ b/meta/cretonne/types.py @@ -1,17 +1,21 @@ -"""Predefined types.""" +""" +The cretonne.types module predefines all the Cretonne scalar types. +""" from . import ScalarType, IntType, FloatType +#: Boolean. bool = ScalarType('bool', 0, """ A boolean value that is either true or false. """) -i8 = IntType(8) -i16 = IntType(16) -i32 = IntType(32) -i64 = IntType(64) +i8 = IntType(8) #: 8-bit int. +i16 = IntType(16) #: 16-bit int. +i32 = IntType(32) #: 32-bit int. +i64 = IntType(64) #: 64-bit int. +#: IEEE single precision. f32 = FloatType(32, """ A 32-bit floating point type represented in the IEEE 754-2008 *binary32* @@ -19,15 +23,10 @@ f32 = FloatType(32, C implementations. """) +#: IEEE double precision. f64 = FloatType(64, """ A 64-bit floating point type represented in the IEEE 754-2008 *binary64* interchange format. This corresponds to the :c:type:`double` type in most C implementations. """) - -i8x16 = i8.by(16) - -f32x4 = f32.by(4) -f64x2 = f64.by(2) - From 9d1fbfd6497c46e27551407dfa0c558dbd2b0cf0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 11 Mar 2016 15:54:28 -0800 Subject: [PATCH 0023/3084] Implement type representation in Rust. Start the Cretonne library as a Rust crate. --- cranelift/.gitignore | 2 + cranelift/src/libcretonne/Cargo.toml | 10 ++ cranelift/src/libcretonne/lib.rs | 8 + cranelift/src/libcretonne/types.rs | 230 +++++++++++++++++++++++++++ 4 files changed, 250 insertions(+) create mode 100644 cranelift/src/libcretonne/Cargo.toml create mode 100644 cranelift/src/libcretonne/lib.rs create mode 100644 cranelift/src/libcretonne/types.rs diff --git a/cranelift/.gitignore b/cranelift/.gitignore index 0d20b6487c..25eadf4e60 100644 --- a/cranelift/.gitignore +++ b/cranelift/.gitignore @@ -1 +1,3 @@ *.pyc +Cargo.lock +target diff --git a/cranelift/src/libcretonne/Cargo.toml b/cranelift/src/libcretonne/Cargo.toml new file mode 100644 index 0000000000..58a67fe569 --- /dev/null +++ b/cranelift/src/libcretonne/Cargo.toml @@ -0,0 +1,10 @@ +[package] +authors = ["The cwCretonneRust Project Developers"] +name = "cretonne" +version = "0.0.0" + +[lib] +name = "cretonne" +path = "lib.rs" + +[dependencies] diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs new file mode 100644 index 0000000000..e2adcaec67 --- /dev/null +++ b/cranelift/src/libcretonne/lib.rs @@ -0,0 +1,8 @@ + +//====--------------------------------------------------------------------------------------====// +// +// Cretonne code generation library. +// +//====--------------------------------------------------------------------------------------====// + +mod types; diff --git a/cranelift/src/libcretonne/types.rs b/cranelift/src/libcretonne/types.rs new file mode 100644 index 0000000000..49d7cc29ce --- /dev/null +++ b/cranelift/src/libcretonne/types.rs @@ -0,0 +1,230 @@ + +//! Common types for the Cretonne code generator. + +use std::fmt::{self, Display, Formatter, Write}; + +/// The type of an SSA value. +/// +/// The VOID type is only used for instructions that produce no value. It can't be part of a SIMD +/// vector. +/// +/// Basic integer types: `I8`, `I16`, `I32`, and `I64`. These types are sign-agnostic. +/// +/// Basic floating point types: `F32` and `F64`. IEEE single and double precision. +/// +/// Boolean types: `B1`, `B8`, `B16`, `B32`, and `B64`. These all encode 'true' or 'false'. The +/// larger types use redundant bits. +/// +/// SIMD vector types have power-of-two lanes, up to 256. Lanes can be any int/float/bool type. +/// +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct Type(u8); + +pub const VOID: Type = Type(0); + +pub const I8: Type = Type(1); +pub const I16: Type = Type(2); +pub const I32: Type = Type(3); +pub const I64: Type = Type(4); + +pub const F32: Type = Type(5); +pub const F64: Type = Type(6); + +pub const B1: Type = Type(7); +pub const B8: Type = Type(8); +pub const B16: Type = Type(9); +pub const B32: Type = Type(10); +pub const B64: Type = Type(11); + +impl Type { + /// Get the lane type of this SIMD vector type. + /// + /// A scalar type is the same as a SIMD vector type with one lane, so it returns itself. + pub fn lane_type(self) -> Type { + Type(self.0 & 0x0f) + } + + /// Get the number of bits in a lane. + pub fn lane_bits(self) -> u8 { + match self.lane_type() { + B1 => 1, + B8 | I8 => 8, + B16 | I16 => 16, + B32 | I32 | F32 => 32, + B64 | I64 | F64 => 64, + _ => 0, + } + } + + /// Is this the VOID type? + pub fn is_void(self) -> bool { + self == VOID + } + + /// Is this a scalar boolean type? + pub fn is_bool(self) -> bool { + match self { + B1 | B8 | B16 | B32 | B64 => true, + _ => false, + } + } + + /// Is this a scalar integer type? + pub fn is_int(self) -> bool { + match self { + I8 | I16 | I32 | I64 => true, + _ => false, + } + } + + /// Is this a scalar floating point type? + pub fn is_float(self) -> bool { + match self { + F32 | F64 => true, + _ => false, + } + } + + /// Get log2 of the number of lanes in this SIMD vector type. + /// + /// All SIMD types have a lane count that is a power of two and no larger than 256, so this + /// will be a number in the range 0-8. + /// + /// A scalar type is the same as a SIMD vector type with one lane, so it return 0. + pub fn log2_lane_count(self) -> u8 { + self.0 >> 4 + } + + /// Is this a scalar type? (That is, not a SIMD vector type). + /// + /// A scalar type is the same as a SIMD vector type with one lane. + pub fn is_scalar(self) -> bool { + self.log2_lane_count() == 0 + } + + /// Get the number of lanes in this SIMD vector type. + /// + /// A scalar type is the same as a SIMD vector type with one lane, so it returns 1. + pub fn lane_count(self) -> u16 { + 1 << self.log2_lane_count() + } + + /// Get the total number of bits used to represent this type. + pub fn bits(self) -> u16 { + self.lane_bits() as u16 * self.lane_count() + } + + /// Get a SIMD vector type with `n` times more lanes than this one. + /// + /// If this is a scalar type, this produces a SIMD type with this as a lane type and `n` lanes. + /// + /// If this is already a SIMD vector type, this produces a SIMD vector type with `n * + /// self.lane_count()` lanes. + pub fn by(self, n: u16) -> Type { + debug_assert!(self.lane_bits() > 0, + "Can't make SIMD vectors with void lanes."); + debug_assert!(n.is_power_of_two(), + "Number of SIMD lanes must be a power of two"); + let log2_lanes: u32 = n.trailing_zeros(); + let new_type = self.0 as u32 + (log2_lanes << 4); + assert!(new_type < 0x90, "No more than 256 SIMD lanes supported"); + Type(new_type as u8) + } + + /// Get a SIMD vector with half the number of lanes. + pub fn half_vector(self) -> Type { + assert!(!self.is_scalar(), "Expecting a proper SIMD vector type."); + Type(self.0 - 0x10) + } +} + +impl Display for Type { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + if self.is_void() { + write!(f, "void") + } else if self.is_bool() { + write!(f, "b{}", self.lane_bits()) + } else if self.is_int() { + write!(f, "i{}", self.lane_bits()) + } else if self.is_float() { + write!(f, "f{}", self.lane_bits()) + } else if !self.is_scalar() { + write!(f, "{}x{}", self.lane_type(), self.lane_count()) + } else { + panic!("Invalid Type(0x{:x})", self.0) + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn basic_scalars() { + assert_eq!(VOID, VOID.lane_type()); + assert_eq!(0, VOID.bits()); + assert_eq!(B1, B1.lane_type()); + assert_eq!(B8, B8.lane_type()); + assert_eq!(B16, B16.lane_type()); + assert_eq!(B32, B32.lane_type()); + assert_eq!(B64, B64.lane_type()); + assert_eq!(I8, I8.lane_type()); + assert_eq!(I16, I16.lane_type()); + assert_eq!(I32, I32.lane_type()); + assert_eq!(I64, I64.lane_type()); + assert_eq!(F32, F32.lane_type()); + assert_eq!(F64, F64.lane_type()); + + assert_eq!(VOID.lane_bits(), 0); + assert_eq!(B1.lane_bits(), 1); + assert_eq!(B8.lane_bits(), 8); + assert_eq!(B16.lane_bits(), 16); + assert_eq!(B32.lane_bits(), 32); + assert_eq!(B64.lane_bits(), 64); + assert_eq!(I8.lane_bits(), 8); + assert_eq!(I16.lane_bits(), 16); + assert_eq!(I32.lane_bits(), 32); + assert_eq!(I64.lane_bits(), 64); + assert_eq!(F32.lane_bits(), 32); + assert_eq!(F64.lane_bits(), 64); + } + + #[test] + fn vectors() { + let big = F64.by(256); + assert_eq!(big.lane_bits(), 64); + assert_eq!(big.lane_count(), 256); + assert_eq!(big.bits(), 64 * 256); + + assert_eq!(format!("{}", big.half_vector()), "f64x128"); + assert_eq!(format!("{}", B1.by(2).half_vector()), "b1"); + } + + #[test] + fn format_scalars() { + assert_eq!(format!("{}", VOID), "void"); + assert_eq!(format!("{}", B1), "b1"); + assert_eq!(format!("{}", B8), "b8"); + assert_eq!(format!("{}", B16), "b16"); + assert_eq!(format!("{}", B32), "b32"); + assert_eq!(format!("{}", B64), "b64"); + assert_eq!(format!("{}", I8), "i8"); + assert_eq!(format!("{}", I16), "i16"); + assert_eq!(format!("{}", I32), "i32"); + assert_eq!(format!("{}", I64), "i64"); + assert_eq!(format!("{}", F32), "f32"); + assert_eq!(format!("{}", F64), "f64"); + } + + #[test] + fn format_vectors() { + assert_eq!(format!("{}", B1.by(8)), "b1x8"); + assert_eq!(format!("{}", B8.by(1)), "b8"); + assert_eq!(format!("{}", B16.by(256)), "b16x256"); + assert_eq!(format!("{}", B32.by(4).by(2)), "b32x8"); + assert_eq!(format!("{}", B64.by(8)), "b64x8"); + assert_eq!(format!("{}", I8.by(64)), "i8x64"); + assert_eq!(format!("{}", F64.by(2)), "f64x2"); + } +} From d9ba9480c9cef1822b24ad4dc5982a08934419a2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 11 Mar 2016 16:06:14 -0800 Subject: [PATCH 0024/3084] Make the types module public, add documentation comments. --- cranelift/src/libcretonne/lib.rs | 6 +++--- cranelift/src/libcretonne/types.rs | 23 ++++++++++++++++++++++- 2 files changed, 25 insertions(+), 4 deletions(-) diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index e2adcaec67..380209e7f6 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -1,8 +1,8 @@ -//====--------------------------------------------------------------------------------------====// +// ====------------------------------------------------------------------------------------==== // // // Cretonne code generation library. // -//====--------------------------------------------------------------------------------------====// +// ====------------------------------------------------------------------------------------==== // -mod types; +pub mod types; diff --git a/cranelift/src/libcretonne/types.rs b/cranelift/src/libcretonne/types.rs index 49d7cc29ce..88a2b6cad2 100644 --- a/cranelift/src/libcretonne/types.rs +++ b/cranelift/src/libcretonne/types.rs @@ -5,7 +5,7 @@ use std::fmt::{self, Display, Formatter, Write}; /// The type of an SSA value. /// -/// The VOID type is only used for instructions that produce no value. It can't be part of a SIMD +/// The `VOID` type is only used for instructions that produce no value. It can't be part of a SIMD /// vector. /// /// Basic integer types: `I8`, `I16`, `I32`, and `I64`. These types are sign-agnostic. @@ -20,20 +20,41 @@ use std::fmt::{self, Display, Formatter, Write}; #[derive(Copy, Clone, PartialEq, Eq, Debug)] pub struct Type(u8); +/// No type. Used for functions without a return value. Can't be loaded or stored. Can't be part of +/// a SIMD vector. pub const VOID: Type = Type(0); +/// Integer type with 8 bits. pub const I8: Type = Type(1); + +/// Integer type with 16 bits. pub const I16: Type = Type(2); + +/// Integer type with 32 bits. pub const I32: Type = Type(3); + +/// Integer type with 64 bits. pub const I64: Type = Type(4); +/// IEEE single precision floating point type. pub const F32: Type = Type(5); + +/// IEEE double precision floating point type. pub const F64: Type = Type(6); +/// Boolean type. Can't be loaded or stored, but can be used to form SIMD vectors. pub const B1: Type = Type(7); + +/// Boolean type using 8 bits to represent true/false. pub const B8: Type = Type(8); + +/// Boolean type using 16 bits to represent true/false. pub const B16: Type = Type(9); + +/// Boolean type using 32 bits to represent true/false. pub const B32: Type = Type(10); + +/// Boolean type using 64 bits to represent true/false. pub const B64: Type = Type(11); impl Type { From 3d67d4d6b9fc3b49f1ccd87705bf444b155d1c54 Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Fri, 12 Feb 2016 10:11:52 -0800 Subject: [PATCH 0025/3084] Add type variables. --- cranelift/docs/metaref.rst | 10 ++++++++++ meta/cretonne/__init__.py | 16 ++++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 8cdece065c..7a8c156142 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -38,3 +38,13 @@ Predefined types .. automodule:: cretonne.types :members: +Parametric polymorphism +----------------------- +.. currentmodule:: cretonne + +Instruction operands can be defined with *type variables* instead of concrete +types for their operands. This makes the instructions polymorphic. + +.. autoclass:: TypeVar + + diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 216607c9dc..e8cdfe3f55 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -94,3 +94,19 @@ class FloatType(ScalarType): def __repr__(self): return 'FloatType(bits={})'.format(self.bits) + +# +# Parametric polymorphism. +# + +class TypeVar(object): + """ + A Type Variable. + + Type variables can be used in place of concrete types when defining + instructions. This makes the instructions *polymorphic*. + """ + + def __init__(self, name): + self.name = name + From 05de34b1a3b4294e71ff72836c6c4b1259729a99 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 30 Mar 2016 11:36:23 -0700 Subject: [PATCH 0026/3084] Add ImmediateType for declaring immediate operands. --- cranelift/docs/langref.rst | 12 ++++++++++++ cranelift/docs/metaref.rst | 17 ++++++++++++++++- meta/cretonne/__init__.py | 20 ++++++++++++++++++++ meta/cretonne/immediates.py | 12 ++++++++++++ 4 files changed, 60 insertions(+), 1 deletion(-) create mode 100644 meta/cretonne/immediates.py diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index ad2ccb4299..845bc4da37 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -200,6 +200,18 @@ in this reference. Either :type:`bool` or :type:`iN`. +Immediate operand types +----------------------- + +These types are not part of the normal SSA type system. They are used to +indicate the different kinds of immediate operands on an instruction. + +.. type:: imm64 + + A 64-bit immediate integer. The value of this operand is interpreted as a + signed two's complement integer. Instruction encodings may limit the valid + range. + Control flow ============ diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 7a8c156142..b4dda9576d 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -38,13 +38,28 @@ Predefined types .. automodule:: cretonne.types :members: +.. currentmodule:: cretonne + Parametric polymorphism ----------------------- -.. currentmodule:: cretonne Instruction operands can be defined with *type variables* instead of concrete types for their operands. This makes the instructions polymorphic. .. autoclass:: TypeVar +Immediates +---------- +Immediate instruction operands don't correspond to SSA values, but have values +that are encoded directly in the instruction. Immediate operands don't +have types from the :class:`cretonne.Type` type system; they often have +enumerated values of a specific type. The type of an immediate operand is +indicated with an instance of :class:`ImmediateType`. + +.. autoclass:: ImmediateType + +.. automodule:: cretonne.immediates + :members: + +.. currentmodule:: cretonne diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index e8cdfe3f55..60c3748106 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -110,3 +110,23 @@ class TypeVar(object): def __init__(self, name): self.name = name +# +# Immediate operands. +# +# Instances of immediate operand types are provided in the cretonne.immediates +# module. + +class ImmediateType(object): + """ + The type of an immediate instruction operand. + """ + + def __init__(self, name, doc): + self.name = name + self.__doc__ = doc + + def __str__(self): + return self.name + + def __repr__(self): + return 'ImmediateType({})'.format(self.name) diff --git a/meta/cretonne/immediates.py b/meta/cretonne/immediates.py new file mode 100644 index 0000000000..6b499d5215 --- /dev/null +++ b/meta/cretonne/immediates.py @@ -0,0 +1,12 @@ +""" +The cretonne.immdiates module predefines all the Cretonne immediate operand +types. +""" + +from . import ImmediateType + +#: A 64-bit immediate integer operand. +#: +#: This type of immediate integer can interact with SSA values with any +#: :py:class:`cretonne.IntType` type. +imm64 = ImmediateType('imm64', 'A 64-bit immediate integer.') From de12bc0f1aff7212f4860a239e53008d2d4f443a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 1 Apr 2016 09:54:49 -0700 Subject: [PATCH 0027/3084] Implement Imm64 in an 'immediates.rs' module. Format larger immediates as hexadecimal with a multiple of 4 digits and '_' group separators. --- cranelift/src/libcretonne/immediates.rs | 53 +++++++++++++++++++++++++ cranelift/src/libcretonne/lib.rs | 1 + 2 files changed, 54 insertions(+) create mode 100644 cranelift/src/libcretonne/immediates.rs diff --git a/cranelift/src/libcretonne/immediates.rs b/cranelift/src/libcretonne/immediates.rs new file mode 100644 index 0000000000..a60291b164 --- /dev/null +++ b/cranelift/src/libcretonne/immediates.rs @@ -0,0 +1,53 @@ + +//! Immediate operands for Cretonne instructions +//! +//! This module defines the types of immediate operands that can appear on Cretonne instructions. +//! Each type here should have a corresponding definition in the `cretonne.immediates` Python +//! module in the meta language. + +use std::fmt::{self, Display, Formatter}; + +/// 64-bit immediate integer operand. +/// +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct Imm64(i64); + +impl Display for Imm64 { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + let x = self.0; + if -10_000 < x && x < 10_000 { + // Use decimal for small numbers. + write!(f, "{}", x) + } else { + // Hexadecimal with a multiple of 4 digits and group separators: + // + // 0xfff0 + // 0x0001_ffff + // 0xffff_ffff_fff8_4400 + // + let mut pos = (64 - x.leading_zeros() - 1) & 0xf0; + try!(write!(f, "0x{:04x}", (x >> pos) & 0xffff)); + while pos > 0 { + pos -= 16; + try!(write!(f, "_{:04x}", (x >> pos) & 0xffff)); + } + Ok(()) + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn format_imm64() { + assert_eq!(format!("{}", Imm64(0)), "0"); + assert_eq!(format!("{}", Imm64(9999)), "9999"); + assert_eq!(format!("{}", Imm64(10000)), "0x2710"); + assert_eq!(format!("{}", Imm64(-9999)), "-9999"); + assert_eq!(format!("{}", Imm64(-10000)), "0xffff_ffff_ffff_d8f0"); + assert_eq!(format!("{}", Imm64(0xffff)), "0xffff"); + assert_eq!(format!("{}", Imm64(0x10000)), "0x0001_0000"); + } +} diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 380209e7f6..9c837eb0f1 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -6,3 +6,4 @@ // ====------------------------------------------------------------------------------------==== // pub mod types; +pub mod immediates; From 20183554a4b49b71794e7c022fd5b2ae4a7b9a2f Mon Sep 17 00:00:00 2001 From: Jakob Olesen Date: Fri, 12 Feb 2016 14:24:01 -0800 Subject: [PATCH 0028/3084] Add Instruction and Operand classes to the meta language. --- cranelift/docs/cton_domain.py | 44 +++++++++++++++++++++ cranelift/docs/metaref.rst | 9 +++++ meta/cretonne/__init__.py | 72 ++++++++++++++++++++++++++++++++++- 3 files changed, 124 insertions(+), 1 deletion(-) diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index 823ed9c8a5..6f20cf8d32 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -242,8 +242,52 @@ class TypeDocumenter(sphinx.ext.autodoc.Documenter): self.add_line(u':bytes: Can\'t be stored in memory', sourcename) +class InstDocumenter(sphinx.ext.autodoc.Documenter): + # Invoke with .. autoinst:: + objtype = 'inst' + # Convert into cton:inst directives + domain = 'cton' + directivetype = 'inst' + + @classmethod + def can_document_member(cls, member, membername, isattr, parent): + return False + + def resolve_name(self, modname, parents, path, base): + return 'cretonne.base', [ base ] + + def format_signature(self): + inst = self.object + sig = self.format_name() + if len(inst.outs) > 0: + sig = ', '.join([op.name for op in inst.outs]) + ' = ' + sig + if len(inst.ins) > 0: + sig = sig + ' ' + ', '.join([op.name for op in inst.ins]) + return sig + + def add_directive_header(self, sig): + """Add the directive header and options to the generated content.""" + domain = getattr(self, 'domain', 'cton') + directive = getattr(self, 'directivetype', self.objtype) + sourcename = self.get_sourcename() + self.add_line(u'.. %s:%s:: %s' % (domain, directive, sig), sourcename) + if self.options.noindex: + self.add_line(u' :noindex:', sourcename) + + def add_content(self, more_content, no_docstring=False): + super(InstDocumenter, self).add_content(more_content, no_docstring) + sourcename = self.get_sourcename() + + # Add inputs and outputs. + for op in self.object.ins: + self.add_line(u':in {} {}: {}'.format(op.typ.name, op.name, op.get_doc()), sourcename) + for op in self.object.outs: + self.add_line(u':out {} {}: {}'.format(op.typ.name, op.name, op.get_doc()), sourcename) + + def setup(app): app.add_domain(CretonneDomain) app.add_autodocumenter(TypeDocumenter) + app.add_autodocumenter(InstDocumenter) return { 'version' : '0.1' } diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index b4dda9576d..8d58757b81 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -63,3 +63,12 @@ indicated with an instance of :class:`ImmediateType`. :members: .. currentmodule:: cretonne + +Instructions +============ + +New instructions are defined as instances of the :class:`cretonne.Instruction` +class. + +.. autoclass:: Operand +.. autoclass:: Instruction diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 60c3748106..fd0c4664dd 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -107,8 +107,9 @@ class TypeVar(object): instructions. This makes the instructions *polymorphic*. """ - def __init__(self, name): + def __init__(self, name, doc): self.name = name + self.__doc__ = doc # # Immediate operands. @@ -130,3 +131,72 @@ class ImmediateType(object): def __repr__(self): return 'ImmediateType({})'.format(self.name) + +# +# Defining instructions. +# + +class Operand(object): + """ + An instruction operand. + + An instruction operand can be either an *immediate* or an *SSA value*. The + type of the operand is one of: + + 1. A :py:class:`Type` instance indicates an SSA value operand with a + concrete type. + + 2. A :py:class:`TypeVar` instance indicates an SSA value operand, and the + instruction is polymorphic over the possible concrete types that the type + variable can assume. + + 3. An :py:class:`ImmediateType` instance indicates an immediate operand + whose value is encoded in the instruction itself rather than being passed + as an SSA value. + + """ + def __init__(self, name, typ, doc=''): + self.name = name + self.typ = typ + self.__doc__ = doc + + def get_doc(self): + if self.__doc__: + return self.__doc__ + else: + return self.typ.__doc__ + +class Instruction(object): + """ + An instruction. + + The operands to the instruction are specified as two tuples: ``ins`` and + ``outs``. Since the Python singleton tuple syntax is a bit awkward, it is + allowed to specify a singleton as just the operand itself, i.e., `ins=x` and + `ins=(x,)` are both allowed and mean the same thing. + + :param name: Instruction mnemonic, also becomes opcode name. + :param doc: Documentation string. + :param ins: Tuple of input operands. This can be a mix of SSA value operands + and immediate operands. + :param outs: Tuple of output operands. The output operands can't be + immediates. + """ + + def __init__(self, name, doc, ins=(), outs=(), **kwargs): + self.name = name + self.__doc__ = doc + self.ins = self._to_operand_tuple(ins) + self.outs = self._to_operand_tuple(outs) + + @staticmethod + def _to_operand_tuple(x): + # Allow a single Operand instance instead of the awkward singleton tuple + # syntax. + if isinstance(x, Operand): + x = (x,) + else: + x = tuple(x) + for op in x: + assert isinstance(op, Operand) + return x From a2db4b680e49c2244a58f8355a50c5652f9d0a6b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 29 Mar 2016 15:22:16 -0700 Subject: [PATCH 0029/3084] Move instruction definitions into meta. Use the meta language to define instructions, just insert 'autoinst' references in langref. --- cranelift/docs/langref.rst | 279 ++++++------------------------------- meta/cretonne/base.py | 275 ++++++++++++++++++++++++++++++++++++ 2 files changed, 314 insertions(+), 240 deletions(-) create mode 100644 meta/cretonne/base.py diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 845bc4da37..fed88a819b 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -625,7 +625,7 @@ Operations The remaining instruction set is mostly arithmetic. A few instructions have variants that take immediate operands (e.g., -:inst:`and` / :inst:`and_imm`), but in general an instruction is required to +:inst:`band` / :inst:`band_imm`), but in general an instruction is required to load a constant into an SSA value. .. inst:: a = iconst N @@ -747,43 +747,10 @@ Integer operations sle ule Less than or equal ====== ======== ========= -.. inst:: a = iadd x, y - - Wrapping integer addition: :math:`a := x + y \pmod{2^B}`. This instruction - does not depend on the signed/unsigned interpretation of the operands. - - Polymorphic over all integer types (vector and scalar). - -.. inst:: a = iadd_imm x, Imm - - Add immediate integer. - - Same as :inst:`iadd`, but one operand is an immediate constant. - - :arg iN x: Dynamic addend. - :arg Imm: Immediate addend. - - Polymorphic over all scalar integer types. - -.. inst:: a = isub x, y - - Wrapping integer subtraction: :math:`a := x - y \pmod{2^B}`. This - instruction does not depend on the signed/unsigned interpretation of the - operands. - - Polymorphic over all integer types (vector and scalar). - -.. inst:: a = isub_imm Imm, x - - Immediate subtraction. - - Also works as integer negation when :math:`Imm = 0`. Use :inst:`iadd_imm` with a - negative immediate operand for the reverse immediate subtraction. - - :arg Imm: Immediate minuend. - :arg iN x: Dynamic subtrahend. - - Polymorphic over all scalar integer types. +.. autoinst:: iadd +.. autoinst:: iadd_imm +.. autoinst:: isub +.. autoinst:: isub_imm .. todo:: Integer overflow arithmetic @@ -791,78 +758,22 @@ Integer operations implement larger integer types efficiently. It should also be possible to legalize :type:`i64` arithmetic to terms of :type:`i32` operations. -.. inst:: a = imul x, y - - Wrapping integer multiplication: :math:`a := x y \pmod{2^B}`. This - instruction does not depend on the signed/unsigned interpretation of the - operands. - - Polymorphic over all integer types (vector and scalar). - -.. inst:: a = imul_imm x, Imm - - Integer multiplication by immediate constant. - - Polymorphic over all scalar integer types. +.. autoinst:: imul +.. autoinst:: imul_imm .. todo:: Larger multiplication results. For example, ``smulx`` which multiplies :type:`i32` operands to produce a :type:`i64` result. Alternatively, ``smulhi`` and ``smullo`` pairs. -.. inst:: a = udiv x, y - - Unsigned integer division: :math:`a := \lfloor {x \over y} \rfloor`. This - operation traps if the divisor is zero. - -.. inst:: a = udiv_imm x, Imm - - Unsigned integer division by an immediate constant. - - This instruction never traps because a divisor of zero is not allowed. - -.. inst:: a = sdiv x, y - - Signed integer division rounded toward zero: :math:`a := sign(xy) \lfloor - {|x| \over |y|}\rfloor`. This operation traps if the divisor is zero, or if - the result is not representable in :math:`B` bits two's complement. This only - happens when :math:`x = -2^{B-1}, y = -1`. - -.. inst:: a = sdiv_imm x, Imm - - Signed integer division by an immediate constant. - - This instruction never traps because a divisor of -1 or 0 is not allowed. - -.. inst:: a = urem x, y - - Unsigned integer remainder. - - This operation traps if the divisor is zero. - -.. inst:: a = urem_imm x, Imm - - Unsigned integer remainder with immediate divisor. - - This instruction never traps because a divisor of zero is not allowed. - -.. inst:: a = srem x, y - - Signed integer remainder. - - This operation traps if the divisor is zero. - - .. todo:: Integer remainder vs modulus. - - Clarify whether the result has the sign of the divisor or the dividend. - Should we add a ``smod`` instruction for the case where the result has - the same sign as the divisor? - -.. inst:: a = srem_imm x, Imm - - Signed integer remainder with immediate divisor. - - This instruction never traps because a divisor of 0 or -1 is not allowed. +.. autoinst:: udiv +.. autoinst:: udiv_imm +.. autoinst:: sdiv +.. autoinst:: sdiv_imm +.. autoinst:: urem +.. autoinst:: urem_imm +.. autoinst:: srem +.. autoinst:: srem_imm .. todo:: Minimum / maximum. @@ -880,153 +791,41 @@ Integer operations Bitwise operations ------------------ -.. inst:: a = and x, y +The bitwise operations and operate on any value type: Integers, floating point +numbers, and booleans. When operating on integer or floating point types, the +bitwise operations are working on the binary representation of the values. When +operating on boolean values, the bitwise operations work as logical operators. - Bitwise and. - - :rtype: bool, iB, iBxN, fBxN? - -.. inst:: a = or x, y - - Bitwise or. - - :rtype: bool, iB, iBxN, fBxN? - -.. inst:: a = xor x, y - - Bitwise xor. - - :rtype: bool, iB, iBxN, fBxN? - -.. inst:: a = not x - - Bitwise not. - - :rtype: bool, iB, iBxN, fBxN? +.. autoinst:: band +.. autoinst:: bor +.. autoinst:: bxor +.. autoinst:: bnot .. todo:: Redundant bitwise operators. ARM has instructions like ``bic(x,y) = x & ~y``, ``orn(x,y) = x | ~y``, and ``eon(x,y) = x ^ ~y``. -.. inst:: a = rotl x, y +The shift and rotate operations only work on integer types (scalar and vector). +The shift amount does not have to be the same type as the value being shifted. +Only the low `B` bits of the shift amount is significant. - Rotate left. +When operating on an integer vector type, the shift amount is still a scalar +type, and all the lanes are shifted the same amount. The shift amount is masked +to the number of bits in a *lane*, not the full size of the vector type. - Rotate the bits in ``x`` by ``y`` places. +.. autoinst:: rotl +.. autoinst:: rotr +.. autoinst:: ishl +.. autoinst:: ushr +.. autoinst:: sshr - :arg T x: Integer value to be rotated. - :arg iN y: Number of bits to shift. Any scalar integer type, not necessarily - the same type as ``x``. - :rtype: Same type as ``x``. - -.. inst:: a = rotr x, y - - Rotate right. - - Rotate the bits in ``x`` by ``y`` places. - - :arg T x: Integer value to be rotated. - :arg iN y: Number of bits to shift. Any scalar integer type, not necessarily - the same type as ``x``. - :rtype: Same type as ``x``. - -.. inst:: a = ishl x, y - - Integer shift left. Shift the bits in ``x`` towards the MSB by ``y`` - places. Shift in zero bits to the LSB. - - The shift amount is masked to the size of ``x``. - - :arg T x: Integer value to be shifted. - :arg iN y: Number of bits to shift. Any scalar integer type, not necessarily - the same type as ``x``. - :rtype: Same type as ``x``. - - When shifting a B-bits integer type, this instruction computes: - - .. math:: - s &:= y \pmod B, \\ - a &:= x \cdot 2^s \pmod{2^B}. - - .. todo:: Add ``ishl_imm`` variant with an immediate ``y``. - -.. inst:: a = ushr x, y - - Unsigned shift right. Shift bits in ``x`` towards the LSB by ``y`` places, - shifting in zero bits to the MSB. Also called a *logical shift*. - - The shift amount is masked to the size of the register. - - :arg T x: Integer value to be shifted. - :arg iN y: Number of bits to shift. Can be any scalar integer type, not - necessarily the same type as ``x``. - :rtype: Same type as ``x``. - - When shifting a B-bits integer type, this instruction computes: - - .. math:: - s &:= y \pmod B, \\ - a &:= \lfloor x \cdot 2^{-s} \rfloor. - - .. todo:: Add ``ushr_imm`` variant with an immediate ``y``. - -.. inst:: a = sshr x, y - - Signed shift right. Shift bits in ``x`` towards the LSB by ``y`` places, - shifting in sign bits to the MSB. Also called an *arithmetic shift*. - - The shift amount is masked to the size of the register. - - :arg T x: Integer value to be shifted. - :arg iN y: Number of bits to shift. Can be any scalar integer type, not - necessarily the same type as ``x``. - :rtype: Same type as ``x``. - - .. todo:: Add ``sshr_imm`` variant with an immediate ``y``. - -.. inst:: a = clz x - - Count leading zero bits. - - :arg x: Integer value. - :rtype: :type:`i8` - - Starting from the MSB in ``x``, count the number of zero bits before - reaching the first one bit. When ``x`` is zero, returns the size of x in - bits. - -.. inst:: a = cls x - - Count leading sign bits. - - :arg x: Integer value. - :rtype: :type:`i8` - - Starting from the MSB after the sign bit in ``x``, count the number of - consecutive bits identical to the sign bit. When ``x`` is 0 or -1, returns - one less than the size of x in bits. - -.. inst:: a = ctz x - - Count trailing zeros. - - :arg x: Integer value. - :rtype: :type:`i8` - - Starting from the LSB in ``x``, count the number of zero bits before - reaching the first one bit. When ``x`` is zero, returns the size of x in - bits. - -.. inst:: a = popcnt x - - Population count - - :arg x: Integer value. - :rtype: :type:`i8` - - Count the number of one bits in ``x``. +The bit-counting instructions below are scalar only. +.. autoinst:: clz +.. autoinst:: cls +.. autoinst:: ctz +.. autoinst:: popcnt Floating point operations ------------------------- diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py new file mode 100644 index 0000000000..24488e7cc2 --- /dev/null +++ b/meta/cretonne/base.py @@ -0,0 +1,275 @@ +""" +Cretonne base instruction set. + +This module defines the basic Cretonne instruction set that all targets support. +""" +from . import TypeVar, Operand, Instruction +from types import i8 +from immediates import imm64 + +Int = TypeVar('Int', 'A scalar or vector integer type') +iB = TypeVar('iB', 'A scalar integer type') + +a = Operand('a', Int) +x = Operand('x', Int) +y = Operand('y', Int) + +iadd = Instruction('iadd', r""" + Wrapping integer addition: :math:`a := x + y \pmod{2^B}`. + + This instruction does not depend on the signed/unsigned interpretation of + the operands. + """, + ins=(x,y), outs=a) + +isub = Instruction('isub', r""" + Wrapping integer subtraction: :math:`a := x - y \pmod{2^B}`. + + This instruction does not depend on the signed/unsigned interpretation of + the operands. + """, + ins=(x,y), outs=a) + +imul = Instruction('imul', r""" + Wrapping integer multiplication: :math:`a := x y \pmod{2^B}`. + + This instruction does not depend on the signed/unsigned interpretation of + the + operands. + + Polymorphic over all integer types (vector and scalar). + """, + ins=(x,y), outs=a) + +udiv = Instruction('udiv', r""" + Unsigned integer division: :math:`a := \lfloor {x \over y} \rfloor`. + + This operation traps if the divisor is zero. + """, + ins=(x,y), outs=a) + +sdiv = Instruction('sdiv', r""" + Signed integer division rounded toward zero: :math:`a := sign(xy) \lfloor + {|x| \over |y|}\rfloor`. + + This operation traps if the divisor is zero, or if the result is not + representable in :math:`B` bits two's complement. This only happens when + :math:`x = -2^{B-1}, y = -1`. + """, + ins=(x,y), outs=a) + +urem = Instruction('urem', """ + Unsigned integer remainder. + + This operation traps if the divisor is zero. + """, + ins=(x,y), outs=a) + +srem = Instruction('srem', """ + Signed integer remainder. + + This operation traps if the divisor is zero. + + .. todo:: Integer remainder vs modulus. + + Clarify whether the result has the sign of the divisor or the dividend. + Should we add a ``smod`` instruction for the case where the result has + the same sign as the divisor? + """, + ins=(x,y), outs=a) + +a = Operand('a', iB) +x = Operand('x', iB) +Y = Operand('Y', imm64) + +iadd_imm = Instruction('iadd_imm', """ + Add immediate integer. + + Same as :inst:`iadd`, but one operand is an immediate constant. + + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(x,Y), outs=a) + +imul_imm = Instruction('imul_imm', """ + Integer multiplication by immediate constant. + + Polymorphic over all scalar integer types. + """, + ins=(x,Y), outs=a) + +udiv_imm = Instruction('udiv_imm', """ + Unsigned integer division by an immediate constant. + + This instruction never traps because a divisor of zero is not allowed. + """, + ins=(x,Y), outs=a) + +sdiv_imm = Instruction('sdiv_imm', """ + Signed integer division by an immediate constant. + + This instruction never traps because a divisor of -1 or 0 is not allowed. + """, + ins=(x,Y), outs=a) + +urem_imm = Instruction('urem_imm', """ + Unsigned integer remainder with immediate divisor. + + This instruction never traps because a divisor of zero is not allowed. + """, + ins=(x,Y), outs=a) + +srem_imm = Instruction('srem_imm', """ + Signed integer remainder with immediate divisor. + + This instruction never traps because a divisor of 0 or -1 is not allowed. + """, + ins=(x,Y), outs=a) + +# Swap x and y for isub_imm. +X = Operand('X', imm64) +y = Operand('y', iB) + +isub_imm = Instruction('isub_imm', """ + Immediate wrapping subtraction: :math:`a := X - y \pmod{2^B}`. + + Also works as integer negation when :math:`X = 0`. Use :inst:`iadd_imm` with a + negative immediate operand for the reverse immediate subtraction. + + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(X,y), outs=a) + +# +# Bitwise operations. +# + +# TODO: Which types should permit boolean operations? Any reason to restrict? +bits = TypeVar('bits', 'Any integer, float, or boolean scalar or vector type') +x = Operand('x', bits) +y = Operand('y', bits) +a = Operand('a', bits) + +band = Instruction('band', """ + Bitwise and. + """, + ins=(x,y), outs=a) + +bor = Instruction('bor', """ + Bitwise or. + """, + ins=(x,y), outs=a) + +bxor = Instruction('bxor', """ + Bitwise xor. + """, + ins=(x,y), outs=a) + +bnot = Instruction('bnot', """ + Bitwise not. + """, + ins=x, outs=a) + +# Shift/rotate. +x = Operand('x', Int, doc='Scalar or vector value to shift') +y = Operand('y', iB, doc='Number of bits to shift') +a = Operand('a', Int) + +rotl = Instruction('rotl', r""" + Rotate left. + + Rotate the bits in ``x`` by ``y`` places. + """, + ins=(x,y), outs=a) + +rotr = Instruction('rotr', r""" + Rotate right. + + Rotate the bits in ``x`` by ``y`` places. + """, + ins=(x,y), outs=a) + +ishl = Instruction('ishl', r""" + Integer shift left. Shift the bits in ``x`` towards the MSB by ``y`` + places. Shift in zero bits to the LSB. + + The shift amount is masked to the size of ``x``. + + When shifting a B-bits integer type, this instruction computes: + + .. math:: + s &:= y \pmod B, \\ + a &:= x \cdot 2^s \pmod{2^B}. + + .. todo:: Add ``ishl_imm`` variant with an immediate ``y``. + """, + ins=(x,y), outs=a) + +ushr = Instruction('ushr', r""" + Unsigned shift right. Shift bits in ``x`` towards the LSB by ``y`` places, + shifting in zero bits to the MSB. Also called a *logical shift*. + + The shift amount is masked to the size of the register. + + When shifting a B-bits integer type, this instruction computes: + + .. math:: + s &:= y \pmod B, \\ + a &:= \lfloor x \cdot 2^{-s} \rfloor. + + .. todo:: Add ``ushr_imm`` variant with an immediate ``y``. + """, + ins=(x,y), outs=a) + +sshr = Instruction('sshr', r""" + Signed shift right. Shift bits in ``x`` towards the LSB by ``y`` places, + shifting in sign bits to the MSB. Also called an *arithmetic shift*. + + The shift amount is masked to the size of the register. + + .. todo:: Add ``sshr_imm`` variant with an immediate ``y``. + """, + ins=(x,y), outs=a) + +# +# Bit counting. +# + +x = Operand('x', iB) +a = Operand('a', i8) + +clz = Instruction('clz', r""" + Count leading zero bits. + + Starting from the MSB in ``x``, count the number of zero bits before + reaching the first one bit. When ``x`` is zero, returns the size of x in + bits. + """, + ins=x, outs=a) + +cls = Instruction('cls', r""" + Count leading sign bits. + + Starting from the MSB after the sign bit in ``x``, count the number of + consecutive bits identical to the sign bit. When ``x`` is 0 or -1, returns + one less than the size of x in bits. + """, + ins=x, outs=a) + +ctz = Instruction('ctz', r""" + Count trailing zeros. + + Starting from the LSB in ``x``, count the number of zero bits before + reaching the first one bit. When ``x`` is zero, returns the size of x in + bits. + """, + ins=x, outs=a) + +popcnt = Instruction('popcnt', r""" + Population count + + Count the number of one bits in ``x``. + """, + ins=x, outs=a) From ad07f673311c0b76b11a68883ccf8bbbf8c36376 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 31 Mar 2016 14:18:02 -0700 Subject: [PATCH 0030/3084] Move constant instructions into meta. Add new immediate types for floating point and vector immediates. Use new immediates to define the constant value instructions in meta. Split the fconst instruction into two: f32const and f64const. This prevents confusion about the interpretation of 64 immediate bits when generating an f32 constant. Add an immvector ImmediateType. This immediate type is variable length, and provides all the bits of a SIMD vector directly. --- cranelift/docs/cton_lexer.py | 2 +- cranelift/docs/example.cton | 4 +-- cranelift/docs/langref.rst | 46 +++++++++++++++---------------- meta/cretonne/base.py | 52 ++++++++++++++++++++++++++++++++++-- meta/cretonne/immediates.py | 13 +++++++++ 5 files changed, 87 insertions(+), 30 deletions(-) diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py index 33f769dfc6..c12db5aea7 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/cton_lexer.py @@ -23,7 +23,7 @@ class CretonneLexer(RegexLexer): # Numbers. (r'[-+]?0[xX][0-9a-fA-F]+', Number.Hex), (r'[-+]?0[xX][0-9a-fA-F]*\.[0-9a-fA-F]*([pP]\d+)?', Number.Hex), - (r'[-+]?\d+\.\d+([eE]\d+)?', Number.Float), + (r'[-+]?(\d+\.\d+([eE]\d+)?|[sq]NaN|Inf)', Number.Float), (r'[-+]?\d+', Number.Integer), # Reserved words. (keywords('function', 'entry'), Keyword), diff --git a/cranelift/docs/example.cton b/cranelift/docs/example.cton index c0001fe7c6..190d3a25fd 100644 --- a/cranelift/docs/example.cton +++ b/cranelift/docs/example.cton @@ -2,7 +2,7 @@ function average(i32, i32) -> f32 { ss1 = stack_slot 8, align 4 ; Stack slot for ``sum``. entry ebb1(v1: i32, v2: i32): - v3 = fconst.f64 0.0 + v3 = f64const 0x0.0 stack_store v3, ss1 brz v2, ebb3 ; Handle count == 0. v4 = iconst.i32 0 @@ -26,6 +26,6 @@ ebb2(v5: i32): return v17 ebb3: - v100 = fconst.f32 0x7fc00000 ; 0/0 = NaN + v100 = f32const qNaN return v100 } diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index fed88a819b..3ef460de06 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -212,6 +212,23 @@ indicate the different kinds of immediate operands on an instruction. signed two's complement integer. Instruction encodings may limit the valid range. +.. type:: ieee32 + + A 32-bit immediate floating point number in the IEEE 754-2008 binary32 + interchange format. All bit patterns are allowed. + +.. type:: ieee64 + + A 64-bit immediate floating point number in the IEEE 754-2008 binary64 + interchange format. All bit patterns are allowed. + +.. type:: immvector + + An immediate SIMD vector. This operand supplies all the bits of a SIMD + type, so it can have different sizes depending on the type produced. The + bits of the operand are interpreted as if the SIMD vector was loaded from + memory containing the immediate. + Control flow ============ @@ -628,31 +645,10 @@ A few instructions have variants that take immediate operands (e.g., :inst:`band` / :inst:`band_imm`), but in general an instruction is required to load a constant into an SSA value. -.. inst:: a = iconst N - - Integer constant. - - Create a scalar integer SSA value with an immediate constant value, or an - integer vector where all the lanes have the same value. - - :result Int a: Constant value. - -.. inst:: a = fconst N - - Floating point constant. - - Create a :type:`f32` or :type:`f64` SSA value with an immediate constant - value, or a floating point vector where all the lanes have the same value. - - :result Float a: Constant value. - -.. inst:: a = vconst N - - Vector constant (floating point or integer). - - Create a SIMD vector value where the lanes don't have to be identical. - - :result TxN a: Constant value. +.. autoinst:: iconst +.. autoinst:: f32const +.. autoinst:: f64const +.. autoinst:: vconst .. inst:: a = select c, x, y diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 24488e7cc2..a926829d7c 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -4,11 +4,59 @@ Cretonne base instruction set. This module defines the basic Cretonne instruction set that all targets support. """ from . import TypeVar, Operand, Instruction -from types import i8 -from immediates import imm64 +from types import i8, f32, f64 +from immediates import imm64, ieee32, ieee64, immvector Int = TypeVar('Int', 'A scalar or vector integer type') iB = TypeVar('iB', 'A scalar integer type') +TxN = TypeVar('%Tx%N', 'A SIMD vector type') + +# +# Materializing constants. +# + +N = Operand('N', imm64) +a = Operand('a', Int, doc='A constant integer scalar or vector value') +iconst = Instruction('iconst', r""" + Integer constant. + + Create a scalar integer SSA value with an immediate constant value, or an + integer vector where all the lanes have the same value. + """, + ins=N, outs=a) + +N = Operand('N', ieee32) +a = Operand('a', f32, doc='A constant integer scalar or vector value') +f32const = Instruction('f32const', r""" + Floating point constant. + + Create a :type:`f32` SSA value with an immediate constant value, or a + floating point vector where all the lanes have the same value. + """, + ins=N, outs=a) + +N = Operand('N', ieee64) +a = Operand('a', f64, doc='A constant integer scalar or vector value') +f64const = Instruction('f64const', r""" + Floating point constant. + + Create a :type:`f64` SSA value with an immediate constant value, or a + floating point vector where all the lanes have the same value. + """, + ins=N, outs=a) + +N = Operand('N', immvector) +a = Operand('a', TxN, doc='A constant vector value') +vconst = Instruction('vconst', r""" + Vector constant (floating point or integer). + + Create a SIMD vector value where the lanes don't have to be identical. + """, + ins=N, outs=a) + +# +# Integer arithmetic +# a = Operand('a', Int) x = Operand('x', Int) diff --git a/meta/cretonne/immediates.py b/meta/cretonne/immediates.py index 6b499d5215..2735cd6be1 100644 --- a/meta/cretonne/immediates.py +++ b/meta/cretonne/immediates.py @@ -10,3 +10,16 @@ from . import ImmediateType #: This type of immediate integer can interact with SSA values with any #: :py:class:`cretonne.IntType` type. imm64 = ImmediateType('imm64', 'A 64-bit immediate integer.') + +#: A 32-bit immediate floating point operand. +#: +#: IEEE 754-2008 binary32 interchange format. +ieee32 = ImmediateType('ieee32', 'A 32-bit immediate floating point number.') + +#: A 64-bit immediate floating point operand. +#: +#: IEEE 754-2008 binary64 interchange format. +ieee64 = ImmediateType('ieee64', 'A 64-bit immediate floating point number.') + +#: A large SIMD vector constant. +immvector = ImmediateType('immvector', 'An immediate SIMD vector.') From 524eb77185d2c2e3646e81bbc8fff445a34016f9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 31 Mar 2016 15:22:23 -0700 Subject: [PATCH 0031/3084] Implement IEEE immediates for binary32 and binary64. Clarify the textual encoding of floating point numbers. Don't allow decimal floating point since conversion to/from binary can produce rounding problems on some (buggy) systems. --- cranelift/docs/langref.rst | 35 +++++ cranelift/src/libcretonne/immediates.rs | 184 ++++++++++++++++++++++++ 2 files changed, 219 insertions(+) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 3ef460de06..8acb5e793a 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -212,6 +212,9 @@ indicate the different kinds of immediate operands on an instruction. signed two's complement integer. Instruction encodings may limit the valid range. + In the textual format, :type:`imm64` immediates appear as decimal or + hexadecimal literals using the same syntax as C. + .. type:: ieee32 A 32-bit immediate floating point number in the IEEE 754-2008 binary32 @@ -229,6 +232,38 @@ indicate the different kinds of immediate operands on an instruction. bits of the operand are interpreted as if the SIMD vector was loaded from memory containing the immediate. +The two IEEE floating point immediate types :type:`ieee32` and :type:`ieee64` +are displayed as hexadecimal floating point literals in the textual IL format. +Decimal floating point literals are not allowed because some computer systems +can round differently when converting to binary. The hexadecimal floating point +format is mostly the same as the one used by C99, but extended to represent all +NaN bit patterns: + +Normal numbers + Compatible with C99: ``-0x1.Tpe`` where ``T`` are the trailing + significand bits encoded as hexadecimal, and ``e`` is the unbiased exponent + as a decimal number. :type:`ieee32` has 23 trailing significand bits. They + are padded with an extra LSB to produce 6 hexadecimal digits. This is not + necessary for :type:`ieee64` which has 52 trailing significand bits + forming 13 hexadecimal digits with no padding. + +Subnormal numbers + Compatible with C99: ``-0x0.Tpemin`` where ``T`` are the trailing + significand bits encoded as hexadecimal, and ``emin`` is the minimum exponent + as a decimal number. + +Infinities + Either ``-Inf`` or ``Inf``. + +Quiet NaNs + Quiet NaNs have the MSB of the trailing significand set. If the remaining + bits of the trailing significand are all zero, the value is displayed as + ``-qNaN`` or ``qNaN``. Otherwise, ``-qNaN:0xT`` where ``T`` are the + trailing significand bits encoded as hexadecimal. + +Signaling NaNs + Displayed as ``-sNaN:0xT``. + Control flow ============ diff --git a/cranelift/src/libcretonne/immediates.rs b/cranelift/src/libcretonne/immediates.rs index a60291b164..0e1e94ba07 100644 --- a/cranelift/src/libcretonne/immediates.rs +++ b/cranelift/src/libcretonne/immediates.rs @@ -6,6 +6,7 @@ //! module in the meta language. use std::fmt::{self, Display, Formatter}; +use std::mem; /// 64-bit immediate integer operand. /// @@ -36,9 +37,123 @@ impl Display for Imm64 { } } + +/// An IEEE binary32 immediate floating point value. +/// +/// All bit patterns are allowed. +pub struct Ieee32(f32); + +/// An IEEE binary64 immediate floating point value. +/// +/// All bit patterns are allowed. +pub struct Ieee64(f64); + +// Format a floating point number in a way that is reasonably human-readable, and that can be +// converted back to binary without any rounding issues. The hexadecimal formatting of normal and +// subnormal numbers is compatible with C99 and the printf "%a" format specifier. The NaN and Inf +// formats are not supported by C99. +// +// The encoding parameters are: +// +// w - exponent field width in bits +// t - trailing significand field width in bits +// +fn format_float(bits: u64, w: u8, t: u8, f: &mut Formatter) -> fmt::Result { + assert!(w > 0 && w <= 16, "Invalid exponent range"); + assert!(1 + w + t <= 64, "Too large IEEE format for u64"); + + let max_e_bits = (1u64 << w) - 1; + let t_bits = bits & ((1u64 << t) - 1); // Trailing significand. + let e_bits = (bits >> t) & max_e_bits; // Biased exponent. + let sign_bit = (bits >> w + t) & 1; + + let bias: i32 = (1 << (w - 1)) - 1; + let e = e_bits as i32 - bias; // Unbiased exponent. + let emin = 1 - bias; // Minimum exponent. + + // How many hexadecimal digits are needed for the trailing significand? + let digits = (t + 3) / 4; + // Trailing significand left-aligned in `digits` hexadecimal digits. + let left_t_bits = t_bits << (4 * digits - t); + + // All formats share the leading sign. + if sign_bit != 0 { + try!(write!(f, "-")); + } + + if e_bits == 0 { + if t_bits == 0 { + // Zero. + write!(f, "0.0") + } else { + // Subnormal. + write!(f, "0x0.{0:01$x}p{2}", left_t_bits, digits as usize, emin) + } + } else if e_bits == max_e_bits { + if t_bits == 0 { + // Infinity. + write!(f, "Inf") + } else { + // NaN. + let payload = t_bits & ((1 << (t - 1)) - 1); + if t_bits & (1 << (t - 1)) != 0 { + // Quiet NaN. + if payload != 0 { + write!(f, "qNaN:0x{:x}", payload) + } else { + write!(f, "qNaN") + } + } else { + // Signaling NaN. + write!(f, "sNaN:0x{:x}", payload) + } + } + } else { + // Normal number. + write!(f, "0x1.{0:01$x}p{2}", left_t_bits, digits as usize, e) + } +} + +impl Ieee32 { + pub fn new(x: f32) -> Ieee32 { + Ieee32(x) + } + + /// Construct Ieee32 immediate from raw bits. + pub fn new_from_bits(x: u32) -> Ieee32 { + Ieee32(unsafe { mem::transmute(x) }) + } +} + +impl Display for Ieee32 { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + let bits: u32 = unsafe { mem::transmute(self.0) }; + format_float(bits as u64, 8, 23, f) + } +} + +impl Ieee64 { + pub fn new(x: f64) -> Ieee64 { + Ieee64(x) + } + + /// Construct Ieee64 immediate from raw bits. + pub fn new_from_bits(x: u64) -> Ieee64 { + Ieee64(unsafe { mem::transmute(x) }) + } +} + +impl Display for Ieee64 { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + let bits: u64 = unsafe { mem::transmute(self.0) }; + format_float(bits, 11, 52, f) + } +} + #[cfg(test)] mod tests { use super::*; + use std::{f32, f64}; #[test] fn format_imm64() { @@ -50,4 +165,73 @@ mod tests { assert_eq!(format!("{}", Imm64(0xffff)), "0xffff"); assert_eq!(format!("{}", Imm64(0x10000)), "0x0001_0000"); } + + #[test] + fn format_ieee32() { + assert_eq!(format!("{}", Ieee32::new(0.0)), "0.0"); + assert_eq!(format!("{}", Ieee32::new(-0.0)), "-0.0"); + assert_eq!(format!("{}", Ieee32::new(1.0)), "0x1.000000p0"); + assert_eq!(format!("{}", Ieee32::new(1.5)), "0x1.800000p0"); + assert_eq!(format!("{}", Ieee32::new(0.5)), "0x1.000000p-1"); + assert_eq!(format!("{}", Ieee32::new(f32::EPSILON)), "0x1.000000p-23"); + assert_eq!(format!("{}", Ieee32::new(f32::MIN)), "-0x1.fffffep127"); + assert_eq!(format!("{}", Ieee32::new(f32::MAX)), "0x1.fffffep127"); + // Smallest positive normal number. + assert_eq!(format!("{}", Ieee32::new(f32::MIN_POSITIVE)), + "0x1.000000p-126"); + // Subnormals. + assert_eq!(format!("{}", Ieee32::new(f32::MIN_POSITIVE / 2.0)), + "0x0.800000p-126"); + assert_eq!(format!("{}", Ieee32::new(f32::MIN_POSITIVE * f32::EPSILON)), + "0x0.000002p-126"); + assert_eq!(format!("{}", Ieee32::new(f32::INFINITY)), "Inf"); + assert_eq!(format!("{}", Ieee32::new(f32::NEG_INFINITY)), "-Inf"); + assert_eq!(format!("{}", Ieee32::new(f32::NAN)), "qNaN"); + assert_eq!(format!("{}", Ieee32::new(-f32::NAN)), "-qNaN"); + // Construct some qNaNs with payloads. + assert_eq!(format!("{}", Ieee32::new_from_bits(0x7fc00001)), "qNaN:0x1"); + assert_eq!(format!("{}", Ieee32::new_from_bits(0x7ff00001)), + "qNaN:0x300001"); + // Signaling NaNs. + assert_eq!(format!("{}", Ieee32::new_from_bits(0x7f800001)), "sNaN:0x1"); + assert_eq!(format!("{}", Ieee32::new_from_bits(0x7fa00001)), + "sNaN:0x200001"); + } + + #[test] + fn format_ieee64() { + assert_eq!(format!("{}", Ieee64::new(0.0)), "0.0"); + assert_eq!(format!("{}", Ieee64::new(-0.0)), "-0.0"); + assert_eq!(format!("{}", Ieee64::new(1.0)), "0x1.0000000000000p0"); + assert_eq!(format!("{}", Ieee64::new(1.5)), "0x1.8000000000000p0"); + assert_eq!(format!("{}", Ieee64::new(0.5)), "0x1.0000000000000p-1"); + assert_eq!(format!("{}", Ieee64::new(f64::EPSILON)), + "0x1.0000000000000p-52"); + assert_eq!(format!("{}", Ieee64::new(f64::MIN)), + "-0x1.fffffffffffffp1023"); + assert_eq!(format!("{}", Ieee64::new(f64::MAX)), + "0x1.fffffffffffffp1023"); + // Smallest positive normal number. + assert_eq!(format!("{}", Ieee64::new(f64::MIN_POSITIVE)), + "0x1.0000000000000p-1022"); + // Subnormals. + assert_eq!(format!("{}", Ieee64::new(f64::MIN_POSITIVE / 2.0)), + "0x0.8000000000000p-1022"); + assert_eq!(format!("{}", Ieee64::new(f64::MIN_POSITIVE * f64::EPSILON)), + "0x0.0000000000001p-1022"); + assert_eq!(format!("{}", Ieee64::new(f64::INFINITY)), "Inf"); + assert_eq!(format!("{}", Ieee64::new(f64::NEG_INFINITY)), "-Inf"); + assert_eq!(format!("{}", Ieee64::new(f64::NAN)), "qNaN"); + assert_eq!(format!("{}", Ieee64::new(-f64::NAN)), "-qNaN"); + // Construct some qNaNs with payloads. + assert_eq!(format!("{}", Ieee64::new_from_bits(0x7ff8000000000001)), + "qNaN:0x1"); + assert_eq!(format!("{}", Ieee64::new_from_bits(0x7ffc000000000001)), + "qNaN:0x4000000000001"); + // Signaling NaNs. + assert_eq!(format!("{}", Ieee64::new_from_bits(0x7ff0000000000001)), + "sNaN:0x1"); + assert_eq!(format!("{}", Ieee64::new_from_bits(0x7ff4000000000001)), + "sNaN:0x4000000000001"); + } } From 4b265c2ee32ea9c8fbdc090a2b726d06bb4cf2b9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Apr 2016 10:20:32 -0700 Subject: [PATCH 0032/3084] Display quiet NaNs as 'NaN'. This is recommended by IEEE 754-2008. We still distinguish signaling NaNs with 'sNaN'. --- cranelift/docs/langref.rst | 7 +++++-- cranelift/src/libcretonne/immediates.rs | 20 ++++++++++---------- 2 files changed, 15 insertions(+), 12 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 8acb5e793a..47f7fa8833 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -247,6 +247,9 @@ Normal numbers necessary for :type:`ieee64` which has 52 trailing significand bits forming 13 hexadecimal digits with no padding. +Zeros + Positive and negative zero are displayed as ``0.0`` and ``-0.0`` respectively. + Subnormal numbers Compatible with C99: ``-0x0.Tpemin`` where ``T`` are the trailing significand bits encoded as hexadecimal, and ``emin`` is the minimum exponent @@ -258,8 +261,8 @@ Infinities Quiet NaNs Quiet NaNs have the MSB of the trailing significand set. If the remaining bits of the trailing significand are all zero, the value is displayed as - ``-qNaN`` or ``qNaN``. Otherwise, ``-qNaN:0xT`` where ``T`` are the - trailing significand bits encoded as hexadecimal. + ``-NaN`` or ``NaN``. Otherwise, ``-NaN:0xT`` where ``T`` are the trailing + significand bits encoded as hexadecimal. Signaling NaNs Displayed as ``-sNaN:0xT``. diff --git a/cranelift/src/libcretonne/immediates.rs b/cranelift/src/libcretonne/immediates.rs index 0e1e94ba07..efd84cd606 100644 --- a/cranelift/src/libcretonne/immediates.rs +++ b/cranelift/src/libcretonne/immediates.rs @@ -99,9 +99,9 @@ fn format_float(bits: u64, w: u8, t: u8, f: &mut Formatter) -> fmt::Result { if t_bits & (1 << (t - 1)) != 0 { // Quiet NaN. if payload != 0 { - write!(f, "qNaN:0x{:x}", payload) + write!(f, "NaN:0x{:x}", payload) } else { - write!(f, "qNaN") + write!(f, "NaN") } } else { // Signaling NaN. @@ -186,12 +186,12 @@ mod tests { "0x0.000002p-126"); assert_eq!(format!("{}", Ieee32::new(f32::INFINITY)), "Inf"); assert_eq!(format!("{}", Ieee32::new(f32::NEG_INFINITY)), "-Inf"); - assert_eq!(format!("{}", Ieee32::new(f32::NAN)), "qNaN"); - assert_eq!(format!("{}", Ieee32::new(-f32::NAN)), "-qNaN"); + assert_eq!(format!("{}", Ieee32::new(f32::NAN)), "NaN"); + assert_eq!(format!("{}", Ieee32::new(-f32::NAN)), "-NaN"); // Construct some qNaNs with payloads. - assert_eq!(format!("{}", Ieee32::new_from_bits(0x7fc00001)), "qNaN:0x1"); + assert_eq!(format!("{}", Ieee32::new_from_bits(0x7fc00001)), "NaN:0x1"); assert_eq!(format!("{}", Ieee32::new_from_bits(0x7ff00001)), - "qNaN:0x300001"); + "NaN:0x300001"); // Signaling NaNs. assert_eq!(format!("{}", Ieee32::new_from_bits(0x7f800001)), "sNaN:0x1"); assert_eq!(format!("{}", Ieee32::new_from_bits(0x7fa00001)), @@ -221,13 +221,13 @@ mod tests { "0x0.0000000000001p-1022"); assert_eq!(format!("{}", Ieee64::new(f64::INFINITY)), "Inf"); assert_eq!(format!("{}", Ieee64::new(f64::NEG_INFINITY)), "-Inf"); - assert_eq!(format!("{}", Ieee64::new(f64::NAN)), "qNaN"); - assert_eq!(format!("{}", Ieee64::new(-f64::NAN)), "-qNaN"); + assert_eq!(format!("{}", Ieee64::new(f64::NAN)), "NaN"); + assert_eq!(format!("{}", Ieee64::new(-f64::NAN)), "-NaN"); // Construct some qNaNs with payloads. assert_eq!(format!("{}", Ieee64::new_from_bits(0x7ff8000000000001)), - "qNaN:0x1"); + "NaN:0x1"); assert_eq!(format!("{}", Ieee64::new_from_bits(0x7ffc000000000001)), - "qNaN:0x4000000000001"); + "NaN:0x4000000000001"); // Signaling NaNs. assert_eq!(format!("{}", Ieee64::new_from_bits(0x7ff0000000000001)), "sNaN:0x1"); From f72f47aece6aab5cb6e40e17c19ecf7004395d93 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 1 Apr 2016 15:32:00 -0700 Subject: [PATCH 0033/3084] Replace bool with b1, b8, b16, ... The b1 type is an abstract boolean value. The others are concrete representations. --- cranelift/docs/cton_lexer.py | 2 +- cranelift/docs/langref.rst | 56 +++++++++++++++++++++++++++--------- meta/cretonne/__init__.py | 14 +++++++++ meta/cretonne/types.py | 9 ++++-- 4 files changed, 64 insertions(+), 17 deletions(-) diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py index c12db5aea7..0139de337c 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/cton_lexer.py @@ -30,7 +30,7 @@ class CretonneLexer(RegexLexer): # Known attributes. (keywords('align', 'aligntrap', 'uext', 'sext', 'inreg'), Name.Attribute), # Well known value types. - (r'\b(bool|i\d+|f32|f64)(x\d+)?\b', Keyword.Type), + (r'\b(b\d+|i\d+|f32|f64)(x\d+)?\b', Keyword.Type), # v = value # ss = stack slot (r'(v|ss)\d+', Name.Variable), diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 47f7fa8833..df0e19b20a 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -87,7 +87,36 @@ All SSA values have a type which determines the size and shape (for SIMD vectors) of the value. Many instructions are polymorphic -- they can operate on different types. -.. autoctontype:: bool +Boolean types +------------- + +Boolean values are either true or false. While this only requires a single bit +to represent, more bits are often used when holding a boolean value in a +register or in memory. The :type:`b1` type represents an abstract boolean +value. It can only exist as an SSA value, it can't be stored in memory or +converted to another type. The larger boolean types can be stored in memory. + +.. todo:: Clarify the representation of larger boolean types. + + The multi-bit boolean types can be interpreted in different ways. We could + declare that zero means false and non-zero means true. This may require + unwanted normalization code in some places. + + We could specify a fixed encoding like all ones for true. This would then + lead to undefined behavior if untrusted code uses the multibit booleans + incorrectly. + + Something like this: + + - External code is not allowed to load/store multi-bit booleans or + otherwise expose the representation. + - Each target specifies the exact representation of a multi-bit boolean. + +.. autoctontype:: b1 +.. autoctontype:: b8 +.. autoctontype:: b16 +.. autoctontype:: b32 +.. autoctontype:: b64 Integer types ------------- @@ -115,7 +144,7 @@ SIMD vector types ----------------- A SIMD vector type represents a vector of values from one of the scalar types -(:type:`bool`, integer, and floating point). Each scalar value in a SIMD type is +(boolean, integer, and floating point). Each scalar value in a SIMD type is called a *lane*. The number of lanes must be a power of two in the range 2-256. .. type:: i%Bx%N @@ -146,14 +175,14 @@ called a *lane*. The number of lanes must be a power of two in the range 2-256. The size of a :type:`f64` vector in memory is :math:`8N` bytes. -.. type:: boolx%N +.. type:: b1x%N A boolean SIMD vector. Boolean vectors are used when comparing SIMD vectors. For example, - comparing two :type:`i32x4` values would produce a :type:`boolx4` result. + comparing two :type:`i32x4` values would produce a :type:`b1x4` result. - Like the :type:`bool` type, a boolean vector cannot be stored in memory. + Like the :type:`b1` type, a boolean vector cannot be stored in memory. Pseudo-types and type classes ----------------------------- @@ -194,11 +223,11 @@ in this reference. .. type:: Logic - Either :type:`bool` or :type:`boolxN`. + Either :type:`b1` or :type:`b1xN`. .. type:: Testable - Either :type:`bool` or :type:`iN`. + Either :type:`b1` or :type:`iN`. Immediate operand types ----------------------- @@ -291,7 +320,7 @@ instruction in the EBB. Branch when zero. - If ``x`` is a :type:`bool` value, take the branch when ``x`` is false. If + If ``x`` is a :type:`b1` value, take the branch when ``x`` is false. If ``x`` is an integer value, take the branch when ``x = 0``. :arg Testable x: Value to test. @@ -303,7 +332,7 @@ instruction in the EBB. Branch when non-zero. - If ``x`` is a :type:`bool` value, take the branch when ``x`` is true. If + If ``x`` is a :type:`b1` value, take the branch when ``x`` is true. If ``x`` is an integer value, take the branch when ``x != 0``. :arg Testable x: Value to test. @@ -479,8 +508,7 @@ Cretonne also provides more restricted memory operations that are always safe. Load from memory at ``p + Offset``. This is a polymorphic instruction that can load any value type which has a - memory representation (i.e., everything except :type:`bool` and boolean - vectors). + memory representation. :arg iPtr p: Base address. :arg Offset: Immediate signed offset. @@ -692,7 +720,7 @@ load a constant into an SSA value. Conditional select. - :arg bool c: Controlling flag. + :arg b1 c: Controlling flag. :arg T x: Value to return when ``c`` is true. :arg T y: Value to return when ``c`` is false. Must be same type as ``x``. :result T a: Same type as ``x`` and ``y``. @@ -710,7 +738,7 @@ Vector operations Select lanes from ``x`` or ``y`` controlled by the lanes of the boolean vector ``c``. - :arg boolxN c: Controlling flag vector. + :arg b1xN c: Controlling flag vector. :arg TxN x: Vector with lanes selected by the true lanes of ``c``. Must be a vector type with the same number of lanes as ``c``. :arg TxN y: Vector with lanes selected by the false lanes of ``c``. @@ -872,7 +900,7 @@ These operations generally follow IEEE 754-2008 semantics. :arg Cond: Condition code determining how ``x`` and ``y`` are compared. :arg x,y: Floating point scalar or vector values of the same type. - :rtype: :type:`bool` or :type:`boolxN` with the same number of lanes as + :rtype: :type:`b1` or :type:`b1xN` with the same number of lanes as ``x`` and ``y``. An 'ordered' condition code yields ``false`` if either operand is Nan. diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index fd0c4664dd..bbf0e74063 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -95,6 +95,20 @@ class FloatType(ScalarType): def __repr__(self): return 'FloatType(bits={})'.format(self.bits) +class BoolType(ScalarType): + """A concrete scalar boolean type.""" + + def __init__(self, bits): + assert bits > 0, 'BoolType must have positive number of bits' + super(BoolType, self).__init__( + name='b{:d}'.format(bits), + membytes=bits/8, + doc="A boolean type with {} bits.".format(bits)) + self.bits = bits + + def __repr__(self): + return 'BoolType(bits={})'.format(self.bits) + # # Parametric polymorphism. # diff --git a/meta/cretonne/types.py b/meta/cretonne/types.py index 3d369a60e4..c639d89050 100644 --- a/meta/cretonne/types.py +++ b/meta/cretonne/types.py @@ -2,14 +2,19 @@ The cretonne.types module predefines all the Cretonne scalar types. """ -from . import ScalarType, IntType, FloatType +from . import ScalarType, IntType, FloatType, BoolType #: Boolean. -bool = ScalarType('bool', 0, +b1 = ScalarType('b1', 0, """ A boolean value that is either true or false. """) +b8 = BoolType(8) #: 8-bit bool. +b16 = BoolType(16) #: 16-bit bool. +b32 = BoolType(32) #: 32-bit bool. +b64 = BoolType(64) #: 64-bit bool. + i8 = IntType(8) #: 8-bit int. i16 = IntType(16) #: 16-bit int. i32 = IntType(32) #: 32-bit int. From 807b718358f75ab0d5597cc29bc4ce8ef7c30c65 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Apr 2016 11:11:53 -0700 Subject: [PATCH 0034/3084] Add an autohtml target to docs/Makefile. This runs the convenient sphinx-autobuild web server for docs development. --- cranelift/docs/Makefile | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/cranelift/docs/Makefile b/cranelift/docs/Makefile index 619d2caf24..598a13b496 100644 --- a/cranelift/docs/Makefile +++ b/cranelift/docs/Makefile @@ -4,6 +4,7 @@ # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build +SPHINXABUILD = sphinx-autobuild PAPER = BUILDDIR = _build @@ -56,6 +57,9 @@ html: @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." +autohtml: html + $(SPHINXABUILD) -z ../meta -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html + dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo From 0b8db43bbef3477dd853a458510bf4effc163c36 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Apr 2016 13:21:46 -0700 Subject: [PATCH 0035/3084] Add scaffolding for a Python build script. Hook up a Cargo build script that runs a Python script in the meta directory. --- cranelift/src/libcretonne/Cargo.toml | 1 + cranelift/src/libcretonne/build.rs | 49 ++++++++++++++++++++++++++++ meta/build.py | 11 +++++++ 3 files changed, 61 insertions(+) create mode 100644 cranelift/src/libcretonne/build.rs create mode 100644 meta/build.py diff --git a/cranelift/src/libcretonne/Cargo.toml b/cranelift/src/libcretonne/Cargo.toml index 58a67fe569..38edc811d1 100644 --- a/cranelift/src/libcretonne/Cargo.toml +++ b/cranelift/src/libcretonne/Cargo.toml @@ -2,6 +2,7 @@ authors = ["The cwCretonneRust Project Developers"] name = "cretonne" version = "0.0.0" +build = "build.rs" [lib] name = "cretonne" diff --git a/cranelift/src/libcretonne/build.rs b/cranelift/src/libcretonne/build.rs new file mode 100644 index 0000000000..523ddb3548 --- /dev/null +++ b/cranelift/src/libcretonne/build.rs @@ -0,0 +1,49 @@ + +// Build script. +// +// This program is run by Cargo when building libcretonne. It is used to generate Rust code from +// the language definitions in the meta directory. +// +// Environment: +// +// OUT_DIR +// Directory where generated files should be placed. +// +// The build script expects to be run from the directory where this build.rs file lives. The +// current directory is used to find the sources. + + +use std::env; +use std::process; + +fn main() { + let out_dir = env::var("OUT_DIR").expect("The OUT_DIR environment variable must be set"); + + println!("Build script generating files in {}", out_dir); + + let mut cur_dir = env::current_dir().expect("Can't access current working directory"); + + // We're in src/libcretonne. Find the top-level directory. + assert!(cur_dir.pop(), "No parent 'src' directory"); + assert!(cur_dir.pop(), "No top-level directory"); + let top_dir = cur_dir.as_path(); + + // Scripts are in $top_dir/meta. + let meta_dir = top_dir.join("meta"); + let build_script = meta_dir.join("build.py"); + + // Let Cargo known that this script should be rerun if anything changes in the meta directory. + println!("cargo:rerun-if-changed={}", meta_dir.display()); + + // Launch build script with Python. We'll just find python in the path. + let status = process::Command::new("python") + .current_dir(top_dir) + .arg(build_script) + .arg("--out-dir") + .arg(out_dir) + .status() + .expect("Failed to launch second-level build script"); + if !status.success() { + process::exit(status.code().unwrap()); + } +} diff --git a/meta/build.py b/meta/build.py new file mode 100644 index 0000000000..2cdda53403 --- /dev/null +++ b/meta/build.py @@ -0,0 +1,11 @@ +# Second-level build script. +# +# This script is run from src/libcretonne/build.rs to generate Rust files. + +import argparse + +parser = argparse.ArgumentParser(description='Generate sources for Cretonne.') +parser.add_argument('--out-dir', help='set output directory') + +args = parser.parse_args() +out_dir = args.out_dir From 99b1251b35ccda52bca73443354267dbcb2796a8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Apr 2016 14:35:50 -0700 Subject: [PATCH 0036/3084] Collect all instructions into instruction groups. --- cranelift/docs/metaref.rst | 2 ++ meta/cretonne/__init__.py | 45 ++++++++++++++++++++++++++++++++++++++ meta/cretonne/base.py | 6 ++++- 3 files changed, 52 insertions(+), 1 deletion(-) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 8d58757b81..e9b7c9a6ec 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -72,3 +72,5 @@ class. .. autoclass:: Operand .. autoclass:: Instruction +.. autoclass:: InstructionGroup + :members: diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index bbf0e74063..a5ebb50f04 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -150,6 +150,50 @@ class ImmediateType(object): # Defining instructions. # +class InstructionGroup(object): + """ + An instruction group. + + Every instruction must belong to exactly one instruction group. A given + target architecture can support instructions from multiple groups, and it + does not necessarily support all instructions in a group. + + New instructions are automatically added to the currently open instruction + group. + """ + + # The currently open instruction group. + _current = None + + def open(self): + """ + Open this instruction group such that future new instructions are + added to this group. + """ + assert InstructionGroup._current is None, ( + "Can't open {} since {} is already open".format(self, _current)) + InstructionGroup._current = self + + def close(self): + """ + Close this instruction group. This function should be called before + opening another instruction group. + """ + assert InstructionGroup._current is self, ( + "Can't close {}, the open instuction group is {}".format(self, _current)) + InstructionGroup._current = None + + def __init__(self, name, doc): + self.name = name + self.__doc__ = doc + self.instructions = [] + self.open() + + @staticmethod + def append(inst): + assert InstructionGroup._current, "Open an instruction group before defining instructions." + InstructionGroup._current.instructions.append(inst) + class Operand(object): """ An instruction operand. @@ -202,6 +246,7 @@ class Instruction(object): self.__doc__ = doc self.ins = self._to_operand_tuple(ins) self.outs = self._to_operand_tuple(outs) + InstructionGroup.append(self) @staticmethod def _to_operand_tuple(x): diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index a926829d7c..604687829e 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -3,10 +3,12 @@ Cretonne base instruction set. This module defines the basic Cretonne instruction set that all targets support. """ -from . import TypeVar, Operand, Instruction +from . import TypeVar, Operand, Instruction, InstructionGroup from types import i8, f32, f64 from immediates import imm64, ieee32, ieee64, immvector +instructions = InstructionGroup("base", "Shared base instruction set") + Int = TypeVar('Int', 'A scalar or vector integer type') iB = TypeVar('iB', 'A scalar integer type') TxN = TypeVar('%Tx%N', 'A SIMD vector type') @@ -321,3 +323,5 @@ popcnt = Instruction('popcnt', r""" Count the number of one bits in ``x``. """, ins=x, outs=a) + +instructions.close() From 5388f68437e0bb9f41cb62347a71de25b6b34f4d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Apr 2016 15:28:08 -0700 Subject: [PATCH 0037/3084] Give instructions a CamelCase name. This will be used as the instruction name in Rust code. By making this a property of the instruction, it can be changed by the user if desired. --- meta/cretonne/__init__.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index a5ebb50f04..ff03ed198c 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -5,6 +5,13 @@ This module provides classes and functions used to describe Cretonne instructions. """ +import re + +camel_re = re.compile('(^|_)([a-z])') +def camel_case(s): + """Convert the string s to CamelCase""" + return camel_re.sub(lambda m: m.group(2).upper(), s) + # Concrete types. # # Instances (i8, i32, ...) are provided in the cretonne.types module. @@ -243,6 +250,7 @@ class Instruction(object): def __init__(self, name, doc, ins=(), outs=(), **kwargs): self.name = name + self.camel_name = camel_case(name) self.__doc__ = doc self.ins = self._to_operand_tuple(ins) self.outs = self._to_operand_tuple(outs) From 6e2e7bfb736f6ba144519a2c75608be7253ce580 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 6 Apr 2016 11:32:43 -0700 Subject: [PATCH 0038/3084] Add a RISC-V target. Flesh out the directory structure for defining target instruction set architectures. Use RISC-V as a startgin point because it is so simple. --- cranelift/.gitignore | 2 -- cranelift/docs/metaref.rst | 18 +++++++++++++++++- cranelift/src/.gitignore | 2 ++ meta/cretonne/__init__.py | 18 ++++++++++++++++++ meta/target/__init__.py | 16 ++++++++++++++++ meta/target/riscv/__init__.py | 30 ++++++++++++++++++++++++++++++ 6 files changed, 83 insertions(+), 3 deletions(-) create mode 100644 cranelift/src/.gitignore create mode 100644 meta/target/__init__.py create mode 100644 meta/target/riscv/__init__.py diff --git a/cranelift/.gitignore b/cranelift/.gitignore index 25eadf4e60..0d20b6487c 100644 --- a/cranelift/.gitignore +++ b/cranelift/.gitignore @@ -1,3 +1 @@ *.pyc -Cargo.lock -target diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index e9b7c9a6ec..c8f8b0e265 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -11,7 +11,7 @@ domain specific language embedded in Python. An instruction set is described by a Python module under the :file:`meta` directory that has a global variable called ``instructions``. The basic Cretonne instruction set described in :doc:`langref` is defined by the Python -module :mod:`cretonne.instrs`. +module :mod:`cretonne.base`. .. module:: cretonne @@ -74,3 +74,19 @@ class. .. autoclass:: Instruction .. autoclass:: InstructionGroup :members: + +Targets +======= + +Cretonne can be compiled with support for multiple target instruction set +architectures. Each ISA is represented by a :py:class`cretonne.Target` instance. + +.. autoclass:: Target + +The definitions for each supported target live in a package under +:file:`meta/target`. + +.. automodule:: target + :members: + +.. automodule:: target.riscv diff --git a/cranelift/src/.gitignore b/cranelift/src/.gitignore new file mode 100644 index 0000000000..a9d37c560c --- /dev/null +++ b/cranelift/src/.gitignore @@ -0,0 +1,2 @@ +target +Cargo.lock diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index ff03ed198c..d69d5c35c4 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -267,3 +267,21 @@ class Instruction(object): for op in x: assert isinstance(op, Operand) return x + +# +# Defining targets +# +class Target(object): + """ + A target instruction set architecture. + + The `Target` class collects everything known about a target ISA. + + :param name: Short mnemonic name for the ISA. + :param instruction_groups: List of `InstructionGroup` instances that are + relevant for this ISA. + """ + + def __init__(self, name, instrution_groups): + self.name = name + self.instruction_groups = instrution_groups diff --git a/meta/target/__init__.py b/meta/target/__init__.py new file mode 100644 index 0000000000..bed730b207 --- /dev/null +++ b/meta/target/__init__.py @@ -0,0 +1,16 @@ +""" +Cretonne target definitions +--------------------------- + +The :py:mod:`target` package contains sub-packages for each target instruction +set architecture supported by Cretonne. +""" + +from . import riscv + +def all_targets(): + """ + Get a list of all the supported targets. Each target is represented as a + :py:class:`cretonne.Target` instance. + """ + return [riscv.target] diff --git a/meta/target/riscv/__init__.py b/meta/target/riscv/__init__.py new file mode 100644 index 0000000000..e9b9926e31 --- /dev/null +++ b/meta/target/riscv/__init__.py @@ -0,0 +1,30 @@ +""" +RISC-V Target +------------- + +`RISC-V `_ is an open instruction set architecture originally +developed at UC Berkeley. It is a RISC-style ISA with either a 32-bit (RV32I) or +64-bit (RV32I) base instruction set and a number of optional extensions: + +RV32M / RV64M + Integer multiplication and division. + +RV32A / RV64A + Atomics. + +RV32F / RV64F + Single-precision IEEE floating point. + +RV32D / RV64D + Double-precision IEEE floating point. + +RV32G / RV64G + General purpose instruction sets. This represents the union of the I, M, A, + F, and D instruction sets listed above. + +""" + +from cretonne import Target +import cretonne.base + +target = Target('riscv', [cretonne.base.instructions]) From eba396546a76ee720d948ac74172798308da363b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 6 Apr 2016 10:45:06 -0700 Subject: [PATCH 0039/3084] Begin source generation. Start out easy by emiting an opcodes.rs file containing an opcode enumeration. --- meta/build.py | 6 ++++ meta/gen_instr.py | 31 ++++++++++++++++++++ meta/srcgen.py | 73 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 110 insertions(+) create mode 100644 meta/gen_instr.py create mode 100644 meta/srcgen.py diff --git a/meta/build.py b/meta/build.py index 2cdda53403..76673fa62b 100644 --- a/meta/build.py +++ b/meta/build.py @@ -3,9 +3,15 @@ # This script is run from src/libcretonne/build.rs to generate Rust files. import argparse +import target +import gen_instr parser = argparse.ArgumentParser(description='Generate sources for Cretonne.') parser.add_argument('--out-dir', help='set output directory') args = parser.parse_args() out_dir = args.out_dir + +targets = target.all_targets() + +gen_instr.generate(targets, out_dir) diff --git a/meta/gen_instr.py b/meta/gen_instr.py new file mode 100644 index 0000000000..a92609c861 --- /dev/null +++ b/meta/gen_instr.py @@ -0,0 +1,31 @@ +""" +Generate sources with instruction info. +""" + +import srcgen + +def collect_instr_groups(targets): + seen = set() + groups = [] + for t in targets: + for g in t.instruction_groups: + if g not in seen: + groups.append(g) + seen.add(g) + return groups + +def gen_opcodes(groups, out_dir): + """Generate opcode enumerations.""" + fmt = srcgen.Formatter() + fmt.line('enum Opcode {') + fmt.indent_push() + for g in groups: + for i in g.instructions: + fmt.line(i.camel_name + ',') + fmt.indent_pop() + fmt.line('}') + fmt.update_file('opcodes.rs', out_dir) + +def generate(targets, out_dir): + groups = collect_instr_groups(targets) + gen_opcodes(groups, out_dir) diff --git a/meta/srcgen.py b/meta/srcgen.py new file mode 100644 index 0000000000..9778e2c518 --- /dev/null +++ b/meta/srcgen.py @@ -0,0 +1,73 @@ +""" +Source code generator. + +The `srcgen` module contains generic helper routines and classes for generating +source code. + +""" + +import sys +import os + +class Formatter(object): + """ + Source code formatter class. + + - Collect source code to be written to a file. + - Keep track of indentation. + + Indentation example: + + >>> f = Formatter() + >>> f.line('Hello line 1') + >>> f.writelines() + Hello line 1 + >>> f.indent_push() + >>> f.comment('Nested comment') + >>> f.indent_pop() + >>> f.line('Back again') + >>> f.writelines() + Hello line 1 + // Nested comment + Back again + + """ + + shiftwidth = 2 + + def __init__(self): + self.indent = '' + self.lines = [] + + def indent_push(self): + """Increase current indentation level by one.""" + self.indent += ' ' * self.shiftwidth + + def indent_pop(self): + """Decrease indentation by one level.""" + assert self.indent != '', 'Already at top level indentation' + self.indent = self.indent[0:-self.shiftwidth] + + def line(self, s): + """And an indented line.""" + self.lines.append('{}{}\n'.format(self.indent, s)) + + def writelines(self, f=None): + """Write all lines to `f`.""" + if not f: + f = sys.stdout + f.writelines(self.lines) + + def update_file(self, filename, directory): + if directory is not None: + filename = os.path.join(directory, filename) + with open(filename, 'w') as f: + self.writelines(f) + + def comment(self, s): + """Add a comment line.""" + self.line('// ' + s) + +if __name__ == "__main__": + import doctest + doctest.testmod() From ee09a39aefd2bfeb1ea660f07069c95322e2fe0d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 6 Apr 2016 14:55:21 -0700 Subject: [PATCH 0040/3084] Include generated Opcode enum in the immediates module. Generate nice doc comments for the Opcode enum variants that 'cargo doc' will pick up. Include a `Display` trait implementation that prints the lower snake-case version of the opcode name. --- cranelift/src/libcretonne/immediates.rs | 18 ++++++++++ meta/gen_instr.py | 32 +++++++++++++---- meta/srcgen.py | 47 ++++++++++++++++++++++--- 3 files changed, 86 insertions(+), 11 deletions(-) diff --git a/cranelift/src/libcretonne/immediates.rs b/cranelift/src/libcretonne/immediates.rs index efd84cd606..0034139f50 100644 --- a/cranelift/src/libcretonne/immediates.rs +++ b/cranelift/src/libcretonne/immediates.rs @@ -8,6 +8,9 @@ use std::fmt::{self, Display, Formatter}; use std::mem; +// The `Opcode` enum is generated from the meta instruction descriptions. +include!(concat!(env!("OUT_DIR"), "/opcodes.rs")); + /// 64-bit immediate integer operand. /// #[derive(Copy, Clone, PartialEq, Eq, Debug)] @@ -41,11 +44,13 @@ impl Display for Imm64 { /// An IEEE binary32 immediate floating point value. /// /// All bit patterns are allowed. +#[derive(Copy, Clone, Debug)] pub struct Ieee32(f32); /// An IEEE binary64 immediate floating point value. /// /// All bit patterns are allowed. +#[derive(Copy, Clone, Debug)] pub struct Ieee64(f64); // Format a floating point number in a way that is reasonably human-readable, and that can be @@ -155,6 +160,19 @@ mod tests { use super::*; use std::{f32, f64}; + #[test] + fn opcodes() { + let x = Opcode::Iadd; + let mut y = Opcode::Isub; + + assert!(x != y); + y = Opcode::Iadd; + assert_eq!(x, y); + + assert_eq!(format!("{:?}", Opcode::IaddImm), "IaddImm"); + assert_eq!(format!("{}", Opcode::IaddImm), "iadd_imm"); + } + #[test] fn format_imm64() { assert_eq!(format!("{}", Imm64(0)), "0"); diff --git a/meta/gen_instr.py b/meta/gen_instr.py index a92609c861..cd50b4ee8a 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -17,13 +17,31 @@ def collect_instr_groups(targets): def gen_opcodes(groups, out_dir): """Generate opcode enumerations.""" fmt = srcgen.Formatter() - fmt.line('enum Opcode {') - fmt.indent_push() - for g in groups: - for i in g.instructions: - fmt.line(i.camel_name + ',') - fmt.indent_pop() - fmt.line('}') + + fmt.doc_comment('An instruction opcode.') + fmt.doc_comment('') + fmt.doc_comment('All instructions from all supported targets are present.') + fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug)]') + instrs = [] + with fmt.indented('pub enum Opcode {', '}'): + for g in groups: + for i in g.instructions: + instrs.append(i) + # Build a doc comment. + prefix = ', '.join(o.name for o in i.outs) + if prefix: + prefix = prefix + ' = ' + suffix = ', '.join(o.name for o in i.ins) + fmt.doc_comment('`{}{} {}`.'.format(prefix, i.name, suffix)) + # Enum variant itself. + fmt.line(i.camel_name + ',') + + with fmt.indented('impl Display for Opcode {', '}'): + with fmt.indented('fn fmt(&self, f: &mut Formatter) -> fmt::Result {', '}'): + with fmt.indented('f.write_str(match *self {', '})'): + for i in instrs: + fmt.format('Opcode::{} => "{}",', i.camel_name, i.name) + fmt.update_file('opcodes.rs', out_dir) def generate(targets, out_dir): diff --git a/meta/srcgen.py b/meta/srcgen.py index 9778e2c518..125ae0a63a 100644 --- a/meta/srcgen.py +++ b/meta/srcgen.py @@ -25,15 +25,15 @@ class Formatter(object): >>> f.indent_push() >>> f.comment('Nested comment') >>> f.indent_pop() - >>> f.line('Back again') + >>> f.format('Back {} again', 'home') >>> f.writelines() Hello line 1 - // Nested comment - Back again + // Nested comment + Back home again """ - shiftwidth = 2 + shiftwidth = 4 def __init__(self): self.indent = '' @@ -64,10 +64,49 @@ class Formatter(object): with open(filename, 'w') as f: self.writelines(f) + class _IndentedScope(object): + def __init__(self, fmt, after): + self.fmt = fmt + self.after = after + + def __enter__(self): + self.fmt.indent_push(); + + def __exit__(self, t, v, tb): + self.fmt.indent_pop() + if self.after: + self.fmt.line(self.after) + + def indented(self, before=None, after=None): + """ + Return a scope object for use with a `with` statement: + + >>> f = Formatter() + >>> with f.indented('prefix {', '} suffix'): + ... f.line('hello') + >>> f.writelines() + prefix { + hello + } suffix + + The optional `before` and `after` parameters are surrounding lines + which are *not* indented. + """ + if before: + self.line(before) + return self._IndentedScope(self, after) + + def format(self, fmt, *args): + self.line(fmt.format(*args)) + def comment(self, s): """Add a comment line.""" self.line('// ' + s) + def doc_comment(self, s): + """Add a documentation comment line.""" + self.line('/// ' + s) + if __name__ == "__main__": import doctest doctest.testmod() From b0c0dd1b9f5f308c83c903b194adbb662d40011b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 7 Apr 2016 11:09:36 -0700 Subject: [PATCH 0041/3084] Add repr.rs module containing the representation of functions. A function owns instructions and extended basic blocks. References to these entities are implemented as opaque structs indexing into the functions internal tables. This avoids fighting Rust's ownership checking and it also makes references 4 bytes on all platforms. SSA values are identified similarly, but with an optimization for the first value produced by an instruction. Very few instructions will produce more than one value, and there is an extended value table for those. --- cranelift/src/libcretonne/lib.rs | 1 + cranelift/src/libcretonne/repr.rs | 341 ++++++++++++++++++++++++++++++ 2 files changed, 342 insertions(+) create mode 100644 cranelift/src/libcretonne/repr.rs diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 9c837eb0f1..b67ca5df10 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -7,3 +7,4 @@ pub mod types; pub mod immediates; +pub mod repr; diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs new file mode 100644 index 0000000000..1318a0936a --- /dev/null +++ b/cranelift/src/libcretonne/repr.rs @@ -0,0 +1,341 @@ + +//! Representation of Cretonne IL functions. + +use types::Type; +use immediates::*; +use std::fmt::{self, Display, Formatter, Write}; +use std::u32; + +// ====--------------------------------------------------------------------------------------====// +// +// Public data types. +// +// ====--------------------------------------------------------------------------------------====// + +/// An opaque reference to an extended basic block in a function. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct Ebb(u32); + +/// A guaranteed invalid EBB reference. +pub const NO_EBB: Ebb = Ebb(u32::MAX); + +/// An opaque reference to an instruction in a function. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct Inst(u32); + +/// A guaranteed invalid instruction reference. +pub const NO_INST: Inst = Inst(u32::MAX); + +/// An opaque reference to an SSA value. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct Value(u32); + +/// A guaranteed invalid value reference. +pub const NO_VALUE: Value = Value(u32::MAX); + +/// A function. +/// +/// The `Function` struct owns all of its instructions and extended basic blocks, and it works as a +/// container for those objects by implementing both `Index` and `Index`. +/// +pub struct Function { + /// Data about all of the instructions in the function. The instructions in this vector is not + /// necessarily in program order. The `Inst` reference indexes into this vector. + instructions: Vec, + + /// Extended basic blocks in the function, not necessarily in program order. The `Ebb` + /// reference indexes into this vector. + extended_basic_blocks: Vec, + + /// Extended value table. Most `Value` references refer directly to their defining instruction. + /// Others index into this table. + extended_values: Vec, + + /// Return type(s). A function may return zero or more values. + pub return_types: Vec, +} + +/// Contents of an extended basic block. +pub struct EbbData { + /// Arguments for this extended basic block. These values dominate everything in the EBB. + /// All branches to this EBB must provide matching arguments, and the arguments to the entry + /// EBB must match the function arguments. + pub arguments: Vec, +} + +/// Contents on an instruction. +/// +/// Every variant must contain `opcode` and `ty` fields. An instruction that doesn't produce a +/// value should have its `ty` field set to `VOID`. The size of `InstructionData` should be kept at +/// 16 bytes on 64-bit architectures. If more space is needed to represent an instruction, use a +/// `Box` to store the additional information out of line. +pub enum InstructionData { + Nullary { + opcode: Opcode, + ty: Type, + }, + Unary { + opcode: Opcode, + ty: Type, + arg: Value, + }, + UnaryImm { + opcode: Opcode, + ty: Type, + imm: Imm64, + }, + Binary { + opcode: Opcode, + ty: Type, + args: [Value; 2], + }, + BinaryImm { + opcode: Opcode, + ty: Type, + arg: Value, + imm: Imm64, + }, + Call { + opcode: Opcode, + ty: Type, + data: Box, + }, +} + +/// Payload of a call instruction. +pub struct CallData { + // Number of result values. + results: u8, + + // Dynamically sized array containing `results-1` result values (not including the first value) + // followed by the argument values. + values: Vec, +} + + +// ====--------------------------------------------------------------------------------------====// +// +// Extended basic block implementation. +// +// ====--------------------------------------------------------------------------------------====// + +impl Ebb { + fn new(index: usize) -> Ebb { + assert!(index < (u32::MAX as usize)); + Ebb(index as u32) + } + + pub fn index(&self) -> usize { + self.0 as usize + } +} + +/// Display an `Ebb` reference as "ebb12". +impl Display for Ebb { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write!(fmt, "ebb{}", self.0) + } +} + +impl EbbData { + fn new() -> EbbData { + EbbData { arguments: Vec::new() } + } +} + +// ====--------------------------------------------------------------------------------------====// +// +// Instruction implementation. +// +// ====--------------------------------------------------------------------------------------====// + +impl Inst { + fn new(index: usize) -> Inst { + assert!(index < (u32::MAX as usize)); + Inst(index as u32) + } + + pub fn index(&self) -> usize { + self.0 as usize + } +} + +/// Display an `Inst` reference as "inst7". +impl Display for Inst { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write!(fmt, "inst{}", self.0) + } +} + +// ====--------------------------------------------------------------------------------------====// +// +// Value implementation. +// +// ====--------------------------------------------------------------------------------------====// + +// Value references can either reference an instruction directly, or they can refer to the +// extended value table. +enum ExpandedValue { + // This is the first value produced by the referenced instruction. + Direct(Inst), + + // This value is described in the extended value table. + Table(usize), +} + +impl Value { + fn new_direct(i: Inst) -> Value { + let encoding = i.index() * 2; + assert!(encoding < u32::MAX as usize); + Value(encoding as u32) + } + + fn new_table(index: usize) -> Value { + let encoding = index * 2 + 1; + assert!(encoding < u32::MAX as usize); + Value(encoding as u32) + } + + // Expand the internal representation into something useful. + fn expand(&self) -> ExpandedValue { + use self::ExpandedValue::*; + let index = (self.0 / 2) as usize; + if self.0 % 2 == 0 { + Direct(Inst::new(index)) + } else { + Table(index) + } + } +} + +/// Display a `Value` reference as "v7" or "v2x". +impl Display for Value { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + use self::ExpandedValue::*; + match self.expand() { + Direct(i) => write!(fmt, "v{}", i.0), + Table(i) => write!(fmt, "v{}x", i), + } + } +} + +// Most values are simply the first value produced by an instruction. +// Other values have an entry in the value table. +enum ValueData { + // An unused entry in the value table. No instruction should be defining or using this value. + Unused, + + // Value is defined by an instruction, but it is not the first result. + Def { + ty: Type, + num: u8, + def: Inst, + }, + + // Value is an EBB argument. + Argument { + ty: Type, + num: u8, + ebb: Ebb, + }, +} + +impl InstructionData { + /// Get the opcode of this instruction. + pub fn opcode(&self) -> Opcode { + use self::InstructionData::*; + match *self { + Nullary { opcode, .. } => opcode, + Unary { opcode, .. } => opcode, + UnaryImm { opcode, .. } => opcode, + Binary { opcode, .. } => opcode, + BinaryImm { opcode, .. } => opcode, + Call { opcode, .. } => opcode, + } + } + + /// Type of the first result. + pub fn first_type(&self) -> Type { + use self::InstructionData::*; + match *self { + Nullary { ty, .. } => ty, + Unary { ty, .. } => ty, + UnaryImm { ty, .. } => ty, + Binary { ty, .. } => ty, + BinaryImm { ty, .. } => ty, + Call { ty, .. } => ty, + } + } +} + +impl Function { + /// Create a new empty function. + pub fn new() -> Function { + Function { + instructions: Vec::new(), + extended_basic_blocks: Vec::new(), + extended_values: Vec::new(), + return_types: Vec::new(), + } + } + + /// Resolve an instruction reference. + pub fn inst(&self, i: Inst) -> &InstructionData { + &self.instructions[i.0 as usize] + } + + /// Create a new instruction. + pub fn make_inst(&mut self, data: InstructionData) -> Inst { + let iref = Inst::new(self.instructions.len()); + self.instructions.push(data); + // FIXME: Allocate extended value table entries if needed. + iref + } + + /// Create a new basic block. + pub fn make_ebb(&mut self) -> Ebb { + let ebb = Ebb::new(self.extended_basic_blocks.len()); + self.extended_basic_blocks.push(EbbData::new()); + ebb + } + + /// Get the type of a value. + pub fn value_type(&self, v: Value) -> Type { + use self::ExpandedValue::*; + use self::ValueData::*; + match v.expand() { + Direct(i) => self.inst(i).first_type(), + Table(i) => { + match self.extended_values[i] { + Unused => panic!("Can't get type of Unused value {}", v), + Def { ty, .. } => ty, + Argument { ty, .. } => ty, + } + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use types; + use immediates::*; + + #[test] + fn make_inst() { + let mut func = Function::new(); + + let idata = InstructionData::Nullary { + opcode: Opcode::Iconst, + ty: types::I32, + }; + let inst = func.make_inst(idata); + assert_eq!(format!("{}", inst), "inst0"); + + // Immutable reference resolution. + let ins = func.inst(inst); + assert_eq!(ins.opcode(), Opcode::Iconst); + assert_eq!(ins.first_type(), types::I32); + } +} From 85248a4b18cf29ea5dfb500e3e04b837ef1ee33a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 7 Apr 2016 13:49:35 -0700 Subject: [PATCH 0042/3084] Generate an opcode_name() function. This function returning a &'static str is more primitive that the Display implementation. It allows the opcode strings to be reused by the parser. --- cranelift/src/libcretonne/immediates.rs | 9 ++++++++- meta/gen_instr.py | 10 +++++----- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/cranelift/src/libcretonne/immediates.rs b/cranelift/src/libcretonne/immediates.rs index 0034139f50..4b9adcc43a 100644 --- a/cranelift/src/libcretonne/immediates.rs +++ b/cranelift/src/libcretonne/immediates.rs @@ -8,9 +8,16 @@ use std::fmt::{self, Display, Formatter}; use std::mem; -// The `Opcode` enum is generated from the meta instruction descriptions. +// The `Opcode` enum and the `opcode_name` function are generated from the meta instruction +// descriptions. include!(concat!(env!("OUT_DIR"), "/opcodes.rs")); +impl Display for Opcode { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + write!(f, "{}", opcode_name(*self)) + } +} + /// 64-bit immediate integer operand. /// #[derive(Copy, Clone, PartialEq, Eq, Debug)] diff --git a/meta/gen_instr.py b/meta/gen_instr.py index cd50b4ee8a..b0fe4686f4 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -36,11 +36,11 @@ def gen_opcodes(groups, out_dir): # Enum variant itself. fmt.line(i.camel_name + ',') - with fmt.indented('impl Display for Opcode {', '}'): - with fmt.indented('fn fmt(&self, f: &mut Formatter) -> fmt::Result {', '}'): - with fmt.indented('f.write_str(match *self {', '})'): - for i in instrs: - fmt.format('Opcode::{} => "{}",', i.camel_name, i.name) + # Generate a private opcode_name function. + with fmt.indented('fn opcode_name(opc: Opcode) -> &\'static str {', '}'): + with fmt.indented('match opc {', '}'): + for i in instrs: + fmt.format('Opcode::{} => "{}",', i.camel_name, i.name) fmt.update_file('opcodes.rs', out_dir) From 1b7d5d849f4a11c024db18905a677c86e08dc167 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 7 Apr 2016 20:24:21 -0700 Subject: [PATCH 0043/3084] Generate a constant hash table for recognizing opcodes. Use a simple quadratically probed, open addressed hash table. We could use a parfect hash function, but it would take longer to compute in Python, and this is not in the critical path performancewise. --- cranelift/src/libcretonne/immediates.rs | 55 +++++++++++++++++- meta/constant_hash.py | 76 +++++++++++++++++++++++++ meta/gen_instr.py | 13 +++++ 3 files changed, 142 insertions(+), 2 deletions(-) create mode 100644 meta/constant_hash.py diff --git a/cranelift/src/libcretonne/immediates.rs b/cranelift/src/libcretonne/immediates.rs index 4b9adcc43a..911197c349 100644 --- a/cranelift/src/libcretonne/immediates.rs +++ b/cranelift/src/libcretonne/immediates.rs @@ -8,8 +8,12 @@ use std::fmt::{self, Display, Formatter}; use std::mem; -// The `Opcode` enum and the `opcode_name` function are generated from the meta instruction -// descriptions. +// Include code generated by `meta/gen_instr.py`. This file contains: +// +// - The `pub enum Opcode` definition with all known opcodes, +// - The private `fn opcode_name(Opcode) -> &'static str` function, and +// - The hash table `const OPCODE_HASH_TABLE: [Opcode; N]`. +// include!(concat!(env!("OUT_DIR"), "/opcodes.rs")); impl Display for Opcode { @@ -18,6 +22,46 @@ impl Display for Opcode { } } +// A primitive hash function for matching opcodes. +// Must match `meta/constant_hash.py`. +fn simple_hash(s: &str) -> u32 { + let mut h: u32 = 5381; + for c in s.chars() { + h = (h ^ c as u32).wrapping_add(h.rotate_right(6)); + } + h +} + +impl Opcode { + /// Parse an Opcode name from a string. + pub fn from_str(s: &str) -> Option { + let tlen = OPCODE_HASH_TABLE.len(); + assert!(tlen.is_power_of_two()); + let mut idx = simple_hash(s) as usize; + let mut step: usize = 0; + loop { + idx = idx % tlen; + let entry = OPCODE_HASH_TABLE[idx]; + + if entry == Opcode::NotAnOpcode { + return None; + } + + if *opcode_name(entry) == *s { + return Some(entry); + } + + // Quadratic probing. + step += 1; + // When `tlen` is a power of two, it can be proven that idx will visit all entries. + // This means that this loop will always terminate if the hash table has even one + // unused entry. + assert!(step < tlen); + idx += step; + } + } +} + /// 64-bit immediate integer operand. /// #[derive(Copy, Clone, PartialEq, Eq, Debug)] @@ -178,6 +222,13 @@ mod tests { assert_eq!(format!("{:?}", Opcode::IaddImm), "IaddImm"); assert_eq!(format!("{}", Opcode::IaddImm), "iadd_imm"); + + // Check the matcher. + assert_eq!(Opcode::from_str("iadd"), Some(Opcode::Iadd)); + assert_eq!(Opcode::from_str("iadd_imm"), Some(Opcode::IaddImm)); + assert_eq!(Opcode::from_str("iadd\0"), None); + assert_eq!(Opcode::from_str(""), None); + assert_eq!(Opcode::from_str("\0"), None); } #[test] diff --git a/meta/constant_hash.py b/meta/constant_hash.py new file mode 100644 index 0000000000..b1b6104231 --- /dev/null +++ b/meta/constant_hash.py @@ -0,0 +1,76 @@ +""" +Generate constant hash tables. + +The `constant_hash` module can generate constant pre-populated hash tables. We +don't attempt parfect hashing, but simply generate an open addressed +quadratically probed hash table. +""" + +def simple_hash(s): + """ + Compute a primitive hash of a string. + + Example: + >>> hex(simple_hash("Hello")) + '0x2fa70c01' + >>> hex(simple_hash("world")) + '0x5b0c31d5' + """ + h = 5381 + for c in s: + h = ((h ^ ord(c)) + ((h >> 6) + (h << 26))) & 0xffffffff + return h + +def next_power_of_two(x): + """ + Compute the next power of two that is greater than `x`: + >>> next_power_of_two(0) + 1 + >>> next_power_of_two(1) + 2 + >>> next_power_of_two(2) + 4 + >>> next_power_of_two(3) + 4 + >>> next_power_of_two(4) + 8 + """ + s = 1 + while x & (x + 1) != 0: + x |= x >> s + s *= 2 + return x + 1 + +def compute_quadratic(items, hash_function): + """ + Compute an open addressed, quadratically probed hash table containing + `items`. The returned table is a list containing the elements of the + iterable `items` and `None` in unused slots. + + :param items: Iterable set of items to place in hash table. + :param hash_function: Hash function which takes an item and returns a + number. + + Simple example (see hash values above, they collide on slot 1): + >>> compute_quadratic(['Hello', 'world'], simple_hash) + [None, 'Hello', 'world', None] + """ + + items = list(items) + # Table size must be a power of two. Aim for >20% unused slots. + size = next_power_of_two(int(1.20*len(items))) + table = [None] * size + + for i in items: + h = hash_function(i) % size + s = 0 + while table[h] is not None: + s += 1 + h = (h + s) % size + table[h] = i + + return table + +if __name__ == "__main__": + import doctest + doctest.testmod() diff --git a/meta/gen_instr.py b/meta/gen_instr.py index b0fe4686f4..eb1b0a361f 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -3,6 +3,7 @@ Generate sources with instruction info. """ import srcgen +import constant_hash def collect_instr_groups(targets): seen = set() @@ -24,6 +25,7 @@ def gen_opcodes(groups, out_dir): fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug)]') instrs = [] with fmt.indented('pub enum Opcode {', '}'): + fmt.line('NotAnOpcode,') for g in groups: for i in g.instructions: instrs.append(i) @@ -39,9 +41,20 @@ def gen_opcodes(groups, out_dir): # Generate a private opcode_name function. with fmt.indented('fn opcode_name(opc: Opcode) -> &\'static str {', '}'): with fmt.indented('match opc {', '}'): + fmt.line('Opcode::NotAnOpcode => "",') for i in instrs: fmt.format('Opcode::{} => "{}",', i.camel_name, i.name) + # Generate an opcode hash table for looking up opcodes by name. + hash_table = constant_hash.compute_quadratic(instrs, + lambda i: constant_hash.simple_hash(i.name)) + with fmt.indented('const OPCODE_HASH_TABLE: [Opcode; {}] = ['.format(len(hash_table)), '];'): + for i in hash_table: + if i is None: + fmt.line('Opcode::NotAnOpcode,') + else: + fmt.format('Opcode::{},', i.camel_name) + fmt.update_file('opcodes.rs', out_dir) def generate(targets, out_dir): From 53c878f4e1cfcd8bbb821f8ed181fa8d8849a420 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Apr 2016 10:31:14 -0700 Subject: [PATCH 0044/3084] Typo. --- cranelift/src/libcretonne/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/src/libcretonne/Cargo.toml b/cranelift/src/libcretonne/Cargo.toml index 38edc811d1..eccd757c7c 100644 --- a/cranelift/src/libcretonne/Cargo.toml +++ b/cranelift/src/libcretonne/Cargo.toml @@ -1,5 +1,5 @@ [package] -authors = ["The cwCretonneRust Project Developers"] +authors = ["The Cretonne Project Developers"] name = "cretonne" version = "0.0.0" build = "build.rs" From 661ac9e7ad9ae244700a8671b9371218aedf04cb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Apr 2016 11:06:33 -0700 Subject: [PATCH 0045/3084] Implement std::str::FromStr for matching opcodes. Replace the home-grown from_str function. --- cranelift/src/libcretonne/immediates.rs | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/cranelift/src/libcretonne/immediates.rs b/cranelift/src/libcretonne/immediates.rs index 911197c349..79326d9a0b 100644 --- a/cranelift/src/libcretonne/immediates.rs +++ b/cranelift/src/libcretonne/immediates.rs @@ -7,6 +7,7 @@ use std::fmt::{self, Display, Formatter}; use std::mem; +use std::str::FromStr; // Include code generated by `meta/gen_instr.py`. This file contains: // @@ -32,9 +33,11 @@ fn simple_hash(s: &str) -> u32 { h } -impl Opcode { +impl FromStr for Opcode { + type Err = &'static str; + /// Parse an Opcode name from a string. - pub fn from_str(s: &str) -> Option { + fn from_str(s: &str) -> Result { let tlen = OPCODE_HASH_TABLE.len(); assert!(tlen.is_power_of_two()); let mut idx = simple_hash(s) as usize; @@ -44,11 +47,11 @@ impl Opcode { let entry = OPCODE_HASH_TABLE[idx]; if entry == Opcode::NotAnOpcode { - return None; + return Err("Unknown opcode"); } if *opcode_name(entry) == *s { - return Some(entry); + return Ok(entry); } // Quadratic probing. @@ -224,11 +227,11 @@ mod tests { assert_eq!(format!("{}", Opcode::IaddImm), "iadd_imm"); // Check the matcher. - assert_eq!(Opcode::from_str("iadd"), Some(Opcode::Iadd)); - assert_eq!(Opcode::from_str("iadd_imm"), Some(Opcode::IaddImm)); - assert_eq!(Opcode::from_str("iadd\0"), None); - assert_eq!(Opcode::from_str(""), None); - assert_eq!(Opcode::from_str("\0"), None); + assert_eq!("iadd".parse::(), Ok(Opcode::Iadd)); + assert_eq!("iadd_imm".parse::(), Ok(Opcode::IaddImm)); + assert_eq!("iadd\0".parse::(), Err("Unknown opcode")); + assert_eq!("".parse::(), Err("Unknown opcode")); + assert_eq!("\0".parse::(), Err("Unknown opcode")); } #[test] From 9a6f79a9ba5450d3cc749816a5b18a32e61b41f8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Apr 2016 11:22:30 -0700 Subject: [PATCH 0046/3084] Add some scaffolding for building more crates. The src/tools directory contains the cretonne-tools crate which will build binaries for testing cretonne. The src/libctonfile directory contains the ctonfile library crate which provides reading and writing of .cton files. --- cranelift/src/.gitignore | 2 +- cranelift/src/libcretonne/Cargo.toml | 4 ++++ cranelift/src/libctonfile/Cargo.toml | 12 ++++++++++++ cranelift/src/libctonfile/lib.rs | 12 ++++++++++++ cranelift/src/libctonfile/parser.rs | 2 ++ cranelift/src/tools/Cargo.lock | 19 +++++++++++++++++++ cranelift/src/tools/Cargo.toml | 14 ++++++++++++++ cranelift/src/tools/main.rs | 5 +++++ 8 files changed, 69 insertions(+), 1 deletion(-) create mode 100644 cranelift/src/libctonfile/Cargo.toml create mode 100644 cranelift/src/libctonfile/lib.rs create mode 100644 cranelift/src/libctonfile/parser.rs create mode 100644 cranelift/src/tools/Cargo.lock create mode 100644 cranelift/src/tools/Cargo.toml create mode 100644 cranelift/src/tools/main.rs diff --git a/cranelift/src/.gitignore b/cranelift/src/.gitignore index a9d37c560c..d6168f7c42 100644 --- a/cranelift/src/.gitignore +++ b/cranelift/src/.gitignore @@ -1,2 +1,2 @@ target -Cargo.lock +lib*/Cargo.lock diff --git a/cranelift/src/libcretonne/Cargo.toml b/cranelift/src/libcretonne/Cargo.toml index eccd757c7c..ba7edec6d9 100644 --- a/cranelift/src/libcretonne/Cargo.toml +++ b/cranelift/src/libcretonne/Cargo.toml @@ -2,6 +2,10 @@ authors = ["The Cretonne Project Developers"] name = "cretonne" version = "0.0.0" +description = "Low-level code generator library" +documentation = "https://cretonne.readthedocs.org/" +repository = "https://github.com/stoklund/cretonne" +publish = false build = "build.rs" [lib] diff --git a/cranelift/src/libctonfile/Cargo.toml b/cranelift/src/libctonfile/Cargo.toml new file mode 100644 index 0000000000..7b2826b072 --- /dev/null +++ b/cranelift/src/libctonfile/Cargo.toml @@ -0,0 +1,12 @@ +[package] +authors = ["The Cretonne Project Developers"] +name = "ctonfile" +version = "0.0.0" +publish = false + +[lib] +name = "ctonfile" +path = "lib.rs" + +[dependencies] +cretonne = { path = "../libcretonne" } diff --git a/cranelift/src/libctonfile/lib.rs b/cranelift/src/libctonfile/lib.rs new file mode 100644 index 0000000000..3271ad035c --- /dev/null +++ b/cranelift/src/libctonfile/lib.rs @@ -0,0 +1,12 @@ + +// ====------------------------------------------------------------------------------------==== // +// +// Cretonne file read/write library +// +// ====------------------------------------------------------------------------------------==== // +// +// The libctonfile library supports reading and writing .cton files. This functionality is needed +// for testing Cretonne, but is not essential for a JIT compiler. + +extern crate cretonne; +pub mod parser; diff --git a/cranelift/src/libctonfile/parser.rs b/cranelift/src/libctonfile/parser.rs new file mode 100644 index 0000000000..44cf3a2500 --- /dev/null +++ b/cranelift/src/libctonfile/parser.rs @@ -0,0 +1,2 @@ + +use cretonne; diff --git a/cranelift/src/tools/Cargo.lock b/cranelift/src/tools/Cargo.lock new file mode 100644 index 0000000000..90722ef5e7 --- /dev/null +++ b/cranelift/src/tools/Cargo.lock @@ -0,0 +1,19 @@ +[root] +name = "cretonne-tools" +version = "0.0.0" +dependencies = [ + "cretonne 0.0.0", + "ctonfile 0.0.0", +] + +[[package]] +name = "cretonne" +version = "0.0.0" + +[[package]] +name = "ctonfile" +version = "0.0.0" +dependencies = [ + "cretonne 0.0.0", +] + diff --git a/cranelift/src/tools/Cargo.toml b/cranelift/src/tools/Cargo.toml new file mode 100644 index 0000000000..b4477bd1e8 --- /dev/null +++ b/cranelift/src/tools/Cargo.toml @@ -0,0 +1,14 @@ +[package] +authors = ["The Cretonne Project Developers"] +name = "cretonne-tools" +version = "0.0.0" +description = "Binaries for testing the Cretonne library" +publish = false + +[[bin]] +name = "cretonne" +path = "main.rs" + +[dependencies] +cretonne = { path = "../libcretonne" } +ctonfile = { path = "../libctonfile" } diff --git a/cranelift/src/tools/main.rs b/cranelift/src/tools/main.rs new file mode 100644 index 0000000000..33cc1afe81 --- /dev/null +++ b/cranelift/src/tools/main.rs @@ -0,0 +1,5 @@ + +extern crate cretonne; +extern crate ctonfile; + +fn main() {} From 71f8fe1cb1df9dfb45f2279128e26c4c2114b018 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Apr 2016 16:08:08 -0700 Subject: [PATCH 0047/3084] Migrate to readthedocs.io --- README.rst | 4 ++-- cranelift/src/libcretonne/Cargo.toml | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.rst b/README.rst index 8e8105f83b..4e87f615e8 100644 --- a/README.rst +++ b/README.rst @@ -5,8 +5,8 @@ Cretonne Code Generator Cretonne is a low-level retargetable code generator. It translates a target-independent intermediate language into executable machine code. -.. image:: https://readthedocs.org/projects/cretonne/badge/?version=latest - :target: http://cretonne.readthedocs.org/en/latest/?badge=latest +.. image:: https://readthedocs.io/projects/cretonne/badge/?version=latest + :target: http://cretonne.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status Cretonne is designed to be a code generator for WebAssembly with these design diff --git a/cranelift/src/libcretonne/Cargo.toml b/cranelift/src/libcretonne/Cargo.toml index ba7edec6d9..e6158af94f 100644 --- a/cranelift/src/libcretonne/Cargo.toml +++ b/cranelift/src/libcretonne/Cargo.toml @@ -3,7 +3,7 @@ authors = ["The Cretonne Project Developers"] name = "cretonne" version = "0.0.0" description = "Low-level code generator library" -documentation = "https://cretonne.readthedocs.org/" +documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/stoklund/cretonne" publish = false build = "build.rs" From 58f70ef12d77395d660e18f9d3b72a4a6ac5c918 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 12 Apr 2016 14:53:57 -0700 Subject: [PATCH 0048/3084] Implement FromStr for Imm64, Ieee32, Ieee64. These are bitwise exact conversions from string to immediates, implementing the inverse of the Display trait. Only accept hexadecimal floating point numbers to avoid issues with rounding when converting decimal numbers to binary. --- cranelift/src/libcretonne/immediates.rs | 493 +++++++++++++++++++++++- 1 file changed, 480 insertions(+), 13 deletions(-) diff --git a/cranelift/src/libcretonne/immediates.rs b/cranelift/src/libcretonne/immediates.rs index 79326d9a0b..4fefdbb9d0 100644 --- a/cranelift/src/libcretonne/immediates.rs +++ b/cranelift/src/libcretonne/immediates.rs @@ -70,6 +70,12 @@ impl FromStr for Opcode { #[derive(Copy, Clone, PartialEq, Eq, Debug)] pub struct Imm64(i64); +impl Imm64 { + pub fn from_bits(x: u64) -> Imm64 { + Imm64(x as i64) + } +} + impl Display for Imm64 { fn fmt(&self, f: &mut Formatter) -> fmt::Result { let x = self.0; @@ -94,6 +100,80 @@ impl Display for Imm64 { } } +impl FromStr for Imm64 { + type Err = &'static str; + + // Parse a decimal or hexadecimal Imm64, formatted as above. + fn from_str(s: &str) -> Result { + let mut value: u64 = 0; + let mut digits = 0; + let negative = s.starts_with('-'); + let s2 = if negative { + &s[1..] + } else { + s + }; + + if s2.starts_with("0x") { + // Hexadecimal. + for ch in s2[2..].chars() { + match ch.to_digit(16) { + Some(digit) => { + digits += 1; + if digits > 16 { + return Err("Too many hexadecimal digits in Imm64"); + } + // This can't overflow given the digit limit. + value = (value << 4) | digit as u64; + } + None => { + // Allow embedded underscores, but fail on anything else. + if ch != '_' { + return Err("Invalid character in hexadecimal Imm64"); + } + } + } + } + } else { + // Decimal number, possibly negative. + for ch in s2.chars() { + match ch.to_digit(16) { + Some(digit) => { + digits += 1; + match value.checked_mul(10) { + None => return Err("Too large decimal Imm64"), + Some(v) => value = v, + } + match value.checked_add(digit as u64) { + None => return Err("Too large decimal Imm64"), + Some(v) => value = v, + } + } + None => { + // Allow embedded underscores, but fail on anything else. + if ch != '_' { + return Err("Invalid character in decimal Imm64"); + } + } + } + } + } + + if digits == 0 { + return Err("No digits in Imm64"); + } + + // We support the range-and-a-half from -2^63 .. 2^64-1. + if negative { + value = value.wrapping_neg(); + // Don't allow large negative values to wrap around and become positive. + if value as i64 > 0 { + return Err("Negative number too small for Imm64"); + } + } + Ok(Imm64::from_bits(value)) + } +} /// An IEEE binary32 immediate floating point value. /// @@ -118,8 +198,9 @@ pub struct Ieee64(f64); // t - trailing significand field width in bits // fn format_float(bits: u64, w: u8, t: u8, f: &mut Formatter) -> fmt::Result { - assert!(w > 0 && w <= 16, "Invalid exponent range"); - assert!(1 + w + t <= 64, "Too large IEEE format for u64"); + debug_assert!(w > 0 && w <= 16, "Invalid exponent range"); + debug_assert!(1 + w + t <= 64, "Too large IEEE format for u64"); + debug_assert!((t + w + 1).is_power_of_two(), "Unexpected IEEE format size"); let max_e_bits = (1u64 << w) - 1; let t_bits = bits & ((1u64 << t) - 1); // Trailing significand. @@ -173,13 +254,173 @@ fn format_float(bits: u64, w: u8, t: u8, f: &mut Formatter) -> fmt::Result { } } +// Parse a float using the same format as `format_float` above. +// +// The encoding parameters are: +// +// w - exponent field width in bits +// t - trailing significand field width in bits +// +fn parse_float(s: &str, w: u8, t: u8) -> Result { + debug_assert!(w > 0 && w <= 16, "Invalid exponent range"); + debug_assert!(1 + w + t <= 64, "Too large IEEE format for u64"); + debug_assert!((t + w + 1).is_power_of_two(), "Unexpected IEEE format size"); + + let (sign_bit, s2) = if s.starts_with('-') { + (1u64 << t + w, &s[1..]) + } else { + (0, s) + }; + + if !s2.starts_with("0x") { + let max_e_bits = ((1u64 << w) - 1) << t; + let quiet_bit = 1u64 << (t - 1); + + // The only decimal encoding allowed is 0. + if s2 == "0.0" { + return Ok(sign_bit); + } + + if s2 == "Inf" { + // +/- infinity: e = max, t = 0. + return Ok(sign_bit | max_e_bits); + } + if s2 == "NaN" { + // Canonical quiet NaN: e = max, t = quiet. + return Ok(sign_bit | max_e_bits | quiet_bit); + } + if s2.starts_with("NaN:0x") { + // Quiet NaN with payload. + return match u64::from_str_radix(&s2[6..], 16) { + Ok(payload) if payload < quiet_bit => { + Ok(sign_bit | max_e_bits | quiet_bit | payload) + } + _ => Err("Invalid NaN payload"), + }; + } + if s2.starts_with("sNaN:0x") { + // Signaling NaN with payload. + return match u64::from_str_radix(&s2[7..], 16) { + Ok(payload) if 0 < payload && payload < quiet_bit => { + Ok(sign_bit | max_e_bits | payload) + } + _ => Err("Invalid sNaN payload"), + }; + } + + return Err("Float must be hexadecimal"); + } + let s3 = &s2[2..]; + + let mut digits = 0u8; + let mut digits_before_period: Option = None; + let mut significand = 0u64; + let mut exponent = 0i32; + + for (idx, ch) in s3.char_indices() { + match ch { + '.' => { + // This is the radix point. There can only be one. + if digits_before_period != None { + return Err("Multiple radix points"); + } else { + digits_before_period = Some(digits); + } + } + 'p' => { + // The following exponent is a decimal number. + let exp_str = &s3[1 + idx..]; + match exp_str.parse::() { + Ok(e) => { + exponent = e as i32; + break; + } + Err(_) => return Err("Bad exponent"), + } + } + _ => { + match ch.to_digit(16) { + Some(digit) => { + digits += 1; + if digits > 16 { + return Err("Too many digits"); + } + significand = (significand << 4) | digit as u64; + } + None => return Err("Invalid character"), + } + } + + } + } + + if digits == 0 { + return Err("No digits"); + } + + if significand == 0 { + // This is +/- 0.0. + return Ok(sign_bit); + } + + // Number of bits appearing after the radix point. + match digits_before_period { + None => {} // No radix point present. + Some(d) => exponent -= 4 * (digits - d) as i32, + }; + + // Normalize the significand and exponent. + let significant_bits = (64 - significand.leading_zeros()) as u8; + if significant_bits > t + 1 { + let adjust = significant_bits - (t + 1); + if significand & ((1u64 << adjust) - 1) != 0 { + return Err("Too many significant bits"); + } + // Adjust significand down. + significand >>= adjust; + exponent += adjust as i32; + } else { + let adjust = t + 1 - significant_bits; + significand <<= adjust; + exponent -= adjust as i32; + } + assert_eq!(significand >> t, 1); + + // Trailing significand excludes the high bit. + let t_bits = significand & ((1 << t) - 1); + + let max_exp = (1i32 << w) - 2; + let bias: i32 = (1 << (w - 1)) - 1; + exponent += bias + t as i32; + + if exponent > max_exp { + Err("Magnitude too large") + } else if exponent > 0 { + // This is a normal number. + let e_bits = (exponent as u64) << t; + Ok(sign_bit | e_bits | t_bits) + } else if 1 - exponent <= t as i32 { + // This is a subnormal number: e = 0, t = significand bits. + // Renormalize significand for exponent = 1. + let adjust = 1 - exponent; + if significand & ((1u64 << adjust) - 1) != 0 { + Err("Subnormal underflow") + } else { + significand >>= adjust; + Ok(sign_bit | significand) + } + } else { + Err("Magnitude too small") + } +} + impl Ieee32 { pub fn new(x: f32) -> Ieee32 { Ieee32(x) } /// Construct Ieee32 immediate from raw bits. - pub fn new_from_bits(x: u32) -> Ieee32 { + pub fn from_bits(x: u32) -> Ieee32 { Ieee32(unsafe { mem::transmute(x) }) } } @@ -191,13 +432,24 @@ impl Display for Ieee32 { } } +impl FromStr for Ieee32 { + type Err = &'static str; + + fn from_str(s: &str) -> Result { + match parse_float(s, 8, 23) { + Ok(b) => Ok(Ieee32::from_bits(b as u32)), + Err(s) => Err(s), + } + } +} + impl Ieee64 { pub fn new(x: f64) -> Ieee64 { Ieee64(x) } /// Construct Ieee64 immediate from raw bits. - pub fn new_from_bits(x: u64) -> Ieee64 { + pub fn from_bits(x: u64) -> Ieee64 { Ieee64(unsafe { mem::transmute(x) }) } } @@ -209,10 +461,23 @@ impl Display for Ieee64 { } } +impl FromStr for Ieee64 { + type Err = &'static str; + + fn from_str(s: &str) -> Result { + match parse_float(s, 11, 52) { + Ok(b) => Ok(Ieee64::from_bits(b)), + Err(s) => Err(s), + } + } +} + #[cfg(test)] mod tests { use super::*; use std::{f32, f64}; + use std::str::FromStr; + use std::fmt::Display; #[test] fn opcodes() { @@ -245,6 +510,74 @@ mod tests { assert_eq!(format!("{}", Imm64(0x10000)), "0x0001_0000"); } + // Verify that `text` can be parsed as a `T` into a value that displays as `want`. + fn parse_ok(text: &str, want: &str) + where ::Err: Display + { + match text.parse::() { + Err(s) => panic!("\"{}\".parse() error: {}", text, s), + Ok(x) => assert_eq!(format!("{}", x), want), + } + } + + // Verify that `text` fails to parse as `T` with the error `msg`. + fn parse_err(text: &str, msg: &str) + where ::Err: Display + { + match text.parse::() { + Err(s) => assert_eq!(format!("{}", s), msg), + Ok(x) => panic!("Wanted Err({}), but got {}", msg, x), + } + } + + #[test] + fn parse_imm64() { + parse_ok::("0", "0"); + parse_ok::("1", "1"); + parse_ok::("-0", "0"); + parse_ok::("-1", "-1"); + parse_ok::("0x0", "0"); + parse_ok::("0xf", "15"); + parse_ok::("-0x9", "-9"); + + // Probe limits. + parse_ok::("0xffffffff_ffffffff", "-1"); + parse_ok::("0x80000000_00000000", "0x8000_0000_0000_0000"); + parse_ok::("-0x80000000_00000000", "0x8000_0000_0000_0000"); + parse_err::("-0x80000000_00000001", + "Negative number too small for Imm64"); + parse_ok::("18446744073709551615", "-1"); + parse_ok::("-9223372036854775808", "0x8000_0000_0000_0000"); + // Overflow both the checked_add and checked_mul. + parse_err::("18446744073709551616", "Too large decimal Imm64"); + parse_err::("184467440737095516100", "Too large decimal Imm64"); + parse_err::("-9223372036854775809", + "Negative number too small for Imm64"); + + // Underscores are allowed where digits go. + parse_ok::("0_0", "0"); + parse_ok::("-_10_0", "-100"); + parse_ok::("_10_", "10"); + parse_ok::("0x97_88_bb", "0x0097_88bb"); + parse_ok::("0x_97_", "151"); + + parse_err::("", "No digits in Imm64"); + parse_err::("-", "No digits in Imm64"); + parse_err::("_", "No digits in Imm64"); + parse_err::("0x", "No digits in Imm64"); + parse_err::("0x_", "No digits in Imm64"); + parse_err::("-0x", "No digits in Imm64"); + parse_err::(" ", "Invalid character in decimal Imm64"); + parse_err::("0 ", "Invalid character in decimal Imm64"); + parse_err::(" 0", "Invalid character in decimal Imm64"); + parse_err::("--", "Invalid character in decimal Imm64"); + parse_err::("-0x-", "Invalid character in hexadecimal Imm64"); + + // Hex count overflow. + parse_err::("0x0_0000_0000_0000_0000", + "Too many hexadecimal digits in Imm64"); + } + #[test] fn format_ieee32() { assert_eq!(format!("{}", Ieee32::new(0.0)), "0.0"); @@ -268,15 +601,81 @@ mod tests { assert_eq!(format!("{}", Ieee32::new(f32::NAN)), "NaN"); assert_eq!(format!("{}", Ieee32::new(-f32::NAN)), "-NaN"); // Construct some qNaNs with payloads. - assert_eq!(format!("{}", Ieee32::new_from_bits(0x7fc00001)), "NaN:0x1"); - assert_eq!(format!("{}", Ieee32::new_from_bits(0x7ff00001)), - "NaN:0x300001"); + assert_eq!(format!("{}", Ieee32::from_bits(0x7fc00001)), "NaN:0x1"); + assert_eq!(format!("{}", Ieee32::from_bits(0x7ff00001)), "NaN:0x300001"); // Signaling NaNs. - assert_eq!(format!("{}", Ieee32::new_from_bits(0x7f800001)), "sNaN:0x1"); - assert_eq!(format!("{}", Ieee32::new_from_bits(0x7fa00001)), + assert_eq!(format!("{}", Ieee32::from_bits(0x7f800001)), "sNaN:0x1"); + assert_eq!(format!("{}", Ieee32::from_bits(0x7fa00001)), "sNaN:0x200001"); } + #[test] + fn parse_ieee32() { + parse_ok::("0.0", "0.0"); + parse_ok::("-0.0", "-0.0"); + parse_ok::("0x0", "0.0"); + parse_ok::("0x0.0", "0.0"); + parse_ok::("0x.0", "0.0"); + parse_ok::("0x0.", "0.0"); + parse_ok::("0x1", "0x1.000000p0"); + parse_ok::("-0x1", "-0x1.000000p0"); + parse_ok::("0x10", "0x1.000000p4"); + parse_ok::("0x10.0", "0x1.000000p4"); + parse_err::("0.", "Float must be hexadecimal"); + parse_err::(".0", "Float must be hexadecimal"); + parse_err::("0", "Float must be hexadecimal"); + parse_err::("-0", "Float must be hexadecimal"); + parse_err::(".", "Float must be hexadecimal"); + parse_err::("", "Float must be hexadecimal"); + parse_err::("-", "Float must be hexadecimal"); + parse_err::("0x", "No digits"); + parse_err::("0x..", "Multiple radix points"); + + // Check significant bits. + parse_ok::("0x0.ffffff", "0x1.fffffep-1"); + parse_ok::("0x1.fffffe", "0x1.fffffep0"); + parse_ok::("0x3.fffffc", "0x1.fffffep1"); + parse_ok::("0x7.fffff8", "0x1.fffffep2"); + parse_ok::("0xf.fffff0", "0x1.fffffep3"); + parse_err::("0x1.ffffff", "Too many significant bits"); + parse_err::("0x1.fffffe0000000000", "Too many digits"); + + // Exponents. + parse_ok::("0x1p3", "0x1.000000p3"); + parse_ok::("0x1p-3", "0x1.000000p-3"); + parse_ok::("0x1.0p3", "0x1.000000p3"); + parse_ok::("0x2.0p3", "0x1.000000p4"); + parse_ok::("0x1.0p127", "0x1.000000p127"); + parse_ok::("0x1.0p-126", "0x1.000000p-126"); + parse_ok::("0x0.1p-122", "0x1.000000p-126"); + parse_err::("0x2.0p127", "Magnitude too large"); + + // Subnormals. + parse_ok::("0x1.0p-127", "0x0.800000p-126"); + parse_ok::("0x1.0p-149", "0x0.000002p-126"); + parse_ok::("0x0.000002p-126", "0x0.000002p-126"); + parse_err::("0x0.100001p-126", "Subnormal underflow"); + parse_err::("0x1.8p-149", "Subnormal underflow"); + parse_err::("0x1.0p-150", "Magnitude too small"); + + // NaNs and Infs. + parse_ok::("Inf", "Inf"); + parse_ok::("-Inf", "-Inf"); + parse_ok::("NaN", "NaN"); + parse_ok::("-NaN", "-NaN"); + parse_ok::("NaN:0x0", "NaN"); + parse_err::("NaN:", "Float must be hexadecimal"); + parse_err::("NaN:0", "Float must be hexadecimal"); + parse_err::("NaN:0x", "Invalid NaN payload"); + parse_ok::("NaN:0x000001", "NaN:0x1"); + parse_ok::("NaN:0x300001", "NaN:0x300001"); + parse_err::("NaN:0x400001", "Invalid NaN payload"); + parse_ok::("sNaN:0x1", "sNaN:0x1"); + parse_err::("sNaN:0x0", "Invalid sNaN payload"); + parse_ok::("sNaN:0x200001", "sNaN:0x200001"); + parse_err::("sNaN:0x400001", "Invalid sNaN payload"); + } + #[test] fn format_ieee64() { assert_eq!(format!("{}", Ieee64::new(0.0)), "0.0"); @@ -303,14 +702,82 @@ mod tests { assert_eq!(format!("{}", Ieee64::new(f64::NAN)), "NaN"); assert_eq!(format!("{}", Ieee64::new(-f64::NAN)), "-NaN"); // Construct some qNaNs with payloads. - assert_eq!(format!("{}", Ieee64::new_from_bits(0x7ff8000000000001)), + assert_eq!(format!("{}", Ieee64::from_bits(0x7ff8000000000001)), "NaN:0x1"); - assert_eq!(format!("{}", Ieee64::new_from_bits(0x7ffc000000000001)), + assert_eq!(format!("{}", Ieee64::from_bits(0x7ffc000000000001)), "NaN:0x4000000000001"); // Signaling NaNs. - assert_eq!(format!("{}", Ieee64::new_from_bits(0x7ff0000000000001)), + assert_eq!(format!("{}", Ieee64::from_bits(0x7ff0000000000001)), "sNaN:0x1"); - assert_eq!(format!("{}", Ieee64::new_from_bits(0x7ff4000000000001)), + assert_eq!(format!("{}", Ieee64::from_bits(0x7ff4000000000001)), "sNaN:0x4000000000001"); } + + #[test] + fn parse_ieee64() { + parse_ok::("0.0", "0.0"); + parse_ok::("-0.0", "-0.0"); + parse_ok::("0x0", "0.0"); + parse_ok::("0x0.0", "0.0"); + parse_ok::("0x.0", "0.0"); + parse_ok::("0x0.", "0.0"); + parse_ok::("0x1", "0x1.0000000000000p0"); + parse_ok::("-0x1", "-0x1.0000000000000p0"); + parse_ok::("0x10", "0x1.0000000000000p4"); + parse_ok::("0x10.0", "0x1.0000000000000p4"); + parse_err::("0.", "Float must be hexadecimal"); + parse_err::(".0", "Float must be hexadecimal"); + parse_err::("0", "Float must be hexadecimal"); + parse_err::("-0", "Float must be hexadecimal"); + parse_err::(".", "Float must be hexadecimal"); + parse_err::("", "Float must be hexadecimal"); + parse_err::("-", "Float must be hexadecimal"); + parse_err::("0x", "No digits"); + parse_err::("0x..", "Multiple radix points"); + + // Check significant bits. + parse_ok::("0x0.fffffffffffff8", "0x1.fffffffffffffp-1"); + parse_ok::("0x1.fffffffffffff", "0x1.fffffffffffffp0"); + parse_ok::("0x3.ffffffffffffe", "0x1.fffffffffffffp1"); + parse_ok::("0x7.ffffffffffffc", "0x1.fffffffffffffp2"); + parse_ok::("0xf.ffffffffffff8", "0x1.fffffffffffffp3"); + parse_err::("0x3.fffffffffffff", "Too many significant bits"); + parse_err::("0x001.fffffe00000000", "Too many digits"); + + // Exponents. + parse_ok::("0x1p3", "0x1.0000000000000p3"); + parse_ok::("0x1p-3", "0x1.0000000000000p-3"); + parse_ok::("0x1.0p3", "0x1.0000000000000p3"); + parse_ok::("0x2.0p3", "0x1.0000000000000p4"); + parse_ok::("0x1.0p1023", "0x1.0000000000000p1023"); + parse_ok::("0x1.0p-1022", "0x1.0000000000000p-1022"); + parse_ok::("0x0.1p-1018", "0x1.0000000000000p-1022"); + parse_err::("0x2.0p1023", "Magnitude too large"); + + // Subnormals. + parse_ok::("0x1.0p-1023", "0x0.8000000000000p-1022"); + parse_ok::("0x1.0p-1074", "0x0.0000000000001p-1022"); + parse_ok::("0x0.0000000000001p-1022", "0x0.0000000000001p-1022"); + parse_err::("0x0.10000000000008p-1022", "Subnormal underflow"); + parse_err::("0x1.8p-1074", "Subnormal underflow"); + parse_err::("0x1.0p-1075", "Magnitude too small"); + + // NaNs and Infs. + parse_ok::("Inf", "Inf"); + parse_ok::("-Inf", "-Inf"); + parse_ok::("NaN", "NaN"); + parse_ok::("-NaN", "-NaN"); + parse_ok::("NaN:0x0", "NaN"); + parse_err::("NaN:", "Float must be hexadecimal"); + parse_err::("NaN:0", "Float must be hexadecimal"); + parse_err::("NaN:0x", "Invalid NaN payload"); + parse_ok::("NaN:0x000001", "NaN:0x1"); + parse_ok::("NaN:0x4000000000001", "NaN:0x4000000000001"); + parse_err::("NaN:0x8000000000001", "Invalid NaN payload"); + parse_ok::("sNaN:0x1", "sNaN:0x1"); + parse_err::("sNaN:0x0", "Invalid sNaN payload"); + parse_ok::("sNaN:0x4000000000001", "sNaN:0x4000000000001"); + parse_err::("sNaN:0x8000000000001", "Invalid sNaN payload"); + } + } From c0f77f35c73a8c2a475eb1b931363e2970a626ff Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Apr 2016 14:25:54 -0700 Subject: [PATCH 0049/3084] Add function signatures. Describe function argument and return value types along with flags for passing values in an ABI-compliant way. --- cranelift/src/libcretonne/types.rs | 131 +++++++++++++++++++++++++++++ 1 file changed, 131 insertions(+) diff --git a/cranelift/src/libcretonne/types.rs b/cranelift/src/libcretonne/types.rs index 88a2b6cad2..6c8fb90d90 100644 --- a/cranelift/src/libcretonne/types.rs +++ b/cranelift/src/libcretonne/types.rs @@ -3,6 +3,12 @@ use std::fmt::{self, Display, Formatter, Write}; +// ====--------------------------------------------------------------------------------------====// +// +// Value types +// +// ====--------------------------------------------------------------------------------------====// + /// The type of an SSA value. /// /// The `VOID` type is only used for instructions that produce no value. It can't be part of a SIMD @@ -177,6 +183,107 @@ impl Display for Type { } } +// ====--------------------------------------------------------------------------------------====// +// +// Function signatures +// +// ====--------------------------------------------------------------------------------------====// + +/// Function argument extension options. +/// +/// On some architectures, small integer function arguments are extended to the width of a +/// general-purpose register. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub enum ArgumentExtension { + /// No extension, high bits are indeterminate. + None, + /// Unsigned extension: high bits in register are 0. + Uext, + /// Signed extension: high bits in register replicate sign bit. + Sext, +} + +/// Function argument or return value type. +/// +/// This describes the value type being passed to or from a function along with flags that affect +/// how the argument is passed. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct ArgumentType { + pub value_type: Type, + pub extension: ArgumentExtension, + /// Place this argument in a register if possible. + pub inreg: bool, +} + +/// Function signature. +/// +/// The function signature describes the types of arguments and return values along with other +/// details that are needed to call a function correctly. +pub struct Signature { + pub argument_types: Vec, + pub return_types: Vec, +} + +impl ArgumentType { + pub fn new(vt: Type) -> ArgumentType { + ArgumentType { + value_type: vt, + extension: ArgumentExtension::None, + inreg: false, + } + } +} + +impl Display for ArgumentType { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + try!(write!(f, "{}", self.value_type)); + match self.extension { + ArgumentExtension::None => {} + ArgumentExtension::Uext => try!(write!(f, " uext")), + ArgumentExtension::Sext => try!(write!(f, " sext")), + } + if self.inreg { + try!(write!(f, " inreg")); + } + Ok(()) + } +} + +impl Signature { + pub fn new() -> Signature { + Signature { + argument_types: Vec::new(), + return_types: Vec::new(), + } + } +} + +fn write_list(f: &mut Formatter, args: &Vec) -> fmt::Result { + match args.split_first() { + None => {} + Some((first, rest)) => { + try!(write!(f, "{}", first)); + for arg in rest { + try!(write!(f, ", {}", arg)); + } + } + } + Ok(()) +} + +impl Display for Signature { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + try!(write!(f, "(")); + try!(write_list(f, &self.argument_types)); + try!(write!(f, ")")); + if !self.return_types.is_empty() { + try!(write!(f, " -> ")); + try!(write_list(f, &self.return_types)); + } + Ok(()) + } +} + #[cfg(test)] mod tests { use super::*; @@ -248,4 +355,28 @@ mod tests { assert_eq!(format!("{}", I8.by(64)), "i8x64"); assert_eq!(format!("{}", F64.by(2)), "f64x2"); } + + #[test] + fn argument_type() { + let mut t = ArgumentType::new(I32); + assert_eq!(format!("{}", t), "i32"); + t.extension = ArgumentExtension::Uext; + assert_eq!(format!("{}", t), "i32 uext"); + t.inreg = true; + assert_eq!(format!("{}", t), "i32 uext inreg"); + } + + #[test] + fn signatures() { + let mut sig = Signature::new(); + assert_eq!(format!("{}", sig), "()"); + sig.argument_types.push(ArgumentType::new(I32)); + assert_eq!(format!("{}", sig), "(i32)"); + sig.return_types.push(ArgumentType::new(F32)); + assert_eq!(format!("{}", sig), "(i32) -> f32"); + sig.argument_types.push(ArgumentType::new(I32.by(4))); + assert_eq!(format!("{}", sig), "(i32, i32x4) -> f32"); + sig.return_types.push(ArgumentType::new(B8)); + assert_eq!(format!("{}", sig), "(i32, i32x4) -> f32, b8"); + } } From bbeafde243475a923eab544d7407c2f19a0736a4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Apr 2016 14:51:16 -0700 Subject: [PATCH 0050/3084] Type::by() returns an Optional. Don't use assertions to enforce the limits on SIMD lanes in a type, Type is too fundamental for that. Instead, the Vector-forming by() method returns an Optional, and None if the requested SIMD vector is not valid. Same for Type::half_vector(). --- cranelift/src/libcretonne/types.rs | 53 ++++++++++++++++++------------ 1 file changed, 32 insertions(+), 21 deletions(-) diff --git a/cranelift/src/libcretonne/types.rs b/cranelift/src/libcretonne/types.rs index 6c8fb90d90..0aaa30841b 100644 --- a/cranelift/src/libcretonne/types.rs +++ b/cranelift/src/libcretonne/types.rs @@ -147,21 +147,26 @@ impl Type { /// /// If this is already a SIMD vector type, this produces a SIMD vector type with `n * /// self.lane_count()` lanes. - pub fn by(self, n: u16) -> Type { - debug_assert!(self.lane_bits() > 0, - "Can't make SIMD vectors with void lanes."); - debug_assert!(n.is_power_of_two(), - "Number of SIMD lanes must be a power of two"); + pub fn by(self, n: u16) -> Option { + if self.lane_bits() == 0 || !n.is_power_of_two() { + return None; + } let log2_lanes: u32 = n.trailing_zeros(); let new_type = self.0 as u32 + (log2_lanes << 4); - assert!(new_type < 0x90, "No more than 256 SIMD lanes supported"); - Type(new_type as u8) + if new_type < 0x90 { + Some(Type(new_type as u8)) + } else { + None + } } /// Get a SIMD vector with half the number of lanes. - pub fn half_vector(self) -> Type { - assert!(!self.is_scalar(), "Expecting a proper SIMD vector type."); - Type(self.0 - 0x10) + pub fn half_vector(self) -> Option { + if self.is_scalar() { + None + } else { + Some(Type(self.0 - 0x10)) + } } } @@ -320,13 +325,16 @@ mod tests { #[test] fn vectors() { - let big = F64.by(256); + let big = F64.by(256).unwrap(); assert_eq!(big.lane_bits(), 64); assert_eq!(big.lane_count(), 256); assert_eq!(big.bits(), 64 * 256); - assert_eq!(format!("{}", big.half_vector()), "f64x128"); - assert_eq!(format!("{}", B1.by(2).half_vector()), "b1"); + assert_eq!(format!("{}", big.half_vector().unwrap()), "f64x128"); + assert_eq!(format!("{}", B1.by(2).unwrap().half_vector().unwrap()), + "b1"); + assert_eq!(I32.half_vector(), None); + assert_eq!(VOID.half_vector(), None); } #[test] @@ -347,13 +355,16 @@ mod tests { #[test] fn format_vectors() { - assert_eq!(format!("{}", B1.by(8)), "b1x8"); - assert_eq!(format!("{}", B8.by(1)), "b8"); - assert_eq!(format!("{}", B16.by(256)), "b16x256"); - assert_eq!(format!("{}", B32.by(4).by(2)), "b32x8"); - assert_eq!(format!("{}", B64.by(8)), "b64x8"); - assert_eq!(format!("{}", I8.by(64)), "i8x64"); - assert_eq!(format!("{}", F64.by(2)), "f64x2"); + assert_eq!(format!("{}", B1.by(8).unwrap()), "b1x8"); + assert_eq!(format!("{}", B8.by(1).unwrap()), "b8"); + assert_eq!(format!("{}", B16.by(256).unwrap()), "b16x256"); + assert_eq!(format!("{}", B32.by(4).unwrap().by(2).unwrap()), "b32x8"); + assert_eq!(format!("{}", B64.by(8).unwrap()), "b64x8"); + assert_eq!(format!("{}", I8.by(64).unwrap()), "i8x64"); + assert_eq!(format!("{}", F64.by(2).unwrap()), "f64x2"); + assert_eq!(I8.by(3), None); + assert_eq!(I8.by(512), None); + assert_eq!(VOID.by(4), None); } #[test] @@ -374,7 +385,7 @@ mod tests { assert_eq!(format!("{}", sig), "(i32)"); sig.return_types.push(ArgumentType::new(F32)); assert_eq!(format!("{}", sig), "(i32) -> f32"); - sig.argument_types.push(ArgumentType::new(I32.by(4))); + sig.argument_types.push(ArgumentType::new(I32.by(4).unwrap())); assert_eq!(format!("{}", sig), "(i32, i32x4) -> f32"); sig.return_types.push(ArgumentType::new(B8)); assert_eq!(format!("{}", sig), "(i32, i32x4) -> f32, b8"); From 1228abca29a0e7c280cec9295ea1e17c491c5c0e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 11 Apr 2016 16:41:00 -0700 Subject: [PATCH 0051/3084] Begin lexer implementation. Add a lexer module which implements the lexical analysis of .cton files. --- cranelift/src/libctonfile/lexer.rs | 425 +++++++++++++++++++++++++++++ cranelift/src/libctonfile/lib.rs | 6 +- 2 files changed, 430 insertions(+), 1 deletion(-) create mode 100644 cranelift/src/libctonfile/lexer.rs diff --git a/cranelift/src/libctonfile/lexer.rs b/cranelift/src/libctonfile/lexer.rs new file mode 100644 index 0000000000..69d9c98580 --- /dev/null +++ b/cranelift/src/libctonfile/lexer.rs @@ -0,0 +1,425 @@ + +// ====--------------------------------------------------------------------------------------====// +// +// Lexical analysis for .cton files. +// +// ====--------------------------------------------------------------------------------------====// + +use std::str::CharIndices; + +/// The location of a `Token` or `Error`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Location { + pub line_number: usize, +} + +/// A Token returned from the `Lexer`. +/// +/// Some variants may contains references to the original source text, so the `Token` has the same +/// lifetime as the source. +#[derive(Debug, PartialEq, Eq, Clone, Copy)] +pub enum Token<'a> { + Comment(&'a str), + LPar, // '(' + RPar, // ')' + LBrace, // '{' + RBrace, // '}' + Comma, // ',' + Dot, // '.' + Colon, // ':' + Equal, // '=' + Arrow, // '->' + Function, // 'function' + Entry, // 'entry' + Float(&'a str), // Floating point immediate + Integer(&'a str), // Integer immediate + ValueDirect(u32), // v12 + ValueExtended(u32), // vx7 + Ebb(u32), // ebb3 + StackSlot(u32), // ss3 + Identifier(&'a str), // Unrecognized identifier (opcode, enumerator, ...) +} + +/// A `Token` with an associated location. +#[derive(Debug, PartialEq, Eq)] +pub struct LocatedToken<'a> { + pub token: Token<'a>, + pub location: Location, +} + +/// Wrap up a `Token` with the given location. +fn token<'a>(token: Token<'a>, loc: Location) -> Result, LocatedError> { + Ok(LocatedToken { + token: token, + location: loc, + }) +} + +/// An error from the lexical analysis. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Error { + InvalidChar, +} + +/// An `Error` with an associated Location. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct LocatedError { + pub error: Error, + pub location: Location, +} + +/// Wrap up an `Error` with the given location. +fn error<'a>(error: Error, loc: Location) -> Result, LocatedError> { + Err(LocatedError { + error: error, + location: loc, + }) +} + +/// Lexical analysis. +/// +/// A `Lexer` reads text from a `&str` and provides a sequence of tokens. +/// +/// Also keep track of a line number for error reporting. +/// +pub struct Lexer<'a> { + // Complete source being processed. + source: &'a str, + + // Iterator into `source`. + chars: CharIndices<'a>, + + // Next character to be processed, or `None` at the end. + lookahead: Option, + + // Index into `source` of lookahead character. + pos: usize, + + // Current line number. + line_number: usize, +} + +impl<'a> Lexer<'a> { + pub fn new(s: &'a str) -> Lexer { + let mut lex = Lexer { + source: s, + chars: s.char_indices(), + lookahead: None, + pos: 0, + line_number: 1, + }; + // Advance to the first char. + lex.next_ch(); + lex + } + + // Advance to the next character. + // Return the next lookahead character, or None when the end is encountered. + // Always update cur_ch to reflect + fn next_ch(&mut self) -> Option { + if self.lookahead == Some('\n') { + self.line_number += 1; + } + match self.chars.next() { + Some((idx, ch)) => { + self.pos = idx; + self.lookahead = Some(ch); + } + None => { + self.pos = self.source.len(); + self.lookahead = None; + } + } + self.lookahead + } + + // Get the location corresponding to `lookahead`. + fn loc(&self) -> Location { + Location { line_number: self.line_number } + } + + // Starting from `lookahead`, are we looking at `prefix`? + fn looking_at(&self, prefix: &str) -> bool { + self.source[self.pos..].starts_with(prefix) + } + + // Scan a single-char token. + fn scan_char(&mut self, tok: Token<'a>) -> Result, LocatedError> { + assert!(self.lookahead != None); + let loc = self.loc(); + self.next_ch(); + token(tok, loc) + } + + // Scan a multi-char token. + fn scan_chars(&mut self, + count: usize, + tok: Token<'a>) + -> Result, LocatedError> { + let loc = self.loc(); + for _ in 0..count { + assert!(self.lookahead != None); + self.next_ch(); + } + token(tok, loc) + } + + // Scan a comment extending to the end of the current line. + fn scan_comment(&mut self) -> Result, LocatedError> { + let begin = self.pos; + let loc = self.loc(); + loop { + match self.next_ch() { + None | Some('\n') => { + let text = &self.source[begin..self.pos]; + return token(Token::Comment(text), loc); + } + _ => {} + } + } + } + + // Scan a number token which can represent either an integer or floating point number. + // + // Accept the following forms: + // + // - `10`: Integer + // - `-10`: Integer + // - `0xff_00`: Integer + // - `0.0`: Float + // - `0x1.f`: Float + // - `-0x2.4`: Float + // - `0x0.4p-34`: Float + // + // This function does not filter out all invalid numbers. It depends in the context-sensitive + // decoding of the text for that. For example, the number of allowed digits an an Ieee32` and + // an `Ieee64` constant are different. + fn scan_number(&mut self) -> Result, LocatedError> { + let begin = self.pos; + let loc = self.loc(); + let mut is_float = false; + + // Skip a leading sign. + if self.lookahead == Some('-') { + self.next_ch(); + } + + // Check for NaNs with payloads. + if self.looking_at("NaN:") || self.looking_at("sNaN:") { + // Skip the `NaN:` prefix, the loop below won't accept it. + // We expect a hexadecimal number to follow the colon. + while self.next_ch() != Some(':') {} + is_float = true; + } else if self.looking_at("NaN") || self.looking_at("Inf") { + // This is Inf or a default quiet NaN. + is_float = true; + } + + // Look for the end of this number. Detect the radix point if there is one. + loop { + match self.next_ch() { + Some('-') | Some('_') => {} + Some('.') => is_float = true, + Some(ch) if ch.is_alphanumeric() => {} + _ => break, + } + } + let text = &self.source[begin..self.pos]; + if is_float { + token(Token::Float(text), loc) + } else { + token(Token::Integer(text), loc) + } + } + + // Scan a 'word', which is an identifier-like sequence of characters beginning with '_' or an + // alphabetic char, followed by zero or more alphanumeric or '_' characters. + // + // + fn scan_word(&mut self) -> Result, LocatedError> { + let begin = self.pos; + let loc = self.loc(); + let mut trailing_digits = 0usize; + + assert!(self.lookahead == Some('_') || self.lookahead.unwrap().is_alphabetic()); + loop { + match self.next_ch() { + Some(ch) if ch.is_digit(10) => trailing_digits += 1, + Some('_') => trailing_digits = 0, + Some(ch) if ch.is_alphabetic() => trailing_digits = 0, + _ => break, + } + } + let text = &self.source[begin..self.pos]; + + match if trailing_digits == 0 { + Self::keyword(text) + } else { + // Look for numbered well-known entities like ebb15, v45, ... + let (prefix, suffix) = text.split_at(text.len() - trailing_digits); + Self::numbered_entity(prefix, suffix) + } { + Some(t) => token(t, loc), + None => token(Token::Identifier(text), loc), + } + } + + // Recognize a keyword. + fn keyword(text: &str) -> Option> { + match text { + "function" => Some(Token::Function), + "entry" => Some(Token::Entry), + _ => None, + } + } + + // If prefix is a well-known entity prefix and suffix is a valid entity number, return the + // decoded token. + fn numbered_entity(prefix: &str, suffix: &str) -> Option> { + // Reject non-canonical numbers like v0001. + if suffix.len() > 1 && suffix.starts_with('0') { + return None; + } + + let value: u32 = match suffix.parse() { + Ok(v) => v, + _ => return None, + }; + + match prefix { + "v" => Some(Token::ValueDirect(value)), + "vx" => Some(Token::ValueExtended(value)), + "ebb" => Some(Token::Ebb(value)), + "ss" => Some(Token::StackSlot(value)), + _ => None, + } + } + + /// Get the next token or a lexical error. + /// + /// Return None when the end of the source is encountered. + pub fn next(&mut self) -> Option, LocatedError>> { + loop { + let loc = self.loc(); + return match self.lookahead { + None => None, + Some(';') => Some(self.scan_comment()), + Some('(') => Some(self.scan_char(Token::LPar)), + Some(')') => Some(self.scan_char(Token::RPar)), + Some('{') => Some(self.scan_char(Token::LBrace)), + Some('}') => Some(self.scan_char(Token::RBrace)), + Some(',') => Some(self.scan_char(Token::Comma)), + Some('.') => Some(self.scan_char(Token::Dot)), + Some(':') => Some(self.scan_char(Token::Colon)), + Some('=') => Some(self.scan_char(Token::Equal)), + Some('-') => { + if self.looking_at("->") { + Some(self.scan_chars(2, Token::Arrow)) + } else { + Some(self.scan_number()) + } + } + Some(ch) if ch.is_digit(10) => Some(self.scan_number()), + Some(ch) if ch.is_alphabetic() => Some(self.scan_word()), + Some(ch) if ch.is_whitespace() => { + self.next_ch(); + continue; + } + _ => { + // Skip invalid char, return error. + self.next_ch(); + Some(error(Error::InvalidChar, loc)) + } + }; + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn token<'a>(token: Token<'a>, line: usize) -> Option, LocatedError>> { + Some(super::token(token, Location { line_number: line })) + } + + fn error<'a>(error: Error, line: usize) -> Option, LocatedError>> { + Some(super::error(error, Location { line_number: line })) + } + + #[test] + fn make_lexer() { + let mut l1 = Lexer::new(""); + let mut l2 = Lexer::new(" "); + let mut l3 = Lexer::new("\n "); + + assert_eq!(l1.next(), None); + assert_eq!(l2.next(), None); + assert_eq!(l3.next(), None); + } + + #[test] + fn lex_comment() { + let mut lex = Lexer::new("; hello"); + assert_eq!(lex.next(), token(Token::Comment("; hello"), 1)); + assert_eq!(lex.next(), None); + + lex = Lexer::new("\n ;hello\n;foo"); + assert_eq!(lex.next(), token(Token::Comment(";hello"), 2)); + assert_eq!(lex.next(), token(Token::Comment(";foo"), 3)); + assert_eq!(lex.next(), None); + + // Scan a comment after an invalid char. + let mut lex = Lexer::new("#; hello"); + assert_eq!(lex.next(), error(Error::InvalidChar, 1)); + assert_eq!(lex.next(), token(Token::Comment("; hello"), 1)); + assert_eq!(lex.next(), None); + } + + #[test] + fn lex_chars() { + let mut lex = Lexer::new("(); hello\n = :{, }."); + assert_eq!(lex.next(), token(Token::LPar, 1)); + assert_eq!(lex.next(), token(Token::RPar, 1)); + assert_eq!(lex.next(), token(Token::Comment("; hello"), 1)); + assert_eq!(lex.next(), token(Token::Equal, 2)); + assert_eq!(lex.next(), token(Token::Colon, 2)); + assert_eq!(lex.next(), token(Token::LBrace, 2)); + assert_eq!(lex.next(), token(Token::Comma, 2)); + assert_eq!(lex.next(), token(Token::RBrace, 2)); + assert_eq!(lex.next(), token(Token::Dot, 2)); + assert_eq!(lex.next(), None); + } + + #[test] + fn lex_numbers() { + let mut lex = Lexer::new(" 0 2_000 -1,0xf -0x0 0.0 0x0.4p-34"); + assert_eq!(lex.next(), token(Token::Integer("0"), 1)); + assert_eq!(lex.next(), token(Token::Integer("2_000"), 1)); + assert_eq!(lex.next(), token(Token::Integer("-1"), 1)); + assert_eq!(lex.next(), token(Token::Comma, 1)); + assert_eq!(lex.next(), token(Token::Integer("0xf"), 1)); + assert_eq!(lex.next(), token(Token::Integer("-0x0"), 1)); + assert_eq!(lex.next(), token(Token::Float("0.0"), 1)); + assert_eq!(lex.next(), token(Token::Float("0x0.4p-34"), 1)); + assert_eq!(lex.next(), None); + } + + #[test] + fn lex_identifiers() { + let mut lex = Lexer::new("v0 v00 vx01 ebb1234567890 ebb5234567890 entry v1x vx1 vxvx4 \ + function0 function"); + assert_eq!(lex.next(), token(Token::ValueDirect(0), 1)); + assert_eq!(lex.next(), token(Token::Identifier("v00"), 1)); + assert_eq!(lex.next(), token(Token::Identifier("vx01"), 1)); + assert_eq!(lex.next(), token(Token::Ebb(1234567890), 1)); + assert_eq!(lex.next(), token(Token::Identifier("ebb5234567890"), 1)); + assert_eq!(lex.next(), token(Token::Entry, 1)); + assert_eq!(lex.next(), token(Token::Identifier("v1x"), 1)); + assert_eq!(lex.next(), token(Token::ValueExtended(1), 1)); + assert_eq!(lex.next(), token(Token::Identifier("vxvx4"), 1)); + assert_eq!(lex.next(), token(Token::Identifier("function0"), 1)); + assert_eq!(lex.next(), token(Token::Function, 1)); + assert_eq!(lex.next(), None); + } +} diff --git a/cranelift/src/libctonfile/lib.rs b/cranelift/src/libctonfile/lib.rs index 3271ad035c..9dcb169b7d 100644 --- a/cranelift/src/libctonfile/lib.rs +++ b/cranelift/src/libctonfile/lib.rs @@ -1,12 +1,16 @@ // ====------------------------------------------------------------------------------------==== // // -// Cretonne file read/write library +// Cretonne file read/write library. // // ====------------------------------------------------------------------------------------==== // // // The libctonfile library supports reading and writing .cton files. This functionality is needed // for testing Cretonne, but is not essential for a JIT compiler. +// +// ====------------------------------------------------------------------------------------==== // extern crate cretonne; + +pub mod lexer; pub mod parser; From 7ccef63077d4f20948b0a84b0751a90b8345f458 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Apr 2016 15:38:54 -0700 Subject: [PATCH 0052/3084] Handle value type names in the lexer. --- cranelift/src/libctonfile/lexer.rs | 43 ++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/cranelift/src/libctonfile/lexer.rs b/cranelift/src/libctonfile/lexer.rs index 69d9c98580..855dd34056 100644 --- a/cranelift/src/libctonfile/lexer.rs +++ b/cranelift/src/libctonfile/lexer.rs @@ -6,6 +6,7 @@ // ====--------------------------------------------------------------------------------------====// use std::str::CharIndices; +use cretonne::types; /// The location of a `Token` or `Error`. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -33,6 +34,7 @@ pub enum Token<'a> { Entry, // 'entry' Float(&'a str), // Floating point immediate Integer(&'a str), // Integer immediate + Type(types::Type), // i32, f32, b32x4, ... ValueDirect(u32), // v12 ValueExtended(u32), // vx7 Ebb(u32), // ebb3 @@ -257,7 +259,7 @@ impl<'a> Lexer<'a> { } else { // Look for numbered well-known entities like ebb15, v45, ... let (prefix, suffix) = text.split_at(text.len() - trailing_digits); - Self::numbered_entity(prefix, suffix) + Self::numbered_entity(prefix, suffix).or_else(|| Self::value_type(text, prefix, suffix)) } { Some(t) => token(t, loc), None => token(Token::Identifier(text), loc), @@ -295,6 +297,39 @@ impl<'a> Lexer<'a> { } } + // Recognize a scalar or vector type. + fn value_type(text: &str, prefix: &str, suffix: &str) -> Option> { + let is_vector = prefix.ends_with('x'); + let scalar = if is_vector { + &prefix[0..prefix.len() - 1] + } else { + text + }; + let base_type = match scalar { + "i8" => types::I8, + "i16" => types::I16, + "i32" => types::I32, + "i64" => types::I64, + "f32" => types::F32, + "f64" => types::F64, + "b1" => types::B1, + "b8" => types::B8, + "b16" => types::B16, + "b32" => types::B32, + "b64" => types::B64, + _ => return None, + }; + if is_vector { + let lanes: u16 = match suffix.parse() { + Ok(v) => v, + _ => return None, + }; + base_type.by(lanes).map(|t| Token::Type(t)) + } else { + Some(Token::Type(base_type)) + } + } + /// Get the next token or a lexical error. /// /// Return None when the end of the source is encountered. @@ -338,6 +373,7 @@ impl<'a> Lexer<'a> { #[cfg(test)] mod tests { use super::*; + use cretonne::types; fn token<'a>(token: Token<'a>, line: usize) -> Option, LocatedError>> { Some(super::token(token, Location { line_number: line })) @@ -408,7 +444,7 @@ mod tests { #[test] fn lex_identifiers() { let mut lex = Lexer::new("v0 v00 vx01 ebb1234567890 ebb5234567890 entry v1x vx1 vxvx4 \ - function0 function"); + function0 function b1 i32x4 f32x5"); assert_eq!(lex.next(), token(Token::ValueDirect(0), 1)); assert_eq!(lex.next(), token(Token::Identifier("v00"), 1)); assert_eq!(lex.next(), token(Token::Identifier("vx01"), 1)); @@ -420,6 +456,9 @@ mod tests { assert_eq!(lex.next(), token(Token::Identifier("vxvx4"), 1)); assert_eq!(lex.next(), token(Token::Identifier("function0"), 1)); assert_eq!(lex.next(), token(Token::Function, 1)); + assert_eq!(lex.next(), token(Token::Type(types::B1), 1)); + assert_eq!(lex.next(), token(Token::Type(types::I32.by(4).unwrap()), 1)); + assert_eq!(lex.next(), token(Token::Identifier("f32x5"), 1)); assert_eq!(lex.next(), None); } } From c69a21f79e6f6d1874cce8f3d2ccb6532f42d3e8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Apr 2016 13:38:50 -0700 Subject: [PATCH 0053/3084] Parser for .cton files. Recursive descent parser, although with this simple grammar there won't be any recursion. --- cranelift/src/libctonfile/parser.rs | 202 +++++++++++++++++++++++++++- 1 file changed, 201 insertions(+), 1 deletion(-) diff --git a/cranelift/src/libctonfile/parser.rs b/cranelift/src/libctonfile/parser.rs index 44cf3a2500..d75d21ee24 100644 --- a/cranelift/src/libctonfile/parser.rs +++ b/cranelift/src/libctonfile/parser.rs @@ -1,2 +1,202 @@ -use cretonne; +// ====--------------------------------------------------------------------------------------====// +// +// Parser for .cton files. +// +// ====--------------------------------------------------------------------------------------====// + +use std::result; +use lexer::{self, Lexer, Token}; +use cretonne::{types, repr}; + +pub use lexer::Location; + +/// A parse error is returned when the parse failed. +pub struct Error { + pub location: Location, + pub message: String, +} + +pub type Result = result::Result; + +pub struct Parser<'a> { + lex: Lexer<'a>, + + lex_error: Option, + + // Current lookahead token. + lookahead: Option>, + + // Location of lookahead. + location: Location, +} + +impl<'a> Parser<'a> { + // Consume the current lookahead token and return it. + fn consume(&mut self) -> Token<'a> { + self.lookahead.take().expect("No token to consume") + } + + // Get the current lookahead token, after making sure there is one. + fn token(&mut self) -> Option> { + if self.lookahead == None { + match self.lex.next() { + Some(Ok(lexer::LocatedToken { token, location })) => { + self.lookahead = Some(token); + self.location = location; + } + Some(Err(lexer::LocatedError { error, location })) => { + self.lex_error = Some(error); + self.location = location; + } + None => {} + } + } + return self.lookahead; + } + + // Generate an error. + fn error(&self, message: &str) -> Error { + Error { + location: self.location, + message: + // If we have a lexer error latched, report that. + match self.lex_error { + Some(lexer::Error::InvalidChar) => "Invalid character".to_string(), + None => message.to_string(), + } + } + } + + // Match and consume a token without payload. + fn match_token(&mut self, want: Token<'a>, err_msg: &str) -> Result> { + match self.token() { + Some(ref t) if *t == want => Ok(self.consume()), + _ => Err(self.error(err_msg)), + } + } + + // if the next token is a `want`, consume it, otherwise do nothing. + fn optional(&mut self, want: Token<'a>) -> bool { + match self.token() { + Some(t) if t == want => { + self.consume(); + true + } + _ => false, + } + } + + // Parse a whole function definition. + // + // function ::= * "function" name signature { ... } + // + fn parse_function(&mut self) -> Result { + try!(self.match_token(Token::Function, "Expected 'function' keyword")); + + // function ::= "function" * name signature { ... } + let name = try!(self.parse_function_name()); + + // function ::= "function" name * signature { ... } + let sig = try!(self.parse_signature()); + + let mut func = repr::Function::new(); + + try!(self.match_token(Token::LBrace, "Expected '{' before function body")); + try!(self.match_token(Token::RBrace, "Expected '}' after function body")); + + Ok(func) + } + + // Parse a function name. + // + // function ::= "function" * name signature { ... } + // + fn parse_function_name(&mut self) -> Result { + match self.token() { + Some(Token::Identifier(s)) => { + self.consume(); + Ok(s.to_string()) + } + _ => Err(self.error("Expected function name")), + } + } + + // Parse a function signature. + // + // signature ::= * "(" arglist ")" ["->" retlist] [call_conv] + // callconv ::= string + // + // function ::= "function" * name signature { ... } + // + fn parse_signature(&mut self) -> Result { + let mut sig = types::Signature::new(); + + try!(self.match_token(Token::LPar, "Expected function signature: '(' args... ')'")); + // signature ::= "(" * arglist ")" ["->" retlist] [call_conv] + sig.argument_types = try!(self.parse_argument_list()); + try!(self.match_token(Token::RPar, "Expected ')' after function arguments")); + if self.optional(Token::Arrow) { + sig.return_types = try!(self.parse_argument_list()); + if sig.return_types.is_empty() { + return Err(self.error("Missing return type after '->'")); + } + } + + // TBD: calling convention. + + Ok(sig) + } + + // Parse (possibly empty) list of function argument / return value types. + // + // arglist ::= * + // * arg + // * arglist "," arg + fn parse_argument_list(&mut self) -> Result> { + let mut list = Vec::new(); + // arglist ::= * + // * arg + match self.token() { + Some(Token::Type(_)) => list.push(try!(self.parse_argument_type())), + _ => return Ok(list), + } + + // arglist ::= arg * + // arglist * "," arg + while self.token() == Some(Token::Comma) { + // arglist ::= arglist * "," arg + self.consume(); + // arglist ::= arglist "," * arg + list.push(try!(self.parse_argument_type())); + } + + Ok(list) + } + + // Parse a single argument type with flags. + fn parse_argument_type(&mut self) -> Result { + // arg ::= * type + // * arg flag + let mut arg = match self.token() { + Some(Token::Type(t)) => types::ArgumentType::new(t), + _ => return Err(self.error("Expected argument type")), + }; + loop { + self.consume(); + // arg ::= arg * flag + match self.token() { + Some(Token::Identifier(s)) => { + match s { + "uext" => arg.extension = types::ArgumentExtension::Uext, + "sext" => arg.extension = types::ArgumentExtension::Sext, + "inreg" => arg.inreg = true, + _ => break, + } + } + _ => break, + } + } + Ok(arg) + } +} From 615d9825e58970f2a1c27c4af91c3ed7b5aafa54 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 28 Apr 2016 08:06:30 -0700 Subject: [PATCH 0054/3084] Simplify parser. Use 'if let' instead of 'match' where it makes sense. Use EBNF notation for the grammar rules. This simplifies repetition a lot. --- cranelift/docs/langref.rst | 6 +- cranelift/src/libctonfile/parser.rs | 91 +++++++++++++---------------- 2 files changed, 41 insertions(+), 56 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index df0e19b20a..2cad2999c6 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -413,11 +413,9 @@ calling convention: .. productionlist:: signature : "(" [arglist] ")" ["->" retlist] [call_conv] - arglist : arg - : arglist "," arg + arglist : arg { "," arg } retlist : arglist - arg : type - : arg flag + arg : type { flag } flag : "uext" | "sext" | "inreg" callconv : `string` diff --git a/cranelift/src/libctonfile/parser.rs b/cranelift/src/libctonfile/parser.rs index d75d21ee24..060883b77e 100644 --- a/cranelift/src/libctonfile/parser.rs +++ b/cranelift/src/libctonfile/parser.rs @@ -70,20 +70,20 @@ impl<'a> Parser<'a> { // Match and consume a token without payload. fn match_token(&mut self, want: Token<'a>, err_msg: &str) -> Result> { - match self.token() { - Some(ref t) if *t == want => Ok(self.consume()), - _ => Err(self.error(err_msg)), + if self.token() == Some(want) { + Ok(self.consume()) + } else { + Err(self.error(err_msg)) } } - // if the next token is a `want`, consume it, otherwise do nothing. + // If the next token is a `want`, consume it, otherwise do nothing. fn optional(&mut self, want: Token<'a>) -> bool { - match self.token() { - Some(t) if t == want => { - self.consume(); - true - } - _ => false, + if self.token() == Some(want) { + self.consume(); + true + } else { + false } } @@ -124,23 +124,19 @@ impl<'a> Parser<'a> { // Parse a function signature. // - // signature ::= * "(" arglist ")" ["->" retlist] [call_conv] - // callconv ::= string - // - // function ::= "function" * name signature { ... } + // signature ::= * "(" [arglist] ")" ["->" retlist] [call_conv] // fn parse_signature(&mut self) -> Result { let mut sig = types::Signature::new(); try!(self.match_token(Token::LPar, "Expected function signature: '(' args... ')'")); - // signature ::= "(" * arglist ")" ["->" retlist] [call_conv] - sig.argument_types = try!(self.parse_argument_list()); + // signature ::= "(" * [arglist] ")" ["->" retlist] [call_conv] + if self.token() != Some(Token::RPar) { + sig.argument_types = try!(self.parse_argument_list()); + } try!(self.match_token(Token::RPar, "Expected ')' after function arguments")); if self.optional(Token::Arrow) { sig.return_types = try!(self.parse_argument_list()); - if sig.return_types.is_empty() { - return Err(self.error("Missing return type after '->'")); - } } // TBD: calling convention. @@ -148,26 +144,19 @@ impl<'a> Parser<'a> { Ok(sig) } - // Parse (possibly empty) list of function argument / return value types. + // Parse list of function argument / return value types. + // + // arglist ::= * arg { "," arg } // - // arglist ::= * - // * arg - // * arglist "," arg fn parse_argument_list(&mut self) -> Result> { let mut list = Vec::new(); - // arglist ::= * - // * arg - match self.token() { - Some(Token::Type(_)) => list.push(try!(self.parse_argument_type())), - _ => return Ok(list), - } - // arglist ::= arg * - // arglist * "," arg - while self.token() == Some(Token::Comma) { - // arglist ::= arglist * "," arg - self.consume(); - // arglist ::= arglist "," * arg + // arglist ::= * arg { "," arg } + list.push(try!(self.parse_argument_type())); + + // arglist ::= arg * { "," arg } + while self.optional(Token::Comma) { + // arglist ::= arg { "," * arg } list.push(try!(self.parse_argument_type())); } @@ -176,27 +165,25 @@ impl<'a> Parser<'a> { // Parse a single argument type with flags. fn parse_argument_type(&mut self) -> Result { - // arg ::= * type - // * arg flag - let mut arg = match self.token() { - Some(Token::Type(t)) => types::ArgumentType::new(t), - _ => return Err(self.error("Expected argument type")), + // arg ::= * type { flag } + let mut arg = if let Some(Token::Type(t)) = self.token() { + types::ArgumentType::new(t) + } else { + return Err(self.error("Expected argument type")); }; - loop { - self.consume(); - // arg ::= arg * flag - match self.token() { - Some(Token::Identifier(s)) => { - match s { - "uext" => arg.extension = types::ArgumentExtension::Uext, - "sext" => arg.extension = types::ArgumentExtension::Sext, - "inreg" => arg.inreg = true, - _ => break, - } - } + self.consume(); + + // arg ::= type * { flag } + while let Some(Token::Identifier(s)) = self.token() { + match s { + "uext" => arg.extension = types::ArgumentExtension::Uext, + "sext" => arg.extension = types::ArgumentExtension::Sext, + "inreg" => arg.inreg = true, _ => break, } + self.consume(); } + Ok(arg) } } From 65dfef16e9525f0f161f02adc2d6b2af7ef927fb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 28 Apr 2016 09:05:11 -0700 Subject: [PATCH 0055/3084] Begin parser unit tests, add public interface. The main entry point is Parser::parse(). --- cranelift/src/libctonfile/parser.rs | 64 +++++++++++++++++++++++++---- 1 file changed, 56 insertions(+), 8 deletions(-) diff --git a/cranelift/src/libctonfile/parser.rs b/cranelift/src/libctonfile/parser.rs index 060883b77e..5069520518 100644 --- a/cranelift/src/libctonfile/parser.rs +++ b/cranelift/src/libctonfile/parser.rs @@ -12,6 +12,7 @@ use cretonne::{types, repr}; pub use lexer::Location; /// A parse error is returned when the parse failed. +#[derive(Debug)] pub struct Error { pub location: Location, pub message: String, @@ -32,6 +33,21 @@ pub struct Parser<'a> { } impl<'a> Parser<'a> { + /// Create a new `Parser` which reads `text`. The referenced text must outlive the parser. + pub fn new(text: &'a str) -> Parser { + Parser { + lex: Lexer::new(text), + lex_error: None, + lookahead: None, + location: Location { line_number: 0 }, + } + } + + /// Parse the entire string into a list of functions. + pub fn parse(text: &'a str) -> Result> { + Self::new(text).parse_function_list() + } + // Consume the current lookahead token and return it. fn consume(&mut self) -> Token<'a> { self.lookahead.take().expect("No token to consume") @@ -62,7 +78,7 @@ impl<'a> Parser<'a> { message: // If we have a lexer error latched, report that. match self.lex_error { - Some(lexer::Error::InvalidChar) => "Invalid character".to_string(), + Some(lexer::Error::InvalidChar) => "invalid character".to_string(), None => message.to_string(), } } @@ -87,12 +103,23 @@ impl<'a> Parser<'a> { } } + /// Parse a list of function definitions. + /// + /// This is the top-level parse function matching the whole contents of a file. + pub fn parse_function_list(&mut self) -> Result> { + let mut list = Vec::new(); + while self.token().is_some() { + list.push(try!(self.parse_function())); + } + Ok(list) + } + // Parse a whole function definition. // // function ::= * "function" name signature { ... } // fn parse_function(&mut self) -> Result { - try!(self.match_token(Token::Function, "Expected 'function' keyword")); + try!(self.match_token(Token::Function, "expected 'function' keyword")); // function ::= "function" * name signature { ... } let name = try!(self.parse_function_name()); @@ -102,8 +129,8 @@ impl<'a> Parser<'a> { let mut func = repr::Function::new(); - try!(self.match_token(Token::LBrace, "Expected '{' before function body")); - try!(self.match_token(Token::RBrace, "Expected '}' after function body")); + try!(self.match_token(Token::LBrace, "expected '{' before function body")); + try!(self.match_token(Token::RBrace, "expected '}' after function body")); Ok(func) } @@ -118,7 +145,7 @@ impl<'a> Parser<'a> { self.consume(); Ok(s.to_string()) } - _ => Err(self.error("Expected function name")), + _ => Err(self.error("expected function name")), } } @@ -129,12 +156,12 @@ impl<'a> Parser<'a> { fn parse_signature(&mut self) -> Result { let mut sig = types::Signature::new(); - try!(self.match_token(Token::LPar, "Expected function signature: '(' args... ')'")); + try!(self.match_token(Token::LPar, "expected function signature: '(' args... ')'")); // signature ::= "(" * [arglist] ")" ["->" retlist] [call_conv] if self.token() != Some(Token::RPar) { sig.argument_types = try!(self.parse_argument_list()); } - try!(self.match_token(Token::RPar, "Expected ')' after function arguments")); + try!(self.match_token(Token::RPar, "expected ')' after function arguments")); if self.optional(Token::Arrow) { sig.return_types = try!(self.parse_argument_list()); } @@ -169,7 +196,7 @@ impl<'a> Parser<'a> { let mut arg = if let Some(Token::Type(t)) = self.token() { types::ArgumentType::new(t) } else { - return Err(self.error("Expected argument type")); + return Err(self.error("expected argument type")); }; self.consume(); @@ -187,3 +214,24 @@ impl<'a> Parser<'a> { Ok(arg) } } + +#[cfg(test)] +mod tests { + use super::*; + use cretonne::types::{self, ArgumentType, ArgumentExtension}; + + #[test] + fn argument_type() { + let mut p = Parser::new("i32 sext"); + let arg = p.parse_argument_type().unwrap(); + assert_eq!(arg, + ArgumentType { + value_type: types::I32, + extension: ArgumentExtension::Sext, + inreg: false, + }); + let Error { location, message } = p.parse_argument_type().unwrap_err(); + assert_eq!(location.line_number, 1); + assert_eq!(message, "expected argument type"); + } +} From 8067a1797794bfe5df733827d2e0460cb821e9ae Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 28 Apr 2016 09:09:34 -0700 Subject: [PATCH 0056/3084] Fix documentation badge. --- README.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.rst b/README.rst index 4e87f615e8..6fae53072f 100644 --- a/README.rst +++ b/README.rst @@ -5,8 +5,8 @@ Cretonne Code Generator Cretonne is a low-level retargetable code generator. It translates a target-independent intermediate language into executable machine code. -.. image:: https://readthedocs.io/projects/cretonne/badge/?version=latest - :target: http://cretonne.readthedocs.io/en/latest/?badge=latest +.. image:: https://readthedocs.org/projects/cretonne/badge/?version=latest + :target: https://cretonne.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status Cretonne is designed to be a code generator for WebAssembly with these design From 716b427cb1e3de53d78c0d75531ebe5b9fe0bf80 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 28 Apr 2016 12:57:40 -0700 Subject: [PATCH 0057/3084] Add top-level productions to language reference. --- cranelift/docs/langref.rst | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 2cad2999c6..0851ea8fcd 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -1,6 +1,6 @@ -**************************************** -Cretonne Intermediate Language Reference -**************************************** +*************************** +Cretonne Language Reference +*************************** .. default-domain:: cton .. highlight:: cton @@ -45,6 +45,15 @@ After the preample follows the :term:`function body` which consists of block`. Every EBB ends with a :term:`terminator instruction`, so execution can never fall through to the next EBB without an explicit branch. +A ``.cton`` file consists of a sequence of independent function definitions: + +.. productionlist:: + function-list : { function } + function : function-spec "{" preample function-body "}" + function-spec : "function" function-name signature + preamble : { preamble-decl } + function-body : { extended-basic-block } + Static single assignment form ----------------------------- From 90b3e16b560add11de74dce4f6ee42d2cb242823 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 28 Apr 2016 13:16:13 -0700 Subject: [PATCH 0058/3084] Tests for signature parser. --- cranelift/src/libcretonne/types.rs | 1 + cranelift/src/libctonfile/parser.rs | 32 ++++++++++++++++++++++++++++- 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/cranelift/src/libcretonne/types.rs b/cranelift/src/libcretonne/types.rs index 0aaa30841b..d54b3c51d0 100644 --- a/cranelift/src/libcretonne/types.rs +++ b/cranelift/src/libcretonne/types.rs @@ -224,6 +224,7 @@ pub struct ArgumentType { /// /// The function signature describes the types of arguments and return values along with other /// details that are needed to call a function correctly. +#[derive(Clone, PartialEq, Eq, Debug)] pub struct Signature { pub argument_types: Vec, pub return_types: Vec, diff --git a/cranelift/src/libctonfile/parser.rs b/cranelift/src/libctonfile/parser.rs index 5069520518..a56d69df00 100644 --- a/cranelift/src/libctonfile/parser.rs +++ b/cranelift/src/libctonfile/parser.rs @@ -6,6 +6,7 @@ // ====--------------------------------------------------------------------------------------====// use std::result; +use std::fmt::{self, Display, Formatter, Write}; use lexer::{self, Lexer, Token}; use cretonne::{types, repr}; @@ -18,6 +19,12 @@ pub struct Error { pub message: String, } +impl Display for Error { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + write!(f, "{}: {}", self.location.line_number, self.message) + } +} + pub type Result = result::Result; pub struct Parser<'a> { @@ -156,7 +163,7 @@ impl<'a> Parser<'a> { fn parse_signature(&mut self) -> Result { let mut sig = types::Signature::new(); - try!(self.match_token(Token::LPar, "expected function signature: '(' args... ')'")); + try!(self.match_token(Token::LPar, "expected function signature: ( args... )")); // signature ::= "(" * [arglist] ")" ["->" retlist] [call_conv] if self.token() != Some(Token::RPar) { sig.argument_types = try!(self.parse_argument_list()); @@ -234,4 +241,27 @@ mod tests { assert_eq!(location.line_number, 1); assert_eq!(message, "expected argument type"); } + + #[test] + fn signature() { + let sig = Parser::new("()").parse_signature().unwrap(); + assert_eq!(sig.argument_types.len(), 0); + assert_eq!(sig.return_types.len(), 0); + + let sig2 = Parser::new("(i8 inreg uext, f32, f64) -> i32 sext, f64") + .parse_signature() + .unwrap(); + assert_eq!(format!("{}", sig2), + "(i8 uext inreg, f32, f64) -> i32 sext, f64"); + + // `void` is not recognized as a type by the lexer. It should not appear in files. + assert_eq!(format!("{}", + Parser::new("() -> void").parse_signature().unwrap_err()), + "1: expected argument type"); + assert_eq!(format!("{}", Parser::new("i8 -> i8").parse_signature().unwrap_err()), + "1: expected function signature: ( args... )"); + assert_eq!(format!("{}", + Parser::new("(i8 -> i8").parse_signature().unwrap_err()), + "1: expected ')' after function arguments"); + } } From d1c79e891670ceb428373a6458de6ee6da1e7764 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 28 Apr 2016 14:02:16 -0700 Subject: [PATCH 0059/3084] Add FunctionName, Signature to repr::Function. Simplify the uses in parser.rs to avoid too many module qualifiers. --- cranelift/src/libcretonne/repr.rs | 19 ++++++++-- cranelift/src/libcretonne/types.rs | 6 ++++ cranelift/src/libctonfile/parser.rs | 54 ++++++++++++++++++----------- 3 files changed, 55 insertions(+), 24 deletions(-) diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index 1318a0936a..7d11b62bd1 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -1,7 +1,7 @@ //! Representation of Cretonne IL functions. -use types::Type; +use types::{Type, FunctionName, Signature}; use immediates::*; use std::fmt::{self, Display, Formatter, Write}; use std::u32; @@ -39,6 +39,12 @@ pub const NO_VALUE: Value = Value(u32::MAX); /// container for those objects by implementing both `Index` and `Index`. /// pub struct Function { + /// Name of this function. Mostly used by `.cton` files. + name: FunctionName, + + /// Signature of this function. + signature: Signature, + /// Data about all of the instructions in the function. The instructions in this vector is not /// necessarily in program order. The `Inst` reference indexes into this vector. instructions: Vec, @@ -269,9 +275,11 @@ impl InstructionData { } impl Function { - /// Create a new empty function. - pub fn new() -> Function { + /// Create a function with the given name and signature. + pub fn with_name_signature(name: FunctionName, sig: Signature) -> Function { Function { + name: name, + signature: sig, instructions: Vec::new(), extended_basic_blocks: Vec::new(), extended_values: Vec::new(), @@ -279,6 +287,11 @@ impl Function { } } + /// Create a new empty, anomymous function. + pub fn new() -> Function { + Self::with_name_signature(FunctionName::new(), Signature::new()) + } + /// Resolve an instruction reference. pub fn inst(&self, i: Inst) -> &InstructionData { &self.instructions[i.0 as usize] diff --git a/cranelift/src/libcretonne/types.rs b/cranelift/src/libcretonne/types.rs index d54b3c51d0..9d4f2d93af 100644 --- a/cranelift/src/libcretonne/types.rs +++ b/cranelift/src/libcretonne/types.rs @@ -194,6 +194,12 @@ impl Display for Type { // // ====--------------------------------------------------------------------------------------====// +/// The name of a function can be any UTF-8 string. +/// +/// Function names are mostly a testing and debugging tool. In partucular, `.cton` files use +/// function names to identify functions. +pub type FunctionName = String; + /// Function argument extension options. /// /// On some architectures, small integer function arguments are extended to the width of a diff --git a/cranelift/src/libctonfile/parser.rs b/cranelift/src/libctonfile/parser.rs index a56d69df00..22b9adcc93 100644 --- a/cranelift/src/libctonfile/parser.rs +++ b/cranelift/src/libctonfile/parser.rs @@ -8,7 +8,8 @@ use std::result; use std::fmt::{self, Display, Formatter, Write}; use lexer::{self, Lexer, Token}; -use cretonne::{types, repr}; +use cretonne::types::{FunctionName, Signature, ArgumentType, ArgumentExtension}; +use cretonne::repr::Function; pub use lexer::Location; @@ -51,7 +52,7 @@ impl<'a> Parser<'a> { } /// Parse the entire string into a list of functions. - pub fn parse(text: &'a str) -> Result> { + pub fn parse(text: &'a str) -> Result> { Self::new(text).parse_function_list() } @@ -113,7 +114,7 @@ impl<'a> Parser<'a> { /// Parse a list of function definitions. /// /// This is the top-level parse function matching the whole contents of a file. - pub fn parse_function_list(&mut self) -> Result> { + pub fn parse_function_list(&mut self) -> Result> { let mut list = Vec::new(); while self.token().is_some() { list.push(try!(self.parse_function())); @@ -123,25 +124,36 @@ impl<'a> Parser<'a> { // Parse a whole function definition. // - // function ::= * "function" name signature { ... } + // function ::= * function-spec "{" preample function-body "}" // - fn parse_function(&mut self) -> Result { - try!(self.match_token(Token::Function, "expected 'function' keyword")); - - // function ::= "function" * name signature { ... } - let name = try!(self.parse_function_name()); - - // function ::= "function" name * signature { ... } - let sig = try!(self.parse_signature()); - - let mut func = repr::Function::new(); + fn parse_function(&mut self) -> Result { + let (name, sig) = try!(self.parse_function_spec()); + let mut func = Function::with_name_signature(name, sig); + // function ::= function-spec * "{" preample function-body "}" try!(self.match_token(Token::LBrace, "expected '{' before function body")); + // function ::= function-spec "{" preample function-body * "}" try!(self.match_token(Token::RBrace, "expected '}' after function body")); Ok(func) } + // Parse a function spec. + // + // function-spec ::= * "function" name signature + // + fn parse_function_spec(&mut self) -> Result<(FunctionName, Signature)> { + try!(self.match_token(Token::Function, "expected 'function' keyword")); + + // function-spec ::= "function" * name signature + let name = try!(self.parse_function_name()); + + // function-spec ::= "function" name * signature + let sig = try!(self.parse_signature()); + + Ok((name, sig)) + } + // Parse a function name. // // function ::= "function" * name signature { ... } @@ -160,8 +172,8 @@ impl<'a> Parser<'a> { // // signature ::= * "(" [arglist] ")" ["->" retlist] [call_conv] // - fn parse_signature(&mut self) -> Result { - let mut sig = types::Signature::new(); + fn parse_signature(&mut self) -> Result { + let mut sig = Signature::new(); try!(self.match_token(Token::LPar, "expected function signature: ( args... )")); // signature ::= "(" * [arglist] ")" ["->" retlist] [call_conv] @@ -182,7 +194,7 @@ impl<'a> Parser<'a> { // // arglist ::= * arg { "," arg } // - fn parse_argument_list(&mut self) -> Result> { + fn parse_argument_list(&mut self) -> Result> { let mut list = Vec::new(); // arglist ::= * arg { "," arg } @@ -198,10 +210,10 @@ impl<'a> Parser<'a> { } // Parse a single argument type with flags. - fn parse_argument_type(&mut self) -> Result { + fn parse_argument_type(&mut self) -> Result { // arg ::= * type { flag } let mut arg = if let Some(Token::Type(t)) = self.token() { - types::ArgumentType::new(t) + ArgumentType::new(t) } else { return Err(self.error("expected argument type")); }; @@ -210,8 +222,8 @@ impl<'a> Parser<'a> { // arg ::= type * { flag } while let Some(Token::Identifier(s)) = self.token() { match s { - "uext" => arg.extension = types::ArgumentExtension::Uext, - "sext" => arg.extension = types::ArgumentExtension::Sext, + "uext" => arg.extension = ArgumentExtension::Uext, + "sext" => arg.extension = ArgumentExtension::Sext, "inreg" => arg.inreg = true, _ => break, } From ed6677d576a8ece71c785ff8f8a6f6b8323b6dc6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 28 Apr 2016 14:35:52 -0700 Subject: [PATCH 0060/3084] Parse stack slot decls. Add a stack slot array to repr::Function, use repr::StackSlot to reference them. Parse stack slot declarations in the function preamble, add them to the function. Add a new `Context` struct which keeps track of mappings between identifiers used in the file and real references. --- cranelift/src/libcretonne/immediates.rs | 4 + cranelift/src/libcretonne/repr.rs | 121 +++++++++++++++++++- cranelift/src/libctonfile/parser.rs | 141 +++++++++++++++++++++++- 3 files changed, 262 insertions(+), 4 deletions(-) diff --git a/cranelift/src/libcretonne/immediates.rs b/cranelift/src/libcretonne/immediates.rs index 4fefdbb9d0..072a8dcb91 100644 --- a/cranelift/src/libcretonne/immediates.rs +++ b/cranelift/src/libcretonne/immediates.rs @@ -74,6 +74,10 @@ impl Imm64 { pub fn from_bits(x: u64) -> Imm64 { Imm64(x as i64) } + + pub fn to_bits(&self) -> u64 { + self.0 as u64 + } } impl Display for Imm64 { diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index 7d11b62bd1..e982a65ddd 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -4,6 +4,7 @@ use types::{Type, FunctionName, Signature}; use immediates::*; use std::fmt::{self, Display, Formatter, Write}; +use std::ops::Index; use std::u32; // ====--------------------------------------------------------------------------------------====// @@ -33,18 +34,29 @@ pub struct Value(u32); /// A guaranteed invalid value reference. pub const NO_VALUE: Value = Value(u32::MAX); +/// An opaque reference to a stack slot. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct StackSlot(u32); + +/// A guaranteed invalid stack slot reference. +pub const NO_STACK_SLOT: StackSlot = StackSlot(u32::MAX); + /// A function. /// /// The `Function` struct owns all of its instructions and extended basic blocks, and it works as a /// container for those objects by implementing both `Index` and `Index`. /// +#[derive(Debug)] pub struct Function { /// Name of this function. Mostly used by `.cton` files. - name: FunctionName, + pub name: FunctionName, /// Signature of this function. signature: Signature, + /// Stack slots allocated in this function. + stack_slots: Vec, + /// Data about all of the instructions in the function. The instructions in this vector is not /// necessarily in program order. The `Inst` reference indexes into this vector. instructions: Vec, @@ -61,7 +73,15 @@ pub struct Function { pub return_types: Vec, } +/// Contents of a stack slot. +#[derive(Debug)] +pub struct StackSlotData { + /// Size of stack slot in bytes. + pub size: u32, +} + /// Contents of an extended basic block. +#[derive(Debug)] pub struct EbbData { /// Arguments for this extended basic block. These values dominate everything in the EBB. /// All branches to this EBB must provide matching arguments, and the arguments to the entry @@ -75,6 +95,7 @@ pub struct EbbData { /// value should have its `ty` field set to `VOID`. The size of `InstructionData` should be kept at /// 16 bytes on 64-bit architectures. If more space is needed to represent an instruction, use a /// `Box` to store the additional information out of line. +#[derive(Debug)] pub enum InstructionData { Nullary { opcode: Opcode, @@ -109,6 +130,7 @@ pub enum InstructionData { } /// Payload of a call instruction. +#[derive(Debug)] pub struct CallData { // Number of result values. results: u8, @@ -119,6 +141,72 @@ pub struct CallData { } +// ====--------------------------------------------------------------------------------------====// +// +// Stack slot implementation. +// +// ====--------------------------------------------------------------------------------------====// + +impl StackSlot { + fn new(index: usize) -> StackSlot { + assert!(index < (u32::MAX as usize)); + StackSlot(index as u32) + } + + pub fn index(&self) -> usize { + self.0 as usize + } +} + +/// Display a `StackSlot` reference as "ss12". +impl Display for StackSlot { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write!(fmt, "ss{}", self.0) + } +} + +impl StackSlotData { + /// Create a stack slot with the specified byte size. + pub fn new(size: u32) -> StackSlotData { + StackSlotData { size: size } + } +} + +impl Display for StackSlotData { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write!(fmt, "stack_slot {}", self.size) + } +} + +/// Allow immutable access to stack slots via function indexing. +impl Index for Function { + type Output = StackSlotData; + + fn index<'a>(&'a self, ss: StackSlot) -> &'a StackSlotData { + &self.stack_slots[ss.index()] + } +} + +/// Stack slot iterator visits all stack slots in a function, returning `StackSlot` references. +pub struct StackSlotIter { + cur: usize, + end: usize, +} + +impl Iterator for StackSlotIter { + type Item = StackSlot; + + fn next(&mut self) -> Option { + if self.cur < self.end { + let ss = StackSlot::new(self.cur); + self.cur += 1; + Some(ss) + } else { + None + } + } +} + // ====--------------------------------------------------------------------------------------====// // // Extended basic block implementation. @@ -227,6 +315,7 @@ impl Display for Value { // Most values are simply the first value produced by an instruction. // Other values have an entry in the value table. +#[derive(Debug)] enum ValueData { // An unused entry in the value table. No instruction should be defining or using this value. Unused, @@ -280,6 +369,7 @@ impl Function { Function { name: name, signature: sig, + stack_slots: Vec::new(), instructions: Vec::new(), extended_basic_blocks: Vec::new(), extended_values: Vec::new(), @@ -292,6 +382,21 @@ impl Function { Self::with_name_signature(FunctionName::new(), Signature::new()) } + /// Allocate a new stack slot. + pub fn make_stack_slot(&mut self, data: StackSlotData) -> StackSlot { + let ss = StackSlot::new(self.stack_slots.len()); + self.stack_slots.push(data); + ss + } + + /// Iterate over all stack slots in function. + pub fn stack_slot_iter(&self) -> StackSlotIter { + StackSlotIter { + cur: 0, + end: self.stack_slots.len(), + } + } + /// Resolve an instruction reference. pub fn inst(&self, i: Inst) -> &InstructionData { &self.instructions[i.0 as usize] @@ -351,4 +456,18 @@ mod tests { assert_eq!(ins.opcode(), Opcode::Iconst); assert_eq!(ins.first_type(), types::I32); } + + #[test] + fn stack_slot() { + let mut func = Function::new(); + + let ss0 = func.make_stack_slot(StackSlotData::new(4)); + let ss1 = func.make_stack_slot(StackSlotData::new(8)); + assert_eq!(format!("{}", ss0), "ss0"); + assert_eq!(format!("{}", ss1), "ss1"); + + assert_eq!(func[ss0].size, 4); + assert_eq!(func[ss1].size, 8); + } + } diff --git a/cranelift/src/libctonfile/parser.rs b/cranelift/src/libctonfile/parser.rs index 22b9adcc93..a014ca0253 100644 --- a/cranelift/src/libctonfile/parser.rs +++ b/cranelift/src/libctonfile/parser.rs @@ -5,11 +5,14 @@ // // ====--------------------------------------------------------------------------------------====// +use std::collections::HashMap; use std::result; use std::fmt::{self, Display, Formatter, Write}; +use std::u32; use lexer::{self, Lexer, Token}; use cretonne::types::{FunctionName, Signature, ArgumentType, ArgumentExtension}; -use cretonne::repr::Function; +use cretonne::immediates::Imm64; +use cretonne::repr::{Function, StackSlot, StackSlotData}; pub use lexer::Location; @@ -40,6 +43,35 @@ pub struct Parser<'a> { location: Location, } +// Context for resolving references when parsing a single function. +// +// Many entities like values, stack slots, and function signatures are referenced in the `.cton` +// file by number. We need to map these numbers to real references. +struct Context { + function: Function, + stack_slots: HashMap, +} + +impl Context { + fn new(f: Function) -> Context { + Context { + function: f, + stack_slots: HashMap::new(), + } + } + + fn add(&mut self, number: u32, data: StackSlotData, loc: &Location) -> Result<()> { + if self.stack_slots.insert(number, self.function.make_stack_slot(data)).is_some() { + Err(Error { + location: loc.clone(), + message: format!("duplicate stack slot: ss{}", number), + }) + } else { + Ok(()) + } + } +} + impl<'a> Parser<'a> { /// Create a new `Parser` which reads `text`. The referenced text must outlive the parser. pub fn new(text: &'a str) -> Parser { @@ -111,6 +143,38 @@ impl<'a> Parser<'a> { } } + // Match and consume a specific identifier string. + // Used for pseudo-keywords like "stack_slot" that only appear in certain contexts. + fn match_identifier(&mut self, want: &'static str, err_msg: &str) -> Result> { + if self.token() == Some(Token::Identifier(want)) { + Ok(self.consume()) + } else { + Err(self.error(err_msg)) + } + } + + // Match and consume a stack slot reference. + fn match_ss(&mut self, err_msg: &str) -> Result { + if let Some(Token::StackSlot(ss)) = self.token() { + self.consume(); + Ok(ss) + } else { + Err(self.error(err_msg)) + } + } + + // Match and consume an Imm64 immediate. + fn match_imm64(&mut self, err_msg: &str) -> Result { + if let Some(Token::Integer(text)) = self.token() { + self.consume(); + // Lexer just gives us raw text that looks like an integer. + // Parse it as an Imm64 to check for overflow and other issues. + text.parse().map_err(|e| self.error(e)) + } else { + Err(self.error(err_msg)) + } + } + /// Parse a list of function definitions. /// /// This is the top-level parse function matching the whole contents of a file. @@ -128,14 +192,16 @@ impl<'a> Parser<'a> { // fn parse_function(&mut self) -> Result { let (name, sig) = try!(self.parse_function_spec()); - let mut func = Function::with_name_signature(name, sig); + let mut ctx = Context::new(Function::with_name_signature(name, sig)); // function ::= function-spec * "{" preample function-body "}" try!(self.match_token(Token::LBrace, "expected '{' before function body")); + // function ::= function-spec "{" * preample function-body "}" + try!(self.parse_preamble(&mut ctx)); // function ::= function-spec "{" preample function-body * "}" try!(self.match_token(Token::RBrace, "expected '}' after function body")); - Ok(func) + Ok(ctx.function) } // Parse a function spec. @@ -232,6 +298,46 @@ impl<'a> Parser<'a> { Ok(arg) } + + // Parse the function preamble. + // + // preamble ::= * { preamble-decl } + // preamble-decl ::= * stack-slot-decl + // * function-decl + // * signature-decl + // + // The parsed decls are added to `ctx` rather than returned. + fn parse_preamble(&mut self, ctx: &mut Context) -> Result<()> { + loop { + try!(match self.token() { + Some(Token::StackSlot(..)) => { + self.parse_stack_slot_decl() + .and_then(|(num, dat)| ctx.add(num, dat, &self.location)) + } + // More to come.. + _ => return Ok(()), + }); + } + } + + // Parse a stack slot decl, add to `func`. + // + // stack-slot-decl ::= * StackSlot(ss) "=" "stack_slot" Bytes {"," stack-slot-flag} + fn parse_stack_slot_decl(&mut self) -> Result<(u32, StackSlotData)> { + let number = try!(self.match_ss("expected stack slot number: ss«n»")); + try!(self.match_token(Token::Equal, "expected '=' in stack_slot decl")); + try!(self.match_identifier("stack_slot", "expected 'stack_slot'")); + + // stack-slot-decl ::= StackSlot(ss) "=" "stack_slot" * Bytes {"," stack-slot-flag} + let bytes = try!(self.match_imm64("expected byte-size in stack_slot decl")).to_bits(); + if bytes > u32::MAX as u64 { + return Err(self.error("stack slot too large")); + } + let data = StackSlotData::new(bytes as u32); + + // TBD: stack-slot-decl ::= StackSlot(ss) "=" "stack_slot" Bytes * {"," stack-slot-flag} + Ok((number, data)) + } } #[cfg(test)] @@ -276,4 +382,33 @@ mod tests { Parser::new("(i8 -> i8").parse_signature().unwrap_err()), "1: expected ')' after function arguments"); } + + #[test] + fn stack_slot_decl() { + let func = Parser::new("function foo() { + ss3 = stack_slot 13 + ss1 = stack_slot 1 + }") + .parse_function() + .unwrap(); + assert_eq!(func.name, "foo"); + let mut iter = func.stack_slot_iter(); + let ss0 = iter.next().unwrap(); + assert_eq!(format!("{}", ss0), "ss0"); + assert_eq!(func[ss0].size, 13); + let ss1 = iter.next().unwrap(); + assert_eq!(format!("{}", ss1), "ss1"); + assert_eq!(func[ss1].size, 1); + assert_eq!(iter.next(), None); + + // Catch suplicate definitions. + assert_eq!(format!("{}", + Parser::new("function bar() { + ss1 = stack_slot 13 + ss1 = stack_slot 1 + }") + .parse_function() + .unwrap_err()), + "3: duplicate stack slot: ss1"); + } } From 8f65207ed1d242bd3e6bf7b6032b0c2cbefe158c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Apr 2016 11:55:40 -0700 Subject: [PATCH 0061/3084] Use x.to_string() instead of format!("{}", x). Both use the Display trait. --- cranelift/src/libcretonne/immediates.rs | 100 ++++++++++++------------ cranelift/src/libcretonne/repr.rs | 6 +- cranelift/src/libcretonne/types.rs | 59 +++++++------- cranelift/src/libctonfile/parser.rs | 26 +++--- cranelift/src/test-all.sh | 4 + 5 files changed, 96 insertions(+), 99 deletions(-) create mode 100755 cranelift/src/test-all.sh diff --git a/cranelift/src/libcretonne/immediates.rs b/cranelift/src/libcretonne/immediates.rs index 072a8dcb91..619d790bfe 100644 --- a/cranelift/src/libcretonne/immediates.rs +++ b/cranelift/src/libcretonne/immediates.rs @@ -493,7 +493,7 @@ mod tests { assert_eq!(x, y); assert_eq!(format!("{:?}", Opcode::IaddImm), "IaddImm"); - assert_eq!(format!("{}", Opcode::IaddImm), "iadd_imm"); + assert_eq!(Opcode::IaddImm.to_string(), "iadd_imm"); // Check the matcher. assert_eq!("iadd".parse::(), Ok(Opcode::Iadd)); @@ -505,13 +505,13 @@ mod tests { #[test] fn format_imm64() { - assert_eq!(format!("{}", Imm64(0)), "0"); - assert_eq!(format!("{}", Imm64(9999)), "9999"); - assert_eq!(format!("{}", Imm64(10000)), "0x2710"); - assert_eq!(format!("{}", Imm64(-9999)), "-9999"); - assert_eq!(format!("{}", Imm64(-10000)), "0xffff_ffff_ffff_d8f0"); - assert_eq!(format!("{}", Imm64(0xffff)), "0xffff"); - assert_eq!(format!("{}", Imm64(0x10000)), "0x0001_0000"); + assert_eq!(Imm64(0).to_string(), "0"); + assert_eq!(Imm64(9999).to_string(), "9999"); + assert_eq!(Imm64(10000).to_string(), "0x2710"); + assert_eq!(Imm64(-9999).to_string(), "-9999"); + assert_eq!(Imm64(-10000).to_string(), "0xffff_ffff_ffff_d8f0"); + assert_eq!(Imm64(0xffff).to_string(), "0xffff"); + assert_eq!(Imm64(0x10000).to_string(), "0x0001_0000"); } // Verify that `text` can be parsed as a `T` into a value that displays as `want`. @@ -520,7 +520,7 @@ mod tests { { match text.parse::() { Err(s) => panic!("\"{}\".parse() error: {}", text, s), - Ok(x) => assert_eq!(format!("{}", x), want), + Ok(x) => assert_eq!(x.to_string(), want), } } @@ -529,7 +529,7 @@ mod tests { where ::Err: Display { match text.parse::() { - Err(s) => assert_eq!(format!("{}", s), msg), + Err(s) => assert_eq!(s.to_string(), msg), Ok(x) => panic!("Wanted Err({}), but got {}", msg, x), } } @@ -584,33 +584,32 @@ mod tests { #[test] fn format_ieee32() { - assert_eq!(format!("{}", Ieee32::new(0.0)), "0.0"); - assert_eq!(format!("{}", Ieee32::new(-0.0)), "-0.0"); - assert_eq!(format!("{}", Ieee32::new(1.0)), "0x1.000000p0"); - assert_eq!(format!("{}", Ieee32::new(1.5)), "0x1.800000p0"); - assert_eq!(format!("{}", Ieee32::new(0.5)), "0x1.000000p-1"); - assert_eq!(format!("{}", Ieee32::new(f32::EPSILON)), "0x1.000000p-23"); - assert_eq!(format!("{}", Ieee32::new(f32::MIN)), "-0x1.fffffep127"); - assert_eq!(format!("{}", Ieee32::new(f32::MAX)), "0x1.fffffep127"); + assert_eq!(Ieee32::new(0.0).to_string(), "0.0"); + assert_eq!(Ieee32::new(-0.0).to_string(), "-0.0"); + assert_eq!(Ieee32::new(1.0).to_string(), "0x1.000000p0"); + assert_eq!(Ieee32::new(1.5).to_string(), "0x1.800000p0"); + assert_eq!(Ieee32::new(0.5).to_string(), "0x1.000000p-1"); + assert_eq!(Ieee32::new(f32::EPSILON).to_string(), "0x1.000000p-23"); + assert_eq!(Ieee32::new(f32::MIN).to_string(), "-0x1.fffffep127"); + assert_eq!(Ieee32::new(f32::MAX).to_string(), "0x1.fffffep127"); // Smallest positive normal number. - assert_eq!(format!("{}", Ieee32::new(f32::MIN_POSITIVE)), + assert_eq!(Ieee32::new(f32::MIN_POSITIVE).to_string(), "0x1.000000p-126"); // Subnormals. - assert_eq!(format!("{}", Ieee32::new(f32::MIN_POSITIVE / 2.0)), + assert_eq!(Ieee32::new(f32::MIN_POSITIVE / 2.0).to_string(), "0x0.800000p-126"); - assert_eq!(format!("{}", Ieee32::new(f32::MIN_POSITIVE * f32::EPSILON)), + assert_eq!(Ieee32::new(f32::MIN_POSITIVE * f32::EPSILON).to_string(), "0x0.000002p-126"); - assert_eq!(format!("{}", Ieee32::new(f32::INFINITY)), "Inf"); - assert_eq!(format!("{}", Ieee32::new(f32::NEG_INFINITY)), "-Inf"); - assert_eq!(format!("{}", Ieee32::new(f32::NAN)), "NaN"); - assert_eq!(format!("{}", Ieee32::new(-f32::NAN)), "-NaN"); + assert_eq!(Ieee32::new(f32::INFINITY).to_string(), "Inf"); + assert_eq!(Ieee32::new(f32::NEG_INFINITY).to_string(), "-Inf"); + assert_eq!(Ieee32::new(f32::NAN).to_string(), "NaN"); + assert_eq!(Ieee32::new(-f32::NAN).to_string(), "-NaN"); // Construct some qNaNs with payloads. - assert_eq!(format!("{}", Ieee32::from_bits(0x7fc00001)), "NaN:0x1"); - assert_eq!(format!("{}", Ieee32::from_bits(0x7ff00001)), "NaN:0x300001"); + assert_eq!(Ieee32::from_bits(0x7fc00001).to_string(), "NaN:0x1"); + assert_eq!(Ieee32::from_bits(0x7ff00001).to_string(), "NaN:0x300001"); // Signaling NaNs. - assert_eq!(format!("{}", Ieee32::from_bits(0x7f800001)), "sNaN:0x1"); - assert_eq!(format!("{}", Ieee32::from_bits(0x7fa00001)), - "sNaN:0x200001"); + assert_eq!(Ieee32::from_bits(0x7f800001).to_string(), "sNaN:0x1"); + assert_eq!(Ieee32::from_bits(0x7fa00001).to_string(), "sNaN:0x200001"); } #[test] @@ -682,38 +681,35 @@ mod tests { #[test] fn format_ieee64() { - assert_eq!(format!("{}", Ieee64::new(0.0)), "0.0"); - assert_eq!(format!("{}", Ieee64::new(-0.0)), "-0.0"); - assert_eq!(format!("{}", Ieee64::new(1.0)), "0x1.0000000000000p0"); - assert_eq!(format!("{}", Ieee64::new(1.5)), "0x1.8000000000000p0"); - assert_eq!(format!("{}", Ieee64::new(0.5)), "0x1.0000000000000p-1"); - assert_eq!(format!("{}", Ieee64::new(f64::EPSILON)), + assert_eq!(Ieee64::new(0.0).to_string(), "0.0"); + assert_eq!(Ieee64::new(-0.0).to_string(), "-0.0"); + assert_eq!(Ieee64::new(1.0).to_string(), "0x1.0000000000000p0"); + assert_eq!(Ieee64::new(1.5).to_string(), "0x1.8000000000000p0"); + assert_eq!(Ieee64::new(0.5).to_string(), "0x1.0000000000000p-1"); + assert_eq!(Ieee64::new(f64::EPSILON).to_string(), "0x1.0000000000000p-52"); - assert_eq!(format!("{}", Ieee64::new(f64::MIN)), - "-0x1.fffffffffffffp1023"); - assert_eq!(format!("{}", Ieee64::new(f64::MAX)), - "0x1.fffffffffffffp1023"); + assert_eq!(Ieee64::new(f64::MIN).to_string(), "-0x1.fffffffffffffp1023"); + assert_eq!(Ieee64::new(f64::MAX).to_string(), "0x1.fffffffffffffp1023"); // Smallest positive normal number. - assert_eq!(format!("{}", Ieee64::new(f64::MIN_POSITIVE)), + assert_eq!(Ieee64::new(f64::MIN_POSITIVE).to_string(), "0x1.0000000000000p-1022"); // Subnormals. - assert_eq!(format!("{}", Ieee64::new(f64::MIN_POSITIVE / 2.0)), + assert_eq!(Ieee64::new(f64::MIN_POSITIVE / 2.0).to_string(), "0x0.8000000000000p-1022"); - assert_eq!(format!("{}", Ieee64::new(f64::MIN_POSITIVE * f64::EPSILON)), + assert_eq!(Ieee64::new(f64::MIN_POSITIVE * f64::EPSILON).to_string(), "0x0.0000000000001p-1022"); - assert_eq!(format!("{}", Ieee64::new(f64::INFINITY)), "Inf"); - assert_eq!(format!("{}", Ieee64::new(f64::NEG_INFINITY)), "-Inf"); - assert_eq!(format!("{}", Ieee64::new(f64::NAN)), "NaN"); - assert_eq!(format!("{}", Ieee64::new(-f64::NAN)), "-NaN"); + assert_eq!(Ieee64::new(f64::INFINITY).to_string(), "Inf"); + assert_eq!(Ieee64::new(f64::NEG_INFINITY).to_string(), "-Inf"); + assert_eq!(Ieee64::new(f64::NAN).to_string(), "NaN"); + assert_eq!(Ieee64::new(-f64::NAN).to_string(), "-NaN"); // Construct some qNaNs with payloads. - assert_eq!(format!("{}", Ieee64::from_bits(0x7ff8000000000001)), - "NaN:0x1"); - assert_eq!(format!("{}", Ieee64::from_bits(0x7ffc000000000001)), + assert_eq!(Ieee64::from_bits(0x7ff8000000000001).to_string(), "NaN:0x1"); + assert_eq!(Ieee64::from_bits(0x7ffc000000000001).to_string(), "NaN:0x4000000000001"); // Signaling NaNs. - assert_eq!(format!("{}", Ieee64::from_bits(0x7ff0000000000001)), + assert_eq!(Ieee64::from_bits(0x7ff0000000000001).to_string(), "sNaN:0x1"); - assert_eq!(format!("{}", Ieee64::from_bits(0x7ff4000000000001)), + assert_eq!(Ieee64::from_bits(0x7ff4000000000001).to_string(), "sNaN:0x4000000000001"); } diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index e982a65ddd..702a371609 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -449,7 +449,7 @@ mod tests { ty: types::I32, }; let inst = func.make_inst(idata); - assert_eq!(format!("{}", inst), "inst0"); + assert_eq!(inst.to_string(), "inst0"); // Immutable reference resolution. let ins = func.inst(inst); @@ -463,8 +463,8 @@ mod tests { let ss0 = func.make_stack_slot(StackSlotData::new(4)); let ss1 = func.make_stack_slot(StackSlotData::new(8)); - assert_eq!(format!("{}", ss0), "ss0"); - assert_eq!(format!("{}", ss1), "ss1"); + assert_eq!(ss0.to_string(), "ss0"); + assert_eq!(ss1.to_string(), "ss1"); assert_eq!(func[ss0].size, 4); assert_eq!(func[ss1].size, 8); diff --git a/cranelift/src/libcretonne/types.rs b/cranelift/src/libcretonne/types.rs index 9d4f2d93af..967879c14d 100644 --- a/cranelift/src/libcretonne/types.rs +++ b/cranelift/src/libcretonne/types.rs @@ -337,38 +337,37 @@ mod tests { assert_eq!(big.lane_count(), 256); assert_eq!(big.bits(), 64 * 256); - assert_eq!(format!("{}", big.half_vector().unwrap()), "f64x128"); - assert_eq!(format!("{}", B1.by(2).unwrap().half_vector().unwrap()), - "b1"); + assert_eq!(big.half_vector().unwrap().to_string(), "f64x128"); + assert_eq!(B1.by(2).unwrap().half_vector().unwrap().to_string(), "b1"); assert_eq!(I32.half_vector(), None); assert_eq!(VOID.half_vector(), None); } #[test] fn format_scalars() { - assert_eq!(format!("{}", VOID), "void"); - assert_eq!(format!("{}", B1), "b1"); - assert_eq!(format!("{}", B8), "b8"); - assert_eq!(format!("{}", B16), "b16"); - assert_eq!(format!("{}", B32), "b32"); - assert_eq!(format!("{}", B64), "b64"); - assert_eq!(format!("{}", I8), "i8"); - assert_eq!(format!("{}", I16), "i16"); - assert_eq!(format!("{}", I32), "i32"); - assert_eq!(format!("{}", I64), "i64"); - assert_eq!(format!("{}", F32), "f32"); - assert_eq!(format!("{}", F64), "f64"); + assert_eq!(VOID.to_string(), "void"); + assert_eq!(B1.to_string(), "b1"); + assert_eq!(B8.to_string(), "b8"); + assert_eq!(B16.to_string(), "b16"); + assert_eq!(B32.to_string(), "b32"); + assert_eq!(B64.to_string(), "b64"); + assert_eq!(I8.to_string(), "i8"); + assert_eq!(I16.to_string(), "i16"); + assert_eq!(I32.to_string(), "i32"); + assert_eq!(I64.to_string(), "i64"); + assert_eq!(F32.to_string(), "f32"); + assert_eq!(F64.to_string(), "f64"); } #[test] fn format_vectors() { - assert_eq!(format!("{}", B1.by(8).unwrap()), "b1x8"); - assert_eq!(format!("{}", B8.by(1).unwrap()), "b8"); - assert_eq!(format!("{}", B16.by(256).unwrap()), "b16x256"); - assert_eq!(format!("{}", B32.by(4).unwrap().by(2).unwrap()), "b32x8"); - assert_eq!(format!("{}", B64.by(8).unwrap()), "b64x8"); - assert_eq!(format!("{}", I8.by(64).unwrap()), "i8x64"); - assert_eq!(format!("{}", F64.by(2).unwrap()), "f64x2"); + assert_eq!(B1.by(8).unwrap().to_string(), "b1x8"); + assert_eq!(B8.by(1).unwrap().to_string(), "b8"); + assert_eq!(B16.by(256).unwrap().to_string(), "b16x256"); + assert_eq!(B32.by(4).unwrap().by(2).unwrap().to_string(), "b32x8"); + assert_eq!(B64.by(8).unwrap().to_string(), "b64x8"); + assert_eq!(I8.by(64).unwrap().to_string(), "i8x64"); + assert_eq!(F64.by(2).unwrap().to_string(), "f64x2"); assert_eq!(I8.by(3), None); assert_eq!(I8.by(512), None); assert_eq!(VOID.by(4), None); @@ -377,24 +376,24 @@ mod tests { #[test] fn argument_type() { let mut t = ArgumentType::new(I32); - assert_eq!(format!("{}", t), "i32"); + assert_eq!(t.to_string(), "i32"); t.extension = ArgumentExtension::Uext; - assert_eq!(format!("{}", t), "i32 uext"); + assert_eq!(t.to_string(), "i32 uext"); t.inreg = true; - assert_eq!(format!("{}", t), "i32 uext inreg"); + assert_eq!(t.to_string(), "i32 uext inreg"); } #[test] fn signatures() { let mut sig = Signature::new(); - assert_eq!(format!("{}", sig), "()"); + assert_eq!(sig.to_string(), "()"); sig.argument_types.push(ArgumentType::new(I32)); - assert_eq!(format!("{}", sig), "(i32)"); + assert_eq!(sig.to_string(), "(i32)"); sig.return_types.push(ArgumentType::new(F32)); - assert_eq!(format!("{}", sig), "(i32) -> f32"); + assert_eq!(sig.to_string(), "(i32) -> f32"); sig.argument_types.push(ArgumentType::new(I32.by(4).unwrap())); - assert_eq!(format!("{}", sig), "(i32, i32x4) -> f32"); + assert_eq!(sig.to_string(), "(i32, i32x4) -> f32"); sig.return_types.push(ArgumentType::new(B8)); - assert_eq!(format!("{}", sig), "(i32, i32x4) -> f32, b8"); + assert_eq!(sig.to_string(), "(i32, i32x4) -> f32, b8"); } } diff --git a/cranelift/src/libctonfile/parser.rs b/cranelift/src/libctonfile/parser.rs index a014ca0253..e74ae22c38 100644 --- a/cranelift/src/libctonfile/parser.rs +++ b/cranelift/src/libctonfile/parser.rs @@ -369,17 +369,15 @@ mod tests { let sig2 = Parser::new("(i8 inreg uext, f32, f64) -> i32 sext, f64") .parse_signature() .unwrap(); - assert_eq!(format!("{}", sig2), + assert_eq!(sig2.to_string(), "(i8 uext inreg, f32, f64) -> i32 sext, f64"); // `void` is not recognized as a type by the lexer. It should not appear in files. - assert_eq!(format!("{}", - Parser::new("() -> void").parse_signature().unwrap_err()), + assert_eq!(Parser::new("() -> void").parse_signature().unwrap_err().to_string(), "1: expected argument type"); - assert_eq!(format!("{}", Parser::new("i8 -> i8").parse_signature().unwrap_err()), + assert_eq!(Parser::new("i8 -> i8").parse_signature().unwrap_err().to_string(), "1: expected function signature: ( args... )"); - assert_eq!(format!("{}", - Parser::new("(i8 -> i8").parse_signature().unwrap_err()), + assert_eq!(Parser::new("(i8 -> i8").parse_signature().unwrap_err().to_string(), "1: expected ')' after function arguments"); } @@ -394,21 +392,21 @@ mod tests { assert_eq!(func.name, "foo"); let mut iter = func.stack_slot_iter(); let ss0 = iter.next().unwrap(); - assert_eq!(format!("{}", ss0), "ss0"); + assert_eq!(ss0.to_string(), "ss0"); assert_eq!(func[ss0].size, 13); let ss1 = iter.next().unwrap(); - assert_eq!(format!("{}", ss1), "ss1"); + assert_eq!(ss1.to_string(), "ss1"); assert_eq!(func[ss1].size, 1); assert_eq!(iter.next(), None); // Catch suplicate definitions. - assert_eq!(format!("{}", - Parser::new("function bar() { - ss1 = stack_slot 13 - ss1 = stack_slot 1 + assert_eq!(Parser::new("function bar() { + ss1 = stack_slot 13 + ss1 = stack_slot 1 }") - .parse_function() - .unwrap_err()), + .parse_function() + .unwrap_err() + .to_string(), "3: duplicate stack slot: ss1"); } } diff --git a/cranelift/src/test-all.sh b/cranelift/src/test-all.sh new file mode 100755 index 0000000000..6838b1544f --- /dev/null +++ b/cranelift/src/test-all.sh @@ -0,0 +1,4 @@ +#!/bin/bash +cd $(dirname "$0")/tools +cargo test -p cretonne -p ctonfile -p cretonne-tools +cargo doc -p cretonne -p ctonfile -p cretonne-tools From b390b3113a7494060ec3ee7ec646c38c0d802fa6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Apr 2016 13:30:07 -0700 Subject: [PATCH 0062/3084] Add a write.rs module. Convert a function to text. --- cranelift/src/libcretonne/lib.rs | 1 + cranelift/src/libcretonne/repr.rs | 5 ++ cranelift/src/libcretonne/write.rs | 113 +++++++++++++++++++++++++++++ 3 files changed, 119 insertions(+) create mode 100644 cranelift/src/libcretonne/write.rs diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index b67ca5df10..f47b049156 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -8,3 +8,4 @@ pub mod types; pub mod immediates; pub mod repr; +pub mod write; diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index 702a371609..3917fa8f48 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -382,6 +382,11 @@ impl Function { Self::with_name_signature(FunctionName::new(), Signature::new()) } + /// Get the signature of this function. + pub fn own_signature(&self) -> &Signature { + &self.signature + } + /// Allocate a new stack slot. pub fn make_stack_slot(&mut self, data: StackSlotData) -> StackSlot { let ss = StackSlot::new(self.stack_slots.len()); diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs new file mode 100644 index 0000000000..7281294fe6 --- /dev/null +++ b/cranelift/src/libcretonne/write.rs @@ -0,0 +1,113 @@ +//! Converting Cretonne IL to text. +//! +//! The `write` module provides the `write_function` function which converts an IL `Function` to an +//! equivalent textual representation. This textual representation can be read back by the +//! `ctonfile` crate. + +use std::io::{self, Write}; +use repr::Function; + +pub type Result = io::Result<()>; + +/// Write `func` to `w` as equivalent text. +pub fn write_function(w: &mut Write, func: &Function) -> Result { + try!(write_spec(w, func)); + try!(writeln!(w, " {{")); + try!(write_preamble(w, func)); + writeln!(w, "}}") +} + +/// Convert `func` to a string. +pub fn function_to_string(func: &Function) -> String { + let mut buffer: Vec = Vec::new(); + // Any errors here would be out-of-memory, which should not happen with normal functions. + write_function(&mut buffer, func).unwrap(); + // A UTF-8 conversion error is a real bug. + String::from_utf8(buffer).unwrap() +} + +// ====--------------------------------------------------------------------------------------====// +// +// Function spec. +// +// ====--------------------------------------------------------------------------------------====// + +// The function name may need quotes if it doesn't parse as an identifier. +fn needs_quotes(name: &str) -> bool { + let mut iter = name.chars(); + if let Some(ch) = iter.next() { + !ch.is_alphabetic() || !iter.all(char::is_alphanumeric) + } else { + // A blank function name needs quotes. + true + } +} + +// Use Rust's escape_default which provides a few simple \t \r \n \' \" \\ escapes and uses +// \u{xxxx} for anything else outside the ASCII printable range. +fn escaped(name: &str) -> String { + name.chars().flat_map(char::escape_default).collect() +} + +fn write_spec(w: &mut Write, func: &Function) -> Result { + let sig = func.own_signature(); + if !needs_quotes(&func.name) { + write!(w, "function {}{}", func.name, sig) + } else { + write!(w, "function \"{}\" {}", escaped(&func.name), sig) + } +} + +fn write_preamble(w: &mut Write, func: &Function) -> Result { + let mut any = false; + + for ss in func.stack_slot_iter() { + any = true; + try!(writeln!(w, " {} = {}", ss, func[ss])); + } + + // Put a blank line after the preamble unless it was empty. + if any { + writeln!(w, "") + } else { + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use super::{needs_quotes, escaped}; + use repr::{Function, StackSlotData}; + + #[test] + fn quoting() { + assert_eq!(needs_quotes(""), true); + assert_eq!(needs_quotes("x"), false); + assert_eq!(needs_quotes(" "), true); + assert_eq!(needs_quotes("0"), true); + assert_eq!(needs_quotes("x0"), false); + } + + #[test] + fn escaping() { + assert_eq!(escaped(""), ""); + assert_eq!(escaped("x"), "x"); + assert_eq!(escaped(" "), " "); + assert_eq!(escaped(" \n"), " \\n"); + assert_eq!(escaped("a\u{1000}v"), "a\\u{1000}v"); + } + + #[test] + fn basic() { + let mut f = Function::new(); + assert_eq!(function_to_string(&f), "function \"\" () {\n}\n"); + + f.name.push_str("foo"); + assert_eq!(function_to_string(&f), "function foo() {\n}\n"); + + f.make_stack_slot(StackSlotData::new(4)); + assert_eq!(function_to_string(&f), + "function foo() {\n ss0 = stack_slot 4\n\n}\n"); + } +} From 01ed9fc6c8bef152da5b85d16f6d9d4c75a0aacf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Apr 2016 14:32:10 -0700 Subject: [PATCH 0063/3084] Rename libraries libctonfile -> libreader. This library will only provide .cton file reading/parsing services which are not needed after deployment. Code for writing .cton files lives in the main cretonne library because it is fairly small, and because it is useful for extracting test cases from a deployed library. --- cranelift/src/libcretonne/write.rs | 2 +- cranelift/src/{libctonfile => libreader}/Cargo.toml | 4 ++-- cranelift/src/{libctonfile => libreader}/lexer.rs | 0 cranelift/src/{libctonfile => libreader}/lib.rs | 4 ++-- cranelift/src/{libctonfile => libreader}/parser.rs | 0 cranelift/src/test-all.sh | 4 ++-- cranelift/src/tools/Cargo.lock | 4 ++-- cranelift/src/tools/Cargo.toml | 2 +- cranelift/src/tools/main.rs | 2 +- 9 files changed, 11 insertions(+), 11 deletions(-) rename cranelift/src/{libctonfile => libreader}/Cargo.toml (78%) rename cranelift/src/{libctonfile => libreader}/lexer.rs (100%) rename cranelift/src/{libctonfile => libreader}/lib.rs (82%) rename cranelift/src/{libctonfile => libreader}/parser.rs (100%) diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index 7281294fe6..f245d977c2 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -2,7 +2,7 @@ //! //! The `write` module provides the `write_function` function which converts an IL `Function` to an //! equivalent textual representation. This textual representation can be read back by the -//! `ctonfile` crate. +//! `cretonne-reader` crate. use std::io::{self, Write}; use repr::Function; diff --git a/cranelift/src/libctonfile/Cargo.toml b/cranelift/src/libreader/Cargo.toml similarity index 78% rename from cranelift/src/libctonfile/Cargo.toml rename to cranelift/src/libreader/Cargo.toml index 7b2826b072..478ee3b586 100644 --- a/cranelift/src/libctonfile/Cargo.toml +++ b/cranelift/src/libreader/Cargo.toml @@ -1,11 +1,11 @@ [package] authors = ["The Cretonne Project Developers"] -name = "ctonfile" +name = "cretonne-reader" version = "0.0.0" publish = false [lib] -name = "ctonfile" +name = "cton_reader" path = "lib.rs" [dependencies] diff --git a/cranelift/src/libctonfile/lexer.rs b/cranelift/src/libreader/lexer.rs similarity index 100% rename from cranelift/src/libctonfile/lexer.rs rename to cranelift/src/libreader/lexer.rs diff --git a/cranelift/src/libctonfile/lib.rs b/cranelift/src/libreader/lib.rs similarity index 82% rename from cranelift/src/libctonfile/lib.rs rename to cranelift/src/libreader/lib.rs index 9dcb169b7d..2d63e68264 100644 --- a/cranelift/src/libctonfile/lib.rs +++ b/cranelift/src/libreader/lib.rs @@ -1,11 +1,11 @@ // ====------------------------------------------------------------------------------------==== // // -// Cretonne file read/write library. +// Cretonne file reader library. // // ====------------------------------------------------------------------------------------==== // // -// The libctonfile library supports reading and writing .cton files. This functionality is needed +// The cton_reader library supports reading and writing .cton files. This functionality is needed // for testing Cretonne, but is not essential for a JIT compiler. // // ====------------------------------------------------------------------------------------==== // diff --git a/cranelift/src/libctonfile/parser.rs b/cranelift/src/libreader/parser.rs similarity index 100% rename from cranelift/src/libctonfile/parser.rs rename to cranelift/src/libreader/parser.rs diff --git a/cranelift/src/test-all.sh b/cranelift/src/test-all.sh index 6838b1544f..4fe15bdca2 100755 --- a/cranelift/src/test-all.sh +++ b/cranelift/src/test-all.sh @@ -1,4 +1,4 @@ #!/bin/bash cd $(dirname "$0")/tools -cargo test -p cretonne -p ctonfile -p cretonne-tools -cargo doc -p cretonne -p ctonfile -p cretonne-tools +cargo test -p cretonne -p cretonne-reader -p cretonne-tools +cargo doc -p cretonne -p cretonne-reader -p cretonne-tools diff --git a/cranelift/src/tools/Cargo.lock b/cranelift/src/tools/Cargo.lock index 90722ef5e7..91e9a5fae4 100644 --- a/cranelift/src/tools/Cargo.lock +++ b/cranelift/src/tools/Cargo.lock @@ -3,7 +3,7 @@ name = "cretonne-tools" version = "0.0.0" dependencies = [ "cretonne 0.0.0", - "ctonfile 0.0.0", + "cretonne-reader 0.0.0", ] [[package]] @@ -11,7 +11,7 @@ name = "cretonne" version = "0.0.0" [[package]] -name = "ctonfile" +name = "cretonne-reader" version = "0.0.0" dependencies = [ "cretonne 0.0.0", diff --git a/cranelift/src/tools/Cargo.toml b/cranelift/src/tools/Cargo.toml index b4477bd1e8..5811ca9a1a 100644 --- a/cranelift/src/tools/Cargo.toml +++ b/cranelift/src/tools/Cargo.toml @@ -11,4 +11,4 @@ path = "main.rs" [dependencies] cretonne = { path = "../libcretonne" } -ctonfile = { path = "../libctonfile" } +cretonne-reader = { path = "../libreader" } diff --git a/cranelift/src/tools/main.rs b/cranelift/src/tools/main.rs index 33cc1afe81..97b261d22d 100644 --- a/cranelift/src/tools/main.rs +++ b/cranelift/src/tools/main.rs @@ -1,5 +1,5 @@ extern crate cretonne; -extern crate ctonfile; +extern crate cton_reader; fn main() {} From 3d4103bf79c9dc2acf17b0a5f81124ab9c42796a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Apr 2016 15:13:09 -0700 Subject: [PATCH 0064/3084] Rename the 'cretonne' binary. It soome that 'cargo doc' gets confused when there is both a library crate and a binary called 'cretonne'. --- cranelift/src/test-all.sh | 6 ++++-- cranelift/src/tools/Cargo.toml | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/cranelift/src/test-all.sh b/cranelift/src/test-all.sh index 4fe15bdca2..f6dca84322 100755 --- a/cranelift/src/test-all.sh +++ b/cranelift/src/test-all.sh @@ -1,4 +1,6 @@ #!/bin/bash cd $(dirname "$0")/tools -cargo test -p cretonne -p cretonne-reader -p cretonne-tools -cargo doc -p cretonne -p cretonne-reader -p cretonne-tools +PKGS="-p cretonne -p cretonne-reader -p cretonne-tools" +cargo build $PKGS +cargo doc $PKGS +cargo test $PKGS diff --git a/cranelift/src/tools/Cargo.toml b/cranelift/src/tools/Cargo.toml index 5811ca9a1a..702bd635e6 100644 --- a/cranelift/src/tools/Cargo.toml +++ b/cranelift/src/tools/Cargo.toml @@ -6,7 +6,7 @@ description = "Binaries for testing the Cretonne library" publish = false [[bin]] -name = "cretonne" +name = "cton-util" path = "main.rs" [dependencies] From c344dbf4374d84cdce83c73a0927185844003fc3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Apr 2016 15:31:25 -0700 Subject: [PATCH 0065/3084] Set up Travis CI --- .travis.yml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 .travis.yml diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 0000000000..041073c206 --- /dev/null +++ b/.travis.yml @@ -0,0 +1,6 @@ +language: rust +rust: + - stable + - beta + - nightly +script: src/test-all.sh From a6c9d2b339e64f5dabee9d71c25fc44f61af6fd8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Apr 2016 15:33:24 -0700 Subject: [PATCH 0066/3084] Build status badge. --- README.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.rst b/README.rst index 6fae53072f..0a26eae385 100644 --- a/README.rst +++ b/README.rst @@ -9,6 +9,10 @@ target-independent intermediate language into executable machine code. :target: https://cretonne.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status +.. image:: https://travis-ci.org/stoklund/cretonne.svg?branch=master + :target: https://travis-ci.org/stoklund/cretonne + :alt: Build Status + Cretonne is designed to be a code generator for WebAssembly with these design goals: From aec94912d7be4def7566b006f642e96f78af353e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Apr 2016 15:45:48 -0700 Subject: [PATCH 0067/3084] Exit test script on errors. --- cranelift/src/test-all.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/cranelift/src/test-all.sh b/cranelift/src/test-all.sh index f6dca84322..c0d9977ed1 100755 --- a/cranelift/src/test-all.sh +++ b/cranelift/src/test-all.sh @@ -1,5 +1,11 @@ #!/bin/bash + +# Exit immediately on errors. +set -e + +# Run from the src/tools directory which includes all our crates. cd $(dirname "$0")/tools + PKGS="-p cretonne -p cretonne-reader -p cretonne-tools" cargo build $PKGS cargo doc $PKGS From ab50f174120d5ce566f2abdcfaee87fab90fb89b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Apr 2016 17:23:34 -0700 Subject: [PATCH 0068/3084] Implement Index for Function. When Function serves as a container for IL entities, use the Index trait to translate a reference class to a Data object. Works for: - StackSlot -> StackSlotData - Inst -> InstructionData --- cranelift/src/libcretonne/repr.rs | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index 3917fa8f48..c219ecb4b3 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -261,6 +261,15 @@ impl Display for Inst { } } +/// Allow immutable access to instructions via function indexing. +impl Index for Function { + type Output = InstructionData; + + fn index<'a>(&'a self, inst: Inst) -> &'a InstructionData { + &self.instructions[inst.index()] + } +} + // ====--------------------------------------------------------------------------------------====// // // Value implementation. @@ -402,11 +411,6 @@ impl Function { } } - /// Resolve an instruction reference. - pub fn inst(&self, i: Inst) -> &InstructionData { - &self.instructions[i.0 as usize] - } - /// Create a new instruction. pub fn make_inst(&mut self, data: InstructionData) -> Inst { let iref = Inst::new(self.instructions.len()); @@ -427,7 +431,7 @@ impl Function { use self::ExpandedValue::*; use self::ValueData::*; match v.expand() { - Direct(i) => self.inst(i).first_type(), + Direct(i) => self[i].first_type(), Table(i) => { match self.extended_values[i] { Unused => panic!("Can't get type of Unused value {}", v), @@ -457,7 +461,7 @@ mod tests { assert_eq!(inst.to_string(), "inst0"); // Immutable reference resolution. - let ins = func.inst(inst); + let ins = &func[inst]; assert_eq!(ins.opcode(), Opcode::Iconst); assert_eq!(ins.first_type(), types::I32); } From 24970593acbf6e06625242f61ce47ba2a11b3d27 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 2 May 2016 15:25:43 -0700 Subject: [PATCH 0069/3084] Implement value lists. Values that are defined together are represented as a singly linked list. These lists appear in: - Instructions with multiple result values. The first result value is special, and the following results form a linked list of Def extended_value table entries. - EBB arguments are represented as a linked list of Argument extended_value table entries. The EbbData struct has pointers to the first and last argument to allow fast insertion at both ends. Add a Values iterator type whicih can enumerate both kinds of value lists. --- cranelift/src/libcretonne/repr.rs | 287 +++++++++++++++++++++++++++--- 1 file changed, 265 insertions(+), 22 deletions(-) diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index c219ecb4b3..807be698ac 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -3,6 +3,7 @@ use types::{Type, FunctionName, Signature}; use immediates::*; +use std::default::Default; use std::fmt::{self, Display, Formatter, Write}; use std::ops::Index; use std::u32; @@ -81,12 +82,20 @@ pub struct StackSlotData { } /// Contents of an extended basic block. +/// +/// Arguments for an extended basic block are values that dominate everything in the EBB. All +/// branches to this EBB must provide matching arguments, and the arguments to the entry EBB must +/// match the function arguments. #[derive(Debug)] pub struct EbbData { - /// Arguments for this extended basic block. These values dominate everything in the EBB. - /// All branches to this EBB must provide matching arguments, and the arguments to the entry - /// EBB must match the function arguments. - pub arguments: Vec, + /// First argument to this EBB, or `NO_VALUE` if the block has no arguments. + /// + /// The arguments are all ValueData::Argument entries that form a linked list from `first_arg` + /// to `last_arg`. + first_arg: Value, + + /// Last argument to this EBB, or `NO_VALUE` if the block has no arguments. + last_arg: Value, } /// Contents on an instruction. @@ -132,12 +141,11 @@ pub enum InstructionData { /// Payload of a call instruction. #[derive(Debug)] pub struct CallData { - // Number of result values. - results: u8, + /// Second result value for a call producing multiple return values. + second_result: Value, - // Dynamically sized array containing `results-1` result values (not including the first value) - // followed by the argument values. - values: Vec, + // Dynamically sized array containing call argument values. + arguments: Vec, } @@ -233,7 +241,10 @@ impl Display for Ebb { impl EbbData { fn new() -> EbbData { - EbbData { arguments: Vec::new() } + EbbData { + first_arg: NO_VALUE, + last_arg: NO_VALUE, + } } } @@ -284,6 +295,9 @@ enum ExpandedValue { // This value is described in the extended value table. Table(usize), + + // This is NO_VALUE. + None, } impl Value { @@ -302,6 +316,9 @@ impl Value { // Expand the internal representation into something useful. fn expand(&self) -> ExpandedValue { use self::ExpandedValue::*; + if *self == NO_VALUE { + return None; + } let index = (self.0 / 2) as usize; if self.0 % 2 == 0 { Direct(Inst::new(index)) @@ -311,13 +328,20 @@ impl Value { } } +impl Default for Value { + fn default() -> Value { + NO_VALUE + } +} + /// Display a `Value` reference as "v7" or "v2x". impl Display for Value { fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { use self::ExpandedValue::*; match self.expand() { Direct(i) => write!(fmt, "v{}", i.0), - Table(i) => write!(fmt, "v{}x", i), + Table(i) => write!(fmt, "vx{}", i), + None => write!(fmt, "NO_VALUE"), } } } @@ -326,25 +350,67 @@ impl Display for Value { // Other values have an entry in the value table. #[derive(Debug)] enum ValueData { - // An unused entry in the value table. No instruction should be defining or using this value. - Unused, - // Value is defined by an instruction, but it is not the first result. Def { ty: Type, - num: u8, def: Inst, + next: Value, // Next result defined by `def`. }, // Value is an EBB argument. Argument { ty: Type, - num: u8, ebb: Ebb, + next: Value, // Next argument to `ebb`. }, } +/// Iterate through a list of related value references, such as: +/// +/// - All results defined by an instruction. +/// - All arguments to an EBB +/// +/// A value iterator borrows a Function reference. +pub struct Values<'a> { + func: &'a Function, + cur: Value, +} + +impl<'a> Iterator for Values<'a> { + type Item = Value; + + fn next(&mut self) -> Option { + let prev = self.cur; + + // Advance self.cur to the next value, or NO_VALUE. + self.cur = match prev.expand() { + ExpandedValue::Direct(inst) => self.func[inst].second_result().unwrap_or_default(), + ExpandedValue::Table(index) => { + match self.func.extended_values[index] { + ValueData::Def { next, .. } => next, + ValueData::Argument { next, .. } => next, + } + } + ExpandedValue::None => return None, + }; + + Some(prev) + } +} + impl InstructionData { + /// Create data for a call instruction. + pub fn call(opc: Opcode, return_type: Type) -> InstructionData { + InstructionData::Call { + opcode: opc, + ty: return_type, + data: Box::new(CallData { + second_result: NO_VALUE, + arguments: Vec::new(), + }), + } + } + /// Get the opcode of this instruction. pub fn opcode(&self) -> Opcode { use self::InstructionData::*; @@ -370,6 +436,31 @@ impl InstructionData { Call { ty, .. } => ty, } } + + /// Second result value, if any. + fn second_result(&self) -> Option { + use self::InstructionData::*; + match *self { + Nullary { .. } => None, + Unary { .. } => None, + UnaryImm { .. } => None, + Binary { .. } => None, + BinaryImm { .. } => None, + Call { ref data, .. } => Some(data.second_result), + } + } + + fn second_result_mut<'a>(&'a mut self) -> Option<&'a mut Value> { + use self::InstructionData::*; + match *self { + Nullary { .. } => None, + Unary { .. } => None, + UnaryImm { .. } => None, + Binary { .. } => None, + BinaryImm { .. } => None, + Call { ref mut data, .. } => Some(&mut data.second_result), + } + } } impl Function { @@ -396,6 +487,8 @@ impl Function { &self.signature } + // Stack slots. + /// Allocate a new stack slot. pub fn make_stack_slot(&mut self, data: StackSlotData) -> StackSlot { let ss = StackSlot::new(self.stack_slots.len()); @@ -411,14 +504,71 @@ impl Function { } } + // Instructions. + /// Create a new instruction. - pub fn make_inst(&mut self, data: InstructionData) -> Inst { - let iref = Inst::new(self.instructions.len()); + /// + /// The instruction is allowed to produce at most one result as indicated by `data.ty`. Use + /// `make_multi_inst()` to create instructions with multiple results. + pub fn make_inst(&mut self, data: InstructionData, extra_result_types: &[Type]) -> Inst { + let inst = Inst::new(self.instructions.len()); self.instructions.push(data); - // FIXME: Allocate extended value table entries if needed. - iref + inst } + /// Make an instruction that may produce multiple results. + /// + /// The type of the first result is `data.ty`. If the instruction generates more than one + /// result, additional result types are in `extra_result_types`. + /// + /// Not all instruction formats can represent multiple result values. This function will panic + /// if the format of `data` is insufficient. + pub fn make_multi_inst(&mut self, data: InstructionData, extra_result_types: &[Type]) -> Inst { + let inst = self.make_inst(data); + + if !extra_result_types.is_empty() { + // Additional values form a linked list starting from the second result value. Generate + // the list backwards so we don't have to modify value table entries in place. (This + // causes additional result values to be numbered backwards which is not the aestetic + // choice, but since it is only visible in extremely rare instructions with 3+ results, + // we don't care). + let mut head = NO_VALUE; + for ty in extra_result_types.into_iter().rev() { + head = self.make_value(ValueData::Def { + ty: *ty, + def: inst, + next: head, + }); + } + + // Update the second_result pointer in `inst`. + if let Some(second_result_ref) = self.instructions[inst.index()].second_result_mut() { + *second_result_ref = head; + } else { + panic!("Instruction format doesn't allow multiple results."); + } + } + + inst + } + + /// Get the first result of an instruction. + /// + /// If `Inst` doesn't produce any results, this returns a `Value` with a `VOID` type. + pub fn first_result(&self, inst: Inst) -> Value { + Value::new_direct(inst) + } + + /// Iterate through all the results of an instruction. + pub fn inst_results<'a>(&'a self, inst: Inst) -> Values<'a> { + Values { + func: self, + cur: Value::new_direct(inst), + } + } + + // Basic blocks + /// Create a new basic block. pub fn make_ebb(&mut self) -> Ebb { let ebb = Ebb::new(self.extended_basic_blocks.len()); @@ -426,6 +576,60 @@ impl Function { ebb } + /// Reference the representation of an EBB. + fn ebb(&self, ebb: Ebb) -> &EbbData { + &self.extended_basic_blocks[ebb.index()] + } + + /// Mutably reference the representation of an EBB. + fn ebb_mut(&mut self, ebb: Ebb) -> &mut EbbData { + &mut self.extended_basic_blocks[ebb.index()] + } + + /// Append an argument with type `ty` to `ebb`. + pub fn append_ebb_arg(&mut self, ebb: Ebb, ty: Type) -> Value { + let val = self.make_value(ValueData::Argument { + ty: ty, + ebb: ebb, + next: NO_VALUE, + }); + + let last_arg = self.ebb(ebb).last_arg; + match last_arg.expand() { + // If last_arg = NO_VALUE, we're adding the first EBB argument. + ExpandedValue::None => self.ebb_mut(ebb).first_arg = val, + ExpandedValue::Table(index) => { + // Append to linked list of arguments. + if let ValueData::Argument { ref mut next, .. } = self.extended_values[index] { + *next = val; + } else { + panic!("wrong type of extended value referenced by Ebb::last_arg"); + } + } + ExpandedValue::Direct(_) => panic!("Direct value cannot appear as EBB argument"), + } + self.ebb_mut(ebb).last_arg = val; + + val + } + + /// Iterate through the arguments to an EBB. + pub fn ebb_args<'a>(&'a self, ebb: Ebb) -> Values<'a> { + Values { + func: self, + cur: self.ebb(ebb).first_arg, + } + } + + // Values. + + /// Allocate an extended value entry. + fn make_value(&mut self, data: ValueData) -> Value { + let vref = Value::new_table(self.extended_values.len()); + self.extended_values.push(data); + vref + } + /// Get the type of a value. pub fn value_type(&self, v: Value) -> Type { use self::ExpandedValue::*; @@ -434,11 +638,11 @@ impl Function { Direct(i) => self[i].first_type(), Table(i) => { match self.extended_values[i] { - Unused => panic!("Can't get type of Unused value {}", v), Def { ty, .. } => ty, Argument { ty, .. } => ty, } } + None => panic!("NO_VALUE has no type"), } } } @@ -457,7 +661,7 @@ mod tests { opcode: Opcode::Iconst, ty: types::I32, }; - let inst = func.make_inst(idata); + let inst = func.make_inst(idata, &[]); assert_eq!(inst.to_string(), "inst0"); // Immutable reference resolution. @@ -466,6 +670,21 @@ mod tests { assert_eq!(ins.first_type(), types::I32); } + #[test] + fn multiple_results() { + use types::*; + let mut func = Function::new(); + + let idata = InstructionData::call(Opcode::Vconst, I64); + let inst = func.make_inst(idata, &[I8, F64]); + assert_eq!(inst.to_string(), "inst0"); + let results: Vec = func.inst_results(inst).collect(); + assert_eq!(results.len(), 3); + assert_eq!(func.value_type(results[0]), I64); + assert_eq!(func.value_type(results[1]), I8); + assert_eq!(func.value_type(results[2]), F64); + } + #[test] fn stack_slot() { let mut func = Function::new(); @@ -479,4 +698,28 @@ mod tests { assert_eq!(func[ss1].size, 8); } + #[test] + fn ebb() { + let mut func = Function::new(); + + let ebb = func.make_ebb(); + assert_eq!(ebb.to_string(), "ebb0"); + assert_eq!(func.ebb_args(ebb).next(), None); + + let arg1 = func.append_ebb_arg(ebb, types::F32); + assert_eq!(arg1.to_string(), "vx0"); + { + let mut args1 = func.ebb_args(ebb); + assert_eq!(args1.next(), Some(arg1)); + assert_eq!(args1.next(), None); + } + let arg2 = func.append_ebb_arg(ebb, types::I16); + assert_eq!(arg2.to_string(), "vx1"); + { + let mut args2 = func.ebb_args(ebb); + assert_eq!(args2.next(), Some(arg1)); + assert_eq!(args2.next(), Some(arg2)); + assert_eq!(args2.next(), None); + } + } } From 6e17d229d008ff3ca221df11de58cae6b321e9e0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 2 May 2016 16:04:21 -0700 Subject: [PATCH 0070/3084] Fix build. --- cranelift/src/libcretonne/repr.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index 807be698ac..534fd1a7cb 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -510,7 +510,7 @@ impl Function { /// /// The instruction is allowed to produce at most one result as indicated by `data.ty`. Use /// `make_multi_inst()` to create instructions with multiple results. - pub fn make_inst(&mut self, data: InstructionData, extra_result_types: &[Type]) -> Inst { + pub fn make_inst(&mut self, data: InstructionData) -> Inst { let inst = Inst::new(self.instructions.len()); self.instructions.push(data); inst @@ -661,7 +661,7 @@ mod tests { opcode: Opcode::Iconst, ty: types::I32, }; - let inst = func.make_inst(idata, &[]); + let inst = func.make_inst(idata); assert_eq!(inst.to_string(), "inst0"); // Immutable reference resolution. @@ -676,7 +676,7 @@ mod tests { let mut func = Function::new(); let idata = InstructionData::call(Opcode::Vconst, I64); - let inst = func.make_inst(idata, &[I8, F64]); + let inst = func.make_multi_inst(idata, &[I8, F64]); assert_eq!(inst.to_string(), "inst0"); let results: Vec = func.inst_results(inst).collect(); assert_eq!(results.len(), 3); From ea46a17f5661c1d57cf88d8d982f84d90662ff93 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 12 May 2016 13:37:03 -0700 Subject: [PATCH 0071/3084] PEP8 formatting. --- meta/constant_hash.py | 3 + meta/cretonne/__init__.py | 63 ++++-- meta/cretonne/base.py | 405 ++++++++++++++++++---------------- meta/cretonne/types.py | 39 ++-- meta/gen_instr.py | 10 +- meta/srcgen.py | 3 +- meta/target/__init__.py | 1 + meta/target/riscv/__init__.py | 7 +- 8 files changed, 299 insertions(+), 232 deletions(-) diff --git a/meta/constant_hash.py b/meta/constant_hash.py index b1b6104231..27a30476a4 100644 --- a/meta/constant_hash.py +++ b/meta/constant_hash.py @@ -6,6 +6,7 @@ don't attempt parfect hashing, but simply generate an open addressed quadratically probed hash table. """ + def simple_hash(s): """ Compute a primitive hash of a string. @@ -21,6 +22,7 @@ def simple_hash(s): h = ((h ^ ord(c)) + ((h >> 6) + (h << 26))) & 0xffffffff return h + def next_power_of_two(x): """ Compute the next power of two that is greater than `x`: @@ -41,6 +43,7 @@ def next_power_of_two(x): s *= 2 return x + 1 + def compute_quadratic(items, hash_function): """ Compute an open addressed, quadratically probed hash table containing diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index d69d5c35c4..a4ef2830c0 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -7,15 +7,20 @@ instructions. import re + camel_re = re.compile('(^|_)([a-z])') + + def camel_case(s): """Convert the string s to CamelCase""" return camel_re.sub(lambda m: m.group(2).upper(), s) + # Concrete types. # # Instances (i8, i32, ...) are provided in the cretonne.types module. + class Type(object): """A concrete value type.""" @@ -27,6 +32,7 @@ class Type(object): def __str__(self): return self.name + class ScalarType(Type): """ A concrete scalar (not vector) type. @@ -55,6 +61,7 @@ class ScalarType(Type): self._vectors[lanes] = v return v + class VectorType(Type): """ A concrete SIMD vector type. @@ -75,7 +82,9 @@ class VectorType(Type): self.lanes = lanes def __repr__(self): - return 'VectorType(base={}, lanes={})'.format(self.base.name, self.lanes) + return ('VectorType(base={}, lanes={})' + .format(self.base.name, self.lanes)) + class IntType(ScalarType): """A concrete scalar integer type.""" @@ -91,17 +100,20 @@ class IntType(ScalarType): def __repr__(self): return 'IntType(bits={})'.format(self.bits) + class FloatType(ScalarType): """A concrete scalar floating point type.""" def __init__(self, bits, doc): assert bits > 0, 'FloatType must have positive number of bits' - super(FloatType, self).__init__( name='f{:d}'.format(bits), membytes=bits/8, doc=doc) + super(FloatType, self).__init__(name='f{:d}'.format(bits), + membytes=bits/8, doc=doc) self.bits = bits def __repr__(self): return 'FloatType(bits={})'.format(self.bits) + class BoolType(ScalarType): """A concrete scalar boolean type.""" @@ -116,9 +128,9 @@ class BoolType(ScalarType): def __repr__(self): return 'BoolType(bits={})'.format(self.bits) -# + # Parametric polymorphism. -# + class TypeVar(object): """ @@ -132,12 +144,13 @@ class TypeVar(object): self.name = name self.__doc__ = doc -# + # Immediate operands. # # Instances of immediate operand types are provided in the cretonne.immediates # module. + class ImmediateType(object): """ The type of an immediate instruction operand. @@ -153,9 +166,9 @@ class ImmediateType(object): def __repr__(self): return 'ImmediateType({})'.format(self.name) -# + # Defining instructions. -# + class InstructionGroup(object): """ @@ -178,7 +191,8 @@ class InstructionGroup(object): added to this group. """ assert InstructionGroup._current is None, ( - "Can't open {} since {} is already open".format(self, _current)) + "Can't open {} since {} is already open" + .format(self, InstructionGroup._current)) InstructionGroup._current = self def close(self): @@ -187,7 +201,8 @@ class InstructionGroup(object): opening another instruction group. """ assert InstructionGroup._current is self, ( - "Can't close {}, the open instuction group is {}".format(self, _current)) + "Can't close {}, the open instuction group is {}" + .format(self, InstructionGroup._current)) InstructionGroup._current = None def __init__(self, name, doc): @@ -198,9 +213,11 @@ class InstructionGroup(object): @staticmethod def append(inst): - assert InstructionGroup._current, "Open an instruction group before defining instructions." + assert InstructionGroup._current, \ + "Open an instruction group before defining instructions." InstructionGroup._current.instructions.append(inst) + class Operand(object): """ An instruction operand. @@ -212,12 +229,12 @@ class Operand(object): concrete type. 2. A :py:class:`TypeVar` instance indicates an SSA value operand, and the - instruction is polymorphic over the possible concrete types that the type - variable can assume. + instruction is polymorphic over the possible concrete types that the + type variable can assume. 3. An :py:class:`ImmediateType` instance indicates an immediate operand - whose value is encoded in the instruction itself rather than being passed - as an SSA value. + whose value is encoded in the instruction itself rather than being + passed as an SSA value. """ def __init__(self, name, typ, doc=''): @@ -231,19 +248,20 @@ class Operand(object): else: return self.typ.__doc__ + class Instruction(object): """ An instruction. The operands to the instruction are specified as two tuples: ``ins`` and ``outs``. Since the Python singleton tuple syntax is a bit awkward, it is - allowed to specify a singleton as just the operand itself, i.e., `ins=x` and - `ins=(x,)` are both allowed and mean the same thing. + allowed to specify a singleton as just the operand itself, i.e., `ins=x` + and `ins=(x,)` are both allowed and mean the same thing. :param name: Instruction mnemonic, also becomes opcode name. :param doc: Documentation string. - :param ins: Tuple of input operands. This can be a mix of SSA value operands - and immediate operands. + :param ins: Tuple of input operands. This can be a mix of SSA value + operands and immediate operands. :param outs: Tuple of output operands. The output operands can't be immediates. """ @@ -258,8 +276,8 @@ class Instruction(object): @staticmethod def _to_operand_tuple(x): - # Allow a single Operand instance instead of the awkward singleton tuple - # syntax. + # Allow a single Operand instance instead of the awkward singleton + # tuple syntax. if isinstance(x, Operand): x = (x,) else: @@ -268,9 +286,10 @@ class Instruction(object): assert isinstance(op, Operand) return x -# + # Defining targets -# + + class Target(object): """ A target instruction set architecture. diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 604687829e..033be2c148 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -1,7 +1,8 @@ """ Cretonne base instruction set. -This module defines the basic Cretonne instruction set that all targets support. +This module defines the basic Cretonne instruction set that all targets +support. """ from . import TypeVar, Operand, Instruction, InstructionGroup from types import i8, f32, f64 @@ -19,42 +20,46 @@ TxN = TypeVar('%Tx%N', 'A SIMD vector type') N = Operand('N', imm64) a = Operand('a', Int, doc='A constant integer scalar or vector value') -iconst = Instruction('iconst', r""" - Integer constant. +iconst = Instruction( + 'iconst', r""" + Integer constant. - Create a scalar integer SSA value with an immediate constant value, or an - integer vector where all the lanes have the same value. - """, - ins=N, outs=a) + Create a scalar integer SSA value with an immediate constant value, or + an integer vector where all the lanes have the same value. + """, + ins=N, outs=a) N = Operand('N', ieee32) a = Operand('a', f32, doc='A constant integer scalar or vector value') -f32const = Instruction('f32const', r""" - Floating point constant. +f32const = Instruction( + 'f32const', r""" + Floating point constant. - Create a :type:`f32` SSA value with an immediate constant value, or a - floating point vector where all the lanes have the same value. - """, - ins=N, outs=a) + Create a :type:`f32` SSA value with an immediate constant value, or a + floating point vector where all the lanes have the same value. + """, + ins=N, outs=a) N = Operand('N', ieee64) a = Operand('a', f64, doc='A constant integer scalar or vector value') -f64const = Instruction('f64const', r""" - Floating point constant. +f64const = Instruction( + 'f64const', r""" + Floating point constant. - Create a :type:`f64` SSA value with an immediate constant value, or a - floating point vector where all the lanes have the same value. - """, - ins=N, outs=a) + Create a :type:`f64` SSA value with an immediate constant value, or a + floating point vector where all the lanes have the same value. + """, + ins=N, outs=a) N = Operand('N', immvector) a = Operand('a', TxN, doc='A constant vector value') -vconst = Instruction('vconst', r""" - Vector constant (floating point or integer). +vconst = Instruction( + 'vconst', r""" + Vector constant (floating point or integer). - Create a SIMD vector value where the lanes don't have to be identical. - """, - ins=N, outs=a) + Create a SIMD vector value where the lanes don't have to be identical. + """, + ins=N, outs=a) # # Integer arithmetic @@ -64,133 +69,148 @@ a = Operand('a', Int) x = Operand('x', Int) y = Operand('y', Int) -iadd = Instruction('iadd', r""" - Wrapping integer addition: :math:`a := x + y \pmod{2^B}`. +iadd = Instruction( + 'iadd', r""" + Wrapping integer addition: :math:`a := x + y \pmod{2^B}`. - This instruction does not depend on the signed/unsigned interpretation of - the operands. - """, - ins=(x,y), outs=a) + This instruction does not depend on the signed/unsigned interpretation + of the operands. + """, + ins=(x, y), outs=a) -isub = Instruction('isub', r""" - Wrapping integer subtraction: :math:`a := x - y \pmod{2^B}`. +isub = Instruction( + 'isub', r""" + Wrapping integer subtraction: :math:`a := x - y \pmod{2^B}`. - This instruction does not depend on the signed/unsigned interpretation of - the operands. - """, - ins=(x,y), outs=a) + This instruction does not depend on the signed/unsigned interpretation + of the operands. + """, + ins=(x, y), outs=a) -imul = Instruction('imul', r""" - Wrapping integer multiplication: :math:`a := x y \pmod{2^B}`. +imul = Instruction( + 'imul', r""" + Wrapping integer multiplication: :math:`a := x y \pmod{2^B}`. - This instruction does not depend on the signed/unsigned interpretation of - the - operands. + This instruction does not depend on the signed/unsigned interpretation + of the + operands. - Polymorphic over all integer types (vector and scalar). - """, - ins=(x,y), outs=a) + Polymorphic over all integer types (vector and scalar). + """, + ins=(x, y), outs=a) -udiv = Instruction('udiv', r""" - Unsigned integer division: :math:`a := \lfloor {x \over y} \rfloor`. +udiv = Instruction( + 'udiv', r""" + Unsigned integer division: :math:`a := \lfloor {x \over y} \rfloor`. - This operation traps if the divisor is zero. - """, - ins=(x,y), outs=a) + This operation traps if the divisor is zero. + """, + ins=(x, y), outs=a) -sdiv = Instruction('sdiv', r""" - Signed integer division rounded toward zero: :math:`a := sign(xy) \lfloor - {|x| \over |y|}\rfloor`. +sdiv = Instruction( + 'sdiv', r""" + Signed integer division rounded toward zero: :math:`a := sign(xy) + \lfloor {|x| \over |y|}\rfloor`. - This operation traps if the divisor is zero, or if the result is not - representable in :math:`B` bits two's complement. This only happens when - :math:`x = -2^{B-1}, y = -1`. - """, - ins=(x,y), outs=a) + This operation traps if the divisor is zero, or if the result is not + representable in :math:`B` bits two's complement. This only happens + when :math:`x = -2^{B-1}, y = -1`. + """, + ins=(x, y), outs=a) -urem = Instruction('urem', """ - Unsigned integer remainder. +urem = Instruction( + 'urem', """ + Unsigned integer remainder. - This operation traps if the divisor is zero. - """, - ins=(x,y), outs=a) + This operation traps if the divisor is zero. + """, + ins=(x, y), outs=a) -srem = Instruction('srem', """ - Signed integer remainder. +srem = Instruction( + 'srem', """ + Signed integer remainder. - This operation traps if the divisor is zero. + This operation traps if the divisor is zero. - .. todo:: Integer remainder vs modulus. + .. todo:: Integer remainder vs modulus. Clarify whether the result has the sign of the divisor or the dividend. Should we add a ``smod`` instruction for the case where the result has the same sign as the divisor? - """, - ins=(x,y), outs=a) + """, + ins=(x, y), outs=a) a = Operand('a', iB) x = Operand('x', iB) Y = Operand('Y', imm64) -iadd_imm = Instruction('iadd_imm', """ - Add immediate integer. +iadd_imm = Instruction( + 'iadd_imm', """ + Add immediate integer. - Same as :inst:`iadd`, but one operand is an immediate constant. + Same as :inst:`iadd`, but one operand is an immediate constant. - Polymorphic over all scalar integer types, but does not support vector - types. - """, - ins=(x,Y), outs=a) + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(x, Y), outs=a) -imul_imm = Instruction('imul_imm', """ - Integer multiplication by immediate constant. +imul_imm = Instruction( + 'imul_imm', """ + Integer multiplication by immediate constant. - Polymorphic over all scalar integer types. - """, - ins=(x,Y), outs=a) + Polymorphic over all scalar integer types. + """, + ins=(x, Y), outs=a) -udiv_imm = Instruction('udiv_imm', """ - Unsigned integer division by an immediate constant. +udiv_imm = Instruction( + 'udiv_imm', """ + Unsigned integer division by an immediate constant. - This instruction never traps because a divisor of zero is not allowed. - """, - ins=(x,Y), outs=a) + This instruction never traps because a divisor of zero is not allowed. + """, + ins=(x, Y), outs=a) -sdiv_imm = Instruction('sdiv_imm', """ - Signed integer division by an immediate constant. +sdiv_imm = Instruction( + 'sdiv_imm', """ + Signed integer division by an immediate constant. - This instruction never traps because a divisor of -1 or 0 is not allowed. - """, - ins=(x,Y), outs=a) + This instruction never traps because a divisor of -1 or 0 is not + allowed. """, + ins=(x, Y), outs=a) -urem_imm = Instruction('urem_imm', """ - Unsigned integer remainder with immediate divisor. +urem_imm = Instruction( + 'urem_imm', """ + Unsigned integer remainder with immediate divisor. - This instruction never traps because a divisor of zero is not allowed. - """, - ins=(x,Y), outs=a) + This instruction never traps because a divisor of zero is not allowed. + """, + ins=(x, Y), outs=a) -srem_imm = Instruction('srem_imm', """ - Signed integer remainder with immediate divisor. +srem_imm = Instruction( + 'srem_imm', """ + Signed integer remainder with immediate divisor. - This instruction never traps because a divisor of 0 or -1 is not allowed. - """, - ins=(x,Y), outs=a) + This instruction never traps because a divisor of 0 or -1 is not + allowed. """, + ins=(x, Y), outs=a) # Swap x and y for isub_imm. X = Operand('X', imm64) y = Operand('y', iB) -isub_imm = Instruction('isub_imm', """ - Immediate wrapping subtraction: :math:`a := X - y \pmod{2^B}`. +isub_imm = Instruction( + 'isub_imm', """ + Immediate wrapping subtraction: :math:`a := X - y \pmod{2^B}`. - Also works as integer negation when :math:`X = 0`. Use :inst:`iadd_imm` with a - negative immediate operand for the reverse immediate subtraction. + Also works as integer negation when :math:`X = 0`. Use :inst:`iadd_imm` + with a negative immediate operand for the reverse immediate + subtraction. - Polymorphic over all scalar integer types, but does not support vector - types. - """, - ins=(X,y), outs=a) + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(X, y), outs=a) # # Bitwise operations. @@ -202,86 +222,97 @@ x = Operand('x', bits) y = Operand('y', bits) a = Operand('a', bits) -band = Instruction('band', """ - Bitwise and. - """, - ins=(x,y), outs=a) +band = Instruction( + 'band', """ + Bitwise and. + """, + ins=(x, y), outs=a) -bor = Instruction('bor', """ - Bitwise or. - """, - ins=(x,y), outs=a) +bor = Instruction( + 'bor', """ + Bitwise or. + """, + ins=(x, y), outs=a) -bxor = Instruction('bxor', """ - Bitwise xor. - """, - ins=(x,y), outs=a) +bxor = Instruction( + 'bxor', """ + Bitwise xor. + """, + ins=(x, y), outs=a) -bnot = Instruction('bnot', """ - Bitwise not. - """, - ins=x, outs=a) +bnot = Instruction( + 'bnot', """ + Bitwise not. + """, + ins=x, outs=a) # Shift/rotate. x = Operand('x', Int, doc='Scalar or vector value to shift') y = Operand('y', iB, doc='Number of bits to shift') a = Operand('a', Int) -rotl = Instruction('rotl', r""" - Rotate left. +rotl = Instruction( + 'rotl', r""" + Rotate left. - Rotate the bits in ``x`` by ``y`` places. - """, - ins=(x,y), outs=a) + Rotate the bits in ``x`` by ``y`` places. + """, + ins=(x, y), outs=a) -rotr = Instruction('rotr', r""" - Rotate right. +rotr = Instruction( + 'rotr', r""" + Rotate right. - Rotate the bits in ``x`` by ``y`` places. - """, - ins=(x,y), outs=a) + Rotate the bits in ``x`` by ``y`` places. + """, + ins=(x, y), outs=a) -ishl = Instruction('ishl', r""" - Integer shift left. Shift the bits in ``x`` towards the MSB by ``y`` - places. Shift in zero bits to the LSB. +ishl = Instruction( + 'ishl', r""" + Integer shift left. Shift the bits in ``x`` towards the MSB by ``y`` + places. Shift in zero bits to the LSB. - The shift amount is masked to the size of ``x``. + The shift amount is masked to the size of ``x``. - When shifting a B-bits integer type, this instruction computes: + When shifting a B-bits integer type, this instruction computes: - .. math:: + .. math:: s &:= y \pmod B, \\ a &:= x \cdot 2^s \pmod{2^B}. - .. todo:: Add ``ishl_imm`` variant with an immediate ``y``. - """, - ins=(x,y), outs=a) + .. todo:: Add ``ishl_imm`` variant with an immediate ``y``. + """, + ins=(x, y), outs=a) -ushr = Instruction('ushr', r""" - Unsigned shift right. Shift bits in ``x`` towards the LSB by ``y`` places, - shifting in zero bits to the MSB. Also called a *logical shift*. +ushr = Instruction( + 'ushr', r""" + Unsigned shift right. Shift bits in ``x`` towards the LSB by ``y`` + places, shifting in zero bits to the MSB. Also called a *logical + shift*. - The shift amount is masked to the size of the register. + The shift amount is masked to the size of the register. - When shifting a B-bits integer type, this instruction computes: + When shifting a B-bits integer type, this instruction computes: - .. math:: + .. math:: s &:= y \pmod B, \\ a &:= \lfloor x \cdot 2^{-s} \rfloor. - .. todo:: Add ``ushr_imm`` variant with an immediate ``y``. - """, - ins=(x,y), outs=a) + .. todo:: Add ``ushr_imm`` variant with an immediate ``y``. + """, + ins=(x, y), outs=a) -sshr = Instruction('sshr', r""" - Signed shift right. Shift bits in ``x`` towards the LSB by ``y`` places, - shifting in sign bits to the MSB. Also called an *arithmetic shift*. +sshr = Instruction( + 'sshr', r""" + Signed shift right. Shift bits in ``x`` towards the LSB by ``y`` + places, shifting in sign bits to the MSB. Also called an *arithmetic + shift*. - The shift amount is masked to the size of the register. + The shift amount is masked to the size of the register. - .. todo:: Add ``sshr_imm`` variant with an immediate ``y``. - """, - ins=(x,y), outs=a) + .. todo:: Add ``sshr_imm`` variant with an immediate ``y``. + """, + ins=(x, y), outs=a) # # Bit counting. @@ -290,38 +321,42 @@ sshr = Instruction('sshr', r""" x = Operand('x', iB) a = Operand('a', i8) -clz = Instruction('clz', r""" - Count leading zero bits. +clz = Instruction( + 'clz', r""" + Count leading zero bits. - Starting from the MSB in ``x``, count the number of zero bits before - reaching the first one bit. When ``x`` is zero, returns the size of x in - bits. - """, - ins=x, outs=a) + Starting from the MSB in ``x``, count the number of zero bits before + reaching the first one bit. When ``x`` is zero, returns the size of x + in bits. + """, + ins=x, outs=a) -cls = Instruction('cls', r""" - Count leading sign bits. +cls = Instruction( + 'cls', r""" + Count leading sign bits. - Starting from the MSB after the sign bit in ``x``, count the number of - consecutive bits identical to the sign bit. When ``x`` is 0 or -1, returns - one less than the size of x in bits. - """, - ins=x, outs=a) + Starting from the MSB after the sign bit in ``x``, count the number of + consecutive bits identical to the sign bit. When ``x`` is 0 or -1, + returns one less than the size of x in bits. + """, + ins=x, outs=a) -ctz = Instruction('ctz', r""" - Count trailing zeros. +ctz = Instruction( + 'ctz', r""" + Count trailing zeros. - Starting from the LSB in ``x``, count the number of zero bits before - reaching the first one bit. When ``x`` is zero, returns the size of x in - bits. - """, - ins=x, outs=a) + Starting from the LSB in ``x``, count the number of zero bits before + reaching the first one bit. When ``x`` is zero, returns the size of x + in bits. + """, + ins=x, outs=a) -popcnt = Instruction('popcnt', r""" - Population count +popcnt = Instruction( + 'popcnt', r""" + Population count - Count the number of one bits in ``x``. - """, - ins=x, outs=a) + Count the number of one bits in ``x``. + """, + ins=x, outs=a) instructions.close() diff --git a/meta/cretonne/types.py b/meta/cretonne/types.py index c639d89050..d92e6f9113 100644 --- a/meta/cretonne/types.py +++ b/meta/cretonne/types.py @@ -5,33 +5,34 @@ The cretonne.types module predefines all the Cretonne scalar types. from . import ScalarType, IntType, FloatType, BoolType #: Boolean. -b1 = ScalarType('b1', 0, +b1 = ScalarType( + 'b1', 0, """ A boolean value that is either true or false. """) -b8 = BoolType(8) #: 8-bit bool. -b16 = BoolType(16) #: 16-bit bool. -b32 = BoolType(32) #: 32-bit bool. -b64 = BoolType(64) #: 64-bit bool. +b8 = BoolType(8) #: 8-bit bool. +b16 = BoolType(16) #: 16-bit bool. +b32 = BoolType(32) #: 32-bit bool. +b64 = BoolType(64) #: 64-bit bool. -i8 = IntType(8) #: 8-bit int. -i16 = IntType(16) #: 16-bit int. -i32 = IntType(32) #: 32-bit int. -i64 = IntType(64) #: 64-bit int. +i8 = IntType(8) #: 8-bit int. +i16 = IntType(16) #: 16-bit int. +i32 = IntType(32) #: 32-bit int. +i64 = IntType(64) #: 64-bit int. #: IEEE single precision. -f32 = FloatType(32, - """ - A 32-bit floating point type represented in the IEEE 754-2008 *binary32* - interchange format. This corresponds to the :c:type:`float` type in most - C implementations. +f32 = FloatType( + 32, """ + A 32-bit floating point type represented in the IEEE 754-2008 + *binary32* interchange format. This corresponds to the :c:type:`float` + type in most C implementations. """) #: IEEE double precision. -f64 = FloatType(64, - """ - A 64-bit floating point type represented in the IEEE 754-2008 *binary64* - interchange format. This corresponds to the :c:type:`double` type in - most C implementations. +f64 = FloatType( + 64, """ + A 64-bit floating point type represented in the IEEE 754-2008 + *binary64* interchange format. This corresponds to the :c:type:`double` + type in most C implementations. """) diff --git a/meta/gen_instr.py b/meta/gen_instr.py index eb1b0a361f..93b02bc369 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -5,6 +5,7 @@ Generate sources with instruction info. import srcgen import constant_hash + def collect_instr_groups(targets): seen = set() groups = [] @@ -15,6 +16,7 @@ def collect_instr_groups(targets): seen.add(g) return groups + def gen_opcodes(groups, out_dir): """Generate opcode enumerations.""" fmt = srcgen.Formatter() @@ -46,9 +48,12 @@ def gen_opcodes(groups, out_dir): fmt.format('Opcode::{} => "{}",', i.camel_name, i.name) # Generate an opcode hash table for looking up opcodes by name. - hash_table = constant_hash.compute_quadratic(instrs, + hash_table = constant_hash.compute_quadratic( + instrs, lambda i: constant_hash.simple_hash(i.name)) - with fmt.indented('const OPCODE_HASH_TABLE: [Opcode; {}] = ['.format(len(hash_table)), '];'): + with fmt.indented( + 'const OPCODE_HASH_TABLE: [Opcode; {}] = [' + .format(len(hash_table)), '];'): for i in hash_table: if i is None: fmt.line('Opcode::NotAnOpcode,') @@ -57,6 +62,7 @@ def gen_opcodes(groups, out_dir): fmt.update_file('opcodes.rs', out_dir) + def generate(targets, out_dir): groups = collect_instr_groups(targets) gen_opcodes(groups, out_dir) diff --git a/meta/srcgen.py b/meta/srcgen.py index 125ae0a63a..7895b0ea38 100644 --- a/meta/srcgen.py +++ b/meta/srcgen.py @@ -9,6 +9,7 @@ source code. import sys import os + class Formatter(object): """ Source code formatter class. @@ -70,7 +71,7 @@ class Formatter(object): self.after = after def __enter__(self): - self.fmt.indent_push(); + self.fmt.indent_push() def __exit__(self, t, v, tb): self.fmt.indent_pop() diff --git a/meta/target/__init__.py b/meta/target/__init__.py index bed730b207..1f9f7a0565 100644 --- a/meta/target/__init__.py +++ b/meta/target/__init__.py @@ -8,6 +8,7 @@ set architecture supported by Cretonne. from . import riscv + def all_targets(): """ Get a list of all the supported targets. Each target is represented as a diff --git a/meta/target/riscv/__init__.py b/meta/target/riscv/__init__.py index e9b9926e31..bdb5b63091 100644 --- a/meta/target/riscv/__init__.py +++ b/meta/target/riscv/__init__.py @@ -2,9 +2,10 @@ RISC-V Target ------------- -`RISC-V `_ is an open instruction set architecture originally -developed at UC Berkeley. It is a RISC-style ISA with either a 32-bit (RV32I) or -64-bit (RV32I) base instruction set and a number of optional extensions: +`RISC-V `_ is an open instruction set architecture +originally developed at UC Berkeley. It is a RISC-style ISA with either a +32-bit (RV32I) or 64-bit (RV32I) base instruction set and a number of optional +extensions: RV32M / RV64M Integer multiplication and division. From 477fa01bfc98f5387278a09a870af3ab1ca7e2e2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 12 May 2016 15:59:40 -0700 Subject: [PATCH 0072/3084] Add OperandKind to the meta language. We have a two-level type system: OperandKinds and ValueTypes. The value types only apply to value operands, but there are many more kinds of operands: immediate numbers, condition codes, basic block references, etc. --- cranelift/docs/metaref.rst | 46 ++++++++------- meta/cretonne/__init__.py | 115 ++++++++++++++++++++++++++---------- meta/cretonne/base.py | 9 +-- meta/cretonne/immediates.py | 12 ++-- 4 files changed, 120 insertions(+), 62 deletions(-) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index c8f8b0e265..fe46e8af4a 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -15,15 +15,15 @@ module :mod:`cretonne.base`. .. module:: cretonne -Types -===== +Value Types +=========== -Concrete value types are represented as instances of :class:`cretonne.Type`. There are +Concrete value types are represented as instances of :class:`cretonne.ValueType`. There are subclasses to represent scalar and vector types. -.. inheritance-diagram:: Type ScalarType VectorType IntType FloatType +.. inheritance-diagram:: ValueType ScalarType VectorType IntType FloatType :parts: 1 -.. autoclass:: Type +.. autoclass:: ValueType .. autoclass:: ScalarType :members: .. autoclass:: VectorType @@ -48,33 +48,35 @@ types for their operands. This makes the instructions polymorphic. .. autoclass:: TypeVar -Immediates ----------- - -Immediate instruction operands don't correspond to SSA values, but have values -that are encoded directly in the instruction. Immediate operands don't -have types from the :class:`cretonne.Type` type system; they often have -enumerated values of a specific type. The type of an immediate operand is -indicated with an instance of :class:`ImmediateType`. - -.. autoclass:: ImmediateType - -.. automodule:: cretonne.immediates - :members: - -.. currentmodule:: cretonne - Instructions ============ New instructions are defined as instances of the :class:`cretonne.Instruction` class. -.. autoclass:: Operand .. autoclass:: Instruction +.. autoclass:: Operand +.. autoclass:: OperandKind .. autoclass:: InstructionGroup :members: + +Immediates +---------- + +Immediate instruction operands don't correspond to SSA values, but have values +that are encoded directly in the instruction. Immediate operands don't +have types from the :class:`cretonne.ValueType` type system; they often have +enumerated values of a specific type. The type of an immediate operand is +indicated with an instance of :class:`ImmediateKind`. + +.. autoclass:: ImmediateKind + +.. automodule:: cretonne.immediates + :members: + +.. currentmodule:: cretonne + Targets ======= diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index a4ef2830c0..26a3664c64 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -16,13 +16,75 @@ def camel_case(s): return camel_re.sub(lambda m: m.group(2).upper(), s) -# Concrete types. +# Kinds of operands. # -# Instances (i8, i32, ...) are provided in the cretonne.types module. +# Each instruction has an opcode and a number of operands. The opcode +# determines the instruction format, and the format determines the number of +# operands and the kind of each operand. +class OperandKind(object): + """ + The kind of an operand. + + An instance of the `OperandKind` class corresponds to a kind of operand. + Each operand kind has a corresponding type in the Rust representation of an + instruction. + """ + + def __init__(self, name, doc): + self.name = name + self.__doc__ = doc + # The camel-cased name of an operand kind is also the Rust type used to + # represent it. + self.camel_name = camel_case(name) + + def __str__(self): + return self.name + + def __repr__(self): + return 'OperandKind({})'.format(self.name) -class Type(object): - """A concrete value type.""" +#: An SSA value operand. This is a value defined by another instruction. +value = OperandKind( + 'value', """ + An SSA value defined by another instruction. + + This kind of operand can represent any SSA value type, but the + instruction format may restrict the valid value types for a given + operand. + """) + + +# Instances of immediate operand types are provided in the cretonne.immediates +# module. +class ImmediateKind(OperandKind): + """ + The type of an immediate instruction operand. + """ + + def __init__(self, name, doc): + self.name = name + self.__doc__ = doc + + def __repr__(self): + return 'ImmediateKind({})'.format(self.name) + + def operand_kind(self): + """ + An `ImmediateKind` instance can be used directly as the type of an + `Operand` when defining an instruction. + """ + return self + + +# ValueType instances (i8, i32, ...) are provided in the cretonne.types module. +class ValueType(object): + """ + A concrete SSA value type. + + All SSA values have a type that is described by an instance of `ValueType` + or one of its subclasses. + """ def __init__(self, name, membytes, doc): self.name = name @@ -32,8 +94,15 @@ class Type(object): def __str__(self): return self.name + def operand_kind(self): + """ + When a `ValueType` object is used to describe the type of an `Operand` + in an instruction definition, the kind of that operand is an SSA value. + """ + return value -class ScalarType(Type): + +class ScalarType(ValueType): """ A concrete scalar (not vector) type. @@ -62,7 +131,7 @@ class ScalarType(Type): return v -class VectorType(Type): +class VectorType(ValueType): """ A concrete SIMD vector type. @@ -144,27 +213,12 @@ class TypeVar(object): self.name = name self.__doc__ = doc - -# Immediate operands. -# -# Instances of immediate operand types are provided in the cretonne.immediates -# module. - - -class ImmediateType(object): - """ - The type of an immediate instruction operand. - """ - - def __init__(self, name, doc): - self.name = name - self.__doc__ = doc - - def __str__(self): - return self.name - - def __repr__(self): - return 'ImmediateType({})'.format(self.name) + def operand_kind(self): + """ + When a `TypeVar` object is used to describe the type of an `Operand` + in an instruction definition, the kind of that operand is an SSA value. + """ + return value # Defining instructions. @@ -225,14 +279,14 @@ class Operand(object): An instruction operand can be either an *immediate* or an *SSA value*. The type of the operand is one of: - 1. A :py:class:`Type` instance indicates an SSA value operand with a + 1. A :py:class:`ValueType` instance indicates an SSA value operand with a concrete type. 2. A :py:class:`TypeVar` instance indicates an SSA value operand, and the instruction is polymorphic over the possible concrete types that the type variable can assume. - 3. An :py:class:`ImmediateType` instance indicates an immediate operand + 3. An :py:class:`ImmediateKind` instance indicates an immediate operand whose value is encoded in the instruction itself rather than being passed as an SSA value. @@ -241,6 +295,7 @@ class Operand(object): self.name = name self.typ = typ self.__doc__ = doc + self.kind = typ.operand_kind() def get_doc(self): if self.__doc__: @@ -251,7 +306,7 @@ class Operand(object): class Instruction(object): """ - An instruction. + An instruction description. The operands to the instruction are specified as two tuples: ``ins`` and ``outs``. Since the Python singleton tuple syntax is a bit awkward, it is diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 033be2c148..412e7f3abd 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -277,10 +277,11 @@ ishl = Instruction( When shifting a B-bits integer type, this instruction computes: .. math:: - s &:= y \pmod B, \\ - a &:= x \cdot 2^s \pmod{2^B}. + s &:= y \pmod B, \\ + a &:= x \cdot 2^s \pmod{2^B}. .. todo:: Add ``ishl_imm`` variant with an immediate ``y``. + """, ins=(x, y), outs=a) @@ -295,8 +296,8 @@ ushr = Instruction( When shifting a B-bits integer type, this instruction computes: .. math:: - s &:= y \pmod B, \\ - a &:= \lfloor x \cdot 2^{-s} \rfloor. + s &:= y \pmod B, \\ + a &:= \lfloor x \cdot 2^{-s} \rfloor. .. todo:: Add ``ushr_imm`` variant with an immediate ``y``. """, diff --git a/meta/cretonne/immediates.py b/meta/cretonne/immediates.py index 2735cd6be1..af7bd72feb 100644 --- a/meta/cretonne/immediates.py +++ b/meta/cretonne/immediates.py @@ -1,25 +1,25 @@ """ -The cretonne.immdiates module predefines all the Cretonne immediate operand +The cretonne.immediates module predefines all the Cretonne immediate operand types. """ -from . import ImmediateType +from . import ImmediateKind #: A 64-bit immediate integer operand. #: #: This type of immediate integer can interact with SSA values with any #: :py:class:`cretonne.IntType` type. -imm64 = ImmediateType('imm64', 'A 64-bit immediate integer.') +imm64 = ImmediateKind('imm64', 'A 64-bit immediate integer.') #: A 32-bit immediate floating point operand. #: #: IEEE 754-2008 binary32 interchange format. -ieee32 = ImmediateType('ieee32', 'A 32-bit immediate floating point number.') +ieee32 = ImmediateKind('ieee32', 'A 32-bit immediate floating point number.') #: A 64-bit immediate floating point operand. #: #: IEEE 754-2008 binary64 interchange format. -ieee64 = ImmediateType('ieee64', 'A 64-bit immediate floating point number.') +ieee64 = ImmediateKind('ieee64', 'A 64-bit immediate floating point number.') #: A large SIMD vector constant. -immvector = ImmediateType('immvector', 'An immediate SIMD vector.') +immvector = ImmediateKind('immvector', 'An immediate SIMD vector.') From af535acdc6451a4e46b06b482e64a861800d4b05 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 12 May 2016 17:28:01 -0700 Subject: [PATCH 0073/3084] Add an InstructionFormat class to the meta language. Define all known instruction formats in the cretonne.formats module. --- meta/cretonne/__init__.py | 80 +++++++++++++++++++++++++++++++++++++++ meta/cretonne/formats.py | 25 ++++++++++++ meta/gen_instr.py | 4 +- 3 files changed, 108 insertions(+), 1 deletion(-) create mode 100644 meta/cretonne/formats.py diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 26a3664c64..36f96ef6f5 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -6,6 +6,7 @@ instructions. """ import re +import importlib camel_re = re.compile('(^|_)([a-z])') @@ -304,6 +305,80 @@ class Operand(object): return self.typ.__doc__ +class InstructionFormat(object): + """ + An instruction format. + + Every instruction opcode has a corresponding instruction format which + determines the number of operands and their kinds. Instruction formats are + identified structurally, i.e., the format of an instruction is derived from + the kinds of operands used in its declaration. + + Most instruction formats produce a single result, or no result at all. If + an instruction can produce more than one result, the `multiple_results` + flag must be set on its format. All results are of the `value` kind, and + the instruction format does not keep track of how many results are + produced. Some instructions, like `call`, may have a variable number of + results. + + All instruction formats must be predefined in the + :py:mod:`cretonne.formats` module. + + :param kinds: List of `OperandKind` objects describing the operands. + :param name: Instruction format name in CamelCase. This is used as a Rust + variant name in both the `InstructionData` and `InstructionFormat` + enums. + :param multiple_results: Set to `True` if this instruction format allows + more than one result to be produced. + """ + + # Map (multiple_results, kind, kind, ...) -> InstructionFormat + _registry = dict() + + def __init__(self, *kinds, **kwargs): + self.name = kwargs.get('name', None) + self.kinds = kinds + self.multiple_results = kwargs.get('multiple_results', False) + # Compute a signature for the global registry. + sig = (self.multiple_results,) + kinds + if sig in InstructionFormat._registry: + raise RuntimeError( + "Format '{}' has the same signature as existing format '{}'" + .format(self.name, InstructionFormat._registry[sig])) + InstructionFormat._registry[sig] = self + + @staticmethod + def lookup(ins, outs): + """ + Find an existing instruction format that matches the given lists of + instruction inputs and outputs. + + The `ins` and `outs` arguments correspond to the + :py:class:`Instruction` arguments of the same name, except they must be + tuples of :py:`Operand` objects. + """ + multiple_results = len(outs) > 1 + sig = (multiple_results,) + tuple(op.kind for op in ins) + if sig not in InstructionFormat._registry: + raise RuntimeError( + "No instruction format matches ins = ({}){}".format( + ", ".join(map(str, sig[1:])), + "[multiple results]" if multiple_results else "")) + return InstructionFormat._registry[sig] + + @staticmethod + def extract_names(globs): + """ + Given a dict mapping name -> object as returned by `globals()`, find + all the InstructionFormat objects and set their name from the dict key. + This is used to name a bunch of global variables in a module. + """ + for name, obj in globs.iteritems(): + if isinstance(obj, InstructionFormat): + assert obj.name is None + obj.name = name + + class Instruction(object): """ An instruction description. @@ -327,6 +402,7 @@ class Instruction(object): self.__doc__ = doc self.ins = self._to_operand_tuple(ins) self.outs = self._to_operand_tuple(outs) + self.format = InstructionFormat.lookup(self.ins, self.outs) InstructionGroup.append(self) @staticmethod @@ -359,3 +435,7 @@ class Target(object): def __init__(self, name, instrution_groups): self.name = name self.instruction_groups = instrution_groups + +# Import the fixed instruction formats now so they can be added to the +# registry. +importlib.import_module('cretonne.formats') diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py new file mode 100644 index 0000000000..6f34214db9 --- /dev/null +++ b/meta/cretonne/formats.py @@ -0,0 +1,25 @@ +""" +The cretonne.formats defines all instruction formats. + +Every instruction format has a corresponding `InstructionData` variant in the +Rust representation of cretonne IL, so all instruction formats must be defined +in this module. +""" + + +from . import InstructionFormat, value +from immediates import imm64, ieee32, ieee64, immvector + +Unary = InstructionFormat(value) +UnaryImm = InstructionFormat(imm64) +UnaryIeee32 = InstructionFormat(ieee32) +UnaryIeee64 = InstructionFormat(ieee64) +UnaryImmVector = InstructionFormat(immvector) + +Binary = InstructionFormat(value, value) +BinaryImm = InstructionFormat(value, imm64) +BinaryImmRev = InstructionFormat(imm64, value) + + +# Finally extract the names of global variables in this module. +InstructionFormat.extract_names(globals()) diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 93b02bc369..70ae0ee8e7 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -36,7 +36,9 @@ def gen_opcodes(groups, out_dir): if prefix: prefix = prefix + ' = ' suffix = ', '.join(o.name for o in i.ins) - fmt.doc_comment('`{}{} {}`.'.format(prefix, i.name, suffix)) + fmt.doc_comment( + '`{}{} {}`. ({})' + .format(prefix, i.name, suffix, i.format.name)) # Enum variant itself. fmt.line(i.camel_name + ',') From 4109b9fe13304fd8b062f70bba1d2718536cb6c8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 13 May 2016 10:00:38 -0700 Subject: [PATCH 0074/3084] Generate recursive meta language dependencies. Cargo doesn't scan a directory for changed dependencies recursively, so do that as part of the build.py script. --- cranelift/src/libcretonne/build.rs | 3 --- meta/build.py | 2 ++ meta/gen_build_deps.py | 36 ++++++++++++++++++++++++++++++ 3 files changed, 38 insertions(+), 3 deletions(-) create mode 100644 meta/gen_build_deps.py diff --git a/cranelift/src/libcretonne/build.rs b/cranelift/src/libcretonne/build.rs index 523ddb3548..c275e33e53 100644 --- a/cranelift/src/libcretonne/build.rs +++ b/cranelift/src/libcretonne/build.rs @@ -32,9 +32,6 @@ fn main() { let meta_dir = top_dir.join("meta"); let build_script = meta_dir.join("build.py"); - // Let Cargo known that this script should be rerun if anything changes in the meta directory. - println!("cargo:rerun-if-changed={}", meta_dir.display()); - // Launch build script with Python. We'll just find python in the path. let status = process::Command::new("python") .current_dir(top_dir) diff --git a/meta/build.py b/meta/build.py index 76673fa62b..24d9f286ee 100644 --- a/meta/build.py +++ b/meta/build.py @@ -5,6 +5,7 @@ import argparse import target import gen_instr +import gen_build_deps parser = argparse.ArgumentParser(description='Generate sources for Cretonne.') parser.add_argument('--out-dir', help='set output directory') @@ -15,3 +16,4 @@ out_dir = args.out_dir targets = target.all_targets() gen_instr.generate(targets, out_dir) +gen_build_deps.generate() diff --git a/meta/gen_build_deps.py b/meta/gen_build_deps.py new file mode 100644 index 0000000000..4d0c0cb260 --- /dev/null +++ b/meta/gen_build_deps.py @@ -0,0 +1,36 @@ +""" +Generate build dependencies for Cargo. + +The `build.py` script is invoked by cargo when building libcretonne to +generate Rust code from the instruction descriptions. Cargo needs to know when +it is necessary to rerun the build script. + +If the build script outputs lines of the form: + + cargo:rerun-if-changed=/path/to/file + +cargo will rerun the build script when those files have changed since the last +build. +""" + +import os +from os.path import dirname, abspath, join + + +def source_files(top): + """ + Recursively find all interesting source files and directories in the + directory tree starting at top. Yield a path to each file. + """ + for (dirpath, dirnames, filenames) in os.walk(top): + yield dirpath + for f in filenames: + if f.endswith('.py'): + yield join(dirpath, f) + + +def generate(): + print "Dependencies from meta language directory:" + meta = dirname(abspath(__file__)) + for path in source_files(meta): + print "cargo:rerun-if-changed=" + path From 593b7bbd5109f48f70399dfaff1c6a980a018f35 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 13 May 2016 11:54:05 -0700 Subject: [PATCH 0075/3084] Generate an InstructionFormat enum. This is a no-payload enum which will have the same variants as InstructionData. This makes it possible to talk about the format of an instruction without actually creating an InstructionData instance. --- cranelift/src/libcretonne/immediates.rs | 12 ++++++++ meta/cretonne/__init__.py | 4 +++ meta/gen_instr.py | 41 ++++++++++++++++++++++--- meta/srcgen.py | 7 +++-- 4 files changed, 57 insertions(+), 7 deletions(-) diff --git a/cranelift/src/libcretonne/immediates.rs b/cranelift/src/libcretonne/immediates.rs index 619d790bfe..135d3c6b65 100644 --- a/cranelift/src/libcretonne/immediates.rs +++ b/cranelift/src/libcretonne/immediates.rs @@ -23,6 +23,17 @@ impl Display for Opcode { } } +impl Opcode { + /// Get the instruction format for this opcode. + pub fn format(self) -> Option { + if self == Opcode::NotAnOpcode { + None + } else { + Some(OPCODE_FORMAT[self as usize - 1]) + } + } +} + // A primitive hash function for matching opcodes. // Must match `meta/constant_hash.py`. fn simple_hash(s: &str) -> u32 { @@ -491,6 +502,7 @@ mod tests { assert!(x != y); y = Opcode::Iadd; assert_eq!(x, y); + assert_eq!(x.format(), Some(InstructionFormat::Binary)); assert_eq!(format!("{:?}", Opcode::IaddImm), "IaddImm"); assert_eq!(Opcode::IaddImm.to_string(), "iadd_imm"); diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 36f96ef6f5..4b8267d2d1 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -335,6 +335,9 @@ class InstructionFormat(object): # Map (multiple_results, kind, kind, ...) -> InstructionFormat _registry = dict() + # All existing formats. + all_formats = list() + def __init__(self, *kinds, **kwargs): self.name = kwargs.get('name', None) self.kinds = kinds @@ -346,6 +349,7 @@ class InstructionFormat(object): "Format '{}' has the same signature as existing format '{}'" .format(self.name, InstructionFormat._registry[sig])) InstructionFormat._registry[sig] = self + InstructionFormat.all_formats.append(self) @staticmethod def lookup(ins, outs): diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 70ae0ee8e7..8c5ad8a9ce 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -4,6 +4,22 @@ Generate sources with instruction info. import srcgen import constant_hash +import cretonne + + +def gen_formats(fmt): + """Generate an instruction format enumeration""" + + fmt.doc_comment('An instruction format') + fmt.doc_comment('') + fmt.doc_comment('Every opcode has a corresponding instruction format') + fmt.doc_comment('which is represented by both the `InstructionFormat`') + fmt.doc_comment('and the `InstructionData` enums.') + fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug)]') + with fmt.indented('pub enum InstructionFormat {', '}'): + for f in cretonne.InstructionFormat.all_formats: + fmt.line(f.name + ',') + fmt.line() def collect_instr_groups(targets): @@ -17,9 +33,8 @@ def collect_instr_groups(targets): return groups -def gen_opcodes(groups, out_dir): +def gen_opcodes(groups, fmt): """Generate opcode enumerations.""" - fmt = srcgen.Formatter() fmt.doc_comment('An instruction opcode.') fmt.doc_comment('') @@ -41,6 +56,18 @@ def gen_opcodes(groups, out_dir): .format(prefix, i.name, suffix, i.format.name)) # Enum variant itself. fmt.line(i.camel_name + ',') + fmt.line() + + # Generate a private opcode_format table. + with fmt.indented( + 'const OPCODE_FORMAT: [InstructionFormat; {}] = [' + .format(len(instrs)), + '];'): + for i in instrs: + fmt.format( + 'InstructionFormat::{}, // {}', + i.format.name, i.name) + fmt.line() # Generate a private opcode_name function. with fmt.indented('fn opcode_name(opc: Opcode) -> &\'static str {', '}'): @@ -48,6 +75,7 @@ def gen_opcodes(groups, out_dir): fmt.line('Opcode::NotAnOpcode => "",') for i in instrs: fmt.format('Opcode::{} => "{}",', i.camel_name, i.name) + fmt.line() # Generate an opcode hash table for looking up opcodes by name. hash_table = constant_hash.compute_quadratic( @@ -61,10 +89,13 @@ def gen_opcodes(groups, out_dir): fmt.line('Opcode::NotAnOpcode,') else: fmt.format('Opcode::{},', i.camel_name) - - fmt.update_file('opcodes.rs', out_dir) + fmt.line() def generate(targets, out_dir): groups = collect_instr_groups(targets) - gen_opcodes(groups, out_dir) + # opcodes.rs + fmt = srcgen.Formatter() + gen_formats(fmt) + gen_opcodes(groups, fmt) + fmt.update_file('opcodes.rs', out_dir) diff --git a/meta/srcgen.py b/meta/srcgen.py index 7895b0ea38..1ebd745286 100644 --- a/meta/srcgen.py +++ b/meta/srcgen.py @@ -49,9 +49,12 @@ class Formatter(object): assert self.indent != '', 'Already at top level indentation' self.indent = self.indent[0:-self.shiftwidth] - def line(self, s): + def line(self, s=None): """And an indented line.""" - self.lines.append('{}{}\n'.format(self.indent, s)) + if s: + self.lines.append('{}{}\n'.format(self.indent, s)) + else: + self.lines.append('\n') def writelines(self, f=None): """Write all lines to `f`.""" From 62ecbc744817d99d60a35787ad1d5a7644fc7e64 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 13 May 2016 13:32:20 -0700 Subject: [PATCH 0076/3084] Break entity references and instruction info out into new modules. Avoid gathering too much code in repr.rs. The `entities` module contains entity reference types, and the `instructions` module contains instruction opcodes and formats. --- cranelift/src/libcretonne/entities.rs | 185 +++++++++++++++ cranelift/src/libcretonne/immediates.rs | 89 ------- cranelift/src/libcretonne/instructions.rs | 229 ++++++++++++++++++ cranelift/src/libcretonne/lib.rs | 2 + cranelift/src/libcretonne/repr.rs | 269 +--------------------- cranelift/src/libreader/parser.rs | 3 +- 6 files changed, 424 insertions(+), 353 deletions(-) create mode 100644 cranelift/src/libcretonne/entities.rs create mode 100644 cranelift/src/libcretonne/instructions.rs diff --git a/cranelift/src/libcretonne/entities.rs b/cranelift/src/libcretonne/entities.rs new file mode 100644 index 0000000000..bdc76d8040 --- /dev/null +++ b/cranelift/src/libcretonne/entities.rs @@ -0,0 +1,185 @@ +//! IL entity references. +//! +//! Instructions in Cretonne IL need to reference other entities in the function. This can be other +//! parts of the function like extended basic blocks or stack slots, or it can be external entities +//! that are declared in the function preamble in the text format. +//! +//! These entity references in instruction operands are not implemented as Rust references both +//! because Rust's ownership and mutability rules make it difficult, and because 64-bit pointers +//! take up a lot of space, and we want a compact in-memory representation. Instead, entity +//! references are structs wrapping a `u32` index into a table in the `Function` main data +//! structure. There is a separate index type for each entity type, so we don't lose type safety. +//! +//! The `entities` module defines public types for the entity references along with constants +//! representing an invalid reference. We prefer to use `Option` whenever possible, but +//! unfortunately that type is twice as large as the 32-bit index type on its own. Thus, compact +//! data structures use the sentinen constant, while function arguments and return values prefer +//! the more Rust-like `Option` variant. +//! +//! The entity references all implement the `Display` trait in a way that matches the textual IL +//! format. + +use std::default::Default; +use std::fmt::{self, Display, Formatter, Write}; +use std::u32; + +/// An opaque reference to an extended basic block in a function. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct Ebb(u32); + +impl Ebb { + pub fn new(index: usize) -> Ebb { + assert!(index < (u32::MAX as usize)); + Ebb(index as u32) + } + + pub fn index(&self) -> usize { + self.0 as usize + } +} + +/// Display an `Ebb` reference as "ebb12". +impl Display for Ebb { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write!(fmt, "ebb{}", self.0) + } +} + +/// A guaranteed invalid EBB reference. +pub const NO_EBB: Ebb = Ebb(u32::MAX); + +impl Default for Ebb { + fn default() -> Ebb { + NO_EBB + } +} + + +/// An opaque reference to an instruction in a function. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct Inst(u32); + +impl Inst { + pub fn new(index: usize) -> Inst { + assert!(index < (u32::MAX as usize)); + Inst(index as u32) + } + + pub fn index(&self) -> usize { + self.0 as usize + } +} + +/// Display an `Inst` reference as "inst7". +impl Display for Inst { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write!(fmt, "inst{}", self.0) + } +} + +/// A guaranteed invalid instruction reference. +pub const NO_INST: Inst = Inst(u32::MAX); + +impl Default for Inst { + fn default() -> Inst { + NO_INST + } +} + + +/// An opaque reference to an SSA value. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct Value(u32); + +// Value references can either reference an instruction directly, or they can refer to the extended +// value table. +pub enum ExpandedValue { + // This is the first value produced by the referenced instruction. + Direct(Inst), + + // This value is described in the extended value table. + Table(usize), + + // This is NO_VALUE. + None, +} + +impl Value { + pub fn new_direct(i: Inst) -> Value { + let encoding = i.index() * 2; + assert!(encoding < u32::MAX as usize); + Value(encoding as u32) + } + + pub fn new_table(index: usize) -> Value { + let encoding = index * 2 + 1; + assert!(encoding < u32::MAX as usize); + Value(encoding as u32) + } + + // Expand the internal representation into something useful. + pub fn expand(&self) -> ExpandedValue { + use self::ExpandedValue::*; + if *self == NO_VALUE { + return None; + } + let index = (self.0 / 2) as usize; + if self.0 % 2 == 0 { + Direct(Inst::new(index)) + } else { + Table(index) + } + } +} + +/// Display a `Value` reference as "v7" or "v2x". +impl Display for Value { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + use self::ExpandedValue::*; + match self.expand() { + Direct(i) => write!(fmt, "v{}", i.0), + Table(i) => write!(fmt, "vx{}", i), + None => write!(fmt, "NO_VALUE"), + } + } +} + +/// A guaranteed invalid value reference. +pub const NO_VALUE: Value = Value(u32::MAX); + +impl Default for Value { + fn default() -> Value { + NO_VALUE + } +} + +/// An opaque reference to a stack slot. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct StackSlot(u32); + +impl StackSlot { + pub fn new(index: usize) -> StackSlot { + assert!(index < (u32::MAX as usize)); + StackSlot(index as u32) + } + + pub fn index(&self) -> usize { + self.0 as usize + } +} + +/// Display a `StackSlot` reference as "ss12". +impl Display for StackSlot { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write!(fmt, "ss{}", self.0) + } +} + +/// A guaranteed invalid stack slot reference. +pub const NO_STACK_SLOT: StackSlot = StackSlot(u32::MAX); + +impl Default for StackSlot { + fn default() -> StackSlot { + NO_STACK_SLOT + } +} diff --git a/cranelift/src/libcretonne/immediates.rs b/cranelift/src/libcretonne/immediates.rs index 135d3c6b65..a329d74f70 100644 --- a/cranelift/src/libcretonne/immediates.rs +++ b/cranelift/src/libcretonne/immediates.rs @@ -9,73 +9,6 @@ use std::fmt::{self, Display, Formatter}; use std::mem; use std::str::FromStr; -// Include code generated by `meta/gen_instr.py`. This file contains: -// -// - The `pub enum Opcode` definition with all known opcodes, -// - The private `fn opcode_name(Opcode) -> &'static str` function, and -// - The hash table `const OPCODE_HASH_TABLE: [Opcode; N]`. -// -include!(concat!(env!("OUT_DIR"), "/opcodes.rs")); - -impl Display for Opcode { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - write!(f, "{}", opcode_name(*self)) - } -} - -impl Opcode { - /// Get the instruction format for this opcode. - pub fn format(self) -> Option { - if self == Opcode::NotAnOpcode { - None - } else { - Some(OPCODE_FORMAT[self as usize - 1]) - } - } -} - -// A primitive hash function for matching opcodes. -// Must match `meta/constant_hash.py`. -fn simple_hash(s: &str) -> u32 { - let mut h: u32 = 5381; - for c in s.chars() { - h = (h ^ c as u32).wrapping_add(h.rotate_right(6)); - } - h -} - -impl FromStr for Opcode { - type Err = &'static str; - - /// Parse an Opcode name from a string. - fn from_str(s: &str) -> Result { - let tlen = OPCODE_HASH_TABLE.len(); - assert!(tlen.is_power_of_two()); - let mut idx = simple_hash(s) as usize; - let mut step: usize = 0; - loop { - idx = idx % tlen; - let entry = OPCODE_HASH_TABLE[idx]; - - if entry == Opcode::NotAnOpcode { - return Err("Unknown opcode"); - } - - if *opcode_name(entry) == *s { - return Ok(entry); - } - - // Quadratic probing. - step += 1; - // When `tlen` is a power of two, it can be proven that idx will visit all entries. - // This means that this loop will always terminate if the hash table has even one - // unused entry. - assert!(step < tlen); - idx += step; - } - } -} - /// 64-bit immediate integer operand. /// #[derive(Copy, Clone, PartialEq, Eq, Debug)] @@ -494,27 +427,6 @@ mod tests { use std::str::FromStr; use std::fmt::Display; - #[test] - fn opcodes() { - let x = Opcode::Iadd; - let mut y = Opcode::Isub; - - assert!(x != y); - y = Opcode::Iadd; - assert_eq!(x, y); - assert_eq!(x.format(), Some(InstructionFormat::Binary)); - - assert_eq!(format!("{:?}", Opcode::IaddImm), "IaddImm"); - assert_eq!(Opcode::IaddImm.to_string(), "iadd_imm"); - - // Check the matcher. - assert_eq!("iadd".parse::(), Ok(Opcode::Iadd)); - assert_eq!("iadd_imm".parse::(), Ok(Opcode::IaddImm)); - assert_eq!("iadd\0".parse::(), Err("Unknown opcode")); - assert_eq!("".parse::(), Err("Unknown opcode")); - assert_eq!("\0".parse::(), Err("Unknown opcode")); - } - #[test] fn format_imm64() { assert_eq!(Imm64(0).to_string(), "0"); @@ -791,5 +703,4 @@ mod tests { parse_ok::("sNaN:0x4000000000001", "sNaN:0x4000000000001"); parse_err::("sNaN:0x8000000000001", "Invalid sNaN payload"); } - } diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs new file mode 100644 index 0000000000..025e6ebb23 --- /dev/null +++ b/cranelift/src/libcretonne/instructions.rs @@ -0,0 +1,229 @@ +//! Instruction formats and opcodes. +//! +//! The `instructions` module contains definitions for instruction formats, opcodes, and the +//! in-memory representation of IL instructions. +//! +//! A large part of this module is auto-generated from the instruction descriptions in the meta +//! directory. + +use std::fmt::{self, Display, Formatter}; +use std::str::FromStr; + +use entities::*; +use immediates::*; +use types::Type; + +// Include code generated by `meta/gen_instr.py`. This file contains: +// +// - The `pub enum InstructionFormat` enum with all the instruction formats. +// - The `pub enum Opcode` definition with all known opcodes, +// - The `const OPCODE_FORMAT: [InstructionFormat; N]` table. +// - The private `fn opcode_name(Opcode) -> &'static str` function, and +// - The hash table `const OPCODE_HASH_TABLE: [Opcode; N]`. +// +include!(concat!(env!("OUT_DIR"), "/opcodes.rs")); + +impl Display for Opcode { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + write!(f, "{}", opcode_name(*self)) + } +} + +impl Opcode { + /// Get the instruction format for this opcode. + pub fn format(self) -> Option { + if self == Opcode::NotAnOpcode { + None + } else { + Some(OPCODE_FORMAT[self as usize - 1]) + } + } +} + +// A primitive hash function for matching opcodes. +// Must match `meta/constant_hash.py`. +fn simple_hash(s: &str) -> u32 { + let mut h: u32 = 5381; + for c in s.chars() { + h = (h ^ c as u32).wrapping_add(h.rotate_right(6)); + } + h +} + +// This trait really belongs in libreader where it is used by the .cton file parser, but since it +// critically depends on the `opcode_name()` function which is needed here anyway, it lives in this +// module. This also saves us from runing the build script twice to generate code for the two +// separate crates. +impl FromStr for Opcode { + type Err = &'static str; + + /// Parse an Opcode name from a string. + fn from_str(s: &str) -> Result { + let tlen = OPCODE_HASH_TABLE.len(); + assert!(tlen.is_power_of_two()); + let mut idx = simple_hash(s) as usize; + let mut step: usize = 0; + loop { + idx = idx % tlen; + let entry = OPCODE_HASH_TABLE[idx]; + + if entry == Opcode::NotAnOpcode { + return Err("Unknown opcode"); + } + + if *opcode_name(entry) == *s { + return Ok(entry); + } + + // Quadratic probing. + step += 1; + // When `tlen` is a power of two, it can be proven that idx will visit all entries. + // This means that this loop will always terminate if the hash table has even one + // unused entry. + assert!(step < tlen); + idx += step; + } + } +} + +/// Contents on an instruction. +/// +/// Every variant must contain `opcode` and `ty` fields. An instruction that doesn't produce a +/// value should have its `ty` field set to `VOID`. The size of `InstructionData` should be kept at +/// 16 bytes on 64-bit architectures. If more space is needed to represent an instruction, use a +/// `Box` to store the additional information out of line. +#[derive(Debug)] +pub enum InstructionData { + Nullary { + opcode: Opcode, + ty: Type, + }, + Unary { + opcode: Opcode, + ty: Type, + arg: Value, + }, + UnaryImm { + opcode: Opcode, + ty: Type, + imm: Imm64, + }, + Binary { + opcode: Opcode, + ty: Type, + args: [Value; 2], + }, + BinaryImm { + opcode: Opcode, + ty: Type, + arg: Value, + imm: Imm64, + }, + Call { + opcode: Opcode, + ty: Type, + data: Box, + }, +} + +/// Payload of a call instruction. +#[derive(Debug)] +pub struct CallData { + /// Second result value for a call producing multiple return values. + second_result: Value, + + // Dynamically sized array containing call argument values. + arguments: Vec, +} + + +impl InstructionData { + /// Create data for a call instruction. + pub fn call(opc: Opcode, return_type: Type) -> InstructionData { + InstructionData::Call { + opcode: opc, + ty: return_type, + data: Box::new(CallData { + second_result: NO_VALUE, + arguments: Vec::new(), + }), + } + } + + /// Get the opcode of this instruction. + pub fn opcode(&self) -> Opcode { + use self::InstructionData::*; + match *self { + Nullary { opcode, .. } => opcode, + Unary { opcode, .. } => opcode, + UnaryImm { opcode, .. } => opcode, + Binary { opcode, .. } => opcode, + BinaryImm { opcode, .. } => opcode, + Call { opcode, .. } => opcode, + } + } + + /// Type of the first result. + pub fn first_type(&self) -> Type { + use self::InstructionData::*; + match *self { + Nullary { ty, .. } => ty, + Unary { ty, .. } => ty, + UnaryImm { ty, .. } => ty, + Binary { ty, .. } => ty, + BinaryImm { ty, .. } => ty, + Call { ty, .. } => ty, + } + } + + /// Second result value, if any. + pub fn second_result(&self) -> Option { + use self::InstructionData::*; + match *self { + Nullary { .. } => None, + Unary { .. } => None, + UnaryImm { .. } => None, + Binary { .. } => None, + BinaryImm { .. } => None, + Call { ref data, .. } => Some(data.second_result), + } + } + + pub fn second_result_mut<'a>(&'a mut self) -> Option<&'a mut Value> { + use self::InstructionData::*; + match *self { + Nullary { .. } => None, + Unary { .. } => None, + UnaryImm { .. } => None, + Binary { .. } => None, + BinaryImm { .. } => None, + Call { ref mut data, .. } => Some(&mut data.second_result), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn opcodes() { + let x = Opcode::Iadd; + let mut y = Opcode::Isub; + + assert!(x != y); + y = Opcode::Iadd; + assert_eq!(x, y); + assert_eq!(x.format(), Some(InstructionFormat::Binary)); + + assert_eq!(format!("{:?}", Opcode::IaddImm), "IaddImm"); + assert_eq!(Opcode::IaddImm.to_string(), "iadd_imm"); + + // Check the matcher. + assert_eq!("iadd".parse::(), Ok(Opcode::Iadd)); + assert_eq!("iadd_imm".parse::(), Ok(Opcode::IaddImm)); + assert_eq!("iadd\0".parse::(), Err("Unknown opcode")); + assert_eq!("".parse::(), Err("Unknown opcode")); + assert_eq!("\0".parse::(), Err("Unknown opcode")); + } +} diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index f47b049156..248c6bb37f 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -7,5 +7,7 @@ pub mod types; pub mod immediates; +pub mod entities; +pub mod instructions; pub mod repr; pub mod write; diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index 534fd1a7cb..5064a7939f 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -2,11 +2,10 @@ //! Representation of Cretonne IL functions. use types::{Type, FunctionName, Signature}; -use immediates::*; -use std::default::Default; -use std::fmt::{self, Display, Formatter, Write}; +use entities::*; +use instructions::*; +use std::fmt::{self, Display, Formatter}; use std::ops::Index; -use std::u32; // ====--------------------------------------------------------------------------------------====// // @@ -14,34 +13,6 @@ use std::u32; // // ====--------------------------------------------------------------------------------------====// -/// An opaque reference to an extended basic block in a function. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] -pub struct Ebb(u32); - -/// A guaranteed invalid EBB reference. -pub const NO_EBB: Ebb = Ebb(u32::MAX); - -/// An opaque reference to an instruction in a function. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] -pub struct Inst(u32); - -/// A guaranteed invalid instruction reference. -pub const NO_INST: Inst = Inst(u32::MAX); - -/// An opaque reference to an SSA value. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] -pub struct Value(u32); - -/// A guaranteed invalid value reference. -pub const NO_VALUE: Value = Value(u32::MAX); - -/// An opaque reference to a stack slot. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] -pub struct StackSlot(u32); - -/// A guaranteed invalid stack slot reference. -pub const NO_STACK_SLOT: StackSlot = StackSlot(u32::MAX); - /// A function. /// /// The `Function` struct owns all of its instructions and extended basic blocks, and it works as a @@ -98,81 +69,12 @@ pub struct EbbData { last_arg: Value, } -/// Contents on an instruction. -/// -/// Every variant must contain `opcode` and `ty` fields. An instruction that doesn't produce a -/// value should have its `ty` field set to `VOID`. The size of `InstructionData` should be kept at -/// 16 bytes on 64-bit architectures. If more space is needed to represent an instruction, use a -/// `Box` to store the additional information out of line. -#[derive(Debug)] -pub enum InstructionData { - Nullary { - opcode: Opcode, - ty: Type, - }, - Unary { - opcode: Opcode, - ty: Type, - arg: Value, - }, - UnaryImm { - opcode: Opcode, - ty: Type, - imm: Imm64, - }, - Binary { - opcode: Opcode, - ty: Type, - args: [Value; 2], - }, - BinaryImm { - opcode: Opcode, - ty: Type, - arg: Value, - imm: Imm64, - }, - Call { - opcode: Opcode, - ty: Type, - data: Box, - }, -} - -/// Payload of a call instruction. -#[derive(Debug)] -pub struct CallData { - /// Second result value for a call producing multiple return values. - second_result: Value, - - // Dynamically sized array containing call argument values. - arguments: Vec, -} - - // ====--------------------------------------------------------------------------------------====// // // Stack slot implementation. // // ====--------------------------------------------------------------------------------------====// -impl StackSlot { - fn new(index: usize) -> StackSlot { - assert!(index < (u32::MAX as usize)); - StackSlot(index as u32) - } - - pub fn index(&self) -> usize { - self.0 as usize - } -} - -/// Display a `StackSlot` reference as "ss12". -impl Display for StackSlot { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write!(fmt, "ss{}", self.0) - } -} - impl StackSlotData { /// Create a stack slot with the specified byte size. pub fn new(size: u32) -> StackSlotData { @@ -221,24 +123,6 @@ impl Iterator for StackSlotIter { // // ====--------------------------------------------------------------------------------------====// -impl Ebb { - fn new(index: usize) -> Ebb { - assert!(index < (u32::MAX as usize)); - Ebb(index as u32) - } - - pub fn index(&self) -> usize { - self.0 as usize - } -} - -/// Display an `Ebb` reference as "ebb12". -impl Display for Ebb { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write!(fmt, "ebb{}", self.0) - } -} - impl EbbData { fn new() -> EbbData { EbbData { @@ -254,24 +138,6 @@ impl EbbData { // // ====--------------------------------------------------------------------------------------====// -impl Inst { - fn new(index: usize) -> Inst { - assert!(index < (u32::MAX as usize)); - Inst(index as u32) - } - - pub fn index(&self) -> usize { - self.0 as usize - } -} - -/// Display an `Inst` reference as "inst7". -impl Display for Inst { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write!(fmt, "inst{}", self.0) - } -} - /// Allow immutable access to instructions via function indexing. impl Index for Function { type Output = InstructionData; @@ -287,65 +153,6 @@ impl Index for Function { // // ====--------------------------------------------------------------------------------------====// -// Value references can either reference an instruction directly, or they can refer to the -// extended value table. -enum ExpandedValue { - // This is the first value produced by the referenced instruction. - Direct(Inst), - - // This value is described in the extended value table. - Table(usize), - - // This is NO_VALUE. - None, -} - -impl Value { - fn new_direct(i: Inst) -> Value { - let encoding = i.index() * 2; - assert!(encoding < u32::MAX as usize); - Value(encoding as u32) - } - - fn new_table(index: usize) -> Value { - let encoding = index * 2 + 1; - assert!(encoding < u32::MAX as usize); - Value(encoding as u32) - } - - // Expand the internal representation into something useful. - fn expand(&self) -> ExpandedValue { - use self::ExpandedValue::*; - if *self == NO_VALUE { - return None; - } - let index = (self.0 / 2) as usize; - if self.0 % 2 == 0 { - Direct(Inst::new(index)) - } else { - Table(index) - } - } -} - -impl Default for Value { - fn default() -> Value { - NO_VALUE - } -} - -/// Display a `Value` reference as "v7" or "v2x". -impl Display for Value { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - use self::ExpandedValue::*; - match self.expand() { - Direct(i) => write!(fmt, "v{}", i.0), - Table(i) => write!(fmt, "vx{}", i), - None => write!(fmt, "NO_VALUE"), - } - } -} - // Most values are simply the first value produced by an instruction. // Other values have an entry in the value table. #[derive(Debug)] @@ -398,71 +205,6 @@ impl<'a> Iterator for Values<'a> { } } -impl InstructionData { - /// Create data for a call instruction. - pub fn call(opc: Opcode, return_type: Type) -> InstructionData { - InstructionData::Call { - opcode: opc, - ty: return_type, - data: Box::new(CallData { - second_result: NO_VALUE, - arguments: Vec::new(), - }), - } - } - - /// Get the opcode of this instruction. - pub fn opcode(&self) -> Opcode { - use self::InstructionData::*; - match *self { - Nullary { opcode, .. } => opcode, - Unary { opcode, .. } => opcode, - UnaryImm { opcode, .. } => opcode, - Binary { opcode, .. } => opcode, - BinaryImm { opcode, .. } => opcode, - Call { opcode, .. } => opcode, - } - } - - /// Type of the first result. - pub fn first_type(&self) -> Type { - use self::InstructionData::*; - match *self { - Nullary { ty, .. } => ty, - Unary { ty, .. } => ty, - UnaryImm { ty, .. } => ty, - Binary { ty, .. } => ty, - BinaryImm { ty, .. } => ty, - Call { ty, .. } => ty, - } - } - - /// Second result value, if any. - fn second_result(&self) -> Option { - use self::InstructionData::*; - match *self { - Nullary { .. } => None, - Unary { .. } => None, - UnaryImm { .. } => None, - Binary { .. } => None, - BinaryImm { .. } => None, - Call { ref data, .. } => Some(data.second_result), - } - } - - fn second_result_mut<'a>(&'a mut self) -> Option<&'a mut Value> { - use self::InstructionData::*; - match *self { - Nullary { .. } => None, - Unary { .. } => None, - UnaryImm { .. } => None, - Binary { .. } => None, - BinaryImm { .. } => None, - Call { ref mut data, .. } => Some(&mut data.second_result), - } - } -} - impl Function { /// Create a function with the given name and signature. pub fn with_name_signature(name: FunctionName, sig: Signature) -> Function { @@ -632,7 +374,7 @@ impl Function { /// Get the type of a value. pub fn value_type(&self, v: Value) -> Type { - use self::ExpandedValue::*; + use entities::ExpandedValue::*; use self::ValueData::*; match v.expand() { Direct(i) => self[i].first_type(), @@ -651,7 +393,8 @@ impl Function { mod tests { use super::*; use types; - use immediates::*; + use entities::*; + use instructions::*; #[test] fn make_inst() { diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index e74ae22c38..119957d7c4 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -12,7 +12,8 @@ use std::u32; use lexer::{self, Lexer, Token}; use cretonne::types::{FunctionName, Signature, ArgumentType, ArgumentExtension}; use cretonne::immediates::Imm64; -use cretonne::repr::{Function, StackSlot, StackSlotData}; +use cretonne::entities::StackSlot; +use cretonne::repr::{Function, StackSlotData}; pub use lexer::Location; From e73583638390bebec9aa6768926db019a13ca0d0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 13 May 2016 14:27:24 -0700 Subject: [PATCH 0077/3084] Synchronize InstructionFormat and InstructionData. These two enums must have identical variants. One is generated from the instruction formats in meta/cretonne/formats.py, the other defines the contents of an instruction. Emit a conversion from InstructionData to InstructionFormat which also serves to verify the correspondence. Rustc will error is the match is not complete. --- cranelift/src/libcretonne/instructions.rs | 39 ++++++++++++++++++++++- meta/cretonne/__init__.py | 11 +++++++ meta/cretonne/formats.py | 5 ++- meta/gen_instr.py | 14 ++++++++ 4 files changed, 67 insertions(+), 2 deletions(-) diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index 025e6ebb23..325033284a 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -108,6 +108,20 @@ pub enum InstructionData { ty: Type, imm: Imm64, }, + UnaryIeee32 { + opcode: Opcode, + ty: Type, + imm: Ieee32, + }, + UnaryIeee64 { + opcode: Opcode, + ty: Type, + imm: Ieee64, + }, + UnaryImmVector { + opcode: Opcode, + ty: Type, // TBD: imm: Box + }, Binary { opcode: Opcode, ty: Type, @@ -117,7 +131,14 @@ pub enum InstructionData { opcode: Opcode, ty: Type, arg: Value, - imm: Imm64, + rhs: Imm64, + }, + // Same as BinaryImm, but the imediate is the lhs operand. + BinaryImmRev { + opcode: Opcode, + ty: Type, + arg: Value, + lhs: Imm64, }, Call { opcode: Opcode, @@ -157,8 +178,12 @@ impl InstructionData { Nullary { opcode, .. } => opcode, Unary { opcode, .. } => opcode, UnaryImm { opcode, .. } => opcode, + UnaryIeee32 { opcode, .. } => opcode, + UnaryIeee64 { opcode, .. } => opcode, + UnaryImmVector { opcode, .. } => opcode, Binary { opcode, .. } => opcode, BinaryImm { opcode, .. } => opcode, + BinaryImmRev { opcode, .. } => opcode, Call { opcode, .. } => opcode, } } @@ -170,8 +195,12 @@ impl InstructionData { Nullary { ty, .. } => ty, Unary { ty, .. } => ty, UnaryImm { ty, .. } => ty, + UnaryIeee32 { ty, .. } => ty, + UnaryIeee64 { ty, .. } => ty, + UnaryImmVector { ty, .. } => ty, Binary { ty, .. } => ty, BinaryImm { ty, .. } => ty, + BinaryImmRev { ty, .. } => ty, Call { ty, .. } => ty, } } @@ -183,8 +212,12 @@ impl InstructionData { Nullary { .. } => None, Unary { .. } => None, UnaryImm { .. } => None, + UnaryIeee32 { .. } => None, + UnaryIeee64 { .. } => None, + UnaryImmVector { .. } => None, Binary { .. } => None, BinaryImm { .. } => None, + BinaryImmRev { .. } => None, Call { ref data, .. } => Some(data.second_result), } } @@ -195,8 +228,12 @@ impl InstructionData { Nullary { .. } => None, Unary { .. } => None, UnaryImm { .. } => None, + UnaryIeee32 { .. } => None, + UnaryIeee64 { .. } => None, + UnaryImmVector { .. } => None, Binary { .. } => None, BinaryImm { .. } => None, + BinaryImmRev { .. } => None, Call { ref mut data, .. } => Some(&mut data.second_result), } } diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 4b8267d2d1..cbad9185fc 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -55,6 +55,17 @@ value = OperandKind( operand. """) +#: A variable-sizes list of value operands. Use for Ebb and function call +#: arguemnts. +args = OperandKind( + 'args', """ + A variable size list of `value` operands. + + Use this to represent arguemtns passed to a function call, arguments + passed to an extended basic block, or a variable number of results + returned from an instruction. + """) + # Instances of immediate operand types are provided in the cretonne.immediates # module. diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index 6f34214db9..cb02d80998 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -7,9 +7,11 @@ in this module. """ -from . import InstructionFormat, value +from . import InstructionFormat, value, args from immediates import imm64, ieee32, ieee64, immvector +Nullary = InstructionFormat() + Unary = InstructionFormat(value) UnaryImm = InstructionFormat(imm64) UnaryIeee32 = InstructionFormat(ieee32) @@ -20,6 +22,7 @@ Binary = InstructionFormat(value, value) BinaryImm = InstructionFormat(value, imm64) BinaryImmRev = InstructionFormat(imm64, value) +Call = InstructionFormat(args, multiple_results=True) # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 8c5ad8a9ce..8fe63cb932 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -21,6 +21,20 @@ def gen_formats(fmt): fmt.line(f.name + ',') fmt.line() + # Emit a From which also serves to verify that + # InstructionFormat and InstructionData are in sync. + with fmt.indented( + "impl<'a> From<&'a InstructionData> for InstructionFormat {", '}'): + with fmt.indented( + "fn from(inst: &'a InstructionData) -> InstructionFormat {", + '}'): + with fmt.indented('match *inst {', '}'): + for f in cretonne.InstructionFormat.all_formats: + fmt.line(('InstructionData::{} {{ .. }} => ' + + 'InstructionFormat::{},') + .format(f.name, f.name)) + fmt.line() + def collect_instr_groups(targets): seen = set() From dd5c1a1a3f2dde647d7acebc524f3d62d8c575dc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 13 May 2016 17:45:57 -0700 Subject: [PATCH 0078/3084] Track instruction order in an EBB. Place instructions in a doubly linked list and point to the first and last instruction in an EBB. Provide an iterator for all the EBBs too. This doesn't reflect the layout order, but simply the order blocks were created. --- cranelift/src/libcretonne/repr.rs | 483 +++++++++++++++++++----------- 1 file changed, 312 insertions(+), 171 deletions(-) diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index 5064a7939f..a444dd0b3d 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -1,17 +1,10 @@ - //! Representation of Cretonne IL functions. use types::{Type, FunctionName, Signature}; use entities::*; use instructions::*; use std::fmt::{self, Display, Formatter}; -use std::ops::Index; - -// ====--------------------------------------------------------------------------------------====// -// -// Public data types. -// -// ====--------------------------------------------------------------------------------------====// +use std::ops::{Index, IndexMut}; /// A function. /// @@ -41,168 +34,9 @@ pub struct Function { /// Others index into this table. extended_values: Vec, - /// Return type(s). A function may return zero or more values. - pub return_types: Vec, -} - -/// Contents of a stack slot. -#[derive(Debug)] -pub struct StackSlotData { - /// Size of stack slot in bytes. - pub size: u32, -} - -/// Contents of an extended basic block. -/// -/// Arguments for an extended basic block are values that dominate everything in the EBB. All -/// branches to this EBB must provide matching arguments, and the arguments to the entry EBB must -/// match the function arguments. -#[derive(Debug)] -pub struct EbbData { - /// First argument to this EBB, or `NO_VALUE` if the block has no arguments. - /// - /// The arguments are all ValueData::Argument entries that form a linked list from `first_arg` - /// to `last_arg`. - first_arg: Value, - - /// Last argument to this EBB, or `NO_VALUE` if the block has no arguments. - last_arg: Value, -} - -// ====--------------------------------------------------------------------------------------====// -// -// Stack slot implementation. -// -// ====--------------------------------------------------------------------------------------====// - -impl StackSlotData { - /// Create a stack slot with the specified byte size. - pub fn new(size: u32) -> StackSlotData { - StackSlotData { size: size } - } -} - -impl Display for StackSlotData { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write!(fmt, "stack_slot {}", self.size) - } -} - -/// Allow immutable access to stack slots via function indexing. -impl Index for Function { - type Output = StackSlotData; - - fn index<'a>(&'a self, ss: StackSlot) -> &'a StackSlotData { - &self.stack_slots[ss.index()] - } -} - -/// Stack slot iterator visits all stack slots in a function, returning `StackSlot` references. -pub struct StackSlotIter { - cur: usize, - end: usize, -} - -impl Iterator for StackSlotIter { - type Item = StackSlot; - - fn next(&mut self) -> Option { - if self.cur < self.end { - let ss = StackSlot::new(self.cur); - self.cur += 1; - Some(ss) - } else { - None - } - } -} - -// ====--------------------------------------------------------------------------------------====// -// -// Extended basic block implementation. -// -// ====--------------------------------------------------------------------------------------====// - -impl EbbData { - fn new() -> EbbData { - EbbData { - first_arg: NO_VALUE, - last_arg: NO_VALUE, - } - } -} - -// ====--------------------------------------------------------------------------------------====// -// -// Instruction implementation. -// -// ====--------------------------------------------------------------------------------------====// - -/// Allow immutable access to instructions via function indexing. -impl Index for Function { - type Output = InstructionData; - - fn index<'a>(&'a self, inst: Inst) -> &'a InstructionData { - &self.instructions[inst.index()] - } -} - -// ====--------------------------------------------------------------------------------------====// -// -// Value implementation. -// -// ====--------------------------------------------------------------------------------------====// - -// Most values are simply the first value produced by an instruction. -// Other values have an entry in the value table. -#[derive(Debug)] -enum ValueData { - // Value is defined by an instruction, but it is not the first result. - Def { - ty: Type, - def: Inst, - next: Value, // Next result defined by `def`. - }, - - // Value is an EBB argument. - Argument { - ty: Type, - ebb: Ebb, - next: Value, // Next argument to `ebb`. - }, -} - -/// Iterate through a list of related value references, such as: -/// -/// - All results defined by an instruction. -/// - All arguments to an EBB -/// -/// A value iterator borrows a Function reference. -pub struct Values<'a> { - func: &'a Function, - cur: Value, -} - -impl<'a> Iterator for Values<'a> { - type Item = Value; - - fn next(&mut self) -> Option { - let prev = self.cur; - - // Advance self.cur to the next value, or NO_VALUE. - self.cur = match prev.expand() { - ExpandedValue::Direct(inst) => self.func[inst].second_result().unwrap_or_default(), - ExpandedValue::Table(index) => { - match self.func.extended_values[index] { - ValueData::Def { next, .. } => next, - ValueData::Argument { next, .. } => next, - } - } - ExpandedValue::None => return None, - }; - - Some(prev) - } + // Linked list nodes for the layout order of instructions. Forms a double linked list per EBB, + // terminated in both ends by NO_INST. + inst_order: Vec, } impl Function { @@ -215,7 +49,7 @@ impl Function { instructions: Vec::new(), extended_basic_blocks: Vec::new(), extended_values: Vec::new(), - return_types: Vec::new(), + inst_order: Vec::new(), } } @@ -255,6 +89,11 @@ impl Function { pub fn make_inst(&mut self, data: InstructionData) -> Inst { let inst = Inst::new(self.instructions.len()); self.instructions.push(data); + self.inst_order.push(InstNode { + prev: NO_INST, + next: NO_INST, + }); + debug_assert_eq!(self.instructions.len(), self.inst_order.len()); inst } @@ -328,6 +167,14 @@ impl Function { &mut self.extended_basic_blocks[ebb.index()] } + /// Iterate over all the EBBs in order of creation. + pub fn ebbs_numerically(&self) -> NumericalEbbs { + NumericalEbbs { + cur: 0, + limit: self.extended_basic_blocks.len(), + } + } + /// Append an argument with type `ty` to `ebb`. pub fn append_ebb_arg(&mut self, ebb: Ebb, ty: Type) -> Value { let val = self.make_value(ValueData::Argument { @@ -363,6 +210,32 @@ impl Function { } } + /// Append an instruction to a basic block. + pub fn append_inst(&mut self, ebb: Ebb, inst: Inst) { + let old_last = self[ebb].last_inst; + + self.inst_order[inst.index()] = InstNode { + prev: old_last, + next: NO_INST, + }; + + if old_last == NO_INST { + assert!(self[ebb].first_inst == NO_INST); + self[ebb].first_inst = inst; + } else { + self.inst_order[old_last.index()].next = inst; + } + self[ebb].last_inst = inst; + } + + /// Iterate through the instructions in `ebb`. + pub fn ebb_insts<'a>(&'a self, ebb: Ebb) -> EbbInsts<'a> { + EbbInsts { + func: self, + cur: self[ebb].first_inst, + } + } + // Values. /// Allocate an extended value entry. @@ -389,6 +262,238 @@ impl Function { } } +// ====--------------------------------------------------------------------------------------====// +// +// Stack slot implementation. +// +// ====--------------------------------------------------------------------------------------====// + +/// Contents of a stack slot. +#[derive(Debug)] +pub struct StackSlotData { + /// Size of stack slot in bytes. + pub size: u32, +} + +impl StackSlotData { + /// Create a stack slot with the specified byte size. + pub fn new(size: u32) -> StackSlotData { + StackSlotData { size: size } + } +} + +impl Display for StackSlotData { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write!(fmt, "stack_slot {}", self.size) + } +} + +/// Allow immutable access to stack slots via function indexing. +impl Index for Function { + type Output = StackSlotData; + + fn index<'a>(&'a self, ss: StackSlot) -> &'a StackSlotData { + &self.stack_slots[ss.index()] + } +} + +/// Stack slot iterator visits all stack slots in a function, returning `StackSlot` references. +pub struct StackSlotIter { + cur: usize, + end: usize, +} + +impl Iterator for StackSlotIter { + type Item = StackSlot; + + fn next(&mut self) -> Option { + if self.cur < self.end { + let ss = StackSlot::new(self.cur); + self.cur += 1; + Some(ss) + } else { + None + } + } +} + +// ====--------------------------------------------------------------------------------------====// +// +// Extended basic block implementation. +// +// ====--------------------------------------------------------------------------------------====// + +/// Contents of an extended basic block. +/// +/// Arguments for an extended basic block are values that dominate everything in the EBB. All +/// branches to this EBB must provide matching arguments, and the arguments to the entry EBB must +/// match the function arguments. +#[derive(Debug)] +pub struct EbbData { + /// First argument to this EBB, or `NO_VALUE` if the block has no arguments. + /// + /// The arguments are all ValueData::Argument entries that form a linked list from `first_arg` + /// to `last_arg`. + first_arg: Value, + + /// Last argument to this EBB, or `NO_VALUE` if the block has no arguments. + last_arg: Value, + + /// First instruction in this block, or `NO_INST`. + first_inst: Inst, + + /// Last instruction in this block, or `NO_INST`. + last_inst: Inst, +} + +impl EbbData { + fn new() -> EbbData { + EbbData { + first_arg: NO_VALUE, + last_arg: NO_VALUE, + first_inst: NO_INST, + last_inst: NO_INST, + } + } +} + +impl Index for Function { + type Output = EbbData; + + fn index<'a>(&'a self, ebb: Ebb) -> &'a EbbData { + &self.extended_basic_blocks[ebb.index()] + } +} + +impl IndexMut for Function { + fn index_mut<'a>(&'a mut self, ebb: Ebb) -> &'a mut EbbData { + &mut self.extended_basic_blocks[ebb.index()] + } +} + +pub struct EbbInsts<'a> { + func: &'a Function, + cur: Inst, +} + +impl<'a> Iterator for EbbInsts<'a> { + type Item = Inst; + + fn next(&mut self) -> Option { + let prev = self.cur; + if prev == NO_INST { + None + } else { + // Advance self.cur to the next inst. + self.cur = self.func.inst_order[prev.index()].next; + Some(prev) + } + } +} + +/// Iterate through all EBBs in a function in numerical order. +/// This order is stable, but has little significance to the semantics of the function. +pub struct NumericalEbbs { + cur: usize, + limit: usize, +} + +impl Iterator for NumericalEbbs { + type Item = Ebb; + + fn next(&mut self) -> Option { + if self.cur < self.limit { + let prev = Ebb::new(self.cur); + self.cur += 1; + Some(prev) + } else { + None + } + } +} + +// ====--------------------------------------------------------------------------------------====// +// +// Instruction implementation. +// +// The InstructionData layout is defined in the `instructions` module. +// +// ====--------------------------------------------------------------------------------------====// + +/// Allow immutable access to instructions via function indexing. +impl Index for Function { + type Output = InstructionData; + + fn index<'a>(&'a self, inst: Inst) -> &'a InstructionData { + &self.instructions[inst.index()] + } +} + +/// A node in a double linked list of instructions is a basic block. +#[derive(Debug)] +struct InstNode { + prev: Inst, + next: Inst, +} + +// ====--------------------------------------------------------------------------------------====// +// +// Value implementation. +// +// ====--------------------------------------------------------------------------------------====// + +// Most values are simply the first value produced by an instruction. +// Other values have an entry in the value table. +#[derive(Debug)] +enum ValueData { + // Value is defined by an instruction, but it is not the first result. + Def { + ty: Type, + def: Inst, + next: Value, // Next result defined by `def`. + }, + + // Value is an EBB argument. + Argument { + ty: Type, + ebb: Ebb, + next: Value, // Next argument to `ebb`. + }, +} + +/// Iterate through a list of related value references, such as: +/// +/// - All results defined by an instruction. +/// - All arguments to an EBB +/// +/// A value iterator borrows a Function reference. +pub struct Values<'a> { + func: &'a Function, + cur: Value, +} + +impl<'a> Iterator for Values<'a> { + type Item = Value; + + fn next(&mut self) -> Option { + let prev = self.cur; + + // Advance self.cur to the next value, or NO_VALUE. + self.cur = match prev.expand() { + ExpandedValue::Direct(inst) => self.func[inst].second_result().unwrap_or_default(), + ExpandedValue::Table(index) => { + match self.func.extended_values[index] { + ValueData::Def { next, .. } => next, + ValueData::Argument { next, .. } => next, + } + } + ExpandedValue::None => return None, + }; + + Some(prev) + } +} + #[cfg(test)] mod tests { use super::*; @@ -445,6 +550,8 @@ mod tests { fn ebb() { let mut func = Function::new(); + assert_eq!(func.ebbs_numerically().next(), None); + let ebb = func.make_ebb(); assert_eq!(ebb.to_string(), "ebb0"); assert_eq!(func.ebb_args(ebb).next(), None); @@ -464,5 +571,39 @@ mod tests { assert_eq!(args2.next(), Some(arg2)); assert_eq!(args2.next(), None); } + + // The numerical ebb iterator doesn't capture the function. + let mut ebbs = func.ebbs_numerically(); + assert_eq!(ebbs.next(), Some(ebb)); + assert_eq!(ebbs.next(), None); + + assert_eq!(func.ebb_insts(ebb).next(), None); + + let inst = func.make_inst(InstructionData::Nullary { + opcode: Opcode::Iconst, + ty: types::I32, + }); + func.append_inst(ebb, inst); + { + let mut ii = func.ebb_insts(ebb); + assert_eq!(ii.next(), Some(inst)); + assert_eq!(ii.next(), None); + } + assert_eq!(func[ebb].first_inst, inst); + assert_eq!(func[ebb].last_inst, inst); + + let inst2 = func.make_inst(InstructionData::Nullary { + opcode: Opcode::Iconst, + ty: types::I32, + }); + func.append_inst(ebb, inst2); + { + let mut ii = func.ebb_insts(ebb); + assert_eq!(ii.next(), Some(inst)); + assert_eq!(ii.next(), Some(inst2)); + assert_eq!(ii.next(), None); + } + assert_eq!(func[ebb].first_inst, inst); + assert_eq!(func[ebb].last_inst, inst2); } } From 1be81c435200968cd71fa3bb91b88f53f00eadf1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 13 May 2016 15:31:37 -0700 Subject: [PATCH 0079/3084] Implement write_instruction and write_ebb. Use the new iterators to write out the contents of a function. --- cranelift/src/libcretonne/instructions.rs | 4 +- cranelift/src/libcretonne/write.rs | 114 ++++++++++++++++++++-- 2 files changed, 108 insertions(+), 10 deletions(-) diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index 325033284a..39202b000f 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -130,14 +130,14 @@ pub enum InstructionData { BinaryImm { opcode: Opcode, ty: Type, - arg: Value, + lhs: Value, rhs: Imm64, }, // Same as BinaryImm, but the imediate is the lhs operand. BinaryImmRev { opcode: Opcode, ty: Type, - arg: Value, + rhs: Value, lhs: Imm64, }, Call { diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index f245d977c2..9f5c28c0b6 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -6,6 +6,7 @@ use std::io::{self, Write}; use repr::Function; +use entities::{Inst, Ebb, Value}; pub type Result = io::Result<()>; @@ -13,7 +14,14 @@ pub type Result = io::Result<()>; pub fn write_function(w: &mut Write, func: &Function) -> Result { try!(write_spec(w, func)); try!(writeln!(w, " {{")); - try!(write_preamble(w, func)); + let mut any = try!(write_preamble(w, func)); + for ebb in func.ebbs_numerically() { + if !any { + try!(writeln!(w, "")); + } + try!(write_ebb(w, func, ebb)); + any = true; + } writeln!(w, "}}") } @@ -58,7 +66,7 @@ fn write_spec(w: &mut Write, func: &Function) -> Result { } } -fn write_preamble(w: &mut Write, func: &Function) -> Result { +fn write_preamble(w: &mut Write, func: &Function) -> io::Result { let mut any = false; for ss in func.stack_slot_iter() { @@ -66,11 +74,88 @@ fn write_preamble(w: &mut Write, func: &Function) -> Result { try!(writeln!(w, " {} = {}", ss, func[ss])); } - // Put a blank line after the preamble unless it was empty. - if any { - writeln!(w, "") - } else { - Ok(()) + Ok(any) +} + +// ====--------------------------------------------------------------------------------------====// +// +// Basic blocks +// +// ====--------------------------------------------------------------------------------------====// + +pub fn write_arg(w: &mut Write, func: &Function, arg: Value) -> Result { + write!(w, "{}: {}", arg, func.value_type(arg)) +} + +pub fn write_ebb_header(w: &mut Write, func: &Function, ebb: Ebb) -> Result { + // Write out the basic block header, outdented: + // + // ebb1: + // ebb1(vx1: i32): + // ebb10(vx4: f64, vx5: b1): + // + + let mut args = func.ebb_args(ebb); + match args.next() { + None => return writeln!(w, "{}:", ebb), + Some(arg) => { + try!(write!(w, "{}(", ebb)); + try!(write_arg(w, func, arg)); + } + } + // Remaining args. + for arg in args { + try!(write!(w, ", ")); + try!(write_arg(w, func, arg)); + } + writeln!(w, "):") +} + +pub fn write_ebb(w: &mut Write, func: &Function, ebb: Ebb) -> Result { + try!(write_ebb_header(w, func, ebb)); + for inst in func.ebb_insts(ebb) { + try!(write_instruction(w, func, inst)); + } + Ok(()) +} + + +// ====--------------------------------------------------------------------------------------====// +// +// Instructions +// +// ====--------------------------------------------------------------------------------------====// + +pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { + try!(write!(w, " ")); + + // First write out the result values, if any. + let mut has_results = false; + for r in func.inst_results(inst) { + if !has_results { + has_results = true; + try!(write!(w, "{}", r)); + } else { + try!(write!(w, ", {}", r)); + } + } + if has_results { + try!(write!(w, " = ")); + } + + // Then the opcode and operands, depending on format. + use instructions::InstructionData::*; + match func[inst] { + Nullary { opcode, .. } => writeln!(w, "{}", opcode), + Unary { opcode, arg, .. } => writeln!(w, "{} {}", opcode, arg), + UnaryImm { opcode, imm, .. } => writeln!(w, "{} {}", opcode, imm), + UnaryIeee32 { opcode, imm, .. } => writeln!(w, "{} {}", opcode, imm), + UnaryIeee64 { opcode, imm, .. } => writeln!(w, "{} {}", opcode, imm), + UnaryImmVector { opcode, .. } => writeln!(w, "{} [...]", opcode), + Binary { opcode, args, .. } => writeln!(w, "{} {}, {}", opcode, args[0], args[1]), + BinaryImm { opcode, lhs, rhs, .. } => writeln!(w, "{} {}, {}", opcode, lhs, rhs), + BinaryImmRev { opcode, lhs, rhs, .. } => writeln!(w, "{} {}, {}", opcode, lhs, rhs), + Call { opcode, .. } => writeln!(w, "{} [...]", opcode), } } @@ -79,6 +164,7 @@ mod tests { use super::*; use super::{needs_quotes, escaped}; use repr::{Function, StackSlotData}; + use types; #[test] fn quoting() { @@ -108,6 +194,18 @@ mod tests { f.make_stack_slot(StackSlotData::new(4)); assert_eq!(function_to_string(&f), - "function foo() {\n ss0 = stack_slot 4\n\n}\n"); + "function foo() {\n ss0 = stack_slot 4\n}\n"); + + let ebb = f.make_ebb(); + assert_eq!(function_to_string(&f), + "function foo() {\n ss0 = stack_slot 4\nebb0:\n}\n"); + + f.append_ebb_arg(ebb, types::I8); + assert_eq!(function_to_string(&f), + "function foo() {\n ss0 = stack_slot 4\nebb0(vx0: i8):\n}\n"); + + f.append_ebb_arg(ebb, types::F32.by(4).unwrap()); + assert_eq!(function_to_string(&f), + "function foo() {\n ss0 = stack_slot 4\nebb0(vx0: i8, vx1: f32x4):\n}\n"); } } From 639d92177ab9f0e8248143238b908ac192bba7d6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 18 May 2016 11:31:47 -0700 Subject: [PATCH 0080/3084] Reorganize meta language reference. Separate instruction descriptions from instruction formats which deal with the Rust representation. Add type class restrictions to type variables. --- cranelift/docs/metaref.rst | 159 ++++++++++++++++++++++++++----------- meta/cretonne/__init__.py | 39 +++++---- meta/cretonne/base.py | 13 ++- 3 files changed, 145 insertions(+), 66 deletions(-) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index fe46e8af4a..2e47535492 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -4,65 +4,79 @@ Cretonne Meta Language Reference .. default-domain:: py .. highlight:: python - -The Cretonne meta language is used to define instructions for Cretonne. It is a -domain specific language embedded in Python. - -An instruction set is described by a Python module under the :file:`meta` -directory that has a global variable called ``instructions``. The basic -Cretonne instruction set described in :doc:`langref` is defined by the Python -module :mod:`cretonne.base`. - .. module:: cretonne -Value Types -=========== +The Cretonne meta language is used to define instructions for Cretonne. It is a +domain specific language embedded in Python. This document describes the Python +modules that form the embedded DSL. -Concrete value types are represented as instances of :class:`cretonne.ValueType`. There are -subclasses to represent scalar and vector types. +The meta language descriptions are Python modules under the :file:`meta` +top-level directory. The descriptions are processed in two steps: -.. inheritance-diagram:: ValueType ScalarType VectorType IntType FloatType - :parts: 1 -.. autoclass:: ValueType -.. autoclass:: ScalarType - :members: -.. autoclass:: VectorType - :members: -.. autoclass:: IntType - :members: -.. autoclass:: FloatType - :members: +1. The Python modules are imported. This has the effect of building static data + structures in global variables in the modules. These static data structures + use the classes in the :mod:`cretonne` module to describe instruction sets + and other properties. -Predefined types ----------------- -.. automodule:: cretonne.types - :members: +2. The static data structures are processed to produce Rust source code and + constant dables tables. -.. currentmodule:: cretonne +The main driver for this source code generation process is the +:file:`meta/build.py` script which is invoked as part of the build process if +anything in the :file:`meta` directory has changed since the last build. -Parametric polymorphism ------------------------ +Instruction descriptions +======================== -Instruction operands can be defined with *type variables* instead of concrete -types for their operands. This makes the instructions polymorphic. +New instructions are defined as instances of the :class:`Instruction` +class. As instruction instances are created, they are added to the currently +open :class:`InstructionGroup`. -.. autoclass:: TypeVar - -Instructions -============ - -New instructions are defined as instances of the :class:`cretonne.Instruction` -class. - -.. autoclass:: Instruction -.. autoclass:: Operand -.. autoclass:: OperandKind .. autoclass:: InstructionGroup :members: +The basic Cretonne instruction set described in :doc:`langref` is defined by the +Python module :mod:`cretonne.base`. This module has a global variable +:data:`cretonne.base.instructions` which is an :class:`InstructionGroup` +instance containing all the base instructions. -Immediates ----------- +.. autoclass:: Instruction + +An instruction is defined with a set of distinct input and output operands which +must be instances of the :class:`Operand` class. + +.. autoclass:: Operand + +Cretonne uses two separate type systems for immediate operands and SSA values. + +Type variables +-------------- + +Instruction descriptions can be made polymorphic by using :class:`Operand` +instances that refer to a *type variable* instead of a concrete value type. +Polymorphism only works for SSA value operands. Immediate operands have a fixed +operand kind. + +.. autoclass:: TypeVar + +If multiple operands refer to the same type variable they will be required to +have the same concrete type. For example, this defines an integer addition +instruction:: + + Int = TypeVar('Int', 'A scalar or vector integer type', ints=True, simd=True) + a = Operand('a', Int) + x = Operand('x', Int) + y = Operand('y', Int) + + iadd = Instruction('iadd', 'Integer addition', ins=(x, y), outs=a) + +The type variable `Int` is allowed to vary over all scalar and vector integer +value types, but in a given instance of the `iadd` instruction, the two +operands must have the same type, and the result will be the same type as the +inputs. + +Immediate operands +------------------ Immediate instruction operands don't correspond to SSA values, but have values that are encoded directly in the instruction. Immediate operands don't @@ -77,6 +91,59 @@ indicated with an instance of :class:`ImmediateKind`. .. currentmodule:: cretonne + +Value types +----------- + +Concrete value types are represented as instances of :class:`cretonne.ValueType`. There are +subclasses to represent scalar and vector types. + +.. autoclass:: ValueType +.. inheritance-diagram:: ValueType ScalarType VectorType IntType FloatType + :parts: 1 +.. autoclass:: ScalarType + :members: +.. autoclass:: VectorType + :members: +.. autoclass:: IntType + :members: +.. autoclass:: FloatType + :members: + +.. automodule:: cretonne.types + :members: + +.. currentmodule:: cretonne + +There are no predefined vector types, but they can be created as needed with +the :func:`ScalarType.by` function. + + +Instruction representation +========================== + +The Rust in-memory representation of instructions is derived from the +instruction descriptions. Part of the representation is generated, and part is +written as Rust code in the `cretonne.instructions` module. The instruction +representation depends on the input operand kinds and whether the instruction +can produce multiple results. + +.. autoclass:: OperandKind + +Since all SSA value operands are represented as a `Value` in Rust code, value +types don't affect the representation. Two special operand kinds are used to +represent SSA values: + +.. autodata:: value +.. autodata:: args + +When an instruction description is created, it is automatically assigned a +predefined instruction format which is an instance of +:class:`InstructionFormat`: + +.. autoclass:: InstructionFormat + + Targets ======= diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index cbad9185fc..7e6f61d6bf 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -24,8 +24,6 @@ def camel_case(s): # operands and the kind of each operand. class OperandKind(object): """ - The kind of an operand. - An instance of the `OperandKind` class corresponds to a kind of operand. Each operand kind has a corresponding type in the Rust representation of an instruction. @@ -55,8 +53,8 @@ value = OperandKind( operand. """) -#: A variable-sizes list of value operands. Use for Ebb and function call -#: arguemnts. +#: A variable-sized list of value operands. Use for Ebb and function call +#: arguments. args = OperandKind( 'args', """ A variable size list of `value` operands. @@ -71,7 +69,7 @@ args = OperandKind( # module. class ImmediateKind(OperandKind): """ - The type of an immediate instruction operand. + The kind of an immediate instruction operand. """ def __init__(self, name, doc): @@ -215,13 +213,30 @@ class BoolType(ScalarType): class TypeVar(object): """ - A Type Variable. - Type variables can be used in place of concrete types when defining instructions. This makes the instructions *polymorphic*. + + A type variable is restricted to vary over a subset of the value types. + This subset is specified by a set of flags that control the permitted base + types and whether the type variable can assume scalar or vector types, or + both. + + :param name: Short name of type variable used in instruction descriptions. + :param doc: Documentation string. + :param base: Single base type or list of base types. Use this to specify an + exact set of base types if the general categories below are not good + enough. + :param ints: Allow all integer base types. + :param floats: Allow all floating point base types. + :param bools: Allow all boolean base types. + :param scalars: Allow type variable to assume scalar types. + :param simd: Allow type variable to assume vector types. """ - def __init__(self, name, doc): + def __init__( + self, name, doc, base=None, + ints=False, floats=False, bools=False, + scalars=True, simd=False): self.name = name self.__doc__ = doc @@ -238,8 +253,6 @@ class TypeVar(object): class InstructionGroup(object): """ - An instruction group. - Every instruction must belong to exactly one instruction group. A given target architecture can support instructions from multiple groups, and it does not necessarily support all instructions in a group. @@ -286,8 +299,6 @@ class InstructionGroup(object): class Operand(object): """ - An instruction operand. - An instruction operand can be either an *immediate* or an *SSA value*. The type of the operand is one of: @@ -318,8 +329,6 @@ class Operand(object): class InstructionFormat(object): """ - An instruction format. - Every instruction opcode has a corresponding instruction format which determines the number of operands and their kinds. Instruction formats are identified structurally, i.e., the format of an instruction is derived from @@ -396,8 +405,6 @@ class InstructionFormat(object): class Instruction(object): """ - An instruction description. - The operands to the instruction are specified as two tuples: ``ins`` and ``outs``. Since the Python singleton tuple syntax is a bit awkward, it is allowed to specify a singleton as just the operand itself, i.e., `ins=x` diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 412e7f3abd..3fd8554cf3 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -10,9 +10,11 @@ from immediates import imm64, ieee32, ieee64, immvector instructions = InstructionGroup("base", "Shared base instruction set") -Int = TypeVar('Int', 'A scalar or vector integer type') -iB = TypeVar('iB', 'A scalar integer type') -TxN = TypeVar('%Tx%N', 'A SIMD vector type') +Int = TypeVar('Int', 'A scalar or vector integer type', ints=True, simd=True) +iB = TypeVar('iB', 'A scalar integer type', ints=True) +TxN = TypeVar( + '%Tx%N', 'A SIMD vector type', + ints=True, floats=True, bools=True, scalars=False, simd=True) # # Materializing constants. @@ -217,7 +219,10 @@ isub_imm = Instruction( # # TODO: Which types should permit boolean operations? Any reason to restrict? -bits = TypeVar('bits', 'Any integer, float, or boolean scalar or vector type') +bits = TypeVar( + 'bits', 'Any integer, float, or boolean scalar or vector type', + ints=True, floats=True, bools=True, scalars=True, simd=True) + x = Operand('x', bits) y = Operand('y', bits) a = Operand('a', bits) From 74625a9b7984c1aa83587feaf0bcc6d23fd9d927 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 18 May 2016 12:24:14 -0700 Subject: [PATCH 0081/3084] Add restrictions on polymorphism. Also introduce the concept of a derived type variable, and provide two methods for deriving type vars: lane() and as_bool(). --- cranelift/docs/metaref.rst | 39 ++++++++++++++++++++++++++++++++++++++ meta/cretonne/__init__.py | 24 +++++++++++++++++++---- 2 files changed, 59 insertions(+), 4 deletions(-) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 2e47535492..5858ef512e 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -58,6 +58,7 @@ Polymorphism only works for SSA value operands. Immediate operands have a fixed operand kind. .. autoclass:: TypeVar + :members: If multiple operands refer to the same type variable they will be required to have the same concrete type. For example, this defines an integer addition @@ -75,6 +76,9 @@ value types, but in a given instance of the `iadd` instruction, the two operands must have the same type, and the result will be the same type as the inputs. +There are some practical restrictions on the use of type variables, see +:ref:`restricted-polymorphism`. + Immediate operands ------------------ @@ -144,6 +148,41 @@ predefined instruction format which is an instance of .. autoclass:: InstructionFormat +.. _restricted-polymorphism: + +Restricted polymorphism +----------------------- + +The instruction format strictly controls the kinds of operands on an +instruction, but it does not constrain value types at all. A given instruction +description typically does constrain the allowed value types for its value +operands. The type variables give a lot of freedom in describing the value type +constraints, in practice more freedom than what is needed for normal instruction +set architectures. In order to simplify the Rust representation of value type +constraints, some restrictions are imposed on the use of type variables. + +A polymorphic instruction has a single *controlling type variable*. The value +types of instruction results must be one of the following: + +1. A concrete value type. +2. The controlling type variable. +3. A type variable derived from the controlling type variable. + +This means that all result types can be computed from the controlling type +variable. + +Input values to the instruction are allowed a bit more freedom. Input value +types must be one of: + +1. A concrete value type. +2. The controlling type variable. +3. A type variable derived from the controlling type variable. +4. A free type variable that is not used by any other operands. + +This means that the type of an input operand can either be computed from the +controlling type variable, or it can vary independently of the other operands. + + Targets ======= diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 7e6f61d6bf..8ce84394da 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -240,11 +240,27 @@ class TypeVar(object): self.name = name self.__doc__ = doc + def lane(self): + """ + Return a derived type variable that is the scalar lane type of this + type variable. + + When this type variable assumes a scalar type, the derived type will be + the same scalar type. + """ + return TypeVar("Lane type of " + self.name, '', base=self, simd=False) + + def as_bool(self): + """ + Return a derived type variable that has the same vector geometry as + this type variable, but with boolean lanes. Scalar types map to `b1`. + """ + return TypeVar(self.name + " as boolean", '', base=self, bools=True) + def operand_kind(self): - """ - When a `TypeVar` object is used to describe the type of an `Operand` - in an instruction definition, the kind of that operand is an SSA value. - """ + # When a `TypeVar` object is used to describe the type of an `Operand` + # in an instruction definition, the kind of that operand is an SSA + # value. return value From 25e78fdbff5899102306c816d503527377127574 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 17 May 2016 11:54:46 -0700 Subject: [PATCH 0082/3084] Parse basic blocks and instructions. Create map entries for ebbs and values as they are defined, but leave ebb and value operands unresolved on instructions as they are parsed. Instruction operands can refer to ebbs and values that may not have been defined yet. Don't infer or check result types yet. --- cranelift/src/libcretonne/entities.rs | 6 + cranelift/src/libcretonne/repr.rs | 4 + cranelift/src/libreader/lexer.rs | 6 +- cranelift/src/libreader/parser.rs | 371 +++++++++++++++++++++++++- 4 files changed, 371 insertions(+), 16 deletions(-) diff --git a/cranelift/src/libcretonne/entities.rs b/cranelift/src/libcretonne/entities.rs index bdc76d8040..d4e1cb849b 100644 --- a/cranelift/src/libcretonne/entities.rs +++ b/cranelift/src/libcretonne/entities.rs @@ -105,6 +105,12 @@ pub enum ExpandedValue { } impl Value { + pub fn direct_from_number(n: u32) -> Value { + let encoding = n * 2; + assert!(encoding < u32::MAX); + Value(encoding) + } + pub fn new_direct(i: Inst) -> Value { let encoding = i.index() * 2; assert!(encoding < u32::MAX as usize); diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index a444dd0b3d..6e7e6ea80f 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -19,6 +19,9 @@ pub struct Function { /// Signature of this function. signature: Signature, + /// The entry block. + pub entry_block: Ebb, + /// Stack slots allocated in this function. stack_slots: Vec, @@ -45,6 +48,7 @@ impl Function { Function { name: name, signature: sig, + entry_block: NO_EBB, stack_slots: Vec::new(), instructions: Vec::new(), extended_basic_blocks: Vec::new(), diff --git a/cranelift/src/libreader/lexer.rs b/cranelift/src/libreader/lexer.rs index 855dd34056..3611a70f69 100644 --- a/cranelift/src/libreader/lexer.rs +++ b/cranelift/src/libreader/lexer.rs @@ -36,7 +36,7 @@ pub enum Token<'a> { Integer(&'a str), // Integer immediate Type(types::Type), // i32, f32, b32x4, ... ValueDirect(u32), // v12 - ValueExtended(u32), // vx7 + ValueTable(u32), // vx7 Ebb(u32), // ebb3 StackSlot(u32), // ss3 Identifier(&'a str), // Unrecognized identifier (opcode, enumerator, ...) @@ -290,7 +290,7 @@ impl<'a> Lexer<'a> { match prefix { "v" => Some(Token::ValueDirect(value)), - "vx" => Some(Token::ValueExtended(value)), + "vx" => Some(Token::ValueTable(value)), "ebb" => Some(Token::Ebb(value)), "ss" => Some(Token::StackSlot(value)), _ => None, @@ -452,7 +452,7 @@ mod tests { assert_eq!(lex.next(), token(Token::Identifier("ebb5234567890"), 1)); assert_eq!(lex.next(), token(Token::Entry, 1)); assert_eq!(lex.next(), token(Token::Identifier("v1x"), 1)); - assert_eq!(lex.next(), token(Token::ValueExtended(1), 1)); + assert_eq!(lex.next(), token(Token::ValueTable(1), 1)); assert_eq!(lex.next(), token(Token::Identifier("vxvx4"), 1)); assert_eq!(lex.next(), token(Token::Identifier("function0"), 1)); assert_eq!(lex.next(), token(Token::Function, 1)); diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 119957d7c4..f390f0dcfa 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -10,9 +10,10 @@ use std::result; use std::fmt::{self, Display, Formatter, Write}; use std::u32; use lexer::{self, Lexer, Token}; -use cretonne::types::{FunctionName, Signature, ArgumentType, ArgumentExtension}; -use cretonne::immediates::Imm64; -use cretonne::entities::StackSlot; +use cretonne::types::{Type, VOID, FunctionName, Signature, ArgumentType, ArgumentExtension}; +use cretonne::immediates::{Imm64, Ieee32, Ieee64}; +use cretonne::entities::*; +use cretonne::instructions::{Opcode, InstructionFormat, InstructionData}; use cretonne::repr::{Function, StackSlotData}; pub use lexer::Location; @@ -50,7 +51,10 @@ pub struct Parser<'a> { // file by number. We need to map these numbers to real references. struct Context { function: Function, - stack_slots: HashMap, + stack_slots: HashMap, // ssNN + ebbs: HashMap, // ebbNN + value_directs: HashMap, // vNN + value_tables: HashMap, // vxNN } impl Context { @@ -58,10 +62,14 @@ impl Context { Context { function: f, stack_slots: HashMap::new(), + ebbs: HashMap::new(), + value_directs: HashMap::new(), + value_tables: HashMap::new(), } } - fn add(&mut self, number: u32, data: StackSlotData, loc: &Location) -> Result<()> { + // Allocate a new stack slot and add a mapping number -> StackSlot. + fn add_ss(&mut self, number: u32, data: StackSlotData, loc: &Location) -> Result<()> { if self.stack_slots.insert(number, self.function.make_stack_slot(data)).is_some() { Err(Error { location: loc.clone(), @@ -71,6 +79,43 @@ impl Context { Ok(()) } } + + // Allocate a new EBB and add a mapping number -> Ebb. + fn add_ebb(&mut self, number: u32, loc: &Location) -> Result { + let ebb = self.function.make_ebb(); + if self.ebbs.insert(number, ebb).is_some() { + Err(Error { + location: loc.clone(), + message: format!("duplicate EBB: ebb{}", number), + }) + } else { + Ok(ebb) + } + } + + // Add a value mapping number -> data for a direct value (vNN). + fn add_v(&mut self, number: u32, data: Value, loc: &Location) -> Result<()> { + if self.value_directs.insert(number, data).is_some() { + Err(Error { + location: loc.clone(), + message: format!("duplicate value: v{}", number), + }) + } else { + Ok(()) + } + } + + // Add a value mapping number -> data for a table value (vxNN). + fn add_vx(&mut self, number: u32, data: Value, loc: &Location) -> Result<()> { + if self.value_tables.insert(number, data).is_some() { + Err(Error { + location: loc.clone(), + message: format!("duplicate value: vx{}", number), + }) + } else { + Ok(()) + } + } } impl<'a> Parser<'a> { @@ -154,6 +199,16 @@ impl<'a> Parser<'a> { } } + // Match and consume a type. + fn match_type(&mut self, err_msg: &str) -> Result { + if let Some(Token::Type(t)) = self.token() { + self.consume(); + Ok(t) + } else { + Err(self.error(err_msg)) + } + } + // Match and consume a stack slot reference. fn match_ss(&mut self, err_msg: &str) -> Result { if let Some(Token::StackSlot(ss)) = self.token() { @@ -164,6 +219,38 @@ impl<'a> Parser<'a> { } } + // Match and consume an ebb reference. + fn match_ebb(&mut self, err_msg: &str) -> Result { + if let Some(Token::Ebb(ebb)) = self.token() { + self.consume(); + Ok(ebb) + } else { + Err(self.error(err_msg)) + } + } + + // Match and consume a vx reference. + fn match_vx(&mut self, err_msg: &str) -> Result { + if let Some(Token::ValueTable(vx)) = self.token() { + self.consume(); + Ok(vx) + } else { + Err(self.error(err_msg)) + } + } + + // Match and consume a value reference, direct or vtable. + // This does not convert from the source value numbering to our in-memory value numbering. + fn match_value(&mut self, err_msg: &str) -> Result { + let val = match self.token() { + Some(Token::ValueDirect(v)) => Value::direct_from_number(v), + Some(Token::ValueTable(vx)) => Value::new_table(vx as usize), + _ => return Err(self.error(err_msg)), + }; + self.consume(); + Ok(val) + } + // Match and consume an Imm64 immediate. fn match_imm64(&mut self, err_msg: &str) -> Result { if let Some(Token::Integer(text)) = self.token() { @@ -176,6 +263,30 @@ impl<'a> Parser<'a> { } } + // Match and consume an Ieee32 immediate. + fn match_ieee32(&mut self, err_msg: &str) -> Result { + if let Some(Token::Float(text)) = self.token() { + self.consume(); + // Lexer just gives us raw text that looks like a float. + // Parse it as an Ieee32 to check for the right number of digits and other issues. + text.parse().map_err(|e| self.error(e)) + } else { + Err(self.error(err_msg)) + } + } + + // Match and consume an Ieee64 immediate. + fn match_ieee64(&mut self, err_msg: &str) -> Result { + if let Some(Token::Float(text)) = self.token() { + self.consume(); + // Lexer just gives us raw text that looks like a float. + // Parse it as an Ieee64 to check for the right number of digits and other issues. + text.parse().map_err(|e| self.error(e)) + } else { + Err(self.error(err_msg)) + } + } + /// Parse a list of function definitions. /// /// This is the top-level parse function matching the whole contents of a file. @@ -199,6 +310,8 @@ impl<'a> Parser<'a> { try!(self.match_token(Token::LBrace, "expected '{' before function body")); // function ::= function-spec "{" * preample function-body "}" try!(self.parse_preamble(&mut ctx)); + // function ::= function-spec "{" preample * function-body "}" + try!(self.parse_function_body(&mut ctx)); // function ::= function-spec "{" preample function-body * "}" try!(self.match_token(Token::RBrace, "expected '}' after function body")); @@ -279,12 +392,7 @@ impl<'a> Parser<'a> { // Parse a single argument type with flags. fn parse_argument_type(&mut self) -> Result { // arg ::= * type { flag } - let mut arg = if let Some(Token::Type(t)) = self.token() { - ArgumentType::new(t) - } else { - return Err(self.error("expected argument type")); - }; - self.consume(); + let mut arg = ArgumentType::new(try!(self.match_type("expected argument type"))); // arg ::= type * { flag } while let Some(Token::Identifier(s)) = self.token() { @@ -313,7 +421,7 @@ impl<'a> Parser<'a> { try!(match self.token() { Some(Token::StackSlot(..)) => { self.parse_stack_slot_decl() - .and_then(|(num, dat)| ctx.add(num, dat, &self.location)) + .and_then(|(num, dat)| ctx.add_ss(num, dat, &self.location)) } // More to come.. _ => return Ok(()), @@ -339,6 +447,221 @@ impl<'a> Parser<'a> { // TBD: stack-slot-decl ::= StackSlot(ss) "=" "stack_slot" Bytes * {"," stack-slot-flag} Ok((number, data)) } + + // Parse a function body, add contents to `ctx`. + // + // function-body ::= * { extended-basic-block } + // + fn parse_function_body(&mut self, ctx: &mut Context) -> Result<()> { + while self.token() != Some(Token::RBrace) { + try!(self.parse_extended_basic_block(ctx)); + } + Ok(()) + } + + // Parse an extended basic block, add contents to `ctx`. + // + // extended-basic-block ::= * ebb-header { instruction } + // ebb-header ::= ["entry"] Ebb(ebb) [ebb-args] ":" + // + fn parse_extended_basic_block(&mut self, ctx: &mut Context) -> Result<()> { + let is_entry = self.optional(Token::Entry); + let ebb_num = try!(self.match_ebb("expected EBB header")); + let ebb = try!(ctx.add_ebb(ebb_num, &self.location)); + + if is_entry { + if ctx.function.entry_block != NO_EBB { + return Err(self.error("multiple entry blocks in function")); + } + ctx.function.entry_block = ebb; + } + + if !self.optional(Token::Colon) { + // ebb-header ::= ["entry"] Ebb(ebb) [ * ebb-args ] ":" + try!(self.parse_ebb_args(ctx, ebb)); + try!(self.match_token(Token::Colon, "expected ':' after EBB arguments")); + } + + // extended-basic-block ::= ebb-header * { instruction } + while match self.token() { + Some(Token::ValueDirect(_)) => true, + Some(Token::Identifier(_)) => true, + _ => false, + } { + try!(self.parse_instruction(ctx, ebb)); + } + + Ok(()) + } + + // Parse parenthesized list of EBB arguments. Returns a vector of (u32, Type) pairs with the + // source vx numbers of the defined values and the defined types. + // + // ebb-args ::= * "(" ebb-arg { "," ebb-arg } ")" + fn parse_ebb_args(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { + // ebb-args ::= * "(" ebb-arg { "," ebb-arg } ")" + try!(self.match_token(Token::LPar, "expected '(' before EBB arguments")); + + // ebb-args ::= "(" * ebb-arg { "," ebb-arg } ")" + try!(self.parse_ebb_arg(ctx, ebb)); + + // ebb-args ::= "(" ebb-arg * { "," ebb-arg } ")" + while self.optional(Token::Comma) { + // ebb-args ::= "(" ebb-arg { "," * ebb-arg } ")" + try!(self.parse_ebb_arg(ctx, ebb)); + } + + // ebb-args ::= "(" ebb-arg { "," ebb-arg } * ")" + try!(self.match_token(Token::RPar, "expected ')' after EBB arguments")); + + Ok(()) + } + + // Parse a single EBB argument declaration, and append it to `ebb`. + // + // ebb-arg ::= * ValueTable(vx) ":" Type(t) + // + fn parse_ebb_arg(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { + // ebb-arg ::= * ValueTable(vx) ":" Type(t) + let vx = try!(self.match_vx("EBB argument must be a vx value")); + let vx_location = self.location; + // ebb-arg ::= ValueTable(vx) * ":" Type(t) + try!(self.match_token(Token::Colon, "expected ':' after EBB argument")); + // ebb-arg ::= ValueTable(vx) ":" * Type(t) + let t = try!(self.match_type("expected EBB argument type")); + // Allocate the EBB argument and add the mapping. + let value = ctx.function.append_ebb_arg(ebb, t); + ctx.add_vx(vx, value, &vx_location) + } + + // Parse an instruction, append it to `ebb`. + // + // instruction ::= [inst-results "="] Opcode(opc) ... + // inst-results ::= ValueDirect(v) { "," ValueTable(vx) } + // + fn parse_instruction(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { + // Result value numbers. First is a ValueDirect, remaining are ValueTable. + let mut results = Vec::new(); + + // instruction ::= * [inst-results "="] Opcode(opc) ... + if let Some(Token::ValueDirect(v)) = self.token() { + self.consume(); + results.push(v); + + // inst-results ::= ValueDirect(v) * { "," ValueTable(vx) } + while self.optional(Token::Comma) { + // inst-results ::= ValueDirect(v) { "," * ValueTable(vx) } + results.push(try!(self.match_vx("expected vx result value"))); + } + + try!(self.match_token(Token::Equal, "expected '=' before opcode")); + } + + // instruction ::= [inst-results "="] * Opcode(opc) ... + let opcode = if let Some(Token::Identifier(text)) = self.token() { + match text.parse() { + Ok(opc) => opc, + Err(msg) => return Err(self.error(msg)), + } + } else { + return Err(self.error("expected instruction opcode")); + }; + + // instruction ::= [inst-results "="] Opcode(opc) * ... + let inst_data = try!(self.parse_inst_operands(opcode)); + let inst = ctx.function.make_inst(inst_data); + + // TODO: Check that results.len() matches the opcode. + // TODO: Multiple results. + if !results.is_empty() { + assert!(results.len() == 1, "Multiple results not implemented"); + let result = ctx.function.first_result(inst); + try!(ctx.add_v(results[0], result, &self.location)); + } + + ctx.function.append_inst(ebb, inst); + + Ok(()) + } + + // Parse the operands following the instruction opcode. + // This depends on the format of the opcode. + fn parse_inst_operands(&mut self, opcode: Opcode) -> Result { + Ok(match opcode.format().unwrap() { + InstructionFormat::Nullary => { + InstructionData::Nullary { + opcode: opcode, + ty: VOID, + } + } + InstructionFormat::Unary => { + InstructionData::Unary { + opcode: opcode, + ty: VOID, + arg: try!(self.match_value("expected SSA value operand")), + } + } + InstructionFormat::UnaryImm => { + InstructionData::UnaryImm { + opcode: opcode, + ty: VOID, + imm: try!(self.match_imm64("expected immediate integer operand")), + } + } + InstructionFormat::UnaryIeee32 => { + InstructionData::UnaryIeee32 { + opcode: opcode, + ty: VOID, + imm: try!(self.match_ieee32("expected immediate 32-bit float operand")), + } + } + InstructionFormat::UnaryIeee64 => { + InstructionData::UnaryIeee64 { + opcode: opcode, + ty: VOID, + imm: try!(self.match_ieee64("expected immediate 64-bit float operand")), + } + } + InstructionFormat::UnaryImmVector => { + unimplemented!(); + } + InstructionFormat::Binary => { + let lhs = try!(self.match_value("expected SSA value first operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let rhs = try!(self.match_value("expected SSA value second operand")); + InstructionData::Binary { + opcode: opcode, + ty: VOID, + args: [lhs, rhs], + } + } + InstructionFormat::BinaryImm => { + let lhs = try!(self.match_value("expected SSA value first operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let rhs = try!(self.match_imm64("expected immediate integer second operand")); + InstructionData::BinaryImm { + opcode: opcode, + ty: VOID, + lhs: lhs, + rhs: rhs, + } + } + InstructionFormat::BinaryImmRev => { + let lhs = try!(self.match_imm64("expected immediate integer first operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let rhs = try!(self.match_value("expected SSA value second operand")); + InstructionData::BinaryImmRev { + opcode: opcode, + ty: VOID, + lhs: lhs, + rhs: rhs, + } + } + InstructionFormat::Call => { + unimplemented!(); + } + }) + } } #[cfg(test)] @@ -400,7 +723,7 @@ mod tests { assert_eq!(func[ss1].size, 1); assert_eq!(iter.next(), None); - // Catch suplicate definitions. + // Catch duplicate definitions. assert_eq!(Parser::new("function bar() { ss1 = stack_slot 13 ss1 = stack_slot 1 @@ -410,4 +733,26 @@ mod tests { .to_string(), "3: duplicate stack slot: ss1"); } + + #[test] + fn ebb_header() { + let func = Parser::new("function ebbs() { + ebb0: + ebb4(vx3: i32): + }") + .parse_function() + .unwrap(); + assert_eq!(func.name, "ebbs"); + + let mut ebbs = func.ebbs_numerically(); + + let ebb0 = ebbs.next().unwrap(); + assert_eq!(func.ebb_args(ebb0).next(), None); + + let ebb4 = ebbs.next().unwrap(); + let mut ebb4_args = func.ebb_args(ebb4); + let arg0 = ebb4_args.next().unwrap(); + assert_eq!(func.value_type(arg0), types::I32); + assert_eq!(ebb4_args.next(), None); + } } From 27a311701de605d22eb8270de5361ec1673277cd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 18 May 2016 15:30:16 -0700 Subject: [PATCH 0083/3084] Add entity references as a new operand kind. Define known entities in the cretonne.entities module. --- cranelift/docs/langref.rst | 30 +++++++++---------- cranelift/docs/metaref.rst | 19 ++++++++++-- cranelift/src/libreader/parser.rs | 10 +++---- meta/cretonne/__init__.py | 48 +++++++++++++++++++++---------- meta/cretonne/entities.py | 23 +++++++++++++++ meta/cretonne/formats.py | 5 ++-- meta/cretonne/immediates.py | 2 +- 7 files changed, 96 insertions(+), 41 deletions(-) create mode 100644 meta/cretonne/entities.py diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 0851ea8fcd..1d987fb3c7 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -36,11 +36,11 @@ Here is the same function compiled into Cretonne IL: The first line of a function definition provides the function *name* and the :term:`function signature` which declares the argument and return types. -Then follows the :term:`function preample` which declares a number of entities -that can be referenced inside the function. In the example above, the preample +Then follows the :term:`function preamble` which declares a number of entities +that can be referenced inside the function. In the example above, the preamble declares a single local variable, ``ss1``. -After the preample follows the :term:`function body` which consists of +After the preamble follows the :term:`function body` which consists of :term:`extended basic block`\s, one of which is marked as the :term:`entry block`. Every EBB ends with a :term:`terminator instruction`, so execution can never fall through to the next EBB without an explicit branch. @@ -49,7 +49,7 @@ A ``.cton`` file consists of a sequence of independent function definitions: .. productionlist:: function-list : { function } - function : function-spec "{" preample function-body "}" + function : function-spec "{" preamble function-body "}" function-spec : "function" function-name signature preamble : { preamble-decl } function-body : { extended-basic-block } @@ -361,12 +361,12 @@ instruction in the EBB. blocks. Split critical edges as needed to work around this. :arg iN x: Integer index into jump table. - :arg JT: Jump table which was declared in the preample. + :arg JT: Jump table which was declared in the preamble. :result: None. .. inst:: JT = jump_table EBB0, EBB1, ..., EBBn - Declare a jump table in the :term:`function preample`. + Declare a jump table in the :term:`function preamble`. This declares a jump table for use by the :inst:`br_table` indirect branch instruction. Entries in the table are either EBB names, or ``0`` which @@ -433,7 +433,7 @@ dependent. They make it possible to call native functions on the target platform. When calling other Cretonne functions, the flags are not necessary. Functions that are called directly must be declared in the :term:`function -preample`: +preamble`: .. inst:: F = function NAME signature @@ -476,7 +476,7 @@ This simple example illustrates direct function calls and signatures:: return v1 } -Indirect function calls use a signature declared in the preample. +Indirect function calls use a signature declared in the preamble. .. inst:: SIG = signature signature @@ -556,12 +556,12 @@ Local variables One set of restricted memory operations access the current function's stack frame. The stack frame is divided into fixed-size stack slots that are -allocated in the :term:`function preample`. Stack slots are not typed, they +allocated in the :term:`function preamble`. Stack slots are not typed, they simply represent a contiguous sequence of bytes in the stack frame. .. inst:: SS = stack_slot Bytes, Flags... - Allocate a stack slot in the preample. + Allocate a stack slot in the preamble. If no alignment is specified, Cretonne will pick an appropriate alignment for the stack slot based on its size and access patterns. @@ -633,14 +633,14 @@ all process memory. Instead, it is given a small set of memory areas to work in, and all accesses are bounds checked. Cretonne models this through the concept of *heaps*. -A heap is declared in the function preample and can be accessed with restricted +A heap is declared in the function preamble and can be accessed with restricted instructions that trap on out-of-bounds accesses. Heap addresses can be smaller than the native pointer size, for example unsigned :type:`i32` offsets on a 64-bit architecture. .. inst:: H = heap Name - Declare a heap in the function preample. + Declare a heap in the function preamble. This doesn't allocate memory, it just retrieves a handle to a sandbox from the runtime environment. @@ -1034,9 +1034,9 @@ Glossary is not necessary to know when calling it, so it is just an attribute, and not part of the signature. - function preample + function preamble A list of declarations of entities that are used by the function body. - Some of the entities that can be declared in the preample are: + Some of the entities that can be declared in the preamble are: - Local variables. - Functions that are called directly. @@ -1045,7 +1045,7 @@ Glossary function body The extended basic blocks which contain all the executable code in a - function. The function body follows the function preample. + function. The function body follows the function preamble. basic block A maximal sequence of instructions that can only be entered from the diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 5858ef512e..c0b75f4101 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -47,14 +47,14 @@ must be instances of the :class:`Operand` class. .. autoclass:: Operand -Cretonne uses two separate type systems for immediate operands and SSA values. +Cretonne uses two separate type systems for operand kinds and SSA values. Type variables -------------- Instruction descriptions can be made polymorphic by using :class:`Operand` instances that refer to a *type variable* instead of a concrete value type. -Polymorphism only works for SSA value operands. Immediate operands have a fixed +Polymorphism only works for SSA value operands. Other operands have a fixed operand kind. .. autoclass:: TypeVar @@ -95,6 +95,18 @@ indicated with an instance of :class:`ImmediateKind`. .. currentmodule:: cretonne +Entity references +----------------- + +Instruction operands can also refer to other entties in the same function. This +can be extended basic blocks, or entities declared in the function preamble. + +.. autoclass:: EntityRefKind + +.. automodule:: cretonne.entities + :members: + +.. currentmodule:: cretonne Value types ----------- @@ -133,13 +145,14 @@ representation depends on the input operand kinds and whether the instruction can produce multiple results. .. autoclass:: OperandKind +.. inheritance-diagram:: OperandKind ImmediateKind EntityRefkind Since all SSA value operands are represented as a `Value` in Rust code, value types don't affect the representation. Two special operand kinds are used to represent SSA values: .. autodata:: value -.. autodata:: args +.. autodata:: variable_args When an instruction description is created, it is automatically assigned a predefined instruction format which is an instance of diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index f390f0dcfa..ab37e0896a 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -300,19 +300,19 @@ impl<'a> Parser<'a> { // Parse a whole function definition. // - // function ::= * function-spec "{" preample function-body "}" + // function ::= * function-spec "{" preamble function-body "}" // fn parse_function(&mut self) -> Result { let (name, sig) = try!(self.parse_function_spec()); let mut ctx = Context::new(Function::with_name_signature(name, sig)); - // function ::= function-spec * "{" preample function-body "}" + // function ::= function-spec * "{" preamble function-body "}" try!(self.match_token(Token::LBrace, "expected '{' before function body")); - // function ::= function-spec "{" * preample function-body "}" + // function ::= function-spec "{" * preamble function-body "}" try!(self.parse_preamble(&mut ctx)); - // function ::= function-spec "{" preample * function-body "}" + // function ::= function-spec "{" preamble * function-body "}" try!(self.parse_function_body(&mut ctx)); - // function ::= function-spec "{" preample function-body * "}" + // function ::= function-spec "{" preamble function-body * "}" try!(self.match_token(Token::RBrace, "expected '}' after function body")); Ok(ctx.function) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 8ce84394da..0447cdfcc0 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -42,6 +42,12 @@ class OperandKind(object): def __repr__(self): return 'OperandKind({})'.format(self.name) + def operand_kind(self): + """ + An `OperandKind` instance can be used directly as the type of an + `Operand` when defining an instruction. + """ + return self #: An SSA value operand. This is a value defined by another instruction. value = OperandKind( @@ -55,8 +61,8 @@ value = OperandKind( #: A variable-sized list of value operands. Use for Ebb and function call #: arguments. -args = OperandKind( - 'args', """ +variable_args = OperandKind( + 'variable_args', """ A variable size list of `value` operands. Use this to represent arguemtns passed to a function call, arguments @@ -65,8 +71,8 @@ args = OperandKind( """) -# Instances of immediate operand types are provided in the cretonne.immediates -# module. +# Instances of immediate operand types are provided in the +# `cretonne.immediates` module. class ImmediateKind(OperandKind): """ The kind of an immediate instruction operand. @@ -79,12 +85,20 @@ class ImmediateKind(OperandKind): def __repr__(self): return 'ImmediateKind({})'.format(self.name) - def operand_kind(self): - """ - An `ImmediateKind` instance can be used directly as the type of an - `Operand` when defining an instruction. - """ - return self + +# Instances of entity reference operand types are provided in the +# `cretonne.entities` module. +class EntityRefKind(OperandKind): + """ + The kind of an entity reference instruction operand. + """ + + def __init__(self, name, doc): + self.name = name + self.__doc__ = doc + + def __repr__(self): + return 'EntityRefKind({})'.format(self.name) # ValueType instances (i8, i32, ...) are provided in the cretonne.types module. @@ -315,8 +329,8 @@ class InstructionGroup(object): class Operand(object): """ - An instruction operand can be either an *immediate* or an *SSA value*. The - type of the operand is one of: + An instruction operand can be an *immediate*, an *SSA value*, or an *entity + reference*. The type of the operand is one of: 1. A :py:class:`ValueType` instance indicates an SSA value operand with a concrete type. @@ -329,6 +343,10 @@ class Operand(object): whose value is encoded in the instruction itself rather than being passed as an SSA value. + 4. An :py:class:`EntityRefKind` instance indicates an operand that + references another entity in the function, typically something declared + in the function preamble. + """ def __init__(self, name, typ, doc=''): self.name = name @@ -429,9 +447,9 @@ class Instruction(object): :param name: Instruction mnemonic, also becomes opcode name. :param doc: Documentation string. :param ins: Tuple of input operands. This can be a mix of SSA value - operands and immediate operands. - :param outs: Tuple of output operands. The output operands can't be - immediates. + operands and other operand kinds. + :param outs: Tuple of output operands. The output operands must be SSA + values. """ def __init__(self, name, doc, ins=(), outs=(), **kwargs): diff --git a/meta/cretonne/entities.py b/meta/cretonne/entities.py new file mode 100644 index 0000000000..67265ee844 --- /dev/null +++ b/meta/cretonne/entities.py @@ -0,0 +1,23 @@ +""" +The `cretonne.entities` module predefines all the Cretonne entity reference +operand types. Thee are corresponding definitions in the `cretonne.entities` +Rust module. +""" + +from . import EntityRefKind + + +#: A reference to an extended basic block in the same function. +#: This is primarliy used in control flow instructions. +ebb = EntityRefKind('ebb', 'An extended basic block in the same function.') + +#: A reference to a stack slot declared in the function preamble. +stack_slot = EntityRefKind('stack_slot', 'A stack slot.') + +#: A reference to a function sugnature declared in the function preamble. +#: Tbis is used to provide the call signature in an indirect call instruction. +signature = EntityRefKind('signature', 'A function signature.') + +#: A reference to an external function declared in the function preamble. +#: This is used to provide the callee and signature in a call instruction. +function = EntityRefKind('function', 'An external function.') diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index cb02d80998..9ea555a257 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -7,8 +7,9 @@ in this module. """ -from . import InstructionFormat, value, args +from . import InstructionFormat, value, variable_args from immediates import imm64, ieee32, ieee64, immvector +from entities import function Nullary = InstructionFormat() @@ -22,7 +23,7 @@ Binary = InstructionFormat(value, value) BinaryImm = InstructionFormat(value, imm64) BinaryImmRev = InstructionFormat(imm64, value) -Call = InstructionFormat(args, multiple_results=True) +Call = InstructionFormat(function, variable_args, multiple_results=True) # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) diff --git a/meta/cretonne/immediates.py b/meta/cretonne/immediates.py index af7bd72feb..e8e581d64d 100644 --- a/meta/cretonne/immediates.py +++ b/meta/cretonne/immediates.py @@ -1,5 +1,5 @@ """ -The cretonne.immediates module predefines all the Cretonne immediate operand +The `cretonne.immediates` module predefines all the Cretonne immediate operand types. """ From b06f5ef72ff51e54f0f0964cfe95d591ed3eeaf3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 18 May 2016 16:28:21 -0700 Subject: [PATCH 0084/3084] Define control flow instructions. Rename 'br' to 'jump'. We'll use jump/br to mean unconditional/conditional control transfer respectively. --- cranelift/docs/cton_domain.py | 7 +- cranelift/docs/langref.rst | 80 ++-------------- cranelift/src/libcretonne/entities.rs | 31 +++++++ cranelift/src/libcretonne/instructions.rs | 108 +++++++++++++++++++++- cranelift/src/libcretonne/write.rs | 5 +- cranelift/src/libreader/parser.rs | 3 + meta/cretonne/__init__.py | 2 + meta/cretonne/base.py | 78 +++++++++++++++- meta/cretonne/entities.py | 3 + meta/cretonne/formats.py | 6 +- 10 files changed, 244 insertions(+), 79 deletions(-) diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index 6f20cf8d32..9c5c38e17a 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -262,7 +262,12 @@ class InstDocumenter(sphinx.ext.autodoc.Documenter): if len(inst.outs) > 0: sig = ', '.join([op.name for op in inst.outs]) + ' = ' + sig if len(inst.ins) > 0: - sig = sig + ' ' + ', '.join([op.name for op in inst.ins]) + sig = sig + ' ' + inst.ins[0].name + for op in inst.ins[1:]: + if op.typ.operand_kind().name == 'variable_args': + sig += '({}...)'.format(op.name) + else: + sig += ', ' + op.name return sig def add_directive_header(self, sig): diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 1d987fb3c7..ca58a9b273 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -313,56 +313,10 @@ arguments, if it has any. Conditional branches only take the branch if their condition is satisfied, otherwise execution continues at the following instruction in the EBB. -.. inst:: br EBB(args...) - - Branch. - - Unconditionally branch to an extended basic block, passing the specified - EBB arguments. The number and types of arguments must match the destination - EBB. - - :arg EBB: Destination extended basic block. - :arg args...: Zero or more arguments passed to EBB. - :result: None. This is a terminator instruction. - -.. inst:: brz x, EBB(args...) - - Branch when zero. - - If ``x`` is a :type:`b1` value, take the branch when ``x`` is false. If - ``x`` is an integer value, take the branch when ``x = 0``. - - :arg Testable x: Value to test. - :arg EBB: Destination extended basic block. - :arg args...: Arguments passed to EBB. - :result: None. - -.. inst:: brnz x, EBB(args...) - - Branch when non-zero. - - If ``x`` is a :type:`b1` value, take the branch when ``x`` is true. If - ``x`` is an integer value, take the branch when ``x != 0``. - - :arg Testable x: Value to test. - :arg EBB: Destination extended basic block. - :arg args...: Zero or more arguments passed to EBB. - :result: None. - -.. inst:: br_table x, JT - - Jump table branch. - - Use ``x`` as an unsigned index into the jump table ``JT``. If a jump table - entry is found, branch to the corresponding EBB. If no entry was found fall - through to the next instruction. - - Note that this branch instruction can't pass arguments to the targeted - blocks. Split critical edges as needed to work around this. - - :arg iN x: Integer index into jump table. - :arg JT: Jump table which was declared in the preamble. - :result: None. +.. autoinst:: jump +.. autoinst:: brz +.. autoinst:: brnz +.. autoinst:: br_table .. inst:: JT = jump_table EBB0, EBB1, ..., EBBn @@ -386,29 +340,9 @@ explicit trap instructions defined below, but some instructions may also cause traps for certain input value. For example, :inst:`udiv` traps when the divisor is zero. -.. inst:: trap - - Terminate execution unconditionally. - - :result: None. This is a terminator instruction. - -.. inst:: trapz x - - Trap when zero. - - if ``x`` is non-zero, execution continues at the following instruction. - - :arg Testable x: Value to test. - :result: None. - -.. inst:: trapnz x - - Trap when non-zero. - - if ``x`` is zero, execution continues at the following instruction. - - :arg Testable x: Value to test. - :result: None. +.. autoinst:: trap +.. autoinst:: trapz +.. autoinst:: trapnz Function calls diff --git a/cranelift/src/libcretonne/entities.rs b/cranelift/src/libcretonne/entities.rs index d4e1cb849b..8513447ab2 100644 --- a/cranelift/src/libcretonne/entities.rs +++ b/cranelift/src/libcretonne/entities.rs @@ -189,3 +189,34 @@ impl Default for StackSlot { NO_STACK_SLOT } } + +/// An opaque reference to a jump table. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct JumpTable(u32); + +impl JumpTable { + pub fn new(index: usize) -> JumpTable { + assert!(index < (u32::MAX as usize)); + JumpTable(index as u32) + } + + pub fn index(&self) -> usize { + self.0 as usize + } +} + +/// Display a `JumpTable` reference as "jt12". +impl Display for JumpTable { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write!(fmt, "jt{}", self.0) + } +} + +/// A guaranteed invalid jump table reference. +pub const NO_JUMP_TABLE: JumpTable = JumpTable(u32::MAX); + +impl Default for JumpTable { + fn default() -> JumpTable { + NO_JUMP_TABLE + } +} diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index 39202b000f..3cb1ba5378 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -140,6 +140,22 @@ pub enum InstructionData { rhs: Value, lhs: Imm64, }, + Jump { + opcode: Opcode, + ty: Type, + data: Box, + }, + Branch { + opcode: Opcode, + ty: Type, + data: Box, + }, + BranchTable { + opcode: Opcode, + ty: Type, + arg: Value, + table: JumpTable, + }, Call { opcode: Opcode, ty: Type, @@ -147,6 +163,66 @@ pub enum InstructionData { }, } +/// A variable list of `Value` operands used for function call arguments and passing arguments to +/// basic blocks. +#[derive(Debug)] +pub struct VariableArgs(Vec); + +impl VariableArgs { + pub fn new() -> VariableArgs { + VariableArgs(Vec::new()) + } +} + +impl Display for VariableArgs { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + try!(write!(fmt, "(")); + for (i, val) in self.0.iter().enumerate() { + if i == 0 { + try!(write!(fmt, "{}", val)); + } else { + try!(write!(fmt, ", {}", val)); + } + } + write!(fmt, ")") + } +} + +impl Default for VariableArgs { + fn default() -> VariableArgs { + VariableArgs::new() + } +} + +/// Payload data for jump instructions. These need to carry lists of EBB arguments that won't fit +/// in the allowed InstructionData size. +#[derive(Debug)] +pub struct JumpData { + destination: Ebb, + arguments: VariableArgs, +} + +impl Display for JumpData { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + write!(f, "{}{}", self.destination, self.arguments) + } +} + +/// Payload data for branch instructions. These need to carry lists of EBB arguments that won't fit +/// in the allowed InstructionData size. +#[derive(Debug)] +pub struct BranchData { + arg: Value, + destination: Ebb, + arguments: VariableArgs, +} + +impl Display for BranchData { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + write!(f, "{}, {}{}", self.arg, self.destination, self.arguments) + } +} + /// Payload of a call instruction. #[derive(Debug)] pub struct CallData { @@ -154,9 +230,14 @@ pub struct CallData { second_result: Value, // Dynamically sized array containing call argument values. - arguments: Vec, + arguments: VariableArgs, } +impl Display for CallData { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + write!(f, "TBD{}", self.arguments) + } +} impl InstructionData { /// Create data for a call instruction. @@ -166,7 +247,7 @@ impl InstructionData { ty: return_type, data: Box::new(CallData { second_result: NO_VALUE, - arguments: Vec::new(), + arguments: VariableArgs::new(), }), } } @@ -184,6 +265,9 @@ impl InstructionData { Binary { opcode, .. } => opcode, BinaryImm { opcode, .. } => opcode, BinaryImmRev { opcode, .. } => opcode, + Jump { opcode, .. } => opcode, + Branch { opcode, .. } => opcode, + BranchTable { opcode, .. } => opcode, Call { opcode, .. } => opcode, } } @@ -201,6 +285,9 @@ impl InstructionData { Binary { ty, .. } => ty, BinaryImm { ty, .. } => ty, BinaryImmRev { ty, .. } => ty, + Jump { ty, .. } => ty, + Branch { ty, .. } => ty, + BranchTable { ty, .. } => ty, Call { ty, .. } => ty, } } @@ -218,6 +305,9 @@ impl InstructionData { Binary { .. } => None, BinaryImm { .. } => None, BinaryImmRev { .. } => None, + Jump { .. } => None, + Branch { .. } => None, + BranchTable { .. } => None, Call { ref data, .. } => Some(data.second_result), } } @@ -234,6 +324,9 @@ impl InstructionData { Binary { .. } => None, BinaryImm { .. } => None, BinaryImmRev { .. } => None, + Jump { .. } => None, + Branch { .. } => None, + BranchTable { .. } => None, Call { ref mut data, .. } => Some(&mut data.second_result), } } @@ -263,4 +356,15 @@ mod tests { assert_eq!("".parse::(), Err("Unknown opcode")); assert_eq!("\0".parse::(), Err("Unknown opcode")); } + + #[test] + fn instruction_data() { + use std::mem; + // The size of the InstructionData enum is important for performance. It should not exceed + // 16 bytes. Use `Box` out-of-line payloads for instruction formats that require + // more space than that. + // It would be fine with a data structure smaller than 16 bytes, but what are the odds of + // that? + assert_eq!(mem::size_of::(), 16); + } } diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index 9f5c28c0b6..74ece4790e 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -155,7 +155,10 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { Binary { opcode, args, .. } => writeln!(w, "{} {}, {}", opcode, args[0], args[1]), BinaryImm { opcode, lhs, rhs, .. } => writeln!(w, "{} {}, {}", opcode, lhs, rhs), BinaryImmRev { opcode, lhs, rhs, .. } => writeln!(w, "{} {}, {}", opcode, lhs, rhs), - Call { opcode, .. } => writeln!(w, "{} [...]", opcode), + Jump { opcode, ref data, .. } => writeln!(w, "{} {}", opcode, data), + Branch { opcode, ref data, .. } => writeln!(w, "{} {}", opcode, data), + BranchTable { opcode, arg, table, .. } => writeln!(w, "{} {}, {}", opcode, arg, table), + Call { opcode, ref data, .. } => writeln!(w, "{} {}", opcode, data), } } diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index ab37e0896a..713fa7e829 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -657,6 +657,9 @@ impl<'a> Parser<'a> { rhs: rhs, } } + InstructionFormat::Jump | + InstructionFormat::Branch | + InstructionFormat::BranchTable | InstructionFormat::Call => { unimplemented!(); } diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 0447cdfcc0..961a3a500c 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -450,6 +450,8 @@ class Instruction(object): operands and other operand kinds. :param outs: Tuple of output operands. The output operands must be SSA values. + :param is_terminator: This is a terminator instruction. + :param is_branch: This is a branch instruction. """ def __init__(self, name, doc, ins=(), outs=(), **kwargs): diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 3fd8554cf3..9f77fbf860 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -4,18 +4,94 @@ Cretonne base instruction set. This module defines the basic Cretonne instruction set that all targets support. """ -from . import TypeVar, Operand, Instruction, InstructionGroup +from . import TypeVar, Operand, Instruction, InstructionGroup, variable_args from types import i8, f32, f64 from immediates import imm64, ieee32, ieee64, immvector +import entities instructions = InstructionGroup("base", "Shared base instruction set") Int = TypeVar('Int', 'A scalar or vector integer type', ints=True, simd=True) iB = TypeVar('iB', 'A scalar integer type', ints=True) +Testable = TypeVar( + 'Testable', 'A scalar boolean or integer type', + ints=True, bools=True) TxN = TypeVar( '%Tx%N', 'A SIMD vector type', ints=True, floats=True, bools=True, scalars=False, simd=True) +# +# Control flow +# +c = Operand('c', Testable, doc='Controlling value to test') +EBB = Operand('EBB', entities.ebb, doc='Destination extended basic block') +args = Operand('args', variable_args, doc='EBB arguments') + +jump = Instruction( + 'jump', r""" + Jump. + + Unconditionally jump to an extended basic block, passing the specified + EBB arguments. The number and types of arguments must match the + destination EBB. + """, + ins=(EBB, args), is_terminator=True) + +brz = Instruction( + 'brz', r""" + Branch when zero. + + If ``c`` is a :type:`b1` value, take the branch when ``c`` is false. If + ``c`` is an integer value, take the branch when ``c = 0``. + """, + ins=(c, EBB, args), is_branch=True) + +brnz = Instruction( + 'brnz', r""" + Branch when non-zero. + + If ``c`` is a :type:`b1` value, take the branch when ``c`` is true. If + ``c`` is an integer value, take the branch when ``c != 0``. + """, + ins=(c, EBB, args), is_branch=True) + +x = Operand('x', iB, doc='index into jump table') +JT = Operand('JT', entities.jump_table) +br_table = Instruction( + 'br_table', r""" + Indirect branch via jump table. + + Use ``x`` as an unsigned index into the jump table ``JT``. If a jump table + entry is found, branch to the corresponding EBB. If no entry was found fall + through to the next instruction. + + Note that this branch instruction can't pass arguments to the targeted + blocks. Split critical edges as needed to work around this. + """, + ins=(x, JT), is_branch=True) + +trap = Instruction( + 'trap', r""" + Terminate execution unconditionally. + """, + is_terminator=True) + +trapz = Instruction( + 'trapz', r""" + Trap when zero. + + if ``c`` is non-zero, execution continues at the following instruction. + """, + ins=c) + +trapnz = Instruction( + 'trapnz', r""" + Trap when non-zero. + + if ``c`` is zero, execution continues at the following instruction. + """, + ins=c) + # # Materializing constants. # diff --git a/meta/cretonne/entities.py b/meta/cretonne/entities.py index 67265ee844..14587d402f 100644 --- a/meta/cretonne/entities.py +++ b/meta/cretonne/entities.py @@ -21,3 +21,6 @@ signature = EntityRefKind('signature', 'A function signature.') #: A reference to an external function declared in the function preamble. #: This is used to provide the callee and signature in a call instruction. function = EntityRefKind('function', 'An external function.') + +#: A reference to a jump table declared in the function preamble. +jump_table = EntityRefKind('jump_table', 'A jump table.') diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index 9ea555a257..8b32936d7c 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -9,7 +9,7 @@ in this module. from . import InstructionFormat, value, variable_args from immediates import imm64, ieee32, ieee64, immvector -from entities import function +from entities import ebb, function, jump_table Nullary = InstructionFormat() @@ -23,6 +23,10 @@ Binary = InstructionFormat(value, value) BinaryImm = InstructionFormat(value, imm64) BinaryImmRev = InstructionFormat(imm64, value) +Jump = InstructionFormat(ebb, variable_args) +Branch = InstructionFormat(value, ebb, variable_args) +BranchTable = InstructionFormat(value, jump_table) + Call = InstructionFormat(function, variable_args, multiple_results=True) # Finally extract the names of global variables in this module. From 6d0486cced92034db95414bcc8a56ca236711804 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 May 2016 10:16:40 -0700 Subject: [PATCH 0085/3084] Auto-generate boilerplate for 'impl InstructionData'. Accessors for shared fields and multiple results can be generated automatically. Add a 'boxed_storage' flag to the instruction format definitions to enable generated code to access 'data'. --- cranelift/src/libcretonne/instructions.rs | 79 --------------------- meta/cretonne/__init__.py | 4 ++ meta/cretonne/formats.py | 7 +- meta/gen_instr.py | 85 +++++++++++++++++++++++ 4 files changed, 93 insertions(+), 82 deletions(-) diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index 3cb1ba5378..50cd4d8aad 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -251,85 +251,6 @@ impl InstructionData { }), } } - - /// Get the opcode of this instruction. - pub fn opcode(&self) -> Opcode { - use self::InstructionData::*; - match *self { - Nullary { opcode, .. } => opcode, - Unary { opcode, .. } => opcode, - UnaryImm { opcode, .. } => opcode, - UnaryIeee32 { opcode, .. } => opcode, - UnaryIeee64 { opcode, .. } => opcode, - UnaryImmVector { opcode, .. } => opcode, - Binary { opcode, .. } => opcode, - BinaryImm { opcode, .. } => opcode, - BinaryImmRev { opcode, .. } => opcode, - Jump { opcode, .. } => opcode, - Branch { opcode, .. } => opcode, - BranchTable { opcode, .. } => opcode, - Call { opcode, .. } => opcode, - } - } - - /// Type of the first result. - pub fn first_type(&self) -> Type { - use self::InstructionData::*; - match *self { - Nullary { ty, .. } => ty, - Unary { ty, .. } => ty, - UnaryImm { ty, .. } => ty, - UnaryIeee32 { ty, .. } => ty, - UnaryIeee64 { ty, .. } => ty, - UnaryImmVector { ty, .. } => ty, - Binary { ty, .. } => ty, - BinaryImm { ty, .. } => ty, - BinaryImmRev { ty, .. } => ty, - Jump { ty, .. } => ty, - Branch { ty, .. } => ty, - BranchTable { ty, .. } => ty, - Call { ty, .. } => ty, - } - } - - /// Second result value, if any. - pub fn second_result(&self) -> Option { - use self::InstructionData::*; - match *self { - Nullary { .. } => None, - Unary { .. } => None, - UnaryImm { .. } => None, - UnaryIeee32 { .. } => None, - UnaryIeee64 { .. } => None, - UnaryImmVector { .. } => None, - Binary { .. } => None, - BinaryImm { .. } => None, - BinaryImmRev { .. } => None, - Jump { .. } => None, - Branch { .. } => None, - BranchTable { .. } => None, - Call { ref data, .. } => Some(data.second_result), - } - } - - pub fn second_result_mut<'a>(&'a mut self) -> Option<&'a mut Value> { - use self::InstructionData::*; - match *self { - Nullary { .. } => None, - Unary { .. } => None, - UnaryImm { .. } => None, - UnaryIeee32 { .. } => None, - UnaryIeee64 { .. } => None, - UnaryImmVector { .. } => None, - Binary { .. } => None, - BinaryImm { .. } => None, - BinaryImmRev { .. } => None, - Jump { .. } => None, - Branch { .. } => None, - BranchTable { .. } => None, - Call { ref mut data, .. } => Some(&mut data.second_result), - } - } } #[cfg(test)] diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 961a3a500c..5df6931aec 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -384,6 +384,9 @@ class InstructionFormat(object): enums. :param multiple_results: Set to `True` if this instruction format allows more than one result to be produced. + :param boxed_storage: Set to `True` is this instruction format requires a + `data: Box<...>` pointer to additional storage in its `InstructionData` + variant. """ # Map (multiple_results, kind, kind, ...) -> InstructionFormat @@ -396,6 +399,7 @@ class InstructionFormat(object): self.name = kwargs.get('name', None) self.kinds = kinds self.multiple_results = kwargs.get('multiple_results', False) + self.boxed_storage = kwargs.get('boxed_storage', False) # Compute a signature for the global registry. sig = (self.multiple_results,) + kinds if sig in InstructionFormat._registry: diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index 8b32936d7c..319dc6fb69 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -23,11 +23,12 @@ Binary = InstructionFormat(value, value) BinaryImm = InstructionFormat(value, imm64) BinaryImmRev = InstructionFormat(imm64, value) -Jump = InstructionFormat(ebb, variable_args) -Branch = InstructionFormat(value, ebb, variable_args) +Jump = InstructionFormat(ebb, variable_args, boxed_storage=True) +Branch = InstructionFormat(value, ebb, variable_args, boxed_storage=True) BranchTable = InstructionFormat(value, jump_table) -Call = InstructionFormat(function, variable_args, multiple_results=True) +Call = InstructionFormat( + function, variable_args, multiple_results=True, boxed_storage=True) # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 8fe63cb932..c055e87ffa 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -36,6 +36,89 @@ def gen_formats(fmt): fmt.line() +def gen_instruction_data_impl(fmt): + """ + Generate the boring parts of the InstructionData implementation. + + These methods in `impl InstructionData` can be generated automatically from + the instruction formats: + + - `pub fn opcode(&self) -> Opcode` + - `pub fn first_type(&self) -> Type` + - `pub fn second_result(&self) -> Option` + - `pub fn second_result_mut<'a>(&'a mut self) -> Option<&'a mut Value>` + """ + + # The `opcode` and `first_type` methods simply read the `opcode` and `ty` + # members. This is really a workaround for Rust's enum types missing shared + # members. + with fmt.indented('impl InstructionData {', '}'): + fmt.doc_comment('Get the opcode of this instruction.') + with fmt.indented('pub fn opcode(&self) -> Opcode {', '}'): + with fmt.indented('match *self {', '}'): + for f in cretonne.InstructionFormat.all_formats: + fmt.line( + 'InstructionData::{} {{ opcode, .. }} => opcode,' + .format(f.name)) + + fmt.doc_comment('Type of the first result, or `VOID`.') + with fmt.indented('pub fn first_type(&self) -> Type {', '}'): + with fmt.indented('match *self {', '}'): + for f in cretonne.InstructionFormat.all_formats: + fmt.line( + 'InstructionData::{} {{ ty, .. }} => ty,' + .format(f.name)) + + # Generate shared and mutable accessors for `second_result` which only + # applies to instruction formats that can produce multiple results. + # Everything else returns `None`. + fmt.doc_comment('Second result value, if any.') + with fmt.indented( + 'pub fn second_result(&self) -> Option {', '}'): + with fmt.indented('match *self {', '}'): + for f in cretonne.InstructionFormat.all_formats: + if not f.multiple_results: + # Single or no results. + fmt.line( + 'InstructionData::{} {{ .. }} => None,' + .format(f.name)) + elif f.boxed_storage: + # Multiple results, boxed storage. + fmt.line( + 'InstructionData::' + f.name + + ' { ref data, .. }' + + ' => Some(data.second_result),') + else: + # Multiple results, inline storage. + fmt.line( + 'InstructionData::' + f.name + + ' { second_result, .. }' + + ' => Some(second_result),') + + fmt.doc_comment('Mutable reference to second result value, if any.') + with fmt.indented( + "pub fn second_result_mut<'a>(&'a mut self) -> Option<&'a mut Value> {", '}'): + with fmt.indented('match *self {', '}'): + for f in cretonne.InstructionFormat.all_formats: + if not f.multiple_results: + # Single or no results. + fmt.line( + 'InstructionData::{} {{ .. }} => None,' + .format(f.name)) + elif f.boxed_storage: + # Multiple results, boxed storage. + fmt.line( + 'InstructionData::' + f.name + + ' { ref mut data, .. }' + + ' => Some(&mut data.second_result),') + else: + # Multiple results, inline storage. + fmt.line( + 'InstructionData::' + f.name + + ' { ref mut second_result, .. }' + + ' => Some(second_result),') + + def collect_instr_groups(targets): seen = set() groups = [] @@ -108,8 +191,10 @@ def gen_opcodes(groups, fmt): def generate(targets, out_dir): groups = collect_instr_groups(targets) + # opcodes.rs fmt = srcgen.Formatter() gen_formats(fmt) + gen_instruction_data_impl(fmt) gen_opcodes(groups, fmt) fmt.update_file('opcodes.rs', out_dir) From ceb134ac328f16e582a9e0328b5bd80b1d5eda1f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 May 2016 10:43:14 -0700 Subject: [PATCH 0086/3084] Add a BinaryOverflow instruction format. This will eventualy be used for add-with-carry and add-with-overflow type instructions. For now it only serves as a representative of instruction formats that have multiple_results=True and boxed_storage=False at the same time. --- cranelift/src/libcretonne/instructions.rs | 8 +++++++- cranelift/src/libcretonne/write.rs | 1 + cranelift/src/libreader/parser.rs | 11 +++++++++++ meta/cretonne/formats.py | 3 +++ 4 files changed, 22 insertions(+), 1 deletion(-) diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index 50cd4d8aad..cbc6b053dd 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -133,13 +133,19 @@ pub enum InstructionData { lhs: Value, rhs: Imm64, }, - // Same as BinaryImm, but the imediate is the lhs operand. + // Same as BinaryImm, but the immediate is the lhs operand. BinaryImmRev { opcode: Opcode, ty: Type, rhs: Value, lhs: Imm64, }, + BinaryOverflow { + opcode: Opcode, + ty: Type, + second_result: Value, + args: [Value; 2], + }, Jump { opcode: Opcode, ty: Type, diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index 74ece4790e..d3a676293d 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -155,6 +155,7 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { Binary { opcode, args, .. } => writeln!(w, "{} {}, {}", opcode, args[0], args[1]), BinaryImm { opcode, lhs, rhs, .. } => writeln!(w, "{} {}, {}", opcode, lhs, rhs), BinaryImmRev { opcode, lhs, rhs, .. } => writeln!(w, "{} {}, {}", opcode, lhs, rhs), + BinaryOverflow { opcode, args, .. } => writeln!(w, "{} {}, {}", opcode, args[0], args[1]), Jump { opcode, ref data, .. } => writeln!(w, "{} {}", opcode, data), Branch { opcode, ref data, .. } => writeln!(w, "{} {}", opcode, data), BranchTable { opcode, arg, table, .. } => writeln!(w, "{} {}, {}", opcode, arg, table), diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 713fa7e829..2247259530 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -657,6 +657,17 @@ impl<'a> Parser<'a> { rhs: rhs, } } + InstructionFormat::BinaryOverflow => { + let lhs = try!(self.match_value("expected SSA value first operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let rhs = try!(self.match_value("expected SSA value second operand")); + InstructionData::BinaryOverflow { + opcode: opcode, + ty: VOID, + second_result: NO_VALUE, + args: [lhs, rhs], + } + } InstructionFormat::Jump | InstructionFormat::Branch | InstructionFormat::BranchTable | diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index 319dc6fb69..261b400321 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -23,6 +23,9 @@ Binary = InstructionFormat(value, value) BinaryImm = InstructionFormat(value, imm64) BinaryImmRev = InstructionFormat(imm64, value) +# Generate result + overflow flag. +BinaryOverflow = InstructionFormat(value, value, multiple_results=True) + Jump = InstructionFormat(ebb, variable_args, boxed_storage=True) Branch = InstructionFormat(value, ebb, variable_args, boxed_storage=True) BranchTable = InstructionFormat(value, jump_table) From c3b76b67ca156be9d4806e883054997699355e81 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 May 2016 11:21:36 -0700 Subject: [PATCH 0087/3084] Verify restrictions on polymorphism. Add a typevar_operand argument to the InstructionFormat constructor which determines the operand used for inferring the controlling type variable. Identify polymorphic instructions when they are created, determine if the controlling type variable can be inferred from the typevar_operand, and verify the use of type variables in the other operands. Generate type variable summary in the documentation, including how the controlling type variable is inferred. --- cranelift/docs/cton_domain.py | 60 ++++++++++++---- cranelift/docs/metaref.rst | 11 ++- meta/cretonne/__init__.py | 129 +++++++++++++++++++++++++++++++++- meta/gen_instr.py | 6 ++ 4 files changed, 187 insertions(+), 19 deletions(-) diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index 9c5c38e17a..bd207e994d 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -18,13 +18,14 @@ from docutils.parsers.rst import directives from sphinx import addnodes from sphinx.directives import ObjectDescription from sphinx.domains import Domain, ObjType -from sphinx.locale import l_, _ +from sphinx.locale import l_ from sphinx.roles import XRefRole from sphinx.util.docfields import Field, GroupedField, TypedField from sphinx.util.nodes import make_refnode import sphinx.ext.autodoc + class CtonObject(ObjectDescription): """ Any kind of Cretonne IL object. @@ -68,10 +69,11 @@ class CtonObject(ObjectDescription): # Type variables are indicated as %T. typevar = re.compile('(\%[A-Z])') + def parse_type(name, signode): """ Parse a type with embedded type vars and append to signode. - + Return a a string that can be compiled into a regular expression matching the type. """ @@ -92,6 +94,7 @@ def parse_type(name, signode): re_str += re.escape(part) return re_str + class CtonType(CtonObject): """A Cretonne IL type description.""" @@ -103,7 +106,7 @@ class CtonType(CtonObject): """ name = sig.strip() - re_str = parse_type(name, signode) + parse_type(name, signode) return name def get_index_text(self, name): @@ -112,12 +115,14 @@ class CtonType(CtonObject): sep_equal = re.compile('\s*=\s*') sep_comma = re.compile('\s*,\s*') + def parse_params(s, signode): - for i,p in enumerate(sep_comma.split(s)): + for i, p in enumerate(sep_comma.split(s)): if i != 0: signode += nodes.Text(', ') signode += nodes.emphasis(p, p) + class CtonInst(CtonObject): """A Cretonne IL instruction.""" @@ -128,6 +133,7 @@ class CtonInst(CtonObject): TypedField('result', label=l_('Results'), names=('out', 'result'), typerolename='type', typenames=('type',)), + GroupedField('typevar', names=('typevar',), label=l_('Type Variables')), GroupedField('flag', names=('flag',), label=l_('Flags')), Field('resulttype', label=l_('Result type'), has_arg=False, names=('rtype',)), @@ -164,24 +170,25 @@ class CtonInst(CtonObject): def get_index_text(self, name): return name + class CretonneDomain(Domain): """Cretonne domain for intermediate language objects.""" name = 'cton' label = 'Cretonne' object_types = { - 'type' : ObjType(l_('type'), 'type'), - 'inst' : ObjType(l_('instruction'), 'inst') + 'type': ObjType(l_('type'), 'type'), + 'inst': ObjType(l_('instruction'), 'inst') } directives = { - 'type' : CtonType, - 'inst' : CtonInst, + 'type': CtonType, + 'inst': CtonInst, } roles = { - 'type' : XRefRole(), - 'inst' : XRefRole(), + 'type': XRefRole(), + 'inst': XRefRole(), } initial_data = { @@ -230,7 +237,7 @@ class TypeDocumenter(sphinx.ext.autodoc.Documenter): return False def resolve_name(self, modname, parents, path, base): - return 'cretonne.types', [ base ] + return 'cretonne.types', [base] def add_content(self, more_content, no_docstring=False): super(TypeDocumenter, self).add_content(more_content, no_docstring) @@ -254,7 +261,7 @@ class InstDocumenter(sphinx.ext.autodoc.Documenter): return False def resolve_name(self, modname, parents, path, base): - return 'cretonne.base', [ base ] + return 'cretonne.base', [base] def format_signature(self): inst = self.object @@ -285,9 +292,32 @@ class InstDocumenter(sphinx.ext.autodoc.Documenter): # Add inputs and outputs. for op in self.object.ins: - self.add_line(u':in {} {}: {}'.format(op.typ.name, op.name, op.get_doc()), sourcename) + self.add_line(u':in {} {}: {}'.format( + op.typ.name, op.name, op.get_doc()), sourcename) for op in self.object.outs: - self.add_line(u':out {} {}: {}'.format(op.typ.name, op.name, op.get_doc()), sourcename) + self.add_line(u':out {} {}: {}'.format( + op.typ.name, op.name, op.get_doc()), sourcename) + + # Document type inference for polymorphic instructions. + if self.object.is_polymorphic: + if self.object.ctrl_typevar is not None: + if self.object.use_typevar_operand: + self.add_line( + u':typevar {}: inferred from {}' + .format( + self.object.ctrl_typevar.name, + self.object.ins[ + self.object.format.typevar_operand]), + sourcename) + else: + self.add_line( + u':typevar {}: explicitly provided' + .format(self.object.ctrl_typevar.name), + sourcename) + for tv in self.object.other_typevars: + self.add_line( + u':typevar {}: from input operand'.format(tv.name), + sourcename) def setup(app): @@ -295,4 +325,4 @@ def setup(app): app.add_autodocumenter(TypeDocumenter) app.add_autodocumenter(InstDocumenter) - return { 'version' : '0.1' } + return {'version': '0.1'} diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index c0b75f4101..65b8f88654 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -145,7 +145,7 @@ representation depends on the input operand kinds and whether the instruction can produce multiple results. .. autoclass:: OperandKind -.. inheritance-diagram:: OperandKind ImmediateKind EntityRefkind +.. inheritance-diagram:: OperandKind ImmediateKind EntityRefKind Since all SSA value operands are represented as a `Value` in Rust code, value types don't affect the representation. Two special operand kinds are used to @@ -174,8 +174,13 @@ constraints, in practice more freedom than what is needed for normal instruction set architectures. In order to simplify the Rust representation of value type constraints, some restrictions are imposed on the use of type variables. -A polymorphic instruction has a single *controlling type variable*. The value -types of instruction results must be one of the following: +A polymorphic instruction has a single *controlling type variable*. For a given +opcode, this type variable must be the type of the first result or the type of +the input value operand designated by the `typevar_operand` argument to the +:py:class:`InstructionFormat` constructor. By default, this is the first value +operand, which works most of the time. + +The value types of instruction results must be one of the following: 1. A concrete value type. 2. The controlling type variable. diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 5df6931aec..924ce8a109 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -49,6 +49,10 @@ class OperandKind(object): """ return self + def free_typevar(self): + # Return the free typevariable controlling the type of this operand. + return None + #: An SSA value operand. This is a value defined by another instruction. value = OperandKind( 'value', """ @@ -125,6 +129,9 @@ class ValueType(object): """ return value + def free_typevar(self): + return None + class ScalarType(ValueType): """ @@ -253,6 +260,10 @@ class TypeVar(object): scalars=True, simd=False): self.name = name self.__doc__ = doc + self.base = base + + def __str__(self): + return "`{}`".format(self.name) def lane(self): """ @@ -277,6 +288,11 @@ class TypeVar(object): # value. return value + def free_typevar(self): + if isinstance(self.base, TypeVar): + return self.base + else: + return self # Defining instructions. @@ -360,6 +376,9 @@ class Operand(object): else: return self.typ.__doc__ + def __str__(self): + return "`{}`".format(self.name) + class InstructionFormat(object): """ @@ -387,6 +406,9 @@ class InstructionFormat(object): :param boxed_storage: Set to `True` is this instruction format requires a `data: Box<...>` pointer to additional storage in its `InstructionData` variant. + :param typevar_operand: Index of the input operand that is used to infer + the controlling type variable. By default, this is the first `value` + operand. """ # Map (multiple_results, kind, kind, ...) -> InstructionFormat @@ -400,6 +422,20 @@ class InstructionFormat(object): self.kinds = kinds self.multiple_results = kwargs.get('multiple_results', False) self.boxed_storage = kwargs.get('boxed_storage', False) + + # Which of self.kinds are `value`? + self.value_operands = tuple( + i for i, k in enumerate(self.kinds) if k is value) + + # The typevar_operand argument must point to a 'value' operand. + self.typevar_operand = kwargs.get('typevar_operand', None) + if self.typevar_operand is not None: + assert self.kinds[self.typevar_operand] is value, \ + "typevar_operand must indicate a 'value' operand" + elif len(self.value_operands) > 0: + # Default to the first 'value' operand, if there is one. + self.typevar_operand = self.value_operands[0] + # Compute a signature for the global registry. sig = (self.multiple_results,) + kinds if sig in InstructionFormat._registry: @@ -453,7 +489,7 @@ class Instruction(object): :param ins: Tuple of input operands. This can be a mix of SSA value operands and other operand kinds. :param outs: Tuple of output operands. The output operands must be SSA - values. + values or `variable_args`. :param is_terminator: This is a terminator instruction. :param is_branch: This is a branch instruction. """ @@ -465,8 +501,99 @@ class Instruction(object): self.ins = self._to_operand_tuple(ins) self.outs = self._to_operand_tuple(outs) self.format = InstructionFormat.lookup(self.ins, self.outs) + # Indexes into outs for value results. Others are `variable_args`. + self.value_results = tuple( + i for i, o in enumerate(self.outs) if o.kind is value) + self._verify_polymorphic() InstructionGroup.append(self) + def _verify_polymorphic(self): + """ + Check if this instruction is polymorphic, and verify its use of type + variables. + """ + poly_ins = [ + i for i in self.format.value_operands + if self.ins[i].typ.free_typevar()] + poly_outs = [ + i for i, o in enumerate(self.outs) + if o.typ.free_typevar()] + self.is_polymorphic = len(poly_ins) > 0 or len(poly_outs) > 0 + if not self.is_polymorphic: + return + + # Prefer to use the typevar_operand to infer the controlling typevar. + self.use_typevar_operand = False + typevar_error = None + if self.format.typevar_operand is not None: + try: + tv = self.ins[self.format.typevar_operand].typ + if tv is tv.free_typevar(): + self.other_typevars = self._verify_ctrl_typevar(tv) + self.ctrl_typevar = tv + self.use_typevar_operand = True + except RuntimeError as e: + typevar_error = e + + if not self.use_typevar_operand: + # The typevar_operand argument doesn't work. Can we infer from the + # first result instead? + if len(self.outs) == 0: + if typevar_error: + raise typevar_error + else: + raise RuntimeError( + "typevar_operand must be a free type variable") + tv = self.outs[0].typ + if tv is not tv.free_typevar(): + raise RuntimeError("first result must be a free type variable") + self.other_typevars = self._verify_ctrl_typevar(tv) + self.ctrl_typevar = tv + + def _verify_ctrl_typevar(self, ctrl_typevar): + """ + Verify that the use of TypeVars is consistent with `ctrl_typevar` as + the controlling type variable. + + All polymorhic inputs must either be derived from `ctrl_typevar` or be + independent free type variables only used once. + + All polymorphic results must be derived from `ctrl_typevar`. + + Return list of other type variables used, or raise an error. + """ + other_tvs = [] + # Check value inputs. + for opidx in self.format.value_operands: + typ = self.ins[opidx].typ + tv = typ.free_typevar() + # Non-polymorphic or derived form ctrl_typevar is OK. + if tv is None or tv is ctrl_typevar: + continue + # No other derived typevars allowed. + if typ is not tv: + raise RuntimeError( + "type variable {} must not be derived from {}" + .format(typ.name, tv.name)) + # Other free type variables can only be used once each. + if tv in other_tvs: + raise RuntimeError( + "type variable {} can't be used more than once" + .format(tv.name)) + other_tvs.append(tv) + + # Check outputs. + for result in self.outs: + typ = result.typ + tv = typ.free_typevar() + # Non-polymorphic or derived from ctrl_typevar is OK. + if tv is None or tv is ctrl_typevar: + continue + raise RuntimeError( + "type variable in output not derived from ctrl_typevar") + + return other_tvs + @staticmethod def _to_operand_tuple(x): # Allow a single Operand instance instead of the awkward singleton diff --git a/meta/gen_instr.py b/meta/gen_instr.py index c055e87ffa..08cd81f1b0 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -151,6 +151,12 @@ def gen_opcodes(groups, fmt): fmt.doc_comment( '`{}{} {}`. ({})' .format(prefix, i.name, suffix, i.format.name)) + # Document polymorphism. + if i.is_polymorphic: + if i.use_typevar_operand: + fmt.doc_comment( + 'Type inferred from {}.' + .format(i.ins[i.format.typevar_operand])) # Enum variant itself. fmt.line(i.camel_name + ',') fmt.line() From ad01af40e4855b2e880a7028b309ad699e9c7f11 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 20 May 2016 10:45:02 -0700 Subject: [PATCH 0088/3084] Generate value type constraints. Add an Opcode::constraints() method which returns an OpcodeConstraints object. This object provides information on instruction polymorphism and how many results is produced. Generate a list of TypeSet objects for checking free type variables. The type sets are parametrized rather than being represented as fully general sets. Add UniqueTable and UniqueSeqTable classes to the meta code generator. Use for compressing tabular data by removing duplicates. --- cranelift/src/libcretonne/instructions.rs | 163 +++++++++++++++++++++- cranelift/src/libcretonne/types.rs | 14 ++ meta/cretonne/__init__.py | 40 +++++- meta/cretonne/base.py | 6 +- meta/gen_instr.py | 132 +++++++++++++++++- meta/unique_table.py | 68 +++++++++ 6 files changed, 413 insertions(+), 10 deletions(-) create mode 100644 meta/unique_table.py diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index cbc6b053dd..c6b353ae9c 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -11,7 +11,7 @@ use std::str::FromStr; use entities::*; use immediates::*; -use types::Type; +use types::{self, Type}; // Include code generated by `meta/gen_instr.py`. This file contains: // @@ -21,6 +21,12 @@ use types::Type; // - The private `fn opcode_name(Opcode) -> &'static str` function, and // - The hash table `const OPCODE_HASH_TABLE: [Opcode; N]`. // +// For value type constraints: +// +// - The `const OPCODE_CONSTRAINTS : [OpcodeConstraints; N]` table. +// - The `const TYPE_SETS : [ValueTypeSet; N]` table. +// - The `const OPERAND_CONSTRAINTS : [OperandConstraint; N]` table. +// include!(concat!(env!("OUT_DIR"), "/opcodes.rs")); impl Display for Opcode { @@ -38,6 +44,12 @@ impl Opcode { Some(OPCODE_FORMAT[self as usize - 1]) } } + + /// Get the constraint descriptor for this opcode. + /// Panic if this is called on `NotAnOpcode`. + pub fn constraints(self) -> OpcodeConstraints { + OPCODE_CONSTRAINTS[self as usize - 1] + } } // A primitive hash function for matching opcodes. @@ -259,6 +271,155 @@ impl InstructionData { } } + +/// Value type constraints for a given opcode. +/// +/// The `InstructionFormat` determines the constraints on most operands, but `Value` operands and +/// results are not determined by the format. Every `Opcode` has an associated +/// `OpcodeConstraints` object that provides the missing details. +/// +/// Since there can be a lot of opcodes, the `OpcodeConstraints` object is encoded as a bit field +/// by the `meta/gen_instr.py` script. +/// +/// The bit field bits are: +/// +/// Bits 0-2: +/// Number of fixed result values. This does not include `variable_args` results as are +/// produced by call instructions. +/// +/// Bit 3: +/// This opcode is polymorphic and the controlling type variable can be inferred from the +/// designated input operand. This is the `typevar_operand` index given to the +/// `InstructionFormat` meta language object. When bit 0 is not set, the controlling type +/// variable must be the first output value instead. +/// +/// Bits 4-7: +/// Permitted set of types for the controlling type variable as an index into `TYPE_SETS`. +/// +/// Bits 8-15: +/// Offset into `OPERAND_CONSTRAINT` table of the descriptors for this opcode. The first +/// `fixed_results()` entries describe the result constraints, then follows constraints for the +/// fixed `Value` input operands. The number of `Value` inputs isdetermined by the instruction +/// format. +/// +#[derive(Clone, Copy)] +pub struct OpcodeConstraints(u16); + +impl OpcodeConstraints { + /// Can the controlling type variable for this opcode be inferred from the designated value + /// input operand? + /// This also implies that this opcode is polymorphic. + pub fn use_typevar_operand(self) -> bool { + (self.0 & 0x8) != 0 + } + + /// Get the number of *fixed* result values produced by this opcode. + /// This does not include `variable_args` produced by calls. + pub fn fixed_results(self) -> usize { + (self.0 & 0x7) as usize + } + + /// Get the offset into `TYPE_SETS` for the controlling type variable. + /// Returns `None` if the instruction is not polymorphic. + fn typeset_offset(self) -> Option { + let offset = ((self.0 & 0xff) >> 4) as usize; + if offset < TYPE_SETS.len() { + Some(offset) + } else { + None + } + } + + /// Get the offset into OPERAND_CONSTRAINTS where the descriptors for this opcode begin. + fn constraint_offset(self) -> usize { + (self.0 >> 8) as usize + } + + /// Get the value type of result number `n`, having resolved the controlling type variable to + /// `ctrl_type`. + pub fn result_type(self, n: usize, ctrl_type: Type) -> Type { + assert!(n < self.fixed_results(), "Invalid result index"); + OPERAND_CONSTRAINTS[self.constraint_offset() + n] + .resolve(ctrl_type) + .expect("Result constraints can't be free") + } + + /// Get the typeset of allowed types for the controlling type variable in a polymorphic + /// instruction. + pub fn ctrl_typeset(self) -> Option { + self.typeset_offset().map(|offset| TYPE_SETS[offset]) + } +} + +/// A value type set describes the permitted set of types for a type variable. +#[derive(Clone, Copy)] +pub struct ValueTypeSet { + allow_scalars: bool, + allow_simd: bool, + + base: Type, + all_ints: bool, + all_floats: bool, + all_bools: bool, +} + +impl ValueTypeSet { + /// Is `scalar` part of the base type set? + /// + /// Note that the base type set does not have to be included in the type set proper. + fn is_base_type(&self, scalar: Type) -> bool { + scalar == self.base || (self.all_ints && scalar.is_int()) || + (self.all_floats && scalar.is_float()) || (self.all_bools && scalar.is_bool()) + } + + /// Does `typ` belong to this set? + pub fn contains(&self, typ: Type) -> bool { + let allowed = if typ.is_scalar() { + self.allow_scalars + } else { + self.allow_simd + }; + allowed && self.is_base_type(typ.lane_type()) + } +} + +/// Operand constraints. This describes the value type constraints on a single `Value` operand. +enum OperandConstraint { + /// This operand has a concrete value type. + Concrete(Type), + + /// This operand can vary freely within the given type set. + /// The type set is identified by its index into the TYPE_SETS constant table. + Free(u8), + + /// This operand is the same type as the controlling type variable. + Same, + + /// This operand is `ctrlType.lane_type()`. + Lane, + + /// This operand is `ctrlType.as_bool()`. + AsBool, +} + +impl OperandConstraint { + /// Resolve this operand constraint into a concrete value type, given the value of the + /// controlling type variable. + /// Returns `None` if this is a free operand which is independent of the controlling type + /// variable. + pub fn resolve(&self, ctrl_type: Type) -> Option { + use self::OperandConstraint::*; + match *self { + Concrete(t) => Some(t), + Free(_) => None, + Same => Some(ctrl_type), + Lane => Some(ctrl_type.lane_type()), + AsBool => Some(ctrl_type.as_bool()), + } + } +} + + #[cfg(test)] mod tests { use super::*; diff --git a/cranelift/src/libcretonne/types.rs b/cranelift/src/libcretonne/types.rs index 967879c14d..a4ff443541 100644 --- a/cranelift/src/libcretonne/types.rs +++ b/cranelift/src/libcretonne/types.rs @@ -83,6 +83,20 @@ impl Type { } } + /// Get a type with the same number of lanes as this type, but with the lanes replaces by + /// booleans of the same size. + pub fn as_bool(self) -> Type { + // Replace the low 4 bits with the boolean version, preserve the high 4 bits. + let lane = match self.lane_type() { + B8 | I8 => B8, + B16 | I16 => B16, + B32 | I32 | F32 => B32, + B64 | I64 | F64 => B64, + _ => B1, + }; + Type(lane.0 | (self.0 & 0xf0)) + } + /// Is this the VOID type? pub fn is_void(self) -> bool { self == VOID diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 924ce8a109..2c737c2e7e 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -7,6 +7,7 @@ instructions. import re import importlib +from collections import namedtuple camel_re = re.compile('(^|_)([a-z])') @@ -148,6 +149,9 @@ class ScalarType(ValueType): def __repr__(self): return 'ScalarType({})'.format(self.name) + def rust_name(self): + return 'types::' + self.name.upper() + def by(self, lanes): """ Get a vector type with this type as the lane type. @@ -232,6 +236,19 @@ class BoolType(ScalarType): # Parametric polymorphism. +#: A `TypeSet` represents a set of types. We don't allow arbitrary subsets of +#: types, but use a parametrized approach instead. +#: This is represented as a named tuple so it can be used as a dictionary key. +TypeSet = namedtuple( + 'TypeSet', [ + 'allow_scalars', + 'allow_simd', + 'base', + 'all_ints', + 'all_floats', + 'all_bools']) + + class TypeVar(object): """ Type variables can be used in place of concrete types when defining @@ -257,10 +274,23 @@ class TypeVar(object): def __init__( self, name, doc, base=None, ints=False, floats=False, bools=False, - scalars=True, simd=False): + scalars=True, simd=False, + derived_func=None): self.name = name self.__doc__ = doc self.base = base + self.is_derived = isinstance(base, TypeVar) + if self.is_derived: + assert derived_func + self.derived_func = derived_func + else: + self.type_set = TypeSet( + allow_scalars=scalars, + allow_simd=simd, + base=base, + all_ints=ints, + all_floats=floats, + all_bools=bools) def __str__(self): return "`{}`".format(self.name) @@ -273,14 +303,18 @@ class TypeVar(object): When this type variable assumes a scalar type, the derived type will be the same scalar type. """ - return TypeVar("Lane type of " + self.name, '', base=self, simd=False) + return TypeVar( + "Lane type of " + self.name, '', + base=self, derived_func='Lane') def as_bool(self): """ Return a derived type variable that has the same vector geometry as this type variable, but with boolean lanes. Scalar types map to `b1`. """ - return TypeVar(self.name + " as boolean", '', base=self, bools=True) + return TypeVar( + self.name + " as boolean", '', + base=self, derived_func='AsBool') def operand_kind(self): # When a `TypeVar` object is used to describe the type of an `Operand` diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 9f77fbf860..9e13377ed6 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -61,9 +61,9 @@ br_table = Instruction( 'br_table', r""" Indirect branch via jump table. - Use ``x`` as an unsigned index into the jump table ``JT``. If a jump table - entry is found, branch to the corresponding EBB. If no entry was found fall - through to the next instruction. + Use ``x`` as an unsigned index into the jump table ``JT``. If a jump + table entry is found, branch to the corresponding EBB. If no entry was + found fall through to the next instruction. Note that this branch instruction can't pass arguments to the targeted blocks. Split critical edges as needed to work around this. diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 08cd81f1b0..1cd0ea8b4f 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -4,6 +4,7 @@ Generate sources with instruction info. import srcgen import constant_hash +from unique_table import UniqueTable, UniqueSeqTable import cretonne @@ -97,7 +98,8 @@ def gen_instruction_data_impl(fmt): fmt.doc_comment('Mutable reference to second result value, if any.') with fmt.indented( - "pub fn second_result_mut<'a>(&'a mut self) -> Option<&'a mut Value> {", '}'): + "pub fn second_result_mut<'a>(&'a mut self)" + + " -> Option<&'a mut Value> {", '}'): with fmt.indented('match *self {', '}'): for f in cretonne.InstructionFormat.all_formats: if not f.multiple_results: @@ -131,7 +133,11 @@ def collect_instr_groups(targets): def gen_opcodes(groups, fmt): - """Generate opcode enumerations.""" + """ + Generate opcode enumerations. + + Return a list of all instructions. + """ fmt.doc_comment('An instruction opcode.') fmt.doc_comment('') @@ -193,6 +199,125 @@ def gen_opcodes(groups, fmt): else: fmt.format('Opcode::{},', i.camel_name) fmt.line() + return instrs + + +def get_constraint(op, ctrl_typevar, type_sets): + """ + Get the value type constraint for an SSA value operand, where + `ctrl_typevar` is the controlling type variable. + + Each operand constraint is represented as a string, one of: + + - `Concrete(vt)`, where `vt` is a value type name. + - `Free(idx)` where `idx` is an index into `type_sets`. + - `Same`, `Lane`, `AsBool` for controlling typevar-derived constraints. + """ + t = op.typ + assert t.operand_kind() is cretonne.value + + # A concrete value type. + if isinstance(t, cretonne.ValueType): + return 'Concrete({})'.format(t.rust_name()) + + if t.free_typevar() is not ctrl_typevar: + assert not t.is_derived + return 'Free({})'.format(type_sets.add(t.type_set)) + + if t.is_derived: + assert t.base is ctrl_typevar, "Not derived directly from ctrl_typevar" + return t.derived_func + + assert t is ctrl_typevar + return 'Same' + + +def gen_type_constraints(fmt, instrs): + """ + Generate value type constraints for all instructions. + + - Emit a compact constant table of ValueTypeSet objects. + - Emit a compact constant table of OperandConstraint objects. + - Emit an opcode-indexed table of instruction constraints. + + """ + + # Table of TypeSet instances. + type_sets = UniqueTable() + + # Table of operand constraint sequences (as tuples). Each operand + # constraint is represented as a string, one of: + # - `Concrete(vt)`, where `vt` is a value type name. + # - `Free(idx)` where `idx` isan index into `type_sets`. + # - `Same`, `Lane`, `AsBool` for controlling typevar-derived constraints. + operand_seqs = UniqueSeqTable() + + # Preload table with constraints for typical binops. + operand_seqs.add(['Same'] * 3) + + # TypeSet indexes are encoded in 3 bits, with `111` reserved. + typeset_limit = 7 + + fmt.comment('Table of opcode constraints.') + with fmt.indented( + 'const OPCODE_CONSTRAINTS : [OpcodeConstraints; {}] = [' + .format(len(instrs)), '];'): + for i in instrs: + # Collect constraints for the value results, not including + # `variable_args` results which are always special cased. + constraints = list() + ctrl_typevar = None + ctrl_typeset = typeset_limit + if i.is_polymorphic: + ctrl_typevar = i.ctrl_typevar + ctrl_typeset = type_sets.add(ctrl_typevar.type_set) + for idx in i.value_results: + constraints.append( + get_constraint(i.outs[idx], ctrl_typevar, type_sets)) + for idx in i.format.value_operands: + constraints.append( + get_constraint(i.ins[idx], ctrl_typevar, type_sets)) + offset = operand_seqs.add(constraints) + fixed_results = len(i.value_results) + use_typevar_operand = i.is_polymorphic and i.use_typevar_operand + fmt.comment( + '{}: fixed_results={}, use_typevar_operand={}' + .format(i.camel_name, fixed_results, use_typevar_operand)) + fmt.comment('Constraints={}'.format(constraints)) + if i.is_polymorphic: + fmt.comment( + 'Polymorphic over {}'.format(ctrl_typevar.type_set)) + # Compute the bit field encoding, c.f. instructions.rs. + assert fixed_results < 8, "Bit field encoding too tight" + bits = (offset << 8) | (ctrl_typeset << 4) | fixed_results + if use_typevar_operand: + bits |= 8 + assert bits < 0x10000, "Constraint table too large for bit field" + fmt.line('OpcodeConstraints({:#06x}),'.format(bits)) + + fmt.comment('Table of value type sets.') + assert len(type_sets.table) <= typeset_limit, "Too many type sets" + with fmt.indented( + 'const TYPE_SETS : [ValueTypeSet; {}] = [' + .format(len(type_sets.table)), '];'): + for ts in type_sets.table: + with fmt.indented('ValueTypeSet {', '},'): + if ts.base: + fmt.line('base: {},'.format(ts.base.rust_name())) + else: + fmt.line('base: types::VOID,') + for field in ts._fields: + if field == 'base': + continue + fmt.line('{}: {},'.format( + field, str(getattr(ts, field)).lower())) + + fmt.comment('Table of operand constraint sequences.') + with fmt.indented( + 'const OPERAND_CONSTRAINTS : [OperandConstraint; {}] = [' + .format(len(operand_seqs.table)), '];'): + for c in operand_seqs.table: + fmt.line('OperandConstraint::{},'.format(c)) def generate(targets, out_dir): @@ -202,5 +327,6 @@ def generate(targets, out_dir): fmt = srcgen.Formatter() gen_formats(fmt) gen_instruction_data_impl(fmt) - gen_opcodes(groups, fmt) + instrs = gen_opcodes(groups, fmt) + gen_type_constraints(fmt, instrs) fmt.update_file('opcodes.rs', out_dir) diff --git a/meta/unique_table.py b/meta/unique_table.py new file mode 100644 index 0000000000..cbc45af200 --- /dev/null +++ b/meta/unique_table.py @@ -0,0 +1,68 @@ +""" +Generate a table of unique items. + +The `UniqueTable` class collects items into an array, removing duplicates. Each +item is mapped to its offset in the final array. + +This is a compression technique for compile-time generated tables. +""" + + +class UniqueTable: + """ + Collect items into the `table` list, removing duplicates. + """ + def __init__(self): + # List of items added in order. + self.table = list() + # Map item -> index. + self.index = dict() + + def add(self, item): + """ + Add a single item to the table if it isn't already there. + + Return the offset into `self.table` of the item. + """ + if item in self.index: + return self.index[item] + + idx = len(self.table) + self.index[item] = idx + self.table.append(item) + return idx + + +class UniqueSeqTable: + """ + Collect sequences into the `table` list, removing duplicates. + + Sequences don't have to be of the same length. + """ + def __init__(self): + self.table = list() + # Map seq -> index. + self.index = dict() + + def add(self, seq): + """ + Add a sequence of items to the table. If the table already contains the + items in `seq` in the same order, use those instead. + + Return the offset into `self.table` of the beginning of `seq`. + """ + if len(seq) == 0: + return 0 + seq = tuple(seq) + if seq in self.index: + return self.index[seq] + + idx = len(self.table) + self.table.extend(seq) + + # Add seq and all sub-sequences to `index`. + for length in range(1, len(seq) + 1): + for offset in range(len(seq) - length + 1): + self.index[seq[offset:offset+length]] = idx + offset + + return idx From cc71744b74ffe276baafbdaf25fadd8a2ce87304 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 20 May 2016 15:10:31 -0700 Subject: [PATCH 0089/3084] Implement select and vselect instructions. This gives us the opportunity to use the AsBool derived type variables and a Select instruction format with a non-default typevar_operand setting. --- cranelift/docs/langref.rst | 27 ++-------------- cranelift/src/libcretonne/instructions.rs | 9 ++++-- cranelift/src/libcretonne/write.rs | 3 ++ cranelift/src/libreader/parser.rs | 1 + meta/cretonne/__init__.py | 13 +++----- meta/cretonne/base.py | 38 +++++++++++++++++++++++ meta/cretonne/formats.py | 4 +++ 7 files changed, 60 insertions(+), 35 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index ca58a9b273..05ebfa9e05 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -656,35 +656,12 @@ load a constant into an SSA value. .. autoinst:: f32const .. autoinst:: f64const .. autoinst:: vconst - -.. inst:: a = select c, x, y - - Conditional select. - - :arg b1 c: Controlling flag. - :arg T x: Value to return when ``c`` is true. - :arg T y: Value to return when ``c`` is false. Must be same type as ``x``. - :result T a: Same type as ``x`` and ``y``. - - This instruction selects whole values. Use :inst:`vselect` for lane-wise - selection. +.. autoinst:: select Vector operations ----------------- -.. inst:: a = vselect c, x, y - - Vector lane select. - - Select lanes from ``x`` or ``y`` controlled by the lanes of the boolean - vector ``c``. - - :arg b1xN c: Controlling flag vector. - :arg TxN x: Vector with lanes selected by the true lanes of ``c``. - Must be a vector type with the same number of lanes as ``c``. - :arg TxN y: Vector with lanes selected by the false lanes of ``c``. - Must be same type as ``x``. - :result TxN a: Same type as ``x`` and ``y``. +.. autoinst:: vselect .. inst:: a = vbuild x, y, z, ... diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index c6b353ae9c..d1861fb669 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -158,6 +158,11 @@ pub enum InstructionData { second_result: Value, args: [Value; 2], }, + Select { + opcode: Opcode, + ty: Type, + args: [Value; 3], + }, Jump { opcode: Opcode, ty: Type, @@ -295,13 +300,13 @@ impl InstructionData { /// /// Bits 4-7: /// Permitted set of types for the controlling type variable as an index into `TYPE_SETS`. -/// +/// /// Bits 8-15: /// Offset into `OPERAND_CONSTRAINT` table of the descriptors for this opcode. The first /// `fixed_results()` entries describe the result constraints, then follows constraints for the /// fixed `Value` input operands. The number of `Value` inputs isdetermined by the instruction /// format. -/// +/// #[derive(Clone, Copy)] pub struct OpcodeConstraints(u16); diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index d3a676293d..e6eb3a477f 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -156,6 +156,9 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { BinaryImm { opcode, lhs, rhs, .. } => writeln!(w, "{} {}, {}", opcode, lhs, rhs), BinaryImmRev { opcode, lhs, rhs, .. } => writeln!(w, "{} {}, {}", opcode, lhs, rhs), BinaryOverflow { opcode, args, .. } => writeln!(w, "{} {}, {}", opcode, args[0], args[1]), + Select { opcode, args, .. } => { + writeln!(w, "{} {}, {}, {}", opcode, args[0], args[1], args[2]) + } Jump { opcode, ref data, .. } => writeln!(w, "{} {}", opcode, data), Branch { opcode, ref data, .. } => writeln!(w, "{} {}", opcode, data), BranchTable { opcode, arg, table, .. } => writeln!(w, "{} {}, {}", opcode, arg, table), diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 2247259530..6aab089330 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -668,6 +668,7 @@ impl<'a> Parser<'a> { args: [lhs, rhs], } } + InstructionFormat::Select | InstructionFormat::Jump | InstructionFormat::Branch | InstructionFormat::BranchTable | diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 2c737c2e7e..c0eec9e970 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -283,6 +283,7 @@ class TypeVar(object): if self.is_derived: assert derived_func self.derived_func = derived_func + self.name = '{}({})'.format(derived_func, base.name) else: self.type_set = TypeSet( allow_scalars=scalars, @@ -303,18 +304,14 @@ class TypeVar(object): When this type variable assumes a scalar type, the derived type will be the same scalar type. """ - return TypeVar( - "Lane type of " + self.name, '', - base=self, derived_func='Lane') + return TypeVar(None, None, base=self, derived_func='Lane') def as_bool(self): """ Return a derived type variable that has the same vector geometry as this type variable, but with boolean lanes. Scalar types map to `b1`. """ - return TypeVar( - self.name + " as boolean", '', - base=self, derived_func='AsBool') + return TypeVar(None, None, base=self, derived_func='AsBool') def operand_kind(self): # When a `TypeVar` object is used to describe the type of an `Operand` @@ -607,8 +604,8 @@ class Instruction(object): # No other derived typevars allowed. if typ is not tv: raise RuntimeError( - "type variable {} must not be derived from {}" - .format(typ.name, tv.name)) + "{}: type variable {} must be derived from {}" + .format(self.ins[opidx], typ.name, ctrl_typevar)) # Other free type variables can only be used once each. if tv in other_tvs: raise RuntimeError( diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 9e13377ed6..c28c3724a2 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -19,6 +19,9 @@ Testable = TypeVar( TxN = TypeVar( '%Tx%N', 'A SIMD vector type', ints=True, floats=True, bools=True, scalars=False, simd=True) +Any = TypeVar( + 'Any', 'Any integer, float, or boolean scalar or vector type', + ints=True, floats=True, bools=True, scalars=True, simd=True) # # Control flow @@ -140,6 +143,41 @@ vconst = Instruction( ins=N, outs=a) # +# Generics. +# + +c = Operand('c', Testable, doc='Controlling value to test') +x = Operand('x', Any, doc='Value to use when `c` is true') +y = Operand('y', Any, doc='Value to use when `c` is false') +a = Operand('a', Any) + +select = Instruction( + 'select', r""" + Conditional select. + + This instruction selects whole values. Use :inst:`vselect` for + lane-wise selection. + """, + ins=(c, x, y), outs=a) + +# +# Vector operations +# + +c = Operand('c', TxN.as_bool(), doc='Controlling vector') +x = Operand('x', TxN, doc='Value to use where `c` is true') +y = Operand('y', TxN, doc='Value to use where `c` is false') +a = Operand('a', TxN) + +vselect = Instruction( + 'vselect', r""" + Vector lane select. + + Select lanes from ``x`` or ``y`` controlled by the lanes of the boolean + vector ``c``. + """, + ins=(c, x, y), outs=a) +# # Integer arithmetic # diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index 261b400321..4843befea9 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -26,6 +26,10 @@ BinaryImmRev = InstructionFormat(imm64, value) # Generate result + overflow flag. BinaryOverflow = InstructionFormat(value, value, multiple_results=True) +# The select instructions are controlled by the second value operand. +# The first value operand is the controlling flag whisch has a derived type. +Select = InstructionFormat(value, value, value, typevar_operand=1) + Jump = InstructionFormat(ebb, variable_args, boxed_storage=True) Branch = InstructionFormat(value, ebb, variable_args, boxed_storage=True) BranchTable = InstructionFormat(value, jump_table) From 96cfb405077667824c3969168aed681b80675e16 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 20 May 2016 15:36:03 -0700 Subject: [PATCH 0090/3084] Add vector instructions. Use derived type variables with the 'LaneOf' function. Add u8 immediates to be used for lane indexes and bit shifts. --- cranelift/docs/langref.rst | 34 ++----------------- cranelift/src/libcretonne/instructions.rs | 16 +++++++-- cranelift/src/libcretonne/write.rs | 4 +++ cranelift/src/libreader/parser.rs | 2 ++ meta/cretonne/__init__.py | 4 +-- meta/cretonne/base.py | 40 +++++++++++++++++++++-- meta/cretonne/formats.py | 5 ++- meta/cretonne/immediates.py | 6 ++++ 8 files changed, 73 insertions(+), 38 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 05ebfa9e05..8bf890b080 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -669,37 +669,9 @@ Vector operations Build a vector value from the provided lanes. -.. inst:: a = splat x - - Vector splat. - - Return a vector whose lanes are all ``x``. - - :arg T x: Scalar value to be replicated. - :result TxN a: Vector with identical lanes. - -.. inst:: a = insertlane x, Idx, y - - Insert ``y`` as lane ``Idx`` in x. - - The lane index, ``Idx``, is an immediate value, not an SSA value. It must - indicate a valid lane index for the type of ``x``. - - :arg TxN x: Vector to modify. - :arg Idx: Lane index smaller than N. - :arg T y: New lane value. - :result TxN y: Updated vector. - -.. inst:: a = extractlane x, Idx - - Extract lane ``Idx`` from ``x``. - - The lane index, ``Idx``, is an immediate value, not an SSA value. It must - indicate a valid lane index for the type of ``x``. - - :arg TxN x: Source vector - :arg Idx: Lane index - :result T a: Lane value. +.. autoinst:: splat +.. autoinst:: insertlane +.. autoinst:: extractlane Integer operations ------------------ diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index d1861fb669..338bf3f28b 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -163,6 +163,18 @@ pub enum InstructionData { ty: Type, args: [Value; 3], }, + InsertLane { + opcode: Opcode, + ty: Type, + lane: u8, + args: [Value; 2], + }, + ExtractLane { + opcode: Opcode, + ty: Type, + lane: u8, + arg: Value, + }, Jump { opcode: Opcode, ty: Type, @@ -401,7 +413,7 @@ enum OperandConstraint { Same, /// This operand is `ctrlType.lane_type()`. - Lane, + LaneOf, /// This operand is `ctrlType.as_bool()`. AsBool, @@ -418,7 +430,7 @@ impl OperandConstraint { Concrete(t) => Some(t), Free(_) => None, Same => Some(ctrl_type), - Lane => Some(ctrl_type.lane_type()), + LaneOf => Some(ctrl_type.lane_type()), AsBool => Some(ctrl_type.as_bool()), } } diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index e6eb3a477f..406296c2a3 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -159,6 +159,10 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { Select { opcode, args, .. } => { writeln!(w, "{} {}, {}, {}", opcode, args[0], args[1], args[2]) } + InsertLane { opcode, lane, args, .. } => { + writeln!(w, "{} {}, {}, {}", opcode, args[0], lane, args[1]) + } + ExtractLane { opcode, lane, arg, .. } => writeln!(w, "{} {}, {}", opcode, arg, lane), Jump { opcode, ref data, .. } => writeln!(w, "{} {}", opcode, data), Branch { opcode, ref data, .. } => writeln!(w, "{} {}", opcode, data), BranchTable { opcode, arg, table, .. } => writeln!(w, "{} {}, {}", opcode, arg, table), diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 6aab089330..0344b6238f 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -669,6 +669,8 @@ impl<'a> Parser<'a> { } } InstructionFormat::Select | + InstructionFormat::InsertLane | + InstructionFormat::ExtractLane | InstructionFormat::Jump | InstructionFormat::Branch | InstructionFormat::BranchTable | diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index c0eec9e970..9660e9e602 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -296,7 +296,7 @@ class TypeVar(object): def __str__(self): return "`{}`".format(self.name) - def lane(self): + def lane_of(self): """ Return a derived type variable that is the scalar lane type of this type variable. @@ -304,7 +304,7 @@ class TypeVar(object): When this type variable assumes a scalar type, the derived type will be the same scalar type. """ - return TypeVar(None, None, base=self, derived_func='Lane') + return TypeVar(None, None, base=self, derived_func='LaneOf') def as_bool(self): """ diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index c28c3724a2..b41225ae00 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -6,7 +6,7 @@ support. """ from . import TypeVar, Operand, Instruction, InstructionGroup, variable_args from types import i8, f32, f64 -from immediates import imm64, ieee32, ieee64, immvector +from immediates import imm64, uimm8, ieee32, ieee64, immvector import entities instructions = InstructionGroup("base", "Shared base instruction set") @@ -17,7 +17,7 @@ Testable = TypeVar( 'Testable', 'A scalar boolean or integer type', ints=True, bools=True) TxN = TypeVar( - '%Tx%N', 'A SIMD vector type', + 'TxN', 'A SIMD vector type', ints=True, floats=True, bools=True, scalars=False, simd=True) Any = TypeVar( 'Any', 'Any integer, float, or boolean scalar or vector type', @@ -177,6 +177,42 @@ vselect = Instruction( vector ``c``. """, ins=(c, x, y), outs=a) + +x = Operand('x', TxN.lane_of()) + +splat = Instruction( + 'splat', r""" + Vector splat. + + Return a vector whose lanes are all ``x``. + """, + ins=x, outs=a) + +x = Operand('x', TxN, doc='SIMD vector to modify') +y = Operand('y', TxN.lane_of(), doc='New lane value') +Idx = Operand('Idx', uimm8, doc='Lane index') + +insertlane = Instruction( + 'insertlane', r""" + Insert ``y`` as lane ``Idx`` in x. + + The lane index, ``Idx``, is an immediate value, not an SSA value. It + must indicate a valid lane index for the type of ``x``. + """, + ins=(x, Idx, y), outs=a) + +x = Operand('x', TxN) +a = Operand('a', TxN.lane_of()) + +extractlane = Instruction( + 'extractlane', r""" + Extract lane ``Idx`` from ``x``. + + The lane index, ``Idx``, is an immediate value, not an SSA value. It + must indicate a valid lane index for the type of ``x``. + """, + ins=(x, Idx), outs=a) + # # Integer arithmetic # diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index 4843befea9..6f09c4b5af 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -8,7 +8,7 @@ in this module. from . import InstructionFormat, value, variable_args -from immediates import imm64, ieee32, ieee64, immvector +from immediates import imm64, uimm8, ieee32, ieee64, immvector from entities import ebb, function, jump_table Nullary = InstructionFormat() @@ -30,6 +30,9 @@ BinaryOverflow = InstructionFormat(value, value, multiple_results=True) # The first value operand is the controlling flag whisch has a derived type. Select = InstructionFormat(value, value, value, typevar_operand=1) +InsertLane = InstructionFormat(value, uimm8, value) +ExtractLane = InstructionFormat(value, uimm8) + Jump = InstructionFormat(ebb, variable_args, boxed_storage=True) Branch = InstructionFormat(value, ebb, variable_args, boxed_storage=True) BranchTable = InstructionFormat(value, jump_table) diff --git a/meta/cretonne/immediates.py b/meta/cretonne/immediates.py index e8e581d64d..299b340bce 100644 --- a/meta/cretonne/immediates.py +++ b/meta/cretonne/immediates.py @@ -11,6 +11,12 @@ from . import ImmediateKind #: :py:class:`cretonne.IntType` type. imm64 = ImmediateKind('imm64', 'A 64-bit immediate integer.') +#: An unsigned 8-bit immediate integer operand. +#: +#: This small operand is used to indicate lane indexes in SIMD vectors and +#: immediate bit counts on shift instructions. +uimm8 = ImmediateKind('uimm8', 'An 8-bit immediate unsigned integer.') + #: A 32-bit immediate floating point operand. #: #: IEEE 754-2008 binary32 interchange format. From 840b48397220887a47a922f609cf95ee0610487a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 27 May 2016 12:03:09 -0700 Subject: [PATCH 0091/3084] Generate Value and Ebb references in lexer. During parsing, it is possible to see instruction operands that reference values or EBBs that have not been created yet. These references have to be resolved by a second pass following parsing once all EBBs and values have been created. To prepare for this second pass, start creating Ebb and Value references that use the numbering from the source file rather than the in-memory real references. Maintain Value -> Value and Ebb -> Ebb mappings. This makes it possible to store source-numbered Ebb and Value references in instructions. All other entities are created in the preamble, so they should have been created before they are referenced. --- cranelift/src/libcretonne/entities.rs | 73 +++++++++++++++++++--- cranelift/src/libreader/lexer.rs | 22 ++++--- cranelift/src/libreader/parser.rs | 89 ++++++++++----------------- 3 files changed, 108 insertions(+), 76 deletions(-) diff --git a/cranelift/src/libcretonne/entities.rs b/cranelift/src/libcretonne/entities.rs index 8513447ab2..dcdff9e822 100644 --- a/cranelift/src/libcretonne/entities.rs +++ b/cranelift/src/libcretonne/entities.rs @@ -24,7 +24,7 @@ use std::fmt::{self, Display, Formatter, Write}; use std::u32; /// An opaque reference to an extended basic block in a function. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct Ebb(u32); impl Ebb { @@ -33,6 +33,15 @@ impl Ebb { Ebb(index as u32) } + /// Create a new EBB reference from its number. This corresponds to the ebbNN representation. + pub fn with_number(n: u32) -> Option { + if n < u32::MAX { + Some(Ebb(n)) + } else { + None + } + } + pub fn index(&self) -> usize { self.0 as usize } @@ -54,9 +63,8 @@ impl Default for Ebb { } } - /// An opaque reference to an instruction in a function. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct Inst(u32); impl Inst { @@ -88,7 +96,7 @@ impl Default for Inst { /// An opaque reference to an SSA value. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct Value(u32); // Value references can either reference an instruction directly, or they can refer to the extended @@ -105,12 +113,29 @@ pub enum ExpandedValue { } impl Value { - pub fn direct_from_number(n: u32) -> Value { - let encoding = n * 2; - assert!(encoding < u32::MAX); - Value(encoding) + /// Create a `Direct` value from its number representation. + /// This is the number in the vNN notation. + pub fn direct_with_number(n: u32) -> Option { + if n < u32::MAX / 2 { + let encoding = n * 2; + assert!(encoding < u32::MAX); + Some(Value(encoding)) + } else { + None + } } + /// Create a `Table` value from its number representation. + /// This is the number in the vxNN notation. + pub fn table_with_number(n: u32) -> Option { + if n < u32::MAX / 2 { + let encoding = n * 2 + 1; + assert!(encoding < u32::MAX); + Some(Value(encoding)) + } else { + None + } + } pub fn new_direct(i: Inst) -> Value { let encoding = i.index() * 2; assert!(encoding < u32::MAX as usize); @@ -160,7 +185,7 @@ impl Default for Value { } /// An opaque reference to a stack slot. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct StackSlot(u32); impl StackSlot { @@ -191,7 +216,7 @@ impl Default for StackSlot { } /// An opaque reference to a jump table. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct JumpTable(u32); impl JumpTable { @@ -220,3 +245,31 @@ impl Default for JumpTable { NO_JUMP_TABLE } } + +#[cfg(test)] +mod tests { + use super::*; + use std::u32; + + #[test] + fn value_with_number() { + assert_eq!(Value::direct_with_number(0).unwrap().to_string(), "v0"); + assert_eq!(Value::direct_with_number(1).unwrap().to_string(), "v1"); + assert_eq!(Value::table_with_number(0).unwrap().to_string(), "vx0"); + assert_eq!(Value::table_with_number(1).unwrap().to_string(), "vx1"); + + assert_eq!(Value::direct_with_number(u32::MAX / 2), None); + assert_eq!(match Value::direct_with_number(u32::MAX / 2 - 1).unwrap().expand() { + ExpandedValue::Direct(i) => i.index() as u32, + _ => u32::MAX, + }, + u32::MAX / 2 - 1); + + assert_eq!(Value::table_with_number(u32::MAX / 2), None); + assert_eq!(match Value::table_with_number(u32::MAX / 2 - 1).unwrap().expand() { + ExpandedValue::Table(i) => i as u32, + _ => u32::MAX, + }, + u32::MAX / 2 - 1); + } +} diff --git a/cranelift/src/libreader/lexer.rs b/cranelift/src/libreader/lexer.rs index 3611a70f69..4d3dfdf7c1 100644 --- a/cranelift/src/libreader/lexer.rs +++ b/cranelift/src/libreader/lexer.rs @@ -7,6 +7,7 @@ use std::str::CharIndices; use cretonne::types; +use cretonne::entities::{Value, Ebb}; /// The location of a `Token` or `Error`. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -35,9 +36,8 @@ pub enum Token<'a> { Float(&'a str), // Floating point immediate Integer(&'a str), // Integer immediate Type(types::Type), // i32, f32, b32x4, ... - ValueDirect(u32), // v12 - ValueTable(u32), // vx7 - Ebb(u32), // ebb3 + Value(Value), // v12, vx7 + Ebb(Ebb), // ebb3 StackSlot(u32), // ss3 Identifier(&'a str), // Unrecognized identifier (opcode, enumerator, ...) } @@ -289,9 +289,9 @@ impl<'a> Lexer<'a> { }; match prefix { - "v" => Some(Token::ValueDirect(value)), - "vx" => Some(Token::ValueTable(value)), - "ebb" => Some(Token::Ebb(value)), + "v" => Value::direct_with_number(value).map(|v| Token::Value(v)), + "vx" => Value::table_with_number(value).map(|v| Token::Value(v)), + "ebb" => Ebb::with_number(value).map(|ebb| Token::Ebb(ebb)), "ss" => Some(Token::StackSlot(value)), _ => None, } @@ -374,6 +374,7 @@ impl<'a> Lexer<'a> { mod tests { use super::*; use cretonne::types; + use cretonne::entities::{Value, Ebb}; fn token<'a>(token: Token<'a>, line: usize) -> Option, LocatedError>> { Some(super::token(token, Location { line_number: line })) @@ -445,14 +446,17 @@ mod tests { fn lex_identifiers() { let mut lex = Lexer::new("v0 v00 vx01 ebb1234567890 ebb5234567890 entry v1x vx1 vxvx4 \ function0 function b1 i32x4 f32x5"); - assert_eq!(lex.next(), token(Token::ValueDirect(0), 1)); + assert_eq!(lex.next(), + token(Token::Value(Value::direct_with_number(0).unwrap()), 1)); assert_eq!(lex.next(), token(Token::Identifier("v00"), 1)); assert_eq!(lex.next(), token(Token::Identifier("vx01"), 1)); - assert_eq!(lex.next(), token(Token::Ebb(1234567890), 1)); + assert_eq!(lex.next(), + token(Token::Ebb(Ebb::with_number(1234567890).unwrap()), 1)); assert_eq!(lex.next(), token(Token::Identifier("ebb5234567890"), 1)); assert_eq!(lex.next(), token(Token::Entry, 1)); assert_eq!(lex.next(), token(Token::Identifier("v1x"), 1)); - assert_eq!(lex.next(), token(Token::ValueTable(1), 1)); + assert_eq!(lex.next(), + token(Token::Value(Value::table_with_number(1).unwrap()), 1)); assert_eq!(lex.next(), token(Token::Identifier("vxvx4"), 1)); assert_eq!(lex.next(), token(Token::Identifier("function0"), 1)); assert_eq!(lex.next(), token(Token::Function, 1)); diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 0344b6238f..ff52940941 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -52,9 +52,8 @@ pub struct Parser<'a> { struct Context { function: Function, stack_slots: HashMap, // ssNN - ebbs: HashMap, // ebbNN - value_directs: HashMap, // vNN - value_tables: HashMap, // vxNN + ebbs: HashMap, // ebbNN + values: HashMap, // vNN, vxNN } impl Context { @@ -63,8 +62,7 @@ impl Context { function: f, stack_slots: HashMap::new(), ebbs: HashMap::new(), - value_directs: HashMap::new(), - value_tables: HashMap::new(), + values: HashMap::new(), } } @@ -80,37 +78,25 @@ impl Context { } } - // Allocate a new EBB and add a mapping number -> Ebb. - fn add_ebb(&mut self, number: u32, loc: &Location) -> Result { + // Allocate a new EBB and add a mapping src_ebb -> Ebb. + fn add_ebb(&mut self, src_ebb: Ebb, loc: &Location) -> Result { let ebb = self.function.make_ebb(); - if self.ebbs.insert(number, ebb).is_some() { + if self.ebbs.insert(src_ebb, ebb).is_some() { Err(Error { location: loc.clone(), - message: format!("duplicate EBB: ebb{}", number), + message: format!("duplicate EBB: {}", src_ebb), }) } else { Ok(ebb) } } - // Add a value mapping number -> data for a direct value (vNN). - fn add_v(&mut self, number: u32, data: Value, loc: &Location) -> Result<()> { - if self.value_directs.insert(number, data).is_some() { + // Add a value mapping src_val -> data. + fn add_value(&mut self, src_val: Value, data: Value, loc: &Location) -> Result<()> { + if self.values.insert(src_val, data).is_some() { Err(Error { location: loc.clone(), - message: format!("duplicate value: v{}", number), - }) - } else { - Ok(()) - } - } - - // Add a value mapping number -> data for a table value (vxNN). - fn add_vx(&mut self, number: u32, data: Value, loc: &Location) -> Result<()> { - if self.value_tables.insert(number, data).is_some() { - Err(Error { - location: loc.clone(), - message: format!("duplicate value: vx{}", number), + message: format!("duplicate value: {}", src_val), }) } else { Ok(()) @@ -220,7 +206,7 @@ impl<'a> Parser<'a> { } // Match and consume an ebb reference. - fn match_ebb(&mut self, err_msg: &str) -> Result { + fn match_ebb(&mut self, err_msg: &str) -> Result { if let Some(Token::Ebb(ebb)) = self.token() { self.consume(); Ok(ebb) @@ -229,26 +215,15 @@ impl<'a> Parser<'a> { } } - // Match and consume a vx reference. - fn match_vx(&mut self, err_msg: &str) -> Result { - if let Some(Token::ValueTable(vx)) = self.token() { - self.consume(); - Ok(vx) - } else { - Err(self.error(err_msg)) - } - } - // Match and consume a value reference, direct or vtable. // This does not convert from the source value numbering to our in-memory value numbering. fn match_value(&mut self, err_msg: &str) -> Result { - let val = match self.token() { - Some(Token::ValueDirect(v)) => Value::direct_from_number(v), - Some(Token::ValueTable(vx)) => Value::new_table(vx as usize), - _ => return Err(self.error(err_msg)), - }; - self.consume(); - Ok(val) + if let Some(Token::Value(v)) = self.token() { + self.consume(); + Ok(v) + } else { + Err(self.error(err_msg)) + } } // Match and consume an Imm64 immediate. @@ -484,7 +459,7 @@ impl<'a> Parser<'a> { // extended-basic-block ::= ebb-header * { instruction } while match self.token() { - Some(Token::ValueDirect(_)) => true, + Some(Token::Value(_)) => true, Some(Token::Identifier(_)) => true, _ => false, } { @@ -519,39 +494,39 @@ impl<'a> Parser<'a> { // Parse a single EBB argument declaration, and append it to `ebb`. // - // ebb-arg ::= * ValueTable(vx) ":" Type(t) + // ebb-arg ::= * Value(vx) ":" Type(t) // fn parse_ebb_arg(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { - // ebb-arg ::= * ValueTable(vx) ":" Type(t) - let vx = try!(self.match_vx("EBB argument must be a vx value")); + // ebb-arg ::= * Value(vx) ":" Type(t) + let vx = try!(self.match_value("EBB argument must be a value")); let vx_location = self.location; - // ebb-arg ::= ValueTable(vx) * ":" Type(t) + // ebb-arg ::= Value(vx) * ":" Type(t) try!(self.match_token(Token::Colon, "expected ':' after EBB argument")); - // ebb-arg ::= ValueTable(vx) ":" * Type(t) + // ebb-arg ::= Value(vx) ":" * Type(t) let t = try!(self.match_type("expected EBB argument type")); // Allocate the EBB argument and add the mapping. let value = ctx.function.append_ebb_arg(ebb, t); - ctx.add_vx(vx, value, &vx_location) + ctx.add_value(vx, value, &vx_location) } // Parse an instruction, append it to `ebb`. // // instruction ::= [inst-results "="] Opcode(opc) ... - // inst-results ::= ValueDirect(v) { "," ValueTable(vx) } + // inst-results ::= Value(v) { "," Value(vx) } // fn parse_instruction(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { - // Result value numbers. First is a ValueDirect, remaining are ValueTable. + // Result value numbers. let mut results = Vec::new(); // instruction ::= * [inst-results "="] Opcode(opc) ... - if let Some(Token::ValueDirect(v)) = self.token() { + if let Some(Token::Value(v)) = self.token() { self.consume(); results.push(v); - // inst-results ::= ValueDirect(v) * { "," ValueTable(vx) } + // inst-results ::= Value(v) * { "," Value(vx) } while self.optional(Token::Comma) { - // inst-results ::= ValueDirect(v) { "," * ValueTable(vx) } - results.push(try!(self.match_vx("expected vx result value"))); + // inst-results ::= Value(v) { "," * Value(vx) } + results.push(try!(self.match_value("expected result value"))); } try!(self.match_token(Token::Equal, "expected '=' before opcode")); @@ -576,7 +551,7 @@ impl<'a> Parser<'a> { if !results.is_empty() { assert!(results.len() == 1, "Multiple results not implemented"); let result = ctx.function.first_result(inst); - try!(ctx.add_v(results[0], result, &self.location)); + try!(ctx.add_value(results[0], result, &self.location)); } ctx.function.append_inst(ebb, inst); From 9af18728fa1ff2469c9b5f6a701c404092e44915 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 1 Jun 2016 09:14:01 -0700 Subject: [PATCH 0092/3084] Clean up unused-import warnings. --- cranelift/src/libcretonne/repr.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index 6e7e6ea80f..716887211d 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -1,7 +1,7 @@ //! Representation of Cretonne IL functions. use types::{Type, FunctionName, Signature}; -use entities::*; +use entities::{Ebb, NO_EBB, Inst, NO_INST, Value, NO_VALUE, ExpandedValue, StackSlot}; use instructions::*; use std::fmt::{self, Display, Formatter}; use std::ops::{Index, IndexMut}; @@ -502,7 +502,6 @@ impl<'a> Iterator for Values<'a> { mod tests { use super::*; use types; - use entities::*; use instructions::*; #[test] From 8ebf6e775d0aec0933535ab25530223543c82b47 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 27 May 2016 11:04:55 -0700 Subject: [PATCH 0093/3084] Parse controlling type variable. Do basic type inference. Replace the make_multi_inst() function with a make_inst_results() which uses the constraint system to create the result values. A typevar argument ensures that this function does not infer anything from the instruction data arguments. These arguments may not be valid during parsing. Implement basic type inference in the parser. If the designated value operand on a polymorphic instruction refers to a known value, use that to infer the controlling type variable. This simple method of type inference requires the operand value to be defined above the use in the text. Since reordering the EBBs could place a dominating EBB below the current one, this is a bit fragile. One possibility would be to require the value is defined in the same EBB. In all other cases, the controlling typevar should be explicit. --- cranelift/src/libcretonne/instructions.rs | 13 ++- cranelift/src/libcretonne/repr.rs | 90 +++++++-------- cranelift/src/libcretonne/types.rs | 7 ++ cranelift/src/libcretonne/write.rs | 4 +- cranelift/src/libreader/parser.rs | 129 ++++++++++++++++++---- meta/gen_instr.py | 40 +++++++ 6 files changed, 215 insertions(+), 68 deletions(-) diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index 338bf3f28b..9ac3ac7220 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -142,15 +142,15 @@ pub enum InstructionData { BinaryImm { opcode: Opcode, ty: Type, - lhs: Value, - rhs: Imm64, + arg: Value, + imm: Imm64, }, // Same as BinaryImm, but the immediate is the lhs operand. BinaryImmRev { opcode: Opcode, ty: Type, - rhs: Value, - lhs: Imm64, + arg: Value, + imm: Imm64, }, BinaryOverflow { opcode: Opcode, @@ -366,6 +366,11 @@ impl OpcodeConstraints { pub fn ctrl_typeset(self) -> Option { self.typeset_offset().map(|offset| TYPE_SETS[offset]) } + + /// Is this instruction polymorphic? + pub fn is_polymorphic(self) -> bool { + self.ctrl_typeset().is_some() + } } /// A value type set describes the permitted set of types for a type variable. diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index 716887211d..9c44f524ca 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -88,8 +88,8 @@ impl Function { /// Create a new instruction. /// - /// The instruction is allowed to produce at most one result as indicated by `data.ty`. Use - /// `make_multi_inst()` to create instructions with multiple results. + /// The type of the first result is indicated by `data.ty`. If the instruction produces + /// multiple results, also call `make_inst_results` to allocate value table entries. pub fn make_inst(&mut self, data: InstructionData) -> Inst { let inst = Inst::new(self.instructions.len()); self.instructions.push(data); @@ -101,40 +101,59 @@ impl Function { inst } - /// Make an instruction that may produce multiple results. - /// - /// The type of the first result is `data.ty`. If the instruction generates more than one - /// result, additional result types are in `extra_result_types`. - /// - /// Not all instruction formats can represent multiple result values. This function will panic - /// if the format of `data` is insufficient. - pub fn make_multi_inst(&mut self, data: InstructionData, extra_result_types: &[Type]) -> Inst { - let inst = self.make_inst(data); + fn inst_mut(&mut self, inst: Inst) -> &mut InstructionData { + &mut self.instructions[inst.index()] + } - if !extra_result_types.is_empty() { - // Additional values form a linked list starting from the second result value. Generate - // the list backwards so we don't have to modify value table entries in place. (This - // causes additional result values to be numbered backwards which is not the aestetic - // choice, but since it is only visible in extremely rare instructions with 3+ results, - // we don't care). - let mut head = NO_VALUE; - for ty in extra_result_types.into_iter().rev() { + /// Create result values for an instruction that produces multiple results. + /// + /// Instructions that produce 0 or 1 result values only need to be created with `make_inst`. If + /// the instruction may produce more than 1 result, call `make_inst_results` to allocate + /// `Value` table entries for the additional results. + /// + /// The result value types are determined from the instruction's value type constraints and the + /// provided `ctrl_typevar` type for polymorphic instructions. For non-polymorphic + /// instructions, `ctrl_typevar` is ignored, and `VOID` can be used. + /// + /// The type of the first result value is also set, even if it was already set in the + /// `InstructionData` passed to `make_inst`. If this function is called with a single-result + /// instruction, that is the only effect. + /// + /// Returns the number of results produced by the instruction. + pub fn make_inst_results(&mut self, inst: Inst, ctrl_typevar: Type) -> usize { + let constraints = self[inst].opcode().constraints(); + let fixed_results = constraints.fixed_results(); + + // Additional values form a linked list starting from the second result value. Generate + // the list backwards so we don't have to modify value table entries in place. (This + // causes additional result values to be numbered backwards which is not the aestetic + // choice, but since it is only visible in extremely rare instructions with 3+ results, + // we don't care). + let mut head = NO_VALUE; + let mut first_type = Type::default(); + + // TBD: Function call return values for direct and indirect function calls. + + if fixed_results > 0 { + for res_idx in (1..fixed_results).rev() { head = self.make_value(ValueData::Def { - ty: *ty, + ty: constraints.result_type(res_idx, ctrl_typevar), def: inst, next: head, }); } - - // Update the second_result pointer in `inst`. - if let Some(second_result_ref) = self.instructions[inst.index()].second_result_mut() { - *second_result_ref = head; - } else { - panic!("Instruction format doesn't allow multiple results."); - } + first_type = constraints.result_type(0, ctrl_typevar); } - inst + // Update the second_result pointer in `inst`. + if head != NO_VALUE { + *self.inst_mut(inst) + .second_result_mut() + .expect("instruction format doesn't allow multiple results") = head; + } + *self.inst_mut(inst).first_type_mut() = first_type; + + fixed_results } /// Get the first result of an instruction. @@ -521,21 +540,6 @@ mod tests { assert_eq!(ins.first_type(), types::I32); } - #[test] - fn multiple_results() { - use types::*; - let mut func = Function::new(); - - let idata = InstructionData::call(Opcode::Vconst, I64); - let inst = func.make_multi_inst(idata, &[I8, F64]); - assert_eq!(inst.to_string(), "inst0"); - let results: Vec = func.inst_results(inst).collect(); - assert_eq!(results.len(), 3); - assert_eq!(func.value_type(results[0]), I64); - assert_eq!(func.value_type(results[1]), I8); - assert_eq!(func.value_type(results[2]), F64); - } - #[test] fn stack_slot() { let mut func = Function::new(); diff --git a/cranelift/src/libcretonne/types.rs b/cranelift/src/libcretonne/types.rs index a4ff443541..bc4cecca6b 100644 --- a/cranelift/src/libcretonne/types.rs +++ b/cranelift/src/libcretonne/types.rs @@ -1,6 +1,7 @@ //! Common types for the Cretonne code generator. +use std::default::Default; use std::fmt::{self, Display, Formatter, Write}; // ====--------------------------------------------------------------------------------------====// @@ -202,6 +203,12 @@ impl Display for Type { } } +impl Default for Type { + fn default() -> Type { + VOID + } +} + // ====--------------------------------------------------------------------------------------====// // // Function signatures diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index 406296c2a3..0d615c545b 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -153,8 +153,8 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { UnaryIeee64 { opcode, imm, .. } => writeln!(w, "{} {}", opcode, imm), UnaryImmVector { opcode, .. } => writeln!(w, "{} [...]", opcode), Binary { opcode, args, .. } => writeln!(w, "{} {}, {}", opcode, args[0], args[1]), - BinaryImm { opcode, lhs, rhs, .. } => writeln!(w, "{} {}, {}", opcode, lhs, rhs), - BinaryImmRev { opcode, lhs, rhs, .. } => writeln!(w, "{} {}, {}", opcode, lhs, rhs), + BinaryImm { opcode, arg, imm, .. } => writeln!(w, "{} {}, {}", opcode, arg, imm), + BinaryImmRev { opcode, imm, arg, .. } => writeln!(w, "{} {}, {}", opcode, imm, arg), BinaryOverflow { opcode, args, .. } => writeln!(w, "{} {}, {}", opcode, args[0], args[1]), Select { opcode, args, .. } => { writeln!(w, "{} {}, {}, {}", opcode, args[0], args[1], args[2]) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index ff52940941..f9a66a5235 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -144,18 +144,22 @@ impl<'a> Parser<'a> { } // Generate an error. - fn error(&self, message: &str) -> Error { + fn error_string(&self, message: String) -> Error { Error { location: self.location, message: // If we have a lexer error latched, report that. match self.lex_error { Some(lexer::Error::InvalidChar) => "invalid character".to_string(), - None => message.to_string(), + None => message, } } } + fn error(&self, message: &str) -> Error { + self.error_string(message.to_string()) + } + // Match and consume a token without payload. fn match_token(&mut self, want: Token<'a>, err_msg: &str) -> Result> { if self.token() == Some(want) { @@ -511,14 +515,15 @@ impl<'a> Parser<'a> { // Parse an instruction, append it to `ebb`. // - // instruction ::= [inst-results "="] Opcode(opc) ... + // instruction ::= [inst-results "="] Opcode(opc) ["." Type] ... // inst-results ::= Value(v) { "," Value(vx) } // fn parse_instruction(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { // Result value numbers. let mut results = Vec::new(); - // instruction ::= * [inst-results "="] Opcode(opc) ... + // instruction ::= * [inst-results "="] Opcode(opc) ["." Type] ... + // inst-results ::= * Value(v) { "," Value(vx) } if let Some(Token::Value(v)) = self.token() { self.consume(); results.push(v); @@ -532,7 +537,7 @@ impl<'a> Parser<'a> { try!(self.match_token(Token::Equal, "expected '=' before opcode")); } - // instruction ::= [inst-results "="] * Opcode(opc) ... + // instruction ::= [inst-results "="] * Opcode(opc) ["." Type] ... let opcode = if let Some(Token::Identifier(text)) = self.token() { match text.parse() { Ok(opc) => opc, @@ -541,24 +546,110 @@ impl<'a> Parser<'a> { } else { return Err(self.error("expected instruction opcode")); }; + self.consume(); - // instruction ::= [inst-results "="] Opcode(opc) * ... + // Look for a controlling type variable annotation. + // instruction ::= [inst-results "="] Opcode(opc) * ["." Type] ... + let explicit_ctrl_type = if self.optional(Token::Dot) { + Some(try!(self.match_type("expected type after 'opcode.'"))) + } else { + None + }; + + // instruction ::= [inst-results "="] Opcode(opc) ["." Type] * ... let inst_data = try!(self.parse_inst_operands(opcode)); + + // We're done parsing the instruction now. + // + // We still need to check that the number of result values in the source matches the opcode + // or function call signature. We also need to create values with the right type for all + // the instruction results. + let ctrl_typevar = try!(self.infer_typevar(ctx, opcode, explicit_ctrl_type, &inst_data)); let inst = ctx.function.make_inst(inst_data); - - // TODO: Check that results.len() matches the opcode. - // TODO: Multiple results. - if !results.is_empty() { - assert!(results.len() == 1, "Multiple results not implemented"); - let result = ctx.function.first_result(inst); - try!(ctx.add_value(results[0], result, &self.location)); - } - + let num_results = ctx.function.make_inst_results(inst, ctrl_typevar); ctx.function.append_inst(ebb, inst); + if results.len() != num_results { + let m = format!("instruction produces {} result values, {} given", + num_results, + results.len()); + return Err(self.error_string(m)); + } + + // Now map the source result values to the just created instruction results. + // We need to copy the list of result values to avoid fighting the borrow checker. + let new_results: Vec = ctx.function.inst_results(inst).collect(); + for (src, val) in results.iter().zip(new_results) { + try!(ctx.add_value(*src, val, &self.location)); + } + Ok(()) } + // Type inference for polymorphic instructions. + // + // The controlling type variable can be specified explicitly as 'splat.i32x4 v5', or it can be + // inferred from `inst_data.typevar_operand` for some opcodes. + // + // The value operands in `inst_data` are expected to use source numbering. + // + // Returns the controlling typevar for a polymorphic opcode, or `VOID` for a non-polymorphic + // opcode. + fn infer_typevar(&self, + ctx: &Context, + opcode: Opcode, + explicit_ctrl_type: Option, + inst_data: &InstructionData) + -> Result { + let constraints = opcode.constraints(); + let ctrl_type = match explicit_ctrl_type { + Some(t) => t, + None => { + if constraints.use_typevar_operand() { + // This is an opcode that supports type inference, AND there was no explicit + // type specified. Look up `ctrl_value` to see if it was defined already. + // TBD: If it is defined in another block, the type should have been specified + // explicitly. It is unfortunate that the correctness of IL depends on the + // layout of the blocks. + let ctrl_src_value = inst_data.typevar_operand() + .expect("Constraints <-> Format inconsistency"); + ctx.function.value_type(match ctx.values.get(&ctrl_src_value) { + Some(&v) => v, + None => { + let m = format!("cannot determine type of operand {}", ctrl_src_value); + return Err(self.error_string(m)); + } + }) + } else if constraints.is_polymorphic() { + // This opcode does not support type inference, so the explicit type variable + // is required. + return Err(self.error("type variable required for polymorphic opcode")); + } else { + // This is a non-polymorphic opcode. No typevar needed. + VOID + } + } + }; + + // Verify that `ctrl_type` is valid for the controlling type variable. We don't want to + // attempt deriving types from an incorrect basis. + // This is not a complete type check. The verifier does that. + if let Some(typeset) = constraints.ctrl_typeset() { + // This is a polymorphic opcode. + if !typeset.contains(ctrl_type) { + let m = format!("{} is not a valid typevar for {}", ctrl_type, opcode); + return Err(self.error_string(m)); + } + } else { + // Treat it as a syntax error to speficy a typevar on a non-polymorphic opcode. + if ctrl_type != VOID { + return Err(self.error_string(format!("{} does not take a typevar", opcode))); + } + } + + Ok(ctrl_type) + } + // Parse the operands following the instruction opcode. // This depends on the format of the opcode. fn parse_inst_operands(&mut self, opcode: Opcode) -> Result { @@ -617,8 +708,8 @@ impl<'a> Parser<'a> { InstructionData::BinaryImm { opcode: opcode, ty: VOID, - lhs: lhs, - rhs: rhs, + arg: lhs, + imm: rhs, } } InstructionFormat::BinaryImmRev => { @@ -628,8 +719,8 @@ impl<'a> Parser<'a> { InstructionData::BinaryImmRev { opcode: opcode, ty: VOID, - lhs: lhs, - rhs: rhs, + imm: lhs, + arg: rhs, } } InstructionFormat::BinaryOverflow => { diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 1cd0ea8b4f..590dd13659 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -70,6 +70,14 @@ def gen_instruction_data_impl(fmt): 'InstructionData::{} {{ ty, .. }} => ty,' .format(f.name)) + fmt.doc_comment('Mutable reference to the type of the first result.') + with fmt.indented('pub fn first_type_mut(&mut self) -> &mut Type {', '}'): + with fmt.indented('match *self {', '}'): + for f in cretonne.InstructionFormat.all_formats: + fmt.line( + 'InstructionData::{} {{ ref mut ty, .. }} => ty,' + .format(f.name)) + # Generate shared and mutable accessors for `second_result` which only # applies to instruction formats that can produce multiple results. # Everything else returns `None`. @@ -120,6 +128,38 @@ def gen_instruction_data_impl(fmt): ' { ref mut second_result, .. }' + ' => Some(second_result),') + fmt.doc_comment('Get the controlling type variable operand.') + with fmt.indented( + 'pub fn typevar_operand(&self) -> Option {', '}'): + with fmt.indented('match *self {', '}'): + for f in cretonne.InstructionFormat.all_formats: + n = 'InstructionData::' + f.name + if f.typevar_operand is None: + fmt.line(n + ' { .. } => None,') + elif len(f.value_operands) == 1: + # We have a single value operand called 'arg'. + if f.boxed_storage: + fmt.line( + n + ' { ref data, .. } => Some(data.arg),') + else: + fmt.line(n + ' { arg, .. } => Some(arg),') + else: + # We have multiple value operands and an array `args`. + # Which `args` index to use? + # Map from index into f.kinds into f.value_operands + # index. + i = f.value_operands.index(f.typevar_operand) + if f.boxed_storage: + fmt.line( + n + + ' {{ ref data, .. }} => Some(data.args[{}]),' + .format(i)) + else: + fmt.line( + n + + ' {{ ref args, .. }} => Some(args[{}]),' + .format(i)) + def collect_instr_groups(targets): seen = set() From 441001c1ad6f5455d25e1ce9a55046c9945773aa Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 1 Jun 2016 20:45:58 -0700 Subject: [PATCH 0094/3084] Avoid allocating a temporary Vec in the parser. Wrangle the borrow checker into allowing us to iterate over function result values while mutating the ctx.values table. --- cranelift/src/libreader/parser.rs | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index f9a66a5235..600632a51a 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -577,13 +577,11 @@ impl<'a> Parser<'a> { } // Now map the source result values to the just created instruction results. - // We need to copy the list of result values to avoid fighting the borrow checker. - let new_results: Vec = ctx.function.inst_results(inst).collect(); - for (src, val) in results.iter().zip(new_results) { - try!(ctx.add_value(*src, val, &self.location)); - } - - Ok(()) + // Pass a reference to `ctx.values` instead of `ctx` itself since the `Values` iterator + // holds a reference to `ctx.function`. + self.add_values(&mut ctx.values, + results.into_iter(), + ctx.function.inst_results(inst)) } // Type inference for polymorphic instructions. @@ -650,6 +648,24 @@ impl<'a> Parser<'a> { Ok(ctrl_type) } + // Add mappings for a list of source values to their corresponding new values. + fn add_values(&self, + values: &mut HashMap, + results: S, + new_results: V) + -> Result<()> + where S: Iterator, + V: Iterator + { + for (src, val) in results.zip(new_results) { + if values.insert(src, val).is_some() { + return Err(self.error_string(format!("duplicate result value: {}", src))); + } + } + Ok(()) + } + + // Parse the operands following the instruction opcode. // This depends on the format of the opcode. fn parse_inst_operands(&mut self, opcode: Opcode) -> Result { From 49aa38b15f233ff727e729f5dd714adabb915fee Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 2 Jun 2016 08:40:47 -0700 Subject: [PATCH 0095/3084] Use an err! macro to build parser errors with format! arguments. --- cranelift/src/libreader/parser.rs | 123 +++++++++++++++--------------- 1 file changed, 62 insertions(+), 61 deletions(-) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 600632a51a..7d25fc096b 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -33,6 +33,23 @@ impl Display for Error { pub type Result = result::Result; +// Create an `Err` variant of `Result` from a location and `format!` args. +macro_rules! err { + ( $loc:expr, $msg:expr ) => { + Err(Error { + location: $loc.clone(), + message: String::from($msg), + }) + }; + + ( $loc:expr, $fmt:expr, $( $arg:expr ),+ ) => { + Err(Error { + location: $loc.clone(), + message: format!( $fmt, $( $arg ),+ ), + }) + }; +} + pub struct Parser<'a> { lex: Lexer<'a>, @@ -42,7 +59,7 @@ pub struct Parser<'a> { lookahead: Option>, // Location of lookahead. - location: Location, + loc: Location, } // Context for resolving references when parsing a single function. @@ -69,10 +86,7 @@ impl Context { // Allocate a new stack slot and add a mapping number -> StackSlot. fn add_ss(&mut self, number: u32, data: StackSlotData, loc: &Location) -> Result<()> { if self.stack_slots.insert(number, self.function.make_stack_slot(data)).is_some() { - Err(Error { - location: loc.clone(), - message: format!("duplicate stack slot: ss{}", number), - }) + err!(loc, "duplicate stack slot: ss{}", number) } else { Ok(()) } @@ -82,10 +96,7 @@ impl Context { fn add_ebb(&mut self, src_ebb: Ebb, loc: &Location) -> Result { let ebb = self.function.make_ebb(); if self.ebbs.insert(src_ebb, ebb).is_some() { - Err(Error { - location: loc.clone(), - message: format!("duplicate EBB: {}", src_ebb), - }) + err!(loc, "duplicate EBB: {}", src_ebb) } else { Ok(ebb) } @@ -94,10 +105,7 @@ impl Context { // Add a value mapping src_val -> data. fn add_value(&mut self, src_val: Value, data: Value, loc: &Location) -> Result<()> { if self.values.insert(src_val, data).is_some() { - Err(Error { - location: loc.clone(), - message: format!("duplicate value: {}", src_val), - }) + err!(loc, "duplicate value: {}", src_val) } else { Ok(()) } @@ -111,7 +119,7 @@ impl<'a> Parser<'a> { lex: Lexer::new(text), lex_error: None, lookahead: None, - location: Location { line_number: 0 }, + loc: Location { line_number: 0 }, } } @@ -131,11 +139,11 @@ impl<'a> Parser<'a> { match self.lex.next() { Some(Ok(lexer::LocatedToken { token, location })) => { self.lookahead = Some(token); - self.location = location; + self.loc = location; } Some(Err(lexer::LocatedError { error, location })) => { self.lex_error = Some(error); - self.location = location; + self.loc = location; } None => {} } @@ -143,29 +151,12 @@ impl<'a> Parser<'a> { return self.lookahead; } - // Generate an error. - fn error_string(&self, message: String) -> Error { - Error { - location: self.location, - message: - // If we have a lexer error latched, report that. - match self.lex_error { - Some(lexer::Error::InvalidChar) => "invalid character".to_string(), - None => message, - } - } - } - - fn error(&self, message: &str) -> Error { - self.error_string(message.to_string()) - } - // Match and consume a token without payload. fn match_token(&mut self, want: Token<'a>, err_msg: &str) -> Result> { if self.token() == Some(want) { Ok(self.consume()) } else { - Err(self.error(err_msg)) + err!(self.loc, err_msg) } } @@ -185,7 +176,7 @@ impl<'a> Parser<'a> { if self.token() == Some(Token::Identifier(want)) { Ok(self.consume()) } else { - Err(self.error(err_msg)) + err!(self.loc, err_msg) } } @@ -195,7 +186,7 @@ impl<'a> Parser<'a> { self.consume(); Ok(t) } else { - Err(self.error(err_msg)) + err!(self.loc, err_msg) } } @@ -205,7 +196,7 @@ impl<'a> Parser<'a> { self.consume(); Ok(ss) } else { - Err(self.error(err_msg)) + err!(self.loc, err_msg) } } @@ -215,7 +206,7 @@ impl<'a> Parser<'a> { self.consume(); Ok(ebb) } else { - Err(self.error(err_msg)) + err!(self.loc, err_msg) } } @@ -226,7 +217,14 @@ impl<'a> Parser<'a> { self.consume(); Ok(v) } else { - Err(self.error(err_msg)) + err!(self.loc, err_msg) + } + } + + fn error(&self, message: &str) -> Error { + Error { + location: self.loc.clone(), + message: message.to_string(), } } @@ -238,7 +236,7 @@ impl<'a> Parser<'a> { // Parse it as an Imm64 to check for overflow and other issues. text.parse().map_err(|e| self.error(e)) } else { - Err(self.error(err_msg)) + err!(self.loc, err_msg) } } @@ -250,7 +248,7 @@ impl<'a> Parser<'a> { // Parse it as an Ieee32 to check for the right number of digits and other issues. text.parse().map_err(|e| self.error(e)) } else { - Err(self.error(err_msg)) + err!(self.loc, err_msg) } } @@ -262,7 +260,7 @@ impl<'a> Parser<'a> { // Parse it as an Ieee64 to check for the right number of digits and other issues. text.parse().map_err(|e| self.error(e)) } else { - Err(self.error(err_msg)) + err!(self.loc, err_msg) } } @@ -323,7 +321,7 @@ impl<'a> Parser<'a> { self.consume(); Ok(s.to_string()) } - _ => Err(self.error("expected function name")), + _ => err!(self.loc, "expected function name"), } } @@ -400,7 +398,7 @@ impl<'a> Parser<'a> { try!(match self.token() { Some(Token::StackSlot(..)) => { self.parse_stack_slot_decl() - .and_then(|(num, dat)| ctx.add_ss(num, dat, &self.location)) + .and_then(|(num, dat)| ctx.add_ss(num, dat, &self.loc)) } // More to come.. _ => return Ok(()), @@ -419,7 +417,7 @@ impl<'a> Parser<'a> { // stack-slot-decl ::= StackSlot(ss) "=" "stack_slot" * Bytes {"," stack-slot-flag} let bytes = try!(self.match_imm64("expected byte-size in stack_slot decl")).to_bits(); if bytes > u32::MAX as u64 { - return Err(self.error("stack slot too large")); + return err!(self.loc, "stack slot too large"); } let data = StackSlotData::new(bytes as u32); @@ -446,11 +444,11 @@ impl<'a> Parser<'a> { fn parse_extended_basic_block(&mut self, ctx: &mut Context) -> Result<()> { let is_entry = self.optional(Token::Entry); let ebb_num = try!(self.match_ebb("expected EBB header")); - let ebb = try!(ctx.add_ebb(ebb_num, &self.location)); + let ebb = try!(ctx.add_ebb(ebb_num, &self.loc)); if is_entry { if ctx.function.entry_block != NO_EBB { - return Err(self.error("multiple entry blocks in function")); + return err!(self.loc, "multiple entry blocks in function"); } ctx.function.entry_block = ebb; } @@ -503,7 +501,7 @@ impl<'a> Parser<'a> { fn parse_ebb_arg(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { // ebb-arg ::= * Value(vx) ":" Type(t) let vx = try!(self.match_value("EBB argument must be a value")); - let vx_location = self.location; + let vx_location = self.loc; // ebb-arg ::= Value(vx) * ":" Type(t) try!(self.match_token(Token::Colon, "expected ':' after EBB argument")); // ebb-arg ::= Value(vx) ":" * Type(t) @@ -541,10 +539,10 @@ impl<'a> Parser<'a> { let opcode = if let Some(Token::Identifier(text)) = self.token() { match text.parse() { Ok(opc) => opc, - Err(msg) => return Err(self.error(msg)), + Err(msg) => return err!(self.loc, msg), } } else { - return Err(self.error("expected instruction opcode")); + return err!(self.loc, "expected instruction opcode"); }; self.consume(); @@ -570,10 +568,10 @@ impl<'a> Parser<'a> { ctx.function.append_inst(ebb, inst); if results.len() != num_results { - let m = format!("instruction produces {} result values, {} given", - num_results, - results.len()); - return Err(self.error_string(m)); + return err!(self.loc, + "instruction produces {} result values, {} given", + num_results, + results.len()); } // Now map the source result values to the just created instruction results. @@ -614,14 +612,15 @@ impl<'a> Parser<'a> { ctx.function.value_type(match ctx.values.get(&ctrl_src_value) { Some(&v) => v, None => { - let m = format!("cannot determine type of operand {}", ctrl_src_value); - return Err(self.error_string(m)); + return err!(self.loc, + "cannot determine type of operand {}", + ctrl_src_value); } }) } else if constraints.is_polymorphic() { // This opcode does not support type inference, so the explicit type variable // is required. - return Err(self.error("type variable required for polymorphic opcode")); + return err!(self.loc, "type variable required for polymorphic opcode"); } else { // This is a non-polymorphic opcode. No typevar needed. VOID @@ -635,13 +634,15 @@ impl<'a> Parser<'a> { if let Some(typeset) = constraints.ctrl_typeset() { // This is a polymorphic opcode. if !typeset.contains(ctrl_type) { - let m = format!("{} is not a valid typevar for {}", ctrl_type, opcode); - return Err(self.error_string(m)); + return err!(self.loc, + "{} is not a valid typevar for {}", + ctrl_type, + opcode); } } else { // Treat it as a syntax error to speficy a typevar on a non-polymorphic opcode. if ctrl_type != VOID { - return Err(self.error_string(format!("{} does not take a typevar", opcode))); + return err!(self.loc, "{} does not take a typevar", opcode); } } @@ -659,7 +660,7 @@ impl<'a> Parser<'a> { { for (src, val) in results.zip(new_results) { if values.insert(src, val).is_some() { - return Err(self.error_string(format!("duplicate result value: {}", src))); + return err!(self.loc, "duplicate result value: {}", src); } } Ok(()) From 61094f6909e8a8e863ecb800476be4fad8a6b1db Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 3 Jun 2016 14:56:25 -0700 Subject: [PATCH 0096/3084] Begin a basic command line interface. Add an external dependency to the docopt package and use it for a scaffold command line interface for the cton-util command. I am not too happy about taking external dependencies, and docopt pulls in 13 other packages. However, I really don't want to be writing command line parsers, and as long as the external dependencies are confined to the tools crate, we should be fine. The core cretonne crate should stay free of external dependencies to avoid trouble with embedding it. Implement a basic 'cat' subcommand which currently behaves like unix 'cat'. It will gain parser powers soon. --- cranelift/src/libcretonne/lib.rs | 2 + cranelift/src/tools/Cargo.lock | 101 +++++++++++++++++++++++++++++++ cranelift/src/tools/Cargo.toml | 2 + cranelift/src/tools/cat.rs | 28 +++++++++ cranelift/src/tools/main.rs | 62 ++++++++++++++++++- 5 files changed, 194 insertions(+), 1 deletion(-) create mode 100644 cranelift/src/tools/cat.rs diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 248c6bb37f..8b92e5dbed 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -5,6 +5,8 @@ // // ====------------------------------------------------------------------------------------==== // +pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); + pub mod types; pub mod immediates; pub mod entities; diff --git a/cranelift/src/tools/Cargo.lock b/cranelift/src/tools/Cargo.lock index 91e9a5fae4..0b85735d6c 100644 --- a/cranelift/src/tools/Cargo.lock +++ b/cranelift/src/tools/Cargo.lock @@ -4,6 +4,16 @@ version = "0.0.0" dependencies = [ "cretonne 0.0.0", "cretonne-reader 0.0.0", + "docopt 0.6.80 (registry+https://github.com/rust-lang/crates.io-index)", + "rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)", +] + +[[package]] +name = "aho-corasick" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +dependencies = [ + "memchr 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)", ] [[package]] @@ -17,3 +27,94 @@ dependencies = [ "cretonne 0.0.0", ] +[[package]] +name = "docopt" +version = "0.6.80" +source = "registry+https://github.com/rust-lang/crates.io-index" +dependencies = [ + "regex 0.1.71 (registry+https://github.com/rust-lang/crates.io-index)", + "rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)", + "strsim 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)", +] + +[[package]] +name = "kernel32-sys" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +dependencies = [ + "winapi 0.2.7 (registry+https://github.com/rust-lang/crates.io-index)", + "winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", +] + +[[package]] +name = "libc" +version = "0.2.11" +source = "registry+https://github.com/rust-lang/crates.io-index" + +[[package]] +name = "memchr" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +dependencies = [ + "libc 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)", +] + +[[package]] +name = "regex" +version = "0.1.71" +source = "registry+https://github.com/rust-lang/crates.io-index" +dependencies = [ + "aho-corasick 0.5.2 (registry+https://github.com/rust-lang/crates.io-index)", + "memchr 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)", + "regex-syntax 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)", + "thread_local 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)", + "utf8-ranges 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)", +] + +[[package]] +name = "regex-syntax" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" + +[[package]] +name = "rustc-serialize" +version = "0.3.19" +source = "registry+https://github.com/rust-lang/crates.io-index" + +[[package]] +name = "strsim" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" + +[[package]] +name = "thread-id" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +dependencies = [ + "kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)", + "libc 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)", +] + +[[package]] +name = "thread_local" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +dependencies = [ + "thread-id 2.0.0 (registry+https://github.com/rust-lang/crates.io-index)", +] + +[[package]] +name = "utf8-ranges" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" + +[[package]] +name = "winapi" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" + +[[package]] +name = "winapi-build" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" + diff --git a/cranelift/src/tools/Cargo.toml b/cranelift/src/tools/Cargo.toml index 702bd635e6..8059002f40 100644 --- a/cranelift/src/tools/Cargo.toml +++ b/cranelift/src/tools/Cargo.toml @@ -12,3 +12,5 @@ path = "main.rs" [dependencies] cretonne = { path = "../libcretonne" } cretonne-reader = { path = "../libreader" } +docopt = "0.6.80" +rustc-serialize = "0.3.19" diff --git a/cranelift/src/tools/cat.rs b/cranelift/src/tools/cat.rs new file mode 100644 index 0000000000..c8f30d0a99 --- /dev/null +++ b/cranelift/src/tools/cat.rs @@ -0,0 +1,28 @@ +//! The `cat` sub-command. +//! +//! Read a sequence of Cretonne IL files and print them again to stdout. This has the effect of +//! normalizing formatting and removing comments. + +use CommandResult; +use std::fs::File; +use std::io::Read; + +pub fn run(files: Vec) -> CommandResult { + for (i, f) in files.into_iter().enumerate() { + if i != 0 { + println!(""); + } + try!(cat_one(f)) + } + Ok(()) +} + +fn cat_one(filename: String) -> CommandResult { + let mut file = try!(File::open(&filename).map_err(|e| format!("{}: {}", filename, e))); + let mut buffer = String::new(); + try!(file.read_to_string(&mut buffer) + .map_err(|e| format!("Couldn't read {}: {}", filename, e))); + + print!("{}", buffer); + Ok(()) +} diff --git a/cranelift/src/tools/main.rs b/cranelift/src/tools/main.rs index 97b261d22d..1ea40c201c 100644 --- a/cranelift/src/tools/main.rs +++ b/cranelift/src/tools/main.rs @@ -1,5 +1,65 @@ extern crate cretonne; extern crate cton_reader; +extern crate docopt; +extern crate rustc_serialize; -fn main() {} +use cretonne::VERSION; +use docopt::Docopt; +use std::io::{self, Write}; +use std::process; + + +mod cat; + +const USAGE: &'static str = " +Cretonne code generator utility + +Usage: + cton-util cat ... + cton-util --help | --version + +Options: + -h, --help print this help message + --version print the Cretonne version + +"; + +#[derive(RustcDecodable, Debug)] +struct Args { + cmd_cat: bool, + arg_file: Vec, +} + +/// A command either succeeds or fails with an error message. +pub type CommandResult = Result<(), String>; + +/// Parse the command line arguments and run the requested command. +fn cton_util() -> CommandResult { + // Parse comand line arguments. + let args: Args = Docopt::new(USAGE) + .and_then(|d| { + d.help(true) + .version(Some(format!("Cretonne {}", VERSION))) + .decode() + }) + .unwrap_or_else(|e| e.exit()); + + // Find the sub-command to execute. + if args.cmd_cat { + cat::run(args.arg_file) + } else { + // Debugging / shouldn't happen with proper command line handling above. + Err(format!("Unhandled args: {:?}", args)) + } +} + +fn main() { + if let Err(mut msg) = cton_util() { + if !msg.ends_with('\n') { + msg.push('\n'); + } + io::stderr().write(msg.as_bytes()).unwrap(); + process::exit(1); + } +} From 5ce1a4f0e876381b66685b0989268400c71aa184 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 1 Jul 2016 14:09:34 -0700 Subject: [PATCH 0097/3084] Parse and write IR in the 'cat' subcommand. The 'cton-util cat' command parses the given files and writes them out again to stdout. This has the effect of reformatting and stripping comments. Fix a writer bug that inverted the blank line before the first EBB. --- cranelift/src/libcretonne/write.rs | 8 ++++---- cranelift/src/tools/cat.rs | 19 ++++++++++++++++--- 2 files changed, 20 insertions(+), 7 deletions(-) diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index 0d615c545b..38800bb48a 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -16,7 +16,7 @@ pub fn write_function(w: &mut Write, func: &Function) -> Result { try!(writeln!(w, " {{")); let mut any = try!(write_preamble(w, func)); for ebb in func.ebbs_numerically() { - if !any { + if any { try!(writeln!(w, "")); } try!(write_ebb(w, func, ebb)); @@ -209,14 +209,14 @@ mod tests { let ebb = f.make_ebb(); assert_eq!(function_to_string(&f), - "function foo() {\n ss0 = stack_slot 4\nebb0:\n}\n"); + "function foo() {\n ss0 = stack_slot 4\n\nebb0:\n}\n"); f.append_ebb_arg(ebb, types::I8); assert_eq!(function_to_string(&f), - "function foo() {\n ss0 = stack_slot 4\nebb0(vx0: i8):\n}\n"); + "function foo() {\n ss0 = stack_slot 4\n\nebb0(vx0: i8):\n}\n"); f.append_ebb_arg(ebb, types::F32.by(4).unwrap()); assert_eq!(function_to_string(&f), - "function foo() {\n ss0 = stack_slot 4\nebb0(vx0: i8, vx1: f32x4):\n}\n"); + "function foo() {\n ss0 = stack_slot 4\n\nebb0(vx0: i8, vx1: f32x4):\n}\n"); } } diff --git a/cranelift/src/tools/cat.rs b/cranelift/src/tools/cat.rs index c8f30d0a99..6d740da701 100644 --- a/cranelift/src/tools/cat.rs +++ b/cranelift/src/tools/cat.rs @@ -3,9 +3,13 @@ //! Read a sequence of Cretonne IL files and print them again to stdout. This has the effect of //! normalizing formatting and removing comments. -use CommandResult; use std::fs::File; -use std::io::Read; +use std::io::{self, Read}; + +use CommandResult; + +use cton_reader::parser::Parser; +use cretonne::write::write_function; pub fn run(files: Vec) -> CommandResult { for (i, f) in files.into_iter().enumerate() { @@ -22,7 +26,16 @@ fn cat_one(filename: String) -> CommandResult { let mut buffer = String::new(); try!(file.read_to_string(&mut buffer) .map_err(|e| format!("Couldn't read {}: {}", filename, e))); + let items = try!(Parser::parse(&buffer).map_err(|e| format!("{}: {}", filename, e))); + + for (idx, func) in items.into_iter().enumerate() { + if idx != 0 { + println!(""); + } + let stdout = io::stdout(); + let mut handle = stdout.lock(); + try!(write_function(&mut handle, &func).map_err(|e| format!("{}: {}", filename, e))); + } - print!("{}", buffer); Ok(()) } From 8e1de5c0f8bd850fdbb5076c12331b44509d46d4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 1 Jul 2016 14:23:54 -0700 Subject: [PATCH 0098/3084] Give a better error message for unknown opcodes. Include the name of the unrecognized opcode along with the line number. --- cranelift/src/libreader/parser.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 7d25fc096b..1fe3368241 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -539,7 +539,7 @@ impl<'a> Parser<'a> { let opcode = if let Some(Token::Identifier(text)) = self.token() { match text.parse() { Ok(opc) => opc, - Err(msg) => return err!(self.loc, msg), + Err(msg) => return err!(self.loc, "{}: '{}'", msg, text), } } else { return err!(self.loc, "expected instruction opcode"); From 320d5b369a4be9c6fc54684b9cd31500420d08ec Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 1 Jul 2016 14:26:24 -0700 Subject: [PATCH 0099/3084] rustfmt v0.5.0 --- cranelift/src/libcretonne/repr.rs | 4 ++-- cranelift/src/libcretonne/types.rs | 2 +- cranelift/src/libreader/parser.rs | 14 +++++++------- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index 9c44f524ca..a42b9de20f 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -148,8 +148,8 @@ impl Function { // Update the second_result pointer in `inst`. if head != NO_VALUE { *self.inst_mut(inst) - .second_result_mut() - .expect("instruction format doesn't allow multiple results") = head; + .second_result_mut() + .expect("instruction format doesn't allow multiple results") = head; } *self.inst_mut(inst).first_type_mut() = first_type; diff --git a/cranelift/src/libcretonne/types.rs b/cranelift/src/libcretonne/types.rs index bc4cecca6b..1c44736111 100644 --- a/cranelift/src/libcretonne/types.rs +++ b/cranelift/src/libcretonne/types.rs @@ -66,7 +66,7 @@ pub const B64: Type = Type(11); impl Type { /// Get the lane type of this SIMD vector type. - /// + /// /// A scalar type is the same as a SIMD vector type with one lane, so it returns itself. pub fn lane_type(self) -> Type { Type(self.0 & 0x0f) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 1fe3368241..94e94cbdcd 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -608,7 +608,7 @@ impl<'a> Parser<'a> { // explicitly. It is unfortunate that the correctness of IL depends on the // layout of the blocks. let ctrl_src_value = inst_data.typevar_operand() - .expect("Constraints <-> Format inconsistency"); + .expect("Constraints <-> Format inconsistency"); ctx.function.value_type(match ctx.values.get(&ctrl_src_value) { Some(&v) => v, None => { @@ -791,8 +791,8 @@ mod tests { assert_eq!(sig.return_types.len(), 0); let sig2 = Parser::new("(i8 inreg uext, f32, f64) -> i32 sext, f64") - .parse_signature() - .unwrap(); + .parse_signature() + .unwrap(); assert_eq!(sig2.to_string(), "(i8 uext inreg, f32, f64) -> i32 sext, f64"); @@ -811,8 +811,8 @@ mod tests { ss3 = stack_slot 13 ss1 = stack_slot 1 }") - .parse_function() - .unwrap(); + .parse_function() + .unwrap(); assert_eq!(func.name, "foo"); let mut iter = func.stack_slot_iter(); let ss0 = iter.next().unwrap(); @@ -840,8 +840,8 @@ mod tests { ebb0: ebb4(vx3: i32): }") - .parse_function() - .unwrap(); + .parse_function() + .unwrap(); assert_eq!(func.name, "ebbs"); let mut ebbs = func.ebbs_numerically(); From ab35a6e596a5e8fff91746b9a6e8f5ed9c31c2af Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 1 Jul 2016 14:32:04 -0700 Subject: [PATCH 0100/3084] Add a script for reformatting all sources. This is mostly useful when updating to a new version of rustfmt with different behavior. --- cranelift/src/format-all.sh | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100755 cranelift/src/format-all.sh diff --git a/cranelift/src/format-all.sh b/cranelift/src/format-all.sh new file mode 100755 index 0000000000..3569c460be --- /dev/null +++ b/cranelift/src/format-all.sh @@ -0,0 +1,14 @@ +#!/bin/bash + +# Format all sources using rustfmt. + +# Exit immediately on errors. +set -e + +cd $(dirname "$0") +src=$(pwd) + +for crate in $(find "$src" -name Cargo.toml); do + cd $(dirname "$crate") + cargo fmt +done From 56996adabd0df06ef650531f0ad088a03a525621 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Jul 2016 12:33:19 -0700 Subject: [PATCH 0101/3084] Don't return any values from inst_results() for VOID instructions. Instructions that don't produce any result values are marked with first_type() = VOID. The inst_results() iterator should not return any values for such instructions. --- cranelift/src/libcretonne/repr.rs | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index a42b9de20f..ff8d5d4dee 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -1,6 +1,6 @@ //! Representation of Cretonne IL functions. -use types::{Type, FunctionName, Signature}; +use types::{Type, FunctionName, Signature, VOID}; use entities::{Ebb, NO_EBB, Inst, NO_INST, Value, NO_VALUE, ExpandedValue, StackSlot}; use instructions::*; use std::fmt::{self, Display, Formatter}; @@ -167,7 +167,11 @@ impl Function { pub fn inst_results<'a>(&'a self, inst: Inst) -> Values<'a> { Values { func: self, - cur: Value::new_direct(inst), + cur: if self[inst].first_type() == VOID { + NO_VALUE + } else { + Value::new_direct(inst) + }, } } @@ -538,6 +542,26 @@ mod tests { let ins = &func[inst]; assert_eq!(ins.opcode(), Opcode::Iconst); assert_eq!(ins.first_type(), types::I32); + + // Result iterator. + let mut res = func.inst_results(inst); + assert!(res.next().is_some()); + assert!(res.next().is_none()); + } + + #[test] + fn no_results() { + let mut func = Function::new(); + + let idata = InstructionData::Nullary { + opcode: Opcode::Trap, + ty: types::VOID, + }; + let inst = func.make_inst(idata); + + // Result iterator should be empty. + let mut res = func.inst_results(inst); + assert_eq!(res.next(), None); } #[test] From b4525a329bd125312c6cf6bc1f1ad9b7f03877fe Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Jul 2016 12:49:34 -0700 Subject: [PATCH 0102/3084] Ignore comments in .cton files. The lexer still recognizes comments and generates tokens for them. They may be useful for test annotations at some point. --- cranelift/src/libreader/parser.rs | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 94e94cbdcd..ec584cdcaa 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -135,17 +135,23 @@ impl<'a> Parser<'a> { // Get the current lookahead token, after making sure there is one. fn token(&mut self) -> Option> { - if self.lookahead == None { + while self.lookahead == None { match self.lex.next() { Some(Ok(lexer::LocatedToken { token, location })) => { - self.lookahead = Some(token); + match token { + Token::Comment(_) => { + // Ignore comments. + } + _ => self.lookahead = Some(token), + } self.loc = location; } Some(Err(lexer::LocatedError { error, location })) => { self.lex_error = Some(error); self.loc = location; + break; } - None => {} + None => break, } } return self.lookahead; From 7b03ecfe04170ebce365821251d988ad6b7b179d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Jul 2016 11:40:42 -0700 Subject: [PATCH 0103/3084] Add very basic test framework for parser tests. Start with a shell script that runs .cton files through 'cton-util cat' and compares the output to a reference. This can get fancy later. --- cranelift/tests/parser/README.rst | 9 +++++++ cranelift/tests/parser/run.sh | 40 ++++++++++++++++++++++++++++ cranelift/tests/parser/tiny.cton | 5 ++++ cranelift/tests/parser/tiny.cton.ref | 4 +++ 4 files changed, 58 insertions(+) create mode 100644 cranelift/tests/parser/README.rst create mode 100755 cranelift/tests/parser/run.sh create mode 100644 cranelift/tests/parser/tiny.cton create mode 100644 cranelift/tests/parser/tiny.cton.ref diff --git a/cranelift/tests/parser/README.rst b/cranelift/tests/parser/README.rst new file mode 100644 index 0000000000..9ac4658d13 --- /dev/null +++ b/cranelift/tests/parser/README.rst @@ -0,0 +1,9 @@ +Parser tests +============ + +This directory contains test cases for the Cretonne IL parser. + +Each test case consists of a `foo.cton` input file and a `foo.ref` reference +output file. Each input file is run through the `cton-util cat` command, and the +output is compared against the reference file. If the two are identical, the +test passes. diff --git a/cranelift/tests/parser/run.sh b/cranelift/tests/parser/run.sh new file mode 100755 index 0000000000..e5795df1da --- /dev/null +++ b/cranelift/tests/parser/run.sh @@ -0,0 +1,40 @@ +#!/bin/bash + +# Go to tests directory. +cd $(dirname "$0")/.. + +# The path to cton-util should be in $CTONUTIL. +if [ -z "$CTONUTIL" ]; then + CTONUTIL=../src/tools/target/debug/cton-util +fi + +if [ ! -x "$CTONUTIL" ]; then + echo "Can't fund executable cton-util: $CTONUTIL" 1>&2 + exit 1 +fi + +declare -a fails + +for testcase in $(find parser -name '*.cton'); do + ref="${testcase}.ref" + if [ ! -r "$ref" ]; then + fails=(${fails[@]} "$testcase") + echo MISSING: $ref + elif diff -u "$ref" <("$CTONUTIL" cat "$testcase"); then + echo OK $testcase + else + fails=(${fails[@]} "$testcase") + echo FAIL $testcase + fi +done + +if [ ${#fails[@]} -ne 0 ]; then + echo + echo Failures: + for f in "${fails[@]}"; do + echo " $f" + done + exit 1 +else + echo "All passed" +fi diff --git a/cranelift/tests/parser/tiny.cton b/cranelift/tests/parser/tiny.cton new file mode 100644 index 0000000000..acc8ea7384 --- /dev/null +++ b/cranelift/tests/parser/tiny.cton @@ -0,0 +1,5 @@ +; The smallest possible function. +function minimal() { +ebb0: + trap +} diff --git a/cranelift/tests/parser/tiny.cton.ref b/cranelift/tests/parser/tiny.cton.ref new file mode 100644 index 0000000000..474c30527e --- /dev/null +++ b/cranelift/tests/parser/tiny.cton.ref @@ -0,0 +1,4 @@ +function minimal() { +ebb0: + trap +} From cbf78c294bb5b45eb407dd1f717e7e904ea94157 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Jul 2016 13:45:15 -0700 Subject: [PATCH 0104/3084] Print a type suffix on some polymorphic instructions. --- cranelift/src/libcretonne/write.rs | 78 ++++++++++++++++++++-------- cranelift/tests/parser/tiny.cton | 9 ++++ cranelift/tests/parser/tiny.cton.ref | 7 +++ 3 files changed, 72 insertions(+), 22 deletions(-) diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index 38800bb48a..e0d19aa5a8 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -7,6 +7,7 @@ use std::io::{self, Write}; use repr::Function; use entities::{Inst, Ebb, Value}; +use types::Type; pub type Result = io::Result<()>; @@ -126,6 +127,35 @@ pub fn write_ebb(w: &mut Write, func: &Function, ebb: Ebb) -> Result { // // ====--------------------------------------------------------------------------------------====// +// Should `inst` be printed with a type suffix? +// +// Polymorphic instructions may need a suffix indicating the value of the controlling type variable +// if it can't be trivially inferred. +// +fn type_suffix(func: &Function, inst: Inst) -> Option { + let constraints = func[inst].opcode().constraints(); + + if !constraints.is_polymorphic() { + return None; + } + + // If the controlling type variable can be inferred from the type of the designated value input + // operand, we don't need the type suffix. + // TODO: Should we include the suffix when the input value is defined in another block? The + // parser needs to know the type of the value, so it must be defined in a block that lexically + // comes before this one. + if constraints.use_typevar_operand() { + return None; + } + + // This polymorphic instruction doesn't support basic type inference. + // The controlling type variable is required to be the type of the first result. + let rtype = func.value_type(func.first_result(inst)); + assert!(!rtype.is_void(), + "Polymorphic instruction must produce a result"); + Some(rtype) +} + pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { try!(write!(w, " ")); @@ -143,30 +173,34 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { try!(write!(w, " = ")); } - // Then the opcode and operands, depending on format. + // Then the opcode, possibly with a '.type' suffix. + let opcode = func[inst].opcode(); + + match type_suffix(func, inst) { + Some(suf) => try!(write!(w, "{}.{}", opcode, suf)), + None => try!(write!(w, "{}", opcode)), + } + + // Then the operands, depending on format. use instructions::InstructionData::*; match func[inst] { - Nullary { opcode, .. } => writeln!(w, "{}", opcode), - Unary { opcode, arg, .. } => writeln!(w, "{} {}", opcode, arg), - UnaryImm { opcode, imm, .. } => writeln!(w, "{} {}", opcode, imm), - UnaryIeee32 { opcode, imm, .. } => writeln!(w, "{} {}", opcode, imm), - UnaryIeee64 { opcode, imm, .. } => writeln!(w, "{} {}", opcode, imm), - UnaryImmVector { opcode, .. } => writeln!(w, "{} [...]", opcode), - Binary { opcode, args, .. } => writeln!(w, "{} {}, {}", opcode, args[0], args[1]), - BinaryImm { opcode, arg, imm, .. } => writeln!(w, "{} {}, {}", opcode, arg, imm), - BinaryImmRev { opcode, imm, arg, .. } => writeln!(w, "{} {}, {}", opcode, imm, arg), - BinaryOverflow { opcode, args, .. } => writeln!(w, "{} {}, {}", opcode, args[0], args[1]), - Select { opcode, args, .. } => { - writeln!(w, "{} {}, {}, {}", opcode, args[0], args[1], args[2]) - } - InsertLane { opcode, lane, args, .. } => { - writeln!(w, "{} {}, {}, {}", opcode, args[0], lane, args[1]) - } - ExtractLane { opcode, lane, arg, .. } => writeln!(w, "{} {}, {}", opcode, arg, lane), - Jump { opcode, ref data, .. } => writeln!(w, "{} {}", opcode, data), - Branch { opcode, ref data, .. } => writeln!(w, "{} {}", opcode, data), - BranchTable { opcode, arg, table, .. } => writeln!(w, "{} {}, {}", opcode, arg, table), - Call { opcode, ref data, .. } => writeln!(w, "{} {}", opcode, data), + Nullary { .. } => writeln!(w, ""), + Unary { arg, .. } => writeln!(w, " {}", arg), + UnaryImm { imm, .. } => writeln!(w, " {}", imm), + UnaryIeee32 { imm, .. } => writeln!(w, " {}", imm), + UnaryIeee64 { imm, .. } => writeln!(w, " {}", imm), + UnaryImmVector { .. } => writeln!(w, " [...]"), + Binary { args, .. } => writeln!(w, " {}, {}", args[0], args[1]), + BinaryImm { arg, imm, .. } => writeln!(w, " {}, {}", arg, imm), + BinaryImmRev { imm, arg, .. } => writeln!(w, " {}, {}", imm, arg), + BinaryOverflow { args, .. } => writeln!(w, " {}, {}", args[0], args[1]), + Select { args, .. } => writeln!(w, " {}, {}, {}", args[0], args[1], args[2]), + InsertLane { lane, args, .. } => writeln!(w, " {}, {}, {}", args[0], lane, args[1]), + ExtractLane { lane, arg, .. } => writeln!(w, " {}, {}", arg, lane), + Jump { ref data, .. } => writeln!(w, " {}", data), + Branch { ref data, .. } => writeln!(w, " {}", data), + BranchTable { arg, table, .. } => writeln!(w, " {}, {}", arg, table), + Call { ref data, .. } => writeln!(w, " {}", data), } } diff --git a/cranelift/tests/parser/tiny.cton b/cranelift/tests/parser/tiny.cton index acc8ea7384..9619147f6f 100644 --- a/cranelift/tests/parser/tiny.cton +++ b/cranelift/tests/parser/tiny.cton @@ -3,3 +3,12 @@ function minimal() { ebb0: trap } + +; Create and use values. +; Polymorphic instructions with type suffix. +function ivalues() { +ebb0: + v0 = iconst.i32 2 + v1 = iconst.i8 6 + v2 = ishl v0, v1 +} diff --git a/cranelift/tests/parser/tiny.cton.ref b/cranelift/tests/parser/tiny.cton.ref index 474c30527e..c8c28dd419 100644 --- a/cranelift/tests/parser/tiny.cton.ref +++ b/cranelift/tests/parser/tiny.cton.ref @@ -2,3 +2,10 @@ function minimal() { ebb0: trap } + +function ivalues() { +ebb0: + v0 = iconst.i32 2 + v1 = iconst.i8 6 + v2 = ishl v0, v1 +} From 473e708dce64b3e244bc59db67f4ff90dc9cfe28 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Jul 2016 14:44:21 -0700 Subject: [PATCH 0105/3084] Parse branch and jump instructions. These instruction formats take EBB references with lists of argument values. For EBBs with no arguments, the argument value list may be omitted. --- cranelift/src/libcretonne/instructions.rs | 30 ++++++++--- cranelift/src/libreader/parser.rs | 66 +++++++++++++++++++++-- cranelift/tests/parser/branch.cton | 45 ++++++++++++++++ cranelift/tests/parser/branch.cton.ref | 39 ++++++++++++++ 4 files changed, 170 insertions(+), 10 deletions(-) create mode 100644 cranelift/tests/parser/branch.cton create mode 100644 cranelift/tests/parser/branch.cton.ref diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index 9ac3ac7220..d8880c28ed 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -207,6 +207,14 @@ impl VariableArgs { pub fn new() -> VariableArgs { VariableArgs(Vec::new()) } + + pub fn push(&mut self, v: Value) { + self.0.push(v) + } + + pub fn is_empty(&self) -> bool { + self.0.is_empty() + } } impl Display for VariableArgs { @@ -233,13 +241,17 @@ impl Default for VariableArgs { /// in the allowed InstructionData size. #[derive(Debug)] pub struct JumpData { - destination: Ebb, - arguments: VariableArgs, + pub destination: Ebb, + pub arguments: VariableArgs, } impl Display for JumpData { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - write!(f, "{}{}", self.destination, self.arguments) + if self.arguments.is_empty() { + write!(f, "{}", self.destination) + } else { + write!(f, "{}{}", self.destination, self.arguments) + } } } @@ -247,14 +259,18 @@ impl Display for JumpData { /// in the allowed InstructionData size. #[derive(Debug)] pub struct BranchData { - arg: Value, - destination: Ebb, - arguments: VariableArgs, + pub arg: Value, + pub destination: Ebb, + pub arguments: VariableArgs, } impl Display for BranchData { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - write!(f, "{}, {}{}", self.arg, self.destination, self.arguments) + try!(write!(f, "{}, {}", self.arg, self.destination)); + if !self.arguments.is_empty() { + try!(write!(f, "{}", self.arguments)); + } + Ok(()) } } diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index ec584cdcaa..eb57504a6f 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -13,7 +13,8 @@ use lexer::{self, Lexer, Token}; use cretonne::types::{Type, VOID, FunctionName, Signature, ArgumentType, ArgumentExtension}; use cretonne::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::entities::*; -use cretonne::instructions::{Opcode, InstructionFormat, InstructionData}; +use cretonne::instructions::{Opcode, InstructionFormat, InstructionData, VariableArgs, JumpData, + BranchData}; use cretonne::repr::{Function, StackSlotData}; pub use lexer::Location; @@ -672,6 +673,39 @@ impl<'a> Parser<'a> { Ok(()) } + // Parse comma-separated value list into a VariableArgs struct. + // + // value_list ::= [ value { "," value } ] + // + fn parse_value_list(&mut self) -> Result { + let mut args = VariableArgs::new(); + + if let Some(Token::Value(v)) = self.token() { + args.push(v); + self.consume(); + } else { + return Ok(args); + } + + while self.optional(Token::Comma) { + args.push(try!(self.match_value("expected value in argument list"))); + } + + Ok(args) + } + + // Parse an optional value list enclosed in parantheses. + fn parse_opt_value_list(&mut self) -> Result { + if !self.optional(Token::LPar) { + return Ok(VariableArgs::new()); + } + + let args = try!(self.parse_value_list()); + + try!(self.match_token(Token::RPar, "expected ')' after arguments")); + + Ok(args) + } // Parse the operands following the instruction opcode. // This depends on the format of the opcode. @@ -757,11 +791,37 @@ impl<'a> Parser<'a> { args: [lhs, rhs], } } + InstructionFormat::Jump => { + // Parse the destination EBB number. Don't translate source to local numbers yet. + let ebb_num = try!(self.match_ebb("expected jump destination EBB")); + let args = try!(self.parse_opt_value_list()); + InstructionData::Jump { + opcode: opcode, + ty: VOID, + data: Box::new(JumpData { + destination: ebb_num, + arguments: args, + }), + } + } + InstructionFormat::Branch => { + let ctrl_arg = try!(self.match_value("expected SSA value control operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let ebb_num = try!(self.match_ebb("expected branch destination EBB")); + let args = try!(self.parse_opt_value_list()); + InstructionData::Branch { + opcode: opcode, + ty: VOID, + data: Box::new(BranchData { + arg: ctrl_arg, + destination: ebb_num, + arguments: args, + }), + } + } InstructionFormat::Select | InstructionFormat::InsertLane | InstructionFormat::ExtractLane | - InstructionFormat::Jump | - InstructionFormat::Branch | InstructionFormat::BranchTable | InstructionFormat::Call => { unimplemented!(); diff --git a/cranelift/tests/parser/branch.cton b/cranelift/tests/parser/branch.cton new file mode 100644 index 0000000000..bf49ba8a9f --- /dev/null +++ b/cranelift/tests/parser/branch.cton @@ -0,0 +1,45 @@ +; Parsing branches and jumps. + +; Jumps with no arguments. The '()' empty argument list is optional. +function minimal() { +ebb0: + jump ebb1 + +ebb1: + jump ebb0() +} + +; Jumps with 1 arg. +function onearg(i32) { +ebb0(vx0: i32): + jump ebb1(vx0) + +ebb1(vx1: i32): + jump ebb0(vx1) +} + +; Jumps with 2 args. +function twoargs(i32, f32) { +ebb0(vx0: i32, vx1: f32): + jump ebb1(vx0, vx1) + +ebb1(vx2: i32, vx3: f32): + jump ebb0(vx2, vx3) +} + +; Branches with no arguments. The '()' empty argument list is optional. +function minimal(i32) { +ebb0(vx0: i32): + brz vx0, ebb1 + +ebb1: + brnz vx0, ebb1() +} + +function twoargs(i32, f32) { +ebb0(vx0: i32, vx1: f32): + brz vx0, ebb1(vx0, vx1) + +ebb1(vx2: i32, vx3: f32): + brnz vx0, ebb0(vx2, vx3) +} diff --git a/cranelift/tests/parser/branch.cton.ref b/cranelift/tests/parser/branch.cton.ref new file mode 100644 index 0000000000..02444326b1 --- /dev/null +++ b/cranelift/tests/parser/branch.cton.ref @@ -0,0 +1,39 @@ +function minimal() { +ebb0: + jump ebb1 + +ebb1: + jump ebb0 +} + +function onearg(i32) { +ebb0(vx0: i32): + jump ebb1(vx0) + +ebb1(vx1: i32): + jump ebb0(vx1) +} + +function twoargs(i32, f32) { +ebb0(vx0: i32, vx1: f32): + jump ebb1(vx0, vx1) + +ebb1(vx2: i32, vx3: f32): + jump ebb0(vx2, vx3) +} + +function minimal(i32) { +ebb0(vx0: i32): + brz vx0, ebb1 + +ebb1: + brnz vx0, ebb1 +} + +function twoargs(i32, f32) { +ebb0(vx0: i32, vx1: f32): + brz vx0, ebb1(vx0, vx1) + +ebb1(vx2: i32, vx3: f32): + brnz vx0, ebb0(vx2, vx3) +} From f18e5fe0fadac44f09526df3f9d3c24d94cd1542 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Jul 2016 16:51:26 -0700 Subject: [PATCH 0106/3084] Parse select instructions. --- cranelift/src/libreader/parser.rs | 13 ++++++++++++- cranelift/tests/parser/tiny.cton | 6 ++++++ cranelift/tests/parser/tiny.cton.ref | 5 +++++ 3 files changed, 23 insertions(+), 1 deletion(-) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index eb57504a6f..82e17aea5f 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -791,6 +791,18 @@ impl<'a> Parser<'a> { args: [lhs, rhs], } } + InstructionFormat::Select => { + let ctrl_arg = try!(self.match_value("expected SSA value control operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let true_arg = try!(self.match_value("expected SSA value true operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let false_arg = try!(self.match_value("expected SSA value false operand")); + InstructionData::Select { + opcode: opcode, + ty: VOID, + args: [ctrl_arg, true_arg, false_arg], + } + } InstructionFormat::Jump => { // Parse the destination EBB number. Don't translate source to local numbers yet. let ebb_num = try!(self.match_ebb("expected jump destination EBB")); @@ -819,7 +831,6 @@ impl<'a> Parser<'a> { }), } } - InstructionFormat::Select | InstructionFormat::InsertLane | InstructionFormat::ExtractLane | InstructionFormat::BranchTable | diff --git a/cranelift/tests/parser/tiny.cton b/cranelift/tests/parser/tiny.cton index 9619147f6f..5e38d5efbb 100644 --- a/cranelift/tests/parser/tiny.cton +++ b/cranelift/tests/parser/tiny.cton @@ -12,3 +12,9 @@ ebb0: v1 = iconst.i8 6 v2 = ishl v0, v1 } + +; Polymorphic istruction controlled by second operand. +function select() { +ebb0(vx0: i32, vx1:i32, vx2: b1): + v0 = select vx2, vx0, vx1 +} diff --git a/cranelift/tests/parser/tiny.cton.ref b/cranelift/tests/parser/tiny.cton.ref index c8c28dd419..29b604f29d 100644 --- a/cranelift/tests/parser/tiny.cton.ref +++ b/cranelift/tests/parser/tiny.cton.ref @@ -9,3 +9,8 @@ ebb0: v1 = iconst.i8 6 v2 = ishl v0, v1 } + +function select() { +ebb0(vx0: i32, vx1: i32, vx2: b1): + v0 = select vx2, vx0, vx1 +} From 7f8479e0971e55de6142d3720547cdf3afdca812 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 6 Jul 2016 13:36:15 -0700 Subject: [PATCH 0107/3084] Parse insertlane and extractlane instruction formats. These instruction formats take immediate lane index operands. We store these as u8 fields and require them to be in decimal format in the source. No hexadecimal lane indexes are supported. --- cranelift/src/libreader/parser.rs | 39 ++++++++++++++++++++++++++-- cranelift/tests/parser/tiny.cton | 10 ++++++- cranelift/tests/parser/tiny.cton.ref | 7 +++++ 3 files changed, 53 insertions(+), 3 deletions(-) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 82e17aea5f..ca8f56232b 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -247,6 +247,19 @@ impl<'a> Parser<'a> { } } + // Match and consume a u8 immediate. + // This is used for lane numbers in SIMD vectors. + fn match_u8(&mut self, err_msg: &str) -> Result { + if let Some(Token::Integer(text)) = self.token() { + self.consume(); + // Lexer just gives us raw text that looks like an integer. + // Parse it as a u8 to check for overflow and other issues. + text.parse().map_err(|_| self.error("expected u8 decimal immediate")) + } else { + err!(self.loc, err_msg) + } + } + // Match and consume an Ieee32 immediate. fn match_ieee32(&mut self, err_msg: &str) -> Result { if let Some(Token::Float(text)) = self.token() { @@ -831,8 +844,30 @@ impl<'a> Parser<'a> { }), } } - InstructionFormat::InsertLane | - InstructionFormat::ExtractLane | + InstructionFormat::InsertLane => { + let lhs = try!(self.match_value("expected SSA value first operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let lane = try!(self.match_u8("expected lane number")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let rhs = try!(self.match_value("expected SSA value last operand")); + InstructionData::InsertLane { + opcode: opcode, + ty: VOID, + lane: lane, + args: [lhs, rhs], + } + } + InstructionFormat::ExtractLane => { + let arg = try!(self.match_value("expected SSA value last operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let lane = try!(self.match_u8("expected lane number")); + InstructionData::ExtractLane { + opcode: opcode, + ty: VOID, + lane: lane, + arg: arg, + } + } InstructionFormat::BranchTable | InstructionFormat::Call => { unimplemented!(); diff --git a/cranelift/tests/parser/tiny.cton b/cranelift/tests/parser/tiny.cton index 5e38d5efbb..6be261e273 100644 --- a/cranelift/tests/parser/tiny.cton +++ b/cranelift/tests/parser/tiny.cton @@ -15,6 +15,14 @@ ebb0: ; Polymorphic istruction controlled by second operand. function select() { -ebb0(vx0: i32, vx1:i32, vx2: b1): +ebb0(vx0: i32, vx1: i32, vx2: b1): v0 = select vx2, vx0, vx1 } + +; Lane indexes. +function lanes() { +ebb0: + v0 = iconst.i32x4 2 + v1 = extractlane v0, 3 + v2 = insertlane v0, 1, v1 +} diff --git a/cranelift/tests/parser/tiny.cton.ref b/cranelift/tests/parser/tiny.cton.ref index 29b604f29d..1fc1745a34 100644 --- a/cranelift/tests/parser/tiny.cton.ref +++ b/cranelift/tests/parser/tiny.cton.ref @@ -14,3 +14,10 @@ function select() { ebb0(vx0: i32, vx1: i32, vx2: b1): v0 = select vx2, vx0, vx1 } + +function lanes() { +ebb0: + v0 = iconst.i32x4 2 + v1 = extractlane v0, 3 + v2 = insertlane v0, 1, v1 +} From 70507a3be04d1536b852a7651e8479f908a8c66c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 6 Jul 2016 15:02:12 -0700 Subject: [PATCH 0108/3084] Add enums for condition codes. The icmp and fmp instructions use different kinds of condition codes because integers and floating point values behave differently. Add a CondCode trait implementing shared behavior. --- cranelift/src/libcretonne/condcodes.rs | 325 +++++++++++++++++++++++++ cranelift/src/libcretonne/lib.rs | 1 + 2 files changed, 326 insertions(+) create mode 100644 cranelift/src/libcretonne/condcodes.rs diff --git a/cranelift/src/libcretonne/condcodes.rs b/cranelift/src/libcretonne/condcodes.rs new file mode 100644 index 0000000000..692ecda96f --- /dev/null +++ b/cranelift/src/libcretonne/condcodes.rs @@ -0,0 +1,325 @@ +//! Condition codes for the Cretonne code generator. +//! +//! A condition code here is an enumerated type that determined how to compare two numbers. There +//! are different rules for comparing integers and floating point numbers, so they use different +//! condition codes. + +use std::fmt::{self, Display, Formatter}; +use std::str::FromStr; + +/// Common traits of condition codes. +pub trait CondCode: Copy { + /// Get the inverse condition code of `self`. + /// + /// The inverse condition code produces the opposite result for all comparisons. + /// That is, `cmp CC, x, y` is true if and only if `cmp CC.inverse(), x, y` is false. + fn inverse(self) -> Self; + + /// Get the reversed condition code for `self`. + /// + /// The reversed condition code produces the same result as swapping `x` and `y` in the + /// comparison. That is, `cmp CC, x, y` is the same as `cmp CC.reverse(), y, x`. + fn reverse(self) -> Self; +} + +/// Condition code for comparing integers. +/// +/// This condition code is used by the `icmp` instruction to compare integer values. There are +/// separate codes for comparing the integers as signed or unsigned numbers where it makes a +/// difference. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum IntCC { + Equal, + NotEqual, + SignedLessThan, + SignedGreaterThanOrEqual, + SignedGreaterThan, + SignedLessThanOrEqual, + UnsignedLessThan, + UnsignedGreaterThanOrEqual, + UnsignedGreaterThan, + UnsignedLessThanOrEqual, +} + +impl CondCode for IntCC { + fn inverse(self) -> Self { + use self::IntCC::*; + match self { + Equal => NotEqual, + NotEqual => Equal, + SignedLessThan => SignedGreaterThanOrEqual, + SignedGreaterThanOrEqual => SignedLessThan, + SignedGreaterThan => SignedLessThanOrEqual, + SignedLessThanOrEqual => SignedGreaterThan, + UnsignedLessThan => UnsignedGreaterThanOrEqual, + UnsignedGreaterThanOrEqual => UnsignedLessThan, + UnsignedGreaterThan => UnsignedLessThanOrEqual, + UnsignedLessThanOrEqual => UnsignedGreaterThan, + } + } + + fn reverse(self) -> Self { + use self::IntCC::*; + match self { + Equal => Equal, + NotEqual => NotEqual, + SignedGreaterThan => SignedLessThan, + SignedGreaterThanOrEqual => SignedLessThanOrEqual, + SignedLessThan => SignedGreaterThan, + SignedLessThanOrEqual => SignedGreaterThanOrEqual, + UnsignedGreaterThan => UnsignedLessThan, + UnsignedGreaterThanOrEqual => UnsignedLessThanOrEqual, + UnsignedLessThan => UnsignedGreaterThan, + UnsignedLessThanOrEqual => UnsignedGreaterThanOrEqual, + } + } +} + +impl Display for IntCC { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + use self::IntCC::*; + f.write_str(match self { + &Equal => "eq", + &NotEqual => "ne", + &SignedGreaterThan => "sgt", + &SignedGreaterThanOrEqual => "sge", + &SignedLessThan => "slt", + &SignedLessThanOrEqual => "sle", + &UnsignedGreaterThan => "ugt", + &UnsignedGreaterThanOrEqual => "uge", + &UnsignedLessThan => "ult", + &UnsignedLessThanOrEqual => "ule", + }) + } +} + +impl FromStr for IntCC { + type Err = (); + + fn from_str(s: &str) -> Result { + use self::IntCC::*; + match s { + "eq" => Ok(Equal), + "ne" => Ok(NotEqual), + "sge" => Ok(SignedGreaterThanOrEqual), + "sgt" => Ok(SignedGreaterThan), + "sle" => Ok(SignedLessThanOrEqual), + "slt" => Ok(SignedLessThan), + "uge" => Ok(UnsignedGreaterThanOrEqual), + "ugt" => Ok(UnsignedGreaterThan), + "ule" => Ok(UnsignedLessThanOrEqual), + "ult" => Ok(UnsignedLessThan), + _ => Err(()), + } + } +} + +/// Condition code for comparing floating point numbers. +/// +/// This condition code is used by the `fcmp` instruction to compare floating point values. Two +/// IEEE floating point values relate in exactly one of four ways: +/// +/// 1. `UN` - unordered when either value is NaN. +/// 2. `EQ` - equal numerical value. +/// 3. `LT` - `x` is less than `y`. +/// 4. `GT` - `x` is greater than `y`. +/// +/// Note that `0.0` and `-0.0` relate as `EQ` because they both represent the number 0. +/// +/// The condition codes described here are used to produce a single boolean value from the +/// comparison. The 14 condition codes here cover every possible combination of the relation above +/// except the impossible `!UN & !EQ & !LT & !GT` and the always true `UN | EQ | LT | GT`. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum FloatCC { + Ordered, // EQ | LT | GT + Unordered, // UN + + Equal, // EQ + // The C '!=' operator is the inverse of '==': NotEqual. + NotEqual, // UN | LT | GT + OrderedNotEqual, // LT | GT + UnorderedOrEqual, // UN | EQ + + LessThan, // LT + LessThanOrEqual, // LT | EQ + GreaterThan, // GT + GreaterThanOrEqual, // GT | EQ + + UnorderedOrLessThan, // UN | LT + UnorderedOrLessThanOrEqual, // UN | LT | EQ + UnorderedOrGreaterThan, // UN | GT + UnorderedOrGreaterThanOrEqual, // UN | GT | EQ +} + +impl CondCode for FloatCC { + fn inverse(self) -> Self { + use self::FloatCC::*; + match self { + Ordered => Unordered, + Unordered => Ordered, + Equal => NotEqual, + NotEqual => Equal, + OrderedNotEqual => UnorderedOrEqual, + UnorderedOrEqual => OrderedNotEqual, + LessThan => UnorderedOrGreaterThanOrEqual, + LessThanOrEqual => UnorderedOrGreaterThan, + GreaterThan => UnorderedOrLessThanOrEqual, + GreaterThanOrEqual => UnorderedOrLessThan, + UnorderedOrLessThan => GreaterThanOrEqual, + UnorderedOrLessThanOrEqual => GreaterThan, + UnorderedOrGreaterThan => LessThanOrEqual, + UnorderedOrGreaterThanOrEqual => LessThan, + } + } + fn reverse(self) -> Self { + use self::FloatCC::*; + match self { + Ordered => Ordered, + Unordered => Unordered, + Equal => Equal, + NotEqual => NotEqual, + OrderedNotEqual => OrderedNotEqual, + UnorderedOrEqual => UnorderedOrEqual, + LessThan => GreaterThan, + LessThanOrEqual => GreaterThanOrEqual, + GreaterThan => LessThan, + GreaterThanOrEqual => LessThanOrEqual, + UnorderedOrLessThan => UnorderedOrGreaterThan, + UnorderedOrLessThanOrEqual => UnorderedOrGreaterThanOrEqual, + UnorderedOrGreaterThan => UnorderedOrLessThan, + UnorderedOrGreaterThanOrEqual => UnorderedOrLessThanOrEqual, + } + } +} + +impl Display for FloatCC { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + use self::FloatCC::*; + f.write_str(match self { + &Ordered => "ord", + &Unordered => "uno", + &Equal => "eq", + &NotEqual => "ne", + &OrderedNotEqual => "one", + &UnorderedOrEqual => "ueq", + &LessThan => "lt", + &LessThanOrEqual => "le", + &GreaterThan => "gt", + &GreaterThanOrEqual => "ge", + &UnorderedOrLessThan => "ult", + &UnorderedOrLessThanOrEqual => "ule", + &UnorderedOrGreaterThan => "ugt", + &UnorderedOrGreaterThanOrEqual => "uge", + }) + } +} + +impl FromStr for FloatCC { + type Err = (); + + fn from_str(s: &str) -> Result { + use self::FloatCC::*; + match s { + "ord" => Ok(Ordered), + "uno" => Ok(Unordered), + "eq" => Ok(Equal), + "ne" => Ok(NotEqual), + "one" => Ok(OrderedNotEqual), + "ueq" => Ok(UnorderedOrEqual), + "lt" => Ok(LessThan), + "le" => Ok(LessThanOrEqual), + "gt" => Ok(GreaterThan), + "ge" => Ok(GreaterThanOrEqual), + "ult" => Ok(UnorderedOrLessThan), + "ule" => Ok(UnorderedOrLessThanOrEqual), + "ugt" => Ok(UnorderedOrGreaterThan), + "uge" => Ok(UnorderedOrGreaterThanOrEqual), + _ => Err(()), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + static INT_ALL: [IntCC; 10] = [IntCC::Equal, + IntCC::NotEqual, + IntCC::SignedLessThan, + IntCC::SignedGreaterThanOrEqual, + IntCC::SignedGreaterThan, + IntCC::SignedLessThanOrEqual, + IntCC::UnsignedLessThan, + IntCC::UnsignedGreaterThanOrEqual, + IntCC::UnsignedGreaterThan, + IntCC::UnsignedLessThanOrEqual]; + + #[test] + fn int_inverse() { + for r in &INT_ALL { + let cc = *r; + let inv = cc.inverse(); + assert!(cc != inv); + assert_eq!(inv.inverse(), cc); + } + } + + #[test] + fn int_reverse() { + for r in &INT_ALL { + let cc = *r; + let rev = cc.reverse(); + assert_eq!(rev.reverse(), cc); + } + } + + #[test] + fn int_display() { + for r in &INT_ALL { + let cc = *r; + assert_eq!(cc.to_string().parse(), Ok(cc)); + } + } + + static FLOAT_ALL: [FloatCC; 14] = [FloatCC::Ordered, + FloatCC::Unordered, + FloatCC::Equal, + FloatCC::NotEqual, + FloatCC::OrderedNotEqual, + FloatCC::UnorderedOrEqual, + FloatCC::LessThan, + FloatCC::LessThanOrEqual, + FloatCC::GreaterThan, + FloatCC::GreaterThanOrEqual, + FloatCC::UnorderedOrLessThan, + FloatCC::UnorderedOrLessThanOrEqual, + FloatCC::UnorderedOrGreaterThan, + FloatCC::UnorderedOrGreaterThanOrEqual]; + + #[test] + fn float_inverse() { + for r in &FLOAT_ALL { + let cc = *r; + let inv = cc.inverse(); + assert!(cc != inv); + assert_eq!(inv.inverse(), cc); + } + } + + #[test] + fn float_reverse() { + for r in &FLOAT_ALL { + let cc = *r; + let rev = cc.reverse(); + assert_eq!(rev.reverse(), cc); + } + } + + #[test] + fn float_display() { + for r in &FLOAT_ALL { + let cc = *r; + assert_eq!(cc.to_string().parse(), Ok(cc)); + } + } +} diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 8b92e5dbed..26fc74f7ab 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -8,6 +8,7 @@ pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); pub mod types; +pub mod condcodes; pub mod immediates; pub mod entities; pub mod instructions; From 5d8fb0fdc3936e15d809e73141a7ce4364508fde Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 7 Jul 2016 11:20:56 -0700 Subject: [PATCH 0109/3084] Define icmp and fcmp comparison instructions. Add new intcc and floatcc operand types for the immediate condition codes on these instructions. Add new IntCompare and FloatCompare instruction formats. Add a generic match_enum() parser function that can match any identifier-like enumerated operand kind that implements FromStr. Define the icmp and fcmp instructions in case.py. Include documentation for the condition codes with these two instructions. --- cranelift/docs/langref.rst | 57 ++---------- cranelift/src/libcretonne/instructions.rs | 13 +++ cranelift/src/libcretonne/write.rs | 2 + cranelift/src/libreader/parser.rs | 43 ++++++++- cranelift/tests/parser/tiny.cton | 16 ++++ cranelift/tests/parser/tiny.cton.ref | 14 +++ meta/cretonne/base.py | 107 +++++++++++++++++++++- meta/cretonne/formats.py | 5 +- meta/cretonne/immediates.py | 13 +++ 9 files changed, 218 insertions(+), 52 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 8bf890b080..7df404e02c 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -270,6 +270,14 @@ indicate the different kinds of immediate operands on an instruction. bits of the operand are interpreted as if the SIMD vector was loaded from memory containing the immediate. +.. type:: intcc + + An integer condition code. See the :inst:`icmp` instruction for details. + +.. type:: floatcc + + A floating point condition code. See the :inst:`fcmp` instruction for details. + The two IEEE floating point immediate types :type:`ieee32` and :type:`ieee64` are displayed as hexadecimal floating point literals in the textual IL format. Decimal floating point literals are not allowed because some computer systems @@ -676,29 +684,7 @@ Vector operations Integer operations ------------------ -.. inst:: a = icmp Cond, x, y - - Integer comparison. - - :arg Cond: Condition code determining how ``x`` and ``y`` are compared. - :arg Int x: First value to compare. - :arg Int y: Second value to compare. - :result Logic a: With the same number of lanes as ``x`` and ``y``. - - The condition code determines if the operands are interpreted as signed or - unsigned integers. - - ====== ======== ========= - Signed Unsigned Condition - ====== ======== ========= - eq eq Equal - ne ne Not equal - slt ult Less than - sge uge Greater than or equal - sgt ugt Greater than - sle ule Less than or equal - ====== ======== ========= - +.. autoinst:: icmp .. autoinst:: iadd .. autoinst:: iadd_imm .. autoinst:: isub @@ -784,30 +770,7 @@ Floating point operations These operations generally follow IEEE 754-2008 semantics. -.. inst:: a = fcmp Cond, x, y - - Floating point comparison. - - :arg Cond: Condition code determining how ``x`` and ``y`` are compared. - :arg x,y: Floating point scalar or vector values of the same type. - :rtype: :type:`b1` or :type:`b1xN` with the same number of lanes as - ``x`` and ``y``. - - An 'ordered' condition code yields ``false`` if either operand is Nan. - - An 'unordered' condition code yields ``true`` if either operand is Nan. - - ======= ========= ========= - Ordered Unordered Condition - ======= ========= ========= - ord uno None (ord = no NaNs, uno = some NaNs) - oeq ueq Equal - one une Not equal - olt ult Less than - oge uge Greater than or equal - ogt ugt Greater than - ole ule Less than or equal - ======= ========= ========= +.. autoinst:: fcmp .. inst:: fadd x,y diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index d8880c28ed..106f506e47 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -11,6 +11,7 @@ use std::str::FromStr; use entities::*; use immediates::*; +use condcodes::*; use types::{self, Type}; // Include code generated by `meta/gen_instr.py`. This file contains: @@ -175,6 +176,18 @@ pub enum InstructionData { lane: u8, arg: Value, }, + IntCompare { + opcode: Opcode, + ty: Type, + cond: IntCC, + args: [Value; 2], + }, + FloatCompare { + opcode: Opcode, + ty: Type, + cond: FloatCC, + args: [Value; 2], + }, Jump { opcode: Opcode, ty: Type, diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index e0d19aa5a8..217d670afc 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -197,6 +197,8 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { Select { args, .. } => writeln!(w, " {}, {}, {}", args[0], args[1], args[2]), InsertLane { lane, args, .. } => writeln!(w, " {}, {}, {}", args[0], lane, args[1]), ExtractLane { lane, arg, .. } => writeln!(w, " {}, {}", arg, lane), + IntCompare { cond, args, .. } => writeln!(w, " {}, {}, {}", cond, args[0], args[1]), + FloatCompare { cond, args, .. } => writeln!(w, " {}, {}, {}", cond, args[0], args[1]), Jump { ref data, .. } => writeln!(w, " {}", data), Branch { ref data, .. } => writeln!(w, " {}", data), BranchTable { arg, table, .. } => writeln!(w, " {}, {}", arg, table), diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index ca8f56232b..a17449008e 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -8,6 +8,7 @@ use std::collections::HashMap; use std::result; use std::fmt::{self, Display, Formatter, Write}; +use std::str::FromStr; use std::u32; use lexer::{self, Lexer, Token}; use cretonne::types::{Type, VOID, FunctionName, Signature, ArgumentType, ArgumentExtension}; @@ -249,7 +250,7 @@ impl<'a> Parser<'a> { // Match and consume a u8 immediate. // This is used for lane numbers in SIMD vectors. - fn match_u8(&mut self, err_msg: &str) -> Result { + fn match_uimm8(&mut self, err_msg: &str) -> Result { if let Some(Token::Integer(text)) = self.token() { self.consume(); // Lexer just gives us raw text that looks like an integer. @@ -284,6 +285,16 @@ impl<'a> Parser<'a> { } } + // Match and consume an enumerated immediate, like one of the condition codes. + fn match_enum(&mut self, err_msg: &str) -> Result { + if let Some(Token::Identifier(text)) = self.token() { + self.consume(); + text.parse().map_err(|_| self.error(err_msg)) + } else { + err!(self.loc, err_msg) + } + } + /// Parse a list of function definitions. /// /// This is the top-level parse function matching the whole contents of a file. @@ -847,7 +858,7 @@ impl<'a> Parser<'a> { InstructionFormat::InsertLane => { let lhs = try!(self.match_value("expected SSA value first operand")); try!(self.match_token(Token::Comma, "expected ',' between operands")); - let lane = try!(self.match_u8("expected lane number")); + let lane = try!(self.match_uimm8("expected lane number")); try!(self.match_token(Token::Comma, "expected ',' between operands")); let rhs = try!(self.match_value("expected SSA value last operand")); InstructionData::InsertLane { @@ -860,7 +871,7 @@ impl<'a> Parser<'a> { InstructionFormat::ExtractLane => { let arg = try!(self.match_value("expected SSA value last operand")); try!(self.match_token(Token::Comma, "expected ',' between operands")); - let lane = try!(self.match_u8("expected lane number")); + let lane = try!(self.match_uimm8("expected lane number")); InstructionData::ExtractLane { opcode: opcode, ty: VOID, @@ -868,6 +879,32 @@ impl<'a> Parser<'a> { arg: arg, } } + InstructionFormat::IntCompare => { + let cond = try!(self.match_enum("expected intcc condition code")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let lhs = try!(self.match_value("expected SSA value first operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let rhs = try!(self.match_value("expected SSA value second operand")); + InstructionData::IntCompare { + opcode: opcode, + ty: VOID, + cond: cond, + args: [lhs, rhs], + } + } + InstructionFormat::FloatCompare => { + let cond = try!(self.match_enum("expected floatcc condition code")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let lhs = try!(self.match_value("expected SSA value first operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let rhs = try!(self.match_value("expected SSA value second operand")); + InstructionData::FloatCompare { + opcode: opcode, + ty: VOID, + cond: cond, + args: [lhs, rhs], + } + } InstructionFormat::BranchTable | InstructionFormat::Call => { unimplemented!(); diff --git a/cranelift/tests/parser/tiny.cton b/cranelift/tests/parser/tiny.cton index 6be261e273..e404985bf8 100644 --- a/cranelift/tests/parser/tiny.cton +++ b/cranelift/tests/parser/tiny.cton @@ -26,3 +26,19 @@ ebb0: v1 = extractlane v0, 3 v2 = insertlane v0, 1, v1 } + +; Integer condition codes. +function icmp(i32, i32) { +ebb0(vx0: i32, vx1: i32): + v0 = icmp eq, vx0, vx1 + v1 = icmp ult, vx0, vx1 + v2 = icmp sge, vx0, vx1 +} + +; Floating condition codes. +function fcmp(f32, f32) { +ebb0(vx0: f32, vx1: f32): + v0 = fcmp eq, vx0, vx1 + v1 = fcmp uno, vx0, vx1 + v2 = fcmp lt, vx0, vx1 +} diff --git a/cranelift/tests/parser/tiny.cton.ref b/cranelift/tests/parser/tiny.cton.ref index 1fc1745a34..31f5767335 100644 --- a/cranelift/tests/parser/tiny.cton.ref +++ b/cranelift/tests/parser/tiny.cton.ref @@ -21,3 +21,17 @@ ebb0: v1 = extractlane v0, 3 v2 = insertlane v0, 1, v1 } + +function icmp(i32, i32) { +ebb0(vx0: i32, vx1: i32): + v0 = icmp eq, vx0, vx1 + v1 = icmp ult, vx0, vx1 + v2 = icmp sge, vx0, vx1 +} + +function fcmp(f32, f32) { +ebb0(vx0: f32, vx1: f32): + v0 = fcmp eq, vx0, vx1 + v1 = fcmp uno, vx0, vx1 + v2 = fcmp lt, vx0, vx1 +} diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index b41225ae00..e6d008585b 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -6,7 +6,7 @@ support. """ from . import TypeVar, Operand, Instruction, InstructionGroup, variable_args from types import i8, f32, f64 -from immediates import imm64, uimm8, ieee32, ieee64, immvector +from immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc import entities instructions = InstructionGroup("base", "Shared base instruction set") @@ -217,6 +217,34 @@ extractlane = Instruction( # Integer arithmetic # +a = Operand('a', Int.as_bool()) +Cond = Operand('Cond', intcc) +x = Operand('x', Int) +y = Operand('y', Int) + +icmp = Instruction( + 'icmp', r""" + Integer comparison. + + The condition code determines if the operands are interpreted as signed + or unsigned integers. + + ====== ======== ========= + Signed Unsigned Condition + ====== ======== ========= + eq eq Equal + ne ne Not equal + slt ult Less than + sge uge Greater than or equal + sgt ugt Greater than + sle ule Less than or equal + ====== ======== ========= + + When this instruction compares integer vectors, it returns a boolean + vector of lane-wise comparisons. + """, + ins=(Cond, x, y), outs=a) + a = Operand('a', Int) x = Operand('x', Int) y = Operand('y', Int) @@ -515,4 +543,81 @@ popcnt = Instruction( """, ins=x, outs=a) +# +# Floating point. +# + +Float = TypeVar( + 'Float', 'A scalar or vector floating point type type', + floats=True, simd=True) + +Cond = Operand('Cond', floatcc) +x = Operand('x', Float) +y = Operand('y', Float) +a = Operand('a', Float.as_bool()) + +fcmp = Instruction( + 'fcmp', r""" + Floating point comparison. + + Two IEEE 754-2008 floating point numbers, `x` and `y`, relate to each + other in exactly one of four ways: + + == ========================================== + UN Unordered when one or both numbers is NaN. + EQ When :math:`x = y`. (And :math:`0.0 = -0.0`). + LT When :math:`x < y`. + GT When :math:`x > y`. + == ========================================== + + The 14 :type:`floatcc` condition codes each correspond to a subset of + the four relations, except for the empty set which would always be + false, and the full set which would always be true. + + The condition codes are divided into 7 'ordered' conditions which don't + include UN, and 7 unordered conditions which all include UN. + + +-------+------------+---------+------------+-------------------------+ + |Ordered |Unordered |Condition | + +=======+============+=========+============+=========================+ + |ord |EQ | LT | GT|uno |UN |NaNs absent / present. | + +-------+------------+---------+------------+-------------------------+ + |eq |EQ |ueq |UN | EQ |Equal | + +-------+------------+---------+------------+-------------------------+ + |one |LT | GT |ne |UN | LT | GT|Not equal | + +-------+------------+---------+------------+-------------------------+ + |lt |LT |ult |UN | LT |Less than | + +-------+------------+---------+------------+-------------------------+ + |le |LT | EQ |ule |UN | LT | EQ|Less than or equal | + +-------+------------+---------+------------+-------------------------+ + |gt |GT |ugt |UN | GT |Greater than | + +-------+------------+---------+------------+-------------------------+ + |ge |GT | EQ |uge |UN | GT | EQ|Greater than or equal | + +-------+------------+---------+------------+-------------------------+ + + The standard C comparison operators, `<, <=, >, >=`, are all ordered, + so they are false if either operand is NaN. The C equality operator, + `==`, is ordered, and since inequality is defined as the logical + inverse it is *unordered*. They map to the :type:`floatcc` condition + codes as follows: + + ==== ====== ============ + C `Cond` Subset + ==== ====== ============ + `==` eq EQ + `!=` ne UN | LT | GT + `<` lt LT + `<=` le LT | EQ + `>` gt GT + `>=` ge GT | EQ + ==== ====== ============ + + This subset of condition codes also corresponds to the WebAssembly + floating point comparisons of the same name. + + When this instruction compares floating point vectors, it returns a + boolean vector with the results of lane-wise comparisons. + """, + ins=(Cond, x, y), outs=a) + instructions.close() diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index 6f09c4b5af..62c678f9b9 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -8,7 +8,7 @@ in this module. from . import InstructionFormat, value, variable_args -from immediates import imm64, uimm8, ieee32, ieee64, immvector +from immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc from entities import ebb, function, jump_table Nullary = InstructionFormat() @@ -33,6 +33,9 @@ Select = InstructionFormat(value, value, value, typevar_operand=1) InsertLane = InstructionFormat(value, uimm8, value) ExtractLane = InstructionFormat(value, uimm8) +IntCompare = InstructionFormat(intcc, value, value) +FloatCompare = InstructionFormat(floatcc, value, value) + Jump = InstructionFormat(ebb, variable_args, boxed_storage=True) Branch = InstructionFormat(value, ebb, variable_args, boxed_storage=True) BranchTable = InstructionFormat(value, jump_table) diff --git a/meta/cretonne/immediates.py b/meta/cretonne/immediates.py index 299b340bce..f20a6142d3 100644 --- a/meta/cretonne/immediates.py +++ b/meta/cretonne/immediates.py @@ -29,3 +29,16 @@ ieee64 = ImmediateKind('ieee64', 'A 64-bit immediate floating point number.') #: A large SIMD vector constant. immvector = ImmediateKind('immvector', 'An immediate SIMD vector.') + +#: A condition code for comparing integer values. +#: +#: This enumerated operand kind is used for the :cton:inst:`icmp` instruction +#: and corresponds to the `condcodes::IntCC` Rust type. +intcc = ImmediateKind('intcc', 'An integer comparison condition code.') + +#: A condition code for comparing floating point values. +#: +#: This enumerated operand kind is used for the :cton:inst:`fcmp` instruction +#: and corresponds to the `condcodes::FloatCC` Rust type. +floatcc = ImmediateKind( + 'floatcc', 'A floating point comparison condition code.') From 81ca406dd0a934ca3ba9b04799e58a1b46a50465 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 7 Jul 2016 13:16:24 -0700 Subject: [PATCH 0110/3084] Add meta definitions for floating point operations. Rename the Select instruction format to Ternary since it is also used by the fma instruction. --- cranelift/docs/langref.rst | 89 +++++--------- cranelift/src/libcretonne/instructions.rs | 2 +- cranelift/src/libcretonne/write.rs | 2 +- cranelift/src/libreader/parser.rs | 6 +- meta/cretonne/base.py | 143 +++++++++++++++++++++- meta/cretonne/formats.py | 5 +- 6 files changed, 183 insertions(+), 64 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 7df404e02c..61c534ad69 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -771,73 +771,48 @@ Floating point operations These operations generally follow IEEE 754-2008 semantics. .. autoinst:: fcmp +.. autoinst:: fadd +.. autoinst:: fsub +.. autoinst:: fmul +.. autoinst:: fdiv +.. autoinst:: sqrt +.. autoinst:: fma -.. inst:: fadd x,y +Sign bit manipulations +~~~~~~~~~~~~~~~~~~~~~~ - Floating point addition. +The sign manipulating instructions work as bitwise operations, so they don't +have special behavior for signaling NaN operands. The exponent and trailing +significand bits are always preserved. -.. inst:: fsub x,y +.. autoinst:: fneg +.. autoinst:: fabs +.. autoinst:: fcopysign - Floating point subtraction. +Minimum and maximum +~~~~~~~~~~~~~~~~~~~ -.. inst:: fneg x +These instructions return the larger or smaller of their operands. They differ +in their handling of quiet NaN inputs. Note that signaling NaN operands always +cause a NaN result. - Floating point negation. +When comparing zeroes, these instructions behave as if :math:`-0.0 < 0.0`. - :result: ``x`` with its sign bit inverted. +.. autoinst:: fmin +.. autoinst:: fminnum +.. autoinst:: fmax +.. autoinst:: fmaxnum - Note that this is a pure bitwise operation. +Rounding +~~~~~~~~ -.. inst:: fabs x +These instructions round their argument to a nearby integral value, still +represented as a floating point number. - Floating point absolute value. - - :result: ``x`` with its sign bit cleared. - - Note that this is a pure bitwise operation. - -.. inst:: a = fcopysign x, y - - Floating point copy sign. - - :result: ``x`` with its sign changed to that of ``y``. - - Note that this is a pure bitwise operation. The sign bit from ``y`` is - copied to the sign bit of ``x``. - -.. inst:: a = fmul x, y -.. inst:: a = fdiv x, y -.. inst:: a = fmin x, y -.. inst:: a = fminnum x, y -.. inst:: a = fmax x, y -.. inst:: a = fmaxnum x, y - -.. inst:: a = ceil x - - Round floating point round to integral, towards positive infinity. - -.. inst:: floor x - - Round floating point round to integral, towards negative infinity. - -.. inst:: trunc x - - Round floating point round to integral, towards zero. - -.. inst:: nearest x - - Round floating point round to integral, towards nearest with ties to even. - -.. inst:: sqrt x - - Floating point square root. - -.. inst:: a = fma x, y, z - - Floating point fused multiply-and-add. - - Computes :math:`a := xy+z` wihtout any intermediate rounding of the - product. +.. autoinst:: ceil +.. autoinst:: floor +.. autoinst:: trunc +.. autoinst:: nearest Conversion operations --------------------- diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index 106f506e47..21138e2ce7 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -159,7 +159,7 @@ pub enum InstructionData { second_result: Value, args: [Value; 2], }, - Select { + Ternary { opcode: Opcode, ty: Type, args: [Value; 3], diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index 217d670afc..6a14497f81 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -194,7 +194,7 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { BinaryImm { arg, imm, .. } => writeln!(w, " {}, {}", arg, imm), BinaryImmRev { imm, arg, .. } => writeln!(w, " {}, {}", imm, arg), BinaryOverflow { args, .. } => writeln!(w, " {}, {}", args[0], args[1]), - Select { args, .. } => writeln!(w, " {}, {}, {}", args[0], args[1], args[2]), + Ternary { args, .. } => writeln!(w, " {}, {}, {}", args[0], args[1], args[2]), InsertLane { lane, args, .. } => writeln!(w, " {}, {}, {}", args[0], lane, args[1]), ExtractLane { lane, arg, .. } => writeln!(w, " {}, {}", arg, lane), IntCompare { cond, args, .. } => writeln!(w, " {}, {}, {}", cond, args[0], args[1]), diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index a17449008e..7977ccbf54 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -815,13 +815,15 @@ impl<'a> Parser<'a> { args: [lhs, rhs], } } - InstructionFormat::Select => { + InstructionFormat::Ternary => { + // Names here refer to the `select` instruction. + // This format is also use by `fma`. let ctrl_arg = try!(self.match_value("expected SSA value control operand")); try!(self.match_token(Token::Comma, "expected ',' between operands")); let true_arg = try!(self.match_value("expected SSA value true operand")); try!(self.match_token(Token::Comma, "expected ',' between operands")); let false_arg = try!(self.match_value("expected SSA value false operand")); - InstructionData::Select { + InstructionData::Ternary { opcode: opcode, ty: VOID, args: [ctrl_arg, true_arg, false_arg], diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index e6d008585b..fd3e4fdcb5 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -548,7 +548,7 @@ popcnt = Instruction( # Float = TypeVar( - 'Float', 'A scalar or vector floating point type type', + 'Float', 'A scalar or vector floating point number', floats=True, simd=True) Cond = Operand('Cond', floatcc) @@ -620,4 +620,145 @@ fcmp = Instruction( """, ins=(Cond, x, y), outs=a) +x = Operand('x', Float) +y = Operand('y', Float) +z = Operand('z', Float) +a = Operand('a', Float, 'Result of applying operator to each lane') + +fadd = Instruction( + 'fadd', r""" + Floating point addition. + """, + ins=(x, y), outs=a) + +fsub = Instruction( + 'fsub', r""" + Floating point subtraction. + """, + ins=(x, y), outs=a) + +fmul = Instruction( + 'fmul', r""" + Floating point multiplication. + """, + ins=(x, y), outs=a) + +fdiv = Instruction( + 'fdiv', r""" + Floating point division. + + Unlike the integer division instructions :cton:inst:`sdiv` and + :cton:inst:`udiv`, this can't trap. Division by zero is infinity or + NaN, depending on the dividend. + """, + ins=(x, y), outs=a) + +sqrt = Instruction( + 'sqrt', r""" + Floating point square root. + """, + ins=x, outs=a) + +fma = Instruction( + 'fma', r""" + Floating point fused multiply-and-add. + + Computes :math:`a := xy+z` wihtout any intermediate rounding of the + product. + """, + ins=(x, y, z), outs=a) + +a = Operand('a', Float, '``x`` with its sign bit inverted') +fneg = Instruction( + 'fneg', r""" + Floating point negation. + + Note that this is a pure bitwise operation. + """, + ins=x, outs=a) + +a = Operand('a', Float, '``x`` with its sign bit cleared') +fabs = Instruction( + 'fabs', r""" + Floating point absolute value. + + Note that this is a pure bitwise operation. + """, + ins=x, outs=a) + +a = Operand('a', Float, '``x`` with its sign bit changed to that of ``y``') +fcopysign = Instruction( + 'fcopysign', r""" + Floating point copy sign. + + Note that this is a pure bitwise operation. The sign bit from ``y`` is + copied to the sign bit of ``x``. + """, + ins=(x, y), outs=a) + +a = Operand('a', Float, 'The smaller of ``x`` and ``y``') + +fmin = Instruction( + 'fmin', r""" + Floating point minimum, propagating NaNs. + + If either operand is NaN, this returns a NaN. + """, + ins=(x, y), outs=a) + +fminnum = Instruction( + 'fminnum', r""" + Floating point minimum, suppressing quiet NaNs. + + If either operand is a quiet NaN, the other operand is returned. If + either operand is a signaling NaN, NaN is returned. + """, + ins=(x, y), outs=a) + +a = Operand('a', Float, 'The larger of ``x`` and ``y``') + +fmax = Instruction( + 'fmax', r""" + Floating point maximum, propagating NaNs. + + If either operand is NaN, this returns a NaN. + """, + ins=(x, y), outs=a) + +fmaxnum = Instruction( + 'fmaxnum', r""" + Floating point maximum, suppressing quiet NaNs. + + If either operand is a quiet NaN, the other operand is returned. If + either operand is a signaling NaN, NaN is returned. + """, + ins=(x, y), outs=a) + +a = Operand('a', Float, '``x`` rounded to integral value') + +ceil = Instruction( + 'ceil', r""" + Round floating point round to integral, towards positive infinity. + """, + ins=x, outs=a) + +floor = Instruction( + 'floor', r""" + Round floating point round to integral, towards negative infinity. + """, + ins=x, outs=a) + +trunc = Instruction( + 'trunc', r""" + Round floating point round to integral, towards zero. + """, + ins=x, outs=a) + +nearest = Instruction( + 'nearest', r""" + Round floating point round to integral, towards nearest with ties to + even. + """, + ins=x, outs=a) + instructions.close() diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index 62c678f9b9..5fc6cd7a2d 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -27,8 +27,9 @@ BinaryImmRev = InstructionFormat(imm64, value) BinaryOverflow = InstructionFormat(value, value, multiple_results=True) # The select instructions are controlled by the second value operand. -# The first value operand is the controlling flag whisch has a derived type. -Select = InstructionFormat(value, value, value, typevar_operand=1) +# The first value operand is the controlling flag which has a derived type. +# The fma instruction has the same constraint on all inputs. +Ternary = InstructionFormat(value, value, value, typevar_operand=1) InsertLane = InstructionFormat(value, uimm8, value) ExtractLane = InstructionFormat(value, uimm8) From c82b77231508bcb7556e422c7924ec0ad15b45f1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 7 Jul 2016 13:40:16 -0700 Subject: [PATCH 0111/3084] Add meta definition for bitcast. This instruction uses two type variables: input and output. Make sure that our parser can handle it. The output type variable annotation is mandatory. Add a ValueTypeSet::example() method which is used to provide better diagnostics for a missing type variable. --- cranelift/docs/langref.rst | 9 +-- cranelift/src/libcretonne/instructions.rs | 69 +++++++++++++++++++++++ cranelift/src/libreader/parser.rs | 5 +- cranelift/tests/parser/tiny.cton | 8 +++ cranelift/tests/parser/tiny.cton.ref | 6 ++ meta/cretonne/base.py | 25 ++++++++ 6 files changed, 113 insertions(+), 9 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 61c534ad69..0f49b8f6ed 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -817,14 +817,7 @@ represented as a floating point number. Conversion operations --------------------- -.. inst:: a = bitcast x - - Reinterpret the bits in ``x`` as a different type. - - The input and output types must be storable to memory and of the same size. - A bitcast is equivalent to storing one type and loading the other type from - the same address. - +.. autoinst:: bitcast .. inst:: a = itrunc x .. inst:: a = uext x .. inst:: a = sext x diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index 21138e2ce7..f6ca187511 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -432,6 +432,30 @@ impl ValueTypeSet { }; allowed && self.is_base_type(typ.lane_type()) } + + /// Get an example member of this type set. + /// + /// This is used for error messages to avoid suggesting invalid types. + pub fn example(&self) -> Type { + if self.base != types::VOID { + return self.base; + } + let t = if self.all_ints { + types::I32 + } else if self.all_floats { + types::F32 + } else if self.allow_scalars { + types::B1 + } else { + types::B32 + }; + + if self.allow_scalars { + t + } else { + t.by(4).unwrap() + } + } } /// Operand constraints. This describes the value type constraints on a single `Value` operand. @@ -506,4 +530,49 @@ mod tests { // that? assert_eq!(mem::size_of::(), 16); } + + #[test] + fn value_set() { + use types::*; + + let vts = ValueTypeSet { + allow_scalars: true, + allow_simd: true, + base: VOID, + all_ints: true, + all_floats: false, + all_bools: true, + }; + assert_eq!(vts.example().to_string(), "i32"); + + let vts = ValueTypeSet { + allow_scalars: true, + allow_simd: true, + base: VOID, + all_ints: false, + all_floats: true, + all_bools: true, + }; + assert_eq!(vts.example().to_string(), "f32"); + + let vts = ValueTypeSet { + allow_scalars: false, + allow_simd: true, + base: VOID, + all_ints: false, + all_floats: true, + all_bools: true, + }; + assert_eq!(vts.example().to_string(), "f32x4"); + + let vts = ValueTypeSet { + allow_scalars: false, + allow_simd: true, + base: VOID, + all_ints: false, + all_floats: false, + all_bools: true, + }; + assert_eq!(vts.example().to_string(), "b32x4"); + } } diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 7977ccbf54..cf0efb1ffd 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -651,7 +651,10 @@ impl<'a> Parser<'a> { } else if constraints.is_polymorphic() { // This opcode does not support type inference, so the explicit type variable // is required. - return err!(self.loc, "type variable required for polymorphic opcode"); + return err!(self.loc, + "type variable required for polymorphic opcode, e.g. '{}.{}'", + opcode, + constraints.ctrl_typeset().unwrap().example()); } else { // This is a non-polymorphic opcode. No typevar needed. VOID diff --git a/cranelift/tests/parser/tiny.cton b/cranelift/tests/parser/tiny.cton index e404985bf8..ae3466b618 100644 --- a/cranelift/tests/parser/tiny.cton +++ b/cranelift/tests/parser/tiny.cton @@ -42,3 +42,11 @@ ebb0(vx0: f32, vx1: f32): v1 = fcmp uno, vx0, vx1 v2 = fcmp lt, vx0, vx1 } + +; The bitcast instruction has two type variables: The controlling type variable +; controls the outout type, and the input type is a free variable. +function bitcast(i32, f32) { +ebb0(vx0: i32, vx1: f32): + v0 = bitcast.i8x4 vx0 + v1 = bitcast.i32 vx1 +} diff --git a/cranelift/tests/parser/tiny.cton.ref b/cranelift/tests/parser/tiny.cton.ref index 31f5767335..f2404bbf94 100644 --- a/cranelift/tests/parser/tiny.cton.ref +++ b/cranelift/tests/parser/tiny.cton.ref @@ -35,3 +35,9 @@ ebb0(vx0: f32, vx1: f32): v1 = fcmp uno, vx0, vx1 v2 = fcmp lt, vx0, vx1 } + +function bitcast(i32, f32) { +ebb0(vx0: i32, vx1: f32): + v0 = bitcast.i8x4 vx0 + v1 = bitcast.i32 vx1 +} diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index fd3e4fdcb5..15f23b1abc 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -761,4 +761,29 @@ nearest = Instruction( """, ins=x, outs=a) + +# +# Conversions +# + +Mem = TypeVar( + 'Mem', 'Any type that can be stored in memory', + ints=True, floats=True, simd=True) +MemTo = TypeVar( + 'MemTo', 'Any type that can be stored in memory', + ints=True, floats=True, simd=True) + +x = Operand('x', Mem) +a = Operand('a', MemTo, 'Bits of `x` reinterpreted') + +bitcast = Instruction( + 'bitcast', r""" + Reinterpret the bits in `x` as a different type. + + The input and output types must be storable to memory and of the same + size. A bitcast is equivalent to storing one type and loading the other + type from the same address. + """, + ins=x, outs=a) + instructions.close() From 31520717d3701c98ca80d77e3f8a60e9bc6abf04 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 7 Jul 2016 17:17:30 -0700 Subject: [PATCH 0112/3084] Include parser tests in the test-all.sh script. Move test-all.sh to the top level directory, and also run the parser tests from this script. Use a release build of cton-util to run the parser tests. As we accumulate many tests in the tests directory tree, this will mean they can still be run quickly. Point Travis config to the new test script. --- .travis.yml | 2 +- cranelift/src/test-all.sh | 12 ------------ cranelift/test-all.sh | 39 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 40 insertions(+), 13 deletions(-) delete mode 100755 cranelift/src/test-all.sh create mode 100755 cranelift/test-all.sh diff --git a/.travis.yml b/.travis.yml index 041073c206..1e6f291c4f 100644 --- a/.travis.yml +++ b/.travis.yml @@ -3,4 +3,4 @@ rust: - stable - beta - nightly -script: src/test-all.sh +script: test-all.sh diff --git a/cranelift/src/test-all.sh b/cranelift/src/test-all.sh deleted file mode 100755 index c0d9977ed1..0000000000 --- a/cranelift/src/test-all.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/bash - -# Exit immediately on errors. -set -e - -# Run from the src/tools directory which includes all our crates. -cd $(dirname "$0")/tools - -PKGS="-p cretonne -p cretonne-reader -p cretonne-tools" -cargo build $PKGS -cargo doc $PKGS -cargo test $PKGS diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh new file mode 100755 index 0000000000..502966e55d --- /dev/null +++ b/cranelift/test-all.sh @@ -0,0 +1,39 @@ +#!/bin/bash + +# This is the top-level test script: +# +# - Build documentation for Rust code in 'src/tools/target/doc'. +# - Run unit tests for all Rust crates. +# - Make a debug build of all crates. +# - Make a release build of cton-util. +# - Run file-level tests with the release build of cton-util. +# +# All tests run by this script should be passing at all times. + +# Exit immediately on errors. +set -e + +# Repository top-level directory. +cd $(dirname "$0") +topdir=$(pwd) + +# Run cargo from the src/tools directory which includes all our crates for +# building cton-util. +cd "$topdir/src/tools" +PKGS="-p cretonne -p cretonne-reader -p cretonne-tools" +echo ====== Rust unit tests and debug build ====== +cargo test $PKGS +cargo build $PKGS +cargo doc + +echo ====== Rust release build ====== +cargo build --release + +export CTONUTIL="$topdir/src/tools/target/release/cton-util" + +# Run the parser tests. +echo ====== Parser tests ====== +cd "$topdir/tests" +parser/run.sh + +echo ====== OK ====== From 05826c8a69743eb7bd046ed2328360b64faf5dab Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 7 Jul 2016 17:27:51 -0700 Subject: [PATCH 0113/3084] Fix Travis script path. --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 1e6f291c4f..567d824194 100644 --- a/.travis.yml +++ b/.travis.yml @@ -3,4 +3,4 @@ rust: - stable - beta - nightly -script: test-all.sh +script: ./test-all.sh From 62e96f4d3e2ed9751b802d3f491508724286bd85 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 7 Jul 2016 18:12:01 -0700 Subject: [PATCH 0114/3084] Metadefs for integer reduce and extend operations. Naming is interesting here. Since 'truncate' refers to removing the least significant digits, use 'ireduce' instead. The 'extend' use is fairly established. Don't abbreviate, avoid unfortunate modern vernacular. --- cranelift/docs/langref.rst | 6 ++--- meta/cretonne/base.py | 47 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+), 3 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 0f49b8f6ed..881976a701 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -818,9 +818,9 @@ Conversion operations --------------------- .. autoinst:: bitcast -.. inst:: a = itrunc x -.. inst:: a = uext x -.. inst:: a = sext x +.. autoinst:: ireduce +.. autoinst:: uextend +.. autoinst:: sextend .. inst:: a = ftrunc x .. inst:: a = fext x .. inst:: a = cvt_ftou x diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 15f23b1abc..b560b9bb7a 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -786,4 +786,51 @@ bitcast = Instruction( """, ins=x, outs=a) +Int = TypeVar('Int', 'A scalar or vector integer type', ints=True, simd=True) +IntTo = TypeVar('IntTo', 'A smaller integer type with the same number of lanes', ints=True, simd=True) + +x = Operand('x', Int) +a = Operand('a', IntTo) + +ireduce = Instruction( + 'ireduce', r""" + Convert `x` to a smaller integer type by dropping high bits. + + Each lane in `x` is converted to a smaller integer type by discarding + the most significant bits. This is the same as reducing modulo + :math:`2^n`. + + The result type must have the same number of vector lanes as the input, + and each lane must not have more bits that the input lanes. If the input + and output types are the same, this is a no-op. + """, + ins=x, outs=a) + +uextend = Instruction( + 'uextend', r""" + Convert `x` to a larger integer type by zero-extending. + + Each lane in `x` is converted to a larger integer type by adding zeroes. + The result has the same numerical value as `x` when both are interpreted + as unsigned integers. + + The result type must have the same number of vector lanes as the input, + and each lane must not have fewer bits that the input lanes. If the input + and output types are the same, this is a no-op. + """, + ins=x, outs=a) + +sextend = Instruction( + 'sextend', r""" + Convert `x` to a larger integer type by sign-extending. + + Each lane in `x` is converted to a larger integer type by replicating + the sign bit. The result has the same numerical value as `x` when both + are interpreted as signed integers. + + The result type must have the same number of vector lanes as the input, + and each lane must not have fewer bits that the input lanes. If the input + and output types are the same, this is a no-op. + """, + ins=x, outs=a) instructions.close() From 26ed45160ebe1b9f215fec6db1a701b3203fd259 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 7 Jul 2016 18:24:22 -0700 Subject: [PATCH 0115/3084] Fix rustc warning about unused Write trait. --- cranelift/src/libcretonne/entities.rs | 2 +- cranelift/src/libcretonne/types.rs | 2 +- cranelift/src/libreader/parser.rs | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/cranelift/src/libcretonne/entities.rs b/cranelift/src/libcretonne/entities.rs index dcdff9e822..6294a8e8c7 100644 --- a/cranelift/src/libcretonne/entities.rs +++ b/cranelift/src/libcretonne/entities.rs @@ -20,7 +20,7 @@ //! format. use std::default::Default; -use std::fmt::{self, Display, Formatter, Write}; +use std::fmt::{self, Display, Formatter}; use std::u32; /// An opaque reference to an extended basic block in a function. diff --git a/cranelift/src/libcretonne/types.rs b/cranelift/src/libcretonne/types.rs index 1c44736111..1bc0d0a0b4 100644 --- a/cranelift/src/libcretonne/types.rs +++ b/cranelift/src/libcretonne/types.rs @@ -2,7 +2,7 @@ //! Common types for the Cretonne code generator. use std::default::Default; -use std::fmt::{self, Display, Formatter, Write}; +use std::fmt::{self, Display, Formatter}; // ====--------------------------------------------------------------------------------------====// // diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index cf0efb1ffd..24d46b884d 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -7,7 +7,7 @@ use std::collections::HashMap; use std::result; -use std::fmt::{self, Display, Formatter, Write}; +use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::u32; use lexer::{self, Lexer, Token}; From ed535d90ad0c1186cbb25a2d998dcabd6b78c2ed Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 7 Jul 2016 19:24:08 -0700 Subject: [PATCH 0116/3084] Fix the recommended Sphinx version. The latest Sphinx 1.4.4 produces lots of warnings about four-column indices. We'll wait for Read the Docs to upgrade their systems before moving to the newer Sphinx version. --- README.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 0a26eae385..a80a23a283 100644 --- a/README.rst +++ b/README.rst @@ -42,8 +42,11 @@ Building the documentation To build the Cretonne documentation, you need the `Sphinx documentation generator `_:: - $ pip install sphinx + $ pip install sphinx==1.3.5 sphinx-autobuild $ cd cretonne/docs $ make html $ open _build/html/index.html +The specific Sphinx version is currently used by Read the Docs. Sphinx 1.4 has +been released, but produces lots of warnings about four-column indices. We'll +upgrade when Read the Docs does. From b710dd8464e20dadfc25d827d9aa59c3db39c97a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Jul 2016 11:20:19 -0700 Subject: [PATCH 0117/3084] Define floating point conversion instructions. --- cranelift/docs/langref.rst | 13 ++-- meta/cretonne/base.py | 119 +++++++++++++++++++++++++++++++++---- 2 files changed, 115 insertions(+), 17 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 881976a701..9181f10e26 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -821,13 +821,12 @@ Conversion operations .. autoinst:: ireduce .. autoinst:: uextend .. autoinst:: sextend -.. inst:: a = ftrunc x -.. inst:: a = fext x -.. inst:: a = cvt_ftou x -.. inst:: a = cvt_ftos x -.. inst:: a = cvt_utof x -.. inst:: a = cvt_stof x - +.. autoinst:: fpromote +.. autoinst:: fdemote +.. autoinst:: fcvt_to_uint +.. autoinst:: fcvt_to_sint +.. autoinst:: fcvt_from_uint +.. autoinst:: fcvt_from_sint Glossary ======== diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index b560b9bb7a..894ac2f505 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -787,7 +787,9 @@ bitcast = Instruction( ins=x, outs=a) Int = TypeVar('Int', 'A scalar or vector integer type', ints=True, simd=True) -IntTo = TypeVar('IntTo', 'A smaller integer type with the same number of lanes', ints=True, simd=True) +IntTo = TypeVar( + 'IntTo', 'A smaller integer type with the same number of lanes', + ints=True, simd=True) x = Operand('x', Int) a = Operand('a', IntTo) @@ -801,22 +803,30 @@ ireduce = Instruction( :math:`2^n`. The result type must have the same number of vector lanes as the input, - and each lane must not have more bits that the input lanes. If the input - and output types are the same, this is a no-op. + and each lane must not have more bits that the input lanes. If the + input and output types are the same, this is a no-op. """, ins=x, outs=a) + +IntTo = TypeVar( + 'IntTo', 'A larger integer type with the same number of lanes', + ints=True, simd=True) + +x = Operand('x', Int) +a = Operand('a', IntTo) + uextend = Instruction( 'uextend', r""" Convert `x` to a larger integer type by zero-extending. - Each lane in `x` is converted to a larger integer type by adding zeroes. - The result has the same numerical value as `x` when both are interpreted - as unsigned integers. + Each lane in `x` is converted to a larger integer type by adding + zeroes. The result has the same numerical value as `x` when both are + interpreted as unsigned integers. The result type must have the same number of vector lanes as the input, - and each lane must not have fewer bits that the input lanes. If the input - and output types are the same, this is a no-op. + and each lane must not have fewer bits that the input lanes. If the + input and output types are the same, this is a no-op. """, ins=x, outs=a) @@ -829,8 +839,97 @@ sextend = Instruction( are interpreted as signed integers. The result type must have the same number of vector lanes as the input, - and each lane must not have fewer bits that the input lanes. If the input - and output types are the same, this is a no-op. + and each lane must not have fewer bits that the input lanes. If the + input and output types are the same, this is a no-op. """, ins=x, outs=a) + +FloatTo = TypeVar( + 'FloatTo', 'A scalar or vector floating point number', + floats=True, simd=True) + +x = Operand('x', Float) +a = Operand('a', FloatTo) + +fpromote = Instruction( + 'fcvt_ftof', r""" + Convert `x` to a larger floating point format. + + Each lane in `x` is converted to the destination floating point format. + This is an exact operation. + + Since Cretonne currently only supports two floating point formats, this + instruction always converts :type:`f32` to :type:`f64`. This may change + in the future. + + The result type must have the same number of vector lanes as the input, + and the result lanes must be larger than the input lanes. + """, + ins=x, outs=a) + +fdemote = Instruction( + 'fdemote', r""" + Convert `x` to a smaller floating point format. + + Each lane in `x` is converted to the destination floating point format + by rounding to nearest, ties to even. + + Since Cretonne currently only supports two floating point formats, this + instruction always converts :type:`f64` to :type:`f32`. This may change + in the future. + + The result type must have the same number of vector lanes as the input, + and the result lanes must be smaller than the input lanes. + """, + ins=x, outs=a) + +x = Operand('x', Float) +a = Operand('a', Int) + +fcvt_to_uint = Instruction( + 'fcvt_to_uint', r""" + Convert floating point to unsigned integer. + + Each lane in `x` is converted to an unsigned integer by rounding + towards zero. If `x` is NaN or if the unsigned integral value cannot be + represented in the result type, this instruction traps. + + The result type must have the same number of vector lanes as the input. + """, + ins=x, outs=a) + +fcvt_to_sint = Instruction( + 'fcvt_to_sint', r""" + Convert floating point to signed integer. + + Each lane in `x` is converted to a signed integer by rounding towards + zero. If `x` is NaN or if the signed integral value cannot be + represented in the result type, this instruction traps. + + The result type must have the same number of vector lanes as the input. + """, + ins=x, outs=a) + +fcvt_from_uint = Instruction( + 'fcvt_from_uint', r""" + Convert unsigned integer to floating point. + + Each lane in `x` is interpreted as an unsigned integer and converted to + floating point using round to nearest, ties to even. + + The result type must have the same number of vector lanes as the input. + """, + ins=x, outs=a) + +fcvt_from_sint = Instruction( + 'fcvt_from_sint', r""" + Convert signed integer to floating point. + + Each lane in `x` is interpreted as a signed integer and converted to + floating point using round to nearest, ties to even. + + The result type must have the same number of vector lanes as the input. + """, + ins=x, outs=a) + instructions.close() From 3c5c5a9e40b851751a81446f0c7c2ad46fad0c5c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Jul 2016 11:44:20 -0700 Subject: [PATCH 0118/3084] Don't print a space after quoted function names. --- cranelift/src/libcretonne/write.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index 6a14497f81..22d87ba103 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -63,7 +63,7 @@ fn write_spec(w: &mut Write, func: &Function) -> Result { if !needs_quotes(&func.name) { write!(w, "function {}{}", func.name, sig) } else { - write!(w, "function \"{}\" {}", escaped(&func.name), sig) + write!(w, "function \"{}\"{}", escaped(&func.name), sig) } } @@ -234,7 +234,7 @@ mod tests { #[test] fn basic() { let mut f = Function::new(); - assert_eq!(function_to_string(&f), "function \"\" () {\n}\n"); + assert_eq!(function_to_string(&f), "function \"\"() {\n}\n"); f.name.push_str("foo"); assert_eq!(function_to_string(&f), "function foo() {\n}\n"); From 6587784d7de1db5373f6f194bdab024e9fdc8616 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Jul 2016 15:15:53 -0700 Subject: [PATCH 0119/3084] Rewrite EBB and value references after parsing. We llow forward references to values and EBBs, so it is not possible to rewrite these from the source domain to the in-memory domain during parsing. Instead go through all the instructions after parsing everything and rewrite the value and EBB references when everything has been created and mapped. --- cranelift/src/libcretonne/instructions.rs | 18 +++- cranelift/src/libcretonne/repr.rs | 15 ++-- cranelift/src/libreader/parser.rs | 105 ++++++++++++++++++++++ cranelift/tests/parser/rewrite.cton | 24 +++++ cranelift/tests/parser/rewrite.cton.ref | 13 +++ 5 files changed, 168 insertions(+), 7 deletions(-) create mode 100644 cranelift/tests/parser/rewrite.cton create mode 100644 cranelift/tests/parser/rewrite.cton.ref diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index f6ca187511..9a87e85d71 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -8,6 +8,7 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; +use std::ops::{Deref, DerefMut}; use entities::*; use immediates::*; @@ -230,6 +231,21 @@ impl VariableArgs { } } +// Coerce VariableArgs into a &[Value] slice. +impl Deref for VariableArgs { + type Target = [Value]; + + fn deref<'a>(&'a self) -> &'a [Value] { + &self.0 + } +} + +impl DerefMut for VariableArgs { + fn deref_mut<'a>(&'a mut self) -> &'a mut [Value] { + &mut self.0 + } +} + impl Display for VariableArgs { fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { try!(write!(fmt, "(")); @@ -294,7 +310,7 @@ pub struct CallData { second_result: Value, // Dynamically sized array containing call argument values. - arguments: VariableArgs, + pub arguments: VariableArgs, } impl Display for CallData { diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index ff8d5d4dee..5a36115fbd 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -101,10 +101,6 @@ impl Function { inst } - fn inst_mut(&mut self, inst: Inst) -> &mut InstructionData { - &mut self.instructions[inst.index()] - } - /// Create result values for an instruction that produces multiple results. /// /// Instructions that produce 0 or 1 result values only need to be created with `make_inst`. If @@ -147,11 +143,11 @@ impl Function { // Update the second_result pointer in `inst`. if head != NO_VALUE { - *self.inst_mut(inst) + *self[inst] .second_result_mut() .expect("instruction format doesn't allow multiple results") = head; } - *self.inst_mut(inst).first_type_mut() = first_type; + *self[inst].first_type_mut() = first_type; fixed_results } @@ -456,6 +452,13 @@ impl Index for Function { } } +/// Allow mutable access to instructions via function indexing. +impl IndexMut for Function { + fn index_mut<'a>(&'a mut self, inst: Inst) -> &'a mut InstructionData { + &mut self.instructions[inst.index()] + } +} + /// A node in a double linked list of instructions is a basic block. #[derive(Debug)] struct InstNode { diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 24d46b884d..e55a277d18 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -73,6 +73,9 @@ struct Context { stack_slots: HashMap, // ssNN ebbs: HashMap, // ebbNN values: HashMap, // vNN, vxNN + + // Remember the location of every instruction. + inst_locs: Vec<(Inst, Location)>, } impl Context { @@ -82,6 +85,7 @@ impl Context { stack_slots: HashMap::new(), ebbs: HashMap::new(), values: HashMap::new(), + inst_locs: Vec::new(), } } @@ -112,6 +116,101 @@ impl Context { Ok(()) } } + + // Record the location of an instuction. + fn add_inst_loc(&mut self, inst: Inst, loc: &Location) { + self.inst_locs.push((inst, *loc)); + } + + // The parser creates all instructions with Ebb and Value references using the source file + // numbering. These references need to be rewritten after parsing is complete since forward + // references are allowed. + + // Rewrite an Ebb reference. + fn rewrite_ebb(map: &HashMap, ebb: &mut Ebb, loc: &Location) -> Result<()> { + match map.get(ebb) { + Some(&new) => { + *ebb = new; + Ok(()) + } + None => err!(loc, "undefined reference: {}", ebb), + } + } + + // Rewrite a value reference. + fn rewrite_value(map: &HashMap, val: &mut Value, loc: &Location) -> Result<()> { + match map.get(val) { + Some(&new) => { + *val = new; + Ok(()) + } + None => err!(loc, "undefined reference: {}", val), + } + } + + // Rewrite a slice of value references. + fn rewrite_values(map: &HashMap, + vals: &mut [Value], + loc: &Location) + -> Result<()> { + for val in vals { + try!(Self::rewrite_value(map, val, loc)); + } + Ok(()) + } + + // Rewrite all EBB and value references in the function. + fn rewrite_references(&mut self) -> Result<()> { + for &(inst, loc) in &self.inst_locs { + match self.function[inst] { + InstructionData::Nullary { .. } | + InstructionData::UnaryImm { .. } | + InstructionData::UnaryIeee32 { .. } | + InstructionData::UnaryIeee64 { .. } | + InstructionData::UnaryImmVector { .. } => {} + + InstructionData::Unary { ref mut arg, .. } | + InstructionData::BinaryImm { ref mut arg, .. } | + InstructionData::BinaryImmRev { ref mut arg, .. } | + InstructionData::ExtractLane { ref mut arg, .. } | + InstructionData::BranchTable { ref mut arg, .. } => { + try!(Self::rewrite_value(&self.values, arg, &loc)); + } + + InstructionData::Binary { ref mut args, .. } | + InstructionData::BinaryOverflow { ref mut args, .. } | + InstructionData::InsertLane { ref mut args, .. } | + InstructionData::IntCompare { ref mut args, .. } | + InstructionData::FloatCompare { ref mut args, .. } => { + try!(Self::rewrite_values(&self.values, args, &loc)); + } + + InstructionData::Ternary { ref mut args, .. } => { + try!(Self::rewrite_values(&self.values, args, &loc)); + } + + InstructionData::Jump { ref mut data, .. } => { + try!(Self::rewrite_ebb(&self.ebbs, &mut data.destination, &loc)); + try!(Self::rewrite_values(&self.values, &mut data.arguments, &loc)); + } + + InstructionData::Branch { ref mut data, .. } => { + try!(Self::rewrite_value(&self.values, &mut data.arg, &loc)); + try!(Self::rewrite_ebb(&self.ebbs, &mut data.destination, &loc)); + try!(Self::rewrite_values(&self.values, &mut data.arguments, &loc)); + } + + InstructionData::Call { ref mut data, .. } => { + try!(Self::rewrite_values(&self.values, &mut data.arguments, &loc)); + } + } + } + + // TODO: Rewrite EBB references in jump tables. (Once jump table data structures are + // defined). + + Ok(()) + } } impl<'a> Parser<'a> { @@ -323,6 +422,10 @@ impl<'a> Parser<'a> { // function ::= function-spec "{" preamble function-body * "}" try!(self.match_token(Token::RBrace, "expected '}' after function body")); + // Rewrite references to values and EBBs after parsing everuthing to allow forward + // references. + try!(ctx.rewrite_references()); + Ok(ctx.function) } @@ -575,6 +678,7 @@ impl<'a> Parser<'a> { } else { return err!(self.loc, "expected instruction opcode"); }; + let opcode_loc = self.loc; self.consume(); // Look for a controlling type variable annotation. @@ -597,6 +701,7 @@ impl<'a> Parser<'a> { let inst = ctx.function.make_inst(inst_data); let num_results = ctx.function.make_inst_results(inst, ctrl_typevar); ctx.function.append_inst(ebb, inst); + ctx.add_inst_loc(inst, &opcode_loc); if results.len() != num_results { return err!(self.loc, diff --git a/cranelift/tests/parser/rewrite.cton b/cranelift/tests/parser/rewrite.cton new file mode 100644 index 0000000000..33e5277db6 --- /dev/null +++ b/cranelift/tests/parser/rewrite.cton @@ -0,0 +1,24 @@ +; The .cton parser can't preserve the actual entity numbers in the input file +; since entities are numbered as they are created. For entities declared in the +; preamble, this is no problem, but for EBB and value references, mapping +; source numbers to real numbers can be a problem. +; +; It is possible to refer to instructions and EBBs that have not yet been +; defined in the lexical order, so the parser needs to rewrite these references +; after the fact. + +; Check that defining numbers are rewritten. +function defs() { +ebb100(v20: i32): + v1000 = iconst.i32x8 5 + vx200 = f64const 0x4.0p0 + trap +} + +; Using values. +function use_value() { +ebb100(v20: i32): + v1000 = iadd_imm v20, 5 + vx200 = iadd v20, v1000 + jump ebb100(v1000) +} diff --git a/cranelift/tests/parser/rewrite.cton.ref b/cranelift/tests/parser/rewrite.cton.ref new file mode 100644 index 0000000000..8f379abd64 --- /dev/null +++ b/cranelift/tests/parser/rewrite.cton.ref @@ -0,0 +1,13 @@ +function defs() { +ebb0(vx0: i32): + v0 = iconst.i32x8 5 + v1 = f64const 0x1.0000000000000p2 + trap +} + +function "use_value"() { +ebb0(vx0: i32): + v0 = iadd_imm vx0, 5 + v1 = iadd vx0, v0 + jump ebb0(v0) +} From 3839281414af7e3a8429dd2f0f3d17d042209409 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Jul 2016 16:19:26 -0700 Subject: [PATCH 0120/3084] Define a return instruction. It is possible to return multiple values from a function, so ReturnData contains a VariableArgs instance. We don't want return instructions to appear as 'return (v1)', so tweak the printing of VariableArgs so the parantheses are added externally. --- cranelift/docs/cton_domain.py | 9 ++++++-- cranelift/docs/langref.rst | 11 +--------- cranelift/src/libcretonne/instructions.rs | 25 ++++++++++++++++------- cranelift/src/libcretonne/write.rs | 7 +++++++ cranelift/src/libreader/parser.rs | 16 +++++++++++++-- meta/cretonne/base.py | 12 +++++++++++ meta/cretonne/formats.py | 2 ++ 7 files changed, 61 insertions(+), 21 deletions(-) diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index bd207e994d..aafb7724cf 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -265,12 +265,17 @@ class InstDocumenter(sphinx.ext.autodoc.Documenter): def format_signature(self): inst = self.object - sig = self.format_name() + sig = inst.name if len(inst.outs) > 0: sig = ', '.join([op.name for op in inst.outs]) + ' = ' + sig if len(inst.ins) > 0: - sig = sig + ' ' + inst.ins[0].name + op = inst.ins[0] + sig += ' ' + op.name + # If the first input is variable-args, this is 'return'. No parens. + if op.typ.operand_kind().name == 'variable_args': + sig += '...'.format(op.name) for op in inst.ins[1:]: + # This is a call or branch with args in (...). if op.typ.operand_kind().name == 'variable_args': sig += '({}...)'.format(op.name) else: diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 9181f10e26..e323dd617d 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -393,16 +393,7 @@ preamble`: :arg args...: Function arguments matching the signature of F. :result a,b,...: Return values matching the signature of F. -.. inst:: return args... - - Return from function. - - Unconditionally transfer control to the calling function, passing the - provided return values. - - :arg args: Return values. The list of return values must match the list of - return value types in the function signature. - :result: None. This is a terminator instruction. +.. autoinst:: x_return This simple example illustrates direct function calls and signatures:: diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/instructions.rs index 9a87e85d71..8ebd63d411 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/instructions.rs @@ -210,6 +210,11 @@ pub enum InstructionData { ty: Type, data: Box, }, + Return { + opcode: Opcode, + ty: Type, + data: Box, + }, } /// A variable list of `Value` operands used for function call arguments and passing arguments to @@ -248,7 +253,6 @@ impl DerefMut for VariableArgs { impl Display for VariableArgs { fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - try!(write!(fmt, "(")); for (i, val) in self.0.iter().enumerate() { if i == 0 { try!(write!(fmt, "{}", val)); @@ -256,7 +260,7 @@ impl Display for VariableArgs { try!(write!(fmt, ", {}", val)); } } - write!(fmt, ")") + Ok(()) } } @@ -279,7 +283,7 @@ impl Display for JumpData { if self.arguments.is_empty() { write!(f, "{}", self.destination) } else { - write!(f, "{}{}", self.destination, self.arguments) + write!(f, "{}({})", self.destination, self.arguments) } } } @@ -297,7 +301,7 @@ impl Display for BranchData { fn fmt(&self, f: &mut Formatter) -> fmt::Result { try!(write!(f, "{}, {}", self.arg, self.destination)); if !self.arguments.is_empty() { - try!(write!(f, "{}", self.arguments)); + try!(write!(f, "({})", self.arguments)); } Ok(()) } @@ -310,15 +314,22 @@ pub struct CallData { second_result: Value, // Dynamically sized array containing call argument values. - pub arguments: VariableArgs, + pub args: VariableArgs, } impl Display for CallData { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - write!(f, "TBD{}", self.arguments) + write!(f, "TBD({})", self.args) } } +/// Payload of a return instruction. +#[derive(Debug)] +pub struct ReturnData { + // Dynamically sized array containing return values. + pub args: VariableArgs, +} + impl InstructionData { /// Create data for a call instruction. pub fn call(opc: Opcode, return_type: Type) -> InstructionData { @@ -327,7 +338,7 @@ impl InstructionData { ty: return_type, data: Box::new(CallData { second_result: NO_VALUE, - arguments: VariableArgs::new(), + args: VariableArgs::new(), }), } } diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index 22d87ba103..e49956970a 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -203,6 +203,13 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { Branch { ref data, .. } => writeln!(w, " {}", data), BranchTable { arg, table, .. } => writeln!(w, " {}, {}", arg, table), Call { ref data, .. } => writeln!(w, " {}", data), + Return { ref data, .. } => { + if data.args.is_empty() { + writeln!(w, "") + } else { + writeln!(w, " {}", data.args) + } + } } } diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index e55a277d18..34dd15f735 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -15,7 +15,7 @@ use cretonne::types::{Type, VOID, FunctionName, Signature, ArgumentType, Argumen use cretonne::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::entities::*; use cretonne::instructions::{Opcode, InstructionFormat, InstructionData, VariableArgs, JumpData, - BranchData}; + BranchData, ReturnData}; use cretonne::repr::{Function, StackSlotData}; pub use lexer::Location; @@ -201,7 +201,11 @@ impl Context { } InstructionData::Call { ref mut data, .. } => { - try!(Self::rewrite_values(&self.values, &mut data.arguments, &loc)); + try!(Self::rewrite_values(&self.values, &mut data.args, &loc)); + } + + InstructionData::Return { ref mut data, .. } => { + try!(Self::rewrite_values(&self.values, &mut data.args, &loc)); } } } @@ -1015,6 +1019,14 @@ impl<'a> Parser<'a> { args: [lhs, rhs], } } + InstructionFormat::Return => { + let args = try!(self.parse_value_list()); + InstructionData::Return { + opcode: opcode, + ty: VOID, + data: Box::new(ReturnData { args: args }), + } + } InstructionFormat::BranchTable | InstructionFormat::Call => { unimplemented!(); diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 894ac2f505..1f17bab749 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -95,6 +95,18 @@ trapnz = Instruction( """, ins=c) +rvals = Operand('rvals', variable_args, doc='return values') + +x_return = Instruction( + 'return', r""" + Return from the function. + + Unconditionally transfer control to the calling function, passing the + provided return values. The list of return values must match the + function signature's return types. + """, + ins=rvals) + # # Materializing constants. # diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index 5fc6cd7a2d..4798b9ed25 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -43,6 +43,8 @@ BranchTable = InstructionFormat(value, jump_table) Call = InstructionFormat( function, variable_args, multiple_results=True, boxed_storage=True) +Return = InstructionFormat(variable_args, boxed_storage=True) + # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) From 8cb198b23c73b9049c930be2ded0602cbb571235 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Jul 2016 16:26:11 -0700 Subject: [PATCH 0121/3084] Delete the concept of 'local SSA form'. This was supposed to make verification fast, but WebAssembly is no longer in this form since it's blocks can produce values. Also, computing a flow graph and dominator tree is really fast anyway. --- cranelift/docs/langref.rst | 18 ------------------ 1 file changed, 18 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index e323dd617d..c6e7b3ced6 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -83,11 +83,6 @@ instructions which behave more like variable accesses in a typical programming language. Cretonne can perform the necessary dataflow analysis to convert stack slots to SSA form. -If all values are only used in the same EBB where they are defined, the -function is said to be in :term:`local SSA form`. It is much faster for -Cretonne to verify the correctness of a function in local SSA form since no -complicated control flow analysis is required. - Value types =========== @@ -890,16 +885,3 @@ Glossary stack slot A fixed size memory allocation in the current function's activation frame. Also called a local variable. - - local SSA form - A restricted version of SSA form where all values are defined and used - in the same EBB. A function is in local SSA form iff it is in SSA form - and: - - - No branches pass arguments to their target EBB. - - Only the entry EBB may have arguments. - - This also implies that there are no branches to the entry EBB. - - Local SSA form is easy to generate and fast to verify. It passes data - between EBBs by using stack slots. From 27b2c90a27e7fef920fef9a2c48bd93f167fd908 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Jul 2016 16:28:19 -0700 Subject: [PATCH 0122/3084] Add tests for parsing call and return. --- cranelift/tests/parser/call.cton | 13 +++++++++++++ cranelift/tests/parser/call.cton.ref | 11 +++++++++++ 2 files changed, 24 insertions(+) create mode 100644 cranelift/tests/parser/call.cton create mode 100644 cranelift/tests/parser/call.cton.ref diff --git a/cranelift/tests/parser/call.cton b/cranelift/tests/parser/call.cton new file mode 100644 index 0000000000..12221c3ff1 --- /dev/null +++ b/cranelift/tests/parser/call.cton @@ -0,0 +1,13 @@ +; Parser tests for call and return syntax. + +function mini() { +ebb1: + return +} + +function r1() -> i32, f32 { +ebb1: + v1 = iconst.i32 3 + v2 = f32const 0.0 + return v1, v2 +} diff --git a/cranelift/tests/parser/call.cton.ref b/cranelift/tests/parser/call.cton.ref new file mode 100644 index 0000000000..a6d51166f4 --- /dev/null +++ b/cranelift/tests/parser/call.cton.ref @@ -0,0 +1,11 @@ +function mini() { +ebb0: + return +} + +function r1() -> i32, f32 { +ebb0: + v0 = iconst.i32 3 + v1 = f32const 0.0 + return v0, v1 +} From 942c4b96c962ba08838d00602c6918904336e6da Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Tue, 12 Jul 2016 13:59:27 -0700 Subject: [PATCH 0123/3084] Ignore cargo-fmt and vim related files --- cranelift/.gitignore | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/cranelift/.gitignore b/cranelift/.gitignore index 0d20b6487c..002a368e2b 100644 --- a/cranelift/.gitignore +++ b/cranelift/.gitignore @@ -1 +1,5 @@ *.pyc +*.bk +*.swp +*.swo +tags From e4a9c5c13cc759014d2b561f20930b9f94608c94 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Tue, 12 Jul 2016 14:37:37 -0700 Subject: [PATCH 0124/3084] Add a Control Flow Graph representation. The CFG must be instantiated against an existing function but may be modified after creation --- cranelift/src/libcretonne/cfg.rs | 270 ++++++++++++++++++++++++++ cranelift/src/libcretonne/entities.rs | 4 +- cranelift/src/libcretonne/lib.rs | 1 + 3 files changed, 273 insertions(+), 2 deletions(-) create mode 100644 cranelift/src/libcretonne/cfg.rs diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs new file mode 100644 index 0000000000..a8640d3459 --- /dev/null +++ b/cranelift/src/libcretonne/cfg.rs @@ -0,0 +1,270 @@ +//! A control flow graph represented as mappings of extended basic blocks to their predecessors. +//! Predecessors are denoted by tuples of EBB and branch/jump instructions. Each predecessor +//! tuple corresponds to the end of a basic block. +//! +//!```c +//! Ebb0: +//! ... ; beginning of basic block +//! +//! ... +//! +//! brz vx, Ebb1 ; end of basic block +//! +//! ... ; beginning of basic block +//! +//! ... +//! +//! jmp Ebb2 ; end of basic block +//!``` +//! +//! Here Ebb1 and Ebb2 would each have a single predecessor denoted as (Ebb0, `brz vx, Ebb1`) +//! and (Ebb0, `jmp Ebb2`) respectively. + +use repr::Function; +use entities::{Inst, Ebb}; +use instructions::InstructionData; +use std::collections::{BTreeSet, BTreeMap, btree_map}; + +/// A basic block denoted by its enclosing Ebb and last instruction. +pub type Predecessor = (Ebb, Inst); + +/// Storing predecessors in a BTreeSet ensures that their ordering is +/// stable with no duplicates. +pub type PredecessorSet = BTreeSet; + +/// The Control Flow Graph maintains a mapping of ebbs to their predecessors +/// where predecessors are basic blocks. +#[derive(Debug)] +pub struct ControlFlowGraph { + data: BTreeMap, +} + +impl ControlFlowGraph { + + /// During initialization mappings will be generated for any existing + /// blocks within the CFG's associated function. Basic sanity checks will + /// also be performed to ensure that the blocks are well formed. + pub fn new(func: &Function) -> Result { + let mut cfg = ControlFlowGraph{data: BTreeMap::new()}; + + // Even ebbs without predecessors should show up in the CFG, albeit + // with no entires. + for ebb in func.ebbs_numerically() { + try!(cfg.init_ebb(ebb)); + } + + for ebb in func.ebbs_numerically() { + // Flips to true when a terminating instruction is seen. So that if additional + // instructions occur an error may be returned. + let mut terminated = false; + for inst in func.ebb_insts(ebb) { + if terminated { + return Err(format!("{} contains unreachable instructions.", ebb)); + } + + match func[inst] { + InstructionData::Branch { ty: _, opcode: _, ref data } => { + try!(cfg.add_predecessor(data.destination, (ebb, inst))); + } + InstructionData::Jump { ty: _, opcode: _, ref data } => { + try!(cfg.add_predecessor(data.destination, (ebb, inst))); + terminated = true; + } + InstructionData::Return { ty: _, opcode: _, data: _ } => { + terminated = true; + } + InstructionData::Nullary { ty: _, opcode: _ } => { + terminated = true; + } + _ => () + } + } + } + Ok(cfg) + } + + /// Initializes a predecessor set for some ebb. If an ebb already has an + /// entry it will be clobbered. + pub fn init_ebb(&mut self, ebb: Ebb) -> Result<&mut PredecessorSet, &'static str> { + self.data.insert(ebb, BTreeSet::new()); + match self.data.get_mut(&ebb) { + Some(predecessors) => Ok(predecessors), + None => Err("Ebb initialization failed.") + } + } + + /// Attempts to add a predecessor for some ebb, attempting to initialize + /// any ebb which has no entry. + pub fn add_predecessor(&mut self, ebb: Ebb, predecessor: Predecessor) -> Result<(), &'static str> { + let success = match self.data.get_mut(&ebb) { + Some(predecessors) => predecessors.insert(predecessor), + None => false + }; + + if success { + Ok(()) + } else { + let mut predecessors = try!(self.init_ebb(ebb)); + if predecessors.insert(predecessor) { + return Ok(()); + } + Err("Predecessor insertion failed.") + } + + } + + /// Returns all of the predecessors for some ebb, if it has an entry. + pub fn get_predecessors(&self, ebb: Ebb) -> Option<&PredecessorSet> { + self.data.get(&ebb) + } + + /// An iterator over all of the ebb to predecessor mappings in the CFG. + pub fn iter<'a>(&'a self) -> btree_map::Iter<'a, Ebb, PredecessorSet> { + self.data.iter() + } +} + +#[cfg(test)] +mod tests { + use instructions::*; + use entities::{Ebb, Inst, NO_VALUE}; + use repr::Function; + use super::*; + use types; + + // Some instructions will be re-used in several tests. + + fn nullary(func: &mut Function) -> Inst { + func.make_inst(InstructionData::Nullary { + opcode: Opcode::Iconst, + ty: types::I32, + }) + } + + fn jump(func: &mut Function, dest: Ebb) -> Inst { + func.make_inst(InstructionData::Jump { + opcode: Opcode::Jump, + ty: types::VOID, + data: Box::new(JumpData { + destination: dest, + arguments: VariableArgs::new(), + }), + }) + } + + fn branch(func: &mut Function, dest: Ebb) -> Inst { + func.make_inst(InstructionData::Branch { + opcode: Opcode::Brz, + ty: types::VOID, + data: Box::new(BranchData { + arg: NO_VALUE, + destination: dest, + arguments: VariableArgs::new(), + }), + }) + } + + #[test] + fn empty() { + let func = Function::new(); + let cfg = ControlFlowGraph::new(&func).unwrap(); + assert_eq!(None, cfg.iter().next()); + } + + #[test] + fn no_predecessors() { + let mut func = Function::new(); + func.make_ebb(); + func.make_ebb(); + func.make_ebb(); + let cfg = ControlFlowGraph::new(&func).unwrap(); + let nodes = cfg.iter().collect::>(); + assert_eq!(nodes.len(), 3); + + let mut fun_ebbs = func.ebbs_numerically(); + for (ebb, predecessors) in nodes { + assert_eq!(ebb.index(), fun_ebbs.next().unwrap().index()); + assert_eq!(predecessors.len(), 0); + } + } + + #[test] + #[should_panic(expected = "instructions")] + fn nullable_before_branch() { + // Ensure that branching after a nullary, within an ebb, triggers an error. + let mut func = Function::new(); + let ebb0 = func.make_ebb(); + let ebb1_malformed = func.make_ebb(); + let ebb2 = func.make_ebb(); + + let nullary_inst = nullary(&mut func); + func.append_inst(ebb1_malformed, nullary_inst); + + // This jump should not be recorded since a nullary takes place + // before it appears. + let jmp_ebb1_ebb0 = jump(&mut func, ebb0); + func.append_inst(ebb1_malformed, jmp_ebb1_ebb0); + + let jmp_ebb0_ebb2 = jump(&mut func, ebb2); + func.append_inst(ebb0, jmp_ebb0_ebb2); + + ControlFlowGraph::new(&func).unwrap(); + } + + #[test] + #[should_panic(expected = "instructions")] + fn jump_before_branch() { + // Ensure that branching after a jump, within an ebb, triggers an error. + let mut func = Function::new(); + let ebb0 = func.make_ebb(); + let ebb1_malformed = func.make_ebb(); + let ebb2 = func.make_ebb(); + + let jmp_ebb0_ebb1 = jump(&mut func, ebb2); + func.append_inst(ebb0, jmp_ebb0_ebb1); + + let jmp_ebb1_ebb2 = jump(&mut func, ebb2); + func.append_inst(ebb1_malformed, jmp_ebb1_ebb2); + + // This branch should not be recorded since a jump takes place + // before it appears. + let br_ebb1_ebb0 = branch(&mut func, ebb0); + func.append_inst(ebb1_malformed, br_ebb1_ebb0); + + ControlFlowGraph::new(&func).unwrap(); + } + + #[test] + fn branches_and_jumps() { + let mut func = Function::new(); + let ebb0 = func.make_ebb(); + let ebb1 = func.make_ebb(); + let ebb2 = func.make_ebb(); + + let br_ebb0_ebb2 = branch(&mut func, ebb2); + func.append_inst(ebb0, br_ebb0_ebb2); + + let jmp_ebb0_ebb1 = jump(&mut func, ebb1); + func.append_inst(ebb0, jmp_ebb0_ebb1); + + let br_ebb1_ebb1 = branch(&mut func, ebb1); + func.append_inst(ebb1, br_ebb1_ebb1); + + let jmp_ebb1_ebb2 = jump(&mut func, ebb2); + func.append_inst(ebb1, jmp_ebb1_ebb2); + + let cfg = ControlFlowGraph::new(&func).unwrap(); + let ebb0_predecessors = cfg.get_predecessors(ebb0).unwrap(); + let ebb1_predecessors = cfg.get_predecessors(ebb1).unwrap(); + let ebb2_predecessors = cfg.get_predecessors(ebb2).unwrap(); + assert_eq!(ebb0_predecessors.len(), 0); + assert_eq!(ebb1_predecessors.len(), 2); + assert_eq!(ebb2_predecessors.len(), 2); + + assert_eq!(ebb1_predecessors.contains(&(ebb0, jmp_ebb0_ebb1)), true); + assert_eq!(ebb1_predecessors.contains(&(ebb1, br_ebb1_ebb1)), true); + assert_eq!(ebb2_predecessors.contains(&(ebb0, br_ebb0_ebb2)), true); + assert_eq!(ebb2_predecessors.contains(&(ebb1, jmp_ebb1_ebb2)), true); + } + +} diff --git a/cranelift/src/libcretonne/entities.rs b/cranelift/src/libcretonne/entities.rs index 6294a8e8c7..c87ed6a818 100644 --- a/cranelift/src/libcretonne/entities.rs +++ b/cranelift/src/libcretonne/entities.rs @@ -24,7 +24,7 @@ use std::fmt::{self, Display, Formatter}; use std::u32; /// An opaque reference to an extended basic block in a function. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] pub struct Ebb(u32); impl Ebb { @@ -64,7 +64,7 @@ impl Default for Ebb { } /// An opaque reference to an instruction in a function. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] pub struct Inst(u32); impl Inst { diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 26fc74f7ab..f0bf55af39 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -14,3 +14,4 @@ pub mod entities; pub mod instructions; pub mod repr; pub mod write; +pub mod cfg; From 522227c965fbecb6fbf05c590f90dbf92aafdb1c Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Tue, 12 Jul 2016 14:42:49 -0700 Subject: [PATCH 0125/3084] Cargo-fmt fixes --- cranelift/src/libcretonne/cfg.rs | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index a8640d3459..cf69d0f184 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -2,7 +2,7 @@ //! Predecessors are denoted by tuples of EBB and branch/jump instructions. Each predecessor //! tuple corresponds to the end of a basic block. //! -//!```c +//! ```c //! Ebb0: //! ... ; beginning of basic block //! @@ -15,7 +15,7 @@ //! ... //! //! jmp Ebb2 ; end of basic block -//!``` +//! ``` //! //! Here Ebb1 and Ebb2 would each have a single predecessor denoted as (Ebb0, `brz vx, Ebb1`) //! and (Ebb0, `jmp Ebb2`) respectively. @@ -40,12 +40,11 @@ pub struct ControlFlowGraph { } impl ControlFlowGraph { - /// During initialization mappings will be generated for any existing /// blocks within the CFG's associated function. Basic sanity checks will /// also be performed to ensure that the blocks are well formed. pub fn new(func: &Function) -> Result { - let mut cfg = ControlFlowGraph{data: BTreeMap::new()}; + let mut cfg = ControlFlowGraph { data: BTreeMap::new() }; // Even ebbs without predecessors should show up in the CFG, albeit // with no entires. @@ -76,7 +75,7 @@ impl ControlFlowGraph { InstructionData::Nullary { ty: _, opcode: _ } => { terminated = true; } - _ => () + _ => (), } } } @@ -89,16 +88,19 @@ impl ControlFlowGraph { self.data.insert(ebb, BTreeSet::new()); match self.data.get_mut(&ebb) { Some(predecessors) => Ok(predecessors), - None => Err("Ebb initialization failed.") + None => Err("Ebb initialization failed."), } } /// Attempts to add a predecessor for some ebb, attempting to initialize /// any ebb which has no entry. - pub fn add_predecessor(&mut self, ebb: Ebb, predecessor: Predecessor) -> Result<(), &'static str> { + pub fn add_predecessor(&mut self, + ebb: Ebb, + predecessor: Predecessor) + -> Result<(), &'static str> { let success = match self.data.get_mut(&ebb) { Some(predecessors) => predecessors.insert(predecessor), - None => false + None => false, }; if success { @@ -120,7 +122,7 @@ impl ControlFlowGraph { /// An iterator over all of the ebb to predecessor mappings in the CFG. pub fn iter<'a>(&'a self) -> btree_map::Iter<'a, Ebb, PredecessorSet> { - self.data.iter() + self.data.iter() } } From 24b421bd6834628eaa1f1701de5af461fb4937a1 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Tue, 12 Jul 2016 14:46:24 -0700 Subject: [PATCH 0126/3084] Remove misleading test comments --- cranelift/src/libcretonne/cfg.rs | 4 ---- 1 file changed, 4 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index cf69d0f184..14f0467fcb 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -202,8 +202,6 @@ mod tests { let nullary_inst = nullary(&mut func); func.append_inst(ebb1_malformed, nullary_inst); - // This jump should not be recorded since a nullary takes place - // before it appears. let jmp_ebb1_ebb0 = jump(&mut func, ebb0); func.append_inst(ebb1_malformed, jmp_ebb1_ebb0); @@ -228,8 +226,6 @@ mod tests { let jmp_ebb1_ebb2 = jump(&mut func, ebb2); func.append_inst(ebb1_malformed, jmp_ebb1_ebb2); - // This branch should not be recorded since a jump takes place - // before it appears. let br_ebb1_ebb0 = branch(&mut func, ebb0); func.append_inst(ebb1_malformed, br_ebb1_ebb0); From 985606b12ce0542b060465a30097d81b043ed855 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Wed, 13 Jul 2016 11:04:39 -0700 Subject: [PATCH 0127/3084] Replace Results with assertions in invariant cases. It seems reasonable that certain non-recoverable errors during the building of the CFG should crash. --- cranelift/src/libcretonne/cfg.rs | 107 ++++--------------------------- 1 file changed, 12 insertions(+), 95 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 14f0467fcb..4fc2f1b292 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -43,76 +43,42 @@ impl ControlFlowGraph { /// During initialization mappings will be generated for any existing /// blocks within the CFG's associated function. Basic sanity checks will /// also be performed to ensure that the blocks are well formed. - pub fn new(func: &Function) -> Result { + pub fn new(func: &Function) -> ControlFlowGraph { let mut cfg = ControlFlowGraph { data: BTreeMap::new() }; // Even ebbs without predecessors should show up in the CFG, albeit // with no entires. for ebb in func.ebbs_numerically() { - try!(cfg.init_ebb(ebb)); + cfg.init_ebb(ebb); } for ebb in func.ebbs_numerically() { // Flips to true when a terminating instruction is seen. So that if additional // instructions occur an error may be returned. - let mut terminated = false; for inst in func.ebb_insts(ebb) { - if terminated { - return Err(format!("{} contains unreachable instructions.", ebb)); - } - match func[inst] { InstructionData::Branch { ty: _, opcode: _, ref data } => { - try!(cfg.add_predecessor(data.destination, (ebb, inst))); + cfg.add_predecessor(data.destination, (ebb, inst)); } InstructionData::Jump { ty: _, opcode: _, ref data } => { - try!(cfg.add_predecessor(data.destination, (ebb, inst))); - terminated = true; - } - InstructionData::Return { ty: _, opcode: _, data: _ } => { - terminated = true; - } - InstructionData::Nullary { ty: _, opcode: _ } => { - terminated = true; + cfg.add_predecessor(data.destination, (ebb, inst)); } _ => (), } } } - Ok(cfg) + cfg } /// Initializes a predecessor set for some ebb. If an ebb already has an /// entry it will be clobbered. - pub fn init_ebb(&mut self, ebb: Ebb) -> Result<&mut PredecessorSet, &'static str> { + pub fn init_ebb(&mut self, ebb: Ebb) -> &mut PredecessorSet { self.data.insert(ebb, BTreeSet::new()); - match self.data.get_mut(&ebb) { - Some(predecessors) => Ok(predecessors), - None => Err("Ebb initialization failed."), - } + self.data.get_mut(&ebb).unwrap() } - /// Attempts to add a predecessor for some ebb, attempting to initialize - /// any ebb which has no entry. - pub fn add_predecessor(&mut self, - ebb: Ebb, - predecessor: Predecessor) - -> Result<(), &'static str> { - let success = match self.data.get_mut(&ebb) { - Some(predecessors) => predecessors.insert(predecessor), - None => false, - }; - - if success { - Ok(()) - } else { - let mut predecessors = try!(self.init_ebb(ebb)); - if predecessors.insert(predecessor) { - return Ok(()); - } - Err("Predecessor insertion failed.") - } - + pub fn add_predecessor(&mut self, ebb: Ebb, predecessor: Predecessor) { + self.data.get_mut(&ebb).unwrap().insert(predecessor); } /// Returns all of the predecessors for some ebb, if it has an entry. @@ -136,13 +102,6 @@ mod tests { // Some instructions will be re-used in several tests. - fn nullary(func: &mut Function) -> Inst { - func.make_inst(InstructionData::Nullary { - opcode: Opcode::Iconst, - ty: types::I32, - }) - } - fn jump(func: &mut Function, dest: Ebb) -> Inst { func.make_inst(InstructionData::Jump { opcode: Opcode::Jump, @@ -169,7 +128,7 @@ mod tests { #[test] fn empty() { let func = Function::new(); - let cfg = ControlFlowGraph::new(&func).unwrap(); + let cfg = ControlFlowGraph::new(&func); assert_eq!(None, cfg.iter().next()); } @@ -179,7 +138,7 @@ mod tests { func.make_ebb(); func.make_ebb(); func.make_ebb(); - let cfg = ControlFlowGraph::new(&func).unwrap(); + let cfg = ControlFlowGraph::new(&func); let nodes = cfg.iter().collect::>(); assert_eq!(nodes.len(), 3); @@ -190,48 +149,6 @@ mod tests { } } - #[test] - #[should_panic(expected = "instructions")] - fn nullable_before_branch() { - // Ensure that branching after a nullary, within an ebb, triggers an error. - let mut func = Function::new(); - let ebb0 = func.make_ebb(); - let ebb1_malformed = func.make_ebb(); - let ebb2 = func.make_ebb(); - - let nullary_inst = nullary(&mut func); - func.append_inst(ebb1_malformed, nullary_inst); - - let jmp_ebb1_ebb0 = jump(&mut func, ebb0); - func.append_inst(ebb1_malformed, jmp_ebb1_ebb0); - - let jmp_ebb0_ebb2 = jump(&mut func, ebb2); - func.append_inst(ebb0, jmp_ebb0_ebb2); - - ControlFlowGraph::new(&func).unwrap(); - } - - #[test] - #[should_panic(expected = "instructions")] - fn jump_before_branch() { - // Ensure that branching after a jump, within an ebb, triggers an error. - let mut func = Function::new(); - let ebb0 = func.make_ebb(); - let ebb1_malformed = func.make_ebb(); - let ebb2 = func.make_ebb(); - - let jmp_ebb0_ebb1 = jump(&mut func, ebb2); - func.append_inst(ebb0, jmp_ebb0_ebb1); - - let jmp_ebb1_ebb2 = jump(&mut func, ebb2); - func.append_inst(ebb1_malformed, jmp_ebb1_ebb2); - - let br_ebb1_ebb0 = branch(&mut func, ebb0); - func.append_inst(ebb1_malformed, br_ebb1_ebb0); - - ControlFlowGraph::new(&func).unwrap(); - } - #[test] fn branches_and_jumps() { let mut func = Function::new(); @@ -251,7 +168,7 @@ mod tests { let jmp_ebb1_ebb2 = jump(&mut func, ebb2); func.append_inst(ebb1, jmp_ebb1_ebb2); - let cfg = ControlFlowGraph::new(&func).unwrap(); + let cfg = ControlFlowGraph::new(&func); let ebb0_predecessors = cfg.get_predecessors(ebb0).unwrap(); let ebb1_predecessors = cfg.get_predecessors(ebb1).unwrap(); let ebb2_predecessors = cfg.get_predecessors(ebb2).unwrap(); From 53ed7f5ea39a829d5b3fa8169d4e51d5ca81250f Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Wed, 13 Jul 2016 14:31:05 -0700 Subject: [PATCH 0128/3084] Add a print-cfg subcommand. The command returns parses a .cton file, builds a CFG, and prints it to stdout in graphviz format. --- cranelift/src/tools/main.rs | 5 + cranelift/src/tools/print_cfg.rs | 169 +++++++++++++++++++++++++++++++ 2 files changed, 174 insertions(+) create mode 100644 cranelift/src/tools/print_cfg.rs diff --git a/cranelift/src/tools/main.rs b/cranelift/src/tools/main.rs index 1ea40c201c..abad701af6 100644 --- a/cranelift/src/tools/main.rs +++ b/cranelift/src/tools/main.rs @@ -11,12 +11,14 @@ use std::process; mod cat; +mod print_cfg; const USAGE: &'static str = " Cretonne code generator utility Usage: cton-util cat ... + cton-util print-cfg ... cton-util --help | --version Options: @@ -28,6 +30,7 @@ Options: #[derive(RustcDecodable, Debug)] struct Args { cmd_cat: bool, + cmd_print_cfg: bool, arg_file: Vec, } @@ -48,6 +51,8 @@ fn cton_util() -> CommandResult { // Find the sub-command to execute. if args.cmd_cat { cat::run(args.arg_file) + } else if args.cmd_print_cfg { + print_cfg::run(args.arg_file) } else { // Debugging / shouldn't happen with proper command line handling above. Err(format!("Unhandled args: {:?}", args)) diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs new file mode 100644 index 0000000000..e55033832a --- /dev/null +++ b/cranelift/src/tools/print_cfg.rs @@ -0,0 +1,169 @@ +//! The `print-cfg` sub-command. +//! +//! Read a series of Cretonne IL files and print their control flow graphs +//! in graphviz format. +use std::fs::File; +use std::io::{Read, Write, stdout}; + +use CommandResult; +use cretonne::repr::Function; +use cretonne::cfg::ControlFlowGraph; +use cretonne::instructions::InstructionData; +use cton_reader::parser::Parser; + +pub fn run(files: Vec) -> CommandResult { + for (i, f) in files.into_iter().enumerate() { + if i != 0 { + println!(""); + } + try!(print_cfg(f)) + } + Ok(()) +} + + +struct CFGPrinter { + level: usize, + writer: T, + buffer: String, +} + +impl CFGPrinter { + pub fn new(writer: T) -> CFGPrinter { + CFGPrinter{level: 0, writer: writer, buffer: String::new()} + } + + pub fn print(&mut self, func: &Function) -> Result<(), String> { + self.level = 0; + self.header(); + self.push_indent(); + self.ebb_subgraphs(func); + let cfg = ControlFlowGraph::new(&func); + self.cfg_connections(&cfg); + self.pop_indent(); + self.footer(); + self.write() + } + + fn write(&mut self) -> Result<(), String> { + match self.writer.write(self.buffer.as_bytes()) { + Err(_) => return Err("Write failed!".to_string()), + _ => (), + }; + match self.writer.flush() { + Err(_) => return Err("Flush failed!".to_string()), + _ => (), + }; + Ok(()) + } + + fn append(&mut self, s: &str) { + let mut indent = String::new(); + for _ in 0 .. self.level { + indent = indent + " "; + } + self.buffer.push_str(&(indent + s)); + } + + fn push_indent(&mut self) { + self.level += 1; + } + + fn pop_indent(&mut self) { + if self.level > 0 { + self.level -= 1; + } + } + + fn open_paren(&mut self) { + self.append("{"); + } + + fn close_paren(&mut self) { + self.append("}"); + } + + fn newline(&mut self) { + self.append("\n"); + } + + fn header(&mut self) { + self.append("digraph "); + self.open_paren(); + self.newline(); + self.push_indent(); + self.append("{rank=min; ebb0}"); + self.pop_indent(); + self.newline(); + } + + fn footer(&mut self) { + self.close_paren(); + self.newline(); + } + + fn ebb_subgraphs(&mut self, func: &Function) { + for ebb in func.ebbs_numerically() { + let inst_data = func.ebb_insts(ebb) + .filter(|inst| { + match func[*inst] { + InstructionData::Branch{ ty: _, opcode: _, data: _ } => true, + InstructionData::Jump{ ty: _, opcode: _, data: _ } => true, + _ => false + } + }) + .map(|inst| { + let op = match func[inst] { + InstructionData::Branch{ ty: _, opcode, ref data } => { + Some((opcode, data.destination)) + }, + InstructionData::Jump{ ty: _, opcode, ref data } => { + Some((opcode, data.destination)) + }, + _ => None + }; + (inst, op) + }) + .collect::>(); + + let mut insts = vec![format!("{}", ebb)]; + for (inst, data) in inst_data { + let (op, dest) = data.unwrap(); + insts.push(format!("<{}>{} {}", inst, op, dest)); + } + + self.append(&format!("{} [shape=record, label=\"{}{}{}\"]", + ebb, "{", insts.join(" | "), "}")); + self.newline(); + } + } + + fn cfg_connections(&mut self, cfg: &ControlFlowGraph) { + for (ref ebb, ref predecessors) in cfg.iter() { + for &(parent, inst) in *predecessors { + self.append(&format!("{}:{} -> {}", parent, inst, ebb)); + self.newline(); + } + } + } + +} + +fn print_cfg(filename: String) -> CommandResult { + let mut file = try!(File::open(&filename).map_err(|e| format!("{}: {}", filename, e))); + let mut buffer = String::new(); + try!(file.read_to_string(&mut buffer) + .map_err(|e| format!("Couldn't read {}: {}", filename, e))); + let items = try!(Parser::parse(&buffer).map_err(|e| format!("{}: {}", filename, e))); + + let mut cfg_printer = CFGPrinter::new(stdout()); + for (idx, func) in items.into_iter().enumerate() { + if idx != 0 { + println!(""); + } + + try!(cfg_printer.print(&func)); + } + + Ok(()) +} From 4e74d85056bd7df4478cbc5d1e7d2f4a6bee3b57 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Thu, 14 Jul 2016 12:22:08 -0700 Subject: [PATCH 0129/3084] Id CFG graphs by function name --- cranelift/src/tools/print_cfg.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs index e55033832a..fc85cbebec 100644 --- a/cranelift/src/tools/print_cfg.rs +++ b/cranelift/src/tools/print_cfg.rs @@ -35,7 +35,7 @@ impl CFGPrinter { pub fn print(&mut self, func: &Function) -> Result<(), String> { self.level = 0; - self.header(); + self.header(func); self.push_indent(); self.ebb_subgraphs(func); let cfg = ControlFlowGraph::new(&func); @@ -87,8 +87,8 @@ impl CFGPrinter { self.append("\n"); } - fn header(&mut self) { - self.append("digraph "); + fn header(&mut self, func: &Function) { + self.append(&format!("digraph {} ", func.name)); self.open_paren(); self.newline(); self.push_indent(); From 41d4cdba46855bd11c441c55dad59173bc5bd75c Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Thu, 14 Jul 2016 13:43:11 -0700 Subject: [PATCH 0130/3084] Add print-cfg tests --- cranelift/test-all.sh | 3 ++- cranelift/tests/cfg/README.rst | 14 ++++++++++ cranelift/tests/cfg/loop.cton | 30 ++++++++++++++++++++++ cranelift/tests/cfg/run.sh | 38 ++++++++++++++++++++++++++++ cranelift/tests/cfg/traps_early.cton | 18 +++++++++++++ cranelift/tests/cfg/unused_node.cton | 16 ++++++++++++ 6 files changed, 118 insertions(+), 1 deletion(-) create mode 100644 cranelift/tests/cfg/README.rst create mode 100644 cranelift/tests/cfg/loop.cton create mode 100755 cranelift/tests/cfg/run.sh create mode 100644 cranelift/tests/cfg/traps_early.cton create mode 100644 cranelift/tests/cfg/unused_node.cton diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 502966e55d..31638a6db3 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -7,7 +7,7 @@ # - Make a debug build of all crates. # - Make a release build of cton-util. # - Run file-level tests with the release build of cton-util. -# +# # All tests run by this script should be passing at all times. # Exit immediately on errors. @@ -35,5 +35,6 @@ export CTONUTIL="$topdir/src/tools/target/release/cton-util" echo ====== Parser tests ====== cd "$topdir/tests" parser/run.sh +cfg/run.sh echo ====== OK ====== diff --git a/cranelift/tests/cfg/README.rst b/cranelift/tests/cfg/README.rst new file mode 100644 index 0000000000..c1ee4b1fa3 --- /dev/null +++ b/cranelift/tests/cfg/README.rst @@ -0,0 +1,14 @@ +CFG tests +============ + +This directory contains test cases for the Cretonne cfg printer. + +Each test case consists of a `foo.cton` input file annotated with its expected connections. +Annotations are comments of the form: `ebbx:insty -> ebbz` where ebbx is connected to ebbz via +a branch or jump instruction at line y. Instructions are labeled by line number starting from zero: `inst0` .. `instn`. + + +Each input file is run through the `cton-util print-cfg` command and the +output is compared against the specially formatted comments to ensure that +expected connections exist. This scheme allows for changes to graph style +without the need to update tests. diff --git a/cranelift/tests/cfg/loop.cton b/cranelift/tests/cfg/loop.cton new file mode 100644 index 0000000000..73ad1c4b6a --- /dev/null +++ b/cranelift/tests/cfg/loop.cton @@ -0,0 +1,30 @@ +; For testing cfg generation. This code is nonsense. + +function nonsense(i32, i32) -> f32 { + +ebb0(v1: i32, v2: i32): + v3 = f64const 0x0.0 + brz v2, ebb2 ;;;; ebb0:inst1 -> ebb2 + v4 = iconst.i32 0 + jump ebb1(v4) ;;;; ebb0:inst3 -> ebb1 + +ebb1(v5: i32): + v6 = imul_imm v5, 4 + v7 = iadd v1, v6 + v8 = f32const 0.0 + v9 = f32const 0.0 + v10 = f32const 0.0 + v11 = fadd v9, v10 + v12 = iadd_imm v5, 1 + v13 = icmp ult, v12, v2 + brnz v13, ebb1(v12) ;;;; ebb1:inst12 -> ebb1 + v14 = f64const 0.0 + v15 = f64const 0.0 + v16 = fdiv v14, v15 + v17 = f32const 0.0 + return v17 + +ebb2: + v100 = f32const 0.0 + return v100 +} diff --git a/cranelift/tests/cfg/run.sh b/cranelift/tests/cfg/run.sh new file mode 100755 index 0000000000..2230d4c1ca --- /dev/null +++ b/cranelift/tests/cfg/run.sh @@ -0,0 +1,38 @@ +#!/bin/bash + +# Go to tests directory. +cd $(dirname "$0")/.. + +# The path to cton-util should be in $CTONUTIL. +if [ -z "$CTONUTIL" ]; then + CTONUTIL=../src/tools/target/debug/cton-util +fi + +if [ ! -x "$CTONUTIL" ]; then + echo "Can't fund executable cton-util: $CTONUTIL" 1>&2 + exit 1 +fi + +declare -a fails + +for testcase in $(find cfg -name '*.cton'); do + annotations=$(cat $testcase | awk /';;;;'/ | awk -F ";;;;" '{print $2}' | sort) + connections=$("${CTONUTIL}" print-cfg "$testcase" | awk /"->"/ | sort) + if diff -u <(echo $annotations) <(echo $connections); then + echo OK $testcase + else + fails=(${fails[@]} "$testcase") + echo FAIL $testcase + fi +done + +if [ ${#fails[@]} -ne 0 ]; then + echo + echo Failures: + for f in "${fails[@]}"; do + echo " $f" + done + exit 1 +else + echo "All passed" +fi diff --git a/cranelift/tests/cfg/traps_early.cton b/cranelift/tests/cfg/traps_early.cton new file mode 100644 index 0000000000..6993cc128e --- /dev/null +++ b/cranelift/tests/cfg/traps_early.cton @@ -0,0 +1,18 @@ +; For testing cfg generation. This code explores the implications of encountering +; a terminating instruction before any connections have been made. + +function nonsense(i32) { + +ebb0(v1: i32): + trap + brnz v1, ebb2 ;;;; ebb0:inst1 -> ebb2 + jump ebb1 ;;;; ebb0:inst2 -> ebb1 + +ebb1: + v2 = iconst.i32 0 + v3 = iadd v1, v3 + jump ebb0(v3) ;;;; ebb1:inst5 -> ebb0 + +ebb2: + return v1 +} diff --git a/cranelift/tests/cfg/unused_node.cton b/cranelift/tests/cfg/unused_node.cton new file mode 100644 index 0000000000..3120ffbe68 --- /dev/null +++ b/cranelift/tests/cfg/unused_node.cton @@ -0,0 +1,16 @@ +; For testing cfg generation where some block is never reached. + +function not_reached(i32) -> i32 { + +ebb0(v0: i32): + brnz v0, ebb2 ;;;; ebb0:inst0 -> ebb2 + trap + +ebb1: + v1 = iconst.i32 1 + v2 = iadd v0, v1 + jump ebb0(v2) ;;;; ebb1:inst4 -> ebb0 + +ebb2: + return v0 +} From e87439341907e29b883f378beb60b94c0ac95718 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 15 Jul 2016 15:14:16 -0700 Subject: [PATCH 0131/3084] Add an entity_map module. This supports the pattern of creating structs wrapping a u32 and using them as indexes into a vector of entities. These entity references should implement the EntityRef trait. The EntityMap is a generic map from an EntityRef to some value type. It expects densely indexed entities and uses a Vec to represent the mapping compactly. --- cranelift/src/libcretonne/entity_map.rs | 130 ++++++++++++++++++++++++ cranelift/src/libcretonne/lib.rs | 2 + 2 files changed, 132 insertions(+) create mode 100644 cranelift/src/libcretonne/entity_map.rs diff --git a/cranelift/src/libcretonne/entity_map.rs b/cranelift/src/libcretonne/entity_map.rs new file mode 100644 index 0000000000..3e1f16a595 --- /dev/null +++ b/cranelift/src/libcretonne/entity_map.rs @@ -0,0 +1,130 @@ +//! Densely numbered entity references as mapping keys. +//! +//! This module defines an `EntityRef` trait that should be implemented by reference types wrapping +//! a small integer index. The `EntityMap` data structure uses the dense index space to implement a +//! map with a vector. + +use std::vec::Vec; +use std::default::Default; +use std::marker::PhantomData; +use std::ops::{Index, IndexMut}; + +/// A type wrapping a small integer index should implement `EntityRef` so it can be used as the key +/// of an `EntityMap`. +pub trait EntityRef: Copy { + /// Create a new entity reference from a small integer. + /// This should crash if the requested index is not representable. + fn new(usize) -> Self; + + /// Get the index that was used to create this entity reference. + fn index(self) -> usize; +} + +/// A mapping `K -> V` for densely indexed entity references. +pub struct EntityMap + where K: EntityRef, + V: Clone + Default +{ + elems: Vec, + unused: PhantomData, +} + +impl EntityMap + where K: EntityRef, + V: Clone + Default +{ + /// Create a new empty map. + pub fn new() -> Self { + EntityMap { + elems: Vec::new(), + unused: PhantomData, + } + } + + /// Ensure that `k` is a valid key but adding default entries if necesssary. + pub fn ensure(&mut self, k: K) { + let idx = k.index(); + if idx >= self.elems.len() { + self.elems.resize(idx + 1, V::default()) + } + } + + /// Append `v` to the mapping, assigning a new key which is returned. + pub fn push(&mut self, v: V) -> K { + let k = K::new(self.elems.len()); + self.elems.push(v); + k + } +} + +/// Immutable indexing into an `EntityMap`. +/// The indexed value must have been accessed mutably previously, or the key passed to `ensure()`. +impl Index for EntityMap + where K: EntityRef, + V: Clone + Default +{ + type Output = V; + + fn index(&self, k: K) -> &V { + &self.elems[k.index()] + } +} + +/// Mutable indexing into an `EntityMap`. +/// The map is resized automatically if the key has not been used before. +impl IndexMut for EntityMap + where K: EntityRef, + V: Clone + Default +{ + fn index_mut(&mut self, k: K) -> &mut V { + self.ensure(k); + &mut self.elems[k.index()] + } +} + +#[cfg(test)] +mod tests { + use super::*; + + // EntityRef impl for testing. + #[derive(Clone, Copy)] + struct E(u32); + + impl EntityRef for E { + fn new(i: usize) -> Self { + E(i as u32) + } + fn index(self) -> usize { + self.0 as usize + } + } + + #[test] + fn basic() { + let r0 = E(0); + let r1 = E(1); + let r2 = E(2); + let mut m = EntityMap::new(); + + m[r2] = 3; + m[r1] = 5; + + assert_eq!(m[r1], 5); + assert_eq!(m[r2], 3); + + let shared = &m; + assert_eq!(shared[r0], 0); + assert_eq!(shared[r1], 5); + assert_eq!(shared[r2], 3); + } + + #[test] + fn push() { + let mut m = EntityMap::new(); + let k1: E = m.push(12); + let k2 = m.push(33); + + assert_eq!(m[k1], 12); + assert_eq!(m[k2], 33); + } +} diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index f0bf55af39..55d601d38b 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -15,3 +15,5 @@ pub mod instructions; pub mod repr; pub mod write; pub mod cfg; + +pub mod entity_map; From 4358a4d96ed70d2ac6266bab2385c5cc5b1ea166 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 15 Jul 2016 15:31:13 -0700 Subject: [PATCH 0132/3084] Implement EntityRef for most of the entities module. The only exception is Value which has two dimensions. --- cranelift/src/libcretonne/cfg.rs | 1 + cranelift/src/libcretonne/entities.rs | 34 ++++++++++++++----------- cranelift/src/libcretonne/entity_map.rs | 12 ++++++--- cranelift/src/libcretonne/repr.rs | 1 + 4 files changed, 30 insertions(+), 18 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 4fc2f1b292..8c423c23ac 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -95,6 +95,7 @@ impl ControlFlowGraph { #[cfg(test)] mod tests { use instructions::*; + use entity_map::EntityRef; use entities::{Ebb, Inst, NO_VALUE}; use repr::Function; use super::*; diff --git a/cranelift/src/libcretonne/entities.rs b/cranelift/src/libcretonne/entities.rs index c87ed6a818..33b41e5273 100644 --- a/cranelift/src/libcretonne/entities.rs +++ b/cranelift/src/libcretonne/entities.rs @@ -19,6 +19,7 @@ //! The entity references all implement the `Display` trait in a way that matches the textual IL //! format. +use entity_map::EntityRef; use std::default::Default; use std::fmt::{self, Display, Formatter}; use std::u32; @@ -27,12 +28,18 @@ use std::u32; #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] pub struct Ebb(u32); -impl Ebb { - pub fn new(index: usize) -> Ebb { +impl EntityRef for Ebb { + fn new(index: usize) -> Self { assert!(index < (u32::MAX as usize)); Ebb(index as u32) } + fn index(self) -> usize { + self.0 as usize + } +} + +impl Ebb { /// Create a new EBB reference from its number. This corresponds to the ebbNN representation. pub fn with_number(n: u32) -> Option { if n < u32::MAX { @@ -41,10 +48,6 @@ impl Ebb { None } } - - pub fn index(&self) -> usize { - self.0 as usize - } } /// Display an `Ebb` reference as "ebb12". @@ -67,13 +70,13 @@ impl Default for Ebb { #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] pub struct Inst(u32); -impl Inst { - pub fn new(index: usize) -> Inst { +impl EntityRef for Inst { + fn new(index: usize) -> Self { assert!(index < (u32::MAX as usize)); Inst(index as u32) } - pub fn index(&self) -> usize { + fn index(self) -> usize { self.0 as usize } } @@ -188,13 +191,13 @@ impl Default for Value { #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct StackSlot(u32); -impl StackSlot { - pub fn new(index: usize) -> StackSlot { +impl EntityRef for StackSlot { + fn new(index: usize) -> StackSlot { assert!(index < (u32::MAX as usize)); StackSlot(index as u32) } - pub fn index(&self) -> usize { + fn index(self) -> usize { self.0 as usize } } @@ -219,13 +222,13 @@ impl Default for StackSlot { #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct JumpTable(u32); -impl JumpTable { - pub fn new(index: usize) -> JumpTable { +impl EntityRef for JumpTable { + fn new(index: usize) -> JumpTable { assert!(index < (u32::MAX as usize)); JumpTable(index as u32) } - pub fn index(&self) -> usize { + fn index(self) -> usize { self.0 as usize } } @@ -250,6 +253,7 @@ impl Default for JumpTable { mod tests { use super::*; use std::u32; + use entity_map::EntityRef; #[test] fn value_with_number() { diff --git a/cranelift/src/libcretonne/entity_map.rs b/cranelift/src/libcretonne/entity_map.rs index 3e1f16a595..fbf8109d9f 100644 --- a/cranelift/src/libcretonne/entity_map.rs +++ b/cranelift/src/libcretonne/entity_map.rs @@ -41,11 +41,15 @@ impl EntityMap } } + /// Check if `k` is a valid key in the map. + pub fn is_valid(&self, k: K) -> bool { + k.index() < self.elems.len() + } + /// Ensure that `k` is a valid key but adding default entries if necesssary. pub fn ensure(&mut self, k: K) { - let idx = k.index(); - if idx >= self.elems.len() { - self.elems.resize(idx + 1, V::default()) + if !self.is_valid(k) { + self.elems.resize(k.index() + 1, V::default()) } } @@ -106,7 +110,9 @@ mod tests { let r2 = E(2); let mut m = EntityMap::new(); + assert!(!m.is_valid(r0)); m[r2] = 3; + assert!(m.is_valid(r1)); m[r1] = 5; assert_eq!(m[r1], 5); diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index 5a36115fbd..27de00ea66 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -1,6 +1,7 @@ //! Representation of Cretonne IL functions. use types::{Type, FunctionName, Signature, VOID}; +use entity_map::EntityRef; use entities::{Ebb, NO_EBB, Inst, NO_INST, Value, NO_VALUE, ExpandedValue, StackSlot}; use instructions::*; use std::fmt::{self, Display, Formatter}; From d3ed162bac9980e1e0895a0d8586d6526355c481 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 15 Jul 2016 11:33:10 -0700 Subject: [PATCH 0133/3084] Begin a layout module. The Layout data structure will keep track of the order of EBBs and their instructions. WIP. --- cranelift/src/libcretonne/layout.rs | 142 ++++++++++++++++++++++++++++ cranelift/src/libcretonne/lib.rs | 1 + 2 files changed, 143 insertions(+) create mode 100644 cranelift/src/libcretonne/layout.rs diff --git a/cranelift/src/libcretonne/layout.rs b/cranelift/src/libcretonne/layout.rs new file mode 100644 index 0000000000..a870e56167 --- /dev/null +++ b/cranelift/src/libcretonne/layout.rs @@ -0,0 +1,142 @@ +//! Function layout. +//! +//! The order of extended basic blocks in a function and the order of instructions in an EBB is +//! determined by the `Layout` data structure defined in this module. + +use entity_map::EntityMap; +use entities::{Ebb, NO_EBB, Inst, NO_INST}; + +/// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not +/// contain definitions of instructions or EBBs, but depends on `Inst` and `Ebb` entity references +/// being defined elsewhere. +/// +/// This data structure determines: +/// +/// - The order of EBBs in the function. +/// - Which EBB contains a given instruction. +/// - The order of instructions with an EBB. +/// +/// While data dependencies are not recorded, instruction ordering does affect control +/// dependencies, so part of the semantics of the program are determined by the layout. +/// +pub struct Layout { + // Linked list nodes for the layout order of EBBs Forms a doubly linked list, terminated in + // both ends by NO_EBB. + ebbs: EntityMap, + + // Linked list nodes for the layout order of instructions. Forms a double linked list per EBB, + // terminated in both ends by NO_INST. + insts: EntityMap, + + // First EBB in the layout order, or `NO_EBB` when no EBBs have been laid out. + first_ebb: Ebb, + + // Last EBB in the layout order, or `NO_EBB` when no EBBs have been laid out. + last_ebb: Ebb, +} + +impl Layout { + /// Create a new empty `Layout`. + pub fn new() -> Layout { + Layout { + ebbs: EntityMap::new(), + insts: EntityMap::new(), + first_ebb: NO_EBB, + last_ebb: NO_EBB, + } + } +} + +/// Methods for laying out EBBs. +/// +/// An unknown EBB starts out as *not inserted* in the EBB layout. The layout is a linear order of +/// inserted EBBs. Once an EBB has been inserted in the layout, instructions can be added. An EBB +/// can only be removed from the layout when it is empty. +/// +/// Since every EBB must end with a terminator instruction which cannot fall through, the layout of +/// EBBs does not affect the semantics of the program. +/// +impl Layout { + /// Is `ebb` currently part of the layout? + pub fn is_ebb_inserted(&self, ebb: Ebb) -> bool { + ebb != self.first_ebb && self.ebbs.is_valid(ebb) && self.ebbs[ebb].prev == NO_EBB + } + + /// Insert `ebb` as the last EBB in the layout. + pub fn append_ebb(&mut self, ebb: Ebb) { + assert!(!self.is_ebb_inserted(ebb), + "Cannot append EBB that is already in the layout"); + let node = &mut self.ebbs[ebb]; + assert!(node.first_inst == NO_INST && node.last_inst == NO_INST); + node.prev = self.last_ebb; + node.next = NO_EBB; + self.last_ebb = ebb; + } + + /// Insert `ebb` in the layout before the existing EBB `before`. + pub fn insert_ebb(&mut self, ebb: Ebb, before: Ebb) { + assert!(!self.is_ebb_inserted(ebb), + "Cannot insert EBB that is already in the layout"); + assert!(self.is_ebb_inserted(before), + "EBB Insertion point not in the layout"); + let after = self.ebbs[before].prev; + self.ebbs[ebb].next = before; + self.ebbs[ebb].prev = after; + self.ebbs[before].prev = ebb; + if after != NO_EBB { + self.ebbs[after].next = ebb; + } + } +} + +#[derive(Clone, Debug, Default)] +struct EbbNode { + prev: Ebb, + next: Ebb, + first_inst: Inst, + last_inst: Inst, +} + +/// Methods for arranging instructions. +/// +/// An instruction starts out as *not inserted* in the layout. An instruction can be inserted into +/// an EBB at a given position. +impl Layout { + /// Get the EBB containing `inst`, or `None` if `inst` is not inserted in the layout. + pub fn inst_ebb(&self, inst: Inst) -> Option { + if self.insts.is_valid(inst) { + let ebb = self.insts[inst].ebb; + if ebb == NO_EBB { + None + } else { + Some(ebb) + } + } else { + None + } + } + + /// Append `inst` to the end of `ebb`. + pub fn append_inst(&self, inst: Inst, ebb: Ebb) { + assert_eq!(self.inst_ebb(inst), None); + assert!(self.is_ebb_inserted(ebb), + "Cannot append instructions to EBB not in layout"); + unimplemented!(); + } + + /// Insert `inst` before the instruction `before` in the same EBB. + pub fn insert_inst(&self, inst: Inst, before: Inst) { + assert_eq!(self.inst_ebb(inst), None); + let ebb = self.inst_ebb(before) + .expect("Instruction before insertion point not in the layout"); + assert!(ebb != NO_EBB); + unimplemented!(); + } +} + +#[derive(Clone, Debug, Default)] +struct InstNode { + ebb: Ebb, + prev: Inst, + next: Inst, +} diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 55d601d38b..100d3121e2 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -13,6 +13,7 @@ pub mod immediates; pub mod entities; pub mod instructions; pub mod repr; +pub mod layout; pub mod write; pub mod cfg; From 32f40168b585ad61902757781c2902adec78c7e4 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Sat, 16 Jul 2016 15:33:30 -0700 Subject: [PATCH 0134/3084] Remove extra newline --- cranelift/src/tools/print_cfg.rs | 1 - 1 file changed, 1 deletion(-) diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs index fc85cbebec..002c206e07 100644 --- a/cranelift/src/tools/print_cfg.rs +++ b/cranelift/src/tools/print_cfg.rs @@ -21,7 +21,6 @@ pub fn run(files: Vec) -> CommandResult { Ok(()) } - struct CFGPrinter { level: usize, writer: T, From c76ba3a7f666527b6b9aa107b4e538988d1e7503 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 18 Jul 2016 14:25:28 -0700 Subject: [PATCH 0135/3084] Add an EntityRef::wrap() method. This is the opposite of unwrap(). It converts the ad-hoc null references like NO_EBB and NO_INST into the more standard Option type which unfortunately takes twice as much space in data structures. --- cranelift/src/libcretonne/entity_map.rs | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/cranelift/src/libcretonne/entity_map.rs b/cranelift/src/libcretonne/entity_map.rs index fbf8109d9f..2aa101f5c0 100644 --- a/cranelift/src/libcretonne/entity_map.rs +++ b/cranelift/src/libcretonne/entity_map.rs @@ -11,13 +11,31 @@ use std::ops::{Index, IndexMut}; /// A type wrapping a small integer index should implement `EntityRef` so it can be used as the key /// of an `EntityMap`. -pub trait EntityRef: Copy { +pub trait EntityRef: Copy + Eq { /// Create a new entity reference from a small integer. /// This should crash if the requested index is not representable. fn new(usize) -> Self; /// Get the index that was used to create this entity reference. fn index(self) -> usize; + + /// Convert an `EntityRef` to an `Optional` by using the default value as the null + /// reference. + /// + /// Entity references are often used in compact data structures like linked lists where a + /// sentinel 'null' value is needed. Normally we would use an `Optional` for that, but + /// currently that uses twice the memory of a plain `EntityRef`. + /// + /// This method is called `wrap()` because it is the inverse of `unwrap()`. + fn wrap(self) -> Option + where Self: Default + { + if self == Self::default() { + None + } else { + Some(self) + } + } } /// A mapping `K -> V` for densely indexed entity references. @@ -91,7 +109,7 @@ mod tests { use super::*; // EntityRef impl for testing. - #[derive(Clone, Copy)] + #[derive(Clone, Copy, PartialEq, Eq)] struct E(u32); impl EntityRef for E { From ea3d2f0bbdadfd443015de510ad7f696fa359089 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Mon, 18 Jul 2016 14:28:00 -0700 Subject: [PATCH 0136/3084] Move test utility functions to their own module --- cranelift/src/libcretonne/cfg.rs | 41 ++++--------------- cranelift/src/libcretonne/lib.rs | 2 + .../src/libcretonne/test_utils/make_inst.rs | 35 ++++++++++++++++ cranelift/src/libcretonne/test_utils/mod.rs | 3 ++ 4 files changed, 47 insertions(+), 34 deletions(-) create mode 100644 cranelift/src/libcretonne/test_utils/make_inst.rs create mode 100644 cranelift/src/libcretonne/test_utils/mod.rs diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 8c423c23ac..668f5b51ef 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -94,37 +94,10 @@ impl ControlFlowGraph { #[cfg(test)] mod tests { - use instructions::*; - use entity_map::EntityRef; - use entities::{Ebb, Inst, NO_VALUE}; - use repr::Function; use super::*; - use types; + use repr::Function; - // Some instructions will be re-used in several tests. - - fn jump(func: &mut Function, dest: Ebb) -> Inst { - func.make_inst(InstructionData::Jump { - opcode: Opcode::Jump, - ty: types::VOID, - data: Box::new(JumpData { - destination: dest, - arguments: VariableArgs::new(), - }), - }) - } - - fn branch(func: &mut Function, dest: Ebb) -> Inst { - func.make_inst(InstructionData::Branch { - opcode: Opcode::Brz, - ty: types::VOID, - data: Box::new(BranchData { - arg: NO_VALUE, - destination: dest, - arguments: VariableArgs::new(), - }), - }) - } + use test_utils::make_inst; #[test] fn empty() { @@ -145,7 +118,7 @@ mod tests { let mut fun_ebbs = func.ebbs_numerically(); for (ebb, predecessors) in nodes { - assert_eq!(ebb.index(), fun_ebbs.next().unwrap().index()); + assert_eq!(*ebb, fun_ebbs.next().unwrap()); assert_eq!(predecessors.len(), 0); } } @@ -157,16 +130,16 @@ mod tests { let ebb1 = func.make_ebb(); let ebb2 = func.make_ebb(); - let br_ebb0_ebb2 = branch(&mut func, ebb2); + let br_ebb0_ebb2 = make_inst::branch(&mut func, ebb2); func.append_inst(ebb0, br_ebb0_ebb2); - let jmp_ebb0_ebb1 = jump(&mut func, ebb1); + let jmp_ebb0_ebb1 = make_inst::jump(&mut func, ebb1); func.append_inst(ebb0, jmp_ebb0_ebb1); - let br_ebb1_ebb1 = branch(&mut func, ebb1); + let br_ebb1_ebb1 = make_inst::branch(&mut func, ebb1); func.append_inst(ebb1, br_ebb1_ebb1); - let jmp_ebb1_ebb2 = jump(&mut func, ebb2); + let jmp_ebb1_ebb2 = make_inst::jump(&mut func, ebb2); func.append_inst(ebb1, jmp_ebb1_ebb2); let cfg = ControlFlowGraph::new(&func); diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 100d3121e2..d5c501477c 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -18,3 +18,5 @@ pub mod write; pub mod cfg; pub mod entity_map; + +#[cfg(test)] pub mod test_utils; diff --git a/cranelift/src/libcretonne/test_utils/make_inst.rs b/cranelift/src/libcretonne/test_utils/make_inst.rs new file mode 100644 index 0000000000..167512b8b5 --- /dev/null +++ b/cranelift/src/libcretonne/test_utils/make_inst.rs @@ -0,0 +1,35 @@ +///! Helper functions for generating dummy instructions. + +use repr::Function; +use entities::{Ebb, Inst, NO_VALUE}; +use instructions::{ + InstructionData, + Opcode, + VariableArgs, + JumpData, + BranchData, +}; +use types; + +pub fn jump(func: &mut Function, dest: Ebb) -> Inst { + func.make_inst(InstructionData::Jump { + opcode: Opcode::Jump, + ty: types::VOID, + data: Box::new(JumpData { + destination: dest, + arguments: VariableArgs::new(), + }), + }) +} + +pub fn branch(func: &mut Function, dest: Ebb) -> Inst { + func.make_inst(InstructionData::Branch { + opcode: Opcode::Brz, + ty: types::VOID, + data: Box::new(BranchData { + arg: NO_VALUE, + destination: dest, + arguments: VariableArgs::new(), + }), + }) +} diff --git a/cranelift/src/libcretonne/test_utils/mod.rs b/cranelift/src/libcretonne/test_utils/mod.rs new file mode 100644 index 0000000000..49e6b23be8 --- /dev/null +++ b/cranelift/src/libcretonne/test_utils/mod.rs @@ -0,0 +1,3 @@ +///! Test utility functions. + +pub mod make_inst; From 0d924c67d08bac9fcf4a07de31054fcee1f98450 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 18 Jul 2016 14:06:03 -0700 Subject: [PATCH 0137/3084] Add Layout::ebbs() and the corresponding iterator. Implement some tests, fix bugs in is_ebb_inserted(). --- cranelift/src/libcretonne/layout.rs | 101 +++++++++++++++++++++++++--- 1 file changed, 90 insertions(+), 11 deletions(-) diff --git a/cranelift/src/libcretonne/layout.rs b/cranelift/src/libcretonne/layout.rs index a870e56167..15748eda6a 100644 --- a/cranelift/src/libcretonne/layout.rs +++ b/cranelift/src/libcretonne/layout.rs @@ -3,7 +3,8 @@ //! The order of extended basic blocks in a function and the order of instructions in an EBB is //! determined by the `Layout` data structure defined in this module. -use entity_map::EntityMap; +use std::iter::Iterator; +use entity_map::{EntityMap, EntityRef}; use entities::{Ebb, NO_EBB, Inst, NO_INST}; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not @@ -28,11 +29,11 @@ pub struct Layout { // terminated in both ends by NO_INST. insts: EntityMap, - // First EBB in the layout order, or `NO_EBB` when no EBBs have been laid out. - first_ebb: Ebb, + // First EBB in the layout order, or `None` when no EBBs have been laid out. + first_ebb: Option, - // Last EBB in the layout order, or `NO_EBB` when no EBBs have been laid out. - last_ebb: Ebb, + // Last EBB in the layout order, or `None` when no EBBs have been laid out. + last_ebb: Option, } impl Layout { @@ -41,8 +42,8 @@ impl Layout { Layout { ebbs: EntityMap::new(), insts: EntityMap::new(), - first_ebb: NO_EBB, - last_ebb: NO_EBB, + first_ebb: None, + last_ebb: None, } } } @@ -59,7 +60,7 @@ impl Layout { impl Layout { /// Is `ebb` currently part of the layout? pub fn is_ebb_inserted(&self, ebb: Ebb) -> bool { - ebb != self.first_ebb && self.ebbs.is_valid(ebb) && self.ebbs[ebb].prev == NO_EBB + Some(ebb) == self.first_ebb || (self.ebbs.is_valid(ebb) && self.ebbs[ebb].prev != NO_EBB) } /// Insert `ebb` as the last EBB in the layout. @@ -68,9 +69,12 @@ impl Layout { "Cannot append EBB that is already in the layout"); let node = &mut self.ebbs[ebb]; assert!(node.first_inst == NO_INST && node.last_inst == NO_INST); - node.prev = self.last_ebb; + node.prev = self.last_ebb.unwrap_or_default(); node.next = NO_EBB; - self.last_ebb = ebb; + self.last_ebb = Some(ebb); + if self.first_ebb.is_none() { + self.first_ebb = Some(ebb); + } } /// Insert `ebb` in the layout before the existing EBB `before`. @@ -83,10 +87,20 @@ impl Layout { self.ebbs[ebb].next = before; self.ebbs[ebb].prev = after; self.ebbs[before].prev = ebb; - if after != NO_EBB { + if after == NO_EBB { + self.first_ebb = Some(ebb); + } else { self.ebbs[after].next = ebb; } } + + /// Return an iterator over all EBBs in layout order. + pub fn ebbs<'a>(&'a self) -> Ebbs<'a> { + Ebbs { + layout: self, + next: self.first_ebb, + } + } } #[derive(Clone, Debug, Default)] @@ -97,6 +111,26 @@ struct EbbNode { last_inst: Inst, } +/// Iterate over EBBs in layout order. See `Layout::ebbs()`. +pub struct Ebbs<'a> { + layout: &'a Layout, + next: Option, +} + +impl<'a> Iterator for Ebbs<'a> { + type Item = Ebb; + + fn next(&mut self) -> Option { + match self.next { + Some(ebb) => { + self.next = self.layout.ebbs[ebb].next.wrap(); + Some(ebb) + } + None => None, + } + } +} + /// Methods for arranging instructions. /// /// An instruction starts out as *not inserted* in the layout. An instruction can be inserted into @@ -140,3 +174,48 @@ struct InstNode { prev: Inst, next: Inst, } + +#[cfg(test)] +mod tests { + use super::Layout; + use entity_map::EntityRef; + use entities::Ebb; + + #[test] + fn insert_ebb() { + let mut layout = Layout::new(); + let e0 = Ebb::new(0); + let e1 = Ebb::new(1); + let e2 = Ebb::new(2); + + { + let imm = &layout; + assert!(!imm.is_ebb_inserted(e0)); + assert!(!imm.is_ebb_inserted(e1)); + + let v: Vec = layout.ebbs().collect(); + assert_eq!(v, []); + } + + layout.append_ebb(e1); + assert!(!layout.is_ebb_inserted(e0)); + assert!(layout.is_ebb_inserted(e1)); + assert!(!layout.is_ebb_inserted(e2)); + let v: Vec = layout.ebbs().collect(); + assert_eq!(v, [e1]); + + layout.insert_ebb(e2, e1); + assert!(!layout.is_ebb_inserted(e0)); + assert!(layout.is_ebb_inserted(e1)); + assert!(layout.is_ebb_inserted(e2)); + let v: Vec = layout.ebbs().collect(); + assert_eq!(v, [e2, e1]); + + layout.insert_ebb(e0, e1); + assert!(layout.is_ebb_inserted(e0)); + assert!(layout.is_ebb_inserted(e1)); + assert!(layout.is_ebb_inserted(e2)); + let v: Vec = layout.ebbs().collect(); + assert_eq!(v, [e2, e0, e1]); + } +} From bd2945ab5e22db375758cbcfefca416de4a23549 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 18 Jul 2016 15:04:37 -0700 Subject: [PATCH 0138/3084] Implement instruction order. Each EBB has a linked list of instructions in layout order. --- cranelift/src/libcretonne/layout.rs | 104 ++++++++++++++++++++++++---- 1 file changed, 92 insertions(+), 12 deletions(-) diff --git a/cranelift/src/libcretonne/layout.rs b/cranelift/src/libcretonne/layout.rs index 15748eda6a..d803858d0a 100644 --- a/cranelift/src/libcretonne/layout.rs +++ b/cranelift/src/libcretonne/layout.rs @@ -139,32 +139,53 @@ impl Layout { /// Get the EBB containing `inst`, or `None` if `inst` is not inserted in the layout. pub fn inst_ebb(&self, inst: Inst) -> Option { if self.insts.is_valid(inst) { - let ebb = self.insts[inst].ebb; - if ebb == NO_EBB { - None - } else { - Some(ebb) - } + self.insts[inst].ebb.wrap() } else { None } } /// Append `inst` to the end of `ebb`. - pub fn append_inst(&self, inst: Inst, ebb: Ebb) { + pub fn append_inst(&mut self, inst: Inst, ebb: Ebb) { assert_eq!(self.inst_ebb(inst), None); assert!(self.is_ebb_inserted(ebb), "Cannot append instructions to EBB not in layout"); - unimplemented!(); + let ebb_node = &mut self.ebbs[ebb]; + let inst_node = &mut self.insts[inst]; + inst_node.ebb = ebb; + inst_node.prev = ebb_node.last_inst; + assert_eq!(inst_node.next, NO_INST); + if ebb_node.first_inst == NO_INST { + ebb_node.first_inst = inst; + } } /// Insert `inst` before the instruction `before` in the same EBB. - pub fn insert_inst(&self, inst: Inst, before: Inst) { + pub fn insert_inst(&mut self, inst: Inst, before: Inst) { assert_eq!(self.inst_ebb(inst), None); let ebb = self.inst_ebb(before) .expect("Instruction before insertion point not in the layout"); - assert!(ebb != NO_EBB); - unimplemented!(); + let after = self.insts[before].prev; + { + let inst_node = &mut self.insts[inst]; + inst_node.ebb = ebb; + inst_node.next = before; + inst_node.prev = after; + } + self.insts[before].prev = inst; + if after == NO_INST { + self.ebbs[ebb].first_inst = inst; + } else { + self.insts[after].next = inst; + } + } + + /// Iterate over the instructions in `ebb` in layout order. + pub fn ebb_insts<'a>(&'a self, ebb: Ebb) -> Insts<'a> { + Insts { + layout: self, + next: self.ebbs[ebb].first_inst.wrap(), + } } } @@ -175,11 +196,31 @@ struct InstNode { next: Inst, } +/// Iterate over instructions in an EBB in layout order. See `Layout::ebb_insts()`. +pub struct Insts<'a> { + layout: &'a Layout, + next: Option, +} + +impl<'a> Iterator for Insts<'a> { + type Item = Inst; + + fn next(&mut self) -> Option { + match self.next { + Some(inst) => { + self.next = self.layout.insts[inst].next.wrap(); + Some(inst) + } + None => None, + } + } +} + #[cfg(test)] mod tests { use super::Layout; use entity_map::EntityRef; - use entities::Ebb; + use entities::{Ebb, Inst}; #[test] fn insert_ebb() { @@ -218,4 +259,43 @@ mod tests { let v: Vec = layout.ebbs().collect(); assert_eq!(v, [e2, e0, e1]); } + + #[test] + fn insert_inst() { + let mut layout = Layout::new(); + let e1 = Ebb::new(1); + + layout.append_ebb(e1); + let v: Vec = layout.ebb_insts(e1).collect(); + assert_eq!(v, []); + + let i0 = Inst::new(0); + let i1 = Inst::new(1); + let i2 = Inst::new(2); + + assert_eq!(layout.inst_ebb(i0), None); + assert_eq!(layout.inst_ebb(i1), None); + assert_eq!(layout.inst_ebb(i2), None); + + layout.append_inst(i1, e1); + assert_eq!(layout.inst_ebb(i0), None); + assert_eq!(layout.inst_ebb(i1), Some(e1)); + assert_eq!(layout.inst_ebb(i2), None); + let v: Vec = layout.ebb_insts(e1).collect(); + assert_eq!(v, [i1]); + + layout.insert_inst(i2, i1); + assert_eq!(layout.inst_ebb(i0), None); + assert_eq!(layout.inst_ebb(i1), Some(e1)); + assert_eq!(layout.inst_ebb(i2), Some(e1)); + let v: Vec = layout.ebb_insts(e1).collect(); + assert_eq!(v, [i2, i1]); + + layout.insert_inst(i0, i1); + assert_eq!(layout.inst_ebb(i0), Some(e1)); + assert_eq!(layout.inst_ebb(i1), Some(e1)); + assert_eq!(layout.inst_ebb(i2), Some(e1)); + let v: Vec = layout.ebb_insts(e1).collect(); + assert_eq!(v, [i2, i0, i1]); + } } From 0ef0937016666e39a4c69cfc43177cdefa50aecf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 18 Jul 2016 18:09:31 -0700 Subject: [PATCH 0139/3084] More layout tests and bugfixes. Fix bugs in append methods. Linked lists are hard. --- cranelift/src/libcretonne/layout.rs | 132 +++++++++++++++++++++++++--- 1 file changed, 122 insertions(+), 10 deletions(-) diff --git a/cranelift/src/libcretonne/layout.rs b/cranelift/src/libcretonne/layout.rs index d803858d0a..e0ca2cc3cc 100644 --- a/cranelift/src/libcretonne/layout.rs +++ b/cranelift/src/libcretonne/layout.rs @@ -67,14 +67,18 @@ impl Layout { pub fn append_ebb(&mut self, ebb: Ebb) { assert!(!self.is_ebb_inserted(ebb), "Cannot append EBB that is already in the layout"); - let node = &mut self.ebbs[ebb]; - assert!(node.first_inst == NO_INST && node.last_inst == NO_INST); - node.prev = self.last_ebb.unwrap_or_default(); - node.next = NO_EBB; - self.last_ebb = Some(ebb); - if self.first_ebb.is_none() { + { + let node = &mut self.ebbs[ebb]; + assert!(node.first_inst == NO_INST && node.last_inst == NO_INST); + node.prev = self.last_ebb.unwrap_or_default(); + node.next = NO_EBB; + } + if let Some(last) = self.last_ebb { + self.ebbs[last].next = ebb; + } else { self.first_ebb = Some(ebb); } + self.last_ebb = Some(ebb); } /// Insert `ebb` in the layout before the existing EBB `before`. @@ -151,13 +155,18 @@ impl Layout { assert!(self.is_ebb_inserted(ebb), "Cannot append instructions to EBB not in layout"); let ebb_node = &mut self.ebbs[ebb]; - let inst_node = &mut self.insts[inst]; - inst_node.ebb = ebb; - inst_node.prev = ebb_node.last_inst; - assert_eq!(inst_node.next, NO_INST); + { + let inst_node = &mut self.insts[inst]; + inst_node.ebb = ebb; + inst_node.prev = ebb_node.last_inst; + assert_eq!(inst_node.next, NO_INST); + } if ebb_node.first_inst == NO_INST { ebb_node.first_inst = inst; + } else { + self.insts[ebb_node.last_inst].next = inst; } + ebb_node.last_inst = inst; } /// Insert `inst` before the instruction `before` in the same EBB. @@ -222,6 +231,44 @@ mod tests { use entity_map::EntityRef; use entities::{Ebb, Inst}; + #[test] + fn append_ebb() { + let mut layout = Layout::new(); + let e0 = Ebb::new(0); + let e1 = Ebb::new(1); + let e2 = Ebb::new(2); + + { + let imm = &layout; + assert!(!imm.is_ebb_inserted(e0)); + assert!(!imm.is_ebb_inserted(e1)); + + let v: Vec = layout.ebbs().collect(); + assert_eq!(v, []); + } + + layout.append_ebb(e1); + assert!(!layout.is_ebb_inserted(e0)); + assert!(layout.is_ebb_inserted(e1)); + assert!(!layout.is_ebb_inserted(e2)); + let v: Vec = layout.ebbs().collect(); + assert_eq!(v, [e1]); + + layout.append_ebb(e2); + assert!(!layout.is_ebb_inserted(e0)); + assert!(layout.is_ebb_inserted(e1)); + assert!(layout.is_ebb_inserted(e2)); + let v: Vec = layout.ebbs().collect(); + assert_eq!(v, [e1, e2]); + + layout.append_ebb(e0); + assert!(layout.is_ebb_inserted(e0)); + assert!(layout.is_ebb_inserted(e1)); + assert!(layout.is_ebb_inserted(e2)); + let v: Vec = layout.ebbs().collect(); + assert_eq!(v, [e1, e2, e0]); + } + #[test] fn insert_ebb() { let mut layout = Layout::new(); @@ -260,6 +307,45 @@ mod tests { assert_eq!(v, [e2, e0, e1]); } + #[test] + fn append_inst() { + let mut layout = Layout::new(); + let e1 = Ebb::new(1); + + layout.append_ebb(e1); + let v: Vec = layout.ebb_insts(e1).collect(); + assert_eq!(v, []); + + let i0 = Inst::new(0); + let i1 = Inst::new(1); + let i2 = Inst::new(2); + + assert_eq!(layout.inst_ebb(i0), None); + assert_eq!(layout.inst_ebb(i1), None); + assert_eq!(layout.inst_ebb(i2), None); + + layout.append_inst(i1, e1); + assert_eq!(layout.inst_ebb(i0), None); + assert_eq!(layout.inst_ebb(i1), Some(e1)); + assert_eq!(layout.inst_ebb(i2), None); + let v: Vec = layout.ebb_insts(e1).collect(); + assert_eq!(v, [i1]); + + layout.append_inst(i2, e1); + assert_eq!(layout.inst_ebb(i0), None); + assert_eq!(layout.inst_ebb(i1), Some(e1)); + assert_eq!(layout.inst_ebb(i2), Some(e1)); + let v: Vec = layout.ebb_insts(e1).collect(); + assert_eq!(v, [i1, i2]); + + layout.append_inst(i0, e1); + assert_eq!(layout.inst_ebb(i0), Some(e1)); + assert_eq!(layout.inst_ebb(i1), Some(e1)); + assert_eq!(layout.inst_ebb(i2), Some(e1)); + let v: Vec = layout.ebb_insts(e1).collect(); + assert_eq!(v, [i1, i2, i0]); + } + #[test] fn insert_inst() { let mut layout = Layout::new(); @@ -298,4 +384,30 @@ mod tests { let v: Vec = layout.ebb_insts(e1).collect(); assert_eq!(v, [i2, i0, i1]); } + + #[test] + fn multiple_ebbs() { + let mut layout = Layout::new(); + + let e0 = Ebb::new(0); + let e1 = Ebb::new(1); + + layout.append_ebb(e0); + layout.append_ebb(e1); + + let i0 = Inst::new(0); + let i1 = Inst::new(1); + let i2 = Inst::new(2); + let i3 = Inst::new(3); + + layout.append_inst(i0, e0); + layout.append_inst(i1, e0); + layout.append_inst(i2, e1); + layout.append_inst(i3, e1); + + let v0: Vec = layout.ebb_insts(e0).collect(); + let v1: Vec = layout.ebb_insts(e1).collect(); + assert_eq!(v0, [i0, i1]); + assert_eq!(v1, [i2, i3]); + } } From d73c2c9dc0780195599be475a20310a932d9a57f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 18 Jul 2016 18:23:32 -0700 Subject: [PATCH 0140/3084] Replace inst_order with Layout in Function. The Layout also handles EBB layout, so new append_ebb calls are necessary. - Rewrite callers to use the public data member 'layout'. - Implement Debug for Function in terms of the write module to avoid deriving it. --- cranelift/src/libcretonne/cfg.rs | 23 ++++--- cranelift/src/libcretonne/repr.rs | 101 ++++++----------------------- cranelift/src/libcretonne/write.rs | 3 +- cranelift/src/libreader/parser.rs | 3 +- cranelift/src/tools/print_cfg.rs | 33 ++++++---- 5 files changed, 58 insertions(+), 105 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 668f5b51ef..af877f4f9b 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -55,7 +55,7 @@ impl ControlFlowGraph { for ebb in func.ebbs_numerically() { // Flips to true when a terminating instruction is seen. So that if additional // instructions occur an error may be returned. - for inst in func.ebb_insts(ebb) { + for inst in func.layout.ebb_insts(ebb) { match func[inst] { InstructionData::Branch { ty: _, opcode: _, ref data } => { cfg.add_predecessor(data.destination, (ebb, inst)); @@ -109,9 +109,13 @@ mod tests { #[test] fn no_predecessors() { let mut func = Function::new(); - func.make_ebb(); - func.make_ebb(); - func.make_ebb(); + let ebb0 = func.make_ebb(); + let ebb1 = func.make_ebb(); + let ebb2 = func.make_ebb(); + func.layout.append_ebb(ebb0); + func.layout.append_ebb(ebb1); + func.layout.append_ebb(ebb2); + let cfg = ControlFlowGraph::new(&func); let nodes = cfg.iter().collect::>(); assert_eq!(nodes.len(), 3); @@ -129,18 +133,21 @@ mod tests { let ebb0 = func.make_ebb(); let ebb1 = func.make_ebb(); let ebb2 = func.make_ebb(); + func.layout.append_ebb(ebb0); + func.layout.append_ebb(ebb1); + func.layout.append_ebb(ebb2); let br_ebb0_ebb2 = make_inst::branch(&mut func, ebb2); - func.append_inst(ebb0, br_ebb0_ebb2); + func.layout.append_inst(br_ebb0_ebb2, ebb0); let jmp_ebb0_ebb1 = make_inst::jump(&mut func, ebb1); - func.append_inst(ebb0, jmp_ebb0_ebb1); + func.layout.append_inst(jmp_ebb0_ebb1, ebb0); let br_ebb1_ebb1 = make_inst::branch(&mut func, ebb1); - func.append_inst(ebb1, br_ebb1_ebb1); + func.layout.append_inst(br_ebb1_ebb1, ebb1); let jmp_ebb1_ebb2 = make_inst::jump(&mut func, ebb2); - func.append_inst(ebb1, jmp_ebb1_ebb2); + func.layout.append_inst(jmp_ebb1_ebb2, ebb1); let cfg = ControlFlowGraph::new(&func); let ebb0_predecessors = cfg.get_predecessors(ebb0).unwrap(); diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index 27de00ea66..ac47a540fe 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -2,9 +2,10 @@ use types::{Type, FunctionName, Signature, VOID}; use entity_map::EntityRef; -use entities::{Ebb, NO_EBB, Inst, NO_INST, Value, NO_VALUE, ExpandedValue, StackSlot}; +use entities::{Ebb, NO_EBB, Inst, Value, NO_VALUE, ExpandedValue, StackSlot}; use instructions::*; -use std::fmt::{self, Display, Formatter}; +use layout::Layout; +use std::fmt::{self, Debug, Display, Formatter}; use std::ops::{Index, IndexMut}; /// A function. @@ -12,7 +13,6 @@ use std::ops::{Index, IndexMut}; /// The `Function` struct owns all of its instructions and extended basic blocks, and it works as a /// container for those objects by implementing both `Index` and `Index`. /// -#[derive(Debug)] pub struct Function { /// Name of this function. Mostly used by `.cton` files. pub name: FunctionName, @@ -38,9 +38,8 @@ pub struct Function { /// Others index into this table. extended_values: Vec, - // Linked list nodes for the layout order of instructions. Forms a double linked list per EBB, - // terminated in both ends by NO_INST. - inst_order: Vec, + /// Layout of EBBs and instructions in the function body. + pub layout: Layout, } impl Function { @@ -54,7 +53,7 @@ impl Function { instructions: Vec::new(), extended_basic_blocks: Vec::new(), extended_values: Vec::new(), - inst_order: Vec::new(), + layout: Layout::new(), } } @@ -94,11 +93,6 @@ impl Function { pub fn make_inst(&mut self, data: InstructionData) -> Inst { let inst = Inst::new(self.instructions.len()); self.instructions.push(data); - self.inst_order.push(InstNode { - prev: NO_INST, - next: NO_INST, - }); - debug_assert_eq!(self.instructions.len(), self.inst_order.len()); inst } @@ -234,32 +228,6 @@ impl Function { } } - /// Append an instruction to a basic block. - pub fn append_inst(&mut self, ebb: Ebb, inst: Inst) { - let old_last = self[ebb].last_inst; - - self.inst_order[inst.index()] = InstNode { - prev: old_last, - next: NO_INST, - }; - - if old_last == NO_INST { - assert!(self[ebb].first_inst == NO_INST); - self[ebb].first_inst = inst; - } else { - self.inst_order[old_last.index()].next = inst; - } - self[ebb].last_inst = inst; - } - - /// Iterate through the instructions in `ebb`. - pub fn ebb_insts<'a>(&'a self, ebb: Ebb) -> EbbInsts<'a> { - EbbInsts { - func: self, - cur: self[ebb].first_inst, - } - } - // Values. /// Allocate an extended value entry. @@ -286,6 +254,13 @@ impl Function { } } +impl Debug for Function { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + use write::function_to_string; + fmt.write_str(&function_to_string(self)) + } +} + // ====--------------------------------------------------------------------------------------====// // // Stack slot implementation. @@ -362,12 +337,6 @@ pub struct EbbData { /// Last argument to this EBB, or `NO_VALUE` if the block has no arguments. last_arg: Value, - - /// First instruction in this block, or `NO_INST`. - first_inst: Inst, - - /// Last instruction in this block, or `NO_INST`. - last_inst: Inst, } impl EbbData { @@ -375,8 +344,6 @@ impl EbbData { EbbData { first_arg: NO_VALUE, last_arg: NO_VALUE, - first_inst: NO_INST, - last_inst: NO_INST, } } } @@ -395,26 +362,6 @@ impl IndexMut for Function { } } -pub struct EbbInsts<'a> { - func: &'a Function, - cur: Inst, -} - -impl<'a> Iterator for EbbInsts<'a> { - type Item = Inst; - - fn next(&mut self) -> Option { - let prev = self.cur; - if prev == NO_INST { - None - } else { - // Advance self.cur to the next inst. - self.cur = self.func.inst_order[prev.index()].next; - Some(prev) - } - } -} - /// Iterate through all EBBs in a function in numerical order. /// This order is stable, but has little significance to the semantics of the function. pub struct NumericalEbbs { @@ -460,13 +407,6 @@ impl IndexMut for Function { } } -/// A node in a double linked list of instructions is a basic block. -#[derive(Debug)] -struct InstNode { - prev: Inst, - next: Inst, -} - // ====--------------------------------------------------------------------------------------====// // // Value implementation. @@ -590,6 +530,7 @@ mod tests { let ebb = func.make_ebb(); assert_eq!(ebb.to_string(), "ebb0"); assert_eq!(func.ebb_args(ebb).next(), None); + func.layout.append_ebb(ebb); let arg1 = func.append_ebb_arg(ebb, types::F32); assert_eq!(arg1.to_string(), "vx0"); @@ -612,33 +553,29 @@ mod tests { assert_eq!(ebbs.next(), Some(ebb)); assert_eq!(ebbs.next(), None); - assert_eq!(func.ebb_insts(ebb).next(), None); + assert_eq!(func.layout.ebb_insts(ebb).next(), None); let inst = func.make_inst(InstructionData::Nullary { opcode: Opcode::Iconst, ty: types::I32, }); - func.append_inst(ebb, inst); + func.layout.append_inst(inst, ebb); { - let mut ii = func.ebb_insts(ebb); + let mut ii = func.layout.ebb_insts(ebb); assert_eq!(ii.next(), Some(inst)); assert_eq!(ii.next(), None); } - assert_eq!(func[ebb].first_inst, inst); - assert_eq!(func[ebb].last_inst, inst); let inst2 = func.make_inst(InstructionData::Nullary { opcode: Opcode::Iconst, ty: types::I32, }); - func.append_inst(ebb, inst2); + func.layout.append_inst(inst2, ebb); { - let mut ii = func.ebb_insts(ebb); + let mut ii = func.layout.ebb_insts(ebb); assert_eq!(ii.next(), Some(inst)); assert_eq!(ii.next(), Some(inst2)); assert_eq!(ii.next(), None); } - assert_eq!(func[ebb].first_inst, inst); - assert_eq!(func[ebb].last_inst, inst2); } } diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index e49956970a..73cb341d96 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -114,7 +114,7 @@ pub fn write_ebb_header(w: &mut Write, func: &Function, ebb: Ebb) -> Result { pub fn write_ebb(w: &mut Write, func: &Function, ebb: Ebb) -> Result { try!(write_ebb_header(w, func, ebb)); - for inst in func.ebb_insts(ebb) { + for inst in func.layout.ebb_insts(ebb) { try!(write_instruction(w, func, inst)); } Ok(()) @@ -251,6 +251,7 @@ mod tests { "function foo() {\n ss0 = stack_slot 4\n}\n"); let ebb = f.make_ebb(); + f.layout.append_ebb(ebb); assert_eq!(function_to_string(&f), "function foo() {\n ss0 = stack_slot 4\n\nebb0:\n}\n"); diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 34dd15f735..2ba382871d 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -101,6 +101,7 @@ impl Context { // Allocate a new EBB and add a mapping src_ebb -> Ebb. fn add_ebb(&mut self, src_ebb: Ebb, loc: &Location) -> Result { let ebb = self.function.make_ebb(); + self.function.layout.append_ebb(ebb); if self.ebbs.insert(src_ebb, ebb).is_some() { err!(loc, "duplicate EBB: {}", src_ebb) } else { @@ -704,7 +705,7 @@ impl<'a> Parser<'a> { let ctrl_typevar = try!(self.infer_typevar(ctx, opcode, explicit_ctrl_type, &inst_data)); let inst = ctx.function.make_inst(inst_data); let num_results = ctx.function.make_inst_results(inst, ctrl_typevar); - ctx.function.append_inst(ebb, inst); + ctx.function.layout.append_inst(inst, ebb); ctx.add_inst_loc(inst, &opcode_loc); if results.len() != num_results { diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs index 002c206e07..6fe51f47e1 100644 --- a/cranelift/src/tools/print_cfg.rs +++ b/cranelift/src/tools/print_cfg.rs @@ -29,7 +29,11 @@ struct CFGPrinter { impl CFGPrinter { pub fn new(writer: T) -> CFGPrinter { - CFGPrinter{level: 0, writer: writer, buffer: String::new()} + CFGPrinter { + level: 0, + writer: writer, + buffer: String::new(), + } } pub fn print(&mut self, func: &Function) -> Result<(), String> { @@ -58,7 +62,7 @@ impl CFGPrinter { fn append(&mut self, s: &str) { let mut indent = String::new(); - for _ in 0 .. self.level { + for _ in 0..self.level { indent = indent + " "; } self.buffer.push_str(&(indent + s)); @@ -103,23 +107,24 @@ impl CFGPrinter { fn ebb_subgraphs(&mut self, func: &Function) { for ebb in func.ebbs_numerically() { - let inst_data = func.ebb_insts(ebb) + let inst_data = func.layout + .ebb_insts(ebb) .filter(|inst| { match func[*inst] { - InstructionData::Branch{ ty: _, opcode: _, data: _ } => true, - InstructionData::Jump{ ty: _, opcode: _, data: _ } => true, - _ => false + InstructionData::Branch { ty: _, opcode: _, data: _ } => true, + InstructionData::Jump { ty: _, opcode: _, data: _ } => true, + _ => false, } }) .map(|inst| { let op = match func[inst] { - InstructionData::Branch{ ty: _, opcode, ref data } => { + InstructionData::Branch { ty: _, opcode, ref data } => { Some((opcode, data.destination)) - }, - InstructionData::Jump{ ty: _, opcode, ref data } => { + } + InstructionData::Jump { ty: _, opcode, ref data } => { Some((opcode, data.destination)) - }, - _ => None + } + _ => None, }; (inst, op) }) @@ -132,7 +137,10 @@ impl CFGPrinter { } self.append(&format!("{} [shape=record, label=\"{}{}{}\"]", - ebb, "{", insts.join(" | "), "}")); + ebb, + "{", + insts.join(" | "), + "}")); self.newline(); } } @@ -145,7 +153,6 @@ impl CFGPrinter { } } } - } fn print_cfg(filename: String) -> CommandResult { From 4f7c624ba676bc696cf224d4f0bd3e9807042a4f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 18 Jul 2016 18:47:42 -0700 Subject: [PATCH 0141/3084] Implement IntoIterator for Layout. --- cranelift/src/libcretonne/layout.rs | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/cranelift/src/libcretonne/layout.rs b/cranelift/src/libcretonne/layout.rs index e0ca2cc3cc..3cc2ffc03e 100644 --- a/cranelift/src/libcretonne/layout.rs +++ b/cranelift/src/libcretonne/layout.rs @@ -3,7 +3,7 @@ //! The order of extended basic blocks in a function and the order of instructions in an EBB is //! determined by the `Layout` data structure defined in this module. -use std::iter::Iterator; +use std::iter::{Iterator, IntoIterator}; use entity_map::{EntityMap, EntityRef}; use entities::{Ebb, NO_EBB, Inst, NO_INST}; @@ -135,6 +135,16 @@ impl<'a> Iterator for Ebbs<'a> { } } +/// Use a layout reference in a for loop. +impl<'a> IntoIterator for &'a Layout { + type Item = Ebb; + type IntoIter = Ebbs<'a>; + + fn into_iter(self) -> Ebbs<'a> { + self.ebbs() + } +} + /// Methods for arranging instructions. /// /// An instruction starts out as *not inserted* in the layout. An instruction can be inserted into @@ -267,6 +277,15 @@ mod tests { assert!(layout.is_ebb_inserted(e2)); let v: Vec = layout.ebbs().collect(); assert_eq!(v, [e1, e2, e0]); + + { + let imm = &layout; + let mut v = Vec::new(); + for e in imm { + v.push(e); + } + assert_eq!(v, [e1, e2, e0]); + } } #[test] From 652ebbdc27dca3151cd20f1101f669265d78f723 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 18 Jul 2016 18:52:35 -0700 Subject: [PATCH 0142/3084] Use EBB layout order almost everywhere. The ebbs_numerically() function was a workaround for the unimplemented EBB layout order. --- cranelift/src/libcretonne/cfg.rs | 6 +++--- cranelift/src/libcretonne/write.rs | 2 +- cranelift/src/libreader/parser.rs | 2 +- cranelift/src/tools/print_cfg.rs | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index af877f4f9b..3a5f554375 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -48,11 +48,11 @@ impl ControlFlowGraph { // Even ebbs without predecessors should show up in the CFG, albeit // with no entires. - for ebb in func.ebbs_numerically() { + for ebb in &func.layout { cfg.init_ebb(ebb); } - for ebb in func.ebbs_numerically() { + for ebb in &func.layout { // Flips to true when a terminating instruction is seen. So that if additional // instructions occur an error may be returned. for inst in func.layout.ebb_insts(ebb) { @@ -120,7 +120,7 @@ mod tests { let nodes = cfg.iter().collect::>(); assert_eq!(nodes.len(), 3); - let mut fun_ebbs = func.ebbs_numerically(); + let mut fun_ebbs = func.layout.ebbs(); for (ebb, predecessors) in nodes { assert_eq!(*ebb, fun_ebbs.next().unwrap()); assert_eq!(predecessors.len(), 0); diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index 73cb341d96..c43634a636 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -16,7 +16,7 @@ pub fn write_function(w: &mut Write, func: &Function) -> Result { try!(write_spec(w, func)); try!(writeln!(w, " {{")); let mut any = try!(write_preamble(w, func)); - for ebb in func.ebbs_numerically() { + for ebb in &func.layout { if any { try!(writeln!(w, "")); } diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 2ba382871d..95eb91dfd2 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -1116,7 +1116,7 @@ mod tests { .unwrap(); assert_eq!(func.name, "ebbs"); - let mut ebbs = func.ebbs_numerically(); + let mut ebbs = func.layout.ebbs(); let ebb0 = ebbs.next().unwrap(); assert_eq!(func.ebb_args(ebb0).next(), None); diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs index 6fe51f47e1..0709b358fd 100644 --- a/cranelift/src/tools/print_cfg.rs +++ b/cranelift/src/tools/print_cfg.rs @@ -106,7 +106,7 @@ impl CFGPrinter { } fn ebb_subgraphs(&mut self, func: &Function) { - for ebb in func.ebbs_numerically() { + for ebb in &func.layout { let inst_data = func.layout .ebb_insts(ebb) .filter(|inst| { From 79211cb8a647437f6357eb0422efe4aabff56f08 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Mon, 18 Jul 2016 20:30:33 -0700 Subject: [PATCH 0143/3084] Fix formatting --- cranelift/src/libcretonne/lib.rs | 2 +- cranelift/src/libcretonne/test_utils/make_inst.rs | 10 ++-------- cranelift/src/libcretonne/test_utils/mod.rs | 2 +- 3 files changed, 4 insertions(+), 10 deletions(-) diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index d5c501477c..0223e4a4b0 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -19,4 +19,4 @@ pub mod cfg; pub mod entity_map; -#[cfg(test)] pub mod test_utils; +#[cfg(test)]pub mod test_utils; diff --git a/cranelift/src/libcretonne/test_utils/make_inst.rs b/cranelift/src/libcretonne/test_utils/make_inst.rs index 167512b8b5..a12fb47191 100644 --- a/cranelift/src/libcretonne/test_utils/make_inst.rs +++ b/cranelift/src/libcretonne/test_utils/make_inst.rs @@ -1,14 +1,8 @@ -///! Helper functions for generating dummy instructions. +//! Helper functions for generating dummy instructions. use repr::Function; use entities::{Ebb, Inst, NO_VALUE}; -use instructions::{ - InstructionData, - Opcode, - VariableArgs, - JumpData, - BranchData, -}; +use instructions::{InstructionData, Opcode, VariableArgs, JumpData, BranchData}; use types; pub fn jump(func: &mut Function, dest: Ebb) -> Inst { diff --git a/cranelift/src/libcretonne/test_utils/mod.rs b/cranelift/src/libcretonne/test_utils/mod.rs index 49e6b23be8..6273573d6d 100644 --- a/cranelift/src/libcretonne/test_utils/mod.rs +++ b/cranelift/src/libcretonne/test_utils/mod.rs @@ -1,3 +1,3 @@ -///! Test utility functions. +//! Test utility functions. pub mod make_inst; From 78ce51e166f0bc6c7d8371851dd7c277be79b1dc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 19 Jul 2016 09:25:05 -0700 Subject: [PATCH 0144/3084] Don't require Clone + Default for EntityMap values. There are two kinds of entity maps: - A primary map is used to allocate entity references and store the primary entity data. This map only grows when adding new entities with 'push'. - A secondary map contains additional information about entities in a primary map. This map always grows with 'ensure', making default entries for any unknown primary entities. Only require the 'Default + Clone' traits for values stored in a secondary map. Also remove the 'grow automatically' feature of the IndexMut implementation. This means that clients need to call 'ensure' whenever using a potentially unknown entity reference. The 'grow automatically' feature could not be implemented for the Index trait, and it seems unfortunate to have different semantics for Index and IndexMut. --- cranelift/src/libcretonne/entity_map.rs | 55 +++++++++++++++++-------- cranelift/src/libcretonne/layout.rs | 13 +++--- 2 files changed, 45 insertions(+), 23 deletions(-) diff --git a/cranelift/src/libcretonne/entity_map.rs b/cranelift/src/libcretonne/entity_map.rs index 2aa101f5c0..7954996ed7 100644 --- a/cranelift/src/libcretonne/entity_map.rs +++ b/cranelift/src/libcretonne/entity_map.rs @@ -39,17 +39,23 @@ pub trait EntityRef: Copy + Eq { } /// A mapping `K -> V` for densely indexed entity references. +/// +/// A *primary* `EntityMap` contains the main definition of an entity, and it can be used to +/// allocate new entity references with the `push` method. +/// +/// A *secondary* `EntityMap` contains additional data about entities kept in a primary map. The +/// values need to implement `Clone + Default` traits so the map can be grown with `ensure`. +/// pub struct EntityMap - where K: EntityRef, - V: Clone + Default + where K: EntityRef { elems: Vec, unused: PhantomData, } +/// Shared `EntityMap` implementation for all value types. impl EntityMap - where K: EntityRef, - V: Clone + Default + where K: EntityRef { /// Create a new empty map. pub fn new() -> Self { @@ -64,13 +70,6 @@ impl EntityMap k.index() < self.elems.len() } - /// Ensure that `k` is a valid key but adding default entries if necesssary. - pub fn ensure(&mut self, k: K) { - if !self.is_valid(k) { - self.elems.resize(k.index() + 1, V::default()) - } - } - /// Append `v` to the mapping, assigning a new key which is returned. pub fn push(&mut self, v: V) -> K { let k = K::new(self.elems.len()); @@ -79,11 +78,32 @@ impl EntityMap } } -/// Immutable indexing into an `EntityMap`. -/// The indexed value must have been accessed mutably previously, or the key passed to `ensure()`. -impl Index for EntityMap +/// Additional methods for value types that implement `Clone` and `Default`. +/// +/// When the value type implements these additional traits, the `EntityMap` can be resized +/// explicitly with the `ensure` method. +/// +/// Use this for secondary maps that are mapping keys created by another primary map. +impl EntityMap where K: EntityRef, V: Clone + Default +{ + /// Ensure that `k` is a valid key but adding default entries if necesssary. + /// + /// Return a mutable reference to the corresponding entry. + pub fn ensure(&mut self, k: K) -> &mut V { + if !self.is_valid(k) { + self.elems.resize(k.index() + 1, V::default()) + } + &mut self.elems[k.index()] + } +} + +/// Immutable indexing into an `EntityMap`. +/// The indexed value must be in the map, either because it was created by `push`, or the key was +/// passed to `ensure`. +impl Index for EntityMap + where K: EntityRef { type Output = V; @@ -93,13 +113,11 @@ impl Index for EntityMap } /// Mutable indexing into an `EntityMap`. -/// The map is resized automatically if the key has not been used before. +/// Use `ensure` instead if the key is not known to be valid. impl IndexMut for EntityMap - where K: EntityRef, - V: Clone + Default + where K: EntityRef { fn index_mut(&mut self, k: K) -> &mut V { - self.ensure(k); &mut self.elems[k.index()] } } @@ -129,6 +147,7 @@ mod tests { let mut m = EntityMap::new(); assert!(!m.is_valid(r0)); + m.ensure(r2); m[r2] = 3; assert!(m.is_valid(r1)); m[r1] = 5; diff --git a/cranelift/src/libcretonne/layout.rs b/cranelift/src/libcretonne/layout.rs index 3cc2ffc03e..8ae6657dff 100644 --- a/cranelift/src/libcretonne/layout.rs +++ b/cranelift/src/libcretonne/layout.rs @@ -68,7 +68,7 @@ impl Layout { assert!(!self.is_ebb_inserted(ebb), "Cannot append EBB that is already in the layout"); { - let node = &mut self.ebbs[ebb]; + let node = self.ebbs.ensure(ebb); assert!(node.first_inst == NO_INST && node.last_inst == NO_INST); node.prev = self.last_ebb.unwrap_or_default(); node.next = NO_EBB; @@ -88,8 +88,11 @@ impl Layout { assert!(self.is_ebb_inserted(before), "EBB Insertion point not in the layout"); let after = self.ebbs[before].prev; - self.ebbs[ebb].next = before; - self.ebbs[ebb].prev = after; + { + let node = self.ebbs.ensure(ebb); + node.next = before; + node.prev = after; + } self.ebbs[before].prev = ebb; if after == NO_EBB { self.first_ebb = Some(ebb); @@ -166,7 +169,7 @@ impl Layout { "Cannot append instructions to EBB not in layout"); let ebb_node = &mut self.ebbs[ebb]; { - let inst_node = &mut self.insts[inst]; + let inst_node = self.insts.ensure(inst); inst_node.ebb = ebb; inst_node.prev = ebb_node.last_inst; assert_eq!(inst_node.next, NO_INST); @@ -186,7 +189,7 @@ impl Layout { .expect("Instruction before insertion point not in the layout"); let after = self.insts[before].prev; { - let inst_node = &mut self.insts[inst]; + let inst_node = self.insts.ensure(inst); inst_node.ebb = ebb; inst_node.next = before; inst_node.prev = after; From 8a55ed59deb2f332622fc2b9b6f05bc37d064720 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 19 Jul 2016 10:19:29 -0700 Subject: [PATCH 0145/3084] Implement separate data flow graph module. The DFG keeps track of instruction definitions, values, and EBBs. Store the primary definition of each instruction: Opcode and operands. Track SSA values as either the result of an instruction or EBB arguments. --- cranelift/src/libcretonne/dfg.rs | 420 +++++++++++++++++++++++++++++++ cranelift/src/libcretonne/lib.rs | 1 + 2 files changed, 421 insertions(+) create mode 100644 cranelift/src/libcretonne/dfg.rs diff --git a/cranelift/src/libcretonne/dfg.rs b/cranelift/src/libcretonne/dfg.rs new file mode 100644 index 0000000000..7cfdaa7fbf --- /dev/null +++ b/cranelift/src/libcretonne/dfg.rs @@ -0,0 +1,420 @@ +//! Data flow graph tracking Instructions, Values, and EBBs. + +use entity_map::EntityMap; +use entities::{Ebb, Inst, Value, NO_VALUE, ExpandedValue}; +use instructions::InstructionData; +use types::Type; +use std::ops::{Index, IndexMut}; +use std::u16; + +/// A data flow graph defines all instuctions and extended basic blocks in a function as well as +/// the data flow dependencies between them. The DFG also tracks values which can be either +/// instruction results or EBB arguments. +/// +/// The layout of EBBs in the function and of instructions in each EBB is recorded by the +/// `FunctionLayout` data structure which form the other half of the function representation. +/// +pub struct DataFlowGraph { + /// Data about all of the instructions in the function, including opcodes and operands. + /// The instructions in this map are not in program order. That is tracked by `Layout`, along + /// with the EBB containing each instruction. + insts: EntityMap, + + /// Extended basic blocks in the function and their arguments. + /// This map is not in program order. That is handled by `Layout`, and so is the sequence of + /// instructions contained in each EBB. + ebbs: EntityMap, + + /// Extended value table. Most `Value` references refer directly to their defining instruction. + /// Others index into this table. + /// + /// This is implemented directly with a `Vec` rather than an `EntityMap` because + /// the Value entity references can refer to two things -- an instruction or an extended value. + extended_values: Vec, +} + +impl DataFlowGraph { + /// Create a new empty `DataFlowGraph`. + pub fn new() -> DataFlowGraph { + DataFlowGraph { + insts: EntityMap::new(), + ebbs: EntityMap::new(), + extended_values: Vec::new(), + } + } +} + +/// Handling values. +/// +/// Values are either EBB arguments or instruction results. +impl DataFlowGraph { + // Allocate an extended value entry. + fn make_value(&mut self, data: ValueData) -> Value { + let vref = Value::new_table(self.extended_values.len()); + self.extended_values.push(data); + vref + } + + /// Get the type of a value. + pub fn value_type(&self, v: Value) -> Type { + use entities::ExpandedValue::*; + match v.expand() { + Direct(i) => self.insts[i].first_type(), + Table(i) => { + match self.extended_values[i] { + ValueData::Inst { ty, .. } => ty, + ValueData::Arg { ty, .. } => ty, + } + } + None => panic!("NO_VALUE has no type"), + } + } + + /// Get the definition of a value. + /// + /// This is either the instruction that defined it or the Ebb that has the value as an + /// argument. + pub fn value_def(&self, v: Value) -> ValueDef { + use entities::ExpandedValue::*; + match v.expand() { + Direct(inst) => ValueDef::Res(inst, 0), + Table(idx) => { + match self.extended_values[idx] { + ValueData::Inst { inst, num, .. } => ValueDef::Res(inst, num as usize), + ValueData::Arg { ebb, num, .. } => ValueDef::Arg(ebb, num as usize), + } + } + None => panic!("NO_VALUE has no def"), + } + } +} + +/// Where did a value come from? +#[derive(Debug, PartialEq, Eq)] +pub enum ValueDef { + /// Value is the n'th result of an instruction. + Res(Inst, usize), + /// Value is the n'th argument to an EBB. + Arg(Ebb, usize), +} + +// Internal table storage for extended values. +enum ValueData { + // Value is defined by an instruction, but it is not the first result. + Inst { + ty: Type, + num: u16, // Result number starting from 0. + inst: Inst, + next: Value, // Next result defined by `def`. + }, + + // Value is an EBB argument. + Arg { + ty: Type, + num: u16, // Argument number, starting from 0. + ebb: Ebb, + next: Value, // Next argument to `ebb`. + }, +} + +/// Iterate through a list of related value references, such as: +/// +/// - All results defined by an instruction. See `DataFlowGraph::inst_results`. +/// - All arguments to an EBB. See `DataFlowGraph::ebb_args`. +/// +/// A value iterator borrows a `DataFlowGraph` reference. +pub struct Values<'a> { + dfg: &'a DataFlowGraph, + cur: Value, +} + +impl<'a> Iterator for Values<'a> { + type Item = Value; + + fn next(&mut self) -> Option { + let prev = self.cur; + + // Advance self.cur to the next value, or NO_VALUE. + self.cur = match prev.expand() { + ExpandedValue::Direct(inst) => self.dfg.insts[inst].second_result().unwrap_or_default(), + ExpandedValue::Table(index) => { + match self.dfg.extended_values[index] { + ValueData::Inst { next, .. } => next, + ValueData::Arg { next, .. } => next, + } + } + ExpandedValue::None => return None, + }; + + Some(prev) + } +} + +/// Instructions. +/// +impl DataFlowGraph { + /// Create a new instruction. + /// + /// The type of the first result is indicated by `data.ty`. If the instruction produces + /// multiple results, also call `make_inst_results` to allocate value table entries. + pub fn make_inst(&mut self, data: InstructionData) -> Inst { + self.insts.push(data) + } + + /// Create result values for an instruction that produces multiple results. + /// + /// Instructions that produce 0 or 1 result values only need to be created with `make_inst`. If + /// the instruction may produce more than 1 result, call `make_inst_results` to allocate + /// value table entries for the additional results. + /// + /// The result value types are determined from the instruction's value type constraints and the + /// provided `ctrl_typevar` type for polymorphic instructions. For non-polymorphic + /// instructions, `ctrl_typevar` is ignored, and `VOID` can be used. + /// + /// The type of the first result value is also set, even if it was already set in the + /// `InstructionData` passed to `make_inst`. If this function is called with a single-result + /// instruction, that is the only effect. + /// + /// Returns the number of results produced by the instruction. + pub fn make_inst_results(&mut self, inst: Inst, ctrl_typevar: Type) -> usize { + let constraints = self.insts[inst].opcode().constraints(); + let fixed_results = constraints.fixed_results(); + + // Additional values form a linked list starting from the second result value. Generate + // the list backwards so we don't have to modify value table entries in place. (This + // causes additional result values to be numbered backwards which is not the aestetic + // choice, but since it is only visible in extremely rare instructions with 3+ results, + // we don't care). + let mut head = NO_VALUE; + let mut first_type = Type::default(); + + // TBD: Function call return values for direct and indirect function calls. + + if fixed_results > 0 { + for res_idx in (1..fixed_results).rev() { + head = self.make_value(ValueData::Inst { + ty: constraints.result_type(res_idx, ctrl_typevar), + num: res_idx as u16, + inst: inst, + next: head, + }); + } + first_type = constraints.result_type(0, ctrl_typevar); + } + + // Update the second_result pointer in `inst`. + if head != NO_VALUE { + *self.insts[inst] + .second_result_mut() + .expect("instruction format doesn't allow multiple results") = head; + } + *self.insts[inst].first_type_mut() = first_type; + + fixed_results + } + + /// Get the first result of an instruction. + /// + /// If `Inst` doesn't produce any results, this returns a `Value` with a `VOID` type. + pub fn first_result(&self, inst: Inst) -> Value { + Value::new_direct(inst) + } + + /// Iterate through all the results of an instruction. + pub fn inst_results<'a>(&'a self, inst: Inst) -> Values<'a> { + Values { + dfg: self, + cur: if self.insts[inst].first_type().is_void() { + NO_VALUE + } else { + Value::new_direct(inst) + }, + } + } +} + +/// Allow immutable access to instructions via indexing. +impl Index for DataFlowGraph { + type Output = InstructionData; + + fn index<'a>(&'a self, inst: Inst) -> &'a InstructionData { + &self.insts[inst] + } +} + +/// Allow mutable access to instructions via indexing. +impl IndexMut for DataFlowGraph { + fn index_mut<'a>(&'a mut self, inst: Inst) -> &'a mut InstructionData { + &mut self.insts[inst] + } +} + +/// Extended basic blocks. +impl DataFlowGraph { + /// Create a new basic block. + pub fn make_ebb(&mut self) -> Ebb { + self.ebbs.push(EbbData::new()) + } + + /// Get the number of arguments on `ebb`. + pub fn num_ebb_args(&self, ebb: Ebb) -> usize { + let last_arg = self.ebbs[ebb].last_arg; + match last_arg.expand() { + ExpandedValue::None => 0, + ExpandedValue::Table(idx) => { + if let ValueData::Arg { num, .. } = self.extended_values[idx] { + num as usize + 1 + } else { + panic!("inconsistent value table entry for EBB arg"); + } + } + ExpandedValue::Direct(_) => panic!("inconsistent value table entry for EBB arg"), + } + } + + /// Append an argument with type `ty` to `ebb`. + pub fn append_ebb_arg(&mut self, ebb: Ebb, ty: Type) -> Value { + let num_args = self.num_ebb_args(ebb); + assert!(num_args <= u16::MAX as usize, "Too many arguments to EBB"); + let val = self.make_value(ValueData::Arg { + ty: ty, + ebb: ebb, + num: num_args as u16, + next: NO_VALUE, + }); + let last_arg = self.ebbs[ebb].last_arg; + match last_arg.expand() { + // If last_arg is NO_VALUE, we're adding the first EBB argument. + ExpandedValue::None => { + self.ebbs[ebb].first_arg = val; + } + // Append to linked list of arguments. + ExpandedValue::Table(idx) => { + if let ValueData::Arg { ref mut next, .. } = self.extended_values[idx] { + *next = val; + } else { + panic!("inconsistent value table entry for EBB arg"); + } + } + ExpandedValue::Direct(_) => panic!("inconsistent value table entry for EBB arg"), + }; + self.ebbs[ebb].last_arg = val; + val + } + + /// Iterate through the arguments to an EBB. + pub fn ebb_args<'a>(&'a self, ebb: Ebb) -> Values<'a> { + Values { + dfg: self, + cur: self.ebbs[ebb].first_arg, + } + } +} + +// Contents of an extended basic block. +// +// Arguments for an extended basic block are values that dominate everything in the EBB. All +// branches to this EBB must provide matching arguments, and the arguments to the entry EBB must +// match the function arguments. +struct EbbData { + // First argument to this EBB, or `NO_VALUE` if the block has no arguments. + // + // The arguments are all ValueData::Argument entries that form a linked list from `first_arg` + // to `last_arg`. + first_arg: Value, + + // Last argument to this EBB, or `NO_VALUE` if the block has no arguments. + last_arg: Value, +} + +impl EbbData { + fn new() -> EbbData { + EbbData { + first_arg: NO_VALUE, + last_arg: NO_VALUE, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use types; + use instructions::{Opcode, InstructionData}; + + #[test] + fn make_inst() { + let mut dfg = DataFlowGraph::new(); + + let idata = InstructionData::Nullary { + opcode: Opcode::Iconst, + ty: types::I32, + }; + let inst = dfg.make_inst(idata); + assert_eq!(inst.to_string(), "inst0"); + + // Immutable reference resolution. + { + let immdfg = &dfg; + let ins = &immdfg[inst]; + assert_eq!(ins.opcode(), Opcode::Iconst); + assert_eq!(ins.first_type(), types::I32); + } + + // Result iterator. + let mut res = dfg.inst_results(inst); + let val = res.next().unwrap(); + assert!(res.next().is_none()); + + assert_eq!(dfg.value_def(val), ValueDef::Res(inst, 0)); + assert_eq!(dfg.value_type(val), types::I32); + } + + #[test] + fn no_results() { + let mut dfg = DataFlowGraph::new(); + + let idata = InstructionData::Nullary { + opcode: Opcode::Trap, + ty: types::VOID, + }; + let inst = dfg.make_inst(idata); + + // Result iterator should be empty. + let mut res = dfg.inst_results(inst); + assert_eq!(res.next(), None); + } + + #[test] + fn ebb() { + let mut dfg = DataFlowGraph::new(); + + let ebb = dfg.make_ebb(); + assert_eq!(ebb.to_string(), "ebb0"); + assert_eq!(dfg.num_ebb_args(ebb), 0); + assert_eq!(dfg.ebb_args(ebb).next(), None); + + let arg1 = dfg.append_ebb_arg(ebb, types::F32); + assert_eq!(arg1.to_string(), "vx0"); + assert_eq!(dfg.num_ebb_args(ebb), 1); + { + let mut args1 = dfg.ebb_args(ebb); + assert_eq!(args1.next(), Some(arg1)); + assert_eq!(args1.next(), None); + } + let arg2 = dfg.append_ebb_arg(ebb, types::I16); + assert_eq!(arg2.to_string(), "vx1"); + assert_eq!(dfg.num_ebb_args(ebb), 2); + { + let mut args2 = dfg.ebb_args(ebb); + assert_eq!(args2.next(), Some(arg1)); + assert_eq!(args2.next(), Some(arg2)); + assert_eq!(args2.next(), None); + } + + assert_eq!(dfg.value_def(arg1), ValueDef::Arg(ebb, 0)); + assert_eq!(dfg.value_def(arg2), ValueDef::Arg(ebb, 1)); + assert_eq!(dfg.value_type(arg1), types::F32); + assert_eq!(dfg.value_type(arg2), types::I16); + } +} diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 0223e4a4b0..bc2edd9d59 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -13,6 +13,7 @@ pub mod immediates; pub mod entities; pub mod instructions; pub mod repr; +pub mod dfg; pub mod layout; pub mod write; pub mod cfg; From 28804e0b417c443a32c7e63218641e7179f00e06 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 19 Jul 2016 13:05:28 -0700 Subject: [PATCH 0146/3084] Use DataFlowGraph in Function. Replace the three tables instructions, extended_basic_blocks, and extended_values with a single 'dfg' public member. Clients using Function are changed to refer to func.layout and func.dfg respectively. --- cranelift/src/libcretonne/cfg.rs | 14 +- cranelift/src/libcretonne/repr.rs | 444 +----------------- .../src/libcretonne/test_utils/make_inst.rs | 4 +- cranelift/src/libcretonne/write.rs | 20 +- cranelift/src/libreader/parser.rs | 20 +- cranelift/src/tools/print_cfg.rs | 4 +- 6 files changed, 38 insertions(+), 468 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 3a5f554375..e9ecd9d982 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -56,7 +56,7 @@ impl ControlFlowGraph { // Flips to true when a terminating instruction is seen. So that if additional // instructions occur an error may be returned. for inst in func.layout.ebb_insts(ebb) { - match func[inst] { + match func.dfg[inst] { InstructionData::Branch { ty: _, opcode: _, ref data } => { cfg.add_predecessor(data.destination, (ebb, inst)); } @@ -109,9 +109,9 @@ mod tests { #[test] fn no_predecessors() { let mut func = Function::new(); - let ebb0 = func.make_ebb(); - let ebb1 = func.make_ebb(); - let ebb2 = func.make_ebb(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + let ebb2 = func.dfg.make_ebb(); func.layout.append_ebb(ebb0); func.layout.append_ebb(ebb1); func.layout.append_ebb(ebb2); @@ -130,9 +130,9 @@ mod tests { #[test] fn branches_and_jumps() { let mut func = Function::new(); - let ebb0 = func.make_ebb(); - let ebb1 = func.make_ebb(); - let ebb2 = func.make_ebb(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + let ebb2 = func.dfg.make_ebb(); func.layout.append_ebb(ebb0); func.layout.append_ebb(ebb1); func.layout.append_ebb(ebb2); diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr.rs index ac47a540fe..6f7075ae4d 100644 --- a/cranelift/src/libcretonne/repr.rs +++ b/cranelift/src/libcretonne/repr.rs @@ -1,18 +1,14 @@ //! Representation of Cretonne IL functions. -use types::{Type, FunctionName, Signature, VOID}; +use types::{FunctionName, Signature}; use entity_map::EntityRef; -use entities::{Ebb, NO_EBB, Inst, Value, NO_VALUE, ExpandedValue, StackSlot}; -use instructions::*; +use entities::{Ebb, NO_EBB, StackSlot}; +use dfg::DataFlowGraph; use layout::Layout; use std::fmt::{self, Debug, Display, Formatter}; -use std::ops::{Index, IndexMut}; +use std::ops::Index; /// A function. -/// -/// The `Function` struct owns all of its instructions and extended basic blocks, and it works as a -/// container for those objects by implementing both `Index` and `Index`. -/// pub struct Function { /// Name of this function. Mostly used by `.cton` files. pub name: FunctionName, @@ -26,17 +22,8 @@ pub struct Function { /// Stack slots allocated in this function. stack_slots: Vec, - /// Data about all of the instructions in the function. The instructions in this vector is not - /// necessarily in program order. The `Inst` reference indexes into this vector. - instructions: Vec, - - /// Extended basic blocks in the function, not necessarily in program order. The `Ebb` - /// reference indexes into this vector. - extended_basic_blocks: Vec, - - /// Extended value table. Most `Value` references refer directly to their defining instruction. - /// Others index into this table. - extended_values: Vec, + /// Data flow graph containing the primary definition of all instructions, EBBs and values. + pub dfg: DataFlowGraph, /// Layout of EBBs and instructions in the function body. pub layout: Layout, @@ -50,9 +37,7 @@ impl Function { signature: sig, entry_block: NO_EBB, stack_slots: Vec::new(), - instructions: Vec::new(), - extended_basic_blocks: Vec::new(), - extended_values: Vec::new(), + dfg: DataFlowGraph::new(), layout: Layout::new(), } } @@ -83,175 +68,6 @@ impl Function { end: self.stack_slots.len(), } } - - // Instructions. - - /// Create a new instruction. - /// - /// The type of the first result is indicated by `data.ty`. If the instruction produces - /// multiple results, also call `make_inst_results` to allocate value table entries. - pub fn make_inst(&mut self, data: InstructionData) -> Inst { - let inst = Inst::new(self.instructions.len()); - self.instructions.push(data); - inst - } - - /// Create result values for an instruction that produces multiple results. - /// - /// Instructions that produce 0 or 1 result values only need to be created with `make_inst`. If - /// the instruction may produce more than 1 result, call `make_inst_results` to allocate - /// `Value` table entries for the additional results. - /// - /// The result value types are determined from the instruction's value type constraints and the - /// provided `ctrl_typevar` type for polymorphic instructions. For non-polymorphic - /// instructions, `ctrl_typevar` is ignored, and `VOID` can be used. - /// - /// The type of the first result value is also set, even if it was already set in the - /// `InstructionData` passed to `make_inst`. If this function is called with a single-result - /// instruction, that is the only effect. - /// - /// Returns the number of results produced by the instruction. - pub fn make_inst_results(&mut self, inst: Inst, ctrl_typevar: Type) -> usize { - let constraints = self[inst].opcode().constraints(); - let fixed_results = constraints.fixed_results(); - - // Additional values form a linked list starting from the second result value. Generate - // the list backwards so we don't have to modify value table entries in place. (This - // causes additional result values to be numbered backwards which is not the aestetic - // choice, but since it is only visible in extremely rare instructions with 3+ results, - // we don't care). - let mut head = NO_VALUE; - let mut first_type = Type::default(); - - // TBD: Function call return values for direct and indirect function calls. - - if fixed_results > 0 { - for res_idx in (1..fixed_results).rev() { - head = self.make_value(ValueData::Def { - ty: constraints.result_type(res_idx, ctrl_typevar), - def: inst, - next: head, - }); - } - first_type = constraints.result_type(0, ctrl_typevar); - } - - // Update the second_result pointer in `inst`. - if head != NO_VALUE { - *self[inst] - .second_result_mut() - .expect("instruction format doesn't allow multiple results") = head; - } - *self[inst].first_type_mut() = first_type; - - fixed_results - } - - /// Get the first result of an instruction. - /// - /// If `Inst` doesn't produce any results, this returns a `Value` with a `VOID` type. - pub fn first_result(&self, inst: Inst) -> Value { - Value::new_direct(inst) - } - - /// Iterate through all the results of an instruction. - pub fn inst_results<'a>(&'a self, inst: Inst) -> Values<'a> { - Values { - func: self, - cur: if self[inst].first_type() == VOID { - NO_VALUE - } else { - Value::new_direct(inst) - }, - } - } - - // Basic blocks - - /// Create a new basic block. - pub fn make_ebb(&mut self) -> Ebb { - let ebb = Ebb::new(self.extended_basic_blocks.len()); - self.extended_basic_blocks.push(EbbData::new()); - ebb - } - - /// Reference the representation of an EBB. - fn ebb(&self, ebb: Ebb) -> &EbbData { - &self.extended_basic_blocks[ebb.index()] - } - - /// Mutably reference the representation of an EBB. - fn ebb_mut(&mut self, ebb: Ebb) -> &mut EbbData { - &mut self.extended_basic_blocks[ebb.index()] - } - - /// Iterate over all the EBBs in order of creation. - pub fn ebbs_numerically(&self) -> NumericalEbbs { - NumericalEbbs { - cur: 0, - limit: self.extended_basic_blocks.len(), - } - } - - /// Append an argument with type `ty` to `ebb`. - pub fn append_ebb_arg(&mut self, ebb: Ebb, ty: Type) -> Value { - let val = self.make_value(ValueData::Argument { - ty: ty, - ebb: ebb, - next: NO_VALUE, - }); - - let last_arg = self.ebb(ebb).last_arg; - match last_arg.expand() { - // If last_arg = NO_VALUE, we're adding the first EBB argument. - ExpandedValue::None => self.ebb_mut(ebb).first_arg = val, - ExpandedValue::Table(index) => { - // Append to linked list of arguments. - if let ValueData::Argument { ref mut next, .. } = self.extended_values[index] { - *next = val; - } else { - panic!("wrong type of extended value referenced by Ebb::last_arg"); - } - } - ExpandedValue::Direct(_) => panic!("Direct value cannot appear as EBB argument"), - } - self.ebb_mut(ebb).last_arg = val; - - val - } - - /// Iterate through the arguments to an EBB. - pub fn ebb_args<'a>(&'a self, ebb: Ebb) -> Values<'a> { - Values { - func: self, - cur: self.ebb(ebb).first_arg, - } - } - - // Values. - - /// Allocate an extended value entry. - fn make_value(&mut self, data: ValueData) -> Value { - let vref = Value::new_table(self.extended_values.len()); - self.extended_values.push(data); - vref - } - - /// Get the type of a value. - pub fn value_type(&self, v: Value) -> Type { - use entities::ExpandedValue::*; - use self::ValueData::*; - match v.expand() { - Direct(i) => self[i].first_type(), - Table(i) => { - match self.extended_values[i] { - Def { ty, .. } => ty, - Argument { ty, .. } => ty, - } - } - None => panic!("NO_VALUE has no type"), - } - } } impl Debug for Function { @@ -316,197 +132,9 @@ impl Iterator for StackSlotIter { } } -// ====--------------------------------------------------------------------------------------====// -// -// Extended basic block implementation. -// -// ====--------------------------------------------------------------------------------------====// - -/// Contents of an extended basic block. -/// -/// Arguments for an extended basic block are values that dominate everything in the EBB. All -/// branches to this EBB must provide matching arguments, and the arguments to the entry EBB must -/// match the function arguments. -#[derive(Debug)] -pub struct EbbData { - /// First argument to this EBB, or `NO_VALUE` if the block has no arguments. - /// - /// The arguments are all ValueData::Argument entries that form a linked list from `first_arg` - /// to `last_arg`. - first_arg: Value, - - /// Last argument to this EBB, or `NO_VALUE` if the block has no arguments. - last_arg: Value, -} - -impl EbbData { - fn new() -> EbbData { - EbbData { - first_arg: NO_VALUE, - last_arg: NO_VALUE, - } - } -} - -impl Index for Function { - type Output = EbbData; - - fn index<'a>(&'a self, ebb: Ebb) -> &'a EbbData { - &self.extended_basic_blocks[ebb.index()] - } -} - -impl IndexMut for Function { - fn index_mut<'a>(&'a mut self, ebb: Ebb) -> &'a mut EbbData { - &mut self.extended_basic_blocks[ebb.index()] - } -} - -/// Iterate through all EBBs in a function in numerical order. -/// This order is stable, but has little significance to the semantics of the function. -pub struct NumericalEbbs { - cur: usize, - limit: usize, -} - -impl Iterator for NumericalEbbs { - type Item = Ebb; - - fn next(&mut self) -> Option { - if self.cur < self.limit { - let prev = Ebb::new(self.cur); - self.cur += 1; - Some(prev) - } else { - None - } - } -} - -// ====--------------------------------------------------------------------------------------====// -// -// Instruction implementation. -// -// The InstructionData layout is defined in the `instructions` module. -// -// ====--------------------------------------------------------------------------------------====// - -/// Allow immutable access to instructions via function indexing. -impl Index for Function { - type Output = InstructionData; - - fn index<'a>(&'a self, inst: Inst) -> &'a InstructionData { - &self.instructions[inst.index()] - } -} - -/// Allow mutable access to instructions via function indexing. -impl IndexMut for Function { - fn index_mut<'a>(&'a mut self, inst: Inst) -> &'a mut InstructionData { - &mut self.instructions[inst.index()] - } -} - -// ====--------------------------------------------------------------------------------------====// -// -// Value implementation. -// -// ====--------------------------------------------------------------------------------------====// - -// Most values are simply the first value produced by an instruction. -// Other values have an entry in the value table. -#[derive(Debug)] -enum ValueData { - // Value is defined by an instruction, but it is not the first result. - Def { - ty: Type, - def: Inst, - next: Value, // Next result defined by `def`. - }, - - // Value is an EBB argument. - Argument { - ty: Type, - ebb: Ebb, - next: Value, // Next argument to `ebb`. - }, -} - -/// Iterate through a list of related value references, such as: -/// -/// - All results defined by an instruction. -/// - All arguments to an EBB -/// -/// A value iterator borrows a Function reference. -pub struct Values<'a> { - func: &'a Function, - cur: Value, -} - -impl<'a> Iterator for Values<'a> { - type Item = Value; - - fn next(&mut self) -> Option { - let prev = self.cur; - - // Advance self.cur to the next value, or NO_VALUE. - self.cur = match prev.expand() { - ExpandedValue::Direct(inst) => self.func[inst].second_result().unwrap_or_default(), - ExpandedValue::Table(index) => { - match self.func.extended_values[index] { - ValueData::Def { next, .. } => next, - ValueData::Argument { next, .. } => next, - } - } - ExpandedValue::None => return None, - }; - - Some(prev) - } -} - #[cfg(test)] mod tests { use super::*; - use types; - use instructions::*; - - #[test] - fn make_inst() { - let mut func = Function::new(); - - let idata = InstructionData::Nullary { - opcode: Opcode::Iconst, - ty: types::I32, - }; - let inst = func.make_inst(idata); - assert_eq!(inst.to_string(), "inst0"); - - // Immutable reference resolution. - let ins = &func[inst]; - assert_eq!(ins.opcode(), Opcode::Iconst); - assert_eq!(ins.first_type(), types::I32); - - // Result iterator. - let mut res = func.inst_results(inst); - assert!(res.next().is_some()); - assert!(res.next().is_none()); - } - - #[test] - fn no_results() { - let mut func = Function::new(); - - let idata = InstructionData::Nullary { - opcode: Opcode::Trap, - ty: types::VOID, - }; - let inst = func.make_inst(idata); - - // Result iterator should be empty. - let mut res = func.inst_results(inst); - assert_eq!(res.next(), None); - } #[test] fn stack_slot() { @@ -520,62 +148,4 @@ mod tests { assert_eq!(func[ss0].size, 4); assert_eq!(func[ss1].size, 8); } - - #[test] - fn ebb() { - let mut func = Function::new(); - - assert_eq!(func.ebbs_numerically().next(), None); - - let ebb = func.make_ebb(); - assert_eq!(ebb.to_string(), "ebb0"); - assert_eq!(func.ebb_args(ebb).next(), None); - func.layout.append_ebb(ebb); - - let arg1 = func.append_ebb_arg(ebb, types::F32); - assert_eq!(arg1.to_string(), "vx0"); - { - let mut args1 = func.ebb_args(ebb); - assert_eq!(args1.next(), Some(arg1)); - assert_eq!(args1.next(), None); - } - let arg2 = func.append_ebb_arg(ebb, types::I16); - assert_eq!(arg2.to_string(), "vx1"); - { - let mut args2 = func.ebb_args(ebb); - assert_eq!(args2.next(), Some(arg1)); - assert_eq!(args2.next(), Some(arg2)); - assert_eq!(args2.next(), None); - } - - // The numerical ebb iterator doesn't capture the function. - let mut ebbs = func.ebbs_numerically(); - assert_eq!(ebbs.next(), Some(ebb)); - assert_eq!(ebbs.next(), None); - - assert_eq!(func.layout.ebb_insts(ebb).next(), None); - - let inst = func.make_inst(InstructionData::Nullary { - opcode: Opcode::Iconst, - ty: types::I32, - }); - func.layout.append_inst(inst, ebb); - { - let mut ii = func.layout.ebb_insts(ebb); - assert_eq!(ii.next(), Some(inst)); - assert_eq!(ii.next(), None); - } - - let inst2 = func.make_inst(InstructionData::Nullary { - opcode: Opcode::Iconst, - ty: types::I32, - }); - func.layout.append_inst(inst2, ebb); - { - let mut ii = func.layout.ebb_insts(ebb); - assert_eq!(ii.next(), Some(inst)); - assert_eq!(ii.next(), Some(inst2)); - assert_eq!(ii.next(), None); - } - } } diff --git a/cranelift/src/libcretonne/test_utils/make_inst.rs b/cranelift/src/libcretonne/test_utils/make_inst.rs index a12fb47191..2f45a09cba 100644 --- a/cranelift/src/libcretonne/test_utils/make_inst.rs +++ b/cranelift/src/libcretonne/test_utils/make_inst.rs @@ -6,7 +6,7 @@ use instructions::{InstructionData, Opcode, VariableArgs, JumpData, BranchData}; use types; pub fn jump(func: &mut Function, dest: Ebb) -> Inst { - func.make_inst(InstructionData::Jump { + func.dfg.make_inst(InstructionData::Jump { opcode: Opcode::Jump, ty: types::VOID, data: Box::new(JumpData { @@ -17,7 +17,7 @@ pub fn jump(func: &mut Function, dest: Ebb) -> Inst { } pub fn branch(func: &mut Function, dest: Ebb) -> Inst { - func.make_inst(InstructionData::Branch { + func.dfg.make_inst(InstructionData::Branch { opcode: Opcode::Brz, ty: types::VOID, data: Box::new(BranchData { diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index c43634a636..10181e0f63 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -85,7 +85,7 @@ fn write_preamble(w: &mut Write, func: &Function) -> io::Result { // ====--------------------------------------------------------------------------------------====// pub fn write_arg(w: &mut Write, func: &Function, arg: Value) -> Result { - write!(w, "{}: {}", arg, func.value_type(arg)) + write!(w, "{}: {}", arg, func.dfg.value_type(arg)) } pub fn write_ebb_header(w: &mut Write, func: &Function, ebb: Ebb) -> Result { @@ -96,7 +96,7 @@ pub fn write_ebb_header(w: &mut Write, func: &Function, ebb: Ebb) -> Result { // ebb10(vx4: f64, vx5: b1): // - let mut args = func.ebb_args(ebb); + let mut args = func.dfg.ebb_args(ebb); match args.next() { None => return writeln!(w, "{}:", ebb), Some(arg) => { @@ -133,7 +133,7 @@ pub fn write_ebb(w: &mut Write, func: &Function, ebb: Ebb) -> Result { // if it can't be trivially inferred. // fn type_suffix(func: &Function, inst: Inst) -> Option { - let constraints = func[inst].opcode().constraints(); + let constraints = func.dfg[inst].opcode().constraints(); if !constraints.is_polymorphic() { return None; @@ -150,7 +150,7 @@ fn type_suffix(func: &Function, inst: Inst) -> Option { // This polymorphic instruction doesn't support basic type inference. // The controlling type variable is required to be the type of the first result. - let rtype = func.value_type(func.first_result(inst)); + let rtype = func.dfg.value_type(func.dfg.first_result(inst)); assert!(!rtype.is_void(), "Polymorphic instruction must produce a result"); Some(rtype) @@ -161,7 +161,7 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { // First write out the result values, if any. let mut has_results = false; - for r in func.inst_results(inst) { + for r in func.dfg.inst_results(inst) { if !has_results { has_results = true; try!(write!(w, "{}", r)); @@ -174,7 +174,7 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { } // Then the opcode, possibly with a '.type' suffix. - let opcode = func[inst].opcode(); + let opcode = func.dfg[inst].opcode(); match type_suffix(func, inst) { Some(suf) => try!(write!(w, "{}.{}", opcode, suf)), @@ -183,7 +183,7 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { // Then the operands, depending on format. use instructions::InstructionData::*; - match func[inst] { + match func.dfg[inst] { Nullary { .. } => writeln!(w, ""), Unary { arg, .. } => writeln!(w, " {}", arg), UnaryImm { imm, .. } => writeln!(w, " {}", imm), @@ -250,16 +250,16 @@ mod tests { assert_eq!(function_to_string(&f), "function foo() {\n ss0 = stack_slot 4\n}\n"); - let ebb = f.make_ebb(); + let ebb = f.dfg.make_ebb(); f.layout.append_ebb(ebb); assert_eq!(function_to_string(&f), "function foo() {\n ss0 = stack_slot 4\n\nebb0:\n}\n"); - f.append_ebb_arg(ebb, types::I8); + f.dfg.append_ebb_arg(ebb, types::I8); assert_eq!(function_to_string(&f), "function foo() {\n ss0 = stack_slot 4\n\nebb0(vx0: i8):\n}\n"); - f.append_ebb_arg(ebb, types::F32.by(4).unwrap()); + f.dfg.append_ebb_arg(ebb, types::F32.by(4).unwrap()); assert_eq!(function_to_string(&f), "function foo() {\n ss0 = stack_slot 4\n\nebb0(vx0: i8, vx1: f32x4):\n}\n"); } diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 95eb91dfd2..b79ff16cc3 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -100,7 +100,7 @@ impl Context { // Allocate a new EBB and add a mapping src_ebb -> Ebb. fn add_ebb(&mut self, src_ebb: Ebb, loc: &Location) -> Result { - let ebb = self.function.make_ebb(); + let ebb = self.function.dfg.make_ebb(); self.function.layout.append_ebb(ebb); if self.ebbs.insert(src_ebb, ebb).is_some() { err!(loc, "duplicate EBB: {}", src_ebb) @@ -163,7 +163,7 @@ impl Context { // Rewrite all EBB and value references in the function. fn rewrite_references(&mut self) -> Result<()> { for &(inst, loc) in &self.inst_locs { - match self.function[inst] { + match self.function.dfg[inst] { InstructionData::Nullary { .. } | InstructionData::UnaryImm { .. } | InstructionData::UnaryIeee32 { .. } | @@ -646,7 +646,7 @@ impl<'a> Parser<'a> { // ebb-arg ::= Value(vx) ":" * Type(t) let t = try!(self.match_type("expected EBB argument type")); // Allocate the EBB argument and add the mapping. - let value = ctx.function.append_ebb_arg(ebb, t); + let value = ctx.function.dfg.append_ebb_arg(ebb, t); ctx.add_value(vx, value, &vx_location) } @@ -703,8 +703,8 @@ impl<'a> Parser<'a> { // or function call signature. We also need to create values with the right type for all // the instruction results. let ctrl_typevar = try!(self.infer_typevar(ctx, opcode, explicit_ctrl_type, &inst_data)); - let inst = ctx.function.make_inst(inst_data); - let num_results = ctx.function.make_inst_results(inst, ctrl_typevar); + let inst = ctx.function.dfg.make_inst(inst_data); + let num_results = ctx.function.dfg.make_inst_results(inst, ctrl_typevar); ctx.function.layout.append_inst(inst, ebb); ctx.add_inst_loc(inst, &opcode_loc); @@ -720,7 +720,7 @@ impl<'a> Parser<'a> { // holds a reference to `ctx.function`. self.add_values(&mut ctx.values, results.into_iter(), - ctx.function.inst_results(inst)) + ctx.function.dfg.inst_results(inst)) } // Type inference for polymorphic instructions. @@ -750,7 +750,7 @@ impl<'a> Parser<'a> { // layout of the blocks. let ctrl_src_value = inst_data.typevar_operand() .expect("Constraints <-> Format inconsistency"); - ctx.function.value_type(match ctx.values.get(&ctrl_src_value) { + ctx.function.dfg.value_type(match ctx.values.get(&ctrl_src_value) { Some(&v) => v, None => { return err!(self.loc, @@ -1119,12 +1119,12 @@ mod tests { let mut ebbs = func.layout.ebbs(); let ebb0 = ebbs.next().unwrap(); - assert_eq!(func.ebb_args(ebb0).next(), None); + assert_eq!(func.dfg.ebb_args(ebb0).next(), None); let ebb4 = ebbs.next().unwrap(); - let mut ebb4_args = func.ebb_args(ebb4); + let mut ebb4_args = func.dfg.ebb_args(ebb4); let arg0 = ebb4_args.next().unwrap(); - assert_eq!(func.value_type(arg0), types::I32); + assert_eq!(func.dfg.value_type(arg0), types::I32); assert_eq!(ebb4_args.next(), None); } } diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs index 0709b358fd..1c7f0e7a19 100644 --- a/cranelift/src/tools/print_cfg.rs +++ b/cranelift/src/tools/print_cfg.rs @@ -110,14 +110,14 @@ impl CFGPrinter { let inst_data = func.layout .ebb_insts(ebb) .filter(|inst| { - match func[*inst] { + match func.dfg[*inst] { InstructionData::Branch { ty: _, opcode: _, data: _ } => true, InstructionData::Jump { ty: _, opcode: _, data: _ } => true, _ => false, } }) .map(|inst| { - let op = match func[inst] { + let op = match func.dfg[inst] { InstructionData::Branch { ty: _, opcode, ref data } => { Some((opcode, data.destination)) } From d20fe25f330423a9307e110317623fa108f53013 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 19 Jul 2016 13:53:02 -0700 Subject: [PATCH 0147/3084] Prepare for repr sub-modules. --- cranelift/src/libcretonne/{repr.rs => repr/mod.rs} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename cranelift/src/libcretonne/{repr.rs => repr/mod.rs} (100%) diff --git a/cranelift/src/libcretonne/repr.rs b/cranelift/src/libcretonne/repr/mod.rs similarity index 100% rename from cranelift/src/libcretonne/repr.rs rename to cranelift/src/libcretonne/repr/mod.rs From 28069ff2a0d3067f6561f410342a2e9345f9271e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 19 Jul 2016 14:10:30 -0700 Subject: [PATCH 0148/3084] Move IR modules under repr/. Use the cretonne::repr module as a common namespace for sub-modules defining the in-memory representation of Cretonn IL. --- cranelift/src/libcretonne/cfg.rs | 4 ++-- cranelift/src/libcretonne/lib.rs | 7 ------- .../src/libcretonne/{ => repr}/condcodes.rs | 0 cranelift/src/libcretonne/{ => repr}/dfg.rs | 14 +++++++------- cranelift/src/libcretonne/{ => repr}/entities.rs | 0 .../src/libcretonne/{ => repr}/immediates.rs | 0 .../src/libcretonne/{ => repr}/instructions.rs | 10 +++++----- cranelift/src/libcretonne/{ => repr}/layout.rs | 4 ++-- cranelift/src/libcretonne/repr/mod.rs | 16 ++++++++++++---- cranelift/src/libcretonne/{ => repr}/types.rs | 0 .../src/libcretonne/test_utils/make_inst.rs | 6 +++--- cranelift/src/libcretonne/write.rs | 8 ++++---- cranelift/src/libreader/lexer.rs | 8 ++++---- cranelift/src/libreader/parser.rs | 12 ++++++------ cranelift/src/tools/print_cfg.rs | 2 +- 15 files changed, 46 insertions(+), 45 deletions(-) rename cranelift/src/libcretonne/{ => repr}/condcodes.rs (100%) rename cranelift/src/libcretonne/{ => repr}/dfg.rs (97%) rename cranelift/src/libcretonne/{ => repr}/entities.rs (100%) rename cranelift/src/libcretonne/{ => repr}/immediates.rs (100%) rename cranelift/src/libcretonne/{ => repr}/instructions.rs (99%) rename cranelift/src/libcretonne/{ => repr}/layout.rs (99%) rename cranelift/src/libcretonne/{ => repr}/types.rs (100%) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index e9ecd9d982..c2a2b6dcf2 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -21,8 +21,8 @@ //! and (Ebb0, `jmp Ebb2`) respectively. use repr::Function; -use entities::{Inst, Ebb}; -use instructions::InstructionData; +use repr::entities::{Inst, Ebb}; +use repr::instructions::InstructionData; use std::collections::{BTreeSet, BTreeMap, btree_map}; /// A basic block denoted by its enclosing Ebb and last instruction. diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index bc2edd9d59..0b1ac8678e 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -7,14 +7,7 @@ pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); -pub mod types; -pub mod condcodes; -pub mod immediates; -pub mod entities; -pub mod instructions; pub mod repr; -pub mod dfg; -pub mod layout; pub mod write; pub mod cfg; diff --git a/cranelift/src/libcretonne/condcodes.rs b/cranelift/src/libcretonne/repr/condcodes.rs similarity index 100% rename from cranelift/src/libcretonne/condcodes.rs rename to cranelift/src/libcretonne/repr/condcodes.rs diff --git a/cranelift/src/libcretonne/dfg.rs b/cranelift/src/libcretonne/repr/dfg.rs similarity index 97% rename from cranelift/src/libcretonne/dfg.rs rename to cranelift/src/libcretonne/repr/dfg.rs index 7cfdaa7fbf..29a462f73e 100644 --- a/cranelift/src/libcretonne/dfg.rs +++ b/cranelift/src/libcretonne/repr/dfg.rs @@ -1,9 +1,9 @@ //! Data flow graph tracking Instructions, Values, and EBBs. use entity_map::EntityMap; -use entities::{Ebb, Inst, Value, NO_VALUE, ExpandedValue}; -use instructions::InstructionData; -use types::Type; +use repr::entities::{Ebb, Inst, Value, NO_VALUE, ExpandedValue}; +use repr::instructions::InstructionData; +use repr::types::Type; use std::ops::{Index, IndexMut}; use std::u16; @@ -57,7 +57,7 @@ impl DataFlowGraph { /// Get the type of a value. pub fn value_type(&self, v: Value) -> Type { - use entities::ExpandedValue::*; + use repr::entities::ExpandedValue::*; match v.expand() { Direct(i) => self.insts[i].first_type(), Table(i) => { @@ -75,7 +75,7 @@ impl DataFlowGraph { /// This is either the instruction that defined it or the Ebb that has the value as an /// argument. pub fn value_def(&self, v: Value) -> ValueDef { - use entities::ExpandedValue::*; + use repr::entities::ExpandedValue::*; match v.expand() { Direct(inst) => ValueDef::Res(inst, 0), Table(idx) => { @@ -339,8 +339,8 @@ impl EbbData { #[cfg(test)] mod tests { use super::*; - use types; - use instructions::{Opcode, InstructionData}; + use repr::types; + use repr::instructions::{Opcode, InstructionData}; #[test] fn make_inst() { diff --git a/cranelift/src/libcretonne/entities.rs b/cranelift/src/libcretonne/repr/entities.rs similarity index 100% rename from cranelift/src/libcretonne/entities.rs rename to cranelift/src/libcretonne/repr/entities.rs diff --git a/cranelift/src/libcretonne/immediates.rs b/cranelift/src/libcretonne/repr/immediates.rs similarity index 100% rename from cranelift/src/libcretonne/immediates.rs rename to cranelift/src/libcretonne/repr/immediates.rs diff --git a/cranelift/src/libcretonne/instructions.rs b/cranelift/src/libcretonne/repr/instructions.rs similarity index 99% rename from cranelift/src/libcretonne/instructions.rs rename to cranelift/src/libcretonne/repr/instructions.rs index 8ebd63d411..7ebb71bbf4 100644 --- a/cranelift/src/libcretonne/instructions.rs +++ b/cranelift/src/libcretonne/repr/instructions.rs @@ -10,10 +10,10 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::ops::{Deref, DerefMut}; -use entities::*; -use immediates::*; -use condcodes::*; -use types::{self, Type}; +use repr::entities::*; +use repr::immediates::*; +use repr::condcodes::*; +use repr::types::{self, Type}; // Include code generated by `meta/gen_instr.py`. This file contains: // @@ -560,7 +560,7 @@ mod tests { #[test] fn value_set() { - use types::*; + use repr::types::*; let vts = ValueTypeSet { allow_scalars: true, diff --git a/cranelift/src/libcretonne/layout.rs b/cranelift/src/libcretonne/repr/layout.rs similarity index 99% rename from cranelift/src/libcretonne/layout.rs rename to cranelift/src/libcretonne/repr/layout.rs index 8ae6657dff..36ee2dacf8 100644 --- a/cranelift/src/libcretonne/layout.rs +++ b/cranelift/src/libcretonne/repr/layout.rs @@ -5,7 +5,7 @@ use std::iter::{Iterator, IntoIterator}; use entity_map::{EntityMap, EntityRef}; -use entities::{Ebb, NO_EBB, Inst, NO_INST}; +use repr::entities::{Ebb, NO_EBB, Inst, NO_INST}; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not /// contain definitions of instructions or EBBs, but depends on `Inst` and `Ebb` entity references @@ -242,7 +242,7 @@ impl<'a> Iterator for Insts<'a> { mod tests { use super::Layout; use entity_map::EntityRef; - use entities::{Ebb, Inst}; + use repr::entities::{Ebb, Inst}; #[test] fn append_ebb() { diff --git a/cranelift/src/libcretonne/repr/mod.rs b/cranelift/src/libcretonne/repr/mod.rs index 6f7075ae4d..a62527ccbe 100644 --- a/cranelift/src/libcretonne/repr/mod.rs +++ b/cranelift/src/libcretonne/repr/mod.rs @@ -1,10 +1,18 @@ //! Representation of Cretonne IL functions. -use types::{FunctionName, Signature}; +pub mod entities; +pub mod types; +pub mod condcodes; +pub mod immediates; +pub mod instructions; +pub mod dfg; +pub mod layout; + +use repr::types::{FunctionName, Signature}; use entity_map::EntityRef; -use entities::{Ebb, NO_EBB, StackSlot}; -use dfg::DataFlowGraph; -use layout::Layout; +use repr::entities::{Ebb, NO_EBB, StackSlot}; +use repr::dfg::DataFlowGraph; +use repr::layout::Layout; use std::fmt::{self, Debug, Display, Formatter}; use std::ops::Index; diff --git a/cranelift/src/libcretonne/types.rs b/cranelift/src/libcretonne/repr/types.rs similarity index 100% rename from cranelift/src/libcretonne/types.rs rename to cranelift/src/libcretonne/repr/types.rs diff --git a/cranelift/src/libcretonne/test_utils/make_inst.rs b/cranelift/src/libcretonne/test_utils/make_inst.rs index 2f45a09cba..00955a5be5 100644 --- a/cranelift/src/libcretonne/test_utils/make_inst.rs +++ b/cranelift/src/libcretonne/test_utils/make_inst.rs @@ -1,9 +1,9 @@ //! Helper functions for generating dummy instructions. use repr::Function; -use entities::{Ebb, Inst, NO_VALUE}; -use instructions::{InstructionData, Opcode, VariableArgs, JumpData, BranchData}; -use types; +use repr::entities::{Ebb, Inst, NO_VALUE}; +use repr::instructions::{InstructionData, Opcode, VariableArgs, JumpData, BranchData}; +use repr::types; pub fn jump(func: &mut Function, dest: Ebb) -> Inst { func.dfg.make_inst(InstructionData::Jump { diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index 10181e0f63..5b66b81065 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -6,8 +6,8 @@ use std::io::{self, Write}; use repr::Function; -use entities::{Inst, Ebb, Value}; -use types::Type; +use repr::entities::{Inst, Ebb, Value}; +use repr::types::Type; pub type Result = io::Result<()>; @@ -182,7 +182,7 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { } // Then the operands, depending on format. - use instructions::InstructionData::*; + use repr::instructions::InstructionData::*; match func.dfg[inst] { Nullary { .. } => writeln!(w, ""), Unary { arg, .. } => writeln!(w, " {}", arg), @@ -218,7 +218,7 @@ mod tests { use super::*; use super::{needs_quotes, escaped}; use repr::{Function, StackSlotData}; - use types; + use repr::types; #[test] fn quoting() { diff --git a/cranelift/src/libreader/lexer.rs b/cranelift/src/libreader/lexer.rs index 4d3dfdf7c1..87c7482589 100644 --- a/cranelift/src/libreader/lexer.rs +++ b/cranelift/src/libreader/lexer.rs @@ -6,8 +6,8 @@ // ====--------------------------------------------------------------------------------------====// use std::str::CharIndices; -use cretonne::types; -use cretonne::entities::{Value, Ebb}; +use cretonne::repr::types; +use cretonne::repr::entities::{Value, Ebb}; /// The location of a `Token` or `Error`. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -373,8 +373,8 @@ impl<'a> Lexer<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne::types; - use cretonne::entities::{Value, Ebb}; + use cretonne::repr::types; + use cretonne::repr::entities::{Value, Ebb}; fn token<'a>(token: Token<'a>, line: usize) -> Option, LocatedError>> { Some(super::token(token, Location { line_number: line })) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index b79ff16cc3..0c573558c7 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -11,11 +11,11 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::u32; use lexer::{self, Lexer, Token}; -use cretonne::types::{Type, VOID, FunctionName, Signature, ArgumentType, ArgumentExtension}; -use cretonne::immediates::{Imm64, Ieee32, Ieee64}; -use cretonne::entities::*; -use cretonne::instructions::{Opcode, InstructionFormat, InstructionData, VariableArgs, JumpData, - BranchData, ReturnData}; +use cretonne::repr::types::{Type, VOID, FunctionName, Signature, ArgumentType, ArgumentExtension}; +use cretonne::repr::immediates::{Imm64, Ieee32, Ieee64}; +use cretonne::repr::entities::*; +use cretonne::repr::instructions::{Opcode, InstructionFormat, InstructionData, VariableArgs, + JumpData, BranchData, ReturnData}; use cretonne::repr::{Function, StackSlotData}; pub use lexer::Location; @@ -1039,7 +1039,7 @@ impl<'a> Parser<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne::types::{self, ArgumentType, ArgumentExtension}; + use cretonne::repr::types::{self, ArgumentType, ArgumentExtension}; #[test] fn argument_type() { diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs index 1c7f0e7a19..155de0e4a1 100644 --- a/cranelift/src/tools/print_cfg.rs +++ b/cranelift/src/tools/print_cfg.rs @@ -8,7 +8,7 @@ use std::io::{Read, Write, stdout}; use CommandResult; use cretonne::repr::Function; use cretonne::cfg::ControlFlowGraph; -use cretonne::instructions::InstructionData; +use cretonne::repr::instructions::InstructionData; use cton_reader::parser::Parser; pub fn run(files: Vec) -> CommandResult { From 505d49ec41b1c7a6014edd98d701dafa7d56bf50 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Thu, 21 Jul 2016 12:08:02 -0700 Subject: [PATCH 0149/3084] Use EntityMap instead of BTreeMap --- cranelift/src/libcretonne/cfg.rs | 73 ++++++++++++++++--------- cranelift/src/libcretonne/entity_map.rs | 5 ++ 2 files changed, 53 insertions(+), 25 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index c2a2b6dcf2..c6363ef21a 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -1,5 +1,5 @@ //! A control flow graph represented as mappings of extended basic blocks to their predecessors. -//! Predecessors are denoted by tuples of EBB and branch/jump instructions. Each predecessor +//! BasicBlocks are denoted by tuples of EBB and branch/jump instructions. Each predecessor //! tuple corresponds to the end of a basic block. //! //! ```c @@ -23,20 +23,21 @@ use repr::Function; use repr::entities::{Inst, Ebb}; use repr::instructions::InstructionData; -use std::collections::{BTreeSet, BTreeMap, btree_map}; +use entity_map::EntityMap; +use std::collections::BTreeSet; /// A basic block denoted by its enclosing Ebb and last instruction. -pub type Predecessor = (Ebb, Inst); +pub type BasicBlock = (Ebb, Inst); /// Storing predecessors in a BTreeSet ensures that their ordering is /// stable with no duplicates. -pub type PredecessorSet = BTreeSet; +pub type BasicBlockSet = BTreeSet; /// The Control Flow Graph maintains a mapping of ebbs to their predecessors /// where predecessors are basic blocks. #[derive(Debug)] pub struct ControlFlowGraph { - data: BTreeMap, + data: EntityMap, } impl ControlFlowGraph { @@ -44,12 +45,12 @@ impl ControlFlowGraph { /// blocks within the CFG's associated function. Basic sanity checks will /// also be performed to ensure that the blocks are well formed. pub fn new(func: &Function) -> ControlFlowGraph { - let mut cfg = ControlFlowGraph { data: BTreeMap::new() }; + let mut cfg = ControlFlowGraph { data: EntityMap::new() }; // Even ebbs without predecessors should show up in the CFG, albeit // with no entires. - for ebb in &func.layout { - cfg.init_ebb(ebb); + for _ in &func.layout { + cfg.push_ebb(); } for ebb in &func.layout { @@ -70,25 +71,48 @@ impl ControlFlowGraph { cfg } - /// Initializes a predecessor set for some ebb. If an ebb already has an - /// entry it will be clobbered. - pub fn init_ebb(&mut self, ebb: Ebb) -> &mut PredecessorSet { - self.data.insert(ebb, BTreeSet::new()); - self.data.get_mut(&ebb).unwrap() + pub fn push_ebb(&mut self) { + self.data.push(BTreeSet::new()); } - pub fn add_predecessor(&mut self, ebb: Ebb, predecessor: Predecessor) { - self.data.get_mut(&ebb).unwrap().insert(predecessor); + pub fn add_predecessor(&mut self, ebb: Ebb, predecessor: BasicBlock) { + self.data[ebb].insert(predecessor); } /// Returns all of the predecessors for some ebb, if it has an entry. - pub fn get_predecessors(&self, ebb: Ebb) -> Option<&PredecessorSet> { - self.data.get(&ebb) + pub fn get_predecessors(&self, ebb: Ebb) -> &BasicBlockSet { + &self.data[ebb] } - /// An iterator over all of the ebb to predecessor mappings in the CFG. - pub fn iter<'a>(&'a self) -> btree_map::Iter<'a, Ebb, PredecessorSet> { - self.data.iter() + pub fn len(&self) -> usize { + self.data.len() + } + + pub fn iter(&self) -> CFGIter { + CFGIter { + cur: 0, + cfg: &self, + } + } +} + +pub struct CFGIter<'a> { + cfg: &'a ControlFlowGraph, + cur: usize, +} + +impl<'a> Iterator for CFGIter<'a> { + type Item = (Ebb, &'a BasicBlockSet); + + fn next(&mut self) -> Option { + if self.cur < self.cfg.len() { + let ebb = Ebb::with_number(self.cur as u32).unwrap(); + let bbs = self.cfg.get_predecessors(ebb); + self.cur += 1; + Some((ebb, bbs)) + } else { + None + } } } @@ -122,7 +146,7 @@ mod tests { let mut fun_ebbs = func.layout.ebbs(); for (ebb, predecessors) in nodes { - assert_eq!(*ebb, fun_ebbs.next().unwrap()); + assert_eq!(ebb, fun_ebbs.next().unwrap()); assert_eq!(predecessors.len(), 0); } } @@ -150,9 +174,9 @@ mod tests { func.layout.append_inst(jmp_ebb1_ebb2, ebb1); let cfg = ControlFlowGraph::new(&func); - let ebb0_predecessors = cfg.get_predecessors(ebb0).unwrap(); - let ebb1_predecessors = cfg.get_predecessors(ebb1).unwrap(); - let ebb2_predecessors = cfg.get_predecessors(ebb2).unwrap(); + let ebb0_predecessors = cfg.get_predecessors(ebb0); + let ebb1_predecessors = cfg.get_predecessors(ebb1); + let ebb2_predecessors = cfg.get_predecessors(ebb2); assert_eq!(ebb0_predecessors.len(), 0); assert_eq!(ebb1_predecessors.len(), 2); assert_eq!(ebb2_predecessors.len(), 2); @@ -162,5 +186,4 @@ mod tests { assert_eq!(ebb2_predecessors.contains(&(ebb0, br_ebb0_ebb2)), true); assert_eq!(ebb2_predecessors.contains(&(ebb1, jmp_ebb1_ebb2)), true); } - } diff --git a/cranelift/src/libcretonne/entity_map.rs b/cranelift/src/libcretonne/entity_map.rs index 7954996ed7..0a7faccb41 100644 --- a/cranelift/src/libcretonne/entity_map.rs +++ b/cranelift/src/libcretonne/entity_map.rs @@ -46,6 +46,7 @@ pub trait EntityRef: Copy + Eq { /// A *secondary* `EntityMap` contains additional data about entities kept in a primary map. The /// values need to implement `Clone + Default` traits so the map can be grown with `ensure`. /// +#[derive(Debug)] pub struct EntityMap where K: EntityRef { @@ -76,6 +77,10 @@ impl EntityMap self.elems.push(v); k } + + pub fn len(&self) -> usize { + self.elems.len() + } } /// Additional methods for value types that implement `Clone` and `Default`. From a4126108a033081449558ed29ce7d98e0704dacb Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Thu, 21 Jul 2016 12:36:51 -0700 Subject: [PATCH 0150/3084] Track predecessors as well as successors in the CFG --- cranelift/src/libcretonne/cfg.rs | 77 +++++++++++++++++++++++++------- cranelift/src/tools/print_cfg.rs | 2 +- 2 files changed, 61 insertions(+), 18 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index c6363ef21a..9fabdff160 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -1,4 +1,6 @@ -//! A control flow graph represented as mappings of extended basic blocks to their predecessors. +//! A control flow graph represented as mappings of extended basic blocks to their predecessors +//! and successors. Successors are represented as extended basic blocks while predecessors are +//! represented by basic blocks. //! BasicBlocks are denoted by tuples of EBB and branch/jump instructions. Each predecessor //! tuple corresponds to the end of a basic block. //! @@ -33,17 +35,33 @@ pub type BasicBlock = (Ebb, Inst); /// stable with no duplicates. pub type BasicBlockSet = BTreeSet; +/// A container for the successors and predecessors of some Ebb. +#[derive(Debug)] +pub struct CFGNode { + pub successors: BTreeSet, + pub predecessors: BasicBlockSet, +} + +impl CFGNode { + pub fn new() -> CFGNode { + CFGNode { + successors: BTreeSet::new(), + predecessors: BTreeSet::new(), + } + } +} + /// The Control Flow Graph maintains a mapping of ebbs to their predecessors -/// where predecessors are basic blocks. +/// and successors where predecessors are basic blocks and successors are +/// extended basic blocks. #[derive(Debug)] pub struct ControlFlowGraph { - data: EntityMap, + data: EntityMap, } impl ControlFlowGraph { /// During initialization mappings will be generated for any existing - /// blocks within the CFG's associated function. Basic sanity checks will - /// also be performed to ensure that the blocks are well formed. + /// blocks within the CFG's associated function. pub fn new(func: &Function) -> ControlFlowGraph { let mut cfg = ControlFlowGraph { data: EntityMap::new() }; @@ -54,14 +72,14 @@ impl ControlFlowGraph { } for ebb in &func.layout { - // Flips to true when a terminating instruction is seen. So that if additional - // instructions occur an error may be returned. for inst in func.layout.ebb_insts(ebb) { match func.dfg[inst] { InstructionData::Branch { ty: _, opcode: _, ref data } => { + cfg.add_successor(ebb, data.destination); cfg.add_predecessor(data.destination, (ebb, inst)); } InstructionData::Jump { ty: _, opcode: _, ref data } => { + cfg.add_successor(ebb, data.destination); cfg.add_predecessor(data.destination, (ebb, inst)); } _ => (), @@ -72,36 +90,44 @@ impl ControlFlowGraph { } pub fn push_ebb(&mut self) { - self.data.push(BTreeSet::new()); + self.data.push(CFGNode::new()); + } + + pub fn add_successor(&mut self, from: Ebb, to: Ebb) { + self.data[from].successors.insert(to); } pub fn add_predecessor(&mut self, ebb: Ebb, predecessor: BasicBlock) { - self.data[ebb].insert(predecessor); + self.data[ebb].predecessors.insert(predecessor); } - /// Returns all of the predecessors for some ebb, if it has an entry. pub fn get_predecessors(&self, ebb: Ebb) -> &BasicBlockSet { - &self.data[ebb] + &self.data[ebb].predecessors + } + + pub fn get_successors(&self, ebb: Ebb) -> &BTreeSet { + &self.data[ebb].successors } pub fn len(&self) -> usize { self.data.len() } - pub fn iter(&self) -> CFGIter { - CFGIter { + pub fn predecessors_iter(&self) -> CFGPredecessorsIter { + CFGPredecessorsIter { cur: 0, cfg: &self, } } } -pub struct CFGIter<'a> { +/// Iterate through every mapping of ebb to predecessors in the CFG +pub struct CFGPredecessorsIter<'a> { cfg: &'a ControlFlowGraph, cur: usize, } -impl<'a> Iterator for CFGIter<'a> { +impl<'a> Iterator for CFGPredecessorsIter<'a> { type Item = (Ebb, &'a BasicBlockSet); fn next(&mut self) -> Option { @@ -127,7 +153,7 @@ mod tests { fn empty() { let func = Function::new(); let cfg = ControlFlowGraph::new(&func); - assert_eq!(None, cfg.iter().next()); + assert_eq!(None, cfg.predecessors_iter().next()); } #[test] @@ -141,13 +167,15 @@ mod tests { func.layout.append_ebb(ebb2); let cfg = ControlFlowGraph::new(&func); - let nodes = cfg.iter().collect::>(); + let nodes = cfg.predecessors_iter().collect::>(); assert_eq!(nodes.len(), 3); let mut fun_ebbs = func.layout.ebbs(); for (ebb, predecessors) in nodes { assert_eq!(ebb, fun_ebbs.next().unwrap()); assert_eq!(predecessors.len(), 0); + assert_eq!(predecessors.len(), 0); + assert_eq!(cfg.get_successors(ebb).len(), 0); } } @@ -174,9 +202,15 @@ mod tests { func.layout.append_inst(jmp_ebb1_ebb2, ebb1); let cfg = ControlFlowGraph::new(&func); + let ebb0_predecessors = cfg.get_predecessors(ebb0); let ebb1_predecessors = cfg.get_predecessors(ebb1); let ebb2_predecessors = cfg.get_predecessors(ebb2); + + let ebb0_successors = cfg.get_successors(ebb0); + let ebb1_successors = cfg.get_successors(ebb1); + let ebb2_successors = cfg.get_successors(ebb2); + assert_eq!(ebb0_predecessors.len(), 0); assert_eq!(ebb1_predecessors.len(), 2); assert_eq!(ebb2_predecessors.len(), 2); @@ -185,5 +219,14 @@ mod tests { assert_eq!(ebb1_predecessors.contains(&(ebb1, br_ebb1_ebb1)), true); assert_eq!(ebb2_predecessors.contains(&(ebb0, br_ebb0_ebb2)), true); assert_eq!(ebb2_predecessors.contains(&(ebb1, jmp_ebb1_ebb2)), true); + + assert_eq!(ebb0_successors.len(), 2); + assert_eq!(ebb1_successors.len(), 2); + assert_eq!(ebb2_successors.len(), 0); + + assert_eq!(ebb0_successors.contains(&ebb1), true); + assert_eq!(ebb0_successors.contains(&ebb2), true); + assert_eq!(ebb1_successors.contains(&ebb1), true); + assert_eq!(ebb1_successors.contains(&ebb2), true); } } diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs index 155de0e4a1..dfa0113fee 100644 --- a/cranelift/src/tools/print_cfg.rs +++ b/cranelift/src/tools/print_cfg.rs @@ -146,7 +146,7 @@ impl CFGPrinter { } fn cfg_connections(&mut self, cfg: &ControlFlowGraph) { - for (ref ebb, ref predecessors) in cfg.iter() { + for (ref ebb, ref predecessors) in cfg.predecessors_iter() { for &(parent, inst) in *predecessors { self.append(&format!("{}:{} -> {}", parent, inst, ebb)); self.newline(); From 5037cc4db698705759f3d8df124a13c3108f0d63 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Thu, 21 Jul 2016 15:22:27 -0700 Subject: [PATCH 0151/3084] Add support for postorder traversal of the cfg. --- cranelift/src/libcretonne/cfg.rs | 69 ++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 9fabdff160..e73e0d04be 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -42,6 +42,15 @@ pub struct CFGNode { pub predecessors: BasicBlockSet, } +impl CFGNode { + /// CFG Node successors stripped of loop edges. + pub fn children(&self) -> Vec { + let pred_ebbs = self.predecessors.iter().map(|&(ebb, _)| { ebb }).collect(); + let children = self.successors.difference(&pred_ebbs).cloned().collect(); + children + } +} + impl CFGNode { pub fn new() -> CFGNode { CFGNode { @@ -109,6 +118,26 @@ impl ControlFlowGraph { &self.data[ebb].successors } + pub fn get_children(&self, ebb: Ebb) -> Vec { + self.data[ebb].children() + } + + pub fn postorder_ebbs(&self) -> Vec { + if self.len() < 1 { + return Vec::new(); + } + let mut stack_a = vec![Ebb::with_number(0).unwrap()]; + let mut stack_b = Vec::new(); + while stack_a.len() > 0 { + let cur = stack_a.pop().unwrap(); + for child in self.get_children(cur) { + stack_a.push(child); + } + stack_b.push(cur); + } + stack_b + } + pub fn len(&self) -> usize { self.data.len() } @@ -228,5 +257,45 @@ mod tests { assert_eq!(ebb0_successors.contains(&ebb2), true); assert_eq!(ebb1_successors.contains(&ebb1), true); assert_eq!(ebb1_successors.contains(&ebb2), true); + + assert_eq!(cfg.get_children(ebb0), vec![ebb1, ebb2]); + assert_eq!(cfg.get_children(ebb1), vec![ebb2]); + assert_eq!(cfg.get_children(ebb2), Vec::new()); + } + + #[test] + fn postorder_traversal() { + let mut func = Function::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + let ebb2 = func.dfg.make_ebb(); + let ebb3 = func.dfg.make_ebb(); + let ebb4 = func.dfg.make_ebb(); + let ebb5 = func.dfg.make_ebb(); + + func.layout.append_ebb(ebb0); + func.layout.append_ebb(ebb1); + func.layout.append_ebb(ebb2); + func.layout.append_ebb(ebb3); + func.layout.append_ebb(ebb4); + func.layout.append_ebb(ebb5); + + let br_ebb0_ebb1 = make_inst::branch(&mut func, ebb1); + func.layout.append_inst(br_ebb0_ebb1, ebb0); + + let jmp_ebb0_ebb2 = make_inst::jump(&mut func, ebb2); + func.layout.append_inst(jmp_ebb0_ebb2, ebb0); + + let jmp_ebb1_ebb3 = make_inst::jump(&mut func, ebb3); + func.layout.append_inst(jmp_ebb1_ebb3, ebb1); + + let br_ebb2_ebb4 = make_inst::branch(&mut func, ebb4); + func.layout.append_inst(br_ebb2_ebb4, ebb2); + + let jmp_ebb2_ebb5 = make_inst::jump(&mut func, ebb5); + func.layout.append_inst(jmp_ebb2_ebb5, ebb2); + + let cfg = ControlFlowGraph::new(&func); + assert_eq!(cfg.postorder_ebbs(), vec![ebb0, ebb2, ebb5, ebb4, ebb1, ebb3]); } } From 4de89a7f96c6879aa788ff1f335327c2be6587a0 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Thu, 21 Jul 2016 15:24:07 -0700 Subject: [PATCH 0152/3084] Cargo-fmt fixes --- cranelift/src/libcretonne/cfg.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index e73e0d04be..77f40a6f9a 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -45,7 +45,7 @@ pub struct CFGNode { impl CFGNode { /// CFG Node successors stripped of loop edges. pub fn children(&self) -> Vec { - let pred_ebbs = self.predecessors.iter().map(|&(ebb, _)| { ebb }).collect(); + let pred_ebbs = self.predecessors.iter().map(|&(ebb, _)| ebb).collect(); let children = self.successors.difference(&pred_ebbs).cloned().collect(); children } @@ -296,6 +296,7 @@ mod tests { func.layout.append_inst(jmp_ebb2_ebb5, ebb2); let cfg = ControlFlowGraph::new(&func); - assert_eq!(cfg.postorder_ebbs(), vec![ebb0, ebb2, ebb5, ebb4, ebb1, ebb3]); + assert_eq!(cfg.postorder_ebbs(), + vec![ebb0, ebb2, ebb5, ebb4, ebb1, ebb3]); } } From 3d59a95b0dc83d2682157d63c687701dd4f554f6 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Thu, 21 Jul 2016 22:44:13 -0700 Subject: [PATCH 0153/3084] Replace btreesets with vectors. --- cranelift/src/libcretonne/cfg.rs | 78 ++++++++++++++++++-------------- 1 file changed, 45 insertions(+), 33 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 77f40a6f9a..c018ab4671 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -26,36 +26,22 @@ use repr::Function; use repr::entities::{Inst, Ebb}; use repr::instructions::InstructionData; use entity_map::EntityMap; -use std::collections::BTreeSet; /// A basic block denoted by its enclosing Ebb and last instruction. pub type BasicBlock = (Ebb, Inst); -/// Storing predecessors in a BTreeSet ensures that their ordering is -/// stable with no duplicates. -pub type BasicBlockSet = BTreeSet; - /// A container for the successors and predecessors of some Ebb. #[derive(Debug)] pub struct CFGNode { - pub successors: BTreeSet, - pub predecessors: BasicBlockSet, -} - -impl CFGNode { - /// CFG Node successors stripped of loop edges. - pub fn children(&self) -> Vec { - let pred_ebbs = self.predecessors.iter().map(|&(ebb, _)| ebb).collect(); - let children = self.successors.difference(&pred_ebbs).cloned().collect(); - children - } + pub successors: Vec, + pub predecessors: Vec, } impl CFGNode { pub fn new() -> CFGNode { CFGNode { - successors: BTreeSet::new(), - predecessors: BTreeSet::new(), + successors: Vec::new(), + predecessors: Vec::new(), } } } @@ -103,25 +89,21 @@ impl ControlFlowGraph { } pub fn add_successor(&mut self, from: Ebb, to: Ebb) { - self.data[from].successors.insert(to); + self.data[from].successors.push(to); } pub fn add_predecessor(&mut self, ebb: Ebb, predecessor: BasicBlock) { - self.data[ebb].predecessors.insert(predecessor); + self.data[ebb].predecessors.push(predecessor); } - pub fn get_predecessors(&self, ebb: Ebb) -> &BasicBlockSet { + pub fn get_predecessors(&self, ebb: Ebb) -> &Vec { &self.data[ebb].predecessors } - pub fn get_successors(&self, ebb: Ebb) -> &BTreeSet { + pub fn get_successors(&self, ebb: Ebb) -> &Vec { &self.data[ebb].successors } - pub fn get_children(&self, ebb: Ebb) -> Vec { - self.data[ebb].children() - } - pub fn postorder_ebbs(&self) -> Vec { if self.len() < 1 { return Vec::new(); @@ -130,8 +112,10 @@ impl ControlFlowGraph { let mut stack_b = Vec::new(); while stack_a.len() > 0 { let cur = stack_a.pop().unwrap(); - for child in self.get_children(cur) { - stack_a.push(child); + for child in &self.data[cur].successors { + if *child != cur && !stack_a.contains(child) { + stack_a.push(child.clone()); + } } stack_b.push(cur); } @@ -157,7 +141,7 @@ pub struct CFGPredecessorsIter<'a> { } impl<'a> Iterator for CFGPredecessorsIter<'a> { - type Item = (Ebb, &'a BasicBlockSet); + type Item = (Ebb, &'a Vec); fn next(&mut self) -> Option { if self.cur < self.cfg.len() { @@ -257,10 +241,6 @@ mod tests { assert_eq!(ebb0_successors.contains(&ebb2), true); assert_eq!(ebb1_successors.contains(&ebb1), true); assert_eq!(ebb1_successors.contains(&ebb2), true); - - assert_eq!(cfg.get_children(ebb0), vec![ebb1, ebb2]); - assert_eq!(cfg.get_children(ebb1), vec![ebb2]); - assert_eq!(cfg.get_children(ebb2), Vec::new()); } #[test] @@ -286,6 +266,12 @@ mod tests { let jmp_ebb0_ebb2 = make_inst::jump(&mut func, ebb2); func.layout.append_inst(jmp_ebb0_ebb2, ebb0); + let br_ebb2_ebb2 = make_inst::branch(&mut func, ebb2); + func.layout.append_inst(br_ebb2_ebb2, ebb2); + + let br_ebb2_ebb1 = make_inst::branch(&mut func, ebb1); + func.layout.append_inst(br_ebb2_ebb1, ebb2); + let jmp_ebb1_ebb3 = make_inst::jump(&mut func, ebb3); func.layout.append_inst(jmp_ebb1_ebb3, ebb1); @@ -299,4 +285,30 @@ mod tests { assert_eq!(cfg.postorder_ebbs(), vec![ebb0, ebb2, ebb5, ebb4, ebb1, ebb3]); } + + #[test] + fn loop_edge() { + let mut func = Function::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + let ebb2 = func.dfg.make_ebb(); + let ebb3 = func.dfg.make_ebb(); + func.layout.append_ebb(ebb0); + func.layout.append_ebb(ebb1); + func.layout.append_ebb(ebb2); + func.layout.append_ebb(ebb3); + + let jmp_ebb0_ebb1 = make_inst::jump(&mut func, ebb1); + let br_ebb1_ebb3 = make_inst::branch(&mut func, ebb3); + let jmp_ebb1_ebb2 = make_inst::jump(&mut func, ebb2); + let jmp_ebb2_ebb3 = make_inst::jump(&mut func, ebb3); + + func.layout.append_inst(jmp_ebb0_ebb1, ebb0); + func.layout.append_inst(br_ebb1_ebb3, ebb1); + func.layout.append_inst(jmp_ebb1_ebb2, ebb1); + func.layout.append_inst(jmp_ebb2_ebb3, ebb2); + + let cfg = ControlFlowGraph::new(&func); + assert_eq!(cfg.postorder_ebbs(), vec![ebb0, ebb1, ebb2, ebb3]); + } } From 27cb00ef42dcca40bfe1035c254d61a6e0d61c8b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Jul 2016 09:34:28 -0700 Subject: [PATCH 0154/3084] Rename the 'repr' module to 'ir'. This module and its submodules define the Intermidiate Representation of the Cretonne IL. --- cranelift/src/libcretonne/cfg.rs | 8 ++++---- .../src/libcretonne/{repr => ir}/condcodes.rs | 0 cranelift/src/libcretonne/{repr => ir}/dfg.rs | 14 +++++++------- cranelift/src/libcretonne/{repr => ir}/entities.rs | 0 .../src/libcretonne/{repr => ir}/immediates.rs | 0 .../src/libcretonne/{repr => ir}/instructions.rs | 10 +++++----- cranelift/src/libcretonne/{repr => ir}/layout.rs | 4 ++-- cranelift/src/libcretonne/{repr => ir}/mod.rs | 8 ++++---- cranelift/src/libcretonne/{repr => ir}/types.rs | 0 cranelift/src/libcretonne/lib.rs | 2 +- cranelift/src/libcretonne/test_utils/make_inst.rs | 8 ++++---- cranelift/src/libcretonne/write.rs | 12 ++++++------ cranelift/src/libreader/lexer.rs | 8 ++++---- cranelift/src/libreader/parser.rs | 12 ++++++------ cranelift/src/tools/print_cfg.rs | 4 ++-- 15 files changed, 45 insertions(+), 45 deletions(-) rename cranelift/src/libcretonne/{repr => ir}/condcodes.rs (100%) rename cranelift/src/libcretonne/{repr => ir}/dfg.rs (97%) rename cranelift/src/libcretonne/{repr => ir}/entities.rs (100%) rename cranelift/src/libcretonne/{repr => ir}/immediates.rs (100%) rename cranelift/src/libcretonne/{repr => ir}/instructions.rs (99%) rename cranelift/src/libcretonne/{repr => ir}/layout.rs (99%) rename cranelift/src/libcretonne/{repr => ir}/mod.rs (96%) rename cranelift/src/libcretonne/{repr => ir}/types.rs (100%) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index c018ab4671..970af69afd 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -22,9 +22,9 @@ //! Here Ebb1 and Ebb2 would each have a single predecessor denoted as (Ebb0, `brz vx, Ebb1`) //! and (Ebb0, `jmp Ebb2`) respectively. -use repr::Function; -use repr::entities::{Inst, Ebb}; -use repr::instructions::InstructionData; +use ir::Function; +use ir::entities::{Inst, Ebb}; +use ir::instructions::InstructionData; use entity_map::EntityMap; /// A basic block denoted by its enclosing Ebb and last instruction. @@ -158,7 +158,7 @@ impl<'a> Iterator for CFGPredecessorsIter<'a> { #[cfg(test)] mod tests { use super::*; - use repr::Function; + use ir::Function; use test_utils::make_inst; diff --git a/cranelift/src/libcretonne/repr/condcodes.rs b/cranelift/src/libcretonne/ir/condcodes.rs similarity index 100% rename from cranelift/src/libcretonne/repr/condcodes.rs rename to cranelift/src/libcretonne/ir/condcodes.rs diff --git a/cranelift/src/libcretonne/repr/dfg.rs b/cranelift/src/libcretonne/ir/dfg.rs similarity index 97% rename from cranelift/src/libcretonne/repr/dfg.rs rename to cranelift/src/libcretonne/ir/dfg.rs index 29a462f73e..d377813544 100644 --- a/cranelift/src/libcretonne/repr/dfg.rs +++ b/cranelift/src/libcretonne/ir/dfg.rs @@ -1,9 +1,9 @@ //! Data flow graph tracking Instructions, Values, and EBBs. use entity_map::EntityMap; -use repr::entities::{Ebb, Inst, Value, NO_VALUE, ExpandedValue}; -use repr::instructions::InstructionData; -use repr::types::Type; +use ir::entities::{Ebb, Inst, Value, NO_VALUE, ExpandedValue}; +use ir::instructions::InstructionData; +use ir::types::Type; use std::ops::{Index, IndexMut}; use std::u16; @@ -57,7 +57,7 @@ impl DataFlowGraph { /// Get the type of a value. pub fn value_type(&self, v: Value) -> Type { - use repr::entities::ExpandedValue::*; + use ir::entities::ExpandedValue::*; match v.expand() { Direct(i) => self.insts[i].first_type(), Table(i) => { @@ -75,7 +75,7 @@ impl DataFlowGraph { /// This is either the instruction that defined it or the Ebb that has the value as an /// argument. pub fn value_def(&self, v: Value) -> ValueDef { - use repr::entities::ExpandedValue::*; + use ir::entities::ExpandedValue::*; match v.expand() { Direct(inst) => ValueDef::Res(inst, 0), Table(idx) => { @@ -339,8 +339,8 @@ impl EbbData { #[cfg(test)] mod tests { use super::*; - use repr::types; - use repr::instructions::{Opcode, InstructionData}; + use ir::types; + use ir::instructions::{Opcode, InstructionData}; #[test] fn make_inst() { diff --git a/cranelift/src/libcretonne/repr/entities.rs b/cranelift/src/libcretonne/ir/entities.rs similarity index 100% rename from cranelift/src/libcretonne/repr/entities.rs rename to cranelift/src/libcretonne/ir/entities.rs diff --git a/cranelift/src/libcretonne/repr/immediates.rs b/cranelift/src/libcretonne/ir/immediates.rs similarity index 100% rename from cranelift/src/libcretonne/repr/immediates.rs rename to cranelift/src/libcretonne/ir/immediates.rs diff --git a/cranelift/src/libcretonne/repr/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs similarity index 99% rename from cranelift/src/libcretonne/repr/instructions.rs rename to cranelift/src/libcretonne/ir/instructions.rs index 7ebb71bbf4..a6c7bd03f1 100644 --- a/cranelift/src/libcretonne/repr/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -10,10 +10,10 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::ops::{Deref, DerefMut}; -use repr::entities::*; -use repr::immediates::*; -use repr::condcodes::*; -use repr::types::{self, Type}; +use ir::entities::*; +use ir::immediates::*; +use ir::condcodes::*; +use ir::types::{self, Type}; // Include code generated by `meta/gen_instr.py`. This file contains: // @@ -560,7 +560,7 @@ mod tests { #[test] fn value_set() { - use repr::types::*; + use ir::types::*; let vts = ValueTypeSet { allow_scalars: true, diff --git a/cranelift/src/libcretonne/repr/layout.rs b/cranelift/src/libcretonne/ir/layout.rs similarity index 99% rename from cranelift/src/libcretonne/repr/layout.rs rename to cranelift/src/libcretonne/ir/layout.rs index 36ee2dacf8..a01d49ef53 100644 --- a/cranelift/src/libcretonne/repr/layout.rs +++ b/cranelift/src/libcretonne/ir/layout.rs @@ -5,7 +5,7 @@ use std::iter::{Iterator, IntoIterator}; use entity_map::{EntityMap, EntityRef}; -use repr::entities::{Ebb, NO_EBB, Inst, NO_INST}; +use ir::entities::{Ebb, NO_EBB, Inst, NO_INST}; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not /// contain definitions of instructions or EBBs, but depends on `Inst` and `Ebb` entity references @@ -242,7 +242,7 @@ impl<'a> Iterator for Insts<'a> { mod tests { use super::Layout; use entity_map::EntityRef; - use repr::entities::{Ebb, Inst}; + use ir::entities::{Ebb, Inst}; #[test] fn append_ebb() { diff --git a/cranelift/src/libcretonne/repr/mod.rs b/cranelift/src/libcretonne/ir/mod.rs similarity index 96% rename from cranelift/src/libcretonne/repr/mod.rs rename to cranelift/src/libcretonne/ir/mod.rs index a62527ccbe..fb93e9ec99 100644 --- a/cranelift/src/libcretonne/repr/mod.rs +++ b/cranelift/src/libcretonne/ir/mod.rs @@ -8,11 +8,11 @@ pub mod instructions; pub mod dfg; pub mod layout; -use repr::types::{FunctionName, Signature}; +use ir::types::{FunctionName, Signature}; use entity_map::EntityRef; -use repr::entities::{Ebb, NO_EBB, StackSlot}; -use repr::dfg::DataFlowGraph; -use repr::layout::Layout; +use ir::entities::{Ebb, NO_EBB, StackSlot}; +use ir::dfg::DataFlowGraph; +use ir::layout::Layout; use std::fmt::{self, Debug, Display, Formatter}; use std::ops::Index; diff --git a/cranelift/src/libcretonne/repr/types.rs b/cranelift/src/libcretonne/ir/types.rs similarity index 100% rename from cranelift/src/libcretonne/repr/types.rs rename to cranelift/src/libcretonne/ir/types.rs diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 0b1ac8678e..f81bfb5720 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -7,7 +7,7 @@ pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); -pub mod repr; +pub mod ir; pub mod write; pub mod cfg; diff --git a/cranelift/src/libcretonne/test_utils/make_inst.rs b/cranelift/src/libcretonne/test_utils/make_inst.rs index 00955a5be5..adc45cdb67 100644 --- a/cranelift/src/libcretonne/test_utils/make_inst.rs +++ b/cranelift/src/libcretonne/test_utils/make_inst.rs @@ -1,9 +1,9 @@ //! Helper functions for generating dummy instructions. -use repr::Function; -use repr::entities::{Ebb, Inst, NO_VALUE}; -use repr::instructions::{InstructionData, Opcode, VariableArgs, JumpData, BranchData}; -use repr::types; +use ir::Function; +use ir::entities::{Ebb, Inst, NO_VALUE}; +use ir::instructions::{InstructionData, Opcode, VariableArgs, JumpData, BranchData}; +use ir::types; pub fn jump(func: &mut Function, dest: Ebb) -> Inst { func.dfg.make_inst(InstructionData::Jump { diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index 5b66b81065..b42fdf02fd 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -5,9 +5,9 @@ //! `cretonne-reader` crate. use std::io::{self, Write}; -use repr::Function; -use repr::entities::{Inst, Ebb, Value}; -use repr::types::Type; +use ir::Function; +use ir::entities::{Inst, Ebb, Value}; +use ir::types::Type; pub type Result = io::Result<()>; @@ -182,7 +182,7 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { } // Then the operands, depending on format. - use repr::instructions::InstructionData::*; + use ir::instructions::InstructionData::*; match func.dfg[inst] { Nullary { .. } => writeln!(w, ""), Unary { arg, .. } => writeln!(w, " {}", arg), @@ -217,8 +217,8 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { mod tests { use super::*; use super::{needs_quotes, escaped}; - use repr::{Function, StackSlotData}; - use repr::types; + use ir::{Function, StackSlotData}; + use ir::types; #[test] fn quoting() { diff --git a/cranelift/src/libreader/lexer.rs b/cranelift/src/libreader/lexer.rs index 87c7482589..3803260c05 100644 --- a/cranelift/src/libreader/lexer.rs +++ b/cranelift/src/libreader/lexer.rs @@ -6,8 +6,8 @@ // ====--------------------------------------------------------------------------------------====// use std::str::CharIndices; -use cretonne::repr::types; -use cretonne::repr::entities::{Value, Ebb}; +use cretonne::ir::types; +use cretonne::ir::entities::{Value, Ebb}; /// The location of a `Token` or `Error`. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -373,8 +373,8 @@ impl<'a> Lexer<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne::repr::types; - use cretonne::repr::entities::{Value, Ebb}; + use cretonne::ir::types; + use cretonne::ir::entities::{Value, Ebb}; fn token<'a>(token: Token<'a>, line: usize) -> Option, LocatedError>> { Some(super::token(token, Location { line_number: line })) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 0c573558c7..e333c0d3e9 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -11,12 +11,12 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::u32; use lexer::{self, Lexer, Token}; -use cretonne::repr::types::{Type, VOID, FunctionName, Signature, ArgumentType, ArgumentExtension}; -use cretonne::repr::immediates::{Imm64, Ieee32, Ieee64}; -use cretonne::repr::entities::*; -use cretonne::repr::instructions::{Opcode, InstructionFormat, InstructionData, VariableArgs, +use cretonne::ir::types::{Type, VOID, FunctionName, Signature, ArgumentType, ArgumentExtension}; +use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; +use cretonne::ir::entities::*; +use cretonne::ir::instructions::{Opcode, InstructionFormat, InstructionData, VariableArgs, JumpData, BranchData, ReturnData}; -use cretonne::repr::{Function, StackSlotData}; +use cretonne::ir::{Function, StackSlotData}; pub use lexer::Location; @@ -1039,7 +1039,7 @@ impl<'a> Parser<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne::repr::types::{self, ArgumentType, ArgumentExtension}; + use cretonne::ir::types::{self, ArgumentType, ArgumentExtension}; #[test] fn argument_type() { diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs index dfa0113fee..9a08c0287e 100644 --- a/cranelift/src/tools/print_cfg.rs +++ b/cranelift/src/tools/print_cfg.rs @@ -6,9 +6,9 @@ use std::fs::File; use std::io::{Read, Write, stdout}; use CommandResult; -use cretonne::repr::Function; +use cretonne::ir::Function; use cretonne::cfg::ControlFlowGraph; -use cretonne::repr::instructions::InstructionData; +use cretonne::ir::instructions::InstructionData; use cton_reader::parser::Parser; pub fn run(files: Vec) -> CommandResult { From 8296e92ddc7c04591253023483aea3deb57b11f0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Jul 2016 10:06:51 -0700 Subject: [PATCH 0155/3084] Move entry_block() into Layout. The single entry block in a function is simply the first block in the layout. Remove the 'entry' keyword from the textual IL, the lexer and parser. --- cranelift/docs/cton_lexer.py | 2 +- cranelift/docs/example.cton | 2 +- cranelift/docs/langref.rst | 16 ++++++++-------- cranelift/src/libcretonne/cfg.rs | 12 +++++------- cranelift/src/libcretonne/ir/layout.rs | 9 +++++++++ cranelift/src/libcretonne/ir/mod.rs | 6 +----- cranelift/src/libreader/lexer.rs | 5 +---- cranelift/src/libreader/parser.rs | 14 +++----------- 8 files changed, 29 insertions(+), 37 deletions(-) diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py index 0139de337c..28a1be25b4 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/cton_lexer.py @@ -26,7 +26,7 @@ class CretonneLexer(RegexLexer): (r'[-+]?(\d+\.\d+([eE]\d+)?|[sq]NaN|Inf)', Number.Float), (r'[-+]?\d+', Number.Integer), # Reserved words. - (keywords('function', 'entry'), Keyword), + (keywords('function'), Keyword), # Known attributes. (keywords('align', 'aligntrap', 'uext', 'sext', 'inreg'), Name.Attribute), # Well known value types. diff --git a/cranelift/docs/example.cton b/cranelift/docs/example.cton index 190d3a25fd..cdfc61bf4b 100644 --- a/cranelift/docs/example.cton +++ b/cranelift/docs/example.cton @@ -1,7 +1,7 @@ function average(i32, i32) -> f32 { ss1 = stack_slot 8, align 4 ; Stack slot for ``sum``. -entry ebb1(v1: i32, v2: i32): +ebb1(v1: i32, v2: i32): v3 = f64const 0x0.0 stack_store v3, ss1 brz v2, ebb3 ; Handle count == 0. diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index c6e7b3ced6..848d140db8 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -41,9 +41,9 @@ that can be referenced inside the function. In the example above, the preamble declares a single local variable, ``ss1``. After the preamble follows the :term:`function body` which consists of -:term:`extended basic block`\s, one of which is marked as the :term:`entry -block`. Every EBB ends with a :term:`terminator instruction`, so execution can -never fall through to the next EBB without an explicit branch. +:term:`extended basic block`\s, the first of which is the :term:`entry block`. +Every EBB ends with a :term:`terminator instruction`, so execution can never +fall through to the next EBB without an explicit branch. A ``.cton`` file consists of a sequence of independent function definitions: @@ -395,7 +395,7 @@ This simple example illustrates direct function calls and signatures:: function gcd(i32 uext, i32 uext) -> i32 uext "C" { f1 = function divmod(i32 uext, i32 uext) -> i32 uext, i32 uext - entry ebb1(v1: i32, v2: i32): + ebb1(v1: i32, v2: i32): brz v2, ebb2 v3, v4 = call f1(v1, v2) br ebb1(v2, v4) @@ -625,7 +625,7 @@ A small example using heaps:: function vdup(i32, i32) { h1 = heap "main" - entry ebb1(v1: i32, v2: i32): + ebb1(v1: i32, v2: i32): v3 = heap_load.i32x4 h1, v1, 0 v4 = heap_addr h1, v2, 32 ; Shared range check for two stores. store v3, v4, 0 @@ -878,9 +878,9 @@ Glossary entry block The :term:`EBB` that is executed first in a function. Currently, a - Cretonne function must have exactly one entry block. The types of the - entry block arguments must match the types of arguments in the function - signature. + Cretonne function must have exactly one entry block which must be the + first block in the function. The types of the entry block arguments must + match the types of arguments in the function signature. stack slot A fixed size memory allocation in the current function's activation diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 970af69afd..964a07a572 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -104,11 +104,8 @@ impl ControlFlowGraph { &self.data[ebb].successors } - pub fn postorder_ebbs(&self) -> Vec { - if self.len() < 1 { - return Vec::new(); - } - let mut stack_a = vec![Ebb::with_number(0).unwrap()]; + pub fn postorder_ebbs(&self, entry: Ebb) -> Vec { + let mut stack_a = vec![entry]; let mut stack_b = Vec::new(); while stack_a.len() > 0 { let cur = stack_a.pop().unwrap(); @@ -282,7 +279,7 @@ mod tests { func.layout.append_inst(jmp_ebb2_ebb5, ebb2); let cfg = ControlFlowGraph::new(&func); - assert_eq!(cfg.postorder_ebbs(), + assert_eq!(cfg.postorder_ebbs(func.layout.entry_block().unwrap()), vec![ebb0, ebb2, ebb5, ebb4, ebb1, ebb3]); } @@ -309,6 +306,7 @@ mod tests { func.layout.append_inst(jmp_ebb2_ebb3, ebb2); let cfg = ControlFlowGraph::new(&func); - assert_eq!(cfg.postorder_ebbs(), vec![ebb0, ebb1, ebb2, ebb3]); + assert_eq!(cfg.postorder_ebbs(func.layout.entry_block().unwrap()), + vec![ebb0, ebb1, ebb2, ebb3]); } } diff --git a/cranelift/src/libcretonne/ir/layout.rs b/cranelift/src/libcretonne/ir/layout.rs index a01d49ef53..af21ee5234 100644 --- a/cranelift/src/libcretonne/ir/layout.rs +++ b/cranelift/src/libcretonne/ir/layout.rs @@ -108,6 +108,12 @@ impl Layout { next: self.first_ebb, } } + + /// Get the function's entry block. + /// This is simply the first EBB in the layout order. + pub fn entry_block(&self) -> Option { + self.first_ebb + } } #[derive(Clone, Debug, Default)] @@ -414,8 +420,11 @@ mod tests { let e0 = Ebb::new(0); let e1 = Ebb::new(1); + assert_eq!(layout.entry_block(), None); layout.append_ebb(e0); + assert_eq!(layout.entry_block(), Some(e0)); layout.append_ebb(e1); + assert_eq!(layout.entry_block(), Some(e0)); let i0 = Inst::new(0); let i1 = Inst::new(1); diff --git a/cranelift/src/libcretonne/ir/mod.rs b/cranelift/src/libcretonne/ir/mod.rs index fb93e9ec99..0d7e9da3a4 100644 --- a/cranelift/src/libcretonne/ir/mod.rs +++ b/cranelift/src/libcretonne/ir/mod.rs @@ -10,7 +10,7 @@ pub mod layout; use ir::types::{FunctionName, Signature}; use entity_map::EntityRef; -use ir::entities::{Ebb, NO_EBB, StackSlot}; +use ir::entities::StackSlot; use ir::dfg::DataFlowGraph; use ir::layout::Layout; use std::fmt::{self, Debug, Display, Formatter}; @@ -24,9 +24,6 @@ pub struct Function { /// Signature of this function. signature: Signature, - /// The entry block. - pub entry_block: Ebb, - /// Stack slots allocated in this function. stack_slots: Vec, @@ -43,7 +40,6 @@ impl Function { Function { name: name, signature: sig, - entry_block: NO_EBB, stack_slots: Vec::new(), dfg: DataFlowGraph::new(), layout: Layout::new(), diff --git a/cranelift/src/libreader/lexer.rs b/cranelift/src/libreader/lexer.rs index 3803260c05..61dd7feb9b 100644 --- a/cranelift/src/libreader/lexer.rs +++ b/cranelift/src/libreader/lexer.rs @@ -32,7 +32,6 @@ pub enum Token<'a> { Equal, // '=' Arrow, // '->' Function, // 'function' - Entry, // 'entry' Float(&'a str), // Floating point immediate Integer(&'a str), // Integer immediate Type(types::Type), // i32, f32, b32x4, ... @@ -270,7 +269,6 @@ impl<'a> Lexer<'a> { fn keyword(text: &str) -> Option> { match text { "function" => Some(Token::Function), - "entry" => Some(Token::Entry), _ => None, } } @@ -444,7 +442,7 @@ mod tests { #[test] fn lex_identifiers() { - let mut lex = Lexer::new("v0 v00 vx01 ebb1234567890 ebb5234567890 entry v1x vx1 vxvx4 \ + let mut lex = Lexer::new("v0 v00 vx01 ebb1234567890 ebb5234567890 v1x vx1 vxvx4 \ function0 function b1 i32x4 f32x5"); assert_eq!(lex.next(), token(Token::Value(Value::direct_with_number(0).unwrap()), 1)); @@ -453,7 +451,6 @@ mod tests { assert_eq!(lex.next(), token(Token::Ebb(Ebb::with_number(1234567890).unwrap()), 1)); assert_eq!(lex.next(), token(Token::Identifier("ebb5234567890"), 1)); - assert_eq!(lex.next(), token(Token::Entry, 1)); assert_eq!(lex.next(), token(Token::Identifier("v1x"), 1)); assert_eq!(lex.next(), token(Token::Value(Value::table_with_number(1).unwrap()), 1)); diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index e333c0d3e9..a961ccdf60 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -15,7 +15,7 @@ use cretonne::ir::types::{Type, VOID, FunctionName, Signature, ArgumentType, Arg use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::*; use cretonne::ir::instructions::{Opcode, InstructionFormat, InstructionData, VariableArgs, - JumpData, BranchData, ReturnData}; + JumpData, BranchData, ReturnData}; use cretonne::ir::{Function, StackSlotData}; pub use lexer::Location; @@ -578,22 +578,14 @@ impl<'a> Parser<'a> { // Parse an extended basic block, add contents to `ctx`. // // extended-basic-block ::= * ebb-header { instruction } - // ebb-header ::= ["entry"] Ebb(ebb) [ebb-args] ":" + // ebb-header ::= Ebb(ebb) [ebb-args] ":" // fn parse_extended_basic_block(&mut self, ctx: &mut Context) -> Result<()> { - let is_entry = self.optional(Token::Entry); let ebb_num = try!(self.match_ebb("expected EBB header")); let ebb = try!(ctx.add_ebb(ebb_num, &self.loc)); - if is_entry { - if ctx.function.entry_block != NO_EBB { - return err!(self.loc, "multiple entry blocks in function"); - } - ctx.function.entry_block = ebb; - } - if !self.optional(Token::Colon) { - // ebb-header ::= ["entry"] Ebb(ebb) [ * ebb-args ] ":" + // ebb-header ::= Ebb(ebb) [ * ebb-args ] ":" try!(self.parse_ebb_args(ctx, ebb)); try!(self.match_token(Token::Colon, "expected ':' after EBB arguments")); } From ba76f444abccc720d327f7bffd4f7025572d52e7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Jul 2016 14:10:39 -0700 Subject: [PATCH 0156/3084] Add a keys() iterator to EntityMap. --- cranelift/src/libcretonne/entity_map.rs | 42 ++++++++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-) diff --git a/cranelift/src/libcretonne/entity_map.rs b/cranelift/src/libcretonne/entity_map.rs index 0a7faccb41..9ce39bec19 100644 --- a/cranelift/src/libcretonne/entity_map.rs +++ b/cranelift/src/libcretonne/entity_map.rs @@ -81,6 +81,15 @@ impl EntityMap pub fn len(&self) -> usize { self.elems.len() } + + /// Iterate over all the keys in this map. + pub fn keys(&self) -> Keys { + Keys { + pos: 0, + len: self.elems.len(), + unused: PhantomData, + } + } } /// Additional methods for value types that implement `Clone` and `Default`. @@ -127,12 +136,37 @@ impl IndexMut for EntityMap } } +/// Iterate over all keys in order. +pub struct Keys + where K: EntityRef +{ + pos: usize, + len: usize, + unused: PhantomData, +} + +impl Iterator for Keys + where K: EntityRef +{ + type Item = K; + + fn next(&mut self) -> Option { + if self.pos < self.len { + let k = K::new(self.pos); + self.pos += 1; + Some(k) + } else { + None + } + } +} + #[cfg(test)] mod tests { use super::*; // EntityRef impl for testing. - #[derive(Clone, Copy, PartialEq, Eq)] + #[derive(Clone, Copy, Debug, PartialEq, Eq)] struct E(u32); impl EntityRef for E { @@ -151,6 +185,9 @@ mod tests { let r2 = E(2); let mut m = EntityMap::new(); + let v: Vec = m.keys().collect(); + assert_eq!(v, []); + assert!(!m.is_valid(r0)); m.ensure(r2); m[r2] = 3; @@ -160,6 +197,9 @@ mod tests { assert_eq!(m[r1], 5); assert_eq!(m[r2], 3); + let v: Vec = m.keys().collect(); + assert_eq!(v, [r0, r1, r2]); + let shared = &m; assert_eq!(shared[r0], 0); assert_eq!(shared[r1], 5); From f054d32f50eb21c7a0d098b6c42be5d888ea2aca Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Jul 2016 11:41:30 -0700 Subject: [PATCH 0157/3084] Implement jump tables. - Add a ir::jumptable module with a JumpTableData struct representing the vector of destinations. - Add an entity map of jump tables to the Function. - Parse and write jump tables in the function preamble. - Rewrite EBB references in jumptables after parsing. --- cranelift/src/libcretonne/ir/jumptable.rs | 156 ++++++++++++++++++++++ cranelift/src/libcretonne/ir/mod.rs | 10 +- cranelift/src/libcretonne/write.rs | 5 + cranelift/src/libreader/lexer.rs | 2 + cranelift/src/libreader/parser.rs | 82 +++++++++++- cranelift/tests/parser/branch.cton | 14 ++ cranelift/tests/parser/branch.cton.ref | 17 +++ 7 files changed, 281 insertions(+), 5 deletions(-) create mode 100644 cranelift/src/libcretonne/ir/jumptable.rs diff --git a/cranelift/src/libcretonne/ir/jumptable.rs b/cranelift/src/libcretonne/ir/jumptable.rs new file mode 100644 index 0000000000..31ea86f643 --- /dev/null +++ b/cranelift/src/libcretonne/ir/jumptable.rs @@ -0,0 +1,156 @@ +//! Jump table representation. +//! +//! Jump tables are declared in the preamble and assigned an `ir::entities::JumpTable` reference. +//! The actual table of destinations is stored in a `JumpTableData` struct defined in this module. + +use ir::entities::{Ebb, NO_EBB}; +use std::iter; +use std::slice; +use std::fmt::{self, Display, Formatter}; + +/// Contents of a jump table. +/// +/// All jump tables use 0-based indexing and are expected to be densely populated. They don't need +/// to be completely populated, though. Individual entries can be missing. +pub struct JumpTableData { + // Table entries, using NO_EBB as a placeholder for missing entries. + table: Vec, + + // How many `NO_EBB` holes in table? + holes: usize, +} + +impl JumpTableData { + /// Create a new empty jump table. + pub fn new() -> JumpTableData { + JumpTableData { + table: Vec::new(), + holes: 0, + } + } + + /// Set a table entry. + /// + /// The table will grow as needed to fit 'idx'. + pub fn set_entry(&mut self, idx: usize, dest: Ebb) { + assert!(dest != NO_EBB); + // Resize table to fit `idx`. + if idx >= self.table.len() { + self.holes += idx - self.table.len(); + self.table.resize(idx + 1, NO_EBB); + } else if self.table[idx] == NO_EBB { + // We're filling in an existing hole. + self.holes -= 1; + } + self.table[idx] = dest; + } + + /// Clear a table entry. + /// + /// The `br_table` instruction will fall through if given an index corresponding to a cleared + /// table entry. + pub fn clear_entry(&mut self, idx: usize) { + if idx < self.table.len() && self.table[idx] != NO_EBB { + self.holes += 1; + self.table[idx] = NO_EBB; + } + } + + /// Get the entry for `idx`, or `None`. + pub fn get_entry(&self, idx: usize) -> Option { + if idx < self.table.len() && self.table[idx] != NO_EBB { + Some(self.table[idx]) + } else { + None + } + } + + /// Enumerate over all `(idx, dest)` pairs in the table in order. + /// + /// This returns an iterator that skips any empty slots in the table. + pub fn entries<'a>(&'a self) -> Entries { + Entries(self.table.iter().cloned().enumerate()) + } + + /// Access the whole table as a mutable slice. + pub fn as_mut_slice(&mut self) -> &mut [Ebb] { + self.table.as_mut_slice() + } +} + +/// Enumerate `(idx, dest)` pairs in order. +pub struct Entries<'a>(iter::Enumerate>>); + +impl<'a> Iterator for Entries<'a> { + type Item = (usize, Ebb); + + fn next(&mut self) -> Option { + loop { + if let Some((idx, dest)) = self.0.next() { + if dest != NO_EBB { + return Some((idx, dest)); + } + } else { + return None; + } + } + } +} + +impl Display for JumpTableData { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + let first = self.table.first().cloned().unwrap_or_default(); + if first == NO_EBB { + try!(write!(fmt, "jump_table 0")); + } else { + try!(write!(fmt, "jump_table {}", first)); + } + + for dest in self.table.iter().cloned().skip(1) { + if dest == NO_EBB { + try!(write!(fmt, ", 0")); + } else { + try!(write!(fmt, ", {}", dest)); + } + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::JumpTableData; + use ir::entities::Ebb; + use entity_map::EntityRef; + + #[test] + fn empty() { + let jt = JumpTableData::new(); + + assert_eq!(jt.get_entry(0), None); + assert_eq!(jt.get_entry(10), None); + + assert_eq!(jt.to_string(), "jump_table 0"); + + let v: Vec<(usize, Ebb)> = jt.entries().collect(); + assert_eq!(v, []); + } + + #[test] + fn insert() { + let e1 = Ebb::new(1); + let e2 = Ebb::new(2); + + let mut jt = JumpTableData::new(); + + jt.set_entry(0, e1); + jt.set_entry(0, e2); + jt.set_entry(10, e1); + + assert_eq!(jt.to_string(), + "jump_table ebb2, 0, 0, 0, 0, 0, 0, 0, 0, 0, ebb1"); + + let v: Vec<(usize, Ebb)> = jt.entries().collect(); + assert_eq!(v, [(0, e2), (10, e1)]); + } +} diff --git a/cranelift/src/libcretonne/ir/mod.rs b/cranelift/src/libcretonne/ir/mod.rs index 0d7e9da3a4..8d5ed99742 100644 --- a/cranelift/src/libcretonne/ir/mod.rs +++ b/cranelift/src/libcretonne/ir/mod.rs @@ -5,12 +5,14 @@ pub mod types; pub mod condcodes; pub mod immediates; pub mod instructions; +pub mod jumptable; pub mod dfg; pub mod layout; use ir::types::{FunctionName, Signature}; -use entity_map::EntityRef; -use ir::entities::StackSlot; +use entity_map::{EntityRef, EntityMap}; +use ir::entities::{StackSlot, JumpTable}; +use ir::jumptable::JumpTableData; use ir::dfg::DataFlowGraph; use ir::layout::Layout; use std::fmt::{self, Debug, Display, Formatter}; @@ -27,6 +29,9 @@ pub struct Function { /// Stack slots allocated in this function. stack_slots: Vec, + /// Jump tables used in this function. + pub jump_tables: EntityMap, + /// Data flow graph containing the primary definition of all instructions, EBBs and values. pub dfg: DataFlowGraph, @@ -41,6 +46,7 @@ impl Function { name: name, signature: sig, stack_slots: Vec::new(), + jump_tables: EntityMap::new(), dfg: DataFlowGraph::new(), layout: Layout::new(), } diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index b42fdf02fd..e7b35e806a 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -75,6 +75,11 @@ fn write_preamble(w: &mut Write, func: &Function) -> io::Result { try!(writeln!(w, " {} = {}", ss, func[ss])); } + for jt in func.jump_tables.keys() { + any = true; + try!(writeln!(w, " {} = {}", jt, func.jump_tables[jt])); + } + Ok(any) } diff --git a/cranelift/src/libreader/lexer.rs b/cranelift/src/libreader/lexer.rs index 61dd7feb9b..a19c14adfd 100644 --- a/cranelift/src/libreader/lexer.rs +++ b/cranelift/src/libreader/lexer.rs @@ -38,6 +38,7 @@ pub enum Token<'a> { Value(Value), // v12, vx7 Ebb(Ebb), // ebb3 StackSlot(u32), // ss3 + JumpTable(u32), // jt2 Identifier(&'a str), // Unrecognized identifier (opcode, enumerator, ...) } @@ -291,6 +292,7 @@ impl<'a> Lexer<'a> { "vx" => Value::table_with_number(value).map(|v| Token::Value(v)), "ebb" => Ebb::with_number(value).map(|ebb| Token::Ebb(ebb)), "ss" => Some(Token::StackSlot(value)), + "jt" => Some(Token::JumpTable(value)), _ => None, } } diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index a961ccdf60..f55604f32a 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -17,6 +17,7 @@ use cretonne::ir::entities::*; use cretonne::ir::instructions::{Opcode, InstructionFormat, InstructionData, VariableArgs, JumpData, BranchData, ReturnData}; use cretonne::ir::{Function, StackSlotData}; +use cretonne::ir::jumptable::JumpTableData; pub use lexer::Location; @@ -71,6 +72,7 @@ pub struct Parser<'a> { struct Context { function: Function, stack_slots: HashMap, // ssNN + jump_tables: HashMap, // jtNN ebbs: HashMap, // ebbNN values: HashMap, // vNN, vxNN @@ -83,6 +85,7 @@ impl Context { Context { function: f, stack_slots: HashMap::new(), + jump_tables: HashMap::new(), ebbs: HashMap::new(), values: HashMap::new(), inst_locs: Vec::new(), @@ -98,6 +101,15 @@ impl Context { } } + // Allocate a new jump table and add a mapping number -> JumpTable. + fn add_jt(&mut self, number: u32, data: JumpTableData, loc: &Location) -> Result<()> { + if self.jump_tables.insert(number, self.function.jump_tables.push(data)).is_some() { + err!(loc, "duplicate jump table: jt{}", number) + } else { + Ok(()) + } + } + // Allocate a new EBB and add a mapping src_ebb -> Ebb. fn add_ebb(&mut self, src_ebb: Ebb, loc: &Location) -> Result { let ebb = self.function.dfg.make_ebb(); @@ -211,8 +223,15 @@ impl Context { } } - // TODO: Rewrite EBB references in jump tables. (Once jump table data structures are - // defined). + // Rewrite EBB references in jump tables. + let loc = Location { line_number: 0 }; + for jt in self.function.jump_tables.keys() { + for ebb in self.function.jump_tables[jt].as_mut_slice() { + if *ebb != NO_EBB { + try!(Self::rewrite_ebb(&self.ebbs, ebb, &loc)); + } + } + } Ok(()) } @@ -312,6 +331,16 @@ impl<'a> Parser<'a> { } } + // Match and consume a jump table reference. + fn match_jt(&mut self, err_msg: &str) -> Result { + if let Some(Token::JumpTable(jt)) = self.token() { + self.consume(); + Ok(jt) + } else { + err!(self.loc, err_msg) + } + } + // Match and consume an ebb reference. fn match_ebb(&mut self, err_msg: &str) -> Result { if let Some(Token::Ebb(ebb)) = self.token() { @@ -530,6 +559,7 @@ impl<'a> Parser<'a> { // preamble-decl ::= * stack-slot-decl // * function-decl // * signature-decl + // * jump-table-decl // // The parsed decls are added to `ctx` rather than returned. fn parse_preamble(&mut self, ctx: &mut Context) -> Result<()> { @@ -539,13 +569,17 @@ impl<'a> Parser<'a> { self.parse_stack_slot_decl() .and_then(|(num, dat)| ctx.add_ss(num, dat, &self.loc)) } + Some(Token::JumpTable(..)) => { + self.parse_jump_table_decl() + .and_then(|(num, dat)| ctx.add_jt(num, dat, &self.loc)) + } // More to come.. _ => return Ok(()), }); } } - // Parse a stack slot decl, add to `func`. + // Parse a stack slot decl. // // stack-slot-decl ::= * StackSlot(ss) "=" "stack_slot" Bytes {"," stack-slot-flag} fn parse_stack_slot_decl(&mut self) -> Result<(u32, StackSlotData)> { @@ -564,6 +598,48 @@ impl<'a> Parser<'a> { Ok((number, data)) } + // Parse a jump table decl. + // + // jump-table-decl ::= * JumpTable(jt) "=" "jump_table" jt-entry {"," jt-entry} + fn parse_jump_table_decl(&mut self) -> Result<(u32, JumpTableData)> { + let number = try!(self.match_jt("expected jump table number: jt«n»")); + try!(self.match_token(Token::Equal, "expected '=' in jump_table decl")); + try!(self.match_identifier("jump_table", "expected 'jump_table'")); + + let mut data = JumpTableData::new(); + + // jump-table-decl ::= JumpTable(jt) "=" "jump_table" * jt-entry {"," jt-entry} + for idx in 0usize.. { + if let Some(dest) = try!(self.parse_jump_table_entry()) { + data.set_entry(idx, dest); + } + if !self.optional(Token::Comma) { + return Ok((number, data)); + } + } + + err!(self.loc, "jump_table too long") + } + + // jt-entry ::= * Ebb(dest) | "0" + fn parse_jump_table_entry(&mut self) -> Result> { + match self.token() { + Some(Token::Integer(s)) => { + if s == "0" { + self.consume(); + Ok(None) + } else { + err!(self.loc, "invalid jump_table entry '{}'", s) + } + } + Some(Token::Ebb(dest)) => { + self.consume(); + Ok(Some(dest)) + } + _ => err!(self.loc, "expected jump_table entry"), + } + } + // Parse a function body, add contents to `ctx`. // // function-body ::= * { extended-basic-block } diff --git a/cranelift/tests/parser/branch.cton b/cranelift/tests/parser/branch.cton index bf49ba8a9f..148b59325c 100644 --- a/cranelift/tests/parser/branch.cton +++ b/cranelift/tests/parser/branch.cton @@ -43,3 +43,17 @@ ebb0(vx0: i32, vx1: f32): ebb1(vx2: i32, vx3: f32): brnz vx0, ebb0(vx2, vx3) } + +function jumptable() { + jt200 = jump_table 0, 0 + jt2 = jump_table 0, 0, ebb10, ebb40, ebb20, ebb30 + +ebb10: + trap +ebb20: + trap +ebb30: + trap +ebb40: + trap +} diff --git a/cranelift/tests/parser/branch.cton.ref b/cranelift/tests/parser/branch.cton.ref index 02444326b1..c640a6b793 100644 --- a/cranelift/tests/parser/branch.cton.ref +++ b/cranelift/tests/parser/branch.cton.ref @@ -37,3 +37,20 @@ ebb0(vx0: i32, vx1: f32): ebb1(vx2: i32, vx3: f32): brnz vx0, ebb0(vx2, vx3) } + +function jumptable() { + jt0 = jump_table 0 + jt1 = jump_table 0, 0, ebb0, ebb3, ebb1, ebb2 + +ebb0: + trap + +ebb1: + trap + +ebb2: + trap + +ebb3: + trap +} From 28a5f007c4f9740279ea2755ea8d7a3ae8b18ea5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Jul 2016 15:16:14 -0700 Subject: [PATCH 0158/3084] Parse the BranchTable instruction format. Resolve the jump table reference immediately since all jump tables are declared in the preamble. --- cranelift/src/libreader/parser.rs | 30 ++++++++++++++++++++------ cranelift/tests/parser/branch.cton | 5 +++-- cranelift/tests/parser/branch.cton.ref | 5 +++-- 3 files changed, 30 insertions(+), 10 deletions(-) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index f55604f32a..41742b54e7 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -110,6 +110,14 @@ impl Context { } } + // Resolve a reference to a jump table. + fn get_jt(&self, number: u32, loc: &Location) -> Result { + match self.jump_tables.get(&number) { + Some(&jt) => Ok(jt), + None => err!(loc, "undefined jump table jt{}", number), + } + } + // Allocate a new EBB and add a mapping src_ebb -> Ebb. fn add_ebb(&mut self, src_ebb: Ebb, loc: &Location) -> Result { let ebb = self.function.dfg.make_ebb(); @@ -332,12 +340,12 @@ impl<'a> Parser<'a> { } // Match and consume a jump table reference. - fn match_jt(&mut self, err_msg: &str) -> Result { + fn match_jt(&mut self) -> Result { if let Some(Token::JumpTable(jt)) = self.token() { self.consume(); Ok(jt) } else { - err!(self.loc, err_msg) + err!(self.loc, "expected jump table number: jt«n»") } } @@ -602,7 +610,7 @@ impl<'a> Parser<'a> { // // jump-table-decl ::= * JumpTable(jt) "=" "jump_table" jt-entry {"," jt-entry} fn parse_jump_table_decl(&mut self) -> Result<(u32, JumpTableData)> { - let number = try!(self.match_jt("expected jump table number: jt«n»")); + let number = try!(self.match_jt()); try!(self.match_token(Token::Equal, "expected '=' in jump_table decl")); try!(self.match_identifier("jump_table", "expected 'jump_table'")); @@ -763,7 +771,7 @@ impl<'a> Parser<'a> { }; // instruction ::= [inst-results "="] Opcode(opc) ["." Type] * ... - let inst_data = try!(self.parse_inst_operands(opcode)); + let inst_data = try!(self.parse_inst_operands(ctx, opcode)); // We're done parsing the instruction now. // @@ -914,7 +922,7 @@ impl<'a> Parser<'a> { // Parse the operands following the instruction opcode. // This depends on the format of the opcode. - fn parse_inst_operands(&mut self, opcode: Opcode) -> Result { + fn parse_inst_operands(&mut self, ctx: &Context, opcode: Opcode) -> Result { Ok(match opcode.format().unwrap() { InstructionFormat::Nullary => { InstructionData::Nullary { @@ -1096,7 +1104,17 @@ impl<'a> Parser<'a> { data: Box::new(ReturnData { args: args }), } } - InstructionFormat::BranchTable | + InstructionFormat::BranchTable => { + let arg = try!(self.match_value("expected SSA value operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let table = try!(self.match_jt().and_then(|num| ctx.get_jt(num, &self.loc))); + InstructionData::BranchTable { + opcode: opcode, + ty: VOID, + arg: arg, + table: table, + } + } InstructionFormat::Call => { unimplemented!(); } diff --git a/cranelift/tests/parser/branch.cton b/cranelift/tests/parser/branch.cton index 148b59325c..05526fd910 100644 --- a/cranelift/tests/parser/branch.cton +++ b/cranelift/tests/parser/branch.cton @@ -44,11 +44,12 @@ ebb1(vx2: i32, vx3: f32): brnz vx0, ebb0(vx2, vx3) } -function jumptable() { +function jumptable(i32) { jt200 = jump_table 0, 0 jt2 = jump_table 0, 0, ebb10, ebb40, ebb20, ebb30 -ebb10: +ebb10(v3: i32): + br_table v3, jt2 trap ebb20: trap diff --git a/cranelift/tests/parser/branch.cton.ref b/cranelift/tests/parser/branch.cton.ref index c640a6b793..8f395c885c 100644 --- a/cranelift/tests/parser/branch.cton.ref +++ b/cranelift/tests/parser/branch.cton.ref @@ -38,11 +38,12 @@ ebb1(vx2: i32, vx3: f32): brnz vx0, ebb0(vx2, vx3) } -function jumptable() { +function jumptable(i32) { jt0 = jump_table 0 jt1 = jump_table 0, 0, ebb0, ebb3, ebb1, ebb2 -ebb0: +ebb0(vx0: i32): + br_table vx0, jt1 trap ebb1: From 8d93fe9685b42118a6b4ecb1d3cea99177db0f99 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Jul 2016 15:38:53 -0700 Subject: [PATCH 0159/3084] Add an analyze_branch method to InstructionData. Rather than switching on instruction formats to discover the destination of a branch, use the analyze_branch method which returns a BranchInfo enum with just the relevant information. This makes CFG algorithms independent of future instruction formats for branches. Only analyze_branch needs to be updated when adding a new format. --- cranelift/src/libcretonne/cfg.rs | 20 ++++++----- cranelift/src/libcretonne/ir/instructions.rs | 36 ++++++++++++++++++++ 2 files changed, 47 insertions(+), 9 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 964a07a572..e221f55fcd 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -24,7 +24,7 @@ use ir::Function; use ir::entities::{Inst, Ebb}; -use ir::instructions::InstructionData; +use ir::instructions::BranchInfo; use entity_map::EntityMap; /// A basic block denoted by its enclosing Ebb and last instruction. @@ -68,16 +68,18 @@ impl ControlFlowGraph { for ebb in &func.layout { for inst in func.layout.ebb_insts(ebb) { - match func.dfg[inst] { - InstructionData::Branch { ty: _, opcode: _, ref data } => { - cfg.add_successor(ebb, data.destination); - cfg.add_predecessor(data.destination, (ebb, inst)); + match func.dfg[inst].analyze_branch() { + BranchInfo::SingleDest(dest, _) => { + cfg.add_successor(ebb, dest); + cfg.add_predecessor(dest, (ebb, inst)); } - InstructionData::Jump { ty: _, opcode: _, ref data } => { - cfg.add_successor(ebb, data.destination); - cfg.add_predecessor(data.destination, (ebb, inst)); + BranchInfo::Table(jt) => { + for (_, dest) in func.jump_tables[jt].entries() { + cfg.add_successor(ebb, dest); + cfg.add_predecessor(dest, (ebb, inst)); + } } - _ => (), + BranchInfo::NotABranch => {} } } } diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index a6c7bd03f1..4087e246d5 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -344,6 +344,42 @@ impl InstructionData { } } +/// Analyzing an instruction. +/// +/// Avoid large matches on instruction formats by using the methods efined here to examine +/// instructions. +impl InstructionData { + /// Return information about the destination of a branch or jump instruction. + /// + /// Any instruction that can transfer control to another EBB reveals its possible destinations + /// here. + pub fn analyze_branch<'a>(&'a self) -> BranchInfo<'a> { + match self { + &InstructionData::Jump { ref data, .. } => { + BranchInfo::SingleDest(data.destination, &data.arguments) + } + &InstructionData::Branch { ref data, .. } => { + BranchInfo::SingleDest(data.destination, &data.arguments) + } + &InstructionData::BranchTable { table, .. } => BranchInfo::Table(table), + _ => BranchInfo::NotABranch, + } + } +} + +/// Information about branch and jump instructions. +pub enum BranchInfo<'a> { + /// This is not a branch or jump instruction. + /// This instruction will not transfer control to another EBB in the function, but it may still + /// affect control flow by returning or trapping. + NotABranch, + + /// This is a branch or jump to a single destination EBB, possibly taking value arguments. + SingleDest(Ebb, &'a [Value]), + + /// This is a jump table branch which can have many destination EBBs. + Table(JumpTable), +} /// Value type constraints for a given opcode. /// From aa730ec2c42dd9950fbaebb4af3a098cd7d550a9 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Mon, 25 Jul 2016 00:41:34 -0700 Subject: [PATCH 0160/3084] make postorder_ebbs into actually reverse_postorder_ebbs. This is a more accurate description. Further return the ebbs in a btreemap so that the index of each Ebb (its order of visitation) is quick to lookup. --- cranelift/src/libcretonne/cfg.rs | 112 ++++++++++++++++++++++++------- 1 file changed, 89 insertions(+), 23 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index e221f55fcd..c853d81831 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -24,8 +24,9 @@ use ir::Function; use ir::entities::{Inst, Ebb}; -use ir::instructions::BranchInfo; +use ir::instructions::InstructionData; use entity_map::EntityMap; +use std::collections::{HashSet, BTreeMap}; /// A basic block denoted by its enclosing Ebb and last instruction. pub type BasicBlock = (Ebb, Inst); @@ -51,6 +52,7 @@ impl CFGNode { /// extended basic blocks. #[derive(Debug)] pub struct ControlFlowGraph { + entry_block: Option, data: EntityMap, } @@ -58,7 +60,11 @@ impl ControlFlowGraph { /// During initialization mappings will be generated for any existing /// blocks within the CFG's associated function. pub fn new(func: &Function) -> ControlFlowGraph { - let mut cfg = ControlFlowGraph { data: EntityMap::new() }; + + let mut cfg = ControlFlowGraph { + data: EntityMap::new(), + entry_block: func.layout.entry_block(), + }; // Even ebbs without predecessors should show up in the CFG, albeit // with no entires. @@ -68,18 +74,16 @@ impl ControlFlowGraph { for ebb in &func.layout { for inst in func.layout.ebb_insts(ebb) { - match func.dfg[inst].analyze_branch() { - BranchInfo::SingleDest(dest, _) => { - cfg.add_successor(ebb, dest); - cfg.add_predecessor(dest, (ebb, inst)); + match func.dfg[inst] { + InstructionData::Branch { ty: _, opcode: _, ref data } => { + cfg.add_successor(ebb, data.destination); + cfg.add_predecessor(data.destination, (ebb, inst)); } - BranchInfo::Table(jt) => { - for (_, dest) in func.jump_tables[jt].entries() { - cfg.add_successor(ebb, dest); - cfg.add_predecessor(dest, (ebb, inst)); - } + InstructionData::Jump { ty: _, opcode: _, ref data } => { + cfg.add_successor(ebb, data.destination); + cfg.add_predecessor(data.destination, (ebb, inst)); } - BranchInfo::NotABranch => {} + _ => (), } } } @@ -106,19 +110,30 @@ impl ControlFlowGraph { &self.data[ebb].successors } - pub fn postorder_ebbs(&self, entry: Ebb) -> Vec { - let mut stack_a = vec![entry]; - let mut stack_b = Vec::new(); + /// Return ebbs in reverse postorder along with a mapping of + /// the ebb to its order of visitation. + pub fn reverse_postorder_ebbs(&self) -> BTreeMap { + let entry_block = match self.entry_block { + None => { + return BTreeMap::new(); + } + Some(eb) => eb, + }; + let mut seen = HashSet::new(); + let mut stack_a = vec![entry_block]; + let mut finished = BTreeMap::new(); while stack_a.len() > 0 { let cur = stack_a.pop().unwrap(); for child in &self.data[cur].successors { - if *child != cur && !stack_a.contains(child) { + if *child != cur && !seen.contains(&child) { + seen.insert(child); stack_a.push(child.clone()); } } - stack_b.push(cur); + let index = finished.len(); + finished.insert(cur, index); } - stack_b + finished } pub fn len(&self) -> usize { @@ -281,12 +296,15 @@ mod tests { func.layout.append_inst(jmp_ebb2_ebb5, ebb2); let cfg = ControlFlowGraph::new(&func); - assert_eq!(cfg.postorder_ebbs(func.layout.entry_block().unwrap()), - vec![ebb0, ebb2, ebb5, ebb4, ebb1, ebb3]); + let mut postorder = vec![ebb3, ebb1, ebb4, ebb5, ebb2, ebb0]; + postorder.reverse(); + for (ebb, key) in cfg.reverse_postorder_ebbs() { + assert_eq!(ebb, postorder[key]); + } } #[test] - fn loop_edge() { + fn loops_one() { let mut func = Function::new(); let ebb0 = func.dfg.make_ebb(); let ebb1 = func.dfg.make_ebb(); @@ -308,7 +326,55 @@ mod tests { func.layout.append_inst(jmp_ebb2_ebb3, ebb2); let cfg = ControlFlowGraph::new(&func); - assert_eq!(cfg.postorder_ebbs(func.layout.entry_block().unwrap()), - vec![ebb0, ebb1, ebb2, ebb3]); + let mut postorder = vec![ebb3, ebb2, ebb1, ebb0]; + postorder.reverse(); + for (ebb, key) in cfg.reverse_postorder_ebbs() { + assert_eq!(ebb, postorder[key]); + } + } + + #[test] + fn loops_two() { + let mut func = Function::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + let ebb2 = func.dfg.make_ebb(); + let ebb3 = func.dfg.make_ebb(); + let ebb4 = func.dfg.make_ebb(); + let ebb5 = func.dfg.make_ebb(); + + func.layout.append_ebb(ebb0); + func.layout.append_ebb(ebb1); + func.layout.append_ebb(ebb2); + func.layout.append_ebb(ebb3); + func.layout.append_ebb(ebb4); + func.layout.append_ebb(ebb5); + + let jmp_ebb0_ebb1 = make_inst::jump(&mut func, ebb1); + let jmp_ebb0_ebb2 = make_inst::jump(&mut func, ebb2); + let jmp_ebb1_ebb3 = make_inst::jump(&mut func, ebb3); + let br_ebb2_ebb4 = make_inst::jump(&mut func, ebb4); + let jmp_ebb2_ebb5 = make_inst::jump(&mut func, ebb5); + let jmp_ebb3_ebb4 = make_inst::jump(&mut func, ebb4); + let br_ebb4_ebb3 = make_inst::branch(&mut func, ebb3); + let jmp_ebb4_ebb5 = make_inst::jump(&mut func, ebb5); + let jmp_ebb5_ebb4 = make_inst::jump(&mut func, ebb4); + + func.layout.append_inst(jmp_ebb0_ebb1, ebb0); + func.layout.append_inst(jmp_ebb0_ebb2, ebb0); + func.layout.append_inst(jmp_ebb1_ebb3, ebb1); + func.layout.append_inst(br_ebb2_ebb4, ebb2); + func.layout.append_inst(jmp_ebb2_ebb5, ebb2); + func.layout.append_inst(jmp_ebb3_ebb4, ebb3); + func.layout.append_inst(br_ebb4_ebb3, ebb4); + func.layout.append_inst(jmp_ebb4_ebb5, ebb4); + func.layout.append_inst(jmp_ebb5_ebb4, ebb5); + + let cfg = ControlFlowGraph::new(&func); + let mut postorder = vec![ebb1, ebb3, ebb4, ebb5, ebb2, ebb0]; + postorder.reverse(); + for (ebb, key) in cfg.reverse_postorder_ebbs() { + assert_eq!(ebb, postorder[key]); + } } } From 26c350833a7191d587767fc38d2836dac6546b72 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Mon, 25 Jul 2016 01:05:15 -0700 Subject: [PATCH 0161/3084] Change variable name to something more descriptive. --- cranelift/src/libcretonne/cfg.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index c853d81831..696c364af2 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -120,14 +120,14 @@ impl ControlFlowGraph { Some(eb) => eb, }; let mut seen = HashSet::new(); - let mut stack_a = vec![entry_block]; + let mut open_nodes = vec![entry_block]; let mut finished = BTreeMap::new(); - while stack_a.len() > 0 { - let cur = stack_a.pop().unwrap(); + while open_nodes.len() > 0 { + let cur = open_nodes.pop().unwrap(); for child in &self.data[cur].successors { if *child != cur && !seen.contains(&child) { seen.insert(child); - stack_a.push(child.clone()); + open_nodes.push(child.clone()); } } let index = finished.len(); From d137529bd5ca7afbc3b94abdde2f338ef2c89899 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Mon, 25 Jul 2016 18:49:39 -0700 Subject: [PATCH 0162/3084] Use cton_reader to simplify cfg traversal tests. --- cranelift/src/libcretonne/tests/cfg.rs | 85 ++++++++++++++++++++++++++ cranelift/src/libcretonne/tests/lib.rs | 1 + 2 files changed, 86 insertions(+) create mode 100644 cranelift/src/libcretonne/tests/cfg.rs create mode 100644 cranelift/src/libcretonne/tests/lib.rs diff --git a/cranelift/src/libcretonne/tests/cfg.rs b/cranelift/src/libcretonne/tests/cfg.rs new file mode 100644 index 0000000000..6ad024aefd --- /dev/null +++ b/cranelift/src/libcretonne/tests/cfg.rs @@ -0,0 +1,85 @@ +extern crate cretonne; +extern crate cton_reader; + +use self::cton_reader::parser::Parser; +use self::cretonne::ir::entities::Ebb; +use self::cretonne::cfg::ControlFlowGraph; + +fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) { + let func = &Parser::parse(function_source).unwrap()[0]; + let cfg = ControlFlowGraph::new(&func); + let ebbs = ebb_order.iter().map(|n| Ebb::with_number(*n).unwrap()) + .collect::>(); + for (ebb, key) in cfg.reverse_postorder_ebbs() { + assert_eq!(ebb, ebbs[key]); + } +} + +#[test] +fn simple_traversal() { + test_reverse_postorder_traversal(" + function test(i32) { + ebb0(v0: i32): + brz v0, ebb1 + jump ebb2 + ebb1: + jump ebb3 + ebb2: + v1 = iconst.i32 1 + v2 = iadd v1, v0 + brz v2, ebb2 + v3 = iadd v1, v2 + brz v3, ebb1 + v4 = iadd v1, v3 + brz v4, ebb4 + jump ebb5 + ebb3: + trap + ebb4: + trap + ebb5: + trap + } + ", vec![0, 2, 5, 4, 1, 3]); +} + +#[test] +fn loops_one() { + test_reverse_postorder_traversal(" + function test(i32) { + ebb0(v0: i32): + jump ebb1 + ebb1: + brnz v0, ebb3 + jump ebb2 + ebb2: + jump ebb3 + ebb3: + return + } + ", vec![0, 1, 2, 3]); +} + +#[test] +fn loops_two() { + test_reverse_postorder_traversal(" + function test(i32) { + ebb0(v0: i32): + brz v0, ebb1 + jump ebb2 + ebb1: + jump ebb3 + ebb2: + brz v0, ebb4 + jump ebb5 + ebb3: + jump ebb4 + ebb4: + brz v0, ebb3 + jump ebb5 + ebb5: + brz v0, ebb4 + return + } + ", vec![0, 2, 5, 4, 3, 1]); +} diff --git a/cranelift/src/libcretonne/tests/lib.rs b/cranelift/src/libcretonne/tests/lib.rs new file mode 100644 index 0000000000..26cb1041ac --- /dev/null +++ b/cranelift/src/libcretonne/tests/lib.rs @@ -0,0 +1 @@ +pub mod cfg; From 1f0e344b24c942932364cf1af73f05cec6e1744d Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Mon, 25 Jul 2016 18:50:50 -0700 Subject: [PATCH 0163/3084] Cargo-fmt and cleanup. --- cranelift/src/libcretonne/Cargo.toml | 3 + cranelift/src/libcretonne/cfg.rs | 121 --------------------------- 2 files changed, 3 insertions(+), 121 deletions(-) diff --git a/cranelift/src/libcretonne/Cargo.toml b/cranelift/src/libcretonne/Cargo.toml index e6158af94f..cfd886391c 100644 --- a/cranelift/src/libcretonne/Cargo.toml +++ b/cranelift/src/libcretonne/Cargo.toml @@ -12,4 +12,7 @@ build = "build.rs" name = "cretonne" path = "lib.rs" +[dev-dependencies] +cretonne-reader = { path = "../libreader" } + [dependencies] diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 696c364af2..d30262e4cd 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -256,125 +256,4 @@ mod tests { assert_eq!(ebb1_successors.contains(&ebb1), true); assert_eq!(ebb1_successors.contains(&ebb2), true); } - - #[test] - fn postorder_traversal() { - let mut func = Function::new(); - let ebb0 = func.dfg.make_ebb(); - let ebb1 = func.dfg.make_ebb(); - let ebb2 = func.dfg.make_ebb(); - let ebb3 = func.dfg.make_ebb(); - let ebb4 = func.dfg.make_ebb(); - let ebb5 = func.dfg.make_ebb(); - - func.layout.append_ebb(ebb0); - func.layout.append_ebb(ebb1); - func.layout.append_ebb(ebb2); - func.layout.append_ebb(ebb3); - func.layout.append_ebb(ebb4); - func.layout.append_ebb(ebb5); - - let br_ebb0_ebb1 = make_inst::branch(&mut func, ebb1); - func.layout.append_inst(br_ebb0_ebb1, ebb0); - - let jmp_ebb0_ebb2 = make_inst::jump(&mut func, ebb2); - func.layout.append_inst(jmp_ebb0_ebb2, ebb0); - - let br_ebb2_ebb2 = make_inst::branch(&mut func, ebb2); - func.layout.append_inst(br_ebb2_ebb2, ebb2); - - let br_ebb2_ebb1 = make_inst::branch(&mut func, ebb1); - func.layout.append_inst(br_ebb2_ebb1, ebb2); - - let jmp_ebb1_ebb3 = make_inst::jump(&mut func, ebb3); - func.layout.append_inst(jmp_ebb1_ebb3, ebb1); - - let br_ebb2_ebb4 = make_inst::branch(&mut func, ebb4); - func.layout.append_inst(br_ebb2_ebb4, ebb2); - - let jmp_ebb2_ebb5 = make_inst::jump(&mut func, ebb5); - func.layout.append_inst(jmp_ebb2_ebb5, ebb2); - - let cfg = ControlFlowGraph::new(&func); - let mut postorder = vec![ebb3, ebb1, ebb4, ebb5, ebb2, ebb0]; - postorder.reverse(); - for (ebb, key) in cfg.reverse_postorder_ebbs() { - assert_eq!(ebb, postorder[key]); - } - } - - #[test] - fn loops_one() { - let mut func = Function::new(); - let ebb0 = func.dfg.make_ebb(); - let ebb1 = func.dfg.make_ebb(); - let ebb2 = func.dfg.make_ebb(); - let ebb3 = func.dfg.make_ebb(); - func.layout.append_ebb(ebb0); - func.layout.append_ebb(ebb1); - func.layout.append_ebb(ebb2); - func.layout.append_ebb(ebb3); - - let jmp_ebb0_ebb1 = make_inst::jump(&mut func, ebb1); - let br_ebb1_ebb3 = make_inst::branch(&mut func, ebb3); - let jmp_ebb1_ebb2 = make_inst::jump(&mut func, ebb2); - let jmp_ebb2_ebb3 = make_inst::jump(&mut func, ebb3); - - func.layout.append_inst(jmp_ebb0_ebb1, ebb0); - func.layout.append_inst(br_ebb1_ebb3, ebb1); - func.layout.append_inst(jmp_ebb1_ebb2, ebb1); - func.layout.append_inst(jmp_ebb2_ebb3, ebb2); - - let cfg = ControlFlowGraph::new(&func); - let mut postorder = vec![ebb3, ebb2, ebb1, ebb0]; - postorder.reverse(); - for (ebb, key) in cfg.reverse_postorder_ebbs() { - assert_eq!(ebb, postorder[key]); - } - } - - #[test] - fn loops_two() { - let mut func = Function::new(); - let ebb0 = func.dfg.make_ebb(); - let ebb1 = func.dfg.make_ebb(); - let ebb2 = func.dfg.make_ebb(); - let ebb3 = func.dfg.make_ebb(); - let ebb4 = func.dfg.make_ebb(); - let ebb5 = func.dfg.make_ebb(); - - func.layout.append_ebb(ebb0); - func.layout.append_ebb(ebb1); - func.layout.append_ebb(ebb2); - func.layout.append_ebb(ebb3); - func.layout.append_ebb(ebb4); - func.layout.append_ebb(ebb5); - - let jmp_ebb0_ebb1 = make_inst::jump(&mut func, ebb1); - let jmp_ebb0_ebb2 = make_inst::jump(&mut func, ebb2); - let jmp_ebb1_ebb3 = make_inst::jump(&mut func, ebb3); - let br_ebb2_ebb4 = make_inst::jump(&mut func, ebb4); - let jmp_ebb2_ebb5 = make_inst::jump(&mut func, ebb5); - let jmp_ebb3_ebb4 = make_inst::jump(&mut func, ebb4); - let br_ebb4_ebb3 = make_inst::branch(&mut func, ebb3); - let jmp_ebb4_ebb5 = make_inst::jump(&mut func, ebb5); - let jmp_ebb5_ebb4 = make_inst::jump(&mut func, ebb4); - - func.layout.append_inst(jmp_ebb0_ebb1, ebb0); - func.layout.append_inst(jmp_ebb0_ebb2, ebb0); - func.layout.append_inst(jmp_ebb1_ebb3, ebb1); - func.layout.append_inst(br_ebb2_ebb4, ebb2); - func.layout.append_inst(jmp_ebb2_ebb5, ebb2); - func.layout.append_inst(jmp_ebb3_ebb4, ebb3); - func.layout.append_inst(br_ebb4_ebb3, ebb4); - func.layout.append_inst(jmp_ebb4_ebb5, ebb4); - func.layout.append_inst(jmp_ebb5_ebb4, ebb5); - - let cfg = ControlFlowGraph::new(&func); - let mut postorder = vec![ebb1, ebb3, ebb4, ebb5, ebb2, ebb0]; - postorder.reverse(); - for (ebb, key) in cfg.reverse_postorder_ebbs() { - assert_eq!(ebb, postorder[key]); - } - } } From 9e19326c300bd86653cec1a7b0d3786b88f5509d Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Mon, 25 Jul 2016 19:19:46 -0700 Subject: [PATCH 0164/3084] Fix the test-all script so that it works with directories --- cranelift/test-all.sh | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 31638a6db3..7ddd65a21d 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -17,16 +17,20 @@ set -e cd $(dirname "$0") topdir=$(pwd) -# Run cargo from the src/tools directory which includes all our crates for -# building cton-util. -cd "$topdir/src/tools" -PKGS="-p cretonne -p cretonne-reader -p cretonne-tools" -echo ====== Rust unit tests and debug build ====== -cargo test $PKGS -cargo build $PKGS -cargo doc +PKGS="libcretonne libreader tools" +echo ====== Rust unit tests and debug builds ====== +for PKG in $PKGS +do + pushd $topdir/src/$PKG + cargo test + cargo build + popd +done -echo ====== Rust release build ====== +# Build cton-util for parser testing. +echo ====== Rust release build and documentation ====== +cd "$topdir/src/tools" +cargo doc cargo build --release export CTONUTIL="$topdir/src/tools/target/release/cton-util" From a9842ff363ef016b296aebbd0b2543590f1b4c2f Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Tue, 26 Jul 2016 02:54:42 -0700 Subject: [PATCH 0165/3084] Fix broken reverse_postorder_ebbs implementation. The previous implementation was actually a reverse preorder walk. --- cranelift/src/libcretonne/cfg.rs | 111 +++++++++++++++++++++---- cranelift/src/libcretonne/tests/cfg.rs | 11 ++- 2 files changed, 103 insertions(+), 19 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index d30262e4cd..bc78a3e904 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -26,7 +26,7 @@ use ir::Function; use ir::entities::{Inst, Ebb}; use ir::instructions::InstructionData; use entity_map::EntityMap; -use std::collections::{HashSet, BTreeMap}; +use std::collections::{HashMap, BTreeMap}; /// A basic block denoted by its enclosing Ebb and last instruction. pub type BasicBlock = (Ebb, Inst); @@ -111,7 +111,7 @@ impl ControlFlowGraph { } /// Return ebbs in reverse postorder along with a mapping of - /// the ebb to its order of visitation. + /// the ebb to its [post]order of visitation. pub fn reverse_postorder_ebbs(&self) -> BTreeMap { let entry_block = match self.entry_block { None => { @@ -119,21 +119,14 @@ impl ControlFlowGraph { } Some(eb) => eb, }; - let mut seen = HashSet::new(); - let mut open_nodes = vec![entry_block]; - let mut finished = BTreeMap::new(); - while open_nodes.len() > 0 { - let cur = open_nodes.pop().unwrap(); - for child in &self.data[cur].successors { - if *child != cur && !seen.contains(&child) { - seen.insert(child); - open_nodes.push(child.clone()); - } - } - let index = finished.len(); - finished.insert(cur, index); + let mut postorder = CFGPostorderWalker::new(&self, entry_block).walk(); + postorder.reverse(); + let mut result = BTreeMap::new(); + for (offset, ebb) in postorder.iter().enumerate() { + let i = postorder.len() - offset; + result.insert(ebb.clone(), i); } - finished + result } pub fn len(&self) -> usize { @@ -148,6 +141,92 @@ impl ControlFlowGraph { } } +/// A helper for iteratively walking a CFG in postorder. +pub struct CFGPostorderWalker<'a> { + cfg: &'a ControlFlowGraph, + start: Ebb, + // Ebbs are mapped to a tuple of booleans where the first bool + // is true if the node has been visited, and the second is + // true if its children have been visited. + visited: HashMap, + levels: Vec>, + // Our node index within the current level. + level_index: usize, +} + +impl<'a> CFGPostorderWalker<'a> { + fn new(cfg: &ControlFlowGraph, start: Ebb) -> CFGPostorderWalker { + CFGPostorderWalker { + cfg: cfg, + start: start, + visited: HashMap::new(), + levels: Vec::new(), + level_index: 0, + } + } + + fn visited(&self, ebb: Ebb) -> bool { + match self.visited.get(&ebb) { + Some(b) => b.0, + None => false, + } + } + + fn children_visited(&self, ebb: Ebb) -> bool { + match self.visited.get(&ebb) { + Some(b) => b.1, + None => false, + } + } + + fn mark_visited(&mut self, ebb: Ebb) { + let status = self.visited.entry(ebb).or_insert((false, false)).1; + self.visited.insert(ebb, (true, status)); + } + + fn mark_children_visited(&mut self, ebb: Ebb) { + let status = self.visited.entry(ebb).or_insert((false, false)).0; + self.visited.insert(ebb, (status, true)); + } + + fn walk(&mut self) -> Vec { + let mut postorder = Vec::new(); + + self.levels.push(vec![self.start.clone()]); + while self.levels.len() > 0 { + let level = &self.levels[self.levels.len() - 1].clone(); + + if self.level_index >= level.len() { + self.levels.pop(); + self.level_index = 0; + continue; + } + + let node = level[self.level_index].clone(); + if !self.visited(node) { + if self.children_visited(node) { + self.mark_visited(node); + postorder.push(node.clone()); + self.level_index += 1; + } else { + let edges = self.cfg.get_successors(node); + let outgoing = edges.iter() + .filter(|e| !self.children_visited(**e)) + .cloned() + .collect::>(); + if outgoing.len() > 0 { + self.levels.push(outgoing); + } + self.mark_children_visited(node); + } + } else { + self.level_index += 1; + } + } + postorder + } +} + /// Iterate through every mapping of ebb to predecessors in the CFG pub struct CFGPredecessorsIter<'a> { cfg: &'a ControlFlowGraph, diff --git a/cranelift/src/libcretonne/tests/cfg.rs b/cranelift/src/libcretonne/tests/cfg.rs index 6ad024aefd..3024b8a8fb 100644 --- a/cranelift/src/libcretonne/tests/cfg.rs +++ b/cranelift/src/libcretonne/tests/cfg.rs @@ -10,8 +10,13 @@ fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) let cfg = ControlFlowGraph::new(&func); let ebbs = ebb_order.iter().map(|n| Ebb::with_number(*n).unwrap()) .collect::>(); - for (ebb, key) in cfg.reverse_postorder_ebbs() { - assert_eq!(ebb, ebbs[key]); + + let reverse_order_ebbs = cfg.reverse_postorder_ebbs(); + + assert_eq!(reverse_order_ebbs.len(), ebbs.len()); + + for (ebb, key) in reverse_order_ebbs { + assert_eq!(ebb, ebbs[ebbs.len() - key]); } } @@ -81,5 +86,5 @@ fn loops_two() { brz v0, ebb4 return } - ", vec![0, 2, 5, 4, 3, 1]); + ", vec![0, 2, 1, 3, 4, 5]); } From b67d07e87a8e94220771fe9ba684e8d19d5aadae Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Tue, 26 Jul 2016 12:07:18 -0700 Subject: [PATCH 0166/3084] Give the test module a more apt name. --- .../tests/{cfg.rs => cfg_traversal.rs} | 36 ++++++++++++++++--- cranelift/src/libcretonne/tests/lib.rs | 2 +- 2 files changed, 33 insertions(+), 5 deletions(-) rename cranelift/src/libcretonne/tests/{cfg.rs => cfg_traversal.rs} (71%) diff --git a/cranelift/src/libcretonne/tests/cfg.rs b/cranelift/src/libcretonne/tests/cfg_traversal.rs similarity index 71% rename from cranelift/src/libcretonne/tests/cfg.rs rename to cranelift/src/libcretonne/tests/cfg_traversal.rs index 3024b8a8fb..056b72dc15 100644 --- a/cranelift/src/libcretonne/tests/cfg.rs +++ b/cranelift/src/libcretonne/tests/cfg_traversal.rs @@ -11,11 +11,10 @@ fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) let ebbs = ebb_order.iter().map(|n| Ebb::with_number(*n).unwrap()) .collect::>(); - let reverse_order_ebbs = cfg.reverse_postorder_ebbs(); + let reverse_postorder_ebbs = cfg.reverse_postorder_ebbs(); - assert_eq!(reverse_order_ebbs.len(), ebbs.len()); - - for (ebb, key) in reverse_order_ebbs { + assert_eq!(reverse_postorder_ebbs.len(), ebbs.len()); + for (ebb, key) in reverse_postorder_ebbs { assert_eq!(ebb, ebbs[ebbs.len() - key]); } } @@ -88,3 +87,32 @@ fn loops_two() { } ", vec![0, 2, 1, 3, 4, 5]); } + +#[test] +fn loops_three() { + test_reverse_postorder_traversal(" + function test(i32) { + ebb0(v0: i32): + brz v0, ebb1 + jump ebb2 + ebb1: + jump ebb3 + ebb2: + brz v0, ebb4 + jump ebb5 + ebb3: + jump ebb4 + ebb4: + brz v0, ebb3 + jump ebb6 + brz v0, ebb5 + ebb5: + brz v0, ebb4 + trap + ebb6: + jump ebb7 + ebb7: + return + } + ", vec![0, 2, 1, 3, 4, 5, 6, 7]); +} diff --git a/cranelift/src/libcretonne/tests/lib.rs b/cranelift/src/libcretonne/tests/lib.rs index 26cb1041ac..70d0709a9c 100644 --- a/cranelift/src/libcretonne/tests/lib.rs +++ b/cranelift/src/libcretonne/tests/lib.rs @@ -1 +1 @@ -pub mod cfg; +pub mod cfg_traversal; From 1027fc747215cd6ce2bcdf373e8b2f7c8f92525e Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Tue, 26 Jul 2016 17:13:11 -0700 Subject: [PATCH 0167/3084] Avoid cloning levels --- cranelift/src/libcretonne/cfg.rs | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index bc78a3e904..5d1cbaadfe 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -142,7 +142,7 @@ impl ControlFlowGraph { } /// A helper for iteratively walking a CFG in postorder. -pub struct CFGPostorderWalker<'a> { +struct CFGPostorderWalker<'a> { cfg: &'a ControlFlowGraph, start: Ebb, // Ebbs are mapped to a tuple of booleans where the first bool @@ -194,15 +194,14 @@ impl<'a> CFGPostorderWalker<'a> { self.levels.push(vec![self.start.clone()]); while self.levels.len() > 0 { - let level = &self.levels[self.levels.len() - 1].clone(); - - if self.level_index >= level.len() { + let level_len = self.levels[self.levels.len() - 1].len(); + if self.level_index >= level_len { self.levels.pop(); self.level_index = 0; continue; } - let node = level[self.level_index].clone(); + let node = self.levels[self.levels.len() - 1][self.level_index].clone(); if !self.visited(node) { if self.children_visited(node) { self.mark_visited(node); From 0e046dcbcb8df6f5c7efd6d23e9bef841e74792a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Jul 2016 16:10:02 -0700 Subject: [PATCH 0168/3084] Documentation typos. --- cranelift/docs/metaref.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 65b8f88654..fc9c5e4e29 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -19,7 +19,7 @@ top-level directory. The descriptions are processed in two steps: and other properties. 2. The static data structures are processed to produce Rust source code and - constant dables tables. + constant tables. The main driver for this source code generation process is the :file:`meta/build.py` script which is invoked as part of the build process if @@ -115,7 +115,7 @@ Concrete value types are represented as instances of :class:`cretonne.ValueType` subclasses to represent scalar and vector types. .. autoclass:: ValueType -.. inheritance-diagram:: ValueType ScalarType VectorType IntType FloatType +.. inheritance-diagram:: ValueType ScalarType VectorType IntType FloatType BoolType :parts: 1 .. autoclass:: ScalarType :members: @@ -125,6 +125,8 @@ subclasses to represent scalar and vector types. :members: .. autoclass:: FloatType :members: +.. autoclass:: BoolType + :members: .. automodule:: cretonne.types :members: From 7936444e1f7ff2d7ed91c47991642c65ce8bf0ac Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Thu, 28 Jul 2016 17:49:25 -0700 Subject: [PATCH 0169/3084] Simplify the reverse_postorder_ebbs implementation. --- cranelift/src/libcretonne/cfg.rs | 134 +++++------------- .../src/libcretonne/tests/cfg_traversal.rs | 30 +++- 2 files changed, 64 insertions(+), 100 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 5d1cbaadfe..f9df5e2997 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -24,9 +24,9 @@ use ir::Function; use ir::entities::{Inst, Ebb}; -use ir::instructions::InstructionData; +use ir::instructions::BranchInfo; use entity_map::EntityMap; -use std::collections::{HashMap, BTreeMap}; +use std::collections::{HashSet, BTreeMap}; /// A basic block denoted by its enclosing Ebb and last instruction. pub type BasicBlock = (Ebb, Inst); @@ -74,16 +74,18 @@ impl ControlFlowGraph { for ebb in &func.layout { for inst in func.layout.ebb_insts(ebb) { - match func.dfg[inst] { - InstructionData::Branch { ty: _, opcode: _, ref data } => { - cfg.add_successor(ebb, data.destination); - cfg.add_predecessor(data.destination, (ebb, inst)); + match func.dfg[inst].analyze_branch() { + BranchInfo::SingleDest(dest, _) => { + cfg.add_successor(ebb, dest); + cfg.add_predecessor(dest, (ebb, inst)); } - InstructionData::Jump { ty: _, opcode: _, ref data } => { - cfg.add_successor(ebb, data.destination); - cfg.add_predecessor(data.destination, (ebb, inst)); + BranchInfo::Table(jt) => { + for (_, dest) in func.jump_tables[jt].entries() { + cfg.add_successor(ebb, dest); + cfg.add_predecessor(dest, (ebb, inst)); + } } - _ => (), + BranchInfo::NotABranch => {} } } } @@ -119,8 +121,33 @@ impl ControlFlowGraph { } Some(eb) => eb, }; - let mut postorder = CFGPostorderWalker::new(&self, entry_block).walk(); + + let mut grey = HashSet::new(); + let mut black = HashSet::new(); + let mut stack = vec![entry_block.clone()]; + let mut postorder = Vec::new(); + + while !stack.is_empty() { + let node = stack.pop().unwrap(); + if !grey.contains(&node) { + // This is a white node. Mark it as gray. + grey.insert(node); + stack.push(node); + // Get any children we’ve never seen before. + for child in self.get_successors(node) { + if !grey.contains(child) { + stack.push(child.clone()); + } + } + } else if !black.contains(&node) { + // This is a gray node, now becoming black. + // We don’t need to mark it since we won’t see it again. + postorder.push(node.clone()); + black.insert(node.clone()); + } + } postorder.reverse(); + let mut result = BTreeMap::new(); for (offset, ebb) in postorder.iter().enumerate() { let i = postorder.len() - offset; @@ -141,91 +168,6 @@ impl ControlFlowGraph { } } -/// A helper for iteratively walking a CFG in postorder. -struct CFGPostorderWalker<'a> { - cfg: &'a ControlFlowGraph, - start: Ebb, - // Ebbs are mapped to a tuple of booleans where the first bool - // is true if the node has been visited, and the second is - // true if its children have been visited. - visited: HashMap, - levels: Vec>, - // Our node index within the current level. - level_index: usize, -} - -impl<'a> CFGPostorderWalker<'a> { - fn new(cfg: &ControlFlowGraph, start: Ebb) -> CFGPostorderWalker { - CFGPostorderWalker { - cfg: cfg, - start: start, - visited: HashMap::new(), - levels: Vec::new(), - level_index: 0, - } - } - - fn visited(&self, ebb: Ebb) -> bool { - match self.visited.get(&ebb) { - Some(b) => b.0, - None => false, - } - } - - fn children_visited(&self, ebb: Ebb) -> bool { - match self.visited.get(&ebb) { - Some(b) => b.1, - None => false, - } - } - - fn mark_visited(&mut self, ebb: Ebb) { - let status = self.visited.entry(ebb).or_insert((false, false)).1; - self.visited.insert(ebb, (true, status)); - } - - fn mark_children_visited(&mut self, ebb: Ebb) { - let status = self.visited.entry(ebb).or_insert((false, false)).0; - self.visited.insert(ebb, (status, true)); - } - - fn walk(&mut self) -> Vec { - let mut postorder = Vec::new(); - - self.levels.push(vec![self.start.clone()]); - while self.levels.len() > 0 { - let level_len = self.levels[self.levels.len() - 1].len(); - if self.level_index >= level_len { - self.levels.pop(); - self.level_index = 0; - continue; - } - - let node = self.levels[self.levels.len() - 1][self.level_index].clone(); - if !self.visited(node) { - if self.children_visited(node) { - self.mark_visited(node); - postorder.push(node.clone()); - self.level_index += 1; - } else { - let edges = self.cfg.get_successors(node); - let outgoing = edges.iter() - .filter(|e| !self.children_visited(**e)) - .cloned() - .collect::>(); - if outgoing.len() > 0 { - self.levels.push(outgoing); - } - self.mark_children_visited(node); - } - } else { - self.level_index += 1; - } - } - postorder - } -} - /// Iterate through every mapping of ebb to predecessors in the CFG pub struct CFGPredecessorsIter<'a> { cfg: &'a ControlFlowGraph, diff --git a/cranelift/src/libcretonne/tests/cfg_traversal.rs b/cranelift/src/libcretonne/tests/cfg_traversal.rs index 056b72dc15..8a3ca97594 100644 --- a/cranelift/src/libcretonne/tests/cfg_traversal.rs +++ b/cranelift/src/libcretonne/tests/cfg_traversal.rs @@ -44,7 +44,7 @@ fn simple_traversal() { ebb5: trap } - ", vec![0, 2, 5, 4, 1, 3]); + ", vec![0, 2, 1, 3, 4, 5]); } #[test] @@ -85,7 +85,7 @@ fn loops_two() { brz v0, ebb4 return } - ", vec![0, 2, 1, 3, 4, 5]); + ", vec![0, 1, 2, 5, 4, 3]); } #[test] @@ -104,8 +104,8 @@ fn loops_three() { jump ebb4 ebb4: brz v0, ebb3 + brnz v0, ebb5 jump ebb6 - brz v0, ebb5 ebb5: brz v0, ebb4 trap @@ -114,5 +114,27 @@ fn loops_three() { ebb7: return } - ", vec![0, 2, 1, 3, 4, 5, 6, 7]); + ", vec![0, 1, 2, 5, 4, 3, 6, 7]); +} + +#[test] +fn back_edge_one() { + test_reverse_postorder_traversal(" + function test(i32) { + ebb0(v0: i32): + brz v0, ebb1 + jump ebb2 + ebb1: + jump ebb3 + ebb2: + brz v0, ebb0 + jump ebb4 + ebb3: + brz v0, ebb2 + brnz v0, ebb0 + return + ebb4: + trap + } + ", vec![0, 1, 3, 2, 4]); } From 2f07a3b6752014d4128bc9159fe2260ec334c1d2 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Thu, 28 Jul 2016 17:51:50 -0700 Subject: [PATCH 0170/3084] Remove innacurate comments. --- cranelift/src/libcretonne/cfg.rs | 2 -- 1 file changed, 2 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index f9df5e2997..ff45b59ae8 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -140,8 +140,6 @@ impl ControlFlowGraph { } } } else if !black.contains(&node) { - // This is a gray node, now becoming black. - // We don’t need to mark it since we won’t see it again. postorder.push(node.clone()); black.insert(node.clone()); } From 0ef28f5bde0138ddff63291a0d9f8b2bc6536170 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Jul 2016 12:35:56 -0700 Subject: [PATCH 0171/3084] Use sub-shells instead of pushd / popd. The push and pop commands print the directory stack to stdout, while subshells and cd is quiet. --- cranelift/test-all.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 7ddd65a21d..28013ec2e3 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -21,10 +21,11 @@ PKGS="libcretonne libreader tools" echo ====== Rust unit tests and debug builds ====== for PKG in $PKGS do - pushd $topdir/src/$PKG + ( + cd $topdir/src/$PKG cargo test cargo build - popd + ) done # Build cton-util for parser testing. From dae349371fd94ac165417d3a1adaa1df5d107a4a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Jul 2016 15:59:09 -0700 Subject: [PATCH 0172/3084] Add EntityMap::with_capacity. Create a secondary entity map with space reserved for a known range of entity references. Add dfg.num_ebbs() and dfg.num_insts() methods to provide capacities. --- cranelift/src/libcretonne/entity_map.rs | 15 +++++++++++++++ cranelift/src/libcretonne/ir/dfg.rs | 16 ++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/cranelift/src/libcretonne/entity_map.rs b/cranelift/src/libcretonne/entity_map.rs index 9ce39bec19..a0f4f3fa6b 100644 --- a/cranelift/src/libcretonne/entity_map.rs +++ b/cranelift/src/libcretonne/entity_map.rs @@ -102,6 +102,21 @@ impl EntityMap where K: EntityRef, V: Clone + Default { + /// Create a new secondary `EntityMap` that is prepared to hold `n` elements. + /// + /// Use this when the length of the primary map is known: + /// ``` + /// let secondary_map = EntityMap::with_capacity(primary_map.len()); + /// ``` + pub fn with_capacity(n: usize) -> Self { + let mut map = EntityMap { + elems: Vec::with_capacity(n), + unused: PhantomData, + }; + map.elems.resize(n, V::default()); + map + } + /// Ensure that `k` is a valid key but adding default entries if necesssary. /// /// Return a mutable reference to the corresponding entry. diff --git a/cranelift/src/libcretonne/ir/dfg.rs b/cranelift/src/libcretonne/ir/dfg.rs index d377813544..aba45354d7 100644 --- a/cranelift/src/libcretonne/ir/dfg.rs +++ b/cranelift/src/libcretonne/ir/dfg.rs @@ -42,6 +42,22 @@ impl DataFlowGraph { extended_values: Vec::new(), } } + + /// Get the total number of instructions created in this function, whether they are currently + /// inserted in the layout or not. + /// + /// This is intended for use with `EntityMap::with_capacity`. + pub fn num_insts(&self) -> usize { + self.insts.len() + } + + /// Get the total number of extended basic blocks created in this function, whether they are + /// currently inserted in the layout or not. + /// + /// This is intended for use with `EntityMap::with_capacity`. + pub fn num_ebbs(&self) -> usize { + self.ebbs.len() + } } /// Handling values. From a8beb542a5ef39ab9989c846e251efa68eedcf67 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Sun, 31 Jul 2016 21:31:18 -0700 Subject: [PATCH 0173/3084] Clean up the CFG representation. --- cranelift/src/libcretonne/cfg.rs | 41 ++++++------------- .../src/libcretonne/tests/cfg_traversal.rs | 4 +- 2 files changed, 15 insertions(+), 30 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index ff45b59ae8..25b5f2409a 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -26,13 +26,13 @@ use ir::Function; use ir::entities::{Inst, Ebb}; use ir::instructions::BranchInfo; use entity_map::EntityMap; -use std::collections::{HashSet, BTreeMap}; +use std::collections::HashSet; /// A basic block denoted by its enclosing Ebb and last instruction. pub type BasicBlock = (Ebb, Inst); /// A container for the successors and predecessors of some Ebb. -#[derive(Debug)] +#[derive(Debug, Clone, Default)] pub struct CFGNode { pub successors: Vec, pub predecessors: Vec, @@ -62,27 +62,19 @@ impl ControlFlowGraph { pub fn new(func: &Function) -> ControlFlowGraph { let mut cfg = ControlFlowGraph { - data: EntityMap::new(), + data: EntityMap::with_capacity(func.dfg.num_ebbs()), entry_block: func.layout.entry_block(), }; - // Even ebbs without predecessors should show up in the CFG, albeit - // with no entires. - for _ in &func.layout { - cfg.push_ebb(); - } - for ebb in &func.layout { for inst in func.layout.ebb_insts(ebb) { match func.dfg[inst].analyze_branch() { BranchInfo::SingleDest(dest, _) => { - cfg.add_successor(ebb, dest); - cfg.add_predecessor(dest, (ebb, inst)); + cfg.add_edge((ebb, inst), dest); } BranchInfo::Table(jt) => { for (_, dest) in func.jump_tables[jt].entries() { - cfg.add_successor(ebb, dest); - cfg.add_predecessor(dest, (ebb, inst)); + cfg.add_edge((ebb, inst), dest); } } BranchInfo::NotABranch => {} @@ -92,19 +84,12 @@ impl ControlFlowGraph { cfg } - pub fn push_ebb(&mut self) { - self.data.push(CFGNode::new()); + fn add_edge(&mut self, from: BasicBlock, to: Ebb) { + self.data[from.0].successors.push(to); + self.data[to].predecessors.push(from); } - pub fn add_successor(&mut self, from: Ebb, to: Ebb) { - self.data[from].successors.push(to); - } - - pub fn add_predecessor(&mut self, ebb: Ebb, predecessor: BasicBlock) { - self.data[ebb].predecessors.push(predecessor); - } - - pub fn get_predecessors(&self, ebb: Ebb) -> &Vec { + fn get_predecessors(&self, ebb: Ebb) -> &Vec { &self.data[ebb].predecessors } @@ -114,10 +99,10 @@ impl ControlFlowGraph { /// Return ebbs in reverse postorder along with a mapping of /// the ebb to its [post]order of visitation. - pub fn reverse_postorder_ebbs(&self) -> BTreeMap { + pub fn reverse_postorder_ebbs(&self) -> EntityMap { let entry_block = match self.entry_block { None => { - return BTreeMap::new(); + return EntityMap::new(); } Some(eb) => eb, }; @@ -146,10 +131,10 @@ impl ControlFlowGraph { } postorder.reverse(); - let mut result = BTreeMap::new(); + let mut result = EntityMap::with_capacity(postorder.len()); for (offset, ebb) in postorder.iter().enumerate() { let i = postorder.len() - offset; - result.insert(ebb.clone(), i); + result[ebb.clone()] = i; } result } diff --git a/cranelift/src/libcretonne/tests/cfg_traversal.rs b/cranelift/src/libcretonne/tests/cfg_traversal.rs index 8a3ca97594..ae74445dd4 100644 --- a/cranelift/src/libcretonne/tests/cfg_traversal.rs +++ b/cranelift/src/libcretonne/tests/cfg_traversal.rs @@ -14,8 +14,8 @@ fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) let reverse_postorder_ebbs = cfg.reverse_postorder_ebbs(); assert_eq!(reverse_postorder_ebbs.len(), ebbs.len()); - for (ebb, key) in reverse_postorder_ebbs { - assert_eq!(ebb, ebbs[ebbs.len() - key]); + for ebb in reverse_postorder_ebbs.keys() { + assert_eq!(ebb, ebbs[ebbs.len() - reverse_postorder_ebbs[ebb]]); } } From 967a7c64d4468273f36a3130dbf5e34481e65db9 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Sun, 31 Jul 2016 21:40:11 -0700 Subject: [PATCH 0174/3084] Add a dominator tree implementation. --- cranelift/src/libcretonne/cfg.rs | 2 +- cranelift/src/libcretonne/dominator_tree.rs | 138 ++++++++++++++++++ cranelift/src/libcretonne/lib.rs | 2 +- .../src/libcretonne/tests/dominator_tree.rs | 60 ++++++++ cranelift/src/libcretonne/tests/lib.rs | 1 + 5 files changed, 201 insertions(+), 2 deletions(-) create mode 100644 cranelift/src/libcretonne/dominator_tree.rs create mode 100644 cranelift/src/libcretonne/tests/dominator_tree.rs diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 25b5f2409a..28a39861e2 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -89,7 +89,7 @@ impl ControlFlowGraph { self.data[to].predecessors.push(from); } - fn get_predecessors(&self, ebb: Ebb) -> &Vec { + pub fn get_predecessors(&self, ebb: Ebb) -> &Vec { &self.data[ebb].predecessors } diff --git a/cranelift/src/libcretonne/dominator_tree.rs b/cranelift/src/libcretonne/dominator_tree.rs new file mode 100644 index 0000000000..e4569ea112 --- /dev/null +++ b/cranelift/src/libcretonne/dominator_tree.rs @@ -0,0 +1,138 @@ +/// ! A Dominator Tree represented as mappings of Ebbs to their immediate dominator. + +use cfg::*; +use ir::entities::Ebb; +use entity_map::EntityMap; + +pub struct DominatorTree { + data: EntityMap>, +} + +impl DominatorTree { + pub fn new(cfg: &ControlFlowGraph) -> DominatorTree { + let mut dt = DominatorTree { data: EntityMap::new() }; + dt.build(cfg); + dt + } + + pub fn build(&mut self, cfg: &ControlFlowGraph) { + let reverse_postorder_map = cfg.reverse_postorder_ebbs(); + let ebbs = reverse_postorder_map.keys().collect::>(); + let len = reverse_postorder_map.len(); + + for (i, ebb) in ebbs.iter().enumerate() { + if i > 0 { + self.data.push(None); + } else { + self.data.push(Some(ebb.clone())); + } + } + + let mut changed = len > 0; + + while changed { + changed = false; + for i in 1..len { + let ebb = ebbs[i]; + let preds = cfg.get_predecessors(ebb); + let mut new_idom = None; + + for &(p, _) in preds { + if new_idom == None { + new_idom = Some(p); + continue; + } + if let Some(_) = self.data[p] { + new_idom = + Some(self.intersect(&reverse_postorder_map, p, new_idom.unwrap())); + } + } + match self.data[ebb] { + None => { + self.data[ebb] = new_idom; + changed = true; + } + Some(idom) => { + // Old idom != New idom + if idom != new_idom.unwrap() { + self.data[ebb] = new_idom; + changed = true; + } + } + } + } + } + } + + fn intersect(&self, ordering: &EntityMap, first: Ebb, second: Ebb) -> Ebb { + println!("A {} B {}", first, second); + let mut a = first; + let mut b = second; + while a != b { + while ordering[a] < ordering[b] { + a = self.data[a].unwrap(); + } + while ordering[b] < ordering[a] { + b = self.data[b].unwrap(); + } + } + a + } + + pub fn idom(&self, ebb: Ebb) -> Option { + self.data[ebb].clone() + } + + pub fn len(&self) -> usize { + self.data.len() + } +} + +#[cfg(test)] +mod test { + use super::*; + use ir::Function; + use cfg::ControlFlowGraph; + use test_utils::make_inst; + + #[test] + fn empty() { + let func = Function::new(); + let cfg = ControlFlowGraph::new(&func); + let dtree = DominatorTree::new(&cfg); + assert_eq!(dtree.len(), 0); + } + + #[test] + fn non_zero_entry_block() { + let mut func = Function::new(); + let ebb3 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + let ebb2 = func.dfg.make_ebb(); + let ebb0 = func.dfg.make_ebb(); + func.layout.append_ebb(ebb3); + func.layout.append_ebb(ebb1); + func.layout.append_ebb(ebb2); + func.layout.append_ebb(ebb0); + + let jmp_ebb3_ebb1 = make_inst::jump(&mut func, ebb1); + let br_ebb1_ebb0 = make_inst::branch(&mut func, ebb0); + let jmp_ebb1_ebb2 = make_inst::jump(&mut func, ebb2); + let jmp_ebb2_ebb0 = make_inst::jump(&mut func, ebb0); + + func.layout.append_inst(br_ebb1_ebb0, ebb1); + func.layout.append_inst(jmp_ebb1_ebb2, ebb1); + func.layout.append_inst(jmp_ebb2_ebb0, ebb2); + func.layout.append_inst(jmp_ebb3_ebb1, ebb3); + + let cfg = ControlFlowGraph::new(&func); + let dt = DominatorTree::new(&cfg); + + assert_eq!(func.layout.entry_block().unwrap(), ebb3); + assert_eq!(dt.len(), cfg.len()); + assert_eq!(dt.idom(ebb3).unwrap(), ebb3); + assert_eq!(dt.idom(ebb1).unwrap(), ebb3); + assert_eq!(dt.idom(ebb2).unwrap(), ebb1); + assert_eq!(dt.idom(ebb0).unwrap(), ebb1); + } +} diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index f81bfb5720..34151d187b 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -10,7 +10,7 @@ pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); pub mod ir; pub mod write; pub mod cfg; - +pub mod dominator_tree; pub mod entity_map; #[cfg(test)]pub mod test_utils; diff --git a/cranelift/src/libcretonne/tests/dominator_tree.rs b/cranelift/src/libcretonne/tests/dominator_tree.rs new file mode 100644 index 0000000000..6820acd152 --- /dev/null +++ b/cranelift/src/libcretonne/tests/dominator_tree.rs @@ -0,0 +1,60 @@ +extern crate cretonne; +extern crate cton_reader; + +use self::cton_reader::parser::Parser; +use self::cretonne::ir::entities::Ebb; +use self::cretonne::cfg::ControlFlowGraph; +use self::cretonne::dominator_tree::DominatorTree; + +fn test_dominator_tree(function_source: &str, idoms: Vec) { + let func = &Parser::parse(function_source).unwrap()[0]; + let cfg = ControlFlowGraph::new(&func); + let dtree = DominatorTree::new(&cfg); + assert_eq!(dtree.len(), idoms.len()); + for (i, j) in idoms.iter().enumerate() { + let ebb = Ebb::with_number(i.clone() as u32); + let idom = Ebb::with_number(*j); + assert_eq!(dtree.idom(ebb.unwrap()), idom); + } +} + +#[test] +fn basic() { + test_dominator_tree(" + function test(i32) { + ebb0(v0: i32): + jump ebb1 + ebb1: + brz v0, ebb3 + jump ebb2 + ebb2: + jump ebb3 + ebb3: + return + } + ", vec![0, 0, 1, 1]); +} + +#[test] +fn loops() { + test_dominator_tree(" + function test(i32) { + ebb0(v0: i32): + brz v0, ebb1 + jump ebb2 + ebb1: + jump ebb3 + ebb2: + brz v0, ebb4 + jump ebb5 + ebb3: + jump ebb4 + ebb4: + brz v0, ebb3 + jump ebb5 + ebb5: + brz v0, ebb4 + return + } + ", vec![0, 0, 0, 0, 0, 0]); +} diff --git a/cranelift/src/libcretonne/tests/lib.rs b/cranelift/src/libcretonne/tests/lib.rs index 70d0709a9c..fcd2535269 100644 --- a/cranelift/src/libcretonne/tests/lib.rs +++ b/cranelift/src/libcretonne/tests/lib.rs @@ -1 +1,2 @@ pub mod cfg_traversal; +pub mod dominator_tree; From f2e9c88b4500d9eeb21120da1609d821b4360d84 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Mon, 1 Aug 2016 12:15:08 -0700 Subject: [PATCH 0175/3084] Remove println\! --- cranelift/src/libcretonne/dominator_tree.rs | 1 - 1 file changed, 1 deletion(-) diff --git a/cranelift/src/libcretonne/dominator_tree.rs b/cranelift/src/libcretonne/dominator_tree.rs index e4569ea112..8dcc4e2899 100644 --- a/cranelift/src/libcretonne/dominator_tree.rs +++ b/cranelift/src/libcretonne/dominator_tree.rs @@ -65,7 +65,6 @@ impl DominatorTree { } fn intersect(&self, ordering: &EntityMap, first: Ebb, second: Ebb) -> Ebb { - println!("A {} B {}", first, second); let mut a = first; let mut b = second; while a != b { From e22efd274847062258c528917cadc55e4886747c Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Mon, 1 Aug 2016 15:00:08 -0700 Subject: [PATCH 0176/3084] Improve the structure and comments of the module. --- cranelift/src/libcretonne/dominator_tree.rs | 39 ++++++++++++++------- 1 file changed, 27 insertions(+), 12 deletions(-) diff --git a/cranelift/src/libcretonne/dominator_tree.rs b/cranelift/src/libcretonne/dominator_tree.rs index 8dcc4e2899..c58726a1ab 100644 --- a/cranelift/src/libcretonne/dominator_tree.rs +++ b/cranelift/src/libcretonne/dominator_tree.rs @@ -10,25 +10,25 @@ pub struct DominatorTree { impl DominatorTree { pub fn new(cfg: &ControlFlowGraph) -> DominatorTree { - let mut dt = DominatorTree { data: EntityMap::new() }; + let mut dt = DominatorTree { data: EntityMap::with_capacity(cfg.len()) }; dt.build(cfg); dt } - pub fn build(&mut self, cfg: &ControlFlowGraph) { + + /// Build a dominator tree from a control flow graph using Keith D. Cooper's + /// "Simple, Fast Dominator Algorithm." + fn build(&mut self, cfg: &ControlFlowGraph) { let reverse_postorder_map = cfg.reverse_postorder_ebbs(); let ebbs = reverse_postorder_map.keys().collect::>(); let len = reverse_postorder_map.len(); - for (i, ebb) in ebbs.iter().enumerate() { - if i > 0 { - self.data.push(None); - } else { - self.data.push(Some(ebb.clone())); - } - } + let mut changed = false; - let mut changed = len > 0; + if len > 0 { + self.data[ebbs[0]] = Some(ebbs[0]); + changed = true; + } while changed { changed = false; @@ -42,9 +42,15 @@ impl DominatorTree { new_idom = Some(p); continue; } + // If this predecessor `p` has an idom available find its common + // ancestor with the current value of new_idom. if let Some(_) = self.data[p] { - new_idom = - Some(self.intersect(&reverse_postorder_map, p, new_idom.unwrap())); + new_idom = match new_idom { + Some(cur_idom) => { + Some(self.intersect(&reverse_postorder_map, p, cur_idom)) + } + None => panic!("A 'current idom' should have been set!"), + } } } match self.data[ebb] { @@ -64,9 +70,15 @@ impl DominatorTree { } } + /// Find the common dominator of two ebbs. fn intersect(&self, ordering: &EntityMap, first: Ebb, second: Ebb) -> Ebb { let mut a = first; let mut b = second; + + // Here we use 'ordering', a mapping of ebbs to their postorder + // visitation number, to ensure that we move upward through the tree. + // Walking upward means that we may always expect self.data[a] and + // self.data[b] to contain non-None entries. while a != b { while ordering[a] < ordering[b] { a = self.data[a].unwrap(); @@ -78,10 +90,13 @@ impl DominatorTree { a } + /// Returns the immediate dominator of some ebb or None if the + /// node is unreachable. pub fn idom(&self, ebb: Ebb) -> Option { self.data[ebb].clone() } + /// The total number of nodes in the tree. pub fn len(&self) -> usize { self.data.len() } From bf24ae229cec2889fc1d26ce0195947f030ba3e5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 1 Aug 2016 15:03:56 -0700 Subject: [PATCH 0177/3084] Print CFG edges from func.layout instead of cfg.predecessors_iter. EBBs not in the layout should never be printed as part of the CFG. --- cranelift/src/tools/print_cfg.rs | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs index 9a08c0287e..dd1aa3de18 100644 --- a/cranelift/src/tools/print_cfg.rs +++ b/cranelift/src/tools/print_cfg.rs @@ -41,8 +41,8 @@ impl CFGPrinter { self.header(func); self.push_indent(); self.ebb_subgraphs(func); - let cfg = ControlFlowGraph::new(&func); - self.cfg_connections(&cfg); + let cfg = ControlFlowGraph::new(func); + self.cfg_connections(func, &cfg); self.pop_indent(); self.footer(); self.write() @@ -145,9 +145,9 @@ impl CFGPrinter { } } - fn cfg_connections(&mut self, cfg: &ControlFlowGraph) { - for (ref ebb, ref predecessors) in cfg.predecessors_iter() { - for &(parent, inst) in *predecessors { + fn cfg_connections(&mut self, func: &Function, cfg: &ControlFlowGraph) { + for ebb in &func.layout { + for &(parent, inst) in cfg.get_predecessors(ebb) { self.append(&format!("{}:{} -> {}", parent, inst, ebb)); self.newline(); } From 11f65fff202dbbdd2a78b6440099e24f5bd98b58 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 1 Aug 2016 15:14:32 -0700 Subject: [PATCH 0178/3084] Remove the cfg::predecessors_iter() method and iterator. This iterator enumerates all EBB references whether they are in the layout or not. That is usually not what is needed when working with the CFG. It is better to iterate over EBB referrences in layout order, or in reverse post-order and then call the get_predecessors() method for each Ebb reference. See the new implementation of print_cfg::cfg_connections(). --- cranelift/src/libcretonne/cfg.rs | 33 +++----------------------------- 1 file changed, 3 insertions(+), 30 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 28a39861e2..30f16b5df5 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -142,34 +142,6 @@ impl ControlFlowGraph { pub fn len(&self) -> usize { self.data.len() } - - pub fn predecessors_iter(&self) -> CFGPredecessorsIter { - CFGPredecessorsIter { - cur: 0, - cfg: &self, - } - } -} - -/// Iterate through every mapping of ebb to predecessors in the CFG -pub struct CFGPredecessorsIter<'a> { - cfg: &'a ControlFlowGraph, - cur: usize, -} - -impl<'a> Iterator for CFGPredecessorsIter<'a> { - type Item = (Ebb, &'a Vec); - - fn next(&mut self) -> Option { - if self.cur < self.cfg.len() { - let ebb = Ebb::with_number(self.cur as u32).unwrap(); - let bbs = self.cfg.get_predecessors(ebb); - self.cur += 1; - Some((ebb, bbs)) - } else { - None - } - } } #[cfg(test)] @@ -183,7 +155,7 @@ mod tests { fn empty() { let func = Function::new(); let cfg = ControlFlowGraph::new(&func); - assert_eq!(None, cfg.predecessors_iter().next()); + assert_eq!(cfg.reverse_postorder_ebbs().keys().count(), 0); } #[test] @@ -197,7 +169,8 @@ mod tests { func.layout.append_ebb(ebb2); let cfg = ControlFlowGraph::new(&func); - let nodes = cfg.predecessors_iter().collect::>(); + let nodes = + [ebb0, ebb1, ebb2].iter().map(|&e| (e, cfg.get_predecessors(e))).collect::>(); assert_eq!(nodes.len(), 3); let mut fun_ebbs = func.layout.ebbs(); From 283f3ea48e7eb8a981744c9e793559418420cdad Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Mon, 1 Aug 2016 18:04:25 -0700 Subject: [PATCH 0179/3084] Remove uses of EntityMap::len. --- cranelift/src/libcretonne/cfg.rs | 60 ++++-------------- cranelift/src/libcretonne/dominator_tree.rs | 61 +++++++++++-------- .../src/libcretonne/tests/cfg_traversal.rs | 14 +++-- .../src/libcretonne/tests/dominator_tree.rs | 2 +- 4 files changed, 58 insertions(+), 79 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 28a39861e2..022c3372e4 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -25,7 +25,7 @@ use ir::Function; use ir::entities::{Inst, Ebb}; use ir::instructions::BranchInfo; -use entity_map::EntityMap; +use entity_map::{EntityMap, Keys}; use std::collections::HashSet; /// A basic block denoted by its enclosing Ebb and last instruction. @@ -97,12 +97,11 @@ impl ControlFlowGraph { &self.data[ebb].successors } - /// Return ebbs in reverse postorder along with a mapping of - /// the ebb to its [post]order of visitation. - pub fn reverse_postorder_ebbs(&self) -> EntityMap { + /// Return [reachable] ebbs in postorder. + pub fn postorder_ebbs(&self) -> Vec { let entry_block = match self.entry_block { None => { - return EntityMap::new(); + return Vec::new(); } Some(eb) => eb, }; @@ -129,46 +128,12 @@ impl ControlFlowGraph { black.insert(node.clone()); } } - postorder.reverse(); - - let mut result = EntityMap::with_capacity(postorder.len()); - for (offset, ebb) in postorder.iter().enumerate() { - let i = postorder.len() - offset; - result[ebb.clone()] = i; - } - result + postorder } - pub fn len(&self) -> usize { - self.data.len() - } - - pub fn predecessors_iter(&self) -> CFGPredecessorsIter { - CFGPredecessorsIter { - cur: 0, - cfg: &self, - } - } -} - -/// Iterate through every mapping of ebb to predecessors in the CFG -pub struct CFGPredecessorsIter<'a> { - cfg: &'a ControlFlowGraph, - cur: usize, -} - -impl<'a> Iterator for CFGPredecessorsIter<'a> { - type Item = (Ebb, &'a Vec); - - fn next(&mut self) -> Option { - if self.cur < self.cfg.len() { - let ebb = Ebb::with_number(self.cur as u32).unwrap(); - let bbs = self.cfg.get_predecessors(ebb); - self.cur += 1; - Some((ebb, bbs)) - } else { - None - } + /// An iterator across all of the ebbs stored in the cfg. + pub fn ebbs(&self) -> Keys { + self.data.keys() } } @@ -183,7 +148,7 @@ mod tests { fn empty() { let func = Function::new(); let cfg = ControlFlowGraph::new(&func); - assert_eq!(None, cfg.predecessors_iter().next()); + assert_eq!(None, cfg.ebbs().next()); } #[test] @@ -197,14 +162,13 @@ mod tests { func.layout.append_ebb(ebb2); let cfg = ControlFlowGraph::new(&func); - let nodes = cfg.predecessors_iter().collect::>(); + let nodes = cfg.ebbs().collect::>(); assert_eq!(nodes.len(), 3); let mut fun_ebbs = func.layout.ebbs(); - for (ebb, predecessors) in nodes { + for ebb in nodes { assert_eq!(ebb, fun_ebbs.next().unwrap()); - assert_eq!(predecessors.len(), 0); - assert_eq!(predecessors.len(), 0); + assert_eq!(cfg.get_predecessors(ebb).len(), 0); assert_eq!(cfg.get_successors(ebb).len(), 0); } } diff --git a/cranelift/src/libcretonne/dominator_tree.rs b/cranelift/src/libcretonne/dominator_tree.rs index c58726a1ab..8420bcf541 100644 --- a/cranelift/src/libcretonne/dominator_tree.rs +++ b/cranelift/src/libcretonne/dominator_tree.rs @@ -2,31 +2,33 @@ use cfg::*; use ir::entities::Ebb; -use entity_map::EntityMap; +use entity_map::{EntityMap, Keys}; pub struct DominatorTree { data: EntityMap>, } impl DominatorTree { - pub fn new(cfg: &ControlFlowGraph) -> DominatorTree { - let mut dt = DominatorTree { data: EntityMap::with_capacity(cfg.len()) }; - dt.build(cfg); - dt - } - - /// Build a dominator tree from a control flow graph using Keith D. Cooper's /// "Simple, Fast Dominator Algorithm." - fn build(&mut self, cfg: &ControlFlowGraph) { - let reverse_postorder_map = cfg.reverse_postorder_ebbs(); - let ebbs = reverse_postorder_map.keys().collect::>(); - let len = reverse_postorder_map.len(); + pub fn new(cfg: &ControlFlowGraph) -> DominatorTree { + let mut ebbs = cfg.postorder_ebbs(); + ebbs.reverse(); + + let len = ebbs.len(); + + // The mappings which designate the dominator tree. + let mut data = EntityMap::with_capacity(len); + + let mut postorder_map = EntityMap::with_capacity(len); + for (i, ebb) in ebbs.iter().enumerate() { + postorder_map[ebb.clone()] = len - i; + } let mut changed = false; if len > 0 { - self.data[ebbs[0]] = Some(ebbs[0]); + data[ebbs[0]] = Some(ebbs[0]); changed = true; } @@ -44,34 +46,42 @@ impl DominatorTree { } // If this predecessor `p` has an idom available find its common // ancestor with the current value of new_idom. - if let Some(_) = self.data[p] { + if let Some(_) = data[p] { new_idom = match new_idom { Some(cur_idom) => { - Some(self.intersect(&reverse_postorder_map, p, cur_idom)) + Some(DominatorTree::intersect(&mut data, + &postorder_map, + p, + cur_idom)) } None => panic!("A 'current idom' should have been set!"), } } } - match self.data[ebb] { + match data[ebb] { None => { - self.data[ebb] = new_idom; + data[ebb] = new_idom; changed = true; } Some(idom) => { // Old idom != New idom if idom != new_idom.unwrap() { - self.data[ebb] = new_idom; + data[ebb] = new_idom; changed = true; } } } } } + DominatorTree { data: data } } /// Find the common dominator of two ebbs. - fn intersect(&self, ordering: &EntityMap, first: Ebb, second: Ebb) -> Ebb { + fn intersect(data: &EntityMap>, + ordering: &EntityMap, + first: Ebb, + second: Ebb) + -> Ebb { let mut a = first; let mut b = second; @@ -81,10 +91,10 @@ impl DominatorTree { // self.data[b] to contain non-None entries. while a != b { while ordering[a] < ordering[b] { - a = self.data[a].unwrap(); + a = data[a].unwrap(); } while ordering[b] < ordering[a] { - b = self.data[b].unwrap(); + b = data[b].unwrap(); } } a @@ -96,9 +106,9 @@ impl DominatorTree { self.data[ebb].clone() } - /// The total number of nodes in the tree. - pub fn len(&self) -> usize { - self.data.len() + /// An iterator across all of the ebbs stored in the tree. + pub fn ebbs(&self) -> Keys { + self.data.keys() } } @@ -114,7 +124,7 @@ mod test { let func = Function::new(); let cfg = ControlFlowGraph::new(&func); let dtree = DominatorTree::new(&cfg); - assert_eq!(dtree.len(), 0); + assert_eq!(None, dtree.ebbs().next()); } #[test] @@ -143,7 +153,6 @@ mod test { let dt = DominatorTree::new(&cfg); assert_eq!(func.layout.entry_block().unwrap(), ebb3); - assert_eq!(dt.len(), cfg.len()); assert_eq!(dt.idom(ebb3).unwrap(), ebb3); assert_eq!(dt.idom(ebb1).unwrap(), ebb3); assert_eq!(dt.idom(ebb2).unwrap(), ebb1); diff --git a/cranelift/src/libcretonne/tests/cfg_traversal.rs b/cranelift/src/libcretonne/tests/cfg_traversal.rs index ae74445dd4..8908b279d8 100644 --- a/cranelift/src/libcretonne/tests/cfg_traversal.rs +++ b/cranelift/src/libcretonne/tests/cfg_traversal.rs @@ -4,6 +4,7 @@ extern crate cton_reader; use self::cton_reader::parser::Parser; use self::cretonne::ir::entities::Ebb; use self::cretonne::cfg::ControlFlowGraph; +use self::cretonne::entity_map::EntityMap; fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) { let func = &Parser::parse(function_source).unwrap()[0]; @@ -11,11 +12,16 @@ fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) let ebbs = ebb_order.iter().map(|n| Ebb::with_number(*n).unwrap()) .collect::>(); - let reverse_postorder_ebbs = cfg.reverse_postorder_ebbs(); + let mut postorder_ebbs = cfg.postorder_ebbs(); + let mut postorder_map = EntityMap::with_capacity(postorder_ebbs.len()); + for (i, ebb) in postorder_ebbs.iter().enumerate() { + postorder_map[ebb.clone()] = i + 1; + } + postorder_ebbs.reverse(); - assert_eq!(reverse_postorder_ebbs.len(), ebbs.len()); - for ebb in reverse_postorder_ebbs.keys() { - assert_eq!(ebb, ebbs[ebbs.len() - reverse_postorder_ebbs[ebb]]); + assert_eq!(postorder_ebbs.len(), ebbs.len()); + for ebb in postorder_ebbs { + assert_eq!(ebb, ebbs[ebbs.len() - postorder_map[ebb]]); } } diff --git a/cranelift/src/libcretonne/tests/dominator_tree.rs b/cranelift/src/libcretonne/tests/dominator_tree.rs index 6820acd152..8e10307a21 100644 --- a/cranelift/src/libcretonne/tests/dominator_tree.rs +++ b/cranelift/src/libcretonne/tests/dominator_tree.rs @@ -10,7 +10,7 @@ fn test_dominator_tree(function_source: &str, idoms: Vec) { let func = &Parser::parse(function_source).unwrap()[0]; let cfg = ControlFlowGraph::new(&func); let dtree = DominatorTree::new(&cfg); - assert_eq!(dtree.len(), idoms.len()); + assert_eq!(dtree.ebbs().collect::>().len(), idoms.len()); for (i, j) in idoms.iter().enumerate() { let ebb = Ebb::with_number(i.clone() as u32); let idom = Ebb::with_number(*j); From 82a85961ff68bd86b7ff98c948b6702f095350be Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 1 Aug 2016 14:45:59 -0700 Subject: [PATCH 0180/3084] Add PrimaryEntityData marker trait. Require this trait on the value type stored in an EntityMap to 'unlock' the methods intended for primary entity maps that are allowed to create references with the 'push method. This prevents accidentally depending on these methods in secondary maps. --- cranelift/src/libcretonne/entity_map.rs | 55 ++++++++++++++++--------- cranelift/src/libcretonne/ir/dfg.rs | 5 ++- cranelift/src/libcretonne/ir/mod.rs | 5 ++- 3 files changed, 44 insertions(+), 21 deletions(-) diff --git a/cranelift/src/libcretonne/entity_map.rs b/cranelift/src/libcretonne/entity_map.rs index a0f4f3fa6b..91ff7505a3 100644 --- a/cranelift/src/libcretonne/entity_map.rs +++ b/cranelift/src/libcretonne/entity_map.rs @@ -2,7 +2,13 @@ //! //! This module defines an `EntityRef` trait that should be implemented by reference types wrapping //! a small integer index. The `EntityMap` data structure uses the dense index space to implement a -//! map with a vector. +//! map with a vector. There are primary and secondary entity maps: +//! +//! - A *primary* `EntityMap` contains the main definition of an entity, and it can be used to +//! allocate new entity references with the `push` method. The values stores in a primary map +//! must implement the `PrimaryEntityData` marker trait. +//! - A *secondary* `EntityMap` contains additional data about entities kept in a primary map. The +//! values need to implement `Clone + Default` traits so the map can be grown with `ensure`. use std::vec::Vec; use std::default::Default; @@ -39,13 +45,6 @@ pub trait EntityRef: Copy + Eq { } /// A mapping `K -> V` for densely indexed entity references. -/// -/// A *primary* `EntityMap` contains the main definition of an entity, and it can be used to -/// allocate new entity references with the `push` method. -/// -/// A *secondary* `EntityMap` contains additional data about entities kept in a primary map. The -/// values need to implement `Clone + Default` traits so the map can be grown with `ensure`. -/// #[derive(Debug)] pub struct EntityMap where K: EntityRef @@ -71,17 +70,6 @@ impl EntityMap k.index() < self.elems.len() } - /// Append `v` to the mapping, assigning a new key which is returned. - pub fn push(&mut self, v: V) -> K { - let k = K::new(self.elems.len()); - self.elems.push(v); - k - } - - pub fn len(&self) -> usize { - self.elems.len() - } - /// Iterate over all the keys in this map. pub fn keys(&self) -> Keys { Keys { @@ -92,6 +80,33 @@ impl EntityMap } } +/// A marker trait for data stored in primary entity maps. +/// +/// A primary entity map can be used to allocate new entity references with the `push` method. It +/// is important that entity references can't be created anywhere else, so the data stored in a +/// primary entity map must be tagged as `PrimaryEntityData` to unlock the `push` method. +pub trait PrimaryEntityData {} + +/// Additional methods for primary entry maps only. +/// +/// These are identified by the `PrimaryEntityData` marker trait. +impl EntityMap + where K: EntityRef, + V: PrimaryEntityData +{ + /// Append `v` to the mapping, assigning a new key which is returned. + pub fn push(&mut self, v: V) -> K { + let k = K::new(self.elems.len()); + self.elems.push(v); + k + } + + /// Get the total number of entity references created. + pub fn len(&self) -> usize { + self.elems.len() + } +} + /// Additional methods for value types that implement `Clone` and `Default`. /// /// When the value type implements these additional traits, the `EntityMap` can be resized @@ -193,6 +208,8 @@ mod tests { } } + impl PrimaryEntityData for isize {} + #[test] fn basic() { let r0 = E(0); diff --git a/cranelift/src/libcretonne/ir/dfg.rs b/cranelift/src/libcretonne/ir/dfg.rs index aba45354d7..4776220a6f 100644 --- a/cranelift/src/libcretonne/ir/dfg.rs +++ b/cranelift/src/libcretonne/ir/dfg.rs @@ -1,6 +1,6 @@ //! Data flow graph tracking Instructions, Values, and EBBs. -use entity_map::EntityMap; +use entity_map::{EntityMap, PrimaryEntityData}; use ir::entities::{Ebb, Inst, Value, NO_VALUE, ExpandedValue}; use ir::instructions::InstructionData; use ir::types::Type; @@ -33,6 +33,9 @@ pub struct DataFlowGraph { extended_values: Vec, } +impl PrimaryEntityData for InstructionData {} +impl PrimaryEntityData for EbbData {} + impl DataFlowGraph { /// Create a new empty `DataFlowGraph`. pub fn new() -> DataFlowGraph { diff --git a/cranelift/src/libcretonne/ir/mod.rs b/cranelift/src/libcretonne/ir/mod.rs index 8d5ed99742..f0e8eda741 100644 --- a/cranelift/src/libcretonne/ir/mod.rs +++ b/cranelift/src/libcretonne/ir/mod.rs @@ -10,7 +10,7 @@ pub mod dfg; pub mod layout; use ir::types::{FunctionName, Signature}; -use entity_map::{EntityRef, EntityMap}; +use entity_map::{EntityRef, EntityMap, PrimaryEntityData}; use ir::entities::{StackSlot, JumpTable}; use ir::jumptable::JumpTableData; use ir::dfg::DataFlowGraph; @@ -39,6 +39,9 @@ pub struct Function { pub layout: Layout, } +// Tag JumpTableData as a primary entity so jump_tables above . +impl PrimaryEntityData for JumpTableData {} + impl Function { /// Create a function with the given name and signature. pub fn with_name_signature(name: FunctionName, sig: Signature) -> Function { From a67ec211a451615848ae59cd73238b6b3dd10ea0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Jul 2016 14:50:19 -0700 Subject: [PATCH 0181/3084] Document binary encodings. Describe the meta-language data structures that are built to represent instruction encodings. Begin a metaref glossary. --- cranelift/docs/metaref.rst | 127 ++++++++++++++++++++++++++++++++++++- 1 file changed, 126 insertions(+), 1 deletion(-) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index fc9c5e4e29..058cc6459f 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -203,11 +203,99 @@ This means that the type of an input operand can either be computed from the controlling type variable, or it can vary independently of the other operands. +Encodings +========= + +Encodings describe how Cretonne instructions are mapped to binary machine code +for the target architecture. After the lealization pass, all remaining +instructions are expected to map 1-1 to native instruction encodings. Cretonne +instructions that can't be encoded for the current architecture are called +:term:`illegal instruction`\s. + +Some instruction set architectures have different :term:`CPU mode`\s with +incompatible encodings. For example, a modern ARMv8 CPU might support three +different CPU modes: *A64* where instructions are encoded in 32 bits, *A32* +where all instuctions are 32 bits, and *T32* which has a mix of 16-bit and +32-bit instruction encodings. These are incompatible encoding spaces, and while +an :cton:inst:`iadd` instruction can be encoded in 32 bits in each of them, it's +not the same 32 bits. It's a judgement call if CPU modes should be modelled as +separate targets, or as sub-modes of the same target. In the ARMv8 case, the +different register banks means that it makes sense to model A64 as a separate +target architecture, while A32 and T32 are CPU modes of the 32-bit ARM target. + +In a given CPU mode, there may be multiple valid encodings of the same +instruction. Both RISC-V and ARMv8's T32 mode have 32-bit encodings of all +instructions with 16-bit encodings available for some opcodes if certain +constraints are satisfied. + +Encodings are guarded by :term:`sub-target predicate`\s. For example, the RISC-V +"C" extension which specifies the compressed encodings may not be supported, and +a predicate would be used to disable all of the 16-bit encodings in that case. +This can also affect whether an instruction is legal. For example, x86 has a +predicate that controls the SSE 4.1 instruction encodings. When that predicate +is false, the SSE 4.1 instructions are not available. + +Encodings also have a :term:`instruction predicate` which depends on the +specific values of the instruction's immediate fields. This is used to ensure +that immediate address offsets are within range, for example. The instructions +in the base Cretonne instruction set can often represent a wider range of +immediates than any specific encoding. The fixed-size RISC-style encodings tend +to have more range limitations than CISC-style variable length encodings like +x86. + +The diagram below shows the relationship between the classes involved in +specifying instruction encodings: + +.. digraph:: encoding + + node [shape=record] + CPUMode -> Target + EncRecipe -> CPUMode + EncRecipe -> SubtargetPred + EncRecipe -> InstrFormat + EncRecipe -> InstrPred + Encoding [label="{Encoding|Opcode+TypeVars}"] + Encoding -> EncRecipe [label="+EncBits"] + Encoding -> SubtargetPred + Encoding -> InstrPred + Encoding -> Opcode + Opcode -> InstrFormat + +An :py:class:`Encoding` instance specifies the encoding of a concrete +instruction. The following properties are used to select instructions to be +encoded: + +- An opcode, i.e. :cton:inst:`iadd_imm`, that must match the instruction's + opcode. +- Values for any type variables if the opcode represents a polymorphic + instruction. +- An :term:`instruction predicate` that must be satisfied by the instruction's + immediate operands. +- A :term:`sub-target predicate` that must be satisfied by the currently active + sub-target. +- :term:`Register constraint`\s that must be satisfied by the instruction's value + operands and results. + +An encoding specifies an *encoding recipe* along with some *encoding bits* that +the recipe can use for native opcode fields etc. The encoding recipe has +additional constraints that must be satisfied: + +- The CPU mode that must be active to enable encodings. +- An :py:class:`InstructionFormat` that must match the format required by the + opcodes of any encodings that use this recipe. +- An additional :term:`instruction predicate`. +- An additional :term:`sub-target predicate`. + +The additional predicates in the :py:class:`EncRecipe` are merged with the +per-encoding predicates when generating the encoding matcher code. Often +encodings only need the recipe predicates. + + Targets ======= Cretonne can be compiled with support for multiple target instruction set -architectures. Each ISA is represented by a :py:class`cretonne.Target` instance. +architectures. Each ISA is represented by a :py:class:`cretonne.Target` instance. .. autoclass:: Target @@ -218,3 +306,40 @@ The definitions for each supported target live in a package under :members: .. automodule:: target.riscv + + +Glossary +======== + +.. glossary:: + + Illegal instruction + An instruction is considered illegal if there is no encoding available + for the current CPU mode. The legality of an instruction depends on the + value of :term:`sub-target predicate`\s, so it can't always be + determined ahead of time. + + CPU mode + Every target defines one or more CPU modes that determine how the CPU + decodes binary instructions. Some CPUs can switch modes dynamically with + a branch instruction (like ARM/Thumb), while other modes are + process-wide (like x86 32/64-bit). + + Sub-target predicate + A predicate that depends on the current sub-target configuration. + Examples are "Use SSE 4.1 instructions", "Use RISC-V compressed + encodings". Sub-target predicates can depend on both detected CPU + features and configuration settings. + + Instruction predicate + A predicate that depends on the immediate fields of an instruction. An + example is "the load address offset must be a 10-bit signed integer". + Instruction predicates do not depend on the registers selected for value + operands. + + Register constraint + Value operands and results correspond to machine registers. Encodings may + constrain operands to either a fixed register or a register class. There + may also be register constraints between operands, for example some + encodings require that the result register is one of the input + registers. From eed6adb413e5537fc4ebc6e71e3b480af335246a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 3 Aug 2016 11:20:13 -0700 Subject: [PATCH 0182/3084] Add a CPUMode meta-language class. --- cranelift/docs/metaref.rst | 2 ++ meta/cretonne/__init__.py | 17 +++++++++++++++++ meta/target/riscv/__init__.py | 6 +++++- 3 files changed, 24 insertions(+), 1 deletion(-) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 058cc6459f..b7f607f91f 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -228,6 +228,8 @@ instruction. Both RISC-V and ARMv8's T32 mode have 32-bit encodings of all instructions with 16-bit encodings available for some opcodes if certain constraints are satisfied. +.. autoclass:: CPUMode + Encodings are guarded by :term:`sub-target predicate`\s. For example, the RISC-V "C" extension which specifies the compressed encodings may not be supported, and a predicate would be used to disable all of the 16-bit encodings in that case. diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 9660e9e602..d5e6530280 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -656,6 +656,23 @@ class Target(object): self.name = name self.instruction_groups = instrution_groups + +class CPUMode(object): + """ + A CPU mode determines which instruction encodings are active. + + All instruction encodings are associated with exactly one `CPUMode`, and + all CPU modes are associated with exactly one `Target`. + + :param name: Short mnemonic name for the CPU mode. + :param target: Associated `Target`. + """ + + def __init__(self, name, target): + self.name = name + self.target = target + + # Import the fixed instruction formats now so they can be added to the # registry. importlib.import_module('cretonne.formats') diff --git a/meta/target/riscv/__init__.py b/meta/target/riscv/__init__.py index bdb5b63091..918602c78e 100644 --- a/meta/target/riscv/__init__.py +++ b/meta/target/riscv/__init__.py @@ -25,7 +25,11 @@ RV32G / RV64G """ -from cretonne import Target +from cretonne import Target, CPUMode import cretonne.base target = Target('riscv', [cretonne.base.instructions]) + +# CPU modes for 32-bit and 64-bit operation. +RV32 = CPUMode('RV32', target) +RV64 = CPUMode('RV64', target) From 66f14138bb8795c483cd126ad1b56c6211d3f193 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 3 Aug 2016 12:06:21 -0700 Subject: [PATCH 0183/3084] Add an EncRecipe meta-language class. Move the CPUMode reference from EncRecipe to the Encoding itself, allowing EncRecipes to be shared between CPU modes. At least RISC-V should be able to share some recipes between RV32 and RV64 modes. --- cranelift/docs/metaref.rst | 8 +++++--- meta/cretonne/__init__.py | 16 ++++++++++++++++ 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index b7f607f91f..3b9ac07f70 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -251,17 +251,17 @@ specifying instruction encodings: .. digraph:: encoding node [shape=record] - CPUMode -> Target - EncRecipe -> CPUMode EncRecipe -> SubtargetPred EncRecipe -> InstrFormat EncRecipe -> InstrPred Encoding [label="{Encoding|Opcode+TypeVars}"] Encoding -> EncRecipe [label="+EncBits"] + Encoding -> CPUMode Encoding -> SubtargetPred Encoding -> InstrPred Encoding -> Opcode Opcode -> InstrFormat + CPUMode -> Target An :py:class:`Encoding` instance specifies the encoding of a concrete instruction. The following properties are used to select instructions to be @@ -273,6 +273,7 @@ encoded: instruction. - An :term:`instruction predicate` that must be satisfied by the instruction's immediate operands. +- The CPU mode that must be active. - A :term:`sub-target predicate` that must be satisfied by the currently active sub-target. - :term:`Register constraint`\s that must be satisfied by the instruction's value @@ -282,7 +283,6 @@ An encoding specifies an *encoding recipe* along with some *encoding bits* that the recipe can use for native opcode fields etc. The encoding recipe has additional constraints that must be satisfied: -- The CPU mode that must be active to enable encodings. - An :py:class:`InstructionFormat` that must match the format required by the opcodes of any encodings that use this recipe. - An additional :term:`instruction predicate`. @@ -292,6 +292,8 @@ The additional predicates in the :py:class:`EncRecipe` are merged with the per-encoding predicates when generating the encoding matcher code. Often encodings only need the recipe predicates. +.. autoclass:: EncRecipe + Targets ======= diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index d5e6530280..4ee491f806 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -673,6 +673,22 @@ class CPUMode(object): self.target = target +class EncRecipe(object): + """ + A recipe for encoding instructions with a given format. + + Many different instructions can be encoded by the same recipe, but they + must all have the same instruction format. + + :param name: Short mnemonic name for this recipe. + :param format: All encoded instructions must have this + :py:class:`InstructionFormat`. + """ + + def __init__(self, name, format): + self.name = name + self.format = format + # Import the fixed instruction formats now so they can be added to the # registry. importlib.import_module('cretonne.formats') From 4987282bbbc8916561134b01a7776d8421985335 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 3 Aug 2016 15:58:41 -0700 Subject: [PATCH 0184/3084] Add an Encoding meta-language class. Start adding some RISC-V encodings too as a way of testing the ergonomics. --- meta/cretonne/__init__.py | 50 ++++++++++++++++++++++++++++++++++ meta/target/riscv/__init__.py | 10 +++---- meta/target/riscv/defs.py | 14 ++++++++++ meta/target/riscv/encodings.py | 21 ++++++++++++++ meta/target/riscv/recipes.py | 44 ++++++++++++++++++++++++++++++ 5 files changed, 133 insertions(+), 6 deletions(-) create mode 100644 meta/target/riscv/defs.py create mode 100644 meta/target/riscv/encodings.py create mode 100644 meta/target/riscv/recipes.py diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 4ee491f806..483fb1f8e3 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -671,6 +671,16 @@ class CPUMode(object): def __init__(self, name, target): self.name = name self.target = target + self.encodings = [] + + def enc(self, *args, **kwargs): + """ + Add a new encoding to this CPU mode. + + Arguments are the `Encoding constructor arguments, except for the first + `CPUMode argument which is implied. + """ + self.encodings.append(Encoding(self, *args, **kwargs)) class EncRecipe(object): @@ -689,6 +699,46 @@ class EncRecipe(object): self.name = name self.format = format + +class Encoding(object): + """ + Encoding for a concrete instruction. + + An `Encoding` object ties an instruction opcode with concrete type + variables together with and encoding recipe and encoding bits. + + :param cpumode: The CPU mode where the encoding is active. + :param inst: The :py:class:`Instruction` being encoded. + :param typevars: Concete types for `inst`'s type variables, if any. + :param recipe: The :py:class:`EncRecipe` to use. + :param encbits: Additional encoding bits to be interpreted by `recipe`. + """ + + def __init__(self, cpumode, inst, typevars, recipe, encbits): + assert isinstance(cpumode, CPUMode) + assert isinstance(inst, Instruction) + assert isinstance(recipe, EncRecipe) + self.cpumode = cpumode + assert inst.format == recipe.format, ( + "Format {} must match recipe: {}".format( + inst.format, recipe.format)) + self.inst = inst + self.typevars = self._to_type_tuple(typevars) + self.recipe = recipe + self.encbits = encbits + + @staticmethod + def _to_type_tuple(x): + # Allow a single ValueType instance instead of the awkward singleton + # tuple syntax. + if isinstance(x, ValueType): + x = (x,) + else: + x = tuple(x) + for ty in x: + assert isinstance(ty, ValueType) + return x + # Import the fixed instruction formats now so they can be added to the # registry. importlib.import_module('cretonne.formats') diff --git a/meta/target/riscv/__init__.py b/meta/target/riscv/__init__.py index 918602c78e..48d916f089 100644 --- a/meta/target/riscv/__init__.py +++ b/meta/target/riscv/__init__.py @@ -25,11 +25,9 @@ RV32G / RV64G """ -from cretonne import Target, CPUMode -import cretonne.base +import defs +import encodings -target = Target('riscv', [cretonne.base.instructions]) +# Re-export the primary target definition. +target = defs.target -# CPU modes for 32-bit and 64-bit operation. -RV32 = CPUMode('RV32', target) -RV64 = CPUMode('RV64', target) diff --git a/meta/target/riscv/defs.py b/meta/target/riscv/defs.py new file mode 100644 index 0000000000..aab9279360 --- /dev/null +++ b/meta/target/riscv/defs.py @@ -0,0 +1,14 @@ +""" +RISC-V definitions. + +Commonly used definitions. +""" + +from cretonne import Target, CPUMode +import cretonne.base + +target = Target('riscv', [cretonne.base.instructions]) + +# CPU modes for 32-bit and 64-bit operation. +RV32 = CPUMode('RV32', target) +RV64 = CPUMode('RV64', target) diff --git a/meta/target/riscv/encodings.py b/meta/target/riscv/encodings.py new file mode 100644 index 0000000000..0177cb57f9 --- /dev/null +++ b/meta/target/riscv/encodings.py @@ -0,0 +1,21 @@ +""" +RISC-V Encodings. +""" +from cretonne import base +from cretonne.types import i32, i64 +from defs import RV32, RV64 +from recipes import OP, R + +# Basic arithmetic binary instructions are encoded in an R-type instruction. +for inst, f3, f7 in [ + (base.iadd, 0b000, 0b0000000), + (base.isub, 0b000, 0b0100000), + (base.ishl, 0b001, 0b0000000), + (base.ushr, 0b101, 0b0000000), + (base.sshr, 0b101, 0b0100000), + (base.bxor, 0b100, 0b0000000), + (base.bor, 0b110, 0b0000000), + (base.band, 0b111, 0b0000000) + ]: + RV32.enc(inst, i32, R, OP(f3, f7)) + RV64.enc(inst, i64, R, OP(f3, f7)) diff --git a/meta/target/riscv/recipes.py b/meta/target/riscv/recipes.py new file mode 100644 index 0000000000..ed96836cd4 --- /dev/null +++ b/meta/target/riscv/recipes.py @@ -0,0 +1,44 @@ +""" +RISC-V Encoding recipes. + +The encoding recipes defined here more or less correspond to the RISC-V native +instruction formats described in the reference: + + The RISC-V Instruction Set Manual + Volume I: User-Level ISA + Version 2.1 +""" +from cretonne import EncRecipe +from cretonne.formats import Binary + +# The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit +# instructions have 11 as the two low bits, with bits 6:2 determining the base +# opcode. +# +# Encbits for the 32-bit recipes are opcode[6:2] | (funct3 << 5) | ... +# The functions below encode the encbits. + +def LOAD(funct3): + assert funct3 <= 0b111 + return 0b00000 | (funct3 << 5) + +def STORE(funct3): + assert funct3 <= 0b111 + return 0b01000 | (funct3 << 5) + +def BRANCH(funct3): + assert funct3 <= 0b111 + return 0b11000 | (funct3 << 5) + +def OPIMM(funct3): + assert funct3 <= 0b111 + return 0b00100 | (funct3 << 5) + +def OP(funct3, funct7): + assert funct3 <= 0b111 + assert funct7 <= 0b1111111 + return 0b01100 | (funct3 << 5) | (funct7 << 8) + +# R-type 32-bit instructions: These are mostly binary arithmetic instructions. +# The encbits are `opcode[6:2] | (funct3 << 5) | (funct7 << 8) +R = EncRecipe('R', Binary) From a1cc8af18682f48c1a6da251cdc9199f28373450 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 3 Aug 2016 16:30:47 -0700 Subject: [PATCH 0185/3084] Use dot syntax to bind type variables on instructions. Encodings need to refer to concrete instances of polymorphic instructions by binding type variables. Allow dot syntax like iadd.i32 to do that. --- meta/cretonne/__init__.py | 68 +++++++++++++++++++++++++++++++--- meta/target/riscv/encodings.py | 4 +- 2 files changed, 64 insertions(+), 8 deletions(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 483fb1f8e3..01b74747b5 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -115,10 +115,15 @@ class ValueType(object): or one of its subclasses. """ + # map name -> ValueType. + _registry = dict() + def __init__(self, name, membytes, doc): self.name = name self.membytes = membytes self.__doc__ = doc + assert name not in ValueType._registry + ValueType._registry[name] = self def __str__(self): return self.name @@ -133,6 +138,13 @@ class ValueType(object): def free_typevar(self): return None + @staticmethod + def by_name(name): + if name in ValueType._registry: + return ValueType._registry[name] + else: + raise AttributeError("No type named '{}'".format(name)) + class ScalarType(ValueType): """ @@ -637,6 +649,47 @@ class Instruction(object): assert isinstance(op, Operand) return x + def bind(self, *args): + """ + Bind a polymorphic instruction to a concrete list of type variable + values. + """ + assert self.is_polymorphic + return BoundInstruction(self, args) + + def __getattr__(self, name): + """ + Bind a polymorphic instruction to a single type variable with dot + syntax: + + >>> iadd.i32 + """ + return self.bind(ValueType.by_name(name)) + + +class BoundInstruction(object): + """ + A polymorphic `Instruction` bound to concrete type variables. + """ + + def __init__(self, inst, typevars): + self.inst = inst + self.typevars = typevars + + def bind(self, *args): + """ + Bind additional typevars. + """ + return BoundInstruction(self.inst, self.typevars + args) + + def __getattr__(self, name): + """ + Bind an additional typevar dot syntax: + + >>> uext.i32.i8 + """ + return self.bind(ValueType.by_name(name)) + # Defining targets @@ -708,22 +761,25 @@ class Encoding(object): variables together with and encoding recipe and encoding bits. :param cpumode: The CPU mode where the encoding is active. - :param inst: The :py:class:`Instruction` being encoded. - :param typevars: Concete types for `inst`'s type variables, if any. + :param inst: The :py:class:`Instruction` or :py:class:`BoundInstruction` + being encoded. :param recipe: The :py:class:`EncRecipe` to use. :param encbits: Additional encoding bits to be interpreted by `recipe`. """ - def __init__(self, cpumode, inst, typevars, recipe, encbits): + def __init__(self, cpumode, inst, recipe, encbits): assert isinstance(cpumode, CPUMode) - assert isinstance(inst, Instruction) + if isinstance(inst, BoundInstruction): + real_inst = inst.inst + else: + real_inst = inst + assert isinstance(real_inst, Instruction) assert isinstance(recipe, EncRecipe) self.cpumode = cpumode - assert inst.format == recipe.format, ( + assert real_inst.format == recipe.format, ( "Format {} must match recipe: {}".format( inst.format, recipe.format)) self.inst = inst - self.typevars = self._to_type_tuple(typevars) self.recipe = recipe self.encbits = encbits diff --git a/meta/target/riscv/encodings.py b/meta/target/riscv/encodings.py index 0177cb57f9..fa3f7d055d 100644 --- a/meta/target/riscv/encodings.py +++ b/meta/target/riscv/encodings.py @@ -17,5 +17,5 @@ for inst, f3, f7 in [ (base.bor, 0b110, 0b0000000), (base.band, 0b111, 0b0000000) ]: - RV32.enc(inst, i32, R, OP(f3, f7)) - RV64.enc(inst, i64, R, OP(f3, f7)) + RV32.enc(inst.i32, R, OP(f3, f7)) + RV64.enc(inst.i64, R, OP(f3, f7)) From 5a8293427d38666f6cbf431e7c28df8ee9065014 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 4 Aug 2016 10:21:48 -0700 Subject: [PATCH 0186/3084] Verify that type variables have been fully bound. The shift instructions have two type variables since the shift amount can be a differently sized integer. Fix the RISC-V shift encodings to reflect this, and allow i64 registers to be shifted by an i32 amount. --- meta/cretonne/__init__.py | 37 ++++++++++++++++++++++++++-------- meta/target/riscv/encodings.py | 14 ++++++++++--- 2 files changed, 40 insertions(+), 11 deletions(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 01b74747b5..7214cac75f 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -666,6 +666,17 @@ class Instruction(object): """ return self.bind(ValueType.by_name(name)) + def fully_bound(self): + """ + Verify that all typevars have been bound, and return a + `(inst, typevars)` pair. + + This version in `Instruction` itself allows non-polymorphic + instructions to duck-type as `BoundInstruction`\s. + """ + assert not self.is_polymorphic, self + return (self, ()) + class BoundInstruction(object): """ @@ -675,6 +686,10 @@ class BoundInstruction(object): def __init__(self, inst, typevars): self.inst = inst self.typevars = typevars + assert len(typevars) <= 1 + len(inst.other_typevars) + + def __str__(self): + return '.'.join([self.inst.name,] + map(str, self.typevars)) def bind(self, *args): """ @@ -690,6 +705,17 @@ class BoundInstruction(object): """ return self.bind(ValueType.by_name(name)) + def fully_bound(self): + """ + Verify that all typevars have been bound, and return a + `(inst, typevars)` pair. + """ + if len(self.typevars) < 1 + len(self.inst.other_typevars): + unb = ', '.join(str(tv) for tv in self.inst.other_typevars[len(self.typevars) - 1:]) + raise AssertionError("Unbound typevar {} in {}".format(unb, self)) + assert len(self.typevars) == 1 + len(self.inst.other_typevars) + return (self.inst, self.typevars) + # Defining targets @@ -769,17 +795,12 @@ class Encoding(object): def __init__(self, cpumode, inst, recipe, encbits): assert isinstance(cpumode, CPUMode) - if isinstance(inst, BoundInstruction): - real_inst = inst.inst - else: - real_inst = inst - assert isinstance(real_inst, Instruction) assert isinstance(recipe, EncRecipe) + self.inst, self.typevars = inst.fully_bound() self.cpumode = cpumode - assert real_inst.format == recipe.format, ( + assert self.inst.format == recipe.format, ( "Format {} must match recipe: {}".format( - inst.format, recipe.format)) - self.inst = inst + self.inst.format, recipe.format)) self.recipe = recipe self.encbits = encbits diff --git a/meta/target/riscv/encodings.py b/meta/target/riscv/encodings.py index fa3f7d055d..fb6602184c 100644 --- a/meta/target/riscv/encodings.py +++ b/meta/target/riscv/encodings.py @@ -10,12 +10,20 @@ from recipes import OP, R for inst, f3, f7 in [ (base.iadd, 0b000, 0b0000000), (base.isub, 0b000, 0b0100000), - (base.ishl, 0b001, 0b0000000), - (base.ushr, 0b101, 0b0000000), - (base.sshr, 0b101, 0b0100000), (base.bxor, 0b100, 0b0000000), (base.bor, 0b110, 0b0000000), (base.band, 0b111, 0b0000000) ]: RV32.enc(inst.i32, R, OP(f3, f7)) RV64.enc(inst.i64, R, OP(f3, f7)) + +# Dynamic shifts have the same masking semantics as the cton base instructions +for inst, f3, f7 in [ + (base.ishl, 0b001, 0b0000000), + (base.ushr, 0b101, 0b0000000), + (base.sshr, 0b101, 0b0100000), + ]: + RV32.enc(inst.i32.i32, R, OP(f3, f7)) + RV64.enc(inst.i64.i64, R, OP(f3, f7)) + # Allow i32 shift amounts in 64-bit shifts. + RV64.enc(inst.i64.i32, R, OP(f3, f7)) From afdc4960d7ad1ddfd52be43b8a6b215311dff0cf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 4 Aug 2016 11:28:38 -0700 Subject: [PATCH 0187/3084] Don't sphinx-autobuild on Vim .swp file changes. --- cranelift/docs/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/docs/Makefile b/cranelift/docs/Makefile index 598a13b496..56137e8b2c 100644 --- a/cranelift/docs/Makefile +++ b/cranelift/docs/Makefile @@ -58,7 +58,7 @@ html: @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." autohtml: html - $(SPHINXABUILD) -z ../meta -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html + $(SPHINXABUILD) -z ../meta --ignore '*.swp' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml From 6d786113db9125ca2645af5856f95a717b470f48 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 4 Aug 2016 11:39:25 -0700 Subject: [PATCH 0188/3084] Add an empty isa/riscv module scaffold. Targeted ISAs will be defined as sub-modules of isa. --- cranelift/src/libcretonne/isa/mod.rs | 6 ++++++ cranelift/src/libcretonne/isa/riscv/mod.rs | 1 + cranelift/src/libcretonne/lib.rs | 1 + 3 files changed, 8 insertions(+) create mode 100644 cranelift/src/libcretonne/isa/mod.rs create mode 100644 cranelift/src/libcretonne/isa/riscv/mod.rs diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs new file mode 100644 index 0000000000..e2f48e7bbe --- /dev/null +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -0,0 +1,6 @@ +//! Instruction Set Architectures. +//! +//! The sub-modules of this `isa` module provide definitions for the instruction sets that Cretonne +//! can target. + +pub mod riscv; diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs new file mode 100644 index 0000000000..4c24cbb8b7 --- /dev/null +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -0,0 +1 @@ +//! RISC-V Instruction Set Architecture. diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 34151d187b..9d696fab53 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -8,6 +8,7 @@ pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); pub mod ir; +pub mod isa; pub mod write; pub mod cfg; pub mod dominator_tree; From 1a4d07d437d2685f6fc3faeeefff04f5587dbc0e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 4 Aug 2016 11:50:19 -0700 Subject: [PATCH 0189/3084] Rename meta/target -> meta/isa. Clarify terminology by always referring to a 'Target ISA' instead of just 'Target'. Use 'isa' as a module name instead of 'target' both in Rust and Python code. This is only to clarify terminology and not at all because Cargo insists on using the 'target' sub-directory for build products. Oh, no. Not at all. --- cranelift/docs/metaref.rst | 10 +++++----- meta/build.py | 6 +++--- meta/cretonne/__init__.py | 14 +++++++------- meta/gen_instr.py | 12 ++++++------ meta/isa/__init__.py | 17 +++++++++++++++++ meta/{target => isa}/riscv/__init__.py | 4 ++-- meta/isa/riscv/defs.py | 14 ++++++++++++++ meta/{target => isa}/riscv/encodings.py | 0 meta/{target => isa}/riscv/recipes.py | 0 meta/target/__init__.py | 17 ----------------- meta/target/riscv/defs.py | 14 -------------- 11 files changed, 54 insertions(+), 54 deletions(-) create mode 100644 meta/isa/__init__.py rename meta/{target => isa}/riscv/__init__.py (90%) create mode 100644 meta/isa/riscv/defs.py rename meta/{target => isa}/riscv/encodings.py (100%) rename meta/{target => isa}/riscv/recipes.py (100%) delete mode 100644 meta/target/__init__.py delete mode 100644 meta/target/riscv/defs.py diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 3b9ac07f70..4ac81096cc 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -299,17 +299,17 @@ Targets ======= Cretonne can be compiled with support for multiple target instruction set -architectures. Each ISA is represented by a :py:class:`cretonne.Target` instance. +architectures. Each ISA is represented by a :py:class:`cretonne.TargetISA` instance. -.. autoclass:: Target +.. autoclass:: TargetISA The definitions for each supported target live in a package under -:file:`meta/target`. +:file:`meta/isa`. -.. automodule:: target +.. automodule:: isa :members: -.. automodule:: target.riscv +.. automodule:: isa.riscv Glossary diff --git a/meta/build.py b/meta/build.py index 24d9f286ee..5dadc112e2 100644 --- a/meta/build.py +++ b/meta/build.py @@ -3,7 +3,7 @@ # This script is run from src/libcretonne/build.rs to generate Rust files. import argparse -import target +import isa import gen_instr import gen_build_deps @@ -13,7 +13,7 @@ parser.add_argument('--out-dir', help='set output directory') args = parser.parse_args() out_dir = args.out_dir -targets = target.all_targets() +isas = isa.all_isas() -gen_instr.generate(targets, out_dir) +gen_instr.generate(isas, out_dir) gen_build_deps.generate() diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 7214cac75f..ee523cfec5 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -717,14 +717,14 @@ class BoundInstruction(object): return (self.inst, self.typevars) -# Defining targets +# Defining target ISAs. -class Target(object): +class TargetISA(object): """ A target instruction set architecture. - The `Target` class collects everything known about a target ISA. + The `TargetISA` class collects everything known about a target ISA. :param name: Short mnemonic name for the ISA. :param instruction_groups: List of `InstructionGroup` instances that are @@ -741,15 +741,15 @@ class CPUMode(object): A CPU mode determines which instruction encodings are active. All instruction encodings are associated with exactly one `CPUMode`, and - all CPU modes are associated with exactly one `Target`. + all CPU modes are associated with exactly one `TargetISA`. :param name: Short mnemonic name for the CPU mode. - :param target: Associated `Target`. + :param target: Associated `TargetISA`. """ - def __init__(self, name, target): + def __init__(self, name, isa): self.name = name - self.target = target + self.isa = isa self.encodings = [] def enc(self, *args, **kwargs): diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 590dd13659..64c726ad9d 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -161,11 +161,11 @@ def gen_instruction_data_impl(fmt): .format(i)) -def collect_instr_groups(targets): +def collect_instr_groups(isas): seen = set() groups = [] - for t in targets: - for g in t.instruction_groups: + for isa in isas: + for g in isa.instruction_groups: if g not in seen: groups.append(g) seen.add(g) @@ -181,7 +181,7 @@ def gen_opcodes(groups, fmt): fmt.doc_comment('An instruction opcode.') fmt.doc_comment('') - fmt.doc_comment('All instructions from all supported targets are present.') + fmt.doc_comment('All instructions from all supported ISAs are present.') fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug)]') instrs = [] with fmt.indented('pub enum Opcode {', '}'): @@ -360,8 +360,8 @@ def gen_type_constraints(fmt, instrs): fmt.line('OperandConstraint::{},'.format(c)) -def generate(targets, out_dir): - groups = collect_instr_groups(targets) +def generate(isas, out_dir): + groups = collect_instr_groups(isas) # opcodes.rs fmt = srcgen.Formatter() diff --git a/meta/isa/__init__.py b/meta/isa/__init__.py new file mode 100644 index 0000000000..b566a521ef --- /dev/null +++ b/meta/isa/__init__.py @@ -0,0 +1,17 @@ +""" +Cretonne target ISA definitions +------------------------------- + +The :py:mod:`isa` package contains sub-packages for each target instruction set +architecture supported by Cretonne. +""" + +from . import riscv + + +def all_isas(): + """ + Get a list of all the supported target ISAs. Each target ISA is represented + as a :py:class:`cretonne.TargetISA` instance. + """ + return [riscv.isa] diff --git a/meta/target/riscv/__init__.py b/meta/isa/riscv/__init__.py similarity index 90% rename from meta/target/riscv/__init__.py rename to meta/isa/riscv/__init__.py index 48d916f089..7d51b69b03 100644 --- a/meta/target/riscv/__init__.py +++ b/meta/isa/riscv/__init__.py @@ -28,6 +28,6 @@ RV32G / RV64G import defs import encodings -# Re-export the primary target definition. -target = defs.target +# Re-export the primary target ISA definition. +isa = defs.isa diff --git a/meta/isa/riscv/defs.py b/meta/isa/riscv/defs.py new file mode 100644 index 0000000000..6e009ccdf4 --- /dev/null +++ b/meta/isa/riscv/defs.py @@ -0,0 +1,14 @@ +""" +RISC-V definitions. + +Commonly used definitions. +""" + +from cretonne import TargetISA, CPUMode +import cretonne.base + +isa = TargetISA('riscv', [cretonne.base.instructions]) + +# CPU modes for 32-bit and 64-bit operation. +RV32 = CPUMode('RV32', isa) +RV64 = CPUMode('RV64', isa) diff --git a/meta/target/riscv/encodings.py b/meta/isa/riscv/encodings.py similarity index 100% rename from meta/target/riscv/encodings.py rename to meta/isa/riscv/encodings.py diff --git a/meta/target/riscv/recipes.py b/meta/isa/riscv/recipes.py similarity index 100% rename from meta/target/riscv/recipes.py rename to meta/isa/riscv/recipes.py diff --git a/meta/target/__init__.py b/meta/target/__init__.py deleted file mode 100644 index 1f9f7a0565..0000000000 --- a/meta/target/__init__.py +++ /dev/null @@ -1,17 +0,0 @@ -""" -Cretonne target definitions ---------------------------- - -The :py:mod:`target` package contains sub-packages for each target instruction -set architecture supported by Cretonne. -""" - -from . import riscv - - -def all_targets(): - """ - Get a list of all the supported targets. Each target is represented as a - :py:class:`cretonne.Target` instance. - """ - return [riscv.target] diff --git a/meta/target/riscv/defs.py b/meta/target/riscv/defs.py deleted file mode 100644 index aab9279360..0000000000 --- a/meta/target/riscv/defs.py +++ /dev/null @@ -1,14 +0,0 @@ -""" -RISC-V definitions. - -Commonly used definitions. -""" - -from cretonne import Target, CPUMode -import cretonne.base - -target = Target('riscv', [cretonne.base.instructions]) - -# CPU modes for 32-bit and 64-bit operation. -RV32 = CPUMode('RV32', target) -RV64 = CPUMode('RV64', target) From c679a2e746b836a17ea9a383247ebcbfd3fcb6b7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 5 Aug 2016 09:55:53 -0700 Subject: [PATCH 0190/3084] Scaffold implementation of the TargetIsa trait. More to come here. --- cranelift/src/libcretonne/isa/mod.rs | 42 ++++++++++++++++++++++++++-- 1 file changed, 40 insertions(+), 2 deletions(-) diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index e2f48e7bbe..cd9749e6ee 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -1,6 +1,44 @@ //! Instruction Set Architectures. //! -//! The sub-modules of this `isa` module provide definitions for the instruction sets that Cretonne -//! can target. +//! The `isa` module provides a `TargetIsa` trait which provides the behavior specialization needed +//! by the ISA-independent code generator. +//! +//! The sub-modules of this module provide definitions for the instruction sets that Cretonne +//! can target. Each sub-module has it's own implementation of `TargetIsa`. pub mod riscv; + +use ir::dfg::DataFlowGraph; +use ir::entities::Inst; + +pub trait TargetIsa { + /// Encode an instruction after determining it is legal. + /// + /// If `inst` can legally be encoded in this ISA, produce the corresponding `Encoding` object. + /// Otherwise, return `None`. + /// + /// This is also the main entry point for determining if an instruction is legal. + fn encode(&self, dfg: &DataFlowGraph, inst: &Inst) -> Option; +} + +/// Bits needed to encode an instruction as binary machine code. +/// +/// The encoding consists of two parts, both specific to the target ISA: An encoding *recipe*, and +/// encoding *bits*. The recipe determines the native instruction format and the mapping of +/// operands to encoded bits. The encoding bits provide additional information to the recipe, +/// typically parts of the opcode. +pub struct Encoding(u32); + +impl Encoding { + /// Create a new `Encoding` containing `(recipe, bits)`. The `num_bits` parameter is the + /// ISA-dependent size of `bits`. + pub fn new(recipe: u32, bits: u32, num_bits: u8) -> Encoding { + Encoding((recipe << num_bits) | bits) + } + + /// Split the encoding into two parts: `(recipe, bits)`. Only the target ISA knows how many + /// bits are in each part. + pub fn split(&self, num_bits: u8) -> (u32, u32) { + (self.0 >> num_bits, self.0 & ((1 << num_bits) - 1)) + } +} From 1925c1b2c2ec883b0d61578895587c167eab2499 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 5 Aug 2016 13:38:43 -0700 Subject: [PATCH 0191/3084] Scaffolding for defining settings. Settings can be defined globally or per-ISA. They are available to code through a generated Settings struct with accessor methods per setting. --- cranelift/docs/metaref.rst | 39 +++++++ cranelift/src/libcretonne/lib.rs | 1 + cranelift/src/libcretonne/settings.rs | 7 ++ meta/build.py | 2 + meta/cretonne/__init__.py | 146 +++++++++++++++++++++++++- meta/cretonne/settings.py | 13 +++ meta/gen_settings.py | 104 ++++++++++++++++++ 7 files changed, 310 insertions(+), 2 deletions(-) create mode 100644 cranelift/src/libcretonne/settings.rs create mode 100644 meta/cretonne/settings.py create mode 100644 meta/gen_settings.py diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 4ac81096cc..8df75d80a2 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -25,6 +25,45 @@ The main driver for this source code generation process is the :file:`meta/build.py` script which is invoked as part of the build process if anything in the :file:`meta` directory has changed since the last build. + +Settings +======== + +Settings are used by the environment embedding Cretonne to control the details +of code generation. Each setting is defined in the meta language so a compact +and consistent Rust representation can be generated. Shared settings are defined +in the :mod:`cretonne.settings` module. Some settings are specific to a target +ISA, and defined in a `settings` module under the appropriate :file:`meta/isa/*` +directory. + +Settings can take boolean on/off values, small numbers, or explicitly enumerated +symbolic values. Each type is represented by a sub-class of :class:`Setting`: + +.. inheritance-diagram:: Setting BoolSetting NumSetting EnumSetting + :parts: 1 + +.. autoclass:: Setting +.. autoclass:: BoolSetting +.. autoclass:: NumSetting +.. autoclass:: EnumSetting + +All settings must belong to a *group*, represented by a :class:`SettingGroup` +object. + +.. autoclass:: SettingGroup + +Normally, a setting group corresponds to all settings defined in a module. Such +a module looks like this:: + + group = SettingGroup('example') + + foo = BoolSetting('use the foo') + bar = BoolSetting('enable bars', True) + opt = EnumSetting('optimization level', 'Debug', 'Release') + + group.close(globals()) + + Instruction descriptions ======================== diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 9d696fab53..766c33abcf 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -13,5 +13,6 @@ pub mod write; pub mod cfg; pub mod dominator_tree; pub mod entity_map; +pub mod settings; #[cfg(test)]pub mod test_utils; diff --git a/cranelift/src/libcretonne/settings.rs b/cranelift/src/libcretonne/settings.rs new file mode 100644 index 0000000000..0b21bc2eb0 --- /dev/null +++ b/cranelift/src/libcretonne/settings.rs @@ -0,0 +1,7 @@ +//! Shared settings module. +//! +//! This module defines data structures to access the settings defined in the meta language. + +// Include code generated by `meta/gen_settings.py`. This file contains a public `Settings` struct +// with an impl for all of the settings defined in `meta/cretonne/settings.py`. +include!(concat!(env!("OUT_DIR"), "/settings.rs")); diff --git a/meta/build.py b/meta/build.py index 5dadc112e2..6ca4425823 100644 --- a/meta/build.py +++ b/meta/build.py @@ -5,6 +5,7 @@ import argparse import isa import gen_instr +import gen_settings import gen_build_deps parser = argparse.ArgumentParser(description='Generate sources for Cretonne.') @@ -16,4 +17,5 @@ out_dir = args.out_dir isas = isa.all_isas() gen_instr.generate(isas, out_dir) +gen_settings.generate(isas, out_dir) gen_build_deps.generate() diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index ee523cfec5..54f20cc678 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -18,6 +18,146 @@ def camel_case(s): return camel_re.sub(lambda m: m.group(2).upper(), s) +class Setting(object): + """ + A named setting variable that can be configured externally to Cretonne. + + Settings are normally not named when they are created. They get their name + from the `extract_names` method. + """ + + def __init__(self, doc): + self.name = None # Assigned later by `extract_names()`. + self.__doc__ = doc + # Offset of byte in settings vector containing this setting. + self.byte_offset = None + SettingGroup.append(self) + + @staticmethod + def extract_names(globs): + """ + Given a dict mapping name -> object as returned by `globals()`, find + all the Setting objects and set their name from the dict key. This is + used to name a bunch of global variables in a module. + """ + for name, obj in globs.iteritems(): + if isinstance(obj, Setting): + assert obj.name is None + obj.name = name + + +class BoolSetting(Setting): + """ + A named setting with a boolean on/off value. + + :param doc: Documentation string. + :param default: The default value of this setting. + """ + + def __init__(self, doc, default=False): + super(BoolSetting, self).__init__(doc) + self.default = default + + def default_byte(self): + """ + Get the default value of this setting, as a byte that can be bitwise + or'ed with the other booleans sharing the same byte. + """ + if self.default: + return 1 << self.bit_offset + else: + return 0 + + +class NumSetting(Setting): + """ + A named setting with an integral value in the range 0--255. + + :param doc: Documentation string. + :param default: The default value of this setting. + """ + + def __init__(self, doc, default=0): + super(NumSetting, self).__init__(doc) + assert default == int(default) + assert default >= 0 and default <= 255 + self.default = default + + def default_byte(self): + return self.default + + +class EnumSetting(Setting): + """ + A named setting with an enumerated set of possible values. + + The default value is always the first enumerator. + + :param doc: Documentation string. + :param args: Tuple of unique strings representing the possible values. + """ + + def __init__(self, doc, *args): + super(EnumSetting, self).__init__(doc) + assert len(args) > 0, "EnumSetting must have at least one value" + self.values = tuple(str(x) for x in args) + self.default = self.values[0] + + def default_byte(self): + return 0 + + +class SettingGroup(object): + """ + A group of settings. + + Whenever a :class:`Setting` object is created, it is added to the currently + open group. A setting group must be closed explicitly before another can be + opened. + + :param name: Short mnemonic name for setting group. + """ + + # The currently open setting group. + _current = None + + def __init__(self, name): + self.name = name + self.settings = [] + self.open() + + def open(self): + """ + Open this setting group such that future new settings are added to this + group. + """ + assert SettingGroup._current is None, ( + "Can't open {} since {} is already open" + .format(self, SettingGroup._current)) + SettingGroup._current = self + + def close(self, globs=None): + """ + Close this setting group. This function must be called before opening + another setting group. + + :param globs: Pass in `globals()` to run `extract_names` on all + settings defined in the module. + """ + assert SettingGroup._current is self, ( + "Can't close {}, the open setting group is {}" + .format(self, SettingGroup._current)) + SettingGroup._current = None + if globs: + Setting.extract_names(globs) + + @staticmethod + def append(setting): + assert SettingGroup._current, \ + "Open a setting group before defining settings." + SettingGroup._current.settings.append(setting) + + # Kinds of operands. # # Each instruction has an opcode and a number of operands. The opcode @@ -689,7 +829,7 @@ class BoundInstruction(object): assert len(typevars) <= 1 + len(inst.other_typevars) def __str__(self): - return '.'.join([self.inst.name,] + map(str, self.typevars)) + return '.'.join([self.inst.name, ] + map(str, self.typevars)) def bind(self, *args): """ @@ -711,7 +851,9 @@ class BoundInstruction(object): `(inst, typevars)` pair. """ if len(self.typevars) < 1 + len(self.inst.other_typevars): - unb = ', '.join(str(tv) for tv in self.inst.other_typevars[len(self.typevars) - 1:]) + unb = ', '.join( + str(tv) for tv in + self.inst.other_typevars[len(self.typevars) - 1:]) raise AssertionError("Unbound typevar {} in {}".format(unb, self)) assert len(self.typevars) == 1 + len(self.inst.other_typevars) return (self.inst, self.typevars) diff --git a/meta/cretonne/settings.py b/meta/cretonne/settings.py new file mode 100644 index 0000000000..f06c69bfef --- /dev/null +++ b/meta/cretonne/settings.py @@ -0,0 +1,13 @@ +""" +Cretonne shared settings. + +This module defines settings are are relevant for all code generators. +""" + +from . import SettingGroup, BoolSetting + +group = SettingGroup('shared') + +enable_simd = BoolSetting("Enable the use of SIMD instructions", default=True) + +group.close(globals()) diff --git a/meta/gen_settings.py b/meta/gen_settings.py new file mode 100644 index 0000000000..accd739cf0 --- /dev/null +++ b/meta/gen_settings.py @@ -0,0 +1,104 @@ +""" +Generate sources with settings. +""" + +import srcgen +from cretonne import BoolSetting, NumSetting, settings + + +def layout_group(sgrp): + """ + Layout the settings in sgrp, assigning byte and bit offsets. + + Return the next unused byte offset. + """ + # Byte offset where booleans are allocated. + bool_byte = -1 + # Next available bit number in bool_byte. + bool_bit = 10 + # Next available whole byte. + next_byte = 0 + + for setting in sgrp.settings: + if isinstance(setting, BoolSetting): + # Allocate a bit from bool_byte. + if bool_bit > 7: + bool_byte = next_byte + next_byte += 1 + bool_bit = 0 + setting.byte_offset = bool_byte + setting.bit_offset = bool_bit + bool_bit += 1 + else: + # This is a numerical or enumerated setting. Allocate a single + # byte. + setting.byte_offset = next_byte + next_byte += 1 + + return next_byte + + +def gen_getter(setting, fmt): + """ + Emit a getter function for `setting`. + """ + fmt.doc_comment(setting.__doc__ + '.') + + if isinstance(setting, BoolSetting): + proto = 'pub fn {}(&self) -> bool'.format(setting.name) + with fmt.indented(proto + ' {', '}'): + fmt.line('(self.bytes[{}] & (1 << {})) != 0'.format( + setting.byte_offset, + setting.bit_offset)) + elif isinstance(setting, NumSetting): + proto = 'pub fn {}(&self) -> u8'.format(setting.name) + with fmt.indented(proto + ' {', '}'): + fmt.line('self.bytes[{}]'.format(setting.byte_offset)) + else: + raise AssertionError("Unknown setting kind") + + +def gen_getters(sgrp, fmt): + """ + Emit getter functions for all the settings in fmt. + """ + fmt.doc_comment("User-defined settings.") + with fmt.indented('impl Settings {', '}'): + for setting in sgrp.settings: + gen_getter(setting, fmt) + + +def gen_default(sgrp, byte_size, fmt): + """ + Emit a Default impl for Settings. + """ + v = [0] * byte_size + for setting in sgrp.settings: + v[setting.byte_offset] |= setting.default_byte() + + with fmt.indented('impl Default for Settings {', '}'): + fmt.doc_comment('Return a `Settings` object with default values.') + with fmt.indented('fn default() -> Settings {', '}'): + with fmt.indented('Settings {', '}'): + vs = ', '.join('{:#04x}'.format(x) for x in v) + fmt.line('bytes: [ {} ],'.format(vs)) + + +def gen_group(sgrp, fmt): + """ + Generate a Settings struct representing `sgrp`. + """ + byte_size = layout_group(sgrp) + + fmt.doc_comment('Settings group `{}`.'.format(sgrp.name)) + with fmt.indented('pub struct Settings {', '}'): + fmt.line('bytes: [u8; {}],'.format(byte_size)) + + gen_getters(sgrp, fmt) + gen_default(sgrp, byte_size, fmt) + + +def generate(isas, out_dir): + fmt = srcgen.Formatter() + gen_group(settings.group, fmt) + fmt.update_file('settings.rs', out_dir) From 4efb0efb44aca4953a6bb1d6146959dd980b8d7b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 5 Aug 2016 16:19:46 -0700 Subject: [PATCH 0192/3084] Add ISA-dependent settings for RISC-V. --- cranelift/src/libcretonne/isa/riscv/mod.rs | 2 ++ cranelift/src/libcretonne/isa/riscv/settings.rs | 5 +++++ meta/cretonne/__init__.py | 1 + meta/gen_settings.py | 8 ++++++++ meta/isa/riscv/__init__.py | 2 +- meta/isa/riscv/encodings.py | 1 - meta/isa/riscv/recipes.py | 6 ++++++ meta/isa/riscv/settings.py | 15 +++++++++++++++ 8 files changed, 38 insertions(+), 2 deletions(-) create mode 100644 cranelift/src/libcretonne/isa/riscv/settings.rs create mode 100644 meta/isa/riscv/settings.py diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index 4c24cbb8b7..775154a3d6 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -1 +1,3 @@ //! RISC-V Instruction Set Architecture. + +pub mod settings; diff --git a/cranelift/src/libcretonne/isa/riscv/settings.rs b/cranelift/src/libcretonne/isa/riscv/settings.rs new file mode 100644 index 0000000000..74d02d5aac --- /dev/null +++ b/cranelift/src/libcretonne/isa/riscv/settings.rs @@ -0,0 +1,5 @@ +//! RISC-V Settings. + +// Include code generated by `meta/gen_settings.py`. This file contains a public `Settings` struct +// with an impl for all of the settings defined in `meta/cretonne/settings.py`. +include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs")); diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 54f20cc678..801ac5a171 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -875,6 +875,7 @@ class TargetISA(object): def __init__(self, name, instrution_groups): self.name = name + self.settings = None self.instruction_groups = instrution_groups diff --git a/meta/gen_settings.py b/meta/gen_settings.py index accd739cf0..c2545a8c90 100644 --- a/meta/gen_settings.py +++ b/meta/gen_settings.py @@ -99,6 +99,14 @@ def gen_group(sgrp, fmt): def generate(isas, out_dir): + # Generate shared settings. fmt = srcgen.Formatter() gen_group(settings.group, fmt) fmt.update_file('settings.rs', out_dir) + + # Generate ISA-specific settings. + for isa in isas: + if isa.settings: + fmt = srcgen.Formatter() + gen_group(isa.settings, fmt) + fmt.update_file('settings-{}.rs'.format(isa.name), out_dir) diff --git a/meta/isa/riscv/__init__.py b/meta/isa/riscv/__init__.py index 7d51b69b03..ce8c47ea3d 100644 --- a/meta/isa/riscv/__init__.py +++ b/meta/isa/riscv/__init__.py @@ -27,7 +27,7 @@ RV32G / RV64G import defs import encodings +import settings # Re-export the primary target ISA definition. isa = defs.isa - diff --git a/meta/isa/riscv/encodings.py b/meta/isa/riscv/encodings.py index fb6602184c..c148247524 100644 --- a/meta/isa/riscv/encodings.py +++ b/meta/isa/riscv/encodings.py @@ -2,7 +2,6 @@ RISC-V Encodings. """ from cretonne import base -from cretonne.types import i32, i64 from defs import RV32, RV64 from recipes import OP, R diff --git a/meta/isa/riscv/recipes.py b/meta/isa/riscv/recipes.py index ed96836cd4..9719a4b2ec 100644 --- a/meta/isa/riscv/recipes.py +++ b/meta/isa/riscv/recipes.py @@ -18,27 +18,33 @@ from cretonne.formats import Binary # Encbits for the 32-bit recipes are opcode[6:2] | (funct3 << 5) | ... # The functions below encode the encbits. + def LOAD(funct3): assert funct3 <= 0b111 return 0b00000 | (funct3 << 5) + def STORE(funct3): assert funct3 <= 0b111 return 0b01000 | (funct3 << 5) + def BRANCH(funct3): assert funct3 <= 0b111 return 0b11000 | (funct3 << 5) + def OPIMM(funct3): assert funct3 <= 0b111 return 0b00100 | (funct3 << 5) + def OP(funct3, funct7): assert funct3 <= 0b111 assert funct7 <= 0b1111111 return 0b01100 | (funct3 << 5) | (funct7 << 8) + # R-type 32-bit instructions: These are mostly binary arithmetic instructions. # The encbits are `opcode[6:2] | (funct3 << 5) | (funct7 << 8) R = EncRecipe('R', Binary) diff --git a/meta/isa/riscv/settings.py b/meta/isa/riscv/settings.py new file mode 100644 index 0000000000..d2366ad829 --- /dev/null +++ b/meta/isa/riscv/settings.py @@ -0,0 +1,15 @@ +""" +RISC-V settings. +""" + +from cretonne import SettingGroup, BoolSetting +from defs import isa + +isa.settings = SettingGroup('riscv') + +supports_m = BoolSetting("CPU supports the 'M' extension (mul/div)") +supports_a = BoolSetting("CPU supports the 'A' extension (atomics)") +supports_f = BoolSetting("CPU supports the 'F' extension (float)") +supports_d = BoolSetting("CPU supports the 'D' extension (double)") + +isa.settings.close(globals()) From 24ec62d030cd384130411c1e280888e82a7adff7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 9 Aug 2016 11:28:51 -0700 Subject: [PATCH 0193/3084] Move simple_hash into its own module. --- cranelift/src/libcretonne/ir/instructions.rs | 11 +--------- cranelift/src/libcretonne/lib.rs | 2 ++ cranelift/src/libcretonne/simple_hash.rs | 21 ++++++++++++++++++++ 3 files changed, 24 insertions(+), 10 deletions(-) create mode 100644 cranelift/src/libcretonne/simple_hash.rs diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index 4087e246d5..9d5c59eb7d 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -54,16 +54,6 @@ impl Opcode { } } -// A primitive hash function for matching opcodes. -// Must match `meta/constant_hash.py`. -fn simple_hash(s: &str) -> u32 { - let mut h: u32 = 5381; - for c in s.chars() { - h = (h ^ c as u32).wrapping_add(h.rotate_right(6)); - } - h -} - // This trait really belongs in libreader where it is used by the .cton file parser, but since it // critically depends on the `opcode_name()` function which is needed here anyway, it lives in this // module. This also saves us from runing the build script twice to generate code for the two @@ -73,6 +63,7 @@ impl FromStr for Opcode { /// Parse an Opcode name from a string. fn from_str(s: &str) -> Result { + use simple_hash::simple_hash; let tlen = OPCODE_HASH_TABLE.len(); assert!(tlen.is_power_of_two()); let mut idx = simple_hash(s) as usize; diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 766c33abcf..a8cd09ab8d 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -15,4 +15,6 @@ pub mod dominator_tree; pub mod entity_map; pub mod settings; +mod simple_hash; + #[cfg(test)]pub mod test_utils; diff --git a/cranelift/src/libcretonne/simple_hash.rs b/cranelift/src/libcretonne/simple_hash.rs new file mode 100644 index 0000000000..99cd80015c --- /dev/null +++ b/cranelift/src/libcretonne/simple_hash.rs @@ -0,0 +1,21 @@ +/// A primitive hash function for matching opcodes. +/// Must match `meta/constant_hash.py`. +pub fn simple_hash(s: &str) -> u32 { + let mut h: u32 = 5381; + for c in s.chars() { + h = (h ^ c as u32).wrapping_add(h.rotate_right(6)); + } + h +} + +#[cfg(test)] +mod tests { + use super::simple_hash; + + #[test] + fn basic() { + // c.f. meta/constant_hash.py tests. + assert_eq!(simple_hash("Hello"), 0x2fa70c01); + assert_eq!(simple_hash("world"), 0x5b0c31d5); + } +} From 530234ad32e3f60d4584b531454953fa317e6f42 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 9 Aug 2016 11:52:36 -0700 Subject: [PATCH 0194/3084] Add settings::Stringwise. This trait allows settings to be manipulated as strings, using descriptors and constant hash-table lookups. Amend gen_settings.py to generate the necessary constant tables. --- .../src/libcretonne/isa/riscv/settings.rs | 19 +++ cranelift/src/libcretonne/settings.rs | 127 ++++++++++++++++++ meta/gen_settings.py | 113 ++++++++++++++++ 3 files changed, 259 insertions(+) diff --git a/cranelift/src/libcretonne/isa/riscv/settings.rs b/cranelift/src/libcretonne/isa/riscv/settings.rs index 74d02d5aac..0ee83b3015 100644 --- a/cranelift/src/libcretonne/isa/riscv/settings.rs +++ b/cranelift/src/libcretonne/isa/riscv/settings.rs @@ -1,5 +1,24 @@ //! RISC-V Settings. +use settings::{Descriptor, Detail, Stringwise, Result, Error}; +use std::fmt; + // Include code generated by `meta/gen_settings.py`. This file contains a public `Settings` struct // with an impl for all of the settings defined in `meta/cretonne/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs")); + +#[cfg(test)] +mod tests { + use super::Settings; + + #[test] + fn display_default() { + let s = Settings::default(); + assert_eq!(s.to_string(), + "[riscv]\n\ + supports_m = false\n\ + supports_a = false\n\ + supports_f = false\n\ + supports_d = false\n"); + } +} diff --git a/cranelift/src/libcretonne/settings.rs b/cranelift/src/libcretonne/settings.rs index 0b21bc2eb0..80948e3ed3 100644 --- a/cranelift/src/libcretonne/settings.rs +++ b/cranelift/src/libcretonne/settings.rs @@ -1,7 +1,134 @@ //! Shared settings module. //! //! This module defines data structures to access the settings defined in the meta language. +//! +//! Each settings group is translated to a `Settings` struct either in this module or in its +//! ISA-specific `settings` module. The struct provides individual getter methods for all of the +//! settings. It also implements the `Stringwise` trait which allows settings to be manipulated by +//! name. + +use std::fmt; +use std::result; + +/// A setting descriptor holds the information needed to generically set and print a setting. +/// +/// Each settings group will be represented as a constant DESCRIPTORS array. +pub struct Descriptor { + /// Lower snake-case name of setting as defined in meta. + pub name: &'static str, + + /// Offset of byte containing this setting. + pub offset: u32, + + /// Additional details, depending on the kind of setting. + pub detail: Detail, +} + +/// The different kind of settings along with descriptor bits that depend on the kind. +#[derive(Clone, Copy)] +pub enum Detail { + /// A boolean setting only uses one bit, numbered from LSB. + Bool { + bit: u8, + }, + + /// A numerical setting uses the whole byte. + Num, + + /// An Enum setting uses a range of enumerators. + Enum { + /// Numerical value of last enumerator, allowing for 1-256 enumerators. + last: u8, + + /// First enumerator in the ENUMERATORS table. + enumerators: u16, + }, +} + +#[derive(Debug, PartialEq, Eq)] +pub enum Error { + /// No setting by this name exists. + BadName, + + /// Type mismatch for setting (e.g., setting an enum setting as a bool). + BadType, + + /// This is not a valid value for this setting. + BadValue, +} + +pub type Result = result::Result; + +/// Interface for working with a group of settings as strings. +pub trait Stringwise { + /// Look up a setting by name, return the details of the setting along with a reference to the + /// byte holding the value of the setting. + fn lookup_mut(&mut self, name: &str) -> Result<(Detail, &mut u8)>; + + /// Get an enumerator string from the `Detail::enumerators` value and an offset. + fn enumerator(&self, enums: u16, value: u8) -> &'static str; + + /// Format a setting value as a TOML string. This is mostly for use by the generateed `Display` + /// implementation. + fn format_toml_value(&self, detail: Detail, byte: u8, f: &mut fmt::Formatter) -> fmt::Result { + match detail { + Detail::Bool { bit } => write!(f, "{}", (byte & (1 << bit)) != 0), + Detail::Num => write!(f, "{}", byte), + Detail::Enum { last, enumerators } => { + if byte <= last { + write!(f, "\"{}\"", self.enumerator(enumerators, byte)) + } else { + write!(f, "{}", byte) + } + } + } + } + + /// Set a boolean setting by name. + fn set_bool(&mut self, name: &str, value: bool) -> Result<()> { + let (detail, byte) = try!(self.lookup_mut(name)); + if let Detail::Bool { bit } = detail { + let mask = 1 << bit; + if value { + *byte |= mask; + } else { + *byte &= !mask; + } + Ok(()) + } else { + Err(Error::BadType) + } + } +} // Include code generated by `meta/gen_settings.py`. This file contains a public `Settings` struct // with an impl for all of the settings defined in `meta/cretonne/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings.rs")); + +#[cfg(test)] +mod tests { + use super::Settings; + use super::Error::*; + use super::Stringwise; + + #[test] + fn display_default() { + let s = Settings::default(); + assert_eq!(s.to_string(), + "[shared]\n\ + enable_simd = true\n"); + } + + #[test] + fn modify_bool() { + let mut s = Settings::default(); + assert_eq!(s.enable_simd(), true); + assert_eq!(s.set_bool("not_there", true), Err(BadName)); + + assert_eq!(s.set_bool("enable_simd", true), Ok(())); + assert_eq!(s.enable_simd(), true); + + assert_eq!(s.set_bool("enable_simd", false), Ok(())); + assert_eq!(s.enable_simd(), false); + } +} diff --git a/meta/gen_settings.py b/meta/gen_settings.py index c2545a8c90..ce1e031e97 100644 --- a/meta/gen_settings.py +++ b/meta/gen_settings.py @@ -3,6 +3,8 @@ Generate sources with settings. """ import srcgen +from unique_table import UniqueSeqTable +import constant_hash from cretonne import BoolSetting, NumSetting, settings @@ -84,6 +86,114 @@ def gen_default(sgrp, byte_size, fmt): fmt.line('bytes: [ {} ],'.format(vs)) +def gen_descriptors(sgrp, fmt): + """ + Generate the DESCRIPTORS and ENUMERATORS tables. + """ + + enums = UniqueSeqTable() + + with fmt.indented( + 'const DESCRIPTORS: [Descriptor; {}] = [' + .format(len(sgrp.settings)), + '];'): + for idx, setting in enumerate(sgrp.settings): + setting.descriptor_index = idx + with fmt.indented('Descriptor {', '},'): + fmt.line('name: "{}",'.format(setting.name)) + fmt.line('offset: {},'.format(setting.byte_offset)) + if isinstance(setting, BoolSetting): + fmt.line( + 'detail: Detail::Bool {{ bit: {} }},' + .format(setting.bit_offset)) + elif isinstance(setting, NumSetting): + fmt.line('detail: Detail::Num,') + else: + raise AssertionError("Unknown setting kind") + + with fmt.indented( + 'const ENUMERATORS: [&\'static str; {}] = [' + .format(len(enums.table)), + '];'): + for txt in enums.table: + fmt.line('"{}",'.format(txt)) + + def hash_setting(s): + return constant_hash.simple_hash(s.name) + + hash_table = constant_hash.compute_quadratic(sgrp.settings, hash_setting) + if len(sgrp.settings) > 0xffff: + ty = 'u32' + elif len(sgrp.settings) > 0xff: + ty = 'u16' + else: + ty = 'u8' + + with fmt.indented( + 'const HASH_TABLE: [{}; {}] = [' + .format(ty, len(hash_table)), + '];'): + for h in hash_table: + if h is None: + fmt.line('{},'.format(len(sgrp.settings))) + else: + fmt.line('{},'.format(h.descriptor_index)) + + +def gen_stringwise(sgrp, fmt): + """ + Generate the Stringwise implementation and supporting tables. + """ + + with fmt.indented('impl Stringwise for Settings {', '}'): + with fmt.indented( + 'fn lookup_mut(&mut self, name: &str)' + + '-> Result<(Detail, &mut u8)> {', + '}'): + fmt.line('use simple_hash::simple_hash;') + fmt.line('let tlen = HASH_TABLE.len();') + fmt.line('assert!(tlen.is_power_of_two());') + fmt.line('let mut idx = simple_hash(name) as usize;') + fmt.line('let mut step: usize = 0;') + with fmt.indented('loop {', '}'): + fmt.line('idx = idx % tlen;') + fmt.line('let entry = HASH_TABLE[idx] as usize;') + with fmt.indented('if entry >= DESCRIPTORS.len() {', '}'): + fmt.line('return Err(Error::BadName)') + with fmt.indented('if DESCRIPTORS[entry].name == name {', '}'): + fmt.line( + 'return Ok((DESCRIPTORS[entry].detail, ' + + '&mut self.bytes[DESCRIPTORS[entry].offset ' + + 'as usize]))') + fmt.line('step += 1;') + fmt.line('assert!(step < tlen);') + fmt.line('idx += step;') + + with fmt.indented( + 'fn enumerator(&self, enums: u16, value: u8)' + + '-> &\'static str {', + '}'): + fmt.line('ENUMERATORS[enums as usize + value as usize]') + + +def gen_display(sgrp, fmt): + """ + Generate the Display impl for Settings. + """ + with fmt.indented('impl fmt::Display for Settings {', '}'): + with fmt.indented( + 'fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {', + '}'): + fmt.line('try!(writeln!(f, "[{}]"));'.format(sgrp.name)) + with fmt.indented('for d in &DESCRIPTORS {', '}'): + fmt.line('try!(write!(f, "{} = ", d.name));') + fmt.line( + 'try!(self.format_toml_value(d.detail,' + + 'self.bytes[d.offset as usize], f));') + fmt.line('try!(writeln!(f, ""));') + fmt.line('Ok(())') + + def gen_group(sgrp, fmt): """ Generate a Settings struct representing `sgrp`. @@ -96,6 +206,9 @@ def gen_group(sgrp, fmt): gen_getters(sgrp, fmt) gen_default(sgrp, byte_size, fmt) + gen_descriptors(sgrp, fmt) + gen_stringwise(sgrp, fmt) + gen_display(sgrp, fmt) def generate(isas, out_dir): From 56cb249e13b4908dd1b1dd0bc22bbb4426db9509 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 9 Aug 2016 14:12:36 -0700 Subject: [PATCH 0195/3084] Add support for enumerated settings. The EnumSetting objects can take one of 256 named values. --- cranelift/src/libcretonne/settings.rs | 77 +++++++++++++++++++++++++-- meta/cretonne/settings.py | 16 +++++- meta/gen_settings.py | 44 ++++++++++++--- meta/srcgen.py | 6 ++- 4 files changed, 127 insertions(+), 16 deletions(-) diff --git a/cranelift/src/libcretonne/settings.rs b/cranelift/src/libcretonne/settings.rs index 80948e3ed3..af25dd458a 100644 --- a/cranelift/src/libcretonne/settings.rs +++ b/cranelift/src/libcretonne/settings.rs @@ -59,16 +59,26 @@ pub enum Error { pub type Result = result::Result; +fn parse_bool_value(value: &str) -> Result { + match value { + "true" | "on" | "yes" | "1" => Ok(true), + "false" | "off" | "no" | "0" => Ok(false), + _ => Err(Error::BadValue), + } +} + /// Interface for working with a group of settings as strings. pub trait Stringwise { - /// Look up a setting by name, return the details of the setting along with a reference to the - /// byte holding the value of the setting. - fn lookup_mut(&mut self, name: &str) -> Result<(Detail, &mut u8)>; + /// Look up a setting by name, return the byte offset and details of the setting. + fn lookup(&self, name: &str) -> Result<(usize, Detail)>; /// Get an enumerator string from the `Detail::enumerators` value and an offset. fn enumerator(&self, enums: u16, value: u8) -> &'static str; - /// Format a setting value as a TOML string. This is mostly for use by the generateed `Display` + /// Get the underlying byte array used to store settings. + fn raw_bytes_mut(&mut self) -> &mut [u8]; + + /// Format a setting value as a TOML string. This is mostly for use by the generated `Display` /// implementation. fn format_toml_value(&self, detail: Detail, byte: u8, f: &mut fmt::Formatter) -> fmt::Result { match detail { @@ -86,9 +96,10 @@ pub trait Stringwise { /// Set a boolean setting by name. fn set_bool(&mut self, name: &str, value: bool) -> Result<()> { - let (detail, byte) = try!(self.lookup_mut(name)); + let (offset, detail) = try!(self.lookup(name)); if let Detail::Bool { bit } = detail { let mask = 1 << bit; + let byte = &mut self.raw_bytes_mut()[offset]; if value { *byte |= mask; } else { @@ -99,6 +110,43 @@ pub trait Stringwise { Err(Error::BadType) } } + + /// Set the string value of a named setting. + /// + /// For boolean settings, any of the values accepted by `parse_bool_value` above are accepted + /// (true/false, on/off, yes/no, 1/0). + /// + /// For enumerated settings, the value must match one of the allowed values exactly. + fn set(&mut self, name: &str, value: &str) -> Result<()> { + let (offset, detail) = try!(self.lookup(name)); + match detail { + Detail::Bool { bit } => { + let mask = 1 << bit; + let byte = &mut self.raw_bytes_mut()[offset]; + if try!(parse_bool_value(value)) { + *byte |= mask; + } else { + *byte &= !mask; + } + } + Detail::Num => { + self.raw_bytes_mut()[offset] = try!(value.parse().map_err(|_| Error::BadValue)); + } + Detail::Enum { last, enumerators } => { + // Linear search.. + for i in 0.. { + if value == self.enumerator(enumerators, i) { + self.raw_bytes_mut()[offset] = i; + break; + } + if i == last { + return Err(Error::BadValue); + } + } + } + } + Ok(()) + } } // Include code generated by `meta/gen_settings.py`. This file contains a public `Settings` struct @@ -116,6 +164,7 @@ mod tests { let s = Settings::default(); assert_eq!(s.to_string(), "[shared]\n\ + opt_level = \"default\"\n\ enable_simd = true\n"); } @@ -131,4 +180,22 @@ mod tests { assert_eq!(s.set_bool("enable_simd", false), Ok(())); assert_eq!(s.enable_simd(), false); } + + #[test] + fn modify_string() { + let mut s = Settings::default(); + assert_eq!(s.enable_simd(), true); + assert_eq!(s.opt_level(), super::OptLevel::Default); + + assert_eq!(s.set("not_there", "true"), Err(BadName)); + assert_eq!(s.set("enable_simd", ""), Err(BadValue)); + assert_eq!(s.set("enable_simd", "best"), Err(BadValue)); + assert_eq!(s.set("opt_level", "true"), Err(BadValue)); + + assert_eq!(s.set("enable_simd", "no"), Ok(())); + assert_eq!(s.enable_simd(), false); + + assert_eq!(s.set("opt_level", "best"), Ok(())); + assert_eq!(s.opt_level(), super::OptLevel::Best); + } } diff --git a/meta/cretonne/settings.py b/meta/cretonne/settings.py index f06c69bfef..2876dfd84f 100644 --- a/meta/cretonne/settings.py +++ b/meta/cretonne/settings.py @@ -4,10 +4,22 @@ Cretonne shared settings. This module defines settings are are relevant for all code generators. """ -from . import SettingGroup, BoolSetting +from . import SettingGroup, BoolSetting, EnumSetting group = SettingGroup('shared') -enable_simd = BoolSetting("Enable the use of SIMD instructions", default=True) +opt_level = EnumSetting( + """ + Optimization level: + + - default: Very profitable optimizations enabled, none slow. + - best: Enable all optimizations + - fastest: Optimize for compile time by disabling most optimizations. + """, + 'default', 'best', 'fastest') + +enable_simd = BoolSetting( + """Enable the use of SIMD instructions.""", + default=True) group.close(globals()) diff --git a/meta/gen_settings.py b/meta/gen_settings.py index ce1e031e97..d8b0e4c919 100644 --- a/meta/gen_settings.py +++ b/meta/gen_settings.py @@ -5,7 +5,7 @@ Generate sources with settings. import srcgen from unique_table import UniqueSeqTable import constant_hash -from cretonne import BoolSetting, NumSetting, settings +from cretonne import camel_case, BoolSetting, NumSetting, EnumSetting, settings def layout_group(sgrp): @@ -40,11 +40,25 @@ def layout_group(sgrp): return next_byte +def gen_enum_types(sgrp, fmt): + """ + Emit enum types for any enum settings. + """ + for setting in sgrp.settings: + if not isinstance(setting, EnumSetting): + continue + ty = camel_case(setting.name) + fmt.line('#[derive(Debug, PartialEq, Eq)]') + fmt.line( + 'pub enum {} {{ {} }}' + .format(ty, ", ".join(camel_case(v) for v in setting.values))) + + def gen_getter(setting, fmt): """ Emit a getter function for `setting`. """ - fmt.doc_comment(setting.__doc__ + '.') + fmt.doc_comment(setting.__doc__) if isinstance(setting, BoolSetting): proto = 'pub fn {}(&self) -> bool'.format(setting.name) @@ -56,6 +70,15 @@ def gen_getter(setting, fmt): proto = 'pub fn {}(&self) -> u8'.format(setting.name) with fmt.indented(proto + ' {', '}'): fmt.line('self.bytes[{}]'.format(setting.byte_offset)) + elif isinstance(setting, EnumSetting): + ty = camel_case(setting.name) + proto = 'pub fn {}(&self) -> {}'.format(setting.name, ty) + with fmt.indented(proto + ' {', '}'): + with fmt.indented( + 'match self.bytes[{}] {{'.format(setting.byte_offset), '}'): + for i, v in enumerate(setting.values): + fmt.line( '{} => {}::{},'.format(i, ty, camel_case(v))) + fmt.line( '_ => panic!("Invalid enum value")') else: raise AssertionError("Unknown setting kind") @@ -108,6 +131,11 @@ def gen_descriptors(sgrp, fmt): .format(setting.bit_offset)) elif isinstance(setting, NumSetting): fmt.line('detail: Detail::Num,') + elif isinstance(setting, EnumSetting): + offs = enums.add(setting.values) + fmt.line( + 'detail: Detail::Enum {{ last: {}, enumerators: {} }},' + .format(len(setting.values)-1, offs)) else: raise AssertionError("Unknown setting kind") @@ -147,8 +175,8 @@ def gen_stringwise(sgrp, fmt): with fmt.indented('impl Stringwise for Settings {', '}'): with fmt.indented( - 'fn lookup_mut(&mut self, name: &str)' + - '-> Result<(Detail, &mut u8)> {', + 'fn lookup(&self, name: &str)' + + '-> Result<(usize, Detail)> {', '}'): fmt.line('use simple_hash::simple_hash;') fmt.line('let tlen = HASH_TABLE.len();') @@ -162,9 +190,8 @@ def gen_stringwise(sgrp, fmt): fmt.line('return Err(Error::BadName)') with fmt.indented('if DESCRIPTORS[entry].name == name {', '}'): fmt.line( - 'return Ok((DESCRIPTORS[entry].detail, ' + - '&mut self.bytes[DESCRIPTORS[entry].offset ' + - 'as usize]))') + 'return Ok((DESCRIPTORS[entry].offset as usize, ' + + 'DESCRIPTORS[entry].detail))') fmt.line('step += 1;') fmt.line('assert!(step < tlen);') fmt.line('idx += step;') @@ -175,6 +202,8 @@ def gen_stringwise(sgrp, fmt): '}'): fmt.line('ENUMERATORS[enums as usize + value as usize]') + with fmt.indented('fn raw_bytes_mut(&mut self) -> &mut [u8] {', '}'): + fmt.line('&mut self.bytes') def gen_display(sgrp, fmt): """ @@ -204,6 +233,7 @@ def gen_group(sgrp, fmt): with fmt.indented('pub struct Settings {', '}'): fmt.line('bytes: [u8; {}],'.format(byte_size)) + gen_enum_types(sgrp, fmt) gen_getters(sgrp, fmt) gen_default(sgrp, byte_size, fmt) gen_descriptors(sgrp, fmt) diff --git a/meta/srcgen.py b/meta/srcgen.py index 1ebd745286..42f39c96de 100644 --- a/meta/srcgen.py +++ b/meta/srcgen.py @@ -8,6 +8,7 @@ source code. import sys import os +import re class Formatter(object): @@ -108,8 +109,9 @@ class Formatter(object): self.line('// ' + s) def doc_comment(self, s): - """Add a documentation comment line.""" - self.line('/// ' + s) + """Add a (multi-line) documentation comment.""" + s = re.sub('^', self.indent + '/// ', s, flags=re.M) + '\n' + self.lines.append(s) if __name__ == "__main__": import doctest From 6e6ad1ef52a23a522c7bf02f318d60ae3ea2fe07 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 10 Aug 2016 15:28:17 -0700 Subject: [PATCH 0196/3084] Add a settings::Builder data type. - Move detail data structures into a settings::detail module to avoid polluting the settings namespace. - Rename generated data types to 'Flags' in anticipation of computed predicate flags that can't be set. The Flags struct is immutable. - Use a settings::Builder struct to manipulate settings, then pass it to Flags::new(). --- .../src/libcretonne/isa/riscv/settings.rs | 11 +- cranelift/src/libcretonne/settings.rs | 366 +++++++++++------- meta/gen_settings.py | 132 +++---- 3 files changed, 292 insertions(+), 217 deletions(-) diff --git a/cranelift/src/libcretonne/isa/riscv/settings.rs b/cranelift/src/libcretonne/isa/riscv/settings.rs index 0ee83b3015..335314a3c9 100644 --- a/cranelift/src/libcretonne/isa/riscv/settings.rs +++ b/cranelift/src/libcretonne/isa/riscv/settings.rs @@ -1,20 +1,21 @@ //! RISC-V Settings. -use settings::{Descriptor, Detail, Stringwise, Result, Error}; +use settings::{detail, Builder}; use std::fmt; -// Include code generated by `meta/gen_settings.py`. This file contains a public `Settings` struct +// Include code generated by `meta/gen_settings.py`. This file contains a public `Flags` struct // with an impl for all of the settings defined in `meta/cretonne/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs")); #[cfg(test)] mod tests { - use super::Settings; + use super::{builder, Flags}; #[test] fn display_default() { - let s = Settings::default(); - assert_eq!(s.to_string(), + let b = builder(); + let f = Flags::new(b); + assert_eq!(f.to_string(), "[riscv]\n\ supports_m = false\n\ supports_a = false\n\ diff --git a/cranelift/src/libcretonne/settings.rs b/cranelift/src/libcretonne/settings.rs index af25dd458a..c65c5eeb68 100644 --- a/cranelift/src/libcretonne/settings.rs +++ b/cranelift/src/libcretonne/settings.rs @@ -2,49 +2,151 @@ //! //! This module defines data structures to access the settings defined in the meta language. //! -//! Each settings group is translated to a `Settings` struct either in this module or in its +//! Each settings group is translated to a `Flags` struct either in this module or in its //! ISA-specific `settings` module. The struct provides individual getter methods for all of the -//! settings. It also implements the `Stringwise` trait which allows settings to be manipulated by -//! name. +//! settings as well as computed predicate flags. +//! +//! The `Flags` struct is immutable once it has been created. A `Builder` instance is used to +//! create it. +//! +//! # Example +//! ``` +//! use cretonne::settings::{self, Configurable}; +//! +//! let mut b = settings::builder(); +//! b.set("opt_level", "fastest"); +//! +//! let f = settings::Flags::new(b); +//! assert_eq!(f.opt_level(), settings::OptLevel::Fastest); +//! ``` use std::fmt; use std::result; -/// A setting descriptor holds the information needed to generically set and print a setting. +use simple_hash::simple_hash; + +/// A string-based configurator for settings groups. /// -/// Each settings group will be represented as a constant DESCRIPTORS array. -pub struct Descriptor { - /// Lower snake-case name of setting as defined in meta. - pub name: &'static str, +/// The `Configurable` protocol allows settings to be modified by name before a finished `Flags` +/// struct is created. +pub trait Configurable { + /// Set the string value of any setting by name. + /// + /// This can set any type of setting whether it is numeric, boolean, or enumerated. + fn set(&mut self, name: &str, value: &str) -> Result<()>; - /// Offset of byte containing this setting. - pub offset: u32, - - /// Additional details, depending on the kind of setting. - pub detail: Detail, + /// Set the value of a boolean setting by name. + /// + /// If the identified setting isn't a boolean, a `BadType` error is returned. + fn set_bool(&mut self, name: &str, value: bool) -> Result<()>; } -/// The different kind of settings along with descriptor bits that depend on the kind. -#[derive(Clone, Copy)] -pub enum Detail { - /// A boolean setting only uses one bit, numbered from LSB. - Bool { - bit: u8, - }, - - /// A numerical setting uses the whole byte. - Num, - - /// An Enum setting uses a range of enumerators. - Enum { - /// Numerical value of last enumerator, allowing for 1-256 enumerators. - last: u8, - - /// First enumerator in the ENUMERATORS table. - enumerators: u16, - }, +/// Collect settings values based on a template. +pub struct Builder { + template: &'static detail::Template, + bytes: Vec, } +impl Builder { + /// Create a new builder with defaults and names from the given template. + pub fn new(tmpl: &'static detail::Template) -> Builder { + Builder { + template: tmpl, + bytes: tmpl.defaults.into(), + } + } + + /// Extract contents of builder once everything is configured. + pub fn finish(self, name: &str) -> Vec { + assert_eq!(name, self.template.name); + self.bytes + } + + /// Set the value of a single bit. + fn set_bit(&mut self, offset: usize, bit: u8, value: bool) { + let byte = &mut self.bytes[offset]; + let mask = 1 << bit; + if value { + *byte |= mask; + } else { + *byte &= !mask; + } + } + + /// Look up a descriptor by name. + fn lookup(&self, name: &str) -> Result<(usize, detail::Detail)> { + let table = self.template.hash_table; + let descs = self.template.descriptors; + let mask = table.len() - 1; + assert!((mask + 1).is_power_of_two()); + + let mut idx = simple_hash(name) as usize; + let mut step: usize = 0; + + loop { + idx = idx & mask; + let entry = table[idx] as usize; + if entry >= descs.len() { + return Err(Error::BadName); + } + let desc = &descs[entry]; + if desc.name == name { + return Ok((desc.offset as usize, desc.detail)); + } + step += 1; + assert!(step <= mask); + idx += step; + } + } +} + +fn parse_bool_value(value: &str) -> Result { + match value { + "true" | "on" | "yes" | "1" => Ok(true), + "false" | "off" | "no" | "0" => Ok(false), + _ => Err(Error::BadValue), + } +} + +fn parse_enum_value(value: &str, choices: &[&str]) -> Result { + match choices.iter().position(|&tag| tag == value) { + Some(idx) => Ok(idx as u8), + None => Err(Error::BadValue), + } +} + +impl Configurable for Builder { + fn set_bool(&mut self, name: &str, value: bool) -> Result<()> { + use self::detail::Detail; + let (offset, detail) = try!(self.lookup(name)); + if let Detail::Bool { bit } = detail { + self.set_bit(offset, bit, value); + Ok(()) + } else { + Err(Error::BadType) + } + } + + fn set(&mut self, name: &str, value: &str) -> Result<()> { + use self::detail::Detail; + let (offset, detail) = try!(self.lookup(name)); + match detail { + Detail::Bool { bit } => { + self.set_bit(offset, bit, try!(parse_bool_value(value))); + } + Detail::Num => { + self.bytes[offset] = try!(value.parse().map_err(|_| Error::BadValue)); + } + Detail::Enum { last, enumerators } => { + self.bytes[offset] = try!(parse_enum_value(value, + self.template.enums(last, enumerators))); + } + } + Ok(()) + } +} + +/// An error produced when changing a setting. #[derive(Debug, PartialEq, Eq)] pub enum Error { /// No setting by this name exists. @@ -59,143 +161,133 @@ pub enum Error { pub type Result = result::Result; -fn parse_bool_value(value: &str) -> Result { - match value { - "true" | "on" | "yes" | "1" => Ok(true), - "false" | "off" | "no" | "0" => Ok(false), - _ => Err(Error::BadValue), +/// Implementation details for generated code. +/// +/// This module holds definitions that need to be public so the can be instantiated by generated +/// code in other modules. +pub mod detail { + use std::fmt; + + /// An instruction group template. + pub struct Template { + pub name: &'static str, + pub descriptors: &'static [Descriptor], + pub enumerators: &'static [&'static str], + pub hash_table: &'static [u16], + pub defaults: &'static [u8], } -} -/// Interface for working with a group of settings as strings. -pub trait Stringwise { - /// Look up a setting by name, return the byte offset and details of the setting. - fn lookup(&self, name: &str) -> Result<(usize, Detail)>; - - /// Get an enumerator string from the `Detail::enumerators` value and an offset. - fn enumerator(&self, enums: u16, value: u8) -> &'static str; - - /// Get the underlying byte array used to store settings. - fn raw_bytes_mut(&mut self) -> &mut [u8]; - - /// Format a setting value as a TOML string. This is mostly for use by the generated `Display` - /// implementation. - fn format_toml_value(&self, detail: Detail, byte: u8, f: &mut fmt::Formatter) -> fmt::Result { - match detail { - Detail::Bool { bit } => write!(f, "{}", (byte & (1 << bit)) != 0), - Detail::Num => write!(f, "{}", byte), - Detail::Enum { last, enumerators } => { - if byte <= last { - write!(f, "\"{}\"", self.enumerator(enumerators, byte)) - } else { - write!(f, "{}", byte) - } - } + impl Template { + /// Get enumerators corresponding to a `Details::Enum`. + pub fn enums(&self, last: u8, enumerators: u16) -> &[&'static str] { + let from = enumerators as usize; + let len = last as usize + 1; + &self.enumerators[from..from + len] } - } - /// Set a boolean setting by name. - fn set_bool(&mut self, name: &str, value: bool) -> Result<()> { - let (offset, detail) = try!(self.lookup(name)); - if let Detail::Bool { bit } = detail { - let mask = 1 << bit; - let byte = &mut self.raw_bytes_mut()[offset]; - if value { - *byte |= mask; - } else { - *byte &= !mask; - } - Ok(()) - } else { - Err(Error::BadType) - } - } - - /// Set the string value of a named setting. - /// - /// For boolean settings, any of the values accepted by `parse_bool_value` above are accepted - /// (true/false, on/off, yes/no, 1/0). - /// - /// For enumerated settings, the value must match one of the allowed values exactly. - fn set(&mut self, name: &str, value: &str) -> Result<()> { - let (offset, detail) = try!(self.lookup(name)); - match detail { - Detail::Bool { bit } => { - let mask = 1 << bit; - let byte = &mut self.raw_bytes_mut()[offset]; - if try!(parse_bool_value(value)) { - *byte |= mask; - } else { - *byte &= !mask; - } - } - Detail::Num => { - self.raw_bytes_mut()[offset] = try!(value.parse().map_err(|_| Error::BadValue)); - } - Detail::Enum { last, enumerators } => { - // Linear search.. - for i in 0.. { - if value == self.enumerator(enumerators, i) { - self.raw_bytes_mut()[offset] = i; - break; - } - if i == last { - return Err(Error::BadValue); + /// Format a setting value as a TOML string. This is mostly for use by the generated + /// `Display` implementation. + pub fn format_toml_value(&self, + detail: Detail, + byte: u8, + f: &mut fmt::Formatter) + -> fmt::Result { + match detail { + Detail::Bool { bit } => write!(f, "{}", (byte & (1 << bit)) != 0), + Detail::Num => write!(f, "{}", byte), + Detail::Enum { last, enumerators } => { + if byte <= last { + let tags = self.enums(last, enumerators); + write!(f, "\"{}\"", tags[byte as usize]) + } else { + write!(f, "{}", byte) } } } } - Ok(()) + } + + /// A setting descriptor holds the information needed to generically set and print a setting. + /// + /// Each settings group will be represented as a constant DESCRIPTORS array. + pub struct Descriptor { + /// Lower snake-case name of setting as defined in meta. + pub name: &'static str, + + /// Offset of byte containing this setting. + pub offset: u32, + + /// Additional details, depending on the kind of setting. + pub detail: Detail, + } + + /// The different kind of settings along with descriptor bits that depend on the kind. + #[derive(Clone, Copy)] + pub enum Detail { + /// A boolean setting only uses one bit, numbered from LSB. + Bool { + bit: u8, + }, + + /// A numerical setting uses the whole byte. + Num, + + /// An Enum setting uses a range of enumerators. + Enum { + /// Numerical value of last enumerator, allowing for 1-256 enumerators. + last: u8, + + /// First enumerator in the ENUMERATORS table. + enumerators: u16, + }, } } -// Include code generated by `meta/gen_settings.py`. This file contains a public `Settings` struct +// Include code generated by `meta/gen_settings.py`. This file contains a public `Flags` struct // with an impl for all of the settings defined in `meta/cretonne/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings.rs")); #[cfg(test)] mod tests { - use super::Settings; + use super::{builder, Flags}; use super::Error::*; - use super::Stringwise; + use super::Configurable; #[test] fn display_default() { - let s = Settings::default(); - assert_eq!(s.to_string(), + let b = builder(); + let f = Flags::new(b); + assert_eq!(f.to_string(), "[shared]\n\ opt_level = \"default\"\n\ enable_simd = true\n"); + assert_eq!(f.opt_level(), super::OptLevel::Default); + assert_eq!(f.enable_simd(), true); } #[test] fn modify_bool() { - let mut s = Settings::default(); - assert_eq!(s.enable_simd(), true); - assert_eq!(s.set_bool("not_there", true), Err(BadName)); + let mut b = builder(); + assert_eq!(b.set_bool("not_there", true), Err(BadName)); + assert_eq!(b.set_bool("enable_simd", true), Ok(())); + assert_eq!(b.set_bool("enable_simd", false), Ok(())); - assert_eq!(s.set_bool("enable_simd", true), Ok(())); - assert_eq!(s.enable_simd(), true); - - assert_eq!(s.set_bool("enable_simd", false), Ok(())); - assert_eq!(s.enable_simd(), false); + let f = Flags::new(b); + assert_eq!(f.enable_simd(), false); } #[test] fn modify_string() { - let mut s = Settings::default(); - assert_eq!(s.enable_simd(), true); - assert_eq!(s.opt_level(), super::OptLevel::Default); + let mut b = builder(); + assert_eq!(b.set("not_there", "true"), Err(BadName)); + assert_eq!(b.set("enable_simd", ""), Err(BadValue)); + assert_eq!(b.set("enable_simd", "best"), Err(BadValue)); + assert_eq!(b.set("opt_level", "true"), Err(BadValue)); + assert_eq!(b.set("opt_level", "best"), Ok(())); + assert_eq!(b.set("enable_simd", "0"), Ok(())); - assert_eq!(s.set("not_there", "true"), Err(BadName)); - assert_eq!(s.set("enable_simd", ""), Err(BadValue)); - assert_eq!(s.set("enable_simd", "best"), Err(BadValue)); - assert_eq!(s.set("opt_level", "true"), Err(BadValue)); - - assert_eq!(s.set("enable_simd", "no"), Ok(())); - assert_eq!(s.enable_simd(), false); - - assert_eq!(s.set("opt_level", "best"), Ok(())); - assert_eq!(s.opt_level(), super::OptLevel::Best); + let f = Flags::new(b); + assert_eq!(f.enable_simd(), false); + assert_eq!(f.opt_level(), super::OptLevel::Best); } } diff --git a/meta/gen_settings.py b/meta/gen_settings.py index d8b0e4c919..a10dd5352a 100644 --- a/meta/gen_settings.py +++ b/meta/gen_settings.py @@ -75,10 +75,11 @@ def gen_getter(setting, fmt): proto = 'pub fn {}(&self) -> {}'.format(setting.name, ty) with fmt.indented(proto + ' {', '}'): with fmt.indented( - 'match self.bytes[{}] {{'.format(setting.byte_offset), '}'): + 'match self.bytes[{}] {{' + .format(setting.byte_offset), '}'): for i, v in enumerate(setting.values): - fmt.line( '{} => {}::{},'.format(i, ty, camel_case(v))) - fmt.line( '_ => panic!("Invalid enum value")') + fmt.line('{} => {}::{},'.format(i, ty, camel_case(v))) + fmt.line('_ => panic!("Invalid enum value")') else: raise AssertionError("Unknown setting kind") @@ -88,27 +89,11 @@ def gen_getters(sgrp, fmt): Emit getter functions for all the settings in fmt. """ fmt.doc_comment("User-defined settings.") - with fmt.indented('impl Settings {', '}'): + with fmt.indented('impl Flags {', '}'): for setting in sgrp.settings: gen_getter(setting, fmt) -def gen_default(sgrp, byte_size, fmt): - """ - Emit a Default impl for Settings. - """ - v = [0] * byte_size - for setting in sgrp.settings: - v[setting.byte_offset] |= setting.default_byte() - - with fmt.indented('impl Default for Settings {', '}'): - fmt.doc_comment('Return a `Settings` object with default values.') - with fmt.indented('fn default() -> Settings {', '}'): - with fmt.indented('Settings {', '}'): - vs = ', '.join('{:#04x}'.format(x) for x in v) - fmt.line('bytes: [ {} ],'.format(vs)) - - def gen_descriptors(sgrp, fmt): """ Generate the DESCRIPTORS and ENUMERATORS tables. @@ -117,30 +102,31 @@ def gen_descriptors(sgrp, fmt): enums = UniqueSeqTable() with fmt.indented( - 'const DESCRIPTORS: [Descriptor; {}] = [' + 'static DESCRIPTORS: [detail::Descriptor; {}] = [' .format(len(sgrp.settings)), '];'): for idx, setting in enumerate(sgrp.settings): setting.descriptor_index = idx - with fmt.indented('Descriptor {', '},'): + with fmt.indented('detail::Descriptor {', '},'): fmt.line('name: "{}",'.format(setting.name)) fmt.line('offset: {},'.format(setting.byte_offset)) if isinstance(setting, BoolSetting): fmt.line( - 'detail: Detail::Bool {{ bit: {} }},' + 'detail: detail::Detail::Bool {{ bit: {} }},' .format(setting.bit_offset)) elif isinstance(setting, NumSetting): - fmt.line('detail: Detail::Num,') + fmt.line('detail: detail::Detail::Num,') elif isinstance(setting, EnumSetting): offs = enums.add(setting.values) fmt.line( - 'detail: Detail::Enum {{ last: {}, enumerators: {} }},' + 'detail: detail::Detail::Enum ' + + '{{ last: {}, enumerators: {} }},' .format(len(setting.values)-1, offs)) else: raise AssertionError("Unknown setting kind") with fmt.indented( - 'const ENUMERATORS: [&\'static str; {}] = [' + 'static ENUMERATORS: [&\'static str; {}] = [' .format(len(enums.table)), '];'): for txt in enums.table: @@ -150,66 +136,46 @@ def gen_descriptors(sgrp, fmt): return constant_hash.simple_hash(s.name) hash_table = constant_hash.compute_quadratic(sgrp.settings, hash_setting) - if len(sgrp.settings) > 0xffff: - ty = 'u32' - elif len(sgrp.settings) > 0xff: - ty = 'u16' - else: - ty = 'u8' - with fmt.indented( - 'const HASH_TABLE: [{}; {}] = [' - .format(ty, len(hash_table)), + 'static HASH_TABLE: [u16; {}] = [' + .format(len(hash_table)), '];'): for h in hash_table: if h is None: - fmt.line('{},'.format(len(sgrp.settings))) + fmt.line('0xffff,') else: fmt.line('{},'.format(h.descriptor_index)) -def gen_stringwise(sgrp, fmt): +def gen_template(sgrp, byte_size, fmt): """ - Generate the Stringwise implementation and supporting tables. + Emit a Template constant. """ + v = [0] * byte_size + for setting in sgrp.settings: + v[setting.byte_offset] |= setting.default_byte() - with fmt.indented('impl Stringwise for Settings {', '}'): - with fmt.indented( - 'fn lookup(&self, name: &str)' + - '-> Result<(usize, Detail)> {', - '}'): - fmt.line('use simple_hash::simple_hash;') - fmt.line('let tlen = HASH_TABLE.len();') - fmt.line('assert!(tlen.is_power_of_two());') - fmt.line('let mut idx = simple_hash(name) as usize;') - fmt.line('let mut step: usize = 0;') - with fmt.indented('loop {', '}'): - fmt.line('idx = idx % tlen;') - fmt.line('let entry = HASH_TABLE[idx] as usize;') - with fmt.indented('if entry >= DESCRIPTORS.len() {', '}'): - fmt.line('return Err(Error::BadName)') - with fmt.indented('if DESCRIPTORS[entry].name == name {', '}'): - fmt.line( - 'return Ok((DESCRIPTORS[entry].offset as usize, ' + - 'DESCRIPTORS[entry].detail))') - fmt.line('step += 1;') - fmt.line('assert!(step < tlen);') - fmt.line('idx += step;') + with fmt.indented( + 'static TEMPLATE: detail::Template = detail::Template {', '};'): + fmt.line('name: "{}",'.format(sgrp.name)) + fmt.line('descriptors: &DESCRIPTORS,') + fmt.line('enumerators: &ENUMERATORS,') + fmt.line('hash_table: &HASH_TABLE,') + vs = ', '.join('{:#04x}'.format(x) for x in v) + fmt.line('defaults: &[ {} ],'.format(vs)) - with fmt.indented( - 'fn enumerator(&self, enums: u16, value: u8)' + - '-> &\'static str {', - '}'): - fmt.line('ENUMERATORS[enums as usize + value as usize]') + fmt.doc_comment( + 'Create a `settings::Builder` for the {} settings group.' + .format(sgrp.name)) + with fmt.indented('pub fn builder() -> Builder {', '}'): + fmt.line('Builder::new(&TEMPLATE)') - with fmt.indented('fn raw_bytes_mut(&mut self) -> &mut [u8] {', '}'): - fmt.line('&mut self.bytes') def gen_display(sgrp, fmt): """ - Generate the Display impl for Settings. + Generate the Display impl for Flags. """ - with fmt.indented('impl fmt::Display for Settings {', '}'): + with fmt.indented('impl fmt::Display for Flags {', '}'): with fmt.indented( 'fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {', '}'): @@ -217,27 +183,43 @@ def gen_display(sgrp, fmt): with fmt.indented('for d in &DESCRIPTORS {', '}'): fmt.line('try!(write!(f, "{} = ", d.name));') fmt.line( - 'try!(self.format_toml_value(d.detail,' + + 'try!(TEMPLATE.format_toml_value(d.detail,' + 'self.bytes[d.offset as usize], f));') fmt.line('try!(writeln!(f, ""));') fmt.line('Ok(())') +def gen_constructor(sgrp, byte_size, parent, fmt): + """ + Generate a Flags constructor. + """ + + with fmt.indented('impl Flags {', '}'): + with fmt.indented('pub fn new(builder: Builder) -> Flags {', '}'): + fmt.line('let bvec = builder.finish("{}");'.format(sgrp.name)) + fmt.line('let mut bytes = [0; {}];'.format(byte_size)) + fmt.line('assert_eq!(bytes.len(), bvec.len());') + with fmt.indented( + 'for (i, b) in bvec.into_iter().enumerate() {', '}'): + fmt.line('bytes[i] = b;') + fmt.line('Flags { bytes: bytes }') + + def gen_group(sgrp, fmt): """ - Generate a Settings struct representing `sgrp`. + Generate a Flags struct representing `sgrp`. """ byte_size = layout_group(sgrp) - fmt.doc_comment('Settings group `{}`.'.format(sgrp.name)) - with fmt.indented('pub struct Settings {', '}'): + fmt.doc_comment('Flags group `{}`.'.format(sgrp.name)) + with fmt.indented('pub struct Flags {', '}'): fmt.line('bytes: [u8; {}],'.format(byte_size)) + gen_constructor(sgrp, byte_size, None, fmt) gen_enum_types(sgrp, fmt) gen_getters(sgrp, fmt) - gen_default(sgrp, byte_size, fmt) gen_descriptors(sgrp, fmt) - gen_stringwise(sgrp, fmt) + gen_template(sgrp, byte_size, fmt) gen_display(sgrp, fmt) From 1ae9a37796488b3c1c200e0cf679f81667ead35f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 10 Aug 2016 12:07:32 -0700 Subject: [PATCH 0197/3084] Document ISA builder. --- cranelift/src/libcretonne/isa/mod.rs | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index cd9749e6ee..577f6e400e 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -1,10 +1,30 @@ //! Instruction Set Architectures. //! //! The `isa` module provides a `TargetIsa` trait which provides the behavior specialization needed -//! by the ISA-independent code generator. +//! by the ISA-independent code generator. The sub-modules of this module provide definitions for +//! the instruction sets that Cretonne can target. Each sub-module has it's own implementation of +//! `TargetIsa`. +//! +//! # Constructing a `TargetIsa` instance +//! +//! The target ISA is build from the following information: +//! +//! - The name of the target ISA as a string. Cretonne is a cross-compiler, so the ISA to target +//! can be selected dynamically. Individual ISAs can be left out when Cretonne is compiled, so a +//! string is used to identify the proper sub-module. +//! - Values for settings that apply to all ISAs. This is represented by a `settings::Flags` +//! instance. +//! - Values for ISA-specific settings. +//! +//! The `isa::lookup()` function is the main entry point which returns an `isa::Builder` +//! appropriate for the requested ISA: +//! +//! ```ignore +//! let isa_builder = isa::lookup("riscv").unwrap(); +//! adjust_isa_settings(&mut isa_builder.settings); +//! let isa = builder.finish(shared_settings()); +//! ``` //! -//! The sub-modules of this module provide definitions for the instruction sets that Cretonne -//! can target. Each sub-module has it's own implementation of `TargetIsa`. pub mod riscv; From 1087aa67f0f8401bc6fda23d1f05e0acd02b56f1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 11 Aug 2016 11:39:42 -0700 Subject: [PATCH 0198/3084] Implement the machinery to create a TargetIsa. Add an isa::lookup() function which serves as a target registry for creating Box trait objects. An isa::Builder makes it possible to confugure the trait object before it is created. --- cranelift/src/libcretonne/isa/mod.rs | 64 ++++++++++++++++++++-- cranelift/src/libcretonne/isa/riscv/mod.rs | 34 ++++++++++++ meta/gen_settings.py | 1 + 3 files changed, 94 insertions(+), 5 deletions(-) diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index 577f6e400e..43834be3ef 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -7,7 +7,7 @@ //! //! # Constructing a `TargetIsa` instance //! -//! The target ISA is build from the following information: +//! The target ISA is built from the following information: //! //! - The name of the target ISA as a string. Cretonne is a cross-compiler, so the ISA to target //! can be selected dynamically. Individual ISAs can be left out when Cretonne is compiled, so a @@ -19,18 +19,72 @@ //! The `isa::lookup()` function is the main entry point which returns an `isa::Builder` //! appropriate for the requested ISA: //! -//! ```ignore -//! let isa_builder = isa::lookup("riscv").unwrap(); -//! adjust_isa_settings(&mut isa_builder.settings); -//! let isa = builder.finish(shared_settings()); +//! ``` +//! use cretonne::settings::{self, Configurable}; +//! use cretonne::isa; +//! +//! let shared_builder = settings::builder(); +//! let shared_flags = settings::Flags::new(shared_builder); +//! +//! match isa::lookup("riscv") { +//! None => { +//! // The RISC-V target ISA is not available. +//! } +//! Some(mut isa_builder) => { +//! isa_builder.set("supports_m", "on"); +//! let isa = isa_builder.finish(shared_flags); +//! } +//! } //! ``` //! +//! The configured target ISA trait object is a `Box` which can be used for multiple +//! concurrent function compilations. pub mod riscv; +use settings; use ir::dfg::DataFlowGraph; use ir::entities::Inst; +/// Look for a supported ISA with the given `name`. +/// Return a builder that can create a corresponding `TargetIsa`. +pub fn lookup(name: &str) -> Option { + match name { + "riscv" => riscv_builder(), + _ => None, + } +} + +// Make a builder for RISC-V. +fn riscv_builder() -> Option { + Some(riscv::isa_builder()) +} + +/// Builder for a `TargetIsa`. +/// Modify the ISA-specific settings before creating the `TargetIsa` trait object with `finish`. +pub struct Builder { + setup: settings::Builder, + constructor: fn(settings::Flags, settings::Builder) -> Box, +} + +impl Builder { + /// Combine the ISA-specific settings with the provided ISA-independent settings and allocate a + /// fully configured `TargetIsa` trait object. + pub fn finish(self, shared_flags: settings::Flags) -> Box { + (self.constructor)(shared_flags, self.setup) + } +} + +impl settings::Configurable for Builder { + fn set(&mut self, name: &str, value: &str) -> settings::Result<()> { + self.setup.set(name, value) + } + + fn set_bool(&mut self, name: &str, value: bool) -> settings::Result<()> { + self.setup.set_bool(name, value) + } +} + pub trait TargetIsa { /// Encode an instruction after determining it is legal. /// diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index 775154a3d6..641263b9f6 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -1,3 +1,37 @@ //! RISC-V Instruction Set Architecture. pub mod settings; + +use super::super::settings as shared_settings; +use super::Builder as IsaBuilder; +use super::{TargetIsa, Encoding}; +use ir::dfg::DataFlowGraph; +use ir::entities::Inst; + +#[allow(dead_code)] +struct Isa { + shared_flags: shared_settings::Flags, + isa_flags: settings::Flags, +} + +pub fn isa_builder() -> IsaBuilder { + IsaBuilder { + setup: settings::builder(), + constructor: isa_constructor, + } +} + +fn isa_constructor(shared_flags: shared_settings::Flags, + builder: shared_settings::Builder) + -> Box { + Box::new(Isa { + shared_flags: shared_flags, + isa_flags: settings::Flags::new(builder), + }) +} + +impl TargetIsa for Isa { + fn encode(&self, _: &DataFlowGraph, _: &Inst) -> Option { + unimplemented!() + } +} diff --git a/meta/gen_settings.py b/meta/gen_settings.py index a10dd5352a..486e71371e 100644 --- a/meta/gen_settings.py +++ b/meta/gen_settings.py @@ -211,6 +211,7 @@ def gen_group(sgrp, fmt): """ byte_size = layout_group(sgrp) + fmt.line('#[derive(Clone)]') fmt.doc_comment('Flags group `{}`.'.format(sgrp.name)) with fmt.indented('pub struct Flags {', '}'): fmt.line('bytes: [u8; {}],'.format(byte_size)) From 13d33d5a7a8a8a59786a5b0aaee8a262b623aa4f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 11 Aug 2016 14:22:23 -0700 Subject: [PATCH 0199/3084] Introduce predicates. Predcates are boolean functions. There will be ISA predicates and instruction predicates. The ISA predicates will be turned into member functions on the generated Flags structs. --- meta/cretonne/__init__.py | 37 +++++++++------ meta/cretonne/predicates.py | 94 +++++++++++++++++++++++++++++++++++++ meta/isa/riscv/settings.py | 6 ++- 3 files changed, 121 insertions(+), 16 deletions(-) create mode 100644 meta/cretonne/predicates.py diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 801ac5a171..f576380f65 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -31,19 +31,14 @@ class Setting(object): self.__doc__ = doc # Offset of byte in settings vector containing this setting. self.byte_offset = None - SettingGroup.append(self) + self.group = SettingGroup.append(self) - @staticmethod - def extract_names(globs): + def predicate_context(self): """ - Given a dict mapping name -> object as returned by `globals()`, find - all the Setting objects and set their name from the dict key. This is - used to name a bunch of global variables in a module. + Return the context where this setting can be evaluated as a (leaf) + predicate. """ - for name, obj in globs.iteritems(): - if isinstance(obj, Setting): - assert obj.name is None - obj.name = name + return self.group class BoolSetting(Setting): @@ -116,14 +111,17 @@ class SettingGroup(object): opened. :param name: Short mnemonic name for setting group. + :param parent: Parent settings group. """ # The currently open setting group. _current = None - def __init__(self, name): + def __init__(self, name, parent=None): self.name = name + self.parent = parent self.settings = [] + self.predicates = [] self.open() def open(self): @@ -149,13 +147,22 @@ class SettingGroup(object): .format(self, SettingGroup._current)) SettingGroup._current = None if globs: - Setting.extract_names(globs) + from predicates import Predicate + for name, obj in globs.iteritems(): + if isinstance(obj, Setting): + assert obj.name is None, obj.name + obj.name = name + if isinstance(obj, Predicate): + assert obj.name is None + obj.name = name + self.predicates.append(obj) @staticmethod def append(setting): - assert SettingGroup._current, \ - "Open a setting group before defining settings." - SettingGroup._current.settings.append(setting) + g = SettingGroup._current + assert g, "Open a setting group before defining settings." + g.settings.append(setting) + return g # Kinds of operands. diff --git a/meta/cretonne/predicates.py b/meta/cretonne/predicates.py new file mode 100644 index 0000000000..2c34e489e5 --- /dev/null +++ b/meta/cretonne/predicates.py @@ -0,0 +1,94 @@ +""" +Cretonne predicates. + +A *predicate* is a function that computes a boolean result. The inputs to the +function determine the kind of predicate: + +- An *ISA predicate* is evaluated on the current ISA settings together with the + shared settings defined in the :py:mod:`settings` module. Once a target ISA + has been configured, the value of all ISA predicates is known. + +- An *Instruction predicate* is evaluated on an instruction instance, so it can + inspect all the immediate fields and type variables of the instruction. + Instruction predicates can be evaluatd before register allocation, so they + can not depend on specific register assignments to the value operands or + outputs. + +Predicates can also be computed from other predicates using the `And`, `Or`, +and `Not` combinators defined in this module. + +All predicates have a *context* which determines where they can be evaluated. +For an ISA predicate, the context is the ISA settings group. For an instruction +predicate, the context is the instruction format. +""" + + +def _is_parent(a, b): + """ + Return true if a is a parent of b, or equal to it. + """ + while b and a is not b: + b = getattr(b, 'parent', None) + return a is b + + +def _descendant(a, b): + """ + If a is a parent of b or b is a parent of a, return the descendant of the + two. + + If neiher is a parent of the other, return None. + """ + if _is_parent(a, b): + return b + if _is_parent(b, a): + return a + return None + + +class Predicate(object): + """ + Superclass for all computed predicates. + + Leaf predicates can have other types, such as `Setting`. + + :param parts: Tuple of components in the predicate expression. + """ + + def __init__(self, parts): + self.name = None + self.parts = parts + self.context = reduce( + _descendant, + (p.predicate_context() for p in parts)) + assert self.context, "Incompatible predicate parts" + + def predicate_context(self): + return self.context + + +class And(Predicate): + """ + Computed predicate that is true if all parts are true. + """ + + def __init__(self, *args): + super(And, self).__init__(args) + + +class Or(Predicate): + """ + Computed predicate that is true if any parts are true. + """ + + def __init__(self, *args): + super(Or, self).__init__(args) + + +class Not(Predicate): + """ + Computed predicate that is true if its single part is false. + """ + + def __init__(self, part): + super(Not, self).__init__((part,)) diff --git a/meta/isa/riscv/settings.py b/meta/isa/riscv/settings.py index d2366ad829..ade60372d6 100644 --- a/meta/isa/riscv/settings.py +++ b/meta/isa/riscv/settings.py @@ -3,13 +3,17 @@ RISC-V settings. """ from cretonne import SettingGroup, BoolSetting +from cretonne.predicates import And +import cretonne.settings as shared from defs import isa -isa.settings = SettingGroup('riscv') +isa.settings = SettingGroup('riscv', parent=shared.group) supports_m = BoolSetting("CPU supports the 'M' extension (mul/div)") supports_a = BoolSetting("CPU supports the 'A' extension (atomics)") supports_f = BoolSetting("CPU supports the 'F' extension (float)") supports_d = BoolSetting("CPU supports the 'D' extension (double)") +full_float = And(shared.enable_simd, supports_f, supports_d) + isa.settings.close(globals()) From c998c79fe844fc9a2bc1e5187332f63bbbb11e3e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 11 Aug 2016 17:38:56 -0700 Subject: [PATCH 0200/3084] Generate code to precompute predicates. Each ISA predicate is assigned a bit the the Flags struct, and a corresponding method is generated. --- cranelift/src/libcretonne/isa/riscv/mod.rs | 2 +- .../src/libcretonne/isa/riscv/settings.rs | 27 ++++++- meta/cretonne/__init__.py | 9 +++ meta/cretonne/predicates.py | 31 +++++++ meta/gen_settings.py | 81 ++++++++++++++++--- 5 files changed, 135 insertions(+), 15 deletions(-) diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index 641263b9f6..4938a7ad28 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -25,8 +25,8 @@ fn isa_constructor(shared_flags: shared_settings::Flags, builder: shared_settings::Builder) -> Box { Box::new(Isa { + isa_flags: settings::Flags::new(&shared_flags, builder), shared_flags: shared_flags, - isa_flags: settings::Flags::new(builder), }) } diff --git a/cranelift/src/libcretonne/isa/riscv/settings.rs b/cranelift/src/libcretonne/isa/riscv/settings.rs index 335314a3c9..1dd1adc405 100644 --- a/cranelift/src/libcretonne/isa/riscv/settings.rs +++ b/cranelift/src/libcretonne/isa/riscv/settings.rs @@ -1,6 +1,6 @@ //! RISC-V Settings. -use settings::{detail, Builder}; +use settings::{self, detail, Builder}; use std::fmt; // Include code generated by `meta/gen_settings.py`. This file contains a public `Flags` struct @@ -10,16 +10,39 @@ include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs")); #[cfg(test)] mod tests { use super::{builder, Flags}; + use settings::{self, Configurable}; #[test] fn display_default() { + let shared = settings::Flags::new(settings::builder()); let b = builder(); - let f = Flags::new(b); + let f = Flags::new(&shared, b); assert_eq!(f.to_string(), "[riscv]\n\ supports_m = false\n\ supports_a = false\n\ supports_f = false\n\ supports_d = false\n"); + // Predicates are not part of the Display output. + assert_eq!(f.full_float(), false); + } + + #[test] + fn predicates() { + let shared = settings::Flags::new(settings::builder()); + let mut b = builder(); + b.set_bool("supports_f", true).unwrap(); + b.set_bool("supports_d", true).unwrap(); + let f = Flags::new(&shared, b); + assert_eq!(f.full_float(), true); + + let mut sb = settings::builder(); + sb.set_bool("enable_simd", false).unwrap(); + let shared = settings::Flags::new(sb); + let mut b = builder(); + b.set_bool("supports_f", true).unwrap(); + b.set_bool("supports_d", true).unwrap(); + let f = Flags::new(&shared, b); + assert_eq!(f.full_float(), false); } } diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index f576380f65..cc31f47cf5 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -63,6 +63,15 @@ class BoolSetting(Setting): else: return 0 + def rust_predicate(self, prec): + """ + Return the Rust code to compute the value of this setting. + + The emitted code assumes that the setting group exists as a local + variable. + """ + return '{}.{}()'.format(self.group.name, self.name) + class NumSetting(Setting): """ diff --git a/meta/cretonne/predicates.py b/meta/cretonne/predicates.py index 2c34e489e5..dd50613a37 100644 --- a/meta/cretonne/predicates.py +++ b/meta/cretonne/predicates.py @@ -72,23 +72,54 @@ class And(Predicate): Computed predicate that is true if all parts are true. """ + precedence = 2 + def __init__(self, *args): super(And, self).__init__(args) + def rust_predicate(self, prec): + """ + Return a Rust expression computing the value of this predicate. + + The surrounding precedence determines whether parentheses are needed: + + 0. An `if` statement. + 1. An `||` expression. + 2. An `&&` expression. + 3. A `!` expression. + """ + s = ' && '.join(p.rust_predicate(And.precedence) for p in self.parts) + if prec > And.precedence: + s = '({})'.format(s) + return s + class Or(Predicate): """ Computed predicate that is true if any parts are true. """ + precedence = 1 + def __init__(self, *args): super(Or, self).__init__(args) + def rust_predicate(self, prec): + s = ' || '.join(p.rust_predicate(Or.precedence) for p in self.parts) + if prec > Or.precedence: + s = '({})'.format(s) + return s + class Not(Predicate): """ Computed predicate that is true if its single part is false. """ + precedence = 3 + def __init__(self, part): super(Not, self).__init__((part,)) + + def rust_predicate(self, prec): + return '!' + self.parts[0].rust_predicate(Not.precedence) diff --git a/meta/gen_settings.py b/meta/gen_settings.py index 486e71371e..85c154648d 100644 --- a/meta/gen_settings.py +++ b/meta/gen_settings.py @@ -12,7 +12,8 @@ def layout_group(sgrp): """ Layout the settings in sgrp, assigning byte and bit offsets. - Return the next unused byte offset. + Return the number of bytes needed for settings and the total number of + bytes needed when including predicates. """ # Byte offset where booleans are allocated. bool_byte = -1 @@ -37,7 +38,20 @@ def layout_group(sgrp): setting.byte_offset = next_byte next_byte += 1 - return next_byte + settings_size = next_byte + + # Allocate bits for all the precomputed predicates. + for pred in sgrp.predicates: + # Allocate a bit from bool_byte. + if bool_bit > 7: + bool_byte = next_byte + next_byte += 1 + bool_bit = 0 + pred.byte_offset = bool_byte + pred.bit_offset = bool_bit + bool_bit += 1 + + return (settings_size, next_byte) def gen_enum_types(sgrp, fmt): @@ -84,6 +98,18 @@ def gen_getter(setting, fmt): raise AssertionError("Unknown setting kind") +def gen_pred_getter(pred, fmt): + """ + Emit a getter for a pre-computed predicate. + """ + fmt.doc_comment('Computed predicate `{}`.'.format(pred.rust_predicate(0))); + proto = 'pub fn {}(&self) -> bool'.format(pred.name) + with fmt.indented(proto + ' {', '}'): + fmt.line('(self.bytes[{}] & (1 << {})) != 0'.format( + pred.byte_offset, + pred.bit_offset)) + + def gen_getters(sgrp, fmt): """ Emit getter functions for all the settings in fmt. @@ -92,6 +118,8 @@ def gen_getters(sgrp, fmt): with fmt.indented('impl Flags {', '}'): for setting in sgrp.settings: gen_getter(setting, fmt) + for pred in sgrp.predicates: + gen_pred_getter(pred, fmt) def gen_descriptors(sgrp, fmt): @@ -147,11 +175,11 @@ def gen_descriptors(sgrp, fmt): fmt.line('{},'.format(h.descriptor_index)) -def gen_template(sgrp, byte_size, fmt): +def gen_template(sgrp, settings_size, fmt): """ Emit a Template constant. """ - v = [0] * byte_size + v = [0] * settings_size for setting in sgrp.settings: v[setting.byte_offset] |= setting.default_byte() @@ -189,50 +217,79 @@ def gen_display(sgrp, fmt): fmt.line('Ok(())') -def gen_constructor(sgrp, byte_size, parent, fmt): +def gen_constructor(sgrp, settings_size, byte_size, parent, fmt): """ Generate a Flags constructor. """ with fmt.indented('impl Flags {', '}'): - with fmt.indented('pub fn new(builder: Builder) -> Flags {', '}'): + args = 'builder: Builder' + if sgrp.parent: + p = sgrp.parent + args = '{}: &{}::Flags, {}'.format(p.name, p.qual_mod, args) + with fmt.indented( + 'pub fn new({}) -> Flags {{'.format(args), '}'): fmt.line('let bvec = builder.finish("{}");'.format(sgrp.name)) fmt.line('let mut bytes = [0; {}];'.format(byte_size)) - fmt.line('assert_eq!(bytes.len(), bvec.len());') + fmt.line('assert_eq!(bytes.len(), {});'.format(settings_size)) with fmt.indented( 'for (i, b) in bvec.into_iter().enumerate() {', '}'): fmt.line('bytes[i] = b;') - fmt.line('Flags { bytes: bytes }') + + # Stop here without predicates. + if len(sgrp.predicates) == 0: + fmt.line('Flags { bytes: bytes }') + return + + # Now compute the predicates. + fmt.line( + 'let mut {} = Flags {{ bytes: bytes }};' + .format(sgrp.name)) + + for pred in sgrp.predicates: + fmt.comment('Precompute: {}.'.format(pred.name)) + with fmt.indented( + 'if {} {{'.format(pred.rust_predicate(0)), + '}'): + fmt.line( + '{}.bytes[{}] |= 1 << {};' + .format( + sgrp.name, pred.byte_offset, pred.bit_offset)) + + fmt.line(sgrp.name) def gen_group(sgrp, fmt): """ Generate a Flags struct representing `sgrp`. """ - byte_size = layout_group(sgrp) + settings_size, byte_size = layout_group(sgrp) fmt.line('#[derive(Clone)]') fmt.doc_comment('Flags group `{}`.'.format(sgrp.name)) with fmt.indented('pub struct Flags {', '}'): - fmt.line('bytes: [u8; {}],'.format(byte_size)) + fmt.line('bytes: [u8; {}],'.format(settings_size)) - gen_constructor(sgrp, byte_size, None, fmt) + gen_constructor(sgrp, settings_size, byte_size, None, fmt) gen_enum_types(sgrp, fmt) gen_getters(sgrp, fmt) gen_descriptors(sgrp, fmt) - gen_template(sgrp, byte_size, fmt) + gen_template(sgrp, settings_size, fmt) gen_display(sgrp, fmt) def generate(isas, out_dir): # Generate shared settings. fmt = srcgen.Formatter() + settings.group.qual_mod = 'settings' gen_group(settings.group, fmt) fmt.update_file('settings.rs', out_dir) # Generate ISA-specific settings. for isa in isas: if isa.settings: + isa.settings.qual_mod = 'isa::{}::settings'.format( + isa.settings.name) fmt = srcgen.Formatter() gen_group(isa.settings, fmt) fmt.update_file('settings-{}.rs'.format(isa.name), out_dir) From f7f157fb7122e0a56766ad489f6ce43df8877c2d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 12 Aug 2016 10:27:15 -0700 Subject: [PATCH 0201/3084] Move integration tests into src/tools/tests. The integration tests use both libcretonne and libreader, so moving them avoids the circular dev-dependency. Also go back to building everything under src/tools/target to avoid rebuilding the libraries when cargo is invoked in different subdirectories. This speeds up test-all.sh quite a bit. Finally, skip the pure debug build. We build "cargo test" and "cargo build --release" which should cover everything we need. --- cranelift/src/libcretonne/Cargo.toml | 5 ---- .../tests/cfg_traversal.rs | 0 .../tests/dominator_tree.rs | 0 .../src/{libcretonne => tools}/tests/lib.rs | 0 cranelift/test-all.sh | 24 +++++++++++-------- 5 files changed, 14 insertions(+), 15 deletions(-) rename cranelift/src/{libcretonne => tools}/tests/cfg_traversal.rs (100%) rename cranelift/src/{libcretonne => tools}/tests/dominator_tree.rs (100%) rename cranelift/src/{libcretonne => tools}/tests/lib.rs (100%) diff --git a/cranelift/src/libcretonne/Cargo.toml b/cranelift/src/libcretonne/Cargo.toml index cfd886391c..4d9b59bb43 100644 --- a/cranelift/src/libcretonne/Cargo.toml +++ b/cranelift/src/libcretonne/Cargo.toml @@ -11,8 +11,3 @@ build = "build.rs" [lib] name = "cretonne" path = "lib.rs" - -[dev-dependencies] -cretonne-reader = { path = "../libreader" } - -[dependencies] diff --git a/cranelift/src/libcretonne/tests/cfg_traversal.rs b/cranelift/src/tools/tests/cfg_traversal.rs similarity index 100% rename from cranelift/src/libcretonne/tests/cfg_traversal.rs rename to cranelift/src/tools/tests/cfg_traversal.rs diff --git a/cranelift/src/libcretonne/tests/dominator_tree.rs b/cranelift/src/tools/tests/dominator_tree.rs similarity index 100% rename from cranelift/src/libcretonne/tests/dominator_tree.rs rename to cranelift/src/tools/tests/dominator_tree.rs diff --git a/cranelift/src/libcretonne/tests/lib.rs b/cranelift/src/tools/tests/lib.rs similarity index 100% rename from cranelift/src/libcretonne/tests/lib.rs rename to cranelift/src/tools/tests/lib.rs diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 28013ec2e3..4b65c9b56f 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -17,29 +17,33 @@ set -e cd $(dirname "$0") topdir=$(pwd) -PKGS="libcretonne libreader tools" -echo ====== Rust unit tests and debug builds ====== +function banner() { + echo "====== $@ ======" +} + +PKGS="cretonne cretonne-reader cretonne-tools" +cd "$topdir/src/tools" for PKG in $PKGS do - ( - cd $topdir/src/$PKG - cargo test - cargo build - ) + banner "Rust $PKG unit tests" + cargo test -p $PKG done # Build cton-util for parser testing. -echo ====== Rust release build and documentation ====== cd "$topdir/src/tools" +banner "Rust documentation" +echo "open $topdir/src/tools/target/doc/cretonne/index.html" cargo doc +banner "Rust release build" cargo build --release export CTONUTIL="$topdir/src/tools/target/release/cton-util" # Run the parser tests. -echo ====== Parser tests ====== cd "$topdir/tests" +banner "Parser tests" parser/run.sh +banner "CFG tests" cfg/run.sh -echo ====== OK ====== +banner "OK" From 9d6e30df92933e39e5a4ca87c7d3471202e0b127 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 12 Aug 2016 11:03:28 -0700 Subject: [PATCH 0202/3084] Remove tests/lib.rs to avoid running tests twice. The 'cargo test' command simply compiles each 'tests/*.rs' and runs the enclosed tests. The 'lib.rs' source would get run as an individual test. --- cranelift/src/tools/tests/lib.rs | 2 -- 1 file changed, 2 deletions(-) delete mode 100644 cranelift/src/tools/tests/lib.rs diff --git a/cranelift/src/tools/tests/lib.rs b/cranelift/src/tools/tests/lib.rs deleted file mode 100644 index fcd2535269..0000000000 --- a/cranelift/src/tools/tests/lib.rs +++ /dev/null @@ -1,2 +0,0 @@ -pub mod cfg_traversal; -pub mod dominator_tree; From b266c28767e8804bfc06c72db6aa8462f0da7d6b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 12 Aug 2016 16:11:38 -0700 Subject: [PATCH 0203/3084] Re-export common types in the cretonne::ir module. Clients should not have to navigate the ir sub-modules to find commonly used types. --- cranelift/src/libcretonne/cfg.rs | 3 +-- cranelift/src/libcretonne/dominator_tree.rs | 2 +- cranelift/src/libcretonne/ir/dfg.rs | 9 +++++---- cranelift/src/libcretonne/ir/instructions.rs | 7 ++++--- cranelift/src/libcretonne/ir/jumptable.rs | 2 +- cranelift/src/libcretonne/ir/layout.rs | 2 +- cranelift/src/libcretonne/ir/mod.rs | 12 +++++++----- cranelift/src/libcretonne/isa/mod.rs | 3 +-- cranelift/src/libcretonne/isa/riscv/mod.rs | 3 +-- cranelift/src/libcretonne/test_utils/make_inst.rs | 6 +++--- cranelift/src/libcretonne/write.rs | 4 +--- cranelift/src/libreader/lexer.rs | 4 ++-- cranelift/src/libreader/parser.rs | 11 ++++++----- cranelift/src/tools/tests/cfg_traversal.rs | 2 +- cranelift/src/tools/tests/dominator_tree.rs | 2 +- 15 files changed, 36 insertions(+), 36 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 022c3372e4..8397ca5166 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -22,8 +22,7 @@ //! Here Ebb1 and Ebb2 would each have a single predecessor denoted as (Ebb0, `brz vx, Ebb1`) //! and (Ebb0, `jmp Ebb2`) respectively. -use ir::Function; -use ir::entities::{Inst, Ebb}; +use ir::{Function, Inst, Ebb}; use ir::instructions::BranchInfo; use entity_map::{EntityMap, Keys}; use std::collections::HashSet; diff --git a/cranelift/src/libcretonne/dominator_tree.rs b/cranelift/src/libcretonne/dominator_tree.rs index 8420bcf541..5909f68745 100644 --- a/cranelift/src/libcretonne/dominator_tree.rs +++ b/cranelift/src/libcretonne/dominator_tree.rs @@ -1,7 +1,7 @@ /// ! A Dominator Tree represented as mappings of Ebbs to their immediate dominator. use cfg::*; -use ir::entities::Ebb; +use ir::Ebb; use entity_map::{EntityMap, Keys}; pub struct DominatorTree { diff --git a/cranelift/src/libcretonne/ir/dfg.rs b/cranelift/src/libcretonne/ir/dfg.rs index 4776220a6f..3fc4d596eb 100644 --- a/cranelift/src/libcretonne/ir/dfg.rs +++ b/cranelift/src/libcretonne/ir/dfg.rs @@ -1,9 +1,10 @@ //! Data flow graph tracking Instructions, Values, and EBBs. -use entity_map::{EntityMap, PrimaryEntityData}; -use ir::entities::{Ebb, Inst, Value, NO_VALUE, ExpandedValue}; +use ir::{Ebb, Inst, Value, Type}; +use ir::entities::{NO_VALUE, ExpandedValue}; use ir::instructions::InstructionData; -use ir::types::Type; +use entity_map::{EntityMap, PrimaryEntityData}; + use std::ops::{Index, IndexMut}; use std::u16; @@ -359,7 +360,7 @@ impl EbbData { mod tests { use super::*; use ir::types; - use ir::instructions::{Opcode, InstructionData}; + use ir::{Opcode, InstructionData}; #[test] fn make_inst() { diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index 9d5c59eb7d..eee47b8d66 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -10,10 +10,11 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::ops::{Deref, DerefMut}; -use ir::entities::*; -use ir::immediates::*; +use ir::{Value, Type, Ebb, JumpTable}; +use ir::entities::NO_VALUE; +use ir::immediates::{Imm64, Ieee32, Ieee64}; use ir::condcodes::*; -use ir::types::{self, Type}; +use ir::types; // Include code generated by `meta/gen_instr.py`. This file contains: // diff --git a/cranelift/src/libcretonne/ir/jumptable.rs b/cranelift/src/libcretonne/ir/jumptable.rs index 31ea86f643..8a2308fbe1 100644 --- a/cranelift/src/libcretonne/ir/jumptable.rs +++ b/cranelift/src/libcretonne/ir/jumptable.rs @@ -120,7 +120,7 @@ impl Display for JumpTableData { #[cfg(test)] mod tests { use super::JumpTableData; - use ir::entities::Ebb; + use ir::Ebb; use entity_map::EntityRef; #[test] diff --git a/cranelift/src/libcretonne/ir/layout.rs b/cranelift/src/libcretonne/ir/layout.rs index af21ee5234..a983ca178f 100644 --- a/cranelift/src/libcretonne/ir/layout.rs +++ b/cranelift/src/libcretonne/ir/layout.rs @@ -248,7 +248,7 @@ impl<'a> Iterator for Insts<'a> { mod tests { use super::Layout; use entity_map::EntityRef; - use ir::entities::{Ebb, Inst}; + use ir::{Ebb, Inst}; #[test] fn append_ebb() { diff --git a/cranelift/src/libcretonne/ir/mod.rs b/cranelift/src/libcretonne/ir/mod.rs index f0e8eda741..30fb4d9323 100644 --- a/cranelift/src/libcretonne/ir/mod.rs +++ b/cranelift/src/libcretonne/ir/mod.rs @@ -9,14 +9,16 @@ pub mod jumptable; pub mod dfg; pub mod layout; -use ir::types::{FunctionName, Signature}; -use entity_map::{EntityRef, EntityMap, PrimaryEntityData}; -use ir::entities::{StackSlot, JumpTable}; +pub use ir::types::{Type, FunctionName, Signature}; +pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable}; +pub use ir::instructions::{Opcode, InstructionData}; +pub use ir::dfg::DataFlowGraph; +pub use ir::layout::Layout; + use ir::jumptable::JumpTableData; -use ir::dfg::DataFlowGraph; -use ir::layout::Layout; use std::fmt::{self, Debug, Display, Formatter}; use std::ops::Index; +use entity_map::{EntityRef, EntityMap, PrimaryEntityData}; /// A function. pub struct Function { diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index 43834be3ef..a84efdbc52 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -43,8 +43,7 @@ pub mod riscv; use settings; -use ir::dfg::DataFlowGraph; -use ir::entities::Inst; +use ir::{Inst, DataFlowGraph}; /// Look for a supported ISA with the given `name`. /// Return a builder that can create a corresponding `TargetIsa`. diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index 4938a7ad28..b9ffa4925c 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -5,8 +5,7 @@ pub mod settings; use super::super::settings as shared_settings; use super::Builder as IsaBuilder; use super::{TargetIsa, Encoding}; -use ir::dfg::DataFlowGraph; -use ir::entities::Inst; +use ir::{Inst, DataFlowGraph}; #[allow(dead_code)] struct Isa { diff --git a/cranelift/src/libcretonne/test_utils/make_inst.rs b/cranelift/src/libcretonne/test_utils/make_inst.rs index adc45cdb67..eb75b2fab3 100644 --- a/cranelift/src/libcretonne/test_utils/make_inst.rs +++ b/cranelift/src/libcretonne/test_utils/make_inst.rs @@ -1,8 +1,8 @@ //! Helper functions for generating dummy instructions. -use ir::Function; -use ir::entities::{Ebb, Inst, NO_VALUE}; -use ir::instructions::{InstructionData, Opcode, VariableArgs, JumpData, BranchData}; +use ir::{Function, Ebb, Inst, Opcode}; +use ir::entities::NO_VALUE; +use ir::instructions::{InstructionData, VariableArgs, JumpData, BranchData}; use ir::types; pub fn jump(func: &mut Function, dest: Ebb) -> Inst { diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index e7b35e806a..e21c93c554 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -4,10 +4,8 @@ //! equivalent textual representation. This textual representation can be read back by the //! `cretonne-reader` crate. +use ir::{Function, Ebb, Inst, Value, Type}; use std::io::{self, Write}; -use ir::Function; -use ir::entities::{Inst, Ebb, Value}; -use ir::types::Type; pub type Result = io::Result<()>; diff --git a/cranelift/src/libreader/lexer.rs b/cranelift/src/libreader/lexer.rs index a19c14adfd..150c38da19 100644 --- a/cranelift/src/libreader/lexer.rs +++ b/cranelift/src/libreader/lexer.rs @@ -7,7 +7,7 @@ use std::str::CharIndices; use cretonne::ir::types; -use cretonne::ir::entities::{Value, Ebb}; +use cretonne::ir::{Value, Ebb}; /// The location of a `Token` or `Error`. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -374,7 +374,7 @@ impl<'a> Lexer<'a> { mod tests { use super::*; use cretonne::ir::types; - use cretonne::ir::entities::{Value, Ebb}; + use cretonne::ir::{Value, Ebb}; fn token<'a>(token: Token<'a>, line: usize) -> Option, LocatedError>> { Some(super::token(token, Location { line_number: line })) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 41742b54e7..0fc0ff6ab4 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -11,12 +11,13 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::u32; use lexer::{self, Lexer, Token}; -use cretonne::ir::types::{Type, VOID, FunctionName, Signature, ArgumentType, ArgumentExtension}; +use cretonne::ir::{Function, Ebb, Inst, Opcode, Value, Type, FunctionName, StackSlotData, + JumpTable, StackSlot}; +use cretonne::ir::types::{VOID, Signature, ArgumentType, ArgumentExtension}; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; -use cretonne::ir::entities::*; -use cretonne::ir::instructions::{Opcode, InstructionFormat, InstructionData, VariableArgs, - JumpData, BranchData, ReturnData}; -use cretonne::ir::{Function, StackSlotData}; +use cretonne::ir::entities::{NO_EBB, NO_VALUE}; +use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, JumpData, + BranchData, ReturnData}; use cretonne::ir::jumptable::JumpTableData; pub use lexer::Location; diff --git a/cranelift/src/tools/tests/cfg_traversal.rs b/cranelift/src/tools/tests/cfg_traversal.rs index 8908b279d8..03ed5bd148 100644 --- a/cranelift/src/tools/tests/cfg_traversal.rs +++ b/cranelift/src/tools/tests/cfg_traversal.rs @@ -2,7 +2,7 @@ extern crate cretonne; extern crate cton_reader; use self::cton_reader::parser::Parser; -use self::cretonne::ir::entities::Ebb; +use self::cretonne::ir::Ebb; use self::cretonne::cfg::ControlFlowGraph; use self::cretonne::entity_map::EntityMap; diff --git a/cranelift/src/tools/tests/dominator_tree.rs b/cranelift/src/tools/tests/dominator_tree.rs index 8e10307a21..494414b4a9 100644 --- a/cranelift/src/tools/tests/dominator_tree.rs +++ b/cranelift/src/tools/tests/dominator_tree.rs @@ -2,7 +2,7 @@ extern crate cretonne; extern crate cton_reader; use self::cton_reader::parser::Parser; -use self::cretonne::ir::entities::Ebb; +use self::cretonne::ir::Ebb; use self::cretonne::cfg::ControlFlowGraph; use self::cretonne::dominator_tree::DominatorTree; From f4258ae293bc2ce63ba21ede4d648ff855b3b090 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 12 Aug 2016 16:34:18 -0700 Subject: [PATCH 0204/3084] Use an EntityMap for stack slots. Delete the StackSlots iterator and move the remaining StackSlotData into its own module. --- cranelift/src/libcretonne/ir/mod.rs | 107 ++-------------------- cranelift/src/libcretonne/ir/stackslot.rs | 45 +++++++++ cranelift/src/libcretonne/write.rs | 6 +- cranelift/src/libreader/parser.rs | 13 ++- 4 files changed, 63 insertions(+), 108 deletions(-) create mode 100644 cranelift/src/libcretonne/ir/stackslot.rs diff --git a/cranelift/src/libcretonne/ir/mod.rs b/cranelift/src/libcretonne/ir/mod.rs index 30fb4d9323..806466a90b 100644 --- a/cranelift/src/libcretonne/ir/mod.rs +++ b/cranelift/src/libcretonne/ir/mod.rs @@ -1,10 +1,11 @@ //! Representation of Cretonne IL functions. -pub mod entities; pub mod types; +pub mod entities; pub mod condcodes; pub mod immediates; pub mod instructions; +pub mod stackslot; pub mod jumptable; pub mod dfg; pub mod layout; @@ -12,13 +13,13 @@ pub mod layout; pub use ir::types::{Type, FunctionName, Signature}; pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable}; pub use ir::instructions::{Opcode, InstructionData}; +pub use ir::stackslot::StackSlotData; +pub use ir::jumptable::JumpTableData; pub use ir::dfg::DataFlowGraph; pub use ir::layout::Layout; -use ir::jumptable::JumpTableData; -use std::fmt::{self, Debug, Display, Formatter}; -use std::ops::Index; -use entity_map::{EntityRef, EntityMap, PrimaryEntityData}; +use std::fmt::{self, Debug, Formatter}; +use entity_map::{EntityMap, PrimaryEntityData}; /// A function. pub struct Function { @@ -29,7 +30,7 @@ pub struct Function { signature: Signature, /// Stack slots allocated in this function. - stack_slots: Vec, + pub stack_slots: EntityMap, /// Jump tables used in this function. pub jump_tables: EntityMap, @@ -41,7 +42,7 @@ pub struct Function { pub layout: Layout, } -// Tag JumpTableData as a primary entity so jump_tables above . +impl PrimaryEntityData for StackSlotData {} impl PrimaryEntityData for JumpTableData {} impl Function { @@ -50,7 +51,7 @@ impl Function { Function { name: name, signature: sig, - stack_slots: Vec::new(), + stack_slots: EntityMap::new(), jump_tables: EntityMap::new(), dfg: DataFlowGraph::new(), layout: Layout::new(), @@ -66,23 +67,6 @@ impl Function { pub fn own_signature(&self) -> &Signature { &self.signature } - - // Stack slots. - - /// Allocate a new stack slot. - pub fn make_stack_slot(&mut self, data: StackSlotData) -> StackSlot { - let ss = StackSlot::new(self.stack_slots.len()); - self.stack_slots.push(data); - ss - } - - /// Iterate over all stack slots in function. - pub fn stack_slot_iter(&self) -> StackSlotIter { - StackSlotIter { - cur: 0, - end: self.stack_slots.len(), - } - } } impl Debug for Function { @@ -91,76 +75,3 @@ impl Debug for Function { fmt.write_str(&function_to_string(self)) } } - -// ====--------------------------------------------------------------------------------------====// -// -// Stack slot implementation. -// -// ====--------------------------------------------------------------------------------------====// - -/// Contents of a stack slot. -#[derive(Debug)] -pub struct StackSlotData { - /// Size of stack slot in bytes. - pub size: u32, -} - -impl StackSlotData { - /// Create a stack slot with the specified byte size. - pub fn new(size: u32) -> StackSlotData { - StackSlotData { size: size } - } -} - -impl Display for StackSlotData { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write!(fmt, "stack_slot {}", self.size) - } -} - -/// Allow immutable access to stack slots via function indexing. -impl Index for Function { - type Output = StackSlotData; - - fn index<'a>(&'a self, ss: StackSlot) -> &'a StackSlotData { - &self.stack_slots[ss.index()] - } -} - -/// Stack slot iterator visits all stack slots in a function, returning `StackSlot` references. -pub struct StackSlotIter { - cur: usize, - end: usize, -} - -impl Iterator for StackSlotIter { - type Item = StackSlot; - - fn next(&mut self) -> Option { - if self.cur < self.end { - let ss = StackSlot::new(self.cur); - self.cur += 1; - Some(ss) - } else { - None - } - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn stack_slot() { - let mut func = Function::new(); - - let ss0 = func.make_stack_slot(StackSlotData::new(4)); - let ss1 = func.make_stack_slot(StackSlotData::new(8)); - assert_eq!(ss0.to_string(), "ss0"); - assert_eq!(ss1.to_string(), "ss1"); - - assert_eq!(func[ss0].size, 4); - assert_eq!(func[ss1].size, 8); - } -} diff --git a/cranelift/src/libcretonne/ir/stackslot.rs b/cranelift/src/libcretonne/ir/stackslot.rs new file mode 100644 index 0000000000..31bee66eda --- /dev/null +++ b/cranelift/src/libcretonne/ir/stackslot.rs @@ -0,0 +1,45 @@ +//! Stack slots. +//! +//! The `StackSlotData` struct keeps track of a single stack slot in a function. +//! + +use std::fmt::{self, Display, Formatter}; + +/// Contents of a stack slot. +#[derive(Debug)] +pub struct StackSlotData { + /// Size of stack slot in bytes. + pub size: u32, +} + +impl StackSlotData { + /// Create a stack slot with the specified byte size. + pub fn new(size: u32) -> StackSlotData { + StackSlotData { size: size } + } +} + +impl Display for StackSlotData { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write!(fmt, "stack_slot {}", self.size) + } +} + +#[cfg(test)] +mod tests { + use ir::Function; + use super::StackSlotData; + + #[test] + fn stack_slot() { + let mut func = Function::new(); + + let ss0 = func.stack_slots.push(StackSlotData::new(4)); + let ss1 = func.stack_slots.push(StackSlotData::new(8)); + assert_eq!(ss0.to_string(), "ss0"); + assert_eq!(ss1.to_string(), "ss1"); + + assert_eq!(func.stack_slots[ss0].size, 4); + assert_eq!(func.stack_slots[ss1].size, 8); + } +} diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index e21c93c554..a20007ed05 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -68,9 +68,9 @@ fn write_spec(w: &mut Write, func: &Function) -> Result { fn write_preamble(w: &mut Write, func: &Function) -> io::Result { let mut any = false; - for ss in func.stack_slot_iter() { + for ss in func.stack_slots.keys() { any = true; - try!(writeln!(w, " {} = {}", ss, func[ss])); + try!(writeln!(w, " {} = {}", ss, func.stack_slots[ss])); } for jt in func.jump_tables.keys() { @@ -249,7 +249,7 @@ mod tests { f.name.push_str("foo"); assert_eq!(function_to_string(&f), "function foo() {\n}\n"); - f.make_stack_slot(StackSlotData::new(4)); + f.stack_slots.push(StackSlotData::new(4)); assert_eq!(function_to_string(&f), "function foo() {\n ss0 = stack_slot 4\n}\n"); diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 0fc0ff6ab4..87c336906a 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -11,14 +11,13 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::u32; use lexer::{self, Lexer, Token}; -use cretonne::ir::{Function, Ebb, Inst, Opcode, Value, Type, FunctionName, StackSlotData, - JumpTable, StackSlot}; +use cretonne::ir::{Function, Ebb, Inst, Opcode, Value, Type, FunctionName, StackSlot, + StackSlotData, JumpTable, JumpTableData}; use cretonne::ir::types::{VOID, Signature, ArgumentType, ArgumentExtension}; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::{NO_EBB, NO_VALUE}; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, JumpData, BranchData, ReturnData}; -use cretonne::ir::jumptable::JumpTableData; pub use lexer::Location; @@ -95,7 +94,7 @@ impl Context { // Allocate a new stack slot and add a mapping number -> StackSlot. fn add_ss(&mut self, number: u32, data: StackSlotData, loc: &Location) -> Result<()> { - if self.stack_slots.insert(number, self.function.make_stack_slot(data)).is_some() { + if self.stack_slots.insert(number, self.function.stack_slots.push(data)).is_some() { err!(loc, "duplicate stack slot: ss{}", number) } else { Ok(()) @@ -1173,13 +1172,13 @@ mod tests { .parse_function() .unwrap(); assert_eq!(func.name, "foo"); - let mut iter = func.stack_slot_iter(); + let mut iter = func.stack_slots.keys(); let ss0 = iter.next().unwrap(); assert_eq!(ss0.to_string(), "ss0"); - assert_eq!(func[ss0].size, 13); + assert_eq!(func.stack_slots[ss0].size, 13); let ss1 = iter.next().unwrap(); assert_eq!(ss1.to_string(), "ss1"); - assert_eq!(func[ss1].size, 1); + assert_eq!(func.stack_slots[ss1].size, 1); assert_eq!(iter.next(), None); // Catch duplicate definitions. From f17ea61593d9a1b53f5e7861ac99d3d990af3fdd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 12 Aug 2016 16:41:48 -0700 Subject: [PATCH 0205/3084] Move ir::Function into a sub-module. Keep the top-level ir module free of implementation details that are inadvertently exposed to sub-modules. --- cranelift/src/libcretonne/ir/function.rs | 64 ++++++++++++++++++++++++ cranelift/src/libcretonne/ir/mod.rs | 60 +--------------------- 2 files changed, 66 insertions(+), 58 deletions(-) create mode 100644 cranelift/src/libcretonne/ir/function.rs diff --git a/cranelift/src/libcretonne/ir/function.rs b/cranelift/src/libcretonne/ir/function.rs new file mode 100644 index 0000000000..2f8344d801 --- /dev/null +++ b/cranelift/src/libcretonne/ir/function.rs @@ -0,0 +1,64 @@ +//! Intermediate representation of a function. +//! +//! The `Function` struct defined in this module owns all of its extended basic blocks and +//! instructions. + +use ir::{FunctionName, Signature, StackSlot, StackSlotData, JumpTable, JumpTableData, + DataFlowGraph, Layout}; +use entity_map::{EntityMap, PrimaryEntityData}; +use std::fmt::{self, Debug, Formatter}; + +/// A function. +pub struct Function { + /// Name of this function. Mostly used by `.cton` files. + pub name: FunctionName, + + /// Signature of this function. + signature: Signature, + + /// Stack slots allocated in this function. + pub stack_slots: EntityMap, + + /// Jump tables used in this function. + pub jump_tables: EntityMap, + + /// Data flow graph containing the primary definition of all instructions, EBBs and values. + pub dfg: DataFlowGraph, + + /// Layout of EBBs and instructions in the function body. + pub layout: Layout, +} + +impl PrimaryEntityData for StackSlotData {} +impl PrimaryEntityData for JumpTableData {} + +impl Function { + /// Create a function with the given name and signature. + pub fn with_name_signature(name: FunctionName, sig: Signature) -> Function { + Function { + name: name, + signature: sig, + stack_slots: EntityMap::new(), + jump_tables: EntityMap::new(), + dfg: DataFlowGraph::new(), + layout: Layout::new(), + } + } + + /// Create a new empty, anomymous function. + pub fn new() -> Function { + Self::with_name_signature(FunctionName::new(), Signature::new()) + } + + /// Get the signature of this function. + pub fn own_signature(&self) -> &Signature { + &self.signature + } +} + +impl Debug for Function { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + use write::function_to_string; + fmt.write_str(&function_to_string(self)) + } +} diff --git a/cranelift/src/libcretonne/ir/mod.rs b/cranelift/src/libcretonne/ir/mod.rs index 806466a90b..91f8b2516c 100644 --- a/cranelift/src/libcretonne/ir/mod.rs +++ b/cranelift/src/libcretonne/ir/mod.rs @@ -9,6 +9,7 @@ pub mod stackslot; pub mod jumptable; pub mod dfg; pub mod layout; +pub mod function; pub use ir::types::{Type, FunctionName, Signature}; pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable}; @@ -17,61 +18,4 @@ pub use ir::stackslot::StackSlotData; pub use ir::jumptable::JumpTableData; pub use ir::dfg::DataFlowGraph; pub use ir::layout::Layout; - -use std::fmt::{self, Debug, Formatter}; -use entity_map::{EntityMap, PrimaryEntityData}; - -/// A function. -pub struct Function { - /// Name of this function. Mostly used by `.cton` files. - pub name: FunctionName, - - /// Signature of this function. - signature: Signature, - - /// Stack slots allocated in this function. - pub stack_slots: EntityMap, - - /// Jump tables used in this function. - pub jump_tables: EntityMap, - - /// Data flow graph containing the primary definition of all instructions, EBBs and values. - pub dfg: DataFlowGraph, - - /// Layout of EBBs and instructions in the function body. - pub layout: Layout, -} - -impl PrimaryEntityData for StackSlotData {} -impl PrimaryEntityData for JumpTableData {} - -impl Function { - /// Create a function with the given name and signature. - pub fn with_name_signature(name: FunctionName, sig: Signature) -> Function { - Function { - name: name, - signature: sig, - stack_slots: EntityMap::new(), - jump_tables: EntityMap::new(), - dfg: DataFlowGraph::new(), - layout: Layout::new(), - } - } - - /// Create a new empty, anomymous function. - pub fn new() -> Function { - Self::with_name_signature(FunctionName::new(), Signature::new()) - } - - /// Get the signature of this function. - pub fn own_signature(&self) -> &Signature { - &self.signature - } -} - -impl Debug for Function { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - use write::function_to_string; - fmt.write_str(&function_to_string(self)) - } -} +pub use ir::function::Function; From 722a3a6ae6918a900f54c9d65909f98f6dfb1590 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 18 Aug 2016 12:32:46 -0700 Subject: [PATCH 0206/3084] Add a generic implementation of quadratic hash table probing. We have multiple pre-computed constant hash tables that all use the same quadratic probing algorithm. Add a constant_hash Rust module to match the meta/constant_hash.py module. Move the simple_hash() function into constant_hash. Its Python equivalent is in the constant_hash.py module. --- cranelift/src/libcretonne/constant_hash.rs | 74 ++++++++++++++++++++ cranelift/src/libcretonne/ir/instructions.rs | 33 ++++----- cranelift/src/libcretonne/lib.rs | 2 +- cranelift/src/libcretonne/settings.rs | 2 +- cranelift/src/libcretonne/simple_hash.rs | 21 ------ 5 files changed, 90 insertions(+), 42 deletions(-) create mode 100644 cranelift/src/libcretonne/constant_hash.rs delete mode 100644 cranelift/src/libcretonne/simple_hash.rs diff --git a/cranelift/src/libcretonne/constant_hash.rs b/cranelift/src/libcretonne/constant_hash.rs new file mode 100644 index 0000000000..fd8115c7fa --- /dev/null +++ b/cranelift/src/libcretonne/constant_hash.rs @@ -0,0 +1,74 @@ +//! Runtime support for precomputed constant hash tables. +//! +//! The `meta/constant_hash.py` Python module can generate constant hash tables using open +//! addressing and quadratic probing. The hash tables are arrays that are guaranteed to: +//! +//! - Have a power-of-two size. +//! - Contain at least one empty slot. +//! +//! This module provides runtime support for lookups in these tables. + +/// Trait that must be implemented by the entries in a constant hash table. +pub trait Table { + /// Get the number of entries in this table which must be a power of two. + fn len(&self) -> usize; + + /// Get the key corresponding to the entry at `idx`, or `None` if the entry is empty. + /// The `idx` must be in range. + fn key(&self, idx: usize) -> Option; +} + + +/// Look for `key` in `table`. +/// +/// The provided `hash` value must have been computed from `key` using the same hash function that +/// was used to construct the table. +/// +/// Returns the table index containing the found entry, or `None` if no entry could be found. +pub fn probe + ?Sized>(table: &T, key: K, hash: usize) -> Option { + debug_assert!(table.len().is_power_of_two()); + let mask = table.len() - 1; + + let mut idx = hash; + let mut step = 0; + + loop { + idx &= mask; + + match table.key(idx) { + None => return None, + Some(k) if k == key => return Some(idx), + _ => {} + } + + // Quadratic probing. + step += 1; + // When `table.len()` is a power of two, it can be proven that `idx` will visit all + // entries. This means that this loop will always terminate if the hash table has even + // one unused entry. + debug_assert!(step < table.len()); + idx += step; + } +} + +/// A primitive hash function for matching opcodes. +/// Must match `meta/constant_hash.py`. +pub fn simple_hash(s: &str) -> usize { + let mut h: u32 = 5381; + for c in s.chars() { + h = (h ^ c as u32).wrapping_add(h.rotate_right(6)); + } + h as usize +} + +#[cfg(test)] +mod tests { + use super::simple_hash; + + #[test] + fn basic() { + // c.f. meta/constant_hash.py tests. + assert_eq!(simple_hash("Hello"), 0x2fa70c01); + assert_eq!(simple_hash("world"), 0x5b0c31d5); + } +} diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index eee47b8d66..ca43a3dc31 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -64,30 +64,25 @@ impl FromStr for Opcode { /// Parse an Opcode name from a string. fn from_str(s: &str) -> Result { - use simple_hash::simple_hash; - let tlen = OPCODE_HASH_TABLE.len(); - assert!(tlen.is_power_of_two()); - let mut idx = simple_hash(s) as usize; - let mut step: usize = 0; - loop { - idx = idx % tlen; - let entry = OPCODE_HASH_TABLE[idx]; + use constant_hash::{Table, simple_hash, probe}; - if entry == Opcode::NotAnOpcode { - return Err("Unknown opcode"); + impl<'a> Table<&'a str> for [Opcode] { + fn len(&self) -> usize { + self.len() } - if *opcode_name(entry) == *s { - return Ok(entry); + fn key(&self, idx: usize) -> Option<&'a str> { + if self[idx] == Opcode::NotAnOpcode { + None + } else { + Some(opcode_name(self[idx])) + } } + } - // Quadratic probing. - step += 1; - // When `tlen` is a power of two, it can be proven that idx will visit all entries. - // This means that this loop will always terminate if the hash table has even one - // unused entry. - assert!(step < tlen); - idx += step; + match probe::<&str, [Opcode]>(&OPCODE_HASH_TABLE, s, simple_hash(s)) { + None => Err("Unknown opcode"), + Some(i) => Ok(OPCODE_HASH_TABLE[i]), } } } diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index a8cd09ab8d..a4ed7dfecb 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -15,6 +15,6 @@ pub mod dominator_tree; pub mod entity_map; pub mod settings; -mod simple_hash; +mod constant_hash; #[cfg(test)]pub mod test_utils; diff --git a/cranelift/src/libcretonne/settings.rs b/cranelift/src/libcretonne/settings.rs index c65c5eeb68..72e8f06a58 100644 --- a/cranelift/src/libcretonne/settings.rs +++ b/cranelift/src/libcretonne/settings.rs @@ -23,7 +23,7 @@ use std::fmt; use std::result; -use simple_hash::simple_hash; +use constant_hash::simple_hash; /// A string-based configurator for settings groups. /// diff --git a/cranelift/src/libcretonne/simple_hash.rs b/cranelift/src/libcretonne/simple_hash.rs deleted file mode 100644 index 99cd80015c..0000000000 --- a/cranelift/src/libcretonne/simple_hash.rs +++ /dev/null @@ -1,21 +0,0 @@ -/// A primitive hash function for matching opcodes. -/// Must match `meta/constant_hash.py`. -pub fn simple_hash(s: &str) -> u32 { - let mut h: u32 = 5381; - for c in s.chars() { - h = (h ^ c as u32).wrapping_add(h.rotate_right(6)); - } - h -} - -#[cfg(test)] -mod tests { - use super::simple_hash; - - #[test] - fn basic() { - // c.f. meta/constant_hash.py tests. - assert_eq!(simple_hash("Hello"), 0x2fa70c01); - assert_eq!(simple_hash("world"), 0x5b0c31d5); - } -} From f9850b1405f6cc86b9d46d50f531c052ffcea017 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 18 Aug 2016 13:23:46 -0700 Subject: [PATCH 0207/3084] Use shared quadratic probing for settings. --- cranelift/src/libcretonne/settings.rs | 44 ++++++++++++++------------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/cranelift/src/libcretonne/settings.rs b/cranelift/src/libcretonne/settings.rs index 72e8f06a58..b3cf494954 100644 --- a/cranelift/src/libcretonne/settings.rs +++ b/cranelift/src/libcretonne/settings.rs @@ -23,7 +23,7 @@ use std::fmt; use std::result; -use constant_hash::simple_hash; +use constant_hash::{probe, simple_hash}; /// A string-based configurator for settings groups. /// @@ -75,27 +75,12 @@ impl Builder { /// Look up a descriptor by name. fn lookup(&self, name: &str) -> Result<(usize, detail::Detail)> { - let table = self.template.hash_table; - let descs = self.template.descriptors; - let mask = table.len() - 1; - assert!((mask + 1).is_power_of_two()); - - let mut idx = simple_hash(name) as usize; - let mut step: usize = 0; - - loop { - idx = idx & mask; - let entry = table[idx] as usize; - if entry >= descs.len() { - return Err(Error::BadName); + match probe(self.template, name, simple_hash(name)) { + None => Err(Error::BadName), + Some(entry) => { + let d = &self.template.descriptors[self.template.hash_table[entry] as usize]; + Ok((d.offset as usize, d.detail)) } - let desc = &descs[entry]; - if desc.name == name { - return Ok((desc.offset as usize, desc.detail)); - } - step += 1; - assert!(step <= mask); - idx += step; } } } @@ -167,6 +152,7 @@ pub type Result = result::Result; /// code in other modules. pub mod detail { use std::fmt; + use constant_hash; /// An instruction group template. pub struct Template { @@ -207,6 +193,22 @@ pub mod detail { } } + /// The template contains a hash table for by-name lookup. + impl<'a> constant_hash::Table<&'a str> for Template { + fn len(&self) -> usize { + self.hash_table.len() + } + + fn key(&self, idx: usize) -> Option<&'a str> { + let e = self.hash_table[idx] as usize; + if e < self.descriptors.len() { + Some(self.descriptors[e].name) + } else { + None + } + } + } + /// A setting descriptor holds the information needed to generically set and print a setting. /// /// Each settings group will be represented as a constant DESCRIPTORS array. From 9b8d2a04fbcb8827c95a70ffe4e0691af9af96ef Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Thu, 18 Aug 2016 14:37:32 -0700 Subject: [PATCH 0208/3084] Add basic block information to the dominator tree. To be complete the dominator tree must represent idoms as Ebb, Inst pairs, i.e. bais blocks. --- cranelift/src/libcretonne/dominator_tree.rs | 64 +++++++++++++++------ cranelift/src/tools/tests/dominator_tree.rs | 37 ++++++++++-- 2 files changed, 79 insertions(+), 22 deletions(-) diff --git a/cranelift/src/libcretonne/dominator_tree.rs b/cranelift/src/libcretonne/dominator_tree.rs index 5909f68745..7c32a8e7f4 100644 --- a/cranelift/src/libcretonne/dominator_tree.rs +++ b/cranelift/src/libcretonne/dominator_tree.rs @@ -2,10 +2,11 @@ use cfg::*; use ir::Ebb; +use ir::entities::NO_INST; use entity_map::{EntityMap, Keys}; pub struct DominatorTree { - data: EntityMap>, + data: EntityMap>, } impl DominatorTree { @@ -28,7 +29,7 @@ impl DominatorTree { let mut changed = false; if len > 0 { - data[ebbs[0]] = Some(ebbs[0]); + data[ebbs[0]] = Some((ebbs[0], NO_INST)); changed = true; } @@ -39,20 +40,21 @@ impl DominatorTree { let preds = cfg.get_predecessors(ebb); let mut new_idom = None; - for &(p, _) in preds { + for &(p_ebb, _) in preds { if new_idom == None { - new_idom = Some(p); + new_idom = Some((p_ebb, NO_INST)); continue; } - // If this predecessor `p` has an idom available find its common + // If this predecessor has an idom available find its common // ancestor with the current value of new_idom. - if let Some(_) = data[p] { + if let Some(_) = data[p_ebb] { new_idom = match new_idom { Some(cur_idom) => { - Some(DominatorTree::intersect(&mut data, - &postorder_map, - p, - cur_idom)) + Some((DominatorTree::intersect(&mut data, + &postorder_map, + p_ebb, + cur_idom.0), + NO_INST)) } None => panic!("A 'current idom' should have been set!"), } @@ -73,11 +75,36 @@ impl DominatorTree { } } } + + // At this point the basic blocks in the tree are incomplete + // since they have all been set with NO_INST. Here we add instructions + // by iterating through each Ebb -> BasicBlock mapping in the dominator + // tree and replacing the basic block with a corresponding predecessor + // from the Ebb (on the left hand side). + // + // The predecessor chosen should have the lowest instruction number and + // an Ebb which matches the Ebb from the dummy basic block. Because + // extended basic blocks have a single entry point this will always + // result in the correct basic block being chosen. + for lhs_ebb in ebbs { + let rhs_bb = data[lhs_ebb].unwrap(); + for pred_bb in cfg.get_predecessors(lhs_ebb) { + if rhs_bb.0 == pred_bb.0 { + // Predecessors are added in order while iterating through + // instructions from lowest to highest. Because of this, + // the first match we encounter will have the lowest instruction + // number. + data[lhs_ebb] = Some(pred_bb.clone()); + break; + } + } + } + DominatorTree { data: data } } /// Find the common dominator of two ebbs. - fn intersect(data: &EntityMap>, + fn intersect(data: &EntityMap>, ordering: &EntityMap, first: Ebb, second: Ebb) @@ -91,10 +118,10 @@ impl DominatorTree { // self.data[b] to contain non-None entries. while a != b { while ordering[a] < ordering[b] { - a = data[a].unwrap(); + a = data[a].unwrap().0; } while ordering[b] < ordering[a] { - b = data[b].unwrap(); + b = data[b].unwrap().0; } } a @@ -102,7 +129,7 @@ impl DominatorTree { /// Returns the immediate dominator of some ebb or None if the /// node is unreachable. - pub fn idom(&self, ebb: Ebb) -> Option { + pub fn idom(&self, ebb: Ebb) -> Option { self.data[ebb].clone() } @@ -116,6 +143,7 @@ impl DominatorTree { mod test { use super::*; use ir::Function; + use ir::entities::NO_INST; use cfg::ControlFlowGraph; use test_utils::make_inst; @@ -153,9 +181,9 @@ mod test { let dt = DominatorTree::new(&cfg); assert_eq!(func.layout.entry_block().unwrap(), ebb3); - assert_eq!(dt.idom(ebb3).unwrap(), ebb3); - assert_eq!(dt.idom(ebb1).unwrap(), ebb3); - assert_eq!(dt.idom(ebb2).unwrap(), ebb1); - assert_eq!(dt.idom(ebb0).unwrap(), ebb1); + assert_eq!(dt.idom(ebb3).unwrap(), (ebb3, NO_INST)); + assert_eq!(dt.idom(ebb1).unwrap(), (ebb3, jmp_ebb3_ebb1)); + assert_eq!(dt.idom(ebb2).unwrap(), (ebb1, jmp_ebb1_ebb2)); + assert_eq!(dt.idom(ebb0).unwrap(), (ebb1, br_ebb1_ebb0)); } } diff --git a/cranelift/src/tools/tests/dominator_tree.rs b/cranelift/src/tools/tests/dominator_tree.rs index 494414b4a9..7642c46587 100644 --- a/cranelift/src/tools/tests/dominator_tree.rs +++ b/cranelift/src/tools/tests/dominator_tree.rs @@ -1,9 +1,11 @@ extern crate cretonne; extern crate cton_reader; -use self::cton_reader::parser::Parser; use self::cretonne::ir::Ebb; +use self::cton_reader::parser::Parser; +use self::cretonne::ir::entities::NO_INST; use self::cretonne::cfg::ControlFlowGraph; +use self::cretonne::ir::instructions::BranchInfo; use self::cretonne::dominator_tree::DominatorTree; fn test_dominator_tree(function_source: &str, idoms: Vec) { @@ -12,9 +14,36 @@ fn test_dominator_tree(function_source: &str, idoms: Vec) { let dtree = DominatorTree::new(&cfg); assert_eq!(dtree.ebbs().collect::>().len(), idoms.len()); for (i, j) in idoms.iter().enumerate() { - let ebb = Ebb::with_number(i.clone() as u32); - let idom = Ebb::with_number(*j); - assert_eq!(dtree.idom(ebb.unwrap()), idom); + let ebb = Ebb::with_number(i.clone() as u32).unwrap(); + let idom_ebb = Ebb::with_number(*j).unwrap(); + let mut idom_inst = NO_INST; + + // Find the first branch/jump instruction which points to the idom_ebb + // and use it to denote our idom basic block. + for inst in func.layout.ebb_insts(idom_ebb) { + match func.dfg[inst].analyze_branch() { + BranchInfo::SingleDest(dest, _) => { + if dest == ebb { + idom_inst = inst; + break; + } + } + BranchInfo::Table(jt) => { + for (_, dest) in func.jump_tables[jt].entries() { + if dest == ebb { + idom_inst = inst; + break; + } + } + // We already found our inst! + if idom_inst != NO_INST { + break; + } + } + BranchInfo::NotABranch => {} + } + } + assert_eq!(dtree.idom(ebb).unwrap(), (idom_ebb, idom_inst)); } } From 08168e9d5006fe37f8f4c90bd6486fa8daf75c02 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 19 Aug 2016 14:27:29 -0700 Subject: [PATCH 0209/3084] Add rotate and shift instructions with immediate amounts. --- meta/cretonne/base.py | 45 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 38 insertions(+), 7 deletions(-) diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 1f17bab749..84d595e73b 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -444,6 +444,8 @@ bnot = Instruction( # Shift/rotate. x = Operand('x', Int, doc='Scalar or vector value to shift') y = Operand('y', iB, doc='Number of bits to shift') +Y = Operand('Y', imm64) + a = Operand('a', Int) rotl = Instruction( @@ -462,6 +464,18 @@ rotr = Instruction( """, ins=(x, y), outs=a) +rotl_imm = Instruction( + 'rotl_imm', r""" + Rotate left by immediate. + """, + ins=(x, Y), outs=a) + +rotr_imm = Instruction( + 'rotr_imm', r""" + Rotate right by immediate. + """, + ins=(x, Y), outs=a) + ishl = Instruction( 'ishl', r""" Integer shift left. Shift the bits in ``x`` towards the MSB by ``y`` @@ -474,9 +488,6 @@ ishl = Instruction( .. math:: s &:= y \pmod B, \\ a &:= x \cdot 2^s \pmod{2^B}. - - .. todo:: Add ``ishl_imm`` variant with an immediate ``y``. - """, ins=(x, y), outs=a) @@ -493,8 +504,6 @@ ushr = Instruction( .. math:: s &:= y \pmod B, \\ a &:= \lfloor x \cdot 2^{-s} \rfloor. - - .. todo:: Add ``ushr_imm`` variant with an immediate ``y``. """, ins=(x, y), outs=a) @@ -505,11 +514,33 @@ sshr = Instruction( shift*. The shift amount is masked to the size of the register. - - .. todo:: Add ``sshr_imm`` variant with an immediate ``y``. """, ins=(x, y), outs=a) +ishl_imm = Instruction( + 'ishl_imm', r""" + Integer shift left by immediate. + + The shift amount is masked to the size of ``x``. + """, + ins=(x, Y), outs=a) + +ushr_imm = Instruction( + 'ushr_imm', r""" + Unsigned shift right by immediate. + + The shift amount is masked to the size of the register. + """, + ins=(x, Y), outs=a) + +sshr_imm = Instruction( + 'sshr_imm', r""" + Signed shift right by immediate. + + The shift amount is masked to the size of the register. + """, + ins=(x, Y), outs=a) + # # Bit counting. # From 24870f0db97a6f3deed2f31f5e1b6e841d8f3726 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 19 Aug 2016 14:54:16 -0700 Subject: [PATCH 0210/3084] Add RISC-V encodings for imediate shifts. Also add the 32-bit shift instructions for RV64. --- meta/isa/riscv/encodings.py | 17 ++++++++++++----- meta/isa/riscv/recipes.py | 20 +++++++++++++++++--- 2 files changed, 29 insertions(+), 8 deletions(-) diff --git a/meta/isa/riscv/encodings.py b/meta/isa/riscv/encodings.py index c148247524..da5870422f 100644 --- a/meta/isa/riscv/encodings.py +++ b/meta/isa/riscv/encodings.py @@ -3,7 +3,7 @@ RISC-V Encodings. """ from cretonne import base from defs import RV32, RV64 -from recipes import OP, R +from recipes import OPIMM, OPIMM32, OP, OP32, R, Rshamt # Basic arithmetic binary instructions are encoded in an R-type instruction. for inst, f3, f7 in [ @@ -17,12 +17,19 @@ for inst, f3, f7 in [ RV64.enc(inst.i64, R, OP(f3, f7)) # Dynamic shifts have the same masking semantics as the cton base instructions -for inst, f3, f7 in [ - (base.ishl, 0b001, 0b0000000), - (base.ushr, 0b101, 0b0000000), - (base.sshr, 0b101, 0b0100000), +for inst, inst_imm, f3, f7 in [ + (base.ishl, base.ishl_imm, 0b001, 0b0000000), + (base.ushr, base.ushr_imm, 0b101, 0b0000000), + (base.sshr, base.sshr_imm, 0b101, 0b0100000), ]: RV32.enc(inst.i32.i32, R, OP(f3, f7)) RV64.enc(inst.i64.i64, R, OP(f3, f7)) + RV64.enc(inst.i32.i32, R, OP32(f3, f7)) # Allow i32 shift amounts in 64-bit shifts. RV64.enc(inst.i64.i32, R, OP(f3, f7)) + RV64.enc(inst.i32.i64, R, OP32(f3, f7)) + + # Immediate shifts. + RV32.enc(inst_imm.i32, Rshamt, OPIMM(f3, f7)) + RV64.enc(inst_imm.i64, Rshamt, OPIMM(f3, f7)) + RV64.enc(inst_imm.i32, Rshamt, OPIMM32(f3, f7)) diff --git a/meta/isa/riscv/recipes.py b/meta/isa/riscv/recipes.py index 9719a4b2ec..49352b817a 100644 --- a/meta/isa/riscv/recipes.py +++ b/meta/isa/riscv/recipes.py @@ -9,7 +9,7 @@ instruction formats described in the reference: Version 2.1 """ from cretonne import EncRecipe -from cretonne.formats import Binary +from cretonne.formats import Binary, BinaryImm # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit # instructions have 11 as the two low bits, with bits 6:2 determining the base @@ -34,9 +34,14 @@ def BRANCH(funct3): return 0b11000 | (funct3 << 5) -def OPIMM(funct3): +def OPIMM(funct3, funct7=0): assert funct3 <= 0b111 - return 0b00100 | (funct3 << 5) + return 0b00100 | (funct3 << 5) | (funct7 << 8) + + +def OPIMM32(funct3, funct7=0): + assert funct3 <= 0b111 + return 0b00110 | (funct3 << 5) | (funct7 << 8) def OP(funct3, funct7): @@ -45,6 +50,15 @@ def OP(funct3, funct7): return 0b01100 | (funct3 << 5) | (funct7 << 8) +def OP32(funct3, funct7): + assert funct3 <= 0b111 + assert funct7 <= 0b1111111 + return 0b01110 | (funct3 << 5) | (funct7 << 8) + + # R-type 32-bit instructions: These are mostly binary arithmetic instructions. # The encbits are `opcode[6:2] | (funct3 << 5) | (funct7 << 8) R = EncRecipe('R', Binary) + +# R-type with an immediate shift amount instead of rs2. +Rshamt = EncRecipe('Rshamt', BinaryImm) From 96b648056d88072eb436ecf60b9f200a149e2fbb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 23 Aug 2016 11:01:08 -0700 Subject: [PATCH 0211/3084] Upgrade to rustfmt 0.6.0 --- cranelift/src/libcretonne/ir/entities.rs | 6 +----- cranelift/src/libcretonne/ir/immediates.rs | 6 +----- cranelift/src/libcretonne/ir/instructions.rs | 5 +---- cranelift/src/libcretonne/lib.rs | 3 ++- cranelift/src/libcretonne/settings.rs | 4 +--- 5 files changed, 6 insertions(+), 18 deletions(-) diff --git a/cranelift/src/libcretonne/ir/entities.rs b/cranelift/src/libcretonne/ir/entities.rs index 33b41e5273..1a887452a7 100644 --- a/cranelift/src/libcretonne/ir/entities.rs +++ b/cranelift/src/libcretonne/ir/entities.rs @@ -42,11 +42,7 @@ impl EntityRef for Ebb { impl Ebb { /// Create a new EBB reference from its number. This corresponds to the ebbNN representation. pub fn with_number(n: u32) -> Option { - if n < u32::MAX { - Some(Ebb(n)) - } else { - None - } + if n < u32::MAX { Some(Ebb(n)) } else { None } } } diff --git a/cranelift/src/libcretonne/ir/immediates.rs b/cranelift/src/libcretonne/ir/immediates.rs index a329d74f70..077ea4205f 100644 --- a/cranelift/src/libcretonne/ir/immediates.rs +++ b/cranelift/src/libcretonne/ir/immediates.rs @@ -56,11 +56,7 @@ impl FromStr for Imm64 { let mut value: u64 = 0; let mut digits = 0; let negative = s.starts_with('-'); - let s2 = if negative { - &s[1..] - } else { - s - }; + let s2 = if negative { &s[1..] } else { s }; if s2.starts_with("0x") { // Hexadecimal. diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index ca43a3dc31..9638d660a5 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -95,10 +95,7 @@ impl FromStr for Opcode { /// `Box` to store the additional information out of line. #[derive(Debug)] pub enum InstructionData { - Nullary { - opcode: Opcode, - ty: Type, - }, + Nullary { opcode: Opcode, ty: Type }, Unary { opcode: Opcode, ty: Type, diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index a4ed7dfecb..7bdda395ab 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -17,4 +17,5 @@ pub mod settings; mod constant_hash; -#[cfg(test)]pub mod test_utils; +#[cfg(test)] +pub mod test_utils; diff --git a/cranelift/src/libcretonne/settings.rs b/cranelift/src/libcretonne/settings.rs index b3cf494954..58731c40bf 100644 --- a/cranelift/src/libcretonne/settings.rs +++ b/cranelift/src/libcretonne/settings.rs @@ -227,9 +227,7 @@ pub mod detail { #[derive(Clone, Copy)] pub enum Detail { /// A boolean setting only uses one bit, numbered from LSB. - Bool { - bit: u8, - }, + Bool { bit: u8 }, /// A numerical setting uses the whole byte. Num, From a9e302e8613df6c689b40a33b0a6d8cec8de4a44 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Tue, 23 Aug 2016 13:30:38 -0700 Subject: [PATCH 0212/3084] Modify the dominator tree's intersect method to interact with Basic Blocks Corresponding changes to test cases are also included. --- cranelift/src/libcretonne/dominator_tree.rs | 68 +++++------- cranelift/src/tools/Cargo.toml | 1 + cranelift/src/tools/tests/dominator_tree.rs | 112 ++++++++++++-------- 3 files changed, 99 insertions(+), 82 deletions(-) diff --git a/cranelift/src/libcretonne/dominator_tree.rs b/cranelift/src/libcretonne/dominator_tree.rs index 7c32a8e7f4..1b99db342d 100644 --- a/cranelift/src/libcretonne/dominator_tree.rs +++ b/cranelift/src/libcretonne/dominator_tree.rs @@ -10,6 +10,11 @@ pub struct DominatorTree { } impl DominatorTree { + /// Insert data directly into a dominator tree. + pub fn from_data(data: EntityMap>) -> DominatorTree { + DominatorTree { data: data } + } + /// Build a dominator tree from a control flow graph using Keith D. Cooper's /// "Simple, Fast Dominator Algorithm." pub fn new(cfg: &ControlFlowGraph) -> DominatorTree { @@ -40,21 +45,20 @@ impl DominatorTree { let preds = cfg.get_predecessors(ebb); let mut new_idom = None; - for &(p_ebb, _) in preds { + for pred in preds { if new_idom == None { - new_idom = Some((p_ebb, NO_INST)); + new_idom = Some(pred.clone()); continue; } // If this predecessor has an idom available find its common // ancestor with the current value of new_idom. - if let Some(_) = data[p_ebb] { + if let Some(_) = data[pred.0] { new_idom = match new_idom { Some(cur_idom) => { Some((DominatorTree::intersect(&mut data, &postorder_map, - p_ebb, - cur_idom.0), - NO_INST)) + *pred, + cur_idom))) } None => panic!("A 'current idom' should have been set!"), } @@ -67,7 +71,7 @@ impl DominatorTree { } Some(idom) => { // Old idom != New idom - if idom != new_idom.unwrap() { + if idom.0 != new_idom.unwrap().0 { data[ebb] = new_idom; changed = true; } @@ -76,39 +80,15 @@ impl DominatorTree { } } - // At this point the basic blocks in the tree are incomplete - // since they have all been set with NO_INST. Here we add instructions - // by iterating through each Ebb -> BasicBlock mapping in the dominator - // tree and replacing the basic block with a corresponding predecessor - // from the Ebb (on the left hand side). - // - // The predecessor chosen should have the lowest instruction number and - // an Ebb which matches the Ebb from the dummy basic block. Because - // extended basic blocks have a single entry point this will always - // result in the correct basic block being chosen. - for lhs_ebb in ebbs { - let rhs_bb = data[lhs_ebb].unwrap(); - for pred_bb in cfg.get_predecessors(lhs_ebb) { - if rhs_bb.0 == pred_bb.0 { - // Predecessors are added in order while iterating through - // instructions from lowest to highest. Because of this, - // the first match we encounter will have the lowest instruction - // number. - data[lhs_ebb] = Some(pred_bb.clone()); - break; - } - } - } - DominatorTree { data: data } } /// Find the common dominator of two ebbs. fn intersect(data: &EntityMap>, ordering: &EntityMap, - first: Ebb, - second: Ebb) - -> Ebb { + first: BasicBlock, + second: BasicBlock) + -> BasicBlock { let mut a = first; let mut b = second; @@ -116,15 +96,23 @@ impl DominatorTree { // visitation number, to ensure that we move upward through the tree. // Walking upward means that we may always expect self.data[a] and // self.data[b] to contain non-None entries. - while a != b { - while ordering[a] < ordering[b] { - a = data[a].unwrap().0; + while a.0 != b.0 { + while ordering[a.0] < ordering[b.0] { + a = data[a.0].unwrap(); } - while ordering[b] < ordering[a] { - b = data[b].unwrap().0; + while ordering[b.0] < ordering[a.0] { + b = data[b.0].unwrap(); } } - a + + // TODO: we can't rely on instruction numbers to always be ordered + // from lowest to highest. Given that, it will be necessary to create + // an abolute mapping to determine the instruction order in the future. + if a.1 == NO_INST || a.1 < b.1 { + a + } else { + b + } } /// Returns the immediate dominator of some ebb or None if the diff --git a/cranelift/src/tools/Cargo.toml b/cranelift/src/tools/Cargo.toml index 8059002f40..b5ee718222 100644 --- a/cranelift/src/tools/Cargo.toml +++ b/cranelift/src/tools/Cargo.toml @@ -14,3 +14,4 @@ cretonne = { path = "../libcretonne" } cretonne-reader = { path = "../libreader" } docopt = "0.6.80" rustc-serialize = "0.3.19" +regex = "0.1.73" diff --git a/cranelift/src/tools/tests/dominator_tree.rs b/cranelift/src/tools/tests/dominator_tree.rs index 7642c46587..1514db74fb 100644 --- a/cranelift/src/tools/tests/dominator_tree.rs +++ b/cranelift/src/tools/tests/dominator_tree.rs @@ -1,49 +1,77 @@ extern crate cretonne; extern crate cton_reader; +extern crate regex; +use regex::Regex; use self::cretonne::ir::Ebb; use self::cton_reader::parser::Parser; +use self::cretonne::ir::function::Function; +use self::cretonne::entity_map::EntityMap; use self::cretonne::ir::entities::NO_INST; use self::cretonne::cfg::ControlFlowGraph; -use self::cretonne::ir::instructions::BranchInfo; use self::cretonne::dominator_tree::DominatorTree; -fn test_dominator_tree(function_source: &str, idoms: Vec) { +/// Construct a dominator tree from specially formatted comments in +/// cton source. Each line with a jump/branch instruction should +/// have a comment of the format: `dominates(n, ..., N)`, where each `n` +/// is the Ebb number for which this instruction is the immediate dominator. +fn dominator_tree_from_source(func: &Function, function_source: &str) -> DominatorTree { + let ebb_re = Regex::new("^[ \t]*ebb[0-9]+.*:").unwrap(); + let dom_re = Regex::new("dominates\\(([0-9,]+)\\)").unwrap(); + let inst_re = Regex::new("^[ \t]*[a-zA-Z0-9]+[^{}]*").unwrap(); + let func_re = Regex::new("^[ \t]*function.*").unwrap(); + + let ebbs = func.layout.ebbs().collect::>(); + let mut data = EntityMap::with_capacity(ebbs.len()); + + if ebbs.len() < 1 { + return DominatorTree::from_data(data); + } + + let mut ebb_offset = 0; + let mut inst_offset = 0; + + let mut cur_ebb = ebbs[0]; + let mut insts = func.layout.ebb_insts(ebbs[ebb_offset]).collect::>(); + + for line in function_source.lines() { + if ebb_re.is_match(line) { + cur_ebb = ebbs[ebb_offset]; + insts = func.layout.ebb_insts(cur_ebb).collect::>(); + ebb_offset += 1; + inst_offset = 0; + } else if inst_re.is_match(line) && !func_re.is_match(line) { + inst_offset += 1; + } + match dom_re.captures(line) { + Some(caps) => { + for s in caps.at(1).unwrap().split(",") { + let this_ebb = Ebb::with_number(s.parse::().unwrap()).unwrap(); + let inst = if inst_offset == 0 { + NO_INST + } else { + insts[inst_offset - 1].clone() + }; + data[this_ebb] = Some((cur_ebb.clone(), inst)); + } + }, + None => continue, + }; + + } + DominatorTree::from_data(data) +} + +fn test_dominator_tree(function_source: &str) { + let func = &Parser::parse(function_source).unwrap()[0]; + let src_dtree = dominator_tree_from_source(&func, function_source); + let cfg = ControlFlowGraph::new(&func); let dtree = DominatorTree::new(&cfg); - assert_eq!(dtree.ebbs().collect::>().len(), idoms.len()); - for (i, j) in idoms.iter().enumerate() { - let ebb = Ebb::with_number(i.clone() as u32).unwrap(); - let idom_ebb = Ebb::with_number(*j).unwrap(); - let mut idom_inst = NO_INST; - // Find the first branch/jump instruction which points to the idom_ebb - // and use it to denote our idom basic block. - for inst in func.layout.ebb_insts(idom_ebb) { - match func.dfg[inst].analyze_branch() { - BranchInfo::SingleDest(dest, _) => { - if dest == ebb { - idom_inst = inst; - break; - } - } - BranchInfo::Table(jt) => { - for (_, dest) in func.jump_tables[jt].entries() { - if dest == ebb { - idom_inst = inst; - break; - } - } - // We already found our inst! - if idom_inst != NO_INST { - break; - } - } - BranchInfo::NotABranch => {} - } - } - assert_eq!(dtree.idom(ebb).unwrap(), (idom_ebb, idom_inst)); + for ebb in func.layout.ebbs() { + assert_eq!(dtree.idom(ebb), src_dtree.idom(ebb)); } } @@ -51,26 +79,26 @@ fn test_dominator_tree(function_source: &str, idoms: Vec) { fn basic() { test_dominator_tree(" function test(i32) { - ebb0(v0: i32): - jump ebb1 + ebb0(v0: i32): ; dominates(0) + jump ebb1 ; dominates(1) ebb1: - brz v0, ebb3 - jump ebb2 + brz v0, ebb3 ; dominates(3) + jump ebb2 ; dominates(2) ebb2: jump ebb3 ebb3: return } - ", vec![0, 0, 1, 1]); + "); } #[test] fn loops() { test_dominator_tree(" function test(i32) { - ebb0(v0: i32): - brz v0, ebb1 - jump ebb2 + ebb0(v0: i32): ; dominates(0) + brz v0, ebb1 ; dominates(1,3,4,5) + jump ebb2 ; dominates(2) ebb1: jump ebb3 ebb2: @@ -85,5 +113,5 @@ fn loops() { brz v0, ebb4 return } - ", vec![0, 0, 0, 0, 0, 0]); + "); } From d8c587a1bf9436729d69bf534de316afb2e3ac60 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Tue, 23 Aug 2016 13:33:51 -0700 Subject: [PATCH 0213/3084] Synchronize regex versions --- cranelift/src/tools/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/src/tools/Cargo.toml b/cranelift/src/tools/Cargo.toml index b5ee718222..79e9d2685d 100644 --- a/cranelift/src/tools/Cargo.toml +++ b/cranelift/src/tools/Cargo.toml @@ -14,4 +14,4 @@ cretonne = { path = "../libcretonne" } cretonne-reader = { path = "../libreader" } docopt = "0.6.80" rustc-serialize = "0.3.19" -regex = "0.1.73" +regex = "0.1.71" From bdefbdeccc10f9066defb761eb5ebafb6e5e2a5b Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Tue, 23 Aug 2016 13:37:04 -0700 Subject: [PATCH 0214/3084] rustfmt changes --- cranelift/src/libcretonne/dominator_tree.rs | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/cranelift/src/libcretonne/dominator_tree.rs b/cranelift/src/libcretonne/dominator_tree.rs index 1b99db342d..a3a00b2cc4 100644 --- a/cranelift/src/libcretonne/dominator_tree.rs +++ b/cranelift/src/libcretonne/dominator_tree.rs @@ -108,11 +108,7 @@ impl DominatorTree { // TODO: we can't rely on instruction numbers to always be ordered // from lowest to highest. Given that, it will be necessary to create // an abolute mapping to determine the instruction order in the future. - if a.1 == NO_INST || a.1 < b.1 { - a - } else { - b - } + if a.1 == NO_INST || a.1 < b.1 { a } else { b } } /// Returns the immediate dominator of some ebb or None if the From c0a9a4fe46f1832b5d834e129808279e35cba111 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Tue, 23 Aug 2016 15:42:31 -0700 Subject: [PATCH 0215/3084] Move dominator tree test cases to their own folder. --- cranelift/src/tools/Cargo.lock | 23 ++++++++ cranelift/src/tools/Cargo.toml | 1 + cranelift/src/tools/tests/dominator_tree.rs | 53 ++++++------------- .../tests/dominator_tree_testdata/basic.cton | 11 ++++ .../tests/dominator_tree_testdata/loops.cton | 18 +++++++ 5 files changed, 69 insertions(+), 37 deletions(-) create mode 100644 cranelift/src/tools/tests/dominator_tree_testdata/basic.cton create mode 100644 cranelift/src/tools/tests/dominator_tree_testdata/loops.cton diff --git a/cranelift/src/tools/Cargo.lock b/cranelift/src/tools/Cargo.lock index 0b85735d6c..d7060f22e8 100644 --- a/cranelift/src/tools/Cargo.lock +++ b/cranelift/src/tools/Cargo.lock @@ -5,6 +5,8 @@ dependencies = [ "cretonne 0.0.0", "cretonne-reader 0.0.0", "docopt 0.6.80 (registry+https://github.com/rust-lang/crates.io-index)", + "glob 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)", + "regex 0.1.71 (registry+https://github.com/rust-lang/crates.io-index)", "rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)", ] @@ -37,6 +39,11 @@ dependencies = [ "strsim 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)", ] +[[package]] +name = "glob" +version = "0.2.11" +source = "registry+https://github.com/rust-lang/crates.io-index" + [[package]] name = "kernel32-sys" version = "0.2.2" @@ -118,3 +125,19 @@ name = "winapi-build" version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" +[metadata] +"checksum aho-corasick 0.5.2 (registry+https://github.com/rust-lang/crates.io-index)" = "2b3fb52b09c1710b961acb35390d514be82e4ac96a9969a8e38565a29b878dc9" +"checksum docopt 0.6.80 (registry+https://github.com/rust-lang/crates.io-index)" = "4cc0acb4ce0828c6a5a11d47baa432fe885881c27428c3a4e473e454ffe57a76" +"checksum glob 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)" = "8be18de09a56b60ed0edf84bc9df007e30040691af7acd1c41874faac5895bfb" +"checksum kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "7507624b29483431c0ba2d82aece8ca6cdba9382bff4ddd0f7490560c056098d" +"checksum libc 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)" = "c96061f0c8a2dc27482e394d82e23073569de41d73cd736672ccd3e5c7471bfd" +"checksum memchr 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)" = "d8b629fb514376c675b98c1421e80b151d3817ac42d7c667717d282761418d20" +"checksum regex 0.1.71 (registry+https://github.com/rust-lang/crates.io-index)" = "e58a1b7d2bfecc0746e8587c30a53d01ea7bc0e98fac54e5aaa375b94338a0cc" +"checksum regex-syntax 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)" = "baa04823ba7be7ed0bed3d0704c7b923019d9c4e4931c5af2804c7c7a0e3d00b" +"checksum rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)" = "6159e4e6e559c81bd706afe9c8fd68f547d3e851ce12e76b1de7914bab61691b" +"checksum strsim 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)" = "e4d73a2c36a4d095ed1a6df5cbeac159863173447f7a82b3f4757426844ab825" +"checksum thread-id 2.0.0 (registry+https://github.com/rust-lang/crates.io-index)" = "a9539db560102d1cef46b8b78ce737ff0bb64e7e18d35b2a5688f7d097d0ff03" +"checksum thread_local 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)" = "55dd963dbaeadc08aa7266bf7f91c3154a7805e32bb94b820b769d2ef3b4744d" +"checksum utf8-ranges 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "a1ca13c08c41c9c3e04224ed9ff80461d97e121589ff27c753a16cb10830ae0f" +"checksum winapi 0.2.7 (registry+https://github.com/rust-lang/crates.io-index)" = "3969e500d618a5e974917ddefd0ba152e4bcaae5eb5d9b8c1fbc008e9e28c24e" +"checksum winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "2d315eee3b34aca4797b2da6b13ed88266e6d612562a0c46390af8299fc699bc" diff --git a/cranelift/src/tools/Cargo.toml b/cranelift/src/tools/Cargo.toml index 79e9d2685d..8164a4e4bb 100644 --- a/cranelift/src/tools/Cargo.toml +++ b/cranelift/src/tools/Cargo.toml @@ -15,3 +15,4 @@ cretonne-reader = { path = "../libreader" } docopt = "0.6.80" rustc-serialize = "0.3.19" regex = "0.1.71" +glob = "0.2.11" diff --git a/cranelift/src/tools/tests/dominator_tree.rs b/cranelift/src/tools/tests/dominator_tree.rs index 1514db74fb..c1be367137 100644 --- a/cranelift/src/tools/tests/dominator_tree.rs +++ b/cranelift/src/tools/tests/dominator_tree.rs @@ -1,8 +1,13 @@ extern crate cretonne; extern crate cton_reader; +extern crate glob; extern crate regex; +use std::env; +use glob::glob; use regex::Regex; +use std::fs::File; +use std::io::Read; use self::cretonne::ir::Ebb; use self::cton_reader::parser::Parser; use self::cretonne::ir::function::Function; @@ -76,42 +81,16 @@ fn test_dominator_tree(function_source: &str) { } #[test] -fn basic() { - test_dominator_tree(" - function test(i32) { - ebb0(v0: i32): ; dominates(0) - jump ebb1 ; dominates(1) - ebb1: - brz v0, ebb3 ; dominates(3) - jump ebb2 ; dominates(2) - ebb2: - jump ebb3 - ebb3: - return - } - "); -} +fn test_all() { + let testdir = format!("{}/tests/dominator_tree_testdata/*.cton", + env::current_dir().unwrap().display()); -#[test] -fn loops() { - test_dominator_tree(" - function test(i32) { - ebb0(v0: i32): ; dominates(0) - brz v0, ebb1 ; dominates(1,3,4,5) - jump ebb2 ; dominates(2) - ebb1: - jump ebb3 - ebb2: - brz v0, ebb4 - jump ebb5 - ebb3: - jump ebb4 - ebb4: - brz v0, ebb3 - jump ebb5 - ebb5: - brz v0, ebb4 - return - } - "); + for entry in glob(&testdir).unwrap() { + let path = entry.unwrap(); + println!("Testing {:?}", path); + let mut file = File::open(&path).unwrap(); + let mut buffer = String::new(); + file.read_to_string(&mut buffer).unwrap(); + test_dominator_tree(&buffer); + } } diff --git a/cranelift/src/tools/tests/dominator_tree_testdata/basic.cton b/cranelift/src/tools/tests/dominator_tree_testdata/basic.cton new file mode 100644 index 0000000000..c274336b79 --- /dev/null +++ b/cranelift/src/tools/tests/dominator_tree_testdata/basic.cton @@ -0,0 +1,11 @@ +function test(i32) { + ebb0(v0: i32): ; dominates(0) + jump ebb1 ; dominates(1) + ebb1: + brz v0, ebb3 ; dominates(3) + jump ebb2 ; dominates(2) + ebb2: + jump ebb3 + ebb3: + return +} diff --git a/cranelift/src/tools/tests/dominator_tree_testdata/loops.cton b/cranelift/src/tools/tests/dominator_tree_testdata/loops.cton new file mode 100644 index 0000000000..87d7780d7b --- /dev/null +++ b/cranelift/src/tools/tests/dominator_tree_testdata/loops.cton @@ -0,0 +1,18 @@ +function test(i32) { + ebb0(v0: i32): ; dominates(0) + brz v0, ebb1 ; dominates(1,3,4,5) + jump ebb2 ; dominates(2) + ebb1: + jump ebb3 + ebb2: + brz v0, ebb4 + jump ebb5 + ebb3: + jump ebb4 + ebb4: + brz v0, ebb3 + jump ebb5 + ebb5: + brz v0, ebb4 + return +} From 9da684780563810e136f87d55fd8fd313b1c172d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 23 Aug 2016 15:16:40 -0700 Subject: [PATCH 0216/3084] Python 3 compat. Try to keep our Python sources compatible with both Python 2.7 and 3. Check with 'pylint --py3k' and 'python -3'. --- meta/build.py | 1 + meta/constant_hash.py | 1 + meta/cretonne/__init__.py | 12 ++++++------ meta/cretonne/base.py | 7 ++++--- meta/cretonne/entities.py | 2 +- meta/cretonne/formats.py | 7 +++---- meta/cretonne/immediates.py | 2 +- meta/cretonne/predicates.py | 2 ++ meta/cretonne/settings.py | 2 +- meta/cretonne/types.py | 2 +- meta/gen_build_deps.py | 6 +++--- meta/gen_instr.py | 2 +- meta/gen_settings.py | 4 ++-- meta/isa/__init__.py | 2 +- meta/isa/riscv/__init__.py | 8 ++++---- meta/isa/riscv/defs.py | 2 +- meta/isa/riscv/encodings.py | 5 +++-- meta/isa/riscv/recipes.py | 1 + meta/isa/riscv/settings.py | 4 ++-- meta/srcgen.py | 2 +- 20 files changed, 40 insertions(+), 34 deletions(-) diff --git a/meta/build.py b/meta/build.py index 6ca4425823..2806426e6d 100644 --- a/meta/build.py +++ b/meta/build.py @@ -2,6 +2,7 @@ # # This script is run from src/libcretonne/build.rs to generate Rust files. +from __future__ import absolute_import import argparse import isa import gen_instr diff --git a/meta/constant_hash.py b/meta/constant_hash.py index 27a30476a4..aa43694a50 100644 --- a/meta/constant_hash.py +++ b/meta/constant_hash.py @@ -5,6 +5,7 @@ The `constant_hash` module can generate constant pre-populated hash tables. We don't attempt parfect hashing, but simply generate an open addressed quadratically probed hash table. """ +from __future__ import absolute_import def simple_hash(s): diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index cc31f47cf5..ecfea17989 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -4,7 +4,7 @@ Cretonne meta language module. This module provides classes and functions used to describe Cretonne instructions. """ - +from __future__ import absolute_import import re import importlib from collections import namedtuple @@ -156,7 +156,7 @@ class SettingGroup(object): .format(self, SettingGroup._current)) SettingGroup._current = None if globs: - from predicates import Predicate + from .predicates import Predicate for name, obj in globs.iteritems(): if isinstance(obj, Setting): assert obj.name is None, obj.name @@ -365,7 +365,7 @@ class IntType(ScalarType): assert bits > 0, 'IntType must have positive number of bits' super(IntType, self).__init__( name='i{:d}'.format(bits), - membytes=bits/8, + membytes=bits // 8, doc="An integer type with {} bits.".format(bits)) self.bits = bits @@ -379,7 +379,7 @@ class FloatType(ScalarType): def __init__(self, bits, doc): assert bits > 0, 'FloatType must have positive number of bits' super(FloatType, self).__init__(name='f{:d}'.format(bits), - membytes=bits/8, doc=doc) + membytes=bits // 8, doc=doc) self.bits = bits def __repr__(self): @@ -393,7 +393,7 @@ class BoolType(ScalarType): assert bits > 0, 'BoolType must have positive number of bits' super(BoolType, self).__init__( name='b{:d}'.format(bits), - membytes=bits/8, + membytes=bits // 8, doc="A boolean type with {} bits.".format(bits)) self.bits = bits @@ -845,7 +845,7 @@ class BoundInstruction(object): assert len(typevars) <= 1 + len(inst.other_typevars) def __str__(self): - return '.'.join([self.inst.name, ] + map(str, self.typevars)) + return '.'.join([self.inst.name, ] + list(map(str, self.typevars))) def bind(self, *args): """ diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 84d595e73b..2dd3c1bf95 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -4,10 +4,11 @@ Cretonne base instruction set. This module defines the basic Cretonne instruction set that all targets support. """ +from __future__ import absolute_import from . import TypeVar, Operand, Instruction, InstructionGroup, variable_args -from types import i8, f32, f64 -from immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc -import entities +from .types import i8, f32, f64 +from .immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc +from . import entities instructions = InstructionGroup("base", "Shared base instruction set") diff --git a/meta/cretonne/entities.py b/meta/cretonne/entities.py index 14587d402f..bd55fa6eed 100644 --- a/meta/cretonne/entities.py +++ b/meta/cretonne/entities.py @@ -3,7 +3,7 @@ The `cretonne.entities` module predefines all the Cretonne entity reference operand types. Thee are corresponding definitions in the `cretonne.entities` Rust module. """ - +from __future__ import absolute_import from . import EntityRefKind diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index 4798b9ed25..efd535b693 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -5,11 +5,10 @@ Every instruction format has a corresponding `InstructionData` variant in the Rust representation of cretonne IL, so all instruction formats must be defined in this module. """ - - +from __future__ import absolute_import from . import InstructionFormat, value, variable_args -from immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc -from entities import ebb, function, jump_table +from .immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc +from .entities import ebb, function, jump_table Nullary = InstructionFormat() diff --git a/meta/cretonne/immediates.py b/meta/cretonne/immediates.py index f20a6142d3..61ba418c9c 100644 --- a/meta/cretonne/immediates.py +++ b/meta/cretonne/immediates.py @@ -2,7 +2,7 @@ The `cretonne.immediates` module predefines all the Cretonne immediate operand types. """ - +from __future__ import absolute_import from . import ImmediateKind #: A 64-bit immediate integer operand. diff --git a/meta/cretonne/predicates.py b/meta/cretonne/predicates.py index dd50613a37..caedae14fb 100644 --- a/meta/cretonne/predicates.py +++ b/meta/cretonne/predicates.py @@ -21,6 +21,8 @@ All predicates have a *context* which determines where they can be evaluated. For an ISA predicate, the context is the ISA settings group. For an instruction predicate, the context is the instruction format. """ +from __future__ import absolute_import +from functools import reduce def _is_parent(a, b): diff --git a/meta/cretonne/settings.py b/meta/cretonne/settings.py index 2876dfd84f..ba95bda36c 100644 --- a/meta/cretonne/settings.py +++ b/meta/cretonne/settings.py @@ -3,7 +3,7 @@ Cretonne shared settings. This module defines settings are are relevant for all code generators. """ - +from __future__ import absolute_import from . import SettingGroup, BoolSetting, EnumSetting group = SettingGroup('shared') diff --git a/meta/cretonne/types.py b/meta/cretonne/types.py index d92e6f9113..05dcb53e16 100644 --- a/meta/cretonne/types.py +++ b/meta/cretonne/types.py @@ -1,7 +1,7 @@ """ The cretonne.types module predefines all the Cretonne scalar types. """ - +from __future__ import absolute_import from . import ScalarType, IntType, FloatType, BoolType #: Boolean. diff --git a/meta/gen_build_deps.py b/meta/gen_build_deps.py index 4d0c0cb260..32ae2f7238 100644 --- a/meta/gen_build_deps.py +++ b/meta/gen_build_deps.py @@ -12,7 +12,7 @@ If the build script outputs lines of the form: cargo will rerun the build script when those files have changed since the last build. """ - +from __future__ import absolute_import, print_function import os from os.path import dirname, abspath, join @@ -30,7 +30,7 @@ def source_files(top): def generate(): - print "Dependencies from meta language directory:" + print("Dependencies from meta language directory:") meta = dirname(abspath(__file__)) for path in source_files(meta): - print "cargo:rerun-if-changed=" + path + print("cargo:rerun-if-changed=" + path) diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 64c726ad9d..92808524c5 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -1,7 +1,7 @@ """ Generate sources with instruction info. """ - +from __future__ import absolute_import import srcgen import constant_hash from unique_table import UniqueTable, UniqueSeqTable diff --git a/meta/gen_settings.py b/meta/gen_settings.py index 85c154648d..65bb7c428a 100644 --- a/meta/gen_settings.py +++ b/meta/gen_settings.py @@ -1,7 +1,7 @@ """ Generate sources with settings. """ - +from __future__ import absolute_import import srcgen from unique_table import UniqueSeqTable import constant_hash @@ -102,7 +102,7 @@ def gen_pred_getter(pred, fmt): """ Emit a getter for a pre-computed predicate. """ - fmt.doc_comment('Computed predicate `{}`.'.format(pred.rust_predicate(0))); + fmt.doc_comment('Computed predicate `{}`.'.format(pred.rust_predicate(0))) proto = 'pub fn {}(&self) -> bool'.format(pred.name) with fmt.indented(proto + ' {', '}'): fmt.line('(self.bytes[{}] & (1 << {})) != 0'.format( diff --git a/meta/isa/__init__.py b/meta/isa/__init__.py index b566a521ef..addc70b8a9 100644 --- a/meta/isa/__init__.py +++ b/meta/isa/__init__.py @@ -5,7 +5,7 @@ Cretonne target ISA definitions The :py:mod:`isa` package contains sub-packages for each target instruction set architecture supported by Cretonne. """ - +from __future__ import absolute_import from . import riscv diff --git a/meta/isa/riscv/__init__.py b/meta/isa/riscv/__init__.py index ce8c47ea3d..1f9ebd3f46 100644 --- a/meta/isa/riscv/__init__.py +++ b/meta/isa/riscv/__init__.py @@ -24,10 +24,10 @@ RV32G / RV64G F, and D instruction sets listed above. """ - -import defs -import encodings -import settings +from __future__ import absolute_import +from . import defs +from . import encodings +from . import settings # Re-export the primary target ISA definition. isa = defs.isa diff --git a/meta/isa/riscv/defs.py b/meta/isa/riscv/defs.py index 6e009ccdf4..4fccbf09a6 100644 --- a/meta/isa/riscv/defs.py +++ b/meta/isa/riscv/defs.py @@ -3,7 +3,7 @@ RISC-V definitions. Commonly used definitions. """ - +from __future__ import absolute_import from cretonne import TargetISA, CPUMode import cretonne.base diff --git a/meta/isa/riscv/encodings.py b/meta/isa/riscv/encodings.py index da5870422f..3ce5102760 100644 --- a/meta/isa/riscv/encodings.py +++ b/meta/isa/riscv/encodings.py @@ -1,9 +1,10 @@ """ RISC-V Encodings. """ +from __future__ import absolute_import from cretonne import base -from defs import RV32, RV64 -from recipes import OPIMM, OPIMM32, OP, OP32, R, Rshamt +from .defs import RV32, RV64 +from .recipes import OPIMM, OPIMM32, OP, OP32, R, Rshamt # Basic arithmetic binary instructions are encoded in an R-type instruction. for inst, f3, f7 in [ diff --git a/meta/isa/riscv/recipes.py b/meta/isa/riscv/recipes.py index 49352b817a..25ea74870f 100644 --- a/meta/isa/riscv/recipes.py +++ b/meta/isa/riscv/recipes.py @@ -8,6 +8,7 @@ instruction formats described in the reference: Volume I: User-Level ISA Version 2.1 """ +from __future__ import absolute_import from cretonne import EncRecipe from cretonne.formats import Binary, BinaryImm diff --git a/meta/isa/riscv/settings.py b/meta/isa/riscv/settings.py index ade60372d6..1ab2917036 100644 --- a/meta/isa/riscv/settings.py +++ b/meta/isa/riscv/settings.py @@ -1,11 +1,11 @@ """ RISC-V settings. """ - +from __future__ import absolute_import from cretonne import SettingGroup, BoolSetting from cretonne.predicates import And import cretonne.settings as shared -from defs import isa +from .defs import isa isa.settings = SettingGroup('riscv', parent=shared.group) diff --git a/meta/srcgen.py b/meta/srcgen.py index 42f39c96de..99bb808435 100644 --- a/meta/srcgen.py +++ b/meta/srcgen.py @@ -5,7 +5,7 @@ The `srcgen` module contains generic helper routines and classes for generating source code. """ - +from __future__ import absolute_import import sys import os import re From 2f322e3d0e7b3f7109b1280b4d06aee898b030a7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 23 Aug 2016 16:54:40 -0700 Subject: [PATCH 0217/3084] Add script for Python 3 compat checks. --- meta/check-py3k.sh | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100755 meta/check-py3k.sh diff --git a/meta/check-py3k.sh b/meta/check-py3k.sh new file mode 100755 index 0000000000..19096339bf --- /dev/null +++ b/meta/check-py3k.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +# Check Python sources for Python 3 compatibility using pylint. +# +# Install pylint with 'pip install pylint'. +cd $(dirname "$0") +pylint --py3k --reports=no -- *.py cretonne isa From 9c5637902c98e1fe74ce1cc0704ce9bd1453b016 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 23 Aug 2016 11:39:47 -0700 Subject: [PATCH 0218/3084] Track the default member name for immediate operands. Usually an instruction firmat has only a single immediate operand called 'imm', or 'cond' if it is one of the condigtion codes. Add a 'default_member' field to ImmediateKind to keep track of this default member name in the InstructionData struct. --- meta/cretonne/__init__.py | 6 +++++- meta/cretonne/immediates.py | 9 +++++++-- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index ecfea17989..c79e0f7bdc 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -237,11 +237,15 @@ variable_args = OperandKind( class ImmediateKind(OperandKind): """ The kind of an immediate instruction operand. + + :param default_member: The default member name of this kind the + `InstructionData` data structure. """ - def __init__(self, name, doc): + def __init__(self, name, doc, default_member='imm'): self.name = name self.__doc__ = doc + self.default_member = default_member def __repr__(self): return 'ImmediateKind({})'.format(self.name) diff --git a/meta/cretonne/immediates.py b/meta/cretonne/immediates.py index 61ba418c9c..ca8affded0 100644 --- a/meta/cretonne/immediates.py +++ b/meta/cretonne/immediates.py @@ -34,11 +34,16 @@ immvector = ImmediateKind('immvector', 'An immediate SIMD vector.') #: #: This enumerated operand kind is used for the :cton:inst:`icmp` instruction #: and corresponds to the `condcodes::IntCC` Rust type. -intcc = ImmediateKind('intcc', 'An integer comparison condition code.') +intcc = ImmediateKind( + 'intcc', + 'An integer comparison condition code.', + default_member='cond') #: A condition code for comparing floating point values. #: #: This enumerated operand kind is used for the :cton:inst:`fcmp` instruction #: and corresponds to the `condcodes::FloatCC` Rust type. floatcc = ImmediateKind( - 'floatcc', 'A floating point comparison condition code.') + 'floatcc', + 'A floating point comparison condition code.', + default_member='cond') From 077c5ce1f684808b7940ff428e3fb51833b11643 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 23 Aug 2016 13:29:29 -0700 Subject: [PATCH 0219/3084] Create format fields for immediate operands. Each InstructionFormat instance gets data members corresponding to its immediate operands, so the can be referred to as BinaryImm.imm, for example. This will be used to construct instruction predicates. --- meta/cretonne/__init__.py | 50 +++++++++++++++++++++++++++++++++++++-- meta/cretonne/formats.py | 4 ++-- 2 files changed, 50 insertions(+), 4 deletions(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index c79e0f7bdc..c6693a7c4a 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -622,9 +622,9 @@ class InstructionFormat(object): def __init__(self, *kinds, **kwargs): self.name = kwargs.get('name', None) - self.kinds = kinds self.multiple_results = kwargs.get('multiple_results', False) self.boxed_storage = kwargs.get('boxed_storage', False) + self.kinds = tuple(self._process_member_names(kinds)) # Which of self.kinds are `value`? self.value_operands = tuple( @@ -640,7 +640,7 @@ class InstructionFormat(object): self.typevar_operand = self.value_operands[0] # Compute a signature for the global registry. - sig = (self.multiple_results,) + kinds + sig = (self.multiple_results,) + self.kinds if sig in InstructionFormat._registry: raise RuntimeError( "Format '{}' has the same signature as existing format '{}'" @@ -648,6 +648,31 @@ class InstructionFormat(object): InstructionFormat._registry[sig] = self InstructionFormat.all_formats.append(self) + def _process_member_names(self, kinds): + """ + Extract names of all the immediate operands in the kinds tuple. + + Each entry is either an `OperandKind` instance, or a `(member, kind)` + pair. The member names correspond to members in the Rust + `InstructionData` data structure. + + Yields the operand kinds. + """ + for i, k in enumerate(kinds): + if isinstance(k, tuple): + member, k = k + else: + member = None + yield k + + # Create `FormatField` instances for the immediates. + if isinstance(k, ImmediateKind): + if not member: + member = k.default_member + assert not hasattr(self, member), "Duplicate member name" + field = FormatField(self, i, member) + setattr(self, member, field) + @staticmethod def lookup(ins, outs): """ @@ -680,6 +705,27 @@ class InstructionFormat(object): obj.name = name +class FormatField(object): + """ + A field in an instruction format. + + This corresponds to a single member of a variant of the `InstructionData` + data type. + + :param format: Parent `InstructionFormat`. + :param operand: Operand number in parent. + :param name: Member name in `InstructionData` variant. + """ + + def __init__(self, format, operand, name): + self.format = format + self.operand = operand + self.name = name + + def __str__(self): + return '{}.{}'.format(self.format.name, self.name) + + class Instruction(object): """ The operands to the instruction are specified as two tuples: ``ins`` and diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index efd535b693..06f4593086 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -30,8 +30,8 @@ BinaryOverflow = InstructionFormat(value, value, multiple_results=True) # The fma instruction has the same constraint on all inputs. Ternary = InstructionFormat(value, value, value, typevar_operand=1) -InsertLane = InstructionFormat(value, uimm8, value) -ExtractLane = InstructionFormat(value, uimm8) +InsertLane = InstructionFormat(value, ('lane', uimm8), value) +ExtractLane = InstructionFormat(value, ('lane', uimm8)) IntCompare = InstructionFormat(intcc, value, value) FloatCompare = InstructionFormat(floatcc, value, value) From b8509c6273a189321a4b6c01b5550a71b079569a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 23 Aug 2016 14:42:03 -0700 Subject: [PATCH 0220/3084] Add bitwise operations with an immediate operand. --- meta/cretonne/base.py | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 2dd3c1bf95..30d890b076 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -442,6 +442,29 @@ bnot = Instruction( """, ins=x, outs=a) +# Bitwise binary ops with immediate arg. +x = Operand('x', iB) +Y = Operand('Y', imm64) +a = Operand('a', iB) + +band_imm = Instruction( + 'band_imm', """ + Bitwise and with immediate. + """, + ins=(x, Y), outs=a) + +bor_imm = Instruction( + 'bor_imm', """ + Bitwise or with immediate. + """, + ins=(x, Y), outs=a) + +bxor_imm = Instruction( + 'bxor_imm', """ + Bitwise xor with immediate. + """, + ins=(x, Y), outs=a) + # Shift/rotate. x = Operand('x', Int, doc='Scalar or vector value to shift') y = Operand('y', iB, doc='Number of bits to shift') From 586fbbc79754842076df320a0918985ed47142d3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 23 Aug 2016 14:45:24 -0700 Subject: [PATCH 0221/3084] Add RISC-V arithmetic w/immediate operand encodings. Add new instruction predicates to support the 'I' encoding recipe: IsSignedInt, IsUnsignedInt used to test that an immediate operand is in the allowed range. --- meta/cretonne/__init__.py | 12 +++++- meta/cretonne/predicates.py | 76 +++++++++++++++++++++++++++++++++++++ meta/isa/riscv/encodings.py | 24 ++++++++---- meta/isa/riscv/recipes.py | 3 ++ 4 files changed, 106 insertions(+), 9 deletions(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index c6693a7c4a..a5e842dd8e 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -725,6 +725,12 @@ class FormatField(object): def __str__(self): return '{}.{}'.format(self.format.name, self.name) + def rust_name(self): + if self.format.boxed_storage: + return 'data.' + self.name + else: + return self.name + class Instruction(object): """ @@ -983,9 +989,13 @@ class EncRecipe(object): :py:class:`InstructionFormat`. """ - def __init__(self, name, format): + def __init__(self, name, format, instp=None, isap=None): self.name = name self.format = format + self.instp = instp + self.isap = isap + if instp: + assert instp.predicate_context() == format class Encoding(object): diff --git a/meta/cretonne/predicates.py b/meta/cretonne/predicates.py index caedae14fb..71a17eb01b 100644 --- a/meta/cretonne/predicates.py +++ b/meta/cretonne/predicates.py @@ -125,3 +125,79 @@ class Not(Predicate): def rust_predicate(self, prec): return '!' + self.parts[0].rust_predicate(Not.precedence) + + +class FieldPredicate(object): + """ + An instruction predicate that performs a test on a single `FormatField`. + + :param field: The `FormatField` to be tested. + :param method: Boolean predicate method to call. + :param args: Arguments for the predicate method. + """ + + def __init__(self, field, method, args): + self.field = field + self.method = method + self.args = args + + def predicate_context(self): + """ + This predicate can be evaluated in the context of an instruction + format. + """ + return self.field.format + + def rust_predicate(self, prec): + """ + Return a string of Rust code that evaluates this predicate. + """ + return '{}.{}({})'.format( + self.field.rust_name(), + self.method, + ', '.join(self.args)) + + +class IsSignedInt(FieldPredicate): + """ + Instruction predicate that checks if an immediate instruction format field + is representable as an n-bit two's complement integer. + + :param field: `FormatField` to be checked. + :param width: Number of bits in the allowed range. + :param scale: Number of low bits that must be 0. + + The predicate is true if the field is in the range: + `-2^(width-1) -- 2^(width-1)-1` + and a multiple of `2^scale`. + """ + + def __init__(self, field, width, scale=0): + super(IsSignedInt, self).__init__( + field, 'is_signed_int', (width, scale)) + self.width = width + self.scale = scale + assert width >= 0 and width <= 64 + assert scale >= 0 and scale < width + + +class IsUnsignedInt(FieldPredicate): + """ + Instruction predicate that checks if an immediate instruction format field + is representable as an n-bit unsigned complement integer. + + :param field: `FormatField` to be checked. + :param width: Number of bits in the allowed range. + :param scale: Number of low bits that must be 0. + + The predicate is true if the field is in the range: + `0 -- 2^width - 1` and a multiple of `2^scale`. + """ + + def __init__(self, field, width, scale=0): + super(IsUnsignedInt, self).__init__( + field, 'is_unsigned_int', (width, scale)) + self.width = width + self.scale = scale + assert width >= 0 and width <= 64 + assert scale >= 0 and scale < width diff --git a/meta/isa/riscv/encodings.py b/meta/isa/riscv/encodings.py index 3ce5102760..099b6b5630 100644 --- a/meta/isa/riscv/encodings.py +++ b/meta/isa/riscv/encodings.py @@ -4,20 +4,28 @@ RISC-V Encodings. from __future__ import absolute_import from cretonne import base from .defs import RV32, RV64 -from .recipes import OPIMM, OPIMM32, OP, OP32, R, Rshamt +from .recipes import OPIMM, OPIMM32, OP, OP32, R, Rshamt, I # Basic arithmetic binary instructions are encoded in an R-type instruction. -for inst, f3, f7 in [ - (base.iadd, 0b000, 0b0000000), - (base.isub, 0b000, 0b0100000), - (base.bxor, 0b100, 0b0000000), - (base.bor, 0b110, 0b0000000), - (base.band, 0b111, 0b0000000) +for inst, inst_imm, f3, f7 in [ + (base.iadd, base.iadd_imm, 0b000, 0b0000000), + (base.isub, None, 0b000, 0b0100000), + (base.bxor, base.bxor_imm, 0b100, 0b0000000), + (base.bor, base.bor_imm, 0b110, 0b0000000), + (base.band, base.band_imm, 0b111, 0b0000000) ]: RV32.enc(inst.i32, R, OP(f3, f7)) RV64.enc(inst.i64, R, OP(f3, f7)) -# Dynamic shifts have the same masking semantics as the cton base instructions + # Immediate versions for add/xor/or/and. + if inst_imm: + RV32.enc(inst_imm.i32, I, OPIMM(f3)) + RV64.enc(inst_imm.i64, I, OPIMM(f3)) + +# There are no andiw/oriw/xoriw variations. +RV64.enc(base.iadd_imm.i32, I, OPIMM32(0b000)) + +# Dynamic shifts have the same masking semantics as the cton base instructions. for inst, inst_imm, f3, f7 in [ (base.ishl, base.ishl_imm, 0b001, 0b0000000), (base.ushr, base.ushr_imm, 0b101, 0b0000000), diff --git a/meta/isa/riscv/recipes.py b/meta/isa/riscv/recipes.py index 25ea74870f..db6d7021c8 100644 --- a/meta/isa/riscv/recipes.py +++ b/meta/isa/riscv/recipes.py @@ -11,6 +11,7 @@ instruction formats described in the reference: from __future__ import absolute_import from cretonne import EncRecipe from cretonne.formats import Binary, BinaryImm +from cretonne.predicates import IsSignedInt # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit # instructions have 11 as the two low bits, with bits 6:2 determining the base @@ -63,3 +64,5 @@ R = EncRecipe('R', Binary) # R-type with an immediate shift amount instead of rs2. Rshamt = EncRecipe('Rshamt', BinaryImm) + +I = EncRecipe('I', BinaryImm, instp=IsSignedInt(BinaryImm.imm, 12)) From 125b79c06a8a037fd86735cf86de75bd90651e37 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Wed, 24 Aug 2016 13:13:31 -0700 Subject: [PATCH 0222/3084] Add additional test cases --- .../dominator_tree_testdata/tall-tree.cton | 31 +++++++++++++++ .../dominator_tree_testdata/wide-tree.cton | 39 +++++++++++++++++++ 2 files changed, 70 insertions(+) create mode 100644 cranelift/src/tools/tests/dominator_tree_testdata/tall-tree.cton create mode 100644 cranelift/src/tools/tests/dominator_tree_testdata/wide-tree.cton diff --git a/cranelift/src/tools/tests/dominator_tree_testdata/tall-tree.cton b/cranelift/src/tools/tests/dominator_tree_testdata/tall-tree.cton new file mode 100644 index 0000000000..d2ea0f8a9e --- /dev/null +++ b/cranelift/src/tools/tests/dominator_tree_testdata/tall-tree.cton @@ -0,0 +1,31 @@ +function test(i32) { + ebb0(v0: i32): ; dominates(0) + brz v0, ebb1 ; dominates(1) + brnz v0, ebb2 ; dominates(2,5) + jump ebb3 ; dominates(3) + ebb1: + jump ebb4 ; dominates(4) + ebb2: + jump ebb5 + ebb3: + jump ebb5 + ebb4: + brz v0, ebb6 ; dominates(6,10) + jump ebb7 ; dominates(7) + ebb5: + return + ebb6: + brz v0, ebb8 ; dominates(11,8) + brnz v0, ebb9 ; dominates(9) + jump ebb10 + ebb7: + jump ebb10 + ebb8: + jump ebb11 + ebb9: + jump ebb11 + ebb10: + return + ebb11: + return +} diff --git a/cranelift/src/tools/tests/dominator_tree_testdata/wide-tree.cton b/cranelift/src/tools/tests/dominator_tree_testdata/wide-tree.cton new file mode 100644 index 0000000000..0883ef6514 --- /dev/null +++ b/cranelift/src/tools/tests/dominator_tree_testdata/wide-tree.cton @@ -0,0 +1,39 @@ +function test(i32) { + ebb0(v0: i32): ; dominates(0) + brz v0, ebb13 ; dominates(13) + jump ebb1 ; dominates(1) + ebb1: + brz v0, ebb2 ; dominates(2,7) + brnz v0, ebb3 ; dominates(3) + brz v0, ebb4 ; dominates(4) + brnz v0, ebb5 ; dominates(5) + jump ebb6 ; dominates(6) + ebb2: + jump ebb7 + ebb3: + jump ebb7 + ebb4: + jump ebb7 + ebb5: + jump ebb7 + ebb6: + jump ebb7 + ebb7: + brnz v0, ebb8 ; dominates(8,12) + brz v0, ebb9 ; dominates(9) + brnz v0, ebb10 ; dominates(10) + jump ebb11 ; dominates(11) + ebb8: + jump ebb12 + ebb9: + jump ebb12 + ebb10: + brz v0, ebb13 + jump ebb12 + ebb11: + jump ebb13 + ebb12: + return + ebb13: + return +} From 055c7a03742b87613167da33a751cae2147a7559 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 24 Aug 2016 14:37:32 -0700 Subject: [PATCH 0223/3084] Clarify that Imm64 holds sign-extended values. When representing smaller integer types, immediate values should be sign-extended to i64. --- cranelift/src/libcretonne/ir/immediates.rs | 14 +++++++++----- cranelift/src/libreader/parser.rs | 7 +++++-- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/cranelift/src/libcretonne/ir/immediates.rs b/cranelift/src/libcretonne/ir/immediates.rs index 077ea4205f..99276bfd66 100644 --- a/cranelift/src/libcretonne/ir/immediates.rs +++ b/cranelift/src/libcretonne/ir/immediates.rs @@ -11,16 +11,20 @@ use std::str::FromStr; /// 64-bit immediate integer operand. /// +/// An `Imm64` operand can also be used to represent immediate values of smaller integer types by +/// sign-extending to `i64`. #[derive(Copy, Clone, PartialEq, Eq, Debug)] pub struct Imm64(i64); impl Imm64 { - pub fn from_bits(x: u64) -> Imm64 { - Imm64(x as i64) + pub fn new(x: i64) -> Imm64 { + Imm64(x) } +} - pub fn to_bits(&self) -> u64 { - self.0 as u64 +impl Into for Imm64 { + fn into(self) -> i64 { + self.0 } } @@ -115,7 +119,7 @@ impl FromStr for Imm64 { return Err("Negative number too small for Imm64"); } } - Ok(Imm64::from_bits(value)) + Ok(Imm64::new(value as i64)) } } diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 87c336906a..fa6a445158 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -596,8 +596,11 @@ impl<'a> Parser<'a> { try!(self.match_identifier("stack_slot", "expected 'stack_slot'")); // stack-slot-decl ::= StackSlot(ss) "=" "stack_slot" * Bytes {"," stack-slot-flag} - let bytes = try!(self.match_imm64("expected byte-size in stack_slot decl")).to_bits(); - if bytes > u32::MAX as u64 { + let bytes: i64 = try!(self.match_imm64("expected byte-size in stack_slot decl")).into(); + if bytes < 0 { + return err!(self.loc, "negative stack slot size"); + } + if bytes > u32::MAX as i64 { return err!(self.loc, "stack slot too large"); } let data = StackSlotData::new(bytes as u32); From 1a92876989f382ee545612b2c07de97b37fad525 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 24 Aug 2016 15:15:37 -0700 Subject: [PATCH 0224/3084] Add module with commonly used immediate predicates. --- cranelift/src/libcretonne/lib.rs | 1 + cranelift/src/libcretonne/predicates.rs | 66 +++++++++++++++++++++++++ 2 files changed, 67 insertions(+) create mode 100644 cranelift/src/libcretonne/predicates.rs diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 7bdda395ab..fcddbdb900 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -16,6 +16,7 @@ pub mod entity_map; pub mod settings; mod constant_hash; +mod predicates; #[cfg(test)] pub mod test_utils; diff --git a/cranelift/src/libcretonne/predicates.rs b/cranelift/src/libcretonne/predicates.rs new file mode 100644 index 0000000000..83676be5a0 --- /dev/null +++ b/cranelift/src/libcretonne/predicates.rs @@ -0,0 +1,66 @@ +//! Predicate functions for testing instruction fields. +//! +//! This module defines functions that are used by the instruction predicates defined by +//! `meta/cretonne/predicates.py` classes. +//! +//! The predicates the operate on integer fields use `Into` as a shared trait bound. This +//! bound is implemented by all the native integer types as well as `Imm64`. +//! +//! Some of these predicates may be unused in certain ISA configurations, so we suppress the +//! dead_code warning. + +/// Check that `x` can be represented as a `wd`-bit signed integer with `sc` low zero bits. +#[allow(dead_code)] +pub fn is_signed_int>(x: T, wd: u8, sc: u8) -> bool { + let s = x.into(); + s == (s >> sc << (64 - wd + sc) >> (64 - wd)) +} + +/// Check that `x` can be represented as a `wd`-bit unsigned integer with `sc` low zero bits. +#[allow(dead_code)] +pub fn is_unsigned_int>(x: T, wd: u8, sc: u8) -> bool { + let u = x.into() as u64; + // Bitmask of the permitted bits. + let m = (1 << wd) - (1 << sc); + u == (u & m) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn cvt_u32() { + let x1 = 0u32; + let x2 = 1u32; + let x3 = 0xffff_fff0u32; + + assert!(is_signed_int(x1, 1, 0)); + assert!(is_signed_int(x1, 2, 1)); + assert!(is_signed_int(x2, 2, 0)); + assert!(!is_signed_int(x2, 2, 1)); + + // u32 doesn't sign-extend when converted to i64. + assert!(!is_signed_int(x3, 8, 0)); + + assert!(is_unsigned_int(x1, 1, 0)); + assert!(is_unsigned_int(x1, 8, 4)); + assert!(is_unsigned_int(x2, 1, 0)); + assert!(!is_unsigned_int(x2, 8, 4)); + assert!(!is_unsigned_int(x3, 1, 0)); + assert!(is_unsigned_int(x3, 32, 4)); + } + + #[test] + fn cvt_imm64() { + use ir::immediates::Imm64; + + let x1 = Imm64::new(-8); + let x2 = Imm64::new(8); + + assert!(is_signed_int(x1, 16, 2)); + assert!(is_signed_int(x2, 16, 2)); + assert!(!is_signed_int(x1, 16, 4)); + assert!(!is_signed_int(x2, 16, 4)); + } +} From eb03abe8647abef5f3fafacd9f8dd8cc35f538b9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 24 Aug 2016 16:02:41 -0700 Subject: [PATCH 0225/3084] Allow predicates on both EncRecipe and Encoding. If both specify a predicate, combine them with 'And'. --- meta/cretonne/__init__.py | 8 +++++++- meta/cretonne/predicates.py | 16 ++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index a5e842dd8e..d265009005 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -8,6 +8,7 @@ from __future__ import absolute_import import re import importlib from collections import namedtuple +from .predicates import And camel_re = re.compile('(^|_)([a-z])') @@ -1010,9 +1011,11 @@ class Encoding(object): being encoded. :param recipe: The :py:class:`EncRecipe` to use. :param encbits: Additional encoding bits to be interpreted by `recipe`. + :param instp: Instruction predicate, or `None`. + :param isap: ISA predicate, or `None`. """ - def __init__(self, cpumode, inst, recipe, encbits): + def __init__(self, cpumode, inst, recipe, encbits, instp=None, isap=None): assert isinstance(cpumode, CPUMode) assert isinstance(recipe, EncRecipe) self.inst, self.typevars = inst.fully_bound() @@ -1022,6 +1025,9 @@ class Encoding(object): self.inst.format, recipe.format)) self.recipe = recipe self.encbits = encbits + # Combine recipe predicates with the manually specified ones. + self.instp = And.combine(recipe.instp, instp) + self.isap = And.combine(recipe.isap, instp) @staticmethod def _to_type_tuple(x): diff --git a/meta/cretonne/predicates.py b/meta/cretonne/predicates.py index 71a17eb01b..d6cb8f8b96 100644 --- a/meta/cretonne/predicates.py +++ b/meta/cretonne/predicates.py @@ -95,6 +95,22 @@ class And(Predicate): s = '({})'.format(s) return s + @staticmethod + def combine(*args): + """ + Combine a sequence of predicates, allowing for `None` members. + + Return a predicate that is true when all non-`None` arguments are true, + or `None` if all of the arguments are `None`. + """ + args = tuple(p for p in args if p) + if args == (): + return None + if len(args) == 1: + return args[0] + # We have multiple predicate args. Combine with `And`. + return And(*args) + class Or(Predicate): """ From 33235b398190c26829eb70d28322a73291335536 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 24 Aug 2016 16:29:54 -0700 Subject: [PATCH 0226/3084] Pass arguments on to rustfmt. This allows the usage: src/format-all.sh --write-mode=diff --- cranelift/src/format-all.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/src/format-all.sh b/cranelift/src/format-all.sh index 3569c460be..7d51b7d343 100755 --- a/cranelift/src/format-all.sh +++ b/cranelift/src/format-all.sh @@ -10,5 +10,5 @@ src=$(pwd) for crate in $(find "$src" -name Cargo.toml); do cd $(dirname "$crate") - cargo fmt + cargo fmt -- "$@" done From 46dec8a11d5750cafc9271068f681b0d16b3f470 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 24 Aug 2016 16:36:52 -0700 Subject: [PATCH 0227/3084] Verify Rust source code formatting as part of the unit tests. Only do this is rustfmt is installed, which likely means don't run on Travis CI. --- cranelift/test-all.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 4b65c9b56f..e4914b9541 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -21,6 +21,12 @@ function banner() { echo "====== $@ ======" } +# Run rustfmt if we have it. (Travis probably won't). +if cargo install --list | grep -q '^rustfmt '; then + banner "Rust formatting" + $topdir/src/format-all.sh --write-mode=diff +fi + PKGS="cretonne cretonne-reader cretonne-tools" cd "$topdir/src/tools" for PKG in $PKGS From a24daf5dede79e1c98d1f508c8dc22e73f3d89dd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 25 Aug 2016 09:50:23 -0700 Subject: [PATCH 0228/3084] Add a predicate_leafs() method. This collects all of the leaf predicates that go into a compound predicate. Current leaf predicates are: - Settings for ISA predicates, and - FieldPredicates for instruction predicates. --- meta/cretonne/__init__.py | 3 +++ meta/cretonne/predicates.py | 10 ++++++++++ 2 files changed, 13 insertions(+) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index d265009005..a286c1fb39 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -41,6 +41,9 @@ class Setting(object): """ return self.group + def predicate_leafs(self, leafs): + leafs.add(self) + class BoolSetting(Setting): """ diff --git a/meta/cretonne/predicates.py b/meta/cretonne/predicates.py index d6cb8f8b96..cd2ef242fe 100644 --- a/meta/cretonne/predicates.py +++ b/meta/cretonne/predicates.py @@ -68,6 +68,13 @@ class Predicate(object): def predicate_context(self): return self.context + def predicate_leafs(self, leafs): + """ + Collect all leaf predicates into the `leafs` set. + """ + for part in self.parts: + part.predicate_leafs(leafs) + class And(Predicate): """ @@ -164,6 +171,9 @@ class FieldPredicate(object): """ return self.field.format + def predicate_leafs(self, leafs): + leafs.add(self) + def rust_predicate(self, prec): """ Return a string of Rust code that evaluates this predicate. From fb7997aa38d28d779604682a08628e96a5349696 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 25 Aug 2016 11:36:30 -0700 Subject: [PATCH 0229/3084] Collect list of CPU modes in TargetISA. Fix a typo nearby. --- meta/cretonne/__init__.py | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index a286c1fb39..8dd0535b24 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -949,10 +949,11 @@ class TargetISA(object): relevant for this ISA. """ - def __init__(self, name, instrution_groups): + def __init__(self, name, instruction_groups): self.name = name self.settings = None - self.instruction_groups = instrution_groups + self.instruction_groups = instruction_groups + self.cpumodes = list() class CPUMode(object): @@ -970,6 +971,7 @@ class CPUMode(object): self.name = name self.isa = isa self.encodings = [] + isa.cpumodes.append(self) def enc(self, *args, **kwargs): """ @@ -1032,17 +1034,15 @@ class Encoding(object): self.instp = And.combine(recipe.instp, instp) self.isap = And.combine(recipe.isap, instp) - @staticmethod - def _to_type_tuple(x): - # Allow a single ValueType instance instead of the awkward singleton - # tuple syntax. - if isinstance(x, ValueType): - x = (x,) + def ctrl_typevar(self): + """ + Get the controlling type variable for this encoding or `None`. + """ + if self.typevars: + return self.typevars[0] else: - x = tuple(x) - for ty in x: - assert isinstance(ty, ValueType) - return x + return None + # Import the fixed instruction formats now so they can be added to the # registry. From 8641076369d8e43ed767fd0296d22b2845adc486 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 25 Aug 2016 11:55:57 -0700 Subject: [PATCH 0230/3084] Fix Python3 compat in docs directory. Update copyright. --- cranelift/docs/conf.py | 177 +--------------------------------- cranelift/docs/cton_domain.py | 4 +- cranelift/docs/cton_lexer.py | 21 ++-- 3 files changed, 23 insertions(+), 179 deletions(-) diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py index c4ab93eff0..5602561828 100644 --- a/cranelift/docs/conf.py +++ b/cranelift/docs/conf.py @@ -12,9 +12,10 @@ # All configuration values have a default; values that are commented out # serve to show the default. +from __future__ import absolute_import import sys import os -import shlex + # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the @@ -27,9 +28,6 @@ sys.path.insert(0, os.path.abspath('../meta')) # -- General configuration ------------------------------------------------ -# If your documentation needs a minimal Sphinx version, state it here. -#needs_sphinx = '1.0' - # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. @@ -52,16 +50,13 @@ templates_path = ['_templates'] # source_suffix = ['.rst', '.md'] source_suffix = '.rst' -# The encoding of source files. -#source_encoding = 'utf-8-sig' - # The master toctree document. master_doc = 'index' # General information about the project. project = u'cretonne' -copyright = u'2016, Jakob Stoklund Olesen' -author = u'Jakob Stoklund Olesen' +copyright = u'2016, Cretonne Developers' +author = u'Cretonne Developers' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the @@ -79,40 +74,13 @@ release = u'0.0' # Usually you set "language" from the command line for these cases. language = None -# There are two options for replacing |today|: either, you set today to some -# non-false value, then it is used: -#today = '' -# Else, today_fmt is used as the format for a strftime call. -#today_fmt = '%B %d, %Y' - # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = ['_build'] -# The reST default role (used for this markup: `text`) to use for all -# documents. -#default_role = None - -# If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -#add_module_names = True - -# If true, sectionauthor and moduleauthor directives will be shown in the -# output. They are ignored by default. -#show_authors = False - # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' -# A list of ignored prefixes for module index sorting. -#modindex_common_prefix = [] - -# If true, keep warnings as "system message" paragraphs in the built documents. -#keep_warnings = False - # If true, `todo` and `todoList` produce output, else they produce nothing. todo_include_todos = True @@ -123,112 +91,12 @@ todo_include_todos = True # a list of builtin themes. html_theme = 'sphinx_rtd_theme' -# Theme options are theme-specific and customize the look and feel of a theme -# further. For a list of options available for each theme, see the -# documentation. -#html_theme_options = {} - -# Add any paths that contain custom themes here, relative to this directory. -#html_theme_path = [] - -# The name for this set of Sphinx documents. If None, it defaults to -# " v documentation". -#html_title = None - -# A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None - -# The name of an image file (relative to this directory) to place at the top -# of the sidebar. -#html_logo = None - -# The name of an image file (within the static path) to use as favicon of the -# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 -# pixels large. -#html_favicon = None - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] - -# Add any extra paths that contain custom files (such as robots.txt or -# .htaccess) here, relative to this directory. These files are copied -# directly to the root of the documentation. -#html_extra_path = [] - -# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, -# using the given strftime format. -#html_last_updated_fmt = '%b %d, %Y' - -# If true, SmartyPants will be used to convert quotes and dashes to -# typographically correct entities. -#html_use_smartypants = True - -# Custom sidebar templates, maps document names to template names. -#html_sidebars = {} - -# Additional templates that should be rendered to pages, maps page names to -# template names. -#html_additional_pages = {} - -# If false, no module index is generated. -#html_domain_indices = True - -# If false, no index is generated. -#html_use_index = True - -# If true, the index is split into individual pages for each letter. -#html_split_index = False - -# If true, links to the reST sources are added to the pages. -#html_show_sourcelink = True - -# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. -#html_show_sphinx = True - -# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. -#html_show_copyright = True - -# If true, an OpenSearch description file will be output, and all pages will -# contain a tag referring to it. The value of this option must be the -# base URL from which the finished HTML is served. -#html_use_opensearch = '' - -# This is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = None - -# Language to be used for generating the HTML full-text search index. -# Sphinx supports the following languages: -# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' -# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' -#html_search_language = 'en' - -# A dictionary with options for the search language support, empty by default. -# Now only 'ja' uses this config value -#html_search_options = {'type': 'default'} - -# The name of a javascript file (relative to the configuration directory) that -# implements a search results scorer. If empty, the default will be used. -#html_search_scorer = 'scorer.js' - # Output file base name for HTML help builder. htmlhelp_basename = 'cretonnedoc' # -- Options for LaTeX output --------------------------------------------- latex_elements = { -# The paper size ('letterpaper' or 'a4paper'). -#'papersize': 'letterpaper', - -# The font size ('10pt', '11pt' or '12pt'). -#'pointsize': '10pt', - -# Additional stuff for the LaTeX preamble. -#'preamble': '', - -# Latex figure (float) alignment -#'figure_align': 'htbp', } # Grouping the document tree into LaTeX files. List of tuples @@ -236,29 +104,9 @@ latex_elements = { # author, documentclass [howto, manual, or own class]). latex_documents = [ (master_doc, 'cretonne.tex', u'cretonne Documentation', - u'Jakob Stoklund Olesen', 'manual'), + author, 'manual'), ] -# The name of an image file (relative to this directory) to place at the top of -# the title page. -#latex_logo = None - -# For "manual" documents, if this is true, then toplevel headings are parts, -# not chapters. -#latex_use_parts = False - -# If true, show page references after internal links. -#latex_show_pagerefs = False - -# If true, show URL addresses after external links. -#latex_show_urls = False - -# Documents to append as an appendix to all manuals. -#latex_appendices = [] - -# If false, no module index is generated. -#latex_domain_indices = True - # -- Options for manual page output --------------------------------------- @@ -269,9 +117,6 @@ man_pages = [ [author], 1) ] -# If true, show URL addresses after external links. -#man_show_urls = False - # -- Options for Texinfo output ------------------------------------------- @@ -284,18 +129,6 @@ texinfo_documents = [ 'Miscellaneous'), ] -# Documents to append as an appendix to all manuals. -#texinfo_appendices = [] - -# If false, no module index is generated. -#texinfo_domain_indices = True - -# How to display URL addresses: 'footnote', 'no', or 'inline'. -#texinfo_show_urls = 'footnote' - -# If true, do not generate a @detailmenu in the "Top" node's menu. -#texinfo_no_detailmenu = False - # -- Options for Graphviz ------------------------------------------------- diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index aafb7724cf..ba762b4df3 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -9,6 +9,7 @@ # .. cton:inst:: v1, v2 = inst op1, op2 # Document an IR instruction. # +from __future__ import absolute_import import re @@ -133,7 +134,8 @@ class CtonInst(CtonObject): TypedField('result', label=l_('Results'), names=('out', 'result'), typerolename='type', typenames=('type',)), - GroupedField('typevar', names=('typevar',), label=l_('Type Variables')), + GroupedField( + 'typevar', names=('typevar',), label=l_('Type Variables')), GroupedField('flag', names=('flag',), label=l_('Flags')), Field('resulttype', label=l_('Result type'), has_arg=False, names=('rtype',)), diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py index 28a1be25b4..eaa856c444 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/cton_lexer.py @@ -1,13 +1,17 @@ # -*- coding: utf-8 -*- # # Pygments lexer for Cretonne. +from __future__ import absolute_import from pygments.lexer import RegexLexer, bygroups, words -from pygments.token import * +from pygments.token import Comment, String, Keyword, Whitespace, Number, Name +from pygments.token import Operator, Punctuation, Text + def keywords(*args): return words(args, prefix=r'\b', suffix=r'\b') + class CretonneLexer(RegexLexer): name = 'Cretonne' aliases = ['cton'] @@ -19,7 +23,8 @@ class CretonneLexer(RegexLexer): # Strings are in double quotes, support \xx escapes only. (r'"([^"\\]+|\\[0-9a-fA-F]{2})*"', String), # A naked function name following 'function' is also a string. - (r'\b(function)([ \t]+)(\w+)\b', bygroups(Keyword, Whitespace, String.Symbol)), + (r'\b(function)([ \t]+)(\w+)\b', + bygroups(Keyword, Whitespace, String.Symbol)), # Numbers. (r'[-+]?0[xX][0-9a-fA-F]+', Number.Hex), (r'[-+]?0[xX][0-9a-fA-F]*\.[0-9a-fA-F]*([pP]\d+)?', Number.Hex), @@ -28,7 +33,8 @@ class CretonneLexer(RegexLexer): # Reserved words. (keywords('function'), Keyword), # Known attributes. - (keywords('align', 'aligntrap', 'uext', 'sext', 'inreg'), Name.Attribute), + (keywords('align', 'aligntrap', 'uext', 'sext', 'inreg'), + Name.Attribute), # Well known value types. (r'\b(b\d+|i\d+|f32|f64)(x\d+)?\b', Keyword.Type), # v = value @@ -37,8 +43,10 @@ class CretonneLexer(RegexLexer): # ebb = extended basic block (r'(ebb)\d+', Name.Label), # Match instruction names in context. - (r'(=)( *)([a-z]\w*)', bygroups(Operator, Whitespace, Name.Function)), - (r'^( *)([a-z]\w*\b)(?! *[,=])', bygroups(Whitespace, Name.Function)), + (r'(=)( *)([a-z]\w*)', + bygroups(Operator, Whitespace, Name.Function)), + (r'^( *)([a-z]\w*\b)(?! *[,=])', + bygroups(Whitespace, Name.Function)), # Other names: results and arguments (r'[a-z]\w*', Name), (r'->|=|:', Operator), @@ -47,8 +55,9 @@ class CretonneLexer(RegexLexer): ] } + def setup(app): """Setup Sphinx extension.""" app.add_lexer('cton', CretonneLexer()) - return { 'version' : '0.1' } + return {'version': '0.1'} From c166279bb94465117a7e75bdd864513c65e116b5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 25 Aug 2016 14:24:02 -0700 Subject: [PATCH 0231/3084] Call function in the predicates module. When generating Rust code for an instruction predicate, call the corresponding function in the predicates module, using a qualified name. We don't have methods corresponding to the predicates. --- meta/cretonne/predicates.py | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/meta/cretonne/predicates.py b/meta/cretonne/predicates.py index cd2ef242fe..f99f4cfc5f 100644 --- a/meta/cretonne/predicates.py +++ b/meta/cretonne/predicates.py @@ -155,13 +155,13 @@ class FieldPredicate(object): An instruction predicate that performs a test on a single `FormatField`. :param field: The `FormatField` to be tested. - :param method: Boolean predicate method to call. - :param args: Arguments for the predicate method. + :param function: Boolean predicate function to call. + :param args: Additional arguments for the predicate function. """ - def __init__(self, field, method, args): + def __init__(self, field, function, args): self.field = field - self.method = method + self.function = function self.args = args def predicate_context(self): @@ -178,10 +178,9 @@ class FieldPredicate(object): """ Return a string of Rust code that evaluates this predicate. """ - return '{}.{}({})'.format( - self.field.rust_name(), - self.method, - ', '.join(self.args)) + # Prepend `field` to the predicate function arguments. + args = (self.field.rust_name(),) + tuple(map(str, self.args)) + return 'predicates::{}({})'.format(self.function, ', '.join(args)) class IsSignedInt(FieldPredicate): From 0c6e0e9cb79d3a08fc5dd0909bf5d230f8193422 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 18 Aug 2016 11:47:01 -0700 Subject: [PATCH 0232/3084] Generate encoding tables. (WIP). Amend build script to generate an encodings-.rs file for each target ISA. Emit a function that can evaluate instruction predicates. Describe the 3-level tables used for representing insrruction encoding tables. Add Python classes representing the tables. The generated code is incomplete and not used anywhere yet. --- meta/build.py | 2 + meta/gen_encoding.py | 212 +++++++++++++++++++++++++++++++++++++++++++ meta/srcgen.py | 11 ++- 3 files changed, 224 insertions(+), 1 deletion(-) create mode 100644 meta/gen_encoding.py diff --git a/meta/build.py b/meta/build.py index 2806426e6d..3ccbd49c0f 100644 --- a/meta/build.py +++ b/meta/build.py @@ -8,6 +8,7 @@ import isa import gen_instr import gen_settings import gen_build_deps +import gen_encoding parser = argparse.ArgumentParser(description='Generate sources for Cretonne.') parser.add_argument('--out-dir', help='set output directory') @@ -19,4 +20,5 @@ isas = isa.all_isas() gen_instr.generate(isas, out_dir) gen_settings.generate(isas, out_dir) +gen_encoding.generate(isas, out_dir) gen_build_deps.generate() diff --git a/meta/gen_encoding.py b/meta/gen_encoding.py new file mode 100644 index 0000000000..c1e618e046 --- /dev/null +++ b/meta/gen_encoding.py @@ -0,0 +1,212 @@ +""" +Generate sources for instruction encoding. + +The tables and functions generated here support the `TargetIsa::encode()` +function which determines if a given instruction is legal, and if so, it's +`Encoding` data which consists of a *recipe* and some *encoding* bits. + +The `encode` function doesn't actually generate the binary machine bits. Each +recipe has a corresponding hand-written function to do that after registers +are allocated. + +This is the information available to us: + +- The instruction to be encoded as an `Inst` reference. +- The data-flow graph containing the instruction, giving us access to the + `InstructionData` representation and the types of all values involved. +- A target ISA instance with shared and ISA-specific settings for evaluating + ISA predicates. +- The currently active CPU mode is determined by the ISA. + +## Level 1 table lookup + +The CPU mode provides the first table. The key is the instruction's controlling +type variable. If the instruction is not polymorphic, use `VOID` for the type +variable. The table values are level 2 tables. + +## Level 2 table lookup + +The level 2 table is keyed by the instruction's opcode. The table values are +*encoding lists*. + +The two-level table lookup allows the level 2 tables to be much smaller with +good locality. Code in any given function usually only uses a few different +types, so many of the level 2 tables will be cold. + +## Encoding lists + +An encoding list is a non-empty sequence of list entries. Each entry has +one of these forms: + +1. Instruction predicate, encoding recipe, and encoding bits. If the + instruction predicate is true, use this recipe and bits. +2. ISA predicate and skip-count. If the ISA predicate is false, skip the next + *skip-count* entries in the list. If the skip count is zero, stop + completely. +3. Stop. End of list marker. If this is reached, the instruction does not have + a legal encoding. + +The instruction predicate is also used to distinguish between polymorphic +instructions with different types for secondary type variables. +""" +from __future__ import absolute_import +import srcgen +from collections import OrderedDict + + +def emit_instp(instp, fmt): + """ + Emit code for matching an instruction predicate against an + `InstructionData` reference called `inst`. + + The generated code is a pattern match that falls through if the instruction + has an unexpected format. This should lead to a panic. + """ + iform = instp.predicate_context() + + # Which fiels do we need in the InstructionData pattern match? + if iform.boxed_storage: + fields = 'ref data' + else: + # Collect the leaf predicates + leafs = set() + instp.predicate_leafs(leafs) + # All the leafs are FieldPredicate instances. Here we just care about + # the field names. + fields = ', '.join(sorted(set(p.field.name for p in leafs))) + + with fmt.indented( + 'if let {} {{ {}, .. }} = *inst {{' + .format(iform.name, fields), '}'): + fmt.line('return {};'.format(instp.rust_predicate(0))) + + +def emit_instps(instps, fmt): + """ + Emit a function for matching instruction predicates. + """ + + with fmt.indented( + 'fn check_instp(inst: &InstructionData, instp_idx: u16) -> bool {', + '}'): + with fmt.indented('match instp_idx {', '}'): + for (instp, idx) in instps.items(): + with fmt.indented('{} => {{'.format(idx), '}'): + emit_instp(instp, fmt) + fmt.line('_ => panic!("Invalid instruction predicate")') + + # The match cases will fall through if the instruction format is wrong. + fmt.line('panic!("Bad format {}/{} for instp {}",') + fmt.line(' InstructionFormat::from(inst),') + fmt.line(' inst.opcode(),') + fmt.line(' instp_idx);') + + +def collect_instps(cpumodes): + # Map instp -> number + instps = OrderedDict() + for cpumode in cpumodes: + for enc in cpumode.encodings: + instp = enc.instp + if instp and instp not in instps: + instps[instp] = 1 + len(instps) + return instps + + +class EncList(object): + """ + List of instructions for encoding a given type + opcode pair. + + An encoding list contains a sequence of predicates and encoding recipes, + all encoded as u16 values. + + :param inst: The instruction opcode being encoded. + :param ty: Value of the controlling type variable, or `None`. + """ + + def __init__(self, inst, ty): + self.inst = inst + self.ty = ty + # List of applicable Encoding instances. + # These will have different predicates. + self.encodings = [] + + def name(self): + if self.ty: + return '{}.{}'.format(self.inst.name, self.ty.name) + else: + return self.inst.name + + +class Level2Table(object): + """ + Level 2 table mapping instruction opcodes to `EncList` objects. + + :param ty: Controlling type variable of all entries, or `None`. + """ + + def __init__(self, ty): + self.ty = ty + # Maps inst -> EncList + self.lists = OrderedDict() + + def __getitem__(self, inst): + ls = self.lists.get(inst) + if not ls: + ls = EncList(inst, self.ty) + self.lists[inst] = ls + return ls + + def __iter__(self): + return iter(self.lists.values()) + + +class Level1Table(object): + """ + Level 1 table mapping types to `Level2` objects. + """ + + def __init__(self): + self.tables = OrderedDict() + + def __getitem__(self, ty): + tbl = self.tables.get(ty) + if not tbl: + tbl = Level2Table(ty) + self.tables[ty] = tbl + return tbl + + def __iter__(self): + return iter(self.tables.values()) + + +def make_tables(cpumode): + """ + Generate tables for `cpumode` as described above. + """ + table = Level1Table() + for enc in cpumode.encodings: + ty = enc.ctrl_typevar() + inst = enc.inst + table[ty][inst].encodings.append(enc) + return table + + +def gen_isa(cpumodes, fmt): + # First assign numbers to relevant instruction predicates and generate the + # check_instp() function.. + instps = collect_instps(cpumodes) + emit_instps(instps, fmt) + + for cpumode in cpumodes: + level1 = make_tables(cpumode) + for level2 in level1: + for enclist in level2: + fmt.comment(enclist.name()) + + +def generate(isas, out_dir): + for isa in isas: + fmt = srcgen.Formatter() + gen_isa(isa.cpumodes, fmt) + fmt.update_file('encoding-{}.rs'.format(isa.name), out_dir) diff --git a/meta/srcgen.py b/meta/srcgen.py index 99bb808435..0b8a7a2973 100644 --- a/meta/srcgen.py +++ b/meta/srcgen.py @@ -51,12 +51,21 @@ class Formatter(object): self.indent = self.indent[0:-self.shiftwidth] def line(self, s=None): - """And an indented line.""" + """Add an indented line.""" if s: self.lines.append('{}{}\n'.format(self.indent, s)) else: self.lines.append('\n') + def outdented_line(self, s): + """ + Emit a line outdented one level. + + This is used for '} else {' and similar things inside a single indented + block. + """ + self.lines.append('{}{}\n'.format(self.indent[0:-self.shiftwidth], s)) + def writelines(self, f=None): """Write all lines to `f`.""" if not f: From 7cb975ce63b74685ba584cf98e76b60b12a10138 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 26 Aug 2016 08:50:47 -0700 Subject: [PATCH 0233/3084] Add string conversions for predicates and encodings. This is just used for printing better comments in generated code. --- meta/cretonne/__init__.py | 6 ++++++ meta/cretonne/predicates.py | 12 ++++++++++++ meta/gen_encoding.py | 2 ++ 3 files changed, 20 insertions(+) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 8dd0535b24..88860286dc 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -1003,6 +1003,9 @@ class EncRecipe(object): if instp: assert instp.predicate_context() == format + def __str__(self): + return self.name + class Encoding(object): """ @@ -1034,6 +1037,9 @@ class Encoding(object): self.instp = And.combine(recipe.instp, instp) self.isap = And.combine(recipe.isap, instp) + def __str__(self): + return '[{}/{:02x}]'.format(self.recipe, self.encbits) + def ctrl_typevar(self): """ Get the controlling type variable for this encoding or `None`. diff --git a/meta/cretonne/predicates.py b/meta/cretonne/predicates.py index f99f4cfc5f..6722c820e5 100644 --- a/meta/cretonne/predicates.py +++ b/meta/cretonne/predicates.py @@ -65,6 +65,14 @@ class Predicate(object): (p.predicate_context() for p in parts)) assert self.context, "Incompatible predicate parts" + def __str__(self): + s = '{}({})'.format( + type(self).__name__, + ' ,'.join(map(str, self.parts))) + if self.name: + s = '{}={}'.format(self.name, s) + return s + def predicate_context(self): return self.context @@ -164,6 +172,10 @@ class FieldPredicate(object): self.function = function self.args = args + def __str__(self): + args = (self.field.name,) + tuple(map(str, self.args)) + return '{}({})'.format(self.function, ', '.join(args)) + def predicate_context(self): """ This predicate can be evaluated in the context of an instruction diff --git a/meta/gen_encoding.py b/meta/gen_encoding.py index c1e618e046..a7ac7e2f4a 100644 --- a/meta/gen_encoding.py +++ b/meta/gen_encoding.py @@ -203,6 +203,8 @@ def gen_isa(cpumodes, fmt): for level2 in level1: for enclist in level2: fmt.comment(enclist.name()) + for enc in enclist.encodings: + fmt.comment('{} when {}'.format(enc, enc.instp)) def generate(isas, out_dir): From c1ae0c99ed0b09c7f31834774213e9ca455d997d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 26 Aug 2016 09:31:50 -0700 Subject: [PATCH 0234/3084] Move predicate collection into TargetISA. Add a TargetISA.finish() method which computes derived data structures after the ISA definitions have been loaded. --- meta/cretonne/__init__.py | 31 +++++++++++++++++++++++++++++++ meta/gen_encoding.py | 34 +++++++++++----------------------- meta/isa/riscv/__init__.py | 5 ++--- 3 files changed, 44 insertions(+), 26 deletions(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 88860286dc..bc20bcbcf3 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -955,6 +955,37 @@ class TargetISA(object): self.instruction_groups = instruction_groups self.cpumodes = list() + def finish(self): + """ + Finish the definition of a target ISA after adding all CPU modes and + settings. + + This computes some derived properties that are used in multilple + places. + + :returns self: + """ + self._collect_instruction_predicates() + return self + + def _collect_instruction_predicates(self): + """ + Collect and number all instruction predicates in use. + + Sets `instp.number` for all used instruction predicates and places them + in `self.all_instps` in numerical order. + """ + self.all_instps = list() + instps = set() + for cpumode in self.cpumodes: + for enc in cpumode.encodings: + instp = enc.instp + if instp and instp not in instps: + # assign predicate number starting from 0. + instp.number = len(instps) + instps.add(instp) + self.all_instps.append(instp) + class CPUMode(object): """ diff --git a/meta/gen_encoding.py b/meta/gen_encoding.py index a7ac7e2f4a..c10a4f8a06 100644 --- a/meta/gen_encoding.py +++ b/meta/gen_encoding.py @@ -75,10 +75,11 @@ def emit_instp(instp, fmt): # the field names. fields = ', '.join(sorted(set(p.field.name for p in leafs))) - with fmt.indented( - 'if let {} {{ {}, .. }} = *inst {{' - .format(iform.name, fields), '}'): - fmt.line('return {};'.format(instp.rust_predicate(0))) + with fmt.indented('{} => {{'.format(instp.number), '}'): + with fmt.indented( + 'if let {} {{ {}, .. }} = *inst {{' + .format(iform.name, fields), '}'): + fmt.line('return {};'.format(instp.rust_predicate(0))) def emit_instps(instps, fmt): @@ -90,9 +91,8 @@ def emit_instps(instps, fmt): 'fn check_instp(inst: &InstructionData, instp_idx: u16) -> bool {', '}'): with fmt.indented('match instp_idx {', '}'): - for (instp, idx) in instps.items(): - with fmt.indented('{} => {{'.format(idx), '}'): - emit_instp(instp, fmt) + for instp in instps: + emit_instp(instp, fmt) fmt.line('_ => panic!("Invalid instruction predicate")') # The match cases will fall through if the instruction format is wrong. @@ -102,17 +102,6 @@ def emit_instps(instps, fmt): fmt.line(' instp_idx);') -def collect_instps(cpumodes): - # Map instp -> number - instps = OrderedDict() - for cpumode in cpumodes: - for enc in cpumode.encodings: - instp = enc.instp - if instp and instp not in instps: - instps[instp] = 1 + len(instps) - return instps - - class EncList(object): """ List of instructions for encoding a given type + opcode pair. @@ -192,13 +181,12 @@ def make_tables(cpumode): return table -def gen_isa(cpumodes, fmt): +def gen_isa(isa, fmt): # First assign numbers to relevant instruction predicates and generate the # check_instp() function.. - instps = collect_instps(cpumodes) - emit_instps(instps, fmt) + emit_instps(isa.all_instps, fmt) - for cpumode in cpumodes: + for cpumode in isa.cpumodes: level1 = make_tables(cpumode) for level2 in level1: for enclist in level2: @@ -210,5 +198,5 @@ def gen_isa(cpumodes, fmt): def generate(isas, out_dir): for isa in isas: fmt = srcgen.Formatter() - gen_isa(isa.cpumodes, fmt) + gen_isa(isa, fmt) fmt.update_file('encoding-{}.rs'.format(isa.name), out_dir) diff --git a/meta/isa/riscv/__init__.py b/meta/isa/riscv/__init__.py index 1f9ebd3f46..e187a95c0e 100644 --- a/meta/isa/riscv/__init__.py +++ b/meta/isa/riscv/__init__.py @@ -26,8 +26,7 @@ RV32G / RV64G """ from __future__ import absolute_import from . import defs -from . import encodings -from . import settings +from . import encodings, settings # noqa # Re-export the primary target ISA definition. -isa = defs.isa +isa = defs.isa.finish() From 3e40c3e4546571a39d965cd95a8e01b0922928b5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 26 Aug 2016 09:39:36 -0700 Subject: [PATCH 0235/3084] Flake8 lints. --- meta/check-py3k.sh | 1 + meta/gen_instr.py | 6 ++++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/meta/check-py3k.sh b/meta/check-py3k.sh index 19096339bf..3ac939ea15 100755 --- a/meta/check-py3k.sh +++ b/meta/check-py3k.sh @@ -5,3 +5,4 @@ # Install pylint with 'pip install pylint'. cd $(dirname "$0") pylint --py3k --reports=no -- *.py cretonne isa +flake8 . diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 92808524c5..13bcaa6842 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -71,7 +71,8 @@ def gen_instruction_data_impl(fmt): .format(f.name)) fmt.doc_comment('Mutable reference to the type of the first result.') - with fmt.indented('pub fn first_type_mut(&mut self) -> &mut Type {', '}'): + with fmt.indented( + 'pub fn first_type_mut(&mut self) -> &mut Type {', '}'): with fmt.indented('match *self {', '}'): for f in cretonne.InstructionFormat.all_formats: fmt.line( @@ -152,7 +153,8 @@ def gen_instruction_data_impl(fmt): if f.boxed_storage: fmt.line( n + - ' {{ ref data, .. }} => Some(data.args[{}]),' + ' {{ ref data, .. }} => ' + + 'Some(data.args[{}]),' .format(i)) else: fmt.line( From ea3396c2ff3afe556b24d4574867ba59d20d2dce Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Fri, 26 Aug 2016 13:39:44 -0700 Subject: [PATCH 0236/3084] Add another dominator tree test case --- .../tests/dominator_tree_testdata/loops2.cton | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 cranelift/src/tools/tests/dominator_tree_testdata/loops2.cton diff --git a/cranelift/src/tools/tests/dominator_tree_testdata/loops2.cton b/cranelift/src/tools/tests/dominator_tree_testdata/loops2.cton new file mode 100644 index 0000000000..452641be01 --- /dev/null +++ b/cranelift/src/tools/tests/dominator_tree_testdata/loops2.cton @@ -0,0 +1,29 @@ +function test(i32) { + ebb0(v0: i32): ; dominates(0) + brz v0, ebb1 ; dominates(1,6) + brnz v0, ebb2 ; dominates(2,9) + jump ebb3 ; dominates(3) + ebb1: + jump ebb6 + ebb2: + brz v0, ebb4 ; dominates(4,7,8) + jump ebb5 ; dominates(5) + ebb3: + jump ebb9 + ebb4: + brz v0, ebb4 + brnz v0, ebb6 + jump ebb7 + ebb5: + brz v0, ebb7 + brnz v0, ebb8 + jump ebb9 + ebb6: + return + ebb7: + jump ebb8 + ebb8: + return + ebb9: + return +} From 15b5576d999e3eb91ddc574f31c90f5bc4f7ef7e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 26 Aug 2016 12:43:38 -0700 Subject: [PATCH 0237/3084] Generate type numbers at meta-time. We need to generate hash tables keyed by types, so the Python scripts need to know the index used to represent types in Rust code. To enforce this, add a new gen_types.py script which generates constant definitions for the ir/types module. Also generate constants for common SIMD vector sizes. --- cranelift/src/libcretonne/ir/types.rs | 44 +++++++---------------- meta/build.py | 2 ++ meta/cretonne/__init__.py | 19 +++++++--- meta/gen_types.py | 51 +++++++++++++++++++++++++++ 4 files changed, 80 insertions(+), 36 deletions(-) create mode 100644 meta/gen_types.py diff --git a/cranelift/src/libcretonne/ir/types.rs b/cranelift/src/libcretonne/ir/types.rs index 1bc0d0a0b4..12dfc8c6ec 100644 --- a/cranelift/src/libcretonne/ir/types.rs +++ b/cranelift/src/libcretonne/ir/types.rs @@ -31,38 +31,9 @@ pub struct Type(u8); /// a SIMD vector. pub const VOID: Type = Type(0); -/// Integer type with 8 bits. -pub const I8: Type = Type(1); - -/// Integer type with 16 bits. -pub const I16: Type = Type(2); - -/// Integer type with 32 bits. -pub const I32: Type = Type(3); - -/// Integer type with 64 bits. -pub const I64: Type = Type(4); - -/// IEEE single precision floating point type. -pub const F32: Type = Type(5); - -/// IEEE double precision floating point type. -pub const F64: Type = Type(6); - -/// Boolean type. Can't be loaded or stored, but can be used to form SIMD vectors. -pub const B1: Type = Type(7); - -/// Boolean type using 8 bits to represent true/false. -pub const B8: Type = Type(8); - -/// Boolean type using 16 bits to represent true/false. -pub const B16: Type = Type(9); - -/// Boolean type using 32 bits to represent true/false. -pub const B32: Type = Type(10); - -/// Boolean type using 64 bits to represent true/false. -pub const B64: Type = Type(11); +// Include code generated by `meta/gen_types.py`. This file contains constant definitions for all +// the scalar types as well as common vector types for 64, 128, 256, and 512-bit SID vectors. +include!(concat!(env!("OUT_DIR"), "/types.rs")); impl Type { /// Get the lane type of this SIMD vector type. @@ -183,6 +154,11 @@ impl Type { Some(Type(self.0 - 0x10)) } } + + /// Index of this type, for use with hash tables etc. + pub fn index(self) -> usize { + self.0 as usize + } } impl Display for Type { @@ -362,6 +338,10 @@ mod tests { assert_eq!(B1.by(2).unwrap().half_vector().unwrap().to_string(), "b1"); assert_eq!(I32.half_vector(), None); assert_eq!(VOID.half_vector(), None); + + // Check that the generated constants match the computed vector types. + assert_eq!(I32.by(4), Some(I32X4)); + assert_eq!(F64.by(8), Some(F64X8)); } #[test] diff --git a/meta/build.py b/meta/build.py index 3ccbd49c0f..730d1c9b3f 100644 --- a/meta/build.py +++ b/meta/build.py @@ -5,6 +5,7 @@ from __future__ import absolute_import import argparse import isa +import gen_types import gen_instr import gen_settings import gen_build_deps @@ -18,6 +19,7 @@ out_dir = args.out_dir isas = isa.all_isas() +gen_types.generate(out_dir) gen_instr.generate(isas, out_dir) gen_settings.generate(isas, out_dir) gen_encoding.generate(isas, out_dir) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index bc20bcbcf3..324dc76b4b 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -6,6 +6,7 @@ instructions. """ from __future__ import absolute_import import re +import math import importlib from collections import namedtuple from .predicates import And @@ -279,9 +280,12 @@ class ValueType(object): or one of its subclasses. """ - # map name -> ValueType. + # Map name -> ValueType. _registry = dict() + # List of all the scalar types. + all_scalars = list() + def __init__(self, name, membytes, doc): self.name = name self.membytes = membytes @@ -321,6 +325,10 @@ class ScalarType(ValueType): def __init__(self, name, membytes, doc): super(ScalarType, self).__init__(name, membytes, doc) self._vectors = dict() + # Assign numbers starting from 1. (0 is VOID). + ValueType.all_scalars.append(self) + self.number = len(ValueType.all_scalars) + assert self.number < 16, 'Too many scalar types' def __repr__(self): return 'ScalarType({})'.format(self.name) @@ -356,10 +364,11 @@ class VectorType(ValueType): name='{}x{}'.format(base.name, lanes), membytes=lanes*base.membytes, doc=""" - A SIMD vector with {} lanes containing a {} each. + A SIMD vector with {} lanes containing a `{}` each. """.format(lanes, base.name)) self.base = base self.lanes = lanes + self.number = 16*int(math.log(lanes, 2)) + base.number def __repr__(self): return ('VectorType(base={}, lanes={})' @@ -386,8 +395,10 @@ class FloatType(ScalarType): def __init__(self, bits, doc): assert bits > 0, 'FloatType must have positive number of bits' - super(FloatType, self).__init__(name='f{:d}'.format(bits), - membytes=bits // 8, doc=doc) + super(FloatType, self).__init__( + name='f{:d}'.format(bits), + membytes=bits // 8, + doc=doc) self.bits = bits def __repr__(self): diff --git a/meta/gen_types.py b/meta/gen_types.py new file mode 100644 index 0000000000..51607a61f9 --- /dev/null +++ b/meta/gen_types.py @@ -0,0 +1,51 @@ +""" +Generate sources with type info. + +This generates a `types.rs` file which is included in +`libcretonne/ir/types/rs`. The file provides constant definitions for the most +commonly used types, including all of the scalar types. + +This ensures that Python and Rust use the same type numbering. +""" +from __future__ import absolute_import +import srcgen +from cretonne import ValueType + + +def emit_type(ty, fmt): + """ + Emit a constant definition of a single value type. + """ + name = ty.name.upper() + fmt.doc_comment(ty.__doc__) + fmt.line( + 'pub const {}: Type = Type({:#x});' + .format(name, ty.number)) + + +def emit_vectors(bits, fmt): + """ + Emit definition for all vector types with `bits` total size. + """ + size = bits // 8 + for ty in ValueType.all_scalars: + mb = ty.membytes + if mb == 0 or mb >= size: + continue + emit_type(ty.by(size // mb), fmt) + + +def emit_types(fmt): + for ty in ValueType.all_scalars: + emit_type(ty, fmt) + # Emit vector definitions for common SIMD sizes. + emit_vectors(64, fmt) + emit_vectors(128, fmt) + emit_vectors(256, fmt) + emit_vectors(512, fmt) + + +def generate(out_dir): + fmt = srcgen.Formatter() + emit_types(fmt) + fmt.update_file('types.rs', out_dir) From 6a8a4c43c2bfdca57b3f6470e594cf8fa67cf886 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 26 Aug 2016 14:48:28 -0700 Subject: [PATCH 0238/3084] Collect and number all active encoding recipes. The recipes are shared across CPU modes. --- meta/cretonne/__init__.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 324dc76b4b..2e316982c0 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -976,9 +976,24 @@ class TargetISA(object): :returns self: """ + self._collect_encoding_recipes() self._collect_instruction_predicates() return self + def _collect_encoding_recipes(self): + """ + Collect and number all encoding recipes in use. + """ + self.all_recipes = list() + rcps = set() + for cpumode in self.cpumodes: + for enc in cpumode.encodings: + recipe = enc.recipe + if recipe not in rcps: + recipe.number = len(rcps) + rcps.add(recipe) + self.all_recipes.append(recipe) + def _collect_instruction_predicates(self): """ Collect and number all instruction predicates in use. From 0c8f251beebba7c497a18372f30e034635e6c5b5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 26 Aug 2016 14:44:03 -0700 Subject: [PATCH 0239/3084] Emit encoding lists (WIP). Compute the u16 representation of encoding lists and emit a big table concatenating all of them. Use the UniqueSeqTable to share some table space between CPU modes. --- meta/cretonne/__init__.py | 3 + meta/gen_encoding.py | 112 +++++++++++++++++++++++++++++++++++--- 2 files changed, 106 insertions(+), 9 deletions(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 2e316982c0..b44a238ffd 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -1030,6 +1030,9 @@ class CPUMode(object): self.encodings = [] isa.cpumodes.append(self) + def __str__(self): + return self.name + def enc(self, *args, **kwargs): """ Add a new encoding to this CPU mode. diff --git a/meta/gen_encoding.py b/meta/gen_encoding.py index c10a4f8a06..278ed66da5 100644 --- a/meta/gen_encoding.py +++ b/meta/gen_encoding.py @@ -51,7 +51,8 @@ instructions with different types for secondary type variables. """ from __future__ import absolute_import import srcgen -from collections import OrderedDict +from unique_table import UniqueSeqTable +from collections import OrderedDict, defaultdict def emit_instp(instp, fmt): @@ -102,6 +103,43 @@ def emit_instps(instps, fmt): fmt.line(' instp_idx);') +# Encoding lists are represented as u16 arrays. +CODE_BITS = 16 +PRED_BITS = 12 +PRED_MASK = (1 << PRED_BITS) - 1 + +# 0..CODE_ALWAYS means: Check instruction predicate and use the next two +# entries as a (recipe, encbits) pair if true. CODE_ALWAYS is the always-true +# predicate, smaller numbers refer to instruction predicates. +CODE_ALWAYS = PRED_MASK + +# Codes above CODE_ALWAYS indicate an ISA predicate to be tested. +# `x & PRED_MASK` is the ISA predicate number to test. +# `(x >> PRED_BITS)*3` is the number of u16 table entries to skip if the ISA +# predicate is false. (The factor of three corresponds to the (inst-pred, +# recipe, encbits) triples. +# +# Finally, CODE_FAIL indicates the end of the list. +CODE_FAIL = (1 << CODE_BITS) - 1 + + +def seq_doc(enc): + """ + Return a tuple containing u16 representations of the instruction predicate + an recipe / encbits. + + Also return a doc string. + """ + if enc.instp: + p = enc.instp.number + doc = '--> {} when {}'.format(enc, enc.instp) + else: + p = CODE_ALWAYS + doc = '--> {}'.format(enc) + assert p <= CODE_ALWAYS + return ((p, enc.recipe.number, enc.encbits), doc) + + class EncList(object): """ List of instructions for encoding a given type + opcode pair. @@ -121,10 +159,38 @@ class EncList(object): self.encodings = [] def name(self): + name = self.inst.name if self.ty: - return '{}.{}'.format(self.inst.name, self.ty.name) - else: - return self.inst.name + name = '{}.{}'.format(name, self.ty.name) + if self.encodings: + name += ' ({})'.format(self.encodings[0].cpumode) + return name + + def encode(self, seq_table, doc_table): + """ + Encode this list as a sequence of u16 numbers. + + Adds the sequence to `seq_table` and records the returned offset as + `self.offset`. + + Adds comment lines to `doc_table` keyed by seq_table offsets. + """ + words = list() + docs = list() + + for idx, enc in enumerate(self.encodings): + seq, doc = seq_doc(enc) + docs.append((len(words), doc)) + words.extend(seq) + words.append(CODE_FAIL) + + self.offset = seq_table.add(words) + + # Add doc comments. + doc_table[self.offset].append( + '{:06x}: {}'.format(self.offset, self.name())) + for pos, doc in docs: + doc_table[self.offset + pos].append(doc) class Level2Table(object): @@ -181,18 +247,46 @@ def make_tables(cpumode): return table +def encode_enclists(level1, seq_table, doc_table): + """ + Compute encodings and doc comments for encoding lists in `level1`. + """ + for level2 in level1: + for enclist in level2: + enclist.encode(seq_table, doc_table) + + +def emit_enclists(seq_table, doc_table, fmt): + with fmt.indented( + 'const ENCLISTS: [u16; {}] = ['.format(len(seq_table.table)), + '];'): + line = '' + for idx, entry in enumerate(seq_table.table): + if idx in doc_table: + if line: + fmt.line(line) + line = '' + for doc in doc_table[idx]: + fmt.comment(doc) + line += '{:#06x}, '.format(entry) + if line: + fmt.line(line) + + def gen_isa(isa, fmt): # First assign numbers to relevant instruction predicates and generate the # check_instp() function.. emit_instps(isa.all_instps, fmt) + # Tables for enclists with comments. + seq_table = UniqueSeqTable() + doc_table = defaultdict(list) + for cpumode in isa.cpumodes: level1 = make_tables(cpumode) - for level2 in level1: - for enclist in level2: - fmt.comment(enclist.name()) - for enc in enclist.encodings: - fmt.comment('{} when {}'.format(enc, enc.instp)) + encode_enclists(level1, seq_table, doc_table) + + emit_enclists(seq_table, doc_table, fmt) def generate(isas, out_dir): From a21935a5a24ef1c93361818dddae99a04de93189 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 26 Aug 2016 16:13:22 -0700 Subject: [PATCH 0240/3084] Add 32-bit ops to RV64. The 32-bit arithmetic instructions are encoded differently in the RISC-V 64-bit mode. --- meta/isa/riscv/encodings.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/meta/isa/riscv/encodings.py b/meta/isa/riscv/encodings.py index 099b6b5630..c339859730 100644 --- a/meta/isa/riscv/encodings.py +++ b/meta/isa/riscv/encodings.py @@ -22,6 +22,9 @@ for inst, inst_imm, f3, f7 in [ RV32.enc(inst_imm.i32, I, OPIMM(f3)) RV64.enc(inst_imm.i64, I, OPIMM(f3)) +# 32-bit ops in RV64. +RV64.enc(base.iadd.i32, R, OP32(0b000, 0b0000000)) +RV64.enc(base.isub.i32, R, OP32(0b000, 0b0100000)) # There are no andiw/oriw/xoriw variations. RV64.enc(base.iadd_imm.i32, I, OPIMM32(0b000)) From adde184042e03e7d92f56c52d339fea89a7f946d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 26 Aug 2016 17:15:05 -0700 Subject: [PATCH 0241/3084] Generate level 2 hashtables. All of the level 2 hashtables are concatenated into one constant array per ISA. --- meta/gen_encoding.py | 43 +++++++++++++++++++++++++++++++++++++++++++ meta/gen_instr.py | 1 + 2 files changed, 44 insertions(+) diff --git a/meta/gen_encoding.py b/meta/gen_encoding.py index 278ed66da5..c9e0695443 100644 --- a/meta/gen_encoding.py +++ b/meta/gen_encoding.py @@ -51,6 +51,7 @@ instructions with different types for secondary type variables. """ from __future__ import absolute_import import srcgen +from constant_hash import compute_quadratic from unique_table import UniqueSeqTable from collections import OrderedDict, defaultdict @@ -215,6 +216,21 @@ class Level2Table(object): def __iter__(self): return iter(self.lists.values()) + def layout_hashtable(self, level2_hashtables): + """ + Compute the hash table mapping opcode -> enclist. + + Append the hash table to `level2_hashtables` and record the offset. + """ + hash_table = compute_quadratic( + self.lists.values(), + lambda enclist: enclist.inst.number) + + self.hash_table_offset = len(level2_hashtables) + self.hash_table_len = len(hash_table) + + level2_hashtables.extend(hash_table) + class Level1Table(object): """ @@ -273,6 +289,28 @@ def emit_enclists(seq_table, doc_table, fmt): fmt.line(line) +def encode_level2_hashtables(level1, level2_hashtables): + for level2 in level1: + level2.layout_hashtable(level2_hashtables) + + +def emit_level2_hashtables(level2_hashtables, fmt): + """ + Emit the big concatenation of level 2 hash tables. + """ + with fmt.indented( + 'const LEVEL2: [(Opcode, u32); {}] = [' + .format(len(level2_hashtables)), + '];'): + for entry in level2_hashtables: + if entry: + fmt.line( + '(Opcode::{}, {:#08x}),' + .format(entry.inst.camel_name, entry.offset)) + else: + fmt.line('(Opcode::NotAnOpcode, 0),') + + def gen_isa(isa, fmt): # First assign numbers to relevant instruction predicates and generate the # check_instp() function.. @@ -282,11 +320,16 @@ def gen_isa(isa, fmt): seq_table = UniqueSeqTable() doc_table = defaultdict(list) + # Single table containing all the level2 hash tables. + level2_hashtables = list() + for cpumode in isa.cpumodes: level1 = make_tables(cpumode) encode_enclists(level1, seq_table, doc_table) + encode_level2_hashtables(level1, level2_hashtables) emit_enclists(seq_table, doc_table, fmt) + emit_level2_hashtables(level2_hashtables, fmt) def generate(isas, out_dir): diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 13bcaa6842..139acf59b7 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -191,6 +191,7 @@ def gen_opcodes(groups, fmt): for g in groups: for i in g.instructions: instrs.append(i) + i.number = len(instrs) # Build a doc comment. prefix = ', '.join(o.name for o in i.outs) if prefix: From 26cd358bd46cf57eee6e74f7aa8b39aac08c7708 Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Mon, 29 Aug 2016 13:17:08 -0700 Subject: [PATCH 0242/3084] Rustfmt fixes --- cranelift/src/libcretonne/dominator_tree.rs | 6 +++++- cranelift/src/libcretonne/ir/entities.rs | 6 +++++- cranelift/src/libcretonne/ir/immediates.rs | 6 +++++- cranelift/src/libcretonne/ir/instructions.rs | 5 ++++- cranelift/src/libcretonne/settings.rs | 4 +++- 5 files changed, 22 insertions(+), 5 deletions(-) diff --git a/cranelift/src/libcretonne/dominator_tree.rs b/cranelift/src/libcretonne/dominator_tree.rs index a3a00b2cc4..1b99db342d 100644 --- a/cranelift/src/libcretonne/dominator_tree.rs +++ b/cranelift/src/libcretonne/dominator_tree.rs @@ -108,7 +108,11 @@ impl DominatorTree { // TODO: we can't rely on instruction numbers to always be ordered // from lowest to highest. Given that, it will be necessary to create // an abolute mapping to determine the instruction order in the future. - if a.1 == NO_INST || a.1 < b.1 { a } else { b } + if a.1 == NO_INST || a.1 < b.1 { + a + } else { + b + } } /// Returns the immediate dominator of some ebb or None if the diff --git a/cranelift/src/libcretonne/ir/entities.rs b/cranelift/src/libcretonne/ir/entities.rs index 1a887452a7..33b41e5273 100644 --- a/cranelift/src/libcretonne/ir/entities.rs +++ b/cranelift/src/libcretonne/ir/entities.rs @@ -42,7 +42,11 @@ impl EntityRef for Ebb { impl Ebb { /// Create a new EBB reference from its number. This corresponds to the ebbNN representation. pub fn with_number(n: u32) -> Option { - if n < u32::MAX { Some(Ebb(n)) } else { None } + if n < u32::MAX { + Some(Ebb(n)) + } else { + None + } } } diff --git a/cranelift/src/libcretonne/ir/immediates.rs b/cranelift/src/libcretonne/ir/immediates.rs index 99276bfd66..87433606d7 100644 --- a/cranelift/src/libcretonne/ir/immediates.rs +++ b/cranelift/src/libcretonne/ir/immediates.rs @@ -60,7 +60,11 @@ impl FromStr for Imm64 { let mut value: u64 = 0; let mut digits = 0; let negative = s.starts_with('-'); - let s2 = if negative { &s[1..] } else { s }; + let s2 = if negative { + &s[1..] + } else { + s + }; if s2.starts_with("0x") { // Hexadecimal. diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index 9638d660a5..ca43a3dc31 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -95,7 +95,10 @@ impl FromStr for Opcode { /// `Box` to store the additional information out of line. #[derive(Debug)] pub enum InstructionData { - Nullary { opcode: Opcode, ty: Type }, + Nullary { + opcode: Opcode, + ty: Type, + }, Unary { opcode: Opcode, ty: Type, diff --git a/cranelift/src/libcretonne/settings.rs b/cranelift/src/libcretonne/settings.rs index 58731c40bf..b3cf494954 100644 --- a/cranelift/src/libcretonne/settings.rs +++ b/cranelift/src/libcretonne/settings.rs @@ -227,7 +227,9 @@ pub mod detail { #[derive(Clone, Copy)] pub enum Detail { /// A boolean setting only uses one bit, numbered from LSB. - Bool { bit: u8 }, + Bool { + bit: u8, + }, /// A numerical setting uses the whole byte. Num, From 0357ffe67064cde9d2087c6df213a47dd74bee04 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 30 Aug 2016 07:46:18 -0700 Subject: [PATCH 0243/3084] Require a specific rustfmt version in test-all script. Rustfmt is still so immature that developers can't have different versions installed, or there will be minute differences in formatting causing the tests to fail. Only run rustfmt as part of the test-all script if the expected version is available. --- cranelift/test-all.sh | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index e4914b9541..586f2b76f1 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -21,10 +21,23 @@ function banner() { echo "====== $@ ======" } -# Run rustfmt if we have it. (Travis probably won't). -if cargo install --list | grep -q '^rustfmt '; then +# Run rustfmt if we have it. +# +# Rustfmt is still immature enough that its formatting decisions can change +# between versions. This makes it difficult to enforce a certain style in a +# test script since not all developers will upgrade rustfmt at the same time. +# To work around this, we only verify formatting when a specific version of +# rustfmt is installed. +# +# This version should always be bumped to the newest version available. +RUSTFMT_VERSION="0.5.0" + +if cargo install --list | grep -q "^rustfmt v$RUSTFMT_VERSION"; then banner "Rust formatting" $topdir/src/format-all.sh --write-mode=diff +else + echo "Please install rustfmt v$RUSTFMT_VERSION to verify formatting." + echo "If a newer version of rustfmt is available, update this script." fi PKGS="cretonne cretonne-reader cretonne-tools" From dcf0b49b077405d866f44771bb25fff17fe5408d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 29 Aug 2016 08:43:33 -0700 Subject: [PATCH 0244/3084] Add comments to the level2 hash tables concatenation. The large LEVEL2 table consists on multiple concatenated constant hash tables. Add comments to the generated output delineating the individual tables. --- meta/gen_encoding.py | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/meta/gen_encoding.py b/meta/gen_encoding.py index c9e0695443..648a1289eb 100644 --- a/meta/gen_encoding.py +++ b/meta/gen_encoding.py @@ -216,7 +216,7 @@ class Level2Table(object): def __iter__(self): return iter(self.lists.values()) - def layout_hashtable(self, level2_hashtables): + def layout_hashtable(self, level2_hashtables, level2_doc): """ Compute the hash table mapping opcode -> enclist. @@ -229,6 +229,11 @@ class Level2Table(object): self.hash_table_offset = len(level2_hashtables) self.hash_table_len = len(hash_table) + level2_doc[self.hash_table_offset].append( + '{:06x}: {}, {} entries'.format( + self.hash_table_offset, + self.ty.name, + self.hash_table_len)) level2_hashtables.extend(hash_table) @@ -289,12 +294,12 @@ def emit_enclists(seq_table, doc_table, fmt): fmt.line(line) -def encode_level2_hashtables(level1, level2_hashtables): +def encode_level2_hashtables(level1, level2_hashtables, level2_doc): for level2 in level1: - level2.layout_hashtable(level2_hashtables) + level2.layout_hashtable(level2_hashtables, level2_doc) -def emit_level2_hashtables(level2_hashtables, fmt): +def emit_level2_hashtables(level2_hashtables, level2_doc, fmt): """ Emit the big concatenation of level 2 hash tables. """ @@ -302,7 +307,10 @@ def emit_level2_hashtables(level2_hashtables, fmt): 'const LEVEL2: [(Opcode, u32); {}] = [' .format(len(level2_hashtables)), '];'): - for entry in level2_hashtables: + for offset, entry in enumerate(level2_hashtables): + if offset in level2_doc: + for doc in level2_doc[offset]: + fmt.comment(doc) if entry: fmt.line( '(Opcode::{}, {:#08x}),' @@ -322,14 +330,16 @@ def gen_isa(isa, fmt): # Single table containing all the level2 hash tables. level2_hashtables = list() + level2_doc = defaultdict(list) for cpumode in isa.cpumodes: + level2_doc[len(level2_hashtables)].append(cpumode.name) level1 = make_tables(cpumode) encode_enclists(level1, seq_table, doc_table) - encode_level2_hashtables(level1, level2_hashtables) + encode_level2_hashtables(level1, level2_hashtables, level2_doc) emit_enclists(seq_table, doc_table, fmt) - emit_level2_hashtables(level2_hashtables, fmt) + emit_level2_hashtables(level2_hashtables, level2_doc, fmt) def generate(isas, out_dir): From 6145f4ca13493a40493d5ee13b87dbf915b7a21c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 29 Aug 2016 16:30:03 -0700 Subject: [PATCH 0245/3084] Add an is_64bit shared setting. Many ISAs and 64-bit and 32-bit variants. Use a shared is_64bit setting to distinguish. --- cranelift/src/libcretonne/settings.rs | 1 + meta/cretonne/settings.py | 2 ++ 2 files changed, 3 insertions(+) diff --git a/cranelift/src/libcretonne/settings.rs b/cranelift/src/libcretonne/settings.rs index b3cf494954..186b06fe09 100644 --- a/cranelift/src/libcretonne/settings.rs +++ b/cranelift/src/libcretonne/settings.rs @@ -262,6 +262,7 @@ mod tests { assert_eq!(f.to_string(), "[shared]\n\ opt_level = \"default\"\n\ + is_64bit = false\n\ enable_simd = true\n"); assert_eq!(f.opt_level(), super::OptLevel::Default); assert_eq!(f.enable_simd(), true); diff --git a/meta/cretonne/settings.py b/meta/cretonne/settings.py index ba95bda36c..4147d5a514 100644 --- a/meta/cretonne/settings.py +++ b/meta/cretonne/settings.py @@ -18,6 +18,8 @@ opt_level = EnumSetting( """, 'default', 'best', 'fastest') +is_64bit = BoolSetting("Enable 64-bit code generation") + enable_simd = BoolSetting( """Enable the use of SIMD instructions.""", default=True) From d67bba1e85fbfceabdf233d56b35ba69d47191ba Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 29 Aug 2016 16:31:16 -0700 Subject: [PATCH 0246/3084] Split the Encoding data type into two u16 values. This hardcodes the division line between the recipe bits and the encoding bits. It does not seem that any ISA will need more than 16 bits for either. --- cranelift/src/libcretonne/isa/mod.rs | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index a84efdbc52..fb2a4dadff 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -100,18 +100,11 @@ pub trait TargetIsa { /// encoding *bits*. The recipe determines the native instruction format and the mapping of /// operands to encoded bits. The encoding bits provide additional information to the recipe, /// typically parts of the opcode. -pub struct Encoding(u32); +pub struct Encoding(u16, u16); impl Encoding { - /// Create a new `Encoding` containing `(recipe, bits)`. The `num_bits` parameter is the - /// ISA-dependent size of `bits`. - pub fn new(recipe: u32, bits: u32, num_bits: u8) -> Encoding { - Encoding((recipe << num_bits) | bits) - } - - /// Split the encoding into two parts: `(recipe, bits)`. Only the target ISA knows how many - /// bits are in each part. - pub fn split(&self, num_bits: u8) -> (u32, u32) { - (self.0 >> num_bits, self.0 & ((1 << num_bits) - 1)) + /// Create a new `Encoding` containing `(recipe, bits)`. + pub fn new(recipe: u16, bits: u16) -> Encoding { + Encoding(recipe, bits) } } From fc6cb2eac185913cf06cd6efcdce7017d905aaa1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 29 Aug 2016 14:00:26 -0700 Subject: [PATCH 0247/3084] Add an isa/encoding module. Define data types for the level 1 and level 2 hash tables. These data types are generic over the offset integer type so they can be twice as compact for typically small ISAs. Use these new types when generating encoding hash tables. Emit both level 1 and level 2 hash tables. Define generic functions that perform lookups in the encoding tables. Implement the TargetIsa::encode() method for RISC-V using these building blocks. --- cranelift/src/libcretonne/isa/encoding.rs | 152 ++++++++++++++++++ cranelift/src/libcretonne/isa/mod.rs | 5 +- .../src/libcretonne/isa/riscv/encoding.rs | 14 ++ cranelift/src/libcretonne/isa/riscv/mod.rs | 25 ++- meta/gen_encoding.py | 79 +++++++-- 5 files changed, 259 insertions(+), 16 deletions(-) create mode 100644 cranelift/src/libcretonne/isa/encoding.rs create mode 100644 cranelift/src/libcretonne/isa/riscv/encoding.rs diff --git a/cranelift/src/libcretonne/isa/encoding.rs b/cranelift/src/libcretonne/isa/encoding.rs new file mode 100644 index 0000000000..62aa54e445 --- /dev/null +++ b/cranelift/src/libcretonne/isa/encoding.rs @@ -0,0 +1,152 @@ +//! Support types for generated encoding tables. +//! +//! This module contains types and functions for working with the encoding tables generated by +//! `meta/gen_encoding.py`. +use ir::{Type, Opcode}; +use isa::Encoding; +use constant_hash::{Table, probe}; + +/// Level 1 hash table entry. +/// +/// One level 1 hash table is generated per CPU mode. This table is keyed by the controlling type +/// variable, using `VOID` for non-polymorphic instructions. +/// +/// The hash table values are references to level 2 hash tables, encoded as an offset in `LEVEL2` +/// where the table begins, and the binary logarithm of its length. All the level 2 hash tables +/// have a power-of-two size. +/// +/// Entries are generic over the offset type. It will typically be `u32` or `u16`, depending on the +/// size of the `LEVEL2` table. A `u16` offset allows entries to shrink to 32 bits each, but some +/// ISAs may have tables so large that `u32` offsets are needed. +/// +/// Empty entries are encoded with a 0 `log2len`. This is on the assumption that no level 2 tables +/// have only a single entry. +pub struct Level1Entry + Copy> { + pub ty: Type, + pub log2len: u8, + pub offset: OffT, +} + +impl + Copy> Table for [Level1Entry] { + fn len(&self) -> usize { + self.len() + } + + fn key(&self, idx: usize) -> Option { + if self[idx].log2len != 0 { + Some(self[idx].ty) + } else { + None + } + } +} + +/// Level 2 hash table entry. +/// +/// The second level hash tables are keyed by `Opcode`, and contain an offset into the `ENCLISTS` +/// table where the encoding recipes for the instrution are stored. +/// +/// Entries are generic over the offset type which depends on the size of `ENCLISTS`. A `u16` +/// offset allows the entries to be only 32 bits each. There is no benefit to dropping down to `u8` +/// for tiny ISAs. The entries won't shrink below 32 bits since the opcode is expected to be 16 +/// bits. +/// +/// Empty entries are encoded with a `NotAnOpcode` `opcode` field. +pub struct Level2Entry + Copy> { + pub opcode: Opcode, + pub offset: OffT, +} + +impl + Copy> Table for [Level2Entry] { + fn len(&self) -> usize { + self.len() + } + + fn key(&self, idx: usize) -> Option { + let opc = self[idx].opcode; + if opc != Opcode::NotAnOpcode { + Some(opc) + } else { + None + } + } +} + +/// Two-level hash table lookup. +/// +/// Given the controlling type variable and instruction opcode, find the corresponding encoding +/// list. +/// +/// Returns an offset into the ISA's `ENCLIST` table, or `None` if the opcode/type combination is +/// not legal. +pub fn lookup_enclist(ctrl_typevar: Type, + opcode: Opcode, + level1_table: &[Level1Entry], + level2_table: &[Level2Entry]) + -> Option + where OffT1: Into + Copy, + OffT2: Into + Copy +{ + probe(level1_table, ctrl_typevar, ctrl_typevar.index()).and_then(|l1idx| { + let l1ent = &level1_table[l1idx]; + let l2off = l1ent.offset.into() as usize; + let l2tab = &level2_table[l2off..l2off + (1 << l1ent.log2len)]; + probe(l2tab, opcode, opcode as usize).map(|l2idx| l2tab[l2idx].offset.into() as usize) + }) +} + +/// Encoding list entry. +/// +/// Encoding lists are represented as sequences of u16 words. +pub type EncListEntry = u16; + +/// Number of bits used to represent a predicate. c.f. `meta.gen_encoding.py`. +const PRED_BITS: u8 = 12; +const PRED_MASK: EncListEntry = (1 << PRED_BITS) - 1; + +/// The match-always instruction predicate. c.f. `meta.gen_encoding.py`. +const CODE_ALWAYS: EncListEntry = PRED_MASK; + +/// The encoding list terminator. +const CODE_FAIL: EncListEntry = 0xffff; + +/// Find the most general encoding of `inst`. +/// +/// Given an encoding list offset as returned by `lookup_enclist` above, search the encoding list +/// for the most general encoding that applies to `inst`. The encoding lists are laid out such that +/// this is the last valid entry in the list. +/// +/// This function takes two closures that are used to evaluate predicates: +/// - `instp` is passed an instruction predicate number to be evaluated on the current instruction. +/// - `isap` is passed an ISA predicate number to evaluate. +/// +/// Returns the corresponding encoding, or `None` if no list entries are satisfied by `inst`. +pub fn general_encoding(offset: usize, + enclist: &[EncListEntry], + instp: InstP, + isap: IsaP) + -> Option + where InstP: Fn(EncListEntry) -> bool, + IsaP: Fn(EncListEntry) -> bool +{ + let mut found = None; + let mut pos = offset; + while enclist[pos] != CODE_FAIL { + let pred = enclist[pos]; + if pred <= CODE_ALWAYS { + // This is an instruction predicate followed by recipe and encbits entries. + if pred == CODE_ALWAYS || instp(pred) { + found = Some(Encoding::new(enclist[pos + 1], enclist[pos + 2])) + } + pos += 3; + } else { + // This is an ISA predicate entry. + pos += 1; + if !isap(pred & PRED_MASK) { + // ISA predicate failed, skip the next N entries. + pos += 3 * (pred >> PRED_BITS) as usize; + } + } + } + found +} diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index fb2a4dadff..60028415fd 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -41,9 +41,10 @@ //! concurrent function compilations. pub mod riscv; +mod encoding; use settings; -use ir::{Inst, DataFlowGraph}; +use ir::{InstructionData, DataFlowGraph}; /// Look for a supported ISA with the given `name`. /// Return a builder that can create a corresponding `TargetIsa`. @@ -91,7 +92,7 @@ pub trait TargetIsa { /// Otherwise, return `None`. /// /// This is also the main entry point for determining if an instruction is legal. - fn encode(&self, dfg: &DataFlowGraph, inst: &Inst) -> Option; + fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Option; } /// Bits needed to encode an instruction as binary machine code. diff --git a/cranelift/src/libcretonne/isa/riscv/encoding.rs b/cranelift/src/libcretonne/isa/riscv/encoding.rs new file mode 100644 index 0000000000..760f4786d7 --- /dev/null +++ b/cranelift/src/libcretonne/isa/riscv/encoding.rs @@ -0,0 +1,14 @@ +//! Encoding tables for RISC-V. + +use ir::{Opcode, InstructionData}; +use ir::instructions::InstructionFormat; +use ir::types; +use predicates; +use isa::encoding::{Level1Entry, Level2Entry}; + +// Include the generated encoding tables: +// - `LEVEL1_RV32` +// - `LEVEL1_RV64` +// - `LEVEL2` +// - `ENCLIST` +include!(concat!(env!("OUT_DIR"), "/encoding-riscv.rs")); diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index b9ffa4925c..0042b5e758 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -1,16 +1,19 @@ //! RISC-V Instruction Set Architecture. pub mod settings; +mod encoding; use super::super::settings as shared_settings; +use isa::encoding as shared_encoding; use super::Builder as IsaBuilder; use super::{TargetIsa, Encoding}; -use ir::{Inst, DataFlowGraph}; +use ir::{InstructionData, DataFlowGraph}; #[allow(dead_code)] struct Isa { shared_flags: shared_settings::Flags, isa_flags: settings::Flags, + cpumode: &'static [shared_encoding::Level1Entry], } pub fn isa_builder() -> IsaBuilder { @@ -23,14 +26,30 @@ pub fn isa_builder() -> IsaBuilder { fn isa_constructor(shared_flags: shared_settings::Flags, builder: shared_settings::Builder) -> Box { + let level1 = if shared_flags.is_64bit() { + &encoding::LEVEL1_RV64[..] + } else { + &encoding::LEVEL1_RV32[..] + }; Box::new(Isa { isa_flags: settings::Flags::new(&shared_flags, builder), shared_flags: shared_flags, + cpumode: level1, }) } impl TargetIsa for Isa { - fn encode(&self, _: &DataFlowGraph, _: &Inst) -> Option { - unimplemented!() + fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Option { + shared_encoding::lookup_enclist(inst.first_type(), + inst.opcode(), + self.cpumode, + &encoding::LEVEL2[..]) + .and_then(|enclist_offset| { + shared_encoding::general_encoding(enclist_offset, + &encoding::ENCLISTS[..], + |instp| encoding::check_instp(inst, instp), + // TODO: Implement ISA predicates properly. + |isap| isap != 17) + }) } } diff --git a/meta/gen_encoding.py b/meta/gen_encoding.py index 648a1289eb..8fe8e1e304 100644 --- a/meta/gen_encoding.py +++ b/meta/gen_encoding.py @@ -54,6 +54,7 @@ import srcgen from constant_hash import compute_quadratic from unique_table import UniqueSeqTable from collections import OrderedDict, defaultdict +import math def emit_instp(instp, fmt): @@ -79,7 +80,7 @@ def emit_instp(instp, fmt): with fmt.indented('{} => {{'.format(instp.number), '}'): with fmt.indented( - 'if let {} {{ {}, .. }} = *inst {{' + 'if let InstructionData::{} {{ {}, .. }} = *inst {{' .format(iform.name, fields), '}'): fmt.line('return {};'.format(instp.rust_predicate(0))) @@ -90,15 +91,15 @@ def emit_instps(instps, fmt): """ with fmt.indented( - 'fn check_instp(inst: &InstructionData, instp_idx: u16) -> bool {', - '}'): + 'pub fn check_instp(inst: &InstructionData, instp_idx: u16) ' + + '-> bool {', '}'): with fmt.indented('match instp_idx {', '}'): for instp in instps: emit_instp(instp, fmt) fmt.line('_ => panic!("Invalid instruction predicate")') # The match cases will fall through if the instruction format is wrong. - fmt.line('panic!("Bad format {}/{} for instp {}",') + fmt.line('panic!("Bad format {:?}/{} for instp {}",') fmt.line(' InstructionFormat::from(inst),') fmt.line(' inst.opcode(),') fmt.line(' instp_idx);') @@ -279,7 +280,7 @@ def encode_enclists(level1, seq_table, doc_table): def emit_enclists(seq_table, doc_table, fmt): with fmt.indented( - 'const ENCLISTS: [u16; {}] = ['.format(len(seq_table.table)), + 'pub static ENCLISTS: [u16; {}] = ['.format(len(seq_table.table)), '];'): line = '' for idx, entry in enumerate(seq_table.table): @@ -299,13 +300,13 @@ def encode_level2_hashtables(level1, level2_hashtables, level2_doc): level2.layout_hashtable(level2_hashtables, level2_doc) -def emit_level2_hashtables(level2_hashtables, level2_doc, fmt): +def emit_level2_hashtables(level2_hashtables, offt, level2_doc, fmt): """ Emit the big concatenation of level 2 hash tables. """ with fmt.indented( - 'const LEVEL2: [(Opcode, u32); {}] = [' - .format(len(level2_hashtables)), + 'pub static LEVEL2: [Level2Entry<{}>; {}] = [' + .format(offt, len(level2_hashtables)), '];'): for offset, entry in enumerate(level2_hashtables): if offset in level2_doc: @@ -313,10 +314,54 @@ def emit_level2_hashtables(level2_hashtables, level2_doc, fmt): fmt.comment(doc) if entry: fmt.line( - '(Opcode::{}, {:#08x}),' + 'Level2Entry ' + + '{{ opcode: Opcode::{}, offset: {:#08x} }},' .format(entry.inst.camel_name, entry.offset)) else: - fmt.line('(Opcode::NotAnOpcode, 0),') + fmt.line( + 'Level2Entry ' + + '{ opcode: Opcode::NotAnOpcode, offset: 0 },') + + +def emit_level1_hashtable(cpumode, level1, offt, fmt): + """ + Emit a level 1 hash table for `cpumode`. + """ + hash_table = compute_quadratic( + level1.tables.values(), + lambda level2: level2.ty.number) + + with fmt.indented( + 'pub static LEVEL1_{}: [Level1Entry<{}>; {}] = [' + .format(cpumode.name.upper(), offt, len(hash_table)), '];'): + for level2 in hash_table: + if level2: + l2l = int(math.log(level2.hash_table_len, 2)) + assert l2l > 0, "Hash table too small" + fmt.line( + 'Level1Entry ' + + '{{ ty: types::{}, log2len: {}, offset: {:#08x} }},' + .format( + level2.ty.name.upper(), + l2l, + level2.hash_table_offset)) + else: + # Empty entry. + fmt.line( + 'Level1Entry ' + + '{ ty: types::VOID, log2len: 0, offset: 0 },') + + +def offset_type(length): + """ + Compute an appropriate Rust integer type to use for offsets into a table of + the given length. + """ + if length <= 0x10000: + return 'u16' + else: + assert length <= 0x100000000, "Table too big" + return 'u32' def gen_isa(isa, fmt): @@ -324,6 +369,9 @@ def gen_isa(isa, fmt): # check_instp() function.. emit_instps(isa.all_instps, fmt) + # Level1 tables, one per CPU mode + level1_tables = dict() + # Tables for enclists with comments. seq_table = UniqueSeqTable() doc_table = defaultdict(list) @@ -335,11 +383,20 @@ def gen_isa(isa, fmt): for cpumode in isa.cpumodes: level2_doc[len(level2_hashtables)].append(cpumode.name) level1 = make_tables(cpumode) + level1_tables[cpumode] = level1 encode_enclists(level1, seq_table, doc_table) encode_level2_hashtables(level1, level2_hashtables, level2_doc) + # Level 1 table encodes offsets into the level 2 table. + level1_offt = offset_type(len(level2_hashtables)) + # Level 2 tables encodes offsets into seq_table. + level2_offt = offset_type(len(seq_table.table)) + emit_enclists(seq_table, doc_table, fmt) - emit_level2_hashtables(level2_hashtables, level2_doc, fmt) + emit_level2_hashtables(level2_hashtables, level2_offt, level2_doc, fmt) + for cpumode in isa.cpumodes: + emit_level1_hashtable( + cpumode, level1_tables[cpumode], level1_offt, fmt) def generate(isas, out_dir): From 6fc6e355851b7b3d56700532db818512fd6225cc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 30 Aug 2016 07:56:12 -0700 Subject: [PATCH 0248/3084] Upgrade to rustfmt 0.6.0. --- cranelift/src/libcretonne/dominator_tree.rs | 6 +----- cranelift/src/libcretonne/ir/entities.rs | 6 +----- cranelift/src/libcretonne/ir/immediates.rs | 6 +----- cranelift/src/libcretonne/ir/instructions.rs | 5 +---- cranelift/src/libcretonne/settings.rs | 4 +--- cranelift/test-all.sh | 2 +- 6 files changed, 6 insertions(+), 23 deletions(-) diff --git a/cranelift/src/libcretonne/dominator_tree.rs b/cranelift/src/libcretonne/dominator_tree.rs index 1b99db342d..a3a00b2cc4 100644 --- a/cranelift/src/libcretonne/dominator_tree.rs +++ b/cranelift/src/libcretonne/dominator_tree.rs @@ -108,11 +108,7 @@ impl DominatorTree { // TODO: we can't rely on instruction numbers to always be ordered // from lowest to highest. Given that, it will be necessary to create // an abolute mapping to determine the instruction order in the future. - if a.1 == NO_INST || a.1 < b.1 { - a - } else { - b - } + if a.1 == NO_INST || a.1 < b.1 { a } else { b } } /// Returns the immediate dominator of some ebb or None if the diff --git a/cranelift/src/libcretonne/ir/entities.rs b/cranelift/src/libcretonne/ir/entities.rs index 33b41e5273..1a887452a7 100644 --- a/cranelift/src/libcretonne/ir/entities.rs +++ b/cranelift/src/libcretonne/ir/entities.rs @@ -42,11 +42,7 @@ impl EntityRef for Ebb { impl Ebb { /// Create a new EBB reference from its number. This corresponds to the ebbNN representation. pub fn with_number(n: u32) -> Option { - if n < u32::MAX { - Some(Ebb(n)) - } else { - None - } + if n < u32::MAX { Some(Ebb(n)) } else { None } } } diff --git a/cranelift/src/libcretonne/ir/immediates.rs b/cranelift/src/libcretonne/ir/immediates.rs index 87433606d7..99276bfd66 100644 --- a/cranelift/src/libcretonne/ir/immediates.rs +++ b/cranelift/src/libcretonne/ir/immediates.rs @@ -60,11 +60,7 @@ impl FromStr for Imm64 { let mut value: u64 = 0; let mut digits = 0; let negative = s.starts_with('-'); - let s2 = if negative { - &s[1..] - } else { - s - }; + let s2 = if negative { &s[1..] } else { s }; if s2.starts_with("0x") { // Hexadecimal. diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index ca43a3dc31..9638d660a5 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -95,10 +95,7 @@ impl FromStr for Opcode { /// `Box` to store the additional information out of line. #[derive(Debug)] pub enum InstructionData { - Nullary { - opcode: Opcode, - ty: Type, - }, + Nullary { opcode: Opcode, ty: Type }, Unary { opcode: Opcode, ty: Type, diff --git a/cranelift/src/libcretonne/settings.rs b/cranelift/src/libcretonne/settings.rs index 186b06fe09..c4f42d4a04 100644 --- a/cranelift/src/libcretonne/settings.rs +++ b/cranelift/src/libcretonne/settings.rs @@ -227,9 +227,7 @@ pub mod detail { #[derive(Clone, Copy)] pub enum Detail { /// A boolean setting only uses one bit, numbered from LSB. - Bool { - bit: u8, - }, + Bool { bit: u8 }, /// A numerical setting uses the whole byte. Num, diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 586f2b76f1..64fb83d3a8 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -30,7 +30,7 @@ function banner() { # rustfmt is installed. # # This version should always be bumped to the newest version available. -RUSTFMT_VERSION="0.5.0" +RUSTFMT_VERSION="0.6.0" if cargo install --list | grep -q "^rustfmt v$RUSTFMT_VERSION"; then banner "Rust formatting" From 662e256e3f3ef1b4952092e3909933929245792d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 30 Aug 2016 13:50:56 -0700 Subject: [PATCH 0249/3084] Generate a table of encoding recipe names for each ISA. This will be used to pretty-print encodings in the textual IR. --- cranelift/src/libcretonne/isa/mod.rs | 6 ++++++ cranelift/src/libcretonne/isa/riscv/mod.rs | 4 ++++ meta/gen_encoding.py | 15 +++++++++++++++ 3 files changed, 25 insertions(+) diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index 60028415fd..18723a16a5 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -93,6 +93,12 @@ pub trait TargetIsa { /// /// This is also the main entry point for determining if an instruction is legal. fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Option; + + /// Get a static array of names associated with encoding recipes in this ISA. Encoding recipes + /// are numbered starting from 0, corresponding to indexes into th name array. + /// + /// This is just used for printing and parsing encodings in the textual IL format. + fn recipe_names(&self) -> &'static [&'static str]; } /// Bits needed to encode an instruction as binary machine code. diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index 0042b5e758..5211479f9c 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -52,4 +52,8 @@ impl TargetIsa for Isa { |isap| isap != 17) }) } + + fn recipe_names(&self) -> &'static [&'static str] { + &encoding::RECIPE_NAMES[..] + } } diff --git a/meta/gen_encoding.py b/meta/gen_encoding.py index 8fe8e1e304..e54d15f7fb 100644 --- a/meta/gen_encoding.py +++ b/meta/gen_encoding.py @@ -364,6 +364,19 @@ def offset_type(length): return 'u32' +def emit_recipe_names(isa, fmt): + """ + Emit a table of encoding recipe names keyed by recipe number. + + This is used for pretty-printing encodings. + """ + with fmt.indented( + 'pub static RECIPE_NAMES: [&\'static str; {}] = [' + .format(len(isa.all_recipes)), '];'): + for r in isa.all_recipes: + fmt.line('"{}",'.format(r.name)) + + def gen_isa(isa, fmt): # First assign numbers to relevant instruction predicates and generate the # check_instp() function.. @@ -398,6 +411,8 @@ def gen_isa(isa, fmt): emit_level1_hashtable( cpumode, level1_tables[cpumode], level1_offt, fmt) + emit_recipe_names(isa, fmt) + def generate(isas, out_dir): for isa in isas: From dddcc0b2caae9bdd805e98c002059114d3488a2a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 30 Aug 2016 10:44:33 -0700 Subject: [PATCH 0250/3084] Add an encoding test for RISC-V. Test that the generated encoding tables work as expected. Change isa::Encoding into a struct with named fields so the recipe and bits can be accessed. --- cranelift/src/libcretonne/isa/mod.rs | 21 +++++++- cranelift/src/libcretonne/isa/riscv/mod.rs | 58 ++++++++++++++++++++++ 2 files changed, 77 insertions(+), 2 deletions(-) diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index 18723a16a5..6eaae9fc62 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -107,11 +107,28 @@ pub trait TargetIsa { /// encoding *bits*. The recipe determines the native instruction format and the mapping of /// operands to encoded bits. The encoding bits provide additional information to the recipe, /// typically parts of the opcode. -pub struct Encoding(u16, u16); +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Encoding { + recipe: u16, + bits: u16, +} impl Encoding { /// Create a new `Encoding` containing `(recipe, bits)`. pub fn new(recipe: u16, bits: u16) -> Encoding { - Encoding(recipe, bits) + Encoding { + recipe: recipe, + bits: bits, + } + } + + /// Get the recipe number in this encoding. + pub fn recipe(self) -> usize { + self.recipe as usize + } + + /// Get the recipe-specific encoding bits. + pub fn bits(self) -> u16 { + self.bits } } diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index 5211479f9c..8abb53467b 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -57,3 +57,61 @@ impl TargetIsa for Isa { &encoding::RECIPE_NAMES[..] } } + +#[cfg(test)] +mod tests { + use settings::{self, Configurable}; + use isa; + use ir::{DataFlowGraph, InstructionData, Opcode}; + use ir::{types, immediates}; + + fn encstr(isa: &isa::TargetIsa, enc: isa::Encoding) -> String { + format!("{}/{:02x}", isa.recipe_names()[enc.recipe()], enc.bits()) + } + + #[test] + fn test_64bitenc() { + let mut shared_builder = settings::builder(); + shared_builder.set_bool("is_64bit", true).unwrap(); + let shared_flags = settings::Flags::new(shared_builder); + let isa = isa::lookup("riscv").unwrap().finish(shared_flags); + + let mut dfg = DataFlowGraph::new(); + let ebb = dfg.make_ebb(); + let arg64 = dfg.append_ebb_arg(ebb, types::I64); + let arg32 = dfg.append_ebb_arg(ebb, types::I32); + + // Try to encode iadd_imm.i64 vx1, -10. + let inst64 = InstructionData::BinaryImm { + opcode: Opcode::IaddImm, + ty: types::I64, + arg: arg64, + imm: immediates::Imm64::new(-10), + }; + + // ADDI is I/0b00100 + assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst64).unwrap()), "I/04"); + + // Try to encode iadd_imm.i64 vx1, -10000. + let inst64_large = InstructionData::BinaryImm { + opcode: Opcode::IaddImm, + ty: types::I64, + arg: arg64, + imm: immediates::Imm64::new(-10000), + }; + + // Immediate is out of range for ADDI. + assert_eq!(isa.encode(&dfg, &inst64_large), None); + + // Create an iadd_imm.i32 which is encodable in RV64. + let inst32 = InstructionData::BinaryImm { + opcode: Opcode::IaddImm, + ty: types::I32, + arg: arg32, + imm: immediates::Imm64::new(10), + }; + + // ADDIW is I/0b00110 + assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32).unwrap()), "I/06"); + } +} From cdac6d1c8e1ca54f344f9eb91e7d5ab70bcf2621 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 30 Aug 2016 14:54:18 -0700 Subject: [PATCH 0251/3084] Add encoding tests for RV32. The 32-bit CPU mode uses a different encoding for iadd_imm.i32, and 64-bit instructions are not supported. --- cranelift/src/libcretonne/isa/riscv/mod.rs | 47 ++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index 8abb53467b..ef87ade3f6 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -114,4 +114,51 @@ mod tests { // ADDIW is I/0b00110 assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32).unwrap()), "I/06"); } + + // Same as above, but for RV32. + #[test] + fn test_32bitenc() { + let mut shared_builder = settings::builder(); + shared_builder.set_bool("is_64bit", false).unwrap(); + let shared_flags = settings::Flags::new(shared_builder); + let isa = isa::lookup("riscv").unwrap().finish(shared_flags); + + let mut dfg = DataFlowGraph::new(); + let ebb = dfg.make_ebb(); + let arg64 = dfg.append_ebb_arg(ebb, types::I64); + let arg32 = dfg.append_ebb_arg(ebb, types::I32); + + // Try to encode iadd_imm.i64 vx1, -10. + let inst64 = InstructionData::BinaryImm { + opcode: Opcode::IaddImm, + ty: types::I64, + arg: arg64, + imm: immediates::Imm64::new(-10), + }; + + // ADDI is I/0b00100 + assert_eq!(isa.encode(&dfg, &inst64), None); + + // Try to encode iadd_imm.i64 vx1, -10000. + let inst64_large = InstructionData::BinaryImm { + opcode: Opcode::IaddImm, + ty: types::I64, + arg: arg64, + imm: immediates::Imm64::new(-10000), + }; + + // Immediate is out of range for ADDI. + assert_eq!(isa.encode(&dfg, &inst64_large), None); + + // Create an iadd_imm.i32 which is encodable in RV32. + let inst32 = InstructionData::BinaryImm { + opcode: Opcode::IaddImm, + ty: types::I32, + arg: arg32, + imm: immediates::Imm64::new(10), + }; + + // ADDI is I/0b00100 + assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32).unwrap()), "I/04"); + } } From 9944bcc92805644c8b0479d8f543c6615e271c03 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 30 Aug 2016 15:27:35 -0700 Subject: [PATCH 0252/3084] Fix settings_size vs byte_size confusion in gen_settings.py. --- meta/gen_settings.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/meta/gen_settings.py b/meta/gen_settings.py index 65bb7c428a..c5c48ad4b7 100644 --- a/meta/gen_settings.py +++ b/meta/gen_settings.py @@ -231,7 +231,7 @@ def gen_constructor(sgrp, settings_size, byte_size, parent, fmt): 'pub fn new({}) -> Flags {{'.format(args), '}'): fmt.line('let bvec = builder.finish("{}");'.format(sgrp.name)) fmt.line('let mut bytes = [0; {}];'.format(byte_size)) - fmt.line('assert_eq!(bytes.len(), {});'.format(settings_size)) + fmt.line('assert_eq!(bvec.len(), {});'.format(settings_size)) with fmt.indented( 'for (i, b) in bvec.into_iter().enumerate() {', '}'): fmt.line('bytes[i] = b;') @@ -268,7 +268,7 @@ def gen_group(sgrp, fmt): fmt.line('#[derive(Clone)]') fmt.doc_comment('Flags group `{}`.'.format(sgrp.name)) with fmt.indented('pub struct Flags {', '}'): - fmt.line('bytes: [u8; {}],'.format(settings_size)) + fmt.line('bytes: [u8; {}],'.format(byte_size)) gen_constructor(sgrp, settings_size, byte_size, None, fmt) gen_enum_types(sgrp, fmt) From 09734f203375a4a57b7f65d0704ea9caa06850e0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 30 Aug 2016 15:10:38 -0700 Subject: [PATCH 0253/3084] Add controls for enabling M, F, and D RISC-V extensions. Three predicates affect each extension: - supports_m determines whether the target CPU supports the instruction set. - enable_m determines if the instructions should be used, assuming they're available. - use_m is the predicate used to actually use the instructions. --- cranelift/src/libcretonne/isa/riscv/settings.rs | 3 ++- cranelift/src/libcretonne/settings.rs | 4 +++- meta/cretonne/settings.py | 8 ++++++++ meta/isa/riscv/settings.py | 9 +++++++++ 4 files changed, 22 insertions(+), 2 deletions(-) diff --git a/cranelift/src/libcretonne/isa/riscv/settings.rs b/cranelift/src/libcretonne/isa/riscv/settings.rs index 1dd1adc405..ae765d9c15 100644 --- a/cranelift/src/libcretonne/isa/riscv/settings.rs +++ b/cranelift/src/libcretonne/isa/riscv/settings.rs @@ -22,7 +22,8 @@ mod tests { supports_m = false\n\ supports_a = false\n\ supports_f = false\n\ - supports_d = false\n"); + supports_d = false\n\ + enable_m = true\n"); // Predicates are not part of the Display output. assert_eq!(f.full_float(), false); } diff --git a/cranelift/src/libcretonne/settings.rs b/cranelift/src/libcretonne/settings.rs index c4f42d4a04..c7c0d1eb30 100644 --- a/cranelift/src/libcretonne/settings.rs +++ b/cranelift/src/libcretonne/settings.rs @@ -261,7 +261,9 @@ mod tests { "[shared]\n\ opt_level = \"default\"\n\ is_64bit = false\n\ - enable_simd = true\n"); + enable_float = true\n\ + enable_simd = true\n\ + enable_atomics = true\n"); assert_eq!(f.opt_level(), super::OptLevel::Default); assert_eq!(f.enable_simd(), true); } diff --git a/meta/cretonne/settings.py b/meta/cretonne/settings.py index 4147d5a514..522a649ef3 100644 --- a/meta/cretonne/settings.py +++ b/meta/cretonne/settings.py @@ -20,8 +20,16 @@ opt_level = EnumSetting( is_64bit = BoolSetting("Enable 64-bit code generation") +enable_float = BoolSetting( + """Enable the use of floating-point instructions""", + default=True) + enable_simd = BoolSetting( """Enable the use of SIMD instructions.""", default=True) +enable_atomics = BoolSetting( + """Enable the use of atomic instructions""", + default=True) + group.close(globals()) diff --git a/meta/isa/riscv/settings.py b/meta/isa/riscv/settings.py index 1ab2917036..8aae295342 100644 --- a/meta/isa/riscv/settings.py +++ b/meta/isa/riscv/settings.py @@ -14,6 +14,15 @@ supports_a = BoolSetting("CPU supports the 'A' extension (atomics)") supports_f = BoolSetting("CPU supports the 'F' extension (float)") supports_d = BoolSetting("CPU supports the 'D' extension (double)") +enable_m = BoolSetting( + "Enable the use of 'M' instructions if available", + default=True) + +use_m = And(supports_m, enable_m) +use_a = And(supports_a, shared.enable_atomics) +use_f = And(supports_f, shared.enable_float) +use_d = And(supports_d, shared.enable_float) + full_float = And(shared.enable_simd, supports_f, supports_d) isa.settings.close(globals()) From 997919c6964878c4c372723790a0789e0641a48b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 30 Aug 2016 15:03:32 -0700 Subject: [PATCH 0254/3084] Add encodings for imul instructions to RISC-V. This is just the basic 'imul' the M instruction set also has mulh/mulhu which yield the high bits of a multiplication, and there are div/rem instructions to be implemented. These instructions are gated by the use_m predicate, but ISA predicates are not completely implemented yet. --- meta/isa/riscv/encodings.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/meta/isa/riscv/encodings.py b/meta/isa/riscv/encodings.py index c339859730..807fdafc7a 100644 --- a/meta/isa/riscv/encodings.py +++ b/meta/isa/riscv/encodings.py @@ -5,6 +5,7 @@ from __future__ import absolute_import from cretonne import base from .defs import RV32, RV64 from .recipes import OPIMM, OPIMM32, OP, OP32, R, Rshamt, I +from .settings import use_m # Basic arithmetic binary instructions are encoded in an R-type instruction. for inst, inst_imm, f3, f7 in [ @@ -45,3 +46,9 @@ for inst, inst_imm, f3, f7 in [ RV32.enc(inst_imm.i32, Rshamt, OPIMM(f3, f7)) RV64.enc(inst_imm.i64, Rshamt, OPIMM(f3, f7)) RV64.enc(inst_imm.i32, Rshamt, OPIMM32(f3, f7)) + +# "M" Standard Extension for Integer Multiplication and Division. +# Gated by the `use_m` flag. +RV32.enc(base.imul.i32, R, OP(0b000, 0b0000001), isap=use_m) +RV64.enc(base.imul.i64, R, OP(0b000, 0b0000001), isap=use_m) +RV64.enc(base.imul.i32, R, OP32(0b000, 0b0000001), isap=use_m) From 036fa46b374930b3036b5145b8d12b15ed86e512 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 31 Aug 2016 08:48:31 -0700 Subject: [PATCH 0255/3084] Fix typo in predicate combination. --- meta/cretonne/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index b44a238ffd..cc3a079e5f 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -1095,7 +1095,7 @@ class Encoding(object): self.encbits = encbits # Combine recipe predicates with the manually specified ones. self.instp = And.combine(recipe.instp, instp) - self.isap = And.combine(recipe.isap, instp) + self.isap = And.combine(recipe.isap, isap) def __str__(self): return '[{}/{:02x}]'.format(self.recipe, self.encbits) From 88c8e9a59a08a91e32d80b6ce996ebb544cdab61 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 31 Aug 2016 11:53:37 -0700 Subject: [PATCH 0256/3084] Move byte-vector layout into SettingGroup.layout(). Move all the byte-sized settings to the front of the byte-vector, and add a mechanism for assigning numbers to predicates that have no name as well as predicates from the parent settings group. This way, all the boolean predicates that are used by a target ISA appear as a contiguous bit-vector that is a suffix of the settings byte-vector. This bit-vector can then be indexed linearly when resolving ISA predicates on encodings. Add a numbered_predicate() method to the generated Flags structs that can read a predicate by number dynamically. --- meta/cretonne/__init__.py | 116 ++++++++++++++++++++++++++++++++++++-- meta/gen_settings.py | 109 +++++++++++++---------------------- 2 files changed, 148 insertions(+), 77 deletions(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index cc3a079e5f..23ec938f61 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -8,7 +8,7 @@ from __future__ import absolute_import import re import math import importlib -from collections import namedtuple +from collections import namedtuple, OrderedDict from .predicates import And @@ -135,7 +135,17 @@ class SettingGroup(object): self.name = name self.parent = parent self.settings = [] - self.predicates = [] + # Named predicates computed from settings in this group or its + # parents. + self.named_predicates = [] + # All boolean predicates that can be accessed by number. This includes: + # - All boolean settings in this group. + # - All named predicates. + # - Added anonymous predicates, see `number_predicate()`. + # - Added parent predicates that are replicated in this group. + # Maps predicate -> number. + self.predicate_number = OrderedDict() + self.open() def open(self): @@ -169,7 +179,8 @@ class SettingGroup(object): if isinstance(obj, Predicate): assert obj.name is None obj.name = name - self.predicates.append(obj) + self.named_predicates.append(obj) + self.layout() @staticmethod def append(setting): @@ -178,6 +189,90 @@ class SettingGroup(object): g.settings.append(setting) return g + def number_predicate(self, pred): + """ + Make sure that `pred` has an assigned number, and will be included in + this group's bit vector. + + The numbered predicates include: + - `BoolSetting` settings that belong to this group. + - `Predicate` instances in `named_predicates`. + - `Predicate` instances without a name. + - Settings or computed predicates that belong to the parent group, but + need to be accessible by number in this group. + + The numbered predicates are referenced by the encoding tables as ISA + predicates. See the `isap` field on `Encoding`. + + :returns: The assigned predicate number in this group. + """ + if pred in self.predicate_number: + return self.predicate_number[pred] + else: + number = len(self.predicate_number) + self.predicate_number[pred] = number + return number + + def layout(self): + """ + Compute the layout of the byte vector used to represent this settings + group. + + The byte vector contains the following entries in order: + + 1. Byte-sized settings like `NumSetting` and `EnumSetting`. + 2. `BoolSetting` settings. + 3. Precomputed named predicates. + 4. Other numbered predicates, including anonymous predicates and parent + predicates that need to be accessible by number. + + Set `self.settings_size` to the length of the byte vector prefix that + contains the settings. All bytes after that are computed, not + configured. + + Set `self.boolean_offset` to the beginning of the numbered predicates, + 2. in the list above. + + Assign `byte_offset` and `bit_offset` fields in all settings. + + After calling this method, no more settings can be added, but + additional predicates can be made accessible with `number_predicate()`. + """ + assert len(self.predicate_number) == 0, "Too late for layout" + + # Assign the non-boolean settings. + byte_offset = 0 + for s in self.settings: + if not isinstance(s, BoolSetting): + s.byte_offset = byte_offset + byte_offset += 1 + + # Then the boolean settings. + self.boolean_offset = byte_offset + for s in self.settings: + if isinstance(s, BoolSetting): + number = self.number_predicate(s) + s.byte_offset = byte_offset + number // 8 + s.bit_offset = number % 8 + + # This is the end of the settings. Round up to a whole number of bytes. + self.boolean_settings = len(self.predicate_number) + self.settings_size = self.byte_size() + + # Now assign numbers to all our named predicates. + for p in self.named_predicates: + self.number_predicate(p) + + def byte_size(self): + """ + Compute the number of bytes required to hold all settings and + precomputed predicates. + + This is the size of the byte-sized settings plus all the numbered + predcate bits rounded up to a whole number of bytes. + """ + return self.boolean_offset + (len(self.predicate_number) + 7) // 8 + # Kinds of operands. # @@ -977,7 +1072,7 @@ class TargetISA(object): :returns self: """ self._collect_encoding_recipes() - self._collect_instruction_predicates() + self._collect_predicates() return self def _collect_encoding_recipes(self): @@ -994,12 +1089,15 @@ class TargetISA(object): rcps.add(recipe) self.all_recipes.append(recipe) - def _collect_instruction_predicates(self): + def _collect_predicates(self): """ - Collect and number all instruction predicates in use. + Collect and number all predicates in use. Sets `instp.number` for all used instruction predicates and places them in `self.all_instps` in numerical order. + + Ensures that all ISA predicates have an assigned bit number in + `self.settings`. """ self.all_instps = list() instps = set() @@ -1012,6 +1110,12 @@ class TargetISA(object): instps.add(instp) self.all_instps.append(instp) + # All referenced ISA predicates must have a number in + # `self.settings`. This may cause some parent predicates to be + # replicated here, which is OK. + if enc.isap: + self.settings.number_predicate(enc.isap) + class CPUMode(object): """ diff --git a/meta/gen_settings.py b/meta/gen_settings.py index c5c48ad4b7..588c2cf798 100644 --- a/meta/gen_settings.py +++ b/meta/gen_settings.py @@ -8,52 +8,6 @@ import constant_hash from cretonne import camel_case, BoolSetting, NumSetting, EnumSetting, settings -def layout_group(sgrp): - """ - Layout the settings in sgrp, assigning byte and bit offsets. - - Return the number of bytes needed for settings and the total number of - bytes needed when including predicates. - """ - # Byte offset where booleans are allocated. - bool_byte = -1 - # Next available bit number in bool_byte. - bool_bit = 10 - # Next available whole byte. - next_byte = 0 - - for setting in sgrp.settings: - if isinstance(setting, BoolSetting): - # Allocate a bit from bool_byte. - if bool_bit > 7: - bool_byte = next_byte - next_byte += 1 - bool_bit = 0 - setting.byte_offset = bool_byte - setting.bit_offset = bool_bit - bool_bit += 1 - else: - # This is a numerical or enumerated setting. Allocate a single - # byte. - setting.byte_offset = next_byte - next_byte += 1 - - settings_size = next_byte - - # Allocate bits for all the precomputed predicates. - for pred in sgrp.predicates: - # Allocate a bit from bool_byte. - if bool_bit > 7: - bool_byte = next_byte - next_byte += 1 - bool_bit = 0 - pred.byte_offset = bool_byte - pred.bit_offset = bool_bit - bool_bit += 1 - - return (settings_size, next_byte) - - def gen_enum_types(sgrp, fmt): """ Emit enum types for any enum settings. @@ -68,7 +22,7 @@ def gen_enum_types(sgrp, fmt): .format(ty, ", ".join(camel_case(v) for v in setting.values))) -def gen_getter(setting, fmt): +def gen_getter(setting, sgrp, fmt): """ Emit a getter function for `setting`. """ @@ -77,9 +31,9 @@ def gen_getter(setting, fmt): if isinstance(setting, BoolSetting): proto = 'pub fn {}(&self) -> bool'.format(setting.name) with fmt.indented(proto + ' {', '}'): - fmt.line('(self.bytes[{}] & (1 << {})) != 0'.format( - setting.byte_offset, - setting.bit_offset)) + fmt.line( + 'self.numbered_predicate({})' + .format(sgrp.predicate_number[setting])) elif isinstance(setting, NumSetting): proto = 'pub fn {}(&self) -> u8'.format(setting.name) with fmt.indented(proto + ' {', '}'): @@ -98,16 +52,16 @@ def gen_getter(setting, fmt): raise AssertionError("Unknown setting kind") -def gen_pred_getter(pred, fmt): +def gen_pred_getter(pred, sgrp, fmt): """ Emit a getter for a pre-computed predicate. """ fmt.doc_comment('Computed predicate `{}`.'.format(pred.rust_predicate(0))) proto = 'pub fn {}(&self) -> bool'.format(pred.name) with fmt.indented(proto + ' {', '}'): - fmt.line('(self.bytes[{}] & (1 << {})) != 0'.format( - pred.byte_offset, - pred.bit_offset)) + fmt.line( + 'self.numbered_predicate({})' + .format(sgrp.predicate_number[pred])) def gen_getters(sgrp, fmt): @@ -116,10 +70,16 @@ def gen_getters(sgrp, fmt): """ fmt.doc_comment("User-defined settings.") with fmt.indented('impl Flags {', '}'): + # Dynamic numbered predicate getter. + with fmt.indented( + 'pub fn numbered_predicate(&self, p: usize) -> bool {', '}'): + fmt.line( + 'self.bytes[{} + p/8] & (1 << (p%8)) != 0' + .format(sgrp.boolean_offset)) for setting in sgrp.settings: - gen_getter(setting, fmt) - for pred in sgrp.predicates: - gen_pred_getter(pred, fmt) + gen_getter(setting, sgrp, fmt) + for pred in sgrp.named_predicates: + gen_pred_getter(pred, sgrp, fmt) def gen_descriptors(sgrp, fmt): @@ -175,11 +135,11 @@ def gen_descriptors(sgrp, fmt): fmt.line('{},'.format(h.descriptor_index)) -def gen_template(sgrp, settings_size, fmt): +def gen_template(sgrp, fmt): """ Emit a Template constant. """ - v = [0] * settings_size + v = [0] * sgrp.settings_size for setting in sgrp.settings: v[setting.byte_offset] |= setting.default_byte() @@ -217,7 +177,7 @@ def gen_display(sgrp, fmt): fmt.line('Ok(())') -def gen_constructor(sgrp, settings_size, byte_size, parent, fmt): +def gen_constructor(sgrp, parent, fmt): """ Generate a Flags constructor. """ @@ -230,14 +190,14 @@ def gen_constructor(sgrp, settings_size, byte_size, parent, fmt): with fmt.indented( 'pub fn new({}) -> Flags {{'.format(args), '}'): fmt.line('let bvec = builder.finish("{}");'.format(sgrp.name)) - fmt.line('let mut bytes = [0; {}];'.format(byte_size)) - fmt.line('assert_eq!(bvec.len(), {});'.format(settings_size)) + fmt.line('let mut bytes = [0; {}];'.format(sgrp.byte_size())) + fmt.line('assert_eq!(bvec.len(), {});'.format(sgrp.settings_size)) with fmt.indented( 'for (i, b) in bvec.into_iter().enumerate() {', '}'): fmt.line('bytes[i] = b;') # Stop here without predicates. - if len(sgrp.predicates) == 0: + if len(sgrp.predicate_number) == sgrp.boolean_settings: fmt.line('Flags { bytes: bytes }') return @@ -246,15 +206,24 @@ def gen_constructor(sgrp, settings_size, byte_size, parent, fmt): 'let mut {} = Flags {{ bytes: bytes }};' .format(sgrp.name)) - for pred in sgrp.predicates: - fmt.comment('Precompute: {}.'.format(pred.name)) + for pred, number in sgrp.predicate_number.items(): + # Don't compute our own settings. + if number < sgrp.boolean_settings: + continue + if pred.name: + fmt.comment( + 'Precompute #{} ({}).'.format(number, pred.name)) + else: + fmt.comment('Precompute #{}.'.format(number)) with fmt.indented( 'if {} {{'.format(pred.rust_predicate(0)), '}'): fmt.line( '{}.bytes[{}] |= 1 << {};' .format( - sgrp.name, pred.byte_offset, pred.bit_offset)) + sgrp.name, + sgrp.boolean_offset + number // 8, + number % 8)) fmt.line(sgrp.name) @@ -263,18 +232,16 @@ def gen_group(sgrp, fmt): """ Generate a Flags struct representing `sgrp`. """ - settings_size, byte_size = layout_group(sgrp) - fmt.line('#[derive(Clone)]') fmt.doc_comment('Flags group `{}`.'.format(sgrp.name)) with fmt.indented('pub struct Flags {', '}'): - fmt.line('bytes: [u8; {}],'.format(byte_size)) + fmt.line('bytes: [u8; {}],'.format(sgrp.byte_size())) - gen_constructor(sgrp, settings_size, byte_size, None, fmt) + gen_constructor(sgrp, None, fmt) gen_enum_types(sgrp, fmt) gen_getters(sgrp, fmt) gen_descriptors(sgrp, fmt) - gen_template(sgrp, settings_size, fmt) + gen_template(sgrp, fmt) gen_display(sgrp, fmt) From de330402b8a138a269516b7fd9caa09e3fa5681c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 31 Aug 2016 15:44:36 -0700 Subject: [PATCH 0257/3084] Add casual string representation of named settings and predicates. Use 'group.setting' format for named predicates, only display the expression for anonymous predicates. --- meta/cretonne/__init__.py | 3 +++ meta/cretonne/predicates.py | 10 +++++----- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 23ec938f61..82b0a02be6 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -35,6 +35,9 @@ class Setting(object): self.byte_offset = None self.group = SettingGroup.append(self) + def __str__(self): + return '{}.{}'.format(self.group.name, self.name) + def predicate_context(self): """ Return the context where this setting can be evaluated as a (leaf) diff --git a/meta/cretonne/predicates.py b/meta/cretonne/predicates.py index 6722c820e5..8a1b18807a 100644 --- a/meta/cretonne/predicates.py +++ b/meta/cretonne/predicates.py @@ -66,12 +66,12 @@ class Predicate(object): assert self.context, "Incompatible predicate parts" def __str__(self): - s = '{}({})'.format( - type(self).__name__, - ' ,'.join(map(str, self.parts))) if self.name: - s = '{}={}'.format(self.name, s) - return s + return '{}.{}'.format(self.context.name, self.name) + else: + return '{}({})'.format( + type(self).__name__, + ', '.join(map(str, self.parts))) def predicate_context(self): return self.context From 78a9e67a09fc527da5b6269edf40d5972ebd03cb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 31 Aug 2016 10:29:53 -0700 Subject: [PATCH 0258/3084] Emit ISA predicates in the encoding tables. Use the new ISA predicate numbering to emit ISA predicate instructions in the encoding tables. Properly decode the ISA predicate number in RISC-V and add tests for RV32M iwth and without 'supports_m' enabled. --- cranelift/src/libcretonne/isa/riscv/mod.rs | 53 ++++++++++++++++++---- meta/gen_encoding.py | 48 ++++++++++++++++---- 2 files changed, 84 insertions(+), 17 deletions(-) diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index ef87ade3f6..a4c1ce9885 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -40,16 +40,16 @@ fn isa_constructor(shared_flags: shared_settings::Flags, impl TargetIsa for Isa { fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Option { - shared_encoding::lookup_enclist(inst.first_type(), - inst.opcode(), - self.cpumode, - &encoding::LEVEL2[..]) + use isa::encoding::{lookup_enclist, general_encoding}; + lookup_enclist(inst.first_type(), + inst.opcode(), + self.cpumode, + &encoding::LEVEL2[..]) .and_then(|enclist_offset| { - shared_encoding::general_encoding(enclist_offset, - &encoding::ENCLISTS[..], - |instp| encoding::check_instp(inst, instp), - // TODO: Implement ISA predicates properly. - |isap| isap != 17) + general_encoding(enclist_offset, + &encoding::ENCLISTS[..], + |instp| encoding::check_instp(inst, instp), + |isap| self.isa_flags.numbered_predicate(isap as usize)) }) } @@ -160,5 +160,40 @@ mod tests { // ADDI is I/0b00100 assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32).unwrap()), "I/04"); + + // Create an imul.i32 which is encodable in RV32, but only when use_m is true. + let mul32 = InstructionData::Binary { + opcode: Opcode::Imul, + ty: types::I32, + args: [arg32, arg32], + }; + + assert_eq!(isa.encode(&dfg, &mul32), None); + } + + #[test] + fn test_rv32m() { + let mut shared_builder = settings::builder(); + shared_builder.set_bool("is_64bit", false).unwrap(); + let shared_flags = settings::Flags::new(shared_builder); + + // Set the supports_m stting which in turn enables the use_m predicate that unlocks + // encodings for imul. + let mut isa_builder = isa::lookup("riscv").unwrap(); + isa_builder.set_bool("supports_m", true).unwrap(); + + let isa = isa_builder.finish(shared_flags); + + let mut dfg = DataFlowGraph::new(); + let ebb = dfg.make_ebb(); + let arg32 = dfg.append_ebb_arg(ebb, types::I32); + + // Create an imul.i32 which is encodable in RV32M. + let mul32 = InstructionData::Binary { + opcode: Opcode::Imul, + ty: types::I32, + args: [arg32, arg32], + }; + assert_eq!(encstr(&*isa, isa.encode(&dfg, &mul32).unwrap()), "R/10c"); } } diff --git a/meta/gen_encoding.py b/meta/gen_encoding.py index e54d15f7fb..e404fb79ec 100644 --- a/meta/gen_encoding.py +++ b/meta/gen_encoding.py @@ -55,6 +55,7 @@ from constant_hash import compute_quadratic from unique_table import UniqueSeqTable from collections import OrderedDict, defaultdict import math +import itertools def emit_instp(instp, fmt): @@ -168,7 +169,26 @@ class EncList(object): name += ' ({})'.format(self.encodings[0].cpumode) return name - def encode(self, seq_table, doc_table): + def by_isap(self): + """ + Group the encodings by ISA predicate without reordering them. + + Yield a sequence of `(isap, (encs...))` tuples where `isap` is the ISA + predicate or `None`, and `(encs...)` is a tuple of encodings that all + have the same ISA predicate. + """ + maxlen = CODE_FAIL >> PRED_BITS + for isap, group in itertools.groupby( + self.encodings, lambda enc: enc.isap): + group = tuple(group) + # This probably never happens, but we can't express more than + # maxlen encodings per isap. + while len(group) > maxlen: + yield (isap, group[0..maxlen]) + group = group[maxlen:] + yield (isap, group) + + def encode(self, seq_table, doc_table, isa): """ Encode this list as a sequence of u16 numbers. @@ -180,10 +200,22 @@ class EncList(object): words = list() docs = list() - for idx, enc in enumerate(self.encodings): - seq, doc = seq_doc(enc) - docs.append((len(words), doc)) - words.extend(seq) + # Group our encodings by isap. + for isap, group in self.by_isap(): + if isap: + # We have an ISA predicate covering `glen` encodings. + pnum = isa.settings.predicate_number[isap] + glen = len(group) + doc = 'skip {}x3 unless {}'.format(glen, isap) + docs.append((len(words), doc)) + words.append((glen << PRED_BITS) | pnum) + + for enc in group: + seq, doc = seq_doc(enc) + docs.append((len(words), doc)) + words.extend(seq) + + # Terminate the list. words.append(CODE_FAIL) self.offset = seq_table.add(words) @@ -269,13 +301,13 @@ def make_tables(cpumode): return table -def encode_enclists(level1, seq_table, doc_table): +def encode_enclists(level1, seq_table, doc_table, isa): """ Compute encodings and doc comments for encoding lists in `level1`. """ for level2 in level1: for enclist in level2: - enclist.encode(seq_table, doc_table) + enclist.encode(seq_table, doc_table, isa) def emit_enclists(seq_table, doc_table, fmt): @@ -397,7 +429,7 @@ def gen_isa(isa, fmt): level2_doc[len(level2_hashtables)].append(cpumode.name) level1 = make_tables(cpumode) level1_tables[cpumode] = level1 - encode_enclists(level1, seq_table, doc_table) + encode_enclists(level1, seq_table, doc_table, isa) encode_level2_hashtables(level1, level2_hashtables, level2_doc) # Level 1 table encodes offsets into the level 2 table. From d8712b2ce68f173ec9ed6666d612331dfbaa2c9e Mon Sep 17 00:00:00 2001 From: Morgan Phillips Date: Tue, 6 Sep 2016 14:05:44 -0700 Subject: [PATCH 0259/3084] Add a verifier The current implementation only performs a few basic checks. --- cranelift/src/libcretonne/Cargo.toml | 3 + cranelift/src/libcretonne/ir/instructions.rs | 10 + cranelift/src/libcretonne/ir/layout.rs | 7 +- cranelift/src/libcretonne/ir/mod.rs | 2 +- cranelift/src/libcretonne/lib.rs | 1 + .../src/libcretonne/test_utils/make_inst.rs | 10 +- cranelift/src/libcretonne/verifier.rs | 171 ++++++++++++++++++ cranelift/src/tools/Cargo.lock | 3 + cranelift/src/tools/tests/verifier.rs | 66 +++++++ .../tests/verifier_testdata/bad_layout.cton | 17 ++ 10 files changed, 287 insertions(+), 3 deletions(-) create mode 100644 cranelift/src/libcretonne/verifier.rs create mode 100644 cranelift/src/tools/tests/verifier.rs create mode 100644 cranelift/src/tools/tests/verifier_testdata/bad_layout.cton diff --git a/cranelift/src/libcretonne/Cargo.toml b/cranelift/src/libcretonne/Cargo.toml index 4d9b59bb43..5d3a9511e6 100644 --- a/cranelift/src/libcretonne/Cargo.toml +++ b/cranelift/src/libcretonne/Cargo.toml @@ -11,3 +11,6 @@ build = "build.rs" [lib] name = "cretonne" path = "lib.rs" + +[dependencies] +regex = "0.1.71" diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index 9638d660a5..a6d664170e 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -349,6 +349,16 @@ impl InstructionData { _ => BranchInfo::NotABranch, } } + + /// Return true if an instruction is terminating, or false otherwise. + pub fn is_terminating<'a>(&'a self) -> bool { + match self { + &InstructionData::Jump { .. } => true, + &InstructionData::Return { .. } => true, + &InstructionData::Nullary { .. } => true, + _ => false, + } + } } /// Information about branch and jump instructions. diff --git a/cranelift/src/libcretonne/ir/layout.rs b/cranelift/src/libcretonne/ir/layout.rs index a983ca178f..c519a1c44f 100644 --- a/cranelift/src/libcretonne/ir/layout.rs +++ b/cranelift/src/libcretonne/ir/layout.rs @@ -55,7 +55,7 @@ impl Layout { /// can only be removed from the layout when it is empty. /// /// Since every EBB must end with a terminator instruction which cannot fall through, the layout of -/// EBBs does not affect the semantics of the program. +/// EBBs do not affect the semantics of the program. /// impl Layout { /// Is `ebb` currently part of the layout? @@ -188,6 +188,11 @@ impl Layout { ebb_node.last_inst = inst; } + /// Fetch an ebb's last instruction. + pub fn last_inst(&self, ebb: Ebb) -> Inst { + self.ebbs[ebb].last_inst + } + /// Insert `inst` before the instruction `before` in the same EBB. pub fn insert_inst(&mut self, inst: Inst, before: Inst) { assert_eq!(self.inst_ebb(inst), None); diff --git a/cranelift/src/libcretonne/ir/mod.rs b/cranelift/src/libcretonne/ir/mod.rs index 91f8b2516c..a26e48f15e 100644 --- a/cranelift/src/libcretonne/ir/mod.rs +++ b/cranelift/src/libcretonne/ir/mod.rs @@ -16,6 +16,6 @@ pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable}; pub use ir::instructions::{Opcode, InstructionData}; pub use ir::stackslot::StackSlotData; pub use ir::jumptable::JumpTableData; -pub use ir::dfg::DataFlowGraph; +pub use ir::dfg::{DataFlowGraph, ValueDef}; pub use ir::layout::Layout; pub use ir::function::Function; diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index fcddbdb900..f338aba8ac 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -14,6 +14,7 @@ pub mod cfg; pub mod dominator_tree; pub mod entity_map; pub mod settings; +pub mod verifier; mod constant_hash; mod predicates; diff --git a/cranelift/src/libcretonne/test_utils/make_inst.rs b/cranelift/src/libcretonne/test_utils/make_inst.rs index eb75b2fab3..a072b8ba8f 100644 --- a/cranelift/src/libcretonne/test_utils/make_inst.rs +++ b/cranelift/src/libcretonne/test_utils/make_inst.rs @@ -2,7 +2,7 @@ use ir::{Function, Ebb, Inst, Opcode}; use ir::entities::NO_VALUE; -use ir::instructions::{InstructionData, VariableArgs, JumpData, BranchData}; +use ir::instructions::{InstructionData, ReturnData, VariableArgs, JumpData, BranchData}; use ir::types; pub fn jump(func: &mut Function, dest: Ebb) -> Inst { @@ -27,3 +27,11 @@ pub fn branch(func: &mut Function, dest: Ebb) -> Inst { }), }) } + +pub fn ret(func: &mut Function) -> Inst { + func.dfg.make_inst(InstructionData::Return { + opcode: Opcode::Return, + ty: types::VOID, + data: Box::new(ReturnData { args: VariableArgs::new() }), + }) +} diff --git a/cranelift/src/libcretonne/verifier.rs b/cranelift/src/libcretonne/verifier.rs new file mode 100644 index 0000000000..e134b36285 --- /dev/null +++ b/cranelift/src/libcretonne/verifier.rs @@ -0,0 +1,171 @@ +//! A verifier for ensuring that functions are well formed. +//! It verifies: +//! +//! EBB integrity +//! +//! - All instructions reached from the ebb_insts iterator must belong to +//! the EBB as reported by inst_ebb(). +//! - Every EBB must end in a terminator instruction, and no other instruction +//! can be a terminator. +//! - Every value in the ebb_args iterator belongs to the EBB as reported by value_ebb. +//! +//! Instruction integrity +//! +//! - The instruction format must match the opcode. +//! TODO: +//! - All result values must be created for multi-valued instructions. +//! - Instructions with no results must have a VOID first_type(). +//! - All referenced entities must exist. (Values, EBBs, stack slots, ...) +//! +//! SSA form +//! +//! - Values must be defined by an instruction that exists and that is inserted in +//! an EBB, or be an argument of an existing EBB. +//! - Values used by an instruction must dominate the instruction. +//! Control flow graph and dominator tree integrity: +//! +//! - All predecessors in the CFG must be branches to the EBB. +//! - All branches to an EBB must be present in the CFG. +//! - A recomputed dominator tree is identical to the existing one. +//! +//! Type checking +//! +//! - Compare input and output values against the opcode's type constraints. +//! For polymorphic opcodes, determine the controlling type variable first. +//! - Branches and jumps must pass arguments to destination EBBs that match the +//! expected types excatly. The number of arguments must match. +//! - All EBBs in a jump_table must take no arguments. +//! - Function calls are type checked against their signature. +//! - The entry block must take arguments that match the signature of the current +//! function. +//! - All return instructions must have return value operands matching the current +//! function signature. +//! +//! Ad hoc checking +//! +//! - Stack slot loads and stores must be in-bounds. +//! - Immediate constraints for certain opcodes, like udiv_imm v3, 0. +//! - Extend / truncate instructions have more type constraints: Source type can't be +//! larger / smaller than result type. +//! - Insertlane and extractlane instructions have immediate lane numbers that must be in +//! range for their polymorphic type. +//! - Swizzle and shuffle instructions take a variable number of lane arguments. The number +//! of arguments must match the destination type, and the lane indexes must be in range. + +use ir::{Function, ValueDef, Ebb, Inst}; +use ir::instructions::InstructionFormat; + +pub struct Verifier<'a> { + func: &'a Function, +} + +impl<'a> Verifier<'a> { + pub fn new(func: &'a Function) -> Verifier { + Verifier { func: func } + } + + fn ebb_integrity(&self, ebb: Ebb, inst: Inst) -> Result<(), String> { + + let is_terminator = self.func.dfg[inst].is_terminating(); + let is_last_inst = self.func.layout.last_inst(ebb) == inst; + + if is_terminator && !is_last_inst { + // Terminating instructions only occur at the end of blocks. + return Err(format!("A terminating instruction was encountered before the \ + end of ebb {:?}!", + ebb)); + } + if is_last_inst && !is_terminator { + return Err(format!("Block {:?} does not end in a terminating instruction!", ebb)); + } + + // Instructions belong to the correct ebb. + let inst_ebb = self.func.layout.inst_ebb(inst); + if inst_ebb != Some(ebb) { + return Err(format!("{:?} should belong to {:?} not {:?}", inst, ebb, inst_ebb)); + } + + // Arguments belong to the correct ebb. + for arg in self.func.dfg.ebb_args(ebb) { + match self.func.dfg.value_def(arg) { + ValueDef::Arg(arg_ebb, _) => { + if ebb != arg_ebb { + return Err(format!("{:?} does not belong to {:?}", arg, ebb)); + } + } + _ => { + return Err("Expected an argument, found a result!".to_string()); + } + } + } + + Ok(()) + } + + fn instruction_integrity(&self, inst: Inst) -> Result<(), String> { + let inst_data = &self.func.dfg[inst]; + + // The instruction format matches the opcode + if inst_data.opcode().format() != Some(InstructionFormat::from(inst_data)) { + return Err("Instruction opcode doesn't match instruction format!".to_string()); + } + + Ok(()) + } + + pub fn run(&self) -> Result<(), String> { + for ebb in self.func.layout.ebbs() { + for inst in self.func.layout.ebb_insts(ebb) { + try!(self.ebb_integrity(ebb, inst)); + try!(self.instruction_integrity(inst)); + } + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + extern crate regex; + + use super::*; + use ir::Function; + use ir::instructions::{InstructionData, Opcode}; + use ir::types; + use self::regex::Regex; + + macro_rules! assert_err_with_msg { + ($e:expr, $msg:expr) => ( + let err_re = Regex::new($msg).unwrap(); + match $e { + Ok(_) => { panic!("Expected an error!") }, + Err(err_msg) => { + if !err_re.is_match(&err_msg) { + panic!(format!("'{}' did not contain the pattern '{}'", err_msg, $msg)); + } + } + } + ) + } + + #[test] + fn empty() { + let func = Function::new(); + let verifier = Verifier::new(&func); + assert_eq!(verifier.run(), Ok(())); + } + + #[test] + fn bad_instruction_format() { + let mut func = Function::new(); + let ebb0 = func.dfg.make_ebb(); + func.layout.append_ebb(ebb0); + let nullary_with_bad_opcode = func.dfg.make_inst(InstructionData::Nullary { + opcode: Opcode::Jump, + ty: types::VOID, + }); + func.layout.append_inst(nullary_with_bad_opcode, ebb0); + let verifier = Verifier::new(&func); + assert_err_with_msg!(verifier.run(), "instruction format"); + } +} diff --git a/cranelift/src/tools/Cargo.lock b/cranelift/src/tools/Cargo.lock index d7060f22e8..251f074c32 100644 --- a/cranelift/src/tools/Cargo.lock +++ b/cranelift/src/tools/Cargo.lock @@ -21,6 +21,9 @@ dependencies = [ [[package]] name = "cretonne" version = "0.0.0" +dependencies = [ + "regex 0.1.71 (registry+https://github.com/rust-lang/crates.io-index)", +] [[package]] name = "cretonne-reader" diff --git a/cranelift/src/tools/tests/verifier.rs b/cranelift/src/tools/tests/verifier.rs new file mode 100644 index 0000000000..28bb1bbba5 --- /dev/null +++ b/cranelift/src/tools/tests/verifier.rs @@ -0,0 +1,66 @@ +extern crate cretonne; +extern crate cton_reader; +extern crate glob; +extern crate regex; + +use std::env; +use glob::glob; +use regex::Regex; +use std::fs::File; +use std::io::Read; +use self::cton_reader::parser::Parser; +use self::cretonne::verifier::Verifier; + +/// Compile a function and run verifier tests based on specially formatted +/// comments in the [function's] source. +fn verifier_tests_from_source(function_source: &str) { + let func_re = Regex::new("^[ \t]*function.*").unwrap(); + let err_re = Regex::new(";[ \t]*Err\\((.*)+\\)").unwrap(); + + // Each entry corresponds to an optional regular expression, where + // the index corresponds to the function offset in our source code. + // If no error is expected for a given function its entry will be + // set to None. + let mut verifier_results = Vec::new(); + for line in function_source.lines() { + if func_re.is_match(line) { + match err_re.captures(line) { + Some(caps) => { + verifier_results.push(Some(Regex::new(caps.at(1).unwrap()).unwrap())); + }, + None => { + verifier_results.push(None); + }, + }; + } + } + + // Run the verifier against each function and compare the output + // with the expected result (as determined above). + for (i, func) in Parser::parse(function_source).unwrap().into_iter().enumerate() { + let result = Verifier::new(&func).run(); + match verifier_results[i] { + Some(ref re) => { + assert_eq!(re.is_match(&result.err().unwrap()), true); + }, + None => { + assert_eq!(result, Ok(())); + } + } + } +} + +#[test] +fn test_all() { + let testdir = format!("{}/tests/verifier_testdata/*.cton", + env::current_dir().unwrap().display()); + + for entry in glob(&testdir).unwrap() { + let path = entry.unwrap(); + println!("Testing {:?}", path); + let mut file = File::open(&path).unwrap(); + let mut buffer = String::new(); + file.read_to_string(&mut buffer).unwrap(); + verifier_tests_from_source(&buffer); + } +} diff --git a/cranelift/src/tools/tests/verifier_testdata/bad_layout.cton b/cranelift/src/tools/tests/verifier_testdata/bad_layout.cton new file mode 100644 index 0000000000..680299def7 --- /dev/null +++ b/cranelift/src/tools/tests/verifier_testdata/bad_layout.cton @@ -0,0 +1,17 @@ +function test(i32) { ; Err(terminating) + ebb0(v0: i32): + jump ebb1 + return + ebb1: + jump ebb2 + brz v0, ebb3 + ebb2: + jump ebb3 + ebb3: + return +} + +function test(i32) { ; Ok + ebb0(v0: i32): + return +} From 71b742ec347258a16e40b59b9ccd8ecadcccde7c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 1 Sep 2016 14:38:01 -0700 Subject: [PATCH 0260/3084] Add a libfilecheck crate. This library implements functionality similar to LLVM's FileCheck utility, but in library form. --- cranelift/src/libfilecheck/Cargo.toml | 12 + cranelift/src/libfilecheck/checker.rs | 392 ++++++++++++++++ cranelift/src/libfilecheck/error.rs | 69 +++ cranelift/src/libfilecheck/lib.rs | 246 ++++++++++ cranelift/src/libfilecheck/pattern.rs | 523 ++++++++++++++++++++++ cranelift/src/libfilecheck/tests/basic.rs | 313 +++++++++++++ cranelift/src/libfilecheck/variable.rs | 58 +++ 7 files changed, 1613 insertions(+) create mode 100644 cranelift/src/libfilecheck/Cargo.toml create mode 100644 cranelift/src/libfilecheck/checker.rs create mode 100644 cranelift/src/libfilecheck/error.rs create mode 100644 cranelift/src/libfilecheck/lib.rs create mode 100644 cranelift/src/libfilecheck/pattern.rs create mode 100644 cranelift/src/libfilecheck/tests/basic.rs create mode 100644 cranelift/src/libfilecheck/variable.rs diff --git a/cranelift/src/libfilecheck/Cargo.toml b/cranelift/src/libfilecheck/Cargo.toml new file mode 100644 index 0000000000..4f85778d92 --- /dev/null +++ b/cranelift/src/libfilecheck/Cargo.toml @@ -0,0 +1,12 @@ +[package] +authors = ["The Cretonne Project Developers"] +name = "filecheck" +version = "0.0.0" +publish = false + +[lib] +name = "filecheck" +path = "lib.rs" + +[dependencies] +regex = "0.1.71" diff --git a/cranelift/src/libfilecheck/checker.rs b/cranelift/src/libfilecheck/checker.rs new file mode 100644 index 0000000000..8abcbfdd3d --- /dev/null +++ b/cranelift/src/libfilecheck/checker.rs @@ -0,0 +1,392 @@ +use error::{Error, Result}; +use variable::{VariableMap, Value, varname_prefix}; +use pattern::Pattern; +use regex::{Regex, Captures}; +use std::collections::HashMap; +use std::cmp::max; +use std::fmt::{self, Display, Formatter}; + +// The different kinds of directives we support. +enum Directive { + Check(Pattern), + SameLn(Pattern), + NextLn(Pattern), + Unordered(Pattern), + Not(Pattern), + Regex(String, String), +} + +// Regular expression matching a directive. +// The match groups are: +// +// 1. Keyword. +// 2. Rest of line / pattern. +// +const DIRECTIVE_RX: &'static str = r"\b(check|sameln|nextln|unordered|not|regex):\s+(.*)"; + +impl Directive { + /// Create a new directive from a `DIRECTIVE_RX` match. + fn new(caps: Captures) -> Result { + let cmd = caps.at(1).expect("group 1 must match"); + let rest = caps.at(2).expect("group 2 must match"); + + if cmd == "regex" { + return Directive::regex(rest); + } + + // All other commands are followed by a pattern. + let pat = try!(rest.parse()); + + match cmd { + "check" => Ok(Directive::Check(pat)), + "sameln" => Ok(Directive::SameLn(pat)), + "nextln" => Ok(Directive::NextLn(pat)), + "unordered" => Ok(Directive::Unordered(pat)), + "not" => { + if !pat.defs().is_empty() { + let msg = format!("can't define variables '$({}=...' in not: {}", + pat.defs()[0], + rest); + Err(Error::DuplicateDef(msg)) + } else { + Ok(Directive::Not(pat)) + } + } + _ => panic!("unexpected command {} in regex match", cmd), + } + } + + /// Create a `regex:` directive from a `VAR=...` string. + fn regex(rest: &str) -> Result { + let varlen = varname_prefix(rest); + if varlen == 0 { + return Err(Error::Syntax(format!("invalid variable name in regex: {}", rest))); + } + let var = rest[0..varlen].to_string(); + if !rest[varlen..].starts_with("=") { + return Err(Error::Syntax(format!("expected '=' after variable '{}' in regex: {}", + var, + rest))); + } + Ok(Directive::Regex(var, rest[varlen + 1..].to_string())) + } +} + + +/// Builder for constructing a `Checker` instance. +pub struct CheckerBuilder { + directives: Vec, + linerx: Regex, +} + +impl CheckerBuilder { + /// Create a new, blank `CheckerBuilder`. + pub fn new() -> CheckerBuilder { + CheckerBuilder { + directives: Vec::new(), + linerx: Regex::new(DIRECTIVE_RX).unwrap(), + } + } + + /// Add a potential directive line. + /// + /// Returns true if this is a a directive with one of the known prefixes. + /// Returns false if no known directive was found. + /// Returns an error if there is a problem with the directive. + pub fn directive(&mut self, l: &str) -> Result { + match self.linerx.captures(l) { + Some(caps) => { + self.directives.push(try!(Directive::new(caps))); + Ok(true) + } + None => Ok(false), + } + } + + /// Add multiple directives. + /// + /// The text is split into lines that are added individually as potential directives. + /// This method can be used to parse a whole test file containing multiple directives. + pub fn text(&mut self, t: &str) -> Result<&mut Self> { + for caps in self.linerx.captures_iter(t) { + self.directives.push(try!(Directive::new(caps))); + } + Ok(self) + } + + /// Get the finished `Checker`. + pub fn finish(&mut self) -> Checker { + // Move directives into the new checker, leaving `self.directives` empty and ready for + // building a new checker. + Checker::new(self.directives.split_off(0)) + } +} + +/// Verify a list of directives against a test input. +/// +/// Use a `CheckerBuilder` to construct a `Checker`. Then use the `test` method to verify the list +/// of directives against a test input. +pub struct Checker { + directives: Vec, +} + +impl Checker { + fn new(directives: Vec) -> Checker { + Checker { directives: directives } + } + + /// An empty checker contains no directives, and will match any input string. + pub fn is_empty(&self) -> bool { + self.directives.is_empty() + } + + /// Verify directives against the input text. + /// + /// This returns `true` if the text matches all the directives, `false` if it doesn't. + /// An error is only returned if there is a problem with the directives. + pub fn check(&self, text: &str, vars: &VariableMap) -> Result { + let mut state = State::new(text, vars); + + // For each pending `not:` check, store (begin-offset, regex). + let mut nots = Vec::new(); + + for dct in &self.directives { + let (pat, range) = match *dct { + Directive::Check(ref pat) => (pat, state.check()), + Directive::SameLn(ref pat) => (pat, state.sameln()), + Directive::NextLn(ref pat) => (pat, state.nextln()), + Directive::Unordered(ref pat) => (pat, state.unordered(pat)), + Directive::Not(ref pat) => { + // Resolve `not:` directives immediately to get the right variable values, but + // don't match it until we know the end of the range. + // + // The `not:` directives test the same range as `unordered:` directives. In + // particular, if they refer to defined variables, their range is restricted to + // the text following the match that defined the variable. + nots.push((state.unordered_begin(pat), try!(pat.resolve(&state)))); + continue; + } + Directive::Regex(ref var, ref rx) => { + state.vars.insert(var.clone(), + VarDef { + value: Value::Regex(rx.clone()), + offset: 0, + }); + continue; + } + }; + // Check if `pat` matches in `range`. + if let Some((match_begin, match_end)) = try!(state.match_positive(pat, range)) { + if let &Directive::Unordered(_) = dct { + // This was an unordered unordered match. + // Keep track of the largest matched position, but leave `last_ordered` alone. + state.max_match = max(state.max_match, match_end); + } else { + // Ordered match. + state.last_ordered = match_end; + state.max_match = match_end; + + // Verify any pending `not:` directives now that we know their range. + for (not_begin, rx) in nots.drain(..) { + if let Some(_) = rx.find(&text[not_begin..match_begin]) { + // Matched `not:` pattern. + // TODO: Use matched range for an error message. + return Ok(false); + } + } + } + } else { + // No match! + return Ok(false); + } + } + + // Verify any pending `not:` directives after the last ordered directive. + for (not_begin, rx) in nots.drain(..) { + if let Some(_) = rx.find(&text[not_begin..]) { + // Matched `not:` pattern. + // TODO: Use matched range for an error message. + return Ok(false); + } + } + + Ok(true) + } +} + +/// A local definition of a variable. +pub struct VarDef { + /// The value given to the variable. + value: Value, + /// Offset in input text from where the variable is available. + offset: usize, +} + +struct State<'a> { + env_vars: &'a VariableMap, + text: &'a str, + vars: HashMap, + // Offset after the last ordered match. This does not include recent unordered matches. + last_ordered: usize, + // Largest offset following a positive match, including unordered matches. + max_match: usize, +} + +impl<'a> State<'a> { + fn new(text: &'a str, env_vars: &'a VariableMap) -> State<'a> { + State { + text: text, + env_vars: env_vars, + vars: HashMap::new(), + last_ordered: 0, + max_match: 0, + } + } + + // Get the offset following the match that defined `var`, or 0 if var is an environment + // variable or unknown. + fn def_offset(&self, var: &str) -> usize { + self.vars.get(var).map(|&VarDef { offset, .. }| offset).unwrap_or(0) + } + + // Get the offset of the beginning of the next line after `pos`. + fn bol(&self, pos: usize) -> usize { + if let Some(offset) = self.text[pos..].find('\n') { + pos + offset + 1 + } else { + self.text.len() + } + } + + // Get the range in text to be matched by a `check:`. + fn check(&self) -> (usize, usize) { + (self.max_match, self.text.len()) + } + + // Get the range in text to be matched by a `sameln:`. + fn sameln(&self) -> (usize, usize) { + let b = self.max_match; + let e = self.bol(b); + (b, e) + } + + // Get the range in text to be matched by a `nextln:`. + fn nextln(&self) -> (usize, usize) { + let b = self.bol(self.max_match); + let e = self.bol(b); + (b, e) + } + + // Get the beginning of the range in text to be matched by a `unordered:` or `not:` directive. + // The unordered directive must match after the directives that define the variables used. + fn unordered_begin(&self, pat: &Pattern) -> usize { + let mut from = self.last_ordered; + for part in pat.parts() { + if let Some(var) = part.ref_var() { + from = max(from, self.def_offset(var)); + } + } + from + } + + // Get the range in text to be matched by a `unordered:` directive. + fn unordered(&self, pat: &Pattern) -> (usize, usize) { + (self.unordered_begin(pat), self.text.len()) + } + + // Search for `pat` in `range`, return the range matched. + // After a positive match, update variable definitions, if any. + fn match_positive(&mut self, + pat: &Pattern, + range: (usize, usize)) + -> Result> { + let rx = try!(pat.resolve(self)); + let txt = &self.text[range.0..range.1]; + let defs = pat.defs(); + let matched_range = if defs.is_empty() { + // Pattern defines no variables. Fastest search is `find`. + rx.find(txt) + } else { + // We need the captures to define variables. + rx.captures(txt).map(|caps| { + let matched_range = caps.pos(0).expect("whole expression must match"); + for var in defs { + let vardef = VarDef { + value: Value::Text(caps.name(var).unwrap_or("").to_string()), + // This offset is the end of the whole matched pattern, not just the text + // defining the variable. + offset: range.0 + matched_range.1, + }; + self.vars.insert(var.clone(), vardef); + } + matched_range + }) + }; + Ok(matched_range.map(|(b, e)| (range.0 + b, range.0 + e))) + } +} + +impl<'a> VariableMap for State<'a> { + fn lookup(&self, varname: &str) -> Option { + // First look for a local define. + if let Some(&VarDef { ref value, .. }) = self.vars.get(varname) { + Some(value.clone()) + } else { + // No local, maybe an environment variable? + self.env_vars.lookup(varname) + } + } +} + +impl Display for Directive { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + use self::Directive::*; + match *self { + Check(ref pat) => writeln!(f, "check: {}", pat), + SameLn(ref pat) => writeln!(f, "sameln: {}", pat), + NextLn(ref pat) => writeln!(f, "nextln: {}", pat), + Unordered(ref pat) => writeln!(f, "unordered: {}", pat), + Not(ref pat) => writeln!(f, "not: {}", pat), + Regex(ref var, ref rx) => writeln!(f, "regex: {}={}", var, rx), + } + } +} + +impl Display for Checker { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + for (idx, dir) in self.directives.iter().enumerate() { + try!(write!(f, "#{} {}", idx, dir)); + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::CheckerBuilder; + use error::Error; + + fn e2s(e: Error) -> String { + e.to_string() + } + + #[test] + fn directive() { + let mut b = CheckerBuilder::new(); + + assert_eq!(b.directive("not here: more text").map_err(e2s), Ok(false)); + assert_eq!(b.directive("not here: regex: X=more text").map_err(e2s), + Ok(true)); + assert_eq!(b.directive("regex: X = tommy").map_err(e2s), + Err("expected '=' after variable 'X' in regex: X = tommy".to_string())); + assert_eq!(b.directive("[arm]not: patt $x $(y) here").map_err(e2s), + Ok(true)); + assert_eq!(b.directive("[x86]sameln: $x $(y=[^]]*) there").map_err(e2s), + Ok(true)); + + let c = b.finish(); + assert_eq!(c.to_string(), + "#0 regex: X=more text\n#1 not: patt $(x) $(y) here\n#2 sameln: $(x) \ + $(y=[^]]*) there\n"); + } +} diff --git a/cranelift/src/libfilecheck/error.rs b/cranelift/src/libfilecheck/error.rs new file mode 100644 index 0000000000..1cc17001fc --- /dev/null +++ b/cranelift/src/libfilecheck/error.rs @@ -0,0 +1,69 @@ +use std::result; +use std::convert::From; +use std::error::Error as StdError; +use std::fmt; +use regex; + +/// A result from the filecheck library. +pub type Result = result::Result; + +/// A filecheck error. +#[derive(Debug)] +pub enum Error { + /// A syntax error in a check line. + Syntax(String), + /// A check refers to an undefined variable. + /// + /// The pattern contains `$foo` where the `foo` variable has not yet been defined. + /// Use `$$` to match a literal dollar sign. + UndefVariable(String), + /// A pattern contains a back-reference to a variable that was defined in the same pattern. + /// + /// For example, `check: Hello $(world=.*) $world`. Backreferences are not support. Often the + /// desired effect can be achieved with the `sameln` check: + /// + /// ```text + /// check: Hello $(world=[^ ]*) + /// sameln: $world + /// ``` + Backref(String), + /// A pattern contains multiple definitions of the same variable. + DuplicateDef(String), + /// An error in a regular expression. + /// + /// Use `cause()` to get the underlying `Regex` library error. + Regex(regex::Error), +} + +impl StdError for Error { + fn description(&self) -> &str { + use Error::*; + match *self { + Syntax(ref s) => s, + UndefVariable(ref s) => s, + Backref(ref s) => s, + DuplicateDef(ref s) => s, + Regex(ref err) => err.description(), + } + } + + fn cause(&self) -> Option<&StdError> { + use Error::*; + match *self { + Regex(ref err) => Some(err), + _ => None, + } + } +} + +impl fmt::Display for Error { + fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result { + write!(fmt, "{}", self.description()) + } +} + +impl From for Error { + fn from(e: regex::Error) -> Error { + Error::Regex(e) + } +} diff --git a/cranelift/src/libfilecheck/lib.rs b/cranelift/src/libfilecheck/lib.rs new file mode 100644 index 0000000000..16d6d5028d --- /dev/null +++ b/cranelift/src/libfilecheck/lib.rs @@ -0,0 +1,246 @@ +//! This crate provides a text pattern matching library with functionality similar to the LLVM +//! project's [FileCheck command](http://llvm.org/docs/CommandGuide/FileCheck.html). +//! +//! A list of directives is typically extracted from a file containing a test case. The test case +//! is then run through the program under test, and its output matched against the directives. +//! +//! See the [CheckerBuilder](struct.CheckerBuilder.html) and [Checker](struct.Checker.html) types +//! for the main library API. +//! +//! # Directives +//! +//! These are the directives recognized by *filecheck*: +//! 
+//! check: <pattern>
+//! sameln: <pattern>
+//! nextln: <pattern>
+//! unordered: <pattern>
+//! not: <pattern>
+//! regex: <variable>=<regex>
+//!
+//! Each directive is described in more detail below. +//! +//! ## Example +//! +//! The Rust program below prints the primes less than 100. It has *filecheck* directives embedded +//! in comments: +//! +//! ```rust +//! fn is_prime(x: u32) -> bool { +//! (2..x).all(|d| x % d != 0) +//! } +//! +//! // Check that we get the primes and nothing else: +//! // regex: NUM=\d+ +//! // not: $NUM +//! // check: 2 +//! // nextln: 3 +//! // check: 89 +//! // nextln: 97 +//! // not: $NUM +//! fn main() { +//! for p in (2..10).filter(|&x| is_prime(x)) { +//! println!("{}", p); +//! } +//! } +//! ``` +//! +//! A test driver compiles and runs the program, then pipes the output through *filecheck*: +//! +//! ```sh +//! $ rustc primes.rs +//! $ ./primes | cton-util filecheck -v +//! #0 regex: NUM=\d+ +//! #1 not: $NUM +//! #2 check: 2 +//! #3 nextln: 3 +//! #4 check: 89 +//! #5 nextln: 97 +//! #6 not: $NUM +//! no match #1: \d+ +//! > 2 +//! ~ +//! match #2: \b2\b +//! > 3 +//! ~ +//! match #3: \b3\b +//! > 5 +//! > 7 +//! ... +//! > 79 +//! > 83 +//! > 89 +//! ~~ +//! match #4: \b89\b +//! > 97 +//! ~~ +//! match #5: \b97\b +//! no match #6: \d+ +//! OK +//! ``` +//! +//! ## The `check:` directive +//! +//! Match patterns non-overlapping and in order: +//! +//! ```sh +//! #0 check: one +//! #1 check: two +//! ``` +//! +//! These directives will match the string `"one two"`, but not `"two one"`. The second directive +//! must match after the first one, and it can't overlap. +//! +//! ## The `sameln:` directive +//! +//! Match a pattern in the same line as the previous match. +//! +//! ```sh +//! #0 check: one +//! #1 sameln: two +//! ``` +//! +//! These directives will match the string `"one two"`, but not `"one\ntwo"`. The second match must +//! be in the same line as the first. Like the `check:` directive, the match must also follow the +//! first match, so `"two one" would not be matched. +//! +//! If there is no previous match, `sameln:` matches on the first line of the input. +//! +//! ## The `nextln:` directive +//! +//! Match a pattern in the next line after the previous match. +//! +//! ```sh +//! #0 check: one +//! #1 nextln: two +//! ``` +//! +//! These directives will match the string `"one\ntwo"`, but not `"one two"` or `"one\n\ntwo"`. +//! +//! If there is no previous match, `nextln:` matches on the second line of the input as if there +//! were a previous match on the first line. +//! +//! ## The `unordered:` directive +//! +//! Match patterns in any order, and possibly overlapping each other. +//! +//! ```sh +//! #0 unordered: one +//! #1 unordered: two +//! ``` +//! +//! These directives will match the string `"one two"` *and* the string `"two one"`. +//! +//! When a normal ordered match is inserted into a sequence of `unordered:` directives, it acts as +//! a barrier: +//! +//! ```sh +//! #0 unordered: one +//! #1 unordered: two +//! #2 check: three +//! #3 unordered: four +//! #4 unordered: five +//! ``` +//! +//! These directives will match `"two one three four five"`, but not `"two three one four five"`. +//! The `unordered:` matches are not allowed to cross the ordered `check:` directive. +//! +//! When `unordered:` matches define and use variables, a topological order is enforced. This means +//! that a match referencing a variable must follow the match where the variable was defined: +//! +//! ```sh +//! #0 regex: V=\bv\d+\b +//! #1 unordered: $(va=$V) = load +//! #2 unordered: $(vb=$V) = iadd $va +//! #3 unordered: $(vc=$V) = load +//! #4 unordered: iadd $va, $vc +//! ``` +//! +//! In the above directives, #2 must match after #1, and #4 must match after both #1 and #3, but +//! otherwise they can match in any order. +//! +//! ## The `not:` directive +//! +//! Check that a pattern *does not* appear between matches. +//! +//! ```sh +//! #0 check: one +//! #1 not: two +//! #2 check: three +//! ``` +//! +//! The directives above will match `"one five three"`, but not `"one two three"`. +//! +//! The pattern in a `not:` directive can't define any variables. Since it never matches anything, +//! the variables would not get a value. +//! +//! ## The `regex:` directive +//! +//! Define a shorthand name for a regular expression. +//! +//! ```sh +//! #0 regex: ID=\b[_a-zA-Z][_0-9a-zA-Z]*\b +//! #1 check: $ID + $ID +//! ``` +//! +//! The `regex:` directive gives a name to a regular expression which can then be used as part of a +//! pattern to match. Patterns are otherwise just plain text strings to match, so this is not +//! simple macro expansion. +//! +//! See [the Rust regex crate](../regex/index.html#syntax) for the regular expression syntax. +//! +//! # Patterns and variables +//! +//! Patterns are plain text strings to be matched in the input file. The dollar sign is used as an +//! escape character to expand variables. The following escape sequences are recognized: +//! +//! 
+//! $$                Match single dollar sign.
+//! $()               Match the empty string.
+//! $(=<regex>)       Match regular expression <regex>.
+//! $<var>            Match contents of variable <var>.
+//! $(<var>)          Match contents of variable <var>.
+//! $(<var>=<regex>)  Match <regex>, then
+//!                   define <var> as the matched text.
+//! $(<var>=$<rxvar>) Match regex in <rxvar>, then
+//!                   define <var> as the matched text.
+//!
+//! +//! Variables can contain either plain text or regular expressions. Plain text variables are +//! defined with the `$(var=...)` syntax in a previous directive. They match the same text again. +//! Backreferences within the same pattern are not allowed. When a variable is defined in a +//! pattern, it can't be referenced again in the same pattern. +//! +//! Regular expression variables are defined with the `regex:` directive. They match the regular +//! expression each time they are used, so the matches don't need to be identical. +//! +//! ## Word boundaries +//! +//! If a pattern begins or ends with a (plain text) letter or number, it will only match on a word +//! boundary. Use the `$()` empty string match to prevent this: +//! +//! ```sh +//! check: one$() +//! ``` +//! +//! This will match `"one"` and `"onetwo"`, but not `"zeroone"`. +//! +//! The empty match syntax can also be used to require leading or trailing whitespace: +//! +//! ```sh +//! check: one, $() +//! ``` +//! +//! This will match `"one, two"` , but not `"one,two"`. Without the `$()`, trailing whitespace +//! would be trimmed from the pattern. + +pub use error::{Error, Result}; +pub use variable::{VariableMap, Value, NO_VARIABLES}; +pub use checker::{Checker, CheckerBuilder}; + +extern crate regex; + +mod error; +mod variable; +mod pattern; +mod checker; diff --git a/cranelift/src/libfilecheck/pattern.rs b/cranelift/src/libfilecheck/pattern.rs new file mode 100644 index 0000000000..69464bd7c8 --- /dev/null +++ b/cranelift/src/libfilecheck/pattern.rs @@ -0,0 +1,523 @@ +//! Pattern matching for a single directive. + +use error::{Error, Result}; +use variable::{varname_prefix, VariableMap, Value}; +use std::str::FromStr; +use std::fmt::{self, Display, Formatter, Write}; +use regex::{Regex, RegexBuilder, quote}; + +/// A pattern to match as specified in a directive. +/// +/// Each pattern is broken into a sequence of parts that must match in order. The kinds of parts +/// are: +/// +/// 1. Plain text match. +/// 2. Variable match, `$FOO` or `$(FOO)`. The variable `FOO` may expand to plain text or a regex. +/// 3. Variable definition from literal regex, `$(foo=.*)`. Match the regex and assign matching text +/// to variable `foo`. +/// 4. Variable definition from regex variable, `$(foo=$RX)`. Lookup variable `RX` which should +/// expand to a regex, match the regex, and assign matching text to variable `foo`. +/// +pub struct Pattern { + parts: Vec, + // Variables defined by this pattern. + defs: Vec, +} + +/// One atomic part of a pattern. +#[derive(Debug, PartialEq, Eq)] +pub enum Part { + /// Match a plain string. + Text(String), + /// Match a regular expression. The regex has already been wrapped in a non-capturing group if + /// necessary, so it is safe to concatenate. + Regex(String), + /// Match the contents of a variable, which can be plain text or regex. + Var(String), + /// Match literal regex, then assign match to variable. + /// The regex has already been wrapped in a named capture group. + DefLit { def: usize, regex: String }, + /// Lookup variable `var`, match resulting regex, assign matching text to variable `defs[def]`. + DefVar { def: usize, var: String }, +} + +impl Part { + /// Get the variabled referenced by this part, if any. + pub fn ref_var(&self) -> Option<&str> { + match *self { + Part::Var(ref var) => Some(var), + Part::DefVar { ref var, .. } => Some(var), + _ => None, + } + } +} + +impl Pattern { + /// Create a new blank pattern. Use the `FromStr` trait to generate Patterns with content. + fn new() -> Pattern { + Pattern { + parts: Vec::new(), + defs: Vec::new(), + } + } + + /// Check if the variable `v` is defined by this pattern. + pub fn defines_var(&self, v: &str) -> bool { + self.defs.iter().any(|d| d == v) + } + + /// Add a definition of a new variable. + /// Return the allocated def number. + fn add_def(&mut self, v: &str) -> Result { + if self.defines_var(v) { + Err(Error::DuplicateDef(format!("duplicate definition of ${} in same pattern", v))) + } else { + let idx = self.defs.len(); + self.defs.push(v.to_string()); + Ok(idx) + } + } + + /// Parse a `Part` from a prefix of `s`. + /// Return the part and the number of bytes consumed from `s`. + /// Adds defined variables to `self.defs`. + fn parse_part(&mut self, s: &str) -> Result<(Part, usize)> { + let dollar = s.find('$'); + if dollar != Some(0) { + // String doesn't begin with a dollar sign, so match plain text up to the dollar sign. + let end = dollar.unwrap_or(s.len()); + return Ok((Part::Text(s[0..end].to_string()), end)); + } + + // String starts with a dollar sign. Look for these possibilities: + // + // 1. `$$`. + // 2. `$var`. + // 3. `$(var)`. + // 4. `$(var=regex)`. Where `regex` is a regular expression possibly containing matching + // braces. + // 5. `$(var=$VAR)`. + + // A doubled dollar sign matches a single dollar sign. + if s.starts_with("$$") { + return Ok((Part::Text("$".to_string()), 2)); + } + + // Look for `$var`. + let varname_end = 1 + varname_prefix(&s[1..]); + if varname_end != 1 { + return Ok((Part::Var(s[1..varname_end].to_string()), varname_end)); + } + + // All remaining possibilities start with `$(`. + if s.len() < 2 || !s.starts_with("$(") { + return Err(Error::Syntax("pattern syntax error, use $$ to match a single $" + .to_string())); + } + + // Match the variable name, allowing for an empty varname in `$()`, or `$(=...)`. + let varname_end = 2 + varname_prefix(&s[2..]); + let varname = s[2..varname_end].to_string(); + + match s[varname_end..].chars().next() { + None => { + return Err(Error::Syntax(format!("unterminated $({}...", varname))); + } + Some(')') => { + let part = if varname.is_empty() { + // Match `$()`, turn it into an empty text match. + Part::Text(varname) + } else { + // Match `$(var)`. + Part::Var(varname) + }; + return Ok((part, varname_end + 1)); + } + Some('=') => { + // Variable definition. Fall through. + } + Some(ch) => { + return Err(Error::Syntax(format!("syntax error in $({}... '{}'", varname, ch))); + } + } + + // This is a variable definition of the form `$(var=...`. + + // Allocate a definition index. + let def = if varname.is_empty() { + None + } else { + Some(try!(self.add_def(&varname))) + }; + + // Match `$(var=$PAT)`. + if s[varname_end + 1..].starts_with('$') { + let refname_begin = varname_end + 2; + let refname_end = refname_begin + varname_prefix(&s[refname_begin..]); + if refname_begin == refname_end { + return Err(Error::Syntax(format!("expected variable name in $({}=$...", varname))); + } + if !s[refname_end..].starts_with(')') { + return Err(Error::Syntax(format!("expected ')' after $({}=${}...", + varname, + &s[refname_begin..refname_end]))); + } + let refname = s[refname_begin..refname_end].to_string(); + return if let Some(defidx) = def { + Ok((Part::DefVar { + def: defidx, + var: refname, + }, + refname_end + 1)) + } else { + Err(Error::Syntax(format!("expected variable name in $(=${})", refname))) + }; + } + + // Last case: `$(var=...)` where `...` is a regular expression, possibly containing matched + // parentheses. + let rx_begin = varname_end + 1; + let rx_end = rx_begin + regex_prefix(&s[rx_begin..]); + if s[rx_end..].starts_with(')') { + let part = if let Some(defidx) = def { + // Wrap the regex in a named capture group. + Part::DefLit { + def: defidx, + regex: format!("(?P<{}>{})", varname, &s[rx_begin..rx_end]), + } + } else { + // When the varname is empty just match the regex, don't capture any variables. + // This is `$(=[a-z])`. + // Wrap the regex in a non-capturing group to make it concatenation-safe. + Part::Regex(format!("(?:{})", &s[rx_begin..rx_end])) + }; + Ok((part, rx_end + 1)) + } else { + Err(Error::Syntax(format!("missing ')' after regex in $({}={}", + varname, + &s[rx_begin..rx_end]))) + } + } +} + +/// Compute the length of a regular expression terminated by `)` or `}`. +/// Handle nested and escaped parentheses in the rx, but don't actualy parse it. +/// Return the position of the terminating brace or the length of the string. +fn regex_prefix(s: &str) -> usize { + // The prevous char was a backslash. + let mut escape = false; + // State around parsing charsets. + enum State { + Normal, // Outside any charset. + Curly, // Inside curly braces. + CSFirst, // Immediately after opening `[`. + CSNeg, // Immediately after `[^`. + CSBody, // Inside `[...`. + } + let mut state = State::Normal; + + // Current nesting level of parens. + let mut nest = 0usize; + + for (idx, ch) in s.char_indices() { + if escape { + escape = false; + continue; + } else if ch == '\\' { + escape = true; + continue; + } + match state { + State::Normal => { + match ch { + '[' => state = State::CSFirst, + '{' => state = State::Curly, + '(' => nest += 1, + ')' if nest > 0 => nest -= 1, + ')' | '}' => return idx, + _ => {} + } + } + State::Curly => { + if ch == '}' { + state = State::Normal; + } + } + State::CSFirst => { + state = match ch { + '^' => State::CSNeg, + _ => State::CSBody, + } + } + State::CSNeg => state = State::CSBody, + State::CSBody => { + if ch == ']' { + state = State::Normal; + } + } + } + } + s.len() +} + +impl FromStr for Pattern { + type Err = Error; + + fn from_str(s: &str) -> Result { + // Always remove leading and trailing whitespace. + // Use `$()` to actually include that in a match. + let s = s.trim(); + let mut pat = Pattern::new(); + let mut pos = 0; + while pos < s.len() { + let (part, len) = try!(pat.parse_part(&s[pos..])); + if let Some(v) = part.ref_var() { + if pat.defines_var(v) { + return Err(Error::Backref(format!("unsupported back-reference to '${}' \ + defined in same pattern", + v))); + } + } + pat.parts.push(part); + pos += len; + } + Ok(pat) + } +} + +impl Pattern { + /// Get a list of parts in this pattern. + pub fn parts(&self) -> &[Part] { + &self.parts + } + + /// Get a list of variable names defined when this pattern matches. + pub fn defs(&self) -> &[String] { + &self.defs + } + + /// Resolve all variable references in this pattern, turning it into a regular expression. + pub fn resolve(&self, vmap: &VariableMap) -> Result { + let mut out = String::new(); + + // Add a word boundary check `\b` to the beginning of the regex, but only if the first part + // is a plain text match that starts with a word character. + // + // This behavior can be disabled by starting the pattern with `$()`. + if let Some(&Part::Text(ref s)) = self.parts.first() { + if s.starts_with(char::is_alphanumeric) { + out.push_str(r"\b"); + } + } + + for part in &self.parts { + match *part { + Part::Text(ref s) => { + out.push_str("e(s)); + } + Part::Regex(ref rx) => out.push_str(rx), + Part::Var(ref var) => { + // Resolve the variable. We can handle a plain text expansion. + match vmap.lookup(var) { + None => { + return Err(Error::UndefVariable(format!("undefined variable ${}", var))) + } + Some(Value::Text(s)) => out.push_str("e(&s)), + // Wrap regex in non-capturing group for safe concatenation. + Some(Value::Regex(rx)) => write!(out, "(?:{})", rx).unwrap(), + } + } + Part::DefLit { ref regex, .. } => out.push_str(regex), + Part::DefVar { def, ref var } => { + // Wrap regex in a named capture group. + write!(out, + "(?P<{}>{})", + self.defs[def], + match vmap.lookup(var) { + None => { + return Err(Error::UndefVariable(format!("undefined variable \ + ${}", + var))) + } + Some(Value::Text(s)) => quote(&s), + Some(Value::Regex(rx)) => rx, + }) + .unwrap() + } + } + + } + + // Add a word boundary check `\b` to the end of the regex, but only if the final part + // is a plain text match that ends with a word character. + // + // This behavior can be disabled by ending the pattern with `$()`. + if let Some(&Part::Text(ref s)) = self.parts.last() { + if s.ends_with(char::is_alphanumeric) { + out.push_str(r"\b"); + } + } + + Ok(try!(RegexBuilder::new(&out).multi_line(true).compile())) + } +} + +impl Display for Pattern { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + for part in &self.parts { + use self::Part::*; + try!(match *part { + Text(ref txt) if txt == "" => write!(f, "$()"), + Text(ref txt) if txt == "$" => write!(f, "$$"), + Text(ref txt) => write!(f, "{}", txt), + Regex(ref rx) => write!(f, "$(={})", rx), + Var(ref var) => write!(f, "$({})", var), + DefLit { def, ref regex } => { + let defvar = &self.defs[def]; + // (?P...). + let litrx = ®ex[5 + defvar.len()..regex.len() - 1]; + write!(f, "$({}={})", defvar, litrx) + } + DefVar { def, ref var } => write!(f, "$({}=${})", self.defs[def], var), + }); + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + #[test] + fn regex() { + use super::regex_prefix; + + assert_eq!(regex_prefix(""), 0); + assert_eq!(regex_prefix(")"), 0); + assert_eq!(regex_prefix(")c"), 0); + assert_eq!(regex_prefix("x"), 1); + assert_eq!(regex_prefix("x)x"), 1); + + assert_eq!(regex_prefix("x(c))x"), 4); + assert_eq!(regex_prefix("()x(c))x"), 6); + assert_eq!(regex_prefix("()x(c)"), 6); + + assert_eq!(regex_prefix("x([)]))x"), 6); + assert_eq!(regex_prefix("x[)])x"), 4); + assert_eq!(regex_prefix("x[^)])x"), 5); + assert_eq!(regex_prefix("x[^])x"), 6); + } + + #[test] + fn part() { + use super::{Pattern, Part}; + let mut pat = Pattern::new(); + + // This is dubious, should we panic instead? + assert_eq!(pat.parse_part("").unwrap(), (Part::Text("".to_string()), 0)); + + assert_eq!(pat.parse_part("x").unwrap(), + (Part::Text("x".to_string()), 1)); + assert_eq!(pat.parse_part("x2").unwrap(), + (Part::Text("x2".to_string()), 2)); + assert_eq!(pat.parse_part("x$").unwrap(), + (Part::Text("x".to_string()), 1)); + assert_eq!(pat.parse_part("x$$").unwrap(), + (Part::Text("x".to_string()), 1)); + + assert_eq!(pat.parse_part("$").unwrap_err().to_string(), + "pattern syntax error, use $$ to match a single $"); + + assert_eq!(pat.parse_part("$$").unwrap(), + (Part::Text("$".to_string()), 2)); + assert_eq!(pat.parse_part("$$ ").unwrap(), + (Part::Text("$".to_string()), 2)); + + assert_eq!(pat.parse_part("$0").unwrap(), + (Part::Var("0".to_string()), 2)); + assert_eq!(pat.parse_part("$xx=").unwrap(), + (Part::Var("xx".to_string()), 3)); + assert_eq!(pat.parse_part("$xx$").unwrap(), + (Part::Var("xx".to_string()), 3)); + + assert_eq!(pat.parse_part("$(0)").unwrap(), + (Part::Var("0".to_string()), 4)); + assert_eq!(pat.parse_part("$()").unwrap(), + (Part::Text("".to_string()), 3)); + + assert_eq!(pat.parse_part("$(0").unwrap_err().to_string(), + ("unterminated $(0...")); + assert_eq!(pat.parse_part("$(foo:").unwrap_err().to_string(), + ("syntax error in $(foo... ':'")); + assert_eq!(pat.parse_part("$(foo =").unwrap_err().to_string(), + ("syntax error in $(foo... ' '")); + assert_eq!(pat.parse_part("$(eo0=$bar").unwrap_err().to_string(), + ("expected ')' after $(eo0=$bar...")); + assert_eq!(pat.parse_part("$(eo1=$bar}").unwrap_err().to_string(), + ("expected ')' after $(eo1=$bar...")); + assert_eq!(pat.parse_part("$(eo2=$)").unwrap_err().to_string(), + ("expected variable name in $(eo2=$...")); + assert_eq!(pat.parse_part("$(eo3=$-)").unwrap_err().to_string(), + ("expected variable name in $(eo3=$...")); + } + + #[test] + fn partdefs() { + use super::{Pattern, Part}; + let mut pat = Pattern::new(); + + assert_eq!(pat.parse_part("$(foo=$bar)").unwrap(), + (Part::DefVar { + def: 0, + var: "bar".to_string(), + }, + 11)); + assert_eq!(pat.parse_part("$(foo=$bar)").unwrap_err().to_string(), + "duplicate definition of $foo in same pattern"); + + assert_eq!(pat.parse_part("$(fxo=$bar)x").unwrap(), + (Part::DefVar { + def: 1, + var: "bar".to_string(), + }, + 11)); + + assert_eq!(pat.parse_part("$(fo2=[a-z])").unwrap(), + (Part::DefLit { + def: 2, + regex: "(?P[a-z])".to_string(), + }, + 12)); + assert_eq!(pat.parse_part("$(fo3=[a-)])").unwrap(), + (Part::DefLit { + def: 3, + regex: "(?P[a-)])".to_string(), + }, + 12)); + assert_eq!(pat.parse_part("$(fo4=)").unwrap(), + (Part::DefLit { + def: 4, + regex: "(?P)".to_string(), + }, + 7)); + + assert_eq!(pat.parse_part("$(=.*)").unwrap(), + (Part::Regex("(?:.*)".to_string()), 6)); + + assert_eq!(pat.parse_part("$(=)").unwrap(), + (Part::Regex("(?:)".to_string()), 4)); + assert_eq!(pat.parse_part("$()").unwrap(), + (Part::Text("".to_string()), 3)); + } + + #[test] + fn pattern() { + use super::Pattern; + + let p: Pattern = " Hello world! ".parse().unwrap(); + assert_eq!(format!("{:?}", p.parts), "[Text(\"Hello world!\")]"); + + let p: Pattern = " $foo=$(bar) ".parse().unwrap(); + assert_eq!(format!("{:?}", p.parts), + "[Var(\"foo\"), Text(\"=\"), Var(\"bar\")]"); + } +} diff --git a/cranelift/src/libfilecheck/tests/basic.rs b/cranelift/src/libfilecheck/tests/basic.rs new file mode 100644 index 0000000000..595158192c --- /dev/null +++ b/cranelift/src/libfilecheck/tests/basic.rs @@ -0,0 +1,313 @@ +extern crate filecheck; + +use filecheck::{CheckerBuilder, NO_VARIABLES, Error as FcError}; + +fn e2s(e: FcError) -> String { + e.to_string() +} + +#[test] +fn empty() { + let c = CheckerBuilder::new().finish(); + assert!(c.is_empty()); + + // An empty checker matches anything. + assert_eq!(c.check("", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("hello", NO_VARIABLES).map_err(e2s), Ok(true)); +} + +#[test] +fn no_directives() { + let c = CheckerBuilder::new().text("nothing here").unwrap().finish(); + assert!(c.is_empty()); + + // An empty checker matches anything. + assert_eq!(c.check("", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("hello", NO_VARIABLES).map_err(e2s), Ok(true)); +} + +#[test] +fn no_matches() { + let c = CheckerBuilder::new().text("regex: FOO=bar").unwrap().finish(); + assert!(!c.is_empty()); + + // An empty checker matches anything. + assert_eq!(c.check("", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("hello", NO_VARIABLES).map_err(e2s), Ok(true)); +} + +#[test] +fn simple() { + let c = CheckerBuilder::new() + .text(" + check: one + check: two + ") + .unwrap() + .finish(); + + let t = " + zero + one + and a half + two + three + "; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); + + let t = " + zero + and a half + two + one + three + "; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); +} + +#[test] +fn sameln() { + let c = CheckerBuilder::new() + .text(" + check: one + sameln: two + ") + .unwrap() + .finish(); + + let t = " + zero + one + and a half + two + three + "; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); + + let t = " + zero + one + two + three + "; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); + + let t = " + zero + one two + three + "; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); +} + +#[test] +fn nextln() { + let c = CheckerBuilder::new() + .text(" + check: one + nextln: two + ") + .unwrap() + .finish(); + + let t = " + zero + one + and a half + two + three + "; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); + + let t = " + zero + one + two + three + "; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); + + let t = " + zero + one two + three + "; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); + + let t = " + zero + one + two"; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); +} + +#[test] +fn leading_nextln() { + // A leading nextln directive should match from line 2. + // This is somewhat arbitrary, but consistent with a preceeding 'check: $()' directive. + let c = CheckerBuilder::new() + .text(" + nextln: one + nextln: two + ") + .unwrap() + .finish(); + + let t = "zero + one + two + three + "; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); + + let t = "one + two + three + "; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); +} + +#[test] +fn leading_sameln() { + // A leading sameln directive should match from line 1. + let c = CheckerBuilder::new() + .text(" + sameln: one + sameln: two + ") + .unwrap() + .finish(); + + let t = "zero + one two three + "; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); + + let t = "zero one two three"; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); + + let t = "zero one + two three"; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); +} + +#[test] +fn not() { + let c = CheckerBuilder::new() + .text(" + check: one$() + not: $()eat$() + check: $()two + ") + .unwrap() + .finish(); + + let t = "onetwo"; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); + + let t = "one eat two"; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); + + let t = "oneeattwo"; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); + + let t = "oneatwo"; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); +} + +#[test] +fn notnot() { + let c = CheckerBuilder::new() + .text(" + check: one$() + not: $()eat$() + not: half + check: $()two + ") + .unwrap() + .finish(); + + let t = "onetwo"; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); + + let t = "one eat two"; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); + + let t = "one half two"; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); + + let t = "oneeattwo"; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); + + // The `not: half` pattern only matches whole words, but the bracketing matches are considered + // word boundaries, so it does match in this case. + let t = "onehalftwo"; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); + + let t = "oneatwo"; + assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); +} + +#[test] +fn unordered() { + let c = CheckerBuilder::new() + .text(" + check: one + unordered: two + unordered: three + check: four + ") + .unwrap() + .finish(); + + assert_eq!(c.check("one two three four", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one three two four", NO_VARIABLES).map_err(e2s), Ok(true)); + + assert_eq!(c.check("one two four three four", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one three four two four", NO_VARIABLES).map_err(e2s), Ok(true)); + + assert_eq!(c.check("one two four three", NO_VARIABLES).map_err(e2s), Ok(false)); + assert_eq!(c.check("one three four two", NO_VARIABLES).map_err(e2s), Ok(false)); +} + +#[test] +fn leading_unordered() { + let c = CheckerBuilder::new() + .text(" + unordered: two + unordered: three + check: four + ") + .unwrap() + .finish(); + + assert_eq!(c.check("one two three four", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one three two four", NO_VARIABLES).map_err(e2s), Ok(true)); + + assert_eq!(c.check("one two four three four", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one three four two four", NO_VARIABLES).map_err(e2s), Ok(true)); + + assert_eq!(c.check("one two four three", NO_VARIABLES).map_err(e2s), Ok(false)); + assert_eq!(c.check("one three four two", NO_VARIABLES).map_err(e2s), Ok(false)); +} + +#[test] +fn trailing_unordered() { + let c = CheckerBuilder::new() + .text(" + check: one + unordered: two + unordered: three + ") + .unwrap() + .finish(); + + assert_eq!(c.check("one two three four", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one three two four", NO_VARIABLES).map_err(e2s), Ok(true)); + + assert_eq!(c.check("one two four three four", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one three four two four", NO_VARIABLES).map_err(e2s), Ok(true)); + + assert_eq!(c.check("one two four three", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one three four two", NO_VARIABLES).map_err(e2s), Ok(true)); +} diff --git a/cranelift/src/libfilecheck/variable.rs b/cranelift/src/libfilecheck/variable.rs new file mode 100644 index 0000000000..7238249146 --- /dev/null +++ b/cranelift/src/libfilecheck/variable.rs @@ -0,0 +1,58 @@ +/// A variable name is one or more ASCII alphanumerical characters, including underscore. +/// Note that numerical variable names like `$45` are allowed too. +/// +/// Try to parse a variable name from the begining of `s`. +/// Return the index of the character following the varname. +/// This returns 0 if `s` doesn't have a prefix that is a variable name. +pub fn varname_prefix(s: &str) -> usize { + for (idx, ch) in s.char_indices() { + match ch { + 'a'...'z' | 'A'...'Z' | '0'...'9' | '_' => {} + _ => return idx, + } + } + s.len() +} + +/// A variable can contain either a regular expression or plain text. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Value { + Text(String), + Regex(String), +} + +/// Resolve variables by name. +pub trait VariableMap { + /// Get the value of the variable `varname`, or return `None` for an unknown variable name. + fn lookup(&self, varname: &str) -> Option; +} + +impl VariableMap for () { + fn lookup(&self, _: &str) -> Option { + None + } +} + +/// An empty variable map. +pub const NO_VARIABLES: &'static VariableMap = &(); + +#[cfg(test)] +mod tests { + #[test] + fn varname() { + use super::varname_prefix; + + assert_eq!(varname_prefix(""), 0); + assert_eq!(varname_prefix("\0"), 0); + assert_eq!(varname_prefix("_"), 1); + assert_eq!(varname_prefix("0"), 1); + assert_eq!(varname_prefix("01"), 2); + assert_eq!(varname_prefix("b"), 1); + assert_eq!(varname_prefix("C"), 1); + assert_eq!(varname_prefix("."), 0); + assert_eq!(varname_prefix(".s"), 0); + assert_eq!(varname_prefix("0."), 1); + assert_eq!(varname_prefix("01="), 2); + assert_eq!(varname_prefix("0a)"), 2); + } +} From 42cdebd508815db3c650e7a6d6582bf5806d9590 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 8 Sep 2016 16:20:06 -0700 Subject: [PATCH 0261/3084] Add a cton-util filecheck sub-command. --- cranelift/src/tools/Cargo.lock | 8 ++++++ cranelift/src/tools/Cargo.toml | 1 + cranelift/src/tools/main.rs | 12 +++++++-- cranelift/src/tools/rsfilecheck.rs | 40 ++++++++++++++++++++++++++++++ cranelift/test-all.sh | 2 +- 5 files changed, 60 insertions(+), 3 deletions(-) create mode 100644 cranelift/src/tools/rsfilecheck.rs diff --git a/cranelift/src/tools/Cargo.lock b/cranelift/src/tools/Cargo.lock index 251f074c32..2aca3c591b 100644 --- a/cranelift/src/tools/Cargo.lock +++ b/cranelift/src/tools/Cargo.lock @@ -5,6 +5,7 @@ dependencies = [ "cretonne 0.0.0", "cretonne-reader 0.0.0", "docopt 0.6.80 (registry+https://github.com/rust-lang/crates.io-index)", + "filecheck 0.0.0", "glob 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)", "regex 0.1.71 (registry+https://github.com/rust-lang/crates.io-index)", "rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)", @@ -42,6 +43,13 @@ dependencies = [ "strsim 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)", ] +[[package]] +name = "filecheck" +version = "0.0.0" +dependencies = [ + "regex 0.1.71 (registry+https://github.com/rust-lang/crates.io-index)", +] + [[package]] name = "glob" version = "0.2.11" diff --git a/cranelift/src/tools/Cargo.toml b/cranelift/src/tools/Cargo.toml index 8164a4e4bb..ba465f7d0e 100644 --- a/cranelift/src/tools/Cargo.toml +++ b/cranelift/src/tools/Cargo.toml @@ -12,6 +12,7 @@ path = "main.rs" [dependencies] cretonne = { path = "../libcretonne" } cretonne-reader = { path = "../libreader" } +filecheck = { path = "../libfilecheck" } docopt = "0.6.80" rustc-serialize = "0.3.19" regex = "0.1.71" diff --git a/cranelift/src/tools/main.rs b/cranelift/src/tools/main.rs index abad701af6..bd0999cd4e 100644 --- a/cranelift/src/tools/main.rs +++ b/cranelift/src/tools/main.rs @@ -3,6 +3,7 @@ extern crate cretonne; extern crate cton_reader; extern crate docopt; extern crate rustc_serialize; +extern crate filecheck; use cretonne::VERSION; use docopt::Docopt; @@ -12,26 +13,31 @@ use std::process; mod cat; mod print_cfg; +mod rsfilecheck; const USAGE: &'static str = " Cretonne code generator utility Usage: cton-util cat ... + cton-util filecheck [-v] cton-util print-cfg ... cton-util --help | --version Options: - -h, --help print this help message - --version print the Cretonne version + -v, --verbose be more verbose + -h, --help print this help message + --version print the Cretonne version "; #[derive(RustcDecodable, Debug)] struct Args { cmd_cat: bool, + cmd_filecheck: bool, cmd_print_cfg: bool, arg_file: Vec, + flag_verbose: bool, } /// A command either succeeds or fails with an error message. @@ -51,6 +57,8 @@ fn cton_util() -> CommandResult { // Find the sub-command to execute. if args.cmd_cat { cat::run(args.arg_file) + } else if args.cmd_filecheck { + rsfilecheck::run(args.arg_file, args.flag_verbose) } else if args.cmd_print_cfg { print_cfg::run(args.arg_file) } else { diff --git a/cranelift/src/tools/rsfilecheck.rs b/cranelift/src/tools/rsfilecheck.rs new file mode 100644 index 0000000000..bce661282c --- /dev/null +++ b/cranelift/src/tools/rsfilecheck.rs @@ -0,0 +1,40 @@ +use CommandResult; +use filecheck::{CheckerBuilder, Checker, NO_VARIABLES}; +use std::fs::File; +use std::io::{self, Read}; + +pub fn run(files: Vec, verbose: bool) -> CommandResult { + if files.is_empty() { + return Err("No check files".to_string()); + } + let checker = try!(read_checkfile(&files[0])); + if checker.is_empty() { + return Err(format!("{}: no filecheck directives found", files[0])); + } + + // Print out the directives under --verbose. + if verbose { + println!("{}", checker); + } + + let mut buffer = String::new(); + try!(io::stdin().read_to_string(&mut buffer).map_err(|e| format!("stdin: {}", e))); + + if try!(checker.check(&buffer, NO_VARIABLES).map_err(|e| e.to_string())) { + Ok(()) + } else { + // TODO: We need to do better than this. + Err("Check failed".to_string()) + } +} + +fn read_checkfile(filename: &str) -> Result { + let mut file = try!(File::open(&filename).map_err(|e| format!("{}: {}", filename, e))); + let mut buffer = String::new(); + try!(file.read_to_string(&mut buffer) + .map_err(|e| format!("Couldn't read {}: {}", filename, e))); + + let mut builder = CheckerBuilder::new(); + try!(builder.text(&buffer).map_err(|e| format!("{}: {}", filename, e))); + Ok(builder.finish()) +} diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 64fb83d3a8..cd3916d8c3 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -40,7 +40,7 @@ else echo "If a newer version of rustfmt is available, update this script." fi -PKGS="cretonne cretonne-reader cretonne-tools" +PKGS="cretonne cretonne-reader cretonne-tools filecheck" cd "$topdir/src/tools" for PKG in $PKGS do From 9772398ae455b9dacf0c197729eb9432bb0846fa Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 9 Sep 2016 11:09:54 -0700 Subject: [PATCH 0262/3084] Convert the DFG tests to use filecheck. --- cranelift/tests/cfg/loop.cton | 7 ++++--- cranelift/tests/cfg/run.sh | 4 +--- cranelift/tests/cfg/traps_early.cton | 7 ++++--- cranelift/tests/cfg/unused_node.cton | 8 ++++++-- 4 files changed, 15 insertions(+), 11 deletions(-) diff --git a/cranelift/tests/cfg/loop.cton b/cranelift/tests/cfg/loop.cton index 73ad1c4b6a..07d93926d8 100644 --- a/cranelift/tests/cfg/loop.cton +++ b/cranelift/tests/cfg/loop.cton @@ -1,12 +1,13 @@ ; For testing cfg generation. This code is nonsense. function nonsense(i32, i32) -> f32 { +; check: digraph nonsense { ebb0(v1: i32, v2: i32): v3 = f64const 0x0.0 - brz v2, ebb2 ;;;; ebb0:inst1 -> ebb2 + brz v2, ebb2 ; unordered: ebb0:inst1 -> ebb2 v4 = iconst.i32 0 - jump ebb1(v4) ;;;; ebb0:inst3 -> ebb1 + jump ebb1(v4) ; unordered: ebb0:inst3 -> ebb1 ebb1(v5: i32): v6 = imul_imm v5, 4 @@ -17,7 +18,7 @@ ebb1(v5: i32): v11 = fadd v9, v10 v12 = iadd_imm v5, 1 v13 = icmp ult, v12, v2 - brnz v13, ebb1(v12) ;;;; ebb1:inst12 -> ebb1 + brnz v13, ebb1(v12) ; unordered: ebb1:inst12 -> ebb1 v14 = f64const 0.0 v15 = f64const 0.0 v16 = fdiv v14, v15 diff --git a/cranelift/tests/cfg/run.sh b/cranelift/tests/cfg/run.sh index 2230d4c1ca..cd324cc73f 100755 --- a/cranelift/tests/cfg/run.sh +++ b/cranelift/tests/cfg/run.sh @@ -16,9 +16,7 @@ fi declare -a fails for testcase in $(find cfg -name '*.cton'); do - annotations=$(cat $testcase | awk /';;;;'/ | awk -F ";;;;" '{print $2}' | sort) - connections=$("${CTONUTIL}" print-cfg "$testcase" | awk /"->"/ | sort) - if diff -u <(echo $annotations) <(echo $connections); then + if "${CTONUTIL}" print-cfg "$testcase" | "${CTONUTIL}" filecheck "$testcase"; then echo OK $testcase else fails=(${fails[@]} "$testcase") diff --git a/cranelift/tests/cfg/traps_early.cton b/cranelift/tests/cfg/traps_early.cton index 6993cc128e..71070f4b15 100644 --- a/cranelift/tests/cfg/traps_early.cton +++ b/cranelift/tests/cfg/traps_early.cton @@ -2,16 +2,17 @@ ; a terminating instruction before any connections have been made. function nonsense(i32) { +; check: digraph nonsense { ebb0(v1: i32): trap - brnz v1, ebb2 ;;;; ebb0:inst1 -> ebb2 - jump ebb1 ;;;; ebb0:inst2 -> ebb1 + brnz v1, ebb2 ; unordered: ebb0:inst1 -> ebb2 + jump ebb1 ; unordered: ebb0:inst2 -> ebb1 ebb1: v2 = iconst.i32 0 v3 = iadd v1, v3 - jump ebb0(v3) ;;;; ebb1:inst5 -> ebb0 + jump ebb0(v3) ; unordered: ebb1:inst5 -> ebb0 ebb2: return v1 diff --git a/cranelift/tests/cfg/unused_node.cton b/cranelift/tests/cfg/unused_node.cton index 3120ffbe68..693196ccde 100644 --- a/cranelift/tests/cfg/unused_node.cton +++ b/cranelift/tests/cfg/unused_node.cton @@ -1,15 +1,19 @@ ; For testing cfg generation where some block is never reached. function not_reached(i32) -> i32 { +; check: digraph not_reached { +; check: ebb0 [shape=record, label="{ebb0 | brnz ebb2}"] +; check: ebb1 [shape=record, label="{ebb1 | jump ebb0}"] +; check: ebb2 [shape=record, label="{ebb2}"] ebb0(v0: i32): - brnz v0, ebb2 ;;;; ebb0:inst0 -> ebb2 + brnz v0, ebb2 ; unordered: ebb0:inst0 -> ebb2 trap ebb1: v1 = iconst.i32 1 v2 = iadd v0, v1 - jump ebb0(v2) ;;;; ebb1:inst4 -> ebb0 + jump ebb0(v2) ; unordered: ebb1:inst4 -> ebb0 ebb2: return v0 From d9cceb18d33380057b4abf6bb865d93c3d3fdda3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 9 Sep 2016 14:11:56 -0700 Subject: [PATCH 0263/3084] Add a MatchRange type alias. The regex library also uses (usize, usize) for ranges. The type alias is just to make it clearer what the tuple means. --- cranelift/src/libfilecheck/checker.rs | 14 ++++++-------- cranelift/src/libfilecheck/lib.rs | 3 +++ 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/cranelift/src/libfilecheck/checker.rs b/cranelift/src/libfilecheck/checker.rs index 8abcbfdd3d..ea23f9e0d9 100644 --- a/cranelift/src/libfilecheck/checker.rs +++ b/cranelift/src/libfilecheck/checker.rs @@ -5,6 +5,7 @@ use regex::{Regex, Captures}; use std::collections::HashMap; use std::cmp::max; use std::fmt::{self, Display, Formatter}; +use MatchRange; // The different kinds of directives we support. enum Directive { @@ -259,19 +260,19 @@ impl<'a> State<'a> { } // Get the range in text to be matched by a `check:`. - fn check(&self) -> (usize, usize) { + fn check(&self) -> MatchRange { (self.max_match, self.text.len()) } // Get the range in text to be matched by a `sameln:`. - fn sameln(&self) -> (usize, usize) { + fn sameln(&self) -> MatchRange { let b = self.max_match; let e = self.bol(b); (b, e) } // Get the range in text to be matched by a `nextln:`. - fn nextln(&self) -> (usize, usize) { + fn nextln(&self) -> MatchRange { let b = self.bol(self.max_match); let e = self.bol(b); (b, e) @@ -290,16 +291,13 @@ impl<'a> State<'a> { } // Get the range in text to be matched by a `unordered:` directive. - fn unordered(&self, pat: &Pattern) -> (usize, usize) { + fn unordered(&self, pat: &Pattern) -> MatchRange { (self.unordered_begin(pat), self.text.len()) } // Search for `pat` in `range`, return the range matched. // After a positive match, update variable definitions, if any. - fn match_positive(&mut self, - pat: &Pattern, - range: (usize, usize)) - -> Result> { + fn match_positive(&mut self, pat: &Pattern, range: MatchRange) -> Result> { let rx = try!(pat.resolve(self)); let txt = &self.text[range.0..range.1]; let defs = pat.defs(); diff --git a/cranelift/src/libfilecheck/lib.rs b/cranelift/src/libfilecheck/lib.rs index 16d6d5028d..c38efe0890 100644 --- a/cranelift/src/libfilecheck/lib.rs +++ b/cranelift/src/libfilecheck/lib.rs @@ -244,3 +244,6 @@ mod error; mod variable; mod pattern; mod checker; + +/// The range of a match in the input text. +type MatchRange = (usize, usize); From 1ce5bc3509f85b6c8fe9c10dcc2299cf0122cdb8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 9 Sep 2016 14:32:07 -0700 Subject: [PATCH 0264/3084] pub --- cranelift/src/libfilecheck/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/src/libfilecheck/lib.rs b/cranelift/src/libfilecheck/lib.rs index c38efe0890..8d61a1dfbf 100644 --- a/cranelift/src/libfilecheck/lib.rs +++ b/cranelift/src/libfilecheck/lib.rs @@ -246,4 +246,4 @@ mod pattern; mod checker; /// The range of a match in the input text. -type MatchRange = (usize, usize); +pub type MatchRange = (usize, usize); From 38952eea0070ab65ac46764b3dcca68195fa737f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 9 Sep 2016 17:08:29 -0700 Subject: [PATCH 0265/3084] Add an explainer mode to filecheck. The -c flag to 'cton-util filecheck' will now print out a description of how the directives are matching the input. This explanation is also printed when a match fails. --- cranelift/src/libfilecheck/checker.rs | 52 +++++-- cranelift/src/libfilecheck/explain.rs | 196 ++++++++++++++++++++++++++ cranelift/src/libfilecheck/lib.rs | 1 + cranelift/src/tools/rsfilecheck.rs | 15 +- cranelift/tests/cfg/loop.cton | 6 +- 5 files changed, 255 insertions(+), 15 deletions(-) create mode 100644 cranelift/src/libfilecheck/explain.rs diff --git a/cranelift/src/libfilecheck/checker.rs b/cranelift/src/libfilecheck/checker.rs index ea23f9e0d9..eba5da78a7 100644 --- a/cranelift/src/libfilecheck/checker.rs +++ b/cranelift/src/libfilecheck/checker.rs @@ -6,6 +6,7 @@ use std::collections::HashMap; use std::cmp::max; use std::fmt::{self, Display, Formatter}; use MatchRange; +use explain::{Recorder, Explainer}; // The different kinds of directives we support. enum Directive { @@ -146,12 +147,24 @@ impl Checker { /// This returns `true` if the text matches all the directives, `false` if it doesn't. /// An error is only returned if there is a problem with the directives. pub fn check(&self, text: &str, vars: &VariableMap) -> Result { - let mut state = State::new(text, vars); + self.run(text, vars, &mut ()) + } + + /// Explain how directives are matched against the input text. + pub fn explain(&self, text: &str, vars: &VariableMap) -> Result<(bool, String)> { + let mut expl = Explainer::new(text); + let success = try!(self.run(text, vars, &mut expl)); + expl.finish(); + Ok((success, expl.to_string())) + } + + fn run(&self, text: &str, vars: &VariableMap, recorder: &mut Recorder) -> Result { + let mut state = State::new(text, vars, recorder); // For each pending `not:` check, store (begin-offset, regex). let mut nots = Vec::new(); - for dct in &self.directives { + for (dct_idx, dct) in self.directives.iter().enumerate() { let (pat, range) = match *dct { Directive::Check(ref pat) => (pat, state.check()), Directive::SameLn(ref pat) => (pat, state.sameln()), @@ -164,7 +177,7 @@ impl Checker { // The `not:` directives test the same range as `unordered:` directives. In // particular, if they refer to defined variables, their range is restricted to // the text following the match that defined the variable. - nots.push((state.unordered_begin(pat), try!(pat.resolve(&state)))); + nots.push((dct_idx, state.unordered_begin(pat), try!(pat.resolve(&state)))); continue; } Directive::Regex(ref var, ref rx) => { @@ -177,6 +190,7 @@ impl Checker { } }; // Check if `pat` matches in `range`. + state.recorder.directive(dct_idx); if let Some((match_begin, match_end)) = try!(state.match_positive(pat, range)) { if let &Directive::Unordered(_) = dct { // This was an unordered unordered match. @@ -188,11 +202,14 @@ impl Checker { state.max_match = match_end; // Verify any pending `not:` directives now that we know their range. - for (not_begin, rx) in nots.drain(..) { - if let Some(_) = rx.find(&text[not_begin..match_begin]) { + for (not_idx, not_begin, rx) in nots.drain(..) { + state.recorder.directive(not_idx); + if let Some((s, e)) = rx.find(&text[not_begin..match_begin]) { // Matched `not:` pattern. - // TODO: Use matched range for an error message. + state.recorder.matched_not(rx.as_str(), (not_begin + s, not_begin + e)); return Ok(false); + } else { + state.recorder.missed_not(rx.as_str(), (not_begin, match_begin)); } } } @@ -203,7 +220,8 @@ impl Checker { } // Verify any pending `not:` directives after the last ordered directive. - for (not_begin, rx) in nots.drain(..) { + for (not_idx, not_begin, rx) in nots.drain(..) { + state.recorder.directive(not_idx); if let Some(_) = rx.find(&text[not_begin..]) { // Matched `not:` pattern. // TODO: Use matched range for an error message. @@ -224,8 +242,10 @@ pub struct VarDef { } struct State<'a> { - env_vars: &'a VariableMap, text: &'a str, + env_vars: &'a VariableMap, + recorder: &'a mut Recorder, + vars: HashMap, // Offset after the last ordered match. This does not include recent unordered matches. last_ordered: usize, @@ -234,10 +254,11 @@ struct State<'a> { } impl<'a> State<'a> { - fn new(text: &'a str, env_vars: &'a VariableMap) -> State<'a> { + fn new(text: &'a str, env_vars: &'a VariableMap, recorder: &'a mut Recorder) -> State<'a> { State { text: text, env_vars: env_vars, + recorder: recorder, vars: HashMap::new(), last_ordered: 0, max_match: 0, @@ -309,8 +330,10 @@ impl<'a> State<'a> { rx.captures(txt).map(|caps| { let matched_range = caps.pos(0).expect("whole expression must match"); for var in defs { + let txtval = caps.name(var).unwrap_or(""); + self.recorder.defined_var(var, txtval); let vardef = VarDef { - value: Value::Text(caps.name(var).unwrap_or("").to_string()), + value: Value::Text(txtval.to_string()), // This offset is the end of the whole matched pattern, not just the text // defining the variable. offset: range.0 + matched_range.1, @@ -320,7 +343,14 @@ impl<'a> State<'a> { matched_range }) }; - Ok(matched_range.map(|(b, e)| (range.0 + b, range.0 + e))) + Ok(if let Some((b, e)) = matched_range { + let r = (range.0 + b, range.0 + e); + self.recorder.matched_check(rx.as_str(), r); + Some(r) + } else { + self.recorder.missed_check(rx.as_str(), range); + None + }) } } diff --git a/cranelift/src/libfilecheck/explain.rs b/cranelift/src/libfilecheck/explain.rs new file mode 100644 index 0000000000..53dd4003a7 --- /dev/null +++ b/cranelift/src/libfilecheck/explain.rs @@ -0,0 +1,196 @@ +//! Explaining how *filecheck* matched or failed to match a file. + +use MatchRange; +use std::fmt::{self, Display, Formatter}; +use std::cmp::min; + +/// Record events during matching. +pub trait Recorder { + /// Set the directive we're talking about now. + fn directive(&mut self, dct: usize); + + /// Matched a positive check directive (check/sameln/nextln/unordered). + fn matched_check(&mut self, regex: &str, matched: MatchRange); + + /// Matched a `not:` directive. This means the match will fail. + fn matched_not(&mut self, regex: &str, matched: MatchRange); + + /// Missed a positive check directive. The range given is the range searched for a match. + fn missed_check(&mut self, regex: &str, searched: MatchRange); + + /// Missed `not:` directive (as intended). + fn missed_not(&mut self, regex: &str, searched: MatchRange); + + /// The directive defined a variable. + fn defined_var(&mut self, varname: &str, value: &str); +} + +/// The null recorder just doesn't listen to anything you say. +impl Recorder for () { + fn directive(&mut self, _: usize) {} + fn matched_check(&mut self, _: &str, _: MatchRange) {} + fn matched_not(&mut self, _: &str, _: MatchRange) {} + fn defined_var(&mut self, _: &str, _: &str) {} + fn missed_check(&mut self, _: &str, _: MatchRange) {} + fn missed_not(&mut self, _: &str, _: MatchRange) {} +} + +struct Match { + directive: usize, + is_match: bool, + is_not: bool, + regex: String, + range: MatchRange, +} + +struct VarDef { + directive: usize, + varname: String, + value: String, +} + +/// Record an explanation for the matching process, success or failure. +pub struct Explainer<'a> { + text: &'a str, + directive: usize, + matches: Vec, + vardefs: Vec, +} + +impl<'a> Explainer<'a> { + pub fn new(text: &'a str) -> Explainer { + Explainer { + text: text, + directive: 0, + matches: Vec::new(), + vardefs: Vec::new(), + } + } + + /// Finish up after recording all events in a match. + pub fn finish(&mut self) { + self.matches.sort_by_key(|m| (m.range, m.directive)); + self.vardefs.sort_by_key(|v| v.directive); + } +} + +impl<'a> Display for Explainer<'a> { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + // Offset of beginning of the last line printed. + let mut curln = 0; + // Offset of beginning of the next line to be printed. + let mut nextln = 0; + + for m in &self.matches { + // Emit lines until m.range.0 is visible. + while nextln <= m.range.0 && nextln < self.text.len() { + let newln = self.text[nextln..] + .find('\n') + .map(|d| nextln + d + 1) + .unwrap_or(self.text.len()); + assert!(newln > nextln); + try!(writeln!(f, "> {}", &self.text[nextln..newln - 1])); + curln = nextln; + nextln = newln; + } + + // Emit ~~~ under the part of the match in curln. + if m.is_match { + try!(write!(f, " ")); + let mend = min(m.range.1, nextln - 1); + for pos in curln..mend { + try!(if pos < m.range.0 { + write!(f, " ") + } else if pos == m.range.0 { + write!(f, "^") + } else { + write!(f, "~") + }); + } + try!(writeln!(f, "")); + } + + // Emit the match message itself. + try!(writeln!(f, + "{} #{}{}: {}", + if m.is_match { "Matched" } else { "Missed" }, + m.directive, + if m.is_not { " not" } else { "" }, + m.regex)); + + // Emit any variable definitions. + if let Ok(found) = self.vardefs.binary_search_by_key(&m.directive, |v| v.directive) { + let mut first = found; + while first > 0 && self.vardefs[first - 1].directive == m.directive { + first -= 1; + } + for d in &self.vardefs[first..] { + if d.directive != m.directive { + break; + } + try!(writeln!(f, "Define {}={}", d.varname, d.value)); + } + } + } + + // Emit trailing lines. + for line in self.text[nextln..].lines() { + try!(writeln!(f, "> {}", line)); + } + Ok(()) + } +} + +impl<'a> Recorder for Explainer<'a> { + fn directive(&mut self, dct: usize) { + self.directive = dct; + } + + fn matched_check(&mut self, regex: &str, matched: MatchRange) { + self.matches.push(Match { + directive: self.directive, + is_match: true, + is_not: false, + regex: regex.to_owned(), + range: matched, + }); + } + + fn matched_not(&mut self, regex: &str, matched: MatchRange) { + self.matches.push(Match { + directive: self.directive, + is_match: true, + is_not: true, + regex: regex.to_owned(), + range: matched, + }); + } + + fn missed_check(&mut self, regex: &str, searched: MatchRange) { + self.matches.push(Match { + directive: self.directive, + is_match: false, + is_not: false, + regex: regex.to_owned(), + range: searched, + }); + } + + fn missed_not(&mut self, regex: &str, searched: MatchRange) { + self.matches.push(Match { + directive: self.directive, + is_match: false, + is_not: true, + regex: regex.to_owned(), + range: searched, + }); + } + + fn defined_var(&mut self, varname: &str, value: &str) { + self.vardefs.push(VarDef { + directive: self.directive, + varname: varname.to_owned(), + value: value.to_owned(), + }); + } +} diff --git a/cranelift/src/libfilecheck/lib.rs b/cranelift/src/libfilecheck/lib.rs index 8d61a1dfbf..f6c53c5dcf 100644 --- a/cranelift/src/libfilecheck/lib.rs +++ b/cranelift/src/libfilecheck/lib.rs @@ -244,6 +244,7 @@ mod error; mod variable; mod pattern; mod checker; +mod explain; /// The range of a match in the input text. pub type MatchRange = (usize, usize); diff --git a/cranelift/src/tools/rsfilecheck.rs b/cranelift/src/tools/rsfilecheck.rs index bce661282c..b64a86283e 100644 --- a/cranelift/src/tools/rsfilecheck.rs +++ b/cranelift/src/tools/rsfilecheck.rs @@ -20,10 +20,21 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { let mut buffer = String::new(); try!(io::stdin().read_to_string(&mut buffer).map_err(|e| format!("stdin: {}", e))); - if try!(checker.check(&buffer, NO_VARIABLES).map_err(|e| e.to_string())) { + if verbose { + let (success, explain) = try!(checker.explain(&buffer, NO_VARIABLES) + .map_err(|e| e.to_string())); + print!("{}", explain); + if success { + println!("OK"); + Ok(()) + } else { + Err("Check failed".to_string()) + } + } else if try!(checker.check(&buffer, NO_VARIABLES).map_err(|e| e.to_string())) { Ok(()) } else { - // TODO: We need to do better than this. + let (_, explain) = try!(checker.explain(&buffer, NO_VARIABLES).map_err(|e| e.to_string())); + print!("{}", explain); Err("Check failed".to_string()) } } diff --git a/cranelift/tests/cfg/loop.cton b/cranelift/tests/cfg/loop.cton index 07d93926d8..fd6e4fa6b9 100644 --- a/cranelift/tests/cfg/loop.cton +++ b/cranelift/tests/cfg/loop.cton @@ -2,12 +2,14 @@ function nonsense(i32, i32) -> f32 { ; check: digraph nonsense { +; regex: I=\binst\d+\b +; check: label="{ebb0 | <$(BRZ=$I)>brz ebb2 | <$(JUMP=$I)>jump ebb1}"] ebb0(v1: i32, v2: i32): v3 = f64const 0x0.0 - brz v2, ebb2 ; unordered: ebb0:inst1 -> ebb2 + brz v2, ebb2 ; unordered: ebb0:$BRZ -> ebb2 v4 = iconst.i32 0 - jump ebb1(v4) ; unordered: ebb0:inst3 -> ebb1 + jump ebb1(v4) ; unordered: ebb0:$JUMP -> ebb1 ebb1(v5: i32): v6 = imul_imm v5, 4 From 1c5547970a09791756169d006c40a38496912ee1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Sat, 10 Sep 2016 12:33:58 -0700 Subject: [PATCH 0266/3084] Idiomatic impl of unordered_begin. --- cranelift/src/libfilecheck/checker.rs | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/cranelift/src/libfilecheck/checker.rs b/cranelift/src/libfilecheck/checker.rs index eba5da78a7..6bb458b55e 100644 --- a/cranelift/src/libfilecheck/checker.rs +++ b/cranelift/src/libfilecheck/checker.rs @@ -302,13 +302,11 @@ impl<'a> State<'a> { // Get the beginning of the range in text to be matched by a `unordered:` or `not:` directive. // The unordered directive must match after the directives that define the variables used. fn unordered_begin(&self, pat: &Pattern) -> usize { - let mut from = self.last_ordered; - for part in pat.parts() { - if let Some(var) = part.ref_var() { - from = max(from, self.def_offset(var)); - } - } - from + pat.parts() + .iter() + .filter_map(|part| part.ref_var()) + .map(|var| self.def_offset(var)) + .fold(self.last_ordered, max) } // Get the range in text to be matched by a `unordered:` directive. From b14be8b14bd341c4d0f3785193652c52c9dfa99d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 12 Sep 2016 14:14:41 -0700 Subject: [PATCH 0267/3084] Add an AnyEntity enum type. This type can reference any type of entity in a function. It will be used for the location of verifier error messages and other annotations. --- cranelift/src/libcretonne/ir/entities.rs | 55 ++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/cranelift/src/libcretonne/ir/entities.rs b/cranelift/src/libcretonne/ir/entities.rs index 1a887452a7..5672e5b3d3 100644 --- a/cranelift/src/libcretonne/ir/entities.rs +++ b/cranelift/src/libcretonne/ir/entities.rs @@ -245,6 +245,61 @@ impl Default for JumpTable { } } +/// A reference to any of the entities defined in this module. +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +pub enum AnyEntity { + /// The whole function. + Function, + Ebb(Ebb), + Inst(Inst), + Value(Value), + StackSlot(StackSlot), + JumpTable(JumpTable), +} + +impl Display for AnyEntity { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + match *self { + AnyEntity::Function => write!(fmt, "function"), + AnyEntity::Ebb(r) => r.fmt(fmt), + AnyEntity::Inst(r) => r.fmt(fmt), + AnyEntity::Value(r) => r.fmt(fmt), + AnyEntity::StackSlot(r) => r.fmt(fmt), + AnyEntity::JumpTable(r) => r.fmt(fmt), + } + } +} + +impl From for AnyEntity { + fn from(r: Ebb) -> AnyEntity { + AnyEntity::Ebb(r) + } +} + +impl From for AnyEntity { + fn from(r: Inst) -> AnyEntity { + AnyEntity::Inst(r) + } +} + +impl From for AnyEntity { + fn from(r: Value) -> AnyEntity { + AnyEntity::Value(r) + } +} + +impl From for AnyEntity { + fn from(r: StackSlot) -> AnyEntity { + AnyEntity::StackSlot(r) + } +} + +impl From for AnyEntity { + fn from(r: JumpTable) -> AnyEntity { + AnyEntity::JumpTable(r) + } +} + #[cfg(test)] mod tests { use super::*; From c80934d0846ae87031e4ac9b0279b1dc03ed7727 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 13 Sep 2016 10:51:15 -0700 Subject: [PATCH 0268/3084] Add a data structure representing a parsed test command. It's not used for anything yet. --- cranelift/src/libreader/lib.rs | 18 ++---- cranelift/src/libreader/testcommand.rs | 88 ++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 11 deletions(-) create mode 100644 cranelift/src/libreader/testcommand.rs diff --git a/cranelift/src/libreader/lib.rs b/cranelift/src/libreader/lib.rs index 2d63e68264..4c53c79fa0 100644 --- a/cranelift/src/libreader/lib.rs +++ b/cranelift/src/libreader/lib.rs @@ -1,16 +1,12 @@ - -// ====------------------------------------------------------------------------------------==== // -// -// Cretonne file reader library. -// -// ====------------------------------------------------------------------------------------==== // -// -// The cton_reader library supports reading and writing .cton files. This functionality is needed -// for testing Cretonne, but is not essential for a JIT compiler. -// -// ====------------------------------------------------------------------------------------==== // +//! Cretonne file reader library. +//! +//! The cton_reader library supports reading .cton files. This functionality is needed for testing +//! Cretonne, but is not essential for a JIT compiler. extern crate cretonne; +pub use testcommand::{TestCommand, TestOption}; + pub mod lexer; pub mod parser; +mod testcommand; diff --git a/cranelift/src/libreader/testcommand.rs b/cranelift/src/libreader/testcommand.rs new file mode 100644 index 0000000000..3eba0d0a53 --- /dev/null +++ b/cranelift/src/libreader/testcommand.rs @@ -0,0 +1,88 @@ +//! Test commands. +//! +//! A `.cton` file can begin with one or more *test commands* which specify what is to be tested. +//! The general syntax is: +//! +//! 
+//! test <command> [options]...
+//!
+//! +//! The options are either a single identifier flag, or setting values like `identifier=value`. +//! +//! The parser does not understand the test commands or which options are alid. It simply parses +//! the general format into a `TestCommand` data structure. + +use std::fmt::{self, Display, Formatter}; + +#[derive(Clone, PartialEq, Eq, Debug)] +pub struct TestCommand<'a> { + pub command: &'a str, + pub options: Vec>, +} + +#[derive(Clone, PartialEq, Eq, Debug)] +pub enum TestOption<'a> { + Flag(&'a str), + Value(&'a str, &'a str), +} + +impl<'a> TestCommand<'a> { + pub fn new(s: &'a str) -> TestCommand<'a> { + let mut parts = s.split_whitespace(); + let cmd = parts.next().unwrap_or(""); + TestCommand { + command: cmd, + options: parts.filter(|s| !s.is_empty()).map(TestOption::new).collect(), + } + } +} + +impl<'a> Display for TestCommand<'a> { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + try!(write!(f, "{}", self.command)); + for opt in &self.options { + try!(write!(f, " {}", opt)); + } + writeln!(f, "") + } +} + +impl<'a> TestOption<'a> { + pub fn new(s: &'a str) -> TestOption<'a> { + match s.find('=') { + None => TestOption::Flag(s), + Some(p) => TestOption::Value(&s[0..p], &s[p + 1..]), + } + } +} + +impl<'a> Display for TestOption<'a> { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + match *self { + TestOption::Flag(s) => write!(f, "{}", s), + TestOption::Value(s, v) => write!(f, "{}={}", s, v), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parse_option() { + assert_eq!(TestOption::new(""), TestOption::Flag("")); + assert_eq!(TestOption::new("foo"), TestOption::Flag("foo")); + assert_eq!(TestOption::new("foo=bar"), TestOption::Value("foo", "bar")); + } + + #[test] + fn parse_command() { + assert_eq!(&TestCommand::new("").to_string(), "\n"); + assert_eq!(&TestCommand::new("cat").to_string(), "cat\n"); + assert_eq!(&TestCommand::new("cat ").to_string(), "cat\n"); + assert_eq!(&TestCommand::new("cat 1 ").to_string(), "cat 1\n"); + assert_eq!(&TestCommand::new("cat one=4 two t").to_string(), + "cat one=4 two t\n"); + } +} From fcec517fc5cfa0d3e90f32f021be0ed83e11bfb9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 13 Sep 2016 11:01:21 -0700 Subject: [PATCH 0269/3084] Simplify the interface to cretonne-reader. Export a single function: parse_functions() which results a vector of functions parsed from the source string. Hide the parser and lexer modules. They are not useful to external clients. --- cranelift/src/libreader/lib.rs | 5 +++-- cranelift/src/libreader/parser.rs | 12 +++++++----- cranelift/src/tools/cat.rs | 4 ++-- cranelift/src/tools/print_cfg.rs | 4 ++-- cranelift/src/tools/tests/cfg_traversal.rs | 4 ++-- cranelift/src/tools/tests/dominator_tree.rs | 4 ++-- cranelift/src/tools/tests/verifier.rs | 4 ++-- 7 files changed, 20 insertions(+), 17 deletions(-) diff --git a/cranelift/src/libreader/lib.rs b/cranelift/src/libreader/lib.rs index 4c53c79fa0..519fee7eed 100644 --- a/cranelift/src/libreader/lib.rs +++ b/cranelift/src/libreader/lib.rs @@ -5,8 +5,9 @@ extern crate cretonne; +pub use parser::{Result, parse_functions}; pub use testcommand::{TestCommand, TestOption}; -pub mod lexer; -pub mod parser; +mod lexer; +mod parser; mod testcommand; diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index fa6a445158..2ae1629c80 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -21,6 +21,13 @@ use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArg pub use lexer::Location; +/// Parse the entire `text` into a list of functions. +/// +/// Any test commands or ISA declarations are ignored. +pub fn parse_functions(text: &str) -> Result> { + Parser::new(text).parse_function_list() +} + /// A parse error is returned when the parse failed. #[derive(Debug)] pub struct Error { @@ -256,11 +263,6 @@ impl<'a> Parser<'a> { } } - /// Parse the entire string into a list of functions. - pub fn parse(text: &'a str) -> Result> { - Self::new(text).parse_function_list() - } - // Consume the current lookahead token and return it. fn consume(&mut self) -> Token<'a> { self.lookahead.take().expect("No token to consume") diff --git a/cranelift/src/tools/cat.rs b/cranelift/src/tools/cat.rs index 6d740da701..a41b5b5f27 100644 --- a/cranelift/src/tools/cat.rs +++ b/cranelift/src/tools/cat.rs @@ -8,7 +8,7 @@ use std::io::{self, Read}; use CommandResult; -use cton_reader::parser::Parser; +use cton_reader::parse_functions; use cretonne::write::write_function; pub fn run(files: Vec) -> CommandResult { @@ -26,7 +26,7 @@ fn cat_one(filename: String) -> CommandResult { let mut buffer = String::new(); try!(file.read_to_string(&mut buffer) .map_err(|e| format!("Couldn't read {}: {}", filename, e))); - let items = try!(Parser::parse(&buffer).map_err(|e| format!("{}: {}", filename, e))); + let items = try!(parse_functions(&buffer).map_err(|e| format!("{}: {}", filename, e))); for (idx, func) in items.into_iter().enumerate() { if idx != 0 { diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs index dd1aa3de18..649d87fb9f 100644 --- a/cranelift/src/tools/print_cfg.rs +++ b/cranelift/src/tools/print_cfg.rs @@ -9,7 +9,7 @@ use CommandResult; use cretonne::ir::Function; use cretonne::cfg::ControlFlowGraph; use cretonne::ir::instructions::InstructionData; -use cton_reader::parser::Parser; +use cton_reader::parse_functions; pub fn run(files: Vec) -> CommandResult { for (i, f) in files.into_iter().enumerate() { @@ -160,7 +160,7 @@ fn print_cfg(filename: String) -> CommandResult { let mut buffer = String::new(); try!(file.read_to_string(&mut buffer) .map_err(|e| format!("Couldn't read {}: {}", filename, e))); - let items = try!(Parser::parse(&buffer).map_err(|e| format!("{}: {}", filename, e))); + let items = try!(parse_functions(&buffer).map_err(|e| format!("{}: {}", filename, e))); let mut cfg_printer = CFGPrinter::new(stdout()); for (idx, func) in items.into_iter().enumerate() { diff --git a/cranelift/src/tools/tests/cfg_traversal.rs b/cranelift/src/tools/tests/cfg_traversal.rs index 03ed5bd148..b766f9d96c 100644 --- a/cranelift/src/tools/tests/cfg_traversal.rs +++ b/cranelift/src/tools/tests/cfg_traversal.rs @@ -1,13 +1,13 @@ extern crate cretonne; extern crate cton_reader; -use self::cton_reader::parser::Parser; +use self::cton_reader::parse_functions; use self::cretonne::ir::Ebb; use self::cretonne::cfg::ControlFlowGraph; use self::cretonne::entity_map::EntityMap; fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) { - let func = &Parser::parse(function_source).unwrap()[0]; + let func = &parse_functions(function_source).unwrap()[0]; let cfg = ControlFlowGraph::new(&func); let ebbs = ebb_order.iter().map(|n| Ebb::with_number(*n).unwrap()) .collect::>(); diff --git a/cranelift/src/tools/tests/dominator_tree.rs b/cranelift/src/tools/tests/dominator_tree.rs index c1be367137..71c02dc652 100644 --- a/cranelift/src/tools/tests/dominator_tree.rs +++ b/cranelift/src/tools/tests/dominator_tree.rs @@ -9,7 +9,7 @@ use regex::Regex; use std::fs::File; use std::io::Read; use self::cretonne::ir::Ebb; -use self::cton_reader::parser::Parser; +use self::cton_reader::parse_functions; use self::cretonne::ir::function::Function; use self::cretonne::entity_map::EntityMap; use self::cretonne::ir::entities::NO_INST; @@ -69,7 +69,7 @@ fn dominator_tree_from_source(func: &Function, function_source: &str) -> Dominat fn test_dominator_tree(function_source: &str) { - let func = &Parser::parse(function_source).unwrap()[0]; + let func = &parse_functions(function_source).unwrap()[0]; let src_dtree = dominator_tree_from_source(&func, function_source); let cfg = ControlFlowGraph::new(&func); diff --git a/cranelift/src/tools/tests/verifier.rs b/cranelift/src/tools/tests/verifier.rs index 28bb1bbba5..c26056b8ba 100644 --- a/cranelift/src/tools/tests/verifier.rs +++ b/cranelift/src/tools/tests/verifier.rs @@ -8,7 +8,7 @@ use glob::glob; use regex::Regex; use std::fs::File; use std::io::Read; -use self::cton_reader::parser::Parser; +use self::cton_reader::parse_functions; use self::cretonne::verifier::Verifier; /// Compile a function and run verifier tests based on specially formatted @@ -37,7 +37,7 @@ fn verifier_tests_from_source(function_source: &str) { // Run the verifier against each function and compare the output // with the expected result (as determined above). - for (i, func) in Parser::parse(function_source).unwrap().into_iter().enumerate() { + for (i, func) in parse_functions(function_source).unwrap().into_iter().enumerate() { let result = Verifier::new(&func).run(); match verifier_results[i] { Some(ref re) => { From 1b6623f068ec5bdb74df40fb65b92b88423e4a1e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 13 Sep 2016 12:18:36 -0700 Subject: [PATCH 0270/3084] Add a representation of a parsed test case file. The new exported function `parse_test()` will produce it eventually. --- cranelift/src/libreader/lib.rs | 4 ++- cranelift/src/libreader/parser.rs | 9 +++++++ cranelift/src/libreader/testfile.rs | 40 +++++++++++++++++++++++++++++ 3 files changed, 52 insertions(+), 1 deletion(-) create mode 100644 cranelift/src/libreader/testfile.rs diff --git a/cranelift/src/libreader/lib.rs b/cranelift/src/libreader/lib.rs index 519fee7eed..d435b4ac3b 100644 --- a/cranelift/src/libreader/lib.rs +++ b/cranelift/src/libreader/lib.rs @@ -5,9 +5,11 @@ extern crate cretonne; -pub use parser::{Result, parse_functions}; +pub use parser::{Result, parse_functions, parse_test}; pub use testcommand::{TestCommand, TestOption}; +pub use testfile::TestFile; mod lexer; mod parser; mod testcommand; +mod testfile; diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 2ae1629c80..e9d77c67d7 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -18,6 +18,7 @@ use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::{NO_EBB, NO_VALUE}; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, JumpData, BranchData, ReturnData}; +use testfile::TestFile; pub use lexer::Location; @@ -28,6 +29,14 @@ pub fn parse_functions(text: &str) -> Result> { Parser::new(text).parse_function_list() } +/// Parse the entire `text` as a test case file. +/// +/// The returned `TestFile` contains direct references to substrings of `text`. +pub fn parse_test<'a>(text: &'a str) -> Result> { + Parser::new(text); + unimplemented!() +} + /// A parse error is returned when the parse failed. #[derive(Debug)] pub struct Error { diff --git a/cranelift/src/libreader/testfile.rs b/cranelift/src/libreader/testfile.rs new file mode 100644 index 0000000000..34eddb4c6b --- /dev/null +++ b/cranelift/src/libreader/testfile.rs @@ -0,0 +1,40 @@ +//! Data structures representing a parsed test file. +//! +//! A test file is a `.cton` file which contains test commands and settings for running a +//! file-based test case. +//! + +use cretonne::ir::Function; +use cretonne::ir::entities::AnyEntity; +use testcommand::TestCommand; + +/// A parsed test case. +/// +/// This is the result of parsing a `.cton` file which contains a number of test commands followed +/// by the functions that should be tested. +#[derive(Debug)] +pub struct TestFile<'a> { + pub commands: Vec>, + pub functions: Vec>, +} + +/// A function parsed from a text string along with other details that are useful for running +/// tests. +#[derive(Debug)] +pub struct DetailedFunction<'a> { + pub function: Function, + pub comments: Vec>, +} + +/// A comment in a parsed function. +/// +/// The comment belongs to the immediately preceeding entity, whether that is an EBB header, and +/// instruction, or one of the preamble declarations. +/// +/// Comments appearing inside the function but before the preamble, as well as comments appearing +/// after the function are tagged as `AnyEntity::Function`. +#[derive(Clone, PartialEq, Eq, Debug)] +pub struct Comment<'a> { + pub entity: AnyEntity, + pub text: &'a str, +} From 458ecebe8f591457fd6c91b7b66ca79bd20376bf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 13 Sep 2016 16:14:46 -0700 Subject: [PATCH 0271/3084] Add a next_key() method to primary entity maps. It is sometimes useful to know the entity reference number that will be assigned to the next thing added to a map. --- cranelift/src/libcretonne/entity_map.rs | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/cranelift/src/libcretonne/entity_map.rs b/cranelift/src/libcretonne/entity_map.rs index 91ff7505a3..df486c48ed 100644 --- a/cranelift/src/libcretonne/entity_map.rs +++ b/cranelift/src/libcretonne/entity_map.rs @@ -94,9 +94,14 @@ impl EntityMap where K: EntityRef, V: PrimaryEntityData { + /// Get the key that will be assigned to the next pushed value. + pub fn next_key(&self) -> K { + K::new(self.elems.len()) + } + /// Append `v` to the mapping, assigning a new key which is returned. pub fn push(&mut self, v: V) -> K { - let k = K::new(self.elems.len()); + let k = self.next_key(); self.elems.push(v); k } From 4e3f0fc41d6fbbfd18409c31a7a2dfe601b0beb7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 13 Sep 2016 16:15:42 -0700 Subject: [PATCH 0272/3084] Collect comments while parsing functions. The result from parsing a function is now a DetailedFunction which includes all comments that can be associated with an entity. Comments before the first function are ignored, everything else is associated with the preceeding entity. The parse_functions() function still returns plain functions. --- cranelift/src/libreader/parser.rs | 119 +++++++++++++++++++++++++++--- 1 file changed, 109 insertions(+), 10 deletions(-) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index e9d77c67d7..5e18af59be 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -10,15 +10,16 @@ use std::result; use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::u32; +use std::mem; use lexer::{self, Lexer, Token}; use cretonne::ir::{Function, Ebb, Inst, Opcode, Value, Type, FunctionName, StackSlot, StackSlotData, JumpTable, JumpTableData}; use cretonne::ir::types::{VOID, Signature, ArgumentType, ArgumentExtension}; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; -use cretonne::ir::entities::{NO_EBB, NO_VALUE}; +use cretonne::ir::entities::{AnyEntity, NO_EBB, NO_INST, NO_VALUE}; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, JumpData, BranchData, ReturnData}; -use testfile::TestFile; +use testfile::{TestFile, DetailedFunction, Comment}; pub use lexer::Location; @@ -26,7 +27,9 @@ pub use lexer::Location; /// /// Any test commands or ISA declarations are ignored. pub fn parse_functions(text: &str) -> Result> { - Parser::new(text).parse_function_list() + Parser::new(text) + .parse_function_list() + .map(|list| list.into_iter().map(|dfunc| dfunc.function).collect()) } /// Parse the entire `text` as a test case file. @@ -79,6 +82,13 @@ pub struct Parser<'a> { // Location of lookahead. loc: Location, + + // The currently active entity that should be associated with collected comments, or `None` if + // comments are ignored. + comment_entity: Option, + + // Comments collected so far. + comments: Vec>, } // Context for resolving references when parsing a single function. @@ -269,6 +279,8 @@ impl<'a> Parser<'a> { lex_error: None, lookahead: None, loc: Location { line_number: 0 }, + comment_entity: None, + comments: Vec::new(), } } @@ -283,8 +295,14 @@ impl<'a> Parser<'a> { match self.lex.next() { Some(Ok(lexer::LocatedToken { token, location })) => { match token { - Token::Comment(_) => { - // Ignore comments. + Token::Comment(text) => { + // Gather comments, associate them with `comment_entity`. + if let Some(entity) = self.comment_entity { + self.comments.push(Comment { + entity: entity, + text: text, + }); + } } _ => self.lookahead = Some(token), } @@ -301,6 +319,22 @@ impl<'a> Parser<'a> { return self.lookahead; } + // Begin gathering comments associated with `entity`. + fn gather_comments>(&mut self, entity: E) { + self.comment_entity = Some(entity.into()); + } + + // Rewrite the entity of the last added comments from `old` to `new`. + // Also switch to collecting future comments for `new`. + fn rewrite_last_comment_entities>(&mut self, old: E, new: E) { + let old = old.into(); + let new = new.into(); + for comment in (&mut self.comments).into_iter().rev().take_while(|c| c.entity == old) { + comment.entity = new; + } + self.comment_entity = Some(new); + } + // Match and consume a token without payload. fn match_token(&mut self, want: Token<'a>, err_msg: &str) -> Result> { if self.token() == Some(want) { @@ -450,7 +484,7 @@ impl<'a> Parser<'a> { /// Parse a list of function definitions. /// /// This is the top-level parse function matching the whole contents of a file. - pub fn parse_function_list(&mut self) -> Result> { + pub fn parse_function_list(&mut self) -> Result>> { let mut list = Vec::new(); while self.token().is_some() { list.push(try!(self.parse_function())); @@ -462,7 +496,13 @@ impl<'a> Parser<'a> { // // function ::= * function-spec "{" preamble function-body "}" // - fn parse_function(&mut self) -> Result { + fn parse_function(&mut self) -> Result> { + // Begin gathering comments. + // Make sure we don't include any comments before the `function` keyword. + self.token(); + self.comments.clear(); + self.gather_comments(AnyEntity::Function); + let (name, sig) = try!(self.parse_function_spec()); let mut ctx = Context::new(Function::with_name_signature(name, sig)); @@ -475,11 +515,19 @@ impl<'a> Parser<'a> { // function ::= function-spec "{" preamble function-body * "}" try!(self.match_token(Token::RBrace, "expected '}' after function body")); + // Collect any comments following the end of the function, then stop gathering comments. + self.gather_comments(AnyEntity::Function); + self.token(); + self.comment_entity = None; + // Rewrite references to values and EBBs after parsing everuthing to allow forward // references. try!(ctx.rewrite_references()); - Ok(ctx.function) + Ok(DetailedFunction { + function: ctx.function, + comments: mem::replace(&mut self.comments, Vec::new()), + }) } // Parse a function spec. @@ -585,10 +633,12 @@ impl<'a> Parser<'a> { loop { try!(match self.token() { Some(Token::StackSlot(..)) => { + self.gather_comments(ctx.function.stack_slots.next_key()); self.parse_stack_slot_decl() .and_then(|(num, dat)| ctx.add_ss(num, dat, &self.loc)) } Some(Token::JumpTable(..)) => { + self.gather_comments(ctx.function.jump_tables.next_key()); self.parse_jump_table_decl() .and_then(|(num, dat)| ctx.add_jt(num, dat, &self.loc)) } @@ -681,6 +731,7 @@ impl<'a> Parser<'a> { fn parse_extended_basic_block(&mut self, ctx: &mut Context) -> Result<()> { let ebb_num = try!(self.match_ebb("expected EBB header")); let ebb = try!(ctx.add_ebb(ebb_num, &self.loc)); + self.gather_comments(ebb); if !self.optional(Token::Colon) { // ebb-header ::= Ebb(ebb) [ * ebb-args ] ":" @@ -746,6 +797,10 @@ impl<'a> Parser<'a> { // inst-results ::= Value(v) { "," Value(vx) } // fn parse_instruction(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { + // Collect comments for `NO_INST` while parsing the instruction, then rewrite after we + // allocate an instruction number. + self.gather_comments(NO_INST); + // Result value numbers. let mut results = Vec::new(); @@ -805,6 +860,10 @@ impl<'a> Parser<'a> { results.len()); } + // If we saw any comments while parsing the instruction, they will have been recorded as + // belonging to `NO_INST`. + self.rewrite_last_comment_entities(NO_INST, inst); + // Now map the source result values to the just created instruction results. // Pass a reference to `ctx.values` instead of `ctx` itself since the `Values` iterator // holds a reference to `ctx.function`. @@ -1140,6 +1199,8 @@ impl<'a> Parser<'a> { mod tests { use super::*; use cretonne::ir::types::{self, ArgumentType, ArgumentExtension}; + use cretonne::ir::entities::AnyEntity; + use testfile::Comment; #[test] fn argument_type() { @@ -1184,7 +1245,8 @@ mod tests { ss1 = stack_slot 1 }") .parse_function() - .unwrap(); + .unwrap() + .function; assert_eq!(func.name, "foo"); let mut iter = func.stack_slots.keys(); let ss0 = iter.next().unwrap(); @@ -1213,7 +1275,8 @@ mod tests { ebb4(vx3: i32): }") .parse_function() - .unwrap(); + .unwrap() + .function; assert_eq!(func.name, "ebbs"); let mut ebbs = func.layout.ebbs(); @@ -1227,4 +1290,40 @@ mod tests { assert_eq!(func.dfg.value_type(arg0), types::I32); assert_eq!(ebb4_args.next(), None); } + + #[test] + fn comments() { + let dfunc = Parser::new("; before + function comment() { ; decl + ss10 = stack_slot 13 ; stackslot. + ; Still stackslot. + jt10 = jump_table ebb0 + ; Jumptable + ebb0: ; Basic block + trap ; Instruction + } ; Trailing. + ; More trailing.") + .parse_function() + .unwrap(); + assert_eq!(&dfunc.function.name, "comment"); + assert_eq!(dfunc.comments.len(), 8); // no 'before' comment. + assert_eq!(dfunc.comments[0], + Comment { + entity: AnyEntity::Function, + text: "; decl", + }); + assert_eq!(dfunc.comments[1].entity.to_string(), "ss0"); + assert_eq!(dfunc.comments[2].entity.to_string(), "ss0"); + assert_eq!(dfunc.comments[2].text, "; Still stackslot."); + assert_eq!(dfunc.comments[3].entity.to_string(), "jt0"); + assert_eq!(dfunc.comments[3].text, "; Jumptable"); + assert_eq!(dfunc.comments[4].entity.to_string(), "ebb0"); + assert_eq!(dfunc.comments[4].text, "; Basic block"); + + assert_eq!(dfunc.comments[5].entity.to_string(), "inst0"); + assert_eq!(dfunc.comments[5].text, "; Instruction"); + + assert_eq!(dfunc.comments[6].entity, AnyEntity::Function); + assert_eq!(dfunc.comments[7].entity, AnyEntity::Function); + } } From 21ad1d1e2d99c4107594c8d56114d82a18305207 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 13 Sep 2016 16:50:10 -0700 Subject: [PATCH 0273/3084] Add a public rest_of_line() function to lexer. This is used to grap the tail of a 'test' line which doesn't use the same tokens as a normal .cton file. --- cranelift/src/libreader/lexer.rs | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/cranelift/src/libreader/lexer.rs b/cranelift/src/libreader/lexer.rs index 150c38da19..6b5e881a1f 100644 --- a/cranelift/src/libreader/lexer.rs +++ b/cranelift/src/libreader/lexer.rs @@ -166,21 +166,25 @@ impl<'a> Lexer<'a> { token(tok, loc) } - // Scan a comment extending to the end of the current line. - fn scan_comment(&mut self) -> Result, LocatedError> { + /// Get the rest of the current line. + /// The next token returned by `next()` will be from the following lines. + pub fn rest_of_line(&mut self) -> &'a str { let begin = self.pos; - let loc = self.loc(); loop { match self.next_ch() { - None | Some('\n') => { - let text = &self.source[begin..self.pos]; - return token(Token::Comment(text), loc); - } + None | Some('\n') => return &self.source[begin..self.pos], _ => {} } } } + // Scan a comment extending to the end of the current line. + fn scan_comment(&mut self) -> Result, LocatedError> { + let loc = self.loc(); + let text = self.rest_of_line(); + return token(Token::Comment(text), loc); + } + // Scan a number token which can represent either an integer or floating point number. // // Accept the following forms: From 538b773e6d3fef40fc9fc6a6b1cac10537b18b23 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 14 Sep 2016 09:10:32 -0700 Subject: [PATCH 0274/3084] Parse test commands in a .cton file. The top-level parse_test() function now parses test commands in the preamble of a .cton file. --- cranelift/src/libreader/parser.rs | 39 +++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 5e18af59be..7f8cc543ad 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -20,6 +20,7 @@ use cretonne::ir::entities::{AnyEntity, NO_EBB, NO_INST, NO_VALUE}; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, JumpData, BranchData, ReturnData}; use testfile::{TestFile, DetailedFunction, Comment}; +use testcommand::TestCommand; pub use lexer::Location; @@ -36,8 +37,11 @@ pub fn parse_functions(text: &str) -> Result> { /// /// The returned `TestFile` contains direct references to substrings of `text`. pub fn parse_test<'a>(text: &'a str) -> Result> { - Parser::new(text); - unimplemented!() + let mut parser = Parser::new(text); + Ok(TestFile { + commands: parser.parse_test_commands(), + functions: try!(parser.parse_function_list()), + }) } /// A parse error is returned when the parse failed. @@ -289,6 +293,14 @@ impl<'a> Parser<'a> { self.lookahead.take().expect("No token to consume") } + // Consume the whole line following the current lookahead token. + // Return the text of the line tail. + fn consume_line(&mut self) -> &'a str { + let rest = self.lex.rest_of_line(); + self.consume(); + rest + } + // Get the current lookahead token, after making sure there is one. fn token(&mut self) -> Option> { while self.lookahead == None { @@ -481,6 +493,15 @@ impl<'a> Parser<'a> { } } + /// Parse a list of test commands. + pub fn parse_test_commands(&mut self) -> Vec> { + let mut list = Vec::new(); + while self.token() == Some(Token::Identifier("test")) { + list.push(TestCommand::new(self.consume_line())); + } + list + } + /// Parse a list of function definitions. /// /// This is the top-level parse function matching the whole contents of a file. @@ -1326,4 +1347,18 @@ mod tests { assert_eq!(dfunc.comments[6].entity, AnyEntity::Function); assert_eq!(dfunc.comments[7].entity, AnyEntity::Function); } + + #[test] + fn test_file() { + let tf = parse_test("; before + test cfg option=5 + test verify + function comment() {}") + .unwrap(); + assert_eq!(tf.commands.len(), 2); + assert_eq!(tf.commands[0].command, "cfg"); + assert_eq!(tf.commands[1].command, "verify"); + assert_eq!(tf.functions.len(), 1); + assert_eq!(tf.functions[0].function.name, "comment"); + } } From ddadf2e7c1f71d0641f1179adc6e67bcd12687bc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 14 Sep 2016 12:20:41 -0700 Subject: [PATCH 0275/3084] Add scaffolding for a 'cton-util test' command. This command accepts files and directories containing test cases to run. Recursively searches for test files in any directory it is passed. Actually running tests is not yet implemented. --- cranelift/src/tools/filetest/mod.rs | 33 ++++++++ cranelift/src/tools/filetest/runner.rs | 112 +++++++++++++++++++++++++ cranelift/src/tools/main.rs | 8 +- 3 files changed, 151 insertions(+), 2 deletions(-) create mode 100644 cranelift/src/tools/filetest/mod.rs create mode 100644 cranelift/src/tools/filetest/runner.rs diff --git a/cranelift/src/tools/filetest/mod.rs b/cranelift/src/tools/filetest/mod.rs new file mode 100644 index 0000000000..078e287dde --- /dev/null +++ b/cranelift/src/tools/filetest/mod.rs @@ -0,0 +1,33 @@ +//! File tests. +//! +//! This module contains the main driver for `cton-util test` as well as implementations of the +//! available test commands. + +use std::path::Path; +use CommandResult; +use filetest::runner::TestRunner; + +mod runner; + +/// Main entry point for `cton-util test`. +/// +/// Take a list of filenames which can be either `.cton` files or directories. +/// +/// Files are interpreted as test cases and executed immediately. +/// +/// Directories are scanned recursively for test cases ending in `.cton`. These test cases are +/// executed on background threads. +/// +pub fn run(files: Vec) -> CommandResult { + let mut runner = TestRunner::new(); + + for path in files.iter().map(Path::new) { + if path.is_file() { + runner.push_test(path); + } else { + runner.push_dir(path); + } + } + + runner.run() +} diff --git a/cranelift/src/tools/filetest/runner.rs b/cranelift/src/tools/filetest/runner.rs new file mode 100644 index 0000000000..7f39ec6490 --- /dev/null +++ b/cranelift/src/tools/filetest/runner.rs @@ -0,0 +1,112 @@ +//! Test runner. +//! +//! This module implements the `TestRunner` struct which manages executing tests as well as +//! scanning directories for tests. + +use std::ffi::OsStr; +use std::path::PathBuf; +use std::error::Error; +use CommandResult; + +pub struct TestRunner { + // Directories that have not yet been scanned. + dir_stack: Vec, + + // Filenames of tests to run. + test_list: Vec, + + errors: usize, +} + +impl TestRunner { + /// Create a new blank TrstRunner. + pub fn new() -> TestRunner { + TestRunner { + dir_stack: Vec::new(), + test_list: Vec::new(), + errors: 0, + } + } + + /// Add a directory path to be scanned later. + /// + /// If `dir` turns out to be a regular file, it is silently ignored. + /// Otherwise, any problems reading the directory are reported. + pub fn push_dir>(&mut self, dir: P) { + self.dir_stack.push(dir.into()); + } + + /// Add a test to be executed later. + /// + /// Any problems reading `file` as a test case file will be reported as a test failure. + pub fn push_test>(&mut self, file: P) { + self.test_list.push(file.into()); + } + + /// Scan any directories pushed so far. + /// Push any potential test cases found. + pub fn scan_dirs(&mut self) { + // This recursive search tries to minimize statting in a directory hierarchy containing + // mostly test cases. + // + // - Directory entries with a "cton" extension are presumed to be test case files. + // - Directory entries with no extension are presumed to be subdirectories. + // - Anything else is ignored. + // + while let Some(dir) = self.dir_stack.pop() { + match dir.read_dir() { + Err(err) => { + // Fail silently if `dir` was actually a regular file. + // This lets us skip spurious extensionless files without statting everything + // needlessly. + if !dir.is_file() { + self.path_error(dir, err); + } + } + Ok(entries) => { + // Read all directory entries. Avoid statting. + for entry_result in entries { + match entry_result { + Err(err) => { + // Not sure why this would happen. `read_dir` succeeds, but there's + // a problem with an entry. I/O error during a getdirentries + // syscall seems to be the reason. The implementation in + // libstd/sys/unix/fs.rs seems to suggest that breaking now would + // be a good idea, or the iterator could keep returning the same + // error forever. + self.path_error(dir, err); + break; + } + Ok(entry) => { + let path = entry.path(); + // Recognize directories and tests by extension. + // Yes, this means we ignore directories with '.' in their name. + match path.extension().and_then(OsStr::to_str) { + Some("cton") => self.push_test(path), + Some(_) => {} + None => self.push_dir(path), + } + } + } + } + } + } + } + } + + /// Report an error related to a path. + fn path_error(&mut self, path: PathBuf, err: E) { + self.errors += 1; + println!("path-error {}: {}", path.to_string_lossy(), err); + } + + /// Scan pushed directories for tests and run them. + pub fn run(&mut self) -> CommandResult { + self.scan_dirs(); + match self.errors { + 0 => Ok(()), + 1 => Err("1 failure".to_string()), + n => Err(format!("{} failures", n)), + } + } +} diff --git a/cranelift/src/tools/main.rs b/cranelift/src/tools/main.rs index bd0999cd4e..5abfa89462 100644 --- a/cranelift/src/tools/main.rs +++ b/cranelift/src/tools/main.rs @@ -10,7 +10,7 @@ use docopt::Docopt; use std::io::{self, Write}; use std::process; - +mod filetest; mod cat; mod print_cfg; mod rsfilecheck; @@ -19,6 +19,7 @@ const USAGE: &'static str = " Cretonne code generator utility Usage: + cton-util test ... cton-util cat ... cton-util filecheck [-v] cton-util print-cfg ... @@ -33,6 +34,7 @@ Options: #[derive(RustcDecodable, Debug)] struct Args { + cmd_test: bool, cmd_cat: bool, cmd_filecheck: bool, cmd_print_cfg: bool, @@ -55,7 +57,9 @@ fn cton_util() -> CommandResult { .unwrap_or_else(|e| e.exit()); // Find the sub-command to execute. - if args.cmd_cat { + if args.cmd_test { + filetest::run(args.arg_file) + } else if args.cmd_cat { cat::run(args.arg_file) } else if args.cmd_filecheck { rsfilecheck::run(args.arg_file, args.flag_verbose) From 99c04383c8e1ffd339ad007dad98a1e6b7ec18f4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 14 Sep 2016 13:14:43 -0700 Subject: [PATCH 0276/3084] Add a utility read_to_string() function. --- cranelift/src/tools/cat.rs | 11 +++-------- cranelift/src/tools/main.rs | 1 + cranelift/src/tools/print_cfg.rs | 9 +++------ cranelift/src/tools/rsfilecheck.rs | 8 ++------ cranelift/src/tools/utils.rs | 13 +++++++++++++ 5 files changed, 22 insertions(+), 20 deletions(-) create mode 100644 cranelift/src/tools/utils.rs diff --git a/cranelift/src/tools/cat.rs b/cranelift/src/tools/cat.rs index a41b5b5f27..566997c403 100644 --- a/cranelift/src/tools/cat.rs +++ b/cranelift/src/tools/cat.rs @@ -3,11 +3,9 @@ //! Read a sequence of Cretonne IL files and print them again to stdout. This has the effect of //! normalizing formatting and removing comments. -use std::fs::File; -use std::io::{self, Read}; - +use std::io; use CommandResult; - +use utils::read_to_string; use cton_reader::parse_functions; use cretonne::write::write_function; @@ -22,10 +20,7 @@ pub fn run(files: Vec) -> CommandResult { } fn cat_one(filename: String) -> CommandResult { - let mut file = try!(File::open(&filename).map_err(|e| format!("{}: {}", filename, e))); - let mut buffer = String::new(); - try!(file.read_to_string(&mut buffer) - .map_err(|e| format!("Couldn't read {}: {}", filename, e))); + let buffer = try!(read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))); let items = try!(parse_functions(&buffer).map_err(|e| format!("{}: {}", filename, e))); for (idx, func) in items.into_iter().enumerate() { diff --git a/cranelift/src/tools/main.rs b/cranelift/src/tools/main.rs index 5abfa89462..af7a092394 100644 --- a/cranelift/src/tools/main.rs +++ b/cranelift/src/tools/main.rs @@ -10,6 +10,7 @@ use docopt::Docopt; use std::io::{self, Write}; use std::process; +mod utils; mod filetest; mod cat; mod print_cfg; diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs index 649d87fb9f..1c57710b29 100644 --- a/cranelift/src/tools/print_cfg.rs +++ b/cranelift/src/tools/print_cfg.rs @@ -2,10 +2,10 @@ //! //! Read a series of Cretonne IL files and print their control flow graphs //! in graphviz format. -use std::fs::File; -use std::io::{Read, Write, stdout}; +use std::io::{Write, stdout}; use CommandResult; +use utils::read_to_string; use cretonne::ir::Function; use cretonne::cfg::ControlFlowGraph; use cretonne::ir::instructions::InstructionData; @@ -156,10 +156,7 @@ impl CFGPrinter { } fn print_cfg(filename: String) -> CommandResult { - let mut file = try!(File::open(&filename).map_err(|e| format!("{}: {}", filename, e))); - let mut buffer = String::new(); - try!(file.read_to_string(&mut buffer) - .map_err(|e| format!("Couldn't read {}: {}", filename, e))); + let buffer = try!(read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))); let items = try!(parse_functions(&buffer).map_err(|e| format!("{}: {}", filename, e))); let mut cfg_printer = CFGPrinter::new(stdout()); diff --git a/cranelift/src/tools/rsfilecheck.rs b/cranelift/src/tools/rsfilecheck.rs index b64a86283e..5ff6c596d1 100644 --- a/cranelift/src/tools/rsfilecheck.rs +++ b/cranelift/src/tools/rsfilecheck.rs @@ -1,6 +1,6 @@ use CommandResult; +use utils::read_to_string; use filecheck::{CheckerBuilder, Checker, NO_VARIABLES}; -use std::fs::File; use std::io::{self, Read}; pub fn run(files: Vec, verbose: bool) -> CommandResult { @@ -40,11 +40,7 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { } fn read_checkfile(filename: &str) -> Result { - let mut file = try!(File::open(&filename).map_err(|e| format!("{}: {}", filename, e))); - let mut buffer = String::new(); - try!(file.read_to_string(&mut buffer) - .map_err(|e| format!("Couldn't read {}: {}", filename, e))); - + let buffer = try!(read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))); let mut builder = CheckerBuilder::new(); try!(builder.text(&buffer).map_err(|e| format!("{}: {}", filename, e))); Ok(builder.finish()) diff --git a/cranelift/src/tools/utils.rs b/cranelift/src/tools/utils.rs new file mode 100644 index 0000000000..5973f25814 --- /dev/null +++ b/cranelift/src/tools/utils.rs @@ -0,0 +1,13 @@ +//! Utility functions. + +use std::path::Path; +use std::fs::File; +use std::io::{Result, Read}; + +/// Read an entire file into a string. +pub fn read_to_string>(path: P) -> Result { + let mut file = try!(File::open(path)); + let mut buffer = String::new(); + try!(file.read_to_string(&mut buffer)); + Ok(buffer) +} From 03170927ddda3a0bc4243b8a1073cd3950724e02 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 14 Sep 2016 15:15:02 -0700 Subject: [PATCH 0277/3084] Add a job queue and begin loading test files. This code looks overly complicated because it is anticipating running jobs in a thread pool. The test files are loaded and parsed, but not actually executed yet. Errors are reported back to the test runner. --- cranelift/src/tools/filetest/runner.rs | 118 +++++++++++++++++++++++-- 1 file changed, 113 insertions(+), 5 deletions(-) diff --git a/cranelift/src/tools/filetest/runner.rs b/cranelift/src/tools/filetest/runner.rs index 7f39ec6490..e7e33b735b 100644 --- a/cranelift/src/tools/filetest/runner.rs +++ b/cranelift/src/tools/filetest/runner.rs @@ -4,16 +4,36 @@ //! scanning directories for tests. use std::ffi::OsStr; -use std::path::PathBuf; +use std::path::{Path, PathBuf}; use std::error::Error; +use std::mem; +use std::panic::catch_unwind; +use std::time; use CommandResult; +use utils::read_to_string; +use cton_reader::parse_test; + +type TestResult = Result; + +#[derive(PartialEq, Eq, Debug)] +enum QueueEntry { + New(PathBuf), + Running, + Done(PathBuf, TestResult), +} pub struct TestRunner { // Directories that have not yet been scanned. dir_stack: Vec, // Filenames of tests to run. - test_list: Vec, + tests: Vec, + + // Pointer into `tests` where the `New` entries begin. + new_tests: usize, + + // Number of contiguous finished tests at the front of `tests`. + finished_tests: usize, errors: usize, } @@ -23,7 +43,9 @@ impl TestRunner { pub fn new() -> TestRunner { TestRunner { dir_stack: Vec::new(), - test_list: Vec::new(), + tests: Vec::new(), + new_tests: 0, + finished_tests: 0, errors: 0, } } @@ -40,7 +62,40 @@ impl TestRunner { /// /// Any problems reading `file` as a test case file will be reported as a test failure. pub fn push_test>(&mut self, file: P) { - self.test_list.push(file.into()); + self.tests.push(QueueEntry::New(file.into())); + } + + /// Take a new test for running as a job. + /// Leaves the queue entry marked as `Runnning`. + fn take_job(&mut self) -> Option { + let index = self.new_tests; + if index == self.tests.len() { + return None; + } + self.new_tests += 1; + + let entry = mem::replace(&mut self.tests[index], QueueEntry::Running); + if let QueueEntry::New(path) = entry { + Some(Job::new(index, path)) + } else { + // Oh, sorry about that. Put the entry back. + self.tests[index] = entry; + None + } + } + + /// Report the end of a job. + fn finish_job(&mut self, job: Job, result: TestResult) { + assert_eq!(self.tests[job.index], QueueEntry::Running); + if let Err(ref e) = result { + self.job_error(&job.path, e); + } + self.tests[job.index] = QueueEntry::Done(job.path, result); + if job.index == self.finished_tests { + while let Some(&QueueEntry::Done(_, _)) = self.tests.get(self.finished_tests) { + self.finished_tests += 1; + } + } } /// Scan any directories pushed so far. @@ -91,18 +146,35 @@ impl TestRunner { } } } + // Get the new jobs running before moving on to the next directory. + self.schedule_jobs(); } } /// Report an error related to a path. fn path_error(&mut self, path: PathBuf, err: E) { self.errors += 1; - println!("path-error {}: {}", path.to_string_lossy(), err); + println!("{}: {}", path.to_string_lossy(), err); + } + + /// Report an error related to a job. + fn job_error(&mut self, path: &Path, err: &str) { + self.errors += 1; + println!("FAIL {}: {}", path.to_string_lossy(), err); + } + + /// Schedule and new jobs to run. + fn schedule_jobs(&mut self) { + while let Some(job) = self.take_job() { + let result = job.run(); + self.finish_job(job, result); + } } /// Scan pushed directories for tests and run them. pub fn run(&mut self) -> CommandResult { self.scan_dirs(); + self.schedule_jobs(); match self.errors { 0 => Ok(()), 1 => Err("1 failure".to_string()), @@ -110,3 +182,39 @@ impl TestRunner { } } } + +/// A test file waiting to be run. +struct Job { + index: usize, + path: PathBuf, +} + +impl Job { + pub fn new(index: usize, path: PathBuf) -> Job { + Job { + index: index, + path: path, + } + } + + pub fn run(&self) -> TestResult { + match catch_unwind(|| self.run_or_panic()) { + Err(msg) => Err(format!("panic: {:?}", msg)), + Ok(result) => result, + } + } + + fn run_or_panic(&self) -> TestResult { + let started = time::Instant::now(); + let buffer = try!(read_to_string(&self.path).map_err(|e| e.to_string())); + let testfile = try!(parse_test(&buffer).map_err(|e| e.to_string())); + if testfile.commands.is_empty() { + return Err("no test commands found".to_string()); + } + if testfile.functions.is_empty() { + return Err("no functions found".to_string()); + } + // TODO: Actually run the tests. + Ok(started.elapsed()) + } +} From 7e98985ea626ac3a782f6ba76e409fa64443ad0c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 15 Sep 2016 08:32:03 -0700 Subject: [PATCH 0278/3084] Break DetailedFunction into a tuple. Use (Function, Details) in place of the aggregrate DetailedFunction. It turns out that some tests want to clone and manipulate the function while the details never change. --- cranelift/src/libreader/lib.rs | 2 +- cranelift/src/libreader/parser.rs | 92 ++++++++++++++--------------- cranelift/src/libreader/testfile.rs | 10 ++-- 3 files changed, 50 insertions(+), 54 deletions(-) diff --git a/cranelift/src/libreader/lib.rs b/cranelift/src/libreader/lib.rs index d435b4ac3b..94436e12f1 100644 --- a/cranelift/src/libreader/lib.rs +++ b/cranelift/src/libreader/lib.rs @@ -7,7 +7,7 @@ extern crate cretonne; pub use parser::{Result, parse_functions, parse_test}; pub use testcommand::{TestCommand, TestOption}; -pub use testfile::TestFile; +pub use testfile::{TestFile, Details}; mod lexer; mod parser; diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 7f8cc543ad..4a10f1cec7 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -19,7 +19,7 @@ use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::{AnyEntity, NO_EBB, NO_INST, NO_VALUE}; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, JumpData, BranchData, ReturnData}; -use testfile::{TestFile, DetailedFunction, Comment}; +use testfile::{TestFile, Details, Comment}; use testcommand::TestCommand; pub use lexer::Location; @@ -30,7 +30,7 @@ pub use lexer::Location; pub fn parse_functions(text: &str) -> Result> { Parser::new(text) .parse_function_list() - .map(|list| list.into_iter().map(|dfunc| dfunc.function).collect()) + .map(|list| list.into_iter().map(|(func, _)| func).collect()) } /// Parse the entire `text` as a test case file. @@ -505,7 +505,7 @@ impl<'a> Parser<'a> { /// Parse a list of function definitions. /// /// This is the top-level parse function matching the whole contents of a file. - pub fn parse_function_list(&mut self) -> Result>> { + pub fn parse_function_list(&mut self) -> Result)>> { let mut list = Vec::new(); while self.token().is_some() { list.push(try!(self.parse_function())); @@ -517,7 +517,7 @@ impl<'a> Parser<'a> { // // function ::= * function-spec "{" preamble function-body "}" // - fn parse_function(&mut self) -> Result> { + fn parse_function(&mut self) -> Result<(Function, Details<'a>)> { // Begin gathering comments. // Make sure we don't include any comments before the `function` keyword. self.token(); @@ -545,10 +545,7 @@ impl<'a> Parser<'a> { // references. try!(ctx.rewrite_references()); - Ok(DetailedFunction { - function: ctx.function, - comments: mem::replace(&mut self.comments, Vec::new()), - }) + Ok((ctx.function, Details { comments: mem::replace(&mut self.comments, Vec::new()) })) } // Parse a function spec. @@ -1221,7 +1218,7 @@ mod tests { use super::*; use cretonne::ir::types::{self, ArgumentType, ArgumentExtension}; use cretonne::ir::entities::AnyEntity; - use testfile::Comment; + use testfile::{Details, Comment}; #[test] fn argument_type() { @@ -1261,13 +1258,12 @@ mod tests { #[test] fn stack_slot_decl() { - let func = Parser::new("function foo() { - ss3 = stack_slot 13 - ss1 = stack_slot 1 - }") + let (func, _) = Parser::new("function foo() { + ss3 = stack_slot 13 + ss1 = stack_slot 1 + }") .parse_function() - .unwrap() - .function; + .unwrap(); assert_eq!(func.name, "foo"); let mut iter = func.stack_slots.keys(); let ss0 = iter.next().unwrap(); @@ -1291,13 +1287,12 @@ mod tests { #[test] fn ebb_header() { - let func = Parser::new("function ebbs() { - ebb0: - ebb4(vx3: i32): - }") + let (func, _) = Parser::new("function ebbs() { + ebb0: + ebb4(vx3: i32): + }") .parse_function() - .unwrap() - .function; + .unwrap(); assert_eq!(func.name, "ebbs"); let mut ebbs = func.layout.ebbs(); @@ -1314,38 +1309,39 @@ mod tests { #[test] fn comments() { - let dfunc = Parser::new("; before - function comment() { ; decl - ss10 = stack_slot 13 ; stackslot. - ; Still stackslot. - jt10 = jump_table ebb0 - ; Jumptable - ebb0: ; Basic block - trap ; Instruction - } ; Trailing. - ; More trailing.") - .parse_function() - .unwrap(); - assert_eq!(&dfunc.function.name, "comment"); - assert_eq!(dfunc.comments.len(), 8); // no 'before' comment. - assert_eq!(dfunc.comments[0], + let (func, Details { comments }) = + Parser::new("; before + function comment() { ; decl + ss10 = stack_slot 13 ; stackslot. + ; Still stackslot. + jt10 = jump_table ebb0 + ; Jumptable + ebb0: ; Basic block + trap ; Instruction + } ; Trailing. + ; More trailing.") + .parse_function() + .unwrap(); + assert_eq!(&func.name, "comment"); + assert_eq!(comments.len(), 8); // no 'before' comment. + assert_eq!(comments[0], Comment { entity: AnyEntity::Function, text: "; decl", }); - assert_eq!(dfunc.comments[1].entity.to_string(), "ss0"); - assert_eq!(dfunc.comments[2].entity.to_string(), "ss0"); - assert_eq!(dfunc.comments[2].text, "; Still stackslot."); - assert_eq!(dfunc.comments[3].entity.to_string(), "jt0"); - assert_eq!(dfunc.comments[3].text, "; Jumptable"); - assert_eq!(dfunc.comments[4].entity.to_string(), "ebb0"); - assert_eq!(dfunc.comments[4].text, "; Basic block"); + assert_eq!(comments[1].entity.to_string(), "ss0"); + assert_eq!(comments[2].entity.to_string(), "ss0"); + assert_eq!(comments[2].text, "; Still stackslot."); + assert_eq!(comments[3].entity.to_string(), "jt0"); + assert_eq!(comments[3].text, "; Jumptable"); + assert_eq!(comments[4].entity.to_string(), "ebb0"); + assert_eq!(comments[4].text, "; Basic block"); - assert_eq!(dfunc.comments[5].entity.to_string(), "inst0"); - assert_eq!(dfunc.comments[5].text, "; Instruction"); + assert_eq!(comments[5].entity.to_string(), "inst0"); + assert_eq!(comments[5].text, "; Instruction"); - assert_eq!(dfunc.comments[6].entity, AnyEntity::Function); - assert_eq!(dfunc.comments[7].entity, AnyEntity::Function); + assert_eq!(comments[6].entity, AnyEntity::Function); + assert_eq!(comments[7].entity, AnyEntity::Function); } #[test] @@ -1359,6 +1355,6 @@ mod tests { assert_eq!(tf.commands[0].command, "cfg"); assert_eq!(tf.commands[1].command, "verify"); assert_eq!(tf.functions.len(), 1); - assert_eq!(tf.functions[0].function.name, "comment"); + assert_eq!(tf.functions[0].0.name, "comment"); } } diff --git a/cranelift/src/libreader/testfile.rs b/cranelift/src/libreader/testfile.rs index 34eddb4c6b..3b4f3d4e16 100644 --- a/cranelift/src/libreader/testfile.rs +++ b/cranelift/src/libreader/testfile.rs @@ -15,14 +15,14 @@ use testcommand::TestCommand; #[derive(Debug)] pub struct TestFile<'a> { pub commands: Vec>, - pub functions: Vec>, + pub functions: Vec<(Function, Details<'a>)>, } -/// A function parsed from a text string along with other details that are useful for running -/// tests. +/// Additional details about a function parsed from a text string. +/// These are useful for detecting test commands embedded in comments etc. +/// The details to not affect the semantics of the function. #[derive(Debug)] -pub struct DetailedFunction<'a> { - pub function: Function, +pub struct Details<'a> { pub comments: Vec>, } From 0b8bf530b075085768d0161e35facc9dd6f1817b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 15 Sep 2016 08:41:19 -0700 Subject: [PATCH 0279/3084] Make functions cloneable for testing. It's not super fast to clone a function, but it is faster than re-parsing the test case file it came from. Some tests want to mutate the function, and there may be other tests in the same script that need the original function. --- cranelift/src/libcretonne/entity_map.rs | 2 +- cranelift/src/libcretonne/ir/dfg.rs | 3 +++ cranelift/src/libcretonne/ir/function.rs | 4 ++++ cranelift/src/libcretonne/ir/instructions.rs | 12 ++++++------ cranelift/src/libcretonne/ir/jumptable.rs | 1 + cranelift/src/libcretonne/ir/layout.rs | 1 + cranelift/src/libcretonne/ir/stackslot.rs | 2 +- 7 files changed, 17 insertions(+), 8 deletions(-) diff --git a/cranelift/src/libcretonne/entity_map.rs b/cranelift/src/libcretonne/entity_map.rs index df486c48ed..2a76e3cce9 100644 --- a/cranelift/src/libcretonne/entity_map.rs +++ b/cranelift/src/libcretonne/entity_map.rs @@ -45,7 +45,7 @@ pub trait EntityRef: Copy + Eq { } /// A mapping `K -> V` for densely indexed entity references. -#[derive(Debug)] +#[derive(Debug, Clone)] pub struct EntityMap where K: EntityRef { diff --git a/cranelift/src/libcretonne/ir/dfg.rs b/cranelift/src/libcretonne/ir/dfg.rs index 3fc4d596eb..35bb9c8f7e 100644 --- a/cranelift/src/libcretonne/ir/dfg.rs +++ b/cranelift/src/libcretonne/ir/dfg.rs @@ -15,6 +15,7 @@ use std::u16; /// The layout of EBBs in the function and of instructions in each EBB is recorded by the /// `FunctionLayout` data structure which form the other half of the function representation. /// +#[derive(Clone)] pub struct DataFlowGraph { /// Data about all of the instructions in the function, including opcodes and operands. /// The instructions in this map are not in program order. That is tracked by `Layout`, along @@ -119,6 +120,7 @@ pub enum ValueDef { } // Internal table storage for extended values. +#[derive(Clone)] enum ValueData { // Value is defined by an instruction, but it is not the first result. Inst { @@ -336,6 +338,7 @@ impl DataFlowGraph { // Arguments for an extended basic block are values that dominate everything in the EBB. All // branches to this EBB must provide matching arguments, and the arguments to the entry EBB must // match the function arguments. +#[derive(Clone)] struct EbbData { // First argument to this EBB, or `NO_VALUE` if the block has no arguments. // diff --git a/cranelift/src/libcretonne/ir/function.rs b/cranelift/src/libcretonne/ir/function.rs index 2f8344d801..d6a44edfa6 100644 --- a/cranelift/src/libcretonne/ir/function.rs +++ b/cranelift/src/libcretonne/ir/function.rs @@ -9,6 +9,10 @@ use entity_map::{EntityMap, PrimaryEntityData}; use std::fmt::{self, Debug, Formatter}; /// A function. +/// +/// Functions can be cloned, but it is not a very fast operation. +/// The clone will have all the same entity numbers as the original. +#[derive(Clone)] pub struct Function { /// Name of this function. Mostly used by `.cton` files. pub name: FunctionName, diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index a6d664170e..c8228d1f43 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -93,7 +93,7 @@ impl FromStr for Opcode { /// value should have its `ty` field set to `VOID`. The size of `InstructionData` should be kept at /// 16 bytes on 64-bit architectures. If more space is needed to represent an instruction, use a /// `Box` to store the additional information out of line. -#[derive(Debug)] +#[derive(Clone, Debug)] pub enum InstructionData { Nullary { opcode: Opcode, ty: Type }, Unary { @@ -203,7 +203,7 @@ pub enum InstructionData { /// A variable list of `Value` operands used for function call arguments and passing arguments to /// basic blocks. -#[derive(Debug)] +#[derive(Clone, Debug)] pub struct VariableArgs(Vec); impl VariableArgs { @@ -256,7 +256,7 @@ impl Default for VariableArgs { /// Payload data for jump instructions. These need to carry lists of EBB arguments that won't fit /// in the allowed InstructionData size. -#[derive(Debug)] +#[derive(Clone, Debug)] pub struct JumpData { pub destination: Ebb, pub arguments: VariableArgs, @@ -274,7 +274,7 @@ impl Display for JumpData { /// Payload data for branch instructions. These need to carry lists of EBB arguments that won't fit /// in the allowed InstructionData size. -#[derive(Debug)] +#[derive(Clone, Debug)] pub struct BranchData { pub arg: Value, pub destination: Ebb, @@ -292,7 +292,7 @@ impl Display for BranchData { } /// Payload of a call instruction. -#[derive(Debug)] +#[derive(Clone, Debug)] pub struct CallData { /// Second result value for a call producing multiple return values. second_result: Value, @@ -308,7 +308,7 @@ impl Display for CallData { } /// Payload of a return instruction. -#[derive(Debug)] +#[derive(Clone, Debug)] pub struct ReturnData { // Dynamically sized array containing return values. pub args: VariableArgs, diff --git a/cranelift/src/libcretonne/ir/jumptable.rs b/cranelift/src/libcretonne/ir/jumptable.rs index 8a2308fbe1..cbd8630e46 100644 --- a/cranelift/src/libcretonne/ir/jumptable.rs +++ b/cranelift/src/libcretonne/ir/jumptable.rs @@ -12,6 +12,7 @@ use std::fmt::{self, Display, Formatter}; /// /// All jump tables use 0-based indexing and are expected to be densely populated. They don't need /// to be completely populated, though. Individual entries can be missing. +#[derive(Clone)] pub struct JumpTableData { // Table entries, using NO_EBB as a placeholder for missing entries. table: Vec, diff --git a/cranelift/src/libcretonne/ir/layout.rs b/cranelift/src/libcretonne/ir/layout.rs index c519a1c44f..a872efa5d8 100644 --- a/cranelift/src/libcretonne/ir/layout.rs +++ b/cranelift/src/libcretonne/ir/layout.rs @@ -20,6 +20,7 @@ use ir::entities::{Ebb, NO_EBB, Inst, NO_INST}; /// While data dependencies are not recorded, instruction ordering does affect control /// dependencies, so part of the semantics of the program are determined by the layout. /// +#[derive(Clone)] pub struct Layout { // Linked list nodes for the layout order of EBBs Forms a doubly linked list, terminated in // both ends by NO_EBB. diff --git a/cranelift/src/libcretonne/ir/stackslot.rs b/cranelift/src/libcretonne/ir/stackslot.rs index 31bee66eda..b22b82919c 100644 --- a/cranelift/src/libcretonne/ir/stackslot.rs +++ b/cranelift/src/libcretonne/ir/stackslot.rs @@ -6,7 +6,7 @@ use std::fmt::{self, Display, Formatter}; /// Contents of a stack slot. -#[derive(Debug)] +#[derive(Clone, Debug)] pub struct StackSlotData { /// Size of stack slot in bytes. pub size: u32, From b40a3495fefac98d1b305fb59dd85607fc24e6f7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 15 Sep 2016 13:56:42 -0700 Subject: [PATCH 0280/3084] Use fmt::Write instead of io::Write in write.rs. It is common to represent a function as a String, and previously that required re-validating the UTF-8 in a Vec. The fmt::Write trait writes UTF-8 directly into a String, so no extra checking is required. This also means we can implement Display for Function which gives it a to_string() method. This makes the function_to_string() method redundant, so delete it. The functions in the write module are no longer generally useful, so make the module private. The Display trait on Function is all we need. --- cranelift/src/libcretonne/ir/function.rs | 14 ++++++++---- cranelift/src/libcretonne/lib.rs | 2 +- cranelift/src/libcretonne/write.rs | 29 ++++++++---------------- cranelift/src/tools/cat.rs | 6 +---- 4 files changed, 21 insertions(+), 30 deletions(-) diff --git a/cranelift/src/libcretonne/ir/function.rs b/cranelift/src/libcretonne/ir/function.rs index d6a44edfa6..415e8de9cc 100644 --- a/cranelift/src/libcretonne/ir/function.rs +++ b/cranelift/src/libcretonne/ir/function.rs @@ -6,7 +6,8 @@ use ir::{FunctionName, Signature, StackSlot, StackSlotData, JumpTable, JumpTableData, DataFlowGraph, Layout}; use entity_map::{EntityMap, PrimaryEntityData}; -use std::fmt::{self, Debug, Formatter}; +use std::fmt::{self, Display, Debug, Formatter}; +use write::write_function; /// A function. /// @@ -60,9 +61,14 @@ impl Function { } } -impl Debug for Function { +impl Display for Function { fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - use write::function_to_string; - fmt.write_str(&function_to_string(self)) + write_function(fmt, self) + } +} + +impl Debug for Function { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write_function(fmt, self) } } diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index f338aba8ac..7b90a902c3 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -9,13 +9,13 @@ pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); pub mod ir; pub mod isa; -pub mod write; pub mod cfg; pub mod dominator_tree; pub mod entity_map; pub mod settings; pub mod verifier; +mod write; mod constant_hash; mod predicates; diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index a20007ed05..aa5076f5d5 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -5,9 +5,8 @@ //! `cretonne-reader` crate. use ir::{Function, Ebb, Inst, Value, Type}; -use std::io::{self, Write}; - -pub type Result = io::Result<()>; +use std::fmt::{Result, Error, Write}; +use std::result; /// Write `func` to `w` as equivalent text. pub fn write_function(w: &mut Write, func: &Function) -> Result { @@ -24,15 +23,6 @@ pub fn write_function(w: &mut Write, func: &Function) -> Result { writeln!(w, "}}") } -/// Convert `func` to a string. -pub fn function_to_string(func: &Function) -> String { - let mut buffer: Vec = Vec::new(); - // Any errors here would be out-of-memory, which should not happen with normal functions. - write_function(&mut buffer, func).unwrap(); - // A UTF-8 conversion error is a real bug. - String::from_utf8(buffer).unwrap() -} - // ====--------------------------------------------------------------------------------------====// // // Function spec. @@ -65,7 +55,7 @@ fn write_spec(w: &mut Write, func: &Function) -> Result { } } -fn write_preamble(w: &mut Write, func: &Function) -> io::Result { +fn write_preamble(w: &mut Write, func: &Function) -> result::Result { let mut any = false; for ss in func.stack_slots.keys() { @@ -218,7 +208,6 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { #[cfg(test)] mod tests { - use super::*; use super::{needs_quotes, escaped}; use ir::{Function, StackSlotData}; use ir::types; @@ -244,26 +233,26 @@ mod tests { #[test] fn basic() { let mut f = Function::new(); - assert_eq!(function_to_string(&f), "function \"\"() {\n}\n"); + assert_eq!(f.to_string(), "function \"\"() {\n}\n"); f.name.push_str("foo"); - assert_eq!(function_to_string(&f), "function foo() {\n}\n"); + assert_eq!(f.to_string(), "function foo() {\n}\n"); f.stack_slots.push(StackSlotData::new(4)); - assert_eq!(function_to_string(&f), + assert_eq!(f.to_string(), "function foo() {\n ss0 = stack_slot 4\n}\n"); let ebb = f.dfg.make_ebb(); f.layout.append_ebb(ebb); - assert_eq!(function_to_string(&f), + assert_eq!(f.to_string(), "function foo() {\n ss0 = stack_slot 4\n\nebb0:\n}\n"); f.dfg.append_ebb_arg(ebb, types::I8); - assert_eq!(function_to_string(&f), + assert_eq!(f.to_string(), "function foo() {\n ss0 = stack_slot 4\n\nebb0(vx0: i8):\n}\n"); f.dfg.append_ebb_arg(ebb, types::F32.by(4).unwrap()); - assert_eq!(function_to_string(&f), + assert_eq!(f.to_string(), "function foo() {\n ss0 = stack_slot 4\n\nebb0(vx0: i8, vx1: f32x4):\n}\n"); } } diff --git a/cranelift/src/tools/cat.rs b/cranelift/src/tools/cat.rs index 566997c403..0db69b48db 100644 --- a/cranelift/src/tools/cat.rs +++ b/cranelift/src/tools/cat.rs @@ -3,11 +3,9 @@ //! Read a sequence of Cretonne IL files and print them again to stdout. This has the effect of //! normalizing formatting and removing comments. -use std::io; use CommandResult; use utils::read_to_string; use cton_reader::parse_functions; -use cretonne::write::write_function; pub fn run(files: Vec) -> CommandResult { for (i, f) in files.into_iter().enumerate() { @@ -27,9 +25,7 @@ fn cat_one(filename: String) -> CommandResult { if idx != 0 { println!(""); } - let stdout = io::stdout(); - let mut handle = stdout.lock(); - try!(write_function(&mut handle, &func).map_err(|e| format!("{}: {}", filename, e))); + print!("{}", func); } Ok(()) From 524bf154283f34fcdeb39f2a23472113bd4e2fe5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 15 Sep 2016 08:17:50 -0700 Subject: [PATCH 0281/3084] Add a SubTest trait and filecheck utilities. This trait serves as a shared interface for the different kinds of test commands the 'cton-util test' understands. Many tests produce output that is run through filecheck for validation. Provide a simple run_filecheck() function to help with this. Implement the 'test cat' sub-test which is probably the simplest possible. --- cranelift/src/libcretonne/lib.rs | 2 + cranelift/src/libcretonne/verifier.rs | 4 ++ cranelift/src/tools/cat.rs | 36 +++++++++++- cranelift/src/tools/filetest/mod.rs | 1 + cranelift/src/tools/filetest/runner.rs | 53 ++++++++++++++++- cranelift/src/tools/filetest/subtest.rs | 77 +++++++++++++++++++++++++ 6 files changed, 169 insertions(+), 4 deletions(-) create mode 100644 cranelift/src/tools/filetest/subtest.rs diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 7b90a902c3..de6cf109b5 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -5,6 +5,8 @@ // // ====------------------------------------------------------------------------------------==== // +pub use verifier::verify_function; + pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); pub mod ir; diff --git a/cranelift/src/libcretonne/verifier.rs b/cranelift/src/libcretonne/verifier.rs index e134b36285..91131ee6ef 100644 --- a/cranelift/src/libcretonne/verifier.rs +++ b/cranelift/src/libcretonne/verifier.rs @@ -55,6 +55,10 @@ use ir::{Function, ValueDef, Ebb, Inst}; use ir::instructions::InstructionFormat; +pub fn verify_function(func: &Function) -> Result<(), String> { + Verifier::new(func).run() +} + pub struct Verifier<'a> { func: &'a Function, } diff --git a/cranelift/src/tools/cat.rs b/cranelift/src/tools/cat.rs index 0db69b48db..14871b07c2 100644 --- a/cranelift/src/tools/cat.rs +++ b/cranelift/src/tools/cat.rs @@ -3,9 +3,12 @@ //! Read a sequence of Cretonne IL files and print them again to stdout. This has the effect of //! normalizing formatting and removing comments. +use std::borrow::Cow; +use cretonne::ir::Function; +use cton_reader::{parse_functions, TestCommand}; use CommandResult; use utils::read_to_string; -use cton_reader::parse_functions; +use filetest::subtest::{self, SubTest, Context, Result as STResult}; pub fn run(files: Vec) -> CommandResult { for (i, f) in files.into_iter().enumerate() { @@ -30,3 +33,34 @@ fn cat_one(filename: String) -> CommandResult { Ok(()) } + +/// Object implementing the `test cat` sub-test. +/// +/// This command is used for testing the parser and function printer. It simply parses a function +/// and prints it out again. +/// +/// The result is verified by filecheck. +struct TestCat; + +pub fn subtest(parsed: &TestCommand) -> STResult> { + assert_eq!(parsed.command, "cat"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestCat)) + } +} + +impl SubTest for TestCat { + fn name(&self) -> Cow { + Cow::from("cat") + } + + fn needs_verifier(&self) -> bool { + false + } + + fn run(&self, func: Cow, context: &Context) -> STResult<()> { + subtest::run_filecheck(&func.to_string(), context) + } +} diff --git a/cranelift/src/tools/filetest/mod.rs b/cranelift/src/tools/filetest/mod.rs index 078e287dde..c1019446ca 100644 --- a/cranelift/src/tools/filetest/mod.rs +++ b/cranelift/src/tools/filetest/mod.rs @@ -7,6 +7,7 @@ use std::path::Path; use CommandResult; use filetest::runner::TestRunner; +pub mod subtest; mod runner; /// Main entry point for `cton-util test`. diff --git a/cranelift/src/tools/filetest/runner.rs b/cranelift/src/tools/filetest/runner.rs index e7e33b735b..1f7ac7ee88 100644 --- a/cranelift/src/tools/filetest/runner.rs +++ b/cranelift/src/tools/filetest/runner.rs @@ -7,11 +7,15 @@ use std::ffi::OsStr; use std::path::{Path, PathBuf}; use std::error::Error; use std::mem; +use std::borrow::{Borrow, Cow}; use std::panic::catch_unwind; use std::time; use CommandResult; use utils::read_to_string; use cton_reader::parse_test; +use cretonne::ir::Function; +use cretonne::verify_function; +use filetest::subtest::{self, SubTest, Context}; type TestResult = Result; @@ -208,13 +212,56 @@ impl Job { let started = time::Instant::now(); let buffer = try!(read_to_string(&self.path).map_err(|e| e.to_string())); let testfile = try!(parse_test(&buffer).map_err(|e| e.to_string())); - if testfile.commands.is_empty() { - return Err("no test commands found".to_string()); - } if testfile.functions.is_empty() { return Err("no functions found".to_string()); } + // Parse the test commands. + let mut tests = + try!(testfile.commands.iter().map(subtest::new).collect::>>()); + + // Sort the tests so the mutators are at the end, and those that + // don't need the verifier are at the front + tests.sort_by_key(|st| (st.is_mutating(), st.needs_verifier())); + + // Isolate the last test in the hope that this is the only mutating test. + // If so, we can completely avoid cloning functions. + let last_test = match tests.pop() { + None => return Err("no test commands found".to_string()), + Some(t) => t, + }; + + for (func, details) in testfile.functions { + let mut context = subtest::Context { + details: details, + verified: false, + }; + + for test in &tests { + try!(self.run_one_test(test.borrow(), Cow::Borrowed(&func), &mut context)); + } + // Run the last test with an owned function which means it won't need to clone it + // before mutating. + try!(self.run_one_test(last_test.borrow(), Cow::Owned(func), &mut context)); + } + + // TODO: Actually run the tests. Ok(started.elapsed()) } + + fn run_one_test(&self, + test: &SubTest, + func: Cow, + context: &mut Context) + -> subtest::Result<()> { + let name = format!("{}({})", test.name(), func.name); + + // Should we run the verifier before this test? + if !context.verified && test.needs_verifier() { + try!(verify_function(&func)); + context.verified = true; + } + + test.run(func, context).map_err(|e| format!("{}: {}", name, e)) + } } diff --git a/cranelift/src/tools/filetest/subtest.rs b/cranelift/src/tools/filetest/subtest.rs new file mode 100644 index 0000000000..69ca47aa64 --- /dev/null +++ b/cranelift/src/tools/filetest/subtest.rs @@ -0,0 +1,77 @@ +//! SubTest trait. + +use std::result; +use std::borrow::Cow; +use cretonne::ir::Function; +use cton_reader::{TestCommand, Details}; +use filecheck::{CheckerBuilder, Checker, NO_VARIABLES}; + +pub type Result = result::Result; + +/// Create a new subcommand trait object to match `parsed.command`. +pub fn new(parsed: &TestCommand) -> Result> { + use cat; + match parsed.command { + "cat" => cat::subtest(parsed), + _ => Err(format!("unknown test command '{}'", parsed.command)), + } +} + +/// Context for running a a test on a single function. +pub struct Context<'a> { + /// Additional details about the function from the parser. + pub details: Details<'a>, + + /// Was the function verified before running this test? + pub verified: bool, +} + +/// Common interface for implementations of test commands. +/// +/// Each `.cton` test file may contain multiple test commands, each represented by a `SubTest` +/// trait object. +pub trait SubTest { + /// Name identifying this subtest. Typically the same as the test command. + fn name(&self) -> Cow; + + /// Should the verifier be run on the function before running the test? + fn needs_verifier(&self) -> bool { + true + } + + /// Does this test mutate the function when it runs? + /// This is used as a hint to avoid cloning the function needlessly. + fn is_mutating(&self) -> bool { + false + } + + /// Run this test on `func`. + fn run(&self, func: Cow, context: &Context) -> Result<()>; +} + +/// Run filecheck on `text`, using directives extracted from `context`. +pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { + let checker = try!(build_filechecker(&context.details)); + if try!(checker.check(&text, NO_VARIABLES).map_err(|e| format!("filecheck: {}", e))) { + Ok(()) + } else { + // Filecheck mismatch. Emit an explanation as output. + let (_, explain) = try!(checker.explain(&text, NO_VARIABLES) + .map_err(|e| format!("explain: {}", e))); + Err(format!("filecheck failed:\n{}{}", checker, explain)) + } +} + +/// Build a filechecker using the directives in the function's comments. +pub fn build_filechecker(details: &Details) -> Result { + let mut builder = CheckerBuilder::new(); + for comment in &details.comments { + try!(builder.directive(comment.text).map_err(|e| format!("filecheck: {}", e))); + } + let checker = builder.finish(); + if checker.is_empty() { + Err("no filecheck directives in function".to_string()) + } else { + Ok(checker) + } +} From 086bd601a2a10c722c84cce3cce1d22258f8c6ef Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 15 Sep 2016 15:27:37 -0700 Subject: [PATCH 0282/3084] Convert parser tests to filetests. Create a new directory hierarchy under 'filetests' for all the tests that are run by 'cton-util test'. Convert the parser tests under 'tests/parser' to use 'test cat' and filecheck directives. --- cranelift/filetests/parser/branch.cton | 113 ++++++++++++++++++ cranelift/filetests/parser/call.cton | 24 ++++ .../{tests => filetests}/parser/rewrite.cton | 13 ++ .../{tests => filetests}/parser/tiny.cton | 39 ++++++ cranelift/test-all.sh | 6 +- cranelift/tests/parser/README.rst | 9 -- cranelift/tests/parser/branch.cton | 60 ---------- cranelift/tests/parser/branch.cton.ref | 57 --------- cranelift/tests/parser/call.cton | 13 -- cranelift/tests/parser/call.cton.ref | 11 -- cranelift/tests/parser/rewrite.cton.ref | 13 -- cranelift/tests/parser/run.sh | 40 ------- cranelift/tests/parser/tiny.cton.ref | 43 ------- 13 files changed, 193 insertions(+), 248 deletions(-) create mode 100644 cranelift/filetests/parser/branch.cton create mode 100644 cranelift/filetests/parser/call.cton rename cranelift/{tests => filetests}/parser/rewrite.cton (68%) rename cranelift/{tests => filetests}/parser/tiny.cton (50%) delete mode 100644 cranelift/tests/parser/README.rst delete mode 100644 cranelift/tests/parser/branch.cton delete mode 100644 cranelift/tests/parser/branch.cton.ref delete mode 100644 cranelift/tests/parser/call.cton delete mode 100644 cranelift/tests/parser/call.cton.ref delete mode 100644 cranelift/tests/parser/rewrite.cton.ref delete mode 100755 cranelift/tests/parser/run.sh delete mode 100644 cranelift/tests/parser/tiny.cton.ref diff --git a/cranelift/filetests/parser/branch.cton b/cranelift/filetests/parser/branch.cton new file mode 100644 index 0000000000..46612973af --- /dev/null +++ b/cranelift/filetests/parser/branch.cton @@ -0,0 +1,113 @@ +; Parsing branches and jumps. +test cat + +; Jumps with no arguments. The '()' empty argument list is optional. +function minimal() { +ebb0: + jump ebb1 + +ebb1: + jump ebb0() +} +; sameln: function minimal() { +; nextln: ebb0: +; nextln: jump ebb1 +; nextln: +; nextln: ebb1: +; nextln: jump ebb0 +; nextln: } + +; Jumps with 1 arg. +function onearg(i32) { +ebb0(vx0: i32): + jump ebb1(vx0) + +ebb1(vx1: i32): + jump ebb0(vx1) +} +; sameln: function onearg(i32) { +; nextln: ebb0(vx0: i32): +; nextln: jump ebb1(vx0) +; nextln: +; nextln: ebb1(vx1: i32): +; nextln: jump ebb0(vx1) +; nextln: } + +; Jumps with 2 args. +function twoargs(i32, f32) { +ebb0(vx0: i32, vx1: f32): + jump ebb1(vx0, vx1) + +ebb1(vx2: i32, vx3: f32): + jump ebb0(vx2, vx3) +} +; sameln: function twoargs(i32, f32) { +; nextln: ebb0(vx0: i32, vx1: f32): +; nextln: jump ebb1(vx0, vx1) +; nextln: +; nextln: ebb1(vx2: i32, vx3: f32): +; nextln: jump ebb0(vx2, vx3) +; nextln: } + +; Branches with no arguments. The '()' empty argument list is optional. +function minimal(i32) { +ebb0(vx0: i32): + brz vx0, ebb1 + +ebb1: + brnz vx0, ebb1() +} +; sameln: function minimal(i32) { +; nextln: ebb0(vx0: i32): +; nextln: brz vx0, ebb1 +; nextln: +; nextln: ebb1: +; nextln: brnz vx0, ebb1 +; nextln: } + +function twoargs(i32, f32) { +ebb0(vx0: i32, vx1: f32): + brz vx0, ebb1(vx0, vx1) + +ebb1(vx2: i32, vx3: f32): + brnz vx0, ebb0(vx2, vx3) +} +; sameln: function twoargs(i32, f32) { +; nextln: ebb0(vx0: i32, vx1: f32): +; nextln: brz vx0, ebb1(vx0, vx1) +; nextln: +; nextln: ebb1(vx2: i32, vx3: f32): +; nextln: brnz vx0, ebb0(vx2, vx3) +; nextln: } + +function jumptable(i32) { + jt200 = jump_table 0, 0 + jt2 = jump_table 0, 0, ebb10, ebb40, ebb20, ebb30 + +ebb10(v3: i32): + br_table v3, jt2 + trap +ebb20: + trap +ebb30: + trap +ebb40: + trap +} +; sameln: function jumptable(i32) { +; nextln: jt0 = jump_table 0 +; nextln: jt1 = jump_table 0, 0, ebb0, ebb3, ebb1, ebb2 +; nextln: +; nextln: ebb0(vx0: i32): +; nextln: br_table vx0, jt1 +; nextln: trap +; nextln: +; nextln: ebb1: +; nextln: trap +; nextln: +; nextln: ebb2: +; nextln: trap +; nextln: +; nextln: ebb3: +; nextln: trap +; nextln: } diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton new file mode 100644 index 0000000000..71c7c1efb0 --- /dev/null +++ b/cranelift/filetests/parser/call.cton @@ -0,0 +1,24 @@ +; Parser tests for call and return syntax. +test cat + +function mini() { +ebb1: + return +} +; sameln: function mini() { +; nextln: ebb0: +; nextln: return +; nextln: } + +function r1() -> i32, f32 { +ebb1: + v1 = iconst.i32 3 + v2 = f32const 0.0 + return v1, v2 +} +; sameln: function r1() -> i32, f32 { +; nextln: ebb0: +; nextln: v0 = iconst.i32 3 +; nextln: v1 = f32const 0.0 +; nextln: return v0, v1 +; nextln: } diff --git a/cranelift/tests/parser/rewrite.cton b/cranelift/filetests/parser/rewrite.cton similarity index 68% rename from cranelift/tests/parser/rewrite.cton rename to cranelift/filetests/parser/rewrite.cton index 33e5277db6..969572bdbf 100644 --- a/cranelift/tests/parser/rewrite.cton +++ b/cranelift/filetests/parser/rewrite.cton @@ -6,6 +6,7 @@ ; It is possible to refer to instructions and EBBs that have not yet been ; defined in the lexical order, so the parser needs to rewrite these references ; after the fact. +test cat ; Check that defining numbers are rewritten. function defs() { @@ -14,6 +15,12 @@ ebb100(v20: i32): vx200 = f64const 0x4.0p0 trap } +; sameln: function defs() { +; nextln: ebb0(vx0: i32): +; nextln: v0 = iconst.i32x8 5 +; nextln: v1 = f64const 0x1.0000000000000p2 +; nextln: trap +; nextln: } ; Using values. function use_value() { @@ -22,3 +29,9 @@ ebb100(v20: i32): vx200 = iadd v20, v1000 jump ebb100(v1000) } +; sameln: function "use_value"() { +; nextln: ebb0(vx0: i32): +; nextln: v0 = iadd_imm vx0, 5 +; nextln: v1 = iadd vx0, v0 +; nextln: jump ebb0(v0) +; nextln: } diff --git a/cranelift/tests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton similarity index 50% rename from cranelift/tests/parser/tiny.cton rename to cranelift/filetests/parser/tiny.cton index ae3466b618..588223c521 100644 --- a/cranelift/tests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -1,8 +1,14 @@ +test cat + ; The smallest possible function. function minimal() { ebb0: trap } +; sameln: function minimal() { +; nextln: ebb0: +; nextln: trap +; nextln: } ; Create and use values. ; Polymorphic instructions with type suffix. @@ -12,12 +18,22 @@ ebb0: v1 = iconst.i8 6 v2 = ishl v0, v1 } +; sameln: function ivalues() { +; nextln: ebb0: +; nextln: v0 = iconst.i32 2 +; nextln: v1 = iconst.i8 6 +; nextln: v2 = ishl v0, v1 +; nextln: } ; Polymorphic istruction controlled by second operand. function select() { ebb0(vx0: i32, vx1: i32, vx2: b1): v0 = select vx2, vx0, vx1 } +; sameln: function select() { +; nextln: ebb0(vx0: i32, vx1: i32, vx2: b1): +; nextln: v0 = select vx2, vx0, vx1 +; nextln: } ; Lane indexes. function lanes() { @@ -26,6 +42,12 @@ ebb0: v1 = extractlane v0, 3 v2 = insertlane v0, 1, v1 } +; sameln: function lanes() { +; nextln: ebb0: +; nextln: v0 = iconst.i32x4 2 +; nextln: v1 = extractlane v0, 3 +; nextln: v2 = insertlane v0, 1, v1 +; nextln: } ; Integer condition codes. function icmp(i32, i32) { @@ -34,6 +56,12 @@ ebb0(vx0: i32, vx1: i32): v1 = icmp ult, vx0, vx1 v2 = icmp sge, vx0, vx1 } +; sameln: function icmp(i32, i32) { +; nextln: ebb0(vx0: i32, vx1: i32): +; nextln: v0 = icmp eq, vx0, vx1 +; nextln: v1 = icmp ult, vx0, vx1 +; nextln: v2 = icmp sge, vx0, vx1 +; nextln: } ; Floating condition codes. function fcmp(f32, f32) { @@ -42,6 +70,12 @@ ebb0(vx0: f32, vx1: f32): v1 = fcmp uno, vx0, vx1 v2 = fcmp lt, vx0, vx1 } +; sameln: function fcmp(f32, f32) { +; nextln: ebb0(vx0: f32, vx1: f32): +; nextln: v0 = fcmp eq, vx0, vx1 +; nextln: v1 = fcmp uno, vx0, vx1 +; nextln: v2 = fcmp lt, vx0, vx1 +; nextln: } ; The bitcast instruction has two type variables: The controlling type variable ; controls the outout type, and the input type is a free variable. @@ -50,3 +84,8 @@ ebb0(vx0: i32, vx1: f32): v0 = bitcast.i8x4 vx0 v1 = bitcast.i32 vx1 } +; sameln: function bitcast(i32, f32) { +; nextln: ebb0(vx0: i32, vx1: f32): +; nextln: v0 = bitcast.i8x4 vx0 +; nextln: v1 = bitcast.i32 vx1 +; nextln: } diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index cd3916d8c3..6c8406493a 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -58,10 +58,12 @@ cargo build --release export CTONUTIL="$topdir/src/tools/target/release/cton-util" +cd "$topdir" +banner "File tests" +"$CTONUTIL" test filetests + # Run the parser tests. cd "$topdir/tests" -banner "Parser tests" -parser/run.sh banner "CFG tests" cfg/run.sh diff --git a/cranelift/tests/parser/README.rst b/cranelift/tests/parser/README.rst deleted file mode 100644 index 9ac4658d13..0000000000 --- a/cranelift/tests/parser/README.rst +++ /dev/null @@ -1,9 +0,0 @@ -Parser tests -============ - -This directory contains test cases for the Cretonne IL parser. - -Each test case consists of a `foo.cton` input file and a `foo.ref` reference -output file. Each input file is run through the `cton-util cat` command, and the -output is compared against the reference file. If the two are identical, the -test passes. diff --git a/cranelift/tests/parser/branch.cton b/cranelift/tests/parser/branch.cton deleted file mode 100644 index 05526fd910..0000000000 --- a/cranelift/tests/parser/branch.cton +++ /dev/null @@ -1,60 +0,0 @@ -; Parsing branches and jumps. - -; Jumps with no arguments. The '()' empty argument list is optional. -function minimal() { -ebb0: - jump ebb1 - -ebb1: - jump ebb0() -} - -; Jumps with 1 arg. -function onearg(i32) { -ebb0(vx0: i32): - jump ebb1(vx0) - -ebb1(vx1: i32): - jump ebb0(vx1) -} - -; Jumps with 2 args. -function twoargs(i32, f32) { -ebb0(vx0: i32, vx1: f32): - jump ebb1(vx0, vx1) - -ebb1(vx2: i32, vx3: f32): - jump ebb0(vx2, vx3) -} - -; Branches with no arguments. The '()' empty argument list is optional. -function minimal(i32) { -ebb0(vx0: i32): - brz vx0, ebb1 - -ebb1: - brnz vx0, ebb1() -} - -function twoargs(i32, f32) { -ebb0(vx0: i32, vx1: f32): - brz vx0, ebb1(vx0, vx1) - -ebb1(vx2: i32, vx3: f32): - brnz vx0, ebb0(vx2, vx3) -} - -function jumptable(i32) { - jt200 = jump_table 0, 0 - jt2 = jump_table 0, 0, ebb10, ebb40, ebb20, ebb30 - -ebb10(v3: i32): - br_table v3, jt2 - trap -ebb20: - trap -ebb30: - trap -ebb40: - trap -} diff --git a/cranelift/tests/parser/branch.cton.ref b/cranelift/tests/parser/branch.cton.ref deleted file mode 100644 index 8f395c885c..0000000000 --- a/cranelift/tests/parser/branch.cton.ref +++ /dev/null @@ -1,57 +0,0 @@ -function minimal() { -ebb0: - jump ebb1 - -ebb1: - jump ebb0 -} - -function onearg(i32) { -ebb0(vx0: i32): - jump ebb1(vx0) - -ebb1(vx1: i32): - jump ebb0(vx1) -} - -function twoargs(i32, f32) { -ebb0(vx0: i32, vx1: f32): - jump ebb1(vx0, vx1) - -ebb1(vx2: i32, vx3: f32): - jump ebb0(vx2, vx3) -} - -function minimal(i32) { -ebb0(vx0: i32): - brz vx0, ebb1 - -ebb1: - brnz vx0, ebb1 -} - -function twoargs(i32, f32) { -ebb0(vx0: i32, vx1: f32): - brz vx0, ebb1(vx0, vx1) - -ebb1(vx2: i32, vx3: f32): - brnz vx0, ebb0(vx2, vx3) -} - -function jumptable(i32) { - jt0 = jump_table 0 - jt1 = jump_table 0, 0, ebb0, ebb3, ebb1, ebb2 - -ebb0(vx0: i32): - br_table vx0, jt1 - trap - -ebb1: - trap - -ebb2: - trap - -ebb3: - trap -} diff --git a/cranelift/tests/parser/call.cton b/cranelift/tests/parser/call.cton deleted file mode 100644 index 12221c3ff1..0000000000 --- a/cranelift/tests/parser/call.cton +++ /dev/null @@ -1,13 +0,0 @@ -; Parser tests for call and return syntax. - -function mini() { -ebb1: - return -} - -function r1() -> i32, f32 { -ebb1: - v1 = iconst.i32 3 - v2 = f32const 0.0 - return v1, v2 -} diff --git a/cranelift/tests/parser/call.cton.ref b/cranelift/tests/parser/call.cton.ref deleted file mode 100644 index a6d51166f4..0000000000 --- a/cranelift/tests/parser/call.cton.ref +++ /dev/null @@ -1,11 +0,0 @@ -function mini() { -ebb0: - return -} - -function r1() -> i32, f32 { -ebb0: - v0 = iconst.i32 3 - v1 = f32const 0.0 - return v0, v1 -} diff --git a/cranelift/tests/parser/rewrite.cton.ref b/cranelift/tests/parser/rewrite.cton.ref deleted file mode 100644 index 8f379abd64..0000000000 --- a/cranelift/tests/parser/rewrite.cton.ref +++ /dev/null @@ -1,13 +0,0 @@ -function defs() { -ebb0(vx0: i32): - v0 = iconst.i32x8 5 - v1 = f64const 0x1.0000000000000p2 - trap -} - -function "use_value"() { -ebb0(vx0: i32): - v0 = iadd_imm vx0, 5 - v1 = iadd vx0, v0 - jump ebb0(v0) -} diff --git a/cranelift/tests/parser/run.sh b/cranelift/tests/parser/run.sh deleted file mode 100755 index e5795df1da..0000000000 --- a/cranelift/tests/parser/run.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/bash - -# Go to tests directory. -cd $(dirname "$0")/.. - -# The path to cton-util should be in $CTONUTIL. -if [ -z "$CTONUTIL" ]; then - CTONUTIL=../src/tools/target/debug/cton-util -fi - -if [ ! -x "$CTONUTIL" ]; then - echo "Can't fund executable cton-util: $CTONUTIL" 1>&2 - exit 1 -fi - -declare -a fails - -for testcase in $(find parser -name '*.cton'); do - ref="${testcase}.ref" - if [ ! -r "$ref" ]; then - fails=(${fails[@]} "$testcase") - echo MISSING: $ref - elif diff -u "$ref" <("$CTONUTIL" cat "$testcase"); then - echo OK $testcase - else - fails=(${fails[@]} "$testcase") - echo FAIL $testcase - fi -done - -if [ ${#fails[@]} -ne 0 ]; then - echo - echo Failures: - for f in "${fails[@]}"; do - echo " $f" - done - exit 1 -else - echo "All passed" -fi diff --git a/cranelift/tests/parser/tiny.cton.ref b/cranelift/tests/parser/tiny.cton.ref deleted file mode 100644 index f2404bbf94..0000000000 --- a/cranelift/tests/parser/tiny.cton.ref +++ /dev/null @@ -1,43 +0,0 @@ -function minimal() { -ebb0: - trap -} - -function ivalues() { -ebb0: - v0 = iconst.i32 2 - v1 = iconst.i8 6 - v2 = ishl v0, v1 -} - -function select() { -ebb0(vx0: i32, vx1: i32, vx2: b1): - v0 = select vx2, vx0, vx1 -} - -function lanes() { -ebb0: - v0 = iconst.i32x4 2 - v1 = extractlane v0, 3 - v2 = insertlane v0, 1, v1 -} - -function icmp(i32, i32) { -ebb0(vx0: i32, vx1: i32): - v0 = icmp eq, vx0, vx1 - v1 = icmp ult, vx0, vx1 - v2 = icmp sge, vx0, vx1 -} - -function fcmp(f32, f32) { -ebb0(vx0: f32, vx1: f32): - v0 = fcmp eq, vx0, vx1 - v1 = fcmp uno, vx0, vx1 - v2 = fcmp lt, vx0, vx1 -} - -function bitcast(i32, f32) { -ebb0(vx0: i32, vx1: f32): - v0 = bitcast.i8x4 vx0 - v1 = bitcast.i32 vx1 -} From edc4fff50cde64db315b23a1f552e08b665ea1d7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 15 Sep 2016 16:07:00 -0700 Subject: [PATCH 0283/3084] Ignore test commands in parse_functions(). No point in returning a syntax error if the file contains test commands. --- cranelift/src/libreader/parser.rs | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 4a10f1cec7..63a635b3f7 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -28,9 +28,7 @@ pub use lexer::Location; /// /// Any test commands or ISA declarations are ignored. pub fn parse_functions(text: &str) -> Result> { - Parser::new(text) - .parse_function_list() - .map(|list| list.into_iter().map(|(func, _)| func).collect()) + parse_test(text).map(|file| file.functions.into_iter().map(|(func, _)| func).collect()) } /// Parse the entire `text` as a test case file. From 1342a0fb71f6310bd11e174aee3a9bc22238f6cc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 15 Sep 2016 17:10:21 -0700 Subject: [PATCH 0284/3084] Also use fmt::Write for the print-cfg command. This prepares use for implementing a 'test print-cfg' sub-test. --- cranelift/src/tools/print_cfg.rs | 166 +++++++++---------------------- 1 file changed, 48 insertions(+), 118 deletions(-) diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs index 1c57710b29..d4d6ccf79f 100644 --- a/cranelift/src/tools/print_cfg.rs +++ b/cranelift/src/tools/print_cfg.rs @@ -2,13 +2,13 @@ //! //! Read a series of Cretonne IL files and print their control flow graphs //! in graphviz format. -use std::io::{Write, stdout}; +use std::fmt::{Result, Write, Display, Formatter}; use CommandResult; use utils::read_to_string; use cretonne::ir::Function; use cretonne::cfg::ControlFlowGraph; -use cretonne::ir::instructions::InstructionData; +use cretonne::ir::instructions::BranchInfo; use cton_reader::parse_functions; pub fn run(files: Vec) -> CommandResult { @@ -21,137 +21,69 @@ pub fn run(files: Vec) -> CommandResult { Ok(()) } -struct CFGPrinter { - level: usize, - writer: T, - buffer: String, +struct CFGPrinter<'a> { + func: &'a Function, + cfg: ControlFlowGraph, } -impl CFGPrinter { - pub fn new(writer: T) -> CFGPrinter { +impl<'a> CFGPrinter<'a> { + pub fn new(func: &'a Function) -> CFGPrinter<'a> { CFGPrinter { - level: 0, - writer: writer, - buffer: String::new(), + func: func, + cfg: ControlFlowGraph::new(func), } } - pub fn print(&mut self, func: &Function) -> Result<(), String> { - self.level = 0; - self.header(func); - self.push_indent(); - self.ebb_subgraphs(func); - let cfg = ControlFlowGraph::new(func); - self.cfg_connections(func, &cfg); - self.pop_indent(); - self.footer(); - self.write() + /// Write the CFG for this function to `w`. + pub fn write(&self, w: &mut Write) -> Result { + try!(self.header(w)); + try!(self.ebb_nodes(w)); + try!(self.cfg_connections(w)); + writeln!(w, "}}") } - fn write(&mut self) -> Result<(), String> { - match self.writer.write(self.buffer.as_bytes()) { - Err(_) => return Err("Write failed!".to_string()), - _ => (), - }; - match self.writer.flush() { - Err(_) => return Err("Flush failed!".to_string()), - _ => (), - }; + fn header(&self, w: &mut Write) -> Result { + try!(writeln!(w, "digraph {} {{", self.func.name)); + if let Some(entry) = self.func.layout.entry_block() { + try!(writeln!(w, " {{rank=min; {}}}", entry)); + } Ok(()) } - fn append(&mut self, s: &str) { - let mut indent = String::new(); - for _ in 0..self.level { - indent = indent + " "; - } - self.buffer.push_str(&(indent + s)); - } - - fn push_indent(&mut self) { - self.level += 1; - } - - fn pop_indent(&mut self) { - if self.level > 0 { - self.level -= 1; - } - } - - fn open_paren(&mut self) { - self.append("{"); - } - - fn close_paren(&mut self) { - self.append("}"); - } - - fn newline(&mut self) { - self.append("\n"); - } - - fn header(&mut self, func: &Function) { - self.append(&format!("digraph {} ", func.name)); - self.open_paren(); - self.newline(); - self.push_indent(); - self.append("{rank=min; ebb0}"); - self.pop_indent(); - self.newline(); - } - - fn footer(&mut self) { - self.close_paren(); - self.newline(); - } - - fn ebb_subgraphs(&mut self, func: &Function) { - for ebb in &func.layout { - let inst_data = func.layout - .ebb_insts(ebb) - .filter(|inst| { - match func.dfg[*inst] { - InstructionData::Branch { ty: _, opcode: _, data: _ } => true, - InstructionData::Jump { ty: _, opcode: _, data: _ } => true, - _ => false, + fn ebb_nodes(&self, w: &mut Write) -> Result { + for ebb in &self.func.layout { + try!(write!(w, " {} [shape=record, label=\"{{{}", ebb, ebb)); + // Add all outgoing branch instructions to the label. + for inst in self.func.layout.ebb_insts(ebb) { + let idata = &self.func.dfg[inst]; + match idata.analyze_branch() { + BranchInfo::SingleDest(dest, _) => { + try!(write!(w, " | <{}>{} {}", inst, idata.opcode(), dest)) } - }) - .map(|inst| { - let op = match func.dfg[inst] { - InstructionData::Branch { ty: _, opcode, ref data } => { - Some((opcode, data.destination)) - } - InstructionData::Jump { ty: _, opcode, ref data } => { - Some((opcode, data.destination)) - } - _ => None, - }; - (inst, op) - }) - .collect::>(); - - let mut insts = vec![format!("{}", ebb)]; - for (inst, data) in inst_data { - let (op, dest) = data.unwrap(); - insts.push(format!("<{}>{} {}", inst, op, dest)); + BranchInfo::Table(table) => { + try!(write!(w, " | <{}>{} {}", inst, idata.opcode(), table)) + } + BranchInfo::NotABranch => {} + } } - - self.append(&format!("{} [shape=record, label=\"{}{}{}\"]", - ebb, - "{", - insts.join(" | "), - "}")); - self.newline(); + try!(writeln!(w, "}}\"]")) } + Ok(()) } - fn cfg_connections(&mut self, func: &Function, cfg: &ControlFlowGraph) { - for ebb in &func.layout { - for &(parent, inst) in cfg.get_predecessors(ebb) { - self.append(&format!("{}:{} -> {}", parent, inst, ebb)); - self.newline(); + fn cfg_connections(&self, w: &mut Write) -> Result { + for ebb in &self.func.layout { + for &(parent, inst) in self.cfg.get_predecessors(ebb) { + try!(writeln!(w, " {}:{} -> {}", parent, inst, ebb)); } } + Ok(()) + } +} + +impl<'a> Display for CFGPrinter<'a> { + fn fmt(&self, f: &mut Formatter) -> Result { + self.write(f) } } @@ -159,13 +91,11 @@ fn print_cfg(filename: String) -> CommandResult { let buffer = try!(read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))); let items = try!(parse_functions(&buffer).map_err(|e| format!("{}: {}", filename, e))); - let mut cfg_printer = CFGPrinter::new(stdout()); for (idx, func) in items.into_iter().enumerate() { if idx != 0 { println!(""); } - - try!(cfg_printer.print(&func)); + print!("{}", CFGPrinter::new(&func)); } Ok(()) From 5ac30b00754bdd8f7d359fb8e875b24115d0a3bd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 15 Sep 2016 17:20:39 -0700 Subject: [PATCH 0285/3084] Implement the 'test print-cfg' sub-test. Move the CFG tests into the filetests directory. Remove the tests directory, there are no more shell-driven tests left. --- cranelift/{tests => filetests}/cfg/loop.cton | 1 + .../{tests => filetests}/cfg/traps_early.cton | 1 + .../{tests => filetests}/cfg/unused_node.cton | 1 + cranelift/src/tools/filetest/subtest.rs | 2 ++ cranelift/src/tools/print_cfg.rs | 31 +++++++++++++++- cranelift/test-all.sh | 5 --- cranelift/tests/cfg/README.rst | 14 -------- cranelift/tests/cfg/run.sh | 36 ------------------- 8 files changed, 35 insertions(+), 56 deletions(-) rename cranelift/{tests => filetests}/cfg/loop.cton (98%) rename cranelift/{tests => filetests}/cfg/traps_early.cton (96%) rename cranelift/{tests => filetests}/cfg/unused_node.cton (97%) delete mode 100644 cranelift/tests/cfg/README.rst delete mode 100755 cranelift/tests/cfg/run.sh diff --git a/cranelift/tests/cfg/loop.cton b/cranelift/filetests/cfg/loop.cton similarity index 98% rename from cranelift/tests/cfg/loop.cton rename to cranelift/filetests/cfg/loop.cton index fd6e4fa6b9..5732ac593a 100644 --- a/cranelift/tests/cfg/loop.cton +++ b/cranelift/filetests/cfg/loop.cton @@ -1,4 +1,5 @@ ; For testing cfg generation. This code is nonsense. +test print-cfg function nonsense(i32, i32) -> f32 { ; check: digraph nonsense { diff --git a/cranelift/tests/cfg/traps_early.cton b/cranelift/filetests/cfg/traps_early.cton similarity index 96% rename from cranelift/tests/cfg/traps_early.cton rename to cranelift/filetests/cfg/traps_early.cton index 71070f4b15..a52cac1ba4 100644 --- a/cranelift/tests/cfg/traps_early.cton +++ b/cranelift/filetests/cfg/traps_early.cton @@ -1,5 +1,6 @@ ; For testing cfg generation. This code explores the implications of encountering ; a terminating instruction before any connections have been made. +test print-cfg function nonsense(i32) { ; check: digraph nonsense { diff --git a/cranelift/tests/cfg/unused_node.cton b/cranelift/filetests/cfg/unused_node.cton similarity index 97% rename from cranelift/tests/cfg/unused_node.cton rename to cranelift/filetests/cfg/unused_node.cton index 693196ccde..d4be2deddd 100644 --- a/cranelift/tests/cfg/unused_node.cton +++ b/cranelift/filetests/cfg/unused_node.cton @@ -1,4 +1,5 @@ ; For testing cfg generation where some block is never reached. +test print-cfg function not_reached(i32) -> i32 { ; check: digraph not_reached { diff --git a/cranelift/src/tools/filetest/subtest.rs b/cranelift/src/tools/filetest/subtest.rs index 69ca47aa64..f87eda9b98 100644 --- a/cranelift/src/tools/filetest/subtest.rs +++ b/cranelift/src/tools/filetest/subtest.rs @@ -11,8 +11,10 @@ pub type Result = result::Result; /// Create a new subcommand trait object to match `parsed.command`. pub fn new(parsed: &TestCommand) -> Result> { use cat; + use print_cfg; match parsed.command { "cat" => cat::subtest(parsed), + "print-cfg" => print_cfg::subtest(parsed), _ => Err(format!("unknown test command '{}'", parsed.command)), } } diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/tools/print_cfg.rs index d4d6ccf79f..858ef4de23 100644 --- a/cranelift/src/tools/print_cfg.rs +++ b/cranelift/src/tools/print_cfg.rs @@ -2,14 +2,17 @@ //! //! Read a series of Cretonne IL files and print their control flow graphs //! in graphviz format. + +use std::borrow::Cow; use std::fmt::{Result, Write, Display, Formatter}; use CommandResult; use utils::read_to_string; +use filetest::subtest::{self, SubTest, Context, Result as STResult}; use cretonne::ir::Function; use cretonne::cfg::ControlFlowGraph; use cretonne::ir::instructions::BranchInfo; -use cton_reader::parse_functions; +use cton_reader::{parse_functions, TestCommand}; pub fn run(files: Vec) -> CommandResult { for (i, f) in files.into_iter().enumerate() { @@ -100,3 +103,29 @@ fn print_cfg(filename: String) -> CommandResult { Ok(()) } + +/// Object implementing the `test print-cfg` sub-test. +struct TestPrintCfg; + +pub fn subtest(parsed: &TestCommand) -> STResult> { + assert_eq!(parsed.command, "print-cfg"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestPrintCfg)) + } +} + +impl SubTest for TestPrintCfg { + fn name(&self) -> Cow { + Cow::from("print-cfg") + } + + fn needs_verifier(&self) -> bool { + false + } + + fn run(&self, func: Cow, context: &Context) -> STResult<()> { + subtest::run_filecheck(&CFGPrinter::new(&func).to_string(), context) + } +} diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 6c8406493a..87aea4573a 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -62,9 +62,4 @@ cd "$topdir" banner "File tests" "$CTONUTIL" test filetests -# Run the parser tests. -cd "$topdir/tests" -banner "CFG tests" -cfg/run.sh - banner "OK" diff --git a/cranelift/tests/cfg/README.rst b/cranelift/tests/cfg/README.rst deleted file mode 100644 index c1ee4b1fa3..0000000000 --- a/cranelift/tests/cfg/README.rst +++ /dev/null @@ -1,14 +0,0 @@ -CFG tests -============ - -This directory contains test cases for the Cretonne cfg printer. - -Each test case consists of a `foo.cton` input file annotated with its expected connections. -Annotations are comments of the form: `ebbx:insty -> ebbz` where ebbx is connected to ebbz via -a branch or jump instruction at line y. Instructions are labeled by line number starting from zero: `inst0` .. `instn`. - - -Each input file is run through the `cton-util print-cfg` command and the -output is compared against the specially formatted comments to ensure that -expected connections exist. This scheme allows for changes to graph style -without the need to update tests. diff --git a/cranelift/tests/cfg/run.sh b/cranelift/tests/cfg/run.sh deleted file mode 100755 index cd324cc73f..0000000000 --- a/cranelift/tests/cfg/run.sh +++ /dev/null @@ -1,36 +0,0 @@ -#!/bin/bash - -# Go to tests directory. -cd $(dirname "$0")/.. - -# The path to cton-util should be in $CTONUTIL. -if [ -z "$CTONUTIL" ]; then - CTONUTIL=../src/tools/target/debug/cton-util -fi - -if [ ! -x "$CTONUTIL" ]; then - echo "Can't fund executable cton-util: $CTONUTIL" 1>&2 - exit 1 -fi - -declare -a fails - -for testcase in $(find cfg -name '*.cton'); do - if "${CTONUTIL}" print-cfg "$testcase" | "${CTONUTIL}" filecheck "$testcase"; then - echo OK $testcase - else - fails=(${fails[@]} "$testcase") - echo FAIL $testcase - fi -done - -if [ ${#fails[@]} -ne 0 ]; then - echo - echo Failures: - for f in "${fails[@]}"; do - echo " $f" - done - exit 1 -else - echo "All passed" -fi From fb16762866d332013f248a606c143c9d0cd475a4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 15 Sep 2016 17:28:09 -0700 Subject: [PATCH 0286/3084] Print the number of tests run. Don't let 'cton-util test' finish without printing anything. --- cranelift/src/tools/filetest/runner.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/cranelift/src/tools/filetest/runner.rs b/cranelift/src/tools/filetest/runner.rs index 1f7ac7ee88..ceb1e34c62 100644 --- a/cranelift/src/tools/filetest/runner.rs +++ b/cranelift/src/tools/filetest/runner.rs @@ -179,6 +179,7 @@ impl TestRunner { pub fn run(&mut self) -> CommandResult { self.scan_dirs(); self.schedule_jobs(); + println!("{} tests", self.tests.len()); match self.errors { 0 => Ok(()), 1 => Err("1 failure".to_string()), From d5ed27cce63b1f724e39665dc8294487a009c6b2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Sep 2016 09:36:34 -0700 Subject: [PATCH 0287/3084] Use Cow for the values of filecheck variables. Often, an implementation of VariableMap can return references to internal strings, and Cow::Borrow() allows that without making any copies. We still want to allow VariableMap implementations to return owned strings in case they have to manufacture variable values on demand. The Cow::Owned() variant does exactly that. Switch the internal VariableMap implementations over to Cows. It turns out they can simply store references to substrings of the input test, completely avoiding string allocation for script-defined variables. --- cranelift/src/libfilecheck/checker.rs | 11 ++++++----- cranelift/src/libfilecheck/pattern.rs | 21 ++++++++------------- cranelift/src/libfilecheck/variable.rs | 8 +++++--- 3 files changed, 19 insertions(+), 21 deletions(-) diff --git a/cranelift/src/libfilecheck/checker.rs b/cranelift/src/libfilecheck/checker.rs index 6bb458b55e..a2af0815e7 100644 --- a/cranelift/src/libfilecheck/checker.rs +++ b/cranelift/src/libfilecheck/checker.rs @@ -2,6 +2,7 @@ use error::{Error, Result}; use variable::{VariableMap, Value, varname_prefix}; use pattern::Pattern; use regex::{Regex, Captures}; +use std::borrow::Cow; use std::collections::HashMap; use std::cmp::max; use std::fmt::{self, Display, Formatter}; @@ -183,7 +184,7 @@ impl Checker { Directive::Regex(ref var, ref rx) => { state.vars.insert(var.clone(), VarDef { - value: Value::Regex(rx.clone()), + value: Value::Regex(Cow::Borrowed(rx)), offset: 0, }); continue; @@ -234,9 +235,9 @@ impl Checker { } /// A local definition of a variable. -pub struct VarDef { +pub struct VarDef<'a> { /// The value given to the variable. - value: Value, + value: Value<'a>, /// Offset in input text from where the variable is available. offset: usize, } @@ -246,7 +247,7 @@ struct State<'a> { env_vars: &'a VariableMap, recorder: &'a mut Recorder, - vars: HashMap, + vars: HashMap>, // Offset after the last ordered match. This does not include recent unordered matches. last_ordered: usize, // Largest offset following a positive match, including unordered matches. @@ -331,7 +332,7 @@ impl<'a> State<'a> { let txtval = caps.name(var).unwrap_or(""); self.recorder.defined_var(var, txtval); let vardef = VarDef { - value: Value::Text(txtval.to_string()), + value: Value::Text(Cow::Borrowed(txtval)), // This offset is the end of the whole matched pattern, not just the text // defining the variable. offset: range.0 + matched_range.1, diff --git a/cranelift/src/libfilecheck/pattern.rs b/cranelift/src/libfilecheck/pattern.rs index 69464bd7c8..88c47011f9 100644 --- a/cranelift/src/libfilecheck/pattern.rs +++ b/cranelift/src/libfilecheck/pattern.rs @@ -330,19 +330,14 @@ impl Pattern { Part::DefLit { ref regex, .. } => out.push_str(regex), Part::DefVar { def, ref var } => { // Wrap regex in a named capture group. - write!(out, - "(?P<{}>{})", - self.defs[def], - match vmap.lookup(var) { - None => { - return Err(Error::UndefVariable(format!("undefined variable \ - ${}", - var))) - } - Some(Value::Text(s)) => quote(&s), - Some(Value::Regex(rx)) => rx, - }) - .unwrap() + write!(out, "(?P<{}>", self.defs[def]).unwrap(); + match vmap.lookup(var) { + None => { + return Err(Error::UndefVariable(format!("undefined variable ${}", var))) + } + Some(Value::Text(s)) => write!(out, "{})", quote(&s[..])).unwrap(), + Some(Value::Regex(rx)) => write!(out, "{})", rx).unwrap(), + } } } diff --git a/cranelift/src/libfilecheck/variable.rs b/cranelift/src/libfilecheck/variable.rs index 7238249146..51bc38fc07 100644 --- a/cranelift/src/libfilecheck/variable.rs +++ b/cranelift/src/libfilecheck/variable.rs @@ -1,3 +1,5 @@ +use std::borrow::Cow; + /// A variable name is one or more ASCII alphanumerical characters, including underscore. /// Note that numerical variable names like `$45` are allowed too. /// @@ -16,9 +18,9 @@ pub fn varname_prefix(s: &str) -> usize { /// A variable can contain either a regular expression or plain text. #[derive(Debug, Clone, PartialEq, Eq)] -pub enum Value { - Text(String), - Regex(String), +pub enum Value<'a> { + Text(Cow<'a, str>), + Regex(Cow<'a, str>), } /// Resolve variables by name. From 3796be696fc45c96e441eaaa60338c53771f77e5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Sep 2016 12:14:23 -0700 Subject: [PATCH 0288/3084] Add a SourceMap to libreader. Give crate clients the possiblility of mapping source-level entity names to proper entity references that are valid in the parsed function. --- cranelift/src/libreader/lib.rs | 2 + cranelift/src/libreader/parser.rs | 12 ++- cranelift/src/libreader/sourcemap.rs | 133 +++++++++++++++++++++++++++ cranelift/src/libreader/testfile.rs | 2 + 4 files changed, 146 insertions(+), 3 deletions(-) create mode 100644 cranelift/src/libreader/sourcemap.rs diff --git a/cranelift/src/libreader/lib.rs b/cranelift/src/libreader/lib.rs index 94436e12f1..79f9b23521 100644 --- a/cranelift/src/libreader/lib.rs +++ b/cranelift/src/libreader/lib.rs @@ -8,8 +8,10 @@ extern crate cretonne; pub use parser::{Result, parse_functions, parse_test}; pub use testcommand::{TestCommand, TestOption}; pub use testfile::{TestFile, Details}; +pub use sourcemap::SourceMap; mod lexer; mod parser; mod testcommand; mod testfile; +mod sourcemap; diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 63a635b3f7..54a53310ee 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -21,6 +21,7 @@ use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArg BranchData, ReturnData}; use testfile::{TestFile, Details, Comment}; use testcommand::TestCommand; +use sourcemap; pub use lexer::Location; @@ -539,11 +540,16 @@ impl<'a> Parser<'a> { self.token(); self.comment_entity = None; - // Rewrite references to values and EBBs after parsing everuthing to allow forward + // Rewrite references to values and EBBs after parsing everything to allow forward // references. try!(ctx.rewrite_references()); - Ok((ctx.function, Details { comments: mem::replace(&mut self.comments, Vec::new()) })) + let details = Details { + comments: mem::replace(&mut self.comments, Vec::new()), + map: sourcemap::new(ctx.values, ctx.ebbs, ctx.stack_slots, ctx.jump_tables), + }; + + Ok((ctx.function, details)) } // Parse a function spec. @@ -1307,7 +1313,7 @@ mod tests { #[test] fn comments() { - let (func, Details { comments }) = + let (func, Details { comments, .. }) = Parser::new("; before function comment() { ; decl ss10 = stack_slot 13 ; stackslot. diff --git a/cranelift/src/libreader/sourcemap.rs b/cranelift/src/libreader/sourcemap.rs new file mode 100644 index 0000000000..9a601a3e39 --- /dev/null +++ b/cranelift/src/libreader/sourcemap.rs @@ -0,0 +1,133 @@ +//! Source map for translating source entity names to parsed entities. +//! +//! When the parser reads in a source file, entities like instructions, EBBs, and values get new +//! entity numbers. The parser maintains a mapping from the entity names in the source to the final +//! entity references. +//! +//! The `SourceMap` struct defined in this module makes the same mapping available to parser +//! clients. + +use std::collections::HashMap; +use cretonne::ir::{StackSlot, JumpTable, Ebb, Value}; +use cretonne::ir::entities::AnyEntity; + +/// Mapping from source entity names to entity references that are valid in the parsed function. +#[derive(Debug)] +pub struct SourceMap { + values: HashMap, // vNN, vxNN + ebbs: HashMap, // ebbNN + stack_slots: HashMap, // ssNN + jump_tables: HashMap, // jtNN +} + +/// Read-only interface which is exposed outside the parser crate. +impl SourceMap { + /// Look up an entity by source name. + /// Returns the entity reference corresponding to `name`, if it exists. + pub fn lookup_str(&self, name: &str) -> Option { + split_entity_name(name).and_then(|(ent, num)| { + match ent { + "v" => { + Value::direct_with_number(num) + .and_then(|v| self.values.get(&v).cloned()) + .map(AnyEntity::Value) + } + "vx" => { + Value::table_with_number(num) + .and_then(|v| self.values.get(&v).cloned()) + .map(AnyEntity::Value) + } + "ebb" => { + Ebb::with_number(num) + .and_then(|e| self.ebbs.get(&e).cloned()) + .map(AnyEntity::Ebb) + } + "ss" => self.stack_slots.get(&num).cloned().map(AnyEntity::StackSlot), + "jt" => self.jump_tables.get(&num).cloned().map(AnyEntity::JumpTable), + _ => None, + } + }) + } +} + +/// Get the number of decimal digits at the end of `s`. +fn trailing_digits(s: &str) -> usize { + // It's faster to iterate backwards over bytes, and we're only counting ASCII digits. + s.as_bytes().iter().rev().cloned().take_while(|&b| b'0' <= b && b <= b'9').count() +} + +/// Pre-parse a supposed entity name by splitting it into two parts: A head of lowercase ASCII +/// letters and numeric tail. +fn split_entity_name(name: &str) -> Option<(&str, u32)> { + let (head, tail) = name.split_at(name.len() - trailing_digits(name)); + if tail.len() > 1 && tail.starts_with('0') { + None + } else { + tail.parse().ok().map(|n| (head, n)) + } +} + +/// Create a new SourceMap from all the individual mappings. +pub fn new(values: HashMap, + ebbs: HashMap, + stack_slots: HashMap, + jump_tables: HashMap) + -> SourceMap { + SourceMap { + values: values, + ebbs: ebbs, + stack_slots: stack_slots, + jump_tables: jump_tables, + } +} + +#[cfg(test)] +mod tests { + use super::{trailing_digits, split_entity_name}; + use parse_test; + + #[test] + fn digits() { + assert_eq!(trailing_digits(""), 0); + assert_eq!(trailing_digits("x"), 0); + assert_eq!(trailing_digits("0x"), 0); + assert_eq!(trailing_digits("x1"), 1); + assert_eq!(trailing_digits("1x1"), 1); + assert_eq!(trailing_digits("1x01"), 2); + } + + #[test] + fn entity_name() { + assert_eq!(split_entity_name(""), None); + assert_eq!(split_entity_name("x"), None); + assert_eq!(split_entity_name("x+"), None); + assert_eq!(split_entity_name("x+1"), Some(("x+", 1))); + assert_eq!(split_entity_name("x-1"), Some(("x-", 1))); + assert_eq!(split_entity_name("1"), Some(("", 1))); + assert_eq!(split_entity_name("x1"), Some(("x", 1))); + assert_eq!(split_entity_name("xy0"), Some(("xy", 0))); + // Reject this non-canonical form. + assert_eq!(split_entity_name("inst01"), None); + } + + #[test] + fn details() { + let tf = parse_test("function detail() { + ss10 = stack_slot 13 + jt10 = jump_table ebb0 + ebb0(v4: i32, vx7: i32): + v10 = iadd v4, vx7 + }") + .unwrap(); + let map = &tf.functions[0].1.map; + + assert_eq!(map.lookup_str("v0"), None); + assert_eq!(map.lookup_str("ss1"), None); + assert_eq!(map.lookup_str("ss10").unwrap().to_string(), "ss0"); + assert_eq!(map.lookup_str("jt10").unwrap().to_string(), "jt0"); + assert_eq!(map.lookup_str("ebb0").unwrap().to_string(), "ebb0"); + assert_eq!(map.lookup_str("v4").unwrap().to_string(), "vx0"); + assert_eq!(map.lookup_str("vx7").unwrap().to_string(), "vx1"); + assert_eq!(map.lookup_str("v10").unwrap().to_string(), "v0"); + } +} diff --git a/cranelift/src/libreader/testfile.rs b/cranelift/src/libreader/testfile.rs index 3b4f3d4e16..34cfabce83 100644 --- a/cranelift/src/libreader/testfile.rs +++ b/cranelift/src/libreader/testfile.rs @@ -7,6 +7,7 @@ use cretonne::ir::Function; use cretonne::ir::entities::AnyEntity; use testcommand::TestCommand; +use sourcemap::SourceMap; /// A parsed test case. /// @@ -24,6 +25,7 @@ pub struct TestFile<'a> { #[derive(Debug)] pub struct Details<'a> { pub comments: Vec>, + pub map: SourceMap, } /// A comment in a parsed function. From ec6a9df08a337c53139e133a603b3f2b167f8d03 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Sep 2016 12:49:49 -0700 Subject: [PATCH 0289/3084] Make the source map available as filecheck variables. This makes it possible to refer to entities defined in the source file, using the source names prefixed with $. For example, $v20 refers to the value by that name in the sources, even if it was renumbered to 'vx0' in the parsed file. --- cranelift/filetests/parser/rewrite.cton | 6 +++--- cranelift/src/tools/filetest/subtest.rs | 19 ++++++++++++++++--- 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/cranelift/filetests/parser/rewrite.cton b/cranelift/filetests/parser/rewrite.cton index 969572bdbf..48ad84869c 100644 --- a/cranelift/filetests/parser/rewrite.cton +++ b/cranelift/filetests/parser/rewrite.cton @@ -16,9 +16,9 @@ ebb100(v20: i32): trap } ; sameln: function defs() { -; nextln: ebb0(vx0: i32): -; nextln: v0 = iconst.i32x8 5 -; nextln: v1 = f64const 0x1.0000000000000p2 +; nextln: $ebb100($v20: i32): +; nextln: $v1000 = iconst.i32x8 5 +; nextln: $vx200 = f64const 0x1.0000000000000p2 ; nextln: trap ; nextln: } diff --git a/cranelift/src/tools/filetest/subtest.rs b/cranelift/src/tools/filetest/subtest.rs index f87eda9b98..89a8b830e1 100644 --- a/cranelift/src/tools/filetest/subtest.rs +++ b/cranelift/src/tools/filetest/subtest.rs @@ -4,7 +4,7 @@ use std::result; use std::borrow::Cow; use cretonne::ir::Function; use cton_reader::{TestCommand, Details}; -use filecheck::{CheckerBuilder, Checker, NO_VARIABLES}; +use filecheck::{self, CheckerBuilder, Checker, Value as FCValue}; pub type Result = result::Result; @@ -51,14 +51,27 @@ pub trait SubTest { fn run(&self, func: Cow, context: &Context) -> Result<()>; } +/// Make the parser's source map available as filecheck variables. +/// +/// This means that the filecheck directives can refer to entities like `jump $ebb3`, where `$ebb3` +/// will expand to the EBB number that was assigned to `ebb3` in the input source. +/// +/// The expanded entity names are wrapped in word boundary regex guards so that 'inst1' doesn't +/// match 'inst10'. +impl<'a> filecheck::VariableMap for Context<'a> { + fn lookup(&self, varname: &str) -> Option { + self.details.map.lookup_str(varname).map(|e| FCValue::Regex(format!(r"\b{}\b", e).into())) + } +} + /// Run filecheck on `text`, using directives extracted from `context`. pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { let checker = try!(build_filechecker(&context.details)); - if try!(checker.check(&text, NO_VARIABLES).map_err(|e| format!("filecheck: {}", e))) { + if try!(checker.check(&text, context).map_err(|e| format!("filecheck: {}", e))) { Ok(()) } else { // Filecheck mismatch. Emit an explanation as output. - let (_, explain) = try!(checker.explain(&text, NO_VARIABLES) + let (_, explain) = try!(checker.explain(&text, context) .map_err(|e| format!("explain: {}", e))); Err(format!("filecheck failed:\n{}{}", checker, explain)) } From 770285d5c699f778697e846dcb1ff76cf107ef78 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Sep 2016 14:47:47 -0700 Subject: [PATCH 0290/3084] Implement a domtree sub-test. This test verifies the computed dominator tree against annotations. Move the existing testcases into filetests/ with the new syntax. --- cranelift/filetests/domtree/basic.cton | 13 +++ .../domtree}/loops.cton | 8 +- .../domtree}/loops2.cton | 14 ++- cranelift/filetests/domtree/tall-tree.cton | 33 ++++++ cranelift/filetests/domtree/wide-tree.cton | 41 +++++++ cranelift/src/tools/filetest/domtree.rs | 103 ++++++++++++++++++ cranelift/src/tools/filetest/mod.rs | 1 + cranelift/src/tools/filetest/subtest.rs | 2 + cranelift/src/tools/tests/dominator_tree.rs | 96 ---------------- .../tests/dominator_tree_testdata/basic.cton | 11 -- .../dominator_tree_testdata/tall-tree.cton | 31 ------ .../dominator_tree_testdata/wide-tree.cton | 39 ------- cranelift/src/tools/utils.rs | 26 +++++ 13 files changed, 232 insertions(+), 186 deletions(-) create mode 100644 cranelift/filetests/domtree/basic.cton rename cranelift/{src/tools/tests/dominator_tree_testdata => filetests/domtree}/loops.cton (63%) rename cranelift/{src/tools/tests/dominator_tree_testdata => filetests/domtree}/loops2.cton (55%) create mode 100644 cranelift/filetests/domtree/tall-tree.cton create mode 100644 cranelift/filetests/domtree/wide-tree.cton create mode 100644 cranelift/src/tools/filetest/domtree.rs delete mode 100644 cranelift/src/tools/tests/dominator_tree.rs delete mode 100644 cranelift/src/tools/tests/dominator_tree_testdata/basic.cton delete mode 100644 cranelift/src/tools/tests/dominator_tree_testdata/tall-tree.cton delete mode 100644 cranelift/src/tools/tests/dominator_tree_testdata/wide-tree.cton diff --git a/cranelift/filetests/domtree/basic.cton b/cranelift/filetests/domtree/basic.cton new file mode 100644 index 0000000000..0b8536effd --- /dev/null +++ b/cranelift/filetests/domtree/basic.cton @@ -0,0 +1,13 @@ +test domtree + +function test(i32) { + ebb0(v0: i32): + jump ebb1 ; dominates: ebb1 + ebb1: + brz v0, ebb3 ; dominates: ebb3 + jump ebb2 ; dominates: ebb2 + ebb2: + jump ebb3 + ebb3: + return +} diff --git a/cranelift/src/tools/tests/dominator_tree_testdata/loops.cton b/cranelift/filetests/domtree/loops.cton similarity index 63% rename from cranelift/src/tools/tests/dominator_tree_testdata/loops.cton rename to cranelift/filetests/domtree/loops.cton index 87d7780d7b..cdc88544cc 100644 --- a/cranelift/src/tools/tests/dominator_tree_testdata/loops.cton +++ b/cranelift/filetests/domtree/loops.cton @@ -1,7 +1,9 @@ +test domtree + function test(i32) { - ebb0(v0: i32): ; dominates(0) - brz v0, ebb1 ; dominates(1,3,4,5) - jump ebb2 ; dominates(2) + ebb0(v0: i32): + brz v0, ebb1 ; dominates: ebb1 ebb3 ebb4 ebb5 + jump ebb2 ; dominates: ebb2 ebb1: jump ebb3 ebb2: diff --git a/cranelift/src/tools/tests/dominator_tree_testdata/loops2.cton b/cranelift/filetests/domtree/loops2.cton similarity index 55% rename from cranelift/src/tools/tests/dominator_tree_testdata/loops2.cton rename to cranelift/filetests/domtree/loops2.cton index 452641be01..d2e3ba4f2c 100644 --- a/cranelift/src/tools/tests/dominator_tree_testdata/loops2.cton +++ b/cranelift/filetests/domtree/loops2.cton @@ -1,13 +1,15 @@ +test domtree + function test(i32) { - ebb0(v0: i32): ; dominates(0) - brz v0, ebb1 ; dominates(1,6) - brnz v0, ebb2 ; dominates(2,9) - jump ebb3 ; dominates(3) + ebb0(v0: i32): + brz v0, ebb1 ; dominates: ebb1 ebb6 + brnz v0, ebb2 ; dominates: ebb2 ebb9 + jump ebb3 ; dominates: ebb3 ebb1: jump ebb6 ebb2: - brz v0, ebb4 ; dominates(4,7,8) - jump ebb5 ; dominates(5) + brz v0, ebb4 ; dominates: ebb4 ebb7 ebb8 + jump ebb5 ; dominates: ebb5 ebb3: jump ebb9 ebb4: diff --git a/cranelift/filetests/domtree/tall-tree.cton b/cranelift/filetests/domtree/tall-tree.cton new file mode 100644 index 0000000000..3dd6f51da6 --- /dev/null +++ b/cranelift/filetests/domtree/tall-tree.cton @@ -0,0 +1,33 @@ +test domtree + +function test(i32) { + ebb0(v0: i32): + brz v0, ebb1 ; dominates: ebb1 + brnz v0, ebb2 ; dominates: ebb2 ebb5 + jump ebb3 ; dominates: ebb3 + ebb1: + jump ebb4 ; dominates: ebb4 + ebb2: + jump ebb5 + ebb3: + jump ebb5 + ebb4: + brz v0, ebb6 ; dominates: ebb6 ebb10 + jump ebb7 ; dominates: ebb7 + ebb5: + return + ebb6: + brz v0, ebb8 ; dominates: ebb11 ebb8 + brnz v0, ebb9 ; dominates: ebb9 + jump ebb10 + ebb7: + jump ebb10 + ebb8: + jump ebb11 + ebb9: + jump ebb11 + ebb10: + return + ebb11: + return +} diff --git a/cranelift/filetests/domtree/wide-tree.cton b/cranelift/filetests/domtree/wide-tree.cton new file mode 100644 index 0000000000..f4e822cd81 --- /dev/null +++ b/cranelift/filetests/domtree/wide-tree.cton @@ -0,0 +1,41 @@ +test domtree + +function test(i32) { + ebb0(v0: i32): + brz v0, ebb13 ; dominates: ebb13 + jump ebb1 ; dominates: ebb1 + ebb1: + brz v0, ebb2 ; dominates: ebb2 ebb7 + brnz v0, ebb3 ; dominates: ebb3 + brz v0, ebb4 ; dominates: ebb4 + brnz v0, ebb5 ; dominates: ebb5 + jump ebb6 ; dominates: ebb6 + ebb2: + jump ebb7 + ebb3: + jump ebb7 + ebb4: + jump ebb7 + ebb5: + jump ebb7 + ebb6: + jump ebb7 + ebb7: + brnz v0, ebb8 ; dominates: ebb8 ebb12 + brz v0, ebb9 ; dominates: ebb9 + brnz v0, ebb10 ; dominates: ebb10 + jump ebb11 ; dominates: ebb11 + ebb8: + jump ebb12 + ebb9: + jump ebb12 + ebb10: + brz v0, ebb13 + jump ebb12 + ebb11: + jump ebb13 + ebb12: + return + ebb13: + return +} diff --git a/cranelift/src/tools/filetest/domtree.rs b/cranelift/src/tools/filetest/domtree.rs new file mode 100644 index 0000000000..306fc433e9 --- /dev/null +++ b/cranelift/src/tools/filetest/domtree.rs @@ -0,0 +1,103 @@ + +//! Test command for verifying dominator trees. +//! +//! The `test domtree` test command looks for annotations on instructions like this: +//! +//! jump ebb3 ; dominates: ebb3 +//! +//! This annotation means that the jump instruction is expected to be the immediate dominator of +//! `ebb3`. +//! +//! We verify that the dominator tree annotations are complete and correct. +//! + +use std::collections::HashMap; +use std::borrow::{Borrow, Cow}; +use cretonne::ir::Function; +use cretonne::ir::entities::AnyEntity; +use cretonne::cfg::ControlFlowGraph; +use cretonne::dominator_tree::DominatorTree; +use cton_reader::TestCommand; +use filetest::subtest::{SubTest, Context, Result}; +use utils::match_directive; + +struct TestDomtree; + +pub fn subtest(parsed: &TestCommand) -> Result> { + assert_eq!(parsed.command, "domtree"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestDomtree)) + } +} + +impl SubTest for TestDomtree { + fn name(&self) -> Cow { + Cow::from("domtree") + } + + // Extract our own dominator tree from + fn run(&self, func: Cow, context: &Context) -> Result<()> { + let func = func.borrow(); + let cfg = ControlFlowGraph::new(func); + let domtree = DominatorTree::new(&cfg); + + // Build an expected domtree from the source annotations. + let mut expected = HashMap::new(); + for comment in &context.details.comments { + if let Some(tail) = match_directive(comment.text, "dominates:") { + let inst = match comment.entity { + AnyEntity::Inst(inst) => inst, + _ => { + return Err(format!("annotation on non-inst {}: {}", + comment.entity, + comment.text)) + } + }; + for src_ebb in tail.split_whitespace() { + let ebb = match context.details.map.lookup_str(src_ebb) { + Some(AnyEntity::Ebb(ebb)) => ebb, + _ => return Err(format!("expected EBB: {}", src_ebb)), + }; + + // Annotations say that `inst` is the idom of `ebb`. + if expected.insert(ebb, inst).is_some() { + return Err(format!("multiple dominators for {}", src_ebb)); + } + + // Compare to computed domtree. + match domtree.idom(ebb) { + Some((_, got_inst)) if got_inst != inst => { + return Err(format!("mismatching idoms for {}:\n\ + want: {}, got: {}", + src_ebb, + inst, + got_inst)); + } + None => { + return Err(format!("mismatching idoms for {}:\n\ + want: {}, got: unreachable", + src_ebb, + inst)); + } + _ => {} + } + } + } + } + + // Now we know that everything in `expected` is consistent with `domtree`. + // All other EBB's should be either unreachable or the entry block. + for ebb in func.layout.ebbs().skip(1).filter(|ebb| !expected.contains_key(&ebb)) { + if let Some((_, got_inst)) = domtree.idom(ebb) { + return Err(format!("mismatching idoms for renumbered {}:\n\ + want: unrechable, got: {}", + ebb, + got_inst)); + } + } + + Ok(()) + } +} diff --git a/cranelift/src/tools/filetest/mod.rs b/cranelift/src/tools/filetest/mod.rs index c1019446ca..0a7bdf0bab 100644 --- a/cranelift/src/tools/filetest/mod.rs +++ b/cranelift/src/tools/filetest/mod.rs @@ -9,6 +9,7 @@ use filetest::runner::TestRunner; pub mod subtest; mod runner; +mod domtree; /// Main entry point for `cton-util test`. /// diff --git a/cranelift/src/tools/filetest/subtest.rs b/cranelift/src/tools/filetest/subtest.rs index 89a8b830e1..0359265d3b 100644 --- a/cranelift/src/tools/filetest/subtest.rs +++ b/cranelift/src/tools/filetest/subtest.rs @@ -12,9 +12,11 @@ pub type Result = result::Result; pub fn new(parsed: &TestCommand) -> Result> { use cat; use print_cfg; + use filetest::domtree; match parsed.command { "cat" => cat::subtest(parsed), "print-cfg" => print_cfg::subtest(parsed), + "domtree" => domtree::subtest(parsed), _ => Err(format!("unknown test command '{}'", parsed.command)), } } diff --git a/cranelift/src/tools/tests/dominator_tree.rs b/cranelift/src/tools/tests/dominator_tree.rs deleted file mode 100644 index 71c02dc652..0000000000 --- a/cranelift/src/tools/tests/dominator_tree.rs +++ /dev/null @@ -1,96 +0,0 @@ -extern crate cretonne; -extern crate cton_reader; -extern crate glob; -extern crate regex; - -use std::env; -use glob::glob; -use regex::Regex; -use std::fs::File; -use std::io::Read; -use self::cretonne::ir::Ebb; -use self::cton_reader::parse_functions; -use self::cretonne::ir::function::Function; -use self::cretonne::entity_map::EntityMap; -use self::cretonne::ir::entities::NO_INST; -use self::cretonne::cfg::ControlFlowGraph; -use self::cretonne::dominator_tree::DominatorTree; - -/// Construct a dominator tree from specially formatted comments in -/// cton source. Each line with a jump/branch instruction should -/// have a comment of the format: `dominates(n, ..., N)`, where each `n` -/// is the Ebb number for which this instruction is the immediate dominator. -fn dominator_tree_from_source(func: &Function, function_source: &str) -> DominatorTree { - let ebb_re = Regex::new("^[ \t]*ebb[0-9]+.*:").unwrap(); - let dom_re = Regex::new("dominates\\(([0-9,]+)\\)").unwrap(); - let inst_re = Regex::new("^[ \t]*[a-zA-Z0-9]+[^{}]*").unwrap(); - let func_re = Regex::new("^[ \t]*function.*").unwrap(); - - let ebbs = func.layout.ebbs().collect::>(); - let mut data = EntityMap::with_capacity(ebbs.len()); - - if ebbs.len() < 1 { - return DominatorTree::from_data(data); - } - - let mut ebb_offset = 0; - let mut inst_offset = 0; - - let mut cur_ebb = ebbs[0]; - let mut insts = func.layout.ebb_insts(ebbs[ebb_offset]).collect::>(); - - for line in function_source.lines() { - if ebb_re.is_match(line) { - cur_ebb = ebbs[ebb_offset]; - insts = func.layout.ebb_insts(cur_ebb).collect::>(); - ebb_offset += 1; - inst_offset = 0; - } else if inst_re.is_match(line) && !func_re.is_match(line) { - inst_offset += 1; - } - match dom_re.captures(line) { - Some(caps) => { - for s in caps.at(1).unwrap().split(",") { - let this_ebb = Ebb::with_number(s.parse::().unwrap()).unwrap(); - let inst = if inst_offset == 0 { - NO_INST - } else { - insts[inst_offset - 1].clone() - }; - data[this_ebb] = Some((cur_ebb.clone(), inst)); - } - }, - None => continue, - }; - - } - DominatorTree::from_data(data) -} - -fn test_dominator_tree(function_source: &str) { - - let func = &parse_functions(function_source).unwrap()[0]; - let src_dtree = dominator_tree_from_source(&func, function_source); - - let cfg = ControlFlowGraph::new(&func); - let dtree = DominatorTree::new(&cfg); - - for ebb in func.layout.ebbs() { - assert_eq!(dtree.idom(ebb), src_dtree.idom(ebb)); - } -} - -#[test] -fn test_all() { - let testdir = format!("{}/tests/dominator_tree_testdata/*.cton", - env::current_dir().unwrap().display()); - - for entry in glob(&testdir).unwrap() { - let path = entry.unwrap(); - println!("Testing {:?}", path); - let mut file = File::open(&path).unwrap(); - let mut buffer = String::new(); - file.read_to_string(&mut buffer).unwrap(); - test_dominator_tree(&buffer); - } -} diff --git a/cranelift/src/tools/tests/dominator_tree_testdata/basic.cton b/cranelift/src/tools/tests/dominator_tree_testdata/basic.cton deleted file mode 100644 index c274336b79..0000000000 --- a/cranelift/src/tools/tests/dominator_tree_testdata/basic.cton +++ /dev/null @@ -1,11 +0,0 @@ -function test(i32) { - ebb0(v0: i32): ; dominates(0) - jump ebb1 ; dominates(1) - ebb1: - brz v0, ebb3 ; dominates(3) - jump ebb2 ; dominates(2) - ebb2: - jump ebb3 - ebb3: - return -} diff --git a/cranelift/src/tools/tests/dominator_tree_testdata/tall-tree.cton b/cranelift/src/tools/tests/dominator_tree_testdata/tall-tree.cton deleted file mode 100644 index d2ea0f8a9e..0000000000 --- a/cranelift/src/tools/tests/dominator_tree_testdata/tall-tree.cton +++ /dev/null @@ -1,31 +0,0 @@ -function test(i32) { - ebb0(v0: i32): ; dominates(0) - brz v0, ebb1 ; dominates(1) - brnz v0, ebb2 ; dominates(2,5) - jump ebb3 ; dominates(3) - ebb1: - jump ebb4 ; dominates(4) - ebb2: - jump ebb5 - ebb3: - jump ebb5 - ebb4: - brz v0, ebb6 ; dominates(6,10) - jump ebb7 ; dominates(7) - ebb5: - return - ebb6: - brz v0, ebb8 ; dominates(11,8) - brnz v0, ebb9 ; dominates(9) - jump ebb10 - ebb7: - jump ebb10 - ebb8: - jump ebb11 - ebb9: - jump ebb11 - ebb10: - return - ebb11: - return -} diff --git a/cranelift/src/tools/tests/dominator_tree_testdata/wide-tree.cton b/cranelift/src/tools/tests/dominator_tree_testdata/wide-tree.cton deleted file mode 100644 index 0883ef6514..0000000000 --- a/cranelift/src/tools/tests/dominator_tree_testdata/wide-tree.cton +++ /dev/null @@ -1,39 +0,0 @@ -function test(i32) { - ebb0(v0: i32): ; dominates(0) - brz v0, ebb13 ; dominates(13) - jump ebb1 ; dominates(1) - ebb1: - brz v0, ebb2 ; dominates(2,7) - brnz v0, ebb3 ; dominates(3) - brz v0, ebb4 ; dominates(4) - brnz v0, ebb5 ; dominates(5) - jump ebb6 ; dominates(6) - ebb2: - jump ebb7 - ebb3: - jump ebb7 - ebb4: - jump ebb7 - ebb5: - jump ebb7 - ebb6: - jump ebb7 - ebb7: - brnz v0, ebb8 ; dominates(8,12) - brz v0, ebb9 ; dominates(9) - brnz v0, ebb10 ; dominates(10) - jump ebb11 ; dominates(11) - ebb8: - jump ebb12 - ebb9: - jump ebb12 - ebb10: - brz v0, ebb13 - jump ebb12 - ebb11: - jump ebb13 - ebb12: - return - ebb13: - return -} diff --git a/cranelift/src/tools/utils.rs b/cranelift/src/tools/utils.rs index 5973f25814..f89bc46b43 100644 --- a/cranelift/src/tools/utils.rs +++ b/cranelift/src/tools/utils.rs @@ -11,3 +11,29 @@ pub fn read_to_string>(path: P) -> Result { try!(file.read_to_string(&mut buffer)); Ok(buffer) } + +/// Look for a directive in a comment string. +/// The directive is of the form "foo:" and should follow the leading `;` in the comment: +/// +/// ; dominates: ebb3 ebb4 +/// +/// Return the comment text following the directive. +pub fn match_directive<'a>(comment: &'a str, directive: &str) -> Option<&'a str> { + assert!(directive.ends_with(':'), + "Directive must include trailing colon"); + let text = comment.trim_left_matches(';').trim_left(); + if text.starts_with(directive) { + Some(text[directive.len()..].trim()) + } else { + None + } +} + +#[test] +fn test_match_directive() { + assert_eq!(match_directive("; foo: bar ", "foo:"), Some("bar")); + assert_eq!(match_directive(" foo:bar", "foo:"), Some("bar")); + assert_eq!(match_directive("foo:bar", "foo:"), Some("bar")); + assert_eq!(match_directive(";x foo: bar", "foo:"), None); + assert_eq!(match_directive(";;; foo: bar", "foo:"), Some("bar")); +} From 45559a21c1f50923e3a96efb64f00f4489684bfc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Sep 2016 15:17:19 -0700 Subject: [PATCH 0291/3084] Remove dead public functions from DominatorTree. --- cranelift/src/libcretonne/dominator_tree.rs | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/cranelift/src/libcretonne/dominator_tree.rs b/cranelift/src/libcretonne/dominator_tree.rs index a3a00b2cc4..c741b5992d 100644 --- a/cranelift/src/libcretonne/dominator_tree.rs +++ b/cranelift/src/libcretonne/dominator_tree.rs @@ -3,18 +3,13 @@ use cfg::*; use ir::Ebb; use ir::entities::NO_INST; -use entity_map::{EntityMap, Keys}; +use entity_map::EntityMap; pub struct DominatorTree { data: EntityMap>, } impl DominatorTree { - /// Insert data directly into a dominator tree. - pub fn from_data(data: EntityMap>) -> DominatorTree { - DominatorTree { data: data } - } - /// Build a dominator tree from a control flow graph using Keith D. Cooper's /// "Simple, Fast Dominator Algorithm." pub fn new(cfg: &ControlFlowGraph) -> DominatorTree { @@ -116,11 +111,6 @@ impl DominatorTree { pub fn idom(&self, ebb: Ebb) -> Option { self.data[ebb].clone() } - - /// An iterator across all of the ebbs stored in the tree. - pub fn ebbs(&self) -> Keys { - self.data.keys() - } } #[cfg(test)] @@ -136,7 +126,7 @@ mod test { let func = Function::new(); let cfg = ControlFlowGraph::new(&func); let dtree = DominatorTree::new(&cfg); - assert_eq!(None, dtree.ebbs().next()); + assert_eq!(0, dtree.data.keys().count()); } #[test] From 49202bf21b780aefcca6ad1e1dcdad30554162a6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 12 Sep 2016 14:17:54 -0700 Subject: [PATCH 0292/3084] Add a location to verifier error messages. The verifier reports the 'location' of an error message as a reference to the entity that has a problem. This uses the 'AnyEntity' type to refer to instructions/values/ebbs etc. Also add an err! macro similar to the one used by the parser. --- cranelift/src/libcretonne/verifier.rs | 65 ++++++++++++++----- cranelift/src/tools/filetest/runner.rs | 2 +- cranelift/src/tools/tests/verifier.rs | 2 +- .../tests/verifier_testdata/bad_layout.cton | 2 +- 4 files changed, 53 insertions(+), 18 deletions(-) diff --git a/cranelift/src/libcretonne/verifier.rs b/cranelift/src/libcretonne/verifier.rs index 91131ee6ef..d58283e3ae 100644 --- a/cranelift/src/libcretonne/verifier.rs +++ b/cranelift/src/libcretonne/verifier.rs @@ -54,8 +54,43 @@ use ir::{Function, ValueDef, Ebb, Inst}; use ir::instructions::InstructionFormat; +use ir::entities::AnyEntity; +use std::fmt::{self, Display, Formatter}; +use std::result; -pub fn verify_function(func: &Function) -> Result<(), String> { +/// A verifier error. +#[derive(Debug, PartialEq, Eq)] +pub struct Error { + pub location: AnyEntity, + pub message: String, +} + +impl Display for Error { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + write!(f, "{}: {}", self.location, self.message) + } +} + +pub type Result = result::Result; + +// Create an `Err` variant of `Result` from a location and `format!` args. +macro_rules! err { + ( $loc:expr, $msg:expr ) => { + Err(Error { + location: $loc.into(), + message: String::from($msg), + }) + }; + + ( $loc:expr, $fmt:expr, $( $arg:expr ),+ ) => { + Err(Error { + location: $loc.into(), + message: format!( $fmt, $( $arg ),+ ), + }) + }; +} + +pub fn verify_function(func: &Function) -> Result<()> { Verifier::new(func).run() } @@ -68,25 +103,25 @@ impl<'a> Verifier<'a> { Verifier { func: func } } - fn ebb_integrity(&self, ebb: Ebb, inst: Inst) -> Result<(), String> { + fn ebb_integrity(&self, ebb: Ebb, inst: Inst) -> Result<()> { let is_terminator = self.func.dfg[inst].is_terminating(); let is_last_inst = self.func.layout.last_inst(ebb) == inst; if is_terminator && !is_last_inst { // Terminating instructions only occur at the end of blocks. - return Err(format!("A terminating instruction was encountered before the \ - end of ebb {:?}!", - ebb)); + return err!(inst, + "a terminator instruction was encountered before the end of {}", + ebb); } if is_last_inst && !is_terminator { - return Err(format!("Block {:?} does not end in a terminating instruction!", ebb)); + return err!(ebb, "block does not end in a terminator instruction!"); } // Instructions belong to the correct ebb. let inst_ebb = self.func.layout.inst_ebb(inst); if inst_ebb != Some(ebb) { - return Err(format!("{:?} should belong to {:?} not {:?}", inst, ebb, inst_ebb)); + return err!(inst, "should belong to {} not {:?}", ebb, inst_ebb); } // Arguments belong to the correct ebb. @@ -94,11 +129,11 @@ impl<'a> Verifier<'a> { match self.func.dfg.value_def(arg) { ValueDef::Arg(arg_ebb, _) => { if ebb != arg_ebb { - return Err(format!("{:?} does not belong to {:?}", arg, ebb)); + return err!(arg, "does not belong to {}", ebb); } } _ => { - return Err("Expected an argument, found a result!".to_string()); + return err!(arg, "expected an argument, found a result"); } } } @@ -106,18 +141,18 @@ impl<'a> Verifier<'a> { Ok(()) } - fn instruction_integrity(&self, inst: Inst) -> Result<(), String> { + fn instruction_integrity(&self, inst: Inst) -> Result<()> { let inst_data = &self.func.dfg[inst]; // The instruction format matches the opcode if inst_data.opcode().format() != Some(InstructionFormat::from(inst_data)) { - return Err("Instruction opcode doesn't match instruction format!".to_string()); + return err!(inst, "instruction opcode doesn't match instruction format"); } Ok(()) } - pub fn run(&self) -> Result<(), String> { + pub fn run(&self) -> Result<()> { for ebb in self.func.layout.ebbs() { for inst in self.func.layout.ebb_insts(ebb) { try!(self.ebb_integrity(ebb, inst)); @@ -143,9 +178,9 @@ mod tests { let err_re = Regex::new($msg).unwrap(); match $e { Ok(_) => { panic!("Expected an error!") }, - Err(err_msg) => { - if !err_re.is_match(&err_msg) { - panic!(format!("'{}' did not contain the pattern '{}'", err_msg, $msg)); + Err(Error { location, message } ) => { + if !err_re.is_match(&message) { + panic!(format!("'{}' did not contain the pattern '{}'", message, $msg)); } } } diff --git a/cranelift/src/tools/filetest/runner.rs b/cranelift/src/tools/filetest/runner.rs index ceb1e34c62..5f9fac9713 100644 --- a/cranelift/src/tools/filetest/runner.rs +++ b/cranelift/src/tools/filetest/runner.rs @@ -259,7 +259,7 @@ impl Job { // Should we run the verifier before this test? if !context.verified && test.needs_verifier() { - try!(verify_function(&func)); + try!(verify_function(&func).map_err(|e| e.to_string())); context.verified = true; } diff --git a/cranelift/src/tools/tests/verifier.rs b/cranelift/src/tools/tests/verifier.rs index c26056b8ba..15fcf05ec8 100644 --- a/cranelift/src/tools/tests/verifier.rs +++ b/cranelift/src/tools/tests/verifier.rs @@ -41,7 +41,7 @@ fn verifier_tests_from_source(function_source: &str) { let result = Verifier::new(&func).run(); match verifier_results[i] { Some(ref re) => { - assert_eq!(re.is_match(&result.err().unwrap()), true); + assert_eq!(re.is_match(&result.err().unwrap().message), true); }, None => { assert_eq!(result, Ok(())); diff --git a/cranelift/src/tools/tests/verifier_testdata/bad_layout.cton b/cranelift/src/tools/tests/verifier_testdata/bad_layout.cton index 680299def7..a99b9b5fe4 100644 --- a/cranelift/src/tools/tests/verifier_testdata/bad_layout.cton +++ b/cranelift/src/tools/tests/verifier_testdata/bad_layout.cton @@ -1,4 +1,4 @@ -function test(i32) { ; Err(terminating) +function test(i32) { ; Err(terminator) ebb0(v0: i32): jump ebb1 return From a98d6e5256ed231acfffa7a32bd913aa1694c78c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Sep 2016 16:19:47 -0700 Subject: [PATCH 0293/3084] Add a 'test verifier' sub-test. This test runs the verifier on each function and matches the resulting verifier error against the "error:" annotation. Move the existing verifier test into filetests/verifier/ and use the new syntex. --- cranelift/filetests/cfg/loop.cton | 1 + cranelift/filetests/cfg/traps_early.cton | 3 +- .../verifier}/bad_layout.cton | 6 +- cranelift/src/tools/filetest/mod.rs | 1 + cranelift/src/tools/filetest/subtest.rs | 2 + cranelift/src/tools/filetest/verifier.rs | 78 +++++++++++++++++++ cranelift/src/tools/tests/verifier.rs | 66 ---------------- 7 files changed, 88 insertions(+), 69 deletions(-) rename cranelift/{src/tools/tests/verifier_testdata => filetests/verifier}/bad_layout.cton (71%) create mode 100644 cranelift/src/tools/filetest/verifier.rs delete mode 100644 cranelift/src/tools/tests/verifier.rs diff --git a/cranelift/filetests/cfg/loop.cton b/cranelift/filetests/cfg/loop.cton index 5732ac593a..9a1ca6d7e3 100644 --- a/cranelift/filetests/cfg/loop.cton +++ b/cranelift/filetests/cfg/loop.cton @@ -1,5 +1,6 @@ ; For testing cfg generation. This code is nonsense. test print-cfg +test verifier function nonsense(i32, i32) -> f32 { ; check: digraph nonsense { diff --git a/cranelift/filetests/cfg/traps_early.cton b/cranelift/filetests/cfg/traps_early.cton index a52cac1ba4..9648190bc5 100644 --- a/cranelift/filetests/cfg/traps_early.cton +++ b/cranelift/filetests/cfg/traps_early.cton @@ -1,12 +1,13 @@ ; For testing cfg generation. This code explores the implications of encountering ; a terminating instruction before any connections have been made. test print-cfg +test verifier function nonsense(i32) { ; check: digraph nonsense { ebb0(v1: i32): - trap + trap ; error: terminator instruction was encountered before the end brnz v1, ebb2 ; unordered: ebb0:inst1 -> ebb2 jump ebb1 ; unordered: ebb0:inst2 -> ebb1 diff --git a/cranelift/src/tools/tests/verifier_testdata/bad_layout.cton b/cranelift/filetests/verifier/bad_layout.cton similarity index 71% rename from cranelift/src/tools/tests/verifier_testdata/bad_layout.cton rename to cranelift/filetests/verifier/bad_layout.cton index a99b9b5fe4..ac29452958 100644 --- a/cranelift/src/tools/tests/verifier_testdata/bad_layout.cton +++ b/cranelift/filetests/verifier/bad_layout.cton @@ -1,6 +1,8 @@ -function test(i32) { ; Err(terminator) +test verifier + +function test(i32) { ebb0(v0: i32): - jump ebb1 + jump ebb1 ; error: terminator return ebb1: jump ebb2 diff --git a/cranelift/src/tools/filetest/mod.rs b/cranelift/src/tools/filetest/mod.rs index 0a7bdf0bab..b368959125 100644 --- a/cranelift/src/tools/filetest/mod.rs +++ b/cranelift/src/tools/filetest/mod.rs @@ -10,6 +10,7 @@ use filetest::runner::TestRunner; pub mod subtest; mod runner; mod domtree; +mod verifier; /// Main entry point for `cton-util test`. /// diff --git a/cranelift/src/tools/filetest/subtest.rs b/cranelift/src/tools/filetest/subtest.rs index 0359265d3b..cfc5cec5bf 100644 --- a/cranelift/src/tools/filetest/subtest.rs +++ b/cranelift/src/tools/filetest/subtest.rs @@ -13,10 +13,12 @@ pub fn new(parsed: &TestCommand) -> Result> { use cat; use print_cfg; use filetest::domtree; + use filetest::verifier; match parsed.command { "cat" => cat::subtest(parsed), "print-cfg" => print_cfg::subtest(parsed), "domtree" => domtree::subtest(parsed), + "verifier" => verifier::subtest(parsed), _ => Err(format!("unknown test command '{}'", parsed.command)), } } diff --git a/cranelift/src/tools/filetest/verifier.rs b/cranelift/src/tools/filetest/verifier.rs new file mode 100644 index 0000000000..5812dc9db0 --- /dev/null +++ b/cranelift/src/tools/filetest/verifier.rs @@ -0,0 +1,78 @@ +//! Test command for checking the IL verifier. +//! +//! The `test verifier` test command looks for annotations on instructions like this: +//! +//! jump ebb3 ; error: jump to non-existent EBB +//! +//! This annotation means that the verifier is expected to given an error for the jump instruction +//! containing the substring "jump to non-existent EBB". + +use std::borrow::{Borrow, Cow}; +use cretonne::verify_function; +use cretonne::ir::Function; +use cton_reader::TestCommand; +use filetest::subtest::{SubTest, Context, Result}; +use utils::match_directive; + +struct TestVerifier; + +pub fn subtest(parsed: &TestCommand) -> Result> { + assert_eq!(parsed.command, "verifier"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestVerifier)) + } +} + +impl SubTest for TestVerifier { + fn name(&self) -> Cow { + Cow::from("verifier") + } + + fn needs_verifier(&self) -> bool { + // Running the verifier before this test would defeat its purpose. + false + } + + fn run(&self, func: Cow, context: &Context) -> Result<()> { + let func = func.borrow(); + + // Scan source annotations for "error:" directives. + let mut expected = None; + for comment in &context.details.comments { + if let Some(tail) = match_directive(comment.text, "error:") { + // Currently, the verifier can only report one problem at a time. + // Reject more than one `error:` directives. + if expected.is_some() { + return Err("cannot handle multiple error: directives".to_string()); + } + expected = Some((comment.entity, tail)); + } + } + + match verify_function(func) { + Ok(_) => { + match expected { + None => Ok(()), + Some((_, msg)) => Err(format!("passed, expected error: {}", msg)), + } + } + Err(got) => { + match expected { + None => Err(format!("verifier pass, got {}", got)), + Some((want_loc, want_msg)) if got.message.contains(want_msg) => { + if want_loc == got.location { + Ok(()) + } else { + Err(format!("correct error reported on {}, but wanted {}", + got.location, + want_loc)) + } + } + Some(_) => Err(format!("mismatching error: {}", got)), + } + } + } + } +} diff --git a/cranelift/src/tools/tests/verifier.rs b/cranelift/src/tools/tests/verifier.rs deleted file mode 100644 index 15fcf05ec8..0000000000 --- a/cranelift/src/tools/tests/verifier.rs +++ /dev/null @@ -1,66 +0,0 @@ -extern crate cretonne; -extern crate cton_reader; -extern crate glob; -extern crate regex; - -use std::env; -use glob::glob; -use regex::Regex; -use std::fs::File; -use std::io::Read; -use self::cton_reader::parse_functions; -use self::cretonne::verifier::Verifier; - -/// Compile a function and run verifier tests based on specially formatted -/// comments in the [function's] source. -fn verifier_tests_from_source(function_source: &str) { - let func_re = Regex::new("^[ \t]*function.*").unwrap(); - let err_re = Regex::new(";[ \t]*Err\\((.*)+\\)").unwrap(); - - // Each entry corresponds to an optional regular expression, where - // the index corresponds to the function offset in our source code. - // If no error is expected for a given function its entry will be - // set to None. - let mut verifier_results = Vec::new(); - for line in function_source.lines() { - if func_re.is_match(line) { - match err_re.captures(line) { - Some(caps) => { - verifier_results.push(Some(Regex::new(caps.at(1).unwrap()).unwrap())); - }, - None => { - verifier_results.push(None); - }, - }; - } - } - - // Run the verifier against each function and compare the output - // with the expected result (as determined above). - for (i, func) in parse_functions(function_source).unwrap().into_iter().enumerate() { - let result = Verifier::new(&func).run(); - match verifier_results[i] { - Some(ref re) => { - assert_eq!(re.is_match(&result.err().unwrap().message), true); - }, - None => { - assert_eq!(result, Ok(())); - } - } - } -} - -#[test] -fn test_all() { - let testdir = format!("{}/tests/verifier_testdata/*.cton", - env::current_dir().unwrap().display()); - - for entry in glob(&testdir).unwrap() { - let path = entry.unwrap(); - println!("Testing {:?}", path); - let mut file = File::open(&path).unwrap(); - let mut buffer = String::new(); - file.read_to_string(&mut buffer).unwrap(); - verifier_tests_from_source(&buffer); - } -} From dd9696e2941d5f111bed560283fd18d1746527a1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Sep 2016 16:34:50 -0700 Subject: [PATCH 0294/3084] Remove unnecessary external dependencies. The main libcretonne crate should not have any external dependencies if at all possible. Use simple substring matching instead of regular expressions in the verifier tests to achieve this. The tools crate no longer depends directly on glob and regex. It still has an indirect dependency on regex through libfilecheck. --- cranelift/src/libcretonne/Cargo.toml | 5 ++++- cranelift/src/libcretonne/verifier.rs | 10 +++------- cranelift/src/tools/Cargo.lock | 10 ---------- cranelift/src/tools/Cargo.toml | 2 -- 4 files changed, 7 insertions(+), 20 deletions(-) diff --git a/cranelift/src/libcretonne/Cargo.toml b/cranelift/src/libcretonne/Cargo.toml index 5d3a9511e6..174fb2e0bf 100644 --- a/cranelift/src/libcretonne/Cargo.toml +++ b/cranelift/src/libcretonne/Cargo.toml @@ -13,4 +13,7 @@ name = "cretonne" path = "lib.rs" [dependencies] -regex = "0.1.71" +# It is a goal of the cretonne crate to have minimal external dependencies. +# Please don't add any unless they are essential to the task of creating binary +# machine code. Integration tests that need external dependencies can be +# accomodated in src/tools/tests. diff --git a/cranelift/src/libcretonne/verifier.rs b/cranelift/src/libcretonne/verifier.rs index d58283e3ae..2b544967e2 100644 --- a/cranelift/src/libcretonne/verifier.rs +++ b/cranelift/src/libcretonne/verifier.rs @@ -165,22 +165,18 @@ impl<'a> Verifier<'a> { #[cfg(test)] mod tests { - extern crate regex; - use super::*; use ir::Function; use ir::instructions::{InstructionData, Opcode}; use ir::types; - use self::regex::Regex; macro_rules! assert_err_with_msg { ($e:expr, $msg:expr) => ( - let err_re = Regex::new($msg).unwrap(); match $e { Ok(_) => { panic!("Expected an error!") }, - Err(Error { location, message } ) => { - if !err_re.is_match(&message) { - panic!(format!("'{}' did not contain the pattern '{}'", message, $msg)); + Err(Error { message, .. } ) => { + if !message.contains($msg) { + panic!(format!("'{}' did not contain the substring '{}'", message, $msg)); } } } diff --git a/cranelift/src/tools/Cargo.lock b/cranelift/src/tools/Cargo.lock index 2aca3c591b..50576fd02b 100644 --- a/cranelift/src/tools/Cargo.lock +++ b/cranelift/src/tools/Cargo.lock @@ -6,8 +6,6 @@ dependencies = [ "cretonne-reader 0.0.0", "docopt 0.6.80 (registry+https://github.com/rust-lang/crates.io-index)", "filecheck 0.0.0", - "glob 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)", - "regex 0.1.71 (registry+https://github.com/rust-lang/crates.io-index)", "rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)", ] @@ -22,9 +20,6 @@ dependencies = [ [[package]] name = "cretonne" version = "0.0.0" -dependencies = [ - "regex 0.1.71 (registry+https://github.com/rust-lang/crates.io-index)", -] [[package]] name = "cretonne-reader" @@ -50,11 +45,6 @@ dependencies = [ "regex 0.1.71 (registry+https://github.com/rust-lang/crates.io-index)", ] -[[package]] -name = "glob" -version = "0.2.11" -source = "registry+https://github.com/rust-lang/crates.io-index" - [[package]] name = "kernel32-sys" version = "0.2.2" diff --git a/cranelift/src/tools/Cargo.toml b/cranelift/src/tools/Cargo.toml index ba465f7d0e..b4572f9f49 100644 --- a/cranelift/src/tools/Cargo.toml +++ b/cranelift/src/tools/Cargo.toml @@ -15,5 +15,3 @@ cretonne-reader = { path = "../libreader" } filecheck = { path = "../libfilecheck" } docopt = "0.6.80" rustc-serialize = "0.3.19" -regex = "0.1.71" -glob = "0.2.11" From b26f7be6c99977b07f18293599c4dbd2996aa21f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Sep 2016 16:41:04 -0700 Subject: [PATCH 0295/3084] cargo update. --- cranelift/src/tools/Cargo.lock | 38 ++++++++++++++++++++-------------- 1 file changed, 22 insertions(+), 16 deletions(-) diff --git a/cranelift/src/tools/Cargo.lock b/cranelift/src/tools/Cargo.lock index 50576fd02b..ca97206cc4 100644 --- a/cranelift/src/tools/Cargo.lock +++ b/cranelift/src/tools/Cargo.lock @@ -4,14 +4,14 @@ version = "0.0.0" dependencies = [ "cretonne 0.0.0", "cretonne-reader 0.0.0", - "docopt 0.6.80 (registry+https://github.com/rust-lang/crates.io-index)", + "docopt 0.6.83 (registry+https://github.com/rust-lang/crates.io-index)", "filecheck 0.0.0", "rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)", ] [[package]] name = "aho-corasick" -version = "0.5.2" +version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" dependencies = [ "memchr 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)", @@ -30,19 +30,20 @@ dependencies = [ [[package]] name = "docopt" -version = "0.6.80" +version = "0.6.83" source = "registry+https://github.com/rust-lang/crates.io-index" dependencies = [ - "regex 0.1.71 (registry+https://github.com/rust-lang/crates.io-index)", + "lazy_static 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", + "regex 0.1.77 (registry+https://github.com/rust-lang/crates.io-index)", "rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)", - "strsim 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)", + "strsim 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)", ] [[package]] name = "filecheck" version = "0.0.0" dependencies = [ - "regex 0.1.71 (registry+https://github.com/rust-lang/crates.io-index)", + "regex 0.1.77 (registry+https://github.com/rust-lang/crates.io-index)", ] [[package]] @@ -50,13 +51,18 @@ name = "kernel32-sys" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" dependencies = [ - "winapi 0.2.7 (registry+https://github.com/rust-lang/crates.io-index)", + "winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)", "winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", ] +[[package]] +name = "lazy_static" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" + [[package]] name = "libc" -version = "0.2.11" +version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" [[package]] @@ -64,24 +70,24 @@ name = "memchr" version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" dependencies = [ - "libc 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)", + "libc 0.2.16 (registry+https://github.com/rust-lang/crates.io-index)", ] [[package]] name = "regex" -version = "0.1.71" +version = "0.1.77" source = "registry+https://github.com/rust-lang/crates.io-index" dependencies = [ - "aho-corasick 0.5.2 (registry+https://github.com/rust-lang/crates.io-index)", + "aho-corasick 0.5.3 (registry+https://github.com/rust-lang/crates.io-index)", "memchr 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)", - "regex-syntax 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)", + "regex-syntax 0.3.5 (registry+https://github.com/rust-lang/crates.io-index)", "thread_local 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)", "utf8-ranges 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)", ] [[package]] name = "regex-syntax" -version = "0.3.3" +version = "0.3.5" source = "registry+https://github.com/rust-lang/crates.io-index" [[package]] @@ -91,7 +97,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" [[package]] name = "strsim" -version = "0.3.0" +version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" [[package]] @@ -100,7 +106,7 @@ version = "2.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" dependencies = [ "kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)", + "libc 0.2.16 (registry+https://github.com/rust-lang/crates.io-index)", ] [[package]] @@ -118,7 +124,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" [[package]] name = "winapi" -version = "0.2.7" +version = "0.2.8" source = "registry+https://github.com/rust-lang/crates.io-index" [[package]] From 4bc41bbcffd780c6778b053e7f33d2b058f9bf56 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Sep 2016 17:00:55 -0700 Subject: [PATCH 0296/3084] Move subtest::new() into the parent module. This saves on importing of all the sub-modules that implement new test commands, and it provides a nice overview of what 'cton-util test' can do. --- cranelift/src/tools/filetest/mod.rs | 17 +++++++++++++++++ cranelift/src/tools/filetest/runner.rs | 3 ++- cranelift/src/tools/filetest/subtest.rs | 17 +---------------- 3 files changed, 20 insertions(+), 17 deletions(-) diff --git a/cranelift/src/tools/filetest/mod.rs b/cranelift/src/tools/filetest/mod.rs index b368959125..9126efbfc0 100644 --- a/cranelift/src/tools/filetest/mod.rs +++ b/cranelift/src/tools/filetest/mod.rs @@ -4,7 +4,10 @@ //! available test commands. use std::path::Path; +use cton_reader::TestCommand; use CommandResult; +use cat; +use print_cfg; use filetest::runner::TestRunner; pub mod subtest; @@ -34,3 +37,17 @@ pub fn run(files: Vec) -> CommandResult { runner.run() } + +/// Create a new subcommand trait object to match `parsed.command`. +/// +/// This function knows how to create all of the possible `test ` commands that can appear in +/// a .cton test file. +fn new_subtest(parsed: &TestCommand) -> subtest::Result> { + match parsed.command { + "cat" => cat::subtest(parsed), + "print-cfg" => print_cfg::subtest(parsed), + "domtree" => domtree::subtest(parsed), + "verifier" => verifier::subtest(parsed), + _ => Err(format!("unknown test command '{}'", parsed.command)), + } +} diff --git a/cranelift/src/tools/filetest/runner.rs b/cranelift/src/tools/filetest/runner.rs index 5f9fac9713..8c9ea87810 100644 --- a/cranelift/src/tools/filetest/runner.rs +++ b/cranelift/src/tools/filetest/runner.rs @@ -15,6 +15,7 @@ use utils::read_to_string; use cton_reader::parse_test; use cretonne::ir::Function; use cretonne::verify_function; +use filetest::new_subtest; use filetest::subtest::{self, SubTest, Context}; type TestResult = Result; @@ -218,7 +219,7 @@ impl Job { } // Parse the test commands. let mut tests = - try!(testfile.commands.iter().map(subtest::new).collect::>>()); + try!(testfile.commands.iter().map(new_subtest).collect::>>()); // Sort the tests so the mutators are at the end, and those that // don't need the verifier are at the front diff --git a/cranelift/src/tools/filetest/subtest.rs b/cranelift/src/tools/filetest/subtest.rs index cfc5cec5bf..b958193cd8 100644 --- a/cranelift/src/tools/filetest/subtest.rs +++ b/cranelift/src/tools/filetest/subtest.rs @@ -3,26 +3,11 @@ use std::result; use std::borrow::Cow; use cretonne::ir::Function; -use cton_reader::{TestCommand, Details}; +use cton_reader::Details; use filecheck::{self, CheckerBuilder, Checker, Value as FCValue}; pub type Result = result::Result; -/// Create a new subcommand trait object to match `parsed.command`. -pub fn new(parsed: &TestCommand) -> Result> { - use cat; - use print_cfg; - use filetest::domtree; - use filetest::verifier; - match parsed.command { - "cat" => cat::subtest(parsed), - "print-cfg" => print_cfg::subtest(parsed), - "domtree" => domtree::subtest(parsed), - "verifier" => verifier::subtest(parsed), - _ => Err(format!("unknown test command '{}'", parsed.command)), - } -} - /// Context for running a a test on a single function. pub struct Context<'a> { /// Additional details about the function from the parser. From 91167f0153597404477836cb4e9acc93e4017da0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Sep 2016 21:06:50 -0700 Subject: [PATCH 0297/3084] Move the code to run test into its own module. Loading a file and running the test in it can be separated from the mechanics of running multiple tests and scanning directories for test files. --- cranelift/src/tools/filetest/mod.rs | 5 ++ cranelift/src/tools/filetest/runner.rs | 74 ++------------------------ cranelift/src/tools/filetest/runone.rs | 66 +++++++++++++++++++++++ 3 files changed, 75 insertions(+), 70 deletions(-) create mode 100644 cranelift/src/tools/filetest/runone.rs diff --git a/cranelift/src/tools/filetest/mod.rs b/cranelift/src/tools/filetest/mod.rs index 9126efbfc0..dec7ff6016 100644 --- a/cranelift/src/tools/filetest/mod.rs +++ b/cranelift/src/tools/filetest/mod.rs @@ -4,6 +4,7 @@ //! available test commands. use std::path::Path; +use std::time; use cton_reader::TestCommand; use CommandResult; use cat; @@ -12,9 +13,13 @@ use filetest::runner::TestRunner; pub mod subtest; mod runner; +mod runone; mod domtree; mod verifier; +/// The result of running the test in a file. +pub type TestResult = Result; + /// Main entry point for `cton-util test`. /// /// Take a list of filenames which can be either `.cton` files or directories. diff --git a/cranelift/src/tools/filetest/runner.rs b/cranelift/src/tools/filetest/runner.rs index 8c9ea87810..7de2829296 100644 --- a/cranelift/src/tools/filetest/runner.rs +++ b/cranelift/src/tools/filetest/runner.rs @@ -3,22 +3,13 @@ //! This module implements the `TestRunner` struct which manages executing tests as well as //! scanning directories for tests. -use std::ffi::OsStr; -use std::path::{Path, PathBuf}; use std::error::Error; +use std::ffi::OsStr; use std::mem; -use std::borrow::{Borrow, Cow}; use std::panic::catch_unwind; -use std::time; +use std::path::{Path, PathBuf}; +use filetest::{TestResult, runone}; use CommandResult; -use utils::read_to_string; -use cton_reader::parse_test; -use cretonne::ir::Function; -use cretonne::verify_function; -use filetest::new_subtest; -use filetest::subtest::{self, SubTest, Context}; - -type TestResult = Result; #[derive(PartialEq, Eq, Debug)] enum QueueEntry { @@ -204,66 +195,9 @@ impl Job { } pub fn run(&self) -> TestResult { - match catch_unwind(|| self.run_or_panic()) { + match catch_unwind(|| runone::run(self.path.as_path())) { Err(msg) => Err(format!("panic: {:?}", msg)), Ok(result) => result, } } - - fn run_or_panic(&self) -> TestResult { - let started = time::Instant::now(); - let buffer = try!(read_to_string(&self.path).map_err(|e| e.to_string())); - let testfile = try!(parse_test(&buffer).map_err(|e| e.to_string())); - if testfile.functions.is_empty() { - return Err("no functions found".to_string()); - } - // Parse the test commands. - let mut tests = - try!(testfile.commands.iter().map(new_subtest).collect::>>()); - - // Sort the tests so the mutators are at the end, and those that - // don't need the verifier are at the front - tests.sort_by_key(|st| (st.is_mutating(), st.needs_verifier())); - - // Isolate the last test in the hope that this is the only mutating test. - // If so, we can completely avoid cloning functions. - let last_test = match tests.pop() { - None => return Err("no test commands found".to_string()), - Some(t) => t, - }; - - for (func, details) in testfile.functions { - let mut context = subtest::Context { - details: details, - verified: false, - }; - - for test in &tests { - try!(self.run_one_test(test.borrow(), Cow::Borrowed(&func), &mut context)); - } - // Run the last test with an owned function which means it won't need to clone it - // before mutating. - try!(self.run_one_test(last_test.borrow(), Cow::Owned(func), &mut context)); - } - - - // TODO: Actually run the tests. - Ok(started.elapsed()) - } - - fn run_one_test(&self, - test: &SubTest, - func: Cow, - context: &mut Context) - -> subtest::Result<()> { - let name = format!("{}({})", test.name(), func.name); - - // Should we run the verifier before this test? - if !context.verified && test.needs_verifier() { - try!(verify_function(&func).map_err(|e| e.to_string())); - context.verified = true; - } - - test.run(func, context).map_err(|e| format!("{}: {}", name, e)) - } } diff --git a/cranelift/src/tools/filetest/runone.rs b/cranelift/src/tools/filetest/runone.rs new file mode 100644 index 0000000000..e0d1e9d17b --- /dev/null +++ b/cranelift/src/tools/filetest/runone.rs @@ -0,0 +1,66 @@ +//! Run the tests in a single test file. + +use std::borrow::{Borrow, Cow}; +use std::path::Path; +use std::time; +use cretonne::ir::Function; +use cretonne::verify_function; +use cton_reader::parse_test; +use utils::read_to_string; +use filetest::{TestResult, new_subtest}; +use filetest::subtest::{SubTest, Context, Result}; + +/// Load `path` and run the test in it. +/// +/// If running this test causes a panic, it will propagate as normal. +pub fn run(path: &Path) -> TestResult { + let started = time::Instant::now(); + let buffer = try!(read_to_string(path).map_err(|e| e.to_string())); + let testfile = try!(parse_test(&buffer).map_err(|e| e.to_string())); + if testfile.functions.is_empty() { + return Err("no functions found".to_string()); + } + // Parse the test commands. + let mut tests = try!(testfile.commands.iter().map(new_subtest).collect::>>()); + + // Sort the tests so the mutators are at the end, and those that don't need the verifier are at + // the front. + tests.sort_by_key(|st| (st.is_mutating(), st.needs_verifier())); + + // Isolate the last test in the hope that this is the only mutating test. + // If so, we can completely avoid cloning functions. + let last_test = match tests.pop() { + None => return Err("no test commands found".to_string()), + Some(t) => t, + }; + + for (func, details) in testfile.functions { + let mut context = Context { + details: details, + verified: false, + }; + + for test in &tests { + try!(run_one_test(test.borrow(), Cow::Borrowed(&func), &mut context)); + } + // Run the last test with an owned function which means it won't need to clone it before + // mutating. + try!(run_one_test(last_test.borrow(), Cow::Owned(func), &mut context)); + } + + + // TODO: Actually run the tests. + Ok(started.elapsed()) +} + +fn run_one_test(test: &SubTest, func: Cow, context: &mut Context) -> Result<()> { + let name = format!("{}({})", test.name(), func.name); + + // Should we run the verifier before this test? + if !context.verified && test.needs_verifier() { + try!(verify_function(&func).map_err(|e| e.to_string())); + context.verified = true; + } + + test.run(func, context).map_err(|e| format!("{}: {}", name, e)) +} From 5ea9a439289b6c04b7584db53a562d2e0ae6bc8a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Sep 2016 21:48:20 -0700 Subject: [PATCH 0298/3084] Simplify job queue. Always keep PathBufs for every entry in the test list. When concurrent testing is enabled, we'll want to clone the path for the worker threads. Remove the Job struct for the same reason. --- cranelift/src/tools/filetest/runner.rs | 87 +++++++++++--------------- 1 file changed, 36 insertions(+), 51 deletions(-) diff --git a/cranelift/src/tools/filetest/runner.rs b/cranelift/src/tools/filetest/runner.rs index 7de2829296..44501489dd 100644 --- a/cranelift/src/tools/filetest/runner.rs +++ b/cranelift/src/tools/filetest/runner.rs @@ -5,17 +5,26 @@ use std::error::Error; use std::ffi::OsStr; -use std::mem; -use std::panic::catch_unwind; use std::path::{Path, PathBuf}; use filetest::{TestResult, runone}; use CommandResult; +struct QueueEntry { + path: PathBuf, + state: State, +} + #[derive(PartialEq, Eq, Debug)] -enum QueueEntry { - New(PathBuf), +enum State { + New, Running, - Done(PathBuf, TestResult), + Done(TestResult), +} + +impl QueueEntry { + pub fn path(&self) -> &Path { + self.path.as_path() + } } pub struct TestRunner { @@ -58,37 +67,35 @@ impl TestRunner { /// /// Any problems reading `file` as a test case file will be reported as a test failure. pub fn push_test>(&mut self, file: P) { - self.tests.push(QueueEntry::New(file.into())); + self.tests.push(QueueEntry { + path: file.into(), + state: State::New, + }); } /// Take a new test for running as a job. /// Leaves the queue entry marked as `Runnning`. - fn take_job(&mut self) -> Option { - let index = self.new_tests; - if index == self.tests.len() { + fn take_job(&mut self) -> Option { + let jobid = self.new_tests; + if jobid == self.tests.len() { return None; } self.new_tests += 1; - - let entry = mem::replace(&mut self.tests[index], QueueEntry::Running); - if let QueueEntry::New(path) = entry { - Some(Job::new(index, path)) - } else { - // Oh, sorry about that. Put the entry back. - self.tests[index] = entry; - None - } + assert_eq!(self.tests[jobid].state, State::New); + self.tests[jobid].state = State::Running; + Some(jobid) } /// Report the end of a job. - fn finish_job(&mut self, job: Job, result: TestResult) { - assert_eq!(self.tests[job.index], QueueEntry::Running); + fn finish_job(&mut self, jobid: usize, result: TestResult) { + assert_eq!(self.tests[jobid].state, State::Running); if let Err(ref e) = result { - self.job_error(&job.path, e); + self.job_error(jobid, e); } - self.tests[job.index] = QueueEntry::Done(job.path, result); - if job.index == self.finished_tests { - while let Some(&QueueEntry::Done(_, _)) = self.tests.get(self.finished_tests) { + self.tests[jobid].state = State::Done(result); + if jobid == self.finished_tests { + while let Some(&QueueEntry { state: State::Done(_), .. }) = self.tests + .get(self.finished_tests) { self.finished_tests += 1; } } @@ -154,16 +161,16 @@ impl TestRunner { } /// Report an error related to a job. - fn job_error(&mut self, path: &Path, err: &str) { + fn job_error(&mut self, jobid: usize, err: &str) { self.errors += 1; - println!("FAIL {}: {}", path.to_string_lossy(), err); + println!("FAIL {}: {}", self.tests[jobid].path.to_string_lossy(), err); } /// Schedule and new jobs to run. fn schedule_jobs(&mut self) { - while let Some(job) = self.take_job() { - let result = job.run(); - self.finish_job(job, result); + while let Some(jobid) = self.take_job() { + let result = runone::run(self.tests[jobid].path()); + self.finish_job(jobid, result); } } @@ -179,25 +186,3 @@ impl TestRunner { } } } - -/// A test file waiting to be run. -struct Job { - index: usize, - path: PathBuf, -} - -impl Job { - pub fn new(index: usize, path: PathBuf) -> Job { - Job { - index: index, - path: path, - } - } - - pub fn run(&self) -> TestResult { - match catch_unwind(|| runone::run(self.path.as_path())) { - Err(msg) => Err(format!("panic: {:?}", msg)), - Ok(result) => result, - } - } -} From b2f0dd7da3651d38182a1709486d22ee4e45127a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Sat, 17 Sep 2016 09:51:06 -0700 Subject: [PATCH 0299/3084] Run tests concurrently. Spin up one worker thread per cpu, and run filetests on all of them. Use a reorder buffer in Runner to make sure results are still reported in order. Individual test files given as command line arguments are still run synchronously for easier debugging. Only directories are run on worker threads. The recursive directory traversal is still happening on the main thread. Use a heartbeat thread to send ticks on the reply channel every second, and use the ticks to detect tests that are stuck. When Receiver::recv_timeout() is stabilized, we can probably get rid of the heartbeat thread. Catch panics on the worker threads and report them as test failures. --- cranelift/src/tools/Cargo.lock | 9 ++ cranelift/src/tools/Cargo.toml | 1 + cranelift/src/tools/filetest/concurrent.rs | 153 +++++++++++++++++++++ cranelift/src/tools/filetest/mod.rs | 2 + cranelift/src/tools/filetest/runner.rs | 131 ++++++++++++++---- cranelift/src/tools/main.rs | 1 + 6 files changed, 267 insertions(+), 30 deletions(-) create mode 100644 cranelift/src/tools/filetest/concurrent.rs diff --git a/cranelift/src/tools/Cargo.lock b/cranelift/src/tools/Cargo.lock index ca97206cc4..28d0f79f5b 100644 --- a/cranelift/src/tools/Cargo.lock +++ b/cranelift/src/tools/Cargo.lock @@ -6,6 +6,7 @@ dependencies = [ "cretonne-reader 0.0.0", "docopt 0.6.83 (registry+https://github.com/rust-lang/crates.io-index)", "filecheck 0.0.0", + "num_cpus 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)", "rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)", ] @@ -73,6 +74,14 @@ dependencies = [ "libc 0.2.16 (registry+https://github.com/rust-lang/crates.io-index)", ] +[[package]] +name = "num_cpus" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +dependencies = [ + "libc 0.2.16 (registry+https://github.com/rust-lang/crates.io-index)", +] + [[package]] name = "regex" version = "0.1.77" diff --git a/cranelift/src/tools/Cargo.toml b/cranelift/src/tools/Cargo.toml index b4572f9f49..3a1a8979a7 100644 --- a/cranelift/src/tools/Cargo.toml +++ b/cranelift/src/tools/Cargo.toml @@ -15,3 +15,4 @@ cretonne-reader = { path = "../libreader" } filecheck = { path = "../libfilecheck" } docopt = "0.6.80" rustc-serialize = "0.3.19" +num_cpus = "1.1.0" diff --git a/cranelift/src/tools/filetest/concurrent.rs b/cranelift/src/tools/filetest/concurrent.rs new file mode 100644 index 0000000000..3123124086 --- /dev/null +++ b/cranelift/src/tools/filetest/concurrent.rs @@ -0,0 +1,153 @@ +//! Run tests concurrently. +//! +//! This module provides the `ConcurrentRunner` struct which uses a pool of threads to run tests +//! concurrently. + +use std::panic::catch_unwind; +use std::path::{Path, PathBuf}; +use std::sync::mpsc::{channel, Sender, Receiver}; +use std::sync::{Arc, Mutex}; +use std::thread; +use std::time::Duration; +use num_cpus; +use filetest::{TestResult, runone}; + +// Request sent to worker threads contains jobid and path. +struct Request(usize, PathBuf); + +/// Reply from worker thread, +pub enum Reply { + Starting { jobid: usize, thread_num: usize }, + Done { jobid: usize, result: TestResult }, + Tick, +} + +/// Manage threads that run test jobs concurrently. +pub struct ConcurrentRunner { + // Channel for sending requests to the worker threads. + // The workers are sharing the receiver with an `Arc>`. + // This is `None` when shutting down. + request_tx: Option>, + + // Channel for receiving replies from the workers. + // Workers have their own `Sender`. + reply_rx: Receiver, + + handles: Vec>, +} + +impl ConcurrentRunner { + /// Create a new `ConcurrentRunner` with threads spun up. + pub fn new() -> ConcurrentRunner { + let (request_tx, request_rx) = channel(); + let request_mutex = Arc::new(Mutex::new(request_rx)); + let (reply_tx, reply_rx) = channel(); + + heartbeat_thread(reply_tx.clone()); + + let handles = (0..num_cpus::get()) + .map(|num| worker_thread(num, request_mutex.clone(), reply_tx.clone())) + .collect(); + + ConcurrentRunner { + request_tx: Some(request_tx), + reply_rx: reply_rx, + handles: handles, + } + } + + /// Shut down worker threads orderly. They will finish any queued jobs first. + pub fn shutdown(&mut self) { + self.request_tx = None; + } + + /// Join all the worker threads. + pub fn join(&mut self) { + assert!(self.request_tx.is_none(), "must shutdown before join"); + for h in self.handles.drain(..) { + if let Err(e) = h.join() { + println!("worker panicked: {:?}", e); + } + } + } + + /// Add a new job to the queues. + pub fn put(&mut self, jobid: usize, path: &Path) { + self.request_tx + .as_ref() + .expect("cannot push after shutdown") + .send(Request(jobid, path.to_owned())) + .expect("all the worker threads are gone"); + } + + /// Get a job reply without blocking. + pub fn try_get(&mut self) -> Option { + self.reply_rx.try_recv().ok() + } + + /// Get a job reply, blocking until one is available. + pub fn get(&mut self) -> Option { + self.reply_rx.recv().ok() + } +} + +/// Spawn a heartbeat thread which sends ticks down the reply channel every second. +/// This lets us implement timeouts without the not yet stable `recv_timeout`. +fn heartbeat_thread(replies: Sender) -> thread::JoinHandle<()> { + thread::Builder::new() + .name("heartbeat".to_string()) + .spawn(move || { + while replies.send(Reply::Tick).is_ok() { + thread::sleep(Duration::from_secs(1)); + } + }) + .unwrap() +} + +/// Spawn a worker thread running tests. +fn worker_thread(thread_num: usize, + requests: Arc>>, + replies: Sender) + -> thread::JoinHandle<()> { + thread::Builder::new() + .name(format!("worker #{}", thread_num)) + .spawn(move || { + loop { + // Lock the mutex only long enough to extract a request. + let Request(jobid, path) = match requests.lock().unwrap().recv() { + Err(..) => break, // TX end shuit down. exit thread. + Ok(req) => req, + }; + + // Tell them we're starting this job. + // The receiver should always be present for this as long as we have jobs. + replies.send(Reply::Starting { + jobid: jobid, + thread_num: thread_num, + }) + .unwrap(); + + let result = match catch_unwind(|| runone::run(path.as_path())) { + Ok(r) => r, + Err(e) => { + // The test panicked, leaving us a `Box`. + // Panics are usually strings. + if let Some(msg) = e.downcast_ref::() { + Err(format!("panicked in worker #{}: {}", thread_num, msg)) + } else if let Some(msg) = e.downcast_ref::<&'static str>() { + Err(format!("panicked in worker #{}: {}", thread_num, msg)) + } else { + Err(format!("panicked in worker #{}", thread_num)) + } + } + }; + + replies.send(Reply::Done { + jobid: jobid, + result: result, + }) + .unwrap(); + } + }) + .unwrap() +} diff --git a/cranelift/src/tools/filetest/mod.rs b/cranelift/src/tools/filetest/mod.rs index dec7ff6016..60b6180c05 100644 --- a/cranelift/src/tools/filetest/mod.rs +++ b/cranelift/src/tools/filetest/mod.rs @@ -14,6 +14,7 @@ use filetest::runner::TestRunner; pub mod subtest; mod runner; mod runone; +mod concurrent; mod domtree; mod verifier; @@ -40,6 +41,7 @@ pub fn run(files: Vec) -> CommandResult { } } + runner.start_threads(); runner.run() } diff --git a/cranelift/src/tools/filetest/runner.rs b/cranelift/src/tools/filetest/runner.rs index 44501489dd..c4cfca1ec3 100644 --- a/cranelift/src/tools/filetest/runner.rs +++ b/cranelift/src/tools/filetest/runner.rs @@ -7,8 +7,15 @@ use std::error::Error; use std::ffi::OsStr; use std::path::{Path, PathBuf}; use filetest::{TestResult, runone}; +use filetest::concurrent::{ConcurrentRunner, Reply}; use CommandResult; +// Timeout in seconds when we're not making progress. +const TIMEOUT_PANIC: usize = 10; + +// Timeout for reporting slow tests without panicking. +const TIMEOUT_SLOW: usize = 3; + struct QueueEntry { path: PathBuf, state: State, @@ -17,6 +24,7 @@ struct QueueEntry { #[derive(PartialEq, Eq, Debug)] enum State { New, + Queued, Running, Done(TestResult), } @@ -40,7 +48,13 @@ pub struct TestRunner { // Number of contiguous finished tests at the front of `tests`. finished_tests: usize, + // Number of errors seen so far. errors: usize, + + // Number of ticks received since we saw any progress. + ticks_since_progress: usize, + + threads: Option, } impl TestRunner { @@ -52,6 +66,8 @@ impl TestRunner { new_tests: 0, finished_tests: 0, errors: 0, + ticks_since_progress: 0, + threads: None, } } @@ -73,32 +89,10 @@ impl TestRunner { }); } - /// Take a new test for running as a job. - /// Leaves the queue entry marked as `Runnning`. - fn take_job(&mut self) -> Option { - let jobid = self.new_tests; - if jobid == self.tests.len() { - return None; - } - self.new_tests += 1; - assert_eq!(self.tests[jobid].state, State::New); - self.tests[jobid].state = State::Running; - Some(jobid) - } - - /// Report the end of a job. - fn finish_job(&mut self, jobid: usize, result: TestResult) { - assert_eq!(self.tests[jobid].state, State::Running); - if let Err(ref e) = result { - self.job_error(jobid, e); - } - self.tests[jobid].state = State::Done(result); - if jobid == self.finished_tests { - while let Some(&QueueEntry { state: State::Done(_), .. }) = self.tests - .get(self.finished_tests) { - self.finished_tests += 1; - } - } + /// Begin running tests concurrently. + pub fn start_threads(&mut self) { + assert!(self.threads.is_none()); + self.threads = Some(ConcurrentRunner::new()); } /// Scan any directories pushed so far. @@ -166,11 +160,87 @@ impl TestRunner { println!("FAIL {}: {}", self.tests[jobid].path.to_string_lossy(), err); } - /// Schedule and new jobs to run. + /// Schedule any new jobs to run. fn schedule_jobs(&mut self) { - while let Some(jobid) = self.take_job() { - let result = runone::run(self.tests[jobid].path()); - self.finish_job(jobid, result); + for jobid in self.new_tests..self.tests.len() { + assert_eq!(self.tests[jobid].state, State::New); + if let Some(ref mut conc) = self.threads { + // Queue test for concurrent execution. + self.tests[jobid].state = State::Queued; + conc.put(jobid, self.tests[jobid].path()); + } else { + // Run test synchronously. + self.tests[jobid].state = State::Running; + let result = runone::run(self.tests[jobid].path()); + self.finish_job(jobid, result); + } + self.new_tests = jobid + 1; + } + + // Check for any asynchronous replies without blocking. + while let Some(reply) = self.threads.as_mut().and_then(ConcurrentRunner::try_get) { + self.handle_reply(reply); + } + } + + /// Report the end of a job. + fn finish_job(&mut self, jobid: usize, result: TestResult) { + assert_eq!(self.tests[jobid].state, State::Running); + if let Err(ref e) = result { + self.job_error(jobid, e); + } + self.tests[jobid].state = State::Done(result); + if jobid == self.finished_tests { + while let Some(&QueueEntry { state: State::Done(_), .. }) = self.tests + .get(self.finished_tests) { + self.finished_tests += 1; + } + } + } + + /// Handle a reply from the async threads. + fn handle_reply(&mut self, reply: Reply) { + match reply { + Reply::Starting { jobid, .. } => { + assert_eq!(self.tests[jobid].state, State::Queued); + self.tests[jobid].state = State::Running; + } + Reply::Done { jobid, result } => { + self.ticks_since_progress = 0; + self.finish_job(jobid, result) + } + Reply::Tick => { + self.ticks_since_progress += 1; + if self.ticks_since_progress == TIMEOUT_SLOW { + println!("STALLED for {} seconds with {}/{} tests finished", + self.ticks_since_progress, + self.finished_tests, + self.tests.len()); + for jobid in self.finished_tests..self.tests.len() { + if self.tests[jobid].state == State::Running { + println!("slow: {}", self.tests[jobid].path.to_string_lossy()); + } + } + } + if self.ticks_since_progress >= TIMEOUT_PANIC { + panic!("worker threads stalled for {} seconds.", + self.ticks_since_progress); + } + } + } + } + + /// Drain the async jobs and shut down the threads. + fn drain_threads(&mut self) { + if let Some(mut conc) = self.threads.take() { + conc.shutdown(); + while self.finished_tests < self.tests.len() { + match conc.get() { + Some(reply) => self.handle_reply(reply), + None => break, + } + } + conc.join(); } } @@ -178,6 +248,7 @@ impl TestRunner { pub fn run(&mut self) -> CommandResult { self.scan_dirs(); self.schedule_jobs(); + self.drain_threads(); println!("{} tests", self.tests.len()); match self.errors { 0 => Ok(()), diff --git a/cranelift/src/tools/main.rs b/cranelift/src/tools/main.rs index af7a092394..436b07bb05 100644 --- a/cranelift/src/tools/main.rs +++ b/cranelift/src/tools/main.rs @@ -4,6 +4,7 @@ extern crate cton_reader; extern crate docopt; extern crate rustc_serialize; extern crate filecheck; +extern crate num_cpus; use cretonne::VERSION; use docopt::Docopt; From 81f26422fa9585101ac97ac14e232c4ad7887201 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Sat, 17 Sep 2016 11:29:52 -0700 Subject: [PATCH 0300/3084] Tell Travis to cache Cargo intermediate build products. The CI builds were using a lot of time downloading and building crates. --- .travis.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.travis.yml b/.travis.yml index 567d824194..33eed41b71 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,3 +4,7 @@ rust: - beta - nightly script: ./test-all.sh +cache: + - cargo + - directories: + - src/tools/target From a4774ce87d72e22bb4fe04eaa698de48b3567cf6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Sat, 17 Sep 2016 11:42:08 -0700 Subject: [PATCH 0301/3084] Record the location of parsed functions. Remember the location of the 'function' keyword that starts every function. This can be useful for reporting test failures etc. --- cranelift/src/libreader/parser.rs | 8 +++++--- cranelift/src/libreader/testfile.rs | 2 ++ 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 54a53310ee..6fea02b22b 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -523,7 +523,7 @@ impl<'a> Parser<'a> { self.comments.clear(); self.gather_comments(AnyEntity::Function); - let (name, sig) = try!(self.parse_function_spec()); + let (location, name, sig) = try!(self.parse_function_spec()); let mut ctx = Context::new(Function::with_name_signature(name, sig)); // function ::= function-spec * "{" preamble function-body "}" @@ -545,6 +545,7 @@ impl<'a> Parser<'a> { try!(ctx.rewrite_references()); let details = Details { + location: location, comments: mem::replace(&mut self.comments, Vec::new()), map: sourcemap::new(ctx.values, ctx.ebbs, ctx.stack_slots, ctx.jump_tables), }; @@ -556,8 +557,9 @@ impl<'a> Parser<'a> { // // function-spec ::= * "function" name signature // - fn parse_function_spec(&mut self) -> Result<(FunctionName, Signature)> { + fn parse_function_spec(&mut self) -> Result<(Location, FunctionName, Signature)> { try!(self.match_token(Token::Function, "expected 'function' keyword")); + let location = self.loc; // function-spec ::= "function" * name signature let name = try!(self.parse_function_name()); @@ -565,7 +567,7 @@ impl<'a> Parser<'a> { // function-spec ::= "function" name * signature let sig = try!(self.parse_signature()); - Ok((name, sig)) + Ok((location, name, sig)) } // Parse a function name. diff --git a/cranelift/src/libreader/testfile.rs b/cranelift/src/libreader/testfile.rs index 34cfabce83..8a9f8d3451 100644 --- a/cranelift/src/libreader/testfile.rs +++ b/cranelift/src/libreader/testfile.rs @@ -8,6 +8,7 @@ use cretonne::ir::Function; use cretonne::ir::entities::AnyEntity; use testcommand::TestCommand; use sourcemap::SourceMap; +use parser::Location; /// A parsed test case. /// @@ -24,6 +25,7 @@ pub struct TestFile<'a> { /// The details to not affect the semantics of the function. #[derive(Debug)] pub struct Details<'a> { + pub location: Location, pub comments: Vec>, pub map: SourceMap, } From 23887358dd31d9570bb1d595ac45f99f95b8dbd5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Sat, 17 Sep 2016 12:36:35 -0700 Subject: [PATCH 0302/3084] Simplify with unwrap_or_else(). --- cranelift/src/tools/filetest/concurrent.rs | 23 ++++++++++------------ 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/cranelift/src/tools/filetest/concurrent.rs b/cranelift/src/tools/filetest/concurrent.rs index 3123124086..30abb844d4 100644 --- a/cranelift/src/tools/filetest/concurrent.rs +++ b/cranelift/src/tools/filetest/concurrent.rs @@ -127,20 +127,17 @@ fn worker_thread(thread_num: usize, }) .unwrap(); - let result = match catch_unwind(|| runone::run(path.as_path())) { - Ok(r) => r, - Err(e) => { - // The test panicked, leaving us a `Box`. - // Panics are usually strings. - if let Some(msg) = e.downcast_ref::() { - Err(format!("panicked in worker #{}: {}", thread_num, msg)) - } else if let Some(msg) = e.downcast_ref::<&'static str>() { - Err(format!("panicked in worker #{}: {}", thread_num, msg)) - } else { - Err(format!("panicked in worker #{}", thread_num)) - } + let result = catch_unwind(|| runone::run(path.as_path())).unwrap_or_else(|e| { + // The test panicked, leaving us a `Box`. + // Panics are usually strings. + if let Some(msg) = e.downcast_ref::() { + Err(format!("panicked in worker #{}: {}", thread_num, msg)) + } else if let Some(msg) = e.downcast_ref::<&'static str>() { + Err(format!("panicked in worker #{}: {}", thread_num, msg)) + } else { + Err(format!("panicked in worker #{}", thread_num)) } - }; + }); replies.send(Reply::Done { jobid: jobid, From 67c8ae7f14e13c6d172574d4a75f51c0e1758c14 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Sat, 17 Sep 2016 17:11:17 -0700 Subject: [PATCH 0303/3084] Add --verbose flag to cton-util test. --- cranelift/src/tools/filetest/mod.rs | 4 +- cranelift/src/tools/filetest/runner.rs | 65 ++++++++++++++++++-------- cranelift/src/tools/main.rs | 4 +- 3 files changed, 50 insertions(+), 23 deletions(-) diff --git a/cranelift/src/tools/filetest/mod.rs b/cranelift/src/tools/filetest/mod.rs index 60b6180c05..a924cb04eb 100644 --- a/cranelift/src/tools/filetest/mod.rs +++ b/cranelift/src/tools/filetest/mod.rs @@ -30,8 +30,8 @@ pub type TestResult = Result; /// Directories are scanned recursively for test cases ending in `.cton`. These test cases are /// executed on background threads. /// -pub fn run(files: Vec) -> CommandResult { - let mut runner = TestRunner::new(); +pub fn run(verbose: bool, files: Vec) -> CommandResult { + let mut runner = TestRunner::new(verbose); for path in files.iter().map(Path::new) { if path.is_file() { diff --git a/cranelift/src/tools/filetest/runner.rs b/cranelift/src/tools/filetest/runner.rs index c4cfca1ec3..11a98fb43a 100644 --- a/cranelift/src/tools/filetest/runner.rs +++ b/cranelift/src/tools/filetest/runner.rs @@ -4,6 +4,7 @@ //! scanning directories for tests. use std::error::Error; +use std::fmt::{self, Display}; use std::ffi::OsStr; use std::path::{Path, PathBuf}; use filetest::{TestResult, runone}; @@ -35,7 +36,26 @@ impl QueueEntry { } } +impl Display for QueueEntry { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + let p = self.path.to_string_lossy(); + match self.state { + State::Done(Ok(dur)) => { + write!(f, + "{}.{:03} {}", + dur.as_secs(), + dur.subsec_nanos() / 1000000, + p) + } + State::Done(Err(ref e)) => write!(f, "FAIL {}: {}", p, e), + _ => write!(f, "{}", p), + } + } +} + pub struct TestRunner { + verbose: bool, + // Directories that have not yet been scanned. dir_stack: Vec, @@ -45,8 +65,8 @@ pub struct TestRunner { // Pointer into `tests` where the `New` entries begin. new_tests: usize, - // Number of contiguous finished tests at the front of `tests`. - finished_tests: usize, + // Number of contiguous reported tests at the front of `tests`. + reported_tests: usize, // Number of errors seen so far. errors: usize, @@ -59,12 +79,13 @@ pub struct TestRunner { impl TestRunner { /// Create a new blank TrstRunner. - pub fn new() -> TestRunner { + pub fn new(verbose: bool) -> TestRunner { TestRunner { + verbose: verbose, dir_stack: Vec::new(), tests: Vec::new(), new_tests: 0, - finished_tests: 0, + reported_tests: 0, errors: 0, ticks_since_progress: 0, threads: None, @@ -154,10 +175,17 @@ impl TestRunner { println!("{}: {}", path.to_string_lossy(), err); } - /// Report an error related to a job. - fn job_error(&mut self, jobid: usize, err: &str) { - self.errors += 1; - println!("FAIL {}: {}", self.tests[jobid].path.to_string_lossy(), err); + /// Report on the next in-order job, if it's done. + fn report_job(&self) -> bool { + let jobid = self.reported_tests; + if let Some(&QueueEntry { state: State::Done(ref result), .. }) = self.tests.get(jobid) { + if self.verbose || result.is_err() { + println!("{}", self.tests[jobid]); + } + true + } else { + false + } } /// Schedule any new jobs to run. @@ -186,15 +214,14 @@ impl TestRunner { /// Report the end of a job. fn finish_job(&mut self, jobid: usize, result: TestResult) { assert_eq!(self.tests[jobid].state, State::Running); - if let Err(ref e) = result { - self.job_error(jobid, e); + if result.is_err() { + self.errors += 1; } self.tests[jobid].state = State::Done(result); - if jobid == self.finished_tests { - while let Some(&QueueEntry { state: State::Done(_), .. }) = self.tests - .get(self.finished_tests) { - self.finished_tests += 1; - } + + // Rports jobs in order. + while self.report_job() { + self.reported_tests += 1; } } @@ -214,11 +241,11 @@ impl TestRunner { if self.ticks_since_progress == TIMEOUT_SLOW { println!("STALLED for {} seconds with {}/{} tests finished", self.ticks_since_progress, - self.finished_tests, + self.reported_tests, self.tests.len()); - for jobid in self.finished_tests..self.tests.len() { + for jobid in self.reported_tests..self.tests.len() { if self.tests[jobid].state == State::Running { - println!("slow: {}", self.tests[jobid].path.to_string_lossy()); + println!("slow: {}", self.tests[jobid]); } } } @@ -234,7 +261,7 @@ impl TestRunner { fn drain_threads(&mut self) { if let Some(mut conc) = self.threads.take() { conc.shutdown(); - while self.finished_tests < self.tests.len() { + while self.reported_tests < self.tests.len() { match conc.get() { Some(reply) => self.handle_reply(reply), None => break, diff --git a/cranelift/src/tools/main.rs b/cranelift/src/tools/main.rs index 436b07bb05..35a5c8a564 100644 --- a/cranelift/src/tools/main.rs +++ b/cranelift/src/tools/main.rs @@ -21,7 +21,7 @@ const USAGE: &'static str = " Cretonne code generator utility Usage: - cton-util test ... + cton-util test [-v] ... cton-util cat ... cton-util filecheck [-v] cton-util print-cfg ... @@ -60,7 +60,7 @@ fn cton_util() -> CommandResult { // Find the sub-command to execute. if args.cmd_test { - filetest::run(args.arg_file) + filetest::run(args.flag_verbose, args.arg_file) } else if args.cmd_cat { cat::run(args.arg_file) } else if args.cmd_filecheck { From b5828cb9a3988b3ae2952582f1996b7cdc366ba3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Sun, 18 Sep 2016 11:35:10 -0700 Subject: [PATCH 0304/3084] Print out a report of slow-running tests. The slow tests are computed as those that would be printed as outliers on a boxplot of all the test runtimes. These are more than 1.5 inter-quartile range away from the 75% quartile. --- cranelift/src/tools/filetest/runner.rs | 44 ++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/cranelift/src/tools/filetest/runner.rs b/cranelift/src/tools/filetest/runner.rs index 11a98fb43a..8794de0004 100644 --- a/cranelift/src/tools/filetest/runner.rs +++ b/cranelift/src/tools/filetest/runner.rs @@ -271,11 +271,55 @@ impl TestRunner { } } + /// Print out a report of slow tests. + fn report_slow_tests(&self) { + // Collect runtimes of succeeded tests. + let mut times = self.tests + .iter() + .filter_map(|entry| match *entry { + QueueEntry { state: State::Done(Ok(dur)), .. } => Some(dur), + _ => None, + }) + .collect::>(); + + // Get me some real data, kid. + let len = times.len(); + if len < 4 { + return; + } + + // Compute quartiles. + times.sort(); + let qlen = len / 4; + let q1 = times[qlen]; + let q3 = times[len - 1 - qlen]; + // Inter-quartile range. + let iqr = q3 - q1; + + // Cut-off for what we consider a 'slow' test: 1.5 IQR from the 75% quartile. + // These are the data points that would be plotted as outliers outside a box plot. + let cut = q3 + iqr * 2 / 3; + if cut > *times.last().unwrap() { + return; + } + + for t in self.tests + .iter() + .filter(|entry| match **entry { + QueueEntry { state: State::Done(Ok(dur)), .. } => dur > cut, + _ => false, + }) { + println!("slow: {}", t) + } + + } + /// Scan pushed directories for tests and run them. pub fn run(&mut self) -> CommandResult { self.scan_dirs(); self.schedule_jobs(); self.drain_threads(); + self.report_slow_tests(); println!("{} tests", self.tests.len()); match self.errors { 0 => Ok(()), From ac46de7200202faafbfd045b1f82876fa2c357b2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 19 Sep 2016 10:08:21 -0700 Subject: [PATCH 0305/3084] Extract Result and Error into their own module. Also include the err! macro and make it usable outside the module. --- cranelift/src/libreader/error.rs | 44 +++++++++++++++++++++++++++++ cranelift/src/libreader/lexer.rs | 8 ++---- cranelift/src/libreader/lib.rs | 4 ++- cranelift/src/libreader/parser.rs | 40 ++------------------------ cranelift/src/libreader/testfile.rs | 2 +- 5 files changed, 53 insertions(+), 45 deletions(-) create mode 100644 cranelift/src/libreader/error.rs diff --git a/cranelift/src/libreader/error.rs b/cranelift/src/libreader/error.rs new file mode 100644 index 0000000000..edc6e40649 --- /dev/null +++ b/cranelift/src/libreader/error.rs @@ -0,0 +1,44 @@ +//! Define the `Location`, `Error`, and `Result` types. + +#![macro_use] + +use std::fmt; +use std::result; + +/// The location of a `Token` or `Error`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Location { + pub line_number: usize, +} + +/// A parse error is returned when the parse failed. +#[derive(Debug)] +pub struct Error { + pub location: Location, + pub message: String, +} + +impl fmt::Display for Error { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{}: {}", self.location.line_number, self.message) + } +} + +pub type Result = result::Result; + +// Create an `Err` variant of `Result` from a location and `format!` args. +macro_rules! err { + ( $loc:expr, $msg:expr ) => { + Err($crate::Error { + location: $loc.clone(), + message: String::from($msg), + }) + }; + + ( $loc:expr, $fmt:expr, $( $arg:expr ),+ ) => { + Err($crate::Error { + location: $loc.clone(), + message: format!( $fmt, $( $arg ),+ ), + }) + }; +} diff --git a/cranelift/src/libreader/lexer.rs b/cranelift/src/libreader/lexer.rs index 6b5e881a1f..7631e306ad 100644 --- a/cranelift/src/libreader/lexer.rs +++ b/cranelift/src/libreader/lexer.rs @@ -8,12 +8,7 @@ use std::str::CharIndices; use cretonne::ir::types; use cretonne::ir::{Value, Ebb}; - -/// The location of a `Token` or `Error`. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub struct Location { - pub line_number: usize, -} +use error::Location; /// A Token returned from the `Lexer`. /// @@ -379,6 +374,7 @@ mod tests { use super::*; use cretonne::ir::types; use cretonne::ir::{Value, Ebb}; + use error::Location; fn token<'a>(token: Token<'a>, line: usize) -> Option, LocatedError>> { Some(super::token(token, Location { line_number: line })) diff --git a/cranelift/src/libreader/lib.rs b/cranelift/src/libreader/lib.rs index 79f9b23521..f0f04fa381 100644 --- a/cranelift/src/libreader/lib.rs +++ b/cranelift/src/libreader/lib.rs @@ -5,11 +5,13 @@ extern crate cretonne; -pub use parser::{Result, parse_functions, parse_test}; +pub use error::{Location, Result, Error}; +pub use parser::{parse_functions, parse_test}; pub use testcommand::{TestCommand, TestOption}; pub use testfile::{TestFile, Details}; pub use sourcemap::SourceMap; +mod error; mod lexer; mod parser; mod testcommand; diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 6fea02b22b..68a5e72a4f 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -6,12 +6,9 @@ // ====--------------------------------------------------------------------------------------====// use std::collections::HashMap; -use std::result; -use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::u32; use std::mem; -use lexer::{self, Lexer, Token}; use cretonne::ir::{Function, Ebb, Inst, Opcode, Value, Type, FunctionName, StackSlot, StackSlotData, JumpTable, JumpTableData}; use cretonne::ir::types::{VOID, Signature, ArgumentType, ArgumentExtension}; @@ -20,11 +17,11 @@ use cretonne::ir::entities::{AnyEntity, NO_EBB, NO_INST, NO_VALUE}; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, JumpData, BranchData, ReturnData}; use testfile::{TestFile, Details, Comment}; +use error::{Location, Error, Result}; +use lexer::{self, Lexer, Token}; use testcommand::TestCommand; use sourcemap; -pub use lexer::Location; - /// Parse the entire `text` into a list of functions. /// /// Any test commands or ISA declarations are ignored. @@ -43,38 +40,6 @@ pub fn parse_test<'a>(text: &'a str) -> Result> { }) } -/// A parse error is returned when the parse failed. -#[derive(Debug)] -pub struct Error { - pub location: Location, - pub message: String, -} - -impl Display for Error { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - write!(f, "{}: {}", self.location.line_number, self.message) - } -} - -pub type Result = result::Result; - -// Create an `Err` variant of `Result` from a location and `format!` args. -macro_rules! err { - ( $loc:expr, $msg:expr ) => { - Err(Error { - location: $loc.clone(), - message: String::from($msg), - }) - }; - - ( $loc:expr, $fmt:expr, $( $arg:expr ),+ ) => { - Err(Error { - location: $loc.clone(), - message: format!( $fmt, $( $arg ),+ ), - }) - }; -} - pub struct Parser<'a> { lex: Lexer<'a>, @@ -1225,6 +1190,7 @@ mod tests { use cretonne::ir::types::{self, ArgumentType, ArgumentExtension}; use cretonne::ir::entities::AnyEntity; use testfile::{Details, Comment}; + use error::Error; #[test] fn argument_type() { diff --git a/cranelift/src/libreader/testfile.rs b/cranelift/src/libreader/testfile.rs index 8a9f8d3451..d63b1f9f9d 100644 --- a/cranelift/src/libreader/testfile.rs +++ b/cranelift/src/libreader/testfile.rs @@ -8,7 +8,7 @@ use cretonne::ir::Function; use cretonne::ir::entities::AnyEntity; use testcommand::TestCommand; use sourcemap::SourceMap; -use parser::Location; +use error::Location; /// A parsed test case. /// From b5514748a4bd901dd0614db9ff7c3b6eb50d9aa2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 19 Sep 2016 11:11:30 -0700 Subject: [PATCH 0306/3084] Add an internal MutableSourceMap trait. Use the SourceMap for mapping during parsing too. --- cranelift/src/libreader/parser.rs | 125 ++++++-------------------- cranelift/src/libreader/sourcemap.rs | 130 ++++++++++++++++++++++----- 2 files changed, 138 insertions(+), 117 deletions(-) diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 68a5e72a4f..b2c9f40d6f 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -5,12 +5,11 @@ // // ====--------------------------------------------------------------------------------------====// -use std::collections::HashMap; use std::str::FromStr; use std::u32; use std::mem; -use cretonne::ir::{Function, Ebb, Inst, Opcode, Value, Type, FunctionName, StackSlot, - StackSlotData, JumpTable, JumpTableData}; +use cretonne::ir::{Function, Ebb, Inst, Opcode, Value, Type, FunctionName, StackSlotData, + JumpTable, JumpTableData}; use cretonne::ir::types::{VOID, Signature, ArgumentType, ArgumentExtension}; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::{AnyEntity, NO_EBB, NO_INST, NO_VALUE}; @@ -20,7 +19,7 @@ use testfile::{TestFile, Details, Comment}; use error::{Location, Error, Result}; use lexer::{self, Lexer, Token}; use testcommand::TestCommand; -use sourcemap; +use sourcemap::{SourceMap, MutableSourceMap}; /// Parse the entire `text` into a list of functions. /// @@ -65,10 +64,7 @@ pub struct Parser<'a> { // file by number. We need to map these numbers to real references. struct Context { function: Function, - stack_slots: HashMap, // ssNN - jump_tables: HashMap, // jtNN - ebbs: HashMap, // ebbNN - values: HashMap, // vNN, vxNN + map: SourceMap, // Remember the location of every instruction. inst_locs: Vec<(Inst, Location)>, @@ -78,36 +74,25 @@ impl Context { fn new(f: Function) -> Context { Context { function: f, - stack_slots: HashMap::new(), - jump_tables: HashMap::new(), - ebbs: HashMap::new(), - values: HashMap::new(), + map: SourceMap::new(), inst_locs: Vec::new(), } } // Allocate a new stack slot and add a mapping number -> StackSlot. fn add_ss(&mut self, number: u32, data: StackSlotData, loc: &Location) -> Result<()> { - if self.stack_slots.insert(number, self.function.stack_slots.push(data)).is_some() { - err!(loc, "duplicate stack slot: ss{}", number) - } else { - Ok(()) - } + self.map.def_ss(number, self.function.stack_slots.push(data), loc) } // Allocate a new jump table and add a mapping number -> JumpTable. fn add_jt(&mut self, number: u32, data: JumpTableData, loc: &Location) -> Result<()> { - if self.jump_tables.insert(number, self.function.jump_tables.push(data)).is_some() { - err!(loc, "duplicate jump table: jt{}", number) - } else { - Ok(()) - } + self.map.def_jt(number, self.function.jump_tables.push(data), loc) } // Resolve a reference to a jump table. fn get_jt(&self, number: u32, loc: &Location) -> Result { - match self.jump_tables.get(&number) { - Some(&jt) => Ok(jt), + match self.map.get_jt(number) { + Some(jt) => Ok(jt), None => err!(loc, "undefined jump table jt{}", number), } } @@ -116,20 +101,7 @@ impl Context { fn add_ebb(&mut self, src_ebb: Ebb, loc: &Location) -> Result { let ebb = self.function.dfg.make_ebb(); self.function.layout.append_ebb(ebb); - if self.ebbs.insert(src_ebb, ebb).is_some() { - err!(loc, "duplicate EBB: {}", src_ebb) - } else { - Ok(ebb) - } - } - - // Add a value mapping src_val -> data. - fn add_value(&mut self, src_val: Value, data: Value, loc: &Location) -> Result<()> { - if self.values.insert(src_val, data).is_some() { - err!(loc, "duplicate value: {}", src_val) - } else { - Ok(()) - } + self.map.def_ebb(src_ebb, ebb, loc).and(Ok(ebb)) } // Record the location of an instuction. @@ -140,41 +112,6 @@ impl Context { // The parser creates all instructions with Ebb and Value references using the source file // numbering. These references need to be rewritten after parsing is complete since forward // references are allowed. - - // Rewrite an Ebb reference. - fn rewrite_ebb(map: &HashMap, ebb: &mut Ebb, loc: &Location) -> Result<()> { - match map.get(ebb) { - Some(&new) => { - *ebb = new; - Ok(()) - } - None => err!(loc, "undefined reference: {}", ebb), - } - } - - // Rewrite a value reference. - fn rewrite_value(map: &HashMap, val: &mut Value, loc: &Location) -> Result<()> { - match map.get(val) { - Some(&new) => { - *val = new; - Ok(()) - } - None => err!(loc, "undefined reference: {}", val), - } - } - - // Rewrite a slice of value references. - fn rewrite_values(map: &HashMap, - vals: &mut [Value], - loc: &Location) - -> Result<()> { - for val in vals { - try!(Self::rewrite_value(map, val, loc)); - } - Ok(()) - } - - // Rewrite all EBB and value references in the function. fn rewrite_references(&mut self) -> Result<()> { for &(inst, loc) in &self.inst_locs { match self.function.dfg[inst] { @@ -189,7 +126,7 @@ impl Context { InstructionData::BinaryImmRev { ref mut arg, .. } | InstructionData::ExtractLane { ref mut arg, .. } | InstructionData::BranchTable { ref mut arg, .. } => { - try!(Self::rewrite_value(&self.values, arg, &loc)); + try!(self.map.rewrite_value(arg, &loc)); } InstructionData::Binary { ref mut args, .. } | @@ -197,30 +134,30 @@ impl Context { InstructionData::InsertLane { ref mut args, .. } | InstructionData::IntCompare { ref mut args, .. } | InstructionData::FloatCompare { ref mut args, .. } => { - try!(Self::rewrite_values(&self.values, args, &loc)); + try!(self.map.rewrite_values(args, &loc)); } InstructionData::Ternary { ref mut args, .. } => { - try!(Self::rewrite_values(&self.values, args, &loc)); + try!(self.map.rewrite_values(args, &loc)); } InstructionData::Jump { ref mut data, .. } => { - try!(Self::rewrite_ebb(&self.ebbs, &mut data.destination, &loc)); - try!(Self::rewrite_values(&self.values, &mut data.arguments, &loc)); + try!(self.map.rewrite_ebb(&mut data.destination, &loc)); + try!(self.map.rewrite_values(&mut data.arguments, &loc)); } InstructionData::Branch { ref mut data, .. } => { - try!(Self::rewrite_value(&self.values, &mut data.arg, &loc)); - try!(Self::rewrite_ebb(&self.ebbs, &mut data.destination, &loc)); - try!(Self::rewrite_values(&self.values, &mut data.arguments, &loc)); + try!(self.map.rewrite_value(&mut data.arg, &loc)); + try!(self.map.rewrite_ebb(&mut data.destination, &loc)); + try!(self.map.rewrite_values(&mut data.arguments, &loc)); } InstructionData::Call { ref mut data, .. } => { - try!(Self::rewrite_values(&self.values, &mut data.args, &loc)); + try!(self.map.rewrite_values(&mut data.args, &loc)); } InstructionData::Return { ref mut data, .. } => { - try!(Self::rewrite_values(&self.values, &mut data.args, &loc)); + try!(self.map.rewrite_values(&mut data.args, &loc)); } } } @@ -230,7 +167,7 @@ impl Context { for jt in self.function.jump_tables.keys() { for ebb in self.function.jump_tables[jt].as_mut_slice() { if *ebb != NO_EBB { - try!(Self::rewrite_ebb(&self.ebbs, ebb, &loc)); + try!(self.map.rewrite_ebb(ebb, &loc)); } } } @@ -512,7 +449,7 @@ impl<'a> Parser<'a> { let details = Details { location: location, comments: mem::replace(&mut self.comments, Vec::new()), - map: sourcemap::new(ctx.values, ctx.ebbs, ctx.stack_slots, ctx.jump_tables), + map: ctx.map, }; Ok((ctx.function, details)) @@ -777,7 +714,7 @@ impl<'a> Parser<'a> { let t = try!(self.match_type("expected EBB argument type")); // Allocate the EBB argument and add the mapping. let value = ctx.function.dfg.append_ebb_arg(ebb, t); - ctx.add_value(vx, value, &vx_location) + ctx.map.def_value(vx, value, &vx_location) } // Parse an instruction, append it to `ebb`. @@ -856,7 +793,7 @@ impl<'a> Parser<'a> { // Now map the source result values to the just created instruction results. // Pass a reference to `ctx.values` instead of `ctx` itself since the `Values` iterator // holds a reference to `ctx.function`. - self.add_values(&mut ctx.values, + self.add_values(&mut ctx.map, results.into_iter(), ctx.function.dfg.inst_results(inst)) } @@ -888,8 +825,8 @@ impl<'a> Parser<'a> { // layout of the blocks. let ctrl_src_value = inst_data.typevar_operand() .expect("Constraints <-> Format inconsistency"); - ctx.function.dfg.value_type(match ctx.values.get(&ctrl_src_value) { - Some(&v) => v, + ctx.function.dfg.value_type(match ctx.map.get_value(ctrl_src_value) { + Some(v) => v, None => { return err!(self.loc, "cannot determine type of operand {}", @@ -932,18 +869,12 @@ impl<'a> Parser<'a> { } // Add mappings for a list of source values to their corresponding new values. - fn add_values(&self, - values: &mut HashMap, - results: S, - new_results: V) - -> Result<()> + fn add_values(&self, map: &mut SourceMap, results: S, new_results: V) -> Result<()> where S: Iterator, V: Iterator { for (src, val) in results.zip(new_results) { - if values.insert(src, val).is_some() { - return err!(self.loc, "duplicate result value: {}", src); - } + try!(map.def_value(src, val, &self.loc)); } Ok(()) } diff --git a/cranelift/src/libreader/sourcemap.rs b/cranelift/src/libreader/sourcemap.rs index 9a601a3e39..090e438d54 100644 --- a/cranelift/src/libreader/sourcemap.rs +++ b/cranelift/src/libreader/sourcemap.rs @@ -10,6 +10,7 @@ use std::collections::HashMap; use cretonne::ir::{StackSlot, JumpTable, Ebb, Value}; use cretonne::ir::entities::AnyEntity; +use error::{Result, Location}; /// Mapping from source entity names to entity references that are valid in the parsed function. #[derive(Debug)] @@ -22,6 +23,26 @@ pub struct SourceMap { /// Read-only interface which is exposed outside the parser crate. impl SourceMap { + /// Look up a value entity by its source number. + pub fn get_value(&self, src: Value) -> Option { + self.values.get(&src).cloned() + } + + /// Look up a EBB entity by its source number. + pub fn get_ebb(&self, src: Ebb) -> Option { + self.ebbs.get(&src).cloned() + } + + /// Look up a stack slot entity by its source number. + pub fn get_ss(&self, src_num: u32) -> Option { + self.stack_slots.get(&src_num).cloned() + } + + /// Look up a jump table entity by its source number. + pub fn get_jt(&self, src_num: u32) -> Option { + self.jump_tables.get(&src_num).cloned() + } + /// Look up an entity by source name. /// Returns the entity reference corresponding to `name`, if it exists. pub fn lookup_str(&self, name: &str) -> Option { @@ -29,25 +50,51 @@ impl SourceMap { match ent { "v" => { Value::direct_with_number(num) - .and_then(|v| self.values.get(&v).cloned()) + .and_then(|v| self.get_value(v)) .map(AnyEntity::Value) } "vx" => { Value::table_with_number(num) - .and_then(|v| self.values.get(&v).cloned()) + .and_then(|v| self.get_value(v)) .map(AnyEntity::Value) } - "ebb" => { - Ebb::with_number(num) - .and_then(|e| self.ebbs.get(&e).cloned()) - .map(AnyEntity::Ebb) - } - "ss" => self.stack_slots.get(&num).cloned().map(AnyEntity::StackSlot), - "jt" => self.jump_tables.get(&num).cloned().map(AnyEntity::JumpTable), + "ebb" => Ebb::with_number(num).and_then(|e| self.get_ebb(e)).map(AnyEntity::Ebb), + "ss" => self.get_ss(num).map(AnyEntity::StackSlot), + "jt" => self.get_jt(num).map(AnyEntity::JumpTable), _ => None, } }) } + + /// Rewrite an Ebb reference. + pub fn rewrite_ebb(&self, ebb: &mut Ebb, loc: &Location) -> Result<()> { + match self.get_ebb(*ebb) { + Some(new) => { + *ebb = new; + Ok(()) + } + None => err!(loc, "undefined reference: {}", ebb), + } + } + + /// Rewrite a value reference. + pub fn rewrite_value(&self, val: &mut Value, loc: &Location) -> Result<()> { + match self.get_value(*val) { + Some(new) => { + *val = new; + Ok(()) + } + None => err!(loc, "undefined reference: {}", val), + } + } + + /// Rewrite a slice of value references. + pub fn rewrite_values(&self, vals: &mut [Value], loc: &Location) -> Result<()> { + for val in vals { + try!(self.rewrite_value(val, loc)); + } + Ok(()) + } } /// Get the number of decimal digits at the end of `s`. @@ -67,17 +114,60 @@ fn split_entity_name(name: &str) -> Option<(&str, u32)> { } } -/// Create a new SourceMap from all the individual mappings. -pub fn new(values: HashMap, - ebbs: HashMap, - stack_slots: HashMap, - jump_tables: HashMap) - -> SourceMap { - SourceMap { - values: values, - ebbs: ebbs, - stack_slots: stack_slots, - jump_tables: jump_tables, + +/// Interface for mutating a source map. +/// +/// This interface is provided for the parser itself, it is not made available outside the crate. +pub trait MutableSourceMap { + fn new() -> Self; + + /// Define a value mapping from the source name `src` to the final `entity`. + fn def_value(&mut self, src: Value, entity: Value, loc: &Location) -> Result<()>; + fn def_ebb(&mut self, src: Ebb, entity: Ebb, loc: &Location) -> Result<()>; + fn def_ss(&mut self, src_num: u32, entity: StackSlot, loc: &Location) -> Result<()>; + fn def_jt(&mut self, src_num: u32, entity: JumpTable, loc: &Location) -> Result<()>; +} + +impl MutableSourceMap for SourceMap { + fn new() -> SourceMap { + SourceMap { + values: HashMap::new(), + ebbs: HashMap::new(), + stack_slots: HashMap::new(), + jump_tables: HashMap::new(), + } + } + + fn def_value(&mut self, src: Value, entity: Value, loc: &Location) -> Result<()> { + if self.values.insert(src, entity).is_some() { + err!(loc, "duplicate value: {}", src) + } else { + Ok(()) + } + } + + fn def_ebb(&mut self, src: Ebb, entity: Ebb, loc: &Location) -> Result<()> { + if self.ebbs.insert(src, entity).is_some() { + err!(loc, "duplicate EBB: {}", src) + } else { + Ok(()) + } + } + + fn def_ss(&mut self, src_num: u32, entity: StackSlot, loc: &Location) -> Result<()> { + if self.stack_slots.insert(src_num, entity).is_some() { + err!(loc, "duplicate stack slot: ss{}", src_num) + } else { + Ok(()) + } + } + + fn def_jt(&mut self, src_num: u32, entity: JumpTable, loc: &Location) -> Result<()> { + if self.jump_tables.insert(src_num, entity).is_some() { + err!(loc, "duplicate jump table: jt{}", src_num) + } else { + Ok(()) + } } } From 1a73b4f3f2f955c743924ffc157b084a559010af Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 19 Sep 2016 11:29:55 -0700 Subject: [PATCH 0307/3084] Also record locations for tracked entities. --- cranelift/src/libreader/sourcemap.rs | 32 +++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/cranelift/src/libreader/sourcemap.rs b/cranelift/src/libreader/sourcemap.rs index 090e438d54..e7828a1803 100644 --- a/cranelift/src/libreader/sourcemap.rs +++ b/cranelift/src/libreader/sourcemap.rs @@ -8,7 +8,7 @@ //! clients. use std::collections::HashMap; -use cretonne::ir::{StackSlot, JumpTable, Ebb, Value}; +use cretonne::ir::{StackSlot, JumpTable, Ebb, Value, Inst}; use cretonne::ir::entities::AnyEntity; use error::{Result, Location}; @@ -19,6 +19,9 @@ pub struct SourceMap { ebbs: HashMap, // ebbNN stack_slots: HashMap, // ssNN jump_tables: HashMap, // jtNN + + // Store locations for entities, including instructions. + locations: HashMap, } /// Read-only interface which is exposed outside the parser crate. @@ -66,6 +69,12 @@ impl SourceMap { }) } + /// Get the source location where an entity was defined. + /// This looks up entities in the parsed function, not the source entity numbers. + pub fn location(&self, entity: AnyEntity) -> Option { + self.locations.get(&entity).cloned() + } + /// Rewrite an Ebb reference. pub fn rewrite_ebb(&self, ebb: &mut Ebb, loc: &Location) -> Result<()> { match self.get_ebb(*ebb) { @@ -126,6 +135,10 @@ pub trait MutableSourceMap { fn def_ebb(&mut self, src: Ebb, entity: Ebb, loc: &Location) -> Result<()>; fn def_ss(&mut self, src_num: u32, entity: StackSlot, loc: &Location) -> Result<()>; fn def_jt(&mut self, src_num: u32, entity: JumpTable, loc: &Location) -> Result<()>; + + /// Define an instruction. Since instruction numbers never appear in source, only the location + /// is recorded. + fn def_inst(&mut self, entity: Inst, loc: &Location) -> Result<()>; } impl MutableSourceMap for SourceMap { @@ -135,12 +148,15 @@ impl MutableSourceMap for SourceMap { ebbs: HashMap::new(), stack_slots: HashMap::new(), jump_tables: HashMap::new(), + locations: HashMap::new(), } } fn def_value(&mut self, src: Value, entity: Value, loc: &Location) -> Result<()> { if self.values.insert(src, entity).is_some() { err!(loc, "duplicate value: {}", src) + } else if self.locations.insert(entity.into(), loc.clone()).is_some() { + err!(loc, "duplicate entity: {}", entity) } else { Ok(()) } @@ -149,6 +165,8 @@ impl MutableSourceMap for SourceMap { fn def_ebb(&mut self, src: Ebb, entity: Ebb, loc: &Location) -> Result<()> { if self.ebbs.insert(src, entity).is_some() { err!(loc, "duplicate EBB: {}", src) + } else if self.locations.insert(entity.into(), loc.clone()).is_some() { + err!(loc, "duplicate entity: {}", entity) } else { Ok(()) } @@ -157,6 +175,8 @@ impl MutableSourceMap for SourceMap { fn def_ss(&mut self, src_num: u32, entity: StackSlot, loc: &Location) -> Result<()> { if self.stack_slots.insert(src_num, entity).is_some() { err!(loc, "duplicate stack slot: ss{}", src_num) + } else if self.locations.insert(entity.into(), loc.clone()).is_some() { + err!(loc, "duplicate entity: {}", entity) } else { Ok(()) } @@ -165,6 +185,16 @@ impl MutableSourceMap for SourceMap { fn def_jt(&mut self, src_num: u32, entity: JumpTable, loc: &Location) -> Result<()> { if self.jump_tables.insert(src_num, entity).is_some() { err!(loc, "duplicate jump table: jt{}", src_num) + } else if self.locations.insert(entity.into(), loc.clone()).is_some() { + err!(loc, "duplicate entity: {}", entity) + } else { + Ok(()) + } + } + + fn def_inst(&mut self, entity: Inst, loc: &Location) -> Result<()> { + if self.locations.insert(entity.into(), loc.clone()).is_some() { + err!(loc, "duplicate entity: {}", entity) } else { Ok(()) } From 047bf5ab28c934869f3fdf1cf2808e342c56c799 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 19 Sep 2016 11:42:56 -0700 Subject: [PATCH 0308/3084] Remove the inst_locs vector in the parser. Use the source map to track instruction locations instead. The rewrite methods now take an AnyEntity argument as the location to use for errors. This means that bad EBB references in jump tables are now reported correctly. --- cranelift/src/libreader/error.rs | 2 +- cranelift/src/libreader/parser.rs | 98 +++++++++++++--------------- cranelift/src/libreader/sourcemap.rs | 18 +++-- 3 files changed, 60 insertions(+), 58 deletions(-) diff --git a/cranelift/src/libreader/error.rs b/cranelift/src/libreader/error.rs index edc6e40649..cf8bd12f08 100644 --- a/cranelift/src/libreader/error.rs +++ b/cranelift/src/libreader/error.rs @@ -6,7 +6,7 @@ use std::fmt; use std::result; /// The location of a `Token` or `Error`. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] pub struct Location { pub line_number: usize, } diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index b2c9f40d6f..d76392d0f2 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -8,8 +8,8 @@ use std::str::FromStr; use std::u32; use std::mem; -use cretonne::ir::{Function, Ebb, Inst, Opcode, Value, Type, FunctionName, StackSlotData, - JumpTable, JumpTableData}; +use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotData, JumpTable, + JumpTableData}; use cretonne::ir::types::{VOID, Signature, ArgumentType, ArgumentExtension}; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::{AnyEntity, NO_EBB, NO_INST, NO_VALUE}; @@ -65,9 +65,6 @@ pub struct Parser<'a> { struct Context { function: Function, map: SourceMap, - - // Remember the location of every instruction. - inst_locs: Vec<(Inst, Location)>, } impl Context { @@ -75,7 +72,6 @@ impl Context { Context { function: f, map: SourceMap::new(), - inst_locs: Vec::new(), } } @@ -104,70 +100,68 @@ impl Context { self.map.def_ebb(src_ebb, ebb, loc).and(Ok(ebb)) } - // Record the location of an instuction. - fn add_inst_loc(&mut self, inst: Inst, loc: &Location) { - self.inst_locs.push((inst, *loc)); - } - // The parser creates all instructions with Ebb and Value references using the source file // numbering. These references need to be rewritten after parsing is complete since forward // references are allowed. fn rewrite_references(&mut self) -> Result<()> { - for &(inst, loc) in &self.inst_locs { - match self.function.dfg[inst] { - InstructionData::Nullary { .. } | - InstructionData::UnaryImm { .. } | - InstructionData::UnaryIeee32 { .. } | - InstructionData::UnaryIeee64 { .. } | - InstructionData::UnaryImmVector { .. } => {} + for ebb in self.function.layout.ebbs() { + for inst in self.function.layout.ebb_insts(ebb) { + let loc = inst.into(); + match self.function.dfg[inst] { + InstructionData::Nullary { .. } | + InstructionData::UnaryImm { .. } | + InstructionData::UnaryIeee32 { .. } | + InstructionData::UnaryIeee64 { .. } | + InstructionData::UnaryImmVector { .. } => {} - InstructionData::Unary { ref mut arg, .. } | - InstructionData::BinaryImm { ref mut arg, .. } | - InstructionData::BinaryImmRev { ref mut arg, .. } | - InstructionData::ExtractLane { ref mut arg, .. } | - InstructionData::BranchTable { ref mut arg, .. } => { - try!(self.map.rewrite_value(arg, &loc)); - } + InstructionData::Unary { ref mut arg, .. } | + InstructionData::BinaryImm { ref mut arg, .. } | + InstructionData::BinaryImmRev { ref mut arg, .. } | + InstructionData::ExtractLane { ref mut arg, .. } | + InstructionData::BranchTable { ref mut arg, .. } => { + try!(self.map.rewrite_value(arg, loc)); + } - InstructionData::Binary { ref mut args, .. } | - InstructionData::BinaryOverflow { ref mut args, .. } | - InstructionData::InsertLane { ref mut args, .. } | - InstructionData::IntCompare { ref mut args, .. } | - InstructionData::FloatCompare { ref mut args, .. } => { - try!(self.map.rewrite_values(args, &loc)); - } + InstructionData::Binary { ref mut args, .. } | + InstructionData::BinaryOverflow { ref mut args, .. } | + InstructionData::InsertLane { ref mut args, .. } | + InstructionData::IntCompare { ref mut args, .. } | + InstructionData::FloatCompare { ref mut args, .. } => { + try!(self.map.rewrite_values(args, loc)); + } - InstructionData::Ternary { ref mut args, .. } => { - try!(self.map.rewrite_values(args, &loc)); - } + InstructionData::Ternary { ref mut args, .. } => { + try!(self.map.rewrite_values(args, loc)); + } - InstructionData::Jump { ref mut data, .. } => { - try!(self.map.rewrite_ebb(&mut data.destination, &loc)); - try!(self.map.rewrite_values(&mut data.arguments, &loc)); - } + InstructionData::Jump { ref mut data, .. } => { + try!(self.map.rewrite_ebb(&mut data.destination, loc)); + try!(self.map.rewrite_values(&mut data.arguments, loc)); + } - InstructionData::Branch { ref mut data, .. } => { - try!(self.map.rewrite_value(&mut data.arg, &loc)); - try!(self.map.rewrite_ebb(&mut data.destination, &loc)); - try!(self.map.rewrite_values(&mut data.arguments, &loc)); - } + InstructionData::Branch { ref mut data, .. } => { + try!(self.map.rewrite_value(&mut data.arg, loc)); + try!(self.map.rewrite_ebb(&mut data.destination, loc)); + try!(self.map.rewrite_values(&mut data.arguments, loc)); + } - InstructionData::Call { ref mut data, .. } => { - try!(self.map.rewrite_values(&mut data.args, &loc)); - } + InstructionData::Call { ref mut data, .. } => { + try!(self.map.rewrite_values(&mut data.args, loc)); + } - InstructionData::Return { ref mut data, .. } => { - try!(self.map.rewrite_values(&mut data.args, &loc)); + InstructionData::Return { ref mut data, .. } => { + try!(self.map.rewrite_values(&mut data.args, loc)); + } } } } // Rewrite EBB references in jump tables. - let loc = Location { line_number: 0 }; for jt in self.function.jump_tables.keys() { + let loc = jt.into(); for ebb in self.function.jump_tables[jt].as_mut_slice() { if *ebb != NO_EBB { - try!(self.map.rewrite_ebb(ebb, &loc)); + try!(self.map.rewrite_ebb(ebb, loc)); } } } @@ -777,7 +771,7 @@ impl<'a> Parser<'a> { let inst = ctx.function.dfg.make_inst(inst_data); let num_results = ctx.function.dfg.make_inst_results(inst, ctrl_typevar); ctx.function.layout.append_inst(inst, ebb); - ctx.add_inst_loc(inst, &opcode_loc); + ctx.map.def_inst(inst, &opcode_loc).expect("duplicate inst references created"); if results.len() != num_results { return err!(self.loc, diff --git a/cranelift/src/libreader/sourcemap.rs b/cranelift/src/libreader/sourcemap.rs index e7828a1803..82f5ece750 100644 --- a/cranelift/src/libreader/sourcemap.rs +++ b/cranelift/src/libreader/sourcemap.rs @@ -76,29 +76,37 @@ impl SourceMap { } /// Rewrite an Ebb reference. - pub fn rewrite_ebb(&self, ebb: &mut Ebb, loc: &Location) -> Result<()> { + pub fn rewrite_ebb(&self, ebb: &mut Ebb, loc: AnyEntity) -> Result<()> { match self.get_ebb(*ebb) { Some(new) => { *ebb = new; Ok(()) } - None => err!(loc, "undefined reference: {}", ebb), + None => { + err!(self.location(loc).unwrap_or_default(), + "undefined reference: {}", + ebb) + } } } /// Rewrite a value reference. - pub fn rewrite_value(&self, val: &mut Value, loc: &Location) -> Result<()> { + pub fn rewrite_value(&self, val: &mut Value, loc: AnyEntity) -> Result<()> { match self.get_value(*val) { Some(new) => { *val = new; Ok(()) } - None => err!(loc, "undefined reference: {}", val), + None => { + err!(self.location(loc).unwrap_or_default(), + "undefined reference: {}", + val) + } } } /// Rewrite a slice of value references. - pub fn rewrite_values(&self, vals: &mut [Value], loc: &Location) -> Result<()> { + pub fn rewrite_values(&self, vals: &mut [Value], loc: AnyEntity) -> Result<()> { for val in vals { try!(self.rewrite_value(val, loc)); } From b5ffd3af9253f6b1e279954f74076474144b9dee Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 19 Sep 2016 13:00:16 -0700 Subject: [PATCH 0309/3084] Rename 'encoding' modules to 'enc_tables'. These modules contain encoding tables, not the 'Encoding' struct. --- .../isa/{encoding.rs => enc_tables.rs} | 0 cranelift/src/libcretonne/isa/mod.rs | 2 +- .../isa/riscv/{encoding.rs => enc_tables.rs} | 2 +- cranelift/src/libcretonne/isa/riscv/mod.rs | 23 +++++++++---------- 4 files changed, 13 insertions(+), 14 deletions(-) rename cranelift/src/libcretonne/isa/{encoding.rs => enc_tables.rs} (100%) rename cranelift/src/libcretonne/isa/riscv/{encoding.rs => enc_tables.rs} (86%) diff --git a/cranelift/src/libcretonne/isa/encoding.rs b/cranelift/src/libcretonne/isa/enc_tables.rs similarity index 100% rename from cranelift/src/libcretonne/isa/encoding.rs rename to cranelift/src/libcretonne/isa/enc_tables.rs diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index 6eaae9fc62..74e1590fea 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -41,7 +41,7 @@ //! concurrent function compilations. pub mod riscv; -mod encoding; +mod enc_tables; use settings; use ir::{InstructionData, DataFlowGraph}; diff --git a/cranelift/src/libcretonne/isa/riscv/encoding.rs b/cranelift/src/libcretonne/isa/riscv/enc_tables.rs similarity index 86% rename from cranelift/src/libcretonne/isa/riscv/encoding.rs rename to cranelift/src/libcretonne/isa/riscv/enc_tables.rs index 760f4786d7..f32a313c92 100644 --- a/cranelift/src/libcretonne/isa/riscv/encoding.rs +++ b/cranelift/src/libcretonne/isa/riscv/enc_tables.rs @@ -4,7 +4,7 @@ use ir::{Opcode, InstructionData}; use ir::instructions::InstructionFormat; use ir::types; use predicates; -use isa::encoding::{Level1Entry, Level2Entry}; +use isa::enc_tables::{Level1Entry, Level2Entry}; // Include the generated encoding tables: // - `LEVEL1_RV32` diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index a4c1ce9885..0f8ec7c71b 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -1,19 +1,19 @@ //! RISC-V Instruction Set Architecture. pub mod settings; -mod encoding; +mod enc_tables; use super::super::settings as shared_settings; -use isa::encoding as shared_encoding; -use super::Builder as IsaBuilder; -use super::{TargetIsa, Encoding}; +use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; +use isa::Builder as IsaBuilder; +use isa::{TargetIsa, Encoding}; use ir::{InstructionData, DataFlowGraph}; #[allow(dead_code)] struct Isa { shared_flags: shared_settings::Flags, isa_flags: settings::Flags, - cpumode: &'static [shared_encoding::Level1Entry], + cpumode: &'static [shared_enc_tables::Level1Entry], } pub fn isa_builder() -> IsaBuilder { @@ -27,9 +27,9 @@ fn isa_constructor(shared_flags: shared_settings::Flags, builder: shared_settings::Builder) -> Box { let level1 = if shared_flags.is_64bit() { - &encoding::LEVEL1_RV64[..] + &enc_tables::LEVEL1_RV64[..] } else { - &encoding::LEVEL1_RV32[..] + &enc_tables::LEVEL1_RV32[..] }; Box::new(Isa { isa_flags: settings::Flags::new(&shared_flags, builder), @@ -40,21 +40,20 @@ fn isa_constructor(shared_flags: shared_settings::Flags, impl TargetIsa for Isa { fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Option { - use isa::encoding::{lookup_enclist, general_encoding}; lookup_enclist(inst.first_type(), inst.opcode(), self.cpumode, - &encoding::LEVEL2[..]) + &enc_tables::LEVEL2[..]) .and_then(|enclist_offset| { general_encoding(enclist_offset, - &encoding::ENCLISTS[..], - |instp| encoding::check_instp(inst, instp), + &enc_tables::ENCLISTS[..], + |instp| enc_tables::check_instp(inst, instp), |isap| self.isa_flags.numbered_predicate(isap as usize)) }) } fn recipe_names(&self) -> &'static [&'static str] { - &encoding::RECIPE_NAMES[..] + &enc_tables::RECIPE_NAMES[..] } } From fb1bc5d09683a447dd26e4b3ad12db9a9a7a6844 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 19 Sep 2016 13:07:58 -0700 Subject: [PATCH 0310/3084] Move 'Encoding' into its own module. --- cranelift/src/libcretonne/isa/encoding.rs | 33 +++++++++++++++++++ cranelift/src/libcretonne/isa/mod.rs | 40 +++-------------------- 2 files changed, 38 insertions(+), 35 deletions(-) create mode 100644 cranelift/src/libcretonne/isa/encoding.rs diff --git a/cranelift/src/libcretonne/isa/encoding.rs b/cranelift/src/libcretonne/isa/encoding.rs new file mode 100644 index 0000000000..1b26e2bab9 --- /dev/null +++ b/cranelift/src/libcretonne/isa/encoding.rs @@ -0,0 +1,33 @@ +//! The `Encoding` struct. + +/// Bits needed to encode an instruction as binary machine code. +/// +/// The encoding consists of two parts, both specific to the target ISA: An encoding *recipe*, and +/// encoding *bits*. The recipe determines the native instruction format and the mapping of +/// operands to encoded bits. The encoding bits provide additional information to the recipe, +/// typically parts of the opcode. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Encoding { + recipe: u16, + bits: u16, +} + +impl Encoding { + /// Create a new `Encoding` containing `(recipe, bits)`. + pub fn new(recipe: u16, bits: u16) -> Encoding { + Encoding { + recipe: recipe, + bits: bits, + } + } + + /// Get the recipe number in this encoding. + pub fn recipe(self) -> usize { + self.recipe as usize + } + + /// Get the recipe-specific encoding bits. + pub fn bits(self) -> u16 { + self.bits + } +} diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index 74e1590fea..445893ee15 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -40,12 +40,14 @@ //! The configured target ISA trait object is a `Box` which can be used for multiple //! concurrent function compilations. -pub mod riscv; -mod enc_tables; - +pub use isa::encoding::Encoding; use settings; use ir::{InstructionData, DataFlowGraph}; +pub mod riscv; +mod encoding; +mod enc_tables; + /// Look for a supported ISA with the given `name`. /// Return a builder that can create a corresponding `TargetIsa`. pub fn lookup(name: &str) -> Option { @@ -100,35 +102,3 @@ pub trait TargetIsa { /// This is just used for printing and parsing encodings in the textual IL format. fn recipe_names(&self) -> &'static [&'static str]; } - -/// Bits needed to encode an instruction as binary machine code. -/// -/// The encoding consists of two parts, both specific to the target ISA: An encoding *recipe*, and -/// encoding *bits*. The recipe determines the native instruction format and the mapping of -/// operands to encoded bits. The encoding bits provide additional information to the recipe, -/// typically parts of the opcode. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub struct Encoding { - recipe: u16, - bits: u16, -} - -impl Encoding { - /// Create a new `Encoding` containing `(recipe, bits)`. - pub fn new(recipe: u16, bits: u16) -> Encoding { - Encoding { - recipe: recipe, - bits: bits, - } - } - - /// Get the recipe number in this encoding. - pub fn recipe(self) -> usize { - self.recipe as usize - } - - /// Get the recipe-specific encoding bits. - pub fn bits(self) -> u16 { - self.bits - } -} From 6fb566fa85cf625654746b40368f96c9ba7a3f8e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 19 Sep 2016 13:45:22 -0700 Subject: [PATCH 0311/3084] Add a TargetIsa::display_enc() method. The interpretation of an encoding depends on the target ISA, and so does the proper printing of the encoding. --- cranelift/src/libcretonne/isa/encoding.rs | 45 ++++++++++++++++++++++ cranelift/src/libcretonne/isa/mod.rs | 8 ++++ cranelift/src/libcretonne/isa/riscv/mod.rs | 2 +- 3 files changed, 54 insertions(+), 1 deletion(-) diff --git a/cranelift/src/libcretonne/isa/encoding.rs b/cranelift/src/libcretonne/isa/encoding.rs index 1b26e2bab9..0bb1f6cc62 100644 --- a/cranelift/src/libcretonne/isa/encoding.rs +++ b/cranelift/src/libcretonne/isa/encoding.rs @@ -1,5 +1,7 @@ //! The `Encoding` struct. +use std::fmt; + /// Bits needed to encode an instruction as binary machine code. /// /// The encoding consists of two parts, both specific to the target ISA: An encoding *recipe*, and @@ -30,4 +32,47 @@ impl Encoding { pub fn bits(self) -> u16 { self.bits } + + /// Is this a legal encoding, or the default placeholder? + pub fn is_legal(self) -> bool { + self != Self::default() + } +} + +/// The default encoding is the illegal one. +impl Default for Encoding { + fn default() -> Self { + Self::new(0xffff, 0xffff) + } +} + +/// ISA-independent display of an encoding. +impl fmt::Display for Encoding { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + if self.is_legal() { + write!(f, "#{}/{:02x}", self.recipe, self.bits) + } else { + write!(f, "-") + } + } +} + +/// Temporary object that holds enough context to properly display an encoding. +/// This is meant to be created by `TargetIsa::display_enc()`. +pub struct DisplayEncoding { + pub encoding: Encoding, + pub recipe_names: &'static [&'static str], +} + +impl fmt::Display for DisplayEncoding { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + if self.encoding.is_legal() { + write!(f, + "{}/{:02x}", + self.recipe_names[self.encoding.recipe()], + self.encoding.bits) + } else { + write!(f, "-") + } + } } diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index 445893ee15..1d788ffff1 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -101,4 +101,12 @@ pub trait TargetIsa { /// /// This is just used for printing and parsing encodings in the textual IL format. fn recipe_names(&self) -> &'static [&'static str]; + + /// Create an object that can display an ISA-dependent encoding properly. + fn display_enc(&self, enc: Encoding) -> encoding::DisplayEncoding { + encoding::DisplayEncoding { + encoding: enc, + recipe_names: self.recipe_names(), + } + } } diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index 0f8ec7c71b..b1ee92a4a9 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -65,7 +65,7 @@ mod tests { use ir::{types, immediates}; fn encstr(isa: &isa::TargetIsa, enc: isa::Encoding) -> String { - format!("{}/{:02x}", isa.recipe_names()[enc.recipe()], enc.bits()) + isa.display_enc(enc).to_string() } #[test] From 388154e06babf3ea3f440aacbe0ebb5bb96594f8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 19 Sep 2016 14:52:48 -0700 Subject: [PATCH 0312/3084] Wrap FunctionName in a newtype struct. Function names display differently than normal strings since they need quotes and escaping. Move the FunctionName type into its own module. --- cranelift/filetests/parser/rewrite.cton | 2 +- cranelift/src/libcretonne/ir/funcname.rs | 77 ++++++++++++++++++++++++ cranelift/src/libcretonne/ir/function.rs | 2 +- cranelift/src/libcretonne/ir/mod.rs | 4 +- cranelift/src/libcretonne/ir/types.rs | 6 -- cranelift/src/libcretonne/write.rs | 47 +-------------- cranelift/src/libreader/parser.rs | 12 ++-- 7 files changed, 91 insertions(+), 59 deletions(-) create mode 100644 cranelift/src/libcretonne/ir/funcname.rs diff --git a/cranelift/filetests/parser/rewrite.cton b/cranelift/filetests/parser/rewrite.cton index 48ad84869c..9f95cb93fa 100644 --- a/cranelift/filetests/parser/rewrite.cton +++ b/cranelift/filetests/parser/rewrite.cton @@ -29,7 +29,7 @@ ebb100(v20: i32): vx200 = iadd v20, v1000 jump ebb100(v1000) } -; sameln: function "use_value"() { +; sameln: function use_value() { ; nextln: ebb0(vx0: i32): ; nextln: v0 = iadd_imm vx0, 5 ; nextln: v1 = iadd vx0, v0 diff --git a/cranelift/src/libcretonne/ir/funcname.rs b/cranelift/src/libcretonne/ir/funcname.rs new file mode 100644 index 0000000000..0f6f106365 --- /dev/null +++ b/cranelift/src/libcretonne/ir/funcname.rs @@ -0,0 +1,77 @@ +//! Function names. +//! +//! The name of a function doesn't have any meaning to Cretonne which compiles functions +//! independently. + +use std::fmt::{self, Write}; +use std::ascii::AsciiExt; + +/// The name of a function can be any UTF-8 string. +/// +/// Function names are mostly a testing and debugging tool. +/// In particular, `.cton` files use function names to identify functions. +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub struct FunctionName(String); + +impl FunctionName { + pub fn new>(s: S) -> FunctionName { + FunctionName(s.into()) + } +} + +fn is_id_start(c: char) -> bool { + c.is_ascii() && (c == '_' || c.is_alphabetic()) +} + +fn is_id_continue(c: char) -> bool { + c.is_ascii() && (c == '_' || c.is_alphanumeric()) +} + +// The function name may need quotes if it doesn't parse as an identifier. +fn needs_quotes(name: &str) -> bool { + let mut iter = name.chars(); + if let Some(ch) = iter.next() { + !is_id_start(ch) || !iter.all(is_id_continue) + } else { + // A blank function name needs quotes. + true + } +} + +impl fmt::Display for FunctionName { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + if needs_quotes(&self.0) { + try!(f.write_char('"')); + for c in self.0.chars().flat_map(char::escape_default) { + try!(f.write_char(c)); + } + f.write_char('"') + } else { + f.write_str(&self.0) + } + } +} + +#[cfg(test)] +mod tests { + use super::{needs_quotes, FunctionName}; + + #[test] + fn quoting() { + assert_eq!(needs_quotes(""), true); + assert_eq!(needs_quotes("x"), false); + assert_eq!(needs_quotes(" "), true); + assert_eq!(needs_quotes("0"), true); + assert_eq!(needs_quotes("x0"), false); + } + + #[test] + fn escaping() { + assert_eq!(FunctionName::new("").to_string(), "\"\""); + assert_eq!(FunctionName::new("x").to_string(), "x"); + assert_eq!(FunctionName::new(" ").to_string(), "\" \""); + assert_eq!(FunctionName::new(" \n").to_string(), "\" \\n\""); + assert_eq!(FunctionName::new("a\u{1000}v").to_string(), + "\"a\\u{1000}v\""); + } +} diff --git a/cranelift/src/libcretonne/ir/function.rs b/cranelift/src/libcretonne/ir/function.rs index 415e8de9cc..ac6c962861 100644 --- a/cranelift/src/libcretonne/ir/function.rs +++ b/cranelift/src/libcretonne/ir/function.rs @@ -52,7 +52,7 @@ impl Function { /// Create a new empty, anomymous function. pub fn new() -> Function { - Self::with_name_signature(FunctionName::new(), Signature::new()) + Self::with_name_signature(FunctionName::default(), Signature::new()) } /// Get the signature of this function. diff --git a/cranelift/src/libcretonne/ir/mod.rs b/cranelift/src/libcretonne/ir/mod.rs index a26e48f15e..ae16eae735 100644 --- a/cranelift/src/libcretonne/ir/mod.rs +++ b/cranelift/src/libcretonne/ir/mod.rs @@ -10,8 +10,10 @@ pub mod jumptable; pub mod dfg; pub mod layout; pub mod function; +mod funcname; -pub use ir::types::{Type, FunctionName, Signature}; +pub use ir::funcname::FunctionName; +pub use ir::types::{Type, Signature}; pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable}; pub use ir::instructions::{Opcode, InstructionData}; pub use ir::stackslot::StackSlotData; diff --git a/cranelift/src/libcretonne/ir/types.rs b/cranelift/src/libcretonne/ir/types.rs index 12dfc8c6ec..f4e0a6b75f 100644 --- a/cranelift/src/libcretonne/ir/types.rs +++ b/cranelift/src/libcretonne/ir/types.rs @@ -191,12 +191,6 @@ impl Default for Type { // // ====--------------------------------------------------------------------------------------====// -/// The name of a function can be any UTF-8 string. -/// -/// Function names are mostly a testing and debugging tool. In partucular, `.cton` files use -/// function names to identify functions. -pub type FunctionName = String; - /// Function argument extension options. /// /// On some architectures, small integer function arguments are extended to the width of a diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index aa5076f5d5..c17025d1a8 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -29,30 +29,8 @@ pub fn write_function(w: &mut Write, func: &Function) -> Result { // // ====--------------------------------------------------------------------------------------====// -// The function name may need quotes if it doesn't parse as an identifier. -fn needs_quotes(name: &str) -> bool { - let mut iter = name.chars(); - if let Some(ch) = iter.next() { - !ch.is_alphabetic() || !iter.all(char::is_alphanumeric) - } else { - // A blank function name needs quotes. - true - } -} - -// Use Rust's escape_default which provides a few simple \t \r \n \' \" \\ escapes and uses -// \u{xxxx} for anything else outside the ASCII printable range. -fn escaped(name: &str) -> String { - name.chars().flat_map(char::escape_default).collect() -} - fn write_spec(w: &mut Write, func: &Function) -> Result { - let sig = func.own_signature(); - if !needs_quotes(&func.name) { - write!(w, "function {}{}", func.name, sig) - } else { - write!(w, "function \"{}\"{}", escaped(&func.name), sig) - } + write!(w, "function {}{}", func.name, func.own_signature()) } fn write_preamble(w: &mut Write, func: &Function) -> result::Result { @@ -208,34 +186,15 @@ pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { #[cfg(test)] mod tests { - use super::{needs_quotes, escaped}; - use ir::{Function, StackSlotData}; + use ir::{Function, FunctionName, StackSlotData}; use ir::types; - #[test] - fn quoting() { - assert_eq!(needs_quotes(""), true); - assert_eq!(needs_quotes("x"), false); - assert_eq!(needs_quotes(" "), true); - assert_eq!(needs_quotes("0"), true); - assert_eq!(needs_quotes("x0"), false); - } - - #[test] - fn escaping() { - assert_eq!(escaped(""), ""); - assert_eq!(escaped("x"), "x"); - assert_eq!(escaped(" "), " "); - assert_eq!(escaped(" \n"), " \\n"); - assert_eq!(escaped("a\u{1000}v"), "a\\u{1000}v"); - } - #[test] fn basic() { let mut f = Function::new(); assert_eq!(f.to_string(), "function \"\"() {\n}\n"); - f.name.push_str("foo"); + f.name = FunctionName::new("foo".to_string()); assert_eq!(f.to_string(), "function foo() {\n}\n"); f.stack_slots.push(StackSlotData::new(4)); diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index d76392d0f2..45177aace7 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -470,11 +470,11 @@ impl<'a> Parser<'a> { // // function ::= "function" * name signature { ... } // - fn parse_function_name(&mut self) -> Result { + fn parse_function_name(&mut self) -> Result { match self.token() { Some(Token::Identifier(s)) => { self.consume(); - Ok(s.to_string()) + Ok(FunctionName::new(s)) } _ => err!(self.loc, "expected function name"), } @@ -1161,7 +1161,7 @@ mod tests { }") .parse_function() .unwrap(); - assert_eq!(func.name, "foo"); + assert_eq!(func.name.to_string(), "foo"); let mut iter = func.stack_slots.keys(); let ss0 = iter.next().unwrap(); assert_eq!(ss0.to_string(), "ss0"); @@ -1190,7 +1190,7 @@ mod tests { }") .parse_function() .unwrap(); - assert_eq!(func.name, "ebbs"); + assert_eq!(func.name.to_string(), "ebbs"); let mut ebbs = func.layout.ebbs(); @@ -1219,7 +1219,7 @@ mod tests { ; More trailing.") .parse_function() .unwrap(); - assert_eq!(&func.name, "comment"); + assert_eq!(func.name.to_string(), "comment"); assert_eq!(comments.len(), 8); // no 'before' comment. assert_eq!(comments[0], Comment { @@ -1252,6 +1252,6 @@ mod tests { assert_eq!(tf.commands[0].command, "cfg"); assert_eq!(tf.commands[1].command, "verify"); assert_eq!(tf.functions.len(), 1); - assert_eq!(tf.functions[0].0.name, "comment"); + assert_eq!(tf.functions[0].0.name.to_string(), "comment"); } } From 7417f884b3a90dd3dc8e9ad6283fd0f9306f9a00 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Sep 2016 09:48:00 -0700 Subject: [PATCH 0313/3084] Add clear, is_empty, and resize methods to EntityMap. These are simply forwards from the underlying Vec. --- cranelift/src/libcretonne/entity_map.rs | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/cranelift/src/libcretonne/entity_map.rs b/cranelift/src/libcretonne/entity_map.rs index 2a76e3cce9..74c3cf5c46 100644 --- a/cranelift/src/libcretonne/entity_map.rs +++ b/cranelift/src/libcretonne/entity_map.rs @@ -70,6 +70,16 @@ impl EntityMap k.index() < self.elems.len() } + /// Is this map completely empty? + pub fn is_empty(&self) -> bool { + self.elems.is_empty() + } + + /// Remove all entries from this map. + pub fn clear(&mut self) { + self.elems.clear() + } + /// Iterate over all the keys in this map. pub fn keys(&self) -> Keys { Keys { @@ -137,12 +147,17 @@ impl EntityMap map } + /// Resize the map to have `n` entries by adding default entries as needed. + pub fn resize(&mut self, n: usize) { + self.elems.resize(n, V::default()); + } + /// Ensure that `k` is a valid key but adding default entries if necesssary. /// /// Return a mutable reference to the corresponding entry. pub fn ensure(&mut self, k: K) -> &mut V { if !self.is_valid(k) { - self.elems.resize(k.index() + 1, V::default()) + self.resize(k.index() + 1) } &mut self.elems[k.index()] } From ebadbdbe7e8f3f35327a9cd26c16473d1f5bd7a0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Sep 2016 10:17:16 -0700 Subject: [PATCH 0314/3084] Store instruction encodings in Function. This is a side-table of ISA-dependent information that will initially be filled out by the legalizer. --- cranelift/src/libcretonne/ir/function.rs | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/cranelift/src/libcretonne/ir/function.rs b/cranelift/src/libcretonne/ir/function.rs index ac6c962861..3ff5aea00f 100644 --- a/cranelift/src/libcretonne/ir/function.rs +++ b/cranelift/src/libcretonne/ir/function.rs @@ -3,10 +3,11 @@ //! The `Function` struct defined in this module owns all of its extended basic blocks and //! instructions. -use ir::{FunctionName, Signature, StackSlot, StackSlotData, JumpTable, JumpTableData, - DataFlowGraph, Layout}; -use entity_map::{EntityMap, PrimaryEntityData}; use std::fmt::{self, Display, Debug, Formatter}; +use ir::{FunctionName, Signature, Inst, StackSlot, StackSlotData, JumpTable, JumpTableData, + DataFlowGraph, Layout}; +use isa::Encoding; +use entity_map::{EntityMap, PrimaryEntityData}; use write::write_function; /// A function. @@ -32,6 +33,10 @@ pub struct Function { /// Layout of EBBs and instructions in the function body. pub layout: Layout, + + /// Encoding recipe and bits for the legal instructions. + /// Illegal instructions have the `Encoding::default()` value. + pub encodings: EntityMap, } impl PrimaryEntityData for StackSlotData {} @@ -47,6 +52,7 @@ impl Function { jump_tables: EntityMap::new(), dfg: DataFlowGraph::new(), layout: Layout::new(), + encodings: EntityMap::new(), } } From de35e715fa83097a4d9b5cd3fce606cbd4a08e09 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Sep 2016 10:49:30 -0700 Subject: [PATCH 0315/3084] Add a stub implementation of the legalizer. This basic version only fills in the encoding of already-legal instructions. It doesn't have any way of transforming the illegal instructions yet. --- cranelift/src/libcretonne/legalizer.rs | 54 ++++++++++++++++++++++++++ cranelift/src/libcretonne/lib.rs | 1 + 2 files changed, 55 insertions(+) create mode 100644 cranelift/src/libcretonne/legalizer.rs diff --git a/cranelift/src/libcretonne/legalizer.rs b/cranelift/src/libcretonne/legalizer.rs new file mode 100644 index 0000000000..3358f77bb0 --- /dev/null +++ b/cranelift/src/libcretonne/legalizer.rs @@ -0,0 +1,54 @@ +//! Legalize instructions. +//! +//! A legal instruction is one that can be mapped directly to a machine code instruction for the +//! target ISA. The `legalize_function()` function takes as input any function and transforms it +//! into an equivalent function using only legal instructions. +//! +//! The characteristics of legal instructions depend on the target ISA, so any given instruction +//! can be legal for one ISA and illegal for another. +//! +//! Besides transforming instructions, the legalizer also fills out the `function.encodings` map +//! which provides a legal encoding recipe for every instruction. +//! +//! The legalizer does not deal with register allocation constraints. These constraints are derived +//! from the encoding recipes, and solved later by the register allocator. + +use ir::Function; +use isa::TargetIsa; + +/// Legalize `func` for `isa`. +/// +/// - Transform any instructions that don't have a legal representation in `isa`. +/// - Fill out `func.encodings`. +/// +pub fn legalize_function(func: &mut Function, isa: &TargetIsa) { + // TODO: This is very simplified and incomplete. + func.encodings.resize(func.dfg.num_insts()); + for ebb in func.layout.ebbs() { + for inst in func.layout.ebb_insts(ebb) { + match isa.encode(&func.dfg, &func.dfg[inst]) { + Some(encoding) => func.encodings[inst] = encoding, + None => { + // TODO: We should transform the instruction into legal equivalents. + // Possible strategies are: + // 1. Expand instruction into sequence of legal instructions. Possibly + // iteratively. + // 2. Split the controlling type variable into high and low parts. This applies + // both to SIMD vector types which can be halved and to integer types such + // as `i64` used on a 32-bit ISA. + // 3. Promote the controlling type variable to a larger type. This typically + // means expressing `i8` and `i16` arithmetic in terms if `i32` operations + // on RISC targets. (It may or may not be beneficial to promote small vector + // types versus splitting them.) + // 4. Convert to library calls. For example, floating point operations on an + // ISA with no IEEE 754 support. + // + // The iteration scheme used here is not going to cut it. Transforming + // instructions involves changing `function.layout` which is impossiblr while + // it is referenced by the two iterators. We need a layout cursor that can + // maintain a position *and* permit inserting and replacing instructions. + } + } + } + } +} diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index de6cf109b5..7285f95602 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -16,6 +16,7 @@ pub mod dominator_tree; pub mod entity_map; pub mod settings; pub mod verifier; +pub mod legalizer; mod write; mod constant_hash; From 5aa1db2e4f8ab7404cc69e1998ca6130e4a59393 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Sep 2016 12:57:29 -0700 Subject: [PATCH 0316/3084] Don't have keywords in the lexer and parser. Instead of recognizing "function" as a keyword, simply match it as a context-sensitive keyword in the parser outside functions. --- cranelift/filetests/parser/keywords.cton | 5 ++++ cranelift/src/libreader/lexer.rs | 29 ++++++------------------ cranelift/src/libreader/parser.rs | 2 +- 3 files changed, 13 insertions(+), 23 deletions(-) create mode 100644 cranelift/filetests/parser/keywords.cton diff --git a/cranelift/filetests/parser/keywords.cton b/cranelift/filetests/parser/keywords.cton new file mode 100644 index 0000000000..eb15f2624d --- /dev/null +++ b/cranelift/filetests/parser/keywords.cton @@ -0,0 +1,5 @@ +test cat + +; 'function' is not a keyword, and can be used as the name of a function too. +function function() {} +; check: function function() diff --git a/cranelift/src/libreader/lexer.rs b/cranelift/src/libreader/lexer.rs index 7631e306ad..c57b8290cf 100644 --- a/cranelift/src/libreader/lexer.rs +++ b/cranelift/src/libreader/lexer.rs @@ -26,7 +26,6 @@ pub enum Token<'a> { Colon, // ':' Equal, // '=' Arrow, // '->' - Function, // 'function' Float(&'a str), // Floating point immediate Integer(&'a str), // Integer immediate Type(types::Type), // i32, f32, b32x4, ... @@ -235,8 +234,6 @@ impl<'a> Lexer<'a> { // Scan a 'word', which is an identifier-like sequence of characters beginning with '_' or an // alphabetic char, followed by zero or more alphanumeric or '_' characters. - // - // fn scan_word(&mut self) -> Result, LocatedError> { let begin = self.pos; let loc = self.loc(); @@ -252,25 +249,13 @@ impl<'a> Lexer<'a> { } } let text = &self.source[begin..self.pos]; + let (prefix, suffix) = text.split_at(text.len() - trailing_digits); - match if trailing_digits == 0 { - Self::keyword(text) - } else { - // Look for numbered well-known entities like ebb15, v45, ... - let (prefix, suffix) = text.split_at(text.len() - trailing_digits); - Self::numbered_entity(prefix, suffix).or_else(|| Self::value_type(text, prefix, suffix)) - } { - Some(t) => token(t, loc), - None => token(Token::Identifier(text), loc), - } - } - - // Recognize a keyword. - fn keyword(text: &str) -> Option> { - match text { - "function" => Some(Token::Function), - _ => None, - } + // Look for numbered well-known entities like ebb15, v45, ... + token(Self::numbered_entity(prefix, suffix) + .or_else(|| Self::value_type(text, prefix, suffix)) + .unwrap_or(Token::Identifier(text)), + loc) } // If prefix is a well-known entity prefix and suffix is a valid entity number, return the @@ -458,7 +443,7 @@ mod tests { token(Token::Value(Value::table_with_number(1).unwrap()), 1)); assert_eq!(lex.next(), token(Token::Identifier("vxvx4"), 1)); assert_eq!(lex.next(), token(Token::Identifier("function0"), 1)); - assert_eq!(lex.next(), token(Token::Function, 1)); + assert_eq!(lex.next(), token(Token::Identifier("function"), 1)); assert_eq!(lex.next(), token(Token::Type(types::B1), 1)); assert_eq!(lex.next(), token(Token::Type(types::I32.by(4).unwrap()), 1)); assert_eq!(lex.next(), token(Token::Identifier("f32x5"), 1)); diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 45177aace7..6f65cbafa3 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -454,7 +454,7 @@ impl<'a> Parser<'a> { // function-spec ::= * "function" name signature // fn parse_function_spec(&mut self) -> Result<(Location, FunctionName, Signature)> { - try!(self.match_token(Token::Function, "expected 'function' keyword")); + try!(self.match_identifier("function", "expected 'function'")); let location = self.loc; // function-spec ::= "function" * name signature From a078f6a6b0b9a9821d149388284b6ffec9baa3eb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Sep 2016 13:20:33 -0700 Subject: [PATCH 0317/3084] Share split_entity_name between lexer and sourcemap. There's only one way of parsing entity names correctly. --- cranelift/src/libreader/lexer.rs | 91 +++++++++++++++++++--------- cranelift/src/libreader/sourcemap.rs | 43 +------------ 2 files changed, 63 insertions(+), 71 deletions(-) diff --git a/cranelift/src/libreader/lexer.rs b/cranelift/src/libreader/lexer.rs index c57b8290cf..3454225dc8 100644 --- a/cranelift/src/libreader/lexer.rs +++ b/cranelift/src/libreader/lexer.rs @@ -6,6 +6,7 @@ // ====--------------------------------------------------------------------------------------====// use std::str::CharIndices; +use std::u16; use cretonne::ir::types; use cretonne::ir::{Value, Ebb}; use error::Location; @@ -72,6 +73,23 @@ fn error<'a>(error: Error, loc: Location) -> Result, LocatedErr }) } +/// Get the number of decimal digits at the end of `s`. +fn trailing_digits(s: &str) -> usize { + // It's faster to iterate backwards over bytes, and we're only counting ASCII digits. + s.as_bytes().iter().rev().cloned().take_while(|&b| b'0' <= b && b <= b'9').count() +} + +/// Pre-parse a supposed entity name by splitting it into two parts: A head of lowercase ASCII +/// letters and numeric tail. +pub fn split_entity_name(name: &str) -> Option<(&str, u32)> { + let (head, tail) = name.split_at(name.len() - trailing_digits(name)); + if tail.len() > 1 && tail.starts_with('0') { + None + } else { + tail.parse().ok().map(|n| (head, n)) + } +} + /// Lexical analysis. /// /// A `Lexer` reads text from a `&str` and provides a sequence of tokens. @@ -237,52 +255,42 @@ impl<'a> Lexer<'a> { fn scan_word(&mut self) -> Result, LocatedError> { let begin = self.pos; let loc = self.loc(); - let mut trailing_digits = 0usize; assert!(self.lookahead == Some('_') || self.lookahead.unwrap().is_alphabetic()); loop { match self.next_ch() { - Some(ch) if ch.is_digit(10) => trailing_digits += 1, - Some('_') => trailing_digits = 0, - Some(ch) if ch.is_alphabetic() => trailing_digits = 0, + Some('_') => {} + Some(ch) if ch.is_alphanumeric() => {} _ => break, } } let text = &self.source[begin..self.pos]; - let (prefix, suffix) = text.split_at(text.len() - trailing_digits); // Look for numbered well-known entities like ebb15, v45, ... - token(Self::numbered_entity(prefix, suffix) - .or_else(|| Self::value_type(text, prefix, suffix)) + token(split_entity_name(text) + .and_then(|(prefix, number)| { + Self::numbered_entity(prefix, number) + .or_else(|| Self::value_type(text, prefix, number)) + }) .unwrap_or(Token::Identifier(text)), loc) } // If prefix is a well-known entity prefix and suffix is a valid entity number, return the // decoded token. - fn numbered_entity(prefix: &str, suffix: &str) -> Option> { - // Reject non-canonical numbers like v0001. - if suffix.len() > 1 && suffix.starts_with('0') { - return None; - } - - let value: u32 = match suffix.parse() { - Ok(v) => v, - _ => return None, - }; - + fn numbered_entity(prefix: &str, number: u32) -> Option> { match prefix { - "v" => Value::direct_with_number(value).map(|v| Token::Value(v)), - "vx" => Value::table_with_number(value).map(|v| Token::Value(v)), - "ebb" => Ebb::with_number(value).map(|ebb| Token::Ebb(ebb)), - "ss" => Some(Token::StackSlot(value)), - "jt" => Some(Token::JumpTable(value)), + "v" => Value::direct_with_number(number).map(|v| Token::Value(v)), + "vx" => Value::table_with_number(number).map(|v| Token::Value(v)), + "ebb" => Ebb::with_number(number).map(|ebb| Token::Ebb(ebb)), + "ss" => Some(Token::StackSlot(number)), + "jt" => Some(Token::JumpTable(number)), _ => None, } } // Recognize a scalar or vector type. - fn value_type(text: &str, prefix: &str, suffix: &str) -> Option> { + fn value_type(text: &str, prefix: &str, number: u32) -> Option> { let is_vector = prefix.ends_with('x'); let scalar = if is_vector { &prefix[0..prefix.len() - 1] @@ -304,11 +312,11 @@ impl<'a> Lexer<'a> { _ => return None, }; if is_vector { - let lanes: u16 = match suffix.parse() { - Ok(v) => v, - _ => return None, - }; - base_type.by(lanes).map(|t| Token::Type(t)) + if number <= u16::MAX as u32 { + base_type.by(number as u16).map(|t| Token::Type(t)) + } else { + None + } } else { Some(Token::Type(base_type)) } @@ -356,11 +364,36 @@ impl<'a> Lexer<'a> { #[cfg(test)] mod tests { + use super::trailing_digits; use super::*; use cretonne::ir::types; use cretonne::ir::{Value, Ebb}; use error::Location; + #[test] + fn digits() { + assert_eq!(trailing_digits(""), 0); + assert_eq!(trailing_digits("x"), 0); + assert_eq!(trailing_digits("0x"), 0); + assert_eq!(trailing_digits("x1"), 1); + assert_eq!(trailing_digits("1x1"), 1); + assert_eq!(trailing_digits("1x01"), 2); + } + + #[test] + fn entity_name() { + assert_eq!(split_entity_name(""), None); + assert_eq!(split_entity_name("x"), None); + assert_eq!(split_entity_name("x+"), None); + assert_eq!(split_entity_name("x+1"), Some(("x+", 1))); + assert_eq!(split_entity_name("x-1"), Some(("x-", 1))); + assert_eq!(split_entity_name("1"), Some(("", 1))); + assert_eq!(split_entity_name("x1"), Some(("x", 1))); + assert_eq!(split_entity_name("xy0"), Some(("xy", 0))); + // Reject this non-canonical form. + assert_eq!(split_entity_name("inst01"), None); + } + fn token<'a>(token: Token<'a>, line: usize) -> Option, LocatedError>> { Some(super::token(token, Location { line_number: line })) } diff --git a/cranelift/src/libreader/sourcemap.rs b/cranelift/src/libreader/sourcemap.rs index 82f5ece750..2bff8e4b81 100644 --- a/cranelift/src/libreader/sourcemap.rs +++ b/cranelift/src/libreader/sourcemap.rs @@ -11,6 +11,7 @@ use std::collections::HashMap; use cretonne::ir::{StackSlot, JumpTable, Ebb, Value, Inst}; use cretonne::ir::entities::AnyEntity; use error::{Result, Location}; +use lexer::split_entity_name; /// Mapping from source entity names to entity references that are valid in the parsed function. #[derive(Debug)] @@ -114,23 +115,6 @@ impl SourceMap { } } -/// Get the number of decimal digits at the end of `s`. -fn trailing_digits(s: &str) -> usize { - // It's faster to iterate backwards over bytes, and we're only counting ASCII digits. - s.as_bytes().iter().rev().cloned().take_while(|&b| b'0' <= b && b <= b'9').count() -} - -/// Pre-parse a supposed entity name by splitting it into two parts: A head of lowercase ASCII -/// letters and numeric tail. -fn split_entity_name(name: &str) -> Option<(&str, u32)> { - let (head, tail) = name.split_at(name.len() - trailing_digits(name)); - if tail.len() > 1 && tail.starts_with('0') { - None - } else { - tail.parse().ok().map(|n| (head, n)) - } -} - /// Interface for mutating a source map. /// @@ -211,33 +195,8 @@ impl MutableSourceMap for SourceMap { #[cfg(test)] mod tests { - use super::{trailing_digits, split_entity_name}; use parse_test; - #[test] - fn digits() { - assert_eq!(trailing_digits(""), 0); - assert_eq!(trailing_digits("x"), 0); - assert_eq!(trailing_digits("0x"), 0); - assert_eq!(trailing_digits("x1"), 1); - assert_eq!(trailing_digits("1x1"), 1); - assert_eq!(trailing_digits("1x01"), 2); - } - - #[test] - fn entity_name() { - assert_eq!(split_entity_name(""), None); - assert_eq!(split_entity_name("x"), None); - assert_eq!(split_entity_name("x+"), None); - assert_eq!(split_entity_name("x+1"), Some(("x+", 1))); - assert_eq!(split_entity_name("x-1"), Some(("x-", 1))); - assert_eq!(split_entity_name("1"), Some(("", 1))); - assert_eq!(split_entity_name("x1"), Some(("x", 1))); - assert_eq!(split_entity_name("xy0"), Some(("xy", 0))); - // Reject this non-canonical form. - assert_eq!(split_entity_name("inst01"), None); - } - #[test] fn details() { let tf = parse_test("function detail() { From 8a764cba3f735d54ba698e023e1084bbbb1d492d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Sep 2016 16:09:45 -0700 Subject: [PATCH 0318/3084] Allow settings::Builder to be reused. When constructing the Flags object from the Builder, don't consume it, but take a reference instead. This makes it possible for the parser to accept multiple 'set' lines and apply them to different ISA specifications. --- cranelift/src/libcretonne/isa/mod.rs | 6 +++--- cranelift/src/libcretonne/isa/riscv/mod.rs | 8 ++++---- cranelift/src/libcretonne/isa/riscv/settings.rs | 12 ++++++------ cranelift/src/libcretonne/settings.rs | 12 ++++++------ meta/gen_settings.py | 8 ++++---- 5 files changed, 23 insertions(+), 23 deletions(-) diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index 1d788ffff1..a43b1a5cb4 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -24,7 +24,7 @@ //! use cretonne::isa; //! //! let shared_builder = settings::builder(); -//! let shared_flags = settings::Flags::new(shared_builder); +//! let shared_flags = settings::Flags::new(&shared_builder); //! //! match isa::lookup("riscv") { //! None => { @@ -66,14 +66,14 @@ fn riscv_builder() -> Option { /// Modify the ISA-specific settings before creating the `TargetIsa` trait object with `finish`. pub struct Builder { setup: settings::Builder, - constructor: fn(settings::Flags, settings::Builder) -> Box, + constructor: fn(settings::Flags, &settings::Builder) -> Box, } impl Builder { /// Combine the ISA-specific settings with the provided ISA-independent settings and allocate a /// fully configured `TargetIsa` trait object. pub fn finish(self, shared_flags: settings::Flags) -> Box { - (self.constructor)(shared_flags, self.setup) + (self.constructor)(shared_flags, &self.setup) } } diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index b1ee92a4a9..ea49358aa6 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -24,7 +24,7 @@ pub fn isa_builder() -> IsaBuilder { } fn isa_constructor(shared_flags: shared_settings::Flags, - builder: shared_settings::Builder) + builder: &shared_settings::Builder) -> Box { let level1 = if shared_flags.is_64bit() { &enc_tables::LEVEL1_RV64[..] @@ -72,7 +72,7 @@ mod tests { fn test_64bitenc() { let mut shared_builder = settings::builder(); shared_builder.set_bool("is_64bit", true).unwrap(); - let shared_flags = settings::Flags::new(shared_builder); + let shared_flags = settings::Flags::new(&shared_builder); let isa = isa::lookup("riscv").unwrap().finish(shared_flags); let mut dfg = DataFlowGraph::new(); @@ -119,7 +119,7 @@ mod tests { fn test_32bitenc() { let mut shared_builder = settings::builder(); shared_builder.set_bool("is_64bit", false).unwrap(); - let shared_flags = settings::Flags::new(shared_builder); + let shared_flags = settings::Flags::new(&shared_builder); let isa = isa::lookup("riscv").unwrap().finish(shared_flags); let mut dfg = DataFlowGraph::new(); @@ -174,7 +174,7 @@ mod tests { fn test_rv32m() { let mut shared_builder = settings::builder(); shared_builder.set_bool("is_64bit", false).unwrap(); - let shared_flags = settings::Flags::new(shared_builder); + let shared_flags = settings::Flags::new(&shared_builder); // Set the supports_m stting which in turn enables the use_m predicate that unlocks // encodings for imul. diff --git a/cranelift/src/libcretonne/isa/riscv/settings.rs b/cranelift/src/libcretonne/isa/riscv/settings.rs index ae765d9c15..57d8689ed1 100644 --- a/cranelift/src/libcretonne/isa/riscv/settings.rs +++ b/cranelift/src/libcretonne/isa/riscv/settings.rs @@ -14,9 +14,9 @@ mod tests { #[test] fn display_default() { - let shared = settings::Flags::new(settings::builder()); + let shared = settings::Flags::new(&settings::builder()); let b = builder(); - let f = Flags::new(&shared, b); + let f = Flags::new(&shared, &b); assert_eq!(f.to_string(), "[riscv]\n\ supports_m = false\n\ @@ -30,20 +30,20 @@ mod tests { #[test] fn predicates() { - let shared = settings::Flags::new(settings::builder()); + let shared = settings::Flags::new(&settings::builder()); let mut b = builder(); b.set_bool("supports_f", true).unwrap(); b.set_bool("supports_d", true).unwrap(); - let f = Flags::new(&shared, b); + let f = Flags::new(&shared, &b); assert_eq!(f.full_float(), true); let mut sb = settings::builder(); sb.set_bool("enable_simd", false).unwrap(); - let shared = settings::Flags::new(sb); + let shared = settings::Flags::new(&sb); let mut b = builder(); b.set_bool("supports_f", true).unwrap(); b.set_bool("supports_d", true).unwrap(); - let f = Flags::new(&shared, b); + let f = Flags::new(&shared, &b); assert_eq!(f.full_float(), false); } } diff --git a/cranelift/src/libcretonne/settings.rs b/cranelift/src/libcretonne/settings.rs index c7c0d1eb30..37d09d2ba8 100644 --- a/cranelift/src/libcretonne/settings.rs +++ b/cranelift/src/libcretonne/settings.rs @@ -16,7 +16,7 @@ //! let mut b = settings::builder(); //! b.set("opt_level", "fastest"); //! -//! let f = settings::Flags::new(b); +//! let f = settings::Flags::new(&b); //! assert_eq!(f.opt_level(), settings::OptLevel::Fastest); //! ``` @@ -57,9 +57,9 @@ impl Builder { } /// Extract contents of builder once everything is configured. - pub fn finish(self, name: &str) -> Vec { + pub fn state_for(&self, name: &str) -> &[u8] { assert_eq!(name, self.template.name); - self.bytes + &self.bytes[..] } /// Set the value of a single bit. @@ -256,7 +256,7 @@ mod tests { #[test] fn display_default() { let b = builder(); - let f = Flags::new(b); + let f = Flags::new(&b); assert_eq!(f.to_string(), "[shared]\n\ opt_level = \"default\"\n\ @@ -275,7 +275,7 @@ mod tests { assert_eq!(b.set_bool("enable_simd", true), Ok(())); assert_eq!(b.set_bool("enable_simd", false), Ok(())); - let f = Flags::new(b); + let f = Flags::new(&b); assert_eq!(f.enable_simd(), false); } @@ -289,7 +289,7 @@ mod tests { assert_eq!(b.set("opt_level", "best"), Ok(())); assert_eq!(b.set("enable_simd", "0"), Ok(())); - let f = Flags::new(b); + let f = Flags::new(&b); assert_eq!(f.enable_simd(), false); assert_eq!(f.opt_level(), super::OptLevel::Best); } diff --git a/meta/gen_settings.py b/meta/gen_settings.py index 588c2cf798..c3c4710ea3 100644 --- a/meta/gen_settings.py +++ b/meta/gen_settings.py @@ -183,18 +183,18 @@ def gen_constructor(sgrp, parent, fmt): """ with fmt.indented('impl Flags {', '}'): - args = 'builder: Builder' + args = 'builder: &Builder' if sgrp.parent: p = sgrp.parent args = '{}: &{}::Flags, {}'.format(p.name, p.qual_mod, args) with fmt.indented( 'pub fn new({}) -> Flags {{'.format(args), '}'): - fmt.line('let bvec = builder.finish("{}");'.format(sgrp.name)) + fmt.line('let bvec = builder.state_for("{}");'.format(sgrp.name)) fmt.line('let mut bytes = [0; {}];'.format(sgrp.byte_size())) fmt.line('assert_eq!(bvec.len(), {});'.format(sgrp.settings_size)) with fmt.indented( - 'for (i, b) in bvec.into_iter().enumerate() {', '}'): - fmt.line('bytes[i] = b;') + 'for (i, b) in bvec.iter().enumerate() {', '}'): + fmt.line('bytes[i] = *b;') # Stop here without predicates. if len(sgrp.predicate_number) == sgrp.boolean_settings: From e4a8932962c177ff96b8cc24b1fd9ec5d47d5a0c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Sep 2016 13:38:36 -0700 Subject: [PATCH 0319/3084] Parse ISA specifications between test commands and functions. Some tests are only applicable to specific ISAs. This can be indicated with an ISA specification: test legalizer isa riscv function foo() { .... } The ISA specifications have the same format as the test lines: The name of the ISA following by optional settings until the end of the line. Also parse `set` commands mixed in with the `isa` commands. These are used to set ISA-independent settings as defined in meta/cretonne/settings.py. --- cranelift/src/libcretonne/isa/mod.rs | 3 + cranelift/src/libcretonne/isa/riscv/mod.rs | 4 + cranelift/src/libreader/isaspec.rs | 49 ++++++++++++ cranelift/src/libreader/lib.rs | 1 + cranelift/src/libreader/parser.rs | 91 ++++++++++++++++++++++ cranelift/src/libreader/testfile.rs | 9 ++- 6 files changed, 154 insertions(+), 3 deletions(-) create mode 100644 cranelift/src/libreader/isaspec.rs diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index a43b1a5cb4..fbde0208e6 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -88,6 +88,9 @@ impl settings::Configurable for Builder { } pub trait TargetIsa { + /// Get the name of this ISA. + fn name(&self) -> &'static str; + /// Encode an instruction after determining it is legal. /// /// If `inst` can legally be encoded in this ISA, produce the corresponding `Encoding` object. diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index ea49358aa6..62bca868a6 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -39,6 +39,10 @@ fn isa_constructor(shared_flags: shared_settings::Flags, } impl TargetIsa for Isa { + fn name(&self) -> &'static str { + "riscv" + } + fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Option { lookup_enclist(inst.first_type(), inst.opcode(), diff --git a/cranelift/src/libreader/isaspec.rs b/cranelift/src/libreader/isaspec.rs new file mode 100644 index 0000000000..706c081e3a --- /dev/null +++ b/cranelift/src/libreader/isaspec.rs @@ -0,0 +1,49 @@ +//! Parsed representation of `set` and `isa` commands. +//! +//! A test case file can contain `set` commands that set ISA-independent settings, and it can +//! contain `isa` commands that select an ISA and applies ISA-specific settings. +//! +//! If a test case file contains `isa` commands, the tests will only be run against the specified +//! ISAs. If the file contains no `isa` commands, the tests will be run against all supported ISAs. + +use cretonne::settings::{Flags, Configurable, Error as SetError}; +use cretonne::isa::TargetIsa; +use error::{Result, Location}; +use testcommand::TestOption; + +/// The ISA specifications in a `.cton` file. +pub enum IsaSpec { + /// The parsed file does not contain any `isa` commands, but it may contain `set` commands + /// which are reflected in the finished `Flags` object. + None(Flags), + + /// The parsed file does contains `isa` commands. + /// Each `isa` command is used to configure a `TargetIsa` trait object. + Some(Vec>), +} + +/// Parse an iterator of command line options and apply them to `config`. +pub fn parse_options<'a, I>(iter: I, config: &mut Configurable, loc: &Location) -> Result<()> + where I: Iterator +{ + for opt in iter.map(TestOption::new) { + match opt { + TestOption::Flag(name) => { + match config.set_bool(name, true) { + Ok(_) => {} + Err(SetError::BadName) => return err!(loc, "unknown flag '{}'", opt), + Err(_) => return err!(loc, "not a boolean flag: '{}'", opt), + } + } + TestOption::Value(name, value) => { + match config.set(name, value) { + Ok(_) => {} + Err(SetError::BadName) => return err!(loc, "unknown setting '{}'", opt), + Err(SetError::BadType) => return err!(loc, "invalid setting type: '{}'", opt), + Err(SetError::BadValue) => return err!(loc, "invalid setting value: '{}'", opt), + } + } + } + } + Ok(()) +} diff --git a/cranelift/src/libreader/lib.rs b/cranelift/src/libreader/lib.rs index f0f04fa381..da3cb35685 100644 --- a/cranelift/src/libreader/lib.rs +++ b/cranelift/src/libreader/lib.rs @@ -15,5 +15,6 @@ mod error; mod lexer; mod parser; mod testcommand; +mod isaspec; mod testfile; mod sourcemap; diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 6f65cbafa3..05fa9b831b 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -15,10 +15,13 @@ use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::{AnyEntity, NO_EBB, NO_INST, NO_VALUE}; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, JumpData, BranchData, ReturnData}; +use cretonne::isa; +use cretonne::settings; use testfile::{TestFile, Details, Comment}; use error::{Location, Error, Result}; use lexer::{self, Lexer, Token}; use testcommand::TestCommand; +use isaspec; use sourcemap::{SourceMap, MutableSourceMap}; /// Parse the entire `text` into a list of functions. @@ -35,6 +38,7 @@ pub fn parse_test<'a>(text: &'a str) -> Result> { let mut parser = Parser::new(text); Ok(TestFile { commands: parser.parse_test_commands(), + isa_spec: try!(parser.parse_isa_specs()), functions: try!(parser.parse_function_list()), }) } @@ -397,6 +401,63 @@ impl<'a> Parser<'a> { list } + /// Parse a list of ISA specs. + /// + /// Accept a mix of `isa` and `set` command lines. The `set` commands are cumulative. + /// + pub fn parse_isa_specs(&mut self) -> Result { + // Was there any `isa` commands? + let mut seen_isa = false; + // Location of last `set` command since the last `isa`. + let mut last_set_loc = None; + + let mut isas = Vec::new(); + let mut flag_builder = settings::builder(); + + while let Some(Token::Identifier(command)) = self.token() { + match command { + "set" => { + last_set_loc = Some(self.loc); + try!(isaspec::parse_options(self.consume_line().trim().split_whitespace(), + &mut flag_builder, + &self.loc)); + } + "isa" => { + last_set_loc = None; + seen_isa = true; + let loc = self.loc; + // Grab the whole line so the lexer won't go looking for tokens on the + // following lines. + let mut words = self.consume_line().trim().split_whitespace(); + // Look for `isa foo`. + let isa_name = match words.next() { + None => return err!(loc, "expected ISA name"), + Some(w) => w, + }; + let mut isa_builder = match isa::lookup(isa_name) { + None => return err!(loc, "unknown ISA '{}'", isa_name), + Some(b) => b, + }; + // Apply the ISA-specific settings to `isa_builder`. + try!(isaspec::parse_options(words, &mut isa_builder, &self.loc)); + + // Construct a trait object with the aggregrate settings. + isas.push(isa_builder.finish(settings::Flags::new(&flag_builder))); + } + _ => break, + } + } + if !seen_isa { + // No `isa` commands, but we allow for `set` commands. + Ok(isaspec::IsaSpec::None(settings::Flags::new(&flag_builder))) + } else if let Some(loc) = last_set_loc { + err!(loc, + "dangling 'set' command after ISA specification has no effect.") + } else { + Ok(isaspec::IsaSpec::Some(isas)) + } + } + /// Parse a list of function definitions. /// /// This is the top-level parse function matching the whole contents of a file. @@ -1115,6 +1176,7 @@ mod tests { use cretonne::ir::types::{self, ArgumentType, ArgumentExtension}; use cretonne::ir::entities::AnyEntity; use testfile::{Details, Comment}; + use isaspec::IsaSpec; use error::Error; #[test] @@ -1246,12 +1308,41 @@ mod tests { let tf = parse_test("; before test cfg option=5 test verify + set enable_float=false function comment() {}") .unwrap(); assert_eq!(tf.commands.len(), 2); assert_eq!(tf.commands[0].command, "cfg"); assert_eq!(tf.commands[1].command, "verify"); + match tf.isa_spec { + IsaSpec::None(s) => assert!(!s.enable_float()), + _ => panic!("unexpected ISAs"), + } assert_eq!(tf.functions.len(), 1); assert_eq!(tf.functions[0].0.name.to_string(), "comment"); } + + #[test] + fn isa_spec() { + assert!(parse_test("isa + function foo() {}") + .is_err()); + + assert!(parse_test("isa riscv + set enable_float=false + function foo() {}") + .is_err()); + + match parse_test("set enable_float=false + isa riscv + function foo() {}") + .unwrap() + .isa_spec { + IsaSpec::None(_) => panic!("Expected some ISA"), + IsaSpec::Some(v) => { + assert_eq!(v.len(), 1); + assert_eq!(v[0].name(), "riscv"); + } + } + } } diff --git a/cranelift/src/libreader/testfile.rs b/cranelift/src/libreader/testfile.rs index d63b1f9f9d..30ed59074b 100644 --- a/cranelift/src/libreader/testfile.rs +++ b/cranelift/src/libreader/testfile.rs @@ -7,16 +7,19 @@ use cretonne::ir::Function; use cretonne::ir::entities::AnyEntity; use testcommand::TestCommand; +use isaspec::IsaSpec; use sourcemap::SourceMap; use error::Location; /// A parsed test case. /// -/// This is the result of parsing a `.cton` file which contains a number of test commands followed -/// by the functions that should be tested. -#[derive(Debug)] +/// This is the result of parsing a `.cton` file which contains a number of test commands and ISA +/// specs followed by the functions that should be tested. pub struct TestFile<'a> { + /// `test foo ...` lines. pub commands: Vec>, + /// `isa bar ...` lines. + pub isa_spec: IsaSpec, pub functions: Vec<(Function, Details<'a>)>, } From 16dba4d35bcecc181ad3b0c401ae30fc9267bff4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 21 Sep 2016 12:12:11 -0700 Subject: [PATCH 0320/3084] Pass flags and target ISAs to filetests. Add a `needs_isa()` method to the SubTest trait, and pass a TargetIsa trait object to those sub-tests that request it. When multiple sub-tests and ISAs are specified, test the cross product. If a sub-test requires an ISA, but none are specified, fail the test. In the future, it may be a good idea to generate a default set of ISAs and test against those. --- cranelift/src/libcretonne/isa/mod.rs | 3 ++ cranelift/src/libcretonne/isa/riscv/mod.rs | 4 ++ cranelift/src/libreader/lib.rs | 1 + cranelift/src/tools/filetest/runone.rs | 61 +++++++++++++++++++--- cranelift/src/tools/filetest/subtest.rs | 14 +++++ 5 files changed, 77 insertions(+), 6 deletions(-) diff --git a/cranelift/src/libcretonne/isa/mod.rs b/cranelift/src/libcretonne/isa/mod.rs index fbde0208e6..67d5a91c29 100644 --- a/cranelift/src/libcretonne/isa/mod.rs +++ b/cranelift/src/libcretonne/isa/mod.rs @@ -91,6 +91,9 @@ pub trait TargetIsa { /// Get the name of this ISA. fn name(&self) -> &'static str; + /// Get the ISA-independent flags that were used to make this trait object. + fn flags(&self) -> &settings::Flags; + /// Encode an instruction after determining it is legal. /// /// If `inst` can legally be encoded in this ISA, produce the corresponding `Encoding` object. diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index 62bca868a6..2ff352918d 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -43,6 +43,10 @@ impl TargetIsa for Isa { "riscv" } + fn flags(&self) -> &shared_settings::Flags { + &self.shared_flags + } + fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Option { lookup_enclist(inst.first_type(), inst.opcode(), diff --git a/cranelift/src/libreader/lib.rs b/cranelift/src/libreader/lib.rs index da3cb35685..ef81e7f210 100644 --- a/cranelift/src/libreader/lib.rs +++ b/cranelift/src/libreader/lib.rs @@ -9,6 +9,7 @@ pub use error::{Location, Result, Error}; pub use parser::{parse_functions, parse_test}; pub use testcommand::{TestCommand, TestOption}; pub use testfile::{TestFile, Details}; +pub use isaspec::IsaSpec; pub use sourcemap::SourceMap; mod error; diff --git a/cranelift/src/tools/filetest/runone.rs b/cranelift/src/tools/filetest/runone.rs index e0d1e9d17b..d0199c45cf 100644 --- a/cranelift/src/tools/filetest/runone.rs +++ b/cranelift/src/tools/filetest/runone.rs @@ -1,11 +1,14 @@ //! Run the tests in a single test file. -use std::borrow::{Borrow, Cow}; +use std::borrow::Cow; use std::path::Path; use std::time; use cretonne::ir::Function; +use cretonne::isa::TargetIsa; +use cretonne::settings::Flags; use cretonne::verify_function; use cton_reader::parse_test; +use cton_reader::IsaSpec; use utils::read_to_string; use filetest::{TestResult, new_subtest}; use filetest::subtest::{SubTest, Context, Result}; @@ -20,16 +23,27 @@ pub fn run(path: &Path) -> TestResult { if testfile.functions.is_empty() { return Err("no functions found".to_string()); } + // Parse the test commands. let mut tests = try!(testfile.commands.iter().map(new_subtest).collect::>>()); + // Flags to use for those tests that don't need an ISA. + // This is the cumulative effect of all the `set` commands in the file. + let flags = match testfile.isa_spec { + IsaSpec::None(ref f) => f, + IsaSpec::Some(ref v) => v.last().expect("Empty ISA list").flags(), + }; + // Sort the tests so the mutators are at the end, and those that don't need the verifier are at // the front. tests.sort_by_key(|st| (st.is_mutating(), st.needs_verifier())); + // Expand the tests into (test, flags, isa) tuples. + let mut tuples = try!(test_tuples(&tests, &testfile.isa_spec, flags)); + // Isolate the last test in the hope that this is the only mutating test. // If so, we can completely avoid cloning functions. - let last_test = match tests.pop() { + let last_tuple = match tuples.pop() { None => return Err("no test commands found".to_string()), Some(t) => t, }; @@ -38,14 +52,16 @@ pub fn run(path: &Path) -> TestResult { let mut context = Context { details: details, verified: false, + flags: flags, + isa: None, }; - for test in &tests { - try!(run_one_test(test.borrow(), Cow::Borrowed(&func), &mut context)); + for tuple in &tuples { + try!(run_one_test(*tuple, Cow::Borrowed(&func), &mut context)); } // Run the last test with an owned function which means it won't need to clone it before // mutating. - try!(run_one_test(last_test.borrow(), Cow::Owned(func), &mut context)); + try!(run_one_test(last_tuple, Cow::Owned(func), &mut context)); } @@ -53,9 +69,42 @@ pub fn run(path: &Path) -> TestResult { Ok(started.elapsed()) } -fn run_one_test(test: &SubTest, func: Cow, context: &mut Context) -> Result<()> { +// Given a slice of tests, generate a vector of (test, flags, isa) tuples. +fn test_tuples<'a>(tests: &'a [Box], + isa_spec: &'a IsaSpec, + no_isa_flags: &'a Flags) + -> Result)>> { + let mut out = Vec::new(); + for test in tests { + if test.needs_isa() { + match *isa_spec { + IsaSpec::None(_) => { + // TODO: Generate a list of default ISAs. + return Err(format!("test {} requires an ISA", test.name())); + } + IsaSpec::Some(ref isas) => { + for isa in isas { + out.push((&**test, isa.flags(), Some(&**isa))); + } + } + } + } else { + out.push((&**test, no_isa_flags, None)); + } + } + Ok(out) +} + +fn run_one_test<'a>(tuple: (&'a SubTest, &'a Flags, Option<&'a TargetIsa>), + func: Cow, + context: &mut Context<'a>) + -> Result<()> { + let (test, flags, isa) = tuple; let name = format!("{}({})", test.name(), func.name); + context.flags = flags; + context.isa = isa; + // Should we run the verifier before this test? if !context.verified && test.needs_verifier() { try!(verify_function(&func).map_err(|e| e.to_string())); diff --git a/cranelift/src/tools/filetest/subtest.rs b/cranelift/src/tools/filetest/subtest.rs index b958193cd8..fee9bbef85 100644 --- a/cranelift/src/tools/filetest/subtest.rs +++ b/cranelift/src/tools/filetest/subtest.rs @@ -3,6 +3,8 @@ use std::result; use std::borrow::Cow; use cretonne::ir::Function; +use cretonne::isa::TargetIsa; +use cretonne::settings::Flags; use cton_reader::Details; use filecheck::{self, CheckerBuilder, Checker, Value as FCValue}; @@ -15,6 +17,13 @@ pub struct Context<'a> { /// Was the function verified before running this test? pub verified: bool, + + /// ISA-independent flags for this test. + pub flags: &'a Flags, + + /// Target ISA to test against. Only present for sub-tests whose `needs_isa` method returned + /// true. + pub isa: Option<&'a TargetIsa>, } /// Common interface for implementations of test commands. @@ -36,6 +45,11 @@ pub trait SubTest { false } + /// Does this test need a `TargetIsa` trait object? + fn needs_isa(&self) -> bool { + false + } + /// Run this test on `func`. fn run(&self, func: Cow, context: &Context) -> Result<()>; } From 4e09b48dd48ea119ab15542f571566db79dee389 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 21 Sep 2016 16:53:03 -0700 Subject: [PATCH 0321/3084] Expose Vec::get() in EntityMap. --- cranelift/src/libcretonne/entity_map.rs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/cranelift/src/libcretonne/entity_map.rs b/cranelift/src/libcretonne/entity_map.rs index 74c3cf5c46..ba2075370f 100644 --- a/cranelift/src/libcretonne/entity_map.rs +++ b/cranelift/src/libcretonne/entity_map.rs @@ -70,6 +70,11 @@ impl EntityMap k.index() < self.elems.len() } + /// Get the element at `k` if it exists. + pub fn get(&self, k: K) -> Option<&V> { + self.elems.get(k.index()) + } + /// Is this map completely empty? pub fn is_empty(&self) -> bool { self.elems.is_empty() From 2ec50203fbb510ba71b0a8b9651958a6fa1a200d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 21 Sep 2016 16:57:08 -0700 Subject: [PATCH 0322/3084] Print encodings as [R#10c] instead of [R/10c]. The # is a more conventional prefix for hexadecimal, and when ISA information is not available, there may be a decimal number in front which would be confusing. So prefer [1#10c] for the ISA-less encoding format. Here '1' is decimal and '#10c' is hexadecimal. --- cranelift/src/libcretonne/isa/encoding.rs | 4 ++-- cranelift/src/libcretonne/isa/riscv/mod.rs | 8 ++++---- meta/cretonne/__init__.py | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/cranelift/src/libcretonne/isa/encoding.rs b/cranelift/src/libcretonne/isa/encoding.rs index 0bb1f6cc62..4c8e33a764 100644 --- a/cranelift/src/libcretonne/isa/encoding.rs +++ b/cranelift/src/libcretonne/isa/encoding.rs @@ -50,7 +50,7 @@ impl Default for Encoding { impl fmt::Display for Encoding { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { if self.is_legal() { - write!(f, "#{}/{:02x}", self.recipe, self.bits) + write!(f, "{}#{:02x}", self.recipe, self.bits) } else { write!(f, "-") } @@ -68,7 +68,7 @@ impl fmt::Display for DisplayEncoding { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { if self.encoding.is_legal() { write!(f, - "{}/{:02x}", + "{}#{:02x}", self.recipe_names[self.encoding.recipe()], self.encoding.bits) } else { diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/cranelift/src/libcretonne/isa/riscv/mod.rs index 2ff352918d..cdc49f6973 100644 --- a/cranelift/src/libcretonne/isa/riscv/mod.rs +++ b/cranelift/src/libcretonne/isa/riscv/mod.rs @@ -97,7 +97,7 @@ mod tests { }; // ADDI is I/0b00100 - assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst64).unwrap()), "I/04"); + assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst64).unwrap()), "I#04"); // Try to encode iadd_imm.i64 vx1, -10000. let inst64_large = InstructionData::BinaryImm { @@ -119,7 +119,7 @@ mod tests { }; // ADDIW is I/0b00110 - assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32).unwrap()), "I/06"); + assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32).unwrap()), "I#06"); } // Same as above, but for RV32. @@ -166,7 +166,7 @@ mod tests { }; // ADDI is I/0b00100 - assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32).unwrap()), "I/04"); + assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32).unwrap()), "I#04"); // Create an imul.i32 which is encodable in RV32, but only when use_m is true. let mul32 = InstructionData::Binary { @@ -201,6 +201,6 @@ mod tests { ty: types::I32, args: [arg32, arg32], }; - assert_eq!(encstr(&*isa, isa.encode(&dfg, &mul32).unwrap()), "R/10c"); + assert_eq!(encstr(&*isa, isa.encode(&dfg, &mul32).unwrap()), "R#10c"); } } diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 82b0a02be6..90cbb2f983 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -1205,7 +1205,7 @@ class Encoding(object): self.isap = And.combine(recipe.isap, isap) def __str__(self): - return '[{}/{:02x}]'.format(self.recipe, self.encbits) + return '[{}#{:02x}]'.format(self.recipe, self.encbits) def ctrl_typevar(self): """ From ce0da25ce02db5eba0bee212dcc21ebfab3d5139 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 21 Sep 2016 17:21:13 -0700 Subject: [PATCH 0323/3084] Write out encoding annotations on instructions. All instructions with associated encodings are now annotated with encoding information in a column before the code. When write_function() is givan a TargetIsa reference, the annotations use ISA-specific names. Otherwise everything is numeric. --- cranelift/src/libcretonne/ir/function.rs | 4 +-- cranelift/src/libcretonne/lib.rs | 1 + cranelift/src/libcretonne/write.rs | 38 +++++++++++++++++++----- 3 files changed, 34 insertions(+), 9 deletions(-) diff --git a/cranelift/src/libcretonne/ir/function.rs b/cranelift/src/libcretonne/ir/function.rs index 3ff5aea00f..844101bf95 100644 --- a/cranelift/src/libcretonne/ir/function.rs +++ b/cranelift/src/libcretonne/ir/function.rs @@ -69,12 +69,12 @@ impl Function { impl Display for Function { fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write_function(fmt, self) + write_function(fmt, self, None) } } impl Debug for Function { fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write_function(fmt, self) + write_function(fmt, self, None) } } diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 7285f95602..f0a328ba8f 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -6,6 +6,7 @@ // ====------------------------------------------------------------------------------------==== // pub use verifier::verify_function; +pub use write::write_function; pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index c17025d1a8..bfdd13b32f 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -5,11 +5,13 @@ //! `cretonne-reader` crate. use ir::{Function, Ebb, Inst, Value, Type}; +use isa::TargetIsa; use std::fmt::{Result, Error, Write}; use std::result; /// Write `func` to `w` as equivalent text. -pub fn write_function(w: &mut Write, func: &Function) -> Result { +/// Use `isa` to emit ISA-dependent annotations. +pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) -> Result { try!(write_spec(w, func)); try!(writeln!(w, " {{")); let mut any = try!(write_preamble(w, func)); @@ -17,7 +19,7 @@ pub fn write_function(w: &mut Write, func: &Function) -> Result { if any { try!(writeln!(w, "")); } - try!(write_ebb(w, func, ebb)); + try!(write_ebb(w, func, isa, ebb)); any = true; } writeln!(w, "}}") @@ -67,6 +69,11 @@ pub fn write_ebb_header(w: &mut Write, func: &Function, ebb: Ebb) -> Result { // ebb10(vx4: f64, vx5: b1): // + // If we're writing encoding annotations, shift by 20. + if !func.encodings.is_empty() { + try!(write!(w, " ")); + } + let mut args = func.dfg.ebb_args(ebb); match args.next() { None => return writeln!(w, "{}:", ebb), @@ -83,10 +90,10 @@ pub fn write_ebb_header(w: &mut Write, func: &Function, ebb: Ebb) -> Result { writeln!(w, "):") } -pub fn write_ebb(w: &mut Write, func: &Function, ebb: Ebb) -> Result { +pub fn write_ebb(w: &mut Write, func: &Function, isa: Option<&TargetIsa>, ebb: Ebb) -> Result { try!(write_ebb_header(w, func, ebb)); for inst in func.layout.ebb_insts(ebb) { - try!(write_instruction(w, func, inst)); + try!(write_instruction(w, func, isa, inst)); } Ok(()) } @@ -127,10 +134,27 @@ fn type_suffix(func: &Function, inst: Inst) -> Option { Some(rtype) } -pub fn write_instruction(w: &mut Write, func: &Function, inst: Inst) -> Result { - try!(write!(w, " ")); +fn write_instruction(w: &mut Write, + func: &Function, + isa: Option<&TargetIsa>, + inst: Inst) + -> Result { + // Write out encoding info. + if let Some(enc) = func.encodings.get(inst).cloned() { + let mut s = String::with_capacity(16); + if let Some(isa) = isa { + try!(write!(s, "[{}]", isa.display_enc(enc))); + } else { + try!(write!(s, "[{}]", enc)); + } + // Align instruction following ISA annotation to col 24. + try!(write!(w, "{:23} ", s)); + } else { + // No annotations, simply indent by 4. + try!(write!(w, " ")); + } - // First write out the result values, if any. + // Write out the result values, if any. let mut has_results = false; for r in func.dfg.inst_results(inst) { if !has_results { From 51bcc78cea9b934e64dbedd352d2be0c9fd3eca1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 21 Sep 2016 13:32:46 -0700 Subject: [PATCH 0324/3084] Add a 'test legaliser' filetest command. This test command sends each function through legalize_function() and then filecheck. --- cranelift/filetests/isa/riscv/encoding.cton | 19 +++++++++ cranelift/src/libcretonne/lib.rs | 3 +- cranelift/src/tools/filetest/legalizer.rs | 45 +++++++++++++++++++++ cranelift/src/tools/filetest/mod.rs | 2 + 4 files changed, 68 insertions(+), 1 deletion(-) create mode 100644 cranelift/filetests/isa/riscv/encoding.cton create mode 100644 cranelift/src/tools/filetest/legalizer.rs diff --git a/cranelift/filetests/isa/riscv/encoding.cton b/cranelift/filetests/isa/riscv/encoding.cton new file mode 100644 index 0000000000..2ba06007bb --- /dev/null +++ b/cranelift/filetests/isa/riscv/encoding.cton @@ -0,0 +1,19 @@ +test legalizer +isa riscv supports_m=1 + +function int32(i32, i32) { +ebb0(v1: i32, v2: i32): + v10 = iadd v1, v2 + ; check: [R#0c] + ; sameln: $v10 = iadd + + v11 = isub v1, v2 + ; check: [R#200c] + ; sameln: $v11 = isub + + v12 = imul v1, v2 + ; check: [R#10c] + ; sameln: $v12 = imul + + return +} diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index f0a328ba8f..26a87bd223 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -7,6 +7,7 @@ pub use verifier::verify_function; pub use write::write_function; +pub use legalizer::legalize_function; pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); @@ -17,11 +18,11 @@ pub mod dominator_tree; pub mod entity_map; pub mod settings; pub mod verifier; -pub mod legalizer; mod write; mod constant_hash; mod predicates; +mod legalizer; #[cfg(test)] pub mod test_utils; diff --git a/cranelift/src/tools/filetest/legalizer.rs b/cranelift/src/tools/filetest/legalizer.rs new file mode 100644 index 0000000000..ea53a703b7 --- /dev/null +++ b/cranelift/src/tools/filetest/legalizer.rs @@ -0,0 +1,45 @@ +//! Test command for checking the IL legalizer. +//! +//! The `test legalizer` test command runs each function through `legalize_function()` and sends +//! the result to filecheck. + +use std::borrow::Cow; +use cretonne::{legalize_function, write_function}; +use cretonne::ir::Function; +use cton_reader::TestCommand; +use filetest::subtest::{SubTest, Context, Result, run_filecheck}; + +struct TestLegalizer; + +pub fn subtest(parsed: &TestCommand) -> Result> { + assert_eq!(parsed.command, "legalizer"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestLegalizer)) + } +} + +impl SubTest for TestLegalizer { + fn name(&self) -> Cow { + Cow::from("legalizer") + } + + fn is_mutating(&self) -> bool { + true + } + + fn needs_isa(&self) -> bool { + true + } + + fn run(&self, func: Cow, context: &Context) -> Result<()> { + let mut func = func.into_owned(); + let isa = context.isa.expect("legalizer needs an ISA"); + legalize_function(&mut func, isa); + + let mut text = String::new(); + try!(write_function(&mut text, &func, Some(isa)).map_err(|e| e.to_string())); + run_filecheck(&text, context) + } +} diff --git a/cranelift/src/tools/filetest/mod.rs b/cranelift/src/tools/filetest/mod.rs index a924cb04eb..28a9b94d45 100644 --- a/cranelift/src/tools/filetest/mod.rs +++ b/cranelift/src/tools/filetest/mod.rs @@ -17,6 +17,7 @@ mod runone; mod concurrent; mod domtree; mod verifier; +mod legalizer; /// The result of running the test in a file. pub type TestResult = Result; @@ -55,6 +56,7 @@ fn new_subtest(parsed: &TestCommand) -> subtest::Result> { "print-cfg" => print_cfg::subtest(parsed), "domtree" => domtree::subtest(parsed), "verifier" => verifier::subtest(parsed), + "legalizer" => legalizer::subtest(parsed), _ => Err(format!("unknown test command '{}'", parsed.command)), } } From 5112ddc13d94f48d245d6db411360be5766832d5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 22 Sep 2016 14:07:31 -0700 Subject: [PATCH 0325/3084] Basic *.cton syntax mode for Vim. --- misc/vim/ftdetect/cton.vim | 1 + misc/vim/syntax/cton.vim | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) create mode 100644 misc/vim/ftdetect/cton.vim create mode 100644 misc/vim/syntax/cton.vim diff --git a/misc/vim/ftdetect/cton.vim b/misc/vim/ftdetect/cton.vim new file mode 100644 index 0000000000..9d7754c472 --- /dev/null +++ b/misc/vim/ftdetect/cton.vim @@ -0,0 +1 @@ +au BufRead,BufNewFile *.cton set filetype=cton diff --git a/misc/vim/syntax/cton.vim b/misc/vim/syntax/cton.vim new file mode 100644 index 0000000000..2d2dcd0fef --- /dev/null +++ b/misc/vim/syntax/cton.vim @@ -0,0 +1,34 @@ +" Vim syntax file +" Language: Cretonne +" Maintainer: Jakob Stoklund Olesen / +syn match ctonEntity /\<\(v\|vx\|ss\|jt\|\)\d\+\>/ +syn match ctonLabel /\/ + +syn match ctonNumber /-\?\<\d\+\>/ +syn match ctonNumber /-\?\<0x\x\+\(\.\x*\)\(p[+-]\?\d\+\)\?\>/ + +syn region ctonCommentLine start=";" end="$" contains=ctonFilecheck + +hi def link ctonHeader Keyword +hi def link ctonDecl Keyword +hi def link ctonType Type +hi def link ctonEntity Identifier +hi def link ctonLabel Label +hi def link ctonNumber Number +hi def link ctonCommentLine Comment +hi def link ctonFilecheck SpecialComment + +let b:current_syntax = "cton" From acf41a7c09e4b07c54095416e86025bf67cfe1aa Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 23 Sep 2016 09:38:17 -0700 Subject: [PATCH 0326/3084] Add a Cretonne testing guide. Describe the basics of Rust-level tests, and go into more detail about the file-level tests. --- cranelift/docs/cton_lexer.py | 19 ++- cranelift/docs/index.rst | 3 +- cranelift/docs/langref.rst | 10 +- cranelift/docs/testing.rst | 295 +++++++++++++++++++++++++++++++++++ 4 files changed, 317 insertions(+), 10 deletions(-) create mode 100644 cranelift/docs/testing.rst diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py index eaa856c444..868bb23241 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/cton_lexer.py @@ -19,6 +19,17 @@ class CretonneLexer(RegexLexer): tokens = { 'root': [ + # Test header lines. + (r'^(test|isa|set)(?:( +)([-\w]+)' + + r'(?:(=)(?:(\d+)|(yes|no|true|false|on|off)|(\w+)))?)*' + + r'( *)$', + bygroups(Keyword.Namespace, Whitespace, Name.Attribute, + Operator, Number.Integer, Keyword.Constant, + Name.Constant, Whitespace)), + # Comments with filecheck or other test directive. + (r'(; *)([a-z]+:)(.*?)$', + bygroups(Comment.Single, Comment.Special, Comment.Single)), + # Plain comments. (r';.*?$', Comment.Single), # Strings are in double quotes, support \xx escapes only. (r'"([^"\\]+|\\[0-9a-fA-F]{2})*"', String), @@ -30,16 +41,16 @@ class CretonneLexer(RegexLexer): (r'[-+]?0[xX][0-9a-fA-F]*\.[0-9a-fA-F]*([pP]\d+)?', Number.Hex), (r'[-+]?(\d+\.\d+([eE]\d+)?|[sq]NaN|Inf)', Number.Float), (r'[-+]?\d+', Number.Integer), - # Reserved words. - (keywords('function'), Keyword), # Known attributes. (keywords('align', 'aligntrap', 'uext', 'sext', 'inreg'), Name.Attribute), # Well known value types. (r'\b(b\d+|i\d+|f32|f64)(x\d+)?\b', Keyword.Type), # v = value + # vx = value # ss = stack slot - (r'(v|ss)\d+', Name.Variable), + # jt = jump table + (r'(vx?|ss|jt)\d+', Name.Variable), # ebb = extended basic block (r'(ebb)\d+', Name.Label), # Match instruction names in context. @@ -52,7 +63,7 @@ class CretonneLexer(RegexLexer): (r'->|=|:', Operator), (r'[{}(),.]', Punctuation), (r'[ \t]+', Text), - ] + ], } diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst index 88d9257213..b5a23ab28b 100644 --- a/cranelift/docs/index.rst +++ b/cranelift/docs/index.rst @@ -4,10 +4,11 @@ Cretonne Code Generator Contents: .. toctree:: - :maxdepth: 2 + :maxdepth: 1 langref metaref + testing Indices and tables ================== diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 848d140db8..10385cb7f4 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -48,11 +48,11 @@ fall through to the next EBB without an explicit branch. A ``.cton`` file consists of a sequence of independent function definitions: .. productionlist:: - function-list : { function } - function : function-spec "{" preamble function-body "}" - function-spec : "function" function-name signature - preamble : { preamble-decl } - function-body : { extended-basic-block } + function_list : { function } + function : function_spec "{" preamble function_body "}" + function_spec : "function" function_name signature + preamble : { preamble_decl } + function_body : { extended_basic_block } Static single assignment form ----------------------------- diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst new file mode 100644 index 0000000000..c902dc70f6 --- /dev/null +++ b/cranelift/docs/testing.rst @@ -0,0 +1,295 @@ +**************** +Testing Cretonne +**************** + +Cretonne is tested at multiple levels of abstraction and integration. When +possible, Rust unit tests are used to verify single functions and types. When +testing the interaction between compiler passes, file-level tests are +appropriate. + +The top-level shell script :file:`test-all.sh` runs all of the tests in the +Cretonne repository. + +Rust tests +========== + +.. highlight:: rust + +Rust and Cargo have good support for testing. Cretonne uses unit tests, doc +tests, and integration tests where appropriate. + +Unit tests +---------- + +Unit test live in a ``tests`` sub-module of the code they are testing:: + + pub fn add(x: u32, y: u32) { + x + y + } + + #[cfg(test)] + mod tests { + use super::add; + + #[test] + check_add() { + assert_eq!(add(2, 2), 4); + } + } + +Since sub-modules have access to non-public items in a Rust module, unit tests +can be used to test module-internal functions and types too. + +Doc tests +--------- + +Documentation comments can contain code snippets which are also compiled and +tested:: + + //! The `Flags` struct is immutable once it has been created. A `Builder` instance is used to + //! create it. + //! + //! # Example + //! ``` + //! use cretonne::settings::{self, Configurable}; + //! + //! let mut b = settings::builder(); + //! b.set("opt_level", "fastest"); + //! + //! let f = settings::Flags::new(&b); + //! assert_eq!(f.opt_level(), settings::OptLevel::Fastest); + //! ``` + +These tests are useful for demonstrating how to use an API, and running them +regularly makes sure that they stay up to date. Documentation tests are not +appropriate for lots of assertions, use unit tests for that. + +Integration tests +----------------- + +Integration tests are Rust source files thar are compiled and linked +individually. They are used to exercise the external API of the crates under +test. + +These tests are usually found in the :file:`src/tools/tests` sub-directory where they +have access to all the crates in the Cretonne repository. The +:file:`libcretonne` and :file:`libreader` crates have no external dependencies +which can make testing tedious. + +File tests +========== + +.. highlight:: cton + +Compilers work with large data structures representing programs, and it quickly +gets unwieldy to generate test data programmatically. File-level tests make it +easier to provide substantial input functions for the compiler tests. + +File tests are :file:`*.cton` files in the :file:`filetests/` directory +hierarchy. Each file has a header describing what to test followed by a number +of input functions in the :doc:`Cretonne textual intermediate language +`: + +.. productionlist:: + test_file : test_header `function_list` + test_header : test_commands (`isa_specs` | `settings`) + test_commands : test_command { test_command } + test_command : "test" test_name { option } "\n" + +The available test commands are described below. + +Many test comands only make sense in the context of a target instruction set +architecture. These tests require one or more ISA specifications in the test +header: + +.. productionlist:: + isa_specs : { [`settings`] isa_spec } + isa_spec : "isa" isa_name { `option` } "\n" + +The options given on the ``isa`` line modify the ISA-specific settings defined in +:file:`meta/isa/*/setings.py`. + +All types of tests allow shared Cretonne settings to be modified: + +.. productionlist:: + settings : { setting } + setting : "set" { option } "\n" + option : flag | setting "=" value + +The shared settings available for all target ISAs are defined in +:file:`meta/cretonne/settings.py`. + +The ``set`` lines apply settings cumulatively:: + + test legalizer + set opt_level=best + set is_64bit=1 + isa riscv + set is_64bit=0 + isa riscv supports_m=false + + function foo() {} + +This example will run the legalizer test twice. Both runs will have +``opt_level=best``, but they will have different ``is_64bit`` settings. The 32-bit +run will also have the RISC-V specific flag ``supports_m`` disabled. + +Filecheck +--------- + +Many of the test commands bescribed below use *filecheck* to verify their +output. Filecheck is a Rust implementation of the LLVM tool of the same name. +See the :file:`libfilecheck` documentation for details of its syntax. + +Comments in :file:`.cton` files are associated with the entity they follow. +This typically means and instruction or the whole function. Those tests that +use filecheck will extract comments associated with each function (or its +entities) and scan them for filecheck directives. The test output for each +function is then matched againts the filecheck directives for that function. + +Note that LLVM's file tests don't separate filecheck directives by their +associated function. It verifies the concatenated output against all filecheck +directives in the test file. LLVM's :command:`FileCheck` command has a +``CHECK-LABEL:`` directive to help separate the output from different functions. +Cretonne's tests don't need this. + +Filecheck variables +~~~~~~~~~~~~~~~~~~~ + +Cretonne's IL parser causes entities like values and EBBs to be renumbered. It +maintains a source mapping to resolve references in the text, but when a +function is written out as text as part of a test, all of the entities have the +new numbers. This can complicate the filecheck directives since they need to +refer to the new entity numbers, not the ones in the adjacent source text. + +To help with this, the parser's source-to-entity mapping is made available as +predefined filecheck variables. A value by the source name ``v10`` can be +referenced as the filecheck variable ``$v10``. The variable expands to the +renumbered entity name. + +`test cat` +---------- + +This is one of the simplest file tests, used for testing the conversion to and +from textual IL. The ``test cat`` command simply parses each function and +converts it back to text again. The text of each function is then matched +against the associated filecheck directives. + +Example:: + + function r1() -> i32, f32 { + ebb1: + v10 = iconst.i32 3 + v20 = f32const 0.0 + return v10, v20 + } + ; sameln: function r1() -> i32, f32 { + ; nextln: ebb0: + ; nextln: v0 = iconst.i32 3 + ; nextln: v1 = f32const 0.0 + ; nextln: return v0, v1 + ; nextln: } + +Notice that the values ``v10`` and ``v20`` in the source were renumbered to +``v0`` and ``v1`` respectively during parsing. The equivalent test using +filecheck variables would be:: + + function r1() -> i32, f32 { + ebb1: + v10 = iconst.i32 3 + v20 = f32const 0.0 + return v10, v20 + } + ; sameln: function r1() -> i32, f32 { + ; nextln: ebb0: + ; nextln: $v10 = iconst.i32 3 + ; nextln: $v20 = f32const 0.0 + ; nextln: return $v10, $v20 + ; nextln: } + +`test verifier` +--------------- + +Run each function through the IL verifier and check that it produces the +expected error messages. + +Expected error messages are indicated with an ``error:`` directive *on the +instruction that produces the verifier error*. Both the error message and +reported location of the error is verified:: + + test verifier + + function test(i32) { + ebb0(v0: i32): + jump ebb1 ; error: terminator + return + } + +This example test passed if the verifier fails with an error message containing +the sub-string ``"terminator"`` *and* the error is reported for the ``jump`` +instruction. + +If a function contains no ``error:`` annotations, the test passes if the +function verifies correctly. + +`test print-cfg` +---------------- + +Print the control flow graph of each function as a Graphviz graph, and run +filecheck over the result. See also the :command:`cton-util print-cfg` +command:: + + ; For testing cfg generation. This code is nonsense. + test print-cfg + test verifier + + function nonsense(i32, i32) -> f32 { + ; check: digraph nonsense { + ; regex: I=\binst\d+\b + ; check: label="{ebb0 | <$(BRZ=$I)>brz ebb2 | <$(JUMP=$I)>jump ebb1}"] + + ebb0(v1: i32, v2: i32): + brz v2, ebb2 ; unordered: ebb0:$BRZ -> ebb2 + v4 = iconst.i32 0 + jump ebb1(v4) ; unordered: ebb0:$JUMP -> ebb1 + + ebb1(v5: i32): + return v1 + + ebb2: + v100 = f32const 0.0 + return v100 + } + +`test domtree` +-------------- + +Compute the dominator tree of each function and validate it against the +``dominates:`` annotations:: + + test domtree + + function test(i32) { + ebb0(v0: i32): + jump ebb1 ; dominates: ebb1 + ebb1: + brz v0, ebb3 ; dominates: ebb3 + jump ebb2 ; dominates: ebb2 + ebb2: + jump ebb3 + ebb3: + return + } + +Every reachable extended basic block except for the entry block has an +*immediate dominator* which is a jump or branch instruction. This test passes +if the ``dominates:`` annotations on the immediate dominator instructions are +both correct and complete. + +`test legalizer` +---------------- + +Legalize each function for the specified target ISA and run the resulting +function through filecheck. This test command can be used to validate the +encodings selected for legal instructions as well as the instruction +transformations performed by the legalizer. From b1bd3140dba40cb6caeef5159d74417215a6e46b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 23 Sep 2016 13:32:26 -0700 Subject: [PATCH 0327/3084] Integer add with carry instructions. Integer addition with carry in/out/both. --- cranelift/filetests/parser/ternary.cton | 13 ++++++ cranelift/src/libcretonne/ir/instructions.rs | 18 ++++++++ cranelift/src/libcretonne/write.rs | 1 + cranelift/src/libreader/parser.rs | 24 ++++++++++- meta/cretonne/base.py | 44 +++++++++++++++++++- meta/cretonne/formats.py | 4 ++ meta/gen_instr.py | 5 +-- 7 files changed, 103 insertions(+), 6 deletions(-) create mode 100644 cranelift/filetests/parser/ternary.cton diff --git a/cranelift/filetests/parser/ternary.cton b/cranelift/filetests/parser/ternary.cton new file mode 100644 index 0000000000..5305126c49 --- /dev/null +++ b/cranelift/filetests/parser/ternary.cton @@ -0,0 +1,13 @@ +test cat +test verifier + +function add_i96(i32, i32, i32, i32, i32, i32) -> i32, i32, i32 { +ebb1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32): + v10, v11 = iadd_cout v1, v4 + ;check: $v10, $v11 = iadd_cout $v1, $v4 + v20, v21 = iadd_carry v2, v5, v11 + ; check: $v20, $v21 = iadd_carry $v2, $v5, $v11 + v30 = iadd_cin v3, v6, v21 + ; check: $v30 = iadd_cin $v3, $v6, $v21 + return v10, v20, v30 +} diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index c8228d1f43..bfc1fe65f3 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -149,6 +149,11 @@ pub enum InstructionData { ty: Type, args: [Value; 3], }, + TernaryOverflow { + opcode: Opcode, + ty: Type, + data: Box, + }, InsertLane { opcode: Opcode, ty: Type, @@ -254,6 +259,19 @@ impl Default for VariableArgs { } } +/// Payload data for ternary instructions with multiple results, such as `iadd_carry`. +#[derive(Clone, Debug)] +pub struct TernaryOverflowData { + pub second_result: Value, + pub args: [Value; 3], +} + +impl Display for TernaryOverflowData { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + write!(f, "{}, {}, {}", self.args[0], self.args[1], self.args[2]) + } +} + /// Payload data for jump instructions. These need to carry lists of EBB arguments that won't fit /// in the allowed InstructionData size. #[derive(Clone, Debug)] diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index bfdd13b32f..ea73d18666 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -190,6 +190,7 @@ fn write_instruction(w: &mut Write, BinaryImmRev { imm, arg, .. } => writeln!(w, " {}, {}", imm, arg), BinaryOverflow { args, .. } => writeln!(w, " {}, {}", args[0], args[1]), Ternary { args, .. } => writeln!(w, " {}, {}, {}", args[0], args[1], args[2]), + TernaryOverflow { ref data, .. } => writeln!(w, " {}", data), InsertLane { lane, args, .. } => writeln!(w, " {}, {}, {}", args[0], lane, args[1]), ExtractLane { lane, arg, .. } => writeln!(w, " {}, {}", arg, lane), IntCompare { cond, args, .. } => writeln!(w, " {}, {}, {}", cond, args[0], args[1]), diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 05fa9b831b..723cfb1951 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -13,8 +13,8 @@ use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotDa use cretonne::ir::types::{VOID, Signature, ArgumentType, ArgumentExtension}; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::{AnyEntity, NO_EBB, NO_INST, NO_VALUE}; -use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, JumpData, - BranchData, ReturnData}; +use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, + TernaryOverflowData, JumpData, BranchData, ReturnData}; use cretonne::isa; use cretonne::settings; use testfile::{TestFile, Details, Comment}; @@ -138,6 +138,10 @@ impl Context { try!(self.map.rewrite_values(args, loc)); } + InstructionData::TernaryOverflow { ref mut data, .. } => { + try!(self.map.rewrite_values(&mut data.args, loc)); + } + InstructionData::Jump { ref mut data, .. } => { try!(self.map.rewrite_ebb(&mut data.destination, loc)); try!(self.map.rewrite_values(&mut data.arguments, loc)); @@ -1066,6 +1070,22 @@ impl<'a> Parser<'a> { args: [ctrl_arg, true_arg, false_arg], } } + InstructionFormat::TernaryOverflow => { + // Names here refer to the `iadd_carry` instruction. + let lhs = try!(self.match_value("expected SSA value first operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let rhs = try!(self.match_value("expected SSA value second operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let cin = try!(self.match_value("expected SSA value third operand")); + InstructionData::TernaryOverflow { + opcode: opcode, + ty: VOID, + data: Box::new(TernaryOverflowData { + second_result: NO_VALUE, + args: [lhs, rhs, cin], + }), + } + } InstructionFormat::Jump => { // Parse the destination EBB number. Don't translate source to local numbers yet. let ebb_num = try!(self.match_ebb("expected jump destination EBB")); diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 30d890b076..f94668da4a 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -6,7 +6,7 @@ support. """ from __future__ import absolute_import from . import TypeVar, Operand, Instruction, InstructionGroup, variable_args -from .types import i8, f32, f64 +from .types import i8, f32, f64, b1 from .immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc from . import entities @@ -405,6 +405,48 @@ isub_imm = Instruction( """, ins=(X, y), outs=a) +# +# Integer arithmetic with carry. +# +a = Operand('a', iB) +x = Operand('x', iB) +y = Operand('y', iB) +cin = Operand('cin', b1, doc="Input carry flag") +cout = Operand('cout', b1, doc="Output carry flag") + +iadd_cin = Instruction( + 'iadd_cin', """ + Add integers with carry in. + + Same as :inst:`iadd` with an additional carry input. + + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(x, y, cin), outs=a) + +iadd_cout = Instruction( + 'iadd_cout', """ + Add integers with carry out. + + Same as :inst:`iadd` with an additional carry output. + + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(x, y), outs=(a, cout)) + +iadd_carry = Instruction( + 'iadd_carry', """ + Add integers with carry in and out. + + Same as :inst:`iadd` with an additional carry output. + + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(x, y, cin), outs=(a, cout)) + # # Bitwise operations. # diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index 06f4593086..48b9d978af 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -30,6 +30,10 @@ BinaryOverflow = InstructionFormat(value, value, multiple_results=True) # The fma instruction has the same constraint on all inputs. Ternary = InstructionFormat(value, value, value, typevar_operand=1) +# Carry in *and* carry out for `iadd_carry` and friends. +TernaryOverflow = InstructionFormat( + value, value, value, multiple_results=True, boxed_storage=True) + InsertLane = InstructionFormat(value, ('lane', uimm8), value) ExtractLane = InstructionFormat(value, ('lane', uimm8)) diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 139acf59b7..6c67077cb0 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -153,9 +153,8 @@ def gen_instruction_data_impl(fmt): if f.boxed_storage: fmt.line( n + - ' {{ ref data, .. }} => ' + - 'Some(data.args[{}]),' - .format(i)) + ' { ref data, .. } => ' + + ('Some(data.args[{}]),'.format(i))) else: fmt.line( n + From f34da59babc45e618e424a239a0b9a1ee30c4eb9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 23 Sep 2016 15:47:39 -0700 Subject: [PATCH 0328/3084] Integer subtraction with borrow flags. This is the x86-style of borrow flags. ARM uses subtract-with-carry which inverts the sense of the carry flag. --- cranelift/filetests/parser/ternary.cton | 11 ++++ meta/cretonne/base.py | 79 ++++++++++++++++++++++--- 2 files changed, 82 insertions(+), 8 deletions(-) diff --git a/cranelift/filetests/parser/ternary.cton b/cranelift/filetests/parser/ternary.cton index 5305126c49..99cb15b566 100644 --- a/cranelift/filetests/parser/ternary.cton +++ b/cranelift/filetests/parser/ternary.cton @@ -11,3 +11,14 @@ ebb1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32): ; check: $v30 = iadd_cin $v3, $v6, $v21 return v10, v20, v30 } + +function sub_i96(i32, i32, i32, i32, i32, i32) -> i32, i32, i32 { +ebb1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32): + v10, v11 = isub_bout v1, v4 + ;check: $v10, $v11 = isub_bout $v1, $v4 + v20, v21 = isub_borrow v2, v5, v11 + ; check: $v20, $v21 = isub_borrow $v2, $v5, $v11 + v30 = isub_bin v3, v6, v21 + ; check: $v30 = isub_bin $v3, $v6, $v21 + return v10, v20, v30 +} diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index f94668da4a..150155d908 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -406,24 +406,30 @@ isub_imm = Instruction( ins=(X, y), outs=a) # -# Integer arithmetic with carry. +# Integer arithmetic with carry and/or borrow. # a = Operand('a', iB) x = Operand('x', iB) y = Operand('y', iB) -cin = Operand('cin', b1, doc="Input carry flag") -cout = Operand('cout', b1, doc="Output carry flag") +c_in = Operand('c_in', b1, doc="Input carry flag") +c_out = Operand('c_out', b1, doc="Output carry flag") +b_in = Operand('b_in', b1, doc="Input borrow flag") +b_out = Operand('b_out', b1, doc="Output borrow flag") iadd_cin = Instruction( 'iadd_cin', """ Add integers with carry in. - Same as :inst:`iadd` with an additional carry input. + Same as :inst:`iadd` with an additional carry input. Computes: + + .. math:: + + a = x + y + c_{in} \pmod 2^B Polymorphic over all scalar integer types, but does not support vector types. """, - ins=(x, y, cin), outs=a) + ins=(x, y, c_in), outs=a) iadd_cout = Instruction( 'iadd_cout', """ @@ -431,21 +437,78 @@ iadd_cout = Instruction( Same as :inst:`iadd` with an additional carry output. + .. math:: + + a &= x + y \pmod 2^B \\ + c_{out} &= x+y >= 2^B + Polymorphic over all scalar integer types, but does not support vector types. """, - ins=(x, y), outs=(a, cout)) + ins=(x, y), outs=(a, c_out)) iadd_carry = Instruction( 'iadd_carry', """ Add integers with carry in and out. - Same as :inst:`iadd` with an additional carry output. + Same as :inst:`iadd` with an additional carry input and output. + + .. math:: + + a &= x + y + c_{in} \pmod 2^B \\ + c_{out} &= x + y + c_{in} >= 2^B Polymorphic over all scalar integer types, but does not support vector types. """, - ins=(x, y, cin), outs=(a, cout)) + ins=(x, y, c_in), outs=(a, c_out)) + +isub_bin = Instruction( + 'isub_bin', """ + Subtract integers with borrow in. + + Same as :inst:`isub` with an additional borrow flag input. Computes: + + .. math:: + + a = x - (y + b_{in}) \pmod 2^B + + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(x, y, b_in), outs=a) + +isub_bout = Instruction( + 'isub_bout', """ + Subtract integers with borrow out. + + Same as :inst:`isub` with an additional borrow flag output. + + .. math:: + + a &= x - y \pmod 2^B \\ + b_{out} &= x < y + + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(x, y), outs=(a, b_out)) + +isub_borrow = Instruction( + 'isub_borrow', """ + Subtract integers with borrow in and out. + + Same as :inst:`isub` with an additional borrow flag input and output. + + .. math:: + + a &= x - (y + b_{in}) \pmod 2^B \\ + b_{out} &= x < y + b_{in} + + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(x, y, b_in), outs=(a, b_out)) # # Bitwise operations. From 6d3883cf78aee14660f7234616d15daff25084c4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 23 Sep 2016 16:41:14 -0700 Subject: [PATCH 0329/3084] Add an autoinstgroup Sphinx directive. This directive documents an instruction group and lists all instructions contained in the group, whether they have been documented or not. --- cranelift/docs/cton_domain.py | 36 +++++++++++++++++++++++++++++++++++ cranelift/docs/langref.rst | 13 +++++++++++++ 2 files changed, 49 insertions(+) diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index ba762b4df3..b45aa5e677 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -173,6 +173,10 @@ class CtonInst(CtonObject): return name +class CtonInstGroup(CtonObject): + """A Cretonne IL instruction group.""" + + class CretonneDomain(Domain): """Cretonne domain for intermediate language objects.""" name = 'cton' @@ -186,11 +190,13 @@ class CretonneDomain(Domain): directives = { 'type': CtonType, 'inst': CtonInst, + 'instgroup': CtonInstGroup, } roles = { 'type': XRefRole(), 'inst': XRefRole(), + 'instgroup': XRefRole(), } initial_data = { @@ -327,9 +333,39 @@ class InstDocumenter(sphinx.ext.autodoc.Documenter): sourcename) +class InstGroupDocumenter(sphinx.ext.autodoc.ModuleLevelDocumenter): + # Invoke with .. autoinstgroup:: + objtype = 'instgroup' + # Convert into cton:instgroup directives + domain = 'cton' + directivetype = 'instgroup' + + @classmethod + def can_document_member(cls, member, membername, isattr, parent): + return False + + def format_name(self): + return "{}.{}".format(self.modname, ".".join(self.objpath)) + + def add_content(self, more_content, no_docstring=False): + super(InstGroupDocumenter, self).add_content( + more_content, no_docstring) + sourcename = self.get_sourcename() + indexed = self.env.domaindata['cton']['objects'] + + names = [inst.name for inst in self.object.instructions] + names.sort() + for name in names: + if name in indexed: + self.add_line(u':cton:inst:`{}`'.format(name), sourcename) + else: + self.add_line(u'``{}``'.format(name), sourcename) + + def setup(app): app.add_domain(CretonneDomain) app.add_autodocumenter(TypeDocumenter) app.add_autodocumenter(InstDocumenter) + app.add_autodocumenter(InstGroupDocumenter) return {'version': '0.1'} diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 10385cb7f4..99a6e1ca62 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -308,6 +308,7 @@ Quiet NaNs Signaling NaNs Displayed as ``-sNaN:0xT``. + Control flow ============ @@ -814,6 +815,18 @@ Conversion operations .. autoinst:: fcvt_from_uint .. autoinst:: fcvt_from_sint + +Base instruction group +====================== + +All of the shared instructions are part of the :instgroup:`base` instruction +group. + +.. autoinstgroup:: cretonne.base.instructions + +Target ISAs may define further instructions in their own instruction groups. + + Glossary ======== From 6f243ab35da872f54a38773c42fde5690a7a0679 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 23 Sep 2016 16:49:35 -0700 Subject: [PATCH 0330/3084] Add documentation links to all existing instructions. --- cranelift/docs/langref.rst | 20 ++++++++++++++------ meta/cretonne/base.py | 12 ++++++------ 2 files changed, 20 insertions(+), 12 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 99a6e1ca62..e42d82ecf4 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -674,14 +674,14 @@ Integer operations .. autoinst:: icmp .. autoinst:: iadd .. autoinst:: iadd_imm +.. autoinst:: iadd_cin +.. autoinst:: iadd_cout +.. autoinst:: iadd_carry .. autoinst:: isub .. autoinst:: isub_imm - -.. todo:: Integer overflow arithmetic - - Add instructions for add with carry out / carry in and so on. Enough to - implement larger integer types efficiently. It should also be possible to - legalize :type:`i64` arithmetic to terms of :type:`i32` operations. +.. autoinst:: isub_bin +.. autoinst:: isub_bout +.. autoinst:: isub_borrow .. autoinst:: imul .. autoinst:: imul_imm @@ -722,8 +722,11 @@ bitwise operations are working on the binary representation of the values. When operating on boolean values, the bitwise operations work as logical operators. .. autoinst:: band +.. autoinst:: band_imm .. autoinst:: bor +.. autoinst:: bor_imm .. autoinst:: bxor +.. autoinst:: bxor_imm .. autoinst:: bnot .. todo:: Redundant bitwise operators. @@ -740,10 +743,15 @@ type, and all the lanes are shifted the same amount. The shift amount is masked to the number of bits in a *lane*, not the full size of the vector type. .. autoinst:: rotl +.. autoinst:: rotl_imm .. autoinst:: rotr +.. autoinst:: rotr_imm .. autoinst:: ishl +.. autoinst:: ishl_imm .. autoinst:: ushr +.. autoinst:: ushr_imm .. autoinst:: sshr +.. autoinst:: sshr_imm The bit-counting instructions below are scalar only. diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index 150155d908..f7b6fb06e4 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -417,7 +417,7 @@ b_in = Operand('b_in', b1, doc="Input borrow flag") b_out = Operand('b_out', b1, doc="Output borrow flag") iadd_cin = Instruction( - 'iadd_cin', """ + 'iadd_cin', r""" Add integers with carry in. Same as :inst:`iadd` with an additional carry input. Computes: @@ -432,7 +432,7 @@ iadd_cin = Instruction( ins=(x, y, c_in), outs=a) iadd_cout = Instruction( - 'iadd_cout', """ + 'iadd_cout', r""" Add integers with carry out. Same as :inst:`iadd` with an additional carry output. @@ -448,7 +448,7 @@ iadd_cout = Instruction( ins=(x, y), outs=(a, c_out)) iadd_carry = Instruction( - 'iadd_carry', """ + 'iadd_carry', r""" Add integers with carry in and out. Same as :inst:`iadd` with an additional carry input and output. @@ -464,7 +464,7 @@ iadd_carry = Instruction( ins=(x, y, c_in), outs=(a, c_out)) isub_bin = Instruction( - 'isub_bin', """ + 'isub_bin', r""" Subtract integers with borrow in. Same as :inst:`isub` with an additional borrow flag input. Computes: @@ -479,7 +479,7 @@ isub_bin = Instruction( ins=(x, y, b_in), outs=a) isub_bout = Instruction( - 'isub_bout', """ + 'isub_bout', r""" Subtract integers with borrow out. Same as :inst:`isub` with an additional borrow flag output. @@ -495,7 +495,7 @@ isub_bout = Instruction( ins=(x, y), outs=(a, b_out)) isub_borrow = Instruction( - 'isub_borrow', """ + 'isub_borrow', r""" Subtract integers with borrow in and out. Same as :inst:`isub` with an additional borrow flag input and output. From 8cbaacac4887bfd5b71dedd9a8dec920bddf3fb4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 27 Sep 2016 10:53:53 -0700 Subject: [PATCH 0331/3084] Move TypeVar and TypeSet into their own Python package. These classes are not very entangled with the rest of __init__, and we'll be expanding them a bit. --- cranelift/docs/metaref.rst | 2 +- meta/cretonne/__init__.py | 94 +----------------------------------- meta/cretonne/base.py | 3 +- meta/cretonne/typevar.py | 98 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 102 insertions(+), 95 deletions(-) create mode 100644 meta/cretonne/typevar.py diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 8df75d80a2..4edf708041 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -96,7 +96,7 @@ instances that refer to a *type variable* instead of a concrete value type. Polymorphism only works for SSA value operands. Other operands have a fixed operand kind. -.. autoclass:: TypeVar +.. autoclass:: cretonne.typevar.TypeVar :members: If multiple operands refer to the same type variable they will be required to diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 90cbb2f983..d318b27ef3 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -8,7 +8,7 @@ from __future__ import absolute_import import re import math import importlib -from collections import namedtuple, OrderedDict +from collections import OrderedDict from .predicates import And @@ -518,98 +518,6 @@ class BoolType(ScalarType): return 'BoolType(bits={})'.format(self.bits) -# Parametric polymorphism. - - -#: A `TypeSet` represents a set of types. We don't allow arbitrary subsets of -#: types, but use a parametrized approach instead. -#: This is represented as a named tuple so it can be used as a dictionary key. -TypeSet = namedtuple( - 'TypeSet', [ - 'allow_scalars', - 'allow_simd', - 'base', - 'all_ints', - 'all_floats', - 'all_bools']) - - -class TypeVar(object): - """ - Type variables can be used in place of concrete types when defining - instructions. This makes the instructions *polymorphic*. - - A type variable is restricted to vary over a subset of the value types. - This subset is specified by a set of flags that control the permitted base - types and whether the type variable can assume scalar or vector types, or - both. - - :param name: Short name of type variable used in instruction descriptions. - :param doc: Documentation string. - :param base: Single base type or list of base types. Use this to specify an - exact set of base types if the general categories below are not good - enough. - :param ints: Allow all integer base types. - :param floats: Allow all floating point base types. - :param bools: Allow all boolean base types. - :param scalars: Allow type variable to assume scalar types. - :param simd: Allow type variable to assume vector types. - """ - - def __init__( - self, name, doc, base=None, - ints=False, floats=False, bools=False, - scalars=True, simd=False, - derived_func=None): - self.name = name - self.__doc__ = doc - self.base = base - self.is_derived = isinstance(base, TypeVar) - if self.is_derived: - assert derived_func - self.derived_func = derived_func - self.name = '{}({})'.format(derived_func, base.name) - else: - self.type_set = TypeSet( - allow_scalars=scalars, - allow_simd=simd, - base=base, - all_ints=ints, - all_floats=floats, - all_bools=bools) - - def __str__(self): - return "`{}`".format(self.name) - - def lane_of(self): - """ - Return a derived type variable that is the scalar lane type of this - type variable. - - When this type variable assumes a scalar type, the derived type will be - the same scalar type. - """ - return TypeVar(None, None, base=self, derived_func='LaneOf') - - def as_bool(self): - """ - Return a derived type variable that has the same vector geometry as - this type variable, but with boolean lanes. Scalar types map to `b1`. - """ - return TypeVar(None, None, base=self, derived_func='AsBool') - - def operand_kind(self): - # When a `TypeVar` object is used to describe the type of an `Operand` - # in an instruction definition, the kind of that operand is an SSA - # value. - return value - - def free_typevar(self): - if isinstance(self.base, TypeVar): - return self.base - else: - return self - # Defining instructions. diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index f7b6fb06e4..fdca2484f9 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -5,7 +5,8 @@ This module defines the basic Cretonne instruction set that all targets support. """ from __future__ import absolute_import -from . import TypeVar, Operand, Instruction, InstructionGroup, variable_args +from . import Operand, Instruction, InstructionGroup, variable_args +from .typevar import TypeVar from .types import i8, f32, f64, b1 from .immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc from . import entities diff --git a/meta/cretonne/typevar.py b/meta/cretonne/typevar.py new file mode 100644 index 0000000000..ec2693829b --- /dev/null +++ b/meta/cretonne/typevar.py @@ -0,0 +1,98 @@ +""" +Type variables for Parametric polymorphism. + +Cretonne instructions and instruction transformations can be specified to be +polymorphic by using type variables. +""" +from __future__ import absolute_import +from collections import namedtuple +from . import value + +#: A `TypeSet` represents a set of types. We don't allow arbitrary subsets of +#: types, but use a parametrized approach instead. +#: This is represented as a named tuple so it can be used as a dictionary key. +TypeSet = namedtuple( + 'TypeSet', [ + 'allow_scalars', + 'allow_simd', + 'base', + 'all_ints', + 'all_floats', + 'all_bools']) + + +class TypeVar(object): + """ + Type variables can be used in place of concrete types when defining + instructions. This makes the instructions *polymorphic*. + + A type variable is restricted to vary over a subset of the value types. + This subset is specified by a set of flags that control the permitted base + types and whether the type variable can assume scalar or vector types, or + both. + + :param name: Short name of type variable used in instruction descriptions. + :param doc: Documentation string. + :param base: Single base type or list of base types. Use this to specify an + exact set of base types if the general categories below are not good + enough. + :param ints: Allow all integer base types. + :param floats: Allow all floating point base types. + :param bools: Allow all boolean base types. + :param scalars: Allow type variable to assume scalar types. + :param simd: Allow type variable to assume vector types. + """ + + def __init__( + self, name, doc, base=None, + ints=False, floats=False, bools=False, + scalars=True, simd=False, + derived_func=None): + self.name = name + self.__doc__ = doc + self.base = base + self.is_derived = isinstance(base, TypeVar) + if self.is_derived: + assert derived_func + self.derived_func = derived_func + self.name = '{}({})'.format(derived_func, base.name) + else: + self.type_set = TypeSet( + allow_scalars=scalars, + allow_simd=simd, + base=base, + all_ints=ints, + all_floats=floats, + all_bools=bools) + + def __str__(self): + return "`{}`".format(self.name) + + def lane_of(self): + """ + Return a derived type variable that is the scalar lane type of this + type variable. + + When this type variable assumes a scalar type, the derived type will be + the same scalar type. + """ + return TypeVar(None, None, base=self, derived_func='LaneOf') + + def as_bool(self): + """ + Return a derived type variable that has the same vector geometry as + this type variable, but with boolean lanes. Scalar types map to `b1`. + """ + return TypeVar(None, None, base=self, derived_func='AsBool') + + def operand_kind(self): + # When a `TypeVar` object is used to describe the type of an `Operand` + # in an instruction definition, the kind of that operand is an SSA + # value. + return value + + def free_typevar(self): + if isinstance(self.base, TypeVar): + return self.base + else: + return self From a616a46db76f1e8f8838005ad7dd7b383bfe6f81 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 27 Sep 2016 13:31:31 -0700 Subject: [PATCH 0332/3084] Represent type sets with ranges. Allow limits on the smallest and largest integer type in the set, the highest and lowest number of lanes etc. --- cranelift/src/libcretonne/ir/instructions.rs | 136 ++++++++------ cranelift/src/libcretonne/ir/types.rs | 12 ++ meta/cretonne/typevar.py | 175 +++++++++++++++---- meta/gen_instr.py | 10 +- 4 files changed, 240 insertions(+), 93 deletions(-) diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index bfc1fe65f3..afd6ba319f 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -480,13 +480,14 @@ impl OpcodeConstraints { /// A value type set describes the permitted set of types for a type variable. #[derive(Clone, Copy)] pub struct ValueTypeSet { - allow_scalars: bool, - allow_simd: bool, - - base: Type, - all_ints: bool, - all_floats: bool, - all_bools: bool, + min_lanes: u8, + max_lanes: u8, + min_int: u8, + max_int: u8, + min_float: u8, + max_float: u8, + min_bool: u8, + max_bool: u8, } impl ValueTypeSet { @@ -494,42 +495,38 @@ impl ValueTypeSet { /// /// Note that the base type set does not have to be included in the type set proper. fn is_base_type(&self, scalar: Type) -> bool { - scalar == self.base || (self.all_ints && scalar.is_int()) || - (self.all_floats && scalar.is_float()) || (self.all_bools && scalar.is_bool()) + let l2b = scalar.log2_lane_bits(); + if scalar.is_int() { + self.min_int <= l2b && l2b < self.max_int + } else if scalar.is_float() { + self.min_float <= l2b && l2b < self.max_float + } else if scalar.is_bool() { + self.min_bool <= l2b && l2b < self.max_bool + } else { + false + } } /// Does `typ` belong to this set? pub fn contains(&self, typ: Type) -> bool { - let allowed = if typ.is_scalar() { - self.allow_scalars - } else { - self.allow_simd - }; - allowed && self.is_base_type(typ.lane_type()) + let l2l = typ.log2_lane_count(); + self.min_lanes <= l2l && l2l < self.max_lanes && self.is_base_type(typ.lane_type()) } /// Get an example member of this type set. /// /// This is used for error messages to avoid suggesting invalid types. pub fn example(&self) -> Type { - if self.base != types::VOID { - return self.base; - } - let t = if self.all_ints { + let t = if self.max_int > 5 { types::I32 - } else if self.all_floats { + } else if self.max_float > 5 { types::F32 - } else if self.allow_scalars { - types::B1 - } else { + } else if self.max_bool > 5 { types::B32 - }; - - if self.allow_scalars { - t } else { - t.by(4).unwrap() - } + types::B1 + }; + t.by(1 << self.min_lanes).unwrap() } } @@ -611,43 +608,74 @@ mod tests { use ir::types::*; let vts = ValueTypeSet { - allow_scalars: true, - allow_simd: true, - base: VOID, - all_ints: true, - all_floats: false, - all_bools: true, + min_lanes: 0, + max_lanes: 8, + min_int: 3, + max_int: 7, + min_float: 0, + max_float: 0, + min_bool: 3, + max_bool: 7, }; + assert!(vts.contains(I32)); + assert!(vts.contains(I64)); + assert!(vts.contains(I32X4)); + assert!(!vts.contains(F32)); + assert!(!vts.contains(B1)); + assert!(vts.contains(B8)); + assert!(vts.contains(B64)); assert_eq!(vts.example().to_string(), "i32"); let vts = ValueTypeSet { - allow_scalars: true, - allow_simd: true, - base: VOID, - all_ints: false, - all_floats: true, - all_bools: true, + min_lanes: 0, + max_lanes: 8, + min_int: 0, + max_int: 0, + min_float: 5, + max_float: 7, + min_bool: 3, + max_bool: 7, }; assert_eq!(vts.example().to_string(), "f32"); let vts = ValueTypeSet { - allow_scalars: false, - allow_simd: true, - base: VOID, - all_ints: false, - all_floats: true, - all_bools: true, + min_lanes: 1, + max_lanes: 8, + min_int: 0, + max_int: 0, + min_float: 5, + max_float: 7, + min_bool: 3, + max_bool: 7, }; - assert_eq!(vts.example().to_string(), "f32x4"); + assert_eq!(vts.example().to_string(), "f32x2"); let vts = ValueTypeSet { - allow_scalars: false, - allow_simd: true, - base: VOID, - all_ints: false, - all_floats: false, - all_bools: true, + min_lanes: 2, + max_lanes: 8, + min_int: 0, + max_int: 0, + min_float: 0, + max_float: 0, + min_bool: 3, + max_bool: 7, }; + assert!(!vts.contains(B32X2)); + assert!(vts.contains(B32X4)); assert_eq!(vts.example().to_string(), "b32x4"); + + let vts = ValueTypeSet { + // TypeSet(lanes=(1, 256), ints=(8, 64)) + min_lanes: 0, + max_lanes: 9, + min_int: 3, + max_int: 7, + min_float: 0, + max_float: 0, + min_bool: 0, + max_bool: 0, + }; + assert!(vts.contains(I32)); + assert!(vts.contains(I32X4)); } } diff --git a/cranelift/src/libcretonne/ir/types.rs b/cranelift/src/libcretonne/ir/types.rs index f4e0a6b75f..1eb546bde6 100644 --- a/cranelift/src/libcretonne/ir/types.rs +++ b/cranelift/src/libcretonne/ir/types.rs @@ -43,6 +43,18 @@ impl Type { Type(self.0 & 0x0f) } + /// Get log2 of the number of bits in a lane. + pub fn log2_lane_bits(self) -> u8 { + match self.lane_type() { + B1 => 0, + B8 | I8 => 3, + B16 | I16 => 4, + B32 | I32 | F32 => 5, + B64 | I64 | F64 => 6, + _ => 0, + } + } + /// Get the number of bits in a lane. pub fn lane_bits(self) -> u8 { match self.lane_type() { diff --git a/meta/cretonne/typevar.py b/meta/cretonne/typevar.py index ec2693829b..7ee7182ad7 100644 --- a/meta/cretonne/typevar.py +++ b/meta/cretonne/typevar.py @@ -5,20 +5,127 @@ Cretonne instructions and instruction transformations can be specified to be polymorphic by using type variables. """ from __future__ import absolute_import -from collections import namedtuple +import math from . import value -#: A `TypeSet` represents a set of types. We don't allow arbitrary subsets of -#: types, but use a parametrized approach instead. -#: This is represented as a named tuple so it can be used as a dictionary key. -TypeSet = namedtuple( - 'TypeSet', [ - 'allow_scalars', - 'allow_simd', - 'base', - 'all_ints', - 'all_floats', - 'all_bools']) + +MAX_LANES = 256 +MAX_BITS = 64 + + +def is_power_of_two(x): + return x > 0 and x & (x-1) == 0 + + +def int_log2(x): + return int(math.log(x, 2)) + + +class TypeSet(object): + """ + A set of types. + + We don't allow arbitrary subsets of types, but use a parametrized approach + instead. + + Objects of this class can be used as dictionary keys. + + Parametrized type sets are specified in terms of ranges: + + - The permitted range of vector lanes, where 1 indicates a scalar type. + - The permitted range of integer types. + - The permitted range of floating point types, and + - The permitted range of boolean types. + + The ranges are inclusive from smallest bit-width to largest bit-width. + + :param lanes: `(min, max)` inclusive range of permitted vector lane counts. + :param ints: `(min, max)` inclusive range of permitted scalar integer + widths. + :param floats: `(min, max)` inclusive range of permitted scalar floating + point widths. + :param bools: `(min, max)` inclusive range of permitted scalar boolean + widths. + """ + + def __init__(self, lanes, ints=None, floats=None, bools=None): + self.min_lanes, self.max_lanes = lanes + assert is_power_of_two(self.min_lanes) + assert is_power_of_two(self.max_lanes) + assert self.max_lanes <= MAX_LANES + + if ints: + if ints is True: + ints = (8, MAX_BITS) + self.min_int, self.max_int = ints + assert is_power_of_two(self.min_int) + assert is_power_of_two(self.max_int) + assert self.max_int <= MAX_BITS + else: + self.min_int = None + self.max_int = None + + if floats: + if floats is True: + floats = (32, 64) + self.min_float, self.max_float = floats + assert is_power_of_two(self.min_float) + assert self.min_float >= 32 + assert is_power_of_two(self.max_float) + assert self.max_float <= 64 + else: + self.min_float = None + self.max_float = None + + if bools: + if bools is True: + bools = (1, MAX_BITS) + self.min_bool, self.max_bool = bools + assert is_power_of_two(self.min_bool) + assert is_power_of_two(self.max_bool) + assert self.max_bool <= MAX_BITS + else: + self.min_bool = None + self.max_bool = None + + def typeset_key(self): + """Key tuple used for hashing and equality.""" + return (self.min_lanes, self.max_lanes, + self.min_int, self.max_int, + self.min_float, self.max_float, + self.min_bool, self.max_bool) + + def __hash__(self): + return hash(self.typeset_key()) + + def __eq__(self, other): + return self.typeset_key() == other.typeset_key() + + def __repr__(self): + s = 'TypeSet(lanes=({}, {})'.format(self.min_lanes, self.max_lanes) + if self.min_int is not None: + s += ', ints=({}, {})'.format(self.min_int, self.max_int) + if self.min_float is not None: + s += ', floats=({}, {})'.format(self.min_float, self.max_float) + if self.min_bool is not None: + s += ', bools=({}, {})'.format(self.min_bool, self.max_bool) + return s + ')' + + def emit_fields(self, fmt): + """Emit field initializers for this typeset.""" + fmt.comment(repr(self)) + fields = ('lanes', 'int', 'float', 'bool') + for field in fields: + min_val = getattr(self, 'min_' + field) + max_val = getattr(self, 'max_' + field) + if min_val is None: + fmt.line('min_{}: 0,'.format(field)) + fmt.line('max_{}: 0,'.format(field)) + else: + fmt.line('min_{}: {},'.format( + field, int_log2(min_val))) + fmt.line('max_{}: {},'.format( + field, int_log2(max_val) + 1)) class TypeVar(object): @@ -33,37 +140,45 @@ class TypeVar(object): :param name: Short name of type variable used in instruction descriptions. :param doc: Documentation string. - :param base: Single base type or list of base types. Use this to specify an - exact set of base types if the general categories below are not good - enough. - :param ints: Allow all integer base types. - :param floats: Allow all floating point base types. - :param bools: Allow all boolean base types. + :param ints: Allow all integer base types, or `(min, max)` bit-range. + :param floats: Allow all floating point base types, or `(min, max)` + bit-range. + :param bools: Allow all boolean base types, or `(min, max)` bit-range. :param scalars: Allow type variable to assume scalar types. - :param simd: Allow type variable to assume vector types. + :param simd: Allow type variable to assume vector types, or `(min, max)` + lane count range. """ def __init__( - self, name, doc, base=None, + self, name, doc, ints=False, floats=False, bools=False, scalars=True, simd=False, - derived_func=None): + base=None, derived_func=None): self.name = name self.__doc__ = doc - self.base = base self.is_derived = isinstance(base, TypeVar) - if self.is_derived: + if base: + assert self.is_derived assert derived_func + self.base = base self.derived_func = derived_func self.name = '{}({})'.format(derived_func, base.name) else: + min_lanes = 1 if scalars else 2 + if simd: + if simd is True: + max_lanes = MAX_LANES + else: + min_lanes, max_lanes = simd + assert not scalars or min_lanes <= 2 + else: + max_lanes = 1 + self.type_set = TypeSet( - allow_scalars=scalars, - allow_simd=simd, - base=base, - all_ints=ints, - all_floats=floats, - all_bools=bools) + lanes=(min_lanes, max_lanes), + ints=ints, + floats=floats, + bools=bools) def __str__(self): return "`{}`".format(self.name) @@ -92,7 +207,7 @@ class TypeVar(object): return value def free_typevar(self): - if isinstance(self.base, TypeVar): + if self.is_derived: return self.base else: return self diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 6c67077cb0..cc636cf83a 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -344,15 +344,7 @@ def gen_type_constraints(fmt, instrs): .format(len(type_sets.table)), '];'): for ts in type_sets.table: with fmt.indented('ValueTypeSet {', '},'): - if ts.base: - fmt.line('base: {},'.format(ts.base.rust_name())) - else: - fmt.line('base: types::VOID,') - for field in ts._fields: - if field == 'base': - continue - fmt.line('{}: {},'.format( - field, str(getattr(ts, field)).lower())) + ts.emit_fields(fmt) fmt.comment('Table of operand constraint sequences.') with fmt.indented( From efa1b8a9eccaf933ceccf4190bef69d05d46be12 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 27 Sep 2016 13:45:05 -0700 Subject: [PATCH 0333/3084] Run Python unittests and doctests. Add a meta/check.sh script that runs unit tests before the syntax linters. Add unittest converters for the existing doctests. --- meta/{check-py3k.sh => check.sh} | 6 +++++- meta/constant_hash.py | 4 ---- meta/srcgen.py | 4 ---- meta/test_constant_hash.py | 8 ++++++++ meta/test_srcgen.py | 8 ++++++++ 5 files changed, 21 insertions(+), 9 deletions(-) rename meta/{check-py3k.sh => check.sh} (78%) create mode 100644 meta/test_constant_hash.py create mode 100644 meta/test_srcgen.py diff --git a/meta/check-py3k.sh b/meta/check.sh similarity index 78% rename from meta/check-py3k.sh rename to meta/check.sh index 3ac939ea15..f01653b57e 100755 --- a/meta/check-py3k.sh +++ b/meta/check.sh @@ -1,8 +1,12 @@ #!/bin/bash +set -e +cd $(dirname "$0") + +# Run unit tests. +python -m unittest discover # Check Python sources for Python 3 compatibility using pylint. # # Install pylint with 'pip install pylint'. -cd $(dirname "$0") pylint --py3k --reports=no -- *.py cretonne isa flake8 . diff --git a/meta/constant_hash.py b/meta/constant_hash.py index aa43694a50..38911278ed 100644 --- a/meta/constant_hash.py +++ b/meta/constant_hash.py @@ -74,7 +74,3 @@ def compute_quadratic(items, hash_function): table[h] = i return table - -if __name__ == "__main__": - import doctest - doctest.testmod() diff --git a/meta/srcgen.py b/meta/srcgen.py index 0b8a7a2973..962a881436 100644 --- a/meta/srcgen.py +++ b/meta/srcgen.py @@ -121,7 +121,3 @@ class Formatter(object): """Add a (multi-line) documentation comment.""" s = re.sub('^', self.indent + '/// ', s, flags=re.M) + '\n' self.lines.append(s) - -if __name__ == "__main__": - import doctest - doctest.testmod() diff --git a/meta/test_constant_hash.py b/meta/test_constant_hash.py new file mode 100644 index 0000000000..e76f09aed9 --- /dev/null +++ b/meta/test_constant_hash.py @@ -0,0 +1,8 @@ +from __future__ import absolute_import +import doctest +import constant_hash + + +def load_tests(loader, tests, ignore): + tests.addTests(doctest.DocTestSuite(constant_hash)) + return tests diff --git a/meta/test_srcgen.py b/meta/test_srcgen.py new file mode 100644 index 0000000000..2fb5e0fb61 --- /dev/null +++ b/meta/test_srcgen.py @@ -0,0 +1,8 @@ +from __future__ import absolute_import +import doctest +import srcgen + + +def load_tests(loader, tests, ignore): + tests.addTests(doctest.DocTestSuite(srcgen)) + return tests From f66a6b350910cf4fdb7e5bc3a021126f79ac5fbf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 27 Sep 2016 14:36:02 -0700 Subject: [PATCH 0334/3084] Add some Python tests for TypeSet. --- meta/cretonne/test_typevar.py | 45 +++++++++++++++++++++++++++++++++++ meta/cretonne/typevar.py | 45 ++++++++++++++++++++++++++++++----- 2 files changed, 84 insertions(+), 6 deletions(-) create mode 100644 meta/cretonne/test_typevar.py diff --git a/meta/cretonne/test_typevar.py b/meta/cretonne/test_typevar.py new file mode 100644 index 0000000000..128b32b71f --- /dev/null +++ b/meta/cretonne/test_typevar.py @@ -0,0 +1,45 @@ +from __future__ import absolute_import +from unittest import TestCase +from doctest import DocTestSuite +from . import typevar +from .typevar import TypeSet + + +def load_tests(loader, tests, ignore): + tests.addTests(DocTestSuite(typevar)) + return tests + + +class TestTypeSet(TestCase): + def test_invalid(self): + with self.assertRaises(AssertionError): + TypeSet(lanes=(2, 1)) + with self.assertRaises(AssertionError): + TypeSet(ints=(32, 16)) + with self.assertRaises(AssertionError): + TypeSet(floats=(32, 16)) + with self.assertRaises(AssertionError): + TypeSet(bools=(32, 16)) + with self.assertRaises(AssertionError): + TypeSet(ints=(32, 33)) + + def test_hash(self): + a = TypeSet(lanes=True, ints=True, floats=True) + b = TypeSet(lanes=True, ints=True, floats=True) + c = TypeSet(lanes=True, ints=(8, 16), floats=True) + self.assertEqual(a, b) + self.assertNotEqual(a, c) + s = set() + s.add(a) + self.assertTrue(a in s) + self.assertTrue(b in s) + self.assertFalse(c in s) + + def test_hash_modified(self): + a = TypeSet(lanes=True, ints=True, floats=True) + s = set() + s.add(a) + a.max_int = 32 + # Can't rehash after modification. + with self.assertRaises(AssertionError): + a in s diff --git a/meta/cretonne/typevar.py b/meta/cretonne/typevar.py index 7ee7182ad7..e91c6ab002 100644 --- a/meta/cretonne/typevar.py +++ b/meta/cretonne/typevar.py @@ -39,6 +39,26 @@ class TypeSet(object): The ranges are inclusive from smallest bit-width to largest bit-width. + A typeset representing scalar integer types `i8` through `i32`: + + >>> TypeSet(ints=(8, 32)) + TypeSet(lanes=(1, 1), ints=(8, 32)) + + Passing `True` instead of a range selects all available scalar types: + + >>> TypeSet(ints=True) + TypeSet(lanes=(1, 1), ints=(8, 64)) + >>> TypeSet(floats=True) + TypeSet(lanes=(1, 1), floats=(32, 64)) + >>> TypeSet(bools=True) + TypeSet(lanes=(1, 1), bools=(1, 64)) + + Similarly, passing `True` for the lanes selects all possible scalar and + vector types: + + >>> TypeSet(lanes=True, ints=True) + TypeSet(lanes=(1, 256), ints=(8, 64)) + :param lanes: `(min, max)` inclusive range of permitted vector lane counts. :param ints: `(min, max)` inclusive range of permitted scalar integer widths. @@ -48,11 +68,18 @@ class TypeSet(object): widths. """ - def __init__(self, lanes, ints=None, floats=None, bools=None): - self.min_lanes, self.max_lanes = lanes - assert is_power_of_two(self.min_lanes) - assert is_power_of_two(self.max_lanes) - assert self.max_lanes <= MAX_LANES + def __init__(self, lanes=None, ints=None, floats=None, bools=None): + if lanes: + if lanes is True: + lanes = (1, MAX_LANES) + self.min_lanes, self.max_lanes = lanes + assert is_power_of_two(self.min_lanes) + assert is_power_of_two(self.max_lanes) + assert self.max_lanes <= MAX_LANES + else: + self.min_lanes = 1 + self.max_lanes = 1 + assert self.min_lanes <= self.max_lanes if ints: if ints is True: @@ -61,6 +88,7 @@ class TypeSet(object): assert is_power_of_two(self.min_int) assert is_power_of_two(self.max_int) assert self.max_int <= MAX_BITS + assert self.min_int <= self.max_int else: self.min_int = None self.max_int = None @@ -73,6 +101,7 @@ class TypeSet(object): assert self.min_float >= 32 assert is_power_of_two(self.max_float) assert self.max_float <= 64 + assert self.min_float <= self.max_float else: self.min_float = None self.max_float = None @@ -84,6 +113,7 @@ class TypeSet(object): assert is_power_of_two(self.min_bool) assert is_power_of_two(self.max_bool) assert self.max_bool <= MAX_BITS + assert self.min_bool <= self.max_bool else: self.min_bool = None self.max_bool = None @@ -96,7 +126,10 @@ class TypeSet(object): self.min_bool, self.max_bool) def __hash__(self): - return hash(self.typeset_key()) + h = hash(self.typeset_key()) + assert h == getattr(self, 'prev_hash', h), "TypeSet changed!" + self.prev_hash = h + return h def __eq__(self, other): return self.typeset_key() == other.typeset_key() From a14bb077eece33c9e0540142d9ed628be52cd020 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 27 Sep 2016 14:53:37 -0700 Subject: [PATCH 0335/3084] In-place intersection of type sets. --- meta/cretonne/typevar.py | 41 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/meta/cretonne/typevar.py b/meta/cretonne/typevar.py index e91c6ab002..918933b7c6 100644 --- a/meta/cretonne/typevar.py +++ b/meta/cretonne/typevar.py @@ -160,6 +160,47 @@ class TypeSet(object): fmt.line('max_{}: {},'.format( field, int_log2(max_val) + 1)) + def __iand__(self, other): + """ + Intersect self with other type set. + + >>> a = TypeSet(lanes=True, ints=(16, 32)) + >>> a + TypeSet(lanes=(1, 256), ints=(16, 32)) + >>> b = TypeSet(lanes=(4, 16), ints=True) + >>> a &= b + >>> a + TypeSet(lanes=(4, 16), ints=(16, 32)) + + >>> a = TypeSet(lanes=True, bools=(1, 8)) + >>> b = TypeSet(lanes=True, bools=(16, 32)) + >>> a &= b + >>> a + TypeSet(lanes=(1, 256)) + """ + self.min_lanes = max(self.min_lanes, other.min_lanes) + self.max_lanes = min(self.max_lanes, other.max_lanes) + + self.min_int = max(self.min_int, other.min_int) + self.max_int = min(self.max_int, other.max_int) + if self.min_int > self.max_int: + self.min_int = None + self.max_int = None + + self.min_float = max(self.min_float, other.min_float) + self.max_float = min(self.max_float, other.max_float) + if self.min_float > self.max_float: + self.min_float = None + self.max_float = None + + self.min_bool = max(self.min_bool, other.min_bool) + self.max_bool = min(self.max_bool, other.max_bool) + if self.min_bool > self.max_bool: + self.min_bool = None + self.max_bool = None + + return self + class TypeVar(object): """ From d256c46f6005cf35dda545a16a9b972755cafbf6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 27 Sep 2016 15:39:54 -0700 Subject: [PATCH 0336/3084] Add HalfWidth and DoubleWidth type variable functions. These functions compute types with half or double the number of bits in each lane. --- cranelift/src/libcretonne/ir/instructions.rs | 8 +++ cranelift/src/libcretonne/ir/types.rs | 65 +++++++++++++++++++- meta/cretonne/test_typevar.py | 21 ++++++- meta/cretonne/typevar.py | 30 +++++++++ 4 files changed, 122 insertions(+), 2 deletions(-) diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index afd6ba319f..f60d23c60c 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -547,6 +547,12 @@ enum OperandConstraint { /// This operand is `ctrlType.as_bool()`. AsBool, + + /// This operand is `ctrlType.half_width()`. + HalfWidth, + + /// This operand is `ctrlType.double_width()`. + DoubleWidth, } impl OperandConstraint { @@ -562,6 +568,8 @@ impl OperandConstraint { Same => Some(ctrl_type), LaneOf => Some(ctrl_type.lane_type()), AsBool => Some(ctrl_type.as_bool()), + HalfWidth => Some(ctrl_type.half_width().expect("invalid type for half_width")), + DoubleWidth => Some(ctrl_type.double_width().expect("invalid type for double_width")), } } } diff --git a/cranelift/src/libcretonne/ir/types.rs b/cranelift/src/libcretonne/ir/types.rs index 1eb546bde6..7c9175f460 100644 --- a/cranelift/src/libcretonne/ir/types.rs +++ b/cranelift/src/libcretonne/ir/types.rs @@ -67,7 +67,7 @@ impl Type { } } - /// Get a type with the same number of lanes as this type, but with the lanes replaces by + /// Get a type with the same number of lanes as this type, but with the lanes replaced by /// booleans of the same size. pub fn as_bool(self) -> Type { // Replace the low 4 bits with the boolean version, preserve the high 4 bits. @@ -81,6 +81,38 @@ impl Type { Type(lane.0 | (self.0 & 0xf0)) } + /// Get a type with the same number of lanes as this type, but with lanes that are half the + /// number of bits. + pub fn half_width(self) -> Option { + let lane = match self.lane_type() { + I16 => I8, + I32 => I16, + I64 => I32, + F64 => F32, + B16 => B8, + B32 => B16, + B64 => B32, + _ => return None, + }; + Some(Type(lane.0 | (self.0 & 0xf0))) + } + + /// Get a type with the same number of lanes as this type, but with lanes that are twice the + /// number of bits. + pub fn double_width(self) -> Option { + let lane = match self.lane_type() { + I8 => I16, + I16 => I32, + I32 => I64, + F32 => F64, + B8 => B16, + B16 => B32, + B32 => B64, + _ => return None, + }; + Some(Type(lane.0 | (self.0 & 0xf0))) + } + /// Is this the VOID type? pub fn is_void(self) -> bool { self == VOID @@ -333,6 +365,37 @@ mod tests { assert_eq!(F64.lane_bits(), 64); } + #[test] + fn typevar_functions() { + assert_eq!(VOID.half_width(), None); + assert_eq!(B1.half_width(), None); + assert_eq!(B8.half_width(), None); + assert_eq!(B16.half_width(), Some(B8)); + assert_eq!(B32.half_width(), Some(B16)); + assert_eq!(B64.half_width(), Some(B32)); + assert_eq!(I8.half_width(), None); + assert_eq!(I16.half_width(), Some(I8)); + assert_eq!(I32.half_width(), Some(I16)); + assert_eq!(I32X4.half_width(), Some(I16X4)); + assert_eq!(I64.half_width(), Some(I32)); + assert_eq!(F32.half_width(), None); + assert_eq!(F64.half_width(), Some(F32)); + + assert_eq!(VOID.double_width(), None); + assert_eq!(B1.double_width(), None); + assert_eq!(B8.double_width(), Some(B16)); + assert_eq!(B16.double_width(), Some(B32)); + assert_eq!(B32.double_width(), Some(B64)); + assert_eq!(B64.double_width(), None); + assert_eq!(I8.double_width(), Some(I16)); + assert_eq!(I16.double_width(), Some(I32)); + assert_eq!(I32.double_width(), Some(I64)); + assert_eq!(I32X4.double_width(), Some(I64X4)); + assert_eq!(I64.double_width(), None); + assert_eq!(F32.double_width(), Some(F64)); + assert_eq!(F64.double_width(), None); + } + #[test] fn vectors() { let big = F64.by(256).unwrap(); diff --git a/meta/cretonne/test_typevar.py b/meta/cretonne/test_typevar.py index 128b32b71f..a841acfdb9 100644 --- a/meta/cretonne/test_typevar.py +++ b/meta/cretonne/test_typevar.py @@ -2,7 +2,7 @@ from __future__ import absolute_import from unittest import TestCase from doctest import DocTestSuite from . import typevar -from .typevar import TypeSet +from .typevar import TypeSet, TypeVar def load_tests(loader, tests, ignore): @@ -43,3 +43,22 @@ class TestTypeSet(TestCase): # Can't rehash after modification. with self.assertRaises(AssertionError): a in s + + +class TestTypeVar(TestCase): + def test_functions(self): + x = TypeVar('x', 'all ints', ints=True) + with self.assertRaises(AssertionError): + x.double_width() + with self.assertRaises(AssertionError): + x.half_width() + + x2 = TypeVar('x2', 'i16 and up', ints=(16, 64)) + with self.assertRaises(AssertionError): + x2.double_width() + self.assertEqual(str(x2.half_width()), '`HalfWidth(x2)`') + + x3 = TypeVar('x3', 'up to i32', ints=(8, 32)) + self.assertEqual(str(x3.double_width()), '`DoubleWidth(x3)`') + with self.assertRaises(AssertionError): + x3.half_width() diff --git a/meta/cretonne/typevar.py b/meta/cretonne/typevar.py index 918933b7c6..a29548af38 100644 --- a/meta/cretonne/typevar.py +++ b/meta/cretonne/typevar.py @@ -274,6 +274,36 @@ class TypeVar(object): """ return TypeVar(None, None, base=self, derived_func='AsBool') + def half_width(self): + """ + Return a derived type variable that has the same number of vector lanes + as this one, but the lanes are half the width. + """ + ts = self.type_set + if ts.min_int: + assert ts.min_int > 8, "Can't halve all integer types" + if ts.min_float: + assert ts.min_float > 32, "Can't halve all float types" + if ts.min_bool: + assert ts.min_bool > 8, "Can't halve all boolean types" + + return TypeVar(None, None, base=self, derived_func='HalfWidth') + + def double_width(self): + """ + Return a derived type variable that has the same number of vector lanes + as this one, but the lanes are double the width. + """ + ts = self.type_set + if ts.max_int: + assert ts.max_int < MAX_BITS, "Can't double all integer types." + if ts.max_float: + assert ts.max_float < MAX_BITS, "Can't double all float types." + if ts.max_bool: + assert ts.max_bool < MAX_BITS, "Can't double all boolean types." + + return TypeVar(None, None, base=self, derived_func='DoubleWidth') + def operand_kind(self): # When a `TypeVar` object is used to describe the type of an `Operand` # in an instruction definition, the kind of that operand is an SSA From 2a4aaa3da161dca0e38b599c860c6f9f431b9d58 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 27 Sep 2016 16:09:26 -0700 Subject: [PATCH 0337/3084] Expand OpcodeConstraints to 32 bits. Make room for 255 different type sets and 2^16 entries in the operand constraints table. --- cranelift/src/libcretonne/ir/instructions.rs | 57 ++++++++++---------- meta/gen_instr.py | 15 +++--- 2 files changed, 36 insertions(+), 36 deletions(-) diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index f60d23c60c..9e259273cc 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -398,52 +398,49 @@ pub enum BranchInfo<'a> { /// The `InstructionFormat` determines the constraints on most operands, but `Value` operands and /// results are not determined by the format. Every `Opcode` has an associated /// `OpcodeConstraints` object that provides the missing details. -/// -/// Since there can be a lot of opcodes, the `OpcodeConstraints` object is encoded as a bit field -/// by the `meta/gen_instr.py` script. -/// -/// The bit field bits are: -/// -/// Bits 0-2: -/// Number of fixed result values. This does not include `variable_args` results as are -/// produced by call instructions. -/// -/// Bit 3: -/// This opcode is polymorphic and the controlling type variable can be inferred from the -/// designated input operand. This is the `typevar_operand` index given to the -/// `InstructionFormat` meta language object. When bit 0 is not set, the controlling type -/// variable must be the first output value instead. -/// -/// Bits 4-7: -/// Permitted set of types for the controlling type variable as an index into `TYPE_SETS`. -/// -/// Bits 8-15: -/// Offset into `OPERAND_CONSTRAINT` table of the descriptors for this opcode. The first -/// `fixed_results()` entries describe the result constraints, then follows constraints for the -/// fixed `Value` input operands. The number of `Value` inputs isdetermined by the instruction -/// format. -/// #[derive(Clone, Copy)] -pub struct OpcodeConstraints(u16); +pub struct OpcodeConstraints { + /// Flags for this opcode encoded as a bit field: + /// + /// Bits 0-2: + /// Number of fixed result values. This does not include `variable_args` results as are + /// produced by call instructions. + /// + /// Bit 3: + /// This opcode is polymorphic and the controlling type variable can be inferred from the + /// designated input operand. This is the `typevar_operand` index given to the + /// `InstructionFormat` meta language object. When bit 0 is not set, the controlling type + /// variable must be the first output value instead. + flags: u8, + + /// Permitted set of types for the controlling type variable as an index into `TYPE_SETS`. + typeset_offset: u8, + + /// Offset into `OPERAND_CONSTRAINT` table of the descriptors for this opcode. The first + /// `fixed_results()` entries describe the result constraints, then follows constraints for the + /// fixed `Value` input operands. The number of `Value` inputs is determined by the instruction + /// format. + constraint_offset: u16, +} impl OpcodeConstraints { /// Can the controlling type variable for this opcode be inferred from the designated value /// input operand? /// This also implies that this opcode is polymorphic. pub fn use_typevar_operand(self) -> bool { - (self.0 & 0x8) != 0 + (self.flags & 0x8) != 0 } /// Get the number of *fixed* result values produced by this opcode. /// This does not include `variable_args` produced by calls. pub fn fixed_results(self) -> usize { - (self.0 & 0x7) as usize + (self.flags & 0x7) as usize } /// Get the offset into `TYPE_SETS` for the controlling type variable. /// Returns `None` if the instruction is not polymorphic. fn typeset_offset(self) -> Option { - let offset = ((self.0 & 0xff) >> 4) as usize; + let offset = self.typeset_offset as usize; if offset < TYPE_SETS.len() { Some(offset) } else { @@ -453,7 +450,7 @@ impl OpcodeConstraints { /// Get the offset into OPERAND_CONSTRAINTS where the descriptors for this opcode begin. fn constraint_offset(self) -> usize { - (self.0 >> 8) as usize + self.constraint_offset as usize } /// Get the value type of result number `n`, having resolved the controlling type variable to diff --git a/meta/gen_instr.py b/meta/gen_instr.py index cc636cf83a..582c0477f0 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -297,8 +297,8 @@ def gen_type_constraints(fmt, instrs): # Preload table with constraints for typical binops. operand_seqs.add(['Same'] * 3) - # TypeSet indexes are encoded in 3 bits, with `111` reserved. - typeset_limit = 7 + # TypeSet indexes are encoded in 8 bits, with `0xff` reserved. + typeset_limit = 0xff fmt.comment('Table of opcode constraints.') with fmt.indented( @@ -331,11 +331,14 @@ def gen_type_constraints(fmt, instrs): 'Polymorphic over {}'.format(ctrl_typevar.type_set)) # Compute the bit field encoding, c.f. instructions.rs. assert fixed_results < 8, "Bit field encoding too tight" - bits = (offset << 8) | (ctrl_typeset << 4) | fixed_results + flags = fixed_results if use_typevar_operand: - bits |= 8 - assert bits < 0x10000, "Constraint table too large for bit field" - fmt.line('OpcodeConstraints({:#06x}),'.format(bits)) + flags |= 8 + + with fmt.indented('OpcodeConstraints {', '},'): + fmt.line('flags: {:#04x},'.format(flags)) + fmt.line('typeset_offset: {},'.format(ctrl_typeset)) + fmt.line('constraint_offset: {},'.format(offset)) fmt.comment('Table of value type sets.') assert len(type_sets.table) <= typeset_limit, "Too many type sets" From 4db11d1ae79f4994e7f1361837617d79d4b731d2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 27 Sep 2016 16:22:32 -0700 Subject: [PATCH 0338/3084] Add legalization helper instructions. The isplit_lohi instruction breaks an integer into two halves. This will typically be used to get the two halves of an `i64` value on 32-bit CPUs. The iconcat_lohi is the reverse operation. It reconstructs the `i64` from the low and high bits. --- cranelift/docs/langref.rst | 8 +++++ cranelift/src/libcretonne/ir/instructions.rs | 6 ++++ cranelift/src/libcretonne/write.rs | 1 + cranelift/src/libreader/parser.rs | 9 +++++ meta/cretonne/base.py | 38 ++++++++++++++++++++ meta/cretonne/formats.py | 1 + 6 files changed, 63 insertions(+) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index e42d82ecf4..e4d2d9dbd3 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -823,6 +823,14 @@ Conversion operations .. autoinst:: fcvt_from_uint .. autoinst:: fcvt_from_sint +Legalization operations +----------------------- + +These instructions are used as helpers when legalizing types and operations for +the target ISA. + +.. autoinst:: isplit_lohi +.. autoinst:: iconcat_lohi Base instruction group ====================== diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index 9e259273cc..fd7fd5f25b 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -120,6 +120,12 @@ pub enum InstructionData { opcode: Opcode, ty: Type, // TBD: imm: Box }, + UnarySplit { + opcode: Opcode, + ty: Type, + second_result: Value, + arg: Value, + }, Binary { opcode: Opcode, ty: Type, diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index ea73d18666..8f7acde772 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -185,6 +185,7 @@ fn write_instruction(w: &mut Write, UnaryIeee32 { imm, .. } => writeln!(w, " {}", imm), UnaryIeee64 { imm, .. } => writeln!(w, " {}", imm), UnaryImmVector { .. } => writeln!(w, " [...]"), + UnarySplit { arg, .. } => writeln!(w, " {}", arg), Binary { args, .. } => writeln!(w, " {}, {}", args[0], args[1]), BinaryImm { arg, imm, .. } => writeln!(w, " {}, {}", arg, imm), BinaryImmRev { imm, arg, .. } => writeln!(w, " {}, {}", imm, arg), diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 723cfb1951..258f920b81 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -119,6 +119,7 @@ impl Context { InstructionData::UnaryImmVector { .. } => {} InstructionData::Unary { ref mut arg, .. } | + InstructionData::UnarySplit { ref mut arg, .. } | InstructionData::BinaryImm { ref mut arg, .. } | InstructionData::BinaryImmRev { ref mut arg, .. } | InstructionData::ExtractLane { ref mut arg, .. } | @@ -1013,6 +1014,14 @@ impl<'a> Parser<'a> { InstructionFormat::UnaryImmVector => { unimplemented!(); } + InstructionFormat::UnarySplit => { + InstructionData::UnarySplit { + opcode: opcode, + ty: VOID, + second_result: NO_VALUE, + arg: try!(self.match_value("expected SSA value operand")), + } + } InstructionFormat::Binary => { let lhs = try!(self.match_value("expected SSA value first operand")); try!(self.match_token(Token::Comma, "expected ',' between operands")); diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index fdca2484f9..fc8de5d78f 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -1105,4 +1105,42 @@ fcvt_from_sint = Instruction( """, ins=x, outs=a) +# +# Legalization helper instructions. +# + +WideInt = TypeVar( + 'WideInt', 'A scalar integer type from `i16` upwards', + ints=(16, 64)) +x = Operand('x', WideInt) +lo = Operand( + 'lo', WideInt.half_width(), 'The low bits of `x`') +hi = Operand( + 'hi', WideInt.half_width(), 'The high bits of `x`') + +isplit_lohi = Instruction( + 'isplit_lohi', r""" + Split a scalar integer into low and high parts. + + Returns the low half of `x` and the high half of `x` as two independent + values. + """, + ins=x, outs=(lo, hi)) + + +NarrowInt = TypeVar( + 'NarrowInt', 'A scalar integer type up to `i32`', + ints=(8, 32)) +lo = Operand('lo', NarrowInt) +hi = Operand('hi', NarrowInt) +a = Operand( + 'a', NarrowInt.double_width(), + doc='The concatenation of `lo` and `hi`') + +iconcat_lohi = Instruction( + 'iconcat_lohi', r""" + Concatenate low and high bits to form a larger integer type. + """, + ins=(lo, hi), outs=a) + instructions.close() diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index 48b9d978af..df0019bff4 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -17,6 +17,7 @@ UnaryImm = InstructionFormat(imm64) UnaryIeee32 = InstructionFormat(ieee32) UnaryIeee64 = InstructionFormat(ieee64) UnaryImmVector = InstructionFormat(immvector) +UnarySplit = InstructionFormat(value, multiple_results=True) Binary = InstructionFormat(value, value) BinaryImm = InstructionFormat(value, imm64) From 620f46202f245710f3a6afc08881ef5ae32104e8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 22 Sep 2016 12:59:31 -0700 Subject: [PATCH 0339/3084] Define AST nodes and instruction transformations. Enable syntax: iadd(x, y) which creates an Apply node. Enable syntax: z << iadd(x, y) which creates a Def node. Add an XForm class which represents source and destination patterns as RTL lists. --- meta/cretonne/__init__.py | 26 +++++- meta/cretonne/ast.py | 122 ++++++++++++++++++++++++ meta/cretonne/test_ast.py | 28 ++++++ meta/cretonne/test_xform.py | 59 ++++++++++++ meta/cretonne/xform.py | 182 ++++++++++++++++++++++++++++++++++++ 5 files changed, 416 insertions(+), 1 deletion(-) create mode 100644 meta/cretonne/ast.py create mode 100644 meta/cretonne/test_ast.py create mode 100644 meta/cretonne/test_xform.py create mode 100644 meta/cretonne/xform.py diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index d318b27ef3..a5fe761bce 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -10,6 +10,7 @@ import math import importlib from collections import OrderedDict from .predicates import And +from .ast import Apply camel_re = re.compile('(^|_)([a-z])') @@ -603,6 +604,12 @@ class Operand(object): def __str__(self): return "`{}`".format(self.name) + def is_value(self): + """ + Is this an SSA value operand? + """ + return self.kind is value + class InstructionFormat(object): """ @@ -779,10 +786,13 @@ class Instruction(object): self.format = InstructionFormat.lookup(self.ins, self.outs) # Indexes into outs for value results. Others are `variable_args`. self.value_results = tuple( - i for i, o in enumerate(self.outs) if o.kind is value) + i for i, o in enumerate(self.outs) if o.is_value()) self._verify_polymorphic() InstructionGroup.append(self) + def __str__(self): + return self.name + def _verify_polymorphic(self): """ Check if this instruction is polymorphic, and verify its use of type @@ -910,6 +920,13 @@ class Instruction(object): assert not self.is_polymorphic, self return (self, ()) + def __call__(self, *args): + """ + Create an `ast.Apply` AST node representing the application of this + instruction to the arguments. + """ + return Apply(self, args) + class BoundInstruction(object): """ @@ -951,6 +968,13 @@ class BoundInstruction(object): assert len(self.typevars) == 1 + len(self.inst.other_typevars) return (self.inst, self.typevars) + def __call__(self, *args): + """ + Create an `ast.Apply` AST node representing the application of this + instruction to the arguments. + """ + return Apply(self, args) + # Defining target ISAs. diff --git a/meta/cretonne/ast.py b/meta/cretonne/ast.py new file mode 100644 index 0000000000..8c9c8576cb --- /dev/null +++ b/meta/cretonne/ast.py @@ -0,0 +1,122 @@ +""" +Abstract syntax trees. + +This module defines classes that can be used to create abstract syntax trees +for patern matching an rewriting of cretonne instructions. +""" +from __future__ import absolute_import + + +class Def(object): + """ + An AST definition associates a set of variables with the values produced by + an expression. + + Example: + + >>> from .base import iadd_cout, iconst + >>> x = Var('x') + >>> y = Var('y') + >>> x << iconst(4) + (Var(x),) << Apply(iconst, (4,)) + >>> (x, y) << iadd_cout(4, 5) + (Var(x), Var(y)) << Apply(iadd_cout, (4, 5)) + + The `<<` operator is used to create variable definitions. + + :param defs: Single variable or tuple of variables to be defined. + :param expr: Expression generating the values. + """ + + def __init__(self, defs, expr): + if not isinstance(defs, tuple): + defs = (defs,) + assert isinstance(expr, Expr) + self.defs = defs + self.expr = expr + + def __repr__(self): + return "{} << {!r}".format(self.defs, self.expr) + + def __str__(self): + if len(self.defs) == 1: + return "{!s} << {!s}".format(self.defs[0], self.expr) + else: + return "({}) << {!s}".format(", ".join(self.defs), self.expr) + + +class Expr(object): + """ + An AST expression. + """ + + def __rlshift__(self, other): + """ + Define variables using `var << expr` or `(v1, v2) << expr`. + """ + return Def(other, self) + + +class Var(Expr): + """ + A free variable. + """ + + def __init__(self, name): + self.name = name + # Bitmask of contexts where this variable is defined. + # See XForm._rewrite_defs(). + self.defctx = 0 + + def __str__(self): + return self.name + + def __repr__(self): + s = self.name + if self.defctx: + s += ", d={:02b}".format(self.defctx) + return "Var({})".format(s) + + +class Apply(Expr): + """ + Apply an instruction to arguments. + + An `Apply` AST expression is created by using function call syntax on + instructions. This applies to both bound and unbound polymorphic + instructions: + + >>> from .base import jump, iadd + >>> jump('next', ()) + Apply(jump, ('next', ())) + >>> iadd.i32('x', 'y') + Apply(iadd.i32, ('x', 'y')) + + :param inst: The instruction being applied, an `Instruction` or + `BoundInstruction` instance. + :param args: Tuple of arguments. + """ + + def __init__(self, inst, args): + from . import BoundInstruction + if isinstance(inst, BoundInstruction): + self.inst = inst.inst + self.typevars = inst.typevars + else: + self.inst = inst + self.typevars = () + self.args = args + assert len(self.inst.ins) == len(args) + + def instname(self): + i = self.inst.name + for t in self.typevars: + i += '.{}'.format(t) + return i + + def __repr__(self): + return "Apply({}, {})".format(self.instname(), self.args) + + def __str__(self): + args = ', '.join(map(str, self.args)) + return '{}({})'.format(self.instname(), args) diff --git a/meta/cretonne/test_ast.py b/meta/cretonne/test_ast.py new file mode 100644 index 0000000000..71791db3f6 --- /dev/null +++ b/meta/cretonne/test_ast.py @@ -0,0 +1,28 @@ +from __future__ import absolute_import +from unittest import TestCase +from doctest import DocTestSuite +from . import ast +from .base import jump, iadd + + +def load_tests(loader, tests, ignore): + tests.addTests(DocTestSuite(ast)) + return tests + + +x = 'x' +y = 'y' +a = 'a' + + +class TestPatterns(TestCase): + def test_apply(self): + i = jump(x, y) + self.assertEqual(repr(i), "Apply(jump, ('x', 'y'))") + + i = iadd.i32(x, y) + self.assertEqual(repr(i), "Apply(iadd.i32, ('x', 'y'))") + + def test_single_ins(self): + pat = a << iadd.i32(x, y) + self.assertEqual(repr(pat), "('a',) << Apply(iadd.i32, ('x', 'y'))") diff --git a/meta/cretonne/test_xform.py b/meta/cretonne/test_xform.py new file mode 100644 index 0000000000..b472a8f458 --- /dev/null +++ b/meta/cretonne/test_xform.py @@ -0,0 +1,59 @@ +from __future__ import absolute_import +from unittest import TestCase +from doctest import DocTestSuite +from . import xform +from .base import iadd, iadd_imm, iconst +from .ast import Var +from .xform import Rtl, XForm + + +def load_tests(loader, tests, ignore): + tests.addTests(DocTestSuite(xform)) + return tests + + +x = Var('x') +y = Var('y') +a = Var('a') +c = Var('c') + + +class TestXForm(TestCase): + def test_macro_pattern(self): + src = Rtl(a << iadd_imm(x, y)) + dst = Rtl( + c << iconst(y), + a << iadd(x, c)) + XForm(src, dst) + + def test_def_input(self): + # Src pattern has a def which is an input in dst. + src = Rtl(a << iadd_imm(x, 1)) + dst = Rtl(y << iadd_imm(a, 1)) + with self.assertRaisesRegexp( + AssertionError, + "'a' used as both input and def"): + XForm(src, dst) + + def test_input_def(self): + # Converse of the above. + src = Rtl(y << iadd_imm(a, 1)) + dst = Rtl(a << iadd_imm(x, 1)) + with self.assertRaisesRegexp( + AssertionError, + "'a' used as both input and def"): + XForm(src, dst) + + def test_extra_input(self): + src = Rtl(a << iadd_imm(x, 1)) + dst = Rtl(a << iadd(x, y)) + with self.assertRaisesRegexp(AssertionError, "extra inputs in dst"): + XForm(src, dst) + + def test_double_def(self): + src = Rtl( + a << iadd_imm(x, 1), + a << iadd(x, y)) + dst = Rtl(a << iadd(x, y)) + with self.assertRaisesRegexp(AssertionError, "'a' multiply defined"): + XForm(src, dst) diff --git a/meta/cretonne/xform.py b/meta/cretonne/xform.py new file mode 100644 index 0000000000..16a9eae870 --- /dev/null +++ b/meta/cretonne/xform.py @@ -0,0 +1,182 @@ +""" +Instruction transformations. +""" +from __future__ import absolute_import +from .ast import Def, Var, Apply + + +SRCCTX = 1 +DSTCTX = 2 + + +class Rtl(object): + """ + Register Transfer Language list. + + An RTL object contains a list of register assignments in the form of `Def` + objects and/or Apply objects for side-effecting instructions. + + An RTL list can represent both a source pattern to be matched, or a + destination pattern to be inserted. + """ + + def __init__(self, *args): + self.rtl = args + + def __iter__(self): + return iter(self.rtl) + + +class XForm(object): + """ + An instruction transformation consists of a source and destination pattern. + + Patterns are expressed in *register transfer language* as tuples of + `ast.Def` or `ast.Expr` nodes. + + A legalization pattern must have a source pattern containing only a single + instruction. + + >>> from .base import iconst, iadd, iadd_imm + >>> a = Var('a') + >>> c = Var('c') + >>> v = Var('v') + >>> x = Var('x') + >>> XForm( + ... Rtl(c << iconst(v), + ... a << iadd(x, c)), + ... Rtl(a << iadd_imm(x, v))) + XForm(inputs=[Var(v), Var(x)], defs=[Var(c, d=01), Var(a, d=11)], + c << iconst(v) + a << iadd(x, c) + => + a << iadd_imm(x, v) + ) + """ + + def __init__(self, src, dst): + self.src = src + self.dst = dst + # Variables that are inputs to the source pattern. + self.inputs = list() + # Variables defined in either src or dst. + self.defs = list() + + # Rewrite variables in src and dst RTL lists to our own copies. + # Map name -> private Var. + symtab = dict() + self._rewrite_rtl(src, symtab, SRCCTX) + num_src_inputs = len(self.inputs) + self._rewrite_rtl(dst, symtab, DSTCTX) + + # Check for inconsistently used inputs. + for i in self.inputs: + if i.defctx: + raise AssertionError( + "'{}' used as both input and def".format(i)) + + # Check for spurious inputs in dst. + if len(self.inputs) > num_src_inputs: + raise AssertionError( + "extra inputs in dst RTL: {}".format( + self.inputs[num_src_inputs:])) + + def __repr__(self): + s = "XForm(inputs={}, defs={},\n ".format(self.inputs, self.defs) + s += '\n '.join(str(n) for n in self.src) + s += '\n=>\n ' + s += '\n '.join(str(n) for n in self.dst) + s += '\n)' + return s + + def _rewrite_rtl(self, rtl, symtab, context): + for line in rtl: + if isinstance(line, Def): + line.defs = tuple( + self._rewrite_defs(line.defs, symtab, context)) + expr = line.expr + else: + expr = line + self._rewrite_expr(expr, symtab, context) + + def _rewrite_expr(self, expr, symtab, context): + """ + Find all uses of variables in `expr` and replace them with our own + local symbols. + """ + + # Accept a whole expression tree. + stack = [expr] + while len(stack) > 0: + expr = stack.pop() + expr.args = tuple( + self._rewrite_uses(expr, stack, symtab, context)) + + def _rewrite_defs(self, defs, symtab, context): + """ + Given a tuple of symbols defined in a Def, rewrite them to local + symbols. Yield the new locals. + """ + for sym in defs: + name = str(sym) + if name in symtab: + var = symtab[name] + if var.defctx & context: + raise AssertionError("'{}' multiply defined".format(name)) + else: + var = Var(name) + symtab[name] = var + self.defs.append(var) + var.defctx |= context + yield var + + def _rewrite_uses(self, expr, stack, symtab, context): + """ + Given an `Apply` expr, rewrite all uses in its arguments to local + variables. Yield a sequence of new arguments. + + Append any `Apply` arguments to `stack`. + """ + for arg, operand in zip(expr.args, expr.inst.ins): + # Nested instructions are allowed. Visit recursively. + if isinstance(arg, Apply): + stack.push(arg) + yield arg + continue + if not isinstance(arg, Var): + assert not operand.is_value(), "Value arg must be `Var`" + yield arg + continue + # This is supposed to be a symbolic value reference. + name = str(arg) + if name in symtab: + var = symtab[name] + # The variable must be used consistenty as a def or input. + if var.defctx and (var.defctx & context) == 0: + raise AssertionError( + "'{}' used as both input and def" + .format(name)) + else: + # First time use of variable. + var = Var(name) + symtab[name] = var + self.inputs.append(var) + yield var + + +class XFormGroup(object): + """ + A group of related transformations. + """ + + def __init__(self): + self.xforms = list() + + def legalize(self, src, dst): + """ + Add a legalization pattern to this group. + + :param src: Single `Def` or `Apply` to be legalized. + :param dst: `Rtl` list of replacement instructions. + """ + self.xforms.append(XForm(Rtl(src), dst)) From 3207e60795037d33971dd3841afae7ecc71111a1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 27 Sep 2016 07:59:47 -0700 Subject: [PATCH 0340/3084] Add legalization patterns. --- meta/cretonne/legalize.py | 48 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 meta/cretonne/legalize.py diff --git a/meta/cretonne/legalize.py b/meta/cretonne/legalize.py new file mode 100644 index 0000000000..14eb879abd --- /dev/null +++ b/meta/cretonne/legalize.py @@ -0,0 +1,48 @@ +""" +Patterns for legalizing the `base` instruction set. + +The base Cretonne instruction set is 'fat', and many instructions don't have +legal representations in a given target ISA. This module defines legalization +patterns that describe how base instructions can be transformed to other base +instructions that are legal. +""" +from __future__ import absolute_import +from .base import iadd, iadd_cout, iadd_cin, isplit_lohi, iconcat_lohi +from .base import isub, isub_bin, isub_bout +from .ast import Var +from .xform import Rtl, XFormGroup + + +narrow = XFormGroup() + +x = Var('x') +y = Var('y') +a = Var('a') +b = Var('b') +c = Var('c') +xl = Var('xl') +xh = Var('xh') +yl = Var('yl') +yh = Var('yh') +al = Var('al') +ah = Var('ah') + +narrow.legalize( + a << iadd(x, y), + Rtl( + (xl, xh) << isplit_lohi(x), + (yl, yh) << isplit_lohi(y), + (al, c) << iadd_cout(xl, yl), + ah << iadd_cin(xh, yh, c), + a << iconcat_lohi(al, ah) + )) + +narrow.legalize( + a << isub(x, y), + Rtl( + (xl, xh) << isplit_lohi(x), + (yl, yh) << isplit_lohi(y), + (al, b) << isub_bout(xl, yl), + ah << isub_bin(xh, yh, b), + a << iconcat_lohi(al, ah) + )) From 6b784dd8dcfe346877ce3633ac26cade1f42cc35 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 10 Oct 2016 10:44:33 -0700 Subject: [PATCH 0341/3084] Create a phantom workspace manifest for all crates. Share a single Cargo.lock and target directory at the repo top-level. --- .travis.yml | 5 +- cranelift/.gitignore | 2 + cranelift/Cargo.toml | 3 + cranelift/src/.gitignore | 2 - cranelift/src/tools/Cargo.lock | 159 --------------------------------- cranelift/test-all.sh | 4 +- 6 files changed, 8 insertions(+), 167 deletions(-) create mode 100644 cranelift/Cargo.toml delete mode 100644 cranelift/src/.gitignore delete mode 100644 cranelift/src/tools/Cargo.lock diff --git a/.travis.yml b/.travis.yml index 33eed41b71..034d0e6c55 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,7 +4,4 @@ rust: - beta - nightly script: ./test-all.sh -cache: - - cargo - - directories: - - src/tools/target +cache: cargo diff --git a/cranelift/.gitignore b/cranelift/.gitignore index 002a368e2b..9ceaac3ed5 100644 --- a/cranelift/.gitignore +++ b/cranelift/.gitignore @@ -3,3 +3,5 @@ *.swp *.swo tags +target +Cargo.lock diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml new file mode 100644 index 0000000000..63597ad428 --- /dev/null +++ b/cranelift/Cargo.toml @@ -0,0 +1,3 @@ +# Phantom workspace manifest for all Cretonne crates. +[workspace] +members = ["src/tools"] diff --git a/cranelift/src/.gitignore b/cranelift/src/.gitignore deleted file mode 100644 index d6168f7c42..0000000000 --- a/cranelift/src/.gitignore +++ /dev/null @@ -1,2 +0,0 @@ -target -lib*/Cargo.lock diff --git a/cranelift/src/tools/Cargo.lock b/cranelift/src/tools/Cargo.lock deleted file mode 100644 index 28d0f79f5b..0000000000 --- a/cranelift/src/tools/Cargo.lock +++ /dev/null @@ -1,159 +0,0 @@ -[root] -name = "cretonne-tools" -version = "0.0.0" -dependencies = [ - "cretonne 0.0.0", - "cretonne-reader 0.0.0", - "docopt 0.6.83 (registry+https://github.com/rust-lang/crates.io-index)", - "filecheck 0.0.0", - "num_cpus 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)", - "rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "aho-corasick" -version = "0.5.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "memchr 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "cretonne" -version = "0.0.0" - -[[package]] -name = "cretonne-reader" -version = "0.0.0" -dependencies = [ - "cretonne 0.0.0", -] - -[[package]] -name = "docopt" -version = "0.6.83" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "lazy_static 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", - "regex 0.1.77 (registry+https://github.com/rust-lang/crates.io-index)", - "rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)", - "strsim 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "filecheck" -version = "0.0.0" -dependencies = [ - "regex 0.1.77 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "kernel32-sys" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "lazy_static" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "libc" -version = "0.2.16" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "memchr" -version = "0.1.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "libc 0.2.16 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "num_cpus" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "libc 0.2.16 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "regex" -version = "0.1.77" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "aho-corasick 0.5.3 (registry+https://github.com/rust-lang/crates.io-index)", - "memchr 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)", - "regex-syntax 0.3.5 (registry+https://github.com/rust-lang/crates.io-index)", - "thread_local 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)", - "utf8-ranges 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "regex-syntax" -version = "0.3.5" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "rustc-serialize" -version = "0.3.19" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "strsim" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "thread-id" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.16 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "thread_local" -version = "0.2.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "thread-id 2.0.0 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "utf8-ranges" -version = "0.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "winapi" -version = "0.2.8" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "winapi-build" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[metadata] -"checksum aho-corasick 0.5.2 (registry+https://github.com/rust-lang/crates.io-index)" = "2b3fb52b09c1710b961acb35390d514be82e4ac96a9969a8e38565a29b878dc9" -"checksum docopt 0.6.80 (registry+https://github.com/rust-lang/crates.io-index)" = "4cc0acb4ce0828c6a5a11d47baa432fe885881c27428c3a4e473e454ffe57a76" -"checksum glob 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)" = "8be18de09a56b60ed0edf84bc9df007e30040691af7acd1c41874faac5895bfb" -"checksum kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "7507624b29483431c0ba2d82aece8ca6cdba9382bff4ddd0f7490560c056098d" -"checksum libc 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)" = "c96061f0c8a2dc27482e394d82e23073569de41d73cd736672ccd3e5c7471bfd" -"checksum memchr 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)" = "d8b629fb514376c675b98c1421e80b151d3817ac42d7c667717d282761418d20" -"checksum regex 0.1.71 (registry+https://github.com/rust-lang/crates.io-index)" = "e58a1b7d2bfecc0746e8587c30a53d01ea7bc0e98fac54e5aaa375b94338a0cc" -"checksum regex-syntax 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)" = "baa04823ba7be7ed0bed3d0704c7b923019d9c4e4931c5af2804c7c7a0e3d00b" -"checksum rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)" = "6159e4e6e559c81bd706afe9c8fd68f547d3e851ce12e76b1de7914bab61691b" -"checksum strsim 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)" = "e4d73a2c36a4d095ed1a6df5cbeac159863173447f7a82b3f4757426844ab825" -"checksum thread-id 2.0.0 (registry+https://github.com/rust-lang/crates.io-index)" = "a9539db560102d1cef46b8b78ce737ff0bb64e7e18d35b2a5688f7d097d0ff03" -"checksum thread_local 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)" = "55dd963dbaeadc08aa7266bf7f91c3154a7805e32bb94b820b769d2ef3b4744d" -"checksum utf8-ranges 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "a1ca13c08c41c9c3e04224ed9ff80461d97e121589ff27c753a16cb10830ae0f" -"checksum winapi 0.2.7 (registry+https://github.com/rust-lang/crates.io-index)" = "3969e500d618a5e974917ddefd0ba152e4bcaae5eb5d9b8c1fbc008e9e28c24e" -"checksum winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "2d315eee3b34aca4797b2da6b13ed88266e6d612562a0c46390af8299fc699bc" diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 87aea4573a..4cf436e100 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -51,12 +51,12 @@ done # Build cton-util for parser testing. cd "$topdir/src/tools" banner "Rust documentation" -echo "open $topdir/src/tools/target/doc/cretonne/index.html" +echo "open $topdir/target/doc/cretonne/index.html" cargo doc banner "Rust release build" cargo build --release -export CTONUTIL="$topdir/src/tools/target/release/cton-util" +export CTONUTIL="$topdir/target/release/cton-util" cd "$topdir" banner "File tests" From 6158b7449a5e948ca84c2a52193436ca8603e3f4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Oct 2016 13:41:27 -0700 Subject: [PATCH 0342/3084] Upgrade to rustfmt 0.6.2 --- cranelift/test-all.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 4cf436e100..843b1e82be 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -30,7 +30,7 @@ function banner() { # rustfmt is installed. # # This version should always be bumped to the newest version available. -RUSTFMT_VERSION="0.6.0" +RUSTFMT_VERSION="0.6.2" if cargo install --list | grep -q "^rustfmt v$RUSTFMT_VERSION"; then banner "Rust formatting" From eeb6fe098318e03b69e68004a9f177272444fc6d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Oct 2016 09:48:52 -0700 Subject: [PATCH 0343/3084] Track InstructionData member names. Entity references in instruction format operands also have member names in the InstructionData struct. Track them the same way we track immediate operand member names. Value operands still go in the arg / args members. --- meta/cretonne/__init__.py | 16 ++++++++-------- meta/cretonne/entities.py | 7 +++++-- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index a5fe761bce..825b30a8dc 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -293,6 +293,7 @@ class OperandKind(object): def __init__(self, name, doc): self.name = name self.__doc__ = doc + self.default_member = None # The camel-cased name of an operand kind is also the Rust type used to # represent it. self.camel_name = camel_case(name) @@ -347,8 +348,7 @@ class ImmediateKind(OperandKind): """ def __init__(self, name, doc, default_member='imm'): - self.name = name - self.__doc__ = doc + super(ImmediateKind, self).__init__(name, doc) self.default_member = default_member def __repr__(self): @@ -362,9 +362,9 @@ class EntityRefKind(OperandKind): The kind of an entity reference instruction operand. """ - def __init__(self, name, doc): - self.name = name - self.__doc__ = doc + def __init__(self, name, doc, default_member=None): + super(EntityRefKind, self).__init__(name, doc) + self.default_member = default_member or name def __repr__(self): return 'EntityRefKind({})'.format(self.name) @@ -686,17 +686,17 @@ class InstructionFormat(object): Yields the operand kinds. """ + self.members = list() for i, k in enumerate(kinds): if isinstance(k, tuple): member, k = k else: - member = None + member = k.default_member yield k + self.members.append(member) # Create `FormatField` instances for the immediates. if isinstance(k, ImmediateKind): - if not member: - member = k.default_member assert not hasattr(self, member), "Duplicate member name" field = FormatField(self, i, member) setattr(self, member, field) diff --git a/meta/cretonne/entities.py b/meta/cretonne/entities.py index bd55fa6eed..907fb64359 100644 --- a/meta/cretonne/entities.py +++ b/meta/cretonne/entities.py @@ -9,7 +9,9 @@ from . import EntityRefKind #: A reference to an extended basic block in the same function. #: This is primarliy used in control flow instructions. -ebb = EntityRefKind('ebb', 'An extended basic block in the same function.') +ebb = EntityRefKind( + 'ebb', 'An extended basic block in the same function.', + default_member='destination') #: A reference to a stack slot declared in the function preamble. stack_slot = EntityRefKind('stack_slot', 'A stack slot.') @@ -23,4 +25,5 @@ signature = EntityRefKind('signature', 'A function signature.') function = EntityRefKind('function', 'An external function.') #: A reference to a jump table declared in the function preamble. -jump_table = EntityRefKind('jump_table', 'A jump table.') +jump_table = EntityRefKind( + 'jump_table', 'A jump table.', default_member='table') From e5080fb64e1293b20a7328026235a9c0b16c812f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Oct 2016 10:07:00 -0700 Subject: [PATCH 0344/3084] Use 'varargs' consistently for VariableArgs members. The meta code generators need to be able to infer these too. --- cranelift/src/libcretonne/ir/instructions.rs | 24 +++++++++---------- .../src/libcretonne/test_utils/make_inst.rs | 6 ++--- cranelift/src/libcretonne/write.rs | 4 ++-- cranelift/src/libreader/parser.rs | 14 +++++------ meta/cretonne/__init__.py | 13 +++++----- 5 files changed, 30 insertions(+), 31 deletions(-) diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index fd7fd5f25b..6b3e0e5ffe 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -283,15 +283,15 @@ impl Display for TernaryOverflowData { #[derive(Clone, Debug)] pub struct JumpData { pub destination: Ebb, - pub arguments: VariableArgs, + pub varargs: VariableArgs, } impl Display for JumpData { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - if self.arguments.is_empty() { + if self.varargs.is_empty() { write!(f, "{}", self.destination) } else { - write!(f, "{}({})", self.destination, self.arguments) + write!(f, "{}({})", self.destination, self.varargs) } } } @@ -302,14 +302,14 @@ impl Display for JumpData { pub struct BranchData { pub arg: Value, pub destination: Ebb, - pub arguments: VariableArgs, + pub varargs: VariableArgs, } impl Display for BranchData { fn fmt(&self, f: &mut Formatter) -> fmt::Result { try!(write!(f, "{}, {}", self.arg, self.destination)); - if !self.arguments.is_empty() { - try!(write!(f, "({})", self.arguments)); + if !self.varargs.is_empty() { + try!(write!(f, "({})", self.varargs)); } Ok(()) } @@ -322,12 +322,12 @@ pub struct CallData { second_result: Value, // Dynamically sized array containing call argument values. - pub args: VariableArgs, + pub varargs: VariableArgs, } impl Display for CallData { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - write!(f, "TBD({})", self.args) + write!(f, "TBD({})", self.varargs) } } @@ -335,7 +335,7 @@ impl Display for CallData { #[derive(Clone, Debug)] pub struct ReturnData { // Dynamically sized array containing return values. - pub args: VariableArgs, + pub varargs: VariableArgs, } impl InstructionData { @@ -346,7 +346,7 @@ impl InstructionData { ty: return_type, data: Box::new(CallData { second_result: NO_VALUE, - args: VariableArgs::new(), + varargs: VariableArgs::new(), }), } } @@ -364,10 +364,10 @@ impl InstructionData { pub fn analyze_branch<'a>(&'a self) -> BranchInfo<'a> { match self { &InstructionData::Jump { ref data, .. } => { - BranchInfo::SingleDest(data.destination, &data.arguments) + BranchInfo::SingleDest(data.destination, &data.varargs) } &InstructionData::Branch { ref data, .. } => { - BranchInfo::SingleDest(data.destination, &data.arguments) + BranchInfo::SingleDest(data.destination, &data.varargs) } &InstructionData::BranchTable { table, .. } => BranchInfo::Table(table), _ => BranchInfo::NotABranch, diff --git a/cranelift/src/libcretonne/test_utils/make_inst.rs b/cranelift/src/libcretonne/test_utils/make_inst.rs index a072b8ba8f..9ab4429b76 100644 --- a/cranelift/src/libcretonne/test_utils/make_inst.rs +++ b/cranelift/src/libcretonne/test_utils/make_inst.rs @@ -11,7 +11,7 @@ pub fn jump(func: &mut Function, dest: Ebb) -> Inst { ty: types::VOID, data: Box::new(JumpData { destination: dest, - arguments: VariableArgs::new(), + varargs: VariableArgs::new(), }), }) } @@ -23,7 +23,7 @@ pub fn branch(func: &mut Function, dest: Ebb) -> Inst { data: Box::new(BranchData { arg: NO_VALUE, destination: dest, - arguments: VariableArgs::new(), + varargs: VariableArgs::new(), }), }) } @@ -32,6 +32,6 @@ pub fn ret(func: &mut Function) -> Inst { func.dfg.make_inst(InstructionData::Return { opcode: Opcode::Return, ty: types::VOID, - data: Box::new(ReturnData { args: VariableArgs::new() }), + data: Box::new(ReturnData { varargs: VariableArgs::new() }), }) } diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index 8f7acde772..e49c8382bf 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -201,10 +201,10 @@ fn write_instruction(w: &mut Write, BranchTable { arg, table, .. } => writeln!(w, " {}, {}", arg, table), Call { ref data, .. } => writeln!(w, " {}", data), Return { ref data, .. } => { - if data.args.is_empty() { + if data.varargs.is_empty() { writeln!(w, "") } else { - writeln!(w, " {}", data.args) + writeln!(w, " {}", data.varargs) } } } diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 258f920b81..eb8c78de38 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -145,21 +145,21 @@ impl Context { InstructionData::Jump { ref mut data, .. } => { try!(self.map.rewrite_ebb(&mut data.destination, loc)); - try!(self.map.rewrite_values(&mut data.arguments, loc)); + try!(self.map.rewrite_values(&mut data.varargs, loc)); } InstructionData::Branch { ref mut data, .. } => { try!(self.map.rewrite_value(&mut data.arg, loc)); try!(self.map.rewrite_ebb(&mut data.destination, loc)); - try!(self.map.rewrite_values(&mut data.arguments, loc)); + try!(self.map.rewrite_values(&mut data.varargs, loc)); } InstructionData::Call { ref mut data, .. } => { - try!(self.map.rewrite_values(&mut data.args, loc)); + try!(self.map.rewrite_values(&mut data.varargs, loc)); } InstructionData::Return { ref mut data, .. } => { - try!(self.map.rewrite_values(&mut data.args, loc)); + try!(self.map.rewrite_values(&mut data.varargs, loc)); } } } @@ -1104,7 +1104,7 @@ impl<'a> Parser<'a> { ty: VOID, data: Box::new(JumpData { destination: ebb_num, - arguments: args, + varargs: args, }), } } @@ -1119,7 +1119,7 @@ impl<'a> Parser<'a> { data: Box::new(BranchData { arg: ctrl_arg, destination: ebb_num, - arguments: args, + varargs: args, }), } } @@ -1178,7 +1178,7 @@ impl<'a> Parser<'a> { InstructionData::Return { opcode: opcode, ty: VOID, - data: Box::new(ReturnData { args: args }), + data: Box::new(ReturnData { varargs: args }), } } InstructionFormat::BranchTable => { diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 825b30a8dc..05366b0720 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -290,10 +290,10 @@ class OperandKind(object): instruction. """ - def __init__(self, name, doc): + def __init__(self, name, doc, default_member=None): self.name = name self.__doc__ = doc - self.default_member = None + self.default_member = default_member # The camel-cased name of an operand kind is also the Rust type used to # represent it. self.camel_name = camel_case(name) @@ -334,7 +334,8 @@ variable_args = OperandKind( Use this to represent arguemtns passed to a function call, arguments passed to an extended basic block, or a variable number of results returned from an instruction. - """) + """, + default_member='varargs') # Instances of immediate operand types are provided in the @@ -348,8 +349,7 @@ class ImmediateKind(OperandKind): """ def __init__(self, name, doc, default_member='imm'): - super(ImmediateKind, self).__init__(name, doc) - self.default_member = default_member + super(ImmediateKind, self).__init__(name, doc, default_member) def __repr__(self): return 'ImmediateKind({})'.format(self.name) @@ -363,8 +363,7 @@ class EntityRefKind(OperandKind): """ def __init__(self, name, doc, default_member=None): - super(EntityRefKind, self).__init__(name, doc) - self.default_member = default_member or name + super(EntityRefKind, self).__init__(name, doc, default_member or name) def __repr__(self): return 'EntityRefKind({})'.format(self.name) From 75406ad5bea1560e8fcf32f760e7ec2581f54905 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Oct 2016 13:35:18 -0700 Subject: [PATCH 0345/3084] Move signatures into new ir::extfunc module. This new module will gain more data types dealing with external function calls. --- cranelift/src/libcretonne/ir/extfunc.rs | 135 ++++++++++++++++++++++++ cranelift/src/libcretonne/ir/mod.rs | 4 +- cranelift/src/libcretonne/ir/types.rs | 127 ---------------------- cranelift/src/libreader/parser.rs | 7 +- 4 files changed, 142 insertions(+), 131 deletions(-) create mode 100644 cranelift/src/libcretonne/ir/extfunc.rs diff --git a/cranelift/src/libcretonne/ir/extfunc.rs b/cranelift/src/libcretonne/ir/extfunc.rs new file mode 100644 index 0000000000..78b723398e --- /dev/null +++ b/cranelift/src/libcretonne/ir/extfunc.rs @@ -0,0 +1,135 @@ +//! External function calls. +//! +//! To a Cretonne function, all functions are "external". Directly called functions must be +//! declared in the preamble, and all function calls must have a signature. +//! +//! This module declares the data types used to represent external functions and call signatures. + +use std::fmt::{self, Display, Formatter}; +use ir::Type; + +/// Function signature. +/// +/// The function signature describes the types of arguments and return values along with other +/// details that are needed to call a function correctly. +#[derive(Clone, PartialEq, Eq, Debug)] +pub struct Signature { + pub argument_types: Vec, + pub return_types: Vec, +} + +impl Signature { + pub fn new() -> Signature { + Signature { + argument_types: Vec::new(), + return_types: Vec::new(), + } + } +} + +fn write_list(f: &mut Formatter, args: &Vec) -> fmt::Result { + match args.split_first() { + None => {} + Some((first, rest)) => { + try!(write!(f, "{}", first)); + for arg in rest { + try!(write!(f, ", {}", arg)); + } + } + } + Ok(()) +} + +impl Display for Signature { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + try!(write!(f, "(")); + try!(write_list(f, &self.argument_types)); + try!(write!(f, ")")); + if !self.return_types.is_empty() { + try!(write!(f, " -> ")); + try!(write_list(f, &self.return_types)); + } + Ok(()) + } +} + +/// Function argument or return value type. +/// +/// This describes the value type being passed to or from a function along with flags that affect +/// how the argument is passed. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct ArgumentType { + pub value_type: Type, + pub extension: ArgumentExtension, + /// Place this argument in a register if possible. + pub inreg: bool, +} + +impl ArgumentType { + pub fn new(vt: Type) -> ArgumentType { + ArgumentType { + value_type: vt, + extension: ArgumentExtension::None, + inreg: false, + } + } +} + +impl Display for ArgumentType { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + try!(write!(f, "{}", self.value_type)); + match self.extension { + ArgumentExtension::None => {} + ArgumentExtension::Uext => try!(write!(f, " uext")), + ArgumentExtension::Sext => try!(write!(f, " sext")), + } + if self.inreg { + try!(write!(f, " inreg")); + } + Ok(()) + } +} + +/// Function argument extension options. +/// +/// On some architectures, small integer function arguments are extended to the width of a +/// general-purpose register. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub enum ArgumentExtension { + /// No extension, high bits are indeterminate. + None, + /// Unsigned extension: high bits in register are 0. + Uext, + /// Signed extension: high bits in register replicate sign bit. + Sext, +} + +#[cfg(test)] +mod tests { + use super::*; + use ir::types::{I32, F32, B8}; + + #[test] + fn argument_type() { + let mut t = ArgumentType::new(I32); + assert_eq!(t.to_string(), "i32"); + t.extension = ArgumentExtension::Uext; + assert_eq!(t.to_string(), "i32 uext"); + t.inreg = true; + assert_eq!(t.to_string(), "i32 uext inreg"); + } + + #[test] + fn signatures() { + let mut sig = Signature::new(); + assert_eq!(sig.to_string(), "()"); + sig.argument_types.push(ArgumentType::new(I32)); + assert_eq!(sig.to_string(), "(i32)"); + sig.return_types.push(ArgumentType::new(F32)); + assert_eq!(sig.to_string(), "(i32) -> f32"); + sig.argument_types.push(ArgumentType::new(I32.by(4).unwrap())); + assert_eq!(sig.to_string(), "(i32, i32x4) -> f32"); + sig.return_types.push(ArgumentType::new(B8)); + assert_eq!(sig.to_string(), "(i32, i32x4) -> f32, b8"); + } +} diff --git a/cranelift/src/libcretonne/ir/mod.rs b/cranelift/src/libcretonne/ir/mod.rs index ae16eae735..285fab96e5 100644 --- a/cranelift/src/libcretonne/ir/mod.rs +++ b/cranelift/src/libcretonne/ir/mod.rs @@ -11,9 +11,11 @@ pub mod dfg; pub mod layout; pub mod function; mod funcname; +mod extfunc; pub use ir::funcname::FunctionName; -pub use ir::types::{Type, Signature}; +pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension}; +pub use ir::types::Type; pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable}; pub use ir::instructions::{Opcode, InstructionData}; pub use ir::stackslot::StackSlotData; diff --git a/cranelift/src/libcretonne/ir/types.rs b/cranelift/src/libcretonne/ir/types.rs index 7c9175f460..99390ee22a 100644 --- a/cranelift/src/libcretonne/ir/types.rs +++ b/cranelift/src/libcretonne/ir/types.rs @@ -1,4 +1,3 @@ - //! Common types for the Cretonne code generator. use std::default::Default; @@ -229,108 +228,6 @@ impl Default for Type { } } -// ====--------------------------------------------------------------------------------------====// -// -// Function signatures -// -// ====--------------------------------------------------------------------------------------====// - -/// Function argument extension options. -/// -/// On some architectures, small integer function arguments are extended to the width of a -/// general-purpose register. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] -pub enum ArgumentExtension { - /// No extension, high bits are indeterminate. - None, - /// Unsigned extension: high bits in register are 0. - Uext, - /// Signed extension: high bits in register replicate sign bit. - Sext, -} - -/// Function argument or return value type. -/// -/// This describes the value type being passed to or from a function along with flags that affect -/// how the argument is passed. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] -pub struct ArgumentType { - pub value_type: Type, - pub extension: ArgumentExtension, - /// Place this argument in a register if possible. - pub inreg: bool, -} - -/// Function signature. -/// -/// The function signature describes the types of arguments and return values along with other -/// details that are needed to call a function correctly. -#[derive(Clone, PartialEq, Eq, Debug)] -pub struct Signature { - pub argument_types: Vec, - pub return_types: Vec, -} - -impl ArgumentType { - pub fn new(vt: Type) -> ArgumentType { - ArgumentType { - value_type: vt, - extension: ArgumentExtension::None, - inreg: false, - } - } -} - -impl Display for ArgumentType { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - try!(write!(f, "{}", self.value_type)); - match self.extension { - ArgumentExtension::None => {} - ArgumentExtension::Uext => try!(write!(f, " uext")), - ArgumentExtension::Sext => try!(write!(f, " sext")), - } - if self.inreg { - try!(write!(f, " inreg")); - } - Ok(()) - } -} - -impl Signature { - pub fn new() -> Signature { - Signature { - argument_types: Vec::new(), - return_types: Vec::new(), - } - } -} - -fn write_list(f: &mut Formatter, args: &Vec) -> fmt::Result { - match args.split_first() { - None => {} - Some((first, rest)) => { - try!(write!(f, "{}", first)); - for arg in rest { - try!(write!(f, ", {}", arg)); - } - } - } - Ok(()) -} - -impl Display for Signature { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - try!(write!(f, "(")); - try!(write_list(f, &self.argument_types)); - try!(write!(f, ")")); - if !self.return_types.is_empty() { - try!(write!(f, " -> ")); - try!(write_list(f, &self.return_types)); - } - Ok(()) - } -} - #[cfg(test)] mod tests { use super::*; @@ -442,28 +339,4 @@ mod tests { assert_eq!(I8.by(512), None); assert_eq!(VOID.by(4), None); } - - #[test] - fn argument_type() { - let mut t = ArgumentType::new(I32); - assert_eq!(t.to_string(), "i32"); - t.extension = ArgumentExtension::Uext; - assert_eq!(t.to_string(), "i32 uext"); - t.inreg = true; - assert_eq!(t.to_string(), "i32 uext inreg"); - } - - #[test] - fn signatures() { - let mut sig = Signature::new(); - assert_eq!(sig.to_string(), "()"); - sig.argument_types.push(ArgumentType::new(I32)); - assert_eq!(sig.to_string(), "(i32)"); - sig.return_types.push(ArgumentType::new(F32)); - assert_eq!(sig.to_string(), "(i32) -> f32"); - sig.argument_types.push(ArgumentType::new(I32.by(4).unwrap())); - assert_eq!(sig.to_string(), "(i32, i32x4) -> f32"); - sig.return_types.push(ArgumentType::new(B8)); - assert_eq!(sig.to_string(), "(i32, i32x4) -> f32, b8"); - } } diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index eb8c78de38..5023195e11 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -9,8 +9,8 @@ use std::str::FromStr; use std::u32; use std::mem; use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotData, JumpTable, - JumpTableData}; -use cretonne::ir::types::{VOID, Signature, ArgumentType, ArgumentExtension}; + JumpTableData, Signature, ArgumentType, ArgumentExtension}; +use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::{AnyEntity, NO_EBB, NO_INST, NO_VALUE}; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, @@ -1202,7 +1202,8 @@ impl<'a> Parser<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne::ir::types::{self, ArgumentType, ArgumentExtension}; + use cretonne::ir::{ArgumentType, ArgumentExtension}; + use cretonne::ir::types; use cretonne::ir::entities::AnyEntity; use testfile::{Details, Comment}; use isaspec::IsaSpec; From e35811e120d371eb62ce1561b65c9b2a9209173f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Oct 2016 14:15:29 -0700 Subject: [PATCH 0346/3084] Add FuncRef and SigRef entity references. These refer to external functions and function signatures declared in the preamble. Since we're already using the type names 'Signature' and 'Function', these entity references don't folow the usual EntityData / Entity naming convention. --- cranelift/docs/langref.rst | 4 +- cranelift/src/libcretonne/ir/entities.rs | 78 ++++++++++++++++++++ cranelift/src/libcretonne/ir/instructions.rs | 22 ++---- cranelift/src/libcretonne/ir/mod.rs | 2 +- cranelift/src/libreader/lexer.rs | 4 + meta/cretonne/entities.py | 4 +- meta/cretonne/formats.py | 4 +- 7 files changed, 94 insertions(+), 24 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index e4d2d9dbd3..b2ced79300 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -394,11 +394,11 @@ preamble`: This simple example illustrates direct function calls and signatures:: function gcd(i32 uext, i32 uext) -> i32 uext "C" { - f1 = function divmod(i32 uext, i32 uext) -> i32 uext, i32 uext + fn1 = function divmod(i32 uext, i32 uext) -> i32 uext, i32 uext ebb1(v1: i32, v2: i32): brz v2, ebb2 - v3, v4 = call f1(v1, v2) + v3, v4 = call fn1(v1, v2) br ebb1(v2, v4) ebb2: diff --git a/cranelift/src/libcretonne/ir/entities.rs b/cranelift/src/libcretonne/ir/entities.rs index 5672e5b3d3..bd8913e5bf 100644 --- a/cranelift/src/libcretonne/ir/entities.rs +++ b/cranelift/src/libcretonne/ir/entities.rs @@ -245,6 +245,68 @@ impl Default for JumpTable { } } +/// A reference to an external function. +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +pub struct FuncRef(u32); + +impl EntityRef for FuncRef { + fn new(index: usize) -> FuncRef { + assert!(index < (u32::MAX as usize)); + FuncRef(index as u32) + } + + fn index(self) -> usize { + self.0 as usize + } +} + +/// Display a `FuncRef` reference as "fn12". +impl Display for FuncRef { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write!(fmt, "fn{}", self.0) + } +} + +/// A guaranteed invalid function reference. +pub const NO_FUNC_REF: FuncRef = FuncRef(u32::MAX); + +impl Default for FuncRef { + fn default() -> FuncRef { + NO_FUNC_REF + } +} + +/// A reference to a function signature. +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +pub struct SigRef(u32); + +impl EntityRef for SigRef { + fn new(index: usize) -> SigRef { + assert!(index < (u32::MAX as usize)); + SigRef(index as u32) + } + + fn index(self) -> usize { + self.0 as usize + } +} + +/// Display a `SigRef` reference as "sig12". +impl Display for SigRef { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write!(fmt, "sig{}", self.0) + } +} + +/// A guaranteed invalid function reference. +pub const NO_SIG_REF: SigRef = SigRef(u32::MAX); + +impl Default for SigRef { + fn default() -> SigRef { + NO_SIG_REF + } +} + /// A reference to any of the entities defined in this module. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub enum AnyEntity { @@ -255,6 +317,8 @@ pub enum AnyEntity { Value(Value), StackSlot(StackSlot), JumpTable(JumpTable), + FuncRef(FuncRef), + SigRef(SigRef), } impl Display for AnyEntity { @@ -266,6 +330,8 @@ impl Display for AnyEntity { AnyEntity::Value(r) => r.fmt(fmt), AnyEntity::StackSlot(r) => r.fmt(fmt), AnyEntity::JumpTable(r) => r.fmt(fmt), + AnyEntity::FuncRef(r) => r.fmt(fmt), + AnyEntity::SigRef(r) => r.fmt(fmt), } } } @@ -300,6 +366,18 @@ impl From for AnyEntity { } } +impl From for AnyEntity { + fn from(r: FuncRef) -> AnyEntity { + AnyEntity::FuncRef(r) + } +} + +impl From for AnyEntity { + fn from(r: SigRef) -> AnyEntity { + AnyEntity::SigRef(r) + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index 6b3e0e5ffe..56ce5ed5fd 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -10,8 +10,7 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::ops::{Deref, DerefMut}; -use ir::{Value, Type, Ebb, JumpTable}; -use ir::entities::NO_VALUE; +use ir::{Value, Type, Ebb, JumpTable, FuncRef}; use ir::immediates::{Imm64, Ieee32, Ieee64}; use ir::condcodes::*; use ir::types; @@ -321,7 +320,10 @@ pub struct CallData { /// Second result value for a call producing multiple return values. second_result: Value, - // Dynamically sized array containing call argument values. + /// Callee function. + pub func_ref: FuncRef, + + /// Dynamically sized array containing call argument values. pub varargs: VariableArgs, } @@ -338,20 +340,6 @@ pub struct ReturnData { pub varargs: VariableArgs, } -impl InstructionData { - /// Create data for a call instruction. - pub fn call(opc: Opcode, return_type: Type) -> InstructionData { - InstructionData::Call { - opcode: opc, - ty: return_type, - data: Box::new(CallData { - second_result: NO_VALUE, - varargs: VariableArgs::new(), - }), - } - } -} - /// Analyzing an instruction. /// /// Avoid large matches on instruction formats by using the methods efined here to examine diff --git a/cranelift/src/libcretonne/ir/mod.rs b/cranelift/src/libcretonne/ir/mod.rs index 285fab96e5..90581d5f12 100644 --- a/cranelift/src/libcretonne/ir/mod.rs +++ b/cranelift/src/libcretonne/ir/mod.rs @@ -16,7 +16,7 @@ mod extfunc; pub use ir::funcname::FunctionName; pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension}; pub use ir::types::Type; -pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable}; +pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable, FuncRef, SigRef}; pub use ir::instructions::{Opcode, InstructionData}; pub use ir::stackslot::StackSlotData; pub use ir::jumptable::JumpTableData; diff --git a/cranelift/src/libreader/lexer.rs b/cranelift/src/libreader/lexer.rs index 3454225dc8..04801b2644 100644 --- a/cranelift/src/libreader/lexer.rs +++ b/cranelift/src/libreader/lexer.rs @@ -34,6 +34,8 @@ pub enum Token<'a> { Ebb(Ebb), // ebb3 StackSlot(u32), // ss3 JumpTable(u32), // jt2 + FuncRef(u32), // fn2 + SigRef(u32), // sig2 Identifier(&'a str), // Unrecognized identifier (opcode, enumerator, ...) } @@ -285,6 +287,8 @@ impl<'a> Lexer<'a> { "ebb" => Ebb::with_number(number).map(|ebb| Token::Ebb(ebb)), "ss" => Some(Token::StackSlot(number)), "jt" => Some(Token::JumpTable(number)), + "fn" => Some(Token::FuncRef(number)), + "sig" => Some(Token::SigRef(number)), _ => None, } } diff --git a/meta/cretonne/entities.py b/meta/cretonne/entities.py index 907fb64359..317006f79f 100644 --- a/meta/cretonne/entities.py +++ b/meta/cretonne/entities.py @@ -18,11 +18,11 @@ stack_slot = EntityRefKind('stack_slot', 'A stack slot.') #: A reference to a function sugnature declared in the function preamble. #: Tbis is used to provide the call signature in an indirect call instruction. -signature = EntityRefKind('signature', 'A function signature.') +sig_ref = EntityRefKind('sig_ref', 'A function signature.') #: A reference to an external function declared in the function preamble. #: This is used to provide the callee and signature in a call instruction. -function = EntityRefKind('function', 'An external function.') +func_ref = EntityRefKind('func_ref', 'An external function.') #: A reference to a jump table declared in the function preamble. jump_table = EntityRefKind( diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index df0019bff4..11a1c5ef3b 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -8,7 +8,7 @@ in this module. from __future__ import absolute_import from . import InstructionFormat, value, variable_args from .immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc -from .entities import ebb, function, jump_table +from .entities import ebb, func_ref, jump_table Nullary = InstructionFormat() @@ -46,7 +46,7 @@ Branch = InstructionFormat(value, ebb, variable_args, boxed_storage=True) BranchTable = InstructionFormat(value, jump_table) Call = InstructionFormat( - function, variable_args, multiple_results=True, boxed_storage=True) + func_ref, variable_args, multiple_results=True, boxed_storage=True) Return = InstructionFormat(variable_args, boxed_storage=True) From cf8d628254d68af857249353316b70a031873f37 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Oct 2016 15:24:59 -0700 Subject: [PATCH 0347/3084] Add simple Uimm8 and ImmVector immediate types. Implement the boxed storage of the UnaryImmVector instruction format. --- cranelift/src/libcretonne/ir/immediates.rs | 11 +++++++++ cranelift/src/libcretonne/ir/instructions.rs | 25 ++++++++++++++++---- cranelift/src/libcretonne/write.rs | 2 +- meta/cretonne/formats.py | 2 +- 4 files changed, 34 insertions(+), 6 deletions(-) diff --git a/cranelift/src/libcretonne/ir/immediates.rs b/cranelift/src/libcretonne/ir/immediates.rs index 99276bfd66..e8d4a20773 100644 --- a/cranelift/src/libcretonne/ir/immediates.rs +++ b/cranelift/src/libcretonne/ir/immediates.rs @@ -123,6 +123,11 @@ impl FromStr for Imm64 { } } +/// 8-bit unsigned integer immediate operand. +/// +/// This is used to indicate lane indexes typically. +pub type Uimm8 = u8; + /// An IEEE binary32 immediate floating point value. /// /// All bit patterns are allowed. @@ -420,6 +425,12 @@ impl FromStr for Ieee64 { } } +/// Arbitrary vector immediate. +/// +/// This kind of immediate can represent any kind of SIMD vector constant. +/// The representation is simply the sequence of bytes that would be used to store the vector. +pub type ImmVector = Vec; + #[cfg(test)] mod tests { use super::*; diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index 56ce5ed5fd..b971a05ffa 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -11,7 +11,7 @@ use std::str::FromStr; use std::ops::{Deref, DerefMut}; use ir::{Value, Type, Ebb, JumpTable, FuncRef}; -use ir::immediates::{Imm64, Ieee32, Ieee64}; +use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, ImmVector}; use ir::condcodes::*; use ir::types; @@ -117,7 +117,8 @@ pub enum InstructionData { }, UnaryImmVector { opcode: Opcode, - ty: Type, // TBD: imm: Box + ty: Type, + data: Box, }, UnarySplit { opcode: Opcode, @@ -162,13 +163,13 @@ pub enum InstructionData { InsertLane { opcode: Opcode, ty: Type, - lane: u8, + lane: Uimm8, args: [Value; 2], }, ExtractLane { opcode: Opcode, ty: Type, - lane: u8, + lane: Uimm8, arg: Value, }, IntCompare { @@ -264,6 +265,22 @@ impl Default for VariableArgs { } } +/// Payload data for `vconst`. +#[derive(Clone, Debug)] +pub struct UnaryImmVectorData { + pub imm: ImmVector, +} + +impl Display for UnaryImmVectorData { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + try!(write!(f, "#")); + for b in &self.imm { + try!(write!(f, "{:02x}", b)); + } + Ok(()) + } +} + /// Payload data for ternary instructions with multiple results, such as `iadd_carry`. #[derive(Clone, Debug)] pub struct TernaryOverflowData { diff --git a/cranelift/src/libcretonne/write.rs b/cranelift/src/libcretonne/write.rs index e49c8382bf..3ef72d7178 100644 --- a/cranelift/src/libcretonne/write.rs +++ b/cranelift/src/libcretonne/write.rs @@ -184,7 +184,7 @@ fn write_instruction(w: &mut Write, UnaryImm { imm, .. } => writeln!(w, " {}", imm), UnaryIeee32 { imm, .. } => writeln!(w, " {}", imm), UnaryIeee64 { imm, .. } => writeln!(w, " {}", imm), - UnaryImmVector { .. } => writeln!(w, " [...]"), + UnaryImmVector { ref data, .. } => writeln!(w, " {}", data), UnarySplit { arg, .. } => writeln!(w, " {}", arg), Binary { args, .. } => writeln!(w, " {}, {}", args[0], args[1]), BinaryImm { arg, imm, .. } => writeln!(w, " {}, {}", arg, imm), diff --git a/meta/cretonne/formats.py b/meta/cretonne/formats.py index 11a1c5ef3b..8e5d4e25cb 100644 --- a/meta/cretonne/formats.py +++ b/meta/cretonne/formats.py @@ -16,7 +16,7 @@ Unary = InstructionFormat(value) UnaryImm = InstructionFormat(imm64) UnaryIeee32 = InstructionFormat(ieee32) UnaryIeee64 = InstructionFormat(ieee64) -UnaryImmVector = InstructionFormat(immvector) +UnaryImmVector = InstructionFormat(immvector, boxed_storage=True) UnarySplit = InstructionFormat(value, multiple_results=True) Binary = InstructionFormat(value, value) From 3ff28ed0640f06f4ea96b0744db4d31954bf22be Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Oct 2016 15:34:50 -0700 Subject: [PATCH 0348/3084] Capture the Rust type used to represent an operand kind. The Rust type is usually the camel-cased name of the operand kind, but there are variations, so allow an explicit rust_type='IntCC' when defining operand kinds. --- meta/cretonne/__init__.py | 14 ++++++++------ meta/cretonne/immediates.py | 9 ++++++--- 2 files changed, 14 insertions(+), 9 deletions(-) diff --git a/meta/cretonne/__init__.py b/meta/cretonne/__init__.py index 05366b0720..a6cef5397b 100644 --- a/meta/cretonne/__init__.py +++ b/meta/cretonne/__init__.py @@ -290,13 +290,13 @@ class OperandKind(object): instruction. """ - def __init__(self, name, doc, default_member=None): + def __init__(self, name, doc, default_member=None, rust_type=None): self.name = name self.__doc__ = doc self.default_member = default_member # The camel-cased name of an operand kind is also the Rust type used to # represent it. - self.camel_name = camel_case(name) + self.rust_type = rust_type or camel_case(name) def __str__(self): return self.name @@ -348,8 +348,9 @@ class ImmediateKind(OperandKind): `InstructionData` data structure. """ - def __init__(self, name, doc, default_member='imm'): - super(ImmediateKind, self).__init__(name, doc, default_member) + def __init__(self, name, doc, default_member='imm', rust_type=None): + super(ImmediateKind, self).__init__( + name, doc, default_member, rust_type) def __repr__(self): return 'ImmediateKind({})'.format(self.name) @@ -362,8 +363,9 @@ class EntityRefKind(OperandKind): The kind of an entity reference instruction operand. """ - def __init__(self, name, doc, default_member=None): - super(EntityRefKind, self).__init__(name, doc, default_member or name) + def __init__(self, name, doc, default_member=None, rust_type=None): + super(EntityRefKind, self).__init__( + name, doc, default_member or name, rust_type) def __repr__(self): return 'EntityRefKind({})'.format(self.name) diff --git a/meta/cretonne/immediates.py b/meta/cretonne/immediates.py index ca8affded0..34b81dc79d 100644 --- a/meta/cretonne/immediates.py +++ b/meta/cretonne/immediates.py @@ -28,7 +28,10 @@ ieee32 = ImmediateKind('ieee32', 'A 32-bit immediate floating point number.') ieee64 = ImmediateKind('ieee64', 'A 64-bit immediate floating point number.') #: A large SIMD vector constant. -immvector = ImmediateKind('immvector', 'An immediate SIMD vector.') +immvector = ImmediateKind( + 'immvector', + 'An immediate SIMD vector.', + rust_type='ImmVector') #: A condition code for comparing integer values. #: @@ -37,7 +40,7 @@ immvector = ImmediateKind('immvector', 'An immediate SIMD vector.') intcc = ImmediateKind( 'intcc', 'An integer comparison condition code.', - default_member='cond') + default_member='cond', rust_type='IntCC') #: A condition code for comparing floating point values. #: @@ -46,4 +49,4 @@ intcc = ImmediateKind( floatcc = ImmediateKind( 'floatcc', 'A floating point comparison condition code.', - default_member='cond') + default_member='cond', rust_type='FloatCC') From 93f79d7a488dac4d6e6960d906055d7126db46e5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Oct 2016 16:01:06 -0700 Subject: [PATCH 0349/3084] Move second_result outside boxed storage. In instruction formats that have multiple results AND boxed storage, place the second_result field outside the boxed storage. The 16-byte instruction format has room for opcode, type, second_result in the first 8 bytes, and the boxed pointer in the last 8 bytes. This provides a simpler implementation of the second_result() and second_result_mut() InstructionData methods. --- cranelift/src/libcretonne/ir/instructions.rs | 6 ++-- cranelift/src/libreader/parser.rs | 6 ++-- meta/gen_instr.py | 38 +++++++------------- 3 files changed, 16 insertions(+), 34 deletions(-) diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/cranelift/src/libcretonne/ir/instructions.rs index b971a05ffa..37b1106d36 100644 --- a/cranelift/src/libcretonne/ir/instructions.rs +++ b/cranelift/src/libcretonne/ir/instructions.rs @@ -158,6 +158,7 @@ pub enum InstructionData { TernaryOverflow { opcode: Opcode, ty: Type, + second_result: Value, data: Box, }, InsertLane { @@ -203,6 +204,7 @@ pub enum InstructionData { Call { opcode: Opcode, ty: Type, + second_result: Value, data: Box, }, Return { @@ -284,7 +286,6 @@ impl Display for UnaryImmVectorData { /// Payload data for ternary instructions with multiple results, such as `iadd_carry`. #[derive(Clone, Debug)] pub struct TernaryOverflowData { - pub second_result: Value, pub args: [Value; 3], } @@ -334,9 +335,6 @@ impl Display for BranchData { /// Payload of a call instruction. #[derive(Clone, Debug)] pub struct CallData { - /// Second result value for a call producing multiple return values. - second_result: Value, - /// Callee function. pub func_ref: FuncRef, diff --git a/cranelift/src/libreader/parser.rs b/cranelift/src/libreader/parser.rs index 5023195e11..5c86e4e21a 100644 --- a/cranelift/src/libreader/parser.rs +++ b/cranelift/src/libreader/parser.rs @@ -1089,10 +1089,8 @@ impl<'a> Parser<'a> { InstructionData::TernaryOverflow { opcode: opcode, ty: VOID, - data: Box::new(TernaryOverflowData { - second_result: NO_VALUE, - args: [lhs, rhs, cin], - }), + second_result: NO_VALUE, + data: Box::new(TernaryOverflowData { args: [lhs, rhs, cin] }), } } InstructionFormat::Jump => { diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 582c0477f0..26783ae82f 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -87,23 +87,16 @@ def gen_instruction_data_impl(fmt): 'pub fn second_result(&self) -> Option {', '}'): with fmt.indented('match *self {', '}'): for f in cretonne.InstructionFormat.all_formats: - if not f.multiple_results: - # Single or no results. - fmt.line( - 'InstructionData::{} {{ .. }} => None,' - .format(f.name)) - elif f.boxed_storage: - # Multiple results, boxed storage. - fmt.line( - 'InstructionData::' + f.name + - ' { ref data, .. }' + - ' => Some(data.second_result),') - else: - # Multiple results, inline storage. + if f.multiple_results: fmt.line( 'InstructionData::' + f.name + ' { second_result, .. }' + ' => Some(second_result),') + else: + # Single or no results. + fmt.line( + 'InstructionData::{} {{ .. }} => None,' + .format(f.name)) fmt.doc_comment('Mutable reference to second result value, if any.') with fmt.indented( @@ -111,23 +104,16 @@ def gen_instruction_data_impl(fmt): " -> Option<&'a mut Value> {", '}'): with fmt.indented('match *self {', '}'): for f in cretonne.InstructionFormat.all_formats: - if not f.multiple_results: - # Single or no results. - fmt.line( - 'InstructionData::{} {{ .. }} => None,' - .format(f.name)) - elif f.boxed_storage: - # Multiple results, boxed storage. - fmt.line( - 'InstructionData::' + f.name + - ' { ref mut data, .. }' + - ' => Some(&mut data.second_result),') - else: - # Multiple results, inline storage. + if f.multiple_results: fmt.line( 'InstructionData::' + f.name + ' { ref mut second_result, .. }' + ' => Some(second_result),') + else: + # Single or no results. + fmt.line( + 'InstructionData::{} {{ .. }} => None,' + .format(f.name)) fmt.doc_comment('Get the controlling type variable operand.') with fmt.indented( From 6ce6f25626e177e1485602df64d6118f75c8931e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Oct 2016 10:51:05 -0700 Subject: [PATCH 0350/3084] Generate a Builder data type. WIP. The Builder provides a convenient interface for inserting instructions into an extended basic block. The bulk of the builder methods are generated automatically from the meta language instruction descriptions. Still TODO: Keep track of an insertion position. --- cranelift/src/libcretonne/ir/builder.rs | 27 ++++ cranelift/src/libcretonne/ir/mod.rs | 4 +- meta/cretonne/base.py | 5 +- meta/gen_instr.py | 179 ++++++++++++++++++++++++ 4 files changed, 213 insertions(+), 2 deletions(-) create mode 100644 cranelift/src/libcretonne/ir/builder.rs diff --git a/cranelift/src/libcretonne/ir/builder.rs b/cranelift/src/libcretonne/ir/builder.rs new file mode 100644 index 0000000000..4e81027e9b --- /dev/null +++ b/cranelift/src/libcretonne/ir/builder.rs @@ -0,0 +1,27 @@ +//! Cretonne instruction builder. +//! +//! A `Builder` provides a convenient interface for inserting instructions into a Cretonne +//! function. Many of its methods are generated from the meta language instruction definitions. + +use ir::{types, instructions}; +use ir::{InstructionData, DataFlowGraph}; +use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, VariableArgs, FuncRef}; +use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, ImmVector}; +use ir::condcodes::{IntCC, FloatCC}; + +pub struct Builder<'a> { + dfg: &'a mut DataFlowGraph, +} + +impl<'a> Builder<'a> { + // Create and insert an instruction. + // This method is used by the generated format-specific methods. + fn insert_inst(&mut self, data: InstructionData) -> Inst { + let inst = self.dfg.make_inst(data); + inst + } +} + +// Include code generated by `meta/gen_instr.py`. This file includes `Builder` methods per +// instruction format and per opcode for inserting instructions. +include!(concat!(env!("OUT_DIR"), "/builder.rs")); diff --git a/cranelift/src/libcretonne/ir/mod.rs b/cranelift/src/libcretonne/ir/mod.rs index 90581d5f12..36d17c29aa 100644 --- a/cranelift/src/libcretonne/ir/mod.rs +++ b/cranelift/src/libcretonne/ir/mod.rs @@ -12,14 +12,16 @@ pub mod layout; pub mod function; mod funcname; mod extfunc; +mod builder; pub use ir::funcname::FunctionName; pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension}; pub use ir::types::Type; pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable, FuncRef, SigRef}; -pub use ir::instructions::{Opcode, InstructionData}; +pub use ir::instructions::{Opcode, InstructionData, VariableArgs}; pub use ir::stackslot::StackSlotData; pub use ir::jumptable::JumpTableData; pub use ir::dfg::{DataFlowGraph, ValueDef}; pub use ir::layout::Layout; pub use ir::function::Function; +pub use ir::builder::Builder; diff --git a/meta/cretonne/base.py b/meta/cretonne/base.py index fc8de5d78f..b665cf5d71 100644 --- a/meta/cretonne/base.py +++ b/meta/cretonne/base.py @@ -1057,7 +1057,7 @@ fdemote = Instruction( ins=x, outs=a) x = Operand('x', Float) -a = Operand('a', Int) +a = Operand('a', IntTo) fcvt_to_uint = Instruction( 'fcvt_to_uint', r""" @@ -1083,6 +1083,9 @@ fcvt_to_sint = Instruction( """, ins=x, outs=a) +x = Operand('x', Int) +a = Operand('a', FloatTo) + fcvt_from_uint = Instruction( 'fcvt_from_uint', r""" Convert unsigned integer to floating point. diff --git a/meta/gen_instr.py b/meta/gen_instr.py index 26783ae82f..8ae6d47ef7 100644 --- a/meta/gen_instr.py +++ b/meta/gen_instr.py @@ -343,6 +343,180 @@ def gen_type_constraints(fmt, instrs): fmt.line('OperandConstraint::{},'.format(c)) +def gen_format_constructor(iform, fmt): + """ + Emit a method for creating and inserting inserting an `iform` instruction, + where `iform` is an instruction format. + + Instruction formats that can produce multiple results take a `ctrl_typevar` + argument for deducing the result types. Others take a `result_type` + argument. + """ + + # Construct method arguments. + args = ['&mut self', 'opcode: Opcode'] + + if iform.multiple_results: + args.append('ctrl_typevar: Type') + # `dfg::make_inst_results` will compute the result type. + result_type = 'types::VOID' + else: + args.append('result_type: Type') + result_type = 'result_type' + + # Normal operand arguments. + for idx, kind in enumerate(iform.kinds): + args.append('op{}: {}'.format(idx, kind.rust_type)) + + proto = '{}({}) -> Inst'.format(iform.name, ', '.join(args)) + fmt.line('#[allow(non_snake_case)]') + with fmt.indented('pub fn {} {{'.format(proto), '}'): + # Generate the instruction data. + with fmt.indented( + 'let data = InstructionData::{} {{'.format(iform.name), '};'): + fmt.line('opcode: opcode,') + fmt.line('ty: {},'.format(result_type)) + if iform.multiple_results: + fmt.line('second_result: Value::default(),') + if iform.boxed_storage: + with fmt.indented( + 'data: Box::new(instructions::{}Data {{' + .format(iform.name), '}),'): + gen_member_inits(iform, fmt) + else: + gen_member_inits(iform, fmt) + + # Create result values if necessary. + if iform.multiple_results: + fmt.line('let inst = self.insert_inst(data);') + fmt.line('self.dfg.make_inst_results(inst, ctrl_typevar);') + fmt.line('inst') + else: + fmt.line('self.insert_inst(data)') + + +def gen_member_inits(iform, fmt): + """ + Emit member initializers for an `iform` instruction. + """ + + # Values first. + if len(iform.value_operands) == 1: + fmt.line('arg: op{},'.format(iform.value_operands[0])) + elif len(iform.value_operands) > 1: + fmt.line('args: [{}],'.format( + ', '.join('op{}'.format(i) for i in iform.value_operands))) + + # Immediates and entity references. + for idx, member in enumerate(iform.members): + if member: + fmt.line('{}: op{},'.format(member, idx)) + + +def gen_inst_builder(inst, fmt): + """ + Emit a method for generating the instruction `inst`. + + The method will create and insert an instruction, then return the result + values, or the instruction reference itself for instructions that don't + have results. + """ + + # Construct method arguments. + args = ['&mut self'] + + # The controlling type variable will be inferred from the input values if + # possible. Otherwise, it is the first method argument. + if inst.is_polymorphic and not inst.use_typevar_operand: + args.append('{}: Type'.format(inst.ctrl_typevar.name)) + + tmpl_types = list() + into_args = list() + for op in inst.ins: + if isinstance(op.kind, cretonne.ImmediateKind): + t = 'T{}{}'.format(1 + len(tmpl_types), op.kind.name) + tmpl_types.append('{}: Into<{}>'.format(t, op.kind.rust_type)) + into_args.append(op.name) + else: + t = op.kind.rust_type + args.append('{}: {}'.format(op.name, t)) + + # Return the inst reference for result-less instructions. + if len(inst.value_results) == 0: + rtype = 'Inst' + elif len(inst.value_results) == 1: + rtype = 'Value' + else: + rvals = ', '.join(len(inst.value_results) * ['Value']) + rtype = '({})'.format(rvals) + + method = inst.name + if method == 'return': + # Avoid Rust keywords + method = '_' + method + + if len(tmpl_types) > 0: + tmpl = '<{}>'.format(', '.join(tmpl_types)) + else: + tmpl = '' + proto = '{}{}({}) -> {}'.format(method, tmpl, ', '.join(args), rtype) + + fmt.line('#[allow(non_snake_case)]') + with fmt.indented('pub fn {} {{'.format(proto), '}'): + # Convert all of the `Into<>` arguments. + for arg in into_args: + fmt.line('let {} = {}.into();'.format(arg, arg)) + + # Arguments for instruction constructor. + args = ['Opcode::' + inst.camel_name] + + if inst.is_polymorphic and not inst.use_typevar_operand: + # This was an explicit method argument. + args.append(inst.ctrl_typevar.name) + elif len(inst.value_results) == 0: + args.append('types::VOID') + elif inst.is_polymorphic: + # Infer the controlling type variable from the input operands. + fmt.line( + 'let ctrl_typevar = self.dfg.value_type({});' + .format(inst.ins[inst.format.typevar_operand].name)) + args.append('ctrl_typevar') + else: + # This non-polymorphic instruction has a fixed result type. + args.append( + 'types::' + + inst.outs[inst.value_results[0]].typ.name.upper()) + + args.extend(op.name for op in inst.ins) + args = ', '.join(args) + fmt.line('let inst = self.{}({});'.format(inst.format.name, args)) + + if len(inst.value_results) == 0: + fmt.line('inst') + elif len(inst.value_results) == 1: + fmt.line('self.dfg.first_result(inst)') + else: + fmt.line('let mut results = self.dfg.inst_results(inst);') + fmt.line('({})'.format(', '.join( + len(inst.value_results) * ['results.next().unwrap()']))) + + +def gen_builder(insts, fmt): + """ + Generate a Builder trait with methods for all instructions. + """ + fmt.doc_comment( + 'Methods for inserting instructions by instruction format.') + with fmt.indented("impl<'a> Builder<'a> {", '}'): + for f in cretonne.InstructionFormat.all_formats: + gen_format_constructor(f, fmt) + + fmt.doc_comment('Methods for inserting instructions by opcode.') + with fmt.indented("impl<'a> Builder<'a> {", '}'): + for inst in insts: + gen_inst_builder(inst, fmt) + + def generate(isas, out_dir): groups = collect_instr_groups(isas) @@ -353,3 +527,8 @@ def generate(isas, out_dir): instrs = gen_opcodes(groups, fmt) gen_type_constraints(fmt, instrs) fmt.update_file('opcodes.rs', out_dir) + + # builder.rs + fmt = srcgen.Formatter() + gen_builder(instrs, fmt) + fmt.update_file('builder.rs', out_dir) From 177ac85db53b1f4e5ecc58c23fd3c346ef02e0fb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 13 Oct 2016 14:36:34 -0700 Subject: [PATCH 0351/3084] Add a LAyout Cursor data structure. A layout cursor can be used instead of an iterator to keep track of a position in a basic block *and* permitting instructions to be manipulated. --- cranelift/src/libcretonne/ir/layout.rs | 319 ++++++++++++++++++++++++- cranelift/src/libcretonne/ir/mod.rs | 2 +- 2 files changed, 319 insertions(+), 2 deletions(-) diff --git a/cranelift/src/libcretonne/ir/layout.rs b/cranelift/src/libcretonne/ir/layout.rs index a872efa5d8..1b78386100 100644 --- a/cranelift/src/libcretonne/ir/layout.rs +++ b/cranelift/src/libcretonne/ir/layout.rs @@ -250,9 +250,273 @@ impl<'a> Iterator for Insts<'a> { } } + +/// Layout Cursor. +/// +/// A `Cursor` represents a position in a function layout where instructions can be inserted and +/// removed. It can be used to iterate through the instructions of a function while editing them at +/// the same time. A normal instruction iterator can't do this since it holds an immutable refernce +/// to the Layout. +/// +/// When new instructions are added, the cursor can either apend them to an EBB or insert them +/// before the current instruction. +pub struct Cursor<'a> { + layout: &'a mut Layout, + pos: CursorPosition, +} + +/// The possible positions of a cursor. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum CursorPosition { + /// Cursor is not pointing anywhere. No instructions can be inserted. + Nowhere, + /// Cursor is pointing at an existing instruction. + /// New instructions will be inserted *before* the current instruction. + At(Inst), + /// Cursor is before the beginning of an EBB. No instructions can be inserted. Calling + /// `next_inst()` wil move to the first instruction in the EBB. + Before(Ebb), + /// Cursor is pointing after the end of an EBB. + /// New instructions will be appended to the EBB. + After(Ebb), +} + +impl<'a> Cursor<'a> { + /// Create a new `Cursor` for `layout`. + /// The cursor holds a mutable reference to `layout` for its entire lifetime. + pub fn new(layout: &'a mut Layout) -> Cursor { + Cursor { + layout: layout, + pos: CursorPosition::Nowhere, + } + } + + /// Get the current position. + pub fn position(&self) -> CursorPosition { + self.pos + } + + /// Get the EBB corresponding to the current position. + pub fn current_ebb(&self) -> Option { + use self::CursorPosition::*; + match self.pos { + Nowhere => None, + At(inst) => self.layout.inst_ebb(inst), + Before(ebb) | After(ebb) => Some(ebb), + } + } + + /// Go to a specific instruction which must be inserted in the layout. + /// New instructions will be inserted before `inst`. + pub fn goto_inst(&mut self, inst: Inst) { + assert!(self.layout.inst_ebb(inst).is_some()); + self.pos = CursorPosition::At(inst); + } + + /// Go to the top of `ebb` which must be inserted into the layout. + /// At this position, instructions cannot be inserted, but `next_inst()` will move to the first + /// instruction in `ebb`. + pub fn goto_top(&mut self, ebb: Ebb) { + assert!(self.layout.is_ebb_inserted(ebb)); + self.pos = CursorPosition::Before(ebb); + } + + /// Go to the bottom of `ebb` which must be inserted into the layout. + /// At this position, inserted instructions will be appended to `ebb`. + pub fn goto_bottom(&mut self, ebb: Ebb) { + assert!(self.layout.is_ebb_inserted(ebb)); + self.pos = CursorPosition::After(ebb); + } + + /// Go to the top of the next EBB in layout order and return it. + /// + /// - If the cursor wasn't pointing at anything, go to the top of the first EBB in the + /// function. + /// - If there are no more EBBs, leave the cursor pointing at nothing and return `None`. + /// + /// # Examples + /// + /// The `next_ebb()` method is intended for iterating over the EBBs in layout order: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb}; + /// # use cretonne::ir::layout::Cursor; + /// fn edit_func(func: &mut Function) { + /// let mut cursor = Cursor::new(&mut func.layout); + /// while let Some(ebb) = cursor.next_ebb() { + /// // Edit ebb. + /// } + /// } + /// ``` + pub fn next_ebb(&mut self) -> Option { + let next = if let Some(ebb) = self.current_ebb() { + self.layout.ebbs[ebb].next.wrap() + } else { + self.layout.first_ebb + }; + self.pos = match next { + Some(ebb) => CursorPosition::Before(ebb), + None => CursorPosition::Nowhere, + }; + next + } + + /// Go to the bottom of the previous EBB in layout order and return it. + /// + /// - If the cursor wasn't pointing at anything, go to the bottom of the last EBB in the + /// function. + /// - If there are no more EBBs, leave the cursor pointing at nothing and return `None`. + /// + /// # Examples + /// + /// The `prev_ebb()` method is intended for iterating over the EBBs in backwards layout order: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb}; + /// # use cretonne::ir::layout::Cursor; + /// fn edit_func(func: &mut Function) { + /// let mut cursor = Cursor::new(&mut func.layout); + /// while let Some(ebb) = cursor.prev_ebb() { + /// // Edit ebb. + /// } + /// } + /// ``` + pub fn prev_ebb(&mut self) -> Option { + let prev = if let Some(ebb) = self.current_ebb() { + self.layout.ebbs[ebb].prev.wrap() + } else { + self.layout.last_ebb + }; + self.pos = match prev { + Some(ebb) => CursorPosition::After(ebb), + None => CursorPosition::Nowhere, + }; + prev + } + + /// Move to the next instruction in the same EBB and return it. + /// + /// - If the cursor was positioned before an EBB, go to the first instruction in that EBB. + /// - If there are no more instructions in the EBB, go to the `After(ebb)` position and return + /// `None`. + /// - If the cursor wasn't pointing anywhere, keep doing that. + /// + /// This method will never move the cursor to a different EBB. + /// + /// # Examples + /// + /// The `next_inst()` method is intended for iterating over the instructions in an EBB like + /// this: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb}; + /// # use cretonne::ir::layout::Cursor; + /// fn edit_ebb(func: &mut Function, ebb: Ebb) { + /// let mut cursor = Cursor::new(&mut func.layout); + /// cursor.goto_top(ebb); + /// while let Some(inst) = cursor.next_inst() { + /// // Edit instructions... + /// } + /// } + /// ``` + /// The loop body can insert and remove instructions via the cursor. + /// + /// Iterating over all the instructions in a function looks like this: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb}; + /// # use cretonne::ir::layout::Cursor; + /// fn edit_func(func: &mut Function) { + /// let mut cursor = Cursor::new(&mut func.layout); + /// while let Some(ebb) = cursor.next_ebb() { + /// while let Some(inst) = cursor.next_inst() { + /// // Edit instructions... + /// } + /// } + /// } + /// ``` + pub fn next_inst(&mut self) -> Option { + use self::CursorPosition::*; + match self.pos { + Nowhere | After(..) => None, + At(inst) => { + if let Some(next) = self.layout.insts[inst].next.wrap() { + self.pos = At(next); + Some(next) + } else { + self.pos = + After(self.layout.inst_ebb(inst).expect("current instruction removed?")); + None + } + } + Before(ebb) => { + if let Some(next) = self.layout.ebbs[ebb].first_inst.wrap() { + self.pos = At(next); + Some(next) + } else { + self.pos = After(ebb); + None + } + } + } + } + + /// Move to the previous instruction in the same EBB and return it. + /// + /// - If the cursor was positioned after an EBB, go to the last instruction in that EBB. + /// - If there are no more instructions in the EBB, go to the `Before(ebb)` position and return + /// `None`. + /// - If the cursor wasn't pointing anywhere, keep doing that. + /// + /// This method will never move the cursor to a different EBB. + /// + /// # Examples + /// + /// The `prev_inst()` method is intended for iterating backwards over the instructions in an + /// EBB like this: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb}; + /// # use cretonne::ir::layout::Cursor; + /// fn edit_ebb(func: &mut Function, ebb: Ebb) { + /// let mut cursor = Cursor::new(&mut func.layout); + /// cursor.goto_bottom(ebb); + /// while let Some(inst) = cursor.prev_inst() { + /// // Edit instructions... + /// } + /// } + /// ``` + pub fn prev_inst(&mut self) -> Option { + use self::CursorPosition::*; + match self.pos { + Nowhere | Before(..) => None, + At(inst) => { + if let Some(prev) = self.layout.insts[inst].prev.wrap() { + self.pos = At(prev); + Some(prev) + } else { + self.pos = + Before(self.layout.inst_ebb(inst).expect("current instruction removed?")); + None + } + } + After(ebb) => { + if let Some(prev) = self.layout.ebbs[ebb].last_inst.wrap() { + self.pos = At(prev); + Some(prev) + } else { + self.pos = Before(ebb); + None + } + } + } + } +} + + #[cfg(test)] mod tests { - use super::Layout; + use super::{Layout, Cursor, CursorPosition}; use entity_map::EntityRef; use ir::{Ebb, Inst}; @@ -301,6 +565,35 @@ mod tests { } assert_eq!(v, [e1, e2, e0]); } + + // Test cursor positioning. + let mut cur = Cursor::new(&mut layout); + assert_eq!(cur.position(), CursorPosition::Nowhere); + assert_eq!(cur.next_inst(), None); + assert_eq!(cur.position(), CursorPosition::Nowhere); + assert_eq!(cur.prev_inst(), None); + assert_eq!(cur.position(), CursorPosition::Nowhere); + + assert_eq!(cur.next_ebb(), Some(e1)); + assert_eq!(cur.position(), CursorPosition::Before(e1)); + assert_eq!(cur.next_inst(), None); + assert_eq!(cur.position(), CursorPosition::After(e1)); + assert_eq!(cur.next_inst(), None); + assert_eq!(cur.position(), CursorPosition::After(e1)); + assert_eq!(cur.next_ebb(), Some(e2)); + assert_eq!(cur.prev_inst(), None); + assert_eq!(cur.position(), CursorPosition::Before(e2)); + assert_eq!(cur.next_ebb(), Some(e0)); + assert_eq!(cur.next_ebb(), None); + assert_eq!(cur.position(), CursorPosition::Nowhere); + + // Backwards through the EBBs. + assert_eq!(cur.prev_ebb(), Some(e0)); + assert_eq!(cur.position(), CursorPosition::After(e0)); + assert_eq!(cur.prev_ebb(), Some(e2)); + assert_eq!(cur.prev_ebb(), Some(e1)); + assert_eq!(cur.prev_ebb(), None); + assert_eq!(cur.position(), CursorPosition::Nowhere); } #[test] @@ -378,6 +671,30 @@ mod tests { assert_eq!(layout.inst_ebb(i2), Some(e1)); let v: Vec = layout.ebb_insts(e1).collect(); assert_eq!(v, [i1, i2, i0]); + + // Test cursor positioning. + let mut cur = Cursor::new(&mut layout); + cur.goto_top(e1); + assert_eq!(cur.position(), CursorPosition::Before(e1)); + assert_eq!(cur.prev_inst(), None); + assert_eq!(cur.position(), CursorPosition::Before(e1)); + assert_eq!(cur.next_inst(), Some(i1)); + assert_eq!(cur.position(), CursorPosition::At(i1)); + assert_eq!(cur.next_inst(), Some(i2)); + assert_eq!(cur.next_inst(), Some(i0)); + assert_eq!(cur.prev_inst(), Some(i2)); + assert_eq!(cur.position(), CursorPosition::At(i2)); + assert_eq!(cur.next_inst(), Some(i0)); + assert_eq!(cur.position(), CursorPosition::At(i0)); + assert_eq!(cur.next_inst(), None); + assert_eq!(cur.position(), CursorPosition::After(e1)); + assert_eq!(cur.next_inst(), None); + assert_eq!(cur.position(), CursorPosition::After(e1)); + assert_eq!(cur.prev_inst(), Some(i0)); + assert_eq!(cur.prev_inst(), Some(i2)); + assert_eq!(cur.prev_inst(), Some(i1)); + assert_eq!(cur.prev_inst(), None); + assert_eq!(cur.position(), CursorPosition::Before(e1)); } #[test] diff --git a/cranelift/src/libcretonne/ir/mod.rs b/cranelift/src/libcretonne/ir/mod.rs index 36d17c29aa..bbbc6d26ab 100644 --- a/cranelift/src/libcretonne/ir/mod.rs +++ b/cranelift/src/libcretonne/ir/mod.rs @@ -22,6 +22,6 @@ pub use ir::instructions::{Opcode, InstructionData, VariableArgs}; pub use ir::stackslot::StackSlotData; pub use ir::jumptable::JumpTableData; pub use ir::dfg::{DataFlowGraph, ValueDef}; -pub use ir::layout::Layout; +pub use ir::layout::{Layout, Cursor}; pub use ir::function::Function; pub use ir::builder::Builder; From 54a4b9b0d79a90d9f53adfa503f8682e7770de4d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 14 Oct 2016 10:09:57 -0700 Subject: [PATCH 0352/3084] Add Cursor::insert_inst(). Insert an instruction at thecurrent cursor position. --- cranelift/src/libcretonne/ir/layout.rs | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/cranelift/src/libcretonne/ir/layout.rs b/cranelift/src/libcretonne/ir/layout.rs index 1b78386100..48943ffc1c 100644 --- a/cranelift/src/libcretonne/ir/layout.rs +++ b/cranelift/src/libcretonne/ir/layout.rs @@ -511,6 +511,24 @@ impl<'a> Cursor<'a> { } } } + + /// Insert an instruction at the current position. + /// + /// - If pointing at an instruction, the new instruction is inserted before the current + /// instruction. + /// - If pointing at the bottom of an EBB, the new instruction is appended to the EBB. + /// - Otherwise panic. + /// + /// In either case, the cursor is not moved, such that repeates calls to `insert_inst()` causes + /// instructions to appear in insertion order in the EBB. + pub fn insert_inst(&mut self, inst: Inst) { + use self::CursorPosition::*; + match self.pos { + Nowhere | Before(..) => panic!("Invalid insert_inst position"), + At(cur) => self.layout.insert_inst(inst, cur), + After(ebb) => self.layout.append_inst(inst, ebb), + } + } } From d1fd91021d339155a574eb8c0fa42f2f1dcbcc7b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 14 Oct 2016 12:33:54 -0700 Subject: [PATCH 0353/3084] Add Layout::split_ebb(). This method splits an EBB in two and moves trailing instructions to a new EBB. --- cranelift/src/libcretonne/ir/layout.rs | 140 +++++++++++++++++++++++++ 1 file changed, 140 insertions(+) diff --git a/cranelift/src/libcretonne/ir/layout.rs b/cranelift/src/libcretonne/ir/layout.rs index 48943ffc1c..b8d10739e5 100644 --- a/cranelift/src/libcretonne/ir/layout.rs +++ b/cranelift/src/libcretonne/ir/layout.rs @@ -221,6 +221,71 @@ impl Layout { next: self.ebbs[ebb].first_inst.wrap(), } } + + /// Split the EBB containing `before` in two. + /// + /// Insert `new_ebb` after the old EBB and move `before` and the following instructions to + /// `new_ebb`: + /// + /// ```text + /// old_ebb: + /// i1 + /// i2 + /// i3 << before + /// i4 + /// ``` + /// becomes: + /// + /// ```text + /// old_ebb: + /// i1 + /// i2 + /// new_ebb: + /// i3 << before + /// i4 + /// ``` + pub fn split_ebb(&mut self, new_ebb: Ebb, before: Inst) { + let old_ebb = self.inst_ebb(before) + .expect("The `before` instruction must be in the layout"); + assert!(!self.is_ebb_inserted(new_ebb)); + + // Insert new_ebb after old_ebb. + let next_ebb = self.ebbs[old_ebb].next; + let last_inst = self.ebbs[old_ebb].last_inst; + { + let node = self.ebbs.ensure(new_ebb); + node.prev = old_ebb; + node.next = next_ebb; + node.first_inst = before; + node.last_inst = last_inst; + } + self.ebbs[old_ebb].next = new_ebb; + + // Fix backwards link. + if Some(old_ebb) == self.last_ebb { + self.last_ebb = Some(new_ebb); + } else { + self.ebbs[next_ebb].prev = new_ebb; + } + + // Disconnect the instruction links. + let prev_inst = self.insts[before].prev; + self.insts[before].prev = NO_INST; + self.ebbs[old_ebb].last_inst = prev_inst; + if prev_inst == NO_INST { + self.ebbs[old_ebb].first_inst = NO_INST; + } else { + self.insts[prev_inst].next = NO_INST; + } + + // Fix the instruction -> ebb pointers. + let mut i = before; + while i != NO_INST { + debug_assert_eq!(self.insts[i].ebb, old_ebb); + self.insts[i].ebb = new_ebb; + i = self.insts[i].next; + } + } } #[derive(Clone, Debug, Default)] @@ -782,4 +847,79 @@ mod tests { assert_eq!(v0, [i0, i1]); assert_eq!(v1, [i2, i3]); } + + #[test] + fn split_ebb() { + let mut layout = Layout::new(); + + let e0 = Ebb::new(0); + let e1 = Ebb::new(1); + let e2 = Ebb::new(2); + + let i0 = Inst::new(0); + let i1 = Inst::new(1); + let i2 = Inst::new(2); + let i3 = Inst::new(3); + + layout.append_ebb(e0); + layout.append_inst(i0, e0); + assert_eq!(layout.inst_ebb(i0), Some(e0)); + layout.split_ebb(e1, i0); + assert_eq!(layout.inst_ebb(i0), Some(e1)); + + { + let mut cur = Cursor::new(&mut layout); + assert_eq!(cur.next_ebb(), Some(e0)); + assert_eq!(cur.next_inst(), None); + assert_eq!(cur.next_ebb(), Some(e1)); + assert_eq!(cur.next_inst(), Some(i0)); + assert_eq!(cur.next_inst(), None); + assert_eq!(cur.next_ebb(), None); + + // Check backwards links. + assert_eq!(cur.prev_ebb(), Some(e1)); + assert_eq!(cur.prev_inst(), Some(i0)); + assert_eq!(cur.prev_inst(), None); + assert_eq!(cur.prev_ebb(), Some(e0)); + assert_eq!(cur.prev_inst(), None); + assert_eq!(cur.prev_ebb(), None); + } + + layout.append_inst(i1, e0); + layout.append_inst(i2, e0); + layout.append_inst(i3, e0); + layout.split_ebb(e2, i2); + + assert_eq!(layout.inst_ebb(i0), Some(e1)); + assert_eq!(layout.inst_ebb(i1), Some(e0)); + assert_eq!(layout.inst_ebb(i2), Some(e2)); + assert_eq!(layout.inst_ebb(i3), Some(e2)); + + { + let mut cur = Cursor::new(&mut layout); + assert_eq!(cur.next_ebb(), Some(e0)); + assert_eq!(cur.next_inst(), Some(i1)); + assert_eq!(cur.next_inst(), None); + assert_eq!(cur.next_ebb(), Some(e2)); + assert_eq!(cur.next_inst(), Some(i2)); + assert_eq!(cur.next_inst(), Some(i3)); + assert_eq!(cur.next_inst(), None); + assert_eq!(cur.next_ebb(), Some(e1)); + assert_eq!(cur.next_inst(), Some(i0)); + assert_eq!(cur.next_inst(), None); + assert_eq!(cur.next_ebb(), None); + + assert_eq!(cur.prev_ebb(), Some(e1)); + assert_eq!(cur.prev_inst(), Some(i0)); + assert_eq!(cur.prev_inst(), None); + assert_eq!(cur.prev_ebb(), Some(e2)); + assert_eq!(cur.prev_inst(), Some(i3)); + assert_eq!(cur.prev_inst(), Some(i2)); + assert_eq!(cur.prev_inst(), None); + assert_eq!(cur.prev_ebb(), Some(e0)); + assert_eq!(cur.prev_inst(), Some(i1)); + assert_eq!(cur.prev_inst(), None); + assert_eq!(cur.prev_ebb(), None); + } + } } From a41b34c11ce7c83715e6e73d9db249128342000f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 14 Oct 2016 13:54:26 -0700 Subject: [PATCH 0354/3084] Add Layout::insert_ebb_after() method. Symmetrical with the existing insert_ebb(). --- cranelift/src/libcretonne/ir/layout.rs | 91 ++++++++++++++++++++------ 1 file changed, 72 insertions(+), 19 deletions(-) diff --git a/cranelift/src/libcretonne/ir/layout.rs b/cranelift/src/libcretonne/ir/layout.rs index b8d10739e5..b1263a5b47 100644 --- a/cranelift/src/libcretonne/ir/layout.rs +++ b/cranelift/src/libcretonne/ir/layout.rs @@ -102,6 +102,26 @@ impl Layout { } } + /// Insert `ebb` in the layout *after* the existing EBB `after`. + pub fn insert_ebb_after(&mut self, ebb: Ebb, after: Ebb) { + assert!(!self.is_ebb_inserted(ebb), + "Cannot insert EBB that is already in the layout"); + assert!(self.is_ebb_inserted(after), + "EBB Insertion point not in the layout"); + let before = self.ebbs[after].next; + { + let node = self.ebbs.ensure(ebb); + node.next = before; + node.prev = after; + } + self.ebbs[after].next = ebb; + if before == NO_EBB { + self.last_ebb = Some(ebb); + } else { + self.ebbs[before].prev = ebb; + } + } + /// Return an iterator over all EBBs in layout order. pub fn ebbs<'a>(&'a self) -> Ebbs<'a> { Ebbs { @@ -603,6 +623,37 @@ mod tests { use entity_map::EntityRef; use ir::{Ebb, Inst}; + fn verify(layout: &mut Layout, ebbs: &[(Ebb, &[Inst])]) { + // Check that EBBs are inserted and instructions belong the right places. + // Check forward linkage with iterators. + { + let mut ebb_iter = layout.ebbs(); + for &(ebb, insts) in ebbs { + assert!(layout.is_ebb_inserted(ebb)); + assert_eq!(ebb_iter.next(), Some(ebb)); + + let mut inst_iter = layout.ebb_insts(ebb); + for &inst in insts { + assert_eq!(layout.inst_ebb(inst), Some(ebb)); + assert_eq!(inst_iter.next(), Some(inst)); + } + assert_eq!(inst_iter.next(), None); + } + assert_eq!(ebb_iter.next(), None); + } + + // Check backwards linkage with a cursor. + let mut cur = Cursor::new(layout); + for &(ebb, insts) in ebbs.into_iter().rev() { + assert_eq!(cur.prev_ebb(), Some(ebb)); + for &inst in insts.into_iter().rev() { + assert_eq!(cur.prev_inst(), Some(inst)); + } + assert_eq!(cur.prev_inst(), None); + } + assert_eq!(cur.prev_ebb(), None); + } + #[test] fn append_ebb() { let mut layout = Layout::new(); @@ -614,10 +665,8 @@ mod tests { let imm = &layout; assert!(!imm.is_ebb_inserted(e0)); assert!(!imm.is_ebb_inserted(e1)); - - let v: Vec = layout.ebbs().collect(); - assert_eq!(v, []); } + verify(&mut layout, &[]); layout.append_ebb(e1); assert!(!layout.is_ebb_inserted(e0)); @@ -699,22 +748,34 @@ mod tests { assert!(!layout.is_ebb_inserted(e0)); assert!(layout.is_ebb_inserted(e1)); assert!(!layout.is_ebb_inserted(e2)); - let v: Vec = layout.ebbs().collect(); - assert_eq!(v, [e1]); + verify(&mut layout, &[(e1, &[])]); layout.insert_ebb(e2, e1); assert!(!layout.is_ebb_inserted(e0)); assert!(layout.is_ebb_inserted(e1)); assert!(layout.is_ebb_inserted(e2)); - let v: Vec = layout.ebbs().collect(); - assert_eq!(v, [e2, e1]); + verify(&mut layout, &[(e2, &[]), (e1, &[])]); layout.insert_ebb(e0, e1); assert!(layout.is_ebb_inserted(e0)); assert!(layout.is_ebb_inserted(e1)); assert!(layout.is_ebb_inserted(e2)); - let v: Vec = layout.ebbs().collect(); - assert_eq!(v, [e2, e0, e1]); + verify(&mut layout, &[(e2, &[]), (e0, &[]), (e1, &[])]); + } + + #[test] + fn insert_ebb_after() { + let mut layout = Layout::new(); + let e0 = Ebb::new(0); + let e1 = Ebb::new(1); + let e2 = Ebb::new(2); + + layout.append_ebb(e1); + layout.insert_ebb_after(e2, e1); + verify(&mut layout, &[(e1, &[]), (e2, &[])]); + + layout.insert_ebb_after(e0, e1); + verify(&mut layout, &[(e1, &[]), (e0, &[]), (e2, &[])]); } #[test] @@ -749,11 +810,7 @@ mod tests { assert_eq!(v, [i1, i2]); layout.append_inst(i0, e1); - assert_eq!(layout.inst_ebb(i0), Some(e1)); - assert_eq!(layout.inst_ebb(i1), Some(e1)); - assert_eq!(layout.inst_ebb(i2), Some(e1)); - let v: Vec = layout.ebb_insts(e1).collect(); - assert_eq!(v, [i1, i2, i0]); + verify(&mut layout, &[(e1, &[i1, i2, i0])]); // Test cursor positioning. let mut cur = Cursor::new(&mut layout); @@ -812,11 +869,7 @@ mod tests { assert_eq!(v, [i2, i1]); layout.insert_inst(i0, i1); - assert_eq!(layout.inst_ebb(i0), Some(e1)); - assert_eq!(layout.inst_ebb(i1), Some(e1)); - assert_eq!(layout.inst_ebb(i2), Some(e1)); - let v: Vec = layout.ebb_insts(e1).collect(); - assert_eq!(v, [i2, i0, i1]); + verify(&mut layout, &[(e1, &[i2, i0, i1])]); } #[test] From 7aef6bd535eee62d7a1c827e4476f3cd04e2c0bd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 14 Oct 2016 14:01:49 -0700 Subject: [PATCH 0355/3084] Add Cursor::insert_ebb() method. Insert an EBB at the current position and switch to inserting instructions there. Call the relevant Layout methods to do so. --- cranelift/src/libcretonne/ir/layout.rs | 30 ++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/cranelift/src/libcretonne/ir/layout.rs b/cranelift/src/libcretonne/ir/layout.rs index b1263a5b47..c6543a88e0 100644 --- a/cranelift/src/libcretonne/ir/layout.rs +++ b/cranelift/src/libcretonne/ir/layout.rs @@ -614,6 +614,36 @@ impl<'a> Cursor<'a> { After(ebb) => self.layout.append_inst(inst, ebb), } } + + /// Insert an EBB at the current position and switch to it. + /// + /// As far as possible, this method behaves as if the EBB header were an instruction inserted + /// at the current position. + /// + /// - If the cursor is pointing at an existing instruction, *the current EBB is split in two* + /// and the current instruction becomes the first instruction in the inserted EBB. + /// - If the cursor points at the bottom of an EBB, the new EBB is inserted after the current + /// one, and moved to the bottom of the new EBB where instructions can be appended. + /// - If the cursor points to the top of an EBB, the new EBB is inserted above the current one. + /// - If the cursor is not pointing at anything, the new EBB is placed last in the layout. + /// + /// This means that is is always valid to call this method, and it always leaves the cursor in + /// a state that will insert instructions into the new EBB. + pub fn insert_ebb(&mut self, new_ebb: Ebb) { + use self::CursorPosition::*; + match self.pos { + At(inst) => { + self.layout.split_ebb(new_ebb, inst); + // All other cases move to `After(ebb)`, but in this case we we'll stay `At(inst)`. + return; + } + Nowhere => self.layout.append_ebb(new_ebb), + Before(ebb) => self.layout.insert_ebb(new_ebb, ebb), + After(ebb) => self.layout.insert_ebb_after(new_ebb, ebb), + } + // For everything but `At(inst)` we end up appending to the new EBB. + self.pos = After(new_ebb); + } } From 8fb8a41f8ecd256d1d42005af3c942ca4146e0db Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 14 Oct 2016 15:02:58 -0700 Subject: [PATCH 0356/3084] Give Builder a Cursor. The Builder keeps track of a position in the layout and inserts new instructions there. Add insert_ebb() and ebb() methods to Builder. Use Builder in the cfg tests. --- cranelift/src/libcretonne/cfg.rs | 32 ++++++++++++++----------- cranelift/src/libcretonne/ir/builder.rs | 32 +++++++++++++++++++++++-- 2 files changed, 48 insertions(+), 16 deletions(-) diff --git a/cranelift/src/libcretonne/cfg.rs b/cranelift/src/libcretonne/cfg.rs index 8397ca5166..aab576ae82 100644 --- a/cranelift/src/libcretonne/cfg.rs +++ b/cranelift/src/libcretonne/cfg.rs @@ -139,9 +139,7 @@ impl ControlFlowGraph { #[cfg(test)] mod tests { use super::*; - use ir::Function; - - use test_utils::make_inst; + use ir::{Function, Builder, Cursor, VariableArgs, types}; #[test] fn empty() { @@ -176,23 +174,29 @@ mod tests { fn branches_and_jumps() { let mut func = Function::new(); let ebb0 = func.dfg.make_ebb(); + let cond = func.dfg.append_ebb_arg(ebb0, types::I32); let ebb1 = func.dfg.make_ebb(); let ebb2 = func.dfg.make_ebb(); - func.layout.append_ebb(ebb0); - func.layout.append_ebb(ebb1); - func.layout.append_ebb(ebb2); - let br_ebb0_ebb2 = make_inst::branch(&mut func, ebb2); - func.layout.append_inst(br_ebb0_ebb2, ebb0); + let br_ebb0_ebb2; + let br_ebb1_ebb1; + let jmp_ebb0_ebb1; + let jmp_ebb1_ebb2; - let jmp_ebb0_ebb1 = make_inst::jump(&mut func, ebb1); - func.layout.append_inst(jmp_ebb0_ebb1, ebb0); + { + let mut cursor = Cursor::new(&mut func.layout); + let mut b = Builder::new(&mut func.dfg, &mut cursor); - let br_ebb1_ebb1 = make_inst::branch(&mut func, ebb1); - func.layout.append_inst(br_ebb1_ebb1, ebb1); + b.insert_ebb(ebb0); + br_ebb0_ebb2 = b.brnz(cond, ebb2, VariableArgs::new()); + jmp_ebb0_ebb1 = b.jump(ebb1, VariableArgs::new()); - let jmp_ebb1_ebb2 = make_inst::jump(&mut func, ebb2); - func.layout.append_inst(jmp_ebb1_ebb2, ebb1); + b.insert_ebb(ebb1); + br_ebb1_ebb1 = b.brnz(cond, ebb1, VariableArgs::new()); + jmp_ebb1_ebb2 = b.jump(ebb2, VariableArgs::new()); + + b.insert_ebb(ebb2); + } let cfg = ControlFlowGraph::new(&func); diff --git a/cranelift/src/libcretonne/ir/builder.rs b/cranelift/src/libcretonne/ir/builder.rs index 4e81027e9b..e31be0f2ca 100644 --- a/cranelift/src/libcretonne/ir/builder.rs +++ b/cranelift/src/libcretonne/ir/builder.rs @@ -4,20 +4,48 @@ //! function. Many of its methods are generated from the meta language instruction definitions. use ir::{types, instructions}; -use ir::{InstructionData, DataFlowGraph}; +use ir::{InstructionData, DataFlowGraph, Cursor}; use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, VariableArgs, FuncRef}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, ImmVector}; use ir::condcodes::{IntCC, FloatCC}; +/// Instruction builder. +/// +/// A `Builder` holds mutable references to a data flow graph and a layout cursor. It provides +/// convenience method for creating and inserting instructions at the current cursor position. pub struct Builder<'a> { - dfg: &'a mut DataFlowGraph, + pub dfg: &'a mut DataFlowGraph, + pub pos: &'a mut Cursor<'a>, } impl<'a> Builder<'a> { + /// Create a new builder which inserts instructions at `pos`. + /// The `dfg` and `pos.layout` references should be from the same `Function`. + pub fn new(dfg: &'a mut DataFlowGraph, pos: &'a mut Cursor<'a>) -> Builder<'a> { + Builder { + dfg: dfg, + pos: pos, + } + } + + /// Create and insert an EBB. Further instructions will be inserted into the new EBB. + pub fn ebb(&mut self) -> Ebb { + let ebb = self.dfg.make_ebb(); + self.insert_ebb(ebb); + ebb + } + + /// Insert an existing EBB at the current position. Further instructions will be inserted into + /// the new EBB. + pub fn insert_ebb(&mut self, ebb: Ebb) { + self.pos.insert_ebb(ebb); + } + // Create and insert an instruction. // This method is used by the generated format-specific methods. fn insert_inst(&mut self, data: InstructionData) -> Inst { let inst = self.dfg.make_inst(data); + self.pos.insert_inst(inst); inst } } From 98308177cc5198746e30007feb2a5febe26f6181 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 14 Oct 2016 15:16:29 -0700 Subject: [PATCH 0357/3084] Switch domtree tests to using Builder. --- cranelift/src/libcretonne/dominator_tree.rs | 35 ++++++++++++--------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/cranelift/src/libcretonne/dominator_tree.rs b/cranelift/src/libcretonne/dominator_tree.rs index c741b5992d..0e34a2c021 100644 --- a/cranelift/src/libcretonne/dominator_tree.rs +++ b/cranelift/src/libcretonne/dominator_tree.rs @@ -116,10 +116,9 @@ impl DominatorTree { #[cfg(test)] mod test { use super::*; - use ir::Function; + use ir::{Function, Builder, Cursor, VariableArgs, types}; use ir::entities::NO_INST; use cfg::ControlFlowGraph; - use test_utils::make_inst; #[test] fn empty() { @@ -133,23 +132,31 @@ mod test { fn non_zero_entry_block() { let mut func = Function::new(); let ebb3 = func.dfg.make_ebb(); + let cond = func.dfg.append_ebb_arg(ebb3, types::I32); let ebb1 = func.dfg.make_ebb(); let ebb2 = func.dfg.make_ebb(); let ebb0 = func.dfg.make_ebb(); - func.layout.append_ebb(ebb3); - func.layout.append_ebb(ebb1); - func.layout.append_ebb(ebb2); - func.layout.append_ebb(ebb0); - let jmp_ebb3_ebb1 = make_inst::jump(&mut func, ebb1); - let br_ebb1_ebb0 = make_inst::branch(&mut func, ebb0); - let jmp_ebb1_ebb2 = make_inst::jump(&mut func, ebb2); - let jmp_ebb2_ebb0 = make_inst::jump(&mut func, ebb0); + let jmp_ebb3_ebb1; + let br_ebb1_ebb0; + let jmp_ebb1_ebb2; - func.layout.append_inst(br_ebb1_ebb0, ebb1); - func.layout.append_inst(jmp_ebb1_ebb2, ebb1); - func.layout.append_inst(jmp_ebb2_ebb0, ebb2); - func.layout.append_inst(jmp_ebb3_ebb1, ebb3); + { + let mut cursor = Cursor::new(&mut func.layout); + let mut b = Builder::new(&mut func.dfg, &mut cursor); + + b.insert_ebb(ebb3); + jmp_ebb3_ebb1 = b.jump(ebb1, VariableArgs::new()); + + b.insert_ebb(ebb1); + br_ebb1_ebb0 = b.brnz(cond, ebb0, VariableArgs::new()); + jmp_ebb1_ebb2 = b.jump(ebb2, VariableArgs::new()); + + b.insert_ebb(ebb2); + b.jump(ebb0, VariableArgs::new()); + + b.insert_ebb(ebb0); + } let cfg = ControlFlowGraph::new(&func); let dt = DominatorTree::new(&cfg); From 680387a53ad8b3b68766652cc9167f8d958131c8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 14 Oct 2016 15:17:34 -0700 Subject: [PATCH 0358/3084] Remove test_utils. These test utilities have been subsumed by ir::Builder. --- cranelift/src/libcretonne/lib.rs | 3 -- .../src/libcretonne/test_utils/make_inst.rs | 37 ------------------- cranelift/src/libcretonne/test_utils/mod.rs | 3 -- 3 files changed, 43 deletions(-) delete mode 100644 cranelift/src/libcretonne/test_utils/make_inst.rs delete mode 100644 cranelift/src/libcretonne/test_utils/mod.rs diff --git a/cranelift/src/libcretonne/lib.rs b/cranelift/src/libcretonne/lib.rs index 26a87bd223..ae25157248 100644 --- a/cranelift/src/libcretonne/lib.rs +++ b/cranelift/src/libcretonne/lib.rs @@ -23,6 +23,3 @@ mod write; mod constant_hash; mod predicates; mod legalizer; - -#[cfg(test)] -pub mod test_utils; diff --git a/cranelift/src/libcretonne/test_utils/make_inst.rs b/cranelift/src/libcretonne/test_utils/make_inst.rs deleted file mode 100644 index 9ab4429b76..0000000000 --- a/cranelift/src/libcretonne/test_utils/make_inst.rs +++ /dev/null @@ -1,37 +0,0 @@ -//! Helper functions for generating dummy instructions. - -use ir::{Function, Ebb, Inst, Opcode}; -use ir::entities::NO_VALUE; -use ir::instructions::{InstructionData, ReturnData, VariableArgs, JumpData, BranchData}; -use ir::types; - -pub fn jump(func: &mut Function, dest: Ebb) -> Inst { - func.dfg.make_inst(InstructionData::Jump { - opcode: Opcode::Jump, - ty: types::VOID, - data: Box::new(JumpData { - destination: dest, - varargs: VariableArgs::new(), - }), - }) -} - -pub fn branch(func: &mut Function, dest: Ebb) -> Inst { - func.dfg.make_inst(InstructionData::Branch { - opcode: Opcode::Brz, - ty: types::VOID, - data: Box::new(BranchData { - arg: NO_VALUE, - destination: dest, - varargs: VariableArgs::new(), - }), - }) -} - -pub fn ret(func: &mut Function) -> Inst { - func.dfg.make_inst(InstructionData::Return { - opcode: Opcode::Return, - ty: types::VOID, - data: Box::new(ReturnData { varargs: VariableArgs::new() }), - }) -} diff --git a/cranelift/src/libcretonne/test_utils/mod.rs b/cranelift/src/libcretonne/test_utils/mod.rs deleted file mode 100644 index 6273573d6d..0000000000 --- a/cranelift/src/libcretonne/test_utils/mod.rs +++ /dev/null @@ -1,3 +0,0 @@ -//! Test utility functions. - -pub mod make_inst; From af6355e2ef1f7e57a9277839bb5c0ce1875c2782 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 17 Oct 2016 14:16:43 -0700 Subject: [PATCH 0359/3084] Move the 'meta' dir to 'lib/cretonne/meta'. The 'lib/cretonne' directory will be the new root of a stand-alone cretonne crate containg both Python and Rust sources. This is in preparation for publishing crates on crates.io. --- cranelift/docs/Makefile | 2 +- cranelift/docs/conf.py | 2 +- cranelift/docs/metaref.rst | 16 +++++++++------- cranelift/src/libcretonne/build.rs | 2 +- {meta => lib/cretonne/meta}/build.py | 0 {meta => lib/cretonne/meta}/check.sh | 0 {meta => lib/cretonne/meta}/constant_hash.py | 0 {meta => lib/cretonne/meta}/cretonne/__init__.py | 0 {meta => lib/cretonne/meta}/cretonne/ast.py | 0 {meta => lib/cretonne/meta}/cretonne/base.py | 0 {meta => lib/cretonne/meta}/cretonne/entities.py | 0 {meta => lib/cretonne/meta}/cretonne/formats.py | 0 .../cretonne/meta}/cretonne/immediates.py | 0 {meta => lib/cretonne/meta}/cretonne/legalize.py | 0 .../cretonne/meta}/cretonne/predicates.py | 0 {meta => lib/cretonne/meta}/cretonne/settings.py | 0 {meta => lib/cretonne/meta}/cretonne/test_ast.py | 0 .../cretonne/meta}/cretonne/test_typevar.py | 0 .../cretonne/meta}/cretonne/test_xform.py | 0 {meta => lib/cretonne/meta}/cretonne/types.py | 0 {meta => lib/cretonne/meta}/cretonne/typevar.py | 0 {meta => lib/cretonne/meta}/cretonne/xform.py | 0 {meta => lib/cretonne/meta}/gen_build_deps.py | 0 {meta => lib/cretonne/meta}/gen_encoding.py | 0 {meta => lib/cretonne/meta}/gen_instr.py | 0 {meta => lib/cretonne/meta}/gen_settings.py | 0 {meta => lib/cretonne/meta}/gen_types.py | 0 {meta => lib/cretonne/meta}/isa/__init__.py | 0 .../cretonne/meta}/isa/riscv/__init__.py | 0 {meta => lib/cretonne/meta}/isa/riscv/defs.py | 0 .../cretonne/meta}/isa/riscv/encodings.py | 0 {meta => lib/cretonne/meta}/isa/riscv/recipes.py | 0 .../cretonne/meta}/isa/riscv/settings.py | 0 {meta => lib/cretonne/meta}/srcgen.py | 0 .../cretonne/meta}/test_constant_hash.py | 0 {meta => lib/cretonne/meta}/test_srcgen.py | 0 {meta => lib/cretonne/meta}/unique_table.py | 0 37 files changed, 12 insertions(+), 10 deletions(-) rename {meta => lib/cretonne/meta}/build.py (100%) rename {meta => lib/cretonne/meta}/check.sh (100%) rename {meta => lib/cretonne/meta}/constant_hash.py (100%) rename {meta => lib/cretonne/meta}/cretonne/__init__.py (100%) rename {meta => lib/cretonne/meta}/cretonne/ast.py (100%) rename {meta => lib/cretonne/meta}/cretonne/base.py (100%) rename {meta => lib/cretonne/meta}/cretonne/entities.py (100%) rename {meta => lib/cretonne/meta}/cretonne/formats.py (100%) rename {meta => lib/cretonne/meta}/cretonne/immediates.py (100%) rename {meta => lib/cretonne/meta}/cretonne/legalize.py (100%) rename {meta => lib/cretonne/meta}/cretonne/predicates.py (100%) rename {meta => lib/cretonne/meta}/cretonne/settings.py (100%) rename {meta => lib/cretonne/meta}/cretonne/test_ast.py (100%) rename {meta => lib/cretonne/meta}/cretonne/test_typevar.py (100%) rename {meta => lib/cretonne/meta}/cretonne/test_xform.py (100%) rename {meta => lib/cretonne/meta}/cretonne/types.py (100%) rename {meta => lib/cretonne/meta}/cretonne/typevar.py (100%) rename {meta => lib/cretonne/meta}/cretonne/xform.py (100%) rename {meta => lib/cretonne/meta}/gen_build_deps.py (100%) rename {meta => lib/cretonne/meta}/gen_encoding.py (100%) rename {meta => lib/cretonne/meta}/gen_instr.py (100%) rename {meta => lib/cretonne/meta}/gen_settings.py (100%) rename {meta => lib/cretonne/meta}/gen_types.py (100%) rename {meta => lib/cretonne/meta}/isa/__init__.py (100%) rename {meta => lib/cretonne/meta}/isa/riscv/__init__.py (100%) rename {meta => lib/cretonne/meta}/isa/riscv/defs.py (100%) rename {meta => lib/cretonne/meta}/isa/riscv/encodings.py (100%) rename {meta => lib/cretonne/meta}/isa/riscv/recipes.py (100%) rename {meta => lib/cretonne/meta}/isa/riscv/settings.py (100%) rename {meta => lib/cretonne/meta}/srcgen.py (100%) rename {meta => lib/cretonne/meta}/test_constant_hash.py (100%) rename {meta => lib/cretonne/meta}/test_srcgen.py (100%) rename {meta => lib/cretonne/meta}/unique_table.py (100%) diff --git a/cranelift/docs/Makefile b/cranelift/docs/Makefile index 56137e8b2c..b2443f68b7 100644 --- a/cranelift/docs/Makefile +++ b/cranelift/docs/Makefile @@ -58,7 +58,7 @@ html: @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." autohtml: html - $(SPHINXABUILD) -z ../meta --ignore '*.swp' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html + $(SPHINXABUILD) -z ../lib/cretonne/meta --ignore '*.swp' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py index 5602561828..0603a27bfd 100644 --- a/cranelift/docs/conf.py +++ b/cranelift/docs/conf.py @@ -24,7 +24,7 @@ sys.path.insert(0, os.path.abspath('.')) # Also add the meta directory to sys.path so autodoc can find the Cretonne meta # language definitions. -sys.path.insert(0, os.path.abspath('../meta')) +sys.path.insert(0, os.path.abspath('../lib/cretonne/meta')) # -- General configuration ------------------------------------------------ diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 4edf708041..cc6d7173d6 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -10,8 +10,9 @@ The Cretonne meta language is used to define instructions for Cretonne. It is a domain specific language embedded in Python. This document describes the Python modules that form the embedded DSL. -The meta language descriptions are Python modules under the :file:`meta` -top-level directory. The descriptions are processed in two steps: +The meta language descriptions are Python modules under the +:file:`lib/cretonne/meta` directory. The descriptions are processed in two +steps: 1. The Python modules are imported. This has the effect of building static data structures in global variables in the modules. These static data structures @@ -22,8 +23,9 @@ top-level directory. The descriptions are processed in two steps: constant tables. The main driver for this source code generation process is the -:file:`meta/build.py` script which is invoked as part of the build process if -anything in the :file:`meta` directory has changed since the last build. +:file:`lib/cretonne/meta/build.py` script which is invoked as part of the build +process if anything in the :file:`lib/cretonne/meta` directory has changed +since the last build. Settings @@ -33,8 +35,8 @@ Settings are used by the environment embedding Cretonne to control the details of code generation. Each setting is defined in the meta language so a compact and consistent Rust representation can be generated. Shared settings are defined in the :mod:`cretonne.settings` module. Some settings are specific to a target -ISA, and defined in a `settings` module under the appropriate :file:`meta/isa/*` -directory. +ISA, and defined in a `settings` module under the appropriate +:file:`lib/cretonne/meta/isa/*` directory. Settings can take boolean on/off values, small numbers, or explicitly enumerated symbolic values. Each type is represented by a sub-class of :class:`Setting`: @@ -343,7 +345,7 @@ architectures. Each ISA is represented by a :py:class:`cretonne.TargetISA` insta .. autoclass:: TargetISA The definitions for each supported target live in a package under -:file:`meta/isa`. +:file:`lib/cretonne/meta/isa`. .. automodule:: isa :members: diff --git a/cranelift/src/libcretonne/build.rs b/cranelift/src/libcretonne/build.rs index c275e33e53..2cb13ac959 100644 --- a/cranelift/src/libcretonne/build.rs +++ b/cranelift/src/libcretonne/build.rs @@ -29,7 +29,7 @@ fn main() { let top_dir = cur_dir.as_path(); // Scripts are in $top_dir/meta. - let meta_dir = top_dir.join("meta"); + let meta_dir = top_dir.join("lib/cretonne/meta"); let build_script = meta_dir.join("build.py"); // Launch build script with Python. We'll just find python in the path. diff --git a/meta/build.py b/lib/cretonne/meta/build.py similarity index 100% rename from meta/build.py rename to lib/cretonne/meta/build.py diff --git a/meta/check.sh b/lib/cretonne/meta/check.sh similarity index 100% rename from meta/check.sh rename to lib/cretonne/meta/check.sh diff --git a/meta/constant_hash.py b/lib/cretonne/meta/constant_hash.py similarity index 100% rename from meta/constant_hash.py rename to lib/cretonne/meta/constant_hash.py diff --git a/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py similarity index 100% rename from meta/cretonne/__init__.py rename to lib/cretonne/meta/cretonne/__init__.py diff --git a/meta/cretonne/ast.py b/lib/cretonne/meta/cretonne/ast.py similarity index 100% rename from meta/cretonne/ast.py rename to lib/cretonne/meta/cretonne/ast.py diff --git a/meta/cretonne/base.py b/lib/cretonne/meta/cretonne/base.py similarity index 100% rename from meta/cretonne/base.py rename to lib/cretonne/meta/cretonne/base.py diff --git a/meta/cretonne/entities.py b/lib/cretonne/meta/cretonne/entities.py similarity index 100% rename from meta/cretonne/entities.py rename to lib/cretonne/meta/cretonne/entities.py diff --git a/meta/cretonne/formats.py b/lib/cretonne/meta/cretonne/formats.py similarity index 100% rename from meta/cretonne/formats.py rename to lib/cretonne/meta/cretonne/formats.py diff --git a/meta/cretonne/immediates.py b/lib/cretonne/meta/cretonne/immediates.py similarity index 100% rename from meta/cretonne/immediates.py rename to lib/cretonne/meta/cretonne/immediates.py diff --git a/meta/cretonne/legalize.py b/lib/cretonne/meta/cretonne/legalize.py similarity index 100% rename from meta/cretonne/legalize.py rename to lib/cretonne/meta/cretonne/legalize.py diff --git a/meta/cretonne/predicates.py b/lib/cretonne/meta/cretonne/predicates.py similarity index 100% rename from meta/cretonne/predicates.py rename to lib/cretonne/meta/cretonne/predicates.py diff --git a/meta/cretonne/settings.py b/lib/cretonne/meta/cretonne/settings.py similarity index 100% rename from meta/cretonne/settings.py rename to lib/cretonne/meta/cretonne/settings.py diff --git a/meta/cretonne/test_ast.py b/lib/cretonne/meta/cretonne/test_ast.py similarity index 100% rename from meta/cretonne/test_ast.py rename to lib/cretonne/meta/cretonne/test_ast.py diff --git a/meta/cretonne/test_typevar.py b/lib/cretonne/meta/cretonne/test_typevar.py similarity index 100% rename from meta/cretonne/test_typevar.py rename to lib/cretonne/meta/cretonne/test_typevar.py diff --git a/meta/cretonne/test_xform.py b/lib/cretonne/meta/cretonne/test_xform.py similarity index 100% rename from meta/cretonne/test_xform.py rename to lib/cretonne/meta/cretonne/test_xform.py diff --git a/meta/cretonne/types.py b/lib/cretonne/meta/cretonne/types.py similarity index 100% rename from meta/cretonne/types.py rename to lib/cretonne/meta/cretonne/types.py diff --git a/meta/cretonne/typevar.py b/lib/cretonne/meta/cretonne/typevar.py similarity index 100% rename from meta/cretonne/typevar.py rename to lib/cretonne/meta/cretonne/typevar.py diff --git a/meta/cretonne/xform.py b/lib/cretonne/meta/cretonne/xform.py similarity index 100% rename from meta/cretonne/xform.py rename to lib/cretonne/meta/cretonne/xform.py diff --git a/meta/gen_build_deps.py b/lib/cretonne/meta/gen_build_deps.py similarity index 100% rename from meta/gen_build_deps.py rename to lib/cretonne/meta/gen_build_deps.py diff --git a/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py similarity index 100% rename from meta/gen_encoding.py rename to lib/cretonne/meta/gen_encoding.py diff --git a/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py similarity index 100% rename from meta/gen_instr.py rename to lib/cretonne/meta/gen_instr.py diff --git a/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py similarity index 100% rename from meta/gen_settings.py rename to lib/cretonne/meta/gen_settings.py diff --git a/meta/gen_types.py b/lib/cretonne/meta/gen_types.py similarity index 100% rename from meta/gen_types.py rename to lib/cretonne/meta/gen_types.py diff --git a/meta/isa/__init__.py b/lib/cretonne/meta/isa/__init__.py similarity index 100% rename from meta/isa/__init__.py rename to lib/cretonne/meta/isa/__init__.py diff --git a/meta/isa/riscv/__init__.py b/lib/cretonne/meta/isa/riscv/__init__.py similarity index 100% rename from meta/isa/riscv/__init__.py rename to lib/cretonne/meta/isa/riscv/__init__.py diff --git a/meta/isa/riscv/defs.py b/lib/cretonne/meta/isa/riscv/defs.py similarity index 100% rename from meta/isa/riscv/defs.py rename to lib/cretonne/meta/isa/riscv/defs.py diff --git a/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py similarity index 100% rename from meta/isa/riscv/encodings.py rename to lib/cretonne/meta/isa/riscv/encodings.py diff --git a/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py similarity index 100% rename from meta/isa/riscv/recipes.py rename to lib/cretonne/meta/isa/riscv/recipes.py diff --git a/meta/isa/riscv/settings.py b/lib/cretonne/meta/isa/riscv/settings.py similarity index 100% rename from meta/isa/riscv/settings.py rename to lib/cretonne/meta/isa/riscv/settings.py diff --git a/meta/srcgen.py b/lib/cretonne/meta/srcgen.py similarity index 100% rename from meta/srcgen.py rename to lib/cretonne/meta/srcgen.py diff --git a/meta/test_constant_hash.py b/lib/cretonne/meta/test_constant_hash.py similarity index 100% rename from meta/test_constant_hash.py rename to lib/cretonne/meta/test_constant_hash.py diff --git a/meta/test_srcgen.py b/lib/cretonne/meta/test_srcgen.py similarity index 100% rename from meta/test_srcgen.py rename to lib/cretonne/meta/test_srcgen.py diff --git a/meta/unique_table.py b/lib/cretonne/meta/unique_table.py similarity index 100% rename from meta/unique_table.py rename to lib/cretonne/meta/unique_table.py From 846db00a219eac703892b3caca30babaeb00de77 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 17 Oct 2016 14:44:43 -0700 Subject: [PATCH 0360/3084] Move library crates under 'lib/'. Give these crates each a more standard directory layout with sources in a 'src' sub-sirectory and Cargo.toml in the top lib/foo directory. Add license and description fields to each. The build script for the cretonne crate now lives in 'lib/cretonne/build.rs' separating it from the normal library sources under 'lib/cretonne/src'. --- cranelift/src/libcretonne/build.rs | 46 ------------------- cranelift/src/libreader/Cargo.toml | 12 ----- cranelift/src/tools/Cargo.toml | 6 +-- .../libcretonne => lib/cretonne}/Cargo.toml | 4 +- lib/cretonne/build.rs | 41 +++++++++++++++++ .../libcretonne => lib/cretonne/src}/cfg.rs | 0 .../cretonne/src}/constant_hash.rs | 0 .../cretonne/src}/dominator_tree.rs | 0 .../cretonne/src}/entity_map.rs | 0 .../cretonne/src}/ir/builder.rs | 0 .../cretonne/src}/ir/condcodes.rs | 0 .../cretonne/src}/ir/dfg.rs | 0 .../cretonne/src}/ir/entities.rs | 0 .../cretonne/src}/ir/extfunc.rs | 0 .../cretonne/src}/ir/funcname.rs | 0 .../cretonne/src}/ir/function.rs | 0 .../cretonne/src}/ir/immediates.rs | 0 .../cretonne/src}/ir/instructions.rs | 0 .../cretonne/src}/ir/jumptable.rs | 0 .../cretonne/src}/ir/layout.rs | 0 .../cretonne/src}/ir/mod.rs | 0 .../cretonne/src}/ir/stackslot.rs | 0 .../cretonne/src}/ir/types.rs | 0 .../cretonne/src}/isa/enc_tables.rs | 0 .../cretonne/src}/isa/encoding.rs | 0 .../cretonne/src}/isa/mod.rs | 0 .../cretonne/src}/isa/riscv/enc_tables.rs | 0 .../cretonne/src}/isa/riscv/mod.rs | 0 .../cretonne/src}/isa/riscv/settings.rs | 0 .../cretonne/src}/legalizer.rs | 0 .../libcretonne => lib/cretonne/src}/lib.rs | 0 .../cretonne/src}/predicates.rs | 0 .../cretonne/src}/settings.rs | 0 .../cretonne/src}/verifier.rs | 0 .../libcretonne => lib/cretonne/src}/write.rs | 0 .../libfilecheck => lib/filecheck}/Cargo.toml | 4 +- .../filecheck/src}/checker.rs | 0 .../filecheck/src}/error.rs | 0 .../filecheck/src}/explain.rs | 0 .../libfilecheck => lib/filecheck/src}/lib.rs | 0 .../filecheck/src}/pattern.rs | 0 .../filecheck/src}/tests/basic.rs | 0 .../filecheck/src}/variable.rs | 0 lib/reader/Cargo.toml | 15 ++++++ .../src/libreader => lib/reader/src}/error.rs | 0 .../libreader => lib/reader/src}/isaspec.rs | 0 .../src/libreader => lib/reader/src}/lexer.rs | 0 .../src/libreader => lib/reader/src}/lib.rs | 0 .../libreader => lib/reader/src}/parser.rs | 0 .../libreader => lib/reader/src}/sourcemap.rs | 0 .../reader/src}/testcommand.rs | 0 .../libreader => lib/reader/src}/testfile.rs | 0 52 files changed, 64 insertions(+), 64 deletions(-) delete mode 100644 cranelift/src/libcretonne/build.rs delete mode 100644 cranelift/src/libreader/Cargo.toml rename {cranelift/src/libcretonne => lib/cretonne}/Cargo.toml (91%) create mode 100644 lib/cretonne/build.rs rename {cranelift/src/libcretonne => lib/cretonne/src}/cfg.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/constant_hash.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/dominator_tree.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/entity_map.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/builder.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/condcodes.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/dfg.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/entities.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/extfunc.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/funcname.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/function.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/immediates.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/instructions.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/jumptable.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/layout.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/mod.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/stackslot.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/ir/types.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/isa/enc_tables.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/isa/encoding.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/isa/mod.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/isa/riscv/enc_tables.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/isa/riscv/mod.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/isa/riscv/settings.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/legalizer.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/lib.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/predicates.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/settings.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/verifier.rs (100%) rename {cranelift/src/libcretonne => lib/cretonne/src}/write.rs (100%) rename {cranelift/src/libfilecheck => lib/filecheck}/Cargo.toml (52%) rename {cranelift/src/libfilecheck => lib/filecheck/src}/checker.rs (100%) rename {cranelift/src/libfilecheck => lib/filecheck/src}/error.rs (100%) rename {cranelift/src/libfilecheck => lib/filecheck/src}/explain.rs (100%) rename {cranelift/src/libfilecheck => lib/filecheck/src}/lib.rs (100%) rename {cranelift/src/libfilecheck => lib/filecheck/src}/pattern.rs (100%) rename {cranelift/src/libfilecheck => lib/filecheck/src}/tests/basic.rs (100%) rename {cranelift/src/libfilecheck => lib/filecheck/src}/variable.rs (100%) create mode 100644 lib/reader/Cargo.toml rename {cranelift/src/libreader => lib/reader/src}/error.rs (100%) rename {cranelift/src/libreader => lib/reader/src}/isaspec.rs (100%) rename {cranelift/src/libreader => lib/reader/src}/lexer.rs (100%) rename {cranelift/src/libreader => lib/reader/src}/lib.rs (100%) rename {cranelift/src/libreader => lib/reader/src}/parser.rs (100%) rename {cranelift/src/libreader => lib/reader/src}/sourcemap.rs (100%) rename {cranelift/src/libreader => lib/reader/src}/testcommand.rs (100%) rename {cranelift/src/libreader => lib/reader/src}/testfile.rs (100%) diff --git a/cranelift/src/libcretonne/build.rs b/cranelift/src/libcretonne/build.rs deleted file mode 100644 index 2cb13ac959..0000000000 --- a/cranelift/src/libcretonne/build.rs +++ /dev/null @@ -1,46 +0,0 @@ - -// Build script. -// -// This program is run by Cargo when building libcretonne. It is used to generate Rust code from -// the language definitions in the meta directory. -// -// Environment: -// -// OUT_DIR -// Directory where generated files should be placed. -// -// The build script expects to be run from the directory where this build.rs file lives. The -// current directory is used to find the sources. - - -use std::env; -use std::process; - -fn main() { - let out_dir = env::var("OUT_DIR").expect("The OUT_DIR environment variable must be set"); - - println!("Build script generating files in {}", out_dir); - - let mut cur_dir = env::current_dir().expect("Can't access current working directory"); - - // We're in src/libcretonne. Find the top-level directory. - assert!(cur_dir.pop(), "No parent 'src' directory"); - assert!(cur_dir.pop(), "No top-level directory"); - let top_dir = cur_dir.as_path(); - - // Scripts are in $top_dir/meta. - let meta_dir = top_dir.join("lib/cretonne/meta"); - let build_script = meta_dir.join("build.py"); - - // Launch build script with Python. We'll just find python in the path. - let status = process::Command::new("python") - .current_dir(top_dir) - .arg(build_script) - .arg("--out-dir") - .arg(out_dir) - .status() - .expect("Failed to launch second-level build script"); - if !status.success() { - process::exit(status.code().unwrap()); - } -} diff --git a/cranelift/src/libreader/Cargo.toml b/cranelift/src/libreader/Cargo.toml deleted file mode 100644 index 478ee3b586..0000000000 --- a/cranelift/src/libreader/Cargo.toml +++ /dev/null @@ -1,12 +0,0 @@ -[package] -authors = ["The Cretonne Project Developers"] -name = "cretonne-reader" -version = "0.0.0" -publish = false - -[lib] -name = "cton_reader" -path = "lib.rs" - -[dependencies] -cretonne = { path = "../libcretonne" } diff --git a/cranelift/src/tools/Cargo.toml b/cranelift/src/tools/Cargo.toml index 3a1a8979a7..930256d8bb 100644 --- a/cranelift/src/tools/Cargo.toml +++ b/cranelift/src/tools/Cargo.toml @@ -10,9 +10,9 @@ name = "cton-util" path = "main.rs" [dependencies] -cretonne = { path = "../libcretonne" } -cretonne-reader = { path = "../libreader" } -filecheck = { path = "../libfilecheck" } +cretonne = { path = "../../lib/cretonne" } +cretonne-reader = { path = "../../lib/reader" } +filecheck = { path = "../../lib/filecheck" } docopt = "0.6.80" rustc-serialize = "0.3.19" num_cpus = "1.1.0" diff --git a/cranelift/src/libcretonne/Cargo.toml b/lib/cretonne/Cargo.toml similarity index 91% rename from cranelift/src/libcretonne/Cargo.toml rename to lib/cretonne/Cargo.toml index 174fb2e0bf..60e8eb2be9 100644 --- a/cranelift/src/libcretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -3,6 +3,7 @@ authors = ["The Cretonne Project Developers"] name = "cretonne" version = "0.0.0" description = "Low-level code generator library" +license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/stoklund/cretonne" publish = false @@ -10,10 +11,9 @@ build = "build.rs" [lib] name = "cretonne" -path = "lib.rs" [dependencies] # It is a goal of the cretonne crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be -# accomodated in src/tools/tests. +# accomodated in `tests`. diff --git a/lib/cretonne/build.rs b/lib/cretonne/build.rs new file mode 100644 index 0000000000..47e2890217 --- /dev/null +++ b/lib/cretonne/build.rs @@ -0,0 +1,41 @@ +// Build script. +// +// This program is run by Cargo when building lib/cretonne. It is used to generate Rust code from +// the language definitions in the lib/cretonne/meta directory. +// +// Environment: +// +// OUT_DIR +// Directory where generated files should be placed. +// +// The build script expects to be run from the directory where this build.rs file lives. The +// current directory is used to find the sources. + + +use std::env; +use std::process; + +fn main() { + let out_dir = env::var("OUT_DIR").expect("The OUT_DIR environment variable must be set"); + + println!("Build script generating files in {}", out_dir); + + let cur_dir = env::current_dir().expect("Can't access current working directory"); + let crate_dir = cur_dir.as_path(); + + // Scripts are in `$crate_dir/meta`. + let meta_dir = crate_dir.join("meta"); + let build_script = meta_dir.join("build.py"); + + // Launch build script with Python. We'll just find python in the path. + let status = process::Command::new("python") + .current_dir(crate_dir) + .arg(build_script) + .arg("--out-dir") + .arg(out_dir) + .status() + .expect("Failed to launch second-level build script"); + if !status.success() { + process::exit(status.code().unwrap()); + } +} diff --git a/cranelift/src/libcretonne/cfg.rs b/lib/cretonne/src/cfg.rs similarity index 100% rename from cranelift/src/libcretonne/cfg.rs rename to lib/cretonne/src/cfg.rs diff --git a/cranelift/src/libcretonne/constant_hash.rs b/lib/cretonne/src/constant_hash.rs similarity index 100% rename from cranelift/src/libcretonne/constant_hash.rs rename to lib/cretonne/src/constant_hash.rs diff --git a/cranelift/src/libcretonne/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs similarity index 100% rename from cranelift/src/libcretonne/dominator_tree.rs rename to lib/cretonne/src/dominator_tree.rs diff --git a/cranelift/src/libcretonne/entity_map.rs b/lib/cretonne/src/entity_map.rs similarity index 100% rename from cranelift/src/libcretonne/entity_map.rs rename to lib/cretonne/src/entity_map.rs diff --git a/cranelift/src/libcretonne/ir/builder.rs b/lib/cretonne/src/ir/builder.rs similarity index 100% rename from cranelift/src/libcretonne/ir/builder.rs rename to lib/cretonne/src/ir/builder.rs diff --git a/cranelift/src/libcretonne/ir/condcodes.rs b/lib/cretonne/src/ir/condcodes.rs similarity index 100% rename from cranelift/src/libcretonne/ir/condcodes.rs rename to lib/cretonne/src/ir/condcodes.rs diff --git a/cranelift/src/libcretonne/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs similarity index 100% rename from cranelift/src/libcretonne/ir/dfg.rs rename to lib/cretonne/src/ir/dfg.rs diff --git a/cranelift/src/libcretonne/ir/entities.rs b/lib/cretonne/src/ir/entities.rs similarity index 100% rename from cranelift/src/libcretonne/ir/entities.rs rename to lib/cretonne/src/ir/entities.rs diff --git a/cranelift/src/libcretonne/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs similarity index 100% rename from cranelift/src/libcretonne/ir/extfunc.rs rename to lib/cretonne/src/ir/extfunc.rs diff --git a/cranelift/src/libcretonne/ir/funcname.rs b/lib/cretonne/src/ir/funcname.rs similarity index 100% rename from cranelift/src/libcretonne/ir/funcname.rs rename to lib/cretonne/src/ir/funcname.rs diff --git a/cranelift/src/libcretonne/ir/function.rs b/lib/cretonne/src/ir/function.rs similarity index 100% rename from cranelift/src/libcretonne/ir/function.rs rename to lib/cretonne/src/ir/function.rs diff --git a/cranelift/src/libcretonne/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs similarity index 100% rename from cranelift/src/libcretonne/ir/immediates.rs rename to lib/cretonne/src/ir/immediates.rs diff --git a/cranelift/src/libcretonne/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs similarity index 100% rename from cranelift/src/libcretonne/ir/instructions.rs rename to lib/cretonne/src/ir/instructions.rs diff --git a/cranelift/src/libcretonne/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs similarity index 100% rename from cranelift/src/libcretonne/ir/jumptable.rs rename to lib/cretonne/src/ir/jumptable.rs diff --git a/cranelift/src/libcretonne/ir/layout.rs b/lib/cretonne/src/ir/layout.rs similarity index 100% rename from cranelift/src/libcretonne/ir/layout.rs rename to lib/cretonne/src/ir/layout.rs diff --git a/cranelift/src/libcretonne/ir/mod.rs b/lib/cretonne/src/ir/mod.rs similarity index 100% rename from cranelift/src/libcretonne/ir/mod.rs rename to lib/cretonne/src/ir/mod.rs diff --git a/cranelift/src/libcretonne/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs similarity index 100% rename from cranelift/src/libcretonne/ir/stackslot.rs rename to lib/cretonne/src/ir/stackslot.rs diff --git a/cranelift/src/libcretonne/ir/types.rs b/lib/cretonne/src/ir/types.rs similarity index 100% rename from cranelift/src/libcretonne/ir/types.rs rename to lib/cretonne/src/ir/types.rs diff --git a/cranelift/src/libcretonne/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs similarity index 100% rename from cranelift/src/libcretonne/isa/enc_tables.rs rename to lib/cretonne/src/isa/enc_tables.rs diff --git a/cranelift/src/libcretonne/isa/encoding.rs b/lib/cretonne/src/isa/encoding.rs similarity index 100% rename from cranelift/src/libcretonne/isa/encoding.rs rename to lib/cretonne/src/isa/encoding.rs diff --git a/cranelift/src/libcretonne/isa/mod.rs b/lib/cretonne/src/isa/mod.rs similarity index 100% rename from cranelift/src/libcretonne/isa/mod.rs rename to lib/cretonne/src/isa/mod.rs diff --git a/cranelift/src/libcretonne/isa/riscv/enc_tables.rs b/lib/cretonne/src/isa/riscv/enc_tables.rs similarity index 100% rename from cranelift/src/libcretonne/isa/riscv/enc_tables.rs rename to lib/cretonne/src/isa/riscv/enc_tables.rs diff --git a/cranelift/src/libcretonne/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs similarity index 100% rename from cranelift/src/libcretonne/isa/riscv/mod.rs rename to lib/cretonne/src/isa/riscv/mod.rs diff --git a/cranelift/src/libcretonne/isa/riscv/settings.rs b/lib/cretonne/src/isa/riscv/settings.rs similarity index 100% rename from cranelift/src/libcretonne/isa/riscv/settings.rs rename to lib/cretonne/src/isa/riscv/settings.rs diff --git a/cranelift/src/libcretonne/legalizer.rs b/lib/cretonne/src/legalizer.rs similarity index 100% rename from cranelift/src/libcretonne/legalizer.rs rename to lib/cretonne/src/legalizer.rs diff --git a/cranelift/src/libcretonne/lib.rs b/lib/cretonne/src/lib.rs similarity index 100% rename from cranelift/src/libcretonne/lib.rs rename to lib/cretonne/src/lib.rs diff --git a/cranelift/src/libcretonne/predicates.rs b/lib/cretonne/src/predicates.rs similarity index 100% rename from cranelift/src/libcretonne/predicates.rs rename to lib/cretonne/src/predicates.rs diff --git a/cranelift/src/libcretonne/settings.rs b/lib/cretonne/src/settings.rs similarity index 100% rename from cranelift/src/libcretonne/settings.rs rename to lib/cretonne/src/settings.rs diff --git a/cranelift/src/libcretonne/verifier.rs b/lib/cretonne/src/verifier.rs similarity index 100% rename from cranelift/src/libcretonne/verifier.rs rename to lib/cretonne/src/verifier.rs diff --git a/cranelift/src/libcretonne/write.rs b/lib/cretonne/src/write.rs similarity index 100% rename from cranelift/src/libcretonne/write.rs rename to lib/cretonne/src/write.rs diff --git a/cranelift/src/libfilecheck/Cargo.toml b/lib/filecheck/Cargo.toml similarity index 52% rename from cranelift/src/libfilecheck/Cargo.toml rename to lib/filecheck/Cargo.toml index 4f85778d92..f7cfa926e8 100644 --- a/cranelift/src/libfilecheck/Cargo.toml +++ b/lib/filecheck/Cargo.toml @@ -2,11 +2,13 @@ authors = ["The Cretonne Project Developers"] name = "filecheck" version = "0.0.0" +description = "Library for matching test outputs against filecheck directives" +license = "Apache-2.0" +repository = "https://github.com/stoklund/cretonne" publish = false [lib] name = "filecheck" -path = "lib.rs" [dependencies] regex = "0.1.71" diff --git a/cranelift/src/libfilecheck/checker.rs b/lib/filecheck/src/checker.rs similarity index 100% rename from cranelift/src/libfilecheck/checker.rs rename to lib/filecheck/src/checker.rs diff --git a/cranelift/src/libfilecheck/error.rs b/lib/filecheck/src/error.rs similarity index 100% rename from cranelift/src/libfilecheck/error.rs rename to lib/filecheck/src/error.rs diff --git a/cranelift/src/libfilecheck/explain.rs b/lib/filecheck/src/explain.rs similarity index 100% rename from cranelift/src/libfilecheck/explain.rs rename to lib/filecheck/src/explain.rs diff --git a/cranelift/src/libfilecheck/lib.rs b/lib/filecheck/src/lib.rs similarity index 100% rename from cranelift/src/libfilecheck/lib.rs rename to lib/filecheck/src/lib.rs diff --git a/cranelift/src/libfilecheck/pattern.rs b/lib/filecheck/src/pattern.rs similarity index 100% rename from cranelift/src/libfilecheck/pattern.rs rename to lib/filecheck/src/pattern.rs diff --git a/cranelift/src/libfilecheck/tests/basic.rs b/lib/filecheck/src/tests/basic.rs similarity index 100% rename from cranelift/src/libfilecheck/tests/basic.rs rename to lib/filecheck/src/tests/basic.rs diff --git a/cranelift/src/libfilecheck/variable.rs b/lib/filecheck/src/variable.rs similarity index 100% rename from cranelift/src/libfilecheck/variable.rs rename to lib/filecheck/src/variable.rs diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml new file mode 100644 index 0000000000..030d20ff9f --- /dev/null +++ b/lib/reader/Cargo.toml @@ -0,0 +1,15 @@ +[package] +authors = ["The Cretonne Project Developers"] +name = "cretonne-reader" +version = "0.0.0" +description = "Cretonne textual IL reader" +license = "Apache-2.0" +documentation = "https://cretonne.readthedocs.io/" +repository = "https://github.com/stoklund/cretonne" +publish = false + +[lib] +name = "cton_reader" + +[dependencies] +cretonne = { path = "../cretonne" } diff --git a/cranelift/src/libreader/error.rs b/lib/reader/src/error.rs similarity index 100% rename from cranelift/src/libreader/error.rs rename to lib/reader/src/error.rs diff --git a/cranelift/src/libreader/isaspec.rs b/lib/reader/src/isaspec.rs similarity index 100% rename from cranelift/src/libreader/isaspec.rs rename to lib/reader/src/isaspec.rs diff --git a/cranelift/src/libreader/lexer.rs b/lib/reader/src/lexer.rs similarity index 100% rename from cranelift/src/libreader/lexer.rs rename to lib/reader/src/lexer.rs diff --git a/cranelift/src/libreader/lib.rs b/lib/reader/src/lib.rs similarity index 100% rename from cranelift/src/libreader/lib.rs rename to lib/reader/src/lib.rs diff --git a/cranelift/src/libreader/parser.rs b/lib/reader/src/parser.rs similarity index 100% rename from cranelift/src/libreader/parser.rs rename to lib/reader/src/parser.rs diff --git a/cranelift/src/libreader/sourcemap.rs b/lib/reader/src/sourcemap.rs similarity index 100% rename from cranelift/src/libreader/sourcemap.rs rename to lib/reader/src/sourcemap.rs diff --git a/cranelift/src/libreader/testcommand.rs b/lib/reader/src/testcommand.rs similarity index 100% rename from cranelift/src/libreader/testcommand.rs rename to lib/reader/src/testcommand.rs diff --git a/cranelift/src/libreader/testfile.rs b/lib/reader/src/testfile.rs similarity index 100% rename from cranelift/src/libreader/testfile.rs rename to lib/reader/src/testfile.rs From f9734458f83de11d6e9a44bffa5d263b5a95d3f6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 17 Oct 2016 14:57:42 -0700 Subject: [PATCH 0361/3084] Promote the src/tools crate to the top-level workspace. The 'src' and 'tests' top-level directories now contain tools sources and integration tests for any of the library crates. --- cranelift/Cargo.toml | 24 +++++++++++++++++-- cranelift/{src => }/format-all.sh | 0 cranelift/src/{tools => }/cat.rs | 0 cranelift/src/{tools/main.rs => cton-util.rs} | 0 .../src/{tools => }/filetest/concurrent.rs | 0 cranelift/src/{tools => }/filetest/domtree.rs | 0 .../src/{tools => }/filetest/legalizer.rs | 0 cranelift/src/{tools => }/filetest/mod.rs | 0 cranelift/src/{tools => }/filetest/runner.rs | 0 cranelift/src/{tools => }/filetest/runone.rs | 0 cranelift/src/{tools => }/filetest/subtest.rs | 0 .../src/{tools => }/filetest/verifier.rs | 0 cranelift/src/{tools => }/print_cfg.rs | 0 cranelift/src/{tools => }/rsfilecheck.rs | 0 cranelift/src/tools/Cargo.toml | 18 -------------- cranelift/src/{tools => }/utils.rs | 0 cranelift/test-all.sh | 6 ++--- .../{src/tools => }/tests/cfg_traversal.rs | 0 18 files changed, 25 insertions(+), 23 deletions(-) rename cranelift/{src => }/format-all.sh (100%) rename cranelift/src/{tools => }/cat.rs (100%) rename cranelift/src/{tools/main.rs => cton-util.rs} (100%) rename cranelift/src/{tools => }/filetest/concurrent.rs (100%) rename cranelift/src/{tools => }/filetest/domtree.rs (100%) rename cranelift/src/{tools => }/filetest/legalizer.rs (100%) rename cranelift/src/{tools => }/filetest/mod.rs (100%) rename cranelift/src/{tools => }/filetest/runner.rs (100%) rename cranelift/src/{tools => }/filetest/runone.rs (100%) rename cranelift/src/{tools => }/filetest/subtest.rs (100%) rename cranelift/src/{tools => }/filetest/verifier.rs (100%) rename cranelift/src/{tools => }/print_cfg.rs (100%) rename cranelift/src/{tools => }/rsfilecheck.rs (100%) delete mode 100644 cranelift/src/tools/Cargo.toml rename cranelift/src/{tools => }/utils.rs (100%) rename cranelift/{src/tools => }/tests/cfg_traversal.rs (100%) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 63597ad428..d0da28f918 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,3 +1,23 @@ -# Phantom workspace manifest for all Cretonne crates. +[package] +name = "cretonne-tools" +authors = ["The Cretonne Project Developers"] +version = "0.0.0" +description = "Binaries for testing the Cretonne library" +license = "Apache-2.0" +documentation = "https://cretonne.readthedocs.io/" +repository = "https://github.com/stoklund/cretonne" +publish = false + +[[bin]] +name = "cton-util" +path = "src/cton-util.rs" + +[dependencies] +cretonne = { path = "lib/cretonne" } +cretonne-reader = { path = "lib/reader" } +filecheck = { path = "lib/filecheck" } +docopt = "0.6.86" +rustc-serialize = "0.3.19" +num_cpus = "1.1.0" + [workspace] -members = ["src/tools"] diff --git a/cranelift/src/format-all.sh b/cranelift/format-all.sh similarity index 100% rename from cranelift/src/format-all.sh rename to cranelift/format-all.sh diff --git a/cranelift/src/tools/cat.rs b/cranelift/src/cat.rs similarity index 100% rename from cranelift/src/tools/cat.rs rename to cranelift/src/cat.rs diff --git a/cranelift/src/tools/main.rs b/cranelift/src/cton-util.rs similarity index 100% rename from cranelift/src/tools/main.rs rename to cranelift/src/cton-util.rs diff --git a/cranelift/src/tools/filetest/concurrent.rs b/cranelift/src/filetest/concurrent.rs similarity index 100% rename from cranelift/src/tools/filetest/concurrent.rs rename to cranelift/src/filetest/concurrent.rs diff --git a/cranelift/src/tools/filetest/domtree.rs b/cranelift/src/filetest/domtree.rs similarity index 100% rename from cranelift/src/tools/filetest/domtree.rs rename to cranelift/src/filetest/domtree.rs diff --git a/cranelift/src/tools/filetest/legalizer.rs b/cranelift/src/filetest/legalizer.rs similarity index 100% rename from cranelift/src/tools/filetest/legalizer.rs rename to cranelift/src/filetest/legalizer.rs diff --git a/cranelift/src/tools/filetest/mod.rs b/cranelift/src/filetest/mod.rs similarity index 100% rename from cranelift/src/tools/filetest/mod.rs rename to cranelift/src/filetest/mod.rs diff --git a/cranelift/src/tools/filetest/runner.rs b/cranelift/src/filetest/runner.rs similarity index 100% rename from cranelift/src/tools/filetest/runner.rs rename to cranelift/src/filetest/runner.rs diff --git a/cranelift/src/tools/filetest/runone.rs b/cranelift/src/filetest/runone.rs similarity index 100% rename from cranelift/src/tools/filetest/runone.rs rename to cranelift/src/filetest/runone.rs diff --git a/cranelift/src/tools/filetest/subtest.rs b/cranelift/src/filetest/subtest.rs similarity index 100% rename from cranelift/src/tools/filetest/subtest.rs rename to cranelift/src/filetest/subtest.rs diff --git a/cranelift/src/tools/filetest/verifier.rs b/cranelift/src/filetest/verifier.rs similarity index 100% rename from cranelift/src/tools/filetest/verifier.rs rename to cranelift/src/filetest/verifier.rs diff --git a/cranelift/src/tools/print_cfg.rs b/cranelift/src/print_cfg.rs similarity index 100% rename from cranelift/src/tools/print_cfg.rs rename to cranelift/src/print_cfg.rs diff --git a/cranelift/src/tools/rsfilecheck.rs b/cranelift/src/rsfilecheck.rs similarity index 100% rename from cranelift/src/tools/rsfilecheck.rs rename to cranelift/src/rsfilecheck.rs diff --git a/cranelift/src/tools/Cargo.toml b/cranelift/src/tools/Cargo.toml deleted file mode 100644 index 930256d8bb..0000000000 --- a/cranelift/src/tools/Cargo.toml +++ /dev/null @@ -1,18 +0,0 @@ -[package] -authors = ["The Cretonne Project Developers"] -name = "cretonne-tools" -version = "0.0.0" -description = "Binaries for testing the Cretonne library" -publish = false - -[[bin]] -name = "cton-util" -path = "main.rs" - -[dependencies] -cretonne = { path = "../../lib/cretonne" } -cretonne-reader = { path = "../../lib/reader" } -filecheck = { path = "../../lib/filecheck" } -docopt = "0.6.80" -rustc-serialize = "0.3.19" -num_cpus = "1.1.0" diff --git a/cranelift/src/tools/utils.rs b/cranelift/src/utils.rs similarity index 100% rename from cranelift/src/tools/utils.rs rename to cranelift/src/utils.rs diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 843b1e82be..0ad58aa114 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -34,14 +34,14 @@ RUSTFMT_VERSION="0.6.2" if cargo install --list | grep -q "^rustfmt v$RUSTFMT_VERSION"; then banner "Rust formatting" - $topdir/src/format-all.sh --write-mode=diff + $topdir/format-all.sh --write-mode=diff else echo "Please install rustfmt v$RUSTFMT_VERSION to verify formatting." echo "If a newer version of rustfmt is available, update this script." fi PKGS="cretonne cretonne-reader cretonne-tools filecheck" -cd "$topdir/src/tools" +cd "$topdir" for PKG in $PKGS do banner "Rust $PKG unit tests" @@ -49,7 +49,7 @@ do done # Build cton-util for parser testing. -cd "$topdir/src/tools" +cd "$topdir" banner "Rust documentation" echo "open $topdir/target/doc/cretonne/index.html" cargo doc diff --git a/cranelift/src/tools/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs similarity index 100% rename from cranelift/src/tools/tests/cfg_traversal.rs rename to cranelift/tests/cfg_traversal.rs From 80a6ae203d603e4e68702e55e9d884a23bbf58cf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 18 Oct 2016 09:31:19 -0700 Subject: [PATCH 0362/3084] Generalize def_inst() to def_entity(). Use this source map method for assigning a location to any entity whose source number is not exposed. This could be - Instructions. - Signatures defined implicitly by function decls. These entities only appear in the location map, not the entity number maps. --- lib/reader/src/parser.rs | 2 +- lib/reader/src/sourcemap.rs | 28 ++++++++++------------------ 2 files changed, 11 insertions(+), 19 deletions(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 5c86e4e21a..89786ed2b3 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -837,7 +837,7 @@ impl<'a> Parser<'a> { let inst = ctx.function.dfg.make_inst(inst_data); let num_results = ctx.function.dfg.make_inst_results(inst, ctrl_typevar); ctx.function.layout.append_inst(inst, ebb); - ctx.map.def_inst(inst, &opcode_loc).expect("duplicate inst references created"); + ctx.map.def_entity(inst.into(), &opcode_loc).expect("duplicate inst references created"); if results.len() != num_results { return err!(self.loc, diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 2bff8e4b81..5808670fa4 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -8,7 +8,7 @@ //! clients. use std::collections::HashMap; -use cretonne::ir::{StackSlot, JumpTable, Ebb, Value, Inst}; +use cretonne::ir::{StackSlot, JumpTable, Ebb, Value}; use cretonne::ir::entities::AnyEntity; use error::{Result, Location}; use lexer::split_entity_name; @@ -128,9 +128,9 @@ pub trait MutableSourceMap { fn def_ss(&mut self, src_num: u32, entity: StackSlot, loc: &Location) -> Result<()>; fn def_jt(&mut self, src_num: u32, entity: JumpTable, loc: &Location) -> Result<()>; - /// Define an instruction. Since instruction numbers never appear in source, only the location - /// is recorded. - fn def_inst(&mut self, entity: Inst, loc: &Location) -> Result<()>; + /// Define an entity without an associated source number. This can be used for instructions + /// whose numbers never appear in source, or implicitly defined signatures. + fn def_entity(&mut self, entity: AnyEntity, loc: &Location) -> Result<()>; } impl MutableSourceMap for SourceMap { @@ -147,45 +147,37 @@ impl MutableSourceMap for SourceMap { fn def_value(&mut self, src: Value, entity: Value, loc: &Location) -> Result<()> { if self.values.insert(src, entity).is_some() { err!(loc, "duplicate value: {}", src) - } else if self.locations.insert(entity.into(), loc.clone()).is_some() { - err!(loc, "duplicate entity: {}", entity) } else { - Ok(()) + self.def_entity(entity.into(), loc) } } fn def_ebb(&mut self, src: Ebb, entity: Ebb, loc: &Location) -> Result<()> { if self.ebbs.insert(src, entity).is_some() { err!(loc, "duplicate EBB: {}", src) - } else if self.locations.insert(entity.into(), loc.clone()).is_some() { - err!(loc, "duplicate entity: {}", entity) } else { - Ok(()) + self.def_entity(entity.into(), loc) } } fn def_ss(&mut self, src_num: u32, entity: StackSlot, loc: &Location) -> Result<()> { if self.stack_slots.insert(src_num, entity).is_some() { err!(loc, "duplicate stack slot: ss{}", src_num) - } else if self.locations.insert(entity.into(), loc.clone()).is_some() { - err!(loc, "duplicate entity: {}", entity) } else { - Ok(()) + self.def_entity(entity.into(), loc) } } fn def_jt(&mut self, src_num: u32, entity: JumpTable, loc: &Location) -> Result<()> { if self.jump_tables.insert(src_num, entity).is_some() { err!(loc, "duplicate jump table: jt{}", src_num) - } else if self.locations.insert(entity.into(), loc.clone()).is_some() { - err!(loc, "duplicate entity: {}", entity) } else { - Ok(()) + self.def_entity(entity.into(), loc) } } - fn def_inst(&mut self, entity: Inst, loc: &Location) -> Result<()> { - if self.locations.insert(entity.into(), loc.clone()).is_some() { + fn def_entity(&mut self, entity: AnyEntity, loc: &Location) -> Result<()> { + if self.locations.insert(entity, loc.clone()).is_some() { err!(loc, "duplicate entity: {}", entity) } else { Ok(()) From b8a76822cfd0a41a25ac788a6626f8d15dde0f1f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 18 Oct 2016 09:43:20 -0700 Subject: [PATCH 0363/3084] Track signatures and function references in the source map. --- lib/reader/src/sourcemap.rs | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 5808670fa4..26d874817d 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -8,7 +8,7 @@ //! clients. use std::collections::HashMap; -use cretonne::ir::{StackSlot, JumpTable, Ebb, Value}; +use cretonne::ir::{StackSlot, JumpTable, Ebb, Value, SigRef, FuncRef}; use cretonne::ir::entities::AnyEntity; use error::{Result, Location}; use lexer::split_entity_name; @@ -19,6 +19,8 @@ pub struct SourceMap { values: HashMap, // vNN, vxNN ebbs: HashMap, // ebbNN stack_slots: HashMap, // ssNN + signatures: HashMap, // sigNN + functions: HashMap, // fnNN jump_tables: HashMap, // jtNN // Store locations for entities, including instructions. @@ -42,6 +44,16 @@ impl SourceMap { self.stack_slots.get(&src_num).cloned() } + /// Look up a signature entity by its source number. + pub fn get_sig(&self, src_num: u32) -> Option { + self.signatures.get(&src_num).cloned() + } + + /// Look up a function entity by its source number. + pub fn get_fn(&self, src_num: u32) -> Option { + self.functions.get(&src_num).cloned() + } + /// Look up a jump table entity by its source number. pub fn get_jt(&self, src_num: u32) -> Option { self.jump_tables.get(&src_num).cloned() @@ -64,6 +76,8 @@ impl SourceMap { } "ebb" => Ebb::with_number(num).and_then(|e| self.get_ebb(e)).map(AnyEntity::Ebb), "ss" => self.get_ss(num).map(AnyEntity::StackSlot), + "sig" => self.get_sig(num).map(AnyEntity::SigRef), + "fn" => self.get_fn(num).map(AnyEntity::FuncRef), "jt" => self.get_jt(num).map(AnyEntity::JumpTable), _ => None, } @@ -126,6 +140,8 @@ pub trait MutableSourceMap { fn def_value(&mut self, src: Value, entity: Value, loc: &Location) -> Result<()>; fn def_ebb(&mut self, src: Ebb, entity: Ebb, loc: &Location) -> Result<()>; fn def_ss(&mut self, src_num: u32, entity: StackSlot, loc: &Location) -> Result<()>; + fn def_sig(&mut self, src_num: u32, entity: SigRef, loc: &Location) -> Result<()>; + fn def_fn(&mut self, src_num: u32, entity: FuncRef, loc: &Location) -> Result<()>; fn def_jt(&mut self, src_num: u32, entity: JumpTable, loc: &Location) -> Result<()>; /// Define an entity without an associated source number. This can be used for instructions @@ -139,6 +155,8 @@ impl MutableSourceMap for SourceMap { values: HashMap::new(), ebbs: HashMap::new(), stack_slots: HashMap::new(), + signatures: HashMap::new(), + functions: HashMap::new(), jump_tables: HashMap::new(), locations: HashMap::new(), } @@ -168,6 +186,22 @@ impl MutableSourceMap for SourceMap { } } + fn def_sig(&mut self, src_num: u32, entity: SigRef, loc: &Location) -> Result<()> { + if self.signatures.insert(src_num, entity).is_some() { + err!(loc, "duplicate signature: sig{}", src_num) + } else { + self.def_entity(entity.into(), loc) + } + } + + fn def_fn(&mut self, src_num: u32, entity: FuncRef, loc: &Location) -> Result<()> { + if self.functions.insert(src_num, entity).is_some() { + err!(loc, "duplicate function: fn{}", src_num) + } else { + self.def_entity(entity.into(), loc) + } + } + fn def_jt(&mut self, src_num: u32, entity: JumpTable, loc: &Location) -> Result<()> { if self.jump_tables.insert(src_num, entity).is_some() { err!(loc, "duplicate jump table: jt{}", src_num) From bdc95990d44992aecbd9cb8b8a1ad38198464ad5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 18 Oct 2016 09:48:05 -0700 Subject: [PATCH 0364/3084] Add call and call_indirect instructions. Add a new IndirectCall instruction format which has a value callee as well as the call arguments. Define call and call_indirect instructions. --- cranelift/docs/langref.rst | 22 ++++--------------- lib/cretonne/meta/cretonne/__init__.py | 5 ++++- lib/cretonne/meta/cretonne/base.py | 30 ++++++++++++++++++++++++++ lib/cretonne/meta/cretonne/formats.py | 5 ++++- lib/cretonne/src/ir/builder.rs | 2 +- lib/cretonne/src/ir/instructions.rs | 21 +++++++++++++----- lib/cretonne/src/write.rs | 5 ++++- lib/reader/src/parser.rs | 14 +++++++++--- 8 files changed, 74 insertions(+), 30 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index b2ced79300..8acc2b4ef2 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -373,22 +373,15 @@ platform. When calling other Cretonne functions, the flags are not necessary. Functions that are called directly must be declared in the :term:`function preamble`: -.. inst:: F = function NAME signature +.. inst:: FN = function NAME signature Declare a function so it can be called directly. :arg NAME: Name of the function, passed to the linker for resolution. :arg signature: Function signature. See below. - :result F: A function identifier that can be used with :inst:`call`. - -.. inst:: a, b, ... = call F(args...) - - Direct function call. - - :arg F: Function identifier to call, declared by :inst:`function`. - :arg args...: Function arguments matching the signature of F. - :result a,b,...: Return values matching the signature of F. + :result FN: A function identifier that can be used with :inst:`call`. +.. autoinst:: call .. autoinst:: x_return This simple example illustrates direct function calls and signatures:: @@ -414,14 +407,7 @@ Indirect function calls use a signature declared in the preamble. :arg signature: Function signature. See :token:`signature`. :result SIG: A signature identifier. -.. inst:: a, b, ... = call_indirect SIG, x(args...) - - Indirect function call. - - :arg SIG: A function signature identifier declared with :inst:`signature`. - :arg iPtr x: The address of the function to call. - :arg args...: Function arguments matching SIG. - :result a,b,...: Return values matching SIG. +.. autoinst:: call_indirect .. todo:: Define safe indirect function calls. diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index a6cef5397b..a38126f98f 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -712,7 +712,10 @@ class InstructionFormat(object): :py:class:`Instruction` arguments of the same name, except they must be tuples of :py:`Operand` objects. """ - multiple_results = len(outs) > 1 + if len(outs) == 1: + multiple_results = outs[0].kind == variable_args + else: + multiple_results = len(outs) > 1 sig = (multiple_results,) + tuple(op.kind for op in ins) if sig not in InstructionFormat._registry: raise RuntimeError( diff --git a/lib/cretonne/meta/cretonne/base.py b/lib/cretonne/meta/cretonne/base.py index b665cf5d71..f507526626 100644 --- a/lib/cretonne/meta/cretonne/base.py +++ b/lib/cretonne/meta/cretonne/base.py @@ -15,6 +15,7 @@ instructions = InstructionGroup("base", "Shared base instruction set") Int = TypeVar('Int', 'A scalar or vector integer type', ints=True, simd=True) iB = TypeVar('iB', 'A scalar integer type', ints=True) +iPtr = TypeVar('iB', 'An integer address type', ints=(32, 64)) Testable = TypeVar( 'Testable', 'A scalar boolean or integer type', ints=True, bools=True) @@ -109,6 +110,35 @@ x_return = Instruction( """, ins=rvals) +FN = Operand( + 'FN', + entities.func_ref, + doc='function to call, declared by :inst:`function`') +args = Operand('args', variable_args, doc='call arguments') + +call = Instruction( + 'call', r""" + Direct function call. + + Call a function which has been declared in the preamble. The argument + types must match the function's signature. + """, + ins=(FN, args), + outs=rvals) + +SIG = Operand('SIG', entities.sig_ref, doc='function signature') +callee = Operand('callee', iPtr, doc='address of function to call') + +call_indirect = Instruction( + 'call_indirect', r""" + Indirect function call. + + Call the function pointed to by `callee` with the given arguments. The + called function must match the soecified signature. + """, + ins=(SIG, callee, args), + outs=rvals) + # # Materializing constants. # diff --git a/lib/cretonne/meta/cretonne/formats.py b/lib/cretonne/meta/cretonne/formats.py index 8e5d4e25cb..cb8aed3f98 100644 --- a/lib/cretonne/meta/cretonne/formats.py +++ b/lib/cretonne/meta/cretonne/formats.py @@ -8,7 +8,7 @@ in this module. from __future__ import absolute_import from . import InstructionFormat, value, variable_args from .immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc -from .entities import ebb, func_ref, jump_table +from .entities import ebb, sig_ref, func_ref, jump_table Nullary = InstructionFormat() @@ -47,6 +47,9 @@ BranchTable = InstructionFormat(value, jump_table) Call = InstructionFormat( func_ref, variable_args, multiple_results=True, boxed_storage=True) +IndirectCall = InstructionFormat( + sig_ref, value, variable_args, + multiple_results=True, boxed_storage=True) Return = InstructionFormat(variable_args, boxed_storage=True) diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index e31be0f2ca..f8510bb7f4 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -5,7 +5,7 @@ use ir::{types, instructions}; use ir::{InstructionData, DataFlowGraph, Cursor}; -use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, VariableArgs, FuncRef}; +use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, VariableArgs, SigRef, FuncRef}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, ImmVector}; use ir::condcodes::{IntCC, FloatCC}; diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 37b1106d36..f7fb6f3d94 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -10,7 +10,7 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::ops::{Deref, DerefMut}; -use ir::{Value, Type, Ebb, JumpTable, FuncRef}; +use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, ImmVector}; use ir::condcodes::*; use ir::types; @@ -207,6 +207,12 @@ pub enum InstructionData { second_result: Value, data: Box, }, + IndirectCall { + opcode: Opcode, + ty: Type, + second_result: Value, + data: Box, + }, Return { opcode: Opcode, ty: Type, @@ -342,10 +348,15 @@ pub struct CallData { pub varargs: VariableArgs, } -impl Display for CallData { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - write!(f, "TBD({})", self.varargs) - } +/// Payload of an indirect call instruction. +#[derive(Clone, Debug)] +pub struct IndirectCallData { + /// Callee function. + pub arg: Value, + pub sig_ref: SigRef, + + /// Dynamically sized array containing call argument values. + pub varargs: VariableArgs, } /// Payload of a return instruction. diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 3ef72d7178..dfae305933 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -199,7 +199,10 @@ fn write_instruction(w: &mut Write, Jump { ref data, .. } => writeln!(w, " {}", data), Branch { ref data, .. } => writeln!(w, " {}", data), BranchTable { arg, table, .. } => writeln!(w, " {}, {}", arg, table), - Call { ref data, .. } => writeln!(w, " {}", data), + Call { ref data, .. } => writeln!(w, " {}({})", data.func_ref, data.varargs), + IndirectCall { ref data, .. } => { + writeln!(w, " {}, {}({})", data.sig_ref, data.arg, data.varargs) + } Return { ref data, .. } => { if data.varargs.is_empty() { writeln!(w, "") diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 89786ed2b3..186fd6df1e 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -158,6 +158,11 @@ impl Context { try!(self.map.rewrite_values(&mut data.varargs, loc)); } + InstructionData::IndirectCall { ref mut data, .. } => { + try!(self.map.rewrite_value(&mut data.arg, loc)); + try!(self.map.rewrite_values(&mut data.varargs, loc)); + } + InstructionData::Return { ref mut data, .. } => { try!(self.map.rewrite_values(&mut data.varargs, loc)); } @@ -1171,6 +1176,12 @@ impl<'a> Parser<'a> { args: [lhs, rhs], } } + InstructionFormat::Call => { + unimplemented!(); + } + InstructionFormat::IndirectCall => { + unimplemented!(); + } InstructionFormat::Return => { let args = try!(self.parse_value_list()); InstructionData::Return { @@ -1190,9 +1201,6 @@ impl<'a> Parser<'a> { table: table, } } - InstructionFormat::Call => { - unimplemented!(); - } }) } } From 1bcb8e25a2b5456827028acad344e7463570980d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 18 Oct 2016 10:07:44 -0700 Subject: [PATCH 0365/3084] Add signatures and ext_funcs tables to DataFlowGraph. These two tables are used to keep track of type signatures of function calls as well as external function references used in direct function calls. Also add an ExtFuncData struct representing an external function that can be called directly. --- lib/cretonne/src/ir/dfg.rs | 14 +++++++++++++- lib/cretonne/src/ir/extfunc.rs | 17 ++++++++++++++++- lib/cretonne/src/ir/mod.rs | 2 +- lib/cretonne/src/write.rs | 11 +++++++++++ 4 files changed, 41 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 35bb9c8f7e..b913f505e6 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -1,8 +1,9 @@ //! Data flow graph tracking Instructions, Values, and EBBs. -use ir::{Ebb, Inst, Value, Type}; +use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef}; use ir::entities::{NO_VALUE, ExpandedValue}; use ir::instructions::InstructionData; +use ir::extfunc::ExtFuncData; use entity_map::{EntityMap, PrimaryEntityData}; use std::ops::{Index, IndexMut}; @@ -33,10 +34,19 @@ pub struct DataFlowGraph { /// This is implemented directly with a `Vec` rather than an `EntityMap` because /// the Value entity references can refer to two things -- an instruction or an extended value. extended_values: Vec, + + /// Function signature table. These signatures are referenced by indirect call instructions as + /// well as the external function references. + pub signatures: EntityMap, + + /// External function references. These are functions that can be called directly. + pub ext_funcs: EntityMap, } impl PrimaryEntityData for InstructionData {} impl PrimaryEntityData for EbbData {} +impl PrimaryEntityData for Signature {} +impl PrimaryEntityData for ExtFuncData {} impl DataFlowGraph { /// Create a new empty `DataFlowGraph`. @@ -45,6 +55,8 @@ impl DataFlowGraph { insts: EntityMap::new(), ebbs: EntityMap::new(), extended_values: Vec::new(), + signatures: EntityMap::new(), + ext_funcs: EntityMap::new(), } } diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 78b723398e..6f89d5d307 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -6,7 +6,7 @@ //! This module declares the data types used to represent external functions and call signatures. use std::fmt::{self, Display, Formatter}; -use ir::Type; +use ir::{Type, FunctionName, SigRef}; /// Function signature. /// @@ -104,6 +104,21 @@ pub enum ArgumentExtension { Sext, } +/// An external function. +/// +/// Information about a function that can be called directly with a direct `call` instruction. +#[derive(Clone, Debug)] +pub struct ExtFuncData { + pub name: FunctionName, + pub signature: SigRef, +} + +impl Display for ExtFuncData { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + write!(f, "{} {}", self.signature, self.name) + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index bbbc6d26ab..5056d9b29e 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -15,7 +15,7 @@ mod extfunc; mod builder; pub use ir::funcname::FunctionName; -pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension}; +pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension, ExtFuncData}; pub use ir::types::Type; pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable, FuncRef, SigRef}; pub use ir::instructions::{Opcode, InstructionData, VariableArgs}; diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index dfae305933..ac487c6f37 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -43,6 +43,17 @@ fn write_preamble(w: &mut Write, func: &Function) -> result::Result try!(writeln!(w, " {} = {}", ss, func.stack_slots[ss])); } + // Write out all signatures before functions since function decls can refer to signatures. + for sig in func.dfg.signatures.keys() { + any = true; + try!(writeln!(w, " {} = signature{}", sig, func.dfg.signatures[sig])); + } + + for fnref in func.dfg.ext_funcs.keys() { + any = true; + try!(writeln!(w, " {} = {}", fnref, func.dfg.ext_funcs[fnref])); + } + for jt in func.jump_tables.keys() { any = true; try!(writeln!(w, " {} = {}", jt, func.jump_tables[jt])); From cdb63a547e85e82b1625f0d40768e6343d81e991 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 18 Oct 2016 12:31:37 -0700 Subject: [PATCH 0366/3084] Add result values to call instructions too. The make_inst_results() method now understands direct and indirect calls, and can allocate result values matching the return types of the function call. --- lib/cretonne/src/ir/dfg.rs | 52 +++++++++++++++++++++++------ lib/cretonne/src/ir/instructions.rs | 28 ++++++++++++++++ 2 files changed, 70 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index b913f505e6..35b0af4dc6 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -2,7 +2,7 @@ use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef}; use ir::entities::{NO_VALUE, ExpandedValue}; -use ir::instructions::InstructionData; +use ir::instructions::{InstructionData, CallInfo}; use ir::extfunc::ExtFuncData; use entity_map::{EntityMap, PrimaryEntityData}; @@ -213,6 +213,7 @@ impl DataFlowGraph { pub fn make_inst_results(&mut self, inst: Inst, ctrl_typevar: Type) -> usize { let constraints = self.insts[inst].opcode().constraints(); let fixed_results = constraints.fixed_results(); + let mut total_results = fixed_results; // Additional values form a linked list starting from the second result value. Generate // the list backwards so we don't have to modify value table entries in place. (This @@ -220,20 +221,41 @@ impl DataFlowGraph { // choice, but since it is only visible in extremely rare instructions with 3+ results, // we don't care). let mut head = NO_VALUE; - let mut first_type = Type::default(); + let mut first_type = None; + let mut rev_num = 1; - // TBD: Function call return values for direct and indirect function calls. + // Get the call signature if this is a function call. + if let Some(sig) = self.call_signature(inst) { + // Create result values corresponding to the call return types. + let var_results = self.signatures[sig].return_types.len(); + total_results += var_results; - if fixed_results > 0 { - for res_idx in (1..fixed_results).rev() { + for res_idx in (0..var_results).rev() { + if let Some(ty) = first_type { + head = self.make_value(ValueData::Inst { + ty: ty, + num: (total_results - rev_num) as u16, + inst: inst, + next: head, + }); + rev_num += 1; + } + first_type = Some(self.signatures[sig].return_types[res_idx].value_type); + } + } + + // Then the fixed results whic will appear at the front of the list. + for res_idx in (0..fixed_results).rev() { + if let Some(ty) = first_type { head = self.make_value(ValueData::Inst { - ty: constraints.result_type(res_idx, ctrl_typevar), - num: res_idx as u16, + ty: ty, + num: (total_results - rev_num) as u16, inst: inst, next: head, }); + rev_num += 1; } - first_type = constraints.result_type(0, ctrl_typevar); + first_type = Some(constraints.result_type(res_idx, ctrl_typevar)); } // Update the second_result pointer in `inst`. @@ -242,9 +264,9 @@ impl DataFlowGraph { .second_result_mut() .expect("instruction format doesn't allow multiple results") = head; } - *self.insts[inst].first_type_mut() = first_type; + *self.insts[inst].first_type_mut() = first_type.unwrap_or_default(); - fixed_results + total_results } /// Get the first result of an instruction. @@ -265,6 +287,16 @@ impl DataFlowGraph { }, } } + + /// Get the call signature of a direct or indirect call instruction. + /// Returns `None` if `inst` is not a call instruction. + pub fn call_signature(&self, inst: Inst) -> Option { + match self.insts[inst].analyze_call() { + CallInfo::NotACall => None, + CallInfo::Direct(f, _) => Some(self.ext_funcs[f].signature), + CallInfo::Indirect(s, _) => Some(s), + } + } } /// Allow immutable access to instructions via indexing. diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index f7fb6f3d94..daab56d09c 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -388,6 +388,21 @@ impl InstructionData { } } + /// Return information about a call instruction. + /// + /// Any instruction that can call another function reveals its call signature here. + pub fn analyze_call<'a>(&'a self) -> CallInfo<'a> { + match self { + &InstructionData::Call { ref data, .. } => { + CallInfo::Direct(data.func_ref, &data.varargs) + } + &InstructionData::IndirectCall { ref data, .. } => { + CallInfo::Indirect(data.sig_ref, &data.varargs) + } + _ => CallInfo::NotACall, + } + } + /// Return true if an instruction is terminating, or false otherwise. pub fn is_terminating<'a>(&'a self) -> bool { match self { @@ -413,6 +428,19 @@ pub enum BranchInfo<'a> { Table(JumpTable), } +/// Information about call instructions. +pub enum CallInfo<'a> { + /// This is not a call instruction. + NotACall, + + /// This is a direct call to an external function declared in the preamble. See + /// `DataFlowGraph.ext_funcs`. + Direct(FuncRef, &'a [Value]), + + /// This is an indirect call with the specified signature. See `DataFlowGraph.signatures`. + Indirect(SigRef, &'a [Value]), +} + /// Value type constraints for a given opcode. /// /// The `InstructionFormat` determines the constraints on most operands, but `Value` operands and From df06f19979d751f287321f5f9a9030fd78da9283 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 18 Oct 2016 10:58:59 -0700 Subject: [PATCH 0367/3084] Parse signature and function declarations. Also add support for parsing call and call_indirect instructions. --- cranelift/filetests/parser/call.cton | 46 +++++++++ lib/reader/src/parser.rs | 145 ++++++++++++++++++++++++++- 2 files changed, 187 insertions(+), 4 deletions(-) diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton index 71c7c1efb0..72a61b6e21 100644 --- a/cranelift/filetests/parser/call.cton +++ b/cranelift/filetests/parser/call.cton @@ -22,3 +22,49 @@ ebb1: ; nextln: v1 = f32const 0.0 ; nextln: return v0, v1 ; nextln: } + +function signatures() { + sig10 = signature() + sig11 = signature(i32, f64) -> i32, b1 + fn5 = sig11 foo + fn8 = function bar(i32) -> b1 +} +; sameln: function signatures() { +; nextln: $sig10 = signature() +; nextln: $sig11 = signature(i32, f64) -> i32, b1 +; nextln: sig2 = signature(i32) -> b1 +; nextln: $fn5 = $sig11 foo +; nextln: $fn8 = sig2 bar +; nextln: } + +function direct() { + fn0 = function none() + fn1 = function one() -> i32 + fn2 = function two() -> i32, f32 + +ebb0: + call fn0() + v1 = call fn1() + v2, v3 = call fn2() + return +} +; check: call $fn0() +; check: $v1 = call $fn1() +; check: $v2, $v3 = call $fn2() +; check: return + +function indirect(i64) { + sig0 = signature(i64) + sig1 = signature() -> i32 + sig2 = signature() -> i32, f32 + +ebb0(v0: i64): + v1 = call_indirect sig1, v0() + call_indirect sig0, v1(v0) + v3, v4 = call_indirect sig2, v1() + return +} +; check: $v1 = call_indirect $sig1, $v0() +; check: call_indirect $sig0, $v1($v0) +; check: $v3, $v4 = call_indirect $sig2, $v1() +; check: return diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 186fd6df1e..7b16abe272 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -9,12 +9,14 @@ use std::str::FromStr; use std::u32; use std::mem; use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotData, JumpTable, - JumpTableData, Signature, ArgumentType, ArgumentExtension}; + JumpTableData, Signature, ArgumentType, ArgumentExtension, ExtFuncData, SigRef, + FuncRef}; use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::{AnyEntity, NO_EBB, NO_INST, NO_VALUE}; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, - TernaryOverflowData, JumpData, BranchData, ReturnData}; + TernaryOverflowData, JumpData, BranchData, CallData, + IndirectCallData, ReturnData}; use cretonne::isa; use cretonne::settings; use testfile::{TestFile, Details, Comment}; @@ -84,6 +86,32 @@ impl Context { self.map.def_ss(number, self.function.stack_slots.push(data), loc) } + // Allocate a new signature and add a mapping number -> SigRef. + fn add_sig(&mut self, number: u32, data: Signature, loc: &Location) -> Result<()> { + self.map.def_sig(number, self.function.dfg.signatures.push(data), loc) + } + + // Resolve a reference to a signature. + fn get_sig(&self, number: u32, loc: &Location) -> Result { + match self.map.get_sig(number) { + Some(sig) => Ok(sig), + None => err!(loc, "undefined signature sig{}", number), + } + } + + // Allocate a new external function and add a mapping number -> FuncRef. + fn add_fn(&mut self, number: u32, data: ExtFuncData, loc: &Location) -> Result<()> { + self.map.def_fn(number, self.function.dfg.ext_funcs.push(data), loc) + } + + // Resolve a reference to a function. + fn get_fn(&self, number: u32, loc: &Location) -> Result { + match self.map.get_fn(number) { + Some(fnref) => Ok(fnref), + None => err!(loc, "undefined function fn{}", number), + } + } + // Allocate a new jump table and add a mapping number -> JumpTable. fn add_jt(&mut self, number: u32, data: JumpTableData, loc: &Location) -> Result<()> { self.map.def_jt(number, self.function.jump_tables.push(data), loc) @@ -305,6 +333,26 @@ impl<'a> Parser<'a> { } } + // Match and consume a function reference. + fn match_fn(&mut self, err_msg: &str) -> Result { + if let Some(Token::FuncRef(fnref)) = self.token() { + self.consume(); + Ok(fnref) + } else { + err!(self.loc, err_msg) + } + } + + // Match and consume a signature reference. + fn match_sig(&mut self, err_msg: &str) -> Result { + if let Some(Token::SigRef(sigref)) = self.token() { + self.consume(); + Ok(sigref) + } else { + err!(self.loc, err_msg) + } + } + // Match and consume a jump table reference. fn match_jt(&mut self) -> Result { if let Some(Token::JumpTable(jt)) = self.token() { @@ -628,6 +676,16 @@ impl<'a> Parser<'a> { self.parse_stack_slot_decl() .and_then(|(num, dat)| ctx.add_ss(num, dat, &self.loc)) } + Some(Token::SigRef(..)) => { + self.gather_comments(ctx.function.dfg.signatures.next_key()); + self.parse_signature_decl() + .and_then(|(num, dat)| ctx.add_sig(num, dat, &self.loc)) + } + Some(Token::FuncRef(..)) => { + self.gather_comments(ctx.function.dfg.ext_funcs.next_key()); + self.parse_function_decl(ctx) + .and_then(|(num, dat)| ctx.add_fn(num, dat, &self.loc)) + } Some(Token::JumpTable(..)) => { self.gather_comments(ctx.function.jump_tables.next_key()); self.parse_jump_table_decl() @@ -661,6 +719,56 @@ impl<'a> Parser<'a> { Ok((number, data)) } + // Parse a signature decl. + // + // signature-decl ::= SigRef(sigref) "=" "signature" signature + // + fn parse_signature_decl(&mut self) -> Result<(u32, Signature)> { + let number = try!(self.match_sig("expected signature number: sig«n»")); + try!(self.match_token(Token::Equal, "expected '=' in signature decl")); + try!(self.match_identifier("signature", "expected 'signature'")); + let data = try!(self.parse_signature()); + Ok((number, data)) + } + + // Parse a function decl. + // + // Two variants: + // + // function-decl ::= FuncRef(fnref) "=" function-spec + // FuncRef(fnref) "=" SigRef(sig) name + // + // The first variant allocates a new signature reference. The second references an existing + // signature which must be declared first. + // + fn parse_function_decl(&mut self, ctx: &mut Context) -> Result<(u32, ExtFuncData)> { + let number = try!(self.match_fn("expected function number: fn«n»")); + try!(self.match_token(Token::Equal, "expected '=' in function decl")); + + let data = match self.token() { + Some(Token::Identifier("function")) => { + let (loc, name, sig) = try!(self.parse_function_spec()); + let sigref = ctx.function.dfg.signatures.push(sig); + ctx.map.def_entity(sigref.into(), &loc).expect("duplicate SigRef entities created"); + ExtFuncData { + name: name, + signature: sigref, + } + } + Some(Token::SigRef(sig_src)) => { + let sig = try!(ctx.get_sig(sig_src, &self.loc)); + self.consume(); + let name = try!(self.parse_function_name()); + ExtFuncData { + name: name, + signature: sig, + } + } + _ => return err!(self.loc, "expected 'function' or sig«n» in function decl"), + }; + Ok((number, data)) + } + // Parse a jump table decl. // // jump-table-decl ::= * JumpTable(jt) "=" "jump_table" jt-entry {"," jt-entry} @@ -1177,10 +1285,39 @@ impl<'a> Parser<'a> { } } InstructionFormat::Call => { - unimplemented!(); + let func_ref = try!(self.match_fn("expected function reference") + .and_then(|num| ctx.get_fn(num, &self.loc))); + try!(self.match_token(Token::LPar, "expected '(' before arguments")); + let args = try!(self.parse_value_list()); + try!(self.match_token(Token::RPar, "expected ')' after arguments")); + InstructionData::Call { + opcode: opcode, + ty: VOID, + second_result: NO_VALUE, + data: Box::new(CallData { + func_ref: func_ref, + varargs: args, + }), + } } InstructionFormat::IndirectCall => { - unimplemented!(); + let sig_ref = try!(self.match_sig("expected signature reference") + .and_then(|num| ctx.get_sig(num, &self.loc))); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let callee = try!(self.match_value("expected SSA value callee operand")); + try!(self.match_token(Token::LPar, "expected '(' before arguments")); + let args = try!(self.parse_value_list()); + try!(self.match_token(Token::RPar, "expected ')' after arguments")); + InstructionData::IndirectCall { + opcode: opcode, + ty: VOID, + second_result: NO_VALUE, + data: Box::new(IndirectCallData { + sig_ref: sig_ref, + arg: callee, + varargs: args, + }), + } } InstructionFormat::Return => { let args = try!(self.parse_value_list()); From 1e18e66ebe4f71a1c38f570013bd5ab52a91fc2d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 18 Oct 2016 13:28:38 -0700 Subject: [PATCH 0368/3084] Bump filecheck version to 0.0.1, allow publication. --- lib/filecheck/Cargo.toml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/filecheck/Cargo.toml b/lib/filecheck/Cargo.toml index f7cfa926e8..ee31a8fa42 100644 --- a/lib/filecheck/Cargo.toml +++ b/lib/filecheck/Cargo.toml @@ -1,11 +1,11 @@ [package] authors = ["The Cretonne Project Developers"] name = "filecheck" -version = "0.0.0" +version = "0.0.1" description = "Library for matching test outputs against filecheck directives" license = "Apache-2.0" repository = "https://github.com/stoklund/cretonne" -publish = false +documentation = "https://docs.rs/filecheck" [lib] name = "filecheck" From a5913e64897b09b89c067a1f255e7e347d883d8e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Oct 2016 11:12:09 -0700 Subject: [PATCH 0369/3084] Add more expansion patterns. RISC-V does not have a flags register, and thus no add-with-carry instructions. Neither does MIPS. Add expansions of these instructions in terms of iadd and icmp. --- lib/cretonne/meta/cretonne/legalize.py | 76 +++++++++++++++++++++++++- lib/cretonne/meta/cretonne/xform.py | 3 +- 2 files changed, 75 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/meta/cretonne/legalize.py b/lib/cretonne/meta/cretonne/legalize.py index 14eb879abd..9d7b218755 100644 --- a/lib/cretonne/meta/cretonne/legalize.py +++ b/lib/cretonne/meta/cretonne/legalize.py @@ -7,19 +7,43 @@ patterns that describe how base instructions can be transformed to other base instructions that are legal. """ from __future__ import absolute_import -from .base import iadd, iadd_cout, iadd_cin, isplit_lohi, iconcat_lohi -from .base import isub, isub_bin, isub_bout +from .base import iadd, iadd_cout, iadd_cin, iadd_carry +from .base import isub, isub_bin, isub_bout, isub_borrow +from .base import bor, isplit_lohi, iconcat_lohi +from .base import icmp from .ast import Var from .xform import Rtl, XFormGroup -narrow = XFormGroup() +narrow = XFormGroup(""" + Legalize instructions by narrowing. + + The transformations in the 'narrow' group work by expressing + instructions in terms of smaller types. Operations on vector types are + expressed in terms of vector types with fewer lanes, and integer + operations are expressed in terms of smaller integer types. + """) + +expand = XFormGroup(""" + Legalize instructions by expansion. + + Rewrite instructions in terms of other instructions, generally + operating on the same types as the original instructions. + """) x = Var('x') y = Var('y') a = Var('a') +a1 = Var('a1') +a2 = Var('a2') b = Var('b') +b1 = Var('b1') +b2 = Var('b2') +b_in = Var('b_in') c = Var('c') +c1 = Var('c1') +c2 = Var('c2') +c_in = Var('c_in') xl = Var('xl') xh = Var('xh') yl = Var('yl') @@ -46,3 +70,49 @@ narrow.legalize( ah << isub_bin(xh, yh, b), a << iconcat_lohi(al, ah) )) + +# Expand integer operations with carry for RISC architectures that don't have +# the flags. +expand.legalize( + (a, c) << iadd_cout(x, y), + Rtl( + a << iadd(x, y), + c << icmp('ult', a, x) + )) + +expand.legalize( + (a, b) << isub_bout(x, y), + Rtl( + a << isub(x, y), + b << icmp('ugt', a, x) + )) + +expand.legalize( + a << iadd_cin(x, y, c), + Rtl( + a1 << iadd(x, y), + a << iadd(a1, c) + )) + +expand.legalize( + a << isub_bin(x, y, b), + Rtl( + a1 << isub(x, y), + a << isub(a1, b) + )) + +expand.legalize( + (a, c) << iadd_carry(x, y, c_in), + Rtl( + (a1, c1) << iadd_cout(x, y), + (a, c2) << iadd_cout(a1, c_in), + c << bor(c1, c2) + )) + +expand.legalize( + (a, b) << isub_borrow(x, y, b_in), + Rtl( + (a1, b1) << isub_bout(x, y), + (a, b2) << isub_bout(a1, b_in), + c << bor(c1, c2) + )) diff --git a/lib/cretonne/meta/cretonne/xform.py b/lib/cretonne/meta/cretonne/xform.py index 16a9eae870..8015ee04b0 100644 --- a/lib/cretonne/meta/cretonne/xform.py +++ b/lib/cretonne/meta/cretonne/xform.py @@ -169,8 +169,9 @@ class XFormGroup(object): A group of related transformations. """ - def __init__(self): + def __init__(self, doc): self.xforms = list() + self.__doc__ = doc def legalize(self, src, dst): """ From cdb141d138d857bd82b8223eb6f6c3b98fb11703 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Oct 2016 12:03:50 -0700 Subject: [PATCH 0370/3084] Also rebuild if build.rs itself changes. We already have all the meta/**.py as dependencies. --- lib/cretonne/build.rs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/cretonne/build.rs b/lib/cretonne/build.rs index 47e2890217..2067a37afd 100644 --- a/lib/cretonne/build.rs +++ b/lib/cretonne/build.rs @@ -23,6 +23,11 @@ fn main() { let cur_dir = env::current_dir().expect("Can't access current working directory"); let crate_dir = cur_dir.as_path(); + // Make sure we rebuild is this build script changes. + // I guess that won't happen if you have non-UTF8 bytes in your path names. + // The `build.py` script prints out its own dependencies. + println!("cargo:rerun-if-changed={}", crate_dir.join("build.rs").to_string_lossy()); + // Scripts are in `$crate_dir/meta`. let meta_dir = crate_dir.join("meta"); let build_script = meta_dir.join("build.py"); From 499fefebd9b84ddd950cd1563d2e9214d3168072 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Oct 2016 16:23:08 -0700 Subject: [PATCH 0371/3084] Rename lifetimes in layout.rs to 'f These lifetimes all represent the lifetime of the Function. --- lib/cretonne/src/ir/layout.rs | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index c6543a88e0..864420c4f5 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -123,7 +123,7 @@ impl Layout { } /// Return an iterator over all EBBs in layout order. - pub fn ebbs<'a>(&'a self) -> Ebbs<'a> { + pub fn ebbs<'f>(&'f self) -> Ebbs<'f> { Ebbs { layout: self, next: self.first_ebb, @@ -146,12 +146,12 @@ struct EbbNode { } /// Iterate over EBBs in layout order. See `Layout::ebbs()`. -pub struct Ebbs<'a> { - layout: &'a Layout, +pub struct Ebbs<'f> { + layout: &'f Layout, next: Option, } -impl<'a> Iterator for Ebbs<'a> { +impl<'f> Iterator for Ebbs<'f> { type Item = Ebb; fn next(&mut self) -> Option { @@ -166,11 +166,11 @@ impl<'a> Iterator for Ebbs<'a> { } /// Use a layout reference in a for loop. -impl<'a> IntoIterator for &'a Layout { +impl<'f> IntoIterator for &'f Layout { type Item = Ebb; - type IntoIter = Ebbs<'a>; + type IntoIter = Ebbs<'f>; - fn into_iter(self) -> Ebbs<'a> { + fn into_iter(self) -> Ebbs<'f> { self.ebbs() } } @@ -235,7 +235,7 @@ impl Layout { } /// Iterate over the instructions in `ebb` in layout order. - pub fn ebb_insts<'a>(&'a self, ebb: Ebb) -> Insts<'a> { + pub fn ebb_insts<'f>(&'f self, ebb: Ebb) -> Insts<'f> { Insts { layout: self, next: self.ebbs[ebb].first_inst.wrap(), @@ -316,12 +316,12 @@ struct InstNode { } /// Iterate over instructions in an EBB in layout order. See `Layout::ebb_insts()`. -pub struct Insts<'a> { - layout: &'a Layout, +pub struct Insts<'f> { + layout: &'f Layout, next: Option, } -impl<'a> Iterator for Insts<'a> { +impl<'f> Iterator for Insts<'f> { type Item = Inst; fn next(&mut self) -> Option { @@ -345,8 +345,8 @@ impl<'a> Iterator for Insts<'a> { /// /// When new instructions are added, the cursor can either apend them to an EBB or insert them /// before the current instruction. -pub struct Cursor<'a> { - layout: &'a mut Layout, +pub struct Cursor<'f> { + layout: &'f mut Layout, pos: CursorPosition, } @@ -366,10 +366,10 @@ pub enum CursorPosition { After(Ebb), } -impl<'a> Cursor<'a> { +impl<'f> Cursor<'f> { /// Create a new `Cursor` for `layout`. /// The cursor holds a mutable reference to `layout` for its entire lifetime. - pub fn new(layout: &'a mut Layout) -> Cursor { + pub fn new(layout: &'f mut Layout) -> Cursor { Cursor { layout: layout, pos: CursorPosition::Nowhere, From eef5de1cf03a67e03ccbbe66d054bd90ad4aa7c5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Oct 2016 18:13:37 -0700 Subject: [PATCH 0372/3084] Generate an InstBuilder trait. All of the instruction format an opcode methods are emitted as an InstBuilder trait instead of adding them to the Bulder struct directly. The methods only make use of the InstBuilderBase methods to create new instructions. This makes it possible to reuse the InstBuilder trait for different ways of inserting instructions. --- lib/cretonne/meta/cretonne/__init__.py | 14 ++++++- lib/cretonne/meta/gen_instr.py | 56 +++++++++++++------------ lib/cretonne/src/cfg.rs | 2 +- lib/cretonne/src/dominator_tree.rs | 2 +- lib/cretonne/src/ir/builder.rs | 57 ++++++++++++++++++++++---- lib/cretonne/src/ir/mod.rs | 2 +- 6 files changed, 96 insertions(+), 37 deletions(-) diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index a38126f98f..d9005d6572 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -795,7 +795,19 @@ class Instruction(object): InstructionGroup.append(self) def __str__(self): - return self.name + prefix = ', '.join(o.name for o in self.outs) + if prefix: + prefix = prefix + ' = ' + suffix = ', '.join(o.name for o in self.ins) + return '{}{} {}'.format(prefix, self.name, suffix) + + def blurb(self): + """Get the first line of the doc comment""" + for line in self.__doc__.split('\n'): + line = line.strip() + if line: + return line + return "" def _verify_polymorphic(self): """ diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 8ae6d47ef7..4e854fe32c 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -177,14 +177,7 @@ def gen_opcodes(groups, fmt): for i in g.instructions: instrs.append(i) i.number = len(instrs) - # Build a doc comment. - prefix = ', '.join(o.name for o in i.outs) - if prefix: - prefix = prefix + ' = ' - suffix = ', '.join(o.name for o in i.ins) - fmt.doc_comment( - '`{}{} {}`. ({})' - .format(prefix, i.name, suffix, i.format.name)) + fmt.doc_comment('`{}`. ({})'.format(i, i.format.name)) # Document polymorphism. if i.is_polymorphic: if i.use_typevar_operand: @@ -370,7 +363,7 @@ def gen_format_constructor(iform, fmt): proto = '{}({}) -> Inst'.format(iform.name, ', '.join(args)) fmt.line('#[allow(non_snake_case)]') - with fmt.indented('pub fn {} {{'.format(proto), '}'): + with fmt.indented('fn {} {{'.format(proto), '}'): # Generate the instruction data. with fmt.indented( 'let data = InstructionData::{} {{'.format(iform.name), '};'): @@ -388,11 +381,9 @@ def gen_format_constructor(iform, fmt): # Create result values if necessary. if iform.multiple_results: - fmt.line('let inst = self.insert_inst(data);') - fmt.line('self.dfg.make_inst_results(inst, ctrl_typevar);') - fmt.line('inst') + fmt.line('self.complex_instruction(data, ctrl_typevar)') else: - fmt.line('self.insert_inst(data)') + fmt.line('self.simple_instruction(data)') def gen_member_inits(iform, fmt): @@ -452,8 +443,8 @@ def gen_inst_builder(inst, fmt): method = inst.name if method == 'return': - # Avoid Rust keywords - method = '_' + method + # Avoid Rust keywords by appending '_'. + method += '_' if len(tmpl_types) > 0: tmpl = '<{}>'.format(', '.join(tmpl_types)) @@ -461,8 +452,9 @@ def gen_inst_builder(inst, fmt): tmpl = '' proto = '{}{}({}) -> {}'.format(method, tmpl, ', '.join(args), rtype) + fmt.doc_comment('`{}`\n\n{}'.format(inst, inst.blurb())) fmt.line('#[allow(non_snake_case)]') - with fmt.indented('pub fn {} {{'.format(proto), '}'): + with fmt.indented('fn {} {{'.format(proto), '}'): # Convert all of the `Into<>` arguments. for arg in into_args: fmt.line('let {} = {}.into();'.format(arg, arg)) @@ -478,7 +470,7 @@ def gen_inst_builder(inst, fmt): elif inst.is_polymorphic: # Infer the controlling type variable from the input operands. fmt.line( - 'let ctrl_typevar = self.dfg.value_type({});' + 'let ctrl_typevar = self.data_flow_graph().value_type({});' .format(inst.ins[inst.format.typevar_operand].name)) args.append('ctrl_typevar') else: @@ -494,9 +486,11 @@ def gen_inst_builder(inst, fmt): if len(inst.value_results) == 0: fmt.line('inst') elif len(inst.value_results) == 1: - fmt.line('self.dfg.first_result(inst)') + fmt.line('self.data_flow_graph().first_result(inst)') else: - fmt.line('let mut results = self.dfg.inst_results(inst);') + fmt.line( + 'let mut results = ' + + 'self.data_flow_graph().inst_results(inst);') fmt.line('({})'.format(', '.join( len(inst.value_results) * ['results.next().unwrap()']))) @@ -505,16 +499,26 @@ def gen_builder(insts, fmt): """ Generate a Builder trait with methods for all instructions. """ - fmt.doc_comment( - 'Methods for inserting instructions by instruction format.') - with fmt.indented("impl<'a> Builder<'a> {", '}'): - for f in cretonne.InstructionFormat.all_formats: - gen_format_constructor(f, fmt) + fmt.doc_comment(""" + Convenience methods for building instructions. - fmt.doc_comment('Methods for inserting instructions by opcode.') - with fmt.indented("impl<'a> Builder<'a> {", '}'): + The `InstrBuilder` trait has one method per instruction opcode for + conveniently constructing the instruction with minimum arguments. + Polymorphic instructions infer their result types from the input + arguments when possible. In some cases, an explicit `result_type` + or `ctrl_typevar` argument is required. + + The opcode methods return the new instruction's result values, or + the `Inst` itself for instructions that don't have any results. + + There is also a method per instruction format. These methods all + return an `Inst`. + """) + with fmt.indented("pub trait InstBuilder: InstBuilderBase {", '}'): for inst in insts: gen_inst_builder(inst, fmt) + for f in cretonne.InstructionFormat.all_formats: + gen_format_constructor(f, fmt) def generate(isas, out_dir): diff --git a/lib/cretonne/src/cfg.rs b/lib/cretonne/src/cfg.rs index aab576ae82..59ced57d82 100644 --- a/lib/cretonne/src/cfg.rs +++ b/lib/cretonne/src/cfg.rs @@ -139,7 +139,7 @@ impl ControlFlowGraph { #[cfg(test)] mod tests { use super::*; - use ir::{Function, Builder, Cursor, VariableArgs, types}; + use ir::{Function, Builder, InstBuilder, Cursor, VariableArgs, types}; #[test] fn empty() { diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 0e34a2c021..85d224060d 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -116,7 +116,7 @@ impl DominatorTree { #[cfg(test)] mod test { use super::*; - use ir::{Function, Builder, Cursor, VariableArgs, types}; + use ir::{Function, Builder, InstBuilder, Cursor, VariableArgs, types}; use ir::entities::NO_INST; use cfg::ControlFlowGraph; diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index f8510bb7f4..e8646d0dd1 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -9,6 +9,42 @@ use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, VariableArgs, SigRef, FuncRe use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, ImmVector}; use ir::condcodes::{IntCC, FloatCC}; +/// Base trait for instruction builders. +/// +/// The `InstBuilderBase` trait provides the basic functionality required by the methods of the +/// generated `InstBuilder` trait. These methods should not normally be used directly. Use the +/// methods in the `InstBuilder trait instead. +/// +/// Any data type that implements `InstBuilderBase` also gets all the methods of the `InstBuilder` +/// trait. +pub trait InstBuilderBase { + /// Get an immutable reference to the data flow graph that will hold the constructed + /// instructions. + fn data_flow_graph(&self) -> &DataFlowGraph; + + /// Insert a simple instruction and return a reference to it. + /// + /// A 'simple' instruction has at most one result, and the `data.ty` field must contain the + /// result type or `VOID` for an instruction with no result values. + fn simple_instruction(&mut self, data: InstructionData) -> Inst; + + /// Insert a simple instruction and return a reference to it. + /// + /// A 'complex' instruction may produce multiple results, and the result types may depend on a + /// controlling type variable. For non-polymorphic instructions with multiple results, pass + /// `VOID` for the `ctrl_typevar` argument. + fn complex_instruction(&mut self, data: InstructionData, ctrl_typevar: Type) -> Inst; +} + +// Include trait code generated by `meta/gen_instr.py`. +// +// This file defines the `InstBuilder` trait as an extension of `InstBuilderBase` with methods per +// instruction format and per opcode. +include!(concat!(env!("OUT_DIR"), "/builder.rs")); + +/// Any type implementing `InstBuilderBase` gets all the `InstBuilder` methods for free. +impl InstBuilder for T {} + /// Instruction builder. /// /// A `Builder` holds mutable references to a data flow graph and a layout cursor. It provides @@ -40,16 +76,23 @@ impl<'a> Builder<'a> { pub fn insert_ebb(&mut self, ebb: Ebb) { self.pos.insert_ebb(ebb); } +} - // Create and insert an instruction. - // This method is used by the generated format-specific methods. - fn insert_inst(&mut self, data: InstructionData) -> Inst { +impl<'a> InstBuilderBase for Builder<'a> { + fn data_flow_graph(&self) -> &DataFlowGraph { + self.dfg + } + + fn simple_instruction(&mut self, data: InstructionData) -> Inst { let inst = self.dfg.make_inst(data); self.pos.insert_inst(inst); inst } -} -// Include code generated by `meta/gen_instr.py`. This file includes `Builder` methods per -// instruction format and per opcode for inserting instructions. -include!(concat!(env!("OUT_DIR"), "/builder.rs")); + fn complex_instruction(&mut self, data: InstructionData, ctrl_typevar: Type) -> Inst { + let inst = self.dfg.make_inst(data); + self.dfg.make_inst_results(inst, ctrl_typevar); + self.pos.insert_inst(inst); + inst + } +} diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 5056d9b29e..0812496032 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -24,4 +24,4 @@ pub use ir::jumptable::JumpTableData; pub use ir::dfg::{DataFlowGraph, ValueDef}; pub use ir::layout::{Layout, Cursor}; pub use ir::function::Function; -pub use ir::builder::Builder; +pub use ir::builder::{Builder, InstBuilder}; From 4cd33b210eaa3f0b5c58d144046ab450119a38dd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Oct 2016 18:27:22 -0700 Subject: [PATCH 0373/3084] Use more precise lifetimes for Builder. Distinguish the lifetime of the Cursor and its referenced function layout. Use two separate function lifetimes: 'fc and 'fd. The borrow checker seems to get confused if we don't. --- lib/cretonne/src/ir/builder.rs | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index e8646d0dd1..a360263138 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -49,15 +49,15 @@ impl InstBuilder for T {} /// /// A `Builder` holds mutable references to a data flow graph and a layout cursor. It provides /// convenience method for creating and inserting instructions at the current cursor position. -pub struct Builder<'a> { - pub dfg: &'a mut DataFlowGraph, - pub pos: &'a mut Cursor<'a>, +pub struct Builder<'c, 'fc: 'c, 'fd> { + pub pos: &'c mut Cursor<'fc>, + pub dfg: &'fd mut DataFlowGraph, } -impl<'a> Builder<'a> { +impl<'c, 'fc, 'fd> Builder<'c, 'fc, 'fd> { /// Create a new builder which inserts instructions at `pos`. /// The `dfg` and `pos.layout` references should be from the same `Function`. - pub fn new(dfg: &'a mut DataFlowGraph, pos: &'a mut Cursor<'a>) -> Builder<'a> { + pub fn new(dfg: &'fd mut DataFlowGraph, pos: &'c mut Cursor<'fc>) -> Builder<'c, 'fc, 'fd> { Builder { dfg: dfg, pos: pos, @@ -78,7 +78,7 @@ impl<'a> Builder<'a> { } } -impl<'a> InstBuilderBase for Builder<'a> { +impl<'c, 'fc, 'fd> InstBuilderBase for Builder<'c, 'fc, 'fd> { fn data_flow_graph(&self) -> &DataFlowGraph { self.dfg } From 78312c6c46756441ee276fc9d432706a6888e384 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Oct 2016 18:50:11 -0700 Subject: [PATCH 0374/3084] Add a Cursor::ins() method which constructs a Builder. Rewrite Builder uses in test cases to use this method and construct a new builder for each instruction. This pattern allows us to change the InstBuilder trait to a one-shot implementation that can only create a single instruction. Don't re-export the Builder struct, it is less important than the InstBuilder trait, and we may get more implementations. --- lib/cretonne/src/cfg.rs | 20 ++++++++++---------- lib/cretonne/src/dominator_tree.rs | 22 +++++++++++----------- lib/cretonne/src/ir/layout.rs | 7 +++++++ lib/cretonne/src/ir/mod.rs | 2 +- 4 files changed, 29 insertions(+), 22 deletions(-) diff --git a/lib/cretonne/src/cfg.rs b/lib/cretonne/src/cfg.rs index 59ced57d82..a50d4ff89f 100644 --- a/lib/cretonne/src/cfg.rs +++ b/lib/cretonne/src/cfg.rs @@ -139,7 +139,7 @@ impl ControlFlowGraph { #[cfg(test)] mod tests { use super::*; - use ir::{Function, Builder, InstBuilder, Cursor, VariableArgs, types}; + use ir::{Function, InstBuilder, Cursor, VariableArgs, types}; #[test] fn empty() { @@ -184,18 +184,18 @@ mod tests { let jmp_ebb1_ebb2; { - let mut cursor = Cursor::new(&mut func.layout); - let mut b = Builder::new(&mut func.dfg, &mut cursor); + let mut cur = Cursor::new(&mut func.layout); + let dfg = &mut func.dfg; - b.insert_ebb(ebb0); - br_ebb0_ebb2 = b.brnz(cond, ebb2, VariableArgs::new()); - jmp_ebb0_ebb1 = b.jump(ebb1, VariableArgs::new()); + cur.insert_ebb(ebb0); + br_ebb0_ebb2 = cur.ins(dfg).brnz(cond, ebb2, VariableArgs::new()); + jmp_ebb0_ebb1 = cur.ins(dfg).jump(ebb1, VariableArgs::new()); - b.insert_ebb(ebb1); - br_ebb1_ebb1 = b.brnz(cond, ebb1, VariableArgs::new()); - jmp_ebb1_ebb2 = b.jump(ebb2, VariableArgs::new()); + cur.insert_ebb(ebb1); + br_ebb1_ebb1 = cur.ins(dfg).brnz(cond, ebb1, VariableArgs::new()); + jmp_ebb1_ebb2 = cur.ins(dfg).jump(ebb2, VariableArgs::new()); - b.insert_ebb(ebb2); + cur.insert_ebb(ebb2); } let cfg = ControlFlowGraph::new(&func); diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 85d224060d..d109cc9f9f 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -116,7 +116,7 @@ impl DominatorTree { #[cfg(test)] mod test { use super::*; - use ir::{Function, Builder, InstBuilder, Cursor, VariableArgs, types}; + use ir::{Function, InstBuilder, Cursor, VariableArgs, types}; use ir::entities::NO_INST; use cfg::ControlFlowGraph; @@ -142,20 +142,20 @@ mod test { let jmp_ebb1_ebb2; { - let mut cursor = Cursor::new(&mut func.layout); - let mut b = Builder::new(&mut func.dfg, &mut cursor); + let mut cur = Cursor::new(&mut func.layout); + let dfg = &mut func.dfg; - b.insert_ebb(ebb3); - jmp_ebb3_ebb1 = b.jump(ebb1, VariableArgs::new()); + cur.insert_ebb(ebb3); + jmp_ebb3_ebb1 = cur.ins(dfg).jump(ebb1, VariableArgs::new()); - b.insert_ebb(ebb1); - br_ebb1_ebb0 = b.brnz(cond, ebb0, VariableArgs::new()); - jmp_ebb1_ebb2 = b.jump(ebb2, VariableArgs::new()); + cur.insert_ebb(ebb1); + br_ebb1_ebb0 = cur.ins(dfg).brnz(cond, ebb0, VariableArgs::new()); + jmp_ebb1_ebb2 = cur.ins(dfg).jump(ebb2, VariableArgs::new()); - b.insert_ebb(ebb2); - b.jump(ebb0, VariableArgs::new()); + cur.insert_ebb(ebb2); + cur.ins(dfg).jump(ebb0, VariableArgs::new()); - b.insert_ebb(ebb0); + cur.insert_ebb(ebb0); } let cfg = ControlFlowGraph::new(&func); diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 864420c4f5..3997b93349 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -6,6 +6,8 @@ use std::iter::{Iterator, IntoIterator}; use entity_map::{EntityMap, EntityRef}; use ir::entities::{Ebb, NO_EBB, Inst, NO_INST}; +use ir::dfg::DataFlowGraph; +use ir::builder::Builder; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not /// contain definitions of instructions or EBBs, but depends on `Inst` and `Ebb` entity references @@ -615,6 +617,11 @@ impl<'f> Cursor<'f> { } } + /// Create a builder for inserting an instruction at the current position. + pub fn ins<'c, 'fd>(&'c mut self, dfg: &'fd mut DataFlowGraph) -> Builder<'c, 'f, 'fd> { + Builder::new(dfg, self) + } + /// Insert an EBB at the current position and switch to it. /// /// As far as possible, this method behaves as if the EBB header were an instruction inserted diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 0812496032..b8cda7711c 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -24,4 +24,4 @@ pub use ir::jumptable::JumpTableData; pub use ir::dfg::{DataFlowGraph, ValueDef}; pub use ir::layout::{Layout, Cursor}; pub use ir::function::Function; -pub use ir::builder::{Builder, InstBuilder}; +pub use ir::builder::InstBuilder; From 5f140eddf718d525878f0b326845e4a5cf15be39 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Oct 2016 19:25:28 -0700 Subject: [PATCH 0375/3084] Switch InstrBuilder to the one-shot builder pattern. All the InstrBuilder methods now consume the builder, and the non-leaf methods return the dfg mutable reference they were holding. This makes it possible to construct instruction builders that are only safe to use once because they are doing more advanced value rewriting. --- lib/cretonne/meta/gen_instr.py | 34 ++++++++++++++++++++-------------- lib/cretonne/src/ir/builder.rs | 24 +++++++++++++++--------- 2 files changed, 35 insertions(+), 23 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 4e854fe32c..21bd8bcd85 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -347,7 +347,7 @@ def gen_format_constructor(iform, fmt): """ # Construct method arguments. - args = ['&mut self', 'opcode: Opcode'] + args = ['self', 'opcode: Opcode'] if iform.multiple_results: args.append('ctrl_typevar: Type') @@ -361,7 +361,9 @@ def gen_format_constructor(iform, fmt): for idx, kind in enumerate(iform.kinds): args.append('op{}: {}'.format(idx, kind.rust_type)) - proto = '{}({}) -> Inst'.format(iform.name, ', '.join(args)) + proto = '{}({})'.format(iform.name, ', '.join(args)) + proto += " -> (Inst, &'f mut DataFlowGraph)" + fmt.line('#[allow(non_snake_case)]') with fmt.indented('fn {} {{'.format(proto), '}'): # Generate the instruction data. @@ -414,7 +416,7 @@ def gen_inst_builder(inst, fmt): """ # Construct method arguments. - args = ['&mut self'] + args = ['self'] # The controlling type variable will be inferred from the input values if # possible. Otherwise, it is the first method argument. @@ -481,18 +483,21 @@ def gen_inst_builder(inst, fmt): args.extend(op.name for op in inst.ins) args = ', '.join(args) - fmt.line('let inst = self.{}({});'.format(inst.format.name, args)) + # Call to the format constructor, + fcall = 'self.{}({})'.format(inst.format.name, args) if len(inst.value_results) == 0: - fmt.line('inst') - elif len(inst.value_results) == 1: - fmt.line('self.data_flow_graph().first_result(inst)') - else: - fmt.line( - 'let mut results = ' + - 'self.data_flow_graph().inst_results(inst);') - fmt.line('({})'.format(', '.join( - len(inst.value_results) * ['results.next().unwrap()']))) + fmt.line(fcall + '.0') + return + + if len(inst.value_results) == 1: + fmt.line('Value::new_direct({}.0)'.format(fcall)) + return + + fmt.line('let (inst, dfg) = {};'.format(fcall)) + fmt.line('let mut results = dfg.inst_results(inst);') + fmt.line('({})'.format(', '.join( + len(inst.value_results) * ['results.next().unwrap()']))) def gen_builder(insts, fmt): @@ -514,7 +519,8 @@ def gen_builder(insts, fmt): There is also a method per instruction format. These methods all return an `Inst`. """) - with fmt.indented("pub trait InstBuilder: InstBuilderBase {", '}'): + with fmt.indented( + "pub trait InstBuilder<'f>: InstBuilderBase<'f> {", '}'): for inst in insts: gen_inst_builder(inst, fmt) for f in cretonne.InstructionFormat.all_formats: diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index a360263138..6ad50d9ab8 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -17,7 +17,7 @@ use ir::condcodes::{IntCC, FloatCC}; /// /// Any data type that implements `InstBuilderBase` also gets all the methods of the `InstBuilder` /// trait. -pub trait InstBuilderBase { +pub trait InstBuilderBase<'f>: Sized { /// Get an immutable reference to the data flow graph that will hold the constructed /// instructions. fn data_flow_graph(&self) -> &DataFlowGraph; @@ -26,14 +26,17 @@ pub trait InstBuilderBase { /// /// A 'simple' instruction has at most one result, and the `data.ty` field must contain the /// result type or `VOID` for an instruction with no result values. - fn simple_instruction(&mut self, data: InstructionData) -> Inst; + fn simple_instruction(self, data: InstructionData) -> (Inst, &'f mut DataFlowGraph); /// Insert a simple instruction and return a reference to it. /// /// A 'complex' instruction may produce multiple results, and the result types may depend on a /// controlling type variable. For non-polymorphic instructions with multiple results, pass /// `VOID` for the `ctrl_typevar` argument. - fn complex_instruction(&mut self, data: InstructionData, ctrl_typevar: Type) -> Inst; + fn complex_instruction(self, + data: InstructionData, + ctrl_typevar: Type) + -> (Inst, &'f mut DataFlowGraph); } // Include trait code generated by `meta/gen_instr.py`. @@ -43,7 +46,7 @@ pub trait InstBuilderBase { include!(concat!(env!("OUT_DIR"), "/builder.rs")); /// Any type implementing `InstBuilderBase` gets all the `InstBuilder` methods for free. -impl InstBuilder for T {} +impl<'f, T: InstBuilderBase<'f>> InstBuilder<'f> for T {} /// Instruction builder. /// @@ -78,21 +81,24 @@ impl<'c, 'fc, 'fd> Builder<'c, 'fc, 'fd> { } } -impl<'c, 'fc, 'fd> InstBuilderBase for Builder<'c, 'fc, 'fd> { +impl<'c, 'fc, 'fd> InstBuilderBase<'fd> for Builder<'c, 'fc, 'fd> { fn data_flow_graph(&self) -> &DataFlowGraph { self.dfg } - fn simple_instruction(&mut self, data: InstructionData) -> Inst { + fn simple_instruction(self, data: InstructionData) -> (Inst, &'fd mut DataFlowGraph) { let inst = self.dfg.make_inst(data); self.pos.insert_inst(inst); - inst + (inst, self.dfg) } - fn complex_instruction(&mut self, data: InstructionData, ctrl_typevar: Type) -> Inst { + fn complex_instruction(self, + data: InstructionData, + ctrl_typevar: Type) + -> (Inst, &'fd mut DataFlowGraph) { let inst = self.dfg.make_inst(data); self.dfg.make_inst_results(inst, ctrl_typevar); self.pos.insert_inst(inst); - inst + (inst, self.dfg) } } From d763bedeaa7e29a8562427f792b31400c16811b9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 20 Oct 2016 11:16:58 -0700 Subject: [PATCH 0376/3084] Rename Builder to InsertBuilder. This instruction builder inserts an instruction at the cursor position. We'll add other kinds of builders shortly. --- lib/cretonne/src/ir/builder.rs | 36 ++++++++++++---------------------- lib/cretonne/src/ir/layout.rs | 6 +++--- 2 files changed, 16 insertions(+), 26 deletions(-) diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 6ad50d9ab8..e9f859cd7b 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -48,40 +48,30 @@ include!(concat!(env!("OUT_DIR"), "/builder.rs")); /// Any type implementing `InstBuilderBase` gets all the `InstBuilder` methods for free. impl<'f, T: InstBuilderBase<'f>> InstBuilder<'f> for T {} -/// Instruction builder. +/// Builder that inserts an instruction at the current cursor position. /// -/// A `Builder` holds mutable references to a data flow graph and a layout cursor. It provides -/// convenience method for creating and inserting instructions at the current cursor position. -pub struct Builder<'c, 'fc: 'c, 'fd> { - pub pos: &'c mut Cursor<'fc>, - pub dfg: &'fd mut DataFlowGraph, +/// An `InsertBuilder` holds mutable references to a data flow graph and a layout cursor. It +/// provides convenience methods for creating and inserting instructions at the current cursor +/// position. +pub struct InsertBuilder<'c, 'fc: 'c, 'fd> { + pos: &'c mut Cursor<'fc>, + dfg: &'fd mut DataFlowGraph, } -impl<'c, 'fc, 'fd> Builder<'c, 'fc, 'fd> { +impl<'c, 'fc, 'fd> InsertBuilder<'c, 'fc, 'fd> { /// Create a new builder which inserts instructions at `pos`. /// The `dfg` and `pos.layout` references should be from the same `Function`. - pub fn new(dfg: &'fd mut DataFlowGraph, pos: &'c mut Cursor<'fc>) -> Builder<'c, 'fc, 'fd> { - Builder { + pub fn new(dfg: &'fd mut DataFlowGraph, + pos: &'c mut Cursor<'fc>) + -> InsertBuilder<'c, 'fc, 'fd> { + InsertBuilder { dfg: dfg, pos: pos, } } - - /// Create and insert an EBB. Further instructions will be inserted into the new EBB. - pub fn ebb(&mut self) -> Ebb { - let ebb = self.dfg.make_ebb(); - self.insert_ebb(ebb); - ebb - } - - /// Insert an existing EBB at the current position. Further instructions will be inserted into - /// the new EBB. - pub fn insert_ebb(&mut self, ebb: Ebb) { - self.pos.insert_ebb(ebb); - } } -impl<'c, 'fc, 'fd> InstBuilderBase<'fd> for Builder<'c, 'fc, 'fd> { +impl<'c, 'fc, 'fd> InstBuilderBase<'fd> for InsertBuilder<'c, 'fc, 'fd> { fn data_flow_graph(&self) -> &DataFlowGraph { self.dfg } diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 3997b93349..ae37899f5e 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -7,7 +7,7 @@ use std::iter::{Iterator, IntoIterator}; use entity_map::{EntityMap, EntityRef}; use ir::entities::{Ebb, NO_EBB, Inst, NO_INST}; use ir::dfg::DataFlowGraph; -use ir::builder::Builder; +use ir::builder::InsertBuilder; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not /// contain definitions of instructions or EBBs, but depends on `Inst` and `Ebb` entity references @@ -618,8 +618,8 @@ impl<'f> Cursor<'f> { } /// Create a builder for inserting an instruction at the current position. - pub fn ins<'c, 'fd>(&'c mut self, dfg: &'fd mut DataFlowGraph) -> Builder<'c, 'f, 'fd> { - Builder::new(dfg, self) + pub fn ins<'c, 'fd>(&'c mut self, dfg: &'fd mut DataFlowGraph) -> InsertBuilder<'c, 'f, 'fd> { + InsertBuilder::new(dfg, self) } /// Insert an EBB at the current position and switch to it. From 8422976d785d98796eb0f9b3c73a27a740fb17e6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 20 Oct 2016 13:09:37 -0700 Subject: [PATCH 0377/3084] Add a ReplaceBuilder instruction builder. The DataFlowGraph::replace(inst) method returns an instruction builder that will replace an instruction in-place. This will be used when transforming instructions, replacing an old instruction with a new (legal) way of computing its primary value. Since primary result values are essentially instruction pointers, this is the only way of replacing the definition of a value. If secondary result values match the old instruction in both number and types, they can be reused. If not, added a detach_secondary_results() method for detaching old secondary values. --- lib/cretonne/src/ir/builder.rs | 88 ++++++++++++++++++++++++++++++++++ lib/cretonne/src/ir/dfg.rs | 55 ++++++++++++++++++++- 2 files changed, 141 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index e9f859cd7b..7df5a6aa3c 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -92,3 +92,91 @@ impl<'c, 'fc, 'fd> InstBuilderBase<'fd> for InsertBuilder<'c, 'fc, 'fd> { (inst, self.dfg) } } + +/// Instruction builder that replaces an existing instruction. +/// +/// The inserted instruction will have the same `Inst` number as the old one. This is the only way +/// of rewriting the first result value of an instruction since this is a `ExpandedValue::Direct` +/// variant which encodes the instruction number directly. +/// +/// If the old instruction produced a value, the same value number will refer to the new +/// instruction's first result, so if that value has any uses the type should stay the same. +/// +/// If the old instruction still has secondary result values attached, it is assumed that the new +/// instruction produces the same number and types of results. The old secondary values are +/// preserved. If the replacemant instruction format does not support multiple results, the builder +/// panics. It is a bug to leave result values dangling. +/// +/// If the old instruction was capable of producing secondary results, but the values have been +/// detached, new result values are generated by calling `DataFlowGraph::make_inst_results()`. +pub struct ReplaceBuilder<'f> { + dfg: &'f mut DataFlowGraph, + inst: Inst, +} + +impl<'f> ReplaceBuilder<'f> { + /// Create a `ReplaceBuilder` that will overwrite `inst`. + pub fn new(dfg: &'f mut DataFlowGraph, inst: Inst) -> ReplaceBuilder { + ReplaceBuilder { + dfg: dfg, + inst: inst, + } + } +} + +impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { + fn data_flow_graph(&self) -> &DataFlowGraph { + self.dfg + } + + fn simple_instruction(self, data: InstructionData) -> (Inst, &'f mut DataFlowGraph) { + // The replacement instruction cannot generate multiple results, so verify that the old + // instruction's secondary results have been detached. + let old_second_value = self.dfg[self.inst].second_result().unwrap_or_default(); + assert_eq!(old_second_value, + Value::default(), + "Secondary result values {:?} would be left dangling by replacing {} with {}", + self.dfg.inst_results(self.inst).collect::>(), + self.dfg[self.inst].opcode(), + data.opcode()); + + // Splat the new instruction on top of the old one. + self.dfg[self.inst] = data; + (self.inst, self.dfg) + } + + fn complex_instruction(self, + data: InstructionData, + ctrl_typevar: Type) + -> (Inst, &'f mut DataFlowGraph) { + // If the old instruction still has secondary results attached, we'll keep them. + let old_second_value = self.dfg[self.inst].second_result().unwrap_or_default(); + + // Splat the new instruction on top of the old one. + self.dfg[self.inst] = data; + + if old_second_value == Value::default() { + // The old secondary values were either detached or non-existent. + // Construct new ones and set the first result type too. + self.dfg.make_inst_results(self.inst, ctrl_typevar); + } else { + // Reattach the old secondary values. + if let Some(val_ref) = self.dfg[self.inst].second_result_mut() { + // Don't check types here. Leave that to the verifier. + *val_ref = old_second_value; + } else { + // Actually, this instruction format should have called `simple_instruction()`, but + // we don't have a rule against calling `complex_instruction()` even when it is + // overkill. + panic!("Secondary result values left dangling"); + } + + // Normally, make_inst_results() would also set the first result type, but we're not + // going to call that, so set it manually. + *self.dfg[self.inst].first_type_mut() = + self.dfg.compute_result_type(self.inst, 0, ctrl_typevar).unwrap_or_default(); + } + + (self.inst, self.dfg) + } +} diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 35b0af4dc6..7b81c8deed 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -5,7 +5,9 @@ use ir::entities::{NO_VALUE, ExpandedValue}; use ir::instructions::{InstructionData, CallInfo}; use ir::extfunc::ExtFuncData; use entity_map::{EntityMap, PrimaryEntityData}; +use ir::builder::ReplaceBuilder; +use std::mem; use std::ops::{Index, IndexMut}; use std::u16; @@ -269,6 +271,27 @@ impl DataFlowGraph { total_results } + /// Create a `ReplaceBuilder` that will replace `inst` with a new instruction in place. + pub fn replace(&mut self, inst: Inst) -> ReplaceBuilder { + ReplaceBuilder::new(self, inst) + } + + /// Detach secondary instruction results, and return them as an iterator. + /// + /// If `inst` produces two or more results, detach these secondary result values from `inst`, + /// and return an iterator that will enumerate them. The first result value cannot be detached. + /// + /// Use this method to detach secondary values before using `replace(inst)` to provide an + /// alternate instruction for computing the primary result value. + pub fn detach_secondary_results(&mut self, inst: Inst) -> Values { + let second_result = + self[inst].second_result_mut().map(|r| mem::replace(r, NO_VALUE)).unwrap_or_default(); + Values { + dfg: self, + cur: second_result, + } + } + /// Get the first result of an instruction. /// /// If `Inst` doesn't produce any results, this returns a `Value` with a `VOID` type. @@ -277,7 +300,7 @@ impl DataFlowGraph { } /// Iterate through all the results of an instruction. - pub fn inst_results<'a>(&'a self, inst: Inst) -> Values<'a> { + pub fn inst_results(&self, inst: Inst) -> Values { Values { dfg: self, cur: if self.insts[inst].first_type().is_void() { @@ -297,6 +320,34 @@ impl DataFlowGraph { CallInfo::Indirect(s, _) => Some(s), } } + + /// Compute the type of an instruction result from opcode constraints and call signatures. + /// + /// This computes the same sequence of result types that `make_inst_results()` above would + /// assign to the created result values, but it does not depend on `make_inst_results()` being + /// called first. + /// + /// Returns `None` if asked about a result index that is too large. + pub fn compute_result_type(&self, + inst: Inst, + result_idx: usize, + ctrl_typevar: Type) + -> Option { + let constraints = self.insts[inst].opcode().constraints(); + let fixed_results = constraints.fixed_results(); + + if result_idx < fixed_results { + return Some(constraints.result_type(result_idx, ctrl_typevar)); + } + + // Not a fixed result, try to extract a return type from the call signature. + self.call_signature(inst).and_then(|sigref| { + self.signatures[sigref] + .return_types + .get(result_idx - fixed_results) + .map(|&arg| arg.value_type) + }) + } } /// Allow immutable access to instructions via indexing. @@ -369,7 +420,7 @@ impl DataFlowGraph { } /// Iterate through the arguments to an EBB. - pub fn ebb_args<'a>(&'a self, ebb: Ebb) -> Values<'a> { + pub fn ebb_args(&self, ebb: Ebb) -> Values { Values { dfg: self, cur: self.ebbs[ebb].first_arg, From 0acabc80d09b6931bb840b1342db2c7a7bc6f6e4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 20 Oct 2016 15:34:16 -0700 Subject: [PATCH 0378/3084] Define live range splitting instructions. The copy/spill/fill instructions will be used by the register allocator for splitting live ranges. The copy instruction is also useful when rewriting values: If a primary value is rewritten as a secondary result, a copy instruction can be used instead: a = foo x => t, vx1 = call ... a = copy vx1 Since a primary value must be the first value of an instruction, this doesn't work: a = foo x => t, a = call ... --- cranelift/docs/langref.rst | 26 ++++++++++++++++++++++- lib/cretonne/meta/cretonne/base.py | 34 ++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+), 1 deletion(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 8acc2b4ef2..6fbe3610de 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -633,11 +633,35 @@ A few instructions have variants that take immediate operands (e.g., :inst:`band` / :inst:`band_imm`), but in general an instruction is required to load a constant into an SSA value. +.. autoinst:: select + +Constant materialization +------------------------ + .. autoinst:: iconst .. autoinst:: f32const .. autoinst:: f64const .. autoinst:: vconst -.. autoinst:: select + +Live range splitting +-------------------- + +Cretonne's register allocator assigns each SSA value to a register or a spill +slot on the stack for its entire live range. Since the live range of an SSA +value can be quite large, it is sometimes beneficial to split the live range +into smaller parts. + +A live range is split by creating new SSA values that are copies or the +original value or each other. The copies are created by inserting :inst:`copy`, +:inst:`spill`, or :inst:`fill` instructions, depending on whether the values +are assigned to registers or stack slots. + +This approach permits SSA form to be preserved throughout the register +allocation pass and beyond. + +.. autoinst:: copy +.. autoinst:: spill +.. autoinst:: fill Vector operations ----------------- diff --git a/lib/cretonne/meta/cretonne/base.py b/lib/cretonne/meta/cretonne/base.py index f507526626..1edbce9ecd 100644 --- a/lib/cretonne/meta/cretonne/base.py +++ b/lib/cretonne/meta/cretonne/base.py @@ -204,6 +204,40 @@ select = Instruction( """, ins=(c, x, y), outs=a) +x = Operand('x', Any) + +copy = Instruction( + 'copy', r""" + Register-register copy. + + This instruction copies its input, preserving the value type. + + A pure SSA-form program does not need to copy values, but this + instruction is useful for representing intermediate stages during + instruction transformations, and the register allocator needs a way of + representing register copies. + """, + ins=x, outs=a) + +spill = Instruction( + 'spill', r""" + Spill a register value to a stack slot. + + This instruction behaves exactly like :inst:`copy`, but the result + value is assigned to a spill slot. + """, + ins=x, outs=a) + +fill = Instruction( + 'fill', r""" + Load a register value from a stack slot. + + This instruction behaves exactly like :inst:`copy`, but the input + value is assigned to a spill slot. + """, + ins=x, outs=a) + + # # Vector operations # From 1305283ed8b00fdad5fc34c7bcd89977e62414ce Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Oct 2016 09:44:24 -0700 Subject: [PATCH 0379/3084] Move the 'ins' method to DataFlowGraph. This given us better symmetry between the replace and insert builder operations: dfg.replace(inst).iadd(x, y) dfg.ins(cursor).imul(x, y) --- lib/cretonne/src/cfg.rs | 10 +++++----- lib/cretonne/src/dominator_tree.rs | 10 +++++----- lib/cretonne/src/ir/dfg.rs | 10 +++++++++- lib/cretonne/src/ir/layout.rs | 7 ------- 4 files changed, 19 insertions(+), 18 deletions(-) diff --git a/lib/cretonne/src/cfg.rs b/lib/cretonne/src/cfg.rs index a50d4ff89f..76ccc27965 100644 --- a/lib/cretonne/src/cfg.rs +++ b/lib/cretonne/src/cfg.rs @@ -184,16 +184,16 @@ mod tests { let jmp_ebb1_ebb2; { - let mut cur = Cursor::new(&mut func.layout); let dfg = &mut func.dfg; + let cur = &mut Cursor::new(&mut func.layout); cur.insert_ebb(ebb0); - br_ebb0_ebb2 = cur.ins(dfg).brnz(cond, ebb2, VariableArgs::new()); - jmp_ebb0_ebb1 = cur.ins(dfg).jump(ebb1, VariableArgs::new()); + br_ebb0_ebb2 = dfg.ins(cur).brnz(cond, ebb2, VariableArgs::new()); + jmp_ebb0_ebb1 = dfg.ins(cur).jump(ebb1, VariableArgs::new()); cur.insert_ebb(ebb1); - br_ebb1_ebb1 = cur.ins(dfg).brnz(cond, ebb1, VariableArgs::new()); - jmp_ebb1_ebb2 = cur.ins(dfg).jump(ebb2, VariableArgs::new()); + br_ebb1_ebb1 = dfg.ins(cur).brnz(cond, ebb1, VariableArgs::new()); + jmp_ebb1_ebb2 = dfg.ins(cur).jump(ebb2, VariableArgs::new()); cur.insert_ebb(ebb2); } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index d109cc9f9f..c81e579f6b 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -142,18 +142,18 @@ mod test { let jmp_ebb1_ebb2; { - let mut cur = Cursor::new(&mut func.layout); let dfg = &mut func.dfg; + let cur = &mut Cursor::new(&mut func.layout); cur.insert_ebb(ebb3); - jmp_ebb3_ebb1 = cur.ins(dfg).jump(ebb1, VariableArgs::new()); + jmp_ebb3_ebb1 = dfg.ins(cur).jump(ebb1, VariableArgs::new()); cur.insert_ebb(ebb1); - br_ebb1_ebb0 = cur.ins(dfg).brnz(cond, ebb0, VariableArgs::new()); - jmp_ebb1_ebb2 = cur.ins(dfg).jump(ebb2, VariableArgs::new()); + br_ebb1_ebb0 = dfg.ins(cur).brnz(cond, ebb0, VariableArgs::new()); + jmp_ebb1_ebb2 = dfg.ins(cur).jump(ebb2, VariableArgs::new()); cur.insert_ebb(ebb2); - cur.ins(dfg).jump(ebb0, VariableArgs::new()); + dfg.ins(cur).jump(ebb0, VariableArgs::new()); cur.insert_ebb(ebb0); } diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 7b81c8deed..903fd48ddc 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -5,7 +5,8 @@ use ir::entities::{NO_VALUE, ExpandedValue}; use ir::instructions::{InstructionData, CallInfo}; use ir::extfunc::ExtFuncData; use entity_map::{EntityMap, PrimaryEntityData}; -use ir::builder::ReplaceBuilder; +use ir::builder::{InsertBuilder, ReplaceBuilder}; +use ir::layout::Cursor; use std::mem; use std::ops::{Index, IndexMut}; @@ -271,6 +272,13 @@ impl DataFlowGraph { total_results } + /// Create an `InsertBuilder` that will insert an instruction at the cursor's current position. + pub fn ins<'c, 'fc: 'c, 'fd>(&'fd mut self, + at: &'c mut Cursor<'fc>) + -> InsertBuilder<'c, 'fc, 'fd> { + InsertBuilder::new(self, at) + } + /// Create a `ReplaceBuilder` that will replace `inst` with a new instruction in place. pub fn replace(&mut self, inst: Inst) -> ReplaceBuilder { ReplaceBuilder::new(self, inst) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index ae37899f5e..864420c4f5 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -6,8 +6,6 @@ use std::iter::{Iterator, IntoIterator}; use entity_map::{EntityMap, EntityRef}; use ir::entities::{Ebb, NO_EBB, Inst, NO_INST}; -use ir::dfg::DataFlowGraph; -use ir::builder::InsertBuilder; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not /// contain definitions of instructions or EBBs, but depends on `Inst` and `Ebb` entity references @@ -617,11 +615,6 @@ impl<'f> Cursor<'f> { } } - /// Create a builder for inserting an instruction at the current position. - pub fn ins<'c, 'fd>(&'c mut self, dfg: &'fd mut DataFlowGraph) -> InsertBuilder<'c, 'f, 'fd> { - InsertBuilder::new(dfg, self) - } - /// Insert an EBB at the current position and switch to it. /// /// As far as possible, this method behaves as if the EBB header were an instruction inserted From ce9049af90a6ddf13220b3be28036406770fe3f1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Oct 2016 09:56:55 -0700 Subject: [PATCH 0380/3084] Implement From for Imm64. This makes it possible to use literal integers as arguments to InstBuilder methods. --- lib/cretonne/src/ir/immediates.rs | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index e8d4a20773..0395d13f26 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -28,6 +28,12 @@ impl Into for Imm64 { } } +impl From for Imm64 { + fn from(x: i64) -> Self { + Imm64(x) + } +} + impl Display for Imm64 { fn fmt(&self, f: &mut Formatter) -> fmt::Result { let x = self.0; From 3791e8660d82c3592d41913cdf6977edea3f9d6e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Oct 2016 10:39:05 -0700 Subject: [PATCH 0381/3084] Make Type::as_bool() less pedantic for scalars. All scalar types are mapped to b1 which is usually what you want for a scalar. Vector types have their lanes mapped to the wider boolean types. Add an as_bool_pedantic() methos that produces the scalar sized boolean types as before. Also add a friendlier Debug implementation for Type. --- lib/cretonne/src/ir/types.rs | 49 +++++++++++++++++++++++++++++++++--- 1 file changed, 45 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 99390ee22a..f1e0998403 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -1,7 +1,7 @@ //! Common types for the Cretonne code generator. use std::default::Default; -use std::fmt::{self, Display, Formatter}; +use std::fmt::{self, Display, Debug, Formatter}; // ====--------------------------------------------------------------------------------------====// // @@ -23,7 +23,7 @@ use std::fmt::{self, Display, Formatter}; /// /// SIMD vector types have power-of-two lanes, up to 256. Lanes can be any int/float/bool type. /// -#[derive(Copy, Clone, PartialEq, Eq, Debug)] +#[derive(Copy, Clone, PartialEq, Eq)] pub struct Type(u8); /// No type. Used for functions without a return value. Can't be loaded or stored. Can't be part of @@ -31,7 +31,7 @@ pub struct Type(u8); pub const VOID: Type = Type(0); // Include code generated by `meta/gen_types.py`. This file contains constant definitions for all -// the scalar types as well as common vector types for 64, 128, 256, and 512-bit SID vectors. +// the scalar types as well as common vector types for 64, 128, 256, and 512-bit SIMD vectors. include!(concat!(env!("OUT_DIR"), "/types.rs")); impl Type { @@ -68,7 +68,10 @@ impl Type { /// Get a type with the same number of lanes as this type, but with the lanes replaced by /// booleans of the same size. - pub fn as_bool(self) -> Type { + /// + /// Scalar types are treated as vectors with one lane, so they are converted to the multi-bit + /// boolean types. + pub fn as_bool_pedantic(self) -> Type { // Replace the low 4 bits with the boolean version, preserve the high 4 bits. let lane = match self.lane_type() { B8 | I8 => B8, @@ -80,6 +83,18 @@ impl Type { Type(lane.0 | (self.0 & 0xf0)) } + /// Get a type with the same number of lanes as this type, but with the lanes replaced by + /// booleans of the same size. + /// + /// Scalar types are all converted to `b1` which is usually what you want. + pub fn as_bool(self) -> Type { + if self.is_scalar() { + B1 + } else { + self.as_bool_pedantic() + } + } + /// Get a type with the same number of lanes as this type, but with lanes that are half the /// number of bits. pub fn half_width(self) -> Option { @@ -222,6 +237,24 @@ impl Display for Type { } } +impl Debug for Type { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + if self.is_void() { + write!(f, "types::VOID") + } else if self.is_bool() { + write!(f, "types::B{}", self.lane_bits()) + } else if self.is_int() { + write!(f, "types::I{}", self.lane_bits()) + } else if self.is_float() { + write!(f, "types::F{}", self.lane_bits()) + } else if !self.is_scalar() { + write!(f, "{:?}X{}", self.lane_type(), self.lane_count()) + } else { + write!(f, "Type(0x{:x})", self.0) + } + } +} + impl Default for Type { fn default() -> Type { VOID @@ -339,4 +372,12 @@ mod tests { assert_eq!(I8.by(512), None); assert_eq!(VOID.by(4), None); } + + #[test] + fn as_bool() { + assert_eq!(I32X4.as_bool(), B32X4); + assert_eq!(I32.as_bool(), B1); + assert_eq!(I32X4.as_bool_pedantic(), B32X4); + assert_eq!(I32.as_bool_pedantic(), B32); + } } From e7a17a1b5929608c1c113f8624f6e18f7279a9c4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Oct 2016 11:02:53 -0700 Subject: [PATCH 0382/3084] Properly infer result type for single-result instructions. Polymorphic single-result instructions don't always return the controlling type variable as their first result. They may use a derived type variable, as for example icmp does. --- lib/cretonne/meta/gen_instr.py | 16 +++++++++++++++- lib/cretonne/src/ir/builder.rs | 29 +++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 21bd8bcd85..8a7e813223 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -474,7 +474,21 @@ def gen_inst_builder(inst, fmt): fmt.line( 'let ctrl_typevar = self.data_flow_graph().value_type({});' .format(inst.ins[inst.format.typevar_operand].name)) - args.append('ctrl_typevar') + if inst.format.multiple_results: + # The format constructor will resolve the result types from the + # type var. + args.append('ctrl_typevar') + elif inst.outs[inst.value_results[0]].typ == inst.ctrl_typevar: + # The format constructor expects a simple result type. + # No type transformation needed from the controlling type + # variable. + args.append('ctrl_typevar') + else: + # The format constructor expects a simple result type. + # TODO: This formula could be resolved ahead of time. + args.append( + 'Opcode::{}.constraints().result_type(0, ctrl_typevar)' + .format(inst.camel_name)) else: # This non-polymorphic instruction has a fixed result type. args.append( diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 7df5a6aa3c..575d36cd0b 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -180,3 +180,32 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { (self.inst, self.dfg) } } + +#[cfg(test)] +mod tests { + use ir::{Function, Cursor, InstBuilder}; + use ir::types::*; + use ir::condcodes::*; + + #[test] + fn types() { + let mut func = Function::new(); + let dfg = &mut func.dfg; + let ebb0 = dfg.make_ebb(); + let arg0 = dfg.append_ebb_arg(ebb0, I32); + let pos = &mut Cursor::new(&mut func.layout); + pos.insert_ebb(ebb0); + + // Explicit types. + let v0 = dfg.ins(pos).iconst(I32, 3); + assert_eq!(dfg.value_type(v0), I32); + + // Inferred from inputs. + let v1 = dfg.ins(pos).iadd(arg0, v0); + assert_eq!(dfg.value_type(v1), I32); + + // Formula. + let cmp = dfg.ins(pos).icmp(IntCC::Equal, arg0, v0); + assert_eq!(dfg.value_type(cmp), B1); + } +} From cf1996b036139c26824e409bd1230efc9bb430b0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 20 Oct 2016 17:24:14 -0700 Subject: [PATCH 0383/3084] Introduce value aliases. A extended value can now be changed to a third form: An alias of another value. This is like a copy instruction, but implicit in the value table. Value aliases are used in lieu of use-def chains which would be used to implement replace-all-uses-with. Added new DFG methods: - change_to_alias() changes an existing extended value into an alias. Primay values can't be changed, replace their definition with a copy instruction instead. - resolve_aliases() find the original non-alias value. - resolve_copies() like resolve_aliases(), but also sees through copy/spill/fill instructions. --- lib/cretonne/src/ir/dfg.rs | 150 ++++++++++++++++++++++++++++++++++++- 1 file changed, 148 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 903fd48ddc..0d24bc2d1a 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -2,7 +2,7 @@ use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef}; use ir::entities::{NO_VALUE, ExpandedValue}; -use ir::instructions::{InstructionData, CallInfo}; +use ir::instructions::{Opcode, InstructionData, CallInfo}; use ir::extfunc::ExtFuncData; use entity_map::{EntityMap, PrimaryEntityData}; use ir::builder::{InsertBuilder, ReplaceBuilder}; @@ -100,6 +100,7 @@ impl DataFlowGraph { match self.extended_values[i] { ValueData::Inst { ty, .. } => ty, ValueData::Arg { ty, .. } => ty, + ValueData::Alias { ty, .. } => ty, } } None => panic!("NO_VALUE has no type"), @@ -118,11 +119,104 @@ impl DataFlowGraph { match self.extended_values[idx] { ValueData::Inst { inst, num, .. } => ValueDef::Res(inst, num as usize), ValueData::Arg { ebb, num, .. } => ValueDef::Arg(ebb, num as usize), + ValueData::Alias { original, .. } => { + // Make sure we only recurse one level. `resolve_aliases` has safeguards to + // detect alias loops without overrunning the stack. + self.value_def(self.resolve_aliases(original)) + } } } None => panic!("NO_VALUE has no def"), } } + + /// Resolve value aliases. + /// + /// Find the original SSA value that `value` aliases. + pub fn resolve_aliases(&self, value: Value) -> Value { + use ir::entities::ExpandedValue::Table; + let mut v = value; + + for _ in 0..self.extended_values.len() { + v = match v.expand() { + Table(idx) => { + match self.extended_values[idx] { + ValueData::Alias { original, .. } => { + // Follow alias values. + original + } + _ => return v, + } + } + _ => return v, + }; + } + panic!("Value alias loop detected for {}", value); + } + + /// Resolve value copies. + /// + /// Find the original definition of a value, looking through value aliases as well as + /// copy/spill/fill instructions. + pub fn resolve_copies(&self, value: Value) -> Value { + use ir::entities::ExpandedValue::Direct; + let mut v = self.resolve_aliases(value); + + for _ in 0..self.insts.len() { + v = self.resolve_aliases(match v.expand() { + Direct(inst) => { + match self[inst] { + InstructionData::Unary { opcode, arg, .. } => { + match opcode { + Opcode::Copy | Opcode::Spill | Opcode::Fill => arg, + _ => return v, + } + } + _ => return v, + } + } + _ => return v, + }); + } + panic!("Copy loop detected for {}", value); + } + + /// Turn a value into an alias of another. + /// + /// Change the `dest` value to behave as an alias of `src`. This means that all uses of `dest` + /// will behave as if they used that value `src`. + /// + /// The `dest` value cannot be a direct value defined as the first result of an instruction. To + /// replace a direct value with `src`, its defining instruction should be replaced with a + /// `copy src` instruction. See `replace()`. + pub fn change_to_alias(&mut self, dest: Value, src: Value) { + use ir::entities::ExpandedValue::Table; + + // Try to create short alias chains by finding the original source value. + // This also avoids the creation of loops. + let original = self.resolve_aliases(src); + assert!(dest != original, + "Aliasing {} to {} would create a loop", + dest, + src); + let ty = self.value_type(original); + assert_eq!(self.value_type(dest), + ty, + "Aliasing {} to {} would change its type {} to {}", + dest, + src, + self.value_type(dest), + ty); + + if let Table(idx) = dest.expand() { + self.extended_values[idx] = ValueData::Alias { + ty: ty, + original: original, + }; + } else { + panic!("Cannot change dirrect value {} into an alias", dest); + } + } } /// Where did a value come from? @@ -152,6 +246,11 @@ enum ValueData { ebb: Ebb, next: Value, // Next argument to `ebb`. }, + + // Value is an alias of another value. + // An alias value can't be linked as an instruction result or EBB argument. It is used as a + // placeholder when the original instruction or EBB has been rewritten or modified. + Alias { ty: Type, original: Value }, } /// Iterate through a list of related value references, such as: @@ -178,6 +277,9 @@ impl<'a> Iterator for Values<'a> { match self.dfg.extended_values[index] { ValueData::Inst { next, .. } => next, ValueData::Arg { next, .. } => next, + ValueData::Alias { .. } => { + panic!("Alias value {} appeared in value list", prev) + } } } ExpandedValue::None => return None, @@ -466,7 +568,7 @@ impl EbbData { mod tests { use super::*; use ir::types; - use ir::{Opcode, InstructionData}; + use ir::{Function, Cursor, Opcode, InstructionData}; #[test] fn make_inst() { @@ -543,4 +645,48 @@ mod tests { assert_eq!(dfg.value_type(arg1), types::F32); assert_eq!(dfg.value_type(arg2), types::I16); } + + #[test] + fn aliases() { + use ir::InstBuilder; + use ir::entities::ExpandedValue::Direct; + use ir::condcodes::IntCC; + + let mut func = Function::new(); + let dfg = &mut func.dfg; + let ebb0 = dfg.make_ebb(); + let arg0 = dfg.append_ebb_arg(ebb0, types::I32); + let pos = &mut Cursor::new(&mut func.layout); + pos.insert_ebb(ebb0); + + // Build a little test program. + let v1 = dfg.ins(pos).iconst(types::I32, 42); + let (s, c) = dfg.ins(pos).iadd_cout(v1, arg0); + let iadd = match s.expand() { + Direct(i) => i, + _ => panic!(), + }; + + // Detach the 'c' value from iadd. + { + let mut vals = dfg.detach_secondary_results(iadd); + assert_eq!(vals.next(), Some(c)); + assert_eq!(vals.next(), None); + } + + // Replace iadd_cout with a normal iadd and an icmp. + dfg.replace(iadd).iadd(v1, arg0); + let c2 = dfg.ins(pos).icmp(IntCC::UnsignedLessThan, s, v1); + dfg.change_to_alias(c, c2); + + assert_eq!(dfg.resolve_aliases(c2), c2); + assert_eq!(dfg.resolve_aliases(c), c2); + + // Make a copy of the alias. + let c3 = dfg.ins(pos).copy(c); + // This does not see through copies. + assert_eq!(dfg.resolve_aliases(c3), c3); + // But this goes through both copies and aliases. + assert_eq!(dfg.resolve_copies(c3), c2); + } } From 31e033af49fa498dd0790024e7e0a5dd5aa471ee Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 24 Oct 2016 13:27:10 -0700 Subject: [PATCH 0384/3084] File stale path references. After rearranging the directory layout, some paths in documentation needed updating. Fix some typos too. --- cranelift/docs/testing.rst | 25 ++++++++++++++----------- lib/cretonne/meta/build.py | 2 +- lib/cretonne/meta/gen_build_deps.py | 2 +- lib/cretonne/meta/gen_types.py | 2 +- lib/cretonne/src/constant_hash.rs | 6 +++--- lib/cretonne/src/ir/builder.rs | 2 +- lib/cretonne/src/ir/instructions.rs | 6 +++--- lib/cretonne/src/ir/types.rs | 5 +++-- lib/cretonne/src/isa/enc_tables.rs | 2 +- lib/cretonne/src/isa/riscv/settings.rs | 5 +++-- lib/cretonne/src/predicates.rs | 2 +- 11 files changed, 32 insertions(+), 27 deletions(-) diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index c902dc70f6..0f0527bf06 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -23,7 +23,7 @@ Unit tests Unit test live in a ``tests`` sub-module of the code they are testing:: - pub fn add(x: u32, y: u32) { + pub fn add(x: u32, y: u32) -> u32 { x + y } @@ -62,19 +62,21 @@ tested:: These tests are useful for demonstrating how to use an API, and running them regularly makes sure that they stay up to date. Documentation tests are not -appropriate for lots of assertions, use unit tests for that. +appropriate for lots of assertions; use unit tests for that. Integration tests ----------------- -Integration tests are Rust source files thar are compiled and linked +Integration tests are Rust source files that are compiled and linked individually. They are used to exercise the external API of the crates under test. -These tests are usually found in the :file:`src/tools/tests` sub-directory where they -have access to all the crates in the Cretonne repository. The -:file:`libcretonne` and :file:`libreader` crates have no external dependencies -which can make testing tedious. +These tests are usually found in the :file:`tests` top-level directory where +they have access to all the crates in the Cretonne repository. The +:file:`lib/cretonne` and :file:`lib/reader` crates have no external +dependencies, which can make testing tedious. Integration tests that don't need +to depend on other crates can be placed in :file:`lib/cretonne/tests` and +:file:`lib/reader/tests`. File tests ========== @@ -107,7 +109,7 @@ header: isa_spec : "isa" isa_name { `option` } "\n" The options given on the ``isa`` line modify the ISA-specific settings defined in -:file:`meta/isa/*/setings.py`. +:file:`lib/cretonne/meta/isa/*/settings.py`. All types of tests allow shared Cretonne settings to be modified: @@ -117,7 +119,7 @@ All types of tests allow shared Cretonne settings to be modified: option : flag | setting "=" value The shared settings available for all target ISAs are defined in -:file:`meta/cretonne/settings.py`. +:file:`lib/cretonne/meta/cretonne/settings.py`. The ``set`` lines apply settings cumulatively:: @@ -139,7 +141,8 @@ Filecheck Many of the test commands bescribed below use *filecheck* to verify their output. Filecheck is a Rust implementation of the LLVM tool of the same name. -See the :file:`libfilecheck` documentation for details of its syntax. +See the :file:`lib/filecheck` `documentation `_ for +details of its syntax. Comments in :file:`.cton` files are associated with the entity they follow. This typically means and instruction or the whole function. Those tests that @@ -225,7 +228,7 @@ reported location of the error is verified:: return } -This example test passed if the verifier fails with an error message containing +This example test passes if the verifier fails with an error message containing the sub-string ``"terminator"`` *and* the error is reported for the ``jump`` instruction. diff --git a/lib/cretonne/meta/build.py b/lib/cretonne/meta/build.py index 730d1c9b3f..cfda7ae862 100644 --- a/lib/cretonne/meta/build.py +++ b/lib/cretonne/meta/build.py @@ -1,6 +1,6 @@ # Second-level build script. # -# This script is run from src/libcretonne/build.rs to generate Rust files. +# This script is run from lib/cretonne/build.rs to generate Rust files. from __future__ import absolute_import import argparse diff --git a/lib/cretonne/meta/gen_build_deps.py b/lib/cretonne/meta/gen_build_deps.py index 32ae2f7238..191113a759 100644 --- a/lib/cretonne/meta/gen_build_deps.py +++ b/lib/cretonne/meta/gen_build_deps.py @@ -1,7 +1,7 @@ """ Generate build dependencies for Cargo. -The `build.py` script is invoked by cargo when building libcretonne to +The `build.py` script is invoked by cargo when building lib/cretonne to generate Rust code from the instruction descriptions. Cargo needs to know when it is necessary to rerun the build script. diff --git a/lib/cretonne/meta/gen_types.py b/lib/cretonne/meta/gen_types.py index 51607a61f9..2b284d135b 100644 --- a/lib/cretonne/meta/gen_types.py +++ b/lib/cretonne/meta/gen_types.py @@ -2,7 +2,7 @@ Generate sources with type info. This generates a `types.rs` file which is included in -`libcretonne/ir/types/rs`. The file provides constant definitions for the most +`lib/cretonne/ir/types.rs`. The file provides constant definitions for the most commonly used types, including all of the scalar types. This ensures that Python and Rust use the same type numbering. diff --git a/lib/cretonne/src/constant_hash.rs b/lib/cretonne/src/constant_hash.rs index fd8115c7fa..f76fde7fb7 100644 --- a/lib/cretonne/src/constant_hash.rs +++ b/lib/cretonne/src/constant_hash.rs @@ -1,7 +1,7 @@ //! Runtime support for precomputed constant hash tables. //! -//! The `meta/constant_hash.py` Python module can generate constant hash tables using open -//! addressing and quadratic probing. The hash tables are arrays that are guaranteed to: +//! The `lib/cretonne/meta/constant_hash.py` Python module can generate constant hash tables using +//! open addressing and quadratic probing. The hash tables are arrays that are guaranteed to: //! //! - Have a power-of-two size. //! - Contain at least one empty slot. @@ -52,7 +52,7 @@ pub fn probe + ?Sized>(table: &T, key: K, hash: usize) } /// A primitive hash function for matching opcodes. -/// Must match `meta/constant_hash.py`. +/// Must match `lib/cretonne/meta/constant_hash.py`. pub fn simple_hash(s: &str) -> usize { let mut h: u32 = 5381; for c in s.chars() { diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 575d36cd0b..695bc8e5a3 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -39,7 +39,7 @@ pub trait InstBuilderBase<'f>: Sized { -> (Inst, &'f mut DataFlowGraph); } -// Include trait code generated by `meta/gen_instr.py`. +// Include trait code generated by `lib/cretonne/meta/gen_instr.py`. // // This file defines the `InstBuilder` trait as an extension of `InstBuilderBase` with methods per // instruction format and per opcode. diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index daab56d09c..103dcd4fe5 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -15,7 +15,7 @@ use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, ImmVector}; use ir::condcodes::*; use ir::types; -// Include code generated by `meta/gen_instr.py`. This file contains: +// Include code generated by `lib/cretonne/meta/gen_instr.py`. This file contains: // // - The `pub enum InstructionFormat` enum with all the instruction formats. // - The `pub enum Opcode` definition with all known opcodes, @@ -54,9 +54,9 @@ impl Opcode { } } -// This trait really belongs in libreader where it is used by the .cton file parser, but since it +// This trait really belongs in lib/reader where it is used by the .cton file parser, but since it // critically depends on the `opcode_name()` function which is needed here anyway, it lives in this -// module. This also saves us from runing the build script twice to generate code for the two +// module. This also saves us from running the build script twice to generate code for the two // separate crates. impl FromStr for Opcode { type Err = &'static str; diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index f1e0998403..c001c9f3c7 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -30,8 +30,9 @@ pub struct Type(u8); /// a SIMD vector. pub const VOID: Type = Type(0); -// Include code generated by `meta/gen_types.py`. This file contains constant definitions for all -// the scalar types as well as common vector types for 64, 128, 256, and 512-bit SIMD vectors. +// Include code generated by `lib/cretonne/meta/gen_types.py`. This file contains constant +// definitions for all the scalar types as well as common vector types for 64, 128, 256, and +// 512-bit SIMD vectors. include!(concat!(env!("OUT_DIR"), "/types.rs")); impl Type { diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index 62aa54e445..185e162f7f 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -1,7 +1,7 @@ //! Support types for generated encoding tables. //! //! This module contains types and functions for working with the encoding tables generated by -//! `meta/gen_encoding.py`. +//! `lib/cretonne/meta/gen_encoding.py`. use ir::{Type, Opcode}; use isa::Encoding; use constant_hash::{Table, probe}; diff --git a/lib/cretonne/src/isa/riscv/settings.rs b/lib/cretonne/src/isa/riscv/settings.rs index 57d8689ed1..c070612d08 100644 --- a/lib/cretonne/src/isa/riscv/settings.rs +++ b/lib/cretonne/src/isa/riscv/settings.rs @@ -3,8 +3,9 @@ use settings::{self, detail, Builder}; use std::fmt; -// Include code generated by `meta/gen_settings.py`. This file contains a public `Flags` struct -// with an impl for all of the settings defined in `meta/cretonne/settings.py`. +// Include code generated by `lib/cretonne/meta/gen_settings.py`. This file contains a public +// `Flags` struct with an impl for all of the settings defined in +// `lib/cretonne/meta/cretonne/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs")); #[cfg(test)] diff --git a/lib/cretonne/src/predicates.rs b/lib/cretonne/src/predicates.rs index 83676be5a0..a0e50402bf 100644 --- a/lib/cretonne/src/predicates.rs +++ b/lib/cretonne/src/predicates.rs @@ -1,7 +1,7 @@ //! Predicate functions for testing instruction fields. //! //! This module defines functions that are used by the instruction predicates defined by -//! `meta/cretonne/predicates.py` classes. +//! `lib/cretonne/meta/cretonne/predicates.py` classes. //! //! The predicates the operate on integer fields use `Into` as a shared trait bound. This //! bound is implemented by all the native integer types as well as `Imm64`. From ed917c394f841975c4e383d0bb72c9a5eeadf706 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Oct 2016 12:19:55 -0700 Subject: [PATCH 0385/3084] Add PEP 484 type annotations to a bunch of Python code. Along with the mypy tool, this helps find bugs in the Python code handling the instruction definition data structures. --- lib/cretonne/meta/build.py | 2 +- lib/cretonne/meta/cretonne/__init__.py | 60 ++++++++++++++++++-------- lib/cretonne/meta/cretonne/ast.py | 54 ++++++++++++++++++----- lib/cretonne/meta/cretonne/legalize.py | 6 +-- lib/cretonne/meta/cretonne/xform.py | 32 ++++++++++---- lib/cretonne/meta/isa/__init__.py | 2 + lib/cretonne/meta/isa/riscv/recipes.py | 7 +++ lib/cretonne/meta/srcgen.py | 19 +++++++- 8 files changed, 140 insertions(+), 42 deletions(-) diff --git a/lib/cretonne/meta/build.py b/lib/cretonne/meta/build.py index cfda7ae862..1554b6a54c 100644 --- a/lib/cretonne/meta/build.py +++ b/lib/cretonne/meta/build.py @@ -15,7 +15,7 @@ parser = argparse.ArgumentParser(description='Generate sources for Cretonne.') parser.add_argument('--out-dir', help='set output directory') args = parser.parse_args() -out_dir = args.out_dir +out_dir = args.out_dir # type: ignore isas = isa.all_isas() diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index d9005d6572..e8a82f650e 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -9,14 +9,23 @@ import re import math import importlib from collections import OrderedDict -from .predicates import And -from .ast import Apply +from .predicates import And, Predicate, FieldPredicate # noqa + +# The typing module is only required by mypy, and we don't use these imports +# outside type comments. +try: + from typing import Tuple, Union, Any, Iterable, Sequence # noqa + MaybeBoundInst = Union['Instruction', 'BoundInstruction'] + AnyPredicate = Union['Predicate', 'FieldPredicate'] +except ImportError: + pass camel_re = re.compile('(^|_)([a-z])') def camel_case(s): + # type: (str) -> str """Convert the string s to CamelCase""" return camel_re.sub(lambda m: m.group(2).upper(), s) @@ -133,7 +142,7 @@ class SettingGroup(object): """ # The currently open setting group. - _current = None + _current = None # type: SettingGroup def __init__(self, name, parent=None): self.name = name @@ -175,7 +184,6 @@ class SettingGroup(object): .format(self, SettingGroup._current)) SettingGroup._current = None if globs: - from .predicates import Predicate for name, obj in globs.iteritems(): if isinstance(obj, Setting): assert obj.name is None, obj.name @@ -381,10 +389,10 @@ class ValueType(object): """ # Map name -> ValueType. - _registry = dict() + _registry = dict() # type: Dict[str, ValueType] # List of all the scalar types. - all_scalars = list() + all_scalars = list() # type: List[ValueType] def __init__(self, name, membytes, doc): self.name = name @@ -534,7 +542,7 @@ class InstructionGroup(object): """ # The currently open instruction group. - _current = None + _current = None # type: InstructionGroup def open(self): """ @@ -644,15 +652,17 @@ class InstructionFormat(object): """ # Map (multiple_results, kind, kind, ...) -> InstructionFormat - _registry = dict() + _registry = dict() # type: Dict[Tuple, InstructionFormat] # All existing formats. - all_formats = list() + all_formats = list() # type: List[InstructionFormat] def __init__(self, *kinds, **kwargs): - self.name = kwargs.get('name', None) + # type: (*Union[OperandKind, Tuple[str, OperandKind]], **Any) -> None # noqa + self.name = kwargs.get('name', None) # type: str self.multiple_results = kwargs.get('multiple_results', False) self.boxed_storage = kwargs.get('boxed_storage', False) + self.members = list() # type: List[str] self.kinds = tuple(self._process_member_names(kinds)) # Which of self.kinds are `value`? @@ -660,7 +670,7 @@ class InstructionFormat(object): i for i, k in enumerate(self.kinds) if k is value) # The typevar_operand argument must point to a 'value' operand. - self.typevar_operand = kwargs.get('typevar_operand', None) + self.typevar_operand = kwargs.get('typevar_operand', None) # type: int if self.typevar_operand is not None: assert self.kinds[self.typevar_operand] is value, \ "typevar_operand must indicate a 'value' operand" @@ -678,6 +688,7 @@ class InstructionFormat(object): InstructionFormat.all_formats.append(self) def _process_member_names(self, kinds): + # type: (Sequence[Union[OperandKind, Tuple[str, OperandKind]]]) -> Iterable[OperandKind] # noqa """ Extract names of all the immediate operands in the kinds tuple. @@ -687,14 +698,14 @@ class InstructionFormat(object): Yields the operand kinds. """ - self.members = list() - for i, k in enumerate(kinds): - if isinstance(k, tuple): - member, k = k + for arg in kinds: + if isinstance(arg, OperandKind): + member = arg.default_member + k = arg else: - member = k.default_member - yield k + member, k = arg self.members.append(member) + yield k # Create `FormatField` instances for the immediates. if isinstance(k, ImmediateKind): @@ -704,6 +715,7 @@ class InstructionFormat(object): @staticmethod def lookup(ins, outs): + # type: (Sequence[Operand], Sequence[Operand]) -> InstructionFormat """ Find an existing instruction format that matches the given lists of instruction inputs and outputs. @@ -750,6 +762,7 @@ class FormatField(object): """ def __init__(self, format, operand, name): + # type: (InstructionFormat, int, str) -> None self.format = format self.operand = operand self.name = name @@ -758,6 +771,7 @@ class FormatField(object): return '{}.{}'.format(self.format.name, self.name) def rust_name(self): + # type: () -> str if self.format.boxed_storage: return 'data.' + self.name else: @@ -782,6 +796,7 @@ class Instruction(object): """ def __init__(self, name, doc, ins=(), outs=(), **kwargs): + # type: (str, str, Union[Sequence[Operand], Operand], Union[Sequence[Operand], Operand], **Any) -> None # noqa self.name = name self.camel_name = camel_case(name) self.__doc__ = doc @@ -898,6 +913,7 @@ class Instruction(object): @staticmethod def _to_operand_tuple(x): + # type: (Union[Sequence[Operand], Operand]) -> Tuple[Operand, ...] # Allow a single Operand instance instead of the awkward singleton # tuple syntax. if isinstance(x, Operand): @@ -909,6 +925,7 @@ class Instruction(object): return x def bind(self, *args): + # type: (*ValueType) -> BoundInstruction """ Bind a polymorphic instruction to a concrete list of type variable values. @@ -917,6 +934,7 @@ class Instruction(object): return BoundInstruction(self, args) def __getattr__(self, name): + # type: (str) -> BoundInstruction """ Bind a polymorphic instruction to a single type variable with dot syntax: @@ -926,6 +944,7 @@ class Instruction(object): return self.bind(ValueType.by_name(name)) def fully_bound(self): + # type: () -> Tuple[Instruction, Tuple[ValueType, ...]] """ Verify that all typevars have been bound, and return a `(inst, typevars)` pair. @@ -941,6 +960,7 @@ class Instruction(object): Create an `ast.Apply` AST node representing the application of this instruction to the arguments. """ + from .ast import Apply return Apply(self, args) @@ -950,6 +970,7 @@ class BoundInstruction(object): """ def __init__(self, inst, typevars): + # type: (Instruction, Tuple[ValueType, ...]) -> None self.inst = inst self.typevars = typevars assert len(typevars) <= 1 + len(inst.other_typevars) @@ -958,12 +979,14 @@ class BoundInstruction(object): return '.'.join([self.inst.name, ] + list(map(str, self.typevars))) def bind(self, *args): + # type: (*ValueType) -> BoundInstruction """ Bind additional typevars. """ return BoundInstruction(self.inst, self.typevars + args) def __getattr__(self, name): + # type: (str) -> BoundInstruction """ Bind an additional typevar dot syntax: @@ -972,6 +995,7 @@ class BoundInstruction(object): return self.bind(ValueType.by_name(name)) def fully_bound(self): + # type: () -> Tuple[Instruction, Tuple[ValueType, ...]] """ Verify that all typevars have been bound, and return a `(inst, typevars)` pair. @@ -989,6 +1013,7 @@ class BoundInstruction(object): Create an `ast.Apply` AST node representing the application of this instruction to the arguments. """ + from .ast import Apply return Apply(self, args) @@ -1139,6 +1164,7 @@ class Encoding(object): """ def __init__(self, cpumode, inst, recipe, encbits, instp=None, isap=None): + # type: (CPUMode, MaybeBoundInst, EncRecipe, int, AnyPredicate, AnyPredicate) -> None # noqa assert isinstance(cpumode, CPUMode) assert isinstance(recipe, EncRecipe) self.inst, self.typevars = inst.fully_bound() diff --git a/lib/cretonne/meta/cretonne/ast.py b/lib/cretonne/meta/cretonne/ast.py index 8c9c8576cb..693a0b5a2b 100644 --- a/lib/cretonne/meta/cretonne/ast.py +++ b/lib/cretonne/meta/cretonne/ast.py @@ -5,6 +5,12 @@ This module defines classes that can be used to create abstract syntax trees for patern matching an rewriting of cretonne instructions. """ from __future__ import absolute_import +from . import Instruction, BoundInstruction + +try: + from typing import Union, Tuple # noqa +except ImportError: + pass class Def(object): @@ -29,10 +35,12 @@ class Def(object): """ def __init__(self, defs, expr): + # type: (Union[Var, Tuple[Var, ...]], Apply) -> None if not isinstance(defs, tuple): - defs = (defs,) - assert isinstance(expr, Expr) - self.defs = defs + self.defs = (defs,) # type: Tuple[Var, ...] + else: + self.defs = defs + assert isinstance(expr, Apply) self.expr = expr def __repr__(self): @@ -42,7 +50,18 @@ class Def(object): if len(self.defs) == 1: return "{!s} << {!s}".format(self.defs[0], self.expr) else: - return "({}) << {!s}".format(", ".join(self.defs), self.expr) + return "({}) << {!s}".format( + ', '.join(map(str, self.defs)), self.expr) + + def root_inst(self): + # type: () -> Instruction + """Get the instruction at the root of this tree.""" + return self.expr.root_inst() + + def defs_expr(self): + # type: () -> Tuple[Tuple[Var, ...], Apply] + """Split into a defs tuple and an Apply expr.""" + return (self.defs, self.expr) class Expr(object): @@ -50,12 +69,6 @@ class Expr(object): An AST expression. """ - def __rlshift__(self, other): - """ - Define variables using `var << expr` or `(v1, v2) << expr`. - """ - return Def(other, self) - class Var(Expr): """ @@ -63,6 +76,7 @@ class Var(Expr): """ def __init__(self, name): + # type: (str) -> None self.name = name # Bitmask of contexts where this variable is defined. # See XForm._rewrite_defs(). @@ -98,16 +112,24 @@ class Apply(Expr): """ def __init__(self, inst, args): - from . import BoundInstruction + # type: (Union[Instruction, BoundInstruction], Tuple[Expr, ...]) -> None # noqa if isinstance(inst, BoundInstruction): self.inst = inst.inst self.typevars = inst.typevars else: + assert isinstance(inst, Instruction) self.inst = inst self.typevars = () self.args = args assert len(self.inst.ins) == len(args) + def __rlshift__(self, other): + # type: (Union[Var, Tuple[Var, ...]]) -> Def + """ + Define variables using `var << expr` or `(v1, v2) << expr`. + """ + return Def(other, self) + def instname(self): i = self.inst.name for t in self.typevars: @@ -120,3 +142,13 @@ class Apply(Expr): def __str__(self): args = ', '.join(map(str, self.args)) return '{}({})'.format(self.instname(), args) + + def root_inst(self): + # type: () -> Instruction + """Get the instruction at the root of this tree.""" + return self.inst + + def defs_expr(self): + # type: () -> Tuple[Tuple[Var, ...], Apply] + """Split into a defs tuple and an Apply expr.""" + return ((), self) diff --git a/lib/cretonne/meta/cretonne/legalize.py b/lib/cretonne/meta/cretonne/legalize.py index 9d7b218755..21073d0aec 100644 --- a/lib/cretonne/meta/cretonne/legalize.py +++ b/lib/cretonne/meta/cretonne/legalize.py @@ -15,7 +15,7 @@ from .ast import Var from .xform import Rtl, XFormGroup -narrow = XFormGroup(""" +narrow = XFormGroup('narrow', """ Legalize instructions by narrowing. The transformations in the 'narrow' group work by expressing @@ -24,7 +24,7 @@ narrow = XFormGroup(""" operations are expressed in terms of smaller integer types. """) -expand = XFormGroup(""" +expand = XFormGroup('expand', """ Legalize instructions by expansion. Rewrite instructions in terms of other instructions, generally @@ -114,5 +114,5 @@ expand.legalize( Rtl( (a1, b1) << isub_bout(x, y), (a, b2) << isub_bout(a1, b_in), - c << bor(c1, c2) + c << bor(b1, b2) )) diff --git a/lib/cretonne/meta/cretonne/xform.py b/lib/cretonne/meta/cretonne/xform.py index 8015ee04b0..1100e4721f 100644 --- a/lib/cretonne/meta/cretonne/xform.py +++ b/lib/cretonne/meta/cretonne/xform.py @@ -2,7 +2,13 @@ Instruction transformations. """ from __future__ import absolute_import -from .ast import Def, Var, Apply +from .ast import Def, Var, Apply, Expr # noqa + +try: + from typing import Union, Iterator, Sequence, Iterable # noqa + DefApply = Union[Def, Apply] +except ImportError: + pass SRCCTX = 1 @@ -21,9 +27,11 @@ class Rtl(object): """ def __init__(self, *args): + # type: (*DefApply) -> None self.rtl = args def __iter__(self): + # type: () -> Iterator[DefApply] return iter(self.rtl) @@ -55,16 +63,17 @@ class XForm(object): """ def __init__(self, src, dst): + # type: (Rtl, Rtl) -> None self.src = src self.dst = dst # Variables that are inputs to the source pattern. - self.inputs = list() + self.inputs = list() # type: List[Var] # Variables defined in either src or dst. - self.defs = list() + self.defs = list() # type: List[Var] # Rewrite variables in src and dst RTL lists to our own copies. # Map name -> private Var. - symtab = dict() + symtab = dict() # type: Dict[str, Var] self._rewrite_rtl(src, symtab, SRCCTX) num_src_inputs = len(self.inputs) self._rewrite_rtl(dst, symtab, DSTCTX) @@ -90,7 +99,8 @@ class XForm(object): return s def _rewrite_rtl(self, rtl, symtab, context): - for line in rtl: + # type: (Rtl, Dict[str, Var], int) -> None + for line in rtl.rtl: if isinstance(line, Def): line.defs = tuple( self._rewrite_defs(line.defs, symtab, context)) @@ -100,6 +110,7 @@ class XForm(object): self._rewrite_expr(expr, symtab, context) def _rewrite_expr(self, expr, symtab, context): + # type: (Apply, Dict[str, Var], int) -> None """ Find all uses of variables in `expr` and replace them with our own local symbols. @@ -113,6 +124,7 @@ class XForm(object): self._rewrite_uses(expr, stack, symtab, context)) def _rewrite_defs(self, defs, symtab, context): + # type: (Sequence[Var], Dict[str, Var], int) -> Iterable[Var] """ Given a tuple of symbols defined in a Def, rewrite them to local symbols. Yield the new locals. @@ -131,6 +143,7 @@ class XForm(object): yield var def _rewrite_uses(self, expr, stack, symtab, context): + # type: (Apply, List[Apply], Dict[str, Var], int) -> Iterable[Expr] """ Given an `Apply` expr, rewrite all uses in its arguments to local variables. Yield a sequence of new arguments. @@ -140,7 +153,7 @@ class XForm(object): for arg, operand in zip(expr.args, expr.inst.ins): # Nested instructions are allowed. Visit recursively. if isinstance(arg, Apply): - stack.push(arg) + stack.append(arg) yield arg continue if not isinstance(arg, Var): @@ -169,11 +182,14 @@ class XFormGroup(object): A group of related transformations. """ - def __init__(self, doc): - self.xforms = list() + def __init__(self, name, doc): + # type: (str, str) -> None + self.xforms = list() # type: List[XForm] + self.name = name self.__doc__ = doc def legalize(self, src, dst): + # type: (Union[Def, Apply], Rtl) -> None """ Add a legalization pattern to this group. diff --git a/lib/cretonne/meta/isa/__init__.py b/lib/cretonne/meta/isa/__init__.py index addc70b8a9..9412f6502c 100644 --- a/lib/cretonne/meta/isa/__init__.py +++ b/lib/cretonne/meta/isa/__init__.py @@ -7,9 +7,11 @@ architecture supported by Cretonne. """ from __future__ import absolute_import from . import riscv +from cretonne import TargetISA # noqa def all_isas(): + # type: () -> List[TargetISA] """ Get a list of all the supported target ISAs. Each target ISA is represented as a :py:class:`cretonne.TargetISA` instance. diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index db6d7021c8..375b0e2150 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -22,37 +22,44 @@ from cretonne.predicates import IsSignedInt def LOAD(funct3): + # type: (int) -> int assert funct3 <= 0b111 return 0b00000 | (funct3 << 5) def STORE(funct3): + # type: (int) -> int assert funct3 <= 0b111 return 0b01000 | (funct3 << 5) def BRANCH(funct3): + # type: (int) -> int assert funct3 <= 0b111 return 0b11000 | (funct3 << 5) def OPIMM(funct3, funct7=0): + # type: (int, int) -> int assert funct3 <= 0b111 return 0b00100 | (funct3 << 5) | (funct7 << 8) def OPIMM32(funct3, funct7=0): + # type: (int, int) -> int assert funct3 <= 0b111 return 0b00110 | (funct3 << 5) | (funct7 << 8) def OP(funct3, funct7): + # type: (int, int) -> int assert funct3 <= 0b111 assert funct7 <= 0b1111111 return 0b01100 | (funct3 << 5) | (funct7 << 8) def OP32(funct3, funct7): + # type: (int, int) -> int assert funct3 <= 0b111 assert funct7 <= 0b1111111 return 0b01110 | (funct3 << 5) | (funct7 << 8) diff --git a/lib/cretonne/meta/srcgen.py b/lib/cretonne/meta/srcgen.py index 962a881436..75b102f534 100644 --- a/lib/cretonne/meta/srcgen.py +++ b/lib/cretonne/meta/srcgen.py @@ -10,6 +10,11 @@ import sys import os import re +try: + from typing import Any # noqa +except ImportError: + pass + class Formatter(object): """ @@ -38,19 +43,23 @@ class Formatter(object): shiftwidth = 4 def __init__(self): + # type: () -> None self.indent = '' - self.lines = [] + self.lines = [] # type: List[str] def indent_push(self): + # type: () -> None """Increase current indentation level by one.""" self.indent += ' ' * self.shiftwidth def indent_pop(self): + # type: () -> None """Decrease indentation by one level.""" assert self.indent != '', 'Already at top level indentation' self.indent = self.indent[0:-self.shiftwidth] def line(self, s=None): + # type: (str) -> None """Add an indented line.""" if s: self.lines.append('{}{}\n'.format(self.indent, s)) @@ -58,6 +67,7 @@ class Formatter(object): self.lines.append('\n') def outdented_line(self, s): + # type: (str) -> None """ Emit a line outdented one level. @@ -67,12 +77,14 @@ class Formatter(object): self.lines.append('{}{}\n'.format(self.indent[0:-self.shiftwidth], s)) def writelines(self, f=None): + # type: (Any) -> None """Write all lines to `f`.""" if not f: f = sys.stdout f.writelines(self.lines) def update_file(self, filename, directory): + # type: (str, str) -> None if directory is not None: filename = os.path.join(directory, filename) with open(filename, 'w') as f: @@ -80,10 +92,12 @@ class Formatter(object): class _IndentedScope(object): def __init__(self, fmt, after): + # type: (Formatter, str) -> None self.fmt = fmt self.after = after def __enter__(self): + # type: () -> None self.fmt.indent_push() def __exit__(self, t, v, tb): @@ -92,6 +106,7 @@ class Formatter(object): self.fmt.line(self.after) def indented(self, before=None, after=None): + # type: (str, str) -> Formatter._IndentedScope """ Return a scope object for use with a `with` statement: @@ -108,7 +123,7 @@ class Formatter(object): """ if before: self.line(before) - return self._IndentedScope(self, after) + return Formatter._IndentedScope(self, after) def format(self, fmt, *args): self.line(fmt.format(*args)) From 2a8f8f79fffed3046b32f87705b3808526839971 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Oct 2016 12:24:16 -0700 Subject: [PATCH 0386/3084] Create FormatField attributes on demand. The InstructionFormat objects make their non-value operands available as FormatField attributes for use by predicates etc. Compute these on demand instead of up front. This makes it possible for the mypy tool to infer the types of these attributes from the __getattr__ signature. --- lib/cretonne/meta/cretonne/__init__.py | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index e8a82f650e..a8865e1966 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -707,11 +707,24 @@ class InstructionFormat(object): self.members.append(member) yield k - # Create `FormatField` instances for the immediates. - if isinstance(k, ImmediateKind): - assert not hasattr(self, member), "Duplicate member name" - field = FormatField(self, i, member) - setattr(self, member, field) + def __getattr__(self, attr): + # type: (str) -> FormatField + """ + Make instruction format members available as attributes. + + Each non-value format member becomes a corresponding `FormatField` + attribute. + """ + try: + i = self.members.index(attr) + except ValueError: + raise AttributeError( + '{} is neither a {} member or a ' + .format(attr, self.name) + + 'normal InstructionFormat attribute') + field = FormatField(self, i, attr) + setattr(self, attr, field) + return field @staticmethod def lookup(ins, outs): From d7a479de6da5463182a987056518a495f527d153 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Oct 2016 12:26:23 -0700 Subject: [PATCH 0387/3084] Run mypy if it's available. The Python tools pyliint, flake8, and mypy are only run if they exist in $PATH. --- lib/cretonne/meta/check.sh | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/meta/check.sh b/lib/cretonne/meta/check.sh index f01653b57e..072066ebec 100755 --- a/lib/cretonne/meta/check.sh +++ b/lib/cretonne/meta/check.sh @@ -2,11 +2,21 @@ set -e cd $(dirname "$0") -# Run unit tests. +echo "=== Python unit tests ===" python -m unittest discover +runif() { + if command -v "$1" > /dev/null; then + echo "=== $1 ===" + "$@" + else + echo "$1 not found" + fi +} + # Check Python sources for Python 3 compatibility using pylint. # # Install pylint with 'pip install pylint'. -pylint --py3k --reports=no -- *.py cretonne isa -flake8 . +runif pylint --py3k --reports=no -- *.py cretonne isa +runif flake8 . +runif mypy --py2 build.py From 15f626ccc035c0f912559232af0222087650eeeb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 25 Oct 2016 16:52:15 -0700 Subject: [PATCH 0388/3084] Begin generating code for the legalizer. This is a work in progress. The 'legalizer.rs' file generated by gen_legalizer.py is not used for anything yet. Add PEP 484 type annotations to a bunch of Python code. --- lib/cretonne/meta/build.py | 2 + lib/cretonne/meta/gen_legalizer.py | 121 +++++++++++++++++++++++++++++ lib/cretonne/src/ir/layout.rs | 9 +++ 3 files changed, 132 insertions(+) create mode 100644 lib/cretonne/meta/gen_legalizer.py diff --git a/lib/cretonne/meta/build.py b/lib/cretonne/meta/build.py index 1554b6a54c..51f8e7c056 100644 --- a/lib/cretonne/meta/build.py +++ b/lib/cretonne/meta/build.py @@ -10,6 +10,7 @@ import gen_instr import gen_settings import gen_build_deps import gen_encoding +import gen_legalizer parser = argparse.ArgumentParser(description='Generate sources for Cretonne.') parser.add_argument('--out-dir', help='set output directory') @@ -23,4 +24,5 @@ gen_types.generate(out_dir) gen_instr.generate(isas, out_dir) gen_settings.generate(isas, out_dir) gen_encoding.generate(isas, out_dir) +gen_legalizer.generate(isas, out_dir) gen_build_deps.generate() diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py new file mode 100644 index 0000000000..0fdfc921af --- /dev/null +++ b/lib/cretonne/meta/gen_legalizer.py @@ -0,0 +1,121 @@ +""" +Generate legalizer transformations. + +The transformations defined in the `cretonne.legalize` module are all of the +macro-expansion form where the input pattern is a single instruction. We +generate a Rust function for each `XFormGroup` which takes a `Cursor` pointing +at the instruction to be legalized. The expanded destination pattern replaces +the input instruction. +""" +from __future__ import absolute_import +from srcgen import Formatter +import cretonne.legalize as legalize +from cretonne.ast import Def, Apply # noqa +from cretonne.xform import XForm, XFormGroup # noqa + +try: + from typing import Union # noqa + DefApply = Union[Def, Apply] +except ImportError: + pass + + +def unwrap_inst(iref, node, fmt): + # type: (str, DefApply, Formatter) -> None + """ + Given a `Def` or `Apply` node, emit code that extracts all the instruction + fields from `dfg[iref]`. + + Create local variables named after the `Var` instances in `node`. + + :param iref: Name of the `Inst` reference to unwrap. + :param node: `Def` or `Apply` node providing variable names. + + """ + fmt.comment('Unwrap {}'.format(node)) + defs, expr = node.defs_expr() + iform = expr.inst.format + nvops = len(iform.value_operands) + + # The tuple of locals we're extracting is `expr.args`. + with fmt.indented( + 'let ({}) = if let InstructionData::{} {{' + .format(', '.join(map(str, expr.args)), iform.name), '};'): + if iform.boxed_storage: + # This format indirects to a largish `data` struct. + fmt.line('ref data,') + else: + # Fields are encoded directly. + for m in iform.members: + if m: + fmt.line('{},'.format(m)) + if nvops == 1: + fmt.line('arg,') + elif nvops > 1: + fmt.line('args,') + fmt.line('..') + fmt.outdented_line('} = dfg[inst] {') + # Generate the values for the tuple. + outs = list() + prefix = 'data.' if iform.boxed_storage else '' + for i, m in enumerate(iform.members): + if m: + outs.append(prefix + m) + else: + # This is a value operand. + if nvops == 1: + outs.append(prefix + 'arg') + else: + outs.append( + '{}args[{}]'.format( + prefix, iform.value_operands.index(i))) + fmt.line('({})'.format(', '.join(outs))) + fmt.outdented_line('} else {') + fmt.line('unimplemented!("bad instruction format")') + + +def gen_xform(xform, fmt): + # type: (XForm, Formatter) -> None + """ + Emit code for `xform`, assuming the the opcode of xform's root instruction + has already been matched. + + `inst: Inst` is the variable to be replaced. It is pointed to by `pos: + Cursor`. + `dfg: DataFlowGraph` is available and mutable. + """ + unwrap_inst('inst', xform.src.rtl[0], fmt) + + +def gen_xform_group(xgrp, fmt): + # type: (XFormGroup, Formatter) -> None + fmt.doc_comment(""" + Legalize the instruction pointed to by `pos`. + + Return the first instruction in the expansion, and leave `pos` pointing + at the last instruction in the expansion. + """) + with fmt.indented( + 'fn ' + xgrp.name + + '(pos: &mut Cursor, dfg: &mut DataFlowGraph) -> ' + + 'Option {{', + '}'): + # Gen the instruction to be legalized. The cursor we're passed must be + # pointing at an instruction. + fmt.line('let inst = pos.current_inst().expect("need instruction");') + + with fmt.indented('match dfg[inst].opcode() {', '}'): + for xform in xgrp.xforms: + inst = xform.src.rtl[0].root_inst() + with fmt.indented( + 'Opcode::{} => {{'.format(inst.camel_name), '}'): + gen_xform(xform, fmt) + # We'll assume there are uncovered opcodes. + fmt.line('_ => None,') + + +def generate(isas, out_dir): + fmt = Formatter() + gen_xform_group(legalize.narrow, fmt) + gen_xform_group(legalize.expand, fmt) + fmt.update_file('legalizer.rs', out_dir) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 864420c4f5..fdd2305747 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -391,6 +391,15 @@ impl<'f> Cursor<'f> { } } + /// Get the instruction corresponding to the current position, if any. + pub fn current_inst(&self) -> Option { + use self::CursorPosition::*; + match self.pos { + At(inst) => Some(inst), + _ => None, + } + } + /// Go to a specific instruction which must be inserted in the layout. /// New instructions will be inserted before `inst`. pub fn goto_inst(&mut self, inst: Inst) { From 447baf015e6f68d3a61ee9dabf59a49bd1427ffd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Oct 2016 17:34:45 -0700 Subject: [PATCH 0389/3084] Require documentation on filecheck public items. --- lib/filecheck/src/lib.rs | 2 ++ lib/filecheck/src/variable.rs | 2 ++ 2 files changed, 4 insertions(+) diff --git a/lib/filecheck/src/lib.rs b/lib/filecheck/src/lib.rs index f6c53c5dcf..66fe6036ab 100644 --- a/lib/filecheck/src/lib.rs +++ b/lib/filecheck/src/lib.rs @@ -234,6 +234,8 @@ //! This will match `"one, two"` , but not `"one,two"`. Without the `$()`, trailing whitespace //! would be trimmed from the pattern. +#![deny(missing_docs)] + pub use error::{Error, Result}; pub use variable::{VariableMap, Value, NO_VARIABLES}; pub use checker::{Checker, CheckerBuilder}; diff --git a/lib/filecheck/src/variable.rs b/lib/filecheck/src/variable.rs index 51bc38fc07..1a43f1428a 100644 --- a/lib/filecheck/src/variable.rs +++ b/lib/filecheck/src/variable.rs @@ -19,7 +19,9 @@ pub fn varname_prefix(s: &str) -> usize { /// A variable can contain either a regular expression or plain text. #[derive(Debug, Clone, PartialEq, Eq)] pub enum Value<'a> { + /// Verbatim text. Text(Cow<'a, str>), + /// Regular expression. Regex(Cow<'a, str>), } From e2418c6ec9dd7f911c8de2350b23a5ab1a92c014 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Oct 2016 17:43:18 -0700 Subject: [PATCH 0390/3084] Require documentation on reader public items. --- lib/reader/src/error.rs | 4 ++++ lib/reader/src/lib.rs | 2 ++ lib/reader/src/testcommand.rs | 10 ++++++++++ lib/reader/src/testfile.rs | 5 +++++ 4 files changed, 21 insertions(+) diff --git a/lib/reader/src/error.rs b/lib/reader/src/error.rs index cf8bd12f08..c49e523b72 100644 --- a/lib/reader/src/error.rs +++ b/lib/reader/src/error.rs @@ -8,13 +8,16 @@ use std::result; /// The location of a `Token` or `Error`. #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] pub struct Location { + /// Line number, starting from 1. pub line_number: usize, } /// A parse error is returned when the parse failed. #[derive(Debug)] pub struct Error { + /// Location of the error. pub location: Location, + /// Error message. pub message: String, } @@ -24,6 +27,7 @@ impl fmt::Display for Error { } } +/// Result of a parser operation. The `Error` variant includes a location. pub type Result = result::Result; // Create an `Err` variant of `Result` from a location and `format!` args. diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index ef81e7f210..25aa25aa98 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -3,6 +3,8 @@ //! The cton_reader library supports reading .cton files. This functionality is needed for testing //! Cretonne, but is not essential for a JIT compiler. +#![deny(missing_docs)] + extern crate cretonne; pub use error::{Location, Result, Error}; diff --git a/lib/reader/src/testcommand.rs b/lib/reader/src/testcommand.rs index 3eba0d0a53..012c7033e1 100644 --- a/lib/reader/src/testcommand.rs +++ b/lib/reader/src/testcommand.rs @@ -14,19 +14,27 @@ use std::fmt::{self, Display, Formatter}; +/// A command appearing in a test file. #[derive(Clone, PartialEq, Eq, Debug)] pub struct TestCommand<'a> { + /// The command name as a string. pub command: &'a str, + /// The options following the command name. pub options: Vec>, } +/// An option on a test command. #[derive(Clone, PartialEq, Eq, Debug)] pub enum TestOption<'a> { + /// Single identifier flag: `foo`. Flag(&'a str), + /// A value assigned to an identifier: `foo=bar`. Value(&'a str, &'a str), } impl<'a> TestCommand<'a> { + /// Create a new TestCommand by parsing `s`. + /// The returned command contains references into `s`. pub fn new(s: &'a str) -> TestCommand<'a> { let mut parts = s.split_whitespace(); let cmd = parts.next().unwrap_or(""); @@ -48,6 +56,8 @@ impl<'a> Display for TestCommand<'a> { } impl<'a> TestOption<'a> { + /// Create a new TestOption by parsing `s`. + /// The returned option contains references into `s`. pub fn new(s: &'a str) -> TestOption<'a> { match s.find('=') { None => TestOption::Flag(s), diff --git a/lib/reader/src/testfile.rs b/lib/reader/src/testfile.rs index 30ed59074b..deeaf04e5a 100644 --- a/lib/reader/src/testfile.rs +++ b/lib/reader/src/testfile.rs @@ -20,6 +20,7 @@ pub struct TestFile<'a> { pub commands: Vec>, /// `isa bar ...` lines. pub isa_spec: IsaSpec, + /// Parsed functions and additional details about each function. pub functions: Vec<(Function, Details<'a>)>, } @@ -28,8 +29,12 @@ pub struct TestFile<'a> { /// The details to not affect the semantics of the function. #[derive(Debug)] pub struct Details<'a> { + /// Location of the `function` keyword that begins this function. pub location: Location, + /// Annotation comments that appeared inside or after the function. pub comments: Vec>, + /// Mapping of source entity numbers to parsed entity numbers. + /// Source locations of parsed entities. pub map: SourceMap, } From 80823b5fc4b2038d25180a70ddfc78b9e14463cb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Oct 2016 18:41:39 -0700 Subject: [PATCH 0391/3084] Require documentation on cretonne public items. --- lib/cretonne/meta/cretonne/__init__.py | 5 +++ lib/cretonne/meta/gen_instr.py | 3 ++ lib/cretonne/meta/gen_settings.py | 11 ++++-- lib/cretonne/src/cfg.rs | 13 ++----- lib/cretonne/src/dominator_tree.rs | 3 +- lib/cretonne/src/ir/condcodes.rs | 54 +++++++++++++++++++------- lib/cretonne/src/ir/entities.rs | 23 ++++++++--- lib/cretonne/src/ir/extfunc.rs | 8 ++++ lib/cretonne/src/ir/funcname.rs | 1 + lib/cretonne/src/ir/immediates.rs | 3 ++ lib/cretonne/src/ir/instructions.rs | 15 ++++++- lib/cretonne/src/isa/mod.rs | 1 + lib/cretonne/src/isa/riscv/mod.rs | 1 + lib/cretonne/src/lib.rs | 8 ++-- lib/cretonne/src/settings.rs | 11 +++++- lib/cretonne/src/verifier.rs | 8 +++- 16 files changed, 124 insertions(+), 44 deletions(-) diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index a8865e1966..6672d6d869 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -707,6 +707,11 @@ class InstructionFormat(object): self.members.append(member) yield k + def __str__(self): + args = ', '.join('{}: {}'.format(m, k) if m else str(k) + for m, k in zip(self.members, self.kinds)) + return '{}({})'.format(self.name, args) + def __getattr__(self, attr): # type: (str) -> FormatField """ diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 8a7e813223..bb14d36375 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -19,6 +19,7 @@ def gen_formats(fmt): fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug)]') with fmt.indented('pub enum InstructionFormat {', '}'): for f in cretonne.InstructionFormat.all_formats: + fmt.doc_comment(str(f)) fmt.line(f.name + ',') fmt.line() @@ -172,6 +173,7 @@ def gen_opcodes(groups, fmt): fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug)]') instrs = [] with fmt.indented('pub enum Opcode {', '}'): + fmt.doc_comment('An invalid opcode.') fmt.line('NotAnOpcode,') for g in groups: for i in g.instructions: @@ -364,6 +366,7 @@ def gen_format_constructor(iform, fmt): proto = '{}({})'.format(iform.name, ', '.join(args)) proto += " -> (Inst, &'f mut DataFlowGraph)" + fmt.doc_comment(str(iform)) fmt.line('#[allow(non_snake_case)]') with fmt.indented('fn {} {{'.format(proto), '}'): # Generate the instruction data. diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index c3c4710ea3..8e35d05909 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -16,10 +16,12 @@ def gen_enum_types(sgrp, fmt): if not isinstance(setting, EnumSetting): continue ty = camel_case(setting.name) + fmt.doc_comment('Values for {}.'.format(setting)) fmt.line('#[derive(Debug, PartialEq, Eq)]') - fmt.line( - 'pub enum {} {{ {} }}' - .format(ty, ", ".join(camel_case(v) for v in setting.values))) + with fmt.indented('pub enum {} {{'.format(ty), '}'): + for v in setting.values: + fmt.doc_comment('`{}`.'.format(v)) + fmt.line(camel_case(v) + ',') def gen_getter(setting, sgrp, fmt): @@ -70,7 +72,7 @@ def gen_getters(sgrp, fmt): """ fmt.doc_comment("User-defined settings.") with fmt.indented('impl Flags {', '}'): - # Dynamic numbered predicate getter. + fmt.doc_comment('Dynamic numbered predicate getter.') with fmt.indented( 'pub fn numbered_predicate(&self, p: usize) -> bool {', '}'): fmt.line( @@ -187,6 +189,7 @@ def gen_constructor(sgrp, parent, fmt): if sgrp.parent: p = sgrp.parent args = '{}: &{}::Flags, {}'.format(p.name, p.qual_mod, args) + fmt.doc_comment('Create flags {} settings group.'.format(sgrp.name)) with fmt.indented( 'pub fn new({}) -> Flags {{'.format(args), '}'): fmt.line('let bvec = builder.state_for("{}");'.format(sgrp.name)) diff --git a/lib/cretonne/src/cfg.rs b/lib/cretonne/src/cfg.rs index 76ccc27965..f68d8bcbf4 100644 --- a/lib/cretonne/src/cfg.rs +++ b/lib/cretonne/src/cfg.rs @@ -33,19 +33,12 @@ pub type BasicBlock = (Ebb, Inst); /// A container for the successors and predecessors of some Ebb. #[derive(Debug, Clone, Default)] pub struct CFGNode { + /// EBBs that are the targets of branches and jumps in this EBB. pub successors: Vec, + /// Basic blocks that can branch or jump to this EBB. pub predecessors: Vec, } -impl CFGNode { - pub fn new() -> CFGNode { - CFGNode { - successors: Vec::new(), - predecessors: Vec::new(), - } - } -} - /// The Control Flow Graph maintains a mapping of ebbs to their predecessors /// and successors where predecessors are basic blocks and successors are /// extended basic blocks. @@ -88,10 +81,12 @@ impl ControlFlowGraph { self.data[to].predecessors.push(from); } + /// Get the CFG predecessor basic blocks to `ebb`. pub fn get_predecessors(&self, ebb: Ebb) -> &Vec { &self.data[ebb].predecessors } + /// Get the CFG successors to `ebb`. pub fn get_successors(&self, ebb: Ebb) -> &Vec { &self.data[ebb].successors } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index c81e579f6b..e331f5657b 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -1,10 +1,11 @@ -/// ! A Dominator Tree represented as mappings of Ebbs to their immediate dominator. +//! A Dominator Tree represented as mappings of Ebbs to their immediate dominator. use cfg::*; use ir::Ebb; use ir::entities::NO_INST; use entity_map::EntityMap; +/// The dominator tree for a single function. pub struct DominatorTree { data: EntityMap>, } diff --git a/lib/cretonne/src/ir/condcodes.rs b/lib/cretonne/src/ir/condcodes.rs index 692ecda96f..dfa6b054f5 100644 --- a/lib/cretonne/src/ir/condcodes.rs +++ b/lib/cretonne/src/ir/condcodes.rs @@ -29,15 +29,25 @@ pub trait CondCode: Copy { /// difference. #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub enum IntCC { + /// `==`. Equal, + /// `!=`. NotEqual, + /// Signed `<`. SignedLessThan, + /// Signed `>=`. SignedGreaterThanOrEqual, + /// Signed `>`. SignedGreaterThan, + /// Signed `<=`. SignedLessThanOrEqual, + /// Unsigned `<`. UnsignedLessThan, + /// Unsigned `>=`. UnsignedGreaterThanOrEqual, + /// Unsigned `>`. UnsignedGreaterThan, + /// Unsigned `<=`. UnsignedLessThanOrEqual, } @@ -131,24 +141,38 @@ impl FromStr for IntCC { /// except the impossible `!UN & !EQ & !LT & !GT` and the always true `UN | EQ | LT | GT`. #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub enum FloatCC { - Ordered, // EQ | LT | GT - Unordered, // UN + /// EQ | LT | GT + Ordered, + /// UN + Unordered, - Equal, // EQ - // The C '!=' operator is the inverse of '==': NotEqual. - NotEqual, // UN | LT | GT - OrderedNotEqual, // LT | GT - UnorderedOrEqual, // UN | EQ + /// EQ + Equal, + /// The C '!=' operator is the inverse of '==': `NotEqual`. + /// UN | LT | GT + NotEqual, + /// LT | GT + OrderedNotEqual, + /// UN | EQ + UnorderedOrEqual, - LessThan, // LT - LessThanOrEqual, // LT | EQ - GreaterThan, // GT - GreaterThanOrEqual, // GT | EQ + /// LT + LessThan, + /// LT | EQ + LessThanOrEqual, + /// GT + GreaterThan, + /// GT | EQ + GreaterThanOrEqual, - UnorderedOrLessThan, // UN | LT - UnorderedOrLessThanOrEqual, // UN | LT | EQ - UnorderedOrGreaterThan, // UN | GT - UnorderedOrGreaterThanOrEqual, // UN | GT | EQ + /// UN | LT + UnorderedOrLessThan, + /// UN | LT | EQ + UnorderedOrLessThanOrEqual, + /// UN | GT + UnorderedOrGreaterThan, + /// UN | GT | EQ + UnorderedOrGreaterThanOrEqual, } impl CondCode for FloatCC { diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index bd8913e5bf..384e4d3ae9 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -98,16 +98,16 @@ impl Default for Inst { #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct Value(u32); -// Value references can either reference an instruction directly, or they can refer to the extended -// value table. +/// Value references can either reference an instruction directly, or they can refer to the +/// extended value table. pub enum ExpandedValue { - // This is the first value produced by the referenced instruction. + /// This is the first value produced by the referenced instruction. Direct(Inst), - // This value is described in the extended value table. + /// This value is described in the extended value table. Table(usize), - // This is NO_VALUE. + /// This is NO_VALUE. None, } @@ -135,19 +135,23 @@ impl Value { None } } + /// Create a `Direct` value corresponding to the first value produced by `i`. pub fn new_direct(i: Inst) -> Value { let encoding = i.index() * 2; assert!(encoding < u32::MAX as usize); Value(encoding as u32) } + /// Create a `Table` value referring to entry `i` in the `DataFlowGraph.extended_values` table. + /// This constructor should not be used directly. Use the public `DataFlowGraph` methods to + /// manipulate values. pub fn new_table(index: usize) -> Value { let encoding = index * 2 + 1; assert!(encoding < u32::MAX as usize); Value(encoding as u32) } - // Expand the internal representation into something useful. + /// Expand the internal representation into something useful. pub fn expand(&self) -> ExpandedValue { use self::ExpandedValue::*; if *self == NO_VALUE { @@ -312,12 +316,19 @@ impl Default for SigRef { pub enum AnyEntity { /// The whole function. Function, + /// An extended basic block. Ebb(Ebb), + /// An instruction. Inst(Inst), + /// An SSA value. Value(Value), + /// A stack slot. StackSlot(StackSlot), + /// A jump table. JumpTable(JumpTable), + /// An external function. FuncRef(FuncRef), + /// A function call signature. SigRef(SigRef), } diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 6f89d5d307..a47ff4ad7a 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -14,11 +14,14 @@ use ir::{Type, FunctionName, SigRef}; /// details that are needed to call a function correctly. #[derive(Clone, PartialEq, Eq, Debug)] pub struct Signature { + /// Types of the arguments passed to the function. pub argument_types: Vec, + /// Types returned from the function. pub return_types: Vec, } impl Signature { + /// Create a new blank signature. pub fn new() -> Signature { Signature { argument_types: Vec::new(), @@ -59,13 +62,16 @@ impl Display for Signature { /// how the argument is passed. #[derive(Copy, Clone, PartialEq, Eq, Debug)] pub struct ArgumentType { + /// Type of the argument value. pub value_type: Type, + /// Method for extending argument to a full register. pub extension: ArgumentExtension, /// Place this argument in a register if possible. pub inreg: bool, } impl ArgumentType { + /// Create an argument type with default flags. pub fn new(vt: Type) -> ArgumentType { ArgumentType { value_type: vt, @@ -109,7 +115,9 @@ pub enum ArgumentExtension { /// Information about a function that can be called directly with a direct `call` instruction. #[derive(Clone, Debug)] pub struct ExtFuncData { + /// Name of the external function. pub name: FunctionName, + /// Call signature of function. pub signature: SigRef, } diff --git a/lib/cretonne/src/ir/funcname.rs b/lib/cretonne/src/ir/funcname.rs index 0f6f106365..d84b25ac2b 100644 --- a/lib/cretonne/src/ir/funcname.rs +++ b/lib/cretonne/src/ir/funcname.rs @@ -14,6 +14,7 @@ use std::ascii::AsciiExt; pub struct FunctionName(String); impl FunctionName { + /// Create new function name equal to `s`. pub fn new>(s: S) -> FunctionName { FunctionName(s.into()) } diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 0395d13f26..e2fc0c446d 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -17,6 +17,7 @@ use std::str::FromStr; pub struct Imm64(i64); impl Imm64 { + /// Create a new `Imm64` representing the signed number `x`. pub fn new(x: i64) -> Imm64 { Imm64(x) } @@ -374,6 +375,7 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { } impl Ieee32 { + /// Create a new `Ieee32` representing the number `x`. pub fn new(x: f32) -> Ieee32 { Ieee32(x) } @@ -403,6 +405,7 @@ impl FromStr for Ieee32 { } impl Ieee64 { + /// Create a new `Ieee64` representing the number `x`. pub fn new(x: f64) -> Ieee64 { Ieee64(x) } diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 103dcd4fe5..f601169beb 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -93,6 +93,7 @@ impl FromStr for Opcode { /// 16 bytes on 64-bit architectures. If more space is needed to represent an instruction, use a /// `Box` to store the additional information out of line. #[derive(Clone, Debug)] +#[allow(missing_docs)] pub enum InstructionData { Nullary { opcode: Opcode, ty: Type }, Unary { @@ -226,14 +227,17 @@ pub enum InstructionData { pub struct VariableArgs(Vec); impl VariableArgs { + /// Create an empty argument list. pub fn new() -> VariableArgs { VariableArgs(Vec::new()) } + /// Add an argument to the end. pub fn push(&mut self, v: Value) { self.0.push(v) } + /// Check if the list is empty. pub fn is_empty(&self) -> bool { self.0.is_empty() } @@ -276,6 +280,7 @@ impl Default for VariableArgs { /// Payload data for `vconst`. #[derive(Clone, Debug)] pub struct UnaryImmVectorData { + /// Raw vector data. pub imm: ImmVector, } @@ -292,6 +297,7 @@ impl Display for UnaryImmVectorData { /// Payload data for ternary instructions with multiple results, such as `iadd_carry`. #[derive(Clone, Debug)] pub struct TernaryOverflowData { + /// Value arguments. pub args: [Value; 3], } @@ -305,7 +311,9 @@ impl Display for TernaryOverflowData { /// in the allowed InstructionData size. #[derive(Clone, Debug)] pub struct JumpData { + /// Jump destination EBB. pub destination: Ebb, + /// Arguments passed to destination EBB. pub varargs: VariableArgs, } @@ -323,8 +331,11 @@ impl Display for JumpData { /// in the allowed InstructionData size. #[derive(Clone, Debug)] pub struct BranchData { + /// Value argument controlling the branch. pub arg: Value, + /// Branch destination EBB. pub destination: Ebb, + /// Arguments passed to destination EBB. pub varargs: VariableArgs, } @@ -353,6 +364,8 @@ pub struct CallData { pub struct IndirectCallData { /// Callee function. pub arg: Value, + + /// Signature of the callee function. pub sig_ref: SigRef, /// Dynamically sized array containing call argument values. @@ -362,7 +375,7 @@ pub struct IndirectCallData { /// Payload of a return instruction. #[derive(Clone, Debug)] pub struct ReturnData { - // Dynamically sized array containing return values. + /// Dynamically sized array containing return values. pub varargs: VariableArgs, } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 67d5a91c29..5705fe3a55 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -87,6 +87,7 @@ impl settings::Configurable for Builder { } } +/// Methods that are specialized to a target ISA. pub trait TargetIsa { /// Get the name of this ISA. fn name(&self) -> &'static str; diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index cdc49f6973..06a9c890d0 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -16,6 +16,7 @@ struct Isa { cpumode: &'static [shared_enc_tables::Level1Entry], } +/// Get an ISA builder for creating RISC-V targets. pub fn isa_builder() -> IsaBuilder { IsaBuilder { setup: settings::builder(), diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index ae25157248..aef83bd433 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -1,14 +1,12 @@ +//! Cretonne code generation library. -// ====------------------------------------------------------------------------------------==== // -// -// Cretonne code generation library. -// -// ====------------------------------------------------------------------------------------==== // +#![deny(missing_docs)] pub use verifier::verify_function; pub use write::write_function; pub use legalizer::legalize_function; +/// Version number of the cretonne crate. pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); pub mod ir; diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 37d09d2ba8..291211b1b3 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -144,6 +144,7 @@ pub enum Error { BadValue, } +/// A result returned when changing a setting. pub type Result = result::Result; /// Implementation details for generated code. @@ -156,10 +157,15 @@ pub mod detail { /// An instruction group template. pub struct Template { + /// Name of the instruction group. pub name: &'static str, + /// List of setting descriptors. pub descriptors: &'static [Descriptor], + /// Union of all enumerators. pub enumerators: &'static [&'static str], + /// Hash table of settings. pub hash_table: &'static [u16], + /// Default values. pub defaults: &'static [u8], } @@ -227,7 +233,10 @@ pub mod detail { #[derive(Clone, Copy)] pub enum Detail { /// A boolean setting only uses one bit, numbered from LSB. - Bool { bit: u8 }, + Bool { + /// 0-7. + bit: u8, + }, /// A numerical setting uses the whole byte. Num, diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 2b544967e2..9e044a738a 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -61,7 +61,9 @@ use std::result; /// A verifier error. #[derive(Debug, PartialEq, Eq)] pub struct Error { + /// The entity causing the verifier error. pub location: AnyEntity, + /// Error message. pub message: String, } @@ -71,6 +73,7 @@ impl Display for Error { } } +/// Verifier result. pub type Result = result::Result; // Create an `Err` variant of `Result` from a location and `format!` args. @@ -90,11 +93,12 @@ macro_rules! err { }; } +/// Verify `func`. pub fn verify_function(func: &Function) -> Result<()> { Verifier::new(func).run() } -pub struct Verifier<'a> { +struct Verifier<'a> { func: &'a Function, } @@ -165,7 +169,7 @@ impl<'a> Verifier<'a> { #[cfg(test)] mod tests { - use super::*; + use super::{Verifier, Error}; use ir::Function; use ir::instructions::{InstructionData, Opcode}; use ir::types; From baf919508090381394de0e35c69a3243bb9dccda Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 27 Oct 2016 13:40:51 -0700 Subject: [PATCH 0392/3084] Add a document comparing Cretonne and LLVM. --- cranelift/docs/compare-llvm.rst | 199 ++++++++++++++++++++++++++++++++ cranelift/docs/index.rst | 1 + cranelift/docs/langref.rst | 1 + 3 files changed, 201 insertions(+) create mode 100644 cranelift/docs/compare-llvm.rst diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst new file mode 100644 index 0000000000..550c6c6f5c --- /dev/null +++ b/cranelift/docs/compare-llvm.rst @@ -0,0 +1,199 @@ +************************* +Cretonne compared to LLVM +************************* + +`LLVM `_ is a collection of compiler components implemented as +a set of C++ libraries. It can be used to build both JIT compilers and static +compilers like `Clang `_, and it is deservedly very +popular. `Chris Lattner's chapter about LLVM +`_ in the `Architecture of Open Source +Applications `_ book gives an excellent +overview of the architecture and design of LLVM. + +Cretonne and LLVM are superficially similar projects, so it is worth +highlighting some of the differences and similarities. Both projects: + +- Use an ISA-agnostic input language in order to mostly abstract away the + differences between target instruction set architectures. +- Depend extensively on SSA form. +- Have both textual and in-memory forms of their primary intermediate language. + (LLVM also has a binary bitcode format; Cretonne doesn't.) +- Can target multiple ISAs. +- Can cross-compile by default without rebuilding the code generator. + +Cretonne's scope is much smaller than that of LLVM. The classical three main +parts of a compiler are: + +1. The language-dependent front end parses and type-checks the input program. +2. Common optimizations that are independent of both the input language and the + target ISA. +3. The code generator which depends strongly on the target ISA. + +LLVM provides both common optimizations *and* a code generator. Cretonne only +provides the last part, the code generator. LLVM additionally provides +infrastructure for building assemblers and disassemblers. Cretonne does not +handle assembly at all---it only generates binary machine code. + +Intermediate representations +============================ + +LLVM uses multiple intermediate representations as it translates a program to +binary machine code: + +`LLVM IR `_ + This is the primary intermediate language which has textual, binary, and + in-memory representations. It serves two main purposes: + + - An ISA-agnostic, stable(ish) input language that front ends can generate + easily. + - Intermediate representation for common mid-level optimizations. A large + library of code analysis and transformation passes operate on LLVM IR. + +`SelectionDAG `_ + A graph-based representation of the code in a single basic block is used by + the instruction selector. It has both ISA-agnostic and ISA-specific + opcodes. These main passes are run on the SelectionDAG representation: + + - Type legalization eliminates all value types that don't have a + representation in the target ISA registers. + - Operation legalization eliminates all opcodes that can't be mapped to + target ISA instructions. + - DAG-combine cleans up redundant code after the legalization passes. + - Instruction selection translates ISA-agnostic expressions to ISA-specific + instructions. + + The SelectionDAG representation automatically eliminates common + subexpressions and dead code. + +`MachineInstr `_ + A linear representation of ISA-specific instructions that initially is in + SSA form, but it can also represent non-SSA form during and after register + allocation. Many low-level optimizations run on MI code. The most important + passes are: + + - Scheduling. + - Register allocation. + +`MC `_ + MC serves as the output abstraction layer and is the basis for LLVM's + integrated assembler. It is used for: + + - Branch relaxation. + - Emitting assembly or binary object code. + - Assemblers. + - Disassemblers. + +There is an ongoing "global instruction selection" project to replace the +SelectionDAG representation with ISA-agnostic opcodes on the MachineInstr +representation. Some target ISAs have a fast instruction selector that can +translate simple code directly to MachineInstrs, bypassing SelectionDAG when +possible. + +:doc:`Cretonne ` uses a single intermediate language to cover these +levels of abstraction. This is possible in part because of Cretonne's smaller +scope. + +- Cretonne does not provide assemblers and disassemblers, so it is not + necessary to be able to represent every weird instruction in an ISA. Only + those instructions that the code generator emits have a representation. +- Cretonne's opcodes are ISA-agnostic, but after legalization / instruction + selection, each instruction is annotated with an ISA-specific encoding which + represents a native instruction. +- SSA form is preserved throughout. After register allocation, each SSA value + is annotated with an assigned ISA register or stack slot. + +The Cretonne intermediate language is similar to LLVM IR, but at a slightly +lower level of abstraction. + +Program structure +----------------- + +In LLVM IR, the largest representable unit is the *module* which corresponds +more or less to a C translation unit. It is a collection of functions and +global variables that may contain references to external symbols too. + +In Cretonne IL, the largest representable unit is the *function*. This is so +that functions can easily be compiled in parallel without worrying about +references to shared data structures. Cretonne does not have any +inter-procedural optimizations like inlining. + +An LLVM IR function is a graph of *basic blocks*. A Cretonne IL function is a +graph of *extended basic blocks* that may contain internal branch instructions. +The main difference is that an LLVM conditional branch instruction has two +target basic blocks---a true and a false edge. A Cretonne branch instruction +only has a single target and falls through to the next instruction when its +condition is false. The Cretonne representation is closer to how machine code +works; LLVM's representation is more abstract. + +LLVM uses `phi instructions +`_ in its SSA +representation. Cretonne passes arguments to EBBs instead. The two +representations are equivalent, but the EBB arguments are better suited to +handle EBBs that main contain multiple branches to the same destination block +with different arguments. Passing arguments to an EBB looks a lot like passing +arguments to a function call, and the register allocator treats them very +similarly. Arguments are assigned to registers or stack locations. + +Value types +----------- + +:ref:`Cretonne's type system ` is mostly a subset of LLVM's type +system. It is less abstract and closer to the types that common ISA registers +can hold. + +- Integer types are limited to powers of two from :cton:type:`i8` to + :cton:type:`i64`. LLVM can represent integer types of arbitrary bit width. +- Floating point types are limited to :cton:type:`f32` and :cton:type:`f64` + which is what WebAssembly provides. It is possible that 16-bit and 128-bit + types will be added in the future/ +- Addresses are represented as integers---There are no Cretonne pointer types. + LLVM currently has rich pointer types that include the pointee type. It may + move to a simpler 'address' type in the future. Cretonne may add a single + address type too. +- SIMD vector types are limited to a power-of-two number of vector lanes up to + 256. LLVM allows an arbitrary number of SIMD lanes. +- Cretonne has no aggregrate types. LLVM has named and anonymous struct types as + well as array types. + +Cretonne has multiple boolean types, whereas LLVM simply uses `i1`. The sized +Cretonne boolean types are used to represent SIMD vector masks like ``b32x4`` +where each lane is either all 0 or all 1 bits. + +Cretonne instructions and function calls can return multiple result values. LLVM +instead models this by returning a single value of an aggregrate type. + +Instruction set +--------------- + +LLVM has a small well-defined basic instruction set and a large number of +intrinsics, some of which are ISA-specific. Cretonne has a larger instruction +set and no intrinsics. Some Cretonne instructions are ISA-specific. + +Since Cretonne instructions are used all the way until the binary machine code +is emitted, there are opcodes for every native instruction that can be +generated. There is a lot of overlap between different ISAs, so for example the +:cton:inst:`iadd_imm` instruction is used by every ISA that can add an +immediate integer to a register. A simle RISC ISA like RISC-V can be defined +with only shared instructions, while an Intel ISA needs a number of specific +instructions to model addressing modes. + +Undefined behavior +================== + +Cretonne does not generally exploit undefined behavior in its optimizations. +LLVM's mid-level optimizations do, but it should be noted that LLVM's low-level code +generator rarely needs to make use of undefined behavior either. + +LLVM provides ``nsw`` and ``nuw`` flags for its arithmetic that invoke +undefined behavior on overflow. Cretonne does not provide this functionality. +Its arithmetic instructions either produce a value or a trap. + +LLVM has an ``unreachable`` instruction which is used to indicate impossible +code paths. Cretonne only has an explicit :cton:inst:`trap` instruction. + +Cretonne does make assumptions about aliasing. For example, it assumes that it +has full control of the stack objects in a function, and that they can only be +modified by function calls if their address have escaped. It is quite likely +that Cretonne will admit more detailed aliasing annotations on load/store +instructions in the future. When these annotations are incorrect, undefined +behavior ensues. diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst index b5a23ab28b..8df4042db3 100644 --- a/cranelift/docs/index.rst +++ b/cranelift/docs/index.rst @@ -9,6 +9,7 @@ Contents: langref metaref testing + compare-llvm Indices and tables ================== diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 6fbe3610de..f8b59e560c 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -83,6 +83,7 @@ instructions which behave more like variable accesses in a typical programming language. Cretonne can perform the necessary dataflow analysis to convert stack slots to SSA form. +.. _value-types: Value types =========== From 18f32f8b5600eff91ce7db87f1672255ade2b0fd Mon Sep 17 00:00:00 2001 From: Benjamin Bouvier Date: Fri, 28 Oct 2016 16:04:43 +0200 Subject: [PATCH 0393/3084] Add build instructions in the REAME; --- README.rst | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/README.rst b/README.rst index a80a23a283..1eb43c6981 100644 --- a/README.rst +++ b/README.rst @@ -36,6 +36,26 @@ Predictable performance the quirks of the target architecture. There are no advanced optimizations that sometimes work, somtimes fail. +Building Cretonne +----------------- + +Cretonne is using the Cargo package manager format. First, ensure you have +installed `rust 1.12.0` or above. Then, change the workind directory to your +clone of cretonne and run:: + + cargo build + +This will create a *target/debug* directory where you can find the generated +binary. + +To build the optimized binary for release:: + + cargo build --release + +You can then run tests with:: + + ./test-all.sh + Building the documentation -------------------------- From 65052b845023c3673a08e3eb4b3a88129d6601cf Mon Sep 17 00:00:00 2001 From: Joel Gallant Date: Fri, 28 Oct 2016 08:16:28 -0600 Subject: [PATCH 0394/3084] Typo fix --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 1eb43c6981..3ceef14248 100644 --- a/README.rst +++ b/README.rst @@ -34,7 +34,7 @@ Scalable performance Predictable performance When optimizing, Cretonne focuses on adapting the target-independent IL to the quirks of the target architecture. There are no advanced optimizations - that sometimes work, somtimes fail. + that sometimes work, sometimes fail. Building Cretonne ----------------- From 4d729bf9868fc43a9fd9777af1054993f401ea2c Mon Sep 17 00:00:00 2001 From: Sean Gillespie Date: Thu, 27 Oct 2016 23:12:35 -0700 Subject: [PATCH 0395/3084] Use `items` instead of `iteritems` for Python 3 compatibility --- lib/cretonne/meta/cretonne/__init__.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index 6672d6d869..a3f0634b94 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -184,7 +184,7 @@ class SettingGroup(object): .format(self, SettingGroup._current)) SettingGroup._current = None if globs: - for name, obj in globs.iteritems(): + for name, obj in globs.items(): if isinstance(obj, Setting): assert obj.name is None, obj.name obj.name = name @@ -761,7 +761,7 @@ class InstructionFormat(object): all the InstructionFormat objects and set their name from the dict key. This is used to name a bunch of global variables in a module. """ - for name, obj in globs.iteritems(): + for name, obj in globs.items(): if isinstance(obj, InstructionFormat): assert obj.name is None obj.name = name From 419e6865812e3802020e5c1bfbaceddab2fc2be5 Mon Sep 17 00:00:00 2001 From: Benjamin Bouvier Date: Fri, 28 Oct 2016 18:56:52 +0200 Subject: [PATCH 0396/3084] Fix a few typos in the docs; --- cranelift/docs/compare-llvm.rst | 6 +++--- cranelift/docs/langref.rst | 4 ++-- cranelift/docs/metaref.rst | 6 +++--- cranelift/docs/testing.rst | 4 ++-- lib/cretonne/meta/cretonne/base.py | 10 +++++----- lib/cretonne/meta/cretonne/entities.py | 2 +- 6 files changed, 16 insertions(+), 16 deletions(-) diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst index 550c6c6f5c..dc19a64983 100644 --- a/cranelift/docs/compare-llvm.rst +++ b/cranelift/docs/compare-llvm.rst @@ -129,7 +129,7 @@ LLVM uses `phi instructions `_ in its SSA representation. Cretonne passes arguments to EBBs instead. The two representations are equivalent, but the EBB arguments are better suited to -handle EBBs that main contain multiple branches to the same destination block +handle EBBs that may contain multiple branches to the same destination block with different arguments. Passing arguments to an EBB looks a lot like passing arguments to a function call, and the register allocator treats them very similarly. Arguments are assigned to registers or stack locations. @@ -145,7 +145,7 @@ can hold. :cton:type:`i64`. LLVM can represent integer types of arbitrary bit width. - Floating point types are limited to :cton:type:`f32` and :cton:type:`f64` which is what WebAssembly provides. It is possible that 16-bit and 128-bit - types will be added in the future/ + types will be added in the future. - Addresses are represented as integers---There are no Cretonne pointer types. LLVM currently has rich pointer types that include the pointee type. It may move to a simpler 'address' type in the future. Cretonne may add a single @@ -173,7 +173,7 @@ Since Cretonne instructions are used all the way until the binary machine code is emitted, there are opcodes for every native instruction that can be generated. There is a lot of overlap between different ISAs, so for example the :cton:inst:`iadd_imm` instruction is used by every ISA that can add an -immediate integer to a register. A simle RISC ISA like RISC-V can be defined +immediate integer to a register. A simple RISC ISA like RISC-V can be defined with only shared instructions, while an Intel ISA needs a number of specific instructions to model addressing modes. diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index f8b59e560c..164e6590bb 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -514,7 +514,7 @@ simply represent a contiguous sequence of bytes in the stack frame. :arg SS: Stack slot declared with :inst:`stack_slot`. :arg Offset: Immediate non-negative offset. -The dedicated stack access instructions are easy ofr the compiler to reason +The dedicated stack access instructions are easy for the compiler to reason about because stack slots and offsets are fixed at compile time. For example, the alignment of these stack memory accesses can be inferred from the offsets and stack slot alignments. @@ -583,7 +583,7 @@ than the native pointer size, for example unsigned :type:`i32` offsets on a Trap if the heap access would be out of bounds. - :arg H: Heap indetifier created by :inst:`heap`. + :arg H: Heap identifier created by :inst:`heap`. :arg T x: Value to be stored. :arg iN p: Unsigned base address in heap. :arg Offset: Immediate signed offset. diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index cc6d7173d6..fa3226c51f 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -139,7 +139,7 @@ indicated with an instance of :class:`ImmediateKind`. Entity references ----------------- -Instruction operands can also refer to other entties in the same function. This +Instruction operands can also refer to other entities in the same function. This can be extended basic blocks, or entities declared in the function preamble. .. autoclass:: EntityRefKind @@ -248,7 +248,7 @@ Encodings ========= Encodings describe how Cretonne instructions are mapped to binary machine code -for the target architecture. After the lealization pass, all remaining +for the target architecture. After the legalization pass, all remaining instructions are expected to map 1-1 to native instruction encodings. Cretonne instructions that can't be encoded for the current architecture are called :term:`illegal instruction`\s. @@ -256,7 +256,7 @@ instructions that can't be encoded for the current architecture are called Some instruction set architectures have different :term:`CPU mode`\s with incompatible encodings. For example, a modern ARMv8 CPU might support three different CPU modes: *A64* where instructions are encoded in 32 bits, *A32* -where all instuctions are 32 bits, and *T32* which has a mix of 16-bit and +where all instructions are 32 bits, and *T32* which has a mix of 16-bit and 32-bit instruction encodings. These are incompatible encoding spaces, and while an :cton:inst:`iadd` instruction can be encoded in 32 bits in each of them, it's not the same 32 bits. It's a judgement call if CPU modes should be modelled as diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index 0f0527bf06..419f51ff22 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -145,10 +145,10 @@ See the :file:`lib/filecheck` `documentation `_ for details of its syntax. Comments in :file:`.cton` files are associated with the entity they follow. -This typically means and instruction or the whole function. Those tests that +This typically means an instruction or the whole function. Those tests that use filecheck will extract comments associated with each function (or its entities) and scan them for filecheck directives. The test output for each -function is then matched againts the filecheck directives for that function. +function is then matched against the filecheck directives for that function. Note that LLVM's file tests don't separate filecheck directives by their associated function. It verifies the concatenated output against all filecheck diff --git a/lib/cretonne/meta/cretonne/base.py b/lib/cretonne/meta/cretonne/base.py index 1edbce9ecd..27a8be9ac6 100644 --- a/lib/cretonne/meta/cretonne/base.py +++ b/lib/cretonne/meta/cretonne/base.py @@ -134,7 +134,7 @@ call_indirect = Instruction( Indirect function call. Call the function pointed to by `callee` with the given arguments. The - called function must match the soecified signature. + called function must match the specified signature. """, ins=(SIG, callee, args), outs=rvals) @@ -392,9 +392,9 @@ srem = Instruction( .. todo:: Integer remainder vs modulus. - Clarify whether the result has the sign of the divisor or the dividend. - Should we add a ``smod`` instruction for the case where the result has - the same sign as the divisor? + Clarify whether the result has the sign of the divisor or the dividend. + Should we add a ``smod`` instruction for the case where the result has + the same sign as the divisor? """, ins=(x, y), outs=a) @@ -900,7 +900,7 @@ fma = Instruction( 'fma', r""" Floating point fused multiply-and-add. - Computes :math:`a := xy+z` wihtout any intermediate rounding of the + Computes :math:`a := xy+z` without any intermediate rounding of the product. """, ins=(x, y, z), outs=a) diff --git a/lib/cretonne/meta/cretonne/entities.py b/lib/cretonne/meta/cretonne/entities.py index 317006f79f..afeeea7e98 100644 --- a/lib/cretonne/meta/cretonne/entities.py +++ b/lib/cretonne/meta/cretonne/entities.py @@ -1,6 +1,6 @@ """ The `cretonne.entities` module predefines all the Cretonne entity reference -operand types. Thee are corresponding definitions in the `cretonne.entities` +operand types. There are corresponding definitions in the `cretonne.entities` Rust module. """ from __future__ import absolute_import From cf9ce038e920ff31e808449a7e1bf142686ca42f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 28 Oct 2016 11:49:35 -0700 Subject: [PATCH 0397/3084] Typo --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 3ceef14248..8befcf42e2 100644 --- a/README.rst +++ b/README.rst @@ -40,7 +40,7 @@ Building Cretonne ----------------- Cretonne is using the Cargo package manager format. First, ensure you have -installed `rust 1.12.0` or above. Then, change the workind directory to your +installed `rust 1.12.0` or above. Then, change the working directory to your clone of cretonne and run:: cargo build From 679448af7294b38ad807bd17f115da4912979190 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 1 Nov 2016 11:55:07 -0700 Subject: [PATCH 0398/3084] Run unittests with Python 3 if it is available. The check.sh script always runs the Python unittests with 'python', but if 'python3' is in the path, run it with that too. Fix a Python 3 compat issue and avoid passing None to max() and min(). Use an explicit intersect() function instead to intersect intervals. --- lib/cretonne/meta/check.sh | 9 +++++ lib/cretonne/meta/cretonne/base.py | 6 +-- lib/cretonne/meta/cretonne/typevar.py | 55 +++++++++++++++++++-------- 3 files changed, 52 insertions(+), 18 deletions(-) diff --git a/lib/cretonne/meta/check.sh b/lib/cretonne/meta/check.sh index 072066ebec..e1e8b1f0b2 100755 --- a/lib/cretonne/meta/check.sh +++ b/lib/cretonne/meta/check.sh @@ -18,5 +18,14 @@ runif() { # # Install pylint with 'pip install pylint'. runif pylint --py3k --reports=no -- *.py cretonne isa + +# Then run the unit tests again with Python 3. +# We get deprecation warnings about assertRaisesRegexp which was renamed in +# Python 3, but there doesn't seem to be an easy workaround. +runif python3 -Wignore:Deprecation -m unittest discover + +# Style linting. runif flake8 . + +# Type checking. runif mypy --py2 build.py diff --git a/lib/cretonne/meta/cretonne/base.py b/lib/cretonne/meta/cretonne/base.py index 27a8be9ac6..670d85cc0c 100644 --- a/lib/cretonne/meta/cretonne/base.py +++ b/lib/cretonne/meta/cretonne/base.py @@ -392,9 +392,9 @@ srem = Instruction( .. todo:: Integer remainder vs modulus. - Clarify whether the result has the sign of the divisor or the dividend. - Should we add a ``smod`` instruction for the case where the result has - the same sign as the divisor? + Clarify whether the result has the sign of the divisor or the + dividend. Should we add a ``smod`` instruction for the case where + the result has the same sign as the divisor? """, ins=(x, y), outs=a) diff --git a/lib/cretonne/meta/cretonne/typevar.py b/lib/cretonne/meta/cretonne/typevar.py index a29548af38..f4b227fd7e 100644 --- a/lib/cretonne/meta/cretonne/typevar.py +++ b/lib/cretonne/meta/cretonne/typevar.py @@ -8,19 +8,45 @@ from __future__ import absolute_import import math from . import value +try: + from typing import Tuple # noqa + Interval = Tuple[int, int] +except ImportError: + pass MAX_LANES = 256 MAX_BITS = 64 def is_power_of_two(x): + # type: (int) -> bool return x > 0 and x & (x-1) == 0 def int_log2(x): + # type: (int) -> int return int(math.log(x, 2)) +def intersect(a, b): + # type: (Interval, Interval) -> Interval + """ + Given two `(min, max)` inclusive intervals, compute their intersection. + + Use `(None, None)` to represent the empty interval on input and output. + """ + if a[0] is None or b[0] is None: + return (None, None) + lo = max(a[0], b[0]) + assert lo is not None + hi = min(a[1], b[1]) + assert hi is not None + if lo <= hi: + return (lo, hi) + else: + return (None, None) + + class TypeSet(object): """ A set of types. @@ -69,6 +95,7 @@ class TypeSet(object): """ def __init__(self, lanes=None, ints=None, floats=None, bools=None): + # type: (Interval, Interval, Interval, Interval) -> None if lanes: if lanes is True: lanes = (1, MAX_LANES) @@ -119,6 +146,7 @@ class TypeSet(object): self.max_bool = None def typeset_key(self): + # type: () -> Tuple[int, int, int, int, int, int, int, int] """Key tuple used for hashing and equality.""" return (self.min_lanes, self.max_lanes, self.min_int, self.max_int, @@ -126,6 +154,7 @@ class TypeSet(object): self.min_bool, self.max_bool) def __hash__(self): + # type: () -> int h = hash(self.typeset_key()) assert h == getattr(self, 'prev_hash', h), "TypeSet changed!" self.prev_hash = h @@ -135,6 +164,7 @@ class TypeSet(object): return self.typeset_key() == other.typeset_key() def __repr__(self): + # type: () -> str s = 'TypeSet(lanes=({}, {})'.format(self.min_lanes, self.max_lanes) if self.min_int is not None: s += ', ints=({}, {})'.format(self.min_int, self.max_int) @@ -161,6 +191,7 @@ class TypeSet(object): field, int_log2(max_val) + 1)) def __iand__(self, other): + # type: (TypeSet) -> TypeSet """ Intersect self with other type set. @@ -181,23 +212,17 @@ class TypeSet(object): self.min_lanes = max(self.min_lanes, other.min_lanes) self.max_lanes = min(self.max_lanes, other.max_lanes) - self.min_int = max(self.min_int, other.min_int) - self.max_int = min(self.max_int, other.max_int) - if self.min_int > self.max_int: - self.min_int = None - self.max_int = None + self.min_int, self.max_int = intersect( + (self.min_int, self.max_int), + (other.min_int, other.max_int)) - self.min_float = max(self.min_float, other.min_float) - self.max_float = min(self.max_float, other.max_float) - if self.min_float > self.max_float: - self.min_float = None - self.max_float = None + self.min_float, self.max_float = intersect( + (self.min_float, self.max_float), + (other.min_float, other.max_float)) - self.min_bool = max(self.min_bool, other.min_bool) - self.max_bool = min(self.max_bool, other.max_bool) - if self.min_bool > self.max_bool: - self.min_bool = None - self.max_bool = None + self.min_bool, self.max_bool = intersect( + (self.min_bool, self.max_bool), + (other.min_bool, other.max_bool)) return self From ececaa902d016d4637e990d27550f59e4c427fbc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 1 Nov 2016 14:23:41 -0700 Subject: [PATCH 0399/3084] Add type annotations to TypeVar --- lib/cretonne/meta/cretonne/typevar.py | 109 ++++++++++++-------------- 1 file changed, 48 insertions(+), 61 deletions(-) diff --git a/lib/cretonne/meta/cretonne/typevar.py b/lib/cretonne/meta/cretonne/typevar.py index f4b227fd7e..fa0179304b 100644 --- a/lib/cretonne/meta/cretonne/typevar.py +++ b/lib/cretonne/meta/cretonne/typevar.py @@ -6,11 +6,13 @@ polymorphic by using type variables. """ from __future__ import absolute_import import math -from . import value +from . import OperandKind, value # noqa try: - from typing import Tuple # noqa + from typing import Tuple, Union # noqa Interval = Tuple[int, int] + # An Interval where `True` means 'everything' + BoolInterval = Union[bool, Interval] except ImportError: pass @@ -47,6 +49,34 @@ def intersect(a, b): return (None, None) +def decode_interval(intv, full_range, default=None): + # type: (BoolInterval, Interval, int) -> Interval + """ + Decode an interval specification which can take the following values: + + True + Use the `full_range`. + `False` or `None` + An empty interval + (lo, hi) + An explicit interval + """ + if isinstance(intv, tuple): + # mypy buig here: 'builtins.None' object is not iterable + lo, hi = intv # type: ignore + assert is_power_of_two(lo) + assert is_power_of_two(hi) + assert lo <= hi + assert lo >= full_range[0] + assert hi <= full_range[1] + return intv + + if intv: + return full_range + else: + return (default, default) + + class TypeSet(object): """ A set of types. @@ -95,55 +125,12 @@ class TypeSet(object): """ def __init__(self, lanes=None, ints=None, floats=None, bools=None): - # type: (Interval, Interval, Interval, Interval) -> None - if lanes: - if lanes is True: - lanes = (1, MAX_LANES) - self.min_lanes, self.max_lanes = lanes - assert is_power_of_two(self.min_lanes) - assert is_power_of_two(self.max_lanes) - assert self.max_lanes <= MAX_LANES - else: - self.min_lanes = 1 - self.max_lanes = 1 - assert self.min_lanes <= self.max_lanes - - if ints: - if ints is True: - ints = (8, MAX_BITS) - self.min_int, self.max_int = ints - assert is_power_of_two(self.min_int) - assert is_power_of_two(self.max_int) - assert self.max_int <= MAX_BITS - assert self.min_int <= self.max_int - else: - self.min_int = None - self.max_int = None - - if floats: - if floats is True: - floats = (32, 64) - self.min_float, self.max_float = floats - assert is_power_of_two(self.min_float) - assert self.min_float >= 32 - assert is_power_of_two(self.max_float) - assert self.max_float <= 64 - assert self.min_float <= self.max_float - else: - self.min_float = None - self.max_float = None - - if bools: - if bools is True: - bools = (1, MAX_BITS) - self.min_bool, self.max_bool = bools - assert is_power_of_two(self.min_bool) - assert is_power_of_two(self.max_bool) - assert self.max_bool <= MAX_BITS - assert self.min_bool <= self.max_bool - else: - self.min_bool = None - self.max_bool = None + # type: (BoolInterval, BoolInterval, BoolInterval, BoolInterval) -> None # noqa + self.min_lanes, self.max_lanes = decode_interval( + lanes, (1, MAX_LANES), 1) + self.min_int, self.max_int = decode_interval(ints, (8, MAX_BITS)) + self.min_float, self.max_float = decode_interval(floats, (32, 64)) + self.min_bool, self.max_bool = decode_interval(bools, (1, MAX_BITS)) def typeset_key(self): # type: () -> Tuple[int, int, int, int, int, int, int, int] @@ -253,6 +240,7 @@ class TypeVar(object): ints=False, floats=False, bools=False, scalars=True, simd=False, base=None, derived_func=None): + # type: (str, str, BoolInterval, BoolInterval, BoolInterval, bool, BoolInterval, TypeVar, str) -> None # noqa self.name = name self.__doc__ = doc self.is_derived = isinstance(base, TypeVar) @@ -264,25 +252,19 @@ class TypeVar(object): self.name = '{}({})'.format(derived_func, base.name) else: min_lanes = 1 if scalars else 2 - if simd: - if simd is True: - max_lanes = MAX_LANES - else: - min_lanes, max_lanes = simd - assert not scalars or min_lanes <= 2 - else: - max_lanes = 1 - + lanes = decode_interval(simd, (min_lanes, MAX_LANES), 1) self.type_set = TypeSet( - lanes=(min_lanes, max_lanes), + lanes=lanes, ints=ints, floats=floats, bools=bools) def __str__(self): + # type: () -> str return "`{}`".format(self.name) def lane_of(self): + # type: () -> TypeVar """ Return a derived type variable that is the scalar lane type of this type variable. @@ -293,6 +275,7 @@ class TypeVar(object): return TypeVar(None, None, base=self, derived_func='LaneOf') def as_bool(self): + # type: () -> TypeVar """ Return a derived type variable that has the same vector geometry as this type variable, but with boolean lanes. Scalar types map to `b1`. @@ -300,6 +283,7 @@ class TypeVar(object): return TypeVar(None, None, base=self, derived_func='AsBool') def half_width(self): + # type: () -> TypeVar """ Return a derived type variable that has the same number of vector lanes as this one, but the lanes are half the width. @@ -315,6 +299,7 @@ class TypeVar(object): return TypeVar(None, None, base=self, derived_func='HalfWidth') def double_width(self): + # type: () -> TypeVar """ Return a derived type variable that has the same number of vector lanes as this one, but the lanes are double the width. @@ -330,12 +315,14 @@ class TypeVar(object): return TypeVar(None, None, base=self, derived_func='DoubleWidth') def operand_kind(self): + # type: () -> OperandKind # When a `TypeVar` object is used to describe the type of an `Operand` # in an instruction definition, the kind of that operand is an SSA # value. return value def free_typevar(self): + # type: () -> TypeVar if self.is_derived: return self.base else: From a5675b6138498cea123aa676bd200929fb20832e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 2 Nov 2016 09:58:02 -0700 Subject: [PATCH 0400/3084] Add glossary entries for IL and IR. --- cranelift/docs/langref.rst | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 164e6590bb..5dcf80a893 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -5,10 +5,10 @@ Cretonne Language Reference .. default-domain:: cton .. highlight:: cton -The Cretonne intermediate language has two equivalent representations: an -*in-memory data structure* that the code generator library is using, and -a *text format* which is used for test cases and debug output. Files containing -Cretonne textual IL have the ``.cton`` filename extension. +The Cretonne intermediate language (:term:`IL`) has two equivalent +representations: an *in-memory data structure* that the code generator library +is using, and a *text format* which is used for test cases and debug output. +Files containing Cretonne textual IL have the ``.cton`` filename extension. This reference uses the text format to describe IL semantics but glosses over the finer details of the lexical and syntactic structure of the format. @@ -76,9 +76,9 @@ variable during each iteration. Finally, ``v12`` is computed as the induction variable value for the next iteration. It can be difficult to generate correct SSA form if the program being converted -into Cretonne IL contains multiple assignments to the same variables. Such -variables can be presented to Cretonne as :term:`stack slot`\s instead. Stack -slots are accessed with the :inst:`stack_store` and :inst:`stack_load` +into Cretonne :term:`IL` contains multiple assignments to the same variables. +Such variables can be presented to Cretonne as :term:`stack slot`\s instead. +Stack slots are accessed with the :inst:`stack_store` and :inst:`stack_load` instructions which behave more like variable accesses in a typical programming language. Cretonne can perform the necessary dataflow analysis to convert stack slots to SSA form. @@ -275,11 +275,11 @@ indicate the different kinds of immediate operands on an instruction. A floating point condition code. See the :inst:`fcmp` instruction for details. The two IEEE floating point immediate types :type:`ieee32` and :type:`ieee64` -are displayed as hexadecimal floating point literals in the textual IL format. -Decimal floating point literals are not allowed because some computer systems -can round differently when converting to binary. The hexadecimal floating point -format is mostly the same as the one used by C99, but extended to represent all -NaN bit patterns: +are displayed as hexadecimal floating point literals in the textual :term:`IL` +format. Decimal floating point literals are not allowed because some computer +systems can round differently when converting to binary. The hexadecimal +floating point format is mostly the same as the one used by C99, but extended +to represent all NaN bit patterns: Normal numbers Compatible with C99: ``-0x1.Tpe`` where ``T`` are the trailing @@ -859,6 +859,20 @@ Glossary .. glossary:: + intermediate language + IL + The language used to describe functions to Cretonne. This reference + describes the syntax and semantics of the Cretonne IL. The IL has two + forms: Textual and an in-memory intermediate representation + (:term:`IR`). + + intermediate representation + IR + The in-memory representation of :term:`IL`. The data structures + Cretonne uses to represent a program internally are called the + intermediate representation. Cretonne's IR can be converted to text + losslessly. + function signature A function signature describes how to call a function. It consists of: From e1dae958281117a12abbcf4a97a1c7f25a48626e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 2 Nov 2016 10:45:59 -0700 Subject: [PATCH 0401/3084] Fix inconsistent instruction name. The 'fpromote' instruction was renamed from 'fcvt_ftof', but the name argument was not changed. --- lib/cretonne/meta/cretonne/base.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/meta/cretonne/base.py b/lib/cretonne/meta/cretonne/base.py index 670d85cc0c..adbbf68d28 100644 --- a/lib/cretonne/meta/cretonne/base.py +++ b/lib/cretonne/meta/cretonne/base.py @@ -1089,7 +1089,7 @@ x = Operand('x', Float) a = Operand('a', FloatTo) fpromote = Instruction( - 'fcvt_ftof', r""" + 'fpromote', r""" Convert `x` to a larger floating point format. Each lane in `x` is converted to the destination floating point format. From c7d1f90b71711620f0b32cc758bb459295c68c35 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 2 Nov 2016 14:28:37 -0700 Subject: [PATCH 0402/3084] Canonicalize the objects in an RTL list. Any Apply objects in the input are converted to Defs with empty def lists. --- lib/cretonne/meta/cretonne/xform.py | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/meta/cretonne/xform.py b/lib/cretonne/meta/cretonne/xform.py index 1100e4721f..5149c02991 100644 --- a/lib/cretonne/meta/cretonne/xform.py +++ b/lib/cretonne/meta/cretonne/xform.py @@ -15,12 +15,25 @@ SRCCTX = 1 DSTCTX = 2 +def canonicalize_defapply(node): + # type: (DefApply) -> Def + """ + Canonicalize a `Def` or `Apply` node into a `Def`. + + An `Apply` becomes a `Def` with an empty list of defs. + """ + if isinstance(node, Apply): + return Def((), node) + else: + return node + + class Rtl(object): """ Register Transfer Language list. An RTL object contains a list of register assignments in the form of `Def` - objects and/or Apply objects for side-effecting instructions. + objects. An RTL list can represent both a source pattern to be matched, or a destination pattern to be inserted. @@ -28,10 +41,10 @@ class Rtl(object): def __init__(self, *args): # type: (*DefApply) -> None - self.rtl = args + self.rtl = tuple(map(canonicalize_defapply, args)) def __iter__(self): - # type: () -> Iterator[DefApply] + # type: () -> Iterator[Def] return iter(self.rtl) From 318e3b9b33946aa2b50f64613d94eebd348df204 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 1 Nov 2016 16:10:38 -0700 Subject: [PATCH 0403/3084] Classify Vars in patterns. There's 4 classes of variables, depending on whether they have defs in the source and destination patterns. Add more XForm verification: In a legalize XForm, all source defs must be outputs. Fix a legalize pattern bug caught by this. --- lib/cretonne/meta/check.sh | 17 +++++----- lib/cretonne/meta/cretonne/ast.py | 45 ++++++++++++++++++++++++++ lib/cretonne/meta/cretonne/legalize.py | 2 +- lib/cretonne/meta/cretonne/xform.py | 30 ++++++++++++----- lib/cretonne/meta/gen_legalizer.py | 2 +- 5 files changed, 78 insertions(+), 18 deletions(-) diff --git a/lib/cretonne/meta/check.sh b/lib/cretonne/meta/check.sh index e1e8b1f0b2..b0a6c853db 100755 --- a/lib/cretonne/meta/check.sh +++ b/lib/cretonne/meta/check.sh @@ -2,9 +2,6 @@ set -e cd $(dirname "$0") -echo "=== Python unit tests ===" -python -m unittest discover - runif() { if command -v "$1" > /dev/null; then echo "=== $1 ===" @@ -19,13 +16,17 @@ runif() { # Install pylint with 'pip install pylint'. runif pylint --py3k --reports=no -- *.py cretonne isa -# Then run the unit tests again with Python 3. -# We get deprecation warnings about assertRaisesRegexp which was renamed in -# Python 3, but there doesn't seem to be an easy workaround. -runif python3 -Wignore:Deprecation -m unittest discover - # Style linting. runif flake8 . # Type checking. runif mypy --py2 build.py + +echo "=== Python unit tests ===" +python -m unittest discover + +# Then run the unit tests again with Python 3. +# We get deprecation warnings about assertRaisesRegexp which was renamed in +# Python 3, but there doesn't seem to be an easy workaround. +runif python3 -Wignore:Deprecation -m unittest discover + diff --git a/lib/cretonne/meta/cretonne/ast.py b/lib/cretonne/meta/cretonne/ast.py index 693a0b5a2b..a43ddfadcb 100644 --- a/lib/cretonne/meta/cretonne/ast.py +++ b/lib/cretonne/meta/cretonne/ast.py @@ -73,6 +73,24 @@ class Expr(object): class Var(Expr): """ A free variable. + + When variables are used in `XForms` with source ans destination patterns, + they are classified as follows: + + Input values + Uses in the source pattern with no preceding def. These may appear as + inputs in the destination pattern too, but no new inputs can be + introduced. + Output values + Variables that are defined in both the source and destination pattern. + These values may have uses outside the source pattern, and the + destination pattern must compute the same value. + Intermediate values + Values that are defined in the source pattern, but not in the + destination pattern. These may have uses outside the source pattern, so + the defining instruction can't be deleted immediately. + Temporary values + Values that are defined only in the destination pattern. """ def __init__(self, name): @@ -82,15 +100,42 @@ class Var(Expr): # See XForm._rewrite_defs(). self.defctx = 0 + # Context bits for `defctx` indicating which pattern has defines of this + # var. + SRCCTX = 1 + DSTCTX = 2 + def __str__(self): + # type: () -> str return self.name def __repr__(self): + # type: () -> str s = self.name if self.defctx: s += ", d={:02b}".format(self.defctx) return "Var({})".format(s) + def is_input(self): + # type: () -> bool + """Is this an input value to the source pattern?""" + return self.defctx == 0 + + def is_output(self): + """Is this an output value, defined in both src and dest patterns?""" + # type: () -> bool + return self.defctx == self.SRCCTX | self.DSTCTX + + def is_intermediate(self): + """Is this an intermediate value, defined only in the src pattern?""" + # type: () -> bool + return self.defctx == self.SRCCTX + + def is_temp(self): + """Is this a temp value, defined only in the dest pattern?""" + # type: () -> bool + return self.defctx == self.DSTCTX + class Apply(Expr): """ diff --git a/lib/cretonne/meta/cretonne/legalize.py b/lib/cretonne/meta/cretonne/legalize.py index 21073d0aec..8dfa2e6b97 100644 --- a/lib/cretonne/meta/cretonne/legalize.py +++ b/lib/cretonne/meta/cretonne/legalize.py @@ -114,5 +114,5 @@ expand.legalize( Rtl( (a1, b1) << isub_bout(x, y), (a, b2) << isub_bout(a1, b_in), - c << bor(b1, b2) + b << bor(b1, b2) )) diff --git a/lib/cretonne/meta/cretonne/xform.py b/lib/cretonne/meta/cretonne/xform.py index 5149c02991..2888b9c26c 100644 --- a/lib/cretonne/meta/cretonne/xform.py +++ b/lib/cretonne/meta/cretonne/xform.py @@ -11,10 +11,6 @@ except ImportError: pass -SRCCTX = 1 -DSTCTX = 2 - - def canonicalize_defapply(node): # type: (DefApply) -> Def """ @@ -87,13 +83,13 @@ class XForm(object): # Rewrite variables in src and dst RTL lists to our own copies. # Map name -> private Var. symtab = dict() # type: Dict[str, Var] - self._rewrite_rtl(src, symtab, SRCCTX) + self._rewrite_rtl(src, symtab, Var.SRCCTX) num_src_inputs = len(self.inputs) - self._rewrite_rtl(dst, symtab, DSTCTX) + self._rewrite_rtl(dst, symtab, Var.DSTCTX) # Check for inconsistently used inputs. for i in self.inputs: - if i.defctx: + if not i.is_input(): raise AssertionError( "'{}' used as both input and def".format(i)) @@ -189,6 +185,22 @@ class XForm(object): self.inputs.append(var) yield var + def verify_legalize(self): + # type: () -> None + """ + Verify that this is a valid legalization XForm. + + - The source pattern must describe a single instruction. + - All values defined in the output pattern must be defined in the + destination pattern. + """ + assert len(self.src.rtl) == 1, "Legalize needs single instruction." + defs, expr = self.src.rtl[0].defs_expr() + for d in defs: + if not d.is_output(): + raise AssertionError( + '{} not defined in dest pattern'.format(d)) + class XFormGroup(object): """ @@ -209,4 +221,6 @@ class XFormGroup(object): :param src: Single `Def` or `Apply` to be legalized. :param dst: `Rtl` list of replacement instructions. """ - self.xforms.append(XForm(Rtl(src), dst)) + xform = XForm(Rtl(src), dst) + xform.verify_legalize() + self.xforms.append(xform) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 0fdfc921af..2a2387723f 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -14,7 +14,7 @@ from cretonne.ast import Def, Apply # noqa from cretonne.xform import XForm, XFormGroup # noqa try: - from typing import Union # noqa + from typing import Union DefApply = Union[Def, Apply] except ImportError: pass From f652dcffe44567e2f36b2bf279521e78c0d9ef7e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 2 Nov 2016 14:41:30 -0700 Subject: [PATCH 0404/3084] Set expectations. --- README.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.rst b/README.rst index 8befcf42e2..b98a4893fa 100644 --- a/README.rst +++ b/README.rst @@ -5,6 +5,8 @@ Cretonne Code Generator Cretonne is a low-level retargetable code generator. It translates a target-independent intermediate language into executable machine code. +*This is a work in progress that is not yet functional.* + .. image:: https://readthedocs.org/projects/cretonne/badge/?version=latest :target: https://cretonne.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status From 0543bb049c61b43223aa1aed51ce9c2657bf29b0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 3 Nov 2016 11:24:04 -0700 Subject: [PATCH 0405/3084] Save a reference from a Var to its src and dst defs. When a Var is used in an XForm, it can be defined in the src or dst or both patterns, and it is classified accordingly. When a Var is defined, it is also useful to be able to find the `Def` that defined it. Add src_def and dst_def reference members to Var, and initialize them in the private Var copies that XForm creates for itself. These two members also replace the defctx bitmask. --- lib/cretonne/meta/cretonne/ast.py | 63 ++++++++++++++++++++--------- lib/cretonne/meta/cretonne/xform.py | 18 ++++----- 2 files changed, 54 insertions(+), 27 deletions(-) diff --git a/lib/cretonne/meta/cretonne/ast.py b/lib/cretonne/meta/cretonne/ast.py index a43ddfadcb..1269a31668 100644 --- a/lib/cretonne/meta/cretonne/ast.py +++ b/lib/cretonne/meta/cretonne/ast.py @@ -74,7 +74,7 @@ class Var(Expr): """ A free variable. - When variables are used in `XForms` with source ans destination patterns, + When variables are used in `XForms` with source and destination patterns, they are classified as follows: Input values @@ -96,14 +96,10 @@ class Var(Expr): def __init__(self, name): # type: (str) -> None self.name = name - # Bitmask of contexts where this variable is defined. - # See XForm._rewrite_defs(). - self.defctx = 0 - - # Context bits for `defctx` indicating which pattern has defines of this - # var. - SRCCTX = 1 - DSTCTX = 2 + # The `Def` defining this variable in a source pattern. + self.src_def = None # type: Def + # The `Def` defining this variable in a destination pattern. + self.dst_def = None # type: Def def __str__(self): # type: () -> str @@ -112,29 +108,60 @@ class Var(Expr): def __repr__(self): # type: () -> str s = self.name - if self.defctx: - s += ", d={:02b}".format(self.defctx) + if self.src_def: + s += ", src" + if self.dst_def: + s += ", dst" return "Var({})".format(s) + # Context bits for `set_def` indicating which pattern has defines of this + # var. + SRCCTX = 1 + DSTCTX = 2 + + def set_def(self, context, d): + # type: (int, Def) -> None + """ + Set the `Def` that defines this variable in the given context. + + The `context` must be one of `SRCCTX` or `DSTCTX` + """ + if context == self.SRCCTX: + self.src_def = d + else: + self.dst_def = d + + def get_def(self, context): + # type: (int) -> Def + """ + Get the def of this variable in context. + + The `context` must be one of `SRCCTX` or `DSTCTX` + """ + if context == self.SRCCTX: + return self.src_def + else: + return self.dst_def + def is_input(self): # type: () -> bool - """Is this an input value to the source pattern?""" - return self.defctx == 0 + """Is this an input value to the src pattern?""" + return not self.src_def and not self.dst_def def is_output(self): - """Is this an output value, defined in both src and dest patterns?""" + """Is this an output value, defined in both src and dst patterns?""" # type: () -> bool - return self.defctx == self.SRCCTX | self.DSTCTX + return self.src_def and self.dst_def def is_intermediate(self): """Is this an intermediate value, defined only in the src pattern?""" # type: () -> bool - return self.defctx == self.SRCCTX + return self.src_def and not self.dst_def def is_temp(self): - """Is this a temp value, defined only in the dest pattern?""" + """Is this a temp value, defined only in the dst pattern?""" # type: () -> bool - return self.defctx == self.DSTCTX + return not self.src_def and self.dst_def class Apply(Expr): diff --git a/lib/cretonne/meta/cretonne/xform.py b/lib/cretonne/meta/cretonne/xform.py index 2888b9c26c..4c2f3c9daf 100644 --- a/lib/cretonne/meta/cretonne/xform.py +++ b/lib/cretonne/meta/cretonne/xform.py @@ -63,7 +63,7 @@ class XForm(object): ... Rtl(c << iconst(v), ... a << iadd(x, c)), ... Rtl(a << iadd_imm(x, v))) - XForm(inputs=[Var(v), Var(x)], defs=[Var(c, d=01), Var(a, d=11)], + XForm(inputs=[Var(v), Var(x)], defs=[Var(c, src), Var(a, src, dst)], c << iconst(v) a << iadd(x, c) => @@ -112,7 +112,7 @@ class XForm(object): for line in rtl.rtl: if isinstance(line, Def): line.defs = tuple( - self._rewrite_defs(line.defs, symtab, context)) + self._rewrite_defs(line, symtab, context)) expr = line.expr else: expr = line @@ -132,23 +132,23 @@ class XForm(object): expr.args = tuple( self._rewrite_uses(expr, stack, symtab, context)) - def _rewrite_defs(self, defs, symtab, context): - # type: (Sequence[Var], Dict[str, Var], int) -> Iterable[Var] + def _rewrite_defs(self, line, symtab, context): + # type: (Def, Dict[str, Var], int) -> Iterable[Var] """ Given a tuple of symbols defined in a Def, rewrite them to local symbols. Yield the new locals. """ - for sym in defs: + for sym in line.defs: name = str(sym) if name in symtab: var = symtab[name] - if var.defctx & context: + if var.get_def(context): raise AssertionError("'{}' multiply defined".format(name)) else: var = Var(name) symtab[name] = var self.defs.append(var) - var.defctx |= context + var.set_def(context, line) yield var def _rewrite_uses(self, expr, stack, symtab, context): @@ -173,8 +173,8 @@ class XForm(object): name = str(arg) if name in symtab: var = symtab[name] - # The variable must be used consistenty as a def or input. - if var.defctx and (var.defctx & context) == 0: + # The variable must be used consistently as a def or input. + if not var.is_input() and not var.get_def(context): raise AssertionError( "'{}' used as both input and def" .format(name)) From 814d1728aafea39c6f6a1ca76eee88c73fae1800 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 3 Nov 2016 17:55:41 -0700 Subject: [PATCH 0406/3084] Add a Value::unwrap_direct() method. When it is known that a value is the first result of an instruction, it is safe to unwrap the instruction reference. --- lib/cretonne/src/ir/entities.rs | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 384e4d3ae9..df7d81988f 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -164,6 +164,19 @@ impl Value { Table(index) } } + + /// Assuming that this is a direct value, get the referenced instruction. + /// + /// # Panics + /// + /// If this is not a value created with `new_direct()`. + pub fn unwrap_direct(&self) -> Inst { + if let ExpandedValue::Direct(inst) = self.expand() { + inst + } else { + panic!("{} is not a direct value", self) + } + } } /// Display a `Value` reference as "v7" or "v2x". From 9c02fe3553d094899a0b16cff0efa510da982a10 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 2 Nov 2016 12:56:34 -0700 Subject: [PATCH 0407/3084] Legalization pattern emission WIP. Begin emitting legalization patterns in the form of two functions, 'expand' and 'narrow' that are included in legalizer.rs. The generated code compiles, but it is not fully working yet. We need to deal with the special cases of instructions producing multiple results. --- lib/cretonne/meta/cretonne/__init__.py | 12 ++++ lib/cretonne/meta/cretonne/ast.py | 28 +++----- lib/cretonne/meta/cretonne/legalize.py | 4 +- lib/cretonne/meta/cretonne/xform.py | 3 +- lib/cretonne/meta/gen_instr.py | 8 +-- lib/cretonne/meta/gen_legalizer.py | 90 ++++++++++++++++++++++---- lib/cretonne/src/legalizer.rs | 9 ++- 7 files changed, 113 insertions(+), 41 deletions(-) diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index a3f0634b94..8a46b2d08f 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -834,6 +834,18 @@ class Instruction(object): suffix = ', '.join(o.name for o in self.ins) return '{}{} {}'.format(prefix, self.name, suffix) + def snake_name(self): + # type: () -> str + """ + Get the snake_case name of this instruction. + + Keywords in Rust and Python are altered by appending a '_' + """ + if self.name == 'return': + return 'return_' + else: + return self.name + def blurb(self): """Get the first line of the doc comment""" for line in self.__doc__.split('\n'): diff --git a/lib/cretonne/meta/cretonne/ast.py b/lib/cretonne/meta/cretonne/ast.py index 1269a31668..ac21809317 100644 --- a/lib/cretonne/meta/cretonne/ast.py +++ b/lib/cretonne/meta/cretonne/ast.py @@ -53,16 +53,6 @@ class Def(object): return "({}) << {!s}".format( ', '.join(map(str, self.defs)), self.expr) - def root_inst(self): - # type: () -> Instruction - """Get the instruction at the root of this tree.""" - return self.expr.root_inst() - - def defs_expr(self): - # type: () -> Tuple[Tuple[Var, ...], Apply] - """Split into a defs tuple and an Apply expr.""" - return (self.defs, self.expr) - class Expr(object): """ @@ -215,12 +205,12 @@ class Apply(Expr): args = ', '.join(map(str, self.args)) return '{}({})'.format(self.instname(), args) - def root_inst(self): - # type: () -> Instruction - """Get the instruction at the root of this tree.""" - return self.inst - - def defs_expr(self): - # type: () -> Tuple[Tuple[Var, ...], Apply] - """Split into a defs tuple and an Apply expr.""" - return ((), self) + def rust_builder(self): + # type: () -> str + """ + Return a Rust Builder method call for instantiating this instruction + application. + """ + args = ', '.join(map(str, self.args)) + method = self.inst.snake_name() + return '{}({})'.format(method, args) diff --git a/lib/cretonne/meta/cretonne/legalize.py b/lib/cretonne/meta/cretonne/legalize.py index 8dfa2e6b97..102df5f6a5 100644 --- a/lib/cretonne/meta/cretonne/legalize.py +++ b/lib/cretonne/meta/cretonne/legalize.py @@ -77,14 +77,14 @@ expand.legalize( (a, c) << iadd_cout(x, y), Rtl( a << iadd(x, y), - c << icmp('ult', a, x) + c << icmp('IntCC::UnsignedLessThan', a, x) )) expand.legalize( (a, b) << isub_bout(x, y), Rtl( a << isub(x, y), - b << icmp('ugt', a, x) + b << icmp('IntCC::UnsignedGreaterThan', a, x) )) expand.legalize( diff --git a/lib/cretonne/meta/cretonne/xform.py b/lib/cretonne/meta/cretonne/xform.py index 4c2f3c9daf..6b9b37f513 100644 --- a/lib/cretonne/meta/cretonne/xform.py +++ b/lib/cretonne/meta/cretonne/xform.py @@ -195,8 +195,7 @@ class XForm(object): destination pattern. """ assert len(self.src.rtl) == 1, "Legalize needs single instruction." - defs, expr = self.src.rtl[0].defs_expr() - for d in defs: + for d in self.src.rtl[0].defs: if not d.is_output(): raise AssertionError( '{} not defined in dest pattern'.format(d)) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index bb14d36375..8b4555d9b2 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -446,16 +446,12 @@ def gen_inst_builder(inst, fmt): rvals = ', '.join(len(inst.value_results) * ['Value']) rtype = '({})'.format(rvals) - method = inst.name - if method == 'return': - # Avoid Rust keywords by appending '_'. - method += '_' - if len(tmpl_types) > 0: tmpl = '<{}>'.format(', '.join(tmpl_types)) else: tmpl = '' - proto = '{}{}({}) -> {}'.format(method, tmpl, ', '.join(args), rtype) + proto = '{}{}({}) -> {}'.format( + inst.snake_name(), tmpl, ', '.join(args), rtype) fmt.doc_comment('`{}`\n\n{}'.format(inst, inst.blurb())) fmt.line('#[allow(non_snake_case)]') diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 2a2387723f..d6b0f5eb2c 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -10,30 +10,29 @@ the input instruction. from __future__ import absolute_import from srcgen import Formatter import cretonne.legalize as legalize -from cretonne.ast import Def, Apply # noqa +from cretonne.ast import Def # noqa from cretonne.xform import XForm, XFormGroup # noqa try: - from typing import Union - DefApply = Union[Def, Apply] + from typing import Sequence # noqa except ImportError: pass def unwrap_inst(iref, node, fmt): - # type: (str, DefApply, Formatter) -> None + # type: (str, Def, Formatter) -> None """ - Given a `Def` or `Apply` node, emit code that extracts all the instruction - fields from `dfg[iref]`. + Given a `Def` node, emit code that extracts all the instruction fields from + `dfg[iref]`. Create local variables named after the `Var` instances in `node`. :param iref: Name of the `Inst` reference to unwrap. - :param node: `Def` or `Apply` node providing variable names. + :param node: `Def` node providing variable names. """ fmt.comment('Unwrap {}'.format(node)) - defs, expr = node.defs_expr() + expr = node.expr iform = expr.inst.format nvops = len(iform.value_operands) @@ -71,7 +70,64 @@ def unwrap_inst(iref, node, fmt): prefix, iform.value_operands.index(i))) fmt.line('({})'.format(', '.join(outs))) fmt.outdented_line('} else {') - fmt.line('unimplemented!("bad instruction format")') + fmt.line('unreachable!("bad instruction format")') + + # If the node has multiple results, detach the values. + # Place the secondary values in 'src_{}' locals. + if len(node.defs) > 1: + if node.defs == node.defs[0].dst_def.defs: + # Special case: The instruction replacing node defines the exact + # same values. + fmt.comment( + 'Multiple results handled by {}.' + .format(node.defs[0].dst_def)) + else: + fmt.comment('Detaching secondary results.') + # Boring case: Detach the secondary values, capture them in locals. + for d in node.defs[1:]: + fmt.line('let src_{};'.format(d)) + with fmt.indented('{', '}'): + fmt.line('let mut vals = dfg.detach_secondary_results(inst);') + for d in node.defs[1:]: + fmt.line('src_{} = vals.next().unwrap();'.format(d)) + fmt.line('assert_eq!(vals.next(), None);') + + +def wrap_tup(seq): + # type: (Sequence[object]) -> str + tup = tuple(map(str, seq)) + if len(tup) == 1: + return tup[0] + else: + return '({})'.format(', '.join(tup)) + + +def emit_dst_inst(node, fmt): + # type: (Def, Formatter) -> None + exact_replace = False + if len(node.defs) == 0: + # This node doesn't define any values, so just insert the new + # instruction. + builder = 'dfg.ins(pos)' + else: + src_def0 = node.defs[0].src_def + if src_def0 and node.defs[0] == src_def0.defs[0]: + # The primary result is replacing the primary result of the src + # pattern. + # Replace the whole instruction. + builder = 'let {} = dfg.replace(inst)'.format(wrap_tup(node.defs)) + # Secondary values weren't replaced if this is an exact replacement + # for all the src results. + exact_replace = (node.defs == src_def0.defs) + else: + # Insert a new instruction since its primary def doesn't match the + # src. + builder = 'let {} = dfg.ins(pos)'.format(wrap_tup(node.defs)) + + fmt.line('{}.{};'.format(builder, node.expr.rust_builder())) + + if exact_replace: + fmt.comment('exactreplacement') def gen_xform(xform, fmt): @@ -83,9 +139,20 @@ def gen_xform(xform, fmt): `inst: Inst` is the variable to be replaced. It is pointed to by `pos: Cursor`. `dfg: DataFlowGraph` is available and mutable. + + Produce an `Option` result at the end. """ + # Unwrap the source instruction, create local variables for the input + # variables. unwrap_inst('inst', xform.src.rtl[0], fmt) + # Emit the destination pattern. + for dst in xform.dst.rtl: + emit_dst_inst(dst, fmt) + + # TODO: Return the first replacement instruction. + fmt.line('None') + def gen_xform_group(xgrp, fmt): # type: (XFormGroup, Formatter) -> None @@ -95,10 +162,11 @@ def gen_xform_group(xgrp, fmt): Return the first instruction in the expansion, and leave `pos` pointing at the last instruction in the expansion. """) + fmt.line('#[allow(unused_variables,unused_assignments)]') with fmt.indented( 'fn ' + xgrp.name + '(pos: &mut Cursor, dfg: &mut DataFlowGraph) -> ' + - 'Option {{', + 'Option {', '}'): # Gen the instruction to be legalized. The cursor we're passed must be # pointing at an instruction. @@ -106,7 +174,7 @@ def gen_xform_group(xgrp, fmt): with fmt.indented('match dfg[inst].opcode() {', '}'): for xform in xgrp.xforms: - inst = xform.src.rtl[0].root_inst() + inst = xform.src.rtl[0].expr.inst with fmt.indented( 'Opcode::{} => {{'.format(inst.camel_name), '}'): gen_xform(xform, fmt) diff --git a/lib/cretonne/src/legalizer.rs b/lib/cretonne/src/legalizer.rs index 3358f77bb0..89ba8cd569 100644 --- a/lib/cretonne/src/legalizer.rs +++ b/lib/cretonne/src/legalizer.rs @@ -13,7 +13,8 @@ //! The legalizer does not deal with register allocation constraints. These constraints are derived //! from the encoding recipes, and solved later by the register allocator. -use ir::Function; +use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, Inst, InstBuilder}; +use ir::condcodes::IntCC; use isa::TargetIsa; /// Legalize `func` for `isa`. @@ -52,3 +53,9 @@ pub fn legalize_function(func: &mut Function, isa: &TargetIsa) { } } } + +// Include legalization patterns that were generated by gen_legalizer.py from the XForms in +// meta/cretonne/legalize.py. +// +// Concretely, this defines private functions `narrow()`, and `expand()`. +include!(concat!(env!("OUT_DIR"), "/legalizer.rs")); From e59b47c41a0bf932bca3e4add7346dbd72d3cfc2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 3 Nov 2016 18:59:32 -0700 Subject: [PATCH 0408/3084] Return a Result from the TargetIsa::encode() method. When an instruction can't be encoded, provide a viable legalization action in the form of a Legalize enum. --- lib/cretonne/src/isa/enc_tables.rs | 21 +++++++----- lib/cretonne/src/isa/mod.rs | 15 ++++++++- lib/cretonne/src/isa/riscv/mod.rs | 17 +++++----- lib/cretonne/src/legalizer.rs | 51 ++++++++++++++++-------------- 4 files changed, 64 insertions(+), 40 deletions(-) diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index 185e162f7f..d0a371dbc4 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -3,7 +3,7 @@ //! This module contains types and functions for working with the encoding tables generated by //! `lib/cretonne/meta/gen_encoding.py`. use ir::{Type, Opcode}; -use isa::Encoding; +use isa::{Encoding, Legalize}; use constant_hash::{Table, probe}; /// Level 1 hash table entry. @@ -83,16 +83,21 @@ pub fn lookup_enclist(ctrl_typevar: Type, opcode: Opcode, level1_table: &[Level1Entry], level2_table: &[Level2Entry]) - -> Option + -> Result where OffT1: Into + Copy, OffT2: Into + Copy { - probe(level1_table, ctrl_typevar, ctrl_typevar.index()).and_then(|l1idx| { - let l1ent = &level1_table[l1idx]; - let l2off = l1ent.offset.into() as usize; - let l2tab = &level2_table[l2off..l2off + (1 << l1ent.log2len)]; - probe(l2tab, opcode, opcode as usize).map(|l2idx| l2tab[l2idx].offset.into() as usize) - }) + // TODO: The choice of legalization actions here is naive. This needs to be configurable. + probe(level1_table, ctrl_typevar, ctrl_typevar.index()) + .ok_or(Legalize::Narrow) + .and_then(|l1idx| { + let l1ent = &level1_table[l1idx]; + let l2off = l1ent.offset.into() as usize; + let l2tab = &level2_table[l2off..l2off + (1 << l1ent.log2len)]; + probe(l2tab, opcode, opcode as usize) + .map(|l2idx| l2tab[l2idx].offset.into() as usize) + .ok_or(Legalize::Expand) + }) } /// Encoding list entry. diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 5705fe3a55..aa7b9a76d8 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -87,6 +87,19 @@ impl settings::Configurable for Builder { } } +/// After determining that an instruction doesn't have an encoding, how should we proceed to +/// legalize it? +/// +/// These actions correspond to the transformation groups defined in `meta/cretonne/legalize.py`. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum Legalize { + /// Legalize in terms of narrower types. + Narrow, + + /// Expanding in terms of other instructions using the same types. + Expand, +} + /// Methods that are specialized to a target ISA. pub trait TargetIsa { /// Get the name of this ISA. @@ -101,7 +114,7 @@ pub trait TargetIsa { /// Otherwise, return `None`. /// /// This is also the main entry point for determining if an instruction is legal. - fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Option; + fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Result; /// Get a static array of names associated with encoding recipes in this ISA. Encoding recipes /// are numbered starting from 0, corresponding to indexes into th name array. diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 06a9c890d0..63ab5ccf89 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -6,7 +6,7 @@ mod enc_tables; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, Encoding}; +use isa::{TargetIsa, Encoding, Legalize}; use ir::{InstructionData, DataFlowGraph}; #[allow(dead_code)] @@ -48,7 +48,7 @@ impl TargetIsa for Isa { &self.shared_flags } - fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Option { + fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Result { lookup_enclist(inst.first_type(), inst.opcode(), self.cpumode, @@ -58,6 +58,7 @@ impl TargetIsa for Isa { &enc_tables::ENCLISTS[..], |instp| enc_tables::check_instp(inst, instp), |isap| self.isa_flags.numbered_predicate(isap as usize)) + .ok_or(Legalize::Expand) }) } @@ -109,7 +110,7 @@ mod tests { }; // Immediate is out of range for ADDI. - assert_eq!(isa.encode(&dfg, &inst64_large), None); + assert_eq!(isa.encode(&dfg, &inst64_large), Err(isa::Legalize::Expand)); // Create an iadd_imm.i32 which is encodable in RV64. let inst32 = InstructionData::BinaryImm { @@ -144,8 +145,8 @@ mod tests { imm: immediates::Imm64::new(-10), }; - // ADDI is I/0b00100 - assert_eq!(isa.encode(&dfg, &inst64), None); + // In 32-bit mode, an i64 bit add should be narrowed. + assert_eq!(isa.encode(&dfg, &inst64), Err(isa::Legalize::Narrow)); // Try to encode iadd_imm.i64 vx1, -10000. let inst64_large = InstructionData::BinaryImm { @@ -155,8 +156,8 @@ mod tests { imm: immediates::Imm64::new(-10000), }; - // Immediate is out of range for ADDI. - assert_eq!(isa.encode(&dfg, &inst64_large), None); + // In 32-bit mode, an i64 bit add should be narrowed. + assert_eq!(isa.encode(&dfg, &inst64_large), Err(isa::Legalize::Narrow)); // Create an iadd_imm.i32 which is encodable in RV32. let inst32 = InstructionData::BinaryImm { @@ -176,7 +177,7 @@ mod tests { args: [arg32, arg32], }; - assert_eq!(isa.encode(&dfg, &mul32), None); + assert_eq!(isa.encode(&dfg, &mul32), Err(isa::Legalize::Expand)); } #[test] diff --git a/lib/cretonne/src/legalizer.rs b/lib/cretonne/src/legalizer.rs index 89ba8cd569..2a84ee520c 100644 --- a/lib/cretonne/src/legalizer.rs +++ b/lib/cretonne/src/legalizer.rs @@ -15,7 +15,7 @@ use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, Inst, InstBuilder}; use ir::condcodes::IntCC; -use isa::TargetIsa; +use isa::{TargetIsa, Legalize}; /// Legalize `func` for `isa`. /// @@ -25,30 +25,35 @@ use isa::TargetIsa; pub fn legalize_function(func: &mut Function, isa: &TargetIsa) { // TODO: This is very simplified and incomplete. func.encodings.resize(func.dfg.num_insts()); - for ebb in func.layout.ebbs() { - for inst in func.layout.ebb_insts(ebb) { + let mut pos = Cursor::new(&mut func.layout); + while let Some(_ebb) = pos.next_ebb() { + while let Some(inst) = pos.next_inst() { match isa.encode(&func.dfg, &func.dfg[inst]) { - Some(encoding) => func.encodings[inst] = encoding, - None => { - // TODO: We should transform the instruction into legal equivalents. - // Possible strategies are: - // 1. Expand instruction into sequence of legal instructions. Possibly - // iteratively. - // 2. Split the controlling type variable into high and low parts. This applies - // both to SIMD vector types which can be halved and to integer types such - // as `i64` used on a 32-bit ISA. - // 3. Promote the controlling type variable to a larger type. This typically - // means expressing `i8` and `i16` arithmetic in terms if `i32` operations - // on RISC targets. (It may or may not be beneficial to promote small vector - // types versus splitting them.) - // 4. Convert to library calls. For example, floating point operations on an - // ISA with no IEEE 754 support. - // - // The iteration scheme used here is not going to cut it. Transforming - // instructions involves changing `function.layout` which is impossiblr while - // it is referenced by the two iterators. We need a layout cursor that can - // maintain a position *and* permit inserting and replacing instructions. + Ok(encoding) => func.encodings[inst] = encoding, + Err(Legalize::Expand) => { + expand(&mut pos, &mut func.dfg); } + Err(Legalize::Narrow) => { + narrow(&mut pos, &mut func.dfg); + } + // TODO: We should transform the instruction into legal equivalents. + // Possible strategies are: + // 1. Expand instruction into sequence of legal instructions. Possibly + // iteratively. + // 2. Split the controlling type variable into high and low parts. This applies + // both to SIMD vector types which can be halved and to integer types such + // as `i64` used on a 32-bit ISA. + // 3. Promote the controlling type variable to a larger type. This typically + // means expressing `i8` and `i16` arithmetic in terms if `i32` operations + // on RISC targets. (It may or may not be beneficial to promote small vector + // types versus splitting them.) + // 4. Convert to library calls. For example, floating point operations on an + // ISA with no IEEE 754 support. + // + // The iteration scheme used here is not going to cut it. Transforming + // instructions involves changing `function.layout` which is impossiblr while + // it is referenced by the two iterators. We need a layout cursor that can + // maintain a position *and* permit inserting and replacing instructions. } } } From 1a9abdd158462c263718de239e43bbed466349ca Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Nov 2016 08:02:37 -0700 Subject: [PATCH 0409/3084] Add narrowing legalization patterns for bitwise ops. RISC-V 32-bit tests for band.i64, bor.i64, bxor.i64. --- .../filetests/isa/riscv/legalize-i64.cton | 42 +++++++++++++++++++ lib/cretonne/meta/cretonne/legalize.py | 13 +++++- 2 files changed, 54 insertions(+), 1 deletion(-) create mode 100644 cranelift/filetests/isa/riscv/legalize-i64.cton diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton new file mode 100644 index 0000000000..2aa4be9e58 --- /dev/null +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -0,0 +1,42 @@ +; Test the legalization of i64 arithmetic instructions. +test legalizer +isa riscv supports_m=1 + +function bitwise_and(i64, i64) -> i64 { +ebb0(v1: i64, v2: i64): + v3 = band v1, v2 + return v3 +} +; regex: V=v\d+ +; regex: VX=vx\d+ +; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 +; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 +; check: $(v3l=$V) = band $v1l, $v2l +; check: $(v3h=$V) = band $v1h, $v2h +; check: $v3 = iconcat_lohi $v3l, $v3h + +function bitwise_or(i64, i64) -> i64 { +ebb0(v1: i64, v2: i64): + v3 = bor v1, v2 + return v3 +} +; regex: V=v\d+ +; regex: VX=vx\d+ +; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 +; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 +; check: $(v3l=$V) = bor $v1l, $v2l +; check: $(v3h=$V) = bor $v1h, $v2h +; check: $v3 = iconcat_lohi $v3l, $v3h + +function bitwise_xor(i64, i64) -> i64 { +ebb0(v1: i64, v2: i64): + v3 = bxor v1, v2 + return v3 +} +; regex: V=v\d+ +; regex: VX=vx\d+ +; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 +; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 +; check: $(v3l=$V) = bxor $v1l, $v2l +; check: $(v3h=$V) = bxor $v1h, $v2h +; check: $v3 = iconcat_lohi $v3l, $v3h diff --git a/lib/cretonne/meta/cretonne/legalize.py b/lib/cretonne/meta/cretonne/legalize.py index 102df5f6a5..189685dce1 100644 --- a/lib/cretonne/meta/cretonne/legalize.py +++ b/lib/cretonne/meta/cretonne/legalize.py @@ -9,7 +9,7 @@ instructions that are legal. from __future__ import absolute_import from .base import iadd, iadd_cout, iadd_cin, iadd_carry from .base import isub, isub_bin, isub_bout, isub_borrow -from .base import bor, isplit_lohi, iconcat_lohi +from .base import band, bor, bxor, isplit_lohi, iconcat_lohi from .base import icmp from .ast import Var from .xform import Rtl, XFormGroup @@ -71,6 +71,17 @@ narrow.legalize( a << iconcat_lohi(al, ah) )) +for bitop in [band, bor, bxor]: + narrow.legalize( + a << bitop(x, y), + Rtl( + (xl, xh) << isplit_lohi(x), + (yl, yh) << isplit_lohi(y), + al << bitop(xl, yl), + ah << bitop(xh, yh), + a << iconcat_lohi(al, ah) + )) + # Expand integer operations with carry for RISC architectures that don't have # the flags. expand.legalize( From 1c57f436432dc79706be64880e89f2a111a81237 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Nov 2016 08:44:01 -0700 Subject: [PATCH 0410/3084] Add Cursor::set_position. Make it possible to move a cursor to a new position. In the current implementation of Layout and Cursor, this is a trivial operation, but if we switch to a B-tree based function layout, this involves navigating the tree. --- lib/cretonne/src/ir/layout.rs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index fdd2305747..bdcf4cdd52 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -381,6 +381,11 @@ impl<'f> Cursor<'f> { self.pos } + /// Move the cursor to a new position. + pub fn set_position(&mut self, pos: CursorPosition) { + self.pos = pos; + } + /// Get the EBB corresponding to the current position. pub fn current_ebb(&self) -> Option { use self::CursorPosition::*; From a2b7769a518e10d19320f02b75fc282c3adcb2a2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Nov 2016 09:44:11 -0700 Subject: [PATCH 0411/3084] Revisit expanded instructions for legalization. When an illegal instruction is replaced with other instructions, back up and revisit the expanded instructions. The new instructions need to have encodings assigned too. This also allows for expansions to contain illegal instructions that need to be legalized themselves. --- .../filetests/isa/riscv/legalize-i64.cton | 18 ++++-- lib/cretonne/meta/gen_legalizer.py | 11 +--- lib/cretonne/src/legalizer.rs | 60 +++++++++++-------- 3 files changed, 50 insertions(+), 39 deletions(-) diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton index 2aa4be9e58..916986fe68 100644 --- a/cranelift/filetests/isa/riscv/legalize-i64.cton +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -11,8 +11,10 @@ ebb0(v1: i64, v2: i64): ; regex: VX=vx\d+ ; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 ; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 -; check: $(v3l=$V) = band $v1l, $v2l -; check: $(v3h=$V) = band $v1h, $v2h +; check: [R#ec +; sameln: $(v3l=$V) = band $v1l, $v2l +; check: [R#ec +; sameln: $(v3h=$V) = band $v1h, $v2h ; check: $v3 = iconcat_lohi $v3l, $v3h function bitwise_or(i64, i64) -> i64 { @@ -24,8 +26,10 @@ ebb0(v1: i64, v2: i64): ; regex: VX=vx\d+ ; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 ; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 -; check: $(v3l=$V) = bor $v1l, $v2l -; check: $(v3h=$V) = bor $v1h, $v2h +; check: [R#cc +; sameln: $(v3l=$V) = bor $v1l, $v2l +; check: [R#cc +; sameln: $(v3h=$V) = bor $v1h, $v2h ; check: $v3 = iconcat_lohi $v3l, $v3h function bitwise_xor(i64, i64) -> i64 { @@ -37,6 +41,8 @@ ebb0(v1: i64, v2: i64): ; regex: VX=vx\d+ ; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 ; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 -; check: $(v3l=$V) = bxor $v1l, $v2l -; check: $(v3h=$V) = bxor $v1h, $v2h +; check: [R#8c +; sameln: $(v3l=$V) = bxor $v1l, $v2l +; check: [R#8c +; sameln: $(v3h=$V) = bxor $v1h, $v2h ; check: $v3 = iconcat_lohi $v3l, $v3h diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index d6b0f5eb2c..3efb120124 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -139,8 +139,6 @@ def gen_xform(xform, fmt): `inst: Inst` is the variable to be replaced. It is pointed to by `pos: Cursor`. `dfg: DataFlowGraph` is available and mutable. - - Produce an `Option` result at the end. """ # Unwrap the source instruction, create local variables for the input # variables. @@ -150,9 +148,6 @@ def gen_xform(xform, fmt): for dst in xform.dst.rtl: emit_dst_inst(dst, fmt) - # TODO: Return the first replacement instruction. - fmt.line('None') - def gen_xform_group(xgrp, fmt): # type: (XFormGroup, Formatter) -> None @@ -165,8 +160,7 @@ def gen_xform_group(xgrp, fmt): fmt.line('#[allow(unused_variables,unused_assignments)]') with fmt.indented( 'fn ' + xgrp.name + - '(pos: &mut Cursor, dfg: &mut DataFlowGraph) -> ' + - 'Option {', + '(pos: &mut Cursor, dfg: &mut DataFlowGraph) -> bool {', '}'): # Gen the instruction to be legalized. The cursor we're passed must be # pointing at an instruction. @@ -179,7 +173,8 @@ def gen_xform_group(xgrp, fmt): 'Opcode::{} => {{'.format(inst.camel_name), '}'): gen_xform(xform, fmt) # We'll assume there are uncovered opcodes. - fmt.line('_ => None,') + fmt.line('_ => return false,') + fmt.line('true') def generate(isas, out_dir): diff --git a/lib/cretonne/src/legalizer.rs b/lib/cretonne/src/legalizer.rs index 2a84ee520c..a83485b5e2 100644 --- a/lib/cretonne/src/legalizer.rs +++ b/lib/cretonne/src/legalizer.rs @@ -13,7 +13,7 @@ //! The legalizer does not deal with register allocation constraints. These constraints are derived //! from the encoding recipes, and solved later by the register allocator. -use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, Inst, InstBuilder}; +use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, InstBuilder}; use ir::condcodes::IntCC; use isa::{TargetIsa, Legalize}; @@ -27,34 +27,44 @@ pub fn legalize_function(func: &mut Function, isa: &TargetIsa) { func.encodings.resize(func.dfg.num_insts()); let mut pos = Cursor::new(&mut func.layout); while let Some(_ebb) = pos.next_ebb() { + // Keep track of the cursor position before the instruction being processed, so we can + // double back when replacing instructions. + let mut prev_pos = pos.position(); + while let Some(inst) = pos.next_inst() { match isa.encode(&func.dfg, &func.dfg[inst]) { - Ok(encoding) => func.encodings[inst] = encoding, - Err(Legalize::Expand) => { - expand(&mut pos, &mut func.dfg); + Ok(encoding) => *func.encodings.ensure(inst) = encoding, + Err(action) => { + // We should transform the instruction into legal equivalents. + // Possible strategies are: + // 1. Legalize::Expand: Expand instruction into sequence of legal instructions. + // Possibly iteratively. () + // 2. Legalize::Narrow: Split the controlling type variable into high and low + // parts. This applies both to SIMD vector types which can be halved and to + // integer types such as `i64` used on a 32-bit ISA. (). + // 3. TODO: Promote the controlling type variable to a larger type. This + // typically means expressing `i8` and `i16` arithmetic in terms if `i32` + // operations on RISC targets. (It may or may not be beneficial to promote + // small vector types versus splitting them.) + // 4. TODO: Convert to library calls. For example, floating point operations on + // an ISA with no IEEE 754 support. + let changed = match action { + Legalize::Expand => expand(&mut pos, &mut func.dfg), + Legalize::Narrow => narrow(&mut pos, &mut func.dfg), + }; + // If the current instruction was replaced, we need to double back and revisit + // the expanded sequence. This is both to assign encodings and possible to + // expand further. + // There's a risk of infinite looping here if the legalization patterns are + // unsound. Should we attempt to detect that? + if changed { + pos.set_position(prev_pos); + } } - Err(Legalize::Narrow) => { - narrow(&mut pos, &mut func.dfg); - } - // TODO: We should transform the instruction into legal equivalents. - // Possible strategies are: - // 1. Expand instruction into sequence of legal instructions. Possibly - // iteratively. - // 2. Split the controlling type variable into high and low parts. This applies - // both to SIMD vector types which can be halved and to integer types such - // as `i64` used on a 32-bit ISA. - // 3. Promote the controlling type variable to a larger type. This typically - // means expressing `i8` and `i16` arithmetic in terms if `i32` operations - // on RISC targets. (It may or may not be beneficial to promote small vector - // types versus splitting them.) - // 4. Convert to library calls. For example, floating point operations on an - // ISA with no IEEE 754 support. - // - // The iteration scheme used here is not going to cut it. Transforming - // instructions involves changing `function.layout` which is impossiblr while - // it is referenced by the two iterators. We need a layout cursor that can - // maintain a position *and* permit inserting and replacing instructions. } + + // Remember this position in case we need to double back. + prev_pos = pos.position(); } } } From 8d6d59cc7aaca0959f9fcefcd81ab8bf733d2009 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Nov 2016 10:34:14 -0700 Subject: [PATCH 0412/3084] Gather comments in the preamble of a test file. Comments preceding the first function are not associated with any specific entity in the file. Put them in a TestFile::preamble_comments field. --- lib/reader/src/lib.rs | 2 +- lib/reader/src/parser.rs | 14 +++++++++++++- lib/reader/src/testfile.rs | 6 ++++++ 3 files changed, 20 insertions(+), 2 deletions(-) diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index 25aa25aa98..61473afe38 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -10,7 +10,7 @@ extern crate cretonne; pub use error::{Location, Result, Error}; pub use parser::{parse_functions, parse_test}; pub use testcommand::{TestCommand, TestOption}; -pub use testfile::{TestFile, Details}; +pub use testfile::{TestFile, Details, Comment}; pub use isaspec::IsaSpec; pub use sourcemap::SourceMap; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 7b16abe272..c513e1a86a 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -38,9 +38,12 @@ pub fn parse_functions(text: &str) -> Result> { /// The returned `TestFile` contains direct references to substrings of `text`. pub fn parse_test<'a>(text: &'a str) -> Result> { let mut parser = Parser::new(text); + // Gather the preamble comments as 'Function'. + parser.gather_comments(AnyEntity::Function); Ok(TestFile { commands: parser.parse_test_commands(), isa_spec: try!(parser.parse_isa_specs()), + preamble_comments: parser.take_comments(), functions: try!(parser.parse_function_list()), }) } @@ -273,6 +276,11 @@ impl<'a> Parser<'a> { self.comment_entity = Some(entity.into()); } + // Get the comments gathered so far, clearing out the internal list. + fn take_comments(&mut self) -> Vec> { + mem::replace(&mut self.comments, Vec::new()) + } + // Rewrite the entity of the last added comments from `old` to `new`. // Also switch to collecting future comments for `new`. fn rewrite_last_comment_entities>(&mut self, old: E, new: E) { @@ -561,7 +569,7 @@ impl<'a> Parser<'a> { let details = Details { location: location, - comments: mem::replace(&mut self.comments, Vec::new()), + comments: self.take_comments(), map: ctx.map, }; @@ -1482,6 +1490,7 @@ mod tests { test cfg option=5 test verify set enable_float=false + ; still preamble function comment() {}") .unwrap(); assert_eq!(tf.commands.len(), 2); @@ -1491,6 +1500,9 @@ mod tests { IsaSpec::None(s) => assert!(!s.enable_float()), _ => panic!("unexpected ISAs"), } + assert_eq!(tf.preamble_comments.len(), 2); + assert_eq!(tf.preamble_comments[0].text, "; before"); + assert_eq!(tf.preamble_comments[1].text, "; still preamble"); assert_eq!(tf.functions.len(), 1); assert_eq!(tf.functions[0].0.name.to_string(), "comment"); } diff --git a/lib/reader/src/testfile.rs b/lib/reader/src/testfile.rs index deeaf04e5a..fcb11590e8 100644 --- a/lib/reader/src/testfile.rs +++ b/lib/reader/src/testfile.rs @@ -20,6 +20,9 @@ pub struct TestFile<'a> { pub commands: Vec>, /// `isa bar ...` lines. pub isa_spec: IsaSpec, + /// Comments appearing before the first function. + /// These are all tagged as 'Function' scope for lack of a better entity. + pub preamble_comments: Vec>, /// Parsed functions and additional details about each function. pub functions: Vec<(Function, Details<'a>)>, } @@ -47,6 +50,9 @@ pub struct Details<'a> { /// after the function are tagged as `AnyEntity::Function`. #[derive(Clone, PartialEq, Eq, Debug)] pub struct Comment<'a> { + /// The entity this comment is attached to. + /// Comments always follow their entity. pub entity: AnyEntity, + /// Text of the comment, including the leading `;`. pub text: &'a str, } From cb718b869c555ae8874c549e7bd9a01033ee446c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Nov 2016 10:49:09 -0700 Subject: [PATCH 0413/3084] TestFile preamble comments apply to all functions. Include the test file preamble comments when building a filecheck instance for every function in the file. This makes it possible to define common regex variables in the preamble and use these definitions for all the functions. --- cranelift/docs/testing.rst | 4 ++++ cranelift/filetests/isa/riscv/legalize-i64.cton | 9 +++------ cranelift/src/filetest/runone.rs | 1 + cranelift/src/filetest/subtest.rs | 17 ++++++++++++----- 4 files changed, 20 insertions(+), 11 deletions(-) diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index 419f51ff22..d52b180c99 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -150,6 +150,10 @@ use filecheck will extract comments associated with each function (or its entities) and scan them for filecheck directives. The test output for each function is then matched against the filecheck directives for that function. +Comments appearing before the first function in a file apply to every function. +This is useful for defining common regular expression variables with the +``regex:`` directive, for example. + Note that LLVM's file tests don't separate filecheck directives by their associated function. It verifies the concatenated output against all filecheck directives in the test file. LLVM's :command:`FileCheck` command has a diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton index 916986fe68..d4f3e6b05e 100644 --- a/cranelift/filetests/isa/riscv/legalize-i64.cton +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -2,13 +2,14 @@ test legalizer isa riscv supports_m=1 +; regex: V=v\d+ +; regex: VX=vx\d+ + function bitwise_and(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): v3 = band v1, v2 return v3 } -; regex: V=v\d+ -; regex: VX=vx\d+ ; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 ; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 ; check: [R#ec @@ -22,8 +23,6 @@ ebb0(v1: i64, v2: i64): v3 = bor v1, v2 return v3 } -; regex: V=v\d+ -; regex: VX=vx\d+ ; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 ; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 ; check: [R#cc @@ -37,8 +36,6 @@ ebb0(v1: i64, v2: i64): v3 = bxor v1, v2 return v3 } -; regex: V=v\d+ -; regex: VX=vx\d+ ; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 ; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 ; check: [R#8c diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index d0199c45cf..d84f0f7041 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -50,6 +50,7 @@ pub fn run(path: &Path) -> TestResult { for (func, details) in testfile.functions { let mut context = Context { + preamble_comments: &testfile.preamble_comments, details: details, verified: false, flags: flags, diff --git a/cranelift/src/filetest/subtest.rs b/cranelift/src/filetest/subtest.rs index fee9bbef85..f78ac7a6b1 100644 --- a/cranelift/src/filetest/subtest.rs +++ b/cranelift/src/filetest/subtest.rs @@ -5,13 +5,16 @@ use std::borrow::Cow; use cretonne::ir::Function; use cretonne::isa::TargetIsa; use cretonne::settings::Flags; -use cton_reader::Details; +use cton_reader::{Details, Comment}; use filecheck::{self, CheckerBuilder, Checker, Value as FCValue}; pub type Result = result::Result; /// Context for running a a test on a single function. pub struct Context<'a> { + /// Comments from the preamble f the test file. These apply to all functions. + pub preamble_comments: &'a [Comment<'a>], + /// Additional details about the function from the parser. pub details: Details<'a>, @@ -69,7 +72,7 @@ impl<'a> filecheck::VariableMap for Context<'a> { /// Run filecheck on `text`, using directives extracted from `context`. pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { - let checker = try!(build_filechecker(&context.details)); + let checker = try!(build_filechecker(context)); if try!(checker.check(&text, context).map_err(|e| format!("filecheck: {}", e))) { Ok(()) } else { @@ -80,10 +83,14 @@ pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { } } -/// Build a filechecker using the directives in the function's comments. -pub fn build_filechecker(details: &Details) -> Result { +/// Build a filechecker using the directives in the file preamble and the function's comments. +pub fn build_filechecker(context: &Context) -> Result { let mut builder = CheckerBuilder::new(); - for comment in &details.comments { + // Preamble comments apply to all functions. + for comment in context.preamble_comments { + try!(builder.directive(comment.text).map_err(|e| format!("filecheck: {}", e))); + } + for comment in &context.details.comments { try!(builder.directive(comment.text).map_err(|e| format!("filecheck: {}", e))); } let checker = builder.finish(); From b63d62c9e8f727df87c795f217b33d4279ffda5d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Nov 2016 11:54:42 -0700 Subject: [PATCH 0414/3084] Advance the insertion cursor after replacinf an instruction. When expanding iadd_cout, the original instruction is replaced with an iadd, and an icmp is inserted after the iadd. Make sure we advance the insertion position after replacing iadd_cout so the icmp gets inserted *after* iadd. --- cranelift/filetests/isa/riscv/expand-i32.cton | 18 ++++++++++++++++++ lib/cretonne/meta/gen_legalizer.py | 17 +++++++++++------ 2 files changed, 29 insertions(+), 6 deletions(-) create mode 100644 cranelift/filetests/isa/riscv/expand-i32.cton diff --git a/cranelift/filetests/isa/riscv/expand-i32.cton b/cranelift/filetests/isa/riscv/expand-i32.cton new file mode 100644 index 0000000000..b2473aa608 --- /dev/null +++ b/cranelift/filetests/isa/riscv/expand-i32.cton @@ -0,0 +1,18 @@ +; Test the legalization of i32 instructions that don't have RISC-V versions. +test legalizer + +set is_64bit=0 +isa riscv supports_m=1 + +set is_64bit=1 +isa riscv supports_m=1 + +; regex: V=vx?\d+ + +function carry_out(i32, i32) -> i32, b1 { +ebb0(v1: i32, v2: i32): + v3, v4 = iadd_cout v1, v2 + return v3, v4 +} +; check: $v3 = iadd $v1, $v2 +; check: $(cout=$V) = icmp ult, $v3, $v1 diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 3efb120124..bc662dca8b 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -105,6 +105,7 @@ def wrap_tup(seq): def emit_dst_inst(node, fmt): # type: (Def, Formatter) -> None exact_replace = False + replaced_inst = None # type: str if len(node.defs) == 0: # This node doesn't define any values, so just insert the new # instruction. @@ -116,6 +117,7 @@ def emit_dst_inst(node, fmt): # pattern. # Replace the whole instruction. builder = 'let {} = dfg.replace(inst)'.format(wrap_tup(node.defs)) + replaced_inst = 'inst' # Secondary values weren't replaced if this is an exact replacement # for all the src results. exact_replace = (node.defs == src_def0.defs) @@ -126,6 +128,14 @@ def emit_dst_inst(node, fmt): fmt.line('{}.{};'.format(builder, node.expr.rust_builder())) + # If we just replaced an instruction, we need to bump the cursor so + # following instructions are inserted *after* the replaced insruction. + if replaced_inst: + with fmt.indented( + 'if pos.current_inst() == Some({}) {{' + .format(replaced_inst), '}'): + fmt.line('pos.next_inst();') + if exact_replace: fmt.comment('exactreplacement') @@ -151,12 +161,7 @@ def gen_xform(xform, fmt): def gen_xform_group(xgrp, fmt): # type: (XFormGroup, Formatter) -> None - fmt.doc_comment(""" - Legalize the instruction pointed to by `pos`. - - Return the first instruction in the expansion, and leave `pos` pointing - at the last instruction in the expansion. - """) + fmt.doc_comment("Legalize the instruction pointed to by `pos`.") fmt.line('#[allow(unused_variables,unused_assignments)]') with fmt.indented( 'fn ' + xgrp.name + From dc2afb24d92cdb790a4565c5d2aca1b22f32bee4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Nov 2016 12:30:51 -0700 Subject: [PATCH 0415/3084] Add a ref_slice module. Utility functions for converting &T to an &[T] slice with a single element. --- lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/ref_slice.rs | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+) create mode 100644 lib/cretonne/src/ref_slice.rs diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index aef83bd433..51a8d9b367 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -21,3 +21,4 @@ mod write; mod constant_hash; mod predicates; mod legalizer; +mod ref_slice; diff --git a/lib/cretonne/src/ref_slice.rs b/lib/cretonne/src/ref_slice.rs new file mode 100644 index 0000000000..b9dbd55dd1 --- /dev/null +++ b/lib/cretonne/src/ref_slice.rs @@ -0,0 +1,18 @@ +//! Functions for converting a reference into a singleton slice. +//! +//! See also the ref_slice crate on crates.io. +//! +//! We define the functions here to avoid external dependencies, and to ensure that they are +//! inlined in this crate. +//! +//! Despite their using an unsafe block, these functions are completely safe. + +use std::slice; + +pub fn ref_slice(s: &T) -> &[T] { + unsafe { slice::from_raw_parts(s, 1) } +} + +pub fn ref_slice_mut(s: &mut T) -> &mut [T] { + unsafe { slice::from_raw_parts_mut(s, 1) } +} From eb2b56c20ade8a479d771c3566cb8193200ab06d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Nov 2016 13:25:40 -0700 Subject: [PATCH 0416/3084] Add arguments() and arguments_mut() methods. Provide a generic way of accessing the value arguments on an instruction. This is provided as two slice references. One for the fixed arguments and one for any VariableArgs. The arguments() methods return an array of two slices which is a bit awkward. Also provide an each_arg() method which passes each argument value to a closure. --- lib/cretonne/meta/gen_instr.py | 74 +++++++++++++++++++++++++++++ lib/cretonne/src/ir/instructions.rs | 52 +++++++++++++++++++- 2 files changed, 125 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 8b4555d9b2..569eca71b7 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -9,6 +9,7 @@ import cretonne def gen_formats(fmt): + # type: (srcgen.Formatter) -> None """Generate an instruction format enumeration""" fmt.doc_comment('An instruction format') @@ -38,7 +39,63 @@ def gen_formats(fmt): fmt.line() +def gen_arguments_method(fmt, is_mut): + # type: (srcgen.Formatter, bool) -> None + method = 'arguments' + mut = '' + rslice = 'ref_slice' + if is_mut: + method += '_mut' + mut = 'mut ' + rslice += '_mut' + + with fmt.indented( + 'pub fn {f}(&{m}self) -> [&{m}[Value]; 2] {{' + .format(f=method, m=mut), '}'): + with fmt.indented('match *self {', '}'): + for f in cretonne.InstructionFormat.all_formats: + n = 'InstructionData::' + f.name + has_varargs = cretonne.variable_args in f.kinds + # Formats with both fixed and variable arguments delegate to + # the data struct. We need to work around borrow checker quirks + # when extracting two mutable references. + if has_varargs and len(f.value_operands) > 0: + fmt.line( + '{} {{ ref {}data, .. }} => data.{}(),' + .format(n, mut, method)) + continue + # Fixed args. + if len(f.value_operands) == 0: + arg = '&{}[]'.format(mut) + capture = '' + elif len(f.value_operands) == 1: + if f.boxed_storage: + capture = 'ref {}data, '.format(mut) + arg = '{}(&{}data.arg)'.format(rslice, mut) + else: + capture = 'ref {}arg, '.format(mut) + arg = '{}(arg)'.format(rslice) + else: + if f.boxed_storage: + capture = 'ref {}data, '.format(mut) + arg = '&{}data.args'.format(mut) + else: + capture = 'ref {}args, '.format(mut) + arg = 'args' + # Varargs. + if cretonne.variable_args in f.kinds: + varg = '&{}data.varargs'.format(mut) + capture = 'ref {}data, '.format(mut) + else: + varg = '&{}[]'.format(mut) + + fmt.line( + '{} {{ {} .. }} => [{}, {}],' + .format(n, capture, arg, varg)) + + def gen_instruction_data_impl(fmt): + # type: (srcgen.Formatter) -> None """ Generate the boring parts of the InstructionData implementation. @@ -49,6 +106,7 @@ def gen_instruction_data_impl(fmt): - `pub fn first_type(&self) -> Type` - `pub fn second_result(&self) -> Option` - `pub fn second_result_mut<'a>(&'a mut self) -> Option<&'a mut Value>` + - `pub fn arguments(&self) -> (&[Value], &[Value])` """ # The `opcode` and `first_type` methods simply read the `opcode` and `ty` @@ -148,6 +206,22 @@ def gen_instruction_data_impl(fmt): ' {{ ref args, .. }} => Some(args[{}]),' .format(i)) + fmt.doc_comment( + """ + Get the value arguments to this instruction. + + This is returned as two `Value` slices. The first one + represents the fixed arguments, the second any variable + arguments. + """) + gen_arguments_method(fmt, False) + fmt.doc_comment( + """ + Get mutable references to the value arguments to this + instruction. + """) + gen_arguments_method(fmt, True) + def collect_instr_groups(isas): seen = set() diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index f601169beb..b8c5bdc4fa 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -15,6 +15,8 @@ use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, ImmVector}; use ir::condcodes::*; use ir::types; +use ref_slice::*; + // Include code generated by `lib/cretonne/meta/gen_instr.py`. This file contains: // // - The `pub enum InstructionFormat` enum with all the instruction formats. @@ -339,6 +341,18 @@ pub struct BranchData { pub varargs: VariableArgs, } +impl BranchData { + /// Get references to the arguments. + pub fn arguments(&self) -> [&[Value]; 2] { + [ref_slice(&self.arg), &self.varargs] + } + + /// Get mutable references to the arguments. + pub fn arguments_mut(&mut self) -> [&mut [Value]; 2] { + [ref_slice_mut(&mut self.arg), &mut self.varargs] + } +} + impl Display for BranchData { fn fmt(&self, f: &mut Formatter) -> fmt::Result { try!(write!(f, "{}, {}", self.arg, self.destination)); @@ -372,6 +386,18 @@ pub struct IndirectCallData { pub varargs: VariableArgs, } +impl IndirectCallData { + /// Get references to the arguments. + pub fn arguments(&self) -> [&[Value]; 2] { + [ref_slice(&self.arg), &self.varargs] + } + + /// Get mutable references to the arguments. + pub fn arguments_mut(&mut self) -> [&mut [Value]; 2] { + [ref_slice_mut(&mut self.arg), &mut self.varargs] + } +} + /// Payload of a return instruction. #[derive(Clone, Debug)] pub struct ReturnData { @@ -381,9 +407,33 @@ pub struct ReturnData { /// Analyzing an instruction. /// -/// Avoid large matches on instruction formats by using the methods efined here to examine +/// Avoid large matches on instruction formats by using the methods defined here to examine /// instructions. impl InstructionData { + /// Execute a closure once for each argument to this instruction. + /// See also the `arguments()` method. + pub fn each_arg(&self, func: F) + where F: Fn(Value) + { + for part in &self.arguments() { + for &arg in part.iter() { + func(arg); + } + } + } + + /// Execute a closure with a mutable reference to each argument to this instruction. + /// See also the `arguments_mut()` method. + pub fn each_arg_mut(&mut self, func: F) + where F: Fn(&mut Value) + { + for part in &mut self.arguments_mut() { + for arg in part.iter_mut() { + func(arg); + } + } + } + /// Return information about the destination of a branch or jump instruction. /// /// Any instruction that can transfer control to another EBB reveals its possible destinations From 55bc5599cc0c4bc2b200a014c4bd2c206bc4416f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Nov 2016 15:27:34 -0700 Subject: [PATCH 0417/3084] Fix off-by-one in resolve_values. When the extended_values table is empty, the value to resolve is definitely not an alias, but we still need as least one trip in the loop to determine that. --- lib/cretonne/src/ir/dfg.rs | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 0d24bc2d1a..7e843d0c87 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -137,7 +137,8 @@ impl DataFlowGraph { use ir::entities::ExpandedValue::Table; let mut v = value; - for _ in 0..self.extended_values.len() { + // Note that extended_values may be empty here. + for _ in 0..1 + self.extended_values.len() { v = match v.expand() { Table(idx) => { match self.extended_values[idx] { @@ -655,12 +656,16 @@ mod tests { let mut func = Function::new(); let dfg = &mut func.dfg; let ebb0 = dfg.make_ebb(); - let arg0 = dfg.append_ebb_arg(ebb0, types::I32); let pos = &mut Cursor::new(&mut func.layout); pos.insert_ebb(ebb0); // Build a little test program. let v1 = dfg.ins(pos).iconst(types::I32, 42); + + // Make sure we can resolve value aliases even when extended_values is empty. + assert_eq!(dfg.resolve_aliases(v1), v1); + + let arg0 = dfg.append_ebb_arg(ebb0, types::I32); let (s, c) = dfg.ins(pos).iadd_cout(v1, arg0); let iadd = match s.expand() { Direct(i) => i, From 4460adbfc206d372d7a9d52591531985cc4aa096 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Nov 2016 15:19:44 -0700 Subject: [PATCH 0418/3084] Write out value aliases when writing instructions. If an instruction uses any values that are aliases of other values, print out the alias mappings on lines preceding the instruction. This is necessary to reconstruct the data flow graph. We don't make any attempt to only write out each alias mapping once. The parser does not yet support value aliases. --- lib/cretonne/src/write.rs | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index ac487c6f37..d8db064cd6 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -145,11 +145,28 @@ fn type_suffix(func: &Function, inst: Inst) -> Option { Some(rtype) } +// Write out any value aliases appearing in `inst`. +fn write_value_aliases(w: &mut Write, func: &Function, inst: Inst, indent: usize) -> Result { + for &arg in func.dfg[inst].arguments().iter().flat_map(|x| x.iter()) { + let resolved = func.dfg.resolve_aliases(arg); + if resolved != arg { + try!(writeln!(w, "{1:0$}{2} -> {3}", indent, "", arg, resolved)); + } + } + Ok(()) +} + fn write_instruction(w: &mut Write, func: &Function, isa: Option<&TargetIsa>, inst: Inst) -> Result { + // Indent all instructions to col 24 if any encodings are present. + let indent = if func.encodings.is_empty() { 4 } else { 24 }; + + // Value aliases come out on lines before the instruction using them. + try!(write_value_aliases(w, func, inst, indent)); + // Write out encoding info. if let Some(enc) = func.encodings.get(inst).cloned() { let mut s = String::with_capacity(16); @@ -161,8 +178,8 @@ fn write_instruction(w: &mut Write, // Align instruction following ISA annotation to col 24. try!(write!(w, "{:23} ", s)); } else { - // No annotations, simply indent by 4. - try!(write!(w, " ")); + // No annotations, simply indent. + try!(write!(w, "{1:0$}", indent, "")); } // Write out the result values, if any. From 453a1b2d170b7e8ca00f68e191ed809fb9bf10fc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Nov 2016 14:49:31 -0700 Subject: [PATCH 0419/3084] Create value aliases when necessary. If a secondary value in the source pattern becomes a primary value in the destination pattern, it is not possible to overwrite the definition of the source value. Instead, change the original source value to an alias to the new promary value. --- cranelift/filetests/isa/riscv/expand-i32.cton | 3 +++ lib/cretonne/meta/gen_legalizer.py | 10 ++++++++++ 2 files changed, 13 insertions(+) diff --git a/cranelift/filetests/isa/riscv/expand-i32.cton b/cranelift/filetests/isa/riscv/expand-i32.cton index b2473aa608..c0ca7be56f 100644 --- a/cranelift/filetests/isa/riscv/expand-i32.cton +++ b/cranelift/filetests/isa/riscv/expand-i32.cton @@ -16,3 +16,6 @@ ebb0(v1: i32, v2: i32): } ; check: $v3 = iadd $v1, $v2 ; check: $(cout=$V) = icmp ult, $v3, $v1 +; It's possible the legalizer will rewrite these value aliases in the future. +; check: $v4 -> $cout +; check: return $v3, $v4 diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index bc662dca8b..6c154adc56 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -106,6 +106,7 @@ def emit_dst_inst(node, fmt): # type: (Def, Formatter) -> None exact_replace = False replaced_inst = None # type: str + fixup_first_result = False if len(node.defs) == 0: # This node doesn't define any values, so just insert the new # instruction. @@ -125,6 +126,7 @@ def emit_dst_inst(node, fmt): # Insert a new instruction since its primary def doesn't match the # src. builder = 'let {} = dfg.ins(pos)'.format(wrap_tup(node.defs)) + fixup_first_result = node.defs[0].is_output() fmt.line('{}.{};'.format(builder, node.expr.rust_builder())) @@ -139,6 +141,14 @@ def emit_dst_inst(node, fmt): if exact_replace: fmt.comment('exactreplacement') + # Fix up any output vars. + if fixup_first_result: + # The first result of the instruction just inserted is an output var, + # but it was not a primary result in the source pattern. + # We need to change the original value to an alias of the primary one + # we just inserted. + fmt.line('dfg.change_to_alias(src_{0}, {0});'.format(node.defs[0])) + def gen_xform(xform, fmt): # type: (XForm, Formatter) -> None From be3577ad7e5ee7b3416dccb947a9ee0aaf3467f3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Nov 2016 15:54:58 -0700 Subject: [PATCH 0420/3084] Resolve value aliases when legalizing instructions. Since we're deconstructing an instruction anyway, go ahead and resolve any value aliases on its arguments before we construct the replacement instructions. --- .../filetests/isa/riscv/legalize-i64.cton | 20 +++++++++++++++++++ lib/cretonne/meta/gen_legalizer.py | 8 ++++---- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton index d4f3e6b05e..41bc2e2947 100644 --- a/cranelift/filetests/isa/riscv/legalize-i64.cton +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -43,3 +43,23 @@ ebb0(v1: i64, v2: i64): ; check: [R#8c ; sameln: $(v3h=$V) = bxor $v1h, $v2h ; check: $v3 = iconcat_lohi $v3l, $v3h + +function arith_add(i64, i64) -> i64 { +; Legalizing iadd.i64 requires two steps: +; 1. Narrow to iadd_cout.i32, then +; 2. Expand iadd_cout.i32 since RISC-V has no carry flag. +ebb0(v1: i64, v2: i64): + v3 = iadd v1, v2 + return v3 +} +; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 +; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 +; check: [R#0c +; sameln: $(v3l=$V) = iadd $v1l, $v2l +; check: $(c=$V) = icmp ult, $v3l, $v1l +; check: [R#0c +; sameln: $(v3h1=$V) = iadd $v1h, $v2h +; TODO: This doesn't typecheck. We need to convert the b1 result to i32. +; check: [R#0c +; sameln: $(v3h=$V) = iadd $v3h1, $c +; check: $v3 = iconcat_lohi $v3l, $v3h diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 6c154adc56..ca7a81e069 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -63,11 +63,11 @@ def unwrap_inst(iref, node, fmt): else: # This is a value operand. if nvops == 1: - outs.append(prefix + 'arg') + arg = prefix + 'arg' else: - outs.append( - '{}args[{}]'.format( - prefix, iform.value_operands.index(i))) + arg = '{}args[{}]'.format( + prefix, iform.value_operands.index(i)) + outs.append('dfg.resolve_aliases({})'.format(arg)) fmt.line('({})'.format(', '.join(outs))) fmt.outdented_line('} else {') fmt.line('unreachable!("bad instruction format")') From 15c635f537bd8481708b53b0b59e000a395ebc91 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Nov 2016 17:39:20 -0700 Subject: [PATCH 0421/3084] Check for unsupported value transformations. Add an assertion for the value placements that we don't support yet. 1. A primary result in the source pattern becomes a secondary result in the destination. 2. A secondary result becomes a secondary result, and the destination instruction is not exactly matching the source. --- lib/cretonne/meta/gen_legalizer.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index ca7a81e069..bd4ff926eb 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -138,9 +138,6 @@ def emit_dst_inst(node, fmt): .format(replaced_inst), '}'): fmt.line('pos.next_inst();') - if exact_replace: - fmt.comment('exactreplacement') - # Fix up any output vars. if fixup_first_result: # The first result of the instruction just inserted is an output var, @@ -149,6 +146,15 @@ def emit_dst_inst(node, fmt): # we just inserted. fmt.line('dfg.change_to_alias(src_{0}, {0});'.format(node.defs[0])) + if not exact_replace: + # We don't support secondary values as outputs yet. Depending on the + # source value, we would need to : + # 1. For a primary source value, replace with a copy instruction. + # 2. For a secondary source value, request that the builder reuses the + # value when making secondary result nodes. + for d in node.defs[1:]: + assert not d.is_output() + def gen_xform(xform, fmt): # type: (XForm, Formatter) -> None From 43890796473fd8e3ae899bd2f7ff88f0143944ea Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 08:52:21 -0800 Subject: [PATCH 0422/3084] Add more Python type annotations. --- lib/cretonne/meta/cretonne/__init__.py | 44 +++++++++++++++++++++++--- lib/cretonne/meta/cretonne/typevar.py | 2 +- 2 files changed, 41 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index 8a46b2d08f..7463f47d69 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -14,11 +14,15 @@ from .predicates import And, Predicate, FieldPredicate # noqa # The typing module is only required by mypy, and we don't use these imports # outside type comments. try: - from typing import Tuple, Union, Any, Iterable, Sequence # noqa + from typing import Tuple, Union, Any, Iterable, Sequence, TYPE_CHECKING # noqa MaybeBoundInst = Union['Instruction', 'BoundInstruction'] AnyPredicate = Union['Predicate', 'FieldPredicate'] + OperandSpec = Union['OperandKind', 'ValueType', 'TypeVar'] except ImportError: - pass + TYPE_CHECKING = False + +if TYPE_CHECKING: + from .typevar import TypeVar # noqa camel_re = re.compile('(^|_)([a-z])') @@ -299,6 +303,7 @@ class OperandKind(object): """ def __init__(self, name, doc, default_member=None, rust_type=None): + # type: (str, str, str, str) -> None self.name = name self.__doc__ = doc self.default_member = default_member @@ -307,12 +312,15 @@ class OperandKind(object): self.rust_type = rust_type or camel_case(name) def __str__(self): + # type: () -> str return self.name def __repr__(self): + # type: () -> str return 'OperandKind({})'.format(self.name) def operand_kind(self): + # type: () -> OperandKind """ An `OperandKind` instance can be used directly as the type of an `Operand` when defining an instruction. @@ -357,10 +365,12 @@ class ImmediateKind(OperandKind): """ def __init__(self, name, doc, default_member='imm', rust_type=None): + # type: (str, str, str, str) -> None super(ImmediateKind, self).__init__( name, doc, default_member, rust_type) def __repr__(self): + # type: () -> str return 'ImmediateKind({})'.format(self.name) @@ -372,10 +382,12 @@ class EntityRefKind(OperandKind): """ def __init__(self, name, doc, default_member=None, rust_type=None): + # type: (str, str, str, str) -> None super(EntityRefKind, self).__init__( name, doc, default_member or name, rust_type) def __repr__(self): + # type: () -> str return 'EntityRefKind({})'.format(self.name) @@ -395,6 +407,7 @@ class ValueType(object): all_scalars = list() # type: List[ValueType] def __init__(self, name, membytes, doc): + # type: (str, int, str) -> None self.name = name self.membytes = membytes self.__doc__ = doc @@ -402,9 +415,11 @@ class ValueType(object): ValueType._registry[name] = self def __str__(self): + # type: () -> str return self.name def operand_kind(self): + # type: () -> OperandKind """ When a `ValueType` object is used to describe the type of an `Operand` in an instruction definition, the kind of that operand is an SSA value. @@ -416,6 +431,7 @@ class ValueType(object): @staticmethod def by_name(name): + # type: (str) -> ValueType if name in ValueType._registry: return ValueType._registry[name] else: @@ -431,20 +447,24 @@ class ScalarType(ValueType): """ def __init__(self, name, membytes, doc): + # type: (str, int, str) -> None super(ScalarType, self).__init__(name, membytes, doc) - self._vectors = dict() + self._vectors = dict() # type: Dict[int, VectorType] # Assign numbers starting from 1. (0 is VOID). ValueType.all_scalars.append(self) self.number = len(ValueType.all_scalars) assert self.number < 16, 'Too many scalar types' def __repr__(self): + # type: () -> str return 'ScalarType({})'.format(self.name) def rust_name(self): + # type: () -> str return 'types::' + self.name.upper() def by(self, lanes): + # type: (int) -> VectorType """ Get a vector type with this type as the lane type. @@ -467,6 +487,7 @@ class VectorType(ValueType): """ def __init__(self, base, lanes): + # type: (ScalarType, int) -> None assert isinstance(base, ScalarType), 'SIMD lanes must be scalar types' super(VectorType, self).__init__( name='{}x{}'.format(base.name, lanes), @@ -479,6 +500,7 @@ class VectorType(ValueType): self.number = 16*int(math.log(lanes, 2)) + base.number def __repr__(self): + # type: () -> str return ('VectorType(base={}, lanes={})' .format(self.base.name, self.lanes)) @@ -487,6 +509,7 @@ class IntType(ScalarType): """A concrete scalar integer type.""" def __init__(self, bits): + # type: (int) -> None assert bits > 0, 'IntType must have positive number of bits' super(IntType, self).__init__( name='i{:d}'.format(bits), @@ -495,6 +518,7 @@ class IntType(ScalarType): self.bits = bits def __repr__(self): + # type: () -> str return 'IntType(bits={})'.format(self.bits) @@ -502,6 +526,7 @@ class FloatType(ScalarType): """A concrete scalar floating point type.""" def __init__(self, bits, doc): + # type: (int, str) -> None assert bits > 0, 'FloatType must have positive number of bits' super(FloatType, self).__init__( name='f{:d}'.format(bits), @@ -510,6 +535,7 @@ class FloatType(ScalarType): self.bits = bits def __repr__(self): + # type: () -> str return 'FloatType(bits={})'.format(self.bits) @@ -517,6 +543,7 @@ class BoolType(ScalarType): """A concrete scalar boolean type.""" def __init__(self, bits): + # type: (int) -> None assert bits > 0, 'BoolType must have positive number of bits' super(BoolType, self).__init__( name='b{:d}'.format(bits), @@ -525,6 +552,7 @@ class BoolType(ScalarType): self.bits = bits def __repr__(self): + # type: () -> str return 'BoolType(bits={})'.format(self.bits) @@ -545,6 +573,7 @@ class InstructionGroup(object): _current = None # type: InstructionGroup def open(self): + # type: () -> None """ Open this instruction group such that future new instructions are added to this group. @@ -555,6 +584,7 @@ class InstructionGroup(object): InstructionGroup._current = self def close(self): + # type: () -> None """ Close this instruction group. This function should be called before opening another instruction group. @@ -565,13 +595,15 @@ class InstructionGroup(object): InstructionGroup._current = None def __init__(self, name, doc): + # type: (str, str) -> None self.name = name self.__doc__ = doc - self.instructions = [] + self.instructions = [] # type: List[Instruction] self.open() @staticmethod def append(inst): + # type: (Instruction) -> None assert InstructionGroup._current, \ "Open an instruction group before defining instructions." InstructionGroup._current.instructions.append(inst) @@ -599,21 +631,25 @@ class Operand(object): """ def __init__(self, name, typ, doc=''): + # type: (str, OperandSpec, str) -> None self.name = name self.typ = typ self.__doc__ = doc self.kind = typ.operand_kind() def get_doc(self): + # type: () -> str if self.__doc__: return self.__doc__ else: return self.typ.__doc__ def __str__(self): + # type: () -> str return "`{}`".format(self.name) def is_value(self): + # type: () -> bool """ Is this an SSA value operand? """ diff --git a/lib/cretonne/meta/cretonne/typevar.py b/lib/cretonne/meta/cretonne/typevar.py index fa0179304b..18f1e6fa2c 100644 --- a/lib/cretonne/meta/cretonne/typevar.py +++ b/lib/cretonne/meta/cretonne/typevar.py @@ -319,7 +319,7 @@ class TypeVar(object): # When a `TypeVar` object is used to describe the type of an `Operand` # in an instruction definition, the kind of that operand is an SSA # value. - return value + return value # type: ignore def free_typevar(self): # type: () -> TypeVar From 6419ba12bc32f79ac3e64655344b09a92acb0fb2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 09:04:40 -0800 Subject: [PATCH 0423/3084] Resolve import cycles. --- lib/cretonne/meta/cretonne/ast.py | 8 ++++---- lib/cretonne/meta/cretonne/typevar.py | 6 +++--- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/lib/cretonne/meta/cretonne/ast.py b/lib/cretonne/meta/cretonne/ast.py index ac21809317..b437148435 100644 --- a/lib/cretonne/meta/cretonne/ast.py +++ b/lib/cretonne/meta/cretonne/ast.py @@ -5,7 +5,7 @@ This module defines classes that can be used to create abstract syntax trees for patern matching an rewriting of cretonne instructions. """ from __future__ import absolute_import -from . import Instruction, BoundInstruction +import cretonne try: from typing import Union, Tuple # noqa @@ -174,12 +174,12 @@ class Apply(Expr): """ def __init__(self, inst, args): - # type: (Union[Instruction, BoundInstruction], Tuple[Expr, ...]) -> None # noqa - if isinstance(inst, BoundInstruction): + # type: (Union[cretonne.Instruction, cretonne.BoundInstruction], Tuple[Expr, ...]) -> None # noqa + if isinstance(inst, cretonne.BoundInstruction): self.inst = inst.inst self.typevars = inst.typevars else: - assert isinstance(inst, Instruction) + assert isinstance(inst, cretonne.Instruction) self.inst = inst self.typevars = () self.args = args diff --git a/lib/cretonne/meta/cretonne/typevar.py b/lib/cretonne/meta/cretonne/typevar.py index 18f1e6fa2c..a6d8d9637f 100644 --- a/lib/cretonne/meta/cretonne/typevar.py +++ b/lib/cretonne/meta/cretonne/typevar.py @@ -6,7 +6,7 @@ polymorphic by using type variables. """ from __future__ import absolute_import import math -from . import OperandKind, value # noqa +import cretonne try: from typing import Tuple, Union # noqa @@ -315,11 +315,11 @@ class TypeVar(object): return TypeVar(None, None, base=self, derived_func='DoubleWidth') def operand_kind(self): - # type: () -> OperandKind + # type: () -> cretonne.OperandKind # When a `TypeVar` object is used to describe the type of an `Operand` # in an instruction definition, the kind of that operand is an SSA # value. - return value # type: ignore + return cretonne.value # type: ignore def free_typevar(self): # type: () -> TypeVar From ac59376c46c9532dec0bf8167e0acb32a2d5fec2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 09:40:19 -0800 Subject: [PATCH 0424/3084] Move ValueType into a new cdsl top-level module. We want to separate the Python classes that make up the DSL used to define the Cretonne language from the concrete definitions. - cdsl.types defines the ValueType class hierarchy. - base.types defines the concrete types. --- cranelift/docs/cton_domain.py | 6 +- cranelift/docs/metaref.rst | 25 +-- lib/cretonne/meta/base/__init__.py | 1 + lib/cretonne/meta/{cretonne => base}/types.py | 4 +- lib/cretonne/meta/cdsl/__init__.py | 6 + lib/cretonne/meta/cdsl/types.py | 160 ++++++++++++++ lib/cretonne/meta/check.sh | 2 +- lib/cretonne/meta/cretonne/__init__.py | 198 ++---------------- lib/cretonne/meta/cretonne/base.py | 2 +- lib/cretonne/meta/gen_instr.py | 5 +- lib/cretonne/meta/gen_types.py | 3 +- 11 files changed, 210 insertions(+), 202 deletions(-) create mode 100644 lib/cretonne/meta/base/__init__.py rename lib/cretonne/meta/{cretonne => base}/types.py (88%) create mode 100644 lib/cretonne/meta/cdsl/__init__.py create mode 100644 lib/cretonne/meta/cdsl/types.py diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index b45aa5e677..00242ff565 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -245,7 +245,7 @@ class TypeDocumenter(sphinx.ext.autodoc.Documenter): return False def resolve_name(self, modname, parents, path, base): - return 'cretonne.types', [base] + return 'base.types', [base] def add_content(self, more_content, no_docstring=False): super(TypeDocumenter, self).add_content(more_content, no_docstring) @@ -280,11 +280,11 @@ class InstDocumenter(sphinx.ext.autodoc.Documenter): op = inst.ins[0] sig += ' ' + op.name # If the first input is variable-args, this is 'return'. No parens. - if op.typ.operand_kind().name == 'variable_args': + if op.kind.name == 'variable_args': sig += '...'.format(op.name) for op in inst.ins[1:]: # This is a call or branch with args in (...). - if op.typ.operand_kind().name == 'variable_args': + if op.kind.name == 'variable_args': sig += '({}...)'.format(op.name) else: sig += ', ' + op.name diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index fa3226c51f..6d8d8ad2b1 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -4,7 +4,7 @@ Cretonne Meta Language Reference .. default-domain:: py .. highlight:: python -.. module:: cretonne +.. module:: cdsl The Cretonne meta language is used to define instructions for Cretonne. It is a domain specific language embedded in Python. This document describes the Python @@ -16,8 +16,8 @@ steps: 1. The Python modules are imported. This has the effect of building static data structures in global variables in the modules. These static data structures - use the classes in the :mod:`cretonne` module to describe instruction sets - and other properties. + in the :mod:`base` and :mod:`isa` packages use the classes in the + :mod:`cdsl` module to describe instruction sets and other properties. 2. The static data structures are processed to produce Rust source code and constant tables. @@ -41,6 +41,7 @@ ISA, and defined in a `settings` module under the appropriate Settings can take boolean on/off values, small numbers, or explicitly enumerated symbolic values. Each type is represented by a sub-class of :class:`Setting`: +.. currentmodule:: cretonne .. inheritance-diagram:: Setting BoolSetting NumSetting EnumSetting :parts: 1 @@ -125,36 +126,36 @@ Immediate operands Immediate instruction operands don't correspond to SSA values, but have values that are encoded directly in the instruction. Immediate operands don't -have types from the :class:`cretonne.ValueType` type system; they often have +have types from the :class:`cdsl.types.ValueType` type system; they often have enumerated values of a specific type. The type of an immediate operand is indicated with an instance of :class:`ImmediateKind`. +.. currentmodule:: cretonne .. autoclass:: ImmediateKind .. automodule:: cretonne.immediates :members: -.. currentmodule:: cretonne - Entity references ----------------- Instruction operands can also refer to other entities in the same function. This can be extended basic blocks, or entities declared in the function preamble. +.. currentmodule:: cretonne + .. autoclass:: EntityRefKind .. automodule:: cretonne.entities :members: -.. currentmodule:: cretonne - Value types ----------- -Concrete value types are represented as instances of :class:`cretonne.ValueType`. There are +Concrete value types are represented as instances of :class:`cdsl.types.ValueType`. There are subclasses to represent scalar and vector types. +.. currentmodule:: cdsl.types .. autoclass:: ValueType .. inheritance-diagram:: ValueType ScalarType VectorType IntType FloatType BoolType :parts: 1 @@ -169,11 +170,9 @@ subclasses to represent scalar and vector types. .. autoclass:: BoolType :members: -.. automodule:: cretonne.types +.. automodule:: base.types :members: -.. currentmodule:: cretonne - There are no predefined vector types, but they can be created as needed with the :func:`ScalarType.by` function. @@ -181,6 +180,8 @@ the :func:`ScalarType.by` function. Instruction representation ========================== +.. currentmodule:: cretonne + The Rust in-memory representation of instructions is derived from the instruction descriptions. Part of the representation is generated, and part is written as Rust code in the `cretonne.instructions` module. The instruction diff --git a/lib/cretonne/meta/base/__init__.py b/lib/cretonne/meta/base/__init__.py new file mode 100644 index 0000000000..132b469ee6 --- /dev/null +++ b/lib/cretonne/meta/base/__init__.py @@ -0,0 +1 @@ +"""Definitions for the base Cretonne language.""" diff --git a/lib/cretonne/meta/cretonne/types.py b/lib/cretonne/meta/base/types.py similarity index 88% rename from lib/cretonne/meta/cretonne/types.py rename to lib/cretonne/meta/base/types.py index 05dcb53e16..a2eb1054b9 100644 --- a/lib/cretonne/meta/cretonne/types.py +++ b/lib/cretonne/meta/base/types.py @@ -1,8 +1,8 @@ """ -The cretonne.types module predefines all the Cretonne scalar types. +The base.types module predefines all the Cretonne scalar types. """ from __future__ import absolute_import -from . import ScalarType, IntType, FloatType, BoolType +from cdsl.types import ScalarType, IntType, FloatType, BoolType #: Boolean. b1 = ScalarType( diff --git a/lib/cretonne/meta/cdsl/__init__.py b/lib/cretonne/meta/cdsl/__init__.py new file mode 100644 index 0000000000..417f800950 --- /dev/null +++ b/lib/cretonne/meta/cdsl/__init__.py @@ -0,0 +1,6 @@ +""" +Cretonne DSL classes. + +This module defines the classes that are used to define Cretonne instructions +and other entitties. +""" diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py new file mode 100644 index 0000000000..3d5a5dcd38 --- /dev/null +++ b/lib/cretonne/meta/cdsl/types.py @@ -0,0 +1,160 @@ +"""Cretonne ValueType hierarchy""" +from __future__ import absolute_import +import math + + +# ValueType instances (i8, i32, ...) are provided in the cretonne.types module. +class ValueType(object): + """ + A concrete SSA value type. + + All SSA values have a type that is described by an instance of `ValueType` + or one of its subclasses. + """ + + # Map name -> ValueType. + _registry = dict() # type: Dict[str, ValueType] + + # List of all the scalar types. + all_scalars = list() # type: List[ValueType] + + def __init__(self, name, membytes, doc): + # type: (str, int, str) -> None + self.name = name + self.membytes = membytes + self.__doc__ = doc + assert name not in ValueType._registry + ValueType._registry[name] = self + + def __str__(self): + # type: () -> str + return self.name + + def free_typevar(self): + return None + + @staticmethod + def by_name(name): + # type: (str) -> ValueType + if name in ValueType._registry: + return ValueType._registry[name] + else: + raise AttributeError("No type named '{}'".format(name)) + + +class ScalarType(ValueType): + """ + A concrete scalar (not vector) type. + + Also tracks a unique set of :py:class:`VectorType` instances with this type + as the lane type. + """ + + def __init__(self, name, membytes, doc): + # type: (str, int, str) -> None + super(ScalarType, self).__init__(name, membytes, doc) + self._vectors = dict() # type: Dict[int, VectorType] + # Assign numbers starting from 1. (0 is VOID). + ValueType.all_scalars.append(self) + self.number = len(ValueType.all_scalars) + assert self.number < 16, 'Too many scalar types' + + def __repr__(self): + # type: () -> str + return 'ScalarType({})'.format(self.name) + + def rust_name(self): + # type: () -> str + return 'types::' + self.name.upper() + + def by(self, lanes): + # type: (int) -> VectorType + """ + Get a vector type with this type as the lane type. + + For example, ``i32.by(4)`` returns the :obj:`i32x4` type. + """ + if lanes in self._vectors: + return self._vectors[lanes] + else: + v = VectorType(self, lanes) + self._vectors[lanes] = v + return v + + +class VectorType(ValueType): + """ + A concrete SIMD vector type. + + A vector type has a lane type which is an instance of :class:`ScalarType`, + and a positive number of lanes. + """ + + def __init__(self, base, lanes): + # type: (ScalarType, int) -> None + assert isinstance(base, ScalarType), 'SIMD lanes must be scalar types' + super(VectorType, self).__init__( + name='{}x{}'.format(base.name, lanes), + membytes=lanes*base.membytes, + doc=""" + A SIMD vector with {} lanes containing a `{}` each. + """.format(lanes, base.name)) + self.base = base + self.lanes = lanes + self.number = 16*int(math.log(lanes, 2)) + base.number + + def __repr__(self): + # type: () -> str + return ('VectorType(base={}, lanes={})' + .format(self.base.name, self.lanes)) + + +class IntType(ScalarType): + """A concrete scalar integer type.""" + + def __init__(self, bits): + # type: (int) -> None + assert bits > 0, 'IntType must have positive number of bits' + super(IntType, self).__init__( + name='i{:d}'.format(bits), + membytes=bits // 8, + doc="An integer type with {} bits.".format(bits)) + self.bits = bits + + def __repr__(self): + # type: () -> str + return 'IntType(bits={})'.format(self.bits) + + +class FloatType(ScalarType): + """A concrete scalar floating point type.""" + + def __init__(self, bits, doc): + # type: (int, str) -> None + assert bits > 0, 'FloatType must have positive number of bits' + super(FloatType, self).__init__( + name='f{:d}'.format(bits), + membytes=bits // 8, + doc=doc) + self.bits = bits + + def __repr__(self): + # type: () -> str + return 'FloatType(bits={})'.format(self.bits) + + +class BoolType(ScalarType): + """A concrete scalar boolean type.""" + + def __init__(self, bits): + # type: (int) -> None + assert bits > 0, 'BoolType must have positive number of bits' + super(BoolType, self).__init__( + name='b{:d}'.format(bits), + membytes=bits // 8, + doc="A boolean type with {} bits.".format(bits)) + self.bits = bits + + def __repr__(self): + # type: () -> str + return 'BoolType(bits={})'.format(self.bits) diff --git a/lib/cretonne/meta/check.sh b/lib/cretonne/meta/check.sh index b0a6c853db..2077716eb7 100755 --- a/lib/cretonne/meta/check.sh +++ b/lib/cretonne/meta/check.sh @@ -14,7 +14,7 @@ runif() { # Check Python sources for Python 3 compatibility using pylint. # # Install pylint with 'pip install pylint'. -runif pylint --py3k --reports=no -- *.py cretonne isa +runif pylint --py3k --reports=no -- *.py cdsl base cretonne isa # Style linting. runif flake8 . diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index 7463f47d69..b1cc0322bd 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -6,10 +6,10 @@ instructions. """ from __future__ import absolute_import import re -import math import importlib from collections import OrderedDict from .predicates import And, Predicate, FieldPredicate # noqa +import cdsl.types # The typing module is only required by mypy, and we don't use these imports # outside type comments. @@ -17,7 +17,7 @@ try: from typing import Tuple, Union, Any, Iterable, Sequence, TYPE_CHECKING # noqa MaybeBoundInst = Union['Instruction', 'BoundInstruction'] AnyPredicate = Union['Predicate', 'FieldPredicate'] - OperandSpec = Union['OperandKind', 'ValueType', 'TypeVar'] + OperandSpec = Union['OperandKind', 'cdsl.types.ValueType', 'TypeVar'] except ImportError: TYPE_CHECKING = False @@ -391,171 +391,6 @@ class EntityRefKind(OperandKind): return 'EntityRefKind({})'.format(self.name) -# ValueType instances (i8, i32, ...) are provided in the cretonne.types module. -class ValueType(object): - """ - A concrete SSA value type. - - All SSA values have a type that is described by an instance of `ValueType` - or one of its subclasses. - """ - - # Map name -> ValueType. - _registry = dict() # type: Dict[str, ValueType] - - # List of all the scalar types. - all_scalars = list() # type: List[ValueType] - - def __init__(self, name, membytes, doc): - # type: (str, int, str) -> None - self.name = name - self.membytes = membytes - self.__doc__ = doc - assert name not in ValueType._registry - ValueType._registry[name] = self - - def __str__(self): - # type: () -> str - return self.name - - def operand_kind(self): - # type: () -> OperandKind - """ - When a `ValueType` object is used to describe the type of an `Operand` - in an instruction definition, the kind of that operand is an SSA value. - """ - return value - - def free_typevar(self): - return None - - @staticmethod - def by_name(name): - # type: (str) -> ValueType - if name in ValueType._registry: - return ValueType._registry[name] - else: - raise AttributeError("No type named '{}'".format(name)) - - -class ScalarType(ValueType): - """ - A concrete scalar (not vector) type. - - Also tracks a unique set of :py:class:`VectorType` instances with this type - as the lane type. - """ - - def __init__(self, name, membytes, doc): - # type: (str, int, str) -> None - super(ScalarType, self).__init__(name, membytes, doc) - self._vectors = dict() # type: Dict[int, VectorType] - # Assign numbers starting from 1. (0 is VOID). - ValueType.all_scalars.append(self) - self.number = len(ValueType.all_scalars) - assert self.number < 16, 'Too many scalar types' - - def __repr__(self): - # type: () -> str - return 'ScalarType({})'.format(self.name) - - def rust_name(self): - # type: () -> str - return 'types::' + self.name.upper() - - def by(self, lanes): - # type: (int) -> VectorType - """ - Get a vector type with this type as the lane type. - - For example, ``i32.by(4)`` returns the :obj:`i32x4` type. - """ - if lanes in self._vectors: - return self._vectors[lanes] - else: - v = VectorType(self, lanes) - self._vectors[lanes] = v - return v - - -class VectorType(ValueType): - """ - A concrete SIMD vector type. - - A vector type has a lane type which is an instance of :class:`ScalarType`, - and a positive number of lanes. - """ - - def __init__(self, base, lanes): - # type: (ScalarType, int) -> None - assert isinstance(base, ScalarType), 'SIMD lanes must be scalar types' - super(VectorType, self).__init__( - name='{}x{}'.format(base.name, lanes), - membytes=lanes*base.membytes, - doc=""" - A SIMD vector with {} lanes containing a `{}` each. - """.format(lanes, base.name)) - self.base = base - self.lanes = lanes - self.number = 16*int(math.log(lanes, 2)) + base.number - - def __repr__(self): - # type: () -> str - return ('VectorType(base={}, lanes={})' - .format(self.base.name, self.lanes)) - - -class IntType(ScalarType): - """A concrete scalar integer type.""" - - def __init__(self, bits): - # type: (int) -> None - assert bits > 0, 'IntType must have positive number of bits' - super(IntType, self).__init__( - name='i{:d}'.format(bits), - membytes=bits // 8, - doc="An integer type with {} bits.".format(bits)) - self.bits = bits - - def __repr__(self): - # type: () -> str - return 'IntType(bits={})'.format(self.bits) - - -class FloatType(ScalarType): - """A concrete scalar floating point type.""" - - def __init__(self, bits, doc): - # type: (int, str) -> None - assert bits > 0, 'FloatType must have positive number of bits' - super(FloatType, self).__init__( - name='f{:d}'.format(bits), - membytes=bits // 8, - doc=doc) - self.bits = bits - - def __repr__(self): - # type: () -> str - return 'FloatType(bits={})'.format(self.bits) - - -class BoolType(ScalarType): - """A concrete scalar boolean type.""" - - def __init__(self, bits): - # type: (int) -> None - assert bits > 0, 'BoolType must have positive number of bits' - super(BoolType, self).__init__( - name='b{:d}'.format(bits), - membytes=bits // 8, - doc="A boolean type with {} bits.".format(bits)) - self.bits = bits - - def __repr__(self): - # type: () -> str - return 'BoolType(bits={})'.format(self.bits) - - # Defining instructions. @@ -633,9 +468,12 @@ class Operand(object): def __init__(self, name, typ, doc=''): # type: (str, OperandSpec, str) -> None self.name = name - self.typ = typ self.__doc__ = doc - self.kind = typ.operand_kind() + self.typ = typ + if isinstance(typ, cdsl.types.ValueType): + self.kind = value + else: + self.kind = typ.operand_kind() def get_doc(self): # type: () -> str @@ -688,7 +526,7 @@ class InstructionFormat(object): """ # Map (multiple_results, kind, kind, ...) -> InstructionFormat - _registry = dict() # type: Dict[Tuple, InstructionFormat] + _registry = dict() # type: Dict[Tuple[bool, Tuple[OperandKind, ...]], InstructionFormat] # noqa # All existing formats. all_formats = list() # type: List[InstructionFormat] @@ -715,7 +553,7 @@ class InstructionFormat(object): self.typevar_operand = self.value_operands[0] # Compute a signature for the global registry. - sig = (self.multiple_results,) + self.kinds + sig = (self.multiple_results, self.kinds) if sig in InstructionFormat._registry: raise RuntimeError( "Format '{}' has the same signature as existing format '{}'" @@ -782,11 +620,11 @@ class InstructionFormat(object): multiple_results = outs[0].kind == variable_args else: multiple_results = len(outs) > 1 - sig = (multiple_results,) + tuple(op.kind for op in ins) + sig = (multiple_results, tuple(op.kind for op in ins)) if sig not in InstructionFormat._registry: raise RuntimeError( "No instruction format matches ins = ({}){}".format( - ", ".join(map(str, sig[1:])), + ", ".join(map(str, sig[1])), "[multiple results]" if multiple_results else "")) return InstructionFormat._registry[sig] @@ -991,7 +829,7 @@ class Instruction(object): return x def bind(self, *args): - # type: (*ValueType) -> BoundInstruction + # type: (*cdsl.types.ValueType) -> BoundInstruction """ Bind a polymorphic instruction to a concrete list of type variable values. @@ -1007,10 +845,10 @@ class Instruction(object): >>> iadd.i32 """ - return self.bind(ValueType.by_name(name)) + return self.bind(cdsl.types.ValueType.by_name(name)) def fully_bound(self): - # type: () -> Tuple[Instruction, Tuple[ValueType, ...]] + # type: () -> Tuple[Instruction, Tuple[cdsl.types.ValueType, ...]] """ Verify that all typevars have been bound, and return a `(inst, typevars)` pair. @@ -1036,7 +874,7 @@ class BoundInstruction(object): """ def __init__(self, inst, typevars): - # type: (Instruction, Tuple[ValueType, ...]) -> None + # type: (Instruction, Tuple[cdsl.types.ValueType, ...]) -> None self.inst = inst self.typevars = typevars assert len(typevars) <= 1 + len(inst.other_typevars) @@ -1045,7 +883,7 @@ class BoundInstruction(object): return '.'.join([self.inst.name, ] + list(map(str, self.typevars))) def bind(self, *args): - # type: (*ValueType) -> BoundInstruction + # type: (*cdsl.types.ValueType) -> BoundInstruction """ Bind additional typevars. """ @@ -1058,10 +896,10 @@ class BoundInstruction(object): >>> uext.i32.i8 """ - return self.bind(ValueType.by_name(name)) + return self.bind(cdsl.types.ValueType.by_name(name)) def fully_bound(self): - # type: () -> Tuple[Instruction, Tuple[ValueType, ...]] + # type: () -> Tuple[Instruction, Tuple[cdsl.types.ValueType, ...]] """ Verify that all typevars have been bound, and return a `(inst, typevars)` pair. diff --git a/lib/cretonne/meta/cretonne/base.py b/lib/cretonne/meta/cretonne/base.py index adbbf68d28..d991228872 100644 --- a/lib/cretonne/meta/cretonne/base.py +++ b/lib/cretonne/meta/cretonne/base.py @@ -7,7 +7,7 @@ support. from __future__ import absolute_import from . import Operand, Instruction, InstructionGroup, variable_args from .typevar import TypeVar -from .types import i8, f32, f64, b1 +from base.types import i8, f32, f64, b1 from .immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc from . import entities diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 569eca71b7..d7afcf1887 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -5,6 +5,7 @@ from __future__ import absolute_import import srcgen import constant_hash from unique_table import UniqueTable, UniqueSeqTable +import cdsl.types import cretonne @@ -310,11 +311,11 @@ def get_constraint(op, ctrl_typevar, type_sets): - `Free(idx)` where `idx` is an index into `type_sets`. - `Same`, `Lane`, `AsBool` for controlling typevar-derived constraints. """ + assert op.kind is cretonne.value t = op.typ - assert t.operand_kind() is cretonne.value # A concrete value type. - if isinstance(t, cretonne.ValueType): + if isinstance(t, cdsl.types.ValueType): return 'Concrete({})'.format(t.rust_name()) if t.free_typevar() is not ctrl_typevar: diff --git a/lib/cretonne/meta/gen_types.py b/lib/cretonne/meta/gen_types.py index 2b284d135b..5ffed14cf4 100644 --- a/lib/cretonne/meta/gen_types.py +++ b/lib/cretonne/meta/gen_types.py @@ -9,7 +9,8 @@ This ensures that Python and Rust use the same type numbering. """ from __future__ import absolute_import import srcgen -from cretonne import ValueType +from cdsl.types import ValueType +import base.types # noqa def emit_type(ty, fmt): From e6b959436d7a4c4be617359dec72231988c2c8ad Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 10:20:07 -0800 Subject: [PATCH 0425/3084] Split out predicates and settings. - cdsl.predicates defines classes for describing predicates. - cdsl.settings defines classes for describing settings. - base.settings defines shared settings. --- cranelift/docs/metaref.rst | 9 +- .../meta/{cretonne => base}/settings.py | 2 +- .../meta/{cretonne => cdsl}/predicates.py | 0 lib/cretonne/meta/cdsl/settings.py | 260 ++++++++++++++++++ lib/cretonne/meta/cretonne/__init__.py | 260 +----------------- lib/cretonne/meta/gen_settings.py | 4 +- lib/cretonne/meta/isa/riscv/recipes.py | 2 +- lib/cretonne/meta/isa/riscv/settings.py | 6 +- 8 files changed, 276 insertions(+), 267 deletions(-) rename lib/cretonne/meta/{cretonne => base}/settings.py (93%) rename lib/cretonne/meta/{cretonne => cdsl}/predicates.py (100%) create mode 100644 lib/cretonne/meta/cdsl/settings.py diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 6d8d8ad2b1..cfb39b00ad 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -28,20 +28,21 @@ process if anything in the :file:`lib/cretonne/meta` directory has changed since the last build. +.. module:: cdsl.settings + Settings ======== Settings are used by the environment embedding Cretonne to control the details of code generation. Each setting is defined in the meta language so a compact and consistent Rust representation can be generated. Shared settings are defined -in the :mod:`cretonne.settings` module. Some settings are specific to a target -ISA, and defined in a `settings` module under the appropriate +in the :mod:`base.settings` module. Some settings are specific to a target ISA, +and defined in a `settings` module under the appropriate :file:`lib/cretonne/meta/isa/*` directory. Settings can take boolean on/off values, small numbers, or explicitly enumerated symbolic values. Each type is represented by a sub-class of :class:`Setting`: -.. currentmodule:: cretonne .. inheritance-diagram:: Setting BoolSetting NumSetting EnumSetting :parts: 1 @@ -70,6 +71,8 @@ a module looks like this:: Instruction descriptions ======================== +.. currentmodule:: cretonne + New instructions are defined as instances of the :class:`Instruction` class. As instruction instances are created, they are added to the currently open :class:`InstructionGroup`. diff --git a/lib/cretonne/meta/cretonne/settings.py b/lib/cretonne/meta/base/settings.py similarity index 93% rename from lib/cretonne/meta/cretonne/settings.py rename to lib/cretonne/meta/base/settings.py index 522a649ef3..c6f87f7fbb 100644 --- a/lib/cretonne/meta/cretonne/settings.py +++ b/lib/cretonne/meta/base/settings.py @@ -4,7 +4,7 @@ Cretonne shared settings. This module defines settings are are relevant for all code generators. """ from __future__ import absolute_import -from . import SettingGroup, BoolSetting, EnumSetting +from cdsl.settings import SettingGroup, BoolSetting, EnumSetting group = SettingGroup('shared') diff --git a/lib/cretonne/meta/cretonne/predicates.py b/lib/cretonne/meta/cdsl/predicates.py similarity index 100% rename from lib/cretonne/meta/cretonne/predicates.py rename to lib/cretonne/meta/cdsl/predicates.py diff --git a/lib/cretonne/meta/cdsl/settings.py b/lib/cretonne/meta/cdsl/settings.py new file mode 100644 index 0000000000..1c1e31643c --- /dev/null +++ b/lib/cretonne/meta/cdsl/settings.py @@ -0,0 +1,260 @@ +"""Classes for describing settings and groups of settings.""" +from __future__ import absolute_import +from collections import OrderedDict +from .predicates import Predicate + + +class Setting(object): + """ + A named setting variable that can be configured externally to Cretonne. + + Settings are normally not named when they are created. They get their name + from the `extract_names` method. + """ + + def __init__(self, doc): + self.name = None # Assigned later by `extract_names()`. + self.__doc__ = doc + # Offset of byte in settings vector containing this setting. + self.byte_offset = None + self.group = SettingGroup.append(self) + + def __str__(self): + return '{}.{}'.format(self.group.name, self.name) + + def predicate_context(self): + """ + Return the context where this setting can be evaluated as a (leaf) + predicate. + """ + return self.group + + def predicate_leafs(self, leafs): + leafs.add(self) + + +class BoolSetting(Setting): + """ + A named setting with a boolean on/off value. + + :param doc: Documentation string. + :param default: The default value of this setting. + """ + + def __init__(self, doc, default=False): + super(BoolSetting, self).__init__(doc) + self.default = default + + def default_byte(self): + """ + Get the default value of this setting, as a byte that can be bitwise + or'ed with the other booleans sharing the same byte. + """ + if self.default: + return 1 << self.bit_offset + else: + return 0 + + def rust_predicate(self, prec): + """ + Return the Rust code to compute the value of this setting. + + The emitted code assumes that the setting group exists as a local + variable. + """ + return '{}.{}()'.format(self.group.name, self.name) + + +class NumSetting(Setting): + """ + A named setting with an integral value in the range 0--255. + + :param doc: Documentation string. + :param default: The default value of this setting. + """ + + def __init__(self, doc, default=0): + super(NumSetting, self).__init__(doc) + assert default == int(default) + assert default >= 0 and default <= 255 + self.default = default + + def default_byte(self): + return self.default + + +class EnumSetting(Setting): + """ + A named setting with an enumerated set of possible values. + + The default value is always the first enumerator. + + :param doc: Documentation string. + :param args: Tuple of unique strings representing the possible values. + """ + + def __init__(self, doc, *args): + super(EnumSetting, self).__init__(doc) + assert len(args) > 0, "EnumSetting must have at least one value" + self.values = tuple(str(x) for x in args) + self.default = self.values[0] + + def default_byte(self): + return 0 + + +class SettingGroup(object): + """ + A group of settings. + + Whenever a :class:`Setting` object is created, it is added to the currently + open group. A setting group must be closed explicitly before another can be + opened. + + :param name: Short mnemonic name for setting group. + :param parent: Parent settings group. + """ + + # The currently open setting group. + _current = None # type: SettingGroup + + def __init__(self, name, parent=None): + self.name = name + self.parent = parent + self.settings = [] + # Named predicates computed from settings in this group or its + # parents. + self.named_predicates = [] + # All boolean predicates that can be accessed by number. This includes: + # - All boolean settings in this group. + # - All named predicates. + # - Added anonymous predicates, see `number_predicate()`. + # - Added parent predicates that are replicated in this group. + # Maps predicate -> number. + self.predicate_number = OrderedDict() + + self.open() + + def open(self): + """ + Open this setting group such that future new settings are added to this + group. + """ + assert SettingGroup._current is None, ( + "Can't open {} since {} is already open" + .format(self, SettingGroup._current)) + SettingGroup._current = self + + def close(self, globs=None): + """ + Close this setting group. This function must be called before opening + another setting group. + + :param globs: Pass in `globals()` to run `extract_names` on all + settings defined in the module. + """ + assert SettingGroup._current is self, ( + "Can't close {}, the open setting group is {}" + .format(self, SettingGroup._current)) + SettingGroup._current = None + if globs: + for name, obj in globs.items(): + if isinstance(obj, Setting): + assert obj.name is None, obj.name + obj.name = name + if isinstance(obj, Predicate): + assert obj.name is None + obj.name = name + self.named_predicates.append(obj) + self.layout() + + @staticmethod + def append(setting): + g = SettingGroup._current + assert g, "Open a setting group before defining settings." + g.settings.append(setting) + return g + + def number_predicate(self, pred): + """ + Make sure that `pred` has an assigned number, and will be included in + this group's bit vector. + + The numbered predicates include: + - `BoolSetting` settings that belong to this group. + - `Predicate` instances in `named_predicates`. + - `Predicate` instances without a name. + - Settings or computed predicates that belong to the parent group, but + need to be accessible by number in this group. + + The numbered predicates are referenced by the encoding tables as ISA + predicates. See the `isap` field on `Encoding`. + + :returns: The assigned predicate number in this group. + """ + if pred in self.predicate_number: + return self.predicate_number[pred] + else: + number = len(self.predicate_number) + self.predicate_number[pred] = number + return number + + def layout(self): + """ + Compute the layout of the byte vector used to represent this settings + group. + + The byte vector contains the following entries in order: + + 1. Byte-sized settings like `NumSetting` and `EnumSetting`. + 2. `BoolSetting` settings. + 3. Precomputed named predicates. + 4. Other numbered predicates, including anonymous predicates and parent + predicates that need to be accessible by number. + + Set `self.settings_size` to the length of the byte vector prefix that + contains the settings. All bytes after that are computed, not + configured. + + Set `self.boolean_offset` to the beginning of the numbered predicates, + 2. in the list above. + + Assign `byte_offset` and `bit_offset` fields in all settings. + + After calling this method, no more settings can be added, but + additional predicates can be made accessible with `number_predicate()`. + """ + assert len(self.predicate_number) == 0, "Too late for layout" + + # Assign the non-boolean settings. + byte_offset = 0 + for s in self.settings: + if not isinstance(s, BoolSetting): + s.byte_offset = byte_offset + byte_offset += 1 + + # Then the boolean settings. + self.boolean_offset = byte_offset + for s in self.settings: + if isinstance(s, BoolSetting): + number = self.number_predicate(s) + s.byte_offset = byte_offset + number // 8 + s.bit_offset = number % 8 + + # This is the end of the settings. Round up to a whole number of bytes. + self.boolean_settings = len(self.predicate_number) + self.settings_size = self.byte_size() + + # Now assign numbers to all our named predicates. + for p in self.named_predicates: + self.number_predicate(p) + + def byte_size(self): + """ + Compute the number of bytes required to hold all settings and + precomputed predicates. + + This is the size of the byte-sized settings plus all the numbered + predcate bits rounded up to a whole number of bytes. + """ + return self.boolean_offset + (len(self.predicate_number) + 7) // 8 diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index b1cc0322bd..d2d4386de5 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -7,14 +7,14 @@ instructions. from __future__ import absolute_import import re import importlib -from collections import OrderedDict -from .predicates import And, Predicate, FieldPredicate # noqa +from cdsl.predicates import And import cdsl.types # The typing module is only required by mypy, and we don't use these imports # outside type comments. try: from typing import Tuple, Union, Any, Iterable, Sequence, TYPE_CHECKING # noqa + from cdsl.predicates import Predicate, FieldPredicate # noqa MaybeBoundInst = Union['Instruction', 'BoundInstruction'] AnyPredicate = Union['Predicate', 'FieldPredicate'] OperandSpec = Union['OperandKind', 'cdsl.types.ValueType', 'TypeVar'] @@ -34,262 +34,6 @@ def camel_case(s): return camel_re.sub(lambda m: m.group(2).upper(), s) -class Setting(object): - """ - A named setting variable that can be configured externally to Cretonne. - - Settings are normally not named when they are created. They get their name - from the `extract_names` method. - """ - - def __init__(self, doc): - self.name = None # Assigned later by `extract_names()`. - self.__doc__ = doc - # Offset of byte in settings vector containing this setting. - self.byte_offset = None - self.group = SettingGroup.append(self) - - def __str__(self): - return '{}.{}'.format(self.group.name, self.name) - - def predicate_context(self): - """ - Return the context where this setting can be evaluated as a (leaf) - predicate. - """ - return self.group - - def predicate_leafs(self, leafs): - leafs.add(self) - - -class BoolSetting(Setting): - """ - A named setting with a boolean on/off value. - - :param doc: Documentation string. - :param default: The default value of this setting. - """ - - def __init__(self, doc, default=False): - super(BoolSetting, self).__init__(doc) - self.default = default - - def default_byte(self): - """ - Get the default value of this setting, as a byte that can be bitwise - or'ed with the other booleans sharing the same byte. - """ - if self.default: - return 1 << self.bit_offset - else: - return 0 - - def rust_predicate(self, prec): - """ - Return the Rust code to compute the value of this setting. - - The emitted code assumes that the setting group exists as a local - variable. - """ - return '{}.{}()'.format(self.group.name, self.name) - - -class NumSetting(Setting): - """ - A named setting with an integral value in the range 0--255. - - :param doc: Documentation string. - :param default: The default value of this setting. - """ - - def __init__(self, doc, default=0): - super(NumSetting, self).__init__(doc) - assert default == int(default) - assert default >= 0 and default <= 255 - self.default = default - - def default_byte(self): - return self.default - - -class EnumSetting(Setting): - """ - A named setting with an enumerated set of possible values. - - The default value is always the first enumerator. - - :param doc: Documentation string. - :param args: Tuple of unique strings representing the possible values. - """ - - def __init__(self, doc, *args): - super(EnumSetting, self).__init__(doc) - assert len(args) > 0, "EnumSetting must have at least one value" - self.values = tuple(str(x) for x in args) - self.default = self.values[0] - - def default_byte(self): - return 0 - - -class SettingGroup(object): - """ - A group of settings. - - Whenever a :class:`Setting` object is created, it is added to the currently - open group. A setting group must be closed explicitly before another can be - opened. - - :param name: Short mnemonic name for setting group. - :param parent: Parent settings group. - """ - - # The currently open setting group. - _current = None # type: SettingGroup - - def __init__(self, name, parent=None): - self.name = name - self.parent = parent - self.settings = [] - # Named predicates computed from settings in this group or its - # parents. - self.named_predicates = [] - # All boolean predicates that can be accessed by number. This includes: - # - All boolean settings in this group. - # - All named predicates. - # - Added anonymous predicates, see `number_predicate()`. - # - Added parent predicates that are replicated in this group. - # Maps predicate -> number. - self.predicate_number = OrderedDict() - - self.open() - - def open(self): - """ - Open this setting group such that future new settings are added to this - group. - """ - assert SettingGroup._current is None, ( - "Can't open {} since {} is already open" - .format(self, SettingGroup._current)) - SettingGroup._current = self - - def close(self, globs=None): - """ - Close this setting group. This function must be called before opening - another setting group. - - :param globs: Pass in `globals()` to run `extract_names` on all - settings defined in the module. - """ - assert SettingGroup._current is self, ( - "Can't close {}, the open setting group is {}" - .format(self, SettingGroup._current)) - SettingGroup._current = None - if globs: - for name, obj in globs.items(): - if isinstance(obj, Setting): - assert obj.name is None, obj.name - obj.name = name - if isinstance(obj, Predicate): - assert obj.name is None - obj.name = name - self.named_predicates.append(obj) - self.layout() - - @staticmethod - def append(setting): - g = SettingGroup._current - assert g, "Open a setting group before defining settings." - g.settings.append(setting) - return g - - def number_predicate(self, pred): - """ - Make sure that `pred` has an assigned number, and will be included in - this group's bit vector. - - The numbered predicates include: - - `BoolSetting` settings that belong to this group. - - `Predicate` instances in `named_predicates`. - - `Predicate` instances without a name. - - Settings or computed predicates that belong to the parent group, but - need to be accessible by number in this group. - - The numbered predicates are referenced by the encoding tables as ISA - predicates. See the `isap` field on `Encoding`. - - :returns: The assigned predicate number in this group. - """ - if pred in self.predicate_number: - return self.predicate_number[pred] - else: - number = len(self.predicate_number) - self.predicate_number[pred] = number - return number - - def layout(self): - """ - Compute the layout of the byte vector used to represent this settings - group. - - The byte vector contains the following entries in order: - - 1. Byte-sized settings like `NumSetting` and `EnumSetting`. - 2. `BoolSetting` settings. - 3. Precomputed named predicates. - 4. Other numbered predicates, including anonymous predicates and parent - predicates that need to be accessible by number. - - Set `self.settings_size` to the length of the byte vector prefix that - contains the settings. All bytes after that are computed, not - configured. - - Set `self.boolean_offset` to the beginning of the numbered predicates, - 2. in the list above. - - Assign `byte_offset` and `bit_offset` fields in all settings. - - After calling this method, no more settings can be added, but - additional predicates can be made accessible with `number_predicate()`. - """ - assert len(self.predicate_number) == 0, "Too late for layout" - - # Assign the non-boolean settings. - byte_offset = 0 - for s in self.settings: - if not isinstance(s, BoolSetting): - s.byte_offset = byte_offset - byte_offset += 1 - - # Then the boolean settings. - self.boolean_offset = byte_offset - for s in self.settings: - if isinstance(s, BoolSetting): - number = self.number_predicate(s) - s.byte_offset = byte_offset + number // 8 - s.bit_offset = number % 8 - - # This is the end of the settings. Round up to a whole number of bytes. - self.boolean_settings = len(self.predicate_number) - self.settings_size = self.byte_size() - - # Now assign numbers to all our named predicates. - for p in self.named_predicates: - self.number_predicate(p) - - def byte_size(self): - """ - Compute the number of bytes required to hold all settings and - precomputed predicates. - - This is the size of the byte-sized settings plus all the numbered - predcate bits rounded up to a whole number of bytes. - """ - return self.boolean_offset + (len(self.predicate_number) + 7) // 8 - - # Kinds of operands. # # Each instruction has an opcode and a number of operands. The opcode diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index 8e35d05909..6a05d1fa03 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -5,7 +5,9 @@ from __future__ import absolute_import import srcgen from unique_table import UniqueSeqTable import constant_hash -from cretonne import camel_case, BoolSetting, NumSetting, EnumSetting, settings +from cdsl.settings import BoolSetting, NumSetting, EnumSetting +from cretonne import camel_case +from base import settings def gen_enum_types(sgrp, fmt): diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 375b0e2150..5818bd3fd9 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -11,7 +11,7 @@ instruction formats described in the reference: from __future__ import absolute_import from cretonne import EncRecipe from cretonne.formats import Binary, BinaryImm -from cretonne.predicates import IsSignedInt +from cdsl.predicates import IsSignedInt # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit # instructions have 11 as the two low bits, with bits 6:2 determining the base diff --git a/lib/cretonne/meta/isa/riscv/settings.py b/lib/cretonne/meta/isa/riscv/settings.py index 8aae295342..7571bf1611 100644 --- a/lib/cretonne/meta/isa/riscv/settings.py +++ b/lib/cretonne/meta/isa/riscv/settings.py @@ -2,9 +2,9 @@ RISC-V settings. """ from __future__ import absolute_import -from cretonne import SettingGroup, BoolSetting -from cretonne.predicates import And -import cretonne.settings as shared +from cdsl.settings import SettingGroup, BoolSetting +from cdsl.predicates import And +import base.settings as shared from .defs import isa isa.settings = SettingGroup('riscv', parent=shared.group) From 19287c1f32278289db4eff86bec7e319f5746c9a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 10:58:23 -0800 Subject: [PATCH 0426/3084] Split out operand descriptions. - cdsl.operands has the Operand and OperandKind classes. --- cranelift/docs/metaref.rst | 14 +-- lib/cretonne/meta/cdsl/__init__.py | 11 ++ lib/cretonne/meta/cdsl/operands.py | 104 +++++++++++++++++++ lib/cretonne/meta/cretonne/__init__.py | 123 ++--------------------- lib/cretonne/meta/cretonne/base.py | 9 +- lib/cretonne/meta/cretonne/entities.py | 2 +- lib/cretonne/meta/cretonne/formats.py | 43 ++++---- lib/cretonne/meta/cretonne/immediates.py | 2 +- lib/cretonne/meta/cretonne/typevar.py | 6 +- lib/cretonne/meta/gen_instr.py | 9 +- 10 files changed, 167 insertions(+), 156 deletions(-) create mode 100644 lib/cretonne/meta/cdsl/operands.py diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index cfb39b00ad..4699a80a43 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -127,13 +127,14 @@ There are some practical restrictions on the use of type variables, see Immediate operands ------------------ +.. currentmodule:: cdsl.operands + Immediate instruction operands don't correspond to SSA values, but have values that are encoded directly in the instruction. Immediate operands don't have types from the :class:`cdsl.types.ValueType` type system; they often have enumerated values of a specific type. The type of an immediate operand is indicated with an instance of :class:`ImmediateKind`. -.. currentmodule:: cretonne .. autoclass:: ImmediateKind .. automodule:: cretonne.immediates @@ -145,8 +146,7 @@ Entity references Instruction operands can also refer to other entities in the same function. This can be extended basic blocks, or entities declared in the function preamble. -.. currentmodule:: cretonne - +.. currentmodule:: cdsl.operands .. autoclass:: EntityRefKind .. automodule:: cretonne.entities @@ -183,7 +183,7 @@ the :func:`ScalarType.by` function. Instruction representation ========================== -.. currentmodule:: cretonne +.. module:: cdsl.operands The Rust in-memory representation of instructions is derived from the instruction descriptions. Part of the representation is generated, and part is @@ -198,8 +198,10 @@ Since all SSA value operands are represented as a `Value` in Rust code, value types don't affect the representation. Two special operand kinds are used to represent SSA values: -.. autodata:: value -.. autodata:: variable_args +.. autodata:: VALUE +.. autodata:: VARIABLE_ARGS + +.. currentmodule:: cretonne When an instruction description is created, it is automatically assigned a predefined instruction format which is an instance of diff --git a/lib/cretonne/meta/cdsl/__init__.py b/lib/cretonne/meta/cdsl/__init__.py index 417f800950..247b970675 100644 --- a/lib/cretonne/meta/cdsl/__init__.py +++ b/lib/cretonne/meta/cdsl/__init__.py @@ -4,3 +4,14 @@ Cretonne DSL classes. This module defines the classes that are used to define Cretonne instructions and other entitties. """ +from __future__ import absolute_import +import re + + +camel_re = re.compile('(^|_)([a-z])') + + +def camel_case(s): + # type: (str) -> str + """Convert the string s to CamelCase""" + return camel_re.sub(lambda m: m.group(2).upper(), s) diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py new file mode 100644 index 0000000000..33157d9e23 --- /dev/null +++ b/lib/cretonne/meta/cdsl/operands.py @@ -0,0 +1,104 @@ +"""Classes for describing instruction operands.""" +from __future__ import absolute_import +from . import camel_case + + +# Kinds of operands. +# +# Each instruction has an opcode and a number of operands. The opcode +# determines the instruction format, and the format determines the number of +# operands and the kind of each operand. +class OperandKind(object): + """ + An instance of the `OperandKind` class corresponds to a kind of operand. + Each operand kind has a corresponding type in the Rust representation of an + instruction. + """ + + def __init__(self, name, doc, default_member=None, rust_type=None): + # type: (str, str, str, str) -> None + self.name = name + self.__doc__ = doc + self.default_member = default_member + # The camel-cased name of an operand kind is also the Rust type used to + # represent it. + self.rust_type = rust_type or camel_case(name) + + def __str__(self): + # type: () -> str + return self.name + + def __repr__(self): + # type: () -> str + return 'OperandKind({})'.format(self.name) + + def operand_kind(self): + # type: () -> OperandKind + """ + An `OperandKind` instance can be used directly as the type of an + `Operand` when defining an instruction. + """ + return self + + def free_typevar(self): + # Return the free typevariable controlling the type of this operand. + return None + +#: An SSA value operand. This is a value defined by another instruction. +VALUE = OperandKind( + 'value', """ + An SSA value defined by another instruction. + + This kind of operand can represent any SSA value type, but the + instruction format may restrict the valid value types for a given + operand. + """) + +#: A variable-sized list of value operands. Use for Ebb and function call +#: arguments. +VARIABLE_ARGS = OperandKind( + 'variable_args', """ + A variable size list of `value` operands. + + Use this to represent arguemtns passed to a function call, arguments + passed to an extended basic block, or a variable number of results + returned from an instruction. + """, + default_member='varargs') + + +# Instances of immediate operand types are provided in the +# `cretonne.immediates` module. +class ImmediateKind(OperandKind): + """ + The kind of an immediate instruction operand. + + :param default_member: The default member name of this kind the + `InstructionData` data structure. + """ + + def __init__(self, name, doc, default_member='imm', rust_type=None): + # type: (str, str, str, str) -> None + super(ImmediateKind, self).__init__( + name, doc, default_member, rust_type) + + def __repr__(self): + # type: () -> str + return 'ImmediateKind({})'.format(self.name) + + +# Instances of entity reference operand types are provided in the +# `cretonne.entities` module. +class EntityRefKind(OperandKind): + """ + The kind of an entity reference instruction operand. + """ + + def __init__(self, name, doc, default_member=None, rust_type=None): + # type: (str, str, str, str) -> None + super(EntityRefKind, self).__init__( + name, doc, default_member or name, rust_type) + + def __repr__(self): + # type: () -> str + return 'EntityRefKind({})'.format(self.name) diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index d2d4386de5..d6b9835443 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -5,10 +5,11 @@ This module provides classes and functions used to describe Cretonne instructions. """ from __future__ import absolute_import -import re import importlib +from cdsl import camel_case from cdsl.predicates import And import cdsl.types +from cdsl.operands import VALUE, VARIABLE_ARGS, OperandKind # The typing module is only required by mypy, and we don't use these imports # outside type comments. @@ -25,116 +26,6 @@ if TYPE_CHECKING: from .typevar import TypeVar # noqa -camel_re = re.compile('(^|_)([a-z])') - - -def camel_case(s): - # type: (str) -> str - """Convert the string s to CamelCase""" - return camel_re.sub(lambda m: m.group(2).upper(), s) - - -# Kinds of operands. -# -# Each instruction has an opcode and a number of operands. The opcode -# determines the instruction format, and the format determines the number of -# operands and the kind of each operand. -class OperandKind(object): - """ - An instance of the `OperandKind` class corresponds to a kind of operand. - Each operand kind has a corresponding type in the Rust representation of an - instruction. - """ - - def __init__(self, name, doc, default_member=None, rust_type=None): - # type: (str, str, str, str) -> None - self.name = name - self.__doc__ = doc - self.default_member = default_member - # The camel-cased name of an operand kind is also the Rust type used to - # represent it. - self.rust_type = rust_type or camel_case(name) - - def __str__(self): - # type: () -> str - return self.name - - def __repr__(self): - # type: () -> str - return 'OperandKind({})'.format(self.name) - - def operand_kind(self): - # type: () -> OperandKind - """ - An `OperandKind` instance can be used directly as the type of an - `Operand` when defining an instruction. - """ - return self - - def free_typevar(self): - # Return the free typevariable controlling the type of this operand. - return None - -#: An SSA value operand. This is a value defined by another instruction. -value = OperandKind( - 'value', """ - An SSA value defined by another instruction. - - This kind of operand can represent any SSA value type, but the - instruction format may restrict the valid value types for a given - operand. - """) - -#: A variable-sized list of value operands. Use for Ebb and function call -#: arguments. -variable_args = OperandKind( - 'variable_args', """ - A variable size list of `value` operands. - - Use this to represent arguemtns passed to a function call, arguments - passed to an extended basic block, or a variable number of results - returned from an instruction. - """, - default_member='varargs') - - -# Instances of immediate operand types are provided in the -# `cretonne.immediates` module. -class ImmediateKind(OperandKind): - """ - The kind of an immediate instruction operand. - - :param default_member: The default member name of this kind the - `InstructionData` data structure. - """ - - def __init__(self, name, doc, default_member='imm', rust_type=None): - # type: (str, str, str, str) -> None - super(ImmediateKind, self).__init__( - name, doc, default_member, rust_type) - - def __repr__(self): - # type: () -> str - return 'ImmediateKind({})'.format(self.name) - - -# Instances of entity reference operand types are provided in the -# `cretonne.entities` module. -class EntityRefKind(OperandKind): - """ - The kind of an entity reference instruction operand. - """ - - def __init__(self, name, doc, default_member=None, rust_type=None): - # type: (str, str, str, str) -> None - super(EntityRefKind, self).__init__( - name, doc, default_member or name, rust_type) - - def __repr__(self): - # type: () -> str - return 'EntityRefKind({})'.format(self.name) - - # Defining instructions. @@ -215,7 +106,7 @@ class Operand(object): self.__doc__ = doc self.typ = typ if isinstance(typ, cdsl.types.ValueType): - self.kind = value + self.kind = VALUE else: self.kind = typ.operand_kind() @@ -235,7 +126,7 @@ class Operand(object): """ Is this an SSA value operand? """ - return self.kind is value + return self.kind is cdsl.operands.VALUE class InstructionFormat(object): @@ -285,12 +176,12 @@ class InstructionFormat(object): # Which of self.kinds are `value`? self.value_operands = tuple( - i for i, k in enumerate(self.kinds) if k is value) + i for i, k in enumerate(self.kinds) if k is VALUE) # The typevar_operand argument must point to a 'value' operand. self.typevar_operand = kwargs.get('typevar_operand', None) # type: int if self.typevar_operand is not None: - assert self.kinds[self.typevar_operand] is value, \ + assert self.kinds[self.typevar_operand] is VALUE, \ "typevar_operand must indicate a 'value' operand" elif len(self.value_operands) > 0: # Default to the first 'value' operand, if there is one. @@ -361,7 +252,7 @@ class InstructionFormat(object): tuples of :py:`Operand` objects. """ if len(outs) == 1: - multiple_results = outs[0].kind == variable_args + multiple_results = outs[0].kind == VARIABLE_ARGS else: multiple_results = len(outs) > 1 sig = (multiple_results, tuple(op.kind for op in ins)) diff --git a/lib/cretonne/meta/cretonne/base.py b/lib/cretonne/meta/cretonne/base.py index d991228872..e2ceadaff4 100644 --- a/lib/cretonne/meta/cretonne/base.py +++ b/lib/cretonne/meta/cretonne/base.py @@ -5,7 +5,8 @@ This module defines the basic Cretonne instruction set that all targets support. """ from __future__ import absolute_import -from . import Operand, Instruction, InstructionGroup, variable_args +from cdsl.operands import VARIABLE_ARGS +from . import Operand, Instruction, InstructionGroup from .typevar import TypeVar from base.types import i8, f32, f64, b1 from .immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc @@ -31,7 +32,7 @@ Any = TypeVar( # c = Operand('c', Testable, doc='Controlling value to test') EBB = Operand('EBB', entities.ebb, doc='Destination extended basic block') -args = Operand('args', variable_args, doc='EBB arguments') +args = Operand('args', VARIABLE_ARGS, doc='EBB arguments') jump = Instruction( 'jump', r""" @@ -98,7 +99,7 @@ trapnz = Instruction( """, ins=c) -rvals = Operand('rvals', variable_args, doc='return values') +rvals = Operand('rvals', VARIABLE_ARGS, doc='return values') x_return = Instruction( 'return', r""" @@ -114,7 +115,7 @@ FN = Operand( 'FN', entities.func_ref, doc='function to call, declared by :inst:`function`') -args = Operand('args', variable_args, doc='call arguments') +args = Operand('args', VARIABLE_ARGS, doc='call arguments') call = Instruction( 'call', r""" diff --git a/lib/cretonne/meta/cretonne/entities.py b/lib/cretonne/meta/cretonne/entities.py index afeeea7e98..327d45e8bc 100644 --- a/lib/cretonne/meta/cretonne/entities.py +++ b/lib/cretonne/meta/cretonne/entities.py @@ -4,7 +4,7 @@ operand types. There are corresponding definitions in the `cretonne.entities` Rust module. """ from __future__ import absolute_import -from . import EntityRefKind +from cdsl.operands import EntityRefKind #: A reference to an extended basic block in the same function. diff --git a/lib/cretonne/meta/cretonne/formats.py b/lib/cretonne/meta/cretonne/formats.py index cb8aed3f98..a4d012020d 100644 --- a/lib/cretonne/meta/cretonne/formats.py +++ b/lib/cretonne/meta/cretonne/formats.py @@ -6,51 +6,52 @@ Rust representation of cretonne IL, so all instruction formats must be defined in this module. """ from __future__ import absolute_import -from . import InstructionFormat, value, variable_args +from cdsl.operands import VALUE, VARIABLE_ARGS +from . import InstructionFormat from .immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc from .entities import ebb, sig_ref, func_ref, jump_table Nullary = InstructionFormat() -Unary = InstructionFormat(value) +Unary = InstructionFormat(VALUE) UnaryImm = InstructionFormat(imm64) UnaryIeee32 = InstructionFormat(ieee32) UnaryIeee64 = InstructionFormat(ieee64) UnaryImmVector = InstructionFormat(immvector, boxed_storage=True) -UnarySplit = InstructionFormat(value, multiple_results=True) +UnarySplit = InstructionFormat(VALUE, multiple_results=True) -Binary = InstructionFormat(value, value) -BinaryImm = InstructionFormat(value, imm64) -BinaryImmRev = InstructionFormat(imm64, value) +Binary = InstructionFormat(VALUE, VALUE) +BinaryImm = InstructionFormat(VALUE, imm64) +BinaryImmRev = InstructionFormat(imm64, VALUE) # Generate result + overflow flag. -BinaryOverflow = InstructionFormat(value, value, multiple_results=True) +BinaryOverflow = InstructionFormat(VALUE, VALUE, multiple_results=True) -# The select instructions are controlled by the second value operand. -# The first value operand is the controlling flag which has a derived type. +# The select instructions are controlled by the second VALUE operand. +# The first VALUE operand is the controlling flag which has a derived type. # The fma instruction has the same constraint on all inputs. -Ternary = InstructionFormat(value, value, value, typevar_operand=1) +Ternary = InstructionFormat(VALUE, VALUE, VALUE, typevar_operand=1) # Carry in *and* carry out for `iadd_carry` and friends. TernaryOverflow = InstructionFormat( - value, value, value, multiple_results=True, boxed_storage=True) + VALUE, VALUE, VALUE, multiple_results=True, boxed_storage=True) -InsertLane = InstructionFormat(value, ('lane', uimm8), value) -ExtractLane = InstructionFormat(value, ('lane', uimm8)) +InsertLane = InstructionFormat(VALUE, ('lane', uimm8), VALUE) +ExtractLane = InstructionFormat(VALUE, ('lane', uimm8)) -IntCompare = InstructionFormat(intcc, value, value) -FloatCompare = InstructionFormat(floatcc, value, value) +IntCompare = InstructionFormat(intcc, VALUE, VALUE) +FloatCompare = InstructionFormat(floatcc, VALUE, VALUE) -Jump = InstructionFormat(ebb, variable_args, boxed_storage=True) -Branch = InstructionFormat(value, ebb, variable_args, boxed_storage=True) -BranchTable = InstructionFormat(value, jump_table) +Jump = InstructionFormat(ebb, VARIABLE_ARGS, boxed_storage=True) +Branch = InstructionFormat(VALUE, ebb, VARIABLE_ARGS, boxed_storage=True) +BranchTable = InstructionFormat(VALUE, jump_table) Call = InstructionFormat( - func_ref, variable_args, multiple_results=True, boxed_storage=True) + func_ref, VARIABLE_ARGS, multiple_results=True, boxed_storage=True) IndirectCall = InstructionFormat( - sig_ref, value, variable_args, + sig_ref, VALUE, VARIABLE_ARGS, multiple_results=True, boxed_storage=True) -Return = InstructionFormat(variable_args, boxed_storage=True) +Return = InstructionFormat(VARIABLE_ARGS, boxed_storage=True) # Finally extract the names of global variables in this module. diff --git a/lib/cretonne/meta/cretonne/immediates.py b/lib/cretonne/meta/cretonne/immediates.py index 34b81dc79d..565dd26850 100644 --- a/lib/cretonne/meta/cretonne/immediates.py +++ b/lib/cretonne/meta/cretonne/immediates.py @@ -3,7 +3,7 @@ The `cretonne.immediates` module predefines all the Cretonne immediate operand types. """ from __future__ import absolute_import -from . import ImmediateKind +from cdsl.operands import ImmediateKind #: A 64-bit immediate integer operand. #: diff --git a/lib/cretonne/meta/cretonne/typevar.py b/lib/cretonne/meta/cretonne/typevar.py index a6d8d9637f..7f58ce73c1 100644 --- a/lib/cretonne/meta/cretonne/typevar.py +++ b/lib/cretonne/meta/cretonne/typevar.py @@ -6,7 +6,7 @@ polymorphic by using type variables. """ from __future__ import absolute_import import math -import cretonne +import cdsl.operands try: from typing import Tuple, Union # noqa @@ -315,11 +315,11 @@ class TypeVar(object): return TypeVar(None, None, base=self, derived_func='DoubleWidth') def operand_kind(self): - # type: () -> cretonne.OperandKind + # type: () -> cdsl.operands.OperandKind # When a `TypeVar` object is used to describe the type of an `Operand` # in an instruction definition, the kind of that operand is an SSA # value. - return cretonne.value # type: ignore + return cdsl.operands.VALUE def free_typevar(self): # type: () -> TypeVar diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index d7afcf1887..aae46b5652 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -6,6 +6,7 @@ import srcgen import constant_hash from unique_table import UniqueTable, UniqueSeqTable import cdsl.types +import cdsl.operands import cretonne @@ -56,7 +57,7 @@ def gen_arguments_method(fmt, is_mut): with fmt.indented('match *self {', '}'): for f in cretonne.InstructionFormat.all_formats: n = 'InstructionData::' + f.name - has_varargs = cretonne.variable_args in f.kinds + has_varargs = cdsl.operands.VARIABLE_ARGS in f.kinds # Formats with both fixed and variable arguments delegate to # the data struct. We need to work around borrow checker quirks # when extracting two mutable references. @@ -84,7 +85,7 @@ def gen_arguments_method(fmt, is_mut): capture = 'ref {}args, '.format(mut) arg = 'args' # Varargs. - if cretonne.variable_args in f.kinds: + if cdsl.operands.VARIABLE_ARGS in f.kinds: varg = '&{}data.varargs'.format(mut) capture = 'ref {}data, '.format(mut) else: @@ -311,7 +312,7 @@ def get_constraint(op, ctrl_typevar, type_sets): - `Free(idx)` where `idx` is an index into `type_sets`. - `Same`, `Lane`, `AsBool` for controlling typevar-derived constraints. """ - assert op.kind is cretonne.value + assert op.kind is cdsl.operands.VALUE t = op.typ # A concrete value type. @@ -504,7 +505,7 @@ def gen_inst_builder(inst, fmt): tmpl_types = list() into_args = list() for op in inst.ins: - if isinstance(op.kind, cretonne.ImmediateKind): + if isinstance(op.kind, cdsl.operands.ImmediateKind): t = 'T{}{}'.format(1 + len(tmpl_types), op.kind.name) tmpl_types.append('{}: Into<{}>'.format(t, op.kind.rust_type)) into_args.append(op.name) From eb688dc72d733f3af2d879c1bf75688c45564d11 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 11:02:07 -0800 Subject: [PATCH 0427/3084] Move formats, entities, and immediates to the base package. - base.formats defines instruction formats. - base.entities defines kinds of entity references. - base.immediates defines kinds of imediate operands. --- cranelift/docs/metaref.rst | 4 ++-- lib/cretonne/meta/{cretonne => base}/entities.py | 0 lib/cretonne/meta/{cretonne => base}/formats.py | 2 +- lib/cretonne/meta/{cretonne => base}/immediates.py | 0 lib/cretonne/meta/cretonne/__init__.py | 2 +- lib/cretonne/meta/cretonne/base.py | 5 +++-- lib/cretonne/meta/isa/riscv/recipes.py | 2 +- 7 files changed, 8 insertions(+), 7 deletions(-) rename lib/cretonne/meta/{cretonne => base}/entities.py (100%) rename lib/cretonne/meta/{cretonne => base}/formats.py (98%) rename lib/cretonne/meta/{cretonne => base}/immediates.py (100%) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 4699a80a43..a073ed5ad8 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -137,7 +137,7 @@ indicated with an instance of :class:`ImmediateKind`. .. autoclass:: ImmediateKind -.. automodule:: cretonne.immediates +.. automodule:: base.immediates :members: Entity references @@ -149,7 +149,7 @@ can be extended basic blocks, or entities declared in the function preamble. .. currentmodule:: cdsl.operands .. autoclass:: EntityRefKind -.. automodule:: cretonne.entities +.. automodule:: base.entities :members: Value types diff --git a/lib/cretonne/meta/cretonne/entities.py b/lib/cretonne/meta/base/entities.py similarity index 100% rename from lib/cretonne/meta/cretonne/entities.py rename to lib/cretonne/meta/base/entities.py diff --git a/lib/cretonne/meta/cretonne/formats.py b/lib/cretonne/meta/base/formats.py similarity index 98% rename from lib/cretonne/meta/cretonne/formats.py rename to lib/cretonne/meta/base/formats.py index a4d012020d..49300c2ed0 100644 --- a/lib/cretonne/meta/cretonne/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -7,7 +7,7 @@ in this module. """ from __future__ import absolute_import from cdsl.operands import VALUE, VARIABLE_ARGS -from . import InstructionFormat +from cretonne import InstructionFormat from .immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc from .entities import ebb, sig_ref, func_ref, jump_table diff --git a/lib/cretonne/meta/cretonne/immediates.py b/lib/cretonne/meta/base/immediates.py similarity index 100% rename from lib/cretonne/meta/cretonne/immediates.py rename to lib/cretonne/meta/base/immediates.py diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index d6b9835443..2429136143 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -732,4 +732,4 @@ class Encoding(object): # Import the fixed instruction formats now so they can be added to the # registry. -importlib.import_module('cretonne.formats') +importlib.import_module('base.formats') diff --git a/lib/cretonne/meta/cretonne/base.py b/lib/cretonne/meta/cretonne/base.py index e2ceadaff4..6ecbdcf86d 100644 --- a/lib/cretonne/meta/cretonne/base.py +++ b/lib/cretonne/meta/cretonne/base.py @@ -9,8 +9,9 @@ from cdsl.operands import VARIABLE_ARGS from . import Operand, Instruction, InstructionGroup from .typevar import TypeVar from base.types import i8, f32, f64, b1 -from .immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc -from . import entities +from base.immediates import imm64, uimm8, ieee32, ieee64, immvector +from base.immediates import intcc, floatcc +from base import entities instructions = InstructionGroup("base", "Shared base instruction set") diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 5818bd3fd9..b4f561f8e2 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -10,8 +10,8 @@ instruction formats described in the reference: """ from __future__ import absolute_import from cretonne import EncRecipe -from cretonne.formats import Binary, BinaryImm from cdsl.predicates import IsSignedInt +from base.formats import Binary, BinaryImm # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit # instructions have 11 as the two low bits, with bits 6:2 determining the base From 2e718c65541330bb7a0c3617febff6dbcd46456d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 11:18:14 -0800 Subject: [PATCH 0428/3084] Split out the typevar module. - cdsl.typevar defines TypeVar and TypeSet classes. --- cranelift/docs/metaref.rst | 2 +- lib/cretonne/meta/{cretonne => cdsl}/test_typevar.py | 0 lib/cretonne/meta/{cretonne => cdsl}/typevar.py | 0 lib/cretonne/meta/cretonne/__init__.py | 2 +- lib/cretonne/meta/cretonne/base.py | 2 +- 5 files changed, 3 insertions(+), 3 deletions(-) rename lib/cretonne/meta/{cretonne => cdsl}/test_typevar.py (100%) rename lib/cretonne/meta/{cretonne => cdsl}/typevar.py (100%) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index a073ed5ad8..31e4ffc9b5 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -102,7 +102,7 @@ instances that refer to a *type variable* instead of a concrete value type. Polymorphism only works for SSA value operands. Other operands have a fixed operand kind. -.. autoclass:: cretonne.typevar.TypeVar +.. autoclass:: cdsl.typevar.TypeVar :members: If multiple operands refer to the same type variable they will be required to diff --git a/lib/cretonne/meta/cretonne/test_typevar.py b/lib/cretonne/meta/cdsl/test_typevar.py similarity index 100% rename from lib/cretonne/meta/cretonne/test_typevar.py rename to lib/cretonne/meta/cdsl/test_typevar.py diff --git a/lib/cretonne/meta/cretonne/typevar.py b/lib/cretonne/meta/cdsl/typevar.py similarity index 100% rename from lib/cretonne/meta/cretonne/typevar.py rename to lib/cretonne/meta/cdsl/typevar.py diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index 2429136143..1753200f70 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -23,7 +23,7 @@ except ImportError: TYPE_CHECKING = False if TYPE_CHECKING: - from .typevar import TypeVar # noqa + from cdsl.typevar import TypeVar # noqa # Defining instructions. diff --git a/lib/cretonne/meta/cretonne/base.py b/lib/cretonne/meta/cretonne/base.py index 6ecbdcf86d..c648633ed9 100644 --- a/lib/cretonne/meta/cretonne/base.py +++ b/lib/cretonne/meta/cretonne/base.py @@ -6,8 +6,8 @@ support. """ from __future__ import absolute_import from cdsl.operands import VARIABLE_ARGS +from cdsl.typevar import TypeVar from . import Operand, Instruction, InstructionGroup -from .typevar import TypeVar from base.types import i8, f32, f64, b1 from base.immediates import imm64, uimm8, ieee32, ieee64, immvector from base.immediates import intcc, floatcc From 0666c8818a3f9c8204fb29195aa3ed5621a44eeb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 11:24:21 -0800 Subject: [PATCH 0429/3084] Get rid of operand_kind() This method caused lots of import cycles when type checking. Use isinstance() in the Operand constructor instead to decipher the OperandSpec union type. --- lib/cretonne/meta/cdsl/operands.py | 8 -------- lib/cretonne/meta/cdsl/typevar.py | 8 -------- lib/cretonne/meta/cretonne/__init__.py | 27 ++++++++++++++------------ 3 files changed, 15 insertions(+), 28 deletions(-) diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index 33157d9e23..279941c9fe 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -32,14 +32,6 @@ class OperandKind(object): # type: () -> str return 'OperandKind({})'.format(self.name) - def operand_kind(self): - # type: () -> OperandKind - """ - An `OperandKind` instance can be used directly as the type of an - `Operand` when defining an instruction. - """ - return self - def free_typevar(self): # Return the free typevariable controlling the type of this operand. return None diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 7f58ce73c1..2c10338bbf 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -6,7 +6,6 @@ polymorphic by using type variables. """ from __future__ import absolute_import import math -import cdsl.operands try: from typing import Tuple, Union # noqa @@ -314,13 +313,6 @@ class TypeVar(object): return TypeVar(None, None, base=self, derived_func='DoubleWidth') - def operand_kind(self): - # type: () -> cdsl.operands.OperandKind - # When a `TypeVar` object is used to describe the type of an `Operand` - # in an instruction definition, the kind of that operand is an SSA - # value. - return cdsl.operands.VALUE - def free_typevar(self): # type: () -> TypeVar if self.is_derived: diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index 1753200f70..4fcf1c9460 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -8,7 +8,8 @@ from __future__ import absolute_import import importlib from cdsl import camel_case from cdsl.predicates import And -import cdsl.types +from cdsl.types import ValueType +from cdsl.typevar import TypeVar from cdsl.operands import VALUE, VARIABLE_ARGS, OperandKind # The typing module is only required by mypy, and we don't use these imports @@ -18,7 +19,7 @@ try: from cdsl.predicates import Predicate, FieldPredicate # noqa MaybeBoundInst = Union['Instruction', 'BoundInstruction'] AnyPredicate = Union['Predicate', 'FieldPredicate'] - OperandSpec = Union['OperandKind', 'cdsl.types.ValueType', 'TypeVar'] + OperandSpec = Union['OperandKind', ValueType, TypeVar] except ImportError: TYPE_CHECKING = False @@ -105,10 +106,12 @@ class Operand(object): self.name = name self.__doc__ = doc self.typ = typ - if isinstance(typ, cdsl.types.ValueType): + if isinstance(typ, ValueType): + self.kind = VALUE + elif isinstance(typ, TypeVar): self.kind = VALUE else: - self.kind = typ.operand_kind() + self.kind = typ def get_doc(self): # type: () -> str @@ -126,7 +129,7 @@ class Operand(object): """ Is this an SSA value operand? """ - return self.kind is cdsl.operands.VALUE + return self.kind is VALUE class InstructionFormat(object): @@ -464,7 +467,7 @@ class Instruction(object): return x def bind(self, *args): - # type: (*cdsl.types.ValueType) -> BoundInstruction + # type: (*ValueType) -> BoundInstruction """ Bind a polymorphic instruction to a concrete list of type variable values. @@ -480,10 +483,10 @@ class Instruction(object): >>> iadd.i32 """ - return self.bind(cdsl.types.ValueType.by_name(name)) + return self.bind(ValueType.by_name(name)) def fully_bound(self): - # type: () -> Tuple[Instruction, Tuple[cdsl.types.ValueType, ...]] + # type: () -> Tuple[Instruction, Tuple[ValueType, ...]] """ Verify that all typevars have been bound, and return a `(inst, typevars)` pair. @@ -509,7 +512,7 @@ class BoundInstruction(object): """ def __init__(self, inst, typevars): - # type: (Instruction, Tuple[cdsl.types.ValueType, ...]) -> None + # type: (Instruction, Tuple[ValueType, ...]) -> None self.inst = inst self.typevars = typevars assert len(typevars) <= 1 + len(inst.other_typevars) @@ -518,7 +521,7 @@ class BoundInstruction(object): return '.'.join([self.inst.name, ] + list(map(str, self.typevars))) def bind(self, *args): - # type: (*cdsl.types.ValueType) -> BoundInstruction + # type: (*ValueType) -> BoundInstruction """ Bind additional typevars. """ @@ -531,10 +534,10 @@ class BoundInstruction(object): >>> uext.i32.i8 """ - return self.bind(cdsl.types.ValueType.by_name(name)) + return self.bind(ValueType.by_name(name)) def fully_bound(self): - # type: () -> Tuple[Instruction, Tuple[cdsl.types.ValueType, ...]] + # type: () -> Tuple[Instruction, Tuple[ValueType, ...]] """ Verify that all typevars have been bound, and return a `(inst, typevars)` pair. From b5e592ad568fd00b0e13ad243222c58c5e447898 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 11:30:31 -0800 Subject: [PATCH 0430/3084] Move Operand itself into cdsl.operands. --- lib/cretonne/meta/cdsl/operands.py | 61 ++++++++++++++++++++++++++ lib/cretonne/meta/cretonne/__init__.py | 54 +---------------------- 2 files changed, 62 insertions(+), 53 deletions(-) diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index 279941c9fe..f379357611 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -1,6 +1,14 @@ """Classes for describing instruction operands.""" from __future__ import absolute_import from . import camel_case +from .types import ValueType +from .typevar import TypeVar + +try: + from typing import Union + OperandSpec = Union['OperandKind', ValueType, TypeVar] +except ImportError: + pass # Kinds of operands. @@ -94,3 +102,56 @@ class EntityRefKind(OperandKind): def __repr__(self): # type: () -> str return 'EntityRefKind({})'.format(self.name) + + +class Operand(object): + """ + An instruction operand can be an *immediate*, an *SSA value*, or an *entity + reference*. The type of the operand is one of: + + 1. A :py:class:`ValueType` instance indicates an SSA value operand with a + concrete type. + + 2. A :py:class:`TypeVar` instance indicates an SSA value operand, and the + instruction is polymorphic over the possible concrete types that the + type variable can assume. + + 3. An :py:class:`ImmediateKind` instance indicates an immediate operand + whose value is encoded in the instruction itself rather than being + passed as an SSA value. + + 4. An :py:class:`EntityRefKind` instance indicates an operand that + references another entity in the function, typically something declared + in the function preamble. + + """ + def __init__(self, name, typ, doc=''): + # type: (str, OperandSpec, str) -> None + self.name = name + self.__doc__ = doc + self.typ = typ + if isinstance(typ, ValueType): + self.kind = VALUE + elif isinstance(typ, TypeVar): + self.kind = VALUE + else: + assert isinstance(typ, OperandKind) + self.kind = typ + + def get_doc(self): + # type: () -> str + if self.__doc__: + return self.__doc__ + else: + return self.typ.__doc__ + + def __str__(self): + # type: () -> str + return "`{}`".format(self.name) + + def is_value(self): + # type: () -> bool + """ + Is this an SSA value operand? + """ + return self.kind is VALUE diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index 4fcf1c9460..ee66dbe00b 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -10,7 +10,7 @@ from cdsl import camel_case from cdsl.predicates import And from cdsl.types import ValueType from cdsl.typevar import TypeVar -from cdsl.operands import VALUE, VARIABLE_ARGS, OperandKind +from cdsl.operands import VALUE, VARIABLE_ARGS, OperandKind, Operand # The typing module is only required by mypy, and we don't use these imports # outside type comments. @@ -80,58 +80,6 @@ class InstructionGroup(object): InstructionGroup._current.instructions.append(inst) -class Operand(object): - """ - An instruction operand can be an *immediate*, an *SSA value*, or an *entity - reference*. The type of the operand is one of: - - 1. A :py:class:`ValueType` instance indicates an SSA value operand with a - concrete type. - - 2. A :py:class:`TypeVar` instance indicates an SSA value operand, and the - instruction is polymorphic over the possible concrete types that the - type variable can assume. - - 3. An :py:class:`ImmediateKind` instance indicates an immediate operand - whose value is encoded in the instruction itself rather than being - passed as an SSA value. - - 4. An :py:class:`EntityRefKind` instance indicates an operand that - references another entity in the function, typically something declared - in the function preamble. - - """ - def __init__(self, name, typ, doc=''): - # type: (str, OperandSpec, str) -> None - self.name = name - self.__doc__ = doc - self.typ = typ - if isinstance(typ, ValueType): - self.kind = VALUE - elif isinstance(typ, TypeVar): - self.kind = VALUE - else: - self.kind = typ - - def get_doc(self): - # type: () -> str - if self.__doc__: - return self.__doc__ - else: - return self.typ.__doc__ - - def __str__(self): - # type: () -> str - return "`{}`".format(self.name) - - def is_value(self): - # type: () -> bool - """ - Is this an SSA value operand? - """ - return self.kind is VALUE - - class InstructionFormat(object): """ Every instruction opcode has a corresponding instruction format which From 87eb1a8ea0e5de8b0c6d7606def19f2cb72d01f6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 11:47:19 -0800 Subject: [PATCH 0431/3084] Split out instruction formats. - cdsl.formats provides classes for describing instruction formats. - base.formats provides concrete instruction format definitions. --- lib/cretonne/meta/base/formats.py | 2 +- lib/cretonne/meta/cdsl/formats.py | 189 +++++++++++++++++++++++++ lib/cretonne/meta/cretonne/__init__.py | 180 +---------------------- lib/cretonne/meta/cretonne/base.py | 1 + 4 files changed, 193 insertions(+), 179 deletions(-) create mode 100644 lib/cretonne/meta/cdsl/formats.py diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 49300c2ed0..421913b0e4 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -6,8 +6,8 @@ Rust representation of cretonne IL, so all instruction formats must be defined in this module. """ from __future__ import absolute_import +from cdsl.formats import InstructionFormat from cdsl.operands import VALUE, VARIABLE_ARGS -from cretonne import InstructionFormat from .immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc from .entities import ebb, sig_ref, func_ref, jump_table diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py new file mode 100644 index 0000000000..32f4463ae0 --- /dev/null +++ b/lib/cretonne/meta/cdsl/formats.py @@ -0,0 +1,189 @@ +"""Classes for describing instruction formats.""" +from __future__ import absolute_import +from .operands import OperandKind, VALUE, VARIABLE_ARGS +from .operands import Operand # noqa + +# The typing module is only required by mypy, and we don't use these imports +# outside type comments. +try: + from typing import Tuple, Union, Any, Sequence, Iterable # noqa +except ImportError: + pass + + +class InstructionFormat(object): + """ + Every instruction opcode has a corresponding instruction format which + determines the number of operands and their kinds. Instruction formats are + identified structurally, i.e., the format of an instruction is derived from + the kinds of operands used in its declaration. + + Most instruction formats produce a single result, or no result at all. If + an instruction can produce more than one result, the `multiple_results` + flag must be set on its format. All results are of the `value` kind, and + the instruction format does not keep track of how many results are + produced. Some instructions, like `call`, may have a variable number of + results. + + All instruction formats must be predefined in the + :py:mod:`cretonne.formats` module. + + :param kinds: List of `OperandKind` objects describing the operands. + :param name: Instruction format name in CamelCase. This is used as a Rust + variant name in both the `InstructionData` and `InstructionFormat` + enums. + :param multiple_results: Set to `True` if this instruction format allows + more than one result to be produced. + :param boxed_storage: Set to `True` is this instruction format requires a + `data: Box<...>` pointer to additional storage in its `InstructionData` + variant. + :param typevar_operand: Index of the input operand that is used to infer + the controlling type variable. By default, this is the first `value` + operand. + """ + + # Map (multiple_results, kind, kind, ...) -> InstructionFormat + _registry = dict() # type: Dict[Tuple[bool, Tuple[OperandKind, ...]], InstructionFormat] # noqa + + # All existing formats. + all_formats = list() # type: List[InstructionFormat] + + def __init__(self, *kinds, **kwargs): + # type: (*Union[OperandKind, Tuple[str, OperandKind]], **Any) -> None # noqa + self.name = kwargs.get('name', None) # type: str + self.multiple_results = kwargs.get('multiple_results', False) + self.boxed_storage = kwargs.get('boxed_storage', False) + self.members = list() # type: List[str] + self.kinds = tuple(self._process_member_names(kinds)) + + # Which of self.kinds are `value`? + self.value_operands = tuple( + i for i, k in enumerate(self.kinds) if k is VALUE) + + # The typevar_operand argument must point to a 'value' operand. + self.typevar_operand = kwargs.get('typevar_operand', None) # type: int + if self.typevar_operand is not None: + assert self.kinds[self.typevar_operand] is VALUE, \ + "typevar_operand must indicate a 'value' operand" + elif len(self.value_operands) > 0: + # Default to the first 'value' operand, if there is one. + self.typevar_operand = self.value_operands[0] + + # Compute a signature for the global registry. + sig = (self.multiple_results, self.kinds) + if sig in InstructionFormat._registry: + raise RuntimeError( + "Format '{}' has the same signature as existing format '{}'" + .format(self.name, InstructionFormat._registry[sig])) + InstructionFormat._registry[sig] = self + InstructionFormat.all_formats.append(self) + + def _process_member_names(self, kinds): + # type: (Sequence[Union[OperandKind, Tuple[str, OperandKind]]]) -> Iterable[OperandKind] # noqa + """ + Extract names of all the immediate operands in the kinds tuple. + + Each entry is either an `OperandKind` instance, or a `(member, kind)` + pair. The member names correspond to members in the Rust + `InstructionData` data structure. + + Yields the operand kinds. + """ + for arg in kinds: + if isinstance(arg, OperandKind): + member = arg.default_member + k = arg + else: + member, k = arg + self.members.append(member) + yield k + + def __str__(self): + # type: () -> str + args = ', '.join('{}: {}'.format(m, k) if m else str(k) + for m, k in zip(self.members, self.kinds)) + return '{}({})'.format(self.name, args) + + def __getattr__(self, attr): + # type: (str) -> FormatField + """ + Make instruction format members available as attributes. + + Each non-value format member becomes a corresponding `FormatField` + attribute. + """ + try: + i = self.members.index(attr) + except ValueError: + raise AttributeError( + '{} is neither a {} member or a ' + .format(attr, self.name) + + 'normal InstructionFormat attribute') + field = FormatField(self, i, attr) + setattr(self, attr, field) + return field + + @staticmethod + def lookup(ins, outs): + # type: (Sequence[Operand], Sequence[Operand]) -> InstructionFormat + """ + Find an existing instruction format that matches the given lists of + instruction inputs and outputs. + + The `ins` and `outs` arguments correspond to the + :py:class:`Instruction` arguments of the same name, except they must be + tuples of :py:`Operand` objects. + """ + if len(outs) == 1: + multiple_results = outs[0].kind == VARIABLE_ARGS + else: + multiple_results = len(outs) > 1 + sig = (multiple_results, tuple(op.kind for op in ins)) + if sig not in InstructionFormat._registry: + raise RuntimeError( + "No instruction format matches ins = ({}){}".format( + ", ".join(map(str, sig[1])), + "[multiple results]" if multiple_results else "")) + return InstructionFormat._registry[sig] + + @staticmethod + def extract_names(globs): + """ + Given a dict mapping name -> object as returned by `globals()`, find + all the InstructionFormat objects and set their name from the dict key. + This is used to name a bunch of global variables in a module. + """ + for name, obj in globs.items(): + if isinstance(obj, InstructionFormat): + assert obj.name is None + obj.name = name + + +class FormatField(object): + """ + A field in an instruction format. + + This corresponds to a single member of a variant of the `InstructionData` + data type. + + :param format: Parent `InstructionFormat`. + :param operand: Operand number in parent. + :param name: Member name in `InstructionData` variant. + """ + + def __init__(self, format, operand, name): + # type: (InstructionFormat, int, str) -> None + self.format = format + self.operand = operand + self.name = name + + def __str__(self): + # type: () -> str + return '{}.{}'.format(self.format.name, self.name) + + def rust_name(self): + # type: () -> str + if self.format.boxed_storage: + return 'data.' + self.name + else: + return self.name diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index ee66dbe00b..0ab55491c9 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -10,7 +10,8 @@ from cdsl import camel_case from cdsl.predicates import And from cdsl.types import ValueType from cdsl.typevar import TypeVar -from cdsl.operands import VALUE, VARIABLE_ARGS, OperandKind, Operand +from cdsl.operands import Operand +from cdsl.formats import InstructionFormat # The typing module is only required by mypy, and we don't use these imports # outside type comments. @@ -19,7 +20,6 @@ try: from cdsl.predicates import Predicate, FieldPredicate # noqa MaybeBoundInst = Union['Instruction', 'BoundInstruction'] AnyPredicate = Union['Predicate', 'FieldPredicate'] - OperandSpec = Union['OperandKind', ValueType, TypeVar] except ImportError: TYPE_CHECKING = False @@ -80,182 +80,6 @@ class InstructionGroup(object): InstructionGroup._current.instructions.append(inst) -class InstructionFormat(object): - """ - Every instruction opcode has a corresponding instruction format which - determines the number of operands and their kinds. Instruction formats are - identified structurally, i.e., the format of an instruction is derived from - the kinds of operands used in its declaration. - - Most instruction formats produce a single result, or no result at all. If - an instruction can produce more than one result, the `multiple_results` - flag must be set on its format. All results are of the `value` kind, and - the instruction format does not keep track of how many results are - produced. Some instructions, like `call`, may have a variable number of - results. - - All instruction formats must be predefined in the - :py:mod:`cretonne.formats` module. - - :param kinds: List of `OperandKind` objects describing the operands. - :param name: Instruction format name in CamelCase. This is used as a Rust - variant name in both the `InstructionData` and `InstructionFormat` - enums. - :param multiple_results: Set to `True` if this instruction format allows - more than one result to be produced. - :param boxed_storage: Set to `True` is this instruction format requires a - `data: Box<...>` pointer to additional storage in its `InstructionData` - variant. - :param typevar_operand: Index of the input operand that is used to infer - the controlling type variable. By default, this is the first `value` - operand. - """ - - # Map (multiple_results, kind, kind, ...) -> InstructionFormat - _registry = dict() # type: Dict[Tuple[bool, Tuple[OperandKind, ...]], InstructionFormat] # noqa - - # All existing formats. - all_formats = list() # type: List[InstructionFormat] - - def __init__(self, *kinds, **kwargs): - # type: (*Union[OperandKind, Tuple[str, OperandKind]], **Any) -> None # noqa - self.name = kwargs.get('name', None) # type: str - self.multiple_results = kwargs.get('multiple_results', False) - self.boxed_storage = kwargs.get('boxed_storage', False) - self.members = list() # type: List[str] - self.kinds = tuple(self._process_member_names(kinds)) - - # Which of self.kinds are `value`? - self.value_operands = tuple( - i for i, k in enumerate(self.kinds) if k is VALUE) - - # The typevar_operand argument must point to a 'value' operand. - self.typevar_operand = kwargs.get('typevar_operand', None) # type: int - if self.typevar_operand is not None: - assert self.kinds[self.typevar_operand] is VALUE, \ - "typevar_operand must indicate a 'value' operand" - elif len(self.value_operands) > 0: - # Default to the first 'value' operand, if there is one. - self.typevar_operand = self.value_operands[0] - - # Compute a signature for the global registry. - sig = (self.multiple_results, self.kinds) - if sig in InstructionFormat._registry: - raise RuntimeError( - "Format '{}' has the same signature as existing format '{}'" - .format(self.name, InstructionFormat._registry[sig])) - InstructionFormat._registry[sig] = self - InstructionFormat.all_formats.append(self) - - def _process_member_names(self, kinds): - # type: (Sequence[Union[OperandKind, Tuple[str, OperandKind]]]) -> Iterable[OperandKind] # noqa - """ - Extract names of all the immediate operands in the kinds tuple. - - Each entry is either an `OperandKind` instance, or a `(member, kind)` - pair. The member names correspond to members in the Rust - `InstructionData` data structure. - - Yields the operand kinds. - """ - for arg in kinds: - if isinstance(arg, OperandKind): - member = arg.default_member - k = arg - else: - member, k = arg - self.members.append(member) - yield k - - def __str__(self): - args = ', '.join('{}: {}'.format(m, k) if m else str(k) - for m, k in zip(self.members, self.kinds)) - return '{}({})'.format(self.name, args) - - def __getattr__(self, attr): - # type: (str) -> FormatField - """ - Make instruction format members available as attributes. - - Each non-value format member becomes a corresponding `FormatField` - attribute. - """ - try: - i = self.members.index(attr) - except ValueError: - raise AttributeError( - '{} is neither a {} member or a ' - .format(attr, self.name) + - 'normal InstructionFormat attribute') - field = FormatField(self, i, attr) - setattr(self, attr, field) - return field - - @staticmethod - def lookup(ins, outs): - # type: (Sequence[Operand], Sequence[Operand]) -> InstructionFormat - """ - Find an existing instruction format that matches the given lists of - instruction inputs and outputs. - - The `ins` and `outs` arguments correspond to the - :py:class:`Instruction` arguments of the same name, except they must be - tuples of :py:`Operand` objects. - """ - if len(outs) == 1: - multiple_results = outs[0].kind == VARIABLE_ARGS - else: - multiple_results = len(outs) > 1 - sig = (multiple_results, tuple(op.kind for op in ins)) - if sig not in InstructionFormat._registry: - raise RuntimeError( - "No instruction format matches ins = ({}){}".format( - ", ".join(map(str, sig[1])), - "[multiple results]" if multiple_results else "")) - return InstructionFormat._registry[sig] - - @staticmethod - def extract_names(globs): - """ - Given a dict mapping name -> object as returned by `globals()`, find - all the InstructionFormat objects and set their name from the dict key. - This is used to name a bunch of global variables in a module. - """ - for name, obj in globs.items(): - if isinstance(obj, InstructionFormat): - assert obj.name is None - obj.name = name - - -class FormatField(object): - """ - A field in an instruction format. - - This corresponds to a single member of a variant of the `InstructionData` - data type. - - :param format: Parent `InstructionFormat`. - :param operand: Operand number in parent. - :param name: Member name in `InstructionData` variant. - """ - - def __init__(self, format, operand, name): - # type: (InstructionFormat, int, str) -> None - self.format = format - self.operand = operand - self.name = name - - def __str__(self): - return '{}.{}'.format(self.format.name, self.name) - - def rust_name(self): - # type: () -> str - if self.format.boxed_storage: - return 'data.' + self.name - else: - return self.name - - class Instruction(object): """ The operands to the instruction are specified as two tuples: ``ins`` and diff --git a/lib/cretonne/meta/cretonne/base.py b/lib/cretonne/meta/cretonne/base.py index c648633ed9..ce747cc203 100644 --- a/lib/cretonne/meta/cretonne/base.py +++ b/lib/cretonne/meta/cretonne/base.py @@ -12,6 +12,7 @@ from base.types import i8, f32, f64, b1 from base.immediates import imm64, uimm8, ieee32, ieee64, immvector from base.immediates import intcc, floatcc from base import entities +import base.formats # noqa instructions = InstructionGroup("base", "Shared base instruction set") From 0b9b956695556e41151992d27a2b22f0e8f0cf8e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 12:08:14 -0800 Subject: [PATCH 0432/3084] Split out instruction definitions. - cdsl.instructions defines the Instruction class. - base.instructions defines the base instruction set. --- cranelift/docs/cton_domain.py | 2 +- cranelift/docs/langref.rst | 2 +- cranelift/docs/metaref.rst | 14 +- .../base.py => base/instructions.py} | 8 +- lib/cretonne/meta/cdsl/instructions.py | 316 ++++++++++++++++++ lib/cretonne/meta/cretonne/__init__.py | 310 +---------------- lib/cretonne/meta/cretonne/ast.py | 12 +- lib/cretonne/meta/cretonne/legalize.py | 8 +- lib/cretonne/meta/cretonne/test_ast.py | 2 +- lib/cretonne/meta/cretonne/test_xform.py | 2 +- lib/cretonne/meta/cretonne/xform.py | 2 +- lib/cretonne/meta/gen_instr.py | 22 +- lib/cretonne/meta/gen_settings.py | 2 +- lib/cretonne/meta/isa/riscv/defs.py | 4 +- lib/cretonne/meta/isa/riscv/encodings.py | 2 +- 15 files changed, 360 insertions(+), 348 deletions(-) rename lib/cretonne/meta/{cretonne/base.py => base/instructions.py} (99%) create mode 100644 lib/cretonne/meta/cdsl/instructions.py diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index 00242ff565..47c7c2996c 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -269,7 +269,7 @@ class InstDocumenter(sphinx.ext.autodoc.Documenter): return False def resolve_name(self, modname, parents, path, base): - return 'cretonne.base', [base] + return 'base.instructions', [base] def format_signature(self): inst = self.object diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 5dcf80a893..a9d5ac4c96 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -849,7 +849,7 @@ Base instruction group All of the shared instructions are part of the :instgroup:`base` instruction group. -.. autoinstgroup:: cretonne.base.instructions +.. autoinstgroup:: base.instructions.GROUP Target ISAs may define further instructions in their own instruction groups. diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 31e4ffc9b5..190945c61d 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -71,7 +71,7 @@ a module looks like this:: Instruction descriptions ======================== -.. currentmodule:: cretonne +.. module:: cdsl.instructions New instructions are defined as instances of the :class:`Instruction` class. As instruction instances are created, they are added to the currently @@ -81,12 +81,14 @@ open :class:`InstructionGroup`. :members: The basic Cretonne instruction set described in :doc:`langref` is defined by the -Python module :mod:`cretonne.base`. This module has a global variable -:data:`cretonne.base.instructions` which is an :class:`InstructionGroup` -instance containing all the base instructions. +Python module :mod:`base.instructions`. This module has a global variable +:data:`base.instructions.GROUP` which is an :class:`InstructionGroup` instance +containing all the base instructions. .. autoclass:: Instruction +.. currentmodule:: cdsl.operands + An instruction is defined with a set of distinct input and output operands which must be instances of the :class:`Operand` class. @@ -201,7 +203,7 @@ represent SSA values: .. autodata:: VALUE .. autodata:: VARIABLE_ARGS -.. currentmodule:: cretonne +.. module:: cdsl.formats When an instruction description is created, it is automatically assigned a predefined instruction format which is an instance of @@ -253,6 +255,8 @@ controlling type variable, or it can vary independently of the other operands. Encodings ========= +.. currentmodule:: cretonne + Encodings describe how Cretonne instructions are mapped to binary machine code for the target architecture. After the legalization pass, all remaining instructions are expected to map 1-1 to native instruction encodings. Cretonne diff --git a/lib/cretonne/meta/cretonne/base.py b/lib/cretonne/meta/base/instructions.py similarity index 99% rename from lib/cretonne/meta/cretonne/base.py rename to lib/cretonne/meta/base/instructions.py index ce747cc203..b5a2c784fa 100644 --- a/lib/cretonne/meta/cretonne/base.py +++ b/lib/cretonne/meta/base/instructions.py @@ -5,16 +5,16 @@ This module defines the basic Cretonne instruction set that all targets support. """ from __future__ import absolute_import -from cdsl.operands import VARIABLE_ARGS +from cdsl.operands import Operand, VARIABLE_ARGS from cdsl.typevar import TypeVar -from . import Operand, Instruction, InstructionGroup +from cdsl.instructions import Instruction, InstructionGroup from base.types import i8, f32, f64, b1 from base.immediates import imm64, uimm8, ieee32, ieee64, immvector from base.immediates import intcc, floatcc from base import entities import base.formats # noqa -instructions = InstructionGroup("base", "Shared base instruction set") +GROUP = InstructionGroup("base", "Shared base instruction set") Int = TypeVar('Int', 'A scalar or vector integer type', ints=True, simd=True) iB = TypeVar('iB', 'A scalar integer type', ints=True) @@ -1213,4 +1213,4 @@ iconcat_lohi = Instruction( """, ins=(lo, hi), outs=a) -instructions.close() +GROUP.close() diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py new file mode 100644 index 0000000000..da9f5bb179 --- /dev/null +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -0,0 +1,316 @@ +"""Classes for defining instructions.""" +from __future__ import absolute_import +from . import camel_case +from .types import ValueType +from .operands import Operand +from .formats import InstructionFormat + +try: + from typing import Union, Sequence + # List of operands for ins/outs: + OpList = Union[Sequence[Operand], Operand] + MaybeBoundInst = Union['Instruction', 'BoundInstruction'] + from typing import Tuple, Any # noqa +except ImportError: + pass + + +class InstructionGroup(object): + """ + Every instruction must belong to exactly one instruction group. A given + target architecture can support instructions from multiple groups, and it + does not necessarily support all instructions in a group. + + New instructions are automatically added to the currently open instruction + group. + """ + + # The currently open instruction group. + _current = None # type: InstructionGroup + + def open(self): + # type: () -> None + """ + Open this instruction group such that future new instructions are + added to this group. + """ + assert InstructionGroup._current is None, ( + "Can't open {} since {} is already open" + .format(self, InstructionGroup._current)) + InstructionGroup._current = self + + def close(self): + # type: () -> None + """ + Close this instruction group. This function should be called before + opening another instruction group. + """ + assert InstructionGroup._current is self, ( + "Can't close {}, the open instuction group is {}" + .format(self, InstructionGroup._current)) + InstructionGroup._current = None + + def __init__(self, name, doc): + # type: (str, str) -> None + self.name = name + self.__doc__ = doc + self.instructions = [] # type: List[Instruction] + self.open() + + @staticmethod + def append(inst): + # type: (Instruction) -> None + assert InstructionGroup._current, \ + "Open an instruction group before defining instructions." + InstructionGroup._current.instructions.append(inst) + + +class Instruction(object): + """ + The operands to the instruction are specified as two tuples: ``ins`` and + ``outs``. Since the Python singleton tuple syntax is a bit awkward, it is + allowed to specify a singleton as just the operand itself, i.e., `ins=x` + and `ins=(x,)` are both allowed and mean the same thing. + + :param name: Instruction mnemonic, also becomes opcode name. + :param doc: Documentation string. + :param ins: Tuple of input operands. This can be a mix of SSA value + operands and other operand kinds. + :param outs: Tuple of output operands. The output operands must be SSA + values or `variable_args`. + :param is_terminator: This is a terminator instruction. + :param is_branch: This is a branch instruction. + """ + + def __init__(self, name, doc, ins=(), outs=(), **kwargs): + # type: (str, str, OpList, OpList, **Any) -> None # noqa + self.name = name + self.camel_name = camel_case(name) + self.__doc__ = doc + self.ins = self._to_operand_tuple(ins) + self.outs = self._to_operand_tuple(outs) + self.format = InstructionFormat.lookup(self.ins, self.outs) + # Indexes into outs for value results. Others are `variable_args`. + self.value_results = tuple( + i for i, o in enumerate(self.outs) if o.is_value()) + self._verify_polymorphic() + InstructionGroup.append(self) + + def __str__(self): + prefix = ', '.join(o.name for o in self.outs) + if prefix: + prefix = prefix + ' = ' + suffix = ', '.join(o.name for o in self.ins) + return '{}{} {}'.format(prefix, self.name, suffix) + + def snake_name(self): + # type: () -> str + """ + Get the snake_case name of this instruction. + + Keywords in Rust and Python are altered by appending a '_' + """ + if self.name == 'return': + return 'return_' + else: + return self.name + + def blurb(self): + """Get the first line of the doc comment""" + for line in self.__doc__.split('\n'): + line = line.strip() + if line: + return line + return "" + + def _verify_polymorphic(self): + """ + Check if this instruction is polymorphic, and verify its use of type + variables. + """ + poly_ins = [ + i for i in self.format.value_operands + if self.ins[i].typ.free_typevar()] + poly_outs = [ + i for i, o in enumerate(self.outs) + if o.typ.free_typevar()] + self.is_polymorphic = len(poly_ins) > 0 or len(poly_outs) > 0 + if not self.is_polymorphic: + return + + # Prefer to use the typevar_operand to infer the controlling typevar. + self.use_typevar_operand = False + typevar_error = None + if self.format.typevar_operand is not None: + try: + tv = self.ins[self.format.typevar_operand].typ + if tv is tv.free_typevar(): + self.other_typevars = self._verify_ctrl_typevar(tv) + self.ctrl_typevar = tv + self.use_typevar_operand = True + except RuntimeError as e: + typevar_error = e + + if not self.use_typevar_operand: + # The typevar_operand argument doesn't work. Can we infer from the + # first result instead? + if len(self.outs) == 0: + if typevar_error: + raise typevar_error + else: + raise RuntimeError( + "typevar_operand must be a free type variable") + tv = self.outs[0].typ + if tv is not tv.free_typevar(): + raise RuntimeError("first result must be a free type variable") + self.other_typevars = self._verify_ctrl_typevar(tv) + self.ctrl_typevar = tv + + def _verify_ctrl_typevar(self, ctrl_typevar): + """ + Verify that the use of TypeVars is consistent with `ctrl_typevar` as + the controlling type variable. + + All polymorhic inputs must either be derived from `ctrl_typevar` or be + independent free type variables only used once. + + All polymorphic results must be derived from `ctrl_typevar`. + + Return list of other type variables used, or raise an error. + """ + other_tvs = [] + # Check value inputs. + for opidx in self.format.value_operands: + typ = self.ins[opidx].typ + tv = typ.free_typevar() + # Non-polymorphic or derived form ctrl_typevar is OK. + if tv is None or tv is ctrl_typevar: + continue + # No other derived typevars allowed. + if typ is not tv: + raise RuntimeError( + "{}: type variable {} must be derived from {}" + .format(self.ins[opidx], typ.name, ctrl_typevar)) + # Other free type variables can only be used once each. + if tv in other_tvs: + raise RuntimeError( + "type variable {} can't be used more than once" + .format(tv.name)) + other_tvs.append(tv) + + # Check outputs. + for result in self.outs: + typ = result.typ + tv = typ.free_typevar() + # Non-polymorphic or derived from ctrl_typevar is OK. + if tv is None or tv is ctrl_typevar: + continue + raise RuntimeError( + "type variable in output not derived from ctrl_typevar") + + return other_tvs + + @staticmethod + def _to_operand_tuple(x): + # type: (Union[Sequence[Operand], Operand]) -> Tuple[Operand, ...] + # Allow a single Operand instance instead of the awkward singleton + # tuple syntax. + if isinstance(x, Operand): + x = (x,) + else: + x = tuple(x) + for op in x: + assert isinstance(op, Operand) + return x + + def bind(self, *args): + # type: (*ValueType) -> BoundInstruction + """ + Bind a polymorphic instruction to a concrete list of type variable + values. + """ + assert self.is_polymorphic + return BoundInstruction(self, args) + + def __getattr__(self, name): + # type: (str) -> BoundInstruction + """ + Bind a polymorphic instruction to a single type variable with dot + syntax: + + >>> iadd.i32 + """ + return self.bind(ValueType.by_name(name)) + + def fully_bound(self): + # type: () -> Tuple[Instruction, Tuple[ValueType, ...]] + """ + Verify that all typevars have been bound, and return a + `(inst, typevars)` pair. + + This version in `Instruction` itself allows non-polymorphic + instructions to duck-type as `BoundInstruction`\s. + """ + assert not self.is_polymorphic, self + return (self, ()) + + def __call__(self, *args): + """ + Create an `ast.Apply` AST node representing the application of this + instruction to the arguments. + """ + from cretonne.ast import Apply + return Apply(self, args) + + +class BoundInstruction(object): + """ + A polymorphic `Instruction` bound to concrete type variables. + """ + + def __init__(self, inst, typevars): + # type: (Instruction, Tuple[ValueType, ...]) -> None + self.inst = inst + self.typevars = typevars + assert len(typevars) <= 1 + len(inst.other_typevars) + + def __str__(self): + return '.'.join([self.inst.name, ] + list(map(str, self.typevars))) + + def bind(self, *args): + # type: (*ValueType) -> BoundInstruction + """ + Bind additional typevars. + """ + return BoundInstruction(self.inst, self.typevars + args) + + def __getattr__(self, name): + # type: (str) -> BoundInstruction + """ + Bind an additional typevar dot syntax: + + >>> uext.i32.i8 + """ + return self.bind(ValueType.by_name(name)) + + def fully_bound(self): + # type: () -> Tuple[Instruction, Tuple[ValueType, ...]] + """ + Verify that all typevars have been bound, and return a + `(inst, typevars)` pair. + """ + if len(self.typevars) < 1 + len(self.inst.other_typevars): + unb = ', '.join( + str(tv) for tv in + self.inst.other_typevars[len(self.typevars) - 1:]) + raise AssertionError("Unbound typevar {} in {}".format(unb, self)) + assert len(self.typevars) == 1 + len(self.inst.other_typevars) + return (self.inst, self.typevars) + + def __call__(self, *args): + """ + Create an `ast.Apply` AST node representing the application of this + instruction to the arguments. + """ + from cretonne.ast import Apply + return Apply(self, args) diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cretonne/__init__.py index 0ab55491c9..fa1da4d302 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cretonne/__init__.py @@ -6,19 +6,15 @@ instructions. """ from __future__ import absolute_import import importlib -from cdsl import camel_case from cdsl.predicates import And -from cdsl.types import ValueType from cdsl.typevar import TypeVar -from cdsl.operands import Operand -from cdsl.formats import InstructionFormat # The typing module is only required by mypy, and we don't use these imports # outside type comments. try: from typing import Tuple, Union, Any, Iterable, Sequence, TYPE_CHECKING # noqa from cdsl.predicates import Predicate, FieldPredicate # noqa - MaybeBoundInst = Union['Instruction', 'BoundInstruction'] + from cdsl.instructions import MaybeBoundInst # noqa AnyPredicate = Union['Predicate', 'FieldPredicate'] except ImportError: TYPE_CHECKING = False @@ -27,310 +23,6 @@ if TYPE_CHECKING: from cdsl.typevar import TypeVar # noqa -# Defining instructions. - - -class InstructionGroup(object): - """ - Every instruction must belong to exactly one instruction group. A given - target architecture can support instructions from multiple groups, and it - does not necessarily support all instructions in a group. - - New instructions are automatically added to the currently open instruction - group. - """ - - # The currently open instruction group. - _current = None # type: InstructionGroup - - def open(self): - # type: () -> None - """ - Open this instruction group such that future new instructions are - added to this group. - """ - assert InstructionGroup._current is None, ( - "Can't open {} since {} is already open" - .format(self, InstructionGroup._current)) - InstructionGroup._current = self - - def close(self): - # type: () -> None - """ - Close this instruction group. This function should be called before - opening another instruction group. - """ - assert InstructionGroup._current is self, ( - "Can't close {}, the open instuction group is {}" - .format(self, InstructionGroup._current)) - InstructionGroup._current = None - - def __init__(self, name, doc): - # type: (str, str) -> None - self.name = name - self.__doc__ = doc - self.instructions = [] # type: List[Instruction] - self.open() - - @staticmethod - def append(inst): - # type: (Instruction) -> None - assert InstructionGroup._current, \ - "Open an instruction group before defining instructions." - InstructionGroup._current.instructions.append(inst) - - -class Instruction(object): - """ - The operands to the instruction are specified as two tuples: ``ins`` and - ``outs``. Since the Python singleton tuple syntax is a bit awkward, it is - allowed to specify a singleton as just the operand itself, i.e., `ins=x` - and `ins=(x,)` are both allowed and mean the same thing. - - :param name: Instruction mnemonic, also becomes opcode name. - :param doc: Documentation string. - :param ins: Tuple of input operands. This can be a mix of SSA value - operands and other operand kinds. - :param outs: Tuple of output operands. The output operands must be SSA - values or `variable_args`. - :param is_terminator: This is a terminator instruction. - :param is_branch: This is a branch instruction. - """ - - def __init__(self, name, doc, ins=(), outs=(), **kwargs): - # type: (str, str, Union[Sequence[Operand], Operand], Union[Sequence[Operand], Operand], **Any) -> None # noqa - self.name = name - self.camel_name = camel_case(name) - self.__doc__ = doc - self.ins = self._to_operand_tuple(ins) - self.outs = self._to_operand_tuple(outs) - self.format = InstructionFormat.lookup(self.ins, self.outs) - # Indexes into outs for value results. Others are `variable_args`. - self.value_results = tuple( - i for i, o in enumerate(self.outs) if o.is_value()) - self._verify_polymorphic() - InstructionGroup.append(self) - - def __str__(self): - prefix = ', '.join(o.name for o in self.outs) - if prefix: - prefix = prefix + ' = ' - suffix = ', '.join(o.name for o in self.ins) - return '{}{} {}'.format(prefix, self.name, suffix) - - def snake_name(self): - # type: () -> str - """ - Get the snake_case name of this instruction. - - Keywords in Rust and Python are altered by appending a '_' - """ - if self.name == 'return': - return 'return_' - else: - return self.name - - def blurb(self): - """Get the first line of the doc comment""" - for line in self.__doc__.split('\n'): - line = line.strip() - if line: - return line - return "" - - def _verify_polymorphic(self): - """ - Check if this instruction is polymorphic, and verify its use of type - variables. - """ - poly_ins = [ - i for i in self.format.value_operands - if self.ins[i].typ.free_typevar()] - poly_outs = [ - i for i, o in enumerate(self.outs) - if o.typ.free_typevar()] - self.is_polymorphic = len(poly_ins) > 0 or len(poly_outs) > 0 - if not self.is_polymorphic: - return - - # Prefer to use the typevar_operand to infer the controlling typevar. - self.use_typevar_operand = False - typevar_error = None - if self.format.typevar_operand is not None: - try: - tv = self.ins[self.format.typevar_operand].typ - if tv is tv.free_typevar(): - self.other_typevars = self._verify_ctrl_typevar(tv) - self.ctrl_typevar = tv - self.use_typevar_operand = True - except RuntimeError as e: - typevar_error = e - - if not self.use_typevar_operand: - # The typevar_operand argument doesn't work. Can we infer from the - # first result instead? - if len(self.outs) == 0: - if typevar_error: - raise typevar_error - else: - raise RuntimeError( - "typevar_operand must be a free type variable") - tv = self.outs[0].typ - if tv is not tv.free_typevar(): - raise RuntimeError("first result must be a free type variable") - self.other_typevars = self._verify_ctrl_typevar(tv) - self.ctrl_typevar = tv - - def _verify_ctrl_typevar(self, ctrl_typevar): - """ - Verify that the use of TypeVars is consistent with `ctrl_typevar` as - the controlling type variable. - - All polymorhic inputs must either be derived from `ctrl_typevar` or be - independent free type variables only used once. - - All polymorphic results must be derived from `ctrl_typevar`. - - Return list of other type variables used, or raise an error. - """ - other_tvs = [] - # Check value inputs. - for opidx in self.format.value_operands: - typ = self.ins[opidx].typ - tv = typ.free_typevar() - # Non-polymorphic or derived form ctrl_typevar is OK. - if tv is None or tv is ctrl_typevar: - continue - # No other derived typevars allowed. - if typ is not tv: - raise RuntimeError( - "{}: type variable {} must be derived from {}" - .format(self.ins[opidx], typ.name, ctrl_typevar)) - # Other free type variables can only be used once each. - if tv in other_tvs: - raise RuntimeError( - "type variable {} can't be used more than once" - .format(tv.name)) - other_tvs.append(tv) - - # Check outputs. - for result in self.outs: - typ = result.typ - tv = typ.free_typevar() - # Non-polymorphic or derived from ctrl_typevar is OK. - if tv is None or tv is ctrl_typevar: - continue - raise RuntimeError( - "type variable in output not derived from ctrl_typevar") - - return other_tvs - - @staticmethod - def _to_operand_tuple(x): - # type: (Union[Sequence[Operand], Operand]) -> Tuple[Operand, ...] - # Allow a single Operand instance instead of the awkward singleton - # tuple syntax. - if isinstance(x, Operand): - x = (x,) - else: - x = tuple(x) - for op in x: - assert isinstance(op, Operand) - return x - - def bind(self, *args): - # type: (*ValueType) -> BoundInstruction - """ - Bind a polymorphic instruction to a concrete list of type variable - values. - """ - assert self.is_polymorphic - return BoundInstruction(self, args) - - def __getattr__(self, name): - # type: (str) -> BoundInstruction - """ - Bind a polymorphic instruction to a single type variable with dot - syntax: - - >>> iadd.i32 - """ - return self.bind(ValueType.by_name(name)) - - def fully_bound(self): - # type: () -> Tuple[Instruction, Tuple[ValueType, ...]] - """ - Verify that all typevars have been bound, and return a - `(inst, typevars)` pair. - - This version in `Instruction` itself allows non-polymorphic - instructions to duck-type as `BoundInstruction`\s. - """ - assert not self.is_polymorphic, self - return (self, ()) - - def __call__(self, *args): - """ - Create an `ast.Apply` AST node representing the application of this - instruction to the arguments. - """ - from .ast import Apply - return Apply(self, args) - - -class BoundInstruction(object): - """ - A polymorphic `Instruction` bound to concrete type variables. - """ - - def __init__(self, inst, typevars): - # type: (Instruction, Tuple[ValueType, ...]) -> None - self.inst = inst - self.typevars = typevars - assert len(typevars) <= 1 + len(inst.other_typevars) - - def __str__(self): - return '.'.join([self.inst.name, ] + list(map(str, self.typevars))) - - def bind(self, *args): - # type: (*ValueType) -> BoundInstruction - """ - Bind additional typevars. - """ - return BoundInstruction(self.inst, self.typevars + args) - - def __getattr__(self, name): - # type: (str) -> BoundInstruction - """ - Bind an additional typevar dot syntax: - - >>> uext.i32.i8 - """ - return self.bind(ValueType.by_name(name)) - - def fully_bound(self): - # type: () -> Tuple[Instruction, Tuple[ValueType, ...]] - """ - Verify that all typevars have been bound, and return a - `(inst, typevars)` pair. - """ - if len(self.typevars) < 1 + len(self.inst.other_typevars): - unb = ', '.join( - str(tv) for tv in - self.inst.other_typevars[len(self.typevars) - 1:]) - raise AssertionError("Unbound typevar {} in {}".format(unb, self)) - assert len(self.typevars) == 1 + len(self.inst.other_typevars) - return (self.inst, self.typevars) - - def __call__(self, *args): - """ - Create an `ast.Apply` AST node representing the application of this - instruction to the arguments. - """ - from .ast import Apply - return Apply(self, args) - - # Defining target ISAs. diff --git a/lib/cretonne/meta/cretonne/ast.py b/lib/cretonne/meta/cretonne/ast.py index b437148435..0212de7a0b 100644 --- a/lib/cretonne/meta/cretonne/ast.py +++ b/lib/cretonne/meta/cretonne/ast.py @@ -5,7 +5,7 @@ This module defines classes that can be used to create abstract syntax trees for patern matching an rewriting of cretonne instructions. """ from __future__ import absolute_import -import cretonne +from cdsl import instructions try: from typing import Union, Tuple # noqa @@ -20,7 +20,7 @@ class Def(object): Example: - >>> from .base import iadd_cout, iconst + >>> from base.instructions import iadd_cout, iconst >>> x = Var('x') >>> y = Var('y') >>> x << iconst(4) @@ -162,7 +162,7 @@ class Apply(Expr): instructions. This applies to both bound and unbound polymorphic instructions: - >>> from .base import jump, iadd + >>> from base.instructions import jump, iadd >>> jump('next', ()) Apply(jump, ('next', ())) >>> iadd.i32('x', 'y') @@ -174,12 +174,12 @@ class Apply(Expr): """ def __init__(self, inst, args): - # type: (Union[cretonne.Instruction, cretonne.BoundInstruction], Tuple[Expr, ...]) -> None # noqa - if isinstance(inst, cretonne.BoundInstruction): + # type: (instructions.MaybeBoundInst, Tuple[Expr, ...]) -> None # noqa + if isinstance(inst, instructions.BoundInstruction): self.inst = inst.inst self.typevars = inst.typevars else: - assert isinstance(inst, cretonne.Instruction) + assert isinstance(inst, instructions.Instruction) self.inst = inst self.typevars = () self.args = args diff --git a/lib/cretonne/meta/cretonne/legalize.py b/lib/cretonne/meta/cretonne/legalize.py index 189685dce1..dd534cf108 100644 --- a/lib/cretonne/meta/cretonne/legalize.py +++ b/lib/cretonne/meta/cretonne/legalize.py @@ -7,10 +7,10 @@ patterns that describe how base instructions can be transformed to other base instructions that are legal. """ from __future__ import absolute_import -from .base import iadd, iadd_cout, iadd_cin, iadd_carry -from .base import isub, isub_bin, isub_bout, isub_borrow -from .base import band, bor, bxor, isplit_lohi, iconcat_lohi -from .base import icmp +from base.instructions import iadd, iadd_cout, iadd_cin, iadd_carry +from base.instructions import isub, isub_bin, isub_bout, isub_borrow +from base.instructions import band, bor, bxor, isplit_lohi, iconcat_lohi +from base.instructions import icmp from .ast import Var from .xform import Rtl, XFormGroup diff --git a/lib/cretonne/meta/cretonne/test_ast.py b/lib/cretonne/meta/cretonne/test_ast.py index 71791db3f6..750142af0a 100644 --- a/lib/cretonne/meta/cretonne/test_ast.py +++ b/lib/cretonne/meta/cretonne/test_ast.py @@ -2,7 +2,7 @@ from __future__ import absolute_import from unittest import TestCase from doctest import DocTestSuite from . import ast -from .base import jump, iadd +from base.instructions import jump, iadd def load_tests(loader, tests, ignore): diff --git a/lib/cretonne/meta/cretonne/test_xform.py b/lib/cretonne/meta/cretonne/test_xform.py index b472a8f458..1609bb6c5f 100644 --- a/lib/cretonne/meta/cretonne/test_xform.py +++ b/lib/cretonne/meta/cretonne/test_xform.py @@ -1,8 +1,8 @@ from __future__ import absolute_import from unittest import TestCase from doctest import DocTestSuite +from base.instructions import iadd, iadd_imm, iconst from . import xform -from .base import iadd, iadd_imm, iconst from .ast import Var from .xform import Rtl, XForm diff --git a/lib/cretonne/meta/cretonne/xform.py b/lib/cretonne/meta/cretonne/xform.py index 6b9b37f513..f8f66de85c 100644 --- a/lib/cretonne/meta/cretonne/xform.py +++ b/lib/cretonne/meta/cretonne/xform.py @@ -54,7 +54,7 @@ class XForm(object): A legalization pattern must have a source pattern containing only a single instruction. - >>> from .base import iconst, iadd, iadd_imm + >>> from base.instructions import iconst, iadd, iadd_imm >>> a = Var('a') >>> c = Var('c') >>> v = Var('v') diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index aae46b5652..56846c90ab 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -7,7 +7,7 @@ import constant_hash from unique_table import UniqueTable, UniqueSeqTable import cdsl.types import cdsl.operands -import cretonne +from cdsl.formats import InstructionFormat def gen_formats(fmt): @@ -21,7 +21,7 @@ def gen_formats(fmt): fmt.doc_comment('and the `InstructionData` enums.') fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug)]') with fmt.indented('pub enum InstructionFormat {', '}'): - for f in cretonne.InstructionFormat.all_formats: + for f in InstructionFormat.all_formats: fmt.doc_comment(str(f)) fmt.line(f.name + ',') fmt.line() @@ -34,7 +34,7 @@ def gen_formats(fmt): "fn from(inst: &'a InstructionData) -> InstructionFormat {", '}'): with fmt.indented('match *inst {', '}'): - for f in cretonne.InstructionFormat.all_formats: + for f in InstructionFormat.all_formats: fmt.line(('InstructionData::{} {{ .. }} => ' + 'InstructionFormat::{},') .format(f.name, f.name)) @@ -55,7 +55,7 @@ def gen_arguments_method(fmt, is_mut): 'pub fn {f}(&{m}self) -> [&{m}[Value]; 2] {{' .format(f=method, m=mut), '}'): with fmt.indented('match *self {', '}'): - for f in cretonne.InstructionFormat.all_formats: + for f in InstructionFormat.all_formats: n = 'InstructionData::' + f.name has_varargs = cdsl.operands.VARIABLE_ARGS in f.kinds # Formats with both fixed and variable arguments delegate to @@ -118,7 +118,7 @@ def gen_instruction_data_impl(fmt): fmt.doc_comment('Get the opcode of this instruction.') with fmt.indented('pub fn opcode(&self) -> Opcode {', '}'): with fmt.indented('match *self {', '}'): - for f in cretonne.InstructionFormat.all_formats: + for f in InstructionFormat.all_formats: fmt.line( 'InstructionData::{} {{ opcode, .. }} => opcode,' .format(f.name)) @@ -126,7 +126,7 @@ def gen_instruction_data_impl(fmt): fmt.doc_comment('Type of the first result, or `VOID`.') with fmt.indented('pub fn first_type(&self) -> Type {', '}'): with fmt.indented('match *self {', '}'): - for f in cretonne.InstructionFormat.all_formats: + for f in InstructionFormat.all_formats: fmt.line( 'InstructionData::{} {{ ty, .. }} => ty,' .format(f.name)) @@ -135,7 +135,7 @@ def gen_instruction_data_impl(fmt): with fmt.indented( 'pub fn first_type_mut(&mut self) -> &mut Type {', '}'): with fmt.indented('match *self {', '}'): - for f in cretonne.InstructionFormat.all_formats: + for f in InstructionFormat.all_formats: fmt.line( 'InstructionData::{} {{ ref mut ty, .. }} => ty,' .format(f.name)) @@ -147,7 +147,7 @@ def gen_instruction_data_impl(fmt): with fmt.indented( 'pub fn second_result(&self) -> Option {', '}'): with fmt.indented('match *self {', '}'): - for f in cretonne.InstructionFormat.all_formats: + for f in InstructionFormat.all_formats: if f.multiple_results: fmt.line( 'InstructionData::' + f.name + @@ -164,7 +164,7 @@ def gen_instruction_data_impl(fmt): "pub fn second_result_mut<'a>(&'a mut self)" + " -> Option<&'a mut Value> {", '}'): with fmt.indented('match *self {', '}'): - for f in cretonne.InstructionFormat.all_formats: + for f in InstructionFormat.all_formats: if f.multiple_results: fmt.line( 'InstructionData::' + f.name + @@ -180,7 +180,7 @@ def gen_instruction_data_impl(fmt): with fmt.indented( 'pub fn typevar_operand(&self) -> Option {', '}'): with fmt.indented('match *self {', '}'): - for f in cretonne.InstructionFormat.all_formats: + for f in InstructionFormat.all_formats: n = 'InstructionData::' + f.name if f.typevar_operand is None: fmt.line(n + ' { .. } => None,') @@ -612,7 +612,7 @@ def gen_builder(insts, fmt): "pub trait InstBuilder<'f>: InstBuilderBase<'f> {", '}'): for inst in insts: gen_inst_builder(inst, fmt) - for f in cretonne.InstructionFormat.all_formats: + for f in InstructionFormat.all_formats: gen_format_constructor(f, fmt) diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index 6a05d1fa03..74aa0dca48 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -5,8 +5,8 @@ from __future__ import absolute_import import srcgen from unique_table import UniqueSeqTable import constant_hash +from cdsl import camel_case from cdsl.settings import BoolSetting, NumSetting, EnumSetting -from cretonne import camel_case from base import settings diff --git a/lib/cretonne/meta/isa/riscv/defs.py b/lib/cretonne/meta/isa/riscv/defs.py index 4fccbf09a6..2a3aa3cc9d 100644 --- a/lib/cretonne/meta/isa/riscv/defs.py +++ b/lib/cretonne/meta/isa/riscv/defs.py @@ -5,9 +5,9 @@ Commonly used definitions. """ from __future__ import absolute_import from cretonne import TargetISA, CPUMode -import cretonne.base +import base.instructions -isa = TargetISA('riscv', [cretonne.base.instructions]) +isa = TargetISA('riscv', [base.instructions.GROUP]) # CPU modes for 32-bit and 64-bit operation. RV32 = CPUMode('RV32', isa) diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index 807fdafc7a..0fc3a3ee8d 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -2,7 +2,7 @@ RISC-V Encodings. """ from __future__ import absolute_import -from cretonne import base +from base import instructions as base from .defs import RV32, RV64 from .recipes import OPIMM, OPIMM32, OP, OP32, R, Rshamt, I from .settings import use_m From cc86964ab3b2925fbb30a9fc7f2a238a58a29e02 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 12:29:17 -0800 Subject: [PATCH 0433/3084] Move ast, xform, and legalize modules. - cdsl.ast defines classes representing abstract syntax trees. - cdsl.xform defines classes for instruction transformations. - base.legalize defines legalization patterns. --- lib/cretonne/meta/{cretonne => base}/legalize.py | 12 ++++++------ lib/cretonne/meta/{cretonne => cdsl}/ast.py | 2 +- lib/cretonne/meta/cdsl/instructions.py | 4 ++-- lib/cretonne/meta/{cretonne => cdsl}/test_ast.py | 0 lib/cretonne/meta/{cretonne => cdsl}/test_xform.py | 0 lib/cretonne/meta/{cretonne => cdsl}/xform.py | 3 ++- lib/cretonne/meta/gen_legalizer.py | 6 +++--- 7 files changed, 14 insertions(+), 13 deletions(-) rename lib/cretonne/meta/{cretonne => base}/legalize.py (91%) rename lib/cretonne/meta/{cretonne => cdsl}/ast.py (99%) rename lib/cretonne/meta/{cretonne => cdsl}/test_ast.py (100%) rename lib/cretonne/meta/{cretonne => cdsl}/test_xform.py (100%) rename lib/cretonne/meta/{cretonne => cdsl}/xform.py (99%) diff --git a/lib/cretonne/meta/cretonne/legalize.py b/lib/cretonne/meta/base/legalize.py similarity index 91% rename from lib/cretonne/meta/cretonne/legalize.py rename to lib/cretonne/meta/base/legalize.py index dd534cf108..f054d16d9e 100644 --- a/lib/cretonne/meta/cretonne/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -7,12 +7,12 @@ patterns that describe how base instructions can be transformed to other base instructions that are legal. """ from __future__ import absolute_import -from base.instructions import iadd, iadd_cout, iadd_cin, iadd_carry -from base.instructions import isub, isub_bin, isub_bout, isub_borrow -from base.instructions import band, bor, bxor, isplit_lohi, iconcat_lohi -from base.instructions import icmp -from .ast import Var -from .xform import Rtl, XFormGroup +from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry +from .instructions import isub, isub_bin, isub_bout, isub_borrow +from .instructions import band, bor, bxor, isplit_lohi, iconcat_lohi +from .instructions import icmp +from cdsl.ast import Var +from cdsl.xform import Rtl, XFormGroup narrow = XFormGroup('narrow', """ diff --git a/lib/cretonne/meta/cretonne/ast.py b/lib/cretonne/meta/cdsl/ast.py similarity index 99% rename from lib/cretonne/meta/cretonne/ast.py rename to lib/cretonne/meta/cdsl/ast.py index 0212de7a0b..853437e4e5 100644 --- a/lib/cretonne/meta/cretonne/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -5,7 +5,7 @@ This module defines classes that can be used to create abstract syntax trees for patern matching an rewriting of cretonne instructions. """ from __future__ import absolute_import -from cdsl import instructions +from . import instructions try: from typing import Union, Tuple # noqa diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index da9f5bb179..8132f144f6 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -259,7 +259,7 @@ class Instruction(object): Create an `ast.Apply` AST node representing the application of this instruction to the arguments. """ - from cretonne.ast import Apply + from .ast import Apply return Apply(self, args) @@ -312,5 +312,5 @@ class BoundInstruction(object): Create an `ast.Apply` AST node representing the application of this instruction to the arguments. """ - from cretonne.ast import Apply + from .ast import Apply return Apply(self, args) diff --git a/lib/cretonne/meta/cretonne/test_ast.py b/lib/cretonne/meta/cdsl/test_ast.py similarity index 100% rename from lib/cretonne/meta/cretonne/test_ast.py rename to lib/cretonne/meta/cdsl/test_ast.py diff --git a/lib/cretonne/meta/cretonne/test_xform.py b/lib/cretonne/meta/cdsl/test_xform.py similarity index 100% rename from lib/cretonne/meta/cretonne/test_xform.py rename to lib/cretonne/meta/cdsl/test_xform.py diff --git a/lib/cretonne/meta/cretonne/xform.py b/lib/cretonne/meta/cdsl/xform.py similarity index 99% rename from lib/cretonne/meta/cretonne/xform.py rename to lib/cretonne/meta/cdsl/xform.py index f8f66de85c..164747fac9 100644 --- a/lib/cretonne/meta/cretonne/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -2,10 +2,11 @@ Instruction transformations. """ from __future__ import absolute_import -from .ast import Def, Var, Apply, Expr # noqa +from .ast import Def, Var, Apply try: from typing import Union, Iterator, Sequence, Iterable # noqa + from .ast import Expr # noqa DefApply = Union[Def, Apply] except ImportError: pass diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index bd4ff926eb..409ebb7cf2 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -9,12 +9,12 @@ the input instruction. """ from __future__ import absolute_import from srcgen import Formatter -import cretonne.legalize as legalize -from cretonne.ast import Def # noqa -from cretonne.xform import XForm, XFormGroup # noqa +from base import legalize try: from typing import Sequence # noqa + from cdsl.ast import Def # noqa + from cdsl.xform import XForm, XFormGroup # noqa except ImportError: pass From 8846cb810559444bdbb4c39b07b9f0d3290bc57f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 13:21:05 -0800 Subject: [PATCH 0434/3084] Move ISA definitions into cdsl.isa. The cretonne package is now split into two: cdsl and base. --- .../{cretonne/__init__.py => cdsl/isa.py} | 57 +++++++++---------- lib/cretonne/meta/isa/__init__.py | 2 +- lib/cretonne/meta/isa/riscv/defs.py | 2 +- lib/cretonne/meta/isa/riscv/recipes.py | 2 +- 4 files changed, 29 insertions(+), 34 deletions(-) rename lib/cretonne/meta/{cretonne/__init__.py => cdsl/isa.py} (82%) diff --git a/lib/cretonne/meta/cretonne/__init__.py b/lib/cretonne/meta/cdsl/isa.py similarity index 82% rename from lib/cretonne/meta/cretonne/__init__.py rename to lib/cretonne/meta/cdsl/isa.py index fa1da4d302..7f59a971db 100644 --- a/lib/cretonne/meta/cretonne/__init__.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -1,29 +1,18 @@ -""" -Cretonne meta language module. - -This module provides classes and functions used to describe Cretonne -instructions. -""" +"""Defining instruction set architectures.""" from __future__ import absolute_import -import importlib -from cdsl.predicates import And -from cdsl.typevar import TypeVar +from .predicates import And # The typing module is only required by mypy, and we don't use these imports # outside type comments. try: from typing import Tuple, Union, Any, Iterable, Sequence, TYPE_CHECKING # noqa - from cdsl.predicates import Predicate, FieldPredicate # noqa - from cdsl.instructions import MaybeBoundInst # noqa - AnyPredicate = Union['Predicate', 'FieldPredicate'] + from .instructions import MaybeBoundInst, InstructionGroup, InstructionFormat # noqa + from .predicates import Predicate, FieldPredicate # noqa + from .settings import SettingGroup # noqa + from .types import ValueType # noqa + AnyPredicate = Union[Predicate, FieldPredicate] except ImportError: - TYPE_CHECKING = False - -if TYPE_CHECKING: - from cdsl.typevar import TypeVar # noqa - - -# Defining target ISAs. + pass class TargetISA(object): @@ -38,12 +27,14 @@ class TargetISA(object): """ def __init__(self, name, instruction_groups): + # type: (str, Sequence[InstructionGroup]) -> None self.name = name - self.settings = None + self.settings = None # type: SettingGroup self.instruction_groups = instruction_groups - self.cpumodes = list() + self.cpumodes = list() # type: List[CPUMode] def finish(self): + # type: () -> TargetISA """ Finish the definition of a target ISA after adding all CPU modes and settings. @@ -58,11 +49,12 @@ class TargetISA(object): return self def _collect_encoding_recipes(self): + # type: () -> None """ Collect and number all encoding recipes in use. """ - self.all_recipes = list() - rcps = set() + self.all_recipes = list() # type: List[EncRecipe] + rcps = set() # type: Set[EncRecipe] for cpumode in self.cpumodes: for enc in cpumode.encodings: recipe = enc.recipe @@ -72,6 +64,7 @@ class TargetISA(object): self.all_recipes.append(recipe) def _collect_predicates(self): + # type: () -> None """ Collect and number all predicates in use. @@ -81,8 +74,8 @@ class TargetISA(object): Ensures that all ISA predicates have an assigned bit number in `self.settings`. """ - self.all_instps = list() - instps = set() + self.all_instps = list() # type: List[AnyPredicate] + instps = set() # type: Set[AnyPredicate] for cpumode in self.cpumodes: for enc in cpumode.encodings: instp = enc.instp @@ -111,12 +104,14 @@ class CPUMode(object): """ def __init__(self, name, isa): + # type: (str, TargetISA) -> None self.name = name self.isa = isa - self.encodings = [] + self.encodings = [] # type: List[Encoding] isa.cpumodes.append(self) def __str__(self): + # type: () -> str return self.name def enc(self, *args, **kwargs): @@ -142,14 +137,17 @@ class EncRecipe(object): """ def __init__(self, name, format, instp=None, isap=None): + # type: (str, InstructionFormat, AnyPredicate, AnyPredicate) -> None self.name = name self.format = format self.instp = instp self.isap = isap if instp: assert instp.predicate_context() == format + self.number = None # type: int def __str__(self): + # type: () -> str return self.name @@ -185,9 +183,11 @@ class Encoding(object): self.isap = And.combine(recipe.isap, isap) def __str__(self): + # type: () -> str return '[{}#{:02x}]'.format(self.recipe, self.encbits) def ctrl_typevar(self): + # type: () -> ValueType """ Get the controlling type variable for this encoding or `None`. """ @@ -195,8 +195,3 @@ class Encoding(object): return self.typevars[0] else: return None - - -# Import the fixed instruction formats now so they can be added to the -# registry. -importlib.import_module('base.formats') diff --git a/lib/cretonne/meta/isa/__init__.py b/lib/cretonne/meta/isa/__init__.py index 9412f6502c..1bef425c58 100644 --- a/lib/cretonne/meta/isa/__init__.py +++ b/lib/cretonne/meta/isa/__init__.py @@ -6,8 +6,8 @@ The :py:mod:`isa` package contains sub-packages for each target instruction set architecture supported by Cretonne. """ from __future__ import absolute_import +from cdsl.isa import TargetISA # noqa from . import riscv -from cretonne import TargetISA # noqa def all_isas(): diff --git a/lib/cretonne/meta/isa/riscv/defs.py b/lib/cretonne/meta/isa/riscv/defs.py index 2a3aa3cc9d..7a487482d3 100644 --- a/lib/cretonne/meta/isa/riscv/defs.py +++ b/lib/cretonne/meta/isa/riscv/defs.py @@ -4,7 +4,7 @@ RISC-V definitions. Commonly used definitions. """ from __future__ import absolute_import -from cretonne import TargetISA, CPUMode +from cdsl.isa import TargetISA, CPUMode import base.instructions isa = TargetISA('riscv', [base.instructions.GROUP]) diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index b4f561f8e2..ae8d5b3add 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -9,7 +9,7 @@ instruction formats described in the reference: Version 2.1 """ from __future__ import absolute_import -from cretonne import EncRecipe +from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt from base.formats import Binary, BinaryImm From a35756bbcca19aebde21fd3a5cb83a95f4a24e7d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 13:30:17 -0800 Subject: [PATCH 0435/3084] Clean up meta-language reference after module splitup. --- cranelift/docs/metaref.rst | 41 +++++++++++++++++++++----------------- 1 file changed, 23 insertions(+), 18 deletions(-) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 190945c61d..ec4555a8d7 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -17,7 +17,7 @@ steps: 1. The Python modules are imported. This has the effect of building static data structures in global variables in the modules. These static data structures in the :mod:`base` and :mod:`isa` packages use the classes in the - :mod:`cdsl` module to describe instruction sets and other properties. + :mod:`cdsl` package to describe instruction sets and other properties. 2. The static data structures are processed to produce Rust source code and constant tables. @@ -37,7 +37,7 @@ Settings are used by the environment embedding Cretonne to control the details of code generation. Each setting is defined in the meta language so a compact and consistent Rust representation can be generated. Shared settings are defined in the :mod:`base.settings` module. Some settings are specific to a target ISA, -and defined in a `settings` module under the appropriate +and defined in a :file:`settings.py` module under the appropriate :file:`lib/cretonne/meta/isa/*` directory. Settings can take boolean on/off values, small numbers, or explicitly enumerated @@ -68,11 +68,11 @@ a module looks like this:: group.close(globals()) +.. module:: cdsl.instructions + Instruction descriptions ======================== -.. module:: cdsl.instructions - New instructions are defined as instances of the :class:`Instruction` class. As instruction instances are created, they are added to the currently open :class:`InstructionGroup`. @@ -96,15 +96,17 @@ must be instances of the :class:`Operand` class. Cretonne uses two separate type systems for operand kinds and SSA values. +.. module:: cdsl.typevar + Type variables -------------- -Instruction descriptions can be made polymorphic by using :class:`Operand` -instances that refer to a *type variable* instead of a concrete value type. -Polymorphism only works for SSA value operands. Other operands have a fixed -operand kind. +Instruction descriptions can be made polymorphic by using +:class:`cdsl.operands.Operand` instances that refer to a *type variable* +instead of a concrete value type. Polymorphism only works for SSA value +operands. Other operands have a fixed operand kind. -.. autoclass:: cdsl.typevar.TypeVar +.. autoclass:: TypeVar :members: If multiple operands refer to the same type variable they will be required to @@ -145,10 +147,11 @@ indicated with an instance of :class:`ImmediateKind`. Entity references ----------------- +.. currentmodule:: cdsl.operands + Instruction operands can also refer to other entities in the same function. This can be extended basic blocks, or entities declared in the function preamble. -.. currentmodule:: cdsl.operands .. autoclass:: EntityRefKind .. automodule:: base.entities @@ -157,10 +160,11 @@ can be extended basic blocks, or entities declared in the function preamble. Value types ----------- -Concrete value types are represented as instances of :class:`cdsl.types.ValueType`. There are -subclasses to represent scalar and vector types. - .. currentmodule:: cdsl.types + +Concrete value types are represented as instances of :class:`ValueType`. There +are subclasses to represent scalar and vector types. + .. autoclass:: ValueType .. inheritance-diagram:: ValueType ScalarType VectorType IntType FloatType BoolType :parts: 1 @@ -182,14 +186,14 @@ There are no predefined vector types, but they can be created as needed with the :func:`ScalarType.by` function. +.. module:: cdsl.operands + Instruction representation ========================== -.. module:: cdsl.operands - The Rust in-memory representation of instructions is derived from the instruction descriptions. Part of the representation is generated, and part is -written as Rust code in the `cretonne.instructions` module. The instruction +written as Rust code in the ``cretonne.instructions`` module. The instruction representation depends on the input operand kinds and whether the instruction can produce multiple results. @@ -255,7 +259,7 @@ controlling type variable, or it can vary independently of the other operands. Encodings ========= -.. currentmodule:: cretonne +.. currentmodule:: cdsl.isa Encodings describe how Cretonne instructions are mapped to binary machine code for the target architecture. After the legalization pass, all remaining @@ -345,12 +349,13 @@ encodings only need the recipe predicates. .. autoclass:: EncRecipe +.. module:: cdsl.isa Targets ======= Cretonne can be compiled with support for multiple target instruction set -architectures. Each ISA is represented by a :py:class:`cretonne.TargetISA` instance. +architectures. Each ISA is represented by a :py:class:`cdsl.isa.TargetISA` instance. .. autoclass:: TargetISA From 5e75b22c01e077a6b989e2ba13e6c73e0a5d1440 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 14:41:13 -0800 Subject: [PATCH 0436/3084] Assign a type variable to all VALUE operands. A few operands have a fixed type assigned. Create a singleton type variable for these exceptions. Most instructions are polymorphic, so this is a little overhead. Eliminate the Operand.typ field and replace it with an Operand.typevar field which is always a TypeVar, but which only exists in VALUE operands. --- lib/cretonne/meta/cdsl/instructions.py | 14 ++++---- lib/cretonne/meta/cdsl/operands.py | 15 +++++---- lib/cretonne/meta/cdsl/test_typevar.py | 16 +++++++++ lib/cretonne/meta/cdsl/types.py | 9 ++---- lib/cretonne/meta/cdsl/typevar.py | 33 +++++++++++++++++++ lib/cretonne/meta/gen_instr.py | 45 ++++++++++++++------------ 6 files changed, 93 insertions(+), 39 deletions(-) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 8132f144f6..ee6b9c4a59 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -130,10 +130,10 @@ class Instruction(object): """ poly_ins = [ i for i in self.format.value_operands - if self.ins[i].typ.free_typevar()] + if self.ins[i].typevar.free_typevar()] poly_outs = [ i for i, o in enumerate(self.outs) - if o.typ.free_typevar()] + if o.is_value() and o.typevar.free_typevar()] self.is_polymorphic = len(poly_ins) > 0 or len(poly_outs) > 0 if not self.is_polymorphic: return @@ -143,7 +143,7 @@ class Instruction(object): typevar_error = None if self.format.typevar_operand is not None: try: - tv = self.ins[self.format.typevar_operand].typ + tv = self.ins[self.format.typevar_operand].typevar if tv is tv.free_typevar(): self.other_typevars = self._verify_ctrl_typevar(tv) self.ctrl_typevar = tv @@ -160,7 +160,7 @@ class Instruction(object): else: raise RuntimeError( "typevar_operand must be a free type variable") - tv = self.outs[0].typ + tv = self.outs[0].typevar if tv is not tv.free_typevar(): raise RuntimeError("first result must be a free type variable") self.other_typevars = self._verify_ctrl_typevar(tv) @@ -181,7 +181,7 @@ class Instruction(object): other_tvs = [] # Check value inputs. for opidx in self.format.value_operands: - typ = self.ins[opidx].typ + typ = self.ins[opidx].typevar tv = typ.free_typevar() # Non-polymorphic or derived form ctrl_typevar is OK. if tv is None or tv is ctrl_typevar: @@ -200,7 +200,9 @@ class Instruction(object): # Check outputs. for result in self.outs: - typ = result.typ + if not result.is_value(): + continue + typ = result.typevar tv = typ.free_typevar() # Non-polymorphic or derived from ctrl_typevar is OK. if tv is None or tv is ctrl_typevar: diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index f379357611..2fcc22b0c0 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -40,10 +40,6 @@ class OperandKind(object): # type: () -> str return 'OperandKind({})'.format(self.name) - def free_typevar(self): - # Return the free typevariable controlling the type of this operand. - return None - #: An SSA value operand. This is a value defined by another instruction. VALUE = OperandKind( 'value', """ @@ -129,11 +125,15 @@ class Operand(object): # type: (str, OperandSpec, str) -> None self.name = name self.__doc__ = doc - self.typ = typ + + # Decode the operand spec and set self.kind. + # Only VALUE operands have a typevar member. if isinstance(typ, ValueType): self.kind = VALUE + self.typevar = TypeVar.singleton(typ) elif isinstance(typ, TypeVar): self.kind = VALUE + self.typevar = typ else: assert isinstance(typ, OperandKind) self.kind = typ @@ -142,8 +142,9 @@ class Operand(object): # type: () -> str if self.__doc__: return self.__doc__ - else: - return self.typ.__doc__ + if self.kind is VALUE: + return self.typevar.__doc__ + return self.kind.__doc__ def __str__(self): # type: () -> str diff --git a/lib/cretonne/meta/cdsl/test_typevar.py b/lib/cretonne/meta/cdsl/test_typevar.py index a841acfdb9..7dae18221a 100644 --- a/lib/cretonne/meta/cdsl/test_typevar.py +++ b/lib/cretonne/meta/cdsl/test_typevar.py @@ -3,6 +3,7 @@ from unittest import TestCase from doctest import DocTestSuite from . import typevar from .typevar import TypeSet, TypeVar +from base.types import i32 def load_tests(loader, tests, ignore): @@ -62,3 +63,18 @@ class TestTypeVar(TestCase): self.assertEqual(str(x3.double_width()), '`DoubleWidth(x3)`') with self.assertRaises(AssertionError): x3.half_width() + + def test_singleton(self): + x = TypeVar.singleton(i32) + self.assertEqual(str(x), '`i32`') + self.assertEqual(x.type_set.min_int, 32) + self.assertEqual(x.type_set.max_int, 32) + self.assertEqual(x.type_set.min_lanes, 1) + self.assertEqual(x.type_set.max_lanes, 1) + + x = TypeVar.singleton(i32.by(4)) + self.assertEqual(str(x), '`i32x4`') + self.assertEqual(x.type_set.min_int, 32) + self.assertEqual(x.type_set.max_int, 32) + self.assertEqual(x.type_set.min_lanes, 4) + self.assertEqual(x.type_set.max_lanes, 4) diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py index 3d5a5dcd38..62334b7dd7 100644 --- a/lib/cretonne/meta/cdsl/types.py +++ b/lib/cretonne/meta/cdsl/types.py @@ -30,8 +30,9 @@ class ValueType(object): # type: () -> str return self.name - def free_typevar(self): - return None + def rust_name(self): + # type: () -> str + return 'types::' + self.name.upper() @staticmethod def by_name(name): @@ -63,10 +64,6 @@ class ScalarType(ValueType): # type: () -> str return 'ScalarType({})'.format(self.name) - def rust_name(self): - # type: () -> str - return 'types::' + self.name.upper() - def by(self, lanes): # type: (int) -> VectorType """ diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 2c10338bbf..9803f1afc6 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -6,6 +6,7 @@ polymorphic by using type variables. """ from __future__ import absolute_import import math +from . import types try: from typing import Tuple, Union # noqa @@ -242,6 +243,7 @@ class TypeVar(object): # type: (str, str, BoolInterval, BoolInterval, BoolInterval, bool, BoolInterval, TypeVar, str) -> None # noqa self.name = name self.__doc__ = doc + self.singleton_type = None # type: types.ValueType self.is_derived = isinstance(base, TypeVar) if base: assert self.is_derived @@ -258,6 +260,34 @@ class TypeVar(object): floats=floats, bools=bools) + @staticmethod + def singleton(typ): + # type: (types.ValueType) -> TypeVar + """Create a type variable that can only assume a single type.""" + if isinstance(typ, types.VectorType): + scalar = typ.base + lanes = (typ.lanes, typ.lanes) + elif isinstance(typ, types.ScalarType): + scalar = typ + lanes = (1, 1) + + ints = None + floats = None + bools = None + + if isinstance(scalar, types.IntType): + ints = (scalar.bits, scalar.bits) + elif isinstance(scalar, types.FloatType): + floats = (scalar.bits, scalar.bits) + elif isinstance(scalar, types.BoolType): + bools = (scalar.bits, scalar.bits) + + tv = TypeVar( + typ.name, 'typeof({})'.format(typ), + ints, floats, bools, simd=lanes) + tv.singleton_type = typ + return tv + def __str__(self): # type: () -> str return "`{}`".format(self.name) @@ -317,5 +347,8 @@ class TypeVar(object): # type: () -> TypeVar if self.is_derived: return self.base + elif self.singleton_type: + # A singleton type variable is not a proper free variable. + return None else: return self diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 56846c90ab..4f240ab6b1 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -5,10 +5,14 @@ from __future__ import absolute_import import srcgen import constant_hash from unique_table import UniqueTable, UniqueSeqTable +from cdsl.operands import ImmediateKind import cdsl.types -import cdsl.operands from cdsl.formats import InstructionFormat +from cdsl.instructions import Instruction # noqa +from cdsl.operands import Operand # noqa +from cdsl.typevar import TypeVar # noqa + def gen_formats(fmt): # type: (srcgen.Formatter) -> None @@ -302,6 +306,7 @@ def gen_opcodes(groups, fmt): def get_constraint(op, ctrl_typevar, type_sets): + # type: (Operand, TypeVar, UniqueTable) -> str """ Get the value type constraint for an SSA value operand, where `ctrl_typevar` is the controlling type variable. @@ -312,22 +317,22 @@ def get_constraint(op, ctrl_typevar, type_sets): - `Free(idx)` where `idx` is an index into `type_sets`. - `Same`, `Lane`, `AsBool` for controlling typevar-derived constraints. """ - assert op.kind is cdsl.operands.VALUE - t = op.typ + assert op.is_value() + tv = op.typevar # A concrete value type. - if isinstance(t, cdsl.types.ValueType): - return 'Concrete({})'.format(t.rust_name()) + if tv.singleton_type: + return 'Concrete({})'.format(tv.singleton_type.rust_name()) - if t.free_typevar() is not ctrl_typevar: - assert not t.is_derived - return 'Free({})'.format(type_sets.add(t.type_set)) + if tv.free_typevar() is not ctrl_typevar: + assert not tv.is_derived + return 'Free({})'.format(type_sets.add(tv.type_set)) - if t.is_derived: - assert t.base is ctrl_typevar, "Not derived directly from ctrl_typevar" - return t.derived_func + if tv.is_derived: + assert tv.base is ctrl_typevar, "Not derived from ctrl_typevar" + return tv.derived_func - assert t is ctrl_typevar + assert tv is ctrl_typevar return 'Same' @@ -486,6 +491,7 @@ def gen_member_inits(iform, fmt): def gen_inst_builder(inst, fmt): + # type: (Instruction, srcgen.Formatter) -> None """ Emit a method for generating the instruction `inst`. @@ -502,10 +508,10 @@ def gen_inst_builder(inst, fmt): if inst.is_polymorphic and not inst.use_typevar_operand: args.append('{}: Type'.format(inst.ctrl_typevar.name)) - tmpl_types = list() - into_args = list() + tmpl_types = list() # type: List[str] + into_args = list() # type: List[str] for op in inst.ins: - if isinstance(op.kind, cdsl.operands.ImmediateKind): + if isinstance(op.kind, ImmediateKind): t = 'T{}{}'.format(1 + len(tmpl_types), op.kind.name) tmpl_types.append('{}: Into<{}>'.format(t, op.kind.rust_type)) into_args.append(op.name) @@ -553,7 +559,7 @@ def gen_inst_builder(inst, fmt): # The format constructor will resolve the result types from the # type var. args.append('ctrl_typevar') - elif inst.outs[inst.value_results[0]].typ == inst.ctrl_typevar: + elif inst.outs[inst.value_results[0]].typevar == inst.ctrl_typevar: # The format constructor expects a simple result type. # No type transformation needed from the controlling type # variable. @@ -567,13 +573,12 @@ def gen_inst_builder(inst, fmt): else: # This non-polymorphic instruction has a fixed result type. args.append( - 'types::' + - inst.outs[inst.value_results[0]].typ.name.upper()) + inst.outs[inst.value_results[0]] + .typevar.singleton_type.rust_name()) args.extend(op.name for op in inst.ins) - args = ', '.join(args) # Call to the format constructor, - fcall = 'self.{}({})'.format(inst.format.name, args) + fcall = 'self.{}({})'.format(inst.format.name, ', '.join(args)) if len(inst.value_results) == 0: fmt.line(fcall + '.0') From cb631dc5789ddd8e8de276cabd4e44d52e91b53b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 8 Nov 2016 15:26:30 -0800 Subject: [PATCH 0437/3084] Fix doc build. --- cranelift/docs/cton_domain.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index 47c7c2996c..2f56b5625e 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -305,11 +305,19 @@ class InstDocumenter(sphinx.ext.autodoc.Documenter): # Add inputs and outputs. for op in self.object.ins: + if op.is_value(): + typ = op.typevar + else: + typ = op.kind self.add_line(u':in {} {}: {}'.format( - op.typ.name, op.name, op.get_doc()), sourcename) + typ, op.name, op.get_doc()), sourcename) for op in self.object.outs: + if op.is_value(): + typ = op.typevar + else: + typ = op.kind self.add_line(u':out {} {}: {}'.format( - op.typ.name, op.name, op.get_doc()), sourcename) + typ, op.name, op.get_doc()), sourcename) # Document type inference for polymorphic instructions. if self.object.is_polymorphic: From d22541dfa201ef3fa0f2b0e8e3fdaa0953f4fdc4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 9 Nov 2016 11:17:55 -0800 Subject: [PATCH 0438/3084] Add TypeVar.derived() function. Add TypeVar constants representing the available type functions, and a TypeVar.derived() static method which creates a derived TypeVar. Keep the existing non-parametric methods for creating derived type variables. Add a method for converting a free type variable to a derived one. --- lib/cretonne/meta/cdsl/typevar.py | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 9803f1afc6..66786f53cf 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -292,6 +292,27 @@ class TypeVar(object): # type: () -> str return "`{}`".format(self.name) + # Supported functions for derived type variables. + SAMEAS = 'SameAs' + LANEOF = 'LaneOf' + ASBOOL = 'AsBool' + HALFWIDTH = 'HalfWidth' + DOUBLEWIDTH = 'DoubleWidth' + + @staticmethod + def derived(base, derived_func): + # type: (TypeVar, str) -> TypeVar + """Create a type variable that is a function of another.""" + return TypeVar(None, None, base=base, derived_func=derived_func) + + def change_to_derived(self, base, derived_func): + # type: (TypeVar, str) -> None + """Change this type variable into a derived one.""" + self.type_set = None + self.is_derived = True + self.base = base + self.derived_func = derived_func + def lane_of(self): # type: () -> TypeVar """ @@ -301,7 +322,7 @@ class TypeVar(object): When this type variable assumes a scalar type, the derived type will be the same scalar type. """ - return TypeVar(None, None, base=self, derived_func='LaneOf') + return TypeVar.derived(self, self.LANEOF) def as_bool(self): # type: () -> TypeVar @@ -309,7 +330,7 @@ class TypeVar(object): Return a derived type variable that has the same vector geometry as this type variable, but with boolean lanes. Scalar types map to `b1`. """ - return TypeVar(None, None, base=self, derived_func='AsBool') + return TypeVar.derived(self, self.ASBOOL) def half_width(self): # type: () -> TypeVar @@ -325,7 +346,7 @@ class TypeVar(object): if ts.min_bool: assert ts.min_bool > 8, "Can't halve all boolean types" - return TypeVar(None, None, base=self, derived_func='HalfWidth') + return TypeVar.derived(self, self.HALFWIDTH) def double_width(self): # type: () -> TypeVar @@ -341,7 +362,7 @@ class TypeVar(object): if ts.max_bool: assert ts.max_bool < MAX_BITS, "Can't double all boolean types." - return TypeVar(None, None, base=self, derived_func='DoubleWidth') + return TypeVar.derived(self, self.DOUBLEWIDTH) def free_typevar(self): # type: () -> TypeVar From 129fd4b06e4e5d39ab24b08395efa85e856d611e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 10 Nov 2016 10:40:53 -0800 Subject: [PATCH 0439/3084] Add TypeVar.strip_sameas(). Strips out any type variable copies from an expression. --- lib/cretonne/meta/cdsl/typevar.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 66786f53cf..c2aeb5de97 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -313,6 +313,19 @@ class TypeVar(object): self.base = base self.derived_func = derived_func + def strip_sameas(self): + # type: () -> TypeVar + """ + Strip any `SAMEAS` functions from this typevar. + + Also rewrite any `SAMEAS` functions nested under this typevar. + """ + if self.is_derived: + self.base = self.base.strip_sameas() + if self.derived_func == self.SAMEAS: + return self.base + return self + def lane_of(self): # type: () -> TypeVar """ From 38f4b9a7c8b59810d92a861bd92e619548f826a7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 9 Nov 2016 15:19:03 -0800 Subject: [PATCH 0440/3084] Add a TypeVar.constrain_types() function. This reduces the set of types a type variable can assume. This implementation is not complete yet, so it may yield type sets that are too large. --- lib/cretonne/meta/cdsl/typevar.py | 32 +++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index c2aeb5de97..e5bf8f42a3 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -379,6 +379,9 @@ class TypeVar(object): def free_typevar(self): # type: () -> TypeVar + """ + Get the free type variable controlling this one. + """ if self.is_derived: return self.base elif self.singleton_type: @@ -386,3 +389,32 @@ class TypeVar(object): return None else: return self + + def constrain_types(self, other): + # type: (TypeVar) -> None + """ + Constrain the range of types this variable can assume to a subset of + those `other` can assume. + + If this is a SAMEAS-derived type variable, constrain the base instead. + """ + a = self.strip_sameas() + b = other.strip_sameas() + if a is b: + return + + if not a.is_derived and not b.is_derived: + a.type_set &= b.type_set + # TODO: What if a.type_set becomes empty? + if not a.singleton_type: + a.singleton_type = b.singleton_type + return + + # TODO: Implement constraints for derived type variables. + # + # If a and b are both derived with the same derived_func, we could say + # `a.base.constrain_types(b.base)`, but unless the derived_func is + # injective, that may constrain `a.base` more than necessary. + # + # For the fully general case, we would need to compute an image typeset + # for `b` and propagate a `a.derived_func` pre-image to `a.base`. From 68c9f20b773791a2db28a71ed7937c264241d755 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 9 Nov 2016 15:05:11 -0800 Subject: [PATCH 0441/3084] Infer type constraints on patterns. Each instruction used in a pattern has constraints on the types of its operands. These constraints are expressed as symbolic type variables. Compute type variables for each variable used in a transformation pattern. Some are free type variables, and some are derived from the free type variables. The type variables associated with variables can be used for computing the result types of replacement instructions that don't support simple forward type inference from their inputs. The type sets computed by this patch are conservatively too large, so they can't yet be used to type check patterns. --- lib/cretonne/meta/cdsl/ast.py | 174 ++++++++++++++++++++++++++++++ lib/cretonne/meta/cdsl/typevar.py | 19 ++++ lib/cretonne/meta/cdsl/xform.py | 61 +++++++++++ 3 files changed, 254 insertions(+) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index 853437e4e5..d8b3b7b875 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -6,6 +6,7 @@ for patern matching an rewriting of cretonne instructions. """ from __future__ import absolute_import from . import instructions +from .typevar import TypeVar try: from typing import Union, Tuple # noqa @@ -90,6 +91,12 @@ class Var(Expr): self.src_def = None # type: Def # The `Def` defining this variable in a destination pattern. self.dst_def = None # type: Def + # TypeVar representing the type of this variable. + self.typevar = None # type: TypeVar + # The original 'typeof(x)' type variable that was created for this Var. + # This one doesn't change. `self.typevar` above may be joined with + # other typevars. + self.original_typevar = None # type: TypeVar def __str__(self): # type: () -> str @@ -153,6 +160,173 @@ class Var(Expr): # type: () -> bool return not self.src_def and self.dst_def + def get_typevar(self): + # type: () -> TypeVar + """Get the type variable representing the type of this variable.""" + if not self.typevar: + # Create a TypeVar allowing all types. + tv = TypeVar( + 'typeof_{}'.format(self), + 'Type of the pattern variable `{}`'.format(self), + ints=True, floats=True, bools=True, + scalars=True, simd=True) + self.original_typevar = tv + self.typevar = tv + return self.typevar + + def link_typevar(self, base, derived_func): + # type: (TypeVar, str) -> None + """ + Link the type variable on this Var to the type variable `base` using + `derived_func`. + """ + self.original_typevar = None + self.typevar.change_to_derived(base, derived_func) + # Possibly eliminate redundant SAMEAS links. + self.typevar = self.typevar.strip_sameas() + + def has_free_typevar(self): + # type: () -> bool + """ + Check if this variable has a free type variable. + + If not, the type of this variable is computed from the type of another + variable. + """ + if not self.typevar or self.typevar.is_derived: + return False + return self.typevar is self.original_typevar + + def constrain_typevar(self, sym_typevar, sym_ctrl, ctrl_var): + # type: (TypeVar, TypeVar, Var) -> None + """ + Constrain the set of allowed types for this variable. + + Merge type variables for the involved variables to minimize the set for + free type variables. + + Suppose we're looking at an instruction defined like this: + + c = Operand('c', TxN.as_bool()) + x = Operand('x', TxN) + y = Operand('y', TxN) + a = Operand('a', TxN) + vselect = Instruction('vselect', ins=(c, x, y), outs=a) + + And suppose the instruction is used in a pattern like this: + + v0 << vselect(v1, v2, v3) + + We want to reconcile the types of the variables v0-v3 with the + constraints from the definition of vselect. This means that v0, v2, and + v3 must all have the same type, and v1 must have the type + `typeof(v2).as_bool()`. + + The types are reconciled by calling this function once for each + input/output operand on the instruction in the pattern with these + arguments. + + :param sym_typevar: Symbolic type variable constraining this variable + in the definition of the instruction. + :param sym_ctrl: Controlling type variable of `sym_typevar` in the + definition of the instruction. + :param ctrl_var: Variable determining the type of `sym_ctrl`. + + When processing `v1` as used in the pattern above, we would get: + + - self: v1 + - sym_typevar: TxN.as_bool() + - sym_ctrl: TxN + - ctrl_var: v2 + + Here, 'v2' represents the controlling variable because of how the + `Ternary` instruction format is defined with `typevar_operand=1`. + """ + # First check if sym_typevar is tied to the controlling type variable + # in the instruction definition. We also allow free type variables on + # instruction inputs that can't be tied to anything else. + # + # This also covers non-polymorphic instructions and other cases where + # we don't have a Var representing the controlling type variable. + sym_free_var = sym_typevar.free_typevar() + if not sym_free_var or sym_free_var is not sym_ctrl or not ctrl_var: + # Just constrain our type to be compatible with the required + # typeset. + self.get_typevar().constrain_types(sym_typevar) + return + + # Now sym_typevar is known to be tied to (or identical to) the + # controlling type variable. + + if not self.typevar: + # If this variable is not yet constrained, just infer its type and + # link it to the controlling type variable. + if not sym_typevar.is_derived: + assert sym_typevar is sym_ctrl + # Identity mapping. + # Note that `self == ctrl_var` is both possible and common. + self.typevar = ctrl_var.get_typevar() + else: + assert self is not ctrl_var, ( + 'Impossible type constraints for {}: {}' + .format(self, sym_typevar)) + # Create a derived type variable identical to sym_typevar, but + # with a different base. + self.typevar = TypeVar.derived( + ctrl_var.get_typevar(), + sym_typevar.derived_func) + # Match the type set constraints of the instruction. + self.typevar.constrain_types(sym_typevar) + return + + # We already have a self.typevar describing our constraints. We need to + # reconcile with the additional constraints. + + # It's likely that ctrl_var and self already share a type + # variable. (Often because `ctrl_var == self`). + if ctrl_var.typevar == self.typevar: + return + + if not sym_typevar.is_derived: + assert sym_typevar is sym_ctrl + # sym_typevar is a direct use of sym_ctrl, so we need to reconcile + # self with ctrl_var. + assert not sym_typevar.is_derived + self.typevar.constrain_types(sym_typevar) + + # It's possible that ctrl_var has not yet been assigned a type + # variable. + if not ctrl_var.typevar: + ctrl_var.typevar = self.typevar + return + + # We can also bind variables with a free type variable to another + # variable. Prefer to do this to temps because they aren't allowed + # to be free, + if self.is_temp() and self.has_free_typevar(): + self.link_typevar(ctrl_var.typevar, TypeVar.SAMEAS) + return + if ctrl_var.is_temp() and ctrl_var.has_free_typevar(): + ctrl_var.link_typevar(self.typevar, TypeVar.SAMEAS) + return + if self.has_free_typevar(): + self.link_typevar(ctrl_var.typevar, TypeVar.SAMEAS) + return + if ctrl_var.has_free_typevar(): + ctrl_var.link_typevar(self.typevar, TypeVar.SAMEAS) + return + + # TODO: Other cases are harder to handle. + # + # - If either variable is an independent free type variable, it + # should be changed to be linked to the other. + # - If both variable are free, we should pick one to link to the + # other. In particular, if one is a temp, it should be linked. + else: + # sym_typevar is derived from sym_ctrl. + # TODO: Other cases are harder to handle. + pass + class Apply(Expr): """ diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index e5bf8f42a3..30986c5db8 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -292,6 +292,25 @@ class TypeVar(object): # type: () -> str return "`{}`".format(self.name) + def __repr__(self): + # type: () -> str + if self.is_derived: + return ( + 'TypeVar({}, base={}, derived_func={})' + .format(self.name, self.base, self.derived_func)) + else: + return ( + 'TypeVar({}, {})' + .format(self.name, self.type_set)) + + def __eq__(self, other): + if self.is_derived and other.is_derived: + return ( + self.derived_func == other.derived_func and + self.base == other.base) + else: + return self is other + # Supported functions for derived type variables. SAMEAS = 'SameAs' LANEOF = 'LaneOf' diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index 164747fac9..ee6710a79f 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -100,6 +100,10 @@ class XForm(object): "extra inputs in dst RTL: {}".format( self.inputs[num_src_inputs:])) + self._infer_types(self.src) + self._infer_types(self.dst) + self._collect_typevars() + def __repr__(self): s = "XForm(inputs={}, defs={},\n ".format(self.inputs, self.defs) s += '\n '.join(str(n) for n in self.src) @@ -201,6 +205,63 @@ class XForm(object): raise AssertionError( '{} not defined in dest pattern'.format(d)) + def _infer_types(self, rtl): + # type: (Rtl) -> None + """Assign type variables to all value variables used in `rtl`.""" + for d in rtl.rtl: + inst = d.expr.inst + + # Get the Var corresponding to the controlling type variable. + ctrl_var = None # type: Var + if inst.is_polymorphic: + if inst.use_typevar_operand: + # Should this be an assertion instead? + # Should all value operands be required to be Vars? + arg = d.expr.args[inst.format.typevar_operand] + if isinstance(arg, Var): + ctrl_var = arg + else: + ctrl_var = d.defs[inst.value_results[0]] + + # Reconcile arguments with the requirements of `inst`. + for opnum in inst.format.value_operands: + inst_tv = inst.ins[opnum].typevar + v = d.expr.args[opnum] + if isinstance(v, Var): + v.constrain_typevar(inst_tv, inst.ctrl_typevar, ctrl_var) + + # Reconcile results with the requirements of `inst`. + for resnum in inst.value_results: + inst_tv = inst.outs[resnum].typevar + v = d.defs[resnum] + v.constrain_typevar(inst_tv, inst.ctrl_typevar, ctrl_var) + + def _collect_typevars(self): + # type: () -> None + """ + Collect a list of variables whose type can be used to infer the types + of all expressions. + + This should be called after `_infer_types()` above has computed type + variables for all the used vars. + """ + fvars = list(v for v in self.inputs if v.has_free_typevar()) + fvars += list(v for v in self.defs if v.has_free_typevar()) + self.free_typevars = fvars + + # When substituting a pattern, we know the types of all variables that + # appear on the source side: inut, output, and intermediate values. + # However, temporary values which appear only on the destination side + # must have their type computed somehow. + # + # Some variables have a fixed type which appears as a type variable + # with a singleton_type field set. That's allowed for temps too. + for v in fvars: + if v.is_temp() and not v.typevar.singleton_type: + raise AssertionError( + "Cannot determine type of temp '{}' in xform:\n{}" + .format(v, self)) + class XFormGroup(object): """ From 36871d3fb15f8c0e38d4096caf99edd78f082747 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 10 Nov 2016 14:46:42 -0800 Subject: [PATCH 0442/3084] Add TypeVar.rust_expr(). Generate a Rust expression that computes the value of a derived type variable. --- lib/cretonne/meta/cdsl/ast.py | 10 +++++ lib/cretonne/meta/cdsl/test_typevar.py | 8 +++- lib/cretonne/meta/cdsl/typevar.py | 56 +++++++++++++++++--------- lib/cretonne/meta/gen_instr.py | 3 +- 4 files changed, 55 insertions(+), 22 deletions(-) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index d8b3b7b875..8db7352333 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -197,6 +197,16 @@ class Var(Expr): return False return self.typevar is self.original_typevar + def rust_type(self): + # type: () -> str + """ + Get a Rust expression that computes the type of this variable. + + It is assumed that local variables exist corresponding to the free type + variables. + """ + return self.typevar.rust_expr() + def constrain_typevar(self, sym_typevar, sym_ctrl, ctrl_var): # type: (TypeVar, TypeVar, Var) -> None """ diff --git a/lib/cretonne/meta/cdsl/test_typevar.py b/lib/cretonne/meta/cdsl/test_typevar.py index 7dae18221a..29db26f583 100644 --- a/lib/cretonne/meta/cdsl/test_typevar.py +++ b/lib/cretonne/meta/cdsl/test_typevar.py @@ -57,10 +57,14 @@ class TestTypeVar(TestCase): x2 = TypeVar('x2', 'i16 and up', ints=(16, 64)) with self.assertRaises(AssertionError): x2.double_width() - self.assertEqual(str(x2.half_width()), '`HalfWidth(x2)`') + self.assertEqual(str(x2.half_width()), '`half_width(x2)`') + self.assertEqual(x2.half_width().rust_expr(), 'x2.half_width()') + self.assertEqual( + x2.half_width().double_width().rust_expr(), + 'x2.half_width().double_width()') x3 = TypeVar('x3', 'up to i32', ints=(8, 32)) - self.assertEqual(str(x3.double_width()), '`DoubleWidth(x3)`') + self.assertEqual(str(x3.double_width()), '`double_width(x3)`') with self.assertRaises(AssertionError): x3.half_width() diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 30986c5db8..d14866cf77 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -312,11 +312,14 @@ class TypeVar(object): return self is other # Supported functions for derived type variables. - SAMEAS = 'SameAs' - LANEOF = 'LaneOf' - ASBOOL = 'AsBool' - HALFWIDTH = 'HalfWidth' - DOUBLEWIDTH = 'DoubleWidth' + # The names here must match the method names on `ir::types::Type`. + # The camel_case of the names must match `enum OperandConstraint` in + # `instructions.rs`. + SAMEAS = 'same_as' + LANEOF = 'lane_of' + ASBOOL = 'as_bool' + HALFWIDTH = 'half_width' + DOUBLEWIDTH = 'double_width' @staticmethod def derived(base, derived_func): @@ -370,13 +373,14 @@ class TypeVar(object): Return a derived type variable that has the same number of vector lanes as this one, but the lanes are half the width. """ - ts = self.type_set - if ts.min_int: - assert ts.min_int > 8, "Can't halve all integer types" - if ts.min_float: - assert ts.min_float > 32, "Can't halve all float types" - if ts.min_bool: - assert ts.min_bool > 8, "Can't halve all boolean types" + if not self.is_derived: + ts = self.type_set + if ts.min_int: + assert ts.min_int > 8, "Can't halve all integer types" + if ts.min_float: + assert ts.min_float > 32, "Can't halve all float types" + if ts.min_bool: + assert ts.min_bool > 8, "Can't halve all boolean types" return TypeVar.derived(self, self.HALFWIDTH) @@ -386,13 +390,14 @@ class TypeVar(object): Return a derived type variable that has the same number of vector lanes as this one, but the lanes are double the width. """ - ts = self.type_set - if ts.max_int: - assert ts.max_int < MAX_BITS, "Can't double all integer types." - if ts.max_float: - assert ts.max_float < MAX_BITS, "Can't double all float types." - if ts.max_bool: - assert ts.max_bool < MAX_BITS, "Can't double all boolean types." + if not self.is_derived: + ts = self.type_set + if ts.max_int: + assert ts.max_int < MAX_BITS, "Can't double all integer types." + if ts.max_float: + assert ts.max_float < MAX_BITS, "Can't double all float types." + if ts.max_bool: + assert ts.max_bool < MAX_BITS, "Can't double all bool types." return TypeVar.derived(self, self.DOUBLEWIDTH) @@ -409,6 +414,19 @@ class TypeVar(object): else: return self + def rust_expr(self): + # type: () -> str + """ + Get a Rust expression that computes the type of this type variable. + """ + if self.is_derived: + return '{}.{}()'.format( + self.base.rust_expr(), self.derived_func) + elif self.singleton_type: + return self.singleton_type.rust_name() + else: + return self.name + def constrain_types(self, other): # type: (TypeVar) -> None """ diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 4f240ab6b1..11aaa4fa0d 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -5,6 +5,7 @@ from __future__ import absolute_import import srcgen import constant_hash from unique_table import UniqueTable, UniqueSeqTable +from cdsl import camel_case from cdsl.operands import ImmediateKind import cdsl.types from cdsl.formats import InstructionFormat @@ -330,7 +331,7 @@ def get_constraint(op, ctrl_typevar, type_sets): if tv.is_derived: assert tv.base is ctrl_typevar, "Not derived from ctrl_typevar" - return tv.derived_func + return camel_case(tv.derived_func) assert tv is ctrl_typevar return 'Same' From 59eef69a4f55c050b2a0de6f2ba27b45c3fd677d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 10 Nov 2016 15:33:16 -0800 Subject: [PATCH 0443/3084] Emit type arguments to builder methods that need it. Use the inferred type variables to construct a type argument for builder methods. This is for those instructions where the result types cannot be computed from the result types. --- lib/cretonne/meta/cdsl/ast.py | 12 +++++++++--- lib/cretonne/meta/gen_legalizer.py | 14 +++++++++++++- 2 files changed, 22 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index 8db7352333..ddcc702575 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -9,7 +9,7 @@ from . import instructions from .typevar import TypeVar try: - from typing import Union, Tuple # noqa + from typing import Union, Tuple, Sequence # noqa except ImportError: pass @@ -389,12 +389,18 @@ class Apply(Expr): args = ', '.join(map(str, self.args)) return '{}({})'.format(self.instname(), args) - def rust_builder(self): - # type: () -> str + def rust_builder(self, defs=None): + # type: (Sequence[Var]) -> str """ Return a Rust Builder method call for instantiating this instruction application. + + The `defs` argument should be a list of variables defined by this + instruction. It is used to construct a result type if necessary. """ args = ', '.join(map(str, self.args)) + # Do we need to pass an explicit type argument? + if self.inst.is_polymorphic and not self.inst.use_typevar_operand: + args = defs[0].rust_type() + ', ' + args method = self.inst.snake_name() return '{}({})'.format(method, args) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 409ebb7cf2..983871ac40 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -10,6 +10,7 @@ the input instruction. from __future__ import absolute_import from srcgen import Formatter from base import legalize +from cdsl.ast import Var try: from typing import Sequence # noqa @@ -72,6 +73,12 @@ def unwrap_inst(iref, node, fmt): fmt.outdented_line('} else {') fmt.line('unreachable!("bad instruction format")') + # Get the types of any variables where it is needed. + for i in iform.value_operands: + v = expr.args[i] + if isinstance(v, Var) and v.has_free_typevar(): + fmt.line('let typeof_{0} = dfg.value_type({0});'.format(v)) + # If the node has multiple results, detach the values. # Place the secondary values in 'src_{}' locals. if len(node.defs) > 1: @@ -91,6 +98,11 @@ def unwrap_inst(iref, node, fmt): for d in node.defs[1:]: fmt.line('src_{} = vals.next().unwrap();'.format(d)) fmt.line('assert_eq!(vals.next(), None);') + for d in node.defs[1:]: + if d.has_free_typevar(): + fmt.line( + 'let typeof_{0} = dfg.value_type(src_{0});' + .format(d)) def wrap_tup(seq): @@ -128,7 +140,7 @@ def emit_dst_inst(node, fmt): builder = 'let {} = dfg.ins(pos)'.format(wrap_tup(node.defs)) fixup_first_result = node.defs[0].is_output() - fmt.line('{}.{};'.format(builder, node.expr.rust_builder())) + fmt.line('{}.{};'.format(builder, node.expr.rust_builder(node.defs))) # If we just replaced an instruction, we need to bump the cursor so # following instructions are inserted *after* the replaced insruction. From 5e4602efb4f89afbd419a4aa091edf566382381c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 7 Nov 2016 13:05:37 -0800 Subject: [PATCH 0444/3084] Add expansion patterns for large immediates. Expand foo_imm into iconst + foo. --- cranelift/filetests/isa/riscv/expand-i32.cton | 11 +++++++++++ lib/cretonne/meta/base/legalize.py | 12 ++++++++++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/isa/riscv/expand-i32.cton b/cranelift/filetests/isa/riscv/expand-i32.cton index c0ca7be56f..166b058eb2 100644 --- a/cranelift/filetests/isa/riscv/expand-i32.cton +++ b/cranelift/filetests/isa/riscv/expand-i32.cton @@ -19,3 +19,14 @@ ebb0(v1: i32, v2: i32): ; It's possible the legalizer will rewrite these value aliases in the future. ; check: $v4 -> $cout ; check: return $v3, $v4 + +; Expanding illegal immediate constants. +; Note that at some point we'll probably expand the iconst as well. +function large_imm(i32) -> i32 { +ebb0(v0: i32): + v1 = iadd_imm v0, 1000000000 + return v1 +} +; check: $(cst=$V) = iconst.i32 0x3b9a_ca00 +; check: $v1 = iadd $v0, $cst +; check: return $v1 diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index f054d16d9e..b2f8fc4f6d 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -7,10 +7,10 @@ patterns that describe how base instructions can be transformed to other base instructions that are legal. """ from __future__ import absolute_import -from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry +from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm from .instructions import isub, isub_bin, isub_bout, isub_borrow from .instructions import band, bor, bxor, isplit_lohi, iconcat_lohi -from .instructions import icmp +from .instructions import icmp, iconst from cdsl.ast import Var from cdsl.xform import Rtl, XFormGroup @@ -127,3 +127,11 @@ expand.legalize( (a, b2) << isub_bout(a1, b_in), b << bor(b1, b2) )) + +# Expansions for immediates that are too large. +expand.legalize( + a << iadd_imm(x, y), + Rtl( + a1 << iconst(y), + a << iadd(x, a1) + )) From 581294dafe55cea5da87e81d0cf57eea2250a93c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 11 Nov 2016 11:17:40 -0800 Subject: [PATCH 0445/3084] Use uppercase for the global riscv.ISA constant. --- lib/cretonne/meta/isa/__init__.py | 2 +- lib/cretonne/meta/isa/riscv/__init__.py | 2 +- lib/cretonne/meta/isa/riscv/defs.py | 6 +++--- lib/cretonne/meta/isa/riscv/settings.py | 6 +++--- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/meta/isa/__init__.py b/lib/cretonne/meta/isa/__init__.py index 1bef425c58..33b787f494 100644 --- a/lib/cretonne/meta/isa/__init__.py +++ b/lib/cretonne/meta/isa/__init__.py @@ -16,4 +16,4 @@ def all_isas(): Get a list of all the supported target ISAs. Each target ISA is represented as a :py:class:`cretonne.TargetISA` instance. """ - return [riscv.isa] + return [riscv.ISA] diff --git a/lib/cretonne/meta/isa/riscv/__init__.py b/lib/cretonne/meta/isa/riscv/__init__.py index e187a95c0e..3d5d19ed99 100644 --- a/lib/cretonne/meta/isa/riscv/__init__.py +++ b/lib/cretonne/meta/isa/riscv/__init__.py @@ -29,4 +29,4 @@ from . import defs from . import encodings, settings # noqa # Re-export the primary target ISA definition. -isa = defs.isa.finish() +ISA = defs.ISA.finish() diff --git a/lib/cretonne/meta/isa/riscv/defs.py b/lib/cretonne/meta/isa/riscv/defs.py index 7a487482d3..485dbd7ef0 100644 --- a/lib/cretonne/meta/isa/riscv/defs.py +++ b/lib/cretonne/meta/isa/riscv/defs.py @@ -7,8 +7,8 @@ from __future__ import absolute_import from cdsl.isa import TargetISA, CPUMode import base.instructions -isa = TargetISA('riscv', [base.instructions.GROUP]) +ISA = TargetISA('riscv', [base.instructions.GROUP]) # CPU modes for 32-bit and 64-bit operation. -RV32 = CPUMode('RV32', isa) -RV64 = CPUMode('RV64', isa) +RV32 = CPUMode('RV32', ISA) +RV64 = CPUMode('RV64', ISA) diff --git a/lib/cretonne/meta/isa/riscv/settings.py b/lib/cretonne/meta/isa/riscv/settings.py index 7571bf1611..e6fe230f89 100644 --- a/lib/cretonne/meta/isa/riscv/settings.py +++ b/lib/cretonne/meta/isa/riscv/settings.py @@ -5,9 +5,9 @@ from __future__ import absolute_import from cdsl.settings import SettingGroup, BoolSetting from cdsl.predicates import And import base.settings as shared -from .defs import isa +from .defs import ISA -isa.settings = SettingGroup('riscv', parent=shared.group) +ISA.settings = SettingGroup('riscv', parent=shared.group) supports_m = BoolSetting("CPU supports the 'M' extension (mul/div)") supports_a = BoolSetting("CPU supports the 'A' extension (atomics)") @@ -25,4 +25,4 @@ use_d = And(supports_d, shared.enable_float) full_float = And(shared.enable_simd, supports_f, supports_d) -isa.settings.close(globals()) +ISA.settings.close(globals()) From 34f7b9348281292aa4b41adc4ad2670496f447d0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 11 Nov 2016 11:27:29 -0800 Subject: [PATCH 0446/3084] Add stubs for Intel and ARM architectures. The Intel ISA handles both 32-bit and 64-bit code. ARM is split into separate arm32 and arm64 ISAs since the architectures have little in common in instruction encodings and register files. --- lib/cretonne/meta/isa/__init__.py | 4 ++-- lib/cretonne/meta/isa/arm32/__init__.py | 13 +++++++++++++ lib/cretonne/meta/isa/arm32/defs.py | 14 ++++++++++++++ lib/cretonne/meta/isa/arm64/__init__.py | 12 ++++++++++++ lib/cretonne/meta/isa/arm64/defs.py | 11 +++++++++++ lib/cretonne/meta/isa/intel/__init__.py | 22 ++++++++++++++++++++++ lib/cretonne/meta/isa/intel/defs.py | 14 ++++++++++++++ 7 files changed, 88 insertions(+), 2 deletions(-) create mode 100644 lib/cretonne/meta/isa/arm32/__init__.py create mode 100644 lib/cretonne/meta/isa/arm32/defs.py create mode 100644 lib/cretonne/meta/isa/arm64/__init__.py create mode 100644 lib/cretonne/meta/isa/arm64/defs.py create mode 100644 lib/cretonne/meta/isa/intel/__init__.py create mode 100644 lib/cretonne/meta/isa/intel/defs.py diff --git a/lib/cretonne/meta/isa/__init__.py b/lib/cretonne/meta/isa/__init__.py index 33b787f494..8f623d18bb 100644 --- a/lib/cretonne/meta/isa/__init__.py +++ b/lib/cretonne/meta/isa/__init__.py @@ -7,7 +7,7 @@ architecture supported by Cretonne. """ from __future__ import absolute_import from cdsl.isa import TargetISA # noqa -from . import riscv +from . import riscv, intel, arm32, arm64 def all_isas(): @@ -16,4 +16,4 @@ def all_isas(): Get a list of all the supported target ISAs. Each target ISA is represented as a :py:class:`cretonne.TargetISA` instance. """ - return [riscv.ISA] + return [riscv.ISA, intel.ISA, arm32.ISA, arm64.ISA] diff --git a/lib/cretonne/meta/isa/arm32/__init__.py b/lib/cretonne/meta/isa/arm32/__init__.py new file mode 100644 index 0000000000..9c992d0c17 --- /dev/null +++ b/lib/cretonne/meta/isa/arm32/__init__.py @@ -0,0 +1,13 @@ +""" +ARM 32-bit Architecture +---------------------- + +This target ISA generates code for ARMv7 and ARMv8 CPUs in 32-bit mode +(AArch32). We support both ARM and Thumb2 instruction encodings. +""" + +from __future__ import absolute_import +from . import defs + +# Re-export the primary target ISA definition. +ISA = defs.ISA.finish() diff --git a/lib/cretonne/meta/isa/arm32/defs.py b/lib/cretonne/meta/isa/arm32/defs.py new file mode 100644 index 0000000000..f90abc2001 --- /dev/null +++ b/lib/cretonne/meta/isa/arm32/defs.py @@ -0,0 +1,14 @@ +""" +ARM 32-bit definitions. + +Commonly used definitions. +""" +from __future__ import absolute_import +from cdsl.isa import TargetISA, CPUMode +import base.instructions + +ISA = TargetISA('arm32', [base.instructions.GROUP]) + +# CPU modes for 32-bit ARM and Thumb2. +A32 = CPUMode('A32', ISA) +T32 = CPUMode('T32', ISA) diff --git a/lib/cretonne/meta/isa/arm64/__init__.py b/lib/cretonne/meta/isa/arm64/__init__.py new file mode 100644 index 0000000000..198fc338b1 --- /dev/null +++ b/lib/cretonne/meta/isa/arm64/__init__.py @@ -0,0 +1,12 @@ +""" +ARM 64-bit Architecture +----------------------- + +ARMv8 CPUs running the Aarch64 architecture. +""" + +from __future__ import absolute_import +from . import defs + +# Re-export the primary target ISA definition. +ISA = defs.ISA.finish() diff --git a/lib/cretonne/meta/isa/arm64/defs.py b/lib/cretonne/meta/isa/arm64/defs.py new file mode 100644 index 0000000000..e493b4424a --- /dev/null +++ b/lib/cretonne/meta/isa/arm64/defs.py @@ -0,0 +1,11 @@ +""" +ARM64 definitions. + +Commonly used definitions. +""" +from __future__ import absolute_import +from cdsl.isa import TargetISA, CPUMode +import base.instructions + +ISA = TargetISA('arm64', [base.instructions.GROUP]) +A64 = CPUMode('A64', ISA) diff --git a/lib/cretonne/meta/isa/intel/__init__.py b/lib/cretonne/meta/isa/intel/__init__.py new file mode 100644 index 0000000000..a97b424099 --- /dev/null +++ b/lib/cretonne/meta/isa/intel/__init__.py @@ -0,0 +1,22 @@ +""" +Intel Target Architecture +------------------------- + +This target ISA generates code for Intel CPUs with two separate CPU modes: + +`I32` + IA-32 architecture, also known as 'x86'. Generates code for the Intel 386 + and later processors in 32-bit mode. +`I64` + Intel 64 architecture, also known as 'x86-64, 'x64', and 'amd64'. Intel and + AMD CPUs running in 64-bit mode. + +Floating point is supported only on CPUs with support for SSE2 or later. There +is no x87 floating point support. +""" + +from __future__ import absolute_import +from . import defs + +# Re-export the primary target ISA definition. +ISA = defs.ISA.finish() diff --git a/lib/cretonne/meta/isa/intel/defs.py b/lib/cretonne/meta/isa/intel/defs.py new file mode 100644 index 0000000000..50251b44af --- /dev/null +++ b/lib/cretonne/meta/isa/intel/defs.py @@ -0,0 +1,14 @@ +""" +Intel definitions. + +Commonly used definitions. +""" +from __future__ import absolute_import +from cdsl.isa import TargetISA, CPUMode +import base.instructions + +ISA = TargetISA('intel', [base.instructions.GROUP]) + +# CPU modes for 32-bit and 64-bit operation. +I32 = CPUMode('I32', ISA) +I64 = CPUMode('I64', ISA) From f16327e6eb84211af506f657071499812cdff001 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 11 Nov 2016 13:04:30 -0800 Subject: [PATCH 0447/3084] Move some utility functions into the cdsl package. - is_power_of_two - next_power_of_two Make sure we run doctests on these functions. --- lib/cretonne/meta/cdsl/__init__.py | 44 +++++++++++++++++++++++++- lib/cretonne/meta/cdsl/test_package.py | 8 +++++ lib/cretonne/meta/cdsl/typevar.py | 7 +--- lib/cretonne/meta/constant_hash.py | 22 +------------ 4 files changed, 53 insertions(+), 28 deletions(-) create mode 100644 lib/cretonne/meta/cdsl/test_package.py diff --git a/lib/cretonne/meta/cdsl/__init__.py b/lib/cretonne/meta/cdsl/__init__.py index 247b970675..44445268ea 100644 --- a/lib/cretonne/meta/cdsl/__init__.py +++ b/lib/cretonne/meta/cdsl/__init__.py @@ -13,5 +13,47 @@ camel_re = re.compile('(^|_)([a-z])') def camel_case(s): # type: (str) -> str - """Convert the string s to CamelCase""" + """Convert the string s to CamelCase: + >>> camel_case('x') + 'X' + >>> camel_case('camel_case') + 'CamelCase' + """ return camel_re.sub(lambda m: m.group(2).upper(), s) + + +def is_power_of_two(x): + # type: (int) -> bool + """Check if `x` is a power of two: + >>> is_power_of_two(0) + False + >>> is_power_of_two(1) + True + >>> is_power_of_two(2) + True + >>> is_power_of_two(3) + False + """ + return x > 0 and x & (x-1) == 0 + + +def next_power_of_two(x): + # type: (int) -> int + """ + Compute the next power of two that is greater than `x`: + >>> next_power_of_two(0) + 1 + >>> next_power_of_two(1) + 2 + >>> next_power_of_two(2) + 4 + >>> next_power_of_two(3) + 4 + >>> next_power_of_two(4) + 8 + """ + s = 1 + while x & (x + 1) != 0: + x |= x >> s + s *= 2 + return x + 1 diff --git a/lib/cretonne/meta/cdsl/test_package.py b/lib/cretonne/meta/cdsl/test_package.py new file mode 100644 index 0000000000..b66d60d694 --- /dev/null +++ b/lib/cretonne/meta/cdsl/test_package.py @@ -0,0 +1,8 @@ +from __future__ import absolute_import +import doctest +import cdsl + + +def load_tests(loader, tests, ignore): + tests.addTests(doctest.DocTestSuite(cdsl)) + return tests diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index d14866cf77..580d367f51 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -6,7 +6,7 @@ polymorphic by using type variables. """ from __future__ import absolute_import import math -from . import types +from . import types, is_power_of_two try: from typing import Tuple, Union # noqa @@ -20,11 +20,6 @@ MAX_LANES = 256 MAX_BITS = 64 -def is_power_of_two(x): - # type: (int) -> bool - return x > 0 and x & (x-1) == 0 - - def int_log2(x): # type: (int) -> int return int(math.log(x, 2)) diff --git a/lib/cretonne/meta/constant_hash.py b/lib/cretonne/meta/constant_hash.py index 38911278ed..8c3c355aa1 100644 --- a/lib/cretonne/meta/constant_hash.py +++ b/lib/cretonne/meta/constant_hash.py @@ -6,6 +6,7 @@ don't attempt parfect hashing, but simply generate an open addressed quadratically probed hash table. """ from __future__ import absolute_import +from cdsl import next_power_of_two def simple_hash(s): @@ -24,27 +25,6 @@ def simple_hash(s): return h -def next_power_of_two(x): - """ - Compute the next power of two that is greater than `x`: - >>> next_power_of_two(0) - 1 - >>> next_power_of_two(1) - 2 - >>> next_power_of_two(2) - 4 - >>> next_power_of_two(3) - 4 - >>> next_power_of_two(4) - 8 - """ - s = 1 - while x & (x + 1) != 0: - x |= x >> s - s *= 2 - return x + 1 - - def compute_quadratic(items, hash_function): """ Compute an open addressed, quadratically probed hash table containing From e1c0171b2c042c613488117a40e67bb5ab8657b8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 11 Nov 2016 14:17:10 -0800 Subject: [PATCH 0448/3084] Define register banks. Add a RegBank class for describing CPU register banks. Define register banks for all the ISA stubs. The ARM32 floating point bank in particular requires attention. --- lib/cretonne/meta/cdsl/isa.py | 2 + lib/cretonne/meta/cdsl/registers.py | 80 ++++++++++++++++++++++++ lib/cretonne/meta/isa/arm32/registers.py | 29 +++++++++ lib/cretonne/meta/isa/arm64/registers.py | 19 ++++++ lib/cretonne/meta/isa/intel/registers.py | 39 ++++++++++++ lib/cretonne/meta/isa/riscv/registers.py | 18 ++++++ 6 files changed, 187 insertions(+) create mode 100644 lib/cretonne/meta/cdsl/registers.py create mode 100644 lib/cretonne/meta/isa/arm32/registers.py create mode 100644 lib/cretonne/meta/isa/arm64/registers.py create mode 100644 lib/cretonne/meta/isa/intel/registers.py create mode 100644 lib/cretonne/meta/isa/riscv/registers.py diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 7f59a971db..6c324603a5 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -10,6 +10,7 @@ try: from .predicates import Predicate, FieldPredicate # noqa from .settings import SettingGroup # noqa from .types import ValueType # noqa + from .registers import RegBank # noqa AnyPredicate = Union[Predicate, FieldPredicate] except ImportError: pass @@ -32,6 +33,7 @@ class TargetISA(object): self.settings = None # type: SettingGroup self.instruction_groups = instruction_groups self.cpumodes = list() # type: List[CPUMode] + self.regbanks = list() # type: List[RegBank] def finish(self): # type: () -> TargetISA diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py new file mode 100644 index 0000000000..6c1d379dac --- /dev/null +++ b/lib/cretonne/meta/cdsl/registers.py @@ -0,0 +1,80 @@ +""" +Register set definitions +------------------------ + +Each ISA defines a separate register set that is used by the register allocator +and the final binary encoding of machine code. + +The CPU registers are first divided into disjoint register banks, represented +by a `RegBank` instance. Registers in different register banks never interfere +with each other. A typical CPU will have a general purpose and a floating point +register bank. + +A register bank consists of a number of *register units* which are the smallest +indivisible units of allocation and interference. A register unit doesn't +necesarily correspond to a particular number of bits in a register, it is more +like a placeholder that can be used to determine of a register is taken or not. + +The register allocator works with *register classes* which can allocate one or +more register units at a time. A register class allocates more than one +register unit at a time when its registers are composed of smaller alocatable +units. For example, the ARM double precision floating point registers are +composed of two single precision registers. """ +from __future__ import absolute_import +from . import is_power_of_two, next_power_of_two + + +try: + from typing import Sequence # noqa + from .isa import TargetISA # noqa +except ImportError: + pass + + +class RegBank(object): + """ + A register bank belonging to an ISA. + + A register bank controls a set of *register units* disjoint from all the + other register banks in the ISA. The register units are numbered uniquely + within the target ISA, and the units in a register bank form a contiguous + sequence starting from a sufficiently aligned point that their low bits can + be used directly when encoding machine code instructions. + + Register units can be given generated names like `r0`, `r1`, ..., or a + tuple of special register unit names can be provided. + + :param name: Name of this register bank. + :param doc: Documentation string. + :param units: Number of register units. + :param prefix: Prefix for generated unit names. + :param names: Special names for the first units. May be shorter than + `units`, the remaining units are named using `prefix`. + """ + + def __init__(self, name, isa, doc, units, prefix='p', names=()): + # type: (str, TargetISA, str, int, str, Sequence[str]) -> None + self.name = name + self.isa = isa + self.first_unit = 0 + self.units = units + self.prefix = prefix + self.names = names + + assert len(names) <= units + + if isa.regbanks: + # Get the next free unit number. + last = isa.regbanks[-1] + u = last.first_unit + last.units + align = units + if not is_power_of_two(align): + align = next_power_of_two(align) + self.first_unit = (u + align - 1) & -align + + isa.regbanks.append(self) + + def __repr__(self): + # type: () -> str + return ('RegBank({}, units={}, first_unit={})' + .format(self.name, self.units, self.first_unit)) diff --git a/lib/cretonne/meta/isa/arm32/registers.py b/lib/cretonne/meta/isa/arm32/registers.py new file mode 100644 index 0000000000..8d2b0480ae --- /dev/null +++ b/lib/cretonne/meta/isa/arm32/registers.py @@ -0,0 +1,29 @@ +""" +ARM32 register banks. +""" +from __future__ import absolute_import +from cdsl.registers import RegBank +from .defs import ISA + + +# Special register units: +# - r15 is the program counter. +# - r14 is the link register. +# - r13 is usually the stack pointer. +IntRegs = RegBank( + 'IntRegs', ISA, + 'General purpose registers', + units=16, prefix='r') + +FloatRegs = RegBank( + 'FloatRegs', ISA, r""" + Floating point registers. + + The floating point register units correspond to the S-registers, but + extended as if there were 64 registers. + + - S registers are one unit each. + - D registers are two units each, even D16 and above. + - Q registers are 4 units each. + """, + units=64, prefix='s') diff --git a/lib/cretonne/meta/isa/arm64/registers.py b/lib/cretonne/meta/isa/arm64/registers.py new file mode 100644 index 0000000000..fa1c87120b --- /dev/null +++ b/lib/cretonne/meta/isa/arm64/registers.py @@ -0,0 +1,19 @@ +""" +Aarch64 register banks. +""" +from __future__ import absolute_import +from cdsl.registers import RegBank +from .defs import ISA + + +# The `x31` regunit serves as the stack pointer / zero register depending on +# context. We reserve it and don't model the difference. +IntRegs = RegBank( + 'IntRegs', ISA, + 'General purpose registers', + units=32, prefix='x') + +FloatRegs = RegBank( + 'FloatRegs', ISA, + 'Floating point registers', + units=32, prefix='v') diff --git a/lib/cretonne/meta/isa/intel/registers.py b/lib/cretonne/meta/isa/intel/registers.py new file mode 100644 index 0000000000..abd6353eb1 --- /dev/null +++ b/lib/cretonne/meta/isa/intel/registers.py @@ -0,0 +1,39 @@ +""" +Intel register banks. + +While the floating-point registers are straight-forward, the general purpose +register bank has a few quirks on Intel architectures. We have these encodings +of the 8-bit registers: + + I32 I64 | 16b 32b 64b + 000 AL AL | AX EAX RAX + 001 CL CL | CX ECX RCX + 010 DL DL | DX EDX RDX + 011 BL BL | BX EBX RBX + 100 AH SPL | SP ESP RSP + 101 CH BPL | BP EBP RBP + 110 DH SIL | SI ESI RSI + 111 BH DIL | DI EDI RDI + +Here, the I64 column refers to the registers you get with a REX prefix. Without +the REX prefix, you get the I32 registers. + +The 8-bit registers are not that useful since WebAssembly only has i32 and i64 +data types, and the H-registers even less so. Rather than trying to model the +H-registers accurately, we'll avoid using them in both I32 and I64 modes. +""" +from __future__ import absolute_import +from cdsl.registers import RegBank +from .defs import ISA + + +IntRegs = RegBank( + 'IntRegs', ISA, + 'General purpose registers', + units=16, prefix='r', + names='rax rcx rdx rbx rsp rbp rsi rdi'.split()) + +FloatRegs = RegBank( + 'FloatRegs', ISA, + 'SSE floating point registers', + units=16, prefix='xmm') diff --git a/lib/cretonne/meta/isa/riscv/registers.py b/lib/cretonne/meta/isa/riscv/registers.py new file mode 100644 index 0000000000..9d1d0eaadb --- /dev/null +++ b/lib/cretonne/meta/isa/riscv/registers.py @@ -0,0 +1,18 @@ +""" +RISC-V register banks. +""" +from __future__ import absolute_import +from cdsl.registers import RegBank +from .defs import ISA + + +# We include `x0`, a.k.a `zero` in the register bank. It will be reserved. +IntRegs = RegBank( + 'IntRegs', ISA, + 'General purpose registers', + units=32, prefix='x') + +FloatRegs = RegBank( + 'FloatRegs', ISA, + 'Floating point registers', + units=32, prefix='f') From 060735adfec98c357a5b6d107cb01345d333bada Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 11 Nov 2016 15:08:12 -0800 Subject: [PATCH 0449/3084] Define register classes for 4 ISAs. --- lib/cretonne/meta/cdsl/registers.py | 53 +++++++++++++++++++++++- lib/cretonne/meta/isa/arm32/registers.py | 7 +++- lib/cretonne/meta/isa/arm64/registers.py | 5 ++- lib/cretonne/meta/isa/intel/registers.py | 5 ++- lib/cretonne/meta/isa/riscv/registers.py | 5 ++- 5 files changed, 70 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index 6c1d379dac..243a72e669 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -19,7 +19,8 @@ The register allocator works with *register classes* which can allocate one or more register units at a time. A register class allocates more than one register unit at a time when its registers are composed of smaller alocatable units. For example, the ARM double precision floating point registers are -composed of two single precision registers. """ +composed of two single precision registers. +""" from __future__ import absolute_import from . import is_power_of_two, next_power_of_two @@ -78,3 +79,53 @@ class RegBank(object): # type: () -> str return ('RegBank({}, units={}, first_unit={})' .format(self.name, self.units, self.first_unit)) + + +class RegClass(object): + """ + A register class is a subset of register units in a RegBank along with a + strategy for allocating registers. + + The *width* parameter determines how many register units are allocated at a + time. Usually it that is one, but for example the ARM D registers are + allocated two units at a time. When multiple units are allocated, it is + always a contiguous set of unit numbers. + + :param name: Name of this register class. + :param bank: The register bank we're allocating from. + :param count: The maximum number of allocations in this register class. By + default, the whole register bank can be allocated. + :param width: How many units to allocate at a time. + :param start: The first unit to allocate, relative to `bank.first.unit`. + :param stride: How many units to skip to get to the next allocation. + Default is `width`. + """ + + def __init__(self, name, bank, count=None, width=1, start=0, stride=None): + # type: (str, RegBank, int, int, int, int) -> None + self.name = name + self.bank = bank + self.start = start + self.width = width + + assert width > 0 + assert start >= 0 and start < bank.units + + if stride is None: + stride = width + assert stride > 0 + self.stride = stride + + if count is None: + count = bank.units / stride + self.count = count + + # When the stride is 1, we can wrap around to the beginning of the + # register bank, but with a larger stride, we wouldn't cover all the + # possible allocations with a simple modulo stride. For example, + # attempting to allocate the even registers before the odd ones + # wouldn't work. Only if stride is coprime to bank.units would it work, + # but that is unlikely since the bank size is almost always a power of + # two. + if start + count*stride > bank.units: + assert stride == 1, 'Wrapping with stride not supported' diff --git a/lib/cretonne/meta/isa/arm32/registers.py b/lib/cretonne/meta/isa/arm32/registers.py index 8d2b0480ae..3084c2bf98 100644 --- a/lib/cretonne/meta/isa/arm32/registers.py +++ b/lib/cretonne/meta/isa/arm32/registers.py @@ -2,7 +2,7 @@ ARM32 register banks. """ from __future__ import absolute_import -from cdsl.registers import RegBank +from cdsl.registers import RegBank, RegClass from .defs import ISA @@ -27,3 +27,8 @@ FloatRegs = RegBank( - Q registers are 4 units each. """, units=64, prefix='s') + +GPR = RegClass('GPR', IntRegs) +S = RegClass('S', FloatRegs, count=32) +D = RegClass('D', FloatRegs, width=2) +Q = RegClass('Q', FloatRegs, width=4) diff --git a/lib/cretonne/meta/isa/arm64/registers.py b/lib/cretonne/meta/isa/arm64/registers.py index fa1c87120b..fec3601367 100644 --- a/lib/cretonne/meta/isa/arm64/registers.py +++ b/lib/cretonne/meta/isa/arm64/registers.py @@ -2,7 +2,7 @@ Aarch64 register banks. """ from __future__ import absolute_import -from cdsl.registers import RegBank +from cdsl.registers import RegBank, RegClass from .defs import ISA @@ -17,3 +17,6 @@ FloatRegs = RegBank( 'FloatRegs', ISA, 'Floating point registers', units=32, prefix='v') + +GPR = RegClass('GPR', IntRegs) +FPR = RegClass('FPR', FloatRegs) diff --git a/lib/cretonne/meta/isa/intel/registers.py b/lib/cretonne/meta/isa/intel/registers.py index abd6353eb1..6a4c59403e 100644 --- a/lib/cretonne/meta/isa/intel/registers.py +++ b/lib/cretonne/meta/isa/intel/registers.py @@ -23,7 +23,7 @@ data types, and the H-registers even less so. Rather than trying to model the H-registers accurately, we'll avoid using them in both I32 and I64 modes. """ from __future__ import absolute_import -from cdsl.registers import RegBank +from cdsl.registers import RegBank, RegClass from .defs import ISA @@ -37,3 +37,6 @@ FloatRegs = RegBank( 'FloatRegs', ISA, 'SSE floating point registers', units=16, prefix='xmm') + +GPR = RegClass('GPR', IntRegs) +FPR = RegClass('FPR', FloatRegs) diff --git a/lib/cretonne/meta/isa/riscv/registers.py b/lib/cretonne/meta/isa/riscv/registers.py index 9d1d0eaadb..f00c9a0f0d 100644 --- a/lib/cretonne/meta/isa/riscv/registers.py +++ b/lib/cretonne/meta/isa/riscv/registers.py @@ -2,7 +2,7 @@ RISC-V register banks. """ from __future__ import absolute_import -from cdsl.registers import RegBank +from cdsl.registers import RegBank, RegClass from .defs import ISA @@ -16,3 +16,6 @@ FloatRegs = RegBank( 'FloatRegs', ISA, 'Floating point registers', units=32, prefix='f') + +GPR = RegClass('GPR', IntRegs) +FPR = RegClass('FPR', FloatRegs) From 7ce834c31235651a6294d5348fc952f48061370d Mon Sep 17 00:00:00 2001 From: Jacob Shaffer Date: Mon, 14 Nov 2016 08:45:10 -0500 Subject: [PATCH 0450/3084] Added link to README about nasal demons. --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index b98a4893fa..5656547e49 100644 --- a/README.rst +++ b/README.rst @@ -19,7 +19,7 @@ Cretonne is designed to be a code generator for WebAssembly with these design goals: No undefined behavior - Cretonne does not have a nasal demons clause, and it won't generate code + Cretonne does not have a `nasal demons clause `_, and it won't generate code with unexpected behavior if invariants are broken. Portable semantics As far as possible, Cretonne's input language has well-defined semantics From f67a1a60cba5d2ef9ac2c7519a2db050aece3f74 Mon Sep 17 00:00:00 2001 From: ranma42 Date: Tue, 15 Nov 2016 15:35:00 +0100 Subject: [PATCH 0451/3084] Fix some typos in the Language Reference --- cranelift/docs/langref.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index a9d5ac4c96..40cc65dbe3 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -913,8 +913,8 @@ Glossary through to the following instructions in the block, but only the first instruction in the EBB can be a branch target. - The last instrution in an EBB must be a :term:`terminator instruction`, - so execion cannot flow through to the next EBB in the function. (But + The last instruction in an EBB must be a :term:`terminator instruction`, + so execution cannot flow through to the next EBB in the function. (But there may be a branch to the next EBB.) Note that some textbooks define an EBB as a maximal *subtree* in the From fd412b49e1d9325c6df62272d2322362aa88f3dc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 21 Nov 2016 13:49:15 -0800 Subject: [PATCH 0452/3084] Start a design document for the Cretonne register allocator. --- cranelift/docs/index.rst | 1 + cranelift/docs/regalloc.rst | 232 ++++++++++++++++++++++++++++++++++++ 2 files changed, 233 insertions(+) create mode 100644 cranelift/docs/regalloc.rst diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst index 8df4042db3..301346a9e3 100644 --- a/cranelift/docs/index.rst +++ b/cranelift/docs/index.rst @@ -9,6 +9,7 @@ Contents: langref metaref testing + regalloc compare-llvm Indices and tables diff --git a/cranelift/docs/regalloc.rst b/cranelift/docs/regalloc.rst new file mode 100644 index 0000000000..18057c2511 --- /dev/null +++ b/cranelift/docs/regalloc.rst @@ -0,0 +1,232 @@ +******************************* +Register Allocation in Cretonne +******************************* + +.. default-domain:: cton +.. highlight:: rust + +Cretonne uses a *decoupled, SSA-based* register allocator. Decoupled means that +register allocation is split into two primary phases: *spilling* and +*coloring*. SSA-based means that the code stays in SSA form throughout the +register allocator, and in fact is still in SSA form after register allocation. + +Before the register allocator is run, all instructions in the function must be +*legalized*, which means that every instruction has an entry in the +``encodings`` table. The encoding entries also provide register class +constraints on the instruction's operands that the register allocator must +satisfy. + +After the register allocator has run, the ``locations`` table provides a +register or stack slot location for all SSA values used by the function. The +register allocator may have inserted :inst:`spill`, :inst:`fill`, and +:inst:`copy` instructions to make that possible. + +SSA-based register allocation +============================= + +The phases of the SSA-based register allocator are: + +Liveness analysis + For each SSA value, determine exactly where it is live. + +Spilling + The process of deciding which SSA values go in a stack slot and which + values go in a register. The spilling phase can also split live ranges by + inserting :inst:`copy` instructions, or transform the code in other ways to + reduce the number of values kept in registers. + + After spilling, the number of live register values never exceeds the number + of available registers. + +Coloring + The process of assigning specific registers to the live values. It's a + property of SSA form that this can be done in a linear scan of the + dominator tree without causing any additional spills. + +EBB argument fixup + The coloring phase does not guarantee that EBB arguments are placed in the + correct registers and/or stack slots before jumping to the EBB. It will + try its best, but not making this guarantee is essential to the speed of + the coloring phase. (EBB arguments correspond to PHI nodes in traditional + SSA form). + + The argument fixup phase inserts 'shuffle code' before jumps and branches + to place the argument values in their expected locations. + +The contract between the spilling and coloring phases is that the number of +values in registers never exceeds the number of available registers. This +sounds simple enough in theory, but in pratice there are some complications. + +Real-world complications to SSA coloring +---------------------------------------- + +In practice, instruction set architectures don't have "K interchangable +registers", and register pressure can't be measured with a single number. There +are complications: + +Different register banks + Most ISAs separate integer registers from floating point registers, and + instructions require their operands to come from a specific bank. This is a + fairly simple problem to deal with since the register banks are completely + disjoint. We simply count the number of integer and floating-point values + that are live independently, and make sure that each number does not exceed + the size of their respective register banks. + +Instructions with fixed operands + Some instructions use a fixed register for an operand. This happens on the + Intel ISAs: + + - Dynamic shift and rotate instructions take the shift amount in CL. + - Division instructions use RAX and RDX for both input and output operands. + - Wide multiply instructions use fixed RAX and RDX registers for input and + output operands. + - A few SSE variable blend instructions use a hardwired XMM0 input operand. + +Operands constrained to register subclasses + Some instructions can only use a subset of the registers for some operands. + For example, the ARM NEON vmla (scalar) instruction requires the scalar + operand to be located in D0-15 or even D0-7, depending on the data type. + The other operands can be from the full D0-31 register set. + +ABI boundaries + Before making a function call, arguments must be placed in specific + registers and stack locations determined by the ABI, and return values + appear in fixed registers. + + Some registers can be clobbered by the call and some are saved by the + callee. In some cases, only the low bits of a register are saved by the + callee. For example, ARM64 callees save only the low 64 bits of v8-15, and + Win64 callees only save the low 128 bits of AVX registers. + + ABI boundaries also affect the location of arguments to the entry block and + return values passed to the :inst:`return` instruction. + +Aliasing registers + Different registers sometimes share the same bits in the register bank. + This can make it difficult to measure register pressure. For example, the + Intel registers RAX, EAX, AX, AL, and AH overlap. + + If only one of the aliasing registers can be used at a time, the aliasing + doesn't cause problems since the registers can simply be counted as one + unit. + +Early clobbers + Sometimes an instruction requires that the register used for an output + operand does not alias any of the input operands. This happens for inline + assembly and in some other special cases. + + +Liveness Analysis +================= + +Both spilling and coloring need to know exactly where SSA values are live. The +liveness analysis computes this information. + +The data structure representing the live range of a value uses the linear +layout of the function. All instructions and EBB headers are assigned a +*program position*. A starting point for a live range can be one of the +following: + +- The instruction where the value is defined. +- The EBB header where the value is an EBB argument. +- An EBB header where the value is live-in because it was defined in a + dominating block. + +The ending point of a live range can be: + +- The last instruction to use the value. +- A branch or jump to an EBB where the value is live-in. + +When all the EBBs in a function are laid out linearly, the live range of a +value doesn't have to be a contiguous interval, although it will be in a +majority of cases. There can be holes in the linear live range. + +The live range of an SSA value is represented as: + +- The earliest program point where the value is live. +- The latest program point where the value is live. +- A (often empty) list of holes, sorted in program order. + +Any value that is only used inside a single EBB will have a live range without +holes. Some values are live across large parts of the function, and this can +often be represented with very few holes. It is important that the live range +data structure doesn't have to grow linearly with the number of EBBs covered by +a live range. + +This representation is very similar to LLVM's ``LiveInterval`` data structure +with a few important differences: + +- The Cretonne ``LiveRange`` only covers a single SSA value, while LLVM's + ``LiveInterval`` represents the union of multiple related SSA values in a + virtual register. This makes Cretonne's representation smaller because + individual segments don't have to annotated with a value number. +- Cretonne stores the min and max program points separately from a list of + holes, while LLVM stores an array of segments. The two representations are + equivalent, but Cretonne optimizes for the common case of a single contiguous + interval. +- LLVM represents a program point as ``SlotIndex`` which holds a pointer to a + 32-byte ``IndexListEntry`` struct. The entries are organized in a double + linked list that mirrors the ordering of instructions in a basic block. This + allows 'tombstone' program points corresponding to instructions that have + been deleted. + + Cretonne uses a 32-bit program point representation that encodes an + instruction or EBB number directly. There are no 'tombstones' for deleted + instructions, and no mirrored linked list of instructions. Live ranges must + be updated when instructions are deleted. + +A consequence of Cretonne's more compact representation is that two program +points can't be compared without the context of a function layout. + + +Spilling algorithm +================== + +There is no one way of implementing spilling, and different tradeoffs between +compilation time and code quality are possible. Any spilling algorithm will +need a way of tracking the register pressure so the colorability condition can +be satisfied. + +Coloring algorithm +================== + +The SSA coloring algorithm is based on a single observation: If two SSA values +interfere, one of the values must be live where the other value is defined. + +We visit the EBBs in a topological order such that all dominating EBBs are +visited before the current EBB. The instructions in an EBB are visited in a +top-down order, and each value define by the instruction is assigned an +available register. With this iteration order, every value that is live at an +instruction has already been assigned to a register. + +This coloring algorith works if the following condition holds: + + At every instruction, consider the values live through the instruction. No + matter how the live values have been assigned to registers, there must be + available registers of the right register classes available for the values + defined by the instruction. + +We'll need to modify this condition in order to deal with the real-world +complications. + +The coloring algorithm needs to keep track of the set of live values at each +instruction. At the top of an EBB, this set can be computed as the union of: + +- The set of live values before the immediately dominating branch or jump + instruction. The topological iteration order guarantees that this set is + available. Values whose live range indicate that they are not live-in to the + current EBB should be filtered out. +- The set of arguments to the EBB. These values should all be live-in, although + it is possible that some are dead and never used anywhere. + +For each live value, we also track its kill point in the current EBB. This is +the last instruction to use the value in the EBB. Values that are live-out +through the EBB terminator don't have a kill point. Note that the kill point +can be a branch to another EBB that uses the value, so the kill instruction +doesn't have to be a use of the value. + +When advancing past an instruction, the live set is updated: + +- Any values whose kill point is the current instruction are removed. +- Any values defined by the instruction are added, unless their kill point is + the current instruction. This corresponds to a dead def which has no uses. From 353caf23cdcf2c9ea09ecbac10288510e31c6e07 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 22 Nov 2016 10:51:42 -0800 Subject: [PATCH 0453/3084] Generate register bank descriptions. Use the information in the ISA's registers.py files to generate a RegInfo Rust data structure. --- lib/cretonne/meta/build.py | 2 + lib/cretonne/meta/gen_registers.py | 51 ++++++++++ lib/cretonne/meta/isa/arm32/__init__.py | 1 + lib/cretonne/meta/isa/arm64/__init__.py | 1 + lib/cretonne/meta/isa/intel/__init__.py | 1 + lib/cretonne/meta/isa/riscv/__init__.py | 2 +- lib/cretonne/src/isa/mod.rs | 5 + lib/cretonne/src/isa/registers.rs | 126 ++++++++++++++++++++++++ lib/cretonne/src/isa/riscv/mod.rs | 7 +- lib/cretonne/src/isa/riscv/registers.py | 1 + lib/cretonne/src/isa/riscv/registers.rs | 37 +++++++ 11 files changed, 232 insertions(+), 2 deletions(-) create mode 100644 lib/cretonne/meta/gen_registers.py create mode 100644 lib/cretonne/src/isa/registers.rs create mode 100644 lib/cretonne/src/isa/riscv/registers.py create mode 100644 lib/cretonne/src/isa/riscv/registers.rs diff --git a/lib/cretonne/meta/build.py b/lib/cretonne/meta/build.py index 51f8e7c056..dd7aeac5f9 100644 --- a/lib/cretonne/meta/build.py +++ b/lib/cretonne/meta/build.py @@ -11,6 +11,7 @@ import gen_settings import gen_build_deps import gen_encoding import gen_legalizer +import gen_registers parser = argparse.ArgumentParser(description='Generate sources for Cretonne.') parser.add_argument('--out-dir', help='set output directory') @@ -25,4 +26,5 @@ gen_instr.generate(isas, out_dir) gen_settings.generate(isas, out_dir) gen_encoding.generate(isas, out_dir) gen_legalizer.generate(isas, out_dir) +gen_registers.generate(isas, out_dir) gen_build_deps.generate() diff --git a/lib/cretonne/meta/gen_registers.py b/lib/cretonne/meta/gen_registers.py new file mode 100644 index 0000000000..323aec6173 --- /dev/null +++ b/lib/cretonne/meta/gen_registers.py @@ -0,0 +1,51 @@ +""" +Generate register bank descriptions for each ISA. +""" + +from __future__ import absolute_import +import srcgen + +try: + from typing import Sequence # noqa + from cdsl.isa import TargetISA # noqa + from cdsl.registers import RegBank # noqa +except ImportError: + pass + + +def gen_regbank(regbank, fmt): + # type: (RegBank, srcgen.Formatter) -> None + """ + Emit a static data definition for regbank. + """ + with fmt.indented( + 'RegBank {{'.format(regbank.name), '},'): + fmt.line('name: "{}",'.format(regbank.name)) + fmt.line('first_unit: {},'.format(regbank.first_unit)) + fmt.line('units: {},'.format(regbank.units)) + fmt.line( + 'names: &[{}],' + .format(', '.join('"{}"'.format(n) for n in regbank.names))) + fmt.line('prefix: "{}",'.format(regbank.prefix)) + + +def gen_isa(isa, fmt): + # type: (TargetISA, srcgen.Formatter) -> None + """ + Generate register tables for isa. + """ + if not isa.regbanks: + print('cargo:warning={} has no register banks'.format(isa.name)) + with fmt.indented('pub static INFO: RegInfo = RegInfo {', '};'): + # Bank descriptors. + with fmt.indented('banks: &[', '],'): + for regbank in isa.regbanks: + gen_regbank(regbank, fmt) + + +def generate(isas, out_dir): + # type: (Sequence[TargetISA], str) -> None + for isa in isas: + fmt = srcgen.Formatter() + gen_isa(isa, fmt) + fmt.update_file('registers-{}.rs'.format(isa.name), out_dir) diff --git a/lib/cretonne/meta/isa/arm32/__init__.py b/lib/cretonne/meta/isa/arm32/__init__.py index 9c992d0c17..52f150b8e1 100644 --- a/lib/cretonne/meta/isa/arm32/__init__.py +++ b/lib/cretonne/meta/isa/arm32/__init__.py @@ -8,6 +8,7 @@ This target ISA generates code for ARMv7 and ARMv8 CPUs in 32-bit mode from __future__ import absolute_import from . import defs +from . import registers # noqa # Re-export the primary target ISA definition. ISA = defs.ISA.finish() diff --git a/lib/cretonne/meta/isa/arm64/__init__.py b/lib/cretonne/meta/isa/arm64/__init__.py index 198fc338b1..75f5e85f38 100644 --- a/lib/cretonne/meta/isa/arm64/__init__.py +++ b/lib/cretonne/meta/isa/arm64/__init__.py @@ -7,6 +7,7 @@ ARMv8 CPUs running the Aarch64 architecture. from __future__ import absolute_import from . import defs +from . import registers # noqa # Re-export the primary target ISA definition. ISA = defs.ISA.finish() diff --git a/lib/cretonne/meta/isa/intel/__init__.py b/lib/cretonne/meta/isa/intel/__init__.py index a97b424099..81948699d2 100644 --- a/lib/cretonne/meta/isa/intel/__init__.py +++ b/lib/cretonne/meta/isa/intel/__init__.py @@ -17,6 +17,7 @@ is no x87 floating point support. from __future__ import absolute_import from . import defs +from . import registers # noqa # Re-export the primary target ISA definition. ISA = defs.ISA.finish() diff --git a/lib/cretonne/meta/isa/riscv/__init__.py b/lib/cretonne/meta/isa/riscv/__init__.py index 3d5d19ed99..cf61cbdcf5 100644 --- a/lib/cretonne/meta/isa/riscv/__init__.py +++ b/lib/cretonne/meta/isa/riscv/__init__.py @@ -26,7 +26,7 @@ RV32G / RV64G """ from __future__ import absolute_import from . import defs -from . import encodings, settings # noqa +from . import encodings, settings, registers # noqa # Re-export the primary target ISA definition. ISA = defs.ISA.finish() diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index aa7b9a76d8..b1b3a8a8df 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -41,12 +41,14 @@ //! concurrent function compilations. pub use isa::encoding::Encoding; +pub use isa::registers::{RegUnit, RegBank, RegInfo}; use settings; use ir::{InstructionData, DataFlowGraph}; pub mod riscv; mod encoding; mod enc_tables; +mod registers; /// Look for a supported ISA with the given `name`. /// Return a builder that can create a corresponding `TargetIsa`. @@ -108,6 +110,9 @@ pub trait TargetIsa { /// Get the ISA-independent flags that were used to make this trait object. fn flags(&self) -> &settings::Flags; + /// Get a data structure describing the registers in this ISA. + fn register_info(&self) -> &RegInfo; + /// Encode an instruction after determining it is legal. /// /// If `inst` can legally be encoded in this ISA, produce the corresponding `Encoding` object. diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs new file mode 100644 index 0000000000..8736f06e84 --- /dev/null +++ b/lib/cretonne/src/isa/registers.rs @@ -0,0 +1,126 @@ +//! Data structures describing the registers in an ISA. + +use std::fmt; + +/// Register units are the smallest units of register allocation. +/// +/// Normally there is a 1-1 correspondence between registers and register units, but when an ISA +/// has aliasing registers, the aliasing can be modeled with registers that cover multiple +/// register units. +/// +/// The register allocator will enforce that each register unit only gets used for one thing. +pub type RegUnit = u16; + +/// The register units in a target ISA are divided into disjoint register banks. Each bank covers a +/// contiguous range of register units. +/// +/// The `RegBank` struct provides a static description of a register bank. +pub struct RegBank { + /// The name of this register bank as defined in the ISA's `registers.py` file. + pub name: &'static str, + + /// The first register unit in this bank. + pub first_unit: RegUnit, + + /// The total number of register units in this bank. + pub units: u16, + + /// Array of specially named register units. This array can be shorter than the number of units + /// in the bank. + pub names: &'static [&'static str], + + /// Name prefix to use for those register units in the bank not covered by the `names` array. + /// The remaining register units will be named this prefix followed by their decimal offset in + /// the bank. So with a prefix `r`, registers will be named `r8`, `r9`, ... + pub prefix: &'static str, +} + +impl RegBank { + /// Does this bank contain `regunit`? + fn contains(&self, regunit: RegUnit) -> bool { + regunit >= self.first_unit && regunit - self.first_unit < self.units + } + + /// Try to parse a regunit name. The name is not expected to begin with `%`. + fn parse_regunit(&self, name: &str) -> Option { + match self.names.iter().position(|&x| x == name) { + Some(offset) => { + // This is one of the special-cased names. + Some(offset as RegUnit) + } + None => { + // Try a regular prefixed name. + if name.starts_with(self.prefix) { + name[self.prefix.len()..].parse().ok() + } else { + None + } + } + } + .and_then(|offset| { + if offset < self.units { + Some(offset + self.first_unit) + } else { + None + } + }) + } + + /// Write `regunit` to `w`, assuming that it belongs to this bank. + /// All regunits are written with a `%` prefix. + fn write_regunit(&self, f: &mut fmt::Formatter, regunit: RegUnit) -> fmt::Result { + let offset = regunit - self.first_unit; + assert!(offset < self.units); + if (offset as usize) < self.names.len() { + write!(f, "%{}", self.names[offset as usize]) + } else { + write!(f, "%{}{}", self.prefix, offset) + } + } +} + +/// Information about the registers in an ISA. +/// +/// The `RegUnit` data structure collects all relevant static information about the registers in an +/// ISA. +pub struct RegInfo { + /// All register banks, ordered by their `first_unit`. The register banks are disjoint, but + /// there may be holes of unused register unit numbers between banks due to alignment. + pub banks: &'static [RegBank], +} + +impl RegInfo { + /// Get the register bank holding `regunit`. + pub fn bank_containing_regunit(&self, regunit: RegUnit) -> Option<&RegBank> { + // We could do a binary search, but most ISAs have only two register banks... + self.banks.iter().find(|b| b.contains(regunit)) + } + + /// Try to parse a regunit name. The name is not expected to begin with `%`. + pub fn parse_regunit(&self, name: &str) -> Option { + self.banks.iter().filter_map(|b| b.parse_regunit(name)).next() + } + + /// Make a temporary object that can display a register unit. + pub fn display_regunit(&self, regunit: RegUnit) -> DisplayRegUnit { + DisplayRegUnit { + regunit: regunit, + reginfo: self, + } + } +} + +/// Temporary object that holds enough information to print a register unit. +pub struct DisplayRegUnit<'a> { + pub regunit: RegUnit, + pub reginfo: &'a RegInfo, +} + +impl<'a> fmt::Display for DisplayRegUnit<'a> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self.reginfo.bank_containing_regunit(self.regunit) { + Some(b) => b.write_regunit(f, self.regunit), + None => write!(f, "%INVALID{}", self.regunit), + } + } +} diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 63ab5ccf89..762df2249c 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -2,11 +2,12 @@ pub mod settings; mod enc_tables; +mod registers; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, Encoding, Legalize}; +use isa::{TargetIsa, RegInfo, Encoding, Legalize}; use ir::{InstructionData, DataFlowGraph}; #[allow(dead_code)] @@ -48,6 +49,10 @@ impl TargetIsa for Isa { &self.shared_flags } + fn register_info(&self) -> &RegInfo { + ®isters::INFO + } + fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Result { lookup_enclist(inst.first_type(), inst.opcode(), diff --git a/lib/cretonne/src/isa/riscv/registers.py b/lib/cretonne/src/isa/riscv/registers.py new file mode 100644 index 0000000000..8b13789179 --- /dev/null +++ b/lib/cretonne/src/isa/riscv/registers.py @@ -0,0 +1 @@ + diff --git a/lib/cretonne/src/isa/riscv/registers.rs b/lib/cretonne/src/isa/riscv/registers.rs new file mode 100644 index 0000000000..a3fe4739e8 --- /dev/null +++ b/lib/cretonne/src/isa/riscv/registers.rs @@ -0,0 +1,37 @@ +//! RISC-V register descriptions. + +use isa::registers::{RegBank, RegInfo}; + +include!(concat!(env!("OUT_DIR"), "/registers-riscv.rs")); + +#[cfg(test)] +mod tests { + use super::INFO; + use isa::RegUnit; + + #[test] + fn unit_encodings() { + assert_eq!(INFO.parse_regunit("x0"), Some(0)); + assert_eq!(INFO.parse_regunit("x31"), Some(31)); + assert_eq!(INFO.parse_regunit("f0"), Some(32)); + assert_eq!(INFO.parse_regunit("f31"), Some(63)); + + assert_eq!(INFO.parse_regunit("x32"), None); + assert_eq!(INFO.parse_regunit("f32"), None); + } + + #[test] + fn unit_names() { + fn uname(ru: RegUnit) -> String { + INFO.display_regunit(ru).to_string() + } + + assert_eq!(uname(0), "%x0"); + assert_eq!(uname(1), "%x1"); + assert_eq!(uname(31), "%x31"); + assert_eq!(uname(32), "%f0"); + assert_eq!(uname(33), "%f1"); + assert_eq!(uname(63), "%f31"); + assert_eq!(uname(64), "%INVALID64"); + } +} From fb4db38dd6ac369611637fa91c056cae11dac1c5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 23 Nov 2016 10:42:07 -0800 Subject: [PATCH 0454/3084] Fill in boilerplate for Intel and ARM targets. The intel, arm32, and arm32 targets were only defined in the meta language previously. Add Rust implementations too. This is mostly boilerplate, except for the unit tests in the registers.rs files. --- lib/cretonne/meta/base/settings.py | 2 + lib/cretonne/meta/gen_encoding.py | 2 + lib/cretonne/meta/gen_settings.py | 12 ++-- lib/cretonne/meta/isa/arm32/__init__.py | 2 +- lib/cretonne/meta/isa/arm32/registers.py | 19 +++--- lib/cretonne/meta/isa/arm32/settings.py | 11 ++++ lib/cretonne/meta/isa/arm64/__init__.py | 2 +- lib/cretonne/meta/isa/arm64/settings.py | 11 ++++ lib/cretonne/meta/isa/intel/__init__.py | 2 +- lib/cretonne/meta/isa/intel/settings.py | 11 ++++ lib/cretonne/src/isa/arm32/enc_tables.rs | 8 +++ lib/cretonne/src/isa/arm32/mod.rs | 73 ++++++++++++++++++++++++ lib/cretonne/src/isa/arm32/registers.rs | 32 +++++++++++ lib/cretonne/src/isa/arm32/settings.rs | 9 +++ lib/cretonne/src/isa/arm64/enc_tables.rs | 8 +++ lib/cretonne/src/isa/arm64/mod.rs | 66 +++++++++++++++++++++ lib/cretonne/src/isa/arm64/registers.rs | 37 ++++++++++++ lib/cretonne/src/isa/arm64/settings.rs | 9 +++ lib/cretonne/src/isa/intel/enc_tables.rs | 8 +++ lib/cretonne/src/isa/intel/mod.rs | 73 ++++++++++++++++++++++++ lib/cretonne/src/isa/intel/registers.rs | 49 ++++++++++++++++ lib/cretonne/src/isa/intel/settings.rs | 9 +++ lib/cretonne/src/isa/mod.rs | 18 ++++++ lib/cretonne/src/settings.rs | 1 + 24 files changed, 456 insertions(+), 18 deletions(-) create mode 100644 lib/cretonne/meta/isa/arm32/settings.py create mode 100644 lib/cretonne/meta/isa/arm64/settings.py create mode 100644 lib/cretonne/meta/isa/intel/settings.py create mode 100644 lib/cretonne/src/isa/arm32/enc_tables.rs create mode 100644 lib/cretonne/src/isa/arm32/mod.rs create mode 100644 lib/cretonne/src/isa/arm32/registers.rs create mode 100644 lib/cretonne/src/isa/arm32/settings.rs create mode 100644 lib/cretonne/src/isa/arm64/enc_tables.rs create mode 100644 lib/cretonne/src/isa/arm64/mod.rs create mode 100644 lib/cretonne/src/isa/arm64/registers.rs create mode 100644 lib/cretonne/src/isa/arm64/settings.rs create mode 100644 lib/cretonne/src/isa/intel/enc_tables.rs create mode 100644 lib/cretonne/src/isa/intel/mod.rs create mode 100644 lib/cretonne/src/isa/intel/registers.rs create mode 100644 lib/cretonne/src/isa/intel/settings.rs diff --git a/lib/cretonne/meta/base/settings.py b/lib/cretonne/meta/base/settings.py index c6f87f7fbb..1f7e21668b 100644 --- a/lib/cretonne/meta/base/settings.py +++ b/lib/cretonne/meta/base/settings.py @@ -20,6 +20,8 @@ opt_level = EnumSetting( is_64bit = BoolSetting("Enable 64-bit code generation") +is_compressed = BoolSetting("Enable compressed instructions") + enable_float = BoolSetting( """Enable the use of floating-point instructions""", default=True) diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index e404fb79ec..8477f31de2 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -91,6 +91,8 @@ def emit_instps(instps, fmt): Emit a function for matching instruction predicates. """ + if not instps: + fmt.line('#[allow(unused_variables)]') with fmt.indented( 'pub fn check_instp(inst: &InstructionData, instp_idx: u16) ' + '-> bool {', '}'): diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index 74aa0dca48..3f435c8fc3 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -192,6 +192,7 @@ def gen_constructor(sgrp, parent, fmt): p = sgrp.parent args = '{}: &{}::Flags, {}'.format(p.name, p.qual_mod, args) fmt.doc_comment('Create flags {} settings group.'.format(sgrp.name)) + fmt.line('#[allow(unused_variables)]') with fmt.indented( 'pub fn new({}) -> Flags {{'.format(args), '}'): fmt.line('let bvec = builder.state_for("{}");'.format(sgrp.name)) @@ -259,9 +260,8 @@ def generate(isas, out_dir): # Generate ISA-specific settings. for isa in isas: - if isa.settings: - isa.settings.qual_mod = 'isa::{}::settings'.format( - isa.settings.name) - fmt = srcgen.Formatter() - gen_group(isa.settings, fmt) - fmt.update_file('settings-{}.rs'.format(isa.name), out_dir) + isa.settings.qual_mod = 'isa::{}::settings'.format( + isa.settings.name) + fmt = srcgen.Formatter() + gen_group(isa.settings, fmt) + fmt.update_file('settings-{}.rs'.format(isa.name), out_dir) diff --git a/lib/cretonne/meta/isa/arm32/__init__.py b/lib/cretonne/meta/isa/arm32/__init__.py index 52f150b8e1..d2de00667a 100644 --- a/lib/cretonne/meta/isa/arm32/__init__.py +++ b/lib/cretonne/meta/isa/arm32/__init__.py @@ -8,7 +8,7 @@ This target ISA generates code for ARMv7 and ARMv8 CPUs in 32-bit mode from __future__ import absolute_import from . import defs -from . import registers # noqa +from . import settings, registers # noqa # Re-export the primary target ISA definition. ISA = defs.ISA.finish() diff --git a/lib/cretonne/meta/isa/arm32/registers.py b/lib/cretonne/meta/isa/arm32/registers.py index 3084c2bf98..af031e145b 100644 --- a/lib/cretonne/meta/isa/arm32/registers.py +++ b/lib/cretonne/meta/isa/arm32/registers.py @@ -6,15 +6,7 @@ from cdsl.registers import RegBank, RegClass from .defs import ISA -# Special register units: -# - r15 is the program counter. -# - r14 is the link register. -# - r13 is usually the stack pointer. -IntRegs = RegBank( - 'IntRegs', ISA, - 'General purpose registers', - units=16, prefix='r') - +# Define the larger float bank first to avoid the alignment gap. FloatRegs = RegBank( 'FloatRegs', ISA, r""" Floating point registers. @@ -28,6 +20,15 @@ FloatRegs = RegBank( """, units=64, prefix='s') +# Special register units: +# - r15 is the program counter. +# - r14 is the link register. +# - r13 is usually the stack pointer. +IntRegs = RegBank( + 'IntRegs', ISA, + 'General purpose registers', + units=16, prefix='r') + GPR = RegClass('GPR', IntRegs) S = RegClass('S', FloatRegs, count=32) D = RegClass('D', FloatRegs, width=2) diff --git a/lib/cretonne/meta/isa/arm32/settings.py b/lib/cretonne/meta/isa/arm32/settings.py new file mode 100644 index 0000000000..5cc948cf2d --- /dev/null +++ b/lib/cretonne/meta/isa/arm32/settings.py @@ -0,0 +1,11 @@ +""" +ARM32 settings. +""" +from __future__ import absolute_import +from cdsl.settings import SettingGroup +import base.settings as shared +from .defs import ISA + +ISA.settings = SettingGroup('arm32', parent=shared.group) + +ISA.settings.close(globals()) diff --git a/lib/cretonne/meta/isa/arm64/__init__.py b/lib/cretonne/meta/isa/arm64/__init__.py index 75f5e85f38..3dd69feb4b 100644 --- a/lib/cretonne/meta/isa/arm64/__init__.py +++ b/lib/cretonne/meta/isa/arm64/__init__.py @@ -7,7 +7,7 @@ ARMv8 CPUs running the Aarch64 architecture. from __future__ import absolute_import from . import defs -from . import registers # noqa +from . import settings, registers # noqa # Re-export the primary target ISA definition. ISA = defs.ISA.finish() diff --git a/lib/cretonne/meta/isa/arm64/settings.py b/lib/cretonne/meta/isa/arm64/settings.py new file mode 100644 index 0000000000..9a2fc13dc7 --- /dev/null +++ b/lib/cretonne/meta/isa/arm64/settings.py @@ -0,0 +1,11 @@ +""" +ARM64 settings. +""" +from __future__ import absolute_import +from cdsl.settings import SettingGroup +import base.settings as shared +from .defs import ISA + +ISA.settings = SettingGroup('arm64', parent=shared.group) + +ISA.settings.close(globals()) diff --git a/lib/cretonne/meta/isa/intel/__init__.py b/lib/cretonne/meta/isa/intel/__init__.py index 81948699d2..6aea0fd288 100644 --- a/lib/cretonne/meta/isa/intel/__init__.py +++ b/lib/cretonne/meta/isa/intel/__init__.py @@ -17,7 +17,7 @@ is no x87 floating point support. from __future__ import absolute_import from . import defs -from . import registers # noqa +from . import settings, registers # noqa # Re-export the primary target ISA definition. ISA = defs.ISA.finish() diff --git a/lib/cretonne/meta/isa/intel/settings.py b/lib/cretonne/meta/isa/intel/settings.py new file mode 100644 index 0000000000..42e1a0ab99 --- /dev/null +++ b/lib/cretonne/meta/isa/intel/settings.py @@ -0,0 +1,11 @@ +""" +Intel settings. +""" +from __future__ import absolute_import +from cdsl.settings import SettingGroup +import base.settings as shared +from .defs import ISA + +ISA.settings = SettingGroup('intel', parent=shared.group) + +ISA.settings.close(globals()) diff --git a/lib/cretonne/src/isa/arm32/enc_tables.rs b/lib/cretonne/src/isa/arm32/enc_tables.rs new file mode 100644 index 0000000000..e40362a32f --- /dev/null +++ b/lib/cretonne/src/isa/arm32/enc_tables.rs @@ -0,0 +1,8 @@ +//! Encoding tables for ARM32 ISA. + +use ir::InstructionData; +use ir::instructions::InstructionFormat; +use ir::types; +use isa::enc_tables::{Level1Entry, Level2Entry}; + +include!(concat!(env!("OUT_DIR"), "/encoding-arm32.rs")); diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs new file mode 100644 index 0000000000..617f52f43e --- /dev/null +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -0,0 +1,73 @@ +//! ARM 32-bit Instruction Set Architecture. + +pub mod settings; +mod enc_tables; +mod registers; + +use super::super::settings as shared_settings; +use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; +use isa::Builder as IsaBuilder; +use isa::{TargetIsa, RegInfo, Encoding, Legalize}; +use ir::{InstructionData, DataFlowGraph}; + +#[allow(dead_code)] +struct Isa { + shared_flags: shared_settings::Flags, + isa_flags: settings::Flags, + cpumode: &'static [shared_enc_tables::Level1Entry], +} + +/// Get an ISA builder for creating ARM32 targets. +pub fn isa_builder() -> IsaBuilder { + IsaBuilder { + setup: settings::builder(), + constructor: isa_constructor, + } +} + +fn isa_constructor(shared_flags: shared_settings::Flags, + builder: &shared_settings::Builder) + -> Box { + let level1 = if shared_flags.is_compressed() { + &enc_tables::LEVEL1_T32[..] + } else { + &enc_tables::LEVEL1_A32[..] + }; + Box::new(Isa { + isa_flags: settings::Flags::new(&shared_flags, builder), + shared_flags: shared_flags, + cpumode: level1, + }) +} + +impl TargetIsa for Isa { + fn name(&self) -> &'static str { + "arm32" + } + + fn flags(&self) -> &shared_settings::Flags { + &self.shared_flags + } + + fn register_info(&self) -> &RegInfo { + ®isters::INFO + } + + fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Result { + lookup_enclist(inst.first_type(), + inst.opcode(), + self.cpumode, + &enc_tables::LEVEL2[..]) + .and_then(|enclist_offset| { + general_encoding(enclist_offset, + &enc_tables::ENCLISTS[..], + |instp| enc_tables::check_instp(inst, instp), + |isap| self.isa_flags.numbered_predicate(isap as usize)) + .ok_or(Legalize::Expand) + }) + } + + fn recipe_names(&self) -> &'static [&'static str] { + &enc_tables::RECIPE_NAMES[..] + } +} diff --git a/lib/cretonne/src/isa/arm32/registers.rs b/lib/cretonne/src/isa/arm32/registers.rs new file mode 100644 index 0000000000..30e7fd58cf --- /dev/null +++ b/lib/cretonne/src/isa/arm32/registers.rs @@ -0,0 +1,32 @@ +//! ARM32 register descriptions. + +use isa::registers::{RegBank, RegInfo}; + +include!(concat!(env!("OUT_DIR"), "/registers-arm32.rs")); + +#[cfg(test)] +mod tests { + use super::INFO; + use isa::RegUnit; + + #[test] + fn unit_encodings() { + assert_eq!(INFO.parse_regunit("s0"), Some(0)); + assert_eq!(INFO.parse_regunit("s31"), Some(31)); + assert_eq!(INFO.parse_regunit("s32"), Some(32)); + assert_eq!(INFO.parse_regunit("r0"), Some(64)); + assert_eq!(INFO.parse_regunit("r15"), Some(79)); + } + + #[test] + fn unit_names() { + fn uname(ru: RegUnit) -> String { + INFO.display_regunit(ru).to_string() + } + + assert_eq!(uname(0), "%s0"); + assert_eq!(uname(1), "%s1"); + assert_eq!(uname(31), "%s31"); + assert_eq!(uname(64), "%r0"); + } +} diff --git a/lib/cretonne/src/isa/arm32/settings.rs b/lib/cretonne/src/isa/arm32/settings.rs new file mode 100644 index 0000000000..e857716a64 --- /dev/null +++ b/lib/cretonne/src/isa/arm32/settings.rs @@ -0,0 +1,9 @@ +//! ARM32 Settings. + +use settings::{self, detail, Builder}; +use std::fmt; + +// Include code generated by `lib/cretonne/meta/gen_settings.py`. This file contains a public +// `Flags` struct with an impl for all of the settings defined in +// `lib/cretonne/meta/cretonne/settings.py`. +include!(concat!(env!("OUT_DIR"), "/settings-arm32.rs")); diff --git a/lib/cretonne/src/isa/arm64/enc_tables.rs b/lib/cretonne/src/isa/arm64/enc_tables.rs new file mode 100644 index 0000000000..dbe65c4e8f --- /dev/null +++ b/lib/cretonne/src/isa/arm64/enc_tables.rs @@ -0,0 +1,8 @@ +//! Encoding tables for ARM64 ISA. + +use ir::InstructionData; +use ir::instructions::InstructionFormat; +use ir::types; +use isa::enc_tables::{Level1Entry, Level2Entry}; + +include!(concat!(env!("OUT_DIR"), "/encoding-arm64.rs")); diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs new file mode 100644 index 0000000000..a4367a878b --- /dev/null +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -0,0 +1,66 @@ +//! ARM 64-bit Instruction Set Architecture. + +pub mod settings; +mod enc_tables; +mod registers; + +use super::super::settings as shared_settings; +use isa::enc_tables::{lookup_enclist, general_encoding}; +use isa::Builder as IsaBuilder; +use isa::{TargetIsa, RegInfo, Encoding, Legalize}; +use ir::{InstructionData, DataFlowGraph}; + +#[allow(dead_code)] +struct Isa { + shared_flags: shared_settings::Flags, + isa_flags: settings::Flags, +} + +/// Get an ISA builder for creating ARM64 targets. +pub fn isa_builder() -> IsaBuilder { + IsaBuilder { + setup: settings::builder(), + constructor: isa_constructor, + } +} + +fn isa_constructor(shared_flags: shared_settings::Flags, + builder: &shared_settings::Builder) + -> Box { + Box::new(Isa { + isa_flags: settings::Flags::new(&shared_flags, builder), + shared_flags: shared_flags, + }) +} + +impl TargetIsa for Isa { + fn name(&self) -> &'static str { + "arm64" + } + + fn flags(&self) -> &shared_settings::Flags { + &self.shared_flags + } + + fn register_info(&self) -> &RegInfo { + ®isters::INFO + } + + fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Result { + lookup_enclist(inst.first_type(), + inst.opcode(), + &enc_tables::LEVEL1_A64[..], + &enc_tables::LEVEL2[..]) + .and_then(|enclist_offset| { + general_encoding(enclist_offset, + &enc_tables::ENCLISTS[..], + |instp| enc_tables::check_instp(inst, instp), + |isap| self.isa_flags.numbered_predicate(isap as usize)) + .ok_or(Legalize::Expand) + }) + } + + fn recipe_names(&self) -> &'static [&'static str] { + &enc_tables::RECIPE_NAMES[..] + } +} diff --git a/lib/cretonne/src/isa/arm64/registers.rs b/lib/cretonne/src/isa/arm64/registers.rs new file mode 100644 index 0000000000..2420db6df1 --- /dev/null +++ b/lib/cretonne/src/isa/arm64/registers.rs @@ -0,0 +1,37 @@ +//! ARM64 register descriptions. + +use isa::registers::{RegBank, RegInfo}; + +include!(concat!(env!("OUT_DIR"), "/registers-arm64.rs")); + +#[cfg(test)] +mod tests { + use super::INFO; + use isa::RegUnit; + + #[test] + fn unit_encodings() { + assert_eq!(INFO.parse_regunit("x0"), Some(0)); + assert_eq!(INFO.parse_regunit("x31"), Some(31)); + assert_eq!(INFO.parse_regunit("v0"), Some(32)); + assert_eq!(INFO.parse_regunit("v31"), Some(63)); + + assert_eq!(INFO.parse_regunit("x32"), None); + assert_eq!(INFO.parse_regunit("v32"), None); + } + + #[test] + fn unit_names() { + fn uname(ru: RegUnit) -> String { + INFO.display_regunit(ru).to_string() + } + + assert_eq!(uname(0), "%x0"); + assert_eq!(uname(1), "%x1"); + assert_eq!(uname(31), "%x31"); + assert_eq!(uname(32), "%v0"); + assert_eq!(uname(33), "%v1"); + assert_eq!(uname(63), "%v31"); + assert_eq!(uname(64), "%INVALID64"); + } +} diff --git a/lib/cretonne/src/isa/arm64/settings.rs b/lib/cretonne/src/isa/arm64/settings.rs new file mode 100644 index 0000000000..6427d7be99 --- /dev/null +++ b/lib/cretonne/src/isa/arm64/settings.rs @@ -0,0 +1,9 @@ +//! ARM64 Settings. + +use settings::{self, detail, Builder}; +use std::fmt; + +// Include code generated by `lib/cretonne/meta/gen_settings.py`. This file contains a public +// `Flags` struct with an impl for all of the settings defined in +// `lib/cretonne/meta/cretonne/settings.py`. +include!(concat!(env!("OUT_DIR"), "/settings-arm64.rs")); diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs new file mode 100644 index 0000000000..842629b058 --- /dev/null +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -0,0 +1,8 @@ +//! Encoding tables for Intel ISAs. + +use ir::InstructionData; +use ir::instructions::InstructionFormat; +use ir::types; +use isa::enc_tables::{Level1Entry, Level2Entry}; + +include!(concat!(env!("OUT_DIR"), "/encoding-intel.rs")); diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs new file mode 100644 index 0000000000..acad76743b --- /dev/null +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -0,0 +1,73 @@ +//! Intel Instruction Set Architectures. + +pub mod settings; +mod enc_tables; +mod registers; + +use super::super::settings as shared_settings; +use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; +use isa::Builder as IsaBuilder; +use isa::{TargetIsa, RegInfo, Encoding, Legalize}; +use ir::{InstructionData, DataFlowGraph}; + +#[allow(dead_code)] +struct Isa { + shared_flags: shared_settings::Flags, + isa_flags: settings::Flags, + cpumode: &'static [shared_enc_tables::Level1Entry], +} + +/// Get an ISA builder for creating Intel targets. +pub fn isa_builder() -> IsaBuilder { + IsaBuilder { + setup: settings::builder(), + constructor: isa_constructor, + } +} + +fn isa_constructor(shared_flags: shared_settings::Flags, + builder: &shared_settings::Builder) + -> Box { + let level1 = if shared_flags.is_64bit() { + &enc_tables::LEVEL1_I64[..] + } else { + &enc_tables::LEVEL1_I32[..] + }; + Box::new(Isa { + isa_flags: settings::Flags::new(&shared_flags, builder), + shared_flags: shared_flags, + cpumode: level1, + }) +} + +impl TargetIsa for Isa { + fn name(&self) -> &'static str { + "intel" + } + + fn flags(&self) -> &shared_settings::Flags { + &self.shared_flags + } + + fn register_info(&self) -> &RegInfo { + ®isters::INFO + } + + fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Result { + lookup_enclist(inst.first_type(), + inst.opcode(), + self.cpumode, + &enc_tables::LEVEL2[..]) + .and_then(|enclist_offset| { + general_encoding(enclist_offset, + &enc_tables::ENCLISTS[..], + |instp| enc_tables::check_instp(inst, instp), + |isap| self.isa_flags.numbered_predicate(isap as usize)) + .ok_or(Legalize::Expand) + }) + } + + fn recipe_names(&self) -> &'static [&'static str] { + &enc_tables::RECIPE_NAMES[..] + } +} diff --git a/lib/cretonne/src/isa/intel/registers.rs b/lib/cretonne/src/isa/intel/registers.rs new file mode 100644 index 0000000000..1946d0fc57 --- /dev/null +++ b/lib/cretonne/src/isa/intel/registers.rs @@ -0,0 +1,49 @@ +//! Intel register descriptions. + +use isa::registers::{RegBank, RegInfo}; + +include!(concat!(env!("OUT_DIR"), "/registers-intel.rs")); + +#[cfg(test)] +mod tests { + use super::INFO; + use isa::RegUnit; + + #[test] + fn unit_encodings() { + // The encoding of integer registers is not alphabetical. + assert_eq!(INFO.parse_regunit("rax"), Some(0)); + assert_eq!(INFO.parse_regunit("rbx"), Some(3)); + assert_eq!(INFO.parse_regunit("rcx"), Some(1)); + assert_eq!(INFO.parse_regunit("rdx"), Some(2)); + assert_eq!(INFO.parse_regunit("rsi"), Some(6)); + assert_eq!(INFO.parse_regunit("rdi"), Some(7)); + assert_eq!(INFO.parse_regunit("rbp"), Some(5)); + assert_eq!(INFO.parse_regunit("rsp"), Some(4)); + assert_eq!(INFO.parse_regunit("r8"), Some(8)); + assert_eq!(INFO.parse_regunit("r15"), Some(15)); + + assert_eq!(INFO.parse_regunit("xmm0"), Some(16)); + assert_eq!(INFO.parse_regunit("xmm15"), Some(31)); + } + + #[test] + fn unit_names() { + fn uname(ru: RegUnit) -> String { + INFO.display_regunit(ru).to_string() + } + + assert_eq!(uname(0), "%rax"); + assert_eq!(uname(3), "%rbx"); + assert_eq!(uname(1), "%rcx"); + assert_eq!(uname(2), "%rdx"); + assert_eq!(uname(6), "%rsi"); + assert_eq!(uname(7), "%rdi"); + assert_eq!(uname(5), "%rbp"); + assert_eq!(uname(4), "%rsp"); + assert_eq!(uname(8), "%r8"); + assert_eq!(uname(15), "%r15"); + assert_eq!(uname(16), "%xmm0"); + assert_eq!(uname(31), "%xmm15"); + } +} diff --git a/lib/cretonne/src/isa/intel/settings.rs b/lib/cretonne/src/isa/intel/settings.rs new file mode 100644 index 0000000000..341eb2dcc9 --- /dev/null +++ b/lib/cretonne/src/isa/intel/settings.rs @@ -0,0 +1,9 @@ +//! Intel Settings. + +use settings::{self, detail, Builder}; +use std::fmt; + +// Include code generated by `lib/cretonne/meta/gen_settings.py`. This file contains a public +// `Flags` struct with an impl for all of the settings defined in +// `lib/cretonne/meta/cretonne/settings.py`. +include!(concat!(env!("OUT_DIR"), "/settings-intel.rs")); diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index b1b3a8a8df..dc66355904 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -46,6 +46,9 @@ use settings; use ir::{InstructionData, DataFlowGraph}; pub mod riscv; +pub mod intel; +pub mod arm32; +pub mod arm64; mod encoding; mod enc_tables; mod registers; @@ -55,6 +58,9 @@ mod registers; pub fn lookup(name: &str) -> Option { match name { "riscv" => riscv_builder(), + "intel" => intel_builder(), + "arm32" => arm32_builder(), + "arm64" => arm64_builder(), _ => None, } } @@ -64,6 +70,18 @@ fn riscv_builder() -> Option { Some(riscv::isa_builder()) } +fn intel_builder() -> Option { + Some(intel::isa_builder()) +} + +fn arm32_builder() -> Option { + Some(arm32::isa_builder()) +} + +fn arm64_builder() -> Option { + Some(arm64::isa_builder()) +} + /// Builder for a `TargetIsa`. /// Modify the ISA-specific settings before creating the `TargetIsa` trait object with `finish`. pub struct Builder { diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 291211b1b3..30ecff68e4 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -270,6 +270,7 @@ mod tests { "[shared]\n\ opt_level = \"default\"\n\ is_64bit = false\n\ + is_compressed = false\n\ enable_float = true\n\ enable_simd = true\n\ enable_atomics = true\n"); From 8285f2a672561dabf13a5c25247ab566e3239e0f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dominik=20Inf=C3=BChr?= Date: Thu, 1 Dec 2016 00:11:06 +0100 Subject: [PATCH 0455/3084] added Opcode flags methods generate `is_branch`, `is_terminator`, `can_trap` methods for `enum Opcode`. --- lib/cretonne/meta/base/instructions.py | 20 ++++++++++---------- lib/cretonne/meta/cdsl/instructions.py | 4 ++++ lib/cretonne/meta/gen_instr.py | 16 ++++++++++++++++ 3 files changed, 30 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index b5a2c784fa..eb98d175ef 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -83,7 +83,7 @@ trap = Instruction( 'trap', r""" Terminate execution unconditionally. """, - is_terminator=True) + is_terminator=True, can_trap=True) trapz = Instruction( 'trapz', r""" @@ -91,7 +91,7 @@ trapz = Instruction( if ``c`` is non-zero, execution continues at the following instruction. """, - ins=c) + ins=c, can_trap=True) trapnz = Instruction( 'trapnz', r""" @@ -99,7 +99,7 @@ trapnz = Instruction( if ``c`` is zero, execution continues at the following instruction. """, - ins=c) + ins=c, can_trap=True) rvals = Operand('rvals', VARIABLE_ARGS, doc='return values') @@ -111,7 +111,7 @@ x_return = Instruction( provided return values. The list of return values must match the function signature's return types. """, - ins=rvals) + ins=rvals, is_terminator=True) FN = Operand( 'FN', @@ -366,7 +366,7 @@ udiv = Instruction( This operation traps if the divisor is zero. """, - ins=(x, y), outs=a) + ins=(x, y), outs=a, can_trap=True) sdiv = Instruction( 'sdiv', r""" @@ -377,7 +377,7 @@ sdiv = Instruction( representable in :math:`B` bits two's complement. This only happens when :math:`x = -2^{B-1}, y = -1`. """, - ins=(x, y), outs=a) + ins=(x, y), outs=a, can_trap=True) urem = Instruction( 'urem', """ @@ -385,7 +385,7 @@ urem = Instruction( This operation traps if the divisor is zero. """, - ins=(x, y), outs=a) + ins=(x, y), outs=a, can_trap=True) srem = Instruction( 'srem', """ @@ -399,7 +399,7 @@ srem = Instruction( dividend. Should we add a ``smod`` instruction for the case where the result has the same sign as the divisor? """, - ins=(x, y), outs=a) + ins=(x, y), outs=a, can_trap=True) a = Operand('a', iB) x = Operand('x', iB) @@ -1136,7 +1136,7 @@ fcvt_to_uint = Instruction( The result type must have the same number of vector lanes as the input. """, - ins=x, outs=a) + ins=x, outs=a, can_trap=True) fcvt_to_sint = Instruction( 'fcvt_to_sint', r""" @@ -1148,7 +1148,7 @@ fcvt_to_sint = Instruction( The result type must have the same number of vector lanes as the input. """, - ins=x, outs=a) + ins=x, outs=a, can_trap=True) x = Operand('x', Int) a = Operand('a', FloatTo) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index ee6b9c4a59..84968c1b6b 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -80,6 +80,7 @@ class Instruction(object): values or `variable_args`. :param is_terminator: This is a terminator instruction. :param is_branch: This is a branch instruction. + :param can_trap: This instruction can trap. """ def __init__(self, name, doc, ins=(), outs=(), **kwargs): @@ -94,6 +95,9 @@ class Instruction(object): self.value_results = tuple( i for i, o in enumerate(self.outs) if o.is_value()) self._verify_polymorphic() + self.is_branch = 'is_branch' in kwargs + self.is_terminator = 'is_terminator' in kwargs + self.can_trap = 'can_trap' in kwargs InstructionGroup.append(self) def __str__(self): diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 11aaa4fa0d..799c080473 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -271,6 +271,22 @@ def gen_opcodes(groups, fmt): fmt.line(i.camel_name + ',') fmt.line() + with fmt.indented('impl Opcode {', '}'): + attrs = ['is_branch', 'is_terminator', 'can_trap'] + + for attr in attrs: + if attr != attrs[0]: + fmt.line() + + with fmt.indented('fn {}(self) -> bool {{' + .format(attr), '}'): + with fmt.indented('match self {', '}'): + for i in instrs: + if getattr(i, attr): + fmt.format('Opcode::{} => true,', i.camel_name, i.name) + + fmt.line('_ => false') + # Generate a private opcode_format table. with fmt.indented( 'const OPCODE_FORMAT: [InstructionFormat; {}] = [' From de0ea3694219f5faa104f1d3a6ee61f5f8e084e4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dominik=20Inf=C3=BChr?= Date: Wed, 7 Dec 2016 19:32:58 +0100 Subject: [PATCH 0456/3084] remove old `is_terminating` function for InstructionData. Use generated `is_terminator()` for `Opcode` instead. `is_terminator`, `can_trap` and `is_branch` functions are now public. fix syntax error --- lib/cretonne/meta/gen_instr.py | 13 +++++++++---- lib/cretonne/src/ir/instructions.rs | 10 ---------- lib/cretonne/src/verifier.rs | 2 +- 3 files changed, 10 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 799c080473..e0055bcf98 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -272,17 +272,22 @@ def gen_opcodes(groups, fmt): fmt.line() with fmt.indented('impl Opcode {', '}'): - attrs = ['is_branch', 'is_terminator', 'can_trap'] + attrs = [ + { 'name': 'is_branch', 'comment': 'True for all branch instructions.' }, + { 'name': 'is_terminator', 'comment': 'True for instructions that terminate EBB.' }, + { 'name': 'can_trap', 'comment': 'True if instruction could trap.' } + ] for attr in attrs: if attr != attrs[0]: fmt.line() - with fmt.indented('fn {}(self) -> bool {{' - .format(attr), '}'): + fmt.doc_comment(attr['comment']) + with fmt.indented('pub fn {}(self) -> bool {{' + .format(attr['name']), '}'): with fmt.indented('match self {', '}'): for i in instrs: - if getattr(i, attr): + if getattr(i, attr['name']): fmt.format('Opcode::{} => true,', i.camel_name, i.name) fmt.line('_ => false') diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index b8c5bdc4fa..21f9b92c3c 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -465,16 +465,6 @@ impl InstructionData { _ => CallInfo::NotACall, } } - - /// Return true if an instruction is terminating, or false otherwise. - pub fn is_terminating<'a>(&'a self) -> bool { - match self { - &InstructionData::Jump { .. } => true, - &InstructionData::Return { .. } => true, - &InstructionData::Nullary { .. } => true, - _ => false, - } - } } /// Information about branch and jump instructions. diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 9e044a738a..939d722e60 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -109,7 +109,7 @@ impl<'a> Verifier<'a> { fn ebb_integrity(&self, ebb: Ebb, inst: Inst) -> Result<()> { - let is_terminator = self.func.dfg[inst].is_terminating(); + let is_terminator = self.func.dfg[inst].opcode().is_terminator(); let is_last_inst = self.func.layout.last_inst(ebb) == inst; if is_terminator && !is_last_inst { From a08a470a1840f3a607d7482177784e3fc29c82e1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 8 Dec 2016 13:30:40 -1000 Subject: [PATCH 0457/3084] Fix Python formatting to keep flake8 happy. Python coding style is verified by the lib/cretonne/meta/check.sh script which is not run as part of test-all.sh. cc @dinfuehr --- lib/cretonne/meta/gen_instr.py | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index e0055bcf98..ac4650bc96 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -273,9 +273,18 @@ def gen_opcodes(groups, fmt): with fmt.indented('impl Opcode {', '}'): attrs = [ - { 'name': 'is_branch', 'comment': 'True for all branch instructions.' }, - { 'name': 'is_terminator', 'comment': 'True for instructions that terminate EBB.' }, - { 'name': 'can_trap', 'comment': 'True if instruction could trap.' } + { + 'name': 'is_branch', + 'comment': 'True for all branch instructions.' + }, + { + 'name': 'is_terminator', + 'comment': 'True for instructions that terminate EBB.' + }, + { + 'name': 'can_trap', + 'comment': 'True if instruction could trap.' + } ] for attr in attrs: @@ -288,7 +297,9 @@ def gen_opcodes(groups, fmt): with fmt.indented('match self {', '}'): for i in instrs: if getattr(i, attr['name']): - fmt.format('Opcode::{} => true,', i.camel_name, i.name) + fmt.format( + 'Opcode::{} => true,', + i.camel_name, i.name) fmt.line('_ => false') From e1c5abaff5f80df0aafeb7f6d01fcd16bffbcb52 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 8 Dec 2016 13:57:28 -1000 Subject: [PATCH 0458/3084] Add a ValueLoc type and the locations table. This table holds the result of register allocation. --- lib/cretonne/src/ir/entities.rs | 11 +++++++++++ lib/cretonne/src/ir/function.rs | 8 ++++++-- lib/cretonne/src/ir/mod.rs | 2 ++ lib/cretonne/src/ir/valueloc.rs | 24 ++++++++++++++++++++++++ 4 files changed, 43 insertions(+), 2 deletions(-) create mode 100644 lib/cretonne/src/ir/valueloc.rs diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index df7d81988f..4570a21009 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -98,6 +98,17 @@ impl Default for Inst { #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct Value(u32); +impl EntityRef for Value { + fn new(index: usize) -> Value { + assert!(index < (u32::MAX as usize)); + Value(index as u32) + } + + fn index(self) -> usize { + self.0 as usize + } +} + /// Value references can either reference an instruction directly, or they can refer to the /// extended value table. pub enum ExpandedValue { diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 844101bf95..db6c811b5a 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -4,8 +4,8 @@ //! instructions. use std::fmt::{self, Display, Debug, Formatter}; -use ir::{FunctionName, Signature, Inst, StackSlot, StackSlotData, JumpTable, JumpTableData, - DataFlowGraph, Layout}; +use ir::{FunctionName, Signature, Value, Inst, StackSlot, StackSlotData, JumpTable, JumpTableData, + ValueLoc, DataFlowGraph, Layout}; use isa::Encoding; use entity_map::{EntityMap, PrimaryEntityData}; use write::write_function; @@ -37,6 +37,9 @@ pub struct Function { /// Encoding recipe and bits for the legal instructions. /// Illegal instructions have the `Encoding::default()` value. pub encodings: EntityMap, + + /// Location assigned to every value. + pub locations: EntityMap, } impl PrimaryEntityData for StackSlotData {} @@ -53,6 +56,7 @@ impl Function { dfg: DataFlowGraph::new(), layout: Layout::new(), encodings: EntityMap::new(), + locations: EntityMap::new(), } } diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index b8cda7711c..738c6336d5 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -13,6 +13,7 @@ pub mod function; mod funcname; mod extfunc; mod builder; +mod valueloc; pub use ir::funcname::FunctionName; pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension, ExtFuncData}; @@ -21,6 +22,7 @@ pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable, FuncRef, SigRef}; pub use ir::instructions::{Opcode, InstructionData, VariableArgs}; pub use ir::stackslot::StackSlotData; pub use ir::jumptable::JumpTableData; +pub use ir::valueloc::ValueLoc; pub use ir::dfg::{DataFlowGraph, ValueDef}; pub use ir::layout::{Layout, Cursor}; pub use ir::function::Function; diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/cretonne/src/ir/valueloc.rs new file mode 100644 index 0000000000..3526c58190 --- /dev/null +++ b/lib/cretonne/src/ir/valueloc.rs @@ -0,0 +1,24 @@ +//! Value locations. +//! +//! The register allocator assigns every SSA value to either a register or a stack slot. This +//! assignment is represented by a `ValueLoc` object. + +use isa::RegUnit; +use ir::StackSlot; + +/// Value location. +#[derive(Copy, Clone, Debug)] +pub enum ValueLoc { + /// This value has not been assigned to a location yet. + Unassigned, + /// Value is assigned to a register. + Reg(RegUnit), + /// Value is assigned to a stack slot. + Stack(StackSlot), +} + +impl Default for ValueLoc { + fn default() -> Self { + ValueLoc::Unassigned + } +} From 29276679b13b32863319b86757956fc31e816487 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Dec 2016 15:50:29 -0800 Subject: [PATCH 0459/3084] Add program points. Program points are used to represent a linear position in a function. Thus will be used for the live ranges of values. --- lib/cretonne/src/ir/mod.rs | 2 + lib/cretonne/src/ir/progpoint.rs | 97 ++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+) create mode 100644 lib/cretonne/src/ir/progpoint.rs diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 738c6336d5..7e05622645 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -14,6 +14,7 @@ mod funcname; mod extfunc; mod builder; mod valueloc; +mod progpoint; pub use ir::funcname::FunctionName; pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension, ExtFuncData}; @@ -27,3 +28,4 @@ pub use ir::dfg::{DataFlowGraph, ValueDef}; pub use ir::layout::{Layout, Cursor}; pub use ir::function::Function; pub use ir::builder::InstBuilder; +pub use ir::progpoint::{ProgramPoint, ProgramOrder}; diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs new file mode 100644 index 0000000000..c609a03d8f --- /dev/null +++ b/lib/cretonne/src/ir/progpoint.rs @@ -0,0 +1,97 @@ +//! Program points. + +use entity_map::EntityRef; +use ir::{Ebb, Inst}; +use std::fmt; +use std::u32; +use std::cmp; + +/// A `ProgramPoint` represents a position in a function where the live range of an SSA value can +/// begin or end. It can be either: +/// +/// 1. An instruction or +/// 2. An EBB header. +/// +/// This corresponds more or less to the lines in the textual representation of Cretonne IL. +#[derive(PartialEq, Eq, Clone, Copy)] +pub struct ProgramPoint(u32); + +impl From for ProgramPoint { + fn from(inst: Inst) -> ProgramPoint { + let idx = inst.index(); + assert!(idx < (u32::MAX / 2) as usize); + ProgramPoint((idx * 2) as u32) + } +} + +impl From for ProgramPoint { + fn from(ebb: Ebb) -> ProgramPoint { + let idx = ebb.index(); + assert!(idx < (u32::MAX / 2) as usize); + ProgramPoint((idx * 2 + 1) as u32) + } +} + +/// An expanded program point directly exposes the variants, but takes twice the space to +/// represent. +pub enum ExpandedProgramPoint { + // An instruction in the function. + Inst(Inst), + // An EBB header. + Ebb(Ebb), +} + +impl ProgramPoint { + /// Expand compact program point representation. + pub fn expand(self) -> ExpandedProgramPoint { + if self.0 & 1 == 0 { + ExpandedProgramPoint::Inst(Inst::new((self.0 / 2) as usize)) + } else { + ExpandedProgramPoint::Ebb(Ebb::new((self.0 / 2) as usize)) + } + } +} + +impl fmt::Display for ProgramPoint { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self.expand() { + ExpandedProgramPoint::Inst(x) => write!(f, "{}", x), + ExpandedProgramPoint::Ebb(x) => write!(f, "{}", x), + } + } +} + +impl fmt::Debug for ProgramPoint { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "ProgramPoint({})", self) + } +} + +/// Context for ordering program points. +/// +/// `ProgramPoint` objects don't carry enough information to be ordered independently, they need a +/// context providing the program order. +pub trait ProgramOrder { + /// Compare the program points `a` and `b` relative to this program order. Return `Less` if `a` + /// appears in the program before `b`. + fn cmp(&self, a: ProgramPoint, b: ProgramPoint) -> cmp::Ordering; +} + +#[cfg(test)] +mod tests { + use super::*; + use entity_map::EntityRef; + use ir::{Inst, Ebb}; + + #[test] + fn convert() { + let i5 = Inst::new(5); + let b3 = Ebb::new(3); + + let pp1: ProgramPoint = i5.into(); + let pp2: ProgramPoint = b3.into(); + + assert_eq!(pp1.to_string(), "inst5"); + assert_eq!(pp2.to_string(), "ebb3"); + } +} From a61a6e888e799b233ec53b2b02e2652d0907e119 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 21 Dec 2016 10:05:14 -0800 Subject: [PATCH 0460/3084] Upgrade to rustfmt 0.6.3 --- cranelift/test-all.sh | 2 +- cranelift/tests/cfg_traversal.rs | 20 +++++++++++------- lib/filecheck/src/pattern.rs | 36 ++++++++++++++++---------------- 3 files changed, 32 insertions(+), 26 deletions(-) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 0ad58aa114..f55464ce3b 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -30,7 +30,7 @@ function banner() { # rustfmt is installed. # # This version should always be bumped to the newest version available. -RUSTFMT_VERSION="0.6.2" +RUSTFMT_VERSION="0.6.3" if cargo install --list | grep -q "^rustfmt v$RUSTFMT_VERSION"; then banner "Rust formatting" diff --git a/cranelift/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs index b766f9d96c..2bdeab2347 100644 --- a/cranelift/tests/cfg_traversal.rs +++ b/cranelift/tests/cfg_traversal.rs @@ -9,8 +9,9 @@ use self::cretonne::entity_map::EntityMap; fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) { let func = &parse_functions(function_source).unwrap()[0]; let cfg = ControlFlowGraph::new(&func); - let ebbs = ebb_order.iter().map(|n| Ebb::with_number(*n).unwrap()) - .collect::>(); + let ebbs = ebb_order.iter() + .map(|n| Ebb::with_number(*n).unwrap()) + .collect::>(); let mut postorder_ebbs = cfg.postorder_ebbs(); let mut postorder_map = EntityMap::with_capacity(postorder_ebbs.len()); @@ -50,7 +51,8 @@ fn simple_traversal() { ebb5: trap } - ", vec![0, 2, 1, 3, 4, 5]); + ", + vec![0, 2, 1, 3, 4, 5]); } #[test] @@ -67,7 +69,8 @@ fn loops_one() { ebb3: return } - ", vec![0, 1, 2, 3]); + ", + vec![0, 1, 2, 3]); } #[test] @@ -91,7 +94,8 @@ fn loops_two() { brz v0, ebb4 return } - ", vec![0, 1, 2, 5, 4, 3]); + ", + vec![0, 1, 2, 5, 4, 3]); } #[test] @@ -120,7 +124,8 @@ fn loops_three() { ebb7: return } - ", vec![0, 1, 2, 5, 4, 3, 6, 7]); + ", + vec![0, 1, 2, 5, 4, 3, 6, 7]); } #[test] @@ -142,5 +147,6 @@ fn back_edge_one() { ebb4: trap } - ", vec![0, 1, 3, 2, 4]); + ", + vec![0, 1, 3, 2, 4]); } diff --git a/lib/filecheck/src/pattern.rs b/lib/filecheck/src/pattern.rs index 88c47011f9..030758d41e 100644 --- a/lib/filecheck/src/pattern.rs +++ b/lib/filecheck/src/pattern.rs @@ -165,9 +165,9 @@ impl Pattern { let refname = s[refname_begin..refname_end].to_string(); return if let Some(defidx) = def { Ok((Part::DefVar { - def: defidx, - var: refname, - }, + def: defidx, + var: refname, + }, refname_end + 1)) } else { Err(Error::Syntax(format!("expected variable name in $(=${})", refname))) @@ -462,37 +462,37 @@ mod tests { assert_eq!(pat.parse_part("$(foo=$bar)").unwrap(), (Part::DefVar { - def: 0, - var: "bar".to_string(), - }, + def: 0, + var: "bar".to_string(), + }, 11)); assert_eq!(pat.parse_part("$(foo=$bar)").unwrap_err().to_string(), "duplicate definition of $foo in same pattern"); assert_eq!(pat.parse_part("$(fxo=$bar)x").unwrap(), (Part::DefVar { - def: 1, - var: "bar".to_string(), - }, + def: 1, + var: "bar".to_string(), + }, 11)); assert_eq!(pat.parse_part("$(fo2=[a-z])").unwrap(), (Part::DefLit { - def: 2, - regex: "(?P[a-z])".to_string(), - }, + def: 2, + regex: "(?P[a-z])".to_string(), + }, 12)); assert_eq!(pat.parse_part("$(fo3=[a-)])").unwrap(), (Part::DefLit { - def: 3, - regex: "(?P[a-)])".to_string(), - }, + def: 3, + regex: "(?P[a-)])".to_string(), + }, 12)); assert_eq!(pat.parse_part("$(fo4=)").unwrap(), (Part::DefLit { - def: 4, - regex: "(?P)".to_string(), - }, + def: 4, + regex: "(?P)".to_string(), + }, 7)); assert_eq!(pat.parse_part("$(=.*)").unwrap(), From 7b04f5bb31fc38c831475625335d460db8962b23 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 28 Dec 2016 17:05:00 -0800 Subject: [PATCH 0461/3084] Implement ProgramOrder for Layout. Assign sequence numbers to both instructions and EBB headers such that their relative position can be determined in constant time simply by comparing sequence numbers. Implement the sequence numbers with a scheme similar to the line numbers used in BASIC programs. Start out with 10, 20, 30, ... and pick numbers in the gaps as long as possible. Renumber locally when needed. --- lib/cretonne/src/ir/layout.rs | 249 ++++++++++++++++++++++++++++++++-- 1 file changed, 237 insertions(+), 12 deletions(-) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index bdcf4cdd52..16f5aa5483 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -3,9 +3,11 @@ //! The order of extended basic blocks in a function and the order of instructions in an EBB is //! determined by the `Layout` data structure defined in this module. +use std::cmp; use std::iter::{Iterator, IntoIterator}; use entity_map::{EntityMap, EntityRef}; use ir::entities::{Ebb, NO_EBB, Inst, NO_INST}; +use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not /// contain definitions of instructions or EBBs, but depends on `Inst` and `Ebb` entity references @@ -49,6 +51,206 @@ impl Layout { } } +// Sequence numbers. +// +// All instructions and EBBs are given a sequence number that can be used to quickly determine +// their relative position in the layout. The sequence numbers are not contiguous, but are assigned +// like line numbers in BASIC: 10, 20, 30, ... +// +// The EBB sequence numbers are strictly increasing, and so are the instruction sequence numbers +// within an EBB. The instruction sequence numbers are all between the sequence number of their +// containing EBB and the following EBB. +// +// The result is that sequence numbers work like BASIC line numbers for the textual representation +// of the IL. +type SequenceNumber = u32; + +// Initial stride assigned to new sequence numbers. +const MAJOR_STRIDE: SequenceNumber = 10; + +// Secondary stride used when renumbering locally. +const MINOR_STRIDE: SequenceNumber = 2; + +// Compute the midpoint between `a` and `b`. +// Return `None` if the midpoint would be equal to either. +fn midpoint(a: SequenceNumber, b: SequenceNumber) -> Option { + assert!(a < b); + // Avoid integer overflow. + let m = a + (b - a) / 2; + if m > a { Some(m) } else { None } +} + +#[test] +fn test_midpoint() { + assert_eq!(midpoint(0, 1), None); + assert_eq!(midpoint(0, 2), Some(1)); + assert_eq!(midpoint(0, 3), Some(1)); + assert_eq!(midpoint(0, 4), Some(2)); + assert_eq!(midpoint(1, 4), Some(2)); + assert_eq!(midpoint(2, 4), Some(3)); + assert_eq!(midpoint(3, 4), None); + assert_eq!(midpoint(3, 4), None); +} + +impl ProgramOrder for Layout { + fn cmp(&self, a: ProgramPoint, b: ProgramPoint) -> cmp::Ordering { + let a_seq = self.pp_seq(a); + let b_seq = self.pp_seq(b); + a_seq.cmp(&b_seq) + } +} + +// Private methods for dealing with sequence numbers. +impl Layout { + /// Get the sequence number of a program point that must correspond to an entity in the layout. + fn pp_seq(&self, pp: ProgramPoint) -> SequenceNumber { + match pp.expand() { + ExpandedProgramPoint::Ebb(ebb) => self.ebbs[ebb].seq, + ExpandedProgramPoint::Inst(inst) => self.insts[inst].seq, + } + } + + /// Get the last sequence number in `ebb`. + fn last_ebb_seq(&self, ebb: Ebb) -> SequenceNumber { + // Get the seq of the last instruction if it exists, otherwise use the EBB header seq. + self.ebbs[ebb] + .last_inst + .wrap() + .map(|inst| self.insts[inst].seq) + .unwrap_or(self.ebbs[ebb].seq) + } + + /// Assign a valid sequence number to `ebb` such that the numbers are still monotonic. This may + /// require renumbering. + fn assign_ebb_seq(&mut self, ebb: Ebb) { + assert!(self.is_ebb_inserted(ebb)); + + // Get the sequence number immediately before `ebb`, or 0. + let prev_seq = + self.ebbs[ebb].prev.wrap().map(|prev_ebb| self.last_ebb_seq(prev_ebb)).unwrap_or(0); + + // Get the sequence number immediately following `ebb`. + let next_seq = if let Some(inst) = self.ebbs[ebb].first_inst.wrap() { + self.insts[inst].seq + } else if let Some(next_ebb) = self.ebbs[ebb].next.wrap() { + self.ebbs[next_ebb].seq + } else { + // There is nothing after `ebb`. We can just use a major stride. + self.ebbs[ebb].seq = prev_seq + MAJOR_STRIDE; + return; + }; + + // Check if there is room between these sequence numbers. + if let Some(seq) = midpoint(prev_seq, next_seq) { + self.ebbs[ebb].seq = seq; + } else { + // No available integers between `prev_seq` and `next_seq`. We have to renumber. + self.renumber_from_ebb(ebb, prev_seq + MINOR_STRIDE); + } + } + + /// Assign a valid sequence number to `inst` such that the numbers are still monotonic. This may + /// require renumbering. + fn assign_inst_seq(&mut self, inst: Inst) { + let ebb = self.inst_ebb(inst).expect("inst must be inserted before assigning an seq"); + + // Get the sequence number immediately before `inst`. + let prev_seq = match self.insts[inst].prev.wrap() { + Some(prev_inst) => self.insts[prev_inst].seq, + None => self.ebbs[ebb].seq, + }; + + // Get the sequence number immediately following `inst`. + let next_seq = if let Some(next_inst) = self.insts[inst].next.wrap() { + self.insts[next_inst].seq + } else if let Some(next_ebb) = self.ebbs[ebb].next.wrap() { + self.ebbs[next_ebb].seq + } else { + // There is nothing after `inst`. We can just use a major stride. + self.insts[inst].seq = prev_seq + MAJOR_STRIDE; + return; + }; + + // Check if there is room between these sequence numbers. + if let Some(seq) = midpoint(prev_seq, next_seq) { + self.insts[inst].seq = seq; + } else { + // No available integers between `prev_seq` and `next_seq`. We have to renumber. + self.renumber_from_inst(inst, prev_seq + MINOR_STRIDE); + } + } + + /// Renumber instructions starting from `inst` until the end of the EBB or until numbers catch + /// up. + /// + /// Return `None` if renumbering has caught up and the sequence is monotonic again. Otherwise + /// return the last used sequence number. + fn renumber_insts(&mut self, inst: Inst, seq: SequenceNumber) -> Option { + let mut inst = inst; + let mut seq = seq; + + loop { + self.insts[inst].seq = seq; + + // Next instruction. + inst = match self.insts[inst].next.wrap() { + None => return Some(seq), + Some(next) => next, + }; + + if seq < self.insts[inst].seq { + // Sequence caught up. + return None; + } + + seq += MINOR_STRIDE; + } + } + + /// Renumber starting from `ebb` to `seq` and continuing until the sequence numbers are + /// monotonic again. + fn renumber_from_ebb(&mut self, ebb: Ebb, first_seq: SequenceNumber) { + let mut ebb = ebb; + let mut seq = first_seq; + + loop { + self.ebbs[ebb].seq = seq; + + // Renumber instructions in `ebb`. Stop when the numbers catch up. + if let Some(inst) = self.ebbs[ebb].first_inst.wrap() { + seq = match self.renumber_insts(inst, seq + MINOR_STRIDE) { + Some(s) => s, + None => return, + } + } + + // Advance to the next EBB. + ebb = match self.ebbs[ebb].next.wrap() { + Some(next) => next, + None => return, + }; + + // Stop renumbering once the numbers catch up. + if seq < self.ebbs[ebb].seq { + return; + } + + seq += MINOR_STRIDE; + } + } + + /// Renumber starting from `inst` to `seq` and continuing until the sequence numbers are + /// monotonic again. + fn renumber_from_inst(&mut self, inst: Inst, first_seq: SequenceNumber) { + if let Some(seq) = self.renumber_insts(inst, first_seq) { + // Renumbering spills over into next EBB. + if let Some(next_ebb) = self.ebbs[self.inst_ebb(inst).unwrap()].next.wrap() { + self.renumber_from_ebb(next_ebb, seq + MINOR_STRIDE); + } + } + } +} + /// Methods for laying out EBBs. /// /// An unknown EBB starts out as *not inserted* in the EBB layout. The layout is a linear order of @@ -80,6 +282,7 @@ impl Layout { self.first_ebb = Some(ebb); } self.last_ebb = Some(ebb); + self.assign_ebb_seq(ebb); } /// Insert `ebb` in the layout before the existing EBB `before`. @@ -100,6 +303,7 @@ impl Layout { } else { self.ebbs[after].next = ebb; } + self.assign_ebb_seq(ebb); } /// Insert `ebb` in the layout *after* the existing EBB `after`. @@ -120,6 +324,7 @@ impl Layout { } else { self.ebbs[before].prev = ebb; } + self.assign_ebb_seq(ebb); } /// Return an iterator over all EBBs in layout order. @@ -143,6 +348,7 @@ struct EbbNode { next: Ebb, first_inst: Inst, last_inst: Inst, + seq: SequenceNumber, } /// Iterate over EBBs in layout order. See `Layout::ebbs()`. @@ -194,19 +400,22 @@ impl Layout { assert_eq!(self.inst_ebb(inst), None); assert!(self.is_ebb_inserted(ebb), "Cannot append instructions to EBB not in layout"); - let ebb_node = &mut self.ebbs[ebb]; { - let inst_node = self.insts.ensure(inst); - inst_node.ebb = ebb; - inst_node.prev = ebb_node.last_inst; - assert_eq!(inst_node.next, NO_INST); + let ebb_node = &mut self.ebbs[ebb]; + { + let inst_node = self.insts.ensure(inst); + inst_node.ebb = ebb; + inst_node.prev = ebb_node.last_inst; + assert_eq!(inst_node.next, NO_INST); + } + if ebb_node.first_inst == NO_INST { + ebb_node.first_inst = inst; + } else { + self.insts[ebb_node.last_inst].next = inst; + } + ebb_node.last_inst = inst; } - if ebb_node.first_inst == NO_INST { - ebb_node.first_inst = inst; - } else { - self.insts[ebb_node.last_inst].next = inst; - } - ebb_node.last_inst = inst; + self.assign_inst_seq(inst); } /// Fetch an ebb's last instruction. @@ -232,6 +441,7 @@ impl Layout { } else { self.insts[after].next = inst; } + self.assign_inst_seq(inst); } /// Iterate over the instructions in `ebb` in layout order. @@ -305,6 +515,8 @@ impl Layout { self.insts[i].ebb = new_ebb; i = self.insts[i].next; } + + self.assign_ebb_seq(new_ebb); } } @@ -313,6 +525,7 @@ struct InstNode { ebb: Ebb, prev: Inst, next: Inst, + seq: SequenceNumber, } /// Iterate over instructions in an EBB in layout order. See `Layout::ebb_insts()`. @@ -665,21 +878,28 @@ impl<'f> Cursor<'f> { mod tests { use super::{Layout, Cursor, CursorPosition}; use entity_map::EntityRef; - use ir::{Ebb, Inst}; + use ir::{Ebb, Inst, ProgramOrder}; + use std::cmp::Ordering; fn verify(layout: &mut Layout, ebbs: &[(Ebb, &[Inst])]) { // Check that EBBs are inserted and instructions belong the right places. // Check forward linkage with iterators. + // Check that layout sequence numbers are strictly monotonic. { + let mut seq = 0; let mut ebb_iter = layout.ebbs(); for &(ebb, insts) in ebbs { assert!(layout.is_ebb_inserted(ebb)); assert_eq!(ebb_iter.next(), Some(ebb)); + assert!(layout.ebbs[ebb].seq > seq); + seq = layout.ebbs[ebb].seq; let mut inst_iter = layout.ebb_insts(ebb); for &inst in insts { assert_eq!(layout.inst_ebb(inst), Some(ebb)); assert_eq!(inst_iter.next(), Some(inst)); + assert!(layout.insts[inst].seq > seq); + seq = layout.insts[inst].seq; } assert_eq!(inst_iter.next(), None); } @@ -1018,5 +1238,10 @@ mod tests { assert_eq!(cur.prev_inst(), None); assert_eq!(cur.prev_ebb(), None); } + + // Check ProgramOrder. + assert_eq!(layout.cmp(e2.into(), e2.into()), Ordering::Equal); + assert_eq!(layout.cmp(e2.into(), i2.into()), Ordering::Less); + assert_eq!(layout.cmp(i3.into(), i2.into()), Ordering::Greater); } } From ae28ef90ef672dff53d986fa944d78642e59d0cc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 5 Jan 2017 14:03:09 -0800 Subject: [PATCH 0462/3084] Encourage better optimization of ProgramOrder::cmp. The ProgramOrder::cmp() comparison is often used where one or both arguments are statically known to be an Inst or Ebb. Give the compiler a better chance to discover this via inlining and other optimizations. - Make cmp() generic with Into bounds. - Implement the natural From traits for ExpandedProgramPoint. - Make Layout::pp_seq() generic with the same bound. Now, with inlining and constant folding, passing an Inst argument to PO::cmp() will result in a call to a monomorphized Layout::seq::() which can avoid the dynamic match to select a table for looking up the sequence number. The result is that comparing two program points of statically known type results in two direct table lookups and a sequence number comparison. This all uses ExpandedProgramPoint because it is more likely to be transparent to the constant folder than the bit-packed ProgramPoint type. --- lib/cretonne/src/ir/layout.rs | 22 ++++++++++-------- lib/cretonne/src/ir/progpoint.rs | 39 ++++++++++++++++++++++++-------- 2 files changed, 42 insertions(+), 19 deletions(-) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 16f5aa5483..cecc8645ef 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -7,7 +7,7 @@ use std::cmp; use std::iter::{Iterator, IntoIterator}; use entity_map::{EntityMap, EntityRef}; use ir::entities::{Ebb, NO_EBB, Inst, NO_INST}; -use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; +use ir::progpoint::{ProgramOrder, ExpandedProgramPoint}; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not /// contain definitions of instructions or EBBs, but depends on `Inst` and `Ebb` entity references @@ -93,9 +93,12 @@ fn test_midpoint() { } impl ProgramOrder for Layout { - fn cmp(&self, a: ProgramPoint, b: ProgramPoint) -> cmp::Ordering { - let a_seq = self.pp_seq(a); - let b_seq = self.pp_seq(b); + fn cmp(&self, a: A, b: B) -> cmp::Ordering + where A: Into, + B: Into + { + let a_seq = self.seq(a); + let b_seq = self.seq(b); a_seq.cmp(&b_seq) } } @@ -103,8 +106,9 @@ impl ProgramOrder for Layout { // Private methods for dealing with sequence numbers. impl Layout { /// Get the sequence number of a program point that must correspond to an entity in the layout. - fn pp_seq(&self, pp: ProgramPoint) -> SequenceNumber { - match pp.expand() { + fn seq>(&self, pp: PP) -> SequenceNumber { + // When `PP = Inst` or `PP = Ebb`, we expect this dynamic type check to be optimized out. + match pp.into() { ExpandedProgramPoint::Ebb(ebb) => self.ebbs[ebb].seq, ExpandedProgramPoint::Inst(inst) => self.insts[inst].seq, } @@ -1240,8 +1244,8 @@ mod tests { } // Check ProgramOrder. - assert_eq!(layout.cmp(e2.into(), e2.into()), Ordering::Equal); - assert_eq!(layout.cmp(e2.into(), i2.into()), Ordering::Less); - assert_eq!(layout.cmp(i3.into(), i2.into()), Ordering::Greater); + assert_eq!(layout.cmp(e2, e2), Ordering::Equal); + assert_eq!(layout.cmp(e2, i2), Ordering::Less); + assert_eq!(layout.cmp(i3, i2), Ordering::Greater); } } diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index c609a03d8f..ec2aa562cf 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -34,6 +34,7 @@ impl From for ProgramPoint { /// An expanded program point directly exposes the variants, but takes twice the space to /// represent. +#[derive(PartialEq, Eq, Clone, Copy)] pub enum ExpandedProgramPoint { // An instruction in the function. Inst(Inst), @@ -41,20 +42,31 @@ pub enum ExpandedProgramPoint { Ebb(Ebb), } -impl ProgramPoint { - /// Expand compact program point representation. - pub fn expand(self) -> ExpandedProgramPoint { - if self.0 & 1 == 0 { - ExpandedProgramPoint::Inst(Inst::new((self.0 / 2) as usize)) +impl From for ExpandedProgramPoint { + fn from(inst: Inst) -> ExpandedProgramPoint { + ExpandedProgramPoint::Inst(inst) + } +} + +impl From for ExpandedProgramPoint { + fn from(ebb: Ebb) -> ExpandedProgramPoint { + ExpandedProgramPoint::Ebb(ebb) + } +} + +impl From for ExpandedProgramPoint { + fn from(pp: ProgramPoint) -> ExpandedProgramPoint { + if pp.0 & 1 == 0 { + ExpandedProgramPoint::Inst(Inst::new((pp.0 / 2) as usize)) } else { - ExpandedProgramPoint::Ebb(Ebb::new((self.0 / 2) as usize)) + ExpandedProgramPoint::Ebb(Ebb::new((pp.0 / 2) as usize)) } } } impl fmt::Display for ProgramPoint { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match self.expand() { + match (*self).into() { ExpandedProgramPoint::Inst(x) => write!(f, "{}", x), ExpandedProgramPoint::Ebb(x) => write!(f, "{}", x), } @@ -72,9 +84,16 @@ impl fmt::Debug for ProgramPoint { /// `ProgramPoint` objects don't carry enough information to be ordered independently, they need a /// context providing the program order. pub trait ProgramOrder { - /// Compare the program points `a` and `b` relative to this program order. Return `Less` if `a` - /// appears in the program before `b`. - fn cmp(&self, a: ProgramPoint, b: ProgramPoint) -> cmp::Ordering; + /// Compare the program points `a` and `b` relative to this program order. + /// + /// Return `Less` if `a` appears in the program before `b`. + /// + /// This is declared as a generic such that it can be called with `Inst` and `Ebb` arguments + /// directly. Depending on the implementation, there is a good chance performance will be + /// improved for those cases where the type of either argument is known statically. + fn cmp(&self, a: A, b: B) -> cmp::Ordering + where A: Into, + B: Into; } #[cfg(test)] From b6c2d4588f591903c2639cf22db27e4cc99e4d9f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 5 Jan 2017 10:34:19 -0800 Subject: [PATCH 0463/3084] Add a LiveRange data structure. We will track live ranges separately for each SSA value, rather than per virtual register like LLVM does. This is the basis for a register allocator, so place it in a new regalloc module. --- lib/cretonne/src/ir/mod.rs | 2 +- lib/cretonne/src/ir/progpoint.rs | 4 +- lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/regalloc/liverange.rs | 510 +++++++++++++++++++++++++ lib/cretonne/src/regalloc/mod.rs | 5 + 5 files changed, 519 insertions(+), 3 deletions(-) create mode 100644 lib/cretonne/src/regalloc/liverange.rs create mode 100644 lib/cretonne/src/regalloc/mod.rs diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 7e05622645..f4b2a337b4 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -28,4 +28,4 @@ pub use ir::dfg::{DataFlowGraph, ValueDef}; pub use ir::layout::{Layout, Cursor}; pub use ir::function::Function; pub use ir::builder::InstBuilder; -pub use ir::progpoint::{ProgramPoint, ProgramOrder}; +pub use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index ec2aa562cf..83c565fca7 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -36,9 +36,9 @@ impl From for ProgramPoint { /// represent. #[derive(PartialEq, Eq, Clone, Copy)] pub enum ExpandedProgramPoint { - // An instruction in the function. + /// An instruction in the function. Inst(Inst), - // An EBB header. + /// An EBB header. Ebb(Ebb), } diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 51a8d9b367..6445d7866b 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -16,6 +16,7 @@ pub mod dominator_tree; pub mod entity_map; pub mod settings; pub mod verifier; +pub mod regalloc; mod write; mod constant_hash; diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs new file mode 100644 index 0000000000..b26dc8e8b1 --- /dev/null +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -0,0 +1,510 @@ +//! Data structure representing the live range of an SSA value. +//! +//! Live ranges are tracked per SSA value, not per variable or virtual register. The live range of +//! an SSA value begins where it is defined and extends to all program points where the value is +//! still needed. +//! +//! # Local Live Ranges +//! +//! Inside a single extended basic block, the live range of a value is always an interval between +//! two program points (if the value is live in the EBB at all). The starting point is either: +//! +//! 1. The instruction that defines the value, or +//! 2. The EBB header, because the value is an argument to the EBB, or +//! 3. The EBB header, because the value is defined in another EBB and live-in to this one. +//! +//! The ending point of the local live range is the last of the following program points in the +//! EBB: +//! +//! 1. The last use in the EBB, where a *use* is an instruction that has the value as an argument. +//! 2. The last branch or jump instruction in the EBB that can reach a use. +//! 3. If the value has no uses anywhere (a *dead value*), the program point that defines it. +//! +//! Note that 2. includes loop back-edges to the same EBB. In general, if a value is defined +//! outside a loop and used inside the loop, it will be live in the entire loop. +//! +//! # Global Live Ranges +//! +//! Values that appear in more than one EBB have a *global live range* which can be seen as the +//! disjoint union of the per-EBB local intervals for all of the EBBs where the value is live. +//! Together with a `ProgramOrder` which provides a linear ordering of the EBBs, the global live +//! range becomes a linear sequence of disjoint intervals, at most one per EBB. +//! +//! In the special case of a dead value, the global live range is a single interval where the start +//! and end points are the same. The global live range of a value is never completely empty. +//! +//! # Register interference +//! +//! The register allocator uses live ranges to determine if values *interfere*, which means that +//! they can't be stored in the same register. Two live ranges interfere if and only if any of +//! their intervals overlap. +//! +//! If one live range ends at an instruction that defines another live range, those two live ranges +//! are not considered to interfere. This is because most ISAs allow instructions to reuse an input +//! register for an output value. If Cretonne gets support for inline assembly, we will need to +//! handle *early clobbers* which are output registers that are not allowed to alias any input +//! registers. +//! +//! If i1 < i2 < i3 are program points, we have: +//! +//! - i1-i2 and i1-i3 interfere because the intervals overlap. +//! - i1-i2 and i2-i3 don't interfere. +//! - i1-i3 and i2-i2 do interfere because the dead def would clobber the register. +//! - i1-i2 and i2-i2 don't interfere. +//! - i2-i3 and i2-i2 do interfere. +//! +//! Because of this behavior around interval end points, live range interference is not completely +//! equivalent to mathematical intersection of open or half-open intervals. +//! +//! # Implementation notes +//! +//! A few notes about the implementation of this data structure. This should not concern someone +//! only looking to use the public interface. +//! +//! ## EBB ordering +//! +//! The relative order of EBBs is used to maintain a sorted list of live-in intervals and to +//! coalesce adjacent live-in intervals when the prior interval covers the whole EBB. This doesn't +//! depend on any property of the program order, so alternative orderings are possible: +//! +//! 1. The EBB layout order. This is what we currently use. +//! 2. A topological order of the dominator tree. All the live-in intervals would come after the +//! def interval. +//! 3. A numerical order by EBB number. Performant because it doesn't need to indirect through the +//! `ProgramOrder` for comparisons. +//! +//! These orderings will cause small differences in coalescing opportinities, but all of them would +//! do a decent job of compressing a long live range. The numerical order might be preferable +//! beacuse: +//! +//! - It has better performance because EBB numbers can be compared directly without any table +//! lookups. +//! - If EBB numbers are not reused, it is safe to allocate new EBBs without getting spurious +//! live-in intervals from any coalesced representations that happen to cross a new EBB. +//! +//! For comparing instructions, the layout order is always what we want. +//! +//! ## Alternative representation +//! +//! Since a local live-in interval always begins at its EBB header, it is uniquely described by its +//! end point instruction alone. We can use the layout to look up the EBB containing the end point. +//! This means that a sorted `Vec` would be enough to represent the set of live-in intervals. +//! +//! Coalescing is an important compression technique because some live ranges can span thousands of +//! EBBs. We can represent that by switching to a sorted `Vec` representation where +//! an `[Ebb, Inst]` pair represents a coalesced range, while an `Inst` entry without a preceeding +//! `Ebb` entry represents a single live-in interval. +//! +//! This representation is more compact for a live range with many uncoalesced live-in intervals. +//! It is more complicated to work with, though, so it is probably not worth it. The performance +//! benefits of switching to a numerical EBB order only appears if the binary search is doing +//! EBB-EBB comparisons. +//! +//! ## B-tree representation +//! +//! A `BTreeMap` could also be used for the live-in intervals. It looks like the +//! standard library B-tree doesn't provide the necessary interface for an efficient implementation +//! of coalescing, so we would need to roll our own. +//! + +use std::cmp::Ordering; +use ir::{Inst, Ebb, ProgramPoint, ProgramOrder}; + +/// Global live range of a single SSA value. +/// +/// As [explained in the module documentation](index.html#local-live-ranges), the live range of an +/// SSA value is the disjoint union of a set of intervals, each local to a single EBB, and with at +/// most one interval per EBB. We further distinguish between: +/// +/// 1. The *def interval* is the local interval in the EBB where the value is defined, and +/// 2. The *live-in intervals* are the local intervals in the remaining EBBs. +/// +/// A live-in interval always begins at the EBB header, while the def interval can begin at the +/// defining instruction, or at the EBB header for an EBB argument value. +/// +/// All values have a def interval, but a large proportion of values don't have any live-in +/// intervals. These are called *local live ranges*. +/// +/// # Program order requirements +/// +/// The internal representation of a `LiveRange` depends on a consistent `ProgramOrder` both for +/// ordering instructions inside an EBB *and* for ordering EBBs. The methods that depend on the +/// ordering take an explicit `ProgramOrder` object, and it is the caller's responsibility to +/// ensure that the provided ordering is consistent between calls. +/// +/// In particular, changing the order of EBBs or inserting new EBBs will invalidate live ranges. +/// +/// Inserting new instructions in the layout is safe, but removing instructions is not. Besides the +/// instructions using or defining their value, `LiveRange` structs can contain references to +/// branch and jump instructions. +pub struct LiveRange { + /// The instruction or EBB header where this value is defined. + def_begin: ProgramPoint, + + /// The end point of the def interval. This must always belong to the same EBB as `def_begin`. + /// + /// We always have `def_begin <= def_end` with equality implying a dead def live range with no + /// uses. + def_end: ProgramPoint, + + /// Additional live-in intervals sorted in program order. + /// + /// This vector is empty for most values which are only used in one EBB. + /// + /// Invariants: + /// + /// - Sorted, disjoint: For all `i < j`: `liveins[i].end < liveins[j].begin`. + /// - Not overlapping defining EBB: For all `i`: + /// `liveins[i].end < def_begin` or `liveins[i].begin > def_end`. + liveins: Vec, +} + +/// An additional contiguous interval of a global live range. +/// +/// This represents a live-in interval for a single EBB, or a coalesced set of live-in intervals +/// for contiguous EBBs where all but the last live-in inteval covers the whole EBB. +/// +struct Interval { + /// Interval starting point. + /// + /// Since this interval does not represent the def of the value, it must begin at an EBB header + /// where the value is live-in. + begin: Ebb, + + /// Interval end point. + /// + /// The only interval end point that can be an EBB header is `def_end` above in a dead def + /// live range for an unused EBB argument. All other intervals must end at an instruction -- + /// either the last use in the EBB or the last branch/jump that can reach a use. + /// + /// When this represents multiple contiguous live-in intervals, this is the end point of the + /// last interval. The other intervals end at the terminator instructions of their respective + /// EBB. + end: Inst, +} + +impl Interval { + /// Extend the interval end point to reach `to`, but only if it would make the interval longer. + fn extend_to(&mut self, to: Inst, order: &PO) { + if order.cmp(to, self.end) == Ordering::Greater { + self.end = to; + } + } +} + +impl LiveRange { + /// Create a new live range defined at `def`. + /// + /// The live range will be created as dead, but it can be extended with `extend_in_ebb()`. + pub fn new>(def: PP) -> LiveRange { + let def = def.into(); + LiveRange { + def_begin: def, + def_end: def, + liveins: Vec::new(), + } + } + + /// Find the live-in interval containing `ebb`, if any. + /// + /// Return `Ok(n)` if `liveins[n]` already contains `ebb`. + /// Otherwise, return `Err(n)` with the index where such an interval should be inserted. + fn find_ebb_interval(&self, ebb: Ebb, order: &PO) -> Result { + self.liveins + .binary_search_by(|intv| order.cmp(intv.begin, ebb)) + .or_else(|n| { + // The interval at `n-1` may cover `ebb`. + if n > 0 && order.cmp(self.liveins[n - 1].end, ebb) == Ordering::Greater { + Ok(n - 1) + } else { + Err(n) + } + }) + } + + /// Extend the local interval for `ebb` so it reaches `to` which must belong to `ebb`. + /// Create a live-in interval if necessary. + /// + /// If the live range already has a local interval in `ebb`, extend its end point so it + /// includes `to`, and return false. + /// + /// If the live range did not previously have a local interval in `ebb`, add one so the value + /// is live-in to `ebb`, extending to `to`. Return true. + /// + /// The return value can be used to detect if we just learned that the value is live-in to + /// `ebb`. This can trigger recursive extensions in `ebb`'s CFG precedessor blocks. + pub fn extend_in_ebb(&mut self, ebb: Ebb, to: Inst, order: &PO) -> bool { + // First check if we're extending the def interval. + // + // We're assuming here that `to` never precedes `def_begin` in the same EBB, but we can't + // check it without a method for getting `to`'s EBB. + if order.cmp(ebb, self.def_end) != Ordering::Greater && + order.cmp(to, self.def_begin) != Ordering::Less { + let to_pp = to.into(); + assert_ne!(to_pp, + self.def_begin, + "Can't use value in the defining instruction."); + if order.cmp(to, self.def_end) == Ordering::Greater { + self.def_end = to_pp; + } + return false; + } + + // Now check if we're extending any of the existing live-in intervals. + match self.find_ebb_interval(ebb, order) { + Ok(n) => { + // We have an interval that contains `ebb`, so we can simply extend it. + self.liveins[n].extend_to(to, order); + // TODO: Check if this interval can be coalesced with the `n+1` one. + false + } + Err(n) => { + // Insert a new live-in interval at `n`. + // TODO: Check if this interval can be coalesced with the `n-1` or `n+1` one (or + // both). + self.liveins.insert(n, + Interval { + begin: ebb, + end: to, + }); + true + } + } + } + + /// Is this the live range of a dead value? + /// + /// A dead value has no uses, and its live range ends at the same program point where it is + /// defined. + pub fn is_dead(&self) -> bool { + self.def_begin == self.def_end + } + + /// Is this a local live range? + /// + /// A local live range is only used in the same EBB where it was defined. It is allowed to span + /// multiple basic blocks within that EBB. + pub fn is_local(&self) -> bool { + self.liveins.is_empty() + } + + /// Get the program point where this live range is defined. + /// + /// This will be an EBB header when the value is an EBB argument, otherwise it is the defining + /// instruction. + pub fn def(&self) -> ProgramPoint { + self.def_begin + } + + /// Get the local end-point of this live range in the EBB where it is defined. + /// + /// This can be the EBB header itself in the case of a dead EBB argument. + /// Otherwise, it will be the last local use or branch/jump that can reach a use. + pub fn def_local_end(&self) -> ProgramPoint { + self.def_end + } + + /// Get the local end-point of this live range in an EBB where it is live-in. + /// + /// If this live range is not live-in to `ebb`, return `None`. Otherwise, return the end-point + /// of this live range's local interval in `ebb`. + /// + /// If the live range is live through all of `ebb`, the terminator of `ebb` is a correct + /// answer, but it is also possible that an even later program point is returned. So don't + /// depend on the returned `Inst` to belong to `ebb`. + pub fn livein_local_end(&self, ebb: Ebb, order: &PO) -> Option { + self.find_ebb_interval(ebb, order).ok().map(|n| self.liveins[n].end) + } +} + +#[cfg(test)] +mod tests { + use super::LiveRange; + use ir::{Inst, Ebb}; + use entity_map::EntityRef; + use ir::{ProgramOrder, ExpandedProgramPoint}; + use std::cmp::Ordering; + + // Dummy program order which simply compares indexes. + // It is assumed that EBBs have indexes that are multiples of 10, and instructions have indexes + // in between. + struct ProgOrder {} + + impl ProgramOrder for ProgOrder { + fn cmp(&self, a: A, b: B) -> Ordering + where A: Into, + B: Into + { + fn idx(pp: ExpandedProgramPoint) -> usize { + match pp { + ExpandedProgramPoint::Inst(i) => i.index(), + ExpandedProgramPoint::Ebb(e) => e.index(), + } + } + + let ia = idx(a.into()); + let ib = idx(b.into()); + ia.cmp(&ib) + } + } + + impl ProgOrder { + // Get the EBB corresponding to `inst`. + fn inst_ebb(&self, inst: Inst) -> Ebb { + let i = inst.index(); + Ebb::new(i - i % 10) + } + + // Get the EBB of a program point. + fn pp_ebb>(&self, pp: PP) -> Ebb { + match pp.into() { + ExpandedProgramPoint::Inst(i) => self.inst_ebb(i), + ExpandedProgramPoint::Ebb(e) => e, + } + } + + // Validate the live range invariants. + fn validate(&self, lr: &LiveRange) { + // The def interval must cover a single EBB. + let def_ebb = self.pp_ebb(lr.def_begin); + assert_eq!(def_ebb, self.pp_ebb(lr.def_end)); + + // Check that the def interval isn't backwards. + match self.cmp(lr.def_begin, lr.def_end) { + Ordering::Equal => assert!(lr.liveins.is_empty()), + Ordering::Greater => { + panic!("Backwards def interval: {}-{}", lr.def_begin, lr.def_end) + } + Ordering::Less => {} + } + + // Check the live-in intervals. + let mut prev_end = None; + for li in &lr.liveins { + assert_eq!(self.cmp(li.begin, li.end), Ordering::Less); + if let Some(e) = prev_end { + assert_eq!(self.cmp(e, li.begin), Ordering::Less); + } + + assert!(self.cmp(lr.def_end, li.begin) == Ordering::Less || + self.cmp(lr.def_begin, li.end) == Ordering::Greater, + "Interval can't overlap the def EBB"); + + // Save for next round. + prev_end = Some(li.end); + } + + } + } + + // Singleton ProgramOrder for tests below. + const PO: &'static ProgOrder = &ProgOrder {}; + + #[test] + fn dead_def_range() { + let i1 = Inst::new(1); + let e2 = Ebb::new(2); + let lr = LiveRange::new(i1); + assert!(lr.is_dead()); + assert!(lr.is_local()); + assert_eq!(lr.def(), i1.into()); + assert_eq!(lr.def_local_end(), i1.into()); + assert_eq!(lr.livein_local_end(e2, PO), None); + PO.validate(&lr); + } + + #[test] + fn dead_arg_range() { + let e2 = Ebb::new(2); + let lr = LiveRange::new(e2); + assert!(lr.is_dead()); + assert!(lr.is_local()); + assert_eq!(lr.def(), e2.into()); + assert_eq!(lr.def_local_end(), e2.into()); + // The def interval of an EBB arg does not count as live-in. + assert_eq!(lr.livein_local_end(e2, PO), None); + PO.validate(&lr); + } + + #[test] + fn local_def() { + let e10 = Ebb::new(10); + let i11 = Inst::new(11); + let i12 = Inst::new(12); + let i13 = Inst::new(13); + let mut lr = LiveRange::new(i11); + + assert_eq!(lr.extend_in_ebb(e10, i13, PO), false); + PO.validate(&lr); + assert!(!lr.is_dead()); + assert!(lr.is_local()); + assert_eq!(lr.def(), i11.into()); + assert_eq!(lr.def_local_end(), i13.into()); + + // Extending to an already covered inst should not change anything. + assert_eq!(lr.extend_in_ebb(e10, i12, PO), false); + PO.validate(&lr); + assert_eq!(lr.def(), i11.into()); + assert_eq!(lr.def_local_end(), i13.into()); + } + + #[test] + fn local_arg() { + let e10 = Ebb::new(10); + let i11 = Inst::new(11); + let i12 = Inst::new(12); + let i13 = Inst::new(13); + let mut lr = LiveRange::new(e10); + + // Extending a dead EBB arg in its own block should not indicate that a live-in interval + // was created. + assert_eq!(lr.extend_in_ebb(e10, i12, PO), false); + PO.validate(&lr); + assert!(!lr.is_dead()); + assert!(lr.is_local()); + assert_eq!(lr.def(), e10.into()); + assert_eq!(lr.def_local_end(), i12.into()); + + // Extending to an already covered inst should not change anything. + assert_eq!(lr.extend_in_ebb(e10, i11, PO), false); + PO.validate(&lr); + assert_eq!(lr.def(), e10.into()); + assert_eq!(lr.def_local_end(), i12.into()); + + // Extending further. + assert_eq!(lr.extend_in_ebb(e10, i13, PO), false); + PO.validate(&lr); + assert_eq!(lr.def(), e10.into()); + assert_eq!(lr.def_local_end(), i13.into()); + } + + #[test] + fn global_def() { + let e10 = Ebb::new(10); + let i11 = Inst::new(11); + let i12 = Inst::new(12); + let e20 = Ebb::new(20); + let i21 = Inst::new(21); + let i22 = Inst::new(22); + let i23 = Inst::new(23); + let mut lr = LiveRange::new(i11); + + assert_eq!(lr.extend_in_ebb(e10, i12, PO), false); + + // Adding a live-in interval. + assert_eq!(lr.extend_in_ebb(e20, i22, PO), true); + PO.validate(&lr); + assert_eq!(lr.livein_local_end(e20, PO), Some(i22)); + + // Non-extending the live-in. + assert_eq!(lr.extend_in_ebb(e20, i21, PO), false); + assert_eq!(lr.livein_local_end(e20, PO), Some(i22)); + + // Extending the existing live-in. + assert_eq!(lr.extend_in_ebb(e20, i23, PO), false); + PO.validate(&lr); + assert_eq!(lr.livein_local_end(e20, PO), Some(i23)); + } + + // TODO: Add more tests that exercise the binary search and the coalescing algorithm. +} diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs new file mode 100644 index 0000000000..0d6c9c9574 --- /dev/null +++ b/lib/cretonne/src/regalloc/mod.rs @@ -0,0 +1,5 @@ +//! Register allocation. +//! +//! This module contains data structures and algorithms used for register allocation. + +pub mod liverange; From d9b63bf22749c75987bd937ebe39992a1ebaaac3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 10 Jan 2017 10:01:05 -0800 Subject: [PATCH 0464/3084] Implement a SparseMap data structure. This implements the classic Briggs/Torczon sparse set construct. Adapt it to our existing EntityRef infrastructure so we can use types keys instead of just integers like the original paper does. Also provide a SparseSet type alias which implements a sparse set of entity refeences. This is actually closer to what the original paper describes. --- lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/sparse_map.rs | 297 +++++++++++++++++++++++++++++++++ 2 files changed, 298 insertions(+) create mode 100644 lib/cretonne/src/sparse_map.rs diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 6445d7866b..67694e139f 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -14,6 +14,7 @@ pub mod isa; pub mod cfg; pub mod dominator_tree; pub mod entity_map; +pub mod sparse_map; pub mod settings; pub mod verifier; pub mod regalloc; diff --git a/lib/cretonne/src/sparse_map.rs b/lib/cretonne/src/sparse_map.rs new file mode 100644 index 0000000000..5194d8a760 --- /dev/null +++ b/lib/cretonne/src/sparse_map.rs @@ -0,0 +1,297 @@ +//! Sparse mapping of entity references to larger value types. +//! +//! This module provides a `SparseMap` data structure which implements a sparse mapping from an +//! `EntityRef` key to a value type that may be on the larger side. This implementation is based on +//! the paper: +//! +//! > Briggs, Torczon, *An efficient representation for sparse sets*, +//! ACM Letters on Programming Languages and Systems, Volume 2, Issue 1-4, March-Dec. 1993. +//! +//! A `SparseMap` map provides: +//! +//! - Memory usage equivalent to `EntityMap` + `Vec`, so much smaller than +//! `EntityMap` for sparse mappings of larger `V` types. +//! - Constant time lookup, slightly slower than `EntityMap`. +//! - A very fast, constant time `clear()` operation. +//! - Fast insert and erase operations. +//! - Stable iteration that is as fast as a `Vec`. +//! +//! # Compared to `EntityMap` +//! +//! When should we use a `SparseMap` instead of a secondary `EntityMap`? First of all, `SparseMap` +//! does not provide the functionality of a primary `EntityMap` which can allocate and assign +//! entity references to objects as they are pushed onto the map. It is only the secondary +//! entity maps that can be replaced with a `SparseMap`. +//! +//! - A secondary entity map requires its values to implement `Default`, and it is a bit loose +//! about creating new mappings to the default value. It doesn't distinguish clearly between an +//! unmapped key and one that maps to the default value. `SparseMap` does not require `Default` +//! values, and it tracks accurately if a key has been mapped or not. +//! - Iterating over the contants of an `EntityMap` is linear in the size of the *key space*, while +//! iterating over a `SparseMap` is linear in the number of elements in the mapping. This is an +//! advantage precisely when the mapping is sparse. +//! - `SparseMap::clear()` is constant time and super-fast. `EntityMap::clear()` is linear in the +//! size of the key space. (Or, rather the required `resize()` call following the `clear()` is). +//! - `SparseMap` requires the values to implement `SparseMapValue` which means that they must +//! contain their own key. + +use entity_map::{EntityRef, EntityMap}; +use std::mem; +use std::u32; + +/// Trait for extracting keys from values stored in a `SparseMap`. +/// +/// All values stored in a `SparseMap` must keep track of their own key in the map and implement +/// this trait to provide access to the key. +pub trait SparseMapValue { + /// Get the key of this sparse map value. This key is not alowed to change while the value + /// is a member of the map. + fn key(&self) -> K; +} + +/// A sparse mapping of entity references. +pub struct SparseMap + where K: EntityRef, + V: SparseMapValue +{ + sparse: EntityMap, + dense: Vec, +} + +impl SparseMap + where K: EntityRef, + V: SparseMapValue +{ + /// Create a new empty mapping. + pub fn new() -> Self { + SparseMap { + sparse: EntityMap::new(), + dense: Vec::new(), + } + } + + /// Returns the number of elements in the map. + pub fn len(&self) -> usize { + self.dense.len() + } + + /// Returns true is the map contains no elements. + pub fn is_empty(&self) -> bool { + self.dense.is_empty() + } + + /// Returns a reference to the value corresponding to the key. + pub fn get(&self, key: K) -> Option<&V> { + if let Some(idx) = self.sparse.get(key).cloned() { + if let Some(entry) = self.dense.get(idx as usize) { + if entry.key() == key { + return Some(entry); + } + } + } + None + } + + /// Returns a mutable reference to the value corresponding to the key. + /// + /// Note that the returned value must not be mutated in a way that would change its key. This + /// would invalidate the sparse set data structure. + pub fn get_mut(&mut self, key: K) -> Option<&mut V> { + if let Some(idx) = self.sparse.get(key).cloned() { + if let Some(entry) = self.dense.get_mut(idx as usize) { + if entry.key() == key { + return Some(entry); + } + } + } + None + } + + /// Return the index into `dense` of the value corresponding to `key`. + fn index(&self, key: K) -> Option { + if let Some(idx) = self.sparse.get(key).cloned() { + let idx = idx as usize; + if let Some(entry) = self.dense.get(idx) { + if entry.key() == key { + return Some(idx); + } + } + } + None + } + + /// Insert a value into the map. + /// + /// If the map did not have this key present, `None` is returned. + /// + /// If the map did have this key present, the value is updated, and the old value is returned. + /// + /// It is not necessary to provide a key since the value knows its own key already. + pub fn insert(&mut self, value: V) -> Option { + let key = value.key(); + + // Replace the existing entry for `key` if there is one. + if let Some(entry) = self.get_mut(key) { + return Some(mem::replace(entry, value)); + } + + // There was no previous entry for `key`. Add it to the end of `dense`. + let idx = self.dense.len(); + assert!(idx <= u32::MAX as usize, "SparseMap overflow"); + self.dense.push(value); + *self.sparse.ensure(key) = idx as u32; + None + } + + /// Remove a value from the map and return it. + pub fn remove(&mut self, key: K) -> Option { + if let Some(idx) = self.index(key) { + let back = self.dense.pop().unwrap(); + + // Are we popping the back of `dense`? + if idx == self.dense.len() { + return Some(back); + } + + // We're removing an element from the middle of `dense`. + // Replace the element at `idx` with the back of `dense`. + // Repair `sparse` first. + self.sparse[back.key()] = idx as u32; + return Some(mem::replace(&mut self.dense[idx], back)); + } + + // Nothing to remove. + None + } +} + +/// Any `EntityRef` can be used as a sparse map value representing itself. +impl SparseMapValue for T + where T: EntityRef +{ + fn key(&self) -> T { + *self + } +} + +/// A sparse set of entity references. +/// +/// Any type that implements `EntityRef` can be used as a sparse set value too. +pub type SparseSet = SparseMap; + +#[cfg(test)] +mod tests { + use super::*; + use entity_map::EntityRef; + use ir::Inst; + + // Mock key-value object for testing. + #[derive(PartialEq, Eq, Debug)] + struct Obj(Inst, &'static str); + + impl SparseMapValue for Obj { + fn key(&self) -> Inst { + self.0 + } + } + + #[test] + fn empty_immutable_map() { + let i1 = Inst::new(1); + let map: SparseMap = SparseMap::new(); + + assert!(map.is_empty()); + assert_eq!(map.len(), 0); + assert_eq!(map.get(i1), None); + } + + #[test] + fn single_entry() { + let i0 = Inst::new(0); + let i1 = Inst::new(1); + let i2 = Inst::new(2); + let mut map = SparseMap::new(); + + assert!(map.is_empty()); + assert_eq!(map.len(), 0); + assert_eq!(map.get(i1), None); + assert_eq!(map.get_mut(i1), None); + assert_eq!(map.remove(i1), None); + + assert_eq!(map.insert(Obj(i1, "hi")), None); + assert!(!map.is_empty()); + assert_eq!(map.len(), 1); + assert_eq!(map.get(i0), None); + assert_eq!(map.get(i1), Some(&Obj(i1, "hi"))); + assert_eq!(map.get(i2), None); + assert_eq!(map.get_mut(i0), None); + assert_eq!(map.get_mut(i1), Some(&mut Obj(i1, "hi"))); + assert_eq!(map.get_mut(i2), None); + + assert_eq!(map.remove(i0), None); + assert_eq!(map.remove(i2), None); + assert_eq!(map.remove(i1), Some(Obj(i1, "hi"))); + assert_eq!(map.len(), 0); + assert_eq!(map.get(i1), None); + assert_eq!(map.get_mut(i1), None); + assert_eq!(map.remove(i0), None); + assert_eq!(map.remove(i1), None); + assert_eq!(map.remove(i2), None); + } + + #[test] + fn multiple_entries() { + let i0 = Inst::new(0); + let i1 = Inst::new(1); + let i2 = Inst::new(2); + let i3 = Inst::new(3); + let mut map = SparseMap::new(); + + assert_eq!(map.insert(Obj(i2, "foo")), None); + assert_eq!(map.insert(Obj(i1, "bar")), None); + assert_eq!(map.insert(Obj(i0, "baz")), None); + + assert_eq!(map.len(), 3); + assert_eq!(map.get(i0), Some(&Obj(i0, "baz"))); + assert_eq!(map.get(i1), Some(&Obj(i1, "bar"))); + assert_eq!(map.get(i2), Some(&Obj(i2, "foo"))); + assert_eq!(map.get(i3), None); + + // Remove front object, causing back to be swapped down. + assert_eq!(map.remove(i1), Some(Obj(i1, "bar"))); + assert_eq!(map.len(), 2); + assert_eq!(map.get(i0), Some(&Obj(i0, "baz"))); + assert_eq!(map.get(i1), None); + assert_eq!(map.get(i2), Some(&Obj(i2, "foo"))); + assert_eq!(map.get(i3), None); + + // Reinsert something at a previously used key. + assert_eq!(map.insert(Obj(i1, "barbar")), None); + assert_eq!(map.len(), 3); + assert_eq!(map.get(i0), Some(&Obj(i0, "baz"))); + assert_eq!(map.get(i1), Some(&Obj(i1, "barbar"))); + assert_eq!(map.get(i2), Some(&Obj(i2, "foo"))); + assert_eq!(map.get(i3), None); + + // Replace an entry. + assert_eq!(map.insert(Obj(i0, "bazbaz")), Some(Obj(i0, "baz"))); + assert_eq!(map.len(), 3); + assert_eq!(map.get(i0), Some(&Obj(i0, "bazbaz"))); + assert_eq!(map.get(i1), Some(&Obj(i1, "barbar"))); + assert_eq!(map.get(i2), Some(&Obj(i2, "foo"))); + assert_eq!(map.get(i3), None); + } + + #[test] + fn entity_set() { + let i0 = Inst::new(0); + let i1 = Inst::new(1); + let mut set = SparseSet::new(); + + assert_eq!(set.insert(i0), None); + assert_eq!(set.insert(i0), Some(i0)); + assert_eq!(set.insert(i1), None); + assert_eq!(set.get(i0), Some(&i0)); + assert_eq!(set.get(i1), Some(&i1)); + } +} From 748fbd993ea3fd83ae7575e1a6786dc40efc877b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 10 Jan 2017 13:51:20 -0800 Subject: [PATCH 0465/3084] Add iteration support to SparseMap. This is simply the slice iterator for the dense vector. - map.values() returns an iterator with references to the values. - for i in &map iterates over references to the values. --- lib/cretonne/src/sparse_map.rs | 35 ++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/lib/cretonne/src/sparse_map.rs b/lib/cretonne/src/sparse_map.rs index 5194d8a760..9f0d93a55a 100644 --- a/lib/cretonne/src/sparse_map.rs +++ b/lib/cretonne/src/sparse_map.rs @@ -37,6 +37,7 @@ use entity_map::{EntityRef, EntityMap}; use std::mem; +use std::slice; use std::u32; /// Trait for extracting keys from values stored in a `SparseMap`. @@ -163,6 +164,28 @@ impl SparseMap // Nothing to remove. None } + + /// Get an iterator over the values in the map. + /// + /// The iteration order is entirely determined by the preceding sequence of `insert` and + /// `remove` operations. In particular, if no elements were removed, this is the insertion + /// order. + pub fn values(&self) -> slice::Iter { + self.dense.iter() + } +} + +/// Iterating over the elements of a set. +impl<'a, K, V> IntoIterator for &'a SparseMap + where K: EntityRef, + V: SparseMapValue +{ + type Item = &'a V; + type IntoIter = slice::Iter<'a, V>; + + fn into_iter(self) -> Self::IntoIter { + self.values() + } } /// Any `EntityRef` can be used as a sparse map value representing itself. @@ -203,6 +226,7 @@ mod tests { assert!(map.is_empty()); assert_eq!(map.len(), 0); assert_eq!(map.get(i1), None); + assert_eq!(map.values().count(), 0); } #[test] @@ -251,6 +275,10 @@ mod tests { assert_eq!(map.insert(Obj(i1, "bar")), None); assert_eq!(map.insert(Obj(i0, "baz")), None); + // Iteration order = insertion order when nothing has been removed yet. + assert_eq!(map.values().map(|obj| obj.1).collect::>(), + ["foo", "bar", "baz"]); + assert_eq!(map.len(), 3); assert_eq!(map.get(i0), Some(&Obj(i0, "baz"))); assert_eq!(map.get(i1), Some(&Obj(i1, "bar"))); @@ -280,6 +308,13 @@ mod tests { assert_eq!(map.get(i1), Some(&Obj(i1, "barbar"))); assert_eq!(map.get(i2), Some(&Obj(i2, "foo"))); assert_eq!(map.get(i3), None); + + // Check the reference `IntoIter` impl. + let mut v = Vec::new(); + for i in &map { + v.push(i.1); + } + assert_eq!(v.len(), map.len()); } #[test] From b5715d5c48ca492dd839f4cfd8badd4554fe03be Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 10 Jan 2017 14:21:56 -0800 Subject: [PATCH 0466/3084] Allow live ranges to be values in a SparseMap. This requires the value number to be stored in the live range itself. --- lib/cretonne/src/regalloc/liverange.rs | 36 +++++++++++++++++++------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index b26dc8e8b1..e192efdc4d 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -108,7 +108,8 @@ //! use std::cmp::Ordering; -use ir::{Inst, Ebb, ProgramPoint, ProgramOrder}; +use ir::{Inst, Ebb, Value, ProgramPoint, ProgramOrder}; +use sparse_map::SparseMapValue; /// Global live range of a single SSA value. /// @@ -138,6 +139,10 @@ use ir::{Inst, Ebb, ProgramPoint, ProgramOrder}; /// instructions using or defining their value, `LiveRange` structs can contain references to /// branch and jump instructions. pub struct LiveRange { + /// The value described by this live range. + /// This member can't be modified in case the live range is stored in a `SparseMap`. + value: Value, + /// The instruction or EBB header where this value is defined. def_begin: ProgramPoint, @@ -193,12 +198,13 @@ impl Interval { } impl LiveRange { - /// Create a new live range defined at `def`. + /// Create a new live range for `value` defined at `def`. /// /// The live range will be created as dead, but it can be extended with `extend_in_ebb()`. - pub fn new>(def: PP) -> LiveRange { + pub fn new>(value: Value, def: PP) -> LiveRange { let def = def.into(); LiveRange { + value: value, def_begin: def, def_end: def, liveins: Vec::new(), @@ -317,10 +323,17 @@ impl LiveRange { } } +/// Allow a `LiveRange` to be stored in a `SparseMap` indexed by values. +impl SparseMapValue for LiveRange { + fn key(&self) -> Value { + self.value + } +} + #[cfg(test)] mod tests { use super::LiveRange; - use ir::{Inst, Ebb}; + use ir::{Inst, Ebb, Value}; use entity_map::EntityRef; use ir::{ProgramOrder, ExpandedProgramPoint}; use std::cmp::Ordering; @@ -402,9 +415,10 @@ mod tests { #[test] fn dead_def_range() { + let v0 = Value::new(0); let i1 = Inst::new(1); let e2 = Ebb::new(2); - let lr = LiveRange::new(i1); + let lr = LiveRange::new(v0, i1); assert!(lr.is_dead()); assert!(lr.is_local()); assert_eq!(lr.def(), i1.into()); @@ -415,8 +429,9 @@ mod tests { #[test] fn dead_arg_range() { + let v0 = Value::new(0); let e2 = Ebb::new(2); - let lr = LiveRange::new(e2); + let lr = LiveRange::new(v0, e2); assert!(lr.is_dead()); assert!(lr.is_local()); assert_eq!(lr.def(), e2.into()); @@ -428,11 +443,12 @@ mod tests { #[test] fn local_def() { + let v0 = Value::new(0); let e10 = Ebb::new(10); let i11 = Inst::new(11); let i12 = Inst::new(12); let i13 = Inst::new(13); - let mut lr = LiveRange::new(i11); + let mut lr = LiveRange::new(v0, i11); assert_eq!(lr.extend_in_ebb(e10, i13, PO), false); PO.validate(&lr); @@ -450,11 +466,12 @@ mod tests { #[test] fn local_arg() { + let v0 = Value::new(0); let e10 = Ebb::new(10); let i11 = Inst::new(11); let i12 = Inst::new(12); let i13 = Inst::new(13); - let mut lr = LiveRange::new(e10); + let mut lr = LiveRange::new(v0, e10); // Extending a dead EBB arg in its own block should not indicate that a live-in interval // was created. @@ -480,6 +497,7 @@ mod tests { #[test] fn global_def() { + let v0 = Value::new(0); let e10 = Ebb::new(10); let i11 = Inst::new(11); let i12 = Inst::new(12); @@ -487,7 +505,7 @@ mod tests { let i21 = Inst::new(21); let i22 = Inst::new(22); let i23 = Inst::new(23); - let mut lr = LiveRange::new(i11); + let mut lr = LiveRange::new(v0, i11); assert_eq!(lr.extend_in_ebb(e10, i12, PO), false); From 05e06cb876a3194a2bdced22d7b0bce63f565ecc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 10 Jan 2017 15:33:03 -0800 Subject: [PATCH 0467/3084] Implement DoubleEndedIterator for the ebb_insts() iterator. Make it possible to iterate backwards over the instructions in an EBB. --- lib/cretonne/src/ir/layout.rs | 38 ++++++++++++++++++++++++++++------- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index cecc8645ef..7f2497df1b 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -452,7 +452,8 @@ impl Layout { pub fn ebb_insts<'f>(&'f self, ebb: Ebb) -> Insts<'f> { Insts { layout: self, - next: self.ebbs[ebb].first_inst.wrap(), + head: self.ebbs[ebb].first_inst.wrap(), + tail: self.ebbs[ebb].last_inst.wrap(), } } @@ -535,20 +536,39 @@ struct InstNode { /// Iterate over instructions in an EBB in layout order. See `Layout::ebb_insts()`. pub struct Insts<'f> { layout: &'f Layout, - next: Option, + head: Option, + tail: Option, } impl<'f> Iterator for Insts<'f> { type Item = Inst; fn next(&mut self) -> Option { - match self.next { - Some(inst) => { - self.next = self.layout.insts[inst].next.wrap(); - Some(inst) + let rval = self.head; + if let Some(inst) = rval { + if self.head == self.tail { + self.head = None; + self.tail = None; + } else { + self.head = Some(self.layout.insts[inst].next); } - None => None, } + rval + } +} + +impl<'f> DoubleEndedIterator for Insts<'f> { + fn next_back(&mut self) -> Option { + let rval = self.tail; + if let Some(inst) = rval { + if self.head == self.tail { + self.head = None; + self.tail = None; + } else { + self.tail = Some(self.layout.insts[inst].prev); + } + } + rval } } @@ -1077,6 +1097,10 @@ mod tests { let v: Vec = layout.ebb_insts(e1).collect(); assert_eq!(v, [i1, i2]); + // Test double-ended instruction iterator. + let v: Vec = layout.ebb_insts(e1).rev().collect(); + assert_eq!(v, [i2, i1]); + layout.append_inst(i0, e1); verify(&mut layout, &[(e1, &[i1, i2, i0])]); From aec53ec3a9070da8d5754c6969481e8442b4f8c8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 13 Jan 2017 11:42:26 -0800 Subject: [PATCH 0468/3084] Add a liveness analysis. This code is best tested with larger functions with more EBBs. Perhaps a new file-test category is in order? --- lib/cretonne/src/ir/instructions.rs | 8 +- lib/cretonne/src/regalloc/liveness.rs | 294 ++++++++++++++++++++++++++ lib/cretonne/src/regalloc/mod.rs | 1 + lib/cretonne/src/sparse_map.rs | 5 + 4 files changed, 304 insertions(+), 4 deletions(-) create mode 100644 lib/cretonne/src/regalloc/liveness.rs diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 21f9b92c3c..eb0b1e5bd4 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -412,8 +412,8 @@ pub struct ReturnData { impl InstructionData { /// Execute a closure once for each argument to this instruction. /// See also the `arguments()` method. - pub fn each_arg(&self, func: F) - where F: Fn(Value) + pub fn each_arg(&self, mut func: F) + where F: FnMut(Value) { for part in &self.arguments() { for &arg in part.iter() { @@ -424,8 +424,8 @@ impl InstructionData { /// Execute a closure with a mutable reference to each argument to this instruction. /// See also the `arguments_mut()` method. - pub fn each_arg_mut(&mut self, func: F) - where F: Fn(&mut Value) + pub fn each_arg_mut(&mut self, mut func: F) + where F: FnMut(&mut Value) { for part in &mut self.arguments_mut() { for arg in part.iter_mut() { diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs new file mode 100644 index 0000000000..c7646ed701 --- /dev/null +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -0,0 +1,294 @@ +//! Liveness analysis for SSA values. +//! +//! This module computes the live range of all the SSA values in a function and produces a +//! `LiveRange` instance for each. +//! +//! +//! # Liveness consumers +//! +//! The primary consumer of the liveness analysis is the SSA coloring pass which goes through each +//! EBB and assigns a register to the defined values. This algorithm needs to maintain a set of the +//! curently live values as it is iterating down the instructions in the EBB. It asks the following +//! questions: +//! +//! - What is the set of live values at the entry to the EBB? +//! - When moving past a use of a value, is that value still alive in the EBB, or was that the last +//! use? +//! - When moving past a branch, which of the live values are still live below the branch? +//! +//! The set of `LiveRange` instances can answer these questions through their `def_local_end` and +//! `livein_local_end` queries. The coloring algorithm visits EBBs in a topological order of the +//! dominator tree, so it can compute the set of live values at the begining of an EBB by starting +//! from the set of live values at the dominating branch instruction and filtering it with +//! `livein_local_end`. These sets do not need to be stored in the liveness analysis. +//! +//! The secondary consumer of the liveness analysis is the spilling pass which needs to count the +//! number of live values at every program point and insert spill code until the number of +//! registers needed is small enough. +//! +//! +//! # Alternative algorithms +//! +//! A number of different liveness analysis algorithms exist, so it is worthwhile to look at a few +//! alternatives. +//! +//! ## Dataflow equations +//! +//! The classic *live variables analysis* that you will find in all compiler books from the +//! previous century does not depend on SSA form. It is typically implemented by iteratively +//! solving dataflow equations on bitvectors of variables. The result is a live-out bitvector of +//! variables for every basic block in the program. +//! +//! This algorithm has some disadvantages that makes us look elsewhere: +//! +//! - Quadratic memory use. We need a bit per variable per basic block in the function. +//! - Sparse representation. In practice, the majority of SSA values never leave their basic block, +//! and those that do span basic blocks rarely span a large number of basic blocks. This makes +//! the bitvectors quite sparse. +//! - Traditionally, the dataflow equations were solved for real program *variables* which does not +//! include temporaries used in evaluating expressions. We have an SSA form program which blurs +//! the distinction between temporaries and variables. This makes the quadratic memory problem +//! worse because there are many more SSA values than there was variables in the original +//! program, and we don't know a priori which SSA values leave their basic block. +//! - Missing last-use information. For values that are not live-out of a basic block, we would +//! need to store information about the last use in the block somewhere. LLVM stores this +//! information as a 'kill bit' on the last use in the IR. Maintaining these kill bits has been a +//! source of problems for LLVM's register allocator. +//! +//! Dataflow equations can detect when a variable is used uninitialized, and they can handle +//! multiple definitions of the same variable. We don't need this generality since we already have +//! a program in SSA form. +//! +//! ## LLVM's liveness analysis +//! +//! LLVM's register allocator computes liveness per *virtual register*, where a virtual register is +//! a disjoint union of related SSA values that should be assigned to the same physical register. +//! It uses a compact data structure very similar to our `LiveRange`. The important difference is +//! that Cretonne's `LiveRange` only describes a single SSA value, while LLVM's `LiveInterval` +//! describes the live range of a virtual register *and* which one of the related SSA values is +//! live at any given program point. +//! +//! LLVM computes the live range of each virtual register independently by using the use-def chains +//! that are baked into its IR. The algorithm for a single virtual register is: +//! +//! 1. Initialize the live range with a single-instruction snippet of liveness at each def, using +//! the def-chain. This does not include any phi-values. +//! 2. Go through the virtual register's use chain and perform the following steps at each use: +//! 3. Perform an exhaustive depth-first traversal up the CFG from the use. Look for basic blocks +//! that already contain some liveness and extend the last live SSA value in the block to be +//! live-out. Also build a list of new basic blocks where the register needs to be live-in. +//! 4. Iteratively propagate live-out SSA values to the new live-in blocks. This may require new +//! PHI values to be created when different SSA values can reach the same block. +//! +//! The iterative SSA form reconstruction can be skipped if the depth-first search only encountered +//! one SSA value. +//! +//! This algorithm has some advantages compared to the dataflow equations: +//! +//! - The live ranges of local virtual registers are computed very quickly without ever traversing +//! the CFG. The memory needed to store these live ranges is independent of the number of basic +//! blocks in the program. +//! - The time to compute the live range of a global virtual register is proportional to the number +//! of basic blocks covered. Many virtual registers only cover a few blocks, even in very large +//! functions. +//! - A single live range can be recomputed after making modifications to the IR. No global +//! algorithm is necessary. This feature depends on having use-def chains for virtual registers +//! which Cretonne doesn't. +//! +//! Cretonne uses a very similar data structures and algorithms to LLVM, with the important +//! difference that live ranges are computed per SSA value instead of per virtual register, and the +//! uses in Cretonne IR refers to SSA values instead of virtual registers. This means that Cretonne +//! can skip the last step of reconstructing SSA form for the virtual register uses. +//! +//! ## Fast Liveness Checking for SSA-Form Programs +//! +//! A liveness analysis that is often brought up in the context of SSA-based register allocation +//! was presented at CGO 2008: +//! +//! > Boissinot, B., Hack, S., Grund, D., de Dinechin, B. D., & Rastello, F. (2008). *Fast Liveness +//! Checking for SSA-Form Programs.* CGO. +//! +//! This analysis uses a global precomputation that only depends on the CFG of the function. It +//! then allows liveness queries for any (value, program point) pair. Each query traverses the use +//! chain of the value and performs lookups in the precomputed bitvectors. +//! +//! I did not seriously consider this analysis for Cretonne because: +//! +//! - It depends critically on use chains which Cretonne doesn't have. +//! - Popular variables like the `this` pointer in a C++ method can have very large use chains. +//! Traversing such a long use chain on every liveness lookup has the potential for some nasty +//! quadratic behavior in unfortunate cases. +//! - It says "fast" in the title, but the paper only claims to be 16% faster than a dataflow based +//! approach, which isn't that impressive. +//! +//! Nevertheless, the property of only depending in the CFG structure is very useful. If Cretonne +//! gains use chains, this approach would be worth a proper evaluation. +//! +//! +//! # Cretonne's liveness analysis +//! +//! The algorithm implemented in this module is similar to LLVM's with these differences: +//! +//! - The `LiveRange` data structure describes the liveness of a single SSA value, not a virtual +//! register. +//! - Instructions in Cretonne IR contains references to SSA values, not virtual registers. +//! - All live ranges are computed in one traversal of the program. Cretonne doesn't have use +//! chains, so it is not possible to compute the live range for a single SSA value independently. +//! +//! The liveness computation visits all instructions in the program. The order is not important for +//! the algorithm to be correct. At each instruction, the used values are examined. +//! +//! - The first time a value is encountered, its live range is constructed as a dead live range +//! containing only the defining program point. +//! - The local interval of the value's live range is extended so it reaches the use. This may +//! require creating a new live-in local interval for the EBB. +//! - If the live range became live-in to the EBB, add the EBB to a work-list. +//! - While the work-list is non-empty pop a live-in EBB and repeat the two steps above, using each +//! of the live-in EBB's CFG predecessor instructions as a 'use'. +//! +//! The effect of this algorithm is to extend the live range of each to reach uses as they are +//! visited. No data about each value beyond the live range is needed between visiting uses, so +//! nothing is lost by computing the live range of all values simultaneously. +//! +//! ## Cache efficiency of Cretonne vs LLVM +//! +//! Since LLVM computes the complete live range of a virtual register in one go, it can keep the +//! whole `LiveInterval` for the register in L1 cache. Since it is visiting the instructions in use +//! chain order, some cache thrashing can occur as a result of pulling instructions into cache +//! somewhat chaotically. +//! +//! Cretonne uses a transposed algorithm, visiting instructions in order. This means that each +//! instruction is brought into cache only once, and it is likely that the other instructions on +//! the same cache line will be visited before the line is evicted. +//! +//! Cretonne's problem is that the `LiveRange` structs are visited many times and not always +//! regularly. We should strive to make the `LiveRange` struct as small as possible such that +//! multiple related values can live on the same cache line. +//! +//! - Local values should fit in a 16-byte `LiveRange` struct or smaller. The current +//! implementation contains a 24-byte `Vec` object and a redundant `value` member pushing the +//! size to 32 bytes. +//! - Related values should be stored on the same cache line. The current sparse set implementation +//! does a decent job of that. +//! - For global values, the list of live-in intervals is very likely to fit on a single cache +//! line. These lists are very likely ot be found in L2 cache at least. +//! +//! There is some room for improvement. + +use regalloc::liverange::LiveRange; +use ir::{Function, Value, Inst, Ebb, ProgramPoint}; +use ir::dfg::{DataFlowGraph, ValueDef}; +use cfg::ControlFlowGraph; +use sparse_map::SparseMap; + +/// A set of live ranges, indexed by value number. +struct LiveRangeSet(SparseMap); + +impl LiveRangeSet { + pub fn new() -> LiveRangeSet { + LiveRangeSet(SparseMap::new()) + } + + pub fn clear(&mut self) { + self.0.clear(); + } + + /// Get a mutable reference to the live range for `value`. + /// Create it if necessary. + pub fn get_or_create(&mut self, value: Value, dfg: &DataFlowGraph) -> &mut LiveRange { + // It would be better to use `get_mut()` here, but that leads to borrow checker fighting + // which can probably only be resolved by non-lexical lifetimes. + // https://github.com/rust-lang/rfcs/issues/811 + if self.0.get(value).is_none() { + // Create a live range for value. We need the program point that defines it. + let def: ProgramPoint = match dfg.value_def(value) { + ValueDef::Res(inst, _) => inst.into(), + ValueDef::Arg(ebb, _) => ebb.into(), + }; + self.0.insert(LiveRange::new(value, def)); + } + self.0.get_mut(value).unwrap() + } +} + +/// Liveness analysis for a function. +/// +/// Compute a live range for every SSA value used in the function. +pub struct Liveness { + /// The live ranges that have been computed so far. + ranges: LiveRangeSet, + + /// Working space for the `extend_to_use` algorithm. + /// This vector is always empty, except for inside that function. + /// It lives here to avoid repeated allocation of scratch memory. + worklist: Vec, +} + +impl Liveness { + /// Create a new empty liveness analysis. + /// + /// The memory allocated for this analysis can be reused for multiple functions. Use the + /// `compute` method to actually runs the analysis for a function. + pub fn new() -> Liveness { + Liveness { + ranges: LiveRangeSet::new(), + worklist: Vec::new(), + } + } + + /// Compute the live ranges of all SSA values used in `func`. + /// This clears out any existing analysis stored in this data structure. + pub fn compute(&mut self, func: &Function, cfg: &ControlFlowGraph) { + self.ranges.clear(); + + // The liveness computation needs to visit all uses, but the order doesn't matter. + // TODO: Perhaps this traversal of the function could be combined with a dead code + // elimination pass if we visit a post-order of the dominator tree? + // TODO: Resolve value aliases while we're visiting instructions? + for ebb in func.layout.ebbs() { + for inst in func.layout.ebb_insts(ebb) { + func.dfg[inst].each_arg(|arg| self.extend_to_use(arg, ebb, inst, func, cfg)); + } + } + } + + /// Extend the live range for `value` so it reaches `to` which must live in `ebb`. + fn extend_to_use(&mut self, + value: Value, + ebb: Ebb, + to: Inst, + func: &Function, + cfg: &ControlFlowGraph) { + // Get the live range, create it as a dead range if necessary. + let lr = self.ranges.get_or_create(value, &func.dfg); + + // This is our scratch working space, and we'll leave it empty when we return. + assert!(self.worklist.is_empty()); + + // Extend the range locally in `ebb`. + // If there already was a live interval in that block, we're done. + if lr.extend_in_ebb(ebb, to, &func.layout) { + self.worklist.push(ebb); + } + + // The worklist contains those EBBs where we have learned that the value needs to be + // live-in. + // + // This algorithm bcomes a depth-first traversal up the CFG, enumerating all paths through + // the CFG from the existing live range to `ebb`. + // + // Extend the live range as we go. The live range itself also serves as a visited set since + // `extend_in_ebb` will never return true twice for the same EBB. + // + while let Some(livein) = self.worklist.pop() { + // We've learned that the value needs to be live-in to the `livein` EBB. + // Make sure it is also live at all predecessor branches to `livein`. + for &(pred, branch) in cfg.get_predecessors(livein) { + if lr.extend_in_ebb(pred, branch, &func.layout) { + // This predecessor EBB also became live-in. We need to process it later. + self.worklist.push(pred); + } + } + } + } +} diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index 0d6c9c9574..dff29bc5bc 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -3,3 +3,4 @@ //! This module contains data structures and algorithms used for register allocation. pub mod liverange; +pub mod liveness; diff --git a/lib/cretonne/src/sparse_map.rs b/lib/cretonne/src/sparse_map.rs index 9f0d93a55a..48844c026a 100644 --- a/lib/cretonne/src/sparse_map.rs +++ b/lib/cretonne/src/sparse_map.rs @@ -81,6 +81,11 @@ impl SparseMap self.dense.is_empty() } + /// Remove all elements from the mapping. + pub fn clear(&mut self) { + self.dense.clear(); + } + /// Returns a reference to the value corresponding to the key. pub fn get(&self, key: K) -> Option<&V> { if let Some(idx) = self.sparse.get(key).cloned() { From 32193d21a213e0096a071e31c44d6e63db59b2c4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Jan 2017 11:04:11 -0800 Subject: [PATCH 0469/3084] Update regalloc document to reflect implementation. --- cranelift/docs/regalloc.rst | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/cranelift/docs/regalloc.rst b/cranelift/docs/regalloc.rst index 18057c2511..1d662e9a63 100644 --- a/cranelift/docs/regalloc.rst +++ b/cranelift/docs/regalloc.rst @@ -141,17 +141,18 @@ When all the EBBs in a function are laid out linearly, the live range of a value doesn't have to be a contiguous interval, although it will be in a majority of cases. There can be holes in the linear live range. -The live range of an SSA value is represented as: +The part of a value's live range that falls inside a single EBB will always be +an interval without any holes. This follows from the dominance requirements of +SSA. A live range is represented as: -- The earliest program point where the value is live. -- The latest program point where the value is live. -- A (often empty) list of holes, sorted in program order. +- The interval inside the EBB where the value is defined. +- A set of intervals for EBBs where the value is live-in. -Any value that is only used inside a single EBB will have a live range without -holes. Some values are live across large parts of the function, and this can -often be represented with very few holes. It is important that the live range -data structure doesn't have to grow linearly with the number of EBBs covered by -a live range. +Any value that is only used inside a single EBB will have an empty set of +live-in intervals. Some values are live across large parts of the function, and +this can often be represented with coalesced live-in intervals covering many +EBBs. It is important that the live range data structure doesn't have to grow +linearly with the number of EBBs covered by a live range. This representation is very similar to LLVM's ``LiveInterval`` data structure with a few important differences: @@ -160,10 +161,16 @@ with a few important differences: ``LiveInterval`` represents the union of multiple related SSA values in a virtual register. This makes Cretonne's representation smaller because individual segments don't have to annotated with a value number. -- Cretonne stores the min and max program points separately from a list of - holes, while LLVM stores an array of segments. The two representations are - equivalent, but Cretonne optimizes for the common case of a single contiguous - interval. +- Cretonne stores the def-interval separately from a list of coalesced live-in + intervals, while LLVM stores an array of segments. The two representations + are equivalent, but Cretonne optimizes for the common case of a value that is + only used locally. +- It is simpler to check if two live ranges are overlapping. The dominance + properties of SSA form means that it is only necessary to check the + def-interval of each live range against the intervals of the other range. It + is not necessary to check for overlap between the two sets of live-in + intervals. This makes the overlap check logarithmic in the number of live-in + intervals instead of linear. - LLVM represents a program point as ``SlotIndex`` which holds a pointer to a 32-byte ``IndexListEntry`` struct. The entries are organized in a double linked list that mirrors the ordering of instructions in a basic block. This From b42faea98099e75f751cb58bd0c7809ae1e4ebb4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Jan 2017 12:13:00 -0800 Subject: [PATCH 0470/3084] Implement PackedOption to address #19. The PackedOption struct uses the same amount of memory as T, but can represent None via a reserved value. --- lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/packed_option.rs | 121 ++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) create mode 100644 lib/cretonne/src/packed_option.rs diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 67694e139f..25feb3c0db 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -24,3 +24,4 @@ mod constant_hash; mod predicates; mod legalizer; mod ref_slice; +mod packed_option; diff --git a/lib/cretonne/src/packed_option.rs b/lib/cretonne/src/packed_option.rs new file mode 100644 index 0000000000..2c93b38ead --- /dev/null +++ b/lib/cretonne/src/packed_option.rs @@ -0,0 +1,121 @@ +//! Compact representation of `Option` for types with a reserved value. +//! +//! Small Cretonne types like the 32-bit entity references are often used in tables and linked +//! lists where an `Option` is needed. Unfortunately, that would double the size of the tables +//! because `Option` is twice as big as `T`. +//! +//! This module provides a `PackedOption` for types that have a reserved value that can be used +//! to represent `None`. + +use std::fmt; + +/// Types that have a reserved value which can't be created any other way. +pub trait ReservedValue: Eq { + /// Create an instance of the reserved value. + fn reserved_value() -> Self; +} + +/// Packed representation of `Option`. +#[derive(Clone, Copy, PartialEq, PartialOrd, Eq, Ord, Hash)] +pub struct PackedOption(T); + +impl PackedOption { + /// Returns `true` if the packed option is a `None` value. + pub fn is_none(&self) -> bool { + self.0 == T::reserved_value() + } + + /// Expand the packed option into a normal `Option`. + pub fn expand(self) -> Option { + if self.is_none() { None } else { Some(self.0) } + } +} + +impl Default for PackedOption { + /// Create a default packed option representing `None`. + fn default() -> PackedOption { + PackedOption(T::reserved_value()) + } +} + +impl From for PackedOption { + /// Convert `t` into a packed `Some(x)`. + fn from(t: T) -> PackedOption { + debug_assert!(t != T::reserved_value(), + "Can't make a PackedOption from the reserved value."); + PackedOption(t) + } +} + +impl From> for PackedOption { + /// Convert an option into its packed equivalent. + fn from(opt: Option) -> PackedOption { + match opt { + None => Self::default(), + Some(t) => t.into(), + } + } +} + +impl Into> for PackedOption { + fn into(self) -> Option { + self.expand() + } +} + +impl fmt::Debug for PackedOption + where T: ReservedValue + fmt::Debug +{ + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + if self.is_none() { + write!(f, "None") + } else { + write!(f, "Some({:?})", self.0) + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + // Dummy entity class, with no Copy or Clone. + #[derive(Debug, PartialEq, Eq)] + struct NoC(u32); + + impl ReservedValue for NoC { + fn reserved_value() -> Self { + NoC(13) + } + } + + #[test] + fn moves() { + let x = NoC(3); + let somex: PackedOption = x.into(); + assert!(!somex.is_none()); + assert_eq!(somex.expand(), Some(NoC(3))); + + let none: PackedOption = None.into(); + assert!(none.is_none()); + assert_eq!(none.expand(), None); + } + + // Dummy entity class, with Copy. + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + struct Ent(u32); + + impl ReservedValue for Ent { + fn reserved_value() -> Self { + Ent(13) + } + } + + #[test] + fn copies() { + let x = Ent(2); + let some: PackedOption = x.into(); + assert_eq!(some.expand(), x.into()); + assert_eq!(some, x.into()); + } +} From 8a30bae9092076540f03eff0924613a91624678c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Jan 2017 12:34:18 -0800 Subject: [PATCH 0471/3084] Move duplicated entity code into a macro. Implement ReservedValue for all the entities. --- lib/cretonne/src/ir/entities.rs | 162 +++++++++----------------------- 1 file changed, 43 insertions(+), 119 deletions(-) diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 4570a21009..ed42b33633 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -20,24 +20,50 @@ //! format. use entity_map::EntityRef; +use packed_option::ReservedValue; use std::default::Default; use std::fmt::{self, Display, Formatter}; use std::u32; +// Implement the common traits for a 32-bit entity reference. +macro_rules! entity_impl { + // Basic traits. + ($entity:ident) => { + impl EntityRef for $entity { + fn new(index: usize) -> Self { + assert!(index < (u32::MAX as usize)); + $entity(index as u32) + } + + fn index(self) -> usize { + self.0 as usize + } + } + + impl ReservedValue for $entity { + fn reserved_value() -> $entity { + $entity(u32::MAX) + } + } + }; + + // Include basic `Display` impl using the given display prefix. + // Display an `Ebb` reference as "ebb12". + ($entity:ident, $display_prefix:expr) => { + entity_impl!($entity); + + impl Display for $entity { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write!(fmt, "{}{}", $display_prefix, self.0) + } + } + } +} + /// An opaque reference to an extended basic block in a function. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] pub struct Ebb(u32); - -impl EntityRef for Ebb { - fn new(index: usize) -> Self { - assert!(index < (u32::MAX as usize)); - Ebb(index as u32) - } - - fn index(self) -> usize { - self.0 as usize - } -} +entity_impl!(Ebb, "ebb"); impl Ebb { /// Create a new EBB reference from its number. This corresponds to the ebbNN representation. @@ -46,13 +72,6 @@ impl Ebb { } } -/// Display an `Ebb` reference as "ebb12". -impl Display for Ebb { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write!(fmt, "ebb{}", self.0) - } -} - /// A guaranteed invalid EBB reference. pub const NO_EBB: Ebb = Ebb(u32::MAX); @@ -65,24 +84,7 @@ impl Default for Ebb { /// An opaque reference to an instruction in a function. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] pub struct Inst(u32); - -impl EntityRef for Inst { - fn new(index: usize) -> Self { - assert!(index < (u32::MAX as usize)); - Inst(index as u32) - } - - fn index(self) -> usize { - self.0 as usize - } -} - -/// Display an `Inst` reference as "inst7". -impl Display for Inst { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write!(fmt, "inst{}", self.0) - } -} +entity_impl!(Inst, "inst"); /// A guaranteed invalid instruction reference. pub const NO_INST: Inst = Inst(u32::MAX); @@ -97,17 +99,7 @@ impl Default for Inst { /// An opaque reference to an SSA value. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct Value(u32); - -impl EntityRef for Value { - fn new(index: usize) -> Value { - assert!(index < (u32::MAX as usize)); - Value(index as u32) - } - - fn index(self) -> usize { - self.0 as usize - } -} +entity_impl!(Value); /// Value references can either reference an instruction directly, or they can refer to the /// extended value table. @@ -214,24 +206,7 @@ impl Default for Value { /// An opaque reference to a stack slot. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct StackSlot(u32); - -impl EntityRef for StackSlot { - fn new(index: usize) -> StackSlot { - assert!(index < (u32::MAX as usize)); - StackSlot(index as u32) - } - - fn index(self) -> usize { - self.0 as usize - } -} - -/// Display a `StackSlot` reference as "ss12". -impl Display for StackSlot { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write!(fmt, "ss{}", self.0) - } -} +entity_impl!(StackSlot, "ss"); /// A guaranteed invalid stack slot reference. pub const NO_STACK_SLOT: StackSlot = StackSlot(u32::MAX); @@ -245,24 +220,7 @@ impl Default for StackSlot { /// An opaque reference to a jump table. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct JumpTable(u32); - -impl EntityRef for JumpTable { - fn new(index: usize) -> JumpTable { - assert!(index < (u32::MAX as usize)); - JumpTable(index as u32) - } - - fn index(self) -> usize { - self.0 as usize - } -} - -/// Display a `JumpTable` reference as "jt12". -impl Display for JumpTable { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write!(fmt, "jt{}", self.0) - } -} +entity_impl!(JumpTable, "jt"); /// A guaranteed invalid jump table reference. pub const NO_JUMP_TABLE: JumpTable = JumpTable(u32::MAX); @@ -276,24 +234,7 @@ impl Default for JumpTable { /// A reference to an external function. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct FuncRef(u32); - -impl EntityRef for FuncRef { - fn new(index: usize) -> FuncRef { - assert!(index < (u32::MAX as usize)); - FuncRef(index as u32) - } - - fn index(self) -> usize { - self.0 as usize - } -} - -/// Display a `FuncRef` reference as "fn12". -impl Display for FuncRef { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write!(fmt, "fn{}", self.0) - } -} +entity_impl!(FuncRef, "fn"); /// A guaranteed invalid function reference. pub const NO_FUNC_REF: FuncRef = FuncRef(u32::MAX); @@ -307,24 +248,7 @@ impl Default for FuncRef { /// A reference to a function signature. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct SigRef(u32); - -impl EntityRef for SigRef { - fn new(index: usize) -> SigRef { - assert!(index < (u32::MAX as usize)); - SigRef(index as u32) - } - - fn index(self) -> usize { - self.0 as usize - } -} - -/// Display a `SigRef` reference as "sig12". -impl Display for SigRef { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write!(fmt, "sig{}", self.0) - } -} +entity_impl!(SigRef, "sig"); /// A guaranteed invalid function reference. pub const NO_SIG_REF: SigRef = SigRef(u32::MAX); From 7b8239d0760ffe29da088ec5b171434f15f75e9d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Jan 2017 13:15:50 -0800 Subject: [PATCH 0472/3084] Remove Default implementations from many entity references. These types can be wrapped in a PackedOption now, so we don't need the NO_* constants and default values. --- lib/cretonne/src/ir/entities.rs | 36 --------------------------------- 1 file changed, 36 deletions(-) diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index ed42b33633..0b01fa9a79 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -208,57 +208,21 @@ impl Default for Value { pub struct StackSlot(u32); entity_impl!(StackSlot, "ss"); -/// A guaranteed invalid stack slot reference. -pub const NO_STACK_SLOT: StackSlot = StackSlot(u32::MAX); - -impl Default for StackSlot { - fn default() -> StackSlot { - NO_STACK_SLOT - } -} - /// An opaque reference to a jump table. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct JumpTable(u32); entity_impl!(JumpTable, "jt"); -/// A guaranteed invalid jump table reference. -pub const NO_JUMP_TABLE: JumpTable = JumpTable(u32::MAX); - -impl Default for JumpTable { - fn default() -> JumpTable { - NO_JUMP_TABLE - } -} - /// A reference to an external function. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct FuncRef(u32); entity_impl!(FuncRef, "fn"); -/// A guaranteed invalid function reference. -pub const NO_FUNC_REF: FuncRef = FuncRef(u32::MAX); - -impl Default for FuncRef { - fn default() -> FuncRef { - NO_FUNC_REF - } -} - /// A reference to a function signature. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct SigRef(u32); entity_impl!(SigRef, "sig"); -/// A guaranteed invalid function reference. -pub const NO_SIG_REF: SigRef = SigRef(u32::MAX); - -impl Default for SigRef { - fn default() -> SigRef { - NO_SIG_REF - } -} - /// A reference to any of the entities defined in this module. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub enum AnyEntity { From 2f6a33f16d89ada671659d3d66b2e1aae47a95ae Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Jan 2017 13:41:56 -0800 Subject: [PATCH 0473/3084] Use PackedOption to represent jump tables. Avoid NO_EBB. --- lib/cretonne/src/ir/jumptable.rs | 51 +++++++++++++------------------ lib/cretonne/src/packed_option.rs | 5 +++ lib/reader/src/parser.rs | 10 +++--- 3 files changed, 33 insertions(+), 33 deletions(-) diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index cbd8630e46..824e2dcf23 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -3,7 +3,8 @@ //! Jump tables are declared in the preamble and assigned an `ir::entities::JumpTable` reference. //! The actual table of destinations is stored in a `JumpTableData` struct defined in this module. -use ir::entities::{Ebb, NO_EBB}; +use packed_option::PackedOption; +use ir::entities::Ebb; use std::iter; use std::slice; use std::fmt::{self, Display, Formatter}; @@ -14,10 +15,10 @@ use std::fmt::{self, Display, Formatter}; /// to be completely populated, though. Individual entries can be missing. #[derive(Clone)] pub struct JumpTableData { - // Table entries, using NO_EBB as a placeholder for missing entries. - table: Vec, + // Table entries, using `None` as a placeholder for missing entries. + table: Vec>, - // How many `NO_EBB` holes in table? + // How many `None` holes in table? holes: usize, } @@ -34,16 +35,15 @@ impl JumpTableData { /// /// The table will grow as needed to fit 'idx'. pub fn set_entry(&mut self, idx: usize, dest: Ebb) { - assert!(dest != NO_EBB); // Resize table to fit `idx`. if idx >= self.table.len() { self.holes += idx - self.table.len(); - self.table.resize(idx + 1, NO_EBB); - } else if self.table[idx] == NO_EBB { + self.table.resize(idx + 1, None.into()); + } else if self.table[idx].is_none() { // We're filling in an existing hole. self.holes -= 1; } - self.table[idx] = dest; + self.table[idx] = dest.into(); } /// Clear a table entry. @@ -51,19 +51,15 @@ impl JumpTableData { /// The `br_table` instruction will fall through if given an index corresponding to a cleared /// table entry. pub fn clear_entry(&mut self, idx: usize) { - if idx < self.table.len() && self.table[idx] != NO_EBB { + if idx < self.table.len() && self.table[idx].is_some() { self.holes += 1; - self.table[idx] = NO_EBB; + self.table[idx] = None.into(); } } /// Get the entry for `idx`, or `None`. pub fn get_entry(&self, idx: usize) -> Option { - if idx < self.table.len() && self.table[idx] != NO_EBB { - Some(self.table[idx]) - } else { - None - } + self.table.get(idx).and_then(|e| e.expand()) } /// Enumerate over all `(idx, dest)` pairs in the table in order. @@ -74,13 +70,13 @@ impl JumpTableData { } /// Access the whole table as a mutable slice. - pub fn as_mut_slice(&mut self) -> &mut [Ebb] { + pub fn as_mut_slice(&mut self) -> &mut [PackedOption] { self.table.as_mut_slice() } } /// Enumerate `(idx, dest)` pairs in order. -pub struct Entries<'a>(iter::Enumerate>>); +pub struct Entries<'a>(iter::Enumerate>>>); impl<'a> Iterator for Entries<'a> { type Item = (usize, Ebb); @@ -88,8 +84,8 @@ impl<'a> Iterator for Entries<'a> { fn next(&mut self) -> Option { loop { if let Some((idx, dest)) = self.0.next() { - if dest != NO_EBB { - return Some((idx, dest)); + if let Some(ebb) = dest.expand() { + return Some((idx, ebb)); } } else { return None; @@ -100,18 +96,15 @@ impl<'a> Iterator for Entries<'a> { impl Display for JumpTableData { fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - let first = self.table.first().cloned().unwrap_or_default(); - if first == NO_EBB { - try!(write!(fmt, "jump_table 0")); - } else { - try!(write!(fmt, "jump_table {}", first)); + match self.table.first().and_then(|e| e.expand()) { + None => try!(write!(fmt, "jump_table 0")), + Some(first) => try!(write!(fmt, "jump_table {}", first)), } - for dest in self.table.iter().cloned().skip(1) { - if dest == NO_EBB { - try!(write!(fmt, ", 0")); - } else { - try!(write!(fmt, ", {}", dest)); + for dest in self.table.iter().skip(1).map(|e| e.expand()) { + match dest { + None => try!(write!(fmt, ", 0")), + Some(ebb) => try!(write!(fmt, ", {}", ebb)), } } Ok(()) diff --git a/lib/cretonne/src/packed_option.rs b/lib/cretonne/src/packed_option.rs index 2c93b38ead..3bc7a3ed5f 100644 --- a/lib/cretonne/src/packed_option.rs +++ b/lib/cretonne/src/packed_option.rs @@ -25,6 +25,11 @@ impl PackedOption { self.0 == T::reserved_value() } + /// Returns `true` if the packed option is a `Some` value. + pub fn is_some(&self) -> bool { + !self.is_none() + } + /// Expand the packed option into a normal `Option`. pub fn expand(self) -> Option { if self.is_none() { None } else { Some(self.0) } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index c513e1a86a..f73bfc7a3c 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -13,7 +13,7 @@ use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotDa FuncRef}; use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; -use cretonne::ir::entities::{AnyEntity, NO_EBB, NO_INST, NO_VALUE}; +use cretonne::ir::entities::{AnyEntity, NO_INST, NO_VALUE}; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, TernaryOverflowData, JumpData, BranchData, CallData, IndirectCallData, ReturnData}; @@ -204,9 +204,11 @@ impl Context { // Rewrite EBB references in jump tables. for jt in self.function.jump_tables.keys() { let loc = jt.into(); - for ebb in self.function.jump_tables[jt].as_mut_slice() { - if *ebb != NO_EBB { - try!(self.map.rewrite_ebb(ebb, loc)); + for ebb_ref in self.function.jump_tables[jt].as_mut_slice() { + if let Some(mut ebb) = ebb_ref.expand() { + try!(self.map.rewrite_ebb(&mut ebb, loc)); + // Convert back to a packed option. + *ebb_ref = ebb.into(); } } } From b8200d7be9a8a5ca34b1b9558cd89aeacea50c2d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Jan 2017 13:59:57 -0800 Subject: [PATCH 0474/3084] Use PackedOption in the Layout implementation. The doubly linked lists are terminated with None. Remove NO_EBB and the Default impl for Ebb. --- lib/cretonne/src/ir/entities.rs | 9 ---- lib/cretonne/src/ir/layout.rs | 75 +++++++++++++++---------------- lib/cretonne/src/packed_option.rs | 12 +++++ 3 files changed, 49 insertions(+), 47 deletions(-) diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 0b01fa9a79..119873777b 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -72,15 +72,6 @@ impl Ebb { } } -/// A guaranteed invalid EBB reference. -pub const NO_EBB: Ebb = Ebb(u32::MAX); - -impl Default for Ebb { - fn default() -> Ebb { - NO_EBB - } -} - /// An opaque reference to an instruction in a function. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] pub struct Inst(u32); diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 7f2497df1b..2092329fba 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -6,7 +6,8 @@ use std::cmp; use std::iter::{Iterator, IntoIterator}; use entity_map::{EntityMap, EntityRef}; -use ir::entities::{Ebb, NO_EBB, Inst, NO_INST}; +use packed_option::PackedOption; +use ir::entities::{Ebb, Inst, NO_INST}; use ir::progpoint::{ProgramOrder, ExpandedProgramPoint}; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not @@ -25,7 +26,7 @@ use ir::progpoint::{ProgramOrder, ExpandedProgramPoint}; #[derive(Clone)] pub struct Layout { // Linked list nodes for the layout order of EBBs Forms a doubly linked list, terminated in - // both ends by NO_EBB. + // both ends by `None`. ebbs: EntityMap, // Linked list nodes for the layout order of instructions. Forms a double linked list per EBB, @@ -130,13 +131,12 @@ impl Layout { assert!(self.is_ebb_inserted(ebb)); // Get the sequence number immediately before `ebb`, or 0. - let prev_seq = - self.ebbs[ebb].prev.wrap().map(|prev_ebb| self.last_ebb_seq(prev_ebb)).unwrap_or(0); + let prev_seq = self.ebbs[ebb].prev.map(|prev_ebb| self.last_ebb_seq(prev_ebb)).unwrap_or(0); // Get the sequence number immediately following `ebb`. let next_seq = if let Some(inst) = self.ebbs[ebb].first_inst.wrap() { self.insts[inst].seq - } else if let Some(next_ebb) = self.ebbs[ebb].next.wrap() { + } else if let Some(next_ebb) = self.ebbs[ebb].next.expand() { self.ebbs[next_ebb].seq } else { // There is nothing after `ebb`. We can just use a major stride. @@ -167,7 +167,7 @@ impl Layout { // Get the sequence number immediately following `inst`. let next_seq = if let Some(next_inst) = self.insts[inst].next.wrap() { self.insts[next_inst].seq - } else if let Some(next_ebb) = self.ebbs[ebb].next.wrap() { + } else if let Some(next_ebb) = self.ebbs[ebb].next.expand() { self.ebbs[next_ebb].seq } else { // There is nothing after `inst`. We can just use a major stride. @@ -229,7 +229,7 @@ impl Layout { } // Advance to the next EBB. - ebb = match self.ebbs[ebb].next.wrap() { + ebb = match self.ebbs[ebb].next.expand() { Some(next) => next, None => return, }; @@ -248,7 +248,7 @@ impl Layout { fn renumber_from_inst(&mut self, inst: Inst, first_seq: SequenceNumber) { if let Some(seq) = self.renumber_insts(inst, first_seq) { // Renumbering spills over into next EBB. - if let Some(next_ebb) = self.ebbs[self.inst_ebb(inst).unwrap()].next.wrap() { + if let Some(next_ebb) = self.ebbs[self.inst_ebb(inst).unwrap()].next.expand() { self.renumber_from_ebb(next_ebb, seq + MINOR_STRIDE); } } @@ -267,7 +267,7 @@ impl Layout { impl Layout { /// Is `ebb` currently part of the layout? pub fn is_ebb_inserted(&self, ebb: Ebb) -> bool { - Some(ebb) == self.first_ebb || (self.ebbs.is_valid(ebb) && self.ebbs[ebb].prev != NO_EBB) + Some(ebb) == self.first_ebb || (self.ebbs.is_valid(ebb) && self.ebbs[ebb].prev.is_some()) } /// Insert `ebb` as the last EBB in the layout. @@ -277,11 +277,11 @@ impl Layout { { let node = self.ebbs.ensure(ebb); assert!(node.first_inst == NO_INST && node.last_inst == NO_INST); - node.prev = self.last_ebb.unwrap_or_default(); - node.next = NO_EBB; + node.prev = self.last_ebb.into(); + node.next = None.into(); } if let Some(last) = self.last_ebb { - self.ebbs[last].next = ebb; + self.ebbs[last].next = ebb.into(); } else { self.first_ebb = Some(ebb); } @@ -298,14 +298,13 @@ impl Layout { let after = self.ebbs[before].prev; { let node = self.ebbs.ensure(ebb); - node.next = before; + node.next = before.into(); node.prev = after; } - self.ebbs[before].prev = ebb; - if after == NO_EBB { - self.first_ebb = Some(ebb); - } else { - self.ebbs[after].next = ebb; + self.ebbs[before].prev = ebb.into(); + match after.expand() { + None => self.first_ebb = Some(ebb), + Some(a) => self.ebbs[a].next = ebb.into(), } self.assign_ebb_seq(ebb); } @@ -320,13 +319,12 @@ impl Layout { { let node = self.ebbs.ensure(ebb); node.next = before; - node.prev = after; + node.prev = after.into(); } - self.ebbs[after].next = ebb; - if before == NO_EBB { - self.last_ebb = Some(ebb); - } else { - self.ebbs[before].prev = ebb; + self.ebbs[after].next = ebb.into(); + match before.expand() { + None => self.last_ebb = Some(ebb), + Some(b) => self.ebbs[b].prev = ebb.into(), } self.assign_ebb_seq(ebb); } @@ -348,8 +346,8 @@ impl Layout { #[derive(Clone, Debug, Default)] struct EbbNode { - prev: Ebb, - next: Ebb, + prev: PackedOption, + next: PackedOption, first_inst: Inst, last_inst: Inst, seq: SequenceNumber, @@ -367,7 +365,7 @@ impl<'f> Iterator for Ebbs<'f> { fn next(&mut self) -> Option { match self.next { Some(ebb) => { - self.next = self.layout.ebbs[ebb].next.wrap(); + self.next = self.layout.ebbs[ebb].next.expand(); Some(ebb) } None => None, @@ -393,7 +391,7 @@ impl Layout { /// Get the EBB containing `inst`, or `None` if `inst` is not inserted in the layout. pub fn inst_ebb(&self, inst: Inst) -> Option { if self.insts.is_valid(inst) { - self.insts[inst].ebb.wrap() + self.insts[inst].ebb.into() } else { None } @@ -408,7 +406,7 @@ impl Layout { let ebb_node = &mut self.ebbs[ebb]; { let inst_node = self.insts.ensure(inst); - inst_node.ebb = ebb; + inst_node.ebb = ebb.into(); inst_node.prev = ebb_node.last_inst; assert_eq!(inst_node.next, NO_INST); } @@ -435,7 +433,7 @@ impl Layout { let after = self.insts[before].prev; { let inst_node = self.insts.ensure(inst); - inst_node.ebb = ebb; + inst_node.ebb = ebb.into(); inst_node.next = before; inst_node.prev = after; } @@ -489,18 +487,18 @@ impl Layout { let last_inst = self.ebbs[old_ebb].last_inst; { let node = self.ebbs.ensure(new_ebb); - node.prev = old_ebb; + node.prev = old_ebb.into(); node.next = next_ebb; node.first_inst = before; node.last_inst = last_inst; } - self.ebbs[old_ebb].next = new_ebb; + self.ebbs[old_ebb].next = new_ebb.into(); // Fix backwards link. if Some(old_ebb) == self.last_ebb { self.last_ebb = Some(new_ebb); } else { - self.ebbs[next_ebb].prev = new_ebb; + self.ebbs[next_ebb.unwrap()].prev = new_ebb.into(); } // Disconnect the instruction links. @@ -516,8 +514,8 @@ impl Layout { // Fix the instruction -> ebb pointers. let mut i = before; while i != NO_INST { - debug_assert_eq!(self.insts[i].ebb, old_ebb); - self.insts[i].ebb = new_ebb; + debug_assert_eq!(self.insts[i].ebb.expand(), Some(old_ebb)); + self.insts[i].ebb = new_ebb.into(); i = self.insts[i].next; } @@ -527,7 +525,8 @@ impl Layout { #[derive(Clone, Debug, Default)] struct InstNode { - ebb: Ebb, + // The Ebb containing this instruction, or `None` if the instruction is not yet inserted. + ebb: PackedOption, prev: Inst, next: Inst, seq: SequenceNumber, @@ -686,7 +685,7 @@ impl<'f> Cursor<'f> { /// ``` pub fn next_ebb(&mut self) -> Option { let next = if let Some(ebb) = self.current_ebb() { - self.layout.ebbs[ebb].next.wrap() + self.layout.ebbs[ebb].next.expand() } else { self.layout.first_ebb }; @@ -719,7 +718,7 @@ impl<'f> Cursor<'f> { /// ``` pub fn prev_ebb(&mut self) -> Option { let prev = if let Some(ebb) = self.current_ebb() { - self.layout.ebbs[ebb].prev.wrap() + self.layout.ebbs[ebb].prev.expand() } else { self.layout.last_ebb }; diff --git a/lib/cretonne/src/packed_option.rs b/lib/cretonne/src/packed_option.rs index 3bc7a3ed5f..a36676a4b4 100644 --- a/lib/cretonne/src/packed_option.rs +++ b/lib/cretonne/src/packed_option.rs @@ -34,6 +34,18 @@ impl PackedOption { pub fn expand(self) -> Option { if self.is_none() { None } else { Some(self.0) } } + + /// Maps a `PackedOption` to `Option` by applying a function to a contained value. + pub fn map(self, f: F) -> Option + where F: FnOnce(T) -> U + { + self.expand().map(f) + } + + /// Unwrap a packed `Some` value or panic. + pub fn unwrap(self) -> T { + self.expand().unwrap() + } } impl Default for PackedOption { From bdc0f53c91515250590c35066ae52cfc3521b36b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Jan 2017 14:26:47 -0800 Subject: [PATCH 0475/3084] Use PackedOption in the Layout implementation. This also revealed that the last_inst() method should return an Option. --- lib/cretonne/src/ir/layout.rs | 85 +++++++++++++++++------------------ lib/cretonne/src/verifier.rs | 2 +- 2 files changed, 42 insertions(+), 45 deletions(-) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 2092329fba..2a8d1ba4ac 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -5,9 +5,9 @@ use std::cmp; use std::iter::{Iterator, IntoIterator}; -use entity_map::{EntityMap, EntityRef}; +use entity_map::EntityMap; use packed_option::PackedOption; -use ir::entities::{Ebb, Inst, NO_INST}; +use ir::entities::{Ebb, Inst}; use ir::progpoint::{ProgramOrder, ExpandedProgramPoint}; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not @@ -30,7 +30,7 @@ pub struct Layout { ebbs: EntityMap, // Linked list nodes for the layout order of instructions. Forms a double linked list per EBB, - // terminated in both ends by NO_INST. + // terminated in both ends by `None`. insts: EntityMap, // First EBB in the layout order, or `None` when no EBBs have been laid out. @@ -120,7 +120,6 @@ impl Layout { // Get the seq of the last instruction if it exists, otherwise use the EBB header seq. self.ebbs[ebb] .last_inst - .wrap() .map(|inst| self.insts[inst].seq) .unwrap_or(self.ebbs[ebb].seq) } @@ -134,7 +133,7 @@ impl Layout { let prev_seq = self.ebbs[ebb].prev.map(|prev_ebb| self.last_ebb_seq(prev_ebb)).unwrap_or(0); // Get the sequence number immediately following `ebb`. - let next_seq = if let Some(inst) = self.ebbs[ebb].first_inst.wrap() { + let next_seq = if let Some(inst) = self.ebbs[ebb].first_inst.expand() { self.insts[inst].seq } else if let Some(next_ebb) = self.ebbs[ebb].next.expand() { self.ebbs[next_ebb].seq @@ -159,13 +158,13 @@ impl Layout { let ebb = self.inst_ebb(inst).expect("inst must be inserted before assigning an seq"); // Get the sequence number immediately before `inst`. - let prev_seq = match self.insts[inst].prev.wrap() { + let prev_seq = match self.insts[inst].prev.expand() { Some(prev_inst) => self.insts[prev_inst].seq, None => self.ebbs[ebb].seq, }; // Get the sequence number immediately following `inst`. - let next_seq = if let Some(next_inst) = self.insts[inst].next.wrap() { + let next_seq = if let Some(next_inst) = self.insts[inst].next.expand() { self.insts[next_inst].seq } else if let Some(next_ebb) = self.ebbs[ebb].next.expand() { self.ebbs[next_ebb].seq @@ -197,7 +196,7 @@ impl Layout { self.insts[inst].seq = seq; // Next instruction. - inst = match self.insts[inst].next.wrap() { + inst = match self.insts[inst].next.expand() { None => return Some(seq), Some(next) => next, }; @@ -221,7 +220,7 @@ impl Layout { self.ebbs[ebb].seq = seq; // Renumber instructions in `ebb`. Stop when the numbers catch up. - if let Some(inst) = self.ebbs[ebb].first_inst.wrap() { + if let Some(inst) = self.ebbs[ebb].first_inst.expand() { seq = match self.renumber_insts(inst, seq + MINOR_STRIDE) { Some(s) => s, None => return, @@ -276,7 +275,7 @@ impl Layout { "Cannot append EBB that is already in the layout"); { let node = self.ebbs.ensure(ebb); - assert!(node.first_inst == NO_INST && node.last_inst == NO_INST); + assert!(node.first_inst.is_none() && node.last_inst.is_none()); node.prev = self.last_ebb.into(); node.next = None.into(); } @@ -348,8 +347,8 @@ impl Layout { struct EbbNode { prev: PackedOption, next: PackedOption, - first_inst: Inst, - last_inst: Inst, + first_inst: PackedOption, + last_inst: PackedOption, seq: SequenceNumber, } @@ -408,21 +407,21 @@ impl Layout { let inst_node = self.insts.ensure(inst); inst_node.ebb = ebb.into(); inst_node.prev = ebb_node.last_inst; - assert_eq!(inst_node.next, NO_INST); + assert!(inst_node.next.is_none()); } - if ebb_node.first_inst == NO_INST { - ebb_node.first_inst = inst; + if ebb_node.first_inst.is_none() { + ebb_node.first_inst = inst.into(); } else { - self.insts[ebb_node.last_inst].next = inst; + self.insts[ebb_node.last_inst.unwrap()].next = inst.into(); } - ebb_node.last_inst = inst; + ebb_node.last_inst = inst.into(); } self.assign_inst_seq(inst); } /// Fetch an ebb's last instruction. - pub fn last_inst(&self, ebb: Ebb) -> Inst { - self.ebbs[ebb].last_inst + pub fn last_inst(&self, ebb: Ebb) -> Option { + self.ebbs[ebb].last_inst.into() } /// Insert `inst` before the instruction `before` in the same EBB. @@ -434,14 +433,13 @@ impl Layout { { let inst_node = self.insts.ensure(inst); inst_node.ebb = ebb.into(); - inst_node.next = before; + inst_node.next = before.into(); inst_node.prev = after; } - self.insts[before].prev = inst; - if after == NO_INST { - self.ebbs[ebb].first_inst = inst; - } else { - self.insts[after].next = inst; + self.insts[before].prev = inst.into(); + match after.expand() { + None => self.ebbs[ebb].first_inst = inst.into(), + Some(a) => self.insts[a].next = inst.into(), } self.assign_inst_seq(inst); } @@ -450,8 +448,8 @@ impl Layout { pub fn ebb_insts<'f>(&'f self, ebb: Ebb) -> Insts<'f> { Insts { layout: self, - head: self.ebbs[ebb].first_inst.wrap(), - tail: self.ebbs[ebb].last_inst.wrap(), + head: self.ebbs[ebb].first_inst.into(), + tail: self.ebbs[ebb].last_inst.into(), } } @@ -489,7 +487,7 @@ impl Layout { let node = self.ebbs.ensure(new_ebb); node.prev = old_ebb.into(); node.next = next_ebb; - node.first_inst = before; + node.first_inst = before.into(); node.last_inst = last_inst; } self.ebbs[old_ebb].next = new_ebb.into(); @@ -503,20 +501,19 @@ impl Layout { // Disconnect the instruction links. let prev_inst = self.insts[before].prev; - self.insts[before].prev = NO_INST; + self.insts[before].prev = None.into(); self.ebbs[old_ebb].last_inst = prev_inst; - if prev_inst == NO_INST { - self.ebbs[old_ebb].first_inst = NO_INST; - } else { - self.insts[prev_inst].next = NO_INST; + match prev_inst.expand() { + None => self.ebbs[old_ebb].first_inst = None.into(), + Some(pi) => self.insts[pi].next = None.into(), } // Fix the instruction -> ebb pointers. - let mut i = before; - while i != NO_INST { + let mut opt_i = Some(before); + while let Some(i) = opt_i { debug_assert_eq!(self.insts[i].ebb.expand(), Some(old_ebb)); self.insts[i].ebb = new_ebb.into(); - i = self.insts[i].next; + opt_i = self.insts[i].next.into(); } self.assign_ebb_seq(new_ebb); @@ -527,8 +524,8 @@ impl Layout { struct InstNode { // The Ebb containing this instruction, or `None` if the instruction is not yet inserted. ebb: PackedOption, - prev: Inst, - next: Inst, + prev: PackedOption, + next: PackedOption, seq: SequenceNumber, } @@ -549,7 +546,7 @@ impl<'f> Iterator for Insts<'f> { self.head = None; self.tail = None; } else { - self.head = Some(self.layout.insts[inst].next); + self.head = self.layout.insts[inst].next.into(); } } rval @@ -564,7 +561,7 @@ impl<'f> DoubleEndedIterator for Insts<'f> { self.head = None; self.tail = None; } else { - self.tail = Some(self.layout.insts[inst].prev); + self.tail = self.layout.insts[inst].prev.into(); } } rval @@ -775,7 +772,7 @@ impl<'f> Cursor<'f> { match self.pos { Nowhere | After(..) => None, At(inst) => { - if let Some(next) = self.layout.insts[inst].next.wrap() { + if let Some(next) = self.layout.insts[inst].next.expand() { self.pos = At(next); Some(next) } else { @@ -785,7 +782,7 @@ impl<'f> Cursor<'f> { } } Before(ebb) => { - if let Some(next) = self.layout.ebbs[ebb].first_inst.wrap() { + if let Some(next) = self.layout.ebbs[ebb].first_inst.expand() { self.pos = At(next); Some(next) } else { @@ -826,7 +823,7 @@ impl<'f> Cursor<'f> { match self.pos { Nowhere | Before(..) => None, At(inst) => { - if let Some(prev) = self.layout.insts[inst].prev.wrap() { + if let Some(prev) = self.layout.insts[inst].prev.expand() { self.pos = At(prev); Some(prev) } else { @@ -836,7 +833,7 @@ impl<'f> Cursor<'f> { } } After(ebb) => { - if let Some(prev) = self.layout.ebbs[ebb].last_inst.wrap() { + if let Some(prev) = self.layout.ebbs[ebb].last_inst.expand() { self.pos = At(prev); Some(prev) } else { diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 939d722e60..f0a4b66b32 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -110,7 +110,7 @@ impl<'a> Verifier<'a> { fn ebb_integrity(&self, ebb: Ebb, inst: Inst) -> Result<()> { let is_terminator = self.func.dfg[inst].opcode().is_terminator(); - let is_last_inst = self.func.layout.last_inst(ebb) == inst; + let is_last_inst = self.func.layout.last_inst(ebb) == Some(inst); if is_terminator && !is_last_inst { // Terminating instructions only occur at the end of blocks. From 1389a51c7a002f2ea231a382b85f40fbd3568097 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Jan 2017 14:49:55 -0800 Subject: [PATCH 0476/3084] Avoid using NO_INST in the parser. This was only used for the comment rewrite mechanism, and we can just predict the next allocated instruction number instead. See the other uses of next_key() for gather_comments(). --- lib/cretonne/src/ir/dfg.rs | 8 ++++++++ lib/reader/src/parser.rs | 22 +++------------------- 2 files changed, 11 insertions(+), 19 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 7e843d0c87..7ab9ee24d8 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -301,6 +301,14 @@ impl DataFlowGraph { self.insts.push(data) } + /// Get the instruction reference that will be assigned to the next instruction created by + /// `make_inst`. + /// + /// This is only really useful to the parser. + pub fn next_inst(&self) -> Inst { + self.insts.next_key() + } + /// Create result values for an instruction that produces multiple results. /// /// Instructions that produce 0 or 1 result values only need to be created with `make_inst`. If diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index f73bfc7a3c..2d1d85f93d 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -13,7 +13,7 @@ use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotDa FuncRef}; use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; -use cretonne::ir::entities::{AnyEntity, NO_INST, NO_VALUE}; +use cretonne::ir::entities::{AnyEntity, NO_VALUE}; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, TernaryOverflowData, JumpData, BranchData, CallData, IndirectCallData, ReturnData}; @@ -283,17 +283,6 @@ impl<'a> Parser<'a> { mem::replace(&mut self.comments, Vec::new()) } - // Rewrite the entity of the last added comments from `old` to `new`. - // Also switch to collecting future comments for `new`. - fn rewrite_last_comment_entities>(&mut self, old: E, new: E) { - let old = old.into(); - let new = new.into(); - for comment in (&mut self.comments).into_iter().rev().take_while(|c| c.entity == old) { - comment.entity = new; - } - self.comment_entity = Some(new); - } - // Match and consume a token without payload. fn match_token(&mut self, want: Token<'a>, err_msg: &str) -> Result> { if self.token() == Some(want) { @@ -906,9 +895,8 @@ impl<'a> Parser<'a> { // inst-results ::= Value(v) { "," Value(vx) } // fn parse_instruction(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { - // Collect comments for `NO_INST` while parsing the instruction, then rewrite after we - // allocate an instruction number. - self.gather_comments(NO_INST); + // Collect comments for the next instruction to be allocated. + self.gather_comments(ctx.function.dfg.next_inst()); // Result value numbers. let mut results = Vec::new(); @@ -969,10 +957,6 @@ impl<'a> Parser<'a> { results.len()); } - // If we saw any comments while parsing the instruction, they will have been recorded as - // belonging to `NO_INST`. - self.rewrite_last_comment_entities(NO_INST, inst); - // Now map the source result values to the just created instruction results. // Pass a reference to `ctx.values` instead of `ctx` itself since the `Values` iterator // holds a reference to `ctx.function`. From 0d77b1970831b28bd6c9cd1522a859113a341085 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Jan 2017 15:52:29 -0800 Subject: [PATCH 0477/3084] Use PackedOption instead of NO_VALUE. - Remove NO_VALUE and ExpandedValue::None. - Remove the Default implelmentation for Value. - InstructionData::second_result() returns an Option. - InstructionData::second_result() returns a reference to the packed option. --- lib/cretonne/meta/gen_instr.py | 9 +- lib/cretonne/src/ir/builder.rs | 10 +- lib/cretonne/src/ir/dfg.rs | 136 ++++++++++++++-------------- lib/cretonne/src/ir/entities.rs | 17 +--- lib/cretonne/src/ir/instructions.rs | 11 ++- lib/cretonne/src/packed_option.rs | 6 ++ lib/reader/src/parser.rs | 12 +-- 7 files changed, 97 insertions(+), 104 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index ac4650bc96..d05bef96df 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -112,7 +112,8 @@ def gen_instruction_data_impl(fmt): - `pub fn opcode(&self) -> Opcode` - `pub fn first_type(&self) -> Type` - `pub fn second_result(&self) -> Option` - - `pub fn second_result_mut<'a>(&'a mut self) -> Option<&'a mut Value>` + - `pub fn second_result_mut<'a>(&'a mut self) + -> Option<&'a mut PackedOption>` - `pub fn arguments(&self) -> (&[Value], &[Value])` """ @@ -157,7 +158,7 @@ def gen_instruction_data_impl(fmt): fmt.line( 'InstructionData::' + f.name + ' { second_result, .. }' + - ' => Some(second_result),') + ' => second_result.into(),') else: # Single or no results. fmt.line( @@ -167,7 +168,7 @@ def gen_instruction_data_impl(fmt): fmt.doc_comment('Mutable reference to second result value, if any.') with fmt.indented( "pub fn second_result_mut<'a>(&'a mut self)" + - " -> Option<&'a mut Value> {", '}'): + " -> Option<&'a mut PackedOption> {", '}'): with fmt.indented('match *self {', '}'): for f in InstructionFormat.all_formats: if f.multiple_results: @@ -489,7 +490,7 @@ def gen_format_constructor(iform, fmt): fmt.line('opcode: opcode,') fmt.line('ty: {},'.format(result_type)) if iform.multiple_results: - fmt.line('second_result: Value::default(),') + fmt.line('second_result: None.into(),') if iform.boxed_storage: with fmt.indented( 'data: Box::new(instructions::{}Data {{' diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 695bc8e5a3..ad3f8e22fe 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -132,9 +132,9 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { fn simple_instruction(self, data: InstructionData) -> (Inst, &'f mut DataFlowGraph) { // The replacement instruction cannot generate multiple results, so verify that the old // instruction's secondary results have been detached. - let old_second_value = self.dfg[self.inst].second_result().unwrap_or_default(); + let old_second_value = self.dfg[self.inst].second_result(); assert_eq!(old_second_value, - Value::default(), + None, "Secondary result values {:?} would be left dangling by replacing {} with {}", self.dfg.inst_results(self.inst).collect::>(), self.dfg[self.inst].opcode(), @@ -150,12 +150,12 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { ctrl_typevar: Type) -> (Inst, &'f mut DataFlowGraph) { // If the old instruction still has secondary results attached, we'll keep them. - let old_second_value = self.dfg[self.inst].second_result().unwrap_or_default(); + let old_second_value = self.dfg[self.inst].second_result(); // Splat the new instruction on top of the old one. self.dfg[self.inst] = data; - if old_second_value == Value::default() { + if old_second_value.is_none() { // The old secondary values were either detached or non-existent. // Construct new ones and set the first result type too. self.dfg.make_inst_results(self.inst, ctrl_typevar); @@ -163,7 +163,7 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { // Reattach the old secondary values. if let Some(val_ref) = self.dfg[self.inst].second_result_mut() { // Don't check types here. Leave that to the verifier. - *val_ref = old_second_value; + *val_ref = old_second_value.into(); } else { // Actually, this instruction format should have called `simple_instruction()`, but // we don't have a rule against calling `complex_instruction()` even when it is diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 7ab9ee24d8..a01a4dd654 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -1,14 +1,14 @@ //! Data flow graph tracking Instructions, Values, and EBBs. use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef}; -use ir::entities::{NO_VALUE, ExpandedValue}; +use ir::entities::ExpandedValue; use ir::instructions::{Opcode, InstructionData, CallInfo}; use ir::extfunc::ExtFuncData; use entity_map::{EntityMap, PrimaryEntityData}; use ir::builder::{InsertBuilder, ReplaceBuilder}; use ir::layout::Cursor; +use packed_option::PackedOption; -use std::mem; use std::ops::{Index, IndexMut}; use std::u16; @@ -103,7 +103,6 @@ impl DataFlowGraph { ValueData::Alias { ty, .. } => ty, } } - None => panic!("NO_VALUE has no type"), } } @@ -126,7 +125,6 @@ impl DataFlowGraph { } } } - None => panic!("NO_VALUE has no def"), } } @@ -237,7 +235,7 @@ enum ValueData { ty: Type, num: u16, // Result number starting from 0. inst: Inst, - next: Value, // Next result defined by `def`. + next: PackedOption, // Next result defined by `def`. }, // Value is an EBB argument. @@ -245,7 +243,7 @@ enum ValueData { ty: Type, num: u16, // Argument number, starting from 0. ebb: Ebb, - next: Value, // Next argument to `ebb`. + next: PackedOption, // Next argument to `ebb`. }, // Value is an alias of another value. @@ -262,31 +260,30 @@ enum ValueData { /// A value iterator borrows a `DataFlowGraph` reference. pub struct Values<'a> { dfg: &'a DataFlowGraph, - cur: Value, + cur: Option, } impl<'a> Iterator for Values<'a> { type Item = Value; fn next(&mut self) -> Option { - let prev = self.cur; - - // Advance self.cur to the next value, or NO_VALUE. - self.cur = match prev.expand() { - ExpandedValue::Direct(inst) => self.dfg.insts[inst].second_result().unwrap_or_default(), - ExpandedValue::Table(index) => { - match self.dfg.extended_values[index] { - ValueData::Inst { next, .. } => next, - ValueData::Arg { next, .. } => next, - ValueData::Alias { .. } => { - panic!("Alias value {} appeared in value list", prev) + let rval = self.cur; + if let Some(prev) = rval { + // Advance self.cur to the next value, or `None`. + self.cur = match prev.expand() { + ExpandedValue::Direct(inst) => self.dfg.insts[inst].second_result(), + ExpandedValue::Table(index) => { + match self.dfg.extended_values[index] { + ValueData::Inst { next, .. } => next.into(), + ValueData::Arg { next, .. } => next.into(), + ValueData::Alias { .. } => { + panic!("Alias value {} appeared in value list", prev) + } } } - } - ExpandedValue::None => return None, - }; - - Some(prev) + }; + } + rval } } @@ -334,7 +331,7 @@ impl DataFlowGraph { // causes additional result values to be numbered backwards which is not the aestetic // choice, but since it is only visible in extremely rare instructions with 3+ results, // we don't care). - let mut head = NO_VALUE; + let mut head = None; let mut first_type = None; let mut rev_num = 1; @@ -346,37 +343,37 @@ impl DataFlowGraph { for res_idx in (0..var_results).rev() { if let Some(ty) = first_type { - head = self.make_value(ValueData::Inst { + head = Some(self.make_value(ValueData::Inst { ty: ty, num: (total_results - rev_num) as u16, inst: inst, - next: head, - }); + next: head.into(), + })); rev_num += 1; } first_type = Some(self.signatures[sig].return_types[res_idx].value_type); } } - // Then the fixed results whic will appear at the front of the list. + // Then the fixed results which will appear at the front of the list. for res_idx in (0..fixed_results).rev() { if let Some(ty) = first_type { - head = self.make_value(ValueData::Inst { + head = Some(self.make_value(ValueData::Inst { ty: ty, num: (total_results - rev_num) as u16, inst: inst, - next: head, - }); + next: head.into(), + })); rev_num += 1; } first_type = Some(constraints.result_type(res_idx, ctrl_typevar)); } // Update the second_result pointer in `inst`. - if head != NO_VALUE { + if head.is_some() { *self.insts[inst] .second_result_mut() - .expect("instruction format doesn't allow multiple results") = head; + .expect("instruction format doesn't allow multiple results") = head.into(); } *self.insts[inst].first_type_mut() = first_type.unwrap_or_default(); @@ -403,8 +400,7 @@ impl DataFlowGraph { /// Use this method to detach secondary values before using `replace(inst)` to provide an /// alternate instruction for computing the primary result value. pub fn detach_secondary_results(&mut self, inst: Inst) -> Values { - let second_result = - self[inst].second_result_mut().map(|r| mem::replace(r, NO_VALUE)).unwrap_or_default(); + let second_result = self[inst].second_result_mut().and_then(|r| r.take()); Values { dfg: self, cur: second_result, @@ -423,9 +419,9 @@ impl DataFlowGraph { Values { dfg: self, cur: if self.insts[inst].first_type().is_void() { - NO_VALUE + None } else { - Value::new_direct(inst) + Some(Value::new_direct(inst)) }, } } @@ -494,17 +490,16 @@ impl DataFlowGraph { /// Get the number of arguments on `ebb`. pub fn num_ebb_args(&self, ebb: Ebb) -> usize { - let last_arg = self.ebbs[ebb].last_arg; - match last_arg.expand() { - ExpandedValue::None => 0, - ExpandedValue::Table(idx) => { - if let ValueData::Arg { num, .. } = self.extended_values[idx] { - num as usize + 1 - } else { - panic!("inconsistent value table entry for EBB arg"); + match self.ebbs[ebb].last_arg.expand() { + None => 0, + Some(last_arg) => { + if let ExpandedValue::Table(idx) = last_arg.expand() { + if let ValueData::Arg { num, .. } = self.extended_values[idx] { + return num as usize + 1; + } } + panic!("inconsistent value table entry for EBB arg"); } - ExpandedValue::Direct(_) => panic!("inconsistent value table entry for EBB arg"), } } @@ -516,25 +511,30 @@ impl DataFlowGraph { ty: ty, ebb: ebb, num: num_args as u16, - next: NO_VALUE, + next: None.into(), }); - let last_arg = self.ebbs[ebb].last_arg; - match last_arg.expand() { - // If last_arg is NO_VALUE, we're adding the first EBB argument. - ExpandedValue::None => { - self.ebbs[ebb].first_arg = val; + match self.ebbs[ebb].last_arg.expand() { + // If last_arg is `None`, we're adding the first EBB argument. + None => { + self.ebbs[ebb].first_arg = val.into(); } - // Append to linked list of arguments. - ExpandedValue::Table(idx) => { - if let ValueData::Arg { ref mut next, .. } = self.extended_values[idx] { - *next = val; - } else { - panic!("inconsistent value table entry for EBB arg"); + Some(last_arg) => { + match last_arg.expand() { + // Append to linked list of arguments. + ExpandedValue::Table(idx) => { + if let ValueData::Arg { ref mut next, .. } = self.extended_values[idx] { + *next = val.into(); + } else { + panic!("inconsistent value table entry for EBB arg"); + } + } + ExpandedValue::Direct(_) => { + panic!("inconsistent value table entry for EBB arg") + } } } - ExpandedValue::Direct(_) => panic!("inconsistent value table entry for EBB arg"), - }; - self.ebbs[ebb].last_arg = val; + } + self.ebbs[ebb].last_arg = val.into(); val } @@ -542,7 +542,7 @@ impl DataFlowGraph { pub fn ebb_args(&self, ebb: Ebb) -> Values { Values { dfg: self, - cur: self.ebbs[ebb].first_arg, + cur: self.ebbs[ebb].first_arg.into(), } } } @@ -554,21 +554,21 @@ impl DataFlowGraph { // match the function arguments. #[derive(Clone)] struct EbbData { - // First argument to this EBB, or `NO_VALUE` if the block has no arguments. + // First argument to this EBB, or `None` if the block has no arguments. // // The arguments are all ValueData::Argument entries that form a linked list from `first_arg` // to `last_arg`. - first_arg: Value, + first_arg: PackedOption, - // Last argument to this EBB, or `NO_VALUE` if the block has no arguments. - last_arg: Value, + // Last argument to this EBB, or `None` if the block has no arguments. + last_arg: PackedOption, } impl EbbData { fn new() -> EbbData { EbbData { - first_arg: NO_VALUE, - last_arg: NO_VALUE, + first_arg: None.into(), + last_arg: None.into(), } } } diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 119873777b..e0b1477af0 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -100,9 +100,6 @@ pub enum ExpandedValue { /// This value is described in the extended value table. Table(usize), - - /// This is NO_VALUE. - None, } impl Value { @@ -129,6 +126,7 @@ impl Value { None } } + /// Create a `Direct` value corresponding to the first value produced by `i`. pub fn new_direct(i: Inst) -> Value { let encoding = i.index() * 2; @@ -148,9 +146,6 @@ impl Value { /// Expand the internal representation into something useful. pub fn expand(&self) -> ExpandedValue { use self::ExpandedValue::*; - if *self == NO_VALUE { - return None; - } let index = (self.0 / 2) as usize; if self.0 % 2 == 0 { Direct(Inst::new(index)) @@ -180,20 +175,10 @@ impl Display for Value { match self.expand() { Direct(i) => write!(fmt, "v{}", i.0), Table(i) => write!(fmt, "vx{}", i), - None => write!(fmt, "NO_VALUE"), } } } -/// A guaranteed invalid value reference. -pub const NO_VALUE: Value = Value(u32::MAX); - -impl Default for Value { - fn default() -> Value { - NO_VALUE - } -} - /// An opaque reference to a stack slot. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct StackSlot(u32); diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index eb0b1e5bd4..f25c58d197 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -16,6 +16,7 @@ use ir::condcodes::*; use ir::types; use ref_slice::*; +use packed_option::PackedOption; // Include code generated by `lib/cretonne/meta/gen_instr.py`. This file contains: // @@ -126,7 +127,7 @@ pub enum InstructionData { UnarySplit { opcode: Opcode, ty: Type, - second_result: Value, + second_result: PackedOption, arg: Value, }, Binary { @@ -150,7 +151,7 @@ pub enum InstructionData { BinaryOverflow { opcode: Opcode, ty: Type, - second_result: Value, + second_result: PackedOption, args: [Value; 2], }, Ternary { @@ -161,7 +162,7 @@ pub enum InstructionData { TernaryOverflow { opcode: Opcode, ty: Type, - second_result: Value, + second_result: PackedOption, data: Box, }, InsertLane { @@ -207,13 +208,13 @@ pub enum InstructionData { Call { opcode: Opcode, ty: Type, - second_result: Value, + second_result: PackedOption, data: Box, }, IndirectCall { opcode: Opcode, ty: Type, - second_result: Value, + second_result: PackedOption, data: Box, }, Return { diff --git a/lib/cretonne/src/packed_option.rs b/lib/cretonne/src/packed_option.rs index a36676a4b4..c647c3793c 100644 --- a/lib/cretonne/src/packed_option.rs +++ b/lib/cretonne/src/packed_option.rs @@ -8,6 +8,7 @@ //! to represent `None`. use std::fmt; +use std::mem; /// Types that have a reserved value which can't be created any other way. pub trait ReservedValue: Eq { @@ -46,6 +47,11 @@ impl PackedOption { pub fn unwrap(self) -> T { self.expand().unwrap() } + + /// Takes the value out of the packed option, leaving a `None` in its place. + pub fn take(&mut self) -> Option { + mem::replace(self, None.into()).expand() + } } impl Default for PackedOption { diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 2d1d85f93d..9decd562b0 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -13,7 +13,7 @@ use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotDa FuncRef}; use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; -use cretonne::ir::entities::{AnyEntity, NO_VALUE}; +use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, TernaryOverflowData, JumpData, BranchData, CallData, IndirectCallData, ReturnData}; @@ -1125,7 +1125,7 @@ impl<'a> Parser<'a> { InstructionData::UnarySplit { opcode: opcode, ty: VOID, - second_result: NO_VALUE, + second_result: None.into(), arg: try!(self.match_value("expected SSA value operand")), } } @@ -1168,7 +1168,7 @@ impl<'a> Parser<'a> { InstructionData::BinaryOverflow { opcode: opcode, ty: VOID, - second_result: NO_VALUE, + second_result: None.into(), args: [lhs, rhs], } } @@ -1196,7 +1196,7 @@ impl<'a> Parser<'a> { InstructionData::TernaryOverflow { opcode: opcode, ty: VOID, - second_result: NO_VALUE, + second_result: None.into(), data: Box::new(TernaryOverflowData { args: [lhs, rhs, cin] }), } } @@ -1287,7 +1287,7 @@ impl<'a> Parser<'a> { InstructionData::Call { opcode: opcode, ty: VOID, - second_result: NO_VALUE, + second_result: None.into(), data: Box::new(CallData { func_ref: func_ref, varargs: args, @@ -1305,7 +1305,7 @@ impl<'a> Parser<'a> { InstructionData::IndirectCall { opcode: opcode, ty: VOID, - second_result: NO_VALUE, + second_result: None.into(), data: Box::new(IndirectCallData { sig_ref: sig_ref, arg: callee, From 02bf84431b19e093a1627aaadb049fe76d970e29 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Jan 2017 18:44:33 -0800 Subject: [PATCH 0478/3084] Use PackedOption in the dominator tree. Also rework the algorithm to be more robust against unreachable blocks. - Add an is_reachable(ebb) method. - Change idom(ebb) to just return an instruction. - Make idom() return None for the entry block as well as unreachable blocks. --- cranelift/src/filetest/domtree.rs | 6 +- lib/cretonne/src/dominator_tree.rs | 270 +++++++++++++++++++---------- lib/cretonne/src/packed_option.rs | 5 + 3 files changed, 184 insertions(+), 97 deletions(-) diff --git a/cranelift/src/filetest/domtree.rs b/cranelift/src/filetest/domtree.rs index 306fc433e9..0a570e3253 100644 --- a/cranelift/src/filetest/domtree.rs +++ b/cranelift/src/filetest/domtree.rs @@ -41,7 +41,7 @@ impl SubTest for TestDomtree { fn run(&self, func: Cow, context: &Context) -> Result<()> { let func = func.borrow(); let cfg = ControlFlowGraph::new(func); - let domtree = DominatorTree::new(&cfg); + let domtree = DominatorTree::new(func, &cfg); // Build an expected domtree from the source annotations. let mut expected = HashMap::new(); @@ -68,7 +68,7 @@ impl SubTest for TestDomtree { // Compare to computed domtree. match domtree.idom(ebb) { - Some((_, got_inst)) if got_inst != inst => { + Some(got_inst) if got_inst != inst => { return Err(format!("mismatching idoms for {}:\n\ want: {}, got: {}", src_ebb, @@ -90,7 +90,7 @@ impl SubTest for TestDomtree { // Now we know that everything in `expected` is consistent with `domtree`. // All other EBB's should be either unreachable or the entry block. for ebb in func.layout.ebbs().skip(1).filter(|ebb| !expected.contains_key(&ebb)) { - if let Some((_, got_inst)) = domtree.idom(ebb) { + if let Some(got_inst) = domtree.idom(ebb) { return Err(format!("mismatching idoms for renumbered {}:\n\ want: unrechable, got: {}", ebb, diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index e331f5657b..f940d3c910 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -1,116 +1,195 @@ //! A Dominator Tree represented as mappings of Ebbs to their immediate dominator. -use cfg::*; -use ir::Ebb; -use ir::entities::NO_INST; +use cfg::{ControlFlowGraph, BasicBlock}; +use ir::{Ebb, Inst, Function, Layout, ProgramOrder}; use entity_map::EntityMap; +use packed_option::PackedOption; + +use std::cmp::Ordering; + +// Dominator tree node. We keep one of these per EBB. +#[derive(Clone, Default)] +struct DomNode { + // Number of this node in a reverse post-order traversal of the CFG, starting from 1. + // Unreachable nodes get number 0, all others are positive. + rpo_number: u32, + + // The immediate dominator of this EBB, represented as the branch or jump instruction at the + // end of the dominating basic block. + // + // This is `None` for unreachable blocks and the entry block which doesn't have an immediate + // dominator. + idom: PackedOption, +} /// The dominator tree for a single function. pub struct DominatorTree { - data: EntityMap>, + nodes: EntityMap, +} + +/// Methods for querying the dominator tree. +impl DominatorTree { + /// Is `ebb` reachable from the entry block? + pub fn is_reachable(&self, ebb: Ebb) -> bool { + self.nodes[ebb].rpo_number != 0 + } + + /// Returns the immediate dominator of `ebb`. + /// + /// The immediate dominator of an extended basic block is a basic block which we represent by + /// the branch or jump instruction at the end of the basic block. This does not have to be the + /// terminator of its EBB. + /// + /// A branch or jump is said to *dominate* `ebb` if all control flow paths from the function + /// entry to `ebb` must go through the branch. + /// + /// The *immediate dominator* is the dominator that is closest to `ebb`. All other dominators + /// also dominate the immediate dominator. + /// + /// This returns `None` if `ebb` is not reachable from the entry EBB, or if it is the entry EBB + /// which has no dominators. + pub fn idom(&self, ebb: Ebb) -> Option { + self.nodes[ebb].idom.into() + } + + /// Compare two EBBs relative to a reverse pst-order traversal of the control-flow graph. + /// + /// Return `Ordering::Less` if `a` comes before `b` in the RPO. + pub fn rpo_cmp(&self, a: Ebb, b: Ebb) -> Ordering { + self.nodes[a].rpo_number.cmp(&self.nodes[b].rpo_number) + } + + /// Returns `true` if `a` dominates `b`. + /// + /// This means that every control-flow path from the function entry to `b` must go through `a`. + /// + /// Dominance is ill defined for unreachable blocks. This function can always determine + /// dominance for instructions in the same EBB, but otherwise returns `false` if either block + /// is unreachable. + /// + /// An instruction is considered to dominate itself. + pub fn dominates(&self, a: Inst, mut b: Inst, layout: &Layout) -> bool { + let ebb_a = layout.inst_ebb(a).expect("Instruction not in layout."); + let mut ebb_b = layout.inst_ebb(b).expect("Instruction not in layout."); + let rpo_a = self.nodes[ebb_a].rpo_number; + + // Run a finger up the dominator tree from b until we see a. + // Do nothing if b is unreachable. + while rpo_a < self.nodes[ebb_b].rpo_number { + b = self.idom(ebb_b).expect("Shouldn't meet unreachable here."); + ebb_b = layout.inst_ebb(b).expect("Dominator got removed."); + } + + ebb_a == ebb_b && layout.cmp(a, b) != Ordering::Greater + } + + /// Compute the common dominator of two basic blocks. + /// + /// Both basic blocks are assumed to be reachable. + pub fn common_dominator(&self, + mut a: BasicBlock, + mut b: BasicBlock, + layout: &Layout) + -> BasicBlock { + loop { + match self.rpo_cmp(a.0, b.0) { + Ordering::Less => { + // `a` comes before `b` in the RPO. Move `b` up. + let idom = self.nodes[b.0].idom.expect("Unreachable basic block?"); + b = (layout.inst_ebb(idom).expect("Dangling idom instruction"), idom); + } + Ordering::Greater => { + // `b` comes before `a` in the RPO. Move `a` up. + let idom = self.nodes[a.0].idom.expect("Unreachable basic block?"); + a = (layout.inst_ebb(idom).expect("Dangling idom instruction"), idom); + } + Ordering::Equal => break, + } + } + + assert_eq!(a.0, b.0, "Unreachable block passed to common_dominator?"); + + // We're in the same EBB. The common dominator is the earlier instruction. + if layout.cmp(a.1, b.1) == Ordering::Less { + a + } else { + b + } + } } impl DominatorTree { /// Build a dominator tree from a control flow graph using Keith D. Cooper's /// "Simple, Fast Dominator Algorithm." - pub fn new(cfg: &ControlFlowGraph) -> DominatorTree { - let mut ebbs = cfg.postorder_ebbs(); - ebbs.reverse(); + pub fn new(func: &Function, cfg: &ControlFlowGraph) -> DominatorTree { + let mut domtree = DominatorTree { nodes: EntityMap::with_capacity(func.dfg.num_ebbs()) }; - let len = ebbs.len(); + // We'll be iterating over a reverse postorder of the CFG. + // This vector only contains reachable EBBs. + let mut postorder = cfg.postorder_ebbs(); - // The mappings which designate the dominator tree. - let mut data = EntityMap::with_capacity(len); + // Remove the entry block, and abort if the function is empty. + // The last block visited in a post-order traversal must be the entry block. + let entry_block = match postorder.pop() { + Some(ebb) => ebb, + None => return domtree, + }; + assert_eq!(Some(entry_block), func.layout.entry_block()); - let mut postorder_map = EntityMap::with_capacity(len); - for (i, ebb) in ebbs.iter().enumerate() { - postorder_map[ebb.clone()] = len - i; - } - - let mut changed = false; - - if len > 0 { - data[ebbs[0]] = Some((ebbs[0], NO_INST)); - changed = true; + // Do a first pass where we assign RPO numbers to all reachable nodes. + domtree.nodes[entry_block].rpo_number = 1; + for (rpo_idx, &ebb) in postorder.iter().rev().enumerate() { + // Update the current node and give it an RPO number. + // The entry block got 1, the rest start at 2. + // + // Nodes do not appear as reachable until the have an assigned RPO number, and + // `compute_idom` will only look at reachable nodes. This means that the function will + // never see an uninitialized predecessor. + // + // Due to the nature of the post-order traversal, every node we visit will have at + // least one predecessor that has previously been visited during this RPO. + domtree.nodes[ebb] = DomNode { + idom: domtree.compute_idom(ebb, cfg, &func.layout).into(), + rpo_number: rpo_idx as u32 + 2, + } } + // Now that we have RPO numbers for everything and initial idom estimates, iterate until + // convergence. + // + // If the function is free of irreducible control flow, this will exit after one iteration. + let mut changed = true; while changed { changed = false; - for i in 1..len { - let ebb = ebbs[i]; - let preds = cfg.get_predecessors(ebb); - let mut new_idom = None; - - for pred in preds { - if new_idom == None { - new_idom = Some(pred.clone()); - continue; - } - // If this predecessor has an idom available find its common - // ancestor with the current value of new_idom. - if let Some(_) = data[pred.0] { - new_idom = match new_idom { - Some(cur_idom) => { - Some((DominatorTree::intersect(&mut data, - &postorder_map, - *pred, - cur_idom))) - } - None => panic!("A 'current idom' should have been set!"), - } - } - } - match data[ebb] { - None => { - data[ebb] = new_idom; - changed = true; - } - Some(idom) => { - // Old idom != New idom - if idom.0 != new_idom.unwrap().0 { - data[ebb] = new_idom; - changed = true; - } - } + for &ebb in postorder.iter().rev() { + let idom = domtree.compute_idom(ebb, cfg, &func.layout).into(); + if domtree.nodes[ebb].idom != idom { + domtree.nodes[ebb].idom = idom; + changed = true; } } } - DominatorTree { data: data } + domtree } - /// Find the common dominator of two ebbs. - fn intersect(data: &EntityMap>, - ordering: &EntityMap, - first: BasicBlock, - second: BasicBlock) - -> BasicBlock { - let mut a = first; - let mut b = second; + // Compute the idom for `ebb` using the current `idom` states for the reachable nodes. + fn compute_idom(&self, ebb: Ebb, cfg: &ControlFlowGraph, layout: &Layout) -> Inst { + // Get an iterator with just the reachable predecessors to `ebb`. + // Note that during the first pass, `is_reachable` returns false for blocks that haven't + // been visited yet. + let mut reachable_preds = + cfg.get_predecessors(ebb).iter().cloned().filter(|&(ebb, _)| self.is_reachable(ebb)); - // Here we use 'ordering', a mapping of ebbs to their postorder - // visitation number, to ensure that we move upward through the tree. - // Walking upward means that we may always expect self.data[a] and - // self.data[b] to contain non-None entries. - while a.0 != b.0 { - while ordering[a.0] < ordering[b.0] { - a = data[a.0].unwrap(); - } - while ordering[b.0] < ordering[a.0] { - b = data[b.0].unwrap(); - } + // The RPO must visit at least one predecessor before this node. + let mut idom = reachable_preds.next() + .expect("EBB node must have one reachable predecessor"); + + for pred in reachable_preds { + idom = self.common_dominator(idom, pred, layout); } - // TODO: we can't rely on instruction numbers to always be ordered - // from lowest to highest. Given that, it will be necessary to create - // an abolute mapping to determine the instruction order in the future. - if a.1 == NO_INST || a.1 < b.1 { a } else { b } - } - - /// Returns the immediate dominator of some ebb or None if the - /// node is unreachable. - pub fn idom(&self, ebb: Ebb) -> Option { - self.data[ebb].clone() + idom.1 } } @@ -118,15 +197,14 @@ impl DominatorTree { mod test { use super::*; use ir::{Function, InstBuilder, Cursor, VariableArgs, types}; - use ir::entities::NO_INST; use cfg::ControlFlowGraph; #[test] fn empty() { let func = Function::new(); let cfg = ControlFlowGraph::new(&func); - let dtree = DominatorTree::new(&cfg); - assert_eq!(0, dtree.data.keys().count()); + let dtree = DominatorTree::new(&func, &cfg); + assert_eq!(0, dtree.nodes.keys().count()); } #[test] @@ -160,12 +238,16 @@ mod test { } let cfg = ControlFlowGraph::new(&func); - let dt = DominatorTree::new(&cfg); + let dt = DominatorTree::new(&func, &cfg); assert_eq!(func.layout.entry_block().unwrap(), ebb3); - assert_eq!(dt.idom(ebb3).unwrap(), (ebb3, NO_INST)); - assert_eq!(dt.idom(ebb1).unwrap(), (ebb3, jmp_ebb3_ebb1)); - assert_eq!(dt.idom(ebb2).unwrap(), (ebb1, jmp_ebb1_ebb2)); - assert_eq!(dt.idom(ebb0).unwrap(), (ebb1, br_ebb1_ebb0)); + assert_eq!(dt.idom(ebb3), None); + assert_eq!(dt.idom(ebb1).unwrap(), jmp_ebb3_ebb1); + assert_eq!(dt.idom(ebb2).unwrap(), jmp_ebb1_ebb2); + assert_eq!(dt.idom(ebb0).unwrap(), br_ebb1_ebb0); + + assert!(dt.dominates(br_ebb1_ebb0, br_ebb1_ebb0, &func.layout)); + assert!(!dt.dominates(br_ebb1_ebb0, jmp_ebb3_ebb1, &func.layout)); + assert!(dt.dominates(jmp_ebb3_ebb1, br_ebb1_ebb0, &func.layout)); } } diff --git a/lib/cretonne/src/packed_option.rs b/lib/cretonne/src/packed_option.rs index c647c3793c..9082a4846f 100644 --- a/lib/cretonne/src/packed_option.rs +++ b/lib/cretonne/src/packed_option.rs @@ -48,6 +48,11 @@ impl PackedOption { self.expand().unwrap() } + /// Unwrap a packed `Some` value or panic. + pub fn expect(self, msg: &str) -> T { + self.expand().expect(msg) + } + /// Takes the value out of the packed option, leaving a `None` in its place. pub fn take(&mut self) -> Option { mem::replace(self, None.into()).expand() From 2e19fa8e074a917591dd376ba265a1167e3401d3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Jan 2017 16:07:16 -0800 Subject: [PATCH 0479/3084] Remove NO_INST and the Default+Ord impls for Inst. Clean up comments. Add an assertion to check that PackedOption is working as designed. --- lib/cretonne/src/ir/entities.rs | 40 ++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index e0b1477af0..69a9902151 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -13,15 +13,14 @@ //! The `entities` module defines public types for the entity references along with constants //! representing an invalid reference. We prefer to use `Option` whenever possible, but //! unfortunately that type is twice as large as the 32-bit index type on its own. Thus, compact -//! data structures use the sentinen constant, while function arguments and return values prefer -//! the more Rust-like `Option` variant. +//! data structures use the `PackedOption` representation, while function arguments and +//! return values prefer the more Rust-like `Option` variant. //! //! The entity references all implement the `Display` trait in a way that matches the textual IL //! format. use entity_map::EntityRef; use packed_option::ReservedValue; -use std::default::Default; use std::fmt::{self, Display, Formatter}; use std::u32; @@ -67,26 +66,13 @@ entity_impl!(Ebb, "ebb"); impl Ebb { /// Create a new EBB reference from its number. This corresponds to the ebbNN representation. + /// + /// This method is for use by the parser. pub fn with_number(n: u32) -> Option { if n < u32::MAX { Some(Ebb(n)) } else { None } } } -/// An opaque reference to an instruction in a function. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] -pub struct Inst(u32); -entity_impl!(Inst, "inst"); - -/// A guaranteed invalid instruction reference. -pub const NO_INST: Inst = Inst(u32::MAX); - -impl Default for Inst { - fn default() -> Inst { - NO_INST - } -} - - /// An opaque reference to an SSA value. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct Value(u32); @@ -105,6 +91,8 @@ pub enum ExpandedValue { impl Value { /// Create a `Direct` value from its number representation. /// This is the number in the vNN notation. + /// + /// This method is for use by the parser. pub fn direct_with_number(n: u32) -> Option { if n < u32::MAX / 2 { let encoding = n * 2; @@ -117,6 +105,8 @@ impl Value { /// Create a `Table` value from its number representation. /// This is the number in the vxNN notation. + /// + /// This method is for use by the parser. pub fn table_with_number(n: u32) -> Option { if n < u32::MAX / 2 { let encoding = n * 2 + 1; @@ -179,6 +169,11 @@ impl Display for Value { } } +/// An opaque reference to an instruction in a function. +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +pub struct Inst(u32); +entity_impl!(Inst, "inst"); + /// An opaque reference to a stack slot. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct StackSlot(u32); @@ -304,4 +299,13 @@ mod tests { }, u32::MAX / 2 - 1); } + + #[test] + fn memory() { + use std::mem; + use packed_option::PackedOption; + // This is the whole point of `PackedOption`. + assert_eq!(mem::size_of::(), + mem::size_of::>()); + } } From 5eba3db4e01e9d929c2631379e41e7768f8a68d9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Jan 2017 21:01:54 -0800 Subject: [PATCH 0480/3084] Remove EntityRef::wrap(). This has been superceded by PackedOption. --- lib/cretonne/src/entity_map.rs | 18 ------------------ 1 file changed, 18 deletions(-) diff --git a/lib/cretonne/src/entity_map.rs b/lib/cretonne/src/entity_map.rs index ba2075370f..33d355e9f1 100644 --- a/lib/cretonne/src/entity_map.rs +++ b/lib/cretonne/src/entity_map.rs @@ -24,24 +24,6 @@ pub trait EntityRef: Copy + Eq { /// Get the index that was used to create this entity reference. fn index(self) -> usize; - - /// Convert an `EntityRef` to an `Optional` by using the default value as the null - /// reference. - /// - /// Entity references are often used in compact data structures like linked lists where a - /// sentinel 'null' value is needed. Normally we would use an `Optional` for that, but - /// currently that uses twice the memory of a plain `EntityRef`. - /// - /// This method is called `wrap()` because it is the inverse of `unwrap()`. - fn wrap(self) -> Option - where Self: Default - { - if self == Self::default() { - None - } else { - Some(self) - } - } } /// A mapping `K -> V` for densely indexed entity references. From 58c36f71e943045cfb577ec64f07dacc4cabbcd3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 20 Jan 2017 10:33:45 -0800 Subject: [PATCH 0481/3084] Fix flake8 style issue. --- lib/cretonne/meta/cdsl/operands.py | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index 2fcc22b0c0..f7ec85eafa 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -40,6 +40,7 @@ class OperandKind(object): # type: () -> str return 'OperandKind({})'.format(self.name) + #: An SSA value operand. This is a value defined by another instruction. VALUE = OperandKind( 'value', """ From 1d21422032abed44becdeb3df0e76a5d1e676846 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 20 Jan 2017 11:27:56 -0800 Subject: [PATCH 0482/3084] Generate register class descriptors. Add a mechanism for defining sub-classes of register classes. --- lib/cretonne/meta/cdsl/registers.py | 81 ++++++++++++++++++------ lib/cretonne/meta/gen_registers.py | 36 ++++++++++- lib/cretonne/meta/isa/arm32/registers.py | 10 +-- lib/cretonne/meta/isa/arm64/registers.py | 6 +- lib/cretonne/meta/isa/intel/registers.py | 7 +- lib/cretonne/meta/isa/riscv/registers.py | 6 +- lib/cretonne/src/isa/arm32/registers.rs | 2 +- lib/cretonne/src/isa/arm64/registers.rs | 2 +- lib/cretonne/src/isa/intel/registers.rs | 2 +- lib/cretonne/src/isa/registers.rs | 37 ++++++++++- lib/cretonne/src/isa/riscv/registers.rs | 2 +- 11 files changed, 152 insertions(+), 39 deletions(-) diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index 243a72e669..b27c6128ec 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -32,6 +32,15 @@ except ImportError: pass +# The number of 32-bit elements in a register unit mask +MASK_LEN = 3 + +# The maximum total number of register units allowed. +# This limit can be raised by also adjusting the RegUnitMask type in +# src/isa/registers.rs. +MAX_UNITS = MASK_LEN * 32 + + class RegBank(object): """ A register bank belonging to an ISA. @@ -61,6 +70,7 @@ class RegBank(object): self.units = units self.prefix = prefix self.names = names + self.classes = list() # type: List[RegClass] assert len(names) <= units @@ -91,19 +101,16 @@ class RegClass(object): allocated two units at a time. When multiple units are allocated, it is always a contiguous set of unit numbers. - :param name: Name of this register class. :param bank: The register bank we're allocating from. :param count: The maximum number of allocations in this register class. By default, the whole register bank can be allocated. :param width: How many units to allocate at a time. :param start: The first unit to allocate, relative to `bank.first.unit`. - :param stride: How many units to skip to get to the next allocation. - Default is `width`. """ - def __init__(self, name, bank, count=None, width=1, start=0, stride=None): - # type: (str, RegBank, int, int, int, int) -> None - self.name = name + def __init__(self, bank, count=None, width=1, start=0): + # type: (RegBank, int, int, int) -> None + self.name = None # type: str self.bank = bank self.start = start self.width = width @@ -111,21 +118,53 @@ class RegClass(object): assert width > 0 assert start >= 0 and start < bank.units - if stride is None: - stride = width - assert stride > 0 - self.stride = stride - if count is None: - count = bank.units / stride + count = bank.units // width self.count = count - # When the stride is 1, we can wrap around to the beginning of the - # register bank, but with a larger stride, we wouldn't cover all the - # possible allocations with a simple modulo stride. For example, - # attempting to allocate the even registers before the odd ones - # wouldn't work. Only if stride is coprime to bank.units would it work, - # but that is unlikely since the bank size is almost always a power of - # two. - if start + count*stride > bank.units: - assert stride == 1, 'Wrapping with stride not supported' + bank.classes.append(self) + + def __getitem__(self, sliced): + """ + Create a sub-class of a register class using slice notation. The slice + indexes refer to allocations in the parent register class, not register + units. + """ + assert isinstance(sliced, slice), "RegClass slicing can't be 1 reg" + # We could add strided sub-classes if needed. + assert sliced.step is None, 'Subclass striding not supported' + + w = self.width + s = self.start + sliced.start * w + c = sliced.stop - sliced.start + assert c > 1, "Can't have single-register classes" + + return RegClass(self.bank, count=c, width=w, start=s) + + def mask(self): + """ + Compute a bit-mask of the register units allocated by this register + class. + + Return as a list of 32-bit integers. + """ + mask = [0] * MASK_LEN + + start = self.bank.first_unit + self.start + for a in range(self.count): + u = start + a * self.width + mask[u // 32] |= 1 << (u % 32) + + return mask + + @staticmethod + def extract_names(globs): + """ + Given a dict mapping name -> object as returned by `globals()`, find + all the RegClass objects and set their name from the dict key. + This is used to name a bunch of global variables in a module. + """ + for name, obj in globs.items(): + if isinstance(obj, RegClass): + assert obj.name is None + obj.name = name diff --git a/lib/cretonne/meta/gen_registers.py b/lib/cretonne/meta/gen_registers.py index 323aec6173..d3afe3ca4c 100644 --- a/lib/cretonne/meta/gen_registers.py +++ b/lib/cretonne/meta/gen_registers.py @@ -8,7 +8,7 @@ import srcgen try: from typing import Sequence # noqa from cdsl.isa import TargetISA # noqa - from cdsl.registers import RegBank # noqa + from cdsl.registers import RegBank, RegClass # noqa except ImportError: pass @@ -18,8 +18,7 @@ def gen_regbank(regbank, fmt): """ Emit a static data definition for regbank. """ - with fmt.indented( - 'RegBank {{'.format(regbank.name), '},'): + with fmt.indented('RegBank {', '},'): fmt.line('name: "{}",'.format(regbank.name)) fmt.line('first_unit: {},'.format(regbank.first_unit)) fmt.line('units: {},'.format(regbank.units)) @@ -29,6 +28,19 @@ def gen_regbank(regbank, fmt): fmt.line('prefix: "{}",'.format(regbank.prefix)) +def gen_regclass(idx, rc, fmt): + # type: (int, RegClass, srcgen.Formatter) -> None + """ + Emit a static data definition for a register class. + """ + fmt.comment(rc.name) + with fmt.indented('RegClassData {', '},'): + fmt.line('index: {},'.format(idx)) + fmt.line('width: {},'.format(rc.width)) + mask = ', '.join('0x{:08x}'.format(x) for x in rc.mask()) + fmt.line('mask: [{}],'.format(mask)) + + def gen_isa(isa, fmt): # type: (TargetISA, srcgen.Formatter) -> None """ @@ -36,11 +48,29 @@ def gen_isa(isa, fmt): """ if not isa.regbanks: print('cargo:warning={} has no register banks'.format(isa.name)) + + rcs = list() # type: List[RegClass] with fmt.indented('pub static INFO: RegInfo = RegInfo {', '};'): # Bank descriptors. with fmt.indented('banks: &[', '],'): for regbank in isa.regbanks: gen_regbank(regbank, fmt) + rcs += regbank.classes + fmt.line('classes: &CLASSES,') + + # Register class descriptors. + with fmt.indented( + 'const CLASSES: [RegClassData; {}] = ['.format(len(rcs)), '];'): + for idx, rc in enumerate(rcs): + gen_regclass(idx, rc, fmt) + + # Emit constants referencing the register classes. + for idx, rc in enumerate(rcs): + if rc.name: + fmt.line('#[allow(dead_code)]') + fmt.line( + 'pub const {}: RegClass = &CLASSES[{}];' + .format(rc.name, idx)) def generate(isas, out_dir): diff --git a/lib/cretonne/meta/isa/arm32/registers.py b/lib/cretonne/meta/isa/arm32/registers.py index af031e145b..9522057e34 100644 --- a/lib/cretonne/meta/isa/arm32/registers.py +++ b/lib/cretonne/meta/isa/arm32/registers.py @@ -29,7 +29,9 @@ IntRegs = RegBank( 'General purpose registers', units=16, prefix='r') -GPR = RegClass('GPR', IntRegs) -S = RegClass('S', FloatRegs, count=32) -D = RegClass('D', FloatRegs, width=2) -Q = RegClass('Q', FloatRegs, width=4) +GPR = RegClass(IntRegs) +S = RegClass(FloatRegs, count=32) +D = RegClass(FloatRegs, width=2) +Q = RegClass(FloatRegs, width=4) + +RegClass.extract_names(globals()) diff --git a/lib/cretonne/meta/isa/arm64/registers.py b/lib/cretonne/meta/isa/arm64/registers.py index fec3601367..b39bc917a1 100644 --- a/lib/cretonne/meta/isa/arm64/registers.py +++ b/lib/cretonne/meta/isa/arm64/registers.py @@ -18,5 +18,7 @@ FloatRegs = RegBank( 'Floating point registers', units=32, prefix='v') -GPR = RegClass('GPR', IntRegs) -FPR = RegClass('FPR', FloatRegs) +GPR = RegClass(IntRegs) +FPR = RegClass(FloatRegs) + +RegClass.extract_names(globals()) diff --git a/lib/cretonne/meta/isa/intel/registers.py b/lib/cretonne/meta/isa/intel/registers.py index 6a4c59403e..92e76191fa 100644 --- a/lib/cretonne/meta/isa/intel/registers.py +++ b/lib/cretonne/meta/isa/intel/registers.py @@ -38,5 +38,8 @@ FloatRegs = RegBank( 'SSE floating point registers', units=16, prefix='xmm') -GPR = RegClass('GPR', IntRegs) -FPR = RegClass('FPR', FloatRegs) +GPR = RegClass(IntRegs) +ABCD = GPR[0:4] +FPR = RegClass(FloatRegs) + +RegClass.extract_names(globals()) diff --git a/lib/cretonne/meta/isa/riscv/registers.py b/lib/cretonne/meta/isa/riscv/registers.py index f00c9a0f0d..d9d43f0432 100644 --- a/lib/cretonne/meta/isa/riscv/registers.py +++ b/lib/cretonne/meta/isa/riscv/registers.py @@ -17,5 +17,7 @@ FloatRegs = RegBank( 'Floating point registers', units=32, prefix='f') -GPR = RegClass('GPR', IntRegs) -FPR = RegClass('FPR', FloatRegs) +GPR = RegClass(IntRegs) +FPR = RegClass(FloatRegs) + +RegClass.extract_names(globals()) diff --git a/lib/cretonne/src/isa/arm32/registers.rs b/lib/cretonne/src/isa/arm32/registers.rs index 30e7fd58cf..69571976cb 100644 --- a/lib/cretonne/src/isa/arm32/registers.rs +++ b/lib/cretonne/src/isa/arm32/registers.rs @@ -1,6 +1,6 @@ //! ARM32 register descriptions. -use isa::registers::{RegBank, RegInfo}; +use isa::registers::{RegBank, RegClass, RegClassData, RegInfo}; include!(concat!(env!("OUT_DIR"), "/registers-arm32.rs")); diff --git a/lib/cretonne/src/isa/arm64/registers.rs b/lib/cretonne/src/isa/arm64/registers.rs index 2420db6df1..0a0e043a20 100644 --- a/lib/cretonne/src/isa/arm64/registers.rs +++ b/lib/cretonne/src/isa/arm64/registers.rs @@ -1,6 +1,6 @@ //! ARM64 register descriptions. -use isa::registers::{RegBank, RegInfo}; +use isa::registers::{RegBank, RegClass, RegClassData, RegInfo}; include!(concat!(env!("OUT_DIR"), "/registers-arm64.rs")); diff --git a/lib/cretonne/src/isa/intel/registers.rs b/lib/cretonne/src/isa/intel/registers.rs index 1946d0fc57..82c5b9886e 100644 --- a/lib/cretonne/src/isa/intel/registers.rs +++ b/lib/cretonne/src/isa/intel/registers.rs @@ -1,6 +1,6 @@ //! Intel register descriptions. -use isa::registers::{RegBank, RegInfo}; +use isa::registers::{RegBank, RegClass, RegClassData, RegInfo}; include!(concat!(env!("OUT_DIR"), "/registers-intel.rs")); diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 8736f06e84..26dc65b7d1 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -11,6 +11,14 @@ use std::fmt; /// The register allocator will enforce that each register unit only gets used for one thing. pub type RegUnit = u16; +/// A bit mask indexed by register units. +/// +/// The size of this type is determined by the target ISA that has the most register units defined. +/// Currently that is arm32 which has 64+16 units. +/// +/// This type should be coordinated with meta/cdsl/registers.py. +pub type RegUnitMask = [u32; 3]; + /// The register units in a target ISA are divided into disjoint register banks. Each bank covers a /// contiguous range of register units. /// @@ -23,7 +31,7 @@ pub struct RegBank { pub first_unit: RegUnit, /// The total number of register units in this bank. - pub units: u16, + pub units: RegUnit, /// Array of specially named register units. This array can be shorter than the number of units /// in the bank. @@ -79,6 +87,30 @@ impl RegBank { } } +/// A register class reference. +/// +/// All register classes are statically defined in tables generated from the meta descriptions. +pub type RegClass = &'static RegClassData; + +/// Data about a register class. +/// +/// A register class represents a subset of the registers in a bank. It describes the set of +/// permitted registers for a register operand in a given encoding of an instruction. +/// +/// A register class can be a subset of another register class. The top-level register classes are +/// disjoint. +pub struct RegClassData { + /// The index of this class in the ISA's RegInfo description. + pub index: u8, + + /// How many register units to allocate per register. + pub width: u8, + + /// Mask of register units in the class. If `width > 1`, the mask only has a bit set for the + /// first register unit in each allocatable register. + pub mask: RegUnitMask, +} + /// Information about the registers in an ISA. /// /// The `RegUnit` data structure collects all relevant static information about the registers in an @@ -87,6 +119,9 @@ pub struct RegInfo { /// All register banks, ordered by their `first_unit`. The register banks are disjoint, but /// there may be holes of unused register unit numbers between banks due to alignment. pub banks: &'static [RegBank], + + /// All register classes ordered topologically so a sub-class always follows its parent. + pub classes: &'static [RegClassData], } impl RegInfo { diff --git a/lib/cretonne/src/isa/riscv/registers.rs b/lib/cretonne/src/isa/riscv/registers.rs index a3fe4739e8..7deef9251a 100644 --- a/lib/cretonne/src/isa/riscv/registers.rs +++ b/lib/cretonne/src/isa/riscv/registers.rs @@ -1,6 +1,6 @@ //! RISC-V register descriptions. -use isa::registers::{RegBank, RegInfo}; +use isa::registers::{RegBank, RegClass, RegClassData, RegInfo}; include!(concat!(env!("OUT_DIR"), "/registers-riscv.rs")); From 3b83496edbd2918f5eb05e0c741c830e1d209d96 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 20 Jan 2017 14:41:06 -0800 Subject: [PATCH 0483/3084] Add an AllocatableSet for registers. This set of available register units also manages register aliasing in an efficient way. Detect if the units in a register straddles mask words. The algorithm for allocating multi-unit registers expect the whole register to be inside a single mask word. We could handle this if necessary, but so far no ISAs need it. --- lib/cretonne/meta/cdsl/registers.py | 7 +- lib/cretonne/src/isa/mod.rs | 4 +- lib/cretonne/src/isa/registers.rs | 4 +- lib/cretonne/src/regalloc/allocatable_set.rs | 176 +++++++++++++++++++ lib/cretonne/src/regalloc/mod.rs | 1 + 5 files changed, 187 insertions(+), 5 deletions(-) create mode 100644 lib/cretonne/src/regalloc/allocatable_set.rs diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index b27c6128ec..dc6885c296 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -153,7 +153,12 @@ class RegClass(object): start = self.bank.first_unit + self.start for a in range(self.count): u = start + a * self.width - mask[u // 32] |= 1 << (u % 32) + b = u % 32 + # We need fancier masking code if a register can straddle mask + # words. This will only happen with widths that are not powers of + # two. + assert b + self.width <= 32, 'Register straddles words' + mask[u // 32] |= 1 << b return mask diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index dc66355904..6d22d5dd71 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -41,7 +41,7 @@ //! concurrent function compilations. pub use isa::encoding::Encoding; -pub use isa::registers::{RegUnit, RegBank, RegInfo}; +pub use isa::registers::{RegInfo, RegUnit, RegClass}; use settings; use ir::{InstructionData, DataFlowGraph}; @@ -49,9 +49,9 @@ pub mod riscv; pub mod intel; pub mod arm32; pub mod arm64; +pub mod registers; mod encoding; mod enc_tables; -mod registers; /// Look for a supported ISA with the given `name`. /// Return a builder that can create a corresponding `TargetIsa`. diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 26dc65b7d1..48173c50bd 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -147,8 +147,8 @@ impl RegInfo { /// Temporary object that holds enough information to print a register unit. pub struct DisplayRegUnit<'a> { - pub regunit: RegUnit, - pub reginfo: &'a RegInfo, + regunit: RegUnit, + reginfo: &'a RegInfo, } impl<'a> fmt::Display for DisplayRegUnit<'a> { diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs new file mode 100644 index 0000000000..34145cd01a --- /dev/null +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -0,0 +1,176 @@ +//! Set of allocatable registers as a bit vector of register units. +//! +//! While allocating registers, we need to keep track of which registers are available and which +//! registers are in use. Since registers can alias in different ways, we track this via the +//! "register unit" abstraction. Every register contains one or more register units. Registers that +//! share a register unit can't be in use at the same time. + +use std::mem::size_of_val; +use isa::registers::{RegUnit, RegUnitMask, RegClass}; + +/// Set of registers available for allocation. +#[derive(Clone)] +pub struct AllocatableSet { + avail: RegUnitMask, +} + +// Given a register class and a register unit in the class, compute a word index and a bit mask of +// register units representing that register. +// +// Note that a register is not allowed to straddle words. +fn bitmask(rc: RegClass, reg: RegUnit) -> (usize, u32) { + // Bit mask representing the register. It is `rc.width` consecutive units. + let width_bits = (1 << rc.width) - 1; + // Index into avail[] of the word containing `reg`. + let word_index = (reg / 32) as usize; + // The actual bits in the word that cover `reg`. + let reg_bits = width_bits << (reg % 32); + + (word_index, reg_bits) +} + +impl AllocatableSet { + /// Create a new register set with all registers available. + /// + /// Note that this includes *all* registers. Query the `TargetIsa` object to get a set of + /// allocatable registers where reserved registers have been filtered out. + pub fn new() -> AllocatableSet { + AllocatableSet { avail: [!0; 3] } + } + + /// Returns `true` if the spoecified register is available. + pub fn is_avail(&self, rc: RegClass, reg: RegUnit) -> bool { + let (idx, bits) = bitmask(rc, reg); + (self.avail[idx] & bits) == bits + } + + /// Allocate `reg` from `rc` so it is no longer available. + /// + /// It is an error to take a register that doesn't have all of its register units available. + pub fn take(&mut self, rc: RegClass, reg: RegUnit) { + let (idx, bits) = bitmask(rc, reg); + debug_assert!((self.avail[idx] & bits) == bits, "Not available"); + self.avail[idx] &= !bits; + } + + /// Make `reg` available for allocation again. + pub fn free(&mut self, rc: RegClass, reg: RegUnit) { + let (idx, bits) = bitmask(rc, reg); + debug_assert!((self.avail[idx] & bits) == 0, "Not allocated"); + self.avail[idx] |= bits; + } + + /// Return an iterator over all available registers belonging to the register class `rc`. + /// + /// This doesn't allocate anything from the set; use `take()` for that. + pub fn iter(&self, rc: RegClass) -> RegSetIter { + // Start by copying the RC mask. It is a single set bit for each register in the class. + let mut rsi = RegSetIter { regs: rc.mask }; + + // Mask out the unavailable units. + for idx in 0..self.avail.len() { + // If a single unit in a register is unavailable, the whole register can't be used. + // If a register straddles a word boundary, it will be marked as unavailable. + // There's an assertion in cdsl/registers.py to check for that. + for i in 0..rc.width { + rsi.regs[idx] &= self.avail[idx] >> i; + } + } + rsi + } +} + +/// Iterator over available registers in a register class. +pub struct RegSetIter { + regs: RegUnitMask, +} + +impl Iterator for RegSetIter { + type Item = RegUnit; + + fn next(&mut self) -> Option { + let mut unit_offset = 0; + + // Find the first set bit in `self.regs`. + for word in &mut self.regs { + if *word != 0 { + // Compute the register unit number from the lowest set bit in the word. + let unit = unit_offset + word.trailing_zeros() as RegUnit; + + // Clear that lowest bit so we won't find it again. + *word = *word & (*word - 1); + + return Some(unit); + } + // How many register units was there in the word? This is a constant 32 for u32 etc. + unit_offset += 8 * size_of_val(word) as RegUnit; + } + + // All of `self.regs` is 0. + None + } +} + +#[cfg(test)] +mod tests { + use super::*; + use isa::registers::{RegClass, RegClassData}; + + // Register classes for testing. + const GPR: RegClass = &RegClassData { + index: 0, + width: 1, + mask: [0xf0000000, 0x0000000f, 0], + }; + const DPR: RegClass = &RegClassData { + index: 0, + width: 2, + mask: [0x50000000, 0x0000000a, 0], + }; + + #[test] + fn put_and_take() { + let mut regs = AllocatableSet::new(); + + // GPR has units 28-36. + assert_eq!(regs.iter(GPR).count(), 8); + assert_eq!(regs.iter(DPR).collect::>(), [28, 30, 33, 35]); + + assert!(regs.is_avail(GPR, 29)); + regs.take(&GPR, 29); + assert!(!regs.is_avail(GPR, 29)); + + assert_eq!(regs.iter(GPR).count(), 7); + assert_eq!(regs.iter(DPR).collect::>(), [30, 33, 35]); + + assert!(regs.is_avail(GPR, 30)); + regs.take(&GPR, 30); + assert!(!regs.is_avail(GPR, 30)); + + assert_eq!(regs.iter(GPR).count(), 6); + assert_eq!(regs.iter(DPR).collect::>(), [33, 35]); + + assert!(regs.is_avail(GPR, 32)); + regs.take(&GPR, 32); + assert!(!regs.is_avail(GPR, 32)); + + assert_eq!(regs.iter(GPR).count(), 5); + assert_eq!(regs.iter(DPR).collect::>(), [33, 35]); + + regs.free(&GPR, 30); + assert!(regs.is_avail(GPR, 30)); + assert!(!regs.is_avail(GPR, 29)); + assert!(!regs.is_avail(GPR, 32)); + + assert_eq!(regs.iter(GPR).count(), 6); + assert_eq!(regs.iter(DPR).collect::>(), [30, 33, 35]); + + regs.free(&GPR, 32); + assert!(regs.is_avail(GPR, 31)); + assert!(!regs.is_avail(GPR, 29)); + assert!(regs.is_avail(GPR, 32)); + + assert_eq!(regs.iter(GPR).count(), 7); + assert_eq!(regs.iter(DPR).collect::>(), [30, 33, 35]); + } +} diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index dff29bc5bc..9e0f7b820d 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -4,3 +4,4 @@ pub mod liverange; pub mod liveness; +pub mod allocatable_set; From 2390e3e3f0f891d62e00a0178e5f6d55216681ca Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 24 Jan 2017 11:19:31 -0800 Subject: [PATCH 0484/3084] Add operand register constraints. Every encoding recipe must specify register constraints on input and output values. Generate recipe constraint tables along with the other encoding tables. --- cranelift/docs/metaref.rst | 82 +++++++++++++++++++++++- lib/cretonne/meta/cdsl/isa.py | 37 ++++++++++- lib/cretonne/meta/cdsl/registers.py | 26 +++++++- lib/cretonne/meta/gen_encoding.py | 53 ++++++++++++++- lib/cretonne/meta/isa/riscv/recipes.py | 9 ++- lib/cretonne/src/isa/arm32/enc_tables.rs | 1 + lib/cretonne/src/isa/arm32/mod.rs | 6 +- lib/cretonne/src/isa/arm64/enc_tables.rs | 1 + lib/cretonne/src/isa/arm64/mod.rs | 6 +- lib/cretonne/src/isa/constraints.rs | 66 +++++++++++++++++++ lib/cretonne/src/isa/intel/enc_tables.rs | 1 + lib/cretonne/src/isa/intel/mod.rs | 6 +- lib/cretonne/src/isa/mod.rs | 11 +++- lib/cretonne/src/isa/riscv/enc_tables.rs | 2 + lib/cretonne/src/isa/riscv/mod.rs | 6 +- 15 files changed, 299 insertions(+), 14 deletions(-) create mode 100644 lib/cretonne/src/isa/constraints.rs diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index ec4555a8d7..05050f8f40 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -331,8 +331,6 @@ encoded: - The CPU mode that must be active. - A :term:`sub-target predicate` that must be satisfied by the currently active sub-target. -- :term:`Register constraint`\s that must be satisfied by the instruction's value - operands and results. An encoding specifies an *encoding recipe* along with some *encoding bits* that the recipe can use for native opcode fields etc. The encoding recipe has @@ -349,6 +347,83 @@ encodings only need the recipe predicates. .. autoclass:: EncRecipe +Register constraints +==================== + +After an encoding recipe has been chosen for an instruction, it is the register +allocator's job to make sure that the recipe's :term:`Register constraint`\s +are satisfied. Most ISAs have separate integer and floating point registers, +and instructions can usually only use registers from one of the banks. Some +instruction encodings are even more constrained and can only use a subset of +the registers in a bank. These constraints are expressed in terms of register +classes. + +Sometimes the result of an instruction is placed in a register that must be the +same as one of the input registers. Some instructions even use a fixed register +for inputs or results. + +Each encoding recipe specifies separate constraints for its value operands and +result. These constraints are separate from the instruction predicate which can +only evaluate the instruction's immediate operands. + +.. module:: cdsl.registers +.. autoclass:: RegBank + +Register class constraints +-------------------------- + +The most common type of register constraint is the register class. It specifies +that an operand or result must be allocated one of the registers from the given +register class:: + + IntRegs = RegBank('IntRegs', ISA, 'General purpose registers', units=16, prefix='r') + GPR = RegClass(IntRegs) + R = EncRecipe('R', Binary, ins=(GPR, GPR), outs=GPR) + +This defines an encoding recipe for the ``Binary`` instruction format where +both input operands must be allocated from the ``GPR`` register class. + +.. autoclass:: RegClass + +Tied register operands +---------------------- + +In more compact machine code encodings, it is common to require that the result +register is the same as one of the inputs. This is represented with tied +operands:: + + CR = EncRecipe('CR', Binary, ins=(GPR, GPR), outs=0) + +This indicates that the result value must be allocated to the same register as +the first input value. Tied operand constraints can only be used for result +values, so the number always refers to one of the input values. + +Fixed register operands +----------------------- + +Some instructions use hard-coded input and output registers for some value +operands. An example is the ``pblendvb`` Intel SSE instruction which takes one +of its three value operands in the hard-coded ``%xmm0`` register:: + + XMM0 = FPR[0] + SSE66_XMM0 = EncRecipe('SSE66_XMM0', Ternary, ins=(FPR, FPR, XMM0), outs=0) + +The syntax ``FPR[0]`` selects the first register from the ``FPR`` register +class which consists of all the XMM registers. + +Stack operands +-------------- + +Cretonne's register allocator can assign an SSA value to a stack slot if there +isn't enough registers. It will insert :cton:inst:`spill` and :cton:inst:`fill` +instructions as needed to satisfy instruction operand constraints, but it is +also possible to have instructions that can access stack slots directly:: + + CSS = EncRecipe('CSS', Unary, ins=GPR, outs=Stack(GPR)) + +An output stack value implies a store to the stack, an input value implies a +load. + .. module:: cdsl.isa Targets @@ -366,6 +441,9 @@ The definitions for each supported target live in a package under :members: .. automodule:: isa.riscv +.. automodule:: isa.intel +.. automodule:: isa.arm32 +.. automodule:: isa.arm64 Glossary diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 6c324603a5..8e1423315c 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -1,6 +1,7 @@ """Defining instruction set architectures.""" from __future__ import absolute_import from .predicates import And +from .registers import RegClass, Register # The typing module is only required by mypy, and we don't use these imports # outside type comments. @@ -12,6 +13,8 @@ try: from .types import ValueType # noqa from .registers import RegBank # noqa AnyPredicate = Union[Predicate, FieldPredicate] + OperandConstraint = Union[RegClass, Register, int] + ConstraintSeq = Union[OperandConstraint, Tuple[OperandConstraint, ...]] except ImportError: pass @@ -133,13 +136,24 @@ class EncRecipe(object): Many different instructions can be encoded by the same recipe, but they must all have the same instruction format. + The `ins` and `outs` arguments are tuples specifying the register + allocation constraints for the value operands and results respectively. The + possible constraints for an operand are: + + - A `RegClass` specifying the set of allowed registers. + - A `Register` specifying a fixed-register operand. + - An integer indicating that this result is tied to a value operand, so + they must use the same register. + :param name: Short mnemonic name for this recipe. :param format: All encoded instructions must have this :py:class:`InstructionFormat`. + :param: ins Tuple of register constraints for value operands. + :param: outs Tuple of register constraints for results. """ - def __init__(self, name, format, instp=None, isap=None): - # type: (str, InstructionFormat, AnyPredicate, AnyPredicate) -> None + def __init__(self, name, format, ins, outs, instp=None, isap=None): + # type: (str, InstructionFormat, ConstraintSeq, ConstraintSeq, AnyPredicate, AnyPredicate) -> None # noqa self.name = name self.format = format self.instp = instp @@ -148,10 +162,29 @@ class EncRecipe(object): assert instp.predicate_context() == format self.number = None # type: int + self.ins = self._verify_constraints(ins) + assert len(self.ins) == len(format.value_operands) + self.outs = self._verify_constraints(outs) + if len(self.outs) > 1: + assert format.multiple_results + def __str__(self): # type: () -> str return self.name + def _verify_constraints(self, seq): + # (ConstraintSeq) -> Sequence[OperandConstraint] + if not isinstance(seq, tuple): + seq = (seq,) + for c in seq: + if isinstance(c, int): + # An integer constraint is bound to a value operand. + # Check that it is in range. + assert c >= 0 and c < len(self.format.value_operands) + else: + assert isinstance(c, RegClass) or isinstance(c, Register) + return seq + class Encoding(object): """ diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index dc6885c296..828e6a2704 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -62,7 +62,7 @@ class RegBank(object): `units`, the remaining units are named using `prefix`. """ - def __init__(self, name, isa, doc, units, prefix='p', names=()): + def __init__(self, name, isa, doc, units, prefix='r', names=()): # type: (str, TargetISA, str, int, str, Sequence[str]) -> None self.name = name self.isa = isa @@ -124,12 +124,16 @@ class RegClass(object): bank.classes.append(self) + def __str__(self): + return self.name + def __getitem__(self, sliced): """ Create a sub-class of a register class using slice notation. The slice indexes refer to allocations in the parent register class, not register units. """ + print(repr(sliced)) assert isinstance(sliced, slice), "RegClass slicing can't be 1 reg" # We could add strided sub-classes if needed. assert sliced.step is None, 'Subclass striding not supported' @@ -142,6 +146,7 @@ class RegClass(object): return RegClass(self.bank, count=c, width=w, start=s) def mask(self): + # type: () -> List[int] """ Compute a bit-mask of the register units allocated by this register class. @@ -173,3 +178,22 @@ class RegClass(object): if isinstance(obj, RegClass): assert obj.name is None obj.name = name + + +class Register(object): + """ + A specific register in a register class. + + A register is identified by the top-level register class it belongs to and + its first register unit. + + Specific registers are used to describe constraints on instructions where + some operands must use a fixed register. + + Register objects should be created using the indexing syntax on the + register class. + """ + def __init__(self, rc, unit): + # type: (RegClass, int) -> None + self.regclass = rc + self.unit = unit diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 8477f31de2..30d1ed3370 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -1,7 +1,7 @@ """ Generate sources for instruction encoding. -The tables and functions generated here support the `TargetIsa::encode()` +The tables and functions generated here support the `TargetISA::encode()` function which determines if a given instruction is legal, and if so, it's `Encoding` data which consists of a *recipe* and some *encoding* bits. @@ -56,6 +56,13 @@ from unique_table import UniqueSeqTable from collections import OrderedDict, defaultdict import math import itertools +from cdsl.registers import RegClass, Register + +try: + from typing import Sequence # noqa + from cdsl.isa import TargetISA, OperandConstraint # noqa +except ImportError: + pass def emit_instp(instp, fmt): @@ -399,6 +406,7 @@ def offset_type(length): def emit_recipe_names(isa, fmt): + # type: (TargetISA, srcgen.Formatter) -> None """ Emit a table of encoding recipe names keyed by recipe number. @@ -411,6 +419,48 @@ def emit_recipe_names(isa, fmt): fmt.line('"{}",'.format(r.name)) +def emit_recipe_constraints(isa, fmt): + # type: (TargetISA, srcgen.Formatter) -> None + """ + Emit a table of encoding recipe operand constraints keyed by recipe number. + + These are used by the register allocator to pick registers that can be + properly encoded. + """ + with fmt.indented( + 'pub static RECIPE_CONSTRAINTS: [RecipeConstraints; {}] = [' + .format(len(isa.all_recipes)), '];'): + for r in isa.all_recipes: + fmt.comment(r.name) + with fmt.indented('RecipeConstraints {', '},'): + emit_operand_constraints(r.ins, 'ins', fmt) + emit_operand_constraints(r.outs, 'outs', fmt) + + +def emit_operand_constraints(seq, field, fmt): + # type: (Sequence[OperandConstraint], str, srcgen.Formatter) -> None + """ + Emit a struct field initializer for an array of operand constraints. + """ + if len(seq) == 0: + fmt.line('{}: &[],'.format(field)) + return + with fmt.indented('{}: &['.format(field), '],'): + for cons in seq: + with fmt.indented('OperandConstraint {', '},'): + if isinstance(cons, RegClass): + fmt.line('kind: ConstraintKind::Reg,') + fmt.line('regclass: {},'.format(cons)) + elif isinstance(cons, Register): + fmt.line( + 'kind: ConstraintKind::FixedReg({}),' + .format(cons.unit)) + fmt.line('regclass: {},'.format(cons.regclass)) + else: + raise AssertionError( + 'Unsupported constraint {}'.format(cons)) + + def gen_isa(isa, fmt): # First assign numbers to relevant instruction predicates and generate the # check_instp() function.. @@ -446,6 +496,7 @@ def gen_isa(isa, fmt): cpumode, level1_tables[cpumode], level1_offt, fmt) emit_recipe_names(isa, fmt) + emit_recipe_constraints(isa, fmt) def generate(isas, out_dir): diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index ae8d5b3add..fe0618d6cc 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -12,6 +12,7 @@ from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt from base.formats import Binary, BinaryImm +from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit # instructions have 11 as the two low bits, with bits 6:2 determining the base @@ -67,9 +68,11 @@ def OP32(funct3, funct7): # R-type 32-bit instructions: These are mostly binary arithmetic instructions. # The encbits are `opcode[6:2] | (funct3 << 5) | (funct7 << 8) -R = EncRecipe('R', Binary) +R = EncRecipe('R', Binary, ins=(GPR, GPR), outs=GPR) # R-type with an immediate shift amount instead of rs2. -Rshamt = EncRecipe('Rshamt', BinaryImm) +Rshamt = EncRecipe('Rshamt', BinaryImm, ins=GPR, outs=GPR) -I = EncRecipe('I', BinaryImm, instp=IsSignedInt(BinaryImm.imm, 12)) +I = EncRecipe( + 'I', BinaryImm, ins=GPR, outs=GPR, + instp=IsSignedInt(BinaryImm.imm, 12)) diff --git a/lib/cretonne/src/isa/arm32/enc_tables.rs b/lib/cretonne/src/isa/arm32/enc_tables.rs index e40362a32f..3c3ffae695 100644 --- a/lib/cretonne/src/isa/arm32/enc_tables.rs +++ b/lib/cretonne/src/isa/arm32/enc_tables.rs @@ -4,5 +4,6 @@ use ir::InstructionData; use ir::instructions::InstructionFormat; use ir::types; use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::constraints::*; include!(concat!(env!("OUT_DIR"), "/encoding-arm32.rs")); diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 617f52f43e..63f6293516 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -7,7 +7,7 @@ mod registers; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, Encoding, Legalize}; +use isa::{TargetIsa, RegInfo, Encoding, Legalize, RecipeConstraints}; use ir::{InstructionData, DataFlowGraph}; #[allow(dead_code)] @@ -70,4 +70,8 @@ impl TargetIsa for Isa { fn recipe_names(&self) -> &'static [&'static str] { &enc_tables::RECIPE_NAMES[..] } + + fn recipe_constraints(&self) -> &'static [RecipeConstraints] { + &enc_tables::RECIPE_CONSTRAINTS + } } diff --git a/lib/cretonne/src/isa/arm64/enc_tables.rs b/lib/cretonne/src/isa/arm64/enc_tables.rs index dbe65c4e8f..92b5ad58c3 100644 --- a/lib/cretonne/src/isa/arm64/enc_tables.rs +++ b/lib/cretonne/src/isa/arm64/enc_tables.rs @@ -4,5 +4,6 @@ use ir::InstructionData; use ir::instructions::InstructionFormat; use ir::types; use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::constraints::*; include!(concat!(env!("OUT_DIR"), "/encoding-arm64.rs")); diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index a4367a878b..2c2b98437b 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -7,7 +7,7 @@ mod registers; use super::super::settings as shared_settings; use isa::enc_tables::{lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, Encoding, Legalize}; +use isa::{TargetIsa, RegInfo, Encoding, Legalize, RecipeConstraints}; use ir::{InstructionData, DataFlowGraph}; #[allow(dead_code)] @@ -63,4 +63,8 @@ impl TargetIsa for Isa { fn recipe_names(&self) -> &'static [&'static str] { &enc_tables::RECIPE_NAMES[..] } + + fn recipe_constraints(&self) -> &'static [RecipeConstraints] { + &enc_tables::RECIPE_CONSTRAINTS + } } diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs new file mode 100644 index 0000000000..bdd2bea958 --- /dev/null +++ b/lib/cretonne/src/isa/constraints.rs @@ -0,0 +1,66 @@ +//! Register constraints for instruction operands. +//! +//! An encoding recipe specifies how an instruction is encoded as binary machine code, but it only +//! works if the operands and results satisfy certain constraints. Constraints on immediate +//! operands are checked by instruction predicates when the recipe is chosen. +//! +//! It is the register allocator's job to make sure that the register constraints on value operands +//! are satisfied. + +use isa::{RegClass, RegUnit}; + +/// Register constraint for a single value operand or instruction result. +pub struct OperandConstraint { + /// The kind of constraint. + pub kind: ConstraintKind, + + /// The register class of the operand. + /// + /// This applies to all kinds of constraints, but with slightly different meaning. + pub regclass: RegClass, +} + +/// The different kinds of operand constraints. +pub enum ConstraintKind { + /// This operand or result must be a register from the given register class. + Reg, + + /// This operand or result must be a fixed register. + /// + /// The constraint's `regclass` field is the top-level register class containing the fixed + /// register. + FixedReg(RegUnit), + + /// This result value must use the same register as an input value operand. Input operands + /// can't be tied. + /// + /// The associated number is the index of the input value operand this result is tied to. + /// + /// The constraint's `regclass` field is the top-level register class containing the tied + /// operand's register class. + Tied(u8), + + /// This operand must be a value in a stack slot. + /// + /// The constraint's `regclass` field is the register class that would normally be used to load + /// and store values of this type. + Stack, +} + +/// Constraints for an encoding recipe. +pub struct RecipeConstraints { + /// Constraints for the instruction's fixed value operands. + /// + /// If the instruction takes a variable number of operands, the register constraints for those + /// operands must be computed dynamically. + /// + /// - For branches and jumps, EBB arguments must match the expectations of the destination EBB. + /// - For calls and returns, the calling convention ABI specifies constraints. + pub ins: &'static [OperandConstraint], + + /// Constraints for the instruction's fixed results. + /// + /// If the instruction produces a variable number of results, it's probably a call and the + /// constraints must be derived from the calling convention ABI. + pub outs: &'static [OperandConstraint], +} diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 842629b058..0b01f481f2 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -4,5 +4,6 @@ use ir::InstructionData; use ir::instructions::InstructionFormat; use ir::types; use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::constraints::*; include!(concat!(env!("OUT_DIR"), "/encoding-intel.rs")); diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index acad76743b..13b77e1dd2 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -7,7 +7,7 @@ mod registers; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, Encoding, Legalize}; +use isa::{TargetIsa, RegInfo, Encoding, Legalize, RecipeConstraints}; use ir::{InstructionData, DataFlowGraph}; #[allow(dead_code)] @@ -70,4 +70,8 @@ impl TargetIsa for Isa { fn recipe_names(&self) -> &'static [&'static str] { &enc_tables::RECIPE_NAMES[..] } + + fn recipe_constraints(&self) -> &'static [RecipeConstraints] { + &enc_tables::RECIPE_CONSTRAINTS + } } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 6d22d5dd71..0bd9e5e9fa 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -42,6 +42,8 @@ pub use isa::encoding::Encoding; pub use isa::registers::{RegInfo, RegUnit, RegClass}; +pub use isa::constraints::RecipeConstraints; + use settings; use ir::{InstructionData, DataFlowGraph}; @@ -52,6 +54,7 @@ pub mod arm64; pub mod registers; mod encoding; mod enc_tables; +mod constraints; /// Look for a supported ISA with the given `name`. /// Return a builder that can create a corresponding `TargetIsa`. @@ -140,11 +143,17 @@ pub trait TargetIsa { fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Result; /// Get a static array of names associated with encoding recipes in this ISA. Encoding recipes - /// are numbered starting from 0, corresponding to indexes into th name array. + /// are numbered starting from 0, corresponding to indexes into the name array. /// /// This is just used for printing and parsing encodings in the textual IL format. fn recipe_names(&self) -> &'static [&'static str]; + /// Get a static array of value operand constraints associated with encoding recipes in this + /// ISA. + /// + /// The constraints describe which registers can be used with an encoding recipe. + fn recipe_constraints(&self) -> &'static [RecipeConstraints]; + /// Create an object that can display an ISA-dependent encoding properly. fn display_enc(&self, enc: Encoding) -> encoding::DisplayEncoding { encoding::DisplayEncoding { diff --git a/lib/cretonne/src/isa/riscv/enc_tables.rs b/lib/cretonne/src/isa/riscv/enc_tables.rs index f32a313c92..2538911a25 100644 --- a/lib/cretonne/src/isa/riscv/enc_tables.rs +++ b/lib/cretonne/src/isa/riscv/enc_tables.rs @@ -5,6 +5,8 @@ use ir::instructions::InstructionFormat; use ir::types; use predicates; use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::constraints::*; +use super::registers::*; // Include the generated encoding tables: // - `LEVEL1_RV32` diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 762df2249c..34b2a43c99 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -7,7 +7,7 @@ mod registers; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, Encoding, Legalize}; +use isa::{TargetIsa, RegInfo, Encoding, Legalize, RecipeConstraints}; use ir::{InstructionData, DataFlowGraph}; #[allow(dead_code)] @@ -70,6 +70,10 @@ impl TargetIsa for Isa { fn recipe_names(&self) -> &'static [&'static str] { &enc_tables::RECIPE_NAMES[..] } + + fn recipe_constraints(&self) -> &'static [RecipeConstraints] { + &enc_tables::RECIPE_CONSTRAINTS + } } #[cfg(test)] From 130c4acf516686a3a870861388468a2f87c07ba2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Jan 2017 13:57:43 -0800 Subject: [PATCH 0485/3084] Compute register class intersections. Ensure that the set of register classes is closed under intersection. Provide a RegClass::intersect() method which finds the register class representing the intersection of two classes. Generate a bit-mask of subclasses for each register class to be used by the intersect() method. Ensure that register classes are sorted topologically. This is also used by the intersect() method. --- lib/cretonne/meta/cdsl/isa.py | 13 +++ lib/cretonne/meta/cdsl/registers.py | 108 ++++++++++++++++++- lib/cretonne/meta/gen_registers.py | 21 ++-- lib/cretonne/src/isa/intel/registers.rs | 15 ++- lib/cretonne/src/isa/registers.rs | 54 ++++++++++ lib/cretonne/src/regalloc/allocatable_set.rs | 2 + 6 files changed, 200 insertions(+), 13 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 8e1423315c..0065e4e907 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -51,6 +51,7 @@ class TargetISA(object): """ self._collect_encoding_recipes() self._collect_predicates() + self._collect_regclasses() return self def _collect_encoding_recipes(self): @@ -96,6 +97,18 @@ class TargetISA(object): if enc.isap: self.settings.number_predicate(enc.isap) + def _collect_regclasses(self): + """ + Collect and number register classes. + + Every register class needs a unique index, and the classes need to be + topologically ordered. + """ + rc_index = 0 + for bank in self.regbanks: + bank.finish_regclasses(rc_index) + rc_index += len(bank.classes) + class CPUMode(object): """ diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index 828e6a2704..99dd453aec 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -26,8 +26,11 @@ from . import is_power_of_two, next_power_of_two try: - from typing import Sequence # noqa + from typing import Sequence, Tuple # noqa from .isa import TargetISA # noqa + # A tuple uniquely identifying a register class inside a register bank. + # (count, width, start) + RCTup = Tuple[int, int, int] except ImportError: pass @@ -90,6 +93,63 @@ class RegBank(object): return ('RegBank({}, units={}, first_unit={})' .format(self.name, self.units, self.first_unit)) + def finish_regclasses(self, first_index): + # type: (int) -> None + """ + Assign indexes to the register classes in this bank, starting from + `first_index`. + + Verify that the set of register classes satisfies: + + 1. Closed under intersection: The intersection of any two register + classes in the set is either empty or identical to a member of the + set. + 2. There are no identical classes under different names. + 3. Classes are sorted topologically such that all subclasses have a + higher index that the superclass. + + We could reorder classes topologically here instead of just enforcing + the order, but the ordering tends to fall out naturally anyway. + """ + cmap = dict() # type: Dict[RCTup, RegClass] + + for idx, rc in enumerate(self.classes): + # All register classes must be given a name. + assert rc.name, "Anonymous register class found" + + # Assign a unique index. + assert rc.index is None + rc.index = idx + first_index + + # Check for duplicates. + tup = rc.rctup() + if tup in cmap: + raise AssertionError( + '{} and {} are identical register classes' + .format(rc, cmap[tup])) + cmap[tup] = rc + + # Check intersections and topological order. + for idx, rc1 in enumerate(self.classes): + for rc2 in self.classes[0:idx]: + itup = rc1.intersect(rc2) + if itup is None: + continue + if itup not in cmap: + raise AssertionError( + 'intersection of {} and {} missing' + .format(rc1, rc2)) + irc = cmap[itup] + # rc1 > rc2, so rc2 can't be the sub-class. + if irc is rc2: + raise AssertionError( + 'Bad topological order: {}/{}' + .format(rc1, rc2)) + if irc is rc1: + # The intersection of rc1 and rc2 is rc1, so it must be a + # sub-class. + rc2.subclasses.append(rc1) + class RegClass(object): """ @@ -111,10 +171,14 @@ class RegClass(object): def __init__(self, bank, count=None, width=1, start=0): # type: (RegBank, int, int, int) -> None self.name = None # type: str + self.index = None # type: int self.bank = bank self.start = start self.width = width + # This is computed later in `finish_regclasses()`. + self.subclasses = list() # type: List[RegClass] + assert width > 0 assert start >= 0 and start < bank.units @@ -127,13 +191,43 @@ class RegClass(object): def __str__(self): return self.name + def rctup(self): + # type: () -> RCTup + """ + Get a tuple that uniquely identifies the registers in this class. + + The tuple can be used as a dictionary key to ensure that there are no + duplicate register classes. + """ + return (self.count, self.width, self.start) + + def intersect(self, other): + # type: (RegClass) -> RCTup + """ + Get a tuple representing the intersction of two register classes. + + Returns `None` if the two classes are disjoint. + """ + if self.width != other.width: + return None + s_end = self.start + self.count * self.width + o_end = other.start + other.count * other.width + if self.start >= o_end or other.start >= s_end: + return None + + # We have an overlap. + start = max(self.start, other.start) + end = min(s_end, o_end) + count = (end - start) // self.width + assert count > 0 + return (count, self.width, start) + def __getitem__(self, sliced): """ Create a sub-class of a register class using slice notation. The slice indexes refer to allocations in the parent register class, not register units. """ - print(repr(sliced)) assert isinstance(sliced, slice), "RegClass slicing can't be 1 reg" # We could add strided sub-classes if needed. assert sliced.step is None, 'Subclass striding not supported' @@ -167,6 +261,16 @@ class RegClass(object): return mask + def subclass_mask(self): + # type: () -> int + """ + Compute a bit-mask of subclasses, including self. + """ + m = 1 << self.index + for rc in self.subclasses: + m |= 1 << rc.index + return m + @staticmethod def extract_names(globs): """ diff --git a/lib/cretonne/meta/gen_registers.py b/lib/cretonne/meta/gen_registers.py index d3afe3ca4c..c535f44c22 100644 --- a/lib/cretonne/meta/gen_registers.py +++ b/lib/cretonne/meta/gen_registers.py @@ -28,15 +28,16 @@ def gen_regbank(regbank, fmt): fmt.line('prefix: "{}",'.format(regbank.prefix)) -def gen_regclass(idx, rc, fmt): - # type: (int, RegClass, srcgen.Formatter) -> None +def gen_regclass(rc, fmt): + # type: (RegClass, srcgen.Formatter) -> None """ Emit a static data definition for a register class. """ fmt.comment(rc.name) with fmt.indented('RegClassData {', '},'): - fmt.line('index: {},'.format(idx)) + fmt.line('index: {},'.format(rc.index)) fmt.line('width: {},'.format(rc.width)) + fmt.line('subclasses: 0x{:x},'.format(rc.subclass_mask())) mask = ', '.join('0x{:08x}'.format(x) for x in rc.mask()) fmt.line('mask: [{}],'.format(mask)) @@ -62,15 +63,15 @@ def gen_isa(isa, fmt): with fmt.indented( 'const CLASSES: [RegClassData; {}] = ['.format(len(rcs)), '];'): for idx, rc in enumerate(rcs): - gen_regclass(idx, rc, fmt) + assert idx == rc.index + gen_regclass(rc, fmt) # Emit constants referencing the register classes. - for idx, rc in enumerate(rcs): - if rc.name: - fmt.line('#[allow(dead_code)]') - fmt.line( - 'pub const {}: RegClass = &CLASSES[{}];' - .format(rc.name, idx)) + for rc in rcs: + fmt.line('#[allow(dead_code)]') + fmt.line( + 'pub const {}: RegClass = &CLASSES[{}];' + .format(rc.name, rc.index)) def generate(isas, out_dir): diff --git a/lib/cretonne/src/isa/intel/registers.rs b/lib/cretonne/src/isa/intel/registers.rs index 82c5b9886e..14b4050d94 100644 --- a/lib/cretonne/src/isa/intel/registers.rs +++ b/lib/cretonne/src/isa/intel/registers.rs @@ -6,7 +6,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-intel.rs")); #[cfg(test)] mod tests { - use super::INFO; + use super::*; use isa::RegUnit; #[test] @@ -46,4 +46,17 @@ mod tests { assert_eq!(uname(16), "%xmm0"); assert_eq!(uname(31), "%xmm15"); } + + #[test] + fn regclasses() { + assert_eq!(GPR.intersect(GPR), Some(GPR.into())); + assert_eq!(GPR.intersect(ABCD), Some(ABCD.into())); + assert_eq!(GPR.intersect(FPR), None); + assert_eq!(ABCD.intersect(GPR), Some(ABCD.into())); + assert_eq!(ABCD.intersect(ABCD), Some(ABCD.into())); + assert_eq!(ABCD.intersect(FPR), None); + assert_eq!(FPR.intersect(FPR), Some(FPR.into())); + assert_eq!(FPR.intersect(GPR), None); + assert_eq!(FPR.intersect(ABCD), None); + } } diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 48173c50bd..b223a3b5b0 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -1,5 +1,6 @@ //! Data structures describing the registers in an ISA. +use entity_map::EntityRef; use std::fmt; /// Register units are the smallest units of register allocation. @@ -106,11 +107,59 @@ pub struct RegClassData { /// How many register units to allocate per register. pub width: u8, + /// Bit-mask of sub-classes of this register class, including itself. + /// + /// Bits correspond to RC indexes. + pub subclasses: u32, + /// Mask of register units in the class. If `width > 1`, the mask only has a bit set for the /// first register unit in each allocatable register. pub mask: RegUnitMask, } +impl RegClassData { + /// Get the register class corresponding to the intersection of `self` and `other`. + /// + /// This register class is guaranteed to exist if the register classes overlap. If the register + /// classes don't overlap, returns `None`. + pub fn intersect(&self, other: RegClass) -> Option { + // Compute the set of common subclasses. + let mask = self.subclasses & other.subclasses; + + if mask == 0 { + // No overlap. + None + } else { + // Register class indexes are topologically ordered, so the largest common subclass has + // the smallest index. + Some(RegClassIndex(mask.trailing_zeros() as u8)) + } + } +} + +/// A small reference to a register class. +/// +/// Use this when storing register classes in compact data structures. The `RegInfo::rc()` method +/// can be used to get the real register class reference back. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct RegClassIndex(u8); + +impl EntityRef for RegClassIndex { + fn new(idx: usize) -> Self { + RegClassIndex(idx as u8) + } + + fn index(self) -> usize { + self.0 as usize + } +} + +impl From for RegClassIndex { + fn from(rc: RegClass) -> Self { + RegClassIndex(rc.index) + } +} + /// Information about the registers in an ISA. /// /// The `RegUnit` data structure collects all relevant static information about the registers in an @@ -143,6 +192,11 @@ impl RegInfo { reginfo: self, } } + + /// Get the register class corresponding to `idx`. + pub fn rc(&self, idx: RegClassIndex) -> RegClass { + &self.classes[idx.index()] + } } /// Temporary object that holds enough information to print a register unit. diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index 34145cd01a..f2ef0102d4 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -120,11 +120,13 @@ mod tests { const GPR: RegClass = &RegClassData { index: 0, width: 1, + subclasses: 0, mask: [0xf0000000, 0x0000000f, 0], }; const DPR: RegClass = &RegClassData { index: 0, width: 2, + subclasses: 0, mask: [0x50000000, 0x0000000a, 0], }; From 4a0d8aaa3d9d120cdd47cd89bd85e02817b39baf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Jan 2017 14:12:36 -0800 Subject: [PATCH 0486/3084] Run Python checks from test-all.sh The Python style enforcements are easy to miss otherwise. --- cranelift/test-all.sh | 3 +++ lib/cretonne/meta/check.sh | 12 +++--------- 2 files changed, 6 insertions(+), 9 deletions(-) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index f55464ce3b..bf2270efb6 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -40,6 +40,9 @@ else echo "If a newer version of rustfmt is available, update this script." fi +banner "Python checks" +$topdir/lib/cretonne/meta/check.sh + PKGS="cretonne cretonne-reader cretonne-tools filecheck" cd "$topdir" for PKG in $PKGS diff --git a/lib/cretonne/meta/check.sh b/lib/cretonne/meta/check.sh index 2077716eb7..655092e6cb 100755 --- a/lib/cretonne/meta/check.sh +++ b/lib/cretonne/meta/check.sh @@ -4,29 +4,23 @@ cd $(dirname "$0") runif() { if command -v "$1" > /dev/null; then - echo "=== $1 ===" + echo " === $1 ===" "$@" else echo "$1 not found" fi } -# Check Python sources for Python 3 compatibility using pylint. -# -# Install pylint with 'pip install pylint'. -runif pylint --py3k --reports=no -- *.py cdsl base cretonne isa - # Style linting. runif flake8 . # Type checking. runif mypy --py2 build.py -echo "=== Python unit tests ===" -python -m unittest discover +# Python unit tests. +runif python -m unittest discover # Then run the unit tests again with Python 3. # We get deprecation warnings about assertRaisesRegexp which was renamed in # Python 3, but there doesn't seem to be an easy workaround. runif python3 -Wignore:Deprecation -m unittest discover - From 3bbe3f71cbbc3227c433417ffc0c157004fbc65d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Jan 2017 14:20:22 -0800 Subject: [PATCH 0487/3084] Install mypy and flake8 in Travis environment. --- .travis.yml | 5 ++++- cranelift/test-all.sh | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 034d0e6c55..7bd682fcfc 100644 --- a/.travis.yml +++ b/.travis.yml @@ -3,5 +3,8 @@ rust: - stable - beta - nightly +install: pip install --upgrade mypy flake8 script: ./test-all.sh -cache: cargo +cache: + - cargo + - pip diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index bf2270efb6..ef85cd19ee 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -40,7 +40,7 @@ else echo "If a newer version of rustfmt is available, update this script." fi -banner "Python checks" +banner $(python --version 2>&1) $topdir/lib/cretonne/meta/check.sh PKGS="cretonne cretonne-reader cretonne-tools filecheck" From 4c5bca6b0dd520e21924fdc82f10e588cb8718b0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Jan 2017 14:26:28 -0800 Subject: [PATCH 0488/3084] Install Python packages without Travis root user. --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 7bd682fcfc..417d67855b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -3,7 +3,7 @@ rust: - stable - beta - nightly -install: pip install --upgrade mypy flake8 +install: pip install --user --upgrade mypy flake8 script: ./test-all.sh cache: - cargo From 91a7922474a3a318ba0985e5e6c223ddfda916f1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Jan 2017 14:32:52 -0800 Subject: [PATCH 0489/3084] Use Python 3.6 in Travis builds --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index 417d67855b..1f6844c672 100644 --- a/.travis.yml +++ b/.travis.yml @@ -3,6 +3,7 @@ rust: - stable - beta - nightly +python: "3.6" install: pip install --user --upgrade mypy flake8 script: ./test-all.sh cache: From 10c2f397a8ac1a61baa66de13bddad21e942a367 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Jan 2017 14:45:41 -0800 Subject: [PATCH 0490/3084] Pull in a python3 Ubuntu package for Travis CI. Then use pip3 to install dependencies. --- .travis.yml | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 1f6844c672..1910397b6c 100644 --- a/.travis.yml +++ b/.travis.yml @@ -3,8 +3,11 @@ rust: - stable - beta - nightly -python: "3.6" -install: pip install --user --upgrade mypy flake8 +addons: + apt: + packages: + - python3-pip +install: pip3 install --user --upgrade mypy flake8 script: ./test-all.sh cache: - cargo From 361d71a0ab9bba774f1b12fb14fc2beafa0d3919 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Jan 2017 14:51:48 -0800 Subject: [PATCH 0491/3084] The python3-pip package does not exist on Ubuntu 12.04 LTS. Try to go via python3-setuptools instead. --- .travis.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 1910397b6c..ebaf2199f7 100644 --- a/.travis.yml +++ b/.travis.yml @@ -6,8 +6,10 @@ rust: addons: apt: packages: - - python3-pip -install: pip3 install --user --upgrade mypy flake8 + - python3-setuptools +install: + - easy_install3 --user pip + - python3 -m pip install --user --upgrade mypy flake8 script: ./test-all.sh cache: - cargo From 70957cc7cedd76ae768699899779810c1bef5930 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Jan 2017 14:59:48 -0800 Subject: [PATCH 0492/3084] Doesn't work with 12.02 LTS's Python 3.2. Try switching to Trusty to get a newer Python 3. --- .travis.yml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.travis.yml b/.travis.yml index ebaf2199f7..38f42b15b9 100644 --- a/.travis.yml +++ b/.travis.yml @@ -3,13 +3,14 @@ rust: - stable - beta - nightly +dist: trusty +sudo: false addons: apt: packages: - - python3-setuptools + - python3-pip install: - - easy_install3 --user pip - - python3 -m pip install --user --upgrade mypy flake8 + - pip3 install --user --upgrade mypy flake8 script: ./test-all.sh cache: - cargo From 859cca081cfba6fd3e589effd1eb09e65b8d7190 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Jan 2017 15:17:27 -0800 Subject: [PATCH 0493/3084] Upgrade to rustfmt 0.7.1 --- cranelift/src/filetest/concurrent.rs | 6 ++--- cranelift/test-all.sh | 2 +- lib/cretonne/src/isa/registers.rs | 10 ++++---- lib/reader/src/sourcemap.rs | 34 +++++++++++++--------------- 4 files changed, 23 insertions(+), 29 deletions(-) diff --git a/cranelift/src/filetest/concurrent.rs b/cranelift/src/filetest/concurrent.rs index 30abb844d4..e0afc8bead 100644 --- a/cranelift/src/filetest/concurrent.rs +++ b/cranelift/src/filetest/concurrent.rs @@ -96,10 +96,8 @@ impl ConcurrentRunner { fn heartbeat_thread(replies: Sender) -> thread::JoinHandle<()> { thread::Builder::new() .name("heartbeat".to_string()) - .spawn(move || { - while replies.send(Reply::Tick).is_ok() { - thread::sleep(Duration::from_secs(1)); - } + .spawn(move || while replies.send(Reply::Tick).is_ok() { + thread::sleep(Duration::from_secs(1)); }) .unwrap() } diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index ef85cd19ee..8035703488 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -30,7 +30,7 @@ function banner() { # rustfmt is installed. # # This version should always be bumped to the newest version available. -RUSTFMT_VERSION="0.6.3" +RUSTFMT_VERSION="0.7.1" if cargo install --list | grep -q "^rustfmt v$RUSTFMT_VERSION"; then banner "Rust formatting" diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index b223a3b5b0..f8c2ee5731 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -66,12 +66,10 @@ impl RegBank { } } } - .and_then(|offset| { - if offset < self.units { - Some(offset + self.first_unit) - } else { - None - } + .and_then(|offset| if offset < self.units { + Some(offset + self.first_unit) + } else { + None }) } diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 26d874817d..369e732e90 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -62,25 +62,23 @@ impl SourceMap { /// Look up an entity by source name. /// Returns the entity reference corresponding to `name`, if it exists. pub fn lookup_str(&self, name: &str) -> Option { - split_entity_name(name).and_then(|(ent, num)| { - match ent { - "v" => { - Value::direct_with_number(num) - .and_then(|v| self.get_value(v)) - .map(AnyEntity::Value) - } - "vx" => { - Value::table_with_number(num) - .and_then(|v| self.get_value(v)) - .map(AnyEntity::Value) - } - "ebb" => Ebb::with_number(num).and_then(|e| self.get_ebb(e)).map(AnyEntity::Ebb), - "ss" => self.get_ss(num).map(AnyEntity::StackSlot), - "sig" => self.get_sig(num).map(AnyEntity::SigRef), - "fn" => self.get_fn(num).map(AnyEntity::FuncRef), - "jt" => self.get_jt(num).map(AnyEntity::JumpTable), - _ => None, + split_entity_name(name).and_then(|(ent, num)| match ent { + "v" => { + Value::direct_with_number(num) + .and_then(|v| self.get_value(v)) + .map(AnyEntity::Value) } + "vx" => { + Value::table_with_number(num) + .and_then(|v| self.get_value(v)) + .map(AnyEntity::Value) + } + "ebb" => Ebb::with_number(num).and_then(|e| self.get_ebb(e)).map(AnyEntity::Ebb), + "ss" => self.get_ss(num).map(AnyEntity::StackSlot), + "sig" => self.get_sig(num).map(AnyEntity::SigRef), + "fn" => self.get_fn(num).map(AnyEntity::FuncRef), + "jt" => self.get_jt(num).map(AnyEntity::JumpTable), + _ => None, }) } From eecbcf98445a5df3089b1236624d59e6afa59232 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Jan 2017 15:35:58 -0800 Subject: [PATCH 0494/3084] Add pip files to the cache. --- .travis.yml | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/.travis.yml b/.travis.yml index 38f42b15b9..f7b0f62d64 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,17 +1,18 @@ language: rust rust: - - stable - - beta - - nightly + - stable + - beta + - nightly dist: trusty sudo: false addons: - apt: - packages: - - python3-pip + apt: + packages: + - python3-pip install: - - pip3 install --user --upgrade mypy flake8 + - pip3 install --user --upgrade mypy flake8 script: ./test-all.sh cache: - - cargo - - pip + cargo: true + directories: + - $HOME/.cache/pip From 38aff37c1eecc6b4cc6b86f8a6a1da3325577eae Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Jan 2017 15:42:16 -0800 Subject: [PATCH 0495/3084] Install rustfmt when running under Travis CI. The built rustfmt should be cached. --- cranelift/test-all.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 8035703488..aebec1a872 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -35,6 +35,12 @@ RUSTFMT_VERSION="0.7.1" if cargo install --list | grep -q "^rustfmt v$RUSTFMT_VERSION"; then banner "Rust formatting" $topdir/format-all.sh --write-mode=diff +elif [ -n "$TRAVIS" ]; then + # We're running under Travis CI. + # Install rustfmt, it will be cached for the next build. + echo "Installing rustfmt v$RUSTFMT_VERSION." + cargo install --force --vers="$RUSTFMT_VERSION" rustfmt + $topdir/format-all.sh --write-mode=diff else echo "Please install rustfmt v$RUSTFMT_VERSION to verify formatting." echo "If a newer version of rustfmt is available, update this script." From 38bb98cf395665eb9c242a49a25e498b910bb2dd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Jan 2017 15:57:43 -0800 Subject: [PATCH 0496/3084] Make sure we can find rustfmt. --- cranelift/format-all.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/cranelift/format-all.sh b/cranelift/format-all.sh index 7d51b7d343..d9e97f1026 100755 --- a/cranelift/format-all.sh +++ b/cranelift/format-all.sh @@ -8,6 +8,9 @@ set -e cd $(dirname "$0") src=$(pwd) +# Make sure we can find rustfmt. +export PATH="$PATH:$HOME/.cargo/bin" + for crate in $(find "$src" -name Cargo.toml); do cd $(dirname "$crate") cargo fmt -- "$@" From c767f277fa187321ed2789e29772e609ecba6ef6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Jan 2017 16:35:28 -0800 Subject: [PATCH 0497/3084] Stop testing on nightly rust The nightly compiler isn't able to compile rustfmt in 10 minutes. This causes Travis CI to terminate the build. We keep testing on beta and stable. --- .travis.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index f7b0f62d64..1986553bf1 100644 --- a/.travis.yml +++ b/.travis.yml @@ -2,7 +2,6 @@ language: rust rust: - stable - beta - - nightly dist: trusty sudo: false addons: From fd3cd153eddc7426892c3f08fda7d26fa1ff907e Mon Sep 17 00:00:00 2001 From: Andrea Canciani Date: Fri, 27 Jan 2017 18:42:05 +0100 Subject: [PATCH 0498/3084] Fix some typos in the documentation These were found by the spellchecker. --- cranelift/docs/langref.rst | 2 +- cranelift/docs/regalloc.rst | 2 +- cranelift/docs/testing.rst | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 40cc65dbe3..5bd59b18d3 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -882,7 +882,7 @@ Glossary - Type and flags of each argument. - Type and flags of each return value. - Not all function atributes are part of the signature. For example, a + Not all function attributes are part of the signature. For example, a function that never returns could be marked as ``noreturn``, but that is not necessary to know when calling it, so it is just an attribute, and not part of the signature. diff --git a/cranelift/docs/regalloc.rst b/cranelift/docs/regalloc.rst index 1d662e9a63..75477e7375 100644 --- a/cranelift/docs/regalloc.rst +++ b/cranelift/docs/regalloc.rst @@ -206,7 +206,7 @@ top-down order, and each value define by the instruction is assigned an available register. With this iteration order, every value that is live at an instruction has already been assigned to a register. -This coloring algorith works if the following condition holds: +This coloring algorithm works if the following condition holds: At every instruction, consider the values live through the instruction. No matter how the live values have been assigned to registers, there must be diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index d52b180c99..a452dc1b58 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -139,7 +139,7 @@ run will also have the RISC-V specific flag ``supports_m`` disabled. Filecheck --------- -Many of the test commands bescribed below use *filecheck* to verify their +Many of the test commands described below use *filecheck* to verify their output. Filecheck is a Rust implementation of the LLVM tool of the same name. See the :file:`lib/filecheck` `documentation `_ for details of its syntax. From 16f4b4c7d5e04d55391f8f76fcb0df9d05d7dd38 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 26 Jan 2017 14:51:49 -0800 Subject: [PATCH 0499/3084] Implement value affinities for register allocation. An SSA value is usually biased towards a specific register class or a stack slot, depending on the constraints of the instructions using it. Represent this bias as an Affinity enum, and implement a merging algorithm for updating an affinity to satisfy a new constraint. Affinities will be computed as part of the liveness analysis. This is not implemented yet. --- cranelift/.gitignore | 1 + lib/cretonne/src/isa/constraints.rs | 1 + lib/cretonne/src/isa/mod.rs | 4 +- lib/cretonne/src/isa/registers.rs | 6 +++ lib/cretonne/src/regalloc/affinity.rs | 68 ++++++++++++++++++++++++++ lib/cretonne/src/regalloc/liverange.rs | 15 ++++-- lib/cretonne/src/regalloc/mod.rs | 2 + 7 files changed, 90 insertions(+), 7 deletions(-) create mode 100644 lib/cretonne/src/regalloc/affinity.rs diff --git a/cranelift/.gitignore b/cranelift/.gitignore index 9ceaac3ed5..765772864e 100644 --- a/cranelift/.gitignore +++ b/cranelift/.gitignore @@ -5,3 +5,4 @@ tags target Cargo.lock +.*.rustfmt diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index bdd2bea958..43a0ddb271 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -21,6 +21,7 @@ pub struct OperandConstraint { } /// The different kinds of operand constraints. +#[derive(Clone, Copy, PartialEq, Eq)] pub enum ConstraintKind { /// This operand or result must be a register from the given register class. Reg, diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 0bd9e5e9fa..ee33bb4c47 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -41,8 +41,8 @@ //! concurrent function compilations. pub use isa::encoding::Encoding; -pub use isa::registers::{RegInfo, RegUnit, RegClass}; -pub use isa::constraints::RecipeConstraints; +pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex}; +pub use isa::constraints::{RecipeConstraints, OperandConstraint, ConstraintKind}; use settings; use ir::{InstructionData, DataFlowGraph}; diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index f8c2ee5731..5e89110284 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -133,6 +133,12 @@ impl RegClassData { Some(RegClassIndex(mask.trailing_zeros() as u8)) } } + + /// Returns true if `other` is a subclass of this register class. + /// A register class is considerd to be a subclass of itself. + pub fn has_subclass>(&self, other: RCI) -> bool { + self.subclasses & (1 << other.into().0) != 0 + } } /// A small reference to a register class. diff --git a/lib/cretonne/src/regalloc/affinity.rs b/lib/cretonne/src/regalloc/affinity.rs new file mode 100644 index 0000000000..529828759e --- /dev/null +++ b/lib/cretonne/src/regalloc/affinity.rs @@ -0,0 +1,68 @@ +//! Value affinity for register allocation. +//! +//! An SSA value's affinity is a hint used to guide the register allocator. It specifies the class +//! of allocation that is likely to cause the least amount of fixup moves in order to satisfy +//! instruction operand constraints. +//! +//! For values that want to be in registers, the affinity hint includes a register class or +//! subclass. This is just a hint, and the register allocator is allowed to pick a register from a +//! larger register class instead. + +use isa::{RegInfo, RegClassIndex, OperandConstraint, ConstraintKind}; + +/// Preferred register allocation for an SSA value. +#[derive(Clone, Copy)] +pub enum Affinity { + /// Don't care. This value can go anywhere. + Any, + + /// This value should be placed in a spill slot on the stack. + Stack, + + /// This value prefers a register from the given register class. + Reg(RegClassIndex), +} + +impl Default for Affinity { + fn default() -> Self { + Affinity::Any + } +} + +impl Affinity { + /// Create an affinity that satisfies a single constraint. + /// + /// This will never create the indifferent `Affinity::Any`. + /// Use the `Default` implementation for that. + pub fn new(constraint: &OperandConstraint) -> Affinity { + if constraint.kind == ConstraintKind::Stack { + Affinity::Stack + } else { + Affinity::Reg(constraint.regclass.into()) + } + } + + /// Merge an operand constraint into this affinity. + /// + /// Note that this does not guarantee that the register allocator will pick a register that + /// satisfies the constraint. + pub fn merge(&mut self, constraint: &OperandConstraint, reg_info: &RegInfo) { + match *self { + Affinity::Any => *self = Affinity::new(constraint), + Affinity::Reg(rc) => { + // If the preferred register class is a subclass of the constraint, there's no need + // to change anything. + if constraint.kind != ConstraintKind::Stack && + !constraint.regclass.has_subclass(rc) { + // If the register classes don't overlap, `intersect` returns `None`, and we + // just keep our previous affinity. + if let Some(subclass) = constraint.regclass.intersect(reg_info.rc(rc)) { + // This constraint shrinks our preferred register class. + *self = Affinity::Reg(subclass); + } + } + } + Affinity::Stack => {} + } + } +} diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index e192efdc4d..67667a3a47 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -73,9 +73,9 @@ //! 3. A numerical order by EBB number. Performant because it doesn't need to indirect through the //! `ProgramOrder` for comparisons. //! -//! These orderings will cause small differences in coalescing opportinities, but all of them would +//! These orderings will cause small differences in coalescing opportunities, but all of them would //! do a decent job of compressing a long live range. The numerical order might be preferable -//! beacuse: +//! because: //! //! - It has better performance because EBB numbers can be compared directly without any table //! lookups. @@ -92,7 +92,7 @@ //! //! Coalescing is an important compression technique because some live ranges can span thousands of //! EBBs. We can represent that by switching to a sorted `Vec` representation where -//! an `[Ebb, Inst]` pair represents a coalesced range, while an `Inst` entry without a preceeding +//! an `[Ebb, Inst]` pair represents a coalesced range, while an `Inst` entry without a preceding //! `Ebb` entry represents a single live-in interval. //! //! This representation is more compact for a live range with many uncoalesced live-in intervals. @@ -109,6 +109,7 @@ use std::cmp::Ordering; use ir::{Inst, Ebb, Value, ProgramPoint, ProgramOrder}; +use regalloc::affinity::Affinity; use sparse_map::SparseMapValue; /// Global live range of a single SSA value. @@ -143,6 +144,9 @@ pub struct LiveRange { /// This member can't be modified in case the live range is stored in a `SparseMap`. value: Value, + /// The preferred register allocation for this value. + pub affinity: Affinity, + /// The instruction or EBB header where this value is defined. def_begin: ProgramPoint, @@ -167,7 +171,7 @@ pub struct LiveRange { /// An additional contiguous interval of a global live range. /// /// This represents a live-in interval for a single EBB, or a coalesced set of live-in intervals -/// for contiguous EBBs where all but the last live-in inteval covers the whole EBB. +/// for contiguous EBBs where all but the last live-in interval covers the whole EBB. /// struct Interval { /// Interval starting point. @@ -205,6 +209,7 @@ impl LiveRange { let def = def.into(); LiveRange { value: value, + affinity: Default::default(), def_begin: def, def_end: def, liveins: Vec::new(), @@ -238,7 +243,7 @@ impl LiveRange { /// is live-in to `ebb`, extending to `to`. Return true. /// /// The return value can be used to detect if we just learned that the value is live-in to - /// `ebb`. This can trigger recursive extensions in `ebb`'s CFG precedessor blocks. + /// `ebb`. This can trigger recursive extensions in `ebb`'s CFG predecessor blocks. pub fn extend_in_ebb(&mut self, ebb: Ebb, to: Inst, order: &PO) -> bool { // First check if we're extending the def interval. // diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index 9e0f7b820d..bd1c34cd23 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -5,3 +5,5 @@ pub mod liverange; pub mod liveness; pub mod allocatable_set; + +mod affinity; From dab96d8ea29efed59f956ce4f1a653f13793ae71 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 27 Jan 2017 14:31:14 -0800 Subject: [PATCH 0500/3084] Add entity lists. Like a vector, but with a tiny footprint, and allocated from a pool so all memory can be released very quickly. --- lib/cretonne/src/entity_list.rs | 519 ++++++++++++++++++++++++++++++++ lib/cretonne/src/lib.rs | 1 + 2 files changed, 520 insertions(+) create mode 100644 lib/cretonne/src/entity_list.rs diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs new file mode 100644 index 0000000000..6ec23e39e2 --- /dev/null +++ b/lib/cretonne/src/entity_list.rs @@ -0,0 +1,519 @@ +//! Small lists of entity references. +//! +//! This module defines an `EntityList` type which provides similar functionality to `Vec`, +//! but with some important differences in the implementation: +//! +//! 1. Memory is allocated from a `ListPool` instead of the global heap. +//! 2. The footprint of an entity list is 4 bytes, compared with the 24 bytes for `Vec`. +//! 3. An entity list doesn't implement `Drop`, leaving it to the pool to manage memory. +//! +//! The list pool is intended to be used as a LIFO allocator. After building up a larger data +//! structure with many list references, the whole thing can be discarded quickly by clearing the +//! pool. +//! +//! # Safety +//! +//! Entity lists are not as safe to use as `Vec`, but they never jeopardize Rust's memory safety +//! guarantees. These are the problems to be aware of: +//! +//! - If you lose track of an entity list, it's memory won't be recycled until the pool is cleared. +//! This can cause the pool to grow very large with leaked lists. +//! - If entity lists are used after their pool is cleared, they may contain garbage data, and +//! modifying them may corrupt other lists in the pool. +//! - If an entity list is used with two different pool instances, both pools are likely to become +//! corrupted. +//! +//! # Implementation +//! +//! The `EntityList` itself is designed to have the smallest possible footprint. This is important +//! because it is used inside very compact data structures like `InstructionData`. The list +//! contains only a 32-bit index into the pool's memory vector, pointing to the first element of +//! the list. +//! +//! The pool is just a single `Vec` containing all of the allocated lists. Each list is +//! represented as three contiguous parts: +//! +//! 1. The number of elements in the list. +//! 2. The list elements. +//! 3. Excess capacity elements. +//! +//! The total size of the three parts is always a power of two, and the excess capacity is always +//! as small as possible. This means that shrinking a list may cause the excess capacity to shrink +//! if a smaller power-of-two size becomes available. +//! +//! Both growing and shrinking a list may cause it to be reallocated in the pool vector. +//! +//! The index stored in an `EntityList` points to part 2, the list elements. The value 0 is +//! reserved for the empty list which isn't allocated in the vector. + +use std::marker::PhantomData; + +use entity_map::EntityRef; + +/// A small list of entity references allocated from a pool. +/// +/// All of the list methods that take a pool reference must be given the same pool reference every +/// time they are called. Otherwise data structures will be corrupted. +pub struct EntityList { + index: u32, + unused: PhantomData, +} + +/// Create an empty list. +impl Default for EntityList { + fn default() -> Self { + EntityList { + index: 0, + unused: PhantomData, + } + } +} + +/// A memory pool for storing lists of `T`. +pub struct ListPool { + // The main array containing the lists. + data: Vec, + + // Heads of the free lists, one for each size class. + free: Vec, +} + +/// Lists are allocated in sizes that are powers of two, starting from 4. +/// Each power of two is assigned a size class number, so the size is `4 << SizeClass`. +type SizeClass = u8; + +/// Get the size of a given size class. The size includes the length field, so the maximum list +/// length is one less than the class size. +fn sclass_size(sclass: SizeClass) -> usize { + 4 << sclass +} + +/// Get the size class to use for a given list length. +/// This always leaves room for the length element in addition to the list elements. +fn sclass_for_length(len: usize) -> SizeClass { + 30 - (len as u32 | 3).leading_zeros() as SizeClass +} + +/// Is `len` the minimum length in its size class? +fn is_sclass_min_length(len: usize) -> bool { + len > 3 && len.is_power_of_two() +} + +impl ListPool { + /// Create a new list pool. + pub fn new() -> ListPool { + ListPool { + data: Vec::new(), + free: Vec::new(), + } + } + + /// Clear the pool, forgetting about all lists that use it. + /// + /// This invalidates any existing entity lists that used this pool to allocate memory. + /// + /// The pool's memory is not released to the operating system, but kept around for faster + /// allocation in the future. + pub fn clear(&mut self) { + self.data.clear(); + self.free.clear(); + } + + /// Read the length of a list field, if it exists. + fn len_of(&self, list: &EntityList) -> Option { + let idx = list.index as usize; + // `idx` points at the list elements. The list length is encoded in the element immediately + // before the list elements. + // + // The `wrapping_sub` handles the special case 0, which is the empty list. This way, the + // cost of the bounds check that we have to pay anyway is co-opted to handle the special + // case of the empty list. + self.data.get(idx.wrapping_sub(1)).map(|len| len.index()) + } + + /// Allocate a storage block with a size given by `sclass`. + /// + /// Returns the first index of an available segment of `self.data` containing + /// `sclass_size(sclass)` elements. + fn alloc(&mut self, sclass: SizeClass) -> usize { + // First try the free list for this size class. + match self.free.get(sclass as usize).cloned() { + Some(head) if head > 0 => { + // The free list pointers are offset by 1, using 0 to terminate the list. + // A block on the free list has two entries: `[ 0, next ]`. + // The 0 is where the length field would be stored for a block in use. + // The free list heads and the next pointer point at the `next` field. + self.free[sclass as usize] = self.data[head].index(); + head - 1 + } + _ => { + // Nothing on the free list. Allocate more memory. + let offset = self.data.len(); + // We don't want to mess around with uninitialized data. + // Just fill it up with nulls. + self.data.resize(offset + sclass_size(sclass), T::new(0)); + offset + } + } + } + + /// Free a storage block with a size given by `sclass`. + /// + /// This must be a block that was previously allocated by `alloc()` with the same size class. + fn free(&mut self, block: usize, sclass: SizeClass) { + let sclass = sclass as usize; + + // Make sure we have a free-list head for `sclass`. + if self.free.len() <= sclass { + self.free.resize(sclass + 1, 0); + } + + // Make sure the length field is cleared. + self.data[block] = T::new(0); + // Insert the block on the free list which is a single linked list. + self.data[block + 1] = T::new(self.free[sclass]); + self.free[sclass] = block + 1 + } + + /// Returns two mutable slices representing the two requested blocks. + /// + /// The two returned slices can be longer than the blocks. Each block is located at the front + /// the the respective slice. + fn mut_slices(&mut self, block0: usize, block1: usize) -> (&mut [T], &mut [T]) { + if block0 < block1 { + let (s0, s1) = self.data.split_at_mut(block1); + (&mut s0[block0..], s1) + } else { + let (s1, s0) = self.data.split_at_mut(block0); + (s0, &mut s1[block1..]) + } + } + + /// Reallocate a block to a different size class. + /// + /// Copy `elems_to_copy` elements from the old to the new block. + fn realloc(&mut self, + block: usize, + from_sclass: SizeClass, + to_sclass: SizeClass, + elems_to_copy: usize) + -> usize { + assert!(elems_to_copy <= sclass_size(from_sclass)); + assert!(elems_to_copy <= sclass_size(to_sclass)); + let new_block = self.alloc(to_sclass); + + if elems_to_copy > 0 { + let (old, new) = self.mut_slices(block, new_block); + (&mut new[0..elems_to_copy]).copy_from_slice(&old[0..elems_to_copy]); + } + + self.free(block, from_sclass); + new_block + } +} + +impl EntityList { + /// Returns `true` if the list has a length of 0. + pub fn is_empty(&self) -> bool { + // 0 is a magic value for the empty list. Any list in the pool array must have a positive + // length. + self.index == 0 + } + + /// Get the number of elements in the list. + pub fn len(&self, pool: &ListPool) -> usize { + // Both the empty list and any invalidated old lists will return `None`. + pool.len_of(self).unwrap_or(0) + } + + /// Get the list as a slice. + pub fn as_slice<'a>(&'a self, pool: &'a ListPool) -> &'a [T] { + let idx = self.index as usize; + match pool.len_of(self) { + None => &[], + Some(len) => &pool.data[idx..idx + len], + } + } + + /// Get a single element from the list. + pub fn get(&self, index: usize, pool: &ListPool) -> Option { + self.as_slice(pool).get(index).cloned() + } + + /// Get the list as a mutable slice. + pub fn as_mut_slice<'a>(&'a mut self, pool: &'a mut ListPool) -> &'a mut [T] { + let idx = self.index as usize; + match pool.len_of(self) { + None => &mut [], + Some(len) => &mut pool.data[idx..idx + len], + } + } + + /// Get a mutable reference to a single element from the list. + pub fn get_mut<'a>(&'a mut self, index: usize, pool: &'a mut ListPool) -> Option<&'a mut T> { + self.as_mut_slice(pool).get_mut(index) + } + + /// Removes all elements from the list. + /// + /// The memory used by the list is put back in the pool. + pub fn clear(&mut self, pool: &mut ListPool) { + let idx = self.index as usize; + match pool.len_of(self) { + None => assert_eq!(idx, 0, "Invalid pool"), + Some(len) => pool.free(idx - 1, sclass_for_length(len)), + } + // Switch back to the empty list representation which has no storage. + self.index = 0; + } + + /// Appends an element to the back of the list. + pub fn push(&mut self, element: T, pool: &mut ListPool) { + let idx = self.index as usize; + match pool.len_of(self) { + None => { + // This is an empty list. Allocate a block and set length=1. + assert_eq!(idx, 0, "Invalid pool"); + let block = pool.alloc(sclass_for_length(1)); + pool.data[block] = T::new(1); + pool.data[block + 1] = element; + self.index = (block + 1) as u32; + } + Some(len) => { + // Do we need to reallocate? + let new_len = len + 1; + let block; + if is_sclass_min_length(new_len) { + // Reallocate, preserving length + all old elements. + let sclass = sclass_for_length(len); + block = pool.realloc(idx - 1, sclass, sclass + 1, len + 1); + self.index = (block + 1) as u32; + } else { + block = idx - 1; + } + pool.data[block + new_len] = element; + pool.data[block] = T::new(new_len); + } + } + } + + /// Appends multiple elements to the back of the list. + pub fn extend(&mut self, elements: I, pool: &mut ListPool) + where I: IntoIterator + { + // TODO: use `size_hint()` to reduce reallocations. + for x in elements { + self.push(x, pool); + } + } + + /// Inserts an element as position `index` in the list, shifting all elements after it to the + /// right. + pub fn insert(&mut self, index: usize, element: T, pool: &mut ListPool) { + // Increase size by 1. + self.push(element, pool); + + // Move tail elements. + let seq = self.as_mut_slice(pool); + if index < seq.len() { + let tail = &mut seq[index..]; + for i in (1..tail.len()).rev() { + tail[i] = tail[i - 1]; + } + tail[0] = element; + } else { + assert_eq!(index, seq.len()); + } + } + + /// Removes the element at position `index` from the list. + pub fn remove(&mut self, index: usize, pool: &mut ListPool) { + let len; + { + let seq = self.as_mut_slice(pool); + len = seq.len(); + assert!(index < len); + + // Copy elements down. + for i in index..len - 1 { + seq[i] = seq[i + 1]; + } + } + + // Check if we deleted the last element. + if len == 1 { + self.clear(pool); + return; + } + + // Do we need to reallocate to a smaller size class? + let mut block = self.index as usize - 1; + if is_sclass_min_length(len) { + let sclass = sclass_for_length(len); + block = pool.realloc(block, sclass, sclass - 1, len); + self.index = (block + 1) as u32; + } + + // Finally adjust the length. + pool.data[block] = T::new(len - 1); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use super::{sclass_size, sclass_for_length}; + use ir::Inst; + use entity_map::EntityRef; + + #[test] + fn size_classes() { + assert_eq!(sclass_size(0), 4); + assert_eq!(sclass_for_length(0), 0); + assert_eq!(sclass_for_length(1), 0); + assert_eq!(sclass_for_length(2), 0); + assert_eq!(sclass_for_length(3), 0); + assert_eq!(sclass_for_length(4), 1); + assert_eq!(sclass_for_length(7), 1); + assert_eq!(sclass_for_length(8), 2); + assert_eq!(sclass_size(1), 8); + for l in 0..300 { + assert!(sclass_size(sclass_for_length(l)) >= l + 1); + } + } + + #[test] + fn block_allocator() { + let mut pool = ListPool::::new(); + let b1 = pool.alloc(0); + let b2 = pool.alloc(1); + let b3 = pool.alloc(0); + assert_ne!(b1, b2); + assert_ne!(b1, b3); + assert_ne!(b2, b3); + pool.free(b2, 1); + let b2a = pool.alloc(1); + let b2b = pool.alloc(1); + assert_ne!(b2a, b2b); + // One of these should reuse the freed block. + assert!(b2a == b2 || b2b == b2); + + // Check the free lists for a size class smaller than the largest seen so far. + pool.free(b1, 0); + pool.free(b3, 0); + let b1a = pool.alloc(0); + let b3a = pool.alloc(0); + assert_ne!(b1a, b3a); + assert!(b1a == b1 || b1a == b3); + assert!(b3a == b1 || b3a == b3); + } + + #[test] + fn empty_list() { + let pool = &mut ListPool::::new(); + let mut list = EntityList::::default(); + { + let ilist = &list; + assert!(ilist.is_empty()); + assert_eq!(ilist.len(pool), 0); + assert_eq!(ilist.as_slice(pool), &[]); + assert_eq!(ilist.get(0, pool), None); + assert_eq!(ilist.get(100, pool), None); + } + assert_eq!(list.as_mut_slice(pool), &[]); + assert_eq!(list.get_mut(0, pool), None); + assert_eq!(list.get_mut(100, pool), None); + + list.clear(pool); + assert!(list.is_empty()); + assert_eq!(list.len(pool), 0); + assert_eq!(list.as_slice(pool), &[]); + } + + #[test] + fn push() { + let pool = &mut ListPool::::new(); + let mut list = EntityList::::default(); + + let i1 = Inst::new(1); + let i2 = Inst::new(2); + let i3 = Inst::new(3); + let i4 = Inst::new(4); + + list.push(i1, pool); + assert_eq!(list.len(pool), 1); + assert!(!list.is_empty()); + assert_eq!(list.as_slice(pool), &[i1]); + assert_eq!(list.get(0, pool), Some(i1)); + assert_eq!(list.get(1, pool), None); + + list.push(i2, pool); + assert_eq!(list.len(pool), 2); + assert!(!list.is_empty()); + assert_eq!(list.as_slice(pool), &[i1, i2]); + assert_eq!(list.get(0, pool), Some(i1)); + assert_eq!(list.get(1, pool), Some(i2)); + assert_eq!(list.get(2, pool), None); + + list.push(i3, pool); + assert_eq!(list.len(pool), 3); + assert!(!list.is_empty()); + assert_eq!(list.as_slice(pool), &[i1, i2, i3]); + assert_eq!(list.get(0, pool), Some(i1)); + assert_eq!(list.get(1, pool), Some(i2)); + assert_eq!(list.get(2, pool), Some(i3)); + assert_eq!(list.get(3, pool), None); + + // This triggers a reallocation. + list.push(i4, pool); + assert_eq!(list.len(pool), 4); + assert!(!list.is_empty()); + assert_eq!(list.as_slice(pool), &[i1, i2, i3, i4]); + assert_eq!(list.get(0, pool), Some(i1)); + assert_eq!(list.get(1, pool), Some(i2)); + assert_eq!(list.get(2, pool), Some(i3)); + assert_eq!(list.get(3, pool), Some(i4)); + assert_eq!(list.get(4, pool), None); + + list.extend([i1, i1, i2, i2, i3, i3, i4, i4].iter().cloned(), pool); + assert_eq!(list.len(pool), 12); + assert_eq!(list.as_slice(pool), + &[i1, i2, i3, i4, i1, i1, i2, i2, i3, i3, i4, i4]); + } + + #[test] + fn insert_remove() { + let pool = &mut ListPool::::new(); + let mut list = EntityList::::default(); + + let i1 = Inst::new(1); + let i2 = Inst::new(2); + let i3 = Inst::new(3); + let i4 = Inst::new(4); + + list.insert(0, i4, pool); + assert_eq!(list.as_slice(pool), &[i4]); + + list.insert(0, i3, pool); + assert_eq!(list.as_slice(pool), &[i3, i4]); + + list.insert(2, i2, pool); + assert_eq!(list.as_slice(pool), &[i3, i4, i2]); + + list.insert(2, i1, pool); + assert_eq!(list.as_slice(pool), &[i3, i4, i1, i2]); + + list.remove(3, pool); + assert_eq!(list.as_slice(pool), &[i3, i4, i1]); + + list.remove(2, pool); + assert_eq!(list.as_slice(pool), &[i3, i4]); + + list.remove(0, pool); + assert_eq!(list.as_slice(pool), &[i4]); + + list.remove(0, pool); + assert_eq!(list.as_slice(pool), &[]); + assert!(list.is_empty()); + } +} diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 25feb3c0db..436e2829e3 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -14,6 +14,7 @@ pub mod isa; pub mod cfg; pub mod dominator_tree; pub mod entity_map; +pub mod entity_list; pub mod sparse_map; pub mod settings; pub mod verifier; From 933dfc70c1e56235b61052bc5005d8ccec3706f0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 3 Feb 2017 11:25:27 -0800 Subject: [PATCH 0501/3084] Fix a dead code warning from the new Rust compiler. On ISAs with no instruction predicates, just emit an unimplemented!() stub for the check_instp() function. It is unlikely that a finished ISA will not have any instruction predicates. --- lib/cretonne/meta/gen_encoding.py | 12 ++++++++++-- lib/cretonne/src/isa/arm32/enc_tables.rs | 1 - lib/cretonne/src/isa/arm64/enc_tables.rs | 1 - lib/cretonne/src/isa/intel/enc_tables.rs | 1 - lib/cretonne/src/isa/riscv/enc_tables.rs | 1 - 5 files changed, 10 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 30d1ed3370..3d4a00c4cc 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -75,7 +75,7 @@ def emit_instp(instp, fmt): """ iform = instp.predicate_context() - # Which fiels do we need in the InstructionData pattern match? + # Which fields do we need in the InstructionData pattern match? if iform.boxed_storage: fields = 'ref data' else: @@ -99,10 +99,18 @@ def emit_instps(instps, fmt): """ if not instps: - fmt.line('#[allow(unused_variables)]') + # If the ISA has no predicates, just emit a stub. + with fmt.indented( + 'pub fn check_instp(_: &InstructionData, _: u16) ' + + '-> bool {', '}'): + fmt.line('unimplemented!()') + return + with fmt.indented( 'pub fn check_instp(inst: &InstructionData, instp_idx: u16) ' + '-> bool {', '}'): + # The matches emitted by `emit_instp` need this. + fmt.line('use ir::instructions::InstructionFormat;') with fmt.indented('match instp_idx {', '}'): for instp in instps: emit_instp(instp, fmt) diff --git a/lib/cretonne/src/isa/arm32/enc_tables.rs b/lib/cretonne/src/isa/arm32/enc_tables.rs index 3c3ffae695..c464d92f23 100644 --- a/lib/cretonne/src/isa/arm32/enc_tables.rs +++ b/lib/cretonne/src/isa/arm32/enc_tables.rs @@ -1,7 +1,6 @@ //! Encoding tables for ARM32 ISA. use ir::InstructionData; -use ir::instructions::InstructionFormat; use ir::types; use isa::enc_tables::{Level1Entry, Level2Entry}; use isa::constraints::*; diff --git a/lib/cretonne/src/isa/arm64/enc_tables.rs b/lib/cretonne/src/isa/arm64/enc_tables.rs index 92b5ad58c3..064a00f2e2 100644 --- a/lib/cretonne/src/isa/arm64/enc_tables.rs +++ b/lib/cretonne/src/isa/arm64/enc_tables.rs @@ -1,7 +1,6 @@ //! Encoding tables for ARM64 ISA. use ir::InstructionData; -use ir::instructions::InstructionFormat; use ir::types; use isa::enc_tables::{Level1Entry, Level2Entry}; use isa::constraints::*; diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 0b01f481f2..a29394a3a9 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -1,7 +1,6 @@ //! Encoding tables for Intel ISAs. use ir::InstructionData; -use ir::instructions::InstructionFormat; use ir::types; use isa::enc_tables::{Level1Entry, Level2Entry}; use isa::constraints::*; diff --git a/lib/cretonne/src/isa/riscv/enc_tables.rs b/lib/cretonne/src/isa/riscv/enc_tables.rs index 2538911a25..f02ccf80ba 100644 --- a/lib/cretonne/src/isa/riscv/enc_tables.rs +++ b/lib/cretonne/src/isa/riscv/enc_tables.rs @@ -1,7 +1,6 @@ //! Encoding tables for RISC-V. use ir::{Opcode, InstructionData}; -use ir::instructions::InstructionFormat; use ir::types; use predicates; use isa::enc_tables::{Level1Entry, Level2Entry}; From f8e4d4e839bce74f43760105ef749be7494e8947 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 3 Feb 2017 12:28:07 -0800 Subject: [PATCH 0502/3084] Speling. --- lib/cretonne/src/cfg.rs | 19 +++++----- lib/cretonne/src/constant_hash.rs | 2 +- lib/cretonne/src/dominator_tree.rs | 11 +++--- lib/cretonne/src/entity_list.rs | 2 +- lib/cretonne/src/entity_map.rs | 4 +- lib/cretonne/src/ir/builder.rs | 4 +- lib/cretonne/src/ir/dfg.rs | 20 +++++----- lib/cretonne/src/ir/entities.rs | 6 +-- lib/cretonne/src/ir/function.rs | 2 +- lib/cretonne/src/ir/immediates.rs | 10 ++--- lib/cretonne/src/ir/instructions.rs | 23 ++++++----- lib/cretonne/src/ir/jumptable.rs | 2 +- lib/cretonne/src/ir/layout.rs | 12 +++--- lib/cretonne/src/ir/types.rs | 4 +- lib/cretonne/src/legalizer.rs | 4 +- lib/cretonne/src/predicates.rs | 4 +- lib/cretonne/src/regalloc/allocatable_set.rs | 8 ++-- lib/cretonne/src/regalloc/liveness.rs | 40 ++++++++++---------- lib/cretonne/src/regalloc/liverange.rs | 20 +++++----- lib/cretonne/src/sparse_map.rs | 4 +- lib/cretonne/src/verifier.rs | 16 ++++---- lib/cretonne/src/write.rs | 5 ++- 22 files changed, 112 insertions(+), 110 deletions(-) diff --git a/lib/cretonne/src/cfg.rs b/lib/cretonne/src/cfg.rs index f68d8bcbf4..5466355812 100644 --- a/lib/cretonne/src/cfg.rs +++ b/lib/cretonne/src/cfg.rs @@ -1,8 +1,9 @@ //! A control flow graph represented as mappings of extended basic blocks to their predecessors -//! and successors. Successors are represented as extended basic blocks while predecessors are -//! represented by basic blocks. -//! BasicBlocks are denoted by tuples of EBB and branch/jump instructions. Each predecessor -//! tuple corresponds to the end of a basic block. +//! and successors. +//! +//! Successors are represented as extended basic blocks while predecessors are represented by basic +//! blocks. Basic blocks are denoted by tuples of EBB and branch/jump instructions. Each +//! predecessor tuple corresponds to the end of a basic block. //! //! ```c //! Ebb0: @@ -19,8 +20,8 @@ //! jmp Ebb2 ; end of basic block //! ``` //! -//! Here Ebb1 and Ebb2 would each have a single predecessor denoted as (Ebb0, `brz vx, Ebb1`) -//! and (Ebb0, `jmp Ebb2`) respectively. +//! Here `Ebb1` and `Ebb2` would each have a single predecessor denoted as `(Ebb0, brz)` +//! and `(Ebb0, jmp Ebb2)` respectively. use ir::{Function, Inst, Ebb}; use ir::instructions::BranchInfo; @@ -91,7 +92,7 @@ impl ControlFlowGraph { &self.data[ebb].successors } - /// Return [reachable] ebbs in postorder. + /// Return [reachable] ebbs in post-order. pub fn postorder_ebbs(&self) -> Vec { let entry_block = match self.entry_block { None => { @@ -111,7 +112,7 @@ impl ControlFlowGraph { // This is a white node. Mark it as gray. grey.insert(node); stack.push(node); - // Get any children we’ve never seen before. + // Get any children we've never seen before. for child in self.get_successors(node) { if !grey.contains(child) { stack.push(child.clone()); @@ -125,7 +126,7 @@ impl ControlFlowGraph { postorder } - /// An iterator across all of the ebbs stored in the cfg. + /// An iterator across all of the ebbs stored in the CFG. pub fn ebbs(&self) -> Keys { self.data.keys() } diff --git a/lib/cretonne/src/constant_hash.rs b/lib/cretonne/src/constant_hash.rs index f76fde7fb7..08cc2944c8 100644 --- a/lib/cretonne/src/constant_hash.rs +++ b/lib/cretonne/src/constant_hash.rs @@ -67,7 +67,7 @@ mod tests { #[test] fn basic() { - // c.f. meta/constant_hash.py tests. + // c.f. `meta/constant_hash.py` tests. assert_eq!(simple_hash("Hello"), 0x2fa70c01); assert_eq!(simple_hash("world"), 0x5b0c31d5); } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index f940d3c910..eaec78f367 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -52,7 +52,7 @@ impl DominatorTree { self.nodes[ebb].idom.into() } - /// Compare two EBBs relative to a reverse pst-order traversal of the control-flow graph. + /// Compare two EBBs relative to a reverse post-order traversal of the control-flow graph. /// /// Return `Ordering::Less` if `a` comes before `b` in the RPO. pub fn rpo_cmp(&self, a: Ebb, b: Ebb) -> Ordering { @@ -124,7 +124,7 @@ impl DominatorTree { pub fn new(func: &Function, cfg: &ControlFlowGraph) -> DominatorTree { let mut domtree = DominatorTree { nodes: EntityMap::with_capacity(func.dfg.num_ebbs()) }; - // We'll be iterating over a reverse postorder of the CFG. + // We'll be iterating over a reverse post-order of the CFG. // This vector only contains reachable EBBs. let mut postorder = cfg.postorder_ebbs(); @@ -154,8 +154,8 @@ impl DominatorTree { } } - // Now that we have RPO numbers for everything and initial idom estimates, iterate until - // convergence. + // Now that we have RPO numbers for everything and initial immediate dominator estimates, + // iterate until convergence. // // If the function is free of irreducible control flow, this will exit after one iteration. let mut changed = true; @@ -173,7 +173,8 @@ impl DominatorTree { domtree } - // Compute the idom for `ebb` using the current `idom` states for the reachable nodes. + // Compute the immediate dominator for `ebb` using the current `idom` states for the reachable + // nodes. fn compute_idom(&self, ebb: Ebb, cfg: &ControlFlowGraph, layout: &Layout) -> Inst { // Get an iterator with just the reachable predecessors to `ebb`. // Note that during the first pass, `is_reachable` returns false for blocks that haven't diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index 6ec23e39e2..d94795c902 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -178,7 +178,7 @@ impl ListPool { /// Returns two mutable slices representing the two requested blocks. /// /// The two returned slices can be longer than the blocks. Each block is located at the front - /// the the respective slice. + /// of the respective slice. fn mut_slices(&mut self, block0: usize, block1: usize) -> (&mut [T], &mut [T]) { if block0 < block1 { let (s0, s1) = self.data.split_at_mut(block1); diff --git a/lib/cretonne/src/entity_map.rs b/lib/cretonne/src/entity_map.rs index 33d355e9f1..f9ac258f92 100644 --- a/lib/cretonne/src/entity_map.rs +++ b/lib/cretonne/src/entity_map.rs @@ -139,7 +139,7 @@ impl EntityMap self.elems.resize(n, V::default()); } - /// Ensure that `k` is a valid key but adding default entries if necesssary. + /// Ensure that `k` is a valid key but adding default entries if necessary. /// /// Return a mutable reference to the corresponding entry. pub fn ensure(&mut self, k: K) -> &mut V { @@ -202,7 +202,7 @@ impl Iterator for Keys mod tests { use super::*; - // EntityRef impl for testing. + // `EntityRef` impl for testing. #[derive(Clone, Copy, Debug, PartialEq, Eq)] struct E(u32); diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index ad3f8e22fe..639b640c20 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -13,7 +13,7 @@ use ir::condcodes::{IntCC, FloatCC}; /// /// The `InstBuilderBase` trait provides the basic functionality required by the methods of the /// generated `InstBuilder` trait. These methods should not normally be used directly. Use the -/// methods in the `InstBuilder trait instead. +/// methods in the `InstBuilder` trait instead. /// /// Any data type that implements `InstBuilderBase` also gets all the methods of the `InstBuilder` /// trait. @@ -104,7 +104,7 @@ impl<'c, 'fc, 'fd> InstBuilderBase<'fd> for InsertBuilder<'c, 'fc, 'fd> { /// /// If the old instruction still has secondary result values attached, it is assumed that the new /// instruction produces the same number and types of results. The old secondary values are -/// preserved. If the replacemant instruction format does not support multiple results, the builder +/// preserved. If the replacement instruction format does not support multiple results, the builder /// panics. It is a bug to leave result values dangling. /// /// If the old instruction was capable of producing secondary results, but the values have been diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index a01a4dd654..94117095a7 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -12,7 +12,7 @@ use packed_option::PackedOption; use std::ops::{Index, IndexMut}; use std::u16; -/// A data flow graph defines all instuctions and extended basic blocks in a function as well as +/// A data flow graph defines all instructions and extended basic blocks in a function as well as /// the data flow dependencies between them. The DFG also tracks values which can be either /// instruction results or EBB arguments. /// @@ -213,7 +213,7 @@ impl DataFlowGraph { original: original, }; } else { - panic!("Cannot change dirrect value {} into an alias", dest); + panic!("Cannot change direct value {} into an alias", dest); } } } @@ -328,7 +328,7 @@ impl DataFlowGraph { // Additional values form a linked list starting from the second result value. Generate // the list backwards so we don't have to modify value table entries in place. (This - // causes additional result values to be numbered backwards which is not the aestetic + // causes additional result values to be numbered backwards which is not the aesthetic // choice, but since it is only visible in extremely rare instructions with 3+ results, // we don't care). let mut head = None; @@ -498,7 +498,7 @@ impl DataFlowGraph { return num as usize + 1; } } - panic!("inconsistent value table entry for EBB arg"); + panic!("inconsistent value table entry for EBB argument"); } } } @@ -514,7 +514,7 @@ impl DataFlowGraph { next: None.into(), }); match self.ebbs[ebb].last_arg.expand() { - // If last_arg is `None`, we're adding the first EBB argument. + // If last_argument is `None`, we're adding the first EBB argument. None => { self.ebbs[ebb].first_arg = val.into(); } @@ -525,11 +525,11 @@ impl DataFlowGraph { if let ValueData::Arg { ref mut next, .. } = self.extended_values[idx] { *next = val.into(); } else { - panic!("inconsistent value table entry for EBB arg"); + panic!("inconsistent value table entry for EBB argument"); } } ExpandedValue::Direct(_) => { - panic!("inconsistent value table entry for EBB arg") + panic!("inconsistent value table entry for EBB argument") } } } @@ -556,7 +556,7 @@ impl DataFlowGraph { struct EbbData { // First argument to this EBB, or `None` if the block has no arguments. // - // The arguments are all ValueData::Argument entries that form a linked list from `first_arg` + // The arguments are all `ValueData::Argument` entries that form a linked list from `first_arg` // to `last_arg`. first_arg: PackedOption, @@ -680,14 +680,14 @@ mod tests { _ => panic!(), }; - // Detach the 'c' value from iadd. + // Detach the 'c' value from `iadd`. { let mut vals = dfg.detach_secondary_results(iadd); assert_eq!(vals.next(), Some(c)); assert_eq!(vals.next(), None); } - // Replace iadd_cout with a normal iadd and an icmp. + // Replace `iadd_cout` with a normal `iadd` and an `icmp`. dfg.replace(iadd).iadd(v1, arg0); let c2 = dfg.ins(pos).icmp(IntCC::UnsignedLessThan, s, v1); dfg.change_to_alias(c, c2); diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 69a9902151..51ca70ad7a 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -65,7 +65,7 @@ pub struct Ebb(u32); entity_impl!(Ebb, "ebb"); impl Ebb { - /// Create a new EBB reference from its number. This corresponds to the ebbNN representation. + /// Create a new EBB reference from its number. This corresponds to the `ebbNN` representation. /// /// This method is for use by the parser. pub fn with_number(n: u32) -> Option { @@ -90,7 +90,7 @@ pub enum ExpandedValue { impl Value { /// Create a `Direct` value from its number representation. - /// This is the number in the vNN notation. + /// This is the number in the `vNN` notation. /// /// This method is for use by the parser. pub fn direct_with_number(n: u32) -> Option { @@ -104,7 +104,7 @@ impl Value { } /// Create a `Table` value from its number representation. - /// This is the number in the vxNN notation. + /// This is the number in the `vxNN` notation. /// /// This method is for use by the parser. pub fn table_with_number(n: u32) -> Option { diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index db6c811b5a..2828104daa 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -60,7 +60,7 @@ impl Function { } } - /// Create a new empty, anomymous function. + /// Create a new empty, anonymous function. pub fn new() -> Function { Self::with_name_signature(FunctionName::default(), Signature::new()) } diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index e2fc0c446d..bc44e6c022 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -62,7 +62,7 @@ impl Display for Imm64 { impl FromStr for Imm64 { type Err = &'static str; - // Parse a decimal or hexadecimal Imm64, formatted as above. + // Parse a decimal or hexadecimal `Imm64`, formatted as above. fn from_str(s: &str) -> Result { let mut value: u64 = 0; let mut digits = 0; @@ -149,7 +149,7 @@ pub struct Ieee64(f64); // Format a floating point number in a way that is reasonably human-readable, and that can be // converted back to binary without any rounding issues. The hexadecimal formatting of normal and -// subnormal numbers is compatible with C99 and the printf "%a" format specifier. The NaN and Inf +// subnormal numbers is compatible with C99 and the `printf "%a"` format specifier. The NaN and Inf // formats are not supported by C99. // // The encoding parameters are: @@ -380,7 +380,7 @@ impl Ieee32 { Ieee32(x) } - /// Construct Ieee32 immediate from raw bits. + /// Construct `Ieee32` immediate from raw bits. pub fn from_bits(x: u32) -> Ieee32 { Ieee32(unsafe { mem::transmute(x) }) } @@ -410,7 +410,7 @@ impl Ieee64 { Ieee64(x) } - /// Construct Ieee64 immediate from raw bits. + /// Construct `Ieee64` immediate from raw bits. pub fn from_bits(x: u64) -> Ieee64 { Ieee64(unsafe { mem::transmute(x) }) } @@ -496,7 +496,7 @@ mod tests { "Negative number too small for Imm64"); parse_ok::("18446744073709551615", "-1"); parse_ok::("-9223372036854775808", "0x8000_0000_0000_0000"); - // Overflow both the checked_add and checked_mul. + // Overflow both the `checked_add` and `checked_mul`. parse_err::("18446744073709551616", "Too large decimal Imm64"); parse_err::("184467440737095516100", "Too large decimal Imm64"); parse_err::("-9223372036854775809", diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index f25c58d197..17a0ef2c27 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -57,9 +57,9 @@ impl Opcode { } } -// This trait really belongs in lib/reader where it is used by the .cton file parser, but since it -// critically depends on the `opcode_name()` function which is needed here anyway, it lives in this -// module. This also saves us from running the build script twice to generate code for the two +// This trait really belongs in lib/reader where it is used by the `.cton` file parser, but since +// it critically depends on the `opcode_name()` function which is needed here anyway, it lives in +// this module. This also saves us from running the build script twice to generate code for the two // separate crates. impl FromStr for Opcode { type Err = &'static str; @@ -141,7 +141,7 @@ pub enum InstructionData { arg: Value, imm: Imm64, }, - // Same as BinaryImm, but the immediate is the lhs operand. + // Same as `BinaryImm`, but the immediate is the left-hand-side operand. BinaryImmRev { opcode: Opcode, ty: Type, @@ -246,7 +246,7 @@ impl VariableArgs { } } -// Coerce VariableArgs into a &[Value] slice. +// Coerce `VariableArgs` into a `&[Value]` slice. impl Deref for VariableArgs { type Target = [Value]; @@ -311,7 +311,7 @@ impl Display for TernaryOverflowData { } /// Payload data for jump instructions. These need to carry lists of EBB arguments that won't fit -/// in the allowed InstructionData size. +/// in the allowed `InstructionData` size. #[derive(Clone, Debug)] pub struct JumpData { /// Jump destination EBB. @@ -331,7 +331,7 @@ impl Display for JumpData { } /// Payload data for branch instructions. These need to carry lists of EBB arguments that won't fit -/// in the allowed InstructionData size. +/// in the allowed `InstructionData` size. #[derive(Clone, Debug)] pub struct BranchData { /// Value argument controlling the branch. @@ -702,11 +702,10 @@ mod tests { #[test] fn instruction_data() { use std::mem; - // The size of the InstructionData enum is important for performance. It should not exceed - // 16 bytes. Use `Box` out-of-line payloads for instruction formats that require - // more space than that. - // It would be fine with a data structure smaller than 16 bytes, but what are the odds of - // that? + // The size of the `InstructionData` enum is important for performance. It should not + // exceed 16 bytes. Use `Box` out-of-line payloads for instruction formats that + // require more space than that. It would be fine with a data structure smaller than 16 + // bytes, but what are the odds of that? assert_eq!(mem::size_of::(), 16); } diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index 824e2dcf23..7103485094 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -33,7 +33,7 @@ impl JumpTableData { /// Set a table entry. /// - /// The table will grow as needed to fit 'idx'. + /// The table will grow as needed to fit `idx`. pub fn set_entry(&mut self, idx: usize, dest: Ebb) { // Resize table to fit `idx`. if idx >= self.table.len() { diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 2a8d1ba4ac..f914142c7e 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -573,10 +573,10 @@ impl<'f> DoubleEndedIterator for Insts<'f> { /// /// A `Cursor` represents a position in a function layout where instructions can be inserted and /// removed. It can be used to iterate through the instructions of a function while editing them at -/// the same time. A normal instruction iterator can't do this since it holds an immutable refernce -/// to the Layout. +/// the same time. A normal instruction iterator can't do this since it holds an immutable +/// reference to the Layout. /// -/// When new instructions are added, the cursor can either apend them to an EBB or insert them +/// When new instructions are added, the cursor can either append them to an EBB or insert them /// before the current instruction. pub struct Cursor<'f> { layout: &'f mut Layout, @@ -592,7 +592,7 @@ pub enum CursorPosition { /// New instructions will be inserted *before* the current instruction. At(Inst), /// Cursor is before the beginning of an EBB. No instructions can be inserted. Calling - /// `next_inst()` wil move to the first instruction in the EBB. + /// `next_inst()` will move to the first instruction in the EBB. Before(Ebb), /// Cursor is pointing after the end of an EBB. /// New instructions will be appended to the EBB. @@ -851,7 +851,7 @@ impl<'f> Cursor<'f> { /// - If pointing at the bottom of an EBB, the new instruction is appended to the EBB. /// - Otherwise panic. /// - /// In either case, the cursor is not moved, such that repeates calls to `insert_inst()` causes + /// In either case, the cursor is not moved, such that repeated calls to `insert_inst()` causes /// instructions to appear in insertion order in the EBB. pub fn insert_inst(&mut self, inst: Inst) { use self::CursorPosition::*; @@ -1263,7 +1263,7 @@ mod tests { assert_eq!(cur.prev_ebb(), None); } - // Check ProgramOrder. + // Check `ProgramOrder`. assert_eq!(layout.cmp(e2, e2), Ordering::Equal); assert_eq!(layout.cmp(e2, i2), Ordering::Less); assert_eq!(layout.cmp(i3, i2), Ordering::Greater); diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index c001c9f3c7..cdd2207aad 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -43,7 +43,7 @@ impl Type { Type(self.0 & 0x0f) } - /// Get log2 of the number of bits in a lane. + /// Get log_2 of the number of bits in a lane. pub fn log2_lane_bits(self) -> u8 { match self.lane_type() { B1 => 0, @@ -157,7 +157,7 @@ impl Type { } } - /// Get log2 of the number of lanes in this SIMD vector type. + /// Get log_2 of the number of lanes in this SIMD vector type. /// /// All SIMD types have a lane count that is a power of two and no larger than 256, so this /// will be a number in the range 0-8. diff --git a/lib/cretonne/src/legalizer.rs b/lib/cretonne/src/legalizer.rs index a83485b5e2..015bfb9b30 100644 --- a/lib/cretonne/src/legalizer.rs +++ b/lib/cretonne/src/legalizer.rs @@ -69,8 +69,8 @@ pub fn legalize_function(func: &mut Function, isa: &TargetIsa) { } } -// Include legalization patterns that were generated by gen_legalizer.py from the XForms in -// meta/cretonne/legalize.py. +// Include legalization patterns that were generated by `gen_legalizer.py` from the `XForms` in +// `meta/cretonne/legalize.py`. // // Concretely, this defines private functions `narrow()`, and `expand()`. include!(concat!(env!("OUT_DIR"), "/legalizer.rs")); diff --git a/lib/cretonne/src/predicates.rs b/lib/cretonne/src/predicates.rs index a0e50402bf..748a75e23a 100644 --- a/lib/cretonne/src/predicates.rs +++ b/lib/cretonne/src/predicates.rs @@ -20,7 +20,7 @@ pub fn is_signed_int>(x: T, wd: u8, sc: u8) -> bool { #[allow(dead_code)] pub fn is_unsigned_int>(x: T, wd: u8, sc: u8) -> bool { let u = x.into() as u64; - // Bitmask of the permitted bits. + // Bit-mask of the permitted bits. let m = (1 << wd) - (1 << sc); u == (u & m) } @@ -40,7 +40,7 @@ mod tests { assert!(is_signed_int(x2, 2, 0)); assert!(!is_signed_int(x2, 2, 1)); - // u32 doesn't sign-extend when converted to i64. + // `u32` doesn't sign-extend when converted to `i64`. assert!(!is_signed_int(x3, 8, 0)); assert!(is_unsigned_int(x1, 1, 0)); diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index f2ef0102d4..2c608b0a16 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -38,7 +38,7 @@ impl AllocatableSet { AllocatableSet { avail: [!0; 3] } } - /// Returns `true` if the spoecified register is available. + /// Returns `true` if the specified register is available. pub fn is_avail(&self, rc: RegClass, reg: RegUnit) -> bool { let (idx, bits) = bitmask(rc, reg); (self.avail[idx] & bits) == bits @@ -71,7 +71,7 @@ impl AllocatableSet { for idx in 0..self.avail.len() { // If a single unit in a register is unavailable, the whole register can't be used. // If a register straddles a word boundary, it will be marked as unavailable. - // There's an assertion in cdsl/registers.py to check for that. + // There's an assertion in `cdsl/registers.py` to check for that. for i in 0..rc.width { rsi.regs[idx] &= self.avail[idx] >> i; } @@ -102,7 +102,7 @@ impl Iterator for RegSetIter { return Some(unit); } - // How many register units was there in the word? This is a constant 32 for u32 etc. + // How many register units was there in the word? This is a constant 32 for `u32` etc. unit_offset += 8 * size_of_val(word) as RegUnit; } @@ -134,7 +134,7 @@ mod tests { fn put_and_take() { let mut regs = AllocatableSet::new(); - // GPR has units 28-36. + // `GPR` has units 28-36. assert_eq!(regs.iter(GPR).count(), 8); assert_eq!(regs.iter(DPR).collect::>(), [28, 30, 33, 35]); diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index c7646ed701..2e6cb78db4 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -8,8 +8,8 @@ //! //! The primary consumer of the liveness analysis is the SSA coloring pass which goes through each //! EBB and assigns a register to the defined values. This algorithm needs to maintain a set of the -//! curently live values as it is iterating down the instructions in the EBB. It asks the following -//! questions: +//! currently live values as it is iterating down the instructions in the EBB. It asks the +//! following questions: //! //! - What is the set of live values at the entry to the EBB? //! - When moving past a use of a value, is that value still alive in the EBB, or was that the last @@ -18,7 +18,7 @@ //! //! The set of `LiveRange` instances can answer these questions through their `def_local_end` and //! `livein_local_end` queries. The coloring algorithm visits EBBs in a topological order of the -//! dominator tree, so it can compute the set of live values at the begining of an EBB by starting +//! dominator tree, so it can compute the set of live values at the beginning of an EBB by starting //! from the set of live values at the dominating branch instruction and filtering it with //! `livein_local_end`. These sets do not need to be stored in the liveness analysis. //! @@ -32,11 +32,11 @@ //! A number of different liveness analysis algorithms exist, so it is worthwhile to look at a few //! alternatives. //! -//! ## Dataflow equations +//! ## Data-flow equations //! //! The classic *live variables analysis* that you will find in all compiler books from the //! previous century does not depend on SSA form. It is typically implemented by iteratively -//! solving dataflow equations on bitvectors of variables. The result is a live-out bitvector of +//! solving data-flow equations on bit-vectors of variables. The result is a live-out bit-vector of //! variables for every basic block in the program. //! //! This algorithm has some disadvantages that makes us look elsewhere: @@ -44,18 +44,18 @@ //! - Quadratic memory use. We need a bit per variable per basic block in the function. //! - Sparse representation. In practice, the majority of SSA values never leave their basic block, //! and those that do span basic blocks rarely span a large number of basic blocks. This makes -//! the bitvectors quite sparse. -//! - Traditionally, the dataflow equations were solved for real program *variables* which does not -//! include temporaries used in evaluating expressions. We have an SSA form program which blurs -//! the distinction between temporaries and variables. This makes the quadratic memory problem -//! worse because there are many more SSA values than there was variables in the original +//! the bit-vectors quite sparse. +//! - Traditionally, the data-flow equations were solved for real program *variables* which does +//! not include temporaries used in evaluating expressions. We have an SSA form program which +//! blurs the distinction between temporaries and variables. This makes the quadratic memory +//! problem worse because there are many more SSA values than there was variables in the original //! program, and we don't know a priori which SSA values leave their basic block. //! - Missing last-use information. For values that are not live-out of a basic block, we would //! need to store information about the last use in the block somewhere. LLVM stores this //! information as a 'kill bit' on the last use in the IR. Maintaining these kill bits has been a //! source of problems for LLVM's register allocator. //! -//! Dataflow equations can detect when a variable is used uninitialized, and they can handle +//! Data-flow equations can detect when a variable is used uninitialized, and they can handle //! multiple definitions of the same variable. We don't need this generality since we already have //! a program in SSA form. //! @@ -83,7 +83,7 @@ //! The iterative SSA form reconstruction can be skipped if the depth-first search only encountered //! one SSA value. //! -//! This algorithm has some advantages compared to the dataflow equations: +//! This algorithm has some advantages compared to the data-flow equations: //! //! - The live ranges of local virtual registers are computed very quickly without ever traversing //! the CFG. The memory needed to store these live ranges is independent of the number of basic @@ -106,11 +106,11 @@ //! was presented at CGO 2008: //! //! > Boissinot, B., Hack, S., Grund, D., de Dinechin, B. D., & Rastello, F. (2008). *Fast Liveness -//! Checking for SSA-Form Programs.* CGO. +//! Checking for SSA-Form Programs.* CGO. //! -//! This analysis uses a global precomputation that only depends on the CFG of the function. It +//! This analysis uses a global pre-computation that only depends on the CFG of the function. It //! then allows liveness queries for any (value, program point) pair. Each query traverses the use -//! chain of the value and performs lookups in the precomputed bitvectors. +//! chain of the value and performs lookups in the precomputed bit-vectors. //! //! I did not seriously consider this analysis for Cretonne because: //! @@ -118,8 +118,8 @@ //! - Popular variables like the `this` pointer in a C++ method can have very large use chains. //! Traversing such a long use chain on every liveness lookup has the potential for some nasty //! quadratic behavior in unfortunate cases. -//! - It says "fast" in the title, but the paper only claims to be 16% faster than a dataflow based -//! approach, which isn't that impressive. +//! - It says "fast" in the title, but the paper only claims to be 16% faster than a data-flow +//! based approach, which isn't that impressive. //! //! Nevertheless, the property of only depending in the CFG structure is very useful. If Cretonne //! gains use chains, this approach would be worth a proper evaluation. @@ -171,7 +171,7 @@ //! - Related values should be stored on the same cache line. The current sparse set implementation //! does a decent job of that. //! - For global values, the list of live-in intervals is very likely to fit on a single cache -//! line. These lists are very likely ot be found in L2 cache at least. +//! line. These lists are very likely to be found in L2 cache at least. //! //! There is some room for improvement. @@ -271,10 +271,10 @@ impl Liveness { self.worklist.push(ebb); } - // The worklist contains those EBBs where we have learned that the value needs to be + // The work list contains those EBBs where we have learned that the value needs to be // live-in. // - // This algorithm bcomes a depth-first traversal up the CFG, enumerating all paths through + // This algorithm becomes a depth-first traversal up the CFG, enumerating all paths through // the CFG from the existing live range to `ebb`. // // Extend the live range as we go. The live range itself also serves as a visited set since diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 67667a3a47..c132a82478 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -45,13 +45,13 @@ //! handle *early clobbers* which are output registers that are not allowed to alias any input //! registers. //! -//! If i1 < i2 < i3 are program points, we have: +//! If `i1 < i2 < i3` are program points, we have: //! -//! - i1-i2 and i1-i3 interfere because the intervals overlap. -//! - i1-i2 and i2-i3 don't interfere. -//! - i1-i3 and i2-i2 do interfere because the dead def would clobber the register. -//! - i1-i2 and i2-i2 don't interfere. -//! - i2-i3 and i2-i2 do interfere. +//! - `i1-i2` and `i1-i3` interfere because the intervals overlap. +//! - `i1-i2` and `i2-i3` don't interfere. +//! - `i1-i3` and `i2-i2` do interfere because the dead def would clobber the register. +//! - `i1-i2` and `i2-i2` don't interfere. +//! - `i2-i3` and `i2-i2` do interfere. //! //! Because of this behavior around interval end points, live range interference is not completely //! equivalent to mathematical intersection of open or half-open intervals. @@ -415,7 +415,7 @@ mod tests { } } - // Singleton ProgramOrder for tests below. + // Singleton `ProgramOrder` for tests below. const PO: &'static ProgOrder = &ProgOrder {}; #[test] @@ -441,7 +441,7 @@ mod tests { assert!(lr.is_local()); assert_eq!(lr.def(), e2.into()); assert_eq!(lr.def_local_end(), e2.into()); - // The def interval of an EBB arg does not count as live-in. + // The def interval of an EBB argument does not count as live-in. assert_eq!(lr.livein_local_end(e2, PO), None); PO.validate(&lr); } @@ -478,8 +478,8 @@ mod tests { let i13 = Inst::new(13); let mut lr = LiveRange::new(v0, e10); - // Extending a dead EBB arg in its own block should not indicate that a live-in interval - // was created. + // Extending a dead EBB argument in its own block should not indicate that a live-in + // interval was created. assert_eq!(lr.extend_in_ebb(e10, i12, PO), false); PO.validate(&lr); assert!(!lr.is_dead()); diff --git a/lib/cretonne/src/sparse_map.rs b/lib/cretonne/src/sparse_map.rs index 48844c026a..3a1b4dbdb6 100644 --- a/lib/cretonne/src/sparse_map.rs +++ b/lib/cretonne/src/sparse_map.rs @@ -27,7 +27,7 @@ //! about creating new mappings to the default value. It doesn't distinguish clearly between an //! unmapped key and one that maps to the default value. `SparseMap` does not require `Default` //! values, and it tracks accurately if a key has been mapped or not. -//! - Iterating over the contants of an `EntityMap` is linear in the size of the *key space*, while +//! - Iterating over the contents of an `EntityMap` is linear in the size of the *key space*, while //! iterating over a `SparseMap` is linear in the number of elements in the mapping. This is an //! advantage precisely when the mapping is sparse. //! - `SparseMap::clear()` is constant time and super-fast. `EntityMap::clear()` is linear in the @@ -45,7 +45,7 @@ use std::u32; /// All values stored in a `SparseMap` must keep track of their own key in the map and implement /// this trait to provide access to the key. pub trait SparseMapValue { - /// Get the key of this sparse map value. This key is not alowed to change while the value + /// Get the key of this sparse map value. This key is not allowed to change while the value /// is a member of the map. fn key(&self) -> K; } diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index f0a4b66b32..0076933934 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -3,18 +3,18 @@ //! //! EBB integrity //! -//! - All instructions reached from the ebb_insts iterator must belong to -//! the EBB as reported by inst_ebb(). +//! - All instructions reached from the `ebb_insts` iterator must belong to +//! the EBB as reported by `inst_ebb()`. //! - Every EBB must end in a terminator instruction, and no other instruction //! can be a terminator. -//! - Every value in the ebb_args iterator belongs to the EBB as reported by value_ebb. +//! - Every value in the `ebb_args` iterator belongs to the EBB as reported by `value_ebb`. //! //! Instruction integrity //! //! - The instruction format must match the opcode. //! TODO: //! - All result values must be created for multi-valued instructions. -//! - Instructions with no results must have a VOID first_type(). +//! - Instructions with no results must have a VOID `first_type()`. //! - All referenced entities must exist. (Values, EBBs, stack slots, ...) //! //! SSA form @@ -33,7 +33,7 @@ //! - Compare input and output values against the opcode's type constraints. //! For polymorphic opcodes, determine the controlling type variable first. //! - Branches and jumps must pass arguments to destination EBBs that match the -//! expected types excatly. The number of arguments must match. +//! expected types exactly. The number of arguments must match. //! - All EBBs in a jump_table must take no arguments. //! - Function calls are type checked against their signature. //! - The entry block must take arguments that match the signature of the current @@ -44,10 +44,10 @@ //! Ad hoc checking //! //! - Stack slot loads and stores must be in-bounds. -//! - Immediate constraints for certain opcodes, like udiv_imm v3, 0. +//! - Immediate constraints for certain opcodes, like `udiv_imm v3, 0`. //! - Extend / truncate instructions have more type constraints: Source type can't be //! larger / smaller than result type. -//! - Insertlane and extractlane instructions have immediate lane numbers that must be in +//! - `Insertlane` and `extractlane` instructions have immediate lane numbers that must be in //! range for their polymorphic type. //! - Swizzle and shuffle instructions take a variable number of lane arguments. The number //! of arguments must match the destination type, and the lane indexes must be in range. @@ -76,7 +76,7 @@ impl Display for Error { /// Verifier result. pub type Result = result::Result; -// Create an `Err` variant of `Result` from a location and `format!` args. +// Create an `Err` variant of `Result` from a location and `format!` arguments. macro_rules! err { ( $loc:expr, $msg:expr ) => { Err(Error { diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index d8db064cd6..8dfe955e42 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -43,7 +43,8 @@ fn write_preamble(w: &mut Write, func: &Function) -> result::Result try!(writeln!(w, " {} = {}", ss, func.stack_slots[ss])); } - // Write out all signatures before functions since function decls can refer to signatures. + // Write out all signatures before functions since function declarations can refer to + // signatures. for sig in func.dfg.signatures.keys() { any = true; try!(writeln!(w, " {} = signature{}", sig, func.dfg.signatures[sig])); @@ -93,7 +94,7 @@ pub fn write_ebb_header(w: &mut Write, func: &Function, ebb: Ebb) -> Result { try!(write_arg(w, func, arg)); } } - // Remaining args. + // Remaining arguments. for arg in args { try!(write!(w, ", ")); try!(write_arg(w, func, arg)); From f6391c57e8758c1f5649699df137df0a50701f8b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 3 Feb 2017 15:06:05 -0800 Subject: [PATCH 0503/3084] Compute register affinities during liveness analysis. Each live range has an affinity hint containing the preferred register class (or stack slot). Compute the affinity by merging the constraints of the def and all uses. --- lib/cretonne/src/regalloc/liveness.rs | 162 +++++++++++++++---------- lib/cretonne/src/regalloc/liverange.rs | 15 ++- 2 files changed, 105 insertions(+), 72 deletions(-) diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 2e6cb78db4..fe43b6962c 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -175,39 +175,87 @@ //! //! There is some room for improvement. -use regalloc::liverange::LiveRange; -use ir::{Function, Value, Inst, Ebb, ProgramPoint}; -use ir::dfg::{DataFlowGraph, ValueDef}; use cfg::ControlFlowGraph; +use ir::dfg::ValueDef; +use ir::{Function, Value, Inst, Ebb}; +use isa::{TargetIsa, RecipeConstraints}; +use regalloc::liverange::LiveRange; +use regalloc::affinity::Affinity; use sparse_map::SparseMap; /// A set of live ranges, indexed by value number. -struct LiveRangeSet(SparseMap); +type LiveRangeSet = SparseMap; -impl LiveRangeSet { - pub fn new() -> LiveRangeSet { - LiveRangeSet(SparseMap::new()) +/// Get a mutable reference to the live range for `value`. +/// Create it if necessary. +fn get_or_create<'a>(lrset: &'a mut LiveRangeSet, + value: Value, + func: &Function, + recipe_constraints: &[RecipeConstraints]) + -> &'a mut LiveRange { + // It would be better to use `get_mut()` here, but that leads to borrow checker fighting + // which can probably only be resolved by non-lexical lifetimes. + // https://github.com/rust-lang/rfcs/issues/811 + if lrset.get(value).is_none() { + // Create a live range for value. We need the program point that defines it. + let def; + let affinity; + match func.dfg.value_def(value) { + ValueDef::Res(inst, rnum) => { + def = inst.into(); + // Initialize the affinity from the defining instruction's result constraints. + // Don't do this for call return values which are always tied to a single register. + affinity = recipe_constraints.get(func.encodings[inst].recipe()) + .and_then(|rc| rc.outs.get(rnum)) + .map(Affinity::new) + .unwrap_or_default(); + } + ValueDef::Arg(ebb, _) => { + def = ebb.into(); + // Don't apply any affinity to EBB arguments. + // They could be in a register or on the stack. + affinity = Default::default(); + } + }; + lrset.insert(LiveRange::new(value, def, affinity)); + } + lrset.get_mut(value).unwrap() +} + +/// Extend the live range for `value` so it reaches `to` which must live in `ebb`. +fn extend_to_use(lr: &mut LiveRange, + ebb: Ebb, + to: Inst, + worklist: &mut Vec, + func: &Function, + cfg: &ControlFlowGraph) { + // This is our scratch working space, and we'll leave it empty when we return. + assert!(worklist.is_empty()); + + // Extend the range locally in `ebb`. + // If there already was a live interval in that block, we're done. + if lr.extend_in_ebb(ebb, to, &func.layout) { + worklist.push(ebb); } - pub fn clear(&mut self) { - self.0.clear(); - } - - /// Get a mutable reference to the live range for `value`. - /// Create it if necessary. - pub fn get_or_create(&mut self, value: Value, dfg: &DataFlowGraph) -> &mut LiveRange { - // It would be better to use `get_mut()` here, but that leads to borrow checker fighting - // which can probably only be resolved by non-lexical lifetimes. - // https://github.com/rust-lang/rfcs/issues/811 - if self.0.get(value).is_none() { - // Create a live range for value. We need the program point that defines it. - let def: ProgramPoint = match dfg.value_def(value) { - ValueDef::Res(inst, _) => inst.into(), - ValueDef::Arg(ebb, _) => ebb.into(), - }; - self.0.insert(LiveRange::new(value, def)); + // The work list contains those EBBs where we have learned that the value needs to be + // live-in. + // + // This algorithm becomes a depth-first traversal up the CFG, enumerating all paths through the + // CFG from the existing live range to `ebb`. + // + // Extend the live range as we go. The live range itself also serves as a visited set since + // `extend_in_ebb` will never return true twice for the same EBB. + // + while let Some(livein) = worklist.pop() { + // We've learned that the value needs to be live-in to the `livein` EBB. + // Make sure it is also live at all predecessor branches to `livein`. + for &(pred, branch) in cfg.get_predecessors(livein) { + if lr.extend_in_ebb(pred, branch, &func.layout) { + // This predecessor EBB also became live-in. We need to process it later. + worklist.push(pred); + } } - self.0.get_mut(value).unwrap() } } @@ -238,56 +286,42 @@ impl Liveness { /// Compute the live ranges of all SSA values used in `func`. /// This clears out any existing analysis stored in this data structure. - pub fn compute(&mut self, func: &Function, cfg: &ControlFlowGraph) { + pub fn compute(&mut self, func: &Function, cfg: &ControlFlowGraph, isa: &TargetIsa) { self.ranges.clear(); + // Get ISA data structures used for computing live range affinities. + let recipe_constraints = isa.recipe_constraints(); + let reg_info = isa.register_info(); + // The liveness computation needs to visit all uses, but the order doesn't matter. // TODO: Perhaps this traversal of the function could be combined with a dead code // elimination pass if we visit a post-order of the dominator tree? // TODO: Resolve value aliases while we're visiting instructions? for ebb in func.layout.ebbs() { for inst in func.layout.ebb_insts(ebb) { - func.dfg[inst].each_arg(|arg| self.extend_to_use(arg, ebb, inst, func, cfg)); - } - } - } + // The instruction encoding is used to compute affinities. + let recipe = func.encodings[inst].recipe(); + // Iterator of constraints, one per value operand. + // TODO: Should we fail here if the instruction doesn't have a valid encoding? + let mut operand_constraints = + recipe_constraints.get(recipe).map(|c| c.ins).unwrap_or(&[]).iter(); - /// Extend the live range for `value` so it reaches `to` which must live in `ebb`. - fn extend_to_use(&mut self, - value: Value, - ebb: Ebb, - to: Inst, - func: &Function, - cfg: &ControlFlowGraph) { - // Get the live range, create it as a dead range if necessary. - let lr = self.ranges.get_or_create(value, &func.dfg); + func.dfg[inst].each_arg(|arg| { + // Get the live range, create it as a dead range if necessary. + let lr = get_or_create(&mut self.ranges, arg, func, recipe_constraints); - // This is our scratch working space, and we'll leave it empty when we return. - assert!(self.worklist.is_empty()); + // Extend the live range to reach this use. + extend_to_use(lr, ebb, inst, &mut self.worklist, func, cfg); - // Extend the range locally in `ebb`. - // If there already was a live interval in that block, we're done. - if lr.extend_in_ebb(ebb, to, &func.layout) { - self.worklist.push(ebb); - } - - // The work list contains those EBBs where we have learned that the value needs to be - // live-in. - // - // This algorithm becomes a depth-first traversal up the CFG, enumerating all paths through - // the CFG from the existing live range to `ebb`. - // - // Extend the live range as we go. The live range itself also serves as a visited set since - // `extend_in_ebb` will never return true twice for the same EBB. - // - while let Some(livein) = self.worklist.pop() { - // We've learned that the value needs to be live-in to the `livein` EBB. - // Make sure it is also live at all predecessor branches to `livein`. - for &(pred, branch) in cfg.get_predecessors(livein) { - if lr.extend_in_ebb(pred, branch, &func.layout) { - // This predecessor EBB also became live-in. We need to process it later. - self.worklist.push(pred); - } + // Apply operand constraint, ignoring any variable arguments after the fixed + // operands described by `operand_constraints`. Variable arguments are either + // EBB arguments or call/return ABI arguments. EBB arguments need to be + // resolved by the coloring algorithm, and ABI constraints require specific + // registers or stack slots which the affinities don't model anyway. + if let Some(constraint) = operand_constraints.next() { + lr.affinity.merge(constraint, reg_info); + } + }); } } } diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index c132a82478..4441bfcda5 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -205,11 +205,10 @@ impl LiveRange { /// Create a new live range for `value` defined at `def`. /// /// The live range will be created as dead, but it can be extended with `extend_in_ebb()`. - pub fn new>(value: Value, def: PP) -> LiveRange { - let def = def.into(); + pub fn new(value: Value, def: ProgramPoint, affinity: Affinity) -> LiveRange { LiveRange { value: value, - affinity: Default::default(), + affinity: affinity, def_begin: def, def_end: def, liveins: Vec::new(), @@ -423,7 +422,7 @@ mod tests { let v0 = Value::new(0); let i1 = Inst::new(1); let e2 = Ebb::new(2); - let lr = LiveRange::new(v0, i1); + let lr = LiveRange::new(v0, i1.into(), Default::default()); assert!(lr.is_dead()); assert!(lr.is_local()); assert_eq!(lr.def(), i1.into()); @@ -436,7 +435,7 @@ mod tests { fn dead_arg_range() { let v0 = Value::new(0); let e2 = Ebb::new(2); - let lr = LiveRange::new(v0, e2); + let lr = LiveRange::new(v0, e2.into(), Default::default()); assert!(lr.is_dead()); assert!(lr.is_local()); assert_eq!(lr.def(), e2.into()); @@ -453,7 +452,7 @@ mod tests { let i11 = Inst::new(11); let i12 = Inst::new(12); let i13 = Inst::new(13); - let mut lr = LiveRange::new(v0, i11); + let mut lr = LiveRange::new(v0, i11.into(), Default::default()); assert_eq!(lr.extend_in_ebb(e10, i13, PO), false); PO.validate(&lr); @@ -476,7 +475,7 @@ mod tests { let i11 = Inst::new(11); let i12 = Inst::new(12); let i13 = Inst::new(13); - let mut lr = LiveRange::new(v0, e10); + let mut lr = LiveRange::new(v0, e10.into(), Default::default()); // Extending a dead EBB argument in its own block should not indicate that a live-in // interval was created. @@ -510,7 +509,7 @@ mod tests { let i21 = Inst::new(21); let i22 = Inst::new(22); let i23 = Inst::new(23); - let mut lr = LiveRange::new(v0, i11); + let mut lr = LiveRange::new(v0, i11.into(), Default::default()); assert_eq!(lr.extend_in_ebb(e10, i12, PO), false); From 5a1d9561a7da60a362423212791b213dc4a6f360 Mon Sep 17 00:00:00 2001 From: Mikko Perttunen Date: Tue, 14 Feb 2017 14:04:03 +0200 Subject: [PATCH 0504/3084] Coalesce live range intervals in adjacent EBBs LiveRanges represent the live-in range of a value as a sorted list of intervals. Each interval starts at an EBB and continues to an instruction. Before this commit, the LiveRange would store an interval for each EBB. This commit changes the representation such that intervals continuing from one EBB to another are coalesced into one. Fixes #37. --- lib/cretonne/src/ir/layout.rs | 12 +++ lib/cretonne/src/ir/progpoint.rs | 5 ++ lib/cretonne/src/regalloc/liverange.rs | 109 ++++++++++++++++++++++--- 3 files changed, 115 insertions(+), 11 deletions(-) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index f914142c7e..50dfa61f4e 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -102,6 +102,13 @@ impl ProgramOrder for Layout { let b_seq = self.seq(b); a_seq.cmp(&b_seq) } + + fn is_ebb_gap(&self, inst: Inst, ebb: Ebb) -> bool { + let i = &self.insts[inst]; + let e = &self.ebbs[ebb]; + + i.next.is_none() && i.ebb == e.prev + } } // Private methods for dealing with sequence numbers. @@ -1267,5 +1274,10 @@ mod tests { assert_eq!(layout.cmp(e2, e2), Ordering::Equal); assert_eq!(layout.cmp(e2, i2), Ordering::Less); assert_eq!(layout.cmp(i3, i2), Ordering::Greater); + + assert_eq!(layout.is_ebb_gap(i1, e2), true); + assert_eq!(layout.is_ebb_gap(i3, e1), true); + assert_eq!(layout.is_ebb_gap(i1, e1), false); + assert_eq!(layout.is_ebb_gap(i2, e1), false); } } diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index 83c565fca7..7d57ab8cb9 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -94,6 +94,11 @@ pub trait ProgramOrder { fn cmp(&self, a: A, b: B) -> cmp::Ordering where A: Into, B: Into; + + /// Is the range from `inst` to `ebb` just the gap between consecutive EBBs? + /// + /// This returns true if `inst` is the terminator in the EBB immediately before `ebb`. + fn is_ebb_gap(&self, inst: Inst, ebb: Ebb) -> bool; } #[cfg(test)] diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 4441bfcda5..a2d7a8f32e 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -173,6 +173,7 @@ pub struct LiveRange { /// This represents a live-in interval for a single EBB, or a coalesced set of live-in intervals /// for contiguous EBBs where all but the last live-in interval covers the whole EBB. /// +#[derive(Copy, Clone)] struct Interval { /// Interval starting point. /// @@ -265,18 +266,57 @@ impl LiveRange { Ok(n) => { // We have an interval that contains `ebb`, so we can simply extend it. self.liveins[n].extend_to(to, order); - // TODO: Check if this interval can be coalesced with the `n+1` one. + + // If `to` is the terminator and the value lives in the successor EBB, + // coalesce the two intervals. + if let Some(next) = self.liveins.get(n + 1).cloned() { + if order.is_ebb_gap(to, next.begin) { + self.liveins[n].extend_to(next.end, order); + self.liveins.remove(n + 1); + } + } + false } Err(n) => { - // Insert a new live-in interval at `n`. - // TODO: Check if this interval can be coalesced with the `n-1` or `n+1` one (or - // both). - self.liveins.insert(n, - Interval { - begin: ebb, - end: to, - }); + // Insert a new live-in interval at `n`, or coalesce to predecessor or successor + // if possible. + + // Determine if the new live-in range touches the predecessor or successor range + // and can therefore be coalesced to them. + let (coalesce_prev, coalesce_next) = { + let prev = n.checked_sub(1).and_then(|i| self.liveins.get(i)); + let next = self.liveins.get(n); + + (prev.map_or(false, |prev| order.is_ebb_gap(prev.end, ebb)), + next.map_or(false, |next| order.is_ebb_gap(to, next.begin))) + }; + + match (coalesce_prev, coalesce_next) { + // Extend predecessor interval to cover new and successor intervals + (true, true) => { + let end = self.liveins[n].end; + self.liveins[n - 1].extend_to(end, order); + self.liveins.remove(n); + } + // Extend predecessor interval to cover new interval + (true, false) => { + self.liveins[n - 1].extend_to(to, order); + } + // Extend successor interval to cover new interval + (false, true) => { + self.liveins[n].begin = ebb; + } + // Cannot coalesce; insert new interval + (false, false) => { + self.liveins.insert(n, + Interval { + begin: ebb, + end: to, + }); + } + } + true } } @@ -344,7 +384,8 @@ mod tests { // Dummy program order which simply compares indexes. // It is assumed that EBBs have indexes that are multiples of 10, and instructions have indexes - // in between. + // in between. `is_ebb_gap` assumes that terminator instructions have indexes of the form + // ebb * 10 + 1. This is used in the coalesce test. struct ProgOrder {} impl ProgramOrder for ProgOrder { @@ -363,6 +404,10 @@ mod tests { let ib = idx(b.into()); ia.cmp(&ib) } + + fn is_ebb_gap(&self, inst: Inst, ebb: Ebb) -> bool { + inst.index() % 10 == 1 && ebb.index() / 10 == inst.index() / 10 + 1 + } } impl ProgOrder { @@ -528,5 +573,47 @@ mod tests { assert_eq!(lr.livein_local_end(e20, PO), Some(i23)); } - // TODO: Add more tests that exercise the binary search and the coalescing algorithm. + #[test] + fn coalesce() { + let v0 = Value::new(0); + let i11 = Inst::new(11); + let e20 = Ebb::new(20); + let i21 = Inst::new(21); + let e30 = Ebb::new(30); + let i31 = Inst::new(31); + let e40 = Ebb::new(40); + let i41 = Inst::new(41); + let mut lr = LiveRange::new(v0, i11.into(), Default::default()); + + assert_eq!(lr.extend_in_ebb(e30, i31, PO), true); + assert_eq!(lr.liveins.len(), 1); + + // Coalesce to previous + assert_eq!(lr.extend_in_ebb(e40, i41, PO), true); + assert_eq!(lr.liveins.len(), 1); + assert_eq!(lr.liveins[0].begin, e30); + assert_eq!(lr.liveins[0].end, i41); + + // Coalesce to next + assert_eq!(lr.extend_in_ebb(e20, i21, PO), true); + assert_eq!(lr.liveins.len(), 1); + assert_eq!(lr.liveins[0].begin, e20); + assert_eq!(lr.liveins[0].end, i41); + + let mut lr = LiveRange::new(v0, i11.into(), Default::default()); + + assert_eq!(lr.extend_in_ebb(e40, i41, PO), true); + assert_eq!(lr.liveins.len(), 1); + + assert_eq!(lr.extend_in_ebb(e20, i21, PO), true); + assert_eq!(lr.liveins.len(), 2); + + // Coalesce to previous and next + assert_eq!(lr.extend_in_ebb(e30, i31, PO), true); + assert_eq!(lr.liveins.len(), 1); + assert_eq!(lr.liveins[0].begin, e20); + assert_eq!(lr.liveins[0].end, i41); + } + + // TODO: Add more tests that exercise the binary search algorithm. } From 5579e9b4a5055b44b565a6a8f9336a5fe3d8a089 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 6 Feb 2017 18:06:37 +0100 Subject: [PATCH 0505/3084] Add a partition_slice function. Partition the elements in a mutable slice according to a predicate. --- lib/cretonne/src/partition_slice.rs | 75 +++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 lib/cretonne/src/partition_slice.rs diff --git a/lib/cretonne/src/partition_slice.rs b/lib/cretonne/src/partition_slice.rs new file mode 100644 index 0000000000..9626b5fd37 --- /dev/null +++ b/lib/cretonne/src/partition_slice.rs @@ -0,0 +1,75 @@ +//! Rearrange the elements in a slice according to a predicate. + +/// Rearrange the elements of the mutable slice `s` such that elements where `p(t)` is true precede +/// the elements where `p(t)` is false. +/// +/// The order of elements is not preserved, unless the slice is already partitioned. +/// +/// Returns the number of elements where `p(t)` is true. +pub fn partition_slice<'a, T: 'a, F>(s: &'a mut [T], mut p: F) -> usize + where F: FnMut(&T) -> bool +{ + // Count the length of the prefix where `p` returns true. + let mut count = match s.iter().position(|t| !p(t)) { + Some(t) => t, + None => return s.len(), + }; + + // Swap remaining `true` elements into place. + // + // This actually preserves the order of the `true` elements, but the `false` elements get + // shuffled. + for i in count + 1..s.len() { + if p(&s[i]) { + s.swap(count, i); + count += 1; + } + } + + count +} + +#[cfg(test)] +mod tests { + use super::partition_slice; + + fn check(x: &[u32], want: &[u32]) { + assert_eq!(x.len(), want.len()); + let want_count = want.iter().cloned().filter(|&x| x % 10 == 0).count(); + let mut v = Vec::new(); + v.extend(x.iter().cloned()); + let count = partition_slice(&mut v[..], |&x| x % 10 == 0); + assert_eq!(v, want); + assert_eq!(count, want_count); + } + + #[test] + fn empty() { + check(&[], &[]); + } + + #[test] + fn singles() { + check(&[0], &[0]); + check(&[1], &[1]); + check(&[10], &[10]); + } + + #[test] + fn doubles() { + check(&[0, 0], &[0, 0]); + check(&[0, 5], &[0, 5]); + check(&[5, 0], &[0, 5]); + check(&[5, 4], &[5, 4]); + } + + #[test] + fn longer() { + check(&[1, 2, 3], &[1, 2, 3]); + check(&[1, 2, 10], &[10, 2, 1]); // Note: 2, 1 order not required. + check(&[1, 10, 2], &[10, 1, 2]); // Note: 1, 2 order not required. + check(&[1, 20, 10], &[20, 10, 1]); + check(&[1, 20, 3, 10], &[20, 10, 3, 1]); + check(&[20, 3, 10, 1], &[20, 10, 3, 1]); + } +} From 2c310416408cd87c4555136667fcb350af72bfd2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 6 Feb 2017 16:54:29 +0100 Subject: [PATCH 0506/3084] Live Value Tracker. Keep track of which values are live and dead as we move through the instructions in an EBB. --- lib/cretonne/src/lib.rs | 1 + .../src/regalloc/live_value_tracker.rs | 253 ++++++++++++++++++ lib/cretonne/src/regalloc/liveness.rs | 5 + lib/cretonne/src/regalloc/mod.rs | 1 + 4 files changed, 260 insertions(+) create mode 100644 lib/cretonne/src/regalloc/live_value_tracker.rs diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 436e2829e3..bfd986949a 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -25,4 +25,5 @@ mod constant_hash; mod predicates; mod legalizer; mod ref_slice; +mod partition_slice; mod packed_option; diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs new file mode 100644 index 0000000000..02af0969d1 --- /dev/null +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -0,0 +1,253 @@ +//! Track which values are live in an EBB with instruction granularity. +//! +//! The `LiveValueTracker` keeps track of the set of live SSA values at each instruction in an EBB. +//! The sets of live values are computed on the fly as the tracker is moved from instruction to +//! instruction, starting at the EBB header. + +use dominator_tree::DominatorTree; +use entity_list::{EntityList, ListPool}; +use ir::instructions::BranchInfo; +use ir::{Inst, Ebb, Value, DataFlowGraph, ProgramOrder, ExpandedProgramPoint}; +use partition_slice::partition_slice; +use regalloc::liveness::Liveness; + +use std::collections::HashMap; + +type ValueList = EntityList; + +/// Compute and track live values throughout an EBB. +pub struct LiveValueTracker { + /// The set of values that are live at the current program point. + live: LiveValueVec, + + /// Saved set of live values for every jump and branch that can potentially be an immediate + /// dominator of an EBB. + /// + /// This is the set of values that are live *before* the branch. + idom_sets: HashMap, + + /// Memory pool for the live sets. + idom_pool: ListPool, +} + +/// Information about a value that is live at the current program point. +pub struct LiveValue { + /// The live value. + pub value: Value, + + /// The local ending point of the live range in the current EBB, as returned by + /// `LiveRange::def_local_end()` or `LiveRange::livein_local_end()`. + pub endpoint: Inst, +} + +struct LiveValueVec { + /// The set of values that are live at the current program point. + values: Vec, + + /// How many values at the front of `values` are known to be live after `inst`? + /// + /// This is used to pass a much smaller slice to `partition_slice` when its called a second + /// time for the same instruction. + live_prefix: Option<(Inst, usize)>, +} + +impl LiveValueVec { + fn new() -> LiveValueVec { + LiveValueVec { + values: Vec::new(), + live_prefix: None, + } + } + + /// Add a new live value to `values`. + fn push(&mut self, value: Value, endpoint: Inst) { + self.values.push(LiveValue { + value: value, + endpoint: endpoint, + }); + } + + /// Remove all elements. + fn clear(&mut self) { + self.values.clear(); + self.live_prefix = None; + } + + /// Make sure that the values killed by `next_inst` are moved to the end of the `values` + /// vector. + /// + /// Returns the number of values that will be live after `next_inst`. + fn live_after(&mut self, next_inst: Inst) -> usize { + // How many values at the front of the vector are already known to survive `next_inst`? + // We don't need to pass this prefix to `partition_slice()` + let keep = match self.live_prefix { + Some((i, prefix)) if i == next_inst => prefix, + _ => 0, + }; + + // Move the remaining surviving values to the front partition of the vector. + let prefix = keep + partition_slice(&mut self.values[keep..], |v| v.endpoint != next_inst); + + // Remember the new prefix length in case we get called again for the same `next_inst`. + self.live_prefix = Some((next_inst, prefix)); + prefix + } + + /// Remove the values killed by `next_inst`. + fn remove_kill_values(&mut self, next_inst: Inst) { + let keep = self.live_after(next_inst); + self.values.truncate(keep); + } +} + +impl LiveValueTracker { + /// Create a new blank tracker. + pub fn new() -> LiveValueTracker { + LiveValueTracker { + live: LiveValueVec::new(), + idom_sets: HashMap::new(), + idom_pool: ListPool::new(), + } + } + + /// Clear all cached information. + pub fn clear(&mut self) { + self.live.clear(); + self.idom_sets.clear(); + self.idom_pool.clear(); + } + + /// Get the set of currently live values. + /// + /// Between calls to `process_inst()` and `drop_dead()`, this includes both values killed and + /// defined by the current instruction. + pub fn live(&self) -> &[LiveValue] { + &self.live.values + } + + /// Move the current position to the top of `ebb`. + /// + /// This depends on the stored live value set at `ebb`'s immediate dominator, so that must have + /// been visited first. + pub fn ebb_top(&mut self, + ebb: Ebb, + dfg: &DataFlowGraph, + liveness: &Liveness, + program_order: &PO, + domtree: &DominatorTree) { + // Start over, compute the set of live values at the top of the EBB from two sources: + // + // 1. Values that were live before `ebb`'s immediate dominator, filtered for those that are + // actually live-in. + // 2. Arguments to `ebb` that are not dead. + // + self.live.clear(); + + // Compute the live-in values. Start by filtering the set of values that were live before + // the immediate dominator. Just use the empty set if there's no immediate dominator (i.e., + // the entry block or an unreachable block). + if let Some(idom) = domtree.idom(ebb) { + // If the immediate dominator exits, we must have a stored list for it. This is a + // requirement to the order EBBs are visited: All dominators must have been processed + // before the current EBB. + let idom_live_list = + self.idom_sets.get(&idom).expect("No stored live set for dominator"); + // Get just the values that are live-in to `ebb`. + for &value in idom_live_list.as_slice(&self.idom_pool) { + let lr = liveness.get(value) + .expect("Immediate dominator value has no live range"); + + // Check if this value is live-in here. + if let Some(endpoint) = lr.livein_local_end(ebb, program_order) { + self.live.push(value, endpoint); + } + } + } + + // Now add all the live arguments to `ebb`. + for value in dfg.ebb_args(ebb) { + let lr = liveness.get(value).expect("EBB argument value has no live range"); + assert_eq!(lr.def(), ebb.into()); + match lr.def_local_end().into() { + ExpandedProgramPoint::Inst(endpoint) => { + self.live.push(value, endpoint); + } + ExpandedProgramPoint::Ebb(local_ebb) => { + // This is a dead EBB argument which is not even live into the first + // instruction in the EBB. We can ignore it. + assert_eq!(local_ebb, + ebb, + "EBB argument live range ends at wrong EBB header"); + } + } + } + } + + /// Prepare to move past `inst`. + /// + /// Determine the set of already live values that are killed by `inst`, and add the new defined + /// values to the tracked set. + /// + /// Returns `(kills, defs)` as a pair of slices. The `defs` slice is guaranteed to be in the + /// same order as `inst`'s results, and includes dead defines. The order of `kills` is + /// arbitrary. + /// + /// The `drop_dead()` method must be called next to actually remove the dead values from the + /// tracked set after the two returned slices are no longer needed. + pub fn process_inst(&mut self, + inst: Inst, + dfg: &DataFlowGraph, + liveness: &Liveness) + -> (&[LiveValue], &[LiveValue]) { + // Save a copy of the live values before any branches or jumps that could be somebody's + // immediate dominator. + match dfg[inst].analyze_branch() { + BranchInfo::NotABranch => {} + _ => self.save_idom_live_set(inst), + } + + // Move killed values to the end of the vector. + // Don't remove them yet, `drop_dead()` will do that. + let first_kill = self.live.live_after(inst); + + // Add the values defined by `inst`. + let first_def = self.live.values.len(); + for value in dfg.inst_results(inst) { + let lr = liveness.get(value).expect("Instruction result has no live range"); + assert_eq!(lr.def(), inst.into()); + match lr.def_local_end().into() { + ExpandedProgramPoint::Inst(endpoint) => { + self.live.push(value, endpoint); + } + ExpandedProgramPoint::Ebb(ebb) => { + panic!("Instruction result live range can't end at {}", ebb); + } + } + } + + (&self.live.values[first_kill..first_def], &self.live.values[first_def..]) + } + + /// Drop the values that are now dead after moving past `inst`. + /// + /// This removes both live values that were killed by `inst` and dead defines on `inst` itself. + /// + /// This must be called after `process_inst(inst)` and before proceeding to the next + /// instruction. + pub fn drop_dead(&mut self, inst: Inst) { + // Remove both live values that were killed by `inst` and dead defines from `inst`. + self.live.remove_kill_values(inst); + } + + /// Save the current set of live values so it is associated with `idom`. + fn save_idom_live_set(&mut self, idom: Inst) { + let values = self.live.values.iter().map(|lv| lv.value); + let pool = &mut self.idom_pool; + // If there already is a set saved for `idom`, just keep it. + self.idom_sets.entry(idom).or_insert_with(|| { + let mut list = ValueList::default(); + list.extend(values, pool); + list + }); + } +} diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index fe43b6962c..e34ceda915 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -284,6 +284,11 @@ impl Liveness { } } + /// Get the live range for `value`, if it exists. + pub fn get(&self, value: Value) -> Option<&LiveRange> { + self.ranges.get(value) + } + /// Compute the live ranges of all SSA values used in `func`. /// This clears out any existing analysis stored in this data structure. pub fn compute(&mut self, func: &Function, cfg: &ControlFlowGraph, isa: &TargetIsa) { diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index bd1c34cd23..902e7aabb0 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -5,5 +5,6 @@ pub mod liverange; pub mod liveness; pub mod allocatable_set; +pub mod live_value_tracker; mod affinity; From 2317142c753bbe95e63a6e64f3e313fd9de1e089 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 14 Feb 2017 15:52:44 -0800 Subject: [PATCH 0507/3084] Add a Layout::next_ebb() method. This lets us iterate over the blocks in a function without holding a reference to the layout. --- lib/cretonne/src/ir/layout.rs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 50dfa61f4e..d30f336780 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -348,6 +348,11 @@ impl Layout { pub fn entry_block(&self) -> Option { self.first_ebb } + + /// Get the block following `ebb` in the layout order. + pub fn next_ebb(&self, ebb: Ebb) -> Option { + self.ebbs[ebb].next.expand() + } } #[derive(Clone, Debug, Default)] From 96e0a3273cbd5c8147298f38fb680d9d07e68176 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 14 Feb 2017 15:54:33 -0800 Subject: [PATCH 0508/3084] Return slices of live-ins and arguments from ebb_top(). The coloring algorithm needs to process these two types of live values differently, so we may as well provide the needed info. --- lib/cretonne/src/regalloc/live_value_tracker.rs | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 02af0969d1..cdbfebe286 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -129,12 +129,17 @@ impl LiveValueTracker { /// /// This depends on the stored live value set at `ebb`'s immediate dominator, so that must have /// been visited first. + /// + /// Returns `(liveins, args)` as a pair or slices. The first slice is the set of live-in values + /// from the immediate dominator. The second slice is the set of `ebb` arguments that are live. + /// Dead arguments with no uses are ignored and not added to the set. pub fn ebb_top(&mut self, ebb: Ebb, dfg: &DataFlowGraph, liveness: &Liveness, program_order: &PO, - domtree: &DominatorTree) { + domtree: &DominatorTree) + -> (&[LiveValue], &[LiveValue]) { // Start over, compute the set of live values at the top of the EBB from two sources: // // 1. Values that were live before `ebb`'s immediate dominator, filtered for those that are @@ -165,6 +170,7 @@ impl LiveValueTracker { } // Now add all the live arguments to `ebb`. + let first_arg = self.live.values.len(); for value in dfg.ebb_args(ebb) { let lr = liveness.get(value).expect("EBB argument value has no live range"); assert_eq!(lr.def(), ebb.into()); @@ -181,6 +187,8 @@ impl LiveValueTracker { } } } + + self.live.values.split_at(first_arg) } /// Prepare to move past `inst`. From 408dc4e72ebdede00520b63634ae9dbd305e0503 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 14 Feb 2017 15:55:47 -0800 Subject: [PATCH 0509/3084] Add a contains_key method to SparseMap. --- lib/cretonne/src/sparse_map.rs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/cretonne/src/sparse_map.rs b/lib/cretonne/src/sparse_map.rs index 3a1b4dbdb6..ad49ac49fa 100644 --- a/lib/cretonne/src/sparse_map.rs +++ b/lib/cretonne/src/sparse_map.rs @@ -126,6 +126,11 @@ impl SparseMap None } + /// Return `true` if the map contains a value corresponding to `key`. + pub fn contains_key(&self, key: K) -> bool { + self.get(key).is_some() + } + /// Insert a value into the map. /// /// If the map did not have this key present, `None` is returned. From 1fa3ddf0187e0c8d1474d013b3ef193a99532935 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 14 Feb 2017 16:00:33 -0800 Subject: [PATCH 0510/3084] Return RegInfo by value from TargetIsa::register_info(). The struct is just a pair of static references, and we don't need the double indirection. --- lib/cretonne/src/isa/arm32/mod.rs | 4 ++-- lib/cretonne/src/isa/arm64/mod.rs | 4 ++-- lib/cretonne/src/isa/intel/mod.rs | 4 ++-- lib/cretonne/src/isa/mod.rs | 2 +- lib/cretonne/src/isa/registers.rs | 1 + lib/cretonne/src/isa/riscv/mod.rs | 4 ++-- lib/cretonne/src/regalloc/liveness.rs | 2 +- 7 files changed, 11 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 63f6293516..63e9516a13 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -49,8 +49,8 @@ impl TargetIsa for Isa { &self.shared_flags } - fn register_info(&self) -> &RegInfo { - ®isters::INFO + fn register_info(&self) -> RegInfo { + registers::INFO.clone() } fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Result { diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 2c2b98437b..4d00b1d49e 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -42,8 +42,8 @@ impl TargetIsa for Isa { &self.shared_flags } - fn register_info(&self) -> &RegInfo { - ®isters::INFO + fn register_info(&self) -> RegInfo { + registers::INFO.clone() } fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Result { diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 13b77e1dd2..533c264e3d 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -49,8 +49,8 @@ impl TargetIsa for Isa { &self.shared_flags } - fn register_info(&self) -> &RegInfo { - ®isters::INFO + fn register_info(&self) -> RegInfo { + registers::INFO.clone() } fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Result { diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index ee33bb4c47..af689a3947 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -132,7 +132,7 @@ pub trait TargetIsa { fn flags(&self) -> &settings::Flags; /// Get a data structure describing the registers in this ISA. - fn register_info(&self) -> &RegInfo; + fn register_info(&self) -> RegInfo; /// Encode an instruction after determining it is legal. /// diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 5e89110284..588fc0b14d 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -168,6 +168,7 @@ impl From for RegClassIndex { /// /// The `RegUnit` data structure collects all relevant static information about the registers in an /// ISA. +#[derive(Clone)] pub struct RegInfo { /// All register banks, ordered by their `first_unit`. The register banks are disjoint, but /// there may be holes of unused register unit numbers between banks due to alignment. diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 34b2a43c99..b997316733 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -49,8 +49,8 @@ impl TargetIsa for Isa { &self.shared_flags } - fn register_info(&self) -> &RegInfo { - ®isters::INFO + fn register_info(&self) -> RegInfo { + registers::INFO.clone() } fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Result { diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index e34ceda915..da4b2568f9 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -324,7 +324,7 @@ impl Liveness { // resolved by the coloring algorithm, and ABI constraints require specific // registers or stack slots which the affinities don't model anyway. if let Some(constraint) = operand_constraints.next() { - lr.affinity.merge(constraint, reg_info); + lr.affinity.merge(constraint, ®_info); } }); } From b1769ac7e4ad454b2e17edfc2012f4938dd31bd7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 15 Feb 2017 13:53:01 -0800 Subject: [PATCH 0511/3084] Cache the affinity in LiveValue. Most of the register allocator algorithms will only have to look at the currently live values as presented by LiveValueTracker. Many also need the value's affinity which is stored in the LiveRange associated with the value. Save the extra table lookup by caching the affinity value inside LiveValue. --- lib/cretonne/src/regalloc/live_value_tracker.rs | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index cdbfebe286..838c5b6bb9 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -9,6 +9,7 @@ use entity_list::{EntityList, ListPool}; use ir::instructions::BranchInfo; use ir::{Inst, Ebb, Value, DataFlowGraph, ProgramOrder, ExpandedProgramPoint}; use partition_slice::partition_slice; +use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; use std::collections::HashMap; @@ -38,6 +39,12 @@ pub struct LiveValue { /// The local ending point of the live range in the current EBB, as returned by /// `LiveRange::def_local_end()` or `LiveRange::livein_local_end()`. pub endpoint: Inst, + + /// The affinity of the value as represented in its `LiveRange`. + /// + /// This value is simply a copy of the affinity stored in the live range. We copy it because + /// almost all users of `LiveValue` need to look at it. + pub affinity: Affinity, } struct LiveValueVec { @@ -60,10 +67,11 @@ impl LiveValueVec { } /// Add a new live value to `values`. - fn push(&mut self, value: Value, endpoint: Inst) { + fn push(&mut self, value: Value, endpoint: Inst, affinity: Affinity) { self.values.push(LiveValue { value: value, endpoint: endpoint, + affinity: affinity, }); } @@ -164,7 +172,7 @@ impl LiveValueTracker { // Check if this value is live-in here. if let Some(endpoint) = lr.livein_local_end(ebb, program_order) { - self.live.push(value, endpoint); + self.live.push(value, endpoint, lr.affinity); } } } @@ -176,7 +184,7 @@ impl LiveValueTracker { assert_eq!(lr.def(), ebb.into()); match lr.def_local_end().into() { ExpandedProgramPoint::Inst(endpoint) => { - self.live.push(value, endpoint); + self.live.push(value, endpoint, lr.affinity); } ExpandedProgramPoint::Ebb(local_ebb) => { // This is a dead EBB argument which is not even live into the first @@ -225,7 +233,7 @@ impl LiveValueTracker { assert_eq!(lr.def(), inst.into()); match lr.def_local_end().into() { ExpandedProgramPoint::Inst(endpoint) => { - self.live.push(value, endpoint); + self.live.push(value, endpoint, lr.affinity); } ExpandedProgramPoint::Ebb(ebb) => { panic!("Instruction result live range can't end at {}", ebb); From e60d7f179ca0a47b9c21479692d54e5f7c8f3d40 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 16 Feb 2017 13:56:58 -0800 Subject: [PATCH 0512/3084] Give register classes a name. This is just for better error messages etc. --- lib/cretonne/meta/gen_registers.py | 2 +- lib/cretonne/src/isa/registers.rs | 3 +++ lib/cretonne/src/regalloc/allocatable_set.rs | 2 ++ 3 files changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/gen_registers.py b/lib/cretonne/meta/gen_registers.py index c535f44c22..e36434f4d6 100644 --- a/lib/cretonne/meta/gen_registers.py +++ b/lib/cretonne/meta/gen_registers.py @@ -33,8 +33,8 @@ def gen_regclass(rc, fmt): """ Emit a static data definition for a register class. """ - fmt.comment(rc.name) with fmt.indented('RegClassData {', '},'): + fmt.line('name: "{}",'.format(rc.name)) fmt.line('index: {},'.format(rc.index)) fmt.line('width: {},'.format(rc.width)) fmt.line('subclasses: 0x{:x},'.format(rc.subclass_mask())) diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 588fc0b14d..c785ab2ac9 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -99,6 +99,9 @@ pub type RegClass = &'static RegClassData; /// A register class can be a subset of another register class. The top-level register classes are /// disjoint. pub struct RegClassData { + /// The name of the register class. + pub name: &'static str, + /// The index of this class in the ISA's RegInfo description. pub index: u8, diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index 2c608b0a16..39058c04e4 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -118,12 +118,14 @@ mod tests { // Register classes for testing. const GPR: RegClass = &RegClassData { + name: "GPR", index: 0, width: 1, subclasses: 0, mask: [0xf0000000, 0x0000000f, 0], }; const DPR: RegClass = &RegClassData { + name: "DPR", index: 0, width: 2, subclasses: 0, From 1992890f850c5077da503b1af5dadd181cc0d362 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 17 Feb 2017 11:57:32 -0800 Subject: [PATCH 0513/3084] Add a compilation context struct. This will provide main entry points for compiling functions, and it serves as a place for keeping data structures that should be preserved between function compilations to reduce allocator thrashing. So far, Context is just basic scaffolding. More to be added. --- lib/cretonne/src/context.rs | 23 +++++++++++++++++++++++ lib/cretonne/src/lib.rs | 24 +++++++++++++----------- 2 files changed, 36 insertions(+), 11 deletions(-) create mode 100644 lib/cretonne/src/context.rs diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs new file mode 100644 index 0000000000..2360302ea2 --- /dev/null +++ b/lib/cretonne/src/context.rs @@ -0,0 +1,23 @@ +//! Cretonne compilation context and main entry point. +//! +//! When compiling many small functions, it is important to avoid repeatedly allocating and +//! deallocating the data structures needed for compilation. The `Context` struct is used to hold +//! on to memory allocations between function compilations. + +use ir::Function; + +/// Persistent data structures and compilation pipeline. +pub struct Context { + /// The function we're compiling. + pub func: Function, +} + +impl Context { + /// Allocate a new compilation context. + /// + /// The returned instance should be reused for compiling multiple functions in order to avoid + /// needless allocator thrashing. + pub fn new() -> Context { + Context { func: Function::new() } + } +} diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index bfd986949a..47339a39ca 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -2,28 +2,30 @@ #![deny(missing_docs)] +pub use context::Context; +pub use legalizer::legalize_function; pub use verifier::verify_function; pub use write::write_function; -pub use legalizer::legalize_function; /// Version number of the cretonne crate. pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); -pub mod ir; -pub mod isa; pub mod cfg; pub mod dominator_tree; -pub mod entity_map; pub mod entity_list; -pub mod sparse_map; -pub mod settings; -pub mod verifier; +pub mod entity_map; +pub mod ir; +pub mod isa; pub mod regalloc; +pub mod settings; +pub mod sparse_map; +pub mod verifier; -mod write; mod constant_hash; -mod predicates; +mod context; mod legalizer; -mod ref_slice; -mod partition_slice; mod packed_option; +mod partition_slice; +mod predicates; +mod ref_slice; +mod write; From f3fa0fb4e991ca421a55c4e5b63d8baa3b9f7ba6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 17 Feb 2017 12:05:27 -0800 Subject: [PATCH 0514/3084] Return slices instead of &Vec references. We Don't need to expose the internal control flow graph representation. --- lib/cretonne/src/cfg.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/cfg.rs b/lib/cretonne/src/cfg.rs index 5466355812..cfb29844d0 100644 --- a/lib/cretonne/src/cfg.rs +++ b/lib/cretonne/src/cfg.rs @@ -83,12 +83,12 @@ impl ControlFlowGraph { } /// Get the CFG predecessor basic blocks to `ebb`. - pub fn get_predecessors(&self, ebb: Ebb) -> &Vec { + pub fn get_predecessors(&self, ebb: Ebb) -> &[BasicBlock] { &self.data[ebb].predecessors } /// Get the CFG successors to `ebb`. - pub fn get_successors(&self, ebb: Ebb) -> &Vec { + pub fn get_successors(&self, ebb: Ebb) -> &[Ebb] { &self.data[ebb].successors } From 77a7ad88f4a3ad36079040d021488f67e8d87820 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 17 Feb 2017 12:16:48 -0800 Subject: [PATCH 0515/3084] Make the ControlFlowGraph reusable. Move the flow graph computation into a compute method which can be called with multiple functions. This allows us to reuse the ControlFlowGraph memory and keep an instance in the Context. --- cranelift/src/filetest/domtree.rs | 2 +- cranelift/src/print_cfg.rs | 2 +- cranelift/tests/cfg_traversal.rs | 2 +- lib/cretonne/src/cfg.rs | 39 ++++++++++++++++++++---------- lib/cretonne/src/context.rs | 9 ++++++- lib/cretonne/src/dominator_tree.rs | 4 +-- 6 files changed, 39 insertions(+), 19 deletions(-) diff --git a/cranelift/src/filetest/domtree.rs b/cranelift/src/filetest/domtree.rs index 0a570e3253..72a4293df7 100644 --- a/cranelift/src/filetest/domtree.rs +++ b/cranelift/src/filetest/domtree.rs @@ -40,7 +40,7 @@ impl SubTest for TestDomtree { // Extract our own dominator tree from fn run(&self, func: Cow, context: &Context) -> Result<()> { let func = func.borrow(); - let cfg = ControlFlowGraph::new(func); + let cfg = ControlFlowGraph::with_function(func); let domtree = DominatorTree::new(func, &cfg); // Build an expected domtree from the source annotations. diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index 858ef4de23..ca6c9219f7 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -33,7 +33,7 @@ impl<'a> CFGPrinter<'a> { pub fn new(func: &'a Function) -> CFGPrinter<'a> { CFGPrinter { func: func, - cfg: ControlFlowGraph::new(func), + cfg: ControlFlowGraph::with_function(func), } } diff --git a/cranelift/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs index 2bdeab2347..453e0f83b7 100644 --- a/cranelift/tests/cfg_traversal.rs +++ b/cranelift/tests/cfg_traversal.rs @@ -8,7 +8,7 @@ use self::cretonne::entity_map::EntityMap; fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) { let func = &parse_functions(function_source).unwrap()[0]; - let cfg = ControlFlowGraph::new(&func); + let cfg = ControlFlowGraph::with_function(&func); let ebbs = ebb_order.iter() .map(|n| Ebb::with_number(*n).unwrap()) .collect::>(); diff --git a/lib/cretonne/src/cfg.rs b/lib/cretonne/src/cfg.rs index cfb29844d0..9b0f3f15e7 100644 --- a/lib/cretonne/src/cfg.rs +++ b/lib/cretonne/src/cfg.rs @@ -50,31 +50,44 @@ pub struct ControlFlowGraph { } impl ControlFlowGraph { - /// During initialization mappings will be generated for any existing - /// blocks within the CFG's associated function. - pub fn new(func: &Function) -> ControlFlowGraph { + /// Allocate a new blank control flow graph. + pub fn new() -> ControlFlowGraph { + ControlFlowGraph { + entry_block: None, + data: EntityMap::new(), + } + } - let mut cfg = ControlFlowGraph { - data: EntityMap::with_capacity(func.dfg.num_ebbs()), - entry_block: func.layout.entry_block(), - }; + /// Allocate and compute the control flow graph for `func`. + pub fn with_function(func: &Function) -> ControlFlowGraph { + let mut cfg = ControlFlowGraph::new(); + cfg.compute(func); + cfg + } + + /// Compute the control flow graph of `func`. + /// + /// This will clear and overwrite any information already stored in this data structure. + pub fn compute(&mut self, func: &Function) { + self.entry_block = func.layout.entry_block(); + self.data.clear(); + self.data.resize(func.dfg.num_ebbs()); for ebb in &func.layout { for inst in func.layout.ebb_insts(ebb) { match func.dfg[inst].analyze_branch() { BranchInfo::SingleDest(dest, _) => { - cfg.add_edge((ebb, inst), dest); + self.add_edge((ebb, inst), dest); } BranchInfo::Table(jt) => { for (_, dest) in func.jump_tables[jt].entries() { - cfg.add_edge((ebb, inst), dest); + self.add_edge((ebb, inst), dest); } } BranchInfo::NotABranch => {} } } } - cfg } fn add_edge(&mut self, from: BasicBlock, to: Ebb) { @@ -140,7 +153,7 @@ mod tests { #[test] fn empty() { let func = Function::new(); - let cfg = ControlFlowGraph::new(&func); + let cfg = ControlFlowGraph::with_function(&func); assert_eq!(None, cfg.ebbs().next()); } @@ -154,7 +167,7 @@ mod tests { func.layout.append_ebb(ebb1); func.layout.append_ebb(ebb2); - let cfg = ControlFlowGraph::new(&func); + let cfg = ControlFlowGraph::with_function(&func); let nodes = cfg.ebbs().collect::>(); assert_eq!(nodes.len(), 3); @@ -194,7 +207,7 @@ mod tests { cur.insert_ebb(ebb2); } - let cfg = ControlFlowGraph::new(&func); + let cfg = ControlFlowGraph::with_function(&func); let ebb0_predecessors = cfg.get_predecessors(ebb0); let ebb1_predecessors = cfg.get_predecessors(ebb1); diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 2360302ea2..8a94dd83fc 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -4,12 +4,16 @@ //! deallocating the data structures needed for compilation. The `Context` struct is used to hold //! on to memory allocations between function compilations. +use cfg::ControlFlowGraph; use ir::Function; /// Persistent data structures and compilation pipeline. pub struct Context { /// The function we're compiling. pub func: Function, + + /// The control flow graph of `func`. + pub cfg: ControlFlowGraph, } impl Context { @@ -18,6 +22,9 @@ impl Context { /// The returned instance should be reused for compiling multiple functions in order to avoid /// needless allocator thrashing. pub fn new() -> Context { - Context { func: Function::new() } + Context { + func: Function::new(), + cfg: ControlFlowGraph::new(), + } } } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index eaec78f367..02c9cecc84 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -203,7 +203,7 @@ mod test { #[test] fn empty() { let func = Function::new(); - let cfg = ControlFlowGraph::new(&func); + let cfg = ControlFlowGraph::with_function(&func); let dtree = DominatorTree::new(&func, &cfg); assert_eq!(0, dtree.nodes.keys().count()); } @@ -238,7 +238,7 @@ mod test { cur.insert_ebb(ebb0); } - let cfg = ControlFlowGraph::new(&func); + let cfg = ControlFlowGraph::with_function(&func); let dt = DominatorTree::new(&func, &cfg); assert_eq!(func.layout.entry_block().unwrap(), ebb3); From 85fa68023c07fb23b7d58384a9afe92f4b46ec4b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 17 Feb 2017 13:09:41 -0800 Subject: [PATCH 0516/3084] Make the DominatorTree reusable. Add a compute() method which can recompute a dominator tree for different functions. Add a dominator tree data to the cretonne::Context. --- cranelift/src/filetest/domtree.rs | 2 +- lib/cretonne/src/context.rs | 11 +++++++++ lib/cretonne/src/dominator_tree.rs | 38 ++++++++++++++++++++---------- 3 files changed, 37 insertions(+), 14 deletions(-) diff --git a/cranelift/src/filetest/domtree.rs b/cranelift/src/filetest/domtree.rs index 72a4293df7..efa4ee857b 100644 --- a/cranelift/src/filetest/domtree.rs +++ b/cranelift/src/filetest/domtree.rs @@ -41,7 +41,7 @@ impl SubTest for TestDomtree { fn run(&self, func: Cow, context: &Context) -> Result<()> { let func = func.borrow(); let cfg = ControlFlowGraph::with_function(func); - let domtree = DominatorTree::new(func, &cfg); + let domtree = DominatorTree::with_function(func, &cfg); // Build an expected domtree from the source annotations. let mut expected = HashMap::new(); diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 8a94dd83fc..31a1569d5f 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -5,6 +5,7 @@ //! on to memory allocations between function compilations. use cfg::ControlFlowGraph; +use dominator_tree::DominatorTree; use ir::Function; /// Persistent data structures and compilation pipeline. @@ -14,6 +15,9 @@ pub struct Context { /// The control flow graph of `func`. pub cfg: ControlFlowGraph, + + /// Dominator tree for `func`. + pub domtree: DominatorTree, } impl Context { @@ -25,6 +29,13 @@ impl Context { Context { func: Function::new(), cfg: ControlFlowGraph::new(), + domtree: DominatorTree::new(), } } + + /// Recompute the control flow graph and dominator tree. + pub fn flowgraph(&mut self) { + self.cfg.compute(&self.func); + self.domtree.compute(&self.func, &self.cfg); + } } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 02c9cecc84..fd2c6cfb50 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -119,10 +119,24 @@ impl DominatorTree { } impl DominatorTree { + /// Allocate a new blank dominator tree. Use `compute` to compute the dominator tree for a + /// function. + pub fn new() -> DominatorTree { + DominatorTree { nodes: EntityMap::new() } + } + + /// Allocate and compute a dominator tree. + pub fn with_function(func: &Function, cfg: &ControlFlowGraph) -> DominatorTree { + let mut domtree = DominatorTree::new(); + domtree.compute(func, cfg); + domtree + } + /// Build a dominator tree from a control flow graph using Keith D. Cooper's /// "Simple, Fast Dominator Algorithm." - pub fn new(func: &Function, cfg: &ControlFlowGraph) -> DominatorTree { - let mut domtree = DominatorTree { nodes: EntityMap::with_capacity(func.dfg.num_ebbs()) }; + pub fn compute(&mut self, func: &Function, cfg: &ControlFlowGraph) { + self.nodes.clear(); + self.nodes.resize(func.dfg.num_ebbs()); // We'll be iterating over a reverse post-order of the CFG. // This vector only contains reachable EBBs. @@ -132,12 +146,12 @@ impl DominatorTree { // The last block visited in a post-order traversal must be the entry block. let entry_block = match postorder.pop() { Some(ebb) => ebb, - None => return domtree, + None => return, }; assert_eq!(Some(entry_block), func.layout.entry_block()); // Do a first pass where we assign RPO numbers to all reachable nodes. - domtree.nodes[entry_block].rpo_number = 1; + self.nodes[entry_block].rpo_number = 1; for (rpo_idx, &ebb) in postorder.iter().rev().enumerate() { // Update the current node and give it an RPO number. // The entry block got 1, the rest start at 2. @@ -148,8 +162,8 @@ impl DominatorTree { // // Due to the nature of the post-order traversal, every node we visit will have at // least one predecessor that has previously been visited during this RPO. - domtree.nodes[ebb] = DomNode { - idom: domtree.compute_idom(ebb, cfg, &func.layout).into(), + self.nodes[ebb] = DomNode { + idom: self.compute_idom(ebb, cfg, &func.layout).into(), rpo_number: rpo_idx as u32 + 2, } } @@ -162,15 +176,13 @@ impl DominatorTree { while changed { changed = false; for &ebb in postorder.iter().rev() { - let idom = domtree.compute_idom(ebb, cfg, &func.layout).into(); - if domtree.nodes[ebb].idom != idom { - domtree.nodes[ebb].idom = idom; + let idom = self.compute_idom(ebb, cfg, &func.layout).into(); + if self.nodes[ebb].idom != idom { + self.nodes[ebb].idom = idom; changed = true; } } } - - domtree } // Compute the immediate dominator for `ebb` using the current `idom` states for the reachable @@ -204,7 +216,7 @@ mod test { fn empty() { let func = Function::new(); let cfg = ControlFlowGraph::with_function(&func); - let dtree = DominatorTree::new(&func, &cfg); + let dtree = DominatorTree::with_function(&func, &cfg); assert_eq!(0, dtree.nodes.keys().count()); } @@ -239,7 +251,7 @@ mod test { } let cfg = ControlFlowGraph::with_function(&func); - let dt = DominatorTree::new(&func, &cfg); + let dt = DominatorTree::with_function(&func, &cfg); assert_eq!(func.layout.entry_block().unwrap(), ebb3); assert_eq!(dt.idom(ebb3), None); From a20afbefe0d65ae140069c2996f530ed1452e95d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Sat, 18 Feb 2017 10:22:00 -0800 Subject: [PATCH 0517/3084] Improve assertion text for missing live ranges. --- lib/cretonne/src/regalloc/live_value_tracker.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 838c5b6bb9..bcbbb62fe1 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -229,7 +229,10 @@ impl LiveValueTracker { // Add the values defined by `inst`. let first_def = self.live.values.len(); for value in dfg.inst_results(inst) { - let lr = liveness.get(value).expect("Instruction result has no live range"); + let lr = match liveness.get(value) { + Some(lr) => lr, + None => panic!("{} result {} has no live range", dfg[inst].opcode(), value), + }; assert_eq!(lr.def(), inst.into()); match lr.def_local_end().into() { ExpandedProgramPoint::Inst(endpoint) => { From 22bc33fa059cc920251bef994e25492831f7b0b0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Sat, 18 Feb 2017 10:22:27 -0800 Subject: [PATCH 0518/3084] Create live ranges for dead defs. When the liveness pass implements dead code elimination, missing live ranges can be used to indicate unused values that it may be possible to remove. But even then, we may have to keep dead defs around if the instruction has side effects or other live defs. --- lib/cretonne/src/regalloc/liveness.rs | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index da4b2568f9..a8591e8597 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -304,6 +304,13 @@ impl Liveness { // TODO: Resolve value aliases while we're visiting instructions? for ebb in func.layout.ebbs() { for inst in func.layout.ebb_insts(ebb) { + // Make sure we have created live ranges for dead defs. + // TODO: When we implement DCE, we can use the absence of a live range to indicate + // an unused value. + for def in func.dfg.inst_results(inst) { + get_or_create(&mut self.ranges, def, func, recipe_constraints); + } + // The instruction encoding is used to compute affinities. let recipe = func.encodings[inst].recipe(); // Iterator of constraints, one per value operand. From b6fa40d6a3a50d2a309d0bf9a7e148bf76463c9d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 21 Feb 2017 13:05:17 -0800 Subject: [PATCH 0519/3084] Add a return_reg instruction to the base instruction set. Register-style return is used by all RISC architectures, so it is natural to have a shared instruction representation. --- cranelift/docs/langref.rst | 1 + lib/cretonne/meta/base/formats.py | 2 +- lib/cretonne/meta/base/instructions.py | 23 +++++++++++++++++++++-- lib/cretonne/src/ir/instructions.rs | 26 ++++++++++++++++++++++++++ lib/cretonne/src/write.rs | 7 +++++++ lib/reader/src/parser.rs | 20 +++++++++++++++++++- 6 files changed, 75 insertions(+), 4 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 5bd59b18d3..d7bffd870a 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -384,6 +384,7 @@ preamble`: .. autoinst:: call .. autoinst:: x_return +.. autoinst:: return_reg This simple example illustrates direct function calls and signatures:: diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 421913b0e4..89704623fd 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -52,7 +52,7 @@ IndirectCall = InstructionFormat( sig_ref, VALUE, VARIABLE_ARGS, multiple_results=True, boxed_storage=True) Return = InstructionFormat(VARIABLE_ARGS, boxed_storage=True) - +ReturnReg = InstructionFormat(VALUE, VARIABLE_ARGS, boxed_storage=True) # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index eb98d175ef..1693763468 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -18,7 +18,7 @@ GROUP = InstructionGroup("base", "Shared base instruction set") Int = TypeVar('Int', 'A scalar or vector integer type', ints=True, simd=True) iB = TypeVar('iB', 'A scalar integer type', ints=True) -iPtr = TypeVar('iB', 'An integer address type', ints=(32, 64)) +iAddr = TypeVar('iAddr', 'An integer address type', ints=(32, 64)) Testable = TypeVar( 'Testable', 'A scalar boolean or integer type', ints=True, bools=True) @@ -113,6 +113,25 @@ x_return = Instruction( """, ins=rvals, is_terminator=True) +raddr = Operand('raddr', iAddr, doc='Return address') + +return_reg = Instruction( + 'return_reg', r""" + Return from the function to a return address held in a register. + + Unconditionally transfer control to the calling function, passing the + provided return values. The list of return values must match the + function signature's return types. + + This instruction should only be used by ISA-specific epilogue lowering + code. It is equivalent to :inst:`return`, but the return address is + provided explicitly in a register. This style of return instruction is + used by RISC architectures such as ARM and RISC-V. A normal + :inst:`return` will be legalized into this instruction on these + architectures. + """, + ins=(raddr, rvals), is_terminator=True) + FN = Operand( 'FN', entities.func_ref, @@ -130,7 +149,7 @@ call = Instruction( outs=rvals) SIG = Operand('SIG', entities.sig_ref, doc='function signature') -callee = Operand('callee', iPtr, doc='address of function to call') +callee = Operand('callee', iAddr, doc='address of function to call') call_indirect = Instruction( 'call_indirect', r""" diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 17a0ef2c27..d778ddb9a3 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -222,6 +222,11 @@ pub enum InstructionData { ty: Type, data: Box, }, + ReturnReg { + opcode: Opcode, + ty: Type, + data: Box, + }, } /// A variable list of `Value` operands used for function call arguments and passing arguments to @@ -406,6 +411,27 @@ pub struct ReturnData { pub varargs: VariableArgs, } +/// Payload of a return instruction. +#[derive(Clone, Debug)] +pub struct ReturnRegData { + /// Return address. + pub arg: Value, + /// Dynamically sized array containing return values. + pub varargs: VariableArgs, +} + +impl ReturnRegData { + /// Get references to the arguments. + pub fn arguments(&self) -> [&[Value]; 2] { + [ref_slice(&self.arg), &self.varargs] + } + + /// Get mutable references to the arguments. + pub fn arguments_mut(&mut self) -> [&mut [Value]; 2] { + [ref_slice_mut(&mut self.arg), &mut self.varargs] + } +} + /// Analyzing an instruction. /// /// Avoid large matches on instruction formats by using the methods defined here to examine diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 8dfe955e42..7a328a6d30 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -239,6 +239,13 @@ fn write_instruction(w: &mut Write, writeln!(w, " {}", data.varargs) } } + ReturnReg { ref data, .. } => { + if data.varargs.is_empty() { + writeln!(w, "{}", data.arg) + } else { + writeln!(w, "{}, {}", data.arg, data.varargs) + } + } } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 9decd562b0..8651e09d4d 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -16,7 +16,7 @@ use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, TernaryOverflowData, JumpData, BranchData, CallData, - IndirectCallData, ReturnData}; + IndirectCallData, ReturnData, ReturnRegData}; use cretonne::isa; use cretonne::settings; use testfile::{TestFile, Details, Comment}; @@ -197,6 +197,11 @@ impl Context { InstructionData::Return { ref mut data, .. } => { try!(self.map.rewrite_values(&mut data.varargs, loc)); } + + InstructionData::ReturnReg { ref mut data, .. } => { + try!(self.map.rewrite_value(&mut data.arg, loc)); + try!(self.map.rewrite_values(&mut data.varargs, loc)); + } } } } @@ -1321,6 +1326,19 @@ impl<'a> Parser<'a> { data: Box::new(ReturnData { varargs: args }), } } + InstructionFormat::ReturnReg => { + let raddr = try!(self.match_value("expected SSA value return addr operand")); + try!(self.match_token(Token::Comma, "expected ',' between operands")); + let args = try!(self.parse_value_list()); + InstructionData::ReturnReg { + opcode: opcode, + ty: VOID, + data: Box::new(ReturnRegData { + arg: raddr, + varargs: args, + }), + } + } InstructionFormat::BranchTable => { let arg = try!(self.match_value("expected SSA value operand")); try!(self.match_token(Token::Comma, "expected ',' between operands")); From 608d452f0c3ae1a50d07125092ac00a976b38561 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 21 Feb 2017 16:18:45 -0800 Subject: [PATCH 0520/3084] Compute the controlling type variable accurately. Some polymorphic instructions don't return the controlling type variable, so it has to be computed from the designated operand instead. - Add a requires_typevar_operand() method to the operand constraints which indicates that. - Add a ctrl_typevar(dfg) method to InstructionData which computes the controlling type variable correctly, and returns VOID for monomorphic instructions. - Use ctrl_typevar(dfg) to drive the level-1 encoding table lookups. --- lib/cretonne/meta/gen_instr.py | 16 ++++++++++-- lib/cretonne/src/ir/instructions.rs | 40 +++++++++++++++++++++++++++-- lib/cretonne/src/isa/arm32/mod.rs | 4 +-- lib/cretonne/src/isa/arm64/mod.rs | 4 +-- lib/cretonne/src/isa/intel/mod.rs | 4 +-- lib/cretonne/src/isa/riscv/mod.rs | 4 +-- 6 files changed, 60 insertions(+), 12 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index d05bef96df..67ab23bda9 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -417,10 +417,20 @@ def gen_type_constraints(fmt, instrs): get_constraint(i.ins[idx], ctrl_typevar, type_sets)) offset = operand_seqs.add(constraints) fixed_results = len(i.value_results) + # Can the controlling type variable be inferred from the designated + # operand? use_typevar_operand = i.is_polymorphic and i.use_typevar_operand + # Can the controlling type variable be inferred from the result? + use_result = (fixed_results > 0 and + i.outs[i.value_results[0]].typevar != ctrl_typevar) + # Are we required to use the designated operand instead of the + # result? + requires_typevar_operand = use_typevar_operand and not use_result fmt.comment( - '{}: fixed_results={}, use_typevar_operand={}' - .format(i.camel_name, fixed_results, use_typevar_operand)) + ('{}: fixed_results={}, use_typevar_operand={}, ' + + 'requires_typevar_operand={}') + .format(i.camel_name, fixed_results, use_typevar_operand, + requires_typevar_operand)) fmt.comment('Constraints={}'.format(constraints)) if i.is_polymorphic: fmt.comment( @@ -430,6 +440,8 @@ def gen_type_constraints(fmt, instrs): flags = fixed_results if use_typevar_operand: flags |= 8 + if requires_typevar_operand: + flags |= 0x10 with fmt.indented('OpcodeConstraints {', '},'): fmt.line('flags: {:#04x},'.format(flags)) diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index d778ddb9a3..95311d48d8 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -14,6 +14,7 @@ use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, ImmVector}; use ir::condcodes::*; use ir::types; +use ir::DataFlowGraph; use ref_slice::*; use packed_option::PackedOption; @@ -492,6 +493,27 @@ impl InstructionData { _ => CallInfo::NotACall, } } + + /// Get the controlling type variable, or `VOID` if this instruction isn't polymorphic. + /// + /// In most cases, the controlling type variable is the same as the first result type, but some + /// opcodes require us to read the type of the designated type variable operand from `dfg`. + pub fn ctrl_typevar(&self, dfg: &DataFlowGraph) -> Type { + let constraints = self.opcode().constraints(); + + if !constraints.is_polymorphic() { + types::VOID + } else if constraints.requires_typevar_operand() { + // Not all instruction formats have a designated operand, but in that case + // `requires_typevar_operand()` should never be true. + dfg.value_type(self.typevar_operand() + .expect("Instruction format doesn't have a designated operand, bad opcode.")) + } else { + // For locality of reference, we prefer to get the controlling type variable from + // `idata` itself, when possible. + self.first_type() + } + } } /// Information about branch and jump instructions. @@ -537,8 +559,12 @@ pub struct OpcodeConstraints { /// Bit 3: /// This opcode is polymorphic and the controlling type variable can be inferred from the /// designated input operand. This is the `typevar_operand` index given to the - /// `InstructionFormat` meta language object. When bit 0 is not set, the controlling type - /// variable must be the first output value instead. + /// `InstructionFormat` meta language object. When this bit is not set, the controlling + /// type variable must be the first output value instead. + /// + /// Bit 4: + /// This opcode is polymorphic and the controlling type variable does *not* appear as the + /// first result type. flags: u8, /// Permitted set of types for the controlling type variable as an index into `TYPE_SETS`. @@ -559,6 +585,16 @@ impl OpcodeConstraints { (self.flags & 0x8) != 0 } + /// Is it necessary to look at the designated value input operand in order to determine the + /// controlling type variable, or is it good enough to use the first return type? + /// + /// Most polymorphic instructions produce a single result with the type of the controlling type + /// variable. A few polymorphic instructions either don't produce any results, or produce + /// results with a fixed type. These instructions return `true`. + pub fn requires_typevar_operand(self) -> bool { + (self.flags & 0x10) != 0 + } + /// Get the number of *fixed* result values produced by this opcode. /// This does not include `variable_args` produced by calls. pub fn fixed_results(self) -> usize { diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 63e9516a13..9af02883e3 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -53,8 +53,8 @@ impl TargetIsa for Isa { registers::INFO.clone() } - fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Result { - lookup_enclist(inst.first_type(), + fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Result { + lookup_enclist(inst.ctrl_typevar(dfg), inst.opcode(), self.cpumode, &enc_tables::LEVEL2[..]) diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 4d00b1d49e..50f54524a9 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -46,8 +46,8 @@ impl TargetIsa for Isa { registers::INFO.clone() } - fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Result { - lookup_enclist(inst.first_type(), + fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Result { + lookup_enclist(inst.ctrl_typevar(dfg), inst.opcode(), &enc_tables::LEVEL1_A64[..], &enc_tables::LEVEL2[..]) diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 533c264e3d..e5c99521e3 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -53,8 +53,8 @@ impl TargetIsa for Isa { registers::INFO.clone() } - fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Result { - lookup_enclist(inst.first_type(), + fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Result { + lookup_enclist(inst.ctrl_typevar(dfg), inst.opcode(), self.cpumode, &enc_tables::LEVEL2[..]) diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index b997316733..8b27a46119 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -53,8 +53,8 @@ impl TargetIsa for Isa { registers::INFO.clone() } - fn encode(&self, _: &DataFlowGraph, inst: &InstructionData) -> Result { - lookup_enclist(inst.first_type(), + fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Result { + lookup_enclist(inst.ctrl_typevar(dfg), inst.opcode(), self.cpumode, &enc_tables::LEVEL2[..]) From 377550b835f0f91a6834eea71823c6ff87e49a30 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 21 Feb 2017 15:17:33 -0800 Subject: [PATCH 0521/3084] Add return_reg encodings for RISC-V. --- cranelift/filetests/isa/riscv/encoding.cton | 4 +++- lib/cretonne/meta/isa/riscv/encodings.py | 11 ++++++++++- lib/cretonne/meta/isa/riscv/recipes.py | 13 ++++++++++++- lib/cretonne/src/write.rs | 4 ++-- lib/reader/src/parser.rs | 9 ++++++--- 5 files changed, 33 insertions(+), 8 deletions(-) diff --git a/cranelift/filetests/isa/riscv/encoding.cton b/cranelift/filetests/isa/riscv/encoding.cton index 2ba06007bb..f6defe27bf 100644 --- a/cranelift/filetests/isa/riscv/encoding.cton +++ b/cranelift/filetests/isa/riscv/encoding.cton @@ -15,5 +15,7 @@ ebb0(v1: i32, v2: i32): ; check: [R#10c] ; sameln: $v12 = imul - return + return_reg v1 + ; check: [Iret#19] + ; sameln: return_reg } diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index 0fc3a3ee8d..bf84c468b0 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -4,7 +4,7 @@ RISC-V Encodings. from __future__ import absolute_import from base import instructions as base from .defs import RV32, RV64 -from .recipes import OPIMM, OPIMM32, OP, OP32, R, Rshamt, I +from .recipes import OPIMM, OPIMM32, OP, OP32, JALR, R, Rshamt, I, Iret from .settings import use_m # Basic arithmetic binary instructions are encoded in an R-type instruction. @@ -52,3 +52,12 @@ for inst, inst_imm, f3, f7 in [ RV32.enc(base.imul.i32, R, OP(0b000, 0b0000001), isap=use_m) RV64.enc(base.imul.i64, R, OP(0b000, 0b0000001), isap=use_m) RV64.enc(base.imul.i32, R, OP32(0b000, 0b0000001), isap=use_m) + +# Control flow. + +# Returns are a special case of JALR. +# Note: Return stack predictors will only recognize this as a return when the +# return address is provided in `x1`. We may want a special encoding to enforce +# that. +RV32.enc(base.return_reg.i32, Iret, JALR()) +RV64.enc(base.return_reg.i64, Iret, JALR()) diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index fe0618d6cc..ea7e019eb4 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -11,7 +11,7 @@ instruction formats described in the reference: from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt -from base.formats import Binary, BinaryImm +from base.formats import Binary, BinaryImm, ReturnReg from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit @@ -40,6 +40,12 @@ def BRANCH(funct3): return 0b11000 | (funct3 << 5) +def JALR(funct3=0): + # type: (int) -> int + assert funct3 <= 0b111 + return 0b11001 | (funct3 << 5) + + def OPIMM(funct3, funct7=0): # type: (int, int) -> int assert funct3 <= 0b111 @@ -76,3 +82,8 @@ Rshamt = EncRecipe('Rshamt', BinaryImm, ins=GPR, outs=GPR) I = EncRecipe( 'I', BinaryImm, ins=GPR, outs=GPR, instp=IsSignedInt(BinaryImm.imm, 12)) + +# I-type encoding for `jalr` as a return instruction. We won't use the +# immediate offset. +# The variable return values are not encoded. +Iret = EncRecipe('Iret', ReturnReg, ins=GPR, outs=()) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 7a328a6d30..cb5fb8a24f 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -241,9 +241,9 @@ fn write_instruction(w: &mut Write, } ReturnReg { ref data, .. } => { if data.varargs.is_empty() { - writeln!(w, "{}", data.arg) + writeln!(w, " {}", data.arg) } else { - writeln!(w, "{}, {}", data.arg, data.varargs) + writeln!(w, " {}, {}", data.arg, data.varargs) } } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 8651e09d4d..f44cfd4dc8 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1327,9 +1327,12 @@ impl<'a> Parser<'a> { } } InstructionFormat::ReturnReg => { - let raddr = try!(self.match_value("expected SSA value return addr operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let args = try!(self.parse_value_list()); + let raddr = try!(self.match_value("expected SSA value return address operand")); + let args = if self.optional(Token::Comma) { + try!(self.parse_value_list()) + } else { + VariableArgs::new() + }; InstructionData::ReturnReg { opcode: opcode, ty: VOID, From 855c429d31a07231871e26673509dafc0ec785a2 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Wed, 22 Feb 2017 14:48:57 +0000 Subject: [PATCH 0522/3084] Documentation fix for what appears to be a minor copy-paste mistake. --- lib/cretonne/meta/base/instructions.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 1693763468..9cec7d9d58 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -254,8 +254,8 @@ fill = Instruction( 'fill', r""" Load a register value from a stack slot. - This instruction behaves exactly like :inst:`copy`, but the input - value is assigned to a spill slot. + This instruction behaves exactly like :inst:`copy`, but creates a new + SSA value for the spilled input value. """, ins=x, outs=a) From e002011602db2351876bbaae244c247213050432 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Wed, 22 Feb 2017 16:13:48 +0000 Subject: [PATCH 0523/3084] Removed the Opcode::NotAnOpcode variant, replaced its uses with Option, and used the NonZero optimization to maintain the small 1-byte size of an optional Opcode. --- lib/cretonne/meta/gen_encoding.py | 4 ++-- lib/cretonne/meta/gen_instr.py | 23 ++++++++++++++++------- lib/cretonne/src/ir/instructions.rs | 22 ++++++++-------------- lib/cretonne/src/isa/enc_tables.rs | 9 ++------- lib/cretonne/src/verifier.rs | 2 +- lib/reader/src/parser.rs | 2 +- 6 files changed, 30 insertions(+), 32 deletions(-) diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 3d4a00c4cc..c2849c1d6f 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -364,12 +364,12 @@ def emit_level2_hashtables(level2_hashtables, offt, level2_doc, fmt): if entry: fmt.line( 'Level2Entry ' + - '{{ opcode: Opcode::{}, offset: {:#08x} }},' + '{{ opcode: Some(Opcode::{}), offset: {:#08x} }},' .format(entry.inst.camel_name, entry.offset)) else: fmt.line( 'Level2Entry ' + - '{ opcode: Opcode::NotAnOpcode, offset: 0 },') + '{ opcode: None, offset: 0 },') def emit_level1_hashtable(cpumode, level1, offt, fmt): diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 67ab23bda9..1e83a1da2d 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -254,9 +254,13 @@ def gen_opcodes(groups, fmt): fmt.doc_comment('All instructions from all supported ISAs are present.') fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug)]') instrs = [] + + # We explicitly set the discriminant of the first variant to 1, which + # allows us to take advantage of the NonZero optimization, meaning that + # wrapping enums can use the 0 discriminant instead of increasing the size + # if the whole type, and so SIZEOF(Option>) == SIZEOF(Opcode) + is_first_opcode = True with fmt.indented('pub enum Opcode {', '}'): - fmt.doc_comment('An invalid opcode.') - fmt.line('NotAnOpcode,') for g in groups: for i in g.instructions: instrs.append(i) @@ -269,7 +273,13 @@ def gen_opcodes(groups, fmt): 'Type inferred from {}.' .format(i.ins[i.format.typevar_operand])) # Enum variant itself. - fmt.line(i.camel_name + ',') + if is_first_opcode: + fmt.doc_comment('We explicitly set this to 1 to allow the NonZero optimization,') + fmt.doc_comment('meaning that SIZEOF(Option) == SIZEOF(Opcode)') + fmt.line(i.camel_name + ' = 1,') + is_first_opcode = False + else: + fmt.line(i.camel_name + ',') fmt.line() with fmt.indented('impl Opcode {', '}'): @@ -318,7 +328,6 @@ def gen_opcodes(groups, fmt): # Generate a private opcode_name function. with fmt.indented('fn opcode_name(opc: Opcode) -> &\'static str {', '}'): with fmt.indented('match opc {', '}'): - fmt.line('Opcode::NotAnOpcode => "",') for i in instrs: fmt.format('Opcode::{} => "{}",', i.camel_name, i.name) fmt.line() @@ -328,13 +337,13 @@ def gen_opcodes(groups, fmt): instrs, lambda i: constant_hash.simple_hash(i.name)) with fmt.indented( - 'const OPCODE_HASH_TABLE: [Opcode; {}] = [' + 'const OPCODE_HASH_TABLE: [Option; {}] = [' .format(len(hash_table)), '];'): for i in hash_table: if i is None: - fmt.line('Opcode::NotAnOpcode,') + fmt.line('None,') else: - fmt.format('Opcode::{},', i.camel_name) + fmt.format('Some(Opcode::{}),', i.camel_name) fmt.line() return instrs diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 95311d48d8..73d49e7246 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -43,12 +43,8 @@ impl Display for Opcode { impl Opcode { /// Get the instruction format for this opcode. - pub fn format(self) -> Option { - if self == Opcode::NotAnOpcode { - None - } else { - Some(OPCODE_FORMAT[self as usize - 1]) - } + pub fn format(self) -> InstructionFormat { + OPCODE_FORMAT[self as usize - 1] } /// Get the constraint descriptor for this opcode. @@ -69,23 +65,21 @@ impl FromStr for Opcode { fn from_str(s: &str) -> Result { use constant_hash::{Table, simple_hash, probe}; - impl<'a> Table<&'a str> for [Opcode] { + impl<'a> Table<&'a str> for [Option] { fn len(&self) -> usize { self.len() } fn key(&self, idx: usize) -> Option<&'a str> { - if self[idx] == Opcode::NotAnOpcode { - None - } else { - Some(opcode_name(self[idx])) - } + self[idx].map(opcode_name) } } - match probe::<&str, [Opcode]>(&OPCODE_HASH_TABLE, s, simple_hash(s)) { + match probe::<&str, [Option]>(&OPCODE_HASH_TABLE, s, simple_hash(s)) { None => Err("Unknown opcode"), - Some(i) => Ok(OPCODE_HASH_TABLE[i]), + // We unwrap here because probe() should have ensured that the entry + // at this index is not None. + Some(i) => Ok(OPCODE_HASH_TABLE[i].unwrap()), } } } diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index d0a371dbc4..dd4e67875f 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -53,7 +53,7 @@ impl + Copy> Table for [Level1Entry] { /// /// Empty entries are encoded with a `NotAnOpcode` `opcode` field. pub struct Level2Entry + Copy> { - pub opcode: Opcode, + pub opcode: Option, pub offset: OffT, } @@ -63,12 +63,7 @@ impl + Copy> Table for [Level2Entry] { } fn key(&self, idx: usize) -> Option { - let opc = self[idx].opcode; - if opc != Opcode::NotAnOpcode { - Some(opc) - } else { - None - } + self[idx].opcode } } diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 0076933934..ea5941006e 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -149,7 +149,7 @@ impl<'a> Verifier<'a> { let inst_data = &self.func.dfg[inst]; // The instruction format matches the opcode - if inst_data.opcode().format() != Some(InstructionFormat::from(inst_data)) { + if inst_data.opcode().format() != InstructionFormat::from(inst_data) { return err!(inst, "instruction opcode doesn't match instruction format"); } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index f44cfd4dc8..204b78c9e5 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1088,7 +1088,7 @@ impl<'a> Parser<'a> { // Parse the operands following the instruction opcode. // This depends on the format of the opcode. fn parse_inst_operands(&mut self, ctx: &Context, opcode: Opcode) -> Result { - Ok(match opcode.format().unwrap() { + Ok(match opcode.format() { InstructionFormat::Nullary => { InstructionData::Nullary { opcode: opcode, From 41ca00df8dbdb5a2e2d16e8297917192f84dec10 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Wed, 22 Feb 2017 17:07:07 +0000 Subject: [PATCH 0524/3084] Fix test case that I missed before. --- lib/cretonne/src/ir/instructions.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 73d49e7246..216e88b883 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -742,7 +742,7 @@ mod tests { assert!(x != y); y = Opcode::Iadd; assert_eq!(x, y); - assert_eq!(x.format(), Some(InstructionFormat::Binary)); + assert_eq!(x.format(), InstructionFormat::Binary); assert_eq!(format!("{:?}", Opcode::IaddImm), "IaddImm"); assert_eq!(Opcode::IaddImm.to_string(), "iadd_imm"); From b2a3b3402228691a8b0c9cdcdc1d2fb61b2ce088 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Wed, 22 Feb 2017 18:07:14 +0000 Subject: [PATCH 0525/3084] Add assertion that the NonZero optimization works on Option. --- lib/cretonne/src/ir/instructions.rs | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 216e88b883..839219fc2a 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -736,6 +736,8 @@ mod tests { #[test] fn opcodes() { + use std::mem; + let x = Opcode::Iadd; let mut y = Opcode::Isub; @@ -753,6 +755,12 @@ mod tests { assert_eq!("iadd\0".parse::(), Err("Unknown opcode")); assert_eq!("".parse::(), Err("Unknown opcode")); assert_eq!("\0".parse::(), Err("Unknown opcode")); + + // Opcode is a single byte, and because Option originally came to 2 bytes, early on + // Opcode included a variant NotAnOpcode to avoid the unnecessary bloat. Since then the Rust + // compiler has brought in NonZero optimization, meaning that an enum not using the 0 value + // can be optional for no size cost. We want to ensure Option remains small. + assert_eq!(mem::size_of::(), mem::size_of::>()); } #[test] From 1e4096b8b885914fb8932c6b55423ab46109dbe3 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Wed, 22 Feb 2017 18:10:09 +0000 Subject: [PATCH 0526/3084] Removed unnecessary documentation. --- lib/cretonne/meta/gen_instr.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 1e83a1da2d..8cf89e7654 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -274,8 +274,6 @@ def gen_opcodes(groups, fmt): .format(i.ins[i.format.typevar_operand])) # Enum variant itself. if is_first_opcode: - fmt.doc_comment('We explicitly set this to 1 to allow the NonZero optimization,') - fmt.doc_comment('meaning that SIZEOF(Option) == SIZEOF(Opcode)') fmt.line(i.camel_name + ' = 1,') is_first_opcode = False else: From 8e421d666d184a2962a005191957d1abf91c36de Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 15 Feb 2017 13:26:52 -0800 Subject: [PATCH 0527/3084] SSA register coloring pass. This is a bare-bones outline of the SSA coloring pass. Many features are missing, including: - Handling instruction operand constraints beyond simple register classes. - Handling ABI requirements for function arguments and return values. - Generating shuffle code for EBB arguments. --- lib/cretonne/src/isa/constraints.rs | 1 + lib/cretonne/src/regalloc/coloring.rs | 337 ++++++++++++++++++++++++++ lib/cretonne/src/regalloc/mod.rs | 1 + 3 files changed, 339 insertions(+) create mode 100644 lib/cretonne/src/regalloc/coloring.rs diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index 43a0ddb271..2b8700ec4a 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -49,6 +49,7 @@ pub enum ConstraintKind { } /// Constraints for an encoding recipe. +#[derive(Clone)] pub struct RecipeConstraints { /// Constraints for the instruction's fixed value operands. /// diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs new file mode 100644 index 0000000000..11018f664d --- /dev/null +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -0,0 +1,337 @@ +//! Register allocator coloring pass. +//! +//! The coloring pass assigns a physical register to every SSA value with a register affinity, +//! under the assumption that the register pressure has been lowered sufficiently by spilling and +//! splitting. +//! +//! # Preconditions +//! +//! The coloring pass doesn't work on arbitrary code. Certain preconditions must be satisfied: +//! +//! 1. All instructions must be legalized and assigned an encoding. The encoding recipe guides the +//! register assignments and provides exact constraints. +//! +//! 2. Instructions with tied operands must be in a coloring-friendly state. Specifically, the +//! values used by the tied operands must be killed by the instruction. This can be achieved by +//! inserting a `copy` to a new value immediately before the two-address instruction. +//! +//! 3. The register pressure must be lowered sufficiently by inserting spill code. Register +//! operands are allowed to read spilled values, but each such instance must be counted as using +//! a register. +//! +//! # Iteration order +//! +//! The SSA property guarantees that whenever the live range of two values overlap, one of the +//! values will be live at the definition point of the other value. If we visit the instructions in +//! a topological order relative to the dominance relation, we can assign colors to the values +//! defined by the instruction and only consider the colors of other values that are live at the +//! instruction. +//! +//! The topological order of instructions inside an EBB is simply the layout order, starting from +//! the EBB header. A topological order of the EBBs can only visit an EBB once its immediate +//! dominator has been visited. +//! +//! There are many valid topological orders of the EBBs, and the specific order can affect which +//! coloring hints are satisfied and which are broken. +//! + +use entity_map::EntityMap; +use dominator_tree::DominatorTree; +use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph}; +use isa::{TargetIsa, RegInfo, Encoding, RecipeConstraints, ConstraintKind}; +use regalloc::affinity::Affinity; +use regalloc::allocatable_set::AllocatableSet; +use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; +use regalloc::liveness::Liveness; +use sparse_map::SparseSet; + + +/// Data structures for the coloring pass. +/// +/// These are scratch space data structures that can be reused between invocations. +pub struct Coloring { + /// Set of visited EBBs. + visited: SparseSet, + + /// Stack of EBBs to be visited next. + stack: Vec, +} + +/// Bundle of references that the coloring algorithm needs. +/// +/// Some of the needed mutable references are passed around as explicit function arguments so we +/// can avoid many fights with the borrow checker over mutable borrows of `self`. This includes the +/// `Function` and `LiveValueTracker` references. +/// +/// Immutable context information and mutable references that don't need to be borrowed across +/// method calls should go in this struct. +struct Context<'a> { + // Cached ISA information. + // We save it here to avoid frequent virtual function calls on the `TargetIsa` trait object. + reginfo: RegInfo, + recipe_constraints: &'a [RecipeConstraints], + + // References to contextual data structures we need. + domtree: &'a DominatorTree, + liveness: &'a mut Liveness, + + // Pristine set of registers that the allocator can use. + // This set remains immutable, we make clones. + usable_regs: AllocatableSet, +} + +impl Coloring { + /// Allocate scratch space data structures for the coloring pass. + pub fn new() -> Coloring { + Coloring { + visited: SparseSet::new(), + stack: Vec::new(), + } + } + + /// Run the coloring algorithm over `func`. + pub fn run(&mut self, + isa: &TargetIsa, + func: &mut Function, + domtree: &DominatorTree, + liveness: &mut Liveness, + tracker: &mut LiveValueTracker) { + let mut ctx = Context { + reginfo: isa.register_info(), + recipe_constraints: isa.recipe_constraints(), + domtree: domtree, + liveness: liveness, + // TODO: Ask the target ISA about reserved registers etc. + usable_regs: AllocatableSet::new(), + }; + ctx.run(self, func, tracker) + } +} + +impl<'a> Context<'a> { + /// Run the coloring algorithm. + fn run(&mut self, data: &mut Coloring, func: &mut Function, tracker: &mut LiveValueTracker) { + // Just visit blocks in layout order, letting `process_ebb` enforce a topological ordering. + // TODO: Once we have a loop tree, we could visit hot blocks first. + let mut next = func.layout.entry_block(); + while let Some(ebb) = next { + self.process_ebb(ebb, data, func, tracker); + next = func.layout.next_ebb(ebb); + } + } + + /// Process `ebb`, but only after ensuring that the immediate dominator has been processed. + /// + /// This method can be called with the most desired order of visiting the EBBs. It will convert + /// that order into a valid topological order by visiting dominators first. + fn process_ebb(&mut self, + mut ebb: Ebb, + data: &mut Coloring, + func: &mut Function, + tracker: &mut LiveValueTracker) { + // The stack is just a scratch space for this algorithm. We leave it empty when returning. + assert!(data.stack.is_empty()); + + // Trace up the dominator tree until we reach a dominator that has already been visited. + while data.visited.insert(ebb).is_none() { + data.stack.push(ebb); + match self.domtree.idom(ebb) { + Some(idom) => ebb = func.layout.inst_ebb(idom).expect("idom not in layout"), + None => break, + } + } + + // Pop off blocks in topological order. + while let Some(ebb) = data.stack.pop() { + self.visit_ebb(ebb, func, tracker); + } + } + + /// Visit `ebb`, assuming that the immediate dominator has already been visited. + fn visit_ebb(&mut self, ebb: Ebb, func: &mut Function, tracker: &mut LiveValueTracker) { + let mut regs = self.visit_ebb_header(ebb, func, tracker); + + // Now go through the instructions in `ebb` and color the values they define. + let mut pos = Cursor::new(&mut func.layout); + pos.goto_top(ebb); + while let Some(inst) = pos.next_inst() { + let encoding = func.encodings[inst]; + assert!(encoding.is_legal(), "Illegal: {}", func.dfg[inst].opcode()); + self.visit_inst(inst, + encoding, + &mut pos, + &mut func.dfg, + tracker, + &mut regs, + &mut func.locations); + tracker.drop_dead(inst); + } + + } + + /// Visit the `ebb` header. + /// + /// Initialize the set of live registers and color the arguments to `ebb`. + fn visit_ebb_header(&self, + ebb: Ebb, + func: &mut Function, + tracker: &mut LiveValueTracker) + -> AllocatableSet { + // Reposition the live value tracker and deal with the EBB arguments. + let (liveins, args) = + tracker.ebb_top(ebb, &func.dfg, self.liveness, &func.layout, self.domtree); + + // The live-ins have already been assigned a register. Reconstruct the allocatable set. + let mut regs = self.livein_regs(liveins, func); + + // TODO: Arguments to the entry block are pre-colored by the ABI. We should probably call + // a whole other function for that case. + self.color_args(args, &mut regs, &mut func.locations); + + regs + } + + /// Initialize a set of allocatable registers from the values that are live-in to a block. + /// These values must already be colored when the dominating blocks were processed. + fn livein_regs(&self, liveins: &[LiveValue], func: &Function) -> AllocatableSet { + // Start from the registers that are actually usable. We don't want to include any reserved + // registers in the set. + let mut regs = self.usable_regs.clone(); + + for lv in liveins { + let value = lv.value; + let affinity = self.liveness.get(value).expect("No live range for live-in").affinity; + if let Affinity::Reg(rc_index) = affinity { + let regclass = self.reginfo.rc(rc_index); + match func.locations[value] { + ValueLoc::Reg(regunit) => regs.take(regclass, regunit), + ValueLoc::Unassigned => panic!("Live-in {} wasn't assigned", value), + ValueLoc::Stack(ss) => { + panic!("Live-in {} is in {}, should be register", value, ss) + } + } + } + } + + regs + } + + /// Color the live arguments to the current block. + /// + /// It is assumed that any live-in register values have already been taken out of the register + /// set. + fn color_args(&self, + args: &[LiveValue], + regs: &mut AllocatableSet, + locations: &mut EntityMap) { + for lv in args { + // Only look at the register arguments. + if let Affinity::Reg(rc_index) = lv.affinity { + let regclass = self.reginfo.rc(rc_index); + // TODO: Fall back to a top-level super-class. Sub-classes are only hints. + let regunit = regs.iter(regclass).next().expect("Out of registers for arguments"); + regs.take(regclass, regunit); + *locations.ensure(lv.value) = ValueLoc::Reg(regunit); + } + } + } + + /// Color the values defined by `inst` and insert any necessary shuffle code to satisfy + /// instruction constraints. + /// + /// Update `regs` to reflect the allocated registers after `inst`, including removing any dead + /// or killed values from the set. + fn visit_inst(&self, + inst: Inst, + encoding: Encoding, + _pos: &mut Cursor, + dfg: &mut DataFlowGraph, + tracker: &mut LiveValueTracker, + regs: &mut AllocatableSet, + locations: &mut EntityMap) { + // First update the live value tracker with this instruction. + // Get lists of values that are killed and defined by `inst`. + let (kills, defs) = tracker.process_inst(inst, dfg, self.liveness); + + // Get the operand constraints for `inst` that we are trying to satisfy. + let constraints = self.recipe_constraints[encoding.recipe()].clone(); + + // Get rid of the killed values. + for lv in kills { + if let Affinity::Reg(rc_index) = lv.affinity { + let regclass = self.reginfo.rc(rc_index); + if let ValueLoc::Reg(regunit) = locations[lv.value] { + regs.free(regclass, regunit); + } + } + } + + // Process the defined values with fixed constraints. + // TODO: Handle constraints on call return values. + assert_eq!(defs.len(), + constraints.outs.len(), + "Can't handle variable results"); + for (lv, opcst) in defs.iter().zip(constraints.outs) { + match lv.affinity { + // This value should go in a register. + Affinity::Reg(rc_index) => { + // The preferred register class is not a requirement. + let pref_rc = self.reginfo.rc(rc_index); + match opcst.kind { + ConstraintKind::Reg => { + // This is a standard register constraint. The preferred register class + // should have been computed as a subclass of the hard constraint of + // the def. + assert!(opcst.regclass.has_subclass(rc_index), + "{} preference {} is not compatible with the definition \ + constraint {}", + lv.value, + pref_rc.name, + opcst.regclass.name); + // Try to grab a register from the preferred class, but fall back to + // the actual constraint if we have to. + let regunit = regs.iter(pref_rc) + .next() + .or_else(|| regs.iter(opcst.regclass).next()) + .expect("Ran out of registers"); + regs.take(opcst.regclass, regunit); + *locations.ensure(lv.value) = ValueLoc::Reg(regunit); + } + ConstraintKind::Tied(arg_index) => { + // This def must use the same register as a fixed instruction argument. + let loc = locations[dfg[inst].arguments()[0][arg_index as usize]]; + *locations.ensure(lv.value) = loc; + // Mark the reused register. It's not really clear if we support tied + // stack operands. We could do that for some Intel read-modify-write + // encodings. + if let ValueLoc::Reg(regunit) = loc { + // This is going to assert out unless the incoming value at + // `arg_index` was killed. Tied operands must be fixed to + // ensure that before running the coloring pass. + regs.take(opcst.regclass, regunit); + } + } + ConstraintKind::FixedReg(_regunit) => unimplemented!(), + ConstraintKind::Stack => { + panic!("{}:{} should be a stack value", lv.value, pref_rc.name) + } + } + } + Affinity::Stack => unimplemented!(), + Affinity::Any => unimplemented!(), + } + } + + // Get rid of the dead defs. + for lv in defs { + if lv.endpoint == inst { + if let Affinity::Reg(rc_index) = lv.affinity { + let regclass = self.reginfo.rc(rc_index); + if let ValueLoc::Reg(regunit) = locations[lv.value] { + regs.free(regclass, regunit); + } + } + } + } + } +} diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index 902e7aabb0..19bb350499 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -6,5 +6,6 @@ pub mod liverange; pub mod liveness; pub mod allocatable_set; pub mod live_value_tracker; +pub mod coloring; mod affinity; From bf9cf09622d980ebc992ec4e02e7e55a3ae2549a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 17 Feb 2017 10:55:47 -0800 Subject: [PATCH 0528/3084] Add a register allocation context module. Collect the data structures that hang around between function compilations. Provide a main entry point to the register allocator passes. --- lib/cretonne/src/context.rs | 22 +++++++++++ lib/cretonne/src/regalloc/context.rs | 57 +++++++++++++++++++++++++++ lib/cretonne/src/regalloc/liveness.rs | 2 +- lib/cretonne/src/regalloc/mod.rs | 3 ++ 4 files changed, 83 insertions(+), 1 deletion(-) create mode 100644 lib/cretonne/src/regalloc/context.rs diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 31a1569d5f..ce1e8b50ff 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -3,10 +3,18 @@ //! When compiling many small functions, it is important to avoid repeatedly allocating and //! deallocating the data structures needed for compilation. The `Context` struct is used to hold //! on to memory allocations between function compilations. +//! +//! The context does not hold a `TargetIsa` instance which has to be provided as an argument +//! instead. This is because an ISA instance is immutable and can be used by multiple compilation +//! contexts concurrently. Typically, you would have one context per compilation thread and only a +//! single ISA instance. use cfg::ControlFlowGraph; use dominator_tree::DominatorTree; use ir::Function; +use isa::TargetIsa; +use legalize_function; +use regalloc; /// Persistent data structures and compilation pipeline. pub struct Context { @@ -18,6 +26,9 @@ pub struct Context { /// Dominator tree for `func`. pub domtree: DominatorTree, + + /// Register allocation context. + pub regalloc: regalloc::Context, } impl Context { @@ -30,12 +41,23 @@ impl Context { func: Function::new(), cfg: ControlFlowGraph::new(), domtree: DominatorTree::new(), + regalloc: regalloc::Context::new(), } } + /// Run the legalizer for `isa` on the function. + pub fn legalize(&mut self, isa: &TargetIsa) { + legalize_function(&mut self.func, isa); + } + /// Recompute the control flow graph and dominator tree. pub fn flowgraph(&mut self) { self.cfg.compute(&self.func); self.domtree.compute(&self.func, &self.cfg); } + + /// Run the register allocator. + pub fn regalloc(&mut self, isa: &TargetIsa) { + self.regalloc.run(isa, &mut self.func, &self.cfg, &self.domtree); + } } diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs new file mode 100644 index 0000000000..75863fbc63 --- /dev/null +++ b/lib/cretonne/src/regalloc/context.rs @@ -0,0 +1,57 @@ +//! Register allocator context. +//! +//! The `Context` struct contains data structures that should be preserved across invocations of +//! the register allocator algorithm. This doesn't preserve any data between functions, but it +//! avoids allocating data structures independently for each function begin compiled. + +use dominator_tree::DominatorTree; +use ir::Function; +use regalloc::coloring::Coloring; +use regalloc::live_value_tracker::LiveValueTracker; +use regalloc::liveness::Liveness; +use isa::TargetIsa; +use cfg::ControlFlowGraph; + +/// Persistent memory allocations for register allocation. +pub struct Context { + liveness: Liveness, + tracker: LiveValueTracker, + coloring: Coloring, +} + +impl Context { + /// Create a new context for register allocation. + /// + /// This context should be reused for multiple functions in order to avoid repeated memory + /// allocations. + pub fn new() -> Context { + Context { + liveness: Liveness::new(), + tracker: LiveValueTracker::new(), + coloring: Coloring::new(), + } + } + + /// Allocate registers in `func`. + /// + /// After register allocation, all values in `func` have been assigned to a register or stack + /// location that is consistent with instruction encoding constraints. + pub fn run(&mut self, + isa: &TargetIsa, + func: &mut Function, + cfg: &ControlFlowGraph, + domtree: &DominatorTree) { + // `Liveness` and `Coloring` are self-clearing. + // Tracker state (dominator live sets) is actually reused between the spilling and coloring + // phases. + self.tracker.clear(); + + // First pass: Liveness analysis. + self.liveness.compute(isa, func, cfg); + + // TODO: Second pass: Spilling. + + // Third pass: Reload and coloring. + self.coloring.run(isa, func, domtree, &mut self.liveness, &mut self.tracker); + } +} diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index a8591e8597..524b50e8ea 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -291,7 +291,7 @@ impl Liveness { /// Compute the live ranges of all SSA values used in `func`. /// This clears out any existing analysis stored in this data structure. - pub fn compute(&mut self, func: &Function, cfg: &ControlFlowGraph, isa: &TargetIsa) { + pub fn compute(&mut self, isa: &TargetIsa, func: &Function, cfg: &ControlFlowGraph) { self.ranges.clear(); // Get ISA data structures used for computing live range affinities. diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index 19bb350499..fb19286670 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -9,3 +9,6 @@ pub mod live_value_tracker; pub mod coloring; mod affinity; +mod context; + +pub use self::context::Context; From 247be5704282a2759816c0df4a31edab175f38f2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 22 Feb 2017 11:41:30 -0800 Subject: [PATCH 0529/3084] Also write out register assignments in write_instruction. The value locations appear after the encodings: > [R#0c,%x2] v0 = iadd vx0, vx1 > [Iret#19] return_reg v0 --- lib/cretonne/src/write.rs | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index cb5fb8a24f..933ba8822e 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -4,7 +4,7 @@ //! equivalent textual representation. This textual representation can be read back by the //! `cretonne-reader` crate. -use ir::{Function, Ebb, Inst, Value, Type}; +use ir::{Function, Ebb, Inst, Value, Type, ValueLoc}; use isa::TargetIsa; use std::fmt::{Result, Error, Write}; use std::result; @@ -172,7 +172,19 @@ fn write_instruction(w: &mut Write, if let Some(enc) = func.encodings.get(inst).cloned() { let mut s = String::with_capacity(16); if let Some(isa) = isa { - try!(write!(s, "[{}]", isa.display_enc(enc))); + try!(write!(s, "[{}", isa.display_enc(enc))); + // Write value locations, if we have them. + if !func.locations.is_empty() { + let regs = isa.register_info(); + for r in func.dfg.inst_results(inst) { + match func.locations[r] { + ValueLoc::Unassigned => write!(s, ",-")?, + ValueLoc::Reg(ru) => write!(s, ",{}", regs.display_regunit(ru))?, + ValueLoc::Stack(ss) => write!(s, ",{}", ss)?, + } + } + } + try!(write!(s, "]")); } else { try!(write!(s, "[{}]", enc)); } From 04bddd73ba576e1bf6d5c9ee164dba0917e5a986 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 17 Feb 2017 17:07:08 -0800 Subject: [PATCH 0530/3084] Add a 'regalloc' filetest command. Run functions through the register allocator, and then filecheck. --- cranelift/docs/testing.rst | 14 +++++++ cranelift/filetests/regalloc/basic.cton | 12 ++++++ cranelift/src/filetest/mod.rs | 11 +++-- cranelift/src/filetest/regalloc.rs | 55 +++++++++++++++++++++++++ 4 files changed, 88 insertions(+), 4 deletions(-) create mode 100644 cranelift/filetests/regalloc/basic.cton create mode 100644 cranelift/src/filetest/regalloc.rs diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index a452dc1b58..db027f8fbe 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -300,3 +300,17 @@ Legalize each function for the specified target ISA and run the resulting function through filecheck. This test command can be used to validate the encodings selected for legal instructions as well as the instruction transformations performed by the legalizer. + +`test regalloc` +--------------- + +Test the register allocator. + +First, each function is legalized for the specified target ISA. This is +required for register allocation since the instruction encodings provide +register class constraints to the register allocator. + +Second, the register allocator is run on the function, inserting spill code and +assigning registers and stack slots to all values. + +The resulting function is then run through filecheck. diff --git a/cranelift/filetests/regalloc/basic.cton b/cranelift/filetests/regalloc/basic.cton new file mode 100644 index 0000000000..00e204f9b1 --- /dev/null +++ b/cranelift/filetests/regalloc/basic.cton @@ -0,0 +1,12 @@ +test regalloc + +; We can add more ISAs once they have defined encodings. +isa riscv + +function add(i32, i32) { +ebb0(v1: i32, v2: i32): + v3 = iadd v1, v2 +; check: [R#0c,%x0] +; sameln: iadd + return_reg v3 +} diff --git a/cranelift/src/filetest/mod.rs b/cranelift/src/filetest/mod.rs index 28a9b94d45..41b89948a1 100644 --- a/cranelift/src/filetest/mod.rs +++ b/cranelift/src/filetest/mod.rs @@ -12,12 +12,14 @@ use print_cfg; use filetest::runner::TestRunner; pub mod subtest; -mod runner; -mod runone; + mod concurrent; mod domtree; -mod verifier; mod legalizer; +mod regalloc; +mod runner; +mod runone; +mod verifier; /// The result of running the test in a file. pub type TestResult = Result; @@ -49,7 +51,7 @@ pub fn run(verbose: bool, files: Vec) -> CommandResult { /// Create a new subcommand trait object to match `parsed.command`. /// /// This function knows how to create all of the possible `test ` commands that can appear in -/// a .cton test file. +/// a `.cton` test file. fn new_subtest(parsed: &TestCommand) -> subtest::Result> { match parsed.command { "cat" => cat::subtest(parsed), @@ -57,6 +59,7 @@ fn new_subtest(parsed: &TestCommand) -> subtest::Result> { "domtree" => domtree::subtest(parsed), "verifier" => verifier::subtest(parsed), "legalizer" => legalizer::subtest(parsed), + "regalloc" => regalloc::subtest(parsed), _ => Err(format!("unknown test command '{}'", parsed.command)), } } diff --git a/cranelift/src/filetest/regalloc.rs b/cranelift/src/filetest/regalloc.rs new file mode 100644 index 0000000000..af9dbb2b23 --- /dev/null +++ b/cranelift/src/filetest/regalloc.rs @@ -0,0 +1,55 @@ +//! Test command for testing the register allocator. +//! +//! The `regalloc` test command runs each function through the register allocator after ensuring +//! that all instructions are legal for the target. +//! +//! The resulting function is sent to `filecheck`. + +use std::borrow::Cow; +use cretonne::{self, write_function}; +use cretonne::ir::Function; +use cton_reader::TestCommand; +use filetest::subtest::{SubTest, Context, Result, run_filecheck}; + +struct TestRegalloc; + +pub fn subtest(parsed: &TestCommand) -> Result> { + assert_eq!(parsed.command, "regalloc"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestRegalloc)) + } +} + +impl SubTest for TestRegalloc { + fn name(&self) -> Cow { + Cow::from("regalloc") + } + + fn is_mutating(&self) -> bool { + true + } + + fn needs_isa(&self) -> bool { + true + } + + fn run(&self, func: Cow, context: &Context) -> Result<()> { + let isa = context.isa.expect("register allocator needs an ISA"); + + // Create a compilation context, and drop in the function. + let mut comp_ctx = cretonne::Context::new(); + comp_ctx.func = func.into_owned(); + + // TODO: Should we have an option to skip legalization? + comp_ctx.legalize(isa); + + comp_ctx.flowgraph(); + comp_ctx.regalloc(isa); + + let mut text = String::new(); + try!(write_function(&mut text, &comp_ctx.func, Some(isa)).map_err(|e| e.to_string())); + run_filecheck(&text, context) + } +} From a08e1775951e5861fbe2ab3a7bf74f51bff34591 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Thu, 23 Feb 2017 00:28:19 +0000 Subject: [PATCH 0531/3084] Lexer can now scan names, hex sequences, brackets and minus signs. --- lib/reader/src/lexer.rs | 49 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 04801b2644..7e33e8ecd0 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -22,6 +22,9 @@ pub enum Token<'a> { RPar, // ')' LBrace, // '{' RBrace, // '}' + LBracket, // '[' + RBracket, // ']' + Minus, // '-' Comma, // ',' Dot, // '.' Colon, // ':' @@ -36,6 +39,8 @@ pub enum Token<'a> { JumpTable(u32), // jt2 FuncRef(u32), // fn2 SigRef(u32), // sig2 + Name(&'a str), // %9arbitrary_alphanum, %x3, %0, %function ... + HexSequence(&'a str), // #89AF Identifier(&'a str), // Unrecognized identifier (opcode, enumerator, ...) } @@ -222,6 +227,13 @@ impl<'a> Lexer<'a> { // Skip a leading sign. if self.lookahead == Some('-') { self.next_ch(); + + if let Some(c) = self.lookahead { + // If the next character won't parse as a number, we conservatively return Token::Minus + if !c.is_alphanumeric() && c != '.' { + return token(Token::Minus, loc); + } + } } // Check for NaNs with payloads. @@ -326,6 +338,39 @@ impl<'a> Lexer<'a> { } } + fn scan_name(&mut self) -> Result, LocatedError> { + let loc = self.loc(); + let begin = self.pos; + + assert!(self.lookahead == Some('%')); + + while let Some(c) = self.next_ch() { + if !c.is_alphanumeric() && c != '_' { + break; + } + } + + let end = self.pos; + token(Token::Name(&self.source[begin..end]), loc) + } + + fn scan_hex_sequence(&mut self) -> Result, LocatedError> { + let loc = self.loc(); + let begin = self.pos; + + assert!(self.lookahead == Some('#')); + + while let Some(c) = self.next_ch() { + match c { + 'a'...'f' | 'A'...'F' | '0'...'9' => {}, + _ => break, + } + } + + let end = self.pos; + token(Token::HexSequence(&self.source[begin..end]), loc) + } + /// Get the next token or a lexical error. /// /// Return None when the end of the source is encountered. @@ -339,6 +384,8 @@ impl<'a> Lexer<'a> { Some(')') => Some(self.scan_char(Token::RPar)), Some('{') => Some(self.scan_char(Token::LBrace)), Some('}') => Some(self.scan_char(Token::RBrace)), + Some('[') => Some(self.scan_char(Token::LBracket)), + Some(']') => Some(self.scan_char(Token::RBracket)), Some(',') => Some(self.scan_char(Token::Comma)), Some('.') => Some(self.scan_char(Token::Dot)), Some(':') => Some(self.scan_char(Token::Colon)), @@ -352,6 +399,8 @@ impl<'a> Lexer<'a> { } Some(ch) if ch.is_digit(10) => Some(self.scan_number()), Some(ch) if ch.is_alphabetic() => Some(self.scan_word()), + Some('%') => Some(self.scan_name()), + Some('#') => Some(self.scan_hex_sequence()), Some(ch) if ch.is_whitespace() => { self.next_ch(); continue; From 54a53b7ab733e71f916bb7073a49b6ddebb51a73 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Thu, 23 Feb 2017 01:07:25 +0000 Subject: [PATCH 0532/3084] Added tests, some fixes. --- lib/reader/src/lexer.rs | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 7e33e8ecd0..474b8d04e0 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -7,6 +7,7 @@ use std::str::CharIndices; use std::u16; +use std::ascii::AsciiExt; use cretonne::ir::types; use cretonne::ir::{Value, Ebb}; use error::Location; @@ -340,12 +341,12 @@ impl<'a> Lexer<'a> { fn scan_name(&mut self) -> Result, LocatedError> { let loc = self.loc(); - let begin = self.pos; + let begin = self.pos + 1; assert!(self.lookahead == Some('%')); while let Some(c) = self.next_ch() { - if !c.is_alphanumeric() && c != '_' { + if !(c.is_ascii() && c.is_alphanumeric() || c == '_') { break; } } @@ -356,14 +357,13 @@ impl<'a> Lexer<'a> { fn scan_hex_sequence(&mut self) -> Result, LocatedError> { let loc = self.loc(); - let begin = self.pos; + let begin = self.pos + 1; assert!(self.lookahead == Some('#')); while let Some(c) = self.next_ch() { - match c { - 'a'...'f' | 'A'...'F' | '0'...'9' => {}, - _ => break, + if !char::is_digit(c, 16) { + break; } } @@ -478,7 +478,7 @@ mod tests { assert_eq!(lex.next(), None); // Scan a comment after an invalid char. - let mut lex = Lexer::new("#; hello"); + let mut lex = Lexer::new("$; hello"); assert_eq!(lex.next(), error(Error::InvalidChar, 1)); assert_eq!(lex.next(), token(Token::Comment("; hello"), 1)); assert_eq!(lex.next(), None); @@ -535,4 +535,27 @@ mod tests { assert_eq!(lex.next(), token(Token::Identifier("f32x5"), 1)); assert_eq!(lex.next(), None); } + + #[test] + fn lex_hex_sequences() { + let mut lex = Lexer::new("#0 #DEADbeef123 #789"); + + assert_eq!(lex.next(), token(Token::HexSequence("0"), 1)); + assert_eq!(lex.next(), token(Token::HexSequence("DEADbeef123"), 1)); + assert_eq!(lex.next(), token(Token::HexSequence("789"), 1)); + } + + #[test] + fn lex_names() { + let mut lex = Lexer::new("%0 %x3 %function %123_abc %ss0 %v3 %ebb11 %_"); + + assert_eq!(lex.next(), token(Token::Name("0"), 1)); + assert_eq!(lex.next(), token(Token::Name("x3"), 1)); + assert_eq!(lex.next(), token(Token::Name("function"), 1)); + assert_eq!(lex.next(), token(Token::Name("123_abc"), 1)); + assert_eq!(lex.next(), token(Token::Name("ss0"), 1)); + assert_eq!(lex.next(), token(Token::Name("v3"), 1)); + assert_eq!(lex.next(), token(Token::Name("ebb11"), 1)); + assert_eq!(lex.next(), token(Token::Name("_"), 1)); + } } From 65efefff3868efc29eac87850d2919f064ea39bb Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Thu, 23 Feb 2017 01:29:32 +0000 Subject: [PATCH 0533/3084] Shortened comment to pass lint. --- lib/reader/src/lexer.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 474b8d04e0..caf32ed252 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -230,7 +230,7 @@ impl<'a> Lexer<'a> { self.next_ch(); if let Some(c) = self.lookahead { - // If the next character won't parse as a number, we conservatively return Token::Minus + // If the next character won't parse as a number, we return Token::Minus if !c.is_alphanumeric() && c != '.' { return token(Token::Minus, loc); } From db9e4ea9791cd5d8f1b46b1eae4795463d5d5503 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 24 Feb 2017 10:33:08 -0800 Subject: [PATCH 0534/3084] Convert try! to ? in extfunc.rs --- lib/cretonne/src/ir/extfunc.rs | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index a47ff4ad7a..ae012e7530 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -34,9 +34,9 @@ fn write_list(f: &mut Formatter, args: &Vec) -> fmt::Result { match args.split_first() { None => {} Some((first, rest)) => { - try!(write!(f, "{}", first)); + write!(f, "{}", first)?; for arg in rest { - try!(write!(f, ", {}", arg)); + write!(f, ", {}", arg)?; } } } @@ -45,12 +45,12 @@ fn write_list(f: &mut Formatter, args: &Vec) -> fmt::Result { impl Display for Signature { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - try!(write!(f, "(")); - try!(write_list(f, &self.argument_types)); - try!(write!(f, ")")); + write!(f, "(")?; + write_list(f, &self.argument_types)?; + write!(f, ")")?; if !self.return_types.is_empty() { - try!(write!(f, " -> ")); - try!(write_list(f, &self.return_types)); + write!(f, " -> ")?; + write_list(f, &self.return_types)?; } Ok(()) } @@ -83,14 +83,14 @@ impl ArgumentType { impl Display for ArgumentType { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - try!(write!(f, "{}", self.value_type)); + write!(f, "{}", self.value_type)?; match self.extension { ArgumentExtension::None => {} - ArgumentExtension::Uext => try!(write!(f, " uext")), - ArgumentExtension::Sext => try!(write!(f, " sext")), + ArgumentExtension::Uext => write!(f, " uext")?, + ArgumentExtension::Sext => write!(f, " sext")?, } if self.inreg { - try!(write!(f, " inreg")); + write!(f, " inreg")?; } Ok(()) } From aa7e349134c0265ac6d0c5059d620277dd46a585 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 24 Feb 2017 11:04:31 -0800 Subject: [PATCH 0535/3084] Add a section about implementation limits. Fix a few other minor issues with the documentation. --- cranelift/docs/Makefile | 2 +- cranelift/docs/langref.rst | 43 +++++++++++++++++++++++-- lib/cretonne/meta/isa/arm32/__init__.py | 2 +- 3 files changed, 43 insertions(+), 4 deletions(-) diff --git a/cranelift/docs/Makefile b/cranelift/docs/Makefile index b2443f68b7..8cec8c36e6 100644 --- a/cranelift/docs/Makefile +++ b/cranelift/docs/Makefile @@ -58,7 +58,7 @@ html: @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." autohtml: html - $(SPHINXABUILD) -z ../lib/cretonne/meta --ignore '*.swp' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html + $(SPHINXABUILD) -z ../lib/cretonne/meta --ignore '.*.sw?' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index d7bffd870a..5e6d999b09 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -80,7 +80,7 @@ into Cretonne :term:`IL` contains multiple assignments to the same variables. Such variables can be presented to Cretonne as :term:`stack slot`\s instead. Stack slots are accessed with the :inst:`stack_store` and :inst:`stack_load` instructions which behave more like variable accesses in a typical programming -language. Cretonne can perform the necessary dataflow analysis to convert stack +language. Cretonne can perform the necessary data-flow analysis to convert stack slots to SSA form. .. _value-types: @@ -459,7 +459,7 @@ accesses may trap, or they may work. Sometimes, operating systems catch alignment traps and emulate the misaligned memory access. On target architectures like x86 that don't check alignment, Cretonne expands -the aligntrap flag into a conditional trap instruction:: +the `aligntrap` flag into a conditional trap instruction:: v5 = load.i32 v1, 4, align(4), aligntrap ; Becomes: @@ -854,6 +854,45 @@ group. Target ISAs may define further instructions in their own instruction groups. +Implementation limits +===================== + +Cretonne's intermediate representation imposes some limits on the size of +functions and the number of entities allowed. If these limits are exceeded, the +implementation will panic. + +Number of instructions in a function + At most :math:`2^{31} - 1`. + +Number of EBBs in a function + At most :math:`2^{31} - 1`. + + Every EBB needs at least a terminator instruction anyway. + +Number of secondary values in a function + At most :math:`2^{31} - 1`. + + Secondary values are any SSA values that are not the first result of an + instruction. + +Other entities declared in the preamble + At most :math:`2^{32} - 1`. + + This covers things like stack slots, jump tables, external functions, and + function signatures, etc. + +Number of arguments to an EBB + At most :math:`2^{16}`. + +Number of arguments to a function + At most :math:`2^{16}`. + + This follows from the limit on arguments to the entry EBB. Note that + Cretonne may add a handful of ABI register arguments as function signatures + are lowered. This is for representing things like the link register, the + incoming frame pointer, and callee-saved registers that are saved in the + prologue. + Glossary ======== diff --git a/lib/cretonne/meta/isa/arm32/__init__.py b/lib/cretonne/meta/isa/arm32/__init__.py index d2de00667a..9e0ae5a7e1 100644 --- a/lib/cretonne/meta/isa/arm32/__init__.py +++ b/lib/cretonne/meta/isa/arm32/__init__.py @@ -1,6 +1,6 @@ """ ARM 32-bit Architecture ----------------------- +----------------------- This target ISA generates code for ARMv7 and ARMv8 CPUs in 32-bit mode (AArch32). We support both ARM and Thumb2 instruction encodings. From cf7a729dc1dd4d2b0e29d6fe1d710af3035de1e9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 24 Feb 2017 12:04:46 -0800 Subject: [PATCH 0536/3084] Add an ArgumentLoc data type. This will be used to amend function signatures with ABI lowering information. --- lib/cretonne/src/ir/mod.rs | 2 +- lib/cretonne/src/ir/valueloc.rs | 34 +++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index f4b2a337b4..c14473e9f5 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -23,7 +23,7 @@ pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable, FuncRef, SigRef}; pub use ir::instructions::{Opcode, InstructionData, VariableArgs}; pub use ir::stackslot::StackSlotData; pub use ir::jumptable::JumpTableData; -pub use ir::valueloc::ValueLoc; +pub use ir::valueloc::{ValueLoc, ArgumentLoc}; pub use ir::dfg::{DataFlowGraph, ValueDef}; pub use ir::layout::{Layout, Cursor}; pub use ir::function::Function; diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/cretonne/src/ir/valueloc.rs index 3526c58190..ea7dea39d9 100644 --- a/lib/cretonne/src/ir/valueloc.rs +++ b/lib/cretonne/src/ir/valueloc.rs @@ -22,3 +22,37 @@ impl Default for ValueLoc { ValueLoc::Unassigned } } + +/// Function argument location. +/// +/// The ABI specifies how arguments are passed to a function, and where return values appear after +/// the call. Just like a `ValueLoc`, function arguments can be passed in registers or on the +/// stack. +/// +/// Function arguments on the stack are accessed differently for the incoming arguments to the +/// current function and the outgoing arguments to a called external function. For this reason, +/// the location of stack arguments is described as an offset into the array of function arguments +/// on the stack. +/// +/// An `ArgumentLoc` can be translated to a `ValueLoc` only when we know if we're talking about an +/// incoming argument or an outgoing argument. +/// +/// - For stack arguments, different `StackSlot` entities are used to represent incoming and +/// outgoing arguments. +/// - For register arguments, there is usually no difference, but if we ever add support for a +/// register-window ISA like SPARC, register arguments would also need to be translated. +#[derive(Copy, Clone, Debug)] +pub enum ArgumentLoc { + /// This argument has not been assigned to a location yet. + Unassigned, + /// Argument is passed in a register. + Reg(RegUnit), + /// Argument is passed on the stack, at the given byte offset into the argument array. + Stack(u32), +} + +impl Default for ArgumentLoc { + fn default() -> Self { + ArgumentLoc::Unassigned + } +} From cf5701b137f8f3cc1a5ecfa27406a222fa955383 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 24 Feb 2017 13:43:04 -0800 Subject: [PATCH 0537/3084] Add ABI annotations to function signatures. Specify the location of arguments as well as the size of stack argument array needed. The ABI annotations are optional, just like the value locations. Remove the Eq implementation for Signature which was only used by a single parser test. --- cranelift/docs/langref.rst | 5 +++ lib/cretonne/src/ir/extfunc.rs | 60 +++++++++++++++++++++++++++++++--- lib/reader/src/parser.rs | 11 +++---- 3 files changed, 65 insertions(+), 11 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 5e6d999b09..23cc225d7f 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -893,6 +893,11 @@ Number of arguments to a function incoming frame pointer, and callee-saved registers that are saved in the prologue. +Size of function call arguments on the stack + At most :math:`2^{32} - 1` bytes. + + This is probably not possible to achieve given the limit on the number of + arguments, except by requiring extremely large offsets for stack arguments. Glossary ======== diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index ae012e7530..5c4d1d1c71 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -5,19 +5,30 @@ //! //! This module declares the data types used to represent external functions and call signatures. +use ir::{Type, FunctionName, SigRef, ArgumentLoc}; +use std::cmp; use std::fmt::{self, Display, Formatter}; -use ir::{Type, FunctionName, SigRef}; /// Function signature. /// /// The function signature describes the types of arguments and return values along with other /// details that are needed to call a function correctly. -#[derive(Clone, PartialEq, Eq, Debug)] +/// +/// A signature can optionally include ISA-specific ABI information which specifies exactly how +/// arguments and return values are passed. +#[derive(Clone, Debug)] pub struct Signature { /// Types of the arguments passed to the function. pub argument_types: Vec, /// Types returned from the function. pub return_types: Vec, + + /// When the signature has been legalized to a specific ISA, this holds the size of the + /// argument array on the stack. Before legalization, this is `None`. + /// + /// This can be computed from the legalized `argument_types` array as the maximum (offset plus + /// byte size) of the `ArgumentLoc::Stack(offset)` argument. + pub argument_bytes: Option, } impl Signature { @@ -26,8 +37,24 @@ impl Signature { Signature { argument_types: Vec::new(), return_types: Vec::new(), + argument_bytes: None, } } + + /// Compute the size of the stack arguments and mark signature as legalized. + /// + /// Even if there are no stack arguments, this will set `argument_types` to `Some(0)` instead + /// of `None`. This indicates that the signature has been legalized. + pub fn compute_argument_bytes(&mut self) { + let bytes = self.argument_types + .iter() + .filter_map(|arg| match arg.location { + ArgumentLoc::Stack(offset) => Some(offset + arg.value_type.bits() as u32 / 8), + _ => None, + }) + .fold(0, cmp::max); + self.argument_bytes = Some(bytes); + } } fn write_list(f: &mut Formatter, args: &Vec) -> fmt::Result { @@ -60,7 +87,7 @@ impl Display for Signature { /// /// This describes the value type being passed to or from a function along with flags that affect /// how the argument is passed. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] +#[derive(Copy, Clone, Debug)] pub struct ArgumentType { /// Type of the argument value. pub value_type: Type, @@ -68,6 +95,10 @@ pub struct ArgumentType { pub extension: ArgumentExtension, /// Place this argument in a register if possible. pub inreg: bool, + + /// ABI-specific location of this argument, or `Unassigned` for arguments that have not yet + /// been legalized. + pub location: ArgumentLoc, } impl ArgumentType { @@ -77,6 +108,7 @@ impl ArgumentType { value_type: vt, extension: ArgumentExtension::None, inreg: false, + location: Default::default(), } } } @@ -92,7 +124,13 @@ impl Display for ArgumentType { if self.inreg { write!(f, " inreg")?; } - Ok(()) + + // This really needs a `&TargetAbi` so we can print register units correctly. + match self.location { + ArgumentLoc::Reg(ru) => write!(f, " [%{}]", ru), + ArgumentLoc::Stack(offset) => write!(f, " [{}]", offset), + ArgumentLoc::Unassigned => Ok(()), + } } } @@ -154,5 +192,19 @@ mod tests { assert_eq!(sig.to_string(), "(i32, i32x4) -> f32"); sig.return_types.push(ArgumentType::new(B8)); assert_eq!(sig.to_string(), "(i32, i32x4) -> f32, b8"); + + // Test the offset computation algorithm. + assert_eq!(sig.argument_bytes, None); + sig.argument_types[1].location = ArgumentLoc::Stack(8); + sig.compute_argument_bytes(); + // An `i32x4` at offset 8 requires a 24-byte argument array. + assert_eq!(sig.argument_bytes, Some(24)); + // Order does not matter. + sig.argument_types[0].location = ArgumentLoc::Stack(24); + sig.compute_argument_bytes(); + assert_eq!(sig.argument_bytes, Some(28)); + + // Writing ABI-annotated signatures. + assert_eq!(sig.to_string(), "(i32 [24], i32x4 [8]) -> f32, b8"); } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 204b78c9e5..47ff73a89c 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1360,7 +1360,7 @@ impl<'a> Parser<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne::ir::{ArgumentType, ArgumentExtension}; + use cretonne::ir::ArgumentExtension; use cretonne::ir::types; use cretonne::ir::entities::AnyEntity; use testfile::{Details, Comment}; @@ -1371,12 +1371,9 @@ mod tests { fn argument_type() { let mut p = Parser::new("i32 sext"); let arg = p.parse_argument_type().unwrap(); - assert_eq!(arg, - ArgumentType { - value_type: types::I32, - extension: ArgumentExtension::Sext, - inreg: false, - }); + assert_eq!(arg.value_type, types::I32); + assert_eq!(arg.extension, ArgumentExtension::Sext); + assert_eq!(arg.inreg, false); let Error { location, message } = p.parse_argument_type().unwrap_err(); assert_eq!(location.line_number, 1); assert_eq!(message, "expected argument type"); From b23f1fb3474c27de48527f4dc8c9e3573164a663 Mon Sep 17 00:00:00 2001 From: rep-nop Date: Sat, 25 Feb 2017 22:12:33 -0500 Subject: [PATCH 0538/3084] Converts all try! macros to ? syntax. Fixes #46 --- cranelift/src/cat.rs | 6 +- cranelift/src/filetest/legalizer.rs | 2 +- cranelift/src/filetest/regalloc.rs | 2 +- cranelift/src/filetest/runone.rs | 14 +- cranelift/src/filetest/subtest.rs | 12 +- cranelift/src/print_cfg.rs | 26 +-- cranelift/src/rsfilecheck.rs | 16 +- cranelift/src/utils.rs | 4 +- lib/cretonne/meta/gen_settings.py | 10 +- lib/cretonne/src/ir/funcname.rs | 4 +- lib/cretonne/src/ir/immediates.rs | 6 +- lib/cretonne/src/ir/instructions.rs | 12 +- lib/cretonne/src/ir/jumptable.rs | 8 +- lib/cretonne/src/settings.rs | 12 +- lib/cretonne/src/verifier.rs | 4 +- lib/cretonne/src/write.rs | 56 +++--- lib/filecheck/src/checker.rs | 16 +- lib/filecheck/src/explain.rs | 18 +- lib/filecheck/src/pattern.rs | 10 +- lib/reader/src/parser.rs | 296 ++++++++++++++-------------- lib/reader/src/sourcemap.rs | 2 +- lib/reader/src/testcommand.rs | 4 +- 22 files changed, 270 insertions(+), 270 deletions(-) diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs index 14871b07c2..443c37dadc 100644 --- a/cranelift/src/cat.rs +++ b/cranelift/src/cat.rs @@ -15,14 +15,14 @@ pub fn run(files: Vec) -> CommandResult { if i != 0 { println!(""); } - try!(cat_one(f)) + cat_one(f)? } Ok(()) } fn cat_one(filename: String) -> CommandResult { - let buffer = try!(read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))); - let items = try!(parse_functions(&buffer).map_err(|e| format!("{}: {}", filename, e))); + let buffer = read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))?; + let items = parse_functions(&buffer).map_err(|e| format!("{}: {}", filename, e))?; for (idx, func) in items.into_iter().enumerate() { if idx != 0 { diff --git a/cranelift/src/filetest/legalizer.rs b/cranelift/src/filetest/legalizer.rs index ea53a703b7..b7c3345436 100644 --- a/cranelift/src/filetest/legalizer.rs +++ b/cranelift/src/filetest/legalizer.rs @@ -39,7 +39,7 @@ impl SubTest for TestLegalizer { legalize_function(&mut func, isa); let mut text = String::new(); - try!(write_function(&mut text, &func, Some(isa)).map_err(|e| e.to_string())); + write_function(&mut text, &func, Some(isa)).map_err(|e| e.to_string())?; run_filecheck(&text, context) } } diff --git a/cranelift/src/filetest/regalloc.rs b/cranelift/src/filetest/regalloc.rs index af9dbb2b23..90796c1b85 100644 --- a/cranelift/src/filetest/regalloc.rs +++ b/cranelift/src/filetest/regalloc.rs @@ -49,7 +49,7 @@ impl SubTest for TestRegalloc { comp_ctx.regalloc(isa); let mut text = String::new(); - try!(write_function(&mut text, &comp_ctx.func, Some(isa)).map_err(|e| e.to_string())); + write_function(&mut text, &comp_ctx.func, Some(isa)).map_err(|e| e.to_string())?; run_filecheck(&text, context) } } diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index d84f0f7041..06e4cfce6b 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -18,14 +18,14 @@ use filetest::subtest::{SubTest, Context, Result}; /// If running this test causes a panic, it will propagate as normal. pub fn run(path: &Path) -> TestResult { let started = time::Instant::now(); - let buffer = try!(read_to_string(path).map_err(|e| e.to_string())); - let testfile = try!(parse_test(&buffer).map_err(|e| e.to_string())); + let buffer = read_to_string(path).map_err(|e| e.to_string())?; + let testfile = parse_test(&buffer).map_err(|e| e.to_string())?; if testfile.functions.is_empty() { return Err("no functions found".to_string()); } // Parse the test commands. - let mut tests = try!(testfile.commands.iter().map(new_subtest).collect::>>()); + let mut tests = testfile.commands.iter().map(new_subtest).collect::>>()?; // Flags to use for those tests that don't need an ISA. // This is the cumulative effect of all the `set` commands in the file. @@ -39,7 +39,7 @@ pub fn run(path: &Path) -> TestResult { tests.sort_by_key(|st| (st.is_mutating(), st.needs_verifier())); // Expand the tests into (test, flags, isa) tuples. - let mut tuples = try!(test_tuples(&tests, &testfile.isa_spec, flags)); + let mut tuples = test_tuples(&tests, &testfile.isa_spec, flags)?; // Isolate the last test in the hope that this is the only mutating test. // If so, we can completely avoid cloning functions. @@ -58,11 +58,11 @@ pub fn run(path: &Path) -> TestResult { }; for tuple in &tuples { - try!(run_one_test(*tuple, Cow::Borrowed(&func), &mut context)); + run_one_test(*tuple, Cow::Borrowed(&func), &mut context)?; } // Run the last test with an owned function which means it won't need to clone it before // mutating. - try!(run_one_test(last_tuple, Cow::Owned(func), &mut context)); + run_one_test(last_tuple, Cow::Owned(func), &mut context)?; } @@ -108,7 +108,7 @@ fn run_one_test<'a>(tuple: (&'a SubTest, &'a Flags, Option<&'a TargetIsa>), // Should we run the verifier before this test? if !context.verified && test.needs_verifier() { - try!(verify_function(&func).map_err(|e| e.to_string())); + verify_function(&func).map_err(|e| e.to_string())?; context.verified = true; } diff --git a/cranelift/src/filetest/subtest.rs b/cranelift/src/filetest/subtest.rs index f78ac7a6b1..8ba3528a6c 100644 --- a/cranelift/src/filetest/subtest.rs +++ b/cranelift/src/filetest/subtest.rs @@ -72,13 +72,13 @@ impl<'a> filecheck::VariableMap for Context<'a> { /// Run filecheck on `text`, using directives extracted from `context`. pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { - let checker = try!(build_filechecker(context)); - if try!(checker.check(&text, context).map_err(|e| format!("filecheck: {}", e))) { + let checker = build_filechecker(context)?; + if checker.check(&text, context).map_err(|e| format!("filecheck: {}", e))? { Ok(()) } else { // Filecheck mismatch. Emit an explanation as output. - let (_, explain) = try!(checker.explain(&text, context) - .map_err(|e| format!("explain: {}", e))); + let (_, explain) = checker.explain(&text, context) + .map_err(|e| format!("explain: {}", e))?; Err(format!("filecheck failed:\n{}{}", checker, explain)) } } @@ -88,10 +88,10 @@ pub fn build_filechecker(context: &Context) -> Result { let mut builder = CheckerBuilder::new(); // Preamble comments apply to all functions. for comment in context.preamble_comments { - try!(builder.directive(comment.text).map_err(|e| format!("filecheck: {}", e))); + builder.directive(comment.text).map_err(|e| format!("filecheck: {}", e))?; } for comment in &context.details.comments { - try!(builder.directive(comment.text).map_err(|e| format!("filecheck: {}", e))); + builder.directive(comment.text).map_err(|e| format!("filecheck: {}", e))?; } let checker = builder.finish(); if checker.is_empty() { diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index ca6c9219f7..d8261dad10 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -19,7 +19,7 @@ pub fn run(files: Vec) -> CommandResult { if i != 0 { println!(""); } - try!(print_cfg(f)) + print_cfg(f)? } Ok(()) } @@ -39,37 +39,37 @@ impl<'a> CFGPrinter<'a> { /// Write the CFG for this function to `w`. pub fn write(&self, w: &mut Write) -> Result { - try!(self.header(w)); - try!(self.ebb_nodes(w)); - try!(self.cfg_connections(w)); + self.header(w)?; + self.ebb_nodes(w)?; + self.cfg_connections(w)?; writeln!(w, "}}") } fn header(&self, w: &mut Write) -> Result { - try!(writeln!(w, "digraph {} {{", self.func.name)); + writeln!(w, "digraph {} {{", self.func.name)?; if let Some(entry) = self.func.layout.entry_block() { - try!(writeln!(w, " {{rank=min; {}}}", entry)); + writeln!(w, " {{rank=min; {}}}", entry)?; } Ok(()) } fn ebb_nodes(&self, w: &mut Write) -> Result { for ebb in &self.func.layout { - try!(write!(w, " {} [shape=record, label=\"{{{}", ebb, ebb)); + write!(w, " {} [shape=record, label=\"{{{}", ebb, ebb)?; // Add all outgoing branch instructions to the label. for inst in self.func.layout.ebb_insts(ebb) { let idata = &self.func.dfg[inst]; match idata.analyze_branch() { BranchInfo::SingleDest(dest, _) => { - try!(write!(w, " | <{}>{} {}", inst, idata.opcode(), dest)) + write!(w, " | <{}>{} {}", inst, idata.opcode(), dest)? } BranchInfo::Table(table) => { - try!(write!(w, " | <{}>{} {}", inst, idata.opcode(), table)) + write!(w, " | <{}>{} {}", inst, idata.opcode(), table)? } BranchInfo::NotABranch => {} } } - try!(writeln!(w, "}}\"]")) + writeln!(w, "}}\"]")? } Ok(()) } @@ -77,7 +77,7 @@ impl<'a> CFGPrinter<'a> { fn cfg_connections(&self, w: &mut Write) -> Result { for ebb in &self.func.layout { for &(parent, inst) in self.cfg.get_predecessors(ebb) { - try!(writeln!(w, " {}:{} -> {}", parent, inst, ebb)); + writeln!(w, " {}:{} -> {}", parent, inst, ebb)?; } } Ok(()) @@ -91,8 +91,8 @@ impl<'a> Display for CFGPrinter<'a> { } fn print_cfg(filename: String) -> CommandResult { - let buffer = try!(read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))); - let items = try!(parse_functions(&buffer).map_err(|e| format!("{}: {}", filename, e))); + let buffer = read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))?; + let items = parse_functions(&buffer).map_err(|e| format!("{}: {}", filename, e))?; for (idx, func) in items.into_iter().enumerate() { if idx != 0 { diff --git a/cranelift/src/rsfilecheck.rs b/cranelift/src/rsfilecheck.rs index 5ff6c596d1..9794e10813 100644 --- a/cranelift/src/rsfilecheck.rs +++ b/cranelift/src/rsfilecheck.rs @@ -7,7 +7,7 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { if files.is_empty() { return Err("No check files".to_string()); } - let checker = try!(read_checkfile(&files[0])); + let checker = read_checkfile(&files[0])?; if checker.is_empty() { return Err(format!("{}: no filecheck directives found", files[0])); } @@ -18,11 +18,11 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { } let mut buffer = String::new(); - try!(io::stdin().read_to_string(&mut buffer).map_err(|e| format!("stdin: {}", e))); + io::stdin().read_to_string(&mut buffer).map_err(|e| format!("stdin: {}", e))?; if verbose { - let (success, explain) = try!(checker.explain(&buffer, NO_VARIABLES) - .map_err(|e| e.to_string())); + let (success, explain) = checker.explain(&buffer, NO_VARIABLES) + .map_err(|e| e.to_string())?; print!("{}", explain); if success { println!("OK"); @@ -30,18 +30,18 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { } else { Err("Check failed".to_string()) } - } else if try!(checker.check(&buffer, NO_VARIABLES).map_err(|e| e.to_string())) { + } else if checker.check(&buffer, NO_VARIABLES).map_err(|e| e.to_string())? { Ok(()) } else { - let (_, explain) = try!(checker.explain(&buffer, NO_VARIABLES).map_err(|e| e.to_string())); + let (_, explain) = checker.explain(&buffer, NO_VARIABLES).map_err(|e| e.to_string())?; print!("{}", explain); Err("Check failed".to_string()) } } fn read_checkfile(filename: &str) -> Result { - let buffer = try!(read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))); + let buffer = read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))?; let mut builder = CheckerBuilder::new(); - try!(builder.text(&buffer).map_err(|e| format!("{}: {}", filename, e))); + builder.text(&buffer).map_err(|e| format!("{}: {}", filename, e))?; Ok(builder.finish()) } diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index f89bc46b43..9cbe20b4bd 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -6,9 +6,9 @@ use std::io::{Result, Read}; /// Read an entire file into a string. pub fn read_to_string>(path: P) -> Result { - let mut file = try!(File::open(path)); + let mut file = File::open(path)?; let mut buffer = String::new(); - try!(file.read_to_string(&mut buffer)); + file.read_to_string(&mut buffer)?; Ok(buffer) } diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index 3f435c8fc3..6429333d28 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -171,13 +171,13 @@ def gen_display(sgrp, fmt): with fmt.indented( 'fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {', '}'): - fmt.line('try!(writeln!(f, "[{}]"));'.format(sgrp.name)) + fmt.line('writeln!(f, "[{}]")?;'.format(sgrp.name)) with fmt.indented('for d in &DESCRIPTORS {', '}'): - fmt.line('try!(write!(f, "{} = ", d.name));') + fmt.line('write!(f, "{} = ", d.name)?;') fmt.line( - 'try!(TEMPLATE.format_toml_value(d.detail,' + - 'self.bytes[d.offset as usize], f));') - fmt.line('try!(writeln!(f, ""));') + 'TEMPLATE.format_toml_value(d.detail,' + + 'self.bytes[d.offset as usize], f)?;') + fmt.line('writeln!(f, "")?;') fmt.line('Ok(())') diff --git a/lib/cretonne/src/ir/funcname.rs b/lib/cretonne/src/ir/funcname.rs index d84b25ac2b..0074c97502 100644 --- a/lib/cretonne/src/ir/funcname.rs +++ b/lib/cretonne/src/ir/funcname.rs @@ -42,9 +42,9 @@ fn needs_quotes(name: &str) -> bool { impl fmt::Display for FunctionName { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { if needs_quotes(&self.0) { - try!(f.write_char('"')); + f.write_char('"')?; for c in self.0.chars().flat_map(char::escape_default) { - try!(f.write_char(c)); + f.write_char(c)?; } f.write_char('"') } else { diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index bc44e6c022..5673af1163 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -49,10 +49,10 @@ impl Display for Imm64 { // 0xffff_ffff_fff8_4400 // let mut pos = (64 - x.leading_zeros() - 1) & 0xf0; - try!(write!(f, "0x{:04x}", (x >> pos) & 0xffff)); + write!(f, "0x{:04x}", (x >> pos) & 0xffff)?; while pos > 0 { pos -= 16; - try!(write!(f, "_{:04x}", (x >> pos) & 0xffff)); + write!(f, "_{:04x}", (x >> pos) & 0xffff)?; } Ok(()) } @@ -178,7 +178,7 @@ fn format_float(bits: u64, w: u8, t: u8, f: &mut Formatter) -> fmt::Result { // All formats share the leading sign. if sign_bit != 0 { - try!(write!(f, "-")); + write!(f, "-")?; } if e_bits == 0 { diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 839219fc2a..c4ac3b0c20 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -265,9 +265,9 @@ impl Display for VariableArgs { fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { for (i, val) in self.0.iter().enumerate() { if i == 0 { - try!(write!(fmt, "{}", val)); + write!(fmt, "{}", val)?; } else { - try!(write!(fmt, ", {}", val)); + write!(fmt, ", {}", val)?; } } Ok(()) @@ -289,9 +289,9 @@ pub struct UnaryImmVectorData { impl Display for UnaryImmVectorData { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - try!(write!(f, "#")); + write!(f, "#")?; for b in &self.imm { - try!(write!(f, "{:02x}", b)); + write!(f, "{:02x}", b)?; } Ok(()) } @@ -356,9 +356,9 @@ impl BranchData { impl Display for BranchData { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - try!(write!(f, "{}, {}", self.arg, self.destination)); + write!(f, "{}, {}", self.arg, self.destination)?; if !self.varargs.is_empty() { - try!(write!(f, "({})", self.varargs)); + write!(f, "({})", self.varargs)?; } Ok(()) } diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index 7103485094..29da4456bc 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -97,14 +97,14 @@ impl<'a> Iterator for Entries<'a> { impl Display for JumpTableData { fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { match self.table.first().and_then(|e| e.expand()) { - None => try!(write!(fmt, "jump_table 0")), - Some(first) => try!(write!(fmt, "jump_table {}", first)), + None => write!(fmt, "jump_table 0")?, + Some(first) => write!(fmt, "jump_table {}", first)?, } for dest in self.table.iter().skip(1).map(|e| e.expand()) { match dest { - None => try!(write!(fmt, ", 0")), - Some(ebb) => try!(write!(fmt, ", {}", ebb)), + None => write!(fmt, ", 0")?, + Some(ebb) => write!(fmt, ", {}", ebb)?, } } Ok(()) diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 30ecff68e4..0d81c38dce 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -103,7 +103,7 @@ fn parse_enum_value(value: &str, choices: &[&str]) -> Result { impl Configurable for Builder { fn set_bool(&mut self, name: &str, value: bool) -> Result<()> { use self::detail::Detail; - let (offset, detail) = try!(self.lookup(name)); + let (offset, detail) = self.lookup(name)?; if let Detail::Bool { bit } = detail { self.set_bit(offset, bit, value); Ok(()) @@ -114,17 +114,17 @@ impl Configurable for Builder { fn set(&mut self, name: &str, value: &str) -> Result<()> { use self::detail::Detail; - let (offset, detail) = try!(self.lookup(name)); + let (offset, detail) = self.lookup(name)?; match detail { Detail::Bool { bit } => { - self.set_bit(offset, bit, try!(parse_bool_value(value))); + self.set_bit(offset, bit, parse_bool_value(value))?; } Detail::Num => { - self.bytes[offset] = try!(value.parse().map_err(|_| Error::BadValue)); + self.bytes[offset] = value.parse().map_err(|_| Error::BadValue)?; } Detail::Enum { last, enumerators } => { - self.bytes[offset] = try!(parse_enum_value(value, - self.template.enums(last, enumerators))); + self.bytes[offset] = parse_enum_value(value, + self.template.enums(last, enumerators))?; } } Ok(()) diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index ea5941006e..2610475759 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -159,8 +159,8 @@ impl<'a> Verifier<'a> { pub fn run(&self) -> Result<()> { for ebb in self.func.layout.ebbs() { for inst in self.func.layout.ebb_insts(ebb) { - try!(self.ebb_integrity(ebb, inst)); - try!(self.instruction_integrity(inst)); + self.ebb_integrity(ebb, inst)?; + self.instruction_integrity(inst)?; } } Ok(()) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 933ba8822e..eca9909961 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -12,14 +12,14 @@ use std::result; /// Write `func` to `w` as equivalent text. /// Use `isa` to emit ISA-dependent annotations. pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) -> Result { - try!(write_spec(w, func)); - try!(writeln!(w, " {{")); - let mut any = try!(write_preamble(w, func)); + write_spec(w, func)?; + writeln!(w, " {{")?; + let mut any = write_preamble(w, func)?; for ebb in &func.layout { if any { - try!(writeln!(w, "")); + writeln!(w, "")?; } - try!(write_ebb(w, func, isa, ebb)); + write_ebb(w, func, isa, ebb)?; any = true; } writeln!(w, "}}") @@ -40,24 +40,24 @@ fn write_preamble(w: &mut Write, func: &Function) -> result::Result for ss in func.stack_slots.keys() { any = true; - try!(writeln!(w, " {} = {}", ss, func.stack_slots[ss])); + writeln!(w, " {} = {}", ss, func.stack_slots[ss])?; } // Write out all signatures before functions since function declarations can refer to // signatures. for sig in func.dfg.signatures.keys() { any = true; - try!(writeln!(w, " {} = signature{}", sig, func.dfg.signatures[sig])); + writeln!(w, " {} = signature{}", sig, func.dfg.signatures[sig])?; } for fnref in func.dfg.ext_funcs.keys() { any = true; - try!(writeln!(w, " {} = {}", fnref, func.dfg.ext_funcs[fnref])); + writeln!(w, " {} = {}", fnref, func.dfg.ext_funcs[fnref])?; } for jt in func.jump_tables.keys() { any = true; - try!(writeln!(w, " {} = {}", jt, func.jump_tables[jt])); + writeln!(w, " {} = {}", jt, func.jump_tables[jt])?; } Ok(any) @@ -83,29 +83,29 @@ pub fn write_ebb_header(w: &mut Write, func: &Function, ebb: Ebb) -> Result { // If we're writing encoding annotations, shift by 20. if !func.encodings.is_empty() { - try!(write!(w, " ")); + write!(w, " ")?; } let mut args = func.dfg.ebb_args(ebb); match args.next() { None => return writeln!(w, "{}:", ebb), Some(arg) => { - try!(write!(w, "{}(", ebb)); - try!(write_arg(w, func, arg)); + write!(w, "{}(", ebb)?; + write_arg(w, func, arg)?; } } // Remaining arguments. for arg in args { - try!(write!(w, ", ")); - try!(write_arg(w, func, arg)); + write!(w, ", ")?; + write_arg(w, func, arg)?; } writeln!(w, "):") } pub fn write_ebb(w: &mut Write, func: &Function, isa: Option<&TargetIsa>, ebb: Ebb) -> Result { - try!(write_ebb_header(w, func, ebb)); + write_ebb_header(w, func, ebb)?; for inst in func.layout.ebb_insts(ebb) { - try!(write_instruction(w, func, isa, inst)); + write_instruction(w, func, isa, inst)?; } Ok(()) } @@ -151,7 +151,7 @@ fn write_value_aliases(w: &mut Write, func: &Function, inst: Inst, indent: usize for &arg in func.dfg[inst].arguments().iter().flat_map(|x| x.iter()) { let resolved = func.dfg.resolve_aliases(arg); if resolved != arg { - try!(writeln!(w, "{1:0$}{2} -> {3}", indent, "", arg, resolved)); + writeln!(w, "{1:0$}{2} -> {3}", indent, "", arg, resolved)?; } } Ok(()) @@ -166,13 +166,13 @@ fn write_instruction(w: &mut Write, let indent = if func.encodings.is_empty() { 4 } else { 24 }; // Value aliases come out on lines before the instruction using them. - try!(write_value_aliases(w, func, inst, indent)); + write_value_aliases(w, func, inst, indent)?; // Write out encoding info. if let Some(enc) = func.encodings.get(inst).cloned() { let mut s = String::with_capacity(16); if let Some(isa) = isa { - try!(write!(s, "[{}", isa.display_enc(enc))); + write!(s, "[{}", isa.display_enc(enc))?; // Write value locations, if we have them. if !func.locations.is_empty() { let regs = isa.register_info(); @@ -184,15 +184,15 @@ fn write_instruction(w: &mut Write, } } } - try!(write!(s, "]")); + write!(s, "]")?; } else { - try!(write!(s, "[{}]", enc)); + write!(s, "[{}]", enc)?; } // Align instruction following ISA annotation to col 24. - try!(write!(w, "{:23} ", s)); + write!(w, "{:23} ", s)?; } else { // No annotations, simply indent. - try!(write!(w, "{1:0$}", indent, "")); + write!(w, "{1:0$}", indent, "")?; } // Write out the result values, if any. @@ -200,21 +200,21 @@ fn write_instruction(w: &mut Write, for r in func.dfg.inst_results(inst) { if !has_results { has_results = true; - try!(write!(w, "{}", r)); + write!(w, "{}", r)?; } else { - try!(write!(w, ", {}", r)); + write!(w, ", {}", r)?; } } if has_results { - try!(write!(w, " = ")); + write!(w, " = ")?; } // Then the opcode, possibly with a '.type' suffix. let opcode = func.dfg[inst].opcode(); match type_suffix(func, inst) { - Some(suf) => try!(write!(w, "{}.{}", opcode, suf)), - None => try!(write!(w, "{}", opcode)), + Some(suf) => write!(w, "{}.{}", opcode, suf)?, + None => write!(w, "{}", opcode)?, } // Then the operands, depending on format. diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs index a2af0815e7..2474742fb1 100644 --- a/lib/filecheck/src/checker.rs +++ b/lib/filecheck/src/checker.rs @@ -38,7 +38,7 @@ impl Directive { } // All other commands are followed by a pattern. - let pat = try!(rest.parse()); + let pat = rest.parse()?; match cmd { "check" => Ok(Directive::Check(pat)), @@ -99,7 +99,7 @@ impl CheckerBuilder { pub fn directive(&mut self, l: &str) -> Result { match self.linerx.captures(l) { Some(caps) => { - self.directives.push(try!(Directive::new(caps))); + self.directives.push(Directive::new(caps)?); Ok(true) } None => Ok(false), @@ -112,7 +112,7 @@ impl CheckerBuilder { /// This method can be used to parse a whole test file containing multiple directives. pub fn text(&mut self, t: &str) -> Result<&mut Self> { for caps in self.linerx.captures_iter(t) { - self.directives.push(try!(Directive::new(caps))); + self.directives.push(Directive::new(caps)?); } Ok(self) } @@ -154,7 +154,7 @@ impl Checker { /// Explain how directives are matched against the input text. pub fn explain(&self, text: &str, vars: &VariableMap) -> Result<(bool, String)> { let mut expl = Explainer::new(text); - let success = try!(self.run(text, vars, &mut expl)); + let success = self.run(text, vars, &mut expl)?; expl.finish(); Ok((success, expl.to_string())) } @@ -178,7 +178,7 @@ impl Checker { // The `not:` directives test the same range as `unordered:` directives. In // particular, if they refer to defined variables, their range is restricted to // the text following the match that defined the variable. - nots.push((dct_idx, state.unordered_begin(pat), try!(pat.resolve(&state)))); + nots.push((dct_idx, state.unordered_begin(pat), pat.resolve(&state)?)); continue; } Directive::Regex(ref var, ref rx) => { @@ -192,7 +192,7 @@ impl Checker { }; // Check if `pat` matches in `range`. state.recorder.directive(dct_idx); - if let Some((match_begin, match_end)) = try!(state.match_positive(pat, range)) { + if let Some((match_begin, match_end)) = state.match_positive(pat, range)? { if let &Directive::Unordered(_) = dct { // This was an unordered unordered match. // Keep track of the largest matched position, but leave `last_ordered` alone. @@ -318,7 +318,7 @@ impl<'a> State<'a> { // Search for `pat` in `range`, return the range matched. // After a positive match, update variable definitions, if any. fn match_positive(&mut self, pat: &Pattern, range: MatchRange) -> Result> { - let rx = try!(pat.resolve(self)); + let rx = pat.resolve(self)?; let txt = &self.text[range.0..range.1]; let defs = pat.defs(); let matched_range = if defs.is_empty() { @@ -382,7 +382,7 @@ impl Display for Directive { impl Display for Checker { fn fmt(&self, f: &mut Formatter) -> fmt::Result { for (idx, dir) in self.directives.iter().enumerate() { - try!(write!(f, "#{} {}", idx, dir)); + write!(f, "#{} {}", idx, dir)?; } Ok(()) } diff --git a/lib/filecheck/src/explain.rs b/lib/filecheck/src/explain.rs index 53dd4003a7..825f162079 100644 --- a/lib/filecheck/src/explain.rs +++ b/lib/filecheck/src/explain.rs @@ -89,34 +89,34 @@ impl<'a> Display for Explainer<'a> { .map(|d| nextln + d + 1) .unwrap_or(self.text.len()); assert!(newln > nextln); - try!(writeln!(f, "> {}", &self.text[nextln..newln - 1])); + writeln!(f, "> {}", &self.text[nextln..newln - 1])?; curln = nextln; nextln = newln; } // Emit ~~~ under the part of the match in curln. if m.is_match { - try!(write!(f, " ")); + write!(f, " ")?; let mend = min(m.range.1, nextln - 1); for pos in curln..mend { - try!(if pos < m.range.0 { + if pos < m.range.0 { write!(f, " ") } else if pos == m.range.0 { write!(f, "^") } else { write!(f, "~") - }); + }?; } - try!(writeln!(f, "")); + writeln!(f, "")?; } // Emit the match message itself. - try!(writeln!(f, + writeln!(f, "{} #{}{}: {}", if m.is_match { "Matched" } else { "Missed" }, m.directive, if m.is_not { " not" } else { "" }, - m.regex)); + m.regex)?; // Emit any variable definitions. if let Ok(found) = self.vardefs.binary_search_by_key(&m.directive, |v| v.directive) { @@ -128,14 +128,14 @@ impl<'a> Display for Explainer<'a> { if d.directive != m.directive { break; } - try!(writeln!(f, "Define {}={}", d.varname, d.value)); + writeln!(f, "Define {}={}", d.varname, d.value)?; } } } // Emit trailing lines. for line in self.text[nextln..].lines() { - try!(writeln!(f, "> {}", line)); + writeln!(f, "> {}", line)?; } Ok(()) } diff --git a/lib/filecheck/src/pattern.rs b/lib/filecheck/src/pattern.rs index 030758d41e..9f54a80cf6 100644 --- a/lib/filecheck/src/pattern.rs +++ b/lib/filecheck/src/pattern.rs @@ -147,7 +147,7 @@ impl Pattern { let def = if varname.is_empty() { None } else { - Some(try!(self.add_def(&varname))) + Some(self.add_def(&varname)?) }; // Match `$(var=$PAT)`. @@ -270,7 +270,7 @@ impl FromStr for Pattern { let mut pat = Pattern::new(); let mut pos = 0; while pos < s.len() { - let (part, len) = try!(pat.parse_part(&s[pos..])); + let (part, len) = pat.parse_part(&s[pos..])?; if let Some(v) = part.ref_var() { if pat.defines_var(v) { return Err(Error::Backref(format!("unsupported back-reference to '${}' \ @@ -353,7 +353,7 @@ impl Pattern { } } - Ok(try!(RegexBuilder::new(&out).multi_line(true).compile())) + Ok(RegexBuilder::new(&out).multi_line(true).compile()?) } } @@ -361,7 +361,7 @@ impl Display for Pattern { fn fmt(&self, f: &mut Formatter) -> fmt::Result { for part in &self.parts { use self::Part::*; - try!(match *part { + match *part { Text(ref txt) if txt == "" => write!(f, "$()"), Text(ref txt) if txt == "$" => write!(f, "$$"), Text(ref txt) => write!(f, "{}", txt), @@ -374,7 +374,7 @@ impl Display for Pattern { write!(f, "$({}={})", defvar, litrx) } DefVar { def, ref var } => write!(f, "$({}=${})", self.defs[def], var), - }); + }?; } Ok(()) } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 47ff73a89c..e9a5a26a02 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -42,9 +42,9 @@ pub fn parse_test<'a>(text: &'a str) -> Result> { parser.gather_comments(AnyEntity::Function); Ok(TestFile { commands: parser.parse_test_commands(), - isa_spec: try!(parser.parse_isa_specs()), + isa_spec: parser.parse_isa_specs()?, preamble_comments: parser.take_comments(), - functions: try!(parser.parse_function_list()), + functions: parser.parse_function_list()?, }) } @@ -155,7 +155,7 @@ impl Context { InstructionData::BinaryImmRev { ref mut arg, .. } | InstructionData::ExtractLane { ref mut arg, .. } | InstructionData::BranchTable { ref mut arg, .. } => { - try!(self.map.rewrite_value(arg, loc)); + self.map.rewrite_value(arg, loc)?; } InstructionData::Binary { ref mut args, .. } | @@ -163,44 +163,44 @@ impl Context { InstructionData::InsertLane { ref mut args, .. } | InstructionData::IntCompare { ref mut args, .. } | InstructionData::FloatCompare { ref mut args, .. } => { - try!(self.map.rewrite_values(args, loc)); + self.map.rewrite_values(args, loc)?; } InstructionData::Ternary { ref mut args, .. } => { - try!(self.map.rewrite_values(args, loc)); + self.map.rewrite_values(args, loc)?; } InstructionData::TernaryOverflow { ref mut data, .. } => { - try!(self.map.rewrite_values(&mut data.args, loc)); + self.map.rewrite_values(&mut data.args, loc)?; } InstructionData::Jump { ref mut data, .. } => { - try!(self.map.rewrite_ebb(&mut data.destination, loc)); - try!(self.map.rewrite_values(&mut data.varargs, loc)); + self.map.rewrite_ebb(&mut data.destination, loc)?; + self.map.rewrite_values(&mut data.varargs, loc)?; } InstructionData::Branch { ref mut data, .. } => { - try!(self.map.rewrite_value(&mut data.arg, loc)); - try!(self.map.rewrite_ebb(&mut data.destination, loc)); - try!(self.map.rewrite_values(&mut data.varargs, loc)); + self.map.rewrite_value(&mut data.arg, loc)?; + self.map.rewrite_ebb(&mut data.destination, loc)?; + self.map.rewrite_values(&mut data.varargs, loc)?; } InstructionData::Call { ref mut data, .. } => { - try!(self.map.rewrite_values(&mut data.varargs, loc)); + self.map.rewrite_values(&mut data.varargs, loc)?; } InstructionData::IndirectCall { ref mut data, .. } => { - try!(self.map.rewrite_value(&mut data.arg, loc)); - try!(self.map.rewrite_values(&mut data.varargs, loc)); + self.map.rewrite_value(&mut data.arg, loc)?; + self.map.rewrite_values(&mut data.varargs, loc)?; } InstructionData::Return { ref mut data, .. } => { - try!(self.map.rewrite_values(&mut data.varargs, loc)); + self.map.rewrite_values(&mut data.varargs, loc)?; } InstructionData::ReturnReg { ref mut data, .. } => { - try!(self.map.rewrite_value(&mut data.arg, loc)); - try!(self.map.rewrite_values(&mut data.varargs, loc)); + self.map.rewrite_value(&mut data.arg, loc)?; + self.map.rewrite_values(&mut data.varargs, loc)?; } } } @@ -211,7 +211,7 @@ impl Context { let loc = jt.into(); for ebb_ref in self.function.jump_tables[jt].as_mut_slice() { if let Some(mut ebb) = ebb_ref.expand() { - try!(self.map.rewrite_ebb(&mut ebb, loc)); + self.map.rewrite_ebb(&mut ebb, loc)?; // Convert back to a packed option. *ebb_ref = ebb.into(); } @@ -480,9 +480,9 @@ impl<'a> Parser<'a> { match command { "set" => { last_set_loc = Some(self.loc); - try!(isaspec::parse_options(self.consume_line().trim().split_whitespace(), + isaspec::parse_options(self.consume_line().trim().split_whitespace(), &mut flag_builder, - &self.loc)); + &self.loc)?; } "isa" => { last_set_loc = None; @@ -501,7 +501,7 @@ impl<'a> Parser<'a> { Some(b) => b, }; // Apply the ISA-specific settings to `isa_builder`. - try!(isaspec::parse_options(words, &mut isa_builder, &self.loc)); + isaspec::parse_options(words, &mut isa_builder, &self.loc)?; // Construct a trait object with the aggregrate settings. isas.push(isa_builder.finish(settings::Flags::new(&flag_builder))); @@ -526,7 +526,7 @@ impl<'a> Parser<'a> { pub fn parse_function_list(&mut self) -> Result)>> { let mut list = Vec::new(); while self.token().is_some() { - list.push(try!(self.parse_function())); + list.push(self.parse_function()?); } Ok(list) } @@ -542,17 +542,17 @@ impl<'a> Parser<'a> { self.comments.clear(); self.gather_comments(AnyEntity::Function); - let (location, name, sig) = try!(self.parse_function_spec()); + let (location, name, sig) = self.parse_function_spec()?; let mut ctx = Context::new(Function::with_name_signature(name, sig)); // function ::= function-spec * "{" preamble function-body "}" - try!(self.match_token(Token::LBrace, "expected '{' before function body")); + self.match_token(Token::LBrace, "expected '{' before function body")?; // function ::= function-spec "{" * preamble function-body "}" - try!(self.parse_preamble(&mut ctx)); + self.parse_preamble(&mut ctx)?; // function ::= function-spec "{" preamble * function-body "}" - try!(self.parse_function_body(&mut ctx)); + self.parse_function_body(&mut ctx)?; // function ::= function-spec "{" preamble function-body * "}" - try!(self.match_token(Token::RBrace, "expected '}' after function body")); + self.match_token(Token::RBrace, "expected '}' after function body")?; // Collect any comments following the end of the function, then stop gathering comments. self.gather_comments(AnyEntity::Function); @@ -561,7 +561,7 @@ impl<'a> Parser<'a> { // Rewrite references to values and EBBs after parsing everything to allow forward // references. - try!(ctx.rewrite_references()); + ctx.rewrite_references()?; let details = Details { location: location, @@ -577,14 +577,14 @@ impl<'a> Parser<'a> { // function-spec ::= * "function" name signature // fn parse_function_spec(&mut self) -> Result<(Location, FunctionName, Signature)> { - try!(self.match_identifier("function", "expected 'function'")); + self.match_identifier("function", "expected 'function'")?; let location = self.loc; // function-spec ::= "function" * name signature - let name = try!(self.parse_function_name()); + let name = self.parse_function_name()?; // function-spec ::= "function" name * signature - let sig = try!(self.parse_signature()); + let sig = self.parse_signature()?; Ok((location, name, sig)) } @@ -610,14 +610,14 @@ impl<'a> Parser<'a> { fn parse_signature(&mut self) -> Result { let mut sig = Signature::new(); - try!(self.match_token(Token::LPar, "expected function signature: ( args... )")); + self.match_token(Token::LPar, "expected function signature: ( args... )")?; // signature ::= "(" * [arglist] ")" ["->" retlist] [call_conv] if self.token() != Some(Token::RPar) { - sig.argument_types = try!(self.parse_argument_list()); + sig.argument_types = self.parse_argument_list()?; } - try!(self.match_token(Token::RPar, "expected ')' after function arguments")); + self.match_token(Token::RPar, "expected ')' after function arguments")?; if self.optional(Token::Arrow) { - sig.return_types = try!(self.parse_argument_list()); + sig.return_types = self.parse_argument_list()?; } // TBD: calling convention. @@ -633,12 +633,12 @@ impl<'a> Parser<'a> { let mut list = Vec::new(); // arglist ::= * arg { "," arg } - list.push(try!(self.parse_argument_type())); + list.push(self.parse_argument_type()?); // arglist ::= arg * { "," arg } while self.optional(Token::Comma) { // arglist ::= arg { "," * arg } - list.push(try!(self.parse_argument_type())); + list.push(self.parse_argument_type()?); } Ok(list) @@ -647,7 +647,7 @@ impl<'a> Parser<'a> { // Parse a single argument type with flags. fn parse_argument_type(&mut self) -> Result { // arg ::= * type { flag } - let mut arg = ArgumentType::new(try!(self.match_type("expected argument type"))); + let mut arg = ArgumentType::new(self.match_type("expected argument type")?); // arg ::= type * { flag } while let Some(Token::Identifier(s)) = self.token() { @@ -674,7 +674,7 @@ impl<'a> Parser<'a> { // The parsed decls are added to `ctx` rather than returned. fn parse_preamble(&mut self, ctx: &mut Context) -> Result<()> { loop { - try!(match self.token() { + match self.token() { Some(Token::StackSlot(..)) => { self.gather_comments(ctx.function.stack_slots.next_key()); self.parse_stack_slot_decl() @@ -697,7 +697,7 @@ impl<'a> Parser<'a> { } // More to come.. _ => return Ok(()), - }); + }?; } } @@ -705,12 +705,12 @@ impl<'a> Parser<'a> { // // stack-slot-decl ::= * StackSlot(ss) "=" "stack_slot" Bytes {"," stack-slot-flag} fn parse_stack_slot_decl(&mut self) -> Result<(u32, StackSlotData)> { - let number = try!(self.match_ss("expected stack slot number: ss«n»")); - try!(self.match_token(Token::Equal, "expected '=' in stack_slot decl")); - try!(self.match_identifier("stack_slot", "expected 'stack_slot'")); + let number = self.match_ss("expected stack slot number: ss«n»")?; + self.match_token(Token::Equal, "expected '=' in stack_slot decl")?; + self.match_identifier("stack_slot", "expected 'stack_slot'")?; // stack-slot-decl ::= StackSlot(ss) "=" "stack_slot" * Bytes {"," stack-slot-flag} - let bytes: i64 = try!(self.match_imm64("expected byte-size in stack_slot decl")).into(); + let bytes: i64 = self.match_imm64("expected byte-size in stack_slot decl")?.into(); if bytes < 0 { return err!(self.loc, "negative stack slot size"); } @@ -728,10 +728,10 @@ impl<'a> Parser<'a> { // signature-decl ::= SigRef(sigref) "=" "signature" signature // fn parse_signature_decl(&mut self) -> Result<(u32, Signature)> { - let number = try!(self.match_sig("expected signature number: sig«n»")); - try!(self.match_token(Token::Equal, "expected '=' in signature decl")); - try!(self.match_identifier("signature", "expected 'signature'")); - let data = try!(self.parse_signature()); + let number = self.match_sig("expected signature number: sig«n»")?; + self.match_token(Token::Equal, "expected '=' in signature decl")?; + self.match_identifier("signature", "expected 'signature'")?; + let data = self.parse_signature()?; Ok((number, data)) } @@ -746,12 +746,12 @@ impl<'a> Parser<'a> { // signature which must be declared first. // fn parse_function_decl(&mut self, ctx: &mut Context) -> Result<(u32, ExtFuncData)> { - let number = try!(self.match_fn("expected function number: fn«n»")); - try!(self.match_token(Token::Equal, "expected '=' in function decl")); + let number = self.match_fn("expected function number: fn«n»")?; + self.match_token(Token::Equal, "expected '=' in function decl")?; let data = match self.token() { Some(Token::Identifier("function")) => { - let (loc, name, sig) = try!(self.parse_function_spec()); + let (loc, name, sig) = self.parse_function_spec()?; let sigref = ctx.function.dfg.signatures.push(sig); ctx.map.def_entity(sigref.into(), &loc).expect("duplicate SigRef entities created"); ExtFuncData { @@ -760,9 +760,9 @@ impl<'a> Parser<'a> { } } Some(Token::SigRef(sig_src)) => { - let sig = try!(ctx.get_sig(sig_src, &self.loc)); + let sig = ctx.get_sig(sig_src, &self.loc)?; self.consume(); - let name = try!(self.parse_function_name()); + let name = self.parse_function_name()?; ExtFuncData { name: name, signature: sig, @@ -777,15 +777,15 @@ impl<'a> Parser<'a> { // // jump-table-decl ::= * JumpTable(jt) "=" "jump_table" jt-entry {"," jt-entry} fn parse_jump_table_decl(&mut self) -> Result<(u32, JumpTableData)> { - let number = try!(self.match_jt()); - try!(self.match_token(Token::Equal, "expected '=' in jump_table decl")); - try!(self.match_identifier("jump_table", "expected 'jump_table'")); + let number = self.match_jt()?; + self.match_token(Token::Equal, "expected '=' in jump_table decl")?; + self.match_identifier("jump_table", "expected 'jump_table'")?; let mut data = JumpTableData::new(); // jump-table-decl ::= JumpTable(jt) "=" "jump_table" * jt-entry {"," jt-entry} for idx in 0usize.. { - if let Some(dest) = try!(self.parse_jump_table_entry()) { + if let Some(dest) = self.parse_jump_table_entry()? { data.set_entry(idx, dest); } if !self.optional(Token::Comma) { @@ -821,7 +821,7 @@ impl<'a> Parser<'a> { // fn parse_function_body(&mut self, ctx: &mut Context) -> Result<()> { while self.token() != Some(Token::RBrace) { - try!(self.parse_extended_basic_block(ctx)); + self.parse_extended_basic_block(ctx)?; } Ok(()) } @@ -832,14 +832,14 @@ impl<'a> Parser<'a> { // ebb-header ::= Ebb(ebb) [ebb-args] ":" // fn parse_extended_basic_block(&mut self, ctx: &mut Context) -> Result<()> { - let ebb_num = try!(self.match_ebb("expected EBB header")); - let ebb = try!(ctx.add_ebb(ebb_num, &self.loc)); + let ebb_num = self.match_ebb("expected EBB header")?; + let ebb = ctx.add_ebb(ebb_num, &self.loc)?; self.gather_comments(ebb); if !self.optional(Token::Colon) { // ebb-header ::= Ebb(ebb) [ * ebb-args ] ":" - try!(self.parse_ebb_args(ctx, ebb)); - try!(self.match_token(Token::Colon, "expected ':' after EBB arguments")); + self.parse_ebb_args(ctx, ebb)?; + self.match_token(Token::Colon, "expected ':' after EBB arguments")?; } // extended-basic-block ::= ebb-header * { instruction } @@ -848,7 +848,7 @@ impl<'a> Parser<'a> { Some(Token::Identifier(_)) => true, _ => false, } { - try!(self.parse_instruction(ctx, ebb)); + self.parse_instruction(ctx, ebb)?; } Ok(()) @@ -860,19 +860,19 @@ impl<'a> Parser<'a> { // ebb-args ::= * "(" ebb-arg { "," ebb-arg } ")" fn parse_ebb_args(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { // ebb-args ::= * "(" ebb-arg { "," ebb-arg } ")" - try!(self.match_token(Token::LPar, "expected '(' before EBB arguments")); + self.match_token(Token::LPar, "expected '(' before EBB arguments")?; // ebb-args ::= "(" * ebb-arg { "," ebb-arg } ")" - try!(self.parse_ebb_arg(ctx, ebb)); + self.parse_ebb_arg(ctx, ebb)?; // ebb-args ::= "(" ebb-arg * { "," ebb-arg } ")" while self.optional(Token::Comma) { // ebb-args ::= "(" ebb-arg { "," * ebb-arg } ")" - try!(self.parse_ebb_arg(ctx, ebb)); + self.parse_ebb_arg(ctx, ebb)?; } // ebb-args ::= "(" ebb-arg { "," ebb-arg } * ")" - try!(self.match_token(Token::RPar, "expected ')' after EBB arguments")); + self.match_token(Token::RPar, "expected ')' after EBB arguments")?; Ok(()) } @@ -883,12 +883,12 @@ impl<'a> Parser<'a> { // fn parse_ebb_arg(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { // ebb-arg ::= * Value(vx) ":" Type(t) - let vx = try!(self.match_value("EBB argument must be a value")); + let vx = self.match_value("EBB argument must be a value")?; let vx_location = self.loc; // ebb-arg ::= Value(vx) * ":" Type(t) - try!(self.match_token(Token::Colon, "expected ':' after EBB argument")); + self.match_token(Token::Colon, "expected ':' after EBB argument")?; // ebb-arg ::= Value(vx) ":" * Type(t) - let t = try!(self.match_type("expected EBB argument type")); + let t = self.match_type("expected EBB argument type")?; // Allocate the EBB argument and add the mapping. let value = ctx.function.dfg.append_ebb_arg(ebb, t); ctx.map.def_value(vx, value, &vx_location) @@ -915,10 +915,10 @@ impl<'a> Parser<'a> { // inst-results ::= Value(v) * { "," Value(vx) } while self.optional(Token::Comma) { // inst-results ::= Value(v) { "," * Value(vx) } - results.push(try!(self.match_value("expected result value"))); + results.push(self.match_value("expected result value")?); } - try!(self.match_token(Token::Equal, "expected '=' before opcode")); + self.match_token(Token::Equal, "expected '=' before opcode")?; } // instruction ::= [inst-results "="] * Opcode(opc) ["." Type] ... @@ -936,20 +936,20 @@ impl<'a> Parser<'a> { // Look for a controlling type variable annotation. // instruction ::= [inst-results "="] Opcode(opc) * ["." Type] ... let explicit_ctrl_type = if self.optional(Token::Dot) { - Some(try!(self.match_type("expected type after 'opcode.'"))) + Some(self.match_type("expected type after 'opcode.'")?) } else { None }; // instruction ::= [inst-results "="] Opcode(opc) ["." Type] * ... - let inst_data = try!(self.parse_inst_operands(ctx, opcode)); + let inst_data = self.parse_inst_operands(ctx, opcode)?; // We're done parsing the instruction now. // // We still need to check that the number of result values in the source matches the opcode // or function call signature. We also need to create values with the right type for all // the instruction results. - let ctrl_typevar = try!(self.infer_typevar(ctx, opcode, explicit_ctrl_type, &inst_data)); + let ctrl_typevar = self.infer_typevar(ctx, opcode, explicit_ctrl_type, &inst_data)?; let inst = ctx.function.dfg.make_inst(inst_data); let num_results = ctx.function.dfg.make_inst_results(inst, ctrl_typevar); ctx.function.layout.append_inst(inst, ebb); @@ -1046,7 +1046,7 @@ impl<'a> Parser<'a> { V: Iterator { for (src, val) in results.zip(new_results) { - try!(map.def_value(src, val, &self.loc)); + map.def_value(src, val, &self.loc)?; } Ok(()) } @@ -1066,7 +1066,7 @@ impl<'a> Parser<'a> { } while self.optional(Token::Comma) { - args.push(try!(self.match_value("expected value in argument list"))); + args.push(self.match_value("expected value in argument list")?); } Ok(args) @@ -1078,9 +1078,9 @@ impl<'a> Parser<'a> { return Ok(VariableArgs::new()); } - let args = try!(self.parse_value_list()); + let args = self.parse_value_list()?; - try!(self.match_token(Token::RPar, "expected ')' after arguments")); + self.match_token(Token::RPar, "expected ')' after arguments")?; Ok(args) } @@ -1099,28 +1099,28 @@ impl<'a> Parser<'a> { InstructionData::Unary { opcode: opcode, ty: VOID, - arg: try!(self.match_value("expected SSA value operand")), + arg: self.match_value("expected SSA value operand")?, } } InstructionFormat::UnaryImm => { InstructionData::UnaryImm { opcode: opcode, ty: VOID, - imm: try!(self.match_imm64("expected immediate integer operand")), + imm: self.match_imm64("expected immediate integer operand")?, } } InstructionFormat::UnaryIeee32 => { InstructionData::UnaryIeee32 { opcode: opcode, ty: VOID, - imm: try!(self.match_ieee32("expected immediate 32-bit float operand")), + imm: self.match_ieee32("expected immediate 32-bit float operand")?, } } InstructionFormat::UnaryIeee64 => { InstructionData::UnaryIeee64 { opcode: opcode, ty: VOID, - imm: try!(self.match_ieee64("expected immediate 64-bit float operand")), + imm: self.match_ieee64("expected immediate 64-bit float operand")?, } } InstructionFormat::UnaryImmVector => { @@ -1131,13 +1131,13 @@ impl<'a> Parser<'a> { opcode: opcode, ty: VOID, second_result: None.into(), - arg: try!(self.match_value("expected SSA value operand")), + arg: self.match_value("expected SSA value operand")?, } } InstructionFormat::Binary => { - let lhs = try!(self.match_value("expected SSA value first operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let rhs = try!(self.match_value("expected SSA value second operand")); + let lhs = self.match_value("expected SSA value first operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let rhs = self.match_value("expected SSA value second operand")?; InstructionData::Binary { opcode: opcode, ty: VOID, @@ -1145,9 +1145,9 @@ impl<'a> Parser<'a> { } } InstructionFormat::BinaryImm => { - let lhs = try!(self.match_value("expected SSA value first operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let rhs = try!(self.match_imm64("expected immediate integer second operand")); + let lhs = self.match_value("expected SSA value first operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let rhs = self.match_imm64("expected immediate integer second operand")?; InstructionData::BinaryImm { opcode: opcode, ty: VOID, @@ -1156,9 +1156,9 @@ impl<'a> Parser<'a> { } } InstructionFormat::BinaryImmRev => { - let lhs = try!(self.match_imm64("expected immediate integer first operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let rhs = try!(self.match_value("expected SSA value second operand")); + let lhs = self.match_imm64("expected immediate integer first operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let rhs = self.match_value("expected SSA value second operand")?; InstructionData::BinaryImmRev { opcode: opcode, ty: VOID, @@ -1167,9 +1167,9 @@ impl<'a> Parser<'a> { } } InstructionFormat::BinaryOverflow => { - let lhs = try!(self.match_value("expected SSA value first operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let rhs = try!(self.match_value("expected SSA value second operand")); + let lhs = self.match_value("expected SSA value first operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let rhs = self.match_value("expected SSA value second operand")?; InstructionData::BinaryOverflow { opcode: opcode, ty: VOID, @@ -1180,11 +1180,11 @@ impl<'a> Parser<'a> { InstructionFormat::Ternary => { // Names here refer to the `select` instruction. // This format is also use by `fma`. - let ctrl_arg = try!(self.match_value("expected SSA value control operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let true_arg = try!(self.match_value("expected SSA value true operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let false_arg = try!(self.match_value("expected SSA value false operand")); + let ctrl_arg = self.match_value("expected SSA value control operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let true_arg = self.match_value("expected SSA value true operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let false_arg = self.match_value("expected SSA value false operand")?; InstructionData::Ternary { opcode: opcode, ty: VOID, @@ -1193,11 +1193,11 @@ impl<'a> Parser<'a> { } InstructionFormat::TernaryOverflow => { // Names here refer to the `iadd_carry` instruction. - let lhs = try!(self.match_value("expected SSA value first operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let rhs = try!(self.match_value("expected SSA value second operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let cin = try!(self.match_value("expected SSA value third operand")); + let lhs = self.match_value("expected SSA value first operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let rhs = self.match_value("expected SSA value second operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let cin = self.match_value("expected SSA value third operand")?; InstructionData::TernaryOverflow { opcode: opcode, ty: VOID, @@ -1207,8 +1207,8 @@ impl<'a> Parser<'a> { } InstructionFormat::Jump => { // Parse the destination EBB number. Don't translate source to local numbers yet. - let ebb_num = try!(self.match_ebb("expected jump destination EBB")); - let args = try!(self.parse_opt_value_list()); + let ebb_num = self.match_ebb("expected jump destination EBB")?; + let args = self.parse_opt_value_list()?; InstructionData::Jump { opcode: opcode, ty: VOID, @@ -1219,10 +1219,10 @@ impl<'a> Parser<'a> { } } InstructionFormat::Branch => { - let ctrl_arg = try!(self.match_value("expected SSA value control operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let ebb_num = try!(self.match_ebb("expected branch destination EBB")); - let args = try!(self.parse_opt_value_list()); + let ctrl_arg = self.match_value("expected SSA value control operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let ebb_num = self.match_ebb("expected branch destination EBB")?; + let args = self.parse_opt_value_list()?; InstructionData::Branch { opcode: opcode, ty: VOID, @@ -1234,11 +1234,11 @@ impl<'a> Parser<'a> { } } InstructionFormat::InsertLane => { - let lhs = try!(self.match_value("expected SSA value first operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let lane = try!(self.match_uimm8("expected lane number")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let rhs = try!(self.match_value("expected SSA value last operand")); + let lhs = self.match_value("expected SSA value first operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let lane = self.match_uimm8("expected lane number")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let rhs = self.match_value("expected SSA value last operand")?; InstructionData::InsertLane { opcode: opcode, ty: VOID, @@ -1247,9 +1247,9 @@ impl<'a> Parser<'a> { } } InstructionFormat::ExtractLane => { - let arg = try!(self.match_value("expected SSA value last operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let lane = try!(self.match_uimm8("expected lane number")); + let arg = self.match_value("expected SSA value last operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let lane = self.match_uimm8("expected lane number")?; InstructionData::ExtractLane { opcode: opcode, ty: VOID, @@ -1258,11 +1258,11 @@ impl<'a> Parser<'a> { } } InstructionFormat::IntCompare => { - let cond = try!(self.match_enum("expected intcc condition code")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let lhs = try!(self.match_value("expected SSA value first operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let rhs = try!(self.match_value("expected SSA value second operand")); + let cond = self.match_enum("expected intcc condition code")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let lhs = self.match_value("expected SSA value first operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let rhs = self.match_value("expected SSA value second operand")?; InstructionData::IntCompare { opcode: opcode, ty: VOID, @@ -1271,11 +1271,11 @@ impl<'a> Parser<'a> { } } InstructionFormat::FloatCompare => { - let cond = try!(self.match_enum("expected floatcc condition code")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let lhs = try!(self.match_value("expected SSA value first operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let rhs = try!(self.match_value("expected SSA value second operand")); + let cond = self.match_enum("expected floatcc condition code")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let lhs = self.match_value("expected SSA value first operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let rhs = self.match_value("expected SSA value second operand")?; InstructionData::FloatCompare { opcode: opcode, ty: VOID, @@ -1284,11 +1284,11 @@ impl<'a> Parser<'a> { } } InstructionFormat::Call => { - let func_ref = try!(self.match_fn("expected function reference") - .and_then(|num| ctx.get_fn(num, &self.loc))); - try!(self.match_token(Token::LPar, "expected '(' before arguments")); - let args = try!(self.parse_value_list()); - try!(self.match_token(Token::RPar, "expected ')' after arguments")); + let func_ref = self.match_fn("expected function reference") + .and_then(|num| ctx.get_fn(num, &self.loc))?; + self.match_token(Token::LPar, "expected '(' before arguments")?; + let args = self.parse_value_list()?; + self.match_token(Token::RPar, "expected ')' after arguments")?; InstructionData::Call { opcode: opcode, ty: VOID, @@ -1300,13 +1300,13 @@ impl<'a> Parser<'a> { } } InstructionFormat::IndirectCall => { - let sig_ref = try!(self.match_sig("expected signature reference") - .and_then(|num| ctx.get_sig(num, &self.loc))); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let callee = try!(self.match_value("expected SSA value callee operand")); - try!(self.match_token(Token::LPar, "expected '(' before arguments")); - let args = try!(self.parse_value_list()); - try!(self.match_token(Token::RPar, "expected ')' after arguments")); + let sig_ref = self.match_sig("expected signature reference") + .and_then(|num| ctx.get_sig(num, &self.loc))?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let callee = self.match_value("expected SSA value callee operand")?; + self.match_token(Token::LPar, "expected '(' before arguments")?; + let args = self.parse_value_list()?; + self.match_token(Token::RPar, "expected ')' after arguments")?; InstructionData::IndirectCall { opcode: opcode, ty: VOID, @@ -1319,7 +1319,7 @@ impl<'a> Parser<'a> { } } InstructionFormat::Return => { - let args = try!(self.parse_value_list()); + let args = self.parse_value_list()?; InstructionData::Return { opcode: opcode, ty: VOID, @@ -1327,9 +1327,9 @@ impl<'a> Parser<'a> { } } InstructionFormat::ReturnReg => { - let raddr = try!(self.match_value("expected SSA value return address operand")); + let raddr = self.match_value("expected SSA value return address operand")?; let args = if self.optional(Token::Comma) { - try!(self.parse_value_list()) + self.parse_value_list()? } else { VariableArgs::new() }; @@ -1343,9 +1343,9 @@ impl<'a> Parser<'a> { } } InstructionFormat::BranchTable => { - let arg = try!(self.match_value("expected SSA value operand")); - try!(self.match_token(Token::Comma, "expected ',' between operands")); - let table = try!(self.match_jt().and_then(|num| ctx.get_jt(num, &self.loc))); + let arg = self.match_value("expected SSA value operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let table = self.match_jt().and_then(|num| ctx.get_jt(num, &self.loc))?; InstructionData::BranchTable { opcode: opcode, ty: VOID, diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 369e732e90..dda6420220 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -121,7 +121,7 @@ impl SourceMap { /// Rewrite a slice of value references. pub fn rewrite_values(&self, vals: &mut [Value], loc: AnyEntity) -> Result<()> { for val in vals { - try!(self.rewrite_value(val, loc)); + self.rewrite_value(val, loc)?; } Ok(()) } diff --git a/lib/reader/src/testcommand.rs b/lib/reader/src/testcommand.rs index 012c7033e1..6d9dd9969e 100644 --- a/lib/reader/src/testcommand.rs +++ b/lib/reader/src/testcommand.rs @@ -47,9 +47,9 @@ impl<'a> TestCommand<'a> { impl<'a> Display for TestCommand<'a> { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - try!(write!(f, "{}", self.command)); + write!(f, "{}", self.command)?; for opt in &self.options { - try!(write!(f, " {}", opt)); + write!(f, " {}", opt)?; } writeln!(f, "") } From af3f4703b99653c271e7ef28bcba81e5d7c2b869 Mon Sep 17 00:00:00 2001 From: rep-nop Date: Sat, 25 Feb 2017 22:21:03 -0500 Subject: [PATCH 0539/3084] Fixes formatting for settings.rs --- lib/cretonne/src/settings.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 0d81c38dce..5ffe5dc6a7 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -124,7 +124,7 @@ impl Configurable for Builder { } Detail::Enum { last, enumerators } => { self.bytes[offset] = parse_enum_value(value, - self.template.enums(last, enumerators))?; + self.template.enums(last, enumerators))?; } } Ok(()) From 9f00a40b5277bc92c293210f58d18141d7304ac6 Mon Sep 17 00:00:00 2001 From: rep-nop Date: Sat, 25 Feb 2017 23:56:16 -0500 Subject: [PATCH 0540/3084] Ran rustfmt --- lib/filecheck/src/explain.rs | 10 +++++----- lib/reader/src/parser.rs | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/lib/filecheck/src/explain.rs b/lib/filecheck/src/explain.rs index 825f162079..75fe6f9286 100644 --- a/lib/filecheck/src/explain.rs +++ b/lib/filecheck/src/explain.rs @@ -112,11 +112,11 @@ impl<'a> Display for Explainer<'a> { // Emit the match message itself. writeln!(f, - "{} #{}{}: {}", - if m.is_match { "Matched" } else { "Missed" }, - m.directive, - if m.is_not { " not" } else { "" }, - m.regex)?; + "{} #{}{}: {}", + if m.is_match { "Matched" } else { "Missed" }, + m.directive, + if m.is_not { " not" } else { "" }, + m.regex)?; // Emit any variable definitions. if let Ok(found) = self.vardefs.binary_search_by_key(&m.directive, |v| v.directive) { diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index e9a5a26a02..bd6fd8fce8 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -481,8 +481,8 @@ impl<'a> Parser<'a> { "set" => { last_set_loc = Some(self.loc); isaspec::parse_options(self.consume_line().trim().split_whitespace(), - &mut flag_builder, - &self.loc)?; + &mut flag_builder, + &self.loc)?; } "isa" => { last_set_loc = None; From 0bc27fca9a19d9aa84b32f81e00ebaf9023679f7 Mon Sep 17 00:00:00 2001 From: rep-nop Date: Sun, 26 Feb 2017 00:14:05 -0500 Subject: [PATCH 0541/3084] Fixes error on propagating a Result<()> with the `?` operator --- lib/cretonne/src/settings.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 5ffe5dc6a7..23ff7d688d 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -117,7 +117,9 @@ impl Configurable for Builder { let (offset, detail) = self.lookup(name)?; match detail { Detail::Bool { bit } => { - self.set_bit(offset, bit, parse_bool_value(value))?; + // Cannot currently propagate Result<()> up on functions returning () + // with the `?` operator + self.set_bit(offset, bit, parse_bool_value(value)?); } Detail::Num => { self.bytes[offset] = value.parse().map_err(|_| Error::BadValue)?; From 1f76521580b100d97b56b84fb502a34e14fd9bec Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 27 Feb 2017 15:03:06 -0800 Subject: [PATCH 0542/3084] Require a current Rust version. Rust 1.12 did work at one point, but Travis is testing against the current stable and beta releases, so that is the only versions we can claim to support. Fixes #51. --- README.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.rst b/README.rst index 5656547e49..4a46f9267c 100644 --- a/README.rst +++ b/README.rst @@ -42,8 +42,9 @@ Building Cretonne ----------------- Cretonne is using the Cargo package manager format. First, ensure you have -installed `rust 1.12.0` or above. Then, change the working directory to your -clone of cretonne and run:: +installed a current stable rust (stable, beta, and nightly should all work, but +only stable and beta are tested consistently). Then, change the working +directory to your clone of cretonne and run:: cargo build From 51702191257e10cc8d18bd583590a66bc9bf4a61 Mon Sep 17 00:00:00 2001 From: Davide Italiano Date: Tue, 28 Feb 2017 12:39:50 -0800 Subject: [PATCH 0543/3084] [EntityList] Fix typo. No functional change. --- lib/cretonne/src/entity_list.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index d94795c902..b2c75bd9ed 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -16,7 +16,7 @@ //! Entity lists are not as safe to use as `Vec`, but they never jeopardize Rust's memory safety //! guarantees. These are the problems to be aware of: //! -//! - If you lose track of an entity list, it's memory won't be recycled until the pool is cleared. +//! - If you lose track of an entity list, its memory won't be recycled until the pool is cleared. //! This can cause the pool to grow very large with leaked lists. //! - If entity lists are used after their pool is cleared, they may contain garbage data, and //! modifying them may corrupt other lists in the pool. From c01ff670ed860d0e08e1065c91356aa984bf3c8f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 3 Mar 2017 09:04:29 -0800 Subject: [PATCH 0544/3084] Fixed for mypy 0.501. The List and Dict types are no longer implicitly available. They must be imported from typing. Type annotations must appear before the doc comment in a function. Also fix type errors in these functions that weren't detected before. --- lib/cretonne/meta/cdsl/ast.py | 14 +++++++------- lib/cretonne/meta/cdsl/formats.py | 2 +- lib/cretonne/meta/cdsl/instructions.py | 2 +- lib/cretonne/meta/cdsl/isa.py | 2 +- lib/cretonne/meta/cdsl/registers.py | 2 +- lib/cretonne/meta/cdsl/types.py | 5 +++++ lib/cretonne/meta/cdsl/xform.py | 2 +- lib/cretonne/meta/gen_instr.py | 7 +++++++ lib/cretonne/meta/gen_registers.py | 2 +- lib/cretonne/meta/isa/__init__.py | 5 +++++ lib/cretonne/meta/srcgen.py | 2 +- 11 files changed, 31 insertions(+), 14 deletions(-) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index ddcc702575..931cc1e3a2 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -143,22 +143,22 @@ class Var(Expr): def is_input(self): # type: () -> bool """Is this an input value to the src pattern?""" - return not self.src_def and not self.dst_def + return self.src_def is None and self.dst_def is None def is_output(self): - """Is this an output value, defined in both src and dst patterns?""" # type: () -> bool - return self.src_def and self.dst_def + """Is this an output value, defined in both src and dst patterns?""" + return self.src_def is not None and self.dst_def is not None def is_intermediate(self): - """Is this an intermediate value, defined only in the src pattern?""" # type: () -> bool - return self.src_def and not self.dst_def + """Is this an intermediate value, defined only in the src pattern?""" + return self.src_def is not None and self.dst_def is None def is_temp(self): - """Is this a temp value, defined only in the dst pattern?""" # type: () -> bool - return not self.src_def and self.dst_def + """Is this a temp value, defined only in the dst pattern?""" + return self.src_def is None and self.dst_def is not None def get_typevar(self): # type: () -> TypeVar diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index 32f4463ae0..974ca80541 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -6,7 +6,7 @@ from .operands import Operand # noqa # The typing module is only required by mypy, and we don't use these imports # outside type comments. try: - from typing import Tuple, Union, Any, Sequence, Iterable # noqa + from typing import Dict, List, Tuple, Union, Any, Sequence, Iterable # noqa except ImportError: pass diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 84968c1b6b..4040e5338b 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -6,7 +6,7 @@ from .operands import Operand from .formats import InstructionFormat try: - from typing import Union, Sequence + from typing import Union, Sequence, List # noqa # List of operands for ins/outs: OpList = Union[Sequence[Operand], Operand] MaybeBoundInst = Union['Instruction', 'BoundInstruction'] diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 0065e4e907..7e4c09e58d 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -6,7 +6,7 @@ from .registers import RegClass, Register # The typing module is only required by mypy, and we don't use these imports # outside type comments. try: - from typing import Tuple, Union, Any, Iterable, Sequence, TYPE_CHECKING # noqa + from typing import Tuple, Union, Any, Iterable, Sequence, List, Set, TYPE_CHECKING # noqa from .instructions import MaybeBoundInst, InstructionGroup, InstructionFormat # noqa from .predicates import Predicate, FieldPredicate # noqa from .settings import SettingGroup # noqa diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index 99dd453aec..003ce6379c 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -26,7 +26,7 @@ from . import is_power_of_two, next_power_of_two try: - from typing import Sequence, Tuple # noqa + from typing import Sequence, Tuple, List, Dict # noqa from .isa import TargetISA # noqa # A tuple uniquely identifying a register class inside a register bank. # (count, width, start) diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py index 62334b7dd7..f96c5a2ef0 100644 --- a/lib/cretonne/meta/cdsl/types.py +++ b/lib/cretonne/meta/cdsl/types.py @@ -2,6 +2,11 @@ from __future__ import absolute_import import math +try: + from typing import Dict, List # noqa +except ImportError: + pass + # ValueType instances (i8, i32, ...) are provided in the cretonne.types module. class ValueType(object): diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index ee6710a79f..deca293e50 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -5,7 +5,7 @@ from __future__ import absolute_import from .ast import Def, Var, Apply try: - from typing import Union, Iterator, Sequence, Iterable # noqa + from typing import Union, Iterator, Sequence, Iterable, List, Dict # noqa from .ast import Expr # noqa DefApply = Union[Def, Apply] except ImportError: diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 8cf89e7654..166967698e 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -14,6 +14,13 @@ from cdsl.instructions import Instruction # noqa from cdsl.operands import Operand # noqa from cdsl.typevar import TypeVar # noqa +# The typing module is only required by mypy, and we don't use these imports +# outside type comments. +try: + from typing import List # noqa +except ImportError: + pass + def gen_formats(fmt): # type: (srcgen.Formatter) -> None diff --git a/lib/cretonne/meta/gen_registers.py b/lib/cretonne/meta/gen_registers.py index e36434f4d6..7dfdc45d00 100644 --- a/lib/cretonne/meta/gen_registers.py +++ b/lib/cretonne/meta/gen_registers.py @@ -6,7 +6,7 @@ from __future__ import absolute_import import srcgen try: - from typing import Sequence # noqa + from typing import Sequence, List # noqa from cdsl.isa import TargetISA # noqa from cdsl.registers import RegBank, RegClass # noqa except ImportError: diff --git a/lib/cretonne/meta/isa/__init__.py b/lib/cretonne/meta/isa/__init__.py index 8f623d18bb..bad91c5c90 100644 --- a/lib/cretonne/meta/isa/__init__.py +++ b/lib/cretonne/meta/isa/__init__.py @@ -9,6 +9,11 @@ from __future__ import absolute_import from cdsl.isa import TargetISA # noqa from . import riscv, intel, arm32, arm64 +try: + from typing import List # noqa +except ImportError: + pass + def all_isas(): # type: () -> List[TargetISA] diff --git a/lib/cretonne/meta/srcgen.py b/lib/cretonne/meta/srcgen.py index 75b102f534..e87c16a577 100644 --- a/lib/cretonne/meta/srcgen.py +++ b/lib/cretonne/meta/srcgen.py @@ -11,7 +11,7 @@ import os import re try: - from typing import Any # noqa + from typing import Any, List # noqa except ImportError: pass From 99c7e18fbf0199f58520dfaf4e1e5e4c20174c0e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 3 Mar 2017 10:12:49 -0800 Subject: [PATCH 0545/3084] Add {ValueLoc,ArgumentLoc}::display(). These functions return an object containing the necessary ISA info to print a ValueLoc or ArgumentLoc correctly. --- lib/cretonne/src/ir/valueloc.rs | 61 ++++++++++++++++++++++++++++++++- lib/cretonne/src/write.rs | 8 ++--- 2 files changed, 62 insertions(+), 7 deletions(-) diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/cretonne/src/ir/valueloc.rs index ea7dea39d9..db1ee2baf2 100644 --- a/lib/cretonne/src/ir/valueloc.rs +++ b/lib/cretonne/src/ir/valueloc.rs @@ -3,8 +3,9 @@ //! The register allocator assigns every SSA value to either a register or a stack slot. This //! assignment is represented by a `ValueLoc` object. -use isa::RegUnit; +use isa::{RegInfo, RegUnit}; use ir::StackSlot; +use std::fmt; /// Value location. #[derive(Copy, Clone, Debug)] @@ -23,6 +24,35 @@ impl Default for ValueLoc { } } +impl ValueLoc { + /// Return an object that can display this value location, using the register info from the + /// target ISA. + pub fn display<'a, R: Into>>(self, regs: R) -> DisplayValueLoc<'a> { + DisplayValueLoc(self, regs.into()) + } +} + +/// Displaying a `ValueLoc` correctly requires the associated `RegInfo` from the target ISA. +/// Without the register info, register units are simply show as numbers. +/// +/// The `DisplayValueLoc` type can display the contained `ValueLoc`. +pub struct DisplayValueLoc<'a>(ValueLoc, Option<&'a RegInfo>); + +impl<'a> fmt::Display for DisplayValueLoc<'a> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self.0 { + ValueLoc::Unassigned => write!(f, "-"), + ValueLoc::Reg(ru) => { + match self.1 { + Some(regs) => write!(f, "{}", regs.display_regunit(ru)), + None => write!(f, "%{}", ru), + } + } + ValueLoc::Stack(ss) => write!(f, "{}", ss), + } + } +} + /// Function argument location. /// /// The ABI specifies how arguments are passed to a function, and where return values appear after @@ -56,3 +86,32 @@ impl Default for ArgumentLoc { ArgumentLoc::Unassigned } } + +impl ArgumentLoc { + /// Return an object that can display this argument location, using the register info from the + /// target ISA. + pub fn display<'a, R: Into>>(self, regs: R) -> DisplayArgumentLoc<'a> { + DisplayArgumentLoc(self, regs.into()) + } +} + +/// Displaying a `ArgumentLoc` correctly requires the associated `RegInfo` from the target ISA. +/// Without the register info, register units are simply show as numbers. +/// +/// The `DisplayArgumentLoc` type can display the contained `ArgumentLoc`. +pub struct DisplayArgumentLoc<'a>(ArgumentLoc, Option<&'a RegInfo>); + +impl<'a> fmt::Display for DisplayArgumentLoc<'a> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self.0 { + ArgumentLoc::Unassigned => write!(f, "-"), + ArgumentLoc::Reg(ru) => { + match self.1 { + Some(regs) => write!(f, "{}", regs.display_regunit(ru)), + None => write!(f, "%{}", ru), + } + } + ArgumentLoc::Stack(offset) => write!(f, "{}", offset), + } + } +} diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index eca9909961..957f8a036d 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -4,7 +4,7 @@ //! equivalent textual representation. This textual representation can be read back by the //! `cretonne-reader` crate. -use ir::{Function, Ebb, Inst, Value, Type, ValueLoc}; +use ir::{Function, Ebb, Inst, Value, Type}; use isa::TargetIsa; use std::fmt::{Result, Error, Write}; use std::result; @@ -177,11 +177,7 @@ fn write_instruction(w: &mut Write, if !func.locations.is_empty() { let regs = isa.register_info(); for r in func.dfg.inst_results(inst) { - match func.locations[r] { - ValueLoc::Unassigned => write!(s, ",-")?, - ValueLoc::Reg(ru) => write!(s, ",{}", regs.display_regunit(ru))?, - ValueLoc::Stack(ss) => write!(s, ",{}", ss)?, - } + write!(s, ",{}", func.locations[r].display(®s))? } } write!(s, "]")?; From cae4368a8adb9b51d2973be6534d0a691b4af482 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 3 Mar 2017 10:34:29 -0800 Subject: [PATCH 0546/3084] Use ISA information to display function signatures. The argument locations contains register unit references that we want to display with their correct names. --- lib/cretonne/src/ir/extfunc.rs | 73 +++++++++++++++++++++++---------- lib/cretonne/src/ir/function.rs | 2 +- lib/cretonne/src/ir/valueloc.rs | 8 ++++ lib/cretonne/src/write.rs | 23 +++++++---- 4 files changed, 77 insertions(+), 29 deletions(-) diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 5c4d1d1c71..9f81d11442 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -6,8 +6,9 @@ //! This module declares the data types used to represent external functions and call signatures. use ir::{Type, FunctionName, SigRef, ArgumentLoc}; +use isa::RegInfo; use std::cmp; -use std::fmt::{self, Display, Formatter}; +use std::fmt; /// Function signature. /// @@ -55,34 +56,51 @@ impl Signature { .fold(0, cmp::max); self.argument_bytes = Some(bytes); } + + /// Return an object that can display `self` with correct register names. + pub fn display<'a, R: Into>>(&'a self, regs: R) -> DisplaySignature<'a> { + DisplaySignature(self, regs.into()) + } } -fn write_list(f: &mut Formatter, args: &Vec) -> fmt::Result { +/// Wrapper type capable of displaying a `Signature` with correct register names. +pub struct DisplaySignature<'a>(&'a Signature, Option<&'a RegInfo>); + +fn write_list(f: &mut fmt::Formatter, + args: &Vec, + regs: Option<&RegInfo>) + -> fmt::Result { match args.split_first() { None => {} Some((first, rest)) => { - write!(f, "{}", first)?; + write!(f, "{}", first.display(regs))?; for arg in rest { - write!(f, ", {}", arg)?; + write!(f, ", {}", arg.display(regs))?; } } } Ok(()) } -impl Display for Signature { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { +impl<'a> fmt::Display for DisplaySignature<'a> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "(")?; - write_list(f, &self.argument_types)?; + write_list(f, &self.0.argument_types, self.1)?; write!(f, ")")?; - if !self.return_types.is_empty() { + if !self.0.return_types.is_empty() { write!(f, " -> ")?; - write_list(f, &self.return_types)?; + write_list(f, &self.0.return_types, self.1)?; } Ok(()) } } +impl fmt::Display for Signature { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + self.display(None).fmt(f) + } +} + /// Function argument or return value type. /// /// This describes the value type being passed to or from a function along with flags that affect @@ -111,26 +129,39 @@ impl ArgumentType { location: Default::default(), } } + + /// Return an object that can display `self` with correct register names. + pub fn display<'a, R: Into>>(&'a self, regs: R) -> DisplayArgumentType<'a> { + DisplayArgumentType(self, regs.into()) + } } -impl Display for ArgumentType { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - write!(f, "{}", self.value_type)?; - match self.extension { +/// Wrapper type capable of displaying an `ArgumentType` with correct register names. +pub struct DisplayArgumentType<'a>(&'a ArgumentType, Option<&'a RegInfo>); + +impl<'a> fmt::Display for DisplayArgumentType<'a> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{}", self.0.value_type)?; + match self.0.extension { ArgumentExtension::None => {} ArgumentExtension::Uext => write!(f, " uext")?, ArgumentExtension::Sext => write!(f, " sext")?, } - if self.inreg { + if self.0.inreg { write!(f, " inreg")?; } - // This really needs a `&TargetAbi` so we can print register units correctly. - match self.location { - ArgumentLoc::Reg(ru) => write!(f, " [%{}]", ru), - ArgumentLoc::Stack(offset) => write!(f, " [{}]", offset), - ArgumentLoc::Unassigned => Ok(()), + if self.0.location.is_assigned() { + write!(f, " [{}]", self.0.location.display(self.1))?; } + + Ok(()) + } +} + +impl fmt::Display for ArgumentType { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + self.display(None).fmt(f) } } @@ -159,8 +190,8 @@ pub struct ExtFuncData { pub signature: SigRef, } -impl Display for ExtFuncData { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { +impl fmt::Display for ExtFuncData { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "{} {}", self.signature, self.name) } } diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 2828104daa..b1ae83dc28 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -20,7 +20,7 @@ pub struct Function { pub name: FunctionName, /// Signature of this function. - signature: Signature, + pub signature: Signature, /// Stack slots allocated in this function. pub stack_slots: EntityMap, diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/cretonne/src/ir/valueloc.rs index db1ee2baf2..253c5ceb33 100644 --- a/lib/cretonne/src/ir/valueloc.rs +++ b/lib/cretonne/src/ir/valueloc.rs @@ -88,6 +88,14 @@ impl Default for ArgumentLoc { } impl ArgumentLoc { + /// Is this an assigned location? (That is, not `Unassigned`). + pub fn is_assigned(&self) -> bool { + match self { + &ArgumentLoc::Unassigned => false, + _ => true, + } + } + /// Return an object that can display this argument location, using the register info from the /// target ISA. pub fn display<'a, R: Into>>(self, regs: R) -> DisplayArgumentLoc<'a> { diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 957f8a036d..40b625d6e0 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -5,16 +5,19 @@ //! `cretonne-reader` crate. use ir::{Function, Ebb, Inst, Value, Type}; -use isa::TargetIsa; +use isa::{TargetIsa, RegInfo}; use std::fmt::{Result, Error, Write}; use std::result; /// Write `func` to `w` as equivalent text. /// Use `isa` to emit ISA-dependent annotations. pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) -> Result { - write_spec(w, func)?; + let regs = isa.map(TargetIsa::register_info); + let regs = regs.as_ref(); + + write_spec(w, func, regs)?; writeln!(w, " {{")?; - let mut any = write_preamble(w, func)?; + let mut any = write_preamble(w, func, regs)?; for ebb in &func.layout { if any { writeln!(w, "")?; @@ -31,11 +34,14 @@ pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) - // // ====--------------------------------------------------------------------------------------====// -fn write_spec(w: &mut Write, func: &Function) -> Result { - write!(w, "function {}{}", func.name, func.own_signature()) +fn write_spec(w: &mut Write, func: &Function, regs: Option<&RegInfo>) -> Result { + write!(w, "function {}{}", func.name, func.signature.display(regs)) } -fn write_preamble(w: &mut Write, func: &Function) -> result::Result { +fn write_preamble(w: &mut Write, + func: &Function, + regs: Option<&RegInfo>) + -> result::Result { let mut any = false; for ss in func.stack_slots.keys() { @@ -47,7 +53,10 @@ fn write_preamble(w: &mut Write, func: &Function) -> result::Result // signatures. for sig in func.dfg.signatures.keys() { any = true; - writeln!(w, " {} = signature{}", sig, func.dfg.signatures[sig])?; + writeln!(w, + " {} = signature{}", + sig, + func.dfg.signatures[sig].display(regs))?; } for fnref in func.dfg.ext_funcs.keys() { From 07f459fb93a03ce6ac9689be0822d5355d80f18f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 1 Mar 2017 11:53:52 -0800 Subject: [PATCH 0547/3084] Add a legalize_signature method to TargetIsa. This entry point will be used for controlling ABI conventions when legalizing. Provide an empty implementation for RISC-V and let the other ISAs crash in legalization. This is just the scaffolding. We still need to: - Rewrite the entry block arguments to match the legalized signature. - Rewrite call and return instructions. - Implement the legalize_signature() function for all ISAs. - Add shared generic types to help with the legalize_signature() functions. --- lib/cretonne/src/ir/function.rs | 5 ----- lib/cretonne/src/isa/mod.rs | 22 +++++++++++++++++++++- lib/cretonne/src/isa/riscv/abi.rs | 12 ++++++++++++ lib/cretonne/src/isa/riscv/mod.rs | 8 +++++++- lib/cretonne/src/legalizer.rs | 14 ++++++++++++++ 5 files changed, 54 insertions(+), 7 deletions(-) create mode 100644 lib/cretonne/src/isa/riscv/abi.rs diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index b1ae83dc28..b0115a70e3 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -64,11 +64,6 @@ impl Function { pub fn new() -> Function { Self::with_name_signature(FunctionName::default(), Signature::new()) } - - /// Get the signature of this function. - pub fn own_signature(&self) -> &Signature { - &self.signature - } } impl Display for Function { diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index af689a3947..37684addb1 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -45,7 +45,7 @@ pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex}; pub use isa::constraints::{RecipeConstraints, OperandConstraint, ConstraintKind}; use settings; -use ir::{InstructionData, DataFlowGraph}; +use ir::{InstructionData, DataFlowGraph, Signature}; pub mod riscv; pub mod intel; @@ -161,4 +161,24 @@ pub trait TargetIsa { recipe_names: self.recipe_names(), } } + + /// Legalize a function signature. + /// + /// This is used to legalize both the signature of the function being compiled and any called + /// functions. The signature should be modified by adding `ArgumentLoc` annotations to all + /// arguments and return values. + /// + /// Arguments with types that are not supported by the ABI can be expanded into multiple + /// arguments: + /// + /// - Integer types that are too large to fit in a register can be broken into multiple + /// arguments of a smaller integer type. + /// - Floating point types can be bit-cast to an integer type of the same size, and possible + /// broken into smaller integer types. + /// - Vector types can be bit-cast and broken down into smaller vectors or scalars. + /// + /// The legalizer will adapt argument and return values as necessary at all ABI boundaries. + fn legalize_signature(&self, _sig: &mut Signature) { + unimplemented!() + } } diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs new file mode 100644 index 0000000000..7025bf8c0e --- /dev/null +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -0,0 +1,12 @@ +//! RISC-V ABI implementation. +//! +//! This module implements the RISC-V calling convention through the primary `legalize_signature()` +//! entry point. + +use ir::Signature; +use settings as shared_settings; + +/// Legalize `sig` for RISC-V. +pub fn legalize_signature(_sig: &mut Signature, _flags: &shared_settings::Flags) { + // TODO: Actually do something. +} diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 8b27a46119..220f062278 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -1,6 +1,7 @@ //! RISC-V Instruction Set Architecture. pub mod settings; +mod abi; mod enc_tables; mod registers; @@ -8,7 +9,7 @@ use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, Encoding, Legalize, RecipeConstraints}; -use ir::{InstructionData, DataFlowGraph}; +use ir::{InstructionData, DataFlowGraph, Signature}; #[allow(dead_code)] struct Isa { @@ -74,6 +75,11 @@ impl TargetIsa for Isa { fn recipe_constraints(&self) -> &'static [RecipeConstraints] { &enc_tables::RECIPE_CONSTRAINTS } + + fn legalize_signature(&self, sig: &mut Signature) { + // We can pass in `self.isa_flags` too, if we need it. + abi::legalize_signature(sig, &self.shared_flags) + } } #[cfg(test)] diff --git a/lib/cretonne/src/legalizer.rs b/lib/cretonne/src/legalizer.rs index 015bfb9b30..3d57c73b95 100644 --- a/lib/cretonne/src/legalizer.rs +++ b/lib/cretonne/src/legalizer.rs @@ -23,6 +23,8 @@ use isa::{TargetIsa, Legalize}; /// - Fill out `func.encodings`. /// pub fn legalize_function(func: &mut Function, isa: &TargetIsa) { + legalize_signatures(func, isa); + // TODO: This is very simplified and incomplete. func.encodings.resize(func.dfg.num_insts()); let mut pos = Cursor::new(&mut func.layout); @@ -74,3 +76,15 @@ pub fn legalize_function(func: &mut Function, isa: &TargetIsa) { // // Concretely, this defines private functions `narrow()`, and `expand()`. include!(concat!(env!("OUT_DIR"), "/legalizer.rs")); + +/// Legalize all the function signatures in `func`. +/// +/// This changes all signatures to be ABI-compliant with full `ArgumentLoc` annotations. It doesn't +/// change the entry block arguments, calls, or return instructions, so this can leave the function +/// in a state with type discrepancies. +fn legalize_signatures(func: &mut Function, isa: &TargetIsa) { + isa.legalize_signature(&mut func.signature); + for sig in func.dfg.signatures.keys() { + isa.legalize_signature(&mut func.dfg.signatures[sig]); + } +} From 42e7021865d16de1b4cc6b503fe9f4e134188e40 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 2 Mar 2017 16:01:56 -0800 Subject: [PATCH 0548/3084] Implement legalize_signature for RISC-V. Add an abi module with code that is probably useful to all ISAs when implementing this function. Add a unit() method to RegClassData which can be used to index the register units in a class. --- cranelift/filetests/isa/riscv/abi.cton | 23 ++++++ lib/cretonne/meta/gen_registers.py | 1 + lib/cretonne/src/abi.rs | 72 +++++++++++++++++++ lib/cretonne/src/isa/registers.rs | 9 +++ lib/cretonne/src/isa/riscv/abi.rs | 76 ++++++++++++++++++-- lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/regalloc/allocatable_set.rs | 2 + 7 files changed, 180 insertions(+), 4 deletions(-) create mode 100644 cranelift/filetests/isa/riscv/abi.cton create mode 100644 lib/cretonne/src/abi.rs diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.cton new file mode 100644 index 0000000000..adb2dfed07 --- /dev/null +++ b/cranelift/filetests/isa/riscv/abi.cton @@ -0,0 +1,23 @@ +; Test the legalization of function signatures. +test legalizer +isa riscv + +function f(i32) { + sig0 = signature(i32) -> i32 + +; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] + + sig1 = signature(i64) -> b1 +; check: sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] + +; The i64 argument must go in an even-odd register pair. + sig2 = signature(f32, i64) -> f64 +; check: sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] + +; Spilling into the stack args. + sig3 = signature(f64, f64, f64, f64, f64, f64, f64, i64) -> f64 +; check: sig3 = signature(f64 [%f10], f64 [%f11], f64 [%f12], f64 [%f13], f64 [%f14], f64 [%f15], f64 [%f16], i32 [0], i32 [4]) -> f64 [%f10] + +ebb0(v0: i32): + return_reg v0 +} diff --git a/lib/cretonne/meta/gen_registers.py b/lib/cretonne/meta/gen_registers.py index 7dfdc45d00..b1532761a6 100644 --- a/lib/cretonne/meta/gen_registers.py +++ b/lib/cretonne/meta/gen_registers.py @@ -37,6 +37,7 @@ def gen_regclass(rc, fmt): fmt.line('name: "{}",'.format(rc.name)) fmt.line('index: {},'.format(rc.index)) fmt.line('width: {},'.format(rc.width)) + fmt.line('first: {},'.format(rc.bank.first_unit + rc.start)) fmt.line('subclasses: 0x{:x},'.format(rc.subclass_mask())) mask = ', '.join('0x{:08x}'.format(x) for x in rc.mask()) fmt.line('mask: [{}],'.format(mask)) diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs new file mode 100644 index 0000000000..897133bc76 --- /dev/null +++ b/lib/cretonne/src/abi.rs @@ -0,0 +1,72 @@ +//! Common helper code for ABI lowering. +//! +//! This module provides functions and data structures that are useful for implementing the +//! `TargetIsa::legalize_signature()` method. + +use ir::{ArgumentLoc, ArgumentType, Type}; + +/// Legalization action to perform on a single argument or return value. +/// +/// An argument may go through a sequence of legalization steps before it reaches the final +/// `Assign` action. +pub enum ArgAction { + /// Assign the argument to the given location. + Assign(ArgumentLoc), + + /// Split the argument into smaller parts, then call again. + /// + /// This action can split an integer type into two smaller integer arguments, or it can split a + /// SIMD vector into halves. + /// + /// Floating point scalar types can't be split. + Split, +} + +/// Common trait for assigning arguments to registers or stack locations. +/// +/// This will be implemented by individual ISAs. +pub trait ArgAssigner { + /// Pick an assignment action for function argument (or return value) `arg`. + fn assign(&mut self, arg: &ArgumentType) -> ArgAction; +} + +/// Legalize the arguments in `args` using the given argument assigner. +/// +/// This function can be used for both arguments and return values. +pub fn legalize_args(args: &mut Vec, aa: &mut AA) { + // Iterate over the arguments. + // We may need to mutate the vector in place, so don't use a normal iterator, and clone the + // argument to avoid holding a reference. + let mut argno = 0; + while let Some(arg) = args.get(argno).cloned() { + // Leave the pre-assigned arguments alone. + // We'll assume that they don't interfere with our assignments. + if arg.location.is_assigned() { + argno += 1; + continue; + } + + match aa.assign(&arg) { + // Assign argument to a location and move on to the next one. + ArgAction::Assign(loc) => { + args[argno].location = loc; + argno += 1; + } + // Split this argument into two smaller ones. Then revisit both. + ArgAction::Split => { + let new_arg = ArgumentType { value_type: split_type(arg.value_type), ..arg }; + args[argno].value_type = new_arg.value_type; + args.insert(argno + 1, new_arg); + } + } + } +} + +/// Given a value type that isn't legal, compute a replacement type half the size. +fn split_type(ty: Type) -> Type { + if ty.is_int() { + ty.half_width().expect("Integer type too small to split") + } else { + ty.half_vector().expect("Can only split integers and vectors") + } +} diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index c785ab2ac9..c51413eb20 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -108,6 +108,9 @@ pub struct RegClassData { /// How many register units to allocate per register. pub width: u8, + /// The first register unit in this class. + pub first: RegUnit, + /// Bit-mask of sub-classes of this register class, including itself. /// /// Bits correspond to RC indexes. @@ -142,6 +145,12 @@ impl RegClassData { pub fn has_subclass>(&self, other: RCI) -> bool { self.subclasses & (1 << other.into().0) != 0 } + + /// Get a specific register unit in this class. + pub fn unit(&self, offset: usize) -> RegUnit { + let uoffset = offset * self.width as usize; + self.first + uoffset as RegUnit + } } /// A small reference to a register class. diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index 7025bf8c0e..5c9630f8f4 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -2,11 +2,79 @@ //! //! This module implements the RISC-V calling convention through the primary `legalize_signature()` //! entry point. +//! +//! This doesn't support the soft-float ABI at the moment. -use ir::Signature; +use abi::{ArgAction, ArgAssigner, legalize_args}; +use ir::{Signature, ArgumentType, ArgumentLoc}; +use isa::riscv::registers::{GPR, FPR}; use settings as shared_settings; -/// Legalize `sig` for RISC-V. -pub fn legalize_signature(_sig: &mut Signature, _flags: &shared_settings::Flags) { - // TODO: Actually do something. +struct Args { + pointer_bits: u16, + pointer_bytes: u32, + regs: u32, + offset: u32, +} + +impl Args { + fn new(bits: u16) -> Args { + Args { + pointer_bits: bits, + pointer_bytes: bits as u32 / 8, + regs: 0, + offset: 0, + } + } +} + +impl ArgAssigner for Args { + fn assign(&mut self, arg: &ArgumentType) -> ArgAction { + fn align(value: u32, to: u32) -> u32 { + (value + to - 1) & !(to - 1) + } + + let ty = arg.value_type; + + // Check for a legal type. + // RISC-V doesn't have SIMD at all, so break all vectors down. + if !ty.is_scalar() { + return ArgAction::Split; + } + + // Large integers and booleans are broken down to fit in a register. + if !ty.is_float() && ty.bits() > self.pointer_bits { + // Align registers and stack to a multiple of two pointers. + self.regs = align(self.regs, 2); + self.offset = align(self.offset, 2 * self.pointer_bytes); + return ArgAction::Split; + } + + if self.regs < 8 { + // Assign to a register. + let reg = if ty.is_float() { + FPR.unit(10 + self.regs as usize) + } else { + GPR.unit(10 + self.regs as usize) + }; + self.regs += 1; + ArgAction::Assign(ArgumentLoc::Reg(reg)) + } else { + // Assign a stack location. + let loc = ArgumentLoc::Stack(self.offset); + self.offset += self.pointer_bytes; + ArgAction::Assign(loc) + } + } +} + +/// Legalize `sig` for RISC-V. +pub fn legalize_signature(sig: &mut Signature, flags: &shared_settings::Flags) { + let bits = if flags.is_64bit() { 64 } else { 32 }; + + let mut args = Args::new(bits); + legalize_args(&mut sig.argument_types, &mut args); + + let mut rets = Args::new(bits); + legalize_args(&mut sig.return_types, &mut rets); } diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 47339a39ca..2839ecfc1f 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -21,6 +21,7 @@ pub mod settings; pub mod sparse_map; pub mod verifier; +mod abi; mod constant_hash; mod context; mod legalizer; diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index 39058c04e4..40ef3c3636 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -121,6 +121,7 @@ mod tests { name: "GPR", index: 0, width: 1, + first: 28, subclasses: 0, mask: [0xf0000000, 0x0000000f, 0], }; @@ -128,6 +129,7 @@ mod tests { name: "DPR", index: 0, width: 2, + first: 28, subclasses: 0, mask: [0x50000000, 0x0000000a, 0], }; From 77492aa4637f22740bfc1ed6f396d4098d03d03f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 6 Mar 2017 14:51:59 -0800 Subject: [PATCH 0549/3084] Add take_ebb_args(), put_ebb_arg() method pair. These two methods can be use to rewrite the argument values to an EBB. In particular, we need to rewrite the arguments to the entry block to be compatible with a legalized function signature. Reuse the put_ebb_arg() method in the implementation of append_ebb_arg(). --- lib/cretonne/src/ir/dfg.rs | 111 ++++++++++++++++++++++++++++--------- 1 file changed, 86 insertions(+), 25 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 94117095a7..593d3a7907 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -505,36 +505,13 @@ impl DataFlowGraph { /// Append an argument with type `ty` to `ebb`. pub fn append_ebb_arg(&mut self, ebb: Ebb, ty: Type) -> Value { - let num_args = self.num_ebb_args(ebb); - assert!(num_args <= u16::MAX as usize, "Too many arguments to EBB"); let val = self.make_value(ValueData::Arg { ty: ty, ebb: ebb, - num: num_args as u16, + num: 0, next: None.into(), }); - match self.ebbs[ebb].last_arg.expand() { - // If last_argument is `None`, we're adding the first EBB argument. - None => { - self.ebbs[ebb].first_arg = val.into(); - } - Some(last_arg) => { - match last_arg.expand() { - // Append to linked list of arguments. - ExpandedValue::Table(idx) => { - if let ValueData::Arg { ref mut next, .. } = self.extended_values[idx] { - *next = val.into(); - } else { - panic!("inconsistent value table entry for EBB argument"); - } - } - ExpandedValue::Direct(_) => { - panic!("inconsistent value table entry for EBB argument") - } - } - } - } - self.ebbs[ebb].last_arg = val.into(); + self.put_ebb_arg(ebb, val); val } @@ -545,6 +522,74 @@ impl DataFlowGraph { cur: self.ebbs[ebb].first_arg.into(), } } + + /// Given a value that is known to be an EBB argument, return the next EBB argument. + pub fn next_ebb_arg(&self, arg: Value) -> Option { + if let ExpandedValue::Table(index) = arg.expand() { + if let ValueData::Arg { next, .. } = self.extended_values[index] { + return next.into(); + } + } + panic!("{} is not an EBB argument", arg); + } + + /// Detach all the arguments from `ebb` and return the first one. + /// + /// The whole list of detached arguments can be traversed with `next_ebb_arg`. This method does + /// not return a `Values` iterator since it is often necessary to mutate the DFG while + /// processing the list of arguments. + /// + /// This is a quite low-level operation. Sensible things to do with the detached EBB arguments + /// is to put them back on the same EBB with `put_ebb_arg()` or change them into aliases + /// with `change_to_alias()`. + pub fn take_ebb_args(&mut self, ebb: Ebb) -> Option { + let first = self.ebbs[ebb].first_arg.into(); + self.ebbs[ebb].first_arg = None.into(); + self.ebbs[ebb].last_arg = None.into(); + first + } + + /// Append an existing argument value to `ebb`. + /// + /// The appended value should already be an EBB argument belonging to `ebb`, but it can't be + /// attached. In practice, this means that it should be one of the values returned from + /// `take_ebb_args()`. + /// + /// In almost all cases, you should be using `append_ebb_arg()` instead of this method. + pub fn put_ebb_arg(&mut self, ebb: Ebb, arg: Value) { + let arg_num = match self.ebbs[ebb].last_arg.map(|v| v.expand()) { + // If last_argument is `None`, we're adding the first EBB argument. + None => { + self.ebbs[ebb].first_arg = arg.into(); + 0 + } + // Append to the linked list of arguments. + Some(ExpandedValue::Table(idx)) => { + if let ValueData::Arg { ref mut next, num, .. } = self.extended_values[idx] { + *next = arg.into(); + assert!(num < u16::MAX, "Too many arguments to EBB"); + num + 1 + } else { + panic!("inconsistent value table entry for EBB argument"); + } + } + _ => panic!("inconsistent value table entry for EBB argument"), + }; + self.ebbs[ebb].last_arg = arg.into(); + + // Now update `arg` itself. + let arg_ebb = ebb; + if let ExpandedValue::Table(idx) = arg.expand() { + if let ValueData::Arg { ref mut num, ebb, ref mut next, .. } = + self.extended_values[idx] { + *num = arg_num; + *next = None.into(); + assert_eq!(arg_ebb, ebb, "{} should already belong to EBB", arg); + return; + } + } + panic!("{} must be an EBB argument value", arg); + } } // Contents of an extended basic block. @@ -630,6 +675,9 @@ mod tests { assert_eq!(ebb.to_string(), "ebb0"); assert_eq!(dfg.num_ebb_args(ebb), 0); assert_eq!(dfg.ebb_args(ebb).next(), None); + assert_eq!(dfg.take_ebb_args(ebb), None); + assert_eq!(dfg.num_ebb_args(ebb), 0); + assert_eq!(dfg.ebb_args(ebb).next(), None); let arg1 = dfg.append_ebb_arg(ebb, types::F32); assert_eq!(arg1.to_string(), "vx0"); @@ -653,6 +701,19 @@ mod tests { assert_eq!(dfg.value_def(arg2), ValueDef::Arg(ebb, 1)); assert_eq!(dfg.value_type(arg1), types::F32); assert_eq!(dfg.value_type(arg2), types::I16); + + // Swap the two EBB arguments. + let take1 = dfg.take_ebb_args(ebb).unwrap(); + assert_eq!(dfg.num_ebb_args(ebb), 0); + assert_eq!(dfg.ebb_args(ebb).next(), None); + let take2 = dfg.next_ebb_arg(take1).unwrap(); + assert_eq!(take1, arg1); + assert_eq!(take2, arg2); + assert_eq!(dfg.next_ebb_arg(take2), None); + dfg.put_ebb_arg(ebb, take2); + let arg3 = dfg.append_ebb_arg(ebb, types::I32); + dfg.put_ebb_arg(ebb, take1); + assert_eq!(dfg.ebb_args(ebb).collect::>(), [take2, arg3, take1]); } #[test] From 37b2e94c72391ba64d4d38f7b0fe002717f191c3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 6 Mar 2017 13:57:36 -0800 Subject: [PATCH 0550/3084] Legalize entry block arguments to match ABI types. Insert conversion code that reconstructs the original function argument types from the legalized ABI signature. Add abi::legalize_abi_value(). This function is used when adapting code to a legalized function signature. --- cranelift/filetests/isa/riscv/abi.cton | 11 ++ .../filetests/isa/riscv/legalize-i64.cton | 16 +- lib/cretonne/src/abi.rs | 156 ++++++++++++++++-- lib/cretonne/src/ir/types.rs | 11 ++ lib/cretonne/src/isa/riscv/abi.rs | 6 +- lib/cretonne/src/legalizer.rs | 115 ++++++++++++- 6 files changed, 288 insertions(+), 27 deletions(-) diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.cton index adb2dfed07..e3bafb83e1 100644 --- a/cranelift/filetests/isa/riscv/abi.cton +++ b/cranelift/filetests/isa/riscv/abi.cton @@ -2,6 +2,9 @@ test legalizer isa riscv +; regex: V=v\d+ +; regex: VX=vx\d+ + function f(i32) { sig0 = signature(i32) -> i32 @@ -21,3 +24,11 @@ function f(i32) { ebb0(v0: i32): return_reg v0 } + +function int_split_args(i64) -> i64 { +ebb0(v0: i64): +; check: $ebb0($(v0l=$VX): i32, $(v0h=$VX): i32): +; check: iconcat_lohi $v0l, $v0h + v1 = iadd_imm v0, 1 + return v0 +} diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton index 41bc2e2947..fb8d412d8c 100644 --- a/cranelift/filetests/isa/riscv/legalize-i64.cton +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -10,8 +10,8 @@ ebb0(v1: i64, v2: i64): v3 = band v1, v2 return v3 } -; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 -; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 +; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi +; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi ; check: [R#ec ; sameln: $(v3l=$V) = band $v1l, $v2l ; check: [R#ec @@ -23,8 +23,8 @@ ebb0(v1: i64, v2: i64): v3 = bor v1, v2 return v3 } -; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 -; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 +; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi +; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi ; check: [R#cc ; sameln: $(v3l=$V) = bor $v1l, $v2l ; check: [R#cc @@ -36,8 +36,8 @@ ebb0(v1: i64, v2: i64): v3 = bxor v1, v2 return v3 } -; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 -; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 +; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi +; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi ; check: [R#8c ; sameln: $(v3l=$V) = bxor $v1l, $v2l ; check: [R#8c @@ -52,8 +52,8 @@ ebb0(v1: i64, v2: i64): v3 = iadd v1, v2 return v3 } -; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 -; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi $v2 +; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi +; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi ; check: [R#0c ; sameln: $(v3l=$V) = iadd $v1l, $v2l ; check: $(c=$V) = icmp ult, $v3l, $v1l diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index 897133bc76..8d02de4aca 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -3,23 +3,65 @@ //! This module provides functions and data structures that are useful for implementing the //! `TargetIsa::legalize_signature()` method. -use ir::{ArgumentLoc, ArgumentType, Type}; +use ir::{ArgumentLoc, ArgumentType, ArgumentExtension, Type}; +use std::cmp::Ordering; -/// Legalization action to perform on a single argument or return value. +/// Legalization action to perform on a single argument or return value when converting a +/// signature. /// /// An argument may go through a sequence of legalization steps before it reaches the final /// `Assign` action. +#[derive(Clone, Copy)] pub enum ArgAction { /// Assign the argument to the given location. Assign(ArgumentLoc), - /// Split the argument into smaller parts, then call again. + /// Convert the argument, then call again. /// /// This action can split an integer type into two smaller integer arguments, or it can split a /// SIMD vector into halves. - /// - /// Floating point scalar types can't be split. - Split, + Convert(ValueConversion), +} + +/// Legalization action to be applied to a value that is being passed to or from a legalized ABI. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ValueConversion { + /// Split an integer types into low and high parts, using `isplit_lohi`. + IntSplit, + + /// Split a vector type into halves with identical lane types. + VectorSplit, + + /// Bit-cast to an integer type of the same size. + IntBits, + + /// Sign-extend integer value to the required type. + Sext(Type), + + /// Unsigned zero-extend value to the required type. + Uext(Type), +} + +impl ValueConversion { + /// Apply this conversion to a type, return the converted type. + pub fn apply(self, ty: Type) -> Type { + match self { + ValueConversion::IntSplit => ty.half_width().expect("Integer type too small to split"), + ValueConversion::VectorSplit => ty.half_vector().expect("Not a vector"), + ValueConversion::IntBits => Type::int(ty.bits()).expect("Bad integer size"), + ValueConversion::Sext(nty) => nty, + ValueConversion::Uext(nty) => nty, + } + } + + /// Is this a split conversion that results in two arguments? + pub fn is_split(self) -> bool { + match self { + ValueConversion::IntSplit => true, + ValueConversion::VectorSplit => true, + _ => false, + } + } } /// Common trait for assigning arguments to registers or stack locations. @@ -53,20 +95,104 @@ pub fn legalize_args(args: &mut Vec, aa: &mut AA) argno += 1; } // Split this argument into two smaller ones. Then revisit both. - ArgAction::Split => { - let new_arg = ArgumentType { value_type: split_type(arg.value_type), ..arg }; + ArgAction::Convert(conv) => { + let new_arg = ArgumentType { value_type: conv.apply(arg.value_type), ..arg }; args[argno].value_type = new_arg.value_type; - args.insert(argno + 1, new_arg); + if conv.is_split() { + args.insert(argno + 1, new_arg); + } } } } } -/// Given a value type that isn't legal, compute a replacement type half the size. -fn split_type(ty: Type) -> Type { - if ty.is_int() { - ty.half_width().expect("Integer type too small to split") - } else { - ty.half_vector().expect("Can only split integers and vectors") +/// Determine the right action to take when passing a `have` value type to a call signature where +/// the next argument is `arg` which has a different value type. +/// +/// The signature legalization process in `legalize_args` above can replace a single argument value +/// with multiple arguments of smaller types. It can also change the type of an integer argument to +/// a larger integer type, requiring the smaller value to be sign- or zero-extended. +/// +/// The legalizer needs to repair the values at all ABI boundaries: +/// +/// - Incoming function arguments to the entry EBB. +/// - Function arguments passed to a call. +/// - Return values from a call. +/// - Return values passed to a return instruction. +/// +/// The `legalize_abi_value` function helps the legalizer with the process. When the legalizer +/// needs to pass a pre-legalized `have` argument, but the ABI argument `arg` has a different value +/// type, `legalize_abi_value(have, arg)` tells the legalizer how to create the needed value type +/// for the argument. +/// +/// It may be necessary to call `legalize_abi_value` more than once for a given argument before the +/// desired argument type appears. This will happen when a vector or integer type needs to be split +/// more than once, for example. +pub fn legalize_abi_value(have: Type, arg: &ArgumentType) -> ValueConversion { + let have_bits = have.bits(); + let arg_bits = arg.value_type.bits(); + + match have_bits.cmp(&arg_bits) { + // We have fewer bits than the ABI argument. + Ordering::Less => { + assert!(have.is_int() && arg.value_type.is_int(), + "Can only extend integer values"); + match arg.extension { + ArgumentExtension::Uext => ValueConversion::Uext(arg.value_type), + ArgumentExtension::Sext => ValueConversion::Sext(arg.value_type), + _ => panic!("No argument extension specified"), + } + } + // We have the same number of bits as the argument. + Ordering::Equal => { + // This must be an integer vector that is split and then extended. + assert!(arg.value_type.is_int()); + assert!(!have.is_scalar()); + ValueConversion::VectorSplit + } + // We have more bits than the argument. + Ordering::Greater => { + if have.is_scalar() { + if have.is_float() { + // Convert a float to int so it can be split the next time. + // ARM would do this to pass an `f64` in two registers. + ValueConversion::IntBits + } else { + ValueConversion::IntSplit + } + } else { + ValueConversion::VectorSplit + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use ir::types; + use ir::ArgumentType; + + #[test] + fn legalize() { + let mut arg = ArgumentType::new(types::I32); + + assert_eq!(legalize_abi_value(types::I64X2, &arg), + ValueConversion::VectorSplit); + assert_eq!(legalize_abi_value(types::I64, &arg), + ValueConversion::IntSplit); + + // Vector of integers is broken down, then sign-extended. + arg.extension = ArgumentExtension::Sext; + assert_eq!(legalize_abi_value(types::I16X4, &arg), + ValueConversion::VectorSplit); + assert_eq!(legalize_abi_value(types::I16.by(2).unwrap(), &arg), + ValueConversion::VectorSplit); + assert_eq!(legalize_abi_value(types::I16, &arg), + ValueConversion::Sext(types::I32)); + + // 64-bit float is split as an integer. + assert_eq!(legalize_abi_value(types::F64, &arg), + ValueConversion::IntBits); } } diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index cdd2207aad..2391ca6072 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -67,6 +67,17 @@ impl Type { } } + /// Get an integer type with the requested number of bits. + pub fn int(bits: u16) -> Option { + match bits { + 8 => Some(I8), + 16 => Some(I16), + 32 => Some(I32), + 64 => Some(I64), + _ => None, + } + } + /// Get a type with the same number of lanes as this type, but with the lanes replaced by /// booleans of the same size. /// diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index 5c9630f8f4..7813a59cef 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -5,7 +5,7 @@ //! //! This doesn't support the soft-float ABI at the moment. -use abi::{ArgAction, ArgAssigner, legalize_args}; +use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; use ir::{Signature, ArgumentType, ArgumentLoc}; use isa::riscv::registers::{GPR, FPR}; use settings as shared_settings; @@ -39,7 +39,7 @@ impl ArgAssigner for Args { // Check for a legal type. // RISC-V doesn't have SIMD at all, so break all vectors down. if !ty.is_scalar() { - return ArgAction::Split; + return ArgAction::Convert(ValueConversion::VectorSplit); } // Large integers and booleans are broken down to fit in a register. @@ -47,7 +47,7 @@ impl ArgAssigner for Args { // Align registers and stack to a multiple of two pointers. self.regs = align(self.regs, 2); self.offset = align(self.offset, 2 * self.pointer_bytes); - return ArgAction::Split; + return ArgAction::Convert(ValueConversion::IntSplit); } if self.regs < 8 { diff --git a/lib/cretonne/src/legalizer.rs b/lib/cretonne/src/legalizer.rs index 3d57c73b95..885a034084 100644 --- a/lib/cretonne/src/legalizer.rs +++ b/lib/cretonne/src/legalizer.rs @@ -13,7 +13,9 @@ //! The legalizer does not deal with register allocation constraints. These constraints are derived //! from the encoding recipes, and solved later by the register allocator. -use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, InstBuilder}; +use abi::{legalize_abi_value, ValueConversion}; +use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, InstBuilder, Ebb, Type, Value, + ArgumentType}; use ir::condcodes::IntCC; use isa::{TargetIsa, Legalize}; @@ -87,4 +89,115 @@ fn legalize_signatures(func: &mut Function, isa: &TargetIsa) { for sig in func.dfg.signatures.keys() { isa.legalize_signature(&mut func.dfg.signatures[sig]); } + + if let Some(entry) = func.layout.entry_block() { + legalize_entry_arguments(func, entry); + } +} + +/// Legalize the entry block arguments after `func`'s signature has been legalized. +/// +/// The legalized signature may contain more arguments than the original signature, and the +/// argument types have been changed. This function goes through the arguments to the entry EBB and +/// replaces them with arguments of the right type for the ABI. +/// +/// The original entry EBB arguments are computed from the new ABI arguments by code inserted at +/// the top of the entry block. +fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { + // Insert position for argument conversion code. + // We want to insert instructions before the first instruction in the entry block. + // If the entry block is empty, append instructions to it instead. + let mut pos = Cursor::new(&mut func.layout); + pos.goto_top(entry); + pos.next_inst(); + + // Keep track of the argument types in the ABI-legalized signature. + let abi_types = &func.signature.argument_types; + let mut abi_arg = 0; + + // Process the EBB arguments one at a time, possibly replacing one argument with multiple new + // ones. We do this by detaching the entry EBB arguments first. + let mut next_arg = func.dfg.take_ebb_args(entry); + while let Some(arg) = next_arg { + // Get the next argument before we mutate `arg`. + next_arg = func.dfg.next_ebb_arg(arg); + + let arg_type = func.dfg.value_type(arg); + if arg_type == abi_types[abi_arg].value_type { + // No value translation is necessary, this argument matches the ABI type. + // Just use the original EBB argument value. This is the most common case. + func.dfg.put_ebb_arg(entry, arg); + abi_arg += 1; + } else { + // Compute the value we want for `arg` from the legalized ABI arguments. + let converted = convert_from_abi(&mut func.dfg, + &mut pos, + entry, + &mut abi_arg, + abi_types, + arg_type); + // The old `arg` is no longer an attached EBB argument, but there are probably still + // uses of the value. Make it an alias to the converted value. + func.dfg.change_to_alias(arg, converted); + } + } +} + +/// Compute original value of type `ty` from the legalized ABI arguments beginning at `abi_arg`. +/// +/// Update `abi_arg` to reflect the ABI arguments consumed and return the computed value. +fn convert_from_abi(dfg: &mut DataFlowGraph, + pos: &mut Cursor, + entry: Ebb, + abi_arg: &mut usize, + abi_types: &[ArgumentType], + ty: Type) + -> Value { + // Terminate the recursion when we get the desired type. + if ty == abi_types[*abi_arg].value_type { + return dfg.append_ebb_arg(entry, ty); + } + + // Reconstruct how `ty` was legalized into the argument at `abi_arg`. + let conversion = legalize_abi_value(ty, &abi_types[*abi_arg]); + + // The conversion describes value to ABI argument. We implement the reverse conversion here. + match conversion { + // Construct a `ty` by concatenating two ABI integers. + ValueConversion::IntSplit => { + let abi_ty = ty.half_width().expect("Invalid type for conversion"); + let lo = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + let hi = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + dfg.ins(pos).iconcat_lohi(lo, hi) + } + // Construct a `ty` by concatenating two halves of a vector. + ValueConversion::VectorSplit => { + let abi_ty = ty.half_vector().expect("Invalid type for conversion"); + let _lo = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + let _hi = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + unimplemented!() + } + // Construct a `ty` by bit-casting from an integer type. + ValueConversion::IntBits => { + assert!(!ty.is_int()); + let abi_ty = Type::int(ty.bits()).expect("Invalid type for conversion"); + let arg = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + dfg.ins(pos).bitcast(ty, arg) + } + // ABI argument is a sign-extended version of the value we want. + ValueConversion::Sext(abi_ty) => { + let arg = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + // TODO: Currently, we don't take advantage of the ABI argument being sign-extended. + // We could insert an `assert_sreduce` which would fold with a following `sextend` of + // this value. + dfg.ins(pos).ireduce(ty, arg) + } + ValueConversion::Uext(abi_ty) => { + let arg = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + // TODO: Currently, we don't take advantage of the ABI argument being sign-extended. + // We could insert an `assert_ureduce` which would fold with a following `uextend` of + // this value. + dfg.ins(pos).ireduce(ty, arg) + } + } } From fd58b7cc293c95627c076fa177a9f84b96fbe5dc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 7 Mar 2017 14:15:55 -0800 Subject: [PATCH 0551/3084] Add vsplit and vconcat instructions. Add support for two new type variable functions: half_vector() and double_vector(). Use these two instructions to break down unsupported SIMD types and build them up again. --- cranelift/docs/langref.rst | 9 ++----- cranelift/filetests/isa/riscv/abi.cton | 22 +++++++++++++++++ lib/cretonne/meta/base/instructions.py | 34 ++++++++++++++++++++++++++ lib/cretonne/meta/cdsl/typevar.py | 26 ++++++++++++++++++++ lib/cretonne/src/ir/instructions.rs | 8 ++++++ lib/cretonne/src/ir/types.rs | 2 ++ lib/cretonne/src/legalizer.rs | 6 ++--- 7 files changed, 97 insertions(+), 10 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 23cc225d7f..5fb42c44a2 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -668,14 +668,9 @@ allocation pass and beyond. Vector operations ----------------- +.. autoinst:: vsplit +.. autoinst:: vconcat .. autoinst:: vselect - -.. inst:: a = vbuild x, y, z, ... - - Vector build. - - Build a vector value from the provided lanes. - .. autoinst:: splat .. autoinst:: insertlane .. autoinst:: extractlane diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.cton index e3bafb83e1..bb27ac20ae 100644 --- a/cranelift/filetests/isa/riscv/abi.cton +++ b/cranelift/filetests/isa/riscv/abi.cton @@ -21,6 +21,14 @@ function f(i32) { sig3 = signature(f64, f64, f64, f64, f64, f64, f64, i64) -> f64 ; check: sig3 = signature(f64 [%f10], f64 [%f11], f64 [%f12], f64 [%f13], f64 [%f14], f64 [%f15], f64 [%f16], i32 [0], i32 [4]) -> f64 [%f10] +; Splitting vectors. + sig4 = signature(i32x4) +; check: sig4 = signature(i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13]) + +; Splitting vectors, then splitting ints. + sig5 = signature(i64x4) +; check: sig5 = signature(i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [%x16], i32 [%x17]) + ebb0(v0: i32): return_reg v0 } @@ -32,3 +40,17 @@ ebb0(v0: i64): v1 = iadd_imm v0, 1 return v0 } + +function vector_split_args(i64x4) -> i64 { +ebb0(v0: i64x4): +; check: $ebb0($(v0al=$VX): i32, $(v0ah=$VX): i32, $(v0bl=$VX): i32, $(v0bh=$VX): i32, $(v0cl=$VX): i32, $(v0ch=$VX): i32, $(v0dl=$VX): i32, $(v0dh=$VX): i32): +; check: $(v0a=$V) = iconcat_lohi $v0al, $v0ah +; check: $(v0b=$V) = iconcat_lohi $v0bl, $v0bh +; check: $(v0ab=$V) = vconcat $v0a, $v0b +; check: $(v0c=$V) = iconcat_lohi $v0cl, $v0ch +; check: $(v0d=$V) = iconcat_lohi $v0dl, $v0dh +; check: $(v0cd=$V) = vconcat $v0c, $v0d +; check: $(v0abcd=$V) = vconcat $v0ab, $v0cd + v1 = iadd v0, v0 + return v0 +} diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 9cec7d9d58..09cd2c75c7 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -264,6 +264,40 @@ fill = Instruction( # Vector operations # +x = Operand('x', TxN, doc='Vector to split') +lo = Operand('lo', TxN.half_vector(), doc='Low-numbered lanes of `x`') +hi = Operand('hi', TxN.half_vector(), doc='High-numbered lanes of `x`') + +vsplit = Instruction( + 'vsplit', r""" + Split a vector into two halves. + + Split the vector `x` into two separate values, each containing half of + the lanes from ``x``. The result may be two scalars if ``x`` only had + two lanes. + """, + ins=x, outs=(lo, hi)) + +Any128 = TypeVar( + 'Any128', 'Any scalar or vector type with as most 128 lanes', + ints=True, floats=True, bools=True, scalars=True, simd=(1, 128)) +x = Operand('x', Any128, doc='Low-numbered lanes') +y = Operand('y', Any128, doc='High-numbered lanes') +a = Operand('a', Any128.double_vector(), doc='Concatenation of `x` and `y`') + +vconcat = Instruction( + 'vconcat', r""" + Vector concatenation. + + Return a vector formed by concatenating ``x`` and ``y``. The resulting + vector type has twice as many lanes as each of the inputs. The lanes of + ``x`` appear as the low-numbered lanes, and the lanes of ``y`` become + the high-numbered lanes of ``a``. + + It is possible to form a vector by concatenating two scalars. + """, + ins=(x, y), outs=a) + c = Operand('c', TxN.as_bool(), doc='Controlling vector') x = Operand('x', TxN, doc='Value to use where `c` is true') y = Operand('y', TxN, doc='Value to use where `c` is false') diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 580d367f51..908ebbd8cd 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -315,6 +315,8 @@ class TypeVar(object): ASBOOL = 'as_bool' HALFWIDTH = 'half_width' DOUBLEWIDTH = 'double_width' + HALFVECTOR = 'half_vector' + DOUBLEVECTOR = 'double_vector' @staticmethod def derived(base, derived_func): @@ -396,6 +398,30 @@ class TypeVar(object): return TypeVar.derived(self, self.DOUBLEWIDTH) + def half_vector(self): + # type: () -> TypeVar + """ + Return a derived type variable that has half the number of vector lanes + as this one, with the same lane type. + """ + if not self.is_derived: + ts = self.type_set + assert ts.min_lanes > 1, "Can't halve a scalar type" + + return TypeVar.derived(self, self.HALFVECTOR) + + def double_vector(self): + # type: () -> TypeVar + """ + Return a derived type variable that has twice the number of vector + lanes as this one, with the same lane type. + """ + if not self.is_derived: + ts = self.type_set + assert ts.max_lanes < 256, "Can't double 256 lanes." + + return TypeVar.derived(self, self.DOUBLEVECTOR) + def free_typevar(self): # type: () -> TypeVar """ diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index c4ac3b0c20..2a956ffc86 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -708,6 +708,12 @@ enum OperandConstraint { /// This operand is `ctrlType.double_width()`. DoubleWidth, + + /// This operand is `ctrlType.half_vector()`. + HalfVector, + + /// This operand is `ctrlType.double_vector()`. + DoubleVector, } impl OperandConstraint { @@ -725,6 +731,8 @@ impl OperandConstraint { AsBool => Some(ctrl_type.as_bool()), HalfWidth => Some(ctrl_type.half_width().expect("invalid type for half_width")), DoubleWidth => Some(ctrl_type.double_width().expect("invalid type for double_width")), + HalfVector => Some(ctrl_type.half_vector().expect("invalid type for half_vector")), + DoubleVector => Some(ctrl_type.by(2).expect("invalid type for double_vector")), } } } diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 2391ca6072..550f25804b 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -217,6 +217,8 @@ impl Type { } /// Get a SIMD vector with half the number of lanes. + /// + /// There is no `double_vector()` method. Use `t.by(2)` instead. pub fn half_vector(self) -> Option { if self.is_scalar() { None diff --git a/lib/cretonne/src/legalizer.rs b/lib/cretonne/src/legalizer.rs index 885a034084..9223a174b0 100644 --- a/lib/cretonne/src/legalizer.rs +++ b/lib/cretonne/src/legalizer.rs @@ -173,9 +173,9 @@ fn convert_from_abi(dfg: &mut DataFlowGraph, // Construct a `ty` by concatenating two halves of a vector. ValueConversion::VectorSplit => { let abi_ty = ty.half_vector().expect("Invalid type for conversion"); - let _lo = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); - let _hi = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); - unimplemented!() + let lo = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + let hi = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + dfg.ins(pos).vconcat(lo, hi) } // Construct a `ty` by bit-casting from an integer type. ValueConversion::IntBits => { From 40fc5da3d85e44f7324309e4f9dc9bf86d76aac4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 7 Mar 2017 15:07:00 -0800 Subject: [PATCH 0552/3084] Heed uext and sext annotations on RISC-V arguments. Translate the small integer arguments to i32 or i64 with the appropriate extend and ireduce instructions. --- cranelift/filetests/isa/riscv/abi.cton | 7 +++++++ lib/cretonne/src/isa/riscv/abi.rs | 17 ++++++++++++++++- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.cton index bb27ac20ae..0282e94874 100644 --- a/cranelift/filetests/isa/riscv/abi.cton +++ b/cranelift/filetests/isa/riscv/abi.cton @@ -41,6 +41,13 @@ ebb0(v0: i64): return v0 } +function int_ext(i8, i8 sext, i8 uext) -> i8 { +ebb0(v1: i8, v2: i8, v3: i8): +; check: $ebb0($v1: i8, $(v2x=$VX): i32, $(v3x=$VX): i32): +; check: ireduce.i8 $v2x +; check: ireduce.i8 $v3x +} + function vector_split_args(i64x4) -> i64 { ebb0(v0: i64x4): ; check: $ebb0($(v0al=$VX): i32, $(v0ah=$VX): i32, $(v0bl=$VX): i32, $(v0bh=$VX): i32, $(v0cl=$VX): i32, $(v0ch=$VX): i32, $(v0dl=$VX): i32, $(v0dh=$VX): i32): diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index 7813a59cef..9d78abc007 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -6,13 +6,14 @@ //! This doesn't support the soft-float ABI at the moment. use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; -use ir::{Signature, ArgumentType, ArgumentLoc}; +use ir::{Signature, Type, ArgumentType, ArgumentLoc, ArgumentExtension}; use isa::riscv::registers::{GPR, FPR}; use settings as shared_settings; struct Args { pointer_bits: u16, pointer_bytes: u32, + pointer_type: Type, regs: u32, offset: u32, } @@ -22,6 +23,7 @@ impl Args { Args { pointer_bits: bits, pointer_bytes: bits as u32 / 8, + pointer_type: Type::int(bits).unwrap(), regs: 0, offset: 0, } @@ -50,6 +52,19 @@ impl ArgAssigner for Args { return ArgAction::Convert(ValueConversion::IntSplit); } + // Small integers are extended to the size of a pointer register. + if ty.is_int() && ty.bits() < self.pointer_bits { + match arg.extension { + ArgumentExtension::None => {} + ArgumentExtension::Uext => { + return ArgAction::Convert(ValueConversion::Uext(self.pointer_type)) + } + ArgumentExtension::Sext => { + return ArgAction::Convert(ValueConversion::Sext(self.pointer_type)) + } + } + } + if self.regs < 8 { // Assign to a register. let reg = if ty.is_float() { From 2de210ddb61d44bc3916927c8e0fbd6b980d246a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 7 Mar 2017 15:13:55 -0800 Subject: [PATCH 0553/3084] Implement From traits on ArgAction for convenience. --- lib/cretonne/src/abi.rs | 12 ++++++++++++ lib/cretonne/src/isa/riscv/abi.rs | 16 ++++++---------- 2 files changed, 18 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index 8d02de4aca..d011311e7c 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -23,6 +23,18 @@ pub enum ArgAction { Convert(ValueConversion), } +impl From for ArgAction { + fn from(x: ArgumentLoc) -> ArgAction { + ArgAction::Assign(x) + } +} + +impl From for ArgAction { + fn from(x: ValueConversion) -> ArgAction { + ArgAction::Convert(x) + } +} + /// Legalization action to be applied to a value that is being passed to or from a legalized ABI. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum ValueConversion { diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index 9d78abc007..fa158dceb5 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -41,7 +41,7 @@ impl ArgAssigner for Args { // Check for a legal type. // RISC-V doesn't have SIMD at all, so break all vectors down. if !ty.is_scalar() { - return ArgAction::Convert(ValueConversion::VectorSplit); + return ValueConversion::VectorSplit.into(); } // Large integers and booleans are broken down to fit in a register. @@ -49,19 +49,15 @@ impl ArgAssigner for Args { // Align registers and stack to a multiple of two pointers. self.regs = align(self.regs, 2); self.offset = align(self.offset, 2 * self.pointer_bytes); - return ArgAction::Convert(ValueConversion::IntSplit); + return ValueConversion::IntSplit.into(); } // Small integers are extended to the size of a pointer register. if ty.is_int() && ty.bits() < self.pointer_bits { match arg.extension { ArgumentExtension::None => {} - ArgumentExtension::Uext => { - return ArgAction::Convert(ValueConversion::Uext(self.pointer_type)) - } - ArgumentExtension::Sext => { - return ArgAction::Convert(ValueConversion::Sext(self.pointer_type)) - } + ArgumentExtension::Uext => return ValueConversion::Uext(self.pointer_type).into(), + ArgumentExtension::Sext => return ValueConversion::Sext(self.pointer_type).into(), } } @@ -73,12 +69,12 @@ impl ArgAssigner for Args { GPR.unit(10 + self.regs as usize) }; self.regs += 1; - ArgAction::Assign(ArgumentLoc::Reg(reg)) + ArgumentLoc::Reg(reg).into() } else { // Assign a stack location. let loc = ArgumentLoc::Stack(self.offset); self.offset += self.pointer_bytes; - ArgAction::Assign(loc) + loc.into() } } } From 5ac57220bee89e4921d57e5a23ab4bed63ee98cc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 8 Mar 2017 12:14:27 -0800 Subject: [PATCH 0554/3084] Strip trailing white space from regex: directives. This was particularly confusing when reading a file with Windows line endings. The CR would become part of the regex. --- lib/filecheck/src/checker.rs | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs index 2474742fb1..3101865e71 100644 --- a/lib/filecheck/src/checker.rs +++ b/lib/filecheck/src/checker.rs @@ -71,7 +71,8 @@ impl Directive { var, rest))); } - Ok(Directive::Regex(var, rest[varlen + 1..].to_string())) + // Ignore trailing white space in the regex, including CR. + Ok(Directive::Regex(var, rest[varlen + 1..].trim_right().to_string())) } } @@ -410,10 +411,12 @@ mod tests { Ok(true)); assert_eq!(b.directive("[x86]sameln: $x $(y=[^]]*) there").map_err(e2s), Ok(true)); + // Windows line ending sneaking in. + assert_eq!(b.directive("regex: Y=foo\r").map_err(e2s), Ok(true)); let c = b.finish(); assert_eq!(c.to_string(), "#0 regex: X=more text\n#1 not: patt $(x) $(y) here\n#2 sameln: $(x) \ - $(y=[^]]*) there\n"); + $(y=[^]]*) there\n#3 regex: Y=foo\n"); } } From a58ccb112524802c82de74cb3da594ebd9caf6ee Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Fri, 3 Mar 2017 00:06:52 +0000 Subject: [PATCH 0555/3084] Added parsing of instruction encodings and result registers specifications. --- lib/reader/src/parser.rs | 145 +++++++++++++++++++++++++++++++++++---- 1 file changed, 132 insertions(+), 13 deletions(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index bd6fd8fce8..f2061afec7 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -6,7 +6,7 @@ // ====--------------------------------------------------------------------------------------====// use std::str::FromStr; -use std::u32; +use std::{u16, u32}; use std::mem; use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotData, JumpTable, JumpTableData, Signature, ArgumentType, ArgumentExtension, ExtFuncData, SigRef, @@ -17,7 +17,7 @@ use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, TernaryOverflowData, JumpData, BranchData, CallData, IndirectCallData, ReturnData, ReturnRegData}; -use cretonne::isa; +use cretonne::isa::{self, TargetIsa, Encoding}; use cretonne::settings; use testfile::{TestFile, Details, Comment}; use error::{Location, Error, Result}; @@ -40,11 +40,26 @@ pub fn parse_test<'a>(text: &'a str) -> Result> { let mut parser = Parser::new(text); // Gather the preamble comments as 'Function'. parser.gather_comments(AnyEntity::Function); + + let commands = parser.parse_test_commands(); + let isa_spec = parser.parse_isa_specs()?; + let preamble_comments = parser.take_comments(); + + let functions = { + let mut unique_isa = None; + if let isaspec::IsaSpec::Some(ref isa_vec) = isa_spec { + if isa_vec.len() == 1 { + unique_isa = Some(&*isa_vec[0]); + } + } + parser.parse_function_list(unique_isa)? + }; + Ok(TestFile { - commands: parser.parse_test_commands(), - isa_spec: parser.parse_isa_specs()?, - preamble_comments: parser.take_comments(), - functions: parser.parse_function_list()?, + commands: commands, + isa_spec: isa_spec, + preamble_comments: preamble_comments, + functions: functions, }) } @@ -71,16 +86,35 @@ pub struct Parser<'a> { // // Many entities like values, stack slots, and function signatures are referenced in the `.cton` // file by number. We need to map these numbers to real references. -struct Context { +struct Context<'a> { function: Function, map: SourceMap, + + // Reference to the unique_isa for things like parsing ISA-specific instruction encoding + // information. This is only `Some` if exactly one set of `isa` directives were found in the + // prologue (it is valid to have directives for multiple different ISAs, but in that case we + // couldn't know which ISA the provided encodings are intended for) + unique_isa: Option<&'a TargetIsa>, } -impl Context { - fn new(f: Function) -> Context { +impl<'a> Context<'a> { + fn new(f: Function, unique_isa: Option<&'a TargetIsa>) -> Context<'a> { Context { function: f, map: SourceMap::new(), + unique_isa: unique_isa + } + } + + // Get the index of a recipe name if it exists. + fn find_recipe_index(&self, recipe_name: &str) -> Option { + if let Some(unique_isa) = self.unique_isa { + unique_isa.recipe_names() + .iter() + .position(|&name| name == recipe_name) + .map(|idx| idx as u16) + } else { + None } } @@ -454,6 +488,41 @@ impl<'a> Parser<'a> { } } + // Match and consume an identifier. + fn match_any_identifier(&mut self, err_msg: &str) -> Result<&'a str> { + if let Some(Token::Identifier(text)) = self.token() { + self.consume(); + Ok(text) + } else { + err!(self.loc, err_msg) + } + } + + // Match and consume a HexSequence that fits into a u16. + // This is used for instruction encodings. + fn match_hex16(&mut self, err_msg: &str) -> Result { + if let Some(Token::HexSequence(bits_str)) = self.token() { + self.consume(); + // The only error we anticipate from this parse is overflow, the lexer should + // already have ensured that the string doesn't contain invalid characters, and + // isn't empty or negative. + u16::from_str_radix(bits_str, 16) + .map_err(|_| self.error("the hex sequence given overflows the u16 type")) + } else { + err!(self.loc, err_msg) + } + } + + // Match and consume a name. + fn match_name(&mut self, err_msg: &str) -> Result<&'a str> { + if let Some(Token::Name(name)) = self.token() { + self.consume(); + Ok(name) + } else { + err!(self.loc, err_msg) + } + } + /// Parse a list of test commands. pub fn parse_test_commands(&mut self) -> Vec> { let mut list = Vec::new(); @@ -523,10 +592,11 @@ impl<'a> Parser<'a> { /// Parse a list of function definitions. /// /// This is the top-level parse function matching the whole contents of a file. - pub fn parse_function_list(&mut self) -> Result)>> { + pub fn parse_function_list(&mut self, unique_isa: Option<&TargetIsa>) + -> Result)>> { let mut list = Vec::new(); while self.token().is_some() { - list.push(self.parse_function()?); + list.push(self.parse_function(unique_isa)?); } Ok(list) } @@ -535,7 +605,7 @@ impl<'a> Parser<'a> { // // function ::= * function-spec "{" preamble function-body "}" // - fn parse_function(&mut self) -> Result<(Function, Details<'a>)> { + fn parse_function(&mut self, unique_isa: Option<&TargetIsa>) -> Result<(Function, Details<'a>)> { // Begin gathering comments. // Make sure we don't include any comments before the `function` keyword. self.token(); @@ -543,7 +613,7 @@ impl<'a> Parser<'a> { self.gather_comments(AnyEntity::Function); let (location, name, sig) = self.parse_function_spec()?; - let mut ctx = Context::new(Function::with_name_signature(name, sig)); + let mut ctx = Context::new(Function::with_name_signature(name, sig), unique_isa); // function ::= function-spec * "{" preamble function-body "}" self.match_token(Token::LBrace, "expected '{' before function body")?; @@ -894,6 +964,42 @@ impl<'a> Parser<'a> { ctx.map.def_value(vx, value, &vx_location) } + fn parse_instruction_encoding(&mut self, ctx: &Context) + -> Result<(Option, Option>)> { + let (mut encoding, mut result_registers) = (None, None); + + // encoding ::= "[" encoding_literal result_registers "]" + if self.optional(Token::LBracket) { + // encoding_literal ::= "-" | Identifier HexSequence + if !self.optional(Token::Minus) { + let recipe = self.match_any_identifier("expected instruction encoding or '-'")?; + let bits = self.match_hex16("expected a hex sequence")?; + + if let Some(recipe_index) = ctx.find_recipe_index(recipe) { + encoding = Some(Encoding::new(recipe_index, bits)); + } + } + + // result_registers ::= ("," ( "-" | names ) )? + // names ::= Name { "," Name } + if self.optional(Token::Comma) && !self.optional(Token::Minus) { + let mut result = Vec::new(); + + result.push(self.match_name("expected register")?); + while self.optional(Token::Comma) { + result.push(self.match_name("expected register")?); + } + + result_registers = Some(result); + } + + self.match_token(Token::RBracket, + "expected ']' to terminate instruction encoding")?; + } + + Ok((encoding, result_registers)) + } + // Parse an instruction, append it to `ebb`. // // instruction ::= [inst-results "="] Opcode(opc) ["." Type] ... @@ -903,6 +1009,8 @@ impl<'a> Parser<'a> { // Collect comments for the next instruction to be allocated. self.gather_comments(ctx.function.dfg.next_inst()); + let (encoding, result_registers) = self.parse_instruction_encoding(ctx)?; + // Result value numbers. let mut results = Vec::new(); @@ -921,6 +1029,13 @@ impl<'a> Parser<'a> { self.match_token(Token::Equal, "expected '=' before opcode")?; } + if let Some(ref result_registers) = result_registers { + if result_registers.len() != results.len() { + return err!(self.loc, + "must have same number of result registers as results"); + } + } + // instruction ::= [inst-results "="] * Opcode(opc) ["." Type] ... let opcode = if let Some(Token::Identifier(text)) = self.token() { match text.parse() { @@ -955,6 +1070,10 @@ impl<'a> Parser<'a> { ctx.function.layout.append_inst(inst, ebb); ctx.map.def_entity(inst.into(), &opcode_loc).expect("duplicate inst references created"); + if let Some(encoding) = encoding { + *ctx.function.encodings.ensure(inst) = encoding; + } + if results.len() != num_results { return err!(self.loc, "instruction produces {} result values, {} given", From 4525929df26f9ad29a663e2ce7eda70968a30c20 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Fri, 3 Mar 2017 21:30:33 +0000 Subject: [PATCH 0556/3084] Added tests, some refactoring, fixed a parsing bug. --- .../parser/instruction_encoding.cton | 25 +++++++++++ lib/reader/src/isaspec.rs | 12 ++++++ lib/reader/src/parser.rs | 41 ++++++++++--------- 3 files changed, 58 insertions(+), 20 deletions(-) create mode 100644 cranelift/filetests/parser/instruction_encoding.cton diff --git a/cranelift/filetests/parser/instruction_encoding.cton b/cranelift/filetests/parser/instruction_encoding.cton new file mode 100644 index 0000000000..49c1913bcf --- /dev/null +++ b/cranelift/filetests/parser/instruction_encoding.cton @@ -0,0 +1,25 @@ +test cat + +isa riscv + +; regex: WS=[ \t]* + +function foo(i32, i32) { +ebb1(v0: i32, v1: i32): + [-,-] v2 = iadd v0, v1 + [-] trap + [R#1234, %x5, %x11] v6, v7 = iadd_cout v2, v0 + [Rshamt#beef, %x25] v8 = ishl_imm v6, 2 + v9 = iadd v8, v7 + [Iret#5] return v0, v8 +} +; sameln: function foo(i32, i32) { +; nextln: $ebb1($v0: i32, $v1: i32): +; nextln: [-]$WS $v2 = iadd $v0, $v1 +; nextln: [-]$WS trap +; nextln: [0#1234]$WS $v6, $v7 = iadd_cout $v2, $v0 +; TODO Add the full encoding information available: instruction recipe name and architectural registers if specified +; nextln: [2#beef]$WS $v8 = ishl_imm $v6, 2 +; nextln: [-]$WS $v9 = iadd $v8, $v7 +; nextln: [3#05]$WS return $v0, $v8 +; nextln: } \ No newline at end of file diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs index 706c081e3a..b7884bc42c 100644 --- a/lib/reader/src/isaspec.rs +++ b/lib/reader/src/isaspec.rs @@ -22,6 +22,18 @@ pub enum IsaSpec { Some(Vec>), } +impl IsaSpec { + /// If the `IsaSpec` contains exactly 1 `TargetIsa` we return a reference to it + pub fn unique_isa(&self) -> Option<&TargetIsa> { + if let &IsaSpec::Some(ref isa_vec) = self { + if isa_vec.len() == 1 { + return Some(&*isa_vec[0]); + } + } + None + } +} + /// Parse an iterator of command line options and apply them to `config`. pub fn parse_options<'a, I>(iter: I, config: &mut Configurable, loc: &Location) -> Result<()> where I: Iterator diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index f2061afec7..1c71f9f4f5 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -44,16 +44,7 @@ pub fn parse_test<'a>(text: &'a str) -> Result> { let commands = parser.parse_test_commands(); let isa_spec = parser.parse_isa_specs()?; let preamble_comments = parser.take_comments(); - - let functions = { - let mut unique_isa = None; - if let isaspec::IsaSpec::Some(ref isa_vec) = isa_spec { - if isa_vec.len() == 1 { - unique_isa = Some(&*isa_vec[0]); - } - } - parser.parse_function_list(unique_isa)? - }; + let functions = parser.parse_function_list(isa_spec.unique_isa())?; Ok(TestFile { commands: commands, @@ -102,7 +93,7 @@ impl<'a> Context<'a> { Context { function: f, map: SourceMap::new(), - unique_isa: unique_isa + unique_isa: unique_isa, } } @@ -592,8 +583,9 @@ impl<'a> Parser<'a> { /// Parse a list of function definitions. /// /// This is the top-level parse function matching the whole contents of a file. - pub fn parse_function_list(&mut self, unique_isa: Option<&TargetIsa>) - -> Result)>> { + pub fn parse_function_list(&mut self, + unique_isa: Option<&TargetIsa>) + -> Result)>> { let mut list = Vec::new(); while self.token().is_some() { list.push(self.parse_function(unique_isa)?); @@ -605,7 +597,9 @@ impl<'a> Parser<'a> { // // function ::= * function-spec "{" preamble function-body "}" // - fn parse_function(&mut self, unique_isa: Option<&TargetIsa>) -> Result<(Function, Details<'a>)> { + fn parse_function(&mut self, + unique_isa: Option<&TargetIsa>) + -> Result<(Function, Details<'a>)> { // Begin gathering comments. // Make sure we don't include any comments before the `function` keyword. self.token(); @@ -916,6 +910,7 @@ impl<'a> Parser<'a> { while match self.token() { Some(Token::Value(_)) => true, Some(Token::Identifier(_)) => true, + Some(Token::LBracket) => true, _ => false, } { self.parse_instruction(ctx, ebb)?; @@ -964,8 +959,9 @@ impl<'a> Parser<'a> { ctx.map.def_value(vx, value, &vx_location) } - fn parse_instruction_encoding(&mut self, ctx: &Context) - -> Result<(Option, Option>)> { + fn parse_instruction_encoding(&mut self, + ctx: &Context) + -> Result<(Option, Option>)> { let (mut encoding, mut result_registers) = (None, None); // encoding ::= "[" encoding_literal result_registers "]" @@ -977,6 +973,11 @@ impl<'a> Parser<'a> { if let Some(recipe_index) = ctx.find_recipe_index(recipe) { encoding = Some(Encoding::new(recipe_index, bits)); + } else if ctx.unique_isa.is_some() { + return err!(self.loc, "invalid instruction recipe"); + } else { + return err!(self.loc, + "provided instruction encoding for unspecified ISA"); } } @@ -1525,7 +1526,7 @@ mod tests { ss3 = stack_slot 13 ss1 = stack_slot 1 }") - .parse_function() + .parse_function(None) .unwrap(); assert_eq!(func.name.to_string(), "foo"); let mut iter = func.stack_slots.keys(); @@ -1542,7 +1543,7 @@ mod tests { ss1 = stack_slot 13 ss1 = stack_slot 1 }") - .parse_function() + .parse_function(None) .unwrap_err() .to_string(), "3: duplicate stack slot: ss1"); @@ -1554,7 +1555,7 @@ mod tests { ebb0: ebb4(vx3: i32): }") - .parse_function() + .parse_function(None) .unwrap(); assert_eq!(func.name.to_string(), "ebbs"); @@ -1583,7 +1584,7 @@ mod tests { trap ; Instruction } ; Trailing. ; More trailing.") - .parse_function() + .parse_function(None) .unwrap(); assert_eq!(func.name.to_string(), "comment"); assert_eq!(comments.len(), 8); // no 'before' comment. From cbbf5cc88b8dbb8fce20366a11f5cfdab03d5cf8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 8 Mar 2017 12:50:43 -0800 Subject: [PATCH 0557/3084] Use a unique ISA in 'test cat' file tests. Add a Function::display() method which can include ISA-specific information when printing the function. If a test file has a unique ISA, use that in the `test cat` implementation. --- .../filetests/parser/instruction_encoding.cton | 8 ++++---- cranelift/src/cat.rs | 2 +- cranelift/src/filetest/runone.rs | 5 ++++- cranelift/src/filetest/subtest.rs | 6 +++--- lib/cretonne/src/ir/function.rs | 16 +++++++++++++++- 5 files changed, 27 insertions(+), 10 deletions(-) diff --git a/cranelift/filetests/parser/instruction_encoding.cton b/cranelift/filetests/parser/instruction_encoding.cton index 49c1913bcf..0e40d0b2f1 100644 --- a/cranelift/filetests/parser/instruction_encoding.cton +++ b/cranelift/filetests/parser/instruction_encoding.cton @@ -17,9 +17,9 @@ ebb1(v0: i32, v1: i32): ; nextln: $ebb1($v0: i32, $v1: i32): ; nextln: [-]$WS $v2 = iadd $v0, $v1 ; nextln: [-]$WS trap -; nextln: [0#1234]$WS $v6, $v7 = iadd_cout $v2, $v0 +; nextln: [R#1234]$WS $v6, $v7 = iadd_cout $v2, $v0 ; TODO Add the full encoding information available: instruction recipe name and architectural registers if specified -; nextln: [2#beef]$WS $v8 = ishl_imm $v6, 2 +; nextln: [Rshamt#beef]$WS $v8 = ishl_imm $v6, 2 ; nextln: [-]$WS $v9 = iadd $v8, $v7 -; nextln: [3#05]$WS return $v0, $v8 -; nextln: } \ No newline at end of file +; nextln: [Iret#05]$WS return $v0, $v8 +; nextln: } diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs index 443c37dadc..fd49722ec0 100644 --- a/cranelift/src/cat.rs +++ b/cranelift/src/cat.rs @@ -61,6 +61,6 @@ impl SubTest for TestCat { } fn run(&self, func: Cow, context: &Context) -> STResult<()> { - subtest::run_filecheck(&func.to_string(), context) + subtest::run_filecheck(&func.display(context.isa).to_string(), context) } } diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index 06e4cfce6b..2067a245b9 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -90,7 +90,10 @@ fn test_tuples<'a>(tests: &'a [Box], } } } else { - out.push((&**test, no_isa_flags, None)); + // This test doesn't require an ISA, and we only want to run one instance of it. + // Still, give it an ISA ref if we happen to have a unique one. + // For example, `test cat` can use this to print encodings and register names. + out.push((&**test, no_isa_flags, isa_spec.unique_isa())); } } Ok(out) diff --git a/cranelift/src/filetest/subtest.rs b/cranelift/src/filetest/subtest.rs index 8ba3528a6c..170490fb3a 100644 --- a/cranelift/src/filetest/subtest.rs +++ b/cranelift/src/filetest/subtest.rs @@ -10,7 +10,7 @@ use filecheck::{self, CheckerBuilder, Checker, Value as FCValue}; pub type Result = result::Result; -/// Context for running a a test on a single function. +/// Context for running a test on a single function. pub struct Context<'a> { /// Comments from the preamble f the test file. These apply to all functions. pub preamble_comments: &'a [Comment<'a>], @@ -24,8 +24,8 @@ pub struct Context<'a> { /// ISA-independent flags for this test. pub flags: &'a Flags, - /// Target ISA to test against. Only present for sub-tests whose `needs_isa` method returned - /// true. + /// Target ISA to test against. Only guaranteed to be present for sub-tests whose `needs_isa` + /// method returned `true`. For other sub-tests, this is set if the test file has a unique ISA. pub isa: Option<&'a TargetIsa>, } diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index b0115a70e3..1abf9c4b35 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -6,7 +6,7 @@ use std::fmt::{self, Display, Debug, Formatter}; use ir::{FunctionName, Signature, Value, Inst, StackSlot, StackSlotData, JumpTable, JumpTableData, ValueLoc, DataFlowGraph, Layout}; -use isa::Encoding; +use isa::{TargetIsa, Encoding}; use entity_map::{EntityMap, PrimaryEntityData}; use write::write_function; @@ -64,6 +64,20 @@ impl Function { pub fn new() -> Function { Self::with_name_signature(FunctionName::default(), Signature::new()) } + + /// Return an object that can display this function with correct ISA-specific annotations. + pub fn display<'a, I: Into>>(&'a self, isa: I) -> DisplayFunction<'a> { + DisplayFunction(self, isa.into()) + } +} + +/// Wrapper type capable of displaying a `Function` with correct ISA annotations. +pub struct DisplayFunction<'a>(&'a Function, Option<&'a TargetIsa>); + +impl<'a> Display for DisplayFunction<'a> { + fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { + write_function(fmt, self.0, self.1) + } } impl Display for Function { From ecc46e56e11da2a701b488962426cc6700ba8ca4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 8 Mar 2017 14:48:50 -0800 Subject: [PATCH 0558/3084] Add is_call and is_return instruction attributes. --- lib/cretonne/meta/base/instructions.py | 10 ++++------ lib/cretonne/meta/cdsl/instructions.py | 18 +++++++++++++++--- lib/cretonne/meta/gen_instr.py | 26 ++++---------------------- 3 files changed, 23 insertions(+), 31 deletions(-) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 09cd2c75c7..a61f98cefe 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -111,7 +111,7 @@ x_return = Instruction( provided return values. The list of return values must match the function signature's return types. """, - ins=rvals, is_terminator=True) + ins=rvals, is_return=True, is_terminator=True) raddr = Operand('raddr', iAddr, doc='Return address') @@ -130,7 +130,7 @@ return_reg = Instruction( :inst:`return` will be legalized into this instruction on these architectures. """, - ins=(raddr, rvals), is_terminator=True) + ins=(raddr, rvals), is_return=True, is_terminator=True) FN = Operand( 'FN', @@ -145,8 +145,7 @@ call = Instruction( Call a function which has been declared in the preamble. The argument types must match the function's signature. """, - ins=(FN, args), - outs=rvals) + ins=(FN, args), outs=rvals, is_call=True) SIG = Operand('SIG', entities.sig_ref, doc='function signature') callee = Operand('callee', iAddr, doc='address of function to call') @@ -158,8 +157,7 @@ call_indirect = Instruction( Call the function pointed to by `callee` with the given arguments. The called function must match the specified signature. """, - ins=(SIG, callee, args), - outs=rvals) + ins=(SIG, callee, args), outs=rvals, is_call=True) # # Materializing constants. diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 4040e5338b..fcce1684fd 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -80,9 +80,22 @@ class Instruction(object): values or `variable_args`. :param is_terminator: This is a terminator instruction. :param is_branch: This is a branch instruction. + :param is_call: This is a call instruction. + :param is_return: This is a return instruction. :param can_trap: This instruction can trap. """ + # Boolean instruction attributes that can be passed as keyword arguments to + # the constructor. Map attribute name to doc comment for generated Rust + # code. + ATTRIBS = { + 'is_terminator': 'True for instructions that terminate the EBB.', + 'is_branch': 'True for all branch or jump instructions.', + 'is_call': 'Is this a call instruction?', + 'is_return': 'Is this a return instruction?', + 'can_trap': 'Can this instruction cause a trap?', + } + def __init__(self, name, doc, ins=(), outs=(), **kwargs): # type: (str, str, OpList, OpList, **Any) -> None # noqa self.name = name @@ -95,9 +108,8 @@ class Instruction(object): self.value_results = tuple( i for i, o in enumerate(self.outs) if o.is_value()) self._verify_polymorphic() - self.is_branch = 'is_branch' in kwargs - self.is_terminator = 'is_terminator' in kwargs - self.can_trap = 'can_trap' in kwargs + for attr in Instruction.ATTRIBS: + setattr(self, attr, not not kwargs.get(attr, False)) InstructionGroup.append(self) def __str__(self): diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 166967698e..353fa60551 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -288,31 +288,13 @@ def gen_opcodes(groups, fmt): fmt.line() with fmt.indented('impl Opcode {', '}'): - attrs = [ - { - 'name': 'is_branch', - 'comment': 'True for all branch instructions.' - }, - { - 'name': 'is_terminator', - 'comment': 'True for instructions that terminate EBB.' - }, - { - 'name': 'can_trap', - 'comment': 'True if instruction could trap.' - } - ] - - for attr in attrs: - if attr != attrs[0]: - fmt.line() - - fmt.doc_comment(attr['comment']) + for attr in sorted(Instruction.ATTRIBS.keys()): + fmt.doc_comment(Instruction.ATTRIBS[attr]) with fmt.indented('pub fn {}(self) -> bool {{' - .format(attr['name']), '}'): + .format(attr), '}'): with fmt.indented('match self {', '}'): for i in instrs: - if getattr(i, attr['name']): + if getattr(i, attr): fmt.format( 'Opcode::{} => true,', i.camel_name, i.name) From e0f7243523bca0fb34a71d7f501899162d5ccaf1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 8 Mar 2017 19:43:24 -0800 Subject: [PATCH 0559/3084] Handle a half-full func.locations map. If func.locations has not been properly resized to have an entry for all values, we should just full in the default location for the missing values instead of crashing. --- lib/cretonne/src/write.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 40b625d6e0..b0ef3a03d3 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -186,7 +186,9 @@ fn write_instruction(w: &mut Write, if !func.locations.is_empty() { let regs = isa.register_info(); for r in func.dfg.inst_results(inst) { - write!(s, ",{}", func.locations[r].display(®s))? + write!(s, + ",{}", + func.locations.get(r).cloned().unwrap_or_default().display(®s))? } } write!(s, "]")?; From 6fba5c4e0a5d1830c8ab0e96d9cbf8ee91fbafbd Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Thu, 9 Mar 2017 02:45:20 +0000 Subject: [PATCH 0560/3084] We now parse and record a ValueLoc for each SSA value result of each instruction. Code currently not passing tests. --- .../parser/instruction_encoding.cton | 9 ++- lib/reader/src/parser.rs | 67 +++++++++++++++---- 2 files changed, 57 insertions(+), 19 deletions(-) diff --git a/cranelift/filetests/parser/instruction_encoding.cton b/cranelift/filetests/parser/instruction_encoding.cton index 0e40d0b2f1..4971cba510 100644 --- a/cranelift/filetests/parser/instruction_encoding.cton +++ b/cranelift/filetests/parser/instruction_encoding.cton @@ -15,11 +15,10 @@ ebb1(v0: i32, v1: i32): } ; sameln: function foo(i32, i32) { ; nextln: $ebb1($v0: i32, $v1: i32): -; nextln: [-]$WS $v2 = iadd $v0, $v1 +; nextln: [-,-]$WS $v2 = iadd $v0, $v1 ; nextln: [-]$WS trap -; nextln: [R#1234]$WS $v6, $v7 = iadd_cout $v2, $v0 -; TODO Add the full encoding information available: instruction recipe name and architectural registers if specified -; nextln: [Rshamt#beef]$WS $v8 = ishl_imm $v6, 2 -; nextln: [-]$WS $v9 = iadd $v8, $v7 +; nextln: [R#1234,%x5,%x11]$WS $v6, $v7 = iadd_cout $v2, $v0 +; nextln: [Rshamt#beef,%x25]$WS $v8 = ishl_imm $v6, 2 +; nextln: [-,-]$WS $v9 = iadd $v8, $v7 ; nextln: [Iret#05]$WS return $v0, $v8 ; nextln: } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 1c71f9f4f5..7427139a4d 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -10,7 +10,7 @@ use std::{u16, u32}; use std::mem; use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotData, JumpTable, JumpTableData, Signature, ArgumentType, ArgumentExtension, ExtFuncData, SigRef, - FuncRef}; + FuncRef, ValueLoc}; use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; @@ -959,9 +959,32 @@ impl<'a> Parser<'a> { ctx.map.def_value(vx, value, &vx_location) } + fn parse_value_location(&mut self, ctx: &Context, isa: &TargetIsa) -> Result { + let reginfo = isa.register_info(); + match self.token() { + Some(Token::StackSlot(src_num)) => { + self.consume(); + if let Some(ss) = ctx.map.get_ss(src_num) { + Ok(ValueLoc::Stack(ss)) + } else { + err!(self.loc, + "attempted to use undefined stack slot ss{}", + src_num) + } + } + Some(Token::Name(name)) => { + self.consume(); + reginfo.parse_regunit(name) + .map(ValueLoc::Reg) + .ok_or(self.error("invalid register value location")) + } + _ => err!(self.loc, "invalid value location"), + } + } + fn parse_instruction_encoding(&mut self, ctx: &Context) - -> Result<(Option, Option>)> { + -> Result<(Option, Option>)> { let (mut encoding, mut result_registers) = (None, None); // encoding ::= "[" encoding_literal result_registers "]" @@ -984,11 +1007,13 @@ impl<'a> Parser<'a> { // result_registers ::= ("," ( "-" | names ) )? // names ::= Name { "," Name } if self.optional(Token::Comma) && !self.optional(Token::Minus) { + let isa = ctx.unique_isa + .ok_or(self.error("value locations specified without a unique isa"))?; let mut result = Vec::new(); - result.push(self.match_name("expected register")?); + result.push(self.parse_value_location(ctx, isa)?); while self.optional(Token::Comma) { - result.push(self.match_name("expected register")?); + result.push(self.parse_value_location(ctx, isa)?); } result_registers = Some(result); @@ -1010,7 +1035,7 @@ impl<'a> Parser<'a> { // Collect comments for the next instruction to be allocated. self.gather_comments(ctx.function.dfg.next_inst()); - let (encoding, result_registers) = self.parse_instruction_encoding(ctx)?; + let (encoding, result_locations) = self.parse_instruction_encoding(ctx)?; // Result value numbers. let mut results = Vec::new(); @@ -1030,13 +1055,6 @@ impl<'a> Parser<'a> { self.match_token(Token::Equal, "expected '=' before opcode")?; } - if let Some(ref result_registers) = result_registers { - if result_registers.len() != results.len() { - return err!(self.loc, - "must have same number of result registers as results"); - } - } - // instruction ::= [inst-results "="] * Opcode(opc) ["." Type] ... let opcode = if let Some(Token::Identifier(text)) = self.token() { match text.parse() { @@ -1086,8 +1104,29 @@ impl<'a> Parser<'a> { // Pass a reference to `ctx.values` instead of `ctx` itself since the `Values` iterator // holds a reference to `ctx.function`. self.add_values(&mut ctx.map, - results.into_iter(), - ctx.function.dfg.inst_results(inst)) + results.iter().map(|v| *v), + ctx.function.dfg.inst_results(inst))?; + + if let Some(ref result_locations) = result_locations { + if result_locations.len() != results.len() { + return err!(self.loc, + "must have same number of result locations as results"); + } else { + for (&src_value, &loc) in results.iter().zip(result_locations) { + // We are safe to unwrap here because all values should have been added to the + // map in the above call to self.add_values() + let value = ctx.map.get_value(src_value).unwrap(); + *ctx.function.locations.ensure(value) = loc; + } + } + } else { + for &src_value in results.iter() { + let value = ctx.map.get_value(src_value).unwrap(); + *ctx.function.locations.ensure(value) = ValueLoc::Unassigned; + } + } + + Ok(()) } // Type inference for polymorphic instructions. From 20d6713f93e4df4218ab3a5a3b039ca57c769a43 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Thu, 9 Mar 2017 12:09:28 +0000 Subject: [PATCH 0561/3084] Some refactoring, relaxed error handling so we allow encoding specifiers even without a unique ISA. --- lib/reader/src/parser.rs | 84 +++++++++++++++++++--------------------- 1 file changed, 39 insertions(+), 45 deletions(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 7427139a4d..b83f8c0b38 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -504,16 +504,6 @@ impl<'a> Parser<'a> { } } - // Match and consume a name. - fn match_name(&mut self, err_msg: &str) -> Result<&'a str> { - if let Some(Token::Name(name)) = self.token() { - self.consume(); - Ok(name) - } else { - err!(self.loc, err_msg) - } - } - /// Parse a list of test commands. pub fn parse_test_commands(&mut self) -> Vec> { let mut list = Vec::new(); @@ -959,8 +949,7 @@ impl<'a> Parser<'a> { ctx.map.def_value(vx, value, &vx_location) } - fn parse_value_location(&mut self, ctx: &Context, isa: &TargetIsa) -> Result { - let reginfo = isa.register_info(); + fn parse_value_location(&mut self, ctx: &Context) -> Result { match self.token() { Some(Token::StackSlot(src_num)) => { self.consume(); @@ -974,9 +963,19 @@ impl<'a> Parser<'a> { } Some(Token::Name(name)) => { self.consume(); - reginfo.parse_regunit(name) - .map(ValueLoc::Reg) - .ok_or(self.error("invalid register value location")) + if let Some(isa) = ctx.unique_isa { + isa.register_info() + .parse_regunit(name) + .map(ValueLoc::Reg) + .ok_or(self.error("invalid register value location")) + } else { + // For convenience we ignore value locations when no unique ISA is specified + Ok(ValueLoc::Unassigned) + } + } + Some(Token::Minus) => { + self.consume(); + Ok(ValueLoc::Unassigned) } _ => err!(self.loc, "invalid value location"), } @@ -985,9 +984,9 @@ impl<'a> Parser<'a> { fn parse_instruction_encoding(&mut self, ctx: &Context) -> Result<(Option, Option>)> { - let (mut encoding, mut result_registers) = (None, None); + let (mut encoding, mut result_locations) = (None, None); - // encoding ::= "[" encoding_literal result_registers "]" + // encoding ::= "[" encoding_literal result_locations "]" if self.optional(Token::LBracket) { // encoding_literal ::= "-" | Identifier HexSequence if !self.optional(Token::Minus) { @@ -999,31 +998,29 @@ impl<'a> Parser<'a> { } else if ctx.unique_isa.is_some() { return err!(self.loc, "invalid instruction recipe"); } else { - return err!(self.loc, - "provided instruction encoding for unspecified ISA"); + // We allow encodings to be specified when there's no unique ISA purely + // for convenience, eg when copy-pasting code for a test. } } - // result_registers ::= ("," ( "-" | names ) )? + // result_locations ::= ("," ( "-" | names ) )? // names ::= Name { "," Name } - if self.optional(Token::Comma) && !self.optional(Token::Minus) { - let isa = ctx.unique_isa - .ok_or(self.error("value locations specified without a unique isa"))?; - let mut result = Vec::new(); + if self.optional(Token::Comma) { + let mut results = Vec::new(); - result.push(self.parse_value_location(ctx, isa)?); + results.push(self.parse_value_location(ctx)?); while self.optional(Token::Comma) { - result.push(self.parse_value_location(ctx, isa)?); + results.push(self.parse_value_location(ctx)?); } - result_registers = Some(result); + result_locations = Some(results); } self.match_token(Token::RBracket, "expected ']' to terminate instruction encoding")?; } - Ok((encoding, result_registers)) + Ok((encoding, result_locations)) } // Parse an instruction, append it to `ebb`. @@ -1100,29 +1097,26 @@ impl<'a> Parser<'a> { results.len()); } + if let Some(ref result_locations) = result_locations { + if results.len() != result_locations.len() { + return err!(self.loc, + "instruction produces {} result values, but {} locations were \ + specified", + results.len(), + result_locations.len()); + } + } + // Now map the source result values to the just created instruction results. // Pass a reference to `ctx.values` instead of `ctx` itself since the `Values` iterator // holds a reference to `ctx.function`. self.add_values(&mut ctx.map, - results.iter().map(|v| *v), + results.into_iter(), ctx.function.dfg.inst_results(inst))?; - if let Some(ref result_locations) = result_locations { - if result_locations.len() != results.len() { - return err!(self.loc, - "must have same number of result locations as results"); - } else { - for (&src_value, &loc) in results.iter().zip(result_locations) { - // We are safe to unwrap here because all values should have been added to the - // map in the above call to self.add_values() - let value = ctx.map.get_value(src_value).unwrap(); - *ctx.function.locations.ensure(value) = loc; - } - } - } else { - for &src_value in results.iter() { - let value = ctx.map.get_value(src_value).unwrap(); - *ctx.function.locations.ensure(value) = ValueLoc::Unassigned; + if let Some(result_locations) = result_locations { + for (value, loc) in ctx.function.dfg.inst_results(inst).zip(result_locations) { + *ctx.function.locations.ensure(value) = loc; } } From cfe58d799437cd2b06bc64898b3a207e74ac5813 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 9 Mar 2017 10:05:55 -0800 Subject: [PATCH 0562/3084] Upgrade to Sphinx 1.5.3 Read the Docs is now using the latest version of Sphinx, so upgrade our recommended version too. As of Sphinx 1.4, index entries are 5-tuples instead of 4-tuples. Update the Cretonne Sphinx domain to generate the new 5-tuples. Since we're over this compatibility bump, there's no reason to recommend a specific Sphinx version, so just go back to 'current'. --- README.rst | 7 +++---- cranelift/docs/cton_domain.py | 4 +++- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/README.rst b/README.rst index 4a46f9267c..8bd8dd13e6 100644 --- a/README.rst +++ b/README.rst @@ -65,11 +65,10 @@ Building the documentation To build the Cretonne documentation, you need the `Sphinx documentation generator `_:: - $ pip install sphinx==1.3.5 sphinx-autobuild + $ pip install sphinx>=1.4 sphinx-autobuild sphinx_rtd_theme $ cd cretonne/docs $ make html $ open _build/html/index.html -The specific Sphinx version is currently used by Read the Docs. Sphinx 1.4 has -been released, but produces lots of warnings about four-column indices. We'll -upgrade when Read the Docs does. +We don't support Sphinx versions before 1.4 since the format of index tuples +has changed. diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index 2f56b5625e..23204f02e7 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -65,7 +65,8 @@ class CtonObject(ObjectDescription): indextext = self.get_index_text(name) if indextext: self.indexnode['entries'].append(('single', indextext, - targetname, '')) + targetname, '', None)) + # Type variables are indicated as %T. typevar = re.compile('(\%[A-Z])') @@ -113,6 +114,7 @@ class CtonType(CtonObject): def get_index_text(self, name): return name + ' (IL type)' + sep_equal = re.compile('\s*=\s*') sep_comma = re.compile('\s*,\s*') From e21bf444dc44e0c01849543d937d09673f2b74a4 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Thu, 9 Mar 2017 20:19:15 +0000 Subject: [PATCH 0563/3084] Added parsing of argument locations for functions and signatures. --- cranelift/filetests/isa/riscv/abi.cton | 33 ++++++++ lib/reader/src/parser.rs | 108 +++++++++++++++++++------ 2 files changed, 117 insertions(+), 24 deletions(-) diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.cton index 0282e94874..0b47fe6727 100644 --- a/cranelift/filetests/isa/riscv/abi.cton +++ b/cranelift/filetests/isa/riscv/abi.cton @@ -61,3 +61,36 @@ ebb0(v0: i64x4): v1 = iadd v0, v0 return v0 } + +function parse_encoding(i32 [%x5]) -> i32 [%x10] { +; check: function parse_encoding(i32 [%x5]) -> i32 [%x10] { + + sig0 = signature(i32 [%x10]) -> i32 [%x10] +; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] + + sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] +; check: sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] + + sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] +; check: sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] + +; Arguments on stack where not necessary + sig3 = signature(f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] +; check: sig3 = signature(f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] + +; Stack argument before register argument + sig4 = signature(f32 [72], i32 [%x10]) +; check: sig4 = signature(f32 [72], i32 [%x10]) + +; Return value on stack + sig5 = signature() -> f32 [0] +; check: sig5 = signature() -> f32 [0] + +; function + signature + fn15 = function bar(i32 [%x10]) -> b1 [%x10] +; check: sig6 = signature(i32 [%x10]) -> b1 [%x10] +; nextln: fn0 = sig6 bar + +ebb0(v0: i32): + return_reg v0 +} \ No newline at end of file diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index b83f8c0b38..6041039513 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -10,7 +10,7 @@ use std::{u16, u32}; use std::mem; use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotData, JumpTable, JumpTableData, Signature, ArgumentType, ArgumentExtension, ExtFuncData, SigRef, - FuncRef, ValueLoc}; + FuncRef, ValueLoc, ArgumentLoc}; use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; @@ -445,6 +445,19 @@ impl<'a> Parser<'a> { } } + // Match and consume a u32 immediate. + // This is used for stack argument byte offsets. + fn match_uimm32(&mut self, err_msg: &str) -> Result { + if let Some(Token::Integer(text)) = self.token() { + self.consume(); + // Lexer just gives us raw text that looks like an integer. + // Parse it as a u32 to check for overflow and other issues. + text.parse().map_err(|_| self.error("expected u32 decimal immediate")) + } else { + err!(self.loc, err_msg) + } + } + // Match and consume an Ieee32 immediate. fn match_ieee32(&mut self, err_msg: &str) -> Result { if let Some(Token::Float(text)) = self.token() { @@ -596,7 +609,7 @@ impl<'a> Parser<'a> { self.comments.clear(); self.gather_comments(AnyEntity::Function); - let (location, name, sig) = self.parse_function_spec()?; + let (location, name, sig) = self.parse_function_spec(unique_isa)?; let mut ctx = Context::new(Function::with_name_signature(name, sig), unique_isa); // function ::= function-spec * "{" preamble function-body "}" @@ -630,7 +643,9 @@ impl<'a> Parser<'a> { // // function-spec ::= * "function" name signature // - fn parse_function_spec(&mut self) -> Result<(Location, FunctionName, Signature)> { + fn parse_function_spec(&mut self, + unique_isa: Option<&TargetIsa>) + -> Result<(Location, FunctionName, Signature)> { self.match_identifier("function", "expected 'function'")?; let location = self.loc; @@ -638,7 +653,7 @@ impl<'a> Parser<'a> { let name = self.parse_function_name()?; // function-spec ::= "function" name * signature - let sig = self.parse_signature()?; + let sig = self.parse_signature(unique_isa)?; Ok((location, name, sig)) } @@ -661,17 +676,21 @@ impl<'a> Parser<'a> { // // signature ::= * "(" [arglist] ")" ["->" retlist] [call_conv] // - fn parse_signature(&mut self) -> Result { + fn parse_signature(&mut self, unique_isa: Option<&TargetIsa>) -> Result { let mut sig = Signature::new(); self.match_token(Token::LPar, "expected function signature: ( args... )")?; // signature ::= "(" * [arglist] ")" ["->" retlist] [call_conv] if self.token() != Some(Token::RPar) { - sig.argument_types = self.parse_argument_list()?; + sig.argument_types = self.parse_argument_list(unique_isa)?; } self.match_token(Token::RPar, "expected ')' after function arguments")?; if self.optional(Token::Arrow) { - sig.return_types = self.parse_argument_list()?; + sig.return_types = self.parse_argument_list(unique_isa)?; + } + + if sig.argument_types.iter().all(|a| a.location.is_assigned()) { + sig.compute_argument_bytes(); } // TBD: calling convention. @@ -683,27 +702,27 @@ impl<'a> Parser<'a> { // // arglist ::= * arg { "," arg } // - fn parse_argument_list(&mut self) -> Result> { + fn parse_argument_list(&mut self, unique_isa: Option<&TargetIsa>) -> Result> { let mut list = Vec::new(); // arglist ::= * arg { "," arg } - list.push(self.parse_argument_type()?); + list.push(self.parse_argument_type(unique_isa)?); // arglist ::= arg * { "," arg } while self.optional(Token::Comma) { // arglist ::= arg { "," * arg } - list.push(self.parse_argument_type()?); + list.push(self.parse_argument_type(unique_isa)?); } Ok(list) } // Parse a single argument type with flags. - fn parse_argument_type(&mut self) -> Result { - // arg ::= * type { flag } + fn parse_argument_type(&mut self, unique_isa: Option<&TargetIsa>) -> Result { + // arg ::= * type { flag } [ argumentloc ] let mut arg = ArgumentType::new(self.match_type("expected argument type")?); - // arg ::= type * { flag } + // arg ::= type * { flag } [ argumentloc ] while let Some(Token::Identifier(s)) = self.token() { match s { "uext" => arg.extension = ArgumentExtension::Uext, @@ -714,9 +733,50 @@ impl<'a> Parser<'a> { self.consume(); } + // arg ::= type { flag } * [ argumentloc ] + arg.location = self.parse_argument_location(unique_isa)?; + Ok(arg) } + // Parse an argument location specifier; either a register or a byte offset into the stack. + fn parse_argument_location(&mut self, unique_isa: Option<&TargetIsa>) -> Result { + // argumentloc ::= '[' regname | uimm32 ']' + if self.optional(Token::LBracket) { + let result = match self.token() { + Some(Token::Name(name)) => { + self.consume(); + if let Some(isa) = unique_isa { + isa.register_info() + .parse_regunit(name) + .map(ArgumentLoc::Reg) + .ok_or(self.error("invalid register name")) + } else { + // We are unable to parse the register without a TargetISA, so we quietly + // ignore it. + Ok(ArgumentLoc::Unassigned) + } + } + Some(Token::Integer(_)) => { + let offset = self.match_uimm32("expected stack argument byte offset")?; + Ok(ArgumentLoc::Stack(offset)) + } + Some(Token::Minus) => { + self.consume(); + Ok(ArgumentLoc::Unassigned) + } + _ => err!(self.loc, "expected argument location"), + }; + + self.match_token(Token::RBracket, + "expected ']' to end argument location annotation")?; + + result + } else { + Ok(ArgumentLoc::Unassigned) + } + } + // Parse the function preamble. // // preamble ::= * { preamble-decl } @@ -736,7 +796,7 @@ impl<'a> Parser<'a> { } Some(Token::SigRef(..)) => { self.gather_comments(ctx.function.dfg.signatures.next_key()); - self.parse_signature_decl() + self.parse_signature_decl(ctx.unique_isa) .and_then(|(num, dat)| ctx.add_sig(num, dat, &self.loc)) } Some(Token::FuncRef(..)) => { @@ -781,11 +841,11 @@ impl<'a> Parser<'a> { // // signature-decl ::= SigRef(sigref) "=" "signature" signature // - fn parse_signature_decl(&mut self) -> Result<(u32, Signature)> { + fn parse_signature_decl(&mut self, unique_isa: Option<&TargetIsa>) -> Result<(u32, Signature)> { let number = self.match_sig("expected signature number: sig«n»")?; self.match_token(Token::Equal, "expected '=' in signature decl")?; self.match_identifier("signature", "expected 'signature'")?; - let data = self.parse_signature()?; + let data = self.parse_signature(unique_isa)?; Ok((number, data)) } @@ -805,7 +865,7 @@ impl<'a> Parser<'a> { let data = match self.token() { Some(Token::Identifier("function")) => { - let (loc, name, sig) = self.parse_function_spec()?; + let (loc, name, sig) = self.parse_function_spec(ctx.unique_isa)?; let sigref = ctx.function.dfg.signatures.push(sig); ctx.map.def_entity(sigref.into(), &loc).expect("duplicate SigRef entities created"); ExtFuncData { @@ -1523,33 +1583,33 @@ mod tests { #[test] fn argument_type() { let mut p = Parser::new("i32 sext"); - let arg = p.parse_argument_type().unwrap(); + let arg = p.parse_argument_type(None).unwrap(); assert_eq!(arg.value_type, types::I32); assert_eq!(arg.extension, ArgumentExtension::Sext); assert_eq!(arg.inreg, false); - let Error { location, message } = p.parse_argument_type().unwrap_err(); + let Error { location, message } = p.parse_argument_type(None).unwrap_err(); assert_eq!(location.line_number, 1); assert_eq!(message, "expected argument type"); } #[test] fn signature() { - let sig = Parser::new("()").parse_signature().unwrap(); + let sig = Parser::new("()").parse_signature(None).unwrap(); assert_eq!(sig.argument_types.len(), 0); assert_eq!(sig.return_types.len(), 0); let sig2 = Parser::new("(i8 inreg uext, f32, f64) -> i32 sext, f64") - .parse_signature() + .parse_signature(None) .unwrap(); assert_eq!(sig2.to_string(), "(i8 uext inreg, f32, f64) -> i32 sext, f64"); // `void` is not recognized as a type by the lexer. It should not appear in files. - assert_eq!(Parser::new("() -> void").parse_signature().unwrap_err().to_string(), + assert_eq!(Parser::new("() -> void").parse_signature(None).unwrap_err().to_string(), "1: expected argument type"); - assert_eq!(Parser::new("i8 -> i8").parse_signature().unwrap_err().to_string(), + assert_eq!(Parser::new("i8 -> i8").parse_signature(None).unwrap_err().to_string(), "1: expected function signature: ( args... )"); - assert_eq!(Parser::new("(i8 -> i8").parse_signature().unwrap_err().to_string(), + assert_eq!(Parser::new("(i8 -> i8").parse_signature(None).unwrap_err().to_string(), "1: expected ')' after function arguments"); } From a6c2cc71dfba7ee966021f107be257bcef75068c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 9 Mar 2017 13:10:27 -0800 Subject: [PATCH 0564/3084] Use value lists for call arguments. Add a new kind of instruction format that keeps all of its value arguments in a value list. These value lists are all allocated out of the dfg.value_lists memory pool. Instruction formats with the value_list property set store *all* of their value arguments in a single value list. There is no distinction between fixed arguments and variable arguments. Change the Call instruction format to use the value list representation for its arguments. This change is only the beginning. The intent is to eliminate the boxed_storage instruction formats completely. Value lists use less memory, and when the transition is complete, InstructionData will have a trivial Drop implementation. --- lib/cretonne/meta/base/formats.py | 2 +- lib/cretonne/meta/cdsl/formats.py | 3 ++ lib/cretonne/meta/gen_instr.py | 53 +++++++++++++++++++++++++-- lib/cretonne/src/entity_list.rs | 6 +++ lib/cretonne/src/ir/builder.rs | 11 +++++- lib/cretonne/src/ir/dfg.rs | 10 +++-- lib/cretonne/src/ir/instructions.rs | 45 +++++++++++++---------- lib/cretonne/src/ir/mod.rs | 2 +- lib/cretonne/src/regalloc/coloring.rs | 3 +- lib/cretonne/src/regalloc/liveness.rs | 2 +- lib/cretonne/src/write.rs | 27 ++++++++++++-- lib/reader/src/parser.rs | 22 ++++++----- 12 files changed, 142 insertions(+), 44 deletions(-) diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 89704623fd..9b29c54775 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -47,7 +47,7 @@ Branch = InstructionFormat(VALUE, ebb, VARIABLE_ARGS, boxed_storage=True) BranchTable = InstructionFormat(VALUE, jump_table) Call = InstructionFormat( - func_ref, VARIABLE_ARGS, multiple_results=True, boxed_storage=True) + func_ref, VARIABLE_ARGS, multiple_results=True, value_list=True) IndirectCall = InstructionFormat( sig_ref, VALUE, VARIABLE_ARGS, multiple_results=True, boxed_storage=True) diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index 974ca80541..422d84c296 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -34,6 +34,8 @@ class InstructionFormat(object): enums. :param multiple_results: Set to `True` if this instruction format allows more than one result to be produced. + :param value_list: Set to `True` if this instruction format uses a + `ValueList` member to store its value operands. :param boxed_storage: Set to `True` is this instruction format requires a `data: Box<...>` pointer to additional storage in its `InstructionData` variant. @@ -52,6 +54,7 @@ class InstructionFormat(object): # type: (*Union[OperandKind, Tuple[str, OperandKind]], **Any) -> None # noqa self.name = kwargs.get('name', None) # type: str self.multiple_results = kwargs.get('multiple_results', False) + self.has_value_list = kwargs.get('value_list', False) self.boxed_storage = kwargs.get('boxed_storage', False) self.members = list() # type: List[str] self.kinds = tuple(self._process_member_names(kinds)) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 353fa60551..b8afeb3eb7 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -58,17 +58,32 @@ def gen_arguments_method(fmt, is_mut): method = 'arguments' mut = '' rslice = 'ref_slice' + as_slice = 'as_slice' if is_mut: method += '_mut' mut = 'mut ' rslice += '_mut' + as_slice = 'as_mut_slice' with fmt.indented( - 'pub fn {f}(&{m}self) -> [&{m}[Value]; 2] {{' + 'pub fn {f}<\'a>(&\'a {m}self, pool: &\'a {m}ValueListPool) -> ' + '[&{m}[Value]; 2] {{' .format(f=method, m=mut), '}'): with fmt.indented('match *self {', '}'): for f in InstructionFormat.all_formats: n = 'InstructionData::' + f.name + + # Formats with a value list put all of their arguments in the + # list. We don't split them up, just return it all as variable + # arguments. (I expect the distinction to go away). + if f.has_value_list: + arg = ''.format(mut) + fmt.line( + '{} {{ ref {}args, .. }} => ' + '[ &{}[], args.{}(pool) ],' + .format(n, mut, mut, as_slice)) + continue + has_varargs = cdsl.operands.VARIABLE_ARGS in f.kinds # Formats with both fixed and variable arguments delegate to # the data struct. We need to work around borrow checker quirks @@ -472,7 +487,11 @@ def gen_format_constructor(iform, fmt): """ # Construct method arguments. - args = ['self', 'opcode: Opcode'] + if iform.has_value_list: + args = ['mut self'] + else: + args = ['self'] + args.append('opcode: Opcode') if iform.multiple_results: args.append('ctrl_typevar: Type') @@ -484,7 +503,10 @@ def gen_format_constructor(iform, fmt): # Normal operand arguments. for idx, kind in enumerate(iform.kinds): - args.append('op{}: {}'.format(idx, kind.rust_type)) + if kind is cdsl.operands.VARIABLE_ARGS and iform.has_value_list: + args.append('op{}: &[Value]'.format(idx, kind.rust_type)) + else: + args.append('op{}: {}'.format(idx, kind.rust_type)) proto = '{}({})'.format(iform.name, ', '.join(args)) proto += " -> (Inst, &'f mut DataFlowGraph)" @@ -492,6 +514,21 @@ def gen_format_constructor(iform, fmt): fmt.doc_comment(str(iform)) fmt.line('#[allow(non_snake_case)]') with fmt.indented('fn {} {{'.format(proto), '}'): + # Start by constructing a value list with *all* the arguments. + if iform.has_value_list: + fmt.line('let mut vlist = ValueList::default();') + with fmt.indented('{', '}'): + fmt.line( + 'let pool = ' + '&mut self.data_flow_graph_mut().value_lists;') + for idx, kind in enumerate(iform.kinds): + if kind is cdsl.operands.VALUE: + fmt.line('vlist.push(op{}, pool);'.format(idx)) + elif kind is cdsl.operands.VARIABLE_ARGS: + fmt.line( + 'vlist.extend(op{}.iter().cloned(), pool);' + .format(idx)) + # Generate the instruction data. with fmt.indented( 'let data = InstructionData::{} {{'.format(iform.name), '};'): @@ -520,7 +557,10 @@ def gen_member_inits(iform, fmt): """ # Values first. - if len(iform.value_operands) == 1: + if iform.has_value_list: + # Value-list formats put *all* arguments in the list. + fmt.line('args: vlist,') + elif len(iform.value_operands) == 1: fmt.line('arg: op{},'.format(iform.value_operands[0])) elif len(iform.value_operands) > 1: fmt.line('args: [{}],'.format( @@ -528,6 +568,8 @@ def gen_member_inits(iform, fmt): # Immediates and entity references. for idx, member in enumerate(iform.members): + if iform.has_value_list and member == 'varargs': + continue if member: fmt.line('{}: op{},'.format(member, idx)) @@ -557,6 +599,9 @@ def gen_inst_builder(inst, fmt): t = 'T{}{}'.format(1 + len(tmpl_types), op.kind.name) tmpl_types.append('{}: Into<{}>'.format(t, op.kind.rust_type)) into_args.append(op.name) + elif (inst.format.has_value_list and + op.kind is cdsl.operands.VARIABLE_ARGS): + t = '&[Value]' else: t = op.kind.rust_type args.append('{}: {}'.format(op.name, t)) diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index b2c75bd9ed..2b8ebb783e 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -54,6 +54,11 @@ use entity_map::EntityRef; /// /// All of the list methods that take a pool reference must be given the same pool reference every /// time they are called. Otherwise data structures will be corrupted. +/// +/// Entity lists can be cloned, but that operation should only be used as part of cloning the whole +/// function they belong to. *Cloning an entity list does not allocate new memory for the clone*. +/// It creates an alias of the same memory. +#[derive(Clone, Debug)] pub struct EntityList { index: u32, unused: PhantomData, @@ -70,6 +75,7 @@ impl Default for EntityList { } /// A memory pool for storing lists of `T`. +#[derive(Clone)] pub struct ListPool { // The main array containing the lists. data: Vec, diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 639b640c20..ebb8ae2350 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -5,7 +5,7 @@ use ir::{types, instructions}; use ir::{InstructionData, DataFlowGraph, Cursor}; -use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, VariableArgs, SigRef, FuncRef}; +use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, VariableArgs, SigRef, FuncRef, ValueList}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, ImmVector}; use ir::condcodes::{IntCC, FloatCC}; @@ -21,6 +21,7 @@ pub trait InstBuilderBase<'f>: Sized { /// Get an immutable reference to the data flow graph that will hold the constructed /// instructions. fn data_flow_graph(&self) -> &DataFlowGraph; + fn data_flow_graph_mut(&mut self) -> &mut DataFlowGraph; /// Insert a simple instruction and return a reference to it. /// @@ -76,6 +77,10 @@ impl<'c, 'fc, 'fd> InstBuilderBase<'fd> for InsertBuilder<'c, 'fc, 'fd> { self.dfg } + fn data_flow_graph_mut(&mut self) -> &mut DataFlowGraph { + self.dfg + } + fn simple_instruction(self, data: InstructionData) -> (Inst, &'fd mut DataFlowGraph) { let inst = self.dfg.make_inst(data); self.pos.insert_inst(inst); @@ -129,6 +134,10 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { self.dfg } + fn data_flow_graph_mut(&mut self) -> &mut DataFlowGraph { + self.dfg + } + fn simple_instruction(self, data: InstructionData) -> (Inst, &'f mut DataFlowGraph) { // The replacement instruction cannot generate multiple results, so verify that the old // instruction's secondary results have been detached. diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 593d3a7907..07e7dbbdb5 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -1,6 +1,6 @@ //! Data flow graph tracking Instructions, Values, and EBBs. -use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef}; +use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueListPool}; use ir::entities::ExpandedValue; use ir::instructions::{Opcode, InstructionData, CallInfo}; use ir::extfunc::ExtFuncData; @@ -24,7 +24,10 @@ pub struct DataFlowGraph { /// Data about all of the instructions in the function, including opcodes and operands. /// The instructions in this map are not in program order. That is tracked by `Layout`, along /// with the EBB containing each instruction. - insts: EntityMap, + pub insts: EntityMap, + + /// Memory pool of value lists referenced by instructions in `insts`. + pub value_lists: ValueListPool, /// Extended basic blocks in the function and their arguments. /// This map is not in program order. That is handled by `Layout`, and so is the sequence of @@ -56,6 +59,7 @@ impl DataFlowGraph { pub fn new() -> DataFlowGraph { DataFlowGraph { insts: EntityMap::new(), + value_lists: ValueListPool::new(), ebbs: EntityMap::new(), extended_values: Vec::new(), signatures: EntityMap::new(), @@ -429,7 +433,7 @@ impl DataFlowGraph { /// Get the call signature of a direct or indirect call instruction. /// Returns `None` if `inst` is not a call instruction. pub fn call_signature(&self, inst: Inst) -> Option { - match self.insts[inst].analyze_call() { + match self.insts[inst].analyze_call(&self.value_lists) { CallInfo::NotACall => None, CallInfo::Direct(f, _) => Some(self.ext_funcs[f].signature), CallInfo::Indirect(s, _) => Some(s), diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 2a956ffc86..2425b8814a 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -16,8 +16,17 @@ use ir::condcodes::*; use ir::types; use ir::DataFlowGraph; -use ref_slice::*; +use entity_list; use packed_option::PackedOption; +use ref_slice::{ref_slice, ref_slice_mut}; + +/// Some instructions use an external list of argument values because there is not enough space in +/// the 16-byte `InstructionData` struct. These value lists are stored in a memory pool in +/// `dfg.value_lists`. +pub type ValueList = entity_list::EntityList; + +/// Memory pool for holding value lists. See `ValueList`. +pub type ValueListPool = entity_list::ListPool; // Include code generated by `lib/cretonne/meta/gen_instr.py`. This file contains: // @@ -204,7 +213,8 @@ pub enum InstructionData { opcode: Opcode, ty: Type, second_result: PackedOption, - data: Box, + func_ref: FuncRef, + args: ValueList, }, IndirectCall { opcode: Opcode, @@ -244,6 +254,13 @@ impl VariableArgs { pub fn is_empty(&self) -> bool { self.0.is_empty() } + + /// Convert this to a value list in `pool`. + pub fn into_value_list(self, pool: &mut ValueListPool) -> ValueList { + let mut vlist = ValueList::default(); + vlist.extend(self.0, pool); + vlist + } } // Coerce `VariableArgs` into a `&[Value]` slice. @@ -364,16 +381,6 @@ impl Display for BranchData { } } -/// Payload of a call instruction. -#[derive(Clone, Debug)] -pub struct CallData { - /// Callee function. - pub func_ref: FuncRef, - - /// Dynamically sized array containing call argument values. - pub varargs: VariableArgs, -} - /// Payload of an indirect call instruction. #[derive(Clone, Debug)] pub struct IndirectCallData { @@ -434,10 +441,10 @@ impl ReturnRegData { impl InstructionData { /// Execute a closure once for each argument to this instruction. /// See also the `arguments()` method. - pub fn each_arg(&self, mut func: F) + pub fn each_arg(&self, pool: &ValueListPool, mut func: F) where F: FnMut(Value) { - for part in &self.arguments() { + for part in &self.arguments(pool) { for &arg in part.iter() { func(arg); } @@ -446,10 +453,10 @@ impl InstructionData { /// Execute a closure with a mutable reference to each argument to this instruction. /// See also the `arguments_mut()` method. - pub fn each_arg_mut(&mut self, mut func: F) + pub fn each_arg_mut(&mut self, pool: &mut ValueListPool, mut func: F) where F: FnMut(&mut Value) { - for part in &mut self.arguments_mut() { + for part in &mut self.arguments_mut(pool) { for arg in part.iter_mut() { func(arg); } @@ -476,10 +483,10 @@ impl InstructionData { /// Return information about a call instruction. /// /// Any instruction that can call another function reveals its call signature here. - pub fn analyze_call<'a>(&'a self) -> CallInfo<'a> { + pub fn analyze_call<'a>(&'a self, pool: &'a ValueListPool) -> CallInfo<'a> { match self { - &InstructionData::Call { ref data, .. } => { - CallInfo::Direct(data.func_ref, &data.varargs) + &InstructionData::Call { func_ref, ref args, .. } => { + CallInfo::Direct(func_ref, &args.as_slice(pool)) } &InstructionData::IndirectCall { ref data, .. } => { CallInfo::Indirect(data.sig_ref, &data.varargs) diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index c14473e9f5..302978d81b 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -20,7 +20,7 @@ pub use ir::funcname::FunctionName; pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension, ExtFuncData}; pub use ir::types::Type; pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable, FuncRef, SigRef}; -pub use ir::instructions::{Opcode, InstructionData, VariableArgs}; +pub use ir::instructions::{Opcode, InstructionData, VariableArgs, ValueList, ValueListPool}; pub use ir::stackslot::StackSlotData; pub use ir::jumptable::JumpTableData; pub use ir::valueloc::{ValueLoc, ArgumentLoc}; diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 11018f664d..6549c4356f 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -299,7 +299,8 @@ impl<'a> Context<'a> { } ConstraintKind::Tied(arg_index) => { // This def must use the same register as a fixed instruction argument. - let loc = locations[dfg[inst].arguments()[0][arg_index as usize]]; + let arg = dfg[inst].arguments(&dfg.value_lists)[0][arg_index as usize]; + let loc = locations[arg]; *locations.ensure(lv.value) = loc; // Mark the reused register. It's not really clear if we support tied // stack operands. We could do that for some Intel read-modify-write diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 524b50e8ea..49db4a04cd 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -318,7 +318,7 @@ impl Liveness { let mut operand_constraints = recipe_constraints.get(recipe).map(|c| c.ins).unwrap_or(&[]).iter(); - func.dfg[inst].each_arg(|arg| { + func.dfg[inst].each_arg(&func.dfg.value_lists, |arg| { // Get the live range, create it as a dead range if necessary. let lr = get_or_create(&mut self.ranges, arg, func, recipe_constraints); diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index b0ef3a03d3..8cba9757f9 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -6,7 +6,7 @@ use ir::{Function, Ebb, Inst, Value, Type}; use isa::{TargetIsa, RegInfo}; -use std::fmt::{Result, Error, Write}; +use std::fmt::{self, Result, Error, Write}; use std::result; /// Write `func` to `w` as equivalent text. @@ -157,7 +157,7 @@ fn type_suffix(func: &Function, inst: Inst) -> Option { // Write out any value aliases appearing in `inst`. fn write_value_aliases(w: &mut Write, func: &Function, inst: Inst, indent: usize) -> Result { - for &arg in func.dfg[inst].arguments().iter().flat_map(|x| x.iter()) { + for &arg in func.dfg[inst].arguments(&func.dfg.value_lists).iter().flat_map(|x| x.iter()) { let resolved = func.dfg.resolve_aliases(arg); if resolved != arg { writeln!(w, "{1:0$}{2} -> {3}", indent, "", arg, resolved)?; @@ -247,7 +247,12 @@ fn write_instruction(w: &mut Write, Jump { ref data, .. } => writeln!(w, " {}", data), Branch { ref data, .. } => writeln!(w, " {}", data), BranchTable { arg, table, .. } => writeln!(w, " {}, {}", arg, table), - Call { ref data, .. } => writeln!(w, " {}({})", data.func_ref, data.varargs), + Call { func_ref, ref args, .. } => { + writeln!(w, + " {}({})", + func_ref, + DisplayValues(args.as_slice(&func.dfg.value_lists))) + } IndirectCall { ref data, .. } => { writeln!(w, " {}, {}({})", data.sig_ref, data.arg, data.varargs) } @@ -268,6 +273,22 @@ fn write_instruction(w: &mut Write, } } +/// Displayable slice of values. +struct DisplayValues<'a>(&'a [Value]); + +impl<'a> fmt::Display for DisplayValues<'a> { + fn fmt(&self, f: &mut fmt::Formatter) -> Result { + for (i, val) in self.0.iter().enumerate() { + if i == 0 { + write!(f, "{}", val)?; + } else { + write!(f, ", {}", val)?; + } + } + Ok(()) + } +} + #[cfg(test)] mod tests { use ir::{Function, FunctionName, StackSlotData}; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 6041039513..c009270f54 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -15,8 +15,8 @@ use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, - TernaryOverflowData, JumpData, BranchData, CallData, - IndirectCallData, ReturnData, ReturnRegData}; + TernaryOverflowData, JumpData, BranchData, IndirectCallData, + ReturnData, ReturnRegData}; use cretonne::isa::{self, TargetIsa, Encoding}; use cretonne::settings; use testfile::{TestFile, Details, Comment}; @@ -167,7 +167,8 @@ impl<'a> Context<'a> { for ebb in self.function.layout.ebbs() { for inst in self.function.layout.ebb_insts(ebb) { let loc = inst.into(); - match self.function.dfg[inst] { + let value_lists = &mut self.function.dfg.value_lists; + match self.function.dfg.insts[inst] { InstructionData::Nullary { .. } | InstructionData::UnaryImm { .. } | InstructionData::UnaryIeee32 { .. } | @@ -210,8 +211,8 @@ impl<'a> Context<'a> { self.map.rewrite_values(&mut data.varargs, loc)?; } - InstructionData::Call { ref mut data, .. } => { - self.map.rewrite_values(&mut data.varargs, loc)?; + InstructionData::Call { ref mut args, .. } => { + self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; } InstructionData::IndirectCall { ref mut data, .. } => { @@ -1300,7 +1301,10 @@ impl<'a> Parser<'a> { // Parse the operands following the instruction opcode. // This depends on the format of the opcode. - fn parse_inst_operands(&mut self, ctx: &Context, opcode: Opcode) -> Result { + fn parse_inst_operands(&mut self, + ctx: &mut Context, + opcode: Opcode) + -> Result { Ok(match opcode.format() { InstructionFormat::Nullary => { InstructionData::Nullary { @@ -1506,10 +1510,8 @@ impl<'a> Parser<'a> { opcode: opcode, ty: VOID, second_result: None.into(), - data: Box::new(CallData { - func_ref: func_ref, - varargs: args, - }), + func_ref: func_ref, + args: args.into_value_list(&mut ctx.function.dfg.value_lists), } } InstructionFormat::IndirectCall => { From 5170ef6b5f750146c4eb328746b97ae605e748d6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 9 Mar 2017 14:53:31 -0800 Subject: [PATCH 0565/3084] Convert the Branch and Jump instruction formats to value_list. The Branch format also stores its fixed argument in the value list. This requires the value pool to be passed to a few more functions. Note that this actually makes the Branch and Jump variants of InstructionData identical. The instruction format hashing does not yet understand that all value operands are stored in the value list. We'll fix that in a later patch. Also convert IndirectCall, noting that Call and IndirectCall remain separate instruction formats because they have different immediate fields. --- cranelift/src/print_cfg.rs | 2 +- lib/cretonne/meta/base/formats.py | 6 +- lib/cretonne/meta/gen_instr.py | 9 +- lib/cretonne/src/cfg.rs | 12 +- lib/cretonne/src/dominator_tree.rs | 10 +- lib/cretonne/src/ir/instructions.rs | 109 +++--------------- .../src/regalloc/live_value_tracker.rs | 2 +- lib/cretonne/src/write.rs | 34 +++++- lib/reader/src/parser.rs | 45 +++----- 9 files changed, 88 insertions(+), 141 deletions(-) diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index d8261dad10..206f5430c2 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -59,7 +59,7 @@ impl<'a> CFGPrinter<'a> { // Add all outgoing branch instructions to the label. for inst in self.func.layout.ebb_insts(ebb) { let idata = &self.func.dfg[inst]; - match idata.analyze_branch() { + match idata.analyze_branch(&self.func.dfg.value_lists) { BranchInfo::SingleDest(dest, _) => { write!(w, " | <{}>{} {}", inst, idata.opcode(), dest)? } diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 9b29c54775..9f7a805940 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -42,15 +42,15 @@ ExtractLane = InstructionFormat(VALUE, ('lane', uimm8)) IntCompare = InstructionFormat(intcc, VALUE, VALUE) FloatCompare = InstructionFormat(floatcc, VALUE, VALUE) -Jump = InstructionFormat(ebb, VARIABLE_ARGS, boxed_storage=True) -Branch = InstructionFormat(VALUE, ebb, VARIABLE_ARGS, boxed_storage=True) +Jump = InstructionFormat(ebb, VARIABLE_ARGS, value_list=True) +Branch = InstructionFormat(VALUE, ebb, VARIABLE_ARGS, value_list=True) BranchTable = InstructionFormat(VALUE, jump_table) Call = InstructionFormat( func_ref, VARIABLE_ARGS, multiple_results=True, value_list=True) IndirectCall = InstructionFormat( sig_ref, VALUE, VARIABLE_ARGS, - multiple_results=True, boxed_storage=True) + multiple_results=True, value_list=True) Return = InstructionFormat(VARIABLE_ARGS, boxed_storage=True) ReturnReg = InstructionFormat(VALUE, VARIABLE_ARGS, boxed_storage=True) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index b8afeb3eb7..376bfc0d56 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -206,12 +206,19 @@ def gen_instruction_data_impl(fmt): fmt.doc_comment('Get the controlling type variable operand.') with fmt.indented( - 'pub fn typevar_operand(&self) -> Option {', '}'): + 'pub fn typevar_operand(&self, pool: &ValueListPool) -> ' + 'Option {', '}'): with fmt.indented('match *self {', '}'): for f in InstructionFormat.all_formats: n = 'InstructionData::' + f.name if f.typevar_operand is None: fmt.line(n + ' { .. } => None,') + elif f.has_value_list: + # We keep all arguments in a value list. + i = f.value_operands.index(f.typevar_operand) + fmt.line( + '{} {{ ref args, .. }} => ' + 'args.get({}, pool),'.format(n, i)) elif len(f.value_operands) == 1: # We have a single value operand called 'arg'. if f.boxed_storage: diff --git a/lib/cretonne/src/cfg.rs b/lib/cretonne/src/cfg.rs index 9b0f3f15e7..c3cf19efe4 100644 --- a/lib/cretonne/src/cfg.rs +++ b/lib/cretonne/src/cfg.rs @@ -75,7 +75,7 @@ impl ControlFlowGraph { for ebb in &func.layout { for inst in func.layout.ebb_insts(ebb) { - match func.dfg[inst].analyze_branch() { + match func.dfg[inst].analyze_branch(&func.dfg.value_lists) { BranchInfo::SingleDest(dest, _) => { self.add_edge((ebb, inst), dest); } @@ -148,7 +148,7 @@ impl ControlFlowGraph { #[cfg(test)] mod tests { use super::*; - use ir::{Function, InstBuilder, Cursor, VariableArgs, types}; + use ir::{Function, InstBuilder, Cursor, types}; #[test] fn empty() { @@ -197,12 +197,12 @@ mod tests { let cur = &mut Cursor::new(&mut func.layout); cur.insert_ebb(ebb0); - br_ebb0_ebb2 = dfg.ins(cur).brnz(cond, ebb2, VariableArgs::new()); - jmp_ebb0_ebb1 = dfg.ins(cur).jump(ebb1, VariableArgs::new()); + br_ebb0_ebb2 = dfg.ins(cur).brnz(cond, ebb2, &[]); + jmp_ebb0_ebb1 = dfg.ins(cur).jump(ebb1, &[]); cur.insert_ebb(ebb1); - br_ebb1_ebb1 = dfg.ins(cur).brnz(cond, ebb1, VariableArgs::new()); - jmp_ebb1_ebb2 = dfg.ins(cur).jump(ebb2, VariableArgs::new()); + br_ebb1_ebb1 = dfg.ins(cur).brnz(cond, ebb1, &[]); + jmp_ebb1_ebb2 = dfg.ins(cur).jump(ebb2, &[]); cur.insert_ebb(ebb2); } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index fd2c6cfb50..8c90b46160 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -209,7 +209,7 @@ impl DominatorTree { #[cfg(test)] mod test { use super::*; - use ir::{Function, InstBuilder, Cursor, VariableArgs, types}; + use ir::{Function, InstBuilder, Cursor, types}; use cfg::ControlFlowGraph; #[test] @@ -238,14 +238,14 @@ mod test { let cur = &mut Cursor::new(&mut func.layout); cur.insert_ebb(ebb3); - jmp_ebb3_ebb1 = dfg.ins(cur).jump(ebb1, VariableArgs::new()); + jmp_ebb3_ebb1 = dfg.ins(cur).jump(ebb1, &[]); cur.insert_ebb(ebb1); - br_ebb1_ebb0 = dfg.ins(cur).brnz(cond, ebb0, VariableArgs::new()); - jmp_ebb1_ebb2 = dfg.ins(cur).jump(ebb2, VariableArgs::new()); + br_ebb1_ebb0 = dfg.ins(cur).brnz(cond, ebb0, &[]); + jmp_ebb1_ebb2 = dfg.ins(cur).jump(ebb2, &[]); cur.insert_ebb(ebb2); - dfg.ins(cur).jump(ebb0, VariableArgs::new()); + dfg.ins(cur).jump(ebb0, &[]); cur.insert_ebb(ebb0); } diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 2425b8814a..d7393c9293 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -196,12 +196,14 @@ pub enum InstructionData { Jump { opcode: Opcode, ty: Type, - data: Box, + destination: Ebb, + args: ValueList, }, Branch { opcode: Opcode, ty: Type, - data: Box, + destination: Ebb, + args: ValueList, }, BranchTable { opcode: Opcode, @@ -220,7 +222,8 @@ pub enum InstructionData { opcode: Opcode, ty: Type, second_result: PackedOption, - data: Box, + sig_ref: SigRef, + args: ValueList, }, Return { opcode: Opcode, @@ -255,9 +258,10 @@ impl VariableArgs { self.0.is_empty() } - /// Convert this to a value list in `pool`. - pub fn into_value_list(self, pool: &mut ValueListPool) -> ValueList { + /// Convert this to a value list in `pool` with `fixed` prepended. + pub fn into_value_list(self, fixed: &[Value], pool: &mut ValueListPool) -> ValueList { let mut vlist = ValueList::default(); + vlist.extend(fixed.iter().cloned(), pool); vlist.extend(self.0, pool); vlist } @@ -327,85 +331,6 @@ impl Display for TernaryOverflowData { } } -/// Payload data for jump instructions. These need to carry lists of EBB arguments that won't fit -/// in the allowed `InstructionData` size. -#[derive(Clone, Debug)] -pub struct JumpData { - /// Jump destination EBB. - pub destination: Ebb, - /// Arguments passed to destination EBB. - pub varargs: VariableArgs, -} - -impl Display for JumpData { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - if self.varargs.is_empty() { - write!(f, "{}", self.destination) - } else { - write!(f, "{}({})", self.destination, self.varargs) - } - } -} - -/// Payload data for branch instructions. These need to carry lists of EBB arguments that won't fit -/// in the allowed `InstructionData` size. -#[derive(Clone, Debug)] -pub struct BranchData { - /// Value argument controlling the branch. - pub arg: Value, - /// Branch destination EBB. - pub destination: Ebb, - /// Arguments passed to destination EBB. - pub varargs: VariableArgs, -} - -impl BranchData { - /// Get references to the arguments. - pub fn arguments(&self) -> [&[Value]; 2] { - [ref_slice(&self.arg), &self.varargs] - } - - /// Get mutable references to the arguments. - pub fn arguments_mut(&mut self) -> [&mut [Value]; 2] { - [ref_slice_mut(&mut self.arg), &mut self.varargs] - } -} - -impl Display for BranchData { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - write!(f, "{}, {}", self.arg, self.destination)?; - if !self.varargs.is_empty() { - write!(f, "({})", self.varargs)?; - } - Ok(()) - } -} - -/// Payload of an indirect call instruction. -#[derive(Clone, Debug)] -pub struct IndirectCallData { - /// Callee function. - pub arg: Value, - - /// Signature of the callee function. - pub sig_ref: SigRef, - - /// Dynamically sized array containing call argument values. - pub varargs: VariableArgs, -} - -impl IndirectCallData { - /// Get references to the arguments. - pub fn arguments(&self) -> [&[Value]; 2] { - [ref_slice(&self.arg), &self.varargs] - } - - /// Get mutable references to the arguments. - pub fn arguments_mut(&mut self) -> [&mut [Value]; 2] { - [ref_slice_mut(&mut self.arg), &mut self.varargs] - } -} - /// Payload of a return instruction. #[derive(Clone, Debug)] pub struct ReturnData { @@ -467,13 +392,13 @@ impl InstructionData { /// /// Any instruction that can transfer control to another EBB reveals its possible destinations /// here. - pub fn analyze_branch<'a>(&'a self) -> BranchInfo<'a> { + pub fn analyze_branch<'a>(&'a self, pool: &'a ValueListPool) -> BranchInfo<'a> { match self { - &InstructionData::Jump { ref data, .. } => { - BranchInfo::SingleDest(data.destination, &data.varargs) + &InstructionData::Jump { destination, ref args, .. } => { + BranchInfo::SingleDest(destination, &args.as_slice(pool)) } - &InstructionData::Branch { ref data, .. } => { - BranchInfo::SingleDest(data.destination, &data.varargs) + &InstructionData::Branch { destination, ref args, .. } => { + BranchInfo::SingleDest(destination, &args.as_slice(pool)[1..]) } &InstructionData::BranchTable { table, .. } => BranchInfo::Table(table), _ => BranchInfo::NotABranch, @@ -488,8 +413,8 @@ impl InstructionData { &InstructionData::Call { func_ref, ref args, .. } => { CallInfo::Direct(func_ref, &args.as_slice(pool)) } - &InstructionData::IndirectCall { ref data, .. } => { - CallInfo::Indirect(data.sig_ref, &data.varargs) + &InstructionData::IndirectCall { sig_ref, ref args, .. } => { + CallInfo::Indirect(sig_ref, &args.as_slice(pool)) } _ => CallInfo::NotACall, } @@ -507,7 +432,7 @@ impl InstructionData { } else if constraints.requires_typevar_operand() { // Not all instruction formats have a designated operand, but in that case // `requires_typevar_operand()` should never be true. - dfg.value_type(self.typevar_operand() + dfg.value_type(self.typevar_operand(&dfg.value_lists) .expect("Instruction format doesn't have a designated operand, bad opcode.")) } else { // For locality of reference, we prefer to get the controlling type variable from diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index bcbbb62fe1..c45da220e4 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -217,7 +217,7 @@ impl LiveValueTracker { -> (&[LiveValue], &[LiveValue]) { // Save a copy of the live values before any branches or jumps that could be somebody's // immediate dominator. - match dfg[inst].analyze_branch() { + match dfg[inst].analyze_branch(&dfg.value_lists) { BranchInfo::NotABranch => {} _ => self.save_idom_live_set(inst), } diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 8cba9757f9..5e8968c48b 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -225,6 +225,7 @@ fn write_instruction(w: &mut Write, } // Then the operands, depending on format. + let pool = &func.dfg.value_lists; use ir::instructions::InstructionData::*; match func.dfg[inst] { Nullary { .. } => writeln!(w, ""), @@ -244,8 +245,28 @@ fn write_instruction(w: &mut Write, ExtractLane { lane, arg, .. } => writeln!(w, " {}, {}", arg, lane), IntCompare { cond, args, .. } => writeln!(w, " {}, {}, {}", cond, args[0], args[1]), FloatCompare { cond, args, .. } => writeln!(w, " {}, {}, {}", cond, args[0], args[1]), - Jump { ref data, .. } => writeln!(w, " {}", data), - Branch { ref data, .. } => writeln!(w, " {}", data), + Jump { destination, ref args, .. } => { + if args.is_empty() { + writeln!(w, " {}", destination) + } else { + writeln!(w, + " {}({})", + destination, + DisplayValues(args.as_slice(pool))) + } + } + Branch { destination, ref args, .. } => { + let args = args.as_slice(pool); + if args.len() == 1 { + writeln!(w, " {}, {}", args[0], destination) + } else { + writeln!(w, + " {}, {}({})", + args[0], + destination, + DisplayValues(&args[1..])) + } + } BranchTable { arg, table, .. } => writeln!(w, " {}, {}", arg, table), Call { func_ref, ref args, .. } => { writeln!(w, @@ -253,8 +274,13 @@ fn write_instruction(w: &mut Write, func_ref, DisplayValues(args.as_slice(&func.dfg.value_lists))) } - IndirectCall { ref data, .. } => { - writeln!(w, " {}, {}({})", data.sig_ref, data.arg, data.varargs) + IndirectCall { sig_ref, ref args, .. } => { + let args = args.as_slice(pool); + writeln!(w, + " {}, {}({})", + sig_ref, + args[0], + DisplayValues(&args[1..])) } Return { ref data, .. } => { if data.varargs.is_empty() { diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index c009270f54..26fbe234a0 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -15,8 +15,7 @@ use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, - TernaryOverflowData, JumpData, BranchData, IndirectCallData, - ReturnData, ReturnRegData}; + TernaryOverflowData, ReturnData, ReturnRegData}; use cretonne::isa::{self, TargetIsa, Encoding}; use cretonne::settings; use testfile::{TestFile, Details, Comment}; @@ -200,24 +199,22 @@ impl<'a> Context<'a> { self.map.rewrite_values(&mut data.args, loc)?; } - InstructionData::Jump { ref mut data, .. } => { - self.map.rewrite_ebb(&mut data.destination, loc)?; - self.map.rewrite_values(&mut data.varargs, loc)?; + InstructionData::Jump { ref mut destination, ref mut args, .. } => { + self.map.rewrite_ebb(destination, loc)?; + self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; } - InstructionData::Branch { ref mut data, .. } => { - self.map.rewrite_value(&mut data.arg, loc)?; - self.map.rewrite_ebb(&mut data.destination, loc)?; - self.map.rewrite_values(&mut data.varargs, loc)?; + InstructionData::Branch { ref mut destination, ref mut args, .. } => { + self.map.rewrite_ebb(destination, loc)?; + self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; } InstructionData::Call { ref mut args, .. } => { self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; } - InstructionData::IndirectCall { ref mut data, .. } => { - self.map.rewrite_value(&mut data.arg, loc)?; - self.map.rewrite_values(&mut data.varargs, loc)?; + InstructionData::IndirectCall { ref mut args, .. } => { + self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; } InstructionData::Return { ref mut data, .. } => { @@ -1209,7 +1206,7 @@ impl<'a> Parser<'a> { // TBD: If it is defined in another block, the type should have been specified // explicitly. It is unfortunate that the correctness of IL depends on the // layout of the blocks. - let ctrl_src_value = inst_data.typevar_operand() + let ctrl_src_value = inst_data.typevar_operand(&ctx.function.dfg.value_lists) .expect("Constraints <-> Format inconsistency"); ctx.function.dfg.value_type(match ctx.map.get_value(ctrl_src_value) { Some(v) => v, @@ -1429,10 +1426,8 @@ impl<'a> Parser<'a> { InstructionData::Jump { opcode: opcode, ty: VOID, - data: Box::new(JumpData { - destination: ebb_num, - varargs: args, - }), + destination: ebb_num, + args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } } InstructionFormat::Branch => { @@ -1443,11 +1438,8 @@ impl<'a> Parser<'a> { InstructionData::Branch { opcode: opcode, ty: VOID, - data: Box::new(BranchData { - arg: ctrl_arg, - destination: ebb_num, - varargs: args, - }), + destination: ebb_num, + args: args.into_value_list(&[ctrl_arg], &mut ctx.function.dfg.value_lists), } } InstructionFormat::InsertLane => { @@ -1511,7 +1503,7 @@ impl<'a> Parser<'a> { ty: VOID, second_result: None.into(), func_ref: func_ref, - args: args.into_value_list(&mut ctx.function.dfg.value_lists), + args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } } InstructionFormat::IndirectCall => { @@ -1526,11 +1518,8 @@ impl<'a> Parser<'a> { opcode: opcode, ty: VOID, second_result: None.into(), - data: Box::new(IndirectCallData { - sig_ref: sig_ref, - arg: callee, - varargs: args, - }), + sig_ref: sig_ref, + args: args.into_value_list(&[callee], &mut ctx.function.dfg.value_lists), } } InstructionFormat::Return => { From d301fb9f2b3fce6b289ec0e680fee5c2157154c6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 9 Mar 2017 15:56:33 -0800 Subject: [PATCH 0566/3084] Convert return formats to value lists. With the Return and ReturnReg formats converted to using value lists for storing their arguments, thee are no remaining instruction formats with variable argument lists in boxed storage. The Return and ReturnReg formats are also going to be merged since they are identical now. --- lib/cretonne/meta/base/formats.py | 4 ++-- lib/cretonne/meta/cdsl/formats.py | 4 ++++ lib/cretonne/src/ir/builder.rs | 2 +- lib/cretonne/src/ir/instructions.rs | 32 ++--------------------------- lib/cretonne/src/write.rs | 18 ++++++++-------- lib/reader/src/parser.rs | 18 +++++++--------- 6 files changed, 25 insertions(+), 53 deletions(-) diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 9f7a805940..c72d5034f8 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -51,8 +51,8 @@ Call = InstructionFormat( IndirectCall = InstructionFormat( sig_ref, VALUE, VARIABLE_ARGS, multiple_results=True, value_list=True) -Return = InstructionFormat(VARIABLE_ARGS, boxed_storage=True) -ReturnReg = InstructionFormat(VALUE, VARIABLE_ARGS, boxed_storage=True) +Return = InstructionFormat(VARIABLE_ARGS, value_list=True) +ReturnReg = InstructionFormat(VALUE, VARIABLE_ARGS, value_list=True) # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index 422d84c296..98f6964e5c 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -63,6 +63,10 @@ class InstructionFormat(object): self.value_operands = tuple( i for i, k in enumerate(self.kinds) if k is VALUE) + # We require a value list for storage of variable arguments. + if VARIABLE_ARGS in self.kinds: + assert self.has_value_list, "Need a value list for variable args" + # The typevar_operand argument must point to a 'value' operand. self.typevar_operand = kwargs.get('typevar_operand', None) # type: int if self.typevar_operand is not None: diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index ebb8ae2350..fa15587445 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -5,7 +5,7 @@ use ir::{types, instructions}; use ir::{InstructionData, DataFlowGraph, Cursor}; -use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, VariableArgs, SigRef, FuncRef, ValueList}; +use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, ValueList}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, ImmVector}; use ir::condcodes::{IntCC, FloatCC}; diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index d7393c9293..a3cd4509da 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -228,12 +228,12 @@ pub enum InstructionData { Return { opcode: Opcode, ty: Type, - data: Box, + args: ValueList, }, ReturnReg { opcode: Opcode, ty: Type, - data: Box, + args: ValueList, }, } @@ -331,34 +331,6 @@ impl Display for TernaryOverflowData { } } -/// Payload of a return instruction. -#[derive(Clone, Debug)] -pub struct ReturnData { - /// Dynamically sized array containing return values. - pub varargs: VariableArgs, -} - -/// Payload of a return instruction. -#[derive(Clone, Debug)] -pub struct ReturnRegData { - /// Return address. - pub arg: Value, - /// Dynamically sized array containing return values. - pub varargs: VariableArgs, -} - -impl ReturnRegData { - /// Get references to the arguments. - pub fn arguments(&self) -> [&[Value]; 2] { - [ref_slice(&self.arg), &self.varargs] - } - - /// Get mutable references to the arguments. - pub fn arguments_mut(&mut self) -> [&mut [Value]; 2] { - [ref_slice_mut(&mut self.arg), &mut self.varargs] - } -} - /// Analyzing an instruction. /// /// Avoid large matches on instruction formats by using the methods defined here to examine diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 5e8968c48b..59c97d9283 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -282,19 +282,19 @@ fn write_instruction(w: &mut Write, args[0], DisplayValues(&args[1..])) } - Return { ref data, .. } => { - if data.varargs.is_empty() { + Return { ref args, .. } => { + if args.is_empty() { writeln!(w, "") } else { - writeln!(w, " {}", data.varargs) + writeln!(w, + " {}", + DisplayValues(args.as_slice(&func.dfg.value_lists))) } } - ReturnReg { ref data, .. } => { - if data.varargs.is_empty() { - writeln!(w, " {}", data.arg) - } else { - writeln!(w, " {}, {}", data.arg, data.varargs) - } + ReturnReg { ref args, .. } => { + writeln!(w, + " {}", + DisplayValues(args.as_slice(&func.dfg.value_lists))) } } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 26fbe234a0..72f43a043d 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -15,7 +15,7 @@ use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, - TernaryOverflowData, ReturnData, ReturnRegData}; + TernaryOverflowData}; use cretonne::isa::{self, TargetIsa, Encoding}; use cretonne::settings; use testfile::{TestFile, Details, Comment}; @@ -217,13 +217,12 @@ impl<'a> Context<'a> { self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; } - InstructionData::Return { ref mut data, .. } => { - self.map.rewrite_values(&mut data.varargs, loc)?; + InstructionData::Return { ref mut args, .. } => { + self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; } - InstructionData::ReturnReg { ref mut data, .. } => { - self.map.rewrite_value(&mut data.arg, loc)?; - self.map.rewrite_values(&mut data.varargs, loc)?; + InstructionData::ReturnReg { ref mut args, .. } => { + self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; } } } @@ -1527,7 +1526,7 @@ impl<'a> Parser<'a> { InstructionData::Return { opcode: opcode, ty: VOID, - data: Box::new(ReturnData { varargs: args }), + args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } } InstructionFormat::ReturnReg => { @@ -1540,10 +1539,7 @@ impl<'a> Parser<'a> { InstructionData::ReturnReg { opcode: opcode, ty: VOID, - data: Box::new(ReturnRegData { - arg: raddr, - varargs: args, - }), + args: args.into_value_list(&[raddr], &mut ctx.function.dfg.value_lists), } } InstructionFormat::BranchTable => { From ec5ee70a5ce3bb4017eb19c82690fdd8e555974d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 9 Mar 2017 19:12:00 -0800 Subject: [PATCH 0567/3084] Remove some has_value_list workarounds. Now that all variable_args formats use has_value_list, we can remove some workarounds that allowed the old boxed_storage form. --- lib/cretonne/meta/cdsl/operands.py | 2 +- lib/cretonne/meta/gen_instr.py | 10 +--------- 2 files changed, 2 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index f7ec85eafa..277f2597b0 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -61,7 +61,7 @@ VARIABLE_ARGS = OperandKind( passed to an extended basic block, or a variable number of results returned from an instruction. """, - default_member='varargs') + rust_type='&[Value]') # Instances of immediate operand types are provided in the diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 376bfc0d56..b452daff2e 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -510,10 +510,7 @@ def gen_format_constructor(iform, fmt): # Normal operand arguments. for idx, kind in enumerate(iform.kinds): - if kind is cdsl.operands.VARIABLE_ARGS and iform.has_value_list: - args.append('op{}: &[Value]'.format(idx, kind.rust_type)) - else: - args.append('op{}: {}'.format(idx, kind.rust_type)) + args.append('op{}: {}'.format(idx, kind.rust_type)) proto = '{}({})'.format(iform.name, ', '.join(args)) proto += " -> (Inst, &'f mut DataFlowGraph)" @@ -575,8 +572,6 @@ def gen_member_inits(iform, fmt): # Immediates and entity references. for idx, member in enumerate(iform.members): - if iform.has_value_list and member == 'varargs': - continue if member: fmt.line('{}: op{},'.format(member, idx)) @@ -606,9 +601,6 @@ def gen_inst_builder(inst, fmt): t = 'T{}{}'.format(1 + len(tmpl_types), op.kind.name) tmpl_types.append('{}: Into<{}>'.format(t, op.kind.rust_type)) into_args.append(op.name) - elif (inst.format.has_value_list and - op.kind is cdsl.operands.VARIABLE_ARGS): - t = '&[Value]' else: t = op.kind.rust_type args.append('{}: {}'.format(op.name, t)) From f3d74854948cb8698cbe59953b548e11b5a873d3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 9 Mar 2017 21:03:52 -0800 Subject: [PATCH 0568/3084] Python InstructionFormat refactoring. Make some changes that will make it easier to get rid of the 'value_operands' and 'members' fields in the Python InstructionFormat class. This is necessary to be able to combine instruction formats that all use a value list representation, but with different fixed value operands. The goal is to eventually identify formats by a new signature: (multiple_results, imm_kinds, num_value_operands) Start by adding new fields: - imm_members and imm_kinds are lists describing the format operands, excluding any values and variable_args operands. - num_value_operands is the number of fixed value operands, or None in a has_value-list format. Use these new members in preference to the old ones where possible. --- lib/cretonne/meta/cdsl/formats.py | 26 ++++++++++++++++++++++---- lib/cretonne/meta/gen_instr.py | 30 +++++++----------------------- lib/cretonne/meta/gen_legalizer.py | 5 ++--- 3 files changed, 31 insertions(+), 30 deletions(-) diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index 98f6964e5c..506f0d7322 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -57,16 +57,23 @@ class InstructionFormat(object): self.has_value_list = kwargs.get('value_list', False) self.boxed_storage = kwargs.get('boxed_storage', False) self.members = list() # type: List[str] + + # Struct member names for the immediate operands. All other instruction + # operands are values or variable argument lists. They are all handled + # specially. + self.imm_members = list() # type: List[str] + # Operand kinds for the immediate operands. + self.imm_kinds = list() # type: List[OperandKind] + # The number of value operands stored in the format, or `None` when + # `has_value_list` is set. + self.num_value_operands = 0 + self.kinds = tuple(self._process_member_names(kinds)) # Which of self.kinds are `value`? self.value_operands = tuple( i for i, k in enumerate(self.kinds) if k is VALUE) - # We require a value list for storage of variable arguments. - if VARIABLE_ARGS in self.kinds: - assert self.has_value_list, "Need a value list for variable args" - # The typevar_operand argument must point to a 'value' operand. self.typevar_operand = kwargs.get('typevar_operand', None) # type: int if self.typevar_operand is not None: @@ -102,6 +109,17 @@ class InstructionFormat(object): k = arg else: member, k = arg + + # We define 'immediate' as not a value or variable arguments. + if k is VALUE: + self.num_value_operands += 1 + elif k is VARIABLE_ARGS: + # We require a value list for storage of variable arguments. + assert self.has_value_list, "Need a value list" + else: + self.imm_kinds.append(k) + self.imm_members.append(member) + self.members.append(member) yield k diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index b452daff2e..073891807f 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -84,20 +84,11 @@ def gen_arguments_method(fmt, is_mut): .format(n, mut, mut, as_slice)) continue - has_varargs = cdsl.operands.VARIABLE_ARGS in f.kinds - # Formats with both fixed and variable arguments delegate to - # the data struct. We need to work around borrow checker quirks - # when extracting two mutable references. - if has_varargs and len(f.value_operands) > 0: - fmt.line( - '{} {{ ref {}data, .. }} => data.{}(),' - .format(n, mut, method)) - continue # Fixed args. - if len(f.value_operands) == 0: + if f.num_value_operands == 0: arg = '&{}[]'.format(mut) capture = '' - elif len(f.value_operands) == 1: + elif f.num_value_operands == 1: if f.boxed_storage: capture = 'ref {}data, '.format(mut) arg = '{}(&{}data.arg)'.format(rslice, mut) @@ -111,16 +102,9 @@ def gen_arguments_method(fmt, is_mut): else: capture = 'ref {}args, '.format(mut) arg = 'args' - # Varargs. - if cdsl.operands.VARIABLE_ARGS in f.kinds: - varg = '&{}data.varargs'.format(mut) - capture = 'ref {}data, '.format(mut) - else: - varg = '&{}[]'.format(mut) - fmt.line( - '{} {{ {} .. }} => [{}, {}],' - .format(n, capture, arg, varg)) + '{} {{ {} .. }} => [{}, &{}[]],' + .format(n, capture, arg, mut)) def gen_instruction_data_impl(fmt): @@ -219,7 +203,7 @@ def gen_instruction_data_impl(fmt): fmt.line( '{} {{ ref args, .. }} => ' 'args.get({}, pool),'.format(n, i)) - elif len(f.value_operands) == 1: + elif f.num_value_operands == 1: # We have a single value operand called 'arg'. if f.boxed_storage: fmt.line( @@ -564,9 +548,9 @@ def gen_member_inits(iform, fmt): if iform.has_value_list: # Value-list formats put *all* arguments in the list. fmt.line('args: vlist,') - elif len(iform.value_operands) == 1: + elif iform.num_value_operands == 1: fmt.line('arg: op{},'.format(iform.value_operands[0])) - elif len(iform.value_operands) > 1: + elif iform.num_value_operands > 1: fmt.line('args: [{}],'.format( ', '.join('op{}'.format(i) for i in iform.value_operands))) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 983871ac40..3199b3741f 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -46,9 +46,8 @@ def unwrap_inst(iref, node, fmt): fmt.line('ref data,') else: # Fields are encoded directly. - for m in iform.members: - if m: - fmt.line('{},'.format(m)) + for m in iform.imm_members: + fmt.line('{},'.format(m)) if nvops == 1: fmt.line('arg,') elif nvops > 1: From 618fefb7dacfe51afe42adc2d5a78f7a1b0e2e6b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 9 Mar 2017 22:04:13 -0800 Subject: [PATCH 0569/3084] Simplify the arguments() return type. Now that variable arguments are always stored in a value list with the fixed arguments, we no longer need the arcane [&[Value]; 2] return type. Arguments are always stored contiguously, so just return a &[Value] slice. Also remove the each_arg() methods which were just trying to make it easier to work with the old slice pair. --- lib/cretonne/meta/gen_instr.py | 11 +++++------ lib/cretonne/src/ir/instructions.rs | 24 ------------------------ lib/cretonne/src/regalloc/coloring.rs | 2 +- lib/cretonne/src/regalloc/liveness.rs | 4 ++-- lib/cretonne/src/write.rs | 2 +- 5 files changed, 9 insertions(+), 34 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 073891807f..f5d5d07b74 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -67,7 +67,7 @@ def gen_arguments_method(fmt, is_mut): with fmt.indented( 'pub fn {f}<\'a>(&\'a {m}self, pool: &\'a {m}ValueListPool) -> ' - '[&{m}[Value]; 2] {{' + '&{m}[Value] {{' .format(f=method, m=mut), '}'): with fmt.indented('match *self {', '}'): for f in InstructionFormat.all_formats: @@ -79,9 +79,8 @@ def gen_arguments_method(fmt, is_mut): if f.has_value_list: arg = ''.format(mut) fmt.line( - '{} {{ ref {}args, .. }} => ' - '[ &{}[], args.{}(pool) ],' - .format(n, mut, mut, as_slice)) + '{} {{ ref {}args, .. }} => args.{}(pool),' + .format(n, mut, as_slice)) continue # Fixed args. @@ -103,8 +102,8 @@ def gen_arguments_method(fmt, is_mut): capture = 'ref {}args, '.format(mut) arg = 'args' fmt.line( - '{} {{ {} .. }} => [{}, &{}[]],' - .format(n, capture, arg, mut)) + '{} {{ {} .. }} => {},' + .format(n, capture, arg)) def gen_instruction_data_impl(fmt): diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index a3cd4509da..4cabfe6698 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -336,30 +336,6 @@ impl Display for TernaryOverflowData { /// Avoid large matches on instruction formats by using the methods defined here to examine /// instructions. impl InstructionData { - /// Execute a closure once for each argument to this instruction. - /// See also the `arguments()` method. - pub fn each_arg(&self, pool: &ValueListPool, mut func: F) - where F: FnMut(Value) - { - for part in &self.arguments(pool) { - for &arg in part.iter() { - func(arg); - } - } - } - - /// Execute a closure with a mutable reference to each argument to this instruction. - /// See also the `arguments_mut()` method. - pub fn each_arg_mut(&mut self, pool: &mut ValueListPool, mut func: F) - where F: FnMut(&mut Value) - { - for part in &mut self.arguments_mut(pool) { - for arg in part.iter_mut() { - func(arg); - } - } - } - /// Return information about the destination of a branch or jump instruction. /// /// Any instruction that can transfer control to another EBB reveals its possible destinations diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 6549c4356f..24f8c0c7e1 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -299,7 +299,7 @@ impl<'a> Context<'a> { } ConstraintKind::Tied(arg_index) => { // This def must use the same register as a fixed instruction argument. - let arg = dfg[inst].arguments(&dfg.value_lists)[0][arg_index as usize]; + let arg = dfg[inst].arguments(&dfg.value_lists)[arg_index as usize]; let loc = locations[arg]; *locations.ensure(lv.value) = loc; // Mark the reused register. It's not really clear if we support tied diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 49db4a04cd..582cbae48a 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -318,7 +318,7 @@ impl Liveness { let mut operand_constraints = recipe_constraints.get(recipe).map(|c| c.ins).unwrap_or(&[]).iter(); - func.dfg[inst].each_arg(&func.dfg.value_lists, |arg| { + for &arg in func.dfg[inst].arguments(&func.dfg.value_lists) { // Get the live range, create it as a dead range if necessary. let lr = get_or_create(&mut self.ranges, arg, func, recipe_constraints); @@ -333,7 +333,7 @@ impl Liveness { if let Some(constraint) = operand_constraints.next() { lr.affinity.merge(constraint, ®_info); } - }); + } } } } diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 59c97d9283..8f120f447a 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -157,7 +157,7 @@ fn type_suffix(func: &Function, inst: Inst) -> Option { // Write out any value aliases appearing in `inst`. fn write_value_aliases(w: &mut Write, func: &Function, inst: Inst, indent: usize) -> Result { - for &arg in func.dfg[inst].arguments(&func.dfg.value_lists).iter().flat_map(|x| x.iter()) { + for &arg in func.dfg[inst].arguments(&func.dfg.value_lists) { let resolved = func.dfg.resolve_aliases(arg); if resolved != arg { writeln!(w, "{1:0$}{2} -> {3}", indent, "", arg, resolved)?; From 4ad8362e09b4bf3f542cdf50920429664bf8c692 Mon Sep 17 00:00:00 2001 From: Davide Italiano Date: Fri, 10 Mar 2017 09:04:41 -0800 Subject: [PATCH 0570/3084] Initial B-tree interface. --- lib/cretonne/src/btree.rs | 98 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 lib/cretonne/src/btree.rs diff --git a/lib/cretonne/src/btree.rs b/lib/cretonne/src/btree.rs new file mode 100644 index 0000000000..cd34c466b5 --- /dev/null +++ b/lib/cretonne/src/btree.rs @@ -0,0 +1,98 @@ +use std::marker::PhantomData; + +// A Node reference is a direct index to an element of the pool. +type NodeRef = u32; + +/// A B-tree data structure which nodes are allocated from a pool. +pub struct BTree { + index: NodeRef, + unused1: PhantomData, + unused2: PhantomData, +} + +/// An enum representing a B-tree node. +/// Keys and values are required to implement Default. +enum Node { + Inner { + size: u8, + keys: [K; 7], + nodes: [NodeRef; 8], + }, + Leaf { + size: u8, + keys: [K; 7], + values: [V; 7], + }, +} + +/// Memory pool for nodes. +struct NodePool { + // The array containing the nodes. + data: Vec>, + + // A free list + freelist: Vec, +} + +impl NodePool { + /// Create a new NodePool. + pub fn new() -> NodePool { + NodePool { + data: Vec::new(), + freelist: Vec::new(), + } + } + + /// Get a B-tree node. + pub fn get(&self, index: u32) -> Option<&Node> { + unimplemented!() + } +} + +impl BTree { + /// Search for `key` and return a `Cursor` that either points at `key` or the position where it would be inserted. + pub fn search(&mut self, key: K) -> Cursor { + unimplemented!() + } +} + +pub struct Cursor<'a, K: 'a, V: 'a> { + pool: &'a mut NodePool, + height: usize, + path: [(NodeRef, u8); 16], +} + +impl<'a, K: Default, V: Default> Cursor<'a, K, V> { + /// The key at the cursor position. Returns `None` when the cursor points off the end. + pub fn key(&self) -> Option { + unimplemented!() + } + + /// The value at the cursor position. Returns `None` when the cursor points off the end. + pub fn value(&self) -> Option<&V> { + unimplemented!() + } + + /// Move to the next element. + /// Returns `false` if that moves the cursor off the end. + pub fn next(&mut self) -> bool { + unimplemented!() + } + + /// Move to the previous element. + /// Returns `false` if this moves the cursor before the beginning. + pub fn prev(&mut self) -> bool { + unimplemented!() + } + + /// Insert a `(key, value)` pair at the cursor position. + /// It is an error to insert a key that would be out of order at this position. + pub fn insert(&mut self, key: K, value: V) { + unimplemented!() + } + + /// Remove the current element. + pub fn remove(&mut self) { + unimplemented!() + } +} From 515e34f22110c1d7e3c9e4461a7d38050d08fae5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 10 Mar 2017 08:25:02 -0800 Subject: [PATCH 0571/3084] Clean up lifetimes a bit. No functional change. --- lib/cretonne/meta/gen_instr.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index f5d5d07b74..23f28efeb7 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -117,9 +117,9 @@ def gen_instruction_data_impl(fmt): - `pub fn opcode(&self) -> Opcode` - `pub fn first_type(&self) -> Type` - `pub fn second_result(&self) -> Option` - - `pub fn second_result_mut<'a>(&'a mut self) - -> Option<&'a mut PackedOption>` - - `pub fn arguments(&self) -> (&[Value], &[Value])` + - `pub fn second_result_mut(&mut self) -> Option<&mut PackedOption>` + - `pub fn arguments(&self, &pool) -> &[Value]` + - `pub fn arguments_mut(&mut self, &pool) -> &mut [Value]` """ # The `opcode` and `first_type` methods simply read the `opcode` and `ty` @@ -172,8 +172,8 @@ def gen_instruction_data_impl(fmt): fmt.doc_comment('Mutable reference to second result value, if any.') with fmt.indented( - "pub fn second_result_mut<'a>(&'a mut self)" + - " -> Option<&'a mut PackedOption> {", '}'): + "pub fn second_result_mut(&mut self)" + + " -> Option<&mut PackedOption> {", '}'): with fmt.indented('match *self {', '}'): for f in InstructionFormat.all_formats: if f.multiple_results: From 054edeb76543c71e312ee3d14235a07504130899 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 10 Mar 2017 09:23:55 -0800 Subject: [PATCH 0572/3084] Add value_opnums and imm_opnums fields to Instruction. These two tuples contain operand indexes of the explicit value operands and immediate operands respectively. We can no longer use the instruction format value_operands field. --- lib/cretonne/meta/cdsl/instructions.py | 11 ++++++++++- lib/cretonne/meta/cdsl/operands.py | 11 +++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index fcce1684fd..861e6f0883 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -104,9 +104,18 @@ class Instruction(object): self.ins = self._to_operand_tuple(ins) self.outs = self._to_operand_tuple(outs) self.format = InstructionFormat.lookup(self.ins, self.outs) - # Indexes into outs for value results. Others are `variable_args`. + + # Indexes into `self.outs` for value results. + # Other results are `variable_args`. self.value_results = tuple( i for i, o in enumerate(self.outs) if o.is_value()) + # Indexes into `self.ins` for value operands. + self.value_opnums = tuple( + i for i, o in enumerate(self.ins) if o.is_value()) + # Indexes into `self.ins` for non-value operands. + self.imm_opnums = tuple( + i for i, o in enumerate(self.ins) if o.is_immediate()) + self._verify_polymorphic() for attr in Instruction.ATTRIBS: setattr(self, attr, not not kwargs.get(attr, False)) diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index 277f2597b0..dafa18de35 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -157,3 +157,14 @@ class Operand(object): Is this an SSA value operand? """ return self.kind is VALUE + + def is_immediate(self): + # type: () -> bool + """ + Is this an immediate operand? + + Note that this includes both `ImmediateKind` operands *and* entity + references. It is any operand that doesn't represent a value + dependency. + """ + return self.kind is not VALUE and self.kind is not VARIABLE_ARGS From d9c61373e2eb8d351882b727943cc102df72d7e4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 10 Mar 2017 09:05:53 -0800 Subject: [PATCH 0573/3084] Change InstBuilder low-level format constructor signatures. The per-instruction format low-level constructors in InstBuilder should be independent of the relative ordering of value and immediate operands in order to prepare for the future instruction format merger. Reorder their arguments such that all the immediate operands are placed before the value operands. For instruction formats that use a value list representation, just take a single ValueList argument. The value lists are created by the individual instruction constructors. This means that the format constructor doesn't care how many of the instructions operands are 'fixed' and how many are 'variable' arguments. --- lib/cretonne/meta/cdsl/operands.py | 7 +++ lib/cretonne/meta/gen_instr.py | 94 +++++++++++++++++------------- 2 files changed, 62 insertions(+), 39 deletions(-) diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index dafa18de35..49e3e588af 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -158,6 +158,13 @@ class Operand(object): """ return self.kind is VALUE + def is_varargs(self): + # type: () -> bool + """ + Is this a VARIABLE_ARGS operand? + """ + return self.kind is VARIABLE_ARGS + def is_immediate(self): # type: () -> bool """ diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 23f28efeb7..b9e46172d7 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -7,7 +7,6 @@ import constant_hash from unique_table import UniqueTable, UniqueSeqTable from cdsl import camel_case from cdsl.operands import ImmediateKind -import cdsl.types from cdsl.formats import InstructionFormat from cdsl.instructions import Instruction # noqa @@ -477,11 +476,7 @@ def gen_format_constructor(iform, fmt): """ # Construct method arguments. - if iform.has_value_list: - args = ['mut self'] - else: - args = ['self'] - args.append('opcode: Opcode') + args = ['self', 'opcode: Opcode'] if iform.multiple_results: args.append('ctrl_typevar: Type') @@ -491,9 +486,18 @@ def gen_format_constructor(iform, fmt): args.append('result_type: Type') result_type = 'result_type' - # Normal operand arguments. - for idx, kind in enumerate(iform.kinds): - args.append('op{}: {}'.format(idx, kind.rust_type)) + # Normal operand arguments. Start with the immediate operands. + for kind, name in zip(iform.imm_kinds, iform.imm_members): + args.append('{}: {}'.format(name, kind.rust_type)) + # Then the value operands. + if iform.has_value_list: + # Take all value arguments as a finished value list. The value lists + # are created by the individual instruction constructors. + args.append('args: ValueList') + else: + # Take a fixed number of value operands. + for i in range(iform.num_value_operands): + args.append('arg{}: Value'.format(i)) proto = '{}({})'.format(iform.name, ', '.join(args)) proto += " -> (Inst, &'f mut DataFlowGraph)" @@ -501,21 +505,6 @@ def gen_format_constructor(iform, fmt): fmt.doc_comment(str(iform)) fmt.line('#[allow(non_snake_case)]') with fmt.indented('fn {} {{'.format(proto), '}'): - # Start by constructing a value list with *all* the arguments. - if iform.has_value_list: - fmt.line('let mut vlist = ValueList::default();') - with fmt.indented('{', '}'): - fmt.line( - 'let pool = ' - '&mut self.data_flow_graph_mut().value_lists;') - for idx, kind in enumerate(iform.kinds): - if kind is cdsl.operands.VALUE: - fmt.line('vlist.push(op{}, pool);'.format(idx)) - elif kind is cdsl.operands.VARIABLE_ARGS: - fmt.line( - 'vlist.extend(op{}.iter().cloned(), pool);' - .format(idx)) - # Generate the instruction data. with fmt.indented( 'let data = InstructionData::{} {{'.format(iform.name), '};'): @@ -543,20 +532,19 @@ def gen_member_inits(iform, fmt): Emit member initializers for an `iform` instruction. """ - # Values first. - if iform.has_value_list: - # Value-list formats put *all* arguments in the list. - fmt.line('args: vlist,') - elif iform.num_value_operands == 1: - fmt.line('arg: op{},'.format(iform.value_operands[0])) - elif iform.num_value_operands > 1: - fmt.line('args: [{}],'.format( - ', '.join('op{}'.format(i) for i in iform.value_operands))) + # Immediate operands. + # We have local variables with the same names as the members. + for member in iform.imm_members: + fmt.line('{}: {},'.format(member, member)) - # Immediates and entity references. - for idx, member in enumerate(iform.members): - if member: - fmt.line('{}: op{},'.format(member, idx)) + # Value operands. + if iform.has_value_list: + fmt.line('args: args,') + elif iform.num_value_operands == 1: + fmt.line('arg: arg0,') + elif iform.num_value_operands > 1: + args = ('arg{}'.format(i) for i in range(iform.num_value_operands)) + fmt.line('args: [{}],'.format(', '.join(args))) def gen_inst_builder(inst, fmt): @@ -570,7 +558,10 @@ def gen_inst_builder(inst, fmt): """ # Construct method arguments. - args = ['self'] + if inst.format.has_value_list: + args = ['mut self'] + else: + args = ['self'] # The controlling type variable will be inferred from the input values if # possible. Otherwise, it is the first method argument. @@ -645,7 +636,32 @@ def gen_inst_builder(inst, fmt): inst.outs[inst.value_results[0]] .typevar.singleton_type.rust_name()) - args.extend(op.name for op in inst.ins) + # Now add all of the immediate operands to the constructor arguments. + for opnum in inst.imm_opnums: + args.append(inst.ins[opnum].name) + + # Finally, the value operands. + if inst.format.has_value_list: + # We need to build a value list with all the arguments. + fmt.line('let mut vlist = ValueList::default();') + args.append('vlist') + with fmt.indented('{', '}'): + fmt.line( + 'let pool = ' + '&mut self.data_flow_graph_mut().value_lists;') + for op in inst.ins: + if op.is_value(): + fmt.line('vlist.push({}, pool);'.format(op.name)) + elif op.is_varargs(): + fmt.line( + 'vlist.extend({}.iter().cloned(), pool);' + .format(op.name)) + else: + # With no value list, we're guaranteed to just have a set of fixed + # value operands. + for opnum in inst.value_opnums: + args.append(inst.ins[opnum].name) + # Call to the format constructor, fcall = 'self.{}({})'.format(inst.format.name, ', '.join(args)) From af3fa6b83d8103fa2bed5b51aae6e859363e8e62 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 10 Mar 2017 09:56:37 -0800 Subject: [PATCH 0574/3084] Avoid using 'members' and 'value_operands' in the legalizer. Use the new Instruction fields instead. --- lib/cretonne/meta/gen_legalizer.py | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 3199b3741f..7c36700dc7 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -35,7 +35,7 @@ def unwrap_inst(iref, node, fmt): fmt.comment('Unwrap {}'.format(node)) expr = node.expr iform = expr.inst.format - nvops = len(iform.value_operands) + nvops = iform.num_value_operands # The tuple of locals we're extracting is `expr.args`. with fmt.indented( @@ -50,31 +50,31 @@ def unwrap_inst(iref, node, fmt): fmt.line('{},'.format(m)) if nvops == 1: fmt.line('arg,') - elif nvops > 1: + elif nvops != 0: fmt.line('args,') fmt.line('..') fmt.outdented_line('} = dfg[inst] {') # Generate the values for the tuple. outs = list() prefix = 'data.' if iform.boxed_storage else '' - for i, m in enumerate(iform.members): - if m: - outs.append(prefix + m) - else: - # This is a value operand. + for opnum, op in enumerate(expr.inst.ins): + if op.is_immediate(): + n = expr.inst.imm_opnums.index(opnum) + outs.append(prefix + iform.imm_members[n]) + elif op.is_value(): if nvops == 1: arg = prefix + 'arg' else: - arg = '{}args[{}]'.format( - prefix, iform.value_operands.index(i)) + n = expr.inst.value_opnums.index(opnum) + arg = '{}args[{}]'.format(prefix, n) outs.append('dfg.resolve_aliases({})'.format(arg)) fmt.line('({})'.format(', '.join(outs))) fmt.outdented_line('} else {') fmt.line('unreachable!("bad instruction format")') # Get the types of any variables where it is needed. - for i in iform.value_operands: - v = expr.args[i] + for opnum in expr.inst.value_opnums: + v = expr.args[opnum] if isinstance(v, Var) and v.has_free_typevar(): fmt.line('let typeof_{0} = dfg.value_type({0});'.format(v)) From da693f72e24e721c64c6f3e1011dccce44a3560b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 10 Mar 2017 10:04:30 -0800 Subject: [PATCH 0575/3084] Eliminate InstructionFormat.members. This field is no longer needed. We can use imm_members to get the names of immediate fields, and num_value_operands to access values. --- lib/cretonne/meta/cdsl/formats.py | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index 506f0d7322..088461d1d2 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -56,7 +56,6 @@ class InstructionFormat(object): self.multiple_results = kwargs.get('multiple_results', False) self.has_value_list = kwargs.get('value_list', False) self.boxed_storage = kwargs.get('boxed_storage', False) - self.members = list() # type: List[str] # Struct member names for the immediate operands. All other instruction # operands are values or variable argument lists. They are all handled @@ -101,6 +100,8 @@ class InstructionFormat(object): pair. The member names correspond to members in the Rust `InstructionData` data structure. + Update the fields `num_value_operands`, `imm_kinds`, and `imm_members`. + Yields the operand kinds. """ for arg in kinds: @@ -120,25 +121,25 @@ class InstructionFormat(object): self.imm_kinds.append(k) self.imm_members.append(member) - self.members.append(member) yield k def __str__(self): # type: () -> str - args = ', '.join('{}: {}'.format(m, k) if m else str(k) - for m, k in zip(self.members, self.kinds)) - return '{}({})'.format(self.name, args) + args = ', '.join('{}: {}'.format(m, k) + for m, k in zip(self.imm_members, self.imm_kinds)) + return '{}({}, values={})'.format( + self.name, args, self.num_value_operands) def __getattr__(self, attr): # type: (str) -> FormatField """ - Make instruction format members available as attributes. + Make immediate instruction format members available as attributes. Each non-value format member becomes a corresponding `FormatField` attribute. """ try: - i = self.members.index(attr) + i = self.imm_members.index(attr) except ValueError: raise AttributeError( '{} is neither a {} member or a ' @@ -192,7 +193,7 @@ class FormatField(object): data type. :param format: Parent `InstructionFormat`. - :param operand: Operand number in parent. + :param operand: Immediate operand number in parent. :param name: Member name in `InstructionData` variant. """ From cdb4cce3dc9dae1108efd9c02035c86992b49a88 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 10 Mar 2017 10:38:38 -0800 Subject: [PATCH 0576/3084] Change index domain for typevar_operand. An instruction format is now seen as having two separate operand lists: immediates and values. Change InstructionFormat.typevar_operand to be a pure index into the value list. --- lib/cretonne/meta/cdsl/formats.py | 16 +++++++++------- lib/cretonne/meta/cdsl/instructions.py | 3 ++- lib/cretonne/meta/gen_instr.py | 12 ++++++------ 3 files changed, 17 insertions(+), 14 deletions(-) diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index 088461d1d2..4b7d512ff1 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -39,9 +39,10 @@ class InstructionFormat(object): :param boxed_storage: Set to `True` is this instruction format requires a `data: Box<...>` pointer to additional storage in its `InstructionData` variant. - :param typevar_operand: Index of the input operand that is used to infer - the controlling type variable. By default, this is the first `value` - operand. + :param typevar_operand: Index of the value input operand that is used to + infer the controlling type variable. By default, this is `0`, the first + `value` operand. The index is relative to the values only, ignoring + immediate operands. """ # Map (multiple_results, kind, kind, ...) -> InstructionFormat @@ -76,11 +77,12 @@ class InstructionFormat(object): # The typevar_operand argument must point to a 'value' operand. self.typevar_operand = kwargs.get('typevar_operand', None) # type: int if self.typevar_operand is not None: - assert self.kinds[self.typevar_operand] is VALUE, \ - "typevar_operand must indicate a 'value' operand" - elif len(self.value_operands) > 0: + if not self.has_value_list: + assert self.typevar_operand < self.num_value_operands, \ + "typevar_operand must indicate a 'value' operand" + elif self.num_value_operands != 0: # Default to the first 'value' operand, if there is one. - self.typevar_operand = self.value_operands[0] + self.typevar_operand = 0 # Compute a signature for the global registry. sig = (self.multiple_results, self.kinds) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 861e6f0883..85581edde2 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -168,7 +168,8 @@ class Instruction(object): typevar_error = None if self.format.typevar_operand is not None: try: - tv = self.ins[self.format.typevar_operand].typevar + opnum = self.value_opnums[self.format.typevar_operand] + tv = self.ins[opnum].typevar if tv is tv.free_typevar(): self.other_typevars = self._verify_ctrl_typevar(tv) self.ctrl_typevar = tv diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index b9e46172d7..567cfa16f6 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -197,7 +197,7 @@ def gen_instruction_data_impl(fmt): fmt.line(n + ' { .. } => None,') elif f.has_value_list: # We keep all arguments in a value list. - i = f.value_operands.index(f.typevar_operand) + i = f.typevar_operand fmt.line( '{} {{ ref args, .. }} => ' 'args.get({}, pool),'.format(n, i)) @@ -211,9 +211,7 @@ def gen_instruction_data_impl(fmt): else: # We have multiple value operands and an array `args`. # Which `args` index to use? - # Map from index into f.kinds into f.value_operands - # index. - i = f.value_operands.index(f.typevar_operand) + i = f.typevar_operand if f.boxed_storage: fmt.line( n + @@ -280,9 +278,10 @@ def gen_opcodes(groups, fmt): # Document polymorphism. if i.is_polymorphic: if i.use_typevar_operand: + opnum = i.value_opnums[i.format.typevar_operand] fmt.doc_comment( 'Type inferred from {}.' - .format(i.ins[i.format.typevar_operand])) + .format(i.ins[opnum])) # Enum variant itself. if is_first_opcode: fmt.line(i.camel_name + ' = 1,') @@ -612,9 +611,10 @@ def gen_inst_builder(inst, fmt): args.append('types::VOID') elif inst.is_polymorphic: # Infer the controlling type variable from the input operands. + opnum = inst.value_opnums[inst.format.typevar_operand] fmt.line( 'let ctrl_typevar = self.data_flow_graph().value_type({});' - .format(inst.ins[inst.format.typevar_operand].name)) + .format(inst.ins[opnum].name)) if inst.format.multiple_results: # The format constructor will resolve the result types from the # type var. From dcdaeee4af0a5562d390c658669bf9444f7bb024 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 10 Mar 2017 10:25:18 -0800 Subject: [PATCH 0577/3084] Eliminate InstructionFormat.value_operands and .kinds. Part of the refactoring of instruction formats. This list is now stored in the instruction itself as value_opnums. --- lib/cretonne/meta/cdsl/formats.py | 8 ++------ lib/cretonne/meta/cdsl/instructions.py | 8 ++++---- lib/cretonne/meta/cdsl/isa.py | 7 +++++-- lib/cretonne/meta/cdsl/xform.py | 2 +- lib/cretonne/meta/gen_instr.py | 4 ++-- 5 files changed, 14 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index 4b7d512ff1..aa7d384b7c 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -68,11 +68,7 @@ class InstructionFormat(object): # `has_value_list` is set. self.num_value_operands = 0 - self.kinds = tuple(self._process_member_names(kinds)) - - # Which of self.kinds are `value`? - self.value_operands = tuple( - i for i, k in enumerate(self.kinds) if k is VALUE) + sig_kinds = tuple(self._process_member_names(kinds)) # The typevar_operand argument must point to a 'value' operand. self.typevar_operand = kwargs.get('typevar_operand', None) # type: int @@ -85,7 +81,7 @@ class InstructionFormat(object): self.typevar_operand = 0 # Compute a signature for the global registry. - sig = (self.multiple_results, self.kinds) + sig = (self.multiple_results, sig_kinds) if sig in InstructionFormat._registry: raise RuntimeError( "Format '{}' has the same signature as existing format '{}'" diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 85581edde2..30596fd1fb 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -154,7 +154,7 @@ class Instruction(object): variables. """ poly_ins = [ - i for i in self.format.value_operands + i for i in self.value_opnums if self.ins[i].typevar.free_typevar()] poly_outs = [ i for i, o in enumerate(self.outs) @@ -206,8 +206,8 @@ class Instruction(object): """ other_tvs = [] # Check value inputs. - for opidx in self.format.value_operands: - typ = self.ins[opidx].typevar + for opnum in self.value_opnums: + typ = self.ins[opnum].typevar tv = typ.free_typevar() # Non-polymorphic or derived form ctrl_typevar is OK. if tv is None or tv is ctrl_typevar: @@ -216,7 +216,7 @@ class Instruction(object): if typ is not tv: raise RuntimeError( "{}: type variable {} must be derived from {}" - .format(self.ins[opidx], typ.name, ctrl_typevar)) + .format(self.ins[opnum], typ.name, ctrl_typevar)) # Other free type variables can only be used once each. if tv in other_tvs: raise RuntimeError( diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 7e4c09e58d..ef667015ce 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -176,7 +176,8 @@ class EncRecipe(object): self.number = None # type: int self.ins = self._verify_constraints(ins) - assert len(self.ins) == len(format.value_operands) + if not format.has_value_list: + assert len(self.ins) == format.num_value_operands self.outs = self._verify_constraints(outs) if len(self.outs) > 1: assert format.multiple_results @@ -193,7 +194,9 @@ class EncRecipe(object): if isinstance(c, int): # An integer constraint is bound to a value operand. # Check that it is in range. - assert c >= 0 and c < len(self.format.value_operands) + assert c >= 0 + if not format.has_value_list: + assert c < self.format.num_value_operands else: assert isinstance(c, RegClass) or isinstance(c, Register) return seq diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index deca293e50..21c6103bcd 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -224,7 +224,7 @@ class XForm(object): ctrl_var = d.defs[inst.value_results[0]] # Reconcile arguments with the requirements of `inst`. - for opnum in inst.format.value_operands: + for opnum in inst.value_opnums: inst_tv = inst.ins[opnum].typevar v = d.expr.args[opnum] if isinstance(v, Var): diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 567cfa16f6..e6cd265118 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -411,9 +411,9 @@ def gen_type_constraints(fmt, instrs): for idx in i.value_results: constraints.append( get_constraint(i.outs[idx], ctrl_typevar, type_sets)) - for idx in i.format.value_operands: + for opnum in i.value_opnums: constraints.append( - get_constraint(i.ins[idx], ctrl_typevar, type_sets)) + get_constraint(i.ins[opnum], ctrl_typevar, type_sets)) offset = operand_seqs.add(constraints) fixed_results = len(i.value_results) # Can the controlling type variable be inferred from the designated From c50e5f3f66078ca30dab741566768e3bbc355572 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 10 Mar 2017 11:09:49 -0800 Subject: [PATCH 0578/3084] Separate immediate and value operands in the instruction format. Instruction formats are now identified by a signature that doesn't include the ordering of value operands relative to immediate operands. This means that the BinaryRev instruction format becomes redundant, so delete it. The isub_imm instruction was the only one using that format. Rename it to irsub_imm to make it clear what it does now that it is printed as 'irsub_imm v2, 45'. --- cranelift/docs/langref.rst | 2 +- cranelift/filetests/parser/tiny.cton | 2 ++ lib/cretonne/meta/base/formats.py | 1 - lib/cretonne/meta/base/instructions.py | 14 ++++----- lib/cretonne/meta/cdsl/formats.py | 41 +++++++++++++++++--------- lib/cretonne/src/ir/instructions.rs | 7 ----- lib/cretonne/src/write.rs | 1 - lib/reader/src/parser.rs | 12 -------- 8 files changed, 35 insertions(+), 45 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 5fb42c44a2..cff747752a 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -685,7 +685,7 @@ Integer operations .. autoinst:: iadd_cout .. autoinst:: iadd_carry .. autoinst:: isub -.. autoinst:: isub_imm +.. autoinst:: irsub_imm .. autoinst:: isub_bin .. autoinst:: isub_bout .. autoinst:: isub_borrow diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 588223c521..7158441ed2 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -55,12 +55,14 @@ ebb0(vx0: i32, vx1: i32): v0 = icmp eq, vx0, vx1 v1 = icmp ult, vx0, vx1 v2 = icmp sge, vx0, vx1 + v3 = irsub_imm vx1, 45 } ; sameln: function icmp(i32, i32) { ; nextln: ebb0(vx0: i32, vx1: i32): ; nextln: v0 = icmp eq, vx0, vx1 ; nextln: v1 = icmp ult, vx0, vx1 ; nextln: v2 = icmp sge, vx0, vx1 +; nextln: v3 = irsub_imm vx1, 45 ; nextln: } ; Floating condition codes. diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index c72d5034f8..7ed047c330 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -22,7 +22,6 @@ UnarySplit = InstructionFormat(VALUE, multiple_results=True) Binary = InstructionFormat(VALUE, VALUE) BinaryImm = InstructionFormat(VALUE, imm64) -BinaryImmRev = InstructionFormat(imm64, VALUE) # Generate result + overflow flag. BinaryOverflow = InstructionFormat(VALUE, VALUE, multiple_results=True) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index a61f98cefe..41b6ff6c05 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -507,22 +507,18 @@ srem_imm = Instruction( allowed. """, ins=(x, Y), outs=a) -# Swap x and y for isub_imm. -X = Operand('X', imm64) -y = Operand('y', iB) +irsub_imm = Instruction( + 'irsub_imm', """ + Immediate reverse wrapping subtraction: :math:`a := Y - x \pmod{2^B}`. -isub_imm = Instruction( - 'isub_imm', """ - Immediate wrapping subtraction: :math:`a := X - y \pmod{2^B}`. - - Also works as integer negation when :math:`X = 0`. Use :inst:`iadd_imm` + Also works as integer negation when :math:`Y = 0`. Use :inst:`iadd_imm` with a negative immediate operand for the reverse immediate subtraction. Polymorphic over all scalar integer types, but does not support vector types. """, - ins=(X, y), outs=a) + ins=(x, Y), outs=a) # # Integer arithmetic with carry and/or borrow. diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index aa7d384b7c..910afb6052 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -25,6 +25,14 @@ class InstructionFormat(object): produced. Some instructions, like `call`, may have a variable number of results. + The instruction format stores two separate lists of operands: Immediates + and values. Immediate operands (including entity references) are + represented as explicit members in the `InstructionData` variants. The + value operands are stored differently, depending on how many there are. + Beyond a certain point, instruction formats switch to an external value + list for storing value arguments. Value lists can hold an arbitrary number + of values. + All instruction formats must be predefined in the :py:mod:`cretonne.formats` module. @@ -45,8 +53,8 @@ class InstructionFormat(object): immediate operands. """ - # Map (multiple_results, kind, kind, ...) -> InstructionFormat - _registry = dict() # type: Dict[Tuple[bool, Tuple[OperandKind, ...]], InstructionFormat] # noqa + # Map (multiple_results, imm_kinds, num_value_operands) -> format + _registry = dict() # type: Dict[Tuple[bool, Tuple[OperandKind, ...], int, bool], InstructionFormat] # noqa # All existing formats. all_formats = list() # type: List[InstructionFormat] @@ -62,13 +70,11 @@ class InstructionFormat(object): # operands are values or variable argument lists. They are all handled # specially. self.imm_members = list() # type: List[str] - # Operand kinds for the immediate operands. - self.imm_kinds = list() # type: List[OperandKind] # The number of value operands stored in the format, or `None` when # `has_value_list` is set. self.num_value_operands = 0 - - sig_kinds = tuple(self._process_member_names(kinds)) + # Operand kinds for the immediate operands. + self.imm_kinds = tuple(self._process_member_names(kinds)) # The typevar_operand argument must point to a 'value' operand. self.typevar_operand = kwargs.get('typevar_operand', None) # type: int @@ -81,7 +87,10 @@ class InstructionFormat(object): self.typevar_operand = 0 # Compute a signature for the global registry. - sig = (self.multiple_results, sig_kinds) + sig = ( + self.multiple_results, self.imm_kinds, + self.num_value_operands, + self.has_value_list) if sig in InstructionFormat._registry: raise RuntimeError( "Format '{}' has the same signature as existing format '{}'" @@ -98,9 +107,9 @@ class InstructionFormat(object): pair. The member names correspond to members in the Rust `InstructionData` data structure. - Update the fields `num_value_operands`, `imm_kinds`, and `imm_members`. + Update the fields `num_value_operands` and `imm_members`. - Yields the operand kinds. + Yields the immediate operand kinds. """ for arg in kinds: if isinstance(arg, OperandKind): @@ -116,16 +125,14 @@ class InstructionFormat(object): # We require a value list for storage of variable arguments. assert self.has_value_list, "Need a value list" else: - self.imm_kinds.append(k) self.imm_members.append(member) - - yield k + yield k def __str__(self): # type: () -> str args = ', '.join('{}: {}'.format(m, k) for m, k in zip(self.imm_members, self.imm_kinds)) - return '{}({}, values={})'.format( + return '{}(imms=({}), vals={})'.format( self.name, args, self.num_value_operands) def __getattr__(self, attr): @@ -162,7 +169,13 @@ class InstructionFormat(object): multiple_results = outs[0].kind == VARIABLE_ARGS else: multiple_results = len(outs) > 1 - sig = (multiple_results, tuple(op.kind for op in ins)) + + # Construct a signature. + imm_kinds = tuple(op.kind for op in ins if op.is_immediate()) + num_values = sum(1 for op in ins if op.is_value()) + has_varargs = (VARIABLE_ARGS in tuple(op.kind for op in ins)) + + sig = (multiple_results, imm_kinds, num_values, has_varargs) if sig not in InstructionFormat._registry: raise RuntimeError( "No instruction format matches ins = ({}){}".format( diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 4cabfe6698..09167907be 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -145,13 +145,6 @@ pub enum InstructionData { arg: Value, imm: Imm64, }, - // Same as `BinaryImm`, but the immediate is the left-hand-side operand. - BinaryImmRev { - opcode: Opcode, - ty: Type, - arg: Value, - imm: Imm64, - }, BinaryOverflow { opcode: Opcode, ty: Type, diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 8f120f447a..57d31d2e5c 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -237,7 +237,6 @@ fn write_instruction(w: &mut Write, UnarySplit { arg, .. } => writeln!(w, " {}", arg), Binary { args, .. } => writeln!(w, " {}, {}", args[0], args[1]), BinaryImm { arg, imm, .. } => writeln!(w, " {}, {}", arg, imm), - BinaryImmRev { imm, arg, .. } => writeln!(w, " {}, {}", imm, arg), BinaryOverflow { args, .. } => writeln!(w, " {}, {}", args[0], args[1]), Ternary { args, .. } => writeln!(w, " {}, {}, {}", args[0], args[1], args[2]), TernaryOverflow { ref data, .. } => writeln!(w, " {}", data), diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 72f43a043d..1e5e194785 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -177,7 +177,6 @@ impl<'a> Context<'a> { InstructionData::Unary { ref mut arg, .. } | InstructionData::UnarySplit { ref mut arg, .. } | InstructionData::BinaryImm { ref mut arg, .. } | - InstructionData::BinaryImmRev { ref mut arg, .. } | InstructionData::ExtractLane { ref mut arg, .. } | InstructionData::BranchTable { ref mut arg, .. } => { self.map.rewrite_value(arg, loc)?; @@ -1368,17 +1367,6 @@ impl<'a> Parser<'a> { imm: rhs, } } - InstructionFormat::BinaryImmRev => { - let lhs = self.match_imm64("expected immediate integer first operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; - let rhs = self.match_value("expected SSA value second operand")?; - InstructionData::BinaryImmRev { - opcode: opcode, - ty: VOID, - imm: lhs, - arg: rhs, - } - } InstructionFormat::BinaryOverflow => { let lhs = self.match_value("expected SSA value first operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; From 9fbfd0d2a6f1245a8a396960d26ecce889290c0f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 10 Mar 2017 11:57:49 -0800 Subject: [PATCH 0579/3084] Remove the vconst instruction and the UnaryImmVector format. No instruction sets actually have single instructions for materializing vector constants. You always need to use a constant pool. Cretonne doesn't have constant pools yet, but it will in the future, and that is how vector constants should be represented. --- cranelift/docs/langref.rst | 1 - lib/cretonne/meta/base/formats.py | 3 +-- lib/cretonne/meta/base/immediates.py | 6 ------ lib/cretonne/meta/base/instructions.py | 12 +----------- lib/cretonne/src/ir/builder.rs | 2 +- lib/cretonne/src/ir/immediates.rs | 6 ------ lib/cretonne/src/ir/instructions.rs | 24 +----------------------- lib/cretonne/src/write.rs | 1 - lib/reader/src/parser.rs | 6 +----- 9 files changed, 5 insertions(+), 56 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index cff747752a..b1262c0335 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -643,7 +643,6 @@ Constant materialization .. autoinst:: iconst .. autoinst:: f32const .. autoinst:: f64const -.. autoinst:: vconst Live range splitting -------------------- diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 7ed047c330..98547bbbd7 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -8,7 +8,7 @@ in this module. from __future__ import absolute_import from cdsl.formats import InstructionFormat from cdsl.operands import VALUE, VARIABLE_ARGS -from .immediates import imm64, uimm8, ieee32, ieee64, immvector, intcc, floatcc +from .immediates import imm64, uimm8, ieee32, ieee64, intcc, floatcc from .entities import ebb, sig_ref, func_ref, jump_table Nullary = InstructionFormat() @@ -17,7 +17,6 @@ Unary = InstructionFormat(VALUE) UnaryImm = InstructionFormat(imm64) UnaryIeee32 = InstructionFormat(ieee32) UnaryIeee64 = InstructionFormat(ieee64) -UnaryImmVector = InstructionFormat(immvector, boxed_storage=True) UnarySplit = InstructionFormat(VALUE, multiple_results=True) Binary = InstructionFormat(VALUE, VALUE) diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index 565dd26850..05a542b6d9 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -27,12 +27,6 @@ ieee32 = ImmediateKind('ieee32', 'A 32-bit immediate floating point number.') #: IEEE 754-2008 binary64 interchange format. ieee64 = ImmediateKind('ieee64', 'A 64-bit immediate floating point number.') -#: A large SIMD vector constant. -immvector = ImmediateKind( - 'immvector', - 'An immediate SIMD vector.', - rust_type='ImmVector') - #: A condition code for comparing integer values. #: #: This enumerated operand kind is used for the :cton:inst:`icmp` instruction diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 41b6ff6c05..4f5febb0c4 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -9,7 +9,7 @@ from cdsl.operands import Operand, VARIABLE_ARGS from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup from base.types import i8, f32, f64, b1 -from base.immediates import imm64, uimm8, ieee32, ieee64, immvector +from base.immediates import imm64, uimm8, ieee32, ieee64 from base.immediates import intcc, floatcc from base import entities import base.formats # noqa @@ -196,16 +196,6 @@ f64const = Instruction( """, ins=N, outs=a) -N = Operand('N', immvector) -a = Operand('a', TxN, doc='A constant vector value') -vconst = Instruction( - 'vconst', r""" - Vector constant (floating point or integer). - - Create a SIMD vector value where the lanes don't have to be identical. - """, - ins=N, outs=a) - # # Generics. # diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index fa15587445..b6d6a16b6b 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -6,7 +6,7 @@ use ir::{types, instructions}; use ir::{InstructionData, DataFlowGraph, Cursor}; use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, ValueList}; -use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, ImmVector}; +use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64}; use ir::condcodes::{IntCC, FloatCC}; /// Base trait for instruction builders. diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 5673af1163..79229fc3a0 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -434,12 +434,6 @@ impl FromStr for Ieee64 { } } -/// Arbitrary vector immediate. -/// -/// This kind of immediate can represent any kind of SIMD vector constant. -/// The representation is simply the sequence of bytes that would be used to store the vector. -pub type ImmVector = Vec; - #[cfg(test)] mod tests { use super::*; diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 09167907be..1292836d6d 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -11,7 +11,7 @@ use std::str::FromStr; use std::ops::{Deref, DerefMut}; use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef}; -use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, ImmVector}; +use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64}; use ir::condcodes::*; use ir::types; use ir::DataFlowGraph; @@ -123,11 +123,6 @@ pub enum InstructionData { ty: Type, imm: Ieee64, }, - UnaryImmVector { - opcode: Opcode, - ty: Type, - data: Box, - }, UnarySplit { opcode: Opcode, ty: Type, @@ -294,23 +289,6 @@ impl Default for VariableArgs { } } -/// Payload data for `vconst`. -#[derive(Clone, Debug)] -pub struct UnaryImmVectorData { - /// Raw vector data. - pub imm: ImmVector, -} - -impl Display for UnaryImmVectorData { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - write!(f, "#")?; - for b in &self.imm { - write!(f, "{:02x}", b)?; - } - Ok(()) - } -} - /// Payload data for ternary instructions with multiple results, such as `iadd_carry`. #[derive(Clone, Debug)] pub struct TernaryOverflowData { diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 57d31d2e5c..e6880de697 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -233,7 +233,6 @@ fn write_instruction(w: &mut Write, UnaryImm { imm, .. } => writeln!(w, " {}", imm), UnaryIeee32 { imm, .. } => writeln!(w, " {}", imm), UnaryIeee64 { imm, .. } => writeln!(w, " {}", imm), - UnaryImmVector { ref data, .. } => writeln!(w, " {}", data), UnarySplit { arg, .. } => writeln!(w, " {}", arg), Binary { args, .. } => writeln!(w, " {}, {}", args[0], args[1]), BinaryImm { arg, imm, .. } => writeln!(w, " {}, {}", arg, imm), diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 1e5e194785..bd7d65c3ee 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -171,8 +171,7 @@ impl<'a> Context<'a> { InstructionData::Nullary { .. } | InstructionData::UnaryImm { .. } | InstructionData::UnaryIeee32 { .. } | - InstructionData::UnaryIeee64 { .. } | - InstructionData::UnaryImmVector { .. } => {} + InstructionData::UnaryIeee64 { .. } => {} InstructionData::Unary { ref mut arg, .. } | InstructionData::UnarySplit { ref mut arg, .. } | @@ -1335,9 +1334,6 @@ impl<'a> Parser<'a> { imm: self.match_ieee64("expected immediate 64-bit float operand")?, } } - InstructionFormat::UnaryImmVector => { - unimplemented!(); - } InstructionFormat::UnarySplit => { InstructionData::UnarySplit { opcode: opcode, From 910e4e6174640525143c5b76b5b6f61d10ce1542 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 10 Mar 2017 12:17:12 -0800 Subject: [PATCH 0580/3084] Coalesce some formats into MultiAry. Allow some flexibility in the signature matching for instruction formats. In particular, look for a value list format as a second chance option. The Return, ReturnReg, and TernaryOverflow formats all fit the single MultiAry catch-all format for instructions without immediate operands. --- lib/cretonne/meta/base/formats.py | 9 +++-- lib/cretonne/meta/cdsl/formats.py | 28 +++++++++++---- lib/cretonne/meta/gen_legalizer.py | 6 ++-- lib/cretonne/meta/isa/riscv/recipes.py | 4 +-- lib/cretonne/src/ir/builder.rs | 2 +- lib/cretonne/src/ir/instructions.rs | 27 ++------------ lib/cretonne/src/write.rs | 24 +++++-------- lib/reader/src/parser.rs | 49 ++++---------------------- 8 files changed, 50 insertions(+), 99 deletions(-) diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 98547bbbd7..9d0cba38d9 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -30,9 +30,10 @@ BinaryOverflow = InstructionFormat(VALUE, VALUE, multiple_results=True) # The fma instruction has the same constraint on all inputs. Ternary = InstructionFormat(VALUE, VALUE, VALUE, typevar_operand=1) -# Carry in *and* carry out for `iadd_carry` and friends. -TernaryOverflow = InstructionFormat( - VALUE, VALUE, VALUE, multiple_results=True, boxed_storage=True) +# Catch-all for instructions with many outputs and inputs and no immediate +# operands. +MultiAry = InstructionFormat( + VARIABLE_ARGS, multiple_results=True, value_list=True) InsertLane = InstructionFormat(VALUE, ('lane', uimm8), VALUE) ExtractLane = InstructionFormat(VALUE, ('lane', uimm8)) @@ -49,8 +50,6 @@ Call = InstructionFormat( IndirectCall = InstructionFormat( sig_ref, VALUE, VARIABLE_ARGS, multiple_results=True, value_list=True) -Return = InstructionFormat(VARIABLE_ARGS, value_list=True) -ReturnReg = InstructionFormat(VALUE, VARIABLE_ARGS, value_list=True) # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index 910afb6052..b1784498ee 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -82,7 +82,7 @@ class InstructionFormat(object): if not self.has_value_list: assert self.typevar_operand < self.num_value_operands, \ "typevar_operand must indicate a 'value' operand" - elif self.num_value_operands != 0: + elif self.has_value_list or self.num_value_operands > 0: # Default to the first 'value' operand, if there is one. self.typevar_operand = 0 @@ -176,12 +176,26 @@ class InstructionFormat(object): has_varargs = (VARIABLE_ARGS in tuple(op.kind for op in ins)) sig = (multiple_results, imm_kinds, num_values, has_varargs) - if sig not in InstructionFormat._registry: - raise RuntimeError( - "No instruction format matches ins = ({}){}".format( - ", ".join(map(str, sig[1])), - "[multiple results]" if multiple_results else "")) - return InstructionFormat._registry[sig] + if sig in InstructionFormat._registry: + return InstructionFormat._registry[sig] + + # Try another value list format as an alternative. + sig = (True, imm_kinds, num_values, has_varargs) + if sig in InstructionFormat._registry: + return InstructionFormat._registry[sig] + + sig = (multiple_results, imm_kinds, 0, True) + if sig in InstructionFormat._registry: + return InstructionFormat._registry[sig] + + sig = (True, imm_kinds, 0, True) + if sig in InstructionFormat._registry: + return InstructionFormat._registry[sig] + + raise RuntimeError( + 'No instruction format matches multiple_results={},' + 'imms={}, vals={}, varargs={}'.format( + multiple_results, imm_kinds, num_values, has_varargs)) @staticmethod def extract_names(globs): diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 7c36700dc7..9da3e9a985 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -50,10 +50,12 @@ def unwrap_inst(iref, node, fmt): fmt.line('{},'.format(m)) if nvops == 1: fmt.line('arg,') - elif nvops != 0: - fmt.line('args,') + elif iform.has_value_list or nvops > 1: + fmt.line('ref args,') fmt.line('..') fmt.outdented_line('} = dfg[inst] {') + if iform.has_value_list: + fmt.line('let args = args.as_slice(&dfg.value_lists);') # Generate the values for the tuple. outs = list() prefix = 'data.' if iform.boxed_storage else '' diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index ea7e019eb4..0dce091713 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -11,7 +11,7 @@ instruction formats described in the reference: from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt -from base.formats import Binary, BinaryImm, ReturnReg +from base.formats import Binary, BinaryImm, MultiAry from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit @@ -86,4 +86,4 @@ I = EncRecipe( # I-type encoding for `jalr` as a return instruction. We won't use the # immediate offset. # The variable return values are not encoded. -Iret = EncRecipe('Iret', ReturnReg, ins=GPR, outs=()) +Iret = EncRecipe('Iret', MultiAry, ins=GPR, outs=()) diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index b6d6a16b6b..8880351d9b 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -3,7 +3,7 @@ //! A `Builder` provides a convenient interface for inserting instructions into a Cretonne //! function. Many of its methods are generated from the meta language instruction definitions. -use ir::{types, instructions}; +use ir::types; use ir::{InstructionData, DataFlowGraph, Cursor}; use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, ValueList}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64}; diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 1292836d6d..de34a708c9 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -151,11 +151,11 @@ pub enum InstructionData { ty: Type, args: [Value; 3], }, - TernaryOverflow { + MultiAry { opcode: Opcode, ty: Type, second_result: PackedOption, - data: Box, + args: ValueList, }, InsertLane { opcode: Opcode, @@ -213,16 +213,6 @@ pub enum InstructionData { sig_ref: SigRef, args: ValueList, }, - Return { - opcode: Opcode, - ty: Type, - args: ValueList, - }, - ReturnReg { - opcode: Opcode, - ty: Type, - args: ValueList, - }, } /// A variable list of `Value` operands used for function call arguments and passing arguments to @@ -289,19 +279,6 @@ impl Default for VariableArgs { } } -/// Payload data for ternary instructions with multiple results, such as `iadd_carry`. -#[derive(Clone, Debug)] -pub struct TernaryOverflowData { - /// Value arguments. - pub args: [Value; 3], -} - -impl Display for TernaryOverflowData { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - write!(f, "{}, {}, {}", self.args[0], self.args[1], self.args[2]) - } -} - /// Analyzing an instruction. /// /// Avoid large matches on instruction formats by using the methods defined here to examine diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index e6880de697..1f9bfd4524 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -238,7 +238,15 @@ fn write_instruction(w: &mut Write, BinaryImm { arg, imm, .. } => writeln!(w, " {}, {}", arg, imm), BinaryOverflow { args, .. } => writeln!(w, " {}, {}", args[0], args[1]), Ternary { args, .. } => writeln!(w, " {}, {}, {}", args[0], args[1], args[2]), - TernaryOverflow { ref data, .. } => writeln!(w, " {}", data), + MultiAry { ref args, .. } => { + if args.is_empty() { + writeln!(w, "") + } else { + writeln!(w, + " {}", + DisplayValues(args.as_slice(&func.dfg.value_lists))) + } + } InsertLane { lane, args, .. } => writeln!(w, " {}, {}, {}", args[0], lane, args[1]), ExtractLane { lane, arg, .. } => writeln!(w, " {}, {}", arg, lane), IntCompare { cond, args, .. } => writeln!(w, " {}, {}, {}", cond, args[0], args[1]), @@ -280,20 +288,6 @@ fn write_instruction(w: &mut Write, args[0], DisplayValues(&args[1..])) } - Return { ref args, .. } => { - if args.is_empty() { - writeln!(w, "") - } else { - writeln!(w, - " {}", - DisplayValues(args.as_slice(&func.dfg.value_lists))) - } - } - ReturnReg { ref args, .. } => { - writeln!(w, - " {}", - DisplayValues(args.as_slice(&func.dfg.value_lists))) - } } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index bd7d65c3ee..c982a5be0c 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -14,8 +14,7 @@ use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotDa use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; -use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs, - TernaryOverflowData}; +use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs}; use cretonne::isa::{self, TargetIsa, Encoding}; use cretonne::settings; use testfile::{TestFile, Details, Comment}; @@ -193,8 +192,8 @@ impl<'a> Context<'a> { self.map.rewrite_values(args, loc)?; } - InstructionData::TernaryOverflow { ref mut data, .. } => { - self.map.rewrite_values(&mut data.args, loc)?; + InstructionData::MultiAry { ref mut args, .. } => { + self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; } InstructionData::Jump { ref mut destination, ref mut args, .. } => { @@ -214,14 +213,6 @@ impl<'a> Context<'a> { InstructionData::IndirectCall { ref mut args, .. } => { self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; } - - InstructionData::Return { ref mut args, .. } => { - self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; - } - - InstructionData::ReturnReg { ref mut args, .. } => { - self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; - } } } } @@ -1388,18 +1379,13 @@ impl<'a> Parser<'a> { args: [ctrl_arg, true_arg, false_arg], } } - InstructionFormat::TernaryOverflow => { - // Names here refer to the `iadd_carry` instruction. - let lhs = self.match_value("expected SSA value first operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; - let rhs = self.match_value("expected SSA value second operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; - let cin = self.match_value("expected SSA value third operand")?; - InstructionData::TernaryOverflow { + InstructionFormat::MultiAry => { + let args = self.parse_value_list()?; + InstructionData::MultiAry { opcode: opcode, ty: VOID, second_result: None.into(), - data: Box::new(TernaryOverflowData { args: [lhs, rhs, cin] }), + args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } } InstructionFormat::Jump => { @@ -1505,27 +1491,6 @@ impl<'a> Parser<'a> { args: args.into_value_list(&[callee], &mut ctx.function.dfg.value_lists), } } - InstructionFormat::Return => { - let args = self.parse_value_list()?; - InstructionData::Return { - opcode: opcode, - ty: VOID, - args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), - } - } - InstructionFormat::ReturnReg => { - let raddr = self.match_value("expected SSA value return address operand")?; - let args = if self.optional(Token::Comma) { - self.parse_value_list()? - } else { - VariableArgs::new() - }; - InstructionData::ReturnReg { - opcode: opcode, - ty: VOID, - args: args.into_value_list(&[raddr], &mut ctx.function.dfg.value_lists), - } - } InstructionFormat::BranchTable => { let arg = self.match_value("expected SSA value operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; From 667d6f9381ea540c14b4f42552c49b7b19216a17 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 10 Mar 2017 12:43:05 -0800 Subject: [PATCH 0581/3084] Remove the value_list and boxed_storage format flags. The value_list flag can be inferred from the presence of VARIABLE_ARGS in the operand list. The boxed_storage flag is obsolete. We don't need boxed storage anywhere no that we have value lists instead. --- lib/cretonne/meta/base/formats.py | 11 ++++---- lib/cretonne/meta/cdsl/formats.py | 17 +++--------- lib/cretonne/meta/gen_encoding.py | 15 ++++------ lib/cretonne/meta/gen_instr.py | 44 +++++++----------------------- lib/cretonne/meta/gen_legalizer.py | 25 +++++++---------- 5 files changed, 35 insertions(+), 77 deletions(-) diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 9d0cba38d9..5ae806b936 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -32,8 +32,7 @@ Ternary = InstructionFormat(VALUE, VALUE, VALUE, typevar_operand=1) # Catch-all for instructions with many outputs and inputs and no immediate # operands. -MultiAry = InstructionFormat( - VARIABLE_ARGS, multiple_results=True, value_list=True) +MultiAry = InstructionFormat(VARIABLE_ARGS, multiple_results=True) InsertLane = InstructionFormat(VALUE, ('lane', uimm8), VALUE) ExtractLane = InstructionFormat(VALUE, ('lane', uimm8)) @@ -41,15 +40,15 @@ ExtractLane = InstructionFormat(VALUE, ('lane', uimm8)) IntCompare = InstructionFormat(intcc, VALUE, VALUE) FloatCompare = InstructionFormat(floatcc, VALUE, VALUE) -Jump = InstructionFormat(ebb, VARIABLE_ARGS, value_list=True) -Branch = InstructionFormat(VALUE, ebb, VARIABLE_ARGS, value_list=True) +Jump = InstructionFormat(ebb, VARIABLE_ARGS) +Branch = InstructionFormat(VALUE, ebb, VARIABLE_ARGS) BranchTable = InstructionFormat(VALUE, jump_table) Call = InstructionFormat( - func_ref, VARIABLE_ARGS, multiple_results=True, value_list=True) + func_ref, VARIABLE_ARGS, multiple_results=True) IndirectCall = InstructionFormat( sig_ref, VALUE, VARIABLE_ARGS, - multiple_results=True, value_list=True) + multiple_results=True) # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index b1784498ee..6233dbf7a6 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -42,11 +42,6 @@ class InstructionFormat(object): enums. :param multiple_results: Set to `True` if this instruction format allows more than one result to be produced. - :param value_list: Set to `True` if this instruction format uses a - `ValueList` member to store its value operands. - :param boxed_storage: Set to `True` is this instruction format requires a - `data: Box<...>` pointer to additional storage in its `InstructionData` - variant. :param typevar_operand: Index of the value input operand that is used to infer the controlling type variable. By default, this is `0`, the first `value` operand. The index is relative to the values only, ignoring @@ -63,8 +58,6 @@ class InstructionFormat(object): # type: (*Union[OperandKind, Tuple[str, OperandKind]], **Any) -> None # noqa self.name = kwargs.get('name', None) # type: str self.multiple_results = kwargs.get('multiple_results', False) - self.has_value_list = kwargs.get('value_list', False) - self.boxed_storage = kwargs.get('boxed_storage', False) # Struct member names for the immediate operands. All other instruction # operands are values or variable argument lists. They are all handled @@ -73,6 +66,8 @@ class InstructionFormat(object): # The number of value operands stored in the format, or `None` when # `has_value_list` is set. self.num_value_operands = 0 + # Does this format use a value list for storing value operands? + self.has_value_list = False # Operand kinds for the immediate operands. self.imm_kinds = tuple(self._process_member_names(kinds)) @@ -122,8 +117,7 @@ class InstructionFormat(object): if k is VALUE: self.num_value_operands += 1 elif k is VARIABLE_ARGS: - # We require a value list for storage of variable arguments. - assert self.has_value_list, "Need a value list" + self.has_value_list = True else: self.imm_members.append(member) yield k @@ -234,7 +228,4 @@ class FormatField(object): def rust_name(self): # type: () -> str - if self.format.boxed_storage: - return 'data.' + self.name - else: - return self.name + return self.name diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index c2849c1d6f..978519efaf 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -76,15 +76,12 @@ def emit_instp(instp, fmt): iform = instp.predicate_context() # Which fields do we need in the InstructionData pattern match? - if iform.boxed_storage: - fields = 'ref data' - else: - # Collect the leaf predicates - leafs = set() - instp.predicate_leafs(leafs) - # All the leafs are FieldPredicate instances. Here we just care about - # the field names. - fields = ', '.join(sorted(set(p.field.name for p in leafs))) + # Collect the leaf predicates. + leafs = set() + instp.predicate_leafs(leafs) + # All the leafs are FieldPredicate instances. Here we just care about + # the field names. + fields = ', '.join(sorted(set(p.field.name for p in leafs))) with fmt.indented('{} => {{'.format(instp.number), '}'): with fmt.indented( diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index e6cd265118..c05cb51078 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -87,19 +87,11 @@ def gen_arguments_method(fmt, is_mut): arg = '&{}[]'.format(mut) capture = '' elif f.num_value_operands == 1: - if f.boxed_storage: - capture = 'ref {}data, '.format(mut) - arg = '{}(&{}data.arg)'.format(rslice, mut) - else: - capture = 'ref {}arg, '.format(mut) - arg = '{}(arg)'.format(rslice) + capture = 'ref {}arg, '.format(mut) + arg = '{}(arg)'.format(rslice) else: - if f.boxed_storage: - capture = 'ref {}data, '.format(mut) - arg = '&{}data.args'.format(mut) - else: - capture = 'ref {}args, '.format(mut) - arg = 'args' + capture = 'ref {}args, '.format(mut) + arg = 'args' fmt.line( '{} {{ {} .. }} => {},' .format(n, capture, arg)) @@ -203,25 +195,15 @@ def gen_instruction_data_impl(fmt): 'args.get({}, pool),'.format(n, i)) elif f.num_value_operands == 1: # We have a single value operand called 'arg'. - if f.boxed_storage: - fmt.line( - n + ' { ref data, .. } => Some(data.arg),') - else: - fmt.line(n + ' { arg, .. } => Some(arg),') + fmt.line(n + ' { arg, .. } => Some(arg),') else: # We have multiple value operands and an array `args`. # Which `args` index to use? i = f.typevar_operand - if f.boxed_storage: - fmt.line( - n + - ' { ref data, .. } => ' + - ('Some(data.args[{}]),'.format(i))) - else: - fmt.line( - n + - ' {{ ref args, .. }} => Some(args[{}]),' - .format(i)) + fmt.line( + n + + ' {{ ref args, .. }} => Some(args[{}]),' + .format(i)) fmt.doc_comment( """ @@ -511,13 +493,7 @@ def gen_format_constructor(iform, fmt): fmt.line('ty: {},'.format(result_type)) if iform.multiple_results: fmt.line('second_result: None.into(),') - if iform.boxed_storage: - with fmt.indented( - 'data: Box::new(instructions::{}Data {{' - .format(iform.name), '}),'): - gen_member_inits(iform, fmt) - else: - gen_member_inits(iform, fmt) + gen_member_inits(iform, fmt) # Create result values if necessary. if iform.multiple_results: diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 9da3e9a985..85aa715996 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -41,34 +41,29 @@ def unwrap_inst(iref, node, fmt): with fmt.indented( 'let ({}) = if let InstructionData::{} {{' .format(', '.join(map(str, expr.args)), iform.name), '};'): - if iform.boxed_storage: - # This format indirects to a largish `data` struct. - fmt.line('ref data,') - else: - # Fields are encoded directly. - for m in iform.imm_members: - fmt.line('{},'.format(m)) - if nvops == 1: - fmt.line('arg,') - elif iform.has_value_list or nvops > 1: - fmt.line('ref args,') + # Fields are encoded directly. + for m in iform.imm_members: + fmt.line('{},'.format(m)) + if nvops == 1: + fmt.line('arg,') + elif iform.has_value_list or nvops > 1: + fmt.line('ref args,') fmt.line('..') fmt.outdented_line('} = dfg[inst] {') if iform.has_value_list: fmt.line('let args = args.as_slice(&dfg.value_lists);') # Generate the values for the tuple. outs = list() - prefix = 'data.' if iform.boxed_storage else '' for opnum, op in enumerate(expr.inst.ins): if op.is_immediate(): n = expr.inst.imm_opnums.index(opnum) - outs.append(prefix + iform.imm_members[n]) + outs.append(iform.imm_members[n]) elif op.is_value(): if nvops == 1: - arg = prefix + 'arg' + arg = 'arg' else: n = expr.inst.value_opnums.index(opnum) - arg = '{}args[{}]'.format(prefix, n) + arg = 'args[{}]'.format(n) outs.append('dfg.resolve_aliases({})'.format(arg)) fmt.line('({})'.format(', '.join(outs))) fmt.outdented_line('} else {') From 46525b0153565e4b297f727cc48179af34eb17ad Mon Sep 17 00:00:00 2001 From: Davide Italiano Date: Fri, 10 Mar 2017 11:52:11 -0800 Subject: [PATCH 0582/3084] [B-tree] Initial comment to describe the design choices. --- lib/cretonne/src/btree.rs | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/lib/cretonne/src/btree.rs b/lib/cretonne/src/btree.rs index cd34c466b5..b6732788f7 100644 --- a/lib/cretonne/src/btree.rs +++ b/lib/cretonne/src/btree.rs @@ -1,3 +1,16 @@ +//! Generic B-Tree implementation. +//! +//! This module defines a `Btree` type which provides similar functionality to +//! `BtreeMap`, but with some important differences in the implementation: +//! +//! 1. Memory is allocated from a `NodePool` instead of the global heap. +//! 2. The footprint of a BTree is only 4 bytes. +//! 3. A BTree doesn't implement `Drop`, leaving it to the pool to manage memory. +//! +//! The node pool is intended to be used as a LIFO allocator. After building up a larger data +//! structure with many list references, the whole thing can be discarded quickly by clearing the +//! pool. + use std::marker::PhantomData; // A Node reference is a direct index to an element of the pool. From 11a0daa7fdf94541c169ed5c2c3c75892bbfdc55 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Sat, 11 Mar 2017 15:44:49 +0000 Subject: [PATCH 0583/3084] Define boolean conversion instructions. --- cranelift/docs/langref.rst | 4 ++ lib/cretonne/meta/base/instructions.py | 61 ++++++++++++++++++++++++++ 2 files changed, 65 insertions(+) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index b1262c0335..9a08da062d 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -819,6 +819,10 @@ Conversion operations --------------------- .. autoinst:: bitcast +.. autoinst:: breduce +.. autoinst:: bextend +.. autoinst:: bint +.. autoinst:: bmask .. autoinst:: ireduce .. autoinst:: uextend .. autoinst:: sextend diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 4f5febb0c4..e0efafb1f5 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -1063,6 +1063,67 @@ bitcast = Instruction( """, ins=x, outs=a) +Bool = TypeVar( + 'Bool', + 'A scalar or vector boolean type', + bools=True, simd=True) +BoolTo = TypeVar( + 'BoolTo', + 'A smaller boolean type with the same number of lanes', + bools=True, simd=True) + +x = Operand('x', Bool) +a = Operand('a', BoolTo) + +breduce = Instruction( + 'breduce', r""" + Convert `x` to a smaller boolean type in the platform-defined way. + + The result type must have the same number of vector lanes as the input, + and each lane must not have more bits that the input lanes. If the + input and output types are the same, this is a no-op. + """, ins=x, outs=a) + +BoolTo = TypeVar( + 'BoolTo', + 'A larger boolean type with the same number of lanes', + bools=True, simd=True) + +x = Operand('x', Bool) +a = Operand('a', BoolTo) + +bextend = Instruction( + 'bextend', r""" + Convert `x` to a larger boolean type in the platform-defined way. + + The result type must have the same number of vector lanes as the input, + and each lane must not have fewer bits that the input lanes. If the + input and output types are the same, this is a no-op. + """, ins=x, outs=a) + +IntTo = TypeVar( + 'IntTo', 'An integer type with the same number of lanes', + ints=True, simd=True) + +x = Operand('x', Bool) +a = Operand('a', IntTo) + +bint = Instruction( + 'bint', r""" + Convert `x` to an integer. + + True maps to 1 and false maps to 0. The result type must have the same + number of vector lanes as the input. + """, ins=x, outs=a) + +bmask = Instruction( + 'bmask', r""" + Convert `x` to an integer mask. + + True maps to all 1s and false maps to all 0s. The result type must have + the same number of vector lanes as the input. + """, ins=x, outs=a) + Int = TypeVar('Int', 'A scalar or vector integer type', ints=True, simd=True) IntTo = TypeVar( 'IntTo', 'A smaller integer type with the same number of lanes', From 1e1a1a87973f4e1458abfcd44d07e2ca110da2b7 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Sun, 12 Mar 2017 15:20:06 +0000 Subject: [PATCH 0584/3084] Added ControlFlowGraph::recompute_ebb for incremental CFG updates. --- lib/cretonne/src/cfg.rs | 124 ++++++++++++++++++++++++++++++---------- 1 file changed, 94 insertions(+), 30 deletions(-) diff --git a/lib/cretonne/src/cfg.rs b/lib/cretonne/src/cfg.rs index c3cf19efe4..f445d2c8f1 100644 --- a/lib/cretonne/src/cfg.rs +++ b/lib/cretonne/src/cfg.rs @@ -27,6 +27,7 @@ use ir::{Function, Inst, Ebb}; use ir::instructions::BranchInfo; use entity_map::{EntityMap, Keys}; use std::collections::HashSet; +use std::mem; /// A basic block denoted by its enclosing Ebb and last instruction. pub type BasicBlock = (Ebb, Inst); @@ -74,22 +75,49 @@ impl ControlFlowGraph { self.data.resize(func.dfg.num_ebbs()); for ebb in &func.layout { - for inst in func.layout.ebb_insts(ebb) { - match func.dfg[inst].analyze_branch(&func.dfg.value_lists) { - BranchInfo::SingleDest(dest, _) => { + self.compute_ebb(func, ebb); + } + } + + fn compute_ebb(&mut self, func: &Function, ebb: Ebb) { + for inst in func.layout.ebb_insts(ebb) { + match func.dfg[inst].analyze_branch(&func.dfg.value_lists) { + BranchInfo::SingleDest(dest, _) => { + self.add_edge((ebb, inst), dest); + } + BranchInfo::Table(jt) => { + for (_, dest) in func.jump_tables[jt].entries() { self.add_edge((ebb, inst), dest); } - BranchInfo::Table(jt) => { - for (_, dest) in func.jump_tables[jt].entries() { - self.add_edge((ebb, inst), dest); - } - } - BranchInfo::NotABranch => {} } + BranchInfo::NotABranch => {} } } } + fn invalidate_ebb_successors(&mut self, ebb: Ebb) { + // Temporarily take ownership because we need mutable access to self.data inside the loop. + // Unfortunately borrowck cannot see that our mut accesses to predecessors don't alias + // our iteration over successors. + let mut successors = mem::replace(&mut self.data[ebb].successors, Vec::new()); + for suc in successors.iter().cloned() { + self.data[suc].predecessors.retain(|&(e, _)| e != ebb); + } + successors.clear(); + self.data[ebb].successors = successors; + } + + /// Recompute the control flow graph of `ebb`. + /// + /// This is for use after modifying instructions within a specific EBB. It recomputes all edges + /// from `ebb` while leaving edges to `ebb` intact. Its functionality a subset of that of the + /// more expensive `compute`, and should be used when we know we don't need to recompute the CFG + /// from scratch, but rather that our changes have been restricted to specific EBBs. + pub fn recompute_ebb(&mut self, func: &Function, ebb: Ebb) { + self.invalidate_ebb_successors(ebb); + self.compute_ebb(func, ebb); + } + fn add_edge(&mut self, from: BasicBlock, to: Ebb) { self.data[from.0].successors.push(to); self.data[to].predecessors.push(from); @@ -207,32 +235,68 @@ mod tests { cur.insert_ebb(ebb2); } - let cfg = ControlFlowGraph::with_function(&func); + let mut cfg = ControlFlowGraph::with_function(&func); - let ebb0_predecessors = cfg.get_predecessors(ebb0); - let ebb1_predecessors = cfg.get_predecessors(ebb1); - let ebb2_predecessors = cfg.get_predecessors(ebb2); + { + let ebb0_predecessors = cfg.get_predecessors(ebb0); + let ebb1_predecessors = cfg.get_predecessors(ebb1); + let ebb2_predecessors = cfg.get_predecessors(ebb2); - let ebb0_successors = cfg.get_successors(ebb0); - let ebb1_successors = cfg.get_successors(ebb1); - let ebb2_successors = cfg.get_successors(ebb2); + let ebb0_successors = cfg.get_successors(ebb0); + let ebb1_successors = cfg.get_successors(ebb1); + let ebb2_successors = cfg.get_successors(ebb2); - assert_eq!(ebb0_predecessors.len(), 0); - assert_eq!(ebb1_predecessors.len(), 2); - assert_eq!(ebb2_predecessors.len(), 2); + assert_eq!(ebb0_predecessors.len(), 0); + assert_eq!(ebb1_predecessors.len(), 2); + assert_eq!(ebb2_predecessors.len(), 2); - assert_eq!(ebb1_predecessors.contains(&(ebb0, jmp_ebb0_ebb1)), true); - assert_eq!(ebb1_predecessors.contains(&(ebb1, br_ebb1_ebb1)), true); - assert_eq!(ebb2_predecessors.contains(&(ebb0, br_ebb0_ebb2)), true); - assert_eq!(ebb2_predecessors.contains(&(ebb1, jmp_ebb1_ebb2)), true); + assert_eq!(ebb1_predecessors.contains(&(ebb0, jmp_ebb0_ebb1)), true); + assert_eq!(ebb1_predecessors.contains(&(ebb1, br_ebb1_ebb1)), true); + assert_eq!(ebb2_predecessors.contains(&(ebb0, br_ebb0_ebb2)), true); + assert_eq!(ebb2_predecessors.contains(&(ebb1, jmp_ebb1_ebb2)), true); - assert_eq!(ebb0_successors.len(), 2); - assert_eq!(ebb1_successors.len(), 2); - assert_eq!(ebb2_successors.len(), 0); + assert_eq!(ebb0_successors.len(), 2); + assert_eq!(ebb1_successors.len(), 2); + assert_eq!(ebb2_successors.len(), 0); - assert_eq!(ebb0_successors.contains(&ebb1), true); - assert_eq!(ebb0_successors.contains(&ebb2), true); - assert_eq!(ebb1_successors.contains(&ebb1), true); - assert_eq!(ebb1_successors.contains(&ebb2), true); + assert_eq!(ebb0_successors.contains(&ebb1), true); + assert_eq!(ebb0_successors.contains(&ebb2), true); + assert_eq!(ebb1_successors.contains(&ebb1), true); + assert_eq!(ebb1_successors.contains(&ebb2), true); + } + + // Change some instructions and recompute ebb0 + func.dfg.replace(br_ebb0_ebb2).brnz(cond, ebb1, &[]); + func.dfg.replace(jmp_ebb0_ebb1).return_(&[]); + cfg.recompute_ebb(&mut func, ebb0); + let br_ebb0_ebb1 = br_ebb0_ebb2; + + { + let ebb0_predecessors = cfg.get_predecessors(ebb0); + let ebb1_predecessors = cfg.get_predecessors(ebb1); + let ebb2_predecessors = cfg.get_predecessors(ebb2); + + let ebb0_successors = cfg.get_successors(ebb0); + let ebb1_successors = cfg.get_successors(ebb1); + let ebb2_successors = cfg.get_successors(ebb2); + + assert_eq!(ebb0_predecessors.len(), 0); + assert_eq!(ebb1_predecessors.len(), 2); + assert_eq!(ebb2_predecessors.len(), 1); + + assert_eq!(ebb1_predecessors.contains(&(ebb0, br_ebb0_ebb1)), true); + assert_eq!(ebb1_predecessors.contains(&(ebb1, br_ebb1_ebb1)), true); + assert_eq!(ebb2_predecessors.contains(&(ebb0, br_ebb0_ebb2)), false); + assert_eq!(ebb2_predecessors.contains(&(ebb1, jmp_ebb1_ebb2)), true); + + assert_eq!(ebb0_successors.len(), 1); + assert_eq!(ebb1_successors.len(), 2); + assert_eq!(ebb2_successors.len(), 0); + + assert_eq!(ebb0_successors.contains(&ebb1), true); + assert_eq!(ebb0_successors.contains(&ebb2), false); + assert_eq!(ebb1_successors.contains(&ebb1), true); + assert_eq!(ebb1_successors.contains(&ebb2), true); + } } } From 1baedc3ca5bc7e84888317c2c6837fad8cc89692 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 13 Mar 2017 14:07:14 -0700 Subject: [PATCH 0585/3084] Add take_value_list and put_value_list methods. Any code that needs to manipulate a variable argument list on an instruction will need to remove the instruction's value list first, change the list, and then put it back on the instruction. This is required to avoid fighting the borrow checker over mutable locks on the DataFlowGraph and its value list pool. Add a generated InstructionData::take_value_list() method which lifts out and existing value list and returns it, levaing an empty list in its place, like Option::take() does it. Add a generated InstructionData::put_value_list() which puts it back, verifying that no existing value list is overwritten. --- lib/cretonne/meta/gen_instr.py | 41 +++++++++++++++++++++++++++++++++ lib/cretonne/src/entity_list.rs | 8 +++++++ 2 files changed, 49 insertions(+) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index c05cb51078..e6eaa82c32 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -111,6 +111,8 @@ def gen_instruction_data_impl(fmt): - `pub fn second_result_mut(&mut self) -> Option<&mut PackedOption>` - `pub fn arguments(&self, &pool) -> &[Value]` - `pub fn arguments_mut(&mut self, &pool) -> &mut [Value]` + - `pub fn take_value_list(&mut self) -> Option` + - `pub fn put_value_list(&mut self, args: ValueList>` """ # The `opcode` and `first_type` methods simply read the `opcode` and `ty` @@ -221,6 +223,45 @@ def gen_instruction_data_impl(fmt): """) gen_arguments_method(fmt, True) + fmt.doc_comment( + """ + Take out the value list with all the value arguments and return + it. + + This leaves the value list in the instruction empty. Use + `put_value_list` to put the value list back. + """) + with fmt.indented( + 'pub fn take_value_list(&mut self) -> Option {', + '}'): + with fmt.indented('match *self {', '}'): + for f in InstructionFormat.all_formats: + n = 'InstructionData::' + f.name + if f.has_value_list: + fmt.line( + n + ' { ref mut args, .. } => Some(args.take()),') + fmt.line('_ => None,') + + fmt.doc_comment( + """ + Put back a value list. + + After removing a value list with `take_value_list()`, use this + method to put it back. It is required that this instruction has + a format that accepts a value list, and that the existing value + list is empty. This avoids leaking list pool memory. + """) + with fmt.indented( + 'pub fn put_value_list(&mut self, vlist: ValueList) {', '}'): + with fmt.indented('let args = match *self {', '};'): + for f in InstructionFormat.all_formats: + n = 'InstructionData::' + f.name + if f.has_value_list: + fmt.line(n + ' { ref mut args, .. } => args,') + fmt.line('_ => panic!("No value list: {:?}", self),') + fmt.line('assert!(args.is_empty(), "Value list already in use");') + fmt.line('*args = vlist;') + def collect_instr_groups(isas): seen = set() diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index 2b8ebb783e..64a0a4c830 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -47,6 +47,7 @@ //! reserved for the empty list which isn't allocated in the vector. use std::marker::PhantomData; +use std::mem; use entity_map::EntityRef; @@ -273,6 +274,13 @@ impl EntityList { self.index = 0; } + /// Take all elements from this list and return them as a new list. Leave this list empty. + /// + /// This is the equivalent of `Option::take()`. + pub fn take(&mut self) -> EntityList { + mem::replace(self, Default::default()) + } + /// Appends an element to the back of the list. pub fn push(&mut self, element: T, pool: &mut ListPool) { let idx = self.index as usize; From d3df198747dbe5b320ec2f09962628bc51bc1036 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 13 Mar 2017 16:37:48 -0700 Subject: [PATCH 0586/3084] Add a grow_at() method to EntityList. This method opens up a hole in the middle of a list where new elements can be written. --- lib/cretonne/src/entity_list.rs | 86 +++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index 64a0a4c830..423b0ed155 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -311,6 +311,41 @@ impl EntityList { } } + /// Grow list by adding `count` uninitialized elements at the end. + /// + /// Returns a mutable slice representing the whole list. + fn grow<'a>(&'a mut self, count: usize, pool: &'a mut ListPool) -> &'a mut [T] { + let idx = self.index as usize; + let new_len; + let block; + match pool.len_of(self) { + None => { + // This is an empty list. Allocate a block. + assert_eq!(idx, 0, "Invalid pool"); + if count == 0 { + return &mut []; + } + new_len = count; + block = pool.alloc(sclass_for_length(new_len)); + self.index = (block + 1) as u32; + } + Some(len) => { + // Do we need to reallocate? + let sclass = sclass_for_length(len); + new_len = len + count; + let new_sclass = sclass_for_length(new_len); + if new_sclass != sclass { + block = pool.realloc(idx - 1, sclass, new_sclass, len + 1); + self.index = (block + 1) as u32; + } else { + block = idx - 1; + } + } + } + pool.data[block] = T::new(new_len); + &mut pool.data[block + 1..block + 1 + new_len] + } + /// Appends multiple elements to the back of the list. pub fn extend(&mut self, elements: I, pool: &mut ListPool) where I: IntoIterator @@ -371,6 +406,20 @@ impl EntityList { // Finally adjust the length. pool.data[block] = T::new(len - 1); } + + /// Grow the list by inserting `count` elements at `index`. + /// + /// The new elements are not initialized, they will contain whatever happened to be in memory. + /// Since the memory comes from the pool, this will be either zero entity references or + /// whatever where in a previously deallocated list. + pub fn grow_at(&mut self, index: usize, count: usize, pool: &mut ListPool) { + let mut data = self.grow(count, pool); + + // Copy elements after `index` up. + for i in (index + count..data.len()).rev() { + data[i] = data[i - count]; + } + } } #[cfg(test)] @@ -530,4 +579,41 @@ mod tests { assert_eq!(list.as_slice(pool), &[]); assert!(list.is_empty()); } + + #[test] + fn growing() { + let pool = &mut ListPool::::new(); + let mut list = EntityList::::default(); + + let i1 = Inst::new(1); + let i2 = Inst::new(2); + let i3 = Inst::new(3); + let i4 = Inst::new(4); + + // This is not supposed to change the list. + list.grow_at(0, 0, pool); + assert_eq!(list.len(pool), 0); + assert!(list.is_empty()); + + list.grow_at(0, 2, pool); + assert_eq!(list.len(pool), 2); + + list.as_mut_slice(pool).copy_from_slice(&[i2, i3]); + + list.grow_at(1, 0, pool); + assert_eq!(list.as_slice(pool), &[i2, i3]); + + list.grow_at(1, 1, pool); + list.as_mut_slice(pool)[1] = i1; + assert_eq!(list.as_slice(pool), &[i2, i1, i3]); + + // Append nothing at the end. + list.grow_at(3, 0, pool); + assert_eq!(list.as_slice(pool), &[i2, i1, i3]); + + // Append something at the end. + list.grow_at(3, 1, pool); + list.as_mut_slice(pool)[3] = i4; + assert_eq!(list.as_slice(pool), &[i2, i1, i3, i4]); + } } From 477fb4d5da12b5af4d128439cd030696b6489c1a Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Mon, 13 Mar 2017 21:51:16 +0000 Subject: [PATCH 0587/3084] Expanded instruction integrity checking in the verifier, now verifying result types and entity references. --- lib/cretonne/src/entity_list.rs | 6 ++ lib/cretonne/src/ir/dfg.rs | 13 +++ lib/cretonne/src/verifier.rs | 136 +++++++++++++++++++++++++++++++- 3 files changed, 152 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index 423b0ed155..1a05464c21 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -233,6 +233,12 @@ impl EntityList { pool.len_of(self).unwrap_or(0) } + /// Returns `true` if the list is valid + pub fn is_valid(&self, pool: &ListPool) -> bool { + // We consider an empty list to be valid + self.is_empty() || pool.len_of(self) != None + } + /// Get the list as a slice. pub fn as_slice<'a>(&'a self, pool: &'a ListPool) -> &'a [T] { let idx = self.index as usize; diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 07e7dbbdb5..8183a04cc5 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -82,6 +82,11 @@ impl DataFlowGraph { pub fn num_ebbs(&self) -> usize { self.ebbs.len() } + + /// Returns `true` if the given ebb reference is valid. + pub fn ebb_is_valid(&self, ebb: Ebb) -> bool { + self.ebbs.is_valid(ebb) + } } /// Handling values. @@ -95,6 +100,14 @@ impl DataFlowGraph { vref } + /// Check if a value reference is valid. + pub fn value_is_valid(&self, v: Value) -> bool { + match v.expand() { + ExpandedValue::Direct(inst) => self.insts.is_valid(inst), + ExpandedValue::Table(index) => index < self.extended_values.len(), + } + } + /// Get the type of a value. pub fn value_type(&self, v: Value) -> Type { use ir::entities::ExpandedValue::*; diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 2610475759..90bf2dda90 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -12,17 +12,18 @@ //! Instruction integrity //! //! - The instruction format must match the opcode. -//! TODO: //! - All result values must be created for multi-valued instructions. //! - Instructions with no results must have a VOID `first_type()`. //! - All referenced entities must exist. (Values, EBBs, stack slots, ...) //! //! SSA form //! +//! TODO: //! - Values must be defined by an instruction that exists and that is inserted in //! an EBB, or be an argument of an existing EBB. //! - Values used by an instruction must dominate the instruction. -//! Control flow graph and dominator tree integrity: +//! +//! Control flow graph and dominator tree integrity: //! //! - All predecessors in the CFG must be branches to the EBB. //! - All branches to an EBB must be present in the CFG. @@ -52,7 +53,7 @@ //! - Swizzle and shuffle instructions take a variable number of lane arguments. The number //! of arguments must match the destination type, and the lane indexes must be in range. -use ir::{Function, ValueDef, Ebb, Inst}; +use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, Value}; use ir::instructions::InstructionFormat; use ir::entities::AnyEntity; use std::fmt::{self, Display, Formatter}; @@ -147,15 +148,144 @@ impl<'a> Verifier<'a> { fn instruction_integrity(&self, inst: Inst) -> Result<()> { let inst_data = &self.func.dfg[inst]; + let dfg = &self.func.dfg; // The instruction format matches the opcode if inst_data.opcode().format() != InstructionFormat::from(inst_data) { return err!(inst, "instruction opcode doesn't match instruction format"); } + let fixed_results = inst_data.opcode().constraints().fixed_results(); + // var_results is 0 if we aren't a call instruction + let var_results = dfg.call_signature(inst) + .map(|sig| dfg.signatures[sig].return_types.len()) + .unwrap_or(0); + let total_results = fixed_results + var_results; + + if total_results == 0 { + // Instructions with no results have a NULL `first_type()` + let ret_type = inst_data.first_type(); + if ret_type != types::VOID { + return err!(inst, + "instruction expected to have NULL return value, found {}", + ret_type); + } + } else { + // All result values for multi-valued instructions are created + let got_results = dfg.inst_results(inst).count(); + if got_results != total_results { + return err!(inst, + "expected {} result values, found {}", + total_results, + got_results); + } + } + + self.verify_entity_references(inst) + } + + fn verify_entity_references(&self, inst: Inst) -> Result<()> { + use ir::instructions::InstructionData::*; + + for &arg in self.func.dfg[inst].arguments(&self.func.dfg.value_lists) { + self.verify_value(inst, arg)?; + } + + for res in self.func.dfg.inst_results(inst) { + self.verify_value(inst, res)?; + } + + match &self.func.dfg[inst] { + &MultiAry { ref args, .. } => { + self.verify_value_list(inst, args)?; + } + &Jump { destination, ref args, .. } => { + self.verify_ebb(inst, destination)?; + self.verify_value_list(inst, args)?; + } + &Branch { destination, ref args, .. } => { + self.verify_ebb(inst, destination)?; + self.verify_value_list(inst, args)?; + } + &BranchTable { table, .. } => { + self.verify_jump_table(inst, table)?; + } + &Call { func_ref, ref args, .. } => { + self.verify_func_ref(inst, func_ref)?; + self.verify_value_list(inst, args)?; + } + &IndirectCall { sig_ref, ref args, .. } => { + self.verify_sig_ref(inst, sig_ref)?; + self.verify_value_list(inst, args)?; + } + // Exhaustive list so we can't forget to add new formats + &Nullary { .. } | + &Unary { .. } | + &UnaryImm { .. } | + &UnaryIeee32 { .. } | + &UnaryIeee64 { .. } | + &UnarySplit { .. } | + &Binary { .. } | + &BinaryImm { .. } | + &BinaryOverflow { .. } | + &Ternary { .. } | + &InsertLane { .. } | + &ExtractLane { .. } | + &IntCompare { .. } | + &FloatCompare { .. } => {} + } + Ok(()) } + fn verify_ebb(&self, inst: Inst, e: Ebb) -> Result<()> { + if !self.func.dfg.ebb_is_valid(e) { + err!(inst, "invalid ebb reference {}", e) + } else { + Ok(()) + } + } + + fn verify_sig_ref(&self, inst: Inst, s: SigRef) -> Result<()> { + if !self.func.dfg.signatures.is_valid(s) { + err!(inst, "invalid signature reference {}", s) + } else { + Ok(()) + } + } + + fn verify_func_ref(&self, inst: Inst, f: FuncRef) -> Result<()> { + if !self.func.dfg.ext_funcs.is_valid(f) { + err!(inst, "invalid function reference {}", f) + } else { + Ok(()) + } + } + + fn verify_value_list(&self, inst: Inst, l: &ValueList) -> Result<()> { + if !l.is_valid(&self.func.dfg.value_lists) { + err!(inst, "invalid value list reference {:?}", l) + } else { + Ok(()) + } + } + + fn verify_jump_table(&self, inst: Inst, j: JumpTable) -> Result<()> { + if !self.func.jump_tables.is_valid(j) { + err!(inst, "invalid jump table reference {}", j) + } else { + Ok(()) + } + } + + fn verify_value(&self, inst: Inst, v: Value) -> Result<()> { + if !self.func.dfg.value_is_valid(v) { + err!(inst, "invalid value reference {}", v) + } else { + Ok(()) + } + } + pub fn run(&self) -> Result<()> { for ebb in self.func.layout.ebbs() { for inst in self.func.layout.ebb_insts(ebb) { From 010861d58e97b12a87a9ed9a6e25824f6afa7201 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 14 Mar 2017 10:48:05 -0700 Subject: [PATCH 0588/3084] Upgrade to rustfmt 0.8.0. Lots of changes this time. Worked around what looks like a rustfmt bug in parse_inst_operands where a large match was nested inside Ok(). --- cranelift/src/cton-util.rs | 6 +- cranelift/src/filetest/concurrent.rs | 16 +-- cranelift/src/filetest/domtree.rs | 5 +- cranelift/src/filetest/runner.rs | 24 ++-- cranelift/src/filetest/runone.rs | 5 +- cranelift/src/filetest/subtest.rs | 8 +- cranelift/src/rsfilecheck.rs | 3 +- cranelift/test-all.sh | 2 +- cranelift/tests/cfg_traversal.rs | 4 +- lib/cretonne/build.rs | 3 +- lib/cretonne/src/dominator_tree.rs | 10 +- lib/cretonne/src/ir/condcodes.rs | 52 +++---- lib/cretonne/src/ir/dfg.rs | 67 ++++----- lib/cretonne/src/ir/extfunc.rs | 8 +- lib/cretonne/src/ir/immediates.rs | 20 +-- lib/cretonne/src/ir/jumptable.rs | 10 +- lib/cretonne/src/ir/layout.rs | 13 +- lib/cretonne/src/ir/types.rs | 14 +- lib/cretonne/src/isa/arm32/mod.rs | 22 +-- lib/cretonne/src/isa/arm64/mod.rs | 20 +-- lib/cretonne/src/isa/intel/mod.rs | 22 +-- lib/cretonne/src/isa/registers.rs | 13 +- lib/cretonne/src/isa/riscv/mod.rs | 22 +-- lib/cretonne/src/partition_slice.rs | 5 +- lib/cretonne/src/regalloc/allocatable_set.rs | 28 ++-- lib/cretonne/src/regalloc/coloring.rs | 5 +- .../src/regalloc/live_value_tracker.rs | 24 ++-- lib/cretonne/src/regalloc/liveness.rs | 6 +- lib/cretonne/src/regalloc/liverange.rs | 18 ++- lib/cretonne/src/verifier.rs | 21 +-- lib/cretonne/src/write.rs | 6 +- lib/filecheck/src/checker.rs | 19 +-- lib/filecheck/src/explain.rs | 56 ++++---- lib/filecheck/src/pattern.rs | 18 +-- lib/reader/src/lexer.rs | 89 ++++++------ lib/reader/src/parser.rs | 135 +++++++++++------- lib/reader/src/sourcemap.rs | 40 +++--- 37 files changed, 462 insertions(+), 377 deletions(-) diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 35a5c8a564..89fee3c5a5 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -51,11 +51,7 @@ pub type CommandResult = Result<(), String>; fn cton_util() -> CommandResult { // Parse comand line arguments. let args: Args = Docopt::new(USAGE) - .and_then(|d| { - d.help(true) - .version(Some(format!("Cretonne {}", VERSION))) - .decode() - }) + .and_then(|d| d.help(true).version(Some(format!("Cretonne {}", VERSION))).decode()) .unwrap_or_else(|e| e.exit()); // Find the sub-command to execute. diff --git a/cranelift/src/filetest/concurrent.rs b/cranelift/src/filetest/concurrent.rs index e0afc8bead..aa0fba8f1d 100644 --- a/cranelift/src/filetest/concurrent.rs +++ b/cranelift/src/filetest/concurrent.rs @@ -97,8 +97,8 @@ fn heartbeat_thread(replies: Sender) -> thread::JoinHandle<()> { thread::Builder::new() .name("heartbeat".to_string()) .spawn(move || while replies.send(Reply::Tick).is_ok() { - thread::sleep(Duration::from_secs(1)); - }) + thread::sleep(Duration::from_secs(1)); + }) .unwrap() } @@ -120,9 +120,9 @@ fn worker_thread(thread_num: usize, // Tell them we're starting this job. // The receiver should always be present for this as long as we have jobs. replies.send(Reply::Starting { - jobid: jobid, - thread_num: thread_num, - }) + jobid: jobid, + thread_num: thread_num, + }) .unwrap(); let result = catch_unwind(|| runone::run(path.as_path())).unwrap_or_else(|e| { @@ -138,9 +138,9 @@ fn worker_thread(thread_num: usize, }); replies.send(Reply::Done { - jobid: jobid, - result: result, - }) + jobid: jobid, + result: result, + }) .unwrap(); } }) diff --git a/cranelift/src/filetest/domtree.rs b/cranelift/src/filetest/domtree.rs index efa4ee857b..84b63177ec 100644 --- a/cranelift/src/filetest/domtree.rs +++ b/cranelift/src/filetest/domtree.rs @@ -89,7 +89,10 @@ impl SubTest for TestDomtree { // Now we know that everything in `expected` is consistent with `domtree`. // All other EBB's should be either unreachable or the entry block. - for ebb in func.layout.ebbs().skip(1).filter(|ebb| !expected.contains_key(&ebb)) { + for ebb in func.layout + .ebbs() + .skip(1) + .filter(|ebb| !expected.contains_key(&ebb)) { if let Some(got_inst) = domtree.idom(ebb) { return Err(format!("mismatching idoms for renumbered {}:\n\ want: unrechable, got: {}", diff --git a/cranelift/src/filetest/runner.rs b/cranelift/src/filetest/runner.rs index 8794de0004..91331a0299 100644 --- a/cranelift/src/filetest/runner.rs +++ b/cranelift/src/filetest/runner.rs @@ -105,9 +105,9 @@ impl TestRunner { /// Any problems reading `file` as a test case file will be reported as a test failure. pub fn push_test>(&mut self, file: P) { self.tests.push(QueueEntry { - path: file.into(), - state: State::New, - }); + path: file.into(), + state: State::New, + }); } /// Begin running tests concurrently. @@ -277,9 +277,9 @@ impl TestRunner { let mut times = self.tests .iter() .filter_map(|entry| match *entry { - QueueEntry { state: State::Done(Ok(dur)), .. } => Some(dur), - _ => None, - }) + QueueEntry { state: State::Done(Ok(dur)), .. } => Some(dur), + _ => None, + }) .collect::>(); // Get me some real data, kid. @@ -303,12 +303,12 @@ impl TestRunner { return; } - for t in self.tests - .iter() - .filter(|entry| match **entry { - QueueEntry { state: State::Done(Ok(dur)), .. } => dur > cut, - _ => false, - }) { + for t in self.tests.iter().filter(|entry| match **entry { + QueueEntry { state: State::Done(Ok(dur)), .. } => { + dur > cut + } + _ => false, + }) { println!("slow: {}", t) } diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index 2067a245b9..8e707d4ff5 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -25,7 +25,10 @@ pub fn run(path: &Path) -> TestResult { } // Parse the test commands. - let mut tests = testfile.commands.iter().map(new_subtest).collect::>>()?; + let mut tests = testfile.commands + .iter() + .map(new_subtest) + .collect::>>()?; // Flags to use for those tests that don't need an ISA. // This is the cumulative effect of all the `set` commands in the file. diff --git a/cranelift/src/filetest/subtest.rs b/cranelift/src/filetest/subtest.rs index 170490fb3a..ede013ef9f 100644 --- a/cranelift/src/filetest/subtest.rs +++ b/cranelift/src/filetest/subtest.rs @@ -66,7 +66,10 @@ pub trait SubTest { /// match 'inst10'. impl<'a> filecheck::VariableMap for Context<'a> { fn lookup(&self, varname: &str) -> Option { - self.details.map.lookup_str(varname).map(|e| FCValue::Regex(format!(r"\b{}\b", e).into())) + self.details + .map + .lookup_str(varname) + .map(|e| FCValue::Regex(format!(r"\b{}\b", e).into())) } } @@ -77,8 +80,7 @@ pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { Ok(()) } else { // Filecheck mismatch. Emit an explanation as output. - let (_, explain) = checker.explain(&text, context) - .map_err(|e| format!("explain: {}", e))?; + let (_, explain) = checker.explain(&text, context).map_err(|e| format!("explain: {}", e))?; Err(format!("filecheck failed:\n{}{}", checker, explain)) } } diff --git a/cranelift/src/rsfilecheck.rs b/cranelift/src/rsfilecheck.rs index 9794e10813..734c067a36 100644 --- a/cranelift/src/rsfilecheck.rs +++ b/cranelift/src/rsfilecheck.rs @@ -21,8 +21,7 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { io::stdin().read_to_string(&mut buffer).map_err(|e| format!("stdin: {}", e))?; if verbose { - let (success, explain) = checker.explain(&buffer, NO_VARIABLES) - .map_err(|e| e.to_string())?; + let (success, explain) = checker.explain(&buffer, NO_VARIABLES).map_err(|e| e.to_string())?; print!("{}", explain); if success { println!("OK"); diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index aebec1a872..6e2ceb2de1 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -30,7 +30,7 @@ function banner() { # rustfmt is installed. # # This version should always be bumped to the newest version available. -RUSTFMT_VERSION="0.7.1" +RUSTFMT_VERSION="0.8.0" if cargo install --list | grep -q "^rustfmt v$RUSTFMT_VERSION"; then banner "Rust formatting" diff --git a/cranelift/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs index 453e0f83b7..ff5cf6020c 100644 --- a/cranelift/tests/cfg_traversal.rs +++ b/cranelift/tests/cfg_traversal.rs @@ -9,9 +9,7 @@ use self::cretonne::entity_map::EntityMap; fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) { let func = &parse_functions(function_source).unwrap()[0]; let cfg = ControlFlowGraph::with_function(&func); - let ebbs = ebb_order.iter() - .map(|n| Ebb::with_number(*n).unwrap()) - .collect::>(); + let ebbs = ebb_order.iter().map(|n| Ebb::with_number(*n).unwrap()).collect::>(); let mut postorder_ebbs = cfg.postorder_ebbs(); let mut postorder_map = EntityMap::with_capacity(postorder_ebbs.len()); diff --git a/lib/cretonne/build.rs b/lib/cretonne/build.rs index 2067a37afd..4766a032e0 100644 --- a/lib/cretonne/build.rs +++ b/lib/cretonne/build.rs @@ -26,7 +26,8 @@ fn main() { // Make sure we rebuild is this build script changes. // I guess that won't happen if you have non-UTF8 bytes in your path names. // The `build.py` script prints out its own dependencies. - println!("cargo:rerun-if-changed={}", crate_dir.join("build.rs").to_string_lossy()); + println!("cargo:rerun-if-changed={}", + crate_dir.join("build.rs").to_string_lossy()); // Scripts are in `$crate_dir/meta`. let meta_dir = crate_dir.join("meta"); diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 8c90b46160..17bd3445ab 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -191,12 +191,14 @@ impl DominatorTree { // Get an iterator with just the reachable predecessors to `ebb`. // Note that during the first pass, `is_reachable` returns false for blocks that haven't // been visited yet. - let mut reachable_preds = - cfg.get_predecessors(ebb).iter().cloned().filter(|&(ebb, _)| self.is_reachable(ebb)); + let mut reachable_preds = cfg.get_predecessors(ebb) + .iter() + .cloned() + .filter(|&(ebb, _)| self.is_reachable(ebb)); // The RPO must visit at least one predecessor before this node. - let mut idom = reachable_preds.next() - .expect("EBB node must have one reachable predecessor"); + let mut idom = + reachable_preds.next().expect("EBB node must have one reachable predecessor"); for pred in reachable_preds { idom = self.common_dominator(idom, pred, layout); diff --git a/lib/cretonne/src/ir/condcodes.rs b/lib/cretonne/src/ir/condcodes.rs index dfa6b054f5..5117e39d9d 100644 --- a/lib/cretonne/src/ir/condcodes.rs +++ b/lib/cretonne/src/ir/condcodes.rs @@ -89,17 +89,17 @@ impl Display for IntCC { fn fmt(&self, f: &mut Formatter) -> fmt::Result { use self::IntCC::*; f.write_str(match self { - &Equal => "eq", - &NotEqual => "ne", - &SignedGreaterThan => "sgt", - &SignedGreaterThanOrEqual => "sge", - &SignedLessThan => "slt", - &SignedLessThanOrEqual => "sle", - &UnsignedGreaterThan => "ugt", - &UnsignedGreaterThanOrEqual => "uge", - &UnsignedLessThan => "ult", - &UnsignedLessThanOrEqual => "ule", - }) + &Equal => "eq", + &NotEqual => "ne", + &SignedGreaterThan => "sgt", + &SignedGreaterThanOrEqual => "sge", + &SignedLessThan => "slt", + &SignedLessThanOrEqual => "sle", + &UnsignedGreaterThan => "ugt", + &UnsignedGreaterThanOrEqual => "uge", + &UnsignedLessThan => "ult", + &UnsignedLessThanOrEqual => "ule", + }) } } @@ -220,21 +220,21 @@ impl Display for FloatCC { fn fmt(&self, f: &mut Formatter) -> fmt::Result { use self::FloatCC::*; f.write_str(match self { - &Ordered => "ord", - &Unordered => "uno", - &Equal => "eq", - &NotEqual => "ne", - &OrderedNotEqual => "one", - &UnorderedOrEqual => "ueq", - &LessThan => "lt", - &LessThanOrEqual => "le", - &GreaterThan => "gt", - &GreaterThanOrEqual => "ge", - &UnorderedOrLessThan => "ult", - &UnorderedOrLessThanOrEqual => "ule", - &UnorderedOrGreaterThan => "ugt", - &UnorderedOrGreaterThanOrEqual => "uge", - }) + &Ordered => "ord", + &Unordered => "uno", + &Equal => "eq", + &NotEqual => "ne", + &OrderedNotEqual => "one", + &UnorderedOrEqual => "ueq", + &LessThan => "lt", + &LessThanOrEqual => "le", + &GreaterThan => "gt", + &GreaterThanOrEqual => "ge", + &UnorderedOrLessThan => "ult", + &UnorderedOrLessThanOrEqual => "ule", + &UnorderedOrGreaterThan => "ugt", + &UnorderedOrGreaterThanOrEqual => "uge", + }) } } diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 8183a04cc5..f5d48ad81f 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -180,19 +180,20 @@ impl DataFlowGraph { for _ in 0..self.insts.len() { v = self.resolve_aliases(match v.expand() { - Direct(inst) => { - match self[inst] { - InstructionData::Unary { opcode, arg, .. } => { - match opcode { - Opcode::Copy | Opcode::Spill | Opcode::Fill => arg, - _ => return v, - } - } - _ => return v, - } - } - _ => return v, - }); + Direct(inst) => { + match self[inst] { + InstructionData::Unary { opcode, arg, .. } => { + match opcode { + Opcode::Copy | Opcode::Spill | + Opcode::Fill => arg, + _ => return v, + } + } + _ => return v, + } + } + _ => return v, + }); } panic!("Copy loop detected for {}", value); } @@ -361,11 +362,11 @@ impl DataFlowGraph { for res_idx in (0..var_results).rev() { if let Some(ty) = first_type { head = Some(self.make_value(ValueData::Inst { - ty: ty, - num: (total_results - rev_num) as u16, - inst: inst, - next: head.into(), - })); + ty: ty, + num: (total_results - rev_num) as u16, + inst: inst, + next: head.into(), + })); rev_num += 1; } first_type = Some(self.signatures[sig].return_types[res_idx].value_type); @@ -376,11 +377,11 @@ impl DataFlowGraph { for res_idx in (0..fixed_results).rev() { if let Some(ty) = first_type { head = Some(self.make_value(ValueData::Inst { - ty: ty, - num: (total_results - rev_num) as u16, - inst: inst, - next: head.into(), - })); + ty: ty, + num: (total_results - rev_num) as u16, + inst: inst, + next: head.into(), + })); rev_num += 1; } first_type = Some(constraints.result_type(res_idx, ctrl_typevar)); @@ -474,11 +475,11 @@ impl DataFlowGraph { // Not a fixed result, try to extract a return type from the call signature. self.call_signature(inst).and_then(|sigref| { - self.signatures[sigref] - .return_types - .get(result_idx - fixed_results) - .map(|&arg| arg.value_type) - }) + self.signatures[sigref] + .return_types + .get(result_idx - fixed_results) + .map(|&arg| arg.value_type) + }) } } @@ -523,11 +524,11 @@ impl DataFlowGraph { /// Append an argument with type `ty` to `ebb`. pub fn append_ebb_arg(&mut self, ebb: Ebb, ty: Type) -> Value { let val = self.make_value(ValueData::Arg { - ty: ty, - ebb: ebb, - num: 0, - next: None.into(), - }); + ty: ty, + ebb: ebb, + num: 0, + next: None.into(), + }); self.put_ebb_arg(ebb, val); val } diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 9f81d11442..bc293d1169 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -50,9 +50,11 @@ impl Signature { let bytes = self.argument_types .iter() .filter_map(|arg| match arg.location { - ArgumentLoc::Stack(offset) => Some(offset + arg.value_type.bits() as u32 / 8), - _ => None, - }) + ArgumentLoc::Stack(offset) => { + Some(offset + arg.value_type.bits() as u32 / 8) + } + _ => None, + }) .fold(0, cmp::max); self.argument_bytes = Some(bytes); } diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 79229fc3a0..84f5978429 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -252,20 +252,20 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { if s2.starts_with("NaN:0x") { // Quiet NaN with payload. return match u64::from_str_radix(&s2[6..], 16) { - Ok(payload) if payload < quiet_bit => { - Ok(sign_bit | max_e_bits | quiet_bit | payload) - } - _ => Err("Invalid NaN payload"), - }; + Ok(payload) if payload < quiet_bit => { + Ok(sign_bit | max_e_bits | quiet_bit | payload) + } + _ => Err("Invalid NaN payload"), + }; } if s2.starts_with("sNaN:0x") { // Signaling NaN with payload. return match u64::from_str_radix(&s2[7..], 16) { - Ok(payload) if 0 < payload && payload < quiet_bit => { - Ok(sign_bit | max_e_bits | payload) - } - _ => Err("Invalid sNaN payload"), - }; + Ok(payload) if 0 < payload && payload < quiet_bit => { + Ok(sign_bit | max_e_bits | payload) + } + _ => Err("Invalid sNaN payload"), + }; } return Err("Float must be hexadecimal"); diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index 29da4456bc..69a44ef8fb 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -66,7 +66,10 @@ impl JumpTableData { /// /// This returns an iterator that skips any empty slots in the table. pub fn entries<'a>(&'a self) -> Entries { - Entries(self.table.iter().cloned().enumerate()) + Entries(self.table + .iter() + .cloned() + .enumerate()) } /// Access the whole table as a mutable slice. @@ -101,7 +104,10 @@ impl Display for JumpTableData { Some(first) => write!(fmt, "jump_table {}", first)?, } - for dest in self.table.iter().skip(1).map(|e| e.expand()) { + for dest in self.table + .iter() + .skip(1) + .map(|e| e.expand()) { match dest { None => write!(fmt, ", 0")?, Some(ebb) => write!(fmt, ", {}", ebb)?, diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index d30f336780..83edbea76f 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -125,10 +125,7 @@ impl Layout { /// Get the last sequence number in `ebb`. fn last_ebb_seq(&self, ebb: Ebb) -> SequenceNumber { // Get the seq of the last instruction if it exists, otherwise use the EBB header seq. - self.ebbs[ebb] - .last_inst - .map(|inst| self.insts[inst].seq) - .unwrap_or(self.ebbs[ebb].seq) + self.ebbs[ebb].last_inst.map(|inst| self.insts[inst].seq).unwrap_or(self.ebbs[ebb].seq) } /// Assign a valid sequence number to `ebb` such that the numbers are still monotonic. This may @@ -439,8 +436,8 @@ impl Layout { /// Insert `inst` before the instruction `before` in the same EBB. pub fn insert_inst(&mut self, inst: Inst, before: Inst) { assert_eq!(self.inst_ebb(inst), None); - let ebb = self.inst_ebb(before) - .expect("Instruction before insertion point not in the layout"); + let ebb = + self.inst_ebb(before).expect("Instruction before insertion point not in the layout"); let after = self.insts[before].prev; { let inst_node = self.insts.ensure(inst); @@ -488,8 +485,8 @@ impl Layout { /// i4 /// ``` pub fn split_ebb(&mut self, new_ebb: Ebb, before: Inst) { - let old_ebb = self.inst_ebb(before) - .expect("The `before` instruction must be in the layout"); + let old_ebb = + self.inst_ebb(before).expect("The `before` instruction must be in the layout"); assert!(!self.is_ebb_inserted(new_ebb)); // Insert new_ebb after old_ebb. diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 550f25804b..1321636add 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -348,7 +348,12 @@ mod tests { assert_eq!(big.bits(), 64 * 256); assert_eq!(big.half_vector().unwrap().to_string(), "f64x128"); - assert_eq!(B1.by(2).unwrap().half_vector().unwrap().to_string(), "b1"); + assert_eq!(B1.by(2) + .unwrap() + .half_vector() + .unwrap() + .to_string(), + "b1"); assert_eq!(I32.half_vector(), None); assert_eq!(VOID.half_vector(), None); @@ -378,7 +383,12 @@ mod tests { assert_eq!(B1.by(8).unwrap().to_string(), "b1x8"); assert_eq!(B8.by(1).unwrap().to_string(), "b8"); assert_eq!(B16.by(256).unwrap().to_string(), "b16x256"); - assert_eq!(B32.by(4).unwrap().by(2).unwrap().to_string(), "b32x8"); + assert_eq!(B32.by(4) + .unwrap() + .by(2) + .unwrap() + .to_string(), + "b32x8"); assert_eq!(B64.by(8).unwrap().to_string(), "b64x8"); assert_eq!(I8.by(64).unwrap().to_string(), "i8x64"); assert_eq!(F64.by(2).unwrap().to_string(), "f64x2"); diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 9af02883e3..7cc6a93358 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -34,10 +34,10 @@ fn isa_constructor(shared_flags: shared_settings::Flags, &enc_tables::LEVEL1_A32[..] }; Box::new(Isa { - isa_flags: settings::Flags::new(&shared_flags, builder), - shared_flags: shared_flags, - cpumode: level1, - }) + isa_flags: settings::Flags::new(&shared_flags, builder), + shared_flags: shared_flags, + cpumode: level1, + }) } impl TargetIsa for Isa { @@ -58,13 +58,13 @@ impl TargetIsa for Isa { inst.opcode(), self.cpumode, &enc_tables::LEVEL2[..]) - .and_then(|enclist_offset| { - general_encoding(enclist_offset, - &enc_tables::ENCLISTS[..], - |instp| enc_tables::check_instp(inst, instp), - |isap| self.isa_flags.numbered_predicate(isap as usize)) - .ok_or(Legalize::Expand) - }) + .and_then(|enclist_offset| { + general_encoding(enclist_offset, + &enc_tables::ENCLISTS[..], + |instp| enc_tables::check_instp(inst, instp), + |isap| self.isa_flags.numbered_predicate(isap as usize)) + .ok_or(Legalize::Expand) + }) } fn recipe_names(&self) -> &'static [&'static str] { diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 50f54524a9..5f253c39c2 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -28,9 +28,9 @@ fn isa_constructor(shared_flags: shared_settings::Flags, builder: &shared_settings::Builder) -> Box { Box::new(Isa { - isa_flags: settings::Flags::new(&shared_flags, builder), - shared_flags: shared_flags, - }) + isa_flags: settings::Flags::new(&shared_flags, builder), + shared_flags: shared_flags, + }) } impl TargetIsa for Isa { @@ -51,13 +51,13 @@ impl TargetIsa for Isa { inst.opcode(), &enc_tables::LEVEL1_A64[..], &enc_tables::LEVEL2[..]) - .and_then(|enclist_offset| { - general_encoding(enclist_offset, - &enc_tables::ENCLISTS[..], - |instp| enc_tables::check_instp(inst, instp), - |isap| self.isa_flags.numbered_predicate(isap as usize)) - .ok_or(Legalize::Expand) - }) + .and_then(|enclist_offset| { + general_encoding(enclist_offset, + &enc_tables::ENCLISTS[..], + |instp| enc_tables::check_instp(inst, instp), + |isap| self.isa_flags.numbered_predicate(isap as usize)) + .ok_or(Legalize::Expand) + }) } fn recipe_names(&self) -> &'static [&'static str] { diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index e5c99521e3..4a5ea8201a 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -34,10 +34,10 @@ fn isa_constructor(shared_flags: shared_settings::Flags, &enc_tables::LEVEL1_I32[..] }; Box::new(Isa { - isa_flags: settings::Flags::new(&shared_flags, builder), - shared_flags: shared_flags, - cpumode: level1, - }) + isa_flags: settings::Flags::new(&shared_flags, builder), + shared_flags: shared_flags, + cpumode: level1, + }) } impl TargetIsa for Isa { @@ -58,13 +58,13 @@ impl TargetIsa for Isa { inst.opcode(), self.cpumode, &enc_tables::LEVEL2[..]) - .and_then(|enclist_offset| { - general_encoding(enclist_offset, - &enc_tables::ENCLISTS[..], - |instp| enc_tables::check_instp(inst, instp), - |isap| self.isa_flags.numbered_predicate(isap as usize)) - .ok_or(Legalize::Expand) - }) + .and_then(|enclist_offset| { + general_encoding(enclist_offset, + &enc_tables::ENCLISTS[..], + |instp| enc_tables::check_instp(inst, instp), + |isap| self.isa_flags.numbered_predicate(isap as usize)) + .ok_or(Legalize::Expand) + }) } fn recipe_names(&self) -> &'static [&'static str] { diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index c51413eb20..864b950f63 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -67,10 +67,10 @@ impl RegBank { } } .and_then(|offset| if offset < self.units { - Some(offset + self.first_unit) - } else { - None - }) + Some(offset + self.first_unit) + } else { + None + }) } /// Write `regunit` to `w`, assuming that it belongs to this bank. @@ -199,7 +199,10 @@ impl RegInfo { /// Try to parse a regunit name. The name is not expected to begin with `%`. pub fn parse_regunit(&self, name: &str) -> Option { - self.banks.iter().filter_map(|b| b.parse_regunit(name)).next() + self.banks + .iter() + .filter_map(|b| b.parse_regunit(name)) + .next() } /// Make a temporary object that can display a register unit. diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 220f062278..ce04fe70b6 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -35,10 +35,10 @@ fn isa_constructor(shared_flags: shared_settings::Flags, &enc_tables::LEVEL1_RV32[..] }; Box::new(Isa { - isa_flags: settings::Flags::new(&shared_flags, builder), - shared_flags: shared_flags, - cpumode: level1, - }) + isa_flags: settings::Flags::new(&shared_flags, builder), + shared_flags: shared_flags, + cpumode: level1, + }) } impl TargetIsa for Isa { @@ -59,13 +59,13 @@ impl TargetIsa for Isa { inst.opcode(), self.cpumode, &enc_tables::LEVEL2[..]) - .and_then(|enclist_offset| { - general_encoding(enclist_offset, - &enc_tables::ENCLISTS[..], - |instp| enc_tables::check_instp(inst, instp), - |isap| self.isa_flags.numbered_predicate(isap as usize)) - .ok_or(Legalize::Expand) - }) + .and_then(|enclist_offset| { + general_encoding(enclist_offset, + &enc_tables::ENCLISTS[..], + |instp| enc_tables::check_instp(inst, instp), + |isap| self.isa_flags.numbered_predicate(isap as usize)) + .ok_or(Legalize::Expand) + }) } fn recipe_names(&self) -> &'static [&'static str] { diff --git a/lib/cretonne/src/partition_slice.rs b/lib/cretonne/src/partition_slice.rs index 9626b5fd37..0986613040 100644 --- a/lib/cretonne/src/partition_slice.rs +++ b/lib/cretonne/src/partition_slice.rs @@ -35,7 +35,10 @@ mod tests { fn check(x: &[u32], want: &[u32]) { assert_eq!(x.len(), want.len()); - let want_count = want.iter().cloned().filter(|&x| x % 10 == 0).count(); + let want_count = want.iter() + .cloned() + .filter(|&x| x % 10 == 0) + .count(); let mut v = Vec::new(); v.extend(x.iter().cloned()); let count = partition_slice(&mut v[..], |&x| x % 10 == 0); diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index 40ef3c3636..29a7e26964 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -118,21 +118,21 @@ mod tests { // Register classes for testing. const GPR: RegClass = &RegClassData { - name: "GPR", - index: 0, - width: 1, - first: 28, - subclasses: 0, - mask: [0xf0000000, 0x0000000f, 0], - }; + name: "GPR", + index: 0, + width: 1, + first: 28, + subclasses: 0, + mask: [0xf0000000, 0x0000000f, 0], + }; const DPR: RegClass = &RegClassData { - name: "DPR", - index: 0, - width: 2, - first: 28, - subclasses: 0, - mask: [0x50000000, 0x0000000a, 0], - }; + name: "DPR", + index: 0, + width: 2, + first: 28, + subclasses: 0, + mask: [0x50000000, 0x0000000a, 0], + }; #[test] fn put_and_take() { diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 24f8c0c7e1..19800600b4 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -200,7 +200,10 @@ impl<'a> Context<'a> { for lv in liveins { let value = lv.value; - let affinity = self.liveness.get(value).expect("No live range for live-in").affinity; + let affinity = self.liveness + .get(value) + .expect("No live range for live-in") + .affinity; if let Affinity::Reg(rc_index) = affinity { let regclass = self.reginfo.rc(rc_index); match func.locations[value] { diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index c45da220e4..19ed62e8fb 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -69,10 +69,10 @@ impl LiveValueVec { /// Add a new live value to `values`. fn push(&mut self, value: Value, endpoint: Inst, affinity: Affinity) { self.values.push(LiveValue { - value: value, - endpoint: endpoint, - affinity: affinity, - }); + value: value, + endpoint: endpoint, + affinity: affinity, + }); } /// Remove all elements. @@ -167,8 +167,7 @@ impl LiveValueTracker { self.idom_sets.get(&idom).expect("No stored live set for dominator"); // Get just the values that are live-in to `ebb`. for &value in idom_live_list.as_slice(&self.idom_pool) { - let lr = liveness.get(value) - .expect("Immediate dominator value has no live range"); + let lr = liveness.get(value).expect("Immediate dominator value has no live range"); // Check if this value is live-in here. if let Some(endpoint) = lr.livein_local_end(ebb, program_order) { @@ -260,13 +259,16 @@ impl LiveValueTracker { /// Save the current set of live values so it is associated with `idom`. fn save_idom_live_set(&mut self, idom: Inst) { - let values = self.live.values.iter().map(|lv| lv.value); + let values = self.live + .values + .iter() + .map(|lv| lv.value); let pool = &mut self.idom_pool; // If there already is a set saved for `idom`, just keep it. self.idom_sets.entry(idom).or_insert_with(|| { - let mut list = ValueList::default(); - list.extend(values, pool); - list - }); + let mut list = ValueList::default(); + list.extend(values, pool); + list + }); } } diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 582cbae48a..989d3bb8b7 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -315,8 +315,10 @@ impl Liveness { let recipe = func.encodings[inst].recipe(); // Iterator of constraints, one per value operand. // TODO: Should we fail here if the instruction doesn't have a valid encoding? - let mut operand_constraints = - recipe_constraints.get(recipe).map(|c| c.ins).unwrap_or(&[]).iter(); + let mut operand_constraints = recipe_constraints.get(recipe) + .map(|c| c.ins) + .unwrap_or(&[]) + .iter(); for &arg in func.dfg[inst].arguments(&func.dfg.value_lists) { // Get the live range, create it as a dead range if necessary. diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index a2d7a8f32e..116ccce4aa 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -221,16 +221,14 @@ impl LiveRange { /// Return `Ok(n)` if `liveins[n]` already contains `ebb`. /// Otherwise, return `Err(n)` with the index where such an interval should be inserted. fn find_ebb_interval(&self, ebb: Ebb, order: &PO) -> Result { - self.liveins - .binary_search_by(|intv| order.cmp(intv.begin, ebb)) - .or_else(|n| { - // The interval at `n-1` may cover `ebb`. - if n > 0 && order.cmp(self.liveins[n - 1].end, ebb) == Ordering::Greater { - Ok(n - 1) - } else { - Err(n) - } - }) + self.liveins.binary_search_by(|intv| order.cmp(intv.begin, ebb)).or_else(|n| { + // The interval at `n-1` may cover `ebb`. + if n > 0 && order.cmp(self.liveins[n - 1].end, ebb) == Ordering::Greater { + Ok(n - 1) + } else { + Err(n) + } + }) } /// Extend the local interval for `ebb` so it reaches `to` which must belong to `ebb`. diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 90bf2dda90..f63db1919f 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -157,9 +157,8 @@ impl<'a> Verifier<'a> { let fixed_results = inst_data.opcode().constraints().fixed_results(); // var_results is 0 if we aren't a call instruction - let var_results = dfg.call_signature(inst) - .map(|sig| dfg.signatures[sig].return_types.len()) - .unwrap_or(0); + let var_results = + dfg.call_signature(inst).map(|sig| dfg.signatures[sig].return_types.len()).unwrap_or(0); let total_results = fixed_results + var_results; if total_results == 0 { @@ -247,7 +246,10 @@ impl<'a> Verifier<'a> { } fn verify_sig_ref(&self, inst: Inst, s: SigRef) -> Result<()> { - if !self.func.dfg.signatures.is_valid(s) { + if !self.func + .dfg + .signatures + .is_valid(s) { err!(inst, "invalid signature reference {}", s) } else { Ok(()) @@ -255,7 +257,10 @@ impl<'a> Verifier<'a> { } fn verify_func_ref(&self, inst: Inst, f: FuncRef) -> Result<()> { - if !self.func.dfg.ext_funcs.is_valid(f) { + if !self.func + .dfg + .ext_funcs + .is_valid(f) { err!(inst, "invalid function reference {}", f) } else { Ok(()) @@ -330,9 +335,9 @@ mod tests { let ebb0 = func.dfg.make_ebb(); func.layout.append_ebb(ebb0); let nullary_with_bad_opcode = func.dfg.make_inst(InstructionData::Nullary { - opcode: Opcode::Jump, - ty: types::VOID, - }); + opcode: Opcode::Jump, + ty: types::VOID, + }); func.layout.append_inst(nullary_with_bad_opcode, ebb0); let verifier = Verifier::new(&func); assert_err_with_msg!(verifier.run(), "instruction format"); diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 1f9bfd4524..081da35617 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -188,7 +188,11 @@ fn write_instruction(w: &mut Write, for r in func.dfg.inst_results(inst) { write!(s, ",{}", - func.locations.get(r).cloned().unwrap_or_default().display(®s))? + func.locations + .get(r) + .cloned() + .unwrap_or_default() + .display(®s))? } } write!(s, "]")?; diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs index 3101865e71..c854f3b6b8 100644 --- a/lib/filecheck/src/checker.rs +++ b/lib/filecheck/src/checker.rs @@ -270,7 +270,10 @@ impl<'a> State<'a> { // Get the offset following the match that defined `var`, or 0 if var is an environment // variable or unknown. fn def_offset(&self, var: &str) -> usize { - self.vars.get(var).map(|&VarDef { offset, .. }| offset).unwrap_or(0) + self.vars + .get(var) + .map(|&VarDef { offset, .. }| offset) + .unwrap_or(0) } // Get the offset of the beginning of the next line after `pos`. @@ -344,13 +347,13 @@ impl<'a> State<'a> { }) }; Ok(if let Some((b, e)) = matched_range { - let r = (range.0 + b, range.0 + e); - self.recorder.matched_check(rx.as_str(), r); - Some(r) - } else { - self.recorder.missed_check(rx.as_str(), range); - None - }) + let r = (range.0 + b, range.0 + e); + self.recorder.matched_check(rx.as_str(), r); + Some(r) + } else { + self.recorder.missed_check(rx.as_str(), range); + None + }) } } diff --git a/lib/filecheck/src/explain.rs b/lib/filecheck/src/explain.rs index 75fe6f9286..04a6cd233e 100644 --- a/lib/filecheck/src/explain.rs +++ b/lib/filecheck/src/explain.rs @@ -148,49 +148,49 @@ impl<'a> Recorder for Explainer<'a> { fn matched_check(&mut self, regex: &str, matched: MatchRange) { self.matches.push(Match { - directive: self.directive, - is_match: true, - is_not: false, - regex: regex.to_owned(), - range: matched, - }); + directive: self.directive, + is_match: true, + is_not: false, + regex: regex.to_owned(), + range: matched, + }); } fn matched_not(&mut self, regex: &str, matched: MatchRange) { self.matches.push(Match { - directive: self.directive, - is_match: true, - is_not: true, - regex: regex.to_owned(), - range: matched, - }); + directive: self.directive, + is_match: true, + is_not: true, + regex: regex.to_owned(), + range: matched, + }); } fn missed_check(&mut self, regex: &str, searched: MatchRange) { self.matches.push(Match { - directive: self.directive, - is_match: false, - is_not: false, - regex: regex.to_owned(), - range: searched, - }); + directive: self.directive, + is_match: false, + is_not: false, + regex: regex.to_owned(), + range: searched, + }); } fn missed_not(&mut self, regex: &str, searched: MatchRange) { self.matches.push(Match { - directive: self.directive, - is_match: false, - is_not: true, - regex: regex.to_owned(), - range: searched, - }); + directive: self.directive, + is_match: false, + is_not: true, + regex: regex.to_owned(), + range: searched, + }); } fn defined_var(&mut self, varname: &str, value: &str) { self.vardefs.push(VarDef { - directive: self.directive, - varname: varname.to_owned(), - value: value.to_owned(), - }); + directive: self.directive, + varname: varname.to_owned(), + value: value.to_owned(), + }); } } diff --git a/lib/filecheck/src/pattern.rs b/lib/filecheck/src/pattern.rs index 9f54a80cf6..67ec359c4a 100644 --- a/lib/filecheck/src/pattern.rs +++ b/lib/filecheck/src/pattern.rs @@ -112,7 +112,7 @@ impl Pattern { // All remaining possibilities start with `$(`. if s.len() < 2 || !s.starts_with("$(") { return Err(Error::Syntax("pattern syntax error, use $$ to match a single $" - .to_string())); + .to_string())); } // Match the variable name, allowing for an empty varname in `$()`, or `$(=...)`. @@ -164,14 +164,14 @@ impl Pattern { } let refname = s[refname_begin..refname_end].to_string(); return if let Some(defidx) = def { - Ok((Part::DefVar { - def: defidx, - var: refname, - }, - refname_end + 1)) - } else { - Err(Error::Syntax(format!("expected variable name in $(=${})", refname))) - }; + Ok((Part::DefVar { + def: defidx, + var: refname, + }, + refname_end + 1)) + } else { + Err(Error::Syntax(format!("expected variable name in $(=${})", refname))) + }; } // Last case: `$(var=...)` where `...` is a regular expression, possibly containing matched diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index caf32ed252..e1e4cc31c4 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -55,9 +55,9 @@ pub struct LocatedToken<'a> { /// Wrap up a `Token` with the given location. fn token<'a>(token: Token<'a>, loc: Location) -> Result, LocatedError> { Ok(LocatedToken { - token: token, - location: loc, - }) + token: token, + location: loc, + }) } /// An error from the lexical analysis. @@ -76,15 +76,20 @@ pub struct LocatedError { /// Wrap up an `Error` with the given location. fn error<'a>(error: Error, loc: Location) -> Result, LocatedError> { Err(LocatedError { - error: error, - location: loc, - }) + error: error, + location: loc, + }) } /// Get the number of decimal digits at the end of `s`. fn trailing_digits(s: &str) -> usize { // It's faster to iterate backwards over bytes, and we're only counting ASCII digits. - s.as_bytes().iter().rev().cloned().take_while(|&b| b'0' <= b && b <= b'9').count() + s.as_bytes() + .iter() + .rev() + .cloned() + .take_while(|&b| b'0' <= b && b <= b'9') + .count() } /// Pre-parse a supposed entity name by splitting it into two parts: A head of lowercase ASCII @@ -284,9 +289,9 @@ impl<'a> Lexer<'a> { // Look for numbered well-known entities like ebb15, v45, ... token(split_entity_name(text) .and_then(|(prefix, number)| { - Self::numbered_entity(prefix, number) + Self::numbered_entity(prefix, number) .or_else(|| Self::value_type(text, prefix, number)) - }) + }) .unwrap_or(Token::Identifier(text)), loc) } @@ -378,39 +383,39 @@ impl<'a> Lexer<'a> { loop { let loc = self.loc(); return match self.lookahead { - None => None, - Some(';') => Some(self.scan_comment()), - Some('(') => Some(self.scan_char(Token::LPar)), - Some(')') => Some(self.scan_char(Token::RPar)), - Some('{') => Some(self.scan_char(Token::LBrace)), - Some('}') => Some(self.scan_char(Token::RBrace)), - Some('[') => Some(self.scan_char(Token::LBracket)), - Some(']') => Some(self.scan_char(Token::RBracket)), - Some(',') => Some(self.scan_char(Token::Comma)), - Some('.') => Some(self.scan_char(Token::Dot)), - Some(':') => Some(self.scan_char(Token::Colon)), - Some('=') => Some(self.scan_char(Token::Equal)), - Some('-') => { - if self.looking_at("->") { - Some(self.scan_chars(2, Token::Arrow)) - } else { - Some(self.scan_number()) - } - } - Some(ch) if ch.is_digit(10) => Some(self.scan_number()), - Some(ch) if ch.is_alphabetic() => Some(self.scan_word()), - Some('%') => Some(self.scan_name()), - Some('#') => Some(self.scan_hex_sequence()), - Some(ch) if ch.is_whitespace() => { - self.next_ch(); - continue; - } - _ => { - // Skip invalid char, return error. - self.next_ch(); - Some(error(Error::InvalidChar, loc)) - } - }; + None => None, + Some(';') => Some(self.scan_comment()), + Some('(') => Some(self.scan_char(Token::LPar)), + Some(')') => Some(self.scan_char(Token::RPar)), + Some('{') => Some(self.scan_char(Token::LBrace)), + Some('}') => Some(self.scan_char(Token::RBrace)), + Some('[') => Some(self.scan_char(Token::LBracket)), + Some(']') => Some(self.scan_char(Token::RBracket)), + Some(',') => Some(self.scan_char(Token::Comma)), + Some('.') => Some(self.scan_char(Token::Dot)), + Some(':') => Some(self.scan_char(Token::Colon)), + Some('=') => Some(self.scan_char(Token::Equal)), + Some('-') => { + if self.looking_at("->") { + Some(self.scan_chars(2, Token::Arrow)) + } else { + Some(self.scan_number()) + } + } + Some(ch) if ch.is_digit(10) => Some(self.scan_number()), + Some(ch) if ch.is_alphabetic() => Some(self.scan_word()), + Some('%') => Some(self.scan_name()), + Some('#') => Some(self.scan_hex_sequence()), + Some(ch) if ch.is_whitespace() => { + self.next_ch(); + continue; + } + _ => { + // Skip invalid char, return error. + self.next_ch(); + Some(error(Error::InvalidChar, loc)) + } + }; } } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index c982a5be0c..44b72afc4a 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -28,7 +28,12 @@ use sourcemap::{SourceMap, MutableSourceMap}; /// /// Any test commands or ISA declarations are ignored. pub fn parse_functions(text: &str) -> Result> { - parse_test(text).map(|file| file.functions.into_iter().map(|(func, _)| func).collect()) + parse_test(text).map(|file| { + file.functions + .into_iter() + .map(|(func, _)| func) + .collect() + }) } /// Parse the entire `text` as a test case file. @@ -45,11 +50,11 @@ pub fn parse_test<'a>(text: &'a str) -> Result> { let functions = parser.parse_function_list(isa_spec.unique_isa())?; Ok(TestFile { - commands: commands, - isa_spec: isa_spec, - preamble_comments: preamble_comments, - functions: functions, - }) + commands: commands, + isa_spec: isa_spec, + preamble_comments: preamble_comments, + functions: functions, + }) } pub struct Parser<'a> { @@ -114,7 +119,12 @@ impl<'a> Context<'a> { // Allocate a new signature and add a mapping number -> SigRef. fn add_sig(&mut self, number: u32, data: Signature, loc: &Location) -> Result<()> { - self.map.def_sig(number, self.function.dfg.signatures.push(data), loc) + self.map.def_sig(number, + self.function + .dfg + .signatures + .push(data), + loc) } // Resolve a reference to a signature. @@ -127,7 +137,12 @@ impl<'a> Context<'a> { // Allocate a new external function and add a mapping number -> FuncRef. fn add_fn(&mut self, number: u32, data: ExtFuncData, loc: &Location) -> Result<()> { - self.map.def_fn(number, self.function.dfg.ext_funcs.push(data), loc) + self.map.def_fn(number, + self.function + .dfg + .ext_funcs + .push(data), + loc) } // Resolve a reference to a function. @@ -269,9 +284,9 @@ impl<'a> Parser<'a> { // Gather comments, associate them with `comment_entity`. if let Some(entity) = self.comment_entity { self.comments.push(Comment { - entity: entity, - text: text, - }); + entity: entity, + text: text, + }); } } _ => self.lookahead = Some(token), @@ -777,23 +792,35 @@ impl<'a> Parser<'a> { match self.token() { Some(Token::StackSlot(..)) => { self.gather_comments(ctx.function.stack_slots.next_key()); - self.parse_stack_slot_decl() - .and_then(|(num, dat)| ctx.add_ss(num, dat, &self.loc)) + self.parse_stack_slot_decl().and_then(|(num, dat)| { + ctx.add_ss(num, dat, &self.loc) + }) } Some(Token::SigRef(..)) => { - self.gather_comments(ctx.function.dfg.signatures.next_key()); - self.parse_signature_decl(ctx.unique_isa) - .and_then(|(num, dat)| ctx.add_sig(num, dat, &self.loc)) + self.gather_comments(ctx.function + .dfg + .signatures + .next_key()); + self.parse_signature_decl(ctx.unique_isa).and_then(|(num, dat)| { + ctx.add_sig(num, + dat, + &self.loc) + }) } Some(Token::FuncRef(..)) => { - self.gather_comments(ctx.function.dfg.ext_funcs.next_key()); - self.parse_function_decl(ctx) - .and_then(|(num, dat)| ctx.add_fn(num, dat, &self.loc)) + self.gather_comments(ctx.function + .dfg + .ext_funcs + .next_key()); + self.parse_function_decl(ctx).and_then(|(num, dat)| { + ctx.add_fn(num, dat, &self.loc) + }) } Some(Token::JumpTable(..)) => { self.gather_comments(ctx.function.jump_tables.next_key()); - self.parse_jump_table_decl() - .and_then(|(num, dat)| ctx.add_jt(num, dat, &self.loc)) + self.parse_jump_table_decl().and_then(|(num, dat)| { + ctx.add_jt(num, dat, &self.loc) + }) } // More to come.. _ => return Ok(()), @@ -852,7 +879,10 @@ impl<'a> Parser<'a> { let data = match self.token() { Some(Token::Identifier("function")) => { let (loc, name, sig) = self.parse_function_spec(ctx.unique_isa)?; - let sigref = ctx.function.dfg.signatures.push(sig); + let sigref = ctx.function + .dfg + .signatures + .push(sig); ctx.map.def_entity(sigref.into(), &loc).expect("duplicate SigRef entities created"); ExtFuncData { name: name, @@ -944,11 +974,11 @@ impl<'a> Parser<'a> { // extended-basic-block ::= ebb-header * { instruction } while match self.token() { - Some(Token::Value(_)) => true, - Some(Token::Identifier(_)) => true, - Some(Token::LBracket) => true, - _ => false, - } { + Some(Token::Value(_)) => true, + Some(Token::Identifier(_)) => true, + Some(Token::LBracket) => true, + _ => false, + } { self.parse_instruction(ctx, ebb)?; } @@ -1161,7 +1191,10 @@ impl<'a> Parser<'a> { ctx.function.dfg.inst_results(inst))?; if let Some(result_locations) = result_locations { - for (value, loc) in ctx.function.dfg.inst_results(inst).zip(result_locations) { + for (value, loc) in ctx.function + .dfg + .inst_results(inst) + .zip(result_locations) { *ctx.function.locations.ensure(value) = loc; } } @@ -1197,13 +1230,13 @@ impl<'a> Parser<'a> { let ctrl_src_value = inst_data.typevar_operand(&ctx.function.dfg.value_lists) .expect("Constraints <-> Format inconsistency"); ctx.function.dfg.value_type(match ctx.map.get_value(ctrl_src_value) { - Some(v) => v, - None => { - return err!(self.loc, - "cannot determine type of operand {}", - ctrl_src_value); - } - }) + Some(v) => v, + None => { + return err!(self.loc, + "cannot determine type of operand {}", + ctrl_src_value); + } + }) } else if constraints.is_polymorphic() { // This opcode does not support type inference, so the explicit type variable // is required. @@ -1290,7 +1323,7 @@ impl<'a> Parser<'a> { ctx: &mut Context, opcode: Opcode) -> Result { - Ok(match opcode.format() { + let idata = match opcode.format() { InstructionFormat::Nullary => { InstructionData::Nullary { opcode: opcode, @@ -1502,7 +1535,8 @@ impl<'a> Parser<'a> { table: table, } } - }) + }; + Ok(idata) } } @@ -1555,8 +1589,8 @@ mod tests { ss3 = stack_slot 13 ss1 = stack_slot 1 }") - .parse_function(None) - .unwrap(); + .parse_function(None) + .unwrap(); assert_eq!(func.name.to_string(), "foo"); let mut iter = func.stack_slots.keys(); let ss0 = iter.next().unwrap(); @@ -1572,9 +1606,9 @@ mod tests { ss1 = stack_slot 13 ss1 = stack_slot 1 }") - .parse_function(None) - .unwrap_err() - .to_string(), + .parse_function(None) + .unwrap_err() + .to_string(), "3: duplicate stack slot: ss1"); } @@ -1584,8 +1618,8 @@ mod tests { ebb0: ebb4(vx3: i32): }") - .parse_function(None) - .unwrap(); + .parse_function(None) + .unwrap(); assert_eq!(func.name.to_string(), "ebbs"); let mut ebbs = func.layout.ebbs(); @@ -1602,8 +1636,7 @@ mod tests { #[test] fn comments() { - let (func, Details { comments, .. }) = - Parser::new("; before + let (func, Details { comments, .. }) = Parser::new("; before function comment() { ; decl ss10 = stack_slot 13 ; stackslot. ; Still stackslot. @@ -1645,7 +1678,7 @@ mod tests { set enable_float=false ; still preamble function comment() {}") - .unwrap(); + .unwrap(); assert_eq!(tf.commands.len(), 2); assert_eq!(tf.commands[0].command, "cfg"); assert_eq!(tf.commands[1].command, "verify"); @@ -1664,18 +1697,18 @@ mod tests { fn isa_spec() { assert!(parse_test("isa function foo() {}") - .is_err()); + .is_err()); assert!(parse_test("isa riscv set enable_float=false function foo() {}") - .is_err()); + .is_err()); match parse_test("set enable_float=false isa riscv function foo() {}") - .unwrap() - .isa_spec { + .unwrap() + .isa_spec { IsaSpec::None(_) => panic!("Expected some ISA"), IsaSpec::Some(v) => { assert_eq!(v.len(), 1); diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index dda6420220..cbea4b02a2 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -63,23 +63,27 @@ impl SourceMap { /// Returns the entity reference corresponding to `name`, if it exists. pub fn lookup_str(&self, name: &str) -> Option { split_entity_name(name).and_then(|(ent, num)| match ent { - "v" => { - Value::direct_with_number(num) - .and_then(|v| self.get_value(v)) - .map(AnyEntity::Value) - } - "vx" => { - Value::table_with_number(num) - .and_then(|v| self.get_value(v)) - .map(AnyEntity::Value) - } - "ebb" => Ebb::with_number(num).and_then(|e| self.get_ebb(e)).map(AnyEntity::Ebb), - "ss" => self.get_ss(num).map(AnyEntity::StackSlot), - "sig" => self.get_sig(num).map(AnyEntity::SigRef), - "fn" => self.get_fn(num).map(AnyEntity::FuncRef), - "jt" => self.get_jt(num).map(AnyEntity::JumpTable), - _ => None, - }) + "v" => { + Value::direct_with_number(num) + .and_then(|v| self.get_value(v)) + .map(AnyEntity::Value) + } + "vx" => { + Value::table_with_number(num) + .and_then(|v| self.get_value(v)) + .map(AnyEntity::Value) + } + "ebb" => { + Ebb::with_number(num) + .and_then(|e| self.get_ebb(e)) + .map(AnyEntity::Ebb) + } + "ss" => self.get_ss(num).map(AnyEntity::StackSlot), + "sig" => self.get_sig(num).map(AnyEntity::SigRef), + "fn" => self.get_fn(num).map(AnyEntity::FuncRef), + "jt" => self.get_jt(num).map(AnyEntity::JumpTable), + _ => None, + }) } /// Get the source location where an entity was defined. @@ -229,7 +233,7 @@ mod tests { ebb0(v4: i32, vx7: i32): v10 = iadd v4, vx7 }") - .unwrap(); + .unwrap(); let map = &tf.functions[0].1.map; assert_eq!(map.lookup_str("v0"), None); From 50e043477ee1104c2abb482823eb44c1a1b7c5d5 Mon Sep 17 00:00:00 2001 From: Keith Yeung Date: Sun, 8 Jan 2017 18:58:12 -0800 Subject: [PATCH 0589/3084] Parse instruction results separately from instructions --- lib/reader/src/parser.rs | 61 ++++++++++++++++++++++++++++++++-------- 1 file changed, 50 insertions(+), 11 deletions(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 44b72afc4a..7c310cbf1d 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -979,7 +979,26 @@ impl<'a> Parser<'a> { Some(Token::LBracket) => true, _ => false, } { - self.parse_instruction(ctx, ebb)?; + let (encoding, result_locations) = self.parse_instruction_encoding(ctx)?; + + // We need to parse instruction results here because they are shared + // between the parsing of value aliases and the parsing of instructions. + // + // inst-results ::= Value(v) { "," Value(vx) } + let results = self.parse_inst_results()?; + + match self.token() { + Some(Token::Arrow) => { + self.consume(); + self.parse_value_alias(results, ctx)?; + } + Some(Token::Equal) => { + self.consume(); + self.parse_instruction(results, encoding, result_locations, ctx, ebb)?; + } + _ if results.len() != 0 => return err!(self.loc, "expected -> or ="), + _ => self.parse_instruction(results, encoding, result_locations, ctx, ebb)?, + } } Ok(()) @@ -1099,17 +1118,11 @@ impl<'a> Parser<'a> { Ok((encoding, result_locations)) } - // Parse an instruction, append it to `ebb`. + // Parse instruction results and return them. // - // instruction ::= [inst-results "="] Opcode(opc) ["." Type] ... // inst-results ::= Value(v) { "," Value(vx) } // - fn parse_instruction(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { - // Collect comments for the next instruction to be allocated. - self.gather_comments(ctx.function.dfg.next_inst()); - - let (encoding, result_locations) = self.parse_instruction_encoding(ctx)?; - + fn parse_inst_results(&mut self) -> Result> { // Result value numbers. let mut results = Vec::new(); @@ -1124,10 +1137,36 @@ impl<'a> Parser<'a> { // inst-results ::= Value(v) { "," * Value(vx) } results.push(self.match_value("expected result value")?); } - - self.match_token(Token::Equal, "expected '=' before opcode")?; } + Ok(results) + } + + // Parse a value alias, and append it to `ebb`. + // + // value_alias ::= [inst-results] "->" Value(vx) + // + fn parse_value_alias(&mut self, results: Vec, ctx: &mut Context) -> Result<()> { + if results.len() != 1 { + return err!(self.loc, "wrong number of aliases"); + } + Ok(()) + } + + // Parse an instruction, append it to `ebb`. + // + // instruction ::= [inst-results "="] Opcode(opc) ["." Type] ... + // + fn parse_instruction(&mut self, + results: Vec, + encoding: Option, + result_locations: Option>, + ctx: &mut Context, + ebb: Ebb) + -> Result<()> { + // Collect comments for the next instruction to be allocated. + self.gather_comments(ctx.function.dfg.next_inst()); + // instruction ::= [inst-results "="] * Opcode(opc) ["." Type] ... let opcode = if let Some(Token::Identifier(text)) = self.token() { match text.parse() { From 8e38c7907d281511aedc16df3e284593b0add957 Mon Sep 17 00:00:00 2001 From: Keith Yeung Date: Mon, 13 Mar 2017 21:29:22 -0700 Subject: [PATCH 0590/3084] Create alias HashMap in parser context --- lib/reader/src/parser.rs | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 7c310cbf1d..858049efc8 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -5,6 +5,7 @@ // // ====--------------------------------------------------------------------------------------====// +use std::collections::HashMap; use std::str::FromStr; use std::{u16, u32}; use std::mem; @@ -83,6 +84,8 @@ pub struct Parser<'a> { struct Context<'a> { function: Function, map: SourceMap, + // Store aliases until the values can be reliably looked up. + aliases: HashMap, // Reference to the unique_isa for things like parsing ISA-specific instruction encoding // information. This is only `Some` if exactly one set of `isa` directives were found in the @@ -96,6 +99,7 @@ impl<'a> Context<'a> { Context { function: f, map: SourceMap::new(), + aliases: HashMap::new(), unique_isa: unique_isa, } } @@ -173,6 +177,13 @@ impl<'a> Context<'a> { self.map.def_ebb(src_ebb, ebb, loc).and(Ok(ebb)) } + fn add_alias(&mut self, src: Value, dest: Value, loc: Location) -> Result<()> { + match self.aliases.insert(src, (dest, loc)) { + Some((v, _)) if v != dest => err!(loc, "duplicate alias: {} -> {}", src, dest), + _ => Ok(()), + } + } + // The parser creates all instructions with Ebb and Value references using the source file // numbering. These references need to be rewritten after parsing is complete since forward // references are allowed. @@ -1150,7 +1161,8 @@ impl<'a> Parser<'a> { if results.len() != 1 { return err!(self.loc, "wrong number of aliases"); } - Ok(()) + let dest = self.match_value("expected value alias")?; + ctx.add_alias(results[0], dest, self.loc) } // Parse an instruction, append it to `ebb`. From d93c37a826222b958a50aa20839b97036f83c28d Mon Sep 17 00:00:00 2001 From: Keith Yeung Date: Tue, 17 Jan 2017 01:17:29 -0800 Subject: [PATCH 0591/3084] Make value aliases during references rewriting --- lib/cretonne/src/ir/dfg.rs | 13 +++++++++++++ lib/reader/src/parser.rs | 18 ++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index f5d48ad81f..c944582031 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -234,6 +234,19 @@ impl DataFlowGraph { panic!("Cannot change direct value {} into an alias", dest); } } + + /// Create a new value alias. + /// + /// Note that this function should only be called by the parser. + pub fn make_value_alias(&mut self, src: Value) -> Value { + let ty = self.value_type(src); + + let data = ValueData::Alias { + ty: ty, + original: src, + }; + self.make_value(data) + } } /// Where did a value come from? diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 858049efc8..21de8bcda7 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -188,6 +188,24 @@ impl<'a> Context<'a> { // numbering. These references need to be rewritten after parsing is complete since forward // references are allowed. fn rewrite_references(&mut self) -> Result<()> { + for (&source_from, &(source_to, source_loc)) in &self.aliases { + let ir_to = match self.map.get_value(source_to) { + Some(v) => v, + None => { + return err!(source_loc, + "IR destination value alias not found for {}", + source_to); + } + }; + let dest_loc = self.map + .location(AnyEntity::from(ir_to)) + .expect(&*format!("Error in looking up location of IR destination value alias \ + for {}", + ir_to)); + let ir_from = self.function.dfg.make_value_alias(ir_to); + self.map.def_value(source_from, ir_from, &dest_loc)?; + } + for ebb in self.function.layout.ebbs() { for inst in self.function.layout.ebb_insts(ebb) { let loc = inst.into(); From f18eddc9eddf87de7f1fde9b4245b4f801930613 Mon Sep 17 00:00:00 2001 From: Keith Yeung Date: Tue, 14 Mar 2017 10:25:30 -0700 Subject: [PATCH 0592/3084] Allow type inference to go through value aliasing --- lib/reader/src/parser.rs | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 21de8bcda7..9f29e54090 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1299,13 +1299,20 @@ impl<'a> Parser<'a> { let ctrl_src_value = inst_data.typevar_operand(&ctx.function.dfg.value_lists) .expect("Constraints <-> Format inconsistency"); ctx.function.dfg.value_type(match ctx.map.get_value(ctrl_src_value) { - Some(v) => v, - None => { - return err!(self.loc, - "cannot determine type of operand {}", - ctrl_src_value); - } - }) + Some(v) => v, + None => { + if let Some(v) = ctx.aliases + .get(&ctrl_src_value) + .and_then(|&(aliased, _)| ctx.map.get_value(aliased)) + { + v + } else { + return err!(self.loc, + "cannot determine type of operand {}", + ctrl_src_value); + } + } + }) } else if constraints.is_polymorphic() { // This opcode does not support type inference, so the explicit type variable // is required. From 98e8a964127680dd9379d524a4d3294c538ddbee Mon Sep 17 00:00:00 2001 From: Keith Yeung Date: Tue, 14 Mar 2017 11:17:20 -0700 Subject: [PATCH 0593/3084] Add unit test for value aliasing --- lib/reader/src/parser.rs | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 9f29e54090..414dd291e2 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1638,6 +1638,25 @@ mod tests { assert_eq!(message, "expected argument type"); } + #[test] + fn aliases() { + let (func, details) = Parser::new("function qux() { + ebb0: + v4 = iconst.i8 6 + vx3 -> v4 + v1 = iadd_imm vx3, 17 + }") + .parse_function(None) + .unwrap(); + assert_eq!(func.name.to_string(), "qux"); + let v4 = details.map.lookup_str("v4").unwrap(); + assert_eq!(v4.to_string(), "v0"); + let vx3 = details.map.lookup_str("vx3").unwrap(); + assert_eq!(vx3.to_string(), "vx0"); + let aliased_to = func.dfg.resolve_aliases(Value::table_with_number(0).unwrap()); + assert_eq!(aliased_to.to_string(), "v0"); + } + #[test] fn signature() { let sig = Parser::new("()").parse_signature(None).unwrap(); From 7db4b1b73a7ab0f9d5b8cdd5b3417bcd6b843a6b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 14 Mar 2017 13:17:21 -0700 Subject: [PATCH 0594/3084] Add OpcodeConstraints::fixed_value_arguments() Now that some instruction formats put all of their value arguments in a value list, we need to know how many value are fixed and how many are variable_args. CC @angusholder who may need this information in the verifier. --- lib/cretonne/meta/gen_instr.py | 9 ++++++--- lib/cretonne/src/ir/instructions.rs | 30 +++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index e6eaa82c32..02677bfdbd 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -439,6 +439,7 @@ def gen_type_constraints(fmt, instrs): get_constraint(i.ins[opnum], ctrl_typevar, type_sets)) offset = operand_seqs.add(constraints) fixed_results = len(i.value_results) + fixed_values = len(i.value_opnums) # Can the controlling type variable be inferred from the designated # operand? use_typevar_operand = i.is_polymorphic and i.use_typevar_operand @@ -449,10 +450,10 @@ def gen_type_constraints(fmt, instrs): # result? requires_typevar_operand = use_typevar_operand and not use_result fmt.comment( - ('{}: fixed_results={}, use_typevar_operand={}, ' + - 'requires_typevar_operand={}') + '{}: fixed_results={}, use_typevar_operand={}, ' + 'requires_typevar_operand={}, fixed_values={}' .format(i.camel_name, fixed_results, use_typevar_operand, - requires_typevar_operand)) + requires_typevar_operand, fixed_values)) fmt.comment('Constraints={}'.format(constraints)) if i.is_polymorphic: fmt.comment( @@ -464,6 +465,8 @@ def gen_type_constraints(fmt, instrs): flags |= 8 if requires_typevar_operand: flags |= 0x10 + assert fixed_values < 8, "Bit field encoding too tight" + flags |= fixed_values << 5 with fmt.indented('OpcodeConstraints {', '},'): fmt.line('flags: {:#04x},'.format(flags)) diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index de34a708c9..7e970f73b8 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -387,6 +387,9 @@ pub struct OpcodeConstraints { /// Bit 4: /// This opcode is polymorphic and the controlling type variable does *not* appear as the /// first result type. + /// + /// Bits 5-7: + /// Number of fixed value arguments. The minimum required number of value operands. flags: u8, /// Permitted set of types for the controlling type variable as an index into `TYPE_SETS`. @@ -423,6 +426,17 @@ impl OpcodeConstraints { (self.flags & 0x7) as usize } + /// Get the number of *fixed* input values required by this opcode. + /// + /// This does not include `variable_args` arguments on call and branch instructions. + /// + /// The number of fixed input values is usually implied by the instruction format, but + /// instruction formats that use a `ValueList` put both fixed and variable arguments in the + /// list. This method returns the *minimum* number of values required in the value list. + pub fn fixed_value_arguments(self) -> usize { + ((self.flags >> 5) & 0x7) as usize + } + /// Get the offset into `TYPE_SETS` for the controlling type variable. /// Returns `None` if the instruction is not polymorphic. fn typeset_offset(self) -> Option { @@ -609,6 +623,22 @@ mod tests { assert_eq!(mem::size_of::(), 16); } + #[test] + fn constraints() { + let a = Opcode::Iadd.constraints(); + assert!(a.use_typevar_operand()); + assert_eq!(a.fixed_results(), 1); + assert_eq!(a.fixed_value_arguments(), 2); + + let c = Opcode::Call.constraints(); + assert_eq!(c.fixed_results(), 0); + assert_eq!(c.fixed_value_arguments(), 0); + + let i = Opcode::CallIndirect.constraints(); + assert_eq!(i.fixed_results(), 0); + assert_eq!(i.fixed_value_arguments(), 1); + } + #[test] fn value_set() { use ir::types::*; From d604855b27f4656423465cdf7774631c1134c8aa Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 14 Mar 2017 13:31:01 -0700 Subject: [PATCH 0595/3084] Fix logic bug in requires_typevar_operand. The Python code computing this property had a bug. The property has only been used for optimizing the ctrl_typevar() method so far. --- lib/cretonne/meta/gen_instr.py | 2 +- lib/cretonne/src/ir/instructions.rs | 13 +++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 02677bfdbd..f3c36a716f 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -445,7 +445,7 @@ def gen_type_constraints(fmt, instrs): use_typevar_operand = i.is_polymorphic and i.use_typevar_operand # Can the controlling type variable be inferred from the result? use_result = (fixed_results > 0 and - i.outs[i.value_results[0]].typevar != ctrl_typevar) + i.outs[i.value_results[0]].typevar == ctrl_typevar) # Are we required to use the designated operand instead of the # result? requires_typevar_operand = use_typevar_operand and not use_result diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 7e970f73b8..c51afcc118 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -627,9 +627,16 @@ mod tests { fn constraints() { let a = Opcode::Iadd.constraints(); assert!(a.use_typevar_operand()); + assert!(!a.requires_typevar_operand()); assert_eq!(a.fixed_results(), 1); assert_eq!(a.fixed_value_arguments(), 2); + let b = Opcode::Bitcast.constraints(); + assert!(!b.use_typevar_operand()); + assert!(!b.requires_typevar_operand()); + assert_eq!(b.fixed_results(), 1); + assert_eq!(b.fixed_value_arguments(), 1); + let c = Opcode::Call.constraints(); assert_eq!(c.fixed_results(), 0); assert_eq!(c.fixed_value_arguments(), 0); @@ -637,6 +644,12 @@ mod tests { let i = Opcode::CallIndirect.constraints(); assert_eq!(i.fixed_results(), 0); assert_eq!(i.fixed_value_arguments(), 1); + + let cmp = Opcode::Icmp.constraints(); + assert!(cmp.use_typevar_operand()); + assert!(cmp.requires_typevar_operand()); + assert_eq!(cmp.fixed_results(), 1); + assert_eq!(cmp.fixed_value_arguments(), 2); } #[test] From 9549cf603cbf168370a7e6118668e2de45b36a15 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 15 Mar 2017 11:04:38 -0700 Subject: [PATCH 0596/3084] Add a primitive debug tracing facility. When the CRETONNE_DBG environment variable is set, send debug messages to a file named cretonne.dbg.*. The trace facility is only enabled when debug assertions are on. --- cranelift/.gitignore | 1 + cranelift/src/cton-util.rs | 2 +- cranelift/src/filetest/concurrent.rs | 4 ++ cranelift/src/filetest/runone.rs | 2 + lib/cretonne/src/dbg.rs | 100 +++++++++++++++++++++++++++ lib/cretonne/src/lib.rs | 3 + 6 files changed, 111 insertions(+), 1 deletion(-) create mode 100644 lib/cretonne/src/dbg.rs diff --git a/cranelift/.gitignore b/cranelift/.gitignore index 765772864e..5f3b8f2899 100644 --- a/cranelift/.gitignore +++ b/cranelift/.gitignore @@ -6,3 +6,4 @@ tags target Cargo.lock .*.rustfmt +cretonne.dbg* diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 89fee3c5a5..681ab5b8bc 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -1,4 +1,4 @@ - +#[macro_use(dbg)] extern crate cretonne; extern crate cton_reader; extern crate docopt; diff --git a/cranelift/src/filetest/concurrent.rs b/cranelift/src/filetest/concurrent.rs index aa0fba8f1d..effe454d38 100644 --- a/cranelift/src/filetest/concurrent.rs +++ b/cranelift/src/filetest/concurrent.rs @@ -137,6 +137,10 @@ fn worker_thread(thread_num: usize, } }); + if let &Err(ref msg) = &result { + dbg!("FAIL: {}", msg); + } + replies.send(Reply::Done { jobid: jobid, result: result, diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index 8e707d4ff5..ed4cae1ee7 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -17,6 +17,7 @@ use filetest::subtest::{SubTest, Context, Result}; /// /// If running this test causes a panic, it will propagate as normal. pub fn run(path: &Path) -> TestResult { + dbg!("---\nFile: {}", path.to_string_lossy()); let started = time::Instant::now(); let buffer = read_to_string(path).map_err(|e| e.to_string())?; let testfile = parse_test(&buffer).map_err(|e| e.to_string())?; @@ -108,6 +109,7 @@ fn run_one_test<'a>(tuple: (&'a SubTest, &'a Flags, Option<&'a TargetIsa>), -> Result<()> { let (test, flags, isa) = tuple; let name = format!("{}({})", test.name(), func.name); + dbg!("Test: {} {}", name, isa.map(TargetIsa::name).unwrap_or("-")); context.flags = flags; context.isa = isa; diff --git a/lib/cretonne/src/dbg.rs b/lib/cretonne/src/dbg.rs new file mode 100644 index 0000000000..06723fa018 --- /dev/null +++ b/lib/cretonne/src/dbg.rs @@ -0,0 +1,100 @@ +//! Debug tracing macros. +//! +//! This module defines the `dbg!` macro which works like `println!` except it writes to the +//! Cretonne tracing output file if enabled. +//! +//! Tracing can be enabled by setting the `CRETONNE_DBG` environment variable to something +/// other than `0`. +/// +/// The output will appear in files named `cretonne.dbg.*`, where the suffix is named after the +/// thread doing the logging. + +use std::ascii::AsciiExt; +use std::cell::RefCell; +use std::env; +use std::ffi::OsStr; +use std::fs::File; +use std::io::{Write, BufWriter}; +use std::sync::atomic; +use std::thread; + +static STATE: atomic::AtomicIsize = atomic::ATOMIC_ISIZE_INIT; + +/// Is debug tracing enabled? +/// +/// Debug tracing can be enabled by setting the `CRETONNE_DBG` environment variable to something +/// other than `0`. +/// +/// This inline function turns into a constant `false` when debug assertions are disabled. +#[inline] +pub fn enabled() -> bool { + if cfg!(debug_assertions) { + match STATE.load(atomic::Ordering::Relaxed) { + 0 => initialize(), + s => s > 0, + } + } else { + false + } +} + +/// Initialize `STATE` from the environment variable. +fn initialize() -> bool { + let enable = match env::var_os("CRETONNE_DBG") { + Some(s) => s != OsStr::new("0"), + None => false, + }; + + if enable { + STATE.store(1, atomic::Ordering::Relaxed); + } else { + STATE.store(-1, atomic::Ordering::Relaxed); + } + + enable +} + +thread_local! { + static WRITER : RefCell> = RefCell::new(open_file()); +} + +/// Execute a closure with mutable access to the tracing file writer. +/// +/// This is for use by the `dbg!` macro. +pub fn with_writer(f: F) -> R + where F: FnOnce(&mut Write) -> R +{ + WRITER.with(|rc| f(&mut *rc.borrow_mut())) +} + +/// Open the tracing file for the current thread. +fn open_file() -> BufWriter { + let file = match thread::current().name() { + None => File::create("cretonne.dbg"), + Some(name) => { + let mut path = "cretonne.dbg.".to_owned(); + for ch in name.chars() { + if ch.is_ascii() && ch.is_alphanumeric() { + path.push(ch); + } + } + File::create(path) + } + } + .expect("Can't open tracing file"); + BufWriter::new(file) +} + +/// Write a line to the debug trace file if tracing is enabled. +/// +/// Arguments are the same as for `printf!`. +#[macro_export] +macro_rules! dbg { + ($($arg:tt)+) => { + if $crate::dbg::enabled() { + // Drop the error result so we don't get compiler errors for ignoring it. + // What are you going to do, log the error? + $crate::dbg::with_writer(|w| { writeln!(w, $($arg)+).ok(); }) + } + } +} diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 2839ecfc1f..733506a605 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -21,6 +21,9 @@ pub mod settings; pub mod sparse_map; pub mod verifier; +#[macro_use] +pub mod dbg; + mod abi; mod constant_hash; mod context; From 01cb51ece97e493a555e7c7af16924fe1091bd52 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 15 Mar 2017 13:40:58 -0700 Subject: [PATCH 0597/3084] Add DataFlowGraph::display_inst(). This method returns an object that can display an instruction in the standard textual format, but without any encoding annotations. --- lib/cretonne/src/ir/dfg.rs | 37 ++++++++++++++++ lib/cretonne/src/write.rs | 86 +++++++++++++++++++------------------- 2 files changed, 80 insertions(+), 43 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index c944582031..699f2a2f1a 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -8,7 +8,9 @@ use entity_map::{EntityMap, PrimaryEntityData}; use ir::builder::{InsertBuilder, ReplaceBuilder}; use ir::layout::Cursor; use packed_option::PackedOption; +use write::write_operands; +use std::fmt; use std::ops::{Index, IndexMut}; use std::u16; @@ -337,6 +339,11 @@ impl DataFlowGraph { self.insts.next_key() } + /// Returns an object that displays `inst`. + pub fn display_inst(&self, inst: Inst) -> DisplayInst { + DisplayInst(self, inst) + } + /// Create result values for an instruction that produces multiple results. /// /// Instructions that produce 0 or 1 result values only need to be created with `make_inst`. If @@ -649,6 +656,34 @@ impl EbbData { } } +/// Object that can display an instruction. +pub struct DisplayInst<'a>(&'a DataFlowGraph, Inst); + +impl<'a> fmt::Display for DisplayInst<'a> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + let dfg = self.0; + let inst = &dfg[self.1]; + + let mut results = dfg.inst_results(self.1); + if let Some(first) = results.next() { + write!(f, "{}", first)?; + for v in results { + write!(f, ", {}", v)?; + } + write!(f, " = ")?; + } + + + let typevar = inst.ctrl_typevar(dfg); + if typevar.is_void() { + write!(f, "{}", inst.opcode())?; + } else { + write!(f, "{}.{}", inst.opcode(), typevar)?; + } + write_operands(f, dfg, self.1) + } +} + #[cfg(test)] mod tests { use super::*; @@ -665,6 +700,7 @@ mod tests { }; let inst = dfg.make_inst(idata); assert_eq!(inst.to_string(), "inst0"); + assert_eq!(dfg.display_inst(inst).to_string(), "v0 = iconst.i32"); // Immutable reference resolution. { @@ -692,6 +728,7 @@ mod tests { ty: types::VOID, }; let inst = dfg.make_inst(idata); + assert_eq!(dfg.display_inst(inst).to_string(), "trap"); // Result iterator should be empty. let mut res = dfg.inst_results(inst); diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 081da35617..dfa7688f20 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -4,7 +4,7 @@ //! equivalent textual representation. This textual representation can be read back by the //! `cretonne-reader` crate. -use ir::{Function, Ebb, Inst, Value, Type}; +use ir::{Function, DataFlowGraph, Ebb, Inst, Value, Type}; use isa::{TargetIsa, RegInfo}; use std::fmt::{self, Result, Error, Write}; use std::result; @@ -228,69 +228,69 @@ fn write_instruction(w: &mut Write, None => write!(w, "{}", opcode)?, } - // Then the operands, depending on format. - let pool = &func.dfg.value_lists; + write_operands(w, &func.dfg, inst)?; + writeln!(w, "") +} + +/// Write the operands of `inst` to `w` with a prepended space. +pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result { + let pool = &dfg.value_lists; use ir::instructions::InstructionData::*; - match func.dfg[inst] { - Nullary { .. } => writeln!(w, ""), - Unary { arg, .. } => writeln!(w, " {}", arg), - UnaryImm { imm, .. } => writeln!(w, " {}", imm), - UnaryIeee32 { imm, .. } => writeln!(w, " {}", imm), - UnaryIeee64 { imm, .. } => writeln!(w, " {}", imm), - UnarySplit { arg, .. } => writeln!(w, " {}", arg), - Binary { args, .. } => writeln!(w, " {}, {}", args[0], args[1]), - BinaryImm { arg, imm, .. } => writeln!(w, " {}, {}", arg, imm), - BinaryOverflow { args, .. } => writeln!(w, " {}, {}", args[0], args[1]), - Ternary { args, .. } => writeln!(w, " {}, {}, {}", args[0], args[1], args[2]), + match dfg[inst] { + Nullary { .. } => write!(w, ""), + Unary { arg, .. } => write!(w, " {}", arg), + UnaryImm { imm, .. } => write!(w, " {}", imm), + UnaryIeee32 { imm, .. } => write!(w, " {}", imm), + UnaryIeee64 { imm, .. } => write!(w, " {}", imm), + UnarySplit { arg, .. } => write!(w, " {}", arg), + Binary { args, .. } => write!(w, " {}, {}", args[0], args[1]), + BinaryImm { arg, imm, .. } => write!(w, " {}, {}", arg, imm), + BinaryOverflow { args, .. } => write!(w, " {}, {}", args[0], args[1]), + Ternary { args, .. } => write!(w, " {}, {}, {}", args[0], args[1], args[2]), MultiAry { ref args, .. } => { if args.is_empty() { - writeln!(w, "") + write!(w, "") } else { - writeln!(w, - " {}", - DisplayValues(args.as_slice(&func.dfg.value_lists))) + write!(w, " {}", DisplayValues(args.as_slice(pool))) } } - InsertLane { lane, args, .. } => writeln!(w, " {}, {}, {}", args[0], lane, args[1]), - ExtractLane { lane, arg, .. } => writeln!(w, " {}, {}", arg, lane), - IntCompare { cond, args, .. } => writeln!(w, " {}, {}, {}", cond, args[0], args[1]), - FloatCompare { cond, args, .. } => writeln!(w, " {}, {}, {}", cond, args[0], args[1]), + InsertLane { lane, args, .. } => write!(w, " {}, {}, {}", args[0], lane, args[1]), + ExtractLane { lane, arg, .. } => write!(w, " {}, {}", arg, lane), + IntCompare { cond, args, .. } => write!(w, " {}, {}, {}", cond, args[0], args[1]), + FloatCompare { cond, args, .. } => write!(w, " {}, {}, {}", cond, args[0], args[1]), Jump { destination, ref args, .. } => { if args.is_empty() { - writeln!(w, " {}", destination) + write!(w, " {}", destination) } else { - writeln!(w, - " {}({})", - destination, - DisplayValues(args.as_slice(pool))) + write!(w, + " {}({})", + destination, + DisplayValues(args.as_slice(pool))) } } Branch { destination, ref args, .. } => { let args = args.as_slice(pool); if args.len() == 1 { - writeln!(w, " {}, {}", args[0], destination) + write!(w, " {}, {}", args[0], destination) } else { - writeln!(w, - " {}, {}({})", - args[0], - destination, - DisplayValues(&args[1..])) + write!(w, + " {}, {}({})", + args[0], + destination, + DisplayValues(&args[1..])) } } - BranchTable { arg, table, .. } => writeln!(w, " {}, {}", arg, table), + BranchTable { arg, table, .. } => write!(w, " {}, {}", arg, table), Call { func_ref, ref args, .. } => { - writeln!(w, - " {}({})", - func_ref, - DisplayValues(args.as_slice(&func.dfg.value_lists))) + write!(w, " {}({})", func_ref, DisplayValues(args.as_slice(pool))) } IndirectCall { sig_ref, ref args, .. } => { let args = args.as_slice(pool); - writeln!(w, - " {}, {}({})", - sig_ref, - args[0], - DisplayValues(&args[1..])) + write!(w, + " {}, {}({})", + sig_ref, + args[0], + DisplayValues(&args[1..])) } } } From d15f25844afb7018dcc8a140a7a3803eebbb5d02 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 8 Mar 2017 17:33:20 -0800 Subject: [PATCH 0598/3084] Legalize ABI arguments to call and return instructions. The type signatures of functions can change when they are legalized for a specific ABI. This means that all call and return instructions need to be rewritten to use the correct arguments. - Fix arguments to call instructions. - Fix arguments to return instructions. TBD: - Fix return values from call instructions. --- cranelift/filetests/isa/riscv/abi.cton | 98 +++++----- lib/cretonne/src/legalizer.rs | 249 ++++++++++++++++++++++++- 2 files changed, 302 insertions(+), 45 deletions(-) diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.cton index 0b47fe6727..50af69638f 100644 --- a/cranelift/filetests/isa/riscv/abi.cton +++ b/cranelift/filetests/isa/riscv/abi.cton @@ -7,27 +7,26 @@ isa riscv function f(i32) { sig0 = signature(i32) -> i32 - -; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] + ; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] sig1 = signature(i64) -> b1 -; check: sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] + ; check: sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] -; The i64 argument must go in an even-odd register pair. + ; The i64 argument must go in an even-odd register pair. sig2 = signature(f32, i64) -> f64 -; check: sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] + ; check: sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] -; Spilling into the stack args. + ; Spilling into the stack args. sig3 = signature(f64, f64, f64, f64, f64, f64, f64, i64) -> f64 -; check: sig3 = signature(f64 [%f10], f64 [%f11], f64 [%f12], f64 [%f13], f64 [%f14], f64 [%f15], f64 [%f16], i32 [0], i32 [4]) -> f64 [%f10] + ; check: sig3 = signature(f64 [%f10], f64 [%f11], f64 [%f12], f64 [%f13], f64 [%f14], f64 [%f15], f64 [%f16], i32 [0], i32 [4]) -> f64 [%f10] -; Splitting vectors. + ; Splitting vectors. sig4 = signature(i32x4) -; check: sig4 = signature(i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13]) + ; check: sig4 = signature(i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13]) -; Splitting vectors, then splitting ints. + ; Splitting vectors, then splitting ints. sig5 = signature(i64x4) -; check: sig5 = signature(i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [%x16], i32 [%x17]) + ; check: sig5 = signature(i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [%x16], i32 [%x17]) ebb0(v0: i32): return_reg v0 @@ -35,62 +34,75 @@ ebb0(v0: i32): function int_split_args(i64) -> i64 { ebb0(v0: i64): -; check: $ebb0($(v0l=$VX): i32, $(v0h=$VX): i32): -; check: iconcat_lohi $v0l, $v0h + ; check: $ebb0($(v0l=$VX): i32, $(v0h=$VX): i32): + ; check: iconcat_lohi $v0l, $v0h v1 = iadd_imm v0, 1 - return v0 + ; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 + ; check: return $v1l, $v1h + return v1 } -function int_ext(i8, i8 sext, i8 uext) -> i8 { +function int_ext(i8, i8 sext, i8 uext) -> i8 uext { ebb0(v1: i8, v2: i8, v3: i8): -; check: $ebb0($v1: i8, $(v2x=$VX): i32, $(v3x=$VX): i32): -; check: ireduce.i8 $v2x -; check: ireduce.i8 $v3x + ; check: $ebb0($v1: i8, $(v2x=$VX): i32, $(v3x=$VX): i32): + ; check: ireduce.i8 $v2x + ; check: ireduce.i8 $v3x + ; check: $(v1x=$V) = uextend.i32 $v1 + ; check: return $v1x + return v1 } -function vector_split_args(i64x4) -> i64 { +function vector_split_args(i64x4) -> i64x4 { ebb0(v0: i64x4): -; check: $ebb0($(v0al=$VX): i32, $(v0ah=$VX): i32, $(v0bl=$VX): i32, $(v0bh=$VX): i32, $(v0cl=$VX): i32, $(v0ch=$VX): i32, $(v0dl=$VX): i32, $(v0dh=$VX): i32): -; check: $(v0a=$V) = iconcat_lohi $v0al, $v0ah -; check: $(v0b=$V) = iconcat_lohi $v0bl, $v0bh -; check: $(v0ab=$V) = vconcat $v0a, $v0b -; check: $(v0c=$V) = iconcat_lohi $v0cl, $v0ch -; check: $(v0d=$V) = iconcat_lohi $v0dl, $v0dh -; check: $(v0cd=$V) = vconcat $v0c, $v0d -; check: $(v0abcd=$V) = vconcat $v0ab, $v0cd + ; check: $ebb0($(v0al=$VX): i32, $(v0ah=$VX): i32, $(v0bl=$VX): i32, $(v0bh=$VX): i32, $(v0cl=$VX): i32, $(v0ch=$VX): i32, $(v0dl=$VX): i32, $(v0dh=$VX): i32): + ; check: $(v0a=$V) = iconcat_lohi $v0al, $v0ah + ; check: $(v0b=$V) = iconcat_lohi $v0bl, $v0bh + ; check: $(v0ab=$V) = vconcat $v0a, $v0b + ; check: $(v0c=$V) = iconcat_lohi $v0cl, $v0ch + ; check: $(v0d=$V) = iconcat_lohi $v0dl, $v0dh + ; check: $(v0cd=$V) = vconcat $v0c, $v0d + ; check: $(v0abcd=$V) = vconcat $v0ab, $v0cd v1 = iadd v0, v0 - return v0 + ; check: $(v1ab=$V), $(v1cd=$VX) = vsplit + ; check: $(v1a=$V), $(v1b=$VX) = vsplit $v1ab + ; check: $(v1al=$V), $(v1ah=$VX) = isplit_lohi $v1a + ; check: $(v1bl=$V), $(v1bh=$VX) = isplit_lohi $v1b + ; check: $(v1c=$V), $(v1d=$VX) = vsplit $v1cd + ; check: $(v1cl=$V), $(v1ch=$VX) = isplit_lohi $v1c + ; check: $(v1dl=$V), $(v1dh=$VX) = isplit_lohi $v1d + ; check: return $v1al, $v1ah, $v1bl, $v1bh, $v1cl, $v1ch, $v1dl, $v1dh + return v1 } function parse_encoding(i32 [%x5]) -> i32 [%x10] { -; check: function parse_encoding(i32 [%x5]) -> i32 [%x10] { + ; check: function parse_encoding(i32 [%x5]) -> i32 [%x10] { sig0 = signature(i32 [%x10]) -> i32 [%x10] -; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] + ; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] -; check: sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] + ; check: sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] -; check: sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] + ; check: sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] -; Arguments on stack where not necessary + ; Arguments on stack where not necessary sig3 = signature(f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] -; check: sig3 = signature(f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] + ; check: sig3 = signature(f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] -; Stack argument before register argument + ; Stack argument before register argument sig4 = signature(f32 [72], i32 [%x10]) -; check: sig4 = signature(f32 [72], i32 [%x10]) + ; check: sig4 = signature(f32 [72], i32 [%x10]) -; Return value on stack + ; Return value on stack sig5 = signature() -> f32 [0] -; check: sig5 = signature() -> f32 [0] + ; check: sig5 = signature() -> f32 [0] -; function + signature + ; function + signature fn15 = function bar(i32 [%x10]) -> b1 [%x10] -; check: sig6 = signature(i32 [%x10]) -> b1 [%x10] -; nextln: fn0 = sig6 bar + ; check: sig6 = signature(i32 [%x10]) -> b1 [%x10] + ; nextln: fn0 = sig6 bar ebb0(v0: i32): - return_reg v0 -} \ No newline at end of file + return v0 +} diff --git a/lib/cretonne/src/legalizer.rs b/lib/cretonne/src/legalizer.rs index 9223a174b0..e22cb6c95e 100644 --- a/lib/cretonne/src/legalizer.rs +++ b/lib/cretonne/src/legalizer.rs @@ -14,9 +14,10 @@ //! from the encoding recipes, and solved later by the register allocator. use abi::{legalize_abi_value, ValueConversion}; -use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, InstBuilder, Ebb, Type, Value, - ArgumentType}; +use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, Inst, InstBuilder, Ebb, Type, + Value, Signature, SigRef, ArgumentType}; use ir::condcodes::IntCC; +use ir::instructions::CallInfo; use isa::{TargetIsa, Legalize}; /// Legalize `func` for `isa`. @@ -36,6 +37,21 @@ pub fn legalize_function(func: &mut Function, isa: &TargetIsa) { let mut prev_pos = pos.position(); while let Some(inst) = pos.next_inst() { + let opcode = func.dfg[inst].opcode(); + + // Check for ABI boundaries that need to be converted to the legalized signature. + if opcode.is_call() && handle_call_abi(&mut func.dfg, &mut pos) { + // Go back and legalize the inserted argument conversion instructions. + pos.set_position(prev_pos); + continue; + } + + if opcode.is_return() && handle_return_abi(&mut func.dfg, &mut pos, &func.signature) { + // Go back and legalize the inserted return value conversion instructions. + pos.set_position(prev_pos); + continue; + } + match isa.encode(&func.dfg, &func.dfg[inst]) { Ok(encoding) => *func.encodings.ensure(inst) = encoding, Err(action) => { @@ -201,3 +217,232 @@ fn convert_from_abi(dfg: &mut DataFlowGraph, } } } + +/// Convert `value` to match an ABI signature by inserting instructions at `pos`. +/// +/// This may require expanding the value to multiple ABI arguments. The conversion process is +/// recursive and controlled by the `put_arg` closure. When a candidate argument value is presented +/// to the closure, it will perform one of two actions: +/// +/// 1. If the suggested argument has an acceptable value type, consume it by adding it to the list +/// of arguments and return `None`. +/// 2. If the suggested argument doesn't have the right value type, don't change anything, but +/// return the `ArgumentType` that is needed. +/// +fn convert_to_abi(dfg: &mut DataFlowGraph, + pos: &mut Cursor, + value: Value, + put_arg: &mut PutArg) + where PutArg: FnMut(&mut DataFlowGraph, Value) -> Option +{ + // Start by invoking the closure to either terminate the recursion or get the argument type + // we're trying to match. + let arg_type = match put_arg(dfg, value) { + None => return, + Some(t) => t, + }; + + let ty = dfg.value_type(value); + match legalize_abi_value(ty, &arg_type) { + ValueConversion::IntSplit => { + let (lo, hi) = dfg.ins(pos).isplit_lohi(value); + convert_to_abi(dfg, pos, lo, put_arg); + convert_to_abi(dfg, pos, hi, put_arg); + } + ValueConversion::VectorSplit => { + let (lo, hi) = dfg.ins(pos).vsplit(value); + convert_to_abi(dfg, pos, lo, put_arg); + convert_to_abi(dfg, pos, hi, put_arg); + } + ValueConversion::IntBits => { + assert!(!ty.is_int()); + let abi_ty = Type::int(ty.bits()).expect("Invalid type for conversion"); + let arg = dfg.ins(pos).bitcast(abi_ty, value); + convert_to_abi(dfg, pos, arg, put_arg); + } + ValueConversion::Sext(abi_ty) => { + let arg = dfg.ins(pos).sextend(abi_ty, value); + convert_to_abi(dfg, pos, arg, put_arg); + } + ValueConversion::Uext(abi_ty) => { + let arg = dfg.ins(pos).uextend(abi_ty, value); + convert_to_abi(dfg, pos, arg, put_arg); + } + } +} + +/// Check if a sequence of arguments match a desired sequence of argument types. +fn check_arg_types(dfg: &DataFlowGraph, args: Args, types: &[ArgumentType]) -> bool + where Args: IntoIterator +{ + let mut n = 0; + for arg in args { + match types.get(n) { + Some(&ArgumentType { value_type, .. }) => { + if dfg.value_type(arg) != value_type { + return false; + } + } + None => return false, + } + n += 1 + } + + // Also verify that the number of arguments matches. + n == types.len() +} + +/// Check if the arguments of the call `inst` match the signature. +/// +/// Returns `None` if the signature matches and no changes are needed, or `Some(sig_ref)` if the +/// signature doesn't match. +fn check_call_signature(dfg: &DataFlowGraph, inst: Inst) -> Option { + // Extract the signature and argument values. + let (sig_ref, args) = match dfg[inst].analyze_call(&dfg.value_lists) { + CallInfo::Direct(func, args) => (dfg.ext_funcs[func].signature, args), + CallInfo::Indirect(sig_ref, args) => (sig_ref, args), + CallInfo::NotACall => panic!("Expected call, got {:?}", dfg[inst]), + }; + let sig = &dfg.signatures[sig_ref]; + + if check_arg_types(dfg, args.iter().cloned(), &sig.argument_types[..]) && + check_arg_types(dfg, dfg.inst_results(inst), &sig.return_types[..]) { + // All types check out. + None + } else { + // Call types need fixing. + Some(sig_ref) + } +} + +/// Insert ABI conversion code for the arguments to the call or return instruction at `pos`. +/// +/// - `abi_args` is the number of arguments that the ABI signature requires. +/// - `get_abi_type` is a closure that can provide the desired `ArgumentType` for a given ABI +/// argument number in `0..abi_args`. +/// +fn legalize_inst_arguments(dfg: &mut DataFlowGraph, + pos: &mut Cursor, + abi_args: usize, + mut get_abi_type: ArgType) + where ArgType: FnMut(&DataFlowGraph, usize) -> ArgumentType +{ + let inst = pos.current_inst().expect("Cursor must point to a call instruction"); + + // Lift the value list out of the call instruction so we modify it. + let mut vlist = dfg[inst].take_value_list().expect("Call must have a value list"); + + // The value list contains all arguments to the instruction, including the callee on an + // indirect call which isn't part of the call arguments that must match the ABI signature. + // Figure out how many fixed values are at the front of the list. We won't touch those. + let fixed_values = dfg[inst].opcode().constraints().fixed_value_arguments(); + let have_args = vlist.len(&dfg.value_lists) - fixed_values; + + // Grow the value list to the right size and shift all the existing arguments to the right. + // This lets us write the new argument values into the list without overwriting the old + // arguments. + // + // Before: + // + // <--> fixed_values + // <-----------> have_args + // [FFFFOOOOOOOOOOOOO] + // + // After grow_at(): + // + // <--> fixed_values + // <-----------> have_args + // <------------------> abi_args + // [FFFF-------OOOOOOOOOOOOO] + // ^ + // old_arg_offset + // + // After writing the new arguments: + // + // <--> fixed_values + // <------------------> abi_args + // [FFFFNNNNNNNNNNNNNNNNNNNN] + // + vlist.grow_at(fixed_values, abi_args - have_args, &mut dfg.value_lists); + let old_arg_offset = fixed_values + abi_args - have_args; + + let mut abi_arg = 0; + for old_arg in 0..have_args { + let old_value = vlist.get(old_arg_offset + old_arg, &dfg.value_lists).unwrap(); + convert_to_abi(dfg, + pos, + old_value, + &mut |dfg, arg| { + let abi_type = get_abi_type(dfg, abi_arg); + if dfg.value_type(arg) == abi_type.value_type { + // This is the argument type we need. + vlist.as_mut_slice(&mut dfg.value_lists)[fixed_values + abi_arg] = arg; + abi_arg += 1; + None + } else { + // Nope, `arg` needs to be converted. + Some(abi_type) + } + }); + } + + // Put the modified value list back. + dfg[inst].put_value_list(vlist); +} + +/// Insert ABI conversion code before and after the call instruction at `pos`. +/// +/// Instructions inserted before the call will compute the appropriate ABI values for the +/// callee's new ABI-legalized signature. The function call arguments are rewritten in place to +/// match the new signature. +/// +/// Instructions will be inserted after the call to convert returned ABI values back to the +/// original return values. The call's result values will be adapted to match the new signature. +/// +/// Returns `true` if any instructions were inserted. +fn handle_call_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor) -> bool { + let inst = pos.current_inst().expect("Cursor must point to a call instruction"); + + // Start by checking if the argument types already match the signature. + let sig_ref = match check_call_signature(dfg, inst) { + None => return false, + Some(s) => s, + }; + + // OK, we need to fix the call arguments to match the ABI signature. + let abi_args = dfg.signatures[sig_ref].argument_types.len(); + legalize_inst_arguments(dfg, + pos, + abi_args, + |dfg, abi_arg| dfg.signatures[sig_ref].argument_types[abi_arg]); + + // TODO: Convert return values. + + // Yes, we changed stuff. + true +} + +/// Insert ABI conversion code before and after the call instruction at `pos`. +/// +/// Return `true` if any instructions were inserted. +fn handle_return_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor, sig: &Signature) -> bool { + let inst = pos.current_inst().expect("Cursor must point to a return instruction"); + + // Check if the returned types already match the signature. + let fixed_values = dfg[inst].opcode().constraints().fixed_value_arguments(); + if check_arg_types(dfg, + dfg[inst] + .arguments(&dfg.value_lists) + .iter() + .skip(fixed_values) + .cloned(), + &sig.return_types[..]) { + return false; + } + + let abi_args = sig.return_types.len(); + legalize_inst_arguments(dfg, pos, abi_args, |_, abi_arg| sig.return_types[abi_arg]); + + // Yes, we changed stuff. + true +} From db5aead1a5dca6ca6e87a9b6caab5622148abd03 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 15 Mar 2017 15:21:57 -0700 Subject: [PATCH 0599/3084] Rename take_ebb_args to detach_ebb_args() This better matches the detach_secondary_results cousin. Also rename the converse put_ebb_arg -> attach_ebb_arg. --- lib/cretonne/src/ir/dfg.rs | 18 +++++++++--------- lib/cretonne/src/legalizer.rs | 4 ++-- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 699f2a2f1a..1b1ff418e8 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -549,7 +549,7 @@ impl DataFlowGraph { num: 0, next: None.into(), }); - self.put_ebb_arg(ebb, val); + self.attach_ebb_arg(ebb, val); val } @@ -578,9 +578,9 @@ impl DataFlowGraph { /// processing the list of arguments. /// /// This is a quite low-level operation. Sensible things to do with the detached EBB arguments - /// is to put them back on the same EBB with `put_ebb_arg()` or change them into aliases + /// is to put them back on the same EBB with `attach_ebb_arg()` or change them into aliases /// with `change_to_alias()`. - pub fn take_ebb_args(&mut self, ebb: Ebb) -> Option { + pub fn detach_ebb_args(&mut self, ebb: Ebb) -> Option { let first = self.ebbs[ebb].first_arg.into(); self.ebbs[ebb].first_arg = None.into(); self.ebbs[ebb].last_arg = None.into(); @@ -591,10 +591,10 @@ impl DataFlowGraph { /// /// The appended value should already be an EBB argument belonging to `ebb`, but it can't be /// attached. In practice, this means that it should be one of the values returned from - /// `take_ebb_args()`. + /// `detach_ebb_args()`. /// /// In almost all cases, you should be using `append_ebb_arg()` instead of this method. - pub fn put_ebb_arg(&mut self, ebb: Ebb, arg: Value) { + pub fn attach_ebb_arg(&mut self, ebb: Ebb, arg: Value) { let arg_num = match self.ebbs[ebb].last_arg.map(|v| v.expand()) { // If last_argument is `None`, we're adding the first EBB argument. None => { @@ -743,7 +743,7 @@ mod tests { assert_eq!(ebb.to_string(), "ebb0"); assert_eq!(dfg.num_ebb_args(ebb), 0); assert_eq!(dfg.ebb_args(ebb).next(), None); - assert_eq!(dfg.take_ebb_args(ebb), None); + assert_eq!(dfg.detach_ebb_args(ebb), None); assert_eq!(dfg.num_ebb_args(ebb), 0); assert_eq!(dfg.ebb_args(ebb).next(), None); @@ -771,16 +771,16 @@ mod tests { assert_eq!(dfg.value_type(arg2), types::I16); // Swap the two EBB arguments. - let take1 = dfg.take_ebb_args(ebb).unwrap(); + let take1 = dfg.detach_ebb_args(ebb).unwrap(); assert_eq!(dfg.num_ebb_args(ebb), 0); assert_eq!(dfg.ebb_args(ebb).next(), None); let take2 = dfg.next_ebb_arg(take1).unwrap(); assert_eq!(take1, arg1); assert_eq!(take2, arg2); assert_eq!(dfg.next_ebb_arg(take2), None); - dfg.put_ebb_arg(ebb, take2); + dfg.attach_ebb_arg(ebb, take2); let arg3 = dfg.append_ebb_arg(ebb, types::I32); - dfg.put_ebb_arg(ebb, take1); + dfg.attach_ebb_arg(ebb, take1); assert_eq!(dfg.ebb_args(ebb).collect::>(), [take2, arg3, take1]); } diff --git a/lib/cretonne/src/legalizer.rs b/lib/cretonne/src/legalizer.rs index e22cb6c95e..d6894321fe 100644 --- a/lib/cretonne/src/legalizer.rs +++ b/lib/cretonne/src/legalizer.rs @@ -133,7 +133,7 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { // Process the EBB arguments one at a time, possibly replacing one argument with multiple new // ones. We do this by detaching the entry EBB arguments first. - let mut next_arg = func.dfg.take_ebb_args(entry); + let mut next_arg = func.dfg.detach_ebb_args(entry); while let Some(arg) = next_arg { // Get the next argument before we mutate `arg`. next_arg = func.dfg.next_ebb_arg(arg); @@ -142,7 +142,7 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { if arg_type == abi_types[abi_arg].value_type { // No value translation is necessary, this argument matches the ABI type. // Just use the original EBB argument value. This is the most common case. - func.dfg.put_ebb_arg(entry, arg); + func.dfg.attach_ebb_arg(entry, arg); abi_arg += 1; } else { // Compute the value we want for `arg` from the legalized ABI arguments. From 2119c85224d65764404409a0ce166949ce1f7ee7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 15 Mar 2017 15:38:47 -0700 Subject: [PATCH 0600/3084] Don't return a Values iterator from detach_secondary_results(). Instead, just return the first of the detached values, and provide a next_secondary_result() method for traversing the list. This is equivalent to how detach_ebb_args() works, and it allows the data flow graph to be modified while traversing the list of results. --- lib/cretonne/meta/gen_legalizer.py | 15 +++++++++++---- lib/cretonne/src/ir/dfg.rs | 31 +++++++++++++++++++----------- 2 files changed, 31 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 85aa715996..95d16d3663 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -90,10 +90,17 @@ def unwrap_inst(iref, node, fmt): for d in node.defs[1:]: fmt.line('let src_{};'.format(d)) with fmt.indented('{', '}'): - fmt.line('let mut vals = dfg.detach_secondary_results(inst);') - for d in node.defs[1:]: - fmt.line('src_{} = vals.next().unwrap();'.format(d)) - fmt.line('assert_eq!(vals.next(), None);') + fmt.line( + 'src_{} = dfg.detach_secondary_results(inst).unwrap();' + .format(node.defs[1])) + for i in range(2, len(node.defs)): + fmt.line( + 'src_{} = dfg.next_secondary_result(src_{})' + '.unwrap();' + .format(node.defs[i], node.defs[i - 1])) + fmt.line( + 'assert_eq!(dfg.next_secondary_result(src_{}), None);' + .format(node.defs[-1])) for d in node.defs[1:]: if d.has_free_typevar(): fmt.line( diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 1b1ff418e8..5a70c92608 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -430,19 +430,29 @@ impl DataFlowGraph { ReplaceBuilder::new(self, inst) } - /// Detach secondary instruction results, and return them as an iterator. + /// Detach secondary instruction results, and return the first of them. /// - /// If `inst` produces two or more results, detach these secondary result values from `inst`, - /// and return an iterator that will enumerate them. The first result value cannot be detached. + /// If `inst` produces two or more results, detach these secondary result values from `inst`. + /// The first result value cannot be detached. + /// The full list of secondary results can be traversed with `next_secondary_result()`. /// /// Use this method to detach secondary values before using `replace(inst)` to provide an /// alternate instruction for computing the primary result value. - pub fn detach_secondary_results(&mut self, inst: Inst) -> Values { - let second_result = self[inst].second_result_mut().and_then(|r| r.take()); - Values { - dfg: self, - cur: second_result, + pub fn detach_secondary_results(&mut self, inst: Inst) -> Option { + self[inst].second_result_mut().and_then(|r| r.take()) + } + + /// Get the next secondary result after `value`. + /// + /// Use this function to traverse the full list of instruction results returned from + /// `detach_secondary_results()`. + pub fn next_secondary_result(&self, value: Value) -> Option { + if let ExpandedValue::Table(index) = value.expand() { + if let ValueData::Inst { next, .. } = self.extended_values[index] { + return next.into(); + } } + panic!("{} is not a secondary result value", value); } /// Get the first result of an instruction. @@ -811,9 +821,8 @@ mod tests { // Detach the 'c' value from `iadd`. { - let mut vals = dfg.detach_secondary_results(iadd); - assert_eq!(vals.next(), Some(c)); - assert_eq!(vals.next(), None); + assert_eq!(dfg.detach_secondary_results(iadd), Some(c)); + assert_eq!(dfg.next_secondary_result(c), None); } // Replace `iadd_cout` with a normal `iadd` and an `icmp`. From e90e59cedb1a7f6f3ee182c877a92535586db891 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 15 Mar 2017 17:04:58 -0700 Subject: [PATCH 0601/3084] Add DataFlowGraph::redefine_first_value() This makes it possible to compute the first result of an instruction in a different way without overwriting the original instruction with replace(). --- lib/cretonne/src/ir/dfg.rs | 49 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 47 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 5a70c92608..a1eb864af2 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -5,7 +5,7 @@ use ir::entities::ExpandedValue; use ir::instructions::{Opcode, InstructionData, CallInfo}; use ir::extfunc::ExtFuncData; use entity_map::{EntityMap, PrimaryEntityData}; -use ir::builder::{InsertBuilder, ReplaceBuilder}; +use ir::builder::{InsertBuilder, ReplaceBuilder, InstBuilder}; use ir::layout::Cursor; use packed_option::PackedOption; use write::write_operands; @@ -455,6 +455,41 @@ impl DataFlowGraph { panic!("{} is not a secondary result value", value); } + /// Move the instruction at `pos` to a new `Inst` reference so its first result can be + /// redefined without overwriting the original instruction. + /// + /// The first result value of an instruction is intrinsically tied to the `Inst` reference, so + /// it is not possible to detach the value and attach it to something else. This function + /// copies the instruction pointed to by `pos` to a new `Inst` reference, making the original + /// `Inst` reference available to be redefined with `dfg.replace(inst)` above. + /// + /// Before: + /// + /// inst1: v1, vx2 = foo <-- pos + /// + /// After: + /// + /// inst7: v7, vx2 = foo + /// inst1: v1 = copy v7 <-- pos + /// + /// Returns the new `Inst` reference where the original instruction has been moved. + pub fn redefine_first_value(&mut self, pos: &mut Cursor) -> Inst { + let orig = pos.current_inst().expect("Cursor must point at an instruction"); + let data = self[orig].clone(); + // After cloning, any secondary values are attached to both copies. Don't do that, we only + // want them on the new clone. + self.detach_secondary_results(orig); + let new = self.make_inst(data); + pos.insert_inst(new); + // Replace the original instruction with a copy of the new value. + // This is likely to be immediately overwritten by something else, but this way we avoid + // leaving the DFG in a state with multiple references to secondary results and value + // lists. It also means that this method doesn't change the semantics of the program. + let new_value = self.first_result(new); + self.replace(orig).copy(new_value); + new + } + /// Get the first result of an instruction. /// /// If `Inst` doesn't produce any results, this returns a `Value` with a `VOID` type. @@ -819,9 +854,19 @@ mod tests { _ => panic!(), }; + // Redefine the first value out of `iadd_cout`. + assert_eq!(pos.prev_inst(), Some(iadd)); + let new_iadd = dfg.redefine_first_value(pos); + let new_s = dfg.first_result(new_iadd); + assert_eq!(dfg[iadd].opcode(), Opcode::Copy); + assert_eq!(dfg.inst_results(iadd).collect::>(), [s]); + assert_eq!(dfg.inst_results(new_iadd).collect::>(), [new_s, c]); + assert_eq!(dfg.resolve_copies(s), new_s); + pos.next_inst(); + // Detach the 'c' value from `iadd`. { - assert_eq!(dfg.detach_secondary_results(iadd), Some(c)); + assert_eq!(dfg.detach_secondary_results(new_iadd), Some(c)); assert_eq!(dfg.next_secondary_result(c), None); } From 496450278259d68ceef5459071a6df378aff0742 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 17 Mar 2017 10:05:56 -0700 Subject: [PATCH 0602/3084] Break out differently purposed tests from abi.cton. - abi.cton is for testing the actual RISC-V ABI. - legalize-abi.cton is for testing the legalizer around ABI boundaries. - parse-encoding.cton is for testing the parser's handling of RISC-V encoding and register annotations. --- cranelift/filetests/isa/riscv/abi.cton | 75 ------------------- .../filetests/isa/riscv/legalize-abi.cton | 47 ++++++++++++ .../filetests/isa/riscv/parse-encoding.cton | 36 +++++++++ 3 files changed, 83 insertions(+), 75 deletions(-) create mode 100644 cranelift/filetests/isa/riscv/legalize-abi.cton create mode 100644 cranelift/filetests/isa/riscv/parse-encoding.cton diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.cton index 50af69638f..d75813e220 100644 --- a/cranelift/filetests/isa/riscv/abi.cton +++ b/cranelift/filetests/isa/riscv/abi.cton @@ -31,78 +31,3 @@ function f(i32) { ebb0(v0: i32): return_reg v0 } - -function int_split_args(i64) -> i64 { -ebb0(v0: i64): - ; check: $ebb0($(v0l=$VX): i32, $(v0h=$VX): i32): - ; check: iconcat_lohi $v0l, $v0h - v1 = iadd_imm v0, 1 - ; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi $v1 - ; check: return $v1l, $v1h - return v1 -} - -function int_ext(i8, i8 sext, i8 uext) -> i8 uext { -ebb0(v1: i8, v2: i8, v3: i8): - ; check: $ebb0($v1: i8, $(v2x=$VX): i32, $(v3x=$VX): i32): - ; check: ireduce.i8 $v2x - ; check: ireduce.i8 $v3x - ; check: $(v1x=$V) = uextend.i32 $v1 - ; check: return $v1x - return v1 -} - -function vector_split_args(i64x4) -> i64x4 { -ebb0(v0: i64x4): - ; check: $ebb0($(v0al=$VX): i32, $(v0ah=$VX): i32, $(v0bl=$VX): i32, $(v0bh=$VX): i32, $(v0cl=$VX): i32, $(v0ch=$VX): i32, $(v0dl=$VX): i32, $(v0dh=$VX): i32): - ; check: $(v0a=$V) = iconcat_lohi $v0al, $v0ah - ; check: $(v0b=$V) = iconcat_lohi $v0bl, $v0bh - ; check: $(v0ab=$V) = vconcat $v0a, $v0b - ; check: $(v0c=$V) = iconcat_lohi $v0cl, $v0ch - ; check: $(v0d=$V) = iconcat_lohi $v0dl, $v0dh - ; check: $(v0cd=$V) = vconcat $v0c, $v0d - ; check: $(v0abcd=$V) = vconcat $v0ab, $v0cd - v1 = iadd v0, v0 - ; check: $(v1ab=$V), $(v1cd=$VX) = vsplit - ; check: $(v1a=$V), $(v1b=$VX) = vsplit $v1ab - ; check: $(v1al=$V), $(v1ah=$VX) = isplit_lohi $v1a - ; check: $(v1bl=$V), $(v1bh=$VX) = isplit_lohi $v1b - ; check: $(v1c=$V), $(v1d=$VX) = vsplit $v1cd - ; check: $(v1cl=$V), $(v1ch=$VX) = isplit_lohi $v1c - ; check: $(v1dl=$V), $(v1dh=$VX) = isplit_lohi $v1d - ; check: return $v1al, $v1ah, $v1bl, $v1bh, $v1cl, $v1ch, $v1dl, $v1dh - return v1 -} - -function parse_encoding(i32 [%x5]) -> i32 [%x10] { - ; check: function parse_encoding(i32 [%x5]) -> i32 [%x10] { - - sig0 = signature(i32 [%x10]) -> i32 [%x10] - ; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] - - sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] - ; check: sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] - - sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] - ; check: sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] - - ; Arguments on stack where not necessary - sig3 = signature(f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] - ; check: sig3 = signature(f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] - - ; Stack argument before register argument - sig4 = signature(f32 [72], i32 [%x10]) - ; check: sig4 = signature(f32 [72], i32 [%x10]) - - ; Return value on stack - sig5 = signature() -> f32 [0] - ; check: sig5 = signature() -> f32 [0] - - ; function + signature - fn15 = function bar(i32 [%x10]) -> b1 [%x10] - ; check: sig6 = signature(i32 [%x10]) -> b1 [%x10] - ; nextln: fn0 = sig6 bar - -ebb0(v0: i32): - return v0 -} diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton new file mode 100644 index 0000000000..bf93596128 --- /dev/null +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -0,0 +1,47 @@ +; Test legalizer's handling of ABI boundaries. +test legalizer +isa riscv + +; regex: V=vx?\d+ + +function int_split_args(i64) -> i64 { +ebb0(v0: i64): + ; check: $ebb0($(v0l=$V): i32, $(v0h=$V): i32): + ; check: iconcat_lohi $v0l, $v0h + v1 = iadd_imm v0, 1 + ; check: $(v1l=$V), $(v1h=$V) = isplit_lohi $v1 + ; check: return $v1l, $v1h + return v1 +} + +function int_ext(i8, i8 sext, i8 uext) -> i8 uext { +ebb0(v1: i8, v2: i8, v3: i8): + ; check: $ebb0($v1: i8, $(v2x=$V): i32, $(v3x=$V): i32): + ; check: ireduce.i8 $v2x + ; check: ireduce.i8 $v3x + ; check: $(v1x=$V) = uextend.i32 $v1 + ; check: return $v1x + return v1 +} + +function vector_split_args(i64x4) -> i64x4 { +ebb0(v0: i64x4): + ; check: $ebb0($(v0al=$V): i32, $(v0ah=$V): i32, $(v0bl=$V): i32, $(v0bh=$V): i32, $(v0cl=$V): i32, $(v0ch=$V): i32, $(v0dl=$V): i32, $(v0dh=$V): i32): + ; check: $(v0a=$V) = iconcat_lohi $v0al, $v0ah + ; check: $(v0b=$V) = iconcat_lohi $v0bl, $v0bh + ; check: $(v0ab=$V) = vconcat $v0a, $v0b + ; check: $(v0c=$V) = iconcat_lohi $v0cl, $v0ch + ; check: $(v0d=$V) = iconcat_lohi $v0dl, $v0dh + ; check: $(v0cd=$V) = vconcat $v0c, $v0d + ; check: $(v0abcd=$V) = vconcat $v0ab, $v0cd + v1 = iadd v0, v0 + ; check: $(v1ab=$V), $(v1cd=$V) = vsplit + ; check: $(v1a=$V), $(v1b=$V) = vsplit $v1ab + ; check: $(v1al=$V), $(v1ah=$V) = isplit_lohi $v1a + ; check: $(v1bl=$V), $(v1bh=$V) = isplit_lohi $v1b + ; check: $(v1c=$V), $(v1d=$V) = vsplit $v1cd + ; check: $(v1cl=$V), $(v1ch=$V) = isplit_lohi $v1c + ; check: $(v1dl=$V), $(v1dh=$V) = isplit_lohi $v1d + ; check: return $v1al, $v1ah, $v1bl, $v1bh, $v1cl, $v1ch, $v1dl, $v1dh + return v1 +} diff --git a/cranelift/filetests/isa/riscv/parse-encoding.cton b/cranelift/filetests/isa/riscv/parse-encoding.cton new file mode 100644 index 0000000000..d3cc6eee4b --- /dev/null +++ b/cranelift/filetests/isa/riscv/parse-encoding.cton @@ -0,0 +1,36 @@ +; Test the parser's support for encoding annotations. +test legalizer +isa riscv + +function parse_encoding(i32 [%x5]) -> i32 [%x10] { + ; check: function parse_encoding(i32 [%x5]) -> i32 [%x10] { + + sig0 = signature(i32 [%x10]) -> i32 [%x10] + ; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] + + sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] + ; check: sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] + + sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] + ; check: sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] + + ; Arguments on stack where not necessary + sig3 = signature(f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] + ; check: sig3 = signature(f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] + + ; Stack argument before register argument + sig4 = signature(f32 [72], i32 [%x10]) + ; check: sig4 = signature(f32 [72], i32 [%x10]) + + ; Return value on stack + sig5 = signature() -> f32 [0] + ; check: sig5 = signature() -> f32 [0] + + ; function + signature + fn15 = function bar(i32 [%x10]) -> b1 [%x10] + ; check: sig6 = signature(i32 [%x10]) -> b1 [%x10] + ; nextln: fn0 = sig6 bar + +ebb0(v0: i32): + return v0 +} From 6dc11d12670941265ee66dff186c382fee7c974d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 16 Mar 2017 14:42:41 -0700 Subject: [PATCH 0603/3084] Use a closure to control the convert_from_abi() function. This will be used for converting function return types soon, so generalize it a bit. --- lib/cretonne/src/legalizer.rs | 87 +++++++++++++++++++---------------- 1 file changed, 47 insertions(+), 40 deletions(-) diff --git a/lib/cretonne/src/legalizer.rs b/lib/cretonne/src/legalizer.rs index d6894321fe..6814e3fe2a 100644 --- a/lib/cretonne/src/legalizer.rs +++ b/lib/cretonne/src/legalizer.rs @@ -146,12 +146,16 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { abi_arg += 1; } else { // Compute the value we want for `arg` from the legalized ABI arguments. - let converted = convert_from_abi(&mut func.dfg, - &mut pos, - entry, - &mut abi_arg, - abi_types, - arg_type); + let mut get_arg = |dfg: &mut DataFlowGraph, ty| { + let abi_type = abi_types[abi_arg]; + if ty == abi_type.value_type { + abi_arg += 1; + Ok(dfg.append_ebb_arg(entry, ty)) + } else { + Err(abi_type) + } + }; + let converted = convert_from_abi(&mut func.dfg, &mut pos, arg_type, &mut get_arg); // The old `arg` is no longer an attached EBB argument, but there are probably still // uses of the value. Make it an alias to the converted value. func.dfg.change_to_alias(arg, converted); @@ -159,57 +163,63 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { } } -/// Compute original value of type `ty` from the legalized ABI arguments beginning at `abi_arg`. +/// Compute original value of type `ty` from the legalized ABI arguments. /// -/// Update `abi_arg` to reflect the ABI arguments consumed and return the computed value. -fn convert_from_abi(dfg: &mut DataFlowGraph, - pos: &mut Cursor, - entry: Ebb, - abi_arg: &mut usize, - abi_types: &[ArgumentType], - ty: Type) - -> Value { +/// The conversion is recursive, controlled by the `get_arg` closure which is called to retrieve an +/// ABI argument. It returns: +/// +/// - `Ok(arg)` if the requested type matches the next ABI argument. +/// - `Err(arg_type)` if further conversions are needed from the ABI argument `arg_type`. +/// +fn convert_from_abi(dfg: &mut DataFlowGraph, + pos: &mut Cursor, + ty: Type, + get_arg: &mut GetArg) + -> Value + where GetArg: FnMut(&mut DataFlowGraph, Type) -> Result +{ // Terminate the recursion when we get the desired type. - if ty == abi_types[*abi_arg].value_type { - return dfg.append_ebb_arg(entry, ty); - } + let arg_type = match get_arg(dfg, ty) { + Ok(v) => return v, + Err(t) => t, + }; - // Reconstruct how `ty` was legalized into the argument at `abi_arg`. - let conversion = legalize_abi_value(ty, &abi_types[*abi_arg]); + // Reconstruct how `ty` was legalized into the `arg_type` argument. + let conversion = legalize_abi_value(ty, &arg_type); // The conversion describes value to ABI argument. We implement the reverse conversion here. match conversion { // Construct a `ty` by concatenating two ABI integers. ValueConversion::IntSplit => { let abi_ty = ty.half_width().expect("Invalid type for conversion"); - let lo = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); - let hi = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + let lo = convert_from_abi(dfg, pos, abi_ty, get_arg); + let hi = convert_from_abi(dfg, pos, abi_ty, get_arg); dfg.ins(pos).iconcat_lohi(lo, hi) } // Construct a `ty` by concatenating two halves of a vector. ValueConversion::VectorSplit => { let abi_ty = ty.half_vector().expect("Invalid type for conversion"); - let lo = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); - let hi = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + let lo = convert_from_abi(dfg, pos, abi_ty, get_arg); + let hi = convert_from_abi(dfg, pos, abi_ty, get_arg); dfg.ins(pos).vconcat(lo, hi) } // Construct a `ty` by bit-casting from an integer type. ValueConversion::IntBits => { assert!(!ty.is_int()); let abi_ty = Type::int(ty.bits()).expect("Invalid type for conversion"); - let arg = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + let arg = convert_from_abi(dfg, pos, abi_ty, get_arg); dfg.ins(pos).bitcast(ty, arg) } // ABI argument is a sign-extended version of the value we want. ValueConversion::Sext(abi_ty) => { - let arg = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + let arg = convert_from_abi(dfg, pos, abi_ty, get_arg); // TODO: Currently, we don't take advantage of the ABI argument being sign-extended. // We could insert an `assert_sreduce` which would fold with a following `sextend` of // this value. dfg.ins(pos).ireduce(ty, arg) } ValueConversion::Uext(abi_ty) => { - let arg = convert_from_abi(dfg, pos, entry, abi_arg, abi_types, abi_ty); + let arg = convert_from_abi(dfg, pos, abi_ty, get_arg); // TODO: Currently, we don't take advantage of the ABI argument being sign-extended. // We could insert an `assert_ureduce` which would fold with a following `uextend` of // this value. @@ -225,21 +235,21 @@ fn convert_from_abi(dfg: &mut DataFlowGraph, /// to the closure, it will perform one of two actions: /// /// 1. If the suggested argument has an acceptable value type, consume it by adding it to the list -/// of arguments and return `None`. +/// of arguments and return `Ok(())`. /// 2. If the suggested argument doesn't have the right value type, don't change anything, but -/// return the `ArgumentType` that is needed. +/// return the `Err(ArgumentType)` that is needed. /// fn convert_to_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor, value: Value, put_arg: &mut PutArg) - where PutArg: FnMut(&mut DataFlowGraph, Value) -> Option + where PutArg: FnMut(&mut DataFlowGraph, Value) -> Result<(), ArgumentType> { // Start by invoking the closure to either terminate the recursion or get the argument type // we're trying to match. let arg_type = match put_arg(dfg, value) { - None => return, - Some(t) => t, + Ok(_) => return, + Err(t) => t, }; let ty = dfg.value_type(value); @@ -369,21 +379,18 @@ fn legalize_inst_arguments(dfg: &mut DataFlowGraph, let mut abi_arg = 0; for old_arg in 0..have_args { let old_value = vlist.get(old_arg_offset + old_arg, &dfg.value_lists).unwrap(); - convert_to_abi(dfg, - pos, - old_value, - &mut |dfg, arg| { + let mut put_arg = |dfg: &mut DataFlowGraph, arg| { let abi_type = get_abi_type(dfg, abi_arg); if dfg.value_type(arg) == abi_type.value_type { // This is the argument type we need. vlist.as_mut_slice(&mut dfg.value_lists)[fixed_values + abi_arg] = arg; abi_arg += 1; - None + Ok(()) } else { - // Nope, `arg` needs to be converted. - Some(abi_type) + Err(abi_type) } - }); + }; + convert_to_abi(dfg, pos, old_value, &mut put_arg); } // Put the modified value list back. From 56d12b14ff05b4be0094d095b765f037aeedd633 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 16 Mar 2017 16:54:26 -0700 Subject: [PATCH 0604/3084] Add attach_secondary_result and append_secondary_result. These low-level functions allow us to build up a list of instruction results incrementally. They are equivalent to the existing attach_ebb_arg and append_ebb_arg. --- lib/cretonne/src/ir/dfg.rs | 57 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index a1eb864af2..8977cc6751 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -455,6 +455,63 @@ impl DataFlowGraph { panic!("{} is not a secondary result value", value); } + /// Attach an existing value as a secondary result after `last_res` which must be the last + /// result of an instruction. + /// + /// This is a very low-level operation. Usually, instruction results with the correct types are + /// created automatically. The `res` value must be a secondary instruction result detached from + /// somewhere else. + pub fn attach_secondary_result(&mut self, last_res: Value, res: Value) { + let (res_inst, res_num) = match last_res.expand() { + ExpandedValue::Direct(inst) => { + // We're adding the second value to `inst`. + let next = self[inst].second_result_mut().expect("bad inst format"); + assert!(next.is_none(), "last_res is not the last result"); + *next = res.into(); + (inst, 1) + } + ExpandedValue::Table(idx) => { + if let ValueData::Inst { num, inst, ref mut next, .. } = self.extended_values[idx] { + assert!(next.is_none(), "last_res is not the last result"); + *next = res.into(); + assert!(num < u16::MAX, "Too many arguments to EBB"); + (inst, num + 1) + } else { + panic!("last_res is not an instruction result"); + } + } + }; + + // Now update `res` itself. + if let ExpandedValue::Table(idx) = res.expand() { + if let ValueData::Inst { ref mut num, ref mut inst, ref mut next, .. } = + self.extended_values[idx] { + *num = res_num; + *inst = res_inst; + *next = None.into(); + return; + } + } + panic!("{} must be a result", res); + } + + /// Append a new instruction result value after `last_res`. + /// + /// The `last_res` value must be the last value on an instruction. + pub fn append_secondary_result(&mut self, last_res: Value, ty: Type) -> Value { + // The only member that matters is `ty`. The rest is filled in by + // `attach_secondary_result`. + use entity_map::EntityRef; + let res = self.make_value(ValueData::Inst { + ty: ty, + inst: Inst::new(0), + num: 0, + next: None.into(), + }); + self.attach_secondary_result(last_res, res); + res + } + /// Move the instruction at `pos` to a new `Inst` reference so its first result can be /// redefined without overwriting the original instruction. /// From d881ed331b57e3e39b0dde959ac5984fbd4a7f07 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 16 Mar 2017 17:50:49 -0700 Subject: [PATCH 0605/3084] Legalize return values from call instructions. Like the entry block arguments, the return values from a call instruction need to be converted back from their ABI representation. Add tests of call instruction legalization. --- .../filetests/isa/riscv/legalize-abi.cton | 60 +++++++++++++ lib/cretonne/src/legalizer.rs | 89 ++++++++++++++++++- 2 files changed, 148 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index bf93596128..eada8ebda9 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -14,6 +14,50 @@ ebb0(v0: i64): return v1 } +function split_call_arg(i32) { + fn1 = function foo(i64) + fn2 = function foo(i32, i64) +ebb0(v0: i32): + v1 = uextend.i64 v0 + call fn1(v1) + ; check: $(v1l=$V), $(v1h=$V) = isplit_lohi $v1 + ; check: call $fn1($v1l, $v1h) + call fn2(v0, v1) + ; check: call $fn2($v0, $V, $V) + return +} + +function split_ret_val() { + fn1 = function foo() -> i64 +ebb0: + v1 = call fn1() + ; check: $ebb0: + ; nextln: $(v1l=$V), $(v1h=$V) = call $fn1() + ; check: $(v1new=$V) = iconcat_lohi $v1l, $v1h + ; check: $v1 = copy $v1new + jump ebb1(v1) + ; check: jump $ebb1($v1) + +ebb1(v10: i64): + jump ebb1(v10) +} + +; First return value is fine, second one is expanded. +function split_ret_val2() { + fn1 = function foo() -> i32, i64 +ebb0: + v1, v2 = call fn1() + ; check: $ebb0: + ; nextln: $v1, $(v2l=$V), $(v2h=$V) = call $fn1() + ; check: $(v2new=$V) = iconcat_lohi $v2l, $v2h + ; check: $v2 -> $v2new + jump ebb1(v1, v2) + ; check: jump $ebb1($v1, $v2) + +ebb1(v9: i32, v10: i64): + jump ebb1(v9, v10) +} + function int_ext(i8, i8 sext, i8 uext) -> i8 uext { ebb0(v1: i8, v2: i8, v3: i8): ; check: $ebb0($v1: i8, $(v2x=$V): i32, $(v3x=$V): i32): @@ -24,6 +68,22 @@ ebb0(v1: i8, v2: i8, v3: i8): return v1 } +; Function produces single return value, still need to copy. +function ext_ret_val() { + fn1 = function foo() -> i8 sext +ebb0: + v1 = call fn1() + ; check: $ebb0: + ; nextln: $(rv=$V) = call $fn1() + ; check: $(v1new=$V) = ireduce.i8 $rv + ; check: $v1 = copy $v1new + jump ebb1(v1) + ; check: jump $ebb1($v1) + +ebb1(v10: i8): + jump ebb1(v10) +} + function vector_split_args(i64x4) -> i64x4 { ebb0(v0: i64x4): ; check: $ebb0($(v0al=$V): i32, $(v0ah=$V): i32, $(v0bl=$V): i32, $(v0bh=$V): i32, $(v0cl=$V): i32, $(v0ch=$V): i32, $(v0dl=$V): i32, $(v0dh=$V): i32): diff --git a/lib/cretonne/src/legalizer.rs b/lib/cretonne/src/legalizer.rs index 6814e3fe2a..49651491fd 100644 --- a/lib/cretonne/src/legalizer.rs +++ b/lib/cretonne/src/legalizer.rs @@ -163,6 +163,89 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { } } +/// Legalize the results returned from a call instruction to match the ABI signature. +/// +/// The cursor `pos` points to a call instruction with at least one return value. The cursor will +/// be left pointing after the instructions inserted to convert the return values. +/// +/// This function is very similar to the `legalize_entry_arguments` function above. +fn legalize_inst_results(dfg: &mut DataFlowGraph, + pos: &mut Cursor, + mut get_abi_type: ResType) + where ResType: FnMut(&DataFlowGraph, usize) -> ArgumentType +{ + let mut call = pos.current_inst().expect("Cursor must point to a call instruction"); + + // We theoretically allow for call instructions that return a number of fixed results before + // the call return values. In practice, it doesn't happen. + let fixed_results = dfg[call].opcode().constraints().fixed_results(); + assert_eq!(fixed_results, 0, "Fixed results on calls not supported"); + + let mut next_res = dfg.detach_secondary_results(call); + // The currently last result on the call instruction. + let mut last_res = dfg.first_result(call); + let mut abi_res = 0; + + // The first result requires special handling. + let first_ty = dfg.value_type(last_res); + if first_ty != get_abi_type(dfg, abi_res).value_type { + // Move the call out of the way, so we can redefine the first result. + let copy = call; + call = dfg.redefine_first_value(pos); + last_res = dfg.first_result(call); + // Set up a closure that can attach new results to `call`. + let mut get_res = |dfg: &mut DataFlowGraph, ty| { + let abi_type = get_abi_type(dfg, abi_res); + if ty == abi_type.value_type { + // Don't append the first result - it's not detachable. + if fixed_results + abi_res == 0 { + *dfg[call].first_type_mut() = ty; + debug_assert_eq!(last_res, dfg.first_result(call)); + } else { + last_res = dfg.append_secondary_result(last_res, ty); + } + abi_res += 1; + Ok(last_res) + } else { + Err(abi_type) + } + }; + + let v = convert_from_abi(dfg, pos, first_ty, &mut get_res); + dfg.replace(copy).copy(v); + } + + // Point immediately after the call and any instructions dealing with the first result. + pos.next_inst(); + + // Now do the secondary results. + while let Some(res) = next_res { + next_res = dfg.next_secondary_result(res); + + let res_type = dfg.value_type(res); + if res_type == get_abi_type(dfg, abi_res).value_type { + // No value translation is necessary, this result matches the ABI type. + dfg.attach_secondary_result(last_res, res); + last_res = res; + abi_res += 1; + } else { + let mut get_res = |dfg: &mut DataFlowGraph, ty| { + let abi_type = get_abi_type(dfg, abi_res); + if ty == abi_type.value_type { + last_res = dfg.append_secondary_result(last_res, ty); + abi_res += 1; + Ok(last_res) + } else { + Err(abi_type) + } + }; + let v = convert_from_abi(dfg, pos, res_type, &mut get_res); + // The old `res` is no longer an attached result. + dfg.change_to_alias(res, v); + } + } +} + /// Compute original value of type `ty` from the legalized ABI arguments. /// /// The conversion is recursive, controlled by the `get_arg` closure which is called to retrieve an @@ -423,7 +506,11 @@ fn handle_call_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor) -> bool { abi_args, |dfg, abi_arg| dfg.signatures[sig_ref].argument_types[abi_arg]); - // TODO: Convert return values. + if !dfg.signatures[sig_ref].return_types.is_empty() { + legalize_inst_results(dfg, + pos, + |dfg, abi_res| dfg.signatures[sig_ref].return_types[abi_res]); + } // Yes, we changed stuff. true From f15651132bac7a1e40ff58c1e22fed92b5ec47d1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 17 Mar 2017 12:34:03 -0700 Subject: [PATCH 0606/3084] Fix a bug in analyze_call(). The call arguments on call_indirect should not include the fixed callee argument. Add legalizer assertions to verify that signatures are actually valid after legalization. If not, we would get infinite legalizer loops. --- .../filetests/isa/riscv/legalize-abi.cton | 16 ++++++ lib/cretonne/src/ir/instructions.rs | 2 +- lib/cretonne/src/legalizer.rs | 57 +++++++++++++------ 3 files changed, 56 insertions(+), 19 deletions(-) diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index eada8ebda9..c092c3f2f4 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -105,3 +105,19 @@ ebb0(v0: i64x4): ; check: return $v1al, $v1ah, $v1bl, $v1bh, $v1cl, $v1ch, $v1dl, $v1dh return v1 } + +function indirect(i32) { + sig1 = signature() +ebb0(v0: i32): + call_indirect sig1, v0() + return +} + +; The first argument to call_indirect doesn't get altered. +function indirect_arg(i32, f32x2) { + sig1 = signature(f32x2) +ebb0(v0: i32, v1: f32x2): + call_indirect sig1, v0(v1) + ; check: call_indirect $sig1, $v0($V, $V) + return +} diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index c51afcc118..58801ce85f 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -310,7 +310,7 @@ impl InstructionData { CallInfo::Direct(func_ref, &args.as_slice(pool)) } &InstructionData::IndirectCall { sig_ref, ref args, .. } => { - CallInfo::Indirect(sig_ref, &args.as_slice(pool)) + CallInfo::Indirect(sig_ref, &args.as_slice(pool)[1..]) } _ => CallInfo::NotACall, } diff --git a/lib/cretonne/src/legalizer.rs b/lib/cretonne/src/legalizer.rs index 49651491fd..88bf0c9ac4 100644 --- a/lib/cretonne/src/legalizer.rs +++ b/lib/cretonne/src/legalizer.rs @@ -169,9 +169,12 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { /// be left pointing after the instructions inserted to convert the return values. /// /// This function is very similar to the `legalize_entry_arguments` function above. +/// +/// Returns the possibly new instruction representing the call. fn legalize_inst_results(dfg: &mut DataFlowGraph, pos: &mut Cursor, mut get_abi_type: ResType) + -> Inst where ResType: FnMut(&DataFlowGraph, usize) -> ArgumentType { let mut call = pos.current_inst().expect("Cursor must point to a call instruction"); @@ -244,6 +247,8 @@ fn legalize_inst_results(dfg: &mut DataFlowGraph, dfg.change_to_alias(res, v); } } + + call } /// Compute original value of type `ty` from the legalized ABI arguments. @@ -387,9 +392,9 @@ fn check_arg_types(dfg: &DataFlowGraph, args: Args, types: &[ArgumentType] /// Check if the arguments of the call `inst` match the signature. /// -/// Returns `None` if the signature matches and no changes are needed, or `Some(sig_ref)` if the +/// Returns `Ok(())` if the signature matches and no changes are needed, or `Err(sig_ref)` if the /// signature doesn't match. -fn check_call_signature(dfg: &DataFlowGraph, inst: Inst) -> Option { +fn check_call_signature(dfg: &DataFlowGraph, inst: Inst) -> Result<(), SigRef> { // Extract the signature and argument values. let (sig_ref, args) = match dfg[inst].analyze_call(&dfg.value_lists) { CallInfo::Direct(func, args) => (dfg.ext_funcs[func].signature, args), @@ -401,13 +406,25 @@ fn check_call_signature(dfg: &DataFlowGraph, inst: Inst) -> Option { if check_arg_types(dfg, args.iter().cloned(), &sig.argument_types[..]) && check_arg_types(dfg, dfg.inst_results(inst), &sig.return_types[..]) { // All types check out. - None + Ok(()) } else { // Call types need fixing. - Some(sig_ref) + Err(sig_ref) } } +/// Check if the arguments of the return `inst` match the signature. +fn check_return_signature(dfg: &DataFlowGraph, inst: Inst, sig: &Signature) -> bool { + let fixed_values = dfg[inst].opcode().constraints().fixed_value_arguments(); + check_arg_types(dfg, + dfg[inst] + .arguments(&dfg.value_lists) + .iter() + .skip(fixed_values) + .cloned(), + &sig.return_types) +} + /// Insert ABI conversion code for the arguments to the call or return instruction at `pos`. /// /// - `abi_args` is the number of arguments that the ABI signature requires. @@ -491,12 +508,12 @@ fn legalize_inst_arguments(dfg: &mut DataFlowGraph, /// /// Returns `true` if any instructions were inserted. fn handle_call_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor) -> bool { - let inst = pos.current_inst().expect("Cursor must point to a call instruction"); + let mut inst = pos.current_inst().expect("Cursor must point to a call instruction"); // Start by checking if the argument types already match the signature. let sig_ref = match check_call_signature(dfg, inst) { - None => return false, - Some(s) => s, + Ok(_) => return false, + Err(s) => s, }; // OK, we need to fix the call arguments to match the ABI signature. @@ -507,11 +524,17 @@ fn handle_call_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor) -> bool { |dfg, abi_arg| dfg.signatures[sig_ref].argument_types[abi_arg]); if !dfg.signatures[sig_ref].return_types.is_empty() { - legalize_inst_results(dfg, - pos, - |dfg, abi_res| dfg.signatures[sig_ref].return_types[abi_res]); + inst = legalize_inst_results(dfg, + pos, + |dfg, abi_res| dfg.signatures[sig_ref].return_types[abi_res]); } + debug_assert!(check_call_signature(dfg, inst).is_ok(), + "Signature still wrong: {}, {}{}", + dfg.display_inst(inst), + sig_ref, + dfg.signatures[sig_ref]); + // Yes, we changed stuff. true } @@ -523,20 +546,18 @@ fn handle_return_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor, sig: &Signature) let inst = pos.current_inst().expect("Cursor must point to a return instruction"); // Check if the returned types already match the signature. - let fixed_values = dfg[inst].opcode().constraints().fixed_value_arguments(); - if check_arg_types(dfg, - dfg[inst] - .arguments(&dfg.value_lists) - .iter() - .skip(fixed_values) - .cloned(), - &sig.return_types[..]) { + if check_return_signature(dfg, inst, sig) { return false; } let abi_args = sig.return_types.len(); legalize_inst_arguments(dfg, pos, abi_args, |_, abi_arg| sig.return_types[abi_arg]); + debug_assert!(check_return_signature(dfg, inst, sig), + "Signature still wrong: {}, sig{}", + dfg.display_inst(inst), + sig); + // Yes, we changed stuff. true } From c50bf2aa68dcbf5d952d2d0fc979ad2061d4678c Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Tue, 14 Mar 2017 20:13:44 +0000 Subject: [PATCH 0607/3084] Verify that values have a valid reference to either an instruction inserted in an EBB, or an EBB inserted in the layout. --- lib/cretonne/src/verifier.rs | 45 ++++++++++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 7 deletions(-) diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index f63db1919f..ddddb9c509 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -18,9 +18,9 @@ //! //! SSA form //! -//! TODO: //! - Values must be defined by an instruction that exists and that is inserted in //! an EBB, or be an argument of an existing EBB. +//! TODO: //! - Values used by an instruction must dominate the instruction. //! //! Control flow graph and dominator tree integrity: @@ -166,7 +166,7 @@ impl<'a> Verifier<'a> { let ret_type = inst_data.first_type(); if ret_type != types::VOID { return err!(inst, - "instruction expected to have NULL return value, found {}", + "instruction with no results expects NULL return type, found {}", ret_type); } } else { @@ -283,12 +283,43 @@ impl<'a> Verifier<'a> { } } - fn verify_value(&self, inst: Inst, v: Value) -> Result<()> { - if !self.func.dfg.value_is_valid(v) { - err!(inst, "invalid value reference {}", v) - } else { - Ok(()) + fn verify_value(&self, loc_inst: Inst, v: Value) -> Result<()> { + let dfg = &self.func.dfg; + if !dfg.value_is_valid(v) { + return err!(loc_inst, "invalid value reference {}", v); } + + // SSA form + match dfg.value_def(v) { + // Value is defined by an instruction that exists and is inserted in an EBB. + ValueDef::Res(def_inst, _) => { + if !dfg.insts.is_valid(def_inst) { + return err!(loc_inst, + "{} is defined by invalid instruction {}", + v, + def_inst); + } + if self.func.layout.inst_ebb(def_inst) == None { + return err!(loc_inst, + "{} is defined by {} which has no EBB", + v, + def_inst); + } + } + // Value is defined by an existing EBB which is inserted in the layout. + ValueDef::Arg(ebb, _) => { + if !dfg.ebb_is_valid(ebb) { + return err!(loc_inst, "{} is defined by invalid EBB {}", v, ebb); + } + if !self.func.layout.is_ebb_inserted(ebb) { + return err!(loc_inst, + "{} is defined by {} which is not in the layout", + v, + ebb); + } + } + } + Ok(()) } pub fn run(&self) -> Result<()> { From 8b300a9efbeb96d795d228685fc6eec423383fd2 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Mon, 20 Mar 2017 10:02:06 +0000 Subject: [PATCH 0608/3084] Verify that values are defined by an EBB/instruction that dominates the instruction that uses them. --- lib/cretonne/src/dominator_tree.rs | 16 ++++++++++++++-- lib/cretonne/src/verifier.rs | 28 ++++++++++++++++++++++++---- 2 files changed, 38 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 17bd3445ab..a426fedbea 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -68,8 +68,20 @@ impl DominatorTree { /// is unreachable. /// /// An instruction is considered to dominate itself. - pub fn dominates(&self, a: Inst, mut b: Inst, layout: &Layout) -> bool { + pub fn dominates(&self, a: Inst, b: Inst, layout: &Layout) -> bool { let ebb_a = layout.inst_ebb(a).expect("Instruction not in layout."); + self.ebb_dominates(ebb_a, b, layout) && layout.cmp(a, b) != Ordering::Greater + } + + /// Returns `true` if `ebb_a` dominates `b`. + /// + /// This means that every control-flow path from the function entry to `b` must go through + /// `ebb_a`. + /// + /// Dominance is ill defined for unreachable blocks. This function can always determine + /// dominance for instructions in the same EBB, but otherwise returns `false` if either block + /// is unreachable. + pub fn ebb_dominates(&self, ebb_a: Ebb, mut b: Inst, layout: &Layout) -> bool { let mut ebb_b = layout.inst_ebb(b).expect("Instruction not in layout."); let rpo_a = self.nodes[ebb_a].rpo_number; @@ -80,7 +92,7 @@ impl DominatorTree { ebb_b = layout.inst_ebb(b).expect("Dominator got removed."); } - ebb_a == ebb_b && layout.cmp(a, b) != Ordering::Greater + ebb_a == ebb_b } /// Compute the common dominator of two basic blocks. diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index ddddb9c509..d4715805ea 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -20,11 +20,11 @@ //! //! - Values must be defined by an instruction that exists and that is inserted in //! an EBB, or be an argument of an existing EBB. -//! TODO: //! - Values used by an instruction must dominate the instruction. //! //! Control flow graph and dominator tree integrity: //! +//! TODO: //! - All predecessors in the CFG must be branches to the EBB. //! - All branches to an EBB must be present in the CFG. //! - A recomputed dominator tree is identical to the existing one. @@ -56,6 +56,8 @@ use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, Value}; use ir::instructions::InstructionFormat; use ir::entities::AnyEntity; +use cfg::ControlFlowGraph; +use dominator_tree::DominatorTree; use std::fmt::{self, Display, Formatter}; use std::result; @@ -101,11 +103,19 @@ pub fn verify_function(func: &Function) -> Result<()> { struct Verifier<'a> { func: &'a Function, + cfg: ControlFlowGraph, + domtree: DominatorTree, } impl<'a> Verifier<'a> { pub fn new(func: &'a Function) -> Verifier { - Verifier { func: func } + let cfg = ControlFlowGraph::with_function(func); + let domtree = DominatorTree::with_function(func, &cfg); + Verifier { + func: func, + cfg: cfg, + domtree: domtree, + } } fn ebb_integrity(&self, ebb: Ebb, inst: Inst) -> Result<()> { @@ -291,32 +301,42 @@ impl<'a> Verifier<'a> { // SSA form match dfg.value_def(v) { - // Value is defined by an instruction that exists and is inserted in an EBB. ValueDef::Res(def_inst, _) => { + // Value is defined by an instruction that exists. if !dfg.insts.is_valid(def_inst) { return err!(loc_inst, "{} is defined by invalid instruction {}", v, def_inst); } + // Defining instruction is inserted in an EBB. if self.func.layout.inst_ebb(def_inst) == None { return err!(loc_inst, "{} is defined by {} which has no EBB", v, def_inst); } + // Defining instruction dominates the instruction that uses the value. + if !self.domtree.dominates(def_inst, loc_inst, &self.func.layout) { + return err!(loc_inst, "uses value from non-dominating {}", def_inst); + } } - // Value is defined by an existing EBB which is inserted in the layout. ValueDef::Arg(ebb, _) => { + // Value is defined by an existing EBB. if !dfg.ebb_is_valid(ebb) { return err!(loc_inst, "{} is defined by invalid EBB {}", v, ebb); } + // Defining EBB is inserted in the layout if !self.func.layout.is_ebb_inserted(ebb) { return err!(loc_inst, "{} is defined by {} which is not in the layout", v, ebb); } + // The defining EBB dominates the instruction using this value. + if !self.domtree.ebb_dominates(ebb, loc_inst, &self.func.layout) { + return err!(loc_inst, "uses value arg from non-dominating {}", ebb); + } } } Ok(()) From d383a4185107271e40ff57d224fb48c082a5b9a6 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Mon, 20 Mar 2017 11:07:50 +0000 Subject: [PATCH 0609/3084] Verify that all predecessors to an EBB are valid branches, and have the EBB recorded as a successor. --- lib/cretonne/src/ir/jumptable.rs | 5 ++++ lib/cretonne/src/verifier.rs | 40 ++++++++++++++++++++++++++++++-- 2 files changed, 43 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index 69a44ef8fb..e7f98a329d 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -72,6 +72,11 @@ impl JumpTableData { .enumerate()) } + /// Checks if any of the entries branch to `ebb`. + pub fn branches_to(&self, ebb: Ebb) -> bool { + self.table.iter().any(|target_ebb| target_ebb.expand() == Some(ebb)) + } + /// Access the whole table as a mutable slice. pub fn as_mut_slice(&mut self) -> &mut [PackedOption] { self.table.as_mut_slice() diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index d4715805ea..dbb57a5b3e 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -24,9 +24,9 @@ //! //! Control flow graph and dominator tree integrity: //! -//! TODO: //! - All predecessors in the CFG must be branches to the EBB. //! - All branches to an EBB must be present in the CFG. +//! TODO: //! - A recomputed dominator tree is identical to the existing one. //! //! Type checking @@ -54,7 +54,7 @@ //! of arguments must match the destination type, and the lane indexes must be in range. use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, Value}; -use ir::instructions::InstructionFormat; +use ir::instructions::{InstructionFormat, BranchInfo}; use ir::entities::AnyEntity; use cfg::ControlFlowGraph; use dominator_tree::DominatorTree; @@ -342,12 +342,48 @@ impl<'a> Verifier<'a> { Ok(()) } + fn cfg_integrity(&self, ebb: Ebb) -> Result<()> { + for &(pred_ebb, pred_inst) in self.cfg.get_predecessors(ebb) { + // All predecessors in the CFG must be branches to the EBB + match self.func.dfg[pred_inst].analyze_branch(&self.func.dfg.value_lists) { + BranchInfo::SingleDest(target_ebb, _) => { + if target_ebb != ebb { + return err!(ebb, + "has predecessor {} in {} which does not branch here", + pred_inst, + pred_ebb); + } + } + BranchInfo::Table(jt) => { + if !self.func.jump_tables[jt].branches_to(ebb) { + return err!(ebb, + "has predecessor {} using {} in {} which never branches here", + pred_inst, + jt, + pred_ebb); + } + } + BranchInfo::NotABranch => { + return err!(ebb, "has predecessor {} which is not a branch", pred_inst); + } + } + // All EBBs branching to `ebb` have it recorded as a successor in the CFG. + if !self.cfg.get_successors(pred_ebb).contains(&ebb) { + return err!(ebb, + "predecessor {} does not have this EBB recorded as a successor", + pred_ebb); + } + } + Ok(()) + } + pub fn run(&self) -> Result<()> { for ebb in self.func.layout.ebbs() { for inst in self.func.layout.ebb_insts(ebb) { self.ebb_integrity(ebb, inst)?; self.instruction_integrity(inst)?; } + self.cfg_integrity(ebb)?; } Ok(()) } From 34e5675d1716ade8fe8500dc48abeb135a18afab Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 20 Mar 2017 14:47:09 -0700 Subject: [PATCH 0610/3084] Add OpcodeConstraints::value_argument_constraint(). As discussed in #3. Once we know the controlling type variable of a polymorphic instruction, the types of input operands are either bound to known types, or they can vary freely. --- lib/cretonne/src/ir/instructions.rs | 68 +++++++++++++++++++++-------- 1 file changed, 51 insertions(+), 17 deletions(-) diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 58801ce85f..fe76791211 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -397,7 +397,7 @@ pub struct OpcodeConstraints { /// Offset into `OPERAND_CONSTRAINT` table of the descriptors for this opcode. The first /// `fixed_results()` entries describe the result constraints, then follows constraints for the - /// fixed `Value` input operands. The number of `Value` inputs is determined by the instruction + /// fixed `Value` input operands. (`fixed_value_arguments()` of them). /// format. constraint_offset: u16, } @@ -457,9 +457,24 @@ impl OpcodeConstraints { /// `ctrl_type`. pub fn result_type(self, n: usize, ctrl_type: Type) -> Type { assert!(n < self.fixed_results(), "Invalid result index"); - OPERAND_CONSTRAINTS[self.constraint_offset() + n] - .resolve(ctrl_type) - .expect("Result constraints can't be free") + if let ResolvedConstraint::Bound(t) = + OPERAND_CONSTRAINTS[self.constraint_offset() + n].resolve(ctrl_type) { + t + } else { + panic!("Result constraints can't be free"); + } + } + + /// Get the value type of input value number `n`, having resolved the controlling type variable + /// to `ctrl_type`. + /// + /// Unlike results, it is possible for some input values to vary freely within a specific + /// `ValueTypeSet`. This is represented with the `ArgumentConstraint::Free` variant. + pub fn value_argument_constraint(self, n: usize, ctrl_type: Type) -> ResolvedConstraint { + assert!(n < self.fixed_value_arguments(), + "Invalid value argument index"); + let offset = self.constraint_offset() + self.fixed_results(); + OPERAND_CONSTRAINTS[offset + n].resolve(ctrl_type) } /// Get the typeset of allowed types for the controlling type variable in a polymorphic @@ -475,7 +490,7 @@ impl OpcodeConstraints { } /// A value type set describes the permitted set of types for a type variable. -#[derive(Clone, Copy)] +#[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct ValueTypeSet { min_lanes: u8, max_lanes: u8, @@ -561,24 +576,31 @@ enum OperandConstraint { impl OperandConstraint { /// Resolve this operand constraint into a concrete value type, given the value of the /// controlling type variable. - /// Returns `None` if this is a free operand which is independent of the controlling type - /// variable. - pub fn resolve(&self, ctrl_type: Type) -> Option { + pub fn resolve(&self, ctrl_type: Type) -> ResolvedConstraint { use self::OperandConstraint::*; + use self::ResolvedConstraint::Bound; match *self { - Concrete(t) => Some(t), - Free(_) => None, - Same => Some(ctrl_type), - LaneOf => Some(ctrl_type.lane_type()), - AsBool => Some(ctrl_type.as_bool()), - HalfWidth => Some(ctrl_type.half_width().expect("invalid type for half_width")), - DoubleWidth => Some(ctrl_type.double_width().expect("invalid type for double_width")), - HalfVector => Some(ctrl_type.half_vector().expect("invalid type for half_vector")), - DoubleVector => Some(ctrl_type.by(2).expect("invalid type for double_vector")), + Concrete(t) => Bound(t), + Free(vts) => ResolvedConstraint::Free(TYPE_SETS[vts as usize]), + Same => Bound(ctrl_type), + LaneOf => Bound(ctrl_type.lane_type()), + AsBool => Bound(ctrl_type.as_bool()), + HalfWidth => Bound(ctrl_type.half_width().expect("invalid type for half_width")), + DoubleWidth => Bound(ctrl_type.double_width().expect("invalid type for double_width")), + HalfVector => Bound(ctrl_type.half_vector().expect("invalid type for half_vector")), + DoubleVector => Bound(ctrl_type.by(2).expect("invalid type for double_vector")), } } } +/// The type constraint on a value argument once the controlling type variable is known. +#[derive(Copy, Clone, Debug, PartialEq, Eq)] +pub enum ResolvedConstraint { + /// The operand is bound to a known type. + Bound(Type), + /// The operand type can vary freely within the given set. + Free(ValueTypeSet), +} #[cfg(test)] mod tests { @@ -630,12 +652,24 @@ mod tests { assert!(!a.requires_typevar_operand()); assert_eq!(a.fixed_results(), 1); assert_eq!(a.fixed_value_arguments(), 2); + assert_eq!(a.result_type(0, types::I32), types::I32); + assert_eq!(a.result_type(0, types::I8), types::I8); + assert_eq!(a.value_argument_constraint(0, types::I32), + ResolvedConstraint::Bound(types::I32)); + assert_eq!(a.value_argument_constraint(1, types::I32), + ResolvedConstraint::Bound(types::I32)); let b = Opcode::Bitcast.constraints(); assert!(!b.use_typevar_operand()); assert!(!b.requires_typevar_operand()); assert_eq!(b.fixed_results(), 1); assert_eq!(b.fixed_value_arguments(), 1); + assert_eq!(b.result_type(0, types::I32), types::I32); + assert_eq!(b.result_type(0, types::I8), types::I8); + match b.value_argument_constraint(0, types::I32) { + ResolvedConstraint::Free(vts) => assert!(vts.contains(types::F32)), + _ => panic!("Unexpected constraint from value_argument_constraint"), + } let c = Opcode::Call.constraints(); assert_eq!(c.fixed_results(), 0); From dfdc1ed514142cc16b0100fb2883e8d2bca60e9a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 20 Mar 2017 15:14:51 -0700 Subject: [PATCH 0611/3084] Move ABI boundary legalization into a sub-module. Keep things organized. --- .../{legalizer.rs => legalizer/boundary.rs} | 112 ++++-------------- lib/cretonne/src/legalizer/mod.rs | 96 +++++++++++++++ 2 files changed, 116 insertions(+), 92 deletions(-) rename lib/cretonne/src/{legalizer.rs => legalizer/boundary.rs} (79%) create mode 100644 lib/cretonne/src/legalizer/mod.rs diff --git a/lib/cretonne/src/legalizer.rs b/lib/cretonne/src/legalizer/boundary.rs similarity index 79% rename from lib/cretonne/src/legalizer.rs rename to lib/cretonne/src/legalizer/boundary.rs index 88bf0c9ac4..1dc638a086 100644 --- a/lib/cretonne/src/legalizer.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -1,106 +1,34 @@ -//! Legalize instructions. +//! Legalize ABI boundaries. //! -//! A legal instruction is one that can be mapped directly to a machine code instruction for the -//! target ISA. The `legalize_function()` function takes as input any function and transforms it -//! into an equivalent function using only legal instructions. +//! This legalizer sub-module contains code for dealing with ABI boundaries: //! -//! The characteristics of legal instructions depend on the target ISA, so any given instruction -//! can be legal for one ISA and illegal for another. +//! - Function arguments passed to the entry block. +//! - Function arguments passed to call instructions. +//! - Return values from call instructions. +//! - Return values passed to return instructions. //! -//! Besides transforming instructions, the legalizer also fills out the `function.encodings` map -//! which provides a legal encoding recipe for every instruction. +//! The ABI boundary legalization happens in two phases: //! -//! The legalizer does not deal with register allocation constraints. These constraints are derived -//! from the encoding recipes, and solved later by the register allocator. +//! 1. The `legalize_signatures` function rewrites all the preamble signatures with ABI information +//! and possibly new argument types. It also rewrites the entry block arguments to match. +//! 2. The `handle_call_abi` and `handle_return_abi` functions rewrite call and return instructions +//! to match the new ABI signatures. +//! +//! Between the two phases, preamble signatures and call/return arguments don't match. This +//! intermediate state doesn't type check. use abi::{legalize_abi_value, ValueConversion}; -use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, Inst, InstBuilder, Ebb, Type, - Value, Signature, SigRef, ArgumentType}; -use ir::condcodes::IntCC; +use ir::{Function, Cursor, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, Signature, SigRef, + ArgumentType}; use ir::instructions::CallInfo; -use isa::{TargetIsa, Legalize}; - -/// Legalize `func` for `isa`. -/// -/// - Transform any instructions that don't have a legal representation in `isa`. -/// - Fill out `func.encodings`. -/// -pub fn legalize_function(func: &mut Function, isa: &TargetIsa) { - legalize_signatures(func, isa); - - // TODO: This is very simplified and incomplete. - func.encodings.resize(func.dfg.num_insts()); - let mut pos = Cursor::new(&mut func.layout); - while let Some(_ebb) = pos.next_ebb() { - // Keep track of the cursor position before the instruction being processed, so we can - // double back when replacing instructions. - let mut prev_pos = pos.position(); - - while let Some(inst) = pos.next_inst() { - let opcode = func.dfg[inst].opcode(); - - // Check for ABI boundaries that need to be converted to the legalized signature. - if opcode.is_call() && handle_call_abi(&mut func.dfg, &mut pos) { - // Go back and legalize the inserted argument conversion instructions. - pos.set_position(prev_pos); - continue; - } - - if opcode.is_return() && handle_return_abi(&mut func.dfg, &mut pos, &func.signature) { - // Go back and legalize the inserted return value conversion instructions. - pos.set_position(prev_pos); - continue; - } - - match isa.encode(&func.dfg, &func.dfg[inst]) { - Ok(encoding) => *func.encodings.ensure(inst) = encoding, - Err(action) => { - // We should transform the instruction into legal equivalents. - // Possible strategies are: - // 1. Legalize::Expand: Expand instruction into sequence of legal instructions. - // Possibly iteratively. () - // 2. Legalize::Narrow: Split the controlling type variable into high and low - // parts. This applies both to SIMD vector types which can be halved and to - // integer types such as `i64` used on a 32-bit ISA. (). - // 3. TODO: Promote the controlling type variable to a larger type. This - // typically means expressing `i8` and `i16` arithmetic in terms if `i32` - // operations on RISC targets. (It may or may not be beneficial to promote - // small vector types versus splitting them.) - // 4. TODO: Convert to library calls. For example, floating point operations on - // an ISA with no IEEE 754 support. - let changed = match action { - Legalize::Expand => expand(&mut pos, &mut func.dfg), - Legalize::Narrow => narrow(&mut pos, &mut func.dfg), - }; - // If the current instruction was replaced, we need to double back and revisit - // the expanded sequence. This is both to assign encodings and possible to - // expand further. - // There's a risk of infinite looping here if the legalization patterns are - // unsound. Should we attempt to detect that? - if changed { - pos.set_position(prev_pos); - } - } - } - - // Remember this position in case we need to double back. - prev_pos = pos.position(); - } - } -} - -// Include legalization patterns that were generated by `gen_legalizer.py` from the `XForms` in -// `meta/cretonne/legalize.py`. -// -// Concretely, this defines private functions `narrow()`, and `expand()`. -include!(concat!(env!("OUT_DIR"), "/legalizer.rs")); +use isa::TargetIsa; /// Legalize all the function signatures in `func`. /// /// This changes all signatures to be ABI-compliant with full `ArgumentLoc` annotations. It doesn't /// change the entry block arguments, calls, or return instructions, so this can leave the function /// in a state with type discrepancies. -fn legalize_signatures(func: &mut Function, isa: &TargetIsa) { +pub fn legalize_signatures(func: &mut Function, isa: &TargetIsa) { isa.legalize_signature(&mut func.signature); for sig in func.dfg.signatures.keys() { isa.legalize_signature(&mut func.dfg.signatures[sig]); @@ -507,7 +435,7 @@ fn legalize_inst_arguments(dfg: &mut DataFlowGraph, /// original return values. The call's result values will be adapted to match the new signature. /// /// Returns `true` if any instructions were inserted. -fn handle_call_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor) -> bool { +pub fn handle_call_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor) -> bool { let mut inst = pos.current_inst().expect("Cursor must point to a call instruction"); // Start by checking if the argument types already match the signature. @@ -542,7 +470,7 @@ fn handle_call_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor) -> bool { /// Insert ABI conversion code before and after the call instruction at `pos`. /// /// Return `true` if any instructions were inserted. -fn handle_return_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor, sig: &Signature) -> bool { +pub fn handle_return_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor, sig: &Signature) -> bool { let inst = pos.current_inst().expect("Cursor must point to a return instruction"); // Check if the returned types already match the signature. diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs new file mode 100644 index 0000000000..a315cf069f --- /dev/null +++ b/lib/cretonne/src/legalizer/mod.rs @@ -0,0 +1,96 @@ +//! Legalize instructions. +//! +//! A legal instruction is one that can be mapped directly to a machine code instruction for the +//! target ISA. The `legalize_function()` function takes as input any function and transforms it +//! into an equivalent function using only legal instructions. +//! +//! The characteristics of legal instructions depend on the target ISA, so any given instruction +//! can be legal for one ISA and illegal for another. +//! +//! Besides transforming instructions, the legalizer also fills out the `function.encodings` map +//! which provides a legal encoding recipe for every instruction. +//! +//! The legalizer does not deal with register allocation constraints. These constraints are derived +//! from the encoding recipes, and solved later by the register allocator. + +use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, InstBuilder}; +use ir::condcodes::IntCC; +use isa::{TargetIsa, Legalize}; + +mod boundary; + +/// Legalize `func` for `isa`. +/// +/// - Transform any instructions that don't have a legal representation in `isa`. +/// - Fill out `func.encodings`. +/// +pub fn legalize_function(func: &mut Function, isa: &TargetIsa) { + boundary::legalize_signatures(func, isa); + + // TODO: This is very simplified and incomplete. + func.encodings.resize(func.dfg.num_insts()); + let mut pos = Cursor::new(&mut func.layout); + while let Some(_ebb) = pos.next_ebb() { + // Keep track of the cursor position before the instruction being processed, so we can + // double back when replacing instructions. + let mut prev_pos = pos.position(); + + while let Some(inst) = pos.next_inst() { + let opcode = func.dfg[inst].opcode(); + + // Check for ABI boundaries that need to be converted to the legalized signature. + if opcode.is_call() && boundary::handle_call_abi(&mut func.dfg, &mut pos) { + // Go back and legalize the inserted argument conversion instructions. + pos.set_position(prev_pos); + continue; + } + + if opcode.is_return() && + boundary::handle_return_abi(&mut func.dfg, &mut pos, &func.signature) { + // Go back and legalize the inserted return value conversion instructions. + pos.set_position(prev_pos); + continue; + } + + match isa.encode(&func.dfg, &func.dfg[inst]) { + Ok(encoding) => *func.encodings.ensure(inst) = encoding, + Err(action) => { + // We should transform the instruction into legal equivalents. + // Possible strategies are: + // 1. Legalize::Expand: Expand instruction into sequence of legal instructions. + // Possibly iteratively. () + // 2. Legalize::Narrow: Split the controlling type variable into high and low + // parts. This applies both to SIMD vector types which can be halved and to + // integer types such as `i64` used on a 32-bit ISA. (). + // 3. TODO: Promote the controlling type variable to a larger type. This + // typically means expressing `i8` and `i16` arithmetic in terms if `i32` + // operations on RISC targets. (It may or may not be beneficial to promote + // small vector types versus splitting them.) + // 4. TODO: Convert to library calls. For example, floating point operations on + // an ISA with no IEEE 754 support. + let changed = match action { + Legalize::Expand => expand(&mut pos, &mut func.dfg), + Legalize::Narrow => narrow(&mut pos, &mut func.dfg), + }; + // If the current instruction was replaced, we need to double back and revisit + // the expanded sequence. This is both to assign encodings and possible to + // expand further. + // There's a risk of infinite looping here if the legalization patterns are + // unsound. Should we attempt to detect that? + if changed { + pos.set_position(prev_pos); + } + } + } + + // Remember this position in case we need to double back. + prev_pos = pos.position(); + } + } +} + +// Include legalization patterns that were generated by `gen_legalizer.py` from the `XForms` in +// `meta/cretonne/legalize.py`. +// +// Concretely, this defines private functions `narrow()`, and `expand()`. +include!(concat!(env!("OUT_DIR"), "/legalizer.rs")); From 2a321f42fbdd013e7a30d7f6dbfb2181c4f4e8d9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 21 Mar 2017 13:08:17 -0700 Subject: [PATCH 0612/3084] Strip the _lohi suffix from the isplit instructions. For symmetry with the vector splitting instructions, we now have: isplit iconcat vsplit vconcat No functional change. --- cranelift/docs/langref.rst | 4 +-- .../filetests/isa/riscv/legalize-abi.cton | 26 +++++++++---------- .../filetests/isa/riscv/legalize-i64.cton | 24 ++++++++--------- lib/cretonne/meta/base/instructions.py | 8 +++--- lib/cretonne/meta/base/legalize.py | 20 +++++++------- lib/cretonne/src/abi.rs | 4 +-- lib/cretonne/src/legalizer/boundary.rs | 4 +-- 7 files changed, 45 insertions(+), 45 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 9a08da062d..19e17490a2 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -839,8 +839,8 @@ Legalization operations These instructions are used as helpers when legalizing types and operations for the target ISA. -.. autoinst:: isplit_lohi -.. autoinst:: iconcat_lohi +.. autoinst:: isplit +.. autoinst:: iconcat Base instruction group ====================== diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index c092c3f2f4..926a3c2820 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -7,9 +7,9 @@ isa riscv function int_split_args(i64) -> i64 { ebb0(v0: i64): ; check: $ebb0($(v0l=$V): i32, $(v0h=$V): i32): - ; check: iconcat_lohi $v0l, $v0h + ; check: iconcat $v0l, $v0h v1 = iadd_imm v0, 1 - ; check: $(v1l=$V), $(v1h=$V) = isplit_lohi $v1 + ; check: $(v1l=$V), $(v1h=$V) = isplit $v1 ; check: return $v1l, $v1h return v1 } @@ -20,7 +20,7 @@ function split_call_arg(i32) { ebb0(v0: i32): v1 = uextend.i64 v0 call fn1(v1) - ; check: $(v1l=$V), $(v1h=$V) = isplit_lohi $v1 + ; check: $(v1l=$V), $(v1h=$V) = isplit $v1 ; check: call $fn1($v1l, $v1h) call fn2(v0, v1) ; check: call $fn2($v0, $V, $V) @@ -33,7 +33,7 @@ ebb0: v1 = call fn1() ; check: $ebb0: ; nextln: $(v1l=$V), $(v1h=$V) = call $fn1() - ; check: $(v1new=$V) = iconcat_lohi $v1l, $v1h + ; check: $(v1new=$V) = iconcat $v1l, $v1h ; check: $v1 = copy $v1new jump ebb1(v1) ; check: jump $ebb1($v1) @@ -49,7 +49,7 @@ ebb0: v1, v2 = call fn1() ; check: $ebb0: ; nextln: $v1, $(v2l=$V), $(v2h=$V) = call $fn1() - ; check: $(v2new=$V) = iconcat_lohi $v2l, $v2h + ; check: $(v2new=$V) = iconcat $v2l, $v2h ; check: $v2 -> $v2new jump ebb1(v1, v2) ; check: jump $ebb1($v1, $v2) @@ -87,21 +87,21 @@ ebb1(v10: i8): function vector_split_args(i64x4) -> i64x4 { ebb0(v0: i64x4): ; check: $ebb0($(v0al=$V): i32, $(v0ah=$V): i32, $(v0bl=$V): i32, $(v0bh=$V): i32, $(v0cl=$V): i32, $(v0ch=$V): i32, $(v0dl=$V): i32, $(v0dh=$V): i32): - ; check: $(v0a=$V) = iconcat_lohi $v0al, $v0ah - ; check: $(v0b=$V) = iconcat_lohi $v0bl, $v0bh + ; check: $(v0a=$V) = iconcat $v0al, $v0ah + ; check: $(v0b=$V) = iconcat $v0bl, $v0bh ; check: $(v0ab=$V) = vconcat $v0a, $v0b - ; check: $(v0c=$V) = iconcat_lohi $v0cl, $v0ch - ; check: $(v0d=$V) = iconcat_lohi $v0dl, $v0dh + ; check: $(v0c=$V) = iconcat $v0cl, $v0ch + ; check: $(v0d=$V) = iconcat $v0dl, $v0dh ; check: $(v0cd=$V) = vconcat $v0c, $v0d ; check: $(v0abcd=$V) = vconcat $v0ab, $v0cd v1 = iadd v0, v0 ; check: $(v1ab=$V), $(v1cd=$V) = vsplit ; check: $(v1a=$V), $(v1b=$V) = vsplit $v1ab - ; check: $(v1al=$V), $(v1ah=$V) = isplit_lohi $v1a - ; check: $(v1bl=$V), $(v1bh=$V) = isplit_lohi $v1b + ; check: $(v1al=$V), $(v1ah=$V) = isplit $v1a + ; check: $(v1bl=$V), $(v1bh=$V) = isplit $v1b ; check: $(v1c=$V), $(v1d=$V) = vsplit $v1cd - ; check: $(v1cl=$V), $(v1ch=$V) = isplit_lohi $v1c - ; check: $(v1dl=$V), $(v1dh=$V) = isplit_lohi $v1d + ; check: $(v1cl=$V), $(v1ch=$V) = isplit $v1c + ; check: $(v1dl=$V), $(v1dh=$V) = isplit $v1d ; check: return $v1al, $v1ah, $v1bl, $v1bh, $v1cl, $v1ch, $v1dl, $v1dh return v1 } diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton index fb8d412d8c..1d446f8a47 100644 --- a/cranelift/filetests/isa/riscv/legalize-i64.cton +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -10,39 +10,39 @@ ebb0(v1: i64, v2: i64): v3 = band v1, v2 return v3 } -; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi -; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi +; check: $(v1l=$V), $(v1h=$VX) = isplit +; check: $(v2l=$V), $(v2h=$VX) = isplit ; check: [R#ec ; sameln: $(v3l=$V) = band $v1l, $v2l ; check: [R#ec ; sameln: $(v3h=$V) = band $v1h, $v2h -; check: $v3 = iconcat_lohi $v3l, $v3h +; check: $v3 = iconcat $v3l, $v3h function bitwise_or(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): v3 = bor v1, v2 return v3 } -; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi -; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi +; check: $(v1l=$V), $(v1h=$VX) = isplit +; check: $(v2l=$V), $(v2h=$VX) = isplit ; check: [R#cc ; sameln: $(v3l=$V) = bor $v1l, $v2l ; check: [R#cc ; sameln: $(v3h=$V) = bor $v1h, $v2h -; check: $v3 = iconcat_lohi $v3l, $v3h +; check: $v3 = iconcat $v3l, $v3h function bitwise_xor(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): v3 = bxor v1, v2 return v3 } -; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi -; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi +; check: $(v1l=$V), $(v1h=$VX) = isplit +; check: $(v2l=$V), $(v2h=$VX) = isplit ; check: [R#8c ; sameln: $(v3l=$V) = bxor $v1l, $v2l ; check: [R#8c ; sameln: $(v3h=$V) = bxor $v1h, $v2h -; check: $v3 = iconcat_lohi $v3l, $v3h +; check: $v3 = iconcat $v3l, $v3h function arith_add(i64, i64) -> i64 { ; Legalizing iadd.i64 requires two steps: @@ -52,8 +52,8 @@ ebb0(v1: i64, v2: i64): v3 = iadd v1, v2 return v3 } -; check: $(v1l=$V), $(v1h=$VX) = isplit_lohi -; check: $(v2l=$V), $(v2h=$VX) = isplit_lohi +; check: $(v1l=$V), $(v1h=$VX) = isplit +; check: $(v2l=$V), $(v2h=$VX) = isplit ; check: [R#0c ; sameln: $(v3l=$V) = iadd $v1l, $v2l ; check: $(c=$V) = icmp ult, $v3l, $v1l @@ -62,4 +62,4 @@ ebb0(v1: i64, v2: i64): ; TODO: This doesn't typecheck. We need to convert the b1 result to i32. ; check: [R#0c ; sameln: $(v3h=$V) = iadd $v3h1, $c -; check: $v3 = iconcat_lohi $v3l, $v3h +; check: $v3 = iconcat $v3l, $v3h diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index e0efafb1f5..66dc0e9090 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -1286,8 +1286,8 @@ lo = Operand( hi = Operand( 'hi', WideInt.half_width(), 'The high bits of `x`') -isplit_lohi = Instruction( - 'isplit_lohi', r""" +isplit = Instruction( + 'isplit', r""" Split a scalar integer into low and high parts. Returns the low half of `x` and the high half of `x` as two independent @@ -1305,8 +1305,8 @@ a = Operand( 'a', NarrowInt.double_width(), doc='The concatenation of `lo` and `hi`') -iconcat_lohi = Instruction( - 'iconcat_lohi', r""" +iconcat = Instruction( + 'iconcat', r""" Concatenate low and high bits to form a larger integer type. """, ins=(lo, hi), outs=a) diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index b2f8fc4f6d..9b0def9ae9 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -9,7 +9,7 @@ instructions that are legal. from __future__ import absolute_import from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm from .instructions import isub, isub_bin, isub_bout, isub_borrow -from .instructions import band, bor, bxor, isplit_lohi, iconcat_lohi +from .instructions import band, bor, bxor, isplit, iconcat from .instructions import icmp, iconst from cdsl.ast import Var from cdsl.xform import Rtl, XFormGroup @@ -54,32 +54,32 @@ ah = Var('ah') narrow.legalize( a << iadd(x, y), Rtl( - (xl, xh) << isplit_lohi(x), - (yl, yh) << isplit_lohi(y), + (xl, xh) << isplit(x), + (yl, yh) << isplit(y), (al, c) << iadd_cout(xl, yl), ah << iadd_cin(xh, yh, c), - a << iconcat_lohi(al, ah) + a << iconcat(al, ah) )) narrow.legalize( a << isub(x, y), Rtl( - (xl, xh) << isplit_lohi(x), - (yl, yh) << isplit_lohi(y), + (xl, xh) << isplit(x), + (yl, yh) << isplit(y), (al, b) << isub_bout(xl, yl), ah << isub_bin(xh, yh, b), - a << iconcat_lohi(al, ah) + a << iconcat(al, ah) )) for bitop in [band, bor, bxor]: narrow.legalize( a << bitop(x, y), Rtl( - (xl, xh) << isplit_lohi(x), - (yl, yh) << isplit_lohi(y), + (xl, xh) << isplit(x), + (yl, yh) << isplit(y), al << bitop(xl, yl), ah << bitop(xh, yh), - a << iconcat_lohi(al, ah) + a << iconcat(al, ah) )) # Expand integer operations with carry for RISC architectures that don't have diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index d011311e7c..14a034b8fb 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -38,10 +38,10 @@ impl From for ArgAction { /// Legalization action to be applied to a value that is being passed to or from a legalized ABI. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum ValueConversion { - /// Split an integer types into low and high parts, using `isplit_lohi`. + /// Split an integer types into low and high parts, using `isplit`. IntSplit, - /// Split a vector type into halves with identical lane types. + /// Split a vector type into halves with identical lane types, using `vsplit`. VectorSplit, /// Bit-cast to an integer type of the same size. diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 1dc638a086..3c98009aed 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -210,7 +210,7 @@ fn convert_from_abi(dfg: &mut DataFlowGraph, let abi_ty = ty.half_width().expect("Invalid type for conversion"); let lo = convert_from_abi(dfg, pos, abi_ty, get_arg); let hi = convert_from_abi(dfg, pos, abi_ty, get_arg); - dfg.ins(pos).iconcat_lohi(lo, hi) + dfg.ins(pos).iconcat(lo, hi) } // Construct a `ty` by concatenating two halves of a vector. ValueConversion::VectorSplit => { @@ -271,7 +271,7 @@ fn convert_to_abi(dfg: &mut DataFlowGraph, let ty = dfg.value_type(value); match legalize_abi_value(ty, &arg_type) { ValueConversion::IntSplit => { - let (lo, hi) = dfg.ins(pos).isplit_lohi(value); + let (lo, hi) = dfg.ins(pos).isplit(value); convert_to_abi(dfg, pos, lo, put_arg); convert_to_abi(dfg, pos, hi, put_arg); } From 697246658d9811a4b5c70f61580829ed39d21309 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 21 Mar 2017 13:25:08 -0700 Subject: [PATCH 0613/3084] Avoid generating value split instructions. The legalizer often splits values into parts with the vsplit and isplit_lohi instructions. Avoid doing that for values that are already defined by the corresponding concatenation instructions. This reduces the number of instructions created during legalization, and it simplifies later optimizations. A number of dead concatenation instructions are left behind. They can be trivially cleaned up by a dead code elimination pass. --- .../filetests/isa/riscv/legalize-i64.cton | 19 ++- lib/cretonne/meta/gen_legalizer.py | 71 +++++++---- lib/cretonne/src/legalizer/boundary.rs | 5 +- lib/cretonne/src/legalizer/mod.rs | 1 + lib/cretonne/src/legalizer/split.rs | 120 ++++++++++++++++++ 5 files changed, 181 insertions(+), 35 deletions(-) create mode 100644 lib/cretonne/src/legalizer/split.rs diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton index 1d446f8a47..dfa78447af 100644 --- a/cranelift/filetests/isa/riscv/legalize-i64.cton +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -2,47 +2,46 @@ test legalizer isa riscv supports_m=1 -; regex: V=v\d+ -; regex: VX=vx\d+ +; regex: V=vx?\d+ function bitwise_and(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): v3 = band v1, v2 return v3 } -; check: $(v1l=$V), $(v1h=$VX) = isplit -; check: $(v2l=$V), $(v2h=$VX) = isplit +; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32): ; check: [R#ec ; sameln: $(v3l=$V) = band $v1l, $v2l ; check: [R#ec ; sameln: $(v3h=$V) = band $v1h, $v2h ; check: $v3 = iconcat $v3l, $v3h +; check: return $v3l, $v3h function bitwise_or(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): v3 = bor v1, v2 return v3 } -; check: $(v1l=$V), $(v1h=$VX) = isplit -; check: $(v2l=$V), $(v2h=$VX) = isplit +; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32): ; check: [R#cc ; sameln: $(v3l=$V) = bor $v1l, $v2l ; check: [R#cc ; sameln: $(v3h=$V) = bor $v1h, $v2h ; check: $v3 = iconcat $v3l, $v3h +; check: return $v3l, $v3h function bitwise_xor(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): v3 = bxor v1, v2 return v3 } -; check: $(v1l=$V), $(v1h=$VX) = isplit -; check: $(v2l=$V), $(v2h=$VX) = isplit +; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32): ; check: [R#8c ; sameln: $(v3l=$V) = bxor $v1l, $v2l ; check: [R#8c ; sameln: $(v3h=$V) = bxor $v1h, $v2h ; check: $v3 = iconcat $v3l, $v3h +; check: return $v3l, $v3h function arith_add(i64, i64) -> i64 { ; Legalizing iadd.i64 requires two steps: @@ -52,8 +51,7 @@ ebb0(v1: i64, v2: i64): v3 = iadd v1, v2 return v3 } -; check: $(v1l=$V), $(v1h=$VX) = isplit -; check: $(v2l=$V), $(v2h=$VX) = isplit +; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32): ; check: [R#0c ; sameln: $(v3l=$V) = iadd $v1l, $v2l ; check: $(c=$V) = icmp ult, $v3l, $v1l @@ -63,3 +61,4 @@ ebb0(v1: i64, v2: i64): ; check: [R#0c ; sameln: $(v3h=$V) = iadd $v3h1, $c ; check: $v3 = iconcat $v3l, $v3h +; check: return $v3l, $v3h diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 95d16d3663..10e9641df7 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -9,7 +9,7 @@ the input instruction. """ from __future__ import absolute_import from srcgen import Formatter -from base import legalize +from base import legalize, instructions from cdsl.ast import Var try: @@ -117,36 +117,61 @@ def wrap_tup(seq): return '({})'.format(', '.join(tup)) +def is_value_split(node): + # type: (Def) -> bool + """ + Determine if `node` represents one of the value splitting instructions: + `isplit` or `vsplit. These instructions are lowered specially by the + `legalize::split` module. + """ + if len(node.defs) != 2: + return False + return node.expr.inst in (instructions.isplit, instructions.vsplit) + + def emit_dst_inst(node, fmt): # type: (Def, Formatter) -> None exact_replace = False replaced_inst = None # type: str fixup_first_result = False - if len(node.defs) == 0: - # This node doesn't define any values, so just insert the new - # instruction. - builder = 'dfg.ins(pos)' - else: - src_def0 = node.defs[0].src_def - if src_def0 and node.defs[0] == src_def0.defs[0]: - # The primary result is replacing the primary result of the src - # pattern. - # Replace the whole instruction. - builder = 'let {} = dfg.replace(inst)'.format(wrap_tup(node.defs)) - replaced_inst = 'inst' - # Secondary values weren't replaced if this is an exact replacement - # for all the src results. - exact_replace = (node.defs == src_def0.defs) - else: - # Insert a new instruction since its primary def doesn't match the - # src. - builder = 'let {} = dfg.ins(pos)'.format(wrap_tup(node.defs)) - fixup_first_result = node.defs[0].is_output() - fmt.line('{}.{};'.format(builder, node.expr.rust_builder(node.defs))) + if is_value_split(node): + # Split instructions are not emitted with the builder, but by calling + # special functions in the `legalizer::split` module. These functions + # will eliminate concat-split patterns. + fmt.line( + 'let {} = split::{}(dfg, pos, {});' + .format( + wrap_tup(node.defs), + node.expr.inst.snake_name(), + node.expr.args[0])) + else: + if len(node.defs) == 0: + # This node doesn't define any values, so just insert the new + # instruction. + builder = 'dfg.ins(pos)' + else: + src_def0 = node.defs[0].src_def + if src_def0 and node.defs[0] == src_def0.defs[0]: + # The primary result is replacing the primary result of the + # source pattern. + # Replace the whole instruction. + builder = 'let {} = dfg.replace(inst)'.format( + wrap_tup(node.defs)) + replaced_inst = 'inst' + # Secondary values weren't replaced if this is an exact + # replacement for all the source results. + exact_replace = (node.defs == src_def0.defs) + else: + # Insert a new instruction since its primary def doesn't match + # the source. + builder = 'let {} = dfg.ins(pos)'.format(wrap_tup(node.defs)) + fixup_first_result = node.defs[0].is_output() + + fmt.line('{}.{};'.format(builder, node.expr.rust_builder(node.defs))) # If we just replaced an instruction, we need to bump the cursor so - # following instructions are inserted *after* the replaced insruction. + # following instructions are inserted *after* the replaced instruction. if replaced_inst: with fmt.indented( 'if pos.current_inst() == Some({}) {{' diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 3c98009aed..a53526232a 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -22,6 +22,7 @@ use ir::{Function, Cursor, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, S ArgumentType}; use ir::instructions::CallInfo; use isa::TargetIsa; +use legalizer::split::{isplit, vsplit}; /// Legalize all the function signatures in `func`. /// @@ -271,12 +272,12 @@ fn convert_to_abi(dfg: &mut DataFlowGraph, let ty = dfg.value_type(value); match legalize_abi_value(ty, &arg_type) { ValueConversion::IntSplit => { - let (lo, hi) = dfg.ins(pos).isplit(value); + let (lo, hi) = isplit(dfg, pos, value); convert_to_abi(dfg, pos, lo, put_arg); convert_to_abi(dfg, pos, hi, put_arg); } ValueConversion::VectorSplit => { - let (lo, hi) = dfg.ins(pos).vsplit(value); + let (lo, hi) = vsplit(dfg, pos, value); convert_to_abi(dfg, pos, lo, put_arg); convert_to_abi(dfg, pos, hi, put_arg); } diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index a315cf069f..56e3e10dd6 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -18,6 +18,7 @@ use ir::condcodes::IntCC; use isa::{TargetIsa, Legalize}; mod boundary; +mod split; /// Legalize `func` for `isa`. /// diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs new file mode 100644 index 0000000000..b8323a59fb --- /dev/null +++ b/lib/cretonne/src/legalizer/split.rs @@ -0,0 +1,120 @@ +//! Value splitting. +//! +//! Some value types are too large to fit in registers, so they need to be split into smaller parts +//! that the ISA can operate on. There's two dimensions of splitting, represented by two +//! complementary instruction pairs: +//! +//! - `isplit` and `iconcat` for splitting integer types into smaller integers. +//! - `vsplit` and `vconcat` for splitting vector types into smaller vector types with the same +//! lane types. +//! +//! There is no floating point splitting. If an ISA doesn't support `f64` values, they probably +//! have to be bit-cast to `i64` and possibly split into two `i32` values that fit in registers. +//! This breakdown is handled by the ABI lowering. +//! +//! When legalizing a single instruction, it is wrapped in splits and concatenations: +//! +//!```cton +//! v1 = bxor.i64 v2, v3 +//! ``` +//! +//! becomes: +//! +//!```cton +//! v20, v21 = isplit v2 +//! v30, v31 = isplit v3 +//! v10 = bxor.i32 v20, v30 +//! v11 = bxor.i32 v21, v31 +//! v1 = iconcat v10, v11 +//! ``` +//! +//! This local expansion approach still leaves the original `i64` values in the code as operands on +//! the `split` and `concat` instructions. It also creates a lot of redundant code to clean up as +//! values are constantly split and concatenated. +//! +//! # Optimized splitting +//! +//! We can eliminate a lot of the splitting code quite easily. Whenever we need to split a value, +//! first check if the value is defined by the corresponding concatenation. If so, then just use +//! the two concatenation inputs directly: +//! +//! ```cton +//! v4 = iadd_imm.i64 v1, 1 +//! ``` +//! +//! becomes, using the expanded code from above: +//! +//! ```cton +//! v40, v5 = iadd_imm_cout.i32 v10, 1 +//! v6 = bint.i32 +//! v41 = iadd.i32 v11, v6 +//! v4 = iconcat v40, v41 +//! ``` +//! +//! This means that the `iconcat` instructions defining `v1` and `v4` end up with no uses, so they +//! can be trivially deleted by a dead code elimination pass. +//! +//! # EBB arguments +//! +//! If all instructions that produce an `i64` value are legalized as above, we will eventually end +//! up with no `i64` values anywhere, except for EBB arguments. We can work around this by +//! iteratively splitting EBB arguments too. That should leave us with no illegal value types +//! anywhere. +//! +//! It is possible to have circular dependencies of EBB arguments that are never used by any real +//! instructions. These loops will remain in the program. + +use ir::{DataFlowGraph, Cursor, Value, Opcode, ValueDef, InstructionData, InstBuilder}; + +/// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values +/// if possible. +pub fn isplit(dfg: &mut DataFlowGraph, pos: &mut Cursor, value: Value) -> (Value, Value) { + split_value(dfg, pos, value, Opcode::Iconcat) +} + +/// Split `value` into halves using the `vsplit` semantics. Do this by reusing existing values if +/// possible. +pub fn vsplit(dfg: &mut DataFlowGraph, pos: &mut Cursor, value: Value) -> (Value, Value) { + split_value(dfg, pos, value, Opcode::Vconcat) +} + +/// Split a single value using the integer or vector semantics given by the `concat` opcode. +/// +/// If the value is defined by a `concat` instruction, just reuse the operand values of that +/// instruction. +/// +/// Return the two new values representing the parts of `value`. +fn split_value(dfg: &mut DataFlowGraph, + pos: &mut Cursor, + value: Value, + concat: Opcode) + -> (Value, Value) { + let value = dfg.resolve_copies(value); + let mut reuse = None; + + match dfg.value_def(value) { + ValueDef::Res(inst, num) => { + // This is an instruction result. See if the value was created by a `concat` + // instruction. + if let InstructionData::Binary { opcode, args, .. } = dfg[inst] { + assert_eq!(num, 0); + if opcode == concat { + reuse = Some((args[0], args[1])); + } + } + } + ValueDef::Arg(_ebb, _num) => {} + } + + // Did the code above succeed in finding values we can reuse? + if let Some(pair) = reuse { + pair + } else { + // No, we'll just have to insert the requested split instruction at `pos`. + match concat { + Opcode::Iconcat => dfg.ins(pos).isplit(value), + Opcode::Vconcat => dfg.ins(pos).vsplit(value), + _ => panic!("Unhandled concat opcode: {}", concat), + } + } +} From a9056f699e7a11cdecfeb7c44c039222bbe77737 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 21 Mar 2017 15:33:23 -0700 Subject: [PATCH 0614/3084] Rename the 'cfg' module to 'flowgraph'. The 'cfg' name was easy to confuse with 'configuration'. --- cranelift/src/filetest/domtree.rs | 8 ++++---- cranelift/src/print_cfg.rs | 6 +++--- cranelift/tests/cfg_traversal.rs | 6 +++--- lib/cretonne/src/context.rs | 2 +- lib/cretonne/src/dominator_tree.rs | 8 ++++---- lib/cretonne/src/{cfg.rs => flowgraph.rs} | 0 lib/cretonne/src/lib.rs | 2 +- lib/cretonne/src/regalloc/context.rs | 4 ++-- lib/cretonne/src/regalloc/liveness.rs | 4 ++-- lib/cretonne/src/verifier.rs | 8 ++++---- 10 files changed, 24 insertions(+), 24 deletions(-) rename lib/cretonne/src/{cfg.rs => flowgraph.rs} (100%) diff --git a/cranelift/src/filetest/domtree.rs b/cranelift/src/filetest/domtree.rs index 84b63177ec..29bde6d13f 100644 --- a/cranelift/src/filetest/domtree.rs +++ b/cranelift/src/filetest/domtree.rs @@ -11,14 +11,14 @@ //! We verify that the dominator tree annotations are complete and correct. //! -use std::collections::HashMap; -use std::borrow::{Borrow, Cow}; +use cretonne::dominator_tree::DominatorTree; +use cretonne::flowgraph::ControlFlowGraph; use cretonne::ir::Function; use cretonne::ir::entities::AnyEntity; -use cretonne::cfg::ControlFlowGraph; -use cretonne::dominator_tree::DominatorTree; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result}; +use std::borrow::{Borrow, Cow}; +use std::collections::HashMap; use utils::match_directive; struct TestDomtree; diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index 206f5430c2..d5b87d806b 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -7,12 +7,12 @@ use std::borrow::Cow; use std::fmt::{Result, Write, Display, Formatter}; use CommandResult; -use utils::read_to_string; -use filetest::subtest::{self, SubTest, Context, Result as STResult}; +use cretonne::flowgraph::ControlFlowGraph; use cretonne::ir::Function; -use cretonne::cfg::ControlFlowGraph; use cretonne::ir::instructions::BranchInfo; use cton_reader::{parse_functions, TestCommand}; +use filetest::subtest::{self, SubTest, Context, Result as STResult}; +use utils::read_to_string; pub fn run(files: Vec) -> CommandResult { for (i, f) in files.into_iter().enumerate() { diff --git a/cranelift/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs index ff5cf6020c..f7fb119463 100644 --- a/cranelift/tests/cfg_traversal.rs +++ b/cranelift/tests/cfg_traversal.rs @@ -1,10 +1,10 @@ extern crate cretonne; extern crate cton_reader; -use self::cton_reader::parse_functions; -use self::cretonne::ir::Ebb; -use self::cretonne::cfg::ControlFlowGraph; use self::cretonne::entity_map::EntityMap; +use self::cretonne::flowgraph::ControlFlowGraph; +use self::cretonne::ir::Ebb; +use self::cton_reader::parse_functions; fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) { let func = &parse_functions(function_source).unwrap()[0]; diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index ce1e8b50ff..a4fc155c41 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -9,8 +9,8 @@ //! contexts concurrently. Typically, you would have one context per compilation thread and only a //! single ISA instance. -use cfg::ControlFlowGraph; use dominator_tree::DominatorTree; +use flowgraph::ControlFlowGraph; use ir::Function; use isa::TargetIsa; use legalize_function; diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index a426fedbea..b014e85ec1 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -1,8 +1,8 @@ //! A Dominator Tree represented as mappings of Ebbs to their immediate dominator. -use cfg::{ControlFlowGraph, BasicBlock}; -use ir::{Ebb, Inst, Function, Layout, ProgramOrder}; use entity_map::EntityMap; +use flowgraph::{ControlFlowGraph, BasicBlock}; +use ir::{Ebb, Inst, Function, Layout, ProgramOrder}; use packed_option::PackedOption; use std::cmp::Ordering; @@ -222,9 +222,9 @@ impl DominatorTree { #[cfg(test)] mod test { - use super::*; + use flowgraph::ControlFlowGraph; use ir::{Function, InstBuilder, Cursor, types}; - use cfg::ControlFlowGraph; + use super::*; #[test] fn empty() { diff --git a/lib/cretonne/src/cfg.rs b/lib/cretonne/src/flowgraph.rs similarity index 100% rename from lib/cretonne/src/cfg.rs rename to lib/cretonne/src/flowgraph.rs diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 733506a605..079b9a4559 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -10,7 +10,7 @@ pub use write::write_function; /// Version number of the cretonne crate. pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); -pub mod cfg; +pub mod flowgraph; pub mod dominator_tree; pub mod entity_list; pub mod entity_map; diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 75863fbc63..195c480372 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -5,12 +5,12 @@ //! avoids allocating data structures independently for each function begin compiled. use dominator_tree::DominatorTree; +use flowgraph::ControlFlowGraph; use ir::Function; +use isa::TargetIsa; use regalloc::coloring::Coloring; use regalloc::live_value_tracker::LiveValueTracker; use regalloc::liveness::Liveness; -use isa::TargetIsa; -use cfg::ControlFlowGraph; /// Persistent memory allocations for register allocation. pub struct Context { diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 989d3bb8b7..0e384017c2 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -175,12 +175,12 @@ //! //! There is some room for improvement. -use cfg::ControlFlowGraph; +use flowgraph::ControlFlowGraph; use ir::dfg::ValueDef; use ir::{Function, Value, Inst, Ebb}; use isa::{TargetIsa, RecipeConstraints}; -use regalloc::liverange::LiveRange; use regalloc::affinity::Affinity; +use regalloc::liverange::LiveRange; use sparse_map::SparseMap; /// A set of live ranges, indexed by value number. diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index dbb57a5b3e..0e59b14e2e 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -53,11 +53,11 @@ //! - Swizzle and shuffle instructions take a variable number of lane arguments. The number //! of arguments must match the destination type, and the lane indexes must be in range. -use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, Value}; -use ir::instructions::{InstructionFormat, BranchInfo}; -use ir::entities::AnyEntity; -use cfg::ControlFlowGraph; use dominator_tree::DominatorTree; +use flowgraph::ControlFlowGraph; +use ir::entities::AnyEntity; +use ir::instructions::{InstructionFormat, BranchInfo}; +use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, Value}; use std::fmt::{self, Display, Formatter}; use std::result; From ca6e402b90941f36ffd7983dd1f8de6f9245ce0b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 21 Mar 2017 15:48:08 -0700 Subject: [PATCH 0615/3084] Add a ControlFlowGraph argument to legalize_function. Legalizing some instructions may require modifications to the control flow graph, and some operations need to use the CFG analysis. The CFG reference is threaded through all the legalization functions to reach the generated expansion functions as well as the legalizer::split module where it will be used first. --- cranelift/src/filetest/legalizer.rs | 4 ++- cranelift/src/filetest/regalloc.rs | 3 +- lib/cretonne/meta/gen_legalizer.py | 9 +++--- lib/cretonne/src/context.rs | 2 +- lib/cretonne/src/legalizer/boundary.rs | 38 +++++++++++++++++--------- lib/cretonne/src/legalizer/mod.rs | 11 ++++---- lib/cretonne/src/legalizer/split.rs | 13 +++++++-- 7 files changed, 52 insertions(+), 28 deletions(-) diff --git a/cranelift/src/filetest/legalizer.rs b/cranelift/src/filetest/legalizer.rs index b7c3345436..25247a5aba 100644 --- a/cranelift/src/filetest/legalizer.rs +++ b/cranelift/src/filetest/legalizer.rs @@ -5,6 +5,7 @@ use std::borrow::Cow; use cretonne::{legalize_function, write_function}; +use cretonne::flowgraph::ControlFlowGraph; use cretonne::ir::Function; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result, run_filecheck}; @@ -36,7 +37,8 @@ impl SubTest for TestLegalizer { fn run(&self, func: Cow, context: &Context) -> Result<()> { let mut func = func.into_owned(); let isa = context.isa.expect("legalizer needs an ISA"); - legalize_function(&mut func, isa); + let mut cfg = ControlFlowGraph::with_function(&func); + legalize_function(&mut func, &mut cfg, isa); let mut text = String::new(); write_function(&mut text, &func, Some(isa)).map_err(|e| e.to_string())?; diff --git a/cranelift/src/filetest/regalloc.rs b/cranelift/src/filetest/regalloc.rs index 90796c1b85..f64d7bb830 100644 --- a/cranelift/src/filetest/regalloc.rs +++ b/cranelift/src/filetest/regalloc.rs @@ -42,10 +42,9 @@ impl SubTest for TestRegalloc { let mut comp_ctx = cretonne::Context::new(); comp_ctx.func = func.into_owned(); + comp_ctx.flowgraph(); // TODO: Should we have an option to skip legalization? comp_ctx.legalize(isa); - - comp_ctx.flowgraph(); comp_ctx.regalloc(isa); let mut text = String::new(); diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 10e9641df7..9131b5d19e 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -140,7 +140,7 @@ def emit_dst_inst(node, fmt): # special functions in the `legalizer::split` module. These functions # will eliminate concat-split patterns. fmt.line( - 'let {} = split::{}(dfg, pos, {});' + 'let {} = split::{}(dfg, cfg, pos, {});' .format( wrap_tup(node.defs), node.expr.inst.snake_name(), @@ -220,9 +220,10 @@ def gen_xform_group(xgrp, fmt): fmt.doc_comment("Legalize the instruction pointed to by `pos`.") fmt.line('#[allow(unused_variables,unused_assignments)]') with fmt.indented( - 'fn ' + xgrp.name + - '(pos: &mut Cursor, dfg: &mut DataFlowGraph) -> bool {', - '}'): + 'fn {}(dfg: &mut DataFlowGraph, ' + 'cfg: &mut ControlFlowGraph, pos: &mut Cursor) -> ' + 'bool {{'.format(xgrp.name), '}'): + # Gen the instruction to be legalized. The cursor we're passed must be # pointing at an instruction. fmt.line('let inst = pos.current_inst().expect("need instruction");') diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index a4fc155c41..9ce9d4912e 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -47,7 +47,7 @@ impl Context { /// Run the legalizer for `isa` on the function. pub fn legalize(&mut self, isa: &TargetIsa) { - legalize_function(&mut self.func, isa); + legalize_function(&mut self.func, &mut self.cfg, isa); } /// Recompute the control flow graph and dominator tree. diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index a53526232a..893e9097b7 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -18,6 +18,7 @@ //! intermediate state doesn't type check. use abi::{legalize_abi_value, ValueConversion}; +use flowgraph::ControlFlowGraph; use ir::{Function, Cursor, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, Signature, SigRef, ArgumentType}; use ir::instructions::CallInfo; @@ -257,6 +258,7 @@ fn convert_from_abi(dfg: &mut DataFlowGraph, /// return the `Err(ArgumentType)` that is needed. /// fn convert_to_abi(dfg: &mut DataFlowGraph, + cfg: &ControlFlowGraph, pos: &mut Cursor, value: Value, put_arg: &mut PutArg) @@ -272,28 +274,28 @@ fn convert_to_abi(dfg: &mut DataFlowGraph, let ty = dfg.value_type(value); match legalize_abi_value(ty, &arg_type) { ValueConversion::IntSplit => { - let (lo, hi) = isplit(dfg, pos, value); - convert_to_abi(dfg, pos, lo, put_arg); - convert_to_abi(dfg, pos, hi, put_arg); + let (lo, hi) = isplit(dfg, cfg, pos, value); + convert_to_abi(dfg, cfg, pos, lo, put_arg); + convert_to_abi(dfg, cfg, pos, hi, put_arg); } ValueConversion::VectorSplit => { - let (lo, hi) = vsplit(dfg, pos, value); - convert_to_abi(dfg, pos, lo, put_arg); - convert_to_abi(dfg, pos, hi, put_arg); + let (lo, hi) = vsplit(dfg, cfg, pos, value); + convert_to_abi(dfg, cfg, pos, lo, put_arg); + convert_to_abi(dfg, cfg, pos, hi, put_arg); } ValueConversion::IntBits => { assert!(!ty.is_int()); let abi_ty = Type::int(ty.bits()).expect("Invalid type for conversion"); let arg = dfg.ins(pos).bitcast(abi_ty, value); - convert_to_abi(dfg, pos, arg, put_arg); + convert_to_abi(dfg, cfg, pos, arg, put_arg); } ValueConversion::Sext(abi_ty) => { let arg = dfg.ins(pos).sextend(abi_ty, value); - convert_to_abi(dfg, pos, arg, put_arg); + convert_to_abi(dfg, cfg, pos, arg, put_arg); } ValueConversion::Uext(abi_ty) => { let arg = dfg.ins(pos).uextend(abi_ty, value); - convert_to_abi(dfg, pos, arg, put_arg); + convert_to_abi(dfg, cfg, pos, arg, put_arg); } } } @@ -361,6 +363,7 @@ fn check_return_signature(dfg: &DataFlowGraph, inst: Inst, sig: &Signature) -> b /// argument number in `0..abi_args`. /// fn legalize_inst_arguments(dfg: &mut DataFlowGraph, + cfg: &ControlFlowGraph, pos: &mut Cursor, abi_args: usize, mut get_abi_type: ArgType) @@ -419,7 +422,7 @@ fn legalize_inst_arguments(dfg: &mut DataFlowGraph, Err(abi_type) } }; - convert_to_abi(dfg, pos, old_value, &mut put_arg); + convert_to_abi(dfg, cfg, pos, old_value, &mut put_arg); } // Put the modified value list back. @@ -436,7 +439,7 @@ fn legalize_inst_arguments(dfg: &mut DataFlowGraph, /// original return values. The call's result values will be adapted to match the new signature. /// /// Returns `true` if any instructions were inserted. -pub fn handle_call_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor) -> bool { +pub fn handle_call_abi(dfg: &mut DataFlowGraph, cfg: &ControlFlowGraph, pos: &mut Cursor) -> bool { let mut inst = pos.current_inst().expect("Cursor must point to a call instruction"); // Start by checking if the argument types already match the signature. @@ -448,6 +451,7 @@ pub fn handle_call_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor) -> bool { // OK, we need to fix the call arguments to match the ABI signature. let abi_args = dfg.signatures[sig_ref].argument_types.len(); legalize_inst_arguments(dfg, + cfg, pos, abi_args, |dfg, abi_arg| dfg.signatures[sig_ref].argument_types[abi_arg]); @@ -471,7 +475,11 @@ pub fn handle_call_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor) -> bool { /// Insert ABI conversion code before and after the call instruction at `pos`. /// /// Return `true` if any instructions were inserted. -pub fn handle_return_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor, sig: &Signature) -> bool { +pub fn handle_return_abi(dfg: &mut DataFlowGraph, + cfg: &ControlFlowGraph, + pos: &mut Cursor, + sig: &Signature) + -> bool { let inst = pos.current_inst().expect("Cursor must point to a return instruction"); // Check if the returned types already match the signature. @@ -480,7 +488,11 @@ pub fn handle_return_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor, sig: &Signat } let abi_args = sig.return_types.len(); - legalize_inst_arguments(dfg, pos, abi_args, |_, abi_arg| sig.return_types[abi_arg]); + legalize_inst_arguments(dfg, + cfg, + pos, + abi_args, + |_, abi_arg| sig.return_types[abi_arg]); debug_assert!(check_return_signature(dfg, inst, sig), "Signature still wrong: {}, sig{}", diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 56e3e10dd6..deb0bbc6b1 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -13,6 +13,7 @@ //! The legalizer does not deal with register allocation constraints. These constraints are derived //! from the encoding recipes, and solved later by the register allocator. +use flowgraph::ControlFlowGraph; use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, InstBuilder}; use ir::condcodes::IntCC; use isa::{TargetIsa, Legalize}; @@ -25,7 +26,7 @@ mod split; /// - Transform any instructions that don't have a legal representation in `isa`. /// - Fill out `func.encodings`. /// -pub fn legalize_function(func: &mut Function, isa: &TargetIsa) { +pub fn legalize_function(func: &mut Function, cfg: &mut ControlFlowGraph, isa: &TargetIsa) { boundary::legalize_signatures(func, isa); // TODO: This is very simplified and incomplete. @@ -40,14 +41,14 @@ pub fn legalize_function(func: &mut Function, isa: &TargetIsa) { let opcode = func.dfg[inst].opcode(); // Check for ABI boundaries that need to be converted to the legalized signature. - if opcode.is_call() && boundary::handle_call_abi(&mut func.dfg, &mut pos) { + if opcode.is_call() && boundary::handle_call_abi(&mut func.dfg, cfg, &mut pos) { // Go back and legalize the inserted argument conversion instructions. pos.set_position(prev_pos); continue; } if opcode.is_return() && - boundary::handle_return_abi(&mut func.dfg, &mut pos, &func.signature) { + boundary::handle_return_abi(&mut func.dfg, cfg, &mut pos, &func.signature) { // Go back and legalize the inserted return value conversion instructions. pos.set_position(prev_pos); continue; @@ -70,8 +71,8 @@ pub fn legalize_function(func: &mut Function, isa: &TargetIsa) { // 4. TODO: Convert to library calls. For example, floating point operations on // an ISA with no IEEE 754 support. let changed = match action { - Legalize::Expand => expand(&mut pos, &mut func.dfg), - Legalize::Narrow => narrow(&mut pos, &mut func.dfg), + Legalize::Expand => expand(&mut func.dfg, cfg, &mut pos), + Legalize::Narrow => narrow(&mut func.dfg, cfg, &mut pos), }; // If the current instruction was replaced, we need to double back and revisit // the expanded sequence. This is both to assign encodings and possible to diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index b8323a59fb..7b36c4997e 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -64,17 +64,26 @@ //! It is possible to have circular dependencies of EBB arguments that are never used by any real //! instructions. These loops will remain in the program. +use flowgraph::ControlFlowGraph; use ir::{DataFlowGraph, Cursor, Value, Opcode, ValueDef, InstructionData, InstBuilder}; /// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values /// if possible. -pub fn isplit(dfg: &mut DataFlowGraph, pos: &mut Cursor, value: Value) -> (Value, Value) { +pub fn isplit(dfg: &mut DataFlowGraph, + _cfg: &ControlFlowGraph, + pos: &mut Cursor, + value: Value) + -> (Value, Value) { split_value(dfg, pos, value, Opcode::Iconcat) } /// Split `value` into halves using the `vsplit` semantics. Do this by reusing existing values if /// possible. -pub fn vsplit(dfg: &mut DataFlowGraph, pos: &mut Cursor, value: Value) -> (Value, Value) { +pub fn vsplit(dfg: &mut DataFlowGraph, + _cfg: &ControlFlowGraph, + pos: &mut Cursor, + value: Value) + -> (Value, Value) { split_value(dfg, pos, value, Opcode::Vconcat) } From d32c19c81da3da9e99c6dc1eddf1efde9338f9e4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 22 Mar 2017 10:36:42 -0700 Subject: [PATCH 0616/3084] Add a DataFlowGraph::replace_ebb_arg() method. This is needed for rewriting EBB arguments with legalized types. --- lib/cretonne/src/ir/dfg.rs | 93 +++++++++++++++++++++++++++++++++++++- 1 file changed, 92 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 8977cc6751..e86b87de32 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -261,7 +261,7 @@ pub enum ValueDef { } // Internal table storage for extended values. -#[derive(Clone)] +#[derive(Clone, Debug)] enum ValueData { // Value is defined by an instruction, but it is not the first result. Inst { @@ -663,6 +663,67 @@ impl DataFlowGraph { } } + /// Replace an EBB argument with a new value of type `ty`. + /// + /// The `old_value` must be an attached EBB argument. It is removed from its place in the list + /// of arguments and replaced by a new value of type `new_type`. The new value gets the same + /// position in the list, and other arguments are not disturbed. + /// + /// The old value is left detached, so it should probably be changed into something else. + /// + /// Returns the new value. + pub fn replace_ebb_arg(&mut self, old_arg: Value, new_type: Type) -> Value { + let old_data = if let ExpandedValue::Table(index) = old_arg.expand() { + self.extended_values[index].clone() + } else { + panic!("old_arg: {} must be an EBB argument", old_arg); + }; + + // Create new value identical to the old one except for the type. + let (ebb, num, new_arg) = if let ValueData::Arg { num, ebb, next, .. } = old_data { + (ebb, + num, + self.make_value(ValueData::Arg { + ty: new_type, + num: num, + ebb: ebb, + next: next, + })) + } else { + panic!("old_arg: {} must be an EBB argument: {:?}", + old_arg, + old_data); + }; + + // Now fix up the linked lists. + if self.ebbs[ebb].last_arg.expand() == Some(old_arg) { + self.ebbs[ebb].last_arg = new_arg.into(); + } + + if self.ebbs[ebb].first_arg.expand() == Some(old_arg) { + assert_eq!(num, 0); + self.ebbs[ebb].first_arg = new_arg.into(); + } else { + // We need to find the num-1 argument value and change its next link. + let mut arg = self.ebbs[ebb].first_arg.expect("EBB has no arguments"); + for _ in 1..num { + arg = self.next_ebb_arg(arg).expect("Too few EBB arguments"); + } + if let ExpandedValue::Table(index) = arg.expand() { + if let ValueData::Arg { ref mut next, .. } = self.extended_values[index] { + assert_eq!(next.expand(), Some(old_arg)); + *next = new_arg.into(); + } else { + panic!("{} is not an EBB argument", arg); + } + } else { + panic!("{} is not an EBB argument", arg); + } + } + + new_arg + } + /// Given a value that is known to be an EBB argument, return the next EBB argument. pub fn next_ebb_arg(&self, arg: Value) -> Option { if let ExpandedValue::Table(index) = arg.expand() { @@ -886,6 +947,36 @@ mod tests { assert_eq!(dfg.ebb_args(ebb).collect::>(), [take2, arg3, take1]); } + #[test] + fn replace_ebb_arguments() { + let mut dfg = DataFlowGraph::new(); + + let ebb = dfg.make_ebb(); + let arg1 = dfg.append_ebb_arg(ebb, types::F32); + + let new1 = dfg.replace_ebb_arg(arg1, types::I64); + assert_eq!(dfg.value_type(arg1), types::F32); + assert_eq!(dfg.value_type(new1), types::I64); + assert_eq!(dfg.ebb_args(ebb).collect::>(), [new1]); + + dfg.attach_ebb_arg(ebb, arg1); + assert_eq!(dfg.ebb_args(ebb).collect::>(), [new1, arg1]); + + let new2 = dfg.replace_ebb_arg(arg1, types::I8); + assert_eq!(dfg.value_type(arg1), types::F32); + assert_eq!(dfg.value_type(new2), types::I8); + assert_eq!(dfg.ebb_args(ebb).collect::>(), [new1, new2]); + + dfg.attach_ebb_arg(ebb, arg1); + assert_eq!(dfg.ebb_args(ebb).collect::>(), [new1, new2, arg1]); + + let new3 = dfg.replace_ebb_arg(new2, types::I16); + assert_eq!(dfg.value_type(new1), types::I64); + assert_eq!(dfg.value_type(new2), types::I8); + assert_eq!(dfg.value_type(new3), types::I16); + assert_eq!(dfg.ebb_args(ebb).collect::>(), [new1, new3, arg1]); + } + #[test] fn aliases() { use ir::InstBuilder; From 272df6489c989e79f39ae21f65d7f9871019fb58 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 21 Mar 2017 14:38:42 -0700 Subject: [PATCH 0617/3084] Iteratively split EBB arguments. When the legalizer splits a value into halves, it would previously stop if the value was an EBB argument. With this change, we also split EBB arguments and iteratively split arguments on branches to the EBB. The iterative splitting stops when we hit the entry block arguments or an instruction that isn't one of the concatenation instructions. --- cranelift/filetests/isa/riscv/split-args.cton | 40 +++++ lib/cretonne/meta/base/instructions.py | 2 +- lib/cretonne/src/ir/layout.rs | 3 +- lib/cretonne/src/legalizer/split.rs | 152 +++++++++++++++++- 4 files changed, 187 insertions(+), 10 deletions(-) create mode 100644 cranelift/filetests/isa/riscv/split-args.cton diff --git a/cranelift/filetests/isa/riscv/split-args.cton b/cranelift/filetests/isa/riscv/split-args.cton new file mode 100644 index 0000000000..1cecb85c10 --- /dev/null +++ b/cranelift/filetests/isa/riscv/split-args.cton @@ -0,0 +1,40 @@ +; Test the legalization of EBB arguments that are split. +test legalizer +isa riscv + +; regex: V=vx?\d+ + +function simple(i64, i64) -> i64 { +ebb0(v1: i64, v2: i64): +; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32): + jump ebb1(v1) + ; check: jump $ebb1($v1l, $v1h) + +ebb1(v3: i64): +; check: $ebb1($(v3l=$V): i32, $(v3h=$V): i32): + v4 = band v3, v2 + ; check: $(v4l=$V) = band $v3l, $v2l + ; check: $(v4h=$V) = band $v3h, $v2h + return v4 + ; check: return $v4l, $v4h +} + +function multi(i64) -> i64 { +ebb1(v1: i64): +; check: $ebb1($(v1l=$V): i32, $(v1h=$V): i32): + jump ebb2(v1, v1) + ; check: jump $ebb2($v1l, $v1l, $v1h, $v1h) + +ebb2(v2: i64, v3: i64): +; check: $ebb2($(v2l=$V): i32, $(v3l=$V): i32, $(v2h=$V): i32, $(v3h=$V): i32): + jump ebb3(v2) + ; check: jump $ebb3($v2l, $v2h) + +ebb3(v4: i64): +; check: $ebb3($(v4l=$V): i32, $(v4h=$V): i32): + v5 = band v4, v3 + ; check: $(v5l=$V) = band $v4l, $v3l + ; check: $(v5h=$V) = band $v4h, $v3h + return v5 + ; check: return $v5l, $v5h +} diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 66dc0e9090..ade9b540bd 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -44,7 +44,7 @@ jump = Instruction( EBB arguments. The number and types of arguments must match the destination EBB. """, - ins=(EBB, args), is_terminator=True) + ins=(EBB, args), is_branch=True, is_terminator=True) brz = Instruction( 'brz', r""" diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 83edbea76f..c34d6a0aba 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -588,7 +588,8 @@ impl<'f> DoubleEndedIterator for Insts<'f> { /// When new instructions are added, the cursor can either append them to an EBB or insert them /// before the current instruction. pub struct Cursor<'f> { - layout: &'f mut Layout, + /// Borrowed function layout. Public so it can be re-borrowed from this cursor. + pub layout: &'f mut Layout, pos: CursorPosition, } diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index 7b36c4997e..d58c7ed285 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -65,26 +65,103 @@ //! instructions. These loops will remain in the program. use flowgraph::ControlFlowGraph; -use ir::{DataFlowGraph, Cursor, Value, Opcode, ValueDef, InstructionData, InstBuilder}; +use ir::{DataFlowGraph, Ebb, Cursor, Value, Type, Opcode, ValueDef, InstructionData, InstBuilder}; +use std::iter; /// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values /// if possible. pub fn isplit(dfg: &mut DataFlowGraph, - _cfg: &ControlFlowGraph, + cfg: &ControlFlowGraph, pos: &mut Cursor, value: Value) -> (Value, Value) { - split_value(dfg, pos, value, Opcode::Iconcat) + split_any(dfg, cfg, pos, value, Opcode::Iconcat) } /// Split `value` into halves using the `vsplit` semantics. Do this by reusing existing values if /// possible. pub fn vsplit(dfg: &mut DataFlowGraph, - _cfg: &ControlFlowGraph, + cfg: &ControlFlowGraph, pos: &mut Cursor, value: Value) -> (Value, Value) { - split_value(dfg, pos, value, Opcode::Vconcat) + split_any(dfg, cfg, pos, value, Opcode::Vconcat) +} + +/// After splitting an EBB argument, we need to go back and fix up all of the predecessor +/// instructions. This is potentially a recursive operation, but we don't implement it recursively +/// since that could use up too muck stack. +/// +/// Instead, the repairs are deferred and placed on a work list in stack form. +struct Repair { + concat: Opcode, + // The argument type after splitting. + split_type: Type, + // The destination EBB whose arguments have been split. + ebb: Ebb, + // Number of the original EBB argument which has been replaced by the low part. + num: usize, + // Number of the new EBB argument which represents the high part after the split. + hi_num: usize, +} + +/// Generic version of `isplit` and `vsplit` controlled by the `concat` opcode. +fn split_any(dfg: &mut DataFlowGraph, + cfg: &ControlFlowGraph, + pos: &mut Cursor, + value: Value, + concat: Opcode) + -> (Value, Value) { + let saved_pos = pos.position(); + let mut repairs = Vec::new(); + let result = split_value(dfg, pos, value, concat, &mut repairs); + + // We have split the value requested, and now we may need to fix some EBB predecessors. + while let Some(repair) = repairs.pop() { + for &(_, inst) in cfg.get_predecessors(repair.ebb) { + let branch_opc = dfg[inst].opcode(); + assert!(branch_opc.is_branch(), + "Predecessor not a branch: {}", + dfg.display_inst(inst)); + let fixed_args = branch_opc.constraints().fixed_value_arguments(); + let mut args = dfg[inst].take_value_list().expect("Branches must have value lists."); + let num_args = args.len(&dfg.value_lists); + // Get the old value passed to the EBB argument we're repairing. + let old_arg = args.get(fixed_args + repair.num, &dfg.value_lists) + .expect("Too few branch arguments"); + + // It's possible that the CFG's predecessor list has duplicates. Detect them here. + if dfg.value_type(old_arg) == repair.split_type { + dfg[inst].put_value_list(args); + continue; + } + + // Split the old argument, possibly causing more repairs to be scheduled. + pos.goto_inst(inst); + let (lo, hi) = split_value(dfg, pos, old_arg, repair.concat, &mut repairs); + + // The `lo` part replaces the original argument. + *args.get_mut(fixed_args + repair.num, &mut dfg.value_lists).unwrap() = lo; + + // The `hi` part goes at the end. Since multiple repairs may have been scheduled to the + // same EBB, there could be multiple arguments missing. + if num_args > fixed_args + repair.hi_num { + *args.get_mut(fixed_args + repair.hi_num, &mut dfg.value_lists).unwrap() = hi; + } else { + // We need to append one or more arguments. If we're adding more than one argument, + // there must be pending repairs on the stack that will fill in the correct values + // instead of `hi`. + args.extend(iter::repeat(hi).take(1 + fixed_args + repair.hi_num - num_args), + &mut dfg.value_lists); + } + + // Put the value list back after manipulating it. + dfg[inst].put_value_list(args); + } + } + + pos.set_position(saved_pos); + result } /// Split a single value using the integer or vector semantics given by the `concat` opcode. @@ -96,7 +173,8 @@ pub fn vsplit(dfg: &mut DataFlowGraph, fn split_value(dfg: &mut DataFlowGraph, pos: &mut Cursor, value: Value, - concat: Opcode) + concat: Opcode, + repairs: &mut Vec) -> (Value, Value) { let value = dfg.resolve_copies(value); let mut reuse = None; @@ -112,14 +190,56 @@ fn split_value(dfg: &mut DataFlowGraph, } } } - ValueDef::Arg(_ebb, _num) => {} + ValueDef::Arg(ebb, num) => { + // This is an EBB argument. We can split the argument value unless this is the entry + // block. + if pos.layout.entry_block() != Some(ebb) { + // We are going to replace the argument at `num` with two new arguments. + // Determine the new value types. + let ty = dfg.value_type(value); + let split_type = match concat { + Opcode::Iconcat => ty.half_width().expect("Invalid type for isplit"), + Opcode::Vconcat => ty.half_vector().expect("Invalid type for vsplit"), + _ => panic!("Unhandled concat opcode: {}", concat), + }; + + // Since the `repairs` stack potentially contains other argument numbers for `ebb`, + // avoid shifting and renumbering EBB arguments. It could invalidate other + // `repairs` entries. + // + // Replace the original `value` with the low part, and append the high part at the + // end of the argument list. + let lo = dfg.replace_ebb_arg(value, split_type); + let hi_num = dfg.num_ebb_args(ebb); + let hi = dfg.append_ebb_arg(ebb, split_type); + reuse = Some((lo, hi)); + + + // Now the original value is dangling. Insert a concatenation instruction that can + // compute it from the two new arguments. This also serves as a record of what we + // did so a future call to this function doesn't have to redo the work. + // + // Note that it is safe to move `pos` here since `reuse` was set above, so we don't + // need to insert a split instruction before returning. + pos.goto_top(ebb); + pos.next_inst(); + let concat_inst = dfg.ins(pos).Binary(concat, ty, lo, hi).0; + let concat_val = dfg.first_result(concat_inst); + dfg.change_to_alias(value, concat_val); + + // Finally, splitting the EBB argument is not enough. We also have to repair all + // of the predecessor instructions that branch here. + add_repair(concat, split_type, ebb, num, hi_num, repairs); + } + } } // Did the code above succeed in finding values we can reuse? if let Some(pair) = reuse { pair } else { - // No, we'll just have to insert the requested split instruction at `pos`. + // No, we'll just have to insert the requested split instruction at `pos`. Note that `pos` + // has not been moved by the EBB argument code above when `reuse` is `None`. match concat { Opcode::Iconcat => dfg.ins(pos).isplit(value), Opcode::Vconcat => dfg.ins(pos).vsplit(value), @@ -127,3 +247,19 @@ fn split_value(dfg: &mut DataFlowGraph, } } } + +// Add a repair entry to the work list. +fn add_repair(concat: Opcode, + split_type: Type, + ebb: Ebb, + num: usize, + hi_num: usize, + repairs: &mut Vec) { + repairs.push(Repair { + concat: concat, + split_type: split_type, + ebb: ebb, + num: num, + hi_num: hi_num, + }); +} From 26ee382b57b37ebc331bcb85543441209de36637 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 22 Mar 2017 13:17:25 -0700 Subject: [PATCH 0618/3084] Fix weird indentation. --- lib/cretonne/src/ir/dfg.rs | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index e86b87de32..ae5e10f6ab 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -178,24 +178,24 @@ impl DataFlowGraph { /// copy/spill/fill instructions. pub fn resolve_copies(&self, value: Value) -> Value { use ir::entities::ExpandedValue::Direct; - let mut v = self.resolve_aliases(value); + let mut v = value; for _ in 0..self.insts.len() { - v = self.resolve_aliases(match v.expand() { - Direct(inst) => { - match self[inst] { - InstructionData::Unary { opcode, arg, .. } => { - match opcode { - Opcode::Copy | Opcode::Spill | - Opcode::Fill => arg, - _ => return v, - } - } - _ => return v, - } - } - _ => return v, - }); + v = self.resolve_aliases(v); + v = match v.expand() { + Direct(inst) => { + match self[inst] { + InstructionData::Unary { opcode, arg, .. } => { + match opcode { + Opcode::Copy | Opcode::Spill | Opcode::Fill => arg, + _ => return v, + } + } + _ => return v, + } + } + _ => return v, + }; } panic!("Copy loop detected for {}", value); } From c7e25e6a351738ab7dadf23921076fa484f6e3f2 Mon Sep 17 00:00:00 2001 From: Keith Yeung Date: Wed, 22 Mar 2017 06:02:07 +0800 Subject: [PATCH 0619/3084] Limit type inference for controlling type variables in write.rs --- cranelift/filetests/parser/branch.cton | 4 ++-- lib/cretonne/src/write.rs | 21 ++++++++++++--------- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/cranelift/filetests/parser/branch.cton b/cranelift/filetests/parser/branch.cton index 46612973af..1aa0ca185a 100644 --- a/cranelift/filetests/parser/branch.cton +++ b/cranelift/filetests/parser/branch.cton @@ -62,7 +62,7 @@ ebb1: ; nextln: brz vx0, ebb1 ; nextln: ; nextln: ebb1: -; nextln: brnz vx0, ebb1 +; nextln: brnz.i32 vx0, ebb1 ; nextln: } function twoargs(i32, f32) { @@ -77,7 +77,7 @@ ebb1(vx2: i32, vx3: f32): ; nextln: brz vx0, ebb1(vx0, vx1) ; nextln: ; nextln: ebb1(vx2: i32, vx3: f32): -; nextln: brnz vx0, ebb0(vx2, vx3) +; nextln: brnz.i32 vx0, ebb0(vx2, vx3) ; nextln: } function jumptable(i32) { diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index dfa7688f20..5ff48ec91c 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -4,7 +4,7 @@ //! equivalent textual representation. This textual representation can be read back by the //! `cretonne-reader` crate. -use ir::{Function, DataFlowGraph, Ebb, Inst, Value, Type}; +use ir::{Function, DataFlowGraph, Ebb, Inst, Value, ValueDef, Type}; use isa::{TargetIsa, RegInfo}; use std::fmt::{self, Result, Error, Write}; use std::result; @@ -132,7 +132,8 @@ pub fn write_ebb(w: &mut Write, func: &Function, isa: Option<&TargetIsa>, ebb: E // if it can't be trivially inferred. // fn type_suffix(func: &Function, inst: Inst) -> Option { - let constraints = func.dfg[inst].opcode().constraints(); + let inst_data = &func.dfg[inst]; + let constraints = inst_data.opcode().constraints(); if !constraints.is_polymorphic() { return None; @@ -140,16 +141,18 @@ fn type_suffix(func: &Function, inst: Inst) -> Option { // If the controlling type variable can be inferred from the type of the designated value input // operand, we don't need the type suffix. - // TODO: Should we include the suffix when the input value is defined in another block? The - // parser needs to know the type of the value, so it must be defined in a block that lexically - // comes before this one. if constraints.use_typevar_operand() { - return None; + let ctrl_var = inst_data.typevar_operand(&func.dfg.value_lists).unwrap(); + let def_ebb = match func.dfg.value_def(ctrl_var) { + ValueDef::Res(instr, _) => func.layout.inst_ebb(instr), + ValueDef::Arg(ebb, _) => Some(ebb), + }; + if def_ebb.is_some() && def_ebb == func.layout.inst_ebb(inst) { + return None; + } } - // This polymorphic instruction doesn't support basic type inference. - // The controlling type variable is required to be the type of the first result. - let rtype = func.dfg.value_type(func.dfg.first_result(inst)); + let rtype = inst_data.ctrl_typevar(&func.dfg); assert!(!rtype.is_void(), "Polymorphic instruction must produce a result"); Some(rtype) From d9f325f6cd81b7b356b89aef278f9a84d5c1d304 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 22 Mar 2017 15:26:06 -0700 Subject: [PATCH 0620/3084] Simplify branch arguments after splitting EBB arguments. The EBB argument splitting may generate concat-split dependencies when it repairs branch arguments in EBBs that have not yet been fully legalized. Add a branch argument simplification step that can resolve these dependency chains. This means that all split and concatenation instructions will be dead after legalization for types that have no legal instructions using them. --- .../filetests/isa/riscv/legalize-abi.cton | 10 +-- cranelift/filetests/isa/riscv/split-args.cton | 15 +++++ lib/cretonne/src/legalizer/mod.rs | 4 ++ lib/cretonne/src/legalizer/split.rs | 62 ++++++++++++++++++- 4 files changed, 86 insertions(+), 5 deletions(-) diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index 926a3c2820..06fcfd5f6c 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -36,7 +36,8 @@ ebb0: ; check: $(v1new=$V) = iconcat $v1l, $v1h ; check: $v1 = copy $v1new jump ebb1(v1) - ; check: jump $ebb1($v1) + ; The v1 copy gets resolved by split::simplify_branch_arguments(). + ; check: jump $ebb1($v1new) ebb1(v10: i64): jump ebb1(v10) @@ -50,9 +51,9 @@ ebb0: ; check: $ebb0: ; nextln: $v1, $(v2l=$V), $(v2h=$V) = call $fn1() ; check: $(v2new=$V) = iconcat $v2l, $v2h - ; check: $v2 -> $v2new jump ebb1(v1, v2) - ; check: jump $ebb1($v1, $v2) + ; The v2 -> v2new alias is resolved by split::simplify_branch_arguments(). + ; check: jump $ebb1($v1, $v2new) ebb1(v9: i32, v10: i64): jump ebb1(v9, v10) @@ -78,7 +79,8 @@ ebb0: ; check: $(v1new=$V) = ireduce.i8 $rv ; check: $v1 = copy $v1new jump ebb1(v1) - ; check: jump $ebb1($v1) + ; The v1 copy gets resolved by split::simplify_branch_arguments(). + ; check: jump $ebb1($v1new) ebb1(v10: i8): jump ebb1(v10) diff --git a/cranelift/filetests/isa/riscv/split-args.cton b/cranelift/filetests/isa/riscv/split-args.cton index 1cecb85c10..8c69378f42 100644 --- a/cranelift/filetests/isa/riscv/split-args.cton +++ b/cranelift/filetests/isa/riscv/split-args.cton @@ -38,3 +38,18 @@ ebb3(v4: i64): return v5 ; check: return $v5l, $v5h } + +function loop(i64, i64) -> i64 { +ebb0(v1: i64, v2: i64): +; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32): + jump ebb1(v1) + ; check: jump $ebb1($v1l, $v1h) + +ebb1(v3: i64): +; check: $ebb1($(v3l=$V): i32, $(v3h=$V): i32): + v4 = band v3, v2 + ; check: $(v4l=$V) = band $v3l, $v2l + ; check: $(v4h=$V) = band $v3h, $v2h + jump ebb1(v4) + ; check: jump $ebb1($v4l, $v4h) +} diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index deb0bbc6b1..b9dc493527 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -54,6 +54,10 @@ pub fn legalize_function(func: &mut Function, cfg: &mut ControlFlowGraph, isa: & continue; } + if opcode.is_branch() { + split::simplify_branch_arguments(&mut func.dfg, inst); + } + match isa.encode(&func.dfg, &func.dfg[inst]) { Ok(encoding) => *func.encodings.ensure(inst) = encoding, Err(action) => { diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index d58c7ed285..43a2c4d385 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -65,7 +65,8 @@ //! instructions. These loops will remain in the program. use flowgraph::ControlFlowGraph; -use ir::{DataFlowGraph, Ebb, Cursor, Value, Type, Opcode, ValueDef, InstructionData, InstBuilder}; +use ir::{DataFlowGraph, Ebb, Inst, Cursor, Value, Type, Opcode, ValueDef, InstructionData, + InstBuilder}; use std::iter; /// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values @@ -263,3 +264,62 @@ fn add_repair(concat: Opcode, hi_num: hi_num, }); } + +/// Strip concat-split chains. Return a simpler way of computing the same value. +/// +/// Given this input: +/// +/// ```cton +/// v10 = iconcat v1, v2 +/// v11, v12 = isplit v10 +/// ``` +/// +/// This function resolves `v11` to `v1` and `v12` to `v2`. +fn resolve_splits(dfg: &DataFlowGraph, value: Value) -> Value { + let value = dfg.resolve_copies(value); + + // Deconstruct a split instruction. + let split_res; + let concat_opc; + let split_arg; + if let ValueDef::Res(inst, num) = dfg.value_def(value) { + split_res = num; + concat_opc = match dfg[inst].opcode() { + Opcode::Isplit => Opcode::Iconcat, + Opcode::Vsplit => Opcode::Vconcat, + _ => return value, + }; + split_arg = dfg[inst].arguments(&dfg.value_lists)[0]; + } else { + return value; + } + + // See if split_arg is defined by a concatenation instruction. + if let ValueDef::Res(inst, _) = dfg.value_def(split_arg) { + if dfg[inst].opcode() == concat_opc { + return dfg[inst].arguments(&dfg.value_lists)[split_res]; + } + } + + value +} + +/// Simplify the arguments to a branch *after* the instructions leading up to the branch have been +/// legalized. +/// +/// The branch argument repairs performed by `split_any()` above may be performed on branches that +/// have not yet been legalized. The repaired arguments can be defined by actual split +/// instructions in that case. +/// +/// After legalizing the instructions computing the value that was split, it is likely that we can +/// avoid depending on the split instruction. Its input probably comes from a concatenation. +pub fn simplify_branch_arguments(dfg: &mut DataFlowGraph, branch: Inst) { + let mut new_args = Vec::new(); + + for &arg in dfg[branch].arguments(&dfg.value_lists) { + let new_arg = resolve_splits(dfg, arg); + new_args.push(new_arg); + } + + dfg.insts[branch].arguments_mut(&mut dfg.value_lists).copy_from_slice(&new_args); +} From cc5c4932ca9aaf3cc8793b048734d8bca60ba6a1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 23 Mar 2017 09:42:36 -0700 Subject: [PATCH 0621/3084] Remove spurious shell redirections from install commands. --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 8bd8dd13e6..6f343fb258 100644 --- a/README.rst +++ b/README.rst @@ -65,7 +65,7 @@ Building the documentation To build the Cretonne documentation, you need the `Sphinx documentation generator `_:: - $ pip install sphinx>=1.4 sphinx-autobuild sphinx_rtd_theme + $ pip install sphinx sphinx-autobuild sphinx_rtd_theme $ cd cretonne/docs $ make html $ open _build/html/index.html From 6fdbbf2b1af26c465d779de093d9e7c341e7a01e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 23 Mar 2017 09:50:26 -0700 Subject: [PATCH 0622/3084] Legalize EBBs in a reverse post-order. This means that whenever we need to split a value, it is either already defined by a concatenation instruction in a previously processed EBB, or it's an EBB argument. --- lib/cretonne/src/legalizer/mod.rs | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index b9dc493527..00354822a1 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -29,10 +29,16 @@ mod split; pub fn legalize_function(func: &mut Function, cfg: &mut ControlFlowGraph, isa: &TargetIsa) { boundary::legalize_signatures(func, isa); - // TODO: This is very simplified and incomplete. func.encodings.resize(func.dfg.num_insts()); + + // Process EBBs in a reverse post-order. This minimizes the number of split instructions we + // need. + let mut postorder = cfg.postorder_ebbs(); let mut pos = Cursor::new(&mut func.layout); - while let Some(_ebb) = pos.next_ebb() { + + while let Some(ebb) = postorder.pop() { + pos.goto_top(ebb); + // Keep track of the cursor position before the instruction being processed, so we can // double back when replacing instructions. let mut prev_pos = pos.position(); From 5027e096b5c41452607aae834ddeab8da635fe7f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 23 Mar 2017 12:39:05 -0700 Subject: [PATCH 0623/3084] Add dfg.inst_args(_mut) methods. A shortcut for calling arguments() directly that goes with the existing inst_results() method. --- lib/cretonne/src/ir/dfg.rs | 10 ++++++++++ lib/cretonne/src/legalizer/boundary.rs | 3 +-- lib/cretonne/src/legalizer/split.rs | 8 ++++---- lib/cretonne/src/regalloc/coloring.rs | 2 +- lib/cretonne/src/regalloc/liveness.rs | 2 +- lib/cretonne/src/verifier.rs | 2 +- lib/cretonne/src/write.rs | 2 +- 7 files changed, 19 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index ae5e10f6ab..24727362b8 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -344,6 +344,16 @@ impl DataFlowGraph { DisplayInst(self, inst) } + /// Get the value arguments on `inst` as a slice. + pub fn inst_args(&self, inst: Inst) -> &[Value] { + self.insts[inst].arguments(&self.value_lists) + } + + /// Get the value arguments on `inst` as a mutable slice. + pub fn inst_args_mut(&mut self, inst: Inst) -> &mut [Value] { + self.insts[inst].arguments_mut(&mut self.value_lists) + } + /// Create result values for an instruction that produces multiple results. /// /// Instructions that produce 0 or 1 result values only need to be created with `make_inst`. If diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 893e9097b7..8ec8ffeb9c 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -348,8 +348,7 @@ fn check_call_signature(dfg: &DataFlowGraph, inst: Inst) -> Result<(), SigRef> { fn check_return_signature(dfg: &DataFlowGraph, inst: Inst, sig: &Signature) -> bool { let fixed_values = dfg[inst].opcode().constraints().fixed_value_arguments(); check_arg_types(dfg, - dfg[inst] - .arguments(&dfg.value_lists) + dfg.inst_args(inst) .iter() .skip(fixed_values) .cloned(), diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index 43a2c4d385..3dda54abe7 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -289,7 +289,7 @@ fn resolve_splits(dfg: &DataFlowGraph, value: Value) -> Value { Opcode::Vsplit => Opcode::Vconcat, _ => return value, }; - split_arg = dfg[inst].arguments(&dfg.value_lists)[0]; + split_arg = dfg.inst_args(inst)[0]; } else { return value; } @@ -297,7 +297,7 @@ fn resolve_splits(dfg: &DataFlowGraph, value: Value) -> Value { // See if split_arg is defined by a concatenation instruction. if let ValueDef::Res(inst, _) = dfg.value_def(split_arg) { if dfg[inst].opcode() == concat_opc { - return dfg[inst].arguments(&dfg.value_lists)[split_res]; + return dfg.inst_args(inst)[split_res]; } } @@ -316,10 +316,10 @@ fn resolve_splits(dfg: &DataFlowGraph, value: Value) -> Value { pub fn simplify_branch_arguments(dfg: &mut DataFlowGraph, branch: Inst) { let mut new_args = Vec::new(); - for &arg in dfg[branch].arguments(&dfg.value_lists) { + for &arg in dfg.inst_args(branch) { let new_arg = resolve_splits(dfg, arg); new_args.push(new_arg); } - dfg.insts[branch].arguments_mut(&mut dfg.value_lists).copy_from_slice(&new_args); + dfg.inst_args_mut(branch).copy_from_slice(&new_args); } diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 19800600b4..9bc10c30b3 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -302,7 +302,7 @@ impl<'a> Context<'a> { } ConstraintKind::Tied(arg_index) => { // This def must use the same register as a fixed instruction argument. - let arg = dfg[inst].arguments(&dfg.value_lists)[arg_index as usize]; + let arg = dfg.inst_args(inst)[arg_index as usize]; let loc = locations[arg]; *locations.ensure(lv.value) = loc; // Mark the reused register. It's not really clear if we support tied diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 0e384017c2..40faff0591 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -320,7 +320,7 @@ impl Liveness { .unwrap_or(&[]) .iter(); - for &arg in func.dfg[inst].arguments(&func.dfg.value_lists) { + for &arg in func.dfg.inst_args(inst) { // Get the live range, create it as a dead range if necessary. let lr = get_or_create(&mut self.ranges, arg, func, recipe_constraints); diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 0e59b14e2e..d3de786b6e 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -196,7 +196,7 @@ impl<'a> Verifier<'a> { fn verify_entity_references(&self, inst: Inst) -> Result<()> { use ir::instructions::InstructionData::*; - for &arg in self.func.dfg[inst].arguments(&self.func.dfg.value_lists) { + for &arg in self.func.dfg.inst_args(inst) { self.verify_value(inst, arg)?; } diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 5ff48ec91c..548f718099 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -160,7 +160,7 @@ fn type_suffix(func: &Function, inst: Inst) -> Option { // Write out any value aliases appearing in `inst`. fn write_value_aliases(w: &mut Write, func: &Function, inst: Inst, indent: usize) -> Result { - for &arg in func.dfg[inst].arguments(&func.dfg.value_lists) { + for &arg in func.dfg.inst_args(inst) { let resolved = func.dfg.resolve_aliases(arg); if resolved != arg { writeln!(w, "{1:0$}{2} -> {3}", indent, "", arg, resolved)?; From 19710af5b728c85c838011cc3f3dfcef3469dd1a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 23 Mar 2017 12:31:29 -0700 Subject: [PATCH 0624/3084] Start the binemit module. This module will provide supporting code for emitting binary machine code with relocations. --- lib/cretonne/src/binemit/mod.rs | 33 +++++++++++++++++++++++++++++++++ lib/cretonne/src/lib.rs | 1 + 2 files changed, 34 insertions(+) create mode 100644 lib/cretonne/src/binemit/mod.rs diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs new file mode 100644 index 0000000000..d3c97ec539 --- /dev/null +++ b/lib/cretonne/src/binemit/mod.rs @@ -0,0 +1,33 @@ +//! Binary machine code emission. +//! +//! The `binemit` module contains code for translating Cretonne's intermediate representation into +//! binary machine code. + +use ir::{FuncRef, JumpTable}; + +/// Relocation kinds depend on the current ISA. +pub struct Reloc(u16); + +/// Abstract interface for adding bytes to the code segment. +/// +/// A `CodeSink` will receive all of the machine code for a function. It also accepts relocations +/// which are locations in the code section that need to be fixed up when linking. +pub trait CodeSink { + /// Add 1 byte to the code section. + fn put1(&mut self, u8); + + /// Add 2 bytes to the code section. + fn put2(&mut self, u16); + + /// Add 4 bytes to the code section. + fn put4(&mut self, u32); + + /// Add 8 bytes to the code section. + fn put8(&mut self, u64); + + /// Add a relocation referencing an external function at the current offset. + fn reloc_func(&mut self, Reloc, FuncRef); + + /// Add a relocation referencing a jump table. + fn reloc_jt(&mut self, Reloc, JumpTable); +} diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 079b9a4559..b4c163b14d 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -10,6 +10,7 @@ pub use write::write_function; /// Version number of the cretonne crate. pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); +pub mod binemit; pub mod flowgraph; pub mod dominator_tree; pub mod entity_list; From 0619d6f8270c89ad5a8b8adfa715f34d502c6bab Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 27 Mar 2017 16:11:54 -0700 Subject: [PATCH 0625/3084] Generate binemit::emit_inst() functions. Use the meta language encoding recipes to generate an emit_inst() function for each ISA. The generated calls into recipe_*() functions that must be implemented by hand. Implement recipe_*() functions for the RISC-V recipes. Add the TargetIsa::emit_inst() entry point which emits an instruction to a CodeSink trait object. --- lib/cretonne/meta/build.py | 2 ++ lib/cretonne/meta/gen_binemit.py | 46 ++++++++++++++++++++++++ lib/cretonne/src/binemit/mod.rs | 10 +++++- lib/cretonne/src/ir/valueloc.rs | 8 +++++ lib/cretonne/src/isa/arm32/binemit.rs | 6 ++++ lib/cretonne/src/isa/arm32/mod.rs | 13 +++++-- lib/cretonne/src/isa/arm64/binemit.rs | 6 ++++ lib/cretonne/src/isa/arm64/mod.rs | 13 +++++-- lib/cretonne/src/isa/intel/binemit.rs | 6 ++++ lib/cretonne/src/isa/intel/mod.rs | 13 +++++-- lib/cretonne/src/isa/mod.rs | 9 ++++- lib/cretonne/src/isa/riscv/binemit.rs | 52 +++++++++++++++++++++++++++ lib/cretonne/src/isa/riscv/mod.rs | 8 ++++- 13 files changed, 183 insertions(+), 9 deletions(-) create mode 100644 lib/cretonne/meta/gen_binemit.py create mode 100644 lib/cretonne/src/isa/arm32/binemit.rs create mode 100644 lib/cretonne/src/isa/arm64/binemit.rs create mode 100644 lib/cretonne/src/isa/intel/binemit.rs create mode 100644 lib/cretonne/src/isa/riscv/binemit.rs diff --git a/lib/cretonne/meta/build.py b/lib/cretonne/meta/build.py index dd7aeac5f9..08ba14a358 100644 --- a/lib/cretonne/meta/build.py +++ b/lib/cretonne/meta/build.py @@ -12,6 +12,7 @@ import gen_build_deps import gen_encoding import gen_legalizer import gen_registers +import gen_binemit parser = argparse.ArgumentParser(description='Generate sources for Cretonne.') parser.add_argument('--out-dir', help='set output directory') @@ -27,4 +28,5 @@ gen_settings.generate(isas, out_dir) gen_encoding.generate(isas, out_dir) gen_legalizer.generate(isas, out_dir) gen_registers.generate(isas, out_dir) +gen_binemit.generate(isas, out_dir) gen_build_deps.generate() diff --git a/lib/cretonne/meta/gen_binemit.py b/lib/cretonne/meta/gen_binemit.py new file mode 100644 index 0000000000..8d1954ce55 --- /dev/null +++ b/lib/cretonne/meta/gen_binemit.py @@ -0,0 +1,46 @@ +""" +Generate binary emission code for each ISA. +""" + +from __future__ import absolute_import +import srcgen + +try: + from typing import Sequence, List # noqa + from cdsl.isa import TargetISA # noqa +except ImportError: + pass + + +def gen_isa(isa, fmt): + # type: (TargetISA, srcgen.Formatter) -> None + """ + Generate Binary emission code for `isa`. + """ + fmt.doc_comment( + ''' + Emit binary machine code for `inst` for the {} ISA. + '''.format(isa.name)) + if len(isa.all_recipes) == 0: + # No encoding recipes: Emit a stub. + with fmt.indented( + 'pub fn emit_inst' + '(func: &Function, inst: Inst, _sink: &mut CS) {', '}'): + fmt.line('bad_encoding(func, inst)') + else: + with fmt.indented( + 'pub fn emit_inst' + '(func: &Function, inst: Inst, sink: &mut CS) {', '}'): + with fmt.indented('match func.encodings[inst].recipe() {', '}'): + for i, recipe in enumerate(isa.all_recipes): + fmt.line('{} => recipe_{}(func, inst, sink),'.format( + i, recipe.name.lower())) + fmt.line('_ => bad_encoding(func, inst),') + + +def generate(isas, out_dir): + # type: (Sequence[TargetISA], str) -> None + for isa in isas: + fmt = srcgen.Formatter() + gen_isa(isa, fmt) + fmt.update_file('binemit-{}.rs'.format(isa.name), out_dir) diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index d3c97ec539..0b8b3475b7 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -3,7 +3,7 @@ //! The `binemit` module contains code for translating Cretonne's intermediate representation into //! binary machine code. -use ir::{FuncRef, JumpTable}; +use ir::{FuncRef, JumpTable, Function, Inst}; /// Relocation kinds depend on the current ISA. pub struct Reloc(u16); @@ -31,3 +31,11 @@ pub trait CodeSink { /// Add a relocation referencing a jump table. fn reloc_jt(&mut self, Reloc, JumpTable); } + +/// Report a bad encoding error. +#[inline(never)] +pub fn bad_encoding(func: &Function, inst: Inst) -> ! { + panic!("Bad encoding {} for {}", + func.encodings[inst], + func.dfg.display_inst(inst)); +} diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/cretonne/src/ir/valueloc.rs index 253c5ceb33..2d6fc644b1 100644 --- a/lib/cretonne/src/ir/valueloc.rs +++ b/lib/cretonne/src/ir/valueloc.rs @@ -25,6 +25,14 @@ impl Default for ValueLoc { } impl ValueLoc { + /// Get the register unit of this location, or panic. + pub fn unwrap_reg(self) -> RegUnit { + match self { + ValueLoc::Reg(ru) => ru, + _ => panic!("Expected register: {:?}", self), + } + } + /// Return an object that can display this value location, using the register info from the /// target ISA. pub fn display<'a, R: Into>>(self, regs: R) -> DisplayValueLoc<'a> { diff --git a/lib/cretonne/src/isa/arm32/binemit.rs b/lib/cretonne/src/isa/arm32/binemit.rs new file mode 100644 index 0000000000..dcc76331a2 --- /dev/null +++ b/lib/cretonne/src/isa/arm32/binemit.rs @@ -0,0 +1,6 @@ +//! Emitting binary ARM32 machine code. + +use binemit::{CodeSink, bad_encoding}; +use ir::{Function, Inst}; + +include!(concat!(env!("OUT_DIR"), "/binemit-arm32.rs")); diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 7cc6a93358..df4f4c2570 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -1,14 +1,16 @@ //! ARM 32-bit Instruction Set Architecture. pub mod settings; +mod binemit; mod enc_tables; mod registers; +use binemit::CodeSink; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, Encoding, Legalize, RecipeConstraints}; -use ir::{InstructionData, DataFlowGraph}; +use ir; #[allow(dead_code)] struct Isa { @@ -53,7 +55,10 @@ impl TargetIsa for Isa { registers::INFO.clone() } - fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Result { + fn encode(&self, + dfg: &ir::DataFlowGraph, + inst: &ir::InstructionData) + -> Result { lookup_enclist(inst.ctrl_typevar(dfg), inst.opcode(), self.cpumode, @@ -74,4 +79,8 @@ impl TargetIsa for Isa { fn recipe_constraints(&self) -> &'static [RecipeConstraints] { &enc_tables::RECIPE_CONSTRAINTS } + + fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { + binemit::emit_inst(func, inst, sink) + } } diff --git a/lib/cretonne/src/isa/arm64/binemit.rs b/lib/cretonne/src/isa/arm64/binemit.rs new file mode 100644 index 0000000000..13885307d4 --- /dev/null +++ b/lib/cretonne/src/isa/arm64/binemit.rs @@ -0,0 +1,6 @@ +//! Emitting binary ARM64 machine code. + +use binemit::{CodeSink, bad_encoding}; +use ir::{Function, Inst}; + +include!(concat!(env!("OUT_DIR"), "/binemit-arm64.rs")); diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 5f253c39c2..2c11b450bb 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -1,14 +1,16 @@ //! ARM 64-bit Instruction Set Architecture. pub mod settings; +mod binemit; mod enc_tables; mod registers; +use binemit::CodeSink; use super::super::settings as shared_settings; use isa::enc_tables::{lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, Encoding, Legalize, RecipeConstraints}; -use ir::{InstructionData, DataFlowGraph}; +use ir; #[allow(dead_code)] struct Isa { @@ -46,7 +48,10 @@ impl TargetIsa for Isa { registers::INFO.clone() } - fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Result { + fn encode(&self, + dfg: &ir::DataFlowGraph, + inst: &ir::InstructionData) + -> Result { lookup_enclist(inst.ctrl_typevar(dfg), inst.opcode(), &enc_tables::LEVEL1_A64[..], @@ -67,4 +72,8 @@ impl TargetIsa for Isa { fn recipe_constraints(&self) -> &'static [RecipeConstraints] { &enc_tables::RECIPE_CONSTRAINTS } + + fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { + binemit::emit_inst(func, inst, sink) + } } diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs new file mode 100644 index 0000000000..8870abd8f1 --- /dev/null +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -0,0 +1,6 @@ +//! Emitting binary Intel machine code. + +use binemit::{CodeSink, bad_encoding}; +use ir::{Function, Inst}; + +include!(concat!(env!("OUT_DIR"), "/binemit-intel.rs")); diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 4a5ea8201a..46aa101ce0 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -1,14 +1,16 @@ //! Intel Instruction Set Architectures. pub mod settings; +mod binemit; mod enc_tables; mod registers; +use binemit::CodeSink; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, Encoding, Legalize, RecipeConstraints}; -use ir::{InstructionData, DataFlowGraph}; +use ir; #[allow(dead_code)] struct Isa { @@ -53,7 +55,10 @@ impl TargetIsa for Isa { registers::INFO.clone() } - fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Result { + fn encode(&self, + dfg: &ir::DataFlowGraph, + inst: &ir::InstructionData) + -> Result { lookup_enclist(inst.ctrl_typevar(dfg), inst.opcode(), self.cpumode, @@ -74,4 +79,8 @@ impl TargetIsa for Isa { fn recipe_constraints(&self) -> &'static [RecipeConstraints] { &enc_tables::RECIPE_CONSTRAINTS } + + fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { + binemit::emit_inst(func, inst, sink) + } } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 37684addb1..776d5e4bca 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -44,8 +44,9 @@ pub use isa::encoding::Encoding; pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex}; pub use isa::constraints::{RecipeConstraints, OperandConstraint, ConstraintKind}; +use binemit::CodeSink; use settings; -use ir::{InstructionData, DataFlowGraph, Signature}; +use ir::{Function, Inst, InstructionData, DataFlowGraph, Signature}; pub mod riscv; pub mod intel; @@ -181,4 +182,10 @@ pub trait TargetIsa { fn legalize_signature(&self, _sig: &mut Signature) { unimplemented!() } + + /// Emit binary machine code for a single instruction into the `sink` trait object. + /// + /// Note that this will call `put*` methods on the trait object via its vtable which is not the + /// fastest way of emitting code. + fn emit_inst(&self, func: &Function, inst: Inst, sink: &mut CodeSink); } diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs new file mode 100644 index 0000000000..e7dcc6021f --- /dev/null +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -0,0 +1,52 @@ +//! Emitting binary RISC-V machine code. + +use binemit::{CodeSink, bad_encoding}; +use ir::{Function, Inst, InstructionData}; + +include!(concat!(env!("OUT_DIR"), "/binemit-riscv.rs")); + +/// R-type instructions. +/// +/// 31 24 19 14 11 6 +/// funct7 rs2 rs1 funct3 rd opcode +/// 25 20 15 12 7 0 +/// +/// Encoding bits: `opcode[6:2] | (funct3 << 5) | (funct7 << 8)`. +fn recipe_r(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Binary { args, .. } = func.dfg[inst] { + let bits = func.encodings[inst].bits(); + let rs1 = func.locations[args[0]].unwrap_reg(); + let rs2 = func.locations[args[1]].unwrap_reg(); + let rd = func.locations[func.dfg.first_result(inst)].unwrap_reg(); + + // 0-6: opcode + let mut i = 0x3; + i |= (bits as u32 & 0x1f) << 2; + // 7-11: rd + i |= (rd as u32 & 0x1f) << 7; + // 12-14: funct3 + i |= ((bits as u32 >> 5) & 0x7) << 12; + // 15-19: rs1 + i |= (rs1 as u32 & 0x1f) << 15; + // 20-24: rs1 + i |= (rs2 as u32 & 0x1f) << 20; + // 25-31: funct7 + i |= ((bits as u32 >> 8) & 0x7f) << 25; + + sink.put4(i); + } else { + panic!("Expected Binary format: {:?}", func.dfg[inst]); + } +} + +fn recipe_rshamt(_func: &Function, _inst: Inst, _sink: &mut CS) { + unimplemented!() +} + +fn recipe_i(_func: &Function, _inst: Inst, _sink: &mut CS) { + unimplemented!() +} + +fn recipe_iret(_func: &Function, _inst: Inst, _sink: &mut CS) { + unimplemented!() +} diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index ce04fe70b6..94808bfb70 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -2,14 +2,16 @@ pub mod settings; mod abi; +mod binemit; mod enc_tables; mod registers; use super::super::settings as shared_settings; +use binemit::CodeSink; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, Encoding, Legalize, RecipeConstraints}; -use ir::{InstructionData, DataFlowGraph, Signature}; +use ir::{Function, Inst, InstructionData, DataFlowGraph, Signature}; #[allow(dead_code)] struct Isa { @@ -80,6 +82,10 @@ impl TargetIsa for Isa { // We can pass in `self.isa_flags` too, if we need it. abi::legalize_signature(sig, &self.shared_flags) } + + fn emit_inst(&self, func: &Function, inst: Inst, sink: &mut CodeSink) { + binemit::emit_inst(func, inst, sink) + } } #[cfg(test)] From f540cb06649ab4d1fcca066b2c6c872009ba74d2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 28 Mar 2017 15:43:41 -0700 Subject: [PATCH 0626/3084] Add a binemit test command. This makes it possible to write file tests that verify the binary encoding of machine code. --- cranelift/docs/testing.rst | 22 ++++ cranelift/filetests/isa/riscv/binary32.cton | 13 +++ cranelift/src/filetest/binemit.rs | 107 ++++++++++++++++++++ cranelift/src/filetest/mod.rs | 2 + 4 files changed, 144 insertions(+) create mode 100644 cranelift/filetests/isa/riscv/binary32.cton create mode 100644 cranelift/src/filetest/binemit.rs diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index db027f8fbe..9a59f2c7be 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -314,3 +314,25 @@ Second, the register allocator is run on the function, inserting spill code and assigning registers and stack slots to all values. The resulting function is then run through filecheck. + +`test binemit` +-------------- + +Test the emission of binary machine code. + +The functions must contains instructions that are annotated with both encodings +and value locations (registers or stack slots). For instructions that are +annotated with a `bin:` directive, the emitted hexadecimal machine code for +that instruction is compared to the directive:: + + test binemit + isa riscv + + function int32() { + ebb0: + [-,%x5] v1 = iconst.i32 1 + [-,%x6] v2 = iconst.i32 2 + [R#0c,%x7] v10 = iadd v1, v2 ; bin: 006283b3 + [R#200c,%x8] v11 = isub v1, v2 ; bin: 40628433 + return + } diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton new file mode 100644 index 0000000000..0d2805681a --- /dev/null +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -0,0 +1,13 @@ +; Binary emission of 32-bit code. +test binemit +isa riscv + +function int32() { +ebb0: + [-,%x5] v1 = iconst.i32 1 + [-,%x6] v2 = iconst.i32 2 + [R#0c,%x7] v10 = iadd v1, v2 ; bin: 006283b3 + [R#200c,%x8] v11 = isub v1, v2 ; bin: 40628433 + [R#10c] v12 = imul v1, v2 + return +} diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs new file mode 100644 index 0000000000..2c6a725431 --- /dev/null +++ b/cranelift/src/filetest/binemit.rs @@ -0,0 +1,107 @@ +//! Test command for testing the binary machine code emission. +//! +//! The `binemit` test command generates binary machine code for every instruction in the input +//! functions and compares the results to the expected output. + +use std::borrow::{Borrow, Cow}; +use std::fmt::Write; +use cretonne::binemit; +use cretonne::ir; +use cretonne::ir::entities::AnyEntity; +use cton_reader::TestCommand; +use filetest::subtest::{SubTest, Context, Result}; +use utils::match_directive; + +struct TestBinEmit; + +pub fn subtest(parsed: &TestCommand) -> Result> { + assert_eq!(parsed.command, "binemit"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestBinEmit)) + } +} + +// Code sink that generates text. +struct TextSink { + text: String, +} + +impl binemit::CodeSink for TextSink { + fn put1(&mut self, x: u8) { + write!(self.text, "{:02x} ", x).unwrap(); + } + + fn put2(&mut self, x: u16) { + write!(self.text, "{:04x} ", x).unwrap(); + } + + fn put4(&mut self, x: u32) { + write!(self.text, "{:08x} ", x).unwrap(); + } + + fn put8(&mut self, x: u64) { + write!(self.text, "{:016x} ", x).unwrap(); + } + + fn reloc_func(&mut self, _: binemit::Reloc, _: ir::FuncRef) { + unimplemented!() + } + + fn reloc_jt(&mut self, _: binemit::Reloc, _: ir::JumpTable) { + unimplemented!() + } +} + +impl SubTest for TestBinEmit { + fn name(&self) -> Cow { + Cow::from("binemit") + } + + fn is_mutating(&self) -> bool { + false + } + + fn needs_isa(&self) -> bool { + true + } + + fn run(&self, func: Cow, context: &Context) -> Result<()> { + let isa = context.isa.expect("binemit needs an ISA"); + // TODO: Run a verifier pass over the code first to detect any bad encodings or missing/bad + // value locations. The current error reporting is just crashing... + let func = func.borrow(); + + let mut sink = TextSink { text: String::new() }; + + for comment in &context.details.comments { + if let Some(want) = match_directive(comment.text, "bin:") { + let inst = match comment.entity { + AnyEntity::Inst(inst) => inst, + _ => { + return Err(format!("annotation on non-inst {}: {}", + comment.entity, + comment.text)) + } + }; + sink.text.clear(); + isa.emit_inst(&func, inst, &mut sink); + let have = sink.text.trim(); + if have != want { + return Err(format!("Bad machine code for {}: {}\nWant: {}\nGot: {}", + inst, + func.dfg.display_inst(inst), + want, + have)); + } + } + } + + if sink.text.is_empty() { + Err("No bin: directives found".to_string()) + } else { + Ok(()) + } + } +} diff --git a/cranelift/src/filetest/mod.rs b/cranelift/src/filetest/mod.rs index 41b89948a1..7a379ec0fe 100644 --- a/cranelift/src/filetest/mod.rs +++ b/cranelift/src/filetest/mod.rs @@ -13,6 +13,7 @@ use filetest::runner::TestRunner; pub mod subtest; +mod binemit; mod concurrent; mod domtree; mod legalizer; @@ -60,6 +61,7 @@ fn new_subtest(parsed: &TestCommand) -> subtest::Result> { "verifier" => verifier::subtest(parsed), "legalizer" => legalizer::subtest(parsed), "regalloc" => regalloc::subtest(parsed), + "binemit" => binemit::subtest(parsed), _ => Err(format!("unknown test command '{}'", parsed.command)), } } From 6ad34f90aaf04a80a9a77c77b7574afd4a0f8d2a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 29 Mar 2017 09:48:25 -0700 Subject: [PATCH 0627/3084] Allow for unencoded instructions in the binemit tests. If an instruction doesn't have an associated encoding, use the standard TargetIsa hook to encode it. The test still fails if an instruction can't be encoded. There is no legalization step. --- cranelift/docs/testing.rst | 8 ++++++++ cranelift/filetests/isa/riscv/binary32.cton | 2 +- cranelift/src/filetest/binemit.rs | 22 ++++++++++++++++++--- 3 files changed, 28 insertions(+), 4 deletions(-) diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index 9a59f2c7be..f58a413033 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -336,3 +336,11 @@ that instruction is compared to the directive:: [R#200c,%x8] v11 = isub v1, v2 ; bin: 40628433 return } + +If any instructions are unencoded (indicated with a `[-]` encoding field), they +will be encoded using the same mechanism as the legalizer uses. However, +illegal instructions for the ISA won't be expanded into other instruction +sequences. Instead the test will fail. + +Value locations must be present if they are required to compute the binary +bits. Missing value locations will cause the test to crash. diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 0d2805681a..c2f117f5da 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -6,7 +6,7 @@ function int32() { ebb0: [-,%x5] v1 = iconst.i32 1 [-,%x6] v2 = iconst.i32 2 - [R#0c,%x7] v10 = iadd v1, v2 ; bin: 006283b3 + [-,%x7] v10 = iadd v1, v2 ; bin: 006283b3 [R#200c,%x8] v11 = isub v1, v2 ; bin: 40628433 [R#10c] v12 = imul v1, v2 return diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 2c6a725431..16c2613b20 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -3,7 +3,7 @@ //! The `binemit` test command generates binary machine code for every instruction in the input //! functions and compares the results to the expected output. -use std::borrow::{Borrow, Cow}; +use std::borrow::Cow; use std::fmt::Write; use cretonne::binemit; use cretonne::ir; @@ -60,7 +60,7 @@ impl SubTest for TestBinEmit { } fn is_mutating(&self) -> bool { - false + true } fn needs_isa(&self) -> bool { @@ -71,7 +71,7 @@ impl SubTest for TestBinEmit { let isa = context.isa.expect("binemit needs an ISA"); // TODO: Run a verifier pass over the code first to detect any bad encodings or missing/bad // value locations. The current error reporting is just crashing... - let func = func.borrow(); + let mut func = func.into_owned(); let mut sink = TextSink { text: String::new() }; @@ -85,6 +85,22 @@ impl SubTest for TestBinEmit { comment.text)) } }; + + // Compute an encoding for `inst` if one wasn't provided. + if !func.encodings + .get(inst) + .map(|e| e.is_legal()) + .unwrap_or(false) { + match isa.encode(&func.dfg, &func.dfg[inst]) { + Ok(enc) => *func.encodings.ensure(inst) = enc, + Err(_) => { + return Err(format!("{} can't be encoded: {}", + inst, + func.dfg.display_inst(inst))) + } + } + } + sink.text.clear(); isa.emit_inst(&func, inst, &mut sink); let have = sink.text.trim(); From 892ad25b1bd61cdb71ac572609e34846334e9f0a Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Wed, 29 Mar 2017 21:14:42 +0100 Subject: [PATCH 0628/3084] Type checking and Dominator Tree integrity checks in Verifier (#66) * Verify that a recomputed dominator tree is identical to the existing one. * The verifier now typechecks instruction results and arguments. * The verifier now typechecks instruction results and arguments. * The verifier now typechecks instruction results and arguments. * Added `inst_{fixed,variable}_args` accessor functions. * Improved error messages in verifier. * Type check return statements against the function signature. --- lib/cretonne/meta/gen_instr.py | 4 - lib/cretonne/src/ir/dfg.rs | 28 ++- lib/cretonne/src/ir/instructions.rs | 1 - lib/cretonne/src/legalizer/boundary.rs | 6 +- lib/cretonne/src/verifier.rs | 245 ++++++++++++++++++++++++- 5 files changed, 269 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index f3c36a716f..0b473e60f3 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -210,10 +210,6 @@ def gen_instruction_data_impl(fmt): fmt.doc_comment( """ Get the value arguments to this instruction. - - This is returned as two `Value` slices. The first one - represents the fixed arguments, the second any variable - arguments. """) gen_arguments_method(fmt, False) fmt.doc_comment( diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 24727362b8..f7d214182b 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -344,16 +344,40 @@ impl DataFlowGraph { DisplayInst(self, inst) } - /// Get the value arguments on `inst` as a slice. + /// Get all value arguments on `inst` as a slice. pub fn inst_args(&self, inst: Inst) -> &[Value] { self.insts[inst].arguments(&self.value_lists) } - /// Get the value arguments on `inst` as a mutable slice. + /// Get all value arguments on `inst` as a mutable slice. pub fn inst_args_mut(&mut self, inst: Inst) -> &mut [Value] { self.insts[inst].arguments_mut(&mut self.value_lists) } + /// Get the fixed value arguments on `inst` as a slice. + pub fn inst_fixed_args(&self, inst: Inst) -> &[Value] { + let fixed_args = self[inst].opcode().constraints().fixed_value_arguments(); + &self.inst_args(inst)[..fixed_args] + } + + /// Get the fixed value arguments on `inst` as a mutable slice. + pub fn inst_fixed_args_mut(&mut self, inst: Inst) -> &mut [Value] { + let fixed_args = self[inst].opcode().constraints().fixed_value_arguments(); + &mut self.inst_args_mut(inst)[..fixed_args] + } + + /// Get the variable value arguments on `inst` as a slice. + pub fn inst_variable_args(&self, inst: Inst) -> &[Value] { + let fixed_args = self[inst].opcode().constraints().fixed_value_arguments(); + &self.inst_args(inst)[fixed_args..] + } + + /// Get the variable value arguments on `inst` as a mutable slice. + pub fn inst_variable_args_mut(&mut self, inst: Inst) -> &mut [Value] { + let fixed_args = self[inst].opcode().constraints().fixed_value_arguments(); + &mut self.inst_args_mut(inst)[fixed_args..] + } + /// Create result values for an instruction that produces multiple results. /// /// Instructions that produce 0 or 1 result values only need to be created with `make_inst`. If diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index fe76791211..f5fd0c0a98 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -398,7 +398,6 @@ pub struct OpcodeConstraints { /// Offset into `OPERAND_CONSTRAINT` table of the descriptors for this opcode. The first /// `fixed_results()` entries describe the result constraints, then follows constraints for the /// fixed `Value` input operands. (`fixed_value_arguments()` of them). - /// format. constraint_offset: u16, } diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 8ec8ffeb9c..2e3766057d 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -346,12 +346,8 @@ fn check_call_signature(dfg: &DataFlowGraph, inst: Inst) -> Result<(), SigRef> { /// Check if the arguments of the return `inst` match the signature. fn check_return_signature(dfg: &DataFlowGraph, inst: Inst, sig: &Signature) -> bool { - let fixed_values = dfg[inst].opcode().constraints().fixed_value_arguments(); check_arg_types(dfg, - dfg.inst_args(inst) - .iter() - .skip(fixed_values) - .cloned(), + dfg.inst_variable_args(inst).iter().cloned(), &sig.return_types) } diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index d3de786b6e..433b4bc84d 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -26,7 +26,6 @@ //! //! - All predecessors in the CFG must be branches to the EBB. //! - All branches to an EBB must be present in the CFG. -//! TODO: //! - A recomputed dominator tree is identical to the existing one. //! //! Type checking @@ -42,6 +41,7 @@ //! - All return instructions must have return value operands matching the current //! function signature. //! +//! TODO: //! Ad hoc checking //! //! - Stack slot loads and stores must be in-bounds. @@ -56,8 +56,9 @@ use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir::entities::AnyEntity; -use ir::instructions::{InstructionFormat, BranchInfo}; -use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, Value}; +use ir::instructions::{InstructionFormat, BranchInfo, ResolvedConstraint, CallInfo}; +use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, Value, Type}; +use Context; use std::fmt::{self, Display, Formatter}; use std::result; @@ -101,6 +102,13 @@ pub fn verify_function(func: &Function) -> Result<()> { Verifier::new(func).run() } +/// Verify `ctx`. +pub fn verify_context(ctx: &Context) -> Result<()> { + let verifier = Verifier::new(&ctx.func); + verifier.domtree_integrity(&ctx.domtree)?; + verifier.run() +} + struct Verifier<'a> { func: &'a Function, cfg: ControlFlowGraph, @@ -377,11 +385,242 @@ impl<'a> Verifier<'a> { Ok(()) } + fn domtree_integrity(&self, domtree: &DominatorTree) -> Result<()> { + // We consider two `DominatorTree`s to be equal if they return the same immediate + // dominator for each EBB. Therefore the current domtree is valid if it matches the freshly + // computed one. + for ebb in self.func.layout.ebbs() { + let expected = domtree.idom(ebb); + let got = self.domtree.idom(ebb); + if got != expected { + return err!(ebb, + "invalid domtree, expected idom({}) = {:?}, got {:?}", + ebb, + expected, + got); + } + } + Ok(()) + } + + fn typecheck_entry_block_arguments(&self) -> Result<()> { + if let Some(ebb) = self.func.layout.entry_block() { + let expected_types = &self.func.signature.argument_types; + let ebb_arg_count = self.func.dfg.num_ebb_args(ebb); + + if ebb_arg_count != expected_types.len() { + return err!(ebb, "entry block arguments must match function signature"); + } + + for (i, arg) in self.func + .dfg + .ebb_args(ebb) + .enumerate() { + let arg_type = self.func.dfg.value_type(arg); + if arg_type != expected_types[i].value_type { + return err!(ebb, + "entry block argument {} expected to have type {}, got {}", + i, + expected_types[i], + arg_type); + } + } + } + Ok(()) + } + + fn typecheck(&self, inst: Inst) -> Result<()> { + let inst_data = &self.func.dfg[inst]; + let constraints = inst_data.opcode().constraints(); + + let ctrl_type = if let Some(value_typeset) = constraints.ctrl_typeset() { + // For polymorphic opcodes, determine the controlling type variable first. + let ctrl_type = inst_data.ctrl_typevar(&self.func.dfg); + + if !value_typeset.contains(ctrl_type) { + return err!(inst, "has an invalid controlling type {}", ctrl_type); + } + + ctrl_type + } else { + // Non-polymorphic instructions don't check the controlling type variable, so `Option` + // is unnecessary and we can just make it `VOID`. + types::VOID + }; + + self.typecheck_results(inst, ctrl_type)?; + self.typecheck_fixed_args(inst, ctrl_type)?; + self.typecheck_variable_args(inst)?; + self.typecheck_return(inst)?; + + Ok(()) + } + + fn typecheck_results(&self, inst: Inst, ctrl_type: Type) -> Result<()> { + let mut i = 0; + for result in self.func.dfg.inst_results(inst) { + let result_type = self.func.dfg.value_type(result); + let expected_type = self.func.dfg.compute_result_type(inst, i, ctrl_type); + if let Some(expected_type) = expected_type { + if result_type != expected_type { + return err!(inst, + "expected result {} ({}) to have type {}, found {}", + i, + result, + expected_type, + result_type); + } + } else { + return err!(inst, "has more result values than expected"); + } + i += 1; + } + + // There aren't any more result types left. + if self.func.dfg.compute_result_type(inst, i, ctrl_type) != None { + return err!(inst, "has fewer result values than expected"); + } + Ok(()) + } + + fn typecheck_fixed_args(&self, inst: Inst, ctrl_type: Type) -> Result<()> { + let constraints = self.func.dfg[inst].opcode().constraints(); + + for (i, &arg) in self.func + .dfg + .inst_fixed_args(inst) + .iter() + .enumerate() { + let arg_type = self.func.dfg.value_type(arg); + match constraints.value_argument_constraint(i, ctrl_type) { + ResolvedConstraint::Bound(expected_type) => { + if arg_type != expected_type { + return err!(inst, + "arg {} ({}) has type {}, expected {}", + i, + arg, + arg_type, + expected_type); + } + } + ResolvedConstraint::Free(type_set) => { + if !type_set.contains(arg_type) { + return err!(inst, + "arg {} ({}) with type {} failed to satisfy type set {:?}", + i, + arg, + arg_type, + type_set); + } + } + } + } + Ok(()) + } + + fn typecheck_variable_args(&self, inst: Inst) -> Result<()> { + match self.func.dfg[inst].analyze_branch(&self.func.dfg.value_lists) { + BranchInfo::SingleDest(ebb, _) => { + let iter = self.func + .dfg + .ebb_args(ebb) + .map(|v| self.func.dfg.value_type(v)); + self.typecheck_variable_args_iterator(inst, iter)?; + } + BranchInfo::Table(table) => { + for (_, ebb) in self.func.jump_tables[table].entries() { + let arg_count = self.func.dfg.num_ebb_args(ebb); + if arg_count != 0 { + return err!(inst, + "takes no arguments, but had target {} with {} arguments", + ebb, + arg_count); + } + } + } + BranchInfo::NotABranch => {} + } + + match self.func.dfg[inst].analyze_call(&self.func.dfg.value_lists) { + CallInfo::Direct(func_ref, _) => { + let sig_ref = self.func.dfg.ext_funcs[func_ref].signature; + let arg_types = + self.func.dfg.signatures[sig_ref].argument_types.iter().map(|a| a.value_type); + self.typecheck_variable_args_iterator(inst, arg_types)?; + } + CallInfo::Indirect(sig_ref, _) => { + let arg_types = + self.func.dfg.signatures[sig_ref].argument_types.iter().map(|a| a.value_type); + self.typecheck_variable_args_iterator(inst, arg_types)?; + } + CallInfo::NotACall => {} + } + Ok(()) + } + + fn typecheck_variable_args_iterator>(&self, + inst: Inst, + iter: I) + -> Result<()> { + let variable_args = self.func.dfg.inst_variable_args(inst); + let mut i = 0; + + for expected_type in iter { + if i >= variable_args.len() { + // Result count mismatch handled below, we want the full argument count first though + i += 1; + continue; + } + let arg = variable_args[i]; + let arg_type = self.func.dfg.value_type(arg); + if expected_type != arg_type { + return err!(inst, + "arg {} ({}) has type {}, expected {}", + i, + variable_args[i], + arg_type, + expected_type); + } + i += 1; + } + if i != variable_args.len() { + return err!(inst, + "mismatched argument count, got {}, expected {}", + variable_args.len(), + i); + } + Ok(()) + } + + fn typecheck_return(&self, inst: Inst) -> Result<()> { + if self.func.dfg[inst].opcode().is_return() { + let args = self.func.dfg.inst_variable_args(inst); + let expected_types = &self.func.signature.return_types; + if args.len() != expected_types.len() { + return err!(inst, "arguments of return must match function signature"); + } + for (i, (&arg, &expected_type)) in args.iter().zip(expected_types).enumerate() { + let arg_type = self.func.dfg.value_type(arg); + if arg_type != expected_type.value_type { + return err!(inst, + "arg {} ({}) has type {}, must match function signature of {}", + i, + arg, + arg_type, + expected_type); + } + } + } + Ok(()) + } + pub fn run(&self) -> Result<()> { + self.typecheck_entry_block_arguments()?; for ebb in self.func.layout.ebbs() { for inst in self.func.layout.ebb_insts(ebb) { self.ebb_integrity(ebb, inst)?; self.instruction_integrity(inst)?; + self.typecheck(inst)?; } self.cfg_integrity(ebb)?; } From 269f6d3dd0ba76bb6ef563990c03f116599da86a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 29 Mar 2017 10:30:22 -0700 Subject: [PATCH 0629/3084] Add Vim syntax support for Name and HexSequence tokens. Also disable spell checking for .cton files. They tend to contain very few words, and even comments are pretty cryptic. --- misc/vim/syntax/cton.vim | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/misc/vim/syntax/cton.vim b/misc/vim/syntax/cton.vim index 2d2dcd0fef..7d6bdab08c 100644 --- a/misc/vim/syntax/cton.vim +++ b/misc/vim/syntax/cton.vim @@ -9,16 +9,22 @@ elseif exists("b:current_syntax") finish endif +" Disable spell checking even in comments. +" They tend to refer to weird stuff like assembler mnemonics anyway. +syn spell notoplevel + syn keyword ctonHeader test isa set syn keyword ctonDecl function stack_slot jump_table syn keyword ctonFilecheck check sameln nextln unordered not regex contained syn match ctonType /\<[bif]\d\+\(x\d\+\)\?\>/ -syn match ctonEntity /\<\(v\|vx\|ss\|jt\|\)\d\+\>/ +syn match ctonEntity /\<\(v\|vx\|ss\|jt\|fn\|sig\)\d\+\>/ syn match ctonLabel /\/ +syn match ctonName /%\w\+\>/ syn match ctonNumber /-\?\<\d\+\>/ syn match ctonNumber /-\?\<0x\x\+\(\.\x*\)\(p[+-]\?\d\+\)\?\>/ +syn match ctonHexSeq /#\x\+\>/ syn region ctonCommentLine start=";" end="$" contains=ctonFilecheck @@ -27,7 +33,9 @@ hi def link ctonDecl Keyword hi def link ctonType Type hi def link ctonEntity Identifier hi def link ctonLabel Label +hi def link ctonName String hi def link ctonNumber Number +hi def link ctonHexSeq Number hi def link ctonCommentLine Comment hi def link ctonFilecheck SpecialComment From 4613cbcf18bea59adb27334fa3beb14f2f81fe4c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 29 Mar 2017 11:00:05 -0700 Subject: [PATCH 0630/3084] Test binary encodings of some RV32I instructions. These are the R-format instructions in RV32I. --- cranelift/filetests/isa/riscv/binary32.cton | 36 +++++++++++++++++---- 1 file changed, 30 insertions(+), 6 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index c2f117f5da..ede783b7df 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -2,12 +2,36 @@ test binemit isa riscv -function int32() { +function RV32I() { ebb0: - [-,%x5] v1 = iconst.i32 1 - [-,%x6] v2 = iconst.i32 2 - [-,%x7] v10 = iadd v1, v2 ; bin: 006283b3 - [R#200c,%x8] v11 = isub v1, v2 ; bin: 40628433 - [R#10c] v12 = imul v1, v2 + [-,%x10] v1 = iconst.i32 1 + [-,%x21] v2 = iconst.i32 2 + + ; Integer Register-Register Operations. + ; add + [-,%x7] v10 = iadd v1, v2 ; bin: 015503b3 + [-,%x16] v11 = iadd v2, v1 ; bin: 00aa8833 + ; sub + [-,%x7] v12 = isub v1, v2 ; bin: 415503b3 + [-,%x16] v13 = isub v2, v1 ; bin: 40aa8833 + ; TBD: slt/sltu + ; and + [-,%x7] v20 = band v1, v2 ; bin: 015573b3 + [-,%x16] v21 = band v2, v1 ; bin: 00aaf833 + ; or + [-,%x7] v22 = bor v1, v2 ; bin: 015563b3 + [-,%x16] v23 = bor v2, v1 ; bin: 00aae833 + ; xor + [-,%x7] v24 = bxor v1, v2 ; bin: 015543b3 + [-,%x16] v25 = bxor v2, v1 ; bin: 00aac833 + ; sll + [-,%x7] v30 = ishl v1, v2 ; bin: 015513b3 + [-,%x16] v31 = ishl v2, v1 ; bin: 00aa9833 + ; srl + [-,%x7] v32 = ushr v1, v2 ; bin: 015553b3 + [-,%x16] v33 = ushr v2, v1 ; bin: 00aad833 + ; sra + [-,%x7] v34 = sshr v1, v2 ; bin: 415553b3 + [-,%x16] v35 = sshr v2, v1 ; bin: 40aad833 return } From b5d40465918a1e8e334f71d4d019d2053bbb182d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 29 Mar 2017 14:40:12 -0700 Subject: [PATCH 0631/3084] Fix a type error in the legalizer patterns. The carry and borrow values are boolean, so we have to convert them to an integer type with bint(c) before we can add them to the result. Also tweak the default legalizer action for unsupported types: Only attempt a narrowing pattern for lane types > 32 bits. This was found by @angusholder's new type checks in the verifier. --- cranelift/filetests/isa/riscv/legalize-abi.cton | 2 +- cranelift/filetests/isa/riscv/legalize-i64.cton | 4 ++-- lib/cretonne/meta/base/legalize.py | 16 +++++++++++----- lib/cretonne/src/isa/enc_tables.rs | 6 +++++- 4 files changed, 19 insertions(+), 9 deletions(-) diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index 06fcfd5f6c..2e7da079bf 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -96,7 +96,7 @@ ebb0(v0: i64x4): ; check: $(v0d=$V) = iconcat $v0dl, $v0dh ; check: $(v0cd=$V) = vconcat $v0c, $v0d ; check: $(v0abcd=$V) = vconcat $v0ab, $v0cd - v1 = iadd v0, v0 + v1 = bxor v0, v0 ; check: $(v1ab=$V), $(v1cd=$V) = vsplit ; check: $(v1a=$V), $(v1b=$V) = vsplit $v1ab ; check: $(v1al=$V), $(v1ah=$V) = isplit $v1a diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton index dfa78447af..e2d5a763a2 100644 --- a/cranelift/filetests/isa/riscv/legalize-i64.cton +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -57,8 +57,8 @@ ebb0(v1: i64, v2: i64): ; check: $(c=$V) = icmp ult, $v3l, $v1l ; check: [R#0c ; sameln: $(v3h1=$V) = iadd $v1h, $v2h -; TODO: This doesn't typecheck. We need to convert the b1 result to i32. +; check: $(c_int=$V) = bint.i32 $c ; check: [R#0c -; sameln: $(v3h=$V) = iadd $v3h1, $c +; sameln: $(v3h=$V) = iadd $v3h1, $c_int ; check: $v3 = iconcat $v3l, $v3h ; check: return $v3l, $v3h diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index 9b0def9ae9..dc8185daf2 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -10,7 +10,7 @@ from __future__ import absolute_import from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm from .instructions import isub, isub_bin, isub_bout, isub_borrow from .instructions import band, bor, bxor, isplit, iconcat -from .instructions import icmp, iconst +from .instructions import icmp, iconst, bint from cdsl.ast import Var from cdsl.xform import Rtl, XFormGroup @@ -40,10 +40,12 @@ b = Var('b') b1 = Var('b1') b2 = Var('b2') b_in = Var('b_in') +b_int = Var('b_int') c = Var('c') c1 = Var('c1') c2 = Var('c2') c_in = Var('c_in') +c_int = Var('c_int') xl = Var('xl') xh = Var('xh') yl = Var('yl') @@ -102,21 +104,24 @@ expand.legalize( a << iadd_cin(x, y, c), Rtl( a1 << iadd(x, y), - a << iadd(a1, c) + c_int << bint(c), + a << iadd(a1, c_int) )) expand.legalize( a << isub_bin(x, y, b), Rtl( a1 << isub(x, y), - a << isub(a1, b) + b_int << bint(b), + a << isub(a1, b_int) )) expand.legalize( (a, c) << iadd_carry(x, y, c_in), Rtl( (a1, c1) << iadd_cout(x, y), - (a, c2) << iadd_cout(a1, c_in), + c_int << bint(c_in), + (a, c2) << iadd_cout(a1, c_int), c << bor(c1, c2) )) @@ -124,7 +129,8 @@ expand.legalize( (a, b) << isub_borrow(x, y, b_in), Rtl( (a1, b1) << isub_bout(x, y), - (a, b2) << isub_bout(a1, b_in), + b_int << bint(b_in), + (a, b2) << isub_bout(a1, b_int), b << bor(b1, b2) )) diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index dd4e67875f..71a3eba2e0 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -84,7 +84,11 @@ pub fn lookup_enclist(ctrl_typevar: Type, { // TODO: The choice of legalization actions here is naive. This needs to be configurable. probe(level1_table, ctrl_typevar, ctrl_typevar.index()) - .ok_or(Legalize::Narrow) + .ok_or_else(|| if ctrl_typevar.lane_type().bits() > 32 { + Legalize::Narrow + } else { + Legalize::Expand + }) .and_then(|l1idx| { let l1ent = &level1_table[l1idx]; let l2off = l1ent.offset.into() as usize; From 7a0092754d7df0274dfae1deb915fe3dd5149ff6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 29 Mar 2017 15:13:19 -0700 Subject: [PATCH 0632/3084] Allow vector types for isplit and iconcat. These two instructions make sense for vector types by simply performing the same operation on each lane, like most other vector operations. Problem found by @angusholder's verifier. --- lib/cretonne/meta/base/instructions.py | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index ade9b540bd..541889f7d5 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -1278,8 +1278,8 @@ fcvt_from_sint = Instruction( # WideInt = TypeVar( - 'WideInt', 'A scalar integer type from `i16` upwards', - ints=(16, 64)) + 'WideInt', 'An integer type with lanes from `i16` upwards', + ints=(16, 64), simd=True) x = Operand('x', WideInt) lo = Operand( 'lo', WideInt.half_width(), 'The low bits of `x`') @@ -1288,7 +1288,10 @@ hi = Operand( isplit = Instruction( 'isplit', r""" - Split a scalar integer into low and high parts. + Split an integer into low and high parts. + + Vectors of integers are split lane-wise, so the results have the same + number of lanes as the input, but the lanes are half the size. Returns the low half of `x` and the high half of `x` as two independent values. @@ -1297,8 +1300,8 @@ isplit = Instruction( NarrowInt = TypeVar( - 'NarrowInt', 'A scalar integer type up to `i32`', - ints=(8, 32)) + 'NarrowInt', 'An integer type with lanes type to `i32`', + ints=(8, 32), simd=True) lo = Operand('lo', NarrowInt) hi = Operand('hi', NarrowInt) a = Operand( @@ -1308,6 +1311,10 @@ a = Operand( iconcat = Instruction( 'iconcat', r""" Concatenate low and high bits to form a larger integer type. + + Vectors of integers are concatenated lane-wise such that the result has + the same number of lanes as the inputs, but the lanes are twice the + size. """, ins=(lo, hi), outs=a) From 4c3e590bb92870a53f74103ec57b656ac10d7c1a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 29 Mar 2017 15:16:59 -0700 Subject: [PATCH 0633/3084] Run verifier after legalizer and regalloc file tests. Run the verify_contexti() function after invoking the legalize() and regalloc() context functions. This will help catch bad code produced by these passes. --- cranelift/src/filetest/legalizer.rs | 15 +++++++++------ cranelift/src/filetest/regalloc.rs | 6 ++++-- cranelift/src/utils.rs | 18 +++++++++++++++++- lib/cretonne/src/context.rs | 11 +++++++++++ 4 files changed, 41 insertions(+), 9 deletions(-) diff --git a/cranelift/src/filetest/legalizer.rs b/cranelift/src/filetest/legalizer.rs index 25247a5aba..39854769ac 100644 --- a/cranelift/src/filetest/legalizer.rs +++ b/cranelift/src/filetest/legalizer.rs @@ -4,11 +4,11 @@ //! the result to filecheck. use std::borrow::Cow; -use cretonne::{legalize_function, write_function}; -use cretonne::flowgraph::ControlFlowGraph; +use cretonne::{self, write_function}; use cretonne::ir::Function; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result, run_filecheck}; +use utils::pretty_verifier_error; struct TestLegalizer; @@ -35,13 +35,16 @@ impl SubTest for TestLegalizer { } fn run(&self, func: Cow, context: &Context) -> Result<()> { - let mut func = func.into_owned(); + let mut comp_ctx = cretonne::Context::new(); + comp_ctx.func = func.into_owned(); let isa = context.isa.expect("legalizer needs an ISA"); - let mut cfg = ControlFlowGraph::with_function(&func); - legalize_function(&mut func, &mut cfg, isa); + + comp_ctx.flowgraph(); + comp_ctx.legalize(isa); + comp_ctx.verify(isa).map_err(|e| pretty_verifier_error(&comp_ctx.func, e))?; let mut text = String::new(); - write_function(&mut text, &func, Some(isa)).map_err(|e| e.to_string())?; + write_function(&mut text, &comp_ctx.func, Some(isa)).map_err(|e| e.to_string())?; run_filecheck(&text, context) } } diff --git a/cranelift/src/filetest/regalloc.rs b/cranelift/src/filetest/regalloc.rs index f64d7bb830..5f70f51100 100644 --- a/cranelift/src/filetest/regalloc.rs +++ b/cranelift/src/filetest/regalloc.rs @@ -5,11 +5,12 @@ //! //! The resulting function is sent to `filecheck`. -use std::borrow::Cow; -use cretonne::{self, write_function}; use cretonne::ir::Function; +use cretonne::{self, write_function}; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result, run_filecheck}; +use std::borrow::Cow; +use utils::pretty_verifier_error; struct TestRegalloc; @@ -46,6 +47,7 @@ impl SubTest for TestRegalloc { // TODO: Should we have an option to skip legalization? comp_ctx.legalize(isa); comp_ctx.regalloc(isa); + comp_ctx.verify(isa).map_err(|e| pretty_verifier_error(&comp_ctx.func, e))?; let mut text = String::new(); write_function(&mut text, &comp_ctx.func, Some(isa)).map_err(|e| e.to_string())?; diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index 9cbe20b4bd..d2a6fae3c7 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -1,8 +1,11 @@ //! Utility functions. -use std::path::Path; +use cretonne::ir::entities::AnyEntity; +use cretonne::{ir, verifier, write_function}; +use std::fmt::Write; use std::fs::File; use std::io::{Result, Read}; +use std::path::Path; /// Read an entire file into a string. pub fn read_to_string>(path: P) -> Result { @@ -29,6 +32,19 @@ pub fn match_directive<'a>(comment: &'a str, directive: &str) -> Option<&'a str> } } +/// Pretty-print a verifier error. +pub fn pretty_verifier_error(func: &ir::Function, err: verifier::Error) -> String { + let mut msg = err.to_string(); + match err.location { + AnyEntity::Inst(inst) => { + write!(msg, "\n{}: {}\n\n", inst, func.dfg.display_inst(inst)).unwrap() + } + _ => {} + } + write_function(&mut msg, func, None).unwrap(); + msg +} + #[test] fn test_match_directive() { assert_eq!(match_directive("; foo: bar ", "foo:"), Some("bar")); diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 9ce9d4912e..2d5631f62c 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -15,6 +15,7 @@ use ir::Function; use isa::TargetIsa; use legalize_function; use regalloc; +use verifier; /// Persistent data structures and compilation pipeline. pub struct Context { @@ -45,6 +46,16 @@ impl Context { } } + /// Run the verifier on the function. + /// + /// Also check that the dominator tree and control flow graph are consistent with the function. + /// + /// The `TargetIsa` argument is currently unused, but the verifier will soon be able to also + /// check ISA-dependent constraints. + pub fn verify<'a, ISA: Into>>(&self, _isa: ISA) -> verifier::Result<()> { + verifier::verify_context(self) + } + /// Run the legalizer for `isa` on the function. pub fn legalize(&mut self, isa: &TargetIsa) { legalize_function(&mut self.func, &mut self.cfg, isa); From cb33f93fcd48c7bc57a2328cc78c1ed586f44657 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 29 Mar 2017 15:30:44 -0700 Subject: [PATCH 0634/3084] Use pretty_verifier_error in runone too. --- cranelift/src/filetest/runone.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index ed4cae1ee7..ec61c98240 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -9,7 +9,7 @@ use cretonne::settings::Flags; use cretonne::verify_function; use cton_reader::parse_test; use cton_reader::IsaSpec; -use utils::read_to_string; +use utils::{read_to_string, pretty_verifier_error}; use filetest::{TestResult, new_subtest}; use filetest::subtest::{SubTest, Context, Result}; @@ -116,7 +116,7 @@ fn run_one_test<'a>(tuple: (&'a SubTest, &'a Flags, Option<&'a TargetIsa>), // Should we run the verifier before this test? if !context.verified && test.needs_verifier() { - verify_function(&func).map_err(|e| e.to_string())?; + verify_function(&func).map_err(|e| pretty_verifier_error(&func, e))?; context.verified = true; } From aa06e404569adfa6b8fb9aa8d08dff18e610d361 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 30 Mar 2017 11:33:28 -0700 Subject: [PATCH 0635/3084] Allow dot syntax notation for enumerated immediate operands. The meta language patterns sometimes need to refer to specific values of enumerated immediate operands. The dot syntax provides a namespaced, typed way of doing that: icmp(intcc.ult, a, x). Add an ast.Enumerator class for representing this kind of AST leaf node. Add value definitions for the intcc and floatcc immediate operand kinds. --- lib/cretonne/meta/base/immediates.py | 32 ++++++++++++++++++++++-- lib/cretonne/meta/base/legalize.py | 5 ++-- lib/cretonne/meta/cdsl/ast.py | 36 ++++++++++++++++++++++++++- lib/cretonne/meta/cdsl/operands.py | 37 +++++++++++++++++++++++++--- 4 files changed, 102 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index 05a542b6d9..ff2eb972a5 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -34,7 +34,19 @@ ieee64 = ImmediateKind('ieee64', 'A 64-bit immediate floating point number.') intcc = ImmediateKind( 'intcc', 'An integer comparison condition code.', - default_member='cond', rust_type='IntCC') + default_member='cond', rust_type='IntCC', + values={ + 'eq': 'Equal', + 'ne': 'NotEqual', + 'sge': 'SignedGreaterThanOrEqual', + 'sgt': 'SignedGreaterThan', + 'sle': 'SignedLessThanOrEqual', + 'slt': 'SignedLessThan', + 'uge': 'UnsignedGreaterThanOrEqual', + 'ugt': 'UnsignedGreaterThan', + 'ule': 'UnsignedLessThanOrEqual', + 'ult': 'UnsignedLessThan', + }) #: A condition code for comparing floating point values. #: @@ -43,4 +55,20 @@ intcc = ImmediateKind( floatcc = ImmediateKind( 'floatcc', 'A floating point comparison condition code.', - default_member='cond', rust_type='FloatCC') + default_member='cond', rust_type='FloatCC', + values={ + 'ord': 'Ordered', + 'uno': 'Unordered', + 'eq': 'Equal', + 'ne': 'NotEqual', + 'one': 'OrderedNotEqual', + 'ueq': 'UnorderedOrEqual', + 'lt': 'LessThan', + 'le': 'LessThanOrEqual', + 'gt': 'GreaterThan', + 'ge': 'GreaterThanOrEqual', + 'ult': 'UnorderedOrLessThan', + 'ule': 'UnorderedOrLessThanOrEqual', + 'ugt': 'UnorderedOrGreaterThan', + 'uge': 'UnorderedOrGreaterThanOrEqual', + }) diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index dc8185daf2..8211a0039f 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -7,6 +7,7 @@ patterns that describe how base instructions can be transformed to other base instructions that are legal. """ from __future__ import absolute_import +from .immediates import intcc from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm from .instructions import isub, isub_bin, isub_bout, isub_borrow from .instructions import band, bor, bxor, isplit, iconcat @@ -90,14 +91,14 @@ expand.legalize( (a, c) << iadd_cout(x, y), Rtl( a << iadd(x, y), - c << icmp('IntCC::UnsignedLessThan', a, x) + c << icmp(intcc.ult, a, x) )) expand.legalize( (a, b) << isub_bout(x, y), Rtl( a << isub(x, y), - b << icmp('IntCC::UnsignedGreaterThan', a, x) + b << icmp(intcc.ugt, a, x) )) expand.legalize( diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index 931cc1e3a2..e70e36388a 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -9,7 +9,9 @@ from . import instructions from .typevar import TypeVar try: - from typing import Union, Tuple, Sequence # noqa + from typing import Union, Tuple, Sequence, TYPE_CHECKING # noqa + if TYPE_CHECKING: + from .operands import ImmediateKind # noqa except ImportError: pass @@ -404,3 +406,35 @@ class Apply(Expr): args = defs[0].rust_type() + ', ' + args method = self.inst.snake_name() return '{}({})'.format(method, args) + + +class Enumerator(Expr): + """ + A value of an enumerated immediate operand. + + Some immediate operand kinds like `intcc` and `floatcc` have an enumerated + range of values corresponding to a Rust enum type. An `Enumerator` object + is an AST leaf node representing one of the values. + + :param kind: The enumerated `ImmediateKind` containing the value. + :param value: The textual IL representation of the value. + + `Enumerator` nodes are not usually created directly. They are created by + using the dot syntax on immediate kinds: `intcc.ult`. + """ + + def __init__(self, kind, value): + # type: (ImmediateKind, str) -> None + self.kind = kind + self.value = value + + def __str__(self): + # type: () -> str + """ + Get the Rust expression form of this enumerator. + """ + return self.kind.rust_enumerator(self.value) + + def __repr__(self): + # type: () -> str + return '{}.{}'.format(self.kind, self.value) diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index 49e3e588af..44bae5d500 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -5,8 +5,10 @@ from .types import ValueType from .typevar import TypeVar try: - from typing import Union + from typing import Union, Dict, TYPE_CHECKING # noqa OperandSpec = Union['OperandKind', ValueType, TypeVar] + if TYPE_CHECKING: + from .ast import Enumerator # noqa except ImportError: pass @@ -74,15 +76,44 @@ class ImmediateKind(OperandKind): `InstructionData` data structure. """ - def __init__(self, name, doc, default_member='imm', rust_type=None): - # type: (str, str, str, str) -> None + def __init__( + self, name, doc, + default_member='imm', + rust_type=None, + values=None): + # type: (str, str, str, str, Dict[str, str]) -> None super(ImmediateKind, self).__init__( name, doc, default_member, rust_type) + self.values = values def __repr__(self): # type: () -> str return 'ImmediateKind({})'.format(self.name) + def __getattr__(self, value): + # type: (str) -> Enumerator + """ + Enumerated immediate kinds allow the use of dot syntax to produce + `Enumerator` AST nodes: `icmp.i32(intcc.ult, a, b)`. + """ + from .ast import Enumerator # noqa + if not self.values: + raise AssertionError( + '{n} is not an enumerated operand kind: {n}.{a}'.format( + n=self.name, a=value)) + if value not in self.values: + raise AssertionError( + 'No such {n} enumerator: {n}.{a}'.format( + n=self.name, a=value)) + return Enumerator(self, value) + + def rust_enumerator(self, value): + # type: (str) -> str + """ + Get the qualified Rust name of the enumerator value `value`. + """ + return '{}::{}'.format(self.rust_type, self.values[value]) + # Instances of entity reference operand types are provided in the # `cretonne.entities` module. From 02051c4764ce4864df23a0f3f2e2cab4bfe27f86 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 30 Mar 2017 14:11:19 -0700 Subject: [PATCH 0636/3084] Add mypy annotations to cdsl.predicates, settings. Wherein we learned that only BoolSettings can be used as predicates. --- lib/cretonne/meta/cdsl/isa.py | 11 ++++--- lib/cretonne/meta/cdsl/predicates.py | 39 +++++++++++++++++++++++-- lib/cretonne/meta/cdsl/settings.py | 43 ++++++++++++++++++++++------ 3 files changed, 77 insertions(+), 16 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index ef667015ce..0109c1f12f 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -8,11 +8,10 @@ from .registers import RegClass, Register try: from typing import Tuple, Union, Any, Iterable, Sequence, List, Set, TYPE_CHECKING # noqa from .instructions import MaybeBoundInst, InstructionGroup, InstructionFormat # noqa - from .predicates import Predicate, FieldPredicate # noqa + from .predicates import PredNode # noqa from .settings import SettingGroup # noqa from .types import ValueType # noqa from .registers import RegBank # noqa - AnyPredicate = Union[Predicate, FieldPredicate] OperandConstraint = Union[RegClass, Register, int] ConstraintSeq = Union[OperandConstraint, Tuple[OperandConstraint, ...]] except ImportError: @@ -80,8 +79,8 @@ class TargetISA(object): Ensures that all ISA predicates have an assigned bit number in `self.settings`. """ - self.all_instps = list() # type: List[AnyPredicate] - instps = set() # type: Set[AnyPredicate] + self.all_instps = list() # type: List[PredNode] + instps = set() # type: Set[PredNode] for cpumode in self.cpumodes: for enc in cpumode.encodings: instp = enc.instp @@ -166,7 +165,7 @@ class EncRecipe(object): """ def __init__(self, name, format, ins, outs, instp=None, isap=None): - # type: (str, InstructionFormat, ConstraintSeq, ConstraintSeq, AnyPredicate, AnyPredicate) -> None # noqa + # type: (str, InstructionFormat, ConstraintSeq, ConstraintSeq, PredNode, PredNode) -> None # noqa self.name = name self.format = format self.instp = instp @@ -219,7 +218,7 @@ class Encoding(object): """ def __init__(self, cpumode, inst, recipe, encbits, instp=None, isap=None): - # type: (CPUMode, MaybeBoundInst, EncRecipe, int, AnyPredicate, AnyPredicate) -> None # noqa + # type: (CPUMode, MaybeBoundInst, EncRecipe, int, PredNode, PredNode) -> None # noqa assert isinstance(cpumode, CPUMode) assert isinstance(recipe, EncRecipe) self.inst, self.typevars = inst.fully_bound() diff --git a/lib/cretonne/meta/cdsl/predicates.py b/lib/cretonne/meta/cdsl/predicates.py index 8a1b18807a..82a54c6cf7 100644 --- a/lib/cretonne/meta/cdsl/predicates.py +++ b/lib/cretonne/meta/cdsl/predicates.py @@ -10,7 +10,7 @@ function determine the kind of predicate: - An *Instruction predicate* is evaluated on an instruction instance, so it can inspect all the immediate fields and type variables of the instruction. - Instruction predicates can be evaluatd before register allocation, so they + Instruction predicates can be evaluated before register allocation, so they can not depend on specific register assignments to the value operands or outputs. @@ -24,6 +24,17 @@ predicate, the context is the instruction format. from __future__ import absolute_import from functools import reduce +try: + from typing import Sequence, Tuple, Set, Any, Union, TYPE_CHECKING # noqa + if TYPE_CHECKING: + from .formats import InstructionFormat, FormatField # noqa + from .settings import BoolSetting, SettingGroup # noqa + PredContext = Union[SettingGroup, InstructionFormat] + PredLeaf = Union[BoolSetting, 'FieldPredicate'] + PredNode = Union[PredLeaf, 'Predicate'] +except ImportError: + pass + def _is_parent(a, b): """ @@ -58,7 +69,9 @@ class Predicate(object): """ def __init__(self, parts): - self.name = None + # type: (Sequence[PredNode]) -> None + self.name = None # type: str + self.number = None # type: int self.parts = parts self.context = reduce( _descendant, @@ -66,6 +79,7 @@ class Predicate(object): assert self.context, "Incompatible predicate parts" def __str__(self): + # type: () -> str if self.name: return '{}.{}'.format(self.context.name, self.name) else: @@ -74,15 +88,21 @@ class Predicate(object): ', '.join(map(str, self.parts))) def predicate_context(self): + # type: () -> PredContext return self.context def predicate_leafs(self, leafs): + # type: (Set[PredLeaf]) -> None """ Collect all leaf predicates into the `leafs` set. """ for part in self.parts: part.predicate_leafs(leafs) + def rust_predicate(self, prec): + # type: (int) -> str + raise NotImplementedError("rust_predicate is an abstract method") + class And(Predicate): """ @@ -92,9 +112,11 @@ class And(Predicate): precedence = 2 def __init__(self, *args): + # type: (*PredNode) -> None super(And, self).__init__(args) def rust_predicate(self, prec): + # type: (int) -> str """ Return a Rust expression computing the value of this predicate. @@ -112,6 +134,7 @@ class And(Predicate): @staticmethod def combine(*args): + # type: (*PredNode) -> PredNode """ Combine a sequence of predicates, allowing for `None` members. @@ -135,9 +158,11 @@ class Or(Predicate): precedence = 1 def __init__(self, *args): + # type: (*PredNode) -> None super(Or, self).__init__(args) def rust_predicate(self, prec): + # type: (int) -> str s = ' || '.join(p.rust_predicate(Or.precedence) for p in self.parts) if prec > Or.precedence: s = '({})'.format(s) @@ -152,9 +177,11 @@ class Not(Predicate): precedence = 3 def __init__(self, part): + # type: (PredNode) -> None super(Not, self).__init__((part,)) def rust_predicate(self, prec): + # type: (int) -> str return '!' + self.parts[0].rust_predicate(Not.precedence) @@ -168,15 +195,19 @@ class FieldPredicate(object): """ def __init__(self, field, function, args): + # type: (FormatField, str, Sequence[Any]) -> None + self.number = None # type: int self.field = field self.function = function self.args = args def __str__(self): + # type: () -> str args = (self.field.name,) + tuple(map(str, self.args)) return '{}({})'.format(self.function, ', '.join(args)) def predicate_context(self): + # type: () -> PredContext """ This predicate can be evaluated in the context of an instruction format. @@ -184,9 +215,11 @@ class FieldPredicate(object): return self.field.format def predicate_leafs(self, leafs): + # type: (Set[PredLeaf]) -> None leafs.add(self) def rust_predicate(self, prec): + # type: (int) -> str """ Return a string of Rust code that evaluates this predicate. """ @@ -210,6 +243,7 @@ class IsSignedInt(FieldPredicate): """ def __init__(self, field, width, scale=0): + # type: (FormatField, int, int) -> None super(IsSignedInt, self).__init__( field, 'is_signed_int', (width, scale)) self.width = width @@ -232,6 +266,7 @@ class IsUnsignedInt(FieldPredicate): """ def __init__(self, field, width, scale=0): + # type: (FormatField, int, int) -> None super(IsUnsignedInt, self).__init__( field, 'is_unsigned_int', (width, scale)) self.width = width diff --git a/lib/cretonne/meta/cdsl/settings.py b/lib/cretonne/meta/cdsl/settings.py index 1c1e31643c..ea50afa44b 100644 --- a/lib/cretonne/meta/cdsl/settings.py +++ b/lib/cretonne/meta/cdsl/settings.py @@ -3,6 +3,13 @@ from __future__ import absolute_import from collections import OrderedDict from .predicates import Predicate +try: + from typing import Set, List, Dict, Any, TYPE_CHECKING # noqa + if TYPE_CHECKING: + from .predicates import PredLeaf, PredNode # noqa +except ImportError: + pass + class Setting(object): """ @@ -13,25 +20,26 @@ class Setting(object): """ def __init__(self, doc): - self.name = None # Assigned later by `extract_names()`. + # type: (str) -> None + self.name = None # type: str # Assigned later by `extract_names()`. + self.number = None # type: int self.__doc__ = doc # Offset of byte in settings vector containing this setting. - self.byte_offset = None + self.byte_offset = None # type: int self.group = SettingGroup.append(self) def __str__(self): + # type: () -> str return '{}.{}'.format(self.group.name, self.name) def predicate_context(self): + # type: () -> SettingGroup """ Return the context where this setting can be evaluated as a (leaf) predicate. """ return self.group - def predicate_leafs(self, leafs): - leafs.add(self) - class BoolSetting(Setting): """ @@ -42,10 +50,13 @@ class BoolSetting(Setting): """ def __init__(self, doc, default=False): + # type: (str, bool) -> None super(BoolSetting, self).__init__(doc) self.default = default + self.bit_offset = None # type: int def default_byte(self): + # type: () -> int """ Get the default value of this setting, as a byte that can be bitwise or'ed with the other booleans sharing the same byte. @@ -55,7 +66,12 @@ class BoolSetting(Setting): else: return 0 + def predicate_leafs(self, leafs): + # type: (Set[PredLeaf]) -> None + leafs.add(self) + def rust_predicate(self, prec): + # type: (int) -> str """ Return the Rust code to compute the value of this setting. @@ -74,12 +90,14 @@ class NumSetting(Setting): """ def __init__(self, doc, default=0): + # type: (str, int) -> None super(NumSetting, self).__init__(doc) assert default == int(default) assert default >= 0 and default <= 255 self.default = default def default_byte(self): + # type: () -> int return self.default @@ -94,12 +112,14 @@ class EnumSetting(Setting): """ def __init__(self, doc, *args): + # type: (str, *str) -> None super(EnumSetting, self).__init__(doc) assert len(args) > 0, "EnumSetting must have at least one value" self.values = tuple(str(x) for x in args) self.default = self.values[0] def default_byte(self): + # type: () -> int return 0 @@ -119,23 +139,25 @@ class SettingGroup(object): _current = None # type: SettingGroup def __init__(self, name, parent=None): + # type: (str, SettingGroup) -> None self.name = name self.parent = parent - self.settings = [] + self.settings = [] # type: List[Setting] # Named predicates computed from settings in this group or its # parents. - self.named_predicates = [] + self.named_predicates = [] # type: List[Predicate] # All boolean predicates that can be accessed by number. This includes: # - All boolean settings in this group. # - All named predicates. # - Added anonymous predicates, see `number_predicate()`. # - Added parent predicates that are replicated in this group. # Maps predicate -> number. - self.predicate_number = OrderedDict() + self.predicate_number = OrderedDict() # type: OrderedDict[PredNode, int] # noqa self.open() def open(self): + # type: () -> None """ Open this setting group such that future new settings are added to this group. @@ -146,6 +168,7 @@ class SettingGroup(object): SettingGroup._current = self def close(self, globs=None): + # type: (Dict[str, Any]) -> None """ Close this setting group. This function must be called before opening another setting group. @@ -170,12 +193,14 @@ class SettingGroup(object): @staticmethod def append(setting): + # type: (Setting) -> SettingGroup g = SettingGroup._current assert g, "Open a setting group before defining settings." g.settings.append(setting) return g def number_predicate(self, pred): + # type: (PredNode) -> int """ Make sure that `pred` has an assigned number, and will be included in this group's bit vector. @@ -200,6 +225,7 @@ class SettingGroup(object): return number def layout(self): + # type: () -> None """ Compute the layout of the byte vector used to represent this settings group. @@ -250,6 +276,7 @@ class SettingGroup(object): self.number_predicate(p) def byte_size(self): + # type: () -> int """ Compute the number of bytes required to hold all settings and precomputed predicates. From cfe2c7f46f1163198b9a05219eea426c64a600c3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 30 Mar 2017 15:15:53 -0700 Subject: [PATCH 0637/3084] Add more mypy annotations. --- lib/cretonne/meta/cdsl/ast.py | 5 +++++ lib/cretonne/meta/cdsl/formats.py | 1 + lib/cretonne/meta/cdsl/instructions.py | 25 +++++++++++++++++-------- lib/cretonne/meta/cdsl/isa.py | 6 ++++-- lib/cretonne/meta/cdsl/predicates.py | 2 ++ lib/cretonne/meta/cdsl/registers.py | 14 +++++++++----- lib/cretonne/meta/cdsl/types.py | 3 ++- lib/cretonne/meta/cdsl/typevar.py | 20 +++++++++++++++----- lib/cretonne/meta/cdsl/xform.py | 9 +++------ lib/cretonne/meta/constant_hash.py | 9 ++++++++- lib/cretonne/meta/gen_build_deps.py | 7 +++++++ lib/cretonne/meta/gen_legalizer.py | 2 ++ lib/cretonne/meta/gen_types.py | 9 +++++++++ lib/cretonne/meta/srcgen.py | 4 ++++ lib/cretonne/meta/unique_table.py | 17 +++++++++++++---- 15 files changed, 101 insertions(+), 32 deletions(-) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index e70e36388a..d2296e8fdb 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -47,9 +47,11 @@ class Def(object): self.expr = expr def __repr__(self): + # type: () -> str return "{} << {!r}".format(self.defs, self.expr) def __str__(self): + # type: () -> str if len(self.defs) == 1: return "{!s} << {!s}".format(self.defs[0], self.expr) else: @@ -379,15 +381,18 @@ class Apply(Expr): return Def(other, self) def instname(self): + # type: () -> str i = self.inst.name for t in self.typevars: i += '.{}'.format(t) return i def __repr__(self): + # type: () -> str return "Apply({}, {})".format(self.instname(), self.args) def __str__(self): + # type: () -> str args = ', '.join(map(str, self.args)) return '{}({})'.format(self.instname(), args) diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index 6233dbf7a6..9549eac1e0 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -193,6 +193,7 @@ class InstructionFormat(object): @staticmethod def extract_names(globs): + # type: (Dict[str, Any]) -> None """ Given a dict mapping name -> object as returned by `globals()`, find all the InstructionFormat objects and set their name from the dict key. diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 30596fd1fb..a4b0f7fe1f 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -6,11 +6,13 @@ from .operands import Operand from .formats import InstructionFormat try: - from typing import Union, Sequence, List # noqa - # List of operands for ins/outs: - OpList = Union[Sequence[Operand], Operand] - MaybeBoundInst = Union['Instruction', 'BoundInstruction'] - from typing import Tuple, Any # noqa + from typing import Union, Sequence, List, Tuple, Any, TYPE_CHECKING # noqa + if TYPE_CHECKING: + from .ast import Expr, Apply # noqa + from .typevar import TypeVar # noqa + # List of operands for ins/outs: + OpList = Union[Sequence[Operand], Operand] + MaybeBoundInst = Union['Instruction', 'BoundInstruction'] except ImportError: pass @@ -122,6 +124,7 @@ class Instruction(object): InstructionGroup.append(self) def __str__(self): + # type: () -> str prefix = ', '.join(o.name for o in self.outs) if prefix: prefix = prefix + ' = ' @@ -141,6 +144,7 @@ class Instruction(object): return self.name def blurb(self): + # type: () -> str """Get the first line of the doc comment""" for line in self.__doc__.split('\n'): line = line.strip() @@ -149,6 +153,7 @@ class Instruction(object): return "" def _verify_polymorphic(self): + # type: () -> None """ Check if this instruction is polymorphic, and verify its use of type variables. @@ -193,6 +198,7 @@ class Instruction(object): self.ctrl_typevar = tv def _verify_ctrl_typevar(self, ctrl_typevar): + # type: (TypeVar) -> List[TypeVar] """ Verify that the use of TypeVars is consistent with `ctrl_typevar` as the controlling type variable. @@ -204,7 +210,7 @@ class Instruction(object): Return list of other type variables used, or raise an error. """ - other_tvs = [] + other_tvs = [] # type: List[TypeVar] # Check value inputs. for opnum in self.value_opnums: typ = self.ins[opnum].typevar @@ -283,11 +289,12 @@ class Instruction(object): return (self, ()) def __call__(self, *args): + # type: (*Expr) -> Apply """ Create an `ast.Apply` AST node representing the application of this instruction to the arguments. """ - from .ast import Apply + from .ast import Apply # noqa return Apply(self, args) @@ -303,6 +310,7 @@ class BoundInstruction(object): assert len(typevars) <= 1 + len(inst.other_typevars) def __str__(self): + # type: () -> str return '.'.join([self.inst.name, ] + list(map(str, self.typevars))) def bind(self, *args): @@ -336,9 +344,10 @@ class BoundInstruction(object): return (self.inst, self.typevars) def __call__(self, *args): + # type: (*Expr) -> Apply """ Create an `ast.Apply` AST node representing the application of this instruction to the arguments. """ - from .ast import Apply + from .ast import Apply # noqa return Apply(self, args) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 0109c1f12f..18abd9807a 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -97,6 +97,7 @@ class TargetISA(object): self.settings.number_predicate(enc.isap) def _collect_regclasses(self): + # type: () -> None """ Collect and number register classes. @@ -132,6 +133,7 @@ class CPUMode(object): return self.name def enc(self, *args, **kwargs): + # type: (*Any, **Any) -> None """ Add a new encoding to this CPU mode. @@ -186,7 +188,7 @@ class EncRecipe(object): return self.name def _verify_constraints(self, seq): - # (ConstraintSeq) -> Sequence[OperandConstraint] + # type: (ConstraintSeq) -> Sequence[OperandConstraint] if not isinstance(seq, tuple): seq = (seq,) for c in seq: @@ -194,7 +196,7 @@ class EncRecipe(object): # An integer constraint is bound to a value operand. # Check that it is in range. assert c >= 0 - if not format.has_value_list: + if not self.format.has_value_list: assert c < self.format.num_value_operands else: assert isinstance(c, RegClass) or isinstance(c, Register) diff --git a/lib/cretonne/meta/cdsl/predicates.py b/lib/cretonne/meta/cdsl/predicates.py index 82a54c6cf7..4cfe8fffb5 100644 --- a/lib/cretonne/meta/cdsl/predicates.py +++ b/lib/cretonne/meta/cdsl/predicates.py @@ -37,6 +37,7 @@ except ImportError: def _is_parent(a, b): + # type: (PredContext, PredContext) -> bool """ Return true if a is a parent of b, or equal to it. """ @@ -46,6 +47,7 @@ def _is_parent(a, b): def _descendant(a, b): + # type: (PredContext, PredContext) -> PredContext """ If a is a parent of b or b is a parent of a, return the descendant of the two. diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index 003ce6379c..b21d2348a6 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -26,11 +26,12 @@ from . import is_power_of_two, next_power_of_two try: - from typing import Sequence, Tuple, List, Dict # noqa - from .isa import TargetISA # noqa - # A tuple uniquely identifying a register class inside a register bank. - # (count, width, start) - RCTup = Tuple[int, int, int] + from typing import Sequence, Tuple, List, Dict, Any, TYPE_CHECKING # noqa + if TYPE_CHECKING: + from .isa import TargetISA # noqa + # A tuple uniquely identifying a register class inside a register bank. + # (count, width, start) + RCTup = Tuple[int, int, int] except ImportError: pass @@ -189,6 +190,7 @@ class RegClass(object): bank.classes.append(self) def __str__(self): + # type: () -> str return self.name def rctup(self): @@ -223,6 +225,7 @@ class RegClass(object): return (count, self.width, start) def __getitem__(self, sliced): + # type: (slice) -> RegClass """ Create a sub-class of a register class using slice notation. The slice indexes refer to allocations in the parent register class, not register @@ -273,6 +276,7 @@ class RegClass(object): @staticmethod def extract_names(globs): + # type: (Dict[str, Any]) -> None """ Given a dict mapping name -> object as returned by `globals()`, find all the RegClass objects and set their name from the dict key. diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py index f96c5a2ef0..e42460af28 100644 --- a/lib/cretonne/meta/cdsl/types.py +++ b/lib/cretonne/meta/cdsl/types.py @@ -21,11 +21,12 @@ class ValueType(object): _registry = dict() # type: Dict[str, ValueType] # List of all the scalar types. - all_scalars = list() # type: List[ValueType] + all_scalars = list() # type: List[ScalarType] def __init__(self, name, membytes, doc): # type: (str, int, str) -> None self.name = name + self.number = None # type: int self.membytes = membytes self.__doc__ = doc assert name not in ValueType._registry diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 908ebbd8cd..84d099657c 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -9,10 +9,12 @@ import math from . import types, is_power_of_two try: - from typing import Tuple, Union # noqa - Interval = Tuple[int, int] - # An Interval where `True` means 'everything' - BoolInterval = Union[bool, Interval] + from typing import Tuple, Union, TYPE_CHECKING # noqa + if TYPE_CHECKING: + from srcgen import Formatter # noqa + Interval = Tuple[int, int] + # An Interval where `True` means 'everything' + BoolInterval = Union[bool, Interval] except ImportError: pass @@ -143,7 +145,11 @@ class TypeSet(object): return h def __eq__(self, other): - return self.typeset_key() == other.typeset_key() + # type: (object) -> bool + if isinstance(other, TypeSet): + return self.typeset_key() == other.typeset_key() + else: + return False def __repr__(self): # type: () -> str @@ -157,6 +163,7 @@ class TypeSet(object): return s + ')' def emit_fields(self, fmt): + # type: (Formatter) -> None """Emit field initializers for this typeset.""" fmt.comment(repr(self)) fields = ('lanes', 'int', 'float', 'bool') @@ -299,6 +306,9 @@ class TypeVar(object): .format(self.name, self.type_set)) def __eq__(self, other): + # type: (object) -> bool + if not isinstance(other, TypeVar): + return False if self.is_derived and other.is_derived: return ( self.derived_func == other.derived_func and diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index 21c6103bcd..9d284bc1e2 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -40,10 +40,6 @@ class Rtl(object): # type: (*DefApply) -> None self.rtl = tuple(map(canonicalize_defapply, args)) - def __iter__(self): - # type: () -> Iterator[Def] - return iter(self.rtl) - class XForm(object): """ @@ -105,10 +101,11 @@ class XForm(object): self._collect_typevars() def __repr__(self): + # type: () -> str s = "XForm(inputs={}, defs={},\n ".format(self.inputs, self.defs) - s += '\n '.join(str(n) for n in self.src) + s += '\n '.join(str(n) for n in self.src.rtl) s += '\n=>\n ' - s += '\n '.join(str(n) for n in self.dst) + s += '\n '.join(str(n) for n in self.dst.rtl) s += '\n)' return s diff --git a/lib/cretonne/meta/constant_hash.py b/lib/cretonne/meta/constant_hash.py index 8c3c355aa1..96560ffc87 100644 --- a/lib/cretonne/meta/constant_hash.py +++ b/lib/cretonne/meta/constant_hash.py @@ -8,8 +8,14 @@ quadratically probed hash table. from __future__ import absolute_import from cdsl import next_power_of_two +try: + from typing import Any, List, Sequence, Callable # noqa +except ImportError: + pass + def simple_hash(s): + # type: (str) -> int """ Compute a primitive hash of a string. @@ -26,6 +32,7 @@ def simple_hash(s): def compute_quadratic(items, hash_function): + # type: (Sequence[Any], Callable[[Any], int]) -> List[Any] """ Compute an open addressed, quadratically probed hash table containing `items`. The returned table is a list containing the elements of the @@ -43,7 +50,7 @@ def compute_quadratic(items, hash_function): items = list(items) # Table size must be a power of two. Aim for >20% unused slots. size = next_power_of_two(int(1.20*len(items))) - table = [None] * size + table = [None] * size # type: List[Any] for i in items: h = hash_function(i) % size diff --git a/lib/cretonne/meta/gen_build_deps.py b/lib/cretonne/meta/gen_build_deps.py index 191113a759..5e1419284c 100644 --- a/lib/cretonne/meta/gen_build_deps.py +++ b/lib/cretonne/meta/gen_build_deps.py @@ -16,8 +16,14 @@ from __future__ import absolute_import, print_function import os from os.path import dirname, abspath, join +try: + from typing import Iterable # noqa +except ImportError: + pass + def source_files(top): + # type: (str) -> Iterable[str] """ Recursively find all interesting source files and directories in the directory tree starting at top. Yield a path to each file. @@ -30,6 +36,7 @@ def source_files(top): def generate(): + # type: () -> None print("Dependencies from meta language directory:") meta = dirname(abspath(__file__)) for path in source_files(meta): diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 9131b5d19e..93f2bc857c 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -14,6 +14,7 @@ from cdsl.ast import Var try: from typing import Sequence # noqa + from cdsl.isa import TargetISA # noqa from cdsl.ast import Def # noqa from cdsl.xform import XForm, XFormGroup # noqa except ImportError: @@ -240,6 +241,7 @@ def gen_xform_group(xgrp, fmt): def generate(isas, out_dir): + # type: (Sequence[TargetISA], str) -> None fmt = Formatter() gen_xform_group(legalize.narrow, fmt) gen_xform_group(legalize.expand, fmt) diff --git a/lib/cretonne/meta/gen_types.py b/lib/cretonne/meta/gen_types.py index 5ffed14cf4..fa033c5bdb 100644 --- a/lib/cretonne/meta/gen_types.py +++ b/lib/cretonne/meta/gen_types.py @@ -12,8 +12,14 @@ import srcgen from cdsl.types import ValueType import base.types # noqa +try: + from typing import Iterable # noqa +except ImportError: + pass + def emit_type(ty, fmt): + # type: (ValueType, srcgen.Formatter) -> None """ Emit a constant definition of a single value type. """ @@ -25,6 +31,7 @@ def emit_type(ty, fmt): def emit_vectors(bits, fmt): + # type: (int, srcgen.Formatter) -> None """ Emit definition for all vector types with `bits` total size. """ @@ -37,6 +44,7 @@ def emit_vectors(bits, fmt): def emit_types(fmt): + # type: (srcgen.Formatter) -> None for ty in ValueType.all_scalars: emit_type(ty, fmt) # Emit vector definitions for common SIMD sizes. @@ -47,6 +55,7 @@ def emit_types(fmt): def generate(out_dir): + # type: (str) -> None fmt = srcgen.Formatter() emit_types(fmt) fmt.update_file('types.rs', out_dir) diff --git a/lib/cretonne/meta/srcgen.py b/lib/cretonne/meta/srcgen.py index e87c16a577..d91ecc4ab2 100644 --- a/lib/cretonne/meta/srcgen.py +++ b/lib/cretonne/meta/srcgen.py @@ -101,6 +101,7 @@ class Formatter(object): self.fmt.indent_push() def __exit__(self, t, v, tb): + # type: (object, object, object) -> None self.fmt.indent_pop() if self.after: self.fmt.line(self.after) @@ -126,13 +127,16 @@ class Formatter(object): return Formatter._IndentedScope(self, after) def format(self, fmt, *args): + # type: (str, *Any) -> None self.line(fmt.format(*args)) def comment(self, s): + # type: (str) -> None """Add a comment line.""" self.line('// ' + s) def doc_comment(self, s): + # type: (str) -> None """Add a (multi-line) documentation comment.""" s = re.sub('^', self.indent + '/// ', s, flags=re.M) + '\n' self.lines.append(s) diff --git a/lib/cretonne/meta/unique_table.py b/lib/cretonne/meta/unique_table.py index cbc45af200..0e18ae3afd 100644 --- a/lib/cretonne/meta/unique_table.py +++ b/lib/cretonne/meta/unique_table.py @@ -7,18 +7,25 @@ item is mapped to its offset in the final array. This is a compression technique for compile-time generated tables. """ +try: + from typing import Any, List, Dict, Tuple, Sequence # noqa +except ImportError: + pass + class UniqueTable: """ Collect items into the `table` list, removing duplicates. """ def __init__(self): + # type: () -> None # List of items added in order. - self.table = list() + self.table = list() # type: List[Any] # Map item -> index. - self.index = dict() + self.index = dict() # type: Dict[Any, int] def add(self, item): + # type: (Any) -> int """ Add a single item to the table if it isn't already there. @@ -40,11 +47,13 @@ class UniqueSeqTable: Sequences don't have to be of the same length. """ def __init__(self): - self.table = list() + # type: () -> None + self.table = list() # type: List[Any] # Map seq -> index. - self.index = dict() + self.index = dict() # type: Dict[Tuple[Any, ...], int] def add(self, seq): + # type: (Sequence[Any]) -> int """ Add a sequence of items to the table. If the table already contains the items in `seq` in the same order, use those instead. From 353dc72b5b06612966d9d1025e0368a45e79035f Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Thu, 30 Mar 2017 23:34:50 +0100 Subject: [PATCH 0638/3084] Verify integrity of the existing control flow graph of the context. (#70) * Verify integrity of the existing control flow graph of the context. * Make checking more thorough. --- lib/cretonne/src/ir/entities.rs | 2 +- lib/cretonne/src/verifier.rs | 88 +++++++++++++++++++-------------- 2 files changed, 52 insertions(+), 38 deletions(-) diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 51ca70ad7a..42d109bf7a 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -170,7 +170,7 @@ impl Display for Value { } /// An opaque reference to an instruction in a function. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] pub struct Inst(u32); entity_impl!(Inst, "inst"); diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 433b4bc84d..37dc5e20f7 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -61,6 +61,7 @@ use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpT use Context; use std::fmt::{self, Display, Formatter}; use std::result; +use std::collections::BTreeSet; /// A verifier error. #[derive(Debug, PartialEq, Eq)] @@ -106,6 +107,7 @@ pub fn verify_function(func: &Function) -> Result<()> { pub fn verify_context(ctx: &Context) -> Result<()> { let verifier = Verifier::new(&ctx.func); verifier.domtree_integrity(&ctx.domtree)?; + verifier.cfg_integrity(&ctx.cfg)?; verifier.run() } @@ -350,41 +352,6 @@ impl<'a> Verifier<'a> { Ok(()) } - fn cfg_integrity(&self, ebb: Ebb) -> Result<()> { - for &(pred_ebb, pred_inst) in self.cfg.get_predecessors(ebb) { - // All predecessors in the CFG must be branches to the EBB - match self.func.dfg[pred_inst].analyze_branch(&self.func.dfg.value_lists) { - BranchInfo::SingleDest(target_ebb, _) => { - if target_ebb != ebb { - return err!(ebb, - "has predecessor {} in {} which does not branch here", - pred_inst, - pred_ebb); - } - } - BranchInfo::Table(jt) => { - if !self.func.jump_tables[jt].branches_to(ebb) { - return err!(ebb, - "has predecessor {} using {} in {} which never branches here", - pred_inst, - jt, - pred_ebb); - } - } - BranchInfo::NotABranch => { - return err!(ebb, "has predecessor {} which is not a branch", pred_inst); - } - } - // All EBBs branching to `ebb` have it recorded as a successor in the CFG. - if !self.cfg.get_successors(pred_ebb).contains(&ebb) { - return err!(ebb, - "predecessor {} does not have this EBB recorded as a successor", - pred_ebb); - } - } - Ok(()) - } - fn domtree_integrity(&self, domtree: &DominatorTree) -> Result<()> { // We consider two `DominatorTree`s to be equal if they return the same immediate // dominator for each EBB. Therefore the current domtree is valid if it matches the freshly @@ -614,6 +581,54 @@ impl<'a> Verifier<'a> { Ok(()) } + fn cfg_integrity(&self, cfg: &ControlFlowGraph) -> Result<()> { + let mut expected_succs = BTreeSet::::new(); + let mut got_succs = BTreeSet::::new(); + let mut expected_preds = BTreeSet::::new(); + let mut got_preds = BTreeSet::::new(); + + for ebb in self.func.layout.ebbs() { + expected_succs.extend(self.cfg.get_successors(ebb)); + got_succs.extend(cfg.get_successors(ebb)); + + let missing_succs: Vec = expected_succs.difference(&got_succs).cloned().collect(); + if missing_succs.len() != 0 { + return err!(ebb, + "cfg lacked the following successor(s) {:?}", + missing_succs); + } + + let excess_succs: Vec = got_succs.difference(&expected_succs).cloned().collect(); + if excess_succs.len() != 0 { + return err!(ebb, "cfg had unexpected successor(s) {:?}", excess_succs); + } + + expected_preds.extend(self.cfg + .get_predecessors(ebb) + .iter() + .map(|&(_, inst)| inst)); + got_preds.extend(cfg.get_predecessors(ebb).iter().map(|&(_, inst)| inst)); + + let missing_preds: Vec = expected_preds.difference(&got_preds).cloned().collect(); + if missing_preds.len() != 0 { + return err!(ebb, + "cfg lacked the following predecessor(s) {:?}", + missing_preds); + } + + let excess_preds: Vec = got_preds.difference(&expected_preds).cloned().collect(); + if excess_preds.len() != 0 { + return err!(ebb, "cfg had unexpected predecessor(s) {:?}", excess_preds); + } + + expected_succs.clear(); + got_succs.clear(); + expected_preds.clear(); + got_preds.clear(); + } + Ok(()) + } + pub fn run(&self) -> Result<()> { self.typecheck_entry_block_arguments()?; for ebb in self.func.layout.ebbs() { @@ -622,7 +637,6 @@ impl<'a> Verifier<'a> { self.instruction_integrity(inst)?; self.typecheck(inst)?; } - self.cfg_integrity(ebb)?; } Ok(()) } @@ -668,4 +682,4 @@ mod tests { let verifier = Verifier::new(&func); assert_err_with_msg!(verifier.run(), "instruction format"); } -} +} \ No newline at end of file From 1a066c4054432279f879311327d1a111f1e1dfe8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 30 Mar 2017 16:20:40 -0700 Subject: [PATCH 0639/3084] Add mypy types for gen_instr.py. Declare the Instruction.number opcode number field. --- lib/cretonne/meta/cdsl/instructions.py | 3 +++ lib/cretonne/meta/gen_instr.py | 22 ++++++++++++++++------ 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index a4b0f7fe1f..a900c95ab6 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -107,6 +107,9 @@ class Instruction(object): self.outs = self._to_operand_tuple(outs) self.format = InstructionFormat.lookup(self.ins, self.outs) + # Opcode number, assigned by gen_instr.py. + self.number = None # type: int + # Indexes into `self.outs` for value results. # Other results are `variable_args`. self.value_results = tuple( diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 0b473e60f3..dc5257ffef 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -8,15 +8,18 @@ from unique_table import UniqueTable, UniqueSeqTable from cdsl import camel_case from cdsl.operands import ImmediateKind from cdsl.formats import InstructionFormat - -from cdsl.instructions import Instruction # noqa -from cdsl.operands import Operand # noqa -from cdsl.typevar import TypeVar # noqa +from cdsl.instructions import Instruction # The typing module is only required by mypy, and we don't use these imports # outside type comments. try: - from typing import List # noqa + from typing import List, Sequence, Set, TYPE_CHECKING # noqa + if TYPE_CHECKING: + from cdsl.isa import TargetISA # noqa + from cdsl.instructions import InstructionGroup # noqa + from cdsl.operands import Operand # noqa + from cdsl.typevar import TypeVar # noqa + except ImportError: pass @@ -260,7 +263,8 @@ def gen_instruction_data_impl(fmt): def collect_instr_groups(isas): - seen = set() + # type: (Sequence[TargetISA]) -> List[InstructionGroup] + seen = set() # type: Set[InstructionGroup] groups = [] for isa in isas: for g in isa.instruction_groups: @@ -271,6 +275,7 @@ def collect_instr_groups(isas): def gen_opcodes(groups, fmt): + # type: (Sequence[InstructionGroup], srcgen.Formatter) -> Sequence[Instruction] # noqa """ Generate opcode enumerations. @@ -389,6 +394,7 @@ def get_constraint(op, ctrl_typevar, type_sets): def gen_type_constraints(fmt, instrs): + # type: (srcgen.Formatter, Sequence[Instruction]) -> None """ Generate value type constraints for all instructions. @@ -487,6 +493,7 @@ def gen_type_constraints(fmt, instrs): def gen_format_constructor(iform, fmt): + # type: (InstructionFormat, srcgen.Formatter) -> None """ Emit a method for creating and inserting inserting an `iform` instruction, where `iform` is an instruction format. @@ -543,6 +550,7 @@ def gen_format_constructor(iform, fmt): def gen_member_inits(iform, fmt): + # type: (InstructionFormat, srcgen.Formatter) -> None """ Emit member initializers for an `iform` instruction. """ @@ -696,6 +704,7 @@ def gen_inst_builder(inst, fmt): def gen_builder(insts, fmt): + # type: (Sequence[Instruction], srcgen.Formatter) -> None """ Generate a Builder trait with methods for all instructions. """ @@ -723,6 +732,7 @@ def gen_builder(insts, fmt): def generate(isas, out_dir): + # type: (Sequence[TargetISA], str) -> None groups = collect_instr_groups(isas) # opcodes.rs From 469ecd91bf89d598caba512c9d4c91c68eed1a11 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 30 Mar 2017 17:07:24 -0700 Subject: [PATCH 0640/3084] Add mypy types for gen_encoding.py. --- lib/cretonne/meta/gen_encoding.py | 72 ++++++++++++++++++++++--------- 1 file changed, 52 insertions(+), 20 deletions(-) diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 978519efaf..6ee10eb632 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -57,15 +57,21 @@ from collections import OrderedDict, defaultdict import math import itertools from cdsl.registers import RegClass, Register +from cdsl.predicates import FieldPredicate try: - from typing import Sequence # noqa - from cdsl.isa import TargetISA, OperandConstraint # noqa + from typing import Sequence, Set, Tuple, List, Iterable, DefaultDict, TYPE_CHECKING # noqa + if TYPE_CHECKING: + from cdsl.isa import TargetISA, OperandConstraint, Encoding, CPUMode # noqa + from cdsl.predicates import PredNode, PredLeaf # noqa + from cdsl.types import ValueType # noqa + from cdsl.instructions import Instruction # noqa except ImportError: pass def emit_instp(instp, fmt): + # type: (PredNode, srcgen.Formatter) -> None """ Emit code for matching an instruction predicate against an `InstructionData` reference called `inst`. @@ -77,11 +83,15 @@ def emit_instp(instp, fmt): # Which fields do we need in the InstructionData pattern match? # Collect the leaf predicates. - leafs = set() + leafs = set() # type: Set[PredLeaf] instp.predicate_leafs(leafs) # All the leafs are FieldPredicate instances. Here we just care about # the field names. - fields = ', '.join(sorted(set(p.field.name for p in leafs))) + fnames = set() # type: Set[str] + for p in leafs: + assert isinstance(p, FieldPredicate) + fnames.add(p.field.name) + fields = ', '.join(sorted(fnames)) with fmt.indented('{} => {{'.format(instp.number), '}'): with fmt.indented( @@ -91,6 +101,7 @@ def emit_instp(instp, fmt): def emit_instps(instps, fmt): + # type: (Sequence[PredNode], srcgen.Formatter) -> None """ Emit a function for matching instruction predicates. """ @@ -141,6 +152,7 @@ CODE_FAIL = (1 << CODE_BITS) - 1 def seq_doc(enc): + # type: (Encoding) -> Tuple[Tuple[int, int, int], str] """ Return a tuple containing u16 representations of the instruction predicate an recipe / encbits. @@ -169,13 +181,15 @@ class EncList(object): """ def __init__(self, inst, ty): + # type: (Instruction, ValueType) -> None self.inst = inst self.ty = ty # List of applicable Encoding instances. # These will have different predicates. - self.encodings = [] + self.encodings = [] # type: List[Encoding] def name(self): + # type: () -> str name = self.inst.name if self.ty: name = '{}.{}'.format(name, self.ty.name) @@ -184,6 +198,7 @@ class EncList(object): return name def by_isap(self): + # type: () -> Iterable[Tuple[PredNode, Tuple[Encoding, ...]]] """ Group the encodings by ISA predicate without reordering them. @@ -192,17 +207,18 @@ class EncList(object): have the same ISA predicate. """ maxlen = CODE_FAIL >> PRED_BITS - for isap, group in itertools.groupby( + for isap, groupi in itertools.groupby( self.encodings, lambda enc: enc.isap): - group = tuple(group) + group = tuple(groupi) # This probably never happens, but we can't express more than # maxlen encodings per isap. while len(group) > maxlen: - yield (isap, group[0..maxlen]) + yield (isap, group[0:maxlen]) group = group[maxlen:] yield (isap, group) def encode(self, seq_table, doc_table, isa): + # type: (UniqueSeqTable, DefaultDict[int, List[str]], TargetISA) -> None # noqa """ Encode this list as a sequence of u16 numbers. @@ -211,8 +227,8 @@ class EncList(object): Adds comment lines to `doc_table` keyed by seq_table offsets. """ - words = list() - docs = list() + words = list() # type: List[int] + docs = list() # type: List[Tuple[int, str]] # Group our encodings by isap. for isap, group in self.by_isap(): @@ -249,21 +265,25 @@ class Level2Table(object): """ def __init__(self, ty): + # type: (ValueType) -> None self.ty = ty # Maps inst -> EncList - self.lists = OrderedDict() + self.lists = OrderedDict() # type: OrderedDict[Instruction, EncList] def __getitem__(self, inst): + # type: (Instruction) -> EncList ls = self.lists.get(inst) if not ls: ls = EncList(inst, self.ty) self.lists[inst] = ls return ls - def __iter__(self): + def enclists(self): + # type: () -> Iterable[EncList] return iter(self.lists.values()) def layout_hashtable(self, level2_hashtables, level2_doc): + # type: (List[EncList], DefaultDict[int, List[str]]) -> None """ Compute the hash table mapping opcode -> enclist. @@ -290,20 +310,24 @@ class Level1Table(object): """ def __init__(self): - self.tables = OrderedDict() + # type: () -> None + self.tables = OrderedDict() # type: OrderedDict[ValueType, Level2Table] # noqa def __getitem__(self, ty): + # type: (ValueType) -> Level2Table tbl = self.tables.get(ty) if not tbl: tbl = Level2Table(ty) self.tables[ty] = tbl return tbl - def __iter__(self): + def l2tables(self): + # type: () -> Iterable[Level2Table] return iter(self.tables.values()) def make_tables(cpumode): + # type: (CPUMode) -> Level1Table """ Generate tables for `cpumode` as described above. """ @@ -316,15 +340,17 @@ def make_tables(cpumode): def encode_enclists(level1, seq_table, doc_table, isa): + # type: (Level1Table, UniqueSeqTable, DefaultDict[int, List[str]], TargetISA) -> None # noqa """ Compute encodings and doc comments for encoding lists in `level1`. """ - for level2 in level1: - for enclist in level2: + for level2 in level1.l2tables(): + for enclist in level2.enclists(): enclist.encode(seq_table, doc_table, isa) def emit_enclists(seq_table, doc_table, fmt): + # type: (UniqueSeqTable, DefaultDict[int, List[str]], srcgen.Formatter) -> None # noqa with fmt.indented( 'pub static ENCLISTS: [u16; {}] = ['.format(len(seq_table.table)), '];'): @@ -342,11 +368,13 @@ def emit_enclists(seq_table, doc_table, fmt): def encode_level2_hashtables(level1, level2_hashtables, level2_doc): - for level2 in level1: + # type: (Level1Table, List[EncList], DefaultDict[int, List[str]]) -> None + for level2 in level1.l2tables(): level2.layout_hashtable(level2_hashtables, level2_doc) def emit_level2_hashtables(level2_hashtables, offt, level2_doc, fmt): + # type: (List[EncList], str, DefaultDict[int, List[str]], srcgen.Formatter) -> None # noqa """ Emit the big concatenation of level 2 hash tables. """ @@ -370,6 +398,7 @@ def emit_level2_hashtables(level2_hashtables, offt, level2_doc, fmt): def emit_level1_hashtable(cpumode, level1, offt, fmt): + # type: (CPUMode, Level1Table, str, srcgen.Formatter) -> None # noqa """ Emit a level 1 hash table for `cpumode`. """ @@ -399,6 +428,7 @@ def emit_level1_hashtable(cpumode, level1, offt, fmt): def offset_type(length): + # type: (int) -> str """ Compute an appropriate Rust integer type to use for offsets into a table of the given length. @@ -467,6 +497,7 @@ def emit_operand_constraints(seq, field, fmt): def gen_isa(isa, fmt): + # type: (TargetISA, srcgen.Formatter) -> None # First assign numbers to relevant instruction predicates and generate the # check_instp() function.. emit_instps(isa.all_instps, fmt) @@ -476,11 +507,11 @@ def gen_isa(isa, fmt): # Tables for enclists with comments. seq_table = UniqueSeqTable() - doc_table = defaultdict(list) + doc_table = defaultdict(list) # type: DefaultDict[int, List[str]] # Single table containing all the level2 hash tables. - level2_hashtables = list() - level2_doc = defaultdict(list) + level2_hashtables = list() # type: List[EncList] + level2_doc = defaultdict(list) # type: DefaultDict[int, List[str]] for cpumode in isa.cpumodes: level2_doc[len(level2_hashtables)].append(cpumode.name) @@ -505,6 +536,7 @@ def gen_isa(isa, fmt): def generate(isas, out_dir): + # type: (Sequence[TargetISA], str) -> None for isa in isas: fmt = srcgen.Formatter() gen_isa(isa, fmt) From fc979c474f83848958b8ad65123e8027de8a7e0d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 30 Mar 2017 18:42:06 -0700 Subject: [PATCH 0641/3084] Add mypy types for gen_settings.py. --- lib/cretonne/meta/cdsl/predicates.py | 2 ++ lib/cretonne/meta/cdsl/settings.py | 10 ++++++++++ lib/cretonne/meta/gen_settings.py | 20 ++++++++++++++++++++ 3 files changed, 32 insertions(+) diff --git a/lib/cretonne/meta/cdsl/predicates.py b/lib/cretonne/meta/cdsl/predicates.py index 4cfe8fffb5..8ba4c8eb41 100644 --- a/lib/cretonne/meta/cdsl/predicates.py +++ b/lib/cretonne/meta/cdsl/predicates.py @@ -202,6 +202,8 @@ class FieldPredicate(object): self.field = field self.function = function self.args = args + # All PredNode members must have a name field. This will never be set. + self.name = None # type: str def __str__(self): # type: () -> str diff --git a/lib/cretonne/meta/cdsl/settings.py b/lib/cretonne/meta/cdsl/settings.py index ea50afa44b..d9404060dd 100644 --- a/lib/cretonne/meta/cdsl/settings.py +++ b/lib/cretonne/meta/cdsl/settings.py @@ -26,6 +26,9 @@ class Setting(object): self.__doc__ = doc # Offset of byte in settings vector containing this setting. self.byte_offset = None # type: int + # Index into the generated DESCRIPTORS table. + self.descriptor_index = None # type: int + self.group = SettingGroup.append(self) def __str__(self): @@ -40,6 +43,10 @@ class Setting(object): """ return self.group + def default_byte(self): + # type: () -> int + raise NotImplementedError("default_byte is an abstract method") + class BoolSetting(Setting): """ @@ -154,6 +161,9 @@ class SettingGroup(object): # Maps predicate -> number. self.predicate_number = OrderedDict() # type: OrderedDict[PredNode, int] # noqa + # Fully qualified Rust module name. See gen_settings.py. + self.qual_mod = None # type: str + self.open() def open(self): diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index 6429333d28..847a72399d 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -9,8 +9,18 @@ from cdsl import camel_case from cdsl.settings import BoolSetting, NumSetting, EnumSetting from base import settings +try: + from typing import Sequence, Set, Tuple, List, TYPE_CHECKING # noqa + if TYPE_CHECKING: + from cdsl.isa import TargetISA # noqa + from cdsl.settings import Setting, SettingGroup # noqa + from cdsl.predicates import Predicate, PredContext # noqa +except ImportError: + pass + def gen_enum_types(sgrp, fmt): + # type: (SettingGroup, srcgen.Formatter) -> None """ Emit enum types for any enum settings. """ @@ -27,6 +37,7 @@ def gen_enum_types(sgrp, fmt): def gen_getter(setting, sgrp, fmt): + # type: (Setting, SettingGroup, srcgen.Formatter) -> None """ Emit a getter function for `setting`. """ @@ -57,6 +68,7 @@ def gen_getter(setting, sgrp, fmt): def gen_pred_getter(pred, sgrp, fmt): + # type: (Predicate, SettingGroup, srcgen.Formatter) -> None """ Emit a getter for a pre-computed predicate. """ @@ -69,6 +81,7 @@ def gen_pred_getter(pred, sgrp, fmt): def gen_getters(sgrp, fmt): + # type: (SettingGroup, srcgen.Formatter) -> None """ Emit getter functions for all the settings in fmt. """ @@ -87,6 +100,7 @@ def gen_getters(sgrp, fmt): def gen_descriptors(sgrp, fmt): + # type: (SettingGroup, srcgen.Formatter) -> None """ Generate the DESCRIPTORS and ENUMERATORS tables. """ @@ -125,6 +139,7 @@ def gen_descriptors(sgrp, fmt): fmt.line('"{}",'.format(txt)) def hash_setting(s): + # type: (Setting) -> int return constant_hash.simple_hash(s.name) hash_table = constant_hash.compute_quadratic(sgrp.settings, hash_setting) @@ -140,6 +155,7 @@ def gen_descriptors(sgrp, fmt): def gen_template(sgrp, fmt): + # type: (SettingGroup, srcgen.Formatter) -> None """ Emit a Template constant. """ @@ -164,6 +180,7 @@ def gen_template(sgrp, fmt): def gen_display(sgrp, fmt): + # type: (SettingGroup, srcgen.Formatter) -> None """ Generate the Display impl for Flags. """ @@ -182,6 +199,7 @@ def gen_display(sgrp, fmt): def gen_constructor(sgrp, parent, fmt): + # type: (SettingGroup, PredContext, srcgen.Formatter) -> None """ Generate a Flags constructor. """ @@ -235,6 +253,7 @@ def gen_constructor(sgrp, parent, fmt): def gen_group(sgrp, fmt): + # type: (SettingGroup, srcgen.Formatter) -> None """ Generate a Flags struct representing `sgrp`. """ @@ -252,6 +271,7 @@ def gen_group(sgrp, fmt): def generate(isas, out_dir): + # type: (Sequence[TargetISA], str) -> None # Generate shared settings. fmt = srcgen.Formatter() settings.group.qual_mod = 'settings' From 98ecebd85b6a659940ac765e8c77d7c267cbe2d6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 30 Mar 2017 19:52:06 -0700 Subject: [PATCH 0642/3084] Add a mypy.ini file and enable some more warnings. Also require all Python functions to have a type declaration. --- lib/cretonne/meta/build.py | 2 +- lib/cretonne/meta/cdsl/typevar.py | 2 +- lib/cretonne/meta/gen_encoding.py | 14 ++++++++------ lib/cretonne/meta/mypy.ini | 5 +++++ 4 files changed, 15 insertions(+), 8 deletions(-) create mode 100644 lib/cretonne/meta/mypy.ini diff --git a/lib/cretonne/meta/build.py b/lib/cretonne/meta/build.py index 08ba14a358..a94b9ce706 100644 --- a/lib/cretonne/meta/build.py +++ b/lib/cretonne/meta/build.py @@ -18,7 +18,7 @@ parser = argparse.ArgumentParser(description='Generate sources for Cretonne.') parser.add_argument('--out-dir', help='set output directory') args = parser.parse_args() -out_dir = args.out_dir # type: ignore +out_dir = args.out_dir isas = isa.all_isas() diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 84d099657c..2c5141d78c 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -60,7 +60,7 @@ def decode_interval(intv, full_range, default=None): """ if isinstance(intv, tuple): # mypy buig here: 'builtins.None' object is not iterable - lo, hi = intv # type: ignore + lo, hi = intv assert is_power_of_two(lo) assert is_power_of_two(hi) assert lo <= hi diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 6ee10eb632..e78d8942dc 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -289,9 +289,10 @@ class Level2Table(object): Append the hash table to `level2_hashtables` and record the offset. """ - hash_table = compute_quadratic( - self.lists.values(), - lambda enclist: enclist.inst.number) + def hash_func(enclist): + # type: (EncList) -> int + return enclist.inst.number + hash_table = compute_quadratic(self.lists.values(), hash_func) self.hash_table_offset = len(level2_hashtables) self.hash_table_len = len(hash_table) @@ -402,9 +403,10 @@ def emit_level1_hashtable(cpumode, level1, offt, fmt): """ Emit a level 1 hash table for `cpumode`. """ - hash_table = compute_quadratic( - level1.tables.values(), - lambda level2: level2.ty.number) + def hash_func(level2): + # type: (Level2Table) -> int + return level2.ty.number + hash_table = compute_quadratic(level1.tables.values(), hash_func) with fmt.indented( 'pub static LEVEL1_{}: [Level1Entry<{}>; {}] = [' diff --git a/lib/cretonne/meta/mypy.ini b/lib/cretonne/meta/mypy.ini new file mode 100644 index 0000000000..ca7f5e4c00 --- /dev/null +++ b/lib/cretonne/meta/mypy.ini @@ -0,0 +1,5 @@ +[mypy] +python_version = 2.7 +disallow_untyped_defs = True +warn_unused_ignores = True +warn_return_any = True From 2f50ae31661c09038e7e3136b6eeef23be7eeaf2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 31 Mar 2017 10:12:44 -0700 Subject: [PATCH 0643/3084] Add an IsEqual FieldPredicate. Compare an immediate operand to a constant value. --- lib/cretonne/meta/cdsl/predicates.py | 15 +++++++++++++++ lib/cretonne/src/predicates.rs | 6 ++++++ 2 files changed, 21 insertions(+) diff --git a/lib/cretonne/meta/cdsl/predicates.py b/lib/cretonne/meta/cdsl/predicates.py index 8ba4c8eb41..e75920d948 100644 --- a/lib/cretonne/meta/cdsl/predicates.py +++ b/lib/cretonne/meta/cdsl/predicates.py @@ -232,6 +232,21 @@ class FieldPredicate(object): return 'predicates::{}({})'.format(self.function, ', '.join(args)) +class IsEqual(FieldPredicate): + """ + Instruction predicate that checks if an immediate instruction format field + is equal to a constant value. + + :param field: `FormatField` to be checked. + :param value: The constant value to compare against. + """ + + def __init__(self, field, value): + # type: (FormatField, Any) -> None + super(IsEqual, self).__init__(field, 'is_equal', (value,)) + self.value = value + + class IsSignedInt(FieldPredicate): """ Instruction predicate that checks if an immediate instruction format field diff --git a/lib/cretonne/src/predicates.rs b/lib/cretonne/src/predicates.rs index 748a75e23a..770857d100 100644 --- a/lib/cretonne/src/predicates.rs +++ b/lib/cretonne/src/predicates.rs @@ -9,6 +9,12 @@ //! Some of these predicates may be unused in certain ISA configurations, so we suppress the //! dead_code warning. +/// Check that `x` is the same as `y`. +#[allow(dead_code)] +pub fn is_equal(x: T, y: T) -> bool { + x == y +} + /// Check that `x` can be represented as a `wd`-bit signed integer with `sc` low zero bits. #[allow(dead_code)] pub fn is_signed_int>(x: T, wd: u8, sc: u8) -> bool { From 63a6ade0a555bf3797ffeb17d1479eef859746e9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 31 Mar 2017 10:12:37 -0700 Subject: [PATCH 0644/3084] Add InstructionFormat.imm_fields. Consolidate the imm_members and imm_kinds into this list so the FormatField is the single definition of these properties. This makes it easier to access the precomputed FormatFields parametrically, avoiding going through getattr(). This is better for type checking too. --- lib/cretonne/meta/cdsl/formats.py | 73 ++++++++++++++-------------- lib/cretonne/meta/cdsl/predicates.py | 2 +- lib/cretonne/meta/gen_encoding.py | 2 +- lib/cretonne/meta/gen_instr.py | 8 +-- lib/cretonne/meta/gen_legalizer.py | 6 +-- 5 files changed, 46 insertions(+), 45 deletions(-) diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index 9549eac1e0..f08d6fa119 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -59,17 +59,15 @@ class InstructionFormat(object): self.name = kwargs.get('name', None) # type: str self.multiple_results = kwargs.get('multiple_results', False) - # Struct member names for the immediate operands. All other instruction - # operands are values or variable argument lists. They are all handled - # specially. - self.imm_members = list() # type: List[str] # The number of value operands stored in the format, or `None` when # `has_value_list` is set. self.num_value_operands = 0 # Does this format use a value list for storing value operands? self.has_value_list = False - # Operand kinds for the immediate operands. - self.imm_kinds = tuple(self._process_member_names(kinds)) + # Operand fields for the immediate operands. All other instruction + # operands are values or variable argument lists. They are all handled + # specially. + self.imm_fields = tuple(self._process_member_names(kinds)) # The typevar_operand argument must point to a 'value' operand. self.typevar_operand = kwargs.get('typevar_operand', None) # type: int @@ -82,9 +80,9 @@ class InstructionFormat(object): self.typevar_operand = 0 # Compute a signature for the global registry. + imm_kinds = tuple(f.kind for f in self.imm_fields) sig = ( - self.multiple_results, self.imm_kinds, - self.num_value_operands, + self.multiple_results, imm_kinds, self.num_value_operands, self.has_value_list) if sig in InstructionFormat._registry: raise RuntimeError( @@ -94,7 +92,7 @@ class InstructionFormat(object): InstructionFormat.all_formats.append(self) def _process_member_names(self, kinds): - # type: (Sequence[Union[OperandKind, Tuple[str, OperandKind]]]) -> Iterable[OperandKind] # noqa + # type: (Sequence[Union[OperandKind, Tuple[str, OperandKind]]]) -> Iterable[FormatField] # noqa """ Extract names of all the immediate operands in the kinds tuple. @@ -102,10 +100,11 @@ class InstructionFormat(object): pair. The member names correspond to members in the Rust `InstructionData` data structure. - Update the fields `num_value_operands` and `imm_members`. + Updates the fields `self.num_value_operands` and `self.has_value_list`. - Yields the immediate operand kinds. + Yields the immediate operand fields. """ + inum = 0 for arg in kinds: if isinstance(arg, OperandKind): member = arg.default_member @@ -119,13 +118,13 @@ class InstructionFormat(object): elif k is VARIABLE_ARGS: self.has_value_list = True else: - self.imm_members.append(member) - yield k + yield FormatField(self, inum, k, member) + inum += 1 def __str__(self): # type: () -> str - args = ', '.join('{}: {}'.format(m, k) - for m, k in zip(self.imm_members, self.imm_kinds)) + args = ', '.join( + '{}: {}'.format(f.member, f.kind) for f in self.imm_fields) return '{}(imms=({}), vals={})'.format( self.name, args, self.num_value_operands) @@ -137,16 +136,16 @@ class InstructionFormat(object): Each non-value format member becomes a corresponding `FormatField` attribute. """ - try: - i = self.imm_members.index(attr) - except ValueError: - raise AttributeError( - '{} is neither a {} member or a ' - .format(attr, self.name) + - 'normal InstructionFormat attribute') - field = FormatField(self, i, attr) - setattr(self, attr, field) - return field + for f in self.imm_fields: + if f.member == attr: + # Cache this field attribute so we won't have to search again. + setattr(self, attr, f) + return f + + raise AttributeError( + '{} is neither a {} member or a ' + .format(attr, self.name) + + 'normal InstructionFormat attribute') @staticmethod def lookup(ins, outs): @@ -207,26 +206,28 @@ class InstructionFormat(object): class FormatField(object): """ - A field in an instruction format. + An immediate field in an instruction format. This corresponds to a single member of a variant of the `InstructionData` data type. - :param format: Parent `InstructionFormat`. - :param operand: Immediate operand number in parent. - :param name: Member name in `InstructionData` variant. + :param iformat: Parent `InstructionFormat`. + :param immnum: Immediate operand number in parent. + :param kind: Immediate Operand kind. + :param member: Member name in `InstructionData` variant. """ - def __init__(self, format, operand, name): - # type: (InstructionFormat, int, str) -> None - self.format = format - self.operand = operand - self.name = name + def __init__(self, iform, immnum, kind, member): + # type: (InstructionFormat, int, OperandKind, str) -> None + self.format = iform + self.immnum = immnum + self.kind = kind + self.member = member def __str__(self): # type: () -> str - return '{}.{}'.format(self.format.name, self.name) + return '{}.{}'.format(self.format.name, self.member) def rust_name(self): # type: () -> str - return self.name + return self.member diff --git a/lib/cretonne/meta/cdsl/predicates.py b/lib/cretonne/meta/cdsl/predicates.py index e75920d948..273f8be9a1 100644 --- a/lib/cretonne/meta/cdsl/predicates.py +++ b/lib/cretonne/meta/cdsl/predicates.py @@ -207,7 +207,7 @@ class FieldPredicate(object): def __str__(self): # type: () -> str - args = (self.field.name,) + tuple(map(str, self.args)) + args = (self.field.rust_name(),) + tuple(map(str, self.args)) return '{}({})'.format(self.function, ', '.join(args)) def predicate_context(self): diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index e78d8942dc..e4faa6410c 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -90,7 +90,7 @@ def emit_instp(instp, fmt): fnames = set() # type: Set[str] for p in leafs: assert isinstance(p, FieldPredicate) - fnames.add(p.field.name) + fnames.add(p.field.rust_name()) fields = ', '.join(sorted(fnames)) with fmt.indented('{} => {{'.format(instp.number), '}'): diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index dc5257ffef..9ce279dc52 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -515,8 +515,8 @@ def gen_format_constructor(iform, fmt): result_type = 'result_type' # Normal operand arguments. Start with the immediate operands. - for kind, name in zip(iform.imm_kinds, iform.imm_members): - args.append('{}: {}'.format(name, kind.rust_type)) + for f in iform.imm_fields: + args.append('{}: {}'.format(f.member, f.kind.rust_type)) # Then the value operands. if iform.has_value_list: # Take all value arguments as a finished value list. The value lists @@ -557,8 +557,8 @@ def gen_member_inits(iform, fmt): # Immediate operands. # We have local variables with the same names as the members. - for member in iform.imm_members: - fmt.line('{}: {},'.format(member, member)) + for f in iform.imm_fields: + fmt.line('{}: {},'.format(f.member, f.member)) # Value operands. if iform.has_value_list: diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 93f2bc857c..7beda84771 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -43,8 +43,8 @@ def unwrap_inst(iref, node, fmt): 'let ({}) = if let InstructionData::{} {{' .format(', '.join(map(str, expr.args)), iform.name), '};'): # Fields are encoded directly. - for m in iform.imm_members: - fmt.line('{},'.format(m)) + for f in iform.imm_fields: + fmt.line('{},'.format(f.member)) if nvops == 1: fmt.line('arg,') elif iform.has_value_list or nvops > 1: @@ -58,7 +58,7 @@ def unwrap_inst(iref, node, fmt): for opnum, op in enumerate(expr.inst.ins): if op.is_immediate(): n = expr.inst.imm_opnums.index(opnum) - outs.append(iform.imm_members[n]) + outs.append(iform.imm_fields[n].member) elif op.is_value(): if nvops == 1: arg = 'arg' From 0403845c408c08ca737bbf2ea5b604db41002f0a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 31 Mar 2017 10:12:23 -0700 Subject: [PATCH 0645/3084] Add Apply.inst_predicate(). Compute an instruction predicate from any constant values given as arguments for the immediate operands in an instruction pattern. Allows for patterns like icmp.i32(intcc.ugt, x, y) or iadd_imm.i32(x, 1) Trap these predicates in the legalizer code generator since we can't actually handle them yet. --- lib/cretonne/meta/cdsl/ast.py | 28 ++++++++++++++++++++++++++++ lib/cretonne/meta/cdsl/predicates.py | 3 ++- lib/cretonne/meta/gen_legalizer.py | 6 ++++++ 3 files changed, 36 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index d2296e8fdb..2b671fc46e 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -7,11 +7,13 @@ for patern matching an rewriting of cretonne instructions. from __future__ import absolute_import from . import instructions from .typevar import TypeVar +from .predicates import IsEqual, And try: from typing import Union, Tuple, Sequence, TYPE_CHECKING # noqa if TYPE_CHECKING: from .operands import ImmediateKind # noqa + from .predicates import PredNode # noqa except ImportError: pass @@ -412,6 +414,32 @@ class Apply(Expr): method = self.inst.snake_name() return '{}({})'.format(method, args) + def inst_predicate(self): + # type: () -> PredNode + """ + Construct an instruction predicate that verifies the immediate operands + on this instruction. + + Immediate operands in a source pattern can be either free variables or + constants like `Enumerator`. We don't currently support constraints on + free variables, but we may in the future. + """ + pred = None # type: PredNode + iform = self.inst.format + + # Examine all of the immediate operands. + for ffield, opnum in zip(iform.imm_fields, self.inst.imm_opnums): + arg = self.args[opnum] + + # Ignore free variables for now. We may add variable predicates + # later. + if isinstance(arg, Var): + continue + + pred = And.combine(pred, IsEqual(ffield, arg)) + + return pred + class Enumerator(Expr): """ diff --git a/lib/cretonne/meta/cdsl/predicates.py b/lib/cretonne/meta/cdsl/predicates.py index 273f8be9a1..8a1cc147f9 100644 --- a/lib/cretonne/meta/cdsl/predicates.py +++ b/lib/cretonne/meta/cdsl/predicates.py @@ -216,7 +216,8 @@ class FieldPredicate(object): This predicate can be evaluated in the context of an instruction format. """ - return self.field.format + iform = self.field.format # type: InstructionFormat + return iform def predicate_leafs(self, leafs): # type: (Set[PredLeaf]) -> None diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 7beda84771..41cd9bab56 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -211,6 +211,12 @@ def gen_xform(xform, fmt): # variables. unwrap_inst('inst', xform.src.rtl[0], fmt) + # We could support instruction predicates, but not yet. Should we just + # return false if it fails? What about multiple patterns with different + # predicates for the same opcode? + instp = xform.src.rtl[0].expr.inst_predicate() + assert instp is None, "Instruction predicates not supported in legalizer" + # Emit the destination pattern. for dst in xform.dst.rtl: emit_dst_inst(dst, fmt) From 75b156e1c72c0c33468a4c1a2e5c5b6d2ec75e89 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 31 Mar 2017 11:58:59 -0700 Subject: [PATCH 0646/3084] Allow for instructions with operands in encodings. When defining an instruction encoding, allow part of the instruction predicate to be provided as operands on the instruction opcode: icmp.i32(intcc.ult, x, y) This generates an instruction predicate that checks IntCompare.cond == IntCC::UnsignedLessThan --- lib/cretonne/meta/cdsl/isa.py | 39 +++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 9 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 18abd9807a..9db0faeb61 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -2,18 +2,23 @@ from __future__ import absolute_import from .predicates import And from .registers import RegClass, Register +from .ast import Apply # The typing module is only required by mypy, and we don't use these imports # outside type comments. try: from typing import Tuple, Union, Any, Iterable, Sequence, List, Set, TYPE_CHECKING # noqa - from .instructions import MaybeBoundInst, InstructionGroup, InstructionFormat # noqa - from .predicates import PredNode # noqa - from .settings import SettingGroup # noqa - from .types import ValueType # noqa - from .registers import RegBank # noqa - OperandConstraint = Union[RegClass, Register, int] - ConstraintSeq = Union[OperandConstraint, Tuple[OperandConstraint, ...]] + if TYPE_CHECKING: + from .instructions import MaybeBoundInst, InstructionGroup, InstructionFormat # noqa + from .predicates import PredNode # noqa + from .settings import SettingGroup # noqa + from .types import ValueType # noqa + from .registers import RegBank # noqa + OperandConstraint = Union[RegClass, Register, int] + ConstraintSeq = Union[OperandConstraint, Tuple[OperandConstraint, ...]] + # Instruction specification for encodings. Allows for predicated + # instructions. + InstSpec = Union[MaybeBoundInst, Apply] except ImportError: pass @@ -210,6 +215,14 @@ class Encoding(object): An `Encoding` object ties an instruction opcode with concrete type variables together with and encoding recipe and encoding bits. + The concrete instruction can be in three different forms: + + 1. A naked opcode: `trap` for non-polymorphic instructions. + 2. With bound type variables: `iadd.i32` for polymorphic instructions. + 3. With operands providing constraints: `icmp.i32(intcc.eq, x, y)`. + + If the instruction is polymorphic, all type variables must be provided. + :param cpumode: The CPU mode where the encoding is active. :param inst: The :py:class:`Instruction` or :py:class:`BoundInstruction` being encoded. @@ -220,10 +233,18 @@ class Encoding(object): """ def __init__(self, cpumode, inst, recipe, encbits, instp=None, isap=None): - # type: (CPUMode, MaybeBoundInst, EncRecipe, int, PredNode, PredNode) -> None # noqa + # type: (CPUMode, InstSpec, EncRecipe, int, PredNode, PredNode) -> None # noqa assert isinstance(cpumode, CPUMode) assert isinstance(recipe, EncRecipe) - self.inst, self.typevars = inst.fully_bound() + + # Check for possible instruction predicates in `inst`. + if isinstance(inst, Apply): + instp = And.combine(instp, inst.inst_predicate()) + self.inst = inst.inst + self.typevars = inst.typevars + else: + self.inst, self.typevars = inst.fully_bound() + self.cpumode = cpumode assert self.inst.format == recipe.format, ( "Format {} must match recipe: {}".format( From e9907fbdd676d17aaf426b85cf526d20191fe73d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 31 Mar 2017 12:12:09 -0700 Subject: [PATCH 0647/3084] Add RISC-V encodings for supported icmp variants. Only slt and ult variants are in the instruction set. Other condition codes must be synthesized. --- cranelift/filetests/isa/riscv/binary32.cton | 8 ++- lib/cretonne/meta/isa/riscv/encodings.py | 15 ++++- lib/cretonne/meta/isa/riscv/recipes.py | 5 +- lib/cretonne/src/isa/riscv/binemit.rs | 63 ++++++++++++++------- lib/cretonne/src/isa/riscv/enc_tables.rs | 1 + 5 files changed, 69 insertions(+), 23 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index ede783b7df..0c6863eae1 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -14,7 +14,6 @@ ebb0: ; sub [-,%x7] v12 = isub v1, v2 ; bin: 415503b3 [-,%x16] v13 = isub v2, v1 ; bin: 40aa8833 - ; TBD: slt/sltu ; and [-,%x7] v20 = band v1, v2 ; bin: 015573b3 [-,%x16] v21 = band v2, v1 ; bin: 00aaf833 @@ -33,5 +32,12 @@ ebb0: ; sra [-,%x7] v34 = sshr v1, v2 ; bin: 415553b3 [-,%x16] v35 = sshr v2, v1 ; bin: 40aad833 + ; slt + [-,%x7] v42 = icmp slt, v1, v2 ; bin: 015523b3 + [-,%x16] v43 = icmp slt, v2, v1 ; bin: 00aaa833 + ; sltu + [-,%x7] v44 = icmp ult, v1, v2 ; bin: 015533b3 + [-,%x16] v45 = icmp ult, v2, v1 ; bin: 00aab833 + return } diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index bf84c468b0..b97c92386a 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -3,9 +3,15 @@ RISC-V Encodings. """ from __future__ import absolute_import from base import instructions as base +from base.immediates import intcc from .defs import RV32, RV64 -from .recipes import OPIMM, OPIMM32, OP, OP32, JALR, R, Rshamt, I, Iret +from .recipes import OPIMM, OPIMM32, OP, OP32, JALR, R, Rshamt, Ricmp, I, Iret from .settings import use_m +from cdsl.ast import Var + +# Dummies for instruction predicates. +x = Var('x') +y = Var('y') # Basic arithmetic binary instructions are encoded in an R-type instruction. for inst, inst_imm, f3, f7 in [ @@ -47,6 +53,13 @@ for inst, inst_imm, f3, f7 in [ RV64.enc(inst_imm.i64, Rshamt, OPIMM(f3, f7)) RV64.enc(inst_imm.i32, Rshamt, OPIMM32(f3, f7)) +# Signed and unsigned integer 'less than'. There are no 'w' variants for +# comparing 32-bit numbers in RV64. +RV32.enc(base.icmp.i32(intcc.slt, x, y), Ricmp, OP(0b010, 0b0000000)) +RV64.enc(base.icmp.i64(intcc.slt, x, y), Ricmp, OP(0b010, 0b0000000)) +RV32.enc(base.icmp.i32(intcc.ult, x, y), Ricmp, OP(0b011, 0b0000000)) +RV64.enc(base.icmp.i64(intcc.ult, x, y), Ricmp, OP(0b011, 0b0000000)) + # "M" Standard Extension for Integer Multiplication and Division. # Gated by the `use_m` flag. RV32.enc(base.imul.i32, R, OP(0b000, 0b0000001), isap=use_m) diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 0dce091713..2fd326e5cc 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -11,7 +11,7 @@ instruction formats described in the reference: from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt -from base.formats import Binary, BinaryImm, MultiAry +from base.formats import Binary, BinaryImm, MultiAry, IntCompare from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit @@ -79,6 +79,9 @@ R = EncRecipe('R', Binary, ins=(GPR, GPR), outs=GPR) # R-type with an immediate shift amount instead of rs2. Rshamt = EncRecipe('Rshamt', BinaryImm, ins=GPR, outs=GPR) +# R-type encoding of an integer comparison. +Ricmp = EncRecipe('Ricmp', IntCompare, ins=(GPR, GPR), outs=GPR) + I = EncRecipe( 'I', BinaryImm, ins=GPR, outs=GPR, instp=IsSignedInt(BinaryImm.imm, 12)) diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index e7dcc6021f..a2e29a847e 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -2,6 +2,7 @@ use binemit::{CodeSink, bad_encoding}; use ir::{Function, Inst, InstructionData}; +use isa::RegUnit; include!(concat!(env!("OUT_DIR"), "/binemit-riscv.rs")); @@ -12,33 +13,55 @@ include!(concat!(env!("OUT_DIR"), "/binemit-riscv.rs")); /// 25 20 15 12 7 0 /// /// Encoding bits: `opcode[6:2] | (funct3 << 5) | (funct7 << 8)`. +fn put_r(bits: u16, + rs1: RegUnit, + rs2: RegUnit, + rd: RegUnit, + sink: &mut CS) { + let bits = bits as u32; + let opcode5 = bits & 0x1f; + let funct3 = (bits >> 5) & 0x7; + let funct7 = (bits >> 8) & 0x7f; + let rs1 = rs1 as u32 & 0x1f; + let rs2 = rs2 as u32 & 0x1f; + let rd = rd as u32 & 0x1f; + + // 0-6: opcode + let mut i = 0x3; + i |= opcode5 << 2; + i |= rd << 7; + i |= funct3 << 12; + i |= rs1 << 15; + i |= rs2 << 20; + i |= funct7 << 25; + + sink.put4(i); +} + fn recipe_r(func: &Function, inst: Inst, sink: &mut CS) { if let InstructionData::Binary { args, .. } = func.dfg[inst] { - let bits = func.encodings[inst].bits(); - let rs1 = func.locations[args[0]].unwrap_reg(); - let rs2 = func.locations[args[1]].unwrap_reg(); - let rd = func.locations[func.dfg.first_result(inst)].unwrap_reg(); - - // 0-6: opcode - let mut i = 0x3; - i |= (bits as u32 & 0x1f) << 2; - // 7-11: rd - i |= (rd as u32 & 0x1f) << 7; - // 12-14: funct3 - i |= ((bits as u32 >> 5) & 0x7) << 12; - // 15-19: rs1 - i |= (rs1 as u32 & 0x1f) << 15; - // 20-24: rs1 - i |= (rs2 as u32 & 0x1f) << 20; - // 25-31: funct7 - i |= ((bits as u32 >> 8) & 0x7f) << 25; - - sink.put4(i); + put_r(func.encodings[inst].bits(), + func.locations[args[0]].unwrap_reg(), + func.locations[args[1]].unwrap_reg(), + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); } else { panic!("Expected Binary format: {:?}", func.dfg[inst]); } } +fn recipe_ricmp(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::IntCompare { args, .. } = func.dfg[inst] { + put_r(func.encodings[inst].bits(), + func.locations[args[0]].unwrap_reg(), + func.locations[args[1]].unwrap_reg(), + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); + } else { + panic!("Expected IntCompare format: {:?}", func.dfg[inst]); + } +} + fn recipe_rshamt(_func: &Function, _inst: Inst, _sink: &mut CS) { unimplemented!() } diff --git a/lib/cretonne/src/isa/riscv/enc_tables.rs b/lib/cretonne/src/isa/riscv/enc_tables.rs index f02ccf80ba..7940fce004 100644 --- a/lib/cretonne/src/isa/riscv/enc_tables.rs +++ b/lib/cretonne/src/isa/riscv/enc_tables.rs @@ -1,5 +1,6 @@ //! Encoding tables for RISC-V. +use ir::condcodes::IntCC; use ir::{Opcode, InstructionData}; use ir::types; use predicates; From c9a14448b8392247c41b6dfa1de25213e47d9bec Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 31 Mar 2017 15:15:20 -0700 Subject: [PATCH 0648/3084] Emit I-type instructions for RISC-V. These are the BinaryImm formats. --- cranelift/filetests/isa/riscv/binary32.cton | 20 +++++++++++ lib/cretonne/src/isa/riscv/binemit.rs | 37 +++++++++++++++++++-- 2 files changed, 55 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 0c6863eae1..68bc718ea7 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -39,5 +39,25 @@ ebb0: [-,%x7] v44 = icmp ult, v1, v2 ; bin: 015533b3 [-,%x16] v45 = icmp ult, v2, v1 ; bin: 00aab833 + ; Integer Register-Immediate Instructions + + ; addi + [-,%x7] v100 = iadd_imm v1, 1000 ; bin: 3e850393 + [-,%x16] v101 = iadd_imm v2, -905 ; bin: c77a8813 + ; TBD: slti + ; andi + [-,%x7] v110 = band_imm v1, 1000 ; bin: 3e857393 + [-,%x16] v111 = band_imm v2, -905 ; bin: c77af813 + ; ori + [-,%x7] v112 = bor_imm v1, 1000 ; bin: 3e856393 + [-,%x16] v113 = bor_imm v2, -905 ; bin: c77ae813 + ; xori + [-,%x7] v114 = bxor_imm v1, 1000 ; bin: 3e854393 + [-,%x16] v115 = bxor_imm v2, -905 ; bin: c77ac813 + + ; TBD: slli + ; TBD: srli + ; TBD: srai + return } diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index a2e29a847e..480d377c6b 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -66,8 +66,41 @@ fn recipe_rshamt(_func: &Function, _inst: Inst, _sink: &m unimplemented!() } -fn recipe_i(_func: &Function, _inst: Inst, _sink: &mut CS) { - unimplemented!() +/// I-type instructions. +/// +/// 31 19 14 11 6 +/// imm rs1 funct3 rd opcode +/// 20 15 12 7 0 +/// +/// Encoding bits: `opcode[6:2] | (funct3 << 5)` +fn put_i(bits: u16, rs1: RegUnit, imm: i64, rd: RegUnit, sink: &mut CS) { + let bits = bits as u32; + let opcode5 = bits & 0x1f; + let funct3 = (bits >> 5) & 0x7; + let rs1 = rs1 as u32 & 0x1f; + let rd = rd as u32 & 0x1f; + + // 0-6: opcode + let mut i = 0x3; + i |= opcode5 << 2; + i |= rd << 7; + i |= funct3 << 12; + i |= rs1 << 15; + i |= (imm << 20) as u32; + + sink.put4(i); +} + +fn recipe_i(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::BinaryImm { arg, imm, .. } = func.dfg[inst] { + put_i(func.encodings[inst].bits(), + func.locations[arg].unwrap_reg(), + imm.into(), + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); + } else { + panic!("Expected Binary format: {:?}", func.dfg[inst]); + } } fn recipe_iret(_func: &Function, _inst: Inst, _sink: &mut CS) { From 9ddbc5d8611b6178d233e5c974631f0e1ee5b8b7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 31 Mar 2017 15:33:21 -0700 Subject: [PATCH 0649/3084] Emit Rshamt-type instructions for RISC-V. These are the shift-by-immediate instructions. --- cranelift/filetests/isa/riscv/binary32.cton | 12 ++++-- lib/cretonne/src/isa/riscv/binemit.rs | 48 +++++++++++++++++++-- 2 files changed, 54 insertions(+), 6 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 68bc718ea7..fd3d910a18 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -55,9 +55,15 @@ ebb0: [-,%x7] v114 = bxor_imm v1, 1000 ; bin: 3e854393 [-,%x16] v115 = bxor_imm v2, -905 ; bin: c77ac813 - ; TBD: slli - ; TBD: srli - ; TBD: srai + ; slli + [-,%x7] v120 = ishl_imm v1, 31 ; bin: 01f51393 + [-,%x16] v121 = ishl_imm v2, 8 ; bin: 008a9813 + ; srli + [-,%x7] v122 = ushr_imm v1, 31 ; bin: 01f55393 + [-,%x16] v123 = ushr_imm v2, 8 ; bin: 008ad813 + ; srai + [-,%x7] v124 = sshr_imm v1, 31 ; bin: 41f55393 + [-,%x16] v125 = sshr_imm v2, 8 ; bin: 408ad813 return } diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 480d377c6b..a3941af526 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -38,6 +38,40 @@ fn put_r(bits: u16, sink.put4(i); } +/// R-type instructions with a shift amount instead of rs2. +/// +/// 31 25 19 14 11 6 +/// funct7 shamt rs1 funct3 rd opcode +/// 25 20 15 12 7 0 +/// +/// Both funct7 and shamt contribute to bit 25. In RV64, shamt uses it for shifts > 31. +/// +/// Encoding bits: `opcode[6:2] | (funct3 << 5) | (funct7 << 8)`. +fn put_rshamt(bits: u16, + rs1: RegUnit, + shamt: i64, + rd: RegUnit, + sink: &mut CS) { + let bits = bits as u32; + let opcode5 = bits & 0x1f; + let funct3 = (bits >> 5) & 0x7; + let funct7 = (bits >> 8) & 0x7f; + let rs1 = rs1 as u32 & 0x1f; + let shamt = shamt as u32 & 0x3f; + let rd = rd as u32 & 0x1f; + + // 0-6: opcode + let mut i = 0x3; + i |= opcode5 << 2; + i |= rd << 7; + i |= funct3 << 12; + i |= rs1 << 15; + i |= shamt << 20; + i |= funct7 << 25; + + sink.put4(i); +} + fn recipe_r(func: &Function, inst: Inst, sink: &mut CS) { if let InstructionData::Binary { args, .. } = func.dfg[inst] { put_r(func.encodings[inst].bits(), @@ -62,8 +96,16 @@ fn recipe_ricmp(func: &Function, inst: Inst, sink: &mut C } } -fn recipe_rshamt(_func: &Function, _inst: Inst, _sink: &mut CS) { - unimplemented!() +fn recipe_rshamt(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::BinaryImm { arg, imm, .. } = func.dfg[inst] { + put_rshamt(func.encodings[inst].bits(), + func.locations[arg].unwrap_reg(), + imm.into(), + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); + } else { + panic!("Expected BinaryImm format: {:?}", func.dfg[inst]); + } } /// I-type instructions. @@ -99,7 +141,7 @@ fn recipe_i(func: &Function, inst: Inst, sink: &mut CS) { func.locations[func.dfg.first_result(inst)].unwrap_reg(), sink); } else { - panic!("Expected Binary format: {:?}", func.dfg[inst]); + panic!("Expected BinaryImm format: {:?}", func.dfg[inst]); } } From f70ae03b47d57a1eb151f03cfb0f5445fc301400 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 31 Mar 2017 15:45:21 -0700 Subject: [PATCH 0650/3084] Skip the Python checks if the Python files haven't changed. --- cranelift/test-all.sh | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 6e2ceb2de1..14b712d9b4 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -46,8 +46,18 @@ else echo "If a newer version of rustfmt is available, update this script." fi -banner $(python --version 2>&1) -$topdir/lib/cretonne/meta/check.sh +# Check if any Python files have changed since we last checked them. +tsfile=$topdir/target/meta-checked +if [ -f $tsfile ]; then + needcheck=$(find $topdir/lib/cretonne/meta -name '*.py' -newer $tsfile) +else + needcheck=yes +fi +if [ -n "$needcheck" ]; then + banner $(python --version 2>&1) + $topdir/lib/cretonne/meta/check.sh + touch $tsfile +fi PKGS="cretonne cretonne-reader cretonne-tools filecheck" cd "$topdir" From e23d12bbc7babb179060fd6ec49e571324b5e155 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 3 Apr 2017 09:49:44 -0700 Subject: [PATCH 0651/3084] Add an icmp_imm instruction. Compare a scalar integer to an immediate constant. Both Intel and RISC-V ISAs have this operation. This requires the addition of a new IntCompareImm instruction format. --- cranelift/docs/langref.rst | 1 + cranelift/filetests/parser/tiny.cton | 4 ++-- lib/cretonne/meta/base/formats.py | 1 + lib/cretonne/meta/base/instructions.py | 16 ++++++++++++++++ lib/cretonne/src/ir/instructions.rs | 7 +++++++ lib/cretonne/src/verifier.rs | 3 ++- lib/cretonne/src/write.rs | 1 + lib/reader/src/parser.rs | 21 ++++++++++++++++++--- 8 files changed, 48 insertions(+), 6 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 19e17490a2..507cef3cc4 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -678,6 +678,7 @@ Integer operations ------------------ .. autoinst:: icmp +.. autoinst:: icmp_imm .. autoinst:: iadd .. autoinst:: iadd_imm .. autoinst:: iadd_cin diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 7158441ed2..eb6a28c982 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -54,14 +54,14 @@ function icmp(i32, i32) { ebb0(vx0: i32, vx1: i32): v0 = icmp eq, vx0, vx1 v1 = icmp ult, vx0, vx1 - v2 = icmp sge, vx0, vx1 + v2 = icmp_imm sge, vx0, -12 v3 = irsub_imm vx1, 45 } ; sameln: function icmp(i32, i32) { ; nextln: ebb0(vx0: i32, vx1: i32): ; nextln: v0 = icmp eq, vx0, vx1 ; nextln: v1 = icmp ult, vx0, vx1 -; nextln: v2 = icmp sge, vx0, vx1 +; nextln: v2 = icmp_imm sge, vx0, -12 ; nextln: v3 = irsub_imm vx1, 45 ; nextln: } diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 5ae806b936..a80fa03fd9 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -38,6 +38,7 @@ InsertLane = InstructionFormat(VALUE, ('lane', uimm8), VALUE) ExtractLane = InstructionFormat(VALUE, ('lane', uimm8)) IntCompare = InstructionFormat(intcc, VALUE, VALUE) +IntCompareImm = InstructionFormat(intcc, VALUE, imm64) FloatCompare = InstructionFormat(floatcc, VALUE, VALUE) Jump = InstructionFormat(ebb, VARIABLE_ARGS) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 541889f7d5..f618be7a75 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -367,6 +367,22 @@ icmp = Instruction( """, ins=(Cond, x, y), outs=a) +a = Operand('a', b1) +x = Operand('x', iB) +Y = Operand('Y', imm64) + +icmp_imm = Instruction( + 'icmp_imm', r""" + Compare scalar integer to a constant. + + This is the same as the :inst:`icmp` instruction, except one operand is + an immediate constant. + + This instruction can only compare scalars. Use :inst:`icmp` for + lane-wise vector comparisons. + """, + ins=(Cond, x, Y), outs=a) + a = Operand('a', Int) x = Operand('x', Int) y = Operand('y', Int) diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index f5fd0c0a98..8502954cd1 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -175,6 +175,13 @@ pub enum InstructionData { cond: IntCC, args: [Value; 2], }, + IntCompareImm { + opcode: Opcode, + ty: Type, + cond: IntCC, + arg: Value, + imm: Imm64, + }, FloatCompare { opcode: Opcode, ty: Type, diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 37dc5e20f7..0267d297a2 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -251,6 +251,7 @@ impl<'a> Verifier<'a> { &InsertLane { .. } | &ExtractLane { .. } | &IntCompare { .. } | + &IntCompareImm { .. } | &FloatCompare { .. } => {} } @@ -682,4 +683,4 @@ mod tests { let verifier = Verifier::new(&func); assert_err_with_msg!(verifier.run(), "instruction format"); } -} \ No newline at end of file +} diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 548f718099..23889390e7 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -260,6 +260,7 @@ pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result InsertLane { lane, args, .. } => write!(w, " {}, {}, {}", args[0], lane, args[1]), ExtractLane { lane, arg, .. } => write!(w, " {}, {}", arg, lane), IntCompare { cond, args, .. } => write!(w, " {}, {}, {}", cond, args[0], args[1]), + IntCompareImm { cond, arg, imm, .. } => write!(w, " {}, {}, {}", cond, arg, imm), FloatCompare { cond, args, .. } => write!(w, " {}, {}, {}", cond, args[0], args[1]), Jump { destination, ref args, .. } => { if args.is_empty() { diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 414dd291e2..2e523eeb01 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -216,11 +216,12 @@ impl<'a> Context<'a> { InstructionData::UnaryIeee32 { .. } | InstructionData::UnaryIeee64 { .. } => {} - InstructionData::Unary { ref mut arg, .. } | - InstructionData::UnarySplit { ref mut arg, .. } | InstructionData::BinaryImm { ref mut arg, .. } | + InstructionData::BranchTable { ref mut arg, .. } | InstructionData::ExtractLane { ref mut arg, .. } | - InstructionData::BranchTable { ref mut arg, .. } => { + InstructionData::IntCompareImm { ref mut arg, .. } | + InstructionData::Unary { ref mut arg, .. } | + InstructionData::UnarySplit { ref mut arg, .. } => { self.map.rewrite_value(arg, loc)?; } @@ -1557,6 +1558,20 @@ impl<'a> Parser<'a> { args: [lhs, rhs], } } + InstructionFormat::IntCompareImm => { + let cond = self.match_enum("expected intcc condition code")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let lhs = self.match_value("expected SSA value first operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let rhs = self.match_imm64("expected immediate second operand")?; + InstructionData::IntCompareImm { + opcode: opcode, + ty: VOID, + cond: cond, + arg: lhs, + imm: rhs, + } + } InstructionFormat::FloatCompare => { let cond = self.match_enum("expected floatcc condition code")?; self.match_token(Token::Comma, "expected ',' between operands")?; From 3176ccd131783412c92696ff31b0b425ac3a95b3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 3 Apr 2017 09:56:47 -0700 Subject: [PATCH 0652/3084] Use the right operand when documenting type variable inference. The meaning of format.typevar_operand changes recently to be relative to value operands only instead of all operands. The Sphinx cton domain wasn't updated. --- cranelift/docs/cton_domain.py | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index 23204f02e7..3dba386d90 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -304,16 +304,17 @@ class InstDocumenter(sphinx.ext.autodoc.Documenter): def add_content(self, more_content, no_docstring=False): super(InstDocumenter, self).add_content(more_content, no_docstring) sourcename = self.get_sourcename() + inst = self.object # Add inputs and outputs. - for op in self.object.ins: + for op in inst.ins: if op.is_value(): typ = op.typevar else: typ = op.kind self.add_line(u':in {} {}: {}'.format( typ, op.name, op.get_doc()), sourcename) - for op in self.object.outs: + for op in inst.outs: if op.is_value(): typ = op.typevar else: @@ -322,22 +323,22 @@ class InstDocumenter(sphinx.ext.autodoc.Documenter): typ, op.name, op.get_doc()), sourcename) # Document type inference for polymorphic instructions. - if self.object.is_polymorphic: - if self.object.ctrl_typevar is not None: - if self.object.use_typevar_operand: + if inst.is_polymorphic: + if inst.ctrl_typevar is not None: + if inst.use_typevar_operand: + tvopnum = inst.value_opnums[inst.format.typevar_operand] self.add_line( u':typevar {}: inferred from {}' .format( - self.object.ctrl_typevar.name, - self.object.ins[ - self.object.format.typevar_operand]), + inst.ctrl_typevar.name, + inst.ins[tvopnum]), sourcename) else: self.add_line( u':typevar {}: explicitly provided' - .format(self.object.ctrl_typevar.name), + .format(inst.ctrl_typevar.name), sourcename) - for tv in self.object.other_typevars: + for tv in inst.other_typevars: self.add_line( u':typevar {}: from input operand'.format(tv.name), sourcename) From e65be4d8275fdaeb7684aa235aa200afd9d7b421 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 3 Apr 2017 10:06:08 -0700 Subject: [PATCH 0653/3084] Give singleton type variables the type's doc string. This reads better than "typeof(b1)". --- lib/cretonne/meta/cdsl/typevar.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 2c5141d78c..1dc8630f5f 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -285,7 +285,7 @@ class TypeVar(object): bools = (scalar.bits, scalar.bits) tv = TypeVar( - typ.name, 'typeof({})'.format(typ), + typ.name, typ.__doc__, ints, floats, bools, simd=lanes) tv.singleton_type = typ return tv From 4e398164d26e02fda90b91c73c3cab71114c0883 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 3 Apr 2017 10:56:30 -0700 Subject: [PATCH 0654/3084] Add icmp_imm encodings for RISC-V. The ISA has icmp_imm slt/ult with 12-bit signed immediate operands. --- cranelift/filetests/isa/riscv/binary32.cton | 8 +++++++- lib/cretonne/meta/isa/riscv/encodings.py | 8 +++++++- lib/cretonne/meta/isa/riscv/recipes.py | 7 ++++++- lib/cretonne/src/isa/riscv/binemit.rs | 12 ++++++++++++ 4 files changed, 32 insertions(+), 3 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index fd3d910a18..aa0a7b7389 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -44,7 +44,6 @@ ebb0: ; addi [-,%x7] v100 = iadd_imm v1, 1000 ; bin: 3e850393 [-,%x16] v101 = iadd_imm v2, -905 ; bin: c77a8813 - ; TBD: slti ; andi [-,%x7] v110 = band_imm v1, 1000 ; bin: 3e857393 [-,%x16] v111 = band_imm v2, -905 ; bin: c77af813 @@ -65,5 +64,12 @@ ebb0: [-,%x7] v124 = sshr_imm v1, 31 ; bin: 41f55393 [-,%x16] v125 = sshr_imm v2, 8 ; bin: 408ad813 + ; slti + [-,%x7] v130 = icmp_imm slt, v1, 1000 ; bin: 3e852393 + [-,%x16] v131 = icmp_imm slt, v2, -905 ; bin: c77aa813 + ; sltiu + [-,%x7] v132 = icmp_imm ult, v1, 1000 ; bin: 3e853393 + [-,%x16] v133 = icmp_imm ult, v2, -905 ; bin: c77ab813 + return } diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index b97c92386a..1966a576c2 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -5,7 +5,8 @@ from __future__ import absolute_import from base import instructions as base from base.immediates import intcc from .defs import RV32, RV64 -from .recipes import OPIMM, OPIMM32, OP, OP32, JALR, R, Rshamt, Ricmp, I, Iret +from .recipes import OPIMM, OPIMM32, OP, OP32 +from .recipes import JALR, R, Rshamt, Ricmp, I, Iicmp, Iret from .settings import use_m from cdsl.ast import Var @@ -60,6 +61,11 @@ RV64.enc(base.icmp.i64(intcc.slt, x, y), Ricmp, OP(0b010, 0b0000000)) RV32.enc(base.icmp.i32(intcc.ult, x, y), Ricmp, OP(0b011, 0b0000000)) RV64.enc(base.icmp.i64(intcc.ult, x, y), Ricmp, OP(0b011, 0b0000000)) +RV32.enc(base.icmp_imm.i32(intcc.slt, x, y), Iicmp, OPIMM(0b010)) +RV64.enc(base.icmp_imm.i64(intcc.slt, x, y), Iicmp, OPIMM(0b010)) +RV32.enc(base.icmp_imm.i32(intcc.ult, x, y), Iicmp, OPIMM(0b011)) +RV64.enc(base.icmp_imm.i64(intcc.ult, x, y), Iicmp, OPIMM(0b011)) + # "M" Standard Extension for Integer Multiplication and Division. # Gated by the `use_m` flag. RV32.enc(base.imul.i32, R, OP(0b000, 0b0000001), isap=use_m) diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 2fd326e5cc..fb31b5d60a 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -11,7 +11,7 @@ instruction formats described in the reference: from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt -from base.formats import Binary, BinaryImm, MultiAry, IntCompare +from base.formats import Binary, BinaryImm, MultiAry, IntCompare, IntCompareImm from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit @@ -86,6 +86,11 @@ I = EncRecipe( 'I', BinaryImm, ins=GPR, outs=GPR, instp=IsSignedInt(BinaryImm.imm, 12)) +# I-type encoding of an integer comparison. +Iicmp = EncRecipe( + 'Iicmp', IntCompareImm, ins=GPR, outs=GPR, + instp=IsSignedInt(IntCompareImm.imm, 12)) + # I-type encoding for `jalr` as a return instruction. We won't use the # immediate offset. # The variable return values are not encoded. diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index a3941af526..5479a60898 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -145,6 +145,18 @@ fn recipe_i(func: &Function, inst: Inst, sink: &mut CS) { } } +fn recipe_iicmp(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::IntCompareImm { arg, imm, .. } = func.dfg[inst] { + put_i(func.encodings[inst].bits(), + func.locations[arg].unwrap_reg(), + imm.into(), + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); + } else { + panic!("Expected IntCompareImm format: {:?}", func.dfg[inst]); + } +} + fn recipe_iret(_func: &Function, _inst: Inst, _sink: &mut CS) { unimplemented!() } From 0530e822d4d80d72f4c81acefc1c59f957508078 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 3 Apr 2017 12:27:22 -0700 Subject: [PATCH 0655/3084] Add RISC-V encodings for lui. This instruction can materialize constants with the low 12 bits clear. --- cranelift/filetests/isa/riscv/binary32.cton | 4 +++ lib/cretonne/meta/isa/riscv/encodings.py | 9 ++++-- lib/cretonne/meta/isa/riscv/recipes.py | 16 +++++++++++ lib/cretonne/src/isa/riscv/binemit.rs | 32 +++++++++++++++++++++ 4 files changed, 59 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index aa0a7b7389..f7225c234e 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -71,5 +71,9 @@ ebb0: [-,%x7] v132 = icmp_imm ult, v1, 1000 ; bin: 3e853393 [-,%x16] v133 = icmp_imm ult, v2, -905 ; bin: c77ab813 + ; lui + [-,%x7] v140 = iconst.i32 0x12345000 ; bin: 123453b7 + [-,%x16] v141 = iconst.i32 0xffffffff_fedcb000 ; bin: fedcb837 + return } diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index 1966a576c2..e1d186db5d 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -5,8 +5,8 @@ from __future__ import absolute_import from base import instructions as base from base.immediates import intcc from .defs import RV32, RV64 -from .recipes import OPIMM, OPIMM32, OP, OP32 -from .recipes import JALR, R, Rshamt, Ricmp, I, Iicmp, Iret +from .recipes import OPIMM, OPIMM32, OP, OP32, LUI +from .recipes import JALR, R, Rshamt, Ricmp, I, Iicmp, Iret, U from .settings import use_m from cdsl.ast import Var @@ -66,6 +66,11 @@ RV64.enc(base.icmp_imm.i64(intcc.slt, x, y), Iicmp, OPIMM(0b010)) RV32.enc(base.icmp_imm.i32(intcc.ult, x, y), Iicmp, OPIMM(0b011)) RV64.enc(base.icmp_imm.i64(intcc.ult, x, y), Iicmp, OPIMM(0b011)) +# Integer constants with the low 12 bits clear are materialized by lui. +RV32.enc(base.iconst.i32, U, LUI()) +RV64.enc(base.iconst.i32, U, LUI()) +RV64.enc(base.iconst.i64, U, LUI()) + # "M" Standard Extension for Integer Multiplication and Division. # Gated by the `use_m` flag. RV32.enc(base.imul.i32, R, OP(0b000, 0b0000001), isap=use_m) diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index fb31b5d60a..ebeb9f9ee1 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -12,6 +12,7 @@ from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt from base.formats import Binary, BinaryImm, MultiAry, IntCompare, IntCompareImm +from base.formats import UnaryImm from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit @@ -72,6 +73,16 @@ def OP32(funct3, funct7): return 0b01110 | (funct3 << 5) | (funct7 << 8) +def AIUPC(): + # type: () -> int + return 0b00101 + + +def LUI(): + # type: () -> int + return 0b01101 + + # R-type 32-bit instructions: These are mostly binary arithmetic instructions. # The encbits are `opcode[6:2] | (funct3 << 5) | (funct7 << 8) R = EncRecipe('R', Binary, ins=(GPR, GPR), outs=GPR) @@ -95,3 +106,8 @@ Iicmp = EncRecipe( # immediate offset. # The variable return values are not encoded. Iret = EncRecipe('Iret', MultiAry, ins=GPR, outs=()) + +# U-type instructions have a 20-bit immediate that targets bits 12-31. +U = EncRecipe( + 'U', UnaryImm, ins=(), outs=GPR, + instp=IsSignedInt(UnaryImm.imm, 32, 12)) diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 5479a60898..a295a66386 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -160,3 +160,35 @@ fn recipe_iicmp(func: &Function, inst: Inst, sink: &mut C fn recipe_iret(_func: &Function, _inst: Inst, _sink: &mut CS) { unimplemented!() } + +/// U-type instructions. +/// +/// 31 11 6 +/// imm rd opcode +/// 12 7 0 +/// +/// Encoding bits: `opcode[6:2] | (funct3 << 5)` +fn put_u(bits: u16, imm: i64, rd: RegUnit, sink: &mut CS) { + let bits = bits as u32; + let opcode5 = bits & 0x1f; + let rd = rd as u32 & 0x1f; + + // 0-6: opcode + let mut i = 0x3; + i |= opcode5 << 2; + i |= rd << 7; + i |= imm as u32 & 0xfffff000; + + sink.put4(i); +} + +fn recipe_u(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::UnaryImm { imm, .. } = func.dfg[inst] { + put_u(func.encodings[inst].bits(), + imm.into(), + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); + } else { + panic!("Expected UnaryImm format: {:?}", func.dfg[inst]); + } +} From 1b6a6f4e4887904c7e892c28f80112a26af384a5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 3 Apr 2017 13:44:15 -0700 Subject: [PATCH 0656/3084] Add the br_icmp instruction. This instruction behaves like icmp fused with brnz, and it can be used to represent fused compare+branch instruction on Intel when optimizing for macro-op fusion. RISC-V provides compare-and-branch instructions directly, and it is needed there too. --- cranelift/docs/langref.rst | 1 + cranelift/filetests/parser/tiny.cton | 2 ++ lib/cretonne/meta/base/formats.py | 1 + lib/cretonne/meta/base/instructions.py | 23 +++++++++++++++++++ lib/cretonne/src/ir/instructions.rs | 10 +++++++++ lib/cretonne/src/verifier.rs | 8 +++---- lib/cretonne/src/write.rs | 20 ++++++++++------- lib/reader/src/parser.rs | 31 +++++++++++++++++--------- 8 files changed, 73 insertions(+), 23 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 507cef3cc4..47a0dae37c 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -321,6 +321,7 @@ instruction in the EBB. .. autoinst:: jump .. autoinst:: brz .. autoinst:: brnz +.. autoinst:: br_icmp .. autoinst:: br_table .. inst:: JT = jump_table EBB0, EBB1, ..., EBBn diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index eb6a28c982..d5790c0e89 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -56,6 +56,7 @@ ebb0(vx0: i32, vx1: i32): v1 = icmp ult, vx0, vx1 v2 = icmp_imm sge, vx0, -12 v3 = irsub_imm vx1, 45 + br_icmp eq, vx0, vx1, ebb0(vx1, vx0) } ; sameln: function icmp(i32, i32) { ; nextln: ebb0(vx0: i32, vx1: i32): @@ -63,6 +64,7 @@ ebb0(vx0: i32, vx1: i32): ; nextln: v1 = icmp ult, vx0, vx1 ; nextln: v2 = icmp_imm sge, vx0, -12 ; nextln: v3 = irsub_imm vx1, 45 +; nextln: br_icmp eq, vx0, vx1, ebb0(vx1, vx0) ; nextln: } ; Floating condition codes. diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index a80fa03fd9..cf1922627f 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -43,6 +43,7 @@ FloatCompare = InstructionFormat(floatcc, VALUE, VALUE) Jump = InstructionFormat(ebb, VARIABLE_ARGS) Branch = InstructionFormat(VALUE, ebb, VARIABLE_ARGS) +BranchIcmp = InstructionFormat(intcc, VALUE, VALUE, ebb, VARIABLE_ARGS) BranchTable = InstructionFormat(VALUE, jump_table) Call = InstructionFormat( diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index f618be7a75..4776f576c1 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -33,6 +33,9 @@ Any = TypeVar( # Control flow # c = Operand('c', Testable, doc='Controlling value to test') +Cond = Operand('Cond', intcc) +x = Operand('x', iB) +y = Operand('y', iB) EBB = Operand('EBB', entities.ebb, doc='Destination extended basic block') args = Operand('args', VARIABLE_ARGS, doc='EBB arguments') @@ -64,6 +67,26 @@ brnz = Instruction( """, ins=(c, EBB, args), is_branch=True) +br_icmp = Instruction( + 'br_icmp', r""" + Compare scalar integers and branch. + + Compare ``x`` and ``y`` in the same way as the :inst:`icmp` instruction + and take the branch if the condition is true:: + + br_icmp ugt v1, v2, ebb4(v5, v6) + + is semantically equivalent to:: + + v10 = icmp ugt, v1, v2 + brnz v10, ebb4(v5, v6) + + Some RISC architectures like MIPS and RISC-V provide instructions that + implement all or some of the condition codes. The instruction can also + be used to represent *macro-op fusion* on architectures like Intel's. + """, + ins=(Cond, x, y, EBB, args), is_branch=True) + x = Operand('x', iB, doc='index into jump table') JT = Operand('JT', entities.jump_table) br_table = Instruction( diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 8502954cd1..d3ca0bc62f 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -200,6 +200,13 @@ pub enum InstructionData { destination: Ebb, args: ValueList, }, + BranchIcmp { + opcode: Opcode, + ty: Type, + cond: IntCC, + destination: Ebb, + args: ValueList, + }, BranchTable { opcode: Opcode, ty: Type, @@ -303,6 +310,9 @@ impl InstructionData { &InstructionData::Branch { destination, ref args, .. } => { BranchInfo::SingleDest(destination, &args.as_slice(pool)[1..]) } + &InstructionData::BranchIcmp { destination, ref args, .. } => { + BranchInfo::SingleDest(destination, &args.as_slice(pool)[2..]) + } &InstructionData::BranchTable { table, .. } => BranchInfo::Table(table), _ => BranchInfo::NotABranch, } diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 0267d297a2..589f117e73 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -218,11 +218,9 @@ impl<'a> Verifier<'a> { &MultiAry { ref args, .. } => { self.verify_value_list(inst, args)?; } - &Jump { destination, ref args, .. } => { - self.verify_ebb(inst, destination)?; - self.verify_value_list(inst, args)?; - } - &Branch { destination, ref args, .. } => { + &Jump { destination, ref args, .. } | + &Branch { destination, ref args, .. } | + &BranchIcmp { destination, ref args, .. } => { self.verify_ebb(inst, destination)?; self.verify_value_list(inst, args)?; } diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 23889390e7..e1ac42ac09 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -274,15 +274,19 @@ pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result } Branch { destination, ref args, .. } => { let args = args.as_slice(pool); - if args.len() == 1 { - write!(w, " {}, {}", args[0], destination) - } else { - write!(w, - " {}, {}({})", - args[0], - destination, - DisplayValues(&args[1..])) + write!(w, " {}, {}", args[0], destination)?; + if args.len() > 1 { + write!(w, "({})", DisplayValues(&args[1..]))?; } + Ok(()) + } + BranchIcmp { cond, destination, ref args, .. } => { + let args = args.as_slice(pool); + write!(w, " {}, {}, {}, {}", cond, args[0], args[1], destination)?; + if args.len() > 2 { + write!(w, "({})", DisplayValues(&args[2..]))?; + } + Ok(()) } BranchTable { arg, table, .. } => write!(w, " {}, {}", arg, table), Call { func_ref, ref args, .. } => { diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 2e523eeb01..e8d88dde89 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -241,20 +241,14 @@ impl<'a> Context<'a> { self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; } - InstructionData::Jump { ref mut destination, ref mut args, .. } => { + InstructionData::Jump { ref mut destination, ref mut args, .. } | + InstructionData::Branch { ref mut destination, ref mut args, .. } | + InstructionData::BranchIcmp { ref mut destination, ref mut args, .. } => { self.map.rewrite_ebb(destination, loc)?; self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; } - InstructionData::Branch { ref mut destination, ref mut args, .. } => { - self.map.rewrite_ebb(destination, loc)?; - self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; - } - - InstructionData::Call { ref mut args, .. } => { - self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; - } - + InstructionData::Call { ref mut args, .. } | InstructionData::IndirectCall { ref mut args, .. } => { self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; } @@ -1521,6 +1515,23 @@ impl<'a> Parser<'a> { args: args.into_value_list(&[ctrl_arg], &mut ctx.function.dfg.value_lists), } } + InstructionFormat::BranchIcmp => { + let cond = self.match_enum("expected intcc condition code")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let lhs = self.match_value("expected SSA value first operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let rhs = self.match_value("expected SSA value second operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let ebb_num = self.match_ebb("expected branch destination EBB")?; + let args = self.parse_opt_value_list()?; + InstructionData::BranchIcmp { + opcode: opcode, + ty: VOID, + cond: cond, + destination: ebb_num, + args: args.into_value_list(&[lhs, rhs], &mut ctx.function.dfg.value_lists), + } + } InstructionFormat::InsertLane => { let lhs = self.match_value("expected SSA value first operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; From 479ff156c13384de0f94c00a25a3e817a3aa6091 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 3 Apr 2017 15:07:08 -0700 Subject: [PATCH 0657/3084] Add conditional branch encodings for RISC-V. Not all br_icmp opcodes are present in the ISA. The missing ones can be reached by commuting operands. Don't attempt to encode EBB offsets yet. For now just emit an EBB relocation for the branch instruction. --- cranelift/filetests/isa/riscv/binary32.cton | 16 ++++++ cranelift/src/filetest/binemit.rs | 18 +++++-- cranelift/test-all.sh | 2 +- lib/cretonne/meta/isa/riscv/encodings.py | 18 ++++++- lib/cretonne/meta/isa/riscv/recipes.py | 7 ++- lib/cretonne/src/binemit/mod.rs | 7 ++- lib/cretonne/src/isa/mod.rs | 8 +++ lib/cretonne/src/isa/riscv/binemit.rs | 56 ++++++++++++++++++++- lib/cretonne/src/isa/riscv/mod.rs | 4 ++ 9 files changed, 124 insertions(+), 12 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index f7225c234e..c2b300e07c 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -75,5 +75,21 @@ ebb0: [-,%x7] v140 = iconst.i32 0x12345000 ; bin: 123453b7 [-,%x16] v141 = iconst.i32 0xffffffff_fedcb000 ; bin: fedcb837 + + ; Control Transfer Instructions + + ; beq + br_icmp eq, v1, v2, ebb0 ; bin: Branch(ebb0) 01550063 + ; bne + br_icmp ne, v1, v2, ebb0 ; bin: Branch(ebb0) 01551063 + ; blt + br_icmp slt, v1, v2, ebb0 ; bin: Branch(ebb0) 01554063 + ; bge + br_icmp sge, v1, v2, ebb0 ; bin: Branch(ebb0) 01555063 + ; bltu + br_icmp ult, v1, v2, ebb0 ; bin: Branch(ebb0) 01556063 + ; bgeu + br_icmp uge, v1, v2, ebb0 ; bin: Branch(ebb0) 01557063 + return } diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 16c2613b20..3c9fc333e6 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -25,6 +25,7 @@ pub fn subtest(parsed: &TestCommand) -> Result> { // Code sink that generates text. struct TextSink { + rnames: &'static [&'static str], text: String, } @@ -45,12 +46,16 @@ impl binemit::CodeSink for TextSink { write!(self.text, "{:016x} ", x).unwrap(); } - fn reloc_func(&mut self, _: binemit::Reloc, _: ir::FuncRef) { - unimplemented!() + fn reloc_ebb(&mut self, reloc: binemit::Reloc, ebb: ir::Ebb) { + write!(self.text, "{}({}) ", self.rnames[reloc.0 as usize], ebb).unwrap(); } - fn reloc_jt(&mut self, _: binemit::Reloc, _: ir::JumpTable) { - unimplemented!() + fn reloc_func(&mut self, reloc: binemit::Reloc, fref: ir::FuncRef) { + write!(self.text, "{}({}) ", self.rnames[reloc.0 as usize], fref).unwrap(); + } + + fn reloc_jt(&mut self, reloc: binemit::Reloc, jt: ir::JumpTable) { + write!(self.text, "{}({}) ", self.rnames[reloc.0 as usize], jt).unwrap(); } } @@ -73,7 +78,10 @@ impl SubTest for TestBinEmit { // value locations. The current error reporting is just crashing... let mut func = func.into_owned(); - let mut sink = TextSink { text: String::new() }; + let mut sink = TextSink { + rnames: isa.reloc_names(), + text: String::new(), + }; for comment in &context.details.comments { if let Some(want) = match_directive(comment.text, "bin:") { diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 14b712d9b4..7c4938902f 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -56,7 +56,7 @@ fi if [ -n "$needcheck" ]; then banner $(python --version 2>&1) $topdir/lib/cretonne/meta/check.sh - touch $tsfile + touch $tsfile || echo no target directory fi PKGS="cretonne cretonne-reader cretonne-tools filecheck" diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index e1d186db5d..0d677c54ca 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -5,14 +5,16 @@ from __future__ import absolute_import from base import instructions as base from base.immediates import intcc from .defs import RV32, RV64 -from .recipes import OPIMM, OPIMM32, OP, OP32, LUI -from .recipes import JALR, R, Rshamt, Ricmp, I, Iicmp, Iret, U +from .recipes import OPIMM, OPIMM32, OP, OP32, LUI, BRANCH +from .recipes import JALR, R, Rshamt, Ricmp, I, Iicmp, Iret, U, SB from .settings import use_m from cdsl.ast import Var # Dummies for instruction predicates. x = Var('x') y = Var('y') +dest = Var('dest') +args = Var('args') # Basic arithmetic binary instructions are encoded in an R-type instruction. for inst, inst_imm, f3, f7 in [ @@ -79,6 +81,18 @@ RV64.enc(base.imul.i32, R, OP32(0b000, 0b0000001), isap=use_m) # Control flow. +# Conditional branches. +for cond, f3 in [ + (intcc.eq, 0b000), + (intcc.ne, 0b001), + (intcc.slt, 0b100), + (intcc.sge, 0b101), + (intcc.ult, 0b110), + (intcc.uge, 0b111) + ]: + RV32.enc(base.br_icmp.i32(cond, x, y, dest, args), SB, BRANCH(f3)) + RV64.enc(base.br_icmp.i64(cond, x, y, dest, args), SB, BRANCH(f3)) + # Returns are a special case of JALR. # Note: Return stack predictors will only recognize this as a return when the # return address is provided in `x1`. We may want a special encoding to enforce diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index ebeb9f9ee1..9fe3c6f14d 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -12,7 +12,7 @@ from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt from base.formats import Binary, BinaryImm, MultiAry, IntCompare, IntCompareImm -from base.formats import UnaryImm +from base.formats import UnaryImm, BranchIcmp from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit @@ -111,3 +111,8 @@ Iret = EncRecipe('Iret', MultiAry, ins=GPR, outs=()) U = EncRecipe( 'U', UnaryImm, ins=(), outs=GPR, instp=IsSignedInt(UnaryImm.imm, 32, 12)) + +# SB-type branch instructions. +# TODO: These instructions have a +/- 4 KB branch range. How to encode that +# constraint? +SB = EncRecipe('SB', BranchIcmp, ins=(GPR, GPR), outs=()) diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 0b8b3475b7..ccad476f44 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -3,10 +3,10 @@ //! The `binemit` module contains code for translating Cretonne's intermediate representation into //! binary machine code. -use ir::{FuncRef, JumpTable, Function, Inst}; +use ir::{Ebb, FuncRef, JumpTable, Function, Inst}; /// Relocation kinds depend on the current ISA. -pub struct Reloc(u16); +pub struct Reloc(pub u16); /// Abstract interface for adding bytes to the code segment. /// @@ -25,6 +25,9 @@ pub trait CodeSink { /// Add 8 bytes to the code section. fn put8(&mut self, u64); + /// Add a relocation referencing an EBB at the current offset. + fn reloc_ebb(&mut self, Reloc, Ebb); + /// Add a relocation referencing an external function at the current offset. fn reloc_func(&mut self, Reloc, FuncRef); diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 776d5e4bca..4eddae7b14 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -188,4 +188,12 @@ pub trait TargetIsa { /// Note that this will call `put*` methods on the trait object via its vtable which is not the /// fastest way of emitting code. fn emit_inst(&self, func: &Function, inst: Inst, sink: &mut CodeSink); + + /// Get a static array of names associated with relocations in this ISA. + /// + /// This array can be indexed by the contents of `binemit::Reloc` objects passed to a + /// `CodeSink`. + fn reloc_names(&self) -> &'static [&'static str] { + unimplemented!() + } } diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index a295a66386..888fa369ed 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -1,11 +1,25 @@ //! Emitting binary RISC-V machine code. -use binemit::{CodeSink, bad_encoding}; +use binemit::{CodeSink, Reloc, bad_encoding}; use ir::{Function, Inst, InstructionData}; use isa::RegUnit; include!(concat!(env!("OUT_DIR"), "/binemit-riscv.rs")); +/// RISC-V relocation kinds. +pub enum RelocKind { + /// A conditional (SB-type) branch to an EBB. + Branch, +} + +pub static RELOC_NAMES: [&'static str; 1] = ["Branch"]; + +impl Into for RelocKind { + fn into(self) -> Reloc { + Reloc(self as u16) + } +} + /// R-type instructions. /// /// 31 24 19 14 11 6 @@ -192,3 +206,43 @@ fn recipe_u(func: &Function, inst: Inst, sink: &mut CS) { panic!("Expected UnaryImm format: {:?}", func.dfg[inst]); } } + +/// SB-type branch instructions. +/// +/// 31 24 19 14 11 6 +/// imm rs2 rs1 funct3 imm opcode +/// 25 20 15 12 7 0 +/// +/// The imm bits are not encoded by this function. They encode the relative distance to the +/// destination block, handled by a relocation. +/// +/// Encoding bits: `opcode[6:2] | (funct3 << 5)` +fn put_sb(bits: u16, rs1: RegUnit, rs2: RegUnit, sink: &mut CS) { + let bits = bits as u32; + let opcode5 = bits & 0x1f; + let funct3 = (bits >> 5) & 0x7; + let rs1 = rs1 as u32 & 0x1f; + let rs2 = rs2 as u32 & 0x1f; + + // 0-6: opcode + let mut i = 0x3; + i |= opcode5 << 2; + i |= funct3 << 12; + i |= rs1 << 15; + i |= rs2 << 20; + + sink.put4(i); +} + +fn recipe_sb(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::BranchIcmp { destination, ref args, .. } = func.dfg[inst] { + let args = &args.as_slice(&func.dfg.value_lists)[0..2]; + sink.reloc_ebb(RelocKind::Branch.into(), destination); + put_sb(func.encodings[inst].bits(), + func.locations[args[0]].unwrap_reg(), + func.locations[args[1]].unwrap_reg(), + sink); + } else { + panic!("Expected BranchIcmp format: {:?}", func.dfg[inst]); + } +} diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 94808bfb70..348e1af32f 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -86,6 +86,10 @@ impl TargetIsa for Isa { fn emit_inst(&self, func: &Function, inst: Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } + + fn reloc_names(&self) -> &'static [&'static str] { + &binemit::RELOC_NAMES + } } #[cfg(test)] From e641c976708007690c0fa5ec1d47dd7eb69bd23d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 3 Apr 2017 15:20:57 -0700 Subject: [PATCH 0658/3084] Add RISC-V encodings for brz and brnz. These branches compare a register to zero. RISC-V implements this with the %x0 hard-coded zero register. --- cranelift/filetests/isa/riscv/binary32.cton | 5 +++++ lib/cretonne/meta/isa/riscv/encodings.py | 11 ++++++++++- lib/cretonne/meta/isa/riscv/recipes.py | 5 ++++- lib/cretonne/src/isa/riscv/binemit.rs | 13 +++++++++++++ 4 files changed, 32 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index c2b300e07c..93b1a9cc2d 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -91,5 +91,10 @@ ebb0: ; bgeu br_icmp uge, v1, v2, ebb0 ; bin: Branch(ebb0) 01557063 + ; beq x, %x0 + brz v1, ebb0 ; bin: Branch(ebb0) 00050063 + ; bne x, %x0 + brnz v1, ebb0 ; bin: Branch(ebb0) 00051063 + return } diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index 0d677c54ca..162ccbd3ea 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -6,7 +6,7 @@ from base import instructions as base from base.immediates import intcc from .defs import RV32, RV64 from .recipes import OPIMM, OPIMM32, OP, OP32, LUI, BRANCH -from .recipes import JALR, R, Rshamt, Ricmp, I, Iicmp, Iret, U, SB +from .recipes import JALR, R, Rshamt, Ricmp, I, Iicmp, Iret, U, SB, SBzero from .settings import use_m from cdsl.ast import Var @@ -93,6 +93,15 @@ for cond, f3 in [ RV32.enc(base.br_icmp.i32(cond, x, y, dest, args), SB, BRANCH(f3)) RV64.enc(base.br_icmp.i64(cond, x, y, dest, args), SB, BRANCH(f3)) +for inst, f3 in [ + (base.brz, 0b000), + (base.brnz, 0b001) + ]: + RV32.enc(inst.i32, SBzero, BRANCH(f3)) + RV64.enc(inst.i64, SBzero, BRANCH(f3)) + RV32.enc(inst.b1, SBzero, BRANCH(f3)) + RV64.enc(inst.b1, SBzero, BRANCH(f3)) + # Returns are a special case of JALR. # Note: Return stack predictors will only recognize this as a return when the # return address is provided in `x1`. We may want a special encoding to enforce diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 9fe3c6f14d..2be2192002 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -12,7 +12,7 @@ from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt from base.formats import Binary, BinaryImm, MultiAry, IntCompare, IntCompareImm -from base.formats import UnaryImm, BranchIcmp +from base.formats import UnaryImm, BranchIcmp, Branch from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit @@ -116,3 +116,6 @@ U = EncRecipe( # TODO: These instructions have a +/- 4 KB branch range. How to encode that # constraint? SB = EncRecipe('SB', BranchIcmp, ins=(GPR, GPR), outs=()) + +# SB-type branch instruction with rs2 fixed to zero. +SBzero = EncRecipe('SBzero', Branch, ins=(GPR), outs=()) diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 888fa369ed..057fac11aa 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -246,3 +246,16 @@ fn recipe_sb(func: &Function, inst: Inst, sink: &mut CS) panic!("Expected BranchIcmp format: {:?}", func.dfg[inst]); } } + +fn recipe_sbzero(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Branch { destination, ref args, .. } = func.dfg[inst] { + let args = &args.as_slice(&func.dfg.value_lists)[0..1]; + sink.reloc_ebb(RelocKind::Branch.into(), destination); + put_sb(func.encodings[inst].bits(), + func.locations[args[0]].unwrap_reg(), + 0, + sink); + } else { + panic!("Expected Branch format: {:?}", func.dfg[inst]); + } +} From 1984c96f7c7326dc7c5c911d7eaa6c3b075165e8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 5 Apr 2017 09:00:11 -0700 Subject: [PATCH 0659/3084] rustfmt 0.8.1 --- cranelift/src/cton-util.rs | 6 +- cranelift/src/filetest/concurrent.rs | 18 +- cranelift/src/filetest/legalizer.rs | 3 +- cranelift/src/filetest/regalloc.rs | 3 +- cranelift/src/filetest/runner.rs | 25 ++- cranelift/src/filetest/runone.rs | 3 +- cranelift/src/filetest/subtest.rs | 12 +- cranelift/src/rsfilecheck.rs | 15 +- cranelift/test-all.sh | 2 +- cranelift/tests/cfg_traversal.rs | 5 +- lib/cretonne/src/abi.rs | 5 +- lib/cretonne/src/context.rs | 3 +- lib/cretonne/src/dominator_tree.rs | 8 +- lib/cretonne/src/entity_list.rs | 4 +- lib/cretonne/src/ir/builder.rs | 10 +- lib/cretonne/src/ir/dfg.rs | 63 ++++-- lib/cretonne/src/ir/entities.rs | 8 +- lib/cretonne/src/ir/extfunc.rs | 3 +- lib/cretonne/src/ir/instructions.rs | 42 ++-- lib/cretonne/src/ir/jumptable.rs | 14 +- lib/cretonne/src/ir/layout.rs | 31 ++- lib/cretonne/src/ir/types.rs | 14 +- lib/cretonne/src/isa/enc_tables.rs | 14 +- lib/cretonne/src/isa/riscv/binemit.rs | 12 +- lib/cretonne/src/legalizer/boundary.rs | 20 +- lib/cretonne/src/legalizer/split.rs | 10 +- lib/cretonne/src/partition_slice.rs | 5 +- lib/cretonne/src/regalloc/coloring.rs | 4 +- lib/cretonne/src/regalloc/context.rs | 3 +- .../src/regalloc/live_value_tracker.rs | 41 ++-- lib/cretonne/src/regalloc/liveness.rs | 6 +- lib/cretonne/src/regalloc/liverange.rs | 33 +-- lib/cretonne/src/verifier.rs | 71 ++++--- lib/cretonne/src/write.rs | 19 +- lib/filecheck/src/checker.rs | 26 ++- lib/filecheck/src/explain.rs | 74 +++---- lib/reader/src/parser.rs | 191 ++++++++++-------- lib/reader/src/testcommand.rs | 5 +- 38 files changed, 497 insertions(+), 334 deletions(-) diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 681ab5b8bc..fea4d879be 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -51,7 +51,11 @@ pub type CommandResult = Result<(), String>; fn cton_util() -> CommandResult { // Parse comand line arguments. let args: Args = Docopt::new(USAGE) - .and_then(|d| d.help(true).version(Some(format!("Cretonne {}", VERSION))).decode()) + .and_then(|d| { + d.help(true) + .version(Some(format!("Cretonne {}", VERSION))) + .decode() + }) .unwrap_or_else(|e| e.exit()); // Find the sub-command to execute. diff --git a/cranelift/src/filetest/concurrent.rs b/cranelift/src/filetest/concurrent.rs index effe454d38..9ea1114000 100644 --- a/cranelift/src/filetest/concurrent.rs +++ b/cranelift/src/filetest/concurrent.rs @@ -119,10 +119,11 @@ fn worker_thread(thread_num: usize, // Tell them we're starting this job. // The receiver should always be present for this as long as we have jobs. - replies.send(Reply::Starting { - jobid: jobid, - thread_num: thread_num, - }) + replies + .send(Reply::Starting { + jobid: jobid, + thread_num: thread_num, + }) .unwrap(); let result = catch_unwind(|| runone::run(path.as_path())).unwrap_or_else(|e| { @@ -141,10 +142,11 @@ fn worker_thread(thread_num: usize, dbg!("FAIL: {}", msg); } - replies.send(Reply::Done { - jobid: jobid, - result: result, - }) + replies + .send(Reply::Done { + jobid: jobid, + result: result, + }) .unwrap(); } }) diff --git a/cranelift/src/filetest/legalizer.rs b/cranelift/src/filetest/legalizer.rs index 39854769ac..1a5f774664 100644 --- a/cranelift/src/filetest/legalizer.rs +++ b/cranelift/src/filetest/legalizer.rs @@ -41,7 +41,8 @@ impl SubTest for TestLegalizer { comp_ctx.flowgraph(); comp_ctx.legalize(isa); - comp_ctx.verify(isa).map_err(|e| pretty_verifier_error(&comp_ctx.func, e))?; + comp_ctx.verify(isa) + .map_err(|e| pretty_verifier_error(&comp_ctx.func, e))?; let mut text = String::new(); write_function(&mut text, &comp_ctx.func, Some(isa)).map_err(|e| e.to_string())?; diff --git a/cranelift/src/filetest/regalloc.rs b/cranelift/src/filetest/regalloc.rs index 5f70f51100..daf7adcb07 100644 --- a/cranelift/src/filetest/regalloc.rs +++ b/cranelift/src/filetest/regalloc.rs @@ -47,7 +47,8 @@ impl SubTest for TestRegalloc { // TODO: Should we have an option to skip legalization? comp_ctx.legalize(isa); comp_ctx.regalloc(isa); - comp_ctx.verify(isa).map_err(|e| pretty_verifier_error(&comp_ctx.func, e))?; + comp_ctx.verify(isa) + .map_err(|e| pretty_verifier_error(&comp_ctx.func, e))?; let mut text = String::new(); write_function(&mut text, &comp_ctx.func, Some(isa)).map_err(|e| e.to_string())?; diff --git a/cranelift/src/filetest/runner.rs b/cranelift/src/filetest/runner.rs index 91331a0299..1993ca5877 100644 --- a/cranelift/src/filetest/runner.rs +++ b/cranelift/src/filetest/runner.rs @@ -104,10 +104,11 @@ impl TestRunner { /// /// Any problems reading `file` as a test case file will be reported as a test failure. pub fn push_test>(&mut self, file: P) { - self.tests.push(QueueEntry { - path: file.into(), - state: State::New, - }); + self.tests + .push(QueueEntry { + path: file.into(), + state: State::New, + }); } /// Begin running tests concurrently. @@ -206,7 +207,9 @@ impl TestRunner { } // Check for any asynchronous replies without blocking. - while let Some(reply) = self.threads.as_mut().and_then(ConcurrentRunner::try_get) { + while let Some(reply) = self.threads + .as_mut() + .and_then(ConcurrentRunner::try_get) { self.handle_reply(reply); } } @@ -303,12 +306,12 @@ impl TestRunner { return; } - for t in self.tests.iter().filter(|entry| match **entry { - QueueEntry { state: State::Done(Ok(dur)), .. } => { - dur > cut - } - _ => false, - }) { + for t in self.tests + .iter() + .filter(|entry| match **entry { + QueueEntry { state: State::Done(Ok(dur)), .. } => dur > cut, + _ => false, + }) { println!("slow: {}", t) } diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index ec61c98240..bc16725d74 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -120,5 +120,6 @@ fn run_one_test<'a>(tuple: (&'a SubTest, &'a Flags, Option<&'a TargetIsa>), context.verified = true; } - test.run(func, context).map_err(|e| format!("{}: {}", name, e)) + test.run(func, context) + .map_err(|e| format!("{}: {}", name, e)) } diff --git a/cranelift/src/filetest/subtest.rs b/cranelift/src/filetest/subtest.rs index ede013ef9f..923439d490 100644 --- a/cranelift/src/filetest/subtest.rs +++ b/cranelift/src/filetest/subtest.rs @@ -76,11 +76,13 @@ impl<'a> filecheck::VariableMap for Context<'a> { /// Run filecheck on `text`, using directives extracted from `context`. pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { let checker = build_filechecker(context)?; - if checker.check(&text, context).map_err(|e| format!("filecheck: {}", e))? { + if checker.check(&text, context) + .map_err(|e| format!("filecheck: {}", e))? { Ok(()) } else { // Filecheck mismatch. Emit an explanation as output. - let (_, explain) = checker.explain(&text, context).map_err(|e| format!("explain: {}", e))?; + let (_, explain) = checker.explain(&text, context) + .map_err(|e| format!("explain: {}", e))?; Err(format!("filecheck failed:\n{}{}", checker, explain)) } } @@ -90,10 +92,12 @@ pub fn build_filechecker(context: &Context) -> Result { let mut builder = CheckerBuilder::new(); // Preamble comments apply to all functions. for comment in context.preamble_comments { - builder.directive(comment.text).map_err(|e| format!("filecheck: {}", e))?; + builder.directive(comment.text) + .map_err(|e| format!("filecheck: {}", e))?; } for comment in &context.details.comments { - builder.directive(comment.text).map_err(|e| format!("filecheck: {}", e))?; + builder.directive(comment.text) + .map_err(|e| format!("filecheck: {}", e))?; } let checker = builder.finish(); if checker.is_empty() { diff --git a/cranelift/src/rsfilecheck.rs b/cranelift/src/rsfilecheck.rs index 734c067a36..79d913bce9 100644 --- a/cranelift/src/rsfilecheck.rs +++ b/cranelift/src/rsfilecheck.rs @@ -18,10 +18,12 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { } let mut buffer = String::new(); - io::stdin().read_to_string(&mut buffer).map_err(|e| format!("stdin: {}", e))?; + io::stdin().read_to_string(&mut buffer) + .map_err(|e| format!("stdin: {}", e))?; if verbose { - let (success, explain) = checker.explain(&buffer, NO_VARIABLES).map_err(|e| e.to_string())?; + let (success, explain) = checker.explain(&buffer, NO_VARIABLES) + .map_err(|e| e.to_string())?; print!("{}", explain); if success { println!("OK"); @@ -29,10 +31,12 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { } else { Err("Check failed".to_string()) } - } else if checker.check(&buffer, NO_VARIABLES).map_err(|e| e.to_string())? { + } else if checker.check(&buffer, NO_VARIABLES) + .map_err(|e| e.to_string())? { Ok(()) } else { - let (_, explain) = checker.explain(&buffer, NO_VARIABLES).map_err(|e| e.to_string())?; + let (_, explain) = checker.explain(&buffer, NO_VARIABLES) + .map_err(|e| e.to_string())?; print!("{}", explain); Err("Check failed".to_string()) } @@ -41,6 +45,7 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { fn read_checkfile(filename: &str) -> Result { let buffer = read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))?; let mut builder = CheckerBuilder::new(); - builder.text(&buffer).map_err(|e| format!("{}: {}", filename, e))?; + builder.text(&buffer) + .map_err(|e| format!("{}: {}", filename, e))?; Ok(builder.finish()) } diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 7c4938902f..522cf594a2 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -30,7 +30,7 @@ function banner() { # rustfmt is installed. # # This version should always be bumped to the newest version available. -RUSTFMT_VERSION="0.8.0" +RUSTFMT_VERSION="0.8.1" if cargo install --list | grep -q "^rustfmt v$RUSTFMT_VERSION"; then banner "Rust formatting" diff --git a/cranelift/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs index f7fb119463..903b1a1ec1 100644 --- a/cranelift/tests/cfg_traversal.rs +++ b/cranelift/tests/cfg_traversal.rs @@ -9,7 +9,10 @@ use self::cton_reader::parse_functions; fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) { let func = &parse_functions(function_source).unwrap()[0]; let cfg = ControlFlowGraph::with_function(&func); - let ebbs = ebb_order.iter().map(|n| Ebb::with_number(*n).unwrap()).collect::>(); + let ebbs = ebb_order + .iter() + .map(|n| Ebb::with_number(*n).unwrap()) + .collect::>(); let mut postorder_ebbs = cfg.postorder_ebbs(); let mut postorder_map = EntityMap::with_capacity(postorder_ebbs.len()); diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index 14a034b8fb..32210fecfb 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -108,7 +108,10 @@ pub fn legalize_args(args: &mut Vec, aa: &mut AA) } // Split this argument into two smaller ones. Then revisit both. ArgAction::Convert(conv) => { - let new_arg = ArgumentType { value_type: conv.apply(arg.value_type), ..arg }; + let new_arg = ArgumentType { + value_type: conv.apply(arg.value_type), + ..arg + }; args[argno].value_type = new_arg.value_type; if conv.is_split() { args.insert(argno + 1, new_arg); diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 2d5631f62c..051adc47aa 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -69,6 +69,7 @@ impl Context { /// Run the register allocator. pub fn regalloc(&mut self, isa: &TargetIsa) { - self.regalloc.run(isa, &mut self.func, &self.cfg, &self.domtree); + self.regalloc + .run(isa, &mut self.func, &self.cfg, &self.domtree); } } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index b014e85ec1..13d1175d8a 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -88,7 +88,8 @@ impl DominatorTree { // Run a finger up the dominator tree from b until we see a. // Do nothing if b is unreachable. while rpo_a < self.nodes[ebb_b].rpo_number { - b = self.idom(ebb_b).expect("Shouldn't meet unreachable here."); + b = self.idom(ebb_b) + .expect("Shouldn't meet unreachable here."); ebb_b = layout.inst_ebb(b).expect("Dominator got removed."); } @@ -209,8 +210,9 @@ impl DominatorTree { .filter(|&(ebb, _)| self.is_reachable(ebb)); // The RPO must visit at least one predecessor before this node. - let mut idom = - reachable_preds.next().expect("EBB node must have one reachable predecessor"); + let mut idom = reachable_preds + .next() + .expect("EBB node must have one reachable predecessor"); for pred in reachable_preds { idom = self.common_dominator(idom, pred, layout); diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index 1a05464c21..0b90265470 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -135,7 +135,9 @@ impl ListPool { // The `wrapping_sub` handles the special case 0, which is the empty list. This way, the // cost of the bounds check that we have to pay anyway is co-opted to handle the special // case of the empty list. - self.data.get(idx.wrapping_sub(1)).map(|len| len.index()) + self.data + .get(idx.wrapping_sub(1)) + .map(|len| len.index()) } /// Allocate a storage block with a size given by `sclass`. diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 8880351d9b..fb5c4ba20a 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -65,10 +65,7 @@ impl<'c, 'fc, 'fd> InsertBuilder<'c, 'fc, 'fd> { pub fn new(dfg: &'fd mut DataFlowGraph, pos: &'c mut Cursor<'fc>) -> InsertBuilder<'c, 'fc, 'fd> { - InsertBuilder { - dfg: dfg, - pos: pos, - } + InsertBuilder { dfg: dfg, pos: pos } } } @@ -182,8 +179,9 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { // Normally, make_inst_results() would also set the first result type, but we're not // going to call that, so set it manually. - *self.dfg[self.inst].first_type_mut() = - self.dfg.compute_result_type(self.inst, 0, ctrl_typevar).unwrap_or_default(); + *self.dfg[self.inst].first_type_mut() = self.dfg + .compute_result_type(self.inst, 0, ctrl_typevar) + .unwrap_or_default(); } (self.inst, self.dfg) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index f7d214182b..22ce7a116d 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -356,25 +356,37 @@ impl DataFlowGraph { /// Get the fixed value arguments on `inst` as a slice. pub fn inst_fixed_args(&self, inst: Inst) -> &[Value] { - let fixed_args = self[inst].opcode().constraints().fixed_value_arguments(); + let fixed_args = self[inst] + .opcode() + .constraints() + .fixed_value_arguments(); &self.inst_args(inst)[..fixed_args] } /// Get the fixed value arguments on `inst` as a mutable slice. pub fn inst_fixed_args_mut(&mut self, inst: Inst) -> &mut [Value] { - let fixed_args = self[inst].opcode().constraints().fixed_value_arguments(); + let fixed_args = self[inst] + .opcode() + .constraints() + .fixed_value_arguments(); &mut self.inst_args_mut(inst)[..fixed_args] } /// Get the variable value arguments on `inst` as a slice. pub fn inst_variable_args(&self, inst: Inst) -> &[Value] { - let fixed_args = self[inst].opcode().constraints().fixed_value_arguments(); + let fixed_args = self[inst] + .opcode() + .constraints() + .fixed_value_arguments(); &self.inst_args(inst)[fixed_args..] } /// Get the variable value arguments on `inst` as a mutable slice. pub fn inst_variable_args_mut(&mut self, inst: Inst) -> &mut [Value] { - let fixed_args = self[inst].opcode().constraints().fixed_value_arguments(); + let fixed_args = self[inst] + .opcode() + .constraints() + .fixed_value_arguments(); &mut self.inst_args_mut(inst)[fixed_args..] } @@ -444,8 +456,8 @@ impl DataFlowGraph { // Update the second_result pointer in `inst`. if head.is_some() { *self.insts[inst] - .second_result_mut() - .expect("instruction format doesn't allow multiple results") = head.into(); + .second_result_mut() + .expect("instruction format doesn't allow multiple results") = head.into(); } *self.insts[inst].first_type_mut() = first_type.unwrap_or_default(); @@ -505,7 +517,12 @@ impl DataFlowGraph { (inst, 1) } ExpandedValue::Table(idx) => { - if let ValueData::Inst { num, inst, ref mut next, .. } = self.extended_values[idx] { + if let ValueData::Inst { + num, + inst, + ref mut next, + .. + } = self.extended_values[idx] { assert!(next.is_none(), "last_res is not the last result"); *next = res.into(); assert!(num < u16::MAX, "Too many arguments to EBB"); @@ -518,8 +535,12 @@ impl DataFlowGraph { // Now update `res` itself. if let ExpandedValue::Table(idx) = res.expand() { - if let ValueData::Inst { ref mut num, ref mut inst, ref mut next, .. } = - self.extended_values[idx] { + if let ValueData::Inst { + ref mut num, + ref mut inst, + ref mut next, + .. + } = self.extended_values[idx] { *num = res_num; *inst = res_inst; *next = None.into(); @@ -565,7 +586,8 @@ impl DataFlowGraph { /// /// Returns the new `Inst` reference where the original instruction has been moved. pub fn redefine_first_value(&mut self, pos: &mut Cursor) -> Inst { - let orig = pos.current_inst().expect("Cursor must point at an instruction"); + let orig = pos.current_inst() + .expect("Cursor must point at an instruction"); let data = self[orig].clone(); // After cloning, any secondary values are attached to both copies. Don't do that, we only // want them on the new clone. @@ -630,12 +652,13 @@ impl DataFlowGraph { } // Not a fixed result, try to extract a return type from the call signature. - self.call_signature(inst).and_then(|sigref| { - self.signatures[sigref] - .return_types - .get(result_idx - fixed_results) - .map(|&arg| arg.value_type) - }) + self.call_signature(inst) + .and_then(|sigref| { + self.signatures[sigref] + .return_types + .get(result_idx - fixed_results) + .map(|&arg| arg.value_type) + }) } } @@ -815,8 +838,12 @@ impl DataFlowGraph { // Now update `arg` itself. let arg_ebb = ebb; if let ExpandedValue::Table(idx) = arg.expand() { - if let ValueData::Arg { ref mut num, ebb, ref mut next, .. } = - self.extended_values[idx] { + if let ValueData::Arg { + ref mut num, + ebb, + ref mut next, + .. + } = self.extended_values[idx] { *num = arg_num; *next = None.into(); assert_eq!(arg_ebb, ebb, "{} should already belong to EBB", arg); diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 42d109bf7a..876598d13a 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -286,14 +286,18 @@ mod tests { assert_eq!(Value::table_with_number(1).unwrap().to_string(), "vx1"); assert_eq!(Value::direct_with_number(u32::MAX / 2), None); - assert_eq!(match Value::direct_with_number(u32::MAX / 2 - 1).unwrap().expand() { + assert_eq!(match Value::direct_with_number(u32::MAX / 2 - 1) + .unwrap() + .expand() { ExpandedValue::Direct(i) => i.index() as u32, _ => u32::MAX, }, u32::MAX / 2 - 1); assert_eq!(Value::table_with_number(u32::MAX / 2), None); - assert_eq!(match Value::table_with_number(u32::MAX / 2 - 1).unwrap().expand() { + assert_eq!(match Value::table_with_number(u32::MAX / 2 - 1) + .unwrap() + .expand() { ExpandedValue::Table(i) => i as u32, _ => u32::MAX, }, diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index bc293d1169..43daaed192 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -221,7 +221,8 @@ mod tests { assert_eq!(sig.to_string(), "(i32)"); sig.return_types.push(ArgumentType::new(F32)); assert_eq!(sig.to_string(), "(i32) -> f32"); - sig.argument_types.push(ArgumentType::new(I32.by(4).unwrap())); + sig.argument_types + .push(ArgumentType::new(I32.by(4).unwrap())); assert_eq!(sig.to_string(), "(i32, i32x4) -> f32"); sig.return_types.push(ArgumentType::new(B8)); assert_eq!(sig.to_string(), "(i32, i32x4) -> f32, b8"); diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index d3ca0bc62f..1c292f473a 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -304,15 +304,21 @@ impl InstructionData { /// here. pub fn analyze_branch<'a>(&'a self, pool: &'a ValueListPool) -> BranchInfo<'a> { match self { - &InstructionData::Jump { destination, ref args, .. } => { - BranchInfo::SingleDest(destination, &args.as_slice(pool)) - } - &InstructionData::Branch { destination, ref args, .. } => { - BranchInfo::SingleDest(destination, &args.as_slice(pool)[1..]) - } - &InstructionData::BranchIcmp { destination, ref args, .. } => { - BranchInfo::SingleDest(destination, &args.as_slice(pool)[2..]) - } + &InstructionData::Jump { + destination, + ref args, + .. + } => BranchInfo::SingleDest(destination, &args.as_slice(pool)), + &InstructionData::Branch { + destination, + ref args, + .. + } => BranchInfo::SingleDest(destination, &args.as_slice(pool)[1..]), + &InstructionData::BranchIcmp { + destination, + ref args, + .. + } => BranchInfo::SingleDest(destination, &args.as_slice(pool)[2..]), &InstructionData::BranchTable { table, .. } => BranchInfo::Table(table), _ => BranchInfo::NotABranch, } @@ -601,9 +607,21 @@ impl OperandConstraint { Same => Bound(ctrl_type), LaneOf => Bound(ctrl_type.lane_type()), AsBool => Bound(ctrl_type.as_bool()), - HalfWidth => Bound(ctrl_type.half_width().expect("invalid type for half_width")), - DoubleWidth => Bound(ctrl_type.double_width().expect("invalid type for double_width")), - HalfVector => Bound(ctrl_type.half_vector().expect("invalid type for half_vector")), + HalfWidth => { + Bound(ctrl_type + .half_width() + .expect("invalid type for half_width")) + } + DoubleWidth => { + Bound(ctrl_type + .double_width() + .expect("invalid type for double_width")) + } + HalfVector => { + Bound(ctrl_type + .half_vector() + .expect("invalid type for half_vector")) + } DoubleVector => Bound(ctrl_type.by(2).expect("invalid type for double_vector")), } } diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index e7f98a329d..dd9bc5c187 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -66,15 +66,14 @@ impl JumpTableData { /// /// This returns an iterator that skips any empty slots in the table. pub fn entries<'a>(&'a self) -> Entries { - Entries(self.table - .iter() - .cloned() - .enumerate()) + Entries(self.table.iter().cloned().enumerate()) } /// Checks if any of the entries branch to `ebb`. pub fn branches_to(&self, ebb: Ebb) -> bool { - self.table.iter().any(|target_ebb| target_ebb.expand() == Some(ebb)) + self.table + .iter() + .any(|target_ebb| target_ebb.expand() == Some(ebb)) } /// Access the whole table as a mutable slice. @@ -109,10 +108,7 @@ impl Display for JumpTableData { Some(first) => write!(fmt, "jump_table {}", first)?, } - for dest in self.table - .iter() - .skip(1) - .map(|e| e.expand()) { + for dest in self.table.iter().skip(1).map(|e| e.expand()) { match dest { None => write!(fmt, ", 0")?, Some(ebb) => write!(fmt, ", {}", ebb)?, diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index c34d6a0aba..4f5d2355b2 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -125,7 +125,10 @@ impl Layout { /// Get the last sequence number in `ebb`. fn last_ebb_seq(&self, ebb: Ebb) -> SequenceNumber { // Get the seq of the last instruction if it exists, otherwise use the EBB header seq. - self.ebbs[ebb].last_inst.map(|inst| self.insts[inst].seq).unwrap_or(self.ebbs[ebb].seq) + self.ebbs[ebb] + .last_inst + .map(|inst| self.insts[inst].seq) + .unwrap_or(self.ebbs[ebb].seq) } /// Assign a valid sequence number to `ebb` such that the numbers are still monotonic. This may @@ -134,7 +137,10 @@ impl Layout { assert!(self.is_ebb_inserted(ebb)); // Get the sequence number immediately before `ebb`, or 0. - let prev_seq = self.ebbs[ebb].prev.map(|prev_ebb| self.last_ebb_seq(prev_ebb)).unwrap_or(0); + let prev_seq = self.ebbs[ebb] + .prev + .map(|prev_ebb| self.last_ebb_seq(prev_ebb)) + .unwrap_or(0); // Get the sequence number immediately following `ebb`. let next_seq = if let Some(inst) = self.ebbs[ebb].first_inst.expand() { @@ -159,7 +165,8 @@ impl Layout { /// Assign a valid sequence number to `inst` such that the numbers are still monotonic. This may /// require renumbering. fn assign_inst_seq(&mut self, inst: Inst) { - let ebb = self.inst_ebb(inst).expect("inst must be inserted before assigning an seq"); + let ebb = self.inst_ebb(inst) + .expect("inst must be inserted before assigning an seq"); // Get the sequence number immediately before `inst`. let prev_seq = match self.insts[inst].prev.expand() { @@ -436,8 +443,8 @@ impl Layout { /// Insert `inst` before the instruction `before` in the same EBB. pub fn insert_inst(&mut self, inst: Inst, before: Inst) { assert_eq!(self.inst_ebb(inst), None); - let ebb = - self.inst_ebb(before).expect("Instruction before insertion point not in the layout"); + let ebb = self.inst_ebb(before) + .expect("Instruction before insertion point not in the layout"); let after = self.insts[before].prev; { let inst_node = self.insts.ensure(inst); @@ -485,8 +492,8 @@ impl Layout { /// i4 /// ``` pub fn split_ebb(&mut self, new_ebb: Ebb, before: Inst) { - let old_ebb = - self.inst_ebb(before).expect("The `before` instruction must be in the layout"); + let old_ebb = self.inst_ebb(before) + .expect("The `before` instruction must be in the layout"); assert!(!self.is_ebb_inserted(new_ebb)); // Insert new_ebb after old_ebb. @@ -786,8 +793,9 @@ impl<'f> Cursor<'f> { self.pos = At(next); Some(next) } else { - self.pos = - After(self.layout.inst_ebb(inst).expect("current instruction removed?")); + self.pos = After(self.layout + .inst_ebb(inst) + .expect("current instruction removed?")); None } } @@ -837,8 +845,9 @@ impl<'f> Cursor<'f> { self.pos = At(prev); Some(prev) } else { - self.pos = - Before(self.layout.inst_ebb(inst).expect("current instruction removed?")); + self.pos = Before(self.layout + .inst_ebb(inst) + .expect("current instruction removed?")); None } } diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 1321636add..550f25804b 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -348,12 +348,7 @@ mod tests { assert_eq!(big.bits(), 64 * 256); assert_eq!(big.half_vector().unwrap().to_string(), "f64x128"); - assert_eq!(B1.by(2) - .unwrap() - .half_vector() - .unwrap() - .to_string(), - "b1"); + assert_eq!(B1.by(2).unwrap().half_vector().unwrap().to_string(), "b1"); assert_eq!(I32.half_vector(), None); assert_eq!(VOID.half_vector(), None); @@ -383,12 +378,7 @@ mod tests { assert_eq!(B1.by(8).unwrap().to_string(), "b1x8"); assert_eq!(B8.by(1).unwrap().to_string(), "b8"); assert_eq!(B16.by(256).unwrap().to_string(), "b16x256"); - assert_eq!(B32.by(4) - .unwrap() - .by(2) - .unwrap() - .to_string(), - "b32x8"); + assert_eq!(B32.by(4).unwrap().by(2).unwrap().to_string(), "b32x8"); assert_eq!(B64.by(8).unwrap().to_string(), "b64x8"); assert_eq!(I8.by(64).unwrap().to_string(), "i8x64"); assert_eq!(F64.by(2).unwrap().to_string(), "f64x2"); diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index 71a3eba2e0..cfb0066b8c 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -90,13 +90,13 @@ pub fn lookup_enclist(ctrl_typevar: Type, Legalize::Expand }) .and_then(|l1idx| { - let l1ent = &level1_table[l1idx]; - let l2off = l1ent.offset.into() as usize; - let l2tab = &level2_table[l2off..l2off + (1 << l1ent.log2len)]; - probe(l2tab, opcode, opcode as usize) - .map(|l2idx| l2tab[l2idx].offset.into() as usize) - .ok_or(Legalize::Expand) - }) + let l1ent = &level1_table[l1idx]; + let l2off = l1ent.offset.into() as usize; + let l2tab = &level2_table[l2off..l2off + (1 << l1ent.log2len)]; + probe(l2tab, opcode, opcode as usize) + .map(|l2idx| l2tab[l2idx].offset.into() as usize) + .ok_or(Legalize::Expand) + }) } /// Encoding list entry. diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 057fac11aa..213d8a132e 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -235,7 +235,11 @@ fn put_sb(bits: u16, rs1: RegUnit, rs2: RegUnit, sink: &m } fn recipe_sb(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::BranchIcmp { destination, ref args, .. } = func.dfg[inst] { + if let InstructionData::BranchIcmp { + destination, + ref args, + .. + } = func.dfg[inst] { let args = &args.as_slice(&func.dfg.value_lists)[0..2]; sink.reloc_ebb(RelocKind::Branch.into(), destination); put_sb(func.encodings[inst].bits(), @@ -248,7 +252,11 @@ fn recipe_sb(func: &Function, inst: Inst, sink: &mut CS) } fn recipe_sbzero(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Branch { destination, ref args, .. } = func.dfg[inst] { + if let InstructionData::Branch { + destination, + ref args, + .. + } = func.dfg[inst] { let args = &args.as_slice(&func.dfg.value_lists)[0..1]; sink.reloc_ebb(RelocKind::Branch.into(), destination); put_sb(func.encodings[inst].bits(), diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 2e3766057d..19e06b7ba4 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -107,7 +107,8 @@ fn legalize_inst_results(dfg: &mut DataFlowGraph, -> Inst where ResType: FnMut(&DataFlowGraph, usize) -> ArgumentType { - let mut call = pos.current_inst().expect("Cursor must point to a call instruction"); + let mut call = pos.current_inst() + .expect("Cursor must point to a call instruction"); // We theoretically allow for call instructions that return a number of fixed results before // the call return values. In practice, it doesn't happen. @@ -364,10 +365,13 @@ fn legalize_inst_arguments(dfg: &mut DataFlowGraph, mut get_abi_type: ArgType) where ArgType: FnMut(&DataFlowGraph, usize) -> ArgumentType { - let inst = pos.current_inst().expect("Cursor must point to a call instruction"); + let inst = pos.current_inst() + .expect("Cursor must point to a call instruction"); // Lift the value list out of the call instruction so we modify it. - let mut vlist = dfg[inst].take_value_list().expect("Call must have a value list"); + let mut vlist = dfg[inst] + .take_value_list() + .expect("Call must have a value list"); // The value list contains all arguments to the instruction, including the callee on an // indirect call which isn't part of the call arguments that must match the ABI signature. @@ -405,7 +409,9 @@ fn legalize_inst_arguments(dfg: &mut DataFlowGraph, let mut abi_arg = 0; for old_arg in 0..have_args { - let old_value = vlist.get(old_arg_offset + old_arg, &dfg.value_lists).unwrap(); + let old_value = vlist + .get(old_arg_offset + old_arg, &dfg.value_lists) + .unwrap(); let mut put_arg = |dfg: &mut DataFlowGraph, arg| { let abi_type = get_abi_type(dfg, abi_arg); if dfg.value_type(arg) == abi_type.value_type { @@ -435,7 +441,8 @@ fn legalize_inst_arguments(dfg: &mut DataFlowGraph, /// /// Returns `true` if any instructions were inserted. pub fn handle_call_abi(dfg: &mut DataFlowGraph, cfg: &ControlFlowGraph, pos: &mut Cursor) -> bool { - let mut inst = pos.current_inst().expect("Cursor must point to a call instruction"); + let mut inst = pos.current_inst() + .expect("Cursor must point to a call instruction"); // Start by checking if the argument types already match the signature. let sig_ref = match check_call_signature(dfg, inst) { @@ -475,7 +482,8 @@ pub fn handle_return_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor, sig: &Signature) -> bool { - let inst = pos.current_inst().expect("Cursor must point to a return instruction"); + let inst = pos.current_inst() + .expect("Cursor must point to a return instruction"); // Check if the returned types already match the signature. if check_return_signature(dfg, inst, sig) { diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index 3dda54abe7..d1f55a6266 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -125,7 +125,9 @@ fn split_any(dfg: &mut DataFlowGraph, "Predecessor not a branch: {}", dfg.display_inst(inst)); let fixed_args = branch_opc.constraints().fixed_value_arguments(); - let mut args = dfg[inst].take_value_list().expect("Branches must have value lists."); + let mut args = dfg[inst] + .take_value_list() + .expect("Branches must have value lists."); let num_args = args.len(&dfg.value_lists); // Get the old value passed to the EBB argument we're repairing. let old_arg = args.get(fixed_args + repair.num, &dfg.value_lists) @@ -142,12 +144,14 @@ fn split_any(dfg: &mut DataFlowGraph, let (lo, hi) = split_value(dfg, pos, old_arg, repair.concat, &mut repairs); // The `lo` part replaces the original argument. - *args.get_mut(fixed_args + repair.num, &mut dfg.value_lists).unwrap() = lo; + *args.get_mut(fixed_args + repair.num, &mut dfg.value_lists) + .unwrap() = lo; // The `hi` part goes at the end. Since multiple repairs may have been scheduled to the // same EBB, there could be multiple arguments missing. if num_args > fixed_args + repair.hi_num { - *args.get_mut(fixed_args + repair.hi_num, &mut dfg.value_lists).unwrap() = hi; + *args.get_mut(fixed_args + repair.hi_num, &mut dfg.value_lists) + .unwrap() = hi; } else { // We need to append one or more arguments. If we're adding more than one argument, // there must be pending repairs on the stack that will fill in the correct values diff --git a/lib/cretonne/src/partition_slice.rs b/lib/cretonne/src/partition_slice.rs index 0986613040..9626b5fd37 100644 --- a/lib/cretonne/src/partition_slice.rs +++ b/lib/cretonne/src/partition_slice.rs @@ -35,10 +35,7 @@ mod tests { fn check(x: &[u32], want: &[u32]) { assert_eq!(x.len(), want.len()); - let want_count = want.iter() - .cloned() - .filter(|&x| x % 10 == 0) - .count(); + let want_count = want.iter().cloned().filter(|&x| x % 10 == 0).count(); let mut v = Vec::new(); v.extend(x.iter().cloned()); let count = partition_slice(&mut v[..], |&x| x % 10 == 0); diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 9bc10c30b3..b8c08bd278 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -232,7 +232,9 @@ impl<'a> Context<'a> { if let Affinity::Reg(rc_index) = lv.affinity { let regclass = self.reginfo.rc(rc_index); // TODO: Fall back to a top-level super-class. Sub-classes are only hints. - let regunit = regs.iter(regclass).next().expect("Out of registers for arguments"); + let regunit = regs.iter(regclass) + .next() + .expect("Out of registers for arguments"); regs.take(regclass, regunit); *locations.ensure(lv.value) = ValueLoc::Reg(regunit); } diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 195c480372..2c59ab9797 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -52,6 +52,7 @@ impl Context { // TODO: Second pass: Spilling. // Third pass: Reload and coloring. - self.coloring.run(isa, func, domtree, &mut self.liveness, &mut self.tracker); + self.coloring + .run(isa, func, domtree, &mut self.liveness, &mut self.tracker); } } diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 19ed62e8fb..75b18ef019 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -68,11 +68,12 @@ impl LiveValueVec { /// Add a new live value to `values`. fn push(&mut self, value: Value, endpoint: Inst, affinity: Affinity) { - self.values.push(LiveValue { - value: value, - endpoint: endpoint, - affinity: affinity, - }); + self.values + .push(LiveValue { + value: value, + endpoint: endpoint, + affinity: affinity, + }); } /// Remove all elements. @@ -163,11 +164,14 @@ impl LiveValueTracker { // If the immediate dominator exits, we must have a stored list for it. This is a // requirement to the order EBBs are visited: All dominators must have been processed // before the current EBB. - let idom_live_list = - self.idom_sets.get(&idom).expect("No stored live set for dominator"); + let idom_live_list = self.idom_sets + .get(&idom) + .expect("No stored live set for dominator"); // Get just the values that are live-in to `ebb`. for &value in idom_live_list.as_slice(&self.idom_pool) { - let lr = liveness.get(value).expect("Immediate dominator value has no live range"); + let lr = liveness + .get(value) + .expect("Immediate dominator value has no live range"); // Check if this value is live-in here. if let Some(endpoint) = lr.livein_local_end(ebb, program_order) { @@ -179,7 +183,9 @@ impl LiveValueTracker { // Now add all the live arguments to `ebb`. let first_arg = self.live.values.len(); for value in dfg.ebb_args(ebb) { - let lr = liveness.get(value).expect("EBB argument value has no live range"); + let lr = liveness + .get(value) + .expect("EBB argument value has no live range"); assert_eq!(lr.def(), ebb.into()); match lr.def_local_end().into() { ExpandedProgramPoint::Inst(endpoint) => { @@ -259,16 +265,15 @@ impl LiveValueTracker { /// Save the current set of live values so it is associated with `idom`. fn save_idom_live_set(&mut self, idom: Inst) { - let values = self.live - .values - .iter() - .map(|lv| lv.value); + let values = self.live.values.iter().map(|lv| lv.value); let pool = &mut self.idom_pool; // If there already is a set saved for `idom`, just keep it. - self.idom_sets.entry(idom).or_insert_with(|| { - let mut list = ValueList::default(); - list.extend(values, pool); - list - }); + self.idom_sets + .entry(idom) + .or_insert_with(|| { + let mut list = ValueList::default(); + list.extend(values, pool); + list + }); } } diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 40faff0591..3344c5b841 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -205,7 +205,8 @@ fn get_or_create<'a>(lrset: &'a mut LiveRangeSet, def = inst.into(); // Initialize the affinity from the defining instruction's result constraints. // Don't do this for call return values which are always tied to a single register. - affinity = recipe_constraints.get(func.encodings[inst].recipe()) + affinity = recipe_constraints + .get(func.encodings[inst].recipe()) .and_then(|rc| rc.outs.get(rnum)) .map(Affinity::new) .unwrap_or_default(); @@ -315,7 +316,8 @@ impl Liveness { let recipe = func.encodings[inst].recipe(); // Iterator of constraints, one per value operand. // TODO: Should we fail here if the instruction doesn't have a valid encoding? - let mut operand_constraints = recipe_constraints.get(recipe) + let mut operand_constraints = recipe_constraints + .get(recipe) .map(|c| c.ins) .unwrap_or(&[]) .iter(); diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 116ccce4aa..94c16cb9ae 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -221,14 +221,16 @@ impl LiveRange { /// Return `Ok(n)` if `liveins[n]` already contains `ebb`. /// Otherwise, return `Err(n)` with the index where such an interval should be inserted. fn find_ebb_interval(&self, ebb: Ebb, order: &PO) -> Result { - self.liveins.binary_search_by(|intv| order.cmp(intv.begin, ebb)).or_else(|n| { - // The interval at `n-1` may cover `ebb`. - if n > 0 && order.cmp(self.liveins[n - 1].end, ebb) == Ordering::Greater { - Ok(n - 1) - } else { - Err(n) - } - }) + self.liveins + .binary_search_by(|intv| order.cmp(intv.begin, ebb)) + .or_else(|n| { + // The interval at `n-1` may cover `ebb`. + if n > 0 && order.cmp(self.liveins[n - 1].end, ebb) == Ordering::Greater { + Ok(n - 1) + } else { + Err(n) + } + }) } /// Extend the local interval for `ebb` so it reaches `to` which must belong to `ebb`. @@ -307,11 +309,12 @@ impl LiveRange { } // Cannot coalesce; insert new interval (false, false) => { - self.liveins.insert(n, - Interval { - begin: ebb, - end: to, - }); + self.liveins + .insert(n, + Interval { + begin: ebb, + end: to, + }); } } @@ -361,7 +364,9 @@ impl LiveRange { /// answer, but it is also possible that an even later program point is returned. So don't /// depend on the returned `Inst` to belong to `ebb`. pub fn livein_local_end(&self, ebb: Ebb, order: &PO) -> Option { - self.find_ebb_interval(ebb, order).ok().map(|n| self.liveins[n].end) + self.find_ebb_interval(ebb, order) + .ok() + .map(|n| self.liveins[n].end) } } diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 589f117e73..776755d4c4 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -177,8 +177,9 @@ impl<'a> Verifier<'a> { let fixed_results = inst_data.opcode().constraints().fixed_results(); // var_results is 0 if we aren't a call instruction - let var_results = - dfg.call_signature(inst).map(|sig| dfg.signatures[sig].return_types.len()).unwrap_or(0); + let var_results = dfg.call_signature(inst) + .map(|sig| dfg.signatures[sig].return_types.len()) + .unwrap_or(0); let total_results = fixed_results + var_results; if total_results == 0 { @@ -218,9 +219,21 @@ impl<'a> Verifier<'a> { &MultiAry { ref args, .. } => { self.verify_value_list(inst, args)?; } - &Jump { destination, ref args, .. } | - &Branch { destination, ref args, .. } | - &BranchIcmp { destination, ref args, .. } => { + &Jump { + destination, + ref args, + .. + } | + &Branch { + destination, + ref args, + .. + } | + &BranchIcmp { + destination, + ref args, + .. + } => { self.verify_ebb(inst, destination)?; self.verify_value_list(inst, args)?; } @@ -265,10 +278,7 @@ impl<'a> Verifier<'a> { } fn verify_sig_ref(&self, inst: Inst, s: SigRef) -> Result<()> { - if !self.func - .dfg - .signatures - .is_valid(s) { + if !self.func.dfg.signatures.is_valid(s) { err!(inst, "invalid signature reference {}", s) } else { Ok(()) @@ -276,10 +286,7 @@ impl<'a> Verifier<'a> { } fn verify_func_ref(&self, inst: Inst, f: FuncRef) -> Result<()> { - if !self.func - .dfg - .ext_funcs - .is_valid(f) { + if !self.func.dfg.ext_funcs.is_valid(f) { err!(inst, "invalid function reference {}", f) } else { Ok(()) @@ -326,7 +333,8 @@ impl<'a> Verifier<'a> { def_inst); } // Defining instruction dominates the instruction that uses the value. - if !self.domtree.dominates(def_inst, loc_inst, &self.func.layout) { + if !self.domtree + .dominates(def_inst, loc_inst, &self.func.layout) { return err!(loc_inst, "uses value from non-dominating {}", def_inst); } } @@ -343,7 +351,8 @@ impl<'a> Verifier<'a> { ebb); } // The defining EBB dominates the instruction using this value. - if !self.domtree.ebb_dominates(ebb, loc_inst, &self.func.layout) { + if !self.domtree + .ebb_dominates(ebb, loc_inst, &self.func.layout) { return err!(loc_inst, "uses value arg from non-dominating {}", ebb); } } @@ -378,10 +387,7 @@ impl<'a> Verifier<'a> { return err!(ebb, "entry block arguments must match function signature"); } - for (i, arg) in self.func - .dfg - .ebb_args(ebb) - .enumerate() { + for (i, arg) in self.func.dfg.ebb_args(ebb).enumerate() { let arg_type = self.func.dfg.value_type(arg); if arg_type != expected_types[i].value_type { return err!(ebb, @@ -452,11 +458,7 @@ impl<'a> Verifier<'a> { fn typecheck_fixed_args(&self, inst: Inst, ctrl_type: Type) -> Result<()> { let constraints = self.func.dfg[inst].opcode().constraints(); - for (i, &arg) in self.func - .dfg - .inst_fixed_args(inst) - .iter() - .enumerate() { + for (i, &arg) in self.func.dfg.inst_fixed_args(inst).iter().enumerate() { let arg_type = self.func.dfg.value_type(arg); match constraints.value_argument_constraint(i, ctrl_type) { ResolvedConstraint::Bound(expected_type) => { @@ -510,13 +512,17 @@ impl<'a> Verifier<'a> { match self.func.dfg[inst].analyze_call(&self.func.dfg.value_lists) { CallInfo::Direct(func_ref, _) => { let sig_ref = self.func.dfg.ext_funcs[func_ref].signature; - let arg_types = - self.func.dfg.signatures[sig_ref].argument_types.iter().map(|a| a.value_type); + let arg_types = self.func.dfg.signatures[sig_ref] + .argument_types + .iter() + .map(|a| a.value_type); self.typecheck_variable_args_iterator(inst, arg_types)?; } CallInfo::Indirect(sig_ref, _) => { - let arg_types = - self.func.dfg.signatures[sig_ref].argument_types.iter().map(|a| a.value_type); + let arg_types = self.func.dfg.signatures[sig_ref] + .argument_types + .iter() + .map(|a| a.value_type); self.typecheck_variable_args_iterator(inst, arg_types)?; } CallInfo::NotACall => {} @@ -673,10 +679,11 @@ mod tests { let mut func = Function::new(); let ebb0 = func.dfg.make_ebb(); func.layout.append_ebb(ebb0); - let nullary_with_bad_opcode = func.dfg.make_inst(InstructionData::Nullary { - opcode: Opcode::Jump, - ty: types::VOID, - }); + let nullary_with_bad_opcode = func.dfg + .make_inst(InstructionData::Nullary { + opcode: Opcode::Jump, + ty: types::VOID, + }); func.layout.append_inst(nullary_with_bad_opcode, ebb0); let verifier = Verifier::new(&func); assert_err_with_msg!(verifier.run(), "instruction format"); diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index e1ac42ac09..e2cd0fe3f5 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -262,7 +262,11 @@ pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result IntCompare { cond, args, .. } => write!(w, " {}, {}, {}", cond, args[0], args[1]), IntCompareImm { cond, arg, imm, .. } => write!(w, " {}, {}, {}", cond, arg, imm), FloatCompare { cond, args, .. } => write!(w, " {}, {}, {}", cond, args[0], args[1]), - Jump { destination, ref args, .. } => { + Jump { + destination, + ref args, + .. + } => { if args.is_empty() { write!(w, " {}", destination) } else { @@ -272,7 +276,11 @@ pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result DisplayValues(args.as_slice(pool))) } } - Branch { destination, ref args, .. } => { + Branch { + destination, + ref args, + .. + } => { let args = args.as_slice(pool); write!(w, " {}, {}", args[0], destination)?; if args.len() > 1 { @@ -280,7 +288,12 @@ pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result } Ok(()) } - BranchIcmp { cond, destination, ref args, .. } => { + BranchIcmp { + cond, + destination, + ref args, + .. + } => { let args = args.as_slice(pool); write!(w, " {}, {}, {}, {}", cond, args[0], args[1], destination)?; if args.len() > 2 { diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs index c854f3b6b8..031a50d9e3 100644 --- a/lib/filecheck/src/checker.rs +++ b/lib/filecheck/src/checker.rs @@ -183,11 +183,13 @@ impl Checker { continue; } Directive::Regex(ref var, ref rx) => { - state.vars.insert(var.clone(), - VarDef { - value: Value::Regex(Cow::Borrowed(rx)), - offset: 0, - }); + state + .vars + .insert(var.clone(), + VarDef { + value: Value::Regex(Cow::Borrowed(rx)), + offset: 0, + }); continue; } }; @@ -208,10 +210,14 @@ impl Checker { state.recorder.directive(not_idx); if let Some((s, e)) = rx.find(&text[not_begin..match_begin]) { // Matched `not:` pattern. - state.recorder.matched_not(rx.as_str(), (not_begin + s, not_begin + e)); + state + .recorder + .matched_not(rx.as_str(), (not_begin + s, not_begin + e)); return Ok(false); } else { - state.recorder.missed_not(rx.as_str(), (not_begin, match_begin)); + state + .recorder + .missed_not(rx.as_str(), (not_begin, match_begin)); } } } @@ -410,9 +416,11 @@ mod tests { Ok(true)); assert_eq!(b.directive("regex: X = tommy").map_err(e2s), Err("expected '=' after variable 'X' in regex: X = tommy".to_string())); - assert_eq!(b.directive("[arm]not: patt $x $(y) here").map_err(e2s), + assert_eq!(b.directive("[arm]not: patt $x $(y) here") + .map_err(e2s), Ok(true)); - assert_eq!(b.directive("[x86]sameln: $x $(y=[^]]*) there").map_err(e2s), + assert_eq!(b.directive("[x86]sameln: $x $(y=[^]]*) there") + .map_err(e2s), Ok(true)); // Windows line ending sneaking in. assert_eq!(b.directive("regex: Y=foo\r").map_err(e2s), Ok(true)); diff --git a/lib/filecheck/src/explain.rs b/lib/filecheck/src/explain.rs index 04a6cd233e..49780321b5 100644 --- a/lib/filecheck/src/explain.rs +++ b/lib/filecheck/src/explain.rs @@ -119,7 +119,8 @@ impl<'a> Display for Explainer<'a> { m.regex)?; // Emit any variable definitions. - if let Ok(found) = self.vardefs.binary_search_by_key(&m.directive, |v| v.directive) { + if let Ok(found) = self.vardefs + .binary_search_by_key(&m.directive, |v| v.directive) { let mut first = found; while first > 0 && self.vardefs[first - 1].directive == m.directive { first -= 1; @@ -147,50 +148,55 @@ impl<'a> Recorder for Explainer<'a> { } fn matched_check(&mut self, regex: &str, matched: MatchRange) { - self.matches.push(Match { - directive: self.directive, - is_match: true, - is_not: false, - regex: regex.to_owned(), - range: matched, - }); + self.matches + .push(Match { + directive: self.directive, + is_match: true, + is_not: false, + regex: regex.to_owned(), + range: matched, + }); } fn matched_not(&mut self, regex: &str, matched: MatchRange) { - self.matches.push(Match { - directive: self.directive, - is_match: true, - is_not: true, - regex: regex.to_owned(), - range: matched, - }); + self.matches + .push(Match { + directive: self.directive, + is_match: true, + is_not: true, + regex: regex.to_owned(), + range: matched, + }); } fn missed_check(&mut self, regex: &str, searched: MatchRange) { - self.matches.push(Match { - directive: self.directive, - is_match: false, - is_not: false, - regex: regex.to_owned(), - range: searched, - }); + self.matches + .push(Match { + directive: self.directive, + is_match: false, + is_not: false, + regex: regex.to_owned(), + range: searched, + }); } fn missed_not(&mut self, regex: &str, searched: MatchRange) { - self.matches.push(Match { - directive: self.directive, - is_match: false, - is_not: true, - regex: regex.to_owned(), - range: searched, - }); + self.matches + .push(Match { + directive: self.directive, + is_match: false, + is_not: true, + regex: regex.to_owned(), + range: searched, + }); } fn defined_var(&mut self, varname: &str, value: &str) { - self.vardefs.push(VarDef { - directive: self.directive, - varname: varname.to_owned(), - value: value.to_owned(), - }); + self.vardefs + .push(VarDef { + directive: self.directive, + varname: varname.to_owned(), + value: value.to_owned(), + }); } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index e8d88dde89..ca98bd98f2 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -107,7 +107,8 @@ impl<'a> Context<'a> { // Get the index of a recipe name if it exists. fn find_recipe_index(&self, recipe_name: &str) -> Option { if let Some(unique_isa) = self.unique_isa { - unique_isa.recipe_names() + unique_isa + .recipe_names() .iter() .position(|&name| name == recipe_name) .map(|idx| idx as u16) @@ -118,17 +119,14 @@ impl<'a> Context<'a> { // Allocate a new stack slot and add a mapping number -> StackSlot. fn add_ss(&mut self, number: u32, data: StackSlotData, loc: &Location) -> Result<()> { - self.map.def_ss(number, self.function.stack_slots.push(data), loc) + self.map + .def_ss(number, self.function.stack_slots.push(data), loc) } // Allocate a new signature and add a mapping number -> SigRef. fn add_sig(&mut self, number: u32, data: Signature, loc: &Location) -> Result<()> { - self.map.def_sig(number, - self.function - .dfg - .signatures - .push(data), - loc) + self.map + .def_sig(number, self.function.dfg.signatures.push(data), loc) } // Resolve a reference to a signature. @@ -141,12 +139,8 @@ impl<'a> Context<'a> { // Allocate a new external function and add a mapping number -> FuncRef. fn add_fn(&mut self, number: u32, data: ExtFuncData, loc: &Location) -> Result<()> { - self.map.def_fn(number, - self.function - .dfg - .ext_funcs - .push(data), - loc) + self.map + .def_fn(number, self.function.dfg.ext_funcs.push(data), loc) } // Resolve a reference to a function. @@ -159,7 +153,8 @@ impl<'a> Context<'a> { // Allocate a new jump table and add a mapping number -> JumpTable. fn add_jt(&mut self, number: u32, data: JumpTableData, loc: &Location) -> Result<()> { - self.map.def_jt(number, self.function.jump_tables.push(data), loc) + self.map + .def_jt(number, self.function.jump_tables.push(data), loc) } // Resolve a reference to a jump table. @@ -238,19 +233,34 @@ impl<'a> Context<'a> { } InstructionData::MultiAry { ref mut args, .. } => { - self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; + self.map + .rewrite_values(args.as_mut_slice(value_lists), loc)?; } - InstructionData::Jump { ref mut destination, ref mut args, .. } | - InstructionData::Branch { ref mut destination, ref mut args, .. } | - InstructionData::BranchIcmp { ref mut destination, ref mut args, .. } => { + InstructionData::Jump { + ref mut destination, + ref mut args, + .. + } | + InstructionData::Branch { + ref mut destination, + ref mut args, + .. + } | + InstructionData::BranchIcmp { + ref mut destination, + ref mut args, + .. + } => { self.map.rewrite_ebb(destination, loc)?; - self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; + self.map + .rewrite_values(args.as_mut_slice(value_lists), loc)?; } InstructionData::Call { ref mut args, .. } | InstructionData::IndirectCall { ref mut args, .. } => { - self.map.rewrite_values(args.as_mut_slice(value_lists), loc)?; + self.map + .rewrite_values(args.as_mut_slice(value_lists), loc)?; } } } @@ -307,10 +317,11 @@ impl<'a> Parser<'a> { Token::Comment(text) => { // Gather comments, associate them with `comment_entity`. if let Some(entity) = self.comment_entity { - self.comments.push(Comment { - entity: entity, - text: text, - }); + self.comments + .push(Comment { + entity: entity, + text: text, + }); } } _ => self.lookahead = Some(token), @@ -464,7 +475,8 @@ impl<'a> Parser<'a> { self.consume(); // Lexer just gives us raw text that looks like an integer. // Parse it as a u8 to check for overflow and other issues. - text.parse().map_err(|_| self.error("expected u8 decimal immediate")) + text.parse() + .map_err(|_| self.error("expected u8 decimal immediate")) } else { err!(self.loc, err_msg) } @@ -477,7 +489,8 @@ impl<'a> Parser<'a> { self.consume(); // Lexer just gives us raw text that looks like an integer. // Parse it as a u32 to check for overflow and other issues. - text.parse().map_err(|_| self.error("expected u32 decimal immediate")) + text.parse() + .map_err(|_| self.error("expected u32 decimal immediate")) } else { err!(self.loc, err_msg) } @@ -714,7 +727,9 @@ impl<'a> Parser<'a> { sig.return_types = self.parse_argument_list(unique_isa)?; } - if sig.argument_types.iter().all(|a| a.location.is_assigned()) { + if sig.argument_types + .iter() + .all(|a| a.location.is_assigned()) { sig.compute_argument_bytes(); } @@ -816,35 +831,23 @@ impl<'a> Parser<'a> { match self.token() { Some(Token::StackSlot(..)) => { self.gather_comments(ctx.function.stack_slots.next_key()); - self.parse_stack_slot_decl().and_then(|(num, dat)| { - ctx.add_ss(num, dat, &self.loc) - }) + self.parse_stack_slot_decl() + .and_then(|(num, dat)| ctx.add_ss(num, dat, &self.loc)) } Some(Token::SigRef(..)) => { - self.gather_comments(ctx.function - .dfg - .signatures - .next_key()); - self.parse_signature_decl(ctx.unique_isa).and_then(|(num, dat)| { - ctx.add_sig(num, - dat, - &self.loc) - }) + self.gather_comments(ctx.function.dfg.signatures.next_key()); + self.parse_signature_decl(ctx.unique_isa) + .and_then(|(num, dat)| ctx.add_sig(num, dat, &self.loc)) } Some(Token::FuncRef(..)) => { - self.gather_comments(ctx.function - .dfg - .ext_funcs - .next_key()); - self.parse_function_decl(ctx).and_then(|(num, dat)| { - ctx.add_fn(num, dat, &self.loc) - }) + self.gather_comments(ctx.function.dfg.ext_funcs.next_key()); + self.parse_function_decl(ctx) + .and_then(|(num, dat)| ctx.add_fn(num, dat, &self.loc)) } Some(Token::JumpTable(..)) => { self.gather_comments(ctx.function.jump_tables.next_key()); - self.parse_jump_table_decl().and_then(|(num, dat)| { - ctx.add_jt(num, dat, &self.loc) - }) + self.parse_jump_table_decl() + .and_then(|(num, dat)| ctx.add_jt(num, dat, &self.loc)) } // More to come.. _ => return Ok(()), @@ -861,7 +864,8 @@ impl<'a> Parser<'a> { self.match_identifier("stack_slot", "expected 'stack_slot'")?; // stack-slot-decl ::= StackSlot(ss) "=" "stack_slot" * Bytes {"," stack-slot-flag} - let bytes: i64 = self.match_imm64("expected byte-size in stack_slot decl")?.into(); + let bytes: i64 = self.match_imm64("expected byte-size in stack_slot decl")? + .into(); if bytes < 0 { return err!(self.loc, "negative stack slot size"); } @@ -903,11 +907,10 @@ impl<'a> Parser<'a> { let data = match self.token() { Some(Token::Identifier("function")) => { let (loc, name, sig) = self.parse_function_spec(ctx.unique_isa)?; - let sigref = ctx.function - .dfg - .signatures - .push(sig); - ctx.map.def_entity(sigref.into(), &loc).expect("duplicate SigRef entities created"); + let sigref = ctx.function.dfg.signatures.push(sig); + ctx.map + .def_entity(sigref.into(), &loc) + .expect("duplicate SigRef entities created"); ExtFuncData { name: name, signature: sigref, @@ -1224,7 +1227,9 @@ impl<'a> Parser<'a> { let inst = ctx.function.dfg.make_inst(inst_data); let num_results = ctx.function.dfg.make_inst_results(inst, ctrl_typevar); ctx.function.layout.append_inst(inst, ebb); - ctx.map.def_entity(inst.into(), &opcode_loc).expect("duplicate inst references created"); + ctx.map + .def_entity(inst.into(), &opcode_loc) + .expect("duplicate inst references created"); if let Some(encoding) = encoding { *ctx.function.encodings.ensure(inst) = encoding; @@ -1282,18 +1287,21 @@ impl<'a> Parser<'a> { inst_data: &InstructionData) -> Result { let constraints = opcode.constraints(); - let ctrl_type = match explicit_ctrl_type { - Some(t) => t, - None => { - if constraints.use_typevar_operand() { - // This is an opcode that supports type inference, AND there was no explicit - // type specified. Look up `ctrl_value` to see if it was defined already. - // TBD: If it is defined in another block, the type should have been specified - // explicitly. It is unfortunate that the correctness of IL depends on the - // layout of the blocks. - let ctrl_src_value = inst_data.typevar_operand(&ctx.function.dfg.value_lists) - .expect("Constraints <-> Format inconsistency"); - ctx.function.dfg.value_type(match ctx.map.get_value(ctrl_src_value) { + let ctrl_type = + match explicit_ctrl_type { + Some(t) => t, + None => { + if constraints.use_typevar_operand() { + // This is an opcode that supports type inference, AND there was no + // explicit type specified. Look up `ctrl_value` to see if it was defined + // already. + // TBD: If it is defined in another block, the type should have been + // specified explicitly. It is unfortunate that the correctness of IL + // depends on the layout of the blocks. + let ctrl_src_value = inst_data + .typevar_operand(&ctx.function.dfg.value_lists) + .expect("Constraints <-> Format inconsistency"); + ctx.function.dfg.value_type(match ctx.map.get_value(ctrl_src_value) { Some(v) => v, None => { if let Some(v) = ctx.aliases @@ -1308,19 +1316,19 @@ impl<'a> Parser<'a> { } } }) - } else if constraints.is_polymorphic() { - // This opcode does not support type inference, so the explicit type variable - // is required. - return err!(self.loc, - "type variable required for polymorphic opcode, e.g. '{}.{}'", - opcode, - constraints.ctrl_typeset().unwrap().example()); - } else { - // This is a non-polymorphic opcode. No typevar needed. - VOID + } else if constraints.is_polymorphic() { + // This opcode does not support type inference, so the explicit type + // variable is required. + return err!(self.loc, + "type variable required for polymorphic opcode, e.g. '{}.{}'", + opcode, + constraints.ctrl_typeset().unwrap().example()); + } else { + // This is a non-polymorphic opcode. No typevar needed. + VOID + } } - } - }; + }; // Verify that `ctrl_type` is valid for the controlling type variable. We don't want to // attempt deriving types from an incorrect basis. @@ -1629,7 +1637,8 @@ impl<'a> Parser<'a> { InstructionFormat::BranchTable => { let arg = self.match_value("expected SSA value operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; - let table = self.match_jt().and_then(|num| ctx.get_jt(num, &self.loc))?; + let table = self.match_jt() + .and_then(|num| ctx.get_jt(num, &self.loc))?; InstructionData::BranchTable { opcode: opcode, ty: VOID, @@ -1679,7 +1688,8 @@ mod tests { assert_eq!(v4.to_string(), "v0"); let vx3 = details.map.lookup_str("vx3").unwrap(); assert_eq!(vx3.to_string(), "vx0"); - let aliased_to = func.dfg.resolve_aliases(Value::table_with_number(0).unwrap()); + let aliased_to = func.dfg + .resolve_aliases(Value::table_with_number(0).unwrap()); assert_eq!(aliased_to.to_string(), "v0"); } @@ -1696,11 +1706,20 @@ mod tests { "(i8 uext inreg, f32, f64) -> i32 sext, f64"); // `void` is not recognized as a type by the lexer. It should not appear in files. - assert_eq!(Parser::new("() -> void").parse_signature(None).unwrap_err().to_string(), + assert_eq!(Parser::new("() -> void") + .parse_signature(None) + .unwrap_err() + .to_string(), "1: expected argument type"); - assert_eq!(Parser::new("i8 -> i8").parse_signature(None).unwrap_err().to_string(), + assert_eq!(Parser::new("i8 -> i8") + .parse_signature(None) + .unwrap_err() + .to_string(), "1: expected function signature: ( args... )"); - assert_eq!(Parser::new("(i8 -> i8").parse_signature(None).unwrap_err().to_string(), + assert_eq!(Parser::new("(i8 -> i8") + .parse_signature(None) + .unwrap_err() + .to_string(), "1: expected ')' after function arguments"); } diff --git a/lib/reader/src/testcommand.rs b/lib/reader/src/testcommand.rs index 6d9dd9969e..a147487c77 100644 --- a/lib/reader/src/testcommand.rs +++ b/lib/reader/src/testcommand.rs @@ -40,7 +40,10 @@ impl<'a> TestCommand<'a> { let cmd = parts.next().unwrap_or(""); TestCommand { command: cmd, - options: parts.filter(|s| !s.is_empty()).map(TestOption::new).collect(), + options: parts + .filter(|s| !s.is_empty()) + .map(TestOption::new) + .collect(), } } } From 72cb6459a908775246b472d95857c94585367e48 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 5 Apr 2017 09:12:12 -0700 Subject: [PATCH 0660/3084] Add a CodeOffset type and CodeSink::offset() method. We need to keep track of code offsets in order to compute accurate branch displacements. --- cranelift/src/filetest/binemit.rs | 29 ++++++++++++++++++++++++----- lib/cretonne/src/binemit/mod.rs | 9 +++++++++ 2 files changed, 33 insertions(+), 5 deletions(-) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 3c9fc333e6..50949811ec 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -8,6 +8,7 @@ use std::fmt::Write; use cretonne::binemit; use cretonne::ir; use cretonne::ir::entities::AnyEntity; +use cretonne::isa::TargetIsa; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result}; use utils::match_directive; @@ -26,24 +27,46 @@ pub fn subtest(parsed: &TestCommand) -> Result> { // Code sink that generates text. struct TextSink { rnames: &'static [&'static str], + offset: binemit::CodeOffset, text: String, } +impl TextSink { + /// Create a new empty TextSink. + pub fn new(isa: &TargetIsa) -> TextSink { + TextSink { + rnames: isa.reloc_names(), + offset: 0, + text: String::new(), + } + } +} + + + impl binemit::CodeSink for TextSink { + fn offset(&self) -> binemit::CodeOffset { + self.offset + } + fn put1(&mut self, x: u8) { write!(self.text, "{:02x} ", x).unwrap(); + self.offset += 1; } fn put2(&mut self, x: u16) { write!(self.text, "{:04x} ", x).unwrap(); + self.offset += 2; } fn put4(&mut self, x: u32) { write!(self.text, "{:08x} ", x).unwrap(); + self.offset += 4; } fn put8(&mut self, x: u64) { write!(self.text, "{:016x} ", x).unwrap(); + self.offset += 8; } fn reloc_ebb(&mut self, reloc: binemit::Reloc, ebb: ir::Ebb) { @@ -77,11 +100,7 @@ impl SubTest for TestBinEmit { // TODO: Run a verifier pass over the code first to detect any bad encodings or missing/bad // value locations. The current error reporting is just crashing... let mut func = func.into_owned(); - - let mut sink = TextSink { - rnames: isa.reloc_names(), - text: String::new(), - }; + let mut sink = TextSink::new(isa); for comment in &context.details.comments { if let Some(want) = match_directive(comment.text, "bin:") { diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index ccad476f44..9721586033 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -5,6 +5,12 @@ use ir::{Ebb, FuncRef, JumpTable, Function, Inst}; +/// Offset in bytes from the beginning of the function. +/// +/// Cretonne can be used as a cross compiler, so we don't want to use a type like `usize` which +/// depends on the *host* platform, not the *target* platform. +pub type CodeOffset = u32; + /// Relocation kinds depend on the current ISA. pub struct Reloc(pub u16); @@ -13,6 +19,9 @@ pub struct Reloc(pub u16); /// A `CodeSink` will receive all of the machine code for a function. It also accepts relocations /// which are locations in the code section that need to be fixed up when linking. pub trait CodeSink { + /// Get the current position. + fn offset(&self) -> CodeOffset; + /// Add 1 byte to the code section. fn put1(&mut self, u8); From f47c62ba8c493985eee99f7321190c17eb6bc7cf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 5 Apr 2017 10:07:19 -0700 Subject: [PATCH 0661/3084] Move encoding-related information into an EncInfo struct. The tables returned by recipe_names() and recipe_constraints() are now collected into an EncInfo struct that is available from TargetIsa::encoding_info(). This is equivalent to the register bank tables available fro TargetIsa::register_info(). This cleans of the TargetIsa interface and makes it easier to add encoding-related information. --- lib/cretonne/meta/gen_encoding.py | 9 ++++++-- lib/cretonne/src/isa/arm32/enc_tables.rs | 3 ++- lib/cretonne/src/isa/arm32/mod.rs | 14 +++++-------- lib/cretonne/src/isa/arm64/enc_tables.rs | 3 ++- lib/cretonne/src/isa/arm64/mod.rs | 14 +++++-------- lib/cretonne/src/isa/encoding.rs | 26 ++++++++++++++++++++++++ lib/cretonne/src/isa/intel/enc_tables.rs | 3 ++- lib/cretonne/src/isa/intel/mod.rs | 14 +++++-------- lib/cretonne/src/isa/mod.rs | 25 ++++------------------- lib/cretonne/src/isa/riscv/enc_tables.rs | 8 +++++--- lib/cretonne/src/isa/riscv/mod.rs | 16 ++++++--------- lib/cretonne/src/regalloc/coloring.rs | 11 ++++++---- lib/cretonne/src/regalloc/liveness.rs | 20 ++++++++---------- lib/cretonne/src/write.rs | 2 +- lib/reader/src/parser.rs | 3 ++- 15 files changed, 88 insertions(+), 83 deletions(-) diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index e4faa6410c..21b121487f 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -450,7 +450,7 @@ def emit_recipe_names(isa, fmt): This is used for pretty-printing encodings. """ with fmt.indented( - 'pub static RECIPE_NAMES: [&\'static str; {}] = [' + 'static RECIPE_NAMES: [&\'static str; {}] = [' .format(len(isa.all_recipes)), '];'): for r in isa.all_recipes: fmt.line('"{}",'.format(r.name)) @@ -465,7 +465,7 @@ def emit_recipe_constraints(isa, fmt): properly encoded. """ with fmt.indented( - 'pub static RECIPE_CONSTRAINTS: [RecipeConstraints; {}] = [' + 'static RECIPE_CONSTRAINTS: [RecipeConstraints; {}] = [' .format(len(isa.all_recipes)), '];'): for r in isa.all_recipes: fmt.comment(r.name) @@ -536,6 +536,11 @@ def gen_isa(isa, fmt): emit_recipe_names(isa, fmt) emit_recipe_constraints(isa, fmt) + # Finally, tie it all together in an `EncInfo`. + with fmt.indented('pub static INFO: EncInfo = EncInfo {', '};'): + fmt.line('constraints: &RECIPE_CONSTRAINTS,') + fmt.line('names: &RECIPE_NAMES,') + def generate(isas, out_dir): # type: (Sequence[TargetISA], str) -> None diff --git a/lib/cretonne/src/isa/arm32/enc_tables.rs b/lib/cretonne/src/isa/arm32/enc_tables.rs index c464d92f23..67cde87ed4 100644 --- a/lib/cretonne/src/isa/arm32/enc_tables.rs +++ b/lib/cretonne/src/isa/arm32/enc_tables.rs @@ -2,7 +2,8 @@ use ir::InstructionData; use ir::types; -use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::EncInfo; use isa::constraints::*; +use isa::enc_tables::{Level1Entry, Level2Entry}; include!(concat!(env!("OUT_DIR"), "/encoding-arm32.rs")); diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index df4f4c2570..c9c7f2af5c 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -9,7 +9,7 @@ use binemit::CodeSink; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, Encoding, Legalize, RecipeConstraints}; +use isa::{TargetIsa, RegInfo, EncInfo, Encoding, Legalize}; use ir; #[allow(dead_code)] @@ -55,6 +55,10 @@ impl TargetIsa for Isa { registers::INFO.clone() } + fn encoding_info(&self) -> EncInfo { + enc_tables::INFO.clone() + } + fn encode(&self, dfg: &ir::DataFlowGraph, inst: &ir::InstructionData) @@ -72,14 +76,6 @@ impl TargetIsa for Isa { }) } - fn recipe_names(&self) -> &'static [&'static str] { - &enc_tables::RECIPE_NAMES[..] - } - - fn recipe_constraints(&self) -> &'static [RecipeConstraints] { - &enc_tables::RECIPE_CONSTRAINTS - } - fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } diff --git a/lib/cretonne/src/isa/arm64/enc_tables.rs b/lib/cretonne/src/isa/arm64/enc_tables.rs index 064a00f2e2..d092c0eb8f 100644 --- a/lib/cretonne/src/isa/arm64/enc_tables.rs +++ b/lib/cretonne/src/isa/arm64/enc_tables.rs @@ -2,7 +2,8 @@ use ir::InstructionData; use ir::types; -use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::EncInfo; use isa::constraints::*; +use isa::enc_tables::{Level1Entry, Level2Entry}; include!(concat!(env!("OUT_DIR"), "/encoding-arm64.rs")); diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 2c11b450bb..a8cfe834b9 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -9,7 +9,7 @@ use binemit::CodeSink; use super::super::settings as shared_settings; use isa::enc_tables::{lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, Encoding, Legalize, RecipeConstraints}; +use isa::{TargetIsa, RegInfo, EncInfo, Encoding, Legalize}; use ir; #[allow(dead_code)] @@ -48,6 +48,10 @@ impl TargetIsa for Isa { registers::INFO.clone() } + fn encoding_info(&self) -> EncInfo { + enc_tables::INFO.clone() + } + fn encode(&self, dfg: &ir::DataFlowGraph, inst: &ir::InstructionData) @@ -65,14 +69,6 @@ impl TargetIsa for Isa { }) } - fn recipe_names(&self) -> &'static [&'static str] { - &enc_tables::RECIPE_NAMES[..] - } - - fn recipe_constraints(&self) -> &'static [RecipeConstraints] { - &enc_tables::RECIPE_CONSTRAINTS - } - fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } diff --git a/lib/cretonne/src/isa/encoding.rs b/lib/cretonne/src/isa/encoding.rs index 4c8e33a764..88040925ad 100644 --- a/lib/cretonne/src/isa/encoding.rs +++ b/lib/cretonne/src/isa/encoding.rs @@ -1,6 +1,7 @@ //! The `Encoding` struct. use std::fmt; +use isa::constraints::RecipeConstraints; /// Bits needed to encode an instruction as binary machine code. /// @@ -76,3 +77,28 @@ impl fmt::Display for DisplayEncoding { } } } + +/// Information about all the encodings in this ISA. +#[derive(Clone)] +pub struct EncInfo { + /// Constraints on value operands per recipe. + pub constraints: &'static [RecipeConstraints], + + /// Names of encoding recipes. + pub names: &'static [&'static str], +} + +impl EncInfo { + /// Get the value operand constraints for `enc` if it is a legal encoding. + pub fn operand_constraints(&self, enc: Encoding) -> Option<&RecipeConstraints> { + self.constraints.get(enc.recipe()) + } + + /// Create an object that can display an ISA-dependent encoding properly. + pub fn display(&self, enc: Encoding) -> DisplayEncoding { + DisplayEncoding { + encoding: enc, + recipe_names: self.names, + } + } +} diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index a29394a3a9..8abc8cd9d9 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -2,7 +2,8 @@ use ir::InstructionData; use ir::types; -use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::EncInfo; use isa::constraints::*; +use isa::enc_tables::{Level1Entry, Level2Entry}; include!(concat!(env!("OUT_DIR"), "/encoding-intel.rs")); diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 46aa101ce0..b48a119632 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -9,7 +9,7 @@ use binemit::CodeSink; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, Encoding, Legalize, RecipeConstraints}; +use isa::{TargetIsa, RegInfo, EncInfo, Encoding, Legalize}; use ir; #[allow(dead_code)] @@ -55,6 +55,10 @@ impl TargetIsa for Isa { registers::INFO.clone() } + fn encoding_info(&self) -> EncInfo { + enc_tables::INFO.clone() + } + fn encode(&self, dfg: &ir::DataFlowGraph, inst: &ir::InstructionData) @@ -72,14 +76,6 @@ impl TargetIsa for Isa { }) } - fn recipe_names(&self) -> &'static [&'static str] { - &enc_tables::RECIPE_NAMES[..] - } - - fn recipe_constraints(&self) -> &'static [RecipeConstraints] { - &enc_tables::RECIPE_CONSTRAINTS - } - fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 4eddae7b14..4fc08906ae 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -40,9 +40,9 @@ //! The configured target ISA trait object is a `Box` which can be used for multiple //! concurrent function compilations. -pub use isa::encoding::Encoding; -pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex}; pub use isa::constraints::{RecipeConstraints, OperandConstraint, ConstraintKind}; +pub use isa::encoding::{Encoding, EncInfo}; +pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex}; use binemit::CodeSink; use settings; @@ -143,25 +143,8 @@ pub trait TargetIsa { /// This is also the main entry point for determining if an instruction is legal. fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Result; - /// Get a static array of names associated with encoding recipes in this ISA. Encoding recipes - /// are numbered starting from 0, corresponding to indexes into the name array. - /// - /// This is just used for printing and parsing encodings in the textual IL format. - fn recipe_names(&self) -> &'static [&'static str]; - - /// Get a static array of value operand constraints associated with encoding recipes in this - /// ISA. - /// - /// The constraints describe which registers can be used with an encoding recipe. - fn recipe_constraints(&self) -> &'static [RecipeConstraints]; - - /// Create an object that can display an ISA-dependent encoding properly. - fn display_enc(&self, enc: Encoding) -> encoding::DisplayEncoding { - encoding::DisplayEncoding { - encoding: enc, - recipe_names: self.recipe_names(), - } - } + /// Get a data structure describing the instruction encodings in this ISA. + fn encoding_info(&self) -> EncInfo; /// Legalize a function signature. /// diff --git a/lib/cretonne/src/isa/riscv/enc_tables.rs b/lib/cretonne/src/isa/riscv/enc_tables.rs index 7940fce004..87c62bd63c 100644 --- a/lib/cretonne/src/isa/riscv/enc_tables.rs +++ b/lib/cretonne/src/isa/riscv/enc_tables.rs @@ -1,11 +1,12 @@ //! Encoding tables for RISC-V. use ir::condcodes::IntCC; -use ir::{Opcode, InstructionData}; use ir::types; -use predicates; -use isa::enc_tables::{Level1Entry, Level2Entry}; +use ir::{Opcode, InstructionData}; +use isa::EncInfo; use isa::constraints::*; +use isa::enc_tables::{Level1Entry, Level2Entry}; +use predicates; use super::registers::*; // Include the generated encoding tables: @@ -13,4 +14,5 @@ use super::registers::*; // - `LEVEL1_RV64` // - `LEVEL2` // - `ENCLIST` +// - `INFO` include!(concat!(env!("OUT_DIR"), "/encoding-riscv.rs")); diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 348e1af32f..7658db392e 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -10,7 +10,7 @@ use super::super::settings as shared_settings; use binemit::CodeSink; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, Encoding, Legalize, RecipeConstraints}; +use isa::{TargetIsa, RegInfo, EncInfo, Encoding, Legalize}; use ir::{Function, Inst, InstructionData, DataFlowGraph, Signature}; #[allow(dead_code)] @@ -56,6 +56,10 @@ impl TargetIsa for Isa { registers::INFO.clone() } + fn encoding_info(&self) -> EncInfo { + enc_tables::INFO.clone() + } + fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Result { lookup_enclist(inst.ctrl_typevar(dfg), inst.opcode(), @@ -70,14 +74,6 @@ impl TargetIsa for Isa { }) } - fn recipe_names(&self) -> &'static [&'static str] { - &enc_tables::RECIPE_NAMES[..] - } - - fn recipe_constraints(&self) -> &'static [RecipeConstraints] { - &enc_tables::RECIPE_CONSTRAINTS - } - fn legalize_signature(&self, sig: &mut Signature) { // We can pass in `self.isa_flags` too, if we need it. abi::legalize_signature(sig, &self.shared_flags) @@ -100,7 +96,7 @@ mod tests { use ir::{types, immediates}; fn encstr(isa: &isa::TargetIsa, enc: isa::Encoding) -> String { - isa.display_enc(enc).to_string() + isa.encoding_info().display(enc).to_string() } #[test] diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index b8c08bd278..b9e5fe7148 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -38,7 +38,7 @@ use entity_map::EntityMap; use dominator_tree::DominatorTree; use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph}; -use isa::{TargetIsa, RegInfo, Encoding, RecipeConstraints, ConstraintKind}; +use isa::{TargetIsa, RegInfo, Encoding, EncInfo, ConstraintKind}; use regalloc::affinity::Affinity; use regalloc::allocatable_set::AllocatableSet; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; @@ -69,7 +69,7 @@ struct Context<'a> { // Cached ISA information. // We save it here to avoid frequent virtual function calls on the `TargetIsa` trait object. reginfo: RegInfo, - recipe_constraints: &'a [RecipeConstraints], + encinfo: EncInfo, // References to contextual data structures we need. domtree: &'a DominatorTree, @@ -98,7 +98,7 @@ impl Coloring { tracker: &mut LiveValueTracker) { let mut ctx = Context { reginfo: isa.register_info(), - recipe_constraints: isa.recipe_constraints(), + encinfo: isa.encoding_info(), domtree: domtree, liveness: liveness, // TODO: Ask the target ISA about reserved registers etc. @@ -259,7 +259,10 @@ impl<'a> Context<'a> { let (kills, defs) = tracker.process_inst(inst, dfg, self.liveness); // Get the operand constraints for `inst` that we are trying to satisfy. - let constraints = self.recipe_constraints[encoding.recipe()].clone(); + let constraints = self.encinfo + .operand_constraints(encoding) + .expect("Missing instruction encoding") + .clone(); // Get rid of the killed values. for lv in kills { diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 3344c5b841..90b817a825 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -178,7 +178,7 @@ use flowgraph::ControlFlowGraph; use ir::dfg::ValueDef; use ir::{Function, Value, Inst, Ebb}; -use isa::{TargetIsa, RecipeConstraints}; +use isa::{TargetIsa, EncInfo}; use regalloc::affinity::Affinity; use regalloc::liverange::LiveRange; use sparse_map::SparseMap; @@ -191,7 +191,7 @@ type LiveRangeSet = SparseMap; fn get_or_create<'a>(lrset: &'a mut LiveRangeSet, value: Value, func: &Function, - recipe_constraints: &[RecipeConstraints]) + enc_info: &EncInfo) -> &'a mut LiveRange { // It would be better to use `get_mut()` here, but that leads to borrow checker fighting // which can probably only be resolved by non-lexical lifetimes. @@ -205,8 +205,8 @@ fn get_or_create<'a>(lrset: &'a mut LiveRangeSet, def = inst.into(); // Initialize the affinity from the defining instruction's result constraints. // Don't do this for call return values which are always tied to a single register. - affinity = recipe_constraints - .get(func.encodings[inst].recipe()) + affinity = enc_info + .operand_constraints(func.encodings[inst]) .and_then(|rc| rc.outs.get(rnum)) .map(Affinity::new) .unwrap_or_default(); @@ -296,7 +296,7 @@ impl Liveness { self.ranges.clear(); // Get ISA data structures used for computing live range affinities. - let recipe_constraints = isa.recipe_constraints(); + let enc_info = isa.encoding_info(); let reg_info = isa.register_info(); // The liveness computation needs to visit all uses, but the order doesn't matter. @@ -309,22 +309,20 @@ impl Liveness { // TODO: When we implement DCE, we can use the absence of a live range to indicate // an unused value. for def in func.dfg.inst_results(inst) { - get_or_create(&mut self.ranges, def, func, recipe_constraints); + get_or_create(&mut self.ranges, def, func, &enc_info); } - // The instruction encoding is used to compute affinities. - let recipe = func.encodings[inst].recipe(); // Iterator of constraints, one per value operand. // TODO: Should we fail here if the instruction doesn't have a valid encoding? - let mut operand_constraints = recipe_constraints - .get(recipe) + let mut operand_constraints = enc_info + .operand_constraints(func.encodings[inst]) .map(|c| c.ins) .unwrap_or(&[]) .iter(); for &arg in func.dfg.inst_args(inst) { // Get the live range, create it as a dead range if necessary. - let lr = get_or_create(&mut self.ranges, arg, func, recipe_constraints); + let lr = get_or_create(&mut self.ranges, arg, func, &enc_info); // Extend the live range to reach this use. extend_to_use(lr, ebb, inst, &mut self.worklist, func, cfg); diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index e2cd0fe3f5..7dfe743ead 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -184,7 +184,7 @@ fn write_instruction(w: &mut Write, if let Some(enc) = func.encodings.get(inst).cloned() { let mut s = String::with_capacity(16); if let Some(isa) = isa { - write!(s, "[{}", isa.display_enc(enc))?; + write!(s, "[{}", isa.encoding_info().display(enc))?; // Write value locations, if we have them. if !func.locations.is_empty() { let regs = isa.register_info(); diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index ca98bd98f2..1843c78776 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -108,7 +108,8 @@ impl<'a> Context<'a> { fn find_recipe_index(&self, recipe_name: &str) -> Option { if let Some(unique_isa) = self.unique_isa { unique_isa - .recipe_names() + .encoding_info() + .names .iter() .position(|&name| name == recipe_name) .map(|idx| idx as u16) From 598c81c12ef00d0d52a943457e7668e048e31af3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 5 Apr 2017 11:04:17 -0700 Subject: [PATCH 0662/3084] Add encoding size information to EncInfo. Two new pieces of information are available for all encoding recipes: - The size in bytes of an encoded instruction, and - The range of a branch encoded with the recipe, if any. In the meta language, EncRecipe takes two new constructor arguments. The size is required for all encodings and branch_range is required for all recipes used to encode branches. --- lib/cretonne/meta/cdsl/isa.py | 32 ++++++++++++- lib/cretonne/meta/gen_encoding.py | 23 +++++++++ lib/cretonne/meta/isa/riscv/recipes.py | 24 ++++++---- lib/cretonne/src/isa/arm32/enc_tables.rs | 1 + lib/cretonne/src/isa/arm64/enc_tables.rs | 1 + lib/cretonne/src/isa/constraints.rs | 60 +++++++++++++++++++++++- lib/cretonne/src/isa/encoding.rs | 38 ++++++++++++++- lib/cretonne/src/isa/intel/enc_tables.rs | 1 + lib/cretonne/src/isa/mod.rs | 2 +- lib/cretonne/src/isa/riscv/enc_tables.rs | 1 + 10 files changed, 169 insertions(+), 14 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 9db0faeb61..1ead583c5a 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -19,6 +19,7 @@ try: # Instruction specification for encodings. Allows for predicated # instructions. InstSpec = Union[MaybeBoundInst, Apply] + BranchRange = Sequence[int] except ImportError: pass @@ -164,17 +165,38 @@ class EncRecipe(object): - An integer indicating that this result is tied to a value operand, so they must use the same register. + The `branch_range` argument must be provided for recipes that can encode + branch instructions. It is an `(origin, bits)` tuple describing the exact + range that can be encoded in a branch instruction. + :param name: Short mnemonic name for this recipe. :param format: All encoded instructions must have this :py:class:`InstructionFormat`. + :param size: Number of bytes in the binary encoded instruction. :param: ins Tuple of register constraints for value operands. :param: outs Tuple of register constraints for results. + :param: branch_range `(origin, bits)` range for branches. + :param: instp Instruction predicate. + :param: isap ISA predicate. """ - def __init__(self, name, format, ins, outs, instp=None, isap=None): - # type: (str, InstructionFormat, ConstraintSeq, ConstraintSeq, PredNode, PredNode) -> None # noqa + def __init__( + self, + name, # type: str + format, # type: InstructionFormat + size, # type: int + ins, # type: ConstraintSeq + outs, # type: ConstraintSeq + branch_range=None, # type: BranchRange + instp=None, # type: PredNode + isap=None # type: PredNode + ): + # type: (...) -> None self.name = name self.format = format + assert size >= 0 + self.size = size + self.branch_range = branch_range self.instp = instp self.isap = isap if instp: @@ -249,6 +271,12 @@ class Encoding(object): assert self.inst.format == recipe.format, ( "Format {} must match recipe: {}".format( self.inst.format, recipe.format)) + + if self.inst.is_branch: + assert recipe.branch_range, ( + 'Recipe {} for {} must have a branch_range' + .format(recipe, self.inst.name)) + self.recipe = recipe self.encbits = encbits # Combine recipe predicates with the manually specified ones. diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 21b121487f..a68710f540 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -498,6 +498,27 @@ def emit_operand_constraints(seq, field, fmt): 'Unsupported constraint {}'.format(cons)) +def emit_recipe_sizing(isa, fmt): + # type: (TargetISA, srcgen.Formatter) -> None + """ + Emit a table of encoding recipe code size information. + """ + with fmt.indented( + 'static RECIPE_SIZING: [RecipeSizing; {}] = [' + .format(len(isa.all_recipes)), '];'): + for r in isa.all_recipes: + fmt.comment(r.name) + with fmt.indented('RecipeSizing {', '},'): + fmt.format('bytes: {},', r.size) + if r.branch_range: + fmt.format( + 'branch_range: ' + 'Some(BranchRange {{ origin: {}, bits: {} }}),', + *r.branch_range) + else: + fmt.line('branch_range: None,') + + def gen_isa(isa, fmt): # type: (TargetISA, srcgen.Formatter) -> None # First assign numbers to relevant instruction predicates and generate the @@ -535,10 +556,12 @@ def gen_isa(isa, fmt): emit_recipe_names(isa, fmt) emit_recipe_constraints(isa, fmt) + emit_recipe_sizing(isa, fmt) # Finally, tie it all together in an `EncInfo`. with fmt.indented('pub static INFO: EncInfo = EncInfo {', '};'): fmt.line('constraints: &RECIPE_CONSTRAINTS,') + fmt.line('sizing: &RECIPE_SIZING,') fmt.line('names: &RECIPE_NAMES,') diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 2be2192002..63ca9b78ba 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -85,37 +85,43 @@ def LUI(): # R-type 32-bit instructions: These are mostly binary arithmetic instructions. # The encbits are `opcode[6:2] | (funct3 << 5) | (funct7 << 8) -R = EncRecipe('R', Binary, ins=(GPR, GPR), outs=GPR) +R = EncRecipe('R', Binary, size=4, ins=(GPR, GPR), outs=GPR) # R-type with an immediate shift amount instead of rs2. -Rshamt = EncRecipe('Rshamt', BinaryImm, ins=GPR, outs=GPR) +Rshamt = EncRecipe('Rshamt', BinaryImm, size=4, ins=GPR, outs=GPR) # R-type encoding of an integer comparison. -Ricmp = EncRecipe('Ricmp', IntCompare, ins=(GPR, GPR), outs=GPR) +Ricmp = EncRecipe('Ricmp', IntCompare, size=4, ins=(GPR, GPR), outs=GPR) I = EncRecipe( - 'I', BinaryImm, ins=GPR, outs=GPR, + 'I', BinaryImm, size=4, ins=GPR, outs=GPR, instp=IsSignedInt(BinaryImm.imm, 12)) # I-type encoding of an integer comparison. Iicmp = EncRecipe( - 'Iicmp', IntCompareImm, ins=GPR, outs=GPR, + 'Iicmp', IntCompareImm, size=4, ins=GPR, outs=GPR, instp=IsSignedInt(IntCompareImm.imm, 12)) # I-type encoding for `jalr` as a return instruction. We won't use the # immediate offset. # The variable return values are not encoded. -Iret = EncRecipe('Iret', MultiAry, ins=GPR, outs=()) +Iret = EncRecipe('Iret', MultiAry, size=4, ins=GPR, outs=()) # U-type instructions have a 20-bit immediate that targets bits 12-31. U = EncRecipe( - 'U', UnaryImm, ins=(), outs=GPR, + 'U', UnaryImm, size=4, ins=(), outs=GPR, instp=IsSignedInt(UnaryImm.imm, 32, 12)) # SB-type branch instructions. # TODO: These instructions have a +/- 4 KB branch range. How to encode that # constraint? -SB = EncRecipe('SB', BranchIcmp, ins=(GPR, GPR), outs=()) +SB = EncRecipe( + 'SB', BranchIcmp, size=4, + ins=(GPR, GPR), outs=(), + branch_range=(0, 13)) # SB-type branch instruction with rs2 fixed to zero. -SBzero = EncRecipe('SBzero', Branch, ins=(GPR), outs=()) +SBzero = EncRecipe( + 'SBzero', Branch, size=4, + ins=(GPR), outs=(), + branch_range=(0, 13)) diff --git a/lib/cretonne/src/isa/arm32/enc_tables.rs b/lib/cretonne/src/isa/arm32/enc_tables.rs index 67cde87ed4..9fdfbcda95 100644 --- a/lib/cretonne/src/isa/arm32/enc_tables.rs +++ b/lib/cretonne/src/isa/arm32/enc_tables.rs @@ -5,5 +5,6 @@ use ir::types; use isa::EncInfo; use isa::constraints::*; use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::encoding::RecipeSizing; include!(concat!(env!("OUT_DIR"), "/encoding-arm32.rs")); diff --git a/lib/cretonne/src/isa/arm64/enc_tables.rs b/lib/cretonne/src/isa/arm64/enc_tables.rs index d092c0eb8f..cdfc255748 100644 --- a/lib/cretonne/src/isa/arm64/enc_tables.rs +++ b/lib/cretonne/src/isa/arm64/enc_tables.rs @@ -5,5 +5,6 @@ use ir::types; use isa::EncInfo; use isa::constraints::*; use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::encoding::RecipeSizing; include!(concat!(env!("OUT_DIR"), "/encoding-arm64.rs")); diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index 2b8700ec4a..23cd923f93 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -7,6 +7,7 @@ //! It is the register allocator's job to make sure that the register constraints on value operands //! are satisfied. +use binemit::CodeOffset; use isa::{RegClass, RegUnit}; /// Register constraint for a single value operand or instruction result. @@ -48,7 +49,7 @@ pub enum ConstraintKind { Stack, } -/// Constraints for an encoding recipe. +/// Value operand constraints for an encoding recipe. #[derive(Clone)] pub struct RecipeConstraints { /// Constraints for the instruction's fixed value operands. @@ -66,3 +67,60 @@ pub struct RecipeConstraints { /// constraints must be derived from the calling convention ABI. pub outs: &'static [OperandConstraint], } + +/// Constraints on the range of a branch instruction. +/// +/// A branch instruction usually encodes its destination as a signed n-bit offset from an origin. +/// The origin depends on the ISA and the specific instruction: +/// +/// - RISC-V and ARM Aarch64 use the address of the branch instruction, `origin = 0`. +/// - Intel uses the address of the instruction following the branch, `origin = 2` for a 2-byte +/// branch instruction. +/// - ARM's A32 encoding uses the address of the branch instruction + 8 bytes, `origin = 8`. +#[derive(Clone, Copy)] +pub struct BranchRange { + /// Offset in bytes from the address of the branch instruction to the origin used for computing + /// the branch displacement. This is the destination of a branch that encodes a 0 displacement. + pub origin: u8, + + /// Number of bits in the signed byte displacement encoded in the instruction. This does not + /// account for branches that can only target aligned addresses. + pub bits: u8, +} + +impl BranchRange { + /// Determine if this branch range can represent the range from `branch` to `dest`, where + /// `branch` is the code offset of the branch instruction itself and `dest` is the code offset + /// of the destination EBB header. + /// + /// This method does not detect if the range is larger than 2 GB. + pub fn contains(self, branch: CodeOffset, dest: CodeOffset) -> bool { + let d = dest.wrapping_sub(branch + self.origin as CodeOffset) as i32; + let s = 32 - self.bits; + d == d << s >> s + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn branch_range() { + // ARM T1 branch. + let t1 = BranchRange { origin: 4, bits: 9 }; + assert!(t1.contains(0, 0)); + assert!(t1.contains(0, 2)); + assert!(t1.contains(2, 0)); + assert!(t1.contains(1000, 1000)); + + // Forward limit. + assert!(t1.contains(1000, 1258)); + assert!(!t1.contains(1000, 1260)); + + // Backward limit + assert!(t1.contains(1000, 748)); + assert!(!t1.contains(1000, 746)); + + } +} diff --git a/lib/cretonne/src/isa/encoding.rs b/lib/cretonne/src/isa/encoding.rs index 88040925ad..c5151978aa 100644 --- a/lib/cretonne/src/isa/encoding.rs +++ b/lib/cretonne/src/isa/encoding.rs @@ -1,7 +1,8 @@ //! The `Encoding` struct. +use binemit::CodeOffset; +use isa::constraints::{RecipeConstraints, BranchRange}; use std::fmt; -use isa::constraints::RecipeConstraints; /// Bits needed to encode an instruction as binary machine code. /// @@ -78,12 +79,28 @@ impl fmt::Display for DisplayEncoding { } } +/// Code size information for an encoding recipe. +/// +/// All encoding recipes correspond to an exact instruction size. +pub struct RecipeSizing { + /// Size in bytes of instructions encoded with this recipe. + pub bytes: u8, + + /// Allowed branch range in this recipe, if any. + /// + /// All encoding recipes for branches have exact branch range information. + pub branch_range: Option, +} + /// Information about all the encodings in this ISA. #[derive(Clone)] pub struct EncInfo { /// Constraints on value operands per recipe. pub constraints: &'static [RecipeConstraints], + /// Code size information per recipe. + pub sizing: &'static [RecipeSizing], + /// Names of encoding recipes. pub names: &'static [&'static str], } @@ -101,4 +118,23 @@ impl EncInfo { recipe_names: self.names, } } + + /// Get the exact size in bytes of instructions encoded with `enc`. + /// + /// Returns 0 for illegal encodings. + pub fn bytes(&self, enc: Encoding) -> CodeOffset { + self.sizing + .get(enc.recipe()) + .map(|s| s.bytes as CodeOffset) + .unwrap_or(0) + } + + /// Get the branch range that is supported by `enc`, if any. + /// + /// This will never return `None` for a legal branch encoding. + pub fn branch_range(&self, enc: Encoding) -> Option { + self.sizing + .get(enc.recipe()) + .and_then(|s| s.branch_range) + } } diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 8abc8cd9d9..1ee1fbd4b3 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -5,5 +5,6 @@ use ir::types; use isa::EncInfo; use isa::constraints::*; use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::encoding::RecipeSizing; include!(concat!(env!("OUT_DIR"), "/encoding-intel.rs")); diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 4fc08906ae..469142ab56 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -40,7 +40,7 @@ //! The configured target ISA trait object is a `Box` which can be used for multiple //! concurrent function compilations. -pub use isa::constraints::{RecipeConstraints, OperandConstraint, ConstraintKind}; +pub use isa::constraints::{RecipeConstraints, OperandConstraint, ConstraintKind, BranchRange}; pub use isa::encoding::{Encoding, EncInfo}; pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex}; diff --git a/lib/cretonne/src/isa/riscv/enc_tables.rs b/lib/cretonne/src/isa/riscv/enc_tables.rs index 87c62bd63c..8ced0c681f 100644 --- a/lib/cretonne/src/isa/riscv/enc_tables.rs +++ b/lib/cretonne/src/isa/riscv/enc_tables.rs @@ -6,6 +6,7 @@ use ir::{Opcode, InstructionData}; use isa::EncInfo; use isa::constraints::*; use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::encoding::RecipeSizing; use predicates; use super::registers::*; From 81251c300561f9eb02814a19159ecb636ecdf108 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 5 Apr 2017 14:54:17 -0700 Subject: [PATCH 0663/3084] Add a branch relaxation pass for #72. Compute exact EBB header offsets and check that branches are in range. Not implemented yet: Relax branches that are not in range. Invoke the relax_branches() pass from the 'test binemit' file tests so they can verify the proper encoding of branch instructions too. --- cranelift/src/filetest/binemit.rs | 116 +++++++++++++++++-------- lib/cretonne/src/binemit/mod.rs | 4 + lib/cretonne/src/binemit/relaxation.rs | 110 +++++++++++++++++++++++ lib/cretonne/src/ir/function.rs | 17 +++- lib/cretonne/src/ir/instructions.rs | 13 +++ lib/cretonne/src/lib.rs | 6 +- 6 files changed, 224 insertions(+), 42 deletions(-) create mode 100644 lib/cretonne/src/binemit/relaxation.rs diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 50949811ec..eee96e6ee5 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -4,6 +4,7 @@ //! functions and compares the results to the expected output. use std::borrow::Cow; +use std::collections::HashMap; use std::fmt::Write; use cretonne::binemit; use cretonne::ir; @@ -97,54 +98,99 @@ impl SubTest for TestBinEmit { fn run(&self, func: Cow, context: &Context) -> Result<()> { let isa = context.isa.expect("binemit needs an ISA"); + let encinfo = isa.encoding_info(); // TODO: Run a verifier pass over the code first to detect any bad encodings or missing/bad // value locations. The current error reporting is just crashing... let mut func = func.into_owned(); - let mut sink = TextSink::new(isa); - for comment in &context.details.comments { - if let Some(want) = match_directive(comment.text, "bin:") { - let inst = match comment.entity { - AnyEntity::Inst(inst) => inst, - _ => { - return Err(format!("annotation on non-inst {}: {}", - comment.entity, - comment.text)) - } - }; - - // Compute an encoding for `inst` if one wasn't provided. + // Give an encoding to any instruction that doesn't already have one. + for ebb in func.layout.ebbs() { + for inst in func.layout.ebb_insts(ebb) { if !func.encodings .get(inst) .map(|e| e.is_legal()) .unwrap_or(false) { - match isa.encode(&func.dfg, &func.dfg[inst]) { - Ok(enc) => *func.encodings.ensure(inst) = enc, - Err(_) => { - return Err(format!("{} can't be encoded: {}", - inst, - func.dfg.display_inst(inst))) - } + if let Ok(enc) = isa.encode(&func.dfg, &func.dfg[inst]) { + *func.encodings.ensure(inst) = enc; } } - - sink.text.clear(); - isa.emit_inst(&func, inst, &mut sink); - let have = sink.text.trim(); - if have != want { - return Err(format!("Bad machine code for {}: {}\nWant: {}\nGot: {}", - inst, - func.dfg.display_inst(inst), - want, - have)); - } } } - if sink.text.is_empty() { - Err("No bin: directives found".to_string()) - } else { - Ok(()) + // Relax branches and compute EBB offsets based on the encodings. + binemit::relax_branches(&mut func, isa); + + // Collect all of the 'bin:' directives on instructions. + let mut bins = HashMap::new(); + for comment in &context.details.comments { + if let Some(want) = match_directive(comment.text, "bin:") { + match comment.entity { + AnyEntity::Inst(inst) => { + if let Some(prev) = bins.insert(inst, want) { + return Err(format!("multiple 'bin:' directives on {}: '{}' and '{}'", + func.dfg.display_inst(inst), + prev, + want)); + } + } + _ => { + return Err(format!("'bin:' directive on non-inst {}: {}", + comment.entity, + comment.text)) + } + } + } } + if bins.is_empty() { + return Err("No 'bin:' directives found".to_string()); + } + + // Now emit all instructions. + let mut sink = TextSink::new(isa); + for ebb in func.layout.ebbs() { + // Correct header offsets should have been computed by `relax_branches()`. + assert_eq!(sink.offset, + func.offsets[ebb], + "Inconsistent {} header offset", + ebb); + for inst in func.layout.ebb_insts(ebb) { + sink.text.clear(); + let enc = func.encodings.get(inst).cloned().unwrap_or_default(); + + // Send legal encodings into the emitter. + if enc.is_legal() { + let before = sink.offset; + isa.emit_inst(&func, inst, &mut sink); + let emitted = sink.offset - before; + // Verify the encoding recipe sizes against the ISAs emit_inst implementation. + assert_eq!(emitted, + encinfo.bytes(enc), + "Inconsistent size for [{}] {}", + encinfo.display(enc), + func.dfg.display_inst(inst)); + } + + // Check against bin: directives. + if let Some(want) = bins.remove(&inst) { + if !enc.is_legal() { + return Err(format!("{} can't be encoded: {}", + inst, + func.dfg.display_inst(inst))); + } + sink.text.clear(); + isa.emit_inst(&func, inst, &mut sink); + let have = sink.text.trim(); + if have != want { + return Err(format!("Bad machine code for {}: {}\nWant: {}\nGot: {}", + inst, + func.dfg.display_inst(inst), + want, + have)); + } + } + } + } + + Ok(()) } } diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 9721586033..33cce8d2b0 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -3,6 +3,10 @@ //! The `binemit` module contains code for translating Cretonne's intermediate representation into //! binary machine code. +mod relaxation; + +pub use self::relaxation::relax_branches; + use ir::{Ebb, FuncRef, JumpTable, Function, Inst}; /// Offset in bytes from the beginning of the function. diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs new file mode 100644 index 0000000000..ef3bfdb257 --- /dev/null +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -0,0 +1,110 @@ +//! Branch relaxation and offset computation. +//! +//! # EBB header offsets +//! +//! Before we can generate binary machine code for branch instructions, we need to know the final +//! offsets of all the EBB headers in the function. This information is encoded in the +//! `func.offsets` table. +//! +//! # Branch relaxation +//! +//! Branch relaxation is the process of ensuring that all branches in the function have enough +//! range to encode their destination. It is common to have multiple branch encodings in an ISA. +//! For example, Intel branches can have either an 8-bit or a 32-bit displacement. +//! +//! On RISC architectures, it can happen that conditional branches have a shorter range than +//! unconditional branches: +//! +//! ```cton +//! brz v1, ebb17 +//! ``` +//! +//! can be transformed into: +//! +//! ```cton +//! brnz v1, ebb23 +//! jump ebb17 +//! ebb23: +//! ``` + +use binemit::CodeOffset; +use entity_map::EntityMap; +use ir::{Function, DataFlowGraph, Cursor, Inst}; +use isa::{TargetIsa, EncInfo, Encoding}; + +/// Relax branches and compute the final layout of EBB headers in `func`. +/// +/// Fill in the `func.offsets` table so the function is ready for binary emission. +pub fn relax_branches(func: &mut Function, isa: &TargetIsa) { + let encinfo = isa.encoding_info(); + + // Clear all offsets so we can recognize EBBs that haven't been visited yet. + func.offsets.clear(); + func.offsets.resize(func.dfg.num_ebbs()); + + // The relaxation algorithm iterates to convergence. + let mut go_again = true; + while go_again { + go_again = false; + + // Visit all instructions in layout order + let mut offset = 0; + let mut pos = Cursor::new(&mut func.layout); + while let Some(ebb) = pos.next_ebb() { + // Record the offset for `ebb` and make sure we iterate until offsets are stable. + if func.offsets[ebb] != offset { + assert!(func.offsets[ebb] < offset, + "Code shrinking during relaxation"); + func.offsets[ebb] = offset; + go_again = true; + } + + while let Some(inst) = pos.next_inst() { + let enc = func.encodings.get(inst).cloned().unwrap_or_default(); + let size = encinfo.bytes(enc); + + // See if this might be a branch that is out of range. + if let Some(range) = encinfo.branch_range(enc) { + if let Some(dest) = func.dfg[inst].branch_destination() { + let dest_offset = func.offsets[dest]; + if !range.contains(offset, dest_offset) { + // This is an out-of-range branch. + // Relax it unless the destination offset has not been computed yet. + if dest_offset != 0 || Some(dest) == pos.layout.entry_block() { + offset += relax_branch(&mut func.dfg, + &mut func.encodings, + &encinfo, + &mut pos, + offset, + dest_offset); + continue; + } + } + } + } + + offset += size; + } + } + } +} + +/// Relax the branch instruction at `pos` so it can cover the range `offset - dest_offset`. +/// +/// Return the size of the replacement instructions up to and including the location where `pos` is +/// left. +fn relax_branch(dfg: &mut DataFlowGraph, + encodings: &mut EntityMap, + encinfo: &EncInfo, + pos: &mut Cursor, + offset: CodeOffset, + dest_offset: CodeOffset) + -> CodeOffset { + let inst = pos.current_inst().unwrap(); + dbg!("Relaxing [{}] {} for {:#x}-{:#x} range", + encinfo.display(encodings[inst]), + dfg.display_inst(inst), + offset, + dest_offset); + unimplemented!(); +} diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 1abf9c4b35..177ca0b5f1 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -3,11 +3,12 @@ //! The `Function` struct defined in this module owns all of its extended basic blocks and //! instructions. -use std::fmt::{self, Display, Debug, Formatter}; -use ir::{FunctionName, Signature, Value, Inst, StackSlot, StackSlotData, JumpTable, JumpTableData, - ValueLoc, DataFlowGraph, Layout}; -use isa::{TargetIsa, Encoding}; +use binemit::CodeOffset; use entity_map::{EntityMap, PrimaryEntityData}; +use ir::{FunctionName, Signature, Value, Inst, Ebb, StackSlot, StackSlotData, JumpTable, + JumpTableData, ValueLoc, DataFlowGraph, Layout}; +use isa::{TargetIsa, Encoding}; +use std::fmt::{self, Display, Debug, Formatter}; use write::write_function; /// A function. @@ -40,6 +41,13 @@ pub struct Function { /// Location assigned to every value. pub locations: EntityMap, + + /// Code offsets of the EBB headers. + /// + /// This information is only transiently available after the `binemit::relax_branches` function + /// computes it, and it can easily be recomputed by calling that function. It is not included + /// in the textual IL format. + pub offsets: EntityMap, } impl PrimaryEntityData for StackSlotData {} @@ -57,6 +65,7 @@ impl Function { layout: Layout::new(), encodings: EntityMap::new(), locations: EntityMap::new(), + offsets: EntityMap::new(), } } diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 1c292f473a..9816ccbc91 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -324,6 +324,19 @@ impl InstructionData { } } + /// Get the single destination of this branch instruction, if it is a single destination + /// branch or jump. + /// + /// Multi-destination branches like `br_table` return `None`. + pub fn branch_destination(&self) -> Option { + match self { + &InstructionData::Jump { destination, .. } => Some(destination), + &InstructionData::Branch { destination, .. } => Some(destination), + &InstructionData::BranchIcmp { destination, .. } => Some(destination), + _ => None, + } + } + /// Return information about a call instruction. /// /// Any instruction that can call another function reveals its call signature here. diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index b4c163b14d..1c983c74e6 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -10,6 +10,9 @@ pub use write::write_function; /// Version number of the cretonne crate. pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); +#[macro_use] +pub mod dbg; + pub mod binemit; pub mod flowgraph; pub mod dominator_tree; @@ -22,9 +25,6 @@ pub mod settings; pub mod sparse_map; pub mod verifier; -#[macro_use] -pub mod dbg; - mod abi; mod constant_hash; mod context; From 8a3f538441f60b5e0e0a6aa24dec19c734867dc0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 6 Apr 2017 11:13:13 -0700 Subject: [PATCH 0664/3084] Fix a bug in the binemit file test. Only emit each instruction once, or the offset computations go all wrong. --- cranelift/filetests/isa/riscv/binary32.cton | 2 ++ cranelift/src/filetest/binemit.rs | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 93b1a9cc2d..8055289c65 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -75,9 +75,11 @@ ebb0: [-,%x7] v140 = iconst.i32 0x12345000 ; bin: 123453b7 [-,%x16] v141 = iconst.i32 0xffffffff_fedcb000 ; bin: fedcb837 + jump ebb1 ; Control Transfer Instructions +ebb1: ; beq br_icmp eq, v1, v2, ebb0 ; bin: Branch(ebb0) 01550063 ; bne diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index eee96e6ee5..f4ec6262c6 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -177,8 +177,6 @@ impl SubTest for TestBinEmit { inst, func.dfg.display_inst(inst))); } - sink.text.clear(); - isa.emit_inst(&func, inst, &mut sink); let have = sink.text.trim(); if have != want { return Err(format!("Bad machine code for {}: {}\nWant: {}\nGot: {}", From 856a67ca3a6e98795f4248e1b4c4f450f96c532c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 6 Apr 2017 11:18:38 -0700 Subject: [PATCH 0665/3084] Use EBB offsets for encoding RISC-V branches. Stop emitting EBB relocations. Use the offsets computed by relax_branches() to encode the correct displacements immediately. --- cranelift/filetests/isa/riscv/binary32.cton | 45 ++++++++++++++------- lib/cretonne/src/isa/riscv/binemit.rs | 24 ++++++++--- 2 files changed, 49 insertions(+), 20 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 8055289c65..bdcbde9dd2 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -80,23 +80,40 @@ ebb0: ; Control Transfer Instructions ebb1: - ; beq - br_icmp eq, v1, v2, ebb0 ; bin: Branch(ebb0) 01550063 - ; bne - br_icmp ne, v1, v2, ebb0 ; bin: Branch(ebb0) 01551063 - ; blt - br_icmp slt, v1, v2, ebb0 ; bin: Branch(ebb0) 01554063 - ; bge - br_icmp sge, v1, v2, ebb0 ; bin: Branch(ebb0) 01555063 - ; bltu - br_icmp ult, v1, v2, ebb0 ; bin: Branch(ebb0) 01556063 - ; bgeu - br_icmp uge, v1, v2, ebb0 ; bin: Branch(ebb0) 01557063 + ; beq 0x000 + br_icmp eq, v1, v2, ebb1 ; bin: 01550063 + ; bne 0xffc + br_icmp ne, v1, v2, ebb1 ; bin: ff551ee3 + ; blt 0xff8 + br_icmp slt, v1, v2, ebb1 ; bin: ff554ce3 + ; bge 0xff4 + br_icmp sge, v1, v2, ebb1 ; bin: ff555ae3 + ; bltu 0xff0 + br_icmp ult, v1, v2, ebb1 ; bin: ff5568e3 + ; bgeu 0xfec + br_icmp uge, v1, v2, ebb1 ; bin: ff5576e3 + ; Forward branches. + ; beq 0x018 + br_icmp eq, v2, v1, ebb2 ; bin: 00aa8c63 + ; bne 0x014 + br_icmp ne, v2, v1, ebb2 ; bin: 00aa9a63 + ; blt 0x010 + br_icmp slt, v2, v1, ebb2 ; bin: 00aac863 + ; bge 0x00c + br_icmp sge, v2, v1, ebb2 ; bin: 00aad663 + ; bltu 0x008 + br_icmp ult, v2, v1, ebb2 ; bin: 00aae463 + ; bgeu 0x004 + br_icmp uge, v2, v1, ebb2 ; bin: 00aaf263 + + jump ebb2 + +ebb2: ; beq x, %x0 - brz v1, ebb0 ; bin: Branch(ebb0) 00050063 + brz v1, ebb2 ; bin: 00050063 ; bne x, %x0 - brnz v1, ebb0 ; bin: Branch(ebb0) 00051063 + brnz v1, ebb2 ; bin: fe051ee3 return } diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 213d8a132e..a392d76a07 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -3,10 +3,12 @@ use binemit::{CodeSink, Reloc, bad_encoding}; use ir::{Function, Inst, InstructionData}; use isa::RegUnit; +use predicates::is_signed_int; include!(concat!(env!("OUT_DIR"), "/binemit-riscv.rs")); /// RISC-V relocation kinds. +#[allow(dead_code)] pub enum RelocKind { /// A conditional (SB-type) branch to an EBB. Branch, @@ -213,17 +215,17 @@ fn recipe_u(func: &Function, inst: Inst, sink: &mut CS) { /// imm rs2 rs1 funct3 imm opcode /// 25 20 15 12 7 0 /// -/// The imm bits are not encoded by this function. They encode the relative distance to the -/// destination block, handled by a relocation. -/// /// Encoding bits: `opcode[6:2] | (funct3 << 5)` -fn put_sb(bits: u16, rs1: RegUnit, rs2: RegUnit, sink: &mut CS) { +fn put_sb(bits: u16, imm: i64, rs1: RegUnit, rs2: RegUnit, sink: &mut CS) { let bits = bits as u32; let opcode5 = bits & 0x1f; let funct3 = (bits >> 5) & 0x7; let rs1 = rs1 as u32 & 0x1f; let rs2 = rs2 as u32 & 0x1f; + assert!(is_signed_int(imm, 13, 1), "SB out of range {:#x}", imm); + let imm = imm as u32; + // 0-6: opcode let mut i = 0x3; i |= opcode5 << 2; @@ -231,6 +233,12 @@ fn put_sb(bits: u16, rs1: RegUnit, rs2: RegUnit, sink: &m i |= rs1 << 15; i |= rs2 << 20; + // The displacement is completely hashed up. + i |= ((imm >> 11) & 0x1) << 7; + i |= ((imm >> 1) & 0xf) << 8; + i |= ((imm >> 5) & 0x3f) << 25; + i |= ((imm >> 12) & 0x1) << 31; + sink.put4(i); } @@ -240,9 +248,11 @@ fn recipe_sb(func: &Function, inst: Inst, sink: &mut CS) ref args, .. } = func.dfg[inst] { + let dest = func.offsets[destination] as i64; + let disp = dest - sink.offset() as i64; let args = &args.as_slice(&func.dfg.value_lists)[0..2]; - sink.reloc_ebb(RelocKind::Branch.into(), destination); put_sb(func.encodings[inst].bits(), + disp, func.locations[args[0]].unwrap_reg(), func.locations[args[1]].unwrap_reg(), sink); @@ -257,9 +267,11 @@ fn recipe_sbzero(func: &Function, inst: Inst, sink: &mut ref args, .. } = func.dfg[inst] { + let dest = func.offsets[destination] as i64; + let disp = dest - sink.offset() as i64; let args = &args.as_slice(&func.dfg.value_lists)[0..1]; - sink.reloc_ebb(RelocKind::Branch.into(), destination); put_sb(func.encodings[inst].bits(), + disp, func.locations[args[0]].unwrap_reg(), 0, sink); From 5025bb67f4b68c518d0a3cf02f5cf026e330ff83 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 6 Apr 2017 13:58:48 -0700 Subject: [PATCH 0666/3084] Add an iterators module with extra Iterator methods. Start with an adjacent_pairs() iterator adapter. --- lib/cretonne/src/iterators.rs | 90 +++++++++++++++++++++++++++++++++++ lib/cretonne/src/lib.rs | 3 +- 2 files changed, 92 insertions(+), 1 deletion(-) create mode 100644 lib/cretonne/src/iterators.rs diff --git a/lib/cretonne/src/iterators.rs b/lib/cretonne/src/iterators.rs new file mode 100644 index 0000000000..3bdc3cc1cb --- /dev/null +++ b/lib/cretonne/src/iterators.rs @@ -0,0 +1,90 @@ +//! Iterator utilities. + +/// Extra methods for iterators. +pub trait IteratorExtras: Iterator { + /// Create an iterator that produces adjacent pairs of elements from the iterator. + fn adjacent_pairs(mut self) -> AdjacentPairs + where Self: Sized, + Self::Item: Clone + { + let elem = self.next(); + AdjacentPairs { + iter: self, + elem: elem, + } + } +} + +impl IteratorExtras for T where T: Iterator {} + +/// Adjacent pairs iterator returned by `adjacent_pairs()`. +/// +/// This wraps another iterator and produces a sequence of adjacent pairs of elements. +pub struct AdjacentPairs + where I: Iterator, + I::Item: Clone +{ + iter: I, + elem: Option, +} + +impl Iterator for AdjacentPairs + where I: Iterator, + I::Item: Clone +{ + type Item = (I::Item, I::Item); + + fn next(&mut self) -> Option { + self.elem + .take() + .and_then(|e| { + self.elem = self.iter.next(); + self.elem.clone().map(|n| (e, n)) + }) + } +} + +#[cfg(test)] +mod tests { + #[test] + fn adjpairs() { + use super::IteratorExtras; + + assert_eq!([1, 2, 3, 4] + .iter() + .cloned() + .adjacent_pairs() + .collect::>(), + vec![(1, 2), (2, 3), (3, 4)]); + assert_eq!([2, 3, 4] + .iter() + .cloned() + .adjacent_pairs() + .collect::>(), + vec![(2, 3), (3, 4)]); + assert_eq!([2, 3, 4] + .iter() + .cloned() + .adjacent_pairs() + .collect::>(), + vec![(2, 3), (3, 4)]); + assert_eq!([3, 4] + .iter() + .cloned() + .adjacent_pairs() + .collect::>(), + vec![(3, 4)]); + assert_eq!([4] + .iter() + .cloned() + .adjacent_pairs() + .collect::>(), + vec![]); + assert_eq!([] + .iter() + .cloned() + .adjacent_pairs() + .collect::>(), + vec![]); + } +} diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 1c983c74e6..e44899f8c2 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -14,10 +14,10 @@ pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); pub mod dbg; pub mod binemit; -pub mod flowgraph; pub mod dominator_tree; pub mod entity_list; pub mod entity_map; +pub mod flowgraph; pub mod ir; pub mod isa; pub mod regalloc; @@ -28,6 +28,7 @@ pub mod verifier; mod abi; mod constant_hash; mod context; +mod iterators; mod legalizer; mod packed_option; mod partition_slice; From fa4f151b9bdaa1cac0ac8b3fdee6c0b1d08a1252 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 6 Apr 2017 14:22:32 -0700 Subject: [PATCH 0667/3084] Add a fallthrough instruction. Change jumps to fallthroughs in the branch relaxation pass before computing the EBB offsets. --- cranelift/docs/langref.rst | 1 + cranelift/filetests/isa/riscv/binary32.cton | 2 +- lib/cretonne/meta/base/instructions.py | 13 ++++++++ lib/cretonne/src/binemit/relaxation.rs | 37 ++++++++++++++++++++- 4 files changed, 51 insertions(+), 2 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 47a0dae37c..398141fb5b 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -319,6 +319,7 @@ condition is satisfied, otherwise execution continues at the following instruction in the EBB. .. autoinst:: jump +.. autoinst:: fallthrough .. autoinst:: brz .. autoinst:: brnz .. autoinst:: br_icmp diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index bdcbde9dd2..4c9f9439b9 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -107,7 +107,7 @@ ebb1: ; bgeu 0x004 br_icmp uge, v2, v1, ebb2 ; bin: 00aaf263 - jump ebb2 + fallthrough ebb2 ebb2: ; beq x, %x0 diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 4776f576c1..d9eda7a220 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -49,6 +49,19 @@ jump = Instruction( """, ins=(EBB, args), is_branch=True, is_terminator=True) +fallthrough = Instruction( + 'fallthrough', r""" + Fall through to the next EBB. + + This is the same as :inst:`jump`, except the destination EBB must be + the next one in the layout. + + Jumps are turned into fall-through instructions by the branch + relaxation pass. There is no reason to use this instruction outside + that pass. + """, + ins=(EBB, args), is_branch=True, is_terminator=True) + brz = Instruction( 'brz', r""" Branch when zero. diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index ef3bfdb257..179540dad0 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -29,8 +29,9 @@ use binemit::CodeOffset; use entity_map::EntityMap; -use ir::{Function, DataFlowGraph, Cursor, Inst}; +use ir::{Function, DataFlowGraph, Cursor, Inst, InstructionData, Opcode}; use isa::{TargetIsa, EncInfo, Encoding}; +use iterators::IteratorExtras; /// Relax branches and compute the final layout of EBB headers in `func`. /// @@ -42,6 +43,9 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) { func.offsets.clear(); func.offsets.resize(func.dfg.num_ebbs()); + // Start by inserting fall through instructions. + fallthroughs(func); + // The relaxation algorithm iterates to convergence. let mut go_again = true; while go_again { @@ -89,6 +93,37 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) { } } +/// Convert `jump` instructions to `fallthrough` instructions where possible and verify that any +/// existing `fallthrough` instructions are correct. +fn fallthroughs(func: &mut Function) { + for (ebb, succ) in func.layout.ebbs().adjacent_pairs() { + let term = func.layout + .last_inst(ebb) + .expect("EBB has no terminator."); + if let InstructionData::Jump { + ref mut opcode, + destination, + .. + } = func.dfg[term] { + match *opcode { + Opcode::Fallthrough => { + // Somebody used a fall-through instruction before the branch relaxation pass. + // Make sure it is correct, i.e. the destination is the layout successor. + assert_eq!(destination, succ, "Illegal fall-through in {}", ebb) + } + Opcode::Jump => { + // If this is a jump to the successor EBB, change it to a fall-through. + if destination == succ { + *opcode = Opcode::Fallthrough; + func.encodings[term] = Default::default(); + } + } + _ => {} + } + } + } +} + /// Relax the branch instruction at `pos` so it can cover the range `offset - dest_offset`. /// /// Return the size of the replacement instructions up to and including the location where `pos` is From 896ac935b9abc1cc35e598347c529d8c2668dbfc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 6 Apr 2017 15:17:57 -0700 Subject: [PATCH 0668/3084] Add jump encodings to RISC-V. Fix a bug in gen_encoding.py when dealing with non-polymorphic instructions where the type variable is None in Python, VOID in Rust. --- cranelift/filetests/isa/riscv/binary32.cton | 18 ++++++---- lib/cretonne/meta/gen_encoding.py | 7 ++-- lib/cretonne/meta/isa/riscv/encodings.py | 8 +++-- lib/cretonne/meta/isa/riscv/recipes.py | 10 +++++- lib/cretonne/src/isa/riscv/binemit.rs | 39 +++++++++++++++++++++ 5 files changed, 70 insertions(+), 12 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 4c9f9439b9..d34fe34e21 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -75,7 +75,8 @@ ebb0: [-,%x7] v140 = iconst.i32 0x12345000 ; bin: 123453b7 [-,%x16] v141 = iconst.i32 0xffffffff_fedcb000 ; bin: fedcb837 - jump ebb1 + brz v1, ebb3 + fallthrough ebb1 ; Control Transfer Instructions @@ -110,10 +111,15 @@ ebb1: fallthrough ebb2 ebb2: - ; beq x, %x0 - brz v1, ebb2 ; bin: 00050063 - ; bne x, %x0 - brnz v1, ebb2 ; bin: fe051ee3 + ; jal %x0, 0x00000 + jump ebb2 ; bin: 0000006f - return +ebb3: + ; beq x, %x0 + brz v1, ebb3 ; bin: 00050063 + ; bne x, %x0 + brnz v1, ebb3 ; bin: fe051ee3 + + ; jal %x0, 0x1ffff4 + jump ebb2 ; bin: ff5ff06f } diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index a68710f540..706ad46780 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -300,7 +300,7 @@ class Level2Table(object): level2_doc[self.hash_table_offset].append( '{:06x}: {}, {} entries'.format( self.hash_table_offset, - self.ty.name, + self.ty, self.hash_table_len)) level2_hashtables.extend(hash_table) @@ -405,7 +405,7 @@ def emit_level1_hashtable(cpumode, level1, offt, fmt): """ def hash_func(level2): # type: (Level2Table) -> int - return level2.ty.number + return level2.ty.number if level2.ty is not None else 0 hash_table = compute_quadratic(level1.tables.values(), hash_func) with fmt.indented( @@ -415,11 +415,12 @@ def emit_level1_hashtable(cpumode, level1, offt, fmt): if level2: l2l = int(math.log(level2.hash_table_len, 2)) assert l2l > 0, "Hash table too small" + tyname = level2.ty.name if level2.ty is not None else 'void' fmt.line( 'Level1Entry ' + '{{ ty: types::{}, log2len: {}, offset: {:#08x} }},' .format( - level2.ty.name.upper(), + tyname.upper(), l2l, level2.hash_table_offset)) else: diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index 162ccbd3ea..21ff09e177 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -5,8 +5,8 @@ from __future__ import absolute_import from base import instructions as base from base.immediates import intcc from .defs import RV32, RV64 -from .recipes import OPIMM, OPIMM32, OP, OP32, LUI, BRANCH -from .recipes import JALR, R, Rshamt, Ricmp, I, Iicmp, Iret, U, SB, SBzero +from .recipes import OPIMM, OPIMM32, OP, OP32, LUI, BRANCH, JALR, JAL +from .recipes import R, Rshamt, Ricmp, I, Iicmp, Iret, U, UJ, SB, SBzero from .settings import use_m from cdsl.ast import Var @@ -81,6 +81,10 @@ RV64.enc(base.imul.i32, R, OP32(0b000, 0b0000001), isap=use_m) # Control flow. +# Unconditional branches. +RV32.enc(base.jump, UJ, JAL()) +RV64.enc(base.jump, UJ, JAL()) + # Conditional branches. for cond, f3 in [ (intcc.eq, 0b000), diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 63ca9b78ba..e30922613f 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -12,7 +12,7 @@ from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt from base.formats import Binary, BinaryImm, MultiAry, IntCompare, IntCompareImm -from base.formats import UnaryImm, BranchIcmp, Branch +from base.formats import UnaryImm, BranchIcmp, Branch, Jump from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit @@ -47,6 +47,11 @@ def JALR(funct3=0): return 0b11001 | (funct3 << 5) +def JAL(): + # type: () -> int + return 0b11011 + + def OPIMM(funct3, funct7=0): # type: (int, int) -> int assert funct3 <= 0b111 @@ -112,6 +117,9 @@ U = EncRecipe( 'U', UnaryImm, size=4, ins=(), outs=GPR, instp=IsSignedInt(UnaryImm.imm, 32, 12)) +# UJ-type unconditional branch instructions. +UJ = EncRecipe('UJ', Jump, size=4, ins=(), outs=(), branch_range=(0, 21)) + # SB-type branch instructions. # TODO: These instructions have a +/- 4 KB branch range. How to encode that # constraint? diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index a392d76a07..e4cc5c05c6 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -279,3 +279,42 @@ fn recipe_sbzero(func: &Function, inst: Inst, sink: &mut panic!("Expected Branch format: {:?}", func.dfg[inst]); } } + +/// UJ-type jump instructions. +/// +/// 31 11 6 +/// imm rd opcode +/// 12 7 0 +/// +/// Encoding bits: `opcode[6:2]` +fn put_uj(bits: u16, imm: i64, rd: RegUnit, sink: &mut CS) { + let bits = bits as u32; + let opcode5 = bits & 0x1f; + let rd = rd as u32 & 0x1f; + + assert!(is_signed_int(imm, 21, 1), "UJ out of range {:#x}", imm); + let imm = imm as u32; + + // 0-6: opcode + let mut i = 0x3; + i |= opcode5 << 2; + i |= rd << 7; + + // The displacement is completely hashed up. + i |= imm & 0xff000; + i |= ((imm >> 11) & 0x1) << 20; + i |= ((imm >> 1) & 0x3ff) << 21; + i |= ((imm >> 20) & 0x1) << 31; + + sink.put4(i); +} + +fn recipe_uj(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Jump { destination, .. } = func.dfg[inst] { + let dest = func.offsets[destination] as i64; + let disp = dest - sink.offset() as i64; + put_uj(func.encodings[inst].bits(), disp, 0, sink); + } else { + panic!("Expected Jump format: {:?}", func.dfg[inst]); + } +} From 51dea9f78457c04fbeb3a2a86e25392dc8a2a442 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 10 Apr 2017 10:28:37 -0700 Subject: [PATCH 0669/3084] Syntax tweak: Omit comma after an initial enum immediate. This affects the comparison instructions which now read "icmp ult a, b". This mimics LLVM's style and makes it simpler to add instruction flags in the future, such as "load v1" -> "load aligned v1". These enumerated operands and flags feel like opcode modifiers rather than value operands, so displaying them differently makes sense. Value and numeric operands are still comma separated. --- cranelift/filetests/cfg/loop.cton | 2 +- cranelift/filetests/isa/riscv/binary32.cton | 40 +++++++++---------- cranelift/filetests/isa/riscv/expand-i32.cton | 2 +- .../filetests/isa/riscv/legalize-i64.cton | 2 +- cranelift/filetests/parser/tiny.cton | 28 ++++++------- lib/cretonne/src/write.rs | 8 ++-- lib/reader/src/parser.rs | 4 -- 7 files changed, 41 insertions(+), 45 deletions(-) diff --git a/cranelift/filetests/cfg/loop.cton b/cranelift/filetests/cfg/loop.cton index 9a1ca6d7e3..3f7892a9d9 100644 --- a/cranelift/filetests/cfg/loop.cton +++ b/cranelift/filetests/cfg/loop.cton @@ -21,7 +21,7 @@ ebb1(v5: i32): v10 = f32const 0.0 v11 = fadd v9, v10 v12 = iadd_imm v5, 1 - v13 = icmp ult, v12, v2 + v13 = icmp ult v12, v2 brnz v13, ebb1(v12) ; unordered: ebb1:inst12 -> ebb1 v14 = f64const 0.0 v15 = f64const 0.0 diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index d34fe34e21..89771d8165 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -33,11 +33,11 @@ ebb0: [-,%x7] v34 = sshr v1, v2 ; bin: 415553b3 [-,%x16] v35 = sshr v2, v1 ; bin: 40aad833 ; slt - [-,%x7] v42 = icmp slt, v1, v2 ; bin: 015523b3 - [-,%x16] v43 = icmp slt, v2, v1 ; bin: 00aaa833 + [-,%x7] v42 = icmp slt v1, v2 ; bin: 015523b3 + [-,%x16] v43 = icmp slt v2, v1 ; bin: 00aaa833 ; sltu - [-,%x7] v44 = icmp ult, v1, v2 ; bin: 015533b3 - [-,%x16] v45 = icmp ult, v2, v1 ; bin: 00aab833 + [-,%x7] v44 = icmp ult v1, v2 ; bin: 015533b3 + [-,%x16] v45 = icmp ult v2, v1 ; bin: 00aab833 ; Integer Register-Immediate Instructions @@ -65,11 +65,11 @@ ebb0: [-,%x16] v125 = sshr_imm v2, 8 ; bin: 408ad813 ; slti - [-,%x7] v130 = icmp_imm slt, v1, 1000 ; bin: 3e852393 - [-,%x16] v131 = icmp_imm slt, v2, -905 ; bin: c77aa813 + [-,%x7] v130 = icmp_imm slt v1, 1000 ; bin: 3e852393 + [-,%x16] v131 = icmp_imm slt v2, -905 ; bin: c77aa813 ; sltiu - [-,%x7] v132 = icmp_imm ult, v1, 1000 ; bin: 3e853393 - [-,%x16] v133 = icmp_imm ult, v2, -905 ; bin: c77ab813 + [-,%x7] v132 = icmp_imm ult v1, 1000 ; bin: 3e853393 + [-,%x16] v133 = icmp_imm ult v2, -905 ; bin: c77ab813 ; lui [-,%x7] v140 = iconst.i32 0x12345000 ; bin: 123453b7 @@ -82,31 +82,31 @@ ebb0: ebb1: ; beq 0x000 - br_icmp eq, v1, v2, ebb1 ; bin: 01550063 + br_icmp eq v1, v2, ebb1 ; bin: 01550063 ; bne 0xffc - br_icmp ne, v1, v2, ebb1 ; bin: ff551ee3 + br_icmp ne v1, v2, ebb1 ; bin: ff551ee3 ; blt 0xff8 - br_icmp slt, v1, v2, ebb1 ; bin: ff554ce3 + br_icmp slt v1, v2, ebb1 ; bin: ff554ce3 ; bge 0xff4 - br_icmp sge, v1, v2, ebb1 ; bin: ff555ae3 + br_icmp sge v1, v2, ebb1 ; bin: ff555ae3 ; bltu 0xff0 - br_icmp ult, v1, v2, ebb1 ; bin: ff5568e3 + br_icmp ult v1, v2, ebb1 ; bin: ff5568e3 ; bgeu 0xfec - br_icmp uge, v1, v2, ebb1 ; bin: ff5576e3 + br_icmp uge v1, v2, ebb1 ; bin: ff5576e3 ; Forward branches. ; beq 0x018 - br_icmp eq, v2, v1, ebb2 ; bin: 00aa8c63 + br_icmp eq v2, v1, ebb2 ; bin: 00aa8c63 ; bne 0x014 - br_icmp ne, v2, v1, ebb2 ; bin: 00aa9a63 + br_icmp ne v2, v1, ebb2 ; bin: 00aa9a63 ; blt 0x010 - br_icmp slt, v2, v1, ebb2 ; bin: 00aac863 + br_icmp slt v2, v1, ebb2 ; bin: 00aac863 ; bge 0x00c - br_icmp sge, v2, v1, ebb2 ; bin: 00aad663 + br_icmp sge v2, v1, ebb2 ; bin: 00aad663 ; bltu 0x008 - br_icmp ult, v2, v1, ebb2 ; bin: 00aae463 + br_icmp ult v2, v1, ebb2 ; bin: 00aae463 ; bgeu 0x004 - br_icmp uge, v2, v1, ebb2 ; bin: 00aaf263 + br_icmp uge v2, v1, ebb2 ; bin: 00aaf263 fallthrough ebb2 diff --git a/cranelift/filetests/isa/riscv/expand-i32.cton b/cranelift/filetests/isa/riscv/expand-i32.cton index 166b058eb2..0415c6026c 100644 --- a/cranelift/filetests/isa/riscv/expand-i32.cton +++ b/cranelift/filetests/isa/riscv/expand-i32.cton @@ -15,7 +15,7 @@ ebb0(v1: i32, v2: i32): return v3, v4 } ; check: $v3 = iadd $v1, $v2 -; check: $(cout=$V) = icmp ult, $v3, $v1 +; check: $(cout=$V) = icmp ult $v3, $v1 ; It's possible the legalizer will rewrite these value aliases in the future. ; check: $v4 -> $cout ; check: return $v3, $v4 diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton index e2d5a763a2..dbe26f824c 100644 --- a/cranelift/filetests/isa/riscv/legalize-i64.cton +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -54,7 +54,7 @@ ebb0(v1: i64, v2: i64): ; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32): ; check: [R#0c ; sameln: $(v3l=$V) = iadd $v1l, $v2l -; check: $(c=$V) = icmp ult, $v3l, $v1l +; check: $(c=$V) = icmp ult $v3l, $v1l ; check: [R#0c ; sameln: $(v3h1=$V) = iadd $v1h, $v2h ; check: $(c_int=$V) = bint.i32 $c diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index d5790c0e89..21d3a59815 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -52,33 +52,33 @@ ebb0: ; Integer condition codes. function icmp(i32, i32) { ebb0(vx0: i32, vx1: i32): - v0 = icmp eq, vx0, vx1 - v1 = icmp ult, vx0, vx1 - v2 = icmp_imm sge, vx0, -12 + v0 = icmp eq vx0, vx1 + v1 = icmp ult vx0, vx1 + v2 = icmp_imm sge vx0, -12 v3 = irsub_imm vx1, 45 - br_icmp eq, vx0, vx1, ebb0(vx1, vx0) + br_icmp eq vx0, vx1, ebb0(vx1, vx0) } ; sameln: function icmp(i32, i32) { ; nextln: ebb0(vx0: i32, vx1: i32): -; nextln: v0 = icmp eq, vx0, vx1 -; nextln: v1 = icmp ult, vx0, vx1 -; nextln: v2 = icmp_imm sge, vx0, -12 +; nextln: v0 = icmp eq vx0, vx1 +; nextln: v1 = icmp ult vx0, vx1 +; nextln: v2 = icmp_imm sge vx0, -12 ; nextln: v3 = irsub_imm vx1, 45 -; nextln: br_icmp eq, vx0, vx1, ebb0(vx1, vx0) +; nextln: br_icmp eq vx0, vx1, ebb0(vx1, vx0) ; nextln: } ; Floating condition codes. function fcmp(f32, f32) { ebb0(vx0: f32, vx1: f32): - v0 = fcmp eq, vx0, vx1 - v1 = fcmp uno, vx0, vx1 - v2 = fcmp lt, vx0, vx1 + v0 = fcmp eq vx0, vx1 + v1 = fcmp uno vx0, vx1 + v2 = fcmp lt vx0, vx1 } ; sameln: function fcmp(f32, f32) { ; nextln: ebb0(vx0: f32, vx1: f32): -; nextln: v0 = fcmp eq, vx0, vx1 -; nextln: v1 = fcmp uno, vx0, vx1 -; nextln: v2 = fcmp lt, vx0, vx1 +; nextln: v0 = fcmp eq vx0, vx1 +; nextln: v1 = fcmp uno vx0, vx1 +; nextln: v2 = fcmp lt vx0, vx1 ; nextln: } ; The bitcast instruction has two type variables: The controlling type variable diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 7dfe743ead..eaa5248408 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -259,9 +259,9 @@ pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result } InsertLane { lane, args, .. } => write!(w, " {}, {}, {}", args[0], lane, args[1]), ExtractLane { lane, arg, .. } => write!(w, " {}, {}", arg, lane), - IntCompare { cond, args, .. } => write!(w, " {}, {}, {}", cond, args[0], args[1]), - IntCompareImm { cond, arg, imm, .. } => write!(w, " {}, {}, {}", cond, arg, imm), - FloatCompare { cond, args, .. } => write!(w, " {}, {}, {}", cond, args[0], args[1]), + IntCompare { cond, args, .. } => write!(w, " {} {}, {}", cond, args[0], args[1]), + IntCompareImm { cond, arg, imm, .. } => write!(w, " {} {}, {}", cond, arg, imm), + FloatCompare { cond, args, .. } => write!(w, " {} {}, {}", cond, args[0], args[1]), Jump { destination, ref args, @@ -295,7 +295,7 @@ pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result .. } => { let args = args.as_slice(pool); - write!(w, " {}, {}, {}, {}", cond, args[0], args[1], destination)?; + write!(w, " {} {}, {}, {}", cond, args[0], args[1], destination)?; if args.len() > 2 { write!(w, "({})", DisplayValues(&args[2..]))?; } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 1843c78776..b9ee4acd58 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1526,7 +1526,6 @@ impl<'a> Parser<'a> { } InstructionFormat::BranchIcmp => { let cond = self.match_enum("expected intcc condition code")?; - self.match_token(Token::Comma, "expected ',' between operands")?; let lhs = self.match_value("expected SSA value first operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_value("expected SSA value second operand")?; @@ -1567,7 +1566,6 @@ impl<'a> Parser<'a> { } InstructionFormat::IntCompare => { let cond = self.match_enum("expected intcc condition code")?; - self.match_token(Token::Comma, "expected ',' between operands")?; let lhs = self.match_value("expected SSA value first operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_value("expected SSA value second operand")?; @@ -1580,7 +1578,6 @@ impl<'a> Parser<'a> { } InstructionFormat::IntCompareImm => { let cond = self.match_enum("expected intcc condition code")?; - self.match_token(Token::Comma, "expected ',' between operands")?; let lhs = self.match_value("expected SSA value first operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_imm64("expected immediate second operand")?; @@ -1594,7 +1591,6 @@ impl<'a> Parser<'a> { } InstructionFormat::FloatCompare => { let cond = self.match_enum("expected floatcc condition code")?; - self.match_token(Token::Comma, "expected ',' between operands")?; let lhs = self.match_value("expected SSA value first operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_value("expected SSA value second operand")?; From eccc2d0daea2e603b8ecccb2289d41cec7412501 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 10 Apr 2017 11:53:46 -0700 Subject: [PATCH 0670/3084] Add an Offset32 immediate operand kind. This will be used to represent an immediate 32-bit signed address offset for load/store instructions. --- cranelift/docs/langref.rst | 14 +- lib/cretonne/meta/base/immediates.py | 6 + lib/cretonne/src/ir/immediates.rs | 290 ++++++++++++++++++--------- 3 files changed, 208 insertions(+), 102 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 398141fb5b..24890c8ada 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -249,6 +249,13 @@ indicate the different kinds of immediate operands on an instruction. In the textual format, :type:`imm64` immediates appear as decimal or hexadecimal literals using the same syntax as C. +.. type:: offset32 + + A signed 32-bit immediate address offset. + + In the textual format, :type:`offset32` immediates always have an explicit + sign, and a 0 offset may beomitted. + .. type:: ieee32 A 32-bit immediate floating point number in the IEEE 754-2008 binary32 @@ -259,13 +266,6 @@ indicate the different kinds of immediate operands on an instruction. A 64-bit immediate floating point number in the IEEE 754-2008 binary64 interchange format. All bit patterns are allowed. -.. type:: immvector - - An immediate SIMD vector. This operand supplies all the bits of a SIMD - type, so it can have different sizes depending on the type produced. The - bits of the operand are interpreted as if the SIMD vector was loaded from - memory containing the immediate. - .. type:: intcc An integer condition code. See the :inst:`icmp` instruction for details. diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index ff2eb972a5..e11ed78e28 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -17,6 +17,12 @@ imm64 = ImmediateKind('imm64', 'A 64-bit immediate integer.') #: immediate bit counts on shift instructions. uimm8 = ImmediateKind('uimm8', 'An 8-bit immediate unsigned integer.') +#: A 32-bit immediate signed offset. +#: +#: This is used to represent an immediate address offset in load/store +#: instructions. +offset32 = ImmediateKind('offset32', 'A 32-bit immediate signed offset.') + #: A 32-bit immediate floating point operand. #: #: IEEE 754-2008 binary32 interchange format. diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 84f5978429..8b779dde60 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -6,6 +6,7 @@ //! module in the meta language. use std::fmt::{self, Display, Formatter}; +use std::i32; use std::mem; use std::str::FromStr; @@ -35,6 +36,22 @@ impl From for Imm64 { } } +// Hexadecimal with a multiple of 4 digits and group separators: +// +// 0xfff0 +// 0x0001_ffff +// 0xffff_ffff_fff8_4400 +// +fn write_hex(x: i64, f: &mut Formatter) -> fmt::Result { + let mut pos = (64 - x.leading_zeros() - 1) & 0xf0; + write!(f, "0x{:04x}", (x >> pos) & 0xffff)?; + while pos > 0 { + pos -= 16; + write!(f, "_{:04x}", (x >> pos) & 0xffff)?; + } + Ok(()) +} + impl Display for Imm64 { fn fmt(&self, f: &mut Formatter) -> fmt::Result { let x = self.0; @@ -42,91 +59,88 @@ impl Display for Imm64 { // Use decimal for small numbers. write!(f, "{}", x) } else { - // Hexadecimal with a multiple of 4 digits and group separators: - // - // 0xfff0 - // 0x0001_ffff - // 0xffff_ffff_fff8_4400 - // - let mut pos = (64 - x.leading_zeros() - 1) & 0xf0; - write!(f, "0x{:04x}", (x >> pos) & 0xffff)?; - while pos > 0 { - pos -= 16; - write!(f, "_{:04x}", (x >> pos) & 0xffff)?; - } - Ok(()) + write_hex(x, f) } } } +/// Parse a 64-bit number. +fn parse_i64(s: &str) -> Result { + let mut value: u64 = 0; + let mut digits = 0; + let negative = s.starts_with('-'); + let s2 = if negative || s.starts_with('+') { + &s[1..] + } else { + s + }; + + if s2.starts_with("0x") { + // Hexadecimal. + for ch in s2[2..].chars() { + match ch.to_digit(16) { + Some(digit) => { + digits += 1; + if digits > 16 { + return Err("Too many hexadecimal digits"); + } + // This can't overflow given the digit limit. + value = (value << 4) | digit as u64; + } + None => { + // Allow embedded underscores, but fail on anything else. + if ch != '_' { + return Err("Invalid character in hexadecimal number"); + } + } + } + } + } else { + // Decimal number, possibly negative. + for ch in s2.chars() { + match ch.to_digit(16) { + Some(digit) => { + digits += 1; + match value.checked_mul(10) { + None => return Err("Too large decimal number"), + Some(v) => value = v, + } + match value.checked_add(digit as u64) { + None => return Err("Too large decimal number"), + Some(v) => value = v, + } + } + None => { + // Allow embedded underscores, but fail on anything else. + if ch != '_' { + return Err("Invalid character in decimal number"); + } + } + } + } + } + + if digits == 0 { + return Err("No digits in number"); + } + + // We support the range-and-a-half from -2^63 .. 2^64-1. + if negative { + value = value.wrapping_neg(); + // Don't allow large negative values to wrap around and become positive. + if value as i64 > 0 { + return Err("Negative number too small"); + } + } + Ok(value as i64) +} + impl FromStr for Imm64 { type Err = &'static str; // Parse a decimal or hexadecimal `Imm64`, formatted as above. fn from_str(s: &str) -> Result { - let mut value: u64 = 0; - let mut digits = 0; - let negative = s.starts_with('-'); - let s2 = if negative { &s[1..] } else { s }; - - if s2.starts_with("0x") { - // Hexadecimal. - for ch in s2[2..].chars() { - match ch.to_digit(16) { - Some(digit) => { - digits += 1; - if digits > 16 { - return Err("Too many hexadecimal digits in Imm64"); - } - // This can't overflow given the digit limit. - value = (value << 4) | digit as u64; - } - None => { - // Allow embedded underscores, but fail on anything else. - if ch != '_' { - return Err("Invalid character in hexadecimal Imm64"); - } - } - } - } - } else { - // Decimal number, possibly negative. - for ch in s2.chars() { - match ch.to_digit(16) { - Some(digit) => { - digits += 1; - match value.checked_mul(10) { - None => return Err("Too large decimal Imm64"), - Some(v) => value = v, - } - match value.checked_add(digit as u64) { - None => return Err("Too large decimal Imm64"), - Some(v) => value = v, - } - } - None => { - // Allow embedded underscores, but fail on anything else. - if ch != '_' { - return Err("Invalid character in decimal Imm64"); - } - } - } - } - } - - if digits == 0 { - return Err("No digits in Imm64"); - } - - // We support the range-and-a-half from -2^63 .. 2^64-1. - if negative { - value = value.wrapping_neg(); - // Don't allow large negative values to wrap around and become positive. - if value as i64 > 0 { - return Err("Negative number too small for Imm64"); - } - } - Ok(Imm64::new(value as i64)) + parse_i64(s).map(Imm64::new) } } @@ -135,6 +149,68 @@ impl FromStr for Imm64 { /// This is used to indicate lane indexes typically. pub type Uimm8 = u8; +/// 32-bit signed immediate offset. +/// +/// This is used to encode an immediate offset for load/store instructions. All supported ISAs have +/// a maximum load/store offset that fits in an `i32`. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct Offset32(i32); + +impl Offset32 { + /// Create a new `Offset32` representing the signed number `x`. + pub fn new(x: i32) -> Offset32 { + Offset32(x) + } +} + +impl Into for Offset32 { + fn into(self) -> i32 { + self.0 + } +} + +impl From for Offset32 { + fn from(x: i32) -> Self { + Offset32(x) + } +} + +impl Display for Offset32 { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + // 0 displays as an empty offset. + if self.0 == 0 { + return Ok(()); + } + + // Always include a sign. + write!(f, "{}", if self.0 < 0 { '-' } else { '+' })?; + + let val = (self.0 as i64).abs(); + if val < 10_000 { + write!(f, "{}", val) + } else { + write_hex(val, f) + } + + } +} + +impl FromStr for Offset32 { + type Err = &'static str; + + // Parse a decimal or hexadecimal `Offset32`, formatted as above. + fn from_str(s: &str) -> Result { + if !(s.starts_with('-') || s.starts_with('+')) { + return Err("Offset must begin with sign"); + } + parse_i64(s).and_then(|x| if i32::MIN as i64 <= x && x <= i32::MAX as i64 { + Ok(Offset32::new(x as i32)) + } else { + Err("Offset out of range") + }) + } +} + /// An IEEE binary32 immediate floating point value. /// /// All bit patterns are allowed. @@ -486,15 +562,13 @@ mod tests { parse_ok::("0xffffffff_ffffffff", "-1"); parse_ok::("0x80000000_00000000", "0x8000_0000_0000_0000"); parse_ok::("-0x80000000_00000000", "0x8000_0000_0000_0000"); - parse_err::("-0x80000000_00000001", - "Negative number too small for Imm64"); + parse_err::("-0x80000000_00000001", "Negative number too small"); parse_ok::("18446744073709551615", "-1"); parse_ok::("-9223372036854775808", "0x8000_0000_0000_0000"); // Overflow both the `checked_add` and `checked_mul`. - parse_err::("18446744073709551616", "Too large decimal Imm64"); - parse_err::("184467440737095516100", "Too large decimal Imm64"); - parse_err::("-9223372036854775809", - "Negative number too small for Imm64"); + parse_err::("18446744073709551616", "Too large decimal number"); + parse_err::("184467440737095516100", "Too large decimal number"); + parse_err::("-9223372036854775809", "Negative number too small"); // Underscores are allowed where digits go. parse_ok::("0_0", "0"); @@ -503,21 +577,47 @@ mod tests { parse_ok::("0x97_88_bb", "0x0097_88bb"); parse_ok::("0x_97_", "151"); - parse_err::("", "No digits in Imm64"); - parse_err::("-", "No digits in Imm64"); - parse_err::("_", "No digits in Imm64"); - parse_err::("0x", "No digits in Imm64"); - parse_err::("0x_", "No digits in Imm64"); - parse_err::("-0x", "No digits in Imm64"); - parse_err::(" ", "Invalid character in decimal Imm64"); - parse_err::("0 ", "Invalid character in decimal Imm64"); - parse_err::(" 0", "Invalid character in decimal Imm64"); - parse_err::("--", "Invalid character in decimal Imm64"); - parse_err::("-0x-", "Invalid character in hexadecimal Imm64"); + parse_err::("", "No digits in number"); + parse_err::("-", "No digits in number"); + parse_err::("_", "No digits in number"); + parse_err::("0x", "No digits in number"); + parse_err::("0x_", "No digits in number"); + parse_err::("-0x", "No digits in number"); + parse_err::(" ", "Invalid character in decimal number"); + parse_err::("0 ", "Invalid character in decimal number"); + parse_err::(" 0", "Invalid character in decimal number"); + parse_err::("--", "Invalid character in decimal number"); + parse_err::("-0x-", "Invalid character in hexadecimal number"); // Hex count overflow. - parse_err::("0x0_0000_0000_0000_0000", - "Too many hexadecimal digits in Imm64"); + parse_err::("0x0_0000_0000_0000_0000", "Too many hexadecimal digits"); + } + + #[test] + fn format_offset32() { + assert_eq!(Offset32(0).to_string(), ""); + assert_eq!(Offset32(1).to_string(), "+1"); + assert_eq!(Offset32(-1).to_string(), "-1"); + assert_eq!(Offset32(9999).to_string(), "+9999"); + assert_eq!(Offset32(10000).to_string(), "+0x2710"); + assert_eq!(Offset32(-9999).to_string(), "-9999"); + assert_eq!(Offset32(-10000).to_string(), "-0x2710"); + assert_eq!(Offset32(0xffff).to_string(), "+0xffff"); + assert_eq!(Offset32(0x10000).to_string(), "+0x0001_0000"); + } + + #[test] + fn parse_offset32() { + parse_ok::("+0", ""); + parse_ok::("+1", "+1"); + parse_ok::("-0", ""); + parse_ok::("-1", "-1"); + parse_ok::("+0x0", ""); + parse_ok::("+0xf", "+15"); + parse_ok::("-0x9", "-9"); + parse_ok::("-0x8000_0000", "-0x8000_0000"); + + parse_err::("+0x8000_0000", "Offset out of range"); } #[test] From 7c3bc9d19fddf3c327b0858b92f08d4d1bfb4d7f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 10 Apr 2017 13:03:46 -0700 Subject: [PATCH 0671/3084] Add Offset32 support to the parser. --- lib/reader/src/lexer.rs | 24 ++++++++++++++++-------- lib/reader/src/parser.rs | 18 +++++++++++++++++- 2 files changed, 33 insertions(+), 9 deletions(-) diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index e1e4cc31c4..0e8eaccd0f 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -223,7 +223,7 @@ impl<'a> Lexer<'a> { // - `0x0.4p-34`: Float // // This function does not filter out all invalid numbers. It depends in the context-sensitive - // decoding of the text for that. For example, the number of allowed digits an an Ieee32` and + // decoding of the text for that. For example, the number of allowed digits in an `Ieee32` and // an `Ieee64` constant are different. fn scan_number(&mut self) -> Result, LocatedError> { let begin = self.pos; @@ -231,15 +231,21 @@ impl<'a> Lexer<'a> { let mut is_float = false; // Skip a leading sign. - if self.lookahead == Some('-') { - self.next_ch(); + match self.lookahead { + Some('-') => { + self.next_ch(); - if let Some(c) = self.lookahead { - // If the next character won't parse as a number, we return Token::Minus - if !c.is_alphanumeric() && c != '.' { - return token(Token::Minus, loc); + if let Some(c) = self.lookahead { + // If the next character won't parse as a number, we return Token::Minus + if !c.is_alphanumeric() && c != '.' { + return token(Token::Minus, loc); + } } } + Some('+') => { + self.next_ch(); + } + _ => {} } // Check for NaNs with payloads. @@ -395,6 +401,7 @@ impl<'a> Lexer<'a> { Some('.') => Some(self.scan_char(Token::Dot)), Some(':') => Some(self.scan_char(Token::Colon)), Some('=') => Some(self.scan_char(Token::Equal)), + Some('+') => Some(self.scan_number()), Some('-') => { if self.looking_at("->") { Some(self.scan_chars(2, Token::Arrow)) @@ -506,7 +513,7 @@ mod tests { #[test] fn lex_numbers() { - let mut lex = Lexer::new(" 0 2_000 -1,0xf -0x0 0.0 0x0.4p-34"); + let mut lex = Lexer::new(" 0 2_000 -1,0xf -0x0 0.0 0x0.4p-34 +5"); assert_eq!(lex.next(), token(Token::Integer("0"), 1)); assert_eq!(lex.next(), token(Token::Integer("2_000"), 1)); assert_eq!(lex.next(), token(Token::Integer("-1"), 1)); @@ -515,6 +522,7 @@ mod tests { assert_eq!(lex.next(), token(Token::Integer("-0x0"), 1)); assert_eq!(lex.next(), token(Token::Float("0.0"), 1)); assert_eq!(lex.next(), token(Token::Float("0x0.4p-34"), 1)); + assert_eq!(lex.next(), token(Token::Integer("+5"), 1)); assert_eq!(lex.next(), None); } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index b9ee4acd58..741621ed08 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -13,7 +13,7 @@ use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotDa JumpTableData, Signature, ArgumentType, ArgumentExtension, ExtFuncData, SigRef, FuncRef, ValueLoc, ArgumentLoc}; use cretonne::ir::types::VOID; -use cretonne::ir::immediates::{Imm64, Ieee32, Ieee64}; +use cretonne::ir::immediates::{Imm64, Offset32, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs}; use cretonne::isa::{self, TargetIsa, Encoding}; @@ -497,6 +497,22 @@ impl<'a> Parser<'a> { } } + // Match and consume an optional offset32 immediate. + // + // Note that that this will match an empty string as an empty offset, and that if an offset is + // present, it must contain a sign. + fn optional_offset32(&mut self) -> Result { + if let Some(Token::Integer(text)) = self.token() { + self.consume(); + // Lexer just gives us raw text that looks like an integer. + // Parse it as an `Offset32` to check for overflow and other issues. + text.parse().map_err(|e| self.error(e)) + } else { + // An offset32 operand can be absent. + Ok(Offset32::new(0)) + } + } + // Match and consume an Ieee32 immediate. fn match_ieee32(&mut self, err_msg: &str) -> Result { if let Some(Token::Float(text)) = self.token() { From 222ae8af227c8c12d673d03550a736c4e5c628e1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 10 Apr 2017 13:20:41 -0700 Subject: [PATCH 0672/3084] Define stack_load, stack_store, and stack_addr instructions. --- cranelift/docs/langref.rst | 40 ++--------------- cranelift/filetests/parser/tiny.cton | 21 +++++++++ lib/cretonne/meta/base/formats.py | 7 ++- lib/cretonne/meta/base/immediates.py | 5 ++- lib/cretonne/meta/base/instructions.py | 61 ++++++++++++++++++++++---- lib/cretonne/src/ir/builder.rs | 4 +- lib/cretonne/src/ir/instructions.rs | 17 ++++++- lib/cretonne/src/verifier.rs | 16 ++++++- lib/cretonne/src/write.rs | 7 +++ lib/reader/src/parser.rs | 41 +++++++++++++++-- 10 files changed, 163 insertions(+), 56 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 24890c8ada..911714751f 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -489,33 +489,8 @@ simply represent a contiguous sequence of bytes in the stack frame. :flag align(N): Request at least N bytes alignment. :result SS: Stack slot index. -.. inst:: a = stack_load SS, Offset - - Load a value from a stack slot at the constant offset. - - This is a polymorphic instruction that can load any value type which has a - memory representation. - - The offset is an immediate constant, not an SSA value. The memory access - cannot go out of bounds, i.e. ``sizeof(a) + Offset <= sizeof(SS)``. - - :arg SS: Stack slot declared with :inst:`stack_slot`. - :arg Offset: Immediate non-negative offset. - :result T a: Value loaded. - -.. inst:: stack_store x, SS, Offset - - Store a value to a stack slot at a constant offset. - - This is a polymorphic instruction that can store any value type with a - memory representation. - - The offset is an immediate constant, not an SSA value. The memory access - cannot go out of bounds, i.e. ``sizeof(a) + Offset <= sizeof(SS)``. - - :arg T x: Value to be stored. - :arg SS: Stack slot declared with :inst:`stack_slot`. - :arg Offset: Immediate non-negative offset. +.. autoinst:: stack_load +.. autoinst:: stack_store The dedicated stack access instructions are easy for the compiler to reason about because stack slots and offsets are fixed at compile time. For example, @@ -525,16 +500,7 @@ and stack slot alignments. It can be necessary to escape from the safety of the restricted instructions by taking the address of a stack slot. -.. inst:: a = stack_addr SS, Offset - - Get the address of a stack slot. - - Compute the absolute address of a byte in a stack slot. The offset must - refer to a byte inside the stack slot: ``0 <= Offset < sizeof(SS)``. - - :arg SS: Stack slot declared with :inst:`stack_slot`. - :arg Offset: Immediate non-negative offset. - :result iPtr a: Address. +.. autoinst:: stack_addr The :inst:`stack_addr` instruction can be used to macro-expand the stack access instructions before instruction selection:: diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 21d3a59815..e89efb4cbe 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -93,3 +93,24 @@ ebb0(vx0: i32, vx1: f32): ; nextln: v0 = bitcast.i8x4 vx0 ; nextln: v1 = bitcast.i32 vx1 ; nextln: } + +; Stack slot references +function stack() { + ss10 = stack_slot 8 + ss2 = stack_slot 4 + +ebb0: + v1 = stack_load.i32 ss10 + v2 = stack_load.i32 ss10+4 + stack_store v1, ss10+2 + stack_store v2, ss2 +} +; sameln: function stack() { +; nextln: $ss10 = stack_slot 8 +; nextln: $ss2 = stack_slot 4 + +; check: ebb0: +; nextln: $v1 = stack_load.i32 $ss10 +; nextln: $v2 = stack_load.i32 $ss10+4 +; nextln: stack_store $v1, $ss10+2 +; nextln: stack_store $v2, $ss2 diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index cf1922627f..c840e1b360 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -8,8 +8,8 @@ in this module. from __future__ import absolute_import from cdsl.formats import InstructionFormat from cdsl.operands import VALUE, VARIABLE_ARGS -from .immediates import imm64, uimm8, ieee32, ieee64, intcc, floatcc -from .entities import ebb, sig_ref, func_ref, jump_table +from .immediates import imm64, uimm8, ieee32, ieee64, offset32, intcc, floatcc +from .entities import ebb, sig_ref, func_ref, jump_table, stack_slot Nullary = InstructionFormat() @@ -52,5 +52,8 @@ IndirectCall = InstructionFormat( sig_ref, VALUE, VARIABLE_ARGS, multiple_results=True) +StackLoad = InstructionFormat(stack_slot, offset32) +StackStore = InstructionFormat(VALUE, stack_slot, offset32) + # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index e11ed78e28..ad1bda9989 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -21,7 +21,10 @@ uimm8 = ImmediateKind('uimm8', 'An 8-bit immediate unsigned integer.') #: #: This is used to represent an immediate address offset in load/store #: instructions. -offset32 = ImmediateKind('offset32', 'A 32-bit immediate signed offset.') +offset32 = ImmediateKind( + 'offset32', + 'A 32-bit immediate signed offset.', + default_member='offset') #: A 32-bit immediate floating point operand. #: diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index d9eda7a220..29d2f44a16 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -9,7 +9,7 @@ from cdsl.operands import Operand, VARIABLE_ARGS from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup from base.types import i8, f32, f64, b1 -from base.immediates import imm64, uimm8, ieee32, ieee64 +from base.immediates import imm64, uimm8, ieee32, ieee64, offset32 from base.immediates import intcc, floatcc from base import entities import base.formats # noqa @@ -28,6 +28,12 @@ TxN = TypeVar( Any = TypeVar( 'Any', 'Any integer, float, or boolean scalar or vector type', ints=True, floats=True, bools=True, scalars=True, simd=True) +Mem = TypeVar( + 'Mem', 'Any type that can be stored in memory', + ints=True, floats=True, simd=True) +MemTo = TypeVar( + 'MemTo', 'Any type that can be stored in memory', + ints=True, floats=True, simd=True) # # Control flow @@ -195,6 +201,52 @@ call_indirect = Instruction( """, ins=(SIG, callee, args), outs=rvals, is_call=True) +# +# Memory operations +# + +SS = Operand('SS', entities.stack_slot) +Offset = Operand('Offset', offset32, 'In-bounds offset into stack slot') +x = Operand('x', Mem, doc='Value to be stored') +a = Operand('a', Mem, doc='Value loaded') +addr = Operand('addr', iAddr) + +stack_load = Instruction( + 'stack_load', r""" + Load a value from a stack slot at the constant offset. + + This is a polymorphic instruction that can load any value type which + has a memory representation. + + The offset is an immediate constant, not an SSA value. The memory + access cannot go out of bounds, i.e. + :math:`sizeof(a) + Offset <= sizeof(SS)`. + """, + ins=(SS, Offset), outs=a) + +stack_store = Instruction( + 'stack_store', r""" + Store a value to a stack slot at a constant offset. + + This is a polymorphic instruction that can store any value type with a + memory representation. + + The offset is an immediate constant, not an SSA value. The memory + access cannot go out of bounds, i.e. + :math:`sizeof(a) + Offset <= sizeof(SS)`. + """, + ins=(x, SS, Offset)) + +stack_addr = Instruction( + 'stack_addr', r""" + Get the address of a stack slot. + + Compute the absolute address of a byte in a stack slot. The offset must + refer to a byte inside the stack slot: + :math:`0 <= Offset < sizeof(SS)`. + """, + ins=(SS, Offset), outs=addr) + # # Materializing constants. # @@ -1095,13 +1147,6 @@ nearest = Instruction( # Conversions # -Mem = TypeVar( - 'Mem', 'Any type that can be stored in memory', - ints=True, floats=True, simd=True) -MemTo = TypeVar( - 'MemTo', 'Any type that can be stored in memory', - ints=True, floats=True, simd=True) - x = Operand('x', Mem) a = Operand('a', MemTo, 'Bits of `x` reinterpreted') diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index fb5c4ba20a..e15a93bf30 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -5,8 +5,8 @@ use ir::types; use ir::{InstructionData, DataFlowGraph, Cursor}; -use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, ValueList}; -use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64}; +use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, StackSlot, ValueList}; +use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32}; use ir::condcodes::{IntCC, FloatCC}; /// Base trait for instruction builders. diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 9816ccbc91..a0ff3a3be4 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -10,8 +10,8 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::ops::{Deref, DerefMut}; -use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef}; -use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64}; +use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef, StackSlot}; +use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32}; use ir::condcodes::*; use ir::types; use ir::DataFlowGraph; @@ -227,6 +227,19 @@ pub enum InstructionData { sig_ref: SigRef, args: ValueList, }, + StackLoad { + opcode: Opcode, + ty: Type, + stack_slot: StackSlot, + offset: Offset32, + }, + StackStore { + opcode: Opcode, + ty: Type, + arg: Value, + stack_slot: StackSlot, + offset: Offset32, + }, } /// A variable list of `Value` operands used for function call arguments and passing arguments to diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 776755d4c4..29a40d55b9 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -57,7 +57,8 @@ use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir::entities::AnyEntity; use ir::instructions::{InstructionFormat, BranchInfo, ResolvedConstraint, CallInfo}; -use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, Value, Type}; +use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, StackSlot, + Value, Type}; use Context; use std::fmt::{self, Display, Formatter}; use std::result; @@ -248,6 +249,11 @@ impl<'a> Verifier<'a> { self.verify_sig_ref(inst, sig_ref)?; self.verify_value_list(inst, args)?; } + &StackLoad { stack_slot, .. } | + &StackStore { stack_slot, .. } => { + self.verify_stack_slot(inst, stack_slot)?; + } + // Exhaustive list so we can't forget to add new formats &Nullary { .. } | &Unary { .. } | @@ -293,6 +299,14 @@ impl<'a> Verifier<'a> { } } + fn verify_stack_slot(&self, inst: Inst, ss: StackSlot) -> Result<()> { + if !self.func.stack_slots.is_valid(ss) { + err!(inst, "invalid stack slot {}", ss) + } else { + Ok(()) + } + } + fn verify_value_list(&self, inst: Inst, l: &ValueList) -> Result<()> { if !l.is_valid(&self.func.dfg.value_lists) { err!(inst, "invalid value list reference {:?}", l) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index eaa5248408..1ab481cdd1 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -313,6 +313,13 @@ pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result args[0], DisplayValues(&args[1..])) } + StackLoad { stack_slot, offset, .. } => write!(w, " {}{}", stack_slot, offset), + StackStore { + arg, + stack_slot, + offset, + .. + } => write!(w, " {}, {}{}", arg, stack_slot, offset), } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 741621ed08..9a36ec2451 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -11,7 +11,7 @@ use std::{u16, u32}; use std::mem; use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotData, JumpTable, JumpTableData, Signature, ArgumentType, ArgumentExtension, ExtFuncData, SigRef, - FuncRef, ValueLoc, ArgumentLoc}; + FuncRef, StackSlot, ValueLoc, ArgumentLoc}; use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Offset32, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; @@ -124,6 +124,14 @@ impl<'a> Context<'a> { .def_ss(number, self.function.stack_slots.push(data), loc) } + // Resolve a reference to a stack slot. + fn get_ss(&self, number: u32, loc: &Location) -> Result { + match self.map.get_ss(number) { + Some(sig) => Ok(sig), + None => err!(loc, "undefined stack slot ss{}", number), + } + } + // Allocate a new signature and add a mapping number -> SigRef. fn add_sig(&mut self, number: u32, data: Signature, loc: &Location) -> Result<()> { self.map @@ -210,14 +218,16 @@ impl<'a> Context<'a> { InstructionData::Nullary { .. } | InstructionData::UnaryImm { .. } | InstructionData::UnaryIeee32 { .. } | - InstructionData::UnaryIeee64 { .. } => {} + InstructionData::UnaryIeee64 { .. } | + InstructionData::StackLoad { .. } => {} InstructionData::BinaryImm { ref mut arg, .. } | InstructionData::BranchTable { ref mut arg, .. } | InstructionData::ExtractLane { ref mut arg, .. } | InstructionData::IntCompareImm { ref mut arg, .. } | InstructionData::Unary { ref mut arg, .. } | - InstructionData::UnarySplit { ref mut arg, .. } => { + InstructionData::UnarySplit { ref mut arg, .. } | + InstructionData::StackStore { ref mut arg, .. } => { self.map.rewrite_value(arg, loc)?; } @@ -1659,6 +1669,31 @@ impl<'a> Parser<'a> { table: table, } } + InstructionFormat::StackLoad => { + let ss = self.match_ss("expected stack slot number: ss«n»") + .and_then(|num| ctx.get_ss(num, &self.loc))?; + let offset = self.optional_offset32()?; + InstructionData::StackLoad { + opcode: opcode, + ty: VOID, + stack_slot: ss, + offset: offset, + } + } + InstructionFormat::StackStore => { + let arg = self.match_value("expected SSA value operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let ss = self.match_ss("expected stack slot number: ss«n»") + .and_then(|num| ctx.get_ss(num, &self.loc))?; + let offset = self.optional_offset32()?; + InstructionData::StackStore { + opcode: opcode, + ty: VOID, + arg: arg, + stack_slot: ss, + offset: offset, + } + } }; Ok(idata) } From e78e4ea4ec1943816916e6bf21d20fc88d3d18de Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 10 Apr 2017 14:32:06 -0700 Subject: [PATCH 0673/3084] Add a Uoffset32 immediate operand kind. WebAssembly memory instructions encode a 32-bit unsigned offset that is used to compute an effective address. --- lib/cretonne/meta/base/immediates.py | 9 +++ lib/cretonne/src/ir/immediates.rs | 95 +++++++++++++++++++++++++++- lib/reader/src/parser.rs | 18 +++++- 3 files changed, 120 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index ad1bda9989..7f1662f5c1 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -26,6 +26,15 @@ offset32 = ImmediateKind( 'A 32-bit immediate signed offset.', default_member='offset') +#: A 32-bit immediate unsigned offset. +#: +#: This is used to represent an immediate address offset in WebAssembly memory +#: instructions. +uoffset32 = ImmediateKind( + 'uoffset32', + 'A 32-bit immediate unsigned offset.', + default_member='offset') + #: A 32-bit immediate floating point operand. #: #: IEEE 754-2008 binary32 interchange format. diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 8b779dde60..120409f3b8 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -6,7 +6,7 @@ //! module in the meta language. use std::fmt::{self, Display, Formatter}; -use std::i32; +use std::{i32, u32}; use std::mem; use std::str::FromStr; @@ -169,6 +169,12 @@ impl Into for Offset32 { } } +impl Into for Offset32 { + fn into(self) -> i64 { + self.0 as i64 + } +} + impl From for Offset32 { fn from(x: i32) -> Self { Offset32(x) @@ -211,6 +217,71 @@ impl FromStr for Offset32 { } } +/// 32-bit unsigned immediate offset. +/// +/// This is used to encode an immediate offset for WebAssembly heap_load/heap_store instructions. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct Uoffset32(u32); + +impl Uoffset32 { + /// Create a new `Uoffset32` representing the number `x`. + pub fn new(x: u32) -> Uoffset32 { + Uoffset32(x) + } +} + +impl Into for Uoffset32 { + fn into(self) -> u32 { + self.0 + } +} + +impl Into for Uoffset32 { + fn into(self) -> i64 { + self.0 as i64 + } +} + +impl From for Uoffset32 { + fn from(x: u32) -> Self { + Uoffset32(x) + } +} + +impl Display for Uoffset32 { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + // 0 displays as an empty offset. + if self.0 == 0 { + return Ok(()); + } + + // Always include a sign. + if self.0 < 10_000 { + write!(f, "+{}", self.0) + } else { + write!(f, "+")?; + write_hex(self.0 as i64, f) + } + + } +} + +impl FromStr for Uoffset32 { + type Err = &'static str; + + // Parse a decimal or hexadecimal `Uoffset32`, formatted as above. + fn from_str(s: &str) -> Result { + if !s.starts_with('+') { + return Err("Unsigned offset must begin with '+' sign"); + } + parse_i64(s).and_then(|x| if 0 <= x && x <= u32::MAX as i64 { + Ok(Uoffset32::new(x as u32)) + } else { + Err("Offset out of range") + }) + } +} + /// An IEEE binary32 immediate floating point value. /// /// All bit patterns are allowed. @@ -620,6 +691,28 @@ mod tests { parse_err::("+0x8000_0000", "Offset out of range"); } + #[test] + fn format_uoffset32() { + assert_eq!(Uoffset32(0).to_string(), ""); + assert_eq!(Uoffset32(1).to_string(), "+1"); + assert_eq!(Uoffset32(9999).to_string(), "+9999"); + assert_eq!(Uoffset32(10000).to_string(), "+0x2710"); + assert_eq!(Uoffset32(0xffff).to_string(), "+0xffff"); + assert_eq!(Uoffset32(0x10000).to_string(), "+0x0001_0000"); + } + + #[test] + fn parse_uoffset32() { + parse_ok::("+0", ""); + parse_ok::("+1", "+1"); + parse_ok::("+0x0", ""); + parse_ok::("+0xf", "+15"); + parse_ok::("+0x8000_0000", "+0x8000_0000"); + parse_ok::("+0xffff_ffff", "+0xffff_ffff"); + + parse_err::("+0x1_0000_0000", "Offset out of range"); + } + #[test] fn format_ieee32() { assert_eq!(Ieee32::new(0.0).to_string(), "0.0"); diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 9a36ec2451..8775f1f0b4 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -13,7 +13,7 @@ use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotDa JumpTableData, Signature, ArgumentType, ArgumentExtension, ExtFuncData, SigRef, FuncRef, StackSlot, ValueLoc, ArgumentLoc}; use cretonne::ir::types::VOID; -use cretonne::ir::immediates::{Imm64, Offset32, Ieee32, Ieee64}; +use cretonne::ir::immediates::{Imm64, Offset32, Uoffset32, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs}; use cretonne::isa::{self, TargetIsa, Encoding}; @@ -523,6 +523,22 @@ impl<'a> Parser<'a> { } } + // Match and consume an optional uoffset32 immediate. + // + // Note that that this will match an empty string as an empty offset, and that if an offset is + // present, it must contain a `+` sign. + fn optional_uoffset32(&mut self) -> Result { + if let Some(Token::Integer(text)) = self.token() { + self.consume(); + // Lexer just gives us raw text that looks like an integer. + // Parse it as a `Uoffset32` to check for overflow and other issues. + text.parse().map_err(|e| self.error(e)) + } else { + // A uoffset32 operand can be absent. + Ok(Uoffset32::new(0)) + } + } + // Match and consume an Ieee32 immediate. fn match_ieee32(&mut self, err_msg: &str) -> Result { if let Some(Token::Float(text)) = self.token() { From b474485c0ddb561eb1979b4ed25d7f5533a99e08 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 10 Apr 2017 15:03:10 -0700 Subject: [PATCH 0674/3084] Add heap_load, heap_store, and heap_addr instructions. These are used when lowering WebAssembly sandbox code. --- cranelift/docs/langref.rst | 42 ++------------------------ cranelift/filetests/parser/tiny.cton | 14 +++++++++ lib/cretonne/meta/base/formats.py | 8 ++++- lib/cretonne/meta/base/instructions.py | 41 ++++++++++++++++++++++++- lib/cretonne/src/ir/builder.rs | 2 +- lib/cretonne/src/ir/instructions.rs | 14 ++++++++- lib/cretonne/src/verifier.rs | 4 ++- lib/cretonne/src/write.rs | 2 ++ lib/reader/src/parser.rs | 29 ++++++++++++++++-- 9 files changed, 110 insertions(+), 46 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 911714751f..4156a95aa3 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -533,49 +533,13 @@ than the native pointer size, for example unsigned :type:`i32` offsets on a :arg Name: String identifying the heap in the runtime environment. :result H: Heap identifier. -.. inst:: a = heap_load H, p, Offset - - Load a value at the address ``p + Offset`` in the heap H. - - Trap if the heap access would be out of bounds. - - :arg H: Heap identifier created by :inst:`heap`. - :arg iN p: Unsigned base address in heap. - :arg Offset: Immediate signed offset. - :flag align(N): Expected alignment of ``p + Offset``. Power of two. - :flag aligntrap: Always trap if the memory access is misaligned. - :result T a: Loaded value. - -.. inst:: a = heap_store H, x, p, Offset - - Store a value at the address ``p + Offset`` in the heap H. - - Trap if the heap access would be out of bounds. - - :arg H: Heap identifier created by :inst:`heap`. - :arg T x: Value to be stored. - :arg iN p: Unsigned base address in heap. - :arg Offset: Immediate signed offset. - :flag align(N): Expected alignment of ``p + Offset``. Power of two. - :flag aligntrap: Always trap if the memory access is misaligned. +.. autoinst:: heap_load +.. autoinst:: heap_store When optimizing heap accesses, Cretonne may separate the heap bounds checking and address computations from the memory accesses. -.. inst:: a = heap_addr H, p, Size - - Bounds check and compute absolute address of heap memory. - - Verify that the address range ``p .. p + Size - 1`` is valid in the heap H, - and trap if not. - - Convert the heap-relative address in ``p`` to a real absolute address and - return it. - - :arg H: Heap identifier created by :inst:`heap`. - :arg iN p: Unsigned base address in heap. - :arg Size: Immediate unsigned byte count for range to verify. - :result iPtr a: Absolute address corresponding to ``p``. +.. autoinst:: heap_addr A small example using heaps:: diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index e89efb4cbe..92842cd140 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -114,3 +114,17 @@ ebb0: ; nextln: $v2 = stack_load.i32 $ss10+4 ; nextln: stack_store $v1, $ss10+2 ; nextln: stack_store $v2, $ss2 + +; Heap access instructions. +function heap(i32) { + ; TODO: heap0 = heap %foo +ebb0(v1: i32): + v2 = heap_load.f32 v1 + v3 = heap_load.f32 v1+12 + heap_store v3, v1 +} +; sameln: function heap(i32) { +; nextln: ebb0($v1: i32): +; nextln: $v2 = heap_load.f32 $v1 +; nextln: $v3 = heap_load.f32 $v1+12 +; nextln: heap_store $v3, $v1 diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index c840e1b360..767be3be7e 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -8,7 +8,8 @@ in this module. from __future__ import absolute_import from cdsl.formats import InstructionFormat from cdsl.operands import VALUE, VARIABLE_ARGS -from .immediates import imm64, uimm8, ieee32, ieee64, offset32, intcc, floatcc +from .immediates import imm64, uimm8, ieee32, ieee64, offset32, uoffset32 +from .immediates import intcc, floatcc from .entities import ebb, sig_ref, func_ref, jump_table, stack_slot Nullary = InstructionFormat() @@ -55,5 +56,10 @@ IndirectCall = InstructionFormat( StackLoad = InstructionFormat(stack_slot, offset32) StackStore = InstructionFormat(VALUE, stack_slot, offset32) +# Accessing a WebAssembly heap. +# TODO: Add a reference to a `heap` declared in the preamble. +HeapLoad = InstructionFormat(VALUE, uoffset32) +HeapStore = InstructionFormat(VALUE, VALUE, uoffset32) + # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 29d2f44a16..d3de015311 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -9,7 +9,7 @@ from cdsl.operands import Operand, VARIABLE_ARGS from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup from base.types import i8, f32, f64, b1 -from base.immediates import imm64, uimm8, ieee32, ieee64, offset32 +from base.immediates import imm64, uimm8, ieee32, ieee64, offset32, uoffset32 from base.immediates import intcc, floatcc from base import entities import base.formats # noqa @@ -209,6 +209,7 @@ SS = Operand('SS', entities.stack_slot) Offset = Operand('Offset', offset32, 'In-bounds offset into stack slot') x = Operand('x', Mem, doc='Value to be stored') a = Operand('a', Mem, doc='Value loaded') +p = Operand('p', iAddr) addr = Operand('addr', iAddr) stack_load = Instruction( @@ -247,6 +248,44 @@ stack_addr = Instruction( """, ins=(SS, Offset), outs=addr) +# +# WebAssembly bounds-checked heap accesses. +# +# TODO: Add a `heap` operand that selects between multiple heaps. +# TODO: Should the immediate offset be a `u32`? +# TODO: Distinguish between `iAddr` for a heap and for a target address? i.e., +# 32-bit WebAssembly on a 64-bit target has two different types. + +Offset = Operand('Offset', uoffset32, 'Unsigned offset to effective address') + +heap_load = Instruction( + 'heap_load', r""" + Load a value at the address :math:`p + Offset` in the heap H. + + Trap if the heap access would be out of bounds. + """, + ins=(p, Offset), outs=a) + +heap_store = Instruction( + 'heap_store', r""" + Store a value at the address :math:`p + Offset` in the heap H. + + Trap if the heap access would be out of bounds. + """, + ins=(x, p, Offset)) + +heap_addr = Instruction( + 'heap_addr', r""" + Bounds check and compute absolute address of heap memory. + + Verify that the address range ``p .. p + Size - 1`` is valid in the + heap H, and trap if not. + + Convert the heap-relative address in ``p`` to a real absolute address + and return it. + """, + ins=(p, Offset), outs=addr) + # # Materializing constants. # diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index e15a93bf30..8b95f20272 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -6,7 +6,7 @@ use ir::types; use ir::{InstructionData, DataFlowGraph, Cursor}; use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, StackSlot, ValueList}; -use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32}; +use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32, Uoffset32}; use ir::condcodes::{IntCC, FloatCC}; /// Base trait for instruction builders. diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index a0ff3a3be4..5e8d66a09c 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -11,7 +11,7 @@ use std::str::FromStr; use std::ops::{Deref, DerefMut}; use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef, StackSlot}; -use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32}; +use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32, Uoffset32}; use ir::condcodes::*; use ir::types; use ir::DataFlowGraph; @@ -240,6 +240,18 @@ pub enum InstructionData { stack_slot: StackSlot, offset: Offset32, }, + HeapLoad { + opcode: Opcode, + ty: Type, + arg: Value, + offset: Uoffset32, + }, + HeapStore { + opcode: Opcode, + ty: Type, + args: [Value; 2], + offset: Uoffset32, + }, } /// A variable list of `Value` operands used for function call arguments and passing arguments to diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 29a40d55b9..b1b4429d1a 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -269,7 +269,9 @@ impl<'a> Verifier<'a> { &ExtractLane { .. } | &IntCompare { .. } | &IntCompareImm { .. } | - &FloatCompare { .. } => {} + &FloatCompare { .. } | + &HeapLoad { .. } | + &HeapStore { .. } => {} } Ok(()) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 1ab481cdd1..f4b7de5f75 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -320,6 +320,8 @@ pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result offset, .. } => write!(w, " {}, {}{}", arg, stack_slot, offset), + HeapLoad { arg, offset, .. } => write!(w, " {}{}", arg, offset), + HeapStore { args, offset, .. } => write!(w, " {}, {}{}", args[0], args[1], offset), } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 8775f1f0b4..e813404130 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -227,15 +227,18 @@ impl<'a> Context<'a> { InstructionData::IntCompareImm { ref mut arg, .. } | InstructionData::Unary { ref mut arg, .. } | InstructionData::UnarySplit { ref mut arg, .. } | - InstructionData::StackStore { ref mut arg, .. } => { + InstructionData::StackStore { ref mut arg, .. } | + InstructionData::HeapLoad { ref mut arg, .. } => { self.map.rewrite_value(arg, loc)?; } + // `args: Value[2]` InstructionData::Binary { ref mut args, .. } | InstructionData::BinaryOverflow { ref mut args, .. } | InstructionData::InsertLane { ref mut args, .. } | InstructionData::IntCompare { ref mut args, .. } | - InstructionData::FloatCompare { ref mut args, .. } => { + InstructionData::FloatCompare { ref mut args, .. } | + InstructionData::HeapStore { ref mut args, .. } => { self.map.rewrite_values(args, loc)?; } @@ -1710,6 +1713,28 @@ impl<'a> Parser<'a> { offset: offset, } } + InstructionFormat::HeapLoad => { + let addr = self.match_value("expected SSA value address")?; + let offset = self.optional_uoffset32()?; + InstructionData::HeapLoad { + opcode: opcode, + ty: VOID, + arg: addr, + offset: offset, + } + } + InstructionFormat::HeapStore => { + let arg = self.match_value("expected SSA value operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let addr = self.match_value("expected SSA value address")?; + let offset = self.optional_uoffset32()?; + InstructionData::HeapStore { + opcode: opcode, + ty: VOID, + args: [arg, addr], + offset: offset, + } + } }; Ok(idata) } From 0c3771bccbb03d351be3736cc3d76f1338cc3d78 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 10 Apr 2017 12:47:16 -0700 Subject: [PATCH 0675/3084] Ensure that the docs examples verify as Cretonne IL. Any *.cton files in the docs directory are now included when running the test-all.sh script. This is to ensure that the examples are in fact correct IL. Always print NaN and Inf floats with a sign. Print the positive ones as +NaN and +Inf to make them easier to parse. --- cranelift/docs/cton_lexer.py | 2 +- cranelift/docs/example.cton | 14 ++++--- cranelift/docs/langref.rst | 3 +- cranelift/test-all.sh | 2 +- lib/cretonne/src/ir/immediates.rs | 64 ++++++++++++++++++------------- 5 files changed, 49 insertions(+), 36 deletions(-) diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py index 868bb23241..b40cb70e41 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/cton_lexer.py @@ -39,7 +39,7 @@ class CretonneLexer(RegexLexer): # Numbers. (r'[-+]?0[xX][0-9a-fA-F]+', Number.Hex), (r'[-+]?0[xX][0-9a-fA-F]*\.[0-9a-fA-F]*([pP]\d+)?', Number.Hex), - (r'[-+]?(\d+\.\d+([eE]\d+)?|[sq]NaN|Inf)', Number.Float), + (r'[-+]?(\d+\.\d+([eE]\d+)?|s?NaN|Inf)', Number.Float), (r'[-+]?\d+', Number.Integer), # Known attributes. (keywords('align', 'aligntrap', 'uext', 'sext', 'inreg'), diff --git a/cranelift/docs/example.cton b/cranelift/docs/example.cton index cdfc61bf4b..db87e891d1 100644 --- a/cranelift/docs/example.cton +++ b/cranelift/docs/example.cton @@ -1,18 +1,20 @@ +test verifier + function average(i32, i32) -> f32 { - ss1 = stack_slot 8, align 4 ; Stack slot for ``sum``. + ss1 = stack_slot 8 ; Stack slot for ``sum``. ebb1(v1: i32, v2: i32): v3 = f64const 0x0.0 stack_store v3, ss1 brz v2, ebb3 ; Handle count == 0. v4 = iconst.i32 0 - br ebb2(v4) + jump ebb2(v4) ebb2(v5: i32): v6 = imul_imm v5, 4 v7 = iadd v1, v6 v8 = heap_load.f32 v7 ; array[i] - v9 = fext.f64 v8 + v9 = fpromote.f64 v8 v10 = stack_load.f64 ss1 v11 = fadd v9, v10 stack_store v11, ss1 @@ -20,12 +22,12 @@ ebb2(v5: i32): v13 = icmp ult v12, v2 brnz v13, ebb2(v12) ; Loop backedge. v14 = stack_load.f64 ss1 - v15 = cvt_utof.f64 v2 + v15 = fcvt_from_uint.f64 v2 v16 = fdiv v14, v15 - v17 = ftrunc.f32 v16 + v17 = fdemote.f32 v16 return v17 ebb3: - v100 = f32const qNaN + v100 = f32const +NaN return v100 } diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 4156a95aa3..6ded3a8bfd 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -31,8 +31,7 @@ Here is the same function compiled into Cretonne IL: .. literalinclude:: example.cton :language: cton - :linenos: - :emphasize-lines: 2 + :lines: 2- The first line of a function definition provides the function *name* and the :term:`function signature` which declares the argument and return types. diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 522cf594a2..5b718d8bc4 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -79,6 +79,6 @@ export CTONUTIL="$topdir/target/release/cton-util" cd "$topdir" banner "File tests" -"$CTONUTIL" test filetests +"$CTONUTIL" test filetests docs banner "OK" diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 120409f3b8..e061b88482 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -337,6 +337,11 @@ fn format_float(bits: u64, w: u8, t: u8, f: &mut Formatter) -> fmt::Result { write!(f, "0x0.{0:01$x}p{2}", left_t_bits, digits as usize, emin) } } else if e_bits == max_e_bits { + // Always print a `+` or `-` sign for these special values. + // This makes them easier to parse as they can't be confused as identifiers. + if sign_bit == 0 { + write!(f, "+")?; + } if t_bits == 0 { // Infinity. write!(f, "Inf") @@ -375,6 +380,8 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { let (sign_bit, s2) = if s.starts_with('-') { (1u64 << t + w, &s[1..]) + } else if s.starts_with('+') { + (0, &s[1..]) } else { (0, s) }; @@ -731,27 +738,29 @@ mod tests { "0x0.800000p-126"); assert_eq!(Ieee32::new(f32::MIN_POSITIVE * f32::EPSILON).to_string(), "0x0.000002p-126"); - assert_eq!(Ieee32::new(f32::INFINITY).to_string(), "Inf"); + assert_eq!(Ieee32::new(f32::INFINITY).to_string(), "+Inf"); assert_eq!(Ieee32::new(f32::NEG_INFINITY).to_string(), "-Inf"); - assert_eq!(Ieee32::new(f32::NAN).to_string(), "NaN"); + assert_eq!(Ieee32::new(f32::NAN).to_string(), "+NaN"); assert_eq!(Ieee32::new(-f32::NAN).to_string(), "-NaN"); // Construct some qNaNs with payloads. - assert_eq!(Ieee32::from_bits(0x7fc00001).to_string(), "NaN:0x1"); - assert_eq!(Ieee32::from_bits(0x7ff00001).to_string(), "NaN:0x300001"); + assert_eq!(Ieee32::from_bits(0x7fc00001).to_string(), "+NaN:0x1"); + assert_eq!(Ieee32::from_bits(0x7ff00001).to_string(), "+NaN:0x300001"); // Signaling NaNs. - assert_eq!(Ieee32::from_bits(0x7f800001).to_string(), "sNaN:0x1"); - assert_eq!(Ieee32::from_bits(0x7fa00001).to_string(), "sNaN:0x200001"); + assert_eq!(Ieee32::from_bits(0x7f800001).to_string(), "+sNaN:0x1"); + assert_eq!(Ieee32::from_bits(0x7fa00001).to_string(), "+sNaN:0x200001"); } #[test] fn parse_ieee32() { parse_ok::("0.0", "0.0"); + parse_ok::("+0.0", "0.0"); parse_ok::("-0.0", "-0.0"); parse_ok::("0x0", "0.0"); parse_ok::("0x0.0", "0.0"); parse_ok::("0x.0", "0.0"); parse_ok::("0x0.", "0.0"); parse_ok::("0x1", "0x1.000000p0"); + parse_ok::("+0x1", "0x1.000000p0"); parse_ok::("-0x1", "-0x1.000000p0"); parse_ok::("0x10", "0x1.000000p4"); parse_ok::("0x10.0", "0x1.000000p4"); @@ -793,20 +802,22 @@ mod tests { parse_err::("0x1.0p-150", "Magnitude too small"); // NaNs and Infs. - parse_ok::("Inf", "Inf"); + parse_ok::("Inf", "+Inf"); + parse_ok::("+Inf", "+Inf"); parse_ok::("-Inf", "-Inf"); - parse_ok::("NaN", "NaN"); + parse_ok::("NaN", "+NaN"); + parse_ok::("+NaN", "+NaN"); parse_ok::("-NaN", "-NaN"); - parse_ok::("NaN:0x0", "NaN"); + parse_ok::("NaN:0x0", "+NaN"); parse_err::("NaN:", "Float must be hexadecimal"); parse_err::("NaN:0", "Float must be hexadecimal"); parse_err::("NaN:0x", "Invalid NaN payload"); - parse_ok::("NaN:0x000001", "NaN:0x1"); - parse_ok::("NaN:0x300001", "NaN:0x300001"); + parse_ok::("NaN:0x000001", "+NaN:0x1"); + parse_ok::("NaN:0x300001", "+NaN:0x300001"); parse_err::("NaN:0x400001", "Invalid NaN payload"); - parse_ok::("sNaN:0x1", "sNaN:0x1"); + parse_ok::("sNaN:0x1", "+sNaN:0x1"); parse_err::("sNaN:0x0", "Invalid sNaN payload"); - parse_ok::("sNaN:0x200001", "sNaN:0x200001"); + parse_ok::("sNaN:0x200001", "+sNaN:0x200001"); parse_err::("sNaN:0x400001", "Invalid sNaN payload"); } @@ -829,19 +840,20 @@ mod tests { "0x0.8000000000000p-1022"); assert_eq!(Ieee64::new(f64::MIN_POSITIVE * f64::EPSILON).to_string(), "0x0.0000000000001p-1022"); - assert_eq!(Ieee64::new(f64::INFINITY).to_string(), "Inf"); + assert_eq!(Ieee64::new(f64::INFINITY).to_string(), "+Inf"); assert_eq!(Ieee64::new(f64::NEG_INFINITY).to_string(), "-Inf"); - assert_eq!(Ieee64::new(f64::NAN).to_string(), "NaN"); + assert_eq!(Ieee64::new(f64::NAN).to_string(), "+NaN"); assert_eq!(Ieee64::new(-f64::NAN).to_string(), "-NaN"); // Construct some qNaNs with payloads. - assert_eq!(Ieee64::from_bits(0x7ff8000000000001).to_string(), "NaN:0x1"); + assert_eq!(Ieee64::from_bits(0x7ff8000000000001).to_string(), + "+NaN:0x1"); assert_eq!(Ieee64::from_bits(0x7ffc000000000001).to_string(), - "NaN:0x4000000000001"); + "+NaN:0x4000000000001"); // Signaling NaNs. assert_eq!(Ieee64::from_bits(0x7ff0000000000001).to_string(), - "sNaN:0x1"); + "+sNaN:0x1"); assert_eq!(Ieee64::from_bits(0x7ff4000000000001).to_string(), - "sNaN:0x4000000000001"); + "+sNaN:0x4000000000001"); } #[test] @@ -894,20 +906,20 @@ mod tests { parse_err::("0x1.0p-1075", "Magnitude too small"); // NaNs and Infs. - parse_ok::("Inf", "Inf"); + parse_ok::("Inf", "+Inf"); parse_ok::("-Inf", "-Inf"); - parse_ok::("NaN", "NaN"); + parse_ok::("NaN", "+NaN"); parse_ok::("-NaN", "-NaN"); - parse_ok::("NaN:0x0", "NaN"); + parse_ok::("NaN:0x0", "+NaN"); parse_err::("NaN:", "Float must be hexadecimal"); parse_err::("NaN:0", "Float must be hexadecimal"); parse_err::("NaN:0x", "Invalid NaN payload"); - parse_ok::("NaN:0x000001", "NaN:0x1"); - parse_ok::("NaN:0x4000000000001", "NaN:0x4000000000001"); + parse_ok::("NaN:0x000001", "+NaN:0x1"); + parse_ok::("NaN:0x4000000000001", "+NaN:0x4000000000001"); parse_err::("NaN:0x8000000000001", "Invalid NaN payload"); - parse_ok::("sNaN:0x1", "sNaN:0x1"); + parse_ok::("sNaN:0x1", "+sNaN:0x1"); parse_err::("sNaN:0x0", "Invalid sNaN payload"); - parse_ok::("sNaN:0x4000000000001", "sNaN:0x4000000000001"); + parse_ok::("sNaN:0x4000000000001", "+sNaN:0x4000000000001"); parse_err::("sNaN:0x8000000000001", "Invalid sNaN payload"); } } From aad6ebebb5740c8c98cb3d6dd3399b44383294d7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 11 Apr 2017 09:54:55 -0700 Subject: [PATCH 0676/3084] Add load and store instructions. Define a MemFlags class, currently holding a notrap and aligned flag. --- cranelift/docs/langref.rst | 36 +--------- cranelift/filetests/parser/tiny.cton | 27 ++++++++ lib/cretonne/meta/base/formats.py | 5 +- lib/cretonne/meta/base/immediates.py | 6 ++ lib/cretonne/meta/base/instructions.py | 21 +++++- lib/cretonne/src/ir/builder.rs | 3 +- lib/cretonne/src/ir/instructions.rs | 16 ++++- lib/cretonne/src/ir/memflags.rs | 92 ++++++++++++++++++++++++++ lib/cretonne/src/ir/mod.rs | 8 ++- lib/cretonne/src/verifier.rs | 4 +- lib/cretonne/src/write.rs | 7 ++ lib/reader/src/parser.rs | 47 ++++++++++++- 12 files changed, 227 insertions(+), 45 deletions(-) create mode 100644 lib/cretonne/src/ir/memflags.rs diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 6ded3a8bfd..12949ab691 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -428,46 +428,14 @@ accessing memory. However, it can be very complicated to verify the safety of general loads and stores when compiling code for a sandboxed environment, so Cretonne also provides more restricted memory operations that are always safe. -.. inst:: a = load p, Offset, Flags... - - Load from memory at ``p + Offset``. - - This is a polymorphic instruction that can load any value type which has a - memory representation. - - :arg iPtr p: Base address. - :arg Offset: Immediate signed offset. - :flag align(N): Expected alignment of ``p + Offset``. Power of two. - :flag aligntrap: Always trap if the memory access is misaligned. - :result T a: Loaded value. - -.. inst:: store x, p, Offset, Flags... - - Store ``x`` to memory at ``p + Offset``. - - This is a polymorphic instruction that can store any value type with a - memory representation. - - :arg T x: Value to store. - :arg iPtr p: Base address. - :arg Offset: Immediate signed offset. - :flag align(N): Expected alignment of ``p + Offset``. Power of two. - :flag aligntrap: Always trap if the memory access is misaligned. +.. autoinst:: load +.. autoinst:: store Loads and stores are *misaligned* if the resultant address is not a multiple of the expected alignment. Depending on the target architecture, misaligned memory accesses may trap, or they may work. Sometimes, operating systems catch alignment traps and emulate the misaligned memory access. -On target architectures like x86 that don't check alignment, Cretonne expands -the `aligntrap` flag into a conditional trap instruction:: - - v5 = load.i32 v1, 4, align(4), aligntrap - ; Becomes: - v10 = and_imm v1, 3 - trapnz v10 - v5 = load.i32 v1, 4 - Local variables --------------- diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 92842cd140..868a7ba0b1 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -128,3 +128,30 @@ ebb0(v1: i32): ; nextln: $v2 = heap_load.f32 $v1 ; nextln: $v3 = heap_load.f32 $v1+12 ; nextln: heap_store $v3, $v1 + +; Memory access instructions. +function memory(i32) { +ebb0(v1: i32): + v2 = load.i64 v1 + v3 = load.i64 aligned v1 + v4 = load.i64 notrap v1 + v5 = load.i64 notrap aligned v1 + v6 = load.i64 aligned notrap v1 + v7 = load.i64 v1-12 + v8 = load.i64 notrap v1+0x1_0000 + store v2, v1 + store aligned v3, v1+12 + store notrap aligned v3, v1-12 +} +; sameln: function memory(i32) { +; nextln: ebb0($v1: i32): +; nextln: $v2 = load.i64 $v1 +; nextln: $v3 = load.i64 aligned $v1 +; nextln: $v4 = load.i64 notrap $v1 +; nextln: $v5 = load.i64 notrap aligned $v1 +; nextln: $v6 = load.i64 notrap aligned $v1 +; nextln: $v7 = load.i64 $v1-12 +; nextln: $v8 = load.i64 notrap $v1+0x0001_0000 +; nextln: store $v2, $v1 +; nextln: store aligned $v3, $v1+12 +; nextln: store notrap aligned $v3, $v1-12 diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 767be3be7e..acb9ce0131 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -9,7 +9,7 @@ from __future__ import absolute_import from cdsl.formats import InstructionFormat from cdsl.operands import VALUE, VARIABLE_ARGS from .immediates import imm64, uimm8, ieee32, ieee64, offset32, uoffset32 -from .immediates import intcc, floatcc +from .immediates import intcc, floatcc, memflags from .entities import ebb, sig_ref, func_ref, jump_table, stack_slot Nullary = InstructionFormat() @@ -53,6 +53,9 @@ IndirectCall = InstructionFormat( sig_ref, VALUE, VARIABLE_ARGS, multiple_results=True) +Load = InstructionFormat(memflags, VALUE, offset32) +Store = InstructionFormat(memflags, VALUE, VALUE, offset32) + StackLoad = InstructionFormat(stack_slot, offset32) StackStore = InstructionFormat(VALUE, stack_slot, offset32) diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index 7f1662f5c1..2f8803e245 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -90,3 +90,9 @@ floatcc = ImmediateKind( 'ugt': 'UnorderedOrGreaterThan', 'uge': 'UnorderedOrGreaterThanOrEqual', }) + +#: Flags for memory operations like :inst:`load` and :inst:`store`. +memflags = ImmediateKind( + 'memflags', + 'Memory operation flags', + default_member='flags', rust_type='MemFlags') diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index d3de015311..487dca1a89 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -10,7 +10,7 @@ from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup from base.types import i8, f32, f64, b1 from base.immediates import imm64, uimm8, ieee32, ieee64, offset32, uoffset32 -from base.immediates import intcc, floatcc +from base.immediates import intcc, floatcc, memflags from base import entities import base.formats # noqa @@ -211,6 +211,25 @@ x = Operand('x', Mem, doc='Value to be stored') a = Operand('a', Mem, doc='Value loaded') p = Operand('p', iAddr) addr = Operand('addr', iAddr) +Flags = Operand('Flags', memflags) + +load = Instruction( + 'load', r""" + Load from memory at ``p + Offset``. + + This is a polymorphic instruction that can load any value type which + has a memory representation. + """, + ins=(Flags, p, Offset), outs=a) + +store = Instruction( + 'store', r""" + Store ``x`` to memory at ``p + Offset``. + + This is a polymorphic instruction that can store any value type with a + memory representation. + """, + ins=(Flags, x, p, Offset)) stack_load = Instruction( 'stack_load', r""" diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 8b95f20272..dcec1bbb91 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -5,7 +5,8 @@ use ir::types; use ir::{InstructionData, DataFlowGraph, Cursor}; -use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, StackSlot, ValueList}; +use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, StackSlot, ValueList, + MemFlags}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32, Uoffset32}; use ir::condcodes::{IntCC, FloatCC}; diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 5e8d66a09c..f48cbcae6f 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -10,7 +10,7 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::ops::{Deref, DerefMut}; -use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef, StackSlot}; +use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef, StackSlot, MemFlags}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32, Uoffset32}; use ir::condcodes::*; use ir::types; @@ -252,6 +252,20 @@ pub enum InstructionData { args: [Value; 2], offset: Uoffset32, }, + Load { + opcode: Opcode, + ty: Type, + flags: MemFlags, + arg: Value, + offset: Offset32, + }, + Store { + opcode: Opcode, + ty: Type, + flags: MemFlags, + args: [Value; 2], + offset: Offset32, + }, } /// A variable list of `Value` operands used for function call arguments and passing arguments to diff --git a/lib/cretonne/src/ir/memflags.rs b/lib/cretonne/src/ir/memflags.rs new file mode 100644 index 0000000000..8cef887da2 --- /dev/null +++ b/lib/cretonne/src/ir/memflags.rs @@ -0,0 +1,92 @@ +//! Memory operation flags. + +use std::fmt; + +enum FlagBit { + Notrap, + Aligned, +} + +const NAMES: [&'static str; 2] = ["notrap", "aligned"]; + +/// Flags for memory operations like load/store. +/// +/// Each of these flags introduce a limited form of undefined behavior. The flags each enable +/// certain optimizations that need to make additional assumptions. Generally, the semantics of a +/// program does not change when a flag is removed, but adding a flag will. +#[derive(Clone, Copy, Debug)] +pub struct MemFlags { + bits: u8, +} + +impl MemFlags { + /// Create a new empty set of flags. + pub fn new() -> MemFlags { + MemFlags { bits: 0 } + } + + /// Read a flag bit. + fn read(self, bit: FlagBit) -> bool { + self.bits & (1 << bit as usize) != 0 + } + + /// Set a flag bit. + fn set(&mut self, bit: FlagBit) { + self.bits |= 1 << bit as usize + } + + /// Set a flag bit by name. + /// + /// Returns true if the flag was found and set, false for an unknown flag name. + pub fn set_by_name(&mut self, name: &str) -> bool { + match NAMES.iter().position(|&s| s == name) { + Some(bit) => { + self.bits |= 1 << bit; + true + } + None => false, + } + } + + /// Test if the `notrap` flag is set. + /// + /// Normally, trapping is part of the semantics of a load/store operation. If the platform + /// would cause a trap when accessing the effective address, the Cretonne memory operation is + /// also required to trap. + /// + /// The `notrap` flag gives a Cretonne operation permission to not trap. This makes it possible + /// to delete an unused load or a dead store instruction. + pub fn notrap(self) -> bool { + self.read(FlagBit::Notrap) + } + + /// Set the `notrap` flag. + pub fn set_notrap(&mut self) { + self.set(FlagBit::Notrap) + } + + /// Test if the `aligned` flag is set. + /// + /// By default, Cretonne memory instructions work with any unaligned effective address. If the + /// `aligned` flag is set, the instruction is permitted to trap or return a wrong result if the + /// effective address is misaligned. + pub fn aligned(self) -> bool { + self.read(FlagBit::Aligned) + } + + /// Set the `aligned` flag. + pub fn set_aligned(&mut self) { + self.set(FlagBit::Aligned) + } +} + +impl fmt::Display for MemFlags { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + for (i, n) in NAMES.iter().enumerate() { + if self.bits & (1 << i) != 0 { + write!(f, " {}", n)?; + } + } + Ok(()) + } +} diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 302978d81b..488b4dab32 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -10,11 +10,12 @@ pub mod jumptable; pub mod dfg; pub mod layout; pub mod function; -mod funcname; -mod extfunc; mod builder; -mod valueloc; +mod extfunc; +mod funcname; +mod memflags; mod progpoint; +mod valueloc; pub use ir::funcname::FunctionName; pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension, ExtFuncData}; @@ -29,3 +30,4 @@ pub use ir::layout::{Layout, Cursor}; pub use ir::function::Function; pub use ir::builder::InstBuilder; pub use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; +pub use ir::memflags::MemFlags; diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index b1b4429d1a..e6dbf4c86d 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -271,7 +271,9 @@ impl<'a> Verifier<'a> { &IntCompareImm { .. } | &FloatCompare { .. } | &HeapLoad { .. } | - &HeapStore { .. } => {} + &HeapStore { .. } | + &Load { .. } | + &Store { .. } => {} } Ok(()) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index f4b7de5f75..bc9299d796 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -322,6 +322,13 @@ pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result } => write!(w, " {}, {}{}", arg, stack_slot, offset), HeapLoad { arg, offset, .. } => write!(w, " {}{}", arg, offset), HeapStore { args, offset, .. } => write!(w, " {}, {}{}", args[0], args[1], offset), + Load { flags, arg, offset, .. } => write!(w, "{} {}{}", flags, arg, offset), + Store { + flags, + args, + offset, + .. + } => write!(w, "{} {}, {}{}", flags, args[0], args[1], offset), } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index e813404130..a0e1f71719 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -11,7 +11,7 @@ use std::{u16, u32}; use std::mem; use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotData, JumpTable, JumpTableData, Signature, ArgumentType, ArgumentExtension, ExtFuncData, SigRef, - FuncRef, StackSlot, ValueLoc, ArgumentLoc}; + FuncRef, StackSlot, ValueLoc, ArgumentLoc, MemFlags}; use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Offset32, Uoffset32, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; @@ -228,7 +228,8 @@ impl<'a> Context<'a> { InstructionData::Unary { ref mut arg, .. } | InstructionData::UnarySplit { ref mut arg, .. } | InstructionData::StackStore { ref mut arg, .. } | - InstructionData::HeapLoad { ref mut arg, .. } => { + InstructionData::HeapLoad { ref mut arg, .. } | + InstructionData::Load { ref mut arg, .. } => { self.map.rewrite_value(arg, loc)?; } @@ -238,7 +239,8 @@ impl<'a> Context<'a> { InstructionData::InsertLane { ref mut args, .. } | InstructionData::IntCompare { ref mut args, .. } | InstructionData::FloatCompare { ref mut args, .. } | - InstructionData::HeapStore { ref mut args, .. } => { + InstructionData::HeapStore { ref mut args, .. } | + InstructionData::Store { ref mut args, .. } => { self.map.rewrite_values(args, loc)?; } @@ -576,6 +578,19 @@ impl<'a> Parser<'a> { } } + // Match and a consume a possibly empty sequence of memory operation flags. + fn optional_memflags(&mut self) -> MemFlags { + let mut flags = MemFlags::new(); + while let Some(Token::Identifier(text)) = self.token() { + if flags.set_by_name(text) { + self.consume(); + } else { + break; + } + } + flags + } + // Match and consume an identifier. fn match_any_identifier(&mut self, err_msg: &str) -> Result<&'a str> { if let Some(Token::Identifier(text)) = self.token() { @@ -1735,6 +1750,32 @@ impl<'a> Parser<'a> { offset: offset, } } + InstructionFormat::Load => { + let flags = self.optional_memflags(); + let addr = self.match_value("expected SSA value address")?; + let offset = self.optional_offset32()?; + InstructionData::Load { + opcode: opcode, + ty: VOID, + flags: flags, + arg: addr, + offset: offset, + } + } + InstructionFormat::Store => { + let flags = self.optional_memflags(); + let arg = self.match_value("expected SSA value operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let addr = self.match_value("expected SSA value address")?; + let offset = self.optional_offset32()?; + InstructionData::Store { + opcode: opcode, + ty: VOID, + flags: flags, + args: [arg, addr], + offset: offset, + } + } }; Ok(idata) } From b4ac52033297a020ba067884c286ebc86f27c453 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 11 Apr 2017 10:30:03 -0700 Subject: [PATCH 0677/3084] Extending loads and truncating stores --- cranelift/docs/langref.rst | 20 ++++++ lib/cretonne/meta/base/immediates.py | 2 +- lib/cretonne/meta/base/instructions.py | 93 ++++++++++++++++++++++++++ 3 files changed, 114 insertions(+), 1 deletion(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 12949ab691..d4b30780b4 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -437,6 +437,26 @@ accesses may trap, or they may work. Sometimes, operating systems catch alignment traps and emulate the misaligned memory access. +Extending loads and truncating stores +------------------------------------- + +Most ISAs provide instructions that load an integer value smaller than a register +and extends it to the width of the register. Similarly, store instructions that +only write the low bits of an integer register are common. + +Cretonne provides extending loads and truncation stores for 8, 16, and 32-bit +memory accesses. + +.. autoinst:: uload8 +.. autoinst:: sload8 +.. autoinst:: istore8 +.. autoinst:: uload16 +.. autoinst:: sload16 +.. autoinst:: istore16 +.. autoinst:: uload32 +.. autoinst:: sload32 +.. autoinst:: istore32 + Local variables --------------- diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index 2f8803e245..8e643bb9ee 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -91,7 +91,7 @@ floatcc = ImmediateKind( 'uge': 'UnorderedOrGreaterThanOrEqual', }) -#: Flags for memory operations like :inst:`load` and :inst:`store`. +#: Flags for memory operations like :cton:inst:`load` and :cton:inst:`store`. memflags = ImmediateKind( 'memflags', 'Memory operation flags', diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 487dca1a89..a7c549ce93 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -231,6 +231,99 @@ store = Instruction( """, ins=(Flags, x, p, Offset)) +iExt8 = TypeVar( + 'iExt8', 'An integer type with more than 8 bits', + ints=(16, 64)) +x = Operand('x', iExt8) +a = Operand('a', iExt8) + +uload8 = Instruction( + 'uload8', r""" + Load 8 bits from memory at ``p + Offset`` and zero-extend. + + This is equivalent to ``load.i8`` followed by ``uextend``. + """, + ins=(Flags, p, Offset), outs=a) + +sload8 = Instruction( + 'sload8', r""" + Load 8 bits from memory at ``p + Offset`` and sign-extend. + + This is equivalent to ``load.i8`` followed by ``uextend``. + """, + ins=(Flags, p, Offset), outs=a) + +istore8 = Instruction( + 'istore8', r""" + Store the low 8 bits of ``x`` to memory at ``p + Offset``. + + This is equivalent to ``ireduce.i8`` followed by ``store.i8``. + """, + ins=(Flags, x, p, Offset)) + +iExt16 = TypeVar( + 'iExt16', 'An integer type with more than 16 bits', + ints=(32, 64)) +x = Operand('x', iExt16) +a = Operand('a', iExt16) + +uload16 = Instruction( + 'uload16', r""" + Load 16 bits from memory at ``p + Offset`` and zero-extend. + + This is equivalent to ``load.i16`` followed by ``uextend``. + """, + ins=(Flags, p, Offset), outs=a) + +sload16 = Instruction( + 'sload16', r""" + Load 16 bits from memory at ``p + Offset`` and sign-extend. + + This is equivalent to ``load.i16`` followed by ``uextend``. + """, + ins=(Flags, p, Offset), outs=a) + +istore16 = Instruction( + 'istore16', r""" + Store the low 16 bits of ``x`` to memory at ``p + Offset``. + + This is equivalent to ``ireduce.i16`` followed by ``store.i8``. + """, + ins=(Flags, x, p, Offset)) + +iExt32 = TypeVar( + 'iExt32', 'An integer type with more than 32 bits', + ints=(64, 64)) +x = Operand('x', iExt32) +a = Operand('a', iExt32) + +uload32 = Instruction( + 'uload32', r""" + Load 32 bits from memory at ``p + Offset`` and zero-extend. + + This is equivalent to ``load.i32`` followed by ``uextend``. + """, + ins=(Flags, p, Offset), outs=a) + +sload32 = Instruction( + 'sload32', r""" + Load 32 bits from memory at ``p + Offset`` and sign-extend. + + This is equivalent to ``load.i32`` followed by ``uextend``. + """, + ins=(Flags, p, Offset), outs=a) + +istore32 = Instruction( + 'istore32', r""" + Store the low 32 bits of ``x`` to memory at ``p + Offset``. + + This is equivalent to ``ireduce.i32`` followed by ``store.i8``. + """, + ins=(Flags, x, p, Offset)) + +x = Operand('x', Mem, doc='Value to be stored') +a = Operand('a', Mem, doc='Value loaded') + stack_load = Instruction( 'stack_load', r""" Load a value from a stack slot at the constant offset. From 4ee519e62032b8083c680b2b6fccd0a4c510abfb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 6 Apr 2017 15:36:03 -0700 Subject: [PATCH 0678/3084] Add RISC-V call instruction encodings. Calls are jal with a fixed %x1 link register. --- cranelift/filetests/isa/riscv/binary32.cton | 9 +++++++-- lib/cretonne/meta/isa/riscv/encodings.py | 5 ++++- lib/cretonne/meta/isa/riscv/recipes.py | 3 ++- lib/cretonne/src/isa/riscv/binemit.rs | 17 +++++++++++++---- 4 files changed, 26 insertions(+), 8 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 89771d8165..984b7b8a80 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -3,6 +3,8 @@ test binemit isa riscv function RV32I() { + fn0 = function foo() + ebb0: [-,%x10] v1 = iconst.i32 1 [-,%x21] v2 = iconst.i32 2 @@ -75,11 +77,14 @@ ebb0: [-,%x7] v140 = iconst.i32 0x12345000 ; bin: 123453b7 [-,%x16] v141 = iconst.i32 0xffffffff_fedcb000 ; bin: fedcb837 + ; Control Transfer Instructions + + ; jal %x1, fn0 + call fn0() ; bin: Call(fn0) 000000ef + brz v1, ebb3 fallthrough ebb1 - ; Control Transfer Instructions - ebb1: ; beq 0x000 br_icmp eq v1, v2, ebb1 ; bin: 01550063 diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index 21ff09e177..294015b4ea 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -6,7 +6,8 @@ from base import instructions as base from base.immediates import intcc from .defs import RV32, RV64 from .recipes import OPIMM, OPIMM32, OP, OP32, LUI, BRANCH, JALR, JAL -from .recipes import R, Rshamt, Ricmp, I, Iicmp, Iret, U, UJ, SB, SBzero +from .recipes import R, Rshamt, Ricmp, I, Iicmp, Iret +from .recipes import U, UJ, UJcall, SB, SBzero from .settings import use_m from cdsl.ast import Var @@ -84,6 +85,8 @@ RV64.enc(base.imul.i32, R, OP32(0b000, 0b0000001), isap=use_m) # Unconditional branches. RV32.enc(base.jump, UJ, JAL()) RV64.enc(base.jump, UJ, JAL()) +RV32.enc(base.call, UJcall, JAL()) +RV64.enc(base.call, UJcall, JAL()) # Conditional branches. for cond, f3 in [ diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index e30922613f..ad682fa74b 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -12,7 +12,7 @@ from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt from base.formats import Binary, BinaryImm, MultiAry, IntCompare, IntCompareImm -from base.formats import UnaryImm, BranchIcmp, Branch, Jump +from base.formats import UnaryImm, BranchIcmp, Branch, Jump, Call from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit @@ -119,6 +119,7 @@ U = EncRecipe( # UJ-type unconditional branch instructions. UJ = EncRecipe('UJ', Jump, size=4, ins=(), outs=(), branch_range=(0, 21)) +UJcall = EncRecipe('UJcall', Call, size=4, ins=(), outs=()) # SB-type branch instructions. # TODO: These instructions have a +/- 4 KB branch range. How to encode that diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index e4cc5c05c6..94ba8b53bf 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -8,13 +8,12 @@ use predicates::is_signed_int; include!(concat!(env!("OUT_DIR"), "/binemit-riscv.rs")); /// RISC-V relocation kinds. -#[allow(dead_code)] pub enum RelocKind { - /// A conditional (SB-type) branch to an EBB. - Branch, + /// A jal call to a function. + Call, } -pub static RELOC_NAMES: [&'static str; 1] = ["Branch"]; +pub static RELOC_NAMES: [&'static str; 1] = ["Call"]; impl Into for RelocKind { fn into(self) -> Reloc { @@ -318,3 +317,13 @@ fn recipe_uj(func: &Function, inst: Inst, sink: &mut CS) panic!("Expected Jump format: {:?}", func.dfg[inst]); } } + +fn recipe_ujcall(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Call { func_ref, .. } = func.dfg[inst] { + sink.reloc_func(RelocKind::Call.into(), func_ref); + // rd=%x1 is the standard link register. + put_uj(func.encodings[inst].bits(), 0, 1, sink); + } else { + panic!("Expected Call format: {:?}", func.dfg[inst]); + } +} From dd0a61cc9127098b559a9d4b7877cfd3d0f41963 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 11 Apr 2017 11:36:50 -0700 Subject: [PATCH 0679/3084] Keep EBB arguments in a ValueList. This is the first step of the value list refactoring which will replace linked lists of values with value lists. - Keep a ValueList in the EbbData struct containing all the EBB arguments. - Change dfg.ebb_args() to return a slice instead of an iterator. This leaves us in a temporary hybrid state where we maintain both a linked list and a ValueList vector of the EBB arguments. --- lib/cretonne/src/entity_list.rs | 5 ++ lib/cretonne/src/ir/dfg.rs | 63 ++++++++----------- .../src/regalloc/live_value_tracker.rs | 2 +- lib/cretonne/src/verifier.rs | 7 ++- lib/cretonne/src/write.rs | 2 +- lib/reader/src/parser.rs | 9 ++- 6 files changed, 40 insertions(+), 48 deletions(-) diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index 0b90265470..d6f0e49119 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -222,6 +222,11 @@ impl ListPool { } impl EntityList { + /// Create a new empty list. + pub fn new() -> Self { + Default::default() + } + /// Returns `true` if the list has a length of 0. pub fn is_empty(&self) -> bool { // 0 is a magic value for the empty list. Any list in the pool array must have a positive diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 22ce7a116d..ffa842e5a5 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -1,6 +1,6 @@ //! Data flow graph tracking Instructions, Values, and EBBs. -use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueListPool}; +use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueList, ValueListPool}; use ir::entities::ExpandedValue; use ir::instructions::{Opcode, InstructionData, CallInfo}; use ir::extfunc::ExtFuncData; @@ -687,17 +687,7 @@ impl DataFlowGraph { /// Get the number of arguments on `ebb`. pub fn num_ebb_args(&self, ebb: Ebb) -> usize { - match self.ebbs[ebb].last_arg.expand() { - None => 0, - Some(last_arg) => { - if let ExpandedValue::Table(idx) = last_arg.expand() { - if let ValueData::Arg { num, .. } = self.extended_values[idx] { - return num as usize + 1; - } - } - panic!("inconsistent value table entry for EBB argument"); - } - } + self.ebbs[ebb].args.len(&self.value_lists) } /// Append an argument with type `ty` to `ebb`. @@ -712,12 +702,9 @@ impl DataFlowGraph { val } - /// Iterate through the arguments to an EBB. - pub fn ebb_args(&self, ebb: Ebb) -> Values { - Values { - dfg: self, - cur: self.ebbs[ebb].first_arg.into(), - } + /// Get the arguments to an EBB. + pub fn ebb_args(&self, ebb: Ebb) -> &[Value] { + self.ebbs[ebb].args.as_slice(&self.value_lists) } /// Replace an EBB argument with a new value of type `ty`. @@ -752,6 +739,8 @@ impl DataFlowGraph { old_data); }; + self.ebbs[ebb].args.as_mut_slice(&mut self.value_lists)[num as usize] = new_arg; + // Now fix up the linked lists. if self.ebbs[ebb].last_arg.expand() == Some(old_arg) { self.ebbs[ebb].last_arg = new_arg.into(); @@ -804,6 +793,7 @@ impl DataFlowGraph { let first = self.ebbs[ebb].first_arg.into(); self.ebbs[ebb].first_arg = None.into(); self.ebbs[ebb].last_arg = None.into(); + self.ebbs[ebb].args.clear(&mut self.value_lists); first } @@ -815,6 +805,7 @@ impl DataFlowGraph { /// /// In almost all cases, you should be using `append_ebb_arg()` instead of this method. pub fn attach_ebb_arg(&mut self, ebb: Ebb, arg: Value) { + self.ebbs[ebb].args.push(arg, &mut self.value_lists); let arg_num = match self.ebbs[ebb].last_arg.map(|v| v.expand()) { // If last_argument is `None`, we're adding the first EBB argument. None => { @@ -861,6 +852,9 @@ impl DataFlowGraph { // match the function arguments. #[derive(Clone)] struct EbbData { + // List of arguments to this EBB. + args: ValueList, + // First argument to this EBB, or `None` if the block has no arguments. // // The arguments are all `ValueData::Argument` entries that form a linked list from `first_arg` @@ -874,6 +868,7 @@ struct EbbData { impl EbbData { fn new() -> EbbData { EbbData { + args: ValueList::new(), first_arg: None.into(), last_arg: None.into(), } @@ -966,28 +961,20 @@ mod tests { let ebb = dfg.make_ebb(); assert_eq!(ebb.to_string(), "ebb0"); assert_eq!(dfg.num_ebb_args(ebb), 0); - assert_eq!(dfg.ebb_args(ebb).next(), None); + assert_eq!(dfg.ebb_args(ebb), &[]); assert_eq!(dfg.detach_ebb_args(ebb), None); assert_eq!(dfg.num_ebb_args(ebb), 0); - assert_eq!(dfg.ebb_args(ebb).next(), None); + assert_eq!(dfg.ebb_args(ebb), &[]); let arg1 = dfg.append_ebb_arg(ebb, types::F32); assert_eq!(arg1.to_string(), "vx0"); assert_eq!(dfg.num_ebb_args(ebb), 1); - { - let mut args1 = dfg.ebb_args(ebb); - assert_eq!(args1.next(), Some(arg1)); - assert_eq!(args1.next(), None); - } + assert_eq!(dfg.ebb_args(ebb), &[arg1]); + let arg2 = dfg.append_ebb_arg(ebb, types::I16); assert_eq!(arg2.to_string(), "vx1"); assert_eq!(dfg.num_ebb_args(ebb), 2); - { - let mut args2 = dfg.ebb_args(ebb); - assert_eq!(args2.next(), Some(arg1)); - assert_eq!(args2.next(), Some(arg2)); - assert_eq!(args2.next(), None); - } + assert_eq!(dfg.ebb_args(ebb), &[arg1, arg2]); assert_eq!(dfg.value_def(arg1), ValueDef::Arg(ebb, 0)); assert_eq!(dfg.value_def(arg2), ValueDef::Arg(ebb, 1)); @@ -997,7 +984,7 @@ mod tests { // Swap the two EBB arguments. let take1 = dfg.detach_ebb_args(ebb).unwrap(); assert_eq!(dfg.num_ebb_args(ebb), 0); - assert_eq!(dfg.ebb_args(ebb).next(), None); + assert_eq!(dfg.ebb_args(ebb), &[]); let take2 = dfg.next_ebb_arg(take1).unwrap(); assert_eq!(take1, arg1); assert_eq!(take2, arg2); @@ -1005,7 +992,7 @@ mod tests { dfg.attach_ebb_arg(ebb, take2); let arg3 = dfg.append_ebb_arg(ebb, types::I32); dfg.attach_ebb_arg(ebb, take1); - assert_eq!(dfg.ebb_args(ebb).collect::>(), [take2, arg3, take1]); + assert_eq!(dfg.ebb_args(ebb), &[take2, arg3, take1]); } #[test] @@ -1018,24 +1005,24 @@ mod tests { let new1 = dfg.replace_ebb_arg(arg1, types::I64); assert_eq!(dfg.value_type(arg1), types::F32); assert_eq!(dfg.value_type(new1), types::I64); - assert_eq!(dfg.ebb_args(ebb).collect::>(), [new1]); + assert_eq!(dfg.ebb_args(ebb), &[new1]); dfg.attach_ebb_arg(ebb, arg1); - assert_eq!(dfg.ebb_args(ebb).collect::>(), [new1, arg1]); + assert_eq!(dfg.ebb_args(ebb), &[new1, arg1]); let new2 = dfg.replace_ebb_arg(arg1, types::I8); assert_eq!(dfg.value_type(arg1), types::F32); assert_eq!(dfg.value_type(new2), types::I8); - assert_eq!(dfg.ebb_args(ebb).collect::>(), [new1, new2]); + assert_eq!(dfg.ebb_args(ebb), &[new1, new2]); dfg.attach_ebb_arg(ebb, arg1); - assert_eq!(dfg.ebb_args(ebb).collect::>(), [new1, new2, arg1]); + assert_eq!(dfg.ebb_args(ebb), &[new1, new2, arg1]); let new3 = dfg.replace_ebb_arg(new2, types::I16); assert_eq!(dfg.value_type(new1), types::I64); assert_eq!(dfg.value_type(new2), types::I8); assert_eq!(dfg.value_type(new3), types::I16); - assert_eq!(dfg.ebb_args(ebb).collect::>(), [new1, new3, arg1]); + assert_eq!(dfg.ebb_args(ebb), &[new1, new3, arg1]); } #[test] diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 75b18ef019..458fe14e26 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -182,7 +182,7 @@ impl LiveValueTracker { // Now add all the live arguments to `ebb`. let first_arg = self.live.values.len(); - for value in dfg.ebb_args(ebb) { + for &value in dfg.ebb_args(ebb) { let lr = liveness .get(value) .expect("EBB argument value has no live range"); diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index e6dbf4c86d..371ada0440 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -151,7 +151,7 @@ impl<'a> Verifier<'a> { } // Arguments belong to the correct ebb. - for arg in self.func.dfg.ebb_args(ebb) { + for &arg in self.func.dfg.ebb_args(ebb) { match self.func.dfg.value_def(arg) { ValueDef::Arg(arg_ebb, _) => { if ebb != arg_ebb { @@ -405,7 +405,7 @@ impl<'a> Verifier<'a> { return err!(ebb, "entry block arguments must match function signature"); } - for (i, arg) in self.func.dfg.ebb_args(ebb).enumerate() { + for (i, &arg) in self.func.dfg.ebb_args(ebb).iter().enumerate() { let arg_type = self.func.dfg.value_type(arg); if arg_type != expected_types[i].value_type { return err!(ebb, @@ -510,7 +510,8 @@ impl<'a> Verifier<'a> { let iter = self.func .dfg .ebb_args(ebb) - .map(|v| self.func.dfg.value_type(v)); + .iter() + .map(|&v| self.func.dfg.value_type(v)); self.typecheck_variable_args_iterator(inst, iter)?; } BranchInfo::Table(table) => { diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index bc9299d796..09a4410f43 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -95,7 +95,7 @@ pub fn write_ebb_header(w: &mut Write, func: &Function, ebb: Ebb) -> Result { write!(w, " ")?; } - let mut args = func.dfg.ebb_args(ebb); + let mut args = func.dfg.ebb_args(ebb).iter().cloned(); match args.next() { None => return writeln!(w, "{}:", ebb), Some(arg) => { diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index a0e1f71719..a0302a1a30 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1895,13 +1895,12 @@ mod tests { let mut ebbs = func.layout.ebbs(); let ebb0 = ebbs.next().unwrap(); - assert_eq!(func.dfg.ebb_args(ebb0).next(), None); + assert_eq!(func.dfg.ebb_args(ebb0), &[]); let ebb4 = ebbs.next().unwrap(); - let mut ebb4_args = func.dfg.ebb_args(ebb4); - let arg0 = ebb4_args.next().unwrap(); - assert_eq!(func.dfg.value_type(arg0), types::I32); - assert_eq!(ebb4_args.next(), None); + let ebb4_args = func.dfg.ebb_args(ebb4); + assert_eq!(ebb4_args.len(), 1); + assert_eq!(func.dfg.value_type(ebb4_args[0]), types::I32); } #[test] From ca900d6bf88281f944c04f8bf7d82c4004518f87 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 11 Apr 2017 12:14:33 -0700 Subject: [PATCH 0680/3084] Return the whole value list from detach_ebb_args(). Rather than returning the head of a linked list of EBB arguments, just return the whole value list of all the arguments. Delete the next_ebb_arg() method which was only used for traversing that list. --- lib/cretonne/src/ir/dfg.rs | 42 ++++++-------------------- lib/cretonne/src/legalizer/boundary.rs | 8 ++--- 2 files changed, 14 insertions(+), 36 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index ffa842e5a5..45485d63e2 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -751,10 +751,7 @@ impl DataFlowGraph { self.ebbs[ebb].first_arg = new_arg.into(); } else { // We need to find the num-1 argument value and change its next link. - let mut arg = self.ebbs[ebb].first_arg.expect("EBB has no arguments"); - for _ in 1..num { - arg = self.next_ebb_arg(arg).expect("Too few EBB arguments"); - } + let arg = self.ebbs[ebb].args.as_slice(&self.value_lists)[(num - 1) as usize]; if let ExpandedValue::Table(index) = arg.expand() { if let ValueData::Arg { ref mut next, .. } = self.extended_values[index] { assert_eq!(next.expand(), Some(old_arg)); @@ -770,31 +767,15 @@ impl DataFlowGraph { new_arg } - /// Given a value that is known to be an EBB argument, return the next EBB argument. - pub fn next_ebb_arg(&self, arg: Value) -> Option { - if let ExpandedValue::Table(index) = arg.expand() { - if let ValueData::Arg { next, .. } = self.extended_values[index] { - return next.into(); - } - } - panic!("{} is not an EBB argument", arg); - } - - /// Detach all the arguments from `ebb` and return the first one. - /// - /// The whole list of detached arguments can be traversed with `next_ebb_arg`. This method does - /// not return a `Values` iterator since it is often necessary to mutate the DFG while - /// processing the list of arguments. + /// Detach all the arguments from `ebb` and return them as a `ValueList`. /// /// This is a quite low-level operation. Sensible things to do with the detached EBB arguments /// is to put them back on the same EBB with `attach_ebb_arg()` or change them into aliases /// with `change_to_alias()`. - pub fn detach_ebb_args(&mut self, ebb: Ebb) -> Option { - let first = self.ebbs[ebb].first_arg.into(); + pub fn detach_ebb_args(&mut self, ebb: Ebb) -> ValueList { self.ebbs[ebb].first_arg = None.into(); self.ebbs[ebb].last_arg = None.into(); - self.ebbs[ebb].args.clear(&mut self.value_lists); - first + self.ebbs[ebb].args.take() } /// Append an existing argument value to `ebb`. @@ -962,7 +943,7 @@ mod tests { assert_eq!(ebb.to_string(), "ebb0"); assert_eq!(dfg.num_ebb_args(ebb), 0); assert_eq!(dfg.ebb_args(ebb), &[]); - assert_eq!(dfg.detach_ebb_args(ebb), None); + assert!(dfg.detach_ebb_args(ebb).is_empty()); assert_eq!(dfg.num_ebb_args(ebb), 0); assert_eq!(dfg.ebb_args(ebb), &[]); @@ -982,17 +963,14 @@ mod tests { assert_eq!(dfg.value_type(arg2), types::I16); // Swap the two EBB arguments. - let take1 = dfg.detach_ebb_args(ebb).unwrap(); + let vlist = dfg.detach_ebb_args(ebb); assert_eq!(dfg.num_ebb_args(ebb), 0); assert_eq!(dfg.ebb_args(ebb), &[]); - let take2 = dfg.next_ebb_arg(take1).unwrap(); - assert_eq!(take1, arg1); - assert_eq!(take2, arg2); - assert_eq!(dfg.next_ebb_arg(take2), None); - dfg.attach_ebb_arg(ebb, take2); + assert_eq!(vlist.as_slice(&dfg.value_lists), &[arg1, arg2]); + dfg.attach_ebb_arg(ebb, arg2); let arg3 = dfg.append_ebb_arg(ebb, types::I32); - dfg.attach_ebb_arg(ebb, take1); - assert_eq!(dfg.ebb_args(ebb), &[take2, arg3, take1]); + dfg.attach_ebb_arg(ebb, arg1); + assert_eq!(dfg.ebb_args(ebb), &[arg2, arg3, arg1]); } #[test] diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 19e06b7ba4..42386cab83 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -63,10 +63,10 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { // Process the EBB arguments one at a time, possibly replacing one argument with multiple new // ones. We do this by detaching the entry EBB arguments first. - let mut next_arg = func.dfg.detach_ebb_args(entry); - while let Some(arg) = next_arg { - // Get the next argument before we mutate `arg`. - next_arg = func.dfg.next_ebb_arg(arg); + let ebb_args = func.dfg.detach_ebb_args(entry); + let mut old_arg = 0; + while let Some(arg) = ebb_args.get(old_arg, &func.dfg.value_lists) { + old_arg += 1; let arg_type = func.dfg.value_type(arg); if arg_type == abi_types[abi_arg].value_type { From 5209778ae2d419bd00c9b15c6001af7d30d9b558 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 11 Apr 2017 12:30:37 -0700 Subject: [PATCH 0681/3084] Stop maintaining a linked list of EBB arguments. Now that we have a value list of the arguments, we can get rid of: - The first_arg and last_arg members in EbbData, - The next member in the ValueData::Arg variant. --- lib/cretonne/src/entity_list.rs | 13 +++-- lib/cretonne/src/ir/dfg.rs | 95 ++++++--------------------------- 2 files changed, 23 insertions(+), 85 deletions(-) diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index d6f0e49119..7e11019529 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -295,7 +295,8 @@ impl EntityList { } /// Appends an element to the back of the list. - pub fn push(&mut self, element: T, pool: &mut ListPool) { + /// Returns the index where the element was inserted. + pub fn push(&mut self, element: T, pool: &mut ListPool) -> usize { let idx = self.index as usize; match pool.len_of(self) { None => { @@ -305,6 +306,7 @@ impl EntityList { pool.data[block] = T::new(1); pool.data[block + 1] = element; self.index = (block + 1) as u32; + 0 } Some(len) => { // Do we need to reallocate? @@ -320,6 +322,7 @@ impl EntityList { } pool.data[block + new_len] = element; pool.data[block] = T::new(new_len); + len } } } @@ -516,14 +519,14 @@ mod tests { let i3 = Inst::new(3); let i4 = Inst::new(4); - list.push(i1, pool); + assert_eq!(list.push(i1, pool), 0); assert_eq!(list.len(pool), 1); assert!(!list.is_empty()); assert_eq!(list.as_slice(pool), &[i1]); assert_eq!(list.get(0, pool), Some(i1)); assert_eq!(list.get(1, pool), None); - list.push(i2, pool); + assert_eq!(list.push(i2, pool), 1); assert_eq!(list.len(pool), 2); assert!(!list.is_empty()); assert_eq!(list.as_slice(pool), &[i1, i2]); @@ -531,7 +534,7 @@ mod tests { assert_eq!(list.get(1, pool), Some(i2)); assert_eq!(list.get(2, pool), None); - list.push(i3, pool); + assert_eq!(list.push(i3, pool), 2); assert_eq!(list.len(pool), 3); assert!(!list.is_empty()); assert_eq!(list.as_slice(pool), &[i1, i2, i3]); @@ -541,7 +544,7 @@ mod tests { assert_eq!(list.get(3, pool), None); // This triggers a reallocation. - list.push(i4, pool); + assert_eq!(list.push(i4, pool), 3); assert_eq!(list.len(pool), 4); assert!(!list.is_empty()); assert_eq!(list.as_slice(pool), &[i1, i2, i3, i4]); diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 45485d63e2..9178aeeb38 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -276,7 +276,6 @@ enum ValueData { ty: Type, num: u16, // Argument number, starting from 0. ebb: Ebb, - next: PackedOption, // Next argument to `ebb`. }, // Value is an alias of another value. @@ -308,7 +307,9 @@ impl<'a> Iterator for Values<'a> { ExpandedValue::Table(index) => { match self.dfg.extended_values[index] { ValueData::Inst { next, .. } => next.into(), - ValueData::Arg { next, .. } => next.into(), + ValueData::Arg { .. } => { + panic!("EBB argument {} appeared in value list", prev) + } ValueData::Alias { .. } => { panic!("Alias value {} appeared in value list", prev) } @@ -696,7 +697,6 @@ impl DataFlowGraph { ty: ty, ebb: ebb, num: 0, - next: None.into(), }); self.attach_ebb_arg(ebb, val); val @@ -724,46 +724,20 @@ impl DataFlowGraph { }; // Create new value identical to the old one except for the type. - let (ebb, num, new_arg) = if let ValueData::Arg { num, ebb, next, .. } = old_data { - (ebb, - num, - self.make_value(ValueData::Arg { - ty: new_type, - num: num, - ebb: ebb, - next: next, - })) + let (ebb, num) = if let ValueData::Arg { num, ebb, .. } = old_data { + (ebb, num) } else { panic!("old_arg: {} must be an EBB argument: {:?}", old_arg, old_data); }; + let new_arg = self.make_value(ValueData::Arg { + ty: new_type, + num: num, + ebb: ebb, + }); self.ebbs[ebb].args.as_mut_slice(&mut self.value_lists)[num as usize] = new_arg; - - // Now fix up the linked lists. - if self.ebbs[ebb].last_arg.expand() == Some(old_arg) { - self.ebbs[ebb].last_arg = new_arg.into(); - } - - if self.ebbs[ebb].first_arg.expand() == Some(old_arg) { - assert_eq!(num, 0); - self.ebbs[ebb].first_arg = new_arg.into(); - } else { - // We need to find the num-1 argument value and change its next link. - let arg = self.ebbs[ebb].args.as_slice(&self.value_lists)[(num - 1) as usize]; - if let ExpandedValue::Table(index) = arg.expand() { - if let ValueData::Arg { ref mut next, .. } = self.extended_values[index] { - assert_eq!(next.expand(), Some(old_arg)); - *next = new_arg.into(); - } else { - panic!("{} is not an EBB argument", arg); - } - } else { - panic!("{} is not an EBB argument", arg); - } - } - new_arg } @@ -773,8 +747,6 @@ impl DataFlowGraph { /// is to put them back on the same EBB with `attach_ebb_arg()` or change them into aliases /// with `change_to_alias()`. pub fn detach_ebb_args(&mut self, ebb: Ebb) -> ValueList { - self.ebbs[ebb].first_arg = None.into(); - self.ebbs[ebb].last_arg = None.into(); self.ebbs[ebb].args.take() } @@ -786,38 +758,14 @@ impl DataFlowGraph { /// /// In almost all cases, you should be using `append_ebb_arg()` instead of this method. pub fn attach_ebb_arg(&mut self, ebb: Ebb, arg: Value) { - self.ebbs[ebb].args.push(arg, &mut self.value_lists); - let arg_num = match self.ebbs[ebb].last_arg.map(|v| v.expand()) { - // If last_argument is `None`, we're adding the first EBB argument. - None => { - self.ebbs[ebb].first_arg = arg.into(); - 0 - } - // Append to the linked list of arguments. - Some(ExpandedValue::Table(idx)) => { - if let ValueData::Arg { ref mut next, num, .. } = self.extended_values[idx] { - *next = arg.into(); - assert!(num < u16::MAX, "Too many arguments to EBB"); - num + 1 - } else { - panic!("inconsistent value table entry for EBB argument"); - } - } - _ => panic!("inconsistent value table entry for EBB argument"), - }; - self.ebbs[ebb].last_arg = arg.into(); + let arg_num = self.ebbs[ebb].args.push(arg, &mut self.value_lists); + assert!(arg_num <= u16::MAX as usize, "Too many arguments to EBB"); // Now update `arg` itself. let arg_ebb = ebb; if let ExpandedValue::Table(idx) = arg.expand() { - if let ValueData::Arg { - ref mut num, - ebb, - ref mut next, - .. - } = self.extended_values[idx] { - *num = arg_num; - *next = None.into(); + if let ValueData::Arg { ref mut num, ebb, .. } = self.extended_values[idx] { + *num = arg_num as u16; assert_eq!(arg_ebb, ebb, "{} should already belong to EBB", arg); return; } @@ -835,24 +783,11 @@ impl DataFlowGraph { struct EbbData { // List of arguments to this EBB. args: ValueList, - - // First argument to this EBB, or `None` if the block has no arguments. - // - // The arguments are all `ValueData::Argument` entries that form a linked list from `first_arg` - // to `last_arg`. - first_arg: PackedOption, - - // Last argument to this EBB, or `None` if the block has no arguments. - last_arg: PackedOption, } impl EbbData { fn new() -> EbbData { - EbbData { - args: ValueList::new(), - first_arg: None.into(), - last_arg: None.into(), - } + EbbData { args: ValueList::new() } } } From fe0d986110c6f5121ab604eeb1f4826c92fd4cdc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 11 Apr 2017 15:03:43 -0700 Subject: [PATCH 0682/3084] Make the dfg.insts table private again. - Add a dfg.is_inst_valid() method for the verifier. - Use the inst_args_mut() method when rewriting values in the parser. - Add a new branch_destination_mut() to use when rewriting EBBs. This also gets rid of one of the large instruction format switches in the parser. --- lib/cretonne/src/ir/dfg.rs | 7 ++- lib/cretonne/src/ir/instructions.rs | 13 ++++++ lib/cretonne/src/verifier.rs | 2 +- lib/reader/src/parser.rs | 69 ++--------------------------- 4 files changed, 24 insertions(+), 67 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 9178aeeb38..63ae3a5e97 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -26,7 +26,7 @@ pub struct DataFlowGraph { /// Data about all of the instructions in the function, including opcodes and operands. /// The instructions in this map are not in program order. That is tracked by `Layout`, along /// with the EBB containing each instruction. - pub insts: EntityMap, + insts: EntityMap, /// Memory pool of value lists referenced by instructions in `insts`. pub value_lists: ValueListPool, @@ -77,6 +77,11 @@ impl DataFlowGraph { self.insts.len() } + /// Returns `true` if the given instruction reference is valid. + pub fn inst_is_valid(&self, inst: Inst) -> bool { + self.insts.is_valid(inst) + } + /// Get the total number of extended basic blocks created in this function, whether they are /// currently inserted in the layout or not. /// diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index f48cbcae6f..626c130740 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -376,6 +376,19 @@ impl InstructionData { } } + /// Get a mutable reference to the single destination of this branch instruction, if it is a + /// single destination branch or jump. + /// + /// Multi-destination branches like `br_table` return `None`. + pub fn branch_destination_mut(&mut self) -> Option<&mut Ebb> { + match *self { + InstructionData::Jump { ref mut destination, .. } => Some(destination), + InstructionData::Branch { ref mut destination, .. } => Some(destination), + InstructionData::BranchIcmp { ref mut destination, .. } => Some(destination), + _ => None, + } + } + /// Return information about a call instruction. /// /// Any instruction that can call another function reveals its call signature here. diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 371ada0440..ace83235df 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -337,7 +337,7 @@ impl<'a> Verifier<'a> { match dfg.value_def(v) { ValueDef::Res(def_inst, _) => { // Value is defined by an instruction that exists. - if !dfg.insts.is_valid(def_inst) { + if !dfg.inst_is_valid(def_inst) { return err!(loc_inst, "{} is defined by invalid instruction {}", v, diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index a0302a1a30..a5c3b4244c 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -213,71 +213,10 @@ impl<'a> Context<'a> { for ebb in self.function.layout.ebbs() { for inst in self.function.layout.ebb_insts(ebb) { let loc = inst.into(); - let value_lists = &mut self.function.dfg.value_lists; - match self.function.dfg.insts[inst] { - InstructionData::Nullary { .. } | - InstructionData::UnaryImm { .. } | - InstructionData::UnaryIeee32 { .. } | - InstructionData::UnaryIeee64 { .. } | - InstructionData::StackLoad { .. } => {} - - InstructionData::BinaryImm { ref mut arg, .. } | - InstructionData::BranchTable { ref mut arg, .. } | - InstructionData::ExtractLane { ref mut arg, .. } | - InstructionData::IntCompareImm { ref mut arg, .. } | - InstructionData::Unary { ref mut arg, .. } | - InstructionData::UnarySplit { ref mut arg, .. } | - InstructionData::StackStore { ref mut arg, .. } | - InstructionData::HeapLoad { ref mut arg, .. } | - InstructionData::Load { ref mut arg, .. } => { - self.map.rewrite_value(arg, loc)?; - } - - // `args: Value[2]` - InstructionData::Binary { ref mut args, .. } | - InstructionData::BinaryOverflow { ref mut args, .. } | - InstructionData::InsertLane { ref mut args, .. } | - InstructionData::IntCompare { ref mut args, .. } | - InstructionData::FloatCompare { ref mut args, .. } | - InstructionData::HeapStore { ref mut args, .. } | - InstructionData::Store { ref mut args, .. } => { - self.map.rewrite_values(args, loc)?; - } - - InstructionData::Ternary { ref mut args, .. } => { - self.map.rewrite_values(args, loc)?; - } - - InstructionData::MultiAry { ref mut args, .. } => { - self.map - .rewrite_values(args.as_mut_slice(value_lists), loc)?; - } - - InstructionData::Jump { - ref mut destination, - ref mut args, - .. - } | - InstructionData::Branch { - ref mut destination, - ref mut args, - .. - } | - InstructionData::BranchIcmp { - ref mut destination, - ref mut args, - .. - } => { - self.map.rewrite_ebb(destination, loc)?; - self.map - .rewrite_values(args.as_mut_slice(value_lists), loc)?; - } - - InstructionData::Call { ref mut args, .. } | - InstructionData::IndirectCall { ref mut args, .. } => { - self.map - .rewrite_values(args.as_mut_slice(value_lists), loc)?; - } + self.map + .rewrite_values(self.function.dfg.inst_args_mut(inst), loc)?; + if let Some(dest) = self.function.dfg[inst].branch_destination_mut() { + self.map.rewrite_ebb(dest, loc)?; } } } From 7b2becfb2189d6587b01b19c53b836b9658b1b1b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 11 Apr 2017 16:03:49 -0700 Subject: [PATCH 0683/3084] Add EntityList::first(). This is the same as slice::first(), except it returns the first element by value. The implementation can avoid checking the list length since empty lists already have a special representation. --- lib/cretonne/src/entity_list.rs | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index 7e11019529..3b7493094e 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -260,6 +260,15 @@ impl EntityList { self.as_slice(pool).get(index).cloned() } + /// Get the first element from the list. + pub fn first(&self, pool: &ListPool) -> Option { + if self.is_empty() { + None + } else { + Some(pool.data[self.index as usize]) + } + } + /// Get the list as a mutable slice. pub fn as_mut_slice<'a>(&'a mut self, pool: &'a mut ListPool) -> &'a mut [T] { let idx = self.index as usize; @@ -507,6 +516,7 @@ mod tests { assert!(list.is_empty()); assert_eq!(list.len(pool), 0); assert_eq!(list.as_slice(pool), &[]); + assert_eq!(list.first(pool), None); } #[test] @@ -523,6 +533,7 @@ mod tests { assert_eq!(list.len(pool), 1); assert!(!list.is_empty()); assert_eq!(list.as_slice(pool), &[i1]); + assert_eq!(list.first(pool), Some(i1)); assert_eq!(list.get(0, pool), Some(i1)); assert_eq!(list.get(1, pool), None); @@ -530,6 +541,7 @@ mod tests { assert_eq!(list.len(pool), 2); assert!(!list.is_empty()); assert_eq!(list.as_slice(pool), &[i1, i2]); + assert_eq!(list.first(pool), Some(i1)); assert_eq!(list.get(0, pool), Some(i1)); assert_eq!(list.get(1, pool), Some(i2)); assert_eq!(list.get(2, pool), None); @@ -538,6 +550,7 @@ mod tests { assert_eq!(list.len(pool), 3); assert!(!list.is_empty()); assert_eq!(list.as_slice(pool), &[i1, i2, i3]); + assert_eq!(list.first(pool), Some(i1)); assert_eq!(list.get(0, pool), Some(i1)); assert_eq!(list.get(1, pool), Some(i2)); assert_eq!(list.get(2, pool), Some(i3)); @@ -548,6 +561,7 @@ mod tests { assert_eq!(list.len(pool), 4); assert!(!list.is_empty()); assert_eq!(list.as_slice(pool), &[i1, i2, i3, i4]); + assert_eq!(list.first(pool), Some(i1)); assert_eq!(list.get(0, pool), Some(i1)); assert_eq!(list.get(1, pool), Some(i2)); assert_eq!(list.get(2, pool), Some(i3)); From 3a36435c3b5e81abfc1121c54f464734fc725ec7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 11 Apr 2017 15:38:50 -0700 Subject: [PATCH 0684/3084] Maintain a ValueList with the results of each instruction. This is the first step of a larger refactoring to represent instruction results as value lists instead of using linked lists. The refactoring will also eliminate the special treatment of first results such that all result values can be detached and redefined. This change put us in a temporary state where results are represented both as linked lists and ValueList vectors. - Add a dfg.results table. - Add the first result in make_inst(). This behavior will change. - Recompute the result list in make_inst_results(). - Make dfg.first_result(inst) crash if the instruction has no results. --- lib/cretonne/src/ir/dfg.rs | 71 +++++++++++++++++++++++++++++++++----- 1 file changed, 63 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 63ae3a5e97..16de65fb3b 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -28,14 +28,26 @@ pub struct DataFlowGraph { /// with the EBB containing each instruction. insts: EntityMap, - /// Memory pool of value lists referenced by instructions in `insts`. - pub value_lists: ValueListPool, + /// List of result values for each instruction. + /// + /// This map gets resized automatically by `make_inst()` so it is always in sync with the + /// primary `insts` map. + results: EntityMap, /// Extended basic blocks in the function and their arguments. /// This map is not in program order. That is handled by `Layout`, and so is the sequence of /// instructions contained in each EBB. ebbs: EntityMap, + /// Memory pool of value lists. + /// + /// The `ValueList` references into this pool appear in many places: + /// + /// - Instructions in `insts` that don't have room for their entire argument list inline. + /// - Instruction result values in `results`. + /// - EBB arguments in `ebbs`. + pub value_lists: ValueListPool, + /// Extended value table. Most `Value` references refer directly to their defining instruction. /// Others index into this table. /// @@ -61,8 +73,9 @@ impl DataFlowGraph { pub fn new() -> DataFlowGraph { DataFlowGraph { insts: EntityMap::new(), - value_lists: ValueListPool::new(), + results: EntityMap::new(), ebbs: EntityMap::new(), + value_lists: ValueListPool::new(), extended_values: Vec::new(), signatures: EntityMap::new(), ext_funcs: EntityMap::new(), @@ -140,8 +153,20 @@ impl DataFlowGraph { Direct(inst) => ValueDef::Res(inst, 0), Table(idx) => { match self.extended_values[idx] { - ValueData::Inst { inst, num, .. } => ValueDef::Res(inst, num as usize), - ValueData::Arg { ebb, num, .. } => ValueDef::Arg(ebb, num as usize), + ValueData::Inst { inst, num, .. } => { + assert_eq!(Some(v), + self.results[inst].get(num as usize, &self.value_lists), + "Dangling result value {}: {}", + v, + self.display_inst(inst)); + ValueDef::Res(inst, num as usize) + } + ValueData::Arg { ebb, num, .. } => { + assert_eq!(Some(v), + self.ebbs[ebb].args.get(num as usize, &self.value_lists), + "Dangling EBB argument value"); + ValueDef::Arg(ebb, num as usize) + } ValueData::Alias { original, .. } => { // Make sure we only recurse one level. `resolve_aliases` has safeguards to // detect alias loops without overrunning the stack. @@ -334,7 +359,13 @@ impl DataFlowGraph { /// The type of the first result is indicated by `data.ty`. If the instruction produces /// multiple results, also call `make_inst_results` to allocate value table entries. pub fn make_inst(&mut self, data: InstructionData) -> Inst { - self.insts.push(data) + let ty = data.first_type(); + let inst = self.insts.push(data); + let res = self.results.ensure(inst); + if !ty.is_void() { + res.push(Value::new_direct(inst), &mut self.value_lists); + } + inst } /// Get the instruction reference that will be assigned to the next instruction created by @@ -416,6 +447,8 @@ impl DataFlowGraph { let fixed_results = constraints.fixed_results(); let mut total_results = fixed_results; + self.results[inst].clear(&mut self.value_lists); + // Additional values form a linked list starting from the second result value. Generate // the list backwards so we don't have to modify value table entries in place. (This // causes additional result values to be numbered backwards which is not the aesthetic @@ -439,6 +472,7 @@ impl DataFlowGraph { inst: inst, next: head.into(), })); + self.results[inst].push(head.unwrap(), &mut self.value_lists); rev_num += 1; } first_type = Some(self.signatures[sig].return_types[res_idx].value_type); @@ -455,6 +489,7 @@ impl DataFlowGraph { next: head.into(), })); rev_num += 1; + self.results[inst].push(head.unwrap(), &mut self.value_lists); } first_type = Some(constraints.result_type(res_idx, ctrl_typevar)); } @@ -467,6 +502,14 @@ impl DataFlowGraph { } *self.insts[inst].first_type_mut() = first_type.unwrap_or_default(); + // Include the first result in the results vector. + if first_type.is_some() { + self.results[inst].push(Value::new_direct(inst), &mut self.value_lists); + } + self.results[inst] + .as_mut_slice(&mut self.value_lists) + .reverse(); + total_results } @@ -491,6 +534,10 @@ impl DataFlowGraph { /// Use this method to detach secondary values before using `replace(inst)` to provide an /// alternate instruction for computing the primary result value. pub fn detach_secondary_results(&mut self, inst: Inst) -> Option { + self.results[inst].clear(&mut self.value_lists); + if !self.insts[inst].first_type().is_void() { + self.results[inst].push(Value::new_direct(inst), &mut self.value_lists); + } self[inst].second_result_mut().and_then(|r| r.take()) } @@ -539,6 +586,8 @@ impl DataFlowGraph { } }; + self.results[res_inst].push(res, &mut self.value_lists); + // Now update `res` itself. if let ExpandedValue::Table(idx) = res.expand() { if let ValueData::Inst { @@ -597,8 +646,11 @@ impl DataFlowGraph { let data = self[orig].clone(); // After cloning, any secondary values are attached to both copies. Don't do that, we only // want them on the new clone. + let mut results = self.results[orig].take(); self.detach_secondary_results(orig); let new = self.make_inst(data); + results.as_mut_slice(&mut self.value_lists)[0] = Value::new_direct(new); + self.results[new] = results; pos.insert_inst(new); // Replace the original instruction with a copy of the new value. // This is likely to be immediately overwritten by something else, but this way we avoid @@ -611,9 +663,11 @@ impl DataFlowGraph { /// Get the first result of an instruction. /// - /// If `Inst` doesn't produce any results, this returns a `Value` with a `VOID` type. + /// This function panics if the instruction doesn't have any result. pub fn first_result(&self, inst: Inst) -> Value { - Value::new_direct(inst) + self.results[inst] + .first(&self.value_lists) + .expect("Instruction has no results") } /// Iterate through all the results of an instruction. @@ -854,6 +908,7 @@ mod tests { let mut res = dfg.inst_results(inst); let val = res.next().unwrap(); assert!(res.next().is_none()); + assert_eq!(val, dfg.first_result(inst)); assert_eq!(dfg.value_def(val), ValueDef::Res(inst, 0)); assert_eq!(dfg.value_type(val), types::I32); From 322a8db83957d10169f4d238b4d07b647eacaaf2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 11 Apr 2017 16:53:54 -0700 Subject: [PATCH 0685/3084] Change dfg.inst_results to return a slice. Now we can access instruction results and arguments as well as EBB arguments as slices. Delete the Values iterator which was traversing the linked lists of values. It is no longer needed. --- lib/cretonne/meta/gen_instr.py | 13 ++-- lib/cretonne/src/ir/builder.rs | 2 +- lib/cretonne/src/ir/dfg.rs | 70 +++---------------- lib/cretonne/src/legalizer/boundary.rs | 12 ++-- .../src/regalloc/live_value_tracker.rs | 2 +- lib/cretonne/src/regalloc/liveness.rs | 2 +- lib/cretonne/src/verifier.rs | 6 +- lib/cretonne/src/write.rs | 2 +- lib/reader/src/parser.rs | 5 +- 9 files changed, 33 insertions(+), 81 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 9ce279dc52..40456339ae 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -693,14 +693,17 @@ def gen_inst_builder(inst, fmt): fmt.line(fcall + '.0') return + fmt.line('let (inst, dfg) = {};'.format(fcall)) + if len(inst.value_results) == 1: - fmt.line('Value::new_direct({}.0)'.format(fcall)) + fmt.line('dfg.first_result(inst)') return - fmt.line('let (inst, dfg) = {};'.format(fcall)) - fmt.line('let mut results = dfg.inst_results(inst);') - fmt.line('({})'.format(', '.join( - len(inst.value_results) * ['results.next().unwrap()']))) + fmt.format( + 'let results = &dfg.inst_results(inst)[0..{}];', + len(inst.value_results)) + fmt.format('({})', ', '.join( + 'results[{}]'.format(i) for i in range(len(inst.value_results)))) def gen_builder(insts, fmt): diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index dcec1bbb91..7ce771de58 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -143,7 +143,7 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { assert_eq!(old_second_value, None, "Secondary result values {:?} would be left dangling by replacing {} with {}", - self.dfg.inst_results(self.inst).collect::>(), + self.dfg.inst_results(self.inst), self.dfg[self.inst].opcode(), data.opcode()); diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 16de65fb3b..1778453ee2 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -314,43 +314,6 @@ enum ValueData { Alias { ty: Type, original: Value }, } -/// Iterate through a list of related value references, such as: -/// -/// - All results defined by an instruction. See `DataFlowGraph::inst_results`. -/// - All arguments to an EBB. See `DataFlowGraph::ebb_args`. -/// -/// A value iterator borrows a `DataFlowGraph` reference. -pub struct Values<'a> { - dfg: &'a DataFlowGraph, - cur: Option, -} - -impl<'a> Iterator for Values<'a> { - type Item = Value; - - fn next(&mut self) -> Option { - let rval = self.cur; - if let Some(prev) = rval { - // Advance self.cur to the next value, or `None`. - self.cur = match prev.expand() { - ExpandedValue::Direct(inst) => self.dfg.insts[inst].second_result(), - ExpandedValue::Table(index) => { - match self.dfg.extended_values[index] { - ValueData::Inst { next, .. } => next.into(), - ValueData::Arg { .. } => { - panic!("EBB argument {} appeared in value list", prev) - } - ValueData::Alias { .. } => { - panic!("Alias value {} appeared in value list", prev) - } - } - } - }; - } - rval - } -} - /// Instructions. /// impl DataFlowGraph { @@ -671,15 +634,8 @@ impl DataFlowGraph { } /// Iterate through all the results of an instruction. - pub fn inst_results(&self, inst: Inst) -> Values { - Values { - dfg: self, - cur: if self.insts[inst].first_type().is_void() { - None - } else { - Some(Value::new_direct(inst)) - }, - } + pub fn inst_results(&self, inst: Inst) -> &[Value] { + self.results[inst].as_slice(&self.value_lists) } /// Get the call signature of a direct or indirect call instruction. @@ -858,10 +814,9 @@ impl<'a> fmt::Display for DisplayInst<'a> { let dfg = self.0; let inst = &dfg[self.1]; - let mut results = dfg.inst_results(self.1); - if let Some(first) = results.next() { + if let Some((first, rest)) = dfg.inst_results(self.1).split_first() { write!(f, "{}", first)?; - for v in results { + for v in rest { write!(f, ", {}", v)?; } write!(f, " = ")?; @@ -904,11 +859,9 @@ mod tests { assert_eq!(ins.first_type(), types::I32); } - // Result iterator. - let mut res = dfg.inst_results(inst); - let val = res.next().unwrap(); - assert!(res.next().is_none()); - assert_eq!(val, dfg.first_result(inst)); + // Results. + let val = dfg.first_result(inst); + assert_eq!(dfg.inst_results(inst), &[val]); assert_eq!(dfg.value_def(val), ValueDef::Res(inst, 0)); assert_eq!(dfg.value_type(val), types::I32); @@ -925,9 +878,8 @@ mod tests { let inst = dfg.make_inst(idata); assert_eq!(dfg.display_inst(inst).to_string(), "trap"); - // Result iterator should be empty. - let mut res = dfg.inst_results(inst); - assert_eq!(res.next(), None); + // Result slice should be empty. + assert_eq!(dfg.inst_results(inst), &[]); } #[test] @@ -1028,8 +980,8 @@ mod tests { let new_iadd = dfg.redefine_first_value(pos); let new_s = dfg.first_result(new_iadd); assert_eq!(dfg[iadd].opcode(), Opcode::Copy); - assert_eq!(dfg.inst_results(iadd).collect::>(), [s]); - assert_eq!(dfg.inst_results(new_iadd).collect::>(), [new_s, c]); + assert_eq!(dfg.inst_results(iadd), &[s]); + assert_eq!(dfg.inst_results(new_iadd), &[new_s, c]); assert_eq!(dfg.resolve_copies(s), new_s); pos.next_inst(); diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 42386cab83..f80b6574c4 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -302,11 +302,9 @@ fn convert_to_abi(dfg: &mut DataFlowGraph, } /// Check if a sequence of arguments match a desired sequence of argument types. -fn check_arg_types(dfg: &DataFlowGraph, args: Args, types: &[ArgumentType]) -> bool - where Args: IntoIterator -{ +fn check_arg_types(dfg: &DataFlowGraph, args: &[Value], types: &[ArgumentType]) -> bool { let mut n = 0; - for arg in args { + for &arg in args { match types.get(n) { Some(&ArgumentType { value_type, .. }) => { if dfg.value_type(arg) != value_type { @@ -335,7 +333,7 @@ fn check_call_signature(dfg: &DataFlowGraph, inst: Inst) -> Result<(), SigRef> { }; let sig = &dfg.signatures[sig_ref]; - if check_arg_types(dfg, args.iter().cloned(), &sig.argument_types[..]) && + if check_arg_types(dfg, args, &sig.argument_types[..]) && check_arg_types(dfg, dfg.inst_results(inst), &sig.return_types[..]) { // All types check out. Ok(()) @@ -347,9 +345,7 @@ fn check_call_signature(dfg: &DataFlowGraph, inst: Inst) -> Result<(), SigRef> { /// Check if the arguments of the return `inst` match the signature. fn check_return_signature(dfg: &DataFlowGraph, inst: Inst, sig: &Signature) -> bool { - check_arg_types(dfg, - dfg.inst_variable_args(inst).iter().cloned(), - &sig.return_types) + check_arg_types(dfg, dfg.inst_variable_args(inst), &sig.return_types) } /// Insert ABI conversion code for the arguments to the call or return instruction at `pos`. diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 458fe14e26..c9f5146245 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -233,7 +233,7 @@ impl LiveValueTracker { // Add the values defined by `inst`. let first_def = self.live.values.len(); - for value in dfg.inst_results(inst) { + for &value in dfg.inst_results(inst) { let lr = match liveness.get(value) { Some(lr) => lr, None => panic!("{} result {} has no live range", dfg[inst].opcode(), value), diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 90b817a825..cb937d67a2 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -308,7 +308,7 @@ impl Liveness { // Make sure we have created live ranges for dead defs. // TODO: When we implement DCE, we can use the absence of a live range to indicate // an unused value. - for def in func.dfg.inst_results(inst) { + for &def in func.dfg.inst_results(inst) { get_or_create(&mut self.ranges, def, func, &enc_info); } diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index ace83235df..d6018a8f5c 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -193,7 +193,7 @@ impl<'a> Verifier<'a> { } } else { // All result values for multi-valued instructions are created - let got_results = dfg.inst_results(inst).count(); + let got_results = dfg.inst_results(inst).len(); if got_results != total_results { return err!(inst, "expected {} result values, found {}", @@ -212,7 +212,7 @@ impl<'a> Verifier<'a> { self.verify_value(inst, arg)?; } - for res in self.func.dfg.inst_results(inst) { + for &res in self.func.dfg.inst_results(inst) { self.verify_value(inst, res)?; } @@ -448,7 +448,7 @@ impl<'a> Verifier<'a> { fn typecheck_results(&self, inst: Inst, ctrl_type: Type) -> Result<()> { let mut i = 0; - for result in self.func.dfg.inst_results(inst) { + for &result in self.func.dfg.inst_results(inst) { let result_type = self.func.dfg.value_type(result); let expected_type = self.func.dfg.compute_result_type(inst, i, ctrl_type); if let Some(expected_type) = expected_type { diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 09a4410f43..ae4a38284f 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -188,7 +188,7 @@ fn write_instruction(w: &mut Write, // Write value locations, if we have them. if !func.locations.is_empty() { let regs = isa.register_info(); - for r in func.dfg.inst_results(inst) { + for &r in func.dfg.inst_results(inst) { write!(s, ",{}", func.locations diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index a5c3b4244c..5f51b4c342 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1257,12 +1257,13 @@ impl<'a> Parser<'a> { // holds a reference to `ctx.function`. self.add_values(&mut ctx.map, results.into_iter(), - ctx.function.dfg.inst_results(inst))?; + ctx.function.dfg.inst_results(inst).iter().cloned())?; if let Some(result_locations) = result_locations { - for (value, loc) in ctx.function + for (&value, loc) in ctx.function .dfg .inst_results(inst) + .iter() .zip(result_locations) { *ctx.function.locations.ensure(value) = loc; } From 71338bb31f9c04024111b957c4940ada3bb04564 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 08:34:59 -0700 Subject: [PATCH 0686/3084] Simplify the back-end of InstBuilder. We don't want to distinguish between single-result and multiple-result instructions any longer. - Merge the simple_instruction() and complex_instruction() builder methods into a single build() that can handle all cases. - All format constructors now take a ctrl_type argument. Previously, some would take a result_type argument. - Instruction constructors no longer attempt to compute a single result type. Just pass a ctrl_type and let the backend decide. Fix one format constructor call in legalizer/split.rs which now takes a ctrl_type instead of a result type. --- lib/cretonne/meta/gen_instr.py | 56 +++++-------------- lib/cretonne/src/ir/builder.rs | 83 ++++++----------------------- lib/cretonne/src/ir/dfg.rs | 7 ++- lib/cretonne/src/legalizer/split.rs | 2 +- 4 files changed, 36 insertions(+), 112 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 40456339ae..ece67dfb2f 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -498,21 +498,12 @@ def gen_format_constructor(iform, fmt): Emit a method for creating and inserting inserting an `iform` instruction, where `iform` is an instruction format. - Instruction formats that can produce multiple results take a `ctrl_typevar` - argument for deducing the result types. Others take a `result_type` - argument. + All instruction formats take an `opcode` argument and a `ctrl_typevar` + argument for deducing the result types. """ # Construct method arguments. - args = ['self', 'opcode: Opcode'] - - if iform.multiple_results: - args.append('ctrl_typevar: Type') - # `dfg::make_inst_results` will compute the result type. - result_type = 'types::VOID' - else: - args.append('result_type: Type') - result_type = 'result_type' + args = ['self', 'opcode: Opcode', 'ctrl_typevar: Type'] # Normal operand arguments. Start with the immediate operands. for f in iform.imm_fields: @@ -537,16 +528,12 @@ def gen_format_constructor(iform, fmt): with fmt.indented( 'let data = InstructionData::{} {{'.format(iform.name), '};'): fmt.line('opcode: opcode,') - fmt.line('ty: {},'.format(result_type)) + fmt.line('ty: types::VOID,') if iform.multiple_results: fmt.line('second_result: None.into(),') gen_member_inits(iform, fmt) - # Create result values if necessary. - if iform.multiple_results: - fmt.line('self.complex_instruction(data, ctrl_typevar)') - else: - fmt.line('self.simple_instruction(data)') + fmt.line('self.build(data, ctrl_typevar)') def gen_member_inits(iform, fmt): @@ -631,34 +618,19 @@ def gen_inst_builder(inst, fmt): if inst.is_polymorphic and not inst.use_typevar_operand: # This was an explicit method argument. args.append(inst.ctrl_typevar.name) - elif len(inst.value_results) == 0: + elif len(inst.value_results) == 0 or not inst.is_polymorphic: + # No controlling type variable needed. args.append('types::VOID') - elif inst.is_polymorphic: + else: + assert inst.is_polymorphic and inst.use_typevar_operand # Infer the controlling type variable from the input operands. opnum = inst.value_opnums[inst.format.typevar_operand] fmt.line( 'let ctrl_typevar = self.data_flow_graph().value_type({});' .format(inst.ins[opnum].name)) - if inst.format.multiple_results: - # The format constructor will resolve the result types from the - # type var. - args.append('ctrl_typevar') - elif inst.outs[inst.value_results[0]].typevar == inst.ctrl_typevar: - # The format constructor expects a simple result type. - # No type transformation needed from the controlling type - # variable. - args.append('ctrl_typevar') - else: - # The format constructor expects a simple result type. - # TODO: This formula could be resolved ahead of time. - args.append( - 'Opcode::{}.constraints().result_type(0, ctrl_typevar)' - .format(inst.camel_name)) - else: - # This non-polymorphic instruction has a fixed result type. - args.append( - inst.outs[inst.value_results[0]] - .typevar.singleton_type.rust_name()) + # The format constructor will resolve the result types from the + # type var. + args.append('ctrl_typevar') # Now add all of the immediate operands to the constructor arguments. for opnum in inst.imm_opnums: @@ -717,8 +689,8 @@ def gen_builder(insts, fmt): The `InstrBuilder` trait has one method per instruction opcode for conveniently constructing the instruction with minimum arguments. Polymorphic instructions infer their result types from the input - arguments when possible. In some cases, an explicit `result_type` - or `ctrl_typevar` argument is required. + arguments when possible. In some cases, an explicit `ctrl_typevar` + argument is required. The opcode methods return the new instruction's result values, or the `Inst` itself for instructions that don't have any results. diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 7ce771de58..c64ce79dc5 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -24,21 +24,11 @@ pub trait InstBuilderBase<'f>: Sized { fn data_flow_graph(&self) -> &DataFlowGraph; fn data_flow_graph_mut(&mut self) -> &mut DataFlowGraph; - /// Insert a simple instruction and return a reference to it. + /// Insert an instruction and return a reference to it, consuming the builder. /// - /// A 'simple' instruction has at most one result, and the `data.ty` field must contain the - /// result type or `VOID` for an instruction with no result values. - fn simple_instruction(self, data: InstructionData) -> (Inst, &'f mut DataFlowGraph); - - /// Insert a simple instruction and return a reference to it. - /// - /// A 'complex' instruction may produce multiple results, and the result types may depend on a - /// controlling type variable. For non-polymorphic instructions with multiple results, pass - /// `VOID` for the `ctrl_typevar` argument. - fn complex_instruction(self, - data: InstructionData, - ctrl_typevar: Type) - -> (Inst, &'f mut DataFlowGraph); + /// The result types may depend on a controlling type variable. For non-polymorphic + /// instructions with multiple results, pass `VOID` for the `ctrl_typevar` argument. + fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'f mut DataFlowGraph); } // Include trait code generated by `lib/cretonne/meta/gen_instr.py`. @@ -79,16 +69,7 @@ impl<'c, 'fc, 'fd> InstBuilderBase<'fd> for InsertBuilder<'c, 'fc, 'fd> { self.dfg } - fn simple_instruction(self, data: InstructionData) -> (Inst, &'fd mut DataFlowGraph) { - let inst = self.dfg.make_inst(data); - self.pos.insert_inst(inst); - (inst, self.dfg) - } - - fn complex_instruction(self, - data: InstructionData, - ctrl_typevar: Type) - -> (Inst, &'fd mut DataFlowGraph) { + fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'fd mut DataFlowGraph) { let inst = self.dfg.make_inst(data); self.dfg.make_inst_results(inst, ctrl_typevar); self.pos.insert_inst(inst); @@ -98,20 +79,12 @@ impl<'c, 'fc, 'fd> InstBuilderBase<'fd> for InsertBuilder<'c, 'fc, 'fd> { /// Instruction builder that replaces an existing instruction. /// -/// The inserted instruction will have the same `Inst` number as the old one. This is the only way -/// of rewriting the first result value of an instruction since this is a `ExpandedValue::Direct` -/// variant which encodes the instruction number directly. +/// The inserted instruction will have the same `Inst` number as the old one. /// -/// If the old instruction produced a value, the same value number will refer to the new -/// instruction's first result, so if that value has any uses the type should stay the same. -/// -/// If the old instruction still has secondary result values attached, it is assumed that the new -/// instruction produces the same number and types of results. The old secondary values are -/// preserved. If the replacement instruction format does not support multiple results, the builder -/// panics. It is a bug to leave result values dangling. -/// -/// If the old instruction was capable of producing secondary results, but the values have been -/// detached, new result values are generated by calling `DataFlowGraph::make_inst_results()`. +/// If the old instruction still has result values attached, it is assumed that the new instruction +/// produces the same number and types of results. The old result values are preserved. If the +/// replacement instruction format does not support multiple results, the builder panics. It is a +/// bug to leave result values dangling. pub struct ReplaceBuilder<'f> { dfg: &'f mut DataFlowGraph, inst: Inst, @@ -136,46 +109,20 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { self.dfg } - fn simple_instruction(self, data: InstructionData) -> (Inst, &'f mut DataFlowGraph) { - // The replacement instruction cannot generate multiple results, so verify that the old - // instruction's secondary results have been detached. - let old_second_value = self.dfg[self.inst].second_result(); - assert_eq!(old_second_value, - None, - "Secondary result values {:?} would be left dangling by replacing {} with {}", - self.dfg.inst_results(self.inst), - self.dfg[self.inst].opcode(), - data.opcode()); - - // Splat the new instruction on top of the old one. - self.dfg[self.inst] = data; - (self.inst, self.dfg) - } - - fn complex_instruction(self, - data: InstructionData, - ctrl_typevar: Type) - -> (Inst, &'f mut DataFlowGraph) { - // If the old instruction still has secondary results attached, we'll keep them. - let old_second_value = self.dfg[self.inst].second_result(); - + fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'f mut DataFlowGraph) { // Splat the new instruction on top of the old one. self.dfg[self.inst] = data; - if old_second_value.is_none() { - // The old secondary values were either detached or non-existent. - // Construct new ones and set the first result type too. + if !self.dfg.has_results(self.inst) { + // The old result values were either detached or non-existent. + // Construct new ones. self.dfg.make_inst_results(self.inst, ctrl_typevar); } else { // Reattach the old secondary values. + let old_second_value = self.dfg.inst_results(self.inst).get(1).cloned(); if let Some(val_ref) = self.dfg[self.inst].second_result_mut() { // Don't check types here. Leave that to the verifier. *val_ref = old_second_value.into(); - } else { - // Actually, this instruction format should have called `simple_instruction()`, but - // we don't have a rule against calling `complex_instruction()` even when it is - // overkill. - panic!("Secondary result values left dangling"); } // Normally, make_inst_results() would also set the first result type, but we're not diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 1778453ee2..6fe3f91d57 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -633,7 +633,12 @@ impl DataFlowGraph { .expect("Instruction has no results") } - /// Iterate through all the results of an instruction. + /// Test if `inst` has any result values currently. + pub fn has_results(&self, inst: Inst) -> bool { + !self.results[inst].is_empty() + } + + /// Return all the results of an instruction. pub fn inst_results(&self, inst: Inst) -> &[Value] { self.results[inst].as_slice(&self.value_lists) } diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index d1f55a6266..3c03d1d77b 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -228,7 +228,7 @@ fn split_value(dfg: &mut DataFlowGraph, // need to insert a split instruction before returning. pos.goto_top(ebb); pos.next_inst(); - let concat_inst = dfg.ins(pos).Binary(concat, ty, lo, hi).0; + let concat_inst = dfg.ins(pos).Binary(concat, split_type, lo, hi).0; let concat_val = dfg.first_result(concat_inst); dfg.change_to_alias(value, concat_val); From 78a036fa514faa4db48120c2e052e41c934c79f0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 09:08:29 -0700 Subject: [PATCH 0687/3084] Remove the second_result field from instructions. Now that result values are stored in value lists in the DFG, these head-of-list pointers are no longer needed. --- lib/cretonne/meta/gen_instr.py | 40 ----------------------------- lib/cretonne/src/ir/builder.rs | 7 ----- lib/cretonne/src/ir/dfg.rs | 38 ++++++++++----------------- lib/cretonne/src/ir/instructions.rs | 6 ----- lib/reader/src/parser.rs | 5 ---- 5 files changed, 13 insertions(+), 83 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index ece67dfb2f..4d71e1e5b4 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -110,8 +110,6 @@ def gen_instruction_data_impl(fmt): - `pub fn opcode(&self) -> Opcode` - `pub fn first_type(&self) -> Type` - - `pub fn second_result(&self) -> Option` - - `pub fn second_result_mut(&mut self) -> Option<&mut PackedOption>` - `pub fn arguments(&self, &pool) -> &[Value]` - `pub fn arguments_mut(&mut self, &pool) -> &mut [Value]` - `pub fn take_value_list(&mut self) -> Option` @@ -147,42 +145,6 @@ def gen_instruction_data_impl(fmt): 'InstructionData::{} {{ ref mut ty, .. }} => ty,' .format(f.name)) - # Generate shared and mutable accessors for `second_result` which only - # applies to instruction formats that can produce multiple results. - # Everything else returns `None`. - fmt.doc_comment('Second result value, if any.') - with fmt.indented( - 'pub fn second_result(&self) -> Option {', '}'): - with fmt.indented('match *self {', '}'): - for f in InstructionFormat.all_formats: - if f.multiple_results: - fmt.line( - 'InstructionData::' + f.name + - ' { second_result, .. }' + - ' => second_result.into(),') - else: - # Single or no results. - fmt.line( - 'InstructionData::{} {{ .. }} => None,' - .format(f.name)) - - fmt.doc_comment('Mutable reference to second result value, if any.') - with fmt.indented( - "pub fn second_result_mut(&mut self)" + - " -> Option<&mut PackedOption> {", '}'): - with fmt.indented('match *self {', '}'): - for f in InstructionFormat.all_formats: - if f.multiple_results: - fmt.line( - 'InstructionData::' + f.name + - ' { ref mut second_result, .. }' + - ' => Some(second_result),') - else: - # Single or no results. - fmt.line( - 'InstructionData::{} {{ .. }} => None,' - .format(f.name)) - fmt.doc_comment('Get the controlling type variable operand.') with fmt.indented( 'pub fn typevar_operand(&self, pool: &ValueListPool) -> ' @@ -529,8 +491,6 @@ def gen_format_constructor(iform, fmt): 'let data = InstructionData::{} {{'.format(iform.name), '};'): fmt.line('opcode: opcode,') fmt.line('ty: types::VOID,') - if iform.multiple_results: - fmt.line('second_result: None.into(),') gen_member_inits(iform, fmt) fmt.line('self.build(data, ctrl_typevar)') diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index c64ce79dc5..312e5144da 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -118,13 +118,6 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { // Construct new ones. self.dfg.make_inst_results(self.inst, ctrl_typevar); } else { - // Reattach the old secondary values. - let old_second_value = self.dfg.inst_results(self.inst).get(1).cloned(); - if let Some(val_ref) = self.dfg[self.inst].second_result_mut() { - // Don't check types here. Leave that to the verifier. - *val_ref = old_second_value.into(); - } - // Normally, make_inst_results() would also set the first result type, but we're not // going to call that, so set it manually. *self.dfg[self.inst].first_type_mut() = self.dfg diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 6fe3f91d57..dc321030b3 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -457,12 +457,6 @@ impl DataFlowGraph { first_type = Some(constraints.result_type(res_idx, ctrl_typevar)); } - // Update the second_result pointer in `inst`. - if head.is_some() { - *self.insts[inst] - .second_result_mut() - .expect("instruction format doesn't allow multiple results") = head.into(); - } *self.insts[inst].first_type_mut() = first_type.unwrap_or_default(); // Include the first result in the results vector. @@ -497,11 +491,16 @@ impl DataFlowGraph { /// Use this method to detach secondary values before using `replace(inst)` to provide an /// alternate instruction for computing the primary result value. pub fn detach_secondary_results(&mut self, inst: Inst) -> Option { + if !self.has_results(inst) { + return None; + } + + let second = self.results[inst].get(1, &mut self.value_lists); self.results[inst].clear(&mut self.value_lists); if !self.insts[inst].first_type().is_void() { self.results[inst].push(Value::new_direct(inst), &mut self.value_lists); } - self[inst].second_result_mut().and_then(|r| r.take()) + second } /// Get the next secondary result after `value`. @@ -524,32 +523,21 @@ impl DataFlowGraph { /// created automatically. The `res` value must be a secondary instruction result detached from /// somewhere else. pub fn attach_secondary_result(&mut self, last_res: Value, res: Value) { - let (res_inst, res_num) = match last_res.expand() { - ExpandedValue::Direct(inst) => { - // We're adding the second value to `inst`. - let next = self[inst].second_result_mut().expect("bad inst format"); - assert!(next.is_none(), "last_res is not the last result"); - *next = res.into(); - (inst, 1) - } + let res_inst = match last_res.expand() { + ExpandedValue::Direct(inst) => inst, ExpandedValue::Table(idx) => { - if let ValueData::Inst { - num, - inst, - ref mut next, - .. - } = self.extended_values[idx] { + if let ValueData::Inst { inst, ref mut next, .. } = self.extended_values[idx] { assert!(next.is_none(), "last_res is not the last result"); *next = res.into(); - assert!(num < u16::MAX, "Too many arguments to EBB"); - (inst, num + 1) + inst } else { panic!("last_res is not an instruction result"); } } }; - self.results[res_inst].push(res, &mut self.value_lists); + let res_num = self.results[res_inst].push(res, &mut self.value_lists); + assert!(res_num <= u16::MAX as usize, "Too many result values"); // Now update `res` itself. if let ExpandedValue::Table(idx) = res.expand() { @@ -559,7 +547,7 @@ impl DataFlowGraph { ref mut next, .. } = self.extended_values[idx] { - *num = res_num; + *num = res_num as u16; *inst = res_inst; *next = None.into(); return; diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 626c130740..29e3e6662f 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -17,7 +17,6 @@ use ir::types; use ir::DataFlowGraph; use entity_list; -use packed_option::PackedOption; use ref_slice::{ref_slice, ref_slice_mut}; /// Some instructions use an external list of argument values because there is not enough space in @@ -126,7 +125,6 @@ pub enum InstructionData { UnarySplit { opcode: Opcode, ty: Type, - second_result: PackedOption, arg: Value, }, Binary { @@ -143,7 +141,6 @@ pub enum InstructionData { BinaryOverflow { opcode: Opcode, ty: Type, - second_result: PackedOption, args: [Value; 2], }, Ternary { @@ -154,7 +151,6 @@ pub enum InstructionData { MultiAry { opcode: Opcode, ty: Type, - second_result: PackedOption, args: ValueList, }, InsertLane { @@ -216,14 +212,12 @@ pub enum InstructionData { Call { opcode: Opcode, ty: Type, - second_result: PackedOption, func_ref: FuncRef, args: ValueList, }, IndirectCall { opcode: Opcode, ty: Type, - second_result: PackedOption, sig_ref: SigRef, args: ValueList, }, diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 5f51b4c342..f948b1f586 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1442,7 +1442,6 @@ impl<'a> Parser<'a> { InstructionData::UnarySplit { opcode: opcode, ty: VOID, - second_result: None.into(), arg: self.match_value("expected SSA value operand")?, } } @@ -1474,7 +1473,6 @@ impl<'a> Parser<'a> { InstructionData::BinaryOverflow { opcode: opcode, ty: VOID, - second_result: None.into(), args: [lhs, rhs], } } @@ -1497,7 +1495,6 @@ impl<'a> Parser<'a> { InstructionData::MultiAry { opcode: opcode, ty: VOID, - second_result: None.into(), args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } } @@ -1610,7 +1607,6 @@ impl<'a> Parser<'a> { InstructionData::Call { opcode: opcode, ty: VOID, - second_result: None.into(), func_ref: func_ref, args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } @@ -1626,7 +1622,6 @@ impl<'a> Parser<'a> { InstructionData::IndirectCall { opcode: opcode, ty: VOID, - second_result: None.into(), sig_ref: sig_ref, args: args.into_value_list(&[callee], &mut ctx.function.dfg.value_lists), } From 0897d43c88484544324640c5e924927642172756 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 09:45:21 -0700 Subject: [PATCH 0688/3084] Don't attach a first result in make_inst(). That is now the job of make_inst_results(). --- lib/cretonne/src/ir/dfg.rs | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index dc321030b3..63849e1a72 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -322,13 +322,9 @@ impl DataFlowGraph { /// The type of the first result is indicated by `data.ty`. If the instruction produces /// multiple results, also call `make_inst_results` to allocate value table entries. pub fn make_inst(&mut self, data: InstructionData) -> Inst { - let ty = data.first_type(); - let inst = self.insts.push(data); - let res = self.results.ensure(inst); - if !ty.is_void() { - res.push(Value::new_direct(inst), &mut self.value_lists); - } - inst + let n = self.num_insts() + 1; + self.results.resize(n); + self.insts.push(data) } /// Get the instruction reference that will be assigned to the next instruction created by @@ -392,9 +388,8 @@ impl DataFlowGraph { /// Create result values for an instruction that produces multiple results. /// - /// Instructions that produce 0 or 1 result values only need to be created with `make_inst`. If - /// the instruction may produce more than 1 result, call `make_inst_results` to allocate - /// value table entries for the additional results. + /// Instructions that produce no result values only need to be created with `make_inst`, + /// otherwise call `make_inst_results` to allocate value table entries for the results. /// /// The result value types are determined from the instruction's value type constraints and the /// provided `ctrl_typevar` type for polymorphic instructions. For non-polymorphic @@ -838,9 +833,10 @@ mod tests { let idata = InstructionData::Nullary { opcode: Opcode::Iconst, - ty: types::I32, + ty: types::VOID, }; let inst = dfg.make_inst(idata); + dfg.make_inst_results(inst, types::I32); assert_eq!(inst.to_string(), "inst0"); assert_eq!(dfg.display_inst(inst).to_string(), "v0 = iconst.i32"); From f8b473c4285d858851983112019c95e5dd49311d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 10:39:11 -0700 Subject: [PATCH 0689/3084] Add detach_results(), attach_result(), and append_result() methods. These low-level methods for manipulating instruction result value lists will replace the existing secondary_result methods. --- lib/cretonne/src/ir/dfg.rs | 61 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 63849e1a72..583f7cc8a1 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -511,6 +511,67 @@ impl DataFlowGraph { panic!("{} is not a secondary result value", value); } + /// Detach the list of result values from `inst` and return it. + /// + /// This leaves `inst` without any result values. New result values can be created by calling + /// `make_inst_results` or by using a `replace(inst)` builder. + pub fn detach_results(&mut self, inst: Inst) -> ValueList { + self.results[inst].take() + } + + /// Attach an existing value to the result value list for `inst`. + /// + /// The `res` value is appended to the end of the result list. + /// + /// This is a very low-level operation. Usually, instruction results with the correct types are + /// created automatically. The `res` value must not be attached to anything else. + pub fn attach_result(&mut self, inst: Inst, res: Value) { + let res_inst = inst; + if let Some(last_res) = self.results[inst] + .as_slice(&mut self.value_lists) + .last() + .cloned() { + self.attach_secondary_result(last_res, res); + } else { + // This is the first result. + self.results[res_inst].push(res, &mut self.value_lists); + + // Now update `res` itself. + match res.expand() { + ExpandedValue::Table(idx) => { + if let ValueData::Inst { + ref mut num, + ref mut inst, + ref mut next, + .. + } = self.extended_values[idx] { + *num = 0; + *inst = res_inst; + *next = None.into(); + return; + } + } + ExpandedValue::Direct(inst) => { + assert_eq!(inst, res_inst); + return; + } + } + panic!("{} must be a result", res); + } + } + + /// Append a new instruction result value to `inst`. + pub fn append_result(&mut self, inst: Inst, ty: Type) -> Value { + let res = self.make_value(ValueData::Inst { + ty: ty, + inst: inst, + num: 0, + next: None.into(), + }); + self.attach_result(inst, res); + res + } + /// Attach an existing value as a secondary result after `last_res` which must be the last /// result of an instruction. /// From 94e26a845a9191bfa4e91dfd4537a14bb5d85bcd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 10:47:49 -0700 Subject: [PATCH 0690/3084] Make tests less sensitive to specific value numbers. --- cranelift/filetests/parser/branch.cton | 64 +++++++++++----------- cranelift/filetests/parser/call.cton | 6 +-- cranelift/filetests/parser/rewrite.cton | 10 ++-- cranelift/filetests/parser/tiny.cton | 72 ++++++++++++------------- 4 files changed, 76 insertions(+), 76 deletions(-) diff --git a/cranelift/filetests/parser/branch.cton b/cranelift/filetests/parser/branch.cton index 1aa0ca185a..ac979bbb26 100644 --- a/cranelift/filetests/parser/branch.cton +++ b/cranelift/filetests/parser/branch.cton @@ -19,65 +19,65 @@ ebb1: ; Jumps with 1 arg. function onearg(i32) { -ebb0(vx0: i32): - jump ebb1(vx0) +ebb0(v90: i32): + jump ebb1(v90) -ebb1(vx1: i32): - jump ebb0(vx1) +ebb1(v91: i32): + jump ebb0(v91) } ; sameln: function onearg(i32) { -; nextln: ebb0(vx0: i32): -; nextln: jump ebb1(vx0) +; nextln: ebb0($v90: i32): +; nextln: jump ebb1($v90) ; nextln: -; nextln: ebb1(vx1: i32): -; nextln: jump ebb0(vx1) +; nextln: ebb1($v91: i32): +; nextln: jump ebb0($v91) ; nextln: } ; Jumps with 2 args. function twoargs(i32, f32) { -ebb0(vx0: i32, vx1: f32): - jump ebb1(vx0, vx1) +ebb0(v90: i32, v91: f32): + jump ebb1(v90, v91) -ebb1(vx2: i32, vx3: f32): - jump ebb0(vx2, vx3) +ebb1(v92: i32, v93: f32): + jump ebb0(v92, v93) } ; sameln: function twoargs(i32, f32) { -; nextln: ebb0(vx0: i32, vx1: f32): -; nextln: jump ebb1(vx0, vx1) +; nextln: ebb0($v90: i32, $v91: f32): +; nextln: jump ebb1($v90, $v91) ; nextln: -; nextln: ebb1(vx2: i32, vx3: f32): -; nextln: jump ebb0(vx2, vx3) +; nextln: ebb1($v92: i32, $v93: f32): +; nextln: jump ebb0($v92, $v93) ; nextln: } ; Branches with no arguments. The '()' empty argument list is optional. function minimal(i32) { -ebb0(vx0: i32): - brz vx0, ebb1 +ebb0(v90: i32): + brz v90, ebb1 ebb1: - brnz vx0, ebb1() + brnz v90, ebb1() } ; sameln: function minimal(i32) { -; nextln: ebb0(vx0: i32): -; nextln: brz vx0, ebb1 +; nextln: ebb0($v90: i32): +; nextln: brz $v90, ebb1 ; nextln: ; nextln: ebb1: -; nextln: brnz.i32 vx0, ebb1 +; nextln: brnz.i32 $v90, ebb1 ; nextln: } function twoargs(i32, f32) { -ebb0(vx0: i32, vx1: f32): - brz vx0, ebb1(vx0, vx1) +ebb0(v90: i32, v91: f32): + brz v90, ebb1(v90, v91) -ebb1(vx2: i32, vx3: f32): - brnz vx0, ebb0(vx2, vx3) +ebb1(v92: i32, v93: f32): + brnz v90, ebb0(v92, v93) } ; sameln: function twoargs(i32, f32) { -; nextln: ebb0(vx0: i32, vx1: f32): -; nextln: brz vx0, ebb1(vx0, vx1) +; nextln: ebb0($v90: i32, $v91: f32): +; nextln: brz $v90, ebb1($v90, $v91) ; nextln: -; nextln: ebb1(vx2: i32, vx3: f32): -; nextln: brnz.i32 vx0, ebb0(vx2, vx3) +; nextln: ebb1($v92: i32, $v93: f32): +; nextln: brnz.i32 $v90, ebb0($v92, $v93) ; nextln: } function jumptable(i32) { @@ -98,8 +98,8 @@ ebb40: ; nextln: jt0 = jump_table 0 ; nextln: jt1 = jump_table 0, 0, ebb0, ebb3, ebb1, ebb2 ; nextln: -; nextln: ebb0(vx0: i32): -; nextln: br_table vx0, jt1 +; nextln: ebb0($v3: i32): +; nextln: br_table $v3, jt1 ; nextln: trap ; nextln: ; nextln: ebb1: diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton index 72a61b6e21..027749f562 100644 --- a/cranelift/filetests/parser/call.cton +++ b/cranelift/filetests/parser/call.cton @@ -18,9 +18,9 @@ ebb1: } ; sameln: function r1() -> i32, f32 { ; nextln: ebb0: -; nextln: v0 = iconst.i32 3 -; nextln: v1 = f32const 0.0 -; nextln: return v0, v1 +; nextln: $v1 = iconst.i32 3 +; nextln: $v2 = f32const 0.0 +; nextln: return $v1, $v2 ; nextln: } function signatures() { diff --git a/cranelift/filetests/parser/rewrite.cton b/cranelift/filetests/parser/rewrite.cton index 9f95cb93fa..ffd1851114 100644 --- a/cranelift/filetests/parser/rewrite.cton +++ b/cranelift/filetests/parser/rewrite.cton @@ -26,12 +26,12 @@ ebb100(v20: i32): function use_value() { ebb100(v20: i32): v1000 = iadd_imm v20, 5 - vx200 = iadd v20, v1000 + v200 = iadd v20, v1000 jump ebb100(v1000) } ; sameln: function use_value() { -; nextln: ebb0(vx0: i32): -; nextln: v0 = iadd_imm vx0, 5 -; nextln: v1 = iadd vx0, v0 -; nextln: jump ebb0(v0) +; nextln: ebb0($v20: i32): +; nextln: $v1000 = iadd_imm $v20, 5 +; nextln: $v200 = iadd $v20, $v1000 +; nextln: jump ebb0($v1000) ; nextln: } diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 868a7ba0b1..87140d9aad 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -20,19 +20,19 @@ ebb0: } ; sameln: function ivalues() { ; nextln: ebb0: -; nextln: v0 = iconst.i32 2 -; nextln: v1 = iconst.i8 6 -; nextln: v2 = ishl v0, v1 +; nextln: $v0 = iconst.i32 2 +; nextln: $v1 = iconst.i8 6 +; nextln: $v2 = ishl $v0, $v1 ; nextln: } ; Polymorphic istruction controlled by second operand. function select() { -ebb0(vx0: i32, vx1: i32, vx2: b1): - v0 = select vx2, vx0, vx1 +ebb0(v90: i32, v91: i32, v92: b1): + v0 = select v92, v90, v91 } ; sameln: function select() { -; nextln: ebb0(vx0: i32, vx1: i32, vx2: b1): -; nextln: v0 = select vx2, vx0, vx1 +; nextln: ebb0($v90: i32, $v91: i32, $v92: b1): +; nextln: $v0 = select $v92, $v90, $v91 ; nextln: } ; Lane indexes. @@ -44,54 +44,54 @@ ebb0: } ; sameln: function lanes() { ; nextln: ebb0: -; nextln: v0 = iconst.i32x4 2 -; nextln: v1 = extractlane v0, 3 -; nextln: v2 = insertlane v0, 1, v1 +; nextln: $v0 = iconst.i32x4 2 +; nextln: $v1 = extractlane $v0, 3 +; nextln: $v2 = insertlane $v0, 1, $v1 ; nextln: } ; Integer condition codes. function icmp(i32, i32) { -ebb0(vx0: i32, vx1: i32): - v0 = icmp eq vx0, vx1 - v1 = icmp ult vx0, vx1 - v2 = icmp_imm sge vx0, -12 - v3 = irsub_imm vx1, 45 - br_icmp eq vx0, vx1, ebb0(vx1, vx0) +ebb0(v90: i32, v91: i32): + v0 = icmp eq v90, v91 + v1 = icmp ult v90, v91 + v2 = icmp_imm sge v90, -12 + v3 = irsub_imm v91, 45 + br_icmp eq v90, v91, ebb0(v91, v90) } ; sameln: function icmp(i32, i32) { -; nextln: ebb0(vx0: i32, vx1: i32): -; nextln: v0 = icmp eq vx0, vx1 -; nextln: v1 = icmp ult vx0, vx1 -; nextln: v2 = icmp_imm sge vx0, -12 -; nextln: v3 = irsub_imm vx1, 45 -; nextln: br_icmp eq vx0, vx1, ebb0(vx1, vx0) +; nextln: ebb0($v90: i32, $v91: i32): +; nextln: $v0 = icmp eq $v90, $v91 +; nextln: $v1 = icmp ult $v90, $v91 +; nextln: $v2 = icmp_imm sge $v90, -12 +; nextln: $v3 = irsub_imm $v91, 45 +; nextln: br_icmp eq $v90, $v91, ebb0($v91, $v90) ; nextln: } ; Floating condition codes. function fcmp(f32, f32) { -ebb0(vx0: f32, vx1: f32): - v0 = fcmp eq vx0, vx1 - v1 = fcmp uno vx0, vx1 - v2 = fcmp lt vx0, vx1 +ebb0(v90: f32, v91: f32): + v0 = fcmp eq v90, v91 + v1 = fcmp uno v90, v91 + v2 = fcmp lt v90, v91 } ; sameln: function fcmp(f32, f32) { -; nextln: ebb0(vx0: f32, vx1: f32): -; nextln: v0 = fcmp eq vx0, vx1 -; nextln: v1 = fcmp uno vx0, vx1 -; nextln: v2 = fcmp lt vx0, vx1 +; nextln: ebb0($v90: f32, $v91: f32): +; nextln: $v0 = fcmp eq $v90, $v91 +; nextln: $v1 = fcmp uno $v90, $v91 +; nextln: $v2 = fcmp lt $v90, $v91 ; nextln: } ; The bitcast instruction has two type variables: The controlling type variable ; controls the outout type, and the input type is a free variable. function bitcast(i32, f32) { -ebb0(vx0: i32, vx1: f32): - v0 = bitcast.i8x4 vx0 - v1 = bitcast.i32 vx1 +ebb0(v90: i32, v91: f32): + v0 = bitcast.i8x4 v90 + v1 = bitcast.i32 v91 } ; sameln: function bitcast(i32, f32) { -; nextln: ebb0(vx0: i32, vx1: f32): -; nextln: v0 = bitcast.i8x4 vx0 -; nextln: v1 = bitcast.i32 vx1 +; nextln: ebb0($v90: i32, $v91: f32): +; nextln: $v0 = bitcast.i8x4 $v90 +; nextln: $v1 = bitcast.i32 $v91 ; nextln: } ; Stack slot references From d133606fe7c7ab45c2847db9abbfa8fe4a496f22 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 10:48:57 -0700 Subject: [PATCH 0691/3084] Don't assume all first results are direct values. We're about to change that. --- lib/cretonne/src/ir/dfg.rs | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 583f7cc8a1..0b0ea7a8cc 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -207,13 +207,12 @@ impl DataFlowGraph { /// Find the original definition of a value, looking through value aliases as well as /// copy/spill/fill instructions. pub fn resolve_copies(&self, value: Value) -> Value { - use ir::entities::ExpandedValue::Direct; let mut v = value; for _ in 0..self.insts.len() { v = self.resolve_aliases(v); - v = match v.expand() { - Direct(inst) => { + v = match self.value_def(v) { + ValueDef::Res(inst, 0) => { match self[inst] { InstructionData::Unary { opcode, arg, .. } => { match opcode { @@ -1003,7 +1002,6 @@ mod tests { #[test] fn aliases() { use ir::InstBuilder; - use ir::entities::ExpandedValue::Direct; use ir::condcodes::IntCC; let mut func = Function::new(); @@ -1020,8 +1018,8 @@ mod tests { let arg0 = dfg.append_ebb_arg(ebb0, types::I32); let (s, c) = dfg.ins(pos).iadd_cout(v1, arg0); - let iadd = match s.expand() { - Direct(i) => i, + let iadd = match dfg.value_def(s) { + ValueDef::Res(i, 0) => i, _ => panic!(), }; From 8a2398be08b22c91904fe404e15718a91a5c02e6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 11:02:59 -0700 Subject: [PATCH 0692/3084] Stop calling Value::new_direct. We only ever create table values now. Simplify legalizer::legalize_inst_results. Instead of calling detach_secondary_results, just detach all the results and don't treat the first result specially. --- .../filetests/isa/riscv/legalize-abi.cton | 6 +- lib/cretonne/src/ir/dfg.rs | 48 ++++++++++----- lib/cretonne/src/legalizer/boundary.rs | 60 ++++++------------- lib/reader/src/parser.rs | 6 +- lib/reader/src/sourcemap.rs | 2 +- 5 files changed, 59 insertions(+), 63 deletions(-) diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index 2e7da079bf..06d68768bd 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -34,9 +34,8 @@ ebb0: ; check: $ebb0: ; nextln: $(v1l=$V), $(v1h=$V) = call $fn1() ; check: $(v1new=$V) = iconcat $v1l, $v1h - ; check: $v1 = copy $v1new jump ebb1(v1) - ; The v1 copy gets resolved by split::simplify_branch_arguments(). + ; The v1 alias gets resolved by split::simplify_branch_arguments(). ; check: jump $ebb1($v1new) ebb1(v10: i64): @@ -77,9 +76,8 @@ ebb0: ; check: $ebb0: ; nextln: $(rv=$V) = call $fn1() ; check: $(v1new=$V) = ireduce.i8 $rv - ; check: $v1 = copy $v1new jump ebb1(v1) - ; The v1 copy gets resolved by split::simplify_branch_arguments(). + ; The v1 alias gets resolved by split::simplify_branch_arguments(). ; check: jump $ebb1($v1new) ebb1(v10: i8): diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 0b0ea7a8cc..359a6cd089 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -454,8 +454,14 @@ impl DataFlowGraph { *self.insts[inst].first_type_mut() = first_type.unwrap_or_default(); // Include the first result in the results vector. - if first_type.is_some() { - self.results[inst].push(Value::new_direct(inst), &mut self.value_lists); + if let Some(ty) = first_type { + let v = self.make_value(ValueData::Inst { + ty: ty, + num: 0, + inst: inst, + next: head.into(), + }); + self.results[inst].push(v, &mut self.value_lists); } self.results[inst] .as_mut_slice(&mut self.value_lists) @@ -489,11 +495,10 @@ impl DataFlowGraph { return None; } + let first = self.results[inst].first(&mut self.value_lists).unwrap(); let second = self.results[inst].get(1, &mut self.value_lists); self.results[inst].clear(&mut self.value_lists); - if !self.insts[inst].first_type().is_void() { - self.results[inst].push(Value::new_direct(inst), &mut self.value_lists); - } + self.results[inst].push(first, &mut self.value_lists); second } @@ -649,21 +654,36 @@ impl DataFlowGraph { pub fn redefine_first_value(&mut self, pos: &mut Cursor) -> Inst { let orig = pos.current_inst() .expect("Cursor must point at an instruction"); + let first_res = self.first_result(orig); + let first_type = self.value_type(first_res); let data = self[orig].clone(); - // After cloning, any secondary values are attached to both copies. Don't do that, we only - // want them on the new clone. - let mut results = self.results[orig].take(); - self.detach_secondary_results(orig); + let results = self.results[orig].take(); + self.results[orig].push(first_res, &mut self.value_lists); + + let new = self.make_inst(data); - results.as_mut_slice(&mut self.value_lists)[0] = Value::new_direct(new); - self.results[new] = results; + let new_first = self.make_value(ValueData::Inst { + ty: first_type, + num: 0, + inst: new, + next: None.into(), + }); + self.results[new].push(new_first, &mut self.value_lists); + + for i in 1.. { + if let Some(v) = results.get(i, &self.value_lists) { + self.attach_result(new, v); + } else { + break; + } + } + pos.insert_inst(new); // Replace the original instruction with a copy of the new value. // This is likely to be immediately overwritten by something else, but this way we avoid // leaving the DFG in a state with multiple references to secondary results and value // lists. It also means that this method doesn't change the semantics of the program. - let new_value = self.first_result(new); - self.replace(orig).copy(new_value); + self.replace(orig).copy(new_first); new } @@ -898,7 +918,7 @@ mod tests { let inst = dfg.make_inst(idata); dfg.make_inst_results(inst, types::I32); assert_eq!(inst.to_string(), "inst0"); - assert_eq!(dfg.display_inst(inst).to_string(), "v0 = iconst.i32"); + assert_eq!(dfg.display_inst(inst).to_string(), "vx0 = iconst.i32"); // Immutable reference resolution. { diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index f80b6574c4..272a19dfee 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -107,7 +107,7 @@ fn legalize_inst_results(dfg: &mut DataFlowGraph, -> Inst where ResType: FnMut(&DataFlowGraph, usize) -> ArgumentType { - let mut call = pos.current_inst() + let call = pos.current_inst() .expect("Cursor must point to a call instruction"); // We theoretically allow for call instructions that return a number of fixed results before @@ -115,58 +115,26 @@ fn legalize_inst_results(dfg: &mut DataFlowGraph, let fixed_results = dfg[call].opcode().constraints().fixed_results(); assert_eq!(fixed_results, 0, "Fixed results on calls not supported"); - let mut next_res = dfg.detach_secondary_results(call); - // The currently last result on the call instruction. - let mut last_res = dfg.first_result(call); + let results = dfg.detach_results(call); + let mut next_res = 0; let mut abi_res = 0; - // The first result requires special handling. - let first_ty = dfg.value_type(last_res); - if first_ty != get_abi_type(dfg, abi_res).value_type { - // Move the call out of the way, so we can redefine the first result. - let copy = call; - call = dfg.redefine_first_value(pos); - last_res = dfg.first_result(call); - // Set up a closure that can attach new results to `call`. - let mut get_res = |dfg: &mut DataFlowGraph, ty| { - let abi_type = get_abi_type(dfg, abi_res); - if ty == abi_type.value_type { - // Don't append the first result - it's not detachable. - if fixed_results + abi_res == 0 { - *dfg[call].first_type_mut() = ty; - debug_assert_eq!(last_res, dfg.first_result(call)); - } else { - last_res = dfg.append_secondary_result(last_res, ty); - } - abi_res += 1; - Ok(last_res) - } else { - Err(abi_type) - } - }; - - let v = convert_from_abi(dfg, pos, first_ty, &mut get_res); - dfg.replace(copy).copy(v); - } - - // Point immediately after the call and any instructions dealing with the first result. + // Point immediately after the call. pos.next_inst(); - // Now do the secondary results. - while let Some(res) = next_res { - next_res = dfg.next_secondary_result(res); + while let Some(res) = results.get(next_res, &dfg.value_lists) { + next_res += 1; let res_type = dfg.value_type(res); if res_type == get_abi_type(dfg, abi_res).value_type { // No value translation is necessary, this result matches the ABI type. - dfg.attach_secondary_result(last_res, res); - last_res = res; + dfg.attach_result(call, res); abi_res += 1; } else { let mut get_res = |dfg: &mut DataFlowGraph, ty| { let abi_type = get_abi_type(dfg, abi_res); if ty == abi_type.value_type { - last_res = dfg.append_secondary_result(last_res, ty); + let last_res = dfg.append_result(call, ty); abi_res += 1; Ok(last_res) } else { @@ -199,13 +167,18 @@ fn convert_from_abi(dfg: &mut DataFlowGraph, { // Terminate the recursion when we get the desired type. let arg_type = match get_arg(dfg, ty) { - Ok(v) => return v, + Ok(v) => { + debug_assert_eq!(dfg.value_type(v), ty); + return v; + } Err(t) => t, }; // Reconstruct how `ty` was legalized into the `arg_type` argument. let conversion = legalize_abi_value(ty, &arg_type); + dbg!("convert_from_abi({}): {:?}", ty, conversion); + // The conversion describes value to ABI argument. We implement the reverse conversion here. match conversion { // Construct a `ty` by concatenating two ABI integers. @@ -213,6 +186,11 @@ fn convert_from_abi(dfg: &mut DataFlowGraph, let abi_ty = ty.half_width().expect("Invalid type for conversion"); let lo = convert_from_abi(dfg, pos, abi_ty, get_arg); let hi = convert_from_abi(dfg, pos, abi_ty, get_arg); + dbg!("intsplit {}: {}, {}: {}", + lo, + dfg.value_type(lo), + hi, + dfg.value_type(hi)); dfg.ins(pos).iconcat(lo, hi) } // Construct a `ty` by concatenating two halves of a vector. diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index f948b1f586..a501d6aaa3 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1750,12 +1750,12 @@ mod tests { .unwrap(); assert_eq!(func.name.to_string(), "qux"); let v4 = details.map.lookup_str("v4").unwrap(); - assert_eq!(v4.to_string(), "v0"); + assert_eq!(v4.to_string(), "vx0"); let vx3 = details.map.lookup_str("vx3").unwrap(); - assert_eq!(vx3.to_string(), "vx0"); + assert_eq!(vx3.to_string(), "vx2"); let aliased_to = func.dfg .resolve_aliases(Value::table_with_number(0).unwrap()); - assert_eq!(aliased_to.to_string(), "v0"); + assert_eq!(aliased_to.to_string(), "vx0"); } #[test] diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index cbea4b02a2..7f387ccee2 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -243,6 +243,6 @@ mod tests { assert_eq!(map.lookup_str("ebb0").unwrap().to_string(), "ebb0"); assert_eq!(map.lookup_str("v4").unwrap().to_string(), "vx0"); assert_eq!(map.lookup_str("vx7").unwrap().to_string(), "vx1"); - assert_eq!(map.lookup_str("v10").unwrap().to_string(), "v0"); + assert_eq!(map.lookup_str("v10").unwrap().to_string(), "vx2"); } } From e653d44f396e7b099de318a6b4dadddf08825549 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 12:54:11 -0700 Subject: [PATCH 0693/3084] Stop linking result values together. Since results are in a value list, they don't need to form a linked list any longer. - Simplify make_inst_results() to create values in the natural order. - Eliminate the last use of next_secondary_value(). - Delete unused result manipulation methods. --- lib/cretonne/meta/gen_legalizer.py | 15 +- lib/cretonne/src/ir/dfg.rs | 277 +++-------------------------- 2 files changed, 33 insertions(+), 259 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 41cd9bab56..1a8d59d7fa 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -91,17 +91,10 @@ def unwrap_inst(iref, node, fmt): for d in node.defs[1:]: fmt.line('let src_{};'.format(d)) with fmt.indented('{', '}'): - fmt.line( - 'src_{} = dfg.detach_secondary_results(inst).unwrap();' - .format(node.defs[1])) - for i in range(2, len(node.defs)): - fmt.line( - 'src_{} = dfg.next_secondary_result(src_{})' - '.unwrap();' - .format(node.defs[i], node.defs[i - 1])) - fmt.line( - 'assert_eq!(dfg.next_secondary_result(src_{}), None);' - .format(node.defs[-1])) + fmt.line('let r = dfg.inst_results(inst);') + for i in range(1, len(node.defs)): + fmt.line('src_{} = r[{}];'.format(node.defs[i], i)) + fmt.line('dfg.detach_secondary_results(inst);') for d in node.defs[1:]: if d.has_free_typevar(): fmt.line( diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 359a6cd089..e00c1fe951 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -5,9 +5,8 @@ use ir::entities::ExpandedValue; use ir::instructions::{Opcode, InstructionData, CallInfo}; use ir::extfunc::ExtFuncData; use entity_map::{EntityMap, PrimaryEntityData}; -use ir::builder::{InsertBuilder, ReplaceBuilder, InstBuilder}; +use ir::builder::{InsertBuilder, ReplaceBuilder}; use ir::layout::Cursor; -use packed_option::PackedOption; use write::write_operands; use std::fmt; @@ -292,20 +291,11 @@ pub enum ValueDef { // Internal table storage for extended values. #[derive(Clone, Debug)] enum ValueData { - // Value is defined by an instruction, but it is not the first result. - Inst { - ty: Type, - num: u16, // Result number starting from 0. - inst: Inst, - next: PackedOption, // Next result defined by `def`. - }, + // Value is defined by an instruction. + Inst { ty: Type, num: u16, inst: Inst }, // Value is an EBB argument. - Arg { - ty: Type, - num: u16, // Argument number, starting from 0. - ebb: Ebb, - }, + Arg { ty: Type, num: u16, ebb: Ebb }, // Value is an alias of another value. // An alias value can't be linked as an instruction result or EBB argument. It is used as a @@ -397,8 +387,6 @@ impl DataFlowGraph { /// The type of the first result value is also set, even if it was already set in the /// `InstructionData` passed to `make_inst`. If this function is called with a single-result /// instruction, that is the only effect. - /// - /// Returns the number of results produced by the instruction. pub fn make_inst_results(&mut self, inst: Inst, ctrl_typevar: Type) -> usize { let constraints = self.insts[inst].opcode().constraints(); let fixed_results = constraints.fixed_results(); @@ -406,67 +394,27 @@ impl DataFlowGraph { self.results[inst].clear(&mut self.value_lists); - // Additional values form a linked list starting from the second result value. Generate - // the list backwards so we don't have to modify value table entries in place. (This - // causes additional result values to be numbered backwards which is not the aesthetic - // choice, but since it is only visible in extremely rare instructions with 3+ results, - // we don't care). - let mut head = None; - let mut first_type = None; - let mut rev_num = 1; + // The fixed results will appear at the front of the list. + for res_idx in 0..fixed_results { + self.append_result(inst, constraints.result_type(res_idx, ctrl_typevar)); + } // Get the call signature if this is a function call. if let Some(sig) = self.call_signature(inst) { // Create result values corresponding to the call return types. let var_results = self.signatures[sig].return_types.len(); total_results += var_results; - - for res_idx in (0..var_results).rev() { - if let Some(ty) = first_type { - head = Some(self.make_value(ValueData::Inst { - ty: ty, - num: (total_results - rev_num) as u16, - inst: inst, - next: head.into(), - })); - self.results[inst].push(head.unwrap(), &mut self.value_lists); - rev_num += 1; - } - first_type = Some(self.signatures[sig].return_types[res_idx].value_type); + for res_idx in 0..var_results { + let ty = self.signatures[sig].return_types[res_idx].value_type; + self.append_result(inst, ty); } } - // Then the fixed results which will appear at the front of the list. - for res_idx in (0..fixed_results).rev() { - if let Some(ty) = first_type { - head = Some(self.make_value(ValueData::Inst { - ty: ty, - num: (total_results - rev_num) as u16, - inst: inst, - next: head.into(), - })); - rev_num += 1; - self.results[inst].push(head.unwrap(), &mut self.value_lists); - } - first_type = Some(constraints.result_type(res_idx, ctrl_typevar)); + if let Some(v) = self.results[inst].first(&mut self.value_lists) { + let ty = self.value_type(v); + *self[inst].first_type_mut() = ty; } - *self.insts[inst].first_type_mut() = first_type.unwrap_or_default(); - - // Include the first result in the results vector. - if let Some(ty) = first_type { - let v = self.make_value(ValueData::Inst { - ty: ty, - num: 0, - inst: inst, - next: head.into(), - }); - self.results[inst].push(v, &mut self.value_lists); - } - self.results[inst] - .as_mut_slice(&mut self.value_lists) - .reverse(); - total_results } @@ -482,37 +430,18 @@ impl DataFlowGraph { ReplaceBuilder::new(self, inst) } - /// Detach secondary instruction results, and return the first of them. + /// Detach secondary instruction results. /// /// If `inst` produces two or more results, detach these secondary result values from `inst`. /// The first result value cannot be detached. - /// The full list of secondary results can be traversed with `next_secondary_result()`. /// /// Use this method to detach secondary values before using `replace(inst)` to provide an /// alternate instruction for computing the primary result value. - pub fn detach_secondary_results(&mut self, inst: Inst) -> Option { - if !self.has_results(inst) { - return None; + pub fn detach_secondary_results(&mut self, inst: Inst) { + if let Some(first) = self.results[inst].first(&mut self.value_lists) { + self.results[inst].clear(&mut self.value_lists); + self.results[inst].push(first, &mut self.value_lists); } - - let first = self.results[inst].first(&mut self.value_lists).unwrap(); - let second = self.results[inst].get(1, &mut self.value_lists); - self.results[inst].clear(&mut self.value_lists); - self.results[inst].push(first, &mut self.value_lists); - second - } - - /// Get the next secondary result after `value`. - /// - /// Use this function to traverse the full list of instruction results returned from - /// `detach_secondary_results()`. - pub fn next_secondary_result(&self, value: Value) -> Option { - if let ExpandedValue::Table(index) = value.expand() { - if let ValueData::Inst { next, .. } = self.extended_values[index] { - return next.into(); - } - } - panic!("{} is not a secondary result value", value); } /// Detach the list of result values from `inst` and return it. @@ -530,37 +459,17 @@ impl DataFlowGraph { /// This is a very low-level operation. Usually, instruction results with the correct types are /// created automatically. The `res` value must not be attached to anything else. pub fn attach_result(&mut self, inst: Inst, res: Value) { - let res_inst = inst; - if let Some(last_res) = self.results[inst] - .as_slice(&mut self.value_lists) - .last() - .cloned() { - self.attach_secondary_result(last_res, res); + let num = self.results[inst].push(res, &mut self.value_lists); + assert!(num <= u16::MAX as usize, "Too many result values"); + let ty = self.value_type(res); + if let ExpandedValue::Table(idx) = res.expand() { + self.extended_values[idx] = ValueData::Inst { + ty: ty, + num: num as u16, + inst: inst, + }; } else { - // This is the first result. - self.results[res_inst].push(res, &mut self.value_lists); - - // Now update `res` itself. - match res.expand() { - ExpandedValue::Table(idx) => { - if let ValueData::Inst { - ref mut num, - ref mut inst, - ref mut next, - .. - } = self.extended_values[idx] { - *num = 0; - *inst = res_inst; - *next = None.into(); - return; - } - } - ExpandedValue::Direct(inst) => { - assert_eq!(inst, res_inst); - return; - } - } - panic!("{} must be a result", res); + panic!("Unexpected direct value"); } } @@ -570,123 +479,11 @@ impl DataFlowGraph { ty: ty, inst: inst, num: 0, - next: None.into(), }); self.attach_result(inst, res); res } - /// Attach an existing value as a secondary result after `last_res` which must be the last - /// result of an instruction. - /// - /// This is a very low-level operation. Usually, instruction results with the correct types are - /// created automatically. The `res` value must be a secondary instruction result detached from - /// somewhere else. - pub fn attach_secondary_result(&mut self, last_res: Value, res: Value) { - let res_inst = match last_res.expand() { - ExpandedValue::Direct(inst) => inst, - ExpandedValue::Table(idx) => { - if let ValueData::Inst { inst, ref mut next, .. } = self.extended_values[idx] { - assert!(next.is_none(), "last_res is not the last result"); - *next = res.into(); - inst - } else { - panic!("last_res is not an instruction result"); - } - } - }; - - let res_num = self.results[res_inst].push(res, &mut self.value_lists); - assert!(res_num <= u16::MAX as usize, "Too many result values"); - - // Now update `res` itself. - if let ExpandedValue::Table(idx) = res.expand() { - if let ValueData::Inst { - ref mut num, - ref mut inst, - ref mut next, - .. - } = self.extended_values[idx] { - *num = res_num as u16; - *inst = res_inst; - *next = None.into(); - return; - } - } - panic!("{} must be a result", res); - } - - /// Append a new instruction result value after `last_res`. - /// - /// The `last_res` value must be the last value on an instruction. - pub fn append_secondary_result(&mut self, last_res: Value, ty: Type) -> Value { - // The only member that matters is `ty`. The rest is filled in by - // `attach_secondary_result`. - use entity_map::EntityRef; - let res = self.make_value(ValueData::Inst { - ty: ty, - inst: Inst::new(0), - num: 0, - next: None.into(), - }); - self.attach_secondary_result(last_res, res); - res - } - - /// Move the instruction at `pos` to a new `Inst` reference so its first result can be - /// redefined without overwriting the original instruction. - /// - /// The first result value of an instruction is intrinsically tied to the `Inst` reference, so - /// it is not possible to detach the value and attach it to something else. This function - /// copies the instruction pointed to by `pos` to a new `Inst` reference, making the original - /// `Inst` reference available to be redefined with `dfg.replace(inst)` above. - /// - /// Before: - /// - /// inst1: v1, vx2 = foo <-- pos - /// - /// After: - /// - /// inst7: v7, vx2 = foo - /// inst1: v1 = copy v7 <-- pos - /// - /// Returns the new `Inst` reference where the original instruction has been moved. - pub fn redefine_first_value(&mut self, pos: &mut Cursor) -> Inst { - let orig = pos.current_inst() - .expect("Cursor must point at an instruction"); - let first_res = self.first_result(orig); - let first_type = self.value_type(first_res); - let data = self[orig].clone(); - let results = self.results[orig].take(); - self.results[orig].push(first_res, &mut self.value_lists); - - - let new = self.make_inst(data); - let new_first = self.make_value(ValueData::Inst { - ty: first_type, - num: 0, - inst: new, - next: None.into(), - }); - self.results[new].push(new_first, &mut self.value_lists); - - for i in 1.. { - if let Some(v) = results.get(i, &self.value_lists) { - self.attach_result(new, v); - } else { - break; - } - } - - pos.insert_inst(new); - // Replace the original instruction with a copy of the new value. - // This is likely to be immediately overwritten by something else, but this way we avoid - // leaving the DFG in a state with multiple references to secondary results and value - // lists. It also means that this method doesn't change the semantics of the program. - self.replace(orig).copy(new_first); - new - } - /// Get the first result of an instruction. /// /// This function panics if the instruction doesn't have any result. @@ -1043,22 +840,6 @@ mod tests { _ => panic!(), }; - // Redefine the first value out of `iadd_cout`. - assert_eq!(pos.prev_inst(), Some(iadd)); - let new_iadd = dfg.redefine_first_value(pos); - let new_s = dfg.first_result(new_iadd); - assert_eq!(dfg[iadd].opcode(), Opcode::Copy); - assert_eq!(dfg.inst_results(iadd), &[s]); - assert_eq!(dfg.inst_results(new_iadd), &[new_s, c]); - assert_eq!(dfg.resolve_copies(s), new_s); - pos.next_inst(); - - // Detach the 'c' value from `iadd`. - { - assert_eq!(dfg.detach_secondary_results(new_iadd), Some(c)); - assert_eq!(dfg.next_secondary_result(c), None); - } - // Replace `iadd_cout` with a normal `iadd` and an `icmp`. dfg.replace(iadd).iadd(v1, arg0); let c2 = dfg.ins(pos).icmp(IntCC::UnsignedLessThan, s, v1); From 28ff7b792559d441f544950c53b5497bb60b9552 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 13:14:03 -0700 Subject: [PATCH 0694/3084] Move the ctrl_typevar function into dfg. Soon, InstructionData won't have sufficient information to compute this. Give TargetIsa::encode() an explicit ctrl_typevar argument. This function does not require the instruction to be inserted in the DFG tables. --- cranelift/src/filetest/binemit.rs | 4 +++- lib/cretonne/src/ir/dfg.rs | 27 ++++++++++++++++++----- lib/cretonne/src/ir/instructions.rs | 22 ------------------- lib/cretonne/src/isa/arm32/mod.rs | 7 +++--- lib/cretonne/src/isa/arm64/mod.rs | 7 +++--- lib/cretonne/src/isa/intel/mod.rs | 7 +++--- lib/cretonne/src/isa/mod.rs | 8 +++++-- lib/cretonne/src/isa/riscv/mod.rs | 34 +++++++++++++++++++---------- lib/cretonne/src/legalizer/mod.rs | 2 +- lib/cretonne/src/verifier.rs | 2 +- lib/cretonne/src/write.rs | 2 +- 11 files changed, 69 insertions(+), 53 deletions(-) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index f4ec6262c6..449e0887e6 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -110,7 +110,9 @@ impl SubTest for TestBinEmit { .get(inst) .map(|e| e.is_legal()) .unwrap_or(false) { - if let Ok(enc) = isa.encode(&func.dfg, &func.dfg[inst]) { + if let Ok(enc) = isa.encode(&func.dfg, + &func.dfg[inst], + func.dfg.ctrl_typevar(inst)) { *func.encodings.ensure(inst) = enc; } } diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index e00c1fe951..1aebfbb03e 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -1,12 +1,13 @@ //! Data flow graph tracking Instructions, Values, and EBBs. -use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueList, ValueListPool}; -use ir::entities::ExpandedValue; -use ir::instructions::{Opcode, InstructionData, CallInfo}; -use ir::extfunc::ExtFuncData; use entity_map::{EntityMap, PrimaryEntityData}; use ir::builder::{InsertBuilder, ReplaceBuilder}; +use ir::entities::ExpandedValue; +use ir::extfunc::ExtFuncData; +use ir::instructions::{Opcode, InstructionData, CallInfo}; use ir::layout::Cursor; +use ir::types; +use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueList, ValueListPool}; use write::write_operands; use std::fmt; @@ -541,6 +542,22 @@ impl DataFlowGraph { .map(|&arg| arg.value_type) }) } + + /// Get the controlling type variable, or `VOID` if `inst` isn't polymorphic. + pub fn ctrl_typevar(&self, inst: Inst) -> Type { + let constraints = self[inst].opcode().constraints(); + + if !constraints.is_polymorphic() { + types::VOID + } else if constraints.requires_typevar_operand() { + // Not all instruction formats have a designated operand, but in that case + // `requires_typevar_operand()` should never be true. + self.value_type(self[inst].typevar_operand(&self.value_lists) + .expect("Instruction format doesn't have a designated operand, bad opcode.")) + } else { + self.value_type(self.first_result(inst)) + } + } } /// Allow immutable access to instructions via indexing. @@ -688,7 +705,7 @@ impl<'a> fmt::Display for DisplayInst<'a> { } - let typevar = inst.ctrl_typevar(dfg); + let typevar = dfg.ctrl_typevar(self.1); if typevar.is_void() { write!(f, "{}", inst.opcode())?; } else { diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 29e3e6662f..1e1262a507 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -14,7 +14,6 @@ use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef, StackSlot, MemFlags}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32, Uoffset32}; use ir::condcodes::*; use ir::types; -use ir::DataFlowGraph; use entity_list; use ref_slice::{ref_slice, ref_slice_mut}; @@ -397,27 +396,6 @@ impl InstructionData { _ => CallInfo::NotACall, } } - - /// Get the controlling type variable, or `VOID` if this instruction isn't polymorphic. - /// - /// In most cases, the controlling type variable is the same as the first result type, but some - /// opcodes require us to read the type of the designated type variable operand from `dfg`. - pub fn ctrl_typevar(&self, dfg: &DataFlowGraph) -> Type { - let constraints = self.opcode().constraints(); - - if !constraints.is_polymorphic() { - types::VOID - } else if constraints.requires_typevar_operand() { - // Not all instruction formats have a designated operand, but in that case - // `requires_typevar_operand()` should never be true. - dfg.value_type(self.typevar_operand(&dfg.value_lists) - .expect("Instruction format doesn't have a designated operand, bad opcode.")) - } else { - // For locality of reference, we prefer to get the controlling type variable from - // `idata` itself, when possible. - self.first_type() - } - } } /// Information about branch and jump instructions. diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index c9c7f2af5c..b76ffa32cb 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -60,10 +60,11 @@ impl TargetIsa for Isa { } fn encode(&self, - dfg: &ir::DataFlowGraph, - inst: &ir::InstructionData) + _dfg: &ir::DataFlowGraph, + inst: &ir::InstructionData, + ctrl_typevar: ir::Type) -> Result { - lookup_enclist(inst.ctrl_typevar(dfg), + lookup_enclist(ctrl_typevar, inst.opcode(), self.cpumode, &enc_tables::LEVEL2[..]) diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index a8cfe834b9..2499028919 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -53,10 +53,11 @@ impl TargetIsa for Isa { } fn encode(&self, - dfg: &ir::DataFlowGraph, - inst: &ir::InstructionData) + _dfg: &ir::DataFlowGraph, + inst: &ir::InstructionData, + ctrl_typevar: ir::Type) -> Result { - lookup_enclist(inst.ctrl_typevar(dfg), + lookup_enclist(ctrl_typevar, inst.opcode(), &enc_tables::LEVEL1_A64[..], &enc_tables::LEVEL2[..]) diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index b48a119632..60ce3a795d 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -60,10 +60,11 @@ impl TargetIsa for Isa { } fn encode(&self, - dfg: &ir::DataFlowGraph, - inst: &ir::InstructionData) + _dfg: &ir::DataFlowGraph, + inst: &ir::InstructionData, + ctrl_typevar: ir::Type) -> Result { - lookup_enclist(inst.ctrl_typevar(dfg), + lookup_enclist(ctrl_typevar, inst.opcode(), self.cpumode, &enc_tables::LEVEL2[..]) diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 469142ab56..c40a839836 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -46,7 +46,7 @@ pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex}; use binemit::CodeSink; use settings; -use ir::{Function, Inst, InstructionData, DataFlowGraph, Signature}; +use ir::{Function, Inst, InstructionData, DataFlowGraph, Signature, Type}; pub mod riscv; pub mod intel; @@ -141,7 +141,11 @@ pub trait TargetIsa { /// Otherwise, return `None`. /// /// This is also the main entry point for determining if an instruction is legal. - fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Result; + fn encode(&self, + dfg: &DataFlowGraph, + inst: &InstructionData, + ctrl_typevar: Type) + -> Result; /// Get a data structure describing the instruction encodings in this ISA. fn encoding_info(&self) -> EncInfo; diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 7658db392e..b1f399665b 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -11,7 +11,7 @@ use binemit::CodeSink; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, EncInfo, Encoding, Legalize}; -use ir::{Function, Inst, InstructionData, DataFlowGraph, Signature}; +use ir::{Function, Inst, InstructionData, DataFlowGraph, Signature, Type}; #[allow(dead_code)] struct Isa { @@ -60,8 +60,12 @@ impl TargetIsa for Isa { enc_tables::INFO.clone() } - fn encode(&self, dfg: &DataFlowGraph, inst: &InstructionData) -> Result { - lookup_enclist(inst.ctrl_typevar(dfg), + fn encode(&self, + _dfg: &DataFlowGraph, + inst: &InstructionData, + ctrl_typevar: Type) + -> Result { + lookup_enclist(ctrl_typevar, inst.opcode(), self.cpumode, &enc_tables::LEVEL2[..]) @@ -120,7 +124,8 @@ mod tests { }; // ADDI is I/0b00100 - assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst64).unwrap()), "I#04"); + assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst64, types::I64).unwrap()), + "I#04"); // Try to encode iadd_imm.i64 vx1, -10000. let inst64_large = InstructionData::BinaryImm { @@ -131,7 +136,8 @@ mod tests { }; // Immediate is out of range for ADDI. - assert_eq!(isa.encode(&dfg, &inst64_large), Err(isa::Legalize::Expand)); + assert_eq!(isa.encode(&dfg, &inst64_large, types::I64), + Err(isa::Legalize::Expand)); // Create an iadd_imm.i32 which is encodable in RV64. let inst32 = InstructionData::BinaryImm { @@ -142,7 +148,8 @@ mod tests { }; // ADDIW is I/0b00110 - assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32).unwrap()), "I#06"); + assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32, types::I32).unwrap()), + "I#06"); } // Same as above, but for RV32. @@ -167,7 +174,8 @@ mod tests { }; // In 32-bit mode, an i64 bit add should be narrowed. - assert_eq!(isa.encode(&dfg, &inst64), Err(isa::Legalize::Narrow)); + assert_eq!(isa.encode(&dfg, &inst64, types::I64), + Err(isa::Legalize::Narrow)); // Try to encode iadd_imm.i64 vx1, -10000. let inst64_large = InstructionData::BinaryImm { @@ -178,7 +186,8 @@ mod tests { }; // In 32-bit mode, an i64 bit add should be narrowed. - assert_eq!(isa.encode(&dfg, &inst64_large), Err(isa::Legalize::Narrow)); + assert_eq!(isa.encode(&dfg, &inst64_large, types::I64), + Err(isa::Legalize::Narrow)); // Create an iadd_imm.i32 which is encodable in RV32. let inst32 = InstructionData::BinaryImm { @@ -189,7 +198,8 @@ mod tests { }; // ADDI is I/0b00100 - assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32).unwrap()), "I#04"); + assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32, types::I32).unwrap()), + "I#04"); // Create an imul.i32 which is encodable in RV32, but only when use_m is true. let mul32 = InstructionData::Binary { @@ -198,7 +208,8 @@ mod tests { args: [arg32, arg32], }; - assert_eq!(isa.encode(&dfg, &mul32), Err(isa::Legalize::Expand)); + assert_eq!(isa.encode(&dfg, &mul32, types::I32), + Err(isa::Legalize::Expand)); } #[test] @@ -224,6 +235,7 @@ mod tests { ty: types::I32, args: [arg32, arg32], }; - assert_eq!(encstr(&*isa, isa.encode(&dfg, &mul32).unwrap()), "R#10c"); + assert_eq!(encstr(&*isa, isa.encode(&dfg, &mul32, types::I32).unwrap()), + "R#10c"); } } diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 00354822a1..7949aed311 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -64,7 +64,7 @@ pub fn legalize_function(func: &mut Function, cfg: &mut ControlFlowGraph, isa: & split::simplify_branch_arguments(&mut func.dfg, inst); } - match isa.encode(&func.dfg, &func.dfg[inst]) { + match isa.encode(&func.dfg, &func.dfg[inst], func.dfg.ctrl_typevar(inst)) { Ok(encoding) => *func.encodings.ensure(inst) = encoding, Err(action) => { // We should transform the instruction into legal equivalents. diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index d6018a8f5c..0b84ad82bd 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -425,7 +425,7 @@ impl<'a> Verifier<'a> { let ctrl_type = if let Some(value_typeset) = constraints.ctrl_typeset() { // For polymorphic opcodes, determine the controlling type variable first. - let ctrl_type = inst_data.ctrl_typevar(&self.func.dfg); + let ctrl_type = self.func.dfg.ctrl_typevar(inst); if !value_typeset.contains(ctrl_type) { return err!(inst, "has an invalid controlling type {}", ctrl_type); diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index ae4a38284f..023eb4ee7a 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -152,7 +152,7 @@ fn type_suffix(func: &Function, inst: Inst) -> Option { } } - let rtype = inst_data.ctrl_typevar(&func.dfg); + let rtype = func.dfg.ctrl_typevar(inst); assert!(!rtype.is_void(), "Polymorphic instruction must produce a result"); Some(rtype) From b165d9a313faba1aefd38333398344ad059e66a1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 13:43:24 -0700 Subject: [PATCH 0695/3084] Remove the first_type() methods from InstructionData. Also remove the type field from all the variants. The type of the first result value can be recovered from the value table now. --- lib/cretonne/meta/gen_instr.py | 24 +--------- lib/cretonne/src/ir/builder.rs | 6 --- lib/cretonne/src/ir/dfg.rs | 18 ++----- lib/cretonne/src/ir/instructions.rs | 74 ++++------------------------- lib/cretonne/src/isa/riscv/mod.rs | 8 ---- lib/cretonne/src/verifier.rs | 33 ++++--------- lib/reader/src/parser.rs | 34 +------------ 7 files changed, 26 insertions(+), 171 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 4d71e1e5b4..b1b32b7bf3 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -109,16 +109,14 @@ def gen_instruction_data_impl(fmt): the instruction formats: - `pub fn opcode(&self) -> Opcode` - - `pub fn first_type(&self) -> Type` - `pub fn arguments(&self, &pool) -> &[Value]` - `pub fn arguments_mut(&mut self, &pool) -> &mut [Value]` - `pub fn take_value_list(&mut self) -> Option` - `pub fn put_value_list(&mut self, args: ValueList>` """ - # The `opcode` and `first_type` methods simply read the `opcode` and `ty` - # members. This is really a workaround for Rust's enum types missing shared - # members. + # The `opcode` method simply reads the `opcode` members. This is really a + # workaround for Rust's enum types missing shared members. with fmt.indented('impl InstructionData {', '}'): fmt.doc_comment('Get the opcode of this instruction.') with fmt.indented('pub fn opcode(&self) -> Opcode {', '}'): @@ -128,23 +126,6 @@ def gen_instruction_data_impl(fmt): 'InstructionData::{} {{ opcode, .. }} => opcode,' .format(f.name)) - fmt.doc_comment('Type of the first result, or `VOID`.') - with fmt.indented('pub fn first_type(&self) -> Type {', '}'): - with fmt.indented('match *self {', '}'): - for f in InstructionFormat.all_formats: - fmt.line( - 'InstructionData::{} {{ ty, .. }} => ty,' - .format(f.name)) - - fmt.doc_comment('Mutable reference to the type of the first result.') - with fmt.indented( - 'pub fn first_type_mut(&mut self) -> &mut Type {', '}'): - with fmt.indented('match *self {', '}'): - for f in InstructionFormat.all_formats: - fmt.line( - 'InstructionData::{} {{ ref mut ty, .. }} => ty,' - .format(f.name)) - fmt.doc_comment('Get the controlling type variable operand.') with fmt.indented( 'pub fn typevar_operand(&self, pool: &ValueListPool) -> ' @@ -490,7 +471,6 @@ def gen_format_constructor(iform, fmt): with fmt.indented( 'let data = InstructionData::{} {{'.format(iform.name), '};'): fmt.line('opcode: opcode,') - fmt.line('ty: types::VOID,') gen_member_inits(iform, fmt) fmt.line('self.build(data, ctrl_typevar)') diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 312e5144da..0414f7c282 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -117,12 +117,6 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { // The old result values were either detached or non-existent. // Construct new ones. self.dfg.make_inst_results(self.inst, ctrl_typevar); - } else { - // Normally, make_inst_results() would also set the first result type, but we're not - // going to call that, so set it manually. - *self.dfg[self.inst].first_type_mut() = self.dfg - .compute_result_type(self.inst, 0, ctrl_typevar) - .unwrap_or_default(); } (self.inst, self.dfg) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 1aebfbb03e..49392d8bfb 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -132,7 +132,7 @@ impl DataFlowGraph { pub fn value_type(&self, v: Value) -> Type { use ir::entities::ExpandedValue::*; match v.expand() { - Direct(i) => self.insts[i].first_type(), + Direct(_) => panic!("Unexpected direct value"), Table(i) => { match self.extended_values[i] { ValueData::Inst { ty, .. } => ty, @@ -411,11 +411,6 @@ impl DataFlowGraph { } } - if let Some(v) = self.results[inst].first(&mut self.value_lists) { - let ty = self.value_type(v); - *self[inst].first_type_mut() = ty; - } - total_results } @@ -725,10 +720,7 @@ mod tests { fn make_inst() { let mut dfg = DataFlowGraph::new(); - let idata = InstructionData::Nullary { - opcode: Opcode::Iconst, - ty: types::VOID, - }; + let idata = InstructionData::Nullary { opcode: Opcode::Iconst }; let inst = dfg.make_inst(idata); dfg.make_inst_results(inst, types::I32); assert_eq!(inst.to_string(), "inst0"); @@ -739,7 +731,6 @@ mod tests { let immdfg = &dfg; let ins = &immdfg[inst]; assert_eq!(ins.opcode(), Opcode::Iconst); - assert_eq!(ins.first_type(), types::I32); } // Results. @@ -754,10 +745,7 @@ mod tests { fn no_results() { let mut dfg = DataFlowGraph::new(); - let idata = InstructionData::Nullary { - opcode: Opcode::Trap, - ty: types::VOID, - }; + let idata = InstructionData::Nullary { opcode: Opcode::Trap }; let inst = dfg.make_inst(idata); assert_eq!(dfg.display_inst(inst).to_string(), "trap"); diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 1e1262a507..61ed2f1eeb 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -100,161 +100,107 @@ impl FromStr for Opcode { #[derive(Clone, Debug)] #[allow(missing_docs)] pub enum InstructionData { - Nullary { opcode: Opcode, ty: Type }, - Unary { - opcode: Opcode, - ty: Type, - arg: Value, - }, - UnaryImm { - opcode: Opcode, - ty: Type, - imm: Imm64, - }, - UnaryIeee32 { - opcode: Opcode, - ty: Type, - imm: Ieee32, - }, - UnaryIeee64 { - opcode: Opcode, - ty: Type, - imm: Ieee64, - }, - UnarySplit { - opcode: Opcode, - ty: Type, - arg: Value, - }, - Binary { - opcode: Opcode, - ty: Type, - args: [Value; 2], - }, + Nullary { opcode: Opcode }, + Unary { opcode: Opcode, arg: Value }, + UnaryImm { opcode: Opcode, imm: Imm64 }, + UnaryIeee32 { opcode: Opcode, imm: Ieee32 }, + UnaryIeee64 { opcode: Opcode, imm: Ieee64 }, + UnarySplit { opcode: Opcode, arg: Value }, + Binary { opcode: Opcode, args: [Value; 2] }, BinaryImm { opcode: Opcode, - ty: Type, arg: Value, imm: Imm64, }, - BinaryOverflow { - opcode: Opcode, - ty: Type, - args: [Value; 2], - }, - Ternary { - opcode: Opcode, - ty: Type, - args: [Value; 3], - }, - MultiAry { - opcode: Opcode, - ty: Type, - args: ValueList, - }, + BinaryOverflow { opcode: Opcode, args: [Value; 2] }, + Ternary { opcode: Opcode, args: [Value; 3] }, + MultiAry { opcode: Opcode, args: ValueList }, InsertLane { opcode: Opcode, - ty: Type, lane: Uimm8, args: [Value; 2], }, ExtractLane { opcode: Opcode, - ty: Type, lane: Uimm8, arg: Value, }, IntCompare { opcode: Opcode, - ty: Type, cond: IntCC, args: [Value; 2], }, IntCompareImm { opcode: Opcode, - ty: Type, cond: IntCC, arg: Value, imm: Imm64, }, FloatCompare { opcode: Opcode, - ty: Type, cond: FloatCC, args: [Value; 2], }, Jump { opcode: Opcode, - ty: Type, destination: Ebb, args: ValueList, }, Branch { opcode: Opcode, - ty: Type, destination: Ebb, args: ValueList, }, BranchIcmp { opcode: Opcode, - ty: Type, cond: IntCC, destination: Ebb, args: ValueList, }, BranchTable { opcode: Opcode, - ty: Type, arg: Value, table: JumpTable, }, Call { opcode: Opcode, - ty: Type, func_ref: FuncRef, args: ValueList, }, IndirectCall { opcode: Opcode, - ty: Type, sig_ref: SigRef, args: ValueList, }, StackLoad { opcode: Opcode, - ty: Type, stack_slot: StackSlot, offset: Offset32, }, StackStore { opcode: Opcode, - ty: Type, arg: Value, stack_slot: StackSlot, offset: Offset32, }, HeapLoad { opcode: Opcode, - ty: Type, arg: Value, offset: Uoffset32, }, HeapStore { opcode: Opcode, - ty: Type, args: [Value; 2], offset: Uoffset32, }, Load { opcode: Opcode, - ty: Type, flags: MemFlags, arg: Value, offset: Offset32, }, Store { opcode: Opcode, - ty: Type, flags: MemFlags, args: [Value; 2], offset: Offset32, diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index b1f399665b..8bf9d7b732 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -118,7 +118,6 @@ mod tests { // Try to encode iadd_imm.i64 vx1, -10. let inst64 = InstructionData::BinaryImm { opcode: Opcode::IaddImm, - ty: types::I64, arg: arg64, imm: immediates::Imm64::new(-10), }; @@ -130,7 +129,6 @@ mod tests { // Try to encode iadd_imm.i64 vx1, -10000. let inst64_large = InstructionData::BinaryImm { opcode: Opcode::IaddImm, - ty: types::I64, arg: arg64, imm: immediates::Imm64::new(-10000), }; @@ -142,7 +140,6 @@ mod tests { // Create an iadd_imm.i32 which is encodable in RV64. let inst32 = InstructionData::BinaryImm { opcode: Opcode::IaddImm, - ty: types::I32, arg: arg32, imm: immediates::Imm64::new(10), }; @@ -168,7 +165,6 @@ mod tests { // Try to encode iadd_imm.i64 vx1, -10. let inst64 = InstructionData::BinaryImm { opcode: Opcode::IaddImm, - ty: types::I64, arg: arg64, imm: immediates::Imm64::new(-10), }; @@ -180,7 +176,6 @@ mod tests { // Try to encode iadd_imm.i64 vx1, -10000. let inst64_large = InstructionData::BinaryImm { opcode: Opcode::IaddImm, - ty: types::I64, arg: arg64, imm: immediates::Imm64::new(-10000), }; @@ -192,7 +187,6 @@ mod tests { // Create an iadd_imm.i32 which is encodable in RV32. let inst32 = InstructionData::BinaryImm { opcode: Opcode::IaddImm, - ty: types::I32, arg: arg32, imm: immediates::Imm64::new(10), }; @@ -204,7 +198,6 @@ mod tests { // Create an imul.i32 which is encodable in RV32, but only when use_m is true. let mul32 = InstructionData::Binary { opcode: Opcode::Imul, - ty: types::I32, args: [arg32, arg32], }; @@ -232,7 +225,6 @@ mod tests { // Create an imul.i32 which is encodable in RV32M. let mul32 = InstructionData::Binary { opcode: Opcode::Imul, - ty: types::I32, args: [arg32, arg32], }; assert_eq!(encstr(&*isa, isa.encode(&dfg, &mul32, types::I32).unwrap()), diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 0b84ad82bd..7a287a64bd 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -183,23 +183,13 @@ impl<'a> Verifier<'a> { .unwrap_or(0); let total_results = fixed_results + var_results; - if total_results == 0 { - // Instructions with no results have a NULL `first_type()` - let ret_type = inst_data.first_type(); - if ret_type != types::VOID { - return err!(inst, - "instruction with no results expects NULL return type, found {}", - ret_type); - } - } else { - // All result values for multi-valued instructions are created - let got_results = dfg.inst_results(inst).len(); - if got_results != total_results { - return err!(inst, - "expected {} result values, found {}", - total_results, - got_results); - } + // All result values for multi-valued instructions are created + let got_results = dfg.inst_results(inst).len(); + if got_results != total_results { + return err!(inst, + "expected {} result values, found {}", + total_results, + got_results); } self.verify_entity_references(inst) @@ -671,7 +661,6 @@ mod tests { use super::{Verifier, Error}; use ir::Function; use ir::instructions::{InstructionData, Opcode}; - use ir::types; macro_rules! assert_err_with_msg { ($e:expr, $msg:expr) => ( @@ -698,11 +687,9 @@ mod tests { let mut func = Function::new(); let ebb0 = func.dfg.make_ebb(); func.layout.append_ebb(ebb0); - let nullary_with_bad_opcode = func.dfg - .make_inst(InstructionData::Nullary { - opcode: Opcode::Jump, - ty: types::VOID, - }); + let nullary_with_bad_opcode = + func.dfg + .make_inst(InstructionData::Nullary { opcode: Opcode::Jump }); func.layout.append_inst(nullary_with_bad_opcode, ebb0); let verifier = Verifier::new(&func); assert_err_with_msg!(verifier.run(), "instruction format"); diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index a501d6aaa3..dcc76b1b15 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1404,44 +1404,34 @@ impl<'a> Parser<'a> { opcode: Opcode) -> Result { let idata = match opcode.format() { - InstructionFormat::Nullary => { - InstructionData::Nullary { - opcode: opcode, - ty: VOID, - } - } + InstructionFormat::Nullary => InstructionData::Nullary { opcode: opcode }, InstructionFormat::Unary => { InstructionData::Unary { opcode: opcode, - ty: VOID, arg: self.match_value("expected SSA value operand")?, } } InstructionFormat::UnaryImm => { InstructionData::UnaryImm { opcode: opcode, - ty: VOID, imm: self.match_imm64("expected immediate integer operand")?, } } InstructionFormat::UnaryIeee32 => { InstructionData::UnaryIeee32 { opcode: opcode, - ty: VOID, imm: self.match_ieee32("expected immediate 32-bit float operand")?, } } InstructionFormat::UnaryIeee64 => { InstructionData::UnaryIeee64 { opcode: opcode, - ty: VOID, imm: self.match_ieee64("expected immediate 64-bit float operand")?, } } InstructionFormat::UnarySplit => { InstructionData::UnarySplit { opcode: opcode, - ty: VOID, arg: self.match_value("expected SSA value operand")?, } } @@ -1451,7 +1441,6 @@ impl<'a> Parser<'a> { let rhs = self.match_value("expected SSA value second operand")?; InstructionData::Binary { opcode: opcode, - ty: VOID, args: [lhs, rhs], } } @@ -1461,7 +1450,6 @@ impl<'a> Parser<'a> { let rhs = self.match_imm64("expected immediate integer second operand")?; InstructionData::BinaryImm { opcode: opcode, - ty: VOID, arg: lhs, imm: rhs, } @@ -1472,7 +1460,6 @@ impl<'a> Parser<'a> { let rhs = self.match_value("expected SSA value second operand")?; InstructionData::BinaryOverflow { opcode: opcode, - ty: VOID, args: [lhs, rhs], } } @@ -1486,7 +1473,6 @@ impl<'a> Parser<'a> { let false_arg = self.match_value("expected SSA value false operand")?; InstructionData::Ternary { opcode: opcode, - ty: VOID, args: [ctrl_arg, true_arg, false_arg], } } @@ -1494,7 +1480,6 @@ impl<'a> Parser<'a> { let args = self.parse_value_list()?; InstructionData::MultiAry { opcode: opcode, - ty: VOID, args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } } @@ -1504,7 +1489,6 @@ impl<'a> Parser<'a> { let args = self.parse_opt_value_list()?; InstructionData::Jump { opcode: opcode, - ty: VOID, destination: ebb_num, args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } @@ -1516,7 +1500,6 @@ impl<'a> Parser<'a> { let args = self.parse_opt_value_list()?; InstructionData::Branch { opcode: opcode, - ty: VOID, destination: ebb_num, args: args.into_value_list(&[ctrl_arg], &mut ctx.function.dfg.value_lists), } @@ -1531,7 +1514,6 @@ impl<'a> Parser<'a> { let args = self.parse_opt_value_list()?; InstructionData::BranchIcmp { opcode: opcode, - ty: VOID, cond: cond, destination: ebb_num, args: args.into_value_list(&[lhs, rhs], &mut ctx.function.dfg.value_lists), @@ -1545,7 +1527,6 @@ impl<'a> Parser<'a> { let rhs = self.match_value("expected SSA value last operand")?; InstructionData::InsertLane { opcode: opcode, - ty: VOID, lane: lane, args: [lhs, rhs], } @@ -1556,7 +1537,6 @@ impl<'a> Parser<'a> { let lane = self.match_uimm8("expected lane number")?; InstructionData::ExtractLane { opcode: opcode, - ty: VOID, lane: lane, arg: arg, } @@ -1568,7 +1548,6 @@ impl<'a> Parser<'a> { let rhs = self.match_value("expected SSA value second operand")?; InstructionData::IntCompare { opcode: opcode, - ty: VOID, cond: cond, args: [lhs, rhs], } @@ -1580,7 +1559,6 @@ impl<'a> Parser<'a> { let rhs = self.match_imm64("expected immediate second operand")?; InstructionData::IntCompareImm { opcode: opcode, - ty: VOID, cond: cond, arg: lhs, imm: rhs, @@ -1593,7 +1571,6 @@ impl<'a> Parser<'a> { let rhs = self.match_value("expected SSA value second operand")?; InstructionData::FloatCompare { opcode: opcode, - ty: VOID, cond: cond, args: [lhs, rhs], } @@ -1606,7 +1583,6 @@ impl<'a> Parser<'a> { self.match_token(Token::RPar, "expected ')' after arguments")?; InstructionData::Call { opcode: opcode, - ty: VOID, func_ref: func_ref, args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } @@ -1621,7 +1597,6 @@ impl<'a> Parser<'a> { self.match_token(Token::RPar, "expected ')' after arguments")?; InstructionData::IndirectCall { opcode: opcode, - ty: VOID, sig_ref: sig_ref, args: args.into_value_list(&[callee], &mut ctx.function.dfg.value_lists), } @@ -1633,7 +1608,6 @@ impl<'a> Parser<'a> { .and_then(|num| ctx.get_jt(num, &self.loc))?; InstructionData::BranchTable { opcode: opcode, - ty: VOID, arg: arg, table: table, } @@ -1644,7 +1618,6 @@ impl<'a> Parser<'a> { let offset = self.optional_offset32()?; InstructionData::StackLoad { opcode: opcode, - ty: VOID, stack_slot: ss, offset: offset, } @@ -1657,7 +1630,6 @@ impl<'a> Parser<'a> { let offset = self.optional_offset32()?; InstructionData::StackStore { opcode: opcode, - ty: VOID, arg: arg, stack_slot: ss, offset: offset, @@ -1668,7 +1640,6 @@ impl<'a> Parser<'a> { let offset = self.optional_uoffset32()?; InstructionData::HeapLoad { opcode: opcode, - ty: VOID, arg: addr, offset: offset, } @@ -1680,7 +1651,6 @@ impl<'a> Parser<'a> { let offset = self.optional_uoffset32()?; InstructionData::HeapStore { opcode: opcode, - ty: VOID, args: [arg, addr], offset: offset, } @@ -1691,7 +1661,6 @@ impl<'a> Parser<'a> { let offset = self.optional_offset32()?; InstructionData::Load { opcode: opcode, - ty: VOID, flags: flags, arg: addr, offset: offset, @@ -1705,7 +1674,6 @@ impl<'a> Parser<'a> { let offset = self.optional_offset32()?; InstructionData::Store { opcode: opcode, - ty: VOID, flags: flags, args: [arg, addr], offset: offset, From 18b567f88e7d95aa2f42261af4ccc01a8dc6b24d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 14:26:23 -0700 Subject: [PATCH 0696/3084] Flatten the Value reference representation. All values are now references into the value table, so drop the distinction between direct and table values. Direct values don't exist any more. Also remove the parser support for the 'vxNN' syntax. Only 'vNN' values can be parsed now. --- cranelift/docs/cton_lexer.py | 3 +- cranelift/filetests/isa/riscv/abi.cton | 1 - cranelift/filetests/isa/riscv/expand-i32.cton | 2 +- .../filetests/isa/riscv/legalize-abi.cton | 2 +- .../filetests/isa/riscv/legalize-i64.cton | 2 +- cranelift/filetests/isa/riscv/split-args.cton | 2 +- cranelift/filetests/parser/rewrite.cton | 4 +- lib/cretonne/src/ir/dfg.rs | 155 ++++++------------ lib/cretonne/src/ir/entities.rs | 111 +------------ lib/cretonne/src/isa/riscv/mod.rs | 8 +- lib/cretonne/src/write.rs | 8 +- lib/reader/src/lexer.rs | 10 +- lib/reader/src/parser.rs | 50 +++--- lib/reader/src/sourcemap.rs | 19 +-- misc/vim/syntax/cton.vim | 2 +- 15 files changed, 115 insertions(+), 264 deletions(-) diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py index b40cb70e41..1c7224b593 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/cton_lexer.py @@ -47,10 +47,9 @@ class CretonneLexer(RegexLexer): # Well known value types. (r'\b(b\d+|i\d+|f32|f64)(x\d+)?\b', Keyword.Type), # v = value - # vx = value # ss = stack slot # jt = jump table - (r'(vx?|ss|jt)\d+', Name.Variable), + (r'(v|ss|jt)\d+', Name.Variable), # ebb = extended basic block (r'(ebb)\d+', Name.Label), # Match instruction names in context. diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.cton index d75813e220..d168fd768c 100644 --- a/cranelift/filetests/isa/riscv/abi.cton +++ b/cranelift/filetests/isa/riscv/abi.cton @@ -3,7 +3,6 @@ test legalizer isa riscv ; regex: V=v\d+ -; regex: VX=vx\d+ function f(i32) { sig0 = signature(i32) -> i32 diff --git a/cranelift/filetests/isa/riscv/expand-i32.cton b/cranelift/filetests/isa/riscv/expand-i32.cton index 0415c6026c..ce335fb551 100644 --- a/cranelift/filetests/isa/riscv/expand-i32.cton +++ b/cranelift/filetests/isa/riscv/expand-i32.cton @@ -7,7 +7,7 @@ isa riscv supports_m=1 set is_64bit=1 isa riscv supports_m=1 -; regex: V=vx?\d+ +; regex: V=v\d+ function carry_out(i32, i32) -> i32, b1 { ebb0(v1: i32, v2: i32): diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index 06d68768bd..85b06b5755 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -2,7 +2,7 @@ test legalizer isa riscv -; regex: V=vx?\d+ +; regex: V=v\d+ function int_split_args(i64) -> i64 { ebb0(v0: i64): diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton index dbe26f824c..dc8604f358 100644 --- a/cranelift/filetests/isa/riscv/legalize-i64.cton +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -2,7 +2,7 @@ test legalizer isa riscv supports_m=1 -; regex: V=vx?\d+ +; regex: V=v\d+ function bitwise_and(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): diff --git a/cranelift/filetests/isa/riscv/split-args.cton b/cranelift/filetests/isa/riscv/split-args.cton index 8c69378f42..06b09bd6d1 100644 --- a/cranelift/filetests/isa/riscv/split-args.cton +++ b/cranelift/filetests/isa/riscv/split-args.cton @@ -2,7 +2,7 @@ test legalizer isa riscv -; regex: V=vx?\d+ +; regex: V=v\d+ function simple(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): diff --git a/cranelift/filetests/parser/rewrite.cton b/cranelift/filetests/parser/rewrite.cton index ffd1851114..e01391d156 100644 --- a/cranelift/filetests/parser/rewrite.cton +++ b/cranelift/filetests/parser/rewrite.cton @@ -12,13 +12,13 @@ test cat function defs() { ebb100(v20: i32): v1000 = iconst.i32x8 5 - vx200 = f64const 0x4.0p0 + v9200 = f64const 0x4.0p0 trap } ; sameln: function defs() { ; nextln: $ebb100($v20: i32): ; nextln: $v1000 = iconst.i32x8 5 -; nextln: $vx200 = f64const 0x1.0000000000000p2 +; nextln: $v9200 = f64const 0x1.0000000000000p2 ; nextln: trap ; nextln: } diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 49392d8bfb..a6a85ba24c 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -2,7 +2,6 @@ use entity_map::{EntityMap, PrimaryEntityData}; use ir::builder::{InsertBuilder, ReplaceBuilder}; -use ir::entities::ExpandedValue; use ir::extfunc::ExtFuncData; use ir::instructions::{Opcode, InstructionData, CallInfo}; use ir::layout::Cursor; @@ -48,12 +47,8 @@ pub struct DataFlowGraph { /// - EBB arguments in `ebbs`. pub value_lists: ValueListPool, - /// Extended value table. Most `Value` references refer directly to their defining instruction. - /// Others index into this table. - /// - /// This is implemented directly with a `Vec` rather than an `EntityMap` because - /// the Value entity references can refer to two things -- an instruction or an extended value. - extended_values: Vec, + /// Primary value table with entries for all values. + values: EntityMap, /// Function signature table. These signatures are referenced by indirect call instructions as /// well as the external function references. @@ -65,6 +60,7 @@ pub struct DataFlowGraph { impl PrimaryEntityData for InstructionData {} impl PrimaryEntityData for EbbData {} +impl PrimaryEntityData for ValueData {} impl PrimaryEntityData for Signature {} impl PrimaryEntityData for ExtFuncData {} @@ -76,7 +72,7 @@ impl DataFlowGraph { results: EntityMap::new(), ebbs: EntityMap::new(), value_lists: ValueListPool::new(), - extended_values: Vec::new(), + values: EntityMap::new(), signatures: EntityMap::new(), ext_funcs: EntityMap::new(), } @@ -115,31 +111,20 @@ impl DataFlowGraph { impl DataFlowGraph { // Allocate an extended value entry. fn make_value(&mut self, data: ValueData) -> Value { - let vref = Value::new_table(self.extended_values.len()); - self.extended_values.push(data); - vref + self.values.push(data) } /// Check if a value reference is valid. pub fn value_is_valid(&self, v: Value) -> bool { - match v.expand() { - ExpandedValue::Direct(inst) => self.insts.is_valid(inst), - ExpandedValue::Table(index) => index < self.extended_values.len(), - } + self.values.is_valid(v) } /// Get the type of a value. pub fn value_type(&self, v: Value) -> Type { - use ir::entities::ExpandedValue::*; - match v.expand() { - Direct(_) => panic!("Unexpected direct value"), - Table(i) => { - match self.extended_values[i] { - ValueData::Inst { ty, .. } => ty, - ValueData::Arg { ty, .. } => ty, - ValueData::Alias { ty, .. } => ty, - } - } + match self.values[v] { + ValueData::Inst { ty, .. } | + ValueData::Arg { ty, .. } | + ValueData::Alias { ty, .. } => ty, } } @@ -148,31 +133,25 @@ impl DataFlowGraph { /// This is either the instruction that defined it or the Ebb that has the value as an /// argument. pub fn value_def(&self, v: Value) -> ValueDef { - use ir::entities::ExpandedValue::*; - match v.expand() { - Direct(inst) => ValueDef::Res(inst, 0), - Table(idx) => { - match self.extended_values[idx] { - ValueData::Inst { inst, num, .. } => { - assert_eq!(Some(v), - self.results[inst].get(num as usize, &self.value_lists), - "Dangling result value {}: {}", - v, - self.display_inst(inst)); - ValueDef::Res(inst, num as usize) - } - ValueData::Arg { ebb, num, .. } => { - assert_eq!(Some(v), - self.ebbs[ebb].args.get(num as usize, &self.value_lists), - "Dangling EBB argument value"); - ValueDef::Arg(ebb, num as usize) - } - ValueData::Alias { original, .. } => { - // Make sure we only recurse one level. `resolve_aliases` has safeguards to - // detect alias loops without overrunning the stack. - self.value_def(self.resolve_aliases(original)) - } - } + match self.values[v] { + ValueData::Inst { inst, num, .. } => { + assert_eq!(Some(v), + self.results[inst].get(num as usize, &self.value_lists), + "Dangling result value {}: {}", + v, + self.display_inst(inst)); + ValueDef::Res(inst, num as usize) + } + ValueData::Arg { ebb, num, .. } => { + assert_eq!(Some(v), + self.ebbs[ebb].args.get(num as usize, &self.value_lists), + "Dangling EBB argument value"); + ValueDef::Arg(ebb, num as usize) + } + ValueData::Alias { original, .. } => { + // Make sure we only recurse one level. `resolve_aliases` has safeguards to + // detect alias loops without overrunning the stack. + self.value_def(self.resolve_aliases(original)) } } } @@ -181,23 +160,15 @@ impl DataFlowGraph { /// /// Find the original SSA value that `value` aliases. pub fn resolve_aliases(&self, value: Value) -> Value { - use ir::entities::ExpandedValue::Table; let mut v = value; // Note that extended_values may be empty here. - for _ in 0..1 + self.extended_values.len() { - v = match v.expand() { - Table(idx) => { - match self.extended_values[idx] { - ValueData::Alias { original, .. } => { - // Follow alias values. - original - } - _ => return v, - } - } - _ => return v, - }; + for _ in 0..1 + self.values.len() { + if let ValueData::Alias { original, .. } = self.values[v] { + v = original; + } else { + return v; + } } panic!("Value alias loop detected for {}", value); } @@ -233,13 +204,7 @@ impl DataFlowGraph { /// /// Change the `dest` value to behave as an alias of `src`. This means that all uses of `dest` /// will behave as if they used that value `src`. - /// - /// The `dest` value cannot be a direct value defined as the first result of an instruction. To - /// replace a direct value with `src`, its defining instruction should be replaced with a - /// `copy src` instruction. See `replace()`. pub fn change_to_alias(&mut self, dest: Value, src: Value) { - use ir::entities::ExpandedValue::Table; - // Try to create short alias chains by finding the original source value. // This also avoids the creation of loops. let original = self.resolve_aliases(src); @@ -256,14 +221,10 @@ impl DataFlowGraph { self.value_type(dest), ty); - if let Table(idx) = dest.expand() { - self.extended_values[idx] = ValueData::Alias { - ty: ty, - original: original, - }; - } else { - panic!("Cannot change direct value {} into an alias", dest); - } + self.values[dest] = ValueData::Alias { + ty: ty, + original: original, + }; } /// Create a new value alias. @@ -458,15 +419,11 @@ impl DataFlowGraph { let num = self.results[inst].push(res, &mut self.value_lists); assert!(num <= u16::MAX as usize, "Too many result values"); let ty = self.value_type(res); - if let ExpandedValue::Table(idx) = res.expand() { - self.extended_values[idx] = ValueData::Inst { - ty: ty, - num: num as u16, - inst: inst, - }; - } else { - panic!("Unexpected direct value"); - } + self.values[res] = ValueData::Inst { + ty: ty, + num: num as u16, + inst: inst, + }; } /// Append a new instruction result value to `inst`. @@ -609,11 +566,7 @@ impl DataFlowGraph { /// /// Returns the new value. pub fn replace_ebb_arg(&mut self, old_arg: Value, new_type: Type) -> Value { - let old_data = if let ExpandedValue::Table(index) = old_arg.expand() { - self.extended_values[index].clone() - } else { - panic!("old_arg: {} must be an EBB argument", old_arg); - }; + let old_data = self.values[old_arg].clone(); // Create new value identical to the old one except for the type. let (ebb, num) = if let ValueData::Arg { num, ebb, .. } = old_data { @@ -655,12 +608,10 @@ impl DataFlowGraph { // Now update `arg` itself. let arg_ebb = ebb; - if let ExpandedValue::Table(idx) = arg.expand() { - if let ValueData::Arg { ref mut num, ebb, .. } = self.extended_values[idx] { - *num = arg_num as u16; - assert_eq!(arg_ebb, ebb, "{} should already belong to EBB", arg); - return; - } + if let ValueData::Arg { ref mut num, ebb, .. } = self.values[arg] { + *num = arg_num as u16; + assert_eq!(arg_ebb, ebb, "{} should already belong to EBB", arg); + return; } panic!("{} must be an EBB argument value", arg); } @@ -724,7 +675,7 @@ mod tests { let inst = dfg.make_inst(idata); dfg.make_inst_results(inst, types::I32); assert_eq!(inst.to_string(), "inst0"); - assert_eq!(dfg.display_inst(inst).to_string(), "vx0 = iconst.i32"); + assert_eq!(dfg.display_inst(inst).to_string(), "v0 = iconst.i32"); // Immutable reference resolution. { @@ -766,12 +717,12 @@ mod tests { assert_eq!(dfg.ebb_args(ebb), &[]); let arg1 = dfg.append_ebb_arg(ebb, types::F32); - assert_eq!(arg1.to_string(), "vx0"); + assert_eq!(arg1.to_string(), "v0"); assert_eq!(dfg.num_ebb_args(ebb), 1); assert_eq!(dfg.ebb_args(ebb), &[arg1]); let arg2 = dfg.append_ebb_arg(ebb, types::I16); - assert_eq!(arg2.to_string(), "vx1"); + assert_eq!(arg2.to_string(), "v1"); assert_eq!(dfg.num_ebb_args(ebb), 2); assert_eq!(dfg.ebb_args(ebb), &[arg1, arg2]); @@ -835,7 +786,7 @@ mod tests { // Build a little test program. let v1 = dfg.ins(pos).iconst(types::I32, 42); - // Make sure we can resolve value aliases even when extended_values is empty. + // Make sure we can resolve value aliases even when values is empty. assert_eq!(dfg.resolve_aliases(v1), v1); let arg0 = dfg.append_ebb_arg(ebb0, types::I32); diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 876598d13a..813ec978c2 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -76,97 +76,20 @@ impl Ebb { /// An opaque reference to an SSA value. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct Value(u32); -entity_impl!(Value); - -/// Value references can either reference an instruction directly, or they can refer to the -/// extended value table. -pub enum ExpandedValue { - /// This is the first value produced by the referenced instruction. - Direct(Inst), - - /// This value is described in the extended value table. - Table(usize), -} +entity_impl!(Value, "v"); impl Value { - /// Create a `Direct` value from its number representation. + /// Create a value from its number representation. /// This is the number in the `vNN` notation. /// /// This method is for use by the parser. - pub fn direct_with_number(n: u32) -> Option { + pub fn with_number(n: u32) -> Option { if n < u32::MAX / 2 { - let encoding = n * 2; - assert!(encoding < u32::MAX); - Some(Value(encoding)) + Some(Value(n)) } else { None } } - - /// Create a `Table` value from its number representation. - /// This is the number in the `vxNN` notation. - /// - /// This method is for use by the parser. - pub fn table_with_number(n: u32) -> Option { - if n < u32::MAX / 2 { - let encoding = n * 2 + 1; - assert!(encoding < u32::MAX); - Some(Value(encoding)) - } else { - None - } - } - - /// Create a `Direct` value corresponding to the first value produced by `i`. - pub fn new_direct(i: Inst) -> Value { - let encoding = i.index() * 2; - assert!(encoding < u32::MAX as usize); - Value(encoding as u32) - } - - /// Create a `Table` value referring to entry `i` in the `DataFlowGraph.extended_values` table. - /// This constructor should not be used directly. Use the public `DataFlowGraph` methods to - /// manipulate values. - pub fn new_table(index: usize) -> Value { - let encoding = index * 2 + 1; - assert!(encoding < u32::MAX as usize); - Value(encoding as u32) - } - - /// Expand the internal representation into something useful. - pub fn expand(&self) -> ExpandedValue { - use self::ExpandedValue::*; - let index = (self.0 / 2) as usize; - if self.0 % 2 == 0 { - Direct(Inst::new(index)) - } else { - Table(index) - } - } - - /// Assuming that this is a direct value, get the referenced instruction. - /// - /// # Panics - /// - /// If this is not a value created with `new_direct()`. - pub fn unwrap_direct(&self) -> Inst { - if let ExpandedValue::Direct(inst) = self.expand() { - inst - } else { - panic!("{} is not a direct value", self) - } - } -} - -/// Display a `Value` reference as "v7" or "v2x". -impl Display for Value { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - use self::ExpandedValue::*; - match self.expand() { - Direct(i) => write!(fmt, "v{}", i.0), - Table(i) => write!(fmt, "vx{}", i), - } - } } /// An opaque reference to an instruction in a function. @@ -276,32 +199,14 @@ impl From for AnyEntity { mod tests { use super::*; use std::u32; - use entity_map::EntityRef; #[test] fn value_with_number() { - assert_eq!(Value::direct_with_number(0).unwrap().to_string(), "v0"); - assert_eq!(Value::direct_with_number(1).unwrap().to_string(), "v1"); - assert_eq!(Value::table_with_number(0).unwrap().to_string(), "vx0"); - assert_eq!(Value::table_with_number(1).unwrap().to_string(), "vx1"); + assert_eq!(Value::with_number(0).unwrap().to_string(), "v0"); + assert_eq!(Value::with_number(1).unwrap().to_string(), "v1"); - assert_eq!(Value::direct_with_number(u32::MAX / 2), None); - assert_eq!(match Value::direct_with_number(u32::MAX / 2 - 1) - .unwrap() - .expand() { - ExpandedValue::Direct(i) => i.index() as u32, - _ => u32::MAX, - }, - u32::MAX / 2 - 1); - - assert_eq!(Value::table_with_number(u32::MAX / 2), None); - assert_eq!(match Value::table_with_number(u32::MAX / 2 - 1) - .unwrap() - .expand() { - ExpandedValue::Table(i) => i as u32, - _ => u32::MAX, - }, - u32::MAX / 2 - 1); + assert_eq!(Value::with_number(u32::MAX / 2), None); + assert!(Value::with_number(u32::MAX / 2 - 1).is_some()); } #[test] diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 8bf9d7b732..778e3b85aa 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -115,7 +115,7 @@ mod tests { let arg64 = dfg.append_ebb_arg(ebb, types::I64); let arg32 = dfg.append_ebb_arg(ebb, types::I32); - // Try to encode iadd_imm.i64 vx1, -10. + // Try to encode iadd_imm.i64 v1, -10. let inst64 = InstructionData::BinaryImm { opcode: Opcode::IaddImm, arg: arg64, @@ -126,7 +126,7 @@ mod tests { assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst64, types::I64).unwrap()), "I#04"); - // Try to encode iadd_imm.i64 vx1, -10000. + // Try to encode iadd_imm.i64 v1, -10000. let inst64_large = InstructionData::BinaryImm { opcode: Opcode::IaddImm, arg: arg64, @@ -162,7 +162,7 @@ mod tests { let arg64 = dfg.append_ebb_arg(ebb, types::I64); let arg32 = dfg.append_ebb_arg(ebb, types::I32); - // Try to encode iadd_imm.i64 vx1, -10. + // Try to encode iadd_imm.i64 v1, -10. let inst64 = InstructionData::BinaryImm { opcode: Opcode::IaddImm, arg: arg64, @@ -173,7 +173,7 @@ mod tests { assert_eq!(isa.encode(&dfg, &inst64, types::I64), Err(isa::Legalize::Narrow)); - // Try to encode iadd_imm.i64 vx1, -10000. + // Try to encode iadd_imm.i64 v1, -10000. let inst64_large = InstructionData::BinaryImm { opcode: Opcode::IaddImm, arg: arg64, diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 023eb4ee7a..09e5ee6641 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -86,8 +86,8 @@ pub fn write_ebb_header(w: &mut Write, func: &Function, ebb: Ebb) -> Result { // Write out the basic block header, outdented: // // ebb1: - // ebb1(vx1: i32): - // ebb10(vx4: f64, vx5: b1): + // ebb1(v1: i32): + // ebb10(v4: f64, v5: b1): // // If we're writing encoding annotations, shift by 20. @@ -372,10 +372,10 @@ mod tests { f.dfg.append_ebb_arg(ebb, types::I8); assert_eq!(f.to_string(), - "function foo() {\n ss0 = stack_slot 4\n\nebb0(vx0: i8):\n}\n"); + "function foo() {\n ss0 = stack_slot 4\n\nebb0(v0: i8):\n}\n"); f.dfg.append_ebb_arg(ebb, types::F32.by(4).unwrap()); assert_eq!(f.to_string(), - "function foo() {\n ss0 = stack_slot 4\n\nebb0(vx0: i8, vx1: f32x4):\n}\n"); + "function foo() {\n ss0 = stack_slot 4\n\nebb0(v0: i8, v1: f32x4):\n}\n"); } } diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 0e8eaccd0f..14a0fe45a4 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -34,7 +34,7 @@ pub enum Token<'a> { Float(&'a str), // Floating point immediate Integer(&'a str), // Integer immediate Type(types::Type), // i32, f32, b32x4, ... - Value(Value), // v12, vx7 + Value(Value), // v12, v7 Ebb(Ebb), // ebb3 StackSlot(u32), // ss3 JumpTable(u32), // jt2 @@ -306,8 +306,7 @@ impl<'a> Lexer<'a> { // decoded token. fn numbered_entity(prefix: &str, number: u32) -> Option> { match prefix { - "v" => Value::direct_with_number(number).map(|v| Token::Value(v)), - "vx" => Value::table_with_number(number).map(|v| Token::Value(v)), + "v" => Value::with_number(number).map(|v| Token::Value(v)), "ebb" => Ebb::with_number(number).map(|ebb| Token::Ebb(ebb)), "ss" => Some(Token::StackSlot(number)), "jt" => Some(Token::JumpTable(number)), @@ -531,15 +530,14 @@ mod tests { let mut lex = Lexer::new("v0 v00 vx01 ebb1234567890 ebb5234567890 v1x vx1 vxvx4 \ function0 function b1 i32x4 f32x5"); assert_eq!(lex.next(), - token(Token::Value(Value::direct_with_number(0).unwrap()), 1)); + token(Token::Value(Value::with_number(0).unwrap()), 1)); assert_eq!(lex.next(), token(Token::Identifier("v00"), 1)); assert_eq!(lex.next(), token(Token::Identifier("vx01"), 1)); assert_eq!(lex.next(), token(Token::Ebb(Ebb::with_number(1234567890).unwrap()), 1)); assert_eq!(lex.next(), token(Token::Identifier("ebb5234567890"), 1)); assert_eq!(lex.next(), token(Token::Identifier("v1x"), 1)); - assert_eq!(lex.next(), - token(Token::Value(Value::table_with_number(1).unwrap()), 1)); + assert_eq!(lex.next(), token(Token::Identifier("vx1"), 1)); assert_eq!(lex.next(), token(Token::Identifier("vxvx4"), 1)); assert_eq!(lex.next(), token(Token::Identifier("function0"), 1)); assert_eq!(lex.next(), token(Token::Identifier("function"), 1)); diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index dcc76b1b15..ac2754a46c 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1011,7 +1011,7 @@ impl<'a> Parser<'a> { // We need to parse instruction results here because they are shared // between the parsing of value aliases and the parsing of instructions. // - // inst-results ::= Value(v) { "," Value(vx) } + // inst-results ::= Value(v) { "," Value(v) } let results = self.parse_inst_results()?; match self.token() { @@ -1032,7 +1032,7 @@ impl<'a> Parser<'a> { } // Parse parenthesized list of EBB arguments. Returns a vector of (u32, Type) pairs with the - // source vx numbers of the defined values and the defined types. + // source value numbers of the defined values and the defined types. // // ebb-args ::= * "(" ebb-arg { "," ebb-arg } ")" fn parse_ebb_args(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { @@ -1056,19 +1056,19 @@ impl<'a> Parser<'a> { // Parse a single EBB argument declaration, and append it to `ebb`. // - // ebb-arg ::= * Value(vx) ":" Type(t) + // ebb-arg ::= * Value(v) ":" Type(t) // fn parse_ebb_arg(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { - // ebb-arg ::= * Value(vx) ":" Type(t) - let vx = self.match_value("EBB argument must be a value")?; - let vx_location = self.loc; - // ebb-arg ::= Value(vx) * ":" Type(t) + // ebb-arg ::= * Value(v) ":" Type(t) + let v = self.match_value("EBB argument must be a value")?; + let v_location = self.loc; + // ebb-arg ::= Value(v) * ":" Type(t) self.match_token(Token::Colon, "expected ':' after EBB argument")?; - // ebb-arg ::= Value(vx) ":" * Type(t) + // ebb-arg ::= Value(v) ":" * Type(t) let t = self.match_type("expected EBB argument type")?; // Allocate the EBB argument and add the mapping. let value = ctx.function.dfg.append_ebb_arg(ebb, t); - ctx.map.def_value(vx, value, &vx_location) + ctx.map.def_value(v, value, &v_location) } fn parse_value_location(&mut self, ctx: &Context) -> Result { @@ -1147,21 +1147,21 @@ impl<'a> Parser<'a> { // Parse instruction results and return them. // - // inst-results ::= Value(v) { "," Value(vx) } + // inst-results ::= Value(v) { "," Value(v) } // fn parse_inst_results(&mut self) -> Result> { // Result value numbers. let mut results = Vec::new(); // instruction ::= * [inst-results "="] Opcode(opc) ["." Type] ... - // inst-results ::= * Value(v) { "," Value(vx) } + // inst-results ::= * Value(v) { "," Value(v) } if let Some(Token::Value(v)) = self.token() { self.consume(); results.push(v); - // inst-results ::= Value(v) * { "," Value(vx) } + // inst-results ::= Value(v) * { "," Value(v) } while self.optional(Token::Comma) { - // inst-results ::= Value(v) { "," * Value(vx) } + // inst-results ::= Value(v) { "," * Value(v) } results.push(self.match_value("expected result value")?); } } @@ -1171,7 +1171,7 @@ impl<'a> Parser<'a> { // Parse a value alias, and append it to `ebb`. // - // value_alias ::= [inst-results] "->" Value(vx) + // value_alias ::= [inst-results] "->" Value(v) // fn parse_value_alias(&mut self, results: Vec, ctx: &mut Context) -> Result<()> { if results.len() != 1 { @@ -1711,19 +1711,23 @@ mod tests { let (func, details) = Parser::new("function qux() { ebb0: v4 = iconst.i8 6 - vx3 -> v4 - v1 = iadd_imm vx3, 17 + v3 -> v4 + v1 = iadd_imm v3, 17 }") .parse_function(None) .unwrap(); assert_eq!(func.name.to_string(), "qux"); let v4 = details.map.lookup_str("v4").unwrap(); - assert_eq!(v4.to_string(), "vx0"); - let vx3 = details.map.lookup_str("vx3").unwrap(); - assert_eq!(vx3.to_string(), "vx2"); - let aliased_to = func.dfg - .resolve_aliases(Value::table_with_number(0).unwrap()); - assert_eq!(aliased_to.to_string(), "vx0"); + assert_eq!(v4.to_string(), "v0"); + let v3 = details.map.lookup_str("v3").unwrap(); + assert_eq!(v3.to_string(), "v2"); + match v3 { + AnyEntity::Value(v3) => { + let aliased_to = func.dfg.resolve_aliases(v3); + assert_eq!(aliased_to.to_string(), "v0"); + } + _ => panic!("expected value: {}", v3), + } } #[test] @@ -1789,7 +1793,7 @@ mod tests { fn ebb_header() { let (func, _) = Parser::new("function ebbs() { ebb0: - ebb4(vx3: i32): + ebb4(v3: i32): }") .parse_function(None) .unwrap(); diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 7f387ccee2..f84c792d89 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -16,7 +16,7 @@ use lexer::split_entity_name; /// Mapping from source entity names to entity references that are valid in the parsed function. #[derive(Debug)] pub struct SourceMap { - values: HashMap, // vNN, vxNN + values: HashMap, // vNN ebbs: HashMap, // ebbNN stack_slots: HashMap, // ssNN signatures: HashMap, // sigNN @@ -64,12 +64,7 @@ impl SourceMap { pub fn lookup_str(&self, name: &str) -> Option { split_entity_name(name).and_then(|(ent, num)| match ent { "v" => { - Value::direct_with_number(num) - .and_then(|v| self.get_value(v)) - .map(AnyEntity::Value) - } - "vx" => { - Value::table_with_number(num) + Value::with_number(num) .and_then(|v| self.get_value(v)) .map(AnyEntity::Value) } @@ -230,8 +225,8 @@ mod tests { let tf = parse_test("function detail() { ss10 = stack_slot 13 jt10 = jump_table ebb0 - ebb0(v4: i32, vx7: i32): - v10 = iadd v4, vx7 + ebb0(v4: i32, v7: i32): + v10 = iadd v4, v7 }") .unwrap(); let map = &tf.functions[0].1.map; @@ -241,8 +236,8 @@ mod tests { assert_eq!(map.lookup_str("ss10").unwrap().to_string(), "ss0"); assert_eq!(map.lookup_str("jt10").unwrap().to_string(), "jt0"); assert_eq!(map.lookup_str("ebb0").unwrap().to_string(), "ebb0"); - assert_eq!(map.lookup_str("v4").unwrap().to_string(), "vx0"); - assert_eq!(map.lookup_str("vx7").unwrap().to_string(), "vx1"); - assert_eq!(map.lookup_str("v10").unwrap().to_string(), "vx2"); + assert_eq!(map.lookup_str("v4").unwrap().to_string(), "v0"); + assert_eq!(map.lookup_str("v7").unwrap().to_string(), "v1"); + assert_eq!(map.lookup_str("v10").unwrap().to_string(), "v2"); } } diff --git a/misc/vim/syntax/cton.vim b/misc/vim/syntax/cton.vim index 7d6bdab08c..86162d8a50 100644 --- a/misc/vim/syntax/cton.vim +++ b/misc/vim/syntax/cton.vim @@ -18,7 +18,7 @@ syn keyword ctonDecl function stack_slot jump_table syn keyword ctonFilecheck check sameln nextln unordered not regex contained syn match ctonType /\<[bif]\d\+\(x\d\+\)\?\>/ -syn match ctonEntity /\<\(v\|vx\|ss\|jt\|fn\|sig\)\d\+\>/ +syn match ctonEntity /\<\(v\|ss\|jt\|fn\|sig\)\d\+\>/ syn match ctonLabel /\/ syn match ctonName /%\w\+\>/ From 84c15e0e66f11e522165b06748c7b42d7a63539d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 15:12:58 -0700 Subject: [PATCH 0697/3084] Add some safety checks for detached values. These methods are used to reattach detached values: - change_to_alias - attach_result - attach_ebb_arg Add an assertion to all of them to ensure that the provided value is not already attached somewhere else. Use a new value_is_attached() method for the test. Also include a verifier check for uses of detached values. --- lib/cretonne/src/ir/dfg.rs | 56 +++++++++++++++++++++++++++--------- lib/cretonne/src/verifier.rs | 7 ++++- 2 files changed, 48 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index a6a85ba24c..16af22cef6 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -156,6 +156,21 @@ impl DataFlowGraph { } } + /// Determine if `v` is an attached instruction result / EBB argument. + /// + /// An attached value can't be attached to something else without first being detached. + /// + /// Value aliases are not considered to be attached to anything. Use `resolve_aliases()` to + /// determine if the original aliased value is attached. + pub fn value_is_attached(&self, v: Value) -> bool { + use self::ValueData::*; + match self.values[v] { + Inst { inst, num, .. } => Some(&v) == self.inst_results(inst).get(num as usize), + Arg { ebb, num, .. } => Some(&v) == self.ebb_args(ebb).get(num as usize), + Alias { .. } => false, + } + } + /// Resolve value aliases. /// /// Find the original SSA value that `value` aliases. @@ -204,7 +219,10 @@ impl DataFlowGraph { /// /// Change the `dest` value to behave as an alias of `src`. This means that all uses of `dest` /// will behave as if they used that value `src`. + /// + /// The `dest` value can't be attached to an instruction or EBB. pub fn change_to_alias(&mut self, dest: Value, src: Value) { + assert!(!self.value_is_attached(dest)); // Try to create short alias chains by finding the original source value. // This also avoids the creation of loops. let original = self.resolve_aliases(src); @@ -409,6 +427,15 @@ impl DataFlowGraph { self.results[inst].take() } + /// Clear the list of result values from `inst`. + /// + /// This leaves `inst` without any result values. New result values can be created by calling + /// `make_inst_results` or by using a `replace(inst)` builder. + pub fn clear_results(&mut self, inst: Inst) { + self.results[inst].clear(&mut self.value_lists) + } + + /// Attach an existing value to the result value list for `inst`. /// /// The `res` value is appended to the end of the result list. @@ -416,6 +443,7 @@ impl DataFlowGraph { /// This is a very low-level operation. Usually, instruction results with the correct types are /// created automatically. The `res` value must not be attached to anything else. pub fn attach_result(&mut self, inst: Inst, res: Value) { + assert!(!self.value_is_attached(res)); let num = self.results[inst].push(res, &mut self.value_lists); assert!(num <= u16::MAX as usize, "Too many result values"); let ty = self.value_type(res); @@ -597,23 +625,19 @@ impl DataFlowGraph { /// Append an existing argument value to `ebb`. /// - /// The appended value should already be an EBB argument belonging to `ebb`, but it can't be - /// attached. In practice, this means that it should be one of the values returned from - /// `detach_ebb_args()`. + /// The appended value can't already be attached to something else. /// /// In almost all cases, you should be using `append_ebb_arg()` instead of this method. pub fn attach_ebb_arg(&mut self, ebb: Ebb, arg: Value) { - let arg_num = self.ebbs[ebb].args.push(arg, &mut self.value_lists); - assert!(arg_num <= u16::MAX as usize, "Too many arguments to EBB"); - - // Now update `arg` itself. - let arg_ebb = ebb; - if let ValueData::Arg { ref mut num, ebb, .. } = self.values[arg] { - *num = arg_num as u16; - assert_eq!(arg_ebb, ebb, "{} should already belong to EBB", arg); - return; - } - panic!("{} must be an EBB argument value", arg); + assert!(!self.value_is_attached(arg)); + let num = self.ebbs[ebb].args.push(arg, &mut self.value_lists); + assert!(num <= u16::MAX as usize, "Too many arguments to EBB"); + let ty = self.value_type(arg); + self.values[arg] = ValueData::Arg { + ty: ty, + num: num as u16, + ebb: ebb, + }; } } @@ -796,6 +820,10 @@ mod tests { _ => panic!(), }; + // Remove `c` from the result list. + dfg.clear_results(iadd); + dfg.attach_result(iadd, s); + // Replace `iadd_cout` with a normal `iadd` and an `icmp`. dfg.replace(iadd).iadd(v1, arg0); let c2 = dfg.ins(pos).icmp(IntCC::UnsignedLessThan, s, v1); diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index 7a287a64bd..ba2f03ec9d 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -13,7 +13,6 @@ //! //! - The instruction format must match the opcode. //! - All result values must be created for multi-valued instructions. -//! - Instructions with no results must have a VOID `first_type()`. //! - All referenced entities must exist. (Values, EBBs, stack slots, ...) //! //! SSA form @@ -200,6 +199,12 @@ impl<'a> Verifier<'a> { for &arg in self.func.dfg.inst_args(inst) { self.verify_value(inst, arg)?; + + // All used values must be attached to something. + let original = self.func.dfg.resolve_aliases(arg); + if !self.func.dfg.value_is_attached(original) { + return err!(inst, "argument {} -> {} is not attached", arg, original); + } } for &res in self.func.dfg.inst_results(inst) { From 801f4ade66874a4d2847782cf619d5079f67dd6c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 17:06:27 -0700 Subject: [PATCH 0698/3084] Add remove_inst() methods to Cursor and Layout. We didn't have a way of removing instructions again. --- lib/cretonne/src/ir/layout.rs | 48 +++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 4f5d2355b2..aec53422f7 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -460,6 +460,32 @@ impl Layout { self.assign_inst_seq(inst); } + /// Remove `inst` from the layout. + pub fn remove_inst(&mut self, inst: Inst) { + let ebb = self.inst_ebb(inst) + .expect("Instruction already removed."); + // Clear the `inst` node and extract links. + let prev; + let next; + { + let n = &mut self.insts[inst]; + prev = n.prev; + next = n.next; + n.ebb = None.into(); + n.prev = None.into(); + n.next = None.into(); + } + // Fix up links to `inst`. + match prev.expand() { + None => self.ebbs[ebb].first_inst = next, + Some(p) => self.insts[p].next = next, + } + match next.expand() { + None => self.ebbs[ebb].last_inst = prev, + Some(n) => self.insts[n].prev = prev, + } + } + /// Iterate over the instructions in `ebb` in layout order. pub fn ebb_insts<'f>(&'f self, ebb: Ebb) -> Insts<'f> { Insts { @@ -881,6 +907,15 @@ impl<'f> Cursor<'f> { } } + /// Remove the instruction under the cursor. + /// + /// The cursor is left pointing at the position following the current instruction. + pub fn remove_inst(&mut self) { + let inst = self.current_inst().expect("No instruction to remove"); + self.next_inst(); + self.layout.remove_inst(inst); + } + /// Insert an EBB at the current position and switch to it. /// /// As far as possible, this method behaves as if the EBB header were an instruction inserted @@ -1142,6 +1177,19 @@ mod tests { assert_eq!(cur.prev_inst(), Some(i1)); assert_eq!(cur.prev_inst(), None); assert_eq!(cur.position(), CursorPosition::Before(e1)); + + // Test remove_inst. + cur.goto_inst(i2); + cur.remove_inst(); + verify(cur.layout, &[(e1, &[i1, i0])]); + assert_eq!(cur.layout.inst_ebb(i2), None); + cur.remove_inst(); + verify(cur.layout, &[(e1, &[i1])]); + assert_eq!(cur.layout.inst_ebb(i0), None); + assert_eq!(cur.position(), CursorPosition::After(e1)); + cur.layout.remove_inst(i1); + verify(cur.layout, &[(e1, &[])]); + assert_eq!(cur.layout.inst_ebb(i1), None); } #[test] From d4f511ecfcdc28348c6ce7383d8918020e5fc911 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 13 Apr 2017 08:18:16 -0700 Subject: [PATCH 0699/3084] Return the removed instruction from Cursor::remove_inst(). This is convenient for asserting that the right instruction was removed. --- lib/cretonne/src/ir/layout.rs | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index aec53422f7..6e3e793125 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -910,10 +910,13 @@ impl<'f> Cursor<'f> { /// Remove the instruction under the cursor. /// /// The cursor is left pointing at the position following the current instruction. - pub fn remove_inst(&mut self) { + /// + /// Return the instruction that was removed. + pub fn remove_inst(&mut self) -> Inst { let inst = self.current_inst().expect("No instruction to remove"); self.next_inst(); self.layout.remove_inst(inst); + inst } /// Insert an EBB at the current position and switch to it. @@ -1180,10 +1183,10 @@ mod tests { // Test remove_inst. cur.goto_inst(i2); - cur.remove_inst(); + assert_eq!(cur.remove_inst(), i2); verify(cur.layout, &[(e1, &[i1, i0])]); assert_eq!(cur.layout.inst_ebb(i2), None); - cur.remove_inst(); + assert_eq!(cur.remove_inst(), i0); verify(cur.layout, &[(e1, &[i1])]); assert_eq!(cur.layout.inst_ebb(i0), None); assert_eq!(cur.position(), CursorPosition::After(e1)); From 3197f240ff4849808d2c54ff1445758ce662b044 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 13 Apr 2017 08:15:44 -0700 Subject: [PATCH 0700/3084] Add a make_inst_results_reusing() generic method. This is a generalization of the existing make_inst_results() which lets you provide some or all of the result values instead of creating all new ones. --- lib/cretonne/src/ir/dfg.rs | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 16af22cef6..ad22a03fa7 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -10,6 +10,7 @@ use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueList, ValueLis use write::write_operands; use std::fmt; +use std::iter; use std::ops::{Index, IndexMut}; use std::u16; @@ -368,6 +369,22 @@ impl DataFlowGraph { /// `InstructionData` passed to `make_inst`. If this function is called with a single-result /// instruction, that is the only effect. pub fn make_inst_results(&mut self, inst: Inst, ctrl_typevar: Type) -> usize { + self.make_inst_results_reusing(inst, ctrl_typevar, iter::empty()) + } + + /// Create result values for `inst`, reusing the provided detached values. + /// + /// Create a new set of result values for `inst` using `ctrl_typevar` to determine the result + /// types. Any values provided by `reuse` will be reused. When `reuse` is exhausted or when it + /// produces `None`, a new value is created. + pub fn make_inst_results_reusing(&mut self, + inst: Inst, + ctrl_typevar: Type, + reuse: I) + -> usize + where I: Iterator> + { + let mut reuse = reuse.fuse(); let constraints = self.insts[inst].opcode().constraints(); let fixed_results = constraints.fixed_results(); let mut total_results = fixed_results; @@ -376,7 +393,13 @@ impl DataFlowGraph { // The fixed results will appear at the front of the list. for res_idx in 0..fixed_results { - self.append_result(inst, constraints.result_type(res_idx, ctrl_typevar)); + let ty = constraints.result_type(res_idx, ctrl_typevar); + if let Some(Some(v)) = reuse.next() { + debug_assert_eq!(self.value_type(v), ty, "Reused {} is wrong type", ty); + self.attach_result(inst, v); + } else { + self.append_result(inst, ty); + } } // Get the call signature if this is a function call. @@ -386,7 +409,12 @@ impl DataFlowGraph { total_results += var_results; for res_idx in 0..var_results { let ty = self.signatures[sig].return_types[res_idx].value_type; - self.append_result(inst, ty); + if let Some(Some(v)) = reuse.next() { + debug_assert_eq!(self.value_type(v), ty, "Reused {} is wrong type", ty); + self.attach_result(inst, v); + } else { + self.append_result(inst, ty); + } } } From 48b2f421b4151435cbdf84b3ed0ef8fa9e904e5c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 13 Apr 2017 08:50:32 -0700 Subject: [PATCH 0701/3084] Add a with_results() method to the InsertBuilder. This makes it possible to reuse one or more result values in the instruction that is being inserted. Also add a with_result(v) method for the common case of reusing a single result value. This could be specialized in the future. --- lib/cretonne/src/ir/builder.rs | 87 +++++++++++++++++++++++++++++++++- 1 file changed, 86 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 0414f7c282..d9ef0faa25 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -58,6 +58,36 @@ impl<'c, 'fc, 'fd> InsertBuilder<'c, 'fc, 'fd> { -> InsertBuilder<'c, 'fc, 'fd> { InsertBuilder { dfg: dfg, pos: pos } } + + /// Reuse result values in `reuse`. + /// + /// Convert this builder into one that will reuse the provided result values instead of + /// allocating new ones. The provided values for reuse must not be attached to anything. Any + /// missing result values will be allocated as normal. + /// + /// The `reuse` argument is expected to be an array of `Option`. + pub fn with_results(self, reuse: Array) -> InsertReuseBuilder<'c, 'fc, 'fd, Array> + where Array: AsRef<[Option]> + { + InsertReuseBuilder { + dfg: self.dfg, + pos: self.pos, + reuse: reuse, + } + } + + /// Reuse a single result value. + /// + /// Convert this into a builder that will reuse `v` as the single result value. The reused + /// result value `v` must not be attached to anything. + /// + /// This method should only be used when building an instruction with exactly one result. Use + /// `with_results()` for the more general case. + pub fn with_result(self, v: Value) -> InsertReuseBuilder<'c, 'fc, 'fd, [Option; 1]> { + // TODO: Specialize this to return a different builder that just attaches `v` instead of + // calling `make_inst_results_reusing()`. + self.with_results([Some(v)]) + } } impl<'c, 'fc, 'fd> InstBuilderBase<'fd> for InsertBuilder<'c, 'fc, 'fd> { @@ -77,6 +107,37 @@ impl<'c, 'fc, 'fd> InstBuilderBase<'fd> for InsertBuilder<'c, 'fc, 'fd> { } } +/// Builder that inserts a new instruction like `InsertBuilder`, but reusing result values. +pub struct InsertReuseBuilder<'c, 'fc: 'c, 'fd, Array> + where Array: AsRef<[Option]> +{ + pos: &'c mut Cursor<'fc>, + dfg: &'fd mut DataFlowGraph, + reuse: Array, +} + +impl<'c, 'fc, 'fd, Array> InstBuilderBase<'fd> for InsertReuseBuilder<'c, 'fc, 'fd, Array> + where Array: AsRef<[Option]> +{ + fn data_flow_graph(&self) -> &DataFlowGraph { + self.dfg + } + + fn data_flow_graph_mut(&mut self) -> &mut DataFlowGraph { + self.dfg + } + + fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'fd mut DataFlowGraph) { + let inst = self.dfg.make_inst(data); + // Make an `Interator>`. + let ru = self.reuse.as_ref().iter().cloned(); + self.dfg + .make_inst_results_reusing(inst, ctrl_typevar, ru); + self.pos.insert_inst(inst); + (inst, self.dfg) + } +} + /// Instruction builder that replaces an existing instruction. /// /// The inserted instruction will have the same `Inst` number as the old one. @@ -125,7 +186,7 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { #[cfg(test)] mod tests { - use ir::{Function, Cursor, InstBuilder}; + use ir::{Function, Cursor, InstBuilder, ValueDef}; use ir::types::*; use ir::condcodes::*; @@ -150,4 +211,28 @@ mod tests { let cmp = dfg.ins(pos).icmp(IntCC::Equal, arg0, v0); assert_eq!(dfg.value_type(cmp), B1); } + + #[test] + fn reuse_results() { + let mut func = Function::new(); + let dfg = &mut func.dfg; + let ebb0 = dfg.make_ebb(); + let arg0 = dfg.append_ebb_arg(ebb0, I32); + let pos = &mut Cursor::new(&mut func.layout); + pos.insert_ebb(ebb0); + + let v0 = dfg.ins(pos).iadd_imm(arg0, 17); + assert_eq!(dfg.value_type(v0), I32); + let iadd = pos.prev_inst().unwrap(); + assert_eq!(dfg.value_def(v0), ValueDef::Res(iadd, 0)); + + // Detach v0 and reuse it for a different instruction. + dfg.clear_results(iadd); + let v0b = dfg.ins(pos).with_result(v0).iconst(I32, 3); + assert_eq!(v0, v0b); + assert_eq!(pos.current_inst(), Some(iadd)); + let iconst = pos.prev_inst().unwrap(); + assert!(iadd != iconst); + assert_eq!(dfg.value_def(v0), ValueDef::Res(iconst, 0)); + } } From d48f79aa7208450fdd2d9ef65105dcdc3c4d1fa7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Apr 2017 17:19:28 -0700 Subject: [PATCH 0702/3084] Avoid creating aliases when expanding legalizer patterns. Now that we can detach and reuse all values, there is no longer a need to create a lot of alias values during pattern expansion. Instead, reuse the values from the source pattern when emitting instructions in the destination pattern. If a destination instruction produces the exact same values as a source instruction, simply leave the values attached and replace the instruction it. Otherwise, detach the source values, reuse them in the expansion, and remove the source instruction afterwards. --- cranelift/filetests/isa/riscv/expand-i32.cton | 4 +- lib/cretonne/meta/gen_legalizer.py | 86 +++++++++---------- 2 files changed, 43 insertions(+), 47 deletions(-) diff --git a/cranelift/filetests/isa/riscv/expand-i32.cton b/cranelift/filetests/isa/riscv/expand-i32.cton index ce335fb551..fe93521567 100644 --- a/cranelift/filetests/isa/riscv/expand-i32.cton +++ b/cranelift/filetests/isa/riscv/expand-i32.cton @@ -15,9 +15,7 @@ ebb0(v1: i32, v2: i32): return v3, v4 } ; check: $v3 = iadd $v1, $v2 -; check: $(cout=$V) = icmp ult $v3, $v1 -; It's possible the legalizer will rewrite these value aliases in the future. -; check: $v4 -> $cout +; check: $v4 = icmp ult $v3, $v1 ; check: return $v3, $v4 ; Expanding illegal immediate constants. diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 1a8d59d7fa..1376e3bc21 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -22,7 +22,7 @@ except ImportError: def unwrap_inst(iref, node, fmt): - # type: (str, Def, Formatter) -> None + # type: (str, Def, Formatter) -> bool """ Given a `Def` node, emit code that extracts all the instruction fields from `dfg[iref]`. @@ -31,7 +31,8 @@ def unwrap_inst(iref, node, fmt): :param iref: Name of the `Inst` reference to unwrap. :param node: `Def` node providing variable names. - + :returns: True if the instruction arguments were not detached, expecting a + replacement instruction to overwrite the original. """ fmt.comment('Unwrap {}'.format(node)) expr = node.expr @@ -76,31 +77,35 @@ def unwrap_inst(iref, node, fmt): if isinstance(v, Var) and v.has_free_typevar(): fmt.line('let typeof_{0} = dfg.value_type({0});'.format(v)) - # If the node has multiple results, detach the values. - # Place the secondary values in 'src_{}' locals. - if len(node.defs) > 1: + # If the node has results, detach the values. + # Place the values in locals. + replace_inst = False + if len(node.defs) > 0: if node.defs == node.defs[0].dst_def.defs: # Special case: The instruction replacing node defines the exact # same values. fmt.comment( - 'Multiple results handled by {}.' + 'Results handled by {}.' .format(node.defs[0].dst_def)) + replace_inst = True else: - fmt.comment('Detaching secondary results.') - # Boring case: Detach the secondary values, capture them in locals. - for d in node.defs[1:]: - fmt.line('let src_{};'.format(d)) + # Boring case: Detach the result values, capture them in locals. + fmt.comment('Detaching results.') + for d in node.defs: + fmt.line('let {};'.format(d)) with fmt.indented('{', '}'): fmt.line('let r = dfg.inst_results(inst);') - for i in range(1, len(node.defs)): - fmt.line('src_{} = r[{}];'.format(node.defs[i], i)) - fmt.line('dfg.detach_secondary_results(inst);') - for d in node.defs[1:]: + for i in range(len(node.defs)): + fmt.line('{} = r[{}];'.format(node.defs[i], i)) + fmt.line('dfg.clear_results(inst);') + for d in node.defs: if d.has_free_typevar(): fmt.line( - 'let typeof_{0} = dfg.value_type(src_{0});' + 'let typeof_{0} = dfg.value_type({0});' .format(d)) + return replace_inst + def wrap_tup(seq): # type: (Sequence[object]) -> str @@ -125,9 +130,7 @@ def is_value_split(node): def emit_dst_inst(node, fmt): # type: (Def, Formatter) -> None - exact_replace = False replaced_inst = None # type: str - fixup_first_result = False if is_value_split(node): # Split instructions are not emitted with the builder, but by calling @@ -146,21 +149,28 @@ def emit_dst_inst(node, fmt): builder = 'dfg.ins(pos)' else: src_def0 = node.defs[0].src_def - if src_def0 and node.defs[0] == src_def0.defs[0]: - # The primary result is replacing the primary result of the - # source pattern. + if src_def0 and node.defs == src_def0.defs: + # The replacement instruction defines the exact same values as + # the source pattern. Unwrapping would have left the results + # intact. # Replace the whole instruction. builder = 'let {} = dfg.replace(inst)'.format( wrap_tup(node.defs)) replaced_inst = 'inst' - # Secondary values weren't replaced if this is an exact - # replacement for all the source results. - exact_replace = (node.defs == src_def0.defs) else: - # Insert a new instruction since its primary def doesn't match - # the source. + # Insert a new instruction. builder = 'let {} = dfg.ins(pos)'.format(wrap_tup(node.defs)) - fixup_first_result = node.defs[0].is_output() + # We may want to reuse some of the detached output values. + if len(node.defs) == 1 and node.defs[0].is_output(): + # Reuse the single source result value. + builder += '.with_result({})'.format(node.defs[0]) + elif any(d.is_output() for d in node.defs): + # We have some output values to be reused. + array = ', '.join( + ('Some({})'.format(d) if d.is_output() + else 'None') + for d in node.defs) + builder += '.with_results([{}])'.format(array) fmt.line('{}.{};'.format(builder, node.expr.rust_builder(node.defs))) @@ -172,23 +182,6 @@ def emit_dst_inst(node, fmt): .format(replaced_inst), '}'): fmt.line('pos.next_inst();') - # Fix up any output vars. - if fixup_first_result: - # The first result of the instruction just inserted is an output var, - # but it was not a primary result in the source pattern. - # We need to change the original value to an alias of the primary one - # we just inserted. - fmt.line('dfg.change_to_alias(src_{0}, {0});'.format(node.defs[0])) - - if not exact_replace: - # We don't support secondary values as outputs yet. Depending on the - # source value, we would need to : - # 1. For a primary source value, replace with a copy instruction. - # 2. For a secondary source value, request that the builder reuses the - # value when making secondary result nodes. - for d in node.defs[1:]: - assert not d.is_output() - def gen_xform(xform, fmt): # type: (XForm, Formatter) -> None @@ -202,7 +195,7 @@ def gen_xform(xform, fmt): """ # Unwrap the source instruction, create local variables for the input # variables. - unwrap_inst('inst', xform.src.rtl[0], fmt) + replace_inst = unwrap_inst('inst', xform.src.rtl[0], fmt) # We could support instruction predicates, but not yet. Should we just # return false if it fails? What about multiple patterns with different @@ -214,6 +207,11 @@ def gen_xform(xform, fmt): for dst in xform.dst.rtl: emit_dst_inst(dst, fmt) + # Delete the original instruction if we didn't have an opportunity to + # replace it. + if not replace_inst: + fmt.line('assert_eq!(pos.remove_inst(), inst);') + def gen_xform_group(xgrp, fmt): # type: (XFormGroup, Formatter) -> None From 7ad882c154e70bce5c346447130f63c965847160 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 13 Apr 2017 09:45:36 -0700 Subject: [PATCH 0703/3084] Avoid creating value aliases in legalizer/split.rs. When we're splitting an EBB argument, we insert a iconcat/vconcat instruction that computes the original value from the new split arguments. The concat instruction can now define the original value directly, it is not necessary to define a new value and alias the old one. --- lib/cretonne/src/legalizer/split.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index 3c03d1d77b..764448b5bc 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -228,9 +228,9 @@ fn split_value(dfg: &mut DataFlowGraph, // need to insert a split instruction before returning. pos.goto_top(ebb); pos.next_inst(); - let concat_inst = dfg.ins(pos).Binary(concat, split_type, lo, hi).0; - let concat_val = dfg.first_result(concat_inst); - dfg.change_to_alias(value, concat_val); + dfg.ins(pos) + .with_result(value) + .Binary(concat, split_type, lo, hi); // Finally, splitting the EBB argument is not enough. We also have to repair all // of the predecessor instructions that branch here. From 3c8bdba15ead65d41b9a65dd1a29482b363676af Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 13 Apr 2017 10:16:58 -0700 Subject: [PATCH 0704/3084] Don't create value aliases when legalizing ABI boundaries. When converting from ABI types to original program types, the final conversion instruction can place its result into the original value, so it doesn't need to be changed to an alias. --- .../filetests/isa/riscv/legalize-abi.cton | 25 +++++------ lib/cretonne/src/legalizer/boundary.rs | 45 +++++++++++-------- 2 files changed, 38 insertions(+), 32 deletions(-) diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index 85b06b5755..88d35e699a 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -7,7 +7,7 @@ isa riscv function int_split_args(i64) -> i64 { ebb0(v0: i64): ; check: $ebb0($(v0l=$V): i32, $(v0h=$V): i32): - ; check: iconcat $v0l, $v0h + ; check: $v0 = iconcat $v0l, $v0h v1 = iadd_imm v0, 1 ; check: $(v1l=$V), $(v1h=$V) = isplit $v1 ; check: return $v1l, $v1h @@ -33,10 +33,9 @@ ebb0: v1 = call fn1() ; check: $ebb0: ; nextln: $(v1l=$V), $(v1h=$V) = call $fn1() - ; check: $(v1new=$V) = iconcat $v1l, $v1h + ; check: $v1 = iconcat $v1l, $v1h jump ebb1(v1) - ; The v1 alias gets resolved by split::simplify_branch_arguments(). - ; check: jump $ebb1($v1new) + ; check: jump $ebb1($v1) ebb1(v10: i64): jump ebb1(v10) @@ -49,10 +48,9 @@ ebb0: v1, v2 = call fn1() ; check: $ebb0: ; nextln: $v1, $(v2l=$V), $(v2h=$V) = call $fn1() - ; check: $(v2new=$V) = iconcat $v2l, $v2h + ; check: $v2 = iconcat $v2l, $v2h jump ebb1(v1, v2) - ; The v2 -> v2new alias is resolved by split::simplify_branch_arguments(). - ; check: jump $ebb1($v1, $v2new) + ; check: jump $ebb1($v1, $v2) ebb1(v9: i32, v10: i64): jump ebb1(v9, v10) @@ -61,8 +59,8 @@ ebb1(v9: i32, v10: i64): function int_ext(i8, i8 sext, i8 uext) -> i8 uext { ebb0(v1: i8, v2: i8, v3: i8): ; check: $ebb0($v1: i8, $(v2x=$V): i32, $(v3x=$V): i32): - ; check: ireduce.i8 $v2x - ; check: ireduce.i8 $v3x + ; check: $v2 = ireduce.i8 $v2x + ; check: $v3 = ireduce.i8 $v3x ; check: $(v1x=$V) = uextend.i32 $v1 ; check: return $v1x return v1 @@ -75,10 +73,9 @@ ebb0: v1 = call fn1() ; check: $ebb0: ; nextln: $(rv=$V) = call $fn1() - ; check: $(v1new=$V) = ireduce.i8 $rv + ; check: $v1 = ireduce.i8 $rv jump ebb1(v1) - ; The v1 alias gets resolved by split::simplify_branch_arguments(). - ; check: jump $ebb1($v1new) + ; check: jump $ebb1($v1) ebb1(v10: i8): jump ebb1(v10) @@ -93,9 +90,9 @@ ebb0(v0: i64x4): ; check: $(v0c=$V) = iconcat $v0cl, $v0ch ; check: $(v0d=$V) = iconcat $v0dl, $v0dh ; check: $(v0cd=$V) = vconcat $v0c, $v0d - ; check: $(v0abcd=$V) = vconcat $v0ab, $v0cd + ; check: $v0 = vconcat $v0ab, $v0cd v1 = bxor v0, v0 - ; check: $(v1ab=$V), $(v1cd=$V) = vsplit + ; check: $(v1ab=$V), $(v1cd=$V) = vsplit $v1 ; check: $(v1a=$V), $(v1b=$V) = vsplit $v1ab ; check: $(v1al=$V), $(v1ah=$V) = isplit $v1a ; check: $(v1bl=$V), $(v1bh=$V) = isplit $v1b diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 272a19dfee..cb2794a49d 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -85,10 +85,11 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { Err(abi_type) } }; - let converted = convert_from_abi(&mut func.dfg, &mut pos, arg_type, &mut get_arg); + let converted = + convert_from_abi(&mut func.dfg, &mut pos, arg_type, Some(arg), &mut get_arg); // The old `arg` is no longer an attached EBB argument, but there are probably still - // uses of the value. Make it an alias to the converted value. - func.dfg.change_to_alias(arg, converted); + // uses of the value. + assert_eq!(func.dfg.resolve_aliases(arg), converted); } } } @@ -141,9 +142,8 @@ fn legalize_inst_results(dfg: &mut DataFlowGraph, Err(abi_type) } }; - let v = convert_from_abi(dfg, pos, res_type, &mut get_res); - // The old `res` is no longer an attached result. - dfg.change_to_alias(res, v); + let v = convert_from_abi(dfg, pos, res_type, Some(res), &mut get_res); + assert_eq!(dfg.resolve_aliases(res), v); } } @@ -158,9 +158,11 @@ fn legalize_inst_results(dfg: &mut DataFlowGraph, /// - `Ok(arg)` if the requested type matches the next ABI argument. /// - `Err(arg_type)` if further conversions are needed from the ABI argument `arg_type`. /// +/// If the `into_result` value is provided, the converted result will be written into that value. fn convert_from_abi(dfg: &mut DataFlowGraph, pos: &mut Cursor, ty: Type, + into_result: Option, get_arg: &mut GetArg) -> Value where GetArg: FnMut(&mut DataFlowGraph, Type) -> Result @@ -169,6 +171,7 @@ fn convert_from_abi(dfg: &mut DataFlowGraph, let arg_type = match get_arg(dfg, ty) { Ok(v) => { debug_assert_eq!(dfg.value_type(v), ty); + assert_eq!(into_result, None); return v; } Err(t) => t, @@ -184,43 +187,49 @@ fn convert_from_abi(dfg: &mut DataFlowGraph, // Construct a `ty` by concatenating two ABI integers. ValueConversion::IntSplit => { let abi_ty = ty.half_width().expect("Invalid type for conversion"); - let lo = convert_from_abi(dfg, pos, abi_ty, get_arg); - let hi = convert_from_abi(dfg, pos, abi_ty, get_arg); + let lo = convert_from_abi(dfg, pos, abi_ty, None, get_arg); + let hi = convert_from_abi(dfg, pos, abi_ty, None, get_arg); dbg!("intsplit {}: {}, {}: {}", lo, dfg.value_type(lo), hi, dfg.value_type(hi)); - dfg.ins(pos).iconcat(lo, hi) + dfg.ins(pos).with_results([into_result]).iconcat(lo, hi) } // Construct a `ty` by concatenating two halves of a vector. ValueConversion::VectorSplit => { let abi_ty = ty.half_vector().expect("Invalid type for conversion"); - let lo = convert_from_abi(dfg, pos, abi_ty, get_arg); - let hi = convert_from_abi(dfg, pos, abi_ty, get_arg); - dfg.ins(pos).vconcat(lo, hi) + let lo = convert_from_abi(dfg, pos, abi_ty, None, get_arg); + let hi = convert_from_abi(dfg, pos, abi_ty, None, get_arg); + dfg.ins(pos).with_results([into_result]).vconcat(lo, hi) } // Construct a `ty` by bit-casting from an integer type. ValueConversion::IntBits => { assert!(!ty.is_int()); let abi_ty = Type::int(ty.bits()).expect("Invalid type for conversion"); - let arg = convert_from_abi(dfg, pos, abi_ty, get_arg); - dfg.ins(pos).bitcast(ty, arg) + let arg = convert_from_abi(dfg, pos, abi_ty, None, get_arg); + dfg.ins(pos) + .with_results([into_result]) + .bitcast(ty, arg) } // ABI argument is a sign-extended version of the value we want. ValueConversion::Sext(abi_ty) => { - let arg = convert_from_abi(dfg, pos, abi_ty, get_arg); + let arg = convert_from_abi(dfg, pos, abi_ty, None, get_arg); // TODO: Currently, we don't take advantage of the ABI argument being sign-extended. // We could insert an `assert_sreduce` which would fold with a following `sextend` of // this value. - dfg.ins(pos).ireduce(ty, arg) + dfg.ins(pos) + .with_results([into_result]) + .ireduce(ty, arg) } ValueConversion::Uext(abi_ty) => { - let arg = convert_from_abi(dfg, pos, abi_ty, get_arg); + let arg = convert_from_abi(dfg, pos, abi_ty, None, get_arg); // TODO: Currently, we don't take advantage of the ABI argument being sign-extended. // We could insert an `assert_ureduce` which would fold with a following `uextend` of // this value. - dfg.ins(pos).ireduce(ty, arg) + dfg.ins(pos) + .with_results([into_result]) + .ireduce(ty, arg) } } } From e5115d77938ffdd1d8668ec6c2b7fbf59dfd292d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 13 Apr 2017 10:37:58 -0700 Subject: [PATCH 0705/3084] Remove detach_secondary_results() and other cleanups. - The detach_secondary_results() is a leftover from the two-plane value representation. Use detach_results() instead to remove all instruction results. - Make the append_* DFG methods more direct. Don't depend on calling the corresponding attach_* methods. Just create a new value directly, using the values.next_key() trick. --- lib/cretonne/src/ir/dfg.rs | 94 ++++++++++++++++---------------------- 1 file changed, 39 insertions(+), 55 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index ad22a03fa7..9f1a8b5299 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -433,20 +433,6 @@ impl DataFlowGraph { ReplaceBuilder::new(self, inst) } - /// Detach secondary instruction results. - /// - /// If `inst` produces two or more results, detach these secondary result values from `inst`. - /// The first result value cannot be detached. - /// - /// Use this method to detach secondary values before using `replace(inst)` to provide an - /// alternate instruction for computing the primary result value. - pub fn detach_secondary_results(&mut self, inst: Inst) { - if let Some(first) = self.results[inst].first(&mut self.value_lists) { - self.results[inst].clear(&mut self.value_lists); - self.results[inst].push(first, &mut self.value_lists); - } - } - /// Detach the list of result values from `inst` and return it. /// /// This leaves `inst` without any result values. New result values can be created by calling @@ -484,13 +470,14 @@ impl DataFlowGraph { /// Append a new instruction result value to `inst`. pub fn append_result(&mut self, inst: Inst, ty: Type) -> Value { - let res = self.make_value(ValueData::Inst { - ty: ty, - inst: inst, - num: 0, - }); - self.attach_result(inst, res); - res + let res = self.values.next_key(); + let num = self.results[inst].push(res, &mut self.value_lists); + assert!(num <= u16::MAX as usize, "Too many result values"); + self.make_value(ValueData::Inst { + ty: ty, + inst: inst, + num: num as u16, + }) } /// Get the first result of an instruction. @@ -596,22 +583,40 @@ impl DataFlowGraph { self.ebbs[ebb].args.len(&self.value_lists) } - /// Append an argument with type `ty` to `ebb`. - pub fn append_ebb_arg(&mut self, ebb: Ebb, ty: Type) -> Value { - let val = self.make_value(ValueData::Arg { - ty: ty, - ebb: ebb, - num: 0, - }); - self.attach_ebb_arg(ebb, val); - val - } - /// Get the arguments to an EBB. pub fn ebb_args(&self, ebb: Ebb) -> &[Value] { self.ebbs[ebb].args.as_slice(&self.value_lists) } + /// Append an argument with type `ty` to `ebb`. + pub fn append_ebb_arg(&mut self, ebb: Ebb, ty: Type) -> Value { + let arg = self.values.next_key(); + let num = self.ebbs[ebb].args.push(arg, &mut self.value_lists); + assert!(num <= u16::MAX as usize, "Too many arguments to EBB"); + self.make_value(ValueData::Arg { + ty: ty, + num: num as u16, + ebb: ebb, + }) + } + + /// Append an existing argument value to `ebb`. + /// + /// The appended value can't already be attached to something else. + /// + /// In almost all cases, you should be using `append_ebb_arg()` instead of this method. + pub fn attach_ebb_arg(&mut self, ebb: Ebb, arg: Value) { + assert!(!self.value_is_attached(arg)); + let num = self.ebbs[ebb].args.push(arg, &mut self.value_lists); + assert!(num <= u16::MAX as usize, "Too many arguments to EBB"); + let ty = self.value_type(arg); + self.values[arg] = ValueData::Arg { + ty: ty, + num: num as u16, + ebb: ebb, + }; + } + /// Replace an EBB argument with a new value of type `ty`. /// /// The `old_value` must be an attached EBB argument. It is removed from its place in the list @@ -622,15 +627,11 @@ impl DataFlowGraph { /// /// Returns the new value. pub fn replace_ebb_arg(&mut self, old_arg: Value, new_type: Type) -> Value { - let old_data = self.values[old_arg].clone(); - // Create new value identical to the old one except for the type. - let (ebb, num) = if let ValueData::Arg { num, ebb, .. } = old_data { + let (ebb, num) = if let ValueData::Arg { num, ebb, .. } = self.values[old_arg] { (ebb, num) } else { - panic!("old_arg: {} must be an EBB argument: {:?}", - old_arg, - old_data); + panic!("{} must be an EBB argument", old_arg); }; let new_arg = self.make_value(ValueData::Arg { ty: new_type, @@ -650,23 +651,6 @@ impl DataFlowGraph { pub fn detach_ebb_args(&mut self, ebb: Ebb) -> ValueList { self.ebbs[ebb].args.take() } - - /// Append an existing argument value to `ebb`. - /// - /// The appended value can't already be attached to something else. - /// - /// In almost all cases, you should be using `append_ebb_arg()` instead of this method. - pub fn attach_ebb_arg(&mut self, ebb: Ebb, arg: Value) { - assert!(!self.value_is_attached(arg)); - let num = self.ebbs[ebb].args.push(arg, &mut self.value_lists); - assert!(num <= u16::MAX as usize, "Too many arguments to EBB"); - let ty = self.value_type(arg); - self.values[arg] = ValueData::Arg { - ty: ty, - num: num as u16, - ebb: ebb, - }; - } } // Contents of an extended basic block. From b151e942d927a67beea2490031c73c6274047ec2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 13 Apr 2017 12:24:20 -0700 Subject: [PATCH 0706/3084] Stop tracking if instruction formats have multiple results. All instruction formats can represent multiple results now, so a few redundant formats can be removed: UnarySplit and BinaryOverflow. --- lib/cretonne/meta/base/formats.py | 13 +++------- lib/cretonne/meta/cdsl/formats.py | 39 ++++++----------------------- lib/cretonne/meta/cdsl/isa.py | 2 -- lib/cretonne/src/ir/instructions.rs | 2 -- lib/cretonne/src/verifier.rs | 2 -- lib/cretonne/src/write.rs | 2 -- lib/reader/src/parser.rs | 15 ----------- 7 files changed, 10 insertions(+), 65 deletions(-) diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index acb9ce0131..6e64e0e3f2 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -18,14 +18,10 @@ Unary = InstructionFormat(VALUE) UnaryImm = InstructionFormat(imm64) UnaryIeee32 = InstructionFormat(ieee32) UnaryIeee64 = InstructionFormat(ieee64) -UnarySplit = InstructionFormat(VALUE, multiple_results=True) Binary = InstructionFormat(VALUE, VALUE) BinaryImm = InstructionFormat(VALUE, imm64) -# Generate result + overflow flag. -BinaryOverflow = InstructionFormat(VALUE, VALUE, multiple_results=True) - # The select instructions are controlled by the second VALUE operand. # The first VALUE operand is the controlling flag which has a derived type. # The fma instruction has the same constraint on all inputs. @@ -33,7 +29,7 @@ Ternary = InstructionFormat(VALUE, VALUE, VALUE, typevar_operand=1) # Catch-all for instructions with many outputs and inputs and no immediate # operands. -MultiAry = InstructionFormat(VARIABLE_ARGS, multiple_results=True) +MultiAry = InstructionFormat(VARIABLE_ARGS) InsertLane = InstructionFormat(VALUE, ('lane', uimm8), VALUE) ExtractLane = InstructionFormat(VALUE, ('lane', uimm8)) @@ -47,11 +43,8 @@ Branch = InstructionFormat(VALUE, ebb, VARIABLE_ARGS) BranchIcmp = InstructionFormat(intcc, VALUE, VALUE, ebb, VARIABLE_ARGS) BranchTable = InstructionFormat(VALUE, jump_table) -Call = InstructionFormat( - func_ref, VARIABLE_ARGS, multiple_results=True) -IndirectCall = InstructionFormat( - sig_ref, VALUE, VARIABLE_ARGS, - multiple_results=True) +Call = InstructionFormat(func_ref, VARIABLE_ARGS) +IndirectCall = InstructionFormat(sig_ref, VALUE, VARIABLE_ARGS) Load = InstructionFormat(memflags, VALUE, offset32) Store = InstructionFormat(memflags, VALUE, VALUE, offset32) diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index f08d6fa119..0b7a72fe0a 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -18,13 +18,6 @@ class InstructionFormat(object): identified structurally, i.e., the format of an instruction is derived from the kinds of operands used in its declaration. - Most instruction formats produce a single result, or no result at all. If - an instruction can produce more than one result, the `multiple_results` - flag must be set on its format. All results are of the `value` kind, and - the instruction format does not keep track of how many results are - produced. Some instructions, like `call`, may have a variable number of - results. - The instruction format stores two separate lists of operands: Immediates and values. Immediate operands (including entity references) are represented as explicit members in the `InstructionData` variants. The @@ -40,16 +33,14 @@ class InstructionFormat(object): :param name: Instruction format name in CamelCase. This is used as a Rust variant name in both the `InstructionData` and `InstructionFormat` enums. - :param multiple_results: Set to `True` if this instruction format allows - more than one result to be produced. :param typevar_operand: Index of the value input operand that is used to infer the controlling type variable. By default, this is `0`, the first `value` operand. The index is relative to the values only, ignoring immediate operands. """ - # Map (multiple_results, imm_kinds, num_value_operands) -> format - _registry = dict() # type: Dict[Tuple[bool, Tuple[OperandKind, ...], int, bool], InstructionFormat] # noqa + # Map (imm_kinds, num_value_operands) -> format + _registry = dict() # type: Dict[Tuple[Tuple[OperandKind, ...], int, bool], InstructionFormat] # noqa # All existing formats. all_formats = list() # type: List[InstructionFormat] @@ -57,7 +48,6 @@ class InstructionFormat(object): def __init__(self, *kinds, **kwargs): # type: (*Union[OperandKind, Tuple[str, OperandKind]], **Any) -> None # noqa self.name = kwargs.get('name', None) # type: str - self.multiple_results = kwargs.get('multiple_results', False) # The number of value operands stored in the format, or `None` when # `has_value_list` is set. @@ -81,9 +71,7 @@ class InstructionFormat(object): # Compute a signature for the global registry. imm_kinds = tuple(f.kind for f in self.imm_fields) - sig = ( - self.multiple_results, imm_kinds, self.num_value_operands, - self.has_value_list) + sig = (imm_kinds, self.num_value_operands, self.has_value_list) if sig in InstructionFormat._registry: raise RuntimeError( "Format '{}' has the same signature as existing format '{}'" @@ -158,37 +146,24 @@ class InstructionFormat(object): :py:class:`Instruction` arguments of the same name, except they must be tuples of :py:`Operand` objects. """ - if len(outs) == 1: - multiple_results = outs[0].kind == VARIABLE_ARGS - else: - multiple_results = len(outs) > 1 - # Construct a signature. imm_kinds = tuple(op.kind for op in ins if op.is_immediate()) num_values = sum(1 for op in ins if op.is_value()) has_varargs = (VARIABLE_ARGS in tuple(op.kind for op in ins)) - sig = (multiple_results, imm_kinds, num_values, has_varargs) + sig = (imm_kinds, num_values, has_varargs) if sig in InstructionFormat._registry: return InstructionFormat._registry[sig] # Try another value list format as an alternative. - sig = (True, imm_kinds, num_values, has_varargs) - if sig in InstructionFormat._registry: - return InstructionFormat._registry[sig] - - sig = (multiple_results, imm_kinds, 0, True) - if sig in InstructionFormat._registry: - return InstructionFormat._registry[sig] - - sig = (True, imm_kinds, 0, True) + sig = (imm_kinds, 0, True) if sig in InstructionFormat._registry: return InstructionFormat._registry[sig] raise RuntimeError( - 'No instruction format matches multiple_results={},' + 'No instruction format matches ' 'imms={}, vals={}, varargs={}'.format( - multiple_results, imm_kinds, num_values, has_varargs)) + imm_kinds, num_values, has_varargs)) @staticmethod def extract_names(globs): diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 1ead583c5a..a44a1e6333 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -207,8 +207,6 @@ class EncRecipe(object): if not format.has_value_list: assert len(self.ins) == format.num_value_operands self.outs = self._verify_constraints(outs) - if len(self.outs) > 1: - assert format.multiple_results def __str__(self): # type: () -> str diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 61ed2f1eeb..9484110d83 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -105,14 +105,12 @@ pub enum InstructionData { UnaryImm { opcode: Opcode, imm: Imm64 }, UnaryIeee32 { opcode: Opcode, imm: Ieee32 }, UnaryIeee64 { opcode: Opcode, imm: Ieee64 }, - UnarySplit { opcode: Opcode, arg: Value }, Binary { opcode: Opcode, args: [Value; 2] }, BinaryImm { opcode: Opcode, arg: Value, imm: Imm64, }, - BinaryOverflow { opcode: Opcode, args: [Value; 2] }, Ternary { opcode: Opcode, args: [Value; 3] }, MultiAry { opcode: Opcode, args: ValueList }, InsertLane { diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier.rs index ba2f03ec9d..ee78a07501 100644 --- a/lib/cretonne/src/verifier.rs +++ b/lib/cretonne/src/verifier.rs @@ -255,10 +255,8 @@ impl<'a> Verifier<'a> { &UnaryImm { .. } | &UnaryIeee32 { .. } | &UnaryIeee64 { .. } | - &UnarySplit { .. } | &Binary { .. } | &BinaryImm { .. } | - &BinaryOverflow { .. } | &Ternary { .. } | &InsertLane { .. } | &ExtractLane { .. } | diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 09e5ee6641..fcfb843c0c 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -245,10 +245,8 @@ pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result UnaryImm { imm, .. } => write!(w, " {}", imm), UnaryIeee32 { imm, .. } => write!(w, " {}", imm), UnaryIeee64 { imm, .. } => write!(w, " {}", imm), - UnarySplit { arg, .. } => write!(w, " {}", arg), Binary { args, .. } => write!(w, " {}, {}", args[0], args[1]), BinaryImm { arg, imm, .. } => write!(w, " {}, {}", arg, imm), - BinaryOverflow { args, .. } => write!(w, " {}, {}", args[0], args[1]), Ternary { args, .. } => write!(w, " {}, {}, {}", args[0], args[1], args[2]), MultiAry { ref args, .. } => { if args.is_empty() { diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index ac2754a46c..f133732e8e 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1429,12 +1429,6 @@ impl<'a> Parser<'a> { imm: self.match_ieee64("expected immediate 64-bit float operand")?, } } - InstructionFormat::UnarySplit => { - InstructionData::UnarySplit { - opcode: opcode, - arg: self.match_value("expected SSA value operand")?, - } - } InstructionFormat::Binary => { let lhs = self.match_value("expected SSA value first operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; @@ -1454,15 +1448,6 @@ impl<'a> Parser<'a> { imm: rhs, } } - InstructionFormat::BinaryOverflow => { - let lhs = self.match_value("expected SSA value first operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; - let rhs = self.match_value("expected SSA value second operand")?; - InstructionData::BinaryOverflow { - opcode: opcode, - args: [lhs, rhs], - } - } InstructionFormat::Ternary => { // Names here refer to the `select` instruction. // This format is also use by `fma`. From d424589daad47e5485df70b94104f7ac6560d213 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 17 Apr 2017 15:04:00 -0700 Subject: [PATCH 0707/3084] Allow for special purpose function arguments and return values. Enumerate a set of special purposes for function arguments that general purpose code needs to know about. Some of these argument purposes will only appear in the signature of the current function, representing things the prologue and epilogues need to know about like the link register and callee-saved registers. Get rid of the 'inreg' argument flag. Arguments can be pre-assigned to a specific register instead. --- cranelift/docs/cton_lexer.py | 3 +- cranelift/docs/langref.rst | 5 +- cranelift/filetests/parser/call.cton | 10 +++ lib/cretonne/src/ir/extfunc.rs | 97 ++++++++++++++++++++++++++-- lib/cretonne/src/ir/mod.rs | 2 +- lib/reader/src/parser.rs | 17 +++-- 6 files changed, 116 insertions(+), 18 deletions(-) diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py index 1c7224b593..1024765cbe 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/cton_lexer.py @@ -42,8 +42,7 @@ class CretonneLexer(RegexLexer): (r'[-+]?(\d+\.\d+([eE]\d+)?|s?NaN|Inf)', Number.Float), (r'[-+]?\d+', Number.Integer), # Known attributes. - (keywords('align', 'aligntrap', 'uext', 'sext', 'inreg'), - Name.Attribute), + (keywords('uext', 'sext'), Name.Attribute), # Well known value types. (r'\b(b\d+|i\d+|f32|f64)(x\d+)?\b', Keyword.Type), # v = value diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index d4b30780b4..fce84d0cdf 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -364,8 +364,9 @@ calling convention: signature : "(" [arglist] ")" ["->" retlist] [call_conv] arglist : arg { "," arg } retlist : arglist - arg : type { flag } - flag : "uext" | "sext" | "inreg" + arg : type [argext] [argspecial] + argext : "uext" | "sext" + argspecial: "sret" | "link" | "fp" | "csr" callconv : `string` Arguments and return values have flags whose meaning is mostly target diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton index 027749f562..7626524110 100644 --- a/cranelift/filetests/parser/call.cton +++ b/cranelift/filetests/parser/call.cton @@ -68,3 +68,13 @@ ebb0(v0: i64): ; check: call_indirect $sig0, $v1($v0) ; check: $v3, $v4 = call_indirect $sig2, $v1() ; check: return + +; Special purpose function arguments +function special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret { +ebb0(v1: i32, v2: i32, v3: i32): + return v4, v2, v3, v1 +} +; check: function special1(i32 sret, i32 fp, i32 link) -> i32 link, i32 fp, i32 sret { +; check: ebb0($v1: i32, $v2: i32, $v3: i32): +; check: return $v3, $v2, $v1 +; check: } diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 43daaed192..a21b5911a9 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -9,6 +9,7 @@ use ir::{Type, FunctionName, SigRef, ArgumentLoc}; use isa::RegInfo; use std::cmp; use std::fmt; +use std::str::FromStr; /// Function signature. /// @@ -111,10 +112,10 @@ impl fmt::Display for Signature { pub struct ArgumentType { /// Type of the argument value. pub value_type: Type, + /// Special purpose of argument, or `Normal`. + pub purpose: ArgumentPurpose, /// Method for extending argument to a full register. pub extension: ArgumentExtension, - /// Place this argument in a register if possible. - pub inreg: bool, /// ABI-specific location of this argument, or `Unassigned` for arguments that have not yet /// been legalized. @@ -127,7 +128,7 @@ impl ArgumentType { ArgumentType { value_type: vt, extension: ArgumentExtension::None, - inreg: false, + purpose: ArgumentPurpose::Normal, location: Default::default(), } } @@ -149,8 +150,8 @@ impl<'a> fmt::Display for DisplayArgumentType<'a> { ArgumentExtension::Uext => write!(f, " uext")?, ArgumentExtension::Sext => write!(f, " sext")?, } - if self.0.inreg { - write!(f, " inreg")?; + if self.0.purpose != ArgumentPurpose::Normal { + write!(f, " {}", self.0.purpose)?; } if self.0.location.is_assigned() { @@ -181,6 +182,75 @@ pub enum ArgumentExtension { Sext, } +/// The special purpose of a function argument. +/// +/// Function arguments and return values are used to pass user program values between functions, +/// but they are also used to represent special registers with significance to the ABI such as +/// frame pointers and callee-saved registers. +/// +/// The argument purpose is used to indicate any special meaning of an argument or return value. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub enum ArgumentPurpose { + /// A normal user program value passed to or from a function. + Normal, + + /// Struct return pointer. + /// + /// When a function needs to return more data than will fit in registers, the caller passes a + /// pointer to a memory location where the return value can be written. In some ABIs, this + /// struct return pointer is passed in a specific register. + /// + /// This argument kind can also appear as a return value for ABIs that require a function with + /// a `StructReturn` pointer argument to also return that pointer in a register. + StructReturn, + + /// The link register. + /// + /// Most RISC architectures implement calls by saving the return address in a designated + /// register rather than pushing it on the stack. This is represented with a `Link` argument. + /// + /// Similarly, some return instructions expect the return address in a register represented as + /// a `Link` return value. + Link, + + /// The frame pointer. + /// + /// This indicates the frame pointer register which has a special meaning in some ABIs. + /// + /// The frame pointer appears as an argument and as a return value since it is a callee-saved + /// register. + FramePointer, + + /// A callee-saved register. + /// + /// Some calling conventions have registers that must be saved by the callee. These registers + /// are represented as `CalleeSaved` arguments and return values. + CalleeSaved, +} + +/// Text format names of the `ArgumentPurpose` variants. +static PURPOSE_NAMES: [&'static str; 5] = ["normal", "sret", "link", "fp", "csr"]; + +impl fmt::Display for ArgumentPurpose { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + f.write_str(PURPOSE_NAMES[*self as usize]) + } +} + +impl FromStr for ArgumentPurpose { + type Err = (); + fn from_str(s: &str) -> Result { + match s { + "normal" => Ok(ArgumentPurpose::Normal), + "sret" => Ok(ArgumentPurpose::StructReturn), + "link" => Ok(ArgumentPurpose::Link), + "fp" => Ok(ArgumentPurpose::FramePointer), + "csr" => Ok(ArgumentPurpose::CalleeSaved), + _ => Err(()), + } + } +} + /// An external function. /// /// Information about a function that can be called directly with a direct `call` instruction. @@ -209,8 +279,21 @@ mod tests { assert_eq!(t.to_string(), "i32"); t.extension = ArgumentExtension::Uext; assert_eq!(t.to_string(), "i32 uext"); - t.inreg = true; - assert_eq!(t.to_string(), "i32 uext inreg"); + t.purpose = ArgumentPurpose::StructReturn; + assert_eq!(t.to_string(), "i32 uext sret"); + } + + #[test] + fn argument_purpose() { + let all_purpose = [ArgumentPurpose::Normal, + ArgumentPurpose::StructReturn, + ArgumentPurpose::Link, + ArgumentPurpose::FramePointer, + ArgumentPurpose::CalleeSaved]; + for (&e, &n) in all_purpose.iter().zip(PURPOSE_NAMES.iter()) { + assert_eq!(e.to_string(), n); + assert_eq!(Ok(e), n.parse()); + } } #[test] diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 488b4dab32..49617165e9 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -18,7 +18,7 @@ mod progpoint; mod valueloc; pub use ir::funcname::FunctionName; -pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension, ExtFuncData}; +pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension, ArgumentPurpose, ExtFuncData}; pub use ir::types::Type; pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable, FuncRef, SigRef}; pub use ir::instructions::{Opcode, InstructionData, VariableArgs, ValueList, ValueListPool}; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index f133732e8e..f96d60518a 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -767,8 +767,13 @@ impl<'a> Parser<'a> { match s { "uext" => arg.extension = ArgumentExtension::Uext, "sext" => arg.extension = ArgumentExtension::Sext, - "inreg" => arg.inreg = true, - _ => break, + _ => { + if let Ok(purpose) = s.parse() { + arg.purpose = purpose; + } else { + break; + } + } } self.consume(); } @@ -1672,7 +1677,7 @@ impl<'a> Parser<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne::ir::ArgumentExtension; + use cretonne::ir::{ArgumentExtension, ArgumentPurpose}; use cretonne::ir::types; use cretonne::ir::entities::AnyEntity; use testfile::{Details, Comment}; @@ -1685,7 +1690,7 @@ mod tests { let arg = p.parse_argument_type(None).unwrap(); assert_eq!(arg.value_type, types::I32); assert_eq!(arg.extension, ArgumentExtension::Sext); - assert_eq!(arg.inreg, false); + assert_eq!(arg.purpose, ArgumentPurpose::Normal); let Error { location, message } = p.parse_argument_type(None).unwrap_err(); assert_eq!(location.line_number, 1); assert_eq!(message, "expected argument type"); @@ -1721,11 +1726,11 @@ mod tests { assert_eq!(sig.argument_types.len(), 0); assert_eq!(sig.return_types.len(), 0); - let sig2 = Parser::new("(i8 inreg uext, f32, f64) -> i32 sext, f64") + let sig2 = Parser::new("(i8 uext, f32, f64, i32 sret) -> i32 sext, f64") .parse_signature(None) .unwrap(); assert_eq!(sig2.to_string(), - "(i8 uext inreg, f32, f64) -> i32 sext, f64"); + "(i8 uext, f32, f64, i32 sret) -> i32 sext, f64"); // `void` is not recognized as a type by the lexer. It should not appear in files. assert_eq!(Parser::new("() -> void") From 49c120957222b71e4da254ce58aa77b60c28f7d9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 17 Apr 2017 15:45:55 -0700 Subject: [PATCH 0708/3084] Fix broken test. --- cranelift/filetests/parser/call.cton | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton index 7626524110..67472f7efc 100644 --- a/cranelift/filetests/parser/call.cton +++ b/cranelift/filetests/parser/call.cton @@ -71,10 +71,10 @@ ebb0(v0: i64): ; Special purpose function arguments function special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret { -ebb0(v1: i32, v2: i32, v3: i32): +ebb0(v1: i32, v2: i32, v3: i32, v4: i32): return v4, v2, v3, v1 } -; check: function special1(i32 sret, i32 fp, i32 link) -> i32 link, i32 fp, i32 sret { -; check: ebb0($v1: i32, $v2: i32, $v3: i32): -; check: return $v3, $v2, $v1 +; check: function special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret { +; check: ebb0($v1: i32, $v2: i32, $v3: i32, $v4: i32): +; check: return $v4, $v2, $v3, $v1 ; check: } From d9ddf4fc5a7a867ce53aa5062c2523aa54e0fad3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Apr 2017 14:53:14 -0700 Subject: [PATCH 0709/3084] Simplify check_arg_types(). Iterator tricks. --- lib/cretonne/src/legalizer/boundary.rs | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index cb2794a49d..537d6cb9de 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -290,21 +290,9 @@ fn convert_to_abi(dfg: &mut DataFlowGraph, /// Check if a sequence of arguments match a desired sequence of argument types. fn check_arg_types(dfg: &DataFlowGraph, args: &[Value], types: &[ArgumentType]) -> bool { - let mut n = 0; - for &arg in args { - match types.get(n) { - Some(&ArgumentType { value_type, .. }) => { - if dfg.value_type(arg) != value_type { - return false; - } - } - None => return false, - } - n += 1 - } - - // Also verify that the number of arguments matches. - n == types.len() + let arg_types = args.iter().map(|&v| dfg.value_type(v)); + let sig_types = types.iter().map(|&at| at.value_type); + arg_types.eq(sig_types) } /// Check if the arguments of the call `inst` match the signature. From e44a5a4391fcb11c30f70b41bdc864e51a017e32 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Apr 2017 15:44:17 -0700 Subject: [PATCH 0710/3084] Append link and sret arguments in legalize_signature. These special-purpose arguments and return values are only relevant for the function being compiled, so add a `current` flag to legalize_signature(). - Add the necessary argument values to the entry block to represent the special-purpose arguments. - Propagate the link and sret arguments to return instructions if the legalized signature asks for it. --- .../filetests/isa/riscv/legalize-abi.cton | 18 ++-- .../filetests/isa/riscv/legalize-i64.cton | 16 +-- .../filetests/isa/riscv/parse-encoding.cton | 2 +- cranelift/filetests/isa/riscv/split-args.cton | 10 +- lib/cretonne/src/ir/extfunc.rs | 12 ++- lib/cretonne/src/isa/mod.rs | 19 +++- lib/cretonne/src/isa/riscv/abi.rs | 17 +++- lib/cretonne/src/isa/riscv/mod.rs | 4 +- lib/cretonne/src/legalizer/boundary.rs | 98 +++++++++++++++++-- 9 files changed, 161 insertions(+), 35 deletions(-) diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index 88d35e699a..a3632edca3 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -6,11 +6,11 @@ isa riscv function int_split_args(i64) -> i64 { ebb0(v0: i64): - ; check: $ebb0($(v0l=$V): i32, $(v0h=$V): i32): + ; check: $ebb0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32): ; check: $v0 = iconcat $v0l, $v0h v1 = iadd_imm v0, 1 ; check: $(v1l=$V), $(v1h=$V) = isplit $v1 - ; check: return $v1l, $v1h + ; check: return $v1l, $v1h, $link return v1 } @@ -31,7 +31,7 @@ function split_ret_val() { fn1 = function foo() -> i64 ebb0: v1 = call fn1() - ; check: $ebb0: + ; check: $ebb0($(link=$V): i32): ; nextln: $(v1l=$V), $(v1h=$V) = call $fn1() ; check: $v1 = iconcat $v1l, $v1h jump ebb1(v1) @@ -46,7 +46,7 @@ function split_ret_val2() { fn1 = function foo() -> i32, i64 ebb0: v1, v2 = call fn1() - ; check: $ebb0: + ; check: $ebb0($(link=$V): i32): ; nextln: $v1, $(v2l=$V), $(v2h=$V) = call $fn1() ; check: $v2 = iconcat $v2l, $v2h jump ebb1(v1, v2) @@ -58,11 +58,11 @@ ebb1(v9: i32, v10: i64): function int_ext(i8, i8 sext, i8 uext) -> i8 uext { ebb0(v1: i8, v2: i8, v3: i8): - ; check: $ebb0($v1: i8, $(v2x=$V): i32, $(v3x=$V): i32): + ; check: $ebb0($v1: i8, $(v2x=$V): i32, $(v3x=$V): i32, $(link=$V): i32): ; check: $v2 = ireduce.i8 $v2x ; check: $v3 = ireduce.i8 $v3x ; check: $(v1x=$V) = uextend.i32 $v1 - ; check: return $v1x + ; check: return $v1x, $link return v1 } @@ -71,7 +71,7 @@ function ext_ret_val() { fn1 = function foo() -> i8 sext ebb0: v1 = call fn1() - ; check: $ebb0: + ; check: $ebb0($V: i32): ; nextln: $(rv=$V) = call $fn1() ; check: $v1 = ireduce.i8 $rv jump ebb1(v1) @@ -83,7 +83,7 @@ ebb1(v10: i8): function vector_split_args(i64x4) -> i64x4 { ebb0(v0: i64x4): - ; check: $ebb0($(v0al=$V): i32, $(v0ah=$V): i32, $(v0bl=$V): i32, $(v0bh=$V): i32, $(v0cl=$V): i32, $(v0ch=$V): i32, $(v0dl=$V): i32, $(v0dh=$V): i32): + ; check: $ebb0($(v0al=$V): i32, $(v0ah=$V): i32, $(v0bl=$V): i32, $(v0bh=$V): i32, $(v0cl=$V): i32, $(v0ch=$V): i32, $(v0dl=$V): i32, $(v0dh=$V): i32, $(link=$V): i32): ; check: $(v0a=$V) = iconcat $v0al, $v0ah ; check: $(v0b=$V) = iconcat $v0bl, $v0bh ; check: $(v0ab=$V) = vconcat $v0a, $v0b @@ -99,7 +99,7 @@ ebb0(v0: i64x4): ; check: $(v1c=$V), $(v1d=$V) = vsplit $v1cd ; check: $(v1cl=$V), $(v1ch=$V) = isplit $v1c ; check: $(v1dl=$V), $(v1dh=$V) = isplit $v1d - ; check: return $v1al, $v1ah, $v1bl, $v1bh, $v1cl, $v1ch, $v1dl, $v1dh + ; check: return $v1al, $v1ah, $v1bl, $v1bh, $v1cl, $v1ch, $v1dl, $v1dh, $link return v1 } diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton index dc8604f358..516502c4d4 100644 --- a/cranelift/filetests/isa/riscv/legalize-i64.cton +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -9,39 +9,39 @@ ebb0(v1: i64, v2: i64): v3 = band v1, v2 return v3 } -; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32): +; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): ; check: [R#ec ; sameln: $(v3l=$V) = band $v1l, $v2l ; check: [R#ec ; sameln: $(v3h=$V) = band $v1h, $v2h ; check: $v3 = iconcat $v3l, $v3h -; check: return $v3l, $v3h +; check: return $v3l, $v3h, $link function bitwise_or(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): v3 = bor v1, v2 return v3 } -; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32): +; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): ; check: [R#cc ; sameln: $(v3l=$V) = bor $v1l, $v2l ; check: [R#cc ; sameln: $(v3h=$V) = bor $v1h, $v2h ; check: $v3 = iconcat $v3l, $v3h -; check: return $v3l, $v3h +; check: return $v3l, $v3h, $link function bitwise_xor(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): v3 = bxor v1, v2 return v3 } -; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32): +; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): ; check: [R#8c ; sameln: $(v3l=$V) = bxor $v1l, $v2l ; check: [R#8c ; sameln: $(v3h=$V) = bxor $v1h, $v2h ; check: $v3 = iconcat $v3l, $v3h -; check: return $v3l, $v3h +; check: return $v3l, $v3h, $link function arith_add(i64, i64) -> i64 { ; Legalizing iadd.i64 requires two steps: @@ -51,7 +51,7 @@ ebb0(v1: i64, v2: i64): v3 = iadd v1, v2 return v3 } -; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32): +; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): ; check: [R#0c ; sameln: $(v3l=$V) = iadd $v1l, $v2l ; check: $(c=$V) = icmp ult $v3l, $v1l @@ -61,4 +61,4 @@ ebb0(v1: i64, v2: i64): ; check: [R#0c ; sameln: $(v3h=$V) = iadd $v3h1, $c_int ; check: $v3 = iconcat $v3l, $v3h -; check: return $v3l, $v3h +; check: return $v3l, $v3h, $link diff --git a/cranelift/filetests/isa/riscv/parse-encoding.cton b/cranelift/filetests/isa/riscv/parse-encoding.cton index d3cc6eee4b..69a25ce744 100644 --- a/cranelift/filetests/isa/riscv/parse-encoding.cton +++ b/cranelift/filetests/isa/riscv/parse-encoding.cton @@ -3,7 +3,7 @@ test legalizer isa riscv function parse_encoding(i32 [%x5]) -> i32 [%x10] { - ; check: function parse_encoding(i32 [%x5]) -> i32 [%x10] { + ; check: function parse_encoding(i32 [%x5], i32 link [%x1]) -> i32 [%x10], i32 link [%x1] { sig0 = signature(i32 [%x10]) -> i32 [%x10] ; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] diff --git a/cranelift/filetests/isa/riscv/split-args.cton b/cranelift/filetests/isa/riscv/split-args.cton index 06b09bd6d1..a568204658 100644 --- a/cranelift/filetests/isa/riscv/split-args.cton +++ b/cranelift/filetests/isa/riscv/split-args.cton @@ -6,7 +6,7 @@ isa riscv function simple(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): -; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32): +; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): jump ebb1(v1) ; check: jump $ebb1($v1l, $v1h) @@ -16,12 +16,12 @@ ebb1(v3: i64): ; check: $(v4l=$V) = band $v3l, $v2l ; check: $(v4h=$V) = band $v3h, $v2h return v4 - ; check: return $v4l, $v4h + ; check: return $v4l, $v4h, $link } function multi(i64) -> i64 { ebb1(v1: i64): -; check: $ebb1($(v1l=$V): i32, $(v1h=$V): i32): +; check: $ebb1($(v1l=$V): i32, $(v1h=$V): i32, $(link=$V): i32): jump ebb2(v1, v1) ; check: jump $ebb2($v1l, $v1l, $v1h, $v1h) @@ -36,12 +36,12 @@ ebb3(v4: i64): ; check: $(v5l=$V) = band $v4l, $v3l ; check: $(v5h=$V) = band $v4h, $v3h return v5 - ; check: return $v5l, $v5h + ; check: return $v5l, $v5h, $link } function loop(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): -; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32): +; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): jump ebb1(v1) ; check: jump $ebb1($v1l, $v1h) diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index a21b5911a9..0971015f62 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -6,7 +6,7 @@ //! This module declares the data types used to represent external functions and call signatures. use ir::{Type, FunctionName, SigRef, ArgumentLoc}; -use isa::RegInfo; +use isa::{RegInfo, RegUnit}; use std::cmp; use std::fmt; use std::str::FromStr; @@ -133,6 +133,16 @@ impl ArgumentType { } } + /// Create an argument type for a special-purpose register. + pub fn special_reg(vt: Type, purpose: ArgumentPurpose, regunit: RegUnit) -> ArgumentType { + ArgumentType { + value_type: vt, + extension: ArgumentExtension::None, + purpose: purpose, + location: ArgumentLoc::Reg(regunit), + } + } + /// Return an object that can display `self` with correct register names. pub fn display<'a, R: Into>>(&'a self, regs: R) -> DisplayArgumentType<'a> { DisplayArgumentType(self, regs.into()) diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index c40a839836..e951ad31de 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -166,7 +166,24 @@ pub trait TargetIsa { /// - Vector types can be bit-cast and broken down into smaller vectors or scalars. /// /// The legalizer will adapt argument and return values as necessary at all ABI boundaries. - fn legalize_signature(&self, _sig: &mut Signature) { + /// + /// When this function is called to legalize the signature of the function currently begin + /// compiler, `_current` is true. The legalized signature can then also contain special purpose + /// arguments and return values such as: + /// + /// - A `link` argument representing the link registers on RISC architectures that don't push + /// the return address on the stack. + /// - A `link` return value which will receive the value that was passed to the `link` + /// argument. + /// - An `sret` argument can be added if one wasn't present already. This is necessary if the + /// signature returns more values than registers are available for returning values. + /// - An `sret` return value can be added if the ABI requires a function to return its `sret` + /// argument in a register. + /// + /// Arguments and return values for the caller's frame pointer and other callee-saved registers + /// should not be added by this function. These arguments are not added until after register + /// allocation. + fn legalize_signature(&self, _sig: &mut Signature, _current: bool) { unimplemented!() } diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index fa158dceb5..453a9c7be8 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -6,7 +6,7 @@ //! This doesn't support the soft-float ABI at the moment. use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; -use ir::{Signature, Type, ArgumentType, ArgumentLoc, ArgumentExtension}; +use ir::{Signature, Type, ArgumentType, ArgumentLoc, ArgumentExtension, ArgumentPurpose}; use isa::riscv::registers::{GPR, FPR}; use settings as shared_settings; @@ -80,7 +80,7 @@ impl ArgAssigner for Args { } /// Legalize `sig` for RISC-V. -pub fn legalize_signature(sig: &mut Signature, flags: &shared_settings::Flags) { +pub fn legalize_signature(sig: &mut Signature, flags: &shared_settings::Flags, current: bool) { let bits = if flags.is_64bit() { 64 } else { 32 }; let mut args = Args::new(bits); @@ -88,4 +88,17 @@ pub fn legalize_signature(sig: &mut Signature, flags: &shared_settings::Flags) { let mut rets = Args::new(bits); legalize_args(&mut sig.return_types, &mut rets); + + if current { + let ptr = Type::int(bits).unwrap(); + + // Add the link register as an argument and return value. + // + // The `jalr` instruction implementing a return can technically accept the return address + // in any register, but a micro-architecture with a return address predictor will only + // recognize it as a return if the address is in `x1`. + let link = ArgumentType::special_reg(ptr, ArgumentPurpose::Link, GPR.unit(1)); + sig.argument_types.push(link); + sig.return_types.push(link); + } } diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 778e3b85aa..ac5057103c 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -78,9 +78,9 @@ impl TargetIsa for Isa { }) } - fn legalize_signature(&self, sig: &mut Signature) { + fn legalize_signature(&self, sig: &mut Signature, current: bool) { // We can pass in `self.isa_flags` too, if we need it. - abi::legalize_signature(sig, &self.shared_flags) + abi::legalize_signature(sig, &self.shared_flags, current) } fn emit_inst(&self, func: &Function, inst: Inst, sink: &mut CodeSink) { diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 537d6cb9de..e0cf9a2f21 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -20,7 +20,7 @@ use abi::{legalize_abi_value, ValueConversion}; use flowgraph::ControlFlowGraph; use ir::{Function, Cursor, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, Signature, SigRef, - ArgumentType}; + ArgumentType, ArgumentPurpose}; use ir::instructions::CallInfo; use isa::TargetIsa; use legalizer::split::{isplit, vsplit}; @@ -31,9 +31,9 @@ use legalizer::split::{isplit, vsplit}; /// change the entry block arguments, calls, or return instructions, so this can leave the function /// in a state with type discrepancies. pub fn legalize_signatures(func: &mut Function, isa: &TargetIsa) { - isa.legalize_signature(&mut func.signature); + isa.legalize_signature(&mut func.signature, true); for sig in func.dfg.signatures.keys() { - isa.legalize_signature(&mut func.dfg.signatures[sig]); + isa.legalize_signature(&mut func.dfg.signatures[sig], false); } if let Some(entry) = func.layout.entry_block() { @@ -50,6 +50,9 @@ pub fn legalize_signatures(func: &mut Function, isa: &TargetIsa) { /// The original entry EBB arguments are computed from the new ABI arguments by code inserted at /// the top of the entry block. fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { + let mut has_sret = false; + let mut has_link = false; + // Insert position for argument conversion code. // We want to insert instructions before the first instruction in the entry block. // If the entry block is empty, append instructions to it instead. @@ -73,11 +76,22 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { // No value translation is necessary, this argument matches the ABI type. // Just use the original EBB argument value. This is the most common case. func.dfg.attach_ebb_arg(entry, arg); + match abi_types[abi_arg].purpose { + ArgumentPurpose::Normal => {} + ArgumentPurpose::StructReturn => { + assert!(!has_sret, "Multiple sret arguments found"); + has_sret = true; + } + _ => panic!("Unexpected special-purpose arg {}", abi_types[abi_arg]), + } abi_arg += 1; } else { // Compute the value we want for `arg` from the legalized ABI arguments. let mut get_arg = |dfg: &mut DataFlowGraph, ty| { let abi_type = abi_types[abi_arg]; + assert_eq!(abi_type.purpose, + ArgumentPurpose::Normal, + "Can't legalize special-purpose argument"); if ty == abi_type.value_type { abi_arg += 1; Ok(dfg.append_ebb_arg(entry, ty)) @@ -92,6 +106,35 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { assert_eq!(func.dfg.resolve_aliases(arg), converted); } } + + // The legalized signature may contain additional arguments representing special-purpose + // registers. + for &arg in &abi_types[abi_arg..] { + match arg.purpose { + // Any normal arguments should have been processed above. + ArgumentPurpose::Normal => { + panic!("Leftover arg: {}", arg); + } + // The callee-save arguments should not appear until after register allocation is + // done. + ArgumentPurpose::FramePointer | + ArgumentPurpose::CalleeSaved => { + panic!("Premature callee-saved arg {}", arg); + } + // These can be meaningfully added by `legalize_signature()`. + ArgumentPurpose::Link => { + assert!(!has_link, "Multiple link arguments found"); + has_link = true; + } + ArgumentPurpose::StructReturn => { + assert!(!has_sret, "Multiple sret arguments found"); + has_sret = true; + } + } + // Just create entry block values to match here. We will use them in `handle_return_abi()` + // below. + func.dfg.append_ebb_arg(entry, arg.value_type); + } } /// Legalize the results returned from a call instruction to match the ABI signature. @@ -445,7 +488,7 @@ pub fn handle_call_abi(dfg: &mut DataFlowGraph, cfg: &ControlFlowGraph, pos: &mu true } -/// Insert ABI conversion code before and after the call instruction at `pos`. +/// Insert ABI conversion code before and after the return instruction at `pos`. /// /// Return `true` if any instructions were inserted. pub fn handle_return_abi(dfg: &mut DataFlowGraph, @@ -461,15 +504,58 @@ pub fn handle_return_abi(dfg: &mut DataFlowGraph, return false; } - let abi_args = sig.return_types.len(); + // Count the special-purpose return values (`link` and `sret`) that were appended to the + // legalized signature. + let special_args = sig.return_types + .iter() + .rev() + .take_while(|&rt| { + rt.purpose == ArgumentPurpose::Link || + rt.purpose == ArgumentPurpose::StructReturn + }) + .count(); + + let abi_args = sig.return_types.len() - special_args; legalize_inst_arguments(dfg, cfg, pos, abi_args, |_, abi_arg| sig.return_types[abi_arg]); + assert_eq!(dfg.inst_variable_args(inst).len(), abi_args); + + // Append special return arguments for any `sret` and `link` return values added to the + // legalized signature. These values should simply be propagated from the entry block + // arguments. + if special_args > 0 { + dbg!("Adding {} special-purpose arguments to {}", + special_args, + dfg.display_inst(inst)); + let mut vlist = dfg[inst].take_value_list().unwrap(); + for arg in &sig.return_types[abi_args..] { + match arg.purpose { + ArgumentPurpose::Link | + ArgumentPurpose::StructReturn => {} + ArgumentPurpose::Normal => panic!("unexpected return value {}", arg), + _ => panic!("Unsupported special purpose return value {}", arg), + } + // A `link` or `sret` return value can only appear in a signature that has a unique + // matching argument. They are appended at the end, so search the signature from the + // end. + let idx = sig.argument_types + .iter() + .rposition(|t| t.purpose == arg.purpose) + .expect("No matching special purpose argument."); + // Get the corresponding entry block value and add it to the return instruction's + // arguments. + let val = dfg.ebb_args(pos.layout.entry_block().unwrap())[idx]; + debug_assert_eq!(dfg.value_type(val), arg.value_type); + vlist.push(val, &mut dfg.value_lists); + } + dfg[inst].put_value_list(vlist); + } debug_assert!(check_return_signature(dfg, inst, sig), - "Signature still wrong: {}, sig{}", + "Signature still wrong: {} / signature {}", dfg.display_inst(inst), sig); From 0cb36c90319846c273849040a9c880bd61853405 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Apr 2017 16:08:16 -0700 Subject: [PATCH 0711/3084] Remove the return_reg instruction. RISC architectures that take a return address in a register can use a special-purpose `link` return value to do so. --- cranelift/docs/langref.rst | 1 - cranelift/filetests/isa/riscv/abi.cton | 6 +++--- cranelift/filetests/isa/riscv/encoding.cton | 4 ++-- cranelift/filetests/regalloc/basic.cton | 2 +- lib/cretonne/meta/base/instructions.py | 19 ------------------- lib/cretonne/meta/isa/riscv/encodings.py | 11 +++++------ 6 files changed, 11 insertions(+), 32 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index fce84d0cdf..008f2a74b1 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -386,7 +386,6 @@ preamble`: .. autoinst:: call .. autoinst:: x_return -.. autoinst:: return_reg This simple example illustrates direct function calls and signatures:: diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.cton index d168fd768c..eba5609f3e 100644 --- a/cranelift/filetests/isa/riscv/abi.cton +++ b/cranelift/filetests/isa/riscv/abi.cton @@ -4,7 +4,7 @@ isa riscv ; regex: V=v\d+ -function f(i32) { +function f() { sig0 = signature(i32) -> i32 ; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] @@ -27,6 +27,6 @@ function f(i32) { sig5 = signature(i64x4) ; check: sig5 = signature(i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [%x16], i32 [%x17]) -ebb0(v0: i32): - return_reg v0 +ebb0: + return } diff --git a/cranelift/filetests/isa/riscv/encoding.cton b/cranelift/filetests/isa/riscv/encoding.cton index f6defe27bf..fdd3ee4329 100644 --- a/cranelift/filetests/isa/riscv/encoding.cton +++ b/cranelift/filetests/isa/riscv/encoding.cton @@ -15,7 +15,7 @@ ebb0(v1: i32, v2: i32): ; check: [R#10c] ; sameln: $v12 = imul - return_reg v1 + return ; check: [Iret#19] - ; sameln: return_reg + ; sameln: return } diff --git a/cranelift/filetests/regalloc/basic.cton b/cranelift/filetests/regalloc/basic.cton index 00e204f9b1..cbff589431 100644 --- a/cranelift/filetests/regalloc/basic.cton +++ b/cranelift/filetests/regalloc/basic.cton @@ -8,5 +8,5 @@ ebb0(v1: i32, v2: i32): v3 = iadd v1, v2 ; check: [R#0c,%x0] ; sameln: iadd - return_reg v3 + return } diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index a7c549ce93..3035f2b98e 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -155,25 +155,6 @@ x_return = Instruction( """, ins=rvals, is_return=True, is_terminator=True) -raddr = Operand('raddr', iAddr, doc='Return address') - -return_reg = Instruction( - 'return_reg', r""" - Return from the function to a return address held in a register. - - Unconditionally transfer control to the calling function, passing the - provided return values. The list of return values must match the - function signature's return types. - - This instruction should only be used by ISA-specific epilogue lowering - code. It is equivalent to :inst:`return`, but the return address is - provided explicitly in a register. This style of return instruction is - used by RISC architectures such as ARM and RISC-V. A normal - :inst:`return` will be legalized into this instruction on these - architectures. - """, - ins=(raddr, rvals), is_return=True, is_terminator=True) - FN = Operand( 'FN', entities.func_ref, diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index 294015b4ea..ffc848ef06 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -109,9 +109,8 @@ for inst, f3 in [ RV32.enc(inst.b1, SBzero, BRANCH(f3)) RV64.enc(inst.b1, SBzero, BRANCH(f3)) -# Returns are a special case of JALR. -# Note: Return stack predictors will only recognize this as a return when the -# return address is provided in `x1`. We may want a special encoding to enforce -# that. -RV32.enc(base.return_reg.i32, Iret, JALR()) -RV64.enc(base.return_reg.i64, Iret, JALR()) +# Returns are a special case of JALR using %x1 to hold the return address. +# The return address is provided by a special-purpose `link` return value that +# is added by legalize_signature(). +RV32.enc(base.x_return, Iret, JALR()) +RV64.enc(base.x_return, Iret, JALR()) From d66a9d196ea1ef0d226b03c35211f772f2cb6f9e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Apr 2017 16:26:04 -0700 Subject: [PATCH 0712/3084] Implement binary emission of RISC-V return instructions. The return address is now always supplied in %x1, so the return address predictor will recognize the jalr as a return and not some indirect branch. --- cranelift/filetests/isa/riscv/binary32.cton | 9 ++++++--- lib/cretonne/meta/isa/riscv/recipes.py | 2 +- lib/cretonne/src/isa/riscv/binemit.rs | 10 ++++++++-- 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 984b7b8a80..f62a6187e0 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -2,10 +2,10 @@ test binemit isa riscv -function RV32I() { +function RV32I(i32 link [%x1]) -> i32 link [%x1] { fn0 = function foo() -ebb0: +ebb0(v9999: i32): [-,%x10] v1 = iconst.i32 1 [-,%x21] v2 = iconst.i32 2 @@ -83,7 +83,10 @@ ebb0: call fn0() ; bin: Call(fn0) 000000ef brz v1, ebb3 - fallthrough ebb1 + brnz v1, ebb1 + + ; jalr %x0, %x1, 0 + return v9999 ; bin: 00008067 ebb1: ; beq 0x000 diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index ad682fa74b..b5aa56bc85 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -110,7 +110,7 @@ Iicmp = EncRecipe( # I-type encoding for `jalr` as a return instruction. We won't use the # immediate offset. # The variable return values are not encoded. -Iret = EncRecipe('Iret', MultiAry, size=4, ins=GPR, outs=()) +Iret = EncRecipe('Iret', MultiAry, size=4, ins=(), outs=()) # U-type instructions have a 20-bit immediate that targets bits 12-31. U = EncRecipe( diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 94ba8b53bf..12a3eb462f 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -172,8 +172,14 @@ fn recipe_iicmp(func: &Function, inst: Inst, sink: &mut C } } -fn recipe_iret(_func: &Function, _inst: Inst, _sink: &mut CS) { - unimplemented!() +fn recipe_iret(func: &Function, inst: Inst, sink: &mut CS) { + // Return instructions are always a jalr to %x1. + // The return address is provided as a special-purpose link argument. + put_i(func.encodings[inst].bits(), + 1, // rs1 = %x1 + 0, // no offset. + 0, // rd = %x0: no address written. + sink); } /// U-type instructions. From 3005f903f7eeb27d0721665d079fbb3cbd54dce1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 20 Apr 2017 14:38:03 -0700 Subject: [PATCH 0713/3084] Move the verifier into a verifier/mod.rs file. Make room for verifier sub-modules in separate files. --- lib/cretonne/src/{verifier.rs => verifier/mod.rs} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename lib/cretonne/src/{verifier.rs => verifier/mod.rs} (100%) diff --git a/lib/cretonne/src/verifier.rs b/lib/cretonne/src/verifier/mod.rs similarity index 100% rename from lib/cretonne/src/verifier.rs rename to lib/cretonne/src/verifier/mod.rs From 866efd91b7dd5b19aea77a7c270e783492b05c0c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Apr 2017 09:49:03 -0700 Subject: [PATCH 0714/3084] Add an enable_verifier setting. This is off by default, but enabled by the parser when reading a textual IL file. Test files can still override the default to turn off verification. The setting enables IL verifier passes at critical points of the compilation pipeline. --- lib/cretonne/meta/base/settings.py | 8 ++++++++ lib/cretonne/src/settings.rs | 1 + lib/reader/src/parser.rs | 14 ++++++++++++-- 3 files changed, 21 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/meta/base/settings.py b/lib/cretonne/meta/base/settings.py index 1f7e21668b..186c808524 100644 --- a/lib/cretonne/meta/base/settings.py +++ b/lib/cretonne/meta/base/settings.py @@ -18,6 +18,14 @@ opt_level = EnumSetting( """, 'default', 'best', 'fastest') +enable_verifier = BoolSetting( + """ + Run the Cretonne IL verifier at strategic times during compilation. + + This makes compilation slower but catches many bugs. The verifier is + disabled by default, except when reading Cretonne IL from a text file. + """) + is_64bit = BoolSetting("Enable 64-bit code generation") is_compressed = BoolSetting("Enable compressed instructions") diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 23ff7d688d..204588c7c1 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -271,6 +271,7 @@ mod tests { assert_eq!(f.to_string(), "[shared]\n\ opt_level = \"default\"\n\ + enable_verifier = false\n\ is_64bit = false\n\ is_compressed = false\n\ enable_float = true\n\ diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index f96d60518a..eae8c925b0 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -17,7 +17,7 @@ use cretonne::ir::immediates::{Imm64, Offset32, Uoffset32, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs}; use cretonne::isa::{self, TargetIsa, Encoding}; -use cretonne::settings; +use cretonne::settings::{self, Configurable}; use testfile::{TestFile, Details, Comment}; use error::{Location, Error, Result}; use lexer::{self, Lexer, Token}; @@ -577,6 +577,13 @@ impl<'a> Parser<'a> { let mut isas = Vec::new(); let mut flag_builder = settings::builder(); + // Change the default for `enable_verifier` to `true`. It defaults to `false` because it + // would slow down normal compilation, but when we're reading IL from a text file we're + // either testing or debugging Cretonne, and verification makes sense. + flag_builder + .set_bool("enable_verifier", true) + .expect("Missing enable_verifier setting"); + while let Some(Token::Identifier(command)) = self.token() { match command { "set" => { @@ -1849,7 +1856,10 @@ mod tests { assert_eq!(tf.commands[0].command, "cfg"); assert_eq!(tf.commands[1].command, "verify"); match tf.isa_spec { - IsaSpec::None(s) => assert!(!s.enable_float()), + IsaSpec::None(s) => { + assert!(s.enable_verifier()); + assert!(!s.enable_float()); + } _ => panic!("unexpected ISAs"), } assert_eq!(tf.preamble_comments.len(), 2); From bac47f2fb87c2859e5dc5281f1ccae1b70c8b1e3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Apr 2017 10:55:12 -0700 Subject: [PATCH 0715/3084] Add global CtonResult and CtonError types. These are for reporting the overall result of compiling a function. --- lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/result.rs | 68 ++++++++++++++++++++++++++++++++ lib/cretonne/src/verifier/mod.rs | 7 ++++ 3 files changed, 76 insertions(+) create mode 100644 lib/cretonne/src/result.rs diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index e44899f8c2..986c45355b 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -21,6 +21,7 @@ pub mod flowgraph; pub mod ir; pub mod isa; pub mod regalloc; +pub mod result; pub mod settings; pub mod sparse_map; pub mod verifier; diff --git a/lib/cretonne/src/result.rs b/lib/cretonne/src/result.rs new file mode 100644 index 0000000000..6605f1daff --- /dev/null +++ b/lib/cretonne/src/result.rs @@ -0,0 +1,68 @@ +//! Result and error types representing the outcome of compiling a function. + +use verifier; +use std::error::Error as StdError; +use std::fmt; +use std::result; + +/// A compilation error. +/// +/// When Cretonne fails to compile a function, it will return one of these error codes. +#[derive(Debug)] +pub enum CtonError { + /// An IL verifier error. + /// + /// This always represents a bug, either in the code that generated IL for Cretonne, or a bug + /// in Cretonne itself. + Verifier(verifier::Error), + + /// An implementation limit was exceeded. + /// + /// Cretonne can compile very large and complicated functions, but the implementation has + /// limits that cause compilation to fail when they are exceeded. + /// + /// See http://cretonne.readthedocs.io/en/latest/langref.html#implementation-limits + ImplLimitExceeded, + + /// The code size for the function is too large. + /// + /// Different target ISAs may impose a limit on the size of a compiled function. If that limit + /// is exceeded, compilation fails. + CodeTooLarge, +} + +/// A Cretonne compilation result. +pub type CtonResult = result::Result<(), CtonError>; + +impl fmt::Display for CtonError { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match *self { + CtonError::Verifier(ref e) => write!(f, "Verifier error: {}", e), + CtonError::ImplLimitExceeded | + CtonError::CodeTooLarge => f.write_str(self.description()), + } + } +} + +impl StdError for CtonError { + fn description(&self) -> &str { + match *self { + CtonError::Verifier(ref e) => &e.message, + CtonError::ImplLimitExceeded => "Implementation limit exceeded", + CtonError::CodeTooLarge => "Code for function is too large", + } + } + fn cause(&self) -> Option<&StdError> { + match *self { + CtonError::Verifier(ref e) => Some(e), + CtonError::ImplLimitExceeded | + CtonError::CodeTooLarge => None, + } + } +} + +impl From for CtonError { + fn from(e: verifier::Error) -> CtonError { + CtonError::Verifier(e) + } +} diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index ee78a07501..c6c2fe792e 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -59,6 +59,7 @@ use ir::instructions::{InstructionFormat, BranchInfo, ResolvedConstraint, CallIn use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, StackSlot, Value, Type}; use Context; +use std::error as std_error; use std::fmt::{self, Display, Formatter}; use std::result; use std::collections::BTreeSet; @@ -78,6 +79,12 @@ impl Display for Error { } } +impl std_error::Error for Error { + fn description(&self) -> &str { + &self.message + } +} + /// Verifier result. pub type Result = result::Result; From 6e95b08df192aab1de717bd136844f3ba08a7a4a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Apr 2017 12:03:05 -0700 Subject: [PATCH 0716/3084] Verifier results are always void. No need for a type parameter. --- lib/cretonne/src/context.rs | 2 +- lib/cretonne/src/verifier/mod.rs | 46 ++++++++++++++++---------------- 2 files changed, 24 insertions(+), 24 deletions(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 051adc47aa..680ddbffeb 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -52,7 +52,7 @@ impl Context { /// /// The `TargetIsa` argument is currently unused, but the verifier will soon be able to also /// check ISA-dependent constraints. - pub fn verify<'a, ISA: Into>>(&self, _isa: ISA) -> verifier::Result<()> { + pub fn verify<'a, ISA: Into>>(&self, _isa: ISA) -> verifier::Result { verifier::verify_context(self) } diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index c6c2fe792e..b6f8ba8710 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -86,7 +86,7 @@ impl std_error::Error for Error { } /// Verifier result. -pub type Result = result::Result; +pub type Result = result::Result<(), Error>; // Create an `Err` variant of `Result` from a location and `format!` arguments. macro_rules! err { @@ -106,12 +106,12 @@ macro_rules! err { } /// Verify `func`. -pub fn verify_function(func: &Function) -> Result<()> { +pub fn verify_function(func: &Function) -> Result { Verifier::new(func).run() } /// Verify `ctx`. -pub fn verify_context(ctx: &Context) -> Result<()> { +pub fn verify_context(ctx: &Context) -> Result { let verifier = Verifier::new(&ctx.func); verifier.domtree_integrity(&ctx.domtree)?; verifier.cfg_integrity(&ctx.cfg)?; @@ -135,7 +135,7 @@ impl<'a> Verifier<'a> { } } - fn ebb_integrity(&self, ebb: Ebb, inst: Inst) -> Result<()> { + fn ebb_integrity(&self, ebb: Ebb, inst: Inst) -> Result { let is_terminator = self.func.dfg[inst].opcode().is_terminator(); let is_last_inst = self.func.layout.last_inst(ebb) == Some(inst); @@ -173,7 +173,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn instruction_integrity(&self, inst: Inst) -> Result<()> { + fn instruction_integrity(&self, inst: Inst) -> Result { let inst_data = &self.func.dfg[inst]; let dfg = &self.func.dfg; @@ -201,7 +201,7 @@ impl<'a> Verifier<'a> { self.verify_entity_references(inst) } - fn verify_entity_references(&self, inst: Inst) -> Result<()> { + fn verify_entity_references(&self, inst: Inst) -> Result { use ir::instructions::InstructionData::*; for &arg in self.func.dfg.inst_args(inst) { @@ -279,7 +279,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn verify_ebb(&self, inst: Inst, e: Ebb) -> Result<()> { + fn verify_ebb(&self, inst: Inst, e: Ebb) -> Result { if !self.func.dfg.ebb_is_valid(e) { err!(inst, "invalid ebb reference {}", e) } else { @@ -287,7 +287,7 @@ impl<'a> Verifier<'a> { } } - fn verify_sig_ref(&self, inst: Inst, s: SigRef) -> Result<()> { + fn verify_sig_ref(&self, inst: Inst, s: SigRef) -> Result { if !self.func.dfg.signatures.is_valid(s) { err!(inst, "invalid signature reference {}", s) } else { @@ -295,7 +295,7 @@ impl<'a> Verifier<'a> { } } - fn verify_func_ref(&self, inst: Inst, f: FuncRef) -> Result<()> { + fn verify_func_ref(&self, inst: Inst, f: FuncRef) -> Result { if !self.func.dfg.ext_funcs.is_valid(f) { err!(inst, "invalid function reference {}", f) } else { @@ -303,7 +303,7 @@ impl<'a> Verifier<'a> { } } - fn verify_stack_slot(&self, inst: Inst, ss: StackSlot) -> Result<()> { + fn verify_stack_slot(&self, inst: Inst, ss: StackSlot) -> Result { if !self.func.stack_slots.is_valid(ss) { err!(inst, "invalid stack slot {}", ss) } else { @@ -311,7 +311,7 @@ impl<'a> Verifier<'a> { } } - fn verify_value_list(&self, inst: Inst, l: &ValueList) -> Result<()> { + fn verify_value_list(&self, inst: Inst, l: &ValueList) -> Result { if !l.is_valid(&self.func.dfg.value_lists) { err!(inst, "invalid value list reference {:?}", l) } else { @@ -319,7 +319,7 @@ impl<'a> Verifier<'a> { } } - fn verify_jump_table(&self, inst: Inst, j: JumpTable) -> Result<()> { + fn verify_jump_table(&self, inst: Inst, j: JumpTable) -> Result { if !self.func.jump_tables.is_valid(j) { err!(inst, "invalid jump table reference {}", j) } else { @@ -327,7 +327,7 @@ impl<'a> Verifier<'a> { } } - fn verify_value(&self, loc_inst: Inst, v: Value) -> Result<()> { + fn verify_value(&self, loc_inst: Inst, v: Value) -> Result { let dfg = &self.func.dfg; if !dfg.value_is_valid(v) { return err!(loc_inst, "invalid value reference {}", v); @@ -378,7 +378,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn domtree_integrity(&self, domtree: &DominatorTree) -> Result<()> { + fn domtree_integrity(&self, domtree: &DominatorTree) -> Result { // We consider two `DominatorTree`s to be equal if they return the same immediate // dominator for each EBB. Therefore the current domtree is valid if it matches the freshly // computed one. @@ -396,7 +396,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck_entry_block_arguments(&self) -> Result<()> { + fn typecheck_entry_block_arguments(&self) -> Result { if let Some(ebb) = self.func.layout.entry_block() { let expected_types = &self.func.signature.argument_types; let ebb_arg_count = self.func.dfg.num_ebb_args(ebb); @@ -419,7 +419,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck(&self, inst: Inst) -> Result<()> { + fn typecheck(&self, inst: Inst) -> Result { let inst_data = &self.func.dfg[inst]; let constraints = inst_data.opcode().constraints(); @@ -446,7 +446,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck_results(&self, inst: Inst, ctrl_type: Type) -> Result<()> { + fn typecheck_results(&self, inst: Inst, ctrl_type: Type) -> Result { let mut i = 0; for &result in self.func.dfg.inst_results(inst) { let result_type = self.func.dfg.value_type(result); @@ -473,7 +473,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck_fixed_args(&self, inst: Inst, ctrl_type: Type) -> Result<()> { + fn typecheck_fixed_args(&self, inst: Inst, ctrl_type: Type) -> Result { let constraints = self.func.dfg[inst].opcode().constraints(); for (i, &arg) in self.func.dfg.inst_fixed_args(inst).iter().enumerate() { @@ -504,7 +504,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck_variable_args(&self, inst: Inst) -> Result<()> { + fn typecheck_variable_args(&self, inst: Inst) -> Result { match self.func.dfg[inst].analyze_branch(&self.func.dfg.value_lists) { BranchInfo::SingleDest(ebb, _) => { let iter = self.func @@ -552,7 +552,7 @@ impl<'a> Verifier<'a> { fn typecheck_variable_args_iterator>(&self, inst: Inst, iter: I) - -> Result<()> { + -> Result { let variable_args = self.func.dfg.inst_variable_args(inst); let mut i = 0; @@ -583,7 +583,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck_return(&self, inst: Inst) -> Result<()> { + fn typecheck_return(&self, inst: Inst) -> Result { if self.func.dfg[inst].opcode().is_return() { let args = self.func.dfg.inst_variable_args(inst); let expected_types = &self.func.signature.return_types; @@ -605,7 +605,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn cfg_integrity(&self, cfg: &ControlFlowGraph) -> Result<()> { + fn cfg_integrity(&self, cfg: &ControlFlowGraph) -> Result { let mut expected_succs = BTreeSet::::new(); let mut got_succs = BTreeSet::::new(); let mut expected_preds = BTreeSet::::new(); @@ -653,7 +653,7 @@ impl<'a> Verifier<'a> { Ok(()) } - pub fn run(&self) -> Result<()> { + pub fn run(&self) -> Result { self.typecheck_entry_block_arguments()?; for ebb in self.func.layout.ebbs() { for inst in self.func.layout.ebb_insts(ebb) { From c4b794f7cf822223bb031862d33fc720f5a72cd8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Apr 2017 12:36:35 -0700 Subject: [PATCH 0717/3084] Run the verifier in the Context methods when it is enabled. The test drivers can stop calling comp_ctx.verify because legalize() and regalloc() do it themselves now. This also makes it possible for those two passes to return other CtonError codes in the future, not just verifier errors. --- cranelift/src/filetest/legalizer.rs | 7 +++---- cranelift/src/filetest/regalloc.rs | 10 +++++----- cranelift/src/utils.rs | 10 ++++++++++ lib/cretonne/src/context.rs | 16 ++++++++++++++-- 4 files changed, 32 insertions(+), 11 deletions(-) diff --git a/cranelift/src/filetest/legalizer.rs b/cranelift/src/filetest/legalizer.rs index 1a5f774664..c6e08ffe29 100644 --- a/cranelift/src/filetest/legalizer.rs +++ b/cranelift/src/filetest/legalizer.rs @@ -8,7 +8,7 @@ use cretonne::{self, write_function}; use cretonne::ir::Function; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result, run_filecheck}; -use utils::pretty_verifier_error; +use utils::pretty_error; struct TestLegalizer; @@ -40,9 +40,8 @@ impl SubTest for TestLegalizer { let isa = context.isa.expect("legalizer needs an ISA"); comp_ctx.flowgraph(); - comp_ctx.legalize(isa); - comp_ctx.verify(isa) - .map_err(|e| pretty_verifier_error(&comp_ctx.func, e))?; + comp_ctx.legalize(isa) + .map_err(|e| pretty_error(&comp_ctx.func, e))?; let mut text = String::new(); write_function(&mut text, &comp_ctx.func, Some(isa)).map_err(|e| e.to_string())?; diff --git a/cranelift/src/filetest/regalloc.rs b/cranelift/src/filetest/regalloc.rs index daf7adcb07..87045eb764 100644 --- a/cranelift/src/filetest/regalloc.rs +++ b/cranelift/src/filetest/regalloc.rs @@ -10,7 +10,7 @@ use cretonne::{self, write_function}; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result, run_filecheck}; use std::borrow::Cow; -use utils::pretty_verifier_error; +use utils::pretty_error; struct TestRegalloc; @@ -45,10 +45,10 @@ impl SubTest for TestRegalloc { comp_ctx.flowgraph(); // TODO: Should we have an option to skip legalization? - comp_ctx.legalize(isa); - comp_ctx.regalloc(isa); - comp_ctx.verify(isa) - .map_err(|e| pretty_verifier_error(&comp_ctx.func, e))?; + comp_ctx.legalize(isa) + .map_err(|e| pretty_error(&comp_ctx.func, e))?; + comp_ctx.regalloc(isa) + .map_err(|e| pretty_error(&comp_ctx.func, e))?; let mut text = String::new(); write_function(&mut text, &comp_ctx.func, Some(isa)).map_err(|e| e.to_string())?; diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index d2a6fae3c7..3519c55627 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -2,6 +2,7 @@ use cretonne::ir::entities::AnyEntity; use cretonne::{ir, verifier, write_function}; +use cretonne::result::CtonError; use std::fmt::Write; use std::fs::File; use std::io::{Result, Read}; @@ -45,6 +46,15 @@ pub fn pretty_verifier_error(func: &ir::Function, err: verifier::Error) -> Strin msg } +/// Pretty-print a Cretonne error. +pub fn pretty_error(func: &ir::Function, err: CtonError) -> String { + if let CtonError::Verifier(e) = err { + pretty_verifier_error(func, e) + } else { + err.to_string() + } +} + #[test] fn test_match_directive() { assert_eq!(match_directive("; foo: bar ", "foo:"), Some("bar")); diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 680ddbffeb..71dfa4539b 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -15,6 +15,7 @@ use ir::Function; use isa::TargetIsa; use legalize_function; use regalloc; +use result::CtonResult; use verifier; /// Persistent data structures and compilation pipeline. @@ -56,9 +57,19 @@ impl Context { verifier::verify_context(self) } + /// Run the verifier only if the `enable_verifier` setting is true. + pub fn verify_if(&self, isa: &TargetIsa) -> CtonResult { + if isa.flags().enable_verifier() { + self.verify(isa).map_err(Into::into) + } else { + Ok(()) + } + } + /// Run the legalizer for `isa` on the function. - pub fn legalize(&mut self, isa: &TargetIsa) { + pub fn legalize(&mut self, isa: &TargetIsa) -> CtonResult { legalize_function(&mut self.func, &mut self.cfg, isa); + self.verify_if(isa) } /// Recompute the control flow graph and dominator tree. @@ -68,8 +79,9 @@ impl Context { } /// Run the register allocator. - pub fn regalloc(&mut self, isa: &TargetIsa) { + pub fn regalloc(&mut self, isa: &TargetIsa) -> CtonResult { self.regalloc .run(isa, &mut self.func, &self.cfg, &self.domtree); + self.verify_if(isa) } } From c5da572ebbd0b88196ea790454c40fd86237bf16 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Apr 2017 13:35:20 -0700 Subject: [PATCH 0718/3084] Add a liveness verifier. The liveness verifier will check that the live ranges are consistent with the function. It runs as part of the register allocation pipeline when enable_verifier is set. The initial implementation checks the live ranges, but not the ISA-specific constraints and affinities. --- cranelift/src/utils.rs | 2 +- lib/cretonne/src/context.rs | 2 +- lib/cretonne/src/regalloc/context.rs | 11 +- lib/cretonne/src/regalloc/liverange.rs | 11 +- lib/cretonne/src/verifier/liveness.rs | 179 +++++++++++++++++++++++++ lib/cretonne/src/verifier/mod.rs | 38 +++--- 6 files changed, 220 insertions(+), 23 deletions(-) create mode 100644 lib/cretonne/src/verifier/liveness.rs diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index 3519c55627..1bcba4891a 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -40,7 +40,7 @@ pub fn pretty_verifier_error(func: &ir::Function, err: verifier::Error) -> Strin AnyEntity::Inst(inst) => { write!(msg, "\n{}: {}\n\n", inst, func.dfg.display_inst(inst)).unwrap() } - _ => {} + _ => msg.push('\n'), } write_function(&mut msg, func, None).unwrap(); msg diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 71dfa4539b..f8a207c6fc 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -81,7 +81,7 @@ impl Context { /// Run the register allocator. pub fn regalloc(&mut self, isa: &TargetIsa) -> CtonResult { self.regalloc - .run(isa, &mut self.func, &self.cfg, &self.domtree); + .run(isa, &mut self.func, &self.cfg, &self.domtree)?; self.verify_if(isa) } } diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 2c59ab9797..5d91a258fa 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -11,6 +11,8 @@ use isa::TargetIsa; use regalloc::coloring::Coloring; use regalloc::live_value_tracker::LiveValueTracker; use regalloc::liveness::Liveness; +use result::CtonResult; +use verifier::verify_liveness; /// Persistent memory allocations for register allocation. pub struct Context { @@ -40,7 +42,8 @@ impl Context { isa: &TargetIsa, func: &mut Function, cfg: &ControlFlowGraph, - domtree: &DominatorTree) { + domtree: &DominatorTree) + -> CtonResult { // `Liveness` and `Coloring` are self-clearing. // Tracker state (dominator live sets) is actually reused between the spilling and coloring // phases. @@ -49,10 +52,16 @@ impl Context { // First pass: Liveness analysis. self.liveness.compute(isa, func, cfg); + if isa.flags().enable_verifier() { + verify_liveness(isa, func, cfg, &self.liveness)?; + } + // TODO: Second pass: Spilling. // Third pass: Reload and coloring. self.coloring .run(isa, func, domtree, &mut self.liveness, &mut self.tracker); + + Ok(()) } } diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 94c16cb9ae..0594af19ed 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -174,12 +174,12 @@ pub struct LiveRange { /// for contiguous EBBs where all but the last live-in interval covers the whole EBB. /// #[derive(Copy, Clone)] -struct Interval { +pub struct Interval { /// Interval starting point. /// /// Since this interval does not represent the def of the value, it must begin at an EBB header /// where the value is live-in. - begin: Ebb, + pub begin: Ebb, /// Interval end point. /// @@ -190,7 +190,7 @@ struct Interval { /// When this represents multiple contiguous live-in intervals, this is the end point of the /// last interval. The other intervals end at the terminator instructions of their respective /// EBB. - end: Inst, + pub end: Inst, } impl Interval { @@ -368,6 +368,11 @@ impl LiveRange { .ok() .map(|n| self.liveins[n].end) } + + /// Get all the live-in intervals. + pub fn liveins(&self) -> &[Interval] { + &self.liveins + } } /// Allow a `LiveRange` to be stored in a `SparseMap` indexed by values. diff --git a/lib/cretonne/src/verifier/liveness.rs b/lib/cretonne/src/verifier/liveness.rs new file mode 100644 index 0000000000..d80f18a8b3 --- /dev/null +++ b/lib/cretonne/src/verifier/liveness.rs @@ -0,0 +1,179 @@ +//! Liveness verifier. + +use flowgraph::ControlFlowGraph; +use ir::{Function, Inst, Value, ProgramOrder, ProgramPoint, ExpandedProgramPoint}; +use ir::entities::AnyEntity; +use isa::TargetIsa; +use regalloc::liveness::Liveness; +use regalloc::liverange::LiveRange; +use std::cmp::Ordering; +use verifier::Result; + +/// Verify liveness information for `func`. +/// +/// The provided control flow graph is assumed to be sound. +/// +/// - All values in the program must have a live range. +/// - The live range def point must match where the value is defined. +/// - The live range must reach all uses. +/// - When a live range is live-in to an EBB, it must be live at all the predecessors. +/// - The live range affinity must be compatible with encoding constraints. +/// +/// We don't verify that live ranges are minimal. This would require recomputing live ranges for +/// all values. +pub fn verify_liveness(_isa: &TargetIsa, + func: &Function, + cfg: &ControlFlowGraph, + liveness: &Liveness) + -> Result { + let verifier = LivenessVerifier { + func: func, + cfg: cfg, + liveness: liveness, + }; + verifier.check_ebbs()?; + verifier.check_insts()?; + Ok(()) +} + +struct LivenessVerifier<'a> { + func: &'a Function, + cfg: &'a ControlFlowGraph, + liveness: &'a Liveness, +} + +impl<'a> LivenessVerifier<'a> { + /// Check all EBB arguments. + fn check_ebbs(&self) -> Result { + for ebb in self.func.layout.ebbs() { + for &val in self.func.dfg.ebb_args(ebb) { + let lr = match self.liveness.get(val) { + Some(lr) => lr, + None => return err!(ebb, "EBB arg {} has no live range", val), + }; + self.check_lr(ebb.into(), val, lr)?; + } + } + Ok(()) + } + + /// Check all instructions. + fn check_insts(&self) -> Result { + for ebb in self.func.layout.ebbs() { + for inst in self.func.layout.ebb_insts(ebb) { + // Check the defs. + for &val in self.func.dfg.inst_results(inst) { + let lr = match self.liveness.get(val) { + Some(lr) => lr, + None => return err!(inst, "{} has no live range", val), + }; + self.check_lr(inst.into(), val, lr)?; + } + + // Check the uses. + for &val in self.func.dfg.inst_args(inst) { + let lr = match self.liveness.get(val) { + Some(lr) => lr, + None => return err!(inst, "{} has no live range", val), + }; + if !self.live_at_use(lr, inst) { + return err!(inst, "{} is not live at this use", val); + } + } + } + } + Ok(()) + } + + /// Is `lr` live at the use `inst`? + fn live_at_use(&self, lr: &LiveRange, inst: Inst) -> bool { + let l = &self.func.layout; + + // Check if `inst` is in the def range, not including the def itself. + if l.cmp(lr.def(), inst) == Ordering::Less && + l.cmp(inst, lr.def_local_end()) != Ordering::Greater { + return true; + } + + // Otherwise see if `inst` is in one of the live-in ranges. + match lr.livein_local_end(l.inst_ebb(inst).unwrap(), l) { + Some(end) => l.cmp(inst, end) != Ordering::Greater, + None => false, + } + } + + /// Check the integrity of the live range `lr`. + fn check_lr(&self, def: ProgramPoint, val: Value, lr: &LiveRange) -> Result { + let l = &self.func.layout; + + let loc: AnyEntity = match def.into() { + ExpandedProgramPoint::Ebb(e) => e.into(), + ExpandedProgramPoint::Inst(i) => i.into(), + }; + if lr.def() != def { + return err!(loc, "Wrong live range def ({}) for {}", lr.def(), val); + } + if lr.is_dead() { + if !lr.is_local() { + return err!(loc, "Dead live range {} should be local", val); + } else { + return Ok(()); + } + } + let def_ebb = match def.into() { + ExpandedProgramPoint::Ebb(e) => e, + ExpandedProgramPoint::Inst(i) => l.inst_ebb(i).unwrap(), + }; + match lr.def_local_end().into() { + ExpandedProgramPoint::Ebb(e) => { + return err!(loc, "Def local range for {} can't end at {}", val, e) + } + ExpandedProgramPoint::Inst(i) => { + if self.func.layout.inst_ebb(i) != Some(def_ebb) { + return err!(loc, "Def local end for {} in wrong ebb", val); + } + } + } + + // Now check the live-in intervals against the CFG. + for &livein in lr.liveins() { + let mut ebb = livein.begin; + if !l.is_ebb_inserted(ebb) { + return err!(loc, "{} livein at {} which is not in the layout", val, ebb); + } + let end_ebb = match l.inst_ebb(livein.end) { + Some(e) => e, + None => { + return err!(loc, + "{} livein for {} ends at {} which is not in the layout", + val, + ebb, + livein.end) + } + }; + + // Check all the EBBs in the interval independently. + loop { + // If `val` is live-in at `ebb`, it must be live at all the predecessors. + for &(_, pred) in self.cfg.get_predecessors(ebb) { + if !self.live_at_use(lr, pred) { + return err!(pred, + "{} is live in to {} but not live at predecessor", + val, + ebb); + } + } + + if ebb == end_ebb { + break; + } + ebb = match l.next_ebb(ebb) { + Some(e) => e, + None => return err!(loc, "end of {} livein ({}) never reached", val, end_ebb), + }; + } + } + + Ok(()) + } +} diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index b6f8ba8710..f2e8a7fc50 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -64,6 +64,27 @@ use std::fmt::{self, Display, Formatter}; use std::result; use std::collections::BTreeSet; +pub use self::liveness::verify_liveness; + +// Create an `Err` variant of `Result` from a location and `format!` arguments. +macro_rules! err { + ( $loc:expr, $msg:expr ) => { + Err(::verifier::Error { + location: $loc.into(), + message: String::from($msg), + }) + }; + + ( $loc:expr, $fmt:expr, $( $arg:expr ),+ ) => { + Err(::verifier::Error { + location: $loc.into(), + message: format!( $fmt, $( $arg ),+ ), + }) + }; +} + +mod liveness; + /// A verifier error. #[derive(Debug, PartialEq, Eq)] pub struct Error { @@ -88,23 +109,6 @@ impl std_error::Error for Error { /// Verifier result. pub type Result = result::Result<(), Error>; -// Create an `Err` variant of `Result` from a location and `format!` arguments. -macro_rules! err { - ( $loc:expr, $msg:expr ) => { - Err(Error { - location: $loc.into(), - message: String::from($msg), - }) - }; - - ( $loc:expr, $fmt:expr, $( $arg:expr ),+ ) => { - Err(Error { - location: $loc.into(), - message: format!( $fmt, $( $arg ),+ ), - }) - }; -} - /// Verify `func`. pub fn verify_function(func: &Function) -> Result { Verifier::new(func).run() From d0d5f3bb2659e4f5c2ee98ce071a4ccbac18a438 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Apr 2017 16:23:33 -0700 Subject: [PATCH 0719/3084] Run the post-regalloc verification inside the regalloc context. This means that we can verify the basics with verify_context before moving on to verifying the liveness information. Live ranges are now verified immediately after computing them and after register allocation is complete. --- lib/cretonne/src/context.rs | 5 ++--- lib/cretonne/src/regalloc/context.rs | 6 +++++- lib/cretonne/src/verifier/mod.rs | 12 ++++++------ 3 files changed, 13 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index f8a207c6fc..e4ba8a0667 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -54,7 +54,7 @@ impl Context { /// The `TargetIsa` argument is currently unused, but the verifier will soon be able to also /// check ISA-dependent constraints. pub fn verify<'a, ISA: Into>>(&self, _isa: ISA) -> verifier::Result { - verifier::verify_context(self) + verifier::verify_context(&self.func, &self.cfg, &self.domtree) } /// Run the verifier only if the `enable_verifier` setting is true. @@ -81,7 +81,6 @@ impl Context { /// Run the register allocator. pub fn regalloc(&mut self, isa: &TargetIsa) -> CtonResult { self.regalloc - .run(isa, &mut self.func, &self.cfg, &self.domtree)?; - self.verify_if(isa) + .run(isa, &mut self.func, &self.cfg, &self.domtree) } } diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 5d91a258fa..8f399c65de 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -12,7 +12,7 @@ use regalloc::coloring::Coloring; use regalloc::live_value_tracker::LiveValueTracker; use regalloc::liveness::Liveness; use result::CtonResult; -use verifier::verify_liveness; +use verifier::{verify_context, verify_liveness}; /// Persistent memory allocations for register allocation. pub struct Context { @@ -62,6 +62,10 @@ impl Context { self.coloring .run(isa, func, domtree, &mut self.liveness, &mut self.tracker); + if isa.flags().enable_verifier() { + verify_context(func, cfg, domtree)?; + verify_liveness(isa, func, cfg, &self.liveness)?; + } Ok(()) } } diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index f2e8a7fc50..1cc8f740dd 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -58,7 +58,6 @@ use ir::entities::AnyEntity; use ir::instructions::{InstructionFormat, BranchInfo, ResolvedConstraint, CallInfo}; use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, StackSlot, Value, Type}; -use Context; use std::error as std_error; use std::fmt::{self, Display, Formatter}; use std::result; @@ -114,11 +113,12 @@ pub fn verify_function(func: &Function) -> Result { Verifier::new(func).run() } -/// Verify `ctx`. -pub fn verify_context(ctx: &Context) -> Result { - let verifier = Verifier::new(&ctx.func); - verifier.domtree_integrity(&ctx.domtree)?; - verifier.cfg_integrity(&ctx.cfg)?; +/// Verify `func` after checking the integrity of associated context data structures `cfg` and +/// `domtree`. +pub fn verify_context(func: &Function, cfg: &ControlFlowGraph, domtree: &DominatorTree) -> Result { + let verifier = Verifier::new(func); + verifier.cfg_integrity(cfg)?; + verifier.domtree_integrity(domtree)?; verifier.run() } From 43ce26e64bfc8b04d68fd67bc8c7c47a2a762356 Mon Sep 17 00:00:00 2001 From: Eric Anholt Date: Sat, 22 Apr 2017 16:58:29 -0700 Subject: [PATCH 0720/3084] Verify that the instruction encoding matches what the ISA would encode. Fixes #69 --- .../filetests/isa/riscv/verify-encoding.cton | 21 +++++++ cranelift/src/filetest/runone.rs | 2 +- cranelift/src/filetest/verifier.rs | 2 +- lib/cretonne/src/context.rs | 4 +- lib/cretonne/src/regalloc/context.rs | 2 +- lib/cretonne/src/verifier/mod.rs | 59 ++++++++++++++++--- 6 files changed, 78 insertions(+), 12 deletions(-) create mode 100644 cranelift/filetests/isa/riscv/verify-encoding.cton diff --git a/cranelift/filetests/isa/riscv/verify-encoding.cton b/cranelift/filetests/isa/riscv/verify-encoding.cton new file mode 100644 index 0000000000..73d28c842a --- /dev/null +++ b/cranelift/filetests/isa/riscv/verify-encoding.cton @@ -0,0 +1,21 @@ +test verifier +isa riscv + +function RV32I(i32 link [%x1]) -> i32 link [%x1] { + fn0 = function foo() + +ebb0(v9999: i32): + ; iconst.i32 needs legalizing, so it should throw a + [R#0,-] v1 = iconst.i32 1 ; error: Instruction failed to re-encode + return v9999 +} + +function RV32I(i32 link [%x1]) -> i32 link [%x1] { + fn0 = function foo() + +ebb0(v9999: i32): + v1 = iconst.i32 1 + v2 = iconst.i32 2 + [R#0,-] v3 = iadd v1, v2 ; error: Instruction re-encoding + return v9999 +} diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index bc16725d74..60231d22f4 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -116,7 +116,7 @@ fn run_one_test<'a>(tuple: (&'a SubTest, &'a Flags, Option<&'a TargetIsa>), // Should we run the verifier before this test? if !context.verified && test.needs_verifier() { - verify_function(&func).map_err(|e| pretty_verifier_error(&func, e))?; + verify_function(&func, isa).map_err(|e| pretty_verifier_error(&func, e))?; context.verified = true; } diff --git a/cranelift/src/filetest/verifier.rs b/cranelift/src/filetest/verifier.rs index 5812dc9db0..1dd5cc88b4 100644 --- a/cranelift/src/filetest/verifier.rs +++ b/cranelift/src/filetest/verifier.rs @@ -51,7 +51,7 @@ impl SubTest for TestVerifier { } } - match verify_function(func) { + match verify_function(func, context.isa) { Ok(_) => { match expected { None => Ok(()), diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index e4ba8a0667..d8b6af4d0b 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -53,8 +53,8 @@ impl Context { /// /// The `TargetIsa` argument is currently unused, but the verifier will soon be able to also /// check ISA-dependent constraints. - pub fn verify<'a, ISA: Into>>(&self, _isa: ISA) -> verifier::Result { - verifier::verify_context(&self.func, &self.cfg, &self.domtree) + pub fn verify<'a, ISA: Into>>(&self, isa: ISA) -> verifier::Result { + verifier::verify_context(&self.func, &self.cfg, &self.domtree, isa.into()) } /// Run the verifier only if the `enable_verifier` setting is true. diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 8f399c65de..b57edebb5f 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -63,7 +63,7 @@ impl Context { .run(isa, func, domtree, &mut self.liveness, &mut self.tracker); if isa.flags().enable_verifier() { - verify_context(func, cfg, domtree)?; + verify_context(func, cfg, domtree, Some(isa))?; verify_liveness(isa, func, cfg, &self.liveness)?; } Ok(()) diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 1cc8f740dd..6bf9b155c6 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -58,6 +58,7 @@ use ir::entities::AnyEntity; use ir::instructions::{InstructionFormat, BranchInfo, ResolvedConstraint, CallInfo}; use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, StackSlot, Value, Type}; +use isa::TargetIsa; use std::error as std_error; use std::fmt::{self, Display, Formatter}; use std::result; @@ -109,14 +110,18 @@ impl std_error::Error for Error { pub type Result = result::Result<(), Error>; /// Verify `func`. -pub fn verify_function(func: &Function) -> Result { - Verifier::new(func).run() +pub fn verify_function(func: &Function, isa: Option<&TargetIsa>) -> Result { + Verifier::new(func, isa).run() } /// Verify `func` after checking the integrity of associated context data structures `cfg` and /// `domtree`. -pub fn verify_context(func: &Function, cfg: &ControlFlowGraph, domtree: &DominatorTree) -> Result { - let verifier = Verifier::new(func); +pub fn verify_context(func: &Function, + cfg: &ControlFlowGraph, + domtree: &DominatorTree, + isa: Option<&TargetIsa>) + -> Result { + let verifier = Verifier::new(func, isa); verifier.cfg_integrity(cfg)?; verifier.domtree_integrity(domtree)?; verifier.run() @@ -126,16 +131,18 @@ struct Verifier<'a> { func: &'a Function, cfg: ControlFlowGraph, domtree: DominatorTree, + isa: Option<&'a TargetIsa>, } impl<'a> Verifier<'a> { - pub fn new(func: &'a Function) -> Verifier { + pub fn new(func: &'a Function, isa: Option<&'a TargetIsa>) -> Verifier<'a> { let cfg = ControlFlowGraph::with_function(func); let domtree = DominatorTree::with_function(func, &cfg); Verifier { func: func, cfg: cfg, domtree: domtree, + isa: isa, } } @@ -657,6 +664,42 @@ impl<'a> Verifier<'a> { Ok(()) } + /// If the verifier has been set up with an ISA, make sure that the recorded encoding for the + /// instruction (if any) matches how the ISA would encode it. + fn verify_encoding(&self, inst: Inst) -> Result { + if let Some(isa) = self.isa { + let encoding = self.func + .encodings + .get(inst) + .cloned() + .unwrap_or_default(); + if encoding.is_legal() { + let verify_encoding = + isa.encode(&self.func.dfg, + &self.func.dfg[inst], + self.func.dfg.ctrl_typevar(inst)); + match verify_encoding { + Ok(verify_encoding) => { + if verify_encoding != encoding { + return err!(inst, + "Instruction re-encoding {} doesn't match {}", + isa.encoding_info().display(verify_encoding), + isa.encoding_info().display(encoding)); + } + } + Err(e) => { + return err!(inst, + "Instruction failed to re-encode {}: {:?}", + isa.encoding_info().display(encoding), + e) + } + } + } + } + + Ok(()) + } + pub fn run(&self) -> Result { self.typecheck_entry_block_arguments()?; for ebb in self.func.layout.ebbs() { @@ -664,8 +707,10 @@ impl<'a> Verifier<'a> { self.ebb_integrity(ebb, inst)?; self.instruction_integrity(inst)?; self.typecheck(inst)?; + self.verify_encoding(inst)?; } } + Ok(()) } } @@ -692,7 +737,7 @@ mod tests { #[test] fn empty() { let func = Function::new(); - let verifier = Verifier::new(&func); + let verifier = Verifier::new(&func, None); assert_eq!(verifier.run(), Ok(())); } @@ -705,7 +750,7 @@ mod tests { func.dfg .make_inst(InstructionData::Nullary { opcode: Opcode::Jump }); func.layout.append_inst(nullary_with_bad_opcode, ebb0); - let verifier = Verifier::new(&func); + let verifier = Verifier::new(&func, None); assert_err_with_msg!(verifier.run(), "instruction format"); } } From 042418b36754a8230eadbbc41400879fa7b24dd7 Mon Sep 17 00:00:00 2001 From: Eric Anholt Date: Sat, 22 Apr 2017 17:39:43 -0700 Subject: [PATCH 0721/3084] Update a comment for the move of display_enc(). --- lib/cretonne/src/isa/encoding.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/isa/encoding.rs b/lib/cretonne/src/isa/encoding.rs index c5151978aa..3d866727a4 100644 --- a/lib/cretonne/src/isa/encoding.rs +++ b/lib/cretonne/src/isa/encoding.rs @@ -60,7 +60,7 @@ impl fmt::Display for Encoding { } /// Temporary object that holds enough context to properly display an encoding. -/// This is meant to be created by `TargetIsa::display_enc()`. +/// This is meant to be created by `EncInfo::display()`. pub struct DisplayEncoding { pub encoding: Encoding, pub recipe_names: &'static [&'static str], From d47f43df11ad99f637311a0bc3983e6ae852e04b Mon Sep 17 00:00:00 2001 From: Eric Anholt Date: Sun, 23 Apr 2017 11:45:12 -0700 Subject: [PATCH 0722/3084] Make sure we double back after legalizing an instruction. The other legalizer cases have a continue after setting the position to double back, while this one didn't. Make sure that we do, in case another legalizer block is added after this one. --- lib/cretonne/src/legalizer/mod.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 7949aed311..50300027a2 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -91,6 +91,7 @@ pub fn legalize_function(func: &mut Function, cfg: &mut ControlFlowGraph, isa: & // unsound. Should we attempt to detect that? if changed { pos.set_position(prev_pos); + continue; } } } From 440add86e70055e6654f79a4459809507dc2cc75 Mon Sep 17 00:00:00 2001 From: Eric Anholt Date: Sun, 23 Apr 2017 11:55:14 -0700 Subject: [PATCH 0723/3084] Make sure that encodings has entries for all instructions after legalize(). If we generated new instructions as part of legalize, and the new instructions failed to legalize, we'd be left with a func.encodings[] that would panic when you dereferenced the inst. --- lib/cretonne/src/legalizer/mod.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 50300027a2..edd233ccee 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -100,6 +100,7 @@ pub fn legalize_function(func: &mut Function, cfg: &mut ControlFlowGraph, isa: & prev_pos = pos.position(); } } + func.encodings.resize(func.dfg.num_insts()); } // Include legalization patterns that were generated by `gen_legalizer.py` from the `XForms` in From d078c546e57b395697d4b7d4df8fd761da170beb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 25 Apr 2017 15:50:59 -0700 Subject: [PATCH 0724/3084] Add an EntityMap::get_or_default() method. This covers a common pattern for secondary entity maps: Get the value in the map or the default value for out-of-range keys. --- cranelift/src/filetest/binemit.rs | 7 ++----- lib/cretonne/src/binemit/relaxation.rs | 2 +- lib/cretonne/src/entity_map.rs | 5 +++++ lib/cretonne/src/verifier/mod.rs | 6 +----- lib/cretonne/src/write.rs | 8 +------- 5 files changed, 10 insertions(+), 18 deletions(-) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 449e0887e6..2f695bff1e 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -106,10 +106,7 @@ impl SubTest for TestBinEmit { // Give an encoding to any instruction that doesn't already have one. for ebb in func.layout.ebbs() { for inst in func.layout.ebb_insts(ebb) { - if !func.encodings - .get(inst) - .map(|e| e.is_legal()) - .unwrap_or(false) { + if !func.encodings.get_or_default(inst).is_legal() { if let Ok(enc) = isa.encode(&func.dfg, &func.dfg[inst], func.dfg.ctrl_typevar(inst)) { @@ -157,7 +154,7 @@ impl SubTest for TestBinEmit { ebb); for inst in func.layout.ebb_insts(ebb) { sink.text.clear(); - let enc = func.encodings.get(inst).cloned().unwrap_or_default(); + let enc = func.encodings.get_or_default(inst); // Send legal encodings into the emitter. if enc.is_legal() { diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index 179540dad0..74158a61e4 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -64,7 +64,7 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) { } while let Some(inst) = pos.next_inst() { - let enc = func.encodings.get(inst).cloned().unwrap_or_default(); + let enc = func.encodings.get_or_default(inst); let size = encinfo.bytes(enc); // See if this might be a branch that is out of range. diff --git a/lib/cretonne/src/entity_map.rs b/lib/cretonne/src/entity_map.rs index f9ac258f92..cee4215a4e 100644 --- a/lib/cretonne/src/entity_map.rs +++ b/lib/cretonne/src/entity_map.rs @@ -148,6 +148,11 @@ impl EntityMap } &mut self.elems[k.index()] } + + /// Get the element at `k` or the default value if `k` is out of range. + pub fn get_or_default(&self, k: K) -> V { + self.elems.get(k.index()).cloned().unwrap_or_default() + } } /// Immutable indexing into an `EntityMap`. diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 6bf9b155c6..e3d5607c99 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -668,11 +668,7 @@ impl<'a> Verifier<'a> { /// instruction (if any) matches how the ISA would encode it. fn verify_encoding(&self, inst: Inst) -> Result { if let Some(isa) = self.isa { - let encoding = self.func - .encodings - .get(inst) - .cloned() - .unwrap_or_default(); + let encoding = self.func.encodings.get_or_default(inst); if encoding.is_legal() { let verify_encoding = isa.encode(&self.func.dfg, diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index fcfb843c0c..5d46ccb624 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -189,13 +189,7 @@ fn write_instruction(w: &mut Write, if !func.locations.is_empty() { let regs = isa.register_info(); for &r in func.dfg.inst_results(inst) { - write!(s, - ",{}", - func.locations - .get(r) - .cloned() - .unwrap_or_default() - .display(®s))? + write!(s, ",{}", func.locations.get_or_default(r).display(®s))? } } write!(s, "]")?; From e67f5e210fb7bc540f68d5b081fed5b13f072d64 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Apr 2017 10:11:50 -0700 Subject: [PATCH 0725/3084] Add a TargetIsa::regclass_for_abi_type() function. The legalize_signature() function will return ArgumentLoc::Reg arguments that contain a register unit. However, the register also needs to be able to associate a register class with the argument values to fully track the used registers. When values are defined by instructions, the register class is part for the operand constraints for the instruction. For values defined on ABI boundaries like function arguments and return values from a call, the register class is provided by the new regclass_for_abi_type() function. Provide implementations of this function in abi modules of all the targets, even those that don't have a legalize_signature() implementation yet. Since we're adding abi modules to all targets, move the legalize_signature() stubs in there and make the function mandatory in TargetIsa. All targets will eventually need this function. --- lib/cretonne/src/isa/arm32/abi.rs | 27 +++++++++++++++++++++++++++ lib/cretonne/src/isa/arm32/mod.rs | 11 ++++++++++- lib/cretonne/src/isa/arm64/abi.rs | 18 ++++++++++++++++++ lib/cretonne/src/isa/arm64/mod.rs | 11 ++++++++++- lib/cretonne/src/isa/intel/abi.rs | 18 ++++++++++++++++++ lib/cretonne/src/isa/intel/mod.rs | 11 ++++++++++- lib/cretonne/src/isa/mod.rs | 14 ++++++++++---- lib/cretonne/src/isa/riscv/abi.rs | 6 ++++++ lib/cretonne/src/isa/riscv/mod.rs | 6 +++++- 9 files changed, 114 insertions(+), 8 deletions(-) create mode 100644 lib/cretonne/src/isa/arm32/abi.rs create mode 100644 lib/cretonne/src/isa/arm64/abi.rs create mode 100644 lib/cretonne/src/isa/intel/abi.rs diff --git a/lib/cretonne/src/isa/arm32/abi.rs b/lib/cretonne/src/isa/arm32/abi.rs new file mode 100644 index 0000000000..c4e0ffc2ee --- /dev/null +++ b/lib/cretonne/src/isa/arm32/abi.rs @@ -0,0 +1,27 @@ +//! ARM ABI implementation. + +use ir; +use isa::RegClass; +use settings as shared_settings; +use super::registers::{S, D, Q, GPR}; + +/// Legalize `sig`. +pub fn legalize_signature(_sig: &mut ir::Signature, + _flags: &shared_settings::Flags, + _current: bool) { + unimplemented!() +} + +/// Get register class for a type appearing in a legalized signature. +pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { + if ty.is_int() { + GPR + } else { + match ty.bits() { + 32 => S, + 64 => D, + 128 => Q, + _ => panic!("Unexpected {} ABI type for arm32", ty), + } + } +} diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index b76ffa32cb..735fcd4b2c 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -1,6 +1,7 @@ //! ARM 32-bit Instruction Set Architecture. pub mod settings; +mod abi; mod binemit; mod enc_tables; mod registers; @@ -9,7 +10,7 @@ use binemit::CodeSink; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, EncInfo, Encoding, Legalize}; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Encoding, Legalize}; use ir; #[allow(dead_code)] @@ -77,6 +78,14 @@ impl TargetIsa for Isa { }) } + fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { + abi::legalize_signature(sig, &self.shared_flags, current) + } + + fn regclass_for_abi_type(&self, ty: ir::Type) -> RegClass { + abi::regclass_for_abi_type(ty) + } + fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } diff --git a/lib/cretonne/src/isa/arm64/abi.rs b/lib/cretonne/src/isa/arm64/abi.rs new file mode 100644 index 0000000000..8c2e9ed7d6 --- /dev/null +++ b/lib/cretonne/src/isa/arm64/abi.rs @@ -0,0 +1,18 @@ +//! ARM 64 ABI implementation. + +use ir; +use isa::RegClass; +use settings as shared_settings; +use super::registers::{GPR, FPR}; + +/// Legalize `sig`. +pub fn legalize_signature(_sig: &mut ir::Signature, + _flags: &shared_settings::Flags, + _current: bool) { + unimplemented!() +} + +/// Get register class for a type appearing in a legalized signature. +pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { + if ty.is_int() { GPR } else { FPR } +} diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 2499028919..91a8a50f4e 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -1,6 +1,7 @@ //! ARM 64-bit Instruction Set Architecture. pub mod settings; +mod abi; mod binemit; mod enc_tables; mod registers; @@ -9,7 +10,7 @@ use binemit::CodeSink; use super::super::settings as shared_settings; use isa::enc_tables::{lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, EncInfo, Encoding, Legalize}; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Encoding, Legalize}; use ir; #[allow(dead_code)] @@ -70,6 +71,14 @@ impl TargetIsa for Isa { }) } + fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { + abi::legalize_signature(sig, &self.shared_flags, current) + } + + fn regclass_for_abi_type(&self, ty: ir::Type) -> RegClass { + abi::regclass_for_abi_type(ty) + } + fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs new file mode 100644 index 0000000000..437925cd40 --- /dev/null +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -0,0 +1,18 @@ +//! Intel ABI implementation. + +use ir; +use isa::RegClass; +use settings as shared_settings; +use super::registers::{GPR, FPR}; + +/// Legalize `sig`. +pub fn legalize_signature(_sig: &mut ir::Signature, + _flags: &shared_settings::Flags, + _current: bool) { + unimplemented!() +} + +/// Get register class for a type appearing in a legalized signature. +pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { + if ty.is_int() { GPR } else { FPR } +} diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 60ce3a795d..768150eb50 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -1,6 +1,7 @@ //! Intel Instruction Set Architectures. pub mod settings; +mod abi; mod binemit; mod enc_tables; mod registers; @@ -9,7 +10,7 @@ use binemit::CodeSink; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, EncInfo, Encoding, Legalize}; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Encoding, Legalize}; use ir; #[allow(dead_code)] @@ -77,6 +78,14 @@ impl TargetIsa for Isa { }) } + fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { + abi::legalize_signature(sig, &self.shared_flags, current) + } + + fn regclass_for_abi_type(&self, ty: ir::Type) -> RegClass { + abi::regclass_for_abi_type(ty) + } + fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index e951ad31de..bd9e0caf9a 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -168,7 +168,7 @@ pub trait TargetIsa { /// The legalizer will adapt argument and return values as necessary at all ABI boundaries. /// /// When this function is called to legalize the signature of the function currently begin - /// compiler, `_current` is true. The legalized signature can then also contain special purpose + /// compiler, `current` is true. The legalized signature can then also contain special purpose /// arguments and return values such as: /// /// - A `link` argument representing the link registers on RISC architectures that don't push @@ -183,9 +183,15 @@ pub trait TargetIsa { /// Arguments and return values for the caller's frame pointer and other callee-saved registers /// should not be added by this function. These arguments are not added until after register /// allocation. - fn legalize_signature(&self, _sig: &mut Signature, _current: bool) { - unimplemented!() - } + fn legalize_signature(&self, sig: &mut Signature, current: bool); + + /// Get the register class that should be used to represent an ABI argument or return value of + /// type `ty`. This should be the top-level register class that contains the argument + /// registers. + /// + /// This function can assume that it will only be asked to provide register classes for types + /// that `legalize_signature()` produces in `ArgumentLoc::Reg` entries. + fn regclass_for_abi_type(&self, ty: Type) -> RegClass; /// Emit binary machine code for a single instruction into the `sink` trait object. /// diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index 453a9c7be8..f13e6d2801 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -7,6 +7,7 @@ use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; use ir::{Signature, Type, ArgumentType, ArgumentLoc, ArgumentExtension, ArgumentPurpose}; +use isa::RegClass; use isa::riscv::registers::{GPR, FPR}; use settings as shared_settings; @@ -102,3 +103,8 @@ pub fn legalize_signature(sig: &mut Signature, flags: &shared_settings::Flags, c sig.return_types.push(link); } } + +/// Get register class for a type appearing in a legalized signature. +pub fn regclass_for_abi_type(ty: Type) -> RegClass { + if ty.is_float() { FPR } else { GPR } +} diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index ac5057103c..60b795156a 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -10,7 +10,7 @@ use super::super::settings as shared_settings; use binemit::CodeSink; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, EncInfo, Encoding, Legalize}; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Encoding, Legalize}; use ir::{Function, Inst, InstructionData, DataFlowGraph, Signature, Type}; #[allow(dead_code)] @@ -83,6 +83,10 @@ impl TargetIsa for Isa { abi::legalize_signature(sig, &self.shared_flags, current) } + fn regclass_for_abi_type(&self, ty: Type) -> RegClass { + abi::regclass_for_abi_type(ty) + } + fn emit_inst(&self, func: &Function, inst: Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } From a20ae9eade83c307c7719fb66311bd21dd7561fd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Apr 2017 10:47:43 -0700 Subject: [PATCH 0726/3084] Assign an affinity to function argument values. Use the argument types from the function signature to initialize the affinity of register and stack arguments. --- lib/cretonne/src/regalloc/affinity.rs | 12 +++++++++++- lib/cretonne/src/regalloc/liveness.rs | 19 +++++++++++++------ 2 files changed, 24 insertions(+), 7 deletions(-) diff --git a/lib/cretonne/src/regalloc/affinity.rs b/lib/cretonne/src/regalloc/affinity.rs index 529828759e..f56b96d434 100644 --- a/lib/cretonne/src/regalloc/affinity.rs +++ b/lib/cretonne/src/regalloc/affinity.rs @@ -8,7 +8,8 @@ //! subclass. This is just a hint, and the register allocator is allowed to pick a register from a //! larger register class instead. -use isa::{RegInfo, RegClassIndex, OperandConstraint, ConstraintKind}; +use ir::{ArgumentType, ArgumentLoc}; +use isa::{TargetIsa, RegInfo, RegClassIndex, OperandConstraint, ConstraintKind}; /// Preferred register allocation for an SSA value. #[derive(Clone, Copy)] @@ -42,6 +43,15 @@ impl Affinity { } } + /// Create an affinity that matches an ABI argument for `isa`. + pub fn abi(arg: &ArgumentType, isa: &TargetIsa) -> Affinity { + match arg.location { + ArgumentLoc::Unassigned => Affinity::Any, + ArgumentLoc::Reg(_) => Affinity::Reg(isa.regclass_for_abi_type(arg.value_type).into()), + ArgumentLoc::Stack(_) => Affinity::Stack, + } + } + /// Merge an operand constraint into this affinity. /// /// Note that this does not guarantee that the register allocator will pick a register that diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index cb937d67a2..737470b2d1 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -190,6 +190,7 @@ type LiveRangeSet = SparseMap; /// Create it if necessary. fn get_or_create<'a>(lrset: &'a mut LiveRangeSet, value: Value, + isa: &TargetIsa, func: &Function, enc_info: &EncInfo) -> &'a mut LiveRange { @@ -211,11 +212,17 @@ fn get_or_create<'a>(lrset: &'a mut LiveRangeSet, .map(Affinity::new) .unwrap_or_default(); } - ValueDef::Arg(ebb, _) => { + ValueDef::Arg(ebb, num) => { def = ebb.into(); - // Don't apply any affinity to EBB arguments. - // They could be in a register or on the stack. - affinity = Default::default(); + if func.layout.entry_block() == Some(ebb) { + // The affinity for entry block arguments can be inferred from the function + // signature. + affinity = Affinity::abi(&func.signature.argument_types[num], isa); + } else { + // Don't apply any affinity to normal EBB arguments. + // They could be in a register or on the stack. + affinity = Default::default(); + } } }; lrset.insert(LiveRange::new(value, def, affinity)); @@ -309,7 +316,7 @@ impl Liveness { // TODO: When we implement DCE, we can use the absence of a live range to indicate // an unused value. for &def in func.dfg.inst_results(inst) { - get_or_create(&mut self.ranges, def, func, &enc_info); + get_or_create(&mut self.ranges, def, isa, func, &enc_info); } // Iterator of constraints, one per value operand. @@ -322,7 +329,7 @@ impl Liveness { for &arg in func.dfg.inst_args(inst) { // Get the live range, create it as a dead range if necessary. - let lr = get_or_create(&mut self.ranges, arg, func, &enc_info); + let lr = get_or_create(&mut self.ranges, arg, isa, func, &enc_info); // Extend the live range to reach this use. extend_to_use(lr, ebb, inst, &mut self.worklist, func, cfg); From 2f81fbdb7779ca4500cf668129df82e54ea489b2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Apr 2017 11:15:46 -0700 Subject: [PATCH 0727/3084] Add an Affinity::display() method. Make it possible to display affinities given a RegInfo reference. --- lib/cretonne/src/isa/registers.rs | 12 ++++++++++++ lib/cretonne/src/regalloc/affinity.rs | 25 +++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 864b950f63..3c4f1b411a 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -153,6 +153,12 @@ impl RegClassData { } } +impl fmt::Display for RegClassData { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + f.write_str(self.name) + } +} + /// A small reference to a register class. /// /// Use this when storing register classes in compact data structures. The `RegInfo::rc()` method @@ -176,6 +182,12 @@ impl From for RegClassIndex { } } +impl fmt::Display for RegClassIndex { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "rci{}", self.0) + } +} + /// Information about the registers in an ISA. /// /// The `RegUnit` data structure collects all relevant static information about the registers in an diff --git a/lib/cretonne/src/regalloc/affinity.rs b/lib/cretonne/src/regalloc/affinity.rs index f56b96d434..f6ca587485 100644 --- a/lib/cretonne/src/regalloc/affinity.rs +++ b/lib/cretonne/src/regalloc/affinity.rs @@ -8,6 +8,7 @@ //! subclass. This is just a hint, and the register allocator is allowed to pick a register from a //! larger register class instead. +use std::fmt; use ir::{ArgumentType, ArgumentLoc}; use isa::{TargetIsa, RegInfo, RegClassIndex, OperandConstraint, ConstraintKind}; @@ -75,4 +76,28 @@ impl Affinity { Affinity::Stack => {} } } + + /// Return an object that can display this value affinity, using the register info from the + /// target ISA. + pub fn display<'a, R: Into>>(self, regs: R) -> DisplayAffinity<'a> { + DisplayAffinity(self, regs.into()) + } +} + +/// Displaying an `Affinity` correctly requires the associated `RegInfo` from the target ISA. +pub struct DisplayAffinity<'a>(Affinity, Option<&'a RegInfo>); + +impl<'a> fmt::Display for DisplayAffinity<'a> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self.0 { + Affinity::None => write!(f, "none"), + Affinity::Stack => write!(f, "stack"), + Affinity::Reg(rci) => { + match self.1 { + Some(regs) => write!(f, "{}", regs.rc(rci)), + None => write!(f, "{}", rci), + } + } + } + } } From 9db131c3bffc9a02ffc7d6d6e3ed2ff9ac27ddc1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 25 Apr 2017 15:40:06 -0700 Subject: [PATCH 0728/3084] Rename Affinity::Any to Affinity::None. This value affinity doesn't mean "whatever", it means that the value does not interact with any real instructions, where "real" means instructions that have a legal encoding. Update the liveness verifier to check this property: - Encoded instructions can only use real values. - Encoded instructions define real values. - Ghost instructions define ghost values. --- lib/cretonne/src/regalloc/affinity.rs | 23 +++++++++++++----- lib/cretonne/src/regalloc/coloring.rs | 4 +++- lib/cretonne/src/verifier/liveness.rs | 34 ++++++++++++++++++++++++++- 3 files changed, 53 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/src/regalloc/affinity.rs b/lib/cretonne/src/regalloc/affinity.rs index f6ca587485..64d2b640df 100644 --- a/lib/cretonne/src/regalloc/affinity.rs +++ b/lib/cretonne/src/regalloc/affinity.rs @@ -15,8 +15,11 @@ use isa::{TargetIsa, RegInfo, RegClassIndex, OperandConstraint, ConstraintKind}; /// Preferred register allocation for an SSA value. #[derive(Clone, Copy)] pub enum Affinity { - /// Don't care. This value can go anywhere. - Any, + /// No affinity. + /// + /// This indicates a value that is not defined or used by any real instructions. It is a ghost + /// value that won't appear in the final program. + None, /// This value should be placed in a spill slot on the stack. Stack, @@ -27,14 +30,14 @@ pub enum Affinity { impl Default for Affinity { fn default() -> Self { - Affinity::Any + Affinity::None } } impl Affinity { /// Create an affinity that satisfies a single constraint. /// - /// This will never create the indifferent `Affinity::Any`. + /// This will never create an `Affinity::None`. /// Use the `Default` implementation for that. pub fn new(constraint: &OperandConstraint) -> Affinity { if constraint.kind == ConstraintKind::Stack { @@ -47,19 +50,27 @@ impl Affinity { /// Create an affinity that matches an ABI argument for `isa`. pub fn abi(arg: &ArgumentType, isa: &TargetIsa) -> Affinity { match arg.location { - ArgumentLoc::Unassigned => Affinity::Any, + ArgumentLoc::Unassigned => Affinity::None, ArgumentLoc::Reg(_) => Affinity::Reg(isa.regclass_for_abi_type(arg.value_type).into()), ArgumentLoc::Stack(_) => Affinity::Stack, } } + /// Is this the `None` affinity? + pub fn is_none(self) -> bool { + match self { + Affinity::None => true, + _ => false, + } + } + /// Merge an operand constraint into this affinity. /// /// Note that this does not guarantee that the register allocator will pick a register that /// satisfies the constraint. pub fn merge(&mut self, constraint: &OperandConstraint, reg_info: &RegInfo) { match *self { - Affinity::Any => *self = Affinity::new(constraint), + Affinity::None => *self = Affinity::new(constraint), Affinity::Reg(rc) => { // If the preferred register class is a subclass of the constraint, there's no need // to change anything. diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index b9e5fe7148..84e81660d7 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -327,7 +327,9 @@ impl<'a> Context<'a> { } } Affinity::Stack => unimplemented!(), - Affinity::Any => unimplemented!(), + Affinity::None => { + panic!("Encoded instruction defines {} with no affinity", lv.value) + } } } diff --git a/lib/cretonne/src/verifier/liveness.rs b/lib/cretonne/src/verifier/liveness.rs index d80f18a8b3..d3d841b68e 100644 --- a/lib/cretonne/src/verifier/liveness.rs +++ b/lib/cretonne/src/verifier/liveness.rs @@ -21,12 +21,13 @@ use verifier::Result; /// /// We don't verify that live ranges are minimal. This would require recomputing live ranges for /// all values. -pub fn verify_liveness(_isa: &TargetIsa, +pub fn verify_liveness(isa: &TargetIsa, func: &Function, cfg: &ControlFlowGraph, liveness: &Liveness) -> Result { let verifier = LivenessVerifier { + isa: isa, func: func, cfg: cfg, liveness: liveness, @@ -37,6 +38,7 @@ pub fn verify_liveness(_isa: &TargetIsa, } struct LivenessVerifier<'a> { + isa: &'a TargetIsa, func: &'a Function, cfg: &'a ControlFlowGraph, liveness: &'a Liveness, @@ -61,6 +63,8 @@ impl<'a> LivenessVerifier<'a> { fn check_insts(&self) -> Result { for ebb in self.func.layout.ebbs() { for inst in self.func.layout.ebb_insts(ebb) { + let encoding = self.func.encodings.get_or_default(inst); + // Check the defs. for &val in self.func.dfg.inst_results(inst) { let lr = match self.liveness.get(val) { @@ -68,6 +72,24 @@ impl<'a> LivenessVerifier<'a> { None => return err!(inst, "{} has no live range", val), }; self.check_lr(inst.into(), val, lr)?; + + if encoding.is_legal() { + // A legal instruction is not allowed to define ghost values. + if lr.affinity.is_none() { + return err!(inst, + "{} is a ghost value defined by a real [{}] instruction", + val, + self.isa.encoding_info().display(encoding)); + } + } else { + // A non-encoded instruction can only define ghost values. + if !lr.affinity.is_none() { + return err!(inst, + "{} is a real {} value defined by a ghost instruction", + val, + lr.affinity.display(&self.isa.register_info())); + } + } } // Check the uses. @@ -79,6 +101,16 @@ impl<'a> LivenessVerifier<'a> { if !self.live_at_use(lr, inst) { return err!(inst, "{} is not live at this use", val); } + + if encoding.is_legal() { + // A legal instruction is not allowed to depend on ghost values. + if lr.affinity.is_none() { + return err!(inst, + "{} is a ghost value used by a real [{}] instruction", + val, + self.isa.encoding_info().display(encoding)); + } + } } } } From a4acc26d5ab041b060def35e7f0d7911073311d8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Apr 2017 13:13:15 -0700 Subject: [PATCH 0729/3084] Add an enable_e setting for the RV32E instruction set. This limited RISC-V version only has registers %x0 - %x15. Make sure the ABI lowering code doesn't use the banned registers for arguments. --- cranelift/filetests/isa/riscv/abi-e.cton | 14 ++++++++++++++ lib/cretonne/meta/isa/riscv/settings.py | 3 +++ lib/cretonne/src/isa/riscv/abi.rs | 18 ++++++++++++------ lib/cretonne/src/isa/riscv/mod.rs | 3 +-- lib/cretonne/src/isa/riscv/settings.rs | 3 ++- 5 files changed, 32 insertions(+), 9 deletions(-) create mode 100644 cranelift/filetests/isa/riscv/abi-e.cton diff --git a/cranelift/filetests/isa/riscv/abi-e.cton b/cranelift/filetests/isa/riscv/abi-e.cton new file mode 100644 index 0000000000..f770a3558f --- /dev/null +++ b/cranelift/filetests/isa/riscv/abi-e.cton @@ -0,0 +1,14 @@ +; Test the legalization of function signatures for RV32E. +test legalizer +isa riscv enable_e + +; regex: V=v\d+ + +function f() { + ; Spilling into the stack args after %x15 since %16 and up are not + ; available in RV32E. + sig0 = signature(i64, i64, i64, i64) -> i64 + ; check: sig0 = signature(i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [0], i32 [4]) -> i32 [%x10], i32 [%x11] +ebb0: + return +} diff --git a/lib/cretonne/meta/isa/riscv/settings.py b/lib/cretonne/meta/isa/riscv/settings.py index e6fe230f89..c8b88db55e 100644 --- a/lib/cretonne/meta/isa/riscv/settings.py +++ b/lib/cretonne/meta/isa/riscv/settings.py @@ -18,6 +18,9 @@ enable_m = BoolSetting( "Enable the use of 'M' instructions if available", default=True) +enable_e = BoolSetting( + "Enable the 'RV32E' instruction set with only 16 registers") + use_m = And(supports_m, enable_m) use_a = And(supports_a, shared.enable_atomics) use_f = And(supports_f, shared.enable_float) diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index f13e6d2801..0df3806529 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -8,24 +8,27 @@ use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; use ir::{Signature, Type, ArgumentType, ArgumentLoc, ArgumentExtension, ArgumentPurpose}; use isa::RegClass; -use isa::riscv::registers::{GPR, FPR}; use settings as shared_settings; +use super::registers::{GPR, FPR}; +use super::settings; struct Args { pointer_bits: u16, pointer_bytes: u32, pointer_type: Type, regs: u32, + reg_limit: u32, offset: u32, } impl Args { - fn new(bits: u16) -> Args { + fn new(bits: u16, enable_e: bool) -> Args { Args { pointer_bits: bits, pointer_bytes: bits as u32 / 8, pointer_type: Type::int(bits).unwrap(), regs: 0, + reg_limit: if enable_e { 6 } else { 8 }, offset: 0, } } @@ -62,7 +65,7 @@ impl ArgAssigner for Args { } } - if self.regs < 8 { + if self.regs < self.reg_limit { // Assign to a register. let reg = if ty.is_float() { FPR.unit(10 + self.regs as usize) @@ -81,13 +84,16 @@ impl ArgAssigner for Args { } /// Legalize `sig` for RISC-V. -pub fn legalize_signature(sig: &mut Signature, flags: &shared_settings::Flags, current: bool) { +pub fn legalize_signature(sig: &mut Signature, + flags: &shared_settings::Flags, + isa_flags: &settings::Flags, + current: bool) { let bits = if flags.is_64bit() { 64 } else { 32 }; - let mut args = Args::new(bits); + let mut args = Args::new(bits, isa_flags.enable_e()); legalize_args(&mut sig.argument_types, &mut args); - let mut rets = Args::new(bits); + let mut rets = Args::new(bits, isa_flags.enable_e()); legalize_args(&mut sig.return_types, &mut rets); if current { diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 60b795156a..fe267ae870 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -79,8 +79,7 @@ impl TargetIsa for Isa { } fn legalize_signature(&self, sig: &mut Signature, current: bool) { - // We can pass in `self.isa_flags` too, if we need it. - abi::legalize_signature(sig, &self.shared_flags, current) + abi::legalize_signature(sig, &self.shared_flags, &self.isa_flags, current) } fn regclass_for_abi_type(&self, ty: Type) -> RegClass { diff --git a/lib/cretonne/src/isa/riscv/settings.rs b/lib/cretonne/src/isa/riscv/settings.rs index c070612d08..aa66d75fd9 100644 --- a/lib/cretonne/src/isa/riscv/settings.rs +++ b/lib/cretonne/src/isa/riscv/settings.rs @@ -24,7 +24,8 @@ mod tests { supports_a = false\n\ supports_f = false\n\ supports_d = false\n\ - enable_m = true\n"); + enable_m = true\n\ + enable_e = false\n"); // Predicates are not part of the Display output. assert_eq!(f.full_float(), false); } From 6197bfff7249b6dcca9a4a670cd915559a2cec6f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Apr 2017 13:54:40 -0700 Subject: [PATCH 0730/3084] Add a TargetIsa::allocatable_registers() method. This gives the target ISA a chance to reserve registers like the stack pointer or hard-wired 0 registers like %x0 on RISC-V. --- cranelift/filetests/regalloc/basic.cton | 3 ++- lib/cretonne/src/isa/arm32/abi.rs | 6 ++++++ lib/cretonne/src/isa/arm32/mod.rs | 5 +++++ lib/cretonne/src/isa/arm64/abi.rs | 6 ++++++ lib/cretonne/src/isa/arm64/mod.rs | 5 +++++ lib/cretonne/src/isa/intel/abi.rs | 6 ++++++ lib/cretonne/src/isa/intel/mod.rs | 5 +++++ lib/cretonne/src/isa/mod.rs | 21 ++++++++++++++------- lib/cretonne/src/isa/riscv/abi.rs | 24 ++++++++++++++++++++++-- lib/cretonne/src/isa/riscv/mod.rs | 19 ++++++++++++------- lib/cretonne/src/regalloc/coloring.rs | 3 +-- lib/cretonne/src/regalloc/mod.rs | 1 + 12 files changed, 85 insertions(+), 19 deletions(-) diff --git a/cranelift/filetests/regalloc/basic.cton b/cranelift/filetests/regalloc/basic.cton index cbff589431..3fc58dabd9 100644 --- a/cranelift/filetests/regalloc/basic.cton +++ b/cranelift/filetests/regalloc/basic.cton @@ -6,7 +6,8 @@ isa riscv function add(i32, i32) { ebb0(v1: i32, v2: i32): v3 = iadd v1, v2 -; check: [R#0c,%x0] +; TODO: This shouldn't clobber the link register. +; check: [R#0c,%x1] ; sameln: iadd return } diff --git a/lib/cretonne/src/isa/arm32/abi.rs b/lib/cretonne/src/isa/arm32/abi.rs index c4e0ffc2ee..93ae0e7e61 100644 --- a/lib/cretonne/src/isa/arm32/abi.rs +++ b/lib/cretonne/src/isa/arm32/abi.rs @@ -2,6 +2,7 @@ use ir; use isa::RegClass; +use regalloc::AllocatableSet; use settings as shared_settings; use super::registers::{S, D, Q, GPR}; @@ -25,3 +26,8 @@ pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { } } } + +/// Get the set of allocatable registers for `func`. +pub fn allocatable_registers(_func: &ir::Function) -> AllocatableSet { + unimplemented!() +} diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 735fcd4b2c..ec9de51917 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -12,6 +12,7 @@ use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encodin use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Encoding, Legalize}; use ir; +use regalloc; #[allow(dead_code)] struct Isa { @@ -86,6 +87,10 @@ impl TargetIsa for Isa { abi::regclass_for_abi_type(ty) } + fn allocatable_registers(&self, func: &ir::Function) -> regalloc::AllocatableSet { + abi::allocatable_registers(func) + } + fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } diff --git a/lib/cretonne/src/isa/arm64/abi.rs b/lib/cretonne/src/isa/arm64/abi.rs index 8c2e9ed7d6..f5a2dc91c7 100644 --- a/lib/cretonne/src/isa/arm64/abi.rs +++ b/lib/cretonne/src/isa/arm64/abi.rs @@ -2,6 +2,7 @@ use ir; use isa::RegClass; +use regalloc::AllocatableSet; use settings as shared_settings; use super::registers::{GPR, FPR}; @@ -16,3 +17,8 @@ pub fn legalize_signature(_sig: &mut ir::Signature, pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { if ty.is_int() { GPR } else { FPR } } + +/// Get the set of allocatable registers for `func`. +pub fn allocatable_registers(_func: &ir::Function) -> AllocatableSet { + unimplemented!() +} diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 91a8a50f4e..4067142f57 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -12,6 +12,7 @@ use isa::enc_tables::{lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Encoding, Legalize}; use ir; +use regalloc; #[allow(dead_code)] struct Isa { @@ -79,6 +80,10 @@ impl TargetIsa for Isa { abi::regclass_for_abi_type(ty) } + fn allocatable_registers(&self, func: &ir::Function) -> regalloc::AllocatableSet { + abi::allocatable_registers(func) + } + fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 437925cd40..e6c5edd5bf 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -2,6 +2,7 @@ use ir; use isa::RegClass; +use regalloc::AllocatableSet; use settings as shared_settings; use super::registers::{GPR, FPR}; @@ -16,3 +17,8 @@ pub fn legalize_signature(_sig: &mut ir::Signature, pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { if ty.is_int() { GPR } else { FPR } } + +/// Get the set of allocatable registers for `func`. +pub fn allocatable_registers(_func: &ir::Function) -> AllocatableSet { + unimplemented!() +} diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 768150eb50..1812a41354 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -12,6 +12,7 @@ use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encodin use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Encoding, Legalize}; use ir; +use regalloc; #[allow(dead_code)] struct Isa { @@ -86,6 +87,10 @@ impl TargetIsa for Isa { abi::regclass_for_abi_type(ty) } + fn allocatable_registers(&self, func: &ir::Function) -> regalloc::AllocatableSet { + abi::allocatable_registers(func) + } + fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index bd9e0caf9a..4b618e0798 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -46,7 +46,8 @@ pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex}; use binemit::CodeSink; use settings; -use ir::{Function, Inst, InstructionData, DataFlowGraph, Signature, Type}; +use ir; +use regalloc; pub mod riscv; pub mod intel; @@ -142,9 +143,9 @@ pub trait TargetIsa { /// /// This is also the main entry point for determining if an instruction is legal. fn encode(&self, - dfg: &DataFlowGraph, - inst: &InstructionData, - ctrl_typevar: Type) + dfg: &ir::DataFlowGraph, + inst: &ir::InstructionData, + ctrl_typevar: ir::Type) -> Result; /// Get a data structure describing the instruction encodings in this ISA. @@ -183,7 +184,7 @@ pub trait TargetIsa { /// Arguments and return values for the caller's frame pointer and other callee-saved registers /// should not be added by this function. These arguments are not added until after register /// allocation. - fn legalize_signature(&self, sig: &mut Signature, current: bool); + fn legalize_signature(&self, sig: &mut ir::Signature, current: bool); /// Get the register class that should be used to represent an ABI argument or return value of /// type `ty`. This should be the top-level register class that contains the argument @@ -191,13 +192,19 @@ pub trait TargetIsa { /// /// This function can assume that it will only be asked to provide register classes for types /// that `legalize_signature()` produces in `ArgumentLoc::Reg` entries. - fn regclass_for_abi_type(&self, ty: Type) -> RegClass; + fn regclass_for_abi_type(&self, ty: ir::Type) -> RegClass; + + /// Get the set of allocatable registers that can be used when compiling `func`. + /// + /// This set excludes reserved registers like the stack pointer and other special-purpose + /// registers. + fn allocatable_registers(&self, func: &ir::Function) -> regalloc::AllocatableSet; /// Emit binary machine code for a single instruction into the `sink` trait object. /// /// Note that this will call `put*` methods on the trait object via its vtable which is not the /// fastest way of emitting code. - fn emit_inst(&self, func: &Function, inst: Inst, sink: &mut CodeSink); + fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink); /// Get a static array of names associated with relocations in this ISA. /// diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index 0df3806529..ab9e485ed5 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -6,8 +6,9 @@ //! This doesn't support the soft-float ABI at the moment. use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; -use ir::{Signature, Type, ArgumentType, ArgumentLoc, ArgumentExtension, ArgumentPurpose}; +use ir::{self, Type, ArgumentType, ArgumentLoc, ArgumentExtension, ArgumentPurpose}; use isa::RegClass; +use regalloc::AllocatableSet; use settings as shared_settings; use super::registers::{GPR, FPR}; use super::settings; @@ -84,7 +85,7 @@ impl ArgAssigner for Args { } /// Legalize `sig` for RISC-V. -pub fn legalize_signature(sig: &mut Signature, +pub fn legalize_signature(sig: &mut ir::Signature, flags: &shared_settings::Flags, isa_flags: &settings::Flags, current: bool) { @@ -114,3 +115,22 @@ pub fn legalize_signature(sig: &mut Signature, pub fn regclass_for_abi_type(ty: Type) -> RegClass { if ty.is_float() { FPR } else { GPR } } + +pub fn allocatable_registers(_func: &ir::Function, isa_flags: &settings::Flags) -> AllocatableSet { + let mut regs = AllocatableSet::new(); + regs.take(GPR, GPR.unit(0)); // Hard-wired 0. + // %x1 is the link register which is available for allocation. + regs.take(GPR, GPR.unit(2)); // Stack pointer. + regs.take(GPR, GPR.unit(3)); // Global pointer. + regs.take(GPR, GPR.unit(4)); // Thread pointer. + // TODO: %x8 is the frame pointer. Reserve it? + + // Remove %x16 and up for RV32E. + if isa_flags.enable_e() { + for u in 16..32 { + regs.take(GPR, GPR.unit(u)); + } + } + + regs +} diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index fe267ae870..cf5aafc7db 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -11,7 +11,8 @@ use binemit::CodeSink; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Encoding, Legalize}; -use ir::{Function, Inst, InstructionData, DataFlowGraph, Signature, Type}; +use ir; +use regalloc; #[allow(dead_code)] struct Isa { @@ -61,9 +62,9 @@ impl TargetIsa for Isa { } fn encode(&self, - _dfg: &DataFlowGraph, - inst: &InstructionData, - ctrl_typevar: Type) + _dfg: &ir::DataFlowGraph, + inst: &ir::InstructionData, + ctrl_typevar: ir::Type) -> Result { lookup_enclist(ctrl_typevar, inst.opcode(), @@ -78,15 +79,19 @@ impl TargetIsa for Isa { }) } - fn legalize_signature(&self, sig: &mut Signature, current: bool) { + fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { abi::legalize_signature(sig, &self.shared_flags, &self.isa_flags, current) } - fn regclass_for_abi_type(&self, ty: Type) -> RegClass { + fn regclass_for_abi_type(&self, ty: ir::Type) -> RegClass { abi::regclass_for_abi_type(ty) } - fn emit_inst(&self, func: &Function, inst: Inst, sink: &mut CodeSink) { + fn allocatable_registers(&self, func: &ir::Function) -> regalloc::AllocatableSet { + abi::allocatable_registers(func, &self.isa_flags) + } + + fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 84e81660d7..f1fc2348ad 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -101,8 +101,7 @@ impl Coloring { encinfo: isa.encoding_info(), domtree: domtree, liveness: liveness, - // TODO: Ask the target ISA about reserved registers etc. - usable_regs: AllocatableSet::new(), + usable_regs: isa.allocatable_registers(func), }; ctx.run(self, func, tracker) } diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index fb19286670..0682be2fd6 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -11,4 +11,5 @@ pub mod coloring; mod affinity; mod context; +pub use self::allocatable_set::AllocatableSet; pub use self::context::Context; From eaf1ed09fc7ede40e6c78cedeffa5035035baad7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Apr 2017 14:38:16 -0700 Subject: [PATCH 0731/3084] Color entry block arguments using the function signature. The arguments to the entry block arrive in registers determined by the ABI. This information is stored in the signature. Use a separate function for coloring entry block arguments using the signature information. We can't handle stack arguments yet. --- cranelift/filetests/regalloc/basic.cton | 3 +- lib/cretonne/src/regalloc/coloring.rs | 79 +++++++++++++++++++++---- 2 files changed, 69 insertions(+), 13 deletions(-) diff --git a/cranelift/filetests/regalloc/basic.cton b/cranelift/filetests/regalloc/basic.cton index 3fc58dabd9..14fa263ac0 100644 --- a/cranelift/filetests/regalloc/basic.cton +++ b/cranelift/filetests/regalloc/basic.cton @@ -6,8 +6,7 @@ isa riscv function add(i32, i32) { ebb0(v1: i32, v2: i32): v3 = iadd v1, v2 -; TODO: This shouldn't clobber the link register. -; check: [R#0c,%x1] +; check: [R#0c,%x5] ; sameln: iadd return } diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index f1fc2348ad..0a5e9e803a 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -37,7 +37,7 @@ use entity_map::EntityMap; use dominator_tree::DominatorTree; -use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph}; +use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph, Signature, ArgumentLoc}; use isa::{TargetIsa, RegInfo, Encoding, EncInfo, ConstraintKind}; use regalloc::affinity::Affinity; use regalloc::allocatable_set::AllocatableSet; @@ -180,14 +180,15 @@ impl<'a> Context<'a> { let (liveins, args) = tracker.ebb_top(ebb, &func.dfg, self.liveness, &func.layout, self.domtree); - // The live-ins have already been assigned a register. Reconstruct the allocatable set. - let mut regs = self.livein_regs(liveins, func); - - // TODO: Arguments to the entry block are pre-colored by the ABI. We should probably call - // a whole other function for that case. - self.color_args(args, &mut regs, &mut func.locations); - - regs + // Arguments to the entry block have ABI constraints. + if func.layout.entry_block() == Some(ebb) { + assert_eq!(liveins.len(), 0); + self.color_entry_args(&func.signature, args, &mut func.locations) + } else { + // The live-ins have already been assigned a register. Reconstruct the allocatable set. + let regs = self.livein_regs(liveins, func); + self.color_args(args, regs, &mut func.locations) + } } /// Initialize a set of allocatable registers from the values that are live-in to a block. @@ -218,14 +219,68 @@ impl<'a> Context<'a> { regs } + /// Color the arguments to the entry block. + /// + /// These are function arguments that should already have assigned register units in the + /// function signature. + /// + /// Return the set of remaining allocatable registers. + fn color_entry_args(&self, + sig: &Signature, + args: &[LiveValue], + locations: &mut EntityMap) + -> AllocatableSet { + assert_eq!(sig.argument_types.len(), args.len()); + + let mut regs = self.usable_regs.clone(); + + for (lv, abi) in args.iter().zip(&sig.argument_types) { + match lv.affinity { + Affinity::Reg(rc_index) => { + let regclass = self.reginfo.rc(rc_index); + if let ArgumentLoc::Reg(regunit) = abi.location { + regs.take(regclass, regunit); + *locations.ensure(lv.value) = ValueLoc::Reg(regunit); + } else { + // This should have been fixed by the reload pass. + panic!("Entry arg {} has {} affinity, but ABI {}", + lv.value, + lv.affinity.display(&self.reginfo), + abi.display(&self.reginfo)); + } + + } + Affinity::Stack => { + if let ArgumentLoc::Stack(_offset) = abi.location { + // TODO: Allocate a stack slot at incoming offset and assign it. + panic!("Unimplemented {}: {} stack allocation", + lv.value, + abi.display(&self.reginfo)); + } else { + // This should have been fixed by the reload pass. + panic!("Entry arg {} has stack affinity, but ABI {}", + lv.value, + abi.display(&self.reginfo)); + } + } + // This is a ghost value, unused in the function. Don't assign it to a location + // either. + Affinity::None => {} + } + } + + regs + } + /// Color the live arguments to the current block. /// /// It is assumed that any live-in register values have already been taken out of the register /// set. fn color_args(&self, args: &[LiveValue], - regs: &mut AllocatableSet, - locations: &mut EntityMap) { + mut regs: AllocatableSet, + locations: &mut EntityMap) + -> AllocatableSet { for lv in args { // Only look at the register arguments. if let Affinity::Reg(rc_index) = lv.affinity { @@ -238,6 +293,8 @@ impl<'a> Context<'a> { *locations.ensure(lv.value) = ValueLoc::Reg(regunit); } } + + regs } /// Color the values defined by `inst` and insert any necessary shuffle code to satisfy From ee5f035e3155e58bdba338eb21f91cf2de80a1e1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 27 Apr 2017 12:28:18 -0700 Subject: [PATCH 0732/3084] Upgrade to Rust 1.17. - Remove some uses of 'static in const and static globals that are no longer needed. - Use the new struct initialization shorthand. --- cranelift/src/cton-util.rs | 2 +- cranelift/src/filetest/concurrent.rs | 16 +-- cranelift/src/filetest/runner.rs | 2 +- cranelift/src/filetest/runone.rs | 4 +- cranelift/src/print_cfg.rs | 2 +- lib/cretonne/meta/gen_encoding.py | 2 +- lib/cretonne/meta/gen_instr.py | 4 +- lib/cretonne/meta/gen_settings.py | 2 +- lib/cretonne/src/ir/builder.rs | 9 +- lib/cretonne/src/ir/dfg.rs | 30 ++--- lib/cretonne/src/ir/extfunc.rs | 4 +- lib/cretonne/src/ir/function.rs | 2 +- lib/cretonne/src/ir/layout.rs | 2 +- lib/cretonne/src/ir/memflags.rs | 2 +- lib/cretonne/src/isa/arm32/mod.rs | 2 +- lib/cretonne/src/isa/arm64/mod.rs | 2 +- lib/cretonne/src/isa/encoding.rs | 5 +- lib/cretonne/src/isa/intel/mod.rs | 2 +- lib/cretonne/src/isa/registers.rs | 2 +- lib/cretonne/src/isa/riscv/mod.rs | 2 +- lib/cretonne/src/iterators.rs | 5 +- lib/cretonne/src/legalizer/split.rs | 10 +- lib/cretonne/src/regalloc/coloring.rs | 4 +- .../src/regalloc/live_value_tracker.rs | 6 +- lib/cretonne/src/regalloc/liverange.rs | 4 +- lib/cretonne/src/verifier/liveness.rs | 8 +- lib/cretonne/src/verifier/mod.rs | 8 +- lib/filecheck/src/checker.rs | 8 +- lib/filecheck/src/explain.rs | 2 +- lib/reader/src/lexer.rs | 4 +- lib/reader/src/parser.rs | 114 ++++++++---------- 31 files changed, 118 insertions(+), 153 deletions(-) diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index fea4d879be..70fcbc3c84 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -17,7 +17,7 @@ mod cat; mod print_cfg; mod rsfilecheck; -const USAGE: &'static str = " +const USAGE: &str = " Cretonne code generator utility Usage: diff --git a/cranelift/src/filetest/concurrent.rs b/cranelift/src/filetest/concurrent.rs index 9ea1114000..dbdd76f35c 100644 --- a/cranelift/src/filetest/concurrent.rs +++ b/cranelift/src/filetest/concurrent.rs @@ -51,8 +51,8 @@ impl ConcurrentRunner { ConcurrentRunner { request_tx: Some(request_tx), - reply_rx: reply_rx, - handles: handles, + reply_rx, + handles, } } @@ -120,10 +120,7 @@ fn worker_thread(thread_num: usize, // Tell them we're starting this job. // The receiver should always be present for this as long as we have jobs. replies - .send(Reply::Starting { - jobid: jobid, - thread_num: thread_num, - }) + .send(Reply::Starting { jobid, thread_num }) .unwrap(); let result = catch_unwind(|| runone::run(path.as_path())).unwrap_or_else(|e| { @@ -142,12 +139,7 @@ fn worker_thread(thread_num: usize, dbg!("FAIL: {}", msg); } - replies - .send(Reply::Done { - jobid: jobid, - result: result, - }) - .unwrap(); + replies.send(Reply::Done { jobid, result }).unwrap(); } }) .unwrap() diff --git a/cranelift/src/filetest/runner.rs b/cranelift/src/filetest/runner.rs index 1993ca5877..23f94fe78f 100644 --- a/cranelift/src/filetest/runner.rs +++ b/cranelift/src/filetest/runner.rs @@ -81,7 +81,7 @@ impl TestRunner { /// Create a new blank TrstRunner. pub fn new(verbose: bool) -> TestRunner { TestRunner { - verbose: verbose, + verbose, dir_stack: Vec::new(), tests: Vec::new(), new_tests: 0, diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index 60231d22f4..621f2bfd6a 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -55,9 +55,9 @@ pub fn run(path: &Path) -> TestResult { for (func, details) in testfile.functions { let mut context = Context { preamble_comments: &testfile.preamble_comments, - details: details, + details, verified: false, - flags: flags, + flags, isa: None, }; diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index d5b87d806b..1d5deff68c 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -32,7 +32,7 @@ struct CFGPrinter<'a> { impl<'a> CFGPrinter<'a> { pub fn new(func: &'a Function) -> CFGPrinter<'a> { CFGPrinter { - func: func, + func, cfg: ControlFlowGraph::with_function(func), } } diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 706ad46780..bbe1df7591 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -451,7 +451,7 @@ def emit_recipe_names(isa, fmt): This is used for pretty-printing encodings. """ with fmt.indented( - 'static RECIPE_NAMES: [&\'static str; {}] = [' + 'static RECIPE_NAMES: [&str; {}] = [' .format(len(isa.all_recipes)), '];'): for r in isa.all_recipes: fmt.line('"{}",'.format(r.name)) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index b1b32b7bf3..f3a4343ab2 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -470,7 +470,7 @@ def gen_format_constructor(iform, fmt): # Generate the instruction data. with fmt.indented( 'let data = InstructionData::{} {{'.format(iform.name), '};'): - fmt.line('opcode: opcode,') + fmt.line('opcode,') gen_member_inits(iform, fmt) fmt.line('self.build(data, ctrl_typevar)') @@ -489,7 +489,7 @@ def gen_member_inits(iform, fmt): # Value operands. if iform.has_value_list: - fmt.line('args: args,') + fmt.line('args,') elif iform.num_value_operands == 1: fmt.line('arg: arg0,') elif iform.num_value_operands > 1: diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index 847a72399d..23e9a5c7a9 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -132,7 +132,7 @@ def gen_descriptors(sgrp, fmt): raise AssertionError("Unknown setting kind") with fmt.indented( - 'static ENUMERATORS: [&\'static str; {}] = [' + 'static ENUMERATORS: [&str; {}] = [' .format(len(enums.table)), '];'): for txt in enums.table: diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index d9ef0faa25..ae71f754c7 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -56,7 +56,7 @@ impl<'c, 'fc, 'fd> InsertBuilder<'c, 'fc, 'fd> { pub fn new(dfg: &'fd mut DataFlowGraph, pos: &'c mut Cursor<'fc>) -> InsertBuilder<'c, 'fc, 'fd> { - InsertBuilder { dfg: dfg, pos: pos } + InsertBuilder { dfg, pos } } /// Reuse result values in `reuse`. @@ -72,7 +72,7 @@ impl<'c, 'fc, 'fd> InsertBuilder<'c, 'fc, 'fd> { InsertReuseBuilder { dfg: self.dfg, pos: self.pos, - reuse: reuse, + reuse, } } @@ -154,10 +154,7 @@ pub struct ReplaceBuilder<'f> { impl<'f> ReplaceBuilder<'f> { /// Create a `ReplaceBuilder` that will overwrite `inst`. pub fn new(dfg: &'f mut DataFlowGraph, inst: Inst) -> ReplaceBuilder { - ReplaceBuilder { - dfg: dfg, - inst: inst, - } + ReplaceBuilder { dfg, inst } } } diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 9f1a8b5299..063d7cbbd7 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -240,10 +240,7 @@ impl DataFlowGraph { self.value_type(dest), ty); - self.values[dest] = ValueData::Alias { - ty: ty, - original: original, - }; + self.values[dest] = ValueData::Alias { ty, original }; } /// Create a new value alias. @@ -252,10 +249,7 @@ impl DataFlowGraph { pub fn make_value_alias(&mut self, src: Value) -> Value { let ty = self.value_type(src); - let data = ValueData::Alias { - ty: ty, - original: src, - }; + let data = ValueData::Alias { ty, original: src }; self.make_value(data) } } @@ -462,9 +456,9 @@ impl DataFlowGraph { assert!(num <= u16::MAX as usize, "Too many result values"); let ty = self.value_type(res); self.values[res] = ValueData::Inst { - ty: ty, + ty, num: num as u16, - inst: inst, + inst, }; } @@ -474,8 +468,8 @@ impl DataFlowGraph { let num = self.results[inst].push(res, &mut self.value_lists); assert!(num <= u16::MAX as usize, "Too many result values"); self.make_value(ValueData::Inst { - ty: ty, - inst: inst, + ty, + inst, num: num as u16, }) } @@ -594,9 +588,9 @@ impl DataFlowGraph { let num = self.ebbs[ebb].args.push(arg, &mut self.value_lists); assert!(num <= u16::MAX as usize, "Too many arguments to EBB"); self.make_value(ValueData::Arg { - ty: ty, + ty, num: num as u16, - ebb: ebb, + ebb, }) } @@ -611,9 +605,9 @@ impl DataFlowGraph { assert!(num <= u16::MAX as usize, "Too many arguments to EBB"); let ty = self.value_type(arg); self.values[arg] = ValueData::Arg { - ty: ty, + ty, num: num as u16, - ebb: ebb, + ebb, }; } @@ -635,8 +629,8 @@ impl DataFlowGraph { }; let new_arg = self.make_value(ValueData::Arg { ty: new_type, - num: num, - ebb: ebb, + num, + ebb, }); self.ebbs[ebb].args.as_mut_slice(&mut self.value_lists)[num as usize] = new_arg; diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 0971015f62..e0863f8d1d 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -138,7 +138,7 @@ impl ArgumentType { ArgumentType { value_type: vt, extension: ArgumentExtension::None, - purpose: purpose, + purpose, location: ArgumentLoc::Reg(regunit), } } @@ -239,7 +239,7 @@ pub enum ArgumentPurpose { } /// Text format names of the `ArgumentPurpose` variants. -static PURPOSE_NAMES: [&'static str; 5] = ["normal", "sret", "link", "fp", "csr"]; +static PURPOSE_NAMES: [&str; 5] = ["normal", "sret", "link", "fp", "csr"]; impl fmt::Display for ArgumentPurpose { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 177ca0b5f1..075ee7c6fd 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -57,7 +57,7 @@ impl Function { /// Create a function with the given name and signature. pub fn with_name_signature(name: FunctionName, sig: Signature) -> Function { Function { - name: name, + name, signature: sig, stack_slots: EntityMap::new(), jump_tables: EntityMap::new(), diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 6e3e793125..b8ef83e75b 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -647,7 +647,7 @@ impl<'f> Cursor<'f> { /// The cursor holds a mutable reference to `layout` for its entire lifetime. pub fn new(layout: &'f mut Layout) -> Cursor { Cursor { - layout: layout, + layout, pos: CursorPosition::Nowhere, } } diff --git a/lib/cretonne/src/ir/memflags.rs b/lib/cretonne/src/ir/memflags.rs index 8cef887da2..bd2f2bab19 100644 --- a/lib/cretonne/src/ir/memflags.rs +++ b/lib/cretonne/src/ir/memflags.rs @@ -7,7 +7,7 @@ enum FlagBit { Aligned, } -const NAMES: [&'static str; 2] = ["notrap", "aligned"]; +const NAMES: [&str; 2] = ["notrap", "aligned"]; /// Flags for memory operations like load/store. /// diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index ec9de51917..e3128fdc0e 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -39,7 +39,7 @@ fn isa_constructor(shared_flags: shared_settings::Flags, }; Box::new(Isa { isa_flags: settings::Flags::new(&shared_flags, builder), - shared_flags: shared_flags, + shared_flags, cpumode: level1, }) } diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 4067142f57..f6fd70fb7a 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -33,7 +33,7 @@ fn isa_constructor(shared_flags: shared_settings::Flags, -> Box { Box::new(Isa { isa_flags: settings::Flags::new(&shared_flags, builder), - shared_flags: shared_flags, + shared_flags, }) } diff --git a/lib/cretonne/src/isa/encoding.rs b/lib/cretonne/src/isa/encoding.rs index 3d866727a4..93888b109e 100644 --- a/lib/cretonne/src/isa/encoding.rs +++ b/lib/cretonne/src/isa/encoding.rs @@ -19,10 +19,7 @@ pub struct Encoding { impl Encoding { /// Create a new `Encoding` containing `(recipe, bits)`. pub fn new(recipe: u16, bits: u16) -> Encoding { - Encoding { - recipe: recipe, - bits: bits, - } + Encoding { recipe, bits } } /// Get the recipe number in this encoding. diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 1812a41354..5871c8766a 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -39,7 +39,7 @@ fn isa_constructor(shared_flags: shared_settings::Flags, }; Box::new(Isa { isa_flags: settings::Flags::new(&shared_flags, builder), - shared_flags: shared_flags, + shared_flags, cpumode: level1, }) } diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 3c4f1b411a..15a089db17 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -220,7 +220,7 @@ impl RegInfo { /// Make a temporary object that can display a register unit. pub fn display_regunit(&self, regunit: RegUnit) -> DisplayRegUnit { DisplayRegUnit { - regunit: regunit, + regunit, reginfo: self, } } diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index cf5aafc7db..ad64e4fd29 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -39,7 +39,7 @@ fn isa_constructor(shared_flags: shared_settings::Flags, }; Box::new(Isa { isa_flags: settings::Flags::new(&shared_flags, builder), - shared_flags: shared_flags, + shared_flags, cpumode: level1, }) } diff --git a/lib/cretonne/src/iterators.rs b/lib/cretonne/src/iterators.rs index 3bdc3cc1cb..bca8cea2e4 100644 --- a/lib/cretonne/src/iterators.rs +++ b/lib/cretonne/src/iterators.rs @@ -8,10 +8,7 @@ pub trait IteratorExtras: Iterator { Self::Item: Clone { let elem = self.next(); - AdjacentPairs { - iter: self, - elem: elem, - } + AdjacentPairs { iter: self, elem } } } diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index 764448b5bc..cbca25c9c2 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -261,11 +261,11 @@ fn add_repair(concat: Opcode, hi_num: usize, repairs: &mut Vec) { repairs.push(Repair { - concat: concat, - split_type: split_type, - ebb: ebb, - num: num, - hi_num: hi_num, + concat, + split_type, + ebb, + num, + hi_num, }); } diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 0a5e9e803a..544c5b5da0 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -99,8 +99,8 @@ impl Coloring { let mut ctx = Context { reginfo: isa.register_info(), encinfo: isa.encoding_info(), - domtree: domtree, - liveness: liveness, + domtree, + liveness, usable_regs: isa.allocatable_registers(func), }; ctx.run(self, func, tracker) diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index c9f5146245..78085564f7 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -70,9 +70,9 @@ impl LiveValueVec { fn push(&mut self, value: Value, endpoint: Inst, affinity: Affinity) { self.values .push(LiveValue { - value: value, - endpoint: endpoint, - affinity: affinity, + value, + endpoint, + affinity, }); } diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 0594af19ed..a854cfc9ea 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -208,8 +208,8 @@ impl LiveRange { /// The live range will be created as dead, but it can be extended with `extend_in_ebb()`. pub fn new(value: Value, def: ProgramPoint, affinity: Affinity) -> LiveRange { LiveRange { - value: value, - affinity: affinity, + value, + affinity, def_begin: def, def_end: def, liveins: Vec::new(), diff --git a/lib/cretonne/src/verifier/liveness.rs b/lib/cretonne/src/verifier/liveness.rs index d3d841b68e..51911b3f38 100644 --- a/lib/cretonne/src/verifier/liveness.rs +++ b/lib/cretonne/src/verifier/liveness.rs @@ -27,10 +27,10 @@ pub fn verify_liveness(isa: &TargetIsa, liveness: &Liveness) -> Result { let verifier = LivenessVerifier { - isa: isa, - func: func, - cfg: cfg, - liveness: liveness, + isa, + func, + cfg, + liveness, }; verifier.check_ebbs()?; verifier.check_insts()?; diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index e3d5607c99..82e95d7f1f 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -139,10 +139,10 @@ impl<'a> Verifier<'a> { let cfg = ControlFlowGraph::with_function(func); let domtree = DominatorTree::with_function(func, &cfg); Verifier { - func: func, - cfg: cfg, - domtree: domtree, - isa: isa, + func, + cfg, + domtree, + isa, } } diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs index 031a50d9e3..e70768e32c 100644 --- a/lib/filecheck/src/checker.rs +++ b/lib/filecheck/src/checker.rs @@ -25,7 +25,7 @@ enum Directive { // 1. Keyword. // 2. Rest of line / pattern. // -const DIRECTIVE_RX: &'static str = r"\b(check|sameln|nextln|unordered|not|regex):\s+(.*)"; +const DIRECTIVE_RX: &str = r"\b(check|sameln|nextln|unordered|not|regex):\s+(.*)"; impl Directive { /// Create a new directive from a `DIRECTIVE_RX` match. @@ -264,9 +264,9 @@ struct State<'a> { impl<'a> State<'a> { fn new(text: &'a str, env_vars: &'a VariableMap, recorder: &'a mut Recorder) -> State<'a> { State { - text: text, - env_vars: env_vars, - recorder: recorder, + text, + env_vars, + recorder, vars: HashMap::new(), last_ordered: 0, max_match: 0, diff --git a/lib/filecheck/src/explain.rs b/lib/filecheck/src/explain.rs index 49780321b5..f268620d11 100644 --- a/lib/filecheck/src/explain.rs +++ b/lib/filecheck/src/explain.rs @@ -60,7 +60,7 @@ pub struct Explainer<'a> { impl<'a> Explainer<'a> { pub fn new(text: &'a str) -> Explainer { Explainer { - text: text, + text, directive: 0, matches: Vec::new(), vardefs: Vec::new(), diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 14a0fe45a4..1262b63888 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -55,7 +55,7 @@ pub struct LocatedToken<'a> { /// Wrap up a `Token` with the given location. fn token<'a>(token: Token<'a>, loc: Location) -> Result, LocatedError> { Ok(LocatedToken { - token: token, + token, location: loc, }) } @@ -76,7 +76,7 @@ pub struct LocatedError { /// Wrap up an `Error` with the given location. fn error<'a>(error: Error, loc: Location) -> Result, LocatedError> { Err(LocatedError { - error: error, + error, location: loc, }) } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index eae8c925b0..02ec88196e 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -51,10 +51,10 @@ pub fn parse_test<'a>(text: &'a str) -> Result> { let functions = parser.parse_function_list(isa_spec.unique_isa())?; Ok(TestFile { - commands: commands, - isa_spec: isa_spec, - preamble_comments: preamble_comments, - functions: functions, + commands, + isa_spec, + preamble_comments, + functions, }) } @@ -100,7 +100,7 @@ impl<'a> Context<'a> { function: f, map: SourceMap::new(), aliases: HashMap::new(), - unique_isa: unique_isa, + unique_isa, } } @@ -272,11 +272,7 @@ impl<'a> Parser<'a> { Token::Comment(text) => { // Gather comments, associate them with `comment_entity`. if let Some(entity) = self.comment_entity { - self.comments - .push(Comment { - entity: entity, - text: text, - }); + self.comments.push(Comment { entity, text }); } } _ => self.lookahead = Some(token), @@ -676,7 +672,7 @@ impl<'a> Parser<'a> { ctx.rewrite_references()?; let details = Details { - location: location, + location, comments: self.take_comments(), map: ctx.map, }; @@ -924,7 +920,7 @@ impl<'a> Parser<'a> { .def_entity(sigref.into(), &loc) .expect("duplicate SigRef entities created"); ExtFuncData { - name: name, + name, signature: sigref, } } @@ -933,7 +929,7 @@ impl<'a> Parser<'a> { self.consume(); let name = self.parse_function_name()?; ExtFuncData { - name: name, + name, signature: sig, } } @@ -1416,28 +1412,28 @@ impl<'a> Parser<'a> { opcode: Opcode) -> Result { let idata = match opcode.format() { - InstructionFormat::Nullary => InstructionData::Nullary { opcode: opcode }, + InstructionFormat::Nullary => InstructionData::Nullary { opcode }, InstructionFormat::Unary => { InstructionData::Unary { - opcode: opcode, + opcode, arg: self.match_value("expected SSA value operand")?, } } InstructionFormat::UnaryImm => { InstructionData::UnaryImm { - opcode: opcode, + opcode, imm: self.match_imm64("expected immediate integer operand")?, } } InstructionFormat::UnaryIeee32 => { InstructionData::UnaryIeee32 { - opcode: opcode, + opcode, imm: self.match_ieee32("expected immediate 32-bit float operand")?, } } InstructionFormat::UnaryIeee64 => { InstructionData::UnaryIeee64 { - opcode: opcode, + opcode, imm: self.match_ieee64("expected immediate 64-bit float operand")?, } } @@ -1446,7 +1442,7 @@ impl<'a> Parser<'a> { self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_value("expected SSA value second operand")?; InstructionData::Binary { - opcode: opcode, + opcode, args: [lhs, rhs], } } @@ -1455,7 +1451,7 @@ impl<'a> Parser<'a> { self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_imm64("expected immediate integer second operand")?; InstructionData::BinaryImm { - opcode: opcode, + opcode, arg: lhs, imm: rhs, } @@ -1469,14 +1465,14 @@ impl<'a> Parser<'a> { self.match_token(Token::Comma, "expected ',' between operands")?; let false_arg = self.match_value("expected SSA value false operand")?; InstructionData::Ternary { - opcode: opcode, + opcode, args: [ctrl_arg, true_arg, false_arg], } } InstructionFormat::MultiAry => { let args = self.parse_value_list()?; InstructionData::MultiAry { - opcode: opcode, + opcode, args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } } @@ -1485,7 +1481,7 @@ impl<'a> Parser<'a> { let ebb_num = self.match_ebb("expected jump destination EBB")?; let args = self.parse_opt_value_list()?; InstructionData::Jump { - opcode: opcode, + opcode, destination: ebb_num, args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } @@ -1496,7 +1492,7 @@ impl<'a> Parser<'a> { let ebb_num = self.match_ebb("expected branch destination EBB")?; let args = self.parse_opt_value_list()?; InstructionData::Branch { - opcode: opcode, + opcode, destination: ebb_num, args: args.into_value_list(&[ctrl_arg], &mut ctx.function.dfg.value_lists), } @@ -1510,8 +1506,8 @@ impl<'a> Parser<'a> { let ebb_num = self.match_ebb("expected branch destination EBB")?; let args = self.parse_opt_value_list()?; InstructionData::BranchIcmp { - opcode: opcode, - cond: cond, + opcode, + cond, destination: ebb_num, args: args.into_value_list(&[lhs, rhs], &mut ctx.function.dfg.value_lists), } @@ -1523,8 +1519,8 @@ impl<'a> Parser<'a> { self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_value("expected SSA value last operand")?; InstructionData::InsertLane { - opcode: opcode, - lane: lane, + opcode, + lane, args: [lhs, rhs], } } @@ -1532,11 +1528,7 @@ impl<'a> Parser<'a> { let arg = self.match_value("expected SSA value last operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; let lane = self.match_uimm8("expected lane number")?; - InstructionData::ExtractLane { - opcode: opcode, - lane: lane, - arg: arg, - } + InstructionData::ExtractLane { opcode, lane, arg } } InstructionFormat::IntCompare => { let cond = self.match_enum("expected intcc condition code")?; @@ -1544,8 +1536,8 @@ impl<'a> Parser<'a> { self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_value("expected SSA value second operand")?; InstructionData::IntCompare { - opcode: opcode, - cond: cond, + opcode, + cond, args: [lhs, rhs], } } @@ -1555,8 +1547,8 @@ impl<'a> Parser<'a> { self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_imm64("expected immediate second operand")?; InstructionData::IntCompareImm { - opcode: opcode, - cond: cond, + opcode, + cond, arg: lhs, imm: rhs, } @@ -1567,8 +1559,8 @@ impl<'a> Parser<'a> { self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_value("expected SSA value second operand")?; InstructionData::FloatCompare { - opcode: opcode, - cond: cond, + opcode, + cond, args: [lhs, rhs], } } @@ -1579,8 +1571,8 @@ impl<'a> Parser<'a> { let args = self.parse_value_list()?; self.match_token(Token::RPar, "expected ')' after arguments")?; InstructionData::Call { - opcode: opcode, - func_ref: func_ref, + opcode, + func_ref, args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } } @@ -1593,8 +1585,8 @@ impl<'a> Parser<'a> { let args = self.parse_value_list()?; self.match_token(Token::RPar, "expected ')' after arguments")?; InstructionData::IndirectCall { - opcode: opcode, - sig_ref: sig_ref, + opcode, + sig_ref, args: args.into_value_list(&[callee], &mut ctx.function.dfg.value_lists), } } @@ -1603,20 +1595,16 @@ impl<'a> Parser<'a> { self.match_token(Token::Comma, "expected ',' between operands")?; let table = self.match_jt() .and_then(|num| ctx.get_jt(num, &self.loc))?; - InstructionData::BranchTable { - opcode: opcode, - arg: arg, - table: table, - } + InstructionData::BranchTable { opcode, arg, table } } InstructionFormat::StackLoad => { let ss = self.match_ss("expected stack slot number: ss«n»") .and_then(|num| ctx.get_ss(num, &self.loc))?; let offset = self.optional_offset32()?; InstructionData::StackLoad { - opcode: opcode, + opcode, stack_slot: ss, - offset: offset, + offset, } } InstructionFormat::StackStore => { @@ -1626,19 +1614,19 @@ impl<'a> Parser<'a> { .and_then(|num| ctx.get_ss(num, &self.loc))?; let offset = self.optional_offset32()?; InstructionData::StackStore { - opcode: opcode, - arg: arg, + opcode, + arg, stack_slot: ss, - offset: offset, + offset, } } InstructionFormat::HeapLoad => { let addr = self.match_value("expected SSA value address")?; let offset = self.optional_uoffset32()?; InstructionData::HeapLoad { - opcode: opcode, + opcode, arg: addr, - offset: offset, + offset, } } InstructionFormat::HeapStore => { @@ -1647,9 +1635,9 @@ impl<'a> Parser<'a> { let addr = self.match_value("expected SSA value address")?; let offset = self.optional_uoffset32()?; InstructionData::HeapStore { - opcode: opcode, + opcode, args: [arg, addr], - offset: offset, + offset, } } InstructionFormat::Load => { @@ -1657,10 +1645,10 @@ impl<'a> Parser<'a> { let addr = self.match_value("expected SSA value address")?; let offset = self.optional_offset32()?; InstructionData::Load { - opcode: opcode, - flags: flags, + opcode, + flags, arg: addr, - offset: offset, + offset, } } InstructionFormat::Store => { @@ -1670,10 +1658,10 @@ impl<'a> Parser<'a> { let addr = self.match_value("expected SSA value address")?; let offset = self.optional_offset32()?; InstructionData::Store { - opcode: opcode, - flags: flags, + opcode, + flags, args: [arg, addr], - offset: offset, + offset, } } }; From 43304e9abcca72364192c1c3f50a8a7cce207017 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 27 Apr 2017 12:52:41 -0700 Subject: [PATCH 0733/3084] Upgrade to rustfmt 0.8.3. --- cranelift/src/cat.rs | 6 ++++-- cranelift/src/filetest/legalizer.rs | 6 ++++-- cranelift/src/filetest/regalloc.rs | 9 ++++++--- cranelift/src/filetest/runone.rs | 6 ++++-- cranelift/src/filetest/subtest.rs | 12 ++++++++---- cranelift/src/print_cfg.rs | 6 ++++-- cranelift/src/rsfilecheck.rs | 18 ++++++++++++------ cranelift/test-all.sh | 2 +- 8 files changed, 43 insertions(+), 22 deletions(-) diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs index fd49722ec0..a53d3b4294 100644 --- a/cranelift/src/cat.rs +++ b/cranelift/src/cat.rs @@ -21,8 +21,10 @@ pub fn run(files: Vec) -> CommandResult { } fn cat_one(filename: String) -> CommandResult { - let buffer = read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))?; - let items = parse_functions(&buffer).map_err(|e| format!("{}: {}", filename, e))?; + let buffer = read_to_string(&filename) + .map_err(|e| format!("{}: {}", filename, e))?; + let items = parse_functions(&buffer) + .map_err(|e| format!("{}: {}", filename, e))?; for (idx, func) in items.into_iter().enumerate() { if idx != 0 { diff --git a/cranelift/src/filetest/legalizer.rs b/cranelift/src/filetest/legalizer.rs index c6e08ffe29..8fd7f45b16 100644 --- a/cranelift/src/filetest/legalizer.rs +++ b/cranelift/src/filetest/legalizer.rs @@ -40,11 +40,13 @@ impl SubTest for TestLegalizer { let isa = context.isa.expect("legalizer needs an ISA"); comp_ctx.flowgraph(); - comp_ctx.legalize(isa) + comp_ctx + .legalize(isa) .map_err(|e| pretty_error(&comp_ctx.func, e))?; let mut text = String::new(); - write_function(&mut text, &comp_ctx.func, Some(isa)).map_err(|e| e.to_string())?; + write_function(&mut text, &comp_ctx.func, Some(isa)) + .map_err(|e| e.to_string())?; run_filecheck(&text, context) } } diff --git a/cranelift/src/filetest/regalloc.rs b/cranelift/src/filetest/regalloc.rs index 87045eb764..79e7ad8e0f 100644 --- a/cranelift/src/filetest/regalloc.rs +++ b/cranelift/src/filetest/regalloc.rs @@ -45,13 +45,16 @@ impl SubTest for TestRegalloc { comp_ctx.flowgraph(); // TODO: Should we have an option to skip legalization? - comp_ctx.legalize(isa) + comp_ctx + .legalize(isa) .map_err(|e| pretty_error(&comp_ctx.func, e))?; - comp_ctx.regalloc(isa) + comp_ctx + .regalloc(isa) .map_err(|e| pretty_error(&comp_ctx.func, e))?; let mut text = String::new(); - write_function(&mut text, &comp_ctx.func, Some(isa)).map_err(|e| e.to_string())?; + write_function(&mut text, &comp_ctx.func, Some(isa)) + .map_err(|e| e.to_string())?; run_filecheck(&text, context) } } diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index 621f2bfd6a..b43d926f0e 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -26,7 +26,8 @@ pub fn run(path: &Path) -> TestResult { } // Parse the test commands. - let mut tests = testfile.commands + let mut tests = testfile + .commands .iter() .map(new_subtest) .collect::>>()?; @@ -116,7 +117,8 @@ fn run_one_test<'a>(tuple: (&'a SubTest, &'a Flags, Option<&'a TargetIsa>), // Should we run the verifier before this test? if !context.verified && test.needs_verifier() { - verify_function(&func, isa).map_err(|e| pretty_verifier_error(&func, e))?; + verify_function(&func, isa) + .map_err(|e| pretty_verifier_error(&func, e))?; context.verified = true; } diff --git a/cranelift/src/filetest/subtest.rs b/cranelift/src/filetest/subtest.rs index 923439d490..9c7159dbb5 100644 --- a/cranelift/src/filetest/subtest.rs +++ b/cranelift/src/filetest/subtest.rs @@ -76,12 +76,14 @@ impl<'a> filecheck::VariableMap for Context<'a> { /// Run filecheck on `text`, using directives extracted from `context`. pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { let checker = build_filechecker(context)?; - if checker.check(&text, context) + if checker + .check(&text, context) .map_err(|e| format!("filecheck: {}", e))? { Ok(()) } else { // Filecheck mismatch. Emit an explanation as output. - let (_, explain) = checker.explain(&text, context) + let (_, explain) = checker + .explain(&text, context) .map_err(|e| format!("explain: {}", e))?; Err(format!("filecheck failed:\n{}{}", checker, explain)) } @@ -92,11 +94,13 @@ pub fn build_filechecker(context: &Context) -> Result { let mut builder = CheckerBuilder::new(); // Preamble comments apply to all functions. for comment in context.preamble_comments { - builder.directive(comment.text) + builder + .directive(comment.text) .map_err(|e| format!("filecheck: {}", e))?; } for comment in &context.details.comments { - builder.directive(comment.text) + builder + .directive(comment.text) .map_err(|e| format!("filecheck: {}", e))?; } let checker = builder.finish(); diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index 1d5deff68c..ec479e3c92 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -91,8 +91,10 @@ impl<'a> Display for CFGPrinter<'a> { } fn print_cfg(filename: String) -> CommandResult { - let buffer = read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))?; - let items = parse_functions(&buffer).map_err(|e| format!("{}: {}", filename, e))?; + let buffer = read_to_string(&filename) + .map_err(|e| format!("{}: {}", filename, e))?; + let items = parse_functions(&buffer) + .map_err(|e| format!("{}: {}", filename, e))?; for (idx, func) in items.into_iter().enumerate() { if idx != 0 { diff --git a/cranelift/src/rsfilecheck.rs b/cranelift/src/rsfilecheck.rs index 79d913bce9..5ee464d20a 100644 --- a/cranelift/src/rsfilecheck.rs +++ b/cranelift/src/rsfilecheck.rs @@ -18,11 +18,13 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { } let mut buffer = String::new(); - io::stdin().read_to_string(&mut buffer) + io::stdin() + .read_to_string(&mut buffer) .map_err(|e| format!("stdin: {}", e))?; if verbose { - let (success, explain) = checker.explain(&buffer, NO_VARIABLES) + let (success, explain) = checker + .explain(&buffer, NO_VARIABLES) .map_err(|e| e.to_string())?; print!("{}", explain); if success { @@ -31,11 +33,13 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { } else { Err("Check failed".to_string()) } - } else if checker.check(&buffer, NO_VARIABLES) + } else if checker + .check(&buffer, NO_VARIABLES) .map_err(|e| e.to_string())? { Ok(()) } else { - let (_, explain) = checker.explain(&buffer, NO_VARIABLES) + let (_, explain) = checker + .explain(&buffer, NO_VARIABLES) .map_err(|e| e.to_string())?; print!("{}", explain); Err("Check failed".to_string()) @@ -43,9 +47,11 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { } fn read_checkfile(filename: &str) -> Result { - let buffer = read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))?; + let buffer = read_to_string(&filename) + .map_err(|e| format!("{}: {}", filename, e))?; let mut builder = CheckerBuilder::new(); - builder.text(&buffer) + builder + .text(&buffer) .map_err(|e| format!("{}: {}", filename, e))?; Ok(builder.finish()) } diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 5b718d8bc4..d387cae46d 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -30,7 +30,7 @@ function banner() { # rustfmt is installed. # # This version should always be bumped to the newest version available. -RUSTFMT_VERSION="0.8.1" +RUSTFMT_VERSION="0.8.3" if cargo install --list | grep -q "^rustfmt v$RUSTFMT_VERSION"; then banner "Rust formatting" From 40488c8e22b3ae214586a6138a34a481d8a02c2a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 27 Apr 2017 13:39:05 -0700 Subject: [PATCH 0734/3084] Install rustfmt as a separate Travis install step. - Add a check-rustfmt.sh script which checks if the right version of rustfmt is installed. - Run check-rustfmt.sh --install as an install step under travis_wait. This is to work around the issue where cargo takes forever to build rustfmt, causing Travis to terminate the build because it hasn't produced any output for 10 minutes. --- .travis.yml | 1 + check-rustfmt.sh | 35 +++++++++++++++++++++++++++++++++++ cranelift/test-all.sh | 21 +-------------------- 3 files changed, 37 insertions(+), 20 deletions(-) create mode 100755 check-rustfmt.sh diff --git a/.travis.yml b/.travis.yml index 1986553bf1..03dd6f76e6 100644 --- a/.travis.yml +++ b/.travis.yml @@ -10,6 +10,7 @@ addons: - python3-pip install: - pip3 install --user --upgrade mypy flake8 + - travis_wait ./check-rustfmt.sh --install script: ./test-all.sh cache: cargo: true diff --git a/check-rustfmt.sh b/check-rustfmt.sh new file mode 100755 index 0000000000..c0c99c9a91 --- /dev/null +++ b/check-rustfmt.sh @@ -0,0 +1,35 @@ +#!/bin/bash +# +# Usage: check-rustfmt.sh [--install] +# +# Check that the desired version of rustfmt is installed. +# +# Rustfmt is still immature enough that its formatting decisions can change +# between versions. This makes it difficult to enforce a certain style in a +# test script since not all developers will upgrade rustfmt at the same time. +# To work around this, we only verify formatting when a specific version of +# rustfmt is installed. +# +# Exits 0 if the right version of rustfmt is installed, 1 otherwise. +# +# With the --install option, also tries to install the right version. + +# This version should always be bumped to the newest version available. +VERS="0.8.3" + +if cargo install --list | grep -q "^rustfmt v$VERS"; then + exit 0 +fi + +if [ "$1" != "--install" ]; then + echo "********************************************************************" + echo "* Please install rustfmt v$VERS to verify formatting. *" + echo "* If a newer version of rustfmt is available, update this script. *" + echo "********************************************************************" + echo "$0 --install" + sleep 1 + exit 1 +fi + +echo "Installing rustfmt v$VERS." +cargo install --force --vers="$VERS" rustfmt diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index d387cae46d..07d6fa2bdd 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -22,28 +22,9 @@ function banner() { } # Run rustfmt if we have it. -# -# Rustfmt is still immature enough that its formatting decisions can change -# between versions. This makes it difficult to enforce a certain style in a -# test script since not all developers will upgrade rustfmt at the same time. -# To work around this, we only verify formatting when a specific version of -# rustfmt is installed. -# -# This version should always be bumped to the newest version available. -RUSTFMT_VERSION="0.8.3" - -if cargo install --list | grep -q "^rustfmt v$RUSTFMT_VERSION"; then +if $topdir/check-rustfmt.sh; then banner "Rust formatting" $topdir/format-all.sh --write-mode=diff -elif [ -n "$TRAVIS" ]; then - # We're running under Travis CI. - # Install rustfmt, it will be cached for the next build. - echo "Installing rustfmt v$RUSTFMT_VERSION." - cargo install --force --vers="$RUSTFMT_VERSION" rustfmt - $topdir/format-all.sh --write-mode=diff -else - echo "Please install rustfmt v$RUSTFMT_VERSION to verify formatting." - echo "If a newer version of rustfmt is available, update this script." fi # Check if any Python files have changed since we last checked them. From 71128611a711ad9820ff65540beedbad0fdeab47 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 27 Apr 2017 17:39:58 -0700 Subject: [PATCH 0735/3084] Extract the topological ordering into a module. Multiple passes will need to iterate over EBBs in a dominator-topological order. Move that functionality into a separate module. --- lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/regalloc/coloring.rs | 54 ++--------- lib/cretonne/src/regalloc/context.rs | 10 ++- lib/cretonne/src/topo_order.rs | 125 ++++++++++++++++++++++++++ 4 files changed, 144 insertions(+), 46 deletions(-) create mode 100644 lib/cretonne/src/topo_order.rs diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 986c45355b..1f51b38d3f 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -35,4 +35,5 @@ mod packed_option; mod partition_slice; mod predicates; mod ref_slice; +mod topo_order; mod write; diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 544c5b5da0..a1b3899577 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -43,19 +43,13 @@ use regalloc::affinity::Affinity; use regalloc::allocatable_set::AllocatableSet; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; -use sparse_map::SparseSet; +use topo_order::TopoOrder; /// Data structures for the coloring pass. /// /// These are scratch space data structures that can be reused between invocations. -pub struct Coloring { - /// Set of visited EBBs. - visited: SparseSet, - - /// Stack of EBBs to be visited next. - stack: Vec, -} +pub struct Coloring {} /// Bundle of references that the coloring algorithm needs. /// @@ -83,10 +77,7 @@ struct Context<'a> { impl Coloring { /// Allocate scratch space data structures for the coloring pass. pub fn new() -> Coloring { - Coloring { - visited: SparseSet::new(), - stack: Vec::new(), - } + Coloring {} } /// Run the coloring algorithm over `func`. @@ -95,6 +86,7 @@ impl Coloring { func: &mut Function, domtree: &DominatorTree, liveness: &mut Liveness, + topo: &mut TopoOrder, tracker: &mut LiveValueTracker) { let mut ctx = Context { reginfo: isa.register_info(), @@ -103,45 +95,17 @@ impl Coloring { liveness, usable_regs: isa.allocatable_registers(func), }; - ctx.run(self, func, tracker) + ctx.run(func, topo, tracker) } } impl<'a> Context<'a> { /// Run the coloring algorithm. - fn run(&mut self, data: &mut Coloring, func: &mut Function, tracker: &mut LiveValueTracker) { - // Just visit blocks in layout order, letting `process_ebb` enforce a topological ordering. + fn run(&mut self, func: &mut Function, topo: &mut TopoOrder, tracker: &mut LiveValueTracker) { + // Just visit blocks in layout order, letting `topo` enforce a topological ordering. // TODO: Once we have a loop tree, we could visit hot blocks first. - let mut next = func.layout.entry_block(); - while let Some(ebb) = next { - self.process_ebb(ebb, data, func, tracker); - next = func.layout.next_ebb(ebb); - } - } - - /// Process `ebb`, but only after ensuring that the immediate dominator has been processed. - /// - /// This method can be called with the most desired order of visiting the EBBs. It will convert - /// that order into a valid topological order by visiting dominators first. - fn process_ebb(&mut self, - mut ebb: Ebb, - data: &mut Coloring, - func: &mut Function, - tracker: &mut LiveValueTracker) { - // The stack is just a scratch space for this algorithm. We leave it empty when returning. - assert!(data.stack.is_empty()); - - // Trace up the dominator tree until we reach a dominator that has already been visited. - while data.visited.insert(ebb).is_none() { - data.stack.push(ebb); - match self.domtree.idom(ebb) { - Some(idom) => ebb = func.layout.inst_ebb(idom).expect("idom not in layout"), - None => break, - } - } - - // Pop off blocks in topological order. - while let Some(ebb) = data.stack.pop() { + topo.reset(func.layout.ebbs()); + while let Some(ebb) = topo.next(&func.layout, self.domtree) { self.visit_ebb(ebb, func, tracker); } } diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index b57edebb5f..b8cebd1bcc 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -12,11 +12,13 @@ use regalloc::coloring::Coloring; use regalloc::live_value_tracker::LiveValueTracker; use regalloc::liveness::Liveness; use result::CtonResult; +use topo_order::TopoOrder; use verifier::{verify_context, verify_liveness}; /// Persistent memory allocations for register allocation. pub struct Context { liveness: Liveness, + topo: TopoOrder, tracker: LiveValueTracker, coloring: Coloring, } @@ -29,6 +31,7 @@ impl Context { pub fn new() -> Context { Context { liveness: Liveness::new(), + topo: TopoOrder::new(), tracker: LiveValueTracker::new(), coloring: Coloring::new(), } @@ -60,7 +63,12 @@ impl Context { // Third pass: Reload and coloring. self.coloring - .run(isa, func, domtree, &mut self.liveness, &mut self.tracker); + .run(isa, + func, + domtree, + &mut self.liveness, + &mut self.topo, + &mut self.tracker); if isa.flags().enable_verifier() { verify_context(func, cfg, domtree, Some(isa))?; diff --git a/lib/cretonne/src/topo_order.rs b/lib/cretonne/src/topo_order.rs new file mode 100644 index 0000000000..55cece2c2f --- /dev/null +++ b/lib/cretonne/src/topo_order.rs @@ -0,0 +1,125 @@ +//! Topological order of EBBs, according to the dominator tree. + +use dominator_tree::DominatorTree; +use ir::{Ebb, Layout}; +use sparse_map::SparseSet; + +/// Present EBBs in a topological order such that all dominating EBBs are guaranteed to be visited +/// before the current EBB. +/// +/// There are many topological orders of the EBBs in a function, so it is possible to provide a +/// preferred order, and the `TopoOrder` will present EBBs in an order that is as close as possible +/// to the preferred order. +pub struct TopoOrder { + /// Preferred order of EBBs to visit. + preferred: Vec, + + /// Next entry to get from `preferred`. + next: usize, + + /// Set of visited EBBs. + visited: SparseSet, + + /// Stack of EBBs to be visited next, already in `visited`. + stack: Vec, +} + +impl TopoOrder { + /// Create a new empty topological order. + pub fn new() -> TopoOrder { + TopoOrder { + preferred: Vec::new(), + next: 0, + visited: SparseSet::new(), + stack: Vec::new(), + } + } + + /// Reset and initialize with a preferred sequence of EBBs. The resulting topological order is + /// guaranteed to contain all of the EBBs in `preferred` as well as any dominators. + pub fn reset(&mut self, preferred: Ebbs) + where Ebbs: IntoIterator + { + self.preferred.clear(); + self.preferred.extend(preferred); + self.next = 0; + self.visited.clear(); + self.stack.clear(); + } + + /// Get the next EBB in the topological order. + /// + /// Two things are guaranteed about the EBBs returned by this function: + /// + /// - All EBBs in the `preferred` iterator given to `reset` will be returned. + /// - All dominators are visited before the EBB returned. + pub fn next(&mut self, layout: &Layout, domtree: &DominatorTree) -> Option { + // Any entries in `stack` should be returned immediately. They have already been added to + // `visited`. + while self.stack.is_empty() { + match self.preferred.get(self.next).cloned() { + None => return None, + Some(mut ebb) => { + // We have the next EBB in the preferred order. + self.next += 1; + // Push it along with any non-visited dominators. + while self.visited.insert(ebb).is_none() { + self.stack.push(ebb); + match domtree.idom(ebb) { + Some(idom) => ebb = layout.inst_ebb(idom).expect("idom not in layout"), + None => break, + } + } + } + } + } + return self.stack.pop(); + } +} + +#[cfg(test)] +mod test { + use flowgraph::ControlFlowGraph; + use dominator_tree::DominatorTree; + use ir::{Function, InstBuilder, Cursor}; + use std::iter; + use super::*; + + #[test] + fn empty() { + let func = Function::new(); + let cfg = ControlFlowGraph::with_function(&func); + let domtree = DominatorTree::with_function(&func, &cfg); + let mut topo = TopoOrder::new(); + + assert_eq!(topo.next(&func.layout, &domtree), None); + topo.reset(func.layout.ebbs()); + assert_eq!(topo.next(&func.layout, &domtree), None); + } + + #[test] + fn simple() { + let mut func = Function::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + + { + let dfg = &mut func.dfg; + let cur = &mut Cursor::new(&mut func.layout); + + cur.insert_ebb(ebb0); + dfg.ins(cur).jump(ebb1, &[]); + cur.insert_ebb(ebb1); + dfg.ins(cur).jump(ebb1, &[]); + } + + let cfg = ControlFlowGraph::with_function(&func); + let domtree = DominatorTree::with_function(&func, &cfg); + let mut topo = TopoOrder::new(); + + topo.reset(iter::once(ebb1)); + assert_eq!(topo.next(&func.layout, &domtree), Some(ebb0)); + assert_eq!(topo.next(&func.layout, &domtree), Some(ebb1)); + assert_eq!(topo.next(&func.layout, &domtree), None); + } +} From 950838c4895ef36b48241eae5ea55850316f94d9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 2 May 2017 11:32:12 -0700 Subject: [PATCH 0736/3084] Add a regmove instruction. This will be used to locally change the register locations of values in order to satisfy instruction constraints. --- cranelift/docs/langref.rst | 5 ++++ cranelift/filetests/parser/tiny.cton | 15 ++++++++++ lib/cretonne/meta/base/formats.py | 4 ++- lib/cretonne/meta/base/immediates.py | 6 ++++ lib/cretonne/meta/base/instructions.py | 19 ++++++++++++- lib/cretonne/src/ir/builder.rs | 1 + lib/cretonne/src/ir/dfg.rs | 2 +- lib/cretonne/src/ir/instructions.rs | 7 +++++ lib/cretonne/src/verifier/mod.rs | 3 +- lib/cretonne/src/write.rs | 21 ++++++++++++-- lib/reader/src/parser.rs | 38 +++++++++++++++++++++++++- 11 files changed, 114 insertions(+), 7 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 008f2a74b1..e6a0e2b3d5 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -583,6 +583,11 @@ allocation pass and beyond. .. autoinst:: spill .. autoinst:: fill +Register values can be temporarily diverted to other registers by the +:inst:`regmove` instruction. + +.. autoinst:: regmove + Vector operations ----------------- diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 87140d9aad..5ad2f7a39d 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -155,3 +155,18 @@ ebb0(v1: i32): ; nextln: store $v2, $v1 ; nextln: store aligned $v3, $v1+12 ; nextln: store notrap aligned $v3, $v1-12 + +; Register diversions. +; This test file has no ISA, so we can unly use register unit numbers. +function diversion(i32) { +ebb0(v1: i32): + regmove v1, %10 -> %20 + regmove v1, %20 -> %10 + return +} +; sameln: function diversion(i32) { +; nextln: ebb0($v1: i32): +; nextln: regmove $v1, %10 -> %20 +; nextln: regmove $v1, %20 -> %10 +; nextln: return +; nextln: } diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 6e64e0e3f2..368fb13078 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -9,7 +9,7 @@ from __future__ import absolute_import from cdsl.formats import InstructionFormat from cdsl.operands import VALUE, VARIABLE_ARGS from .immediates import imm64, uimm8, ieee32, ieee64, offset32, uoffset32 -from .immediates import intcc, floatcc, memflags +from .immediates import intcc, floatcc, memflags, regunit from .entities import ebb, sig_ref, func_ref, jump_table, stack_slot Nullary = InstructionFormat() @@ -57,5 +57,7 @@ StackStore = InstructionFormat(VALUE, stack_slot, offset32) HeapLoad = InstructionFormat(VALUE, uoffset32) HeapStore = InstructionFormat(VALUE, VALUE, uoffset32) +RegMove = InstructionFormat(VALUE, ('src', regunit), ('dst', regunit)) + # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index 8e643bb9ee..2458b76e70 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -96,3 +96,9 @@ memflags = ImmediateKind( 'memflags', 'Memory operation flags', default_member='flags', rust_type='MemFlags') + +#: A register unit in the current target ISA. +regunit = ImmediateKind( + 'regunit', + 'A register unit in the target ISA', + rust_type='RegUnit') diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 3035f2b98e..b9d2d5a5af 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -10,7 +10,7 @@ from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup from base.types import i8, f32, f64, b1 from base.immediates import imm64, uimm8, ieee32, ieee64, offset32, uoffset32 -from base.immediates import intcc, floatcc, memflags +from base.immediates import intcc, floatcc, memflags, regunit from base import entities import base.formats # noqa @@ -467,6 +467,23 @@ fill = Instruction( """, ins=x, outs=a) +src = Operand('src', regunit) +dst = Operand('dst', regunit) + +regmove = Instruction( + 'regmove', r""" + Temporarily divert ``x`` from ``src`` to ``dst``. + + This instruction moves the location of a value from one register to + another without creating a new SSA value. It is used by the register + allocator to temporarily rearrange register assignments in order to + satisfy instruction constraints. + + The register diversions created by this instruction must be undone + before the value leaves the EBB. At the entry to a new EBB, all live + values must be in their originally assigned registers. + """, + ins=(x, src, dst)) # # Vector operations diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index ae71f754c7..a5a3719299 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -9,6 +9,7 @@ use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, StackSlot, MemFlags}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32, Uoffset32}; use ir::condcodes::{IntCC, FloatCC}; +use isa::RegUnit; /// Base trait for instruction builders. /// diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 063d7cbbd7..6e33c9d592 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -687,7 +687,7 @@ impl<'a> fmt::Display for DisplayInst<'a> { } else { write!(f, "{}.{}", inst.opcode(), typevar)?; } - write_operands(f, dfg, self.1) + write_operands(f, dfg, None, self.1) } } diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 9484110d83..0b32929af6 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -14,6 +14,7 @@ use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef, StackSlot, MemFlags}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32, Uoffset32}; use ir::condcodes::*; use ir::types; +use isa::RegUnit; use entity_list; use ref_slice::{ref_slice, ref_slice_mut}; @@ -203,6 +204,12 @@ pub enum InstructionData { args: [Value; 2], offset: Offset32, }, + RegMove { + opcode: Opcode, + arg: Value, + src: RegUnit, + dst: RegUnit, + }, } /// A variable list of `Value` operands used for function call arguments and passing arguments to diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 82e95d7f1f..e1568d5bab 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -284,7 +284,8 @@ impl<'a> Verifier<'a> { &HeapLoad { .. } | &HeapStore { .. } | &Load { .. } | - &Store { .. } => {} + &Store { .. } | + &RegMove { .. } => {} } Ok(()) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 5d46ccb624..af66cc8868 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -225,12 +225,16 @@ fn write_instruction(w: &mut Write, None => write!(w, "{}", opcode)?, } - write_operands(w, &func.dfg, inst)?; + write_operands(w, &func.dfg, isa, inst)?; writeln!(w, "") } /// Write the operands of `inst` to `w` with a prepended space. -pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result { +pub fn write_operands(w: &mut Write, + dfg: &DataFlowGraph, + isa: Option<&TargetIsa>, + inst: Inst) + -> Result { let pool = &dfg.value_lists; use ir::instructions::InstructionData::*; match dfg[inst] { @@ -321,6 +325,19 @@ pub fn write_operands(w: &mut Write, dfg: &DataFlowGraph, inst: Inst) -> Result offset, .. } => write!(w, "{} {}, {}{}", flags, args[0], args[1], offset), + RegMove { arg, src, dst, .. } => { + if let Some(isa) = isa { + let regs = isa.register_info(); + write!(w, + " {}, {} -> {}", + arg, + regs.display_regunit(src), + regs.display_regunit(dst)) + } else { + write!(w, " {}, %{} -> %{}", arg, src, dst) + } + } + } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 02ec88196e..b907ee426f 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -16,7 +16,7 @@ use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Offset32, Uoffset32, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs}; -use cretonne::isa::{self, TargetIsa, Encoding}; +use cretonne::isa::{self, TargetIsa, Encoding, RegUnit}; use cretonne::settings::{self, Configurable}; use testfile::{TestFile, Details, Comment}; use error::{Location, Error, Result}; @@ -551,6 +551,29 @@ impl<'a> Parser<'a> { } } + // Match and consume a register unit either by number `%15` or by name `%rax`. + fn match_regunit(&mut self, isa: Option<&TargetIsa>) -> Result { + if let Some(Token::Name(name)) = self.token() { + self.consume(); + match isa { + Some(isa) => { + isa.register_info() + .parse_regunit(name) + .ok_or_else(|| self.error("invalid register name")) + } + None => { + name.parse() + .map_err(|_| self.error("invalid register number")) + } + } + } else { + match isa { + Some(isa) => err!(self.loc, "Expected {} register unit", isa.name()), + None => err!(self.loc, "Expected register unit number"), + } + } + } + /// Parse a list of test commands. pub fn parse_test_commands(&mut self) -> Vec> { let mut list = Vec::new(); @@ -1664,6 +1687,19 @@ impl<'a> Parser<'a> { offset, } } + InstructionFormat::RegMove => { + let arg = self.match_value("expected SSA value operand")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let src = self.match_regunit(ctx.unique_isa)?; + self.match_token(Token::Arrow, "expected '->' between register units")?; + let dst = self.match_regunit(ctx.unique_isa)?; + InstructionData::RegMove { + opcode, + arg, + src, + dst, + } + } }; Ok(idata) } From 1818bb18b528262c643261cca386c1695326584b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 8 May 2017 12:24:32 -0700 Subject: [PATCH 0737/3084] Ignore .mypy_cache A recent mypy update started writing the .mypy_cache directory which we don't want under version control. The cache is only used by the experimental "mypy --incremental" mode which we don't use, but it is always written anyway. --- cranelift/.gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/cranelift/.gitignore b/cranelift/.gitignore index 5f3b8f2899..a230556d47 100644 --- a/cranelift/.gitignore +++ b/cranelift/.gitignore @@ -7,3 +7,4 @@ target Cargo.lock .*.rustfmt cretonne.dbg* +.mypy_cache From aecd90a1b9b4b7332d8c1974b0e45a72404377c7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 8 May 2017 12:54:09 -0700 Subject: [PATCH 0738/3084] Run mypy in python 3 mode. This still picks up the 2.7 type annotations in comments. Fix the compute_quadratic signature to allow for the ValuewView of an OrderedDict in python 3. --- lib/cretonne/meta/constant_hash.py | 4 ++-- lib/cretonne/meta/mypy.ini | 1 - 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/meta/constant_hash.py b/lib/cretonne/meta/constant_hash.py index 96560ffc87..47f5eebe33 100644 --- a/lib/cretonne/meta/constant_hash.py +++ b/lib/cretonne/meta/constant_hash.py @@ -9,7 +9,7 @@ from __future__ import absolute_import from cdsl import next_power_of_two try: - from typing import Any, List, Sequence, Callable # noqa + from typing import Any, List, Iterable, Callable # noqa except ImportError: pass @@ -32,7 +32,7 @@ def simple_hash(s): def compute_quadratic(items, hash_function): - # type: (Sequence[Any], Callable[[Any], int]) -> List[Any] + # type: (Iterable[Any], Callable[[Any], int]) -> List[Any] """ Compute an open addressed, quadratically probed hash table containing `items`. The returned table is a list containing the elements of the diff --git a/lib/cretonne/meta/mypy.ini b/lib/cretonne/meta/mypy.ini index ca7f5e4c00..7046100b4c 100644 --- a/lib/cretonne/meta/mypy.ini +++ b/lib/cretonne/meta/mypy.ini @@ -1,5 +1,4 @@ [mypy] -python_version = 2.7 disallow_untyped_defs = True warn_unused_ignores = True warn_return_any = True From aaa70a677d3604ee90d03ef240710bab4137752b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 8 May 2017 12:20:35 -0700 Subject: [PATCH 0739/3084] Add a regs_overlap function to the isa module. Test it with the arm32 register banks which have the most interesting properties. Most other registers have a single register unit. --- lib/cretonne/src/isa/arm32/registers.rs | 37 ++++++++++++++++++++++++- lib/cretonne/src/isa/mod.rs | 2 +- lib/cretonne/src/isa/registers.rs | 10 +++++++ 3 files changed, 47 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/isa/arm32/registers.rs b/lib/cretonne/src/isa/arm32/registers.rs index 69571976cb..283f113d1e 100644 --- a/lib/cretonne/src/isa/arm32/registers.rs +++ b/lib/cretonne/src/isa/arm32/registers.rs @@ -6,7 +6,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-arm32.rs")); #[cfg(test)] mod tests { - use super::INFO; + use super::{INFO, GPR, S, D}; use isa::RegUnit; #[test] @@ -29,4 +29,39 @@ mod tests { assert_eq!(uname(31), "%s31"); assert_eq!(uname(64), "%r0"); } + + #[test] + fn overlaps() { + // arm32 has the most interesting register geometries, so test `regs_overlap()` here. + use isa::regs_overlap; + + let r0 = GPR.unit(0); + let r1 = GPR.unit(1); + let r2 = GPR.unit(2); + + assert!(regs_overlap(GPR, r0, GPR, r0)); + assert!(regs_overlap(GPR, r2, GPR, r2)); + assert!(!regs_overlap(GPR, r0, GPR, r1)); + assert!(!regs_overlap(GPR, r1, GPR, r0)); + assert!(!regs_overlap(GPR, r2, GPR, r1)); + assert!(!regs_overlap(GPR, r1, GPR, r2)); + + let s0 = S.unit(0); + let s1 = S.unit(1); + let s2 = S.unit(2); + let s3 = S.unit(3); + let d0 = D.unit(0); + let d1 = D.unit(1); + + assert!(regs_overlap(S, s0, D, d0)); + assert!(regs_overlap(S, s1, D, d0)); + assert!(!regs_overlap(S, s0, D, d1)); + assert!(!regs_overlap(S, s1, D, d1)); + assert!(regs_overlap(S, s2, D, d1)); + assert!(regs_overlap(S, s3, D, d1)); + assert!(!regs_overlap(D, d1, S, s1)); + assert!(regs_overlap(D, d1, S, s2)); + assert!(!regs_overlap(D, d0, D, d1)); + assert!(regs_overlap(D, d1, D, d1)); + } } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 4b618e0798..34d3e21ef6 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -42,7 +42,7 @@ pub use isa::constraints::{RecipeConstraints, OperandConstraint, ConstraintKind, BranchRange}; pub use isa::encoding::{Encoding, EncInfo}; -pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex}; +pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex, regs_overlap}; use binemit::CodeSink; use settings; diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 15a089db17..76fd39685a 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -188,6 +188,16 @@ impl fmt::Display for RegClassIndex { } } +/// Test of two registers overlap. +/// +/// A register is identified as a `(RegClass, RegUnit)` pair. The register class is needed to +/// determine the width (in regunits) of the register. +pub fn regs_overlap(rc1: RegClass, reg1: RegUnit, rc2: RegClass, reg2: RegUnit) -> bool { + let end1 = reg1 + rc1.width as RegUnit; + let end2 = reg2 + rc2.width as RegUnit; + !(end1 <= reg2 || end2 <= reg1) +} + /// Information about the registers in an ISA. /// /// The `RegUnit` data structure collects all relevant static information about the registers in an From f8a3a01f96d126d8550a2ab04bf451953a12ca63 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 2 May 2017 09:59:58 -0700 Subject: [PATCH 0740/3084] Add a few register utilities. --- lib/cretonne/src/isa/registers.rs | 5 +++ lib/cretonne/src/isa/riscv/registers.rs | 14 ++++++++- lib/cretonne/src/regalloc/allocatable_set.rs | 33 ++++++++++++++++++++ 3 files changed, 51 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 76fd39685a..122dec8e71 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -151,6 +151,11 @@ impl RegClassData { let uoffset = offset * self.width as usize; self.first + uoffset as RegUnit } + + /// Does this register class contain `regunit`? + pub fn contains(&self, regunit: RegUnit) -> bool { + self.mask[(regunit / 32) as usize] & (1u32 << (regunit % 32)) != 0 + } } impl fmt::Display for RegClassData { diff --git a/lib/cretonne/src/isa/riscv/registers.rs b/lib/cretonne/src/isa/riscv/registers.rs index 7deef9251a..9447e5ea29 100644 --- a/lib/cretonne/src/isa/riscv/registers.rs +++ b/lib/cretonne/src/isa/riscv/registers.rs @@ -6,7 +6,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-riscv.rs")); #[cfg(test)] mod tests { - use super::INFO; + use super::{INFO, GPR, FPR}; use isa::RegUnit; #[test] @@ -34,4 +34,16 @@ mod tests { assert_eq!(uname(63), "%f31"); assert_eq!(uname(64), "%INVALID64"); } + + #[test] + fn classes() { + assert!(GPR.contains(GPR.unit(0))); + assert!(GPR.contains(GPR.unit(31))); + assert!(!FPR.contains(GPR.unit(0))); + assert!(!FPR.contains(GPR.unit(31))); + assert!(!GPR.contains(FPR.unit(0))); + assert!(!GPR.contains(FPR.unit(31))); + assert!(FPR.contains(FPR.unit(0))); + assert!(FPR.contains(FPR.unit(31))); + } } diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index 29a7e26964..714a82aab7 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -78,6 +78,25 @@ impl AllocatableSet { } rsi } + + /// Check if any register units allocated out of this set interferes with units allocated out + /// of `other`. + /// + /// This assumes that unused bits are 1. + pub fn interferes_with(&self, other: &AllocatableSet) -> bool { + self.avail + .iter() + .zip(&other.avail) + .any(|(&x, &y)| (x | y) != !0) + } + + /// Intersect this set of allocatable registers with `other`. This has the effect of removing + /// any register units from this set that are not in `other`. + pub fn intersect(&mut self, other: &AllocatableSet) { + for (x, &y) in self.avail.iter_mut().zip(&other.avail) { + *x &= y; + } + } } /// Iterator over available registers in a register class. @@ -179,4 +198,18 @@ mod tests { assert_eq!(regs.iter(GPR).count(), 7); assert_eq!(regs.iter(DPR).collect::>(), [30, 33, 35]); } + + #[test] + fn interference() { + let mut regs1 = AllocatableSet::new(); + let mut regs2 = AllocatableSet::new(); + + assert!(!regs1.interferes_with(®s2)); + regs1.take(&GPR, 32); + assert!(!regs1.interferes_with(®s2)); + regs2.take(&GPR, 31); + assert!(!regs1.interferes_with(®s2)); + regs1.intersect(®s2); + assert!(regs1.interferes_with(®s2)); + } } From 402e437a4aaf2d0fe551962d1d996e32dfcb5eb8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 8 May 2017 13:57:39 -0700 Subject: [PATCH 0741/3084] Also return live-through values from process_inst(). The coloring algorithm will need to look at the live-through values to check if they interfere with fixed-register outputs for calls etc. --- lib/cretonne/src/regalloc/coloring.rs | 2 +- .../src/regalloc/live_value_tracker.rs | 21 +++++++++++++------ 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index a1b3899577..f664c43c48 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -276,7 +276,7 @@ impl<'a> Context<'a> { locations: &mut EntityMap) { // First update the live value tracker with this instruction. // Get lists of values that are killed and defined by `inst`. - let (kills, defs) = tracker.process_inst(inst, dfg, self.liveness); + let (_throughs, kills, defs) = tracker.process_inst(inst, dfg, self.liveness); // Get the operand constraints for `inst` that we are trying to satisfy. let constraints = self.encinfo diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 78085564f7..4ef7df8d3a 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -139,7 +139,7 @@ impl LiveValueTracker { /// This depends on the stored live value set at `ebb`'s immediate dominator, so that must have /// been visited first. /// - /// Returns `(liveins, args)` as a pair or slices. The first slice is the set of live-in values + /// Returns `(liveins, args)` as a pair of slices. The first slice is the set of live-in values /// from the immediate dominator. The second slice is the set of `ebb` arguments that are live. /// Dead arguments with no uses are ignored and not added to the set. pub fn ebb_top(&mut self, @@ -209,9 +209,16 @@ impl LiveValueTracker { /// Determine the set of already live values that are killed by `inst`, and add the new defined /// values to the tracked set. /// - /// Returns `(kills, defs)` as a pair of slices. The `defs` slice is guaranteed to be in the - /// same order as `inst`'s results, and includes dead defines. The order of `kills` is - /// arbitrary. + /// Returns `(throughs, kills, defs)` as a tuple of slices: + /// + /// 1. The `throughs` slice is the set of live-through values that are neither defined nor + /// killed by the instruction. + /// 2. The `kills` slice is the set of values that were live before the instruction and are + /// killed at the instruction. This does not include dead defs. + /// 3. The `defs` slice is guaranteed to be in the same order as `inst`'s results, and includes + /// dead defines. + /// + /// The order of `throughs` and `kills` is arbitrary. /// /// The `drop_dead()` method must be called next to actually remove the dead values from the /// tracked set after the two returned slices are no longer needed. @@ -219,7 +226,7 @@ impl LiveValueTracker { inst: Inst, dfg: &DataFlowGraph, liveness: &Liveness) - -> (&[LiveValue], &[LiveValue]) { + -> (&[LiveValue], &[LiveValue], &[LiveValue]) { // Save a copy of the live values before any branches or jumps that could be somebody's // immediate dominator. match dfg[inst].analyze_branch(&dfg.value_lists) { @@ -249,7 +256,9 @@ impl LiveValueTracker { } } - (&self.live.values[first_kill..first_def], &self.live.values[first_def..]) + (&self.live.values[0..first_kill], + &self.live.values[first_kill..first_def], + &self.live.values[first_def..]) } /// Drop the values that are now dead after moving past `inst`. From 0f41cbdee25a60ddf93f9b8606cb801f6cd3e650 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 8 May 2017 15:21:00 -0700 Subject: [PATCH 0742/3084] Add support for tied operand constraints. The register constraint for an output operand can be specified as an integer indicating the input operand number to tie. The tied operands must use the same register. Generate operand constraints using ConstraintKind::Tied(n) for both the tied operands. The n index refers to the opposite array. The input operand refers to the outs array and vice versa. --- lib/cretonne/meta/cdsl/isa.py | 22 ++++++++++++--- lib/cretonne/meta/gen_encoding.py | 42 ++++++++++++++++++++--------- lib/cretonne/src/isa/constraints.rs | 11 ++++---- 3 files changed, 53 insertions(+), 22 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index a44a1e6333..f1125f59da 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -7,7 +7,7 @@ from .ast import Apply # The typing module is only required by mypy, and we don't use these imports # outside type comments. try: - from typing import Tuple, Union, Any, Iterable, Sequence, List, Set, TYPE_CHECKING # noqa + from typing import Tuple, Union, Any, Iterable, Sequence, List, Set, Dict, TYPE_CHECKING # noqa if TYPE_CHECKING: from .instructions import MaybeBoundInst, InstructionGroup, InstructionFormat # noqa from .predicates import PredNode # noqa @@ -220,13 +220,27 @@ class EncRecipe(object): if isinstance(c, int): # An integer constraint is bound to a value operand. # Check that it is in range. - assert c >= 0 - if not self.format.has_value_list: - assert c < self.format.num_value_operands + assert c >= 0 and c < len(self.ins) else: assert isinstance(c, RegClass) or isinstance(c, Register) return seq + def ties(self): + # type: () -> Tuple[Dict[int, int], Dict[int, int]] + """ + Return two dictionaries representing the tied operands. + + The first maps input number to tied output number, the second maps + output number to tied input number. + """ + i2o = dict() # type: Dict[int, int] + o2i = dict() # type: Dict[int, int] + for o, i in enumerate(self.outs): + if isinstance(i, int): + i2o[i] = o + o2i[o] = i + return (i2o, o2i) + class Encoding(object): """ diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index bbe1df7591..f19377b8bd 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -60,9 +60,9 @@ from cdsl.registers import RegClass, Register from cdsl.predicates import FieldPredicate try: - from typing import Sequence, Set, Tuple, List, Iterable, DefaultDict, TYPE_CHECKING # noqa + from typing import Sequence, Set, Tuple, List, Dict, Iterable, DefaultDict, TYPE_CHECKING # noqa if TYPE_CHECKING: - from cdsl.isa import TargetISA, OperandConstraint, Encoding, CPUMode # noqa + from cdsl.isa import TargetISA, OperandConstraint, Encoding, CPUMode, EncRecipe # noqa from cdsl.predicates import PredNode, PredLeaf # noqa from cdsl.types import ValueType # noqa from cdsl.instructions import Instruction # noqa @@ -470,13 +470,20 @@ def emit_recipe_constraints(isa, fmt): .format(len(isa.all_recipes)), '];'): for r in isa.all_recipes: fmt.comment(r.name) + tied_i2o, tied_o2i = r.ties() with fmt.indented('RecipeConstraints {', '},'): - emit_operand_constraints(r.ins, 'ins', fmt) - emit_operand_constraints(r.outs, 'outs', fmt) + emit_operand_constraints(r, r.ins, 'ins', tied_i2o, fmt) + emit_operand_constraints(r, r.outs, 'outs', tied_o2i, fmt) -def emit_operand_constraints(seq, field, fmt): - # type: (Sequence[OperandConstraint], str, srcgen.Formatter) -> None +def emit_operand_constraints( + recipe, # type: EncRecipe + seq, # type: Sequence[OperandConstraint] + field, # type: str + tied, # type: Dict[int, int] + fmt # type: srcgen.Formatter + ): + # type: (...) -> None """ Emit a struct field initializer for an array of operand constraints. """ @@ -484,16 +491,25 @@ def emit_operand_constraints(seq, field, fmt): fmt.line('{}: &[],'.format(field)) return with fmt.indented('{}: &['.format(field), '],'): - for cons in seq: + for n, cons in enumerate(seq): with fmt.indented('OperandConstraint {', '},'): if isinstance(cons, RegClass): - fmt.line('kind: ConstraintKind::Reg,') - fmt.line('regclass: {},'.format(cons)) + if n in tied: + fmt.format('kind: ConstraintKind::Tied({}),', tied[n]) + else: + fmt.line('kind: ConstraintKind::Reg,') + fmt.format('regclass: {},', cons) elif isinstance(cons, Register): - fmt.line( - 'kind: ConstraintKind::FixedReg({}),' - .format(cons.unit)) - fmt.line('regclass: {},'.format(cons.regclass)) + assert n not in tied, "Can't tie fixed register operand" + fmt.format( + 'kind: ConstraintKind::FixedReg({}),', cons.unit) + fmt.format('regclass: {},', cons.regclass) + elif isinstance(cons, int): + # This is a tied output constraint. It should never happen + # for input constraints. + assert cons == tied[n], "Invalid tied constraint" + fmt.format('kind: ConstraintKind::Tied({}),', cons) + fmt.format('regclass: {},', recipe.ins[cons]) else: raise AssertionError( 'Unsupported constraint {}'.format(cons)) diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index 23cd923f93..fa30fd24fe 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -33,13 +33,14 @@ pub enum ConstraintKind { /// register. FixedReg(RegUnit), - /// This result value must use the same register as an input value operand. Input operands - /// can't be tied. + /// This result value must use the same register as an input value operand. /// - /// The associated number is the index of the input value operand this result is tied to. + /// The associated number is the index of the input value operand this result is tied to. The + /// constraint's `regclass` field is the same as the tied operand's register class. /// - /// The constraint's `regclass` field is the top-level register class containing the tied - /// operand's register class. + /// When an (in, out) operand pair is tied, this constraint kind appears in both the `ins` and + /// the `outs` arrays. The constraint for the in operand is `Tied(out)`, and the constraint for + /// the out operand is `Tied(in)`. Tied(u8), /// This operand must be a value in a stack slot. From 39e69ff565e0cded242b9d3631ca6f4b493aae21 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 8 May 2017 12:44:22 -0700 Subject: [PATCH 0743/3084] Add constraint summaries to RecipeConstraints. Most instructions don't have any fixed register constraints. Add boolean summaries that can be used to check if it is worthwhile to scan the constraint lists when looking for a fixed register constraint. Also add a tied_ops summary bool which indicates that the instruction has tied operand constraints. --- lib/cretonne/meta/gen_encoding.py | 9 +++++++++ lib/cretonne/src/isa/constraints.rs | 9 +++++++++ 2 files changed, 18 insertions(+) diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index f19377b8bd..c56db8fc79 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -474,6 +474,15 @@ def emit_recipe_constraints(isa, fmt): with fmt.indented('RecipeConstraints {', '},'): emit_operand_constraints(r, r.ins, 'ins', tied_i2o, fmt) emit_operand_constraints(r, r.outs, 'outs', tied_o2i, fmt) + fmt.format( + 'fixed_ins: {},', + str(any(isinstance(c, Register) + for c in r.ins)).lower()) + fmt.format( + 'fixed_outs: {},', + str(any(isinstance(c, Register) + for c in r.outs)).lower()) + fmt.format('tied_ops: {},', str(bool(tied_i2o)).lower()) def emit_operand_constraints( diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index fa30fd24fe..e38b044535 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -67,6 +67,15 @@ pub struct RecipeConstraints { /// If the instruction produces a variable number of results, it's probably a call and the /// constraints must be derived from the calling convention ABI. pub outs: &'static [OperandConstraint], + + /// Are any of the input constraints `FixedReg`? + pub fixed_ins: bool, + + /// Are any of the output constraints `FixedReg`? + pub fixed_outs: bool, + + /// Are there any tied operands? + pub tied_ops: bool, } /// Constraints on the range of a branch instruction. From 041fda63acdee5a4c2c235f31ab8fef1231fb1dd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Sun, 30 Apr 2017 13:34:51 -0700 Subject: [PATCH 0744/3084] Add the very basics of Intel 32-bit instruction encodings. Tabulate the Intel opcode representations and implement an OP() function which computes the encoding bits. Implement the single-byte opcode with a reg-reg ModR/M byte. --- cranelift/filetests/isa/intel/binary32.cton | 23 +++++++ lib/cretonne/meta/isa/intel/__init__.py | 2 +- lib/cretonne/meta/isa/intel/encodings.py | 10 +++ lib/cretonne/meta/isa/intel/recipes.py | 67 +++++++++++++++++++++ lib/cretonne/src/isa/intel/binemit.rs | 30 ++++++++- lib/cretonne/src/isa/intel/enc_tables.rs | 3 +- lib/cretonne/src/isa/intel/mod.rs | 4 ++ 7 files changed, 136 insertions(+), 3 deletions(-) create mode 100644 cranelift/filetests/isa/intel/binary32.cton create mode 100644 lib/cretonne/meta/isa/intel/encodings.py create mode 100644 lib/cretonne/meta/isa/intel/recipes.py diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton new file mode 100644 index 0000000000..c8440440b9 --- /dev/null +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -0,0 +1,23 @@ +; binary emission of 32-bit code. +test binemit +isa intel + +; The binary encodings can be verified with the command: +; +; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/binary32.cton | llvm-mc -show-encoding -triple=i386 +; + +function I32() { +ebb0: + [-,%rcx] v1 = iconst.i32 1 + [-,%rsi] v2 = iconst.i32 2 + + ; Integer Register-Register Operations. + + ; asm: addl %esi, %ecx + [-,%rcx] v10 = iadd v1, v2 ; bin: 01 f1 + ; asm: addl %ecx, %esi + [-,%rsi] v11 = iadd v2, v1 ; bin: 01 ce + + return +} diff --git a/lib/cretonne/meta/isa/intel/__init__.py b/lib/cretonne/meta/isa/intel/__init__.py index 6aea0fd288..0828d790aa 100644 --- a/lib/cretonne/meta/isa/intel/__init__.py +++ b/lib/cretonne/meta/isa/intel/__init__.py @@ -17,7 +17,7 @@ is no x87 floating point support. from __future__ import absolute_import from . import defs -from . import settings, registers # noqa +from . import encodings, settings, registers # noqa # Re-export the primary target ISA definition. ISA = defs.ISA.finish() diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py new file mode 100644 index 0000000000..5d1ffdb4ec --- /dev/null +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -0,0 +1,10 @@ +""" +Intel Encodings. +""" +from __future__ import absolute_import +from base import instructions as base +from .defs import I32 +from .recipes import Op1rr +from .recipes import OP + +I32.enc(base.iadd.i32, Op1rr, OP(0x01)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py new file mode 100644 index 0000000000..9535598917 --- /dev/null +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -0,0 +1,67 @@ +""" +Intel Encoding recipes. +""" +from __future__ import absolute_import +from cdsl.isa import EncRecipe +# from cdsl.predicates import IsSignedInt +from base.formats import Binary +from .registers import GPR + +# Opcode representation. +# +# Cretonne requires each recipe to have a single encoding size in bytes, and +# Intel opcodes are variable length, so we use separate recipes for different +# styles of opcodes and prefixes. The opcode format is indicated by the recipe +# name prefix: +# +# Op1* OP(op) +# 0F Op2* OP(op) +# 0F 38 Op3* OP38(op) +# 0F 3A Op3* OP3A(op) +# 66 Mp1* MP66(op) +# 66 0F Mp2* MP66(op) +# 66 0F 38 Mp3* MP6638(op) +# 66 0F 3A Mp3* MP663A(op) +# F2 Mp1* MPF2(op) +# F2 0F Mp2* MPF2(op) +# F2 0F 38 Mp3* MPF238(op) +# F2 0F 3A Mp3* MPF23A(op) +# F3 Mp1* MPF3(op) +# F3 0F Mp2* MPF3(op) +# F3 0F 38 Mp3* MPF338(op) +# F3 0F 3A Mp3* MPF33A(op) +# +# VEX/XOP and EVEX prefixes are not yet supported. +# +# The encoding bits are: +# +# 0-7: The opcode byte . +# 8-9: pp, mandatory prefix: +# 00 none (Op*) +# 01 66 (Mp*) +# 10 F3 (Mp*) +# 11 F2 (Mp*) +# 10-11: mm, opcode map: +# 00 (Op1/Mp1) +# 01 0F (Op2/Mp2) +# 10 0F 38 (Op3/Mp3) +# 11 0F 3A (Op3/Mp3) +# 12-14 rrr, opcode bits for the ModR/M byte for certain opcodes. +# 15: REX.W bit (or VEX.W/E) +# +# There is some redundancy between bits 8-11 and the recipe names, but we have +# enough bits, and the pp+mm format is ready for supporting VEX prefixes. + + +def OP(op, pp=0, mm=0, rrr=0, w=0): + # type: (int, int, int, int, int) -> int + assert op <= 0xff + assert pp <= 0b11 + assert mm <= 0b11 + assert rrr <= 0b111 + assert w <= 1 + return op | (pp << 8) | (mm << 10) | (rrr << 12) | (w << 15) + + +# XX /r +Op1rr = EncRecipe('Op1rr', Binary, size=2, ins=(GPR, GPR), outs=0) diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 8870abd8f1..4b322c6969 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -1,6 +1,34 @@ //! Emitting binary Intel machine code. use binemit::{CodeSink, bad_encoding}; -use ir::{Function, Inst}; +use ir::{Function, Inst, InstructionData}; +use isa::RegUnit; include!(concat!(env!("OUT_DIR"), "/binemit-intel.rs")); + +pub static RELOC_NAMES: [&'static str; 1] = ["Call"]; + +fn put_op1(bits: u16, sink: &mut CS) { + debug_assert!(bits & 0x0f00 == 0, "Invalid encoding bits for Op1*"); + sink.put1(bits as u8); +} + +fn modrm_rr(rm: RegUnit, reg: RegUnit, sink: &mut CS) { + let reg = reg as u8 & 7; + let rm = rm as u8 & 7; + let mut b = 0b11000000; + b |= reg << 3; + b |= rm; + sink.put1(b); +} + +fn recipe_op1rr(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Binary { args, .. } = func.dfg[inst] { + put_op1(func.encodings[inst].bits(), sink); + modrm_rr(func.locations[args[0]].unwrap_reg(), + func.locations[args[1]].unwrap_reg(), + sink); + } else { + panic!("Expected Binary format: {:?}", func.dfg[inst]); + } +} diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 1ee1fbd4b3..1ec93f00a9 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -1,10 +1,11 @@ //! Encoding tables for Intel ISAs. -use ir::InstructionData; +use ir::{Opcode, InstructionData}; use ir::types; use isa::EncInfo; use isa::constraints::*; use isa::enc_tables::{Level1Entry, Level2Entry}; use isa::encoding::RecipeSizing; +use super::registers::*; include!(concat!(env!("OUT_DIR"), "/encoding-intel.rs")); diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 5871c8766a..1711be3166 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -94,4 +94,8 @@ impl TargetIsa for Isa { fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } + + fn reloc_names(&self) -> &'static [&'static str] { + &binemit::RELOC_NAMES + } } From 976b22d8161379ccf1fcde7b524b24705fbde211 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 9 May 2017 11:43:58 -0700 Subject: [PATCH 0745/3084] Make `srem` have the sign of the dividend. This is how remainder is defined in C (as of C99), C++ (as of C++11), Rust, and WebAssembly, for example. --- lib/cretonne/meta/base/instructions.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index b9d2d5a5af..5777ed76ea 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -683,14 +683,13 @@ urem = Instruction( srem = Instruction( 'srem', """ - Signed integer remainder. + Signed integer remainder. The result has the sign of the dividend. This operation traps if the divisor is zero. .. todo:: Integer remainder vs modulus. - Clarify whether the result has the sign of the divisor or the - dividend. Should we add a ``smod`` instruction for the case where + Should we add a ``smod`` instruction for the case where the result has the same sign as the divisor? """, ins=(x, y), outs=a, can_trap=True) From a0085434af43ecfc9e59cb7ed4307c78b93c3a4b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 8 May 2017 20:56:08 -0700 Subject: [PATCH 0746/3084] Add encodings for Intel dynamic shift instructions. These instructions have a fixed register constraint; the shift amount is passed in CL. Add meta language syntax so a fixed register can be specified as "GPR.rcx". --- cranelift/filetests/isa/intel/binary32.cton | 15 +++++++++++ lib/cretonne/meta/cdsl/registers.py | 29 ++++++++++++++++++--- lib/cretonne/meta/isa/intel/encodings.py | 9 ++++++- lib/cretonne/meta/isa/intel/recipes.py | 3 +++ lib/cretonne/src/isa/intel/binemit.rs | 21 +++++++++++++++ 5 files changed, 72 insertions(+), 5 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index c8440440b9..d609a0c030 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -19,5 +19,20 @@ ebb0: ; asm: addl %ecx, %esi [-,%rsi] v11 = iadd v2, v1 ; bin: 01 ce + ; Dynamic shifts take the shift amount in %rcx. + + ; asm: shll %cl, %esi + [-,%rsi] v12 = ishl v2, v1 ; bin: d3 e6 + ; asm: shll %cl, %ecx + [-,%rcx] v13 = ishl v1, v1 ; bin: d3 e1 + ; asm: shrl %cl, %esi + [-,%rsi] v14 = ushr v2, v1 ; bin: d3 ee + ; asm: shrl %cl, %ecx + [-,%rcx] v15 = ushr v1, v1 ; bin: d3 e9 + ; asm: sarl %cl, %esi + [-,%rsi] v16 = sshr v2, v1 ; bin: d3 fe + ; asm: sarl %cl, %ecx + [-,%rcx] v17 = sshr v1, v1 ; bin: d3 f9 + return } diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index b21d2348a6..1f6bfa682f 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -12,12 +12,12 @@ register bank. A register bank consists of a number of *register units* which are the smallest indivisible units of allocation and interference. A register unit doesn't -necesarily correspond to a particular number of bits in a register, it is more +necessarily correspond to a particular number of bits in a register, it is more like a placeholder that can be used to determine of a register is taken or not. The register allocator works with *register classes* which can allocate one or more register units at a time. A register class allocates more than one -register unit at a time when its registers are composed of smaller alocatable +register unit at a time when its registers are composed of smaller allocatable units. For example, the ARM double precision floating point registers are composed of two single precision registers. """ @@ -151,6 +151,18 @@ class RegBank(object): # sub-class. rc2.subclasses.append(rc1) + def unit_by_name(self, name): + # type: (str) -> int + """ + Get a register unit in this bank by name. + """ + if name in self.names: + r = self.names.index(name) + elif name.startswith(self.prefix): + r = int(name[len(self.prefix):]) + assert r < self.units, 'Invalid register name: ' + name + return self.first_unit + r + class RegClass(object): """ @@ -242,6 +254,15 @@ class RegClass(object): return RegClass(self.bank, count=c, width=w, start=s) + def __getattr__(self, attr): + # type: (str) -> Register + """ + Get a specific register in the class by name. + + For example: `GPR.r5`. + """ + return Register(self, self.bank.unit_by_name(attr)) + def mask(self): # type: () -> List[int] """ @@ -298,8 +319,8 @@ class Register(object): Specific registers are used to describe constraints on instructions where some operands must use a fixed register. - Register objects should be created using the indexing syntax on the - register class. + Register instances can be created with the constructor, or accessed as + attributes on the register class: `GPR.rcx`. """ def __init__(self, rc, unit): # type: (RegClass, int) -> None diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 5d1ffdb4ec..3896532c85 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -4,7 +4,14 @@ Intel Encodings. from __future__ import absolute_import from base import instructions as base from .defs import I32 -from .recipes import Op1rr +from .recipes import Op1rr, Op1rc from .recipes import OP I32.enc(base.iadd.i32, Op1rr, OP(0x01)) + +# 32-bit shifts and rotates. +# Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit +# and 16-bit shifts would need explicit masking. +I32.enc(base.ishl.i32.i32, Op1rc, OP(0xd3, rrr=4)) +I32.enc(base.ushr.i32.i32, Op1rc, OP(0xd3, rrr=5)) +I32.enc(base.sshr.i32.i32, Op1rc, OP(0xd3, rrr=7)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 9535598917..0ffd7e5c50 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -65,3 +65,6 @@ def OP(op, pp=0, mm=0, rrr=0, w=0): # XX /r Op1rr = EncRecipe('Op1rr', Binary, size=2, ins=(GPR, GPR), outs=0) + +# XX /n with one arg in %rcx, for shifts. +Op1rc = EncRecipe('Op1rc', Binary, size=2, ins=(GPR, GPR.rcx), outs=0) diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 4b322c6969..0644d583d3 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -13,6 +13,7 @@ fn put_op1(bits: u16, sink: &mut CS) { sink.put1(bits as u8); } +/// Emit a ModR/M byte for reg-reg operands. fn modrm_rr(rm: RegUnit, reg: RegUnit, sink: &mut CS) { let reg = reg as u8 & 7; let rm = rm as u8 & 7; @@ -22,6 +23,16 @@ fn modrm_rr(rm: RegUnit, reg: RegUnit, sink: &mut CS) { sink.put1(b); } +/// Emit a ModR/M byte where the reg bits are part of the opcode. +fn modrm_r_bits(rm: RegUnit, bits: u16, sink: &mut CS) { + let reg = (bits >> 12) as u8 & 7; + let rm = rm as u8 & 7; + let mut b = 0b11000000; + b |= reg << 3; + b |= rm; + sink.put1(b); +} + fn recipe_op1rr(func: &Function, inst: Inst, sink: &mut CS) { if let InstructionData::Binary { args, .. } = func.dfg[inst] { put_op1(func.encodings[inst].bits(), sink); @@ -32,3 +43,13 @@ fn recipe_op1rr(func: &Function, inst: Inst, sink: &mut C panic!("Expected Binary format: {:?}", func.dfg[inst]); } } + +fn recipe_op1rc(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Binary { args, .. } = func.dfg[inst] { + let bits = func.encodings[inst].bits(); + put_op1(bits, sink); + modrm_r_bits(func.locations[args[0]].unwrap_reg(), bits, sink); + } else { + panic!("Expected Binary format: {:?}", func.dfg[inst]); + } +} From 2873019779365118ae6f034f4830850de4d79d87 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 2 May 2017 13:48:19 -0700 Subject: [PATCH 0747/3084] Add a register diversion tracker. Keep track of the current location of register values as regmove instructions are encountered throughout an EBB. --- lib/cretonne/src/regalloc/diversion.rs | 114 +++++++++++++++++++++++++ lib/cretonne/src/regalloc/mod.rs | 2 + 2 files changed, 116 insertions(+) create mode 100644 lib/cretonne/src/regalloc/diversion.rs diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs new file mode 100644 index 0000000000..2096af8722 --- /dev/null +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -0,0 +1,114 @@ +//! Register diversions. +//! +//! Normally, a value is assigned to a single register or stack location by the register allocator. +//! Sometimes, it is necessary to move register values to a different register in order to satisfy +//! instruction constraints. +//! +//! These register diversions are local to an EBB. No values can be diverted when entering a new +//! EBB. + +use entity_map::EntityMap; +use ir::{Value, ValueLoc}; +use isa::RegUnit; + +/// A diversion of a value from its original register location to a new register. +/// +/// In IL, a diversion is represented by a `regmove` instruction, possibly a chain of them for the +/// same value. +/// +/// When tracking diversions, the `from` field is the original assigned value location, and `to` is +/// the current one. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Diversion { + /// The value that is diverted. + pub value: Value, + /// The original register value location. + pub from: RegUnit, + /// The current register value location. + pub to: RegUnit, +} + +impl Diversion { + /// Make a new register diversion. + pub fn new(value: Value, from: RegUnit, to: RegUnit) -> Diversion { + Diversion { value, from, to } + } +} + +/// Keep track of register diversions in an EBB. +pub struct RegDiversions { + current: Vec, +} + +impl RegDiversions { + /// Create a new empty diversion tracker. + pub fn new() -> RegDiversions { + RegDiversions { current: Vec::new() } + } + + /// Clear the tracker, preparing for a new EBB. + pub fn clear(&mut self) { + self.current.clear() + } + + /// Get the current diversion of `value`, if any. + pub fn diversion(&self, value: Value) -> Option<&Diversion> { + self.current.iter().find(|d| d.value == value) + } + + /// Get all current diversion. + pub fn all(&self) -> &[Diversion] { + self.current.as_slice() + } + + /// Get the current register location for `value`. Fall back to the assignment map for + /// non-diverted values. + pub fn reg(&self, value: Value, locations: &EntityMap) -> RegUnit { + match self.diversion(value) { + Some(d) => d.to, + None => locations[value].unwrap_reg(), + } + } + + /// Record a register move. + pub fn regmove(&mut self, value: Value, from: RegUnit, to: RegUnit) { + if let Some(i) = self.current.iter().position(|d| d.value == value) { + debug_assert_eq!(self.current[i].to, from, "Bad regmove chain for {}", value); + if self.current[i].from != to { + self.current[i].to = to; + } else { + self.current.swap_remove(i); + } + } else { + self.current.push(Diversion::new(value, from, to)); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use ir::Value; + use entity_map::EntityRef; + + #[test] + fn inserts() { + let mut divs = RegDiversions::new(); + let v1 = Value::new(1); + let v2 = Value::new(2); + + divs.regmove(v1, 10, 12); + assert_eq!(divs.diversion(v1), + Some(&Diversion { + value: v1, + from: 10, + to: 12, + })); + assert_eq!(divs.diversion(v2), None); + + divs.regmove(v1, 12, 11); + assert_eq!(divs.diversion(v1).unwrap().to, 11); + divs.regmove(v1, 11, 10); + assert_eq!(divs.diversion(v1), None); + } +} diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index 0682be2fd6..9803c3af3f 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -10,6 +10,8 @@ pub mod coloring; mod affinity; mod context; +mod diversion; pub use self::allocatable_set::AllocatableSet; pub use self::context::Context; +pub use self::diversion::RegDiversions; From 6b4c28d554dc849557101782cbda9854cc632c1a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 28 Apr 2017 15:41:28 -0700 Subject: [PATCH 0748/3084] Use a constraint solver for register coloring. Most of the time, register coloring is almost trivial: just pick available registers for the values defined by the current instruction. However, some instructions have register operand constraints, and it may be necessary to move live registers around to satisfy the constraints. Sometimes the instruction's own operands can interfere with each other in a way that you can't just pick a register assignment for each output in order. This is complicated enough that it is worthwhile to represent as a constraint satisfaction problem in a separate solver module. The representation is chosen to be very fast in the common case where the constraints are trivial to solve. The current implementation is still incomplete, but as functional as the code it's replacing. Missing features: - Handle tied operand constraints. - Handle ABI constraints on calls and return instructions. - Execute a constraint solution by emitting regmove instructions. - Handling register diversions before leaving the EBB. --- lib/cretonne/src/isa/encoding.rs | 2 +- lib/cretonne/src/regalloc/coloring.rs | 329 +++++++++++----- lib/cretonne/src/regalloc/mod.rs | 1 + lib/cretonne/src/regalloc/solver.rs | 536 ++++++++++++++++++++++++++ 4 files changed, 763 insertions(+), 105 deletions(-) create mode 100644 lib/cretonne/src/regalloc/solver.rs diff --git a/lib/cretonne/src/isa/encoding.rs b/lib/cretonne/src/isa/encoding.rs index 93888b109e..b5062d988c 100644 --- a/lib/cretonne/src/isa/encoding.rs +++ b/lib/cretonne/src/isa/encoding.rs @@ -104,7 +104,7 @@ pub struct EncInfo { impl EncInfo { /// Get the value operand constraints for `enc` if it is a legal encoding. - pub fn operand_constraints(&self, enc: Encoding) -> Option<&RecipeConstraints> { + pub fn operand_constraints(&self, enc: Encoding) -> Option<&'static RecipeConstraints> { self.constraints.get(enc.recipe()) } diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index f664c43c48..871d86e757 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -15,7 +15,11 @@ //! values used by the tied operands must be killed by the instruction. This can be achieved by //! inserting a `copy` to a new value immediately before the two-address instruction. //! -//! 3. The register pressure must be lowered sufficiently by inserting spill code. Register +//! 3. If a value is bound to more than one operand on the same instruction, the operand +//! constraints must be compatible. This can also be achieved by inserting copies so the +//! incompatible operands get different values. +//! +//! 4. The register pressure must be lowered sufficiently by inserting spill code. Register //! operands are allowed to read spilled values, but each such instance must be counted as using //! a register. //! @@ -26,30 +30,28 @@ //! a topological order relative to the dominance relation, we can assign colors to the values //! defined by the instruction and only consider the colors of other values that are live at the //! instruction. -//! -//! The topological order of instructions inside an EBB is simply the layout order, starting from -//! the EBB header. A topological order of the EBBs can only visit an EBB once its immediate -//! dominator has been visited. -//! -//! There are many valid topological orders of the EBBs, and the specific order can affect which -//! coloring hints are satisfied and which are broken. -//! use entity_map::EntityMap; use dominator_tree::DominatorTree; use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph, Signature, ArgumentLoc}; -use isa::{TargetIsa, RegInfo, Encoding, EncInfo, ConstraintKind}; +use isa::{TargetIsa, Encoding, EncInfo, OperandConstraint, ConstraintKind}; +use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; use regalloc::affinity::Affinity; use regalloc::allocatable_set::AllocatableSet; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; +use regalloc::solver::Solver; +use regalloc::RegDiversions; use topo_order::TopoOrder; /// Data structures for the coloring pass. /// /// These are scratch space data structures that can be reused between invocations. -pub struct Coloring {} +pub struct Coloring { + divert: RegDiversions, + solver: Solver, +} /// Bundle of references that the coloring algorithm needs. /// @@ -69,6 +71,13 @@ struct Context<'a> { domtree: &'a DominatorTree, liveness: &'a mut Liveness, + // References to working set data structures. + // If we need to borrow out of a data structure across a method call, it must be passed as a + // function argument instead, see the `LiveValueTracker` arguments. + topo: &'a mut TopoOrder, + divert: &'a mut RegDiversions, + solver: &'a mut Solver, + // Pristine set of registers that the allocator can use. // This set remains immutable, we make clones. usable_regs: AllocatableSet, @@ -77,7 +86,10 @@ struct Context<'a> { impl Coloring { /// Allocate scratch space data structures for the coloring pass. pub fn new() -> Coloring { - Coloring {} + Coloring { + divert: RegDiversions::new(), + solver: Solver::new(), + } } /// Run the coloring algorithm over `func`. @@ -93,26 +105,31 @@ impl Coloring { encinfo: isa.encoding_info(), domtree, liveness, + topo, + divert: &mut self.divert, + solver: &mut self.solver, usable_regs: isa.allocatable_registers(func), }; - ctx.run(func, topo, tracker) + ctx.run(func, tracker) } } impl<'a> Context<'a> { /// Run the coloring algorithm. - fn run(&mut self, func: &mut Function, topo: &mut TopoOrder, tracker: &mut LiveValueTracker) { - // Just visit blocks in layout order, letting `topo` enforce a topological ordering. + fn run(&mut self, func: &mut Function, tracker: &mut LiveValueTracker) { + // Just visit blocks in layout order, letting `self.topo` enforce a topological ordering. // TODO: Once we have a loop tree, we could visit hot blocks first. - topo.reset(func.layout.ebbs()); - while let Some(ebb) = topo.next(&func.layout, self.domtree) { + self.topo.reset(func.layout.ebbs()); + while let Some(ebb) = self.topo.next(&func.layout, self.domtree) { self.visit_ebb(ebb, func, tracker); } } /// Visit `ebb`, assuming that the immediate dominator has already been visited. fn visit_ebb(&mut self, ebb: Ebb, func: &mut Function, tracker: &mut LiveValueTracker) { + dbg!("Coloring {}:", ebb); let mut regs = self.visit_ebb_header(ebb, func, tracker); + self.divert.clear(); // Now go through the instructions in `ebb` and color the values they define. let mut pos = Cursor::new(&mut func.layout); @@ -168,10 +185,15 @@ impl<'a> Context<'a> { .get(value) .expect("No live range for live-in") .affinity; - if let Affinity::Reg(rc_index) = affinity { - let regclass = self.reginfo.rc(rc_index); - match func.locations[value] { - ValueLoc::Reg(regunit) => regs.take(regclass, regunit), + if let Affinity::Reg(rci) = affinity { + let rc = self.reginfo.rc(rci); + let loc = func.locations[value]; + dbg!("Live-in: {}:{} in {}", + lv.value, + rc, + loc.display(&self.reginfo)); + match loc { + ValueLoc::Reg(reg) => regs.take(rc, reg), ValueLoc::Unassigned => panic!("Live-in {} wasn't assigned", value), ValueLoc::Stack(ss) => { panic!("Live-in {} is in {}, should be register", value, ss) @@ -200,11 +222,11 @@ impl<'a> Context<'a> { for (lv, abi) in args.iter().zip(&sig.argument_types) { match lv.affinity { - Affinity::Reg(rc_index) => { - let regclass = self.reginfo.rc(rc_index); - if let ArgumentLoc::Reg(regunit) = abi.location { - regs.take(regclass, regunit); - *locations.ensure(lv.value) = ValueLoc::Reg(regunit); + Affinity::Reg(rci) => { + let rc = self.reginfo.rc(rci); + if let ArgumentLoc::Reg(reg) = abi.location { + regs.take(rc, reg); + *locations.ensure(lv.value) = ValueLoc::Reg(reg); } else { // This should have been fixed by the reload pass. panic!("Entry arg {} has {} affinity, but ABI {}", @@ -247,14 +269,14 @@ impl<'a> Context<'a> { -> AllocatableSet { for lv in args { // Only look at the register arguments. - if let Affinity::Reg(rc_index) = lv.affinity { - let regclass = self.reginfo.rc(rc_index); + if let Affinity::Reg(rci) = lv.affinity { + let rc = self.reginfo.rc(rci); // TODO: Fall back to a top-level super-class. Sub-classes are only hints. - let regunit = regs.iter(regclass) + let reg = regs.iter(rc) .next() .expect("Out of registers for arguments"); - regs.take(regclass, regunit); - *locations.ensure(lv.value) = ValueLoc::Reg(regunit); + regs.take(rc, reg); + *locations.ensure(lv.value) = ValueLoc::Reg(reg); } } @@ -266,7 +288,7 @@ impl<'a> Context<'a> { /// /// Update `regs` to reflect the allocated registers after `inst`, including removing any dead /// or killed values from the set. - fn visit_inst(&self, + fn visit_inst(&mut self, inst: Inst, encoding: Encoding, _pos: &mut Cursor, @@ -274,95 +296,194 @@ impl<'a> Context<'a> { tracker: &mut LiveValueTracker, regs: &mut AllocatableSet, locations: &mut EntityMap) { - // First update the live value tracker with this instruction. - // Get lists of values that are killed and defined by `inst`. - let (_throughs, kills, defs) = tracker.process_inst(inst, dfg, self.liveness); + dbg!("Coloring [{}] {}", + self.encinfo.display(encoding), + dfg.display_inst(inst)); // Get the operand constraints for `inst` that we are trying to satisfy. let constraints = self.encinfo .operand_constraints(encoding) - .expect("Missing instruction encoding") - .clone(); + .expect("Missing instruction encoding"); + + // Program the solver with register constraints for the input side. + self.solver.reset(regs); + self.program_input_constraints(inst, constraints.ins, dfg, locations); + if self.solver.has_fixed_input_conflicts() { + self.divert_fixed_input_conflicts(tracker.live(), locations); + } + self.solver.inputs_done(); + + // Update the live value tracker with this instruction. + let (throughs, kills, defs) = tracker.process_inst(inst, dfg, self.liveness); // Get rid of the killed values. for lv in kills { - if let Affinity::Reg(rc_index) = lv.affinity { - let regclass = self.reginfo.rc(rc_index); - if let ValueLoc::Reg(regunit) = locations[lv.value] { - regs.free(regclass, regunit); - } + if let Affinity::Reg(rci) = lv.affinity { + self.solver + .add_kill(lv.value, + self.reginfo.rc(rci), + self.divert.reg(lv.value, locations)); } } - // Process the defined values with fixed constraints. - // TODO: Handle constraints on call return values. - assert_eq!(defs.len(), - constraints.outs.len(), - "Can't handle variable results"); - for (lv, opcst) in defs.iter().zip(constraints.outs) { - match lv.affinity { - // This value should go in a register. - Affinity::Reg(rc_index) => { - // The preferred register class is not a requirement. - let pref_rc = self.reginfo.rc(rc_index); - match opcst.kind { - ConstraintKind::Reg => { - // This is a standard register constraint. The preferred register class - // should have been computed as a subclass of the hard constraint of - // the def. - assert!(opcst.regclass.has_subclass(rc_index), - "{} preference {} is not compatible with the definition \ - constraint {}", - lv.value, - pref_rc.name, - opcst.regclass.name); - // Try to grab a register from the preferred class, but fall back to - // the actual constraint if we have to. - let regunit = regs.iter(pref_rc) - .next() - .or_else(|| regs.iter(opcst.regclass).next()) - .expect("Ran out of registers"); - regs.take(opcst.regclass, regunit); - *locations.ensure(lv.value) = ValueLoc::Reg(regunit); - } - ConstraintKind::Tied(arg_index) => { - // This def must use the same register as a fixed instruction argument. - let arg = dfg.inst_args(inst)[arg_index as usize]; - let loc = locations[arg]; - *locations.ensure(lv.value) = loc; - // Mark the reused register. It's not really clear if we support tied - // stack operands. We could do that for some Intel read-modify-write - // encodings. - if let ValueLoc::Reg(regunit) = loc { - // This is going to assert out unless the incoming value at - // `arg_index` was killed. Tied operands must be fixed to - // ensure that before running the coloring pass. - regs.take(opcst.regclass, regunit); - } - } - ConstraintKind::FixedReg(_regunit) => unimplemented!(), - ConstraintKind::Stack => { - panic!("{}:{} should be a stack value", lv.value, pref_rc.name) - } - } - } - Affinity::Stack => unimplemented!(), - Affinity::None => { - panic!("Encoded instruction defines {} with no affinity", lv.value) - } - } + // Program the fixed output constraints before the general defines. This allows us to + // detect conflicts between fixed outputs and tied operands where the input value hasn't + // been converted to a solver variable. + if constraints.fixed_outs { + self.program_fixed_output_constraints(inst, + constraints.outs, + defs, + throughs, + dfg, + locations); + } + self.program_output_constraints(inst, constraints.outs, defs, dfg, locations); + + // Finally, we've fully programmed the constraint solver. + // We expect a quick solution in most cases. + let mut output_regs = self.solver + .quick_solve() + .unwrap_or_else(|_| self.iterate_solution()); + + // Apply the solution to the defs. + for v in self.solver.vars().iter().filter(|&v| v.is_define) { + *locations.ensure(v.value) = ValueLoc::Reg(v.solution); } - // Get rid of the dead defs. + // Update `regs` for the next instruction, remove the dead defs. for lv in defs { if lv.endpoint == inst { - if let Affinity::Reg(rc_index) = lv.affinity { - let regclass = self.reginfo.rc(rc_index); - if let ValueLoc::Reg(regunit) = locations[lv.value] { - regs.free(regclass, regunit); + if let Affinity::Reg(rci) = lv.affinity { + let rc = self.reginfo.rc(rci); + let reg = self.divert.reg(lv.value, locations); + output_regs.free(rc, reg); + } + } + } + *regs = output_regs; + } + + /// Program the input-side constraints for `inst` into the constraint solver. + fn program_input_constraints(&mut self, + inst: Inst, + constraints: &[OperandConstraint], + dfg: &mut DataFlowGraph, + locations: &mut EntityMap) { + for (op, &value) in constraints + .iter() + .zip(dfg.inst_args(inst)) + .filter(|&(op, _)| op.kind != ConstraintKind::Stack) { + // Reload pass is supposed to ensure that all arguments to register operands are + // already in a register. + let cur_reg = self.divert.reg(value, locations); + match op.kind { + ConstraintKind::FixedReg(regunit) => { + if regunit != cur_reg { + self.solver + .reassign_in(value, op.regclass, cur_reg, regunit); } } + ConstraintKind::Reg | + ConstraintKind::Tied(_) => { + if !op.regclass.contains(cur_reg) { + self.solver + .add_var(value, op.regclass, cur_reg, &self.reginfo); + } + } + ConstraintKind::Stack => unreachable!(), + } + } + } + + // Find existing live values that conflict with the fixed input register constraints programmed + // into the constraint solver. Convert them to solver variables so they can be diverted. + fn divert_fixed_input_conflicts(&mut self, + live: &[LiveValue], + locations: &mut EntityMap) { + for lv in live { + if let Affinity::Reg(rci) = lv.affinity { + let rc = self.reginfo.rc(rci); + let reg = self.divert.reg(lv.value, locations); + if self.solver.is_fixed_input_conflict(rc, reg) { + self.solver.add_var(lv.value, rc, reg, &self.reginfo); + } } } } + + /// Program any fixed-register output constraints into the solver. This may also detect + /// conflicts between live-through registers and fixed output registers. These live-through + /// values need to be turned into solver variables so they can be reassigned. + fn program_fixed_output_constraints(&mut self, + _inst: Inst, + constraints: &[OperandConstraint], + defs: &[LiveValue], + throughs: &[LiveValue], + _dfg: &mut DataFlowGraph, + locations: &mut EntityMap) { + for (op, lv) in constraints.iter().zip(defs) { + if let ConstraintKind::FixedReg(reg) = op.kind { + self.add_fixed_output(lv.value, op.regclass, reg, throughs, locations); + } + } + } + + /// Add a single fixed output value to the solver. + fn add_fixed_output(&mut self, + value: Value, + rc: RegClass, + reg: RegUnit, + throughs: &[LiveValue], + locations: &mut EntityMap) { + if !self.solver.add_fixed_output(rc, reg) { + // The fixed output conflicts with some of the live-through registers. + for lv in throughs { + if let Affinity::Reg(rci) = lv.affinity { + let rc2 = self.reginfo.rc(rci); + let reg2 = self.divert.reg(lv.value, locations); + if regs_overlap(rc, reg, rc2, reg2) { + // This live-through value is interfering with the fixed output assignment. + // Convert it to a solver variable. + // TODO: Use a looser constraint than the affinity hint. Any allocatable + // register in the top-level register class would be OK. Maybe `add_var` + // should take both a preferred class and a required constraint class. + self.solver.add_var(lv.value, rc2, reg2, &self.reginfo); + } + } + } + + let ok = self.solver.add_fixed_output(rc, reg); + assert!(ok, "Couldn't clear fixed output interference for {}", value); + } + *locations.ensure(value) = ValueLoc::Reg(reg); + } + + /// Program the output-side constraints for `inst` into the constraint solver. + /// + /// It is assumed that all fixed outputs have already been handled. + fn program_output_constraints(&mut self, + _inst: Inst, + constraints: &[OperandConstraint], + defs: &[LiveValue], + _dfg: &mut DataFlowGraph, + _locations: &mut EntityMap) { + for (op, lv) in constraints.iter().zip(defs) { + match op.kind { + ConstraintKind::FixedReg(_) | + ConstraintKind::Stack => continue, + ConstraintKind::Reg => { + self.solver.add_def(lv.value, op.regclass); + } + ConstraintKind::Tied(_) => unimplemented!(), + } + } + } + + /// Try harder to find a solution to the constraint problem since `quick_solve()` failed. + /// + /// We may need to move more registers around before a solution is possible. Use an iterative + /// algorithm that adds one more variable until a solution can be found. + fn iterate_solution(&self) -> AllocatableSet { + unimplemented!(); + } } diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index 9803c3af3f..fd2e78e9c3 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -11,6 +11,7 @@ pub mod coloring; mod affinity; mod context; mod diversion; +mod solver; pub use self::allocatable_set::AllocatableSet; pub use self::context::Context; diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs new file mode 100644 index 0000000000..05751c5155 --- /dev/null +++ b/lib/cretonne/src/regalloc/solver.rs @@ -0,0 +1,536 @@ +//! Constraint solver for register coloring. +//! +//! The coloring phase of SSA-based register allocation is very simple in theory, but in practice +//! it is complicated by the various constraints imposed by individual instructions: +//! +//! - Call and return instructions have to satisfy ABI requirements for arguments and return +//! values. +//! - Values live across a call must be in a callee-saved register. +//! - Some instructions have operand constraints such as register sub-classes, fixed registers, or +//! tied operands. +//! +//! # The instruction register coloring problem +//! +//! The constraint solver addresses the problem of satisfying the constraints of a single +//! instruction. We have: +//! +//! - A set of values that are live in registers before the instruction, with current register +//! assignments. Some are used by the instruction, some are not. +//! - A subset of the live register values that are killed by the instruction. +//! - A set of new register values that are defined by the instruction. +//! The constraint solver addresses the problem of satisfying the constraints of a single +//! instruction. We have: +//! +//! - A set of values that are live in registers before the instruction, with current register +//! assignments. Some are used by the instruction, some are not. +//! - A subset of the live register values that are killed by the instruction. +//! - A set of new register values that are defined by the instruction. +//! +//! We are not concerned with stack values at all. The reload pass ensures that all values required +//! to be in a register by the instruction are already in a register. +//! +//! A solution to the register coloring problem consists of: +//! +//! - Register reassignment prescriptions for a subset of the live register values. +//! - Register assignments for the defined values. +//! +//! The solution ensures that when live registers are reassigned as prescribed before the +//! instruction, all its operand constraints are satisfied, and the definition assignments won't +//! conflict. +//! +//! # Register diversions and global interference +//! +//! We can divert register values temporarily to satisfy constraints, but we need to put the +//! values back into their originally assigned register locations before leaving the EBB. +//! Otherwise, values won't be in the right register at the entry point of other EBBs. +//! +//! Some values are *local*, and we don't need to worry about putting those values back since they +//! are not used in any other EBBs. +//! +//! When we assign register locations to defines, we are assigning both the register used locally +//! immediately after the instruction and the register used globally when the defined value is used +//! in a different EBB. We need to avoid interference both locally at the instruction and globally. +//! +//! We have multiple mappings of values to registers: +//! +//! 1. The initial local mapping before the instruction. This includes any diversions from previous +//! instructions in the EBB, but not diversions for the current instruction. +//! 2. The local mapping after applying the additional reassignments required to satisfy the +//! constraints of the current instruction. +//! 3. The local mapping after the instruction. This excludes values killed by the instruction and +//! includes values defined by the instruction. +//! 4. The global mapping after the instruction. This mapping only contains values with global live +//! ranges, and it does not include any diversions. +//! +//! All four mappings must be kept free of interference. +//! +//! # Problems handled by previous passes. +//! +//! The constraint solver can only reassign registers, it can't create spill code, so some +//! constraints are handled by earlier passes: +//! +//! - There will be enough free registers available for the defines. Ensuring this is the primary +//! purpose of the spilling phase. +//! - When the same value is used for multiple operands, the intersection of operand constraints is +//! non-empty. The spilling phase will insert copies to handle mutually incompatible constraints, +//! such as when the same value is bound to two different function arguments. +//! - Values bound to tied operands must be killed by the instruction. Also enforced by the +//! spiller. +//! - Values used by register operands are in registers, and values used by stack operands are in +//! stack slots. This is enforced by the reload pass. +//! +//! # Solver algorithm +//! +//! The goal of the solver is to satisfy the instruction constraints with a minimal number of +//! register assignments before the instruction. +//! +//! 1. Compute the set of values used by operands with a fixed register constraint that isn't +//! already satisfied. These are mandatory predetermined reassignments. +//! 2. Compute the set of values that don't satisfy their register class constraint. These are +//! mandatory reassignments that we need to solve. +//! 3. Add the set of defines to the set of variables computed in 2. Exclude defines tied to an +//! input operand since their value is pre-determined. +//! +//! The set of values computed in 2. and 3. are the *variables* for the solver. Given a set of +//! variables, we can also compute a set of allocatable registers by removing the variables from +//! the set of assigned registers before the instruction. +//! +//! 1. For each variable, compute its domain as the intersection of the allocatable registers and +//! its register class constraint. +//! 2. Sort the variables in order of increasing domain size. +//! 3. Search for a solution that assigns each variable a register from its domain without +//! interference between variables. +//! +//! If the search fails to find a solution, we may need to reassign more registers. Find an +//! appropriate candidate among the set of live register values, add it as a variable and start +//! over. + +use ir::Value; +use isa::{RegInfo, RegClass, RegUnit}; +use regalloc::allocatable_set::RegSetIter; +use sparse_map::{SparseMap, SparseMapValue}; +use std::fmt; +use super::AllocatableSet; + +/// A variable in the constraint problem. +/// +/// Variables represent register values that can be assigned to any register unit within the +/// constraint register class. This includes live register values that can be reassigned to a new +/// register and values defined by the instruction which must be assigned to a register. +/// +/// Besides satisfying the register class constraint, variables must also be mutually +/// non-interfering in up to three contexts: +/// +/// 1. Input side live registers, after applying all the reassignments. +/// 2. Output side live registers, considering all the local register diversions. +/// 3. Global live register, not considering any local diversions. +/// +pub struct Variable { + /// The value whose register assignment we're looking for. + pub value: Value, + + /// Avoid interference on the input side. + is_input: bool, + + /// Avoid interference on the output side. + is_output: bool, + + /// Avoid interference with the global registers. + is_global: bool, + + /// The value is defined by the current instruction. + pub is_define: bool, + + /// Number of registers available in the domain of this variable. + domain: u16, + + /// The assigned register unit after a full solution was found. + pub solution: RegUnit, + + /// Any solution must belong to the constraint register class. + constraint: RegClass, +} + +impl Variable { + /// Get an iterator over possible register choices, given the available registers on the input + /// and output sides respectively. + fn iter(&self, iregs: &AllocatableSet, oregs: &AllocatableSet) -> RegSetIter { + if self.is_input && self.is_output { + let mut r = iregs.clone(); + r.intersect(oregs); + r.iter(self.constraint) + } else if self.is_input { + iregs.iter(self.constraint) + } else { + oregs.iter(self.constraint) + } + } +} + +impl fmt::Display for Variable { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{}({}", self.value, self.constraint)?; + if self.is_input { + write!(f, ", in")?; + } + if self.is_output { + write!(f, ", out")?; + } + if self.is_global { + write!(f, ", global")?; + } + if self.is_define { + write!(f, ", def")?; + } + if self.domain > 0 { + write!(f, ", {}", self.domain)?; + } + write!(f, ")") + } +} + + +struct Assignment { + value: Value, + from: RegUnit, + to: RegUnit, + rc: RegClass, +} + +impl SparseMapValue for Assignment { + fn key(&self) -> Value { + self.value + } +} + +/// Constraint solver for register allocation around a single instruction. +/// +/// Start by programming in the instruction constraints. +/// +/// 1. Initialize the solver by calling `reset()` with the set of allocatable registers before the +/// instruction. +/// 2. Program the input side constraints: Call `reassign_in()` for all fixed register constraints, +/// and `add_var()` for any input operands whose constraints are not already satisfied. +/// 3. Check for conflicts between fixed input assignments and existing live values by calling +/// `has_fixed_input_conflicts()`. Resolve any conflicts by calling `add_var()` with the +/// conflicting values. +/// 4. Prepare for adding output side constraints by calling `inputs_done()`. +/// 5. Add any killed register values that no longer cause interference on the output side by +/// calling `add_kill()`. +/// 6. Program the output side constraints: Call `add_fixed_output()` for all fixed register +/// constraints and `add_def()` for free defines. Resolve fixed output conflicts by calling +/// `add_var()`. +/// +pub struct Solver { + /// Register reassignments that are required or decided as part of a full solution. + assignments: SparseMap, + + /// Variables are the values that should be reassigned as part of a solution. + /// Values with a fixed register constraints are not considered variables. They are represented + /// in the `assignments` vector if necessary. + vars: Vec, + + /// Are we finished adding input-side constraints? This changes the meaning of the `regs_in` + /// and `regs_out` register sets. + inputs_done: bool, + + /// Available registers on the input side of the instruction. + /// + /// While we're adding input constraints (`!inputs_done`): + /// + /// - Live values on the input side are marked as unavailable. + /// - The 'from' registers of fixed input reassignments are marked as available as they are + /// added. + /// - Input-side variables are marked as available. + /// + /// After finishing input constraints (`inputs_done`): + /// + /// - Live values on the input side are marked as unavailable. + /// - The 'to' registers of fixed input reassignments are marked as unavailable. + /// - Input-side variables are marked as available. + /// + regs_in: AllocatableSet, + + /// Available registers on the output side of the instruction / fixed input scratch space. + /// + /// While we're adding input constraints (`!inputs_done`): + /// + /// - The 'to' registers of fixed input reassignments are marked as unavailable. + /// + /// After finishing input constraints (`inputs_done`): + /// + /// - Live-through values are marked as unavailable. + /// - Fixed output assignments are marked as unavailable. + /// - Live-through variables are marked as available. + /// + regs_out: AllocatableSet, +} + +/// Interface for programming the constraints into the solver. +impl Solver { + /// Create a new empty solver. + pub fn new() -> Solver { + Solver { + assignments: SparseMap::new(), + vars: Vec::new(), + inputs_done: false, + regs_in: AllocatableSet::new(), + regs_out: AllocatableSet::new(), + } + } + + /// Reset the solver state and prepare solving for a new instruction with an initial set of + /// allocatable registers. + /// + /// The `regs` set is the allocatable registers before any reassignments are applied. + pub fn reset(&mut self, regs: &AllocatableSet) { + self.assignments.clear(); + self.vars.clear(); + self.inputs_done = false; + self.regs_in = regs.clone(); + // Used for tracking fixed input assignments while `!inputs_done`: + self.regs_out = AllocatableSet::new(); + } + + /// Add a fixed input reassignment of `value`. + /// + /// This means that `value` must be assigned to `to` and can't become a variable. Call with + /// `from == to` to ensure that `value` is not reassigned from its existing register location. + /// + /// In either case, `to` will not be available for variables on the input side of the + /// instruction. + pub fn reassign_in(&mut self, value: Value, rc: RegClass, from: RegUnit, to: RegUnit) { + debug_assert!(!self.inputs_done); + if self.regs_in.is_avail(rc, from) { + // It looks like `value` was already removed from the register set. It must have been + // added as a variable previously. A fixed constraint beats a variable, so convert it. + if let Some(idx) = self.vars.iter().position(|v| v.value == value) { + let v = self.vars.remove(idx); + dbg!("Converting variable {} to a fixed constraint", v); + // The spiller is responsible for ensuring that all constraints on the uses of a + // value are compatible. + assert!(v.constraint.contains(to), + "Incompatible constraints for {}", + value); + } else { + panic!("Invalid from register for fixed {} constraint", value); + } + } + self.regs_in.free(rc, from); + self.regs_out.take(rc, to); + self.assignments + .insert(Assignment { + value, + rc, + from, + to, + }); + } + + /// Add a variable representing an input side value with an existing register assignment. + /// + /// A variable is a value that should be reassigned to something in the `constraint` register + /// class. + /// + /// It is assumed initially that the value is also live on the output side of the instruction. + /// This can be changed by calling to `add_kill()`. + pub fn add_var(&mut self, + value: Value, + constraint: RegClass, + from: RegUnit, + reginfo: &RegInfo) { + // Check for existing entries for this value. + if self.regs_in.is_avail(constraint, from) { + // There cold be an existing variable entry. + if let Some(v) = self.vars.iter_mut().find(|v| v.value == value) { + // We have an existing variable entry for `value`. Combine the constraints. + if let Some(rci) = v.constraint.intersect(constraint) { + v.constraint = reginfo.rc(rci); + return; + } else { + // The spiller should have made sure the same value is not used with disjoint + // constraints. + panic!("Incompatible constraints: {} + {}", constraint, *v) + } + } + + // No variable, then it must be a fixed reassignment. + if let Some(a) = self.assignments.get(value) { + assert!(constraint.contains(a.to), + "Incompatible constraints for {}", + value); + return; + } + + panic!("Wrong from register for {}", value); + } + self.regs_in.free(constraint, from); + if self.inputs_done { + self.regs_out.free(constraint, from); + } + self.vars + .push(Variable { + value, + constraint, + is_input: true, + is_output: true, + is_global: false, + is_define: false, + domain: 0, + solution: !0, + }); + } + + /// Check for conflicts between fixed input assignments and existing live values. + /// + /// Returns true if one of the live values conflicts with a fixed input assignment. Such a + /// conflicting value must be turned into a variable. + pub fn has_fixed_input_conflicts(&self) -> bool { + debug_assert!(!self.inputs_done); + // The `from` side of the fixed input diversions are taken from `regs_out`. + self.regs_out.interferes_with(&self.regs_in) + } + + /// Check if `rc, reg` specifically conflicts with the fixed input assignments. + pub fn is_fixed_input_conflict(&self, rc: RegClass, reg: RegUnit) -> bool { + debug_assert!(!self.inputs_done); + !self.regs_out.is_avail(rc, reg) + } + + /// Finish adding input side constraints. + /// + /// Call this method to indicate that there will be no more fixed input reassignments added + /// and prepare for the output side constraints. + pub fn inputs_done(&mut self) { + assert!(!self.has_fixed_input_conflicts()); + + // At this point, `regs_out` contains the `to` side of the input reassignments, and the + // `from` side has already been marked as available in `regs_in`. + // + // Remove the `to` assignments from `regs_in` so it now indicates the registers available + // to variables at the input side. + self.regs_in.intersect(&self.regs_out); + + // The meaning of `regs_out` now changes completely to indicate the registers available to + // variables on the output side. + // The initial mask will be modified by `add_kill()` and `add_fixed_output()`. + self.regs_out = self.regs_in.clone(); + + // Now we can't add more fixed input assignments, but `add_var()` is still allowed. + self.inputs_done = true; + } + + /// Record that an input register value is killed by the instruction. + /// + /// Even if a fixed reassignment has been added for the value, the `reg` argument should be the + /// original location before the reassignments. + /// + /// This means that the register is available on the output side. + pub fn add_kill(&mut self, value: Value, rc: RegClass, reg: RegUnit) { + debug_assert!(self.inputs_done); + + // If a fixed assignment is killed, the `to` register becomes available on the output side. + if let Some(a) = self.assignments.get(value) { + debug_assert_eq!(a.from, reg); + self.regs_out.free(a.rc, a.to); + return; + } + + // It's also possible that a variable is killed. That means it doesn't need to satisfy + // interference constraints on the output side. + // Variables representing tied operands will get their `is_output` flag set again later. + if let Some(v) = self.vars.iter_mut().find(|v| v.value == value) { + assert!(v.is_input); + v.is_output = false; + return; + } + + // Alright, this is just a boring value being killed by the instruction. Just reclaim + // the assigned register. + self.regs_out.free(rc, reg); + } + + /// Add a fixed output assignment. + /// + /// This means that `to` will not be available for variables on the output side of the + /// instruction. + /// + /// Returns `false` if a live value conflicts with `to`, so it couldn't be added. Find the + /// conflicting live-through value and turn it into a variable before calling this method + /// again. + #[allow(dead_code)] + pub fn add_fixed_output(&mut self, rc: RegClass, reg: RegUnit) -> bool { + debug_assert!(self.inputs_done); + if self.regs_out.is_avail(rc, reg) { + self.regs_out.take(rc, reg); + true + } else { + false + } + } + + /// Add a defined output value. + /// + /// This is similar to `add_var`, except the value doesn't have a prior register assignment. + pub fn add_def(&mut self, value: Value, constraint: RegClass) { + debug_assert!(self.inputs_done); + self.vars + .push(Variable { + value, + constraint, + is_input: false, + is_output: true, + is_global: false, + is_define: true, + domain: 0, + solution: !0, + }); + } +} + +/// Interface for searching for a solution. +impl Solver { + /// Try a quick-and-dirty solution. + /// + /// This is expected to succeed for most instructions since the constraint problem is almost + /// always trivial. + /// + /// Returns `true` is a solution was found. + pub fn quick_solve(&mut self) -> Result { + self.find_solution() + } + + /// Search for a solution with the current list of variables. + /// + /// If a solution was found, returns `Ok(regs)` with the set of available registers on the + /// output side after the solution. If no solution could be found, returns `Err(rc)` with the + /// constraint register class that needs more available registers. + fn find_solution(&mut self) -> Result { + // Available registers on the input and output sides respectively. + let mut iregs = self.regs_in.clone(); + let mut oregs = self.regs_out.clone(); + + for v in &mut self.vars { + let rc = v.constraint; + let reg = match v.iter(&iregs, &oregs).next() { + None => return Err(rc), + Some(reg) => reg, + }; + + v.solution = reg; + if v.is_input { + iregs.take(rc, reg); + } + if v.is_output { + oregs.take(rc, reg); + } + } + + Ok(oregs) + } + + /// Get all the variables. + pub fn vars(&self) -> &[Variable] { + &self.vars + } +} From ad9a942f5266a76eef3d458f1915f6dfb699f1c8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 10 May 2017 14:58:14 -0700 Subject: [PATCH 0749/3084] Use write! in utility code, rather than calling write_function directly. --- cranelift/src/filetest/legalizer.rs | 5 +++-- cranelift/src/filetest/regalloc.rs | 5 +++-- cranelift/src/utils.rs | 4 ++-- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/cranelift/src/filetest/legalizer.rs b/cranelift/src/filetest/legalizer.rs index 8fd7f45b16..aa6385c223 100644 --- a/cranelift/src/filetest/legalizer.rs +++ b/cranelift/src/filetest/legalizer.rs @@ -4,10 +4,11 @@ //! the result to filecheck. use std::borrow::Cow; -use cretonne::{self, write_function}; +use cretonne::{self}; use cretonne::ir::Function; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result, run_filecheck}; +use std::fmt::Write; use utils::pretty_error; struct TestLegalizer; @@ -45,7 +46,7 @@ impl SubTest for TestLegalizer { .map_err(|e| pretty_error(&comp_ctx.func, e))?; let mut text = String::new(); - write_function(&mut text, &comp_ctx.func, Some(isa)) + write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))) .map_err(|e| e.to_string())?; run_filecheck(&text, context) } diff --git a/cranelift/src/filetest/regalloc.rs b/cranelift/src/filetest/regalloc.rs index 79e7ad8e0f..4173b7c3d8 100644 --- a/cranelift/src/filetest/regalloc.rs +++ b/cranelift/src/filetest/regalloc.rs @@ -6,10 +6,11 @@ //! The resulting function is sent to `filecheck`. use cretonne::ir::Function; -use cretonne::{self, write_function}; +use cretonne::{self}; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result, run_filecheck}; use std::borrow::Cow; +use std::fmt::Write; use utils::pretty_error; struct TestRegalloc; @@ -53,7 +54,7 @@ impl SubTest for TestRegalloc { .map_err(|e| pretty_error(&comp_ctx.func, e))?; let mut text = String::new(); - write_function(&mut text, &comp_ctx.func, Some(isa)) + write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))) .map_err(|e| e.to_string())?; run_filecheck(&text, context) } diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index 1bcba4891a..b637f7ca49 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -1,7 +1,7 @@ //! Utility functions. use cretonne::ir::entities::AnyEntity; -use cretonne::{ir, verifier, write_function}; +use cretonne::{ir, verifier}; use cretonne::result::CtonError; use std::fmt::Write; use std::fs::File; @@ -42,7 +42,7 @@ pub fn pretty_verifier_error(func: &ir::Function, err: verifier::Error) -> Strin } _ => msg.push('\n'), } - write_function(&mut msg, func, None).unwrap(); + write!(msg, "{}", func).unwrap(); msg } From fb9d86f8522ea5f872ba0ffdb8f5a768819984ed Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 11 May 2017 06:27:13 -0700 Subject: [PATCH 0750/3084] Fix rustfmt diffs. --- cranelift/src/filetest/legalizer.rs | 2 +- cranelift/src/filetest/regalloc.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/cranelift/src/filetest/legalizer.rs b/cranelift/src/filetest/legalizer.rs index aa6385c223..806730b4ed 100644 --- a/cranelift/src/filetest/legalizer.rs +++ b/cranelift/src/filetest/legalizer.rs @@ -4,7 +4,7 @@ //! the result to filecheck. use std::borrow::Cow; -use cretonne::{self}; +use cretonne; use cretonne::ir::Function; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result, run_filecheck}; diff --git a/cranelift/src/filetest/regalloc.rs b/cranelift/src/filetest/regalloc.rs index 4173b7c3d8..1699d384a8 100644 --- a/cranelift/src/filetest/regalloc.rs +++ b/cranelift/src/filetest/regalloc.rs @@ -6,7 +6,7 @@ //! The resulting function is sent to `filecheck`. use cretonne::ir::Function; -use cretonne::{self}; +use cretonne; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result, run_filecheck}; use std::borrow::Cow; From 38fa75459e4318806565170f0192e9747fcb05d2 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 11 May 2017 10:21:59 -0700 Subject: [PATCH 0751/3084] Check for unknown instruction attributes. (#80) * Check for unknown instruction attributes. * Avoid has_key, at flake8's advice. * Use AssertionError instead of RuntimeError, per review request. --- lib/cretonne/meta/cdsl/instructions.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index a900c95ab6..6724b775ea 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -122,6 +122,10 @@ class Instruction(object): i for i, o in enumerate(self.ins) if o.is_immediate()) self._verify_polymorphic() + for attr in kwargs: + if attr not in Instruction.ATTRIBS: + raise AssertionError( + "unknown instruction attribute '" + attr + "'") for attr in Instruction.ATTRIBS: setattr(self, attr, not not kwargs.get(attr, False)) InstructionGroup.append(self) From 2c0d4136b0f87beb407256f4cb505d5f321bbe31 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 11 May 2017 11:03:18 -0700 Subject: [PATCH 0752/3084] Propagate a few more LiveRange properties to LiveValue. The live value tracker goes through the trouble of looking up the live range for each value it tracks. We can cache a few more interesting properties from the live range in the LiveValue struct. --- .../src/regalloc/live_value_tracker.rs | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 4ef7df8d3a..76e2fcb0d7 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -11,6 +11,7 @@ use ir::{Inst, Ebb, Value, DataFlowGraph, ProgramOrder, ExpandedProgramPoint}; use partition_slice::partition_slice; use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; +use regalloc::liverange::LiveRange; use std::collections::HashMap; @@ -45,6 +46,12 @@ pub struct LiveValue { /// This value is simply a copy of the affinity stored in the live range. We copy it because /// almost all users of `LiveValue` need to look at it. pub affinity: Affinity, + + /// The live range for this value never leaves its EBB. + pub is_local: bool, + + /// This value is dead - the live range ends immediately. + pub is_dead: bool, } struct LiveValueVec { @@ -66,13 +73,15 @@ impl LiveValueVec { } } - /// Add a new live value to `values`. - fn push(&mut self, value: Value, endpoint: Inst, affinity: Affinity) { + /// Add a new live value to `values`. Copy some properties from `lr`. + fn push(&mut self, value: Value, endpoint: Inst, lr: &LiveRange) { self.values .push(LiveValue { value, endpoint, - affinity, + affinity: lr.affinity, + is_local: lr.is_local(), + is_dead: lr.is_dead(), }); } @@ -175,7 +184,7 @@ impl LiveValueTracker { // Check if this value is live-in here. if let Some(endpoint) = lr.livein_local_end(ebb, program_order) { - self.live.push(value, endpoint, lr.affinity); + self.live.push(value, endpoint, lr); } } } @@ -189,7 +198,7 @@ impl LiveValueTracker { assert_eq!(lr.def(), ebb.into()); match lr.def_local_end().into() { ExpandedProgramPoint::Inst(endpoint) => { - self.live.push(value, endpoint, lr.affinity); + self.live.push(value, endpoint, lr); } ExpandedProgramPoint::Ebb(local_ebb) => { // This is a dead EBB argument which is not even live into the first @@ -248,7 +257,7 @@ impl LiveValueTracker { assert_eq!(lr.def(), inst.into()); match lr.def_local_end().into() { ExpandedProgramPoint::Inst(endpoint) => { - self.live.push(value, endpoint, lr.affinity); + self.live.push(value, endpoint, lr); } ExpandedProgramPoint::Ebb(ebb) => { panic!("Instruction result live range can't end at {}", ebb); From 94944d20576759deadb6415d80e95e4a346bc2b6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 11 May 2017 11:25:35 -0700 Subject: [PATCH 0753/3084] Keep dead EBB arguments around in LiveValueTracker::ebb_top(). Provide a drop_dead_args() function which deletes them instead. We still need to assign a register to dead EBB arguments, so they can't just be ignored. --- lib/cretonne/src/ir/layout.rs | 5 +++ lib/cretonne/src/regalloc/coloring.rs | 13 +++++- .../src/regalloc/live_value_tracker.rs | 40 ++++++++++++++----- 3 files changed, 46 insertions(+), 12 deletions(-) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index b8ef83e75b..abf0f3f482 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -435,6 +435,11 @@ impl Layout { self.assign_inst_seq(inst); } + /// Fetch an ebb's first instruction. + pub fn first_inst(&self, ebb: Ebb) -> Option { + self.ebbs[ebb].first_inst.into() + } + /// Fetch an ebb's last instruction. pub fn last_inst(&self, ebb: Ebb) -> Option { self.ebbs[ebb].last_inst.into() diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 871d86e757..6c0c906f90 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -129,6 +129,7 @@ impl<'a> Context<'a> { fn visit_ebb(&mut self, ebb: Ebb, func: &mut Function, tracker: &mut LiveValueTracker) { dbg!("Coloring {}:", ebb); let mut regs = self.visit_ebb_header(ebb, func, tracker); + tracker.drop_dead_args(); self.divert.clear(); // Now go through the instructions in `ebb` and color the values they define. @@ -162,14 +163,24 @@ impl<'a> Context<'a> { tracker.ebb_top(ebb, &func.dfg, self.liveness, &func.layout, self.domtree); // Arguments to the entry block have ABI constraints. - if func.layout.entry_block() == Some(ebb) { + let mut regs = if func.layout.entry_block() == Some(ebb) { assert_eq!(liveins.len(), 0); self.color_entry_args(&func.signature, args, &mut func.locations) } else { // The live-ins have already been assigned a register. Reconstruct the allocatable set. let regs = self.livein_regs(liveins, func); self.color_args(args, regs, &mut func.locations) + }; + + // Now forget about the dead arguments. + for lv in args.iter().filter(|&lv| lv.is_dead) { + if let Affinity::Reg(rci) = lv.affinity { + let rc = self.reginfo.rc(rci); + let reg = func.locations[lv.value].unwrap_reg(); + regs.free(rc, reg); + } } + regs } /// Initialize a set of allocatable registers from the values that are live-in to a block. diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 76e2fcb0d7..a5ee8197e8 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -7,7 +7,7 @@ use dominator_tree::DominatorTree; use entity_list::{EntityList, ListPool}; use ir::instructions::BranchInfo; -use ir::{Inst, Ebb, Value, DataFlowGraph, ProgramOrder, ExpandedProgramPoint}; +use ir::{Inst, Ebb, Value, DataFlowGraph, Layout, ExpandedProgramPoint}; use partition_slice::partition_slice; use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; @@ -116,6 +116,12 @@ impl LiveValueVec { let keep = self.live_after(next_inst); self.values.truncate(keep); } + + /// Remove any dead values. + fn remove_dead_values(&mut self) { + self.values.retain(|v| !v.is_dead); + self.live_prefix = None; + } } impl LiveValueTracker { @@ -150,14 +156,15 @@ impl LiveValueTracker { /// /// Returns `(liveins, args)` as a pair of slices. The first slice is the set of live-in values /// from the immediate dominator. The second slice is the set of `ebb` arguments that are live. - /// Dead arguments with no uses are ignored and not added to the set. - pub fn ebb_top(&mut self, - ebb: Ebb, - dfg: &DataFlowGraph, - liveness: &Liveness, - program_order: &PO, - domtree: &DominatorTree) - -> (&[LiveValue], &[LiveValue]) { + /// + /// Dead arguments with no uses are included in `args`. Call `drop_dead_args()` to remove them. + pub fn ebb_top(&mut self, + ebb: Ebb, + dfg: &DataFlowGraph, + liveness: &Liveness, + layout: &Layout, + domtree: &DominatorTree) + -> (&[LiveValue], &[LiveValue]) { // Start over, compute the set of live values at the top of the EBB from two sources: // // 1. Values that were live before `ebb`'s immediate dominator, filtered for those that are @@ -183,7 +190,7 @@ impl LiveValueTracker { .expect("Immediate dominator value has no live range"); // Check if this value is live-in here. - if let Some(endpoint) = lr.livein_local_end(ebb, program_order) { + if let Some(endpoint) = lr.livein_local_end(ebb, layout) { self.live.push(value, endpoint, lr); } } @@ -202,10 +209,14 @@ impl LiveValueTracker { } ExpandedProgramPoint::Ebb(local_ebb) => { // This is a dead EBB argument which is not even live into the first - // instruction in the EBB. We can ignore it. + // instruction in the EBB. assert_eq!(local_ebb, ebb, "EBB argument live range ends at wrong EBB header"); + // Give this value a fake endpoint that is the first instruction in the EBB. + // We expect it to be removed by calling `drop_dead_args()`. + self.live + .push(value, layout.first_inst(ebb).expect("Empty EBB"), lr); } } } @@ -281,6 +292,13 @@ impl LiveValueTracker { self.live.remove_kill_values(inst); } + /// Drop any values that are marked as `is_dead`. + /// + /// Use this after calling `ebb_top` to clean out dead EBB arguments. + pub fn drop_dead_args(&mut self) { + self.live.remove_dead_values(); + } + /// Save the current set of live values so it is associated with `idom`. fn save_idom_live_set(&mut self, idom: Inst) { let values = self.live.values.iter().map(|lv| lv.value); From d1390006b18fa65b710bfbe037463b5fa1fda581 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 11 May 2017 11:42:44 -0700 Subject: [PATCH 0754/3084] Always create live ranges for dead EBB arguments. The live value tracker expects them to be there. We may eventually delete dead arguments from internal EBBs, but at least the entry block needs to be able to handle dead function arguments. --- cranelift/filetests/regalloc/basic.cton | 7 +++++++ lib/cretonne/src/regalloc/liveness.rs | 7 +++++++ 2 files changed, 14 insertions(+) diff --git a/cranelift/filetests/regalloc/basic.cton b/cranelift/filetests/regalloc/basic.cton index 14fa263ac0..9db7b7563a 100644 --- a/cranelift/filetests/regalloc/basic.cton +++ b/cranelift/filetests/regalloc/basic.cton @@ -10,3 +10,10 @@ ebb0(v1: i32, v2: i32): ; sameln: iadd return } + +; Function with a dead argument. +function dead_arg(i32, i32) -> i32{ +ebb0(v1: i32, v2: i32): +; check: return $v1 + return v1 +} diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 737470b2d1..f2c9944d4b 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -311,6 +311,13 @@ impl Liveness { // elimination pass if we visit a post-order of the dominator tree? // TODO: Resolve value aliases while we're visiting instructions? for ebb in func.layout.ebbs() { + // Make sure we have created live ranges for dead EBB arguments. + // TODO: If these arguments are really dead, we could remove them, except for the entry + // block which must match the function signature. + for &arg in func.dfg.ebb_args(ebb) { + get_or_create(&mut self.ranges, arg, isa, func, &enc_info); + } + for inst in func.layout.ebb_insts(ebb) { // Make sure we have created live ranges for dead defs. // TODO: When we implement DCE, we can use the absence of a live range to indicate From 21b4c3a00bc2679e06d3e61051b16642225cc41f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 11 May 2017 11:48:58 -0700 Subject: [PATCH 0755/3084] Simplify the dead EBB argument tracking. It is not necessary to to a second pass over the live values to update the set of available registers. The color_args() and color_entry_args() functions can do that in a single pass. --- lib/cretonne/src/regalloc/coloring.rs | 28 +++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 6c0c906f90..521b1c9cba 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -163,24 +163,14 @@ impl<'a> Context<'a> { tracker.ebb_top(ebb, &func.dfg, self.liveness, &func.layout, self.domtree); // Arguments to the entry block have ABI constraints. - let mut regs = if func.layout.entry_block() == Some(ebb) { + if func.layout.entry_block() == Some(ebb) { assert_eq!(liveins.len(), 0); self.color_entry_args(&func.signature, args, &mut func.locations) } else { // The live-ins have already been assigned a register. Reconstruct the allocatable set. let regs = self.livein_regs(liveins, func); self.color_args(args, regs, &mut func.locations) - }; - - // Now forget about the dead arguments. - for lv in args.iter().filter(|&lv| lv.is_dead) { - if let Affinity::Reg(rci) = lv.affinity { - let rc = self.reginfo.rc(rci); - let reg = func.locations[lv.value].unwrap_reg(); - regs.free(rc, reg); - } } - regs } /// Initialize a set of allocatable registers from the values that are live-in to a block. @@ -221,7 +211,7 @@ impl<'a> Context<'a> { /// These are function arguments that should already have assigned register units in the /// function signature. /// - /// Return the set of remaining allocatable registers. + /// Return the set of remaining allocatable registers after filtering out the dead arguments. fn color_entry_args(&self, sig: &Signature, args: &[LiveValue], @@ -236,7 +226,9 @@ impl<'a> Context<'a> { Affinity::Reg(rci) => { let rc = self.reginfo.rc(rci); if let ArgumentLoc::Reg(reg) = abi.location { - regs.take(rc, reg); + if !lv.is_dead { + regs.take(rc, reg); + } *locations.ensure(lv.value) = ValueLoc::Reg(reg); } else { // This should have been fixed by the reload pass. @@ -278,6 +270,9 @@ impl<'a> Context<'a> { mut regs: AllocatableSet, locations: &mut EntityMap) -> AllocatableSet { + // Available registers *after* filtering out the dead arguments. + let mut live_regs = regs.clone(); + for lv in args { // Only look at the register arguments. if let Affinity::Reg(rci) = lv.affinity { @@ -287,11 +282,16 @@ impl<'a> Context<'a> { .next() .expect("Out of registers for arguments"); regs.take(rc, reg); + if !lv.is_dead { + live_regs.take(rc, reg); + } *locations.ensure(lv.value) = ValueLoc::Reg(reg); } } - regs + // All arguments are accounted for in `regs`. We don't care about the dead arguments now + // that we have made sure they don't interfere. + live_regs } /// Color the values defined by `inst` and insert any necessary shuffle code to satisfy From b521254149a9d655814cf01caf47ed50c9c3cf96 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 10 May 2017 12:31:59 -0700 Subject: [PATCH 0756/3084] Solver variables keep track of where they came from. The register constraint solver has two kinds of variables: 1. Live values that were already in a register, and 2. Values defined by the instruction. Make a record of the original register holding the first kind of value. --- lib/cretonne/src/regalloc/coloring.rs | 2 +- lib/cretonne/src/regalloc/solver.rs | 63 ++++++++++++++++----------- 2 files changed, 39 insertions(+), 26 deletions(-) diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 521b1c9cba..d8d3571ad6 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -357,7 +357,7 @@ impl<'a> Context<'a> { .unwrap_or_else(|_| self.iterate_solution()); // Apply the solution to the defs. - for v in self.solver.vars().iter().filter(|&v| v.is_define) { + for v in self.solver.vars().iter().filter(|&v| v.is_define()) { *locations.ensure(v.value) = ValueLoc::Reg(v.solution); } diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 05751c5155..0001169e96 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -129,6 +129,10 @@ pub struct Variable { /// The value whose register assignment we're looking for. pub value: Value, + /// Original register unit holding this live value before the instruction, or `None` for a + /// value that is defined by the instruction. + from: Option, + /// Avoid interference on the input side. is_input: bool, @@ -138,9 +142,6 @@ pub struct Variable { /// Avoid interference with the global registers. is_global: bool, - /// The value is defined by the current instruction. - pub is_define: bool, - /// Number of registers available in the domain of this variable. domain: u16, @@ -152,6 +153,37 @@ pub struct Variable { } impl Variable { + fn new_live(value: Value, constraint: RegClass, from: RegUnit) -> Variable { + Variable { + value, + constraint, + from: Some(from), + is_input: true, + is_output: true, + is_global: false, + domain: 0, + solution: !0, + } + } + + fn new_def(value: Value, constraint: RegClass) -> Variable { + Variable { + value, + constraint, + from: None, + is_input: false, + is_output: true, + is_global: false, + domain: 0, + solution: !0, + } + } + + /// Does this variable represent a value defined by the current instruction? + pub fn is_define(&self) -> bool { + self.from.is_none() + } + /// Get an iterator over possible register choices, given the available registers on the input /// and output sides respectively. fn iter(&self, iregs: &AllocatableSet, oregs: &AllocatableSet) -> RegSetIter { @@ -179,7 +211,7 @@ impl fmt::Display for Variable { if self.is_global { write!(f, ", global")?; } - if self.is_define { + if self.is_define() { write!(f, ", def")?; } if self.domain > 0 { @@ -369,16 +401,7 @@ impl Solver { self.regs_out.free(constraint, from); } self.vars - .push(Variable { - value, - constraint, - is_input: true, - is_output: true, - is_global: false, - is_define: false, - domain: 0, - solution: !0, - }); + .push(Variable::new_live(value, constraint, from)); } /// Check for conflicts between fixed input assignments and existing live values. @@ -474,17 +497,7 @@ impl Solver { /// This is similar to `add_var`, except the value doesn't have a prior register assignment. pub fn add_def(&mut self, value: Value, constraint: RegClass) { debug_assert!(self.inputs_done); - self.vars - .push(Variable { - value, - constraint, - is_input: false, - is_output: true, - is_global: false, - is_define: true, - domain: 0, - solution: !0, - }); + self.vars.push(Variable::new_def(value, constraint)); } } From 9be262e8780bf91911d50b6dcf5e24481db7955f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 10 May 2017 15:36:14 -0700 Subject: [PATCH 0757/3084] Implement a move resolver for the register constraint solver. After finding a register solution, it need to be executed as a sequence of regmove instructions. This often requires a topological ordering of the moves so they don't conflict. When the solution contains cycles, try to grab an available scratch register to implement the copies. Panic if that fails (later, we'll implement emergency spilling in this case). Make sure we handle odd aliasing in the arm32 floating point register bank. Not everything is a simple cycle in that case, so make sure we don't assume so. --- lib/cretonne/src/isa/registers.rs | 6 + lib/cretonne/src/regalloc/coloring.rs | 27 ++- lib/cretonne/src/regalloc/solver.rs | 277 +++++++++++++++++++++++++- 3 files changed, 302 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 122dec8e71..91a25595f1 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -164,6 +164,12 @@ impl fmt::Display for RegClassData { } } +impl fmt::Debug for RegClassData { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + f.write_str(self.name) + } +} + /// A small reference to a register class. /// /// Use this when storing register classes in compact data structures. The `RegInfo::rc()` method diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index d8d3571ad6..b2f1863538 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -34,6 +34,7 @@ use entity_map::EntityMap; use dominator_tree::DominatorTree; use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph, Signature, ArgumentLoc}; +use ir::InstBuilder; use isa::{TargetIsa, Encoding, EncInfo, OperandConstraint, ConstraintKind}; use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; use regalloc::affinity::Affinity; @@ -302,7 +303,7 @@ impl<'a> Context<'a> { fn visit_inst(&mut self, inst: Inst, encoding: Encoding, - _pos: &mut Cursor, + pos: &mut Cursor, dfg: &mut DataFlowGraph, tracker: &mut LiveValueTracker, regs: &mut AllocatableSet, @@ -356,6 +357,11 @@ impl<'a> Context<'a> { .quick_solve() .unwrap_or_else(|_| self.iterate_solution()); + + // The solution and/or fixed input constraints may require us to shuffle the set of live + // registers around. + self.shuffle_inputs(pos, dfg, regs); + // Apply the solution to the defs. for v in self.solver.vars().iter().filter(|&v| v.is_define()) { *locations.ensure(v.value) = ValueLoc::Reg(v.solution); @@ -497,4 +503,23 @@ impl<'a> Context<'a> { fn iterate_solution(&self) -> AllocatableSet { unimplemented!(); } + + /// Emit `regmove` instructions as needed to move the live registers into place before the + /// instruction. Also update `self.divert` accordingly. + /// + /// The `pos` cursor is expected to point at the instruction. The register moves are inserted + /// before. + /// + /// The solver needs to be reminded of the available registers before any moves are inserted. + fn shuffle_inputs(&mut self, + pos: &mut Cursor, + dfg: &mut DataFlowGraph, + regs: &mut AllocatableSet) { + self.solver.schedule_moves(regs); + + for m in self.solver.moves() { + self.divert.regmove(m.value, m.from, m.to); + dfg.ins(pos).regmove(m.value, m.from, m.to); + } + } } diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 0001169e96..5e3c43da5f 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -221,12 +221,12 @@ impl fmt::Display for Variable { } } - -struct Assignment { - value: Value, - from: RegUnit, - to: RegUnit, - rc: RegClass, +#[derive(Clone, Debug)] +pub struct Assignment { + pub value: Value, + pub from: RegUnit, + pub to: RegUnit, + pub rc: RegClass, } impl SparseMapValue for Assignment { @@ -235,6 +235,14 @@ impl SparseMapValue for Assignment { } } +#[cfg(test)] +impl PartialEq for Assignment { + fn eq(&self, other: &Assignment) -> bool { + self.value == other.value && self.from == other.from && self.to == other.to && + self.rc.index == other.rc.index + } +} + /// Constraint solver for register allocation around a single instruction. /// /// Start by programming in the instruction constraints. @@ -296,6 +304,11 @@ pub struct Solver { /// - Live-through variables are marked as available. /// regs_out: AllocatableSet, + + /// List of register moves scheduled to avoid conflicts. + /// + /// This is used as working space by the `schedule_moves()` function. + moves: Vec, } /// Interface for programming the constraints into the solver. @@ -308,6 +321,7 @@ impl Solver { inputs_done: false, regs_in: AllocatableSet::new(), regs_out: AllocatableSet::new(), + moves: Vec::new(), } } @@ -508,7 +522,7 @@ impl Solver { /// This is expected to succeed for most instructions since the constraint problem is almost /// always trivial. /// - /// Returns `true` is a solution was found. + /// Returns `Ok(regs)` if a solution was found. pub fn quick_solve(&mut self) -> Result { self.find_solution() } @@ -547,3 +561,252 @@ impl Solver { &self.vars } } + +/// Interface for working with parallel copies once a solution has been found. +impl Solver { + /// Collect all the register moves we need to execute. + fn collect_moves(&mut self) { + self.moves.clear(); + + // Collect moves from the chosen solution for all non-define variables. + for v in &self.vars { + if let Some(from) = v.from { + self.moves + .push(Assignment { + value: v.value, + from, + to: v.solution, + rc: v.constraint, + }); + } + } + + self.moves.extend(self.assignments.values().cloned()); + } + + /// Try to schedule a sequence of `regmove` instructions that will shuffle registers into + /// place. + /// + /// This may require the use of additional available registers, and it can fail if no + /// additional registers are available. + /// + /// TODO: Handle failure by generating a sequence of register swaps, or by temporarily spilling + /// a register. + /// + /// Returns the number of spills that had to be emitted. + pub fn schedule_moves(&mut self, regs: &AllocatableSet) -> usize { + self.collect_moves(); + + let mut avail = regs.clone(); + let mut i = 0; + while i < self.moves.len() { + // Find the first move that can be executed now. + if let Some(j) = self.moves[i..] + .iter() + .position(|m| avail.is_avail(m.rc, m.to)) { + // This move can be executed now. + self.moves.swap(i, i + j); + let m = &self.moves[i]; + avail.take(m.rc, m.to); + avail.free(m.rc, m.from); + i += 1; + continue; + } + + // When we get here, non of the `moves[i..]` can be executed. This means there are only + // cycles remaining. The cycles can be broken in a few ways: + // + // 1. Grab an available register and use it to break a cycle. + // 2. Move a value temporarily into a stack slot instead of a register. + // 3. Use swap instructions. + // + // TODO: So far we only implement 1. + + // Pick an assignment with the largest possible width. This is more likely to break up + // a cycle than an assignment with fewer register units. For example, it may be + // necessary to move two arm32 S-registers out of the way before a D-register can move + // into place. + // + // We use `min_by_key` and `!` instead of `max_by_key` because it preserves the + // existing order of moves with the same width. + let j = self.moves[i..] + .iter() + .enumerate() + .min_by_key(|&(_, m)| !m.rc.width) + .unwrap() + .0; + self.moves.swap(i, i + j); + + let m = self.moves[i].clone(); + if let Some(reg) = avail.iter(m.rc).next() { + // Alter the move so it is guaranteed to be picked up when we loop. It is important + // that this move is scheduled immediately, otherwise we would have multiple moves + // of the same value, and they would not be commutable. + self.moves[i].to = reg; + // Append a fixup move so we end up in the right place. This move will be scheduled + // later. That's ok because it is the single remaining move of `m.value` after the + // next iteration. + self.moves + .push(Assignment { + value: m.value, + rc: m.rc, + from: reg, + to: m.to, + }); + // TODO: What if allocating an extra register is not enough to break a cycle? This + // can happen when there are registers of different widths in a cycle. For ARM, we + // may have to move two S-registers out of the way before we can resolve a cycle + // involving a D-register. + } else { + panic!("Not enough registers in {} to schedule moves", m.rc); + } + } + + // Spilling not implemented yet. + 0 + } + + /// Borrow the scheduled set of register moves that was computed by `schedule_moves()`. + pub fn moves(&self) -> &[Assignment] { + &self.moves + } +} + +#[cfg(test)] +mod tests { + use entity_map::EntityRef; + use ir::Value; + use isa::{TargetIsa, RegClass, RegUnit}; + use regalloc::AllocatableSet; + use std::borrow::Borrow; + use super::{Solver, Assignment}; + + // Make an arm32 `TargetIsa`, if possible. + fn arm32() -> Option> { + use settings; + use isa; + + let shared_builder = settings::builder(); + let shared_flags = settings::Flags::new(&shared_builder); + + isa::lookup("arm32").map(|b| b.finish(shared_flags)) + } + + // Get a register class by name. + fn rc_by_name(isa: &TargetIsa, name: &str) -> RegClass { + isa.register_info() + .classes + .iter() + .find(|rc| rc.name == name) + .expect("Can't find named register class.") + } + + // Construct a move. + fn mov(value: Value, rc: RegClass, from: RegUnit, to: RegUnit) -> Assignment { + Assignment { + value, + rc, + from, + to, + } + } + + #[test] + fn simple_moves() { + let isa = arm32().expect("This test requires arm32 support"); + let isa = isa.borrow(); + let gpr = rc_by_name(isa, "GPR"); + let r0 = gpr.unit(0); + let r1 = gpr.unit(1); + let r2 = gpr.unit(2); + let mut regs = AllocatableSet::new(); + let mut solver = Solver::new(); + let v10 = Value::new(10); + let v11 = Value::new(11); + + // As simple as it gets: Value is in r1, we want r0. + regs.take(gpr, r1); + solver.reset(®s); + solver.reassign_in(v10, gpr, r1, r0); + solver.inputs_done(); + assert!(solver.quick_solve().is_ok()); + assert_eq!(solver.schedule_moves(®s), 0); + assert_eq!(solver.moves(), &[mov(v10, gpr, r1, r0)]); + + // A bit harder: r0, r1 need to go in r1, r2. + regs.take(gpr, r0); + solver.reset(®s); + solver.reassign_in(v10, gpr, r0, r1); + solver.reassign_in(v11, gpr, r1, r2); + solver.inputs_done(); + assert!(solver.quick_solve().is_ok()); + assert_eq!(solver.schedule_moves(®s), 0); + assert_eq!(solver.moves(), + &[mov(v11, gpr, r1, r2), mov(v10, gpr, r0, r1)]); + + // Swap r0 and r1 in three moves using r2 as a scratch. + solver.reset(®s); + solver.reassign_in(v10, gpr, r0, r1); + solver.reassign_in(v11, gpr, r1, r0); + solver.inputs_done(); + assert!(solver.quick_solve().is_ok()); + assert_eq!(solver.schedule_moves(®s), 0); + assert_eq!(solver.moves(), + &[mov(v10, gpr, r0, r2), + mov(v11, gpr, r1, r0), + mov(v10, gpr, r2, r1)]); + } + + #[test] + fn harder_move_cycles() { + let isa = arm32().expect("This test requires arm32 support"); + let isa = isa.borrow(); + let s = rc_by_name(isa, "S"); + let d = rc_by_name(isa, "D"); + let d0 = d.unit(0); + let d1 = d.unit(1); + let d2 = d.unit(2); + let s0 = s.unit(0); + let s1 = s.unit(1); + let s2 = s.unit(2); + let s3 = s.unit(3); + let mut regs = AllocatableSet::new(); + let mut solver = Solver::new(); + let v10 = Value::new(10); + let v11 = Value::new(11); + let v12 = Value::new(12); + + // Not a simple cycle: Swap d0 <-> (s2, s3) + regs.take(d, d0); + regs.take(d, d1); + solver.reset(®s); + solver.reassign_in(v10, d, d0, d1); + solver.reassign_in(v11, s, s2, s0); + solver.reassign_in(v12, s, s3, s1); + solver.inputs_done(); + assert!(solver.quick_solve().is_ok()); + assert_eq!(solver.schedule_moves(®s), 0); + assert_eq!(solver.moves(), + &[mov(v10, d, d0, d2), + mov(v11, s, s2, s0), + mov(v12, s, s3, s1), + mov(v10, d, d2, d1)]); + + // Same problem in the other direction: Swap (s0, s1) <-> d1. + // + // If we divert the moves in order, we will need to allocate *two* temporary S registers. A + // trivial algorithm might assume that allocating a single temp is enough. + solver.reset(®s); + solver.reassign_in(v11, s, s0, s2); + solver.reassign_in(v12, s, s1, s3); + solver.reassign_in(v10, d, d1, d0); + solver.inputs_done(); + assert!(solver.quick_solve().is_ok()); + assert_eq!(solver.schedule_moves(®s), 0); + assert_eq!(solver.moves(), + &[mov(v10, d, d1, d2), + mov(v12, s, s1, s3), + mov(v11, s, s0, s2), + mov(v10, d, d2, d0)]); + } +} From c713ea8c579ae067de4c3f04ac6f6cf8832d7624 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 11 May 2017 10:11:28 -0700 Subject: [PATCH 0758/3084] Add fixed constraints for ABI arguments and return values. We can start adding some real test cases for the move resolver now. --- cranelift/filetests/regalloc/basic.cton | 23 ++++++ lib/cretonne/src/regalloc/coloring.rs | 94 ++++++++++++++++++++----- 2 files changed, 98 insertions(+), 19 deletions(-) diff --git a/cranelift/filetests/regalloc/basic.cton b/cranelift/filetests/regalloc/basic.cton index 9db7b7563a..9046aeb711 100644 --- a/cranelift/filetests/regalloc/basic.cton +++ b/cranelift/filetests/regalloc/basic.cton @@ -3,6 +3,8 @@ test regalloc ; We can add more ISAs once they have defined encodings. isa riscv +; regex: RX=%x\d+ + function add(i32, i32) { ebb0(v1: i32, v2: i32): v3 = iadd v1, v2 @@ -14,6 +16,27 @@ ebb0(v1: i32, v2: i32): ; Function with a dead argument. function dead_arg(i32, i32) -> i32{ ebb0(v1: i32, v2: i32): +; not: regmove ; check: return $v1 return v1 } + +; Return a value from a different register. +function move1(i32, i32) -> i32 { +ebb0(v1: i32, v2: i32): +; not: regmove +; check: regmove $v2, %x11 -> %x10 +; nextln: return $v2 + return v2 +} + +; Swap two registers. +function swap(i32, i32) -> i32, i32 { +ebb0(v1: i32, v2: i32): +; not: regmove +; check: regmove $v2, %x11 -> $(tmp=$RX) +; nextln: regmove $v1, %x10 -> %x11 +; nextln: regmove $v2, $tmp -> %x10 +; nextln: return $v2, $v1 + return v2, v1 +} diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index b2f1863538..33013aa309 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -33,8 +33,8 @@ use entity_map::EntityMap; use dominator_tree::DominatorTree; -use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph, Signature, ArgumentLoc}; -use ir::InstBuilder; +use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph}; +use ir::{InstBuilder, Signature, ArgumentType, ArgumentLoc}; use isa::{TargetIsa, Encoding, EncInfo, OperandConstraint, ConstraintKind}; use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; use regalloc::affinity::Affinity; @@ -145,7 +145,8 @@ impl<'a> Context<'a> { &mut func.dfg, tracker, &mut regs, - &mut func.locations); + &mut func.locations, + &func.signature); tracker.drop_dead(inst); } @@ -307,7 +308,8 @@ impl<'a> Context<'a> { dfg: &mut DataFlowGraph, tracker: &mut LiveValueTracker, regs: &mut AllocatableSet, - locations: &mut EntityMap) { + locations: &mut EntityMap, + func_signature: &Signature) { dbg!("Coloring [{}] {}", self.encinfo.display(encoding), dfg.display_inst(inst)); @@ -320,6 +322,12 @@ impl<'a> Context<'a> { // Program the solver with register constraints for the input side. self.solver.reset(regs); self.program_input_constraints(inst, constraints.ins, dfg, locations); + let call_sig = dfg.call_signature(inst); + if let Some(sig) = call_sig { + self.program_input_abi(inst, &dfg.signatures[sig].argument_types, dfg, locations); + } else if dfg[inst].opcode().is_return() { + self.program_input_abi(inst, &func_signature.return_types, dfg, locations); + } if self.solver.has_fixed_input_conflicts() { self.divert_fixed_input_conflicts(tracker.live(), locations); } @@ -342,12 +350,11 @@ impl<'a> Context<'a> { // detect conflicts between fixed outputs and tied operands where the input value hasn't // been converted to a solver variable. if constraints.fixed_outs { - self.program_fixed_output_constraints(inst, - constraints.outs, - defs, - throughs, - dfg, - locations); + self.program_fixed_outputs(constraints.outs, defs, throughs, locations); + } + if let Some(sig) = call_sig { + let abi = &dfg.signatures[sig].return_types; + self.program_output_abi(abi, defs, throughs, locations); } self.program_output_constraints(inst, constraints.outs, defs, dfg, locations); @@ -384,8 +391,8 @@ impl<'a> Context<'a> { fn program_input_constraints(&mut self, inst: Inst, constraints: &[OperandConstraint], - dfg: &mut DataFlowGraph, - locations: &mut EntityMap) { + dfg: &DataFlowGraph, + locations: &EntityMap) { for (op, &value) in constraints .iter() .zip(dfg.inst_args(inst)) @@ -412,6 +419,33 @@ impl<'a> Context<'a> { } } + /// Program the input-side ABI constraints for `inst` into the constraint solver. + /// + /// ABI constraints are the fixed register assignments used for calls and returns. + fn program_input_abi(&mut self, + inst: Inst, + abi_types: &[ArgumentType], + dfg: &DataFlowGraph, + locations: &EntityMap) { + for (abi, &value) in abi_types.iter().zip(dfg.inst_variable_args(inst)) { + if let ArgumentLoc::Reg(reg) = abi.location { + let cur_reg = self.divert.reg(value, locations); + if reg != cur_reg { + if let Affinity::Reg(rci) = + self.liveness + .get(value) + .expect("ABI register must have live range") + .affinity { + let rc = self.reginfo.rc(rci); + self.solver.reassign_in(value, rc, cur_reg, reg); + } else { + panic!("ABI argument {} should be in a register", value); + } + } + } + } + } + // Find existing live values that conflict with the fixed input register constraints programmed // into the constraint solver. Convert them to solver variables so they can be diverted. fn divert_fixed_input_conflicts(&mut self, @@ -431,13 +465,11 @@ impl<'a> Context<'a> { /// Program any fixed-register output constraints into the solver. This may also detect /// conflicts between live-through registers and fixed output registers. These live-through /// values need to be turned into solver variables so they can be reassigned. - fn program_fixed_output_constraints(&mut self, - _inst: Inst, - constraints: &[OperandConstraint], - defs: &[LiveValue], - throughs: &[LiveValue], - _dfg: &mut DataFlowGraph, - locations: &mut EntityMap) { + fn program_fixed_outputs(&mut self, + constraints: &[OperandConstraint], + defs: &[LiveValue], + throughs: &[LiveValue], + locations: &mut EntityMap) { for (op, lv) in constraints.iter().zip(defs) { if let ConstraintKind::FixedReg(reg) = op.kind { self.add_fixed_output(lv.value, op.regclass, reg, throughs, locations); @@ -445,6 +477,30 @@ impl<'a> Context<'a> { } } + /// Program the output-side ABI constraints for `inst` into the constraint solver. + /// + /// That means return values for a call instruction. + fn program_output_abi(&mut self, + abi_types: &[ArgumentType], + defs: &[LiveValue], + throughs: &[LiveValue], + locations: &mut EntityMap) { + // It's technically possible for a call instruction to have fixed results before the + // variable list of results, but we have no known instances of that. + // Just assume all results are variable return values. + assert_eq!(defs.len(), abi_types.len()); + for (abi, lv) in abi_types.iter().zip(defs) { + if let ArgumentLoc::Reg(reg) = abi.location { + if let Affinity::Reg(rci) = lv.affinity { + let rc = self.reginfo.rc(rci); + self.add_fixed_output(lv.value, rc, reg, throughs, locations); + } else { + panic!("ABI argument {} should be in a register", lv.value); + } + } + } + } + /// Add a single fixed output value to the solver. fn add_fixed_output(&mut self, value: Value, From 6fd95c1158d2babe326694201766fa84d6dbffcd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 11 May 2017 17:58:11 -0700 Subject: [PATCH 0759/3084] Implement reloc_names() for all targets. This gets rid of the last TargetIsa method with a default implementation. --- lib/cretonne/src/isa/arm32/binemit.rs | 2 ++ lib/cretonne/src/isa/arm32/mod.rs | 4 ++++ lib/cretonne/src/isa/arm64/binemit.rs | 2 ++ lib/cretonne/src/isa/arm64/mod.rs | 4 ++++ lib/cretonne/src/isa/mod.rs | 4 +--- 5 files changed, 13 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/isa/arm32/binemit.rs b/lib/cretonne/src/isa/arm32/binemit.rs index dcc76331a2..cf12bdbde2 100644 --- a/lib/cretonne/src/isa/arm32/binemit.rs +++ b/lib/cretonne/src/isa/arm32/binemit.rs @@ -4,3 +4,5 @@ use binemit::{CodeSink, bad_encoding}; use ir::{Function, Inst}; include!(concat!(env!("OUT_DIR"), "/binemit-arm32.rs")); + +pub static RELOC_NAMES: [&'static str; 1] = ["Call"]; diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index e3128fdc0e..5657c7523b 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -94,4 +94,8 @@ impl TargetIsa for Isa { fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } + + fn reloc_names(&self) -> &'static [&'static str] { + &binemit::RELOC_NAMES + } } diff --git a/lib/cretonne/src/isa/arm64/binemit.rs b/lib/cretonne/src/isa/arm64/binemit.rs index 13885307d4..120115c0d8 100644 --- a/lib/cretonne/src/isa/arm64/binemit.rs +++ b/lib/cretonne/src/isa/arm64/binemit.rs @@ -4,3 +4,5 @@ use binemit::{CodeSink, bad_encoding}; use ir::{Function, Inst}; include!(concat!(env!("OUT_DIR"), "/binemit-arm64.rs")); + +pub static RELOC_NAMES: [&'static str; 1] = ["Call"]; diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index f6fd70fb7a..fe81ea8c09 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -87,4 +87,8 @@ impl TargetIsa for Isa { fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { binemit::emit_inst(func, inst, sink) } + + fn reloc_names(&self) -> &'static [&'static str] { + &binemit::RELOC_NAMES + } } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 34d3e21ef6..edc0034217 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -210,7 +210,5 @@ pub trait TargetIsa { /// /// This array can be indexed by the contents of `binemit::Reloc` objects passed to a /// `CodeSink`. - fn reloc_names(&self) -> &'static [&'static str] { - unimplemented!() - } + fn reloc_names(&self) -> &'static [&'static str]; } From 31b7330b8d7621f67043fd2ef6f95bdda8cdee68 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 12 May 2017 15:31:08 -0700 Subject: [PATCH 0760/3084] Return the first applicable encoding from general_encoding(). We'll arrange encoding lists such that the first suitable encoding is the best choice for the legalizer. This is the most intuitive way of generating the encodings. After register allocation, we may choose a different encoding, but that will require looking at the whole list. --- lib/cretonne/src/isa/enc_tables.rs | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index cfb0066b8c..d7dde71eed 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -114,11 +114,11 @@ const CODE_ALWAYS: EncListEntry = PRED_MASK; /// The encoding list terminator. const CODE_FAIL: EncListEntry = 0xffff; -/// Find the most general encoding of `inst`. +/// Find the first applicable general encoding of `inst`. /// /// Given an encoding list offset as returned by `lookup_enclist` above, search the encoding list -/// for the most general encoding that applies to `inst`. The encoding lists are laid out such that -/// this is the last valid entry in the list. +/// for the most first encoding that applies to `inst`. The encoding lists are laid out such that +/// this is the first valid entry in the list. /// /// This function takes two closures that are used to evaluate predicates: /// - `instp` is passed an instruction predicate number to be evaluated on the current instruction. @@ -133,14 +133,13 @@ pub fn general_encoding(offset: usize, where InstP: Fn(EncListEntry) -> bool, IsaP: Fn(EncListEntry) -> bool { - let mut found = None; let mut pos = offset; while enclist[pos] != CODE_FAIL { let pred = enclist[pos]; if pred <= CODE_ALWAYS { // This is an instruction predicate followed by recipe and encbits entries. if pred == CODE_ALWAYS || instp(pred) { - found = Some(Encoding::new(enclist[pos + 1], enclist[pos + 2])) + return Some(Encoding::new(enclist[pos + 1], enclist[pos + 2])); } pos += 3; } else { @@ -152,5 +151,6 @@ pub fn general_encoding(offset: usize, } } } - found + + None } From f4929825ca71f453a0ff318396884e7d248a327a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 12 May 2017 10:35:18 -0700 Subject: [PATCH 0761/3084] Add subtract and logical instruction encodings to Intel-32. Also add versions with 8-bit and 32-bit immediate operands. --- cranelift/filetests/isa/intel/binary32.cton | 70 +++++++++++++++++++-- lib/cretonne/meta/isa/intel/encodings.py | 16 ++++- lib/cretonne/meta/isa/intel/recipes.py | 14 ++++- lib/cretonne/src/isa/intel/binemit.rs | 24 +++++++ lib/cretonne/src/isa/intel/enc_tables.rs | 3 +- 5 files changed, 117 insertions(+), 10 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index d609a0c030..4509507005 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -18,21 +18,79 @@ ebb0: [-,%rcx] v10 = iadd v1, v2 ; bin: 01 f1 ; asm: addl %ecx, %esi [-,%rsi] v11 = iadd v2, v1 ; bin: 01 ce + ; asm: subl %esi, %ecx + [-,%rcx] v12 = isub v1, v2 ; bin: 29 f1 + ; asm: subl %ecx, %esi + [-,%rsi] v13 = isub v2, v1 ; bin: 29 ce + + ; asm: andl %esi, %ecx + [-,%rcx] v14 = band v1, v2 ; bin: 21 f1 + ; asm: andl %ecx, %esi + [-,%rsi] v15 = band v2, v1 ; bin: 21 ce + ; asm: orl %esi, %ecx + [-,%rcx] v16 = bor v1, v2 ; bin: 09 f1 + ; asm: orl %ecx, %esi + [-,%rsi] v17 = bor v2, v1 ; bin: 09 ce + ; asm: xorl %esi, %ecx + [-,%rcx] v18 = bxor v1, v2 ; bin: 31 f1 + ; asm: xorl %ecx, %esi + [-,%rsi] v19 = bxor v2, v1 ; bin: 31 ce ; Dynamic shifts take the shift amount in %rcx. ; asm: shll %cl, %esi - [-,%rsi] v12 = ishl v2, v1 ; bin: d3 e6 + [-,%rsi] v20 = ishl v2, v1 ; bin: d3 e6 ; asm: shll %cl, %ecx - [-,%rcx] v13 = ishl v1, v1 ; bin: d3 e1 + [-,%rcx] v21 = ishl v1, v1 ; bin: d3 e1 ; asm: shrl %cl, %esi - [-,%rsi] v14 = ushr v2, v1 ; bin: d3 ee + [-,%rsi] v22 = ushr v2, v1 ; bin: d3 ee ; asm: shrl %cl, %ecx - [-,%rcx] v15 = ushr v1, v1 ; bin: d3 e9 + [-,%rcx] v23 = ushr v1, v1 ; bin: d3 e9 ; asm: sarl %cl, %esi - [-,%rsi] v16 = sshr v2, v1 ; bin: d3 fe + [-,%rsi] v24 = sshr v2, v1 ; bin: d3 fe ; asm: sarl %cl, %ecx - [-,%rcx] v17 = sshr v1, v1 ; bin: d3 f9 + [-,%rcx] v25 = sshr v1, v1 ; bin: d3 f9 + + ; Integer Register - Immediate 8-bit operations. + ; The 8-bit immediate is sign-extended. + + ; asm: addl $-128, %ecx + [-,%rcx] v30 = iadd_imm v1, -128 ; bin: 83 c1 80 + ; asm: addl $10, %esi + [-,%rsi] v31 = iadd_imm v2, 10 ; bin: 83 c6 0a + + ; asm: andl $-128, %ecx + [-,%rcx] v32 = band_imm v1, -128 ; bin: 83 e1 80 + ; asm: andl $10, %esi + [-,%rsi] v33 = band_imm v2, 10 ; bin: 83 e6 0a + ; asm: orl $-128, %ecx + [-,%rcx] v34 = bor_imm v1, -128 ; bin: 83 c9 80 + ; asm: orl $10, %esi + [-,%rsi] v35 = bor_imm v2, 10 ; bin: 83 ce 0a + ; asm: xorl $-128, %ecx + [-,%rcx] v36 = bxor_imm v1, -128 ; bin: 83 f1 80 + ; asm: xorl $10, %esi + [-,%rsi] v37 = bxor_imm v2, 10 ; bin: 83 f6 0a + + ; Integer Register - Immediate 32-bit operations. + + ; asm: addl $-128000, %ecx + [-,%rcx] v40 = iadd_imm v1, -128000 ; bin: 81 c1 fffe0c00 + ; asm: addl $1000000, %esi + [-,%rsi] v41 = iadd_imm v2, 1000000 ; bin: 81 c6 000f4240 + + ; asm: andl $-128000, %ecx + [-,%rcx] v42 = band_imm v1, -128000 ; bin: 81 e1 fffe0c00 + ; asm: andl $1000000, %esi + [-,%rsi] v43 = band_imm v2, 1000000 ; bin: 81 e6 000f4240 + ; asm: orl $-128000, %ecx + [-,%rcx] v44 = bor_imm v1, -128000 ; bin: 81 c9 fffe0c00 + ; asm: orl $1000000, %esi + [-,%rsi] v45 = bor_imm v2, 1000000 ; bin: 81 ce 000f4240 + ; asm: xorl $-128000, %ecx + [-,%rcx] v46 = bxor_imm v1, -128000 ; bin: 81 f1 fffe0c00 + ; asm: xorl $1000000, %esi + [-,%rsi] v47 = bxor_imm v2, 1000000 ; bin: 81 f6 000f4240 return } diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 3896532c85..967da0d857 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -4,10 +4,24 @@ Intel Encodings. from __future__ import absolute_import from base import instructions as base from .defs import I32 -from .recipes import Op1rr, Op1rc +from .recipes import Op1rr, Op1rc, Op1rib, Op1rid from .recipes import OP I32.enc(base.iadd.i32, Op1rr, OP(0x01)) +I32.enc(base.isub.i32, Op1rr, OP(0x29)) + +I32.enc(base.band.i32, Op1rr, OP(0x21)) +I32.enc(base.bor.i32, Op1rr, OP(0x09)) +I32.enc(base.bxor.i32, Op1rr, OP(0x31)) + +# Immediate instructions with sign-extended 8-bit and 32-bit immediate. +for inst, r in [ + (base.iadd_imm.i32, 0), + (base.band_imm.i32, 4), + (base.bor_imm.i32, 1), + (base.bxor_imm.i32, 6)]: + I32.enc(inst, Op1rib, OP(0x83, rrr=r)) + I32.enc(inst, Op1rid, OP(0x81, rrr=r)) # 32-bit shifts and rotates. # Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 0ffd7e5c50..d4891fba27 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -3,8 +3,8 @@ Intel Encoding recipes. """ from __future__ import absolute_import from cdsl.isa import EncRecipe -# from cdsl.predicates import IsSignedInt -from base.formats import Binary +from cdsl.predicates import IsSignedInt +from base.formats import Binary, BinaryImm from .registers import GPR # Opcode representation. @@ -68,3 +68,13 @@ Op1rr = EncRecipe('Op1rr', Binary, size=2, ins=(GPR, GPR), outs=0) # XX /n with one arg in %rcx, for shifts. Op1rc = EncRecipe('Op1rc', Binary, size=2, ins=(GPR, GPR.rcx), outs=0) + +# XX /n ib with 8-bit immediate sign-extended. +Op1rib = EncRecipe( + 'Op1rib', BinaryImm, size=3, ins=GPR, outs=0, + instp=IsSignedInt(BinaryImm.imm, 8)) + +# XX /n id with 32-bit immediate sign-extended. +Op1rid = EncRecipe( + 'Op1rid', BinaryImm, size=6, ins=GPR, outs=0, + instp=IsSignedInt(BinaryImm.imm, 32)) diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 0644d583d3..75f8241e9c 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -53,3 +53,27 @@ fn recipe_op1rc(func: &Function, inst: Inst, sink: &mut C panic!("Expected Binary format: {:?}", func.dfg[inst]); } } + +fn recipe_op1rib(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::BinaryImm { arg, imm, .. } = func.dfg[inst] { + let bits = func.encodings[inst].bits(); + put_op1(bits, sink); + modrm_r_bits(func.locations[arg].unwrap_reg(), bits, sink); + let imm: i64 = imm.into(); + sink.put1(imm as u8); + } else { + panic!("Expected BinaryImm format: {:?}", func.dfg[inst]); + } +} + +fn recipe_op1rid(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::BinaryImm { arg, imm, .. } = func.dfg[inst] { + let bits = func.encodings[inst].bits(); + put_op1(bits, sink); + modrm_r_bits(func.locations[arg].unwrap_reg(), bits, sink); + let imm: i64 = imm.into(); + sink.put4(imm as u32); + } else { + panic!("Expected BinaryImm format: {:?}", func.dfg[inst]); + } +} diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 1ec93f00a9..e72da36b1e 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -1,11 +1,12 @@ //! Encoding tables for Intel ISAs. -use ir::{Opcode, InstructionData}; use ir::types; +use ir::{Opcode, InstructionData}; use isa::EncInfo; use isa::constraints::*; use isa::enc_tables::{Level1Entry, Level2Entry}; use isa::encoding::RecipeSizing; +use predicates; use super::registers::*; include!(concat!(env!("OUT_DIR"), "/encoding-intel.rs")); From bd8230411a8977c4495bee915bbd980173286344 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 12 May 2017 14:26:44 -0700 Subject: [PATCH 0762/3084] Encodings for load/store instructions. We don't support the full set of Intel addressing modes yet. So far we have: - Register indirect, no displacement. - Register indirect, 8-bit signed displacement. - Register indirect, 32-bit signed displacement. The SIB addressing modes will need new Cretonne instruction formats to represent. --- cranelift/filetests/isa/intel/binary32.cton | 105 ++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 57 +++-- lib/cretonne/meta/isa/intel/recipes.py | 111 ++++++++-- lib/cretonne/src/isa/intel/binemit.rs | 220 ++++++++++++++++++++ lib/cretonne/src/predicates.rs | 4 +- 5 files changed, 470 insertions(+), 27 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 4509507005..5cac489f77 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -92,5 +92,110 @@ ebb0: ; asm: xorl $1000000, %esi [-,%rsi] v47 = bxor_imm v2, 1000000 ; bin: 81 f6 000f4240 + ; Load/Store instructions. + + ; Register indirect addressing with no displacement. + + ; asm: movl %ecx, (%esi) + store v1, v2 ; bin: 89 0e + ; asm: movl %esi, (%ecx) + store v2, v1 ; bin: 89 31 + ; asm: movw %cx, (%esi) + istore16 v1, v2 ; bin: 66 89 0e + ; asm: movw %si, (%ecx) + istore16 v2, v1 ; bin: 66 89 31 + ; asm: movb %cl, (%esi) + istore8 v1, v2 ; bin: 88 0e + ; Can't store %sil in 32-bit mode (needs REX prefix). + + ; asm: movl (%ecx), %edi + [-,%rdi] v100 = load.i32 v1 ; bin: 8b 39 + ; asm: movl (%esi), %edx + [-,%rdx] v101 = load.i32 v2 ; bin: 8b 16 + ; asm: movzwl (%ecx), %edi + [-,%rdi] v102 = uload16.i32 v1 ; bin: 0f b7 39 + ; asm: movzwl (%esi), %edx + [-,%rdx] v103 = uload16.i32 v2 ; bin: 0f b7 16 + ; asm: movswl (%ecx), %edi + [-,%rdi] v104 = sload16.i32 v1 ; bin: 0f bf 39 + ; asm: movswl (%esi), %edx + [-,%rdx] v105 = sload16.i32 v2 ; bin: 0f bf 16 + ; asm: movzbl (%ecx), %edi + [-,%rdi] v106 = uload8.i32 v1 ; bin: 0f b6 39 + ; asm: movzbl (%esi), %edx + [-,%rdx] v107 = uload8.i32 v2 ; bin: 0f b6 16 + ; asm: movsbl (%ecx), %edi + [-,%rdi] v108 = sload8.i32 v1 ; bin: 0f be 39 + ; asm: movsbl (%esi), %edx + [-,%rdx] v109 = sload8.i32 v2 ; bin: 0f be 16 + + ; Register-indirect with 8-bit signed displacement. + + ; asm: movl %ecx, 100(%esi) + store v1, v2+100 ; bin: 89 4e 64 + ; asm: movl %esi, -100(%ecx) + store v2, v1-100 ; bin: 89 71 9c + ; asm: movw %cx, 100(%esi) + istore16 v1, v2+100 ; bin: 66 89 4e 64 + ; asm: movw %si, -100(%ecx) + istore16 v2, v1-100 ; bin: 66 89 71 9c + ; asm: movb %cl, 100(%esi) + istore8 v1, v2+100 ; bin: 88 4e 64 + + ; asm: movl 50(%ecx), %edi + [-,%rdi] v110 = load.i32 v1+50 ; bin: 8b 79 32 + ; asm: movl -50(%esi), %edx + [-,%rdx] v111 = load.i32 v2-50 ; bin: 8b 56 ce + ; asm: movzwl 50(%ecx), %edi + [-,%rdi] v112 = uload16.i32 v1+50 ; bin: 0f b7 79 32 + ; asm: movzwl -50(%esi), %edx + [-,%rdx] v113 = uload16.i32 v2-50 ; bin: 0f b7 56 ce + ; asm: movswl 50(%ecx), %edi + [-,%rdi] v114 = sload16.i32 v1+50 ; bin: 0f bf 79 32 + ; asm: movswl -50(%esi), %edx + [-,%rdx] v115 = sload16.i32 v2-50 ; bin: 0f bf 56 ce + ; asm: movzbl 50(%ecx), %edi + [-,%rdi] v116 = uload8.i32 v1+50 ; bin: 0f b6 79 32 + ; asm: movzbl -50(%esi), %edx + [-,%rdx] v117 = uload8.i32 v2-50 ; bin: 0f b6 56 ce + ; asm: movsbl 50(%ecx), %edi + [-,%rdi] v118 = sload8.i32 v1+50 ; bin: 0f be 79 32 + ; asm: movsbl -50(%esi), %edx + [-,%rdx] v119 = sload8.i32 v2-50 ; bin: 0f be 56 ce + + ; Register-indirect with 32-bit signed displacement. + + ; asm: movl %ecx, 10000(%esi) + store v1, v2+10000 ; bin: 89 8e 00002710 + ; asm: movl %esi, -10000(%ecx) + store v2, v1-10000 ; bin: 89 b1 ffffd8f0 + ; asm: movw %cx, 10000(%esi) + istore16 v1, v2+10000 ; bin: 66 89 8e 00002710 + ; asm: movw %si, -10000(%ecx) + istore16 v2, v1-10000 ; bin: 66 89 b1 ffffd8f0 + ; asm: movb %cl, 10000(%esi) + istore8 v1, v2+10000 ; bin: 88 8e 00002710 + + ; asm: movl 50000(%ecx), %edi + [-,%rdi] v120 = load.i32 v1+50000 ; bin: 8b b9 0000c350 + ; asm: movl -50000(%esi), %edx + [-,%rdx] v121 = load.i32 v2-50000 ; bin: 8b 96 ffff3cb0 + ; asm: movzwl 50000(%ecx), %edi + [-,%rdi] v122 = uload16.i32 v1+50000 ; bin: 0f b7 b9 0000c350 + ; asm: movzwl -50000(%esi), %edx + [-,%rdx] v123 = uload16.i32 v2-50000 ; bin: 0f b7 96 ffff3cb0 + ; asm: movswl 50000(%ecx), %edi + [-,%rdi] v124 = sload16.i32 v1+50000 ; bin: 0f bf b9 0000c350 + ; asm: movswl -50000(%esi), %edx + [-,%rdx] v125 = sload16.i32 v2-50000 ; bin: 0f bf 96 ffff3cb0 + ; asm: movzbl 50000(%ecx), %edi + [-,%rdi] v126 = uload8.i32 v1+50000 ; bin: 0f b6 b9 0000c350 + ; asm: movzbl -50000(%esi), %edx + [-,%rdx] v127 = uload8.i32 v2-50000 ; bin: 0f b6 96 ffff3cb0 + ; asm: movsbl 50000(%ecx), %edi + [-,%rdi] v128 = sload8.i32 v1+50000 ; bin: 0f be b9 0000c350 + ; asm: movsbl -50000(%esi), %edx + [-,%rdx] v129 = sload8.i32 v2-50000 ; bin: 0f be 96 ffff3cb0 + return } diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 967da0d857..8d6d466aa7 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -4,15 +4,15 @@ Intel Encodings. from __future__ import absolute_import from base import instructions as base from .defs import I32 -from .recipes import Op1rr, Op1rc, Op1rib, Op1rid -from .recipes import OP +from . import recipes as rcp +from .recipes import OP, OP0F, MP66 -I32.enc(base.iadd.i32, Op1rr, OP(0x01)) -I32.enc(base.isub.i32, Op1rr, OP(0x29)) +I32.enc(base.iadd.i32, rcp.Op1rr, OP(0x01)) +I32.enc(base.isub.i32, rcp.Op1rr, OP(0x29)) -I32.enc(base.band.i32, Op1rr, OP(0x21)) -I32.enc(base.bor.i32, Op1rr, OP(0x09)) -I32.enc(base.bxor.i32, Op1rr, OP(0x31)) +I32.enc(base.band.i32, rcp.Op1rr, OP(0x21)) +I32.enc(base.bor.i32, rcp.Op1rr, OP(0x09)) +I32.enc(base.bxor.i32, rcp.Op1rr, OP(0x31)) # Immediate instructions with sign-extended 8-bit and 32-bit immediate. for inst, r in [ @@ -20,12 +20,45 @@ for inst, r in [ (base.band_imm.i32, 4), (base.bor_imm.i32, 1), (base.bxor_imm.i32, 6)]: - I32.enc(inst, Op1rib, OP(0x83, rrr=r)) - I32.enc(inst, Op1rid, OP(0x81, rrr=r)) + I32.enc(inst, rcp.Op1rib, OP(0x83, rrr=r)) + I32.enc(inst, rcp.Op1rid, OP(0x81, rrr=r)) # 32-bit shifts and rotates. # Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit # and 16-bit shifts would need explicit masking. -I32.enc(base.ishl.i32.i32, Op1rc, OP(0xd3, rrr=4)) -I32.enc(base.ushr.i32.i32, Op1rc, OP(0xd3, rrr=5)) -I32.enc(base.sshr.i32.i32, Op1rc, OP(0xd3, rrr=7)) +I32.enc(base.ishl.i32.i32, rcp.Op1rc, OP(0xd3, rrr=4)) +I32.enc(base.ushr.i32.i32, rcp.Op1rc, OP(0xd3, rrr=5)) +I32.enc(base.sshr.i32.i32, rcp.Op1rc, OP(0xd3, rrr=7)) + +# Loads and stores. +I32.enc(base.store.i32.i32, rcp.Op1st, OP(0x89)) +I32.enc(base.store.i32.i32, rcp.Op1stDisp8, OP(0x89)) +I32.enc(base.store.i32.i32, rcp.Op1stDisp32, OP(0x89)) + +I32.enc(base.istore16.i32.i32, rcp.Mp1st, MP66(0x89)) +I32.enc(base.istore16.i32.i32, rcp.Mp1stDisp8, MP66(0x89)) +I32.enc(base.istore16.i32.i32, rcp.Mp1stDisp32, MP66(0x89)) + +I32.enc(base.istore8.i32.i32, rcp.Op1st_abcd, OP(0x88)) +I32.enc(base.istore8.i32.i32, rcp.Op1stDisp8_abcd, OP(0x88)) +I32.enc(base.istore8.i32.i32, rcp.Op1stDisp32_abcd, OP(0x88)) + +I32.enc(base.load.i32.i32, rcp.Op1ld, OP(0x8b)) +I32.enc(base.load.i32.i32, rcp.Op1ldDisp8, OP(0x8b)) +I32.enc(base.load.i32.i32, rcp.Op1ldDisp32, OP(0x8b)) + +I32.enc(base.uload16.i32.i32, rcp.Op2ld, OP0F(0xb7)) +I32.enc(base.uload16.i32.i32, rcp.Op2ldDisp8, OP0F(0xb7)) +I32.enc(base.uload16.i32.i32, rcp.Op2ldDisp32, OP0F(0xb7)) + +I32.enc(base.sload16.i32.i32, rcp.Op2ld, OP0F(0xbf)) +I32.enc(base.sload16.i32.i32, rcp.Op2ldDisp8, OP0F(0xbf)) +I32.enc(base.sload16.i32.i32, rcp.Op2ldDisp32, OP0F(0xbf)) + +I32.enc(base.uload8.i32.i32, rcp.Op2ld, OP0F(0xb6)) +I32.enc(base.uload8.i32.i32, rcp.Op2ldDisp8, OP0F(0xb6)) +I32.enc(base.uload8.i32.i32, rcp.Op2ldDisp32, OP0F(0xb6)) + +I32.enc(base.sload8.i32.i32, rcp.Op2ld, OP0F(0xbe)) +I32.enc(base.sload8.i32.i32, rcp.Op2ldDisp8, OP0F(0xbe)) +I32.enc(base.sload8.i32.i32, rcp.Op2ldDisp32, OP0F(0xbe)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index d4891fba27..6c72fe4690 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -3,9 +3,9 @@ Intel Encoding recipes. """ from __future__ import absolute_import from cdsl.isa import EncRecipe -from cdsl.predicates import IsSignedInt -from base.formats import Binary, BinaryImm -from .registers import GPR +from cdsl.predicates import IsSignedInt, IsEqual +from base.formats import Binary, BinaryImm, Store, Load +from .registers import GPR, ABCD # Opcode representation. # @@ -15,21 +15,21 @@ from .registers import GPR # name prefix: # # Op1* OP(op) -# 0F Op2* OP(op) +# 0F Op2* OP0F(op) # 0F 38 Op3* OP38(op) # 0F 3A Op3* OP3A(op) # 66 Mp1* MP66(op) -# 66 0F Mp2* MP66(op) -# 66 0F 38 Mp3* MP6638(op) -# 66 0F 3A Mp3* MP663A(op) +# 66 0F Mp2* MP660F(op) +# 66 0F 38 Mp3* MP660F38(op) +# 66 0F 3A Mp3* MP660F3A(op) # F2 Mp1* MPF2(op) -# F2 0F Mp2* MPF2(op) -# F2 0F 38 Mp3* MPF238(op) -# F2 0F 3A Mp3* MPF23A(op) +# F2 0F Mp2* MPF20F(op) +# F2 0F 38 Mp3* MPF20F38(op) +# F2 0F 3A Mp3* MPF20F3A(op) # F3 Mp1* MPF3(op) -# F3 0F Mp2* MPF3(op) -# F3 0F 38 Mp3* MPF338(op) -# F3 0F 3A Mp3* MPF33A(op) +# F3 0F Mp2* MP0FF3(op) +# F3 0F 38 Mp3* MPF30F38(op) +# F3 0F 3A Mp3* MPF30F3A(op) # # VEX/XOP and EVEX prefixes are not yet supported. # @@ -63,6 +63,16 @@ def OP(op, pp=0, mm=0, rrr=0, w=0): return op | (pp << 8) | (mm << 10) | (rrr << 12) | (w << 15) +def OP0F(op, rrr=0, w=0): + # type: (int, int, int) -> int + return OP(op, pp=0, mm=1, rrr=rrr, w=w) + + +def MP66(op, rrr=0, w=0): + # type: (int, int, int) -> int + return OP(op, pp=1, mm=0, rrr=rrr, w=w) + + # XX /r Op1rr = EncRecipe('Op1rr', Binary, size=2, ins=(GPR, GPR), outs=0) @@ -78,3 +88,78 @@ Op1rib = EncRecipe( Op1rid = EncRecipe( 'Op1rid', BinaryImm, size=6, ins=GPR, outs=0, instp=IsSignedInt(BinaryImm.imm, 32)) + +# +# Store recipes. +# + +# XX /r register-indirect store with no offset. +Op1st = EncRecipe( + 'Op1st', Store, size=2, ins=(GPR, GPR), outs=(), + instp=IsEqual(Store.offset, 0)) + +# XX /r register-indirect store with no offset. +# Only ABCD allowed for stored value. This is for byte stores. +Op1st_abcd = EncRecipe( + 'Op1st_abcd', Store, size=2, ins=(ABCD, GPR), outs=(), + instp=IsEqual(Store.offset, 0)) + +# XX /r register-indirect store with 8-bit offset. +Op1stDisp8 = EncRecipe( + 'Op1stDisp8', Store, size=3, ins=(GPR, GPR), outs=(), + instp=IsSignedInt(Store.offset, 8)) +Op1stDisp8_abcd = EncRecipe( + 'Op1stDisp8_abcd', Store, size=3, ins=(ABCD, GPR), outs=(), + instp=IsSignedInt(Store.offset, 8)) + +# XX /r register-indirect store with 32-bit offset. +Op1stDisp32 = EncRecipe('Op1stDisp32', Store, size=6, ins=(GPR, GPR), outs=()) +Op1stDisp32_abcd = EncRecipe( + 'Op1stDisp32_abcd', Store, size=6, ins=(ABCD, GPR), outs=()) + +# PP WW /r register-indirect store with no offset. +Mp1st = EncRecipe( + 'Mp1st', Store, size=3, ins=(GPR, GPR), outs=(), + instp=IsEqual(Store.offset, 0)) + +# PP XX /r register-indirect store with 8-bit offset. +Mp1stDisp8 = EncRecipe( + 'Mp1stDisp8', Store, size=4, ins=(GPR, GPR), outs=(), + instp=IsSignedInt(Store.offset, 8)) + +# PP XX /r register-indirect store with 32-bit offset. +Mp1stDisp32 = EncRecipe('Mp1stDisp32', Store, size=7, ins=(GPR, GPR), outs=()) + +# +# Load recipes +# + +# XX /r load with no offset. +Op1ld = EncRecipe( + 'Op1ld', Load, size=2, ins=(GPR), outs=(GPR), + instp=IsEqual(Load.offset, 0)) + +# XX /r load with 8-bit offset. +Op1ldDisp8 = EncRecipe( + 'Op1ldDisp8', Load, size=3, ins=(GPR), outs=(GPR), + instp=IsSignedInt(Load.offset, 8)) + +# XX /r load with 32-bit offset. +Op1ldDisp32 = EncRecipe( + 'Op1ldDisp32', Load, size=6, ins=(GPR), outs=(GPR), + instp=IsSignedInt(Load.offset, 32)) + +# 0F XX /r load with no offset. +Op2ld = EncRecipe( + 'Op2ld', Load, size=3, ins=(GPR), outs=(GPR), + instp=IsEqual(Load.offset, 0)) + +# XX /r load with 8-bit offset. +Op2ldDisp8 = EncRecipe( + 'Op2ldDisp8', Load, size=4, ins=(GPR), outs=(GPR), + instp=IsSignedInt(Load.offset, 8)) + +# XX /r load with 32-bit offset. +Op2ldDisp32 = EncRecipe( + 'Op2ldDisp32', Load, size=7, ins=(GPR), outs=(GPR), + instp=IsSignedInt(Load.offset, 32)) diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 75f8241e9c..834a9848c3 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -8,11 +8,30 @@ include!(concat!(env!("OUT_DIR"), "/binemit-intel.rs")); pub static RELOC_NAMES: [&'static str; 1] = ["Call"]; +// Emit single-byte opcode. fn put_op1(bits: u16, sink: &mut CS) { debug_assert!(bits & 0x0f00 == 0, "Invalid encoding bits for Op1*"); sink.put1(bits as u8); } +// Emit two-byte opcode: 0F XX +fn put_op2(bits: u16, sink: &mut CS) { + debug_assert!(bits & 0x0f00 == 0x0400, "Invalid encoding bits for Op2*"); + sink.put1(0x0f); + sink.put1(bits as u8); +} + +// Mandatory prefix bytes for Mp* opcodes. +const PREFIX: [u8; 3] = [0x66, 0xf3, 0xf2]; + +// Emit single-byte opcode with mandatory prefix. +fn put_mp1(bits: u16, sink: &mut CS) { + debug_assert!(bits & 0x0c00 == 0, "Invalid encoding bits for Mp1*"); + let pp = (bits >> 8) & 3; + sink.put1(PREFIX[(pp - 1) as usize]); + sink.put1(bits as u8); +} + /// Emit a ModR/M byte for reg-reg operands. fn modrm_rr(rm: RegUnit, reg: RegUnit, sink: &mut CS) { let reg = reg as u8 & 7; @@ -33,6 +52,42 @@ fn modrm_r_bits(rm: RegUnit, bits: u16, sink: &mut CS) { sink.put1(b); } +/// Emit a mode 00 ModR/M byte. This is a register-indirect addressing mode with no offset. +/// Registers %rsp and %rbp are invalid for `rm`, %rsp indicates a SIB byte, and %rbp indicates an +/// absolute immediate 32-bit address. +fn modrm_rm(rm: RegUnit, reg: RegUnit, sink: &mut CS) { + let reg = reg as u8 & 7; + let rm = rm as u8 & 7; + let mut b = 0b00000000; + b |= reg << 3; + b |= rm; + sink.put1(b); +} + +/// Emit a mode 01 ModR/M byte. This is a register-indirect addressing mode with 8-bit +/// displacement. +/// Register %rsp is invalid for `rm`. It indicates the presence of a SIB byte. +fn modrm_disp8(rm: RegUnit, reg: RegUnit, sink: &mut CS) { + let reg = reg as u8 & 7; + let rm = rm as u8 & 7; + let mut b = 0b01000000; + b |= reg << 3; + b |= rm; + sink.put1(b); +} + +/// Emit a mode 10 ModR/M byte. This is a register-indirect addressing mode with 32-bit +/// displacement. +/// Register %rsp is invalid for `rm`. It indicates the presence of a SIB byte. +fn modrm_disp32(rm: RegUnit, reg: RegUnit, sink: &mut CS) { + let reg = reg as u8 & 7; + let rm = rm as u8 & 7; + let mut b = 0b10000000; + b |= reg << 3; + b |= rm; + sink.put1(b); +} + fn recipe_op1rr(func: &Function, inst: Inst, sink: &mut CS) { if let InstructionData::Binary { args, .. } = func.dfg[inst] { put_op1(func.encodings[inst].bits(), sink); @@ -77,3 +132,168 @@ fn recipe_op1rid(func: &Function, inst: Inst, sink: &mut panic!("Expected BinaryImm format: {:?}", func.dfg[inst]); } } + +// Store recipes. + +fn recipe_op1st(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Store { args, .. } = func.dfg[inst] { + put_op1(func.encodings[inst].bits(), sink); + modrm_rm(func.locations[args[1]].unwrap_reg(), + func.locations[args[0]].unwrap_reg(), + sink); + } else { + panic!("Expected Store format: {:?}", func.dfg[inst]); + } +} + +// This is just a tighter register class constraint. +fn recipe_op1st_abcd(func: &Function, inst: Inst, sink: &mut CS) { + recipe_op1st(func, inst, sink) +} + +fn recipe_mp1st(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Store { args, .. } = func.dfg[inst] { + put_mp1(func.encodings[inst].bits(), sink); + modrm_rm(func.locations[args[1]].unwrap_reg(), + func.locations[args[0]].unwrap_reg(), + sink); + } else { + panic!("Expected Store format: {:?}", func.dfg[inst]); + } +} + +fn recipe_op1stdisp8(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Store { args, offset, .. } = func.dfg[inst] { + put_op1(func.encodings[inst].bits(), sink); + modrm_disp8(func.locations[args[1]].unwrap_reg(), + func.locations[args[0]].unwrap_reg(), + sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + } else { + panic!("Expected Store format: {:?}", func.dfg[inst]); + } +} + +fn recipe_op1stdisp8_abcd(func: &Function, inst: Inst, sink: &mut CS) { + recipe_op1stdisp8(func, inst, sink) +} + +fn recipe_mp1stdisp8(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Store { args, offset, .. } = func.dfg[inst] { + put_mp1(func.encodings[inst].bits(), sink); + modrm_disp8(func.locations[args[1]].unwrap_reg(), + func.locations[args[0]].unwrap_reg(), + sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + } else { + panic!("Expected Store format: {:?}", func.dfg[inst]); + } +} + +fn recipe_op1stdisp32(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Store { args, offset, .. } = func.dfg[inst] { + put_op1(func.encodings[inst].bits(), sink); + modrm_disp32(func.locations[args[1]].unwrap_reg(), + func.locations[args[0]].unwrap_reg(), + sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + } else { + panic!("Expected Store format: {:?}", func.dfg[inst]); + } +} + +fn recipe_op1stdisp32_abcd(func: &Function, inst: Inst, sink: &mut CS) { + recipe_op1stdisp32(func, inst, sink) +} + +fn recipe_mp1stdisp32(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Store { args, offset, .. } = func.dfg[inst] { + put_mp1(func.encodings[inst].bits(), sink); + modrm_disp32(func.locations[args[1]].unwrap_reg(), + func.locations[args[0]].unwrap_reg(), + sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + } else { + panic!("Expected Store format: {:?}", func.dfg[inst]); + } +} + +// Load recipes + +fn recipe_op1ld(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Load { arg, .. } = func.dfg[inst] { + put_op1(func.encodings[inst].bits(), sink); + modrm_rm(func.locations[arg].unwrap_reg(), + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); + } else { + panic!("Expected Load format: {:?}", func.dfg[inst]); + } +} + +fn recipe_op1lddisp8(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Load { arg, offset, .. } = func.dfg[inst] { + put_op1(func.encodings[inst].bits(), sink); + modrm_disp8(func.locations[arg].unwrap_reg(), + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + } else { + panic!("Expected Load format: {:?}", func.dfg[inst]); + } +} + +fn recipe_op1lddisp32(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Load { arg, offset, .. } = func.dfg[inst] { + put_op1(func.encodings[inst].bits(), sink); + modrm_disp32(func.locations[arg].unwrap_reg(), + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + } else { + panic!("Expected Load format: {:?}", func.dfg[inst]); + } +} + +fn recipe_op2ld(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Load { arg, .. } = func.dfg[inst] { + put_op2(func.encodings[inst].bits(), sink); + modrm_rm(func.locations[arg].unwrap_reg(), + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); + } else { + panic!("Expected Load format: {:?}", func.dfg[inst]); + } +} + +fn recipe_op2lddisp8(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Load { arg, offset, .. } = func.dfg[inst] { + put_op2(func.encodings[inst].bits(), sink); + modrm_disp8(func.locations[arg].unwrap_reg(), + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + } else { + panic!("Expected Load format: {:?}", func.dfg[inst]); + } +} + +fn recipe_op2lddisp32(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Load { arg, offset, .. } = func.dfg[inst] { + put_op2(func.encodings[inst].bits(), sink); + modrm_disp32(func.locations[arg].unwrap_reg(), + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + } else { + panic!("Expected Load format: {:?}", func.dfg[inst]); + } +} diff --git a/lib/cretonne/src/predicates.rs b/lib/cretonne/src/predicates.rs index 770857d100..6ce3e0a799 100644 --- a/lib/cretonne/src/predicates.rs +++ b/lib/cretonne/src/predicates.rs @@ -11,8 +11,8 @@ /// Check that `x` is the same as `y`. #[allow(dead_code)] -pub fn is_equal(x: T, y: T) -> bool { - x == y +pub fn is_equal + Copy>(x: T, y: O) -> bool { + x == y.into() } /// Check that `x` can be represented as a `wd`-bit signed integer with `sc` low zero bits. From 232fb36d8f4571db3c62a5c3b82eaec6bdc5e87a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Sun, 14 May 2017 11:53:44 -0700 Subject: [PATCH 0763/3084] Generate Intel encoding recipes on demand. Cretonne's encoding recipes need to have a fixed size so we can compute accurate branch destination addresses. Intel's instruction encoding has a lot of variance in the number of bytes needed to encode the opcode which leads to a number of duplicated encoding recipes that only differ in the opcode size. Add an Intel-specific TailEnc Python class which represents an abstraction over a set of recipes that are identical except for the opcode encoding. The TailEnc can then generate specific encoding recipes for each opcode format. The opcode format is a prefix of the recipe name, so for example, the 'rr' TailEnc will generate the 'Op1rr', 'Op2rr', 'Mp2rr' etc recipes. The TailEnc class provides a __call__ implementation that simply takes the sequence of opcode bytes as arguments. It then looks up the right prefix for the opcode bytes. --- lib/cretonne/meta/isa/intel/encodings.py | 73 ++++----- lib/cretonne/meta/isa/intel/recipes.py | 200 ++++++++++++++--------- 2 files changed, 155 insertions(+), 118 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 8d6d466aa7..575c36baca 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -4,61 +4,60 @@ Intel Encodings. from __future__ import absolute_import from base import instructions as base from .defs import I32 -from . import recipes as rcp -from .recipes import OP, OP0F, MP66 +from . import recipes as r -I32.enc(base.iadd.i32, rcp.Op1rr, OP(0x01)) -I32.enc(base.isub.i32, rcp.Op1rr, OP(0x29)) +I32.enc(base.iadd.i32, *r.rr(0x01)) +I32.enc(base.isub.i32, *r.rr(0x29)) -I32.enc(base.band.i32, rcp.Op1rr, OP(0x21)) -I32.enc(base.bor.i32, rcp.Op1rr, OP(0x09)) -I32.enc(base.bxor.i32, rcp.Op1rr, OP(0x31)) +I32.enc(base.band.i32, *r.rr(0x21)) +I32.enc(base.bor.i32, *r.rr(0x09)) +I32.enc(base.bxor.i32, *r.rr(0x31)) # Immediate instructions with sign-extended 8-bit and 32-bit immediate. -for inst, r in [ +for inst, rrr in [ (base.iadd_imm.i32, 0), (base.band_imm.i32, 4), (base.bor_imm.i32, 1), (base.bxor_imm.i32, 6)]: - I32.enc(inst, rcp.Op1rib, OP(0x83, rrr=r)) - I32.enc(inst, rcp.Op1rid, OP(0x81, rrr=r)) + I32.enc(inst, *r.rib(0x83, rrr=rrr)) + I32.enc(inst, *r.rid(0x81, rrr=rrr)) # 32-bit shifts and rotates. # Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit # and 16-bit shifts would need explicit masking. -I32.enc(base.ishl.i32.i32, rcp.Op1rc, OP(0xd3, rrr=4)) -I32.enc(base.ushr.i32.i32, rcp.Op1rc, OP(0xd3, rrr=5)) -I32.enc(base.sshr.i32.i32, rcp.Op1rc, OP(0xd3, rrr=7)) +I32.enc(base.ishl.i32.i32, *r.rc(0xd3, rrr=4)) +I32.enc(base.ushr.i32.i32, *r.rc(0xd3, rrr=5)) +I32.enc(base.sshr.i32.i32, *r.rc(0xd3, rrr=7)) # Loads and stores. -I32.enc(base.store.i32.i32, rcp.Op1st, OP(0x89)) -I32.enc(base.store.i32.i32, rcp.Op1stDisp8, OP(0x89)) -I32.enc(base.store.i32.i32, rcp.Op1stDisp32, OP(0x89)) +I32.enc(base.store.i32.i32, *r.st(0x89)) +I32.enc(base.store.i32.i32, *r.stDisp8(0x89)) +I32.enc(base.store.i32.i32, *r.stDisp32(0x89)) -I32.enc(base.istore16.i32.i32, rcp.Mp1st, MP66(0x89)) -I32.enc(base.istore16.i32.i32, rcp.Mp1stDisp8, MP66(0x89)) -I32.enc(base.istore16.i32.i32, rcp.Mp1stDisp32, MP66(0x89)) +I32.enc(base.istore16.i32.i32, *r.st(0x66, 0x89)) +I32.enc(base.istore16.i32.i32, *r.stDisp8(0x66, 0x89)) +I32.enc(base.istore16.i32.i32, *r.stDisp32(0x66, 0x89)) -I32.enc(base.istore8.i32.i32, rcp.Op1st_abcd, OP(0x88)) -I32.enc(base.istore8.i32.i32, rcp.Op1stDisp8_abcd, OP(0x88)) -I32.enc(base.istore8.i32.i32, rcp.Op1stDisp32_abcd, OP(0x88)) +I32.enc(base.istore8.i32.i32, *r.st_abcd(0x88)) +I32.enc(base.istore8.i32.i32, *r.stDisp8_abcd(0x88)) +I32.enc(base.istore8.i32.i32, *r.stDisp32_abcd(0x88)) -I32.enc(base.load.i32.i32, rcp.Op1ld, OP(0x8b)) -I32.enc(base.load.i32.i32, rcp.Op1ldDisp8, OP(0x8b)) -I32.enc(base.load.i32.i32, rcp.Op1ldDisp32, OP(0x8b)) +I32.enc(base.load.i32.i32, *r.ld(0x8b)) +I32.enc(base.load.i32.i32, *r.ldDisp8(0x8b)) +I32.enc(base.load.i32.i32, *r.ldDisp32(0x8b)) -I32.enc(base.uload16.i32.i32, rcp.Op2ld, OP0F(0xb7)) -I32.enc(base.uload16.i32.i32, rcp.Op2ldDisp8, OP0F(0xb7)) -I32.enc(base.uload16.i32.i32, rcp.Op2ldDisp32, OP0F(0xb7)) +I32.enc(base.uload16.i32.i32, *r.ld(0x0f, 0xb7)) +I32.enc(base.uload16.i32.i32, *r.ldDisp8(0x0f, 0xb7)) +I32.enc(base.uload16.i32.i32, *r.ldDisp32(0x0f, 0xb7)) -I32.enc(base.sload16.i32.i32, rcp.Op2ld, OP0F(0xbf)) -I32.enc(base.sload16.i32.i32, rcp.Op2ldDisp8, OP0F(0xbf)) -I32.enc(base.sload16.i32.i32, rcp.Op2ldDisp32, OP0F(0xbf)) +I32.enc(base.sload16.i32.i32, *r.ld(0x0f, 0xbf)) +I32.enc(base.sload16.i32.i32, *r.ldDisp8(0x0f, 0xbf)) +I32.enc(base.sload16.i32.i32, *r.ldDisp32(0x0f, 0xbf)) -I32.enc(base.uload8.i32.i32, rcp.Op2ld, OP0F(0xb6)) -I32.enc(base.uload8.i32.i32, rcp.Op2ldDisp8, OP0F(0xb6)) -I32.enc(base.uload8.i32.i32, rcp.Op2ldDisp32, OP0F(0xb6)) +I32.enc(base.uload8.i32.i32, *r.ld(0x0f, 0xb6)) +I32.enc(base.uload8.i32.i32, *r.ldDisp8(0x0f, 0xb6)) +I32.enc(base.uload8.i32.i32, *r.ldDisp32(0x0f, 0xb6)) -I32.enc(base.sload8.i32.i32, rcp.Op2ld, OP0F(0xbe)) -I32.enc(base.sload8.i32.i32, rcp.Op2ldDisp8, OP0F(0xbe)) -I32.enc(base.sload8.i32.i32, rcp.Op2ldDisp32, OP0F(0xbe)) +I32.enc(base.sload8.i32.i32, *r.ld(0x0f, 0xbe)) +I32.enc(base.sload8.i32.i32, *r.ldDisp8(0x0f, 0xbe)) +I32.enc(base.sload8.i32.i32, *r.ldDisp32(0x0f, 0xbe)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 6c72fe4690..71b8260f42 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -7,30 +7,41 @@ from cdsl.predicates import IsSignedInt, IsEqual from base.formats import Binary, BinaryImm, Store, Load from .registers import GPR, ABCD +try: + from typing import Tuple, Dict # noqa + from cdsl.instructions import InstructionFormat # noqa + from cdsl.isa import ConstraintSeq, BranchRange, PredNode # noqa +except ImportError: + pass + + # Opcode representation. # # Cretonne requires each recipe to have a single encoding size in bytes, and # Intel opcodes are variable length, so we use separate recipes for different # styles of opcodes and prefixes. The opcode format is indicated by the recipe # name prefix: -# -# Op1* OP(op) -# 0F Op2* OP0F(op) -# 0F 38 Op3* OP38(op) -# 0F 3A Op3* OP3A(op) -# 66 Mp1* MP66(op) -# 66 0F Mp2* MP660F(op) -# 66 0F 38 Mp3* MP660F38(op) -# 66 0F 3A Mp3* MP660F3A(op) -# F2 Mp1* MPF2(op) -# F2 0F Mp2* MPF20F(op) -# F2 0F 38 Mp3* MPF20F38(op) -# F2 0F 3A Mp3* MPF20F3A(op) -# F3 Mp1* MPF3(op) -# F3 0F Mp2* MP0FF3(op) -# F3 0F 38 Mp3* MPF30F38(op) -# F3 0F 3A Mp3* MPF30F3A(op) -# + +OPCODE_PREFIX = { + # Prefix bytes Name mmpp + (): ('Op1', 0b0000), + (0x66,): ('Mp1', 0b0001), + (0xf3,): ('Mp1', 0b0010), + (0xf2,): ('Mp1', 0b0011), + (0x0f,): ('Op2', 0b0100), + (0x66, 0x0f): ('Mp2', 0b0101), + (0xf3, 0x0f): ('Mp2', 0b0110), + (0xf2, 0x0f): ('Mp2', 0b0111), + (0x0f, 0x38): ('Op3', 0b1000), + (0x66, 0x0f, 0x38): ('Mp3', 0b1001), + (0xf3, 0x0f, 0x38): ('Mp3', 0b1010), + (0xf2, 0x0f, 0x38): ('Mp3', 0b1011), + (0x0f, 0x3a): ('Op3', 0b1100), + (0x66, 0x0f, 0x3a): ('Mp3', 0b1101), + (0xf3, 0x0f, 0x3a): ('Mp3', 0b1110), + (0xf2, 0x0f, 0x3a): ('Mp3', 0b1111) + } + # VEX/XOP and EVEX prefixes are not yet supported. # # The encoding bits are: @@ -53,40 +64,95 @@ from .registers import GPR, ABCD # enough bits, and the pp+mm format is ready for supporting VEX prefixes. -def OP(op, pp=0, mm=0, rrr=0, w=0): - # type: (int, int, int, int, int) -> int - assert op <= 0xff - assert pp <= 0b11 - assert mm <= 0b11 +def decode_ops(ops, rrr=0, w=0): + # type: (Tuple[int, ...], int, int) -> Tuple[str, int] + """ + Given a sequence of opcode bytes, compute the recipe name prefix and + encoding bits. + """ assert rrr <= 0b111 assert w <= 1 - return op | (pp << 8) | (mm << 10) | (rrr << 12) | (w << 15) + name, mmpp = OPCODE_PREFIX[ops[:-1]] + op = ops[-1] + assert op <= 256 + return (name, op | (mmpp << 8) | (rrr << 12) | (w << 15)) -def OP0F(op, rrr=0, w=0): - # type: (int, int, int) -> int - return OP(op, pp=0, mm=1, rrr=rrr, w=w) +class TailRecipe: + """ + Generate encoding recipes on demand. + Intel encodings are somewhat orthogonal with the opcode representation on + one side and the ModR/M, SIB and immediate fields on the other side. -def MP66(op, rrr=0, w=0): - # type: (int, int, int) -> int - return OP(op, pp=1, mm=0, rrr=rrr, w=w) + A `TailRecipe` represents the part of an encoding that follow the opcode. + It is used to generate full encoding recipes on demand when combined with + an opcode. + + The arguments are the same as for an `EncRecipe`, except for `size` which + does not include the size of the opcode. + """ + + def __init__( + self, + name, # type: str + format, # type: InstructionFormat + size, # type: int + ins, # type: ConstraintSeq + outs, # type: ConstraintSeq + branch_range=None, # type: BranchRange + instp=None, # type: PredNode + isap=None # type: PredNode + ): + # type: (...) -> None + self.name = name + self.format = format + self.size = size + self.ins = ins + self.outs = outs + self.branch_range = branch_range + self.instp = instp + self.isap = isap + + # Cached recipes, keyed by name prefix. + self.recipes = dict() # type: Dict[str, EncRecipe] + + def __call__(self, *ops, **kwargs): + # type: (*int, **int) -> Tuple[EncRecipe, int] + """ + Create an encoding recipe and encoding bits for the opcode bytes in + `ops`. + """ + rrr = kwargs.get('rrr', 0) + w = kwargs.get('w', 0) + name, bits = decode_ops(ops, rrr, w) + if name not in self.recipes: + self.recipes[name] = EncRecipe( + name + self.name, + self.format, + len(ops) + self.size, + ins=self.ins, + outs=self.outs, + branch_range=self.branch_range, + instp=self.instp, + isap=self.isap) + return (self.recipes[name], bits) # XX /r -Op1rr = EncRecipe('Op1rr', Binary, size=2, ins=(GPR, GPR), outs=0) +rr = TailRecipe('rr', Binary, size=1, ins=(GPR, GPR), outs=0) # XX /n with one arg in %rcx, for shifts. -Op1rc = EncRecipe('Op1rc', Binary, size=2, ins=(GPR, GPR.rcx), outs=0) +rc = TailRecipe('rc', Binary, size=1, ins=(GPR, GPR.rcx), outs=0) # XX /n ib with 8-bit immediate sign-extended. -Op1rib = EncRecipe( - 'Op1rib', BinaryImm, size=3, ins=GPR, outs=0, +rib = TailRecipe( + 'rib', BinaryImm, size=2, ins=GPR, outs=0, instp=IsSignedInt(BinaryImm.imm, 8)) # XX /n id with 32-bit immediate sign-extended. -Op1rid = EncRecipe( - 'Op1rid', BinaryImm, size=6, ins=GPR, outs=0, +rid = TailRecipe( + 'rid', BinaryImm, size=5, ins=GPR, outs=0, instp=IsSignedInt(BinaryImm.imm, 32)) # @@ -94,72 +160,44 @@ Op1rid = EncRecipe( # # XX /r register-indirect store with no offset. -Op1st = EncRecipe( - 'Op1st', Store, size=2, ins=(GPR, GPR), outs=(), +st = TailRecipe( + 'st', Store, size=1, ins=(GPR, GPR), outs=(), instp=IsEqual(Store.offset, 0)) # XX /r register-indirect store with no offset. # Only ABCD allowed for stored value. This is for byte stores. -Op1st_abcd = EncRecipe( - 'Op1st_abcd', Store, size=2, ins=(ABCD, GPR), outs=(), +st_abcd = TailRecipe( + 'st_abcd', Store, size=1, ins=(ABCD, GPR), outs=(), instp=IsEqual(Store.offset, 0)) # XX /r register-indirect store with 8-bit offset. -Op1stDisp8 = EncRecipe( - 'Op1stDisp8', Store, size=3, ins=(GPR, GPR), outs=(), +stDisp8 = TailRecipe( + 'stDisp8', Store, size=2, ins=(GPR, GPR), outs=(), instp=IsSignedInt(Store.offset, 8)) -Op1stDisp8_abcd = EncRecipe( - 'Op1stDisp8_abcd', Store, size=3, ins=(ABCD, GPR), outs=(), +stDisp8_abcd = TailRecipe( + 'stDisp8_abcd', Store, size=2, ins=(ABCD, GPR), outs=(), instp=IsSignedInt(Store.offset, 8)) # XX /r register-indirect store with 32-bit offset. -Op1stDisp32 = EncRecipe('Op1stDisp32', Store, size=6, ins=(GPR, GPR), outs=()) -Op1stDisp32_abcd = EncRecipe( - 'Op1stDisp32_abcd', Store, size=6, ins=(ABCD, GPR), outs=()) - -# PP WW /r register-indirect store with no offset. -Mp1st = EncRecipe( - 'Mp1st', Store, size=3, ins=(GPR, GPR), outs=(), - instp=IsEqual(Store.offset, 0)) - -# PP XX /r register-indirect store with 8-bit offset. -Mp1stDisp8 = EncRecipe( - 'Mp1stDisp8', Store, size=4, ins=(GPR, GPR), outs=(), - instp=IsSignedInt(Store.offset, 8)) - -# PP XX /r register-indirect store with 32-bit offset. -Mp1stDisp32 = EncRecipe('Mp1stDisp32', Store, size=7, ins=(GPR, GPR), outs=()) +stDisp32 = TailRecipe('stDisp32', Store, size=5, ins=(GPR, GPR), outs=()) +stDisp32_abcd = TailRecipe( + 'stDisp32_abcd', Store, size=5, ins=(ABCD, GPR), outs=()) # # Load recipes # # XX /r load with no offset. -Op1ld = EncRecipe( - 'Op1ld', Load, size=2, ins=(GPR), outs=(GPR), +ld = TailRecipe( + 'ld', Load, size=1, ins=(GPR), outs=(GPR), instp=IsEqual(Load.offset, 0)) # XX /r load with 8-bit offset. -Op1ldDisp8 = EncRecipe( - 'Op1ldDisp8', Load, size=3, ins=(GPR), outs=(GPR), +ldDisp8 = TailRecipe( + 'ldDisp8', Load, size=2, ins=(GPR), outs=(GPR), instp=IsSignedInt(Load.offset, 8)) # XX /r load with 32-bit offset. -Op1ldDisp32 = EncRecipe( - 'Op1ldDisp32', Load, size=6, ins=(GPR), outs=(GPR), - instp=IsSignedInt(Load.offset, 32)) - -# 0F XX /r load with no offset. -Op2ld = EncRecipe( - 'Op2ld', Load, size=3, ins=(GPR), outs=(GPR), - instp=IsEqual(Load.offset, 0)) - -# XX /r load with 8-bit offset. -Op2ldDisp8 = EncRecipe( - 'Op2ldDisp8', Load, size=4, ins=(GPR), outs=(GPR), - instp=IsSignedInt(Load.offset, 8)) - -# XX /r load with 32-bit offset. -Op2ldDisp32 = EncRecipe( - 'Op2ldDisp32', Load, size=7, ins=(GPR), outs=(GPR), +ldDisp32 = TailRecipe( + 'ldDisp32', Load, size=5, ins=(GPR), outs=(GPR), instp=IsSignedInt(Load.offset, 32)) From 29dc723e25dfeb80fd4684c8b483316a303be79e Mon Sep 17 00:00:00 2001 From: Benjamin Bouvier Date: Tue, 16 May 2017 00:10:47 +0200 Subject: [PATCH 0764/3084] Update rustfmt to 0.8.4; (#81) --- check-rustfmt.sh | 2 +- cranelift/src/filetest/concurrent.rs | 4 +- cranelift/src/filetest/runner.rs | 4 +- lib/cretonne/src/binemit/relaxation.rs | 4 +- lib/cretonne/src/dominator_tree.rs | 3 +- lib/cretonne/src/entity_list.rs | 4 +- lib/cretonne/src/ir/builder.rs | 3 +- lib/cretonne/src/ir/dfg.rs | 20 +--- lib/cretonne/src/ir/instructions.rs | 30 +++--- lib/cretonne/src/ir/layout.rs | 3 +- lib/cretonne/src/isa/encoding.rs | 4 +- lib/cretonne/src/iterators.rs | 12 +-- lib/cretonne/src/legalizer/boundary.rs | 12 +-- lib/cretonne/src/regalloc/allocatable_set.rs | 28 +++--- lib/cretonne/src/regalloc/diversion.rs | 8 +- lib/cretonne/src/regalloc/solver.rs | 3 +- lib/cretonne/src/verifier/mod.rs | 32 +++---- lib/filecheck/src/checker.rs | 6 +- lib/reader/src/lexer.rs | 16 ++-- lib/reader/src/parser.rs | 98 +++++++++----------- 20 files changed, 120 insertions(+), 176 deletions(-) diff --git a/check-rustfmt.sh b/check-rustfmt.sh index c0c99c9a91..6e8563900f 100755 --- a/check-rustfmt.sh +++ b/check-rustfmt.sh @@ -15,7 +15,7 @@ # With the --install option, also tries to install the right version. # This version should always be bumped to the newest version available. -VERS="0.8.3" +VERS="0.8.4" if cargo install --list | grep -q "^rustfmt v$VERS"; then exit 0 diff --git a/cranelift/src/filetest/concurrent.rs b/cranelift/src/filetest/concurrent.rs index dbdd76f35c..7dcbe1b4fc 100644 --- a/cranelift/src/filetest/concurrent.rs +++ b/cranelift/src/filetest/concurrent.rs @@ -119,9 +119,7 @@ fn worker_thread(thread_num: usize, // Tell them we're starting this job. // The receiver should always be present for this as long as we have jobs. - replies - .send(Reply::Starting { jobid, thread_num }) - .unwrap(); + replies.send(Reply::Starting { jobid, thread_num }).unwrap(); let result = catch_unwind(|| runone::run(path.as_path())).unwrap_or_else(|e| { // The test panicked, leaving us a `Box`. diff --git a/cranelift/src/filetest/runner.rs b/cranelift/src/filetest/runner.rs index 23f94fe78f..9188ee00ad 100644 --- a/cranelift/src/filetest/runner.rs +++ b/cranelift/src/filetest/runner.rs @@ -207,9 +207,7 @@ impl TestRunner { } // Check for any asynchronous replies without blocking. - while let Some(reply) = self.threads - .as_mut() - .and_then(ConcurrentRunner::try_get) { + while let Some(reply) = self.threads.as_mut().and_then(ConcurrentRunner::try_get) { self.handle_reply(reply); } } diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index 74158a61e4..517867c725 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -97,9 +97,7 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) { /// existing `fallthrough` instructions are correct. fn fallthroughs(func: &mut Function) { for (ebb, succ) in func.layout.ebbs().adjacent_pairs() { - let term = func.layout - .last_inst(ebb) - .expect("EBB has no terminator."); + let term = func.layout.last_inst(ebb).expect("EBB has no terminator."); if let InstructionData::Jump { ref mut opcode, destination, diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 13d1175d8a..ed6aa4dbd6 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -88,8 +88,7 @@ impl DominatorTree { // Run a finger up the dominator tree from b until we see a. // Do nothing if b is unreachable. while rpo_a < self.nodes[ebb_b].rpo_number { - b = self.idom(ebb_b) - .expect("Shouldn't meet unreachable here."); + b = self.idom(ebb_b).expect("Shouldn't meet unreachable here."); ebb_b = layout.inst_ebb(b).expect("Dominator got removed."); } diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index 3b7493094e..b6cf3099eb 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -135,9 +135,7 @@ impl ListPool { // The `wrapping_sub` handles the special case 0, which is the empty list. This way, the // cost of the bounds check that we have to pay anyway is co-opted to handle the special // case of the empty list. - self.data - .get(idx.wrapping_sub(1)) - .map(|len| len.index()) + self.data.get(idx.wrapping_sub(1)).map(|len| len.index()) } /// Allocate a storage block with a size given by `sclass`. diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index a5a3719299..256845bcfb 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -132,8 +132,7 @@ impl<'c, 'fc, 'fd, Array> InstBuilderBase<'fd> for InsertReuseBuilder<'c, 'fc, ' let inst = self.dfg.make_inst(data); // Make an `Interator>`. let ru = self.reuse.as_ref().iter().cloned(); - self.dfg - .make_inst_results_reusing(inst, ctrl_typevar, ru); + self.dfg.make_inst_results_reusing(inst, ctrl_typevar, ru); self.pos.insert_inst(inst); (inst, self.dfg) } diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 6e33c9d592..d5eb5ab73d 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -316,37 +316,25 @@ impl DataFlowGraph { /// Get the fixed value arguments on `inst` as a slice. pub fn inst_fixed_args(&self, inst: Inst) -> &[Value] { - let fixed_args = self[inst] - .opcode() - .constraints() - .fixed_value_arguments(); + let fixed_args = self[inst].opcode().constraints().fixed_value_arguments(); &self.inst_args(inst)[..fixed_args] } /// Get the fixed value arguments on `inst` as a mutable slice. pub fn inst_fixed_args_mut(&mut self, inst: Inst) -> &mut [Value] { - let fixed_args = self[inst] - .opcode() - .constraints() - .fixed_value_arguments(); + let fixed_args = self[inst].opcode().constraints().fixed_value_arguments(); &mut self.inst_args_mut(inst)[..fixed_args] } /// Get the variable value arguments on `inst` as a slice. pub fn inst_variable_args(&self, inst: Inst) -> &[Value] { - let fixed_args = self[inst] - .opcode() - .constraints() - .fixed_value_arguments(); + let fixed_args = self[inst].opcode().constraints().fixed_value_arguments(); &self.inst_args(inst)[fixed_args..] } /// Get the variable value arguments on `inst` as a mutable slice. pub fn inst_variable_args_mut(&mut self, inst: Inst) -> &mut [Value] { - let fixed_args = self[inst] - .opcode() - .constraints() - .fixed_value_arguments(); + let fixed_args = self[inst].opcode().constraints().fixed_value_arguments(); &mut self.inst_args_mut(inst)[fixed_args..] } diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 0b32929af6..4301c7796c 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -288,20 +288,20 @@ impl InstructionData { pub fn analyze_branch<'a>(&'a self, pool: &'a ValueListPool) -> BranchInfo<'a> { match self { &InstructionData::Jump { - destination, - ref args, - .. - } => BranchInfo::SingleDest(destination, &args.as_slice(pool)), + destination, + ref args, + .. + } => BranchInfo::SingleDest(destination, &args.as_slice(pool)), &InstructionData::Branch { - destination, - ref args, - .. - } => BranchInfo::SingleDest(destination, &args.as_slice(pool)[1..]), + destination, + ref args, + .. + } => BranchInfo::SingleDest(destination, &args.as_slice(pool)[1..]), &InstructionData::BranchIcmp { - destination, - ref args, - .. - } => BranchInfo::SingleDest(destination, &args.as_slice(pool)[2..]), + destination, + ref args, + .. + } => BranchInfo::SingleDest(destination, &args.as_slice(pool)[2..]), &InstructionData::BranchTable { table, .. } => BranchInfo::Table(table), _ => BranchInfo::NotABranch, } @@ -595,11 +595,7 @@ impl OperandConstraint { Same => Bound(ctrl_type), LaneOf => Bound(ctrl_type.lane_type()), AsBool => Bound(ctrl_type.as_bool()), - HalfWidth => { - Bound(ctrl_type - .half_width() - .expect("invalid type for half_width")) - } + HalfWidth => Bound(ctrl_type.half_width().expect("invalid type for half_width")), DoubleWidth => { Bound(ctrl_type .double_width() diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index abf0f3f482..4e6f738653 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -467,8 +467,7 @@ impl Layout { /// Remove `inst` from the layout. pub fn remove_inst(&mut self, inst: Inst) { - let ebb = self.inst_ebb(inst) - .expect("Instruction already removed."); + let ebb = self.inst_ebb(inst).expect("Instruction already removed."); // Clear the `inst` node and extract links. let prev; let next; diff --git a/lib/cretonne/src/isa/encoding.rs b/lib/cretonne/src/isa/encoding.rs index b5062d988c..9bc78d9ef0 100644 --- a/lib/cretonne/src/isa/encoding.rs +++ b/lib/cretonne/src/isa/encoding.rs @@ -130,8 +130,6 @@ impl EncInfo { /// /// This will never return `None` for a legal branch encoding. pub fn branch_range(&self, enc: Encoding) -> Option { - self.sizing - .get(enc.recipe()) - .and_then(|s| s.branch_range) + self.sizing.get(enc.recipe()).and_then(|s| s.branch_range) } } diff --git a/lib/cretonne/src/iterators.rs b/lib/cretonne/src/iterators.rs index bca8cea2e4..70d1165ed7 100644 --- a/lib/cretonne/src/iterators.rs +++ b/lib/cretonne/src/iterators.rs @@ -65,17 +65,9 @@ mod tests { .adjacent_pairs() .collect::>(), vec![(2, 3), (3, 4)]); - assert_eq!([3, 4] - .iter() - .cloned() - .adjacent_pairs() - .collect::>(), + assert_eq!([3, 4].iter().cloned().adjacent_pairs().collect::>(), vec![(3, 4)]); - assert_eq!([4] - .iter() - .cloned() - .adjacent_pairs() - .collect::>(), + assert_eq!([4].iter().cloned().adjacent_pairs().collect::>(), vec![]); assert_eq!([] .iter() diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index e0cf9a2f21..b65eea0984 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -251,9 +251,7 @@ fn convert_from_abi(dfg: &mut DataFlowGraph, assert!(!ty.is_int()); let abi_ty = Type::int(ty.bits()).expect("Invalid type for conversion"); let arg = convert_from_abi(dfg, pos, abi_ty, None, get_arg); - dfg.ins(pos) - .with_results([into_result]) - .bitcast(ty, arg) + dfg.ins(pos).with_results([into_result]).bitcast(ty, arg) } // ABI argument is a sign-extended version of the value we want. ValueConversion::Sext(abi_ty) => { @@ -261,18 +259,14 @@ fn convert_from_abi(dfg: &mut DataFlowGraph, // TODO: Currently, we don't take advantage of the ABI argument being sign-extended. // We could insert an `assert_sreduce` which would fold with a following `sextend` of // this value. - dfg.ins(pos) - .with_results([into_result]) - .ireduce(ty, arg) + dfg.ins(pos).with_results([into_result]).ireduce(ty, arg) } ValueConversion::Uext(abi_ty) => { let arg = convert_from_abi(dfg, pos, abi_ty, None, get_arg); // TODO: Currently, we don't take advantage of the ABI argument being sign-extended. // We could insert an `assert_ureduce` which would fold with a following `uextend` of // this value. - dfg.ins(pos) - .with_results([into_result]) - .ireduce(ty, arg) + dfg.ins(pos).with_results([into_result]).ireduce(ty, arg) } } } diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index 714a82aab7..db5a593901 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -137,21 +137,21 @@ mod tests { // Register classes for testing. const GPR: RegClass = &RegClassData { - name: "GPR", - index: 0, - width: 1, - first: 28, - subclasses: 0, - mask: [0xf0000000, 0x0000000f, 0], - }; + name: "GPR", + index: 0, + width: 1, + first: 28, + subclasses: 0, + mask: [0xf0000000, 0x0000000f, 0], + }; const DPR: RegClass = &RegClassData { - name: "DPR", - index: 0, - width: 2, - first: 28, - subclasses: 0, - mask: [0x50000000, 0x0000000a, 0], - }; + name: "DPR", + index: 0, + width: 2, + first: 28, + subclasses: 0, + mask: [0x50000000, 0x0000000a, 0], + }; #[test] fn put_and_take() { diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index 2096af8722..89decad82f 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -100,10 +100,10 @@ mod tests { divs.regmove(v1, 10, 12); assert_eq!(divs.diversion(v1), Some(&Diversion { - value: v1, - from: 10, - to: 12, - })); + value: v1, + from: 10, + to: 12, + })); assert_eq!(divs.diversion(v2), None); divs.regmove(v1, 12, 11); diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 5e3c43da5f..89aee41317 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -414,8 +414,7 @@ impl Solver { if self.inputs_done { self.regs_out.free(constraint, from); } - self.vars - .push(Variable::new_live(value, constraint, from)); + self.vars.push(Variable::new_live(value, constraint, from)); } /// Check for conflicts between fixed input assignments and existing live values. diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index e1568d5bab..5c283dd6d3 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -234,20 +234,20 @@ impl<'a> Verifier<'a> { self.verify_value_list(inst, args)?; } &Jump { - destination, - ref args, - .. - } | + destination, + ref args, + .. + } | &Branch { - destination, - ref args, - .. - } | + destination, + ref args, + .. + } | &BranchIcmp { - destination, - ref args, - .. - } => { + destination, + ref args, + .. + } => { self.verify_ebb(inst, destination)?; self.verify_value_list(inst, args)?; } @@ -381,8 +381,7 @@ impl<'a> Verifier<'a> { ebb); } // The defining EBB dominates the instruction using this value. - if !self.domtree - .ebb_dominates(ebb, loc_inst, &self.func.layout) { + if !self.domtree.ebb_dominates(ebb, loc_inst, &self.func.layout) { return err!(loc_inst, "uses value arg from non-dominating {}", ebb); } } @@ -639,10 +638,7 @@ impl<'a> Verifier<'a> { return err!(ebb, "cfg had unexpected successor(s) {:?}", excess_succs); } - expected_preds.extend(self.cfg - .get_predecessors(ebb) - .iter() - .map(|&(_, inst)| inst)); + expected_preds.extend(self.cfg.get_predecessors(ebb).iter().map(|&(_, inst)| inst)); got_preds.extend(cfg.get_predecessors(ebb).iter().map(|&(_, inst)| inst)); let missing_preds: Vec = expected_preds.difference(&got_preds).cloned().collect(); diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs index e70768e32c..ee44a769e5 100644 --- a/lib/filecheck/src/checker.rs +++ b/lib/filecheck/src/checker.rs @@ -416,11 +416,9 @@ mod tests { Ok(true)); assert_eq!(b.directive("regex: X = tommy").map_err(e2s), Err("expected '=' after variable 'X' in regex: X = tommy".to_string())); - assert_eq!(b.directive("[arm]not: patt $x $(y) here") - .map_err(e2s), + assert_eq!(b.directive("[arm]not: patt $x $(y) here").map_err(e2s), Ok(true)); - assert_eq!(b.directive("[x86]sameln: $x $(y=[^]]*) there") - .map_err(e2s), + assert_eq!(b.directive("[x86]sameln: $x $(y=[^]]*) there").map_err(e2s), Ok(true)); // Windows line ending sneaking in. assert_eq!(b.directive("regex: Y=foo\r").map_err(e2s), Ok(true)); diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 1262b63888..c5c2aa5d80 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -296,7 +296,7 @@ impl<'a> Lexer<'a> { token(split_entity_name(text) .and_then(|(prefix, number)| { Self::numbered_entity(prefix, number) - .or_else(|| Self::value_type(text, prefix, number)) + .or_else(|| Self::value_type(text, prefix, number)) }) .unwrap_or(Token::Identifier(text)), loc) @@ -413,14 +413,14 @@ impl<'a> Lexer<'a> { Some('%') => Some(self.scan_name()), Some('#') => Some(self.scan_hex_sequence()), Some(ch) if ch.is_whitespace() => { - self.next_ch(); - continue; - } + self.next_ch(); + continue; + } _ => { - // Skip invalid char, return error. - self.next_ch(); - Some(error(Error::InvalidChar, loc)) - } + // Skip invalid char, return error. + self.next_ch(); + Some(error(Error::InvalidChar, loc)) + } }; } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index b907ee426f..fa9f31df5a 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -29,12 +29,7 @@ use sourcemap::{SourceMap, MutableSourceMap}; /// /// Any test commands or ISA declarations are ignored. pub fn parse_functions(text: &str) -> Result> { - parse_test(text).map(|file| { - file.functions - .into_iter() - .map(|(func, _)| func) - .collect() - }) + parse_test(text).map(|file| file.functions.into_iter().map(|(func, _)| func).collect()) } /// Parse the entire `text` as a test case file. @@ -753,9 +748,7 @@ impl<'a> Parser<'a> { sig.return_types = self.parse_argument_list(unique_isa)?; } - if sig.argument_types - .iter() - .all(|a| a.location.is_assigned()) { + if sig.argument_types.iter().all(|a| a.location.is_assigned()) { sig.compute_argument_bytes(); } @@ -1319,48 +1312,50 @@ impl<'a> Parser<'a> { inst_data: &InstructionData) -> Result { let constraints = opcode.constraints(); - let ctrl_type = - match explicit_ctrl_type { - Some(t) => t, - None => { - if constraints.use_typevar_operand() { - // This is an opcode that supports type inference, AND there was no - // explicit type specified. Look up `ctrl_value` to see if it was defined - // already. - // TBD: If it is defined in another block, the type should have been - // specified explicitly. It is unfortunate that the correctness of IL - // depends on the layout of the blocks. - let ctrl_src_value = inst_data - .typevar_operand(&ctx.function.dfg.value_lists) - .expect("Constraints <-> Format inconsistency"); - ctx.function.dfg.value_type(match ctx.map.get_value(ctrl_src_value) { - Some(v) => v, - None => { - if let Some(v) = ctx.aliases - .get(&ctrl_src_value) - .and_then(|&(aliased, _)| ctx.map.get_value(aliased)) - { - v - } else { - return err!(self.loc, - "cannot determine type of operand {}", - ctrl_src_value); - } - } - }) - } else if constraints.is_polymorphic() { - // This opcode does not support type inference, so the explicit type - // variable is required. - return err!(self.loc, - "type variable required for polymorphic opcode, e.g. '{}.{}'", - opcode, - constraints.ctrl_typeset().unwrap().example()); - } else { - // This is a non-polymorphic opcode. No typevar needed. - VOID - } + let ctrl_type = match explicit_ctrl_type { + Some(t) => t, + None => { + if constraints.use_typevar_operand() { + // This is an opcode that supports type inference, AND there was no + // explicit type specified. Look up `ctrl_value` to see if it was defined + // already. + // TBD: If it is defined in another block, the type should have been + // specified explicitly. It is unfortunate that the correctness of IL + // depends on the layout of the blocks. + let ctrl_src_value = inst_data + .typevar_operand(&ctx.function.dfg.value_lists) + .expect("Constraints <-> Format inconsistency"); + ctx.function + .dfg + .value_type(match ctx.map.get_value(ctrl_src_value) { + Some(v) => v, + None => { + if let Some(v) = ctx.aliases + .get(&ctrl_src_value) + .and_then(|&(aliased, _)| { + ctx.map.get_value(aliased) + }) { + v + } else { + return err!(self.loc, + "cannot determine type of operand {}", + ctrl_src_value); + } + } + }) + } else if constraints.is_polymorphic() { + // This opcode does not support type inference, so the explicit type + // variable is required. + return err!(self.loc, + "type variable required for polymorphic opcode, e.g. '{}.{}'", + opcode, + constraints.ctrl_typeset().unwrap().example()); + } else { + // This is a non-polymorphic opcode. No typevar needed. + VOID } - }; + } + }; // Verify that `ctrl_type` is valid for the controlling type variable. We don't want to // attempt deriving types from an incorrect basis. @@ -1616,8 +1611,7 @@ impl<'a> Parser<'a> { InstructionFormat::BranchTable => { let arg = self.match_value("expected SSA value operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; - let table = self.match_jt() - .and_then(|num| ctx.get_jt(num, &self.loc))?; + let table = self.match_jt().and_then(|num| ctx.get_jt(num, &self.loc))?; InstructionData::BranchTable { opcode, arg, table } } InstructionFormat::StackLoad => { From 621f3a2f502f03a5f835daed284d8576097bcceb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 15 May 2017 13:01:41 -0700 Subject: [PATCH 0765/3084] Add a register bank index to RegClassData. This makes it possible to find the register bank that contains a register class. --- lib/cretonne/meta/cdsl/registers.py | 1 + lib/cretonne/meta/gen_registers.py | 13 +++++++------ lib/cretonne/src/isa/registers.rs | 3 +++ lib/cretonne/src/regalloc/allocatable_set.rs | 2 ++ 4 files changed, 13 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index 1f6bfa682f..c16e760a5c 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -87,6 +87,7 @@ class RegBank(object): align = next_power_of_two(align) self.first_unit = (u + align - 1) & -align + self.index = len(isa.regbanks) isa.regbanks.append(self) def __repr__(self): diff --git a/lib/cretonne/meta/gen_registers.py b/lib/cretonne/meta/gen_registers.py index b1532761a6..4199dce268 100644 --- a/lib/cretonne/meta/gen_registers.py +++ b/lib/cretonne/meta/gen_registers.py @@ -34,13 +34,14 @@ def gen_regclass(rc, fmt): Emit a static data definition for a register class. """ with fmt.indented('RegClassData {', '},'): - fmt.line('name: "{}",'.format(rc.name)) - fmt.line('index: {},'.format(rc.index)) - fmt.line('width: {},'.format(rc.width)) - fmt.line('first: {},'.format(rc.bank.first_unit + rc.start)) - fmt.line('subclasses: 0x{:x},'.format(rc.subclass_mask())) + fmt.format('name: "{}",', rc.name) + fmt.format('index: {},', rc.index) + fmt.format('width: {},', rc.width) + fmt.format('bank: {},', rc.bank.index) + fmt.format('first: {},', rc.bank.first_unit + rc.start) + fmt.format('subclasses: 0x{:x},', rc.subclass_mask()) mask = ', '.join('0x{:08x}'.format(x) for x in rc.mask()) - fmt.line('mask: [{}],'.format(mask)) + fmt.format('mask: [{}],', mask) def gen_isa(isa, fmt): diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 91a25595f1..461309a9e9 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -108,6 +108,9 @@ pub struct RegClassData { /// How many register units to allocate per register. pub width: u8, + /// Index of the register bank this class belongs to. + pub bank: u8, + /// The first register unit in this class. pub first: RegUnit, diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index db5a593901..fa3e7a9153 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -140,6 +140,7 @@ mod tests { name: "GPR", index: 0, width: 1, + bank: 0, first: 28, subclasses: 0, mask: [0xf0000000, 0x0000000f, 0], @@ -148,6 +149,7 @@ mod tests { name: "DPR", index: 0, width: 2, + bank: 0, first: 28, subclasses: 0, mask: [0x50000000, 0x0000000a, 0], From 71bb7483b723f8f62b75c1827fc9b333ed2077cd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 15 May 2017 13:39:59 -0700 Subject: [PATCH 0766/3084] Compute top-level register classes for each register bank. A top-level register class is one that has no sub-classes. It is possible to have multiple top-level register classes in the same register bank. For example, ARM's FPR bank has both D and Q top-level register classes. Number register classes such that all top-level register classes appear as a contiguous sequence starting from 0. This will be used by the register allocator when counting used registers per top-level register class. --- lib/cretonne/meta/cdsl/isa.py | 19 ++++++++++-- lib/cretonne/meta/cdsl/registers.py | 31 ++++++++++++++------ lib/cretonne/meta/gen_registers.py | 26 ++++++++-------- lib/cretonne/src/isa/registers.rs | 12 ++++++++ lib/cretonne/src/regalloc/allocatable_set.rs | 2 ++ 5 files changed, 66 insertions(+), 24 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index f1125f59da..9239bf2f70 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -42,6 +42,7 @@ class TargetISA(object): self.instruction_groups = instruction_groups self.cpumodes = list() # type: List[CPUMode] self.regbanks = list() # type: List[RegBank] + self.regclasses = list() # type: List[RegClass] def finish(self): # type: () -> TargetISA @@ -109,11 +110,23 @@ class TargetISA(object): Every register class needs a unique index, and the classes need to be topologically ordered. + + We also want all the top-level register classes to be first. """ - rc_index = 0 + # Compute subclasses and top-level classes in each bank. + # Collect the top-level classes so they get numbered consecutively. for bank in self.regbanks: - bank.finish_regclasses(rc_index) - rc_index += len(bank.classes) + bank.finish_regclasses() + self.regclasses.extend(bank.toprcs) + + # Collect all of the non-top-level register classes. + # They are numbered strictly after the top-level classes. + for bank in self.regbanks: + self.regclasses.extend( + rc for rc in bank.classes if not rc.is_toprc()) + + for idx, rc in enumerate(self.regclasses): + rc.index = idx class CPUMode(object): diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index c16e760a5c..caaacd20e4 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -75,6 +75,8 @@ class RegBank(object): self.prefix = prefix self.names = names self.classes = list() # type: List[RegClass] + self.toprcs = list() # type: List[RegClass] + self.first_toprc_index = None # type: int assert len(names) <= units @@ -95,11 +97,10 @@ class RegBank(object): return ('RegBank({}, units={}, first_unit={})' .format(self.name, self.units, self.first_unit)) - def finish_regclasses(self, first_index): - # type: (int) -> None + def finish_regclasses(self): + # type: () -> None """ - Assign indexes to the register classes in this bank, starting from - `first_index`. + Compute subclasses and the top-level register class. Verify that the set of register classes satisfies: @@ -115,14 +116,10 @@ class RegBank(object): """ cmap = dict() # type: Dict[RCTup, RegClass] - for idx, rc in enumerate(self.classes): + for rc in self.classes: # All register classes must be given a name. assert rc.name, "Anonymous register class found" - # Assign a unique index. - assert rc.index is None - rc.index = idx + first_index - # Check for duplicates. tup = rc.rctup() if tup in cmap: @@ -133,6 +130,7 @@ class RegBank(object): # Check intersections and topological order. for idx, rc1 in enumerate(self.classes): + rc1.toprc = rc1 for rc2 in self.classes[0:idx]: itup = rc1.intersect(rc2) if itup is None: @@ -151,6 +149,10 @@ class RegBank(object): # The intersection of rc1 and rc2 is rc1, so it must be a # sub-class. rc2.subclasses.append(rc1) + rc1.toprc = rc2.toprc + + if rc1.is_toprc(): + self.toprcs.append(rc1) def unit_by_name(self, name): # type: (str) -> int @@ -192,6 +194,7 @@ class RegClass(object): # This is computed later in `finish_regclasses()`. self.subclasses = list() # type: List[RegClass] + self.toprc = None # type: RegClass assert width > 0 assert start >= 0 and start < bank.units @@ -206,6 +209,16 @@ class RegClass(object): # type: () -> str return self.name + def is_toprc(self): + # type: () -> bool + """ + Is this a top-level register class? + + A top-level register class has no sub-classes. This can only be + answered aster running `finish_regclasses()`. + """ + return self.toprc is self + def rctup(self): # type: () -> RCTup """ diff --git a/lib/cretonne/meta/gen_registers.py b/lib/cretonne/meta/gen_registers.py index 4199dce268..463364709a 100644 --- a/lib/cretonne/meta/gen_registers.py +++ b/lib/cretonne/meta/gen_registers.py @@ -19,13 +19,15 @@ def gen_regbank(regbank, fmt): Emit a static data definition for regbank. """ with fmt.indented('RegBank {', '},'): - fmt.line('name: "{}",'.format(regbank.name)) - fmt.line('first_unit: {},'.format(regbank.first_unit)) - fmt.line('units: {},'.format(regbank.units)) - fmt.line( - 'names: &[{}],' - .format(', '.join('"{}"'.format(n) for n in regbank.names))) - fmt.line('prefix: "{}",'.format(regbank.prefix)) + fmt.format('name: "{}",', regbank.name) + fmt.format('first_unit: {},', regbank.first_unit) + fmt.format('units: {},', regbank.units) + fmt.format( + 'names: &[{}],', + ', '.join('"{}"'.format(n) for n in regbank.names)) + fmt.format('prefix: "{}",', regbank.prefix) + fmt.format('first_toprc: {},', regbank.toprcs[0].index) + fmt.format('num_toprcs: {},', len(regbank.toprcs)) def gen_regclass(rc, fmt): @@ -38,6 +40,7 @@ def gen_regclass(rc, fmt): fmt.format('index: {},', rc.index) fmt.format('width: {},', rc.width) fmt.format('bank: {},', rc.bank.index) + fmt.format('toprc: {},', rc.toprc.index) fmt.format('first: {},', rc.bank.first_unit + rc.start) fmt.format('subclasses: 0x{:x},', rc.subclass_mask()) mask = ', '.join('0x{:08x}'.format(x) for x in rc.mask()) @@ -52,24 +55,23 @@ def gen_isa(isa, fmt): if not isa.regbanks: print('cargo:warning={} has no register banks'.format(isa.name)) - rcs = list() # type: List[RegClass] with fmt.indented('pub static INFO: RegInfo = RegInfo {', '};'): # Bank descriptors. with fmt.indented('banks: &[', '],'): for regbank in isa.regbanks: gen_regbank(regbank, fmt) - rcs += regbank.classes fmt.line('classes: &CLASSES,') # Register class descriptors. with fmt.indented( - 'const CLASSES: [RegClassData; {}] = ['.format(len(rcs)), '];'): - for idx, rc in enumerate(rcs): + 'const CLASSES: [RegClassData; {}] = [' + .format(len(isa.regclasses)), '];'): + for idx, rc in enumerate(isa.regclasses): assert idx == rc.index gen_regclass(rc, fmt) # Emit constants referencing the register classes. - for rc in rcs: + for rc in isa.regclasses: fmt.line('#[allow(dead_code)]') fmt.line( 'pub const {}: RegClass = &CLASSES[{}];' diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 461309a9e9..a4527733c5 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -42,6 +42,15 @@ pub struct RegBank { /// The remaining register units will be named this prefix followed by their decimal offset in /// the bank. So with a prefix `r`, registers will be named `r8`, `r9`, ... pub prefix: &'static str, + + /// Index of the first top-level register class in this bank. + pub first_toprc: usize, + + /// Number of top-level register classes in this bank. + /// + /// The top-level register classes in a bank are guaranteed to be numbered sequentially from + /// `first_toprc`, and all top-level register classes across banks come before any sub-classes. + pub num_toprcs: usize, } impl RegBank { @@ -111,6 +120,9 @@ pub struct RegClassData { /// Index of the register bank this class belongs to. pub bank: u8, + /// Index of the top-level register class contains this one. + pub toprc: u8, + /// The first register unit in this class. pub first: RegUnit, diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index fa3e7a9153..756ad1d349 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -141,6 +141,7 @@ mod tests { index: 0, width: 1, bank: 0, + toprc: 0, first: 28, subclasses: 0, mask: [0xf0000000, 0x0000000f, 0], @@ -150,6 +151,7 @@ mod tests { index: 0, width: 2, bank: 0, + toprc: 0, first: 28, subclasses: 0, mask: [0x50000000, 0x0000000a, 0], From 6787a4ea5c7c35c423857fb73c636c7f56860762 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 15 May 2017 16:01:24 -0700 Subject: [PATCH 0767/3084] Add a RegClassMask typedef and a MAX_TOPRCS constant. Avoid spreading u32 as a bitmask of register classes throughout the code. Enforce the limit of 32 register classes total. This could easily be raised if needed. The MAX_TOPRCS constant is the highest possible number of top-level register classes in an ISA. The RegClassData.toprc field is always smaller than this limit. --- lib/cretonne/meta/cdsl/isa.py | 10 ++++++++++ lib/cretonne/src/isa/registers.rs | 14 +++++++++++++- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 9239bf2f70..a2387bc0ff 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -119,6 +119,11 @@ class TargetISA(object): bank.finish_regclasses() self.regclasses.extend(bank.toprcs) + # The limit on the number of top-level register classes can be raised. + # This should be coordinated with the `MAX_TOPRCS` constant in + # `isa/registers.rs`. + assert len(self.regclasses) <= 4, "Too many top-level register classes" + # Collect all of the non-top-level register classes. # They are numbered strictly after the top-level classes. for bank in self.regbanks: @@ -128,6 +133,11 @@ class TargetISA(object): for idx, rc in enumerate(self.regclasses): rc.index = idx + # The limit on the number of register classes can be changed. It should + # be coordinated with the `RegClassMask` and `RegClassIndex` types in + # `isa/registers.rs`. + assert len(self.regclasses) <= 32, "Too many register classes" + class CPUMode(object): """ diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index a4527733c5..5549e98157 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -20,6 +20,18 @@ pub type RegUnit = u16; /// This type should be coordinated with meta/cdsl/registers.py. pub type RegUnitMask = [u32; 3]; +/// A bit mask indexed by register classes. +/// +/// The size of this type is determined by the ISA with the most register classes. +/// +/// This type should be coordinated with meta/cdsl/isa.py. +pub type RegClassMask = u32; + +/// Guaranteed maximum number of top-level register classes in any ISA. +/// +/// This can be increased, but should be coordinated with meta/cdsl/isa.py. +pub const MAX_TOPRCS: usize = 4; + /// The register units in a target ISA are divided into disjoint register banks. Each bank covers a /// contiguous range of register units. /// @@ -129,7 +141,7 @@ pub struct RegClassData { /// Bit-mask of sub-classes of this register class, including itself. /// /// Bits correspond to RC indexes. - pub subclasses: u32, + pub subclasses: RegClassMask, /// Mask of register units in the class. If `width > 1`, the mask only has a bit set for the /// first register unit in each allocatable register. From d1e37def3f0587dffa55cd4a6524df5bfde113ed Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 16 May 2017 10:46:19 -0700 Subject: [PATCH 0768/3084] Implement ExactSizeIterator for RegSetIter. The set of available registers in a register class is known ahead of time. It can be computed with pop-count. --- lib/cretonne/src/regalloc/allocatable_set.rs | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index 756ad1d349..6ad8019ba9 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -5,8 +5,9 @@ //! "register unit" abstraction. Every register contains one or more register units. Registers that //! share a register unit can't be in use at the same time. -use std::mem::size_of_val; use isa::registers::{RegUnit, RegUnitMask, RegClass}; +use std::iter::ExactSizeIterator; +use std::mem::size_of_val; /// Set of registers available for allocation. #[derive(Clone)] @@ -128,8 +129,15 @@ impl Iterator for RegSetIter { // All of `self.regs` is 0. None } + + fn size_hint(&self) -> (usize, Option) { + let bits = self.regs.iter().map(|&w| w.count_ones() as usize).sum(); + (bits, Some(bits)) + } } +impl ExactSizeIterator for RegSetIter {} + #[cfg(test)] mod tests { use super::*; @@ -162,6 +170,7 @@ mod tests { let mut regs = AllocatableSet::new(); // `GPR` has units 28-36. + assert_eq!(regs.iter(GPR).len(), 8); assert_eq!(regs.iter(GPR).count(), 8); assert_eq!(regs.iter(DPR).collect::>(), [28, 30, 33, 35]); From 1d8efaad83f41afae8188a51eeb6d5c8cd97f5b7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 16 May 2017 10:02:57 -0700 Subject: [PATCH 0769/3084] Add a register pressure tracker. The spilling and reload passes need to ensure that the set of live ranges with register affinity can always be assigned registers. The register pressure tracker can count how many registers are in use for each top-level register class and give guidance on the type of registers that need to be spilled when limits are exceeded. Pressure tracking is extra complicated for the arm32 floating point register bank because there are multiple top-level register classes (S, D, Q) competing for the same register units. --- lib/cretonne/src/regalloc/mod.rs | 1 + lib/cretonne/src/regalloc/pressure.rs | 296 ++++++++++++++++++++++++++ 2 files changed, 297 insertions(+) create mode 100644 lib/cretonne/src/regalloc/pressure.rs diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index fd2e78e9c3..473a9fdb28 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -11,6 +11,7 @@ pub mod coloring; mod affinity; mod context; mod diversion; +mod pressure; mod solver; pub use self::allocatable_set::AllocatableSet; diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs new file mode 100644 index 0000000000..14e4441697 --- /dev/null +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -0,0 +1,296 @@ +//! Register pressure tracking. +//! +//! SSA-based register allocation depends on a spilling phase that "lowers register pressure +//! sufficiently". This module defines the data structures needed to measure register pressure +//! accurately enough to guarantee that the coloring phase will not run out of registers. +//! +//! Ideally, measuring register pressure amounts to simply counting the number of live registers at +//! any given program point. This simplistic method has two problems: +//! +//! 1. Registers are not interchangeable. Most ISAs have separate integer and floating-point +//! register banks, so we need to at least count the number of live registers in each register +//! bank separately. +//! +//! 2. Some ISAs have complicated register aliasing properties. In particular, the 32-bit ARM +//! ISA has a floating-point register bank where two 32-bit registers alias one 64-bit register. +//! This makes it difficult to accurately measure register pressure. +//! +//! This module deals with the problems via *register banks* and *top-level register classes*. +//! Register classes in different register banks are completely independent, so we can count +//! registers in one bank without worrying about the other bank at all. +//! +//! All register classes have a unique top-level register class, and we will count registers for +//! each top-level register class individually. However, a register bank can have multiple +//! top-level register classes that interfere with each other, so all top-level counts need to +//! be considered when determining how many more registers can be allocated. +//! +//! Currently, the only register bank with multiple top-level registers is the `arm32` +//! floating-point register bank which has `S`, `D`, and `Q` top-level classes. + +// Remove once we're using the pressure tracker. +#![allow(dead_code)] + +use isa::registers::{RegInfo, MAX_TOPRCS, RegClass, RegClassMask}; +use regalloc::AllocatableSet; +use std::cmp::min; +use std::iter::ExactSizeIterator; + +/// Information per top-level register class. +/// +/// Everything but the count is static information computed from the constructor arguments. +#[derive(Default)] +struct TopRC { + // Number of registers currently used from this register class. + count: u32, + + // Max number of registers that can be allocated. + limit: u32, + + // Register units per register. + width: u8, + + // The first aliasing top-level RC. + first_toprc: u8, + + // The number of aliasing top-level RCs. + num_toprcs: u8, +} + +pub struct Pressure { + // Bit mask of top-level register classes that are aliased by other top-level register classes. + // Unaliased register classes can use a simpler interference algorithm. + aliased: RegClassMask, + + // Current register counts per top-level register class. + toprc: [TopRC; MAX_TOPRCS], +} + +impl Pressure { + /// Create a new register pressure tracker. + pub fn new(reginfo: &RegInfo, usable: &AllocatableSet) -> Pressure { + let mut p = Pressure { + aliased: 0, + toprc: Default::default(), + }; + + // Get the layout of aliasing top-level register classes from the register banks. + for bank in reginfo.banks { + let first = bank.first_toprc; + let num = bank.num_toprcs; + for rc in &mut p.toprc[first..first + num] { + rc.first_toprc = first as u8; + rc.num_toprcs = num as u8; + } + + // Flag the top-level register classes with aliases. + if num > 1 { + p.aliased |= ((1 << num) - 1) << first; + } + } + + // Compute per-class limits from `usable`. + for (toprc, rc) in p.toprc + .iter_mut() + .take_while(|t| t.num_toprcs > 0) + .zip(reginfo.classes) { + toprc.limit = usable.iter(rc).len() as u32; + toprc.width = rc.width; + } + + p + } + + /// Check for an available register in the register class `rc`. + /// + /// If it is possible to allocate one more register from `rc`'s top-level register class, + /// returns 0. + /// + /// If not, returns a bit-mask of top-level register classes that are interfering. Register + /// pressure should be eased in one of the returned top-level register classes before calling + /// `can_take()` to check again. + pub fn check_avail(&self, rc: RegClass) -> RegClassMask { + let entry = &self.toprc[rc.toprc as usize]; + let mask = 1 << rc.toprc; + if self.aliased & mask == 0 { + // This is a simple unaliased top-level register class. + if entry.count < entry.limit { 0 } else { mask } + } else { + // This is the more complicated case. The top-level register class has aliases. + self.check_avail_aliased(entry) + } + } + + /// Check for an available register in a top-level register class that may have aliases. + /// + /// This is the out-of-line slow path for `check_avail()`. + fn check_avail_aliased(&self, entry: &TopRC) -> RegClassMask { + let first = entry.first_toprc as usize; + let num = entry.num_toprcs as usize; + let width = entry.width as u32; + let ulimit = entry.limit * width; + + // Count up the number of available register units. + let mut units = 0; + for (rc, rci) in self.toprc[first..first + num].iter().zip(first..) { + let rcw = rc.width as u32; + // If `rc.width` is smaller than `width`, each register in `rc` could potentially block + // one of ours. This is assuming that none of the smaller registers are straddling the + // bigger ones. + // + // If `rc.width` is larger than `width`, we are also assuming that the registers are + // aligned and `rc.width` is a multiple of `width`. + let u = if rcw < width { + // We can't take more than the total number of register units in the class. + // This matters for arm32 S-registers which can only ever lock out 16 D-registers. + min(rc.count * width, rc.limit * rcw) + } else { + rc.count * rcw + }; + + // If this top-level RC on its own is responsible for exceeding our limit, return it + // early to guarantee that registers here are spilled before spilling other registers + // unnecessarily. + if u >= ulimit { + return 1 << rci; + } + + units += u; + } + + // We've counted up the worst-case number of register units claimed by all aliasing + // classes. Compare to the unit limit in this class. + if units < ulimit { + 0 + } else { + // Registers need to be spilled from any one of the aliasing classes. + ((1 << num) - 1) << first + } + } + + /// Take a register from `rc`. + /// + /// This assumes that `can_take(rc)` already returned 0. + pub fn take(&mut self, rc: RegClass) { + self.toprc[rc.toprc as usize].count += 1 + } + + /// Free a register in `rc`. + pub fn free(&mut self, rc: RegClass) { + self.toprc[rc.toprc as usize].count -= 1 + } + + /// Reset all counts to 0. + pub fn reset(&mut self) { + for e in self.toprc.iter_mut() { + e.count = 0; + } + } +} + +#[cfg(test)] +mod tests { + use isa::{TargetIsa, RegClass}; + use regalloc::AllocatableSet; + use std::borrow::Borrow; + use super::Pressure; + + // Make an arm32 `TargetIsa`, if possible. + fn arm32() -> Option> { + use settings; + use isa; + + let shared_builder = settings::builder(); + let shared_flags = settings::Flags::new(&shared_builder); + + isa::lookup("arm32").map(|b| b.finish(shared_flags)) + } + + // Get a register class by name. + fn rc_by_name(isa: &TargetIsa, name: &str) -> RegClass { + isa.register_info() + .classes + .iter() + .find(|rc| rc.name == name) + .expect("Can't find named register class.") + } + + #[test] + fn basic_counting() { + let isa = arm32().expect("This test requires arm32 support"); + let isa = isa.borrow(); + let gpr = rc_by_name(isa, "GPR"); + let s = rc_by_name(isa, "S"); + let reginfo = isa.register_info(); + let regs = AllocatableSet::new(); + + let mut pressure = Pressure::new(®info, ®s); + let mut count = 0; + while pressure.check_avail(gpr) == 0 { + pressure.take(gpr); + count += 1; + } + assert_eq!(count, 16); + assert_eq!(pressure.check_avail(gpr), 1 << gpr.toprc); + assert_eq!(pressure.check_avail(s), 0); + pressure.free(gpr); + assert_eq!(pressure.check_avail(gpr), 0); + pressure.take(gpr); + assert_eq!(pressure.check_avail(gpr), 1 << gpr.toprc); + assert_eq!(pressure.check_avail(s), 0); + pressure.reset(); + assert_eq!(pressure.check_avail(gpr), 0); + assert_eq!(pressure.check_avail(s), 0); + } + + #[test] + fn arm_float_bank() { + let isa = arm32().expect("This test requires arm32 support"); + let isa = isa.borrow(); + let s = rc_by_name(isa, "S"); + let d = rc_by_name(isa, "D"); + let q = rc_by_name(isa, "Q"); + let reginfo = isa.register_info(); + let regs = AllocatableSet::new(); + + let mut pressure = Pressure::new(®info, ®s); + assert_eq!(pressure.check_avail(s), 0); + assert_eq!(pressure.check_avail(d), 0); + assert_eq!(pressure.check_avail(q), 0); + + // Allocating a single S-register should not affect availability. + pressure.take(s); + assert_eq!(pressure.check_avail(s), 0); + assert_eq!(pressure.check_avail(d), 0); + assert_eq!(pressure.check_avail(q), 0); + + pressure.take(d); + assert_eq!(pressure.check_avail(s), 0); + assert_eq!(pressure.check_avail(d), 0); + assert_eq!(pressure.check_avail(q), 0); + + pressure.take(q); + assert_eq!(pressure.check_avail(s), 0); + assert_eq!(pressure.check_avail(d), 0); + assert_eq!(pressure.check_avail(q), 0); + + // Take a total of 16 S-regs. + for _ in 1..16 { + pressure.take(s); + } + assert_eq!(pressure.check_avail(s), 0); + assert_eq!(pressure.check_avail(d), 0); + assert_eq!(pressure.check_avail(q), 0); + + // We've taken 16 S, 1 D, and 1 Q. There should be 6 more Qs. + for _ in 0..6 { + assert_eq!(pressure.check_avail(d), 0); + assert_eq!(pressure.check_avail(q), 0); + pressure.take(q); + } + + // We've taken 16 S, 1 D, and 7 Qs. + assert!(pressure.check_avail(s) != 0); + assert_eq!(pressure.check_avail(d), 0); + assert!(pressure.check_avail(q) != 0); + } +} From c826aefa0a8be6a32684d9b3ab021409663b3bc2 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 18 May 2017 18:18:57 -0700 Subject: [PATCH 0770/3084] Start a very simple GVN pass (#79) * Skeleton simple_gvn pass. * Basic testing infrastructure for simple-gvn. * Add can_load and can_store flags to instructions. * Move the replace_values function into the DataFlowGraph. * Make InstructionData derive from Hash, PartialEq, and Eq. * Make EntityList's hash and eq functions panic. * Change Ieee32 and Ieee64 to store u32 and u64, respectively. --- cranelift/docs/testing.rst | 8 +++ cranelift/filetests/simple_gvn/basic.cton | 11 ++++ cranelift/src/filetest/mod.rs | 2 + cranelift/src/filetest/simple_gvn.rs | 51 ++++++++++++++++++ lib/cretonne/meta/base/instructions.py | 30 +++++------ lib/cretonne/meta/cdsl/instructions.py | 4 ++ lib/cretonne/meta/gen_instr.py | 2 +- lib/cretonne/src/context.rs | 17 ++++-- lib/cretonne/src/entity_list.rs | 18 +++++++ lib/cretonne/src/ir/condcodes.rs | 4 +- lib/cretonne/src/ir/dfg.rs | 40 ++++++++++++++ lib/cretonne/src/ir/immediates.rs | 56 ++++++++----------- lib/cretonne/src/ir/instructions.rs | 2 +- lib/cretonne/src/ir/memflags.rs | 2 +- lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/simple_gvn.rs | 65 +++++++++++++++++++++++ 16 files changed, 256 insertions(+), 57 deletions(-) create mode 100644 cranelift/filetests/simple_gvn/basic.cton create mode 100644 cranelift/src/filetest/simple_gvn.rs create mode 100644 lib/cretonne/src/simple_gvn.rs diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index f58a413033..d4aa1a881c 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -344,3 +344,11 @@ sequences. Instead the test will fail. Value locations must be present if they are required to compute the binary bits. Missing value locations will cause the test to crash. + +`test simple-gvn` +----------------- + +Test the simple GVN pass. + +The simple GVN pass is run on each function, and then results are run +through filecheck. diff --git a/cranelift/filetests/simple_gvn/basic.cton b/cranelift/filetests/simple_gvn/basic.cton new file mode 100644 index 0000000000..20544666fc --- /dev/null +++ b/cranelift/filetests/simple_gvn/basic.cton @@ -0,0 +1,11 @@ +test simple-gvn + +function simple_redundancy(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = iadd v0, v1 + v3 = iadd v0, v1 +; check: v3 -> v2 + v4 = imul v2, v3 +; check: v4 = imul $v2, $v3 + return v4 +} diff --git a/cranelift/src/filetest/mod.rs b/cranelift/src/filetest/mod.rs index 7a379ec0fe..961c3ff2f2 100644 --- a/cranelift/src/filetest/mod.rs +++ b/cranelift/src/filetest/mod.rs @@ -20,6 +20,7 @@ mod legalizer; mod regalloc; mod runner; mod runone; +mod simple_gvn; mod verifier; /// The result of running the test in a file. @@ -62,6 +63,7 @@ fn new_subtest(parsed: &TestCommand) -> subtest::Result> { "legalizer" => legalizer::subtest(parsed), "regalloc" => regalloc::subtest(parsed), "binemit" => binemit::subtest(parsed), + "simple-gvn" => simple_gvn::subtest(parsed), _ => Err(format!("unknown test command '{}'", parsed.command)), } } diff --git a/cranelift/src/filetest/simple_gvn.rs b/cranelift/src/filetest/simple_gvn.rs new file mode 100644 index 0000000000..f220cbdde7 --- /dev/null +++ b/cranelift/src/filetest/simple_gvn.rs @@ -0,0 +1,51 @@ +//! Test command for testing the simple GVN pass. +//! +//! The `simple-gvn` test command runs each function through the simple GVN pass after ensuring +//! that all instructions are legal for the target. +//! +//! The resulting function is sent to `filecheck`. + +use cretonne::ir::Function; +use cretonne; +use cton_reader::TestCommand; +use filetest::subtest::{SubTest, Context, Result, run_filecheck}; +use std::borrow::Cow; +use std::fmt::Write; +use utils::pretty_error; + +struct TestSimpleGVN; + +pub fn subtest(parsed: &TestCommand) -> Result> { + assert_eq!(parsed.command, "simple-gvn"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestSimpleGVN)) + } +} + +impl SubTest for TestSimpleGVN { + fn name(&self) -> Cow { + Cow::from("simple-gvn") + } + + fn is_mutating(&self) -> bool { + true + } + + fn run(&self, func: Cow, context: &Context) -> Result<()> { + // Create a compilation context, and drop in the function. + let mut comp_ctx = cretonne::Context::new(); + comp_ctx.func = func.into_owned(); + + comp_ctx.flowgraph(); + comp_ctx + .simple_gvn() + .map_err(|e| pretty_error(&comp_ctx.func, e))?; + + let mut text = String::new(); + write!(&mut text, "{}", &comp_ctx.func) + .map_err(|e| e.to_string())?; + run_filecheck(&text, context) + } +} diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 5777ed76ea..b4458ec585 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -201,7 +201,7 @@ load = Instruction( This is a polymorphic instruction that can load any value type which has a memory representation. """, - ins=(Flags, p, Offset), outs=a) + ins=(Flags, p, Offset), outs=a, can_load=True) store = Instruction( 'store', r""" @@ -210,7 +210,7 @@ store = Instruction( This is a polymorphic instruction that can store any value type with a memory representation. """, - ins=(Flags, x, p, Offset)) + ins=(Flags, x, p, Offset), can_store=True) iExt8 = TypeVar( 'iExt8', 'An integer type with more than 8 bits', @@ -224,7 +224,7 @@ uload8 = Instruction( This is equivalent to ``load.i8`` followed by ``uextend``. """, - ins=(Flags, p, Offset), outs=a) + ins=(Flags, p, Offset), outs=a, can_load=True) sload8 = Instruction( 'sload8', r""" @@ -232,7 +232,7 @@ sload8 = Instruction( This is equivalent to ``load.i8`` followed by ``uextend``. """, - ins=(Flags, p, Offset), outs=a) + ins=(Flags, p, Offset), outs=a, can_load=True) istore8 = Instruction( 'istore8', r""" @@ -240,7 +240,7 @@ istore8 = Instruction( This is equivalent to ``ireduce.i8`` followed by ``store.i8``. """, - ins=(Flags, x, p, Offset)) + ins=(Flags, x, p, Offset), can_store=True) iExt16 = TypeVar( 'iExt16', 'An integer type with more than 16 bits', @@ -254,7 +254,7 @@ uload16 = Instruction( This is equivalent to ``load.i16`` followed by ``uextend``. """, - ins=(Flags, p, Offset), outs=a) + ins=(Flags, p, Offset), outs=a, can_load=True) sload16 = Instruction( 'sload16', r""" @@ -262,7 +262,7 @@ sload16 = Instruction( This is equivalent to ``load.i16`` followed by ``uextend``. """, - ins=(Flags, p, Offset), outs=a) + ins=(Flags, p, Offset), outs=a, can_load=True) istore16 = Instruction( 'istore16', r""" @@ -270,7 +270,7 @@ istore16 = Instruction( This is equivalent to ``ireduce.i16`` followed by ``store.i8``. """, - ins=(Flags, x, p, Offset)) + ins=(Flags, x, p, Offset), can_store=True) iExt32 = TypeVar( 'iExt32', 'An integer type with more than 32 bits', @@ -284,7 +284,7 @@ uload32 = Instruction( This is equivalent to ``load.i32`` followed by ``uextend``. """, - ins=(Flags, p, Offset), outs=a) + ins=(Flags, p, Offset), outs=a, can_load=True) sload32 = Instruction( 'sload32', r""" @@ -292,7 +292,7 @@ sload32 = Instruction( This is equivalent to ``load.i32`` followed by ``uextend``. """, - ins=(Flags, p, Offset), outs=a) + ins=(Flags, p, Offset), outs=a, can_load=True) istore32 = Instruction( 'istore32', r""" @@ -300,7 +300,7 @@ istore32 = Instruction( This is equivalent to ``ireduce.i32`` followed by ``store.i8``. """, - ins=(Flags, x, p, Offset)) + ins=(Flags, x, p, Offset), can_store=True) x = Operand('x', Mem, doc='Value to be stored') a = Operand('a', Mem, doc='Value loaded') @@ -316,7 +316,7 @@ stack_load = Instruction( access cannot go out of bounds, i.e. :math:`sizeof(a) + Offset <= sizeof(SS)`. """, - ins=(SS, Offset), outs=a) + ins=(SS, Offset), outs=a, can_load=True) stack_store = Instruction( 'stack_store', r""" @@ -329,7 +329,7 @@ stack_store = Instruction( access cannot go out of bounds, i.e. :math:`sizeof(a) + Offset <= sizeof(SS)`. """, - ins=(x, SS, Offset)) + ins=(x, SS, Offset), can_store=True) stack_addr = Instruction( 'stack_addr', r""" @@ -357,7 +357,7 @@ heap_load = Instruction( Trap if the heap access would be out of bounds. """, - ins=(p, Offset), outs=a) + ins=(p, Offset), outs=a, can_load=True) heap_store = Instruction( 'heap_store', r""" @@ -365,7 +365,7 @@ heap_store = Instruction( Trap if the heap access would be out of bounds. """, - ins=(x, p, Offset)) + ins=(x, p, Offset), can_store=True) heap_addr = Instruction( 'heap_addr', r""" diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 6724b775ea..d8e9d24e5f 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -85,6 +85,8 @@ class Instruction(object): :param is_call: This is a call instruction. :param is_return: This is a return instruction. :param can_trap: This instruction can trap. + :param can_load: This instruction can load from memory. + :param can_store: This instruction can store to memory. """ # Boolean instruction attributes that can be passed as keyword arguments to @@ -95,6 +97,8 @@ class Instruction(object): 'is_branch': 'True for all branch or jump instructions.', 'is_call': 'Is this a call instruction?', 'is_return': 'Is this a return instruction?', + 'can_load': 'Can this instruction read from memory?', + 'can_store': 'Can this instruction write to memory?', 'can_trap': 'Can this instruction cause a trap?', } diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index f3a4343ab2..548cd45a29 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -228,7 +228,7 @@ def gen_opcodes(groups, fmt): fmt.doc_comment('An instruction opcode.') fmt.doc_comment('') fmt.doc_comment('All instructions from all supported ISAs are present.') - fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug)]') + fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)]') instrs = [] # We explicitly set the discriminant of the first variant to 1, which diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index d8b6af4d0b..ce4b269574 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -17,6 +17,7 @@ use legalize_function; use regalloc; use result::CtonResult; use verifier; +use simple_gvn::do_simple_gvn; /// Persistent data structures and compilation pipeline. pub struct Context { @@ -51,16 +52,16 @@ impl Context { /// /// Also check that the dominator tree and control flow graph are consistent with the function. /// - /// The `TargetIsa` argument is currently unused, but the verifier will soon be able to also + /// The `isa` argument is currently unused, but the verifier will soon be able to also /// check ISA-dependent constraints. - pub fn verify<'a, ISA: Into>>(&self, isa: ISA) -> verifier::Result { - verifier::verify_context(&self.func, &self.cfg, &self.domtree, isa.into()) + pub fn verify<'a>(&self, isa: Option<&TargetIsa>) -> verifier::Result { + verifier::verify_context(&self.func, &self.cfg, &self.domtree, isa) } /// Run the verifier only if the `enable_verifier` setting is true. pub fn verify_if(&self, isa: &TargetIsa) -> CtonResult { if isa.flags().enable_verifier() { - self.verify(isa).map_err(Into::into) + self.verify(Some(isa)).map_err(Into::into) } else { Ok(()) } @@ -78,6 +79,14 @@ impl Context { self.domtree.compute(&self.func, &self.cfg); } + /// Perform simple GVN on the function. + pub fn simple_gvn(&mut self) -> CtonResult { + do_simple_gvn(&mut self.func, &mut self.cfg); + // TODO: Factor things such that we can get a Flags and test + // enable_verifier(). + self.verify(None).map_err(Into::into) + } + /// Run the register allocator. pub fn regalloc(&mut self, isa: &TargetIsa) -> CtonResult { self.regalloc diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index b6cf3099eb..5121d8e4cc 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -46,6 +46,7 @@ //! The index stored in an `EntityList` points to part 2, the list elements. The value 0 is //! reserved for the empty list which isn't allocated in the vector. +use std::hash::{Hash, Hasher}; use std::marker::PhantomData; use std::mem; @@ -59,6 +60,10 @@ use entity_map::EntityRef; /// Entity lists can be cloned, but that operation should only be used as part of cloning the whole /// function they belong to. *Cloning an entity list does not allocate new memory for the clone*. /// It creates an alias of the same memory. +/// +/// Entity lists can also be hashed and compared for equality, but those operations just panic if, +/// they're ever actually called, because it's not possible to compare the contents of the list +/// without the pool reference. #[derive(Clone, Debug)] pub struct EntityList { index: u32, @@ -75,6 +80,19 @@ impl Default for EntityList { } } +impl Hash for EntityList { + fn hash(&self, _: &mut H) { + panic!("hash called on EntityList"); + } +} + +impl PartialEq for EntityList { + fn eq(&self, _: &EntityList) -> bool { + panic!("eq called on EntityList"); + } +} +impl Eq for EntityList {} + /// A memory pool for storing lists of `T`. #[derive(Clone)] pub struct ListPool { diff --git a/lib/cretonne/src/ir/condcodes.rs b/lib/cretonne/src/ir/condcodes.rs index 5117e39d9d..520014471d 100644 --- a/lib/cretonne/src/ir/condcodes.rs +++ b/lib/cretonne/src/ir/condcodes.rs @@ -27,7 +27,7 @@ pub trait CondCode: Copy { /// This condition code is used by the `icmp` instruction to compare integer values. There are /// separate codes for comparing the integers as signed or unsigned numbers where it makes a /// difference. -#[derive(Clone, Copy, PartialEq, Eq, Debug)] +#[derive(Clone, Copy, PartialEq, Eq, Debug, Hash)] pub enum IntCC { /// `==`. Equal, @@ -139,7 +139,7 @@ impl FromStr for IntCC { /// The condition codes described here are used to produce a single boolean value from the /// comparison. The 14 condition codes here cover every possible combination of the relation above /// except the impossible `!UN & !EQ & !LT & !GT` and the always true `UN | EQ | LT | GT`. -#[derive(Clone, Copy, PartialEq, Eq, Debug)] +#[derive(Clone, Copy, PartialEq, Eq, Debug, Hash)] pub enum FloatCC { /// EQ | LT | GT Ordered, diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index d5eb5ab73d..81c3142680 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -243,6 +243,46 @@ impl DataFlowGraph { self.values[dest] = ValueData::Alias { ty, original }; } + /// Replace the results of one instruction with aliases to the results of another. + /// + /// Change all the results of `dest_inst` to behave as aliases of + /// corresponding results of `src_inst`, as if calling change_to_alias for + /// each. + /// + /// After calling this instruction, `dest_inst` will have had its results + /// cleared, so it likely needs to be removed from the graph. + /// + pub fn replace_with_aliases(&mut self, dest_inst: Inst, src_inst: Inst) { + debug_assert_ne!(dest_inst, + src_inst, + "Replacing {} with itself would create a loop", + dest_inst); + debug_assert_eq!(self.results[dest_inst].len(&self.value_lists), + self.results[src_inst].len(&self.value_lists), + "Replacing {} with {} would produce a different number of results.", + dest_inst, + src_inst); + + for (&dest, &src) in self.results[dest_inst] + .as_slice(&self.value_lists) + .iter() + .zip(self.results[src_inst].as_slice(&self.value_lists)) { + let original = src; + let ty = self.value_type(original); + assert_eq!(self.value_type(dest), + ty, + "Aliasing {} to {} would change its type {} to {}", + dest, + src, + self.value_type(dest), + ty); + + self.values[dest] = ValueData::Alias { ty, original }; + } + + self.clear_results(dest_inst); + } + /// Create a new value alias. /// /// Note that this function should only be called by the parser. diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index e061b88482..44c4cd3fc3 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -14,7 +14,7 @@ use std::str::FromStr; /// /// An `Imm64` operand can also be used to represent immediate values of smaller integer types by /// sign-extending to `i64`. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)] pub struct Imm64(i64); impl Imm64 { @@ -153,7 +153,7 @@ pub type Uimm8 = u8; /// /// This is used to encode an immediate offset for load/store instructions. All supported ISAs have /// a maximum load/store offset that fits in an `i32`. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)] pub struct Offset32(i32); impl Offset32 { @@ -220,7 +220,7 @@ impl FromStr for Offset32 { /// 32-bit unsigned immediate offset. /// /// This is used to encode an immediate offset for WebAssembly heap_load/heap_store instructions. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)] pub struct Uoffset32(u32); impl Uoffset32 { @@ -282,17 +282,19 @@ impl FromStr for Uoffset32 { } } -/// An IEEE binary32 immediate floating point value. +/// An IEEE binary32 immediate floating point value, represented as a u32 +/// containing the bitpattern. /// /// All bit patterns are allowed. -#[derive(Copy, Clone, Debug)] -pub struct Ieee32(f32); +#[derive(Copy, Clone, Debug, Eq, PartialEq, Hash)] +pub struct Ieee32(u32); -/// An IEEE binary64 immediate floating point value. +/// An IEEE binary64 immediate floating point value, represented as a u64 +/// containing the bitpattern. /// /// All bit patterns are allowed. -#[derive(Copy, Clone, Debug)] -pub struct Ieee64(f64); +#[derive(Copy, Clone, Debug, Eq, PartialEq, Hash)] +pub struct Ieee64(u64); // Format a floating point number in a way that is reasonably human-readable, and that can be // converted back to binary without any rounding issues. The hexadecimal formatting of normal and @@ -531,18 +533,13 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { impl Ieee32 { /// Create a new `Ieee32` representing the number `x`. pub fn new(x: f32) -> Ieee32 { - Ieee32(x) - } - - /// Construct `Ieee32` immediate from raw bits. - pub fn from_bits(x: u32) -> Ieee32 { Ieee32(unsafe { mem::transmute(x) }) } } impl Display for Ieee32 { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - let bits: u32 = unsafe { mem::transmute(self.0) }; + let bits: u32 = self.0; format_float(bits as u64, 8, 23, f) } } @@ -552,7 +549,7 @@ impl FromStr for Ieee32 { fn from_str(s: &str) -> Result { match parse_float(s, 8, 23) { - Ok(b) => Ok(Ieee32::from_bits(b as u32)), + Ok(b) => Ok(Ieee32(b as u32)), Err(s) => Err(s), } } @@ -561,18 +558,13 @@ impl FromStr for Ieee32 { impl Ieee64 { /// Create a new `Ieee64` representing the number `x`. pub fn new(x: f64) -> Ieee64 { - Ieee64(x) - } - - /// Construct `Ieee64` immediate from raw bits. - pub fn from_bits(x: u64) -> Ieee64 { Ieee64(unsafe { mem::transmute(x) }) } } impl Display for Ieee64 { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - let bits: u64 = unsafe { mem::transmute(self.0) }; + let bits: u64 = self.0; format_float(bits, 11, 52, f) } } @@ -582,7 +574,7 @@ impl FromStr for Ieee64 { fn from_str(s: &str) -> Result { match parse_float(s, 11, 52) { - Ok(b) => Ok(Ieee64::from_bits(b)), + Ok(b) => Ok(Ieee64(b)), Err(s) => Err(s), } } @@ -743,11 +735,11 @@ mod tests { assert_eq!(Ieee32::new(f32::NAN).to_string(), "+NaN"); assert_eq!(Ieee32::new(-f32::NAN).to_string(), "-NaN"); // Construct some qNaNs with payloads. - assert_eq!(Ieee32::from_bits(0x7fc00001).to_string(), "+NaN:0x1"); - assert_eq!(Ieee32::from_bits(0x7ff00001).to_string(), "+NaN:0x300001"); + assert_eq!(Ieee32(0x7fc00001).to_string(), "+NaN:0x1"); + assert_eq!(Ieee32(0x7ff00001).to_string(), "+NaN:0x300001"); // Signaling NaNs. - assert_eq!(Ieee32::from_bits(0x7f800001).to_string(), "+sNaN:0x1"); - assert_eq!(Ieee32::from_bits(0x7fa00001).to_string(), "+sNaN:0x200001"); + assert_eq!(Ieee32(0x7f800001).to_string(), "+sNaN:0x1"); + assert_eq!(Ieee32(0x7fa00001).to_string(), "+sNaN:0x200001"); } #[test] @@ -845,14 +837,12 @@ mod tests { assert_eq!(Ieee64::new(f64::NAN).to_string(), "+NaN"); assert_eq!(Ieee64::new(-f64::NAN).to_string(), "-NaN"); // Construct some qNaNs with payloads. - assert_eq!(Ieee64::from_bits(0x7ff8000000000001).to_string(), - "+NaN:0x1"); - assert_eq!(Ieee64::from_bits(0x7ffc000000000001).to_string(), + assert_eq!(Ieee64(0x7ff8000000000001).to_string(), "+NaN:0x1"); + assert_eq!(Ieee64(0x7ffc000000000001).to_string(), "+NaN:0x4000000000001"); // Signaling NaNs. - assert_eq!(Ieee64::from_bits(0x7ff0000000000001).to_string(), - "+sNaN:0x1"); - assert_eq!(Ieee64::from_bits(0x7ff4000000000001).to_string(), + assert_eq!(Ieee64(0x7ff0000000000001).to_string(), "+sNaN:0x1"); + assert_eq!(Ieee64(0x7ff4000000000001).to_string(), "+sNaN:0x4000000000001"); } diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 4301c7796c..66c6d039b0 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -98,7 +98,7 @@ impl FromStr for Opcode { /// value should have its `ty` field set to `VOID`. The size of `InstructionData` should be kept at /// 16 bytes on 64-bit architectures. If more space is needed to represent an instruction, use a /// `Box` to store the additional information out of line. -#[derive(Clone, Debug)] +#[derive(Clone, Debug, Hash, PartialEq, Eq)] #[allow(missing_docs)] pub enum InstructionData { Nullary { opcode: Opcode }, diff --git a/lib/cretonne/src/ir/memflags.rs b/lib/cretonne/src/ir/memflags.rs index bd2f2bab19..8d0694e1a0 100644 --- a/lib/cretonne/src/ir/memflags.rs +++ b/lib/cretonne/src/ir/memflags.rs @@ -14,7 +14,7 @@ const NAMES: [&str; 2] = ["notrap", "aligned"]; /// Each of these flags introduce a limited form of undefined behavior. The flags each enable /// certain optimizations that need to make additional assumptions. Generally, the semantics of a /// program does not change when a flag is removed, but adding a flag will. -#[derive(Clone, Copy, Debug)] +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] pub struct MemFlags { bits: u8, } diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 1f51b38d3f..264e808cbf 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -35,5 +35,6 @@ mod packed_option; mod partition_slice; mod predicates; mod ref_slice; +mod simple_gvn; mod topo_order; mod write; diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs new file mode 100644 index 0000000000..cfd13128fe --- /dev/null +++ b/lib/cretonne/src/simple_gvn.rs @@ -0,0 +1,65 @@ +//! A simple GVN pass. + +use flowgraph::ControlFlowGraph; +use dominator_tree::DominatorTree; +use ir::{Cursor, InstructionData, Function, Inst, Opcode}; +use std::collections::HashMap; + +/// Test whether the given opcode is unsafe to even consider for GVN. +fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { + opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || opcode.is_return() || + opcode.can_trap() +} + +/// Perform simple GVN on `func`. +/// +pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph) { + let mut visible_values: HashMap = HashMap::new(); + + let domtree = DominatorTree::with_function(func, &cfg); + + // Visit EBBs in a reverse post-order. + let mut postorder = cfg.postorder_ebbs(); + let mut pos = Cursor::new(&mut func.layout); + + while let Some(ebb) = postorder.pop() { + pos.goto_top(ebb); + + while let Some(inst) = pos.next_inst() { + let opcode = func.dfg[inst].opcode(); + + if trivially_unsafe_for_gvn(opcode) { + continue; + } + + // TODO: Implement simple redundant-load elimination. + if opcode.can_store() { + continue; + } + if opcode.can_load() { + continue; + } + + let key = func.dfg[inst].clone(); + let entry = visible_values.entry(key); + use std::collections::hash_map::Entry::*; + match entry { + Occupied(mut entry) => { + if domtree.dominates(*entry.get(), inst, &pos.layout) { + func.dfg.replace_with_aliases(inst, *entry.get()); + pos.remove_inst(); + } else { + // The prior instruction doesn't dominate inst, so it + // won't dominate any subsequent instructions we'll + // visit, so just replace it. + *entry.get_mut() = inst; + continue; + } + } + Vacant(entry) => { + entry.insert(inst); + } + } + } + } +} From 0e023e97ea088ae2f6700b110139b2721fcf9467 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 25 May 2017 16:37:31 -0700 Subject: [PATCH 0771/3084] Fix more GVN issues (#83) * Fix GVN skipping the instruction after a deleted instruction. * Teach GVN to resolve aliases as it proceeds. * Clean up an obsolete reference to extended_values. --- cranelift/filetests/simple_gvn/basic.cton | 14 ++++++-- lib/cretonne/src/ir/dfg.rs | 42 +++++++++++++++++------ lib/cretonne/src/ir/layout.rs | 12 +++++++ lib/cretonne/src/simple_gvn.rs | 5 ++- 4 files changed, 59 insertions(+), 14 deletions(-) diff --git a/cranelift/filetests/simple_gvn/basic.cton b/cranelift/filetests/simple_gvn/basic.cton index 20544666fc..06d0989d1e 100644 --- a/cranelift/filetests/simple_gvn/basic.cton +++ b/cranelift/filetests/simple_gvn/basic.cton @@ -4,8 +4,18 @@ function simple_redundancy(i32, i32) -> i32 { ebb0(v0: i32, v1: i32): v2 = iadd v0, v1 v3 = iadd v0, v1 -; check: v3 -> v2 v4 = imul v2, v3 -; check: v4 = imul $v2, $v3 +; check: v4 = imul $v2, $v2 return v4 } + +function cascading_redundancy(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = iadd v0, v1 + v3 = iadd v0, v1 + v4 = imul v2, v3 + v5 = imul v2, v2 + v6 = iadd v4, v5 +; check: v6 = iadd $v4, $v4 + return v6 +} diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 81c3142680..35d7a18fa4 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -106,6 +106,23 @@ impl DataFlowGraph { } } +/// Resolve value aliases. +/// +/// Find the original SSA value that `value` aliases. +fn resolve_aliases(values: &EntityMap, value: Value) -> Value { + let mut v = value; + + // Note that values may be empty here. + for _ in 0..1 + values.len() { + if let ValueData::Alias { original, .. } = values[v] { + v = original; + } else { + return v; + } + } + panic!("Value alias loop detected for {}", value); +} + /// Handling values. /// /// Values are either EBB arguments or instruction results. @@ -176,17 +193,7 @@ impl DataFlowGraph { /// /// Find the original SSA value that `value` aliases. pub fn resolve_aliases(&self, value: Value) -> Value { - let mut v = value; - - // Note that extended_values may be empty here. - for _ in 0..1 + self.values.len() { - if let ValueData::Alias { original, .. } = self.values[v] { - v = original; - } else { - return v; - } - } - panic!("Value alias loop detected for {}", value); + resolve_aliases(&self.values, value) } /// Resolve value copies. @@ -216,6 +223,19 @@ impl DataFlowGraph { panic!("Copy loop detected for {}", value); } + /// Resolve all aliases among inst's arguments. + /// + /// For each argument of inst which is defined by an alias, replace the + /// alias with the aliased value. + pub fn resolve_aliases_in_arguments(&mut self, inst: Inst) { + for arg in self.insts[inst].arguments_mut(&mut self.value_lists) { + let resolved = resolve_aliases(&self.values, *arg); + if resolved != *arg { + *arg = resolved; + } + } + } + /// Turn a value into an alias of another. /// /// Change the `dest` value to behave as an alias of `src`. This means that all uses of `dest` diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 4e6f738653..06ab92612b 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -923,6 +923,18 @@ impl<'f> Cursor<'f> { inst } + /// Remove the instruction under the cursor. + /// + /// The cursor is left pointing at the position preceding the current instruction. + /// + /// Return the instruction that was removed. + pub fn remove_inst_and_step_back(&mut self) -> Inst { + let inst = self.current_inst().expect("No instruction to remove"); + self.prev_inst(); + self.layout.remove_inst(inst); + inst + } + /// Insert an EBB at the current position and switch to it. /// /// As far as possible, this method behaves as if the EBB header were an instruction inserted diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index cfd13128fe..a7665e965d 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -28,6 +28,9 @@ pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph) { while let Some(inst) = pos.next_inst() { let opcode = func.dfg[inst].opcode(); + // Resolve aliases, particularly aliases we created earlier. + func.dfg.resolve_aliases_in_arguments(inst); + if trivially_unsafe_for_gvn(opcode) { continue; } @@ -47,7 +50,7 @@ pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph) { Occupied(mut entry) => { if domtree.dominates(*entry.get(), inst, &pos.layout) { func.dfg.replace_with_aliases(inst, *entry.get()); - pos.remove_inst(); + pos.remove_inst_and_step_back(); } else { // The prior instruction doesn't dominate inst, so it // won't dominate any subsequent instructions we'll From 4f26764e7149be06f278e3cf6cdcb6d3ec20a872 Mon Sep 17 00:00:00 2001 From: Aleksey Kuznetsov Date: Thu, 1 Jun 2017 20:48:36 +0500 Subject: [PATCH 0772/3084] Remove unnecessary cloned() in reader::lexer::trailing_digits() --- lib/reader/src/lexer.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index c5c2aa5d80..bbbb023490 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -87,8 +87,7 @@ fn trailing_digits(s: &str) -> usize { s.as_bytes() .iter() .rev() - .cloned() - .take_while(|&b| b'0' <= b && b <= b'9') + .take_while(|&&b| b'0' <= b && b <= b'9') .count() } From b02ccea8dc601783fe93419ea757c04eb2d79423 Mon Sep 17 00:00:00 2001 From: Denis Merigoux Date: Fri, 2 Jun 2017 15:00:29 -0700 Subject: [PATCH 0773/3084] Loop analysis of the IL * Implemented in two passes * First pass discovers the loops headers (they dominate one of their predecessors) * Second pass traverses the blocks of each loop * Discovers the loop tree structure * Offers a new LoopAnalysis data structure queried from outside the module --- lib/cretonne/src/context.rs | 5 + lib/cretonne/src/entity_list.rs | 2 +- lib/cretonne/src/entity_map.rs | 20 +- lib/cretonne/src/ir/dfg.rs | 1 - lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/loop_analysis.rs | 336 ++++++++++++++++++++++++++++++ 6 files changed, 360 insertions(+), 5 deletions(-) create mode 100644 lib/cretonne/src/loop_analysis.rs diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index ce4b269574..9b0b3b31a7 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -12,6 +12,7 @@ use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir::Function; +use loop_analysis::LoopAnalysis; use isa::TargetIsa; use legalize_function; use regalloc; @@ -32,6 +33,9 @@ pub struct Context { /// Register allocation context. pub regalloc: regalloc::Context, + + /// Loop analysis of `func`. + pub loop_analysis: LoopAnalysis, } impl Context { @@ -45,6 +49,7 @@ impl Context { cfg: ControlFlowGraph::new(), domtree: DominatorTree::new(), regalloc: regalloc::Context::new(), + loop_analysis: LoopAnalysis::new(), } } diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index 5121d8e4cc..946bb7ba04 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -94,7 +94,7 @@ impl PartialEq for EntityList { impl Eq for EntityList {} /// A memory pool for storing lists of `T`. -#[derive(Clone)] +#[derive(Clone, Debug)] pub struct ListPool { // The main array containing the lists. data: Vec, diff --git a/lib/cretonne/src/entity_map.rs b/lib/cretonne/src/entity_map.rs index cee4215a4e..8227a19fb6 100644 --- a/lib/cretonne/src/entity_map.rs +++ b/lib/cretonne/src/entity_map.rs @@ -71,7 +71,7 @@ impl EntityMap pub fn keys(&self) -> Keys { Keys { pos: 0, - len: self.elems.len(), + rev_pos: self.elems.len(), unused: PhantomData, } } @@ -183,7 +183,7 @@ pub struct Keys where K: EntityRef { pos: usize, - len: usize, + rev_pos: usize, unused: PhantomData, } @@ -193,7 +193,7 @@ impl Iterator for Keys type Item = K; fn next(&mut self) -> Option { - if self.pos < self.len { + if self.pos < self.rev_pos { let k = K::new(self.pos); self.pos += 1; Some(k) @@ -203,6 +203,20 @@ impl Iterator for Keys } } +impl DoubleEndedIterator for Keys + where K: EntityRef +{ + fn next_back(&mut self) -> Option { + if self.rev_pos > self.pos { + let k = K::new(self.rev_pos - 1); + self.rev_pos -= 1; + Some(k) + } else { + None + } + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 35d7a18fa4..c3c5422e75 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -8,7 +8,6 @@ use ir::layout::Cursor; use ir::types; use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueList, ValueListPool}; use write::write_operands; - use std::fmt; use std::iter; use std::ops::{Index, IndexMut}; diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 264e808cbf..5ac263d4ff 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -20,6 +20,7 @@ pub mod entity_map; pub mod flowgraph; pub mod ir; pub mod isa; +pub mod loop_analysis; pub mod regalloc; pub mod result; pub mod settings; diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs new file mode 100644 index 0000000000..1e36ced8f9 --- /dev/null +++ b/lib/cretonne/src/loop_analysis.rs @@ -0,0 +1,336 @@ +//! A loop analysis represented as mappings of loops to their header Ebb +//! and parent in the loop tree. + +use ir::{Function, Ebb, Layout}; +use flowgraph::ControlFlowGraph; +use dominator_tree::DominatorTree; +use entity_map::{EntityMap, PrimaryEntityData}; +use packed_option::{PackedOption, ReservedValue}; +use entity_map::{EntityRef, Keys}; +use std::u32; + +/// A opaque reference to a code loop. +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +pub struct Loop(u32); +impl EntityRef for Loop { + fn new(index: usize) -> Self { + assert!(index < (u32::MAX as usize)); + Loop(index as u32) + } + + fn index(self) -> usize { + self.0 as usize + } +} + +impl ReservedValue for Loop { + fn reserved_value() -> Loop { + Loop(u32::MAX) + } +} + + +/// Loop tree information for a single function. +/// +/// Loops are referenced by the Loop object, and for each loop you can access its header EBB, +/// its eventual parent in the loop tree and all the EBB belonging to the loop. +pub struct LoopAnalysis { + loops: EntityMap, + ebb_loop_map: EntityMap>, +} + +struct LoopData { + header: Ebb, + parent: PackedOption, +} + +impl PrimaryEntityData for LoopData {} + +impl LoopData { + /// Creates a `LoopData` object with the loop header and its eventual parent in the loop tree. + pub fn new(header: Ebb, parent: Option) -> LoopData { + LoopData { + header: header, + parent: parent.into(), + } + } +} + +/// Methods for querying the loop analysis. +impl LoopAnalysis { + /// Allocate a new blank loop analysis struct. Use `compute` to compute the loop analysis for + /// a function. + pub fn new() -> LoopAnalysis { + LoopAnalysis { + loops: EntityMap::new(), + ebb_loop_map: EntityMap::new(), + } + } + + /// Returns all the loops contained in a function. + pub fn loops(&self) -> Keys { + self.loops.keys() + } + + /// Returns the header EBB of a particular loop. + /// + /// The characteristic property of a loop header block is that it dominates some of its + /// predecessors. + pub fn loop_header(&self, lp: Loop) -> Ebb { + self.loops[lp].header + } + + /// Return the eventual parent of a loop in the loop tree. + pub fn loop_parent(&self, lp: Loop) -> Option { + self.loops[lp].parent.expand() + } + + /// Determine if an Ebb belongs to a loop by running a finger along the loop tree. + /// + /// Returns `true` if `ebb` is in loop `lp`. + pub fn is_in_loop(&self, ebb: Ebb, lp: Loop) -> bool { + let ebb_loop = self.ebb_loop_map[ebb]; + match ebb_loop.expand() { + None => false, + Some(ebb_loop) => self.is_child_loop(ebb_loop, lp), + } + } + + /// Determines if a loop is contained in another loop. + /// + /// `is_child_loop(child,parent)` returns `true` if and only if `child` is a child loop of + /// `parent` (or `child == parent`). + pub fn is_child_loop(&self, child: Loop, parent: Loop) -> bool { + let mut finger = Some(child); + while let Some(finger_loop) = finger { + if finger_loop == parent { + return true; + } + finger = self.loop_parent(finger_loop); + } + false + } +} + +impl LoopAnalysis { + /// Detects the loops in a function. Needs the control flow graph and the dominator tree. + pub fn compute(&mut self, func: &Function, cfg: &ControlFlowGraph, domtree: &DominatorTree) { + self.loops.clear(); + self.ebb_loop_map.clear(); + self.ebb_loop_map.resize(func.dfg.num_ebbs()); + self.find_loop_headers(cfg, domtree, &func.layout); + self.discover_loop_blocks(cfg, domtree, &func.layout) + } + + // Traverses the CFG in reverse postorder and create a loop object for every EBB having a + // back edge. + fn find_loop_headers(&mut self, + cfg: &ControlFlowGraph, + domtree: &DominatorTree, + layout: &Layout) { + // We traverse the CFg in reverse postorder + for ebb in cfg.postorder_ebbs().iter().rev() { + for &(_, pred_inst) in cfg.get_predecessors(*ebb) { + // If the ebb dominates one of its predecessors it is a back edge + if domtree.ebb_dominates(ebb.clone(), pred_inst, layout) { + // This ebb is a loop header, so we create its associated loop + let lp = self.loops.push(LoopData::new(*ebb, None)); + self.ebb_loop_map[*ebb] = lp.into(); + break; + // We break because we only need one back edge to identify a loop header. + } + } + } + } + + // Intended to be called after `find_loop_headers`. For each detected loop header, + // discovers all the ebb belonging to the loop and its inner loops. After a call to this + // function, the loop tree is fully constructed. + fn discover_loop_blocks(&mut self, + cfg: &ControlFlowGraph, + domtree: &DominatorTree, + layout: &Layout) { + let mut stack: Vec = Vec::new(); + // We handle each loop header in reverse order, corresponding to a pesudo postorder + // traversal of the graph. + for lp in self.loops().rev() { + for &(pred, pred_inst) in cfg.get_predecessors(self.loops[lp].header) { + // We follow the back edges + if domtree.ebb_dominates(self.loops[lp].header, pred_inst, layout) { + stack.push(pred); + } + } + while let Some(node) = stack.pop() { + let continue_dfs: Option; + match self.ebb_loop_map[node].expand() { + None => { + // The node hasn't been visited yet, we tag it as part of the loop + self.ebb_loop_map[node] = PackedOption::from(lp); + continue_dfs = Some(node); + } + Some(node_loop) => { + // We copy the node_loop into a mutable reference passed along the while + let mut node_loop = node_loop; + // The node is part of a loop, which can be lp or an inner loop + let mut node_loop_parent_option = self.loops[node_loop].parent; + while let Some(node_loop_parent) = node_loop_parent_option.expand() { + if node_loop_parent == lp { + // We have encounterd lp so we stop (already visited) + break; + } else { + // + node_loop = node_loop_parent; + // We lookup the parent loop + node_loop_parent_option = self.loops[node_loop].parent; + } + } + // Now node_loop_parent is either: + // - None and node_loop is an new inner loop of lp + // - Some(...) and the initial node_loop was a known inner loop of lp + match node_loop_parent_option.expand() { + Some(_) => continue_dfs = None, + None => { + if node_loop != lp { + self.loops[node_loop].parent = lp.into(); + continue_dfs = Some(self.loops[node_loop].header) + } else { + // If lp is a one-block loop then we make sure we stop + continue_dfs = None + } + } + } + } + } + // Now we have handled the popped node and need to continue the DFS by adding the + // predecessors of that node + if let Some(continue_dfs) = continue_dfs { + for &(pred, _) in cfg.get_predecessors(continue_dfs) { + stack.push(pred) + } + } + } + + } + } +} + +#[cfg(test)] +mod test { + + use ir::{Function, InstBuilder, Cursor, types}; + use loop_analysis::{Loop, LoopAnalysis}; + use flowgraph::ControlFlowGraph; + use dominator_tree::DominatorTree; + + #[test] + fn nested_loops_detection() { + let mut func = Function::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + let ebb2 = func.dfg.make_ebb(); + let ebb3 = func.dfg.make_ebb(); + let cond = func.dfg.append_ebb_arg(ebb0, types::I32); + + { + let dfg = &mut func.dfg; + let cur = &mut Cursor::new(&mut func.layout); + + cur.insert_ebb(ebb0); + dfg.ins(cur).jump(ebb1, &[]); + + cur.insert_ebb(ebb1); + dfg.ins(cur).jump(ebb2, &[]); + + cur.insert_ebb(ebb2); + dfg.ins(cur).brnz(cond, ebb1, &[]); + dfg.ins(cur).jump(ebb3, &[]); + + cur.insert_ebb(ebb3); + dfg.ins(cur).brnz(cond, ebb0, &[]); + + } + + let mut loop_analysis = LoopAnalysis::new(); + let mut cfg = ControlFlowGraph::new(); + let mut domtree = DominatorTree::new(); + cfg.compute(&func); + domtree.compute(&func, &cfg); + loop_analysis.compute(&func, &cfg, &domtree); + + let loops = loop_analysis.loops().collect::>(); + assert_eq!(loops.len(), 2); + assert_eq!(loop_analysis.loop_header(loops[0]), ebb0); + assert_eq!(loop_analysis.loop_header(loops[1]), ebb1); + assert_eq!(loop_analysis.loop_parent(loops[1]), Some(loops[0])); + assert_eq!(loop_analysis.loop_parent(loops[0]), None); + assert_eq!(loop_analysis.is_in_loop(ebb0, loops[0]), true); + assert_eq!(loop_analysis.is_in_loop(ebb0, loops[1]), false); + assert_eq!(loop_analysis.is_in_loop(ebb1, loops[1]), true); + assert_eq!(loop_analysis.is_in_loop(ebb1, loops[0]), true); + assert_eq!(loop_analysis.is_in_loop(ebb2, loops[1]), true); + assert_eq!(loop_analysis.is_in_loop(ebb2, loops[0]), true); + assert_eq!(loop_analysis.is_in_loop(ebb3, loops[0]), true); + assert_eq!(loop_analysis.is_in_loop(ebb0, loops[1]), false); + } + + #[test] + fn complex_loop_detection() { + let mut func = Function::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + let ebb2 = func.dfg.make_ebb(); + let ebb3 = func.dfg.make_ebb(); + let ebb4 = func.dfg.make_ebb(); + let ebb5 = func.dfg.make_ebb(); + let cond = func.dfg.append_ebb_arg(ebb0, types::I32); + + { + let dfg = &mut func.dfg; + let cur = &mut Cursor::new(&mut func.layout); + + cur.insert_ebb(ebb0); + dfg.ins(cur).brnz(cond, ebb1, &[]); + dfg.ins(cur).jump(ebb3, &[]); + + cur.insert_ebb(ebb1); + dfg.ins(cur).jump(ebb2, &[]); + + cur.insert_ebb(ebb2); + dfg.ins(cur).brnz(cond, ebb1, &[]); + dfg.ins(cur).jump(ebb5, &[]); + + cur.insert_ebb(ebb3); + dfg.ins(cur).jump(ebb4, &[]); + + cur.insert_ebb(ebb4); + dfg.ins(cur).brnz(cond, ebb3, &[]); + dfg.ins(cur).jump(ebb5, &[]); + + cur.insert_ebb(ebb5); + dfg.ins(cur).brnz(cond, ebb0, &[]); + + } + + let mut loop_analysis = LoopAnalysis::new(); + let mut cfg = ControlFlowGraph::new(); + let mut domtree = DominatorTree::new(); + cfg.compute(&func); + domtree.compute(&func, &cfg); + loop_analysis.compute(&func, &cfg, &domtree); + + let loops = loop_analysis.loops().collect::>(); + assert_eq!(loops.len(), 3); + assert_eq!(loop_analysis.loop_header(loops[0]), ebb0); + assert_eq!(loop_analysis.loop_header(loops[1]), ebb1); + assert_eq!(loop_analysis.loop_header(loops[2]), ebb3); + assert_eq!(loop_analysis.loop_parent(loops[1]), Some(loops[0])); + assert_eq!(loop_analysis.loop_parent(loops[2]), Some(loops[0])); + assert_eq!(loop_analysis.loop_parent(loops[0]), None); + assert_eq!(loop_analysis.is_in_loop(ebb0, loops[0]), true); + assert_eq!(loop_analysis.is_in_loop(ebb1, loops[1]), true); + assert_eq!(loop_analysis.is_in_loop(ebb2, loops[1]), true); + assert_eq!(loop_analysis.is_in_loop(ebb3, loops[2]), true); + assert_eq!(loop_analysis.is_in_loop(ebb4, loops[2]), true); + assert_eq!(loop_analysis.is_in_loop(ebb5, loops[0]), true); + } +} From 16df1f1cf521c95c16b1a5f8b9294216b60868bd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 2 Jun 2017 15:08:02 -0700 Subject: [PATCH 0774/3084] Compute a CFG post-order when building the dominator tree. The DominatorTree has existing DomNodes per EBB that can be used in lieu of expensive HastSets for the depth-first traversal of the CFG. Make the computed and cached post-order available for other passes through the `cfg_postorder()` method which returns a slice. The post-order algorithm is essentially the same as the one in ControlFlowGraph::postorder_ebbs(), except it will never push a successor node that has already been visited once. This is more efficient, but it generates a different post-order. Change the cfg_traversal tests to check this new algorithm. --- cranelift/tests/cfg_traversal.rs | 37 +++++----- lib/cretonne/src/dominator_tree.rs | 115 +++++++++++++++++++++++------ 2 files changed, 111 insertions(+), 41 deletions(-) diff --git a/cranelift/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs index 903b1a1ec1..21e2a086b2 100644 --- a/cranelift/tests/cfg_traversal.rs +++ b/cranelift/tests/cfg_traversal.rs @@ -1,30 +1,27 @@ extern crate cretonne; extern crate cton_reader; -use self::cretonne::entity_map::EntityMap; use self::cretonne::flowgraph::ControlFlowGraph; +use self::cretonne::dominator_tree::DominatorTree; use self::cretonne::ir::Ebb; use self::cton_reader::parse_functions; fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) { let func = &parse_functions(function_source).unwrap()[0]; let cfg = ControlFlowGraph::with_function(&func); - let ebbs = ebb_order + let domtree = DominatorTree::with_function(&func, &cfg); + + let got = domtree + .cfg_postorder() .iter() - .map(|n| Ebb::with_number(*n).unwrap()) - .collect::>(); - - let mut postorder_ebbs = cfg.postorder_ebbs(); - let mut postorder_map = EntityMap::with_capacity(postorder_ebbs.len()); - for (i, ebb) in postorder_ebbs.iter().enumerate() { - postorder_map[ebb.clone()] = i + 1; - } - postorder_ebbs.reverse(); - - assert_eq!(postorder_ebbs.len(), ebbs.len()); - for ebb in postorder_ebbs { - assert_eq!(ebb, ebbs[ebbs.len() - postorder_map[ebb]]); - } + .rev() + .cloned() + .collect::>(); + let want = ebb_order + .iter() + .map(|&n| Ebb::with_number(n).unwrap()) + .collect::>(); + assert_eq!(got, want); } #[test] @@ -53,7 +50,7 @@ fn simple_traversal() { trap } ", - vec![0, 2, 1, 3, 4, 5]); + vec![0, 1, 3, 2, 4, 5]); } #[test] @@ -71,7 +68,7 @@ fn loops_one() { return } ", - vec![0, 1, 2, 3]); + vec![0, 1, 3, 2]); } #[test] @@ -96,7 +93,7 @@ fn loops_two() { return } ", - vec![0, 1, 2, 5, 4, 3]); + vec![0, 1, 2, 4, 3, 5]); } #[test] @@ -126,7 +123,7 @@ fn loops_three() { return } ", - vec![0, 1, 2, 5, 4, 3, 6, 7]); + vec![0, 1, 2, 4, 3, 6, 7, 5]); } #[test] diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index ed6aa4dbd6..6d42cefb36 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -25,6 +25,12 @@ struct DomNode { /// The dominator tree for a single function. pub struct DominatorTree { nodes: EntityMap, + + // CFG post-order of all reachable EBBs. + postorder: Vec, + + // Scratch memory used by `compute_postorder()`. + stack: Vec, } /// Methods for querying the dominator tree. @@ -34,6 +40,14 @@ impl DominatorTree { self.nodes[ebb].rpo_number != 0 } + /// Get the CFG post-order of EBBs that was used to compute the dominator tree. + /// + /// Note that this post-order is not updated automatically when the CFG is modified. It is + /// computed from scratch and cached by `compute()`. + pub fn cfg_postorder(&self) -> &[Ebb] { + &self.postorder + } + /// Returns the immediate dominator of `ebb`. /// /// The immediate dominator of an extended basic block is a basic block which we represent by @@ -134,7 +148,11 @@ impl DominatorTree { /// Allocate a new blank dominator tree. Use `compute` to compute the dominator tree for a /// function. pub fn new() -> DominatorTree { - DominatorTree { nodes: EntityMap::new() } + DominatorTree { + nodes: EntityMap::new(), + postorder: Vec::new(), + stack: Vec::new(), + } } /// Allocate and compute a dominator tree. @@ -144,39 +162,91 @@ impl DominatorTree { domtree } - /// Build a dominator tree from a control flow graph using Keith D. Cooper's - /// "Simple, Fast Dominator Algorithm." + /// Reset and compute a CFG post-order and dominator tree. pub fn compute(&mut self, func: &Function, cfg: &ControlFlowGraph) { + self.compute_postorder(func, cfg); + self.compute_domtree(func, cfg); + } + + /// Reset all internal data structures and compute a post-order for `cfg`. + /// + /// This leaves `rpo_number == 1` for all reachable EBBs, 0 for unreachable ones. + fn compute_postorder(&mut self, func: &Function, cfg: &ControlFlowGraph) { self.nodes.clear(); self.nodes.resize(func.dfg.num_ebbs()); + self.postorder.clear(); + assert!(self.stack.is_empty()); - // We'll be iterating over a reverse post-order of the CFG. - // This vector only contains reachable EBBs. - let mut postorder = cfg.postorder_ebbs(); + // During this algorithm only, use `rpo_number` to hold the following state: + // + // 0: EBB never reached. + // 2: EBB has been pushed once, so it shouldn't be pushed again. + // 1: EBB has already been popped once, and should be added to the post-order next time. + const SEEN: u32 = 2; + const DONE: u32 = 1; - // Remove the entry block, and abort if the function is empty. - // The last block visited in a post-order traversal must be the entry block. - let entry_block = match postorder.pop() { - Some(ebb) => ebb, + match func.layout.entry_block() { + Some(ebb) => { + self.nodes[ebb].rpo_number = SEEN; + self.stack.push(ebb) + } + None => return, + } + + while let Some(ebb) = self.stack.pop() { + match self.nodes[ebb].rpo_number { + // This is the first time we visit `ebb`, forming a pre-order. + SEEN => { + // Mark it as done and re-queue it to be visited after its children. + self.nodes[ebb].rpo_number = DONE; + self.stack.push(ebb); + for &succ in cfg.get_successors(ebb) { + // Only push children that haven't been seen before. + if self.nodes[succ].rpo_number == 0 { + self.nodes[succ].rpo_number = SEEN; + self.stack.push(succ); + } + } + } + // This is the second time we popped `ebb`, so all its children have been visited. + // This is the post-order. + DONE => self.postorder.push(ebb), + _ => panic!("Inconsistent stack rpo_number"), + } + } + } + + /// Build a dominator tree from a control flow graph using Keith D. Cooper's + /// "Simple, Fast Dominator Algorithm." + fn compute_domtree(&mut self, func: &Function, cfg: &ControlFlowGraph) { + // During this algorithm, `rpo_number` has the following values: + // + // 0: EBB is not reachable. + // 1: EBB is reachable, but has not yet been visited during the first pass. This is set by + // `compute_postorder`. + // 2+: EBB is reachable and has an assigned RPO number. + + // We'll be iterating over a reverse post-order of the CFG, skipping the entry block. + let (entry_block, postorder) = match self.postorder.as_slice().split_last() { + Some((&eb, rest)) => (eb, rest), None => return, }; - assert_eq!(Some(entry_block), func.layout.entry_block()); + debug_assert_eq!(Some(entry_block), func.layout.entry_block()); // Do a first pass where we assign RPO numbers to all reachable nodes. - self.nodes[entry_block].rpo_number = 1; + self.nodes[entry_block].rpo_number = 2; for (rpo_idx, &ebb) in postorder.iter().rev().enumerate() { // Update the current node and give it an RPO number. - // The entry block got 1, the rest start at 2. + // The entry block got 2, the rest start at 3. // - // Nodes do not appear as reachable until the have an assigned RPO number, and - // `compute_idom` will only look at reachable nodes. This means that the function will - // never see an uninitialized predecessor. + // Since `compute_idom` will only look at nodes with an assigned RPO number, the + // function will never see an uninitialized predecessor. // // Due to the nature of the post-order traversal, every node we visit will have at // least one predecessor that has previously been visited during this RPO. self.nodes[ebb] = DomNode { idom: self.compute_idom(ebb, cfg, &func.layout).into(), - rpo_number: rpo_idx as u32 + 2, + rpo_number: rpo_idx as u32 + 3, } } @@ -200,13 +270,13 @@ impl DominatorTree { // Compute the immediate dominator for `ebb` using the current `idom` states for the reachable // nodes. fn compute_idom(&self, ebb: Ebb, cfg: &ControlFlowGraph, layout: &Layout) -> Inst { - // Get an iterator with just the reachable predecessors to `ebb`. - // Note that during the first pass, `is_reachable` returns false for blocks that haven't - // been visited yet. + // Get an iterator with just the reachable, already visited predecessors to `ebb`. + // Note that during the first pass, `rpo_number` is 1 for reachable blocks that haven't + // been visited yet, 0 for unreachable blocks. let mut reachable_preds = cfg.get_predecessors(ebb) .iter() .cloned() - .filter(|&(ebb, _)| self.is_reachable(ebb)); + .filter(|&(pred, _)| self.nodes[pred].rpo_number > 1); // The RPO must visit at least one predecessor before this node. let mut idom = reachable_preds @@ -233,6 +303,7 @@ mod test { let cfg = ControlFlowGraph::with_function(&func); let dtree = DominatorTree::with_function(&func, &cfg); assert_eq!(0, dtree.nodes.keys().count()); + assert_eq!(dtree.cfg_postorder(), &[]); } #[test] @@ -277,5 +348,7 @@ mod test { assert!(dt.dominates(br_ebb1_ebb0, br_ebb1_ebb0, &func.layout)); assert!(!dt.dominates(br_ebb1_ebb0, jmp_ebb3_ebb1, &func.layout)); assert!(dt.dominates(jmp_ebb3_ebb1, br_ebb1_ebb0, &func.layout)); + + assert_eq!(dt.cfg_postorder(), &[ebb2, ebb0, ebb1, ebb3]); } } From 026ed7a4709226b20c3aea6a78e939684619ec97 Mon Sep 17 00:00:00 2001 From: Aleksey Kuznetsov Date: Sat, 3 Jun 2017 16:16:39 +0500 Subject: [PATCH 0775/3084] Move lib/filecheck/src/tests directory to lib/filecheck --- lib/filecheck/{src => }/tests/basic.rs | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename lib/filecheck/{src => }/tests/basic.rs (100%) diff --git a/lib/filecheck/src/tests/basic.rs b/lib/filecheck/tests/basic.rs similarity index 100% rename from lib/filecheck/src/tests/basic.rs rename to lib/filecheck/tests/basic.rs From c1d095de18f28d22fac59a2a8f2a36b2bb3d03cf Mon Sep 17 00:00:00 2001 From: Aleksey Kuznetsov Date: Sat, 3 Jun 2017 16:20:41 +0500 Subject: [PATCH 0776/3084] Run rustfmt on lib/filecheck/tests/basic.rs --- lib/filecheck/tests/basic.rs | 65 +++++++++++++++++++++++++----------- 1 file changed, 46 insertions(+), 19 deletions(-) diff --git a/lib/filecheck/tests/basic.rs b/lib/filecheck/tests/basic.rs index 595158192c..15e74788d7 100644 --- a/lib/filecheck/tests/basic.rs +++ b/lib/filecheck/tests/basic.rs @@ -28,7 +28,10 @@ fn no_directives() { #[test] fn no_matches() { - let c = CheckerBuilder::new().text("regex: FOO=bar").unwrap().finish(); + let c = CheckerBuilder::new() + .text("regex: FOO=bar") + .unwrap() + .finish(); assert!(!c.is_empty()); // An empty checker matches anything. @@ -260,14 +263,22 @@ fn unordered() { .unwrap() .finish(); - assert_eq!(c.check("one two three four", NO_VARIABLES).map_err(e2s), Ok(true)); - assert_eq!(c.check("one three two four", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one two three four", NO_VARIABLES).map_err(e2s), + Ok(true)); + assert_eq!(c.check("one three two four", NO_VARIABLES).map_err(e2s), + Ok(true)); - assert_eq!(c.check("one two four three four", NO_VARIABLES).map_err(e2s), Ok(true)); - assert_eq!(c.check("one three four two four", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one two four three four", NO_VARIABLES) + .map_err(e2s), + Ok(true)); + assert_eq!(c.check("one three four two four", NO_VARIABLES) + .map_err(e2s), + Ok(true)); - assert_eq!(c.check("one two four three", NO_VARIABLES).map_err(e2s), Ok(false)); - assert_eq!(c.check("one three four two", NO_VARIABLES).map_err(e2s), Ok(false)); + assert_eq!(c.check("one two four three", NO_VARIABLES).map_err(e2s), + Ok(false)); + assert_eq!(c.check("one three four two", NO_VARIABLES).map_err(e2s), + Ok(false)); } #[test] @@ -281,14 +292,22 @@ fn leading_unordered() { .unwrap() .finish(); - assert_eq!(c.check("one two three four", NO_VARIABLES).map_err(e2s), Ok(true)); - assert_eq!(c.check("one three two four", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one two three four", NO_VARIABLES).map_err(e2s), + Ok(true)); + assert_eq!(c.check("one three two four", NO_VARIABLES).map_err(e2s), + Ok(true)); - assert_eq!(c.check("one two four three four", NO_VARIABLES).map_err(e2s), Ok(true)); - assert_eq!(c.check("one three four two four", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one two four three four", NO_VARIABLES) + .map_err(e2s), + Ok(true)); + assert_eq!(c.check("one three four two four", NO_VARIABLES) + .map_err(e2s), + Ok(true)); - assert_eq!(c.check("one two four three", NO_VARIABLES).map_err(e2s), Ok(false)); - assert_eq!(c.check("one three four two", NO_VARIABLES).map_err(e2s), Ok(false)); + assert_eq!(c.check("one two four three", NO_VARIABLES).map_err(e2s), + Ok(false)); + assert_eq!(c.check("one three four two", NO_VARIABLES).map_err(e2s), + Ok(false)); } #[test] @@ -302,12 +321,20 @@ fn trailing_unordered() { .unwrap() .finish(); - assert_eq!(c.check("one two three four", NO_VARIABLES).map_err(e2s), Ok(true)); - assert_eq!(c.check("one three two four", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one two three four", NO_VARIABLES).map_err(e2s), + Ok(true)); + assert_eq!(c.check("one three two four", NO_VARIABLES).map_err(e2s), + Ok(true)); - assert_eq!(c.check("one two four three four", NO_VARIABLES).map_err(e2s), Ok(true)); - assert_eq!(c.check("one three four two four", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one two four three four", NO_VARIABLES) + .map_err(e2s), + Ok(true)); + assert_eq!(c.check("one three four two four", NO_VARIABLES) + .map_err(e2s), + Ok(true)); - assert_eq!(c.check("one two four three", NO_VARIABLES).map_err(e2s), Ok(true)); - assert_eq!(c.check("one three four two", NO_VARIABLES).map_err(e2s), Ok(true)); + assert_eq!(c.check("one two four three", NO_VARIABLES).map_err(e2s), + Ok(true)); + assert_eq!(c.check("one three four two", NO_VARIABLES).map_err(e2s), + Ok(true)); } From de0f0e5f0ab7f4083f305683b1ab811e8ead42d7 Mon Sep 17 00:00:00 2001 From: Igor Date: Sat, 3 Jun 2017 22:40:53 +0300 Subject: [PATCH 0777/3084] Updated the regex crate to 0.2.2 as per issue #88 (#90) * Updated the regex crate to 0.2.2 as per issue #88 * Added potential fix for cryptic CI error. * Fixed incorrect handling of regex name call. Fixes #88 --- lib/filecheck/Cargo.toml | 2 +- lib/filecheck/src/checker.rs | 19 ++++++++++--------- lib/filecheck/src/pattern.rs | 10 +++++----- 3 files changed, 16 insertions(+), 15 deletions(-) diff --git a/lib/filecheck/Cargo.toml b/lib/filecheck/Cargo.toml index ee31a8fa42..41effb059d 100644 --- a/lib/filecheck/Cargo.toml +++ b/lib/filecheck/Cargo.toml @@ -11,4 +11,4 @@ documentation = "https://docs.rs/filecheck" name = "filecheck" [dependencies] -regex = "0.1.71" +regex = "0.2.2" diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs index ee44a769e5..6dcbdd2620 100644 --- a/lib/filecheck/src/checker.rs +++ b/lib/filecheck/src/checker.rs @@ -30,8 +30,8 @@ const DIRECTIVE_RX: &str = r"\b(check|sameln|nextln|unordered|not|regex):\s+(.*) impl Directive { /// Create a new directive from a `DIRECTIVE_RX` match. fn new(caps: Captures) -> Result { - let cmd = caps.at(1).expect("group 1 must match"); - let rest = caps.at(2).expect("group 2 must match"); + let cmd = caps.get(1).map(|m| m.as_str()).expect("group 1 must match"); + let rest = caps.get(2).map(|m| m.as_str()).expect("group 2 must match"); if cmd == "regex" { return Directive::regex(rest); @@ -208,11 +208,12 @@ impl Checker { // Verify any pending `not:` directives now that we know their range. for (not_idx, not_begin, rx) in nots.drain(..) { state.recorder.directive(not_idx); - if let Some((s, e)) = rx.find(&text[not_begin..match_begin]) { + if let Some(mat) = rx.find(&text[not_begin..match_begin]) { // Matched `not:` pattern. state .recorder - .matched_not(rx.as_str(), (not_begin + s, not_begin + e)); + .matched_not(rx.as_str(), + (not_begin + mat.start(), not_begin + mat.end())); return Ok(false); } else { state @@ -337,23 +338,23 @@ impl<'a> State<'a> { } else { // We need the captures to define variables. rx.captures(txt).map(|caps| { - let matched_range = caps.pos(0).expect("whole expression must match"); + let matched_range = caps.get(0).expect("whole expression must match"); for var in defs { - let txtval = caps.name(var).unwrap_or(""); + let txtval = caps.name(var).map(|mat| mat.as_str()).unwrap_or(""); self.recorder.defined_var(var, txtval); let vardef = VarDef { value: Value::Text(Cow::Borrowed(txtval)), // This offset is the end of the whole matched pattern, not just the text // defining the variable. - offset: range.0 + matched_range.1, + offset: range.0 + matched_range.end(), }; self.vars.insert(var.clone(), vardef); } matched_range }) }; - Ok(if let Some((b, e)) = matched_range { - let r = (range.0 + b, range.0 + e); + Ok(if let Some(mat) = matched_range { + let r = (range.0 + mat.start(), range.0 + mat.end()); self.recorder.matched_check(rx.as_str(), r); Some(r) } else { diff --git a/lib/filecheck/src/pattern.rs b/lib/filecheck/src/pattern.rs index 67ec359c4a..7eb3c8fa30 100644 --- a/lib/filecheck/src/pattern.rs +++ b/lib/filecheck/src/pattern.rs @@ -4,7 +4,7 @@ use error::{Error, Result}; use variable::{varname_prefix, VariableMap, Value}; use std::str::FromStr; use std::fmt::{self, Display, Formatter, Write}; -use regex::{Regex, RegexBuilder, quote}; +use regex::{Regex, RegexBuilder, escape}; /// A pattern to match as specified in a directive. /// @@ -313,7 +313,7 @@ impl Pattern { for part in &self.parts { match *part { Part::Text(ref s) => { - out.push_str("e(s)); + out.push_str(&escape(s)); } Part::Regex(ref rx) => out.push_str(rx), Part::Var(ref var) => { @@ -322,7 +322,7 @@ impl Pattern { None => { return Err(Error::UndefVariable(format!("undefined variable ${}", var))) } - Some(Value::Text(s)) => out.push_str("e(&s)), + Some(Value::Text(s)) => out.push_str(&escape(&s)), // Wrap regex in non-capturing group for safe concatenation. Some(Value::Regex(rx)) => write!(out, "(?:{})", rx).unwrap(), } @@ -335,7 +335,7 @@ impl Pattern { None => { return Err(Error::UndefVariable(format!("undefined variable ${}", var))) } - Some(Value::Text(s)) => write!(out, "{})", quote(&s[..])).unwrap(), + Some(Value::Text(s)) => write!(out, "{})", escape(&s[..])).unwrap(), Some(Value::Regex(rx)) => write!(out, "{})", rx).unwrap(), } } @@ -353,7 +353,7 @@ impl Pattern { } } - Ok(RegexBuilder::new(&out).multi_line(true).compile()?) + Ok(RegexBuilder::new(&out).multi_line(true).build()?) } } From 402cb8e1f63f85859c28e376012866451d0197ab Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 5 Jun 2017 15:07:48 -0700 Subject: [PATCH 0778/3084] Add a dfg::replace_result() method. This is analogous to replace_ebb_arg(). It replaces an instruction result value with a new value, leaving the old value in a detached state. --- lib/cretonne/src/ir/dfg.rs | 40 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index c3c5422e75..363b80d2c1 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -10,6 +10,7 @@ use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueList, ValueLis use write::write_operands; use std::fmt; use std::iter; +use std::mem; use std::ops::{Index, IndexMut}; use std::u16; @@ -509,6 +510,36 @@ impl DataFlowGraph { }; } + /// Replace an instruction result with a new value of type `new_type`. + /// + /// The `old_value` must be an attached instruction result. + /// + /// The old value is left detached, so it should probably be changed into something else. + /// + /// Returns the new value. + pub fn replace_result(&mut self, old_value: Value, new_type: Type) -> Value { + let (num, inst) = match self.values[old_value] { + ValueData::Inst { num, inst, .. } => (num, inst), + _ => panic!("{} is not an instruction result value", old_value), + }; + let new_value = self.make_value(ValueData::Inst { + ty: new_type, + num, + inst, + }); + let num = num as usize; + let attached = mem::replace(self.results[inst] + .get_mut(num, &mut self.value_lists) + .expect("Replacing detached result"), + new_value); + assert_eq!(attached, + old_value, + "{} wasn't detached from {}", + old_value, + self.display_inst(inst)); + new_value + } + /// Append a new instruction result value to `inst`. pub fn append_result(&mut self, inst: Inst, ty: Type) -> Value { let res = self.values.next_key(); @@ -767,6 +798,15 @@ mod tests { assert_eq!(dfg.value_def(val), ValueDef::Res(inst, 0)); assert_eq!(dfg.value_type(val), types::I32); + + // Replacing results. + assert!(dfg.value_is_attached(val)); + let v2 = dfg.replace_result(val, types::F64); + assert!(!dfg.value_is_attached(val)); + assert!(dfg.value_is_attached(v2)); + assert_eq!(dfg.inst_results(inst), &[v2]); + assert_eq!(dfg.value_def(v2), ValueDef::Res(inst, 0)); + assert_eq!(dfg.value_type(v2), types::F64); } #[test] From e47f4a49fb4e829e19f723344dec15a2f9744444 Mon Sep 17 00:00:00 2001 From: Denis Merigoux Date: Wed, 7 Jun 2017 11:27:22 -0700 Subject: [PATCH 0779/3084] LICM pass (#87) * LICM pass * Uses loop analysis to detect loop tree * For each loop (starting with the inner ones), create a pre-header and move there loop-invariant instructions * An instruction is loop invariant if it does not use as argument a value defined earlier in the loop * File tests to check LICM's correctness * Optimized pre-header creation If the loop already has a natural pre-header, we use it instead of creating a new one. The natural pre-header of a loop is the only predecessor of the header it doesn't dominate. --- cranelift/filetests/licm/basic.cton | 31 +++ cranelift/filetests/licm/complex.cton | 81 +++++++ cranelift/filetests/licm/multiple-blocks.cton | 46 ++++ cranelift/filetests/licm/nested_loops.cton | 52 +++++ cranelift/src/filetest/licm.rs | 51 +++++ cranelift/src/filetest/mod.rs | 2 + lib/cretonne/src/context.rs | 10 + lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/licm.rs | 208 ++++++++++++++++++ lib/cretonne/src/loop_analysis.rs | 10 +- 10 files changed, 487 insertions(+), 5 deletions(-) create mode 100644 cranelift/filetests/licm/basic.cton create mode 100644 cranelift/filetests/licm/complex.cton create mode 100644 cranelift/filetests/licm/multiple-blocks.cton create mode 100644 cranelift/filetests/licm/nested_loops.cton create mode 100644 cranelift/src/filetest/licm.rs create mode 100644 lib/cretonne/src/licm.rs diff --git a/cranelift/filetests/licm/basic.cton b/cranelift/filetests/licm/basic.cton new file mode 100644 index 0000000000..637b910f53 --- /dev/null +++ b/cranelift/filetests/licm/basic.cton @@ -0,0 +1,31 @@ +test licm + +function simple_loop(i32) -> i32 { + +ebb1(v0: i32): + v1 = iconst.i32 1 + v2 = iconst.i32 2 + v3 = iadd v1, v2 + brz v0, ebb2(v0) + v4 = isub v0, v1 + jump ebb1(v4) + +ebb2(v5: i32): + return v5 + +} +; sameln: function simple_loop(i32) -> i32 { +; nextln: ebb2(v6: i32): +; nextln: v1 = iconst.i32 1 +; nextln: v2 = iconst.i32 2 +; nextln: v3 = iadd v1, v2 +; nextln: jump ebb0(v6) +; nextln: +; nextln: ebb0(v0: i32): +; nextln: brz v0, ebb1(v0) +; nextln: v4 = isub v0, v1 +; nextln: jump ebb0(v4) +; nextln: +; nextln: ebb1(v5: i32): +; nextln: return v5 +; nextln: } diff --git a/cranelift/filetests/licm/complex.cton b/cranelift/filetests/licm/complex.cton new file mode 100644 index 0000000000..fead0cf746 --- /dev/null +++ b/cranelift/filetests/licm/complex.cton @@ -0,0 +1,81 @@ +test licm + +function complex(i32) -> i32 { + +ebb0(v0: i32): + v1 = iconst.i32 1 + v19 = iconst.i32 4 + v2 = iadd v1, v0 + brz v0, ebb1(v1) + jump ebb3(v2) + +ebb1(v3: i32): + v4 = iconst.i32 2 + v5 = iadd v3, v2 + v6 = iadd v4, v0 + jump ebb2(v6) + +ebb2(v7: i32): + v8 = iadd v7, v3 + v9 = iadd v0, v2 + brz v0, ebb1(v7) + jump ebb5(v8) + +ebb3(v10: i32): + v11 = iconst.i32 3 + v12 = iadd v10, v11 + v13 = iadd v2, v11 + jump ebb4(v11) + +ebb4(v14: i32): + v15 = iadd v12, v2 + brz v0, ebb3(v14) + jump ebb5(v14) + +ebb5(v16: i32): + v17 = iadd v16, v1 + v18 = iadd v1, v19 + brz v0, ebb0(v18) + return v17 +} + +; sameln: function complex(i32) -> i32 { +; nextln: ebb6(v20: i32): +; nextln: v1 = iconst.i32 1 +; nextln: v2 = iconst.i32 4 +; nextln: v5 = iconst.i32 2 +; nextln: v12 = iconst.i32 3 +; nextln: v19 = iadd v1, v2 +; nextln: jump ebb0(v20) +; nextln: +; nextln: ebb0(v0: i32): +; nextln: v3 = iadd.i32 v1, v0 +; nextln: v7 = iadd.i32 v5, v0 +; nextln: v10 = iadd v0, v3 +; nextln: brz v0, ebb1(v1) +; nextln: v14 = iadd v3, v12 +; nextln: jump ebb3(v3) +; nextln: +; nextln: ebb1(v4: i32): +; nextln: v6 = iadd v4, v3 +; nextln: jump ebb2(v7) +; nextln: +; nextln: ebb2(v8: i32): +; nextln: v9 = iadd v8, v4 +; nextln: brz.i32 v0, ebb1(v8) +; nextln: jump ebb5(v9) +; nextln: +; nextln: ebb3(v11: i32): +; nextln: v13 = iadd v11, v12 +; nextln: jump ebb4(v12) +; nextln: +; nextln: ebb4(v15: i32): +; nextln: v16 = iadd.i32 v13, v3 +; nextln: brz.i32 v0, ebb3(v15) +; nextln: jump ebb5(v15) +; nextln: +; nextln: ebb5(v17: i32): +; nextln: v18 = iadd v17, v1 +; nextln: brz.i32 v0, ebb0(v19) +; nextln: return v18 +; nextln: } diff --git a/cranelift/filetests/licm/multiple-blocks.cton b/cranelift/filetests/licm/multiple-blocks.cton new file mode 100644 index 0000000000..54db640501 --- /dev/null +++ b/cranelift/filetests/licm/multiple-blocks.cton @@ -0,0 +1,46 @@ +test licm + +function multiple_blocks(i32) -> i32 { + +ebb0(v0: i32): + jump ebb1(v0) + +ebb1(v10: i32): + v11 = iconst.i32 1 + v12 = iconst.i32 2 + v13 = iadd v11, v12 + brz v10, ebb2(v10) + v15 = isub v10, v11 + brz v15, ebb3(v15) + v14 = isub v10, v11 + jump ebb1(v14) + +ebb2(v20: i32): + return v20 + +ebb3(v30: i32): + v31 = iadd v11, v13 + jump ebb1(v30) + +} +; sameln:function multiple_blocks(i32) -> i32 { +; nextln: ebb0(v0: i32): +; nextln: v2 = iconst.i32 1 +; nextln: v3 = iconst.i32 2 +; nextln: v4 = iadd v2, v3 +; nextln: v9 = iadd v2, v4 +; nextln: jump ebb1(v0) +; nextln: +; nextln: ebb1(v1: i32): +; nextln: brz v1, ebb2(v1) +; nextln: v5 = isub v1, v2 +; nextln: brz v5, ebb3(v5) +; nextln: v6 = isub v1, v2 +; nextln: jump ebb1(v6) +; nextln: +; nextln: ebb2(v7: i32): +; nextln: return v7 +; nextln: +; nextln: ebb3(v8: i32): +; nextln: jump ebb1(v8) +; nextln: } diff --git a/cranelift/filetests/licm/nested_loops.cton b/cranelift/filetests/licm/nested_loops.cton new file mode 100644 index 0000000000..e2d3846a0f --- /dev/null +++ b/cranelift/filetests/licm/nested_loops.cton @@ -0,0 +1,52 @@ +test licm + +function nested_loops(i32) -> i32 { + +ebb0(v0: i32): + v1 = iconst.i32 1 + v2 = iconst.i32 2 + v3 = iadd v1, v2 + v4 = isub v0, v1 + jump ebb1(v4,v4) + +ebb1(v10: i32,v11: i32): + brz v11, ebb2(v10) + v12 = iconst.i32 1 + v15 = iadd v12, v4 + v13 = isub v11, v12 + jump ebb1(v10,v13) + +ebb2(v20: i32): + brz v20, ebb3(v20) + jump ebb0(v20) + +ebb3(v30: i32): + return v30 + +} + +; sameln:function nested_loops(i32) -> i32 { +; nextln: ebb4(v12: i32): +; nextln: v1 = iconst.i32 1 +; nextln: v2 = iconst.i32 2 +; nextln: v3 = iadd v1, v2 +; nextln: v7 = iconst.i32 1 +; nextln: jump ebb0(v12) +; nextln: +; nextln: ebb0(v0: i32): +; nextln: v4 = isub v0, v1 +; nextln: v8 = iadd.i32 v7, v4 +; nextln: jump ebb1(v4, v4) +; nextln: +; nextln: ebb1(v5: i32, v6: i32): +; nextln: brz v6, ebb2(v5) +; nextln: v9 = isub v6, v7 +; nextln: jump ebb1(v5, v9) +; nextln: +; nextln: ebb2(v10: i32): +; nextln: brz v10, ebb3(v10) +; nextln: jump ebb0(v10) +; nextln: +; nextln: ebb3(v11: i32): +; nextln: return v11 +; nextln: } diff --git a/cranelift/src/filetest/licm.rs b/cranelift/src/filetest/licm.rs new file mode 100644 index 0000000000..dcde7dd7be --- /dev/null +++ b/cranelift/src/filetest/licm.rs @@ -0,0 +1,51 @@ +//! Test command for testing the LICM pass. +//! +//! The `licm` test command runs each function through the LICM pass after ensuring +//! that all instructions are legal for the target. +//! +//! The resulting function is sent to `filecheck`. + +use cretonne::ir::Function; +use cretonne; +use cton_reader::TestCommand; +use filetest::subtest::{SubTest, Context, Result, run_filecheck}; +use std::borrow::Cow; +use std::fmt::Write; +use utils::pretty_error; + +struct TestLICM; + +pub fn subtest(parsed: &TestCommand) -> Result> { + assert_eq!(parsed.command, "licm"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestLICM)) + } +} + +impl SubTest for TestLICM { + fn name(&self) -> Cow { + Cow::from("licm") + } + + fn is_mutating(&self) -> bool { + true + } + + fn run(&self, func: Cow, context: &Context) -> Result<()> { + // Create a compilation context, and drop in the function. + let mut comp_ctx = cretonne::Context::new(); + comp_ctx.func = func.into_owned(); + + comp_ctx.flowgraph(); + comp_ctx + .licm() + .map_err(|e| pretty_error(&comp_ctx.func, e))?; + + let mut text = String::new(); + write!(&mut text, "{}", &comp_ctx.func) + .map_err(|e| e.to_string())?; + run_filecheck(&text, context) + } +} diff --git a/cranelift/src/filetest/mod.rs b/cranelift/src/filetest/mod.rs index 961c3ff2f2..9c03cc6d4a 100644 --- a/cranelift/src/filetest/mod.rs +++ b/cranelift/src/filetest/mod.rs @@ -17,6 +17,7 @@ mod binemit; mod concurrent; mod domtree; mod legalizer; +mod licm; mod regalloc; mod runner; mod runone; @@ -61,6 +62,7 @@ fn new_subtest(parsed: &TestCommand) -> subtest::Result> { "domtree" => domtree::subtest(parsed), "verifier" => verifier::subtest(parsed), "legalizer" => legalizer::subtest(parsed), + "licm" => licm::subtest(parsed), "regalloc" => regalloc::subtest(parsed), "binemit" => binemit::subtest(parsed), "simple-gvn" => simple_gvn::subtest(parsed), diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 9b0b3b31a7..a67a4bde0e 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -19,6 +19,7 @@ use regalloc; use result::CtonResult; use verifier; use simple_gvn::do_simple_gvn; +use licm::do_licm; /// Persistent data structures and compilation pipeline. pub struct Context { @@ -92,6 +93,15 @@ impl Context { self.verify(None).map_err(Into::into) } + /// Perform LICM on the function. + pub fn licm(&mut self) -> CtonResult { + do_licm(&mut self.func, + &mut self.cfg, + &mut self.domtree, + &mut self.loop_analysis); + self.verify(None).map_err(Into::into) + } + /// Run the register allocator. pub fn regalloc(&mut self, isa: &TargetIsa) -> CtonResult { self.regalloc diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 5ac263d4ff..6ac2f7993b 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -32,6 +32,7 @@ mod constant_hash; mod context; mod iterators; mod legalizer; +mod licm; mod packed_option; mod partition_slice; mod predicates; diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs new file mode 100644 index 0000000000..fc4b7e251b --- /dev/null +++ b/lib/cretonne/src/licm.rs @@ -0,0 +1,208 @@ +//! A Loop Invariant Code Motion optimization pass + +use ir::{Function, Ebb, Inst, Value, Cursor, Type, InstBuilder, Layout}; +use flowgraph::ControlFlowGraph; +use std::collections::HashSet; +use dominator_tree::DominatorTree; +use entity_list::{EntityList, ListPool}; +use loop_analysis::{Loop, LoopAnalysis}; + +/// Performs the LICM pass by detecting loops within the CFG and moving +/// loop-invariant instructions out of them. +/// Changes the CFG and domtree in-place during the operation. +pub fn do_licm(func: &mut Function, + cfg: &mut ControlFlowGraph, + domtree: &mut DominatorTree, + loop_analysis: &mut LoopAnalysis) { + loop_analysis.compute(func, cfg, domtree); + for lp in loop_analysis.loops() { + // For each loop that we want to optimize we determine the set of loop-invariant + // instructions + let invariant_inst = remove_loop_invariant_instructions(lp, func, cfg, loop_analysis); + // Then we create the loop's pre-header and fill it with the invariant instructions + // Then we remove the invariant instructions from the loop body + if invariant_inst.len() > 0 { + // If the loop has a natural pre-header we use it, otherwise we create it. + let mut pos; + match has_pre_header(&func.layout, + cfg, + domtree, + loop_analysis.loop_header(lp).clone()) { + None => { + let pre_header = create_pre_header(loop_analysis.loop_header(lp).clone(), + func, + cfg, + domtree); + pos = Cursor::new(&mut func.layout); + pos.goto_bottom(pre_header); + pos.prev_inst(); + } + // If there is a natural pre-header we insert new instructions just before the + // related jumping instruction (which is not necessarily at the end). + Some((_, last_inst)) => { + pos = Cursor::new(&mut func.layout); + pos.goto_inst(last_inst); + } + }; + // The last instruction of the pre-header is the termination instruction (usually + // a jump) so we need to insert just before this. + for inst in invariant_inst.iter() { + pos.insert_inst(inst.clone()); + } + } + } + // We have to recompute the domtree to account for the changes + cfg.compute(func); + domtree.compute(func, cfg); +} + +// Insert a pre-header before the header, modifying the function layout and CFG to reflect it. +// A jump instruction to the header is placed at the end of the pre-header. +fn create_pre_header(header: Ebb, + func: &mut Function, + cfg: &mut ControlFlowGraph, + domtree: &DominatorTree) + -> Ebb { + let pool = &mut ListPool::::new(); + let header_args_values: Vec = func.dfg + .ebb_args(header) + .into_iter() + .map(|val| *val) + .collect(); + let header_args_types: Vec = header_args_values + .clone() + .into_iter() + .map(|val| func.dfg.value_type(val)) + .collect(); + let pre_header = func.dfg.make_ebb(); + let mut pre_header_args_value: EntityList = EntityList::new(); + for typ in header_args_types { + pre_header_args_value.push(func.dfg.append_ebb_arg(pre_header, typ), pool); + } + for &(_, last_inst) in cfg.get_predecessors(header) { + // We only follow normal edges (not the back edges) + if !domtree.ebb_dominates(header.clone(), last_inst, &func.layout) { + change_branch_jump_destination(last_inst, pre_header, func); + } + } + { + let mut pos = Cursor::new(&mut func.layout); + pos.goto_top(header); + // Inserts the pre-header at the right place in the layout. + pos.insert_ebb(pre_header); + pos.next_inst(); + func.dfg + .ins(&mut pos) + .jump(header, pre_header_args_value.as_slice(pool)); + } + pre_header +} + +// Detects if a loop header has a natural pre-header. +// +// A loop header has a pre-header if there is only one predecessor that the header doesn't +// dominate. +// Returns the pre-header Ebb and the instruction jumping to the header. +fn has_pre_header(layout: &Layout, + cfg: &ControlFlowGraph, + domtree: &DominatorTree, + header: Ebb) + -> Option<(Ebb, Inst)> { + let mut result = None; + let mut found = false; + for &(pred_ebb, last_inst) in cfg.get_predecessors(header) { + // We only count normal edges (not the back edges) + if !domtree.ebb_dominates(header.clone(), last_inst, layout) { + if found { + // We have already found one, there are more than one + return None; + } else { + result = Some((pred_ebb, last_inst)); + found = true; + } + } + } + result +} + + +// Change the destination of a jump or branch instruction. Does nothing if called with a non-jump +// or non-branch instruction. +fn change_branch_jump_destination(inst: Inst, new_ebb: Ebb, func: &mut Function) { + match func.dfg[inst].branch_destination_mut() { + None => (), + Some(instruction_dest) => *instruction_dest = new_ebb, + } +} + +// Traverses a loop in reverse post-order from a header EBB and identify lopp-invariant +// instructions. Theseloop-invariant instructions are then removed from the code and returned +// (in reverse post-order) for later use. +fn remove_loop_invariant_instructions(lp: Loop, + func: &mut Function, + cfg: &ControlFlowGraph, + loop_analysis: &LoopAnalysis) + -> Vec { + let mut loop_values: HashSet = HashSet::new(); + let mut invariant_inst: Vec = Vec::new(); + let mut pos = Cursor::new(&mut func.layout); + // We traverse the loop EBB in reverse post-order. + for ebb in postorder_ebbs_loop(loop_analysis, cfg, lp).iter().rev() { + // Arguments of the EBB are loop values + for val in func.dfg.ebb_args(*ebb) { + loop_values.insert(val.clone()); + } + pos.goto_top(*ebb); + while let Some(inst) = pos.next_inst() { + if func.dfg.has_results(inst) && + func.dfg + .inst_args(inst) + .into_iter() + .all(|arg| !loop_values.contains(arg)) { + // If all the instruction's argument are defined outside the loop + // then this instruction is loop-invariant + invariant_inst.push(inst); + // We remove it from the loop + pos.remove_inst(); + pos.prev_inst(); + } else { + // If the instruction is not loop-invariant we push its results in the set of + // loop values + for out in func.dfg.inst_results(inst) { + loop_values.insert(out.clone()); + } + } + } + } + invariant_inst +} + +/// Return ebbs from a loop in post-order, starting from an entry point in the block. +pub fn postorder_ebbs_loop(loop_analysis: &LoopAnalysis, + cfg: &ControlFlowGraph, + lp: Loop) + -> Vec { + let mut grey = HashSet::new(); + let mut black = HashSet::new(); + let mut stack = vec![loop_analysis.loop_header(lp).clone()]; + let mut postorder = Vec::new(); + + while !stack.is_empty() { + let node = stack.pop().unwrap(); + if !grey.contains(&node) { + // This is a white node. Mark it as gray. + grey.insert(node); + stack.push(node); + // Get any children we've never seen before. + for child in cfg.get_successors(node) { + if loop_analysis.is_in_loop(child.clone(), lp) && !grey.contains(child) { + stack.push(child.clone()); + } + } + } else if !black.contains(&node) { + postorder.push(node.clone()); + black.insert(node.clone()); + } + } + postorder +} diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index 1e36ced8f9..f67a18e9d0 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -129,13 +129,13 @@ impl LoopAnalysis { domtree: &DominatorTree, layout: &Layout) { // We traverse the CFg in reverse postorder - for ebb in cfg.postorder_ebbs().iter().rev() { - for &(_, pred_inst) in cfg.get_predecessors(*ebb) { + for &ebb in cfg.postorder_ebbs().iter().rev() { + for &(_, pred_inst) in cfg.get_predecessors(ebb) { // If the ebb dominates one of its predecessors it is a back edge - if domtree.ebb_dominates(ebb.clone(), pred_inst, layout) { + if domtree.ebb_dominates(ebb, pred_inst, layout) { // This ebb is a loop header, so we create its associated loop - let lp = self.loops.push(LoopData::new(*ebb, None)); - self.ebb_loop_map[*ebb] = lp.into(); + let lp = self.loops.push(LoopData::new(ebb, None)); + self.ebb_loop_map[ebb] = lp.into(); break; // We break because we only need one back edge to identify a loop header. } From 9adfd8709b61cf5c9a015efeef65d842b546d58e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 7 Jun 2017 10:39:47 -0700 Subject: [PATCH 0780/3084] Add Liveness methods for updating live ranges. The create_dead() methods can create a live range for a new value, and extend_local() can extend a live range within an EBB where it is already live. This is enough to update liveness for new values as long as they stay local to their EBB. --- lib/cretonne/src/regalloc/liveness.rs | 43 +++++++++++++++++++++++++- lib/cretonne/src/regalloc/liverange.rs | 8 +++++ 2 files changed, 50 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index f2c9944d4b..3ced8ebc00 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -177,7 +177,7 @@ use flowgraph::ControlFlowGraph; use ir::dfg::ValueDef; -use ir::{Function, Value, Inst, Ebb}; +use ir::{Function, Value, Inst, Ebb, Layout, ProgramPoint}; use isa::{TargetIsa, EncInfo}; use regalloc::affinity::Affinity; use regalloc::liverange::LiveRange; @@ -297,6 +297,47 @@ impl Liveness { self.ranges.get(value) } + /// Create a new live range for `value`. + /// + /// The new live range will be defined at `def` with no extent, like a dead value. + /// + /// This asserts that `value` does not have an existing live range. + pub fn create_dead(&mut self, value: Value, def: PP, affinity: Affinity) + where PP: Into + { + let old = self.ranges + .insert(LiveRange::new(value, def.into(), affinity)); + assert!(old.is_none(), "{} already has a live range", value); + } + + /// Move the definition of `value` to `def`. + /// + /// The old and new def points must be in the same EBB, and before the end of the live range. + pub fn move_def_locally(&mut self, value: Value, def: PP) + where PP: Into + { + let mut lr = self.ranges.get_mut(value).expect("Value has no live range"); + lr.move_def_locally(def.into()); + } + + /// Locally extend the live range for `value` to reach `user`. + /// + /// It is assumed the `value` is already live before `user` in `ebb`. + /// + /// Returns a mutable reference to the value's affinity in case that also needs to be updated. + pub fn extend_locally(&mut self, + value: Value, + ebb: Ebb, + user: Inst, + layout: &Layout) + -> &mut Affinity { + debug_assert_eq!(Some(ebb), layout.inst_ebb(user)); + let mut lr = self.ranges.get_mut(value).expect("Value has no live range"); + let livein = lr.extend_in_ebb(ebb, user, layout); + assert!(!livein, "{} should already be live in {}", value, ebb); + &mut lr.affinity + } + /// Compute the live ranges of all SSA values used in `func`. /// This clears out any existing analysis stored in this data structure. pub fn compute(&mut self, isa: &TargetIsa, func: &Function, cfg: &ControlFlowGraph) { diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index a854cfc9ea..1cabd113dd 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -347,6 +347,14 @@ impl LiveRange { self.def_begin } + /// Move the definition of this value to a new program point. + /// + /// It is only valid to move the definition within the same EBB, and it can't be moved beyond + /// `def_local_end()`. + pub fn move_def_locally(&mut self, def: ProgramPoint) { + self.def_begin = def; + } + /// Get the local end-point of this live range in the EBB where it is defined. /// /// This can be the EBB header itself in the case of a dead EBB argument. From dcdfa59aec756ddf2d64d42f912357bd601b4d1b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 7 Jun 2017 11:33:47 -0700 Subject: [PATCH 0781/3084] Implement a conversion from ValueDef into ProgramPoint. A ValueDef is really no more than a program point plus an argument/result number. --- lib/cretonne/src/ir/progpoint.rs | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index 7d57ab8cb9..c99e79dbe6 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -1,7 +1,7 @@ //! Program points. use entity_map::EntityRef; -use ir::{Ebb, Inst}; +use ir::{Ebb, Inst, ValueDef}; use std::fmt; use std::u32; use std::cmp; @@ -32,6 +32,15 @@ impl From for ProgramPoint { } } +impl From for ProgramPoint { + fn from(def: ValueDef) -> ProgramPoint { + match def { + ValueDef::Res(inst, _) => inst.into(), + ValueDef::Arg(ebb, _) => ebb.into(), + } + } +} + /// An expanded program point directly exposes the variants, but takes twice the space to /// represent. #[derive(PartialEq, Eq, Clone, Copy)] From 0d227fd23044b1db0f1922e88634bf1e1955c9b3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 24 May 2017 09:28:15 -0700 Subject: [PATCH 0782/3084] Add a minimalistic reload pass. The reload pass inserts spill and fill instructions as needed so instructions that operate on registers will never see a value with stack affinity. This is a very basic implementation, and we can't write good test cases until we have a spilling pass. --- lib/cretonne/src/ir/dfg.rs | 10 + lib/cretonne/src/regalloc/affinity.rs | 8 + lib/cretonne/src/regalloc/context.rs | 19 +- lib/cretonne/src/regalloc/mod.rs | 1 + lib/cretonne/src/regalloc/reload.rs | 269 ++++++++++++++++++++++++++ 5 files changed, 306 insertions(+), 1 deletion(-) create mode 100644 lib/cretonne/src/regalloc/reload.rs diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 363b80d2c1..a402274d5e 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -323,6 +323,16 @@ pub enum ValueDef { Arg(Ebb, usize), } +impl ValueDef { + /// Unwrap the instruction where the value was defined, or panic. + pub fn unwrap_inst(&self) -> Inst { + match self { + &ValueDef::Res(inst, _) => inst, + _ => panic!("Value is not an instruction result"), + } + } +} + // Internal table storage for extended values. #[derive(Clone, Debug)] enum ValueData { diff --git a/lib/cretonne/src/regalloc/affinity.rs b/lib/cretonne/src/regalloc/affinity.rs index 64d2b640df..e7ee0515ad 100644 --- a/lib/cretonne/src/regalloc/affinity.rs +++ b/lib/cretonne/src/regalloc/affinity.rs @@ -64,6 +64,14 @@ impl Affinity { } } + /// Is this the `Stack` affinity? + pub fn is_stack(self) -> bool { + match self { + Affinity::Stack => true, + _ => false, + } + } + /// Merge an operand constraint into this affinity. /// /// Note that this does not guarantee that the register allocator will pick a register that diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index b8cebd1bcc..3cf2b5dd81 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -11,6 +11,7 @@ use isa::TargetIsa; use regalloc::coloring::Coloring; use regalloc::live_value_tracker::LiveValueTracker; use regalloc::liveness::Liveness; +use regalloc::reload::Reload; use result::CtonResult; use topo_order::TopoOrder; use verifier::{verify_context, verify_liveness}; @@ -20,6 +21,7 @@ pub struct Context { liveness: Liveness, topo: TopoOrder, tracker: LiveValueTracker, + reload: Reload, coloring: Coloring, } @@ -33,6 +35,7 @@ impl Context { liveness: Liveness::new(), topo: TopoOrder::new(), tracker: LiveValueTracker::new(), + reload: Reload::new(), coloring: Coloring::new(), } } @@ -61,7 +64,21 @@ impl Context { // TODO: Second pass: Spilling. - // Third pass: Reload and coloring. + // Third pass: Reload. + self.reload + .run(isa, + func, + domtree, + &mut self.liveness, + &mut self.topo, + &mut self.tracker); + + if isa.flags().enable_verifier() { + verify_context(func, cfg, domtree, Some(isa))?; + verify_liveness(isa, func, cfg, &self.liveness)?; + } + + // Fourth pass: Coloring. self.coloring .run(isa, func, diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index 473a9fdb28..4a86abbe93 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -12,6 +12,7 @@ mod affinity; mod context; mod diversion; mod pressure; +mod reload; mod solver; pub use self::allocatable_set::AllocatableSet; diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs new file mode 100644 index 0000000000..533dd45595 --- /dev/null +++ b/lib/cretonne/src/regalloc/reload.rs @@ -0,0 +1,269 @@ +//! Reload pass +//! +//! The reload pass runs between the spilling and coloring passes. Its primary responsibility is to +//! insert `spill` and `fill` instructions such that instruction operands expecting a register will +//! get a value with register affinity, and operands expecting a stack slot will get a value with +//! stack affinity. +//! +//! The secondary responsibility of the reload pass is to reuse values in registers as much as +//! possible to minimize the number of `fill` instructions needed. This must not cause the register +//! pressure limits to be exceeded. + +use dominator_tree::DominatorTree; +use ir::{Ebb, Inst, Value, Function, DataFlowGraph}; +use ir::layout::{Cursor, CursorPosition}; +use ir::{InstBuilder, ArgumentLoc}; +use isa::RegClass; +use isa::{TargetIsa, Encoding, EncInfo, ConstraintKind}; +use regalloc::affinity::Affinity; +use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; +use regalloc::liveness::Liveness; +use sparse_map::{SparseMap, SparseMapValue}; +use topo_order::TopoOrder; + +/// Reusable data structures for the reload pass. +pub struct Reload { + candidates: Vec, + reloads: SparseMap, +} + +/// Context data structure that gets instantiated once per pass. +struct Context<'a> { + // Cached ISA information. + // We save it here to avoid frequent virtual function calls on the `TargetIsa` trait object. + encinfo: EncInfo, + + // References to contextual data structures we need. + domtree: &'a DominatorTree, + liveness: &'a mut Liveness, + topo: &'a mut TopoOrder, + + candidates: &'a mut Vec, + reloads: &'a mut SparseMap, +} + +impl Reload { + /// Create a new blank reload pass. + pub fn new() -> Reload { + Reload { + candidates: Vec::new(), + reloads: SparseMap::new(), + } + } + + /// Run the reload algorithm over `func`. + pub fn run(&mut self, + isa: &TargetIsa, + func: &mut Function, + domtree: &DominatorTree, + liveness: &mut Liveness, + topo: &mut TopoOrder, + tracker: &mut LiveValueTracker) { + let mut ctx = Context { + encinfo: isa.encoding_info(), + domtree, + liveness, + topo, + candidates: &mut self.candidates, + reloads: &mut self.reloads, + }; + ctx.run(func, tracker) + } +} + +/// A reload candidate. +/// +/// This represents a stack value that is used by the current instruction where a register is +/// needed. +struct ReloadCandidate { + value: Value, + regclass: RegClass, +} + +/// A Reloaded value. +/// +/// This represents a value that has been reloaded into a register value from the stack. +struct ReloadedValue { + stack: Value, + reg: Value, +} + +impl SparseMapValue for ReloadedValue { + fn key(&self) -> Value { + self.stack + } +} + +impl<'a> Context<'a> { + fn run(&mut self, func: &mut Function, tracker: &mut LiveValueTracker) { + self.topo.reset(func.layout.ebbs()); + while let Some(ebb) = self.topo.next(&func.layout, self.domtree) { + self.visit_ebb(ebb, func, tracker); + } + } + + fn visit_ebb(&mut self, ebb: Ebb, func: &mut Function, tracker: &mut LiveValueTracker) { + dbg!("Reloading {}:", ebb); + let start_from = self.visit_ebb_header(ebb, func, tracker); + tracker.drop_dead_args(); + + let mut pos = Cursor::new(&mut func.layout); + pos.set_position(start_from); + while let Some(inst) = pos.current_inst() { + let encoding = func.encodings[inst]; + if encoding.is_legal() { + self.visit_inst(ebb, inst, encoding, &mut pos, &mut func.dfg, tracker); + tracker.drop_dead(inst); + } else { + pos.next_inst(); + } + } + } + + /// Process the EBB parameters. Return the next instruction in the EBB to be processed + fn visit_ebb_header(&self, + ebb: Ebb, + func: &mut Function, + tracker: &mut LiveValueTracker) + -> CursorPosition { + let (liveins, args) = + tracker.ebb_top(ebb, &func.dfg, self.liveness, &func.layout, self.domtree); + + if func.layout.entry_block() == Some(ebb) { + assert_eq!(liveins.len(), 0); + self.visit_entry_args(ebb, func, args) + } else { + self.visit_ebb_args(ebb, func, args) + } + } + + /// Visit the arguments to the entry block. + /// These values have ABI constraints from the function signature. + fn visit_entry_args(&self, + ebb: Ebb, + func: &mut Function, + args: &[LiveValue]) + -> CursorPosition { + assert_eq!(func.signature.argument_types.len(), args.len()); + let mut pos = Cursor::new(&mut func.layout); + pos.goto_top(ebb); + pos.next_inst(); + + for (abi, arg) in func.signature.argument_types.iter().zip(args) { + match abi.location { + ArgumentLoc::Reg(_) => { + if arg.affinity.is_stack() { + // An incoming register parameter was spilled. Replace the parameter value + // with a temporary register value that is immediately spilled. + let reg = func.dfg.replace_ebb_arg(arg.value, abi.value_type); + func.dfg.ins(&mut pos).with_result(arg.value).spill(reg); + // TODO: Update live ranges. + } + } + ArgumentLoc::Stack(_) => { + assert!(arg.affinity.is_stack()); + } + ArgumentLoc::Unassigned => panic!("Unexpected ABI location"), + } + } + pos.position() + } + + fn visit_ebb_args(&self, ebb: Ebb, func: &mut Function, _args: &[LiveValue]) -> CursorPosition { + let mut pos = Cursor::new(&mut func.layout); + pos.goto_top(ebb); + pos.next_inst(); + pos.position() + } + + /// Process the instruction pointed to by `pos`, and advance the cursor to the next instruction + /// that needs processing. + fn visit_inst(&mut self, + ebb: Ebb, + inst: Inst, + encoding: Encoding, + pos: &mut Cursor, + dfg: &mut DataFlowGraph, + tracker: &mut LiveValueTracker) { + // Get the operand constraints for `inst` that we are trying to satisfy. + let constraints = self.encinfo + .operand_constraints(encoding) + .expect("Missing instruction encoding"); + + assert!(self.candidates.is_empty()); + + // Identify reload candidates. + for (op, &arg) in constraints.ins.iter().zip(dfg.inst_args(inst)) { + if op.kind != ConstraintKind::Stack { + let lv = self.liveness.get(arg).expect("Missing live range for arg"); + if lv.affinity.is_stack() { + self.candidates + .push(ReloadCandidate { + value: arg, + regclass: op.regclass, + }) + } + } + } + + // Insert fill instructions before `inst`. + while let Some(cand) = self.candidates.pop() { + if let Some(_reload) = self.reloads.get_mut(cand.value) { + continue; + } + + let reg = dfg.ins(pos).fill(cand.value); + self.reloads + .insert(ReloadedValue { + stack: cand.value, + reg: reg, + }); + + // Create a live range for the new reload. + let affinity = Affinity::Reg(cand.regclass.into()); + self.liveness.create_dead(reg, dfg.value_def(reg), affinity); + self.liveness.extend_locally(reg, ebb, inst, &pos.layout); + } + + // Rewrite arguments. + for arg in dfg.inst_args_mut(inst) { + if let Some(reload) = self.reloads.get(*arg) { + *arg = reload.reg; + } + } + + // TODO: Reuse reloads for future instructions. + self.reloads.clear(); + + let (_throughs, _kills, defs) = tracker.process_inst(inst, dfg, self.liveness); + + // Advance to the next instruction so we can insert any spills after the instruction. + pos.next_inst(); + + // Rewrite register defs that need to be spilled. + // + // Change: + // + // v2 = inst ... + // + // Into: + // + // v7 = inst ... + // v2 = spill v7 + // + // That way, we don't need to rewrite all future uses of v2. + for (lv, op) in defs.iter().zip(constraints.outs) { + if lv.affinity.is_stack() && op.kind != ConstraintKind::Stack { + let value_type = dfg.value_type(lv.value); + let reg = dfg.replace_result(lv.value, value_type); + dfg.ins(pos).with_result(lv.value).spill(reg); + let spill = dfg.value_def(lv.value).unwrap_inst(); + + // Create a live range for reg. + self.liveness.create_dead(reg, inst, Affinity::new(op)); + self.liveness.extend_locally(reg, ebb, spill, &pos.layout); + self.liveness.move_def_locally(lv.value, spill); + } + } + } +} From f22461b4b3d50dc488d9d05ab83895917b105aae Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 7 Jun 2017 13:38:27 -0700 Subject: [PATCH 0783/3084] Stop using cfg.postorder_ebbs(). Switch to the new domtree.cfg_postorder() which returns a reference to a pre-computed post-order instead of allocating memory and computing a new post-order. --- lib/cretonne/src/context.rs | 2 +- lib/cretonne/src/legalizer/mod.rs | 13 ++++++++----- lib/cretonne/src/loop_analysis.rs | 4 ++-- lib/cretonne/src/simple_gvn.rs | 3 +-- 4 files changed, 12 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index a67a4bde0e..90c4948b40 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -75,7 +75,7 @@ impl Context { /// Run the legalizer for `isa` on the function. pub fn legalize(&mut self, isa: &TargetIsa) -> CtonResult { - legalize_function(&mut self.func, &mut self.cfg, isa); + legalize_function(&mut self.func, &mut self.cfg, &self.domtree, isa); self.verify_if(isa) } diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index edd233ccee..803f5808c2 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -13,6 +13,7 @@ //! The legalizer does not deal with register allocation constraints. These constraints are derived //! from the encoding recipes, and solved later by the register allocator. +use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, InstBuilder}; use ir::condcodes::IntCC; @@ -26,17 +27,19 @@ mod split; /// - Transform any instructions that don't have a legal representation in `isa`. /// - Fill out `func.encodings`. /// -pub fn legalize_function(func: &mut Function, cfg: &mut ControlFlowGraph, isa: &TargetIsa) { +pub fn legalize_function(func: &mut Function, + cfg: &mut ControlFlowGraph, + domtree: &DominatorTree, + isa: &TargetIsa) { boundary::legalize_signatures(func, isa); func.encodings.resize(func.dfg.num_insts()); - // Process EBBs in a reverse post-order. This minimizes the number of split instructions we - // need. - let mut postorder = cfg.postorder_ebbs(); let mut pos = Cursor::new(&mut func.layout); - while let Some(ebb) = postorder.pop() { + // Process EBBs in a reverse post-order. This minimizes the number of split instructions we + // need. + for &ebb in domtree.cfg_postorder().iter().rev() { pos.goto_top(ebb); // Keep track of the cursor position before the instruction being processed, so we can diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index f67a18e9d0..f5ea45424e 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -128,8 +128,8 @@ impl LoopAnalysis { cfg: &ControlFlowGraph, domtree: &DominatorTree, layout: &Layout) { - // We traverse the CFg in reverse postorder - for &ebb in cfg.postorder_ebbs().iter().rev() { + // We traverse the CFG in reverse postorder + for &ebb in domtree.cfg_postorder().iter().rev() { for &(_, pred_inst) in cfg.get_predecessors(ebb) { // If the ebb dominates one of its predecessors it is a back edge if domtree.ebb_dominates(ebb, pred_inst, layout) { diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index a7665e965d..e8f89f4a1b 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -19,10 +19,9 @@ pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph) { let domtree = DominatorTree::with_function(func, &cfg); // Visit EBBs in a reverse post-order. - let mut postorder = cfg.postorder_ebbs(); let mut pos = Cursor::new(&mut func.layout); - while let Some(ebb) = postorder.pop() { + for &ebb in domtree.cfg_postorder().iter().rev() { pos.goto_top(ebb); while let Some(inst) = pos.next_inst() { From 6563270c63fa6986053cf5f0522ef8d316a37873 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 7 Jun 2017 13:41:29 -0700 Subject: [PATCH 0784/3084] Remove cfg.postorder_ebbs(). This is now unused. Use domtree.cfg_postorder() instead. Also remove the dead cfg.ebbs(). --- lib/cretonne/src/flowgraph.rs | 49 +++-------------------------------- 1 file changed, 3 insertions(+), 46 deletions(-) diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index f445d2c8f1..45596d41ec 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -25,8 +25,7 @@ use ir::{Function, Inst, Ebb}; use ir::instructions::BranchInfo; -use entity_map::{EntityMap, Keys}; -use std::collections::HashSet; +use entity_map::EntityMap; use std::mem; /// A basic block denoted by its enclosing Ebb and last instruction. @@ -132,45 +131,6 @@ impl ControlFlowGraph { pub fn get_successors(&self, ebb: Ebb) -> &[Ebb] { &self.data[ebb].successors } - - /// Return [reachable] ebbs in post-order. - pub fn postorder_ebbs(&self) -> Vec { - let entry_block = match self.entry_block { - None => { - return Vec::new(); - } - Some(eb) => eb, - }; - - let mut grey = HashSet::new(); - let mut black = HashSet::new(); - let mut stack = vec![entry_block.clone()]; - let mut postorder = Vec::new(); - - while !stack.is_empty() { - let node = stack.pop().unwrap(); - if !grey.contains(&node) { - // This is a white node. Mark it as gray. - grey.insert(node); - stack.push(node); - // Get any children we've never seen before. - for child in self.get_successors(node) { - if !grey.contains(child) { - stack.push(child.clone()); - } - } - } else if !black.contains(&node) { - postorder.push(node.clone()); - black.insert(node.clone()); - } - } - postorder - } - - /// An iterator across all of the ebbs stored in the CFG. - pub fn ebbs(&self) -> Keys { - self.data.keys() - } } #[cfg(test)] @@ -181,8 +141,7 @@ mod tests { #[test] fn empty() { let func = Function::new(); - let cfg = ControlFlowGraph::with_function(&func); - assert_eq!(None, cfg.ebbs().next()); + ControlFlowGraph::with_function(&func); } #[test] @@ -196,11 +155,9 @@ mod tests { func.layout.append_ebb(ebb2); let cfg = ControlFlowGraph::with_function(&func); - let nodes = cfg.ebbs().collect::>(); - assert_eq!(nodes.len(), 3); let mut fun_ebbs = func.layout.ebbs(); - for ebb in nodes { + for ebb in func.layout.ebbs() { assert_eq!(ebb, fun_ebbs.next().unwrap()); assert_eq!(cfg.get_predecessors(ebb).len(), 0); assert_eq!(cfg.get_successors(ebb).len(), 0); From 81284dbd93703d8e16032073a006d5244dd2b663 Mon Sep 17 00:00:00 2001 From: Denis Merigoux Date: Thu, 8 Jun 2017 09:41:57 -0700 Subject: [PATCH 0785/3084] LICM pass: small changes after code review (#94) --- lib/cretonne/src/licm.rs | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index fc4b7e251b..1c9e1930fd 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -46,7 +46,7 @@ pub fn do_licm(func: &mut Function, }; // The last instruction of the pre-header is the termination instruction (usually // a jump) so we need to insert just before this. - for inst in invariant_inst.iter() { + for inst in invariant_inst { pos.insert_inst(inst.clone()); } } @@ -64,11 +64,7 @@ fn create_pre_header(header: Ebb, domtree: &DominatorTree) -> Ebb { let pool = &mut ListPool::::new(); - let header_args_values: Vec = func.dfg - .ebb_args(header) - .into_iter() - .map(|val| *val) - .collect(); + let header_args_values: Vec = func.dfg.ebb_args(header).into_iter().cloned().collect(); let header_args_types: Vec = header_args_values .clone() .into_iter() @@ -135,8 +131,8 @@ fn change_branch_jump_destination(inst: Inst, new_ebb: Ebb, func: &mut Function) } } -// Traverses a loop in reverse post-order from a header EBB and identify lopp-invariant -// instructions. Theseloop-invariant instructions are then removed from the code and returned +// Traverses a loop in reverse post-order from a header EBB and identify loop-invariant +// instructions. These loop-invariant instructions are then removed from the code and returned // (in reverse post-order) for later use. fn remove_loop_invariant_instructions(lp: Loop, func: &mut Function, @@ -163,8 +159,7 @@ fn remove_loop_invariant_instructions(lp: Loop, // then this instruction is loop-invariant invariant_inst.push(inst); // We remove it from the loop - pos.remove_inst(); - pos.prev_inst(); + pos.remove_inst_and_step_back(); } else { // If the instruction is not loop-invariant we push its results in the set of // loop values @@ -178,10 +173,7 @@ fn remove_loop_invariant_instructions(lp: Loop, } /// Return ebbs from a loop in post-order, starting from an entry point in the block. -pub fn postorder_ebbs_loop(loop_analysis: &LoopAnalysis, - cfg: &ControlFlowGraph, - lp: Loop) - -> Vec { +fn postorder_ebbs_loop(loop_analysis: &LoopAnalysis, cfg: &ControlFlowGraph, lp: Loop) -> Vec { let mut grey = HashSet::new(); let mut black = HashSet::new(); let mut stack = vec![loop_analysis.loop_header(lp).clone()]; From 2f33848fcdd9f7b49e04ab76ba6f8c05ce51cd8d Mon Sep 17 00:00:00 2001 From: Denis Merigoux Date: Fri, 9 Jun 2017 16:20:38 -0700 Subject: [PATCH 0786/3084] Improved DFG API (#95) * Improved DFG API with swap_remove_ebb_args and append_inst_arg * Implemented EntityList::swap_remove And used it for dfg::swap_remove_ebb_arg --- lib/cretonne/src/entity_list.rs | 18 ++++++++- lib/cretonne/src/ir/dfg.rs | 66 ++++++++++++++++++++++++++++++++- 2 files changed, 82 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index 946bb7ba04..1ccddc3aed 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -416,7 +416,7 @@ impl EntityList { } } - /// Removes the element at position `index` from the list. + /// Removes the element at position `index` from the list. Potentially linear complexity. pub fn remove(&mut self, index: usize, pool: &mut ListPool) { let len; { @@ -448,6 +448,22 @@ impl EntityList { pool.data[block] = T::new(len - 1); } + /// Removes the element at `index` in constant time by switching it with the last element of + /// the list. + pub fn swap_remove(&mut self, index: usize, pool: &mut ListPool) { + let len = self.len(pool); + assert!(index < len); + if index == len - 1 { + self.remove(index, pool); + } else { + { + let seq = self.as_mut_slice(pool); + seq.swap(index, len - 1); + } + self.remove(len - 1, pool); + } + } + /// Grow the list by inserting `count` elements at `index`. /// /// The new elements are not initialized, they will contain whatever happened to be in memory. diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index a402274d5e..36607e76f1 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -127,7 +127,7 @@ fn resolve_aliases(values: &EntityMap, value: Value) -> Value /// /// Values are either EBB arguments or instruction results. impl DataFlowGraph { - // Allocate an extended value entry. + /// Allocate an extended value entry. fn make_value(&mut self, data: ValueData) -> Value { self.values.push(data) } @@ -562,6 +562,17 @@ impl DataFlowGraph { }) } + /// Append a new value argument to an instruction. + /// + /// Panics if the instruction doesn't support arguments. + pub fn append_inst_arg(&mut self, inst: Inst, new_arg: Value) { + let mut branch_values = self.insts[inst] + .take_value_list() + .expect("the instruction doesn't have value arguments"); + branch_values.push(new_arg, &mut self.value_lists); + self.insts[inst].put_value_list(branch_values) + } + /// Get the first result of an instruction. /// /// This function panics if the instruction doesn't have any result. @@ -682,6 +693,35 @@ impl DataFlowGraph { }) } + /// Removes `val` from `ebb`'s argument by swapping it with the last argument of `ebb`. + /// Returns the position of `val` before removal. + /// + /// *Important*: to ensure O(1) deletion, this method swaps the removed argument with the + /// last `Ebb` argument. This can disrupt all the branch instructions jumping to this + /// `Ebb` for which you have to change the jump argument order if necessary. + /// + /// Panics if `val` is not an `Ebb` argument. Returns `true` if `Ebb` arguments have been + /// swapped. + pub fn swap_remove_ebb_arg(&mut self, val: Value) -> usize { + let (ebb, num) = if let ValueData::Arg { num, ebb, .. } = self.values[val] { + (ebb, num) + } else { + panic!("{} must be an EBB argument", val); + }; + self.ebbs[ebb] + .args + .swap_remove(num as usize, &mut self.value_lists); + if let Some(last_arg_val) = self.ebbs[ebb].args.get(num as usize, &self.value_lists) { + // We update the position of the old last arg. + if let ValueData::Arg { num: ref mut old_num, .. } = self.values[last_arg_val] { + *old_num = num; + } else { + panic!("{} should be an Ebb argument but is not", last_arg_val); + } + } + num as usize + } + /// Append an existing argument value to `ebb`. /// /// The appended value can't already be attached to something else. @@ -899,6 +939,30 @@ mod tests { assert_eq!(dfg.ebb_args(ebb), &[new1, new3, arg1]); } + #[test] + fn swap_remove_ebb_arguments() { + let mut dfg = DataFlowGraph::new(); + + let ebb = dfg.make_ebb(); + let arg1 = dfg.append_ebb_arg(ebb, types::F32); + let arg2 = dfg.append_ebb_arg(ebb, types::F32); + let arg3 = dfg.append_ebb_arg(ebb, types::F32); + assert_eq!(dfg.ebb_args(ebb), &[arg1, arg2, arg3]); + + dfg.swap_remove_ebb_arg(arg1); + assert_eq!(dfg.value_is_attached(arg1), false); + assert_eq!(dfg.value_is_attached(arg2), true); + assert_eq!(dfg.value_is_attached(arg3), true); + assert_eq!(dfg.ebb_args(ebb), &[arg3, arg2]); + dfg.swap_remove_ebb_arg(arg2); + assert_eq!(dfg.value_is_attached(arg2), false); + assert_eq!(dfg.value_is_attached(arg3), true); + assert_eq!(dfg.ebb_args(ebb), &[arg3]); + dfg.swap_remove_ebb_arg(arg3); + assert_eq!(dfg.value_is_attached(arg3), false); + assert_eq!(dfg.ebb_args(ebb), &[]); + } + #[test] fn aliases() { use ir::InstBuilder; From 706eef23d3bfd3dd66c15f31bee3bd55e23071be Mon Sep 17 00:00:00 2001 From: Aleksey Kuznetsov Date: Sat, 10 Jun 2017 22:30:37 +0500 Subject: [PATCH 0787/3084] Binary function names (#91) * Function names should start with % * Create FunctionName from string * Implement displaying of FunctionName as %nnnn with fallback to #xxxx * Run rustfmt and fix FunctionName::with_string in parser * Implement FunctionName::new as a generic function * Binary function names should start with # * Implement NameRepr for function name * Fix examples in docs to reflect that function names start with % * Rebase and fix filecheck tests --- cranelift/docs/example.cton | 2 +- cranelift/docs/langref.rst | 6 +- cranelift/docs/testing.rst | 20 +-- cranelift/filetests/cfg/loop.cton | 4 +- cranelift/filetests/cfg/traps_early.cton | 4 +- cranelift/filetests/cfg/unused_node.cton | 4 +- cranelift/filetests/domtree/basic.cton | 2 +- cranelift/filetests/domtree/loops.cton | 2 +- cranelift/filetests/domtree/loops2.cton | 2 +- cranelift/filetests/domtree/tall-tree.cton | 2 +- cranelift/filetests/domtree/wide-tree.cton | 2 +- cranelift/filetests/isa/intel/binary32.cton | 2 +- cranelift/filetests/isa/riscv/abi-e.cton | 2 +- cranelift/filetests/isa/riscv/abi.cton | 2 +- cranelift/filetests/isa/riscv/binary32.cton | 4 +- cranelift/filetests/isa/riscv/encoding.cton | 2 +- cranelift/filetests/isa/riscv/expand-i32.cton | 4 +- .../filetests/isa/riscv/legalize-abi.cton | 28 ++-- .../filetests/isa/riscv/legalize-i64.cton | 8 +- .../filetests/isa/riscv/parse-encoding.cton | 8 +- cranelift/filetests/isa/riscv/split-args.cton | 6 +- .../filetests/isa/riscv/verify-encoding.cton | 8 +- cranelift/filetests/licm/basic.cton | 4 +- cranelift/filetests/licm/complex.cton | 4 +- cranelift/filetests/licm/multiple-blocks.cton | 4 +- cranelift/filetests/licm/nested_loops.cton | 4 +- cranelift/filetests/parser/branch.cton | 24 ++-- cranelift/filetests/parser/call.cton | 34 ++--- .../parser/instruction_encoding.cton | 4 +- cranelift/filetests/parser/keywords.cton | 4 +- cranelift/filetests/parser/rewrite.cton | 8 +- cranelift/filetests/parser/ternary.cton | 4 +- cranelift/filetests/parser/tiny.cton | 44 +++--- cranelift/filetests/regalloc/basic.cton | 8 +- cranelift/filetests/simple_gvn/basic.cton | 4 +- cranelift/filetests/verifier/bad_layout.cton | 4 +- cranelift/tests/cfg_traversal.rs | 10 +- lib/cretonne/src/ir/funcname.rs | 128 ++++++++++++------ lib/cretonne/src/write.rs | 14 +- lib/reader/src/parser.rs | 82 +++++++++-- lib/reader/src/sourcemap.rs | 2 +- 41 files changed, 306 insertions(+), 208 deletions(-) diff --git a/cranelift/docs/example.cton b/cranelift/docs/example.cton index db87e891d1..8b9c43c6b9 100644 --- a/cranelift/docs/example.cton +++ b/cranelift/docs/example.cton @@ -1,6 +1,6 @@ test verifier -function average(i32, i32) -> f32 { +function %average(i32, i32) -> f32 { ss1 = stack_slot 8 ; Stack slot for ``sum``. ebb1(v1: i32, v2: i32): diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index e6a0e2b3d5..24b5961ec3 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -389,8 +389,8 @@ preamble`: This simple example illustrates direct function calls and signatures:: - function gcd(i32 uext, i32 uext) -> i32 uext "C" { - fn1 = function divmod(i32 uext, i32 uext) -> i32 uext, i32 uext + function %gcd(i32 uext, i32 uext) -> i32 uext "C" { + fn1 = function %divmod(i32 uext, i32 uext) -> i32 uext, i32 uext ebb1(v1: i32, v2: i32): brz v2, ebb2 @@ -530,7 +530,7 @@ and address computations from the memory accesses. A small example using heaps:: - function vdup(i32, i32) { + function %vdup(i32, i32) { h1 = heap "main" ebb1(v1: i32, v2: i32): diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index d4aa1a881c..b72ace08d3 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -130,7 +130,7 @@ The ``set`` lines apply settings cumulatively:: set is_64bit=0 isa riscv supports_m=false - function foo() {} + function %foo() {} This example will run the legalizer test twice. Both runs will have ``opt_level=best``, but they will have different ``is_64bit`` settings. The 32-bit @@ -184,13 +184,13 @@ against the associated filecheck directives. Example:: - function r1() -> i32, f32 { + function %r1() -> i32, f32 { ebb1: v10 = iconst.i32 3 v20 = f32const 0.0 return v10, v20 } - ; sameln: function r1() -> i32, f32 { + ; sameln: function %r1() -> i32, f32 { ; nextln: ebb0: ; nextln: v0 = iconst.i32 3 ; nextln: v1 = f32const 0.0 @@ -201,13 +201,13 @@ Notice that the values ``v10`` and ``v20`` in the source were renumbered to ``v0`` and ``v1`` respectively during parsing. The equivalent test using filecheck variables would be:: - function r1() -> i32, f32 { + function %r1() -> i32, f32 { ebb1: v10 = iconst.i32 3 v20 = f32const 0.0 return v10, v20 } - ; sameln: function r1() -> i32, f32 { + ; sameln: function %r1() -> i32, f32 { ; nextln: ebb0: ; nextln: $v10 = iconst.i32 3 ; nextln: $v20 = f32const 0.0 @@ -226,7 +226,7 @@ reported location of the error is verified:: test verifier - function test(i32) { + function %test(i32) { ebb0(v0: i32): jump ebb1 ; error: terminator return @@ -250,8 +250,8 @@ command:: test print-cfg test verifier - function nonsense(i32, i32) -> f32 { - ; check: digraph nonsense { + function %nonsense(i32, i32) -> f32 { + ; check: digraph %nonsense { ; regex: I=\binst\d+\b ; check: label="{ebb0 | <$(BRZ=$I)>brz ebb2 | <$(JUMP=$I)>jump ebb1}"] @@ -276,7 +276,7 @@ Compute the dominator tree of each function and validate it against the test domtree - function test(i32) { + function %test(i32) { ebb0(v0: i32): jump ebb1 ; dominates: ebb1 ebb1: @@ -328,7 +328,7 @@ that instruction is compared to the directive:: test binemit isa riscv - function int32() { + function %int32() { ebb0: [-,%x5] v1 = iconst.i32 1 [-,%x6] v2 = iconst.i32 2 diff --git a/cranelift/filetests/cfg/loop.cton b/cranelift/filetests/cfg/loop.cton index 3f7892a9d9..06aa848c7f 100644 --- a/cranelift/filetests/cfg/loop.cton +++ b/cranelift/filetests/cfg/loop.cton @@ -2,8 +2,8 @@ test print-cfg test verifier -function nonsense(i32, i32) -> f32 { -; check: digraph nonsense { +function %nonsense(i32, i32) -> f32 { +; check: digraph %nonsense { ; regex: I=\binst\d+\b ; check: label="{ebb0 | <$(BRZ=$I)>brz ebb2 | <$(JUMP=$I)>jump ebb1}"] diff --git a/cranelift/filetests/cfg/traps_early.cton b/cranelift/filetests/cfg/traps_early.cton index 9648190bc5..814e251f51 100644 --- a/cranelift/filetests/cfg/traps_early.cton +++ b/cranelift/filetests/cfg/traps_early.cton @@ -3,8 +3,8 @@ test print-cfg test verifier -function nonsense(i32) { -; check: digraph nonsense { +function %nonsense(i32) { +; check: digraph %nonsense { ebb0(v1: i32): trap ; error: terminator instruction was encountered before the end diff --git a/cranelift/filetests/cfg/unused_node.cton b/cranelift/filetests/cfg/unused_node.cton index d4be2deddd..cbde9757bc 100644 --- a/cranelift/filetests/cfg/unused_node.cton +++ b/cranelift/filetests/cfg/unused_node.cton @@ -1,8 +1,8 @@ ; For testing cfg generation where some block is never reached. test print-cfg -function not_reached(i32) -> i32 { -; check: digraph not_reached { +function %not_reached(i32) -> i32 { +; check: digraph %not_reached { ; check: ebb0 [shape=record, label="{ebb0 | brnz ebb2}"] ; check: ebb1 [shape=record, label="{ebb1 | jump ebb0}"] ; check: ebb2 [shape=record, label="{ebb2}"] diff --git a/cranelift/filetests/domtree/basic.cton b/cranelift/filetests/domtree/basic.cton index 0b8536effd..e46c73e67e 100644 --- a/cranelift/filetests/domtree/basic.cton +++ b/cranelift/filetests/domtree/basic.cton @@ -1,6 +1,6 @@ test domtree -function test(i32) { +function %test(i32) { ebb0(v0: i32): jump ebb1 ; dominates: ebb1 ebb1: diff --git a/cranelift/filetests/domtree/loops.cton b/cranelift/filetests/domtree/loops.cton index cdc88544cc..43ad1e1c08 100644 --- a/cranelift/filetests/domtree/loops.cton +++ b/cranelift/filetests/domtree/loops.cton @@ -1,6 +1,6 @@ test domtree -function test(i32) { +function %test(i32) { ebb0(v0: i32): brz v0, ebb1 ; dominates: ebb1 ebb3 ebb4 ebb5 jump ebb2 ; dominates: ebb2 diff --git a/cranelift/filetests/domtree/loops2.cton b/cranelift/filetests/domtree/loops2.cton index d2e3ba4f2c..eeac8343bd 100644 --- a/cranelift/filetests/domtree/loops2.cton +++ b/cranelift/filetests/domtree/loops2.cton @@ -1,6 +1,6 @@ test domtree -function test(i32) { +function %test(i32) { ebb0(v0: i32): brz v0, ebb1 ; dominates: ebb1 ebb6 brnz v0, ebb2 ; dominates: ebb2 ebb9 diff --git a/cranelift/filetests/domtree/tall-tree.cton b/cranelift/filetests/domtree/tall-tree.cton index 3dd6f51da6..89821d0744 100644 --- a/cranelift/filetests/domtree/tall-tree.cton +++ b/cranelift/filetests/domtree/tall-tree.cton @@ -1,6 +1,6 @@ test domtree -function test(i32) { +function %test(i32) { ebb0(v0: i32): brz v0, ebb1 ; dominates: ebb1 brnz v0, ebb2 ; dominates: ebb2 ebb5 diff --git a/cranelift/filetests/domtree/wide-tree.cton b/cranelift/filetests/domtree/wide-tree.cton index f4e822cd81..3a9b4fff37 100644 --- a/cranelift/filetests/domtree/wide-tree.cton +++ b/cranelift/filetests/domtree/wide-tree.cton @@ -1,6 +1,6 @@ test domtree -function test(i32) { +function %test(i32) { ebb0(v0: i32): brz v0, ebb13 ; dominates: ebb13 jump ebb1 ; dominates: ebb1 diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 5cac489f77..818335d349 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -7,7 +7,7 @@ isa intel ; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/binary32.cton | llvm-mc -show-encoding -triple=i386 ; -function I32() { +function %I32() { ebb0: [-,%rcx] v1 = iconst.i32 1 [-,%rsi] v2 = iconst.i32 2 diff --git a/cranelift/filetests/isa/riscv/abi-e.cton b/cranelift/filetests/isa/riscv/abi-e.cton index f770a3558f..543b55079f 100644 --- a/cranelift/filetests/isa/riscv/abi-e.cton +++ b/cranelift/filetests/isa/riscv/abi-e.cton @@ -4,7 +4,7 @@ isa riscv enable_e ; regex: V=v\d+ -function f() { +function %f() { ; Spilling into the stack args after %x15 since %16 and up are not ; available in RV32E. sig0 = signature(i64, i64, i64, i64) -> i64 diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.cton index eba5609f3e..f26c7686cf 100644 --- a/cranelift/filetests/isa/riscv/abi.cton +++ b/cranelift/filetests/isa/riscv/abi.cton @@ -4,7 +4,7 @@ isa riscv ; regex: V=v\d+ -function f() { +function %f() { sig0 = signature(i32) -> i32 ; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index f62a6187e0..052342f233 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -2,8 +2,8 @@ test binemit isa riscv -function RV32I(i32 link [%x1]) -> i32 link [%x1] { - fn0 = function foo() +function %RV32I(i32 link [%x1]) -> i32 link [%x1] { + fn0 = function %foo() ebb0(v9999: i32): [-,%x10] v1 = iconst.i32 1 diff --git a/cranelift/filetests/isa/riscv/encoding.cton b/cranelift/filetests/isa/riscv/encoding.cton index fdd3ee4329..71961ba353 100644 --- a/cranelift/filetests/isa/riscv/encoding.cton +++ b/cranelift/filetests/isa/riscv/encoding.cton @@ -1,7 +1,7 @@ test legalizer isa riscv supports_m=1 -function int32(i32, i32) { +function %int32(i32, i32) { ebb0(v1: i32, v2: i32): v10 = iadd v1, v2 ; check: [R#0c] diff --git a/cranelift/filetests/isa/riscv/expand-i32.cton b/cranelift/filetests/isa/riscv/expand-i32.cton index fe93521567..2bffad234c 100644 --- a/cranelift/filetests/isa/riscv/expand-i32.cton +++ b/cranelift/filetests/isa/riscv/expand-i32.cton @@ -9,7 +9,7 @@ isa riscv supports_m=1 ; regex: V=v\d+ -function carry_out(i32, i32) -> i32, b1 { +function %carry_out(i32, i32) -> i32, b1 { ebb0(v1: i32, v2: i32): v3, v4 = iadd_cout v1, v2 return v3, v4 @@ -20,7 +20,7 @@ ebb0(v1: i32, v2: i32): ; Expanding illegal immediate constants. ; Note that at some point we'll probably expand the iconst as well. -function large_imm(i32) -> i32 { +function %large_imm(i32) -> i32 { ebb0(v0: i32): v1 = iadd_imm v0, 1000000000 return v1 diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index a3632edca3..4d6f4e6e7f 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -4,7 +4,7 @@ isa riscv ; regex: V=v\d+ -function int_split_args(i64) -> i64 { +function %int_split_args(i64) -> i64 { ebb0(v0: i64): ; check: $ebb0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32): ; check: $v0 = iconcat $v0l, $v0h @@ -14,9 +14,9 @@ ebb0(v0: i64): return v1 } -function split_call_arg(i32) { - fn1 = function foo(i64) - fn2 = function foo(i32, i64) +function %split_call_arg(i32) { + fn1 = function %foo(i64) + fn2 = function %foo(i32, i64) ebb0(v0: i32): v1 = uextend.i64 v0 call fn1(v1) @@ -27,8 +27,8 @@ ebb0(v0: i32): return } -function split_ret_val() { - fn1 = function foo() -> i64 +function %split_ret_val() { + fn1 = function %foo() -> i64 ebb0: v1 = call fn1() ; check: $ebb0($(link=$V): i32): @@ -42,8 +42,8 @@ ebb1(v10: i64): } ; First return value is fine, second one is expanded. -function split_ret_val2() { - fn1 = function foo() -> i32, i64 +function %split_ret_val2() { + fn1 = function %foo() -> i32, i64 ebb0: v1, v2 = call fn1() ; check: $ebb0($(link=$V): i32): @@ -56,7 +56,7 @@ ebb1(v9: i32, v10: i64): jump ebb1(v9, v10) } -function int_ext(i8, i8 sext, i8 uext) -> i8 uext { +function %int_ext(i8, i8 sext, i8 uext) -> i8 uext { ebb0(v1: i8, v2: i8, v3: i8): ; check: $ebb0($v1: i8, $(v2x=$V): i32, $(v3x=$V): i32, $(link=$V): i32): ; check: $v2 = ireduce.i8 $v2x @@ -67,8 +67,8 @@ ebb0(v1: i8, v2: i8, v3: i8): } ; Function produces single return value, still need to copy. -function ext_ret_val() { - fn1 = function foo() -> i8 sext +function %ext_ret_val() { + fn1 = function %foo() -> i8 sext ebb0: v1 = call fn1() ; check: $ebb0($V: i32): @@ -81,7 +81,7 @@ ebb1(v10: i8): jump ebb1(v10) } -function vector_split_args(i64x4) -> i64x4 { +function %vector_split_args(i64x4) -> i64x4 { ebb0(v0: i64x4): ; check: $ebb0($(v0al=$V): i32, $(v0ah=$V): i32, $(v0bl=$V): i32, $(v0bh=$V): i32, $(v0cl=$V): i32, $(v0ch=$V): i32, $(v0dl=$V): i32, $(v0dh=$V): i32, $(link=$V): i32): ; check: $(v0a=$V) = iconcat $v0al, $v0ah @@ -103,7 +103,7 @@ ebb0(v0: i64x4): return v1 } -function indirect(i32) { +function %indirect(i32) { sig1 = signature() ebb0(v0: i32): call_indirect sig1, v0() @@ -111,7 +111,7 @@ ebb0(v0: i32): } ; The first argument to call_indirect doesn't get altered. -function indirect_arg(i32, f32x2) { +function %indirect_arg(i32, f32x2) { sig1 = signature(f32x2) ebb0(v0: i32, v1: f32x2): call_indirect sig1, v0(v1) diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton index 516502c4d4..e7a3441ee9 100644 --- a/cranelift/filetests/isa/riscv/legalize-i64.cton +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -4,7 +4,7 @@ isa riscv supports_m=1 ; regex: V=v\d+ -function bitwise_and(i64, i64) -> i64 { +function %bitwise_and(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): v3 = band v1, v2 return v3 @@ -17,7 +17,7 @@ ebb0(v1: i64, v2: i64): ; check: $v3 = iconcat $v3l, $v3h ; check: return $v3l, $v3h, $link -function bitwise_or(i64, i64) -> i64 { +function %bitwise_or(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): v3 = bor v1, v2 return v3 @@ -30,7 +30,7 @@ ebb0(v1: i64, v2: i64): ; check: $v3 = iconcat $v3l, $v3h ; check: return $v3l, $v3h, $link -function bitwise_xor(i64, i64) -> i64 { +function %bitwise_xor(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): v3 = bxor v1, v2 return v3 @@ -43,7 +43,7 @@ ebb0(v1: i64, v2: i64): ; check: $v3 = iconcat $v3l, $v3h ; check: return $v3l, $v3h, $link -function arith_add(i64, i64) -> i64 { +function %arith_add(i64, i64) -> i64 { ; Legalizing iadd.i64 requires two steps: ; 1. Narrow to iadd_cout.i32, then ; 2. Expand iadd_cout.i32 since RISC-V has no carry flag. diff --git a/cranelift/filetests/isa/riscv/parse-encoding.cton b/cranelift/filetests/isa/riscv/parse-encoding.cton index 69a25ce744..cd02a3ca47 100644 --- a/cranelift/filetests/isa/riscv/parse-encoding.cton +++ b/cranelift/filetests/isa/riscv/parse-encoding.cton @@ -2,8 +2,8 @@ test legalizer isa riscv -function parse_encoding(i32 [%x5]) -> i32 [%x10] { - ; check: function parse_encoding(i32 [%x5], i32 link [%x1]) -> i32 [%x10], i32 link [%x1] { +function %parse_encoding(i32 [%x5]) -> i32 [%x10] { + ; check: function %parse_encoding(i32 [%x5], i32 link [%x1]) -> i32 [%x10], i32 link [%x1] { sig0 = signature(i32 [%x10]) -> i32 [%x10] ; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] @@ -27,9 +27,9 @@ function parse_encoding(i32 [%x5]) -> i32 [%x10] { ; check: sig5 = signature() -> f32 [0] ; function + signature - fn15 = function bar(i32 [%x10]) -> b1 [%x10] + fn15 = function %bar(i32 [%x10]) -> b1 [%x10] ; check: sig6 = signature(i32 [%x10]) -> b1 [%x10] - ; nextln: fn0 = sig6 bar + ; nextln: fn0 = sig6 %bar ebb0(v0: i32): return v0 diff --git a/cranelift/filetests/isa/riscv/split-args.cton b/cranelift/filetests/isa/riscv/split-args.cton index a568204658..be1370dc12 100644 --- a/cranelift/filetests/isa/riscv/split-args.cton +++ b/cranelift/filetests/isa/riscv/split-args.cton @@ -4,7 +4,7 @@ isa riscv ; regex: V=v\d+ -function simple(i64, i64) -> i64 { +function %simple(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): ; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): jump ebb1(v1) @@ -19,7 +19,7 @@ ebb1(v3: i64): ; check: return $v4l, $v4h, $link } -function multi(i64) -> i64 { +function %multi(i64) -> i64 { ebb1(v1: i64): ; check: $ebb1($(v1l=$V): i32, $(v1h=$V): i32, $(link=$V): i32): jump ebb2(v1, v1) @@ -39,7 +39,7 @@ ebb3(v4: i64): ; check: return $v5l, $v5h, $link } -function loop(i64, i64) -> i64 { +function %loop(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): ; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): jump ebb1(v1) diff --git a/cranelift/filetests/isa/riscv/verify-encoding.cton b/cranelift/filetests/isa/riscv/verify-encoding.cton index 73d28c842a..52b8d6d79c 100644 --- a/cranelift/filetests/isa/riscv/verify-encoding.cton +++ b/cranelift/filetests/isa/riscv/verify-encoding.cton @@ -1,8 +1,8 @@ test verifier isa riscv -function RV32I(i32 link [%x1]) -> i32 link [%x1] { - fn0 = function foo() +function %RV32I(i32 link [%x1]) -> i32 link [%x1] { + fn0 = function %foo() ebb0(v9999: i32): ; iconst.i32 needs legalizing, so it should throw a @@ -10,8 +10,8 @@ ebb0(v9999: i32): return v9999 } -function RV32I(i32 link [%x1]) -> i32 link [%x1] { - fn0 = function foo() +function %RV32I(i32 link [%x1]) -> i32 link [%x1] { + fn0 = function %foo() ebb0(v9999: i32): v1 = iconst.i32 1 diff --git a/cranelift/filetests/licm/basic.cton b/cranelift/filetests/licm/basic.cton index 637b910f53..36b7864cfe 100644 --- a/cranelift/filetests/licm/basic.cton +++ b/cranelift/filetests/licm/basic.cton @@ -1,6 +1,6 @@ test licm -function simple_loop(i32) -> i32 { +function %simple_loop(i32) -> i32 { ebb1(v0: i32): v1 = iconst.i32 1 @@ -14,7 +14,7 @@ ebb2(v5: i32): return v5 } -; sameln: function simple_loop(i32) -> i32 { +; sameln: function %simple_loop(i32) -> i32 { ; nextln: ebb2(v6: i32): ; nextln: v1 = iconst.i32 1 ; nextln: v2 = iconst.i32 2 diff --git a/cranelift/filetests/licm/complex.cton b/cranelift/filetests/licm/complex.cton index fead0cf746..b5063d807f 100644 --- a/cranelift/filetests/licm/complex.cton +++ b/cranelift/filetests/licm/complex.cton @@ -1,6 +1,6 @@ test licm -function complex(i32) -> i32 { +function %complex(i32) -> i32 { ebb0(v0: i32): v1 = iconst.i32 1 @@ -39,7 +39,7 @@ ebb5(v16: i32): return v17 } -; sameln: function complex(i32) -> i32 { +; sameln: function %complex(i32) -> i32 { ; nextln: ebb6(v20: i32): ; nextln: v1 = iconst.i32 1 ; nextln: v2 = iconst.i32 4 diff --git a/cranelift/filetests/licm/multiple-blocks.cton b/cranelift/filetests/licm/multiple-blocks.cton index 54db640501..4738081d9a 100644 --- a/cranelift/filetests/licm/multiple-blocks.cton +++ b/cranelift/filetests/licm/multiple-blocks.cton @@ -1,6 +1,6 @@ test licm -function multiple_blocks(i32) -> i32 { +function %multiple_blocks(i32) -> i32 { ebb0(v0: i32): jump ebb1(v0) @@ -23,7 +23,7 @@ ebb3(v30: i32): jump ebb1(v30) } -; sameln:function multiple_blocks(i32) -> i32 { +; sameln:function %multiple_blocks(i32) -> i32 { ; nextln: ebb0(v0: i32): ; nextln: v2 = iconst.i32 1 ; nextln: v3 = iconst.i32 2 diff --git a/cranelift/filetests/licm/nested_loops.cton b/cranelift/filetests/licm/nested_loops.cton index e2d3846a0f..a32dd4b498 100644 --- a/cranelift/filetests/licm/nested_loops.cton +++ b/cranelift/filetests/licm/nested_loops.cton @@ -1,6 +1,6 @@ test licm -function nested_loops(i32) -> i32 { +function %nested_loops(i32) -> i32 { ebb0(v0: i32): v1 = iconst.i32 1 @@ -25,7 +25,7 @@ ebb3(v30: i32): } -; sameln:function nested_loops(i32) -> i32 { +; sameln:function %nested_loops(i32) -> i32 { ; nextln: ebb4(v12: i32): ; nextln: v1 = iconst.i32 1 ; nextln: v2 = iconst.i32 2 diff --git a/cranelift/filetests/parser/branch.cton b/cranelift/filetests/parser/branch.cton index ac979bbb26..4adb4b5d27 100644 --- a/cranelift/filetests/parser/branch.cton +++ b/cranelift/filetests/parser/branch.cton @@ -2,14 +2,14 @@ test cat ; Jumps with no arguments. The '()' empty argument list is optional. -function minimal() { +function %minimal() { ebb0: jump ebb1 ebb1: jump ebb0() } -; sameln: function minimal() { +; sameln: function %minimal() { ; nextln: ebb0: ; nextln: jump ebb1 ; nextln: @@ -18,14 +18,14 @@ ebb1: ; nextln: } ; Jumps with 1 arg. -function onearg(i32) { +function %onearg(i32) { ebb0(v90: i32): jump ebb1(v90) ebb1(v91: i32): jump ebb0(v91) } -; sameln: function onearg(i32) { +; sameln: function %onearg(i32) { ; nextln: ebb0($v90: i32): ; nextln: jump ebb1($v90) ; nextln: @@ -34,14 +34,14 @@ ebb1(v91: i32): ; nextln: } ; Jumps with 2 args. -function twoargs(i32, f32) { +function %twoargs(i32, f32) { ebb0(v90: i32, v91: f32): jump ebb1(v90, v91) ebb1(v92: i32, v93: f32): jump ebb0(v92, v93) } -; sameln: function twoargs(i32, f32) { +; sameln: function %twoargs(i32, f32) { ; nextln: ebb0($v90: i32, $v91: f32): ; nextln: jump ebb1($v90, $v91) ; nextln: @@ -50,14 +50,14 @@ ebb1(v92: i32, v93: f32): ; nextln: } ; Branches with no arguments. The '()' empty argument list is optional. -function minimal(i32) { +function %minimal(i32) { ebb0(v90: i32): brz v90, ebb1 ebb1: brnz v90, ebb1() } -; sameln: function minimal(i32) { +; sameln: function %minimal(i32) { ; nextln: ebb0($v90: i32): ; nextln: brz $v90, ebb1 ; nextln: @@ -65,14 +65,14 @@ ebb1: ; nextln: brnz.i32 $v90, ebb1 ; nextln: } -function twoargs(i32, f32) { +function %twoargs(i32, f32) { ebb0(v90: i32, v91: f32): brz v90, ebb1(v90, v91) ebb1(v92: i32, v93: f32): brnz v90, ebb0(v92, v93) } -; sameln: function twoargs(i32, f32) { +; sameln: function %twoargs(i32, f32) { ; nextln: ebb0($v90: i32, $v91: f32): ; nextln: brz $v90, ebb1($v90, $v91) ; nextln: @@ -80,7 +80,7 @@ ebb1(v92: i32, v93: f32): ; nextln: brnz.i32 $v90, ebb0($v92, $v93) ; nextln: } -function jumptable(i32) { +function %jumptable(i32) { jt200 = jump_table 0, 0 jt2 = jump_table 0, 0, ebb10, ebb40, ebb20, ebb30 @@ -94,7 +94,7 @@ ebb30: ebb40: trap } -; sameln: function jumptable(i32) { +; sameln: function %jumptable(i32) { ; nextln: jt0 = jump_table 0 ; nextln: jt1 = jump_table 0, 0, ebb0, ebb3, ebb1, ebb2 ; nextln: diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton index 67472f7efc..9d7c2c3d16 100644 --- a/cranelift/filetests/parser/call.cton +++ b/cranelift/filetests/parser/call.cton @@ -1,46 +1,46 @@ ; Parser tests for call and return syntax. test cat -function mini() { +function %mini() { ebb1: return } -; sameln: function mini() { +; sameln: function %mini() { ; nextln: ebb0: ; nextln: return ; nextln: } -function r1() -> i32, f32 { +function %r1() -> i32, f32 { ebb1: v1 = iconst.i32 3 v2 = f32const 0.0 return v1, v2 } -; sameln: function r1() -> i32, f32 { +; sameln: function %r1() -> i32, f32 { ; nextln: ebb0: ; nextln: $v1 = iconst.i32 3 ; nextln: $v2 = f32const 0.0 ; nextln: return $v1, $v2 ; nextln: } -function signatures() { +function %signatures() { sig10 = signature() sig11 = signature(i32, f64) -> i32, b1 - fn5 = sig11 foo - fn8 = function bar(i32) -> b1 + fn5 = sig11 %foo + fn8 = function %bar(i32) -> b1 } -; sameln: function signatures() { +; sameln: function %signatures() { ; nextln: $sig10 = signature() ; nextln: $sig11 = signature(i32, f64) -> i32, b1 ; nextln: sig2 = signature(i32) -> b1 -; nextln: $fn5 = $sig11 foo -; nextln: $fn8 = sig2 bar +; nextln: $fn5 = $sig11 %foo +; nextln: $fn8 = sig2 %bar ; nextln: } -function direct() { - fn0 = function none() - fn1 = function one() -> i32 - fn2 = function two() -> i32, f32 +function %direct() { + fn0 = function %none() + fn1 = function %one() -> i32 + fn2 = function %two() -> i32, f32 ebb0: call fn0() @@ -53,7 +53,7 @@ ebb0: ; check: $v2, $v3 = call $fn2() ; check: return -function indirect(i64) { +function %indirect(i64) { sig0 = signature(i64) sig1 = signature() -> i32 sig2 = signature() -> i32, f32 @@ -70,11 +70,11 @@ ebb0(v0: i64): ; check: return ; Special purpose function arguments -function special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret { +function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret { ebb0(v1: i32, v2: i32, v3: i32, v4: i32): return v4, v2, v3, v1 } -; check: function special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret { +; check: function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret { ; check: ebb0($v1: i32, $v2: i32, $v3: i32, $v4: i32): ; check: return $v4, $v2, $v3, $v1 ; check: } diff --git a/cranelift/filetests/parser/instruction_encoding.cton b/cranelift/filetests/parser/instruction_encoding.cton index 4971cba510..bc2e1dd239 100644 --- a/cranelift/filetests/parser/instruction_encoding.cton +++ b/cranelift/filetests/parser/instruction_encoding.cton @@ -4,7 +4,7 @@ isa riscv ; regex: WS=[ \t]* -function foo(i32, i32) { +function %foo(i32, i32) { ebb1(v0: i32, v1: i32): [-,-] v2 = iadd v0, v1 [-] trap @@ -13,7 +13,7 @@ ebb1(v0: i32, v1: i32): v9 = iadd v8, v7 [Iret#5] return v0, v8 } -; sameln: function foo(i32, i32) { +; sameln: function %foo(i32, i32) { ; nextln: $ebb1($v0: i32, $v1: i32): ; nextln: [-,-]$WS $v2 = iadd $v0, $v1 ; nextln: [-]$WS trap diff --git a/cranelift/filetests/parser/keywords.cton b/cranelift/filetests/parser/keywords.cton index eb15f2624d..37d0390a58 100644 --- a/cranelift/filetests/parser/keywords.cton +++ b/cranelift/filetests/parser/keywords.cton @@ -1,5 +1,5 @@ test cat ; 'function' is not a keyword, and can be used as the name of a function too. -function function() {} -; check: function function() +function %function() {} +; check: function %function() diff --git a/cranelift/filetests/parser/rewrite.cton b/cranelift/filetests/parser/rewrite.cton index e01391d156..f7ebfa3876 100644 --- a/cranelift/filetests/parser/rewrite.cton +++ b/cranelift/filetests/parser/rewrite.cton @@ -9,13 +9,13 @@ test cat ; Check that defining numbers are rewritten. -function defs() { +function %defs() { ebb100(v20: i32): v1000 = iconst.i32x8 5 v9200 = f64const 0x4.0p0 trap } -; sameln: function defs() { +; sameln: function %defs() { ; nextln: $ebb100($v20: i32): ; nextln: $v1000 = iconst.i32x8 5 ; nextln: $v9200 = f64const 0x1.0000000000000p2 @@ -23,13 +23,13 @@ ebb100(v20: i32): ; nextln: } ; Using values. -function use_value() { +function %use_value() { ebb100(v20: i32): v1000 = iadd_imm v20, 5 v200 = iadd v20, v1000 jump ebb100(v1000) } -; sameln: function use_value() { +; sameln: function %use_value() { ; nextln: ebb0($v20: i32): ; nextln: $v1000 = iadd_imm $v20, 5 ; nextln: $v200 = iadd $v20, $v1000 diff --git a/cranelift/filetests/parser/ternary.cton b/cranelift/filetests/parser/ternary.cton index 99cb15b566..3f320a61f0 100644 --- a/cranelift/filetests/parser/ternary.cton +++ b/cranelift/filetests/parser/ternary.cton @@ -1,7 +1,7 @@ test cat test verifier -function add_i96(i32, i32, i32, i32, i32, i32) -> i32, i32, i32 { +function %add_i96(i32, i32, i32, i32, i32, i32) -> i32, i32, i32 { ebb1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32): v10, v11 = iadd_cout v1, v4 ;check: $v10, $v11 = iadd_cout $v1, $v4 @@ -12,7 +12,7 @@ ebb1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32): return v10, v20, v30 } -function sub_i96(i32, i32, i32, i32, i32, i32) -> i32, i32, i32 { +function %sub_i96(i32, i32, i32, i32, i32, i32) -> i32, i32, i32 { ebb1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32): v10, v11 = isub_bout v1, v4 ;check: $v10, $v11 = isub_bout $v1, $v4 diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 5ad2f7a39d..1414668a52 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -1,24 +1,24 @@ test cat ; The smallest possible function. -function minimal() { +function %minimal() { ebb0: trap } -; sameln: function minimal() { +; sameln: function %minimal() { ; nextln: ebb0: ; nextln: trap ; nextln: } ; Create and use values. ; Polymorphic instructions with type suffix. -function ivalues() { +function %ivalues() { ebb0: v0 = iconst.i32 2 v1 = iconst.i8 6 v2 = ishl v0, v1 } -; sameln: function ivalues() { +; sameln: function %ivalues() { ; nextln: ebb0: ; nextln: $v0 = iconst.i32 2 ; nextln: $v1 = iconst.i8 6 @@ -26,23 +26,23 @@ ebb0: ; nextln: } ; Polymorphic istruction controlled by second operand. -function select() { +function %select() { ebb0(v90: i32, v91: i32, v92: b1): v0 = select v92, v90, v91 } -; sameln: function select() { +; sameln: function %select() { ; nextln: ebb0($v90: i32, $v91: i32, $v92: b1): ; nextln: $v0 = select $v92, $v90, $v91 ; nextln: } ; Lane indexes. -function lanes() { +function %lanes() { ebb0: v0 = iconst.i32x4 2 v1 = extractlane v0, 3 v2 = insertlane v0, 1, v1 } -; sameln: function lanes() { +; sameln: function %lanes() { ; nextln: ebb0: ; nextln: $v0 = iconst.i32x4 2 ; nextln: $v1 = extractlane $v0, 3 @@ -50,7 +50,7 @@ ebb0: ; nextln: } ; Integer condition codes. -function icmp(i32, i32) { +function %icmp(i32, i32) { ebb0(v90: i32, v91: i32): v0 = icmp eq v90, v91 v1 = icmp ult v90, v91 @@ -58,7 +58,7 @@ ebb0(v90: i32, v91: i32): v3 = irsub_imm v91, 45 br_icmp eq v90, v91, ebb0(v91, v90) } -; sameln: function icmp(i32, i32) { +; sameln: function %icmp(i32, i32) { ; nextln: ebb0($v90: i32, $v91: i32): ; nextln: $v0 = icmp eq $v90, $v91 ; nextln: $v1 = icmp ult $v90, $v91 @@ -68,13 +68,13 @@ ebb0(v90: i32, v91: i32): ; nextln: } ; Floating condition codes. -function fcmp(f32, f32) { +function %fcmp(f32, f32) { ebb0(v90: f32, v91: f32): v0 = fcmp eq v90, v91 v1 = fcmp uno v90, v91 v2 = fcmp lt v90, v91 } -; sameln: function fcmp(f32, f32) { +; sameln: function %fcmp(f32, f32) { ; nextln: ebb0($v90: f32, $v91: f32): ; nextln: $v0 = fcmp eq $v90, $v91 ; nextln: $v1 = fcmp uno $v90, $v91 @@ -83,19 +83,19 @@ ebb0(v90: f32, v91: f32): ; The bitcast instruction has two type variables: The controlling type variable ; controls the outout type, and the input type is a free variable. -function bitcast(i32, f32) { +function %bitcast(i32, f32) { ebb0(v90: i32, v91: f32): v0 = bitcast.i8x4 v90 v1 = bitcast.i32 v91 } -; sameln: function bitcast(i32, f32) { +; sameln: function %bitcast(i32, f32) { ; nextln: ebb0($v90: i32, $v91: f32): ; nextln: $v0 = bitcast.i8x4 $v90 ; nextln: $v1 = bitcast.i32 $v91 ; nextln: } ; Stack slot references -function stack() { +function %stack() { ss10 = stack_slot 8 ss2 = stack_slot 4 @@ -105,7 +105,7 @@ ebb0: stack_store v1, ss10+2 stack_store v2, ss2 } -; sameln: function stack() { +; sameln: function %stack() { ; nextln: $ss10 = stack_slot 8 ; nextln: $ss2 = stack_slot 4 @@ -116,21 +116,21 @@ ebb0: ; nextln: stack_store $v2, $ss2 ; Heap access instructions. -function heap(i32) { +function %heap(i32) { ; TODO: heap0 = heap %foo ebb0(v1: i32): v2 = heap_load.f32 v1 v3 = heap_load.f32 v1+12 heap_store v3, v1 } -; sameln: function heap(i32) { +; sameln: function %heap(i32) { ; nextln: ebb0($v1: i32): ; nextln: $v2 = heap_load.f32 $v1 ; nextln: $v3 = heap_load.f32 $v1+12 ; nextln: heap_store $v3, $v1 ; Memory access instructions. -function memory(i32) { +function %memory(i32) { ebb0(v1: i32): v2 = load.i64 v1 v3 = load.i64 aligned v1 @@ -143,7 +143,7 @@ ebb0(v1: i32): store aligned v3, v1+12 store notrap aligned v3, v1-12 } -; sameln: function memory(i32) { +; sameln: function %memory(i32) { ; nextln: ebb0($v1: i32): ; nextln: $v2 = load.i64 $v1 ; nextln: $v3 = load.i64 aligned $v1 @@ -158,13 +158,13 @@ ebb0(v1: i32): ; Register diversions. ; This test file has no ISA, so we can unly use register unit numbers. -function diversion(i32) { +function %diversion(i32) { ebb0(v1: i32): regmove v1, %10 -> %20 regmove v1, %20 -> %10 return } -; sameln: function diversion(i32) { +; sameln: function %diversion(i32) { ; nextln: ebb0($v1: i32): ; nextln: regmove $v1, %10 -> %20 ; nextln: regmove $v1, %20 -> %10 diff --git a/cranelift/filetests/regalloc/basic.cton b/cranelift/filetests/regalloc/basic.cton index 9046aeb711..36c0e5c81c 100644 --- a/cranelift/filetests/regalloc/basic.cton +++ b/cranelift/filetests/regalloc/basic.cton @@ -5,7 +5,7 @@ isa riscv ; regex: RX=%x\d+ -function add(i32, i32) { +function %add(i32, i32) { ebb0(v1: i32, v2: i32): v3 = iadd v1, v2 ; check: [R#0c,%x5] @@ -14,7 +14,7 @@ ebb0(v1: i32, v2: i32): } ; Function with a dead argument. -function dead_arg(i32, i32) -> i32{ +function %dead_arg(i32, i32) -> i32{ ebb0(v1: i32, v2: i32): ; not: regmove ; check: return $v1 @@ -22,7 +22,7 @@ ebb0(v1: i32, v2: i32): } ; Return a value from a different register. -function move1(i32, i32) -> i32 { +function %move1(i32, i32) -> i32 { ebb0(v1: i32, v2: i32): ; not: regmove ; check: regmove $v2, %x11 -> %x10 @@ -31,7 +31,7 @@ ebb0(v1: i32, v2: i32): } ; Swap two registers. -function swap(i32, i32) -> i32, i32 { +function %swap(i32, i32) -> i32, i32 { ebb0(v1: i32, v2: i32): ; not: regmove ; check: regmove $v2, %x11 -> $(tmp=$RX) diff --git a/cranelift/filetests/simple_gvn/basic.cton b/cranelift/filetests/simple_gvn/basic.cton index 06d0989d1e..c76ec12b88 100644 --- a/cranelift/filetests/simple_gvn/basic.cton +++ b/cranelift/filetests/simple_gvn/basic.cton @@ -1,6 +1,6 @@ test simple-gvn -function simple_redundancy(i32, i32) -> i32 { +function %simple_redundancy(i32, i32) -> i32 { ebb0(v0: i32, v1: i32): v2 = iadd v0, v1 v3 = iadd v0, v1 @@ -9,7 +9,7 @@ ebb0(v0: i32, v1: i32): return v4 } -function cascading_redundancy(i32, i32) -> i32 { +function %cascading_redundancy(i32, i32) -> i32 { ebb0(v0: i32, v1: i32): v2 = iadd v0, v1 v3 = iadd v0, v1 diff --git a/cranelift/filetests/verifier/bad_layout.cton b/cranelift/filetests/verifier/bad_layout.cton index ac29452958..fd597359be 100644 --- a/cranelift/filetests/verifier/bad_layout.cton +++ b/cranelift/filetests/verifier/bad_layout.cton @@ -1,6 +1,6 @@ test verifier -function test(i32) { +function %test(i32) { ebb0(v0: i32): jump ebb1 ; error: terminator return @@ -13,7 +13,7 @@ function test(i32) { return } -function test(i32) { ; Ok +function %test(i32) { ; Ok ebb0(v0: i32): return } diff --git a/cranelift/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs index 21e2a086b2..a0c9a64d90 100644 --- a/cranelift/tests/cfg_traversal.rs +++ b/cranelift/tests/cfg_traversal.rs @@ -27,7 +27,7 @@ fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) #[test] fn simple_traversal() { test_reverse_postorder_traversal(" - function test(i32) { + function %test(i32) { ebb0(v0: i32): brz v0, ebb1 jump ebb2 @@ -56,7 +56,7 @@ fn simple_traversal() { #[test] fn loops_one() { test_reverse_postorder_traversal(" - function test(i32) { + function %test(i32) { ebb0(v0: i32): jump ebb1 ebb1: @@ -74,7 +74,7 @@ fn loops_one() { #[test] fn loops_two() { test_reverse_postorder_traversal(" - function test(i32) { + function %test(i32) { ebb0(v0: i32): brz v0, ebb1 jump ebb2 @@ -99,7 +99,7 @@ fn loops_two() { #[test] fn loops_three() { test_reverse_postorder_traversal(" - function test(i32) { + function %test(i32) { ebb0(v0: i32): brz v0, ebb1 jump ebb2 @@ -129,7 +129,7 @@ fn loops_three() { #[test] fn back_edge_one() { test_reverse_postorder_traversal(" - function test(i32) { + function %test(i32) { ebb0(v0: i32): brz v0, ebb1 jump ebb2 diff --git a/lib/cretonne/src/ir/funcname.rs b/lib/cretonne/src/ir/funcname.rs index 0074c97502..4ab76d2001 100644 --- a/lib/cretonne/src/ir/funcname.rs +++ b/lib/cretonne/src/ir/funcname.rs @@ -6,73 +6,119 @@ use std::fmt::{self, Write}; use std::ascii::AsciiExt; -/// The name of a function can be any UTF-8 string. +/// The name of a function can be any sequence of bytes. /// /// Function names are mostly a testing and debugging tool. /// In particular, `.cton` files use function names to identify functions. #[derive(Debug, Clone, PartialEq, Eq, Default)] -pub struct FunctionName(String); +pub struct FunctionName(NameRepr); impl FunctionName { - /// Create new function name equal to `s`. - pub fn new>(s: S) -> FunctionName { - FunctionName(s.into()) + /// Creates a new function name from a sequence of bytes. + /// + /// # Examples + /// + /// ```rust + /// # use cretonne::ir::FunctionName; + /// // Create `FunctionName` from a string. + /// let name = FunctionName::new("hello"); + /// assert_eq!(name.to_string(), "%hello"); + /// + /// // Create `FunctionName` from a sequence of bytes. + /// let bytes: &[u8] = &[10, 9, 8]; + /// let name = FunctionName::new(bytes); + /// assert_eq!(name.to_string(), "#0a0908"); + /// ``` + pub fn new(v: T) -> FunctionName + where T: Into> + { + let vec = v.into(); + if vec.len() <= NAME_LENGTH_THRESHOLD { + let mut bytes = [0u8; NAME_LENGTH_THRESHOLD]; + for (i, &byte) in vec.iter().enumerate() { + bytes[i] = byte; + } + FunctionName(NameRepr::Short { + length: vec.len() as u8, + bytes: bytes, + }) + } else { + FunctionName(NameRepr::Long(vec)) + } } } -fn is_id_start(c: char) -> bool { - c.is_ascii() && (c == '_' || c.is_alphabetic()) +/// Tries to interpret bytes as ASCII alphanumerical characters and `_`. +fn try_as_name(bytes: &[u8]) -> Option { + let mut name = String::with_capacity(bytes.len()); + for c in bytes.iter().map(|&b| b as char) { + if c.is_ascii() && c.is_alphanumeric() || c == '_' { + name.push(c); + } else { + return None; + } + } + Some(name) } -fn is_id_continue(c: char) -> bool { - c.is_ascii() && (c == '_' || c.is_alphanumeric()) +const NAME_LENGTH_THRESHOLD: usize = 22; + +#[derive(Debug, Clone, PartialEq, Eq)] +enum NameRepr { + Short { + length: u8, + bytes: [u8; NAME_LENGTH_THRESHOLD], + }, + Long(Vec), } -// The function name may need quotes if it doesn't parse as an identifier. -fn needs_quotes(name: &str) -> bool { - let mut iter = name.chars(); - if let Some(ch) = iter.next() { - !is_id_start(ch) || !iter.all(is_id_continue) - } else { - // A blank function name needs quotes. - true +impl AsRef<[u8]> for NameRepr { + fn as_ref(&self) -> &[u8] { + match *self { + NameRepr::Short { length, ref bytes } => &bytes[0..length as usize], + NameRepr::Long(ref vec) => vec.as_ref(), + } + } +} + +impl Default for NameRepr { + fn default() -> Self { + NameRepr::Short { + length: 0, + bytes: [0; NAME_LENGTH_THRESHOLD], + } } } impl fmt::Display for FunctionName { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - if needs_quotes(&self.0) { - f.write_char('"')?; - for c in self.0.chars().flat_map(char::escape_default) { - f.write_char(c)?; - } - f.write_char('"') + if let Some(name) = try_as_name(self.0.as_ref()) { + write!(f, "%{}", name) } else { - f.write_str(&self.0) + f.write_char('#')?; + for byte in self.0.as_ref() { + write!(f, "{:02x}", byte)?; + } + Ok(()) } } } #[cfg(test)] mod tests { - use super::{needs_quotes, FunctionName}; + use super::FunctionName; #[test] - fn quoting() { - assert_eq!(needs_quotes(""), true); - assert_eq!(needs_quotes("x"), false); - assert_eq!(needs_quotes(" "), true); - assert_eq!(needs_quotes("0"), true); - assert_eq!(needs_quotes("x0"), false); - } - - #[test] - fn escaping() { - assert_eq!(FunctionName::new("").to_string(), "\"\""); - assert_eq!(FunctionName::new("x").to_string(), "x"); - assert_eq!(FunctionName::new(" ").to_string(), "\" \""); - assert_eq!(FunctionName::new(" \n").to_string(), "\" \\n\""); - assert_eq!(FunctionName::new("a\u{1000}v").to_string(), - "\"a\\u{1000}v\""); + fn displaying() { + assert_eq!(FunctionName::new("").to_string(), "%"); + assert_eq!(FunctionName::new("x").to_string(), "%x"); + assert_eq!(FunctionName::new("x_1").to_string(), "%x_1"); + assert_eq!(FunctionName::new(" ").to_string(), "#20"); + assert_eq!(FunctionName::new("кретон").to_string(), + "#d0bad180d0b5d182d0bed0bd"); + assert_eq!(FunctionName::new("印花棉布").to_string(), + "#e58db0e88ab1e6a389e5b883"); + assert_eq!(FunctionName::new(vec![0, 1, 2, 3, 4, 5]).to_string(), + "#000102030405"); } } diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index af66cc8868..818e4d9700 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -365,26 +365,26 @@ mod tests { #[test] fn basic() { let mut f = Function::new(); - assert_eq!(f.to_string(), "function \"\"() {\n}\n"); + assert_eq!(f.to_string(), "function %() {\n}\n"); - f.name = FunctionName::new("foo".to_string()); - assert_eq!(f.to_string(), "function foo() {\n}\n"); + f.name = FunctionName::new("foo"); + assert_eq!(f.to_string(), "function %foo() {\n}\n"); f.stack_slots.push(StackSlotData::new(4)); assert_eq!(f.to_string(), - "function foo() {\n ss0 = stack_slot 4\n}\n"); + "function %foo() {\n ss0 = stack_slot 4\n}\n"); let ebb = f.dfg.make_ebb(); f.layout.append_ebb(ebb); assert_eq!(f.to_string(), - "function foo() {\n ss0 = stack_slot 4\n\nebb0:\n}\n"); + "function %foo() {\n ss0 = stack_slot 4\n\nebb0:\n}\n"); f.dfg.append_ebb_arg(ebb, types::I8); assert_eq!(f.to_string(), - "function foo() {\n ss0 = stack_slot 4\n\nebb0(v0: i8):\n}\n"); + "function %foo() {\n ss0 = stack_slot 4\n\nebb0(v0: i8):\n}\n"); f.dfg.append_ebb_arg(ebb, types::F32.by(4).unwrap()); assert_eq!(f.to_string(), - "function foo() {\n ss0 = stack_slot 4\n\nebb0(v0: i8, v1: f32x4):\n}\n"); + "function %foo() {\n ss0 = stack_slot 4\n\nebb0(v0: i8, v1: f32x4):\n}\n"); } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index fa9f31df5a..3529bc95a3 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -723,10 +723,25 @@ impl<'a> Parser<'a> { // fn parse_function_name(&mut self) -> Result { match self.token() { - Some(Token::Identifier(s)) => { + Some(Token::Name(s)) => { self.consume(); Ok(FunctionName::new(s)) } + Some(Token::HexSequence(s)) => { + if s.len() % 2 != 0 { + return err!(self.loc, + "expected binary function name to have length multiple of two"); + } + let mut bin_name = Vec::with_capacity(s.len() / 2); + let mut i = 0; + while i + 2 <= s.len() { + let byte = u8::from_str_radix(&s[i..i + 2], 16).unwrap(); + bin_name.push(byte); + i += 2; + } + self.consume(); + Ok(FunctionName::new(bin_name)) + } _ => err!(self.loc, "expected function name"), } } @@ -1723,7 +1738,7 @@ mod tests { #[test] fn aliases() { - let (func, details) = Parser::new("function qux() { + let (func, details) = Parser::new("function %qux() { ebb0: v4 = iconst.i8 6 v3 -> v4 @@ -1731,7 +1746,7 @@ mod tests { }") .parse_function(None) .unwrap(); - assert_eq!(func.name.to_string(), "qux"); + assert_eq!(func.name.to_string(), "%qux"); let v4 = details.map.lookup_str("v4").unwrap(); assert_eq!(v4.to_string(), "v0"); let v3 = details.map.lookup_str("v3").unwrap(); @@ -1777,13 +1792,13 @@ mod tests { #[test] fn stack_slot_decl() { - let (func, _) = Parser::new("function foo() { + let (func, _) = Parser::new("function %foo() { ss3 = stack_slot 13 ss1 = stack_slot 1 }") .parse_function(None) .unwrap(); - assert_eq!(func.name.to_string(), "foo"); + assert_eq!(func.name.to_string(), "%foo"); let mut iter = func.stack_slots.keys(); let ss0 = iter.next().unwrap(); assert_eq!(ss0.to_string(), "ss0"); @@ -1794,7 +1809,7 @@ mod tests { assert_eq!(iter.next(), None); // Catch duplicate definitions. - assert_eq!(Parser::new("function bar() { + assert_eq!(Parser::new("function %bar() { ss1 = stack_slot 13 ss1 = stack_slot 1 }") @@ -1806,13 +1821,13 @@ mod tests { #[test] fn ebb_header() { - let (func, _) = Parser::new("function ebbs() { + let (func, _) = Parser::new("function %ebbs() { ebb0: ebb4(v3: i32): }") .parse_function(None) .unwrap(); - assert_eq!(func.name.to_string(), "ebbs"); + assert_eq!(func.name.to_string(), "%ebbs"); let mut ebbs = func.layout.ebbs(); @@ -1828,7 +1843,7 @@ mod tests { #[test] fn comments() { let (func, Details { comments, .. }) = Parser::new("; before - function comment() { ; decl + function %comment() { ; decl ss10 = stack_slot 13 ; stackslot. ; Still stackslot. jt10 = jump_table ebb0 @@ -1839,7 +1854,7 @@ mod tests { ; More trailing.") .parse_function(None) .unwrap(); - assert_eq!(func.name.to_string(), "comment"); + assert_eq!(func.name.to_string(), "%comment"); assert_eq!(comments.len(), 8); // no 'before' comment. assert_eq!(comments[0], Comment { @@ -1868,7 +1883,7 @@ mod tests { test verify set enable_float=false ; still preamble - function comment() {}") + function %comment() {}") .unwrap(); assert_eq!(tf.commands.len(), 2); assert_eq!(tf.commands[0].command, "cfg"); @@ -1884,23 +1899,23 @@ mod tests { assert_eq!(tf.preamble_comments[0].text, "; before"); assert_eq!(tf.preamble_comments[1].text, "; still preamble"); assert_eq!(tf.functions.len(), 1); - assert_eq!(tf.functions[0].0.name.to_string(), "comment"); + assert_eq!(tf.functions[0].0.name.to_string(), "%comment"); } #[test] fn isa_spec() { assert!(parse_test("isa - function foo() {}") + function %foo() {}") .is_err()); assert!(parse_test("isa riscv set enable_float=false - function foo() {}") + function %foo() {}") .is_err()); match parse_test("set enable_float=false isa riscv - function foo() {}") + function %foo() {}") .unwrap() .isa_spec { IsaSpec::None(_) => panic!("Expected some ISA"), @@ -1910,4 +1925,41 @@ mod tests { } } } + + #[test] + fn binary_function_name() { + // Valid characters in the name. + let func = Parser::new("function #1234567890AbCdEf() { + ebb0: + trap + }") + .parse_function(None) + .unwrap() + .0; + assert_eq!(func.name.to_string(), "#1234567890abcdef"); + + // Invalid characters in the name. + let mut parser = Parser::new("function #12ww() { + ebb0: + trap + }"); + assert!(parser.parse_function(None).is_err()); + + // The length of binary function name should be multiple of two. + let mut parser = Parser::new("function #1() { + ebb0: + trap + }"); + assert!(parser.parse_function(None).is_err()); + + // Empty binary function name should be valid. + let func = Parser::new("function #() { + ebb0: + trap + }") + .parse_function(None) + .unwrap() + .0; + assert_eq!(func.name.to_string(), "%"); + } } diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index f84c792d89..824eb97621 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -222,7 +222,7 @@ mod tests { #[test] fn details() { - let tf = parse_test("function detail() { + let tf = parse_test("function %detail() { ss10 = stack_slot 13 jt10 = jump_table ebb0 ebb0(v4: i32, v7: i32): From 4f9ff548bdbfb96916bffee13c0b3a3148d67b48 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 12 Jun 2017 13:53:12 -0700 Subject: [PATCH 0788/3084] Generalize rpo_cmp to handle all program points. When comparing instructions in the same EBB, behave like the RPO visits instructions in program order. - Add a Layout::pp_ebb() method for convenience. It gets the EBB containing any program point. - Add a conversion from ValueDef to ExpandedProgramPoint so it can be used with the rpo_cmp method. --- lib/cretonne/src/dominator_tree.rs | 32 +++++++++++++++++++++++++----- lib/cretonne/src/ir/layout.rs | 12 +++++++++++ lib/cretonne/src/ir/progpoint.rs | 9 +++++++++ 3 files changed, 48 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 6d42cefb36..66fdeb2ff0 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -2,7 +2,7 @@ use entity_map::EntityMap; use flowgraph::{ControlFlowGraph, BasicBlock}; -use ir::{Ebb, Inst, Function, Layout, ProgramOrder}; +use ir::{Ebb, Inst, Function, Layout, ProgramOrder, ExpandedProgramPoint}; use packed_option::PackedOption; use std::cmp::Ordering; @@ -66,11 +66,26 @@ impl DominatorTree { self.nodes[ebb].idom.into() } - /// Compare two EBBs relative to a reverse post-order traversal of the control-flow graph. + /// Compare two EBBs relative to the reverse post-order. + fn rpo_cmp_ebb(&self, a: Ebb, b: Ebb) -> Ordering { + + self.nodes[a].rpo_number.cmp(&self.nodes[b].rpo_number) + } + + /// Compare two program points relative to a reverse post-order traversal of the control-flow + /// graph. /// /// Return `Ordering::Less` if `a` comes before `b` in the RPO. - pub fn rpo_cmp(&self, a: Ebb, b: Ebb) -> Ordering { - self.nodes[a].rpo_number.cmp(&self.nodes[b].rpo_number) + /// + /// If `a` and `b` belong to the same EBB, compare their relative position in the EBB. + pub fn rpo_cmp(&self, a: A, b: B, layout: &Layout) -> Ordering + where A: Into, + B: Into + { + let a = a.into(); + let b = b.into(); + self.rpo_cmp_ebb(layout.pp_ebb(a), layout.pp_ebb(b)) + .then(layout.cmp(a, b)) } /// Returns `true` if `a` dominates `b`. @@ -118,7 +133,7 @@ impl DominatorTree { layout: &Layout) -> BasicBlock { loop { - match self.rpo_cmp(a.0, b.0) { + match self.rpo_cmp_ebb(a.0, b.0) { Ordering::Less => { // `a` comes before `b` in the RPO. Move `b` up. let idom = self.nodes[b.0].idom.expect("Unreachable basic block?"); @@ -349,6 +364,13 @@ mod test { assert!(!dt.dominates(br_ebb1_ebb0, jmp_ebb3_ebb1, &func.layout)); assert!(dt.dominates(jmp_ebb3_ebb1, br_ebb1_ebb0, &func.layout)); + assert_eq!(dt.rpo_cmp(ebb3, ebb3, &func.layout), Ordering::Equal); + assert_eq!(dt.rpo_cmp(ebb3, ebb1, &func.layout), Ordering::Less); + assert_eq!(dt.rpo_cmp(ebb3, jmp_ebb3_ebb1, &func.layout), + Ordering::Less); + assert_eq!(dt.rpo_cmp(jmp_ebb3_ebb1, jmp_ebb1_ebb2, &func.layout), + Ordering::Less); + assert_eq!(dt.cfg_postorder(), &[ebb2, ebb0, ebb1, ebb3]); } } diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 06ab92612b..3d569adcc6 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -412,6 +412,18 @@ impl Layout { } } + /// Get the EBB containing the program point `pp`. Panic if `pp` is not in the layout. + pub fn pp_ebb(&self, pp: PP) -> Ebb + where PP: Into + { + match pp.into() { + ExpandedProgramPoint::Ebb(ebb) => ebb, + ExpandedProgramPoint::Inst(inst) => { + self.inst_ebb(inst).expect("Program point not in layout") + } + } + } + /// Append `inst` to the end of `ebb`. pub fn append_inst(&mut self, inst: Inst, ebb: Ebb) { assert_eq!(self.inst_ebb(inst), None); diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index c99e79dbe6..3134a53603 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -63,6 +63,15 @@ impl From for ExpandedProgramPoint { } } +impl From for ExpandedProgramPoint { + fn from(def: ValueDef) -> ExpandedProgramPoint { + match def { + ValueDef::Res(inst, _) => inst.into(), + ValueDef::Arg(ebb, _) => ebb.into(), + } + } +} + impl From for ExpandedProgramPoint { fn from(pp: ProgramPoint) -> ExpandedProgramPoint { if pp.0 & 1 == 0 { From a1dfd0f06f615b3c30a34acc76e5589601f2dafa Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 12 Jun 2017 14:52:42 -0700 Subject: [PATCH 0789/3084] Generalize DominatorTree::dominates. This is now a generic function that can test arbitrary combinations of instructions and EBBs for dominance. It can handle anything that converts into an expanded program point, including a ValueDef. Also fix a bug if the earlier dominates() function which didn't properly handle block layouts that were not topologically ordered. --- lib/cretonne/src/dominator_tree.rs | 120 ++++++++++++++++++++++++++++- 1 file changed, 117 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 66fdeb2ff0..8022ee1096 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -97,9 +97,48 @@ impl DominatorTree { /// is unreachable. /// /// An instruction is considered to dominate itself. - pub fn dominates(&self, a: Inst, b: Inst, layout: &Layout) -> bool { - let ebb_a = layout.inst_ebb(a).expect("Instruction not in layout."); - self.ebb_dominates(ebb_a, b, layout) && layout.cmp(a, b) != Ordering::Greater + pub fn dominates(&self, a: A, b: B, layout: &Layout) -> bool + where A: Into, + B: Into + { + let a = a.into(); + let b = b.into(); + match a { + ExpandedProgramPoint::Ebb(ebb_a) => { + a == b || self.last_dominator(ebb_a, b, layout).is_some() + } + ExpandedProgramPoint::Inst(inst_a) => { + let ebb_a = layout.inst_ebb(inst_a).expect("Instruction not in layout."); + match self.last_dominator(ebb_a, b, layout) { + Some(last) => layout.cmp(inst_a, last) != Ordering::Greater, + None => false, + } + } + } + } + + /// Find the last instruction in `a` that dominates `b`. + /// If no instructions in `a` dominate `b`, return `None`. + fn last_dominator(&self, a: Ebb, b: B, layout: &Layout) -> Option + where B: Into + { + let (mut ebb_b, mut inst_b) = match b.into() { + ExpandedProgramPoint::Ebb(ebb) => (ebb, None), + ExpandedProgramPoint::Inst(inst) => { + (layout.inst_ebb(inst).expect("Instruction not in layout."), Some(inst)) + } + }; + let rpo_a = self.nodes[a].rpo_number; + + // Run a finger up the dominator tree from b until we see a. + // Do nothing if b is unreachable. + while rpo_a < self.nodes[ebb_b].rpo_number { + let idom = self.idom(ebb_b).expect("Shouldn't meet unreachable here."); + ebb_b = layout.inst_ebb(idom).expect("Dominator got removed."); + inst_b = Some(idom); + } + + if a == ebb_b { inst_b } else { None } } /// Returns `true` if `ebb_a` dominates `b`. @@ -373,4 +412,79 @@ mod test { assert_eq!(dt.cfg_postorder(), &[ebb2, ebb0, ebb1, ebb3]); } + + #[test] + fn backwards_layout() { + let mut func = Function::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + let ebb2 = func.dfg.make_ebb(); + + let jmp02; + let jmp21; + let trap; + { + let dfg = &mut func.dfg; + let cur = &mut Cursor::new(&mut func.layout); + + cur.insert_ebb(ebb0); + jmp02 = dfg.ins(cur).jump(ebb2, &[]); + + cur.insert_ebb(ebb1); + trap = dfg.ins(cur).trap(); + + cur.insert_ebb(ebb2); + jmp21 = dfg.ins(cur).jump(ebb1, &[]); + } + + let cfg = ControlFlowGraph::with_function(&func); + let dt = DominatorTree::with_function(&func, &cfg); + + assert_eq!(func.layout.entry_block(), Some(ebb0)); + assert_eq!(dt.idom(ebb0), None); + assert_eq!(dt.idom(ebb1), Some(jmp21)); + assert_eq!(dt.idom(ebb2), Some(jmp02)); + + assert!(dt.dominates(ebb0, ebb0, &func.layout)); + assert!(dt.dominates(ebb0, jmp02, &func.layout)); + assert!(dt.dominates(ebb0, ebb1, &func.layout)); + assert!(dt.dominates(ebb0, trap, &func.layout)); + assert!(dt.dominates(ebb0, ebb2, &func.layout)); + assert!(dt.dominates(ebb0, jmp21, &func.layout)); + + assert!(!dt.dominates(jmp02, ebb0, &func.layout)); + assert!(dt.dominates(jmp02, jmp02, &func.layout)); + assert!(dt.dominates(jmp02, ebb1, &func.layout)); + assert!(dt.dominates(jmp02, trap, &func.layout)); + assert!(dt.dominates(jmp02, ebb2, &func.layout)); + assert!(dt.dominates(jmp02, jmp21, &func.layout)); + + assert!(!dt.dominates(ebb1, ebb0, &func.layout)); + assert!(!dt.dominates(ebb1, jmp02, &func.layout)); + assert!(dt.dominates(ebb1, ebb1, &func.layout)); + assert!(dt.dominates(ebb1, trap, &func.layout)); + assert!(!dt.dominates(ebb1, ebb2, &func.layout)); + assert!(!dt.dominates(ebb1, jmp21, &func.layout)); + + assert!(!dt.dominates(trap, ebb0, &func.layout)); + assert!(!dt.dominates(trap, jmp02, &func.layout)); + assert!(!dt.dominates(trap, ebb1, &func.layout)); + assert!(dt.dominates(trap, trap, &func.layout)); + assert!(!dt.dominates(trap, ebb2, &func.layout)); + assert!(!dt.dominates(trap, jmp21, &func.layout)); + + assert!(!dt.dominates(ebb2, ebb0, &func.layout)); + assert!(!dt.dominates(ebb2, jmp02, &func.layout)); + assert!(dt.dominates(ebb2, ebb1, &func.layout)); + assert!(dt.dominates(ebb2, trap, &func.layout)); + assert!(dt.dominates(ebb2, ebb2, &func.layout)); + assert!(dt.dominates(ebb2, jmp21, &func.layout)); + + assert!(!dt.dominates(jmp21, ebb0, &func.layout)); + assert!(!dt.dominates(jmp21, jmp02, &func.layout)); + assert!(dt.dominates(jmp21, ebb1, &func.layout)); + assert!(dt.dominates(jmp21, trap, &func.layout)); + assert!(!dt.dominates(jmp21, ebb2, &func.layout)); + assert!(dt.dominates(jmp21, jmp21, &func.layout)); + } } From ea8a8a95a89c9c7f403ffa1e054a4b959f077934 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 12 Jun 2017 14:59:34 -0700 Subject: [PATCH 0790/3084] Remove the ebb_dominates function. This is now subsumed by the generic 'dominates' function. --- lib/cretonne/src/dominator_tree.rs | 22 ---------------------- lib/cretonne/src/licm.rs | 4 ++-- lib/cretonne/src/loop_analysis.rs | 4 ++-- lib/cretonne/src/verifier/mod.rs | 2 +- 4 files changed, 5 insertions(+), 27 deletions(-) diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 8022ee1096..14f08e4263 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -141,28 +141,6 @@ impl DominatorTree { if a == ebb_b { inst_b } else { None } } - /// Returns `true` if `ebb_a` dominates `b`. - /// - /// This means that every control-flow path from the function entry to `b` must go through - /// `ebb_a`. - /// - /// Dominance is ill defined for unreachable blocks. This function can always determine - /// dominance for instructions in the same EBB, but otherwise returns `false` if either block - /// is unreachable. - pub fn ebb_dominates(&self, ebb_a: Ebb, mut b: Inst, layout: &Layout) -> bool { - let mut ebb_b = layout.inst_ebb(b).expect("Instruction not in layout."); - let rpo_a = self.nodes[ebb_a].rpo_number; - - // Run a finger up the dominator tree from b until we see a. - // Do nothing if b is unreachable. - while rpo_a < self.nodes[ebb_b].rpo_number { - b = self.idom(ebb_b).expect("Shouldn't meet unreachable here."); - ebb_b = layout.inst_ebb(b).expect("Dominator got removed."); - } - - ebb_a == ebb_b - } - /// Compute the common dominator of two basic blocks. /// /// Both basic blocks are assumed to be reachable. diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 1c9e1930fd..0fc8212460 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -77,7 +77,7 @@ fn create_pre_header(header: Ebb, } for &(_, last_inst) in cfg.get_predecessors(header) { // We only follow normal edges (not the back edges) - if !domtree.ebb_dominates(header.clone(), last_inst, &func.layout) { + if !domtree.dominates(header, last_inst, &func.layout) { change_branch_jump_destination(last_inst, pre_header, func); } } @@ -108,7 +108,7 @@ fn has_pre_header(layout: &Layout, let mut found = false; for &(pred_ebb, last_inst) in cfg.get_predecessors(header) { // We only count normal edges (not the back edges) - if !domtree.ebb_dominates(header.clone(), last_inst, layout) { + if !domtree.dominates(header, last_inst, layout) { if found { // We have already found one, there are more than one return None; diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index f5ea45424e..5ed5f8a4d8 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -132,7 +132,7 @@ impl LoopAnalysis { for &ebb in domtree.cfg_postorder().iter().rev() { for &(_, pred_inst) in cfg.get_predecessors(ebb) { // If the ebb dominates one of its predecessors it is a back edge - if domtree.ebb_dominates(ebb, pred_inst, layout) { + if domtree.dominates(ebb, pred_inst, layout) { // This ebb is a loop header, so we create its associated loop let lp = self.loops.push(LoopData::new(ebb, None)); self.ebb_loop_map[ebb] = lp.into(); @@ -156,7 +156,7 @@ impl LoopAnalysis { for lp in self.loops().rev() { for &(pred, pred_inst) in cfg.get_predecessors(self.loops[lp].header) { // We follow the back edges - if domtree.ebb_dominates(self.loops[lp].header, pred_inst, layout) { + if domtree.dominates(self.loops[lp].header, pred_inst, layout) { stack.push(pred); } } diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 5c283dd6d3..a2f806a447 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -381,7 +381,7 @@ impl<'a> Verifier<'a> { ebb); } // The defining EBB dominates the instruction using this value. - if !self.domtree.ebb_dominates(ebb, loc_inst, &self.func.layout) { + if !self.domtree.dominates(ebb, loc_inst, &self.func.layout) { return err!(loc_inst, "uses value arg from non-dominating {}", ebb); } } From 5336bbd4cc7c357b05db21c55b6fbc7077a2a714 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 13 Jun 2017 13:39:52 -0700 Subject: [PATCH 0791/3084] Add RISC-V encodings for spill and fill. Add a Stack() class for specifying operand constraints for values on the stack. Add encoding recipes for RISC-V spill and fill instructions. Don't implement the encoding recipe functions yet since we don't have the stack slot layout yet. --- lib/cretonne/meta/cdsl/isa.py | 9 ++++++--- lib/cretonne/meta/cdsl/registers.py | 12 ++++++++++++ lib/cretonne/meta/gen_encoding.py | 6 +++++- lib/cretonne/meta/isa/riscv/encodings.py | 11 ++++++++++- lib/cretonne/meta/isa/riscv/recipes.py | 13 ++++++++++++- lib/cretonne/src/isa/riscv/binemit.rs | 8 ++++++++ 6 files changed, 53 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index a2387bc0ff..bc85f7fb8f 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -1,7 +1,7 @@ """Defining instruction set architectures.""" from __future__ import absolute_import from .predicates import And -from .registers import RegClass, Register +from .registers import RegClass, Register, Stack from .ast import Apply # The typing module is only required by mypy, and we don't use these imports @@ -14,7 +14,7 @@ try: from .settings import SettingGroup # noqa from .types import ValueType # noqa from .registers import RegBank # noqa - OperandConstraint = Union[RegClass, Register, int] + OperandConstraint = Union[RegClass, Register, int, Stack] ConstraintSeq = Union[OperandConstraint, Tuple[OperandConstraint, ...]] # Instruction specification for encodings. Allows for predicated # instructions. @@ -187,6 +187,7 @@ class EncRecipe(object): - A `Register` specifying a fixed-register operand. - An integer indicating that this result is tied to a value operand, so they must use the same register. + - A `Stack` specifying a value in a stack slot. The `branch_range` argument must be provided for recipes that can encode branch instructions. It is an `(origin, bits)` tuple describing the exact @@ -245,7 +246,9 @@ class EncRecipe(object): # Check that it is in range. assert c >= 0 and c < len(self.ins) else: - assert isinstance(c, RegClass) or isinstance(c, Register) + assert (isinstance(c, RegClass) + or isinstance(c, Register) + or isinstance(c, Stack)) return seq def ties(self): diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index caaacd20e4..880f9096d4 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -340,3 +340,15 @@ class Register(object): # type: (RegClass, int) -> None self.regclass = rc self.unit = unit + + +class Stack(object): + """ + An operand that must be in a stack slot. + + A `Stack` object can be used to indicate an operand constraint for a value + operand that must live in a stack slot. + """ + def __init__(self, rc): + # type: (RegClass) -> None + self.regclass = rc diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index c56db8fc79..135940dd23 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -56,7 +56,7 @@ from unique_table import UniqueSeqTable from collections import OrderedDict, defaultdict import math import itertools -from cdsl.registers import RegClass, Register +from cdsl.registers import RegClass, Register, Stack from cdsl.predicates import FieldPredicate try: @@ -519,6 +519,10 @@ def emit_operand_constraints( assert cons == tied[n], "Invalid tied constraint" fmt.format('kind: ConstraintKind::Tied({}),', cons) fmt.format('regclass: {},', recipe.ins[cons]) + elif isinstance(cons, Stack): + assert n not in tied, "Can't tie stack operand" + fmt.line('kind: ConstraintKind::Stack,') + fmt.format('regclass: {},', cons.regclass) else: raise AssertionError( 'Unsupported constraint {}'.format(cons)) diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index ffc848ef06..2b59964007 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -6,8 +6,9 @@ from base import instructions as base from base.immediates import intcc from .defs import RV32, RV64 from .recipes import OPIMM, OPIMM32, OP, OP32, LUI, BRANCH, JALR, JAL +from .recipes import LOAD, STORE from .recipes import R, Rshamt, Ricmp, I, Iicmp, Iret -from .recipes import U, UJ, UJcall, SB, SBzero +from .recipes import U, UJ, UJcall, SB, SBzero, GPsp, GPfi from .settings import use_m from cdsl.ast import Var @@ -114,3 +115,11 @@ for inst, f3 in [ # is added by legalize_signature(). RV32.enc(base.x_return, Iret, JALR()) RV64.enc(base.x_return, Iret, JALR()) + +# Spill and fill. +RV32.enc(base.spill.i32, GPsp, STORE(0b010)) +RV64.enc(base.spill.i32, GPsp, STORE(0b010)) +RV64.enc(base.spill.i64, GPsp, STORE(0b011)) +RV32.enc(base.fill.i32, GPfi, LOAD(0b010)) +RV64.enc(base.fill.i32, GPfi, LOAD(0b010)) +RV64.enc(base.fill.i64, GPfi, LOAD(0b011)) diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index b5aa56bc85..1a7e98d0fe 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -11,8 +11,9 @@ instruction formats described in the reference: from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt +from cdsl.registers import Stack from base.formats import Binary, BinaryImm, MultiAry, IntCompare, IntCompareImm -from base.formats import UnaryImm, BranchIcmp, Branch, Jump, Call +from base.formats import Unary, UnaryImm, BranchIcmp, Branch, Jump, Call from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit @@ -134,3 +135,13 @@ SBzero = EncRecipe( 'SBzero', Branch, size=4, ins=(GPR), outs=(), branch_range=(0, 13)) + +# Spill of a GPR. +GPsp = EncRecipe( + 'GPsp', Unary, size=4, + ins=GPR, outs=Stack(GPR)) + +# Fill of a GPR. +GPfi = EncRecipe( + 'GPfi', Unary, size=4, + ins=Stack(GPR), outs=GPR) diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 12a3eb462f..e025b3f9d4 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -333,3 +333,11 @@ fn recipe_ujcall(func: &Function, inst: Inst, sink: &mut panic!("Expected Call format: {:?}", func.dfg[inst]); } } + +fn recipe_gpsp(_func: &Function, _inst: Inst, _sink: &mut CS) { + unimplemented!(); +} + +fn recipe_gpfi(_func: &Function, _inst: Inst, _sink: &mut CS) { + unimplemented!(); +} From 9cea092bc32bd37a6cd49b457c9b94e293932256 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 12 Jun 2017 11:55:01 -0700 Subject: [PATCH 0792/3084] Track transient register counts in Pressure. The register pressure tracker now has to separate register counts: base and transient. The transient counts are used to track spikes of register pressure, such as dead defs or temporary registers needed to satisfy instruction constraints. The base counts represent long-lived variables. --- lib/cretonne/src/regalloc/pressure.rs | 60 ++++++++++++++++++++++----- 1 file changed, 49 insertions(+), 11 deletions(-) diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index 14e4441697..6ddf6bdf4e 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -26,6 +26,12 @@ //! //! Currently, the only register bank with multiple top-level registers is the `arm32` //! floating-point register bank which has `S`, `D`, and `Q` top-level classes. +//! +//! # Base and transient counts +//! +//! We maintain two separate register counts per top-level register class: base counts and +//! transient counts. The base counts are adjusted with the `take` and `free` functions. The +//! transient counts are adjusted with `take_transient` and `free_transient`. // Remove once we're using the pressure tracker. #![allow(dead_code)] @@ -37,11 +43,12 @@ use std::iter::ExactSizeIterator; /// Information per top-level register class. /// -/// Everything but the count is static information computed from the constructor arguments. +/// Everything but the counts is static information computed from the constructor arguments. #[derive(Default)] struct TopRC { // Number of registers currently used from this register class. - count: u32, + base_count: u32, + transient_count: u32, // Max number of registers that can be allocated. limit: u32, @@ -56,6 +63,12 @@ struct TopRC { num_toprcs: u8, } +impl TopRC { + fn total_count(&self) -> u32 { + self.base_count + self.transient_count + } +} + pub struct Pressure { // Bit mask of top-level register classes that are aliased by other top-level register classes. // Unaliased register classes can use a simpler interference algorithm. @@ -108,12 +121,16 @@ impl Pressure { /// If not, returns a bit-mask of top-level register classes that are interfering. Register /// pressure should be eased in one of the returned top-level register classes before calling /// `can_take()` to check again. - pub fn check_avail(&self, rc: RegClass) -> RegClassMask { + fn check_avail(&self, rc: RegClass) -> RegClassMask { let entry = &self.toprc[rc.toprc as usize]; let mask = 1 << rc.toprc; if self.aliased & mask == 0 { // This is a simple unaliased top-level register class. - if entry.count < entry.limit { 0 } else { mask } + if entry.total_count() < entry.limit { + 0 + } else { + mask + } } else { // This is the more complicated case. The top-level register class has aliases. self.check_avail_aliased(entry) @@ -142,9 +159,9 @@ impl Pressure { let u = if rcw < width { // We can't take more than the total number of register units in the class. // This matters for arm32 S-registers which can only ever lock out 16 D-registers. - min(rc.count * width, rc.limit * rcw) + min(rc.total_count() * width, rc.limit * rcw) } else { - rc.count * rcw + rc.total_count() * rcw }; // If this top-level RC on its own is responsible for exceeding our limit, return it @@ -169,20 +186,41 @@ impl Pressure { /// Take a register from `rc`. /// - /// This assumes that `can_take(rc)` already returned 0. + /// This does not check if there are enough registers available. pub fn take(&mut self, rc: RegClass) { - self.toprc[rc.toprc as usize].count += 1 + self.toprc[rc.toprc as usize].base_count += 1 } /// Free a register in `rc`. pub fn free(&mut self, rc: RegClass) { - self.toprc[rc.toprc as usize].count -= 1 + self.toprc[rc.toprc as usize].base_count -= 1 } - /// Reset all counts to 0. + /// Reset all counts to 0, both base and transient. pub fn reset(&mut self) { for e in self.toprc.iter_mut() { - e.count = 0; + e.base_count = 0; + e.transient_count = 0; + } + } + + /// Try to increment a transient counter. + /// + /// This will fail if there are not enough registers available. + pub fn take_transient(&mut self, rc: RegClass) -> Result<(), RegClassMask> { + let mask = self.check_avail(rc); + if mask == 0 { + self.toprc[rc.toprc as usize].transient_count += 1; + Ok(()) + } else { + Err(mask) + } + } + + /// Reset all transient counts to 0. + pub fn reset_transient(&mut self) { + for e in self.toprc.iter_mut() { + e.transient_count = 0; } } } From 63a372fd806a8daf78d6243a43575bf04d2d9337 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 8 Jun 2017 10:33:22 -0700 Subject: [PATCH 0793/3084] Basic spilling implementation. Add a spilling pass which lowers register pressure by assigning SSA values to the stack. Important missing features: - Resolve conflicts where an instruction uses the same value more than once in incompatible ways. - Deal with EBB arguments. Fix bugs in the reload pass exposed by the first test case: - Create live ranges for temporary registers. - Set encodings on created spill and fill instructions. --- cranelift/filetests/regalloc/spill.cton | 44 +++ lib/cretonne/src/regalloc/context.rs | 17 +- .../src/regalloc/live_value_tracker.rs | 15 +- lib/cretonne/src/regalloc/liveness.rs | 8 + lib/cretonne/src/regalloc/mod.rs | 1 + lib/cretonne/src/regalloc/pressure.rs | 11 + lib/cretonne/src/regalloc/reload.rs | 31 ++- lib/cretonne/src/regalloc/spilling.rs | 259 ++++++++++++++++++ 8 files changed, 380 insertions(+), 6 deletions(-) create mode 100644 cranelift/filetests/regalloc/spill.cton create mode 100644 lib/cretonne/src/regalloc/spilling.rs diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton new file mode 100644 index 0000000000..a6565b987e --- /dev/null +++ b/cranelift/filetests/regalloc/spill.cton @@ -0,0 +1,44 @@ +test regalloc + +; Test the spiler on an ISA with few registers. +; RV32E has 16 registers, where: +; - %x0 is hardwired to zero. +; - %x1 is the return address. +; - %x2 is the stack pointer. +; - %x3 is the global pointer. +; - %x4 is the thread pointer. +; - %x10-%x15 are function arguments. +; +; regex: V=v\d+ + +isa riscv enable_e + +; In straight-line code, the first value defined is spilled. +; That is the argument. +function %pyramid(i32) -> i32 { +ebb0(v1: i32): +; check: $v1 = spill $(rv1=$V) + v2 = iadd_imm v1, 12 + v3 = iadd_imm v2, 12 + v4 = iadd_imm v3, 12 + v5 = iadd_imm v4, 12 + v6 = iadd_imm v5, 12 + v7 = iadd_imm v6, 12 + v8 = iadd_imm v7, 12 + v9 = iadd_imm v8, 12 + v10 = iadd_imm v9, 12 + v11 = iadd_imm v10, 12 + v12 = iadd_imm v11, 12 + v31 = iadd v11, v12 + v30 = iadd v31, v10 + v29 = iadd v30, v9 + v28 = iadd v29, v8 + v27 = iadd v28, v7 + v26 = iadd v27, v6 + v25 = iadd v26, v5 + v24 = iadd v25, v4 + v23 = iadd v24, v3 + v22 = iadd v23, v2 + v21 = iadd v22, v1 + return v21 +} diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 3cf2b5dd81..b404602a34 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -12,6 +12,7 @@ use regalloc::coloring::Coloring; use regalloc::live_value_tracker::LiveValueTracker; use regalloc::liveness::Liveness; use regalloc::reload::Reload; +use regalloc::spilling::Spilling; use result::CtonResult; use topo_order::TopoOrder; use verifier::{verify_context, verify_liveness}; @@ -21,6 +22,7 @@ pub struct Context { liveness: Liveness, topo: TopoOrder, tracker: LiveValueTracker, + spilling: Spilling, reload: Reload, coloring: Coloring, } @@ -35,6 +37,7 @@ impl Context { liveness: Liveness::new(), topo: TopoOrder::new(), tracker: LiveValueTracker::new(), + spilling: Spilling::new(), reload: Reload::new(), coloring: Coloring::new(), } @@ -62,7 +65,19 @@ impl Context { verify_liveness(isa, func, cfg, &self.liveness)?; } - // TODO: Second pass: Spilling. + // Second pass: Spilling. + self.spilling + .run(isa, + func, + domtree, + &mut self.liveness, + &mut self.topo, + &mut self.tracker); + + if isa.flags().enable_verifier() { + verify_context(func, cfg, domtree, Some(isa))?; + verify_liveness(isa, func, cfg, &self.liveness)?; + } // Third pass: Reload. self.reload diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index a5ee8197e8..ccaf6c9209 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -12,7 +12,6 @@ use partition_slice::partition_slice; use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; use regalloc::liverange::LiveRange; - use std::collections::HashMap; type ValueList = EntityList; @@ -299,6 +298,20 @@ impl LiveValueTracker { self.live.remove_dead_values(); } + /// Process new spills. + /// + /// Any values where `f` returns true are spilled and will be treated as if their affinity was + /// `Stack`. + pub fn process_spills(&mut self, mut f: F) + where F: FnMut(Value) -> bool + { + for lv in &mut self.live.values { + if f(lv.value) { + lv.affinity = Affinity::Stack; + } + } + } + /// Save the current set of live values so it is associated with `idom`. fn save_idom_live_set(&mut self, idom: Inst) { let values = self.live.values.iter().map(|lv| lv.value); diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 3ced8ebc00..afccbffaa3 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -182,6 +182,7 @@ use isa::{TargetIsa, EncInfo}; use regalloc::affinity::Affinity; use regalloc::liverange::LiveRange; use sparse_map::SparseMap; +use std::mem; /// A set of live ranges, indexed by value number. type LiveRangeSet = SparseMap; @@ -338,6 +339,13 @@ impl Liveness { &mut lr.affinity } + /// Change the affinity of `value` to `Stack` and return the previous affinity. + pub fn spill(&mut self, value: Value) -> Affinity { + let mut lr = self.ranges.get_mut(value).expect("Value has no live range"); + mem::replace(&mut lr.affinity, Affinity::Stack) + } + + /// Compute the live ranges of all SSA values used in `func`. /// This clears out any existing analysis stored in this data structure. pub fn compute(&mut self, isa: &TargetIsa, func: &Function, cfg: &ControlFlowGraph) { diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index 4a86abbe93..59eb341eb0 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -14,6 +14,7 @@ mod diversion; mod pressure; mod reload; mod solver; +mod spilling; pub use self::allocatable_set::AllocatableSet; pub use self::context::Context; diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index 6ddf6bdf4e..1398bff8d6 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -39,6 +39,7 @@ use isa::registers::{RegInfo, MAX_TOPRCS, RegClass, RegClassMask}; use regalloc::AllocatableSet; use std::cmp::min; +use std::fmt; use std::iter::ExactSizeIterator; /// Information per top-level register class. @@ -225,6 +226,16 @@ impl Pressure { } } +impl fmt::Display for Pressure { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "Pressure[")?; + for rc in &self.toprc { + write!(f, " {}+{}/{}", rc.base_count, rc.transient_count, rc.limit)?; + } + write!(f, " ]") + } +} + #[cfg(test)] mod tests { use isa::{TargetIsa, RegClass}; diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 533dd45595..1342625380 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -10,6 +10,7 @@ //! pressure limits to be exceeded. use dominator_tree::DominatorTree; +use entity_map::EntityMap; use ir::{Ebb, Inst, Value, Function, DataFlowGraph}; use ir::layout::{Cursor, CursorPosition}; use ir::{InstBuilder, ArgumentLoc}; @@ -29,6 +30,7 @@ pub struct Reload { /// Context data structure that gets instantiated once per pass. struct Context<'a> { + isa: &'a TargetIsa, // Cached ISA information. // We save it here to avoid frequent virtual function calls on the `TargetIsa` trait object. encinfo: EncInfo, @@ -60,6 +62,7 @@ impl Reload { topo: &mut TopoOrder, tracker: &mut LiveValueTracker) { let mut ctx = Context { + isa, encinfo: isa.encoding_info(), domtree, liveness, @@ -112,7 +115,13 @@ impl<'a> Context<'a> { while let Some(inst) = pos.current_inst() { let encoding = func.encodings[inst]; if encoding.is_legal() { - self.visit_inst(ebb, inst, encoding, &mut pos, &mut func.dfg, tracker); + self.visit_inst(ebb, + inst, + encoding, + &mut pos, + &mut func.dfg, + &mut func.encodings, + tracker); tracker.drop_dead(inst); } else { pos.next_inst(); @@ -121,7 +130,7 @@ impl<'a> Context<'a> { } /// Process the EBB parameters. Return the next instruction in the EBB to be processed - fn visit_ebb_header(&self, + fn visit_ebb_header(&mut self, ebb: Ebb, func: &mut Function, tracker: &mut LiveValueTracker) @@ -139,7 +148,7 @@ impl<'a> Context<'a> { /// Visit the arguments to the entry block. /// These values have ABI constraints from the function signature. - fn visit_entry_args(&self, + fn visit_entry_args(&mut self, ebb: Ebb, func: &mut Function, args: &[LiveValue]) @@ -157,7 +166,15 @@ impl<'a> Context<'a> { // with a temporary register value that is immediately spilled. let reg = func.dfg.replace_ebb_arg(arg.value, abi.value_type); func.dfg.ins(&mut pos).with_result(arg.value).spill(reg); - // TODO: Update live ranges. + let spill = func.dfg.value_def(arg.value).unwrap_inst(); + *func.encodings.ensure(spill) = self.isa + .encode(&func.dfg, &func.dfg[spill], abi.value_type) + .expect("Can't encode spill"); + // Update live ranges. + self.liveness.move_def_locally(arg.value, spill); + let affinity = Affinity::abi(abi, self.isa); + self.liveness.create_dead(reg, ebb, affinity); + self.liveness.extend_locally(reg, ebb, spill, pos.layout); } } ArgumentLoc::Stack(_) => { @@ -184,6 +201,7 @@ impl<'a> Context<'a> { encoding: Encoding, pos: &mut Cursor, dfg: &mut DataFlowGraph, + encodings: &mut EntityMap, tracker: &mut LiveValueTracker) { // Get the operand constraints for `inst` that we are trying to satisfy. let constraints = self.encinfo @@ -213,6 +231,11 @@ impl<'a> Context<'a> { } let reg = dfg.ins(pos).fill(cand.value); + let fill = dfg.value_def(reg).unwrap_inst(); + *encodings.ensure(fill) = self.isa + .encode(dfg, &dfg[fill], dfg.value_type(reg)) + .expect("Can't encode fill"); + self.reloads .insert(ReloadedValue { stack: cand.value, diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs new file mode 100644 index 0000000000..8938944952 --- /dev/null +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -0,0 +1,259 @@ +//! Spilling pass. +//! +//! The spilling pass is the first to run after the liveness analysis. Its primary function is to +//! ensure that the register pressure never exceeds the number of available registers by moving +//! some SSA values to spill slots on the stack. This is encoded in the affinity of the value's +//! live range. + +use dominator_tree::DominatorTree; +use ir::{DataFlowGraph, Layout, Cursor}; +use ir::{Function, Ebb, Inst, Value}; +use isa::{TargetIsa, RegInfo, EncInfo, RecipeConstraints, ConstraintKind}; +use isa::registers::RegClassMask; +use regalloc::affinity::Affinity; +use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; +use regalloc::liveness::Liveness; +use regalloc::pressure::Pressure; +use topo_order::TopoOrder; + +/// Persistent data structures for the spilling pass. +pub struct Spilling { + spills: Vec, +} + +/// Context data structure that gets instantiated once per pass. +struct Context<'a> { + // Cached ISA information. + reginfo: RegInfo, + encinfo: EncInfo, + + // References to contextual data structures we need. + domtree: &'a DominatorTree, + liveness: &'a mut Liveness, + topo: &'a mut TopoOrder, + + // Current register pressure. + pressure: Pressure, + + // Values spilled for the current instruction. These values have already been removed from the + // pressure tracker, but they are still present in the live value tracker and their affinity + // hasn't been changed yet. + spills: &'a mut Vec, +} + +impl Spilling { + /// Create a new spilling data structure. + pub fn new() -> Spilling { + Spilling { spills: Vec::new() } + } + + /// Run the spilling algorithm over `func`. + pub fn run(&mut self, + isa: &TargetIsa, + func: &mut Function, + domtree: &DominatorTree, + liveness: &mut Liveness, + topo: &mut TopoOrder, + tracker: &mut LiveValueTracker) { + dbg!("Spilling for:\n{}", func.display(isa)); + let reginfo = isa.register_info(); + let usable_regs = isa.allocatable_registers(func); + let mut ctx = Context { + reginfo: isa.register_info(), + encinfo: isa.encoding_info(), + domtree, + liveness, + topo, + pressure: Pressure::new(®info, &usable_regs), + spills: &mut self.spills, + }; + ctx.run(func, tracker) + } +} + +impl<'a> Context<'a> { + fn run(&mut self, func: &mut Function, tracker: &mut LiveValueTracker) { + self.topo.reset(func.layout.ebbs()); + while let Some(ebb) = self.topo.next(&func.layout, self.domtree) { + self.visit_ebb(ebb, func, tracker); + } + } + + fn visit_ebb(&mut self, ebb: Ebb, func: &mut Function, tracker: &mut LiveValueTracker) { + dbg!("Spilling {}:", ebb); + self.visit_ebb_header(ebb, func, tracker); + tracker.drop_dead_args(); + + let mut pos = Cursor::new(&mut func.layout); + pos.goto_top(ebb); + while let Some(inst) = pos.next_inst() { + if let Some(constraints) = self.encinfo.operand_constraints(func.encodings[inst]) { + self.visit_inst(inst, constraints, &mut pos, &mut func.dfg, tracker); + tracker.drop_dead(inst); + self.process_spills(tracker); + } + } + } + + // Take all live registers in `regs` from the pressure set. + // This doesn't cause any spilling, it is assumed there are enough registers. + fn take_live_regs(&mut self, regs: &[LiveValue]) { + for lv in regs { + if !lv.is_dead { + if let Affinity::Reg(rci) = lv.affinity { + let rc = self.reginfo.rc(rci); + self.pressure.take(rc); + } + } + } + } + + // Free all registers in `kills` from the pressure set. + fn free_regs(&mut self, kills: &[LiveValue]) { + for lv in kills { + if let Affinity::Reg(rci) = lv.affinity { + let rc = self.reginfo.rc(rci); + self.pressure.free(rc); + } + } + } + + fn visit_ebb_header(&mut self, ebb: Ebb, func: &mut Function, tracker: &mut LiveValueTracker) { + let (liveins, args) = + tracker.ebb_top(ebb, &func.dfg, self.liveness, &func.layout, self.domtree); + + // Count the live-in registers. These should already fit in registers; they did at the + // dominator. + self.pressure.reset(); + self.take_live_regs(liveins); + + // TODO: Process and count EBB arguments. Some may need spilling. + self.take_live_regs(args); + } + + fn visit_inst(&mut self, + inst: Inst, + constraints: &RecipeConstraints, + pos: &mut Cursor, + dfg: &DataFlowGraph, + tracker: &mut LiveValueTracker) { + dbg!("Inst {}, {}", dfg.display_inst(inst), self.pressure); + // TODO: Repair constraint violations by copying input values. + // + // - Tied use of value that is not killed. + // - Inconsistent uses of the same value. + // + // Each inserted copy may increase register pressure. Fix by spilling something not used by + // the instruction. + // + // Count pressure for register uses of spilled values too. + // + // Finally, reset pressure state to level from before the input adjustments, minus spills. + // + // Spills should be removed from tracker. Otherwise they could be double-counted by + // free_regs below. + + // Update the live value tracker with this instruction. + let (throughs, kills, defs) = tracker.process_inst(inst, dfg, self.liveness); + + + // Remove kills from the pressure tracker. + self.free_regs(kills); + + // Make sure we have enough registers for the register defs. + // Dead defs are included here. They need a register too. + // No need to process call return values, they are in fixed registers. + for op in constraints.outs { + if op.kind != ConstraintKind::Stack { + // Add register def to pressure, spill if needed. + while let Err(mask) = self.pressure.take_transient(op.regclass) { + dbg!("Need {} reg from {} throughs", op.regclass, throughs.len()); + self.spill_from(mask, throughs, dfg, &pos.layout); + } + } + } + self.pressure.reset_transient(); + + // Restore pressure state, compute pressure with affinities from `defs`. + // Exclude dead defs. Includes call return values. + // This won't cause spilling. + self.take_live_regs(defs); + } + + // Spill a candidate from `candidates` whose top-level register class is in `mask`. + fn spill_from<'ii, II>(&mut self, + mask: RegClassMask, + candidates: II, + dfg: &DataFlowGraph, + layout: &Layout) + where II: IntoIterator + { + // Find the best viable spill candidate. + // + // The very simple strategy implemented here is to spill the value with the earliest def in + // the reverse post-order. This strategy depends on a good reload pass to generate good + // code. + // + // We know that all candidate defs dominate the current instruction, so one of them will + // dominate the others. That is the earliest def. + let best = candidates + .into_iter() + .filter_map(|lv| { + // Viable candidates are registers in one of the `mask` classes, and not already in + // the spill set. + if let Affinity::Reg(rci) = lv.affinity { + let rc = self.reginfo.rc(rci); + if (mask & (1 << rc.toprc)) != 0 && !self.spills.contains(&lv.value) { + // Here, `lv` is a viable spill candidate. + return Some(lv.value); + } + } + None + }) + .min_by(|&a, &b| { + // Find the minimum candidate according to the RPO of their defs. + self.domtree + .rpo_cmp(dfg.value_def(a), dfg.value_def(b), layout) + }); + + if let Some(value) = best { + // Found a spill candidate. + self.spill_reg(value); + } else { + panic!("Ran out of registers for mask={}", mask); + } + } + + /// Spill `value` immediately by + /// + /// 1. Changing its affinity to `Stack` which marks the spill. + /// 2. Removing the value from the pressure tracker. + /// 3. Adding the value to `self.spills` for later reference by `process_spills`. + /// + /// Note that this does not update the cached affinity in the live value tracker. Call + /// `process_spills` to do that. + fn spill_reg(&mut self, value: Value) { + if let Affinity::Reg(rci) = self.liveness.spill(value) { + let rc = self.reginfo.rc(rci); + self.pressure.free(rc); + self.spills.push(value); + dbg!("Spilled {}:{} -> {}", value, rc, self.pressure); + } else { + panic!("Cannot spill {} that was already on the stack", value); + } + } + + /// Process any pending spills in the `self.spills` vector. + /// + /// It is assumed that spills are removed from the pressure tracker immediately, see + /// `spill_from` above. + /// + /// We also need to update the live range affinity and remove spilled values from the live + /// value tracker. + fn process_spills(&mut self, tracker: &mut LiveValueTracker) { + if !self.spills.is_empty() { + tracker.process_spills(|v| self.spills.contains(&v)); + self.spills.clear() + } + } +} From 396b998dade5343e9def53165b06e7ab140545b2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 13 Jun 2017 15:13:36 -0700 Subject: [PATCH 0794/3084] Handle ABI arguments correctly in the reload pass. Values passed as arguments to calls and return instructions may also be reload candidates. --- cranelift/filetests/regalloc/spill.cton | 15 ++++- lib/cretonne/src/ir/valueloc.rs | 8 +++ lib/cretonne/src/regalloc/coloring.rs | 1 + lib/cretonne/src/regalloc/reload.rs | 82 ++++++++++++++++++++----- 4 files changed, 86 insertions(+), 20 deletions(-) diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index a6565b987e..d3da74ddf0 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -14,10 +14,15 @@ test regalloc isa riscv enable_e ; In straight-line code, the first value defined is spilled. -; That is the argument. +; That is in order: +; 1. The argument v1. +; 2. The link register. function %pyramid(i32) -> i32 { ebb0(v1: i32): -; check: $v1 = spill $(rv1=$V) +; check: $ebb0($(rv1=$V): i32, $(rlink=$V): i32) + ; check: $v1 = spill $rv1 + ; nextln: $(link=$V) = spill $rlink + ; not: spill v2 = iadd_imm v1, 12 v3 = iadd_imm v2, 12 v4 = iadd_imm v3, 12 @@ -29,7 +34,9 @@ ebb0(v1: i32): v10 = iadd_imm v9, 12 v11 = iadd_imm v10, 12 v12 = iadd_imm v11, 12 - v31 = iadd v11, v12 + v13 = iadd_imm v12, 12 + v32 = iadd v12, v13 + v31 = iadd v32, v11 v30 = iadd v31, v10 v29 = iadd v30, v9 v28 = iadd v29, v8 @@ -40,5 +47,7 @@ ebb0(v1: i32): v23 = iadd v24, v3 v22 = iadd v23, v2 v21 = iadd v22, v1 + ; check: $(rlink2=$V) = fill $link return v21 + ; check: return $v21, $rlink2 } diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/cretonne/src/ir/valueloc.rs index 2d6fc644b1..b664fe70e5 100644 --- a/lib/cretonne/src/ir/valueloc.rs +++ b/lib/cretonne/src/ir/valueloc.rs @@ -104,6 +104,14 @@ impl ArgumentLoc { } } + /// Is this a register location? + pub fn is_reg(&self) -> bool { + match self { + &ArgumentLoc::Reg(_) => true, + _ => false, + } + } + /// Return an object that can display this argument location, using the register info from the /// target ISA. pub fn display<'a, R: Into>>(self, regs: R) -> DisplayArgumentLoc<'a> { diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 33013aa309..8fd602fe39 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -101,6 +101,7 @@ impl Coloring { liveness: &mut Liveness, topo: &mut TopoOrder, tracker: &mut LiveValueTracker) { + dbg!("Coloring for:\n{}", func.display(isa)); let mut ctx = Context { reginfo: isa.register_info(), encinfo: isa.encoding_info(), diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 1342625380..9786f45b67 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -11,11 +11,11 @@ use dominator_tree::DominatorTree; use entity_map::EntityMap; -use ir::{Ebb, Inst, Value, Function, DataFlowGraph}; +use ir::{Ebb, Inst, Value, Function, Signature, DataFlowGraph}; use ir::layout::{Cursor, CursorPosition}; -use ir::{InstBuilder, ArgumentLoc}; +use ir::{InstBuilder, ArgumentType, ArgumentLoc}; use isa::RegClass; -use isa::{TargetIsa, Encoding, EncInfo, ConstraintKind}; +use isa::{TargetIsa, Encoding, EncInfo, RecipeConstraints, ConstraintKind}; use regalloc::affinity::Affinity; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; @@ -61,6 +61,7 @@ impl Reload { liveness: &mut Liveness, topo: &mut TopoOrder, tracker: &mut LiveValueTracker) { + dbg!("Reload for:\n{}", func.display(isa)); let mut ctx = Context { isa, encinfo: isa.encoding_info(), @@ -121,6 +122,7 @@ impl<'a> Context<'a> { &mut pos, &mut func.dfg, &mut func.encodings, + &func.signature, tracker); tracker.drop_dead(inst); } else { @@ -202,27 +204,16 @@ impl<'a> Context<'a> { pos: &mut Cursor, dfg: &mut DataFlowGraph, encodings: &mut EntityMap, + func_signature: &Signature, tracker: &mut LiveValueTracker) { // Get the operand constraints for `inst` that we are trying to satisfy. let constraints = self.encinfo .operand_constraints(encoding) .expect("Missing instruction encoding"); - assert!(self.candidates.is_empty()); - // Identify reload candidates. - for (op, &arg) in constraints.ins.iter().zip(dfg.inst_args(inst)) { - if op.kind != ConstraintKind::Stack { - let lv = self.liveness.get(arg).expect("Missing live range for arg"); - if lv.affinity.is_stack() { - self.candidates - .push(ReloadCandidate { - value: arg, - regclass: op.regclass, - }) - } - } - } + assert!(self.candidates.is_empty()); + self.find_candidates(inst, constraints, func_signature, dfg); // Insert fill instructions before `inst`. while let Some(cand) = self.candidates.pop() { @@ -289,4 +280,61 @@ impl<'a> Context<'a> { } } } + + // Find reload candidates for `inst` and add them to `self.condidates`. + // + // These are uses of spilled values where the operand constraint requires a register. + fn find_candidates(&mut self, + inst: Inst, + constraints: &RecipeConstraints, + func_signature: &Signature, + dfg: &DataFlowGraph) { + let args = dfg.inst_args(inst); + + for (op, &arg) in constraints.ins.iter().zip(args) { + if op.kind != ConstraintKind::Stack { + let lv = self.liveness.get(arg).expect("Missing live range for arg"); + if lv.affinity.is_stack() { + self.candidates + .push(ReloadCandidate { + value: arg, + regclass: op.regclass, + }) + } + } + } + + // If we only have the fixed arguments, we're done now. + if args.len() == constraints.ins.len() { + return; + } + let var_args = &args[constraints.ins.len()..]; + + // Handle ABI arguments. + if let Some(sig) = dfg.call_signature(inst) { + self.handle_abi_args(&dfg.signatures[sig].argument_types, var_args); + } else if dfg[inst].opcode().is_return() { + self.handle_abi_args(&func_signature.return_types, var_args); + } + } + + /// Find reload candidates in the instruction's ABI variable arguments. This handles both + /// return values and call arguments. + fn handle_abi_args(&mut self, abi_types: &[ArgumentType], var_args: &[Value]) { + assert_eq!(abi_types.len(), var_args.len()); + for (abi, &arg) in abi_types.iter().zip(var_args) { + if abi.location.is_reg() { + let lv = self.liveness + .get(arg) + .expect("Missing live range for ABI arg"); + if lv.affinity.is_stack() { + self.candidates + .push(ReloadCandidate { + value: arg, + regclass: self.isa.regclass_for_abi_type(abi.value_type), + }); + } + } + } + } } From 0974b4a6e24b172a9614579e6cb49237edecabe8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 13 Jun 2017 15:43:14 -0700 Subject: [PATCH 0795/3084] Extract spill insertion into a reload::insert_spill function. Make sure that spill instructions are generated in the same way everywhere, including adding encoding and updating live ranges. --- cranelift/filetests/regalloc/spill.cton | 13 +++++- lib/cretonne/src/regalloc/reload.rs | 54 ++++++++++++++++++------- 2 files changed, 51 insertions(+), 16 deletions(-) diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index d3da74ddf0..a43ba4cf8f 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -17,6 +17,7 @@ isa riscv enable_e ; That is in order: ; 1. The argument v1. ; 2. The link register. +; 3. The first computed value, v2 function %pyramid(i32) -> i32 { ebb0(v1: i32): ; check: $ebb0($(rv1=$V): i32, $(rlink=$V): i32) @@ -24,6 +25,9 @@ ebb0(v1: i32): ; nextln: $(link=$V) = spill $rlink ; not: spill v2 = iadd_imm v1, 12 + ; check: $(r1v2=$V) = iadd_imm + ; nextln: $v2 = spill $r1v2 + ; not: spill v3 = iadd_imm v2, 12 v4 = iadd_imm v3, 12 v5 = iadd_imm v4, 12 @@ -35,7 +39,10 @@ ebb0(v1: i32): v11 = iadd_imm v10, 12 v12 = iadd_imm v11, 12 v13 = iadd_imm v12, 12 - v32 = iadd v12, v13 + v14 = iadd_imm v13, 12 + v33 = iadd v13, v14 + ; check: iadd $v13 + v32 = iadd v33, v12 v31 = iadd v32, v11 v30 = iadd v31, v10 v29 = iadd v30, v9 @@ -46,7 +53,11 @@ ebb0(v1: i32): v24 = iadd v25, v4 v23 = iadd v24, v3 v22 = iadd v23, v2 + ; check: $(r2v2=$V) = fill $v2 + ; check: $v22 = iadd $v23, $r2v2 v21 = iadd v22, v1 + ; check: $(r2v1=$V) = fill $v1 + ; check: $v21 = iadd $v22, $r2v1 ; check: $(rlink2=$V) = fill $link return v21 ; check: return $v21, $rlink2 diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 9786f45b67..dd623de2b5 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -13,7 +13,7 @@ use dominator_tree::DominatorTree; use entity_map::EntityMap; use ir::{Ebb, Inst, Value, Function, Signature, DataFlowGraph}; use ir::layout::{Cursor, CursorPosition}; -use ir::{InstBuilder, ArgumentType, ArgumentLoc}; +use ir::{InstBuilder, Opcode, ArgumentType, ArgumentLoc}; use isa::RegClass; use isa::{TargetIsa, Encoding, EncInfo, RecipeConstraints, ConstraintKind}; use regalloc::affinity::Affinity; @@ -167,16 +167,14 @@ impl<'a> Context<'a> { // An incoming register parameter was spilled. Replace the parameter value // with a temporary register value that is immediately spilled. let reg = func.dfg.replace_ebb_arg(arg.value, abi.value_type); - func.dfg.ins(&mut pos).with_result(arg.value).spill(reg); - let spill = func.dfg.value_def(arg.value).unwrap_inst(); - *func.encodings.ensure(spill) = self.isa - .encode(&func.dfg, &func.dfg[spill], abi.value_type) - .expect("Can't encode spill"); - // Update live ranges. - self.liveness.move_def_locally(arg.value, spill); let affinity = Affinity::abi(abi, self.isa); self.liveness.create_dead(reg, ebb, affinity); - self.liveness.extend_locally(reg, ebb, spill, pos.layout); + self.insert_spill(ebb, + arg.value, + reg, + &mut pos, + &mut func.encodings, + &mut func.dfg); } } ArgumentLoc::Stack(_) => { @@ -270,13 +268,8 @@ impl<'a> Context<'a> { if lv.affinity.is_stack() && op.kind != ConstraintKind::Stack { let value_type = dfg.value_type(lv.value); let reg = dfg.replace_result(lv.value, value_type); - dfg.ins(pos).with_result(lv.value).spill(reg); - let spill = dfg.value_def(lv.value).unwrap_inst(); - - // Create a live range for reg. self.liveness.create_dead(reg, inst, Affinity::new(op)); - self.liveness.extend_locally(reg, ebb, spill, &pos.layout); - self.liveness.move_def_locally(lv.value, spill); + self.insert_spill(ebb, lv.value, reg, pos, encodings, dfg); } } } @@ -337,4 +330,35 @@ impl<'a> Context<'a> { } } } + + /// Insert a spill at `pos` and update data structures. + /// + /// - Insert `stack = spill reg` at `pos`, and assign an encoding. + /// - Move the `stack` live range starting point to the new instruction. + /// - Extend the `reg` live range to reach the new instruction. + fn insert_spill(&mut self, + ebb: Ebb, + stack: Value, + reg: Value, + pos: &mut Cursor, + encodings: &mut EntityMap, + dfg: &mut DataFlowGraph) { + let ty = dfg.value_type(reg); + + // Insert spill instruction. Use the low-level `Unary` constructor because it returns an + // instruction reference directly rather than a result value (which we know is equal to + // `stack`). + let (inst, _) = dfg.ins(pos) + .with_result(stack) + .Unary(Opcode::Spill, ty, reg); + + // Give it an encoding. + *encodings.ensure(inst) = self.isa + .encode(dfg, &dfg[inst], ty) + .expect("Can't encode spill"); + + // Update live ranges. + self.liveness.move_def_locally(stack, inst); + self.liveness.extend_locally(reg, ebb, inst, pos.layout); + } } From 454910407f99adb1e2fbd554e908ad4e6b8062c8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 14 Jun 2017 08:55:01 -0700 Subject: [PATCH 0796/3084] Spill values live across calls. Calls clobber many registers, so spill everything that is live across a call for now. In the future, we may add support for callee-saved registers. --- cranelift/filetests/regalloc/spill.cton | 13 +++++++++++++ lib/cretonne/src/regalloc/affinity.rs | 8 ++++++++ lib/cretonne/src/regalloc/spilling.rs | 13 ++++++++++++- 3 files changed, 33 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index a43ba4cf8f..eaea391504 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -62,3 +62,16 @@ ebb0(v1: i32): return v21 ; check: return $v21, $rlink2 } + +; All values live across a call must be spilled +function %across_call(i32) { + fn0 = function %foo(i32) +ebb0(v1: i32): + ; check: $v1 = spill + call fn0(v1) + ; check: call $fn0 + call fn0(v1) + ; check: fill $v1 + ; check: call $fn0 + return +} diff --git a/lib/cretonne/src/regalloc/affinity.rs b/lib/cretonne/src/regalloc/affinity.rs index e7ee0515ad..a7c95df99f 100644 --- a/lib/cretonne/src/regalloc/affinity.rs +++ b/lib/cretonne/src/regalloc/affinity.rs @@ -64,6 +64,14 @@ impl Affinity { } } + /// Is this the `Reg` affinity? + pub fn is_reg(self) -> bool { + match self { + Affinity::Reg(_) => true, + _ => false, + } + } + /// Is this the `Stack` affinity? pub fn is_stack(self) -> bool { match self { diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 8938944952..da0453f549 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -152,14 +152,25 @@ impl<'a> Context<'a> { // // Spills should be removed from tracker. Otherwise they could be double-counted by // free_regs below. + let call_sig = dfg.call_signature(inst); // Update the live value tracker with this instruction. let (throughs, kills, defs) = tracker.process_inst(inst, dfg, self.liveness); - // Remove kills from the pressure tracker. self.free_regs(kills); + // If inst is a call, spill all register values that are live across the call. + // This means that we don't currently take advantage of callee-saved registers. + // TODO: Be more sophisticated. + if call_sig.is_some() { + for lv in throughs { + if lv.affinity.is_reg() && !self.spills.contains(&lv.value) { + self.spill_reg(lv.value); + } + } + } + // Make sure we have enough registers for the register defs. // Dead defs are included here. They need a register too. // No need to process call return values, they are in fixed registers. From 897c363714f2e0ae6a34d0f432f91f43f382bf7e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 14 Jun 2017 10:35:01 -0700 Subject: [PATCH 0797/3084] Update docopt dependency to 0.8.0. This breaks our depending on two different versions of the regex library. This updated docopt uses serde instead of rustc_serialize. --- cranelift/Cargo.toml | 7 ++++--- cranelift/src/cton-util.rs | 10 ++++++---- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index d0da28f918..06eed706e9 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -16,8 +16,9 @@ path = "src/cton-util.rs" cretonne = { path = "lib/cretonne" } cretonne-reader = { path = "lib/reader" } filecheck = { path = "lib/filecheck" } -docopt = "0.6.86" -rustc-serialize = "0.3.19" -num_cpus = "1.1.0" +docopt = "0.8.0" +serde = "1.0.8" +serde_derive = "1.0.8" +num_cpus = "1.5.1" [workspace] diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 70fcbc3c84..28b4b90de0 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -2,7 +2,9 @@ extern crate cretonne; extern crate cton_reader; extern crate docopt; -extern crate rustc_serialize; +extern crate serde; +#[macro_use] +extern crate serde_derive; extern crate filecheck; extern crate num_cpus; @@ -34,7 +36,7 @@ Options: "; -#[derive(RustcDecodable, Debug)] +#[derive(Deserialize, Debug)] struct Args { cmd_test: bool, cmd_cat: bool, @@ -49,12 +51,12 @@ pub type CommandResult = Result<(), String>; /// Parse the command line arguments and run the requested command. fn cton_util() -> CommandResult { - // Parse comand line arguments. + // Parse command line arguments. let args: Args = Docopt::new(USAGE) .and_then(|d| { d.help(true) .version(Some(format!("Cretonne {}", VERSION))) - .decode() + .deserialize() }) .unwrap_or_else(|e| e.exit()); From 96f122821109ce2015e2115c37a5cc97c36ce8a3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 14 Jun 2017 12:05:47 -0700 Subject: [PATCH 0798/3084] Always call reassign_in for register ABI arguments. Even if an argument is already in the correct register, make sure that we detect conflicts by registering the no-op move. This also means that the ABI argument register won't be turned into a variable for the solver. --- lib/cretonne/src/regalloc/coloring.rs | 22 ++++++++++------------ lib/cretonne/src/regalloc/solver.rs | 16 +++++++++------- 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 8fd602fe39..53546de07f 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -430,18 +430,16 @@ impl<'a> Context<'a> { locations: &EntityMap) { for (abi, &value) in abi_types.iter().zip(dfg.inst_variable_args(inst)) { if let ArgumentLoc::Reg(reg) = abi.location { - let cur_reg = self.divert.reg(value, locations); - if reg != cur_reg { - if let Affinity::Reg(rci) = - self.liveness - .get(value) - .expect("ABI register must have live range") - .affinity { - let rc = self.reginfo.rc(rci); - self.solver.reassign_in(value, rc, cur_reg, reg); - } else { - panic!("ABI argument {} should be in a register", value); - } + if let Affinity::Reg(rci) = + self.liveness + .get(value) + .expect("ABI register must have live range") + .affinity { + let rc = self.reginfo.rc(rci); + let cur_reg = self.divert.reg(value, locations); + self.solver.reassign_in(value, rc, cur_reg, reg); + } else { + panic!("ABI argument {} should be in a register", value); } } } diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 89aee41317..d94cf3e72d 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -364,13 +364,15 @@ impl Solver { } self.regs_in.free(rc, from); self.regs_out.take(rc, to); - self.assignments - .insert(Assignment { - value, - rc, - from, - to, - }); + if from != to { + self.assignments + .insert(Assignment { + value, + rc, + from, + to, + }); + } } /// Add a variable representing an input side value with an existing register assignment. From 3e1e2b6e5e24465895cbede623fe74c20d196595 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 14 Jun 2017 15:36:25 -0700 Subject: [PATCH 0799/3084] Add RISC-V encodings for copy instructions. --- cranelift/filetests/isa/riscv/binary32.cton | 4 ++++ lib/cretonne/meta/isa/riscv/encodings.py | 7 ++++++- lib/cretonne/meta/isa/riscv/recipes.py | 3 +++ lib/cretonne/src/isa/riscv/binemit.rs | 12 ++++++++++++ 4 files changed, 25 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 052342f233..923b6eb537 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -77,6 +77,10 @@ ebb0(v9999: i32): [-,%x7] v140 = iconst.i32 0x12345000 ; bin: 123453b7 [-,%x16] v141 = iconst.i32 0xffffffff_fedcb000 ; bin: fedcb837 + ; Copies alias to iadd_imm. + [-,%x7] v150 = copy v1 ; bin: 00050393 + [-,%x16] v151 = copy v2 ; bin: 000a8813 + ; Control Transfer Instructions ; jal %x1, fn0 diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index 2b59964007..5d3c121a14 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -7,7 +7,7 @@ from base.immediates import intcc from .defs import RV32, RV64 from .recipes import OPIMM, OPIMM32, OP, OP32, LUI, BRANCH, JALR, JAL from .recipes import LOAD, STORE -from .recipes import R, Rshamt, Ricmp, I, Iicmp, Iret +from .recipes import R, Rshamt, Ricmp, I, Iicmp, Iret, Icopy from .recipes import U, UJ, UJcall, SB, SBzero, GPsp, GPfi from .settings import use_m from cdsl.ast import Var @@ -123,3 +123,8 @@ RV64.enc(base.spill.i64, GPsp, STORE(0b011)) RV32.enc(base.fill.i32, GPfi, LOAD(0b010)) RV64.enc(base.fill.i32, GPfi, LOAD(0b010)) RV64.enc(base.fill.i64, GPfi, LOAD(0b011)) + +# Register copies. +RV32.enc(base.copy.i32, Icopy, OPIMM(0b000)) +RV64.enc(base.copy.i64, Icopy, OPIMM(0b000)) +RV64.enc(base.copy.i32, Icopy, OPIMM32(0b000)) diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 1a7e98d0fe..8a6be46c5f 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -113,6 +113,9 @@ Iicmp = EncRecipe( # The variable return values are not encoded. Iret = EncRecipe('Iret', MultiAry, size=4, ins=(), outs=()) +# Copy of a GPR is implemented as addi x, 0. +Icopy = EncRecipe('Icopy', Unary, size=4, ins=GPR, outs=GPR) + # U-type instructions have a 20-bit immediate that targets bits 12-31. U = EncRecipe( 'U', UnaryImm, size=4, ins=(), outs=GPR, diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index e025b3f9d4..0beaf51a8d 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -182,6 +182,18 @@ fn recipe_iret(func: &Function, inst: Inst, sink: &mut CS sink); } +fn recipe_icopy(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Unary { arg, .. } = func.dfg[inst] { + put_i(func.encodings[inst].bits(), + func.locations[arg].unwrap_reg(), + 0, + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); + } else { + panic!("Expected Unary format: {:?}", func.dfg[inst]); + } +} + /// U-type instructions. /// /// 31 11 6 From 4503306f0e81e6afd18b7040263cc9829a40c509 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 14 Jun 2017 16:14:16 -0700 Subject: [PATCH 0800/3084] Add RISC-V encodings for call_indirect. --- cranelift/filetests/isa/riscv/binary32.cton | 5 +++++ lib/cretonne/meta/isa/riscv/encodings.py | 4 +++- lib/cretonne/meta/isa/riscv/recipes.py | 6 +++++- lib/cretonne/src/isa/riscv/binemit.rs | 9 +++++++++ 4 files changed, 22 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 923b6eb537..209d9922c0 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -4,6 +4,7 @@ isa riscv function %RV32I(i32 link [%x1]) -> i32 link [%x1] { fn0 = function %foo() + sig0 = signature() ebb0(v9999: i32): [-,%x10] v1 = iconst.i32 1 @@ -86,6 +87,10 @@ ebb0(v9999: i32): ; jal %x1, fn0 call fn0() ; bin: Call(fn0) 000000ef + ; jalr %x1, %x10 + call_indirect sig0, v1() ; bin: 000500e7 + call_indirect sig0, v2() ; bin: 000a80e7 + brz v1, ebb3 brnz v1, ebb1 diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index 5d3c121a14..374cc951af 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -7,7 +7,7 @@ from base.immediates import intcc from .defs import RV32, RV64 from .recipes import OPIMM, OPIMM32, OP, OP32, LUI, BRANCH, JALR, JAL from .recipes import LOAD, STORE -from .recipes import R, Rshamt, Ricmp, I, Iicmp, Iret, Icopy +from .recipes import R, Rshamt, Ricmp, I, Iicmp, Iret, Icall, Icopy from .recipes import U, UJ, UJcall, SB, SBzero, GPsp, GPfi from .settings import use_m from cdsl.ast import Var @@ -115,6 +115,8 @@ for inst, f3 in [ # is added by legalize_signature(). RV32.enc(base.x_return, Iret, JALR()) RV64.enc(base.x_return, Iret, JALR()) +RV32.enc(base.call_indirect.i32, Icall, JALR()) +RV64.enc(base.call_indirect.i64, Icall, JALR()) # Spill and fill. RV32.enc(base.spill.i32, GPsp, STORE(0b010)) diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 8a6be46c5f..60a67fbc00 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -13,7 +13,8 @@ from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt from cdsl.registers import Stack from base.formats import Binary, BinaryImm, MultiAry, IntCompare, IntCompareImm -from base.formats import Unary, UnaryImm, BranchIcmp, Branch, Jump, Call +from base.formats import Unary, UnaryImm, BranchIcmp, Branch, Jump +from base.formats import Call, IndirectCall from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit @@ -113,6 +114,9 @@ Iicmp = EncRecipe( # The variable return values are not encoded. Iret = EncRecipe('Iret', MultiAry, size=4, ins=(), outs=()) +# I-type encoding for `jalr` as an indirect call. +Icall = EncRecipe('Icall', IndirectCall, size=4, ins=GPR, outs=()) + # Copy of a GPR is implemented as addi x, 0. Icopy = EncRecipe('Icopy', Unary, size=4, ins=GPR, outs=GPR) diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 0beaf51a8d..0c25f9ec85 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -182,6 +182,15 @@ fn recipe_iret(func: &Function, inst: Inst, sink: &mut CS sink); } +fn recipe_icall(func: &Function, inst: Inst, sink: &mut CS) { + // Indirect instructions are jalr with rd=%x1. + put_i(func.encodings[inst].bits(), + func.locations[func.dfg.inst_args(inst)[0]].unwrap_reg(), + 0, // no offset. + 1, // rd = %x1: link register. + sink); +} + fn recipe_icopy(func: &Function, inst: Inst, sink: &mut CS) { if let InstructionData::Unary { arg, .. } = func.dfg[inst] { put_i(func.encodings[inst].bits(), From db62f435f8827a101b3f8e341da2294d858179b3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 14 Jun 2017 13:56:06 -0700 Subject: [PATCH 0801/3084] Make register copies for incompatible operands. An instruction may have fixed operand constraints that make it impossibly to use a single register value to satisfy two at a time. Detect when the same value is used for multiple fixed register operands and insert copies during the spilling pass. --- cranelift/filetests/regalloc/spill.cton | 29 +++ lib/cretonne/src/ir/entities.rs | 2 +- lib/cretonne/src/regalloc/spilling.rs | 235 +++++++++++++++++++++--- 3 files changed, 235 insertions(+), 31 deletions(-) diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index eaea391504..f163c478b3 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -75,3 +75,32 @@ ebb0(v1: i32): ; check: call $fn0 return } + +; The same value used for two function arguments. +function %doubleuse(i32) { + fn0 = function %xx(i32, i32) +ebb0(v0: i32): + ; check: $(c=$V) = copy $v0 + call fn0(v0, v0) + ; check: call $fn0($v0, $c) + return +} + +; The same value used as indirect callee and argument. +function %doubleuse_icall1(i32) { + sig0 = signature(i32) +ebb0(v0: i32): + ; not:copy + call_indirect sig0, v0(v0) + return +} + +; The same value used as indirect callee and two arguments. +function %doubleuse_icall2(i32) { + sig0 = signature(i32, i32) +ebb0(v0: i32): + ; check: $(c=$V) = copy $v0 + call_indirect sig0, v0(v0, v0) + ; check: call_indirect $sig0, $v0($v0, $c) + return +} diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 813ec978c2..04b54e0488 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -74,7 +74,7 @@ impl Ebb { } /// An opaque reference to an SSA value. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] pub struct Value(u32); entity_impl!(Value, "v"); diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index da0453f549..23d4799dad 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -4,12 +4,23 @@ //! ensure that the register pressure never exceeds the number of available registers by moving //! some SSA values to spill slots on the stack. This is encoded in the affinity of the value's //! live range. +//! +//! Some instruction operand constraints may require additional registers to resolve. Since this +//! can cause spilling, the spilling pass is also responsible for resolving those constraints by +//! inserting copies. The extra constraints are: +//! +//! 1. A value used by a tied operand must be killed by the instruction. This is resolved by +//! inserting a copy to a temporary value when necessary. +//! 2. When the same value is used more than once by an instruction, the operand constraints must +//! be compatible. Otherwise, the value must be copied into a new register for some of the +//! operands. use dominator_tree::DominatorTree; -use ir::{DataFlowGraph, Layout, Cursor}; -use ir::{Function, Ebb, Inst, Value}; -use isa::{TargetIsa, RegInfo, EncInfo, RecipeConstraints, ConstraintKind}; -use isa::registers::RegClassMask; +use entity_map::EntityMap; +use ir::{DataFlowGraph, Layout, Cursor, InstBuilder}; +use ir::{Function, Ebb, Inst, Value, SigRef}; +use isa::registers::{RegClass, RegClassMask}; +use isa::{TargetIsa, RegInfo, EncInfo, Encoding, RecipeConstraints, ConstraintKind}; use regalloc::affinity::Affinity; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; @@ -19,14 +30,19 @@ use topo_order::TopoOrder; /// Persistent data structures for the spilling pass. pub struct Spilling { spills: Vec, + reg_uses: Vec, } /// Context data structure that gets instantiated once per pass. struct Context<'a> { + isa: &'a TargetIsa, // Cached ISA information. reginfo: RegInfo, encinfo: EncInfo, + // References to parts of the current function. + encodings: &'a mut EntityMap, + // References to contextual data structures we need. domtree: &'a DominatorTree, liveness: &'a mut Liveness, @@ -39,12 +55,18 @@ struct Context<'a> { // pressure tracker, but they are still present in the live value tracker and their affinity // hasn't been changed yet. spills: &'a mut Vec, + + // Uses of register values in the current instruction. + reg_uses: &'a mut Vec, } impl Spilling { /// Create a new spilling data structure. pub fn new() -> Spilling { - Spilling { spills: Vec::new() } + Spilling { + spills: Vec::new(), + reg_uses: Vec::new(), + } } /// Run the spilling algorithm over `func`. @@ -59,36 +81,46 @@ impl Spilling { let reginfo = isa.register_info(); let usable_regs = isa.allocatable_registers(func); let mut ctx = Context { + isa, reginfo: isa.register_info(), encinfo: isa.encoding_info(), + encodings: &mut func.encodings, domtree, liveness, topo, pressure: Pressure::new(®info, &usable_regs), spills: &mut self.spills, + reg_uses: &mut self.reg_uses, }; - ctx.run(func, tracker) + ctx.run(&mut func.layout, &mut func.dfg, tracker) } } impl<'a> Context<'a> { - fn run(&mut self, func: &mut Function, tracker: &mut LiveValueTracker) { - self.topo.reset(func.layout.ebbs()); - while let Some(ebb) = self.topo.next(&func.layout, self.domtree) { - self.visit_ebb(ebb, func, tracker); + fn run(&mut self, + layout: &mut Layout, + dfg: &mut DataFlowGraph, + tracker: &mut LiveValueTracker) { + self.topo.reset(layout.ebbs()); + while let Some(ebb) = self.topo.next(layout, self.domtree) { + self.visit_ebb(ebb, layout, dfg, tracker); } } - fn visit_ebb(&mut self, ebb: Ebb, func: &mut Function, tracker: &mut LiveValueTracker) { + fn visit_ebb(&mut self, + ebb: Ebb, + layout: &mut Layout, + dfg: &mut DataFlowGraph, + tracker: &mut LiveValueTracker) { dbg!("Spilling {}:", ebb); - self.visit_ebb_header(ebb, func, tracker); + self.visit_ebb_header(ebb, layout, dfg, tracker); tracker.drop_dead_args(); - let mut pos = Cursor::new(&mut func.layout); + let mut pos = Cursor::new(layout); pos.goto_top(ebb); while let Some(inst) = pos.next_inst() { - if let Some(constraints) = self.encinfo.operand_constraints(func.encodings[inst]) { - self.visit_inst(inst, constraints, &mut pos, &mut func.dfg, tracker); + if let Some(constraints) = self.encinfo.operand_constraints(self.encodings[inst]) { + self.visit_inst(inst, constraints, &mut pos, dfg, tracker); tracker.drop_dead(inst); self.process_spills(tracker); } @@ -118,9 +150,12 @@ impl<'a> Context<'a> { } } - fn visit_ebb_header(&mut self, ebb: Ebb, func: &mut Function, tracker: &mut LiveValueTracker) { - let (liveins, args) = - tracker.ebb_top(ebb, &func.dfg, self.liveness, &func.layout, self.domtree); + fn visit_ebb_header(&mut self, + ebb: Ebb, + layout: &mut Layout, + dfg: &mut DataFlowGraph, + tracker: &mut LiveValueTracker) { + let (liveins, args) = tracker.ebb_top(ebb, dfg, self.liveness, layout, self.domtree); // Count the live-in registers. These should already fit in registers; they did at the // dominator. @@ -135,24 +170,32 @@ impl<'a> Context<'a> { inst: Inst, constraints: &RecipeConstraints, pos: &mut Cursor, - dfg: &DataFlowGraph, + dfg: &mut DataFlowGraph, tracker: &mut LiveValueTracker) { dbg!("Inst {}, {}", dfg.display_inst(inst), self.pressure); // TODO: Repair constraint violations by copying input values. // // - Tied use of value that is not killed. - // - Inconsistent uses of the same value. - // - // Each inserted copy may increase register pressure. Fix by spilling something not used by - // the instruction. - // - // Count pressure for register uses of spilled values too. - // - // Finally, reset pressure state to level from before the input adjustments, minus spills. - // - // Spills should be removed from tracker. Otherwise they could be double-counted by - // free_regs below. + // - Count pressure for register uses of spilled values too. + + assert!(self.reg_uses.is_empty()); + + // If the instruction has any fixed register operands, we may need to resolve register + // constraints. + if constraints.fixed_ins { + self.collect_reg_uses(inst, constraints, dfg); + } + + // Calls usually have fixed register uses. let call_sig = dfg.call_signature(inst); + if let Some(sig) = call_sig { + self.collect_abi_reg_uses(inst, sig, dfg); + } + + if !self.reg_uses.is_empty() { + self.process_reg_uses(inst, pos, dfg, tracker); + } + // Update the live value tracker with this instruction. let (throughs, kills, defs) = tracker.process_inst(inst, dfg, self.liveness); @@ -190,6 +233,85 @@ impl<'a> Context<'a> { // This won't cause spilling. self.take_live_regs(defs); } + // Collect register uses from the fixed input constraints. + // + // We are assuming here that if a value is used both by a fixed register operand and a register + // class operand, they two are compatible. We are also assuming that two register class + // operands are always compatible. + fn collect_reg_uses(&mut self, + inst: Inst, + constraints: &RecipeConstraints, + dfg: &DataFlowGraph) { + let args = dfg.inst_args(inst); + for (idx, (op, &arg)) in constraints.ins.iter().zip(args).enumerate() { + match op.kind { + ConstraintKind::FixedReg(_) => { + self.reg_uses.push(RegUse::new(arg, idx)); + } + _ => {} + } + } + } + + // Collect register uses from the ABI input constraints. + fn collect_abi_reg_uses(&mut self, inst: Inst, sig: SigRef, dfg: &DataFlowGraph) { + let fixed_args = dfg[inst].opcode().constraints().fixed_value_arguments(); + let args = dfg.inst_variable_args(inst); + for (idx, (abi, &arg)) in + dfg.signatures[sig] + .argument_types + .iter() + .zip(args) + .enumerate() { + if abi.location.is_reg() { + self.reg_uses.push(RegUse::new(arg, fixed_args + idx)); + } + } + } + + // Process multiple register uses to resolve potential conflicts. + // + // Look for multiple uses of the same value in `self.reg_uses` and insert copies as necessary. + // Trigger spilling if any of the temporaries cause the register pressure to become too high. + // + // Leave `self.reg_uses` empty. + fn process_reg_uses(&mut self, + inst: Inst, + pos: &mut Cursor, + dfg: &mut DataFlowGraph, + tracker: &LiveValueTracker) { + // We're looking for multiple uses of the same value, so start by sorting by value. The + // secondary `opidx` key makes it possible to use an unstable sort once that is available + // outside nightly Rust. + self.reg_uses.sort_by_key(|u| (u.value, u.opidx)); + + // We are assuming that `reg_uses` has an entry per fixed register operand, and that any + // non-fixed register operands are compatible with one of the fixed uses of the value. + for i in 1..self.reg_uses.len() { + let ru = self.reg_uses[i]; + if self.reg_uses[i - 1].value != ru.value { + continue; + } + + // We have two fixed uses of the same value. Make a copy. + let (copy, rc) = self.insert_copy(ru.value, pos, dfg); + dfg.inst_args_mut(inst)[ru.opidx as usize] = copy; + + // Make sure the new copy doesn't blow the register pressure. + while let Err(mask) = self.pressure.take_transient(rc) { + dbg!("Copy of {} reg causes spill", rc); + // Spill a live register that is *not* used by the current instruction. + // Spilling a use wouldn't help. + let args = dfg.inst_args(inst); + self.spill_from(mask, + tracker.live().iter().filter(|lv| !args.contains(&lv.value)), + dfg, + &pos.layout); + } + } + self.pressure.reset_transient(); + self.reg_uses.clear() + } // Spill a candidate from `candidates` whose top-level register class is in `mask`. fn spill_from<'ii, II>(&mut self, @@ -267,4 +389,57 @@ impl<'a> Context<'a> { self.spills.clear() } } + + /// Insert a `copy value` before `pos` and give it a live range extending to `pos`. + /// + /// Returns the new local value created and its register class. + fn insert_copy(&mut self, + value: Value, + pos: &mut Cursor, + dfg: &mut DataFlowGraph) + -> (Value, RegClass) { + let copy = dfg.ins(pos).copy(value); + let inst = dfg.value_def(copy).unwrap_inst(); + let ty = dfg.value_type(copy); + + // Give it an encoding. + let encoding = self.isa + .encode(dfg, &dfg[inst], ty) + .expect("Can't encode copy"); + *self.encodings.ensure(inst) = encoding; + + // Update live ranges. + let rc = self.encinfo + .operand_constraints(encoding) + .expect("Bad copy encoding") + .outs + [0] + .regclass; + self.liveness + .create_dead(copy, inst, Affinity::Reg(rc.into())); + self.liveness + .extend_locally(copy, + pos.layout.pp_ebb(inst), + pos.current_inst().expect("must be at an instruction"), + pos.layout); + + (copy, rc) + } +} + +// Struct representing a register use of a value. +// Used to detect multiple uses of the same value with incompatible register constraints. +#[derive(Clone, Copy)] +struct RegUse { + value: Value, + opidx: u16, +} + +impl RegUse { + fn new(value: Value, idx: usize) -> RegUse { + RegUse { + value, + opidx: idx as u16, + } + } } From 7b979339965c6d4c898eabb354be845a4bba5f65 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Jun 2017 10:35:01 -0700 Subject: [PATCH 0802/3084] Track stack slot kinds. Add a StackSlotKind enumeration to help keep track of the different kinds of stack slots supported: - Incoming and outgoing function arguments on the stack. - Spill slots and locals. Change the text format syntax for declaring a stack slot to use a kind keyword rather than just 'stack_slot'. --- cranelift/docs/example.cton | 2 +- cranelift/docs/langref.rst | 4 +- cranelift/filetests/parser/tiny.cton | 12 +++-- lib/cretonne/src/ir/mod.rs | 2 +- lib/cretonne/src/ir/stackslot.rs | 79 ++++++++++++++++++++++++---- lib/cretonne/src/write.rs | 14 ++--- lib/reader/src/parser.rs | 29 ++++++---- lib/reader/src/sourcemap.rs | 2 +- misc/vim/syntax/cton.vim | 4 +- 9 files changed, 110 insertions(+), 38 deletions(-) diff --git a/cranelift/docs/example.cton b/cranelift/docs/example.cton index 8b9c43c6b9..493794da50 100644 --- a/cranelift/docs/example.cton +++ b/cranelift/docs/example.cton @@ -1,7 +1,7 @@ test verifier function %average(i32, i32) -> f32 { - ss1 = stack_slot 8 ; Stack slot for ``sum``. + ss1 = local 8 ; Stack slot for ``sum``. ebb1(v1: i32, v2: i32): v3 = f64const 0x0.0 diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 24b5961ec3..ac20591c36 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -465,9 +465,9 @@ frame. The stack frame is divided into fixed-size stack slots that are allocated in the :term:`function preamble`. Stack slots are not typed, they simply represent a contiguous sequence of bytes in the stack frame. -.. inst:: SS = stack_slot Bytes, Flags... +.. inst:: SS = local Bytes, Flags... - Allocate a stack slot in the preamble. + Allocate a stack slot for a local variable in the preamble. If no alignment is specified, Cretonne will pick an appropriate alignment for the stack slot based on its size and access patterns. diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 1414668a52..704477e6c2 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -96,8 +96,10 @@ ebb0(v90: i32, v91: f32): ; Stack slot references function %stack() { - ss10 = stack_slot 8 - ss2 = stack_slot 4 + ss10 = spill_slot 8 + ss2 = local 4 + ss3 = incoming_arg 4 + ss4 = outgoing_arg 4 ebb0: v1 = stack_load.i32 ss10 @@ -106,8 +108,10 @@ ebb0: stack_store v2, ss2 } ; sameln: function %stack() { -; nextln: $ss10 = stack_slot 8 -; nextln: $ss2 = stack_slot 4 +; nextln: $ss10 = spill_slot 8 +; nextln: $ss2 = local 4 +; nextln: $ss3 = incoming_arg 4 +; nextln: $ss4 = outgoing_arg 4 ; check: ebb0: ; nextln: $v1 = stack_load.i32 $ss10 diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 49617165e9..56c972c429 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -22,7 +22,7 @@ pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension, ArgumentPurpos pub use ir::types::Type; pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable, FuncRef, SigRef}; pub use ir::instructions::{Opcode, InstructionData, VariableArgs, ValueList, ValueListPool}; -pub use ir::stackslot::StackSlotData; +pub use ir::stackslot::{StackSlotKind, StackSlotData}; pub use ir::jumptable::JumpTableData; pub use ir::valueloc::{ValueLoc, ArgumentLoc}; pub use ir::dfg::{DataFlowGraph, ValueDef}; diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index b22b82919c..a205f6d410 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -3,43 +3,104 @@ //! The `StackSlotData` struct keeps track of a single stack slot in a function. //! -use std::fmt::{self, Display, Formatter}; +use std::fmt; +use std::str::FromStr; + +/// The kind of a stack slot. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum StackSlotKind { + /// A spill slot. This is a stack slot created by the register allocator. + SpillSlot, + + /// A local variable. This is a chunk of local stack memory for use by the `stack_load` and + /// `stack_store` instructions. + Local, + + /// An incoming function argument. + /// + /// If the current function has more arguments than fits in registers, the remaining arguments + /// are passed on the stack by the caller. These incoming arguments are represented as SSA + /// values assigned to incoming stack slots. + IncomingArg, + + /// An outgoing function argument. + /// + /// When preparing to call a function whose arguments don't fit in registers, outgoing argument + /// stack slots are used to represent individual arguments in the outgoing call frame. These + /// stack slots are only valid while setting up a call. + OutgoingArg, +} + +impl FromStr for StackSlotKind { + type Err = (); + + fn from_str(s: &str) -> Result { + use self::StackSlotKind::*; + match s { + "local" => Ok(Local), + "spill_slot" => Ok(SpillSlot), + "incoming_arg" => Ok(IncomingArg), + "outgoing_arg" => Ok(OutgoingArg), + _ => Err(()), + } + } +} + +impl fmt::Display for StackSlotKind { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + use self::StackSlotKind::*; + f.write_str(match *self { + Local => "local", + SpillSlot => "spill_slot", + IncomingArg => "incoming_arg", + OutgoingArg => "outgoing_arg", + }) + } +} /// Contents of a stack slot. #[derive(Clone, Debug)] pub struct StackSlotData { + /// The kind of stack slot. + pub kind: StackSlotKind, + /// Size of stack slot in bytes. pub size: u32, } impl StackSlotData { /// Create a stack slot with the specified byte size. - pub fn new(size: u32) -> StackSlotData { - StackSlotData { size: size } + pub fn new(kind: StackSlotKind, size: u32) -> StackSlotData { + StackSlotData { kind, size } } } -impl Display for StackSlotData { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write!(fmt, "stack_slot {}", self.size) +impl fmt::Display for StackSlotData { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{} {}", self.kind, self.size) } } #[cfg(test)] mod tests { use ir::Function; - use super::StackSlotData; + use super::*; #[test] fn stack_slot() { let mut func = Function::new(); - let ss0 = func.stack_slots.push(StackSlotData::new(4)); - let ss1 = func.stack_slots.push(StackSlotData::new(8)); + let ss0 = func.stack_slots + .push(StackSlotData::new(StackSlotKind::IncomingArg, 4)); + let ss1 = func.stack_slots + .push(StackSlotData::new(StackSlotKind::SpillSlot, 8)); assert_eq!(ss0.to_string(), "ss0"); assert_eq!(ss1.to_string(), "ss1"); assert_eq!(func.stack_slots[ss0].size, 4); assert_eq!(func.stack_slots[ss1].size, 8); + + assert_eq!(func.stack_slots[ss0].to_string(), "incoming_arg 4"); + assert_eq!(func.stack_slots[ss1].to_string(), "spill_slot 8"); } } diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 818e4d9700..f015b7d6df 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -359,7 +359,7 @@ impl<'a> fmt::Display for DisplayValues<'a> { #[cfg(test)] mod tests { - use ir::{Function, FunctionName, StackSlotData}; + use ir::{Function, FunctionName, StackSlotData, StackSlotKind}; use ir::types; #[test] @@ -370,21 +370,21 @@ mod tests { f.name = FunctionName::new("foo"); assert_eq!(f.to_string(), "function %foo() {\n}\n"); - f.stack_slots.push(StackSlotData::new(4)); - assert_eq!(f.to_string(), - "function %foo() {\n ss0 = stack_slot 4\n}\n"); + f.stack_slots + .push(StackSlotData::new(StackSlotKind::Local, 4)); + assert_eq!(f.to_string(), "function %foo() {\n ss0 = local 4\n}\n"); let ebb = f.dfg.make_ebb(); f.layout.append_ebb(ebb); assert_eq!(f.to_string(), - "function %foo() {\n ss0 = stack_slot 4\n\nebb0:\n}\n"); + "function %foo() {\n ss0 = local 4\n\nebb0:\n}\n"); f.dfg.append_ebb_arg(ebb, types::I8); assert_eq!(f.to_string(), - "function %foo() {\n ss0 = stack_slot 4\n\nebb0(v0: i8):\n}\n"); + "function %foo() {\n ss0 = local 4\n\nebb0(v0: i8):\n}\n"); f.dfg.append_ebb_arg(ebb, types::F32.by(4).unwrap()); assert_eq!(f.to_string(), - "function %foo() {\n ss0 = stack_slot 4\n\nebb0(v0: i8, v1: f32x4):\n}\n"); + "function %foo() {\n ss0 = local 4\n\nebb0(v0: i8, v1: f32x4):\n}\n"); } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 3529bc95a3..bc8ebcfa0d 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -896,13 +896,17 @@ impl<'a> Parser<'a> { // Parse a stack slot decl. // - // stack-slot-decl ::= * StackSlot(ss) "=" "stack_slot" Bytes {"," stack-slot-flag} + // stack-slot-decl ::= * StackSlot(ss) "=" stack-slot-kind Bytes {"," stack-slot-flag} + // stack-slot-kind ::= "local" + // | "spill_slot" + // | "incoming_arg" + // | "outgoing_arg" fn parse_stack_slot_decl(&mut self) -> Result<(u32, StackSlotData)> { let number = self.match_ss("expected stack slot number: ss«n»")?; - self.match_token(Token::Equal, "expected '=' in stack_slot decl")?; - self.match_identifier("stack_slot", "expected 'stack_slot'")?; + self.match_token(Token::Equal, "expected '=' in stack slot declaration")?; + let kind = self.match_enum("expected stack slot kind")?; - // stack-slot-decl ::= StackSlot(ss) "=" "stack_slot" * Bytes {"," stack-slot-flag} + // stack-slot-decl ::= StackSlot(ss) "=" stack-slot-kind * Bytes {"," stack-slot-flag} let bytes: i64 = self.match_imm64("expected byte-size in stack_slot decl")? .into(); if bytes < 0 { @@ -911,9 +915,9 @@ impl<'a> Parser<'a> { if bytes > u32::MAX as i64 { return err!(self.loc, "stack slot too large"); } - let data = StackSlotData::new(bytes as u32); + let data = StackSlotData::new(kind, bytes as u32); - // TBD: stack-slot-decl ::= StackSlot(ss) "=" "stack_slot" Bytes * {"," stack-slot-flag} + // TBD: stack-slot-decl ::= StackSlot(ss) "=" stack-slot-kind Bytes * {"," stack-slot-flag} Ok((number, data)) } @@ -1719,6 +1723,7 @@ mod tests { use super::*; use cretonne::ir::{ArgumentExtension, ArgumentPurpose}; use cretonne::ir::types; + use cretonne::ir::StackSlotKind; use cretonne::ir::entities::AnyEntity; use testfile::{Details, Comment}; use isaspec::IsaSpec; @@ -1793,8 +1798,8 @@ mod tests { #[test] fn stack_slot_decl() { let (func, _) = Parser::new("function %foo() { - ss3 = stack_slot 13 - ss1 = stack_slot 1 + ss3 = incoming_arg 13 + ss1 = spill_slot 1 }") .parse_function(None) .unwrap(); @@ -1802,16 +1807,18 @@ mod tests { let mut iter = func.stack_slots.keys(); let ss0 = iter.next().unwrap(); assert_eq!(ss0.to_string(), "ss0"); + assert_eq!(func.stack_slots[ss0].kind, StackSlotKind::IncomingArg); assert_eq!(func.stack_slots[ss0].size, 13); let ss1 = iter.next().unwrap(); assert_eq!(ss1.to_string(), "ss1"); + assert_eq!(func.stack_slots[ss1].kind, StackSlotKind::SpillSlot); assert_eq!(func.stack_slots[ss1].size, 1); assert_eq!(iter.next(), None); // Catch duplicate definitions. assert_eq!(Parser::new("function %bar() { - ss1 = stack_slot 13 - ss1 = stack_slot 1 + ss1 = spill_slot 13 + ss1 = spill_slot 1 }") .parse_function(None) .unwrap_err() @@ -1844,7 +1851,7 @@ mod tests { fn comments() { let (func, Details { comments, .. }) = Parser::new("; before function %comment() { ; decl - ss10 = stack_slot 13 ; stackslot. + ss10 = outgoing_arg 13 ; stackslot. ; Still stackslot. jt10 = jump_table ebb0 ; Jumptable diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 824eb97621..d905cdb9ab 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -223,7 +223,7 @@ mod tests { #[test] fn details() { let tf = parse_test("function %detail() { - ss10 = stack_slot 13 + ss10 = incoming_arg 13 jt10 = jump_table ebb0 ebb0(v4: i32, v7: i32): v10 = iadd v4, v7 diff --git a/misc/vim/syntax/cton.vim b/misc/vim/syntax/cton.vim index 86162d8a50..003029e1bb 100644 --- a/misc/vim/syntax/cton.vim +++ b/misc/vim/syntax/cton.vim @@ -1,7 +1,7 @@ " Vim syntax file " Language: Cretonne " Maintainer: Jakob Stoklund Olesen / From 605bda292565a77d701086dc83cb5b039aa9cc3c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Jun 2017 11:51:41 -0700 Subject: [PATCH 0803/3084] Add a stack frame manager. Use a new StackSlots struct to keep track of a function's stack slots instead of just an entity map. This let's us build more internal data structures for tracking the stack slots if necessary. Start by adding a make_spill_slot() function that will be used by the register allocator. --- lib/cretonne/src/ir/function.rs | 9 ++--- lib/cretonne/src/ir/mod.rs | 2 +- lib/cretonne/src/ir/stackslot.rs | 68 ++++++++++++++++++++++++++++++++ 3 files changed, 73 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 075ee7c6fd..3cb547fdff 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -5,8 +5,8 @@ use binemit::CodeOffset; use entity_map::{EntityMap, PrimaryEntityData}; -use ir::{FunctionName, Signature, Value, Inst, Ebb, StackSlot, StackSlotData, JumpTable, - JumpTableData, ValueLoc, DataFlowGraph, Layout}; +use ir::{FunctionName, Signature, Value, Inst, Ebb, StackSlots, JumpTable, JumpTableData, + ValueLoc, DataFlowGraph, Layout}; use isa::{TargetIsa, Encoding}; use std::fmt::{self, Display, Debug, Formatter}; use write::write_function; @@ -24,7 +24,7 @@ pub struct Function { pub signature: Signature, /// Stack slots allocated in this function. - pub stack_slots: EntityMap, + pub stack_slots: StackSlots, /// Jump tables used in this function. pub jump_tables: EntityMap, @@ -50,7 +50,6 @@ pub struct Function { pub offsets: EntityMap, } -impl PrimaryEntityData for StackSlotData {} impl PrimaryEntityData for JumpTableData {} impl Function { @@ -59,7 +58,7 @@ impl Function { Function { name, signature: sig, - stack_slots: EntityMap::new(), + stack_slots: StackSlots::new(), jump_tables: EntityMap::new(), dfg: DataFlowGraph::new(), layout: Layout::new(), diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 56c972c429..de5e1515b6 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -22,7 +22,7 @@ pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension, ArgumentPurpos pub use ir::types::Type; pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable, FuncRef, SigRef}; pub use ir::instructions::{Opcode, InstructionData, VariableArgs, ValueList, ValueListPool}; -pub use ir::stackslot::{StackSlotKind, StackSlotData}; +pub use ir::stackslot::{StackSlots, StackSlotKind, StackSlotData}; pub use ir::jumptable::JumpTableData; pub use ir::valueloc::{ValueLoc, ArgumentLoc}; pub use ir::dfg::{DataFlowGraph, ValueDef}; diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index a205f6d410..a1a5f0cb94 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -3,7 +3,10 @@ //! The `StackSlotData` struct keeps track of a single stack slot in a function. //! +use entity_map::{EntityMap, PrimaryEntityData, Keys}; +use ir::{Type, StackSlot}; use std::fmt; +use std::ops::Index; use std::str::FromStr; /// The kind of a stack slot. @@ -81,6 +84,71 @@ impl fmt::Display for StackSlotData { } } +impl PrimaryEntityData for StackSlotData {} + +/// Stack frame manager. +/// +/// Keep track of all the stack slots used by a function. +#[derive(Clone, Debug)] +pub struct StackSlots { + slots: EntityMap, +} + +/// Stack slot manager functions that behave mostly like an entity map. +impl StackSlots { + /// Create an empty stack slot manager. + pub fn new() -> StackSlots { + StackSlots { slots: EntityMap::new() } + } + + /// Clear out everything. + pub fn clear(&mut self) { + self.slots.clear(); + } + + /// Allocate a new stack slot. + /// + /// This function should be primarily used by the text format parser. There are more convenient + /// functions for creating specific kinds of stack slots below. + pub fn push(&mut self, data: StackSlotData) -> StackSlot { + self.slots.push(data) + } + + /// Check if `ss` is a valid stack slot reference. + pub fn is_valid(&self, ss: StackSlot) -> bool { + self.slots.is_valid(ss) + } + + /// Get an iterator over all the stack slot keys. + pub fn keys(&self) -> Keys { + self.slots.keys() + } + + /// Get a reference to the next stack slot that would be created by `push()`. + /// + /// This should just be used by the parser. + pub fn next_key(&self) -> StackSlot { + self.slots.next_key() + } +} + +/// Higher-level stack frame manipulation functions. +impl StackSlots { + /// Create a new spill slot for spilling values of type `ty`. + pub fn make_spill_slot(&mut self, ty: Type) -> StackSlot { + let bytes = (ty.bits() as u32 + 7) / 8; + self.push(StackSlotData::new(StackSlotKind::SpillSlot, bytes)) + } +} + +impl Index for StackSlots { + type Output = StackSlotData; + + fn index(&self, ss: StackSlot) -> &StackSlotData { + &self.slots[ss] + } +} + #[cfg(test)] mod tests { use ir::Function; From ddbf46bef4697bf4c2a21e70f1faa632c8024c0f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Jun 2017 13:08:13 -0700 Subject: [PATCH 0804/3084] Add typedefs for the common entity maps. The various entity maps in a function end up being referenced in multiple places, so create typedefs for them. --- lib/cretonne/src/binemit/relaxation.rs | 7 +++---- lib/cretonne/src/ir/function.rs | 29 +++++++++++++------------- lib/cretonne/src/ir/mod.rs | 16 ++++++++++++++ lib/cretonne/src/regalloc/coloring.rs | 18 ++++++++-------- lib/cretonne/src/regalloc/reload.rs | 7 +++---- 5 files changed, 45 insertions(+), 32 deletions(-) diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index 517867c725..53b7fd0112 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -28,9 +28,8 @@ //! ``` use binemit::CodeOffset; -use entity_map::EntityMap; -use ir::{Function, DataFlowGraph, Cursor, Inst, InstructionData, Opcode}; -use isa::{TargetIsa, EncInfo, Encoding}; +use ir::{Function, DataFlowGraph, Cursor, InstructionData, Opcode, InstEncodings}; +use isa::{TargetIsa, EncInfo}; use iterators::IteratorExtras; /// Relax branches and compute the final layout of EBB headers in `func`. @@ -127,7 +126,7 @@ fn fallthroughs(func: &mut Function) { /// Return the size of the replacement instructions up to and including the location where `pos` is /// left. fn relax_branch(dfg: &mut DataFlowGraph, - encodings: &mut EntityMap, + encodings: &mut InstEncodings, encinfo: &EncInfo, pos: &mut Cursor, offset: CodeOffset, diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 3cb547fdff..59213aff13 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -3,12 +3,11 @@ //! The `Function` struct defined in this module owns all of its extended basic blocks and //! instructions. -use binemit::CodeOffset; use entity_map::{EntityMap, PrimaryEntityData}; -use ir::{FunctionName, Signature, Value, Inst, Ebb, StackSlots, JumpTable, JumpTableData, - ValueLoc, DataFlowGraph, Layout}; -use isa::{TargetIsa, Encoding}; -use std::fmt::{self, Display, Debug, Formatter}; +use ir::{FunctionName, Signature, JumpTableData, DataFlowGraph, Layout}; +use ir::{JumpTables, InstEncodings, ValueLocations, StackSlots, EbbOffsets}; +use isa::TargetIsa; +use std::fmt; use write::write_function; /// A function. @@ -27,7 +26,7 @@ pub struct Function { pub stack_slots: StackSlots, /// Jump tables used in this function. - pub jump_tables: EntityMap, + pub jump_tables: JumpTables, /// Data flow graph containing the primary definition of all instructions, EBBs and values. pub dfg: DataFlowGraph, @@ -37,17 +36,17 @@ pub struct Function { /// Encoding recipe and bits for the legal instructions. /// Illegal instructions have the `Encoding::default()` value. - pub encodings: EntityMap, + pub encodings: InstEncodings, /// Location assigned to every value. - pub locations: EntityMap, + pub locations: ValueLocations, /// Code offsets of the EBB headers. /// /// This information is only transiently available after the `binemit::relax_branches` function /// computes it, and it can easily be recomputed by calling that function. It is not included /// in the textual IL format. - pub offsets: EntityMap, + pub offsets: EbbOffsets, } impl PrimaryEntityData for JumpTableData {} @@ -82,20 +81,20 @@ impl Function { /// Wrapper type capable of displaying a `Function` with correct ISA annotations. pub struct DisplayFunction<'a>(&'a Function, Option<&'a TargetIsa>); -impl<'a> Display for DisplayFunction<'a> { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { +impl<'a> fmt::Display for DisplayFunction<'a> { + fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result { write_function(fmt, self.0, self.1) } } -impl Display for Function { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { +impl fmt::Display for Function { + fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result { write_function(fmt, self, None) } } -impl Debug for Function { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { +impl fmt::Debug for Function { + fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result { write_function(fmt, self, None) } } diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index de5e1515b6..c79edfa4c6 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -31,3 +31,19 @@ pub use ir::function::Function; pub use ir::builder::InstBuilder; pub use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; pub use ir::memflags::MemFlags; + +use binemit; +use entity_map::EntityMap; +use isa; + +/// Map of value locations. +pub type ValueLocations = EntityMap; + +/// Map of jump tables. +pub type JumpTables = EntityMap; + +/// Map of instruction encodings. +pub type InstEncodings = EntityMap; + +/// Code offsets for EBBs. +pub type EbbOffsets = EntityMap; diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 53546de07f..c71218e7cc 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -33,7 +33,7 @@ use entity_map::EntityMap; use dominator_tree::DominatorTree; -use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph}; +use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph, ValueLocations}; use ir::{InstBuilder, Signature, ArgumentType, ArgumentLoc}; use isa::{TargetIsa, Encoding, EncInfo, OperandConstraint, ConstraintKind}; use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; @@ -218,7 +218,7 @@ impl<'a> Context<'a> { fn color_entry_args(&self, sig: &Signature, args: &[LiveValue], - locations: &mut EntityMap) + locations: &mut ValueLocations) -> AllocatableSet { assert_eq!(sig.argument_types.len(), args.len()); @@ -271,7 +271,7 @@ impl<'a> Context<'a> { fn color_args(&self, args: &[LiveValue], mut regs: AllocatableSet, - locations: &mut EntityMap) + locations: &mut ValueLocations) -> AllocatableSet { // Available registers *after* filtering out the dead arguments. let mut live_regs = regs.clone(); @@ -309,7 +309,7 @@ impl<'a> Context<'a> { dfg: &mut DataFlowGraph, tracker: &mut LiveValueTracker, regs: &mut AllocatableSet, - locations: &mut EntityMap, + locations: &mut ValueLocations, func_signature: &Signature) { dbg!("Coloring [{}] {}", self.encinfo.display(encoding), @@ -449,7 +449,7 @@ impl<'a> Context<'a> { // into the constraint solver. Convert them to solver variables so they can be diverted. fn divert_fixed_input_conflicts(&mut self, live: &[LiveValue], - locations: &mut EntityMap) { + locations: &mut ValueLocations) { for lv in live { if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); @@ -468,7 +468,7 @@ impl<'a> Context<'a> { constraints: &[OperandConstraint], defs: &[LiveValue], throughs: &[LiveValue], - locations: &mut EntityMap) { + locations: &mut ValueLocations) { for (op, lv) in constraints.iter().zip(defs) { if let ConstraintKind::FixedReg(reg) = op.kind { self.add_fixed_output(lv.value, op.regclass, reg, throughs, locations); @@ -483,7 +483,7 @@ impl<'a> Context<'a> { abi_types: &[ArgumentType], defs: &[LiveValue], throughs: &[LiveValue], - locations: &mut EntityMap) { + locations: &mut ValueLocations) { // It's technically possible for a call instruction to have fixed results before the // variable list of results, but we have no known instances of that. // Just assume all results are variable return values. @@ -506,7 +506,7 @@ impl<'a> Context<'a> { rc: RegClass, reg: RegUnit, throughs: &[LiveValue], - locations: &mut EntityMap) { + locations: &mut ValueLocations) { if !self.solver.add_fixed_output(rc, reg) { // The fixed output conflicts with some of the live-through registers. for lv in throughs { @@ -538,7 +538,7 @@ impl<'a> Context<'a> { constraints: &[OperandConstraint], defs: &[LiveValue], _dfg: &mut DataFlowGraph, - _locations: &mut EntityMap) { + _locations: &mut ValueLocations) { for (op, lv) in constraints.iter().zip(defs) { match op.kind { ConstraintKind::FixedReg(_) | diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index dd623de2b5..2226566799 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -10,8 +10,7 @@ //! pressure limits to be exceeded. use dominator_tree::DominatorTree; -use entity_map::EntityMap; -use ir::{Ebb, Inst, Value, Function, Signature, DataFlowGraph}; +use ir::{Ebb, Inst, Value, Function, Signature, DataFlowGraph, InstEncodings}; use ir::layout::{Cursor, CursorPosition}; use ir::{InstBuilder, Opcode, ArgumentType, ArgumentLoc}; use isa::RegClass; @@ -201,7 +200,7 @@ impl<'a> Context<'a> { encoding: Encoding, pos: &mut Cursor, dfg: &mut DataFlowGraph, - encodings: &mut EntityMap, + encodings: &mut InstEncodings, func_signature: &Signature, tracker: &mut LiveValueTracker) { // Get the operand constraints for `inst` that we are trying to satisfy. @@ -341,7 +340,7 @@ impl<'a> Context<'a> { stack: Value, reg: Value, pos: &mut Cursor, - encodings: &mut EntityMap, + encodings: &mut InstEncodings, dfg: &mut DataFlowGraph) { let ty = dfg.value_type(reg); From 1dd8d913aa20dfe1e549a9550a3a1a227c1ebc31 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Jun 2017 13:32:21 -0700 Subject: [PATCH 0805/3084] Assign spill slots to spilled values. As soon as a value is spilled, also assign it to a spill slot. For now, create a new spill slot for each spilled value. In the future, values will be sharing spill slots of they are phi-related. --- cranelift/filetests/regalloc/spill.cton | 11 ++++++++--- lib/cretonne/src/regalloc/spilling.rs | 25 +++++++++++++++++-------- 2 files changed, 25 insertions(+), 11 deletions(-) diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index f163c478b3..23ddf7129b 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -10,6 +10,7 @@ test regalloc ; - %x10-%x15 are function arguments. ; ; regex: V=v\d+ +; regex: WS=\s+ isa riscv enable_e @@ -19,14 +20,18 @@ isa riscv enable_e ; 2. The link register. ; 3. The first computed value, v2 function %pyramid(i32) -> i32 { +; check: ss0 = spill_slot 4 +; check: ss1 = spill_slot 4 +; check: ss2 = spill_slot 4 +; not: spill_slot ebb0(v1: i32): ; check: $ebb0($(rv1=$V): i32, $(rlink=$V): i32) - ; check: $v1 = spill $rv1 - ; nextln: $(link=$V) = spill $rlink + ; check: ,ss0]$WS $v1 = spill $rv1 + ; nextln: ,ss1]$WS $(link=$V) = spill $rlink ; not: spill v2 = iadd_imm v1, 12 ; check: $(r1v2=$V) = iadd_imm - ; nextln: $v2 = spill $r1v2 + ; nextln: ,ss2]$WS $v2 = spill $r1v2 ; not: spill v3 = iadd_imm v2, 12 v4 = iadd_imm v3, 12 diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 23d4799dad..f21e6e6c1a 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -16,11 +16,11 @@ //! operands. use dominator_tree::DominatorTree; -use entity_map::EntityMap; use ir::{DataFlowGraph, Layout, Cursor, InstBuilder}; -use ir::{Function, Ebb, Inst, Value, SigRef}; +use ir::{Function, Ebb, Inst, Value, ValueLoc, SigRef}; +use ir::{InstEncodings, StackSlots, ValueLocations}; use isa::registers::{RegClass, RegClassMask}; -use isa::{TargetIsa, RegInfo, EncInfo, Encoding, RecipeConstraints, ConstraintKind}; +use isa::{TargetIsa, RegInfo, EncInfo, RecipeConstraints, ConstraintKind}; use regalloc::affinity::Affinity; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; @@ -41,7 +41,9 @@ struct Context<'a> { encinfo: EncInfo, // References to parts of the current function. - encodings: &'a mut EntityMap, + encodings: &'a mut InstEncodings, + stack_slots: &'a mut StackSlots, + locations: &'a mut ValueLocations, // References to contextual data structures we need. domtree: &'a DominatorTree, @@ -85,6 +87,8 @@ impl Spilling { reginfo: isa.register_info(), encinfo: isa.encoding_info(), encodings: &mut func.encodings, + stack_slots: &mut func.stack_slots, + locations: &mut func.locations, domtree, liveness, topo, @@ -209,7 +213,7 @@ impl<'a> Context<'a> { if call_sig.is_some() { for lv in throughs { if lv.affinity.is_reg() && !self.spills.contains(&lv.value) { - self.spill_reg(lv.value); + self.spill_reg(lv.value, dfg); } } } @@ -351,7 +355,7 @@ impl<'a> Context<'a> { if let Some(value) = best { // Found a spill candidate. - self.spill_reg(value); + self.spill_reg(value, dfg); } else { panic!("Ran out of registers for mask={}", mask); } @@ -365,12 +369,17 @@ impl<'a> Context<'a> { /// /// Note that this does not update the cached affinity in the live value tracker. Call /// `process_spills` to do that. - fn spill_reg(&mut self, value: Value) { + fn spill_reg(&mut self, value: Value, dfg: &DataFlowGraph) { if let Affinity::Reg(rci) = self.liveness.spill(value) { let rc = self.reginfo.rc(rci); self.pressure.free(rc); self.spills.push(value); - dbg!("Spilled {}:{} -> {}", value, rc, self.pressure); + + // Assign a spill slot. + // TODO: phi-related values should use the same spill slot. + let ss = self.stack_slots.make_spill_slot(dfg.value_type(value)); + *self.locations.ensure(value) = ValueLoc::Stack(ss); + dbg!("Spilled {}:{} to {} -> {}", value, rc, ss, self.pressure); } else { panic!("Cannot spill {} that was already on the stack", value); } From 3693735874c4c116fc2f81af581c6990e1a925d1 Mon Sep 17 00:00:00 2001 From: Aleksey Kuznetsov Date: Mon, 19 Jun 2017 20:52:19 +0500 Subject: [PATCH 0806/3084] Implement an iterator over encodings (#96) * Implement an iterator over encodings * Implement TargetIsa::legal_encodings * Exclude non-boolean settings of isa flags bytes * Address flake8 long line error --- lib/cretonne/meta/gen_settings.py | 4 ++ lib/cretonne/src/isa/arm32/mod.rs | 26 +++---- lib/cretonne/src/isa/arm64/mod.rs | 26 +++---- lib/cretonne/src/isa/enc_tables.rs | 108 +++++++++++++++++++---------- lib/cretonne/src/isa/intel/mod.rs | 26 +++---- lib/cretonne/src/isa/mod.rs | 15 +++- lib/cretonne/src/isa/riscv/mod.rs | 26 +++---- 7 files changed, 139 insertions(+), 92 deletions(-) diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index 23e9a5c7a9..ae16b337c0 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -87,6 +87,10 @@ def gen_getters(sgrp, fmt): """ fmt.doc_comment("User-defined settings.") with fmt.indented('impl Flags {', '}'): + fmt.doc_comment('Returns inner slice of bytes.') + fmt.doc_comment('The byte-sized settings are not included.') + with fmt.indented('pub fn predicate_bytes(&self) -> &[u8] {', '}'): + fmt.line('&self.bytes[{}..]'.format(sgrp.boolean_offset)) fmt.doc_comment('Dynamic numbered predicate getter.') with fmt.indented( 'pub fn numbered_predicate(&self, p: usize) -> bool {', '}'): diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 5657c7523b..d986b9e0e2 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -8,9 +8,9 @@ mod registers; use binemit::CodeSink; use super::super::settings as shared_settings; -use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; +use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Encoding, Legalize}; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Legalize}; use ir; use regalloc; @@ -61,22 +61,22 @@ impl TargetIsa for Isa { enc_tables::INFO.clone() } - fn encode(&self, - _dfg: &ir::DataFlowGraph, - inst: &ir::InstructionData, - ctrl_typevar: ir::Type) - -> Result { + fn legal_encodings<'a, 'b>(&'a self, + _dfg: &'b ir::DataFlowGraph, + inst: &'b ir::InstructionData, + ctrl_typevar: ir::Type) + -> Result, Legalize> { lookup_enclist(ctrl_typevar, inst.opcode(), self.cpumode, &enc_tables::LEVEL2[..]) .and_then(|enclist_offset| { - general_encoding(enclist_offset, - &enc_tables::ENCLISTS[..], - |instp| enc_tables::check_instp(inst, instp), - |isap| self.isa_flags.numbered_predicate(isap as usize)) - .ok_or(Legalize::Expand) - }) + Ok(Encodings::new(enclist_offset, + &enc_tables::ENCLISTS[..], + inst, + enc_tables::check_instp, + self.isa_flags.predicate_bytes())) + }) } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index fe81ea8c09..c8c2de3cf8 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -8,9 +8,9 @@ mod registers; use binemit::CodeSink; use super::super::settings as shared_settings; -use isa::enc_tables::{lookup_enclist, general_encoding}; +use isa::enc_tables::{lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Encoding, Legalize}; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Legalize}; use ir; use regalloc; @@ -54,22 +54,22 @@ impl TargetIsa for Isa { enc_tables::INFO.clone() } - fn encode(&self, - _dfg: &ir::DataFlowGraph, - inst: &ir::InstructionData, - ctrl_typevar: ir::Type) - -> Result { + fn legal_encodings<'a, 'b>(&'a self, + _dfg: &'b ir::DataFlowGraph, + inst: &'b ir::InstructionData, + ctrl_typevar: ir::Type) + -> Result, Legalize> { lookup_enclist(ctrl_typevar, inst.opcode(), &enc_tables::LEVEL1_A64[..], &enc_tables::LEVEL2[..]) .and_then(|enclist_offset| { - general_encoding(enclist_offset, - &enc_tables::ENCLISTS[..], - |instp| enc_tables::check_instp(inst, instp), - |isap| self.isa_flags.numbered_predicate(isap as usize)) - .ok_or(Legalize::Expand) - }) + Ok(Encodings::new(enclist_offset, + &enc_tables::ENCLISTS[..], + inst, + enc_tables::check_instp, + self.isa_flags.predicate_bytes())) + }) } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index d7dde71eed..3467a2bdfd 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -2,7 +2,7 @@ //! //! This module contains types and functions for working with the encoding tables generated by //! `lib/cretonne/meta/gen_encoding.py`. -use ir::{Type, Opcode}; +use ir::{Type, Opcode, InstructionData}; use isa::{Encoding, Legalize}; use constant_hash::{Table, probe}; @@ -114,43 +114,75 @@ const CODE_ALWAYS: EncListEntry = PRED_MASK; /// The encoding list terminator. const CODE_FAIL: EncListEntry = 0xffff; -/// Find the first applicable general encoding of `inst`. -/// -/// Given an encoding list offset as returned by `lookup_enclist` above, search the encoding list -/// for the most first encoding that applies to `inst`. The encoding lists are laid out such that -/// this is the first valid entry in the list. -/// -/// This function takes two closures that are used to evaluate predicates: -/// - `instp` is passed an instruction predicate number to be evaluated on the current instruction. -/// - `isap` is passed an ISA predicate number to evaluate. -/// -/// Returns the corresponding encoding, or `None` if no list entries are satisfied by `inst`. -pub fn general_encoding(offset: usize, - enclist: &[EncListEntry], - instp: InstP, - isap: IsaP) - -> Option - where InstP: Fn(EncListEntry) -> bool, - IsaP: Fn(EncListEntry) -> bool -{ - let mut pos = offset; - while enclist[pos] != CODE_FAIL { - let pred = enclist[pos]; - if pred <= CODE_ALWAYS { - // This is an instruction predicate followed by recipe and encbits entries. - if pred == CODE_ALWAYS || instp(pred) { - return Some(Encoding::new(enclist[pos + 1], enclist[pos + 2])); - } - pos += 3; - } else { - // This is an ISA predicate entry. - pos += 1; - if !isap(pred & PRED_MASK) { - // ISA predicate failed, skip the next N entries. - pos += 3 * (pred >> PRED_BITS) as usize; - } +/// An iterator over legal encodings for the instruction. +pub struct Encodings<'a, 'b> { + offset: usize, + enclist: &'b [EncListEntry], + inst: &'b InstructionData, + instp: fn(&InstructionData, EncListEntry) -> bool, + isa_predicate_bytes: &'a [u8], +} + +impl<'a, 'b> Encodings<'a, 'b> { + /// Creates a new instance of `Encodings`. + /// + /// # Parameters + /// + /// - `offset` an offset into encoding list returned by `lookup_enclist` function. + /// - `inst` the current instruction. + /// - `enclist` a list of encoding entries. + /// - `instp` an instruction predicate number to be evaluated on the current instruction. + /// - `isa_predicate_bytes` an ISA flags as a slice of bytes to evaluate an ISA predicate number + /// on the current instruction. + /// + /// This iterator provides search for encodings that applies to the given instruction. The + /// encoding lists are laid out such that first call to `next` returns valid entry in the list + /// or `None`. + pub fn new(offset: usize, + enclist: &'b [EncListEntry], + inst: &'b InstructionData, + instp: fn(&InstructionData, EncListEntry) -> bool, + isa_predicate_bytes: &'a [u8]) + -> Self { + Encodings { + offset, + enclist, + inst, + instp, + isa_predicate_bytes, } } - - None +} + +impl<'a, 'b> Iterator for Encodings<'a, 'b> { + type Item = Encoding; + + fn next(&mut self) -> Option { + fn numbered_predicate(bytes: &[u8], p: usize) -> bool { + bytes[p / 8] & (1 << (p % 8)) != 0 + } + + while self.enclist[self.offset] != CODE_FAIL { + let pred = self.enclist[self.offset]; + if pred <= CODE_ALWAYS { + // This is an instruction predicate followed by recipe and encbits entries. + if pred == CODE_ALWAYS || (self.instp)(self.inst, pred) { + let encoding = Encoding::new(self.enclist[self.offset + 1], + self.enclist[self.offset + 2]); + self.offset += 3; + return Some(encoding); + } else { + self.offset += 3; + } + } else { + // This is an ISA predicate entry. + self.offset += 1; + if !numbered_predicate(self.isa_predicate_bytes, (pred & PRED_MASK) as usize) { + // ISA predicate failed, skip the next N entries. + self.offset += 3 * (pred >> PRED_BITS) as usize; + } + } + } + None + } } diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 1711be3166..6cfd0b4b85 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -8,9 +8,9 @@ mod registers; use binemit::CodeSink; use super::super::settings as shared_settings; -use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; +use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Encoding, Legalize}; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Legalize}; use ir; use regalloc; @@ -61,22 +61,22 @@ impl TargetIsa for Isa { enc_tables::INFO.clone() } - fn encode(&self, - _dfg: &ir::DataFlowGraph, - inst: &ir::InstructionData, - ctrl_typevar: ir::Type) - -> Result { + fn legal_encodings<'a, 'b>(&'a self, + _dfg: &'b ir::DataFlowGraph, + inst: &'b ir::InstructionData, + ctrl_typevar: ir::Type) + -> Result, Legalize> { lookup_enclist(ctrl_typevar, inst.opcode(), self.cpumode, &enc_tables::LEVEL2[..]) .and_then(|enclist_offset| { - general_encoding(enclist_offset, - &enc_tables::ENCLISTS[..], - |instp| enc_tables::check_instp(inst, instp), - |isap| self.isa_flags.numbered_predicate(isap as usize)) - .ok_or(Legalize::Expand) - }) + Ok(Encodings::new(enclist_offset, + &enc_tables::ENCLISTS[..], + inst, + enc_tables::check_instp, + self.isa_flags.predicate_bytes())) + }) } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index edc0034217..1ec80e790c 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -48,6 +48,7 @@ use binemit::CodeSink; use settings; use ir; use regalloc; +use isa::enc_tables::Encodings; pub mod riscv; pub mod intel; @@ -136,17 +137,27 @@ pub trait TargetIsa { /// Get a data structure describing the registers in this ISA. fn register_info(&self) -> RegInfo; + /// Returns an iterartor over legal encodings for the instruction. + fn legal_encodings<'a, 'b>(&'a self, + dfg: &'b ir::DataFlowGraph, + inst: &'b ir::InstructionData, + ctrl_typevar: ir::Type) + -> Result, Legalize>; + /// Encode an instruction after determining it is legal. /// /// If `inst` can legally be encoded in this ISA, produce the corresponding `Encoding` object. - /// Otherwise, return `None`. + /// Otherwise, return `Legalize` action. /// /// This is also the main entry point for determining if an instruction is legal. fn encode(&self, dfg: &ir::DataFlowGraph, inst: &ir::InstructionData, ctrl_typevar: ir::Type) - -> Result; + -> Result { + self.legal_encodings(dfg, inst, ctrl_typevar) + .and_then(|mut iter| iter.next().ok_or(Legalize::Expand)) + } /// Get a data structure describing the instruction encodings in this ISA. fn encoding_info(&self) -> EncInfo; diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index ad64e4fd29..5f9cd771a2 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -8,9 +8,9 @@ mod registers; use super::super::settings as shared_settings; use binemit::CodeSink; -use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, general_encoding}; +use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Encoding, Legalize}; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Legalize}; use ir; use regalloc; @@ -61,22 +61,22 @@ impl TargetIsa for Isa { enc_tables::INFO.clone() } - fn encode(&self, - _dfg: &ir::DataFlowGraph, - inst: &ir::InstructionData, - ctrl_typevar: ir::Type) - -> Result { + fn legal_encodings<'a, 'b>(&'a self, + _dfg: &'b ir::DataFlowGraph, + inst: &'b ir::InstructionData, + ctrl_typevar: ir::Type) + -> Result, Legalize> { lookup_enclist(ctrl_typevar, inst.opcode(), self.cpumode, &enc_tables::LEVEL2[..]) .and_then(|enclist_offset| { - general_encoding(enclist_offset, - &enc_tables::ENCLISTS[..], - |instp| enc_tables::check_instp(inst, instp), - |isap| self.isa_flags.numbered_predicate(isap as usize)) - .ok_or(Legalize::Expand) - }) + Ok(Encodings::new(enclist_offset, + &enc_tables::ENCLISTS[..], + inst, + enc_tables::check_instp, + self.isa_flags.predicate_bytes())) + }) } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { From 0c7316ae28d17f436e5c362a259bdd1c94180228 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 19 Jun 2017 16:24:10 -0700 Subject: [PATCH 0807/3084] Lint fixes (#99) * Replace a single-character string literal with a character literal. * Use is_some() instead of comparing with Some(_). * Add code-quotes around type names in comments. * Use !...is_empty() instead of len() != 0. * Tidy up redundant returns. * Remove redundant .clone() calls. * Remove unnecessary explicit lifetime parameters. * Tidy up unnecessary '&'s. * Add parens to make operator precedence explicit. * Use debug_assert_eq instead of debug_assert with ==. * Replace a &Vec argument with a &[...]. * Replace `a = a op b` with `a op= b`. * Avoid unnecessary closures. * Avoid .iter() and .iter_mut() for iterating over containers. * Remove unneeded qualification. --- lib/cretonne/src/context.rs | 2 +- lib/cretonne/src/ir/condcodes.rs | 52 ++++++++-------- lib/cretonne/src/ir/dfg.rs | 13 ++-- lib/cretonne/src/ir/extfunc.rs | 2 +- lib/cretonne/src/ir/immediates.rs | 4 +- lib/cretonne/src/ir/instructions.rs | 28 ++++----- lib/cretonne/src/ir/jumptable.rs | 2 +- lib/cretonne/src/ir/valueloc.rs | 8 +-- lib/cretonne/src/isa/intel/binemit.rs | 6 +- lib/cretonne/src/licm.rs | 29 ++++----- lib/cretonne/src/regalloc/allocatable_set.rs | 6 +- lib/cretonne/src/regalloc/pressure.rs | 4 +- lib/cretonne/src/regalloc/reload.rs | 2 +- lib/cretonne/src/regalloc/spilling.rs | 2 +- lib/cretonne/src/result.rs | 3 +- lib/cretonne/src/simple_gvn.rs | 4 +- lib/cretonne/src/topo_order.rs | 2 +- lib/cretonne/src/verifier/mod.rs | 64 ++++++++++---------- lib/filecheck/src/checker.rs | 6 +- lib/filecheck/src/lib.rs | 4 +- lib/reader/src/isaspec.rs | 2 +- lib/reader/src/lexer.rs | 16 ++--- lib/reader/src/parser.rs | 6 +- lib/reader/src/sourcemap.rs | 2 +- 24 files changed, 132 insertions(+), 137 deletions(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 90c4948b40..5e23ed3efc 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -60,7 +60,7 @@ impl Context { /// /// The `isa` argument is currently unused, but the verifier will soon be able to also /// check ISA-dependent constraints. - pub fn verify<'a>(&self, isa: Option<&TargetIsa>) -> verifier::Result { + pub fn verify(&self, isa: Option<&TargetIsa>) -> verifier::Result { verifier::verify_context(&self.func, &self.cfg, &self.domtree, isa) } diff --git a/lib/cretonne/src/ir/condcodes.rs b/lib/cretonne/src/ir/condcodes.rs index 520014471d..f63bd60938 100644 --- a/lib/cretonne/src/ir/condcodes.rs +++ b/lib/cretonne/src/ir/condcodes.rs @@ -88,17 +88,17 @@ impl CondCode for IntCC { impl Display for IntCC { fn fmt(&self, f: &mut Formatter) -> fmt::Result { use self::IntCC::*; - f.write_str(match self { - &Equal => "eq", - &NotEqual => "ne", - &SignedGreaterThan => "sgt", - &SignedGreaterThanOrEqual => "sge", - &SignedLessThan => "slt", - &SignedLessThanOrEqual => "sle", - &UnsignedGreaterThan => "ugt", - &UnsignedGreaterThanOrEqual => "uge", - &UnsignedLessThan => "ult", - &UnsignedLessThanOrEqual => "ule", + f.write_str(match *self { + Equal => "eq", + NotEqual => "ne", + SignedGreaterThan => "sgt", + SignedGreaterThanOrEqual => "sge", + SignedLessThan => "slt", + SignedLessThanOrEqual => "sle", + UnsignedGreaterThan => "ugt", + UnsignedGreaterThanOrEqual => "uge", + UnsignedLessThan => "ult", + UnsignedLessThanOrEqual => "ule", }) } } @@ -219,21 +219,21 @@ impl CondCode for FloatCC { impl Display for FloatCC { fn fmt(&self, f: &mut Formatter) -> fmt::Result { use self::FloatCC::*; - f.write_str(match self { - &Ordered => "ord", - &Unordered => "uno", - &Equal => "eq", - &NotEqual => "ne", - &OrderedNotEqual => "one", - &UnorderedOrEqual => "ueq", - &LessThan => "lt", - &LessThanOrEqual => "le", - &GreaterThan => "gt", - &GreaterThanOrEqual => "ge", - &UnorderedOrLessThan => "ult", - &UnorderedOrLessThanOrEqual => "ule", - &UnorderedOrGreaterThan => "ugt", - &UnorderedOrGreaterThanOrEqual => "uge", + f.write_str(match *self { + Ordered => "ord", + Unordered => "uno", + Equal => "eq", + NotEqual => "ne", + OrderedNotEqual => "one", + UnorderedOrEqual => "ueq", + LessThan => "lt", + LessThanOrEqual => "le", + GreaterThan => "gt", + GreaterThanOrEqual => "ge", + UnorderedOrLessThan => "ult", + UnorderedOrLessThanOrEqual => "ule", + UnorderedOrGreaterThan => "ugt", + UnorderedOrGreaterThanOrEqual => "uge", }) } } diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 36607e76f1..40f209a49b 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -247,10 +247,11 @@ impl DataFlowGraph { // Try to create short alias chains by finding the original source value. // This also avoids the creation of loops. let original = self.resolve_aliases(src); - assert!(dest != original, - "Aliasing {} to {} would create a loop", - dest, - src); + assert_ne!(dest, + original, + "Aliasing {} to {} would create a loop", + dest, + src); let ty = self.value_type(original); assert_eq!(self.value_type(dest), ty, @@ -326,8 +327,8 @@ pub enum ValueDef { impl ValueDef { /// Unwrap the instruction where the value was defined, or panic. pub fn unwrap_inst(&self) -> Inst { - match self { - &ValueDef::Res(inst, _) => inst, + match *self { + ValueDef::Res(inst, _) => inst, _ => panic!("Value is not an instruction result"), } } diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index e0863f8d1d..57346df5f2 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -70,7 +70,7 @@ impl Signature { pub struct DisplaySignature<'a>(&'a Signature, Option<&'a RegInfo>); fn write_list(f: &mut fmt::Formatter, - args: &Vec, + args: &[ArgumentType], regs: Option<&RegInfo>) -> fmt::Result { match args.split_first() { diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 44c4cd3fc3..fa6185cb8b 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -314,7 +314,7 @@ fn format_float(bits: u64, w: u8, t: u8, f: &mut Formatter) -> fmt::Result { let max_e_bits = (1u64 << w) - 1; let t_bits = bits & ((1u64 << t) - 1); // Trailing significand. let e_bits = (bits >> t) & max_e_bits; // Biased exponent. - let sign_bit = (bits >> w + t) & 1; + let sign_bit = (bits >> (w + t)) & 1; let bias: i32 = (1 << (w - 1)) - 1; let e = e_bits as i32 - bias; // Unbiased exponent. @@ -381,7 +381,7 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { debug_assert!((t + w + 1).is_power_of_two(), "Unexpected IEEE format size"); let (sign_bit, s2) = if s.starts_with('-') { - (1u64 << t + w, &s[1..]) + (1u64 << (t + w), &s[1..]) } else if s.starts_with('+') { (0, &s[1..]) } else { diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 66c6d039b0..ad4dd6fa21 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -286,23 +286,23 @@ impl InstructionData { /// Any instruction that can transfer control to another EBB reveals its possible destinations /// here. pub fn analyze_branch<'a>(&'a self, pool: &'a ValueListPool) -> BranchInfo<'a> { - match self { - &InstructionData::Jump { + match *self { + InstructionData::Jump { destination, ref args, .. - } => BranchInfo::SingleDest(destination, &args.as_slice(pool)), - &InstructionData::Branch { + } => BranchInfo::SingleDest(destination, args.as_slice(pool)), + InstructionData::Branch { destination, ref args, .. } => BranchInfo::SingleDest(destination, &args.as_slice(pool)[1..]), - &InstructionData::BranchIcmp { + InstructionData::BranchIcmp { destination, ref args, .. } => BranchInfo::SingleDest(destination, &args.as_slice(pool)[2..]), - &InstructionData::BranchTable { table, .. } => BranchInfo::Table(table), + InstructionData::BranchTable { table, .. } => BranchInfo::Table(table), _ => BranchInfo::NotABranch, } } @@ -312,10 +312,10 @@ impl InstructionData { /// /// Multi-destination branches like `br_table` return `None`. pub fn branch_destination(&self) -> Option { - match self { - &InstructionData::Jump { destination, .. } => Some(destination), - &InstructionData::Branch { destination, .. } => Some(destination), - &InstructionData::BranchIcmp { destination, .. } => Some(destination), + match *self { + InstructionData::Jump { destination, .. } => Some(destination), + InstructionData::Branch { destination, .. } => Some(destination), + InstructionData::BranchIcmp { destination, .. } => Some(destination), _ => None, } } @@ -337,11 +337,11 @@ impl InstructionData { /// /// Any instruction that can call another function reveals its call signature here. pub fn analyze_call<'a>(&'a self, pool: &'a ValueListPool) -> CallInfo<'a> { - match self { - &InstructionData::Call { func_ref, ref args, .. } => { - CallInfo::Direct(func_ref, &args.as_slice(pool)) + match *self { + InstructionData::Call { func_ref, ref args, .. } => { + CallInfo::Direct(func_ref, args.as_slice(pool)) } - &InstructionData::IndirectCall { sig_ref, ref args, .. } => { + InstructionData::IndirectCall { sig_ref, ref args, .. } => { CallInfo::Indirect(sig_ref, &args.as_slice(pool)[1..]) } _ => CallInfo::NotACall, diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index dd9bc5c187..24dad02056 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -65,7 +65,7 @@ impl JumpTableData { /// Enumerate over all `(idx, dest)` pairs in the table in order. /// /// This returns an iterator that skips any empty slots in the table. - pub fn entries<'a>(&'a self) -> Entries { + pub fn entries(&self) -> Entries { Entries(self.table.iter().cloned().enumerate()) } diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/cretonne/src/ir/valueloc.rs index b664fe70e5..8c5ea50dfa 100644 --- a/lib/cretonne/src/ir/valueloc.rs +++ b/lib/cretonne/src/ir/valueloc.rs @@ -98,16 +98,16 @@ impl Default for ArgumentLoc { impl ArgumentLoc { /// Is this an assigned location? (That is, not `Unassigned`). pub fn is_assigned(&self) -> bool { - match self { - &ArgumentLoc::Unassigned => false, + match *self { + ArgumentLoc::Unassigned => false, _ => true, } } /// Is this a register location? pub fn is_reg(&self) -> bool { - match self { - &ArgumentLoc::Reg(_) => true, + match *self { + ArgumentLoc::Reg(_) => true, _ => false, } } diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 834a9848c3..87587463ca 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -10,13 +10,13 @@ pub static RELOC_NAMES: [&'static str; 1] = ["Call"]; // Emit single-byte opcode. fn put_op1(bits: u16, sink: &mut CS) { - debug_assert!(bits & 0x0f00 == 0, "Invalid encoding bits for Op1*"); + debug_assert_eq!(bits & 0x0f00, 0, "Invalid encoding bits for Op1*"); sink.put1(bits as u8); } // Emit two-byte opcode: 0F XX fn put_op2(bits: u16, sink: &mut CS) { - debug_assert!(bits & 0x0f00 == 0x0400, "Invalid encoding bits for Op2*"); + debug_assert_eq!(bits & 0x0f00, 0x0400, "Invalid encoding bits for Op2*"); sink.put1(0x0f); sink.put1(bits as u8); } @@ -26,7 +26,7 @@ const PREFIX: [u8; 3] = [0x66, 0xf3, 0xf2]; // Emit single-byte opcode with mandatory prefix. fn put_mp1(bits: u16, sink: &mut CS) { - debug_assert!(bits & 0x0c00 == 0, "Invalid encoding bits for Mp1*"); + debug_assert_eq!(bits & 0x0c00, 0, "Invalid encoding bits for Mp1*"); let pp = (bits >> 8) & 3; sink.put1(PREFIX[(pp - 1) as usize]); sink.put1(bits as u8); diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 0fc8212460..10e0558999 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -21,18 +21,13 @@ pub fn do_licm(func: &mut Function, let invariant_inst = remove_loop_invariant_instructions(lp, func, cfg, loop_analysis); // Then we create the loop's pre-header and fill it with the invariant instructions // Then we remove the invariant instructions from the loop body - if invariant_inst.len() > 0 { + if !invariant_inst.is_empty() { // If the loop has a natural pre-header we use it, otherwise we create it. let mut pos; - match has_pre_header(&func.layout, - cfg, - domtree, - loop_analysis.loop_header(lp).clone()) { + match has_pre_header(&func.layout, cfg, domtree, loop_analysis.loop_header(lp)) { None => { - let pre_header = create_pre_header(loop_analysis.loop_header(lp).clone(), - func, - cfg, - domtree); + let pre_header = + create_pre_header(loop_analysis.loop_header(lp), func, cfg, domtree); pos = Cursor::new(&mut func.layout); pos.goto_bottom(pre_header); pos.prev_inst(); @@ -47,7 +42,7 @@ pub fn do_licm(func: &mut Function, // The last instruction of the pre-header is the termination instruction (usually // a jump) so we need to insert just before this. for inst in invariant_inst { - pos.insert_inst(inst.clone()); + pos.insert_inst(inst); } } } @@ -146,7 +141,7 @@ fn remove_loop_invariant_instructions(lp: Loop, for ebb in postorder_ebbs_loop(loop_analysis, cfg, lp).iter().rev() { // Arguments of the EBB are loop values for val in func.dfg.ebb_args(*ebb) { - loop_values.insert(val.clone()); + loop_values.insert(*val); } pos.goto_top(*ebb); while let Some(inst) = pos.next_inst() { @@ -164,7 +159,7 @@ fn remove_loop_invariant_instructions(lp: Loop, // If the instruction is not loop-invariant we push its results in the set of // loop values for out in func.dfg.inst_results(inst) { - loop_values.insert(out.clone()); + loop_values.insert(*out); } } } @@ -176,7 +171,7 @@ fn remove_loop_invariant_instructions(lp: Loop, fn postorder_ebbs_loop(loop_analysis: &LoopAnalysis, cfg: &ControlFlowGraph, lp: Loop) -> Vec { let mut grey = HashSet::new(); let mut black = HashSet::new(); - let mut stack = vec![loop_analysis.loop_header(lp).clone()]; + let mut stack = vec![loop_analysis.loop_header(lp)]; let mut postorder = Vec::new(); while !stack.is_empty() { @@ -187,13 +182,13 @@ fn postorder_ebbs_loop(loop_analysis: &LoopAnalysis, cfg: &ControlFlowGraph, lp: stack.push(node); // Get any children we've never seen before. for child in cfg.get_successors(node) { - if loop_analysis.is_in_loop(child.clone(), lp) && !grey.contains(child) { - stack.push(child.clone()); + if loop_analysis.is_in_loop(*child, lp) && !grey.contains(child) { + stack.push(*child); } } } else if !black.contains(&node) { - postorder.push(node.clone()); - black.insert(node.clone()); + postorder.push(node); + black.insert(node); } } postorder diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index 6ad8019ba9..aecd338e7f 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -50,14 +50,14 @@ impl AllocatableSet { /// It is an error to take a register that doesn't have all of its register units available. pub fn take(&mut self, rc: RegClass, reg: RegUnit) { let (idx, bits) = bitmask(rc, reg); - debug_assert!((self.avail[idx] & bits) == bits, "Not available"); + debug_assert_eq!(self.avail[idx] & bits, bits, "Not available"); self.avail[idx] &= !bits; } /// Make `reg` available for allocation again. pub fn free(&mut self, rc: RegClass, reg: RegUnit) { let (idx, bits) = bitmask(rc, reg); - debug_assert!((self.avail[idx] & bits) == 0, "Not allocated"); + debug_assert_eq!(self.avail[idx] & bits, 0, "Not allocated"); self.avail[idx] |= bits; } @@ -118,7 +118,7 @@ impl Iterator for RegSetIter { let unit = unit_offset + word.trailing_zeros() as RegUnit; // Clear that lowest bit so we won't find it again. - *word = *word & (*word - 1); + *word &= *word - 1; return Some(unit); } diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index 1398bff8d6..4839c7451a 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -199,7 +199,7 @@ impl Pressure { /// Reset all counts to 0, both base and transient. pub fn reset(&mut self) { - for e in self.toprc.iter_mut() { + for e in &mut self.toprc { e.base_count = 0; e.transient_count = 0; } @@ -220,7 +220,7 @@ impl Pressure { /// Reset all transient counts to 0. pub fn reset_transient(&mut self) { - for e in self.toprc.iter_mut() { + for e in &mut self.toprc { e.transient_count = 0; } } diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 2226566799..e8327b5a73 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -233,7 +233,7 @@ impl<'a> Context<'a> { // Create a live range for the new reload. let affinity = Affinity::Reg(cand.regclass.into()); self.liveness.create_dead(reg, dfg.value_def(reg), affinity); - self.liveness.extend_locally(reg, ebb, inst, &pos.layout); + self.liveness.extend_locally(reg, ebb, inst, pos.layout); } // Rewrite arguments. diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index f21e6e6c1a..7311de5d5d 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -226,7 +226,7 @@ impl<'a> Context<'a> { // Add register def to pressure, spill if needed. while let Err(mask) = self.pressure.take_transient(op.regclass) { dbg!("Need {} reg from {} throughs", op.regclass, throughs.len()); - self.spill_from(mask, throughs, dfg, &pos.layout); + self.spill_from(mask, throughs, dfg, pos.layout); } } } diff --git a/lib/cretonne/src/result.rs b/lib/cretonne/src/result.rs index 6605f1daff..960bd021d4 100644 --- a/lib/cretonne/src/result.rs +++ b/lib/cretonne/src/result.rs @@ -3,7 +3,6 @@ use verifier; use std::error::Error as StdError; use std::fmt; -use std::result; /// A compilation error. /// @@ -32,7 +31,7 @@ pub enum CtonError { } /// A Cretonne compilation result. -pub type CtonResult = result::Result<(), CtonError>; +pub type CtonResult = Result<(), CtonError>; impl fmt::Display for CtonError { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index e8f89f4a1b..bb339fbd4d 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -16,7 +16,7 @@ fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph) { let mut visible_values: HashMap = HashMap::new(); - let domtree = DominatorTree::with_function(func, &cfg); + let domtree = DominatorTree::with_function(func, cfg); // Visit EBBs in a reverse post-order. let mut pos = Cursor::new(&mut func.layout); @@ -47,7 +47,7 @@ pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph) { use std::collections::hash_map::Entry::*; match entry { Occupied(mut entry) => { - if domtree.dominates(*entry.get(), inst, &pos.layout) { + if domtree.dominates(*entry.get(), inst, pos.layout) { func.dfg.replace_with_aliases(inst, *entry.get()); pos.remove_inst_and_step_back(); } else { diff --git a/lib/cretonne/src/topo_order.rs b/lib/cretonne/src/topo_order.rs index 55cece2c2f..323df99ef0 100644 --- a/lib/cretonne/src/topo_order.rs +++ b/lib/cretonne/src/topo_order.rs @@ -73,7 +73,7 @@ impl TopoOrder { } } } - return self.stack.pop(); + self.stack.pop() } } diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index a2f806a447..08aeb9249f 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -229,21 +229,21 @@ impl<'a> Verifier<'a> { self.verify_value(inst, res)?; } - match &self.func.dfg[inst] { - &MultiAry { ref args, .. } => { + match self.func.dfg[inst] { + MultiAry { ref args, .. } => { self.verify_value_list(inst, args)?; } - &Jump { + Jump { destination, ref args, .. } | - &Branch { + Branch { destination, ref args, .. } | - &BranchIcmp { + BranchIcmp { destination, ref args, .. @@ -251,41 +251,41 @@ impl<'a> Verifier<'a> { self.verify_ebb(inst, destination)?; self.verify_value_list(inst, args)?; } - &BranchTable { table, .. } => { + BranchTable { table, .. } => { self.verify_jump_table(inst, table)?; } - &Call { func_ref, ref args, .. } => { + Call { func_ref, ref args, .. } => { self.verify_func_ref(inst, func_ref)?; self.verify_value_list(inst, args)?; } - &IndirectCall { sig_ref, ref args, .. } => { + IndirectCall { sig_ref, ref args, .. } => { self.verify_sig_ref(inst, sig_ref)?; self.verify_value_list(inst, args)?; } - &StackLoad { stack_slot, .. } | - &StackStore { stack_slot, .. } => { + StackLoad { stack_slot, .. } | + StackStore { stack_slot, .. } => { self.verify_stack_slot(inst, stack_slot)?; } // Exhaustive list so we can't forget to add new formats - &Nullary { .. } | - &Unary { .. } | - &UnaryImm { .. } | - &UnaryIeee32 { .. } | - &UnaryIeee64 { .. } | - &Binary { .. } | - &BinaryImm { .. } | - &Ternary { .. } | - &InsertLane { .. } | - &ExtractLane { .. } | - &IntCompare { .. } | - &IntCompareImm { .. } | - &FloatCompare { .. } | - &HeapLoad { .. } | - &HeapStore { .. } | - &Load { .. } | - &Store { .. } | - &RegMove { .. } => {} + Nullary { .. } | + Unary { .. } | + UnaryImm { .. } | + UnaryIeee32 { .. } | + UnaryIeee64 { .. } | + Binary { .. } | + BinaryImm { .. } | + Ternary { .. } | + InsertLane { .. } | + ExtractLane { .. } | + IntCompare { .. } | + IntCompareImm { .. } | + FloatCompare { .. } | + HeapLoad { .. } | + HeapStore { .. } | + Load { .. } | + Store { .. } | + RegMove { .. } => {} } Ok(()) @@ -627,14 +627,14 @@ impl<'a> Verifier<'a> { got_succs.extend(cfg.get_successors(ebb)); let missing_succs: Vec = expected_succs.difference(&got_succs).cloned().collect(); - if missing_succs.len() != 0 { + if !missing_succs.is_empty() { return err!(ebb, "cfg lacked the following successor(s) {:?}", missing_succs); } let excess_succs: Vec = got_succs.difference(&expected_succs).cloned().collect(); - if excess_succs.len() != 0 { + if !excess_succs.is_empty() { return err!(ebb, "cfg had unexpected successor(s) {:?}", excess_succs); } @@ -642,14 +642,14 @@ impl<'a> Verifier<'a> { got_preds.extend(cfg.get_predecessors(ebb).iter().map(|&(_, inst)| inst)); let missing_preds: Vec = expected_preds.difference(&got_preds).cloned().collect(); - if missing_preds.len() != 0 { + if !missing_preds.is_empty() { return err!(ebb, "cfg lacked the following predecessor(s) {:?}", missing_preds); } let excess_preds: Vec = got_preds.difference(&expected_preds).cloned().collect(); - if excess_preds.len() != 0 { + if !excess_preds.is_empty() { return err!(ebb, "cfg had unexpected predecessor(s) {:?}", excess_preds); } diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs index 6dcbdd2620..21d9049fee 100644 --- a/lib/filecheck/src/checker.rs +++ b/lib/filecheck/src/checker.rs @@ -66,7 +66,7 @@ impl Directive { return Err(Error::Syntax(format!("invalid variable name in regex: {}", rest))); } let var = rest[0..varlen].to_string(); - if !rest[varlen..].starts_with("=") { + if !rest[varlen..].starts_with('=') { return Err(Error::Syntax(format!("expected '=' after variable '{}' in regex: {}", var, rest))); @@ -196,7 +196,7 @@ impl Checker { // Check if `pat` matches in `range`. state.recorder.directive(dct_idx); if let Some((match_begin, match_end)) = state.match_positive(pat, range)? { - if let &Directive::Unordered(_) = dct { + if let Directive::Unordered(_) = *dct { // This was an unordered unordered match. // Keep track of the largest matched position, but leave `last_ordered` alone. state.max_match = max(state.max_match, match_end); @@ -231,7 +231,7 @@ impl Checker { // Verify any pending `not:` directives after the last ordered directive. for (not_idx, not_begin, rx) in nots.drain(..) { state.recorder.directive(not_idx); - if let Some(_) = rx.find(&text[not_begin..]) { + if rx.find(&text[not_begin..]).is_some() { // Matched `not:` pattern. // TODO: Use matched range for an error message. return Ok(false); diff --git a/lib/filecheck/src/lib.rs b/lib/filecheck/src/lib.rs index 66fe6036ab..8cca73736b 100644 --- a/lib/filecheck/src/lib.rs +++ b/lib/filecheck/src/lib.rs @@ -4,8 +4,8 @@ //! A list of directives is typically extracted from a file containing a test case. The test case //! is then run through the program under test, and its output matched against the directives. //! -//! See the [CheckerBuilder](struct.CheckerBuilder.html) and [Checker](struct.Checker.html) types -//! for the main library API. +//! See the [`CheckerBuilder`](struct.CheckerBuilder.html) and [`Checker`](struct.Checker.html) +//! types for the main library API. //! //! # Directives //! diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs index b7884bc42c..3715f4ed6b 100644 --- a/lib/reader/src/isaspec.rs +++ b/lib/reader/src/isaspec.rs @@ -25,7 +25,7 @@ pub enum IsaSpec { impl IsaSpec { /// If the `IsaSpec` contains exactly 1 `TargetIsa` we return a reference to it pub fn unique_isa(&self) -> Option<&TargetIsa> { - if let &IsaSpec::Some(ref isa_vec) = self { + if let IsaSpec::Some(ref isa_vec) = *self { if isa_vec.len() == 1 { return Some(&*isa_vec[0]); } diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index bbbb023490..d4392a5bd9 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -171,7 +171,7 @@ impl<'a> Lexer<'a> { // Scan a single-char token. fn scan_char(&mut self, tok: Token<'a>) -> Result, LocatedError> { - assert!(self.lookahead != None); + assert_ne!(self.lookahead, None); let loc = self.loc(); self.next_ch(); token(tok, loc) @@ -184,7 +184,7 @@ impl<'a> Lexer<'a> { -> Result, LocatedError> { let loc = self.loc(); for _ in 0..count { - assert!(self.lookahead != None); + assert_ne!(self.lookahead, None); self.next_ch(); } token(tok, loc) @@ -206,7 +206,7 @@ impl<'a> Lexer<'a> { fn scan_comment(&mut self) -> Result, LocatedError> { let loc = self.loc(); let text = self.rest_of_line(); - return token(Token::Comment(text), loc); + token(Token::Comment(text), loc) } // Scan a number token which can represent either an integer or floating point number. @@ -305,8 +305,8 @@ impl<'a> Lexer<'a> { // decoded token. fn numbered_entity(prefix: &str, number: u32) -> Option> { match prefix { - "v" => Value::with_number(number).map(|v| Token::Value(v)), - "ebb" => Ebb::with_number(number).map(|ebb| Token::Ebb(ebb)), + "v" => Value::with_number(number).map(Token::Value), + "ebb" => Ebb::with_number(number).map(Token::Ebb), "ss" => Some(Token::StackSlot(number)), "jt" => Some(Token::JumpTable(number)), "fn" => Some(Token::FuncRef(number)), @@ -339,7 +339,7 @@ impl<'a> Lexer<'a> { }; if is_vector { if number <= u16::MAX as u32 { - base_type.by(number as u16).map(|t| Token::Type(t)) + base_type.by(number as u16).map(Token::Type) } else { None } @@ -352,7 +352,7 @@ impl<'a> Lexer<'a> { let loc = self.loc(); let begin = self.pos + 1; - assert!(self.lookahead == Some('%')); + assert_eq!(self.lookahead, Some('%')); while let Some(c) = self.next_ch() { if !(c.is_ascii() && c.is_alphanumeric() || c == '_') { @@ -368,7 +368,7 @@ impl<'a> Lexer<'a> { let loc = self.loc(); let begin = self.pos + 1; - assert!(self.lookahead == Some('#')); + assert_eq!(self.lookahead, Some('#')); while let Some(c) = self.next_ch() { if !char::is_digit(c, 16) { diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index bc8ebcfa0d..3745bd849e 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -282,7 +282,7 @@ impl<'a> Parser<'a> { None => break, } } - return self.lookahead; + self.lookahead } // Begin gathering comments associated with `entity`. @@ -397,7 +397,7 @@ impl<'a> Parser<'a> { fn error(&self, message: &str) -> Error { Error { - location: self.loc.clone(), + location: self.loc, message: message.to_string(), } } @@ -1066,7 +1066,7 @@ impl<'a> Parser<'a> { self.consume(); self.parse_instruction(results, encoding, result_locations, ctx, ebb)?; } - _ if results.len() != 0 => return err!(self.loc, "expected -> or ="), + _ if !results.is_empty() => return err!(self.loc, "expected -> or ="), _ => self.parse_instruction(results, encoding, result_locations, ctx, ebb)?, } } diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index d905cdb9ab..d590a5220f 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -208,7 +208,7 @@ impl MutableSourceMap for SourceMap { } fn def_entity(&mut self, entity: AnyEntity, loc: &Location) -> Result<()> { - if self.locations.insert(entity, loc.clone()).is_some() { + if self.locations.insert(entity, *loc).is_some() { err!(loc, "duplicate entity: {}", entity) } else { Ok(()) From b6cff6a98a95d23127a40445bbced31d22134f6b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Jun 2017 10:07:23 -0700 Subject: [PATCH 0808/3084] Move EntityRef and entity_impl! into a new module. The EntityRef trait is used by more than just the EntityMap now, so it should live in its own module. Also move the entity_impl! macro into the new module so it can be used for defining new entity references anywhere. --- lib/cretonne/src/entity_list.rs | 5 +-- lib/cretonne/src/entity_map.rs | 19 ++------- lib/cretonne/src/entity_ref.rs | 51 ++++++++++++++++++++++ lib/cretonne/src/ir/entities.rs | 59 +++++--------------------- lib/cretonne/src/ir/jumptable.rs | 2 +- lib/cretonne/src/ir/layout.rs | 2 +- lib/cretonne/src/ir/progpoint.rs | 4 +- lib/cretonne/src/isa/registers.rs | 2 +- lib/cretonne/src/lib.rs | 2 + lib/cretonne/src/loop_analysis.rs | 28 +++--------- lib/cretonne/src/regalloc/diversion.rs | 2 +- lib/cretonne/src/regalloc/liverange.rs | 2 +- lib/cretonne/src/regalloc/solver.rs | 2 +- lib/cretonne/src/sparse_map.rs | 5 ++- 14 files changed, 85 insertions(+), 100 deletions(-) create mode 100644 lib/cretonne/src/entity_ref.rs diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index 1ccddc3aed..4b535e3f56 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -46,12 +46,11 @@ //! The index stored in an `EntityList` points to part 2, the list elements. The value 0 is //! reserved for the empty list which isn't allocated in the vector. +use entity_ref::EntityRef; use std::hash::{Hash, Hasher}; use std::marker::PhantomData; use std::mem; -use entity_map::EntityRef; - /// A small list of entity references allocated from a pool. /// /// All of the list methods that take a pool reference must be given the same pool reference every @@ -484,7 +483,7 @@ mod tests { use super::*; use super::{sclass_size, sclass_for_length}; use ir::Inst; - use entity_map::EntityRef; + use entity_ref::EntityRef; #[test] fn size_classes() { diff --git a/lib/cretonne/src/entity_map.rs b/lib/cretonne/src/entity_map.rs index 8227a19fb6..be73b790e1 100644 --- a/lib/cretonne/src/entity_map.rs +++ b/lib/cretonne/src/entity_map.rs @@ -1,8 +1,7 @@ //! Densely numbered entity references as mapping keys. //! -//! This module defines an `EntityRef` trait that should be implemented by reference types wrapping -//! a small integer index. The `EntityMap` data structure uses the dense index space to implement a -//! map with a vector. There are primary and secondary entity maps: +//! The `EntityMap` data structure uses the dense index space to implement a map with a vector. +//! There are primary and secondary entity maps: //! //! - A *primary* `EntityMap` contains the main definition of an entity, and it can be used to //! allocate new entity references with the `push` method. The values stores in a primary map @@ -10,22 +9,10 @@ //! - A *secondary* `EntityMap` contains additional data about entities kept in a primary map. The //! values need to implement `Clone + Default` traits so the map can be grown with `ensure`. -use std::vec::Vec; -use std::default::Default; +use entity_ref::EntityRef; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; -/// A type wrapping a small integer index should implement `EntityRef` so it can be used as the key -/// of an `EntityMap`. -pub trait EntityRef: Copy + Eq { - /// Create a new entity reference from a small integer. - /// This should crash if the requested index is not representable. - fn new(usize) -> Self; - - /// Get the index that was used to create this entity reference. - fn index(self) -> usize; -} - /// A mapping `K -> V` for densely indexed entity references. #[derive(Debug, Clone)] pub struct EntityMap diff --git a/lib/cretonne/src/entity_ref.rs b/lib/cretonne/src/entity_ref.rs new file mode 100644 index 0000000000..9990b521f0 --- /dev/null +++ b/lib/cretonne/src/entity_ref.rs @@ -0,0 +1,51 @@ +//! Densely numbered entity references as mapping keys. +//! +//! This module defines an `EntityRef` trait that should be implemented by reference types wrapping +//! a small integer index. + +/// A type wrapping a small integer index should implement `EntityRef` so it can be used as the key +/// of an `EntityMap` or `SparseMap`. +pub trait EntityRef: Copy + Eq { + /// Create a new entity reference from a small integer. + /// This should crash if the requested index is not representable. + fn new(usize) -> Self; + + /// Get the index that was used to create this entity reference. + fn index(self) -> usize; +} + +/// Macro which provides the common implementation of a 32-bit entity reference. +#[macro_export] +macro_rules! entity_impl { + // Basic traits. + ($entity:ident) => { + impl $crate::entity_ref::EntityRef for $entity { + fn new(index: usize) -> Self { + assert!(index < (::std::u32::MAX as usize)); + $entity(index as u32) + } + + fn index(self) -> usize { + self.0 as usize + } + } + + impl $crate::packed_option::ReservedValue for $entity { + fn reserved_value() -> $entity { + $entity(::std::u32::MAX) + } + } + }; + + // Include basic `Display` impl using the given display prefix. + // Display an `Ebb` reference as "ebb12". + ($entity:ident, $display_prefix:expr) => { + entity_impl!($entity); + + impl ::std::fmt::Display for $entity { + fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result { + write!(f, "{}{}", $display_prefix, self.0) + } + } + } +} diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 04b54e0488..98a29c7e61 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -19,46 +19,9 @@ //! The entity references all implement the `Display` trait in a way that matches the textual IL //! format. -use entity_map::EntityRef; -use packed_option::ReservedValue; -use std::fmt::{self, Display, Formatter}; +use std::fmt; use std::u32; -// Implement the common traits for a 32-bit entity reference. -macro_rules! entity_impl { - // Basic traits. - ($entity:ident) => { - impl EntityRef for $entity { - fn new(index: usize) -> Self { - assert!(index < (u32::MAX as usize)); - $entity(index as u32) - } - - fn index(self) -> usize { - self.0 as usize - } - } - - impl ReservedValue for $entity { - fn reserved_value() -> $entity { - $entity(u32::MAX) - } - } - }; - - // Include basic `Display` impl using the given display prefix. - // Display an `Ebb` reference as "ebb12". - ($entity:ident, $display_prefix:expr) => { - entity_impl!($entity); - - impl Display for $entity { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { - write!(fmt, "{}{}", $display_prefix, self.0) - } - } - } -} - /// An opaque reference to an extended basic block in a function. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] pub struct Ebb(u32); @@ -138,17 +101,17 @@ pub enum AnyEntity { SigRef(SigRef), } -impl Display for AnyEntity { - fn fmt(&self, fmt: &mut Formatter) -> fmt::Result { +impl fmt::Display for AnyEntity { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match *self { - AnyEntity::Function => write!(fmt, "function"), - AnyEntity::Ebb(r) => r.fmt(fmt), - AnyEntity::Inst(r) => r.fmt(fmt), - AnyEntity::Value(r) => r.fmt(fmt), - AnyEntity::StackSlot(r) => r.fmt(fmt), - AnyEntity::JumpTable(r) => r.fmt(fmt), - AnyEntity::FuncRef(r) => r.fmt(fmt), - AnyEntity::SigRef(r) => r.fmt(fmt), + AnyEntity::Function => write!(f, "function"), + AnyEntity::Ebb(r) => r.fmt(f), + AnyEntity::Inst(r) => r.fmt(f), + AnyEntity::Value(r) => r.fmt(f), + AnyEntity::StackSlot(r) => r.fmt(f), + AnyEntity::JumpTable(r) => r.fmt(f), + AnyEntity::FuncRef(r) => r.fmt(f), + AnyEntity::SigRef(r) => r.fmt(f), } } } diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index 24dad02056..aaaa694a50 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -122,7 +122,7 @@ impl Display for JumpTableData { mod tests { use super::JumpTableData; use ir::Ebb; - use entity_map::EntityRef; + use entity_ref::EntityRef; #[test] fn empty() { diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 3d569adcc6..0fa48edc6f 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -982,7 +982,7 @@ impl<'f> Cursor<'f> { #[cfg(test)] mod tests { use super::{Layout, Cursor, CursorPosition}; - use entity_map::EntityRef; + use entity_ref::EntityRef; use ir::{Ebb, Inst, ProgramOrder}; use std::cmp::Ordering; diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index 3134a53603..373bc382ab 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -1,6 +1,6 @@ //! Program points. -use entity_map::EntityRef; +use entity_ref::EntityRef; use ir::{Ebb, Inst, ValueDef}; use std::fmt; use std::u32; @@ -122,7 +122,7 @@ pub trait ProgramOrder { #[cfg(test)] mod tests { use super::*; - use entity_map::EntityRef; + use entity_ref::EntityRef; use ir::{Inst, Ebb}; #[test] diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 5549e98157..2823fbea37 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -1,6 +1,6 @@ //! Data structures describing the registers in an ISA. -use entity_map::EntityRef; +use entity_ref::EntityRef; use std::fmt; /// Register units are the smallest units of register allocation. diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 6ac2f7993b..09bdfc9302 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -12,6 +12,8 @@ pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); #[macro_use] pub mod dbg; +#[macro_use] +pub mod entity_ref; pub mod binemit; pub mod dominator_tree; diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index 5ed5f8a4d8..d0a9d505bd 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -1,34 +1,16 @@ //! A loop analysis represented as mappings of loops to their header Ebb //! and parent in the loop tree. -use ir::{Function, Ebb, Layout}; -use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; -use entity_map::{EntityMap, PrimaryEntityData}; -use packed_option::{PackedOption, ReservedValue}; -use entity_map::{EntityRef, Keys}; -use std::u32; +use entity_map::{EntityMap, PrimaryEntityData, Keys}; +use flowgraph::ControlFlowGraph; +use ir::{Function, Ebb, Layout}; +use packed_option::PackedOption; /// A opaque reference to a code loop. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct Loop(u32); -impl EntityRef for Loop { - fn new(index: usize) -> Self { - assert!(index < (u32::MAX as usize)); - Loop(index as u32) - } - - fn index(self) -> usize { - self.0 as usize - } -} - -impl ReservedValue for Loop { - fn reserved_value() -> Loop { - Loop(u32::MAX) - } -} - +entity_impl!(Loop, "loop"); /// Loop tree information for a single function. /// diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index 89decad82f..e8e8d3d190 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -89,7 +89,7 @@ impl RegDiversions { mod tests { use super::*; use ir::Value; - use entity_map::EntityRef; + use entity_ref::EntityRef; #[test] fn inserts() { diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 1cabd113dd..9866fabb20 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -394,7 +394,7 @@ impl SparseMapValue for LiveRange { mod tests { use super::LiveRange; use ir::{Inst, Ebb, Value}; - use entity_map::EntityRef; + use entity_ref::EntityRef; use ir::{ProgramOrder, ExpandedProgramPoint}; use std::cmp::Ordering; diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index d94cf3e72d..b2e73bf91f 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -675,7 +675,7 @@ impl Solver { #[cfg(test)] mod tests { - use entity_map::EntityRef; + use entity_ref::EntityRef; use ir::Value; use isa::{TargetIsa, RegClass, RegUnit}; use regalloc::AllocatableSet; diff --git a/lib/cretonne/src/sparse_map.rs b/lib/cretonne/src/sparse_map.rs index ad49ac49fa..b1e844b618 100644 --- a/lib/cretonne/src/sparse_map.rs +++ b/lib/cretonne/src/sparse_map.rs @@ -35,7 +35,8 @@ //! - `SparseMap` requires the values to implement `SparseMapValue` which means that they must //! contain their own key. -use entity_map::{EntityRef, EntityMap}; +use entity_map::EntityMap; +use entity_ref::EntityRef; use std::mem; use std::slice; use std::u32; @@ -215,7 +216,7 @@ pub type SparseSet = SparseMap; #[cfg(test)] mod tests { use super::*; - use entity_map::EntityRef; + use entity_ref::EntityRef; use ir::Inst; // Mock key-value object for testing. From 4a5d48fe11e9f2ed8908531f4fb82f5c95f0ba68 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 22 Jun 2017 12:01:32 -0700 Subject: [PATCH 0809/3084] Documentation fixes (#103) * Clarify that extended basic blocks are abbreviated as EBB. * Fix typo. * Fix a typo. * Fix typos. * Use the same phrase to indicate scalar-only as other places in the doc. * Mention that `band_imm` and friends are scalar-only. And mention that they're equivalent to their respective non-immediate-form counterparts. --- cranelift/docs/compare-llvm.rst | 4 ++-- cranelift/docs/langref.rst | 8 ++++---- cranelift/docs/regalloc.rst | 4 ++-- cranelift/docs/testing.rst | 2 +- lib/cretonne/meta/base/instructions.py | 18 +++++++++++++++++- lib/reader/src/parser.rs | 2 +- 6 files changed, 27 insertions(+), 11 deletions(-) diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst index dc19a64983..03b82dc7f7 100644 --- a/cranelift/docs/compare-llvm.rst +++ b/cranelift/docs/compare-llvm.rst @@ -152,7 +152,7 @@ can hold. address type too. - SIMD vector types are limited to a power-of-two number of vector lanes up to 256. LLVM allows an arbitrary number of SIMD lanes. -- Cretonne has no aggregrate types. LLVM has named and anonymous struct types as +- Cretonne has no aggregate types. LLVM has named and anonymous struct types as well as array types. Cretonne has multiple boolean types, whereas LLVM simply uses `i1`. The sized @@ -160,7 +160,7 @@ Cretonne boolean types are used to represent SIMD vector masks like ``b32x4`` where each lane is either all 0 or all 1 bits. Cretonne instructions and function calls can return multiple result values. LLVM -instead models this by returning a single value of an aggregrate type. +instead models this by returning a single value of an aggregate type. Instruction set --------------- diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index ac20591c36..7f8d3e038f 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -40,9 +40,9 @@ that can be referenced inside the function. In the example above, the preamble declares a single local variable, ``ss1``. After the preamble follows the :term:`function body` which consists of -:term:`extended basic block`\s, the first of which is the :term:`entry block`. -Every EBB ends with a :term:`terminator instruction`, so execution can never -fall through to the next EBB without an explicit branch. +:term:`extended basic block`\s (EBBs), the first of which is the +:term:`entry block`. Every EBB ends with a :term:`terminator instruction`, so +execution can never fall through to the next EBB without an explicit branch. A ``.cton`` file consists of a sequence of independent function definitions: @@ -253,7 +253,7 @@ indicate the different kinds of immediate operands on an instruction. A signed 32-bit immediate address offset. In the textual format, :type:`offset32` immediates always have an explicit - sign, and a 0 offset may beomitted. + sign, and a 0 offset may be omitted. .. type:: ieee32 diff --git a/cranelift/docs/regalloc.rst b/cranelift/docs/regalloc.rst index 75477e7375..8c73df41ec 100644 --- a/cranelift/docs/regalloc.rst +++ b/cranelift/docs/regalloc.rst @@ -55,12 +55,12 @@ EBB argument fixup The contract between the spilling and coloring phases is that the number of values in registers never exceeds the number of available registers. This -sounds simple enough in theory, but in pratice there are some complications. +sounds simple enough in theory, but in practice there are some complications. Real-world complications to SSA coloring ---------------------------------------- -In practice, instruction set architectures don't have "K interchangable +In practice, instruction set architectures don't have "K interchangeable registers", and register pressure can't be measured with a single number. There are complications: diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index b72ace08d3..edc8ee6570 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -100,7 +100,7 @@ of input functions in the :doc:`Cretonne textual intermediate language The available test commands are described below. -Many test comands only make sense in the context of a target instruction set +Many test commands only make sense in the context of a target instruction set architecture. These tests require one or more ISA specifications in the test header: diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index b4458ec585..dc9accf1cc 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -713,7 +713,8 @@ imul_imm = Instruction( 'imul_imm', """ Integer multiplication by immediate constant. - Polymorphic over all scalar integer types. + Polymorphic over all scalar integer types, but does not support vector + types. """, ins=(x, Y), outs=a) @@ -912,18 +913,33 @@ a = Operand('a', iB) band_imm = Instruction( 'band_imm', """ Bitwise and with immediate. + + Same as :inst:`band`, but one operand is an immediate constant. + + Polymorphic over all scalar integer types, but does not support vector + types. """, ins=(x, Y), outs=a) bor_imm = Instruction( 'bor_imm', """ Bitwise or with immediate. + + Same as :inst:`bor`, but one operand is an immediate constant. + + Polymorphic over all scalar integer types, but does not support vector + types. """, ins=(x, Y), outs=a) bxor_imm = Instruction( 'bxor_imm', """ Bitwise xor with immediate. + + Same as :inst:`bxor`, but one operand is an immediate constant. + + Polymorphic over all scalar integer types, but does not support vector + types. """, ins=(x, Y), outs=a) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 3745bd849e..ae88e09873 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -625,7 +625,7 @@ impl<'a> Parser<'a> { // Apply the ISA-specific settings to `isa_builder`. isaspec::parse_options(words, &mut isa_builder, &self.loc)?; - // Construct a trait object with the aggregrate settings. + // Construct a trait object with the aggregate settings. isas.push(isa_builder.finish(settings::Flags::new(&flag_builder))); } _ => break, From e15c332471f6a977410f7e8656b9b9b337ccf2ed Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 22 Jun 2017 12:11:54 -0700 Subject: [PATCH 0810/3084] Encode iconst.i32 for RISC-V. For large constants with the low 12 bits clear, we already have the "lui" encoding. Add "addi %x0" encodings for signed 12-bit constants. --- cranelift/filetests/isa/riscv/binary32.cton | 3 +++ cranelift/filetests/isa/riscv/verify-encoding.cton | 2 +- lib/cretonne/meta/isa/riscv/encodings.py | 7 ++++++- lib/cretonne/meta/isa/riscv/recipes.py | 5 +++++ lib/cretonne/src/isa/riscv/binemit.rs | 12 ++++++++++++ 5 files changed, 27 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 209d9922c0..6575609267 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -77,6 +77,9 @@ ebb0(v9999: i32): ; lui [-,%x7] v140 = iconst.i32 0x12345000 ; bin: 123453b7 [-,%x16] v141 = iconst.i32 0xffffffff_fedcb000 ; bin: fedcb837 + ; addi + [-,%x7] v142 = iconst.i32 1000 ; bin: 3e800393 + [-,%x16] v143 = iconst.i32 -905 ; bin: c7700813 ; Copies alias to iadd_imm. [-,%x7] v150 = copy v1 ; bin: 00050393 diff --git a/cranelift/filetests/isa/riscv/verify-encoding.cton b/cranelift/filetests/isa/riscv/verify-encoding.cton index 52b8d6d79c..b88fdc6402 100644 --- a/cranelift/filetests/isa/riscv/verify-encoding.cton +++ b/cranelift/filetests/isa/riscv/verify-encoding.cton @@ -6,7 +6,7 @@ function %RV32I(i32 link [%x1]) -> i32 link [%x1] { ebb0(v9999: i32): ; iconst.i32 needs legalizing, so it should throw a - [R#0,-] v1 = iconst.i32 1 ; error: Instruction failed to re-encode + [R#0,-] v1 = iconst.i32 0xf0f0f0f0f0 ; error: Instruction failed to re-encode return v9999 } diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index 374cc951af..d8afbab2cf 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -7,7 +7,7 @@ from base.immediates import intcc from .defs import RV32, RV64 from .recipes import OPIMM, OPIMM32, OP, OP32, LUI, BRANCH, JALR, JAL from .recipes import LOAD, STORE -from .recipes import R, Rshamt, Ricmp, I, Iicmp, Iret, Icall, Icopy +from .recipes import R, Rshamt, Ricmp, I, Iz, Iicmp, Iret, Icall, Icopy from .recipes import U, UJ, UJcall, SB, SBzero, GPsp, GPfi from .settings import use_m from cdsl.ast import Var @@ -40,6 +40,11 @@ RV64.enc(base.isub.i32, R, OP32(0b000, 0b0100000)) # There are no andiw/oriw/xoriw variations. RV64.enc(base.iadd_imm.i32, I, OPIMM32(0b000)) +# Use iadd_imm with %x0 to materialize constants. +RV32.enc(base.iconst.i32, Iz, OPIMM(0b000)) +RV64.enc(base.iconst.i32, Iz, OPIMM(0b000)) +RV64.enc(base.iconst.i64, Iz, OPIMM(0b000)) + # Dynamic shifts have the same masking semantics as the cton base instructions. for inst, inst_imm, f3, f7 in [ (base.ishl, base.ishl_imm, 0b001, 0b0000000), diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 60a67fbc00..2ea7f35597 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -104,6 +104,11 @@ I = EncRecipe( 'I', BinaryImm, size=4, ins=GPR, outs=GPR, instp=IsSignedInt(BinaryImm.imm, 12)) +# I-type instruction with a hardcoded %x0 rs1. +Iz = EncRecipe( + 'Iz', UnaryImm, size=4, ins=(), outs=GPR, + instp=IsSignedInt(UnaryImm.imm, 12)) + # I-type encoding of an integer comparison. Iicmp = EncRecipe( 'Iicmp', IntCompareImm, size=4, ins=GPR, outs=GPR, diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 0c25f9ec85..21638980b2 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -160,6 +160,18 @@ fn recipe_i(func: &Function, inst: Inst, sink: &mut CS) { } } +fn recipe_iz(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::UnaryImm { imm, .. } = func.dfg[inst] { + put_i(func.encodings[inst].bits(), + 0, + imm.into(), + func.locations[func.dfg.first_result(inst)].unwrap_reg(), + sink); + } else { + panic!("Expected UnaryImm format: {:?}", func.dfg[inst]); + } +} + fn recipe_iicmp(func: &Function, inst: Inst, sink: &mut CS) { if let InstructionData::IntCompareImm { arg, imm, .. } = func.dfg[inst] { put_i(func.encodings[inst].bits(), From 0f53fe191304179519bf68f3d62fbc3315c3b058 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 22 Jun 2017 14:34:21 -0700 Subject: [PATCH 0811/3084] Add a simple_gvn test that includes some basic control flow. --- cranelift/filetests/simple_gvn/basic.cton | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/cranelift/filetests/simple_gvn/basic.cton b/cranelift/filetests/simple_gvn/basic.cton index c76ec12b88..6ff45d1aef 100644 --- a/cranelift/filetests/simple_gvn/basic.cton +++ b/cranelift/filetests/simple_gvn/basic.cton @@ -19,3 +19,23 @@ ebb0(v0: i32, v1: i32): ; check: v6 = iadd $v4, $v4 return v6 } + +function %redundancies_on_some_paths(i32, i32, i32) -> i32 { +ebb0(v0: i32, v1: i32, v2: i32): + v3 = iadd v0, v1 + brz v3, ebb1 + v4 = iadd v0, v1 + jump ebb2(v4) +; check: jump ebb2(v3) + +ebb1: + v5 = iadd v0, v1 + jump ebb2(v5) +; check: jump ebb2(v3) + +ebb2(v6: i32): + v7 = iadd v0, v1 + v8 = iadd v6, v7 +; check: v8 = iadd v6, v3 + return v8 +} From 10e4b2fa06cee7940541474505a9fbcebd886b95 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 21 Jun 2017 10:03:39 -0700 Subject: [PATCH 0812/3084] Add two interference checking methods to LiveInterval. The overlaps_def() method tests if a definition would conflict with the live range. The reaches_use() method tests if a live range is live at an instruction. --- lib/cretonne/src/regalloc/liverange.rs | 36 +++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 9866fabb20..95dfb49f2e 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -108,7 +108,7 @@ //! use std::cmp::Ordering; -use ir::{Inst, Ebb, Value, ProgramPoint, ProgramOrder}; +use ir::{Inst, Ebb, Value, ProgramPoint, ExpandedProgramPoint, ProgramOrder}; use regalloc::affinity::Affinity; use sparse_map::SparseMapValue; @@ -381,6 +381,40 @@ impl LiveRange { pub fn liveins(&self) -> &[Interval] { &self.liveins } + + /// Check if this live range overlaps a definition in `ebb`. + pub fn overlaps_def(&self, def: ExpandedProgramPoint, ebb: Ebb, order: &PO) -> bool + where PO: ProgramOrder + { + // Check for an overlap with the local range. + if order.cmp(def, self.def_begin) != Ordering::Less && + order.cmp(def, self.def_end) == Ordering::Less { + return true; + } + + // Check for an overlap with a live-in range. + match self.livein_local_end(ebb, order) { + Some(inst) => order.cmp(def, inst) == Ordering::Less, + None => false, + } + } + + /// Check if this live range reaches a use at `inst` in `ebb`. + pub fn reaches_use(&self, user: Inst, ebb: Ebb, order: &PO) -> bool + where PO: ProgramOrder + { + // Check for an overlap with the local range. + if order.cmp(user, self.def_begin) == Ordering::Greater && + order.cmp(user, self.def_end) != Ordering::Greater { + return true; + } + + // Check for an overlap with a live-in range. + match self.livein_local_end(ebb, order) { + Some(inst) => order.cmp(user, inst) != Ordering::Greater, + None => false, + } + } } /// Allow a `LiveRange` to be stored in a `SparseMap` indexed by values. From 03a856f4dafcfa8baa8b31c590a6042c35ccbd0c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 22 Jun 2017 08:44:08 -0700 Subject: [PATCH 0813/3084] Implement Display and Debug for the program point types. --- lib/cretonne/src/ir/progpoint.rs | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index 373bc382ab..7adeb43c5c 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -82,15 +82,28 @@ impl From for ExpandedProgramPoint { } } -impl fmt::Display for ProgramPoint { +impl fmt::Display for ExpandedProgramPoint { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match (*self).into() { + match *self { ExpandedProgramPoint::Inst(x) => write!(f, "{}", x), ExpandedProgramPoint::Ebb(x) => write!(f, "{}", x), } } } +impl fmt::Display for ProgramPoint { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + let epp: ExpandedProgramPoint = (*self).into(); + epp.fmt(f) + } +} + +impl fmt::Debug for ExpandedProgramPoint { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "ExpandedProgramPoint({})", self) + } +} + impl fmt::Debug for ProgramPoint { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "ProgramPoint({})", self) From d079dead9ad434ab8a8a2cd9157b09ec93826736 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 22 Jun 2017 09:31:32 -0700 Subject: [PATCH 0814/3084] Skip ghost instructions when coloring. Ghost instructions don't have an encoding, and don't appear in the output. The values they define do not need to be assigned to registers, so they can be skipped. --- lib/cretonne/src/regalloc/coloring.rs | 40 +++++++++++---------------- 1 file changed, 16 insertions(+), 24 deletions(-) diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index c71218e7cc..56244bcc3c 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -31,18 +31,18 @@ //! defined by the instruction and only consider the colors of other values that are live at the //! instruction. -use entity_map::EntityMap; use dominator_tree::DominatorTree; +use entity_map::EntityMap; use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph, ValueLocations}; use ir::{InstBuilder, Signature, ArgumentType, ArgumentLoc}; -use isa::{TargetIsa, Encoding, EncInfo, OperandConstraint, ConstraintKind}; use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; +use isa::{TargetIsa, EncInfo, RecipeConstraints, OperandConstraint, ConstraintKind}; +use regalloc::RegDiversions; use regalloc::affinity::Affinity; use regalloc::allocatable_set::AllocatableSet; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; use regalloc::solver::Solver; -use regalloc::RegDiversions; use topo_order::TopoOrder; @@ -138,19 +138,18 @@ impl<'a> Context<'a> { let mut pos = Cursor::new(&mut func.layout); pos.goto_top(ebb); while let Some(inst) = pos.next_inst() { - let encoding = func.encodings[inst]; - assert!(encoding.is_legal(), "Illegal: {}", func.dfg[inst].opcode()); - self.visit_inst(inst, - encoding, - &mut pos, - &mut func.dfg, - tracker, - &mut regs, - &mut func.locations, - &func.signature); - tracker.drop_dead(inst); + if let Some(constraints) = self.encinfo.operand_constraints(func.encodings[inst]) { + self.visit_inst(inst, + constraints, + &mut pos, + &mut func.dfg, + tracker, + &mut regs, + &mut func.locations, + &func.signature); + tracker.drop_dead(inst); + } } - } /// Visit the `ebb` header. @@ -304,21 +303,14 @@ impl<'a> Context<'a> { /// or killed values from the set. fn visit_inst(&mut self, inst: Inst, - encoding: Encoding, + constraints: &RecipeConstraints, pos: &mut Cursor, dfg: &mut DataFlowGraph, tracker: &mut LiveValueTracker, regs: &mut AllocatableSet, locations: &mut ValueLocations, func_signature: &Signature) { - dbg!("Coloring [{}] {}", - self.encinfo.display(encoding), - dfg.display_inst(inst)); - - // Get the operand constraints for `inst` that we are trying to satisfy. - let constraints = self.encinfo - .operand_constraints(encoding) - .expect("Missing instruction encoding"); + dbg!("Coloring {}", dfg.display_inst(inst)); // Program the solver with register constraints for the input side. self.solver.reset(regs); From d5055275c4b76c7caa5937b7eeb6b50d1c57afe1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Jun 2017 15:17:19 -0700 Subject: [PATCH 0815/3084] Virtual registers. Add a VirtRegs collection which tracks virtual registers. A virtual register is a set of related SSA values whose live ranges don't interfere. It is advantageous to use the same register or spill slot for al the values in a virtual register. It reduces copies for EBB arguments. --- lib/cretonne/src/regalloc/context.rs | 5 + lib/cretonne/src/regalloc/mod.rs | 1 + lib/cretonne/src/regalloc/virtregs.rs | 135 ++++++++++++++++++++++++++ 3 files changed, 141 insertions(+) create mode 100644 lib/cretonne/src/regalloc/virtregs.rs diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index b404602a34..ea2cdcd924 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -13,6 +13,7 @@ use regalloc::live_value_tracker::LiveValueTracker; use regalloc::liveness::Liveness; use regalloc::reload::Reload; use regalloc::spilling::Spilling; +use regalloc::virtregs::VirtRegs; use result::CtonResult; use topo_order::TopoOrder; use verifier::{verify_context, verify_liveness}; @@ -20,6 +21,7 @@ use verifier::{verify_context, verify_liveness}; /// Persistent memory allocations for register allocation. pub struct Context { liveness: Liveness, + virtregs: VirtRegs, topo: TopoOrder, tracker: LiveValueTracker, spilling: Spilling, @@ -35,6 +37,7 @@ impl Context { pub fn new() -> Context { Context { liveness: Liveness::new(), + virtregs: VirtRegs::new(), topo: TopoOrder::new(), tracker: LiveValueTracker::new(), spilling: Spilling::new(), @@ -54,6 +57,8 @@ impl Context { domtree: &DominatorTree) -> CtonResult { // `Liveness` and `Coloring` are self-clearing. + self.virtregs.clear(); + // Tracker state (dominator live sets) is actually reused between the spilling and coloring // phases. self.tracker.clear(); diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index 59eb341eb0..ca4624b45b 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -15,6 +15,7 @@ mod pressure; mod reload; mod solver; mod spilling; +mod virtregs; pub use self::allocatable_set::AllocatableSet; pub use self::context::Context; diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs new file mode 100644 index 0000000000..6efe196805 --- /dev/null +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -0,0 +1,135 @@ +//! Virtual registers. +//! +//! A virtual register is a set of related SSA values whose live ranges don't interfere. If all the +//! values in a virtual register are assigned to the same location, fewer copies will result in the +//! output. +//! +//! A virtual register is typically built by merging together SSA values that are "phi-related" - +//! that is, one value is passed as an EBB argument to a branch and the other is the EBB parameter +//! value itself. +//! +//! If any values in a virtual register are spilled, they will use the same stack slot. This avoids +//! memory-to-memory copies when a spilled value is passed as an EBB argument. + +use entity_list::{EntityList, ListPool}; +use entity_map::{EntityMap, PrimaryEntityData}; +use ir::Value; +use packed_option::PackedOption; +use ref_slice::ref_slice; + +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] +pub struct VirtReg(u32); +entity_impl!(VirtReg, "vreg"); + +type ValueList = EntityList; +impl PrimaryEntityData for ValueList {} + +/// Collection of virtual registers. +/// +/// Each virtual register is a list of values. Also maintain a map from values to their unique +/// virtual register, if any. +pub struct VirtRegs { + /// Memory pool for the value lists. + pool: ListPool, + + /// The primary table of virtual registers. + /// + /// The list of values ion a virtual register is kept sorted according to the dominator tree's + /// RPO of the value defs. + vregs: EntityMap, + + /// Each value belongs to at most one virtual register. + value_vregs: EntityMap>, +} + +#[allow(dead_code)] +impl VirtRegs { + /// Create a new virtual register collection. + pub fn new() -> VirtRegs { + VirtRegs { + pool: ListPool::new(), + vregs: EntityMap::new(), + value_vregs: EntityMap::new(), + } + } + + /// Clear all virtual registers. + pub fn clear(&mut self) { + self.vregs.clear(); + self.value_vregs.clear(); + self.pool.clear(); + } + + /// Get the virtual register containing `value`, if any. + pub fn get(&self, value: Value) -> Option { + self.value_vregs.get_or_default(value).into() + } + + /// Get the list of values in `vreg`. The values are ordered according to `DomTree::rpo_cmp` of + /// their definition points. + pub fn values(&self, vreg: VirtReg) -> &[Value] { + self.vregs[vreg].as_slice(&self.pool) + } + + /// Get the congruence class of `value`. + /// + /// If `value` belongs to a virtual register, the congruence class is the values of the virtual + /// register. Otherwise it is just the value itself. + pub fn congruence_class<'a, 'b>(&'a self, value: &'b Value) -> &'b [Value] + where 'a: 'b + { + self.get(*value) + .map(|vr| self.values(vr)) + .unwrap_or(ref_slice(value)) + } + + /// Check if `a` and `b` belong to the same congruence class. + pub fn same_class(&self, a: Value, b: Value) -> bool { + match (self.get(a), self.get(b)) { + (Some(va), Some(vb)) => va == vb, + _ => a == b, + } + } + + /// Unify `values` into a single virtual register. + /// + /// The values in the slice can be singletons or they can belong to a virtual register already. + /// If a value belongs to a virtual register, all of the values in that register must be + /// present. + /// + /// The values are assumed to already be in RPO order. + pub fn unify(&mut self, values: &[Value]) -> VirtReg { + // Start by clearing all virtual registers involved. + // Pick a virtual register to reuse (the smallest number) or allocate a new one. + let mut singletons = 0; + let mut cleared = 0; + let vreg = values + .iter() + .filter_map(|&v| { + let vr = self.get(v); + match vr { + None => singletons += 1, + Some(vr) => { + if !self.vregs[vr].is_empty() { + cleared += self.vregs[vr].len(&self.pool); + self.vregs[vr].clear(&mut self.pool); + } + } + } + vr + }) + .min() + .unwrap_or_else(|| self.vregs.push(Default::default())); + + assert_eq!(values.len(), + singletons + cleared, + "Can't unify partial virtual registers"); + + self.vregs[vreg].extend(values.iter().cloned(), &mut self.pool); + for &v in values { + *self.value_vregs.ensure(v) = vreg.into(); + } + + vreg + } +} From 85b624d13bf5249d78ff2047cf2fcac8d1529e08 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 21 Jun 2017 09:24:12 -0700 Subject: [PATCH 0816/3084] Add a coalescing pass to the register allocator. Coalescing means creating virtual registers and transforming the code into conventional SSA form. This means that every value used as a branch argument will belong to the same virtual register as the corresponding EBB argument value. Conventional SSA form makes it easy to avoid memory-memory copies when spilling values, and the virtual registers can be used as hints when picking registers too. This reduces the number of register moves needed for EBB arguments. --- cranelift/filetests/regalloc/coalesce.cton | 91 ++++ lib/cretonne/src/dbg.rs | 21 + lib/cretonne/src/regalloc/coalescing.rs | 530 +++++++++++++++++++++ lib/cretonne/src/regalloc/context.rs | 18 + lib/cretonne/src/regalloc/mod.rs | 1 + 5 files changed, 661 insertions(+) create mode 100644 cranelift/filetests/regalloc/coalesce.cton create mode 100644 lib/cretonne/src/regalloc/coalescing.rs diff --git a/cranelift/filetests/regalloc/coalesce.cton b/cranelift/filetests/regalloc/coalesce.cton new file mode 100644 index 0000000000..8b76c8db6b --- /dev/null +++ b/cranelift/filetests/regalloc/coalesce.cton @@ -0,0 +1,91 @@ +test regalloc +isa riscv + +; Test the coalescer. +; regex: V=v\d+ +; regex: WS=\s+ + +; This function is already CSSA, so no copies should be inserted. +function %cssa(i32) -> i32 { +ebb0(v0: i32): + ; not: copy + ; v0 is used by the branch and passed as an arg - that's no conflict. + brnz v0, ebb1(v0) + ; v0 is live across the branch above. That's no conflict. + v1 = iadd_imm v0, 7 + jump ebb1(v1) + +ebb1(v10: i32): + v11 = iadd_imm v10, 7 + return v11 +} + +function %trivial(i32) -> i32 { +ebb0(v0: i32): + ; check: $(cp1=$V) = copy $v0 + ; nextln: brnz $v0, $ebb1($cp1) + brnz v0, ebb1(v0) + ; not: copy + v1 = iadd_imm v0, 7 + jump ebb1(v1) + +ebb1(v10: i32): + ; Use v0 in the destination EBB causes a conflict. + v11 = iadd v10, v0 + return v11 +} + +; A value is used as an SSA argument twice in the same branch. +function %dualuse(i32) -> i32 { +ebb0(v0: i32): + ; check: $(cp1=$V) = copy $v0 + ; nextln: brnz $v0, $ebb1($v0, $cp1) + brnz v0, ebb1(v0, v0) + ; not: copy + v1 = iadd_imm v0, 7 + v2 = iadd_imm v1, 56 + jump ebb1(v1, v2) + +ebb1(v10: i32, v11: i32): + v12 = iadd v10, v11 + return v12 +} + +; Interference away from the branch +; The interference can be broken with a copy at either branch. +function %interference(i32) -> i32 { +ebb0(v0: i32): + ; not: copy + brnz v0, ebb1(v0) + v1 = iadd_imm v0, 7 + ; v1 and v0 interfere here: + trapnz v0 + ; check: $(cp1=$V) = copy $v1 + ; nextln: jump $ebb1($cp1) + jump ebb1(v1) + +ebb1(v10: i32): + ; not: copy + v11 = iadd_imm v10, 7 + return v11 +} + +; A loop where one induction variable is used as a backedge argument. +function %fibonacci(i32) -> i32 { +ebb0(v0: i32): + ; not: copy + v1 = iconst.i32 1 + v2 = iconst.i32 2 + jump ebb1(v1, v2) + +ebb1(v10: i32, v11: i32): + ; v11 needs to be isolated because it interferes with v10. + ; check: $ebb1($v10: i32, $(nv11a=$V): i32) + ; check: $v11 = copy $nv11a + v12 = iadd v10, v11 + v13 = icmp ult v12, v0 + ; check: $(nv11b=$V) = copy $v11 + ; nextln: brnz $v13, $ebb1($nv11b, $v12) + brnz v13, ebb1(v11, v12) + return v12 +} diff --git a/lib/cretonne/src/dbg.rs b/lib/cretonne/src/dbg.rs index 06723fa018..dc7793f3d9 100644 --- a/lib/cretonne/src/dbg.rs +++ b/lib/cretonne/src/dbg.rs @@ -13,6 +13,7 @@ use std::ascii::AsciiExt; use std::cell::RefCell; use std::env; use std::ffi::OsStr; +use std::fmt; use std::fs::File; use std::io::{Write, BufWriter}; use std::sync::atomic; @@ -98,3 +99,23 @@ macro_rules! dbg { } } } + +/// Helper for printing lists. +pub struct DisplayList<'a, T>(pub &'a [T]) where T: 'a + fmt::Display; + +impl<'a, T> fmt::Display for DisplayList<'a, T> + where T: 'a + fmt::Display +{ + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self.0.split_first() { + None => write!(f, "[]"), + Some((first, rest)) => { + write!(f, "[{}", first)?; + for x in rest { + write!(f, ", {}", x)?; + } + write!(f, "]") + } + } + } +} diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs new file mode 100644 index 0000000000..70a20e3897 --- /dev/null +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -0,0 +1,530 @@ +//! Constructing conventional SSA form. +//! +//! Conventional SSA form is a subset of SSA form where any (transitively) phi-related values do +//! not interfere. We construct CSSA by building virtual registers that are as large as possible +//! and inserting copies where necessary such that all values passed to an EBB argument will belong +//! to the same virtual register as the EBB argument value itself. + +use dbg::DisplayList; +use dominator_tree::DominatorTree; +use flowgraph::{ControlFlowGraph, BasicBlock}; +use ir::{DataFlowGraph, Layout, Cursor, InstBuilder}; +use ir::{Function, Ebb, Inst, Value, ExpandedProgramPoint}; +use regalloc::affinity::Affinity; +use regalloc::liveness::Liveness; +use regalloc::virtregs::VirtRegs; +use std::cmp::Ordering; +use std::iter::Peekable; +use std::mem; +use isa::{TargetIsa, EncInfo}; + +/// Dominator forest. +/// +/// This is a utility type used for merging virtual registers, where each virtual register is a +/// list of values ordered according to `DomTree::rpo_cmp`. +/// +/// A `DomForest` object is used as a buffer for building virtual registers. It lets you merge two +/// sorted lists of values while checking for interference only whee necessary. +/// +/// The idea of a dominator forest was introduced here: +/// +/// Budimlic, Z., Budimlic, Z., Cooper, K. D., Cooper, K. D., Harvey, T. J., Harvey, T. J., et al. +/// (2002). Fast copy coalescing and live-range identification (Vol. 37, pp. 25–32). ACM. +/// http://doi.org/10.1145/543552.512534 +/// +/// The linear stack representation here: +/// +/// Boissinot, B., Darte, A., & Rastello, F. (2009). Revisiting out-of-SSA translation for +/// correctness, code quality and efficiency. Presented at the Proceedings of the 7th …. +struct DomForest { + // The sequence of values that have been merged so far. In RPO order of their defs. + values: Vec, + + // Stack representing the rightmost edge of the dominator forest so far, ending in the last + // element of `values`. At all times, each element in the stack dominates the next one, and all + // elements dominating the end of `values` are on the stack. + stack: Vec, +} + +/// A node in the dominator forest. +#[derive(Clone, Copy, Debug)] +struct Node { + value: Value, + /// Set identifier. Values in the same set are assumed to be non-interfering. + set: u8, + /// The program point where `value` is defined. + def: ExpandedProgramPoint, +} + +impl Node { + /// Create a node for `value`. + pub fn new(value: Value, set: u8, dfg: &DataFlowGraph) -> Node { + Node { + value, + set, + def: dfg.value_def(value).into(), + } + } +} + +/// Push a node to `stack` and update `stack` so it contains all dominator forest ancestors of +/// the pushed value. +/// + +impl DomForest { + /// Create a new empty dominator forest. + pub fn new() -> DomForest { + DomForest { + values: Vec::new(), + stack: Vec::new(), + } + } + + /// Swap the merged list with `buffer`, leaving the dominator forest empty. + /// + /// This is typically called after a successful merge to extract the merged value list. + pub fn swap(&mut self, buffer: &mut Vec) { + buffer.clear(); + mem::swap(&mut self.values, buffer); + } + + /// Add a single node to the forest. + /// + /// Update the stack so its dominance invariants are preserved. Detect a parent node on the + /// stack which is the closest one dominating the new node. + /// + /// If the pushed node's parent in the dominator forest belongs to a different set, returns + /// `Some(parent)`. + fn push_node(&mut self, node: Node, layout: &Layout, domtree: &DominatorTree) -> Option { + self.values.push(node.value); + + // The stack contains the current sequence of dominating defs. Pop elements until we + // find one that dominates `node`. + while let Some(top) = self.stack.pop() { + if domtree.dominates(top.def, node.def, layout) { + // This is the right insertion spot for `node`. + self.stack.push(top); + self.stack.push(node); + // If the parent value comes from a different set, return it for interference + // checking. If the sets are equal, assume that interference is already handled. + if top.set != node.set { + return Some(top.value); + } else { + return None; + } + } + } + + // No dominators, start a new tree in the forest. + self.stack.push(node); + None + } + + /// Try to merge two sorted sets of values. Each slice must already be sorted and free of any + /// interference. + /// + /// It is permitted for a value to appear in both lists. The merged sequence will only have one + /// copy of the value. + /// + /// If an interference is detected, returns `Err((a, b))` with the two conflicting values form + /// `va` and `vb` respectively. + /// + /// If the merge succeeds, returns `Ok(())`. The merged sequence can be extracted with + /// `swap()`. + pub fn try_merge(&mut self, + va: &[Value], + vb: &[Value], + dfg: &DataFlowGraph, + layout: &Layout, + domtree: &DominatorTree, + liveness: &Liveness) + -> Result<(), (Value, Value)> { + self.stack.clear(); + self.values.clear(); + self.values.reserve(va.len() + vb.len()); + + // Convert the two value lists into a merged sequence of nodes. + let merged = MergedNodes { + a: va.iter().map(|&value| Node::new(value, 0, dfg)).peekable(), + b: vb.iter().map(|&value| Node::new(value, 1, dfg)).peekable(), + layout, + domtree, + }; + for node in merged { + if let Some(parent) = self.push_node(node, layout, domtree) { + // Check if `parent` live range contains `node.def`. + let lr = liveness + .get(parent) + .expect("No live range for parent value"); + if lr.overlaps_def(node.def, layout.pp_ebb(node.def), layout) { + // Interference detected. Get the `(a, b)` order right in the error. + return Err(if node.set == 0 { + (node.value, parent) + } else { + (parent, node.value) + }); + } + } + } + + Ok(()) + } +} + +/// Node-merging iterator. +/// +/// Given two ordered sequences of nodes, yield an ordered sequence containing all of them. +/// Duplicates are removed. +struct MergedNodes<'a, IA, IB> + where IA: Iterator, + IB: Iterator +{ + a: Peekable, + b: Peekable, + layout: &'a Layout, + domtree: &'a DominatorTree, +} + +impl<'a, IA, IB> Iterator for MergedNodes<'a, IA, IB> + where IA: Iterator, + IB: Iterator +{ + type Item = Node; + + fn next(&mut self) -> Option { + let ord = match (self.a.peek(), self.b.peek()) { + (Some(a), Some(b)) => { + // If the two values are defined at the same point, compare value numbers instead + // this is going to cause an interference conflict unless its actually the same + // value appearing in both streams. + self.domtree + .rpo_cmp(a.def, b.def, self.layout) + .then(Ord::cmp(&a.value, &b.value)) + } + (Some(_), None) => Ordering::Less, + (None, Some(_)) => Ordering::Greater, + (None, None) => return None, + }; + match ord { + Ordering::Equal => { + // The two iterators produced the same value. Just return the first one. + self.b.next(); + self.a.next() + } + Ordering::Less => self.a.next(), + Ordering::Greater => self.b.next(), + } + } +} + +/// Data structures to be used by the coalescing pass. +pub struct Coalescing { + forest: DomForest, + + // Current set of coalesced values. Kept sorted and interference free. + values: Vec, + + // New values that were created when splitting interferences. + split_values: Vec, +} + +/// One-shot context created once per invocation. +struct Context<'a> { + isa: &'a TargetIsa, + encinfo: EncInfo, + + func: &'a mut Function, + domtree: &'a DominatorTree, + liveness: &'a mut Liveness, + virtregs: &'a mut VirtRegs, + + forest: &'a mut DomForest, + values: &'a mut Vec, + split_values: &'a mut Vec, +} + +impl Coalescing { + /// Create a new coalescing pass. + pub fn new() -> Coalescing { + Coalescing { + forest: DomForest::new(), + values: Vec::new(), + split_values: Vec::new(), + } + + } + + /// Convert `func` to conventional SSA form and build virtual registers in the process. + pub fn conventional_ssa(&mut self, + isa: &TargetIsa, + func: &mut Function, + cfg: &ControlFlowGraph, + domtree: &DominatorTree, + liveness: &mut Liveness, + virtregs: &mut VirtRegs) { + dbg!("Coalescing for:\n{}", func.display(isa)); + let mut context = Context { + isa, + encinfo: isa.encoding_info(), + func, + domtree, + liveness, + virtregs, + forest: &mut self.forest, + values: &mut self.values, + split_values: &mut self.split_values, + }; + + // TODO: The iteration order matters here. We should coalesce in the most important blocks + // first, so they get first pick at forming virtual registers. + for &ebb in domtree.cfg_postorder() { + let preds = cfg.get_predecessors(ebb); + if !preds.is_empty() { + for argnum in 0..context.func.dfg.num_ebb_args(ebb) { + context.coalesce_ebb_arg(ebb, argnum, preds) + } + } + } + } +} + +impl<'a> Context<'a> { + /// Coalesce the `argnum`'th argument to `ebb`. + fn coalesce_ebb_arg(&mut self, ebb: Ebb, argnum: usize, preds: &[BasicBlock]) { + self.split_values.clear(); + let mut succ_val = self.func.dfg.ebb_args(ebb)[argnum]; + dbg!("Processing {}/{}: {}", ebb, argnum, succ_val); + + // We want to merge the virtual register for `succ_val` with the virtual registers for + // the branch arguments in the predecessors. This may not be possible if any live + // ranges interfere, so we can insert copies to break interferences: + // + // pred: + // jump ebb1(v1) + // + // ebb1(v10: i32): + // ... + // + // In the predecessor: + // + // v2 = copy v1 + // jump ebb(v2) + // + // A predecessor copy is always required if the branch argument virtual register is + // live into the successor. + // + // In the successor: + // + // ebb1(v11: i32): + // v10 = copy v11 + // + // A successor copy is always required if the `succ_val` virtual register is live at + // any predecessor branch. + + while let Some(bad_value) = self.try_coalesce(argnum, succ_val, preds) { + dbg!("Isolating interfering value {}", bad_value); + // The bad value has some conflict that can only be reconciled by excluding its + // congruence class from the new virtual register. + // + // Try to catch infinite splitting loops. The values created by splitting should never + // have irreconcilable interferences. + assert!(!self.split_values.contains(&bad_value), + "{} was already isolated", + bad_value); + let split_len = self.split_values.len(); + + // The bad value can be both the successor value and a predecessor value at the same + // time. + if self.virtregs.same_class(bad_value, succ_val) { + succ_val = self.split_succ(ebb, succ_val); + } + + // Check the predecessors. + for &(pred_ebb, pred_inst) in preds { + let pred_val = self.func.dfg.inst_variable_args(pred_inst)[argnum]; + if self.virtregs.same_class(bad_value, pred_val) { + self.split_pred(pred_inst, pred_ebb, argnum, pred_val); + } + } + + // Second loop check. + assert_ne!(split_len, + self.split_values.len(), + "Couldn't isolate {}", + bad_value); + } + + let vreg = self.virtregs.unify(self.values); + dbg!("Coalesced {} arg {} into {} = {}", + ebb, + argnum, + vreg, + DisplayList(self.virtregs.values(vreg))); + } + + /// Reset `self.values` to just the set of split values. + fn reset_values(&mut self) { + self.values.clear(); + self.values.extend_from_slice(self.split_values); + let domtree = &self.domtree; + let func = &self.func; + self.values + .sort_by(|&a, &b| { + domtree.rpo_cmp(func.dfg.value_def(a), func.dfg.value_def(b), &func.layout) + }); + } + + /// Try coalescing predecessors with `succ_val`. + /// + /// Returns a value from a congruence class that needs to be split before starting over, or + /// `None` if everything was successfully coalesced into `self.values`. + fn try_coalesce(&mut self, + argnum: usize, + succ_val: Value, + preds: &[BasicBlock]) + -> Option { + /// Initialize the value list with the split values. These are guaranteed to be + /// interference free, and anything that interferes with them must be split away. + self.reset_values(); + dbg!("Trying {} with split values: {:?}", succ_val, self.values); + + // Start by adding `succ_val` so we can determine if it interferes with any of the new + // split values. If it does, we must split it. + if self.add_class(succ_val).is_err() { + return Some(succ_val); + } + + for &(pred_ebb, pred_inst) in preds { + let pred_val = self.func.dfg.inst_variable_args(pred_inst)[argnum]; + dbg!("Checking {}: {}: {}", + pred_val, + pred_ebb, + self.func.dfg.display_inst(pred_inst)); + if let Err((a, b)) = self.add_class(pred_val) { + dbg!("Found conflict between {} and {}", a, b); + // We have a conflict between the already merged value `a` and one of the new + // values `b`. + // + // Check if the `a` live range is fundamentally incompatible with `pred_inst`. + if self.liveness + .get(a) + .expect("No live range for interfering value") + .reaches_use(pred_inst, pred_ebb, &self.func.layout) { + // Splitting at `pred_inst` wouldn't resolve the interference, so we need to + // start over. + return Some(a); + } + + // The local conflict could be avoided by splitting at this predecessor, so try + // that. This split is not necessarily required, but it allows us to make progress. + let new_val = self.split_pred(pred_inst, pred_ebb, argnum, pred_val); + assert!(self.add_class(new_val).is_ok(), + "Splitting didn't resolve conflict."); + } + } + + None + } + + /// Try merging the congruence class for `value` into `self.values`. + /// + /// Leave `self.values` unchanged on failure. + fn add_class(&mut self, value: Value) -> Result<(), (Value, Value)> { + self.forest + .try_merge(&self.values, + self.virtregs.congruence_class(&value), + &self.func.dfg, + &self.func.layout, + self.domtree, + self.liveness)?; + self.forest.swap(&mut self.values); + Ok(()) + } + + /// Split the congruence class for the `argnum` argument to `pred_inst` by inserting a copy. + fn split_pred(&mut self, + pred_inst: Inst, + pred_ebb: Ebb, + argnum: usize, + pred_val: Value) + -> Value { + let copy; + { + let mut pos = Cursor::new(&mut self.func.layout); + pos.goto_inst(pred_inst); + copy = self.func.dfg.ins(&mut pos).copy(pred_val); + } + let inst = self.func.dfg.value_def(copy).unwrap_inst(); + let ty = self.func.dfg.value_type(copy); + + dbg!("Inserted {}, before {}: {}", + self.func.dfg.display_inst(inst), + pred_ebb, + self.func.dfg.display_inst(pred_inst)); + + // Give it an encoding. + let encoding = self.isa + .encode(&self.func.dfg, &self.func.dfg[inst], ty) + .expect("Can't encode copy"); + *self.func.encodings.ensure(inst) = encoding; + + // Create a live range for the new value. + let affinity = Affinity::new(&self.encinfo + .operand_constraints(encoding) + .expect("Bad copy encoding") + .outs + [0]); + self.liveness.create_dead(copy, inst, affinity); + self.liveness + .extend_locally(copy, pred_ebb, pred_inst, &self.func.layout); + + self.func.dfg.inst_variable_args_mut(pred_inst)[argnum] = copy; + self.split_values.push(copy); + copy + } + + /// Split the congruence class for the successor EBB value itself. + fn split_succ(&mut self, ebb: Ebb, succ_val: Value) -> Value { + let ty = self.func.dfg.value_type(succ_val); + let new_val = self.func.dfg.replace_ebb_arg(succ_val, ty); + + // Insert a copy instruction at the top of ebb. + { + let mut pos = Cursor::new(&mut self.func.layout); + pos.goto_top(ebb); + pos.next_inst(); + self.func + .dfg + .ins(&mut pos) + .with_result(succ_val) + .copy(new_val); + } + let inst = self.func.dfg.value_def(succ_val).unwrap_inst(); + self.liveness.move_def_locally(succ_val, inst); + + dbg!("Inserted {}, following {}({}: {})", + self.func.dfg.display_inst(inst), + ebb, + new_val, + ty); + + // Give it an encoding. + let encoding = self.isa + .encode(&self.func.dfg, &self.func.dfg[inst], ty) + .expect("Can't encode copy"); + *self.func.encodings.ensure(inst) = encoding; + + // Create a live range for the new value. + let affinity = Affinity::new(&self.encinfo + .operand_constraints(encoding) + .expect("Bad copy encoding") + .outs + [0]); + self.liveness.create_dead(new_val, ebb, affinity); + self.liveness + .extend_locally(new_val, ebb, inst, &self.func.layout); + + self.split_values.push(new_val); + new_val + } +} diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index ea2cdcd924..78e1595747 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -8,6 +8,7 @@ use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir::Function; use isa::TargetIsa; +use regalloc::coalescing::Coalescing; use regalloc::coloring::Coloring; use regalloc::live_value_tracker::LiveValueTracker; use regalloc::liveness::Liveness; @@ -22,6 +23,7 @@ use verifier::{verify_context, verify_liveness}; pub struct Context { liveness: Liveness, virtregs: VirtRegs, + coalescing: Coalescing, topo: TopoOrder, tracker: LiveValueTracker, spilling: Spilling, @@ -38,6 +40,7 @@ impl Context { Context { liveness: Liveness::new(), virtregs: VirtRegs::new(), + coalescing: Coalescing::new(), topo: TopoOrder::new(), tracker: LiveValueTracker::new(), spilling: Spilling::new(), @@ -70,6 +73,21 @@ impl Context { verify_liveness(isa, func, cfg, &self.liveness)?; } + // Coalesce and create conventional SSA form. + self.coalescing + .conventional_ssa(isa, + func, + cfg, + domtree, + &mut self.liveness, + &mut self.virtregs); + + if isa.flags().enable_verifier() { + verify_context(func, cfg, domtree, Some(isa))?; + verify_liveness(isa, func, cfg, &self.liveness)?; + } + + // Second pass: Spilling. self.spilling .run(isa, diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index ca4624b45b..f689503c03 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -9,6 +9,7 @@ pub mod live_value_tracker; pub mod coloring; mod affinity; +mod coalescing; mod context; mod diversion; mod pressure; From da24bd422aa5c228f6618fb07b13a9449e7697ab Mon Sep 17 00:00:00 2001 From: d1m0 Date: Thu, 22 Jun 2017 16:47:14 -0700 Subject: [PATCH 0817/3084] Convert interval sets inside TypeSet/ValueTypeSet in general sets (#102) * Convert TypeSet fields to sets; Add BitSet type to rust; Encode ValueTypeSets using BitSet; (still need mypy cleanup) * nits * cleanup nits * forgot mypy type annotations * rustfmt fixes * Round 1 comments: filer b2, b4; doc comments in python; move bitset in its own toplevel module; Use Into * fixes * Revert comment to appease rustfmt --- lib/cretonne/meta/cdsl/test_typevar.py | 22 ++-- lib/cretonne/meta/cdsl/typevar.py | 160 +++++++++++++++---------- lib/cretonne/src/bitset.rs | 147 +++++++++++++++++++++++ lib/cretonne/src/ir/instructions.rs | 93 ++++++-------- lib/cretonne/src/lib.rs | 1 + 5 files changed, 292 insertions(+), 131 deletions(-) create mode 100644 lib/cretonne/src/bitset.rs diff --git a/lib/cretonne/meta/cdsl/test_typevar.py b/lib/cretonne/meta/cdsl/test_typevar.py index 29db26f583..97793f71a5 100644 --- a/lib/cretonne/meta/cdsl/test_typevar.py +++ b/lib/cretonne/meta/cdsl/test_typevar.py @@ -40,7 +40,7 @@ class TestTypeSet(TestCase): a = TypeSet(lanes=True, ints=True, floats=True) s = set() s.add(a) - a.max_int = 32 + a.ints.remove(64) # Can't rehash after modification. with self.assertRaises(AssertionError): a in s @@ -71,14 +71,18 @@ class TestTypeVar(TestCase): def test_singleton(self): x = TypeVar.singleton(i32) self.assertEqual(str(x), '`i32`') - self.assertEqual(x.type_set.min_int, 32) - self.assertEqual(x.type_set.max_int, 32) - self.assertEqual(x.type_set.min_lanes, 1) - self.assertEqual(x.type_set.max_lanes, 1) + self.assertEqual(min(x.type_set.ints), 32) + self.assertEqual(max(x.type_set.ints), 32) + self.assertEqual(min(x.type_set.lanes), 1) + self.assertEqual(max(x.type_set.lanes), 1) + self.assertEqual(len(x.type_set.floats), 0) + self.assertEqual(len(x.type_set.bools), 0) x = TypeVar.singleton(i32.by(4)) self.assertEqual(str(x), '`i32x4`') - self.assertEqual(x.type_set.min_int, 32) - self.assertEqual(x.type_set.max_int, 32) - self.assertEqual(x.type_set.min_lanes, 4) - self.assertEqual(x.type_set.max_lanes, 4) + self.assertEqual(min(x.type_set.ints), 32) + self.assertEqual(max(x.type_set.ints), 32) + self.assertEqual(min(x.type_set.lanes), 4) + self.assertEqual(max(x.type_set.lanes), 4) + self.assertEqual(len(x.type_set.floats), 0) + self.assertEqual(len(x.type_set.bools), 0) diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 1dc8630f5f..119c6bdf01 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -9,7 +9,7 @@ import math from . import types, is_power_of_two try: - from typing import Tuple, Union, TYPE_CHECKING # noqa + from typing import Tuple, Union, Iterable, Any, Set, TYPE_CHECKING # noqa if TYPE_CHECKING: from srcgen import Formatter # noqa Interval = Tuple[int, int] @@ -46,6 +46,32 @@ def intersect(a, b): return (None, None) +def is_empty(intv): + # type: (Interval) -> bool + return intv is None or intv is False or intv == (None, None) + + +def encode_bitset(vals, size): + # type: (Iterable[int], int) -> int + """ + Encode a set of values (each between 0 and size) as a bitset of width size. + """ + res = 0 + assert is_power_of_two(size) and size <= 64 + for v in vals: + assert 0 <= v and v < size + res |= 1 << v + return res + + +def pp_set(s): + # type: (Iterable[Any]) -> str + """ + Return a consistent string representation of a set (ordering is fixed) + """ + return '{' + ', '.join([repr(x) for x in sorted(s)]) + '}' + + def decode_interval(intv, full_range, default=None): # type: (BoolInterval, Interval, int) -> Interval """ @@ -74,6 +100,18 @@ def decode_interval(intv, full_range, default=None): return (default, default) +def interval_to_set(intv): + # type: (Interval) -> Set + if is_empty(intv): + return set() + + (lo, hi) = intv + assert is_power_of_two(lo) + assert is_power_of_two(hi) + assert lo <= hi + return set([2**i for i in range(int_log2(lo), int_log2(hi)+1)]) + + class TypeSet(object): """ A set of types. @@ -95,22 +133,22 @@ class TypeSet(object): A typeset representing scalar integer types `i8` through `i32`: >>> TypeSet(ints=(8, 32)) - TypeSet(lanes=(1, 1), ints=(8, 32)) + TypeSet(lanes={1}, ints={8, 16, 32}) Passing `True` instead of a range selects all available scalar types: >>> TypeSet(ints=True) - TypeSet(lanes=(1, 1), ints=(8, 64)) + TypeSet(lanes={1}, ints={8, 16, 32, 64}) >>> TypeSet(floats=True) - TypeSet(lanes=(1, 1), floats=(32, 64)) + TypeSet(lanes={1}, floats={32, 64}) >>> TypeSet(bools=True) - TypeSet(lanes=(1, 1), bools=(1, 64)) + TypeSet(lanes={1}, bools={1, 8, 16, 32, 64}) Similarly, passing `True` for the lanes selects all possible scalar and vector types: >>> TypeSet(lanes=True, ints=True) - TypeSet(lanes=(1, 256), ints=(8, 64)) + TypeSet(lanes={1, 2, 4, 8, 16, 32, 64, 128, 256}, ints={8, 16, 32, 64}) :param lanes: `(min, max)` inclusive range of permitted vector lane counts. :param ints: `(min, max)` inclusive range of permitted scalar integer @@ -123,19 +161,19 @@ class TypeSet(object): def __init__(self, lanes=None, ints=None, floats=None, bools=None): # type: (BoolInterval, BoolInterval, BoolInterval, BoolInterval) -> None # noqa - self.min_lanes, self.max_lanes = decode_interval( - lanes, (1, MAX_LANES), 1) - self.min_int, self.max_int = decode_interval(ints, (8, MAX_BITS)) - self.min_float, self.max_float = decode_interval(floats, (32, 64)) - self.min_bool, self.max_bool = decode_interval(bools, (1, MAX_BITS)) + self.lanes = interval_to_set(decode_interval(lanes, (1, MAX_LANES), 1)) + self.ints = interval_to_set(decode_interval(ints, (8, MAX_BITS))) + self.floats = interval_to_set(decode_interval(floats, (32, 64))) + self.bools = interval_to_set(decode_interval(bools, (1, MAX_BITS))) + self.bools = set(filter(lambda x: x == 1 or x >= 8, self.bools)) def typeset_key(self): - # type: () -> Tuple[int, int, int, int, int, int, int, int] + # type: () -> Tuple[Tuple, Tuple, Tuple, Tuple] """Key tuple used for hashing and equality.""" - return (self.min_lanes, self.max_lanes, - self.min_int, self.max_int, - self.min_float, self.max_float, - self.min_bool, self.max_bool) + return (tuple(sorted(list(self.lanes))), + tuple(sorted(list(self.ints))), + tuple(sorted(list(self.floats))), + tuple(sorted(list(self.bools)))) def __hash__(self): # type: () -> int @@ -153,31 +191,29 @@ class TypeSet(object): def __repr__(self): # type: () -> str - s = 'TypeSet(lanes=({}, {})'.format(self.min_lanes, self.max_lanes) - if self.min_int is not None: - s += ', ints=({}, {})'.format(self.min_int, self.max_int) - if self.min_float is not None: - s += ', floats=({}, {})'.format(self.min_float, self.max_float) - if self.min_bool is not None: - s += ', bools=({}, {})'.format(self.min_bool, self.max_bool) + s = 'TypeSet(lanes={}'.format(pp_set(self.lanes)) + if len(self.ints) > 0: + s += ', ints={}'.format(pp_set(self.ints)) + if len(self.floats) > 0: + s += ', floats={}'.format(pp_set(self.floats)) + if len(self.bools) > 0: + s += ', bools={}'.format(pp_set(self.bools)) return s + ')' def emit_fields(self, fmt): # type: (Formatter) -> None """Emit field initializers for this typeset.""" fmt.comment(repr(self)) - fields = ('lanes', 'int', 'float', 'bool') - for field in fields: - min_val = getattr(self, 'min_' + field) - max_val = getattr(self, 'max_' + field) - if min_val is None: - fmt.line('min_{}: 0,'.format(field)) - fmt.line('max_{}: 0,'.format(field)) - else: - fmt.line('min_{}: {},'.format( - field, int_log2(min_val))) - fmt.line('max_{}: {},'.format( - field, int_log2(max_val) + 1)) + + fields = (('lanes', 16), + ('ints', 8), + ('floats', 8), + ('bools', 8)) + + for (field, bits) in fields: + vals = [int_log2(x) for x in getattr(self, field)] + fmt.line('{}: BitSet::({}),' + .format(field, bits, encode_bitset(vals, bits))) def __iand__(self, other): # type: (TypeSet) -> TypeSet @@ -186,32 +222,22 @@ class TypeSet(object): >>> a = TypeSet(lanes=True, ints=(16, 32)) >>> a - TypeSet(lanes=(1, 256), ints=(16, 32)) + TypeSet(lanes={1, 2, 4, 8, 16, 32, 64, 128, 256}, ints={16, 32}) >>> b = TypeSet(lanes=(4, 16), ints=True) >>> a &= b >>> a - TypeSet(lanes=(4, 16), ints=(16, 32)) + TypeSet(lanes={4, 8, 16}, ints={16, 32}) >>> a = TypeSet(lanes=True, bools=(1, 8)) >>> b = TypeSet(lanes=True, bools=(16, 32)) >>> a &= b >>> a - TypeSet(lanes=(1, 256)) + TypeSet(lanes={1, 2, 4, 8, 16, 32, 64, 128, 256}) """ - self.min_lanes = max(self.min_lanes, other.min_lanes) - self.max_lanes = min(self.max_lanes, other.max_lanes) - - self.min_int, self.max_int = intersect( - (self.min_int, self.max_int), - (other.min_int, other.max_int)) - - self.min_float, self.max_float = intersect( - (self.min_float, self.max_float), - (other.min_float, other.max_float)) - - self.min_bool, self.max_bool = intersect( - (self.min_bool, self.max_bool), - (other.min_bool, other.max_bool)) + self.lanes.intersection_update(other.lanes) + self.ints.intersection_update(other.ints) + self.floats.intersection_update(other.floats) + self.bools.intersection_update(other.bools) return self @@ -382,12 +408,12 @@ class TypeVar(object): """ if not self.is_derived: ts = self.type_set - if ts.min_int: - assert ts.min_int > 8, "Can't halve all integer types" - if ts.min_float: - assert ts.min_float > 32, "Can't halve all float types" - if ts.min_bool: - assert ts.min_bool > 8, "Can't halve all boolean types" + if len(ts.ints) > 0: + assert min(ts.ints) > 8, "Can't halve all integer types" + if len(ts.floats) > 0: + assert min(ts.floats) > 32, "Can't halve all float types" + if len(ts.bools) > 0: + assert min(ts.bools) > 8, "Can't halve all boolean types" return TypeVar.derived(self, self.HALFWIDTH) @@ -399,12 +425,14 @@ class TypeVar(object): """ if not self.is_derived: ts = self.type_set - if ts.max_int: - assert ts.max_int < MAX_BITS, "Can't double all integer types." - if ts.max_float: - assert ts.max_float < MAX_BITS, "Can't double all float types." - if ts.max_bool: - assert ts.max_bool < MAX_BITS, "Can't double all bool types." + if len(ts.ints) > 0: + assert max(ts.ints) < MAX_BITS,\ + "Can't double all integer types." + if len(ts.floats) > 0: + assert max(ts.floats) < MAX_BITS,\ + "Can't double all float types." + if len(ts.bools) > 0: + assert max(ts.bools) < MAX_BITS, "Can't double all bool types." return TypeVar.derived(self, self.DOUBLEWIDTH) @@ -416,7 +444,7 @@ class TypeVar(object): """ if not self.is_derived: ts = self.type_set - assert ts.min_lanes > 1, "Can't halve a scalar type" + assert min(ts.lanes) > 1, "Can't halve a scalar type" return TypeVar.derived(self, self.HALFVECTOR) @@ -428,7 +456,7 @@ class TypeVar(object): """ if not self.is_derived: ts = self.type_set - assert ts.max_lanes < 256, "Can't double 256 lanes." + assert max(ts.lanes) < MAX_LANES, "Can't double 256 lanes." return TypeVar.derived(self, self.DOUBLEVECTOR) diff --git a/lib/cretonne/src/bitset.rs b/lib/cretonne/src/bitset.rs new file mode 100644 index 0000000000..cfca8371fb --- /dev/null +++ b/lib/cretonne/src/bitset.rs @@ -0,0 +1,147 @@ +//! Small Bitset +//! +//! This module defines a struct BitSet encapsulating a bitset built over the type T. +//! T is intended to be a primitive unsigned type. Currently it can be any type between u8 and u32 +//! +//! If you would like to add support for larger bitsets in the future, you need to change the trait +//! bound Into and the u32 in the implementation of max_bits() +use std::mem::size_of; +use std::ops::{Shl, BitOr, Sub, Add}; +use std::convert::{Into, From}; + +/// A small bitset built on a single primitive integer type +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct BitSet(pub T); + +impl BitSet + where T: Into + From + BitOr + Shl + Sub + + Add + PartialEq + Copy +{ +/// Maximum number of bits supported by this BitSet instance + pub fn bits() -> usize { + size_of::() * 8 + } + +/// Maximum number of bits supported by any bitset instance atm. + pub fn max_bits() -> usize { + size_of::() * 8 + } + +/// Check if this BitSet contains the number num + pub fn contains(&self, num: u8) -> bool { + assert!((num as usize) < Self::bits()); + assert!((num as usize) < Self::max_bits()); + return self.0.into() & (1 << num) != 0; + } + +/// Return the smallest number contained in the bitset or None if empty + pub fn min(&self) -> Option { + if self.0.into() == 0 { + None + } else { + Some(self.0.into().trailing_zeros() as u8) + } + } + +/// Return the largest number contained in the bitset or None if empty + pub fn max(&self) -> Option { + if self.0.into() == 0 { + None + } else { + let leading_zeroes = self.0.into().leading_zeros() as usize; + Some((Self::max_bits() - leading_zeroes - 1) as u8) + } + } + +/// Construct a BitSet with the half-open range [lo,hi) filled in + pub fn from_range(lo: u8, hi: u8) -> BitSet { + assert!(lo <= hi); + assert!((hi as usize) <= Self::bits()); + let one : T = T::from(1); +// I can't just do (one << hi) - one here as the shift may overflow + let hi_rng = if hi >= 1 { + (one << (hi-1)) + ((one << (hi-1)) - one) + } else { + T::from(0) + }; + + let lo_rng = (one << lo) - one; + + BitSet(hi_rng - lo_rng) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn contains() { + let s = BitSet::(255); + for i in 0..7 { + assert!(s.contains(i)); + } + + let s1 = BitSet::(0); + for i in 0..7 { + assert!(!s1.contains(i)); + } + + let s2 = BitSet::(127); + for i in 0..6 { + assert!(s2.contains(i)); + } + assert!(!s2.contains(7)); + + let s3 = BitSet::(2 | 4 | 64); + assert!(!s3.contains(0) && !s3.contains(3) && !s3.contains(4) && !s3.contains(5) && + !s3.contains(7)); + assert!(s3.contains(1) && s3.contains(2) && s3.contains(6)); + + let s4 = BitSet::(4 | 8 | 256 | 1024); + assert!(!s4.contains(0) && !s4.contains(1) && !s4.contains(4) && !s4.contains(5) && + !s4.contains(6) && !s4.contains(7) && + !s4.contains(9) && !s4.contains(11)); + assert!(s4.contains(2) && s4.contains(3) && s4.contains(8) && s4.contains(10)); + } + + #[test] + fn minmax() { + let s = BitSet::(255); + assert_eq!(s.min(), Some(0)); + assert_eq!(s.max(), Some(7)); + assert!(s.min() == Some(0) && s.max() == Some(7)); + let s1 = BitSet::(0); + assert!(s1.min() == None && s1.max() == None); + let s2 = BitSet::(127); + assert!(s2.min() == Some(0) && s2.max() == Some(6)); + let s3 = BitSet::(2 | 4 | 64); + assert!(s3.min() == Some(1) && s3.max() == Some(6)); + let s4 = BitSet::(4 | 8 | 256 | 1024); + assert!(s4.min() == Some(2) && s4.max() == Some(10)); + } + + #[test] + fn from_range() { + let s = BitSet::::from_range(5, 5); + assert!(s.0 == 0); + + let s = BitSet::::from_range(0, 8); + assert!(s.0 == 255); + + let s = BitSet::::from_range(0, 8); + assert!(s.0 == 255u16); + + let s = BitSet::::from_range(0, 16); + assert!(s.0 == 65535u16); + + let s = BitSet::::from_range(5, 6); + assert!(s.0 == 32u8); + + let s = BitSet::::from_range(3, 7); + assert!(s.0 == 8 | 16 | 32 | 64); + + let s = BitSet::::from_range(5, 11); + assert!(s.0 == 32 | 64 | 128 | 256 | 512 | 1024); + } +} diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index ad4dd6fa21..35a200de08 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -17,6 +17,7 @@ use ir::types; use isa::RegUnit; use entity_list; +use bitset::BitSet; use ref_slice::{ref_slice, ref_slice_mut}; /// Some instructions use an external list of argument values because there is not enough space in @@ -499,17 +500,16 @@ impl OpcodeConstraints { } } +type BitSet8 = BitSet; +type BitSet16 = BitSet; + /// A value type set describes the permitted set of types for a type variable. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct ValueTypeSet { - min_lanes: u8, - max_lanes: u8, - min_int: u8, - max_int: u8, - min_float: u8, - max_float: u8, - min_bool: u8, - max_bool: u8, + lanes: BitSet16, + ints: BitSet8, + floats: BitSet8, + bools: BitSet8, } impl ValueTypeSet { @@ -519,11 +519,11 @@ impl ValueTypeSet { fn is_base_type(&self, scalar: Type) -> bool { let l2b = scalar.log2_lane_bits(); if scalar.is_int() { - self.min_int <= l2b && l2b < self.max_int + self.ints.contains(l2b) } else if scalar.is_float() { - self.min_float <= l2b && l2b < self.max_float + self.floats.contains(l2b) } else if scalar.is_bool() { - self.min_bool <= l2b && l2b < self.max_bool + self.bools.contains(l2b) } else { false } @@ -532,23 +532,23 @@ impl ValueTypeSet { /// Does `typ` belong to this set? pub fn contains(&self, typ: Type) -> bool { let l2l = typ.log2_lane_count(); - self.min_lanes <= l2l && l2l < self.max_lanes && self.is_base_type(typ.lane_type()) + self.lanes.contains(l2l) && self.is_base_type(typ.lane_type()) } /// Get an example member of this type set. /// /// This is used for error messages to avoid suggesting invalid types. pub fn example(&self) -> Type { - let t = if self.max_int > 5 { + let t = if self.ints.max().unwrap_or(0) > 5 { types::I32 - } else if self.max_float > 5 { + } else if self.floats.max().unwrap_or(0) > 5 { types::F32 - } else if self.max_bool > 5 { + } else if self.bools.max().unwrap_or(0) > 5 { types::B32 } else { types::B1 }; - t.by(1 << self.min_lanes).unwrap() + t.by(1 << self.lanes.min().unwrap()).unwrap() } } @@ -709,15 +709,12 @@ mod tests { use ir::types::*; let vts = ValueTypeSet { - min_lanes: 0, - max_lanes: 8, - min_int: 3, - max_int: 7, - min_float: 0, - max_float: 0, - min_bool: 3, - max_bool: 7, + lanes: BitSet16::from_range(0, 8), + ints: BitSet8::from_range(4, 7), + floats: BitSet8::from_range(0, 0), + bools: BitSet8::from_range(3, 7), }; + assert!(!vts.contains(I8)); assert!(vts.contains(I32)); assert!(vts.contains(I64)); assert!(vts.contains(I32X4)); @@ -728,38 +725,26 @@ mod tests { assert_eq!(vts.example().to_string(), "i32"); let vts = ValueTypeSet { - min_lanes: 0, - max_lanes: 8, - min_int: 0, - max_int: 0, - min_float: 5, - max_float: 7, - min_bool: 3, - max_bool: 7, + lanes: BitSet16::from_range(0, 8), + ints: BitSet8::from_range(0, 0), + floats: BitSet8::from_range(5, 7), + bools: BitSet8::from_range(3, 7), }; assert_eq!(vts.example().to_string(), "f32"); let vts = ValueTypeSet { - min_lanes: 1, - max_lanes: 8, - min_int: 0, - max_int: 0, - min_float: 5, - max_float: 7, - min_bool: 3, - max_bool: 7, + lanes: BitSet16::from_range(1, 8), + ints: BitSet8::from_range(0, 0), + floats: BitSet8::from_range(5, 7), + bools: BitSet8::from_range(3, 7), }; assert_eq!(vts.example().to_string(), "f32x2"); let vts = ValueTypeSet { - min_lanes: 2, - max_lanes: 8, - min_int: 0, - max_int: 0, - min_float: 0, - max_float: 0, - min_bool: 3, - max_bool: 7, + lanes: BitSet16::from_range(2, 8), + ints: BitSet8::from_range(0, 0), + floats: BitSet8::from_range(0, 0), + bools: BitSet8::from_range(3, 7), }; assert!(!vts.contains(B32X2)); assert!(vts.contains(B32X4)); @@ -767,14 +752,10 @@ mod tests { let vts = ValueTypeSet { // TypeSet(lanes=(1, 256), ints=(8, 64)) - min_lanes: 0, - max_lanes: 9, - min_int: 3, - max_int: 7, - min_float: 0, - max_float: 0, - min_bool: 0, - max_bool: 0, + lanes: BitSet16::from_range(0, 9), + ints: BitSet8::from_range(3, 7), + floats: BitSet8::from_range(0, 0), + bools: BitSet8::from_range(0, 0), }; assert!(vts.contains(I32)); assert!(vts.contains(I32X4)); diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 09bdfc9302..08c1c4354b 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -16,6 +16,7 @@ pub mod dbg; pub mod entity_ref; pub mod binemit; +pub mod bitset; pub mod dominator_tree; pub mod entity_list; pub mod entity_map; From a79703c23f35190f83053ad2fa7eb02425656896 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 22 Jun 2017 14:35:05 -0700 Subject: [PATCH 0818/3084] Add rusty-tags.* to .gitignore. --- cranelift/.gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/cranelift/.gitignore b/cranelift/.gitignore index a230556d47..14318728b6 100644 --- a/cranelift/.gitignore +++ b/cranelift/.gitignore @@ -8,3 +8,4 @@ Cargo.lock .*.rustfmt cretonne.dbg* .mypy_cache +rusty-tags.* From 222ea748303961bea1c6d5e6e32e910210da58bf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 23 Jun 2017 10:43:59 -0700 Subject: [PATCH 0819/3084] Spill whole virtual registers at a time. When the spiller decides to spill a value, bring along all of the values in its virtual register. This ensures that we won't have problems with computing register pressure around EBB arguments. They will always be register-to-register or stack-to-stack with related values using the same stack slot. This also means that the reloading pass won't have to deal with spilled EBB arguments. --- lib/cretonne/src/regalloc/context.rs | 11 ++++++----- lib/cretonne/src/regalloc/spilling.rs | 18 ++++++++++++------ 2 files changed, 18 insertions(+), 11 deletions(-) diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 78e1595747..0908b02a1c 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -66,14 +66,14 @@ impl Context { // phases. self.tracker.clear(); - // First pass: Liveness analysis. + // Pass: Liveness analysis. self.liveness.compute(isa, func, cfg); if isa.flags().enable_verifier() { verify_liveness(isa, func, cfg, &self.liveness)?; } - // Coalesce and create conventional SSA form. + // Pass: Coalesce and create conventional SSA form. self.coalescing .conventional_ssa(isa, func, @@ -88,12 +88,13 @@ impl Context { } - // Second pass: Spilling. + // Pass: Spilling. self.spilling .run(isa, func, domtree, &mut self.liveness, + &self.virtregs, &mut self.topo, &mut self.tracker); @@ -102,7 +103,7 @@ impl Context { verify_liveness(isa, func, cfg, &self.liveness)?; } - // Third pass: Reload. + // Pass: Reload. self.reload .run(isa, func, @@ -116,7 +117,7 @@ impl Context { verify_liveness(isa, func, cfg, &self.liveness)?; } - // Fourth pass: Coloring. + // Pass: Coloring. self.coloring .run(isa, func, diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 7311de5d5d..f1feff3f4b 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -25,6 +25,7 @@ use regalloc::affinity::Affinity; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; use regalloc::pressure::Pressure; +use regalloc::virtregs::VirtRegs; use topo_order::TopoOrder; /// Persistent data structures for the spilling pass. @@ -48,6 +49,7 @@ struct Context<'a> { // References to contextual data structures we need. domtree: &'a DominatorTree, liveness: &'a mut Liveness, + virtregs: &'a VirtRegs, topo: &'a mut TopoOrder, // Current register pressure. @@ -77,6 +79,7 @@ impl Spilling { func: &mut Function, domtree: &DominatorTree, liveness: &mut Liveness, + virtregs: &VirtRegs, topo: &mut TopoOrder, tracker: &mut LiveValueTracker) { dbg!("Spilling for:\n{}", func.display(isa)); @@ -91,6 +94,7 @@ impl Spilling { locations: &mut func.locations, domtree, liveness, + virtregs, topo, pressure: Pressure::new(®info, &usable_regs), spills: &mut self.spills, @@ -374,15 +378,17 @@ impl<'a> Context<'a> { let rc = self.reginfo.rc(rci); self.pressure.free(rc); self.spills.push(value); - - // Assign a spill slot. - // TODO: phi-related values should use the same spill slot. - let ss = self.stack_slots.make_spill_slot(dfg.value_type(value)); - *self.locations.ensure(value) = ValueLoc::Stack(ss); - dbg!("Spilled {}:{} to {} -> {}", value, rc, ss, self.pressure); + dbg!("Spilled {}:{} -> {}", value, rc, self.pressure); } else { panic!("Cannot spill {} that was already on the stack", value); } + + // Assign a spill slot for the whole virtual register. + let ss = self.stack_slots.make_spill_slot(dfg.value_type(value)); + for &v in self.virtregs.congruence_class(&value) { + self.liveness.spill(v); + *self.locations.ensure(v) = ValueLoc::Stack(ss); + } } /// Process any pending spills in the `self.spills` vector. From 7c298078c8de84207e61f54957ba7f36c86f5618 Mon Sep 17 00:00:00 2001 From: d1m0 Date: Fri, 23 Jun 2017 11:57:24 -0700 Subject: [PATCH 0820/3084] Add image computation of typesets; Remove TypeVar.singleton_type - instead derive singleton type from typeset; (#104) --- lib/cretonne/meta/base/types.py | 9 +- lib/cretonne/meta/cdsl/instructions.py | 2 +- lib/cretonne/meta/cdsl/test_typevar.py | 80 ++++++++- lib/cretonne/meta/cdsl/types.py | 28 +++- lib/cretonne/meta/cdsl/typevar.py | 218 +++++++++++++++++++++---- lib/cretonne/meta/cdsl/xform.py | 2 +- lib/cretonne/meta/gen_instr.py | 4 +- 7 files changed, 294 insertions(+), 49 deletions(-) diff --git a/lib/cretonne/meta/base/types.py b/lib/cretonne/meta/base/types.py index a2eb1054b9..7111626009 100644 --- a/lib/cretonne/meta/base/types.py +++ b/lib/cretonne/meta/base/types.py @@ -2,15 +2,10 @@ The base.types module predefines all the Cretonne scalar types. """ from __future__ import absolute_import -from cdsl.types import ScalarType, IntType, FloatType, BoolType +from cdsl.types import IntType, FloatType, BoolType #: Boolean. -b1 = ScalarType( - 'b1', 0, - """ - A boolean value that is either true or false. - """) - +b1 = BoolType(1) #: 1-bit bool. Type is abstract (can't be stored in mem) b8 = BoolType(8) #: 8-bit bool. b16 = BoolType(16) #: 16-bit bool. b32 = BoolType(32) #: 32-bit bool. diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index d8e9d24e5f..22c989bd65 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -186,7 +186,7 @@ class Instruction(object): try: opnum = self.value_opnums[self.format.typevar_operand] tv = self.ins[opnum].typevar - if tv is tv.free_typevar(): + if tv is tv.free_typevar() or tv.singleton_type() is not None: self.other_typevars = self._verify_ctrl_typevar(tv) self.ctrl_typevar = tv self.use_typevar_operand = True diff --git a/lib/cretonne/meta/cdsl/test_typevar.py b/lib/cretonne/meta/cdsl/test_typevar.py index 97793f71a5..5da081eabe 100644 --- a/lib/cretonne/meta/cdsl/test_typevar.py +++ b/lib/cretonne/meta/cdsl/test_typevar.py @@ -3,7 +3,7 @@ from unittest import TestCase from doctest import DocTestSuite from . import typevar from .typevar import TypeSet, TypeVar -from base.types import i32 +from base.types import i32, i16, b1, f64 def load_tests(loader, tests, ignore): @@ -45,6 +45,84 @@ class TestTypeSet(TestCase): with self.assertRaises(AssertionError): a in s + def test_forward_images(self): + a = TypeSet(lanes=(2, 8), ints=(8, 8), floats=(32, 32)) + b = TypeSet(lanes=(1, 8), ints=(8, 8), floats=(32, 32)) + self.assertEqual(a.lane_of(), TypeSet(ints=(8, 8), floats=(32, 32))) + + c = TypeSet(lanes=(2, 8)) + c.bools = set([8, 32]) + + # Test case with disjoint intervals + self.assertEqual(a.as_bool(), c) + + # For as_bool check b1 is present when 1 \in lanes + d = TypeSet(lanes=(1, 8)) + d.bools = set([1, 8, 32]) + self.assertEqual(b.as_bool(), d) + + self.assertEqual(TypeSet(lanes=(1, 32)).half_vector(), + TypeSet(lanes=(1, 16))) + + self.assertEqual(TypeSet(lanes=(1, 32)).double_vector(), + TypeSet(lanes=(2, 64))) + + self.assertEqual(TypeSet(lanes=(128, 256)).double_vector(), + TypeSet(lanes=(256, 256))) + + self.assertEqual(TypeSet(ints=(8, 32)).half_width(), + TypeSet(ints=(8, 16))) + + self.assertEqual(TypeSet(ints=(8, 32)).double_width(), + TypeSet(ints=(16, 64))) + + self.assertEqual(TypeSet(ints=(32, 64)).double_width(), + TypeSet(ints=(64, 64))) + + # Should produce an empty ts + self.assertEqual(TypeSet(floats=(32, 32)).half_width(), + TypeSet()) + + self.assertEqual(TypeSet(floats=(32, 64)).half_width(), + TypeSet(floats=(32, 32))) + + self.assertEqual(TypeSet(floats=(32, 32)).double_width(), + TypeSet(floats=(64, 64))) + + self.assertEqual(TypeSet(floats=(32, 64)).double_width(), + TypeSet(floats=(64, 64))) + + # Bools have trickier behavior around b1 (since b2, b4 don't exist) + self.assertEqual(TypeSet(bools=(1, 8)).half_width(), + TypeSet()) + + t = TypeSet() + t.bools = set([8, 16]) + self.assertEqual(TypeSet(bools=(1, 32)).half_width(), t) + + # double_width() of bools={1, 8, 16} must not include 2 or 8 + t.bools = set([16, 32]) + self.assertEqual(TypeSet(bools=(1, 16)).double_width(), t) + + self.assertEqual(TypeSet(bools=(32, 64)).double_width(), + TypeSet(bools=(64, 64))) + + def test_get_singleton(self): + # Raise error when calling get_singleton() on non-singleton TS + t = TypeSet(lanes=(1, 1), ints=(8, 8), floats=(32, 32)) + with self.assertRaises(AssertionError): + t.get_singleton() + t = TypeSet(lanes=(1, 2), floats=(32, 32)) + + with self.assertRaises(AssertionError): + t.get_singleton() + + self.assertEqual(TypeSet(ints=(16, 16)).get_singleton(), i16) + self.assertEqual(TypeSet(floats=(64, 64)).get_singleton(), f64) + self.assertEqual(TypeSet(bools=(1, 1)).get_singleton(), b1) + self.assertEqual(TypeSet(lanes=(4, 4), ints=(32, 32)).get_singleton(), + i32.by(4)) + class TestTypeVar(TestCase): def test_functions(self): diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py index e42460af28..6feb2112d8 100644 --- a/lib/cretonne/meta/cdsl/types.py +++ b/lib/cretonne/meta/cdsl/types.py @@ -97,7 +97,7 @@ class VectorType(ValueType): # type: (ScalarType, int) -> None assert isinstance(base, ScalarType), 'SIMD lanes must be scalar types' super(VectorType, self).__init__( - name='{}x{}'.format(base.name, lanes), + name=VectorType.get_name(base, lanes), membytes=lanes*base.membytes, doc=""" A SIMD vector with {} lanes containing a `{}` each. @@ -111,6 +111,11 @@ class VectorType(ValueType): return ('VectorType(base={}, lanes={})' .format(self.base.name, self.lanes)) + @staticmethod + def get_name(base, lanes): + # type: (ValueType, int) -> str + return '{}x{}'.format(base.name, lanes) + class IntType(ScalarType): """A concrete scalar integer type.""" @@ -119,7 +124,7 @@ class IntType(ScalarType): # type: (int) -> None assert bits > 0, 'IntType must have positive number of bits' super(IntType, self).__init__( - name='i{:d}'.format(bits), + name=IntType.get_name(bits), membytes=bits // 8, doc="An integer type with {} bits.".format(bits)) self.bits = bits @@ -128,6 +133,11 @@ class IntType(ScalarType): # type: () -> str return 'IntType(bits={})'.format(self.bits) + @staticmethod + def get_name(bits): + # type: (int) -> str + return 'i{:d}'.format(bits) + class FloatType(ScalarType): """A concrete scalar floating point type.""" @@ -136,7 +146,7 @@ class FloatType(ScalarType): # type: (int, str) -> None assert bits > 0, 'FloatType must have positive number of bits' super(FloatType, self).__init__( - name='f{:d}'.format(bits), + name=FloatType.get_name(bits), membytes=bits // 8, doc=doc) self.bits = bits @@ -145,6 +155,11 @@ class FloatType(ScalarType): # type: () -> str return 'FloatType(bits={})'.format(self.bits) + @staticmethod + def get_name(bits): + # type: (int) -> str + return 'f{:d}'.format(bits) + class BoolType(ScalarType): """A concrete scalar boolean type.""" @@ -153,7 +168,7 @@ class BoolType(ScalarType): # type: (int) -> None assert bits > 0, 'BoolType must have positive number of bits' super(BoolType, self).__init__( - name='b{:d}'.format(bits), + name=BoolType.get_name(bits), membytes=bits // 8, doc="A boolean type with {} bits.".format(bits)) self.bits = bits @@ -161,3 +176,8 @@ class BoolType(ScalarType): def __repr__(self): # type: () -> str return 'BoolType(bits={})'.format(self.bits) + + @staticmethod + def get_name(bits): + # type: (int) -> str + return 'b{:d}'.format(bits) diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 119c6bdf01..6ed3ffc86c 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -7,15 +7,19 @@ polymorphic by using type variables. from __future__ import absolute_import import math from . import types, is_power_of_two +from copy import deepcopy +from .types import ValueType, IntType, FloatType, BoolType try: from typing import Tuple, Union, Iterable, Any, Set, TYPE_CHECKING # noqa + from typing import cast if TYPE_CHECKING: from srcgen import Formatter # noqa Interval = Tuple[int, int] # An Interval where `True` means 'everything' BoolInterval = Union[bool, Interval] except ImportError: + TYPE_CHECKING = False pass MAX_LANES = 256 @@ -112,6 +116,16 @@ def interval_to_set(intv): return set([2**i for i in range(int_log2(lo), int_log2(hi)+1)]) +def legal_bool(bits): + # type: (int) -> bool + """ + True iff bits is a legal bit width for a bool type. + bits == 1 || bits \in { 8, 16, .. MAX_BITS } + """ + return bits == 1 or \ + (bits >= 8 and bits <= MAX_BITS and is_power_of_two(bits)) + + class TypeSet(object): """ A set of types. @@ -165,7 +179,15 @@ class TypeSet(object): self.ints = interval_to_set(decode_interval(ints, (8, MAX_BITS))) self.floats = interval_to_set(decode_interval(floats, (32, 64))) self.bools = interval_to_set(decode_interval(bools, (1, MAX_BITS))) - self.bools = set(filter(lambda x: x == 1 or x >= 8, self.bools)) + self.bools = set(filter(legal_bool, self.bools)) + + def copy(self): + # type: (TypeSet) -> TypeSet + """ + Return a copy of our self. deepcopy is sufficient and safe here, since + TypeSet contains only sets of numbers. + """ + return deepcopy(self) def typeset_key(self): # type: () -> Tuple[Tuple, Tuple, Tuple, Tuple] @@ -241,6 +263,109 @@ class TypeSet(object): return self + def lane_of(self): + # type: () -> TypeSet + """ + Return a TypeSet describing the image of self across lane_of + """ + new = self.copy() + new.lanes = set([1]) + return new + + def as_bool(self): + # type: () -> TypeSet + """ + Return a TypeSet describing the image of self across as_bool + """ + new = self.copy() + new.ints = set() + new.floats = set() + new.bools = self.ints.union(self.floats).union(self.bools) + + if 1 in self.lanes: + new.bools.add(1) + return new + + def half_width(self): + # type: () -> TypeSet + """ + Return a TypeSet describing the image of self across halfwidth + """ + new = self.copy() + new.ints = set([x/2 for x in self.ints if x > 8]) + new.floats = set([x/2 for x in self.floats if x > 32]) + new.bools = set([x/2 for x in self.bools if x > 8]) + + return new + + def double_width(self): + # type: () -> TypeSet + """ + Return a TypeSet describing the image of self across doublewidth + """ + new = self.copy() + new.ints = set([x*2 for x in self.ints if x < MAX_BITS]) + new.floats = set([x*2 for x in self.floats if x < MAX_BITS]) + new.bools = set(filter(legal_bool, + set([x*2 for x in self.bools if x < MAX_BITS]))) + + return new + + def half_vector(self): + # type: () -> TypeSet + """ + Return a TypeSet describing the image of self across halfvector + """ + new = self.copy() + new.lanes = set([x/2 for x in self.lanes if x > 1]) + + return new + + def double_vector(self): + # type: () -> TypeSet + """ + Return a TypeSet describing the image of self across doublevector + """ + new = self.copy() + new.lanes = set([x*2 for x in self.lanes if x < MAX_LANES]) + + return new + + def size(self): + # type: () -> int + """ + Return the number of concrete types represented by this typeset + """ + return len(self.lanes) * (len(self.ints) + len(self.floats) + + len(self.bools)) + + def get_singleton(self): + # type: () -> types.ValueType + """ + Return the singleton type represented by self. Can only call on + typesets containing 1 type. + """ + assert self.size() == 1 + if len(self.ints) > 0: + bits = tuple(self.ints)[0] + scalar_type = ValueType.by_name(IntType.get_name(bits)) + elif len(self.floats) > 0: + bits = tuple(self.floats)[0] + scalar_type = ValueType.by_name(FloatType.get_name(bits)) + else: + bits = tuple(self.bools)[0] + scalar_type = ValueType.by_name(BoolType.get_name(bits)) + + nlanes = tuple(self.lanes)[0] + + if nlanes == 1: + return scalar_type + else: + if TYPE_CHECKING: + return cast(types.ScalarType, scalar_type).by(nlanes) + else: + return scalar_type.by(nlanes) + class TypeVar(object): """ @@ -271,7 +396,6 @@ class TypeVar(object): # type: (str, str, BoolInterval, BoolInterval, BoolInterval, bool, BoolInterval, TypeVar, str) -> None # noqa self.name = name self.__doc__ = doc - self.singleton_type = None # type: types.ValueType self.is_derived = isinstance(base, TypeVar) if base: assert self.is_derived @@ -313,7 +437,6 @@ class TypeVar(object): tv = TypeVar( typ.name, typ.__doc__, ints, floats, bools, simd=lanes) - tv.singleton_type = typ return tv def __str__(self): @@ -406,14 +529,13 @@ class TypeVar(object): Return a derived type variable that has the same number of vector lanes as this one, but the lanes are half the width. """ - if not self.is_derived: - ts = self.type_set - if len(ts.ints) > 0: - assert min(ts.ints) > 8, "Can't halve all integer types" - if len(ts.floats) > 0: - assert min(ts.floats) > 32, "Can't halve all float types" - if len(ts.bools) > 0: - assert min(ts.bools) > 8, "Can't halve all boolean types" + ts = self.get_typeset() + if len(ts.ints) > 0: + assert min(ts.ints) > 8, "Can't halve all integer types" + if len(ts.floats) > 0: + assert min(ts.floats) > 32, "Can't halve all float types" + if len(ts.bools) > 0: + assert min(ts.bools) > 8, "Can't halve all boolean types" return TypeVar.derived(self, self.HALFWIDTH) @@ -423,16 +545,13 @@ class TypeVar(object): Return a derived type variable that has the same number of vector lanes as this one, but the lanes are double the width. """ - if not self.is_derived: - ts = self.type_set - if len(ts.ints) > 0: - assert max(ts.ints) < MAX_BITS,\ - "Can't double all integer types." - if len(ts.floats) > 0: - assert max(ts.floats) < MAX_BITS,\ - "Can't double all float types." - if len(ts.bools) > 0: - assert max(ts.bools) < MAX_BITS, "Can't double all bool types." + ts = self.get_typeset() + if len(ts.ints) > 0: + assert max(ts.ints) < MAX_BITS, "Can't double all integer types." + if len(ts.floats) > 0: + assert max(ts.floats) < MAX_BITS, "Can't double all float types." + if len(ts.bools) > 0: + assert max(ts.bools) < MAX_BITS, "Can't double all bool types." return TypeVar.derived(self, self.DOUBLEWIDTH) @@ -442,9 +561,8 @@ class TypeVar(object): Return a derived type variable that has half the number of vector lanes as this one, with the same lane type. """ - if not self.is_derived: - ts = self.type_set - assert min(ts.lanes) > 1, "Can't halve a scalar type" + ts = self.get_typeset() + assert min(ts.lanes) > 1, "Can't halve a scalar type" return TypeVar.derived(self, self.HALFVECTOR) @@ -454,12 +572,23 @@ class TypeVar(object): Return a derived type variable that has twice the number of vector lanes as this one, with the same lane type. """ - if not self.is_derived: - ts = self.type_set - assert max(ts.lanes) < MAX_LANES, "Can't double 256 lanes." + ts = self.get_typeset() + assert max(ts.lanes) < MAX_LANES, "Can't double 256 lanes." return TypeVar.derived(self, self.DOUBLEVECTOR) + def singleton_type(self): + # type: () -> ValueType + """ + If the associated typeset has a single type return it. Otherwise return + None + """ + ts = self.get_typeset() + if ts.size() != 1: + return None + + return ts.get_singleton() + def free_typevar(self): # type: () -> TypeVar """ @@ -467,7 +596,7 @@ class TypeVar(object): """ if self.is_derived: return self.base - elif self.singleton_type: + elif self.singleton_type() is not None: # A singleton type variable is not a proper free variable. return None else: @@ -481,8 +610,8 @@ class TypeVar(object): if self.is_derived: return '{}.{}()'.format( self.base.rust_expr(), self.derived_func) - elif self.singleton_type: - return self.singleton_type.rust_name() + elif self.singleton_type(): + return self.singleton_type().rust_name() else: return self.name @@ -501,9 +630,6 @@ class TypeVar(object): if not a.is_derived and not b.is_derived: a.type_set &= b.type_set - # TODO: What if a.type_set becomes empty? - if not a.singleton_type: - a.singleton_type = b.singleton_type return # TODO: Implement constraints for derived type variables. @@ -514,3 +640,29 @@ class TypeVar(object): # # For the fully general case, we would need to compute an image typeset # for `b` and propagate a `a.derived_func` pre-image to `a.base`. + + def get_typeset(self): + # type: () -> TypeSet + """ + Returns the typeset for this TV. If the TV is derived, computes it + recursively from the derived function and the base's typeset. + """ + if not self.is_derived: + return self.type_set + else: + if (self.derived_func == TypeVar.SAMEAS): + return self.base.get_typeset() + elif (self.derived_func == TypeVar.LANEOF): + return self.base.get_typeset().lane_of() + elif (self.derived_func == TypeVar.ASBOOL): + return self.base.get_typeset().as_bool() + elif (self.derived_func == TypeVar.HALFWIDTH): + return self.base.get_typeset().half_width() + elif (self.derived_func == TypeVar.DOUBLEWIDTH): + return self.base.get_typeset().double_width() + elif (self.derived_func == TypeVar.HALFVECTOR): + return self.base.get_typeset().half_vector() + elif (self.derived_func == TypeVar.DOUBLEVECTOR): + return self.base.get_typeset().double_vector() + else: + assert False, "Unknown derived function: " + self.derived_func diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index 9d284bc1e2..9ce93c9ed9 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -254,7 +254,7 @@ class XForm(object): # Some variables have a fixed type which appears as a type variable # with a singleton_type field set. That's allowed for temps too. for v in fvars: - if v.is_temp() and not v.typevar.singleton_type: + if v.is_temp() and not v.typevar.singleton_type(): raise AssertionError( "Cannot determine type of temp '{}' in xform:\n{}" .format(v, self)) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 548cd45a29..2d67311ae2 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -321,8 +321,8 @@ def get_constraint(op, ctrl_typevar, type_sets): tv = op.typevar # A concrete value type. - if tv.singleton_type: - return 'Concrete({})'.format(tv.singleton_type.rust_name()) + if tv.singleton_type(): + return 'Concrete({})'.format(tv.singleton_type().rust_name()) if tv.free_typevar() is not ctrl_typevar: assert not tv.is_derived From c073d919f429b38f0845195a9e42dd49874e9b15 Mon Sep 17 00:00:00 2001 From: Dimo Date: Fri, 23 Jun 2017 17:46:05 -0700 Subject: [PATCH 0821/3084] Cleanup ValueType.get_names to with_bits form previous PR; Add computation of inverse image of typeset across a derived function - TypeSet.map_inverse; Change TypeVar.constrain_type to perform a more-general computation using inverse images of TypeSets; Tests for map_inverse; --- lib/cretonne/meta/cdsl/test_typevar.py | 108 ++++++++++++++++++++ lib/cretonne/meta/cdsl/types.py | 46 +++++---- lib/cretonne/meta/cdsl/typevar.py | 131 +++++++++++++++++++------ 3 files changed, 237 insertions(+), 48 deletions(-) diff --git a/lib/cretonne/meta/cdsl/test_typevar.py b/lib/cretonne/meta/cdsl/test_typevar.py index 5da081eabe..f655c8966d 100644 --- a/lib/cretonne/meta/cdsl/test_typevar.py +++ b/lib/cretonne/meta/cdsl/test_typevar.py @@ -4,6 +4,8 @@ from doctest import DocTestSuite from . import typevar from .typevar import TypeSet, TypeVar from base.types import i32, i16, b1, f64 +from itertools import product +from functools import reduce def load_tests(loader, tests, ignore): @@ -123,6 +125,58 @@ class TestTypeSet(TestCase): self.assertEqual(TypeSet(lanes=(4, 4), ints=(32, 32)).get_singleton(), i32.by(4)) + def test_map_inverse(self): + t = TypeSet(lanes=(1, 1), ints=(8, 8), floats=(32, 32)) + self.assertEqual(t, t.map_inverse(TypeVar.SAMEAS)) + + # LANEOF + self.assertEqual(TypeSet(lanes=True, ints=(8, 8), floats=(32, 32)), + t.map_inverse(TypeVar.LANEOF)) + # Inverse of empty set is still empty across LANEOF + self.assertEqual(TypeSet(), + TypeSet().map_inverse(TypeVar.LANEOF)) + + # ASBOOL + t = TypeSet(lanes=(1, 4), bools=(1, 64)) + self.assertEqual(t.map_inverse(TypeVar.ASBOOL), + TypeSet(lanes=(1, 4), ints=True, bools=True, + floats=True)) + + # Inverse image across ASBOOL of TS not involving b1 cannot have + # lanes=1 + t = TypeSet(lanes=(1, 4), bools=(16, 32)) + self.assertEqual(t.map_inverse(TypeVar.ASBOOL), + TypeSet(lanes=(2, 4), ints=(16, 32), bools=(16, 32), + floats=(32, 32))) + + # Half/Double Vector + t = TypeSet(lanes=(1, 1), ints=(8, 8)) + t1 = TypeSet(lanes=(256, 256), ints=(8, 8)) + self.assertEqual(t.map_inverse(TypeVar.DOUBLEVECTOR).size(), 0) + self.assertEqual(t1.map_inverse(TypeVar.HALFVECTOR).size(), 0) + + t = TypeSet(lanes=(1, 16), ints=(8, 16), floats=(32, 32)) + t1 = TypeSet(lanes=(64, 256), bools=(1, 32)) + + self.assertEqual(t.map_inverse(TypeVar.DOUBLEVECTOR), + TypeSet(lanes=(1, 8), ints=(8, 16), floats=(32, 32))) + self.assertEqual(t1.map_inverse(TypeVar.HALFVECTOR), + TypeSet(lanes=(128, 256), bools=(1, 32))) + + # Half/Double Width + t = TypeSet(ints=(8, 8), floats=(32, 32), bools=(1, 8)) + t1 = TypeSet(ints=(64, 64), floats=(64, 64), bools=(64, 64)) + self.assertEqual(t.map_inverse(TypeVar.DOUBLEWIDTH).size(), 0) + self.assertEqual(t1.map_inverse(TypeVar.HALFWIDTH).size(), 0) + + t = TypeSet(lanes=(1, 16), ints=(8, 16), floats=(32, 64)) + t1 = TypeSet(lanes=(64, 256), bools=(1, 64)) + + self.assertEqual(t.map_inverse(TypeVar.DOUBLEWIDTH), + TypeSet(lanes=(1, 16), ints=(8, 8), floats=(32, 32))) + self.assertEqual(t1.map_inverse(TypeVar.HALFWIDTH), + TypeSet(lanes=(64, 256), bools=(16, 64))) + class TestTypeVar(TestCase): def test_functions(self): @@ -164,3 +218,57 @@ class TestTypeVar(TestCase): self.assertEqual(max(x.type_set.lanes), 4) self.assertEqual(len(x.type_set.floats), 0) self.assertEqual(len(x.type_set.bools), 0) + + def test_stress_constrain_types(self): + # Get all 49 possible derived vars of lentgh 2. Since we have SAMEAS + # this includes singly derived and non-derived vars + funcs = [TypeVar.SAMEAS, TypeVar.LANEOF, + TypeVar.ASBOOL, TypeVar.HALFVECTOR, TypeVar.DOUBLEVECTOR, + TypeVar.HALFWIDTH, TypeVar.DOUBLEWIDTH] + v = list(product(*[funcs, funcs])) + + # For each pair of derived variables + for (i1, i2) in product(v, v): + # Compute the derived sets for each starting with a full typeset + full_ts = TypeSet(lanes=True, floats=True, ints=True, bools=True) + ts1 = reduce(lambda ts, func: ts.map(func), i1, full_ts) + ts2 = reduce(lambda ts, func: ts.map(func), i2, full_ts) + + # Compute intersection + intersect = ts1.copy() + intersect &= ts2 + + # Propagate instersections backward + ts1_src = reduce(lambda ts, func: ts.map_inverse(func), + reversed(i1), + intersect) + ts2_src = reduce(lambda ts, func: ts.map_inverse(func), + reversed(i2), + intersect) + + # If the intersection or its propagated forms are empty, then these + # two variables can never overlap. For example x.double_vector and + # x.lane_of. + if (intersect.size() == 0 or ts1_src.size() == 0 or + ts2_src.size() == 0): + continue + + # Should be safe to create derived tvs from ts1_src and ts2_src + tv1 = reduce(lambda tv, func: TypeVar.derived(tv, func), + i1, + TypeVar.from_typeset(ts1_src)) + + tv2 = reduce(lambda tv, func: TypeVar.derived(tv, func), + i2, + TypeVar.from_typeset(ts2_src)) + + # The typesets of the two derived variables should be subsets of + # the intersection we computed originally + assert tv1.get_typeset().issubset(intersect) + assert tv2.get_typeset().issubset(intersect) + + # In the absence of AS_BOOL map(map_inverse(f)) == f so the + # typesets of tv1 and tv2 should be exactly intersection + assert (tv1.get_typeset() == tv2.get_typeset() and + tv1.get_typeset() == intersect) or\ + TypeVar.ASBOOL in set(i1 + i2) diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py index 6feb2112d8..eeb5325ae8 100644 --- a/lib/cretonne/meta/cdsl/types.py +++ b/lib/cretonne/meta/cdsl/types.py @@ -3,8 +3,9 @@ from __future__ import absolute_import import math try: - from typing import Dict, List # noqa + from typing import Dict, List, cast, TYPE_CHECKING # noqa except ImportError: + TYPE_CHECKING = False pass @@ -97,7 +98,7 @@ class VectorType(ValueType): # type: (ScalarType, int) -> None assert isinstance(base, ScalarType), 'SIMD lanes must be scalar types' super(VectorType, self).__init__( - name=VectorType.get_name(base, lanes), + name='{}x{}'.format(base.name, lanes), membytes=lanes*base.membytes, doc=""" A SIMD vector with {} lanes containing a `{}` each. @@ -111,11 +112,6 @@ class VectorType(ValueType): return ('VectorType(base={}, lanes={})' .format(self.base.name, self.lanes)) - @staticmethod - def get_name(base, lanes): - # type: (ValueType, int) -> str - return '{}x{}'.format(base.name, lanes) - class IntType(ScalarType): """A concrete scalar integer type.""" @@ -124,7 +120,7 @@ class IntType(ScalarType): # type: (int) -> None assert bits > 0, 'IntType must have positive number of bits' super(IntType, self).__init__( - name=IntType.get_name(bits), + name='i{:d}'.format(bits), membytes=bits // 8, doc="An integer type with {} bits.".format(bits)) self.bits = bits @@ -134,9 +130,13 @@ class IntType(ScalarType): return 'IntType(bits={})'.format(self.bits) @staticmethod - def get_name(bits): - # type: (int) -> str - return 'i{:d}'.format(bits) + def with_bits(bits): + # type: (int) -> IntType + typ = ValueType.by_name('i{:d}'.format(bits)) + if TYPE_CHECKING: + return cast(IntType, typ) + else: + return typ class FloatType(ScalarType): @@ -146,7 +146,7 @@ class FloatType(ScalarType): # type: (int, str) -> None assert bits > 0, 'FloatType must have positive number of bits' super(FloatType, self).__init__( - name=FloatType.get_name(bits), + name='f{:d}'.format(bits), membytes=bits // 8, doc=doc) self.bits = bits @@ -156,9 +156,13 @@ class FloatType(ScalarType): return 'FloatType(bits={})'.format(self.bits) @staticmethod - def get_name(bits): - # type: (int) -> str - return 'f{:d}'.format(bits) + def with_bits(bits): + # type: (int) -> FloatType + typ = ValueType.by_name('f{:d}'.format(bits)) + if TYPE_CHECKING: + return cast(FloatType, typ) + else: + return typ class BoolType(ScalarType): @@ -168,7 +172,7 @@ class BoolType(ScalarType): # type: (int) -> None assert bits > 0, 'BoolType must have positive number of bits' super(BoolType, self).__init__( - name=BoolType.get_name(bits), + name='b{:d}'.format(bits), membytes=bits // 8, doc="A boolean type with {} bits.".format(bits)) self.bits = bits @@ -178,6 +182,10 @@ class BoolType(ScalarType): return 'BoolType(bits={})'.format(self.bits) @staticmethod - def get_name(bits): - # type: (int) -> str - return 'b{:d}'.format(bits) + def with_bits(bits): + # type: (int) -> BoolType + typ = ValueType.by_name('b{:d}'.format(bits)) + if TYPE_CHECKING: + return cast(BoolType, typ) + else: + return typ diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 6ed3ffc86c..69b419bfa3 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -8,18 +8,17 @@ from __future__ import absolute_import import math from . import types, is_power_of_two from copy import deepcopy -from .types import ValueType, IntType, FloatType, BoolType +from .types import IntType, FloatType, BoolType try: from typing import Tuple, Union, Iterable, Any, Set, TYPE_CHECKING # noqa - from typing import cast if TYPE_CHECKING: from srcgen import Formatter # noqa + from .types import ValueType # noqa Interval = Tuple[int, int] # An Interval where `True` means 'everything' BoolInterval = Union[bool, Interval] except ImportError: - TYPE_CHECKING = False pass MAX_LANES = 256 @@ -263,6 +262,16 @@ class TypeSet(object): return self + def issubset(self, other): + # type: (TypeSet) -> bool + """ + Return true iff self is a subset of other + """ + return self.lanes.issubset(other.lanes) and \ + self.ints.issubset(other.ints) and \ + self.floats.issubset(other.floats) and \ + self.bools.issubset(other.bools) + def lane_of(self): # type: () -> TypeSet """ @@ -292,9 +301,9 @@ class TypeSet(object): Return a TypeSet describing the image of self across halfwidth """ new = self.copy() - new.ints = set([x/2 for x in self.ints if x > 8]) - new.floats = set([x/2 for x in self.floats if x > 32]) - new.bools = set([x/2 for x in self.bools if x > 8]) + new.ints = set([x//2 for x in self.ints if x > 8]) + new.floats = set([x//2 for x in self.floats if x > 32]) + new.bools = set([x//2 for x in self.bools if x > 8]) return new @@ -317,7 +326,7 @@ class TypeSet(object): Return a TypeSet describing the image of self across halfvector """ new = self.copy() - new.lanes = set([x/2 for x in self.lanes if x > 1]) + new.lanes = set([x//2 for x in self.lanes if x > 1]) return new @@ -331,6 +340,67 @@ class TypeSet(object): return new + def map(self, func): + # type: (str) -> TypeSet + """ + Return the image of self across the derived function func + """ + if (func == TypeVar.SAMEAS): + return self + elif (func == TypeVar.LANEOF): + return self.lane_of() + elif (func == TypeVar.ASBOOL): + return self.as_bool() + elif (func == TypeVar.HALFWIDTH): + return self.half_width() + elif (func == TypeVar.DOUBLEWIDTH): + return self.double_width() + elif (func == TypeVar.HALFVECTOR): + return self.half_vector() + elif (func == TypeVar.DOUBLEVECTOR): + return self.double_vector() + else: + assert False, "Unknown derived function: " + func + + def map_inverse(self, func): + # type: (str) -> TypeSet + """ + Return the inverse image of self across the derived function func + """ + # The inverse of the empty set is always empty + if (self.size() == 0): + return self + + if (func == TypeVar.SAMEAS): + return self + elif (func == TypeVar.LANEOF): + new = self.copy() + new.lanes = set([2**i for i in range(0, int_log2(MAX_LANES)+1)]) + return new + elif (func == TypeVar.ASBOOL): + new = self.copy() + new.ints = self.bools.difference(set([1])) + new.floats = self.bools.intersection(set([32, 64])) + + if 1 not in self.bools: + try: + # If the range doesn't have b1, then the domain can't + # include scalars, as as_bool(scalar)=b1 + new.lanes.remove(1) + except KeyError: + pass + return new + elif (func == TypeVar.HALFWIDTH): + return self.double_width() + elif (func == TypeVar.DOUBLEWIDTH): + return self.half_width() + elif (func == TypeVar.HALFVECTOR): + return self.double_vector() + elif (func == TypeVar.DOUBLEVECTOR): + return self.half_vector() + else: + assert False, "Unknown derived function: " + func + def size(self): # type: () -> int """ @@ -346,25 +416,20 @@ class TypeSet(object): typesets containing 1 type. """ assert self.size() == 1 + scalar_type = None # type: types.ScalarType if len(self.ints) > 0: - bits = tuple(self.ints)[0] - scalar_type = ValueType.by_name(IntType.get_name(bits)) + scalar_type = IntType.with_bits(tuple(self.ints)[0]) elif len(self.floats) > 0: - bits = tuple(self.floats)[0] - scalar_type = ValueType.by_name(FloatType.get_name(bits)) + scalar_type = FloatType.with_bits(tuple(self.floats)[0]) else: - bits = tuple(self.bools)[0] - scalar_type = ValueType.by_name(BoolType.get_name(bits)) + scalar_type = BoolType.with_bits(tuple(self.bools)[0]) nlanes = tuple(self.lanes)[0] if nlanes == 1: return scalar_type else: - if TYPE_CHECKING: - return cast(types.ScalarType, scalar_type).by(nlanes) - else: - return scalar_type.by(nlanes) + return scalar_type.by(nlanes) class TypeVar(object): @@ -483,6 +548,14 @@ class TypeVar(object): """Create a type variable that is a function of another.""" return TypeVar(None, None, base=base, derived_func=derived_func) + @staticmethod + def from_typeset(ts): + # type: (TypeSet) -> TypeVar + """ Create a type variable from a type set.""" + tv = TypeVar(None, None) + tv.type_set = ts + return tv + def change_to_derived(self, base, derived_func): # type: (TypeVar, str) -> None """Change this type variable into a derived one.""" @@ -615,6 +688,17 @@ class TypeVar(object): else: return self.name + def constrain_types_by_ts(self, ts): + # type: (TypeSet) -> None + """ + Constrain the range of types this variable can assume to a subset of + those in the typeset ts. + """ + if not self.is_derived: + self.type_set &= ts + else: + self.base.constrain_types_by_ts(ts.map_inverse(self.derived_func)) + def constrain_types(self, other): # type: (TypeVar) -> None """ @@ -628,18 +712,7 @@ class TypeVar(object): if a is b: return - if not a.is_derived and not b.is_derived: - a.type_set &= b.type_set - return - - # TODO: Implement constraints for derived type variables. - # - # If a and b are both derived with the same derived_func, we could say - # `a.base.constrain_types(b.base)`, but unless the derived_func is - # injective, that may constrain `a.base` more than necessary. - # - # For the fully general case, we would need to compute an image typeset - # for `b` and propagate a `a.derived_func` pre-image to `a.base`. + a.constrain_types_by_ts(b.get_typeset()) def get_typeset(self): # type: () -> TypeSet From 165e80d9bf2a272614ce33a4aec6d5af0359f705 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 27 Jun 2017 16:04:50 -0700 Subject: [PATCH 0822/3084] Process ghost instruction kills during coloring. Ghost instructions don't generate code, but they can keep registers alive. The coloring pass needs to process values killed by ghost instructions so it knows when the registers are freed up. Also track register pressure changes from ghost kills in the spiller. --- lib/cretonne/src/regalloc/coloring.rs | 40 ++++++++++++++++++- lib/cretonne/src/regalloc/diversion.rs | 15 +++++++ .../src/regalloc/live_value_tracker.rs | 10 +++++ lib/cretonne/src/regalloc/spilling.rs | 7 +++- 4 files changed, 69 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 56244bcc3c..878656ae84 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -147,8 +147,11 @@ impl<'a> Context<'a> { &mut regs, &mut func.locations, &func.signature); - tracker.drop_dead(inst); + } else { + let (_throughs, kills) = tracker.process_ghost(inst); + self.process_ghost_kills(kills, &mut regs, &func.locations); } + tracker.drop_dead(inst); } } @@ -377,6 +380,9 @@ impl<'a> Context<'a> { } } } + + self.forget_diverted(kills); + *regs = output_regs; } @@ -569,4 +575,36 @@ impl<'a> Context<'a> { dfg.ins(pos).regmove(m.value, m.from, m.to); } } + + /// Forget about any register diversions in `kills`. + fn forget_diverted(&mut self, kills: &[LiveValue]) { + if self.divert.is_empty() { + return; + } + + for lv in kills { + if lv.affinity.is_reg() { + self.divert.remove(lv.value); + } + } + } + + /// Process kills on a ghost instruction. + /// - Forget diversions. + /// - Free killed registers. + fn process_ghost_kills(&mut self, + kills: &[LiveValue], + regs: &mut AllocatableSet, + locations: &ValueLocations) { + for lv in kills { + if let Affinity::Reg(rci) = lv.affinity { + let rc = self.reginfo.rc(rci); + let reg = match self.divert.remove(lv.value) { + Some(r) => r, + None => locations[lv.value].unwrap_reg(), + }; + regs.free(rc, reg); + } + } + } } diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index e8e8d3d190..e07b694ab9 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -51,6 +51,11 @@ impl RegDiversions { self.current.clear() } + /// Are there any diversions? + pub fn is_empty(&self) -> bool { + self.current.is_empty() + } + /// Get the current diversion of `value`, if any. pub fn diversion(&self, value: Value) -> Option<&Diversion> { self.current.iter().find(|d| d.value == value) @@ -83,6 +88,16 @@ impl RegDiversions { self.current.push(Diversion::new(value, from, to)); } } + + /// Drop any recorded register move for `value`. + /// + /// Returns the `to` register of the removed diversion. + pub fn remove(&mut self, value: Value) -> Option { + self.current + .iter() + .position(|d| d.value == value) + .map(|i| self.current.swap_remove(i).to) + } } #[cfg(test)] diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index ccaf6c9209..b243b93923 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -280,6 +280,16 @@ impl LiveValueTracker { &self.live.values[first_def..]) } + /// Prepare to move past a ghost instruction. + /// + /// This is like `process_inst`, except any defs are ignored. + /// + /// Returns `(throughs, kills)`. + pub fn process_ghost(&mut self, inst: Inst) -> (&[LiveValue], &[LiveValue]) { + let first_kill = self.live.live_after(inst); + self.live.values.as_slice().split_at(first_kill) + } + /// Drop the values that are now dead after moving past `inst`. /// /// This removes both live values that were killed by `inst` and dead defines on `inst` itself. diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index f1feff3f4b..09559c9d26 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -129,9 +129,12 @@ impl<'a> Context<'a> { while let Some(inst) = pos.next_inst() { if let Some(constraints) = self.encinfo.operand_constraints(self.encodings[inst]) { self.visit_inst(inst, constraints, &mut pos, dfg, tracker); - tracker.drop_dead(inst); - self.process_spills(tracker); + } else { + let (_throughs, kills) = tracker.process_ghost(inst); + self.free_regs(kills); } + tracker.drop_dead(inst); + self.process_spills(tracker); } } From bbdf07a64ec49914397e6b3779ae0a8b81d33544 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 27 Jun 2017 12:59:23 -0700 Subject: [PATCH 0823/3084] Color EBB arguments. When coloring registers for a branch instruction, also make sure that the values passed as EBB arguments are in the registers expected by the EBB. The first time a branch to an EBB is processed, assign the EBB arguments to the registers where the branch arguments already reside so no regmoves are needed. --- cranelift/filetests/regalloc/coalesce.cton | 6 +- lib/cretonne/src/ir/dfg.rs | 5 + lib/cretonne/src/ir/valueloc.rs | 2 +- lib/cretonne/src/regalloc/coloring.rs | 229 +++++++++++++++------ lib/cretonne/src/regalloc/context.rs | 7 +- lib/cretonne/src/regalloc/diversion.rs | 2 +- 6 files changed, 183 insertions(+), 68 deletions(-) diff --git a/cranelift/filetests/regalloc/coalesce.cton b/cranelift/filetests/regalloc/coalesce.cton index 8b76c8db6b..60ef905d15 100644 --- a/cranelift/filetests/regalloc/coalesce.cton +++ b/cranelift/filetests/regalloc/coalesce.cton @@ -61,7 +61,8 @@ ebb0(v0: i32): ; v1 and v0 interfere here: trapnz v0 ; check: $(cp1=$V) = copy $v1 - ; nextln: jump $ebb1($cp1) + ; not: copy + ; check: jump $ebb1($cp1) jump ebb1(v1) ebb1(v10: i32): @@ -85,7 +86,8 @@ ebb1(v10: i32, v11: i32): v12 = iadd v10, v11 v13 = icmp ult v12, v0 ; check: $(nv11b=$V) = copy $v11 - ; nextln: brnz $v13, $ebb1($nv11b, $v12) + ; not: copy + ; check: brnz $v13, $ebb1($nv11b, $v12) brnz v13, ebb1(v11, v12) return v12 } diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 40f209a49b..51ada16775 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -104,6 +104,11 @@ impl DataFlowGraph { pub fn ebb_is_valid(&self, ebb: Ebb) -> bool { self.ebbs.is_valid(ebb) } + + /// Get the total number of values. + pub fn num_values(&self) -> usize { + self.values.len() + } } /// Resolve value aliases. diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/cretonne/src/ir/valueloc.rs index 8c5ea50dfa..83dbcd0010 100644 --- a/lib/cretonne/src/ir/valueloc.rs +++ b/lib/cretonne/src/ir/valueloc.rs @@ -8,7 +8,7 @@ use ir::StackSlot; use std::fmt; /// Value location. -#[derive(Copy, Clone, Debug)] +#[derive(Copy, Clone, Debug, PartialEq, Eq)] pub enum ValueLoc { /// This value has not been assigned to a location yet. Unassigned, diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 878656ae84..cfb918d832 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -23,6 +23,10 @@ //! operands are allowed to read spilled values, but each such instance must be counted as using //! a register. //! +//! 5. The code must be in conventional SSA form. Among other things, this means that values passed +//! as arguments when branching to an EBB must belong to the same virtual register as the +//! corresponding EBB argument value. +//! //! # Iteration order //! //! The SSA property guarantees that whenever the live range of two values overlap, one of the @@ -30,10 +34,16 @@ //! a topological order relative to the dominance relation, we can assign colors to the values //! defined by the instruction and only consider the colors of other values that are live at the //! instruction. +//! +//! The first time we see a branch to an EBB, the EBB's argument values are colored to match the +//! registers currently holding branch argument values passed to the predecessor branch. By +//! visiting EBBs in a CFG topological order, we guarantee that at least one predecessor branch has +//! been visited before the destination EBB. Therefore, the EBB's arguments are already colored. +//! +//! The exception is the entry block whose arguments are colored from the ABI requirements. use dominator_tree::DominatorTree; -use entity_map::EntityMap; -use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph, ValueLocations}; +use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph, Layout, ValueLocations}; use ir::{InstBuilder, Signature, ArgumentType, ArgumentLoc}; use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; use isa::{TargetIsa, EncInfo, RecipeConstraints, OperandConstraint, ConstraintKind}; @@ -42,8 +52,8 @@ use regalloc::affinity::Affinity; use regalloc::allocatable_set::AllocatableSet; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; +use regalloc::liverange::LiveRange; use regalloc::solver::Solver; -use topo_order::TopoOrder; /// Data structures for the coloring pass. @@ -75,7 +85,6 @@ struct Context<'a> { // References to working set data structures. // If we need to borrow out of a data structure across a method call, it must be passed as a // function argument instead, see the `LiveValueTracker` arguments. - topo: &'a mut TopoOrder, divert: &'a mut RegDiversions, solver: &'a mut Solver, @@ -99,7 +108,6 @@ impl Coloring { func: &mut Function, domtree: &DominatorTree, liveness: &mut Liveness, - topo: &mut TopoOrder, tracker: &mut LiveValueTracker) { dbg!("Coloring for:\n{}", func.display(isa)); let mut ctx = Context { @@ -107,7 +115,6 @@ impl Coloring { encinfo: isa.encoding_info(), domtree, liveness, - topo, divert: &mut self.divert, solver: &mut self.solver, usable_regs: isa.allocatable_registers(func), @@ -119,10 +126,11 @@ impl Coloring { impl<'a> Context<'a> { /// Run the coloring algorithm. fn run(&mut self, func: &mut Function, tracker: &mut LiveValueTracker) { - // Just visit blocks in layout order, letting `self.topo` enforce a topological ordering. - // TODO: Once we have a loop tree, we could visit hot blocks first. - self.topo.reset(func.layout.ebbs()); - while let Some(ebb) = self.topo.next(&func.layout, self.domtree) { + func.locations.resize(func.dfg.num_values()); + + // Visit blocks in reverse post-order. We need to ensure that at least one predecessor has + // been visited before each EBB. That guarantees that the EBB arguments have been colored. + for &ebb in self.domtree.cfg_postorder().iter().rev() { self.visit_ebb(ebb, func, tracker); } } @@ -164,28 +172,29 @@ impl<'a> Context<'a> { tracker: &mut LiveValueTracker) -> AllocatableSet { // Reposition the live value tracker and deal with the EBB arguments. - let (liveins, args) = - tracker.ebb_top(ebb, &func.dfg, self.liveness, &func.layout, self.domtree); + tracker.ebb_top(ebb, &func.dfg, self.liveness, &func.layout, self.domtree); - // Arguments to the entry block have ABI constraints. if func.layout.entry_block() == Some(ebb) { - assert_eq!(liveins.len(), 0); - self.color_entry_args(&func.signature, args, &mut func.locations) + // Arguments to the entry block have ABI constraints. + self.color_entry_args(&func.signature, tracker.live(), &mut func.locations) } else { - // The live-ins have already been assigned a register. Reconstruct the allocatable set. - let regs = self.livein_regs(liveins, func); - self.color_args(args, regs, &mut func.locations) + // The live-ins and arguments to a non-entry EBB have already been assigned a register. + // Reconstruct the allocatable set. + self.livein_regs(tracker.live(), func) } } /// Initialize a set of allocatable registers from the values that are live-in to a block. /// These values must already be colored when the dominating blocks were processed. - fn livein_regs(&self, liveins: &[LiveValue], func: &Function) -> AllocatableSet { + /// + /// Also process the EBB arguments which were colored when the first predecessor branch was + /// encountered. + fn livein_regs(&self, live: &[LiveValue], func: &Function) -> AllocatableSet { // Start from the registers that are actually usable. We don't want to include any reserved // registers in the set. let mut regs = self.usable_regs.clone(); - for lv in liveins { + for lv in live.iter().filter(|lv| !lv.is_dead) { let value = lv.value; let affinity = self.liveness .get(value) @@ -234,7 +243,7 @@ impl<'a> Context<'a> { if !lv.is_dead { regs.take(rc, reg); } - *locations.ensure(lv.value) = ValueLoc::Reg(reg); + locations[lv.value] = ValueLoc::Reg(reg); } else { // This should have been fixed by the reload pass. panic!("Entry arg {} has {} affinity, but ABI {}", @@ -266,39 +275,6 @@ impl<'a> Context<'a> { regs } - /// Color the live arguments to the current block. - /// - /// It is assumed that any live-in register values have already been taken out of the register - /// set. - fn color_args(&self, - args: &[LiveValue], - mut regs: AllocatableSet, - locations: &mut ValueLocations) - -> AllocatableSet { - // Available registers *after* filtering out the dead arguments. - let mut live_regs = regs.clone(); - - for lv in args { - // Only look at the register arguments. - if let Affinity::Reg(rci) = lv.affinity { - let rc = self.reginfo.rc(rci); - // TODO: Fall back to a top-level super-class. Sub-classes are only hints. - let reg = regs.iter(rc) - .next() - .expect("Out of registers for arguments"); - regs.take(rc, reg); - if !lv.is_dead { - live_regs.take(rc, reg); - } - *locations.ensure(lv.value) = ValueLoc::Reg(reg); - } - } - - // All arguments are accounted for in `regs`. We don't care about the dead arguments now - // that we have made sure they don't interfere. - live_regs - } - /// Color the values defined by `inst` and insert any necessary shuffle code to satisfy /// instruction constraints. /// @@ -315,6 +291,10 @@ impl<'a> Context<'a> { func_signature: &Signature) { dbg!("Coloring {}", dfg.display_inst(inst)); + // EBB whose arguments should be colored to match the current branch instruction's + // arguments. + let mut color_dest_args = None; + // Program the solver with register constraints for the input side. self.solver.reset(regs); self.program_input_constraints(inst, constraints.ins, dfg, locations); @@ -323,7 +303,25 @@ impl<'a> Context<'a> { self.program_input_abi(inst, &dfg.signatures[sig].argument_types, dfg, locations); } else if dfg[inst].opcode().is_return() { self.program_input_abi(inst, &func_signature.return_types, dfg, locations); + } else if dfg[inst].opcode().is_branch() { + // This is a branch, so we need to make sure that globally live values are in their + // global registers. For EBBs that take arguments, we also need to place the argument + // values in the expected registers. + if let Some(dest) = dfg[inst].branch_destination() { + if self.program_ebb_arguments(inst, dest, dfg, pos.layout, locations) { + color_dest_args = Some(dest); + } + } else { + // This is a multi-way branch like `br_table`. We only support arguments on + // single-destination branches. + assert_eq!(dfg.inst_variable_args(inst).len(), + 0, + "Can't handle EBB arguments: {}", + dfg.display_inst(inst)); + self.undivert_regs(|lr| !lr.is_local()); + } } + if self.solver.has_fixed_input_conflicts() { self.divert_fixed_input_conflicts(tracker.live(), locations); } @@ -365,9 +363,15 @@ impl<'a> Context<'a> { // registers around. self.shuffle_inputs(pos, dfg, regs); + // If this is the first time we branch to `dest`, color its arguments to match the current + // register state. + if let Some(dest) = color_dest_args { + self.color_ebb_arguments(inst, dest, dfg, locations); + } + // Apply the solution to the defs. for v in self.solver.vars().iter().filter(|&v| v.is_define()) { - *locations.ensure(v.value) = ValueLoc::Reg(v.solution); + locations[v.value] = ValueLoc::Reg(v.solution); } // Update `regs` for the next instruction, remove the dead defs. @@ -391,7 +395,7 @@ impl<'a> Context<'a> { inst: Inst, constraints: &[OperandConstraint], dfg: &DataFlowGraph, - locations: &EntityMap) { + locations: &ValueLocations) { for (op, &value) in constraints .iter() .zip(dfg.inst_args(inst)) @@ -425,7 +429,7 @@ impl<'a> Context<'a> { inst: Inst, abi_types: &[ArgumentType], dfg: &DataFlowGraph, - locations: &EntityMap) { + locations: &ValueLocations) { for (abi, &value) in abi_types.iter().zip(dfg.inst_variable_args(inst)) { if let ArgumentLoc::Reg(reg) = abi.location { if let Affinity::Reg(rci) = @@ -443,6 +447,115 @@ impl<'a> Context<'a> { } } + /// Prepare for a branch to `dest`. + /// + /// 1. Any values that are live-in to `dest` must be un-diverted so they live in their globally + /// assigned register. + /// 2. If the `dest` EBB takes arguments, reassign the branch argument values to the matching + /// registers. + /// + /// Returns true if this is the first time a branch to `dest` is seen, so the `dest` argument + /// values should be colored after `shuffle_inputs`. + fn program_ebb_arguments(&mut self, + inst: Inst, + dest: Ebb, + dfg: &DataFlowGraph, + layout: &Layout, + locations: &ValueLocations) + -> bool { + // Find diverted registers that are live-in to `dest` and reassign them to their global + // home. + // + // Values with a global live range that are not live in to `dest` could appear as branch + // arguments, so they can't always be un-diverted. + self.undivert_regs(|lr| lr.livein_local_end(dest, layout).is_some()); + + // Now handle the EBB arguments. + let br_args = dfg.inst_variable_args(inst); + let dest_args = dfg.ebb_args(dest); + assert_eq!(br_args.len(), dest_args.len()); + for (&dest_arg, &br_arg) in dest_args.iter().zip(br_args) { + // The first time we encounter a branch to `dest`, we get to pick the location. The + // following times we see a branch to `dest`, we must follow suit. + match locations[dest_arg] { + ValueLoc::Unassigned => { + // This is the first branch to `dest`, so we should color `dest_arg` instead of + // `br_arg`. However, we don't know where `br_arg` will end up until + // after `shuffle_inputs`. See `color_ebb_arguments` below. + return true; + } + ValueLoc::Reg(dest_reg) => { + // We've branched to `dest` before. Make sure we use the correct argument + // registers by reassigning `br_arg`. + let br_lr = self.liveness + .get(br_arg) + .expect("Missing live range for branch argument"); + if let Affinity::Reg(rci) = br_lr.affinity { + let rc = self.reginfo.rc(rci); + let br_reg = self.divert.reg(br_arg, locations); + self.solver.reassign_in(br_arg, rc, br_reg, dest_reg); + } else { + panic!("Branch argument {} is not in a register", br_arg); + } + } + ValueLoc::Stack(ss) => { + // The spiller should already have given us identical stack slots. + debug_assert_eq!(ValueLoc::Stack(ss), locations[br_arg]); + } + } + } + + // No `dest` arguments need coloring. + false + } + + /// Knowing that we've never seen a branch to `dest` before, color its arguments to match our + /// register state. + /// + /// This function is only called when `program_ebb_arguments()` returned `true`. + fn color_ebb_arguments(&mut self, + inst: Inst, + dest: Ebb, + dfg: &DataFlowGraph, + locations: &mut ValueLocations) { + let br_args = dfg.inst_variable_args(inst); + let dest_args = dfg.ebb_args(dest); + assert_eq!(br_args.len(), dest_args.len()); + for (&dest_arg, &br_arg) in dest_args.iter().zip(br_args) { + match locations[dest_arg] { + ValueLoc::Unassigned => { + let br_reg = self.divert.reg(br_arg, locations); + locations[dest_arg] = ValueLoc::Reg(br_reg); + } + ValueLoc::Reg(_) => panic!("{} arg {} already colored", dest, dest_arg), + // Spilled value consistency is verified by `program_ebb_arguments()` above. + ValueLoc::Stack(_) => {} + } + } + } + + /// Find all diverted registers where `pred` returns `true` and undo their diversion so they + /// are reallocated to their global register assignments. + fn undivert_regs(&mut self, mut pred: Pred) + where Pred: FnMut(&LiveRange) -> bool + { + for rdiv in self.divert.all() { + let lr = self.liveness + .get(rdiv.value) + .expect("Missing live range for diverted register"); + if pred(lr) { + if let Affinity::Reg(rci) = lr.affinity { + let rc = self.reginfo.rc(rci); + self.solver.reassign_in(rdiv.value, rc, rdiv.to, rdiv.from); + } else { + panic!("Diverted register {} with {} affinity", + rdiv.value, + lr.affinity.display(&self.reginfo)); + } + } + } + } + // Find existing live values that conflict with the fixed input register constraints programmed // into the constraint solver. Convert them to solver variables so they can be diverted. fn divert_fixed_input_conflicts(&mut self, @@ -525,7 +638,7 @@ impl<'a> Context<'a> { let ok = self.solver.add_fixed_output(rc, reg); assert!(ok, "Couldn't clear fixed output interference for {}", value); } - *locations.ensure(value) = ValueLoc::Reg(reg); + locations[value] = ValueLoc::Reg(reg); } /// Program the output-side constraints for `inst` into the constraint solver. diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 0908b02a1c..1961add95c 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -119,12 +119,7 @@ impl Context { // Pass: Coloring. self.coloring - .run(isa, - func, - domtree, - &mut self.liveness, - &mut self.topo, - &mut self.tracker); + .run(isa, func, domtree, &mut self.liveness, &mut self.tracker); if isa.flags().enable_verifier() { verify_context(func, cfg, domtree, Some(isa))?; diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index e07b694ab9..4f17d3685d 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -61,7 +61,7 @@ impl RegDiversions { self.current.iter().find(|d| d.value == value) } - /// Get all current diversion. + /// Get all current diversions. pub fn all(&self) -> &[Diversion] { self.current.as_slice() } From 05cf44a1563e64cef0adc1e2c544d52c3c67578f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 28 Jun 2017 14:05:11 -0700 Subject: [PATCH 0824/3084] Add an offset to StackSlotData. The offset is relative to the stack pointer in the calling function, so it excludes the return address pushed by the call instruction itself on Intel ISAs. Change the ArgumentLoc::Stack offset to an i32, so it matches the stack slot offsets. --- cranelift/filetests/parser/tiny.cton | 4 ++-- lib/cretonne/src/ir/extfunc.rs | 4 ++-- lib/cretonne/src/ir/stackslot.rs | 30 ++++++++++++++++++++++++---- lib/cretonne/src/ir/types.rs | 5 +++++ lib/cretonne/src/ir/valueloc.rs | 2 +- lib/cretonne/src/isa/riscv/abi.rs | 3 ++- lib/reader/src/parser.rs | 23 ++++++++++++++------- 7 files changed, 54 insertions(+), 17 deletions(-) diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 704477e6c2..ff3d386947 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -98,7 +98,7 @@ ebb0(v90: i32, v91: f32): function %stack() { ss10 = spill_slot 8 ss2 = local 4 - ss3 = incoming_arg 4 + ss3 = incoming_arg 4, offset 8 ss4 = outgoing_arg 4 ebb0: @@ -110,7 +110,7 @@ ebb0: ; sameln: function %stack() { ; nextln: $ss10 = spill_slot 8 ; nextln: $ss2 = local 4 -; nextln: $ss3 = incoming_arg 4 +; nextln: $ss3 = incoming_arg 4, offset 8 ; nextln: $ss4 = outgoing_arg 4 ; check: ebb0: diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 57346df5f2..e46ef39a54 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -51,8 +51,8 @@ impl Signature { let bytes = self.argument_types .iter() .filter_map(|arg| match arg.location { - ArgumentLoc::Stack(offset) => { - Some(offset + arg.value_type.bits() as u32 / 8) + ArgumentLoc::Stack(offset) if offset > 0 => { + Some(offset as u32 + arg.value_type.bytes()) } _ => None, }) diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index a1a5f0cb94..baeba2dd23 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -69,18 +69,33 @@ pub struct StackSlotData { /// Size of stack slot in bytes. pub size: u32, + + /// Offset of stack slot relative to the stack pointer in the caller. + /// + /// On Intel ISAs, the base address is the stack pointer *before* the return address was + /// pushed. On RISC ISAs, the base address is the value of the stack pointer on entry to the + /// function. + pub offset: i32, } impl StackSlotData { /// Create a stack slot with the specified byte size. pub fn new(kind: StackSlotKind, size: u32) -> StackSlotData { - StackSlotData { kind, size } + StackSlotData { + kind, + size, + offset: 0, + } } } impl fmt::Display for StackSlotData { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - write!(f, "{} {}", self.kind, self.size) + write!(f, "{} {}", self.kind, self.size)?; + if self.offset != 0 { + write!(f, ", offset {}", self.offset)?; + } + Ok(()) } } @@ -136,8 +151,15 @@ impl StackSlots { impl StackSlots { /// Create a new spill slot for spilling values of type `ty`. pub fn make_spill_slot(&mut self, ty: Type) -> StackSlot { - let bytes = (ty.bits() as u32 + 7) / 8; - self.push(StackSlotData::new(StackSlotKind::SpillSlot, bytes)) + self.push(StackSlotData::new(StackSlotKind::SpillSlot, ty.bytes())) + } + + /// Create a stack slot representing an incoming function argument. + pub fn make_incoming_arg(&mut self, ty: Type, offset: u32) -> StackSlot { + let mut data = StackSlotData::new(StackSlotKind::IncomingArg, ty.bytes()); + assert!(offset <= i32::max_value() as u32 - data.size); + data.offset = offset as i32; + self.push(data) } } diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 550f25804b..e6379be2e6 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -197,6 +197,11 @@ impl Type { self.lane_bits() as u16 * self.lane_count() } + /// Get the number of bytes used to store this type in memory. + pub fn bytes(self) -> u32 { + (self.bits() as u32 + 7) / 8 + } + /// Get a SIMD vector type with `n` times more lanes than this one. /// /// If this is a scalar type, this produces a SIMD type with this as a lane type and `n` lanes. diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/cretonne/src/ir/valueloc.rs index 83dbcd0010..a582f1d934 100644 --- a/lib/cretonne/src/ir/valueloc.rs +++ b/lib/cretonne/src/ir/valueloc.rs @@ -86,7 +86,7 @@ pub enum ArgumentLoc { /// Argument is passed in a register. Reg(RegUnit), /// Argument is passed on the stack, at the given byte offset into the argument array. - Stack(u32), + Stack(i32), } impl Default for ArgumentLoc { diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index ab9e485ed5..66d4c2e6b7 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -77,8 +77,9 @@ impl ArgAssigner for Args { ArgumentLoc::Reg(reg).into() } else { // Assign a stack location. - let loc = ArgumentLoc::Stack(self.offset); + let loc = ArgumentLoc::Stack(self.offset as i32); self.offset += self.pointer_bytes; + assert!(self.offset <= i32::max_value() as u32); loc.into() } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index ae88e09873..f698b1f54f 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -428,15 +428,15 @@ impl<'a> Parser<'a> { } } - // Match and consume a u32 immediate. + // Match and consume an i32 immediate. // This is used for stack argument byte offsets. - fn match_uimm32(&mut self, err_msg: &str) -> Result { + fn match_imm32(&mut self, err_msg: &str) -> Result { if let Some(Token::Integer(text)) = self.token() { self.consume(); // Lexer just gives us raw text that looks like an integer. - // Parse it as a u32 to check for overflow and other issues. + // Parse it as a i32 to check for overflow and other issues. text.parse() - .map_err(|_| self.error("expected u32 decimal immediate")) + .map_err(|_| self.error("expected i32 decimal immediate")) } else { err!(self.loc, err_msg) } @@ -837,7 +837,7 @@ impl<'a> Parser<'a> { } } Some(Token::Integer(_)) => { - let offset = self.match_uimm32("expected stack argument byte offset")?; + let offset = self.match_imm32("expected stack argument byte offset")?; Ok(ArgumentLoc::Stack(offset)) } Some(Token::Minus) => { @@ -870,8 +870,9 @@ impl<'a> Parser<'a> { match self.token() { Some(Token::StackSlot(..)) => { self.gather_comments(ctx.function.stack_slots.next_key()); + let loc = self.loc; self.parse_stack_slot_decl() - .and_then(|(num, dat)| ctx.add_ss(num, dat, &self.loc)) + .and_then(|(num, dat)| ctx.add_ss(num, dat, &loc)) } Some(Token::SigRef(..)) => { self.gather_comments(ctx.function.dfg.signatures.next_key()); @@ -915,7 +916,15 @@ impl<'a> Parser<'a> { if bytes > u32::MAX as i64 { return err!(self.loc, "stack slot too large"); } - let data = StackSlotData::new(kind, bytes as u32); + let mut data = StackSlotData::new(kind, bytes as u32); + + // Take additional options. + while self.optional(Token::Comma) { + match self.match_any_identifier("expected stack slot flags")? { + "offset" => data.offset = self.match_imm32("expected byte offset")?, + other => return err!(self.loc, "Unknown stack slot flag '{}'", other), + } + } // TBD: stack-slot-decl ::= StackSlot(ss) "=" stack-slot-kind Bytes * {"," stack-slot-flag} Ok((number, data)) From b2fda76c5fcc9d3efd1c686fc5406d539dc330bf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 28 Jun 2017 14:59:59 -0700 Subject: [PATCH 0825/3084] Assign stack slots to incoming function arguments. Function arguments that don't fit in registers are passed on the stack. Create "incoming_arg" stack slots representing the stack arguments, and assign them to the value arguments during spilling. --- cranelift/filetests/regalloc/spill.cton | 12 ++++++++++ lib/cretonne/src/ir/stackslot.rs | 6 ++--- lib/cretonne/src/ir/valueloc.rs | 8 +++++++ lib/cretonne/src/regalloc/coloring.rs | 15 ++----------- lib/cretonne/src/regalloc/spilling.rs | 29 ++++++++++++++++++++++++- 5 files changed, 53 insertions(+), 17 deletions(-) diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index 23ddf7129b..d30903dc2d 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -109,3 +109,15 @@ ebb0(v0: i32): ; check: call_indirect $sig0, $v0($v0, $c) return } + +; Two arguments on the stack. +function %stackargs(i32, i32, i32, i32, i32, i32, i32, i32) -> i32 { +; check: ss0 = incoming_arg 4 +; check: ss1 = incoming_arg 4, offset 4 +; not: incoming_arg +ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32, v7: i32): + ; unordered: fill $v6 + ; unordered: fill $v7 + v10 = iadd v6, v7 + return v10 +} diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index baeba2dd23..c6fd64ce5c 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -155,10 +155,10 @@ impl StackSlots { } /// Create a stack slot representing an incoming function argument. - pub fn make_incoming_arg(&mut self, ty: Type, offset: u32) -> StackSlot { + pub fn make_incoming_arg(&mut self, ty: Type, offset: i32) -> StackSlot { let mut data = StackSlotData::new(StackSlotKind::IncomingArg, ty.bytes()); - assert!(offset <= i32::max_value() as u32 - data.size); - data.offset = offset as i32; + assert!(offset <= i32::max_value() - data.size as i32); + data.offset = offset; self.push(data) } } diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/cretonne/src/ir/valueloc.rs index a582f1d934..cae3042170 100644 --- a/lib/cretonne/src/ir/valueloc.rs +++ b/lib/cretonne/src/ir/valueloc.rs @@ -112,6 +112,14 @@ impl ArgumentLoc { } } + /// Is this a stack location? + pub fn is_stack(&self) -> bool { + match *self { + ArgumentLoc::Stack(_) => true, + _ => false, + } + } + /// Return an object that can display this argument location, using the register info from the /// target ISA. pub fn display<'a, R: Into>>(self, regs: R) -> DisplayArgumentLoc<'a> { diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index cfb918d832..c2353de40e 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -253,19 +253,8 @@ impl<'a> Context<'a> { } } - Affinity::Stack => { - if let ArgumentLoc::Stack(_offset) = abi.location { - // TODO: Allocate a stack slot at incoming offset and assign it. - panic!("Unimplemented {}: {} stack allocation", - lv.value, - abi.display(&self.reginfo)); - } else { - // This should have been fixed by the reload pass. - panic!("Entry arg {} has stack affinity, but ABI {}", - lv.value, - abi.display(&self.reginfo)); - } - } + // The spiller will have assigned an incoming stack slot already. + Affinity::Stack => assert!(abi.location.is_stack()), // This is a ghost value, unused in the function. Don't assign it to a location // either. Affinity::None => {} diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 09559c9d26..c64fd565e3 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -17,7 +17,7 @@ use dominator_tree::DominatorTree; use ir::{DataFlowGraph, Layout, Cursor, InstBuilder}; -use ir::{Function, Ebb, Inst, Value, ValueLoc, SigRef}; +use ir::{Function, Ebb, Inst, Value, ValueLoc, ArgumentLoc, Signature, SigRef}; use ir::{InstEncodings, StackSlots, ValueLocations}; use isa::registers::{RegClass, RegClassMask}; use isa::{TargetIsa, RegInfo, EncInfo, RecipeConstraints, ConstraintKind}; @@ -45,6 +45,7 @@ struct Context<'a> { encodings: &'a mut InstEncodings, stack_slots: &'a mut StackSlots, locations: &'a mut ValueLocations, + func_signature: &'a Signature, // References to contextual data structures we need. domtree: &'a DominatorTree, @@ -92,6 +93,7 @@ impl Spilling { encodings: &mut func.encodings, stack_slots: &mut func.stack_slots, locations: &mut func.locations, + func_signature: &func.signature, domtree, liveness, virtregs, @@ -109,12 +111,37 @@ impl<'a> Context<'a> { layout: &mut Layout, dfg: &mut DataFlowGraph, tracker: &mut LiveValueTracker) { + if let Some(entry) = layout.entry_block() { + self.spill_entry_arguments(entry, dfg); + } + self.topo.reset(layout.ebbs()); while let Some(ebb) = self.topo.next(layout, self.domtree) { self.visit_ebb(ebb, layout, dfg, tracker); } } + /// Assign stack slots to incoming function arguments on the stack. + fn spill_entry_arguments(&mut self, entry: Ebb, dfg: &DataFlowGraph) { + for (abi, &arg) in self.func_signature + .argument_types + .iter() + .zip(dfg.ebb_args(entry)) { + if let ArgumentLoc::Stack(offset) = abi.location { + // Function arguments passed on the stack can't be part of a virtual register. We + // would need to write other values to the stack slot, but it belongs to the + // caller. (Not that the caller would care, nobody depends on stack arguments being + // preserved across calls). + assert_eq!(self.virtregs.get(arg), + None, + "Stack argument {} can't be part of a virtual register", + arg); + let ss = self.stack_slots.make_incoming_arg(abi.value_type, offset); + *self.locations.ensure(arg) = ValueLoc::Stack(ss); + } + } + } + fn visit_ebb(&mut self, ebb: Ebb, layout: &mut Layout, From 0574dcdeee8c0daa78d424e9811ca165ff0f6504 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 28 Jun 2017 15:37:38 -0700 Subject: [PATCH 0826/3084] Don't coalesce incoming stack arguments. A function parameter in an incoming_arg stack slot should not be coalesced into any virtual registers. We don't want to force the whole virtual register to spill to the incoming_arg slot. --- cranelift/filetests/regalloc/coalesce.cton | 18 ++++++++++++++++++ lib/cretonne/src/regalloc/coalescing.rs | 18 +++++++++++++++++- 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/regalloc/coalesce.cton b/cranelift/filetests/regalloc/coalesce.cton index 60ef905d15..41fc6e9771 100644 --- a/cranelift/filetests/regalloc/coalesce.cton +++ b/cranelift/filetests/regalloc/coalesce.cton @@ -91,3 +91,21 @@ ebb1(v10: i32, v11: i32): brnz v13, ebb1(v11, v12) return v12 } + +; Function arguments passed on the stack aren't allowed to be part of a virtual +; register, at least for now. This is because the other values in the virtual +; register would need to be spilled to the incoming_arg stack slot which we treat +; as belonging to the caller. +function %stackarg(i32, i32, i32, i32, i32, i32, i32, i32, i32) -> i32 { +; check: ss0 = incoming_arg 4 +; not: incoming_arg +ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32, v7: i32, v8: i32): + ; check: fill v8 + ; not: v8 + brnz v0, ebb1(v8) + jump ebb1(v7) + +ebb1(v10: i32): + v11 = iadd_imm v10, 1 + return v11 +} diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 70a20e3897..634f0c0a32 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -8,7 +8,7 @@ use dbg::DisplayList; use dominator_tree::DominatorTree; use flowgraph::{ControlFlowGraph, BasicBlock}; -use ir::{DataFlowGraph, Layout, Cursor, InstBuilder}; +use ir::{DataFlowGraph, Layout, Cursor, InstBuilder, ValueDef}; use ir::{Function, Ebb, Inst, Value, ExpandedProgramPoint}; use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; @@ -400,6 +400,22 @@ impl<'a> Context<'a> { pred_val, pred_ebb, self.func.dfg.display_inst(pred_inst)); + + // Never coalesce incoming function arguments on the stack. These arguments are + // pre-spilled, and the rest of the virtual register would be forced to spill to the + // `incoming_arg` stack slot too. + if let ValueDef::Arg(def_ebb, def_num) = self.func.dfg.value_def(pred_val) { + if Some(def_ebb) == self.func.layout.entry_block() && + self.func.signature.argument_types[def_num] + .location + .is_stack() { + dbg!("Isolating incoming stack parameter {}", pred_val); + let new_val = self.split_pred(pred_inst, pred_ebb, argnum, pred_val); + assert!(self.add_class(new_val).is_ok()); + continue; + } + } + if let Err((a, b)) = self.add_class(pred_val) { dbg!("Found conflict between {} and {}", a, b); // We have a conflict between the already merged value `a` and one of the new From e7a543ea33e906ba1fbafd6e474ec35fd9f91d93 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 28 Jun 2017 19:30:36 -0700 Subject: [PATCH 0827/3084] Make sure return values are assigned an affinity. When an EBB argument value is used only as a return value, it still needs to be given a register affinity. Otherwise it would appear as a ghost value with no affinity. Do the same to call arguments. --- cranelift/filetests/regalloc/basic.cton | 23 +++++++++++++++++++++++ lib/cretonne/src/regalloc/liveness.rs | 17 ++++++++++++----- 2 files changed, 35 insertions(+), 5 deletions(-) diff --git a/cranelift/filetests/regalloc/basic.cton b/cranelift/filetests/regalloc/basic.cton index 36c0e5c81c..dd6cab11cf 100644 --- a/cranelift/filetests/regalloc/basic.cton +++ b/cranelift/filetests/regalloc/basic.cton @@ -40,3 +40,26 @@ ebb0(v1: i32, v2: i32): ; nextln: return $v2, $v1 return v2, v1 } + +; Return an EBB argument. +function %retebb(i32, i32) -> i32 { +ebb0(v1: i32, v2: i32): + brnz v1, ebb1(v1) + jump ebb1(v2) + +ebb1(v10: i32): + return v10 +} + +; Pass an EBB argument as a function argument. +function %callebb(i32, i32) -> i32 { + fn0 = function %foo(i32) -> i32 + +ebb0(v1: i32, v2: i32): + brnz v1, ebb1(v1) + jump ebb1(v2) + +ebb1(v10: i32): + v11 = call fn0(v10) + return v11 +} diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index afccbffaa3..71fa4961e7 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -376,9 +376,9 @@ impl Liveness { } // Iterator of constraints, one per value operand. - // TODO: Should we fail here if the instruction doesn't have a valid encoding? + let encoding = func.encodings[inst]; let mut operand_constraints = enc_info - .operand_constraints(func.encodings[inst]) + .operand_constraints(encoding) .map(|c| c.ins) .unwrap_or(&[]) .iter(); @@ -392,11 +392,18 @@ impl Liveness { // Apply operand constraint, ignoring any variable arguments after the fixed // operands described by `operand_constraints`. Variable arguments are either - // EBB arguments or call/return ABI arguments. EBB arguments need to be - // resolved by the coloring algorithm, and ABI constraints require specific - // registers or stack slots which the affinities don't model anyway. + // EBB arguments or call/return ABI arguments. if let Some(constraint) = operand_constraints.next() { lr.affinity.merge(constraint, ®_info); + } else if lr.affinity.is_none() && encoding.is_legal() && + !func.dfg[inst].opcode().is_branch() { + // This is a real encoded instruction using a value that doesn't yet have a + // concrete affinity. Most likely a call argument or a return value. Give + // the value a register affinity matching the ABI type. + // + // EBB arguments on a branch are not required to have an affinity. + let rc = isa.regclass_for_abi_type(func.dfg.value_type(arg)); + lr.affinity = Affinity::Reg(rc.into()); } } } From 588ef0ad2fdbb5c07df1dcb93e7073cb6987423a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 29 Jun 2017 10:30:26 -0700 Subject: [PATCH 0828/3084] Propagate affinities for EBB arguments. A priory, an EBB argument value only gets an affinity if it is used directly by a non-ghost instruction. A use by a branch passing arguments to an EBB doesn't count. When an EBB argument value does have an affinity, the values passed by all the predecessors must also have affinities. This can cause EBB argument values to get affinities recursively. - Add a second pass to the liveness computation for propagating EBB argument affinities, possibly recursively. - Verify EBB argument affinities correctly: A value passed to a branch must have an affinity only if the corresponding EBB argument value in the destination has an affinity. --- cranelift/filetests/regalloc/basic.cton | 15 +++++++++ lib/cretonne/src/regalloc/liveness.rs | 41 +++++++++++++++++++++++++ lib/cretonne/src/verifier/liveness.rs | 33 ++++++++++++++++++-- 3 files changed, 87 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/regalloc/basic.cton b/cranelift/filetests/regalloc/basic.cton index dd6cab11cf..89e160d75a 100644 --- a/cranelift/filetests/regalloc/basic.cton +++ b/cranelift/filetests/regalloc/basic.cton @@ -63,3 +63,18 @@ ebb1(v10: i32): v11 = call fn0(v10) return v11 } + +; Pass an EBB argument as a jump argument. +function %jumpebb(i32, i32) -> i32 { + fn0 = function %foo(i32) -> i32 + +ebb0(v1: i32, v2: i32): + brnz v1, ebb1(v1, v2) + jump ebb1(v2, v1) + +ebb1(v10: i32, v11: i32): + jump ebb2(v10, v11) + +ebb2(v20: i32, v21: i32): + return v21 +} diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 71fa4961e7..2093b15299 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -279,6 +279,9 @@ pub struct Liveness { /// This vector is always empty, except for inside that function. /// It lives here to avoid repeated allocation of scratch memory. worklist: Vec, + + /// Working space for the `propagate_ebb_arguments` algorithm. + ebb_args: Vec, } impl Liveness { @@ -290,6 +293,7 @@ impl Liveness { Liveness { ranges: LiveRangeSet::new(), worklist: Vec::new(), + ebb_args: Vec::new(), } } @@ -408,5 +412,42 @@ impl Liveness { } } } + + self.propagate_ebb_arguments(func, cfg); + } + + /// Propagate affinities for EBB arguments. + /// + /// If an EBB argument value has an affinity, all predecessors must pass a value with an + /// affinity. + pub fn propagate_ebb_arguments(&mut self, func: &Function, cfg: &ControlFlowGraph) { + assert!(self.ebb_args.is_empty()); + + for ebb in func.layout.ebbs() { + for &arg in func.dfg.ebb_args(ebb).iter() { + let affinity = self.ranges.get(arg).unwrap().affinity; + if affinity.is_none() { + continue; + } + self.ebb_args.push(arg); + + // Now apply the affinity to all predecessors recursively. + while let Some(succ_arg) = self.ebb_args.pop() { + let (succ_ebb, num) = match func.dfg.value_def(succ_arg) { + ValueDef::Arg(e, n) => (e, n), + _ => continue, + }; + + for &(_, pred_branch) in cfg.get_predecessors(succ_ebb) { + let pred_arg = func.dfg.inst_variable_args(pred_branch)[num]; + let pred_affinity = &mut self.ranges.get_mut(pred_arg).unwrap().affinity; + if pred_affinity.is_none() { + *pred_affinity = affinity; + self.ebb_args.push(pred_arg); + } + } + } + } + } } } diff --git a/lib/cretonne/src/verifier/liveness.rs b/lib/cretonne/src/verifier/liveness.rs index 51911b3f38..2dff4a6e3e 100644 --- a/lib/cretonne/src/verifier/liveness.rs +++ b/lib/cretonne/src/verifier/liveness.rs @@ -93,7 +93,7 @@ impl<'a> LivenessVerifier<'a> { } // Check the uses. - for &val in self.func.dfg.inst_args(inst) { + for (idx, &val) in self.func.dfg.inst_args(inst).iter().enumerate() { let lr = match self.liveness.get(val) { Some(lr) => lr, None => return err!(inst, "{} has no live range", val), @@ -104,7 +104,10 @@ impl<'a> LivenessVerifier<'a> { if encoding.is_legal() { // A legal instruction is not allowed to depend on ghost values. - if lr.affinity.is_none() { + // + // A branch argument can be a ghost value if the corresponding destination + // EBB argument is a ghost value. + if lr.affinity.is_none() && !self.is_ghost_branch_argument(inst, idx) { return err!(inst, "{} is a ghost value used by a real [{}] instruction", val, @@ -134,6 +137,32 @@ impl<'a> LivenessVerifier<'a> { } } + /// Is argument `argnum` on `inst` a branch argument that leads to a ghost EBB argument? + fn is_ghost_branch_argument(&self, inst: Inst, argnum: usize) -> bool { + let dest = match self.func.dfg[inst].branch_destination() { + Some(d) => d, + None => return false, + }; + + let fixed_args = self.func.dfg[inst] + .opcode() + .constraints() + .fixed_value_arguments(); + if argnum < fixed_args { + return false; + } + + // If the EBB argument value in the destination is a ghost value, we'll allow a ghost + // branch argument. + self.func + .dfg + .ebb_args(dest) + .get(argnum - fixed_args) + .and_then(|&v| self.liveness.get(v)) + .map(|lr| lr.affinity.is_none()) + .unwrap_or(false) + } + /// Check the integrity of the live range `lr`. fn check_lr(&self, def: ProgramPoint, val: Value, lr: &LiveRange) -> Result { let l = &self.func.layout; From d6f73ac4c878a26e73fb32bd5a3bde805c80fbf3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 29 Jun 2017 11:11:46 -0700 Subject: [PATCH 0829/3084] Split spill_from() into spill_candidate() and spill_reg(). We'll need to pick a spill candidate from a set and allow for the search to fail to find anything. This also allows slightly better panic messages when we run out of registers. --- lib/cretonne/src/regalloc/spilling.rs | 52 ++++++++++++++++----------- 1 file changed, 31 insertions(+), 21 deletions(-) diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index c64fd565e3..1b438a7b50 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -260,7 +260,14 @@ impl<'a> Context<'a> { // Add register def to pressure, spill if needed. while let Err(mask) = self.pressure.take_transient(op.regclass) { dbg!("Need {} reg from {} throughs", op.regclass, throughs.len()); - self.spill_from(mask, throughs, dfg, pos.layout); + match self.spill_candidate(mask, throughs, dfg, pos.layout) { + Some(cand) => self.spill_reg(cand, dfg), + None => { + panic!("Ran out of {} registers for {}", + op.regclass, + dfg.display_inst(inst)) + } + } } } } @@ -341,22 +348,32 @@ impl<'a> Context<'a> { // Spill a live register that is *not* used by the current instruction. // Spilling a use wouldn't help. let args = dfg.inst_args(inst); - self.spill_from(mask, - tracker.live().iter().filter(|lv| !args.contains(&lv.value)), - dfg, - &pos.layout); + match self.spill_candidate(mask, + tracker.live().iter().filter(|lv| { + !args.contains(&lv.value) + }), + dfg, + &pos.layout) { + Some(cand) => self.spill_reg(cand, dfg), + None => { + panic!("Ran out of {} registers when inserting copy before {}", + rc, + dfg.display_inst(inst)) + } + } } } self.pressure.reset_transient(); self.reg_uses.clear() } - // Spill a candidate from `candidates` whose top-level register class is in `mask`. - fn spill_from<'ii, II>(&mut self, - mask: RegClassMask, - candidates: II, - dfg: &DataFlowGraph, - layout: &Layout) + // Find a spill candidate from `candidates` whose top-level register class is in `mask`. + fn spill_candidate<'ii, II>(&self, + mask: RegClassMask, + candidates: II, + dfg: &DataFlowGraph, + layout: &Layout) + -> Option where II: IntoIterator { // Find the best viable spill candidate. @@ -367,7 +384,7 @@ impl<'a> Context<'a> { // // We know that all candidate defs dominate the current instruction, so one of them will // dominate the others. That is the earliest def. - let best = candidates + candidates .into_iter() .filter_map(|lv| { // Viable candidates are registers in one of the `mask` classes, and not already in @@ -385,14 +402,7 @@ impl<'a> Context<'a> { // Find the minimum candidate according to the RPO of their defs. self.domtree .rpo_cmp(dfg.value_def(a), dfg.value_def(b), layout) - }); - - if let Some(value) = best { - // Found a spill candidate. - self.spill_reg(value, dfg); - } else { - panic!("Ran out of registers for mask={}", mask); - } + }) } /// Spill `value` immediately by @@ -424,7 +434,7 @@ impl<'a> Context<'a> { /// Process any pending spills in the `self.spills` vector. /// /// It is assumed that spills are removed from the pressure tracker immediately, see - /// `spill_from` above. + /// `spill_reg` above. /// /// We also need to update the live range affinity and remove spilled values from the live /// value tracker. From ae661631aa9313d7e252753635de07d67e92d6a9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 29 Jun 2017 13:29:43 -0700 Subject: [PATCH 0830/3084] Only color EBB arguments that have register affinity. It is possible to pass a register value as an argument to an EBB that expects a "None" affinity. In that case, the destination EBB value should not be colored. --- lib/cretonne/src/regalloc/coloring.rs | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index c2353de40e..1a21f78f7e 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -200,13 +200,13 @@ impl<'a> Context<'a> { .get(value) .expect("No live range for live-in") .affinity; + dbg!("Live-in: {}:{} in {}", + value, + affinity.display(&self.reginfo), + func.locations[value].display(&self.reginfo)); if let Affinity::Reg(rci) = affinity { let rc = self.reginfo.rc(rci); let loc = func.locations[value]; - dbg!("Live-in: {}:{} in {}", - lv.value, - rc, - loc.display(&self.reginfo)); match loc { ValueLoc::Reg(reg) => regs.take(rc, reg), ValueLoc::Unassigned => panic!("Live-in {} wasn't assigned", value), @@ -471,7 +471,12 @@ impl<'a> Context<'a> { // This is the first branch to `dest`, so we should color `dest_arg` instead of // `br_arg`. However, we don't know where `br_arg` will end up until // after `shuffle_inputs`. See `color_ebb_arguments` below. - return true; + // + // It is possible for `dest_arg` to have no affinity, and then it should simply + // be ignored. + if self.liveness.get(dest_arg).unwrap().affinity.is_reg() { + return true; + } } ValueLoc::Reg(dest_reg) => { // We've branched to `dest` before. Make sure we use the correct argument @@ -513,8 +518,10 @@ impl<'a> Context<'a> { for (&dest_arg, &br_arg) in dest_args.iter().zip(br_args) { match locations[dest_arg] { ValueLoc::Unassigned => { - let br_reg = self.divert.reg(br_arg, locations); - locations[dest_arg] = ValueLoc::Reg(br_reg); + if self.liveness.get(dest_arg).unwrap().affinity.is_reg() { + let br_reg = self.divert.reg(br_arg, locations); + locations[dest_arg] = ValueLoc::Reg(br_reg); + } } ValueLoc::Reg(_) => panic!("{} arg {} already colored", dest, dest_arg), // Spilled value consistency is verified by `program_ebb_arguments()` above. From 138d3c75c6a8c2a9fee88b6a68a2963266e03077 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 29 Jun 2017 14:07:19 -0700 Subject: [PATCH 0831/3084] Spill live-ins and EBB arguments if there are too many. --- cranelift/filetests/regalloc/spill.cton | 23 +++++++++++++++ lib/cretonne/src/regalloc/pressure.rs | 8 ++++++ lib/cretonne/src/regalloc/spilling.rs | 38 +++++++++++++++++++++++-- 3 files changed, 67 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index d30903dc2d..71c16ed50a 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -121,3 +121,26 @@ ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32, v7: i32): v10 = iadd v6, v7 return v10 } + +; More EBB arguments than registers. +function %ebbargs(i32) -> i32 { +ebb0(v1: i32): + ; check: $v1 = spill + v2 = iconst.i32 1 + jump ebb1(v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2) + +ebb1(v10: i32, v11: i32, v12: i32, v13: i32, v14: i32, v15: i32, v16: i32, v17: i32, v18: i32, v19: i32, v20: i32, v21: i32): + v22 = iadd v10, v11 + v23 = iadd v22, v12 + v24 = iadd v23, v13 + v25 = iadd v24, v14 + v26 = iadd v25, v15 + v27 = iadd v26, v16 + v28 = iadd v27, v17 + v29 = iadd v28, v18 + v30 = iadd v29, v19 + v31 = iadd v30, v20 + v32 = iadd v31, v21 + v33 = iadd v32, v1 + return v33 +} diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index 4839c7451a..3bf1e68f54 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -224,6 +224,14 @@ impl Pressure { e.transient_count = 0; } } + + /// Preserve the transient counts by transferring them to the base counts. + pub fn preserve_transient(&mut self) { + for e in &mut self.toprc { + e.base_count += e.transient_count; + e.transient_count = 0; + } + } } impl fmt::Display for Pressure { diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 1b438a7b50..2a05fbb8c3 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -200,8 +200,42 @@ impl<'a> Context<'a> { self.pressure.reset(); self.take_live_regs(liveins); - // TODO: Process and count EBB arguments. Some may need spilling. - self.take_live_regs(args); + // An EBB can have an arbitrary (up to 2^16...) number of EBB arguments, so they are not + // guaranteed to fit in registers. + for lv in args { + if let Affinity::Reg(rci) = lv.affinity { + let rc = self.reginfo.rc(rci); + 'try_take: while let Err(mask) = self.pressure.take_transient(rc) { + dbg!("Need {} reg for EBB argument {} from {} live-ins", + rc, + lv.value, + liveins.len()); + match self.spill_candidate(mask, liveins, dfg, layout) { + Some(cand) => { + dbg!("Spilling live-in {} to make room for {} EBB argument {}", + cand, + rc, + lv.value); + self.spill_reg(cand, dfg); + } + None => { + // We can't spill any of the live-in registers, so we have to spill an + // EBB argument. Since the current spill metric would consider all the + // EBB arguments equal, just spill the present register. + dbg!("Spilling {} EBB argument {}", rc, lv.value); + + // Since `spill_reg` will free a register, add the current one here. + self.pressure.take(rc); + self.spill_reg(lv.value, dfg); + break 'try_take; + } + } + } + } + } + + // The transient pressure counts for the EBB arguments are accurate. Just preserve them. + self.pressure.preserve_transient(); } fn visit_inst(&mut self, From 2b999d9bd67751ced2b091a778830c2f2977df05 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 29 Jun 2017 15:13:04 -0700 Subject: [PATCH 0832/3084] Add an Index implementation to Liveness. Use it to access live ranges that are supposed to be there. --- lib/cretonne/src/regalloc/coloring.rs | 9 +++------ lib/cretonne/src/regalloc/live_value_tracker.rs | 5 +---- lib/cretonne/src/regalloc/liveness.rs | 12 ++++++++++++ lib/cretonne/src/regalloc/reload.rs | 3 +-- 4 files changed, 17 insertions(+), 12 deletions(-) diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 1a21f78f7e..733506ef0d 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -474,17 +474,14 @@ impl<'a> Context<'a> { // // It is possible for `dest_arg` to have no affinity, and then it should simply // be ignored. - if self.liveness.get(dest_arg).unwrap().affinity.is_reg() { + if self.liveness[dest_arg].affinity.is_reg() { return true; } } ValueLoc::Reg(dest_reg) => { // We've branched to `dest` before. Make sure we use the correct argument // registers by reassigning `br_arg`. - let br_lr = self.liveness - .get(br_arg) - .expect("Missing live range for branch argument"); - if let Affinity::Reg(rci) = br_lr.affinity { + if let Affinity::Reg(rci) = self.liveness[br_arg].affinity { let rc = self.reginfo.rc(rci); let br_reg = self.divert.reg(br_arg, locations); self.solver.reassign_in(br_arg, rc, br_reg, dest_reg); @@ -518,7 +515,7 @@ impl<'a> Context<'a> { for (&dest_arg, &br_arg) in dest_args.iter().zip(br_args) { match locations[dest_arg] { ValueLoc::Unassigned => { - if self.liveness.get(dest_arg).unwrap().affinity.is_reg() { + if self.liveness[dest_arg].affinity.is_reg() { let br_reg = self.divert.reg(br_arg, locations); locations[dest_arg] = ValueLoc::Reg(br_reg); } diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index b243b93923..bee25aa287 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -260,10 +260,7 @@ impl LiveValueTracker { // Add the values defined by `inst`. let first_def = self.live.values.len(); for &value in dfg.inst_results(inst) { - let lr = match liveness.get(value) { - Some(lr) => lr, - None => panic!("{} result {} has no live range", dfg[inst].opcode(), value), - }; + let lr = &liveness[value]; assert_eq!(lr.def(), inst.into()); match lr.def_local_end().into() { ExpandedProgramPoint::Inst(endpoint) => { diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 2093b15299..973e278f62 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -183,6 +183,7 @@ use regalloc::affinity::Affinity; use regalloc::liverange::LiveRange; use sparse_map::SparseMap; use std::mem; +use std::ops::Index; /// A set of live ranges, indexed by value number. type LiveRangeSet = SparseMap; @@ -451,3 +452,14 @@ impl Liveness { } } } + +impl Index for Liveness { + type Output = LiveRange; + + fn index(&self, index: Value) -> &LiveRange { + match self.ranges.get(index) { + Some(lr) => lr, + None => panic!("{} has no live range", index), + } + } +} diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index e8327b5a73..93860071d3 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -285,8 +285,7 @@ impl<'a> Context<'a> { for (op, &arg) in constraints.ins.iter().zip(args) { if op.kind != ConstraintKind::Stack { - let lv = self.liveness.get(arg).expect("Missing live range for arg"); - if lv.affinity.is_stack() { + if self.liveness[arg].affinity.is_stack() { self.candidates .push(ReloadCandidate { value: arg, From 18dc420352f0b572c469d771fd536e1e99086ad5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 29 Jun 2017 16:51:05 -0700 Subject: [PATCH 0833/3084] Repair constraint violations during spilling. The following constraints may need to be resolved during spilling because the resolution increases register pressure: - A tied operand whose value is live through the instruction. - A fixed register constraint for a value used more than once. - A register use of a spilled value needs to account for the reload register. --- cranelift/filetests/regalloc/spill.cton | 50 ++++++++ lib/cretonne/src/regalloc/spilling.rs | 152 +++++++++++++++--------- 2 files changed, 147 insertions(+), 55 deletions(-) diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index 71c16ed50a..4c470a73e6 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -144,3 +144,53 @@ ebb1(v10: i32, v11: i32, v12: i32, v13: i32, v14: i32, v15: i32, v16: i32, v17: v33 = iadd v32, v1 return v33 } + +; In straight-line code, the first value defined is spilled. +; That is in order: +; 1. The argument v1. +; 2. The link register. +; 3. The first computed value, v2 +function %use_spilled_value(i32) -> i32 { +; check: ss0 = spill_slot 4 +; check: ss1 = spill_slot 4 +; check: ss2 = spill_slot 4 +ebb0(v1: i32): +; check: $ebb0($(rv1=$V): i32, $(rlink=$V): i32) + ; check: ,ss0]$WS $v1 = spill $rv1 + ; nextln: ,ss1]$WS $(link=$V) = spill $rlink + ; not: spill + v2 = iadd_imm v1, 12 + ; check: $(r1v2=$V) = iadd_imm + ; nextln: ,ss2]$WS $v2 = spill $r1v2 + v3 = iadd_imm v2, 12 + v4 = iadd_imm v3, 12 + v5 = iadd_imm v4, 12 + v6 = iadd_imm v5, 12 + v7 = iadd_imm v6, 12 + v8 = iadd_imm v7, 12 + v9 = iadd_imm v8, 12 + v10 = iadd_imm v9, 12 + v11 = iadd_imm v10, 12 + v12 = iadd_imm v11, 12 + v13 = iadd_imm v12, 12 + v14 = iadd_imm v13, 12 + + ; Here we have maximum register pressure, and v2 has been spilled. + ; What happens if we use it? + v33 = iadd v2, v14 + v32 = iadd v33, v12 + v31 = iadd v32, v11 + v30 = iadd v31, v10 + v29 = iadd v30, v9 + v28 = iadd v29, v8 + v27 = iadd v28, v7 + v26 = iadd v27, v6 + v25 = iadd v26, v5 + v24 = iadd v25, v4 + v23 = iadd v24, v3 + v22 = iadd v23, v2 + v21 = iadd v22, v1 + v20 = iadd v21, v13 + v19 = iadd v20, v2 + return v21 +} diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 2a05fbb8c3..58f8fdc7fa 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -19,7 +19,7 @@ use dominator_tree::DominatorTree; use ir::{DataFlowGraph, Layout, Cursor, InstBuilder}; use ir::{Function, Ebb, Inst, Value, ValueLoc, ArgumentLoc, Signature, SigRef}; use ir::{InstEncodings, StackSlots, ValueLocations}; -use isa::registers::{RegClass, RegClassMask}; +use isa::registers::{RegClassMask, RegClassIndex}; use isa::{TargetIsa, RegInfo, EncInfo, RecipeConstraints, ConstraintKind}; use regalloc::affinity::Affinity; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; @@ -245,18 +245,10 @@ impl<'a> Context<'a> { dfg: &mut DataFlowGraph, tracker: &mut LiveValueTracker) { dbg!("Inst {}, {}", dfg.display_inst(inst), self.pressure); - // TODO: Repair constraint violations by copying input values. - // - // - Tied use of value that is not killed. - // - Count pressure for register uses of spilled values too. + // We may need to resolve register constraints if there are any noteworthy uses. assert!(self.reg_uses.is_empty()); - - // If the instruction has any fixed register operands, we may need to resolve register - // constraints. - if constraints.fixed_ins { - self.collect_reg_uses(inst, constraints, dfg); - } + self.collect_reg_uses(inst, constraints, dfg); // Calls usually have fixed register uses. let call_sig = dfg.call_signature(inst); @@ -268,7 +260,6 @@ impl<'a> Context<'a> { self.process_reg_uses(inst, pos, dfg, tracker); } - // Update the live value tracker with this instruction. let (throughs, kills, defs) = tracker.process_inst(inst, dfg, self.liveness); @@ -312,7 +303,12 @@ impl<'a> Context<'a> { // This won't cause spilling. self.take_live_regs(defs); } - // Collect register uses from the fixed input constraints. + + // Collect register uses that are noteworthy in one of the following ways: + // + // 1. It's a fixed register constraint. + // 2. It's a use of a spilled value. + // 3. It's a tied register constraint and the value isn't killed. // // We are assuming here that if a value is used both by a fixed register operand and a register // class operand, they two are compatible. We are also assuming that two register class @@ -323,11 +319,23 @@ impl<'a> Context<'a> { dfg: &DataFlowGraph) { let args = dfg.inst_args(inst); for (idx, (op, &arg)) in constraints.ins.iter().zip(args).enumerate() { + let mut reguse = RegUse::new(arg, idx, op.regclass.into()); match op.kind { - ConstraintKind::FixedReg(_) => { - self.reg_uses.push(RegUse::new(arg, idx)); + ConstraintKind::Stack => continue, + ConstraintKind::FixedReg(_) => reguse.fixed = true, + ConstraintKind::Tied(_) => { + // TODO: If `arg` isn't killed here, we need a copy } - _ => {} + ConstraintKind::Reg => {} + } + let lr = &self.liveness[arg]; + if lr.affinity.is_stack() { + reguse.spilled = true; + } + + // Only collect the interesting register uses. + if reguse.fixed || reguse.spilled { + self.reg_uses.push(reguse); } } } @@ -343,7 +351,17 @@ impl<'a> Context<'a> { .zip(args) .enumerate() { if abi.location.is_reg() { - self.reg_uses.push(RegUse::new(arg, fixed_args + idx)); + let (rci, spilled) = match self.liveness[arg].affinity { + Affinity::Reg(rci) => (rci, false), + Affinity::Stack => { + (self.isa.regclass_for_abi_type(abi.value_type).into(), true) + } + Affinity::None => panic!("Missing affinity for {}", arg), + }; + let mut reguse = RegUse::new(arg, fixed_args + idx, rci); + reguse.fixed = true; + reguse.spilled = spilled; + self.reg_uses.push(reguse); } } } @@ -364,35 +382,49 @@ impl<'a> Context<'a> { // outside nightly Rust. self.reg_uses.sort_by_key(|u| (u.value, u.opidx)); - // We are assuming that `reg_uses` has an entry per fixed register operand, and that any - // non-fixed register operands are compatible with one of the fixed uses of the value. - for i in 1..self.reg_uses.len() { + for i in 0..self.reg_uses.len() { let ru = self.reg_uses[i]; - if self.reg_uses[i - 1].value != ru.value { - continue; + + // Do we need to insert a copy for this use? + let need_copy = if ru.tied { + true + } else if ru.fixed { + // This is a fixed register use which doesn't necessarily require a copy. + // Make a copy only if this is not the first use of the value. + self.reg_uses + .get(i.wrapping_sub(1)) + .map(|ru2| ru2.value == ru.value) + .unwrap_or(false) + } else { + false + }; + + if need_copy { + let copy = self.insert_copy(ru.value, ru.rci, pos, dfg); + dfg.inst_args_mut(inst)[ru.opidx as usize] = copy; } - // We have two fixed uses of the same value. Make a copy. - let (copy, rc) = self.insert_copy(ru.value, pos, dfg); - dfg.inst_args_mut(inst)[ru.opidx as usize] = copy; - - // Make sure the new copy doesn't blow the register pressure. - while let Err(mask) = self.pressure.take_transient(rc) { - dbg!("Copy of {} reg causes spill", rc); - // Spill a live register that is *not* used by the current instruction. - // Spilling a use wouldn't help. - let args = dfg.inst_args(inst); - match self.spill_candidate(mask, - tracker.live().iter().filter(|lv| { - !args.contains(&lv.value) - }), - dfg, - &pos.layout) { - Some(cand) => self.spill_reg(cand, dfg), - None => { - panic!("Ran out of {} registers when inserting copy before {}", - rc, - dfg.display_inst(inst)) + // Even if we don't insert a copy, we may need to account for register pressure for the + // reload pass. + if need_copy || ru.spilled { + let rc = self.reginfo.rc(ru.rci); + while let Err(mask) = self.pressure.take_transient(rc) { + dbg!("Copy of {} reg causes spill", rc); + // Spill a live register that is *not* used by the current instruction. + // Spilling a use wouldn't help. + let args = dfg.inst_args(inst); + match self.spill_candidate(mask, + tracker.live().iter().filter(|lv| { + !args.contains(&lv.value) + }), + dfg, + &pos.layout) { + Some(cand) => self.spill_reg(cand, dfg), + None => { + panic!("Ran out of {} registers when inserting copy before {}", + rc, + dfg.display_inst(inst)) + } } } } @@ -481,12 +513,13 @@ impl<'a> Context<'a> { /// Insert a `copy value` before `pos` and give it a live range extending to `pos`. /// - /// Returns the new local value created and its register class. + /// Returns the new local value created. fn insert_copy(&mut self, value: Value, + rci: RegClassIndex, pos: &mut Cursor, dfg: &mut DataFlowGraph) - -> (Value, RegClass) { + -> Value { let copy = dfg.ins(pos).copy(value); let inst = dfg.value_def(copy).unwrap_inst(); let ty = dfg.value_type(copy); @@ -498,21 +531,14 @@ impl<'a> Context<'a> { *self.encodings.ensure(inst) = encoding; // Update live ranges. - let rc = self.encinfo - .operand_constraints(encoding) - .expect("Bad copy encoding") - .outs - [0] - .regclass; - self.liveness - .create_dead(copy, inst, Affinity::Reg(rc.into())); + self.liveness.create_dead(copy, inst, Affinity::Reg(rci)); self.liveness .extend_locally(copy, pos.layout.pp_ebb(inst), pos.current_inst().expect("must be at an instruction"), pos.layout); - (copy, rc) + copy } } @@ -522,13 +548,29 @@ impl<'a> Context<'a> { struct RegUse { value: Value, opidx: u16, + + // Register class required by the use. + rci: RegClassIndex, + + // A use with a fixed register constraint. + fixed: bool, + + // A register use of a spilled value. + spilled: bool, + + // A use with a tied register constraint *and* the used value is not killed. + tied: bool, } impl RegUse { - fn new(value: Value, idx: usize) -> RegUse { + fn new(value: Value, idx: usize, rci: RegClassIndex) -> RegUse { RegUse { value, opidx: idx as u16, + rci, + fixed: false, + spilled: false, + tied: false, } } } From aa3bf4467eb47bed39c979c5d5645adacaf135d2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 30 Jun 2017 08:41:54 -0700 Subject: [PATCH 0834/3084] Hook up the handling of tied register constraints. Tests are forthcoming, we need to implement Intel ABI lowering first. --- lib/cretonne/src/regalloc/liverange.rs | 9 ++++++++- lib/cretonne/src/regalloc/spilling.rs | 14 +++++++++----- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 95dfb49f2e..9174e0cdba 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -399,7 +399,7 @@ impl LiveRange { } } - /// Check if this live range reaches a use at `inst` in `ebb`. + /// Check if this live range reaches a use at `user` in `ebb`. pub fn reaches_use(&self, user: Inst, ebb: Ebb, order: &PO) -> bool where PO: ProgramOrder { @@ -415,6 +415,13 @@ impl LiveRange { None => false, } } + + /// Check if this live range is killed at `user` in `ebb`. + pub fn killed_at(&self, user: Inst, ebb: Ebb, order: &PO) -> bool + where PO: ProgramOrder + { + self.def_local_end() == user.into() || self.livein_local_end(ebb, order) == Some(user) + } } /// Allow a `LiveRange` to be stored in a `SparseMap` indexed by values. diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 58f8fdc7fa..37ff870a26 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -155,7 +155,7 @@ impl<'a> Context<'a> { pos.goto_top(ebb); while let Some(inst) = pos.next_inst() { if let Some(constraints) = self.encinfo.operand_constraints(self.encodings[inst]) { - self.visit_inst(inst, constraints, &mut pos, dfg, tracker); + self.visit_inst(inst, ebb, constraints, &mut pos, dfg, tracker); } else { let (_throughs, kills) = tracker.process_ghost(inst); self.free_regs(kills); @@ -240,6 +240,7 @@ impl<'a> Context<'a> { fn visit_inst(&mut self, inst: Inst, + ebb: Ebb, constraints: &RecipeConstraints, pos: &mut Cursor, dfg: &mut DataFlowGraph, @@ -248,7 +249,7 @@ impl<'a> Context<'a> { // We may need to resolve register constraints if there are any noteworthy uses. assert!(self.reg_uses.is_empty()); - self.collect_reg_uses(inst, constraints, dfg); + self.collect_reg_uses(inst, ebb, constraints, dfg, pos.layout); // Calls usually have fixed register uses. let call_sig = dfg.call_signature(inst); @@ -315,20 +316,23 @@ impl<'a> Context<'a> { // operands are always compatible. fn collect_reg_uses(&mut self, inst: Inst, + ebb: Ebb, constraints: &RecipeConstraints, - dfg: &DataFlowGraph) { + dfg: &DataFlowGraph, + layout: &Layout) { let args = dfg.inst_args(inst); for (idx, (op, &arg)) in constraints.ins.iter().zip(args).enumerate() { let mut reguse = RegUse::new(arg, idx, op.regclass.into()); + let lr = &self.liveness[arg]; match op.kind { ConstraintKind::Stack => continue, ConstraintKind::FixedReg(_) => reguse.fixed = true, ConstraintKind::Tied(_) => { - // TODO: If `arg` isn't killed here, we need a copy + // A tied operand must kill the used value. + reguse.tied = !lr.killed_at(inst, ebb, layout); } ConstraintKind::Reg => {} } - let lr = &self.liveness[arg]; if lr.affinity.is_stack() { reguse.spilled = true; } From 983048cdd16594029d0e55920f78a7fcc3a4b4c9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 30 Jun 2017 09:32:42 -0700 Subject: [PATCH 0835/3084] Generate an enum with all the register units in a target. It is sometimes useful to create constant lists of register units by name. The generated RU enum can be used for that. --- lib/cretonne/meta/gen_registers.py | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/lib/cretonne/meta/gen_registers.py b/lib/cretonne/meta/gen_registers.py index 463364709a..6d41a40330 100644 --- a/lib/cretonne/meta/gen_registers.py +++ b/lib/cretonne/meta/gen_registers.py @@ -30,6 +30,19 @@ def gen_regbank(regbank, fmt): fmt.format('num_toprcs: {},', len(regbank.toprcs)) +def gen_regbank_units(regbank, fmt): + # type: (RegBank, srcgen.Formatter) -> None + """ + Emit constants for all the register units in `regbank`. + """ + for unit in range(regbank.units): + v = unit + regbank.first_unit + if unit < len(regbank.names): + fmt.format("{} = {},", regbank.names[unit], v) + else: + fmt.format("{}{} = {},", regbank.prefix, unit, v) + + def gen_regclass(rc, fmt): # type: (RegClass, srcgen.Formatter) -> None """ @@ -77,6 +90,13 @@ def gen_isa(isa, fmt): 'pub const {}: RegClass = &CLASSES[{}];' .format(rc.name, rc.index)) + # Emit constants for all the register units. + fmt.line('#[allow(dead_code, non_camel_case_types)]') + fmt.line('#[derive(Clone, Copy)]') + with fmt.indented('pub enum RU {', '}'): + for regbank in isa.regbanks: + gen_regbank_units(regbank, fmt) + def generate(isas, out_dir): # type: (Sequence[TargetISA], str) -> None From 7bc2e82b1607da4c9a3a147f7253a90c5520cbff Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 30 Jun 2017 10:37:42 -0700 Subject: [PATCH 0836/3084] Implement the basics of the x86-64 ABI. This is just a rough sketch to get us started. There are bound to be some issues. This also legalizes signatures for x86-32, but probably not correctly. It's basically implementing the x86-64 ABI for 32-bit. --- cranelift/filetests/isa/intel/abi64.cton | 20 ++++ lib/cretonne/src/isa/intel/abi.rs | 114 +++++++++++++++++++++-- lib/cretonne/src/isa/intel/mod.rs | 2 +- 3 files changed, 128 insertions(+), 8 deletions(-) create mode 100644 cranelift/filetests/isa/intel/abi64.cton diff --git a/cranelift/filetests/isa/intel/abi64.cton b/cranelift/filetests/isa/intel/abi64.cton new file mode 100644 index 0000000000..ecd3d29fd6 --- /dev/null +++ b/cranelift/filetests/isa/intel/abi64.cton @@ -0,0 +1,20 @@ +; Test the legalization of function signatures. +test legalizer +set is_64bit +isa intel + +; regex: V=v\d+ + +function %f() { + sig0 = signature(i32) -> i32 + ; check: sig0 = signature(i32 [%rdi]) -> i32 [%rax] + + sig1 = signature(i64) -> b1 + ; check: sig1 = signature(i64 [%rdi]) -> b1 [%rax] + + sig2 = signature(f32, i64) -> f64 + ; check: sig2 = signature(f32 [%xmm0], i64 [%rdi]) -> f64 [%xmm0] + +ebb0: + return +} diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index e6c5edd5bf..240e9d327d 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -1,16 +1,102 @@ //! Intel ABI implementation. use ir; -use isa::RegClass; +use isa::{RegClass, RegUnit}; use regalloc::AllocatableSet; use settings as shared_settings; -use super::registers::{GPR, FPR}; +use super::registers::{GPR, FPR, RU}; +use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; +use ir::{ArgumentType, ArgumentLoc, ArgumentExtension}; + +/// Argument registers for x86-64 +static ARG_GPRS: [RU; 6] = [RU::rdi, RU::rsi, RU::rdx, RU::rcx, RU::r8, RU::r9]; + +/// Return value registers. +static RET_GPRS: [RU; 3] = [RU::rax, RU::rdx, RU::rcx]; + +struct Args { + pointer_bytes: u32, + pointer_bits: u16, + pointer_type: ir::Type, + gpr: &'static [RU], + gpr_used: usize, + fpr_limit: usize, + fpr_used: usize, + offset: u32, +} + +impl Args { + fn new(bits: u16, gpr: &'static [RU], fpr_limit: usize) -> Args { + Args { + pointer_bytes: bits as u32 / 8, + pointer_bits: bits, + pointer_type: ir::Type::int(bits).unwrap(), + gpr, + gpr_used: 0, + fpr_limit, + fpr_used: 0, + offset: 0, + } + } +} + +impl ArgAssigner for Args { + fn assign(&mut self, arg: &ArgumentType) -> ArgAction { + let ty = arg.value_type; + + // Check for a legal type. + // We don't support SIMD yet, so break all vectors down. + if !ty.is_scalar() { + return ValueConversion::VectorSplit.into(); + } + + // Large integers and booleans are broken down to fit in a register. + if !ty.is_float() && ty.bits() > self.pointer_bits { + return ValueConversion::IntSplit.into(); + } + + // Small integers are extended to the size of a pointer register. + if ty.is_int() && ty.bits() < self.pointer_bits { + match arg.extension { + ArgumentExtension::None => {} + ArgumentExtension::Uext => return ValueConversion::Uext(self.pointer_type).into(), + ArgumentExtension::Sext => return ValueConversion::Sext(self.pointer_type).into(), + } + } + + // Try to use a GPR. + if !ty.is_float() && self.gpr_used < self.gpr.len() { + let reg = self.gpr[self.gpr_used] as RegUnit; + self.gpr_used += 1; + return ArgumentLoc::Reg(reg).into(); + } + + // Try to use an FPR. + if ty.is_float() && self.fpr_used < self.fpr_limit { + let reg = FPR.unit(self.fpr_used); + self.fpr_used += 1; + return ArgumentLoc::Reg(reg).into(); + } + + // Assign a stack location. + let loc = ArgumentLoc::Stack(self.offset as i32); + self.offset += self.pointer_bytes; + assert!(self.offset <= i32::max_value() as u32); + loc.into() + } +} /// Legalize `sig`. -pub fn legalize_signature(_sig: &mut ir::Signature, - _flags: &shared_settings::Flags, +pub fn legalize_signature(sig: &mut ir::Signature, + flags: &shared_settings::Flags, _current: bool) { - unimplemented!() + let bits = if flags.is_64bit() { 64 } else { 32 }; + + let mut args = Args::new(bits, &ARG_GPRS, 8); + legalize_args(&mut sig.argument_types, &mut args); + + let mut rets = Args::new(bits, &RET_GPRS, 2); + legalize_args(&mut sig.return_types, &mut rets); } /// Get register class for a type appearing in a legalized signature. @@ -19,6 +105,20 @@ pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { } /// Get the set of allocatable registers for `func`. -pub fn allocatable_registers(_func: &ir::Function) -> AllocatableSet { - unimplemented!() +pub fn allocatable_registers(_func: &ir::Function, + flags: &shared_settings::Flags) + -> AllocatableSet { + let mut regs = AllocatableSet::new(); + regs.take(GPR, RU::rsp as RegUnit); + regs.take(GPR, RU::rbp as RegUnit); + + // 32-bit arch only has 8 registers. + if !flags.is_64bit() { + for i in 8..16 { + regs.take(GPR, GPR.unit(i)); + regs.take(FPR, FPR.unit(i)); + } + } + + regs } diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 6cfd0b4b85..88df62c8ac 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -88,7 +88,7 @@ impl TargetIsa for Isa { } fn allocatable_registers(&self, func: &ir::Function) -> regalloc::AllocatableSet { - abi::allocatable_registers(func) + abi::allocatable_registers(func, &self.shared_flags) } fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { From c592d3174f2cc6ff4b9d4e86d4ab02390dd09280 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 30 Jun 2017 11:41:06 -0700 Subject: [PATCH 0837/3084] Add Intel iconst.i32 encoding. --- cranelift/filetests/isa/intel/binary32.cton | 6 ++++-- lib/cretonne/meta/isa/intel/encodings.py | 3 +++ lib/cretonne/meta/isa/intel/recipes.py | 7 ++++++- lib/cretonne/src/isa/intel/binemit.rs | 13 +++++++++++++ 4 files changed, 26 insertions(+), 3 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 818335d349..6d13844497 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -9,8 +9,10 @@ isa intel function %I32() { ebb0: - [-,%rcx] v1 = iconst.i32 1 - [-,%rsi] v2 = iconst.i32 2 + ; asm: movl $1, %ecx + [-,%rcx] v1 = iconst.i32 1 ; bin: b9 00000001 + ; asm: movl $2, %esi + [-,%rsi] v2 = iconst.i32 2 ; bin: be 00000002 ; Integer Register-Register Operations. diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 575c36baca..4010ffb64a 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -22,6 +22,9 @@ for inst, rrr in [ I32.enc(inst, *r.rib(0x83, rrr=rrr)) I32.enc(inst, *r.rid(0x81, rrr=rrr)) +# Immediate constant. +I32.enc(base.iconst.i32, *r.uid(0xb8)) + # 32-bit shifts and rotates. # Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit # and 16-bit shifts would need explicit masking. diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 71b8260f42..70377f8756 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -4,7 +4,7 @@ Intel Encoding recipes. from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt, IsEqual -from base.formats import Binary, BinaryImm, Store, Load +from base.formats import UnaryImm, Binary, BinaryImm, Store, Load from .registers import GPR, ABCD try: @@ -155,6 +155,11 @@ rid = TailRecipe( 'rid', BinaryImm, size=5, ins=GPR, outs=0, instp=IsSignedInt(BinaryImm.imm, 32)) +# XX+rd id unary with 32-bit immediate. +uid = TailRecipe( + 'uid', UnaryImm, size=4, ins=(), outs=GPR, + instp=IsSignedInt(UnaryImm.imm, 32)) + # # Store recipes. # diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 87587463ca..aded1e41e6 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -133,6 +133,19 @@ fn recipe_op1rid(func: &Function, inst: Inst, sink: &mut } } +fn recipe_op1uid(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::UnaryImm { imm, .. } = func.dfg[inst] { + let bits = func.encodings[inst].bits(); + let reg = func.locations[func.dfg.first_result(inst)].unwrap_reg(); + // The destination register is encoded in the low bits of the opcode. No ModR/M + put_op1(bits | (reg & 7), sink); + let imm: i64 = imm.into(); + sink.put4(imm as u32); + } else { + panic!("Expected UnaryImm format: {:?}", func.dfg[inst]); + } +} + // Store recipes. fn recipe_op1st(func: &Function, inst: Inst, sink: &mut CS) { From 811c1059fc6ea9f935475b6b433ffb102a1fc5f1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 30 Jun 2017 12:21:36 -0700 Subject: [PATCH 0838/3084] Add Intel call/return encodings. --- cranelift/filetests/isa/intel/binary32.cton | 14 +++++++- lib/cretonne/meta/isa/intel/encodings.py | 7 ++++ lib/cretonne/meta/isa/intel/recipes.py | 8 +++++ lib/cretonne/src/isa/intel/binemit.rs | 38 +++++++++++++++++++-- 4 files changed, 64 insertions(+), 3 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 6d13844497..2e42d01fcc 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -8,6 +8,9 @@ isa intel ; function %I32() { + fn0 = function %foo() + sig0 = signature() + ebb0: ; asm: movl $1, %ecx [-,%rcx] v1 = iconst.i32 1 ; bin: b9 00000001 @@ -199,5 +202,14 @@ ebb0: ; asm: movsbl -50000(%esi), %edx [-,%rdx] v129 = sload8.i32 v2-50000 ; bin: 0f be 96 ffff3cb0 - return + ; asm: call foo + call fn0() ; bin: e8 PCRel4(fn0) 00000000 + + ; asm: call *%ecx + call_indirect sig0, v1() ; bin: ff d1 + ; asm: call *%esi + call_indirect sig0, v2() ; bin: ff d6 + + ; asm: ret + return ; bin: c3 } diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 4010ffb64a..c05308177a 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -64,3 +64,10 @@ I32.enc(base.uload8.i32.i32, *r.ldDisp32(0x0f, 0xb6)) I32.enc(base.sload8.i32.i32, *r.ld(0x0f, 0xbe)) I32.enc(base.sload8.i32.i32, *r.ldDisp8(0x0f, 0xbe)) I32.enc(base.sload8.i32.i32, *r.ldDisp32(0x0f, 0xbe)) + +# +# Call/return +# +I32.enc(base.call, *r.call_id(0xe8)) +I32.enc(base.call_indirect.i32, *r.call_r(0xff, rrr=2)) +I32.enc(base.x_return, *r.ret(0xc3)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 70377f8756..4e41c9b990 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -5,6 +5,7 @@ from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt, IsEqual from base.formats import UnaryImm, Binary, BinaryImm, Store, Load +from base.formats import MultiAry, Call, IndirectCall from .registers import GPR, ABCD try: @@ -206,3 +207,10 @@ ldDisp8 = TailRecipe( ldDisp32 = TailRecipe( 'ldDisp32', Load, size=5, ins=(GPR), outs=(GPR), instp=IsSignedInt(Load.offset, 32)) + +# +# Call/return +# +call_id = TailRecipe('call_id', Call, size=4, ins=(), outs=()) +call_r = TailRecipe('call_r', IndirectCall, size=1, ins=GPR, outs=()) +ret = TailRecipe('ret', MultiAry, size=0, ins=(), outs=()) diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index aded1e41e6..001a41e009 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -1,12 +1,24 @@ //! Emitting binary Intel machine code. -use binemit::{CodeSink, bad_encoding}; +use binemit::{CodeSink, Reloc, bad_encoding}; use ir::{Function, Inst, InstructionData}; use isa::RegUnit; include!(concat!(env!("OUT_DIR"), "/binemit-intel.rs")); -pub static RELOC_NAMES: [&'static str; 1] = ["Call"]; +/// Intel relocations. +pub enum RelocKind { + /// A 4-byte relative function reference. Based from relocation + 4 bytes. + PCRel4, +} + +pub static RELOC_NAMES: [&'static str; 1] = ["PCRel4"]; + +impl Into for RelocKind { + fn into(self) -> Reloc { + Reloc(self as u16) + } +} // Emit single-byte opcode. fn put_op1(bits: u16, sink: &mut CS) { @@ -310,3 +322,25 @@ fn recipe_op2lddisp32(func: &Function, inst: Inst, sink: panic!("Expected Load format: {:?}", func.dfg[inst]); } } + +fn recipe_op1call_id(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Call { func_ref, .. } = func.dfg[inst] { + put_op1(func.encodings[inst].bits(), sink); + sink.reloc_func(RelocKind::PCRel4.into(), func_ref); + sink.put4(0); + } else { + panic!("Expected Call format: {:?}", func.dfg[inst]); + } +} + +fn recipe_op1call_r(func: &Function, inst: Inst, sink: &mut CS) { + let bits = func.encodings[inst].bits(); + put_op1(bits, sink); + modrm_r_bits(func.locations[func.dfg.inst_args(inst)[0]].unwrap_reg(), + bits, + sink); +} + +fn recipe_op1ret(func: &Function, inst: Inst, sink: &mut CS) { + put_op1(func.encodings[inst].bits(), sink); +} From 0d2d1ea8cf40eea1473ca06751393e8d7bf26ecd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 30 Jun 2017 13:34:18 -0700 Subject: [PATCH 0839/3084] Add support for tied operands. Include a very basic test using an Intel 'sub' instruction. More to follow. --- cranelift/filetests/regalloc/constraints.cton | 15 +++++++++++ lib/cretonne/src/regalloc/coloring.rs | 25 +++++++++++++++--- lib/cretonne/src/regalloc/solver.rs | 26 +++++++++++++++++++ 3 files changed, 62 insertions(+), 4 deletions(-) create mode 100644 cranelift/filetests/regalloc/constraints.cton diff --git a/cranelift/filetests/regalloc/constraints.cton b/cranelift/filetests/regalloc/constraints.cton new file mode 100644 index 0000000000..e88bcc514c --- /dev/null +++ b/cranelift/filetests/regalloc/constraints.cton @@ -0,0 +1,15 @@ +test regalloc +isa intel + +; regex: V=v\d+ + +; Tied operands, both are killed at instruction. +function %tied_easy() -> i32 { +ebb0: + v0 = iconst.i32 12 + v1 = iconst.i32 13 + ; not: copy + ; check: isub + v2 = isub v0, v1 + return v2 +} diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 733506ef0d..4ebf3340f9 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -363,6 +363,17 @@ impl<'a> Context<'a> { locations[v.value] = ValueLoc::Reg(v.solution); } + // Tied defs are not part of the solution above. + // Copy register assignments from tied inputs to tied outputs. + if constraints.tied_ops { + for (op, lv) in constraints.outs.iter().zip(defs) { + if let ConstraintKind::Tied(num) = op.kind { + let arg = dfg.inst_args(inst)[num as usize]; + locations[lv.value] = locations[arg]; + } + } + } + // Update `regs` for the next instruction, remove the dead defs. for lv in defs { if lv.endpoint == inst { @@ -638,11 +649,11 @@ impl<'a> Context<'a> { /// /// It is assumed that all fixed outputs have already been handled. fn program_output_constraints(&mut self, - _inst: Inst, + inst: Inst, constraints: &[OperandConstraint], defs: &[LiveValue], - _dfg: &mut DataFlowGraph, - _locations: &mut ValueLocations) { + dfg: &mut DataFlowGraph, + locations: &mut ValueLocations) { for (op, lv) in constraints.iter().zip(defs) { match op.kind { ConstraintKind::FixedReg(_) | @@ -650,7 +661,13 @@ impl<'a> Context<'a> { ConstraintKind::Reg => { self.solver.add_def(lv.value, op.regclass); } - ConstraintKind::Tied(_) => unimplemented!(), + ConstraintKind::Tied(num) => { + // Find the input operand we're tied to. + // The solver doesn't care about the output value. + let arg = dfg.inst_args(inst)[num as usize]; + self.solver + .add_tied_input(arg, op.regclass, self.divert.reg(arg, locations)); + } } } } diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index b2e73bf91f..b135f051f3 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -488,6 +488,32 @@ impl Solver { self.regs_out.free(rc, reg); } + /// Record that an input register is tied to an output register. + /// + /// It is assumed that `add_kill` was called previously with the same arguments. + /// + /// The output value that must have the same register as the input value is not recorded in the + /// solver. + pub fn add_tied_input(&mut self, value: Value, rc: RegClass, reg: RegUnit) { + debug_assert!(self.inputs_done); + + // If a fixed assignment is tied, the `to` register is not available on the output side. + if let Some(a) = self.assignments.get(value) { + debug_assert_eq!(a.from, reg); + self.regs_out.take(a.rc, a.to); + return; + } + + // Check if a variable was created. + if let Some(v) = self.vars.iter_mut().find(|v| v.value == value) { + assert!(v.is_input); + v.is_output = true; + return; + } + + self.regs_out.take(rc, reg); + } + /// Add a fixed output assignment. /// /// This means that `to` will not be available for variables on the output side of the From e63c581859a622ca14515b976f4133adc7f93052 Mon Sep 17 00:00:00 2001 From: Denis Merigoux Date: Wed, 5 Jul 2017 08:44:51 -0700 Subject: [PATCH 0840/3084] Fixed bug in verifier (#109) * Fixed bug in verifier Does not check variable def for unreachable codex * Check reachability first + file test --- .../filetests/verifier/unreachable_code.cton | 23 +++++++++++++++++++ lib/cretonne/src/verifier/mod.rs | 6 +++-- 2 files changed, 27 insertions(+), 2 deletions(-) create mode 100644 cranelift/filetests/verifier/unreachable_code.cton diff --git a/cranelift/filetests/verifier/unreachable_code.cton b/cranelift/filetests/verifier/unreachable_code.cton new file mode 100644 index 0000000000..474eaec6b4 --- /dev/null +++ b/cranelift/filetests/verifier/unreachable_code.cton @@ -0,0 +1,23 @@ +test verifier + +function %test() -> i32 { ; Ok +ebb0: + v0 = iconst.i32 0 + v1 = iconst.i32 0 + jump ebb2 + +ebb2: + jump ebb4 + +ebb4: + jump ebb2 + +ebb3(v2: i32): + v4 = iadd.i32 v1, v2 + jump ebb9(v4) + +ebb9(v7: i32): + v9 = iadd.i32 v2, v7 + return v9 + +} diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 08aeb9249f..2f817d1a48 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -363,7 +363,8 @@ impl<'a> Verifier<'a> { def_inst); } // Defining instruction dominates the instruction that uses the value. - if !self.domtree + if self.domtree.is_reachable(self.func.layout.pp_ebb(loc_inst)) && + !self.domtree .dominates(def_inst, loc_inst, &self.func.layout) { return err!(loc_inst, "uses value from non-dominating {}", def_inst); } @@ -381,7 +382,8 @@ impl<'a> Verifier<'a> { ebb); } // The defining EBB dominates the instruction using this value. - if !self.domtree.dominates(ebb, loc_inst, &self.func.layout) { + if self.domtree.is_reachable(ebb) && + !self.domtree.dominates(ebb, loc_inst, &self.func.layout) { return err!(loc_inst, "uses value arg from non-dominating {}", ebb); } } From e35398842d711e9630fbb59a34155f463a4a4fa0 Mon Sep 17 00:00:00 2001 From: d1m0 Date: Wed, 5 Jul 2017 09:16:44 -0700 Subject: [PATCH 0841/3084] Add better type inference and encapsulate it in its own file (#110) * Add more rigorous type inference and encapsulate the type inferece code in its own file (ti.py). Add constraints accumulation during type inference, to represent constraints that cannot be expressed using bijective derivation functions between typevars. Add testing for new type inference code. * Additional annotations to appease mypy --- lib/cretonne/meta/cdsl/ast.py | 147 +------ lib/cretonne/meta/cdsl/test_ti.py | 432 +++++++++++++++++++ lib/cretonne/meta/cdsl/test_typevar.py | 62 ++- lib/cretonne/meta/cdsl/ti.py | 556 +++++++++++++++++++++++++ lib/cretonne/meta/cdsl/typevar.py | 125 ++++-- lib/cretonne/meta/cdsl/xform.py | 82 +--- 6 files changed, 1123 insertions(+), 281 deletions(-) create mode 100644 lib/cretonne/meta/cdsl/test_ti.py create mode 100644 lib/cretonne/meta/cdsl/ti.py diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index 2b671fc46e..6efc492cf5 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -100,8 +100,8 @@ class Var(Expr): # TypeVar representing the type of this variable. self.typevar = None # type: TypeVar # The original 'typeof(x)' type variable that was created for this Var. - # This one doesn't change. `self.typevar` above may be joined with - # other typevars. + # This one doesn't change. `self.typevar` above may be changed to + # another typevar by type inference. self.original_typevar = None # type: TypeVar def __str__(self): @@ -180,16 +180,9 @@ class Var(Expr): self.typevar = tv return self.typevar - def link_typevar(self, base, derived_func): - # type: (TypeVar, str) -> None - """ - Link the type variable on this Var to the type variable `base` using - `derived_func`. - """ - self.original_typevar = None - self.typevar.change_to_derived(base, derived_func) - # Possibly eliminate redundant SAMEAS links. - self.typevar = self.typevar.strip_sameas() + def set_typevar(self, tv): + # type: (TypeVar) -> None + self.typevar = tv def has_free_typevar(self): # type: () -> bool @@ -213,136 +206,6 @@ class Var(Expr): """ return self.typevar.rust_expr() - def constrain_typevar(self, sym_typevar, sym_ctrl, ctrl_var): - # type: (TypeVar, TypeVar, Var) -> None - """ - Constrain the set of allowed types for this variable. - - Merge type variables for the involved variables to minimize the set for - free type variables. - - Suppose we're looking at an instruction defined like this: - - c = Operand('c', TxN.as_bool()) - x = Operand('x', TxN) - y = Operand('y', TxN) - a = Operand('a', TxN) - vselect = Instruction('vselect', ins=(c, x, y), outs=a) - - And suppose the instruction is used in a pattern like this: - - v0 << vselect(v1, v2, v3) - - We want to reconcile the types of the variables v0-v3 with the - constraints from the definition of vselect. This means that v0, v2, and - v3 must all have the same type, and v1 must have the type - `typeof(v2).as_bool()`. - - The types are reconciled by calling this function once for each - input/output operand on the instruction in the pattern with these - arguments. - - :param sym_typevar: Symbolic type variable constraining this variable - in the definition of the instruction. - :param sym_ctrl: Controlling type variable of `sym_typevar` in the - definition of the instruction. - :param ctrl_var: Variable determining the type of `sym_ctrl`. - - When processing `v1` as used in the pattern above, we would get: - - - self: v1 - - sym_typevar: TxN.as_bool() - - sym_ctrl: TxN - - ctrl_var: v2 - - Here, 'v2' represents the controlling variable because of how the - `Ternary` instruction format is defined with `typevar_operand=1`. - """ - # First check if sym_typevar is tied to the controlling type variable - # in the instruction definition. We also allow free type variables on - # instruction inputs that can't be tied to anything else. - # - # This also covers non-polymorphic instructions and other cases where - # we don't have a Var representing the controlling type variable. - sym_free_var = sym_typevar.free_typevar() - if not sym_free_var or sym_free_var is not sym_ctrl or not ctrl_var: - # Just constrain our type to be compatible with the required - # typeset. - self.get_typevar().constrain_types(sym_typevar) - return - - # Now sym_typevar is known to be tied to (or identical to) the - # controlling type variable. - - if not self.typevar: - # If this variable is not yet constrained, just infer its type and - # link it to the controlling type variable. - if not sym_typevar.is_derived: - assert sym_typevar is sym_ctrl - # Identity mapping. - # Note that `self == ctrl_var` is both possible and common. - self.typevar = ctrl_var.get_typevar() - else: - assert self is not ctrl_var, ( - 'Impossible type constraints for {}: {}' - .format(self, sym_typevar)) - # Create a derived type variable identical to sym_typevar, but - # with a different base. - self.typevar = TypeVar.derived( - ctrl_var.get_typevar(), - sym_typevar.derived_func) - # Match the type set constraints of the instruction. - self.typevar.constrain_types(sym_typevar) - return - - # We already have a self.typevar describing our constraints. We need to - # reconcile with the additional constraints. - - # It's likely that ctrl_var and self already share a type - # variable. (Often because `ctrl_var == self`). - if ctrl_var.typevar == self.typevar: - return - - if not sym_typevar.is_derived: - assert sym_typevar is sym_ctrl - # sym_typevar is a direct use of sym_ctrl, so we need to reconcile - # self with ctrl_var. - assert not sym_typevar.is_derived - self.typevar.constrain_types(sym_typevar) - - # It's possible that ctrl_var has not yet been assigned a type - # variable. - if not ctrl_var.typevar: - ctrl_var.typevar = self.typevar - return - - # We can also bind variables with a free type variable to another - # variable. Prefer to do this to temps because they aren't allowed - # to be free, - if self.is_temp() and self.has_free_typevar(): - self.link_typevar(ctrl_var.typevar, TypeVar.SAMEAS) - return - if ctrl_var.is_temp() and ctrl_var.has_free_typevar(): - ctrl_var.link_typevar(self.typevar, TypeVar.SAMEAS) - return - if self.has_free_typevar(): - self.link_typevar(ctrl_var.typevar, TypeVar.SAMEAS) - return - if ctrl_var.has_free_typevar(): - ctrl_var.link_typevar(self.typevar, TypeVar.SAMEAS) - return - - # TODO: Other cases are harder to handle. - # - # - If either variable is an independent free type variable, it - # should be changed to be linked to the other. - # - If both variable are free, we should pick one to link to the - # other. In particular, if one is a temp, it should be linked. - else: - # sym_typevar is derived from sym_ctrl. - # TODO: Other cases are harder to handle. - pass - class Apply(Expr): """ diff --git a/lib/cretonne/meta/cdsl/test_ti.py b/lib/cretonne/meta/cdsl/test_ti.py new file mode 100644 index 0000000000..d97902568a --- /dev/null +++ b/lib/cretonne/meta/cdsl/test_ti.py @@ -0,0 +1,432 @@ +from __future__ import absolute_import +from base.instructions import vselect, vsplit, vconcat, iconst, iadd, bint,\ + b1, icmp, iadd_cout, iadd_cin +from base.legalize import narrow, expand +from base.immediates import intcc +from .typevar import TypeVar +from .ast import Var, Def +from .xform import Rtl, XForm +from .ti import ti_rtl, subst, TypeEnv, get_type_env +from unittest import TestCase +from functools import reduce + +try: + from .ti import TypeMap, ConstraintList, VarMap, TypingOrError # noqa + from .ti import Constraint + from typing import List, Dict, Tuple, TYPE_CHECKING, cast # noqa +except ImportError: + TYPE_CHECKING = False + + +def sort_constr(c): + # type: (Constraint) -> Constraint + """ + Sort the 2 typevars in a constraint by name for comparison + """ + r = tuple(sorted(c, key=lambda y: y.name)) + if TYPE_CHECKING: + return cast(Constraint, r) + else: + return r + + +def agree(me, other): + # type: (TypeEnv, TypeEnv) -> bool + """ + Given TypeEnvs me and other, check if they agree. As part of that build + a map m from TVs in me to their corresponding TVs in other. + Specifically: + + 1. Check that all TVs that are keys in me.type_map are also defined + in other.type_map + + 2. For any tv in me.type_map check that: + me[tv].get_typeset() == other[tv].get_typeset() + + 3. Set m[me[tv]] = other[tv] in the substitution m + + 4. If we find another tv1 such that me[tv1] == me[tv], assert that + other[tv1] == m[me[tv1]] == m[me[tv]] = other[tv] + + 5. Check that me and other have the same constraints under the + substitution m + """ + m = {} # type: TypeMap + # Check that our type map and other's agree and built substitution m + for tv in me.type_map: + if (me[tv] not in m): + m[me[tv]] = other[tv] + if me[tv].get_typeset() != other[tv].get_typeset(): + return False + else: + if m[me[tv]] != other[tv]: + return False + + # Tranlsate our constraints using m, and sort + me_equiv_constr = [(subst(a, m), subst(b, m)) for (a, b) in me.constraints] + me_equiv_constr = sorted([sort_constr(x) for x in me_equiv_constr]) + + # Sort other's constraints + other_equiv_constr = sorted([sort_constr(x) for x in other.constraints], + key=lambda y: y[0].name) + + return me_equiv_constr == other_equiv_constr + + +def check_typing(got_or_err, expected, symtab=None): + # type: (TypingOrError, Tuple[VarMap, ConstraintList], Dict[str, Var]) -> None # noqa + """ + Check that a the typying we received (got_or_err) complies with the + expected typing (expected). If symtab is specified, substitute the Vars in + expected using symtab first (used when checking type inference on XForms) + """ + (m, c) = expected + got = get_type_env(got_or_err) + + if (symtab is not None): + # For xforms we first need to re-write our TVs in terms of the tvs + # stored internally in the XForm. Use the symtab passed + subst_m = {k.get_typevar(): symtab[str(k)].get_typevar() + for k in m.keys()} + # Convert m from a Var->TypeVar map to TypeVar->TypeVar map where + # the key TypeVar is re-written to its XForm internal version + tv_m = {subst(k.get_typevar(), subst_m): v for (k, v) in m.items()} + # Rewrite the TVs in the input constraints to their XForm internal + # versions + c = [(subst(a, subst_m), subst(b, subst_m)) for (a, b) in c] + else: + # If no symtab, just convert m from Var->TypeVar map to a + # TypeVar->TypeVar map + tv_m = {k.get_typevar(): v for (k, v) in m.items()} + + expected_typ = TypeEnv((tv_m, c)) + assert agree(expected_typ, got), \ + "typings disagree:\n {} \n {}".format(got.dot(), + expected_typ.dot()) + + +def check_concrete_typing_rtl(var_types, rtl): + # type: (VarMap, Rtl) -> None + """ + Check that a concrete type assignment var_types (Dict[Var, TypeVar]) is + valid for an Rtl rtl. Specifically check that: + + 1) For each Var v \in rtl, v is defined in var_types + + 2) For all v, var_types[v] is a singleton type + + 3) For each v, and each location u, where v is used with expected type + tv_u, var_types[v].get_typeset() is a subset of + subst(tv_u, m).get_typeset() where m is the substitution of + formals->actuals we are building so far. + + 4) If tv_u is non-derived and not in m, set m[tv_u]= var_types[v] + """ + for d in rtl.rtl: + assert isinstance(d, Def) + inst = d.expr.inst + # Accumulate all actual TVs for value defs/opnums in actual_tvs + actual_tvs = [var_types[d.defs[i]] for i in inst.value_results] + for v in [d.expr.args[i] for i in inst.value_opnums]: + assert isinstance(v, Var) + actual_tvs.append(var_types[v]) + + # Accumulate all formal TVs for value defs/opnums in actual_tvs + formal_tvs = [inst.outs[i].typevar for i in inst.value_results] +\ + [inst.ins[i].typevar for i in inst.value_opnums] + m = {} # type: TypeMap + + # For each actual/formal pair check that they agree + for (actual_tv, formal_tv) in zip(actual_tvs, formal_tvs): + # actual should be a singleton + assert actual_tv.singleton_type() is not None + formal_tv = subst(formal_tv, m) + # actual should agree with the concretized formal + assert actual_tv.get_typeset().issubset(formal_tv.get_typeset()) + + if formal_tv not in m and not formal_tv.is_derived: + m[formal_tv] = actual_tv + + +def check_concrete_typing_xform(var_types, xform): + # type: (VarMap, XForm) -> None + """ + Check a concrete type assignment var_types for an XForm xform + """ + check_concrete_typing_rtl(var_types, xform.src) + check_concrete_typing_rtl(var_types, xform.dst) + + +class TypeCheckingBaseTest(TestCase): + def setUp(self): + # type: () -> None + self.v0 = Var("v0") + self.v1 = Var("v1") + self.v2 = Var("v2") + self.v3 = Var("v3") + self.v4 = Var("v4") + self.v5 = Var("v5") + self.v6 = Var("v6") + self.v7 = Var("v7") + self.v8 = Var("v8") + self.v9 = Var("v9") + self.imm0 = Var("imm0") + self.IxN_nonscalar = TypeVar("IxN_nonscalar", "", ints=True, + scalars=False, simd=True) + self.TxN = TypeVar("TxN", "", ints=True, bools=True, floats=True, + scalars=False, simd=True) + self.b1 = TypeVar.singleton(b1) + + +class TestRTL(TypeCheckingBaseTest): + def test_bad_rtl1(self): + # type: () -> None + r = Rtl( + (self.v0, self.v1) << vsplit(self.v2), + self.v3 << vconcat(self.v0, self.v2), + ) + ti = TypeEnv() + self.assertEqual(ti_rtl(r, ti), + "On line 1: fail ti on `typeof_v2` <: `2`: " + + "Error: empty type created when unifying " + + "`typeof_v2` and `half_vector(typeof_v2)`") + + def test_vselect(self): + # type: () -> None + r = Rtl( + self.v0 << vselect(self.v1, self.v2, self.v3), + ) + ti = TypeEnv() + typing = ti_rtl(r, ti) + txn = self.TxN.get_fresh_copy("TxN1") + check_typing(typing, ({ + self.v0: txn, + self.v1: txn.as_bool(), + self.v2: txn, + self.v3: txn + }, [])) + + def test_vselect_icmpimm(self): + # type: () -> None + r = Rtl( + self.v0 << iconst(self.imm0), + self.v1 << icmp(intcc.eq, self.v2, self.v0), + self.v5 << vselect(self.v1, self.v3, self.v4), + ) + ti = TypeEnv() + typing = ti_rtl(r, ti) + ixn = self.IxN_nonscalar.get_fresh_copy("IxN1") + txn = self.TxN.get_fresh_copy("TxN1") + check_typing(typing, ({ + self.v0: ixn, + self.v1: ixn.as_bool(), + self.v2: ixn, + self.v3: txn, + self.v4: txn, + self.v5: txn, + }, [(ixn.as_bool(), txn.as_bool())])) + + def test_vselect_vsplits(self): + # type: () -> None + r = Rtl( + self.v3 << vselect(self.v0, self.v1, self.v2), + (self.v4, self.v5) << vsplit(self.v3), + (self.v6, self.v7) << vsplit(self.v4), + ) + ti = TypeEnv() + typing = ti_rtl(r, ti) + t = TypeVar("t", "", ints=True, bools=True, floats=True, + simd=(4, 256)) + check_typing(typing, ({ + self.v0: t.as_bool(), + self.v1: t, + self.v2: t, + self.v3: t, + self.v4: t.half_vector(), + self.v5: t.half_vector(), + self.v6: t.half_vector().half_vector(), + self.v7: t.half_vector().half_vector(), + }, [])) + + def test_vselect_vconcats(self): + # type: () -> None + r = Rtl( + self.v3 << vselect(self.v0, self.v1, self.v2), + self.v8 << vconcat(self.v3, self.v3), + self.v9 << vconcat(self.v8, self.v8), + ) + ti = TypeEnv() + typing = ti_rtl(r, ti) + t = TypeVar("t", "", ints=True, bools=True, floats=True, + simd=(2, 64)) + check_typing(typing, ({ + self.v0: t.as_bool(), + self.v1: t, + self.v2: t, + self.v3: t, + self.v8: t.double_vector(), + self.v9: t.double_vector().double_vector(), + }, [])) + + def test_vselect_vsplits_vconcats(self): + # type: () -> None + r = Rtl( + self.v3 << vselect(self.v0, self.v1, self.v2), + (self.v4, self.v5) << vsplit(self.v3), + (self.v6, self.v7) << vsplit(self.v4), + self.v8 << vconcat(self.v3, self.v3), + self.v9 << vconcat(self.v8, self.v8), + ) + ti = TypeEnv() + typing = ti_rtl(r, ti) + t = TypeVar("t", "", ints=True, bools=True, floats=True, + simd=(4, 64)) + check_typing(typing, ({ + self.v0: t.as_bool(), + self.v1: t, + self.v2: t, + self.v3: t, + self.v4: t.half_vector(), + self.v5: t.half_vector(), + self.v6: t.half_vector().half_vector(), + self.v7: t.half_vector().half_vector(), + self.v8: t.double_vector(), + self.v9: t.double_vector().double_vector(), + }, [])) + + def test_bint(self): + # type: () -> None + r = Rtl( + self.v4 << iadd(self.v1, self.v2), + self.v5 << bint(self.v3), + self.v0 << iadd(self.v4, self.v5) + ) + ti = TypeEnv() + typing = ti_rtl(r, ti) + itype = TypeVar("t", "", ints=True, simd=(1, 256)) + btype = TypeVar("b", "", bools=True, simd=True) + + # Check that self.v5 gets the same integer type as + # the rest of them + # TODO: Add constraint nlanes(v3) == nlanes(v1) when we + # add that type constraint to bint + check_typing(typing, ({ + self.v1: itype, + self.v2: itype, + self.v4: itype, + self.v5: itype, + self.v3: btype, + self.v0: itype, + }, [])) + + +class TestXForm(TypeCheckingBaseTest): + def test_iadd_cout(self): + # type: () -> None + x = XForm(Rtl((self.v0, self.v1) << iadd_cout(self.v2, self.v3),), + Rtl( + self.v0 << iadd(self.v2, self.v3), + self.v1 << icmp(intcc.ult, self.v0, self.v2) + )) + itype = TypeVar("t", "", ints=True, simd=(1, 1)) + + check_typing(x.ti, ({ + self.v0: itype, + self.v2: itype, + self.v3: itype, + self.v1: itype.as_bool(), + }, []), x.symtab) + + def test_iadd_cin(self): + # type: () -> None + x = XForm(Rtl(self.v0 << iadd_cin(self.v1, self.v2, self.v3)), + Rtl( + self.v4 << iadd(self.v1, self.v2), + self.v5 << bint(self.v3), + self.v0 << iadd(self.v4, self.v5) + )) + itype = TypeVar("t", "", ints=True, simd=(1, 1)) + + check_typing(x.ti, ({ + self.v0: itype, + self.v1: itype, + self.v2: itype, + self.v3: self.b1, + self.v4: itype, + self.v5: itype, + }, []), x.symtab) + + def test_enumeration_with_constraints(self): + # type: () -> None + xform = XForm( + Rtl( + self.v0 << iconst(self.imm0), + self.v1 << icmp(intcc.eq, self.v2, self.v0), + self.v5 << vselect(self.v1, self.v3, self.v4) + ), + Rtl( + self.v0 << iconst(self.imm0), + self.v1 << icmp(intcc.eq, self.v2, self.v0), + self.v5 << vselect(self.v1, self.v3, self.v4) + )) + + # Check all var assigns are correct + assert len(xform.ti.constraints) > 0 + concrete_var_assigns = list(xform.ti.concrete_typings()) + + v0 = xform.symtab[str(self.v0)] + v1 = xform.symtab[str(self.v1)] + v2 = xform.symtab[str(self.v2)] + v3 = xform.symtab[str(self.v3)] + v4 = xform.symtab[str(self.v4)] + v5 = xform.symtab[str(self.v5)] + + for var_m in concrete_var_assigns: + assert var_m[v0] == var_m[v2] and \ + var_m[v3] == var_m[v4] and\ + var_m[v5] == var_m[v3] and\ + var_m[v1] == var_m[v2].as_bool() and\ + var_m[v1].get_typeset() == var_m[v3].as_bool().get_typeset() + check_concrete_typing_xform(var_m, xform) + + # The number of possible typings here is: + # 8 cases for v0 = i8xN times 2 options for v3 - i8, b8 = 16 + # 8 cases for v0 = i16xN times 2 options for v3 - i16, b16 = 16 + # 8 cases for v0 = i32xN times 3 options for v3 - i32, b32, f32 = 24 + # 8 cases for v0 = i64xN times 3 options for v3 - i64, b64, f64 = 24 + # + # (Note we have 8 cases for lanes since vselect prevents scalars) + # Total: 2*16 + 2*24 = 80 + assert len(concrete_var_assigns) == 80 + + def test_base_legalizations_enumeration(self): + # type: () -> None + for xform in narrow.xforms + expand.xforms: + # Any legalization patterns we defined should have at least 1 + # concrete typing + concrete_typings_list = list(xform.ti.concrete_typings()) + assert len(concrete_typings_list) > 0 + + # If there are no free_typevars, this is a non-polymorphic pattern. + # There should be only one possible concrete typing. + if (len(xform.free_typevars) == 0): + assert len(concrete_typings_list) == 1 + continue + + # For any patterns where the type env includes constraints, at + # least one of the "theoretically possible" concrete typings must + # be prevented by the constraints. (i.e. we are not emitting + # unneccessary constraints). + # We check that by asserting that the number of concrete typings is + # less than the number of all possible free typevar assignments + if (len(xform.ti.constraints) > 0): + theoretical_num_typings =\ + reduce(lambda x, y: x*y, + [tv.get_typeset().size() + for tv in xform.free_typevars], 1) + assert len(concrete_typings_list) < theoretical_num_typings + + # Check the validity of each individual concrete typing against the + # xform + for concrete_typing in concrete_typings_list: + check_concrete_typing_xform(concrete_typing, xform) diff --git a/lib/cretonne/meta/cdsl/test_typevar.py b/lib/cretonne/meta/cdsl/test_typevar.py index f655c8966d..d990d7804f 100644 --- a/lib/cretonne/meta/cdsl/test_typevar.py +++ b/lib/cretonne/meta/cdsl/test_typevar.py @@ -125,59 +125,56 @@ class TestTypeSet(TestCase): self.assertEqual(TypeSet(lanes=(4, 4), ints=(32, 32)).get_singleton(), i32.by(4)) - def test_map_inverse(self): + def test_preimage(self): t = TypeSet(lanes=(1, 1), ints=(8, 8), floats=(32, 32)) - self.assertEqual(t, t.map_inverse(TypeVar.SAMEAS)) + self.assertEqual(t, t.preimage(TypeVar.SAMEAS)) # LANEOF self.assertEqual(TypeSet(lanes=True, ints=(8, 8), floats=(32, 32)), - t.map_inverse(TypeVar.LANEOF)) + t.preimage(TypeVar.LANEOF)) # Inverse of empty set is still empty across LANEOF self.assertEqual(TypeSet(), - TypeSet().map_inverse(TypeVar.LANEOF)) + TypeSet().preimage(TypeVar.LANEOF)) # ASBOOL t = TypeSet(lanes=(1, 4), bools=(1, 64)) - self.assertEqual(t.map_inverse(TypeVar.ASBOOL), + self.assertEqual(t.preimage(TypeVar.ASBOOL), TypeSet(lanes=(1, 4), ints=True, bools=True, floats=True)) - # Inverse image across ASBOOL of TS not involving b1 cannot have - # lanes=1 - t = TypeSet(lanes=(1, 4), bools=(16, 32)) - self.assertEqual(t.map_inverse(TypeVar.ASBOOL), - TypeSet(lanes=(2, 4), ints=(16, 32), bools=(16, 32), - floats=(32, 32))) - # Half/Double Vector t = TypeSet(lanes=(1, 1), ints=(8, 8)) t1 = TypeSet(lanes=(256, 256), ints=(8, 8)) - self.assertEqual(t.map_inverse(TypeVar.DOUBLEVECTOR).size(), 0) - self.assertEqual(t1.map_inverse(TypeVar.HALFVECTOR).size(), 0) + self.assertEqual(t.preimage(TypeVar.DOUBLEVECTOR).size(), 0) + self.assertEqual(t1.preimage(TypeVar.HALFVECTOR).size(), 0) t = TypeSet(lanes=(1, 16), ints=(8, 16), floats=(32, 32)) t1 = TypeSet(lanes=(64, 256), bools=(1, 32)) - self.assertEqual(t.map_inverse(TypeVar.DOUBLEVECTOR), + self.assertEqual(t.preimage(TypeVar.DOUBLEVECTOR), TypeSet(lanes=(1, 8), ints=(8, 16), floats=(32, 32))) - self.assertEqual(t1.map_inverse(TypeVar.HALFVECTOR), + self.assertEqual(t1.preimage(TypeVar.HALFVECTOR), TypeSet(lanes=(128, 256), bools=(1, 32))) # Half/Double Width t = TypeSet(ints=(8, 8), floats=(32, 32), bools=(1, 8)) t1 = TypeSet(ints=(64, 64), floats=(64, 64), bools=(64, 64)) - self.assertEqual(t.map_inverse(TypeVar.DOUBLEWIDTH).size(), 0) - self.assertEqual(t1.map_inverse(TypeVar.HALFWIDTH).size(), 0) + self.assertEqual(t.preimage(TypeVar.DOUBLEWIDTH).size(), 0) + self.assertEqual(t1.preimage(TypeVar.HALFWIDTH).size(), 0) t = TypeSet(lanes=(1, 16), ints=(8, 16), floats=(32, 64)) t1 = TypeSet(lanes=(64, 256), bools=(1, 64)) - self.assertEqual(t.map_inverse(TypeVar.DOUBLEWIDTH), + self.assertEqual(t.preimage(TypeVar.DOUBLEWIDTH), TypeSet(lanes=(1, 16), ints=(8, 8), floats=(32, 32))) - self.assertEqual(t1.map_inverse(TypeVar.HALFWIDTH), + self.assertEqual(t1.preimage(TypeVar.HALFWIDTH), TypeSet(lanes=(64, 256), bools=(16, 64))) +def has_non_bijective_derived_f(iterable): + return any(not TypeVar.is_bijection(x) for x in iterable) + + class TestTypeVar(TestCase): def test_functions(self): x = TypeVar('x', 'all ints', ints=True) @@ -220,7 +217,7 @@ class TestTypeVar(TestCase): self.assertEqual(len(x.type_set.bools), 0) def test_stress_constrain_types(self): - # Get all 49 possible derived vars of lentgh 2. Since we have SAMEAS + # Get all 49 possible derived vars of length 2. Since we have SAMEAS # this includes singly derived and non-derived vars funcs = [TypeVar.SAMEAS, TypeVar.LANEOF, TypeVar.ASBOOL, TypeVar.HALFVECTOR, TypeVar.DOUBLEVECTOR, @@ -231,18 +228,18 @@ class TestTypeVar(TestCase): for (i1, i2) in product(v, v): # Compute the derived sets for each starting with a full typeset full_ts = TypeSet(lanes=True, floats=True, ints=True, bools=True) - ts1 = reduce(lambda ts, func: ts.map(func), i1, full_ts) - ts2 = reduce(lambda ts, func: ts.map(func), i2, full_ts) + ts1 = reduce(lambda ts, func: ts.image(func), i1, full_ts) + ts2 = reduce(lambda ts, func: ts.image(func), i2, full_ts) # Compute intersection intersect = ts1.copy() intersect &= ts2 # Propagate instersections backward - ts1_src = reduce(lambda ts, func: ts.map_inverse(func), + ts1_src = reduce(lambda ts, func: ts.preimage(func), reversed(i1), intersect) - ts2_src = reduce(lambda ts, func: ts.map_inverse(func), + ts2_src = reduce(lambda ts, func: ts.preimage(func), reversed(i2), intersect) @@ -262,13 +259,10 @@ class TestTypeVar(TestCase): i2, TypeVar.from_typeset(ts2_src)) - # The typesets of the two derived variables should be subsets of - # the intersection we computed originally - assert tv1.get_typeset().issubset(intersect) - assert tv2.get_typeset().issubset(intersect) - - # In the absence of AS_BOOL map(map_inverse(f)) == f so the + # In the absence of AS_BOOL image(preimage(f)) == f so the # typesets of tv1 and tv2 should be exactly intersection - assert (tv1.get_typeset() == tv2.get_typeset() and - tv1.get_typeset() == intersect) or\ - TypeVar.ASBOOL in set(i1 + i2) + assert tv1.get_typeset() == intersect or\ + has_non_bijective_derived_f(i1) + + assert tv2.get_typeset() == intersect or\ + has_non_bijective_derived_f(i2) diff --git a/lib/cretonne/meta/cdsl/ti.py b/lib/cretonne/meta/cdsl/ti.py new file mode 100644 index 0000000000..5b376f1e5d --- /dev/null +++ b/lib/cretonne/meta/cdsl/ti.py @@ -0,0 +1,556 @@ +""" +Type Inference +""" +from .typevar import TypeVar +from .ast import Def, Var +from copy import copy +from itertools import product + +try: + from typing import Dict, TYPE_CHECKING, Union, Tuple, Optional, Set # noqa + from typing import Iterable # noqa + from typing import cast, List + from .xform import Rtl, XForm # noqa + from .ast import Expr # noqa + if TYPE_CHECKING: + Constraint = Tuple[TypeVar, TypeVar] + ConstraintList = List[Constraint] + TypeMap = Dict[TypeVar, TypeVar] + VarMap = Dict[Var, TypeVar] +except ImportError: + TYPE_CHECKING = False + pass + + +class TypeEnv(object): + """ + Class encapsulating the neccessary book keeping for type inference. + :attribute type_map: dict holding the equivalence relations between tvs + :attribute constraints: a list of accumulated constraints - tuples + (tv1, tv2)) where tv1 and tv2 are equal + :attribute ranks: dictionary recording the (optional) ranks for tvs. + tvs corresponding to real variables have explicitly + specified ranks. + :attribute vars: a set containing all known Vars + :attribute idx: counter used to get fresh ids + """ + def __init__(self, arg=None): + # type: (Optional[Tuple[TypeMap, ConstraintList]]) -> None + self.ranks = {} # type: Dict[TypeVar, int] + self.vars = set() # type: Set[Var] + + if arg is None: + self.type_map = {} # type: TypeMap + self.constraints = [] # type: ConstraintList + else: + self.type_map, self.constraints = arg + + self.idx = 0 + + def __getitem__(self, arg): + # type: (Union[TypeVar, Var]) -> TypeVar + """ + Lookup the canonical representative for a Var/TypeVar. + """ + if (isinstance(arg, Var)): + tv = arg.get_typevar() + else: + assert (isinstance(arg, TypeVar)) + tv = arg + + while tv in self.type_map: + tv = self.type_map[tv] + + if tv.is_derived: + tv = TypeVar.derived(self[tv.base], tv.derived_func) + return tv + + def equivalent(self, tv1, tv2): + # type: (TypeVar, TypeVar) -> None + """ + Record a that the free tv1 is part of the same equivalence class as + tv2. The canonical representative of the merged class is tv2's + cannonical representative. + """ + assert not tv1.is_derived + assert self[tv1] == tv1 + + # Make sure we don't create cycles + if tv2.is_derived: + assert self[tv2.base] != tv1 + + self.type_map[tv1] = tv2 + + def add_constraint(self, tv1, tv2): + # type: (TypeVar, TypeVar) -> None + """ + Add a new equivalence constraint between tv1 and tv2 + """ + self.constraints.append((tv1, tv2)) + + def get_uid(self): + # type: () -> str + r = str(self.idx) + self.idx += 1 + return r + + def __repr__(self): + # type: () -> str + return self.dot() + + def rank(self, tv): + # type: (TypeVar) -> int + """ + Get the rank of tv in the partial order. TVs directly associated with a + Var get their rank from the Var (see register()). + Internally generated non-derived TVs implicitly get the lowest rank (0) + Internal derived variables get the highest rank. + """ + default_rank = 5 if tv.is_derived else 0 + return self.ranks.get(tv, default_rank) + + def register(self, v): + # type: (Var) -> None + """ + Register a new Var v. This computes a rank for the associated TypeVar + for v, which is used to impose a partial order on type variables. + """ + self.vars.add(v) + + if v.is_input(): + r = 4 + elif v.is_intermediate(): + r = 3 + elif v.is_output(): + r = 2 + else: + assert(v.is_temp()) + r = 1 + + self.ranks[v.get_typevar()] = r + + def free_typevars(self): + # type: () -> Set[TypeVar] + """ + Get the free typevars in the current type env. + """ + tvs = set([self[tv].free_typevar() for tv in self.type_map.keys()]) + # Filter out None here due to singleton type vars + return set(filter(lambda x: x is not None, tvs)) + + def normalize(self): + # type: () -> None + """ + Normalize by: + - collapsing any roots that don't correspond to a concrete TV AND + have a single TV derived from them or equivalent to them + + E.g. if we have a root of the tree that looks like: + + typeof_a typeof_b + \ / + typeof_x + | + half_width(1) + | + 1 + + we want to collapse the linear path between 1 and typeof_x. The + resulting graph is: + + typeof_a typeof_b + \ / + typeof_x + """ + source_tvs = set([v.get_typevar() for v in self.vars]) + children = {} # type: Dict[TypeVar, Set[TypeVar]] + for v in self.type_map.values(): + if not v.is_derived: + continue + + t = v.free_typevar() + s = children.get(t, set()) + s.add(v) + children[t] = s + + for (a, b) in self.type_map.items(): + s = children.get(b, set()) + s.add(a) + children[b] = s + + for r in list(self.free_typevars()): + while (r not in source_tvs and r in children and + len(children[r]) == 1): + child = list(children[r])[0] + if child in self.type_map: + assert self.type_map[child] == r + del self.type_map[child] + + r = child + + def extract(self): + # type: () -> TypeEnv + """ + Extract a clean type environment from self, that only mentions + TVs associated with real variables + """ + vars_tvs = set([v.get_typevar() for v in self.vars]) + new_type_map = {tv: self[tv] for tv in vars_tvs if tv != self[tv]} + new_constraints = [(self[tv1], self[tv2]) + for (tv1, tv2) in self.constraints] + + # Sanity: new constraints and the new type_map should only contain + # tvs associated with real vars + for (a, b) in new_constraints: + assert a.free_typevar() in vars_tvs and\ + b.free_typevar() in vars_tvs + + for (k, v) in new_type_map.items(): + assert k in vars_tvs + assert v.free_typevar() is None or v.free_typevar() in vars_tvs + + t = TypeEnv() + t.type_map = new_type_map + t.constraints = new_constraints + # ranks and vars contain only TVs associated with real vars + t.ranks = copy(self.ranks) + t.vars = copy(self.vars) + return t + + def concrete_typings(self): + # type: () -> Iterable[VarMap] + """ + Return an iterable over all possible concrete typings permitted by this + TypeEnv. + """ + free_tvs = self.free_typevars() + free_tv_iters = [tv.get_typeset().concrete_types() for tv in free_tvs] + for concrete_types in product(*free_tv_iters): + # Build type substitutions for all free vars + m = {tv: TypeVar.singleton(typ) + for (tv, typ) in zip(free_tvs, concrete_types)} + + concrete_var_map = {v: subst(self[v.get_typevar()], m) + for v in self.vars} + + # Check if constraints are satisfied for this typing + failed = None + for (tv1, tv2) in self.constraints: + tv1 = subst(tv1, m) + tv2 = subst(tv2, m) + assert tv1.get_typeset().size() == 1 and\ + tv2.get_typeset().size() == 1 + if (tv1.get_typeset() != tv2.get_typeset()): + failed = (tv1, tv2) + break + + if (failed is not None): + continue + + yield concrete_var_map + + def dot(self): + # type: () -> str + """ + Return a representation of self as a graph in dot format. + Nodes correspond to TypeVariables. + Dotted edges correspond to equivalences between TVS + Solid edges correspond to derivation relations between TVs. + Dashed edges correspond to equivalence constraints. + """ + def label(s): + # type: (TypeVar) -> str + return "\"" + str(s) + "\"" + + # Add all registered TVs (as some of them may be singleton nodes not + # appearing in the graph + nodes = set([v.get_typevar() for v in self.vars]) # type: Set[TypeVar] + edges = set() # type: Set[Tuple[TypeVar, TypeVar, str, Optional[str]]] + + for (k, v) in self.type_map.items(): + # Add all intermediate TVs appearing in edges + nodes.add(k) + nodes.add(v) + edges.add((k, v, "dotted", None)) + while (v.is_derived): + nodes.add(v.base) + edges.add((v, v.base, "solid", v.derived_func)) + v = v.base + + for (a, b) in self.constraints: + assert a in nodes and b in nodes + edges.add((a, b, "dashed", None)) + + root_nodes = set([x for x in nodes + if x not in self.type_map and not x.is_derived]) + + r = "digraph {\n" + for n in nodes: + r += label(n) + if n in root_nodes: + r += "[xlabel=\"{}\"]".format(self[n].get_typeset()) + r += ";\n" + + for (n1, n2, style, elabel) in edges: + e = label(n1) + if style == "dashed": + e += '--' + else: + e += '->' + e += label(n2) + e += "[style={}".format(style) + + if elabel is not None: + e += ",label={}".format(elabel) + e += "];\n" + + r += e + r += "}" + + return r + + +if TYPE_CHECKING: + TypingError = str + TypingOrError = Union[TypeEnv, TypingError] + + +def get_error(typing_or_err): + # type: (TypingOrError) -> Optional[TypingError] + """ + Helper function to appease mypy when checking the result of typing. + """ + if isinstance(typing_or_err, str): + if (TYPE_CHECKING): + return cast(TypingError, typing_or_err) + else: + return typing_or_err + else: + return None + + +def get_type_env(typing_or_err): + # type: (TypingOrError) -> TypeEnv + """ + Helper function to appease mypy when checking the result of typing. + """ + assert isinstance(typing_or_err, TypeEnv) + if (TYPE_CHECKING): + return cast(TypeEnv, typing_or_err) + else: + return typing_or_err + + +def subst(tv, tv_map): + # type: (TypeVar, TypeMap) -> TypeVar + """ + Perform substition on the input tv using the TypeMap tv_map. + """ + if tv in tv_map: + return tv_map[tv] + + if tv.is_derived: + return TypeVar.derived(subst(tv.base, tv_map), tv.derived_func) + + return tv + + +def normalize_tv(tv): + # type: (TypeVar) -> TypeVar + """ + Normalize a (potentially derived) TV using the following rules: + - collapse SAMEAS + SAMEAS(base) -> base + + - vector and width derived functions commute + {HALF,DOUBLE}VECTOR({HALF,DOUBLE}WIDTH(base)) -> + {HALF,DOUBLE}WIDTH({HALF,DOUBLE}VECTOR(base)) + + - half/double pairs collapse + {HALF,DOUBLE}WIDTH({DOUBLE,HALF}WIDTH(base)) -> base + {HALF,DOUBLE}VECTOR({DOUBLE,HALF}VECTOR(base)) -> base + """ + vector_derives = [TypeVar.HALFVECTOR, TypeVar.DOUBLEVECTOR] + width_derives = [TypeVar.HALFWIDTH, TypeVar.DOUBLEWIDTH] + + if not tv.is_derived: + return tv + + df = tv.derived_func + + # Collapse SAMEAS edges + if (df == TypeVar.SAMEAS): + return normalize_tv(tv.base) + + if (tv.base.is_derived): + base_df = tv.base.derived_func + + # Reordering: {HALFWIDTH, DOUBLEWIDTH} commute with {HALFVECTOR, + # DOUBLEVECTOR}. Arbitrarily pick WIDTH < VECTOR + if df in vector_derives and base_df in width_derives: + return normalize_tv( + TypeVar.derived( + TypeVar.derived(tv.base.base, df), base_df)) + + # Cancelling: HALFWIDTH, DOUBLEWIDTH and HALFVECTOR, DOUBLEVECTOR + # cancel each other. TODO: Does this cancellation hide type + # overflow/underflow? + + if (df, base_df) in \ + [(TypeVar.HALFVECTOR, TypeVar.DOUBLEVECTOR), + (TypeVar.DOUBLEVECTOR, TypeVar.HALFVECTOR), + (TypeVar.HALFWIDTH, TypeVar.DOUBLEWIDTH), + (TypeVar.DOUBLEWIDTH, TypeVar.HALFWIDTH)]: + return normalize_tv(tv.base.base) + + return TypeVar.derived(normalize_tv(tv.base), df) + + +def constrain_fixpoint(tv1, tv2): + # type: (TypeVar, TypeVar) -> None + """ + Given typevars tv1 and tv2 (which could be derived from one another) + constrain their typesets to be the same. When one is derived from the + other, repeat the constrain process until fixpoint. + """ + # Constrain tv2's typeset as long as tv1's typeset is changing. + while True: + old_tv1_ts = tv1.get_typeset().copy() + tv2.constrain_types(tv1) + if tv1.get_typeset() == old_tv1_ts: + break + + old_tv2_ts = tv2.get_typeset().copy() + tv1.constrain_types(tv2) + assert old_tv2_ts == tv2.get_typeset() + + +def unify(tv1, tv2, typ): + # type: (TypeVar, TypeVar, TypeEnv) -> TypingOrError + """ + Unify tv1 and tv2 in the current type environment typ, and return an + updated type environment or error. + """ + tv1 = normalize_tv(typ[tv1]) + tv2 = normalize_tv(typ[tv2]) + + # Already unified + if tv1 == tv2: + return typ + + if typ.rank(tv2) < typ.rank(tv1): + return unify(tv2, tv1, typ) + + constrain_fixpoint(tv1, tv2) + + if (tv1.get_typeset().size() == 0 or tv2.get_typeset().size() == 0): + return "Error: empty type created when unifying {} and {}"\ + .format(tv1, tv2) + + # Free -> Derived(Free) + if not tv1.is_derived: + typ.equivalent(tv1, tv2) + return typ + + assert tv2.is_derived, "Ordering gives us !tv1.is_derived==>tv2.is_derived" + + if (tv1.is_derived and TypeVar.is_bijection(tv1.derived_func)): + inv_f = TypeVar.inverse_func(tv1.derived_func) + return unify(tv1.base, normalize_tv(TypeVar.derived(tv2, inv_f)), typ) + + typ.add_constraint(tv1, tv2) + return typ + + +def ti_def(definition, typ): + # type: (Def, TypeEnv) -> TypingOrError + """ + Perform type inference on one Def in the current type environment typ and + return an updated type environment or error. + + At a high level this works by creating fresh copies of each formal type var + in the Def's instruction's signature, and unifying the formal tv with the + corresponding actual tv. + """ + expr = definition.expr + inst = expr.inst + + # Create a map m mapping each free typevar in the signature of definition + # to a fresh copy of itself + all_formal_tvs = \ + [inst.outs[i].typevar for i in inst.value_results] +\ + [inst.ins[i].typevar for i in inst.value_opnums] + free_formal_tvs = [tv for tv in all_formal_tvs if not tv.is_derived] + m = {tv: tv.get_fresh_copy(str(typ.get_uid())) for tv in free_formal_tvs} + + # Get fresh copies for each typevar in the signature (both free and + # derived) + fresh_formal_tvs = \ + [subst(inst.outs[i].typevar, m) for i in inst.value_results] +\ + [subst(inst.ins[i].typevar, m) for i in inst.value_opnums] + + # Get the list of actual Vars + actual_vars = [] # type: List[Expr] + actual_vars += [definition.defs[i] for i in inst.value_results] + actual_vars += [expr.args[i] for i in inst.value_opnums] + + # Get the list of the actual TypeVars + actual_tvs = [] + for v in actual_vars: + assert(isinstance(v, Var)) + # Register with TypeEnv that this typevar corresponds ot variable v, + # and thus has a given rank + typ.register(v) + actual_tvs.append(v.get_typevar()) + + # Unify each actual typevar with the correpsonding fresh formal tv + for (actual_tv, formal_tv) in zip(actual_tvs, fresh_formal_tvs): + typ_or_err = unify(actual_tv, formal_tv, typ) + err = get_error(typ_or_err) + if (err): + return "fail ti on {} <: {}: ".format(actual_tv, formal_tv) + err + + typ = get_type_env(typ_or_err) + + return typ + + +def ti_rtl(rtl, typ): + # type: (Rtl, TypeEnv) -> TypingOrError + """ + Perform type inference on an Rtl in a starting type env typ. Return an + updated type environment or error. + """ + for (i, d) in enumerate(rtl.rtl): + assert (isinstance(d, Def)) + typ_or_err = ti_def(d, typ) + err = get_error(typ_or_err) # type: Optional[TypingError] + if (err): + return "On line {}: ".format(i) + err + + typ = get_type_env(typ_or_err) + + return typ + + +def ti_xform(xform, typ): + # type: (XForm, TypeEnv) -> TypingOrError + """ + Perform type inference on an Rtl in a starting type env typ. Return an + updated type environment or error. + """ + typ_or_err = ti_rtl(xform.src, typ) + err = get_error(typ_or_err) # type: Optional[TypingError] + if (err): + return "In src pattern: " + err + + typ = get_type_env(typ_or_err) + + typ_or_err = ti_rtl(xform.dst, typ) + err = get_error(typ_or_err) + if (err): + return "In dst pattern: " + err + + typ = get_type_env(typ_or_err) + + return get_type_env(typ_or_err) diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 69b419bfa3..1ea9831b91 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -8,7 +8,6 @@ from __future__ import absolute_import import math from . import types, is_power_of_two from copy import deepcopy -from .types import IntType, FloatType, BoolType try: from typing import Tuple, Union, Iterable, Any, Set, TYPE_CHECKING # noqa @@ -210,6 +209,10 @@ class TypeSet(object): else: return False + def __ne__(self, other): + # type: (object) -> bool + return not self.__eq__(other) + def __repr__(self): # type: () -> str s = 'TypeSet(lanes={}'.format(pp_set(self.lanes)) @@ -289,7 +292,9 @@ class TypeSet(object): new = self.copy() new.ints = set() new.floats = set() - new.bools = self.ints.union(self.floats).union(self.bools) + + if len(self.lanes.difference(set([1]))) > 0: + new.bools = self.ints.union(self.floats).union(self.bools) if 1 in self.lanes: new.bools.add(1) @@ -340,7 +345,7 @@ class TypeSet(object): return new - def map(self, func): + def image(self, func): # type: (str) -> TypeSet """ Return the image of self across the derived function func @@ -362,7 +367,7 @@ class TypeSet(object): else: assert False, "Unknown derived function: " + func - def map_inverse(self, func): + def preimage(self, func): # type: (str) -> TypeSet """ Return the inverse image of self across the derived function func @@ -379,16 +384,14 @@ class TypeSet(object): return new elif (func == TypeVar.ASBOOL): new = self.copy() - new.ints = self.bools.difference(set([1])) - new.floats = self.bools.intersection(set([32, 64])) if 1 not in self.bools: - try: - # If the range doesn't have b1, then the domain can't - # include scalars, as as_bool(scalar)=b1 - new.lanes.remove(1) - except KeyError: - pass + new.ints = self.bools.difference(set([1])) + new.floats = self.bools.intersection(set([32, 64])) + else: + new.ints = set([2**x for x in range(3, 7)]) + new.floats = set([32, 64]) + return new elif (func == TypeVar.HALFWIDTH): return self.double_width() @@ -409,27 +412,32 @@ class TypeSet(object): return len(self.lanes) * (len(self.ints) + len(self.floats) + len(self.bools)) + def concrete_types(self): + # type: () -> Iterable[types.ValueType] + def by(scalar, lanes): + # type: (types.ScalarType, int) -> types.ValueType + if (lanes == 1): + return scalar + else: + return scalar.by(lanes) + + for nlanes in self.lanes: + for bits in self.ints: + yield by(types.IntType.with_bits(bits), nlanes) + for bits in self.floats: + yield by(types.FloatType.with_bits(bits), nlanes) + for bits in self.bools: + yield by(types.BoolType.with_bits(bits), nlanes) + def get_singleton(self): # type: () -> types.ValueType """ Return the singleton type represented by self. Can only call on typesets containing 1 type. """ - assert self.size() == 1 - scalar_type = None # type: types.ScalarType - if len(self.ints) > 0: - scalar_type = IntType.with_bits(tuple(self.ints)[0]) - elif len(self.floats) > 0: - scalar_type = FloatType.with_bits(tuple(self.floats)[0]) - else: - scalar_type = BoolType.with_bits(tuple(self.bools)[0]) - - nlanes = tuple(self.lanes)[0] - - if nlanes == 1: - return scalar_type - else: - return scalar_type.by(nlanes) + types = list(self.concrete_types()) + assert len(types) == 1 + return types[0] class TypeVar(object): @@ -519,6 +527,13 @@ class TypeVar(object): 'TypeVar({}, {})' .format(self.name, self.type_set)) + def __hash__(self): + # type: () -> int + if (not self.is_derived): + return object.__hash__(self) + + return hash((self.derived_func, self.base)) + def __eq__(self, other): # type: (object) -> bool if not isinstance(other, TypeVar): @@ -530,6 +545,10 @@ class TypeVar(object): else: return self is other + def __ne__(self, other): + # type: (object) -> bool + return not self.__eq__(other) + # Supported functions for derived type variables. # The names here must match the method names on `ir::types::Type`. # The camel_case of the names must match `enum OperandConstraint` in @@ -542,6 +561,27 @@ class TypeVar(object): HALFVECTOR = 'half_vector' DOUBLEVECTOR = 'double_vector' + @staticmethod + def is_bijection(func): + # type: (str) -> bool + return func in [ + TypeVar.SAMEAS, + TypeVar.HALFWIDTH, + TypeVar.DOUBLEWIDTH, + TypeVar.HALFVECTOR, + TypeVar.DOUBLEVECTOR] + + @staticmethod + def inverse_func(func): + # type: (str) -> str + return { + TypeVar.SAMEAS: TypeVar.SAMEAS, + TypeVar.HALFWIDTH: TypeVar.DOUBLEWIDTH, + TypeVar.DOUBLEWIDTH: TypeVar.HALFWIDTH, + TypeVar.HALFVECTOR: TypeVar.DOUBLEVECTOR, + TypeVar.DOUBLEVECTOR: TypeVar.HALFVECTOR + }[func] + @staticmethod def derived(base, derived_func): # type: (TypeVar, str) -> TypeVar @@ -668,7 +708,7 @@ class TypeVar(object): Get the free type variable controlling this one. """ if self.is_derived: - return self.base + return self.base.free_typevar() elif self.singleton_type() is not None: # A singleton type variable is not a proper free variable. return None @@ -697,7 +737,7 @@ class TypeVar(object): if not self.is_derived: self.type_set &= ts else: - self.base.constrain_types_by_ts(ts.map_inverse(self.derived_func)) + self.base.constrain_types_by_ts(ts.preimage(self.derived_func)) def constrain_types(self, other): # type: (TypeVar) -> None @@ -723,19 +763,14 @@ class TypeVar(object): if not self.is_derived: return self.type_set else: - if (self.derived_func == TypeVar.SAMEAS): - return self.base.get_typeset() - elif (self.derived_func == TypeVar.LANEOF): - return self.base.get_typeset().lane_of() - elif (self.derived_func == TypeVar.ASBOOL): - return self.base.get_typeset().as_bool() - elif (self.derived_func == TypeVar.HALFWIDTH): - return self.base.get_typeset().half_width() - elif (self.derived_func == TypeVar.DOUBLEWIDTH): - return self.base.get_typeset().double_width() - elif (self.derived_func == TypeVar.HALFVECTOR): - return self.base.get_typeset().half_vector() - elif (self.derived_func == TypeVar.DOUBLEVECTOR): - return self.base.get_typeset().double_vector() - else: - assert False, "Unknown derived function: " + self.derived_func + return self.base.get_typeset().image(self.derived_func) + + def get_fresh_copy(self, name): + # type: (str) -> TypeVar + """ + Get a fresh copy of self. Can only be called on free typevars. + """ + assert not self.is_derived + tv = TypeVar.from_typeset(self.type_set.copy()) + tv.name = name + return tv diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index 9ce93c9ed9..a8fbb7f66e 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -3,6 +3,7 @@ Instruction transformations. """ from __future__ import absolute_import from .ast import Def, Var, Apply +from .ti import ti_xform, TypeEnv, get_type_env try: from typing import Union, Iterator, Sequence, Iterable, List, Dict # noqa @@ -83,6 +84,8 @@ class XForm(object): self._rewrite_rtl(src, symtab, Var.SRCCTX) num_src_inputs = len(self.inputs) self._rewrite_rtl(dst, symtab, Var.DSTCTX) + # Needed for testing type inference on XForms + self.symtab = symtab # Check for inconsistently used inputs. for i in self.inputs: @@ -96,9 +99,25 @@ class XForm(object): "extra inputs in dst RTL: {}".format( self.inputs[num_src_inputs:])) - self._infer_types(self.src) - self._infer_types(self.dst) - self._collect_typevars() + # Perform type inference and cleanup + raw_ti = get_type_env(ti_xform(self, TypeEnv())) + raw_ti.normalize() + self.ti = raw_ti.extract() + + # Sanity: The set of inferred free typevars should be a subset of the + # TVs corresponding to Vars appearing in src + self.free_typevars = self.ti.free_typevars() + src_vars = set(self.inputs).union( + [x for x in self.defs if not x.is_temp()]) + src_tvs = set([v.get_typevar() for v in src_vars]) + if (not self.free_typevars.issubset(src_tvs)): + raise AssertionError( + "Some free vars don't appear in src - {}" + .format(self.free_typevars.difference(src_tvs))) + + # Update the type vars for each Var to their inferred values + for v in self.inputs + self.defs: + v.set_typevar(self.ti[v.get_typevar()]) def __repr__(self): # type: () -> str @@ -202,63 +221,6 @@ class XForm(object): raise AssertionError( '{} not defined in dest pattern'.format(d)) - def _infer_types(self, rtl): - # type: (Rtl) -> None - """Assign type variables to all value variables used in `rtl`.""" - for d in rtl.rtl: - inst = d.expr.inst - - # Get the Var corresponding to the controlling type variable. - ctrl_var = None # type: Var - if inst.is_polymorphic: - if inst.use_typevar_operand: - # Should this be an assertion instead? - # Should all value operands be required to be Vars? - arg = d.expr.args[inst.format.typevar_operand] - if isinstance(arg, Var): - ctrl_var = arg - else: - ctrl_var = d.defs[inst.value_results[0]] - - # Reconcile arguments with the requirements of `inst`. - for opnum in inst.value_opnums: - inst_tv = inst.ins[opnum].typevar - v = d.expr.args[opnum] - if isinstance(v, Var): - v.constrain_typevar(inst_tv, inst.ctrl_typevar, ctrl_var) - - # Reconcile results with the requirements of `inst`. - for resnum in inst.value_results: - inst_tv = inst.outs[resnum].typevar - v = d.defs[resnum] - v.constrain_typevar(inst_tv, inst.ctrl_typevar, ctrl_var) - - def _collect_typevars(self): - # type: () -> None - """ - Collect a list of variables whose type can be used to infer the types - of all expressions. - - This should be called after `_infer_types()` above has computed type - variables for all the used vars. - """ - fvars = list(v for v in self.inputs if v.has_free_typevar()) - fvars += list(v for v in self.defs if v.has_free_typevar()) - self.free_typevars = fvars - - # When substituting a pattern, we know the types of all variables that - # appear on the source side: inut, output, and intermediate values. - # However, temporary values which appear only on the destination side - # must have their type computed somehow. - # - # Some variables have a fixed type which appears as a type variable - # with a singleton_type field set. That's allowed for temps too. - for v in fvars: - if v.is_temp() and not v.typevar.singleton_type(): - raise AssertionError( - "Cannot determine type of temp '{}' in xform:\n{}" - .format(v, self)) - class XFormGroup(object): """ From 9108716cb78349d8d218b1b1921be1a39e579bbd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 30 Jun 2017 19:09:10 -0700 Subject: [PATCH 0842/3084] Implement fmt::Display for AllocatableSet. Also add a display() method which accepts a RegInfo reference. --- lib/cretonne/src/regalloc/allocatable_set.rs | 48 +++++++++++++++++++- lib/cretonne/src/regalloc/coloring.rs | 4 +- 2 files changed, 50 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index aecd338e7f..063c8e63ac 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -5,7 +5,8 @@ //! "register unit" abstraction. Every register contains one or more register units. Registers that //! share a register unit can't be in use at the same time. -use isa::registers::{RegUnit, RegUnitMask, RegClass}; +use isa::registers::{RegInfo, RegUnit, RegUnitMask, RegClass}; +use std::fmt; use std::iter::ExactSizeIterator; use std::mem::size_of_val; @@ -98,6 +99,12 @@ impl AllocatableSet { *x &= y; } } + + /// Return an object that can display this register set, using the register info from the + /// target ISA. + pub fn display<'a, R: Into>>(&self, regs: R) -> DisplayAllocatableSet<'a> { + DisplayAllocatableSet(self.clone(), regs.into()) + } } /// Iterator over available registers in a register class. @@ -138,6 +145,45 @@ impl Iterator for RegSetIter { impl ExactSizeIterator for RegSetIter {} +/// Displaying an `AllocatableSet` correctly requires the associated `RegInfo` from the target ISA. +pub struct DisplayAllocatableSet<'a>(AllocatableSet, Option<&'a RegInfo>); + +impl<'a> fmt::Display for DisplayAllocatableSet<'a> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "[")?; + match self.1 { + None => { + for w in &self.0.avail { + write!(f, " #{:08x}", w)?; + } + } + Some(reginfo) => { + let toprcs = reginfo + .banks + .iter() + .map(|b| b.first_toprc + b.num_toprcs) + .max() + .expect("No register banks"); + for rc in ®info.classes[0..toprcs] { + if rc.width == 1 { + write!(f, " {}:", rc)?; + for u in self.0.iter(rc) { + write!(f, " {}", reginfo.display_regunit(u))?; + } + } + } + } + } + write!(f, " ]") + } +} + +impl fmt::Display for AllocatableSet { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + self.display(None).fmt(f) + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 4ebf3340f9..e4e5a6599f 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -278,7 +278,9 @@ impl<'a> Context<'a> { regs: &mut AllocatableSet, locations: &mut ValueLocations, func_signature: &Signature) { - dbg!("Coloring {}", dfg.display_inst(inst)); + dbg!("Coloring {}\n {}", + dfg.display_inst(inst), + regs.display(&self.reginfo)); // EBB whose arguments should be colored to match the current branch instruction's // arguments. From e7db3f2b3a401e3bb25a5f6e02d5da39f420d243 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 5 Jul 2017 11:14:08 -0700 Subject: [PATCH 0843/3084] Add a test with a fixed register constraint. Make sure we use the diverted register location for tied operands. --- cranelift/filetests/regalloc/constraints.cton | 13 +++++++++++++ lib/cretonne/src/regalloc/coloring.rs | 3 ++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/regalloc/constraints.cton b/cranelift/filetests/regalloc/constraints.cton index e88bcc514c..c6108b0673 100644 --- a/cranelift/filetests/regalloc/constraints.cton +++ b/cranelift/filetests/regalloc/constraints.cton @@ -13,3 +13,16 @@ ebb0: v2 = isub v0, v1 return v2 } + +; Fixed register constraint. +function %fixed_op() -> i32 { +ebb0: + ; check: ,%rax] + ; sameln: $v0 = iconst.i32 12 + v0 = iconst.i32 12 + v1 = iconst.i32 13 + ; The dynamic shift amount must be in %rcx + ; check: regmove $v0, %rax -> %rcx + v2 = ishl v1, v0 + return v2 +} diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index e4e5a6599f..f61e88e9b7 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -371,7 +371,8 @@ impl<'a> Context<'a> { for (op, lv) in constraints.outs.iter().zip(defs) { if let ConstraintKind::Tied(num) = op.kind { let arg = dfg.inst_args(inst)[num as usize]; - locations[lv.value] = locations[arg]; + let reg = self.divert.reg(arg, locations); + locations[lv.value] = ValueLoc::Reg(reg); } } } From b7917fe404d6e23f0f9a2ee0c54d17a3f1f2d030 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 5 Jul 2017 12:19:55 -0700 Subject: [PATCH 0844/3084] Test two consecutive fixed operands. We need to move the previous value out of the way first. --- cranelift/filetests/regalloc/constraints.cton | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/cranelift/filetests/regalloc/constraints.cton b/cranelift/filetests/regalloc/constraints.cton index c6108b0673..28758a3bdd 100644 --- a/cranelift/filetests/regalloc/constraints.cton +++ b/cranelift/filetests/regalloc/constraints.cton @@ -2,6 +2,7 @@ test regalloc isa intel ; regex: V=v\d+ +; regex: REG=%r([abcd]x|[sd]i) ; Tied operands, both are killed at instruction. function %tied_easy() -> i32 { @@ -26,3 +27,20 @@ ebb0: v2 = ishl v1, v0 return v2 } + +; Fixed register constraint twice. +function %fixed_op_twice() -> i32 { +ebb0: + ; check: ,%rax] + ; sameln: $v0 = iconst.i32 12 + v0 = iconst.i32 12 + v1 = iconst.i32 13 + ; The dynamic shift amount must be in %rcx + ; check: regmove $v0, %rax -> %rcx + v2 = ishl v1, v0 + ; check: regmove $v0, %rcx -> $REG + ; check: regmove $v2, $REG -> %rcx + v3 = ishl v0, v2 + + return v3 +} From 01abbcbebeae07881c2768c4ba7b1edb359b9ed2 Mon Sep 17 00:00:00 2001 From: d1m0 Date: Wed, 5 Jul 2017 15:47:44 -0700 Subject: [PATCH 0845/3084] Cleanup typos; Remove SAMEAS; More descriptive rank comments; Introduce explicit sorting in free_typevars() (#111) As per the comment in TypeEnv.normalize_tv about cancellation, whenever we create a TypeVar we must assert that there is no under/overflow. To make sure this always happen move the safety checks to TypeVar.derived() from the other helper methods --- lib/cretonne/meta/cdsl/test_ti.py | 8 +-- lib/cretonne/meta/cdsl/test_typevar.py | 8 +-- lib/cretonne/meta/cdsl/ti.py | 44 +++++++------ lib/cretonne/meta/cdsl/typevar.py | 90 +++++++++----------------- lib/cretonne/meta/cdsl/xform.py | 6 +- 5 files changed, 66 insertions(+), 90 deletions(-) diff --git a/lib/cretonne/meta/cdsl/test_ti.py b/lib/cretonne/meta/cdsl/test_ti.py index d97902568a..396d5ef844 100644 --- a/lib/cretonne/meta/cdsl/test_ti.py +++ b/lib/cretonne/meta/cdsl/test_ti.py @@ -62,7 +62,7 @@ def agree(me, other): if m[me[tv]] != other[tv]: return False - # Tranlsate our constraints using m, and sort + # Translate our constraints using m, and sort me_equiv_constr = [(subst(a, m), subst(b, m)) for (a, b) in me.constraints] me_equiv_constr = sorted([sort_constr(x) for x in me_equiv_constr]) @@ -76,7 +76,7 @@ def agree(me, other): def check_typing(got_or_err, expected, symtab=None): # type: (TypingOrError, Tuple[VarMap, ConstraintList], Dict[str, Var]) -> None # noqa """ - Check that a the typying we received (got_or_err) complies with the + Check that a the typing we received (got_or_err) complies with the expected typing (expected). If symtab is specified, substitute the Vars in expected using symtab first (used when checking type inference on XForms) """ @@ -409,7 +409,7 @@ class TestXForm(TypeCheckingBaseTest): # If there are no free_typevars, this is a non-polymorphic pattern. # There should be only one possible concrete typing. - if (len(xform.free_typevars) == 0): + if (len(xform.ti.free_typevars()) == 0): assert len(concrete_typings_list) == 1 continue @@ -423,7 +423,7 @@ class TestXForm(TypeCheckingBaseTest): theoretical_num_typings =\ reduce(lambda x, y: x*y, [tv.get_typeset().size() - for tv in xform.free_typevars], 1) + for tv in xform.ti.free_typevars()], 1) assert len(concrete_typings_list) < theoretical_num_typings # Check the validity of each individual concrete typing against the diff --git a/lib/cretonne/meta/cdsl/test_typevar.py b/lib/cretonne/meta/cdsl/test_typevar.py index d990d7804f..dca5ba1b48 100644 --- a/lib/cretonne/meta/cdsl/test_typevar.py +++ b/lib/cretonne/meta/cdsl/test_typevar.py @@ -127,7 +127,6 @@ class TestTypeSet(TestCase): def test_preimage(self): t = TypeSet(lanes=(1, 1), ints=(8, 8), floats=(32, 32)) - self.assertEqual(t, t.preimage(TypeVar.SAMEAS)) # LANEOF self.assertEqual(TypeSet(lanes=True, ints=(8, 8), floats=(32, 32)), @@ -217,12 +216,11 @@ class TestTypeVar(TestCase): self.assertEqual(len(x.type_set.bools), 0) def test_stress_constrain_types(self): - # Get all 49 possible derived vars of length 2. Since we have SAMEAS - # this includes singly derived and non-derived vars - funcs = [TypeVar.SAMEAS, TypeVar.LANEOF, + # Get all 43 possible derived vars of length up to 2 + funcs = [TypeVar.LANEOF, TypeVar.ASBOOL, TypeVar.HALFVECTOR, TypeVar.DOUBLEVECTOR, TypeVar.HALFWIDTH, TypeVar.DOUBLEWIDTH] - v = list(product(*[funcs, funcs])) + v = [()] + [(x,) for x in funcs] + list(product(*[funcs, funcs])) # For each pair of derived variables for (i1, i2) in product(v, v): diff --git a/lib/cretonne/meta/cdsl/ti.py b/lib/cretonne/meta/cdsl/ti.py index 5b376f1e5d..ce2f05c750 100644 --- a/lib/cretonne/meta/cdsl/ti.py +++ b/lib/cretonne/meta/cdsl/ti.py @@ -29,11 +29,19 @@ class TypeEnv(object): :attribute constraints: a list of accumulated constraints - tuples (tv1, tv2)) where tv1 and tv2 are equal :attribute ranks: dictionary recording the (optional) ranks for tvs. - tvs corresponding to real variables have explicitly - specified ranks. + 'rank' is a partial ordering on TVs based on their + origin. See comments in rank() and register(). :attribute vars: a set containing all known Vars :attribute idx: counter used to get fresh ids """ + + RANK_DERIVED = 5 + RANK_INPUT = 4 + RANK_INTERMEDIATE = 3 + RANK_OUTPUT = 2 + RANK_TEMP = 1 + RANK_INTERNAL = 0 + def __init__(self, arg=None): # type: (Optional[Tuple[TypeMap, ConstraintList]]) -> None self.ranks = {} # type: Dict[TypeVar, int] @@ -104,9 +112,10 @@ class TypeEnv(object): Get the rank of tv in the partial order. TVs directly associated with a Var get their rank from the Var (see register()). Internally generated non-derived TVs implicitly get the lowest rank (0) - Internal derived variables get the highest rank. + Derived variables get the highest rank. """ - default_rank = 5 if tv.is_derived else 0 + default_rank = TypeEnv.RANK_DERIVED if tv.is_derived else\ + TypeEnv.RANK_INTERNAL return self.ranks.get(tv, default_rank) def register(self, v): @@ -118,25 +127,26 @@ class TypeEnv(object): self.vars.add(v) if v.is_input(): - r = 4 + r = TypeEnv.RANK_INPUT elif v.is_intermediate(): - r = 3 + r = TypeEnv.RANK_INTERMEDIATE elif v.is_output(): - r = 2 + r = TypeEnv.RANK_OUTPUT else: assert(v.is_temp()) - r = 1 + r = TypeEnv.RANK_TEMP self.ranks[v.get_typevar()] = r def free_typevars(self): - # type: () -> Set[TypeVar] + # type: () -> List[TypeVar] """ Get the free typevars in the current type env. """ tvs = set([self[tv].free_typevar() for tv in self.type_map.keys()]) # Filter out None here due to singleton type vars - return set(filter(lambda x: x is not None, tvs)) + return sorted(filter(lambda x: x is not None, tvs), + key=lambda x: x.name) def normalize(self): # type: () -> None @@ -178,7 +188,7 @@ class TypeEnv(object): s.add(a) children[b] = s - for r in list(self.free_typevars()): + for r in self.free_typevars(): while (r not in source_tvs and r in children and len(children[r]) == 1): child = list(children[r])[0] @@ -359,9 +369,6 @@ def normalize_tv(tv): # type: (TypeVar) -> TypeVar """ Normalize a (potentially derived) TV using the following rules: - - collapse SAMEAS - SAMEAS(base) -> base - - vector and width derived functions commute {HALF,DOUBLE}VECTOR({HALF,DOUBLE}WIDTH(base)) -> {HALF,DOUBLE}WIDTH({HALF,DOUBLE}VECTOR(base)) @@ -378,10 +385,6 @@ def normalize_tv(tv): df = tv.derived_func - # Collapse SAMEAS edges - if (df == TypeVar.SAMEAS): - return normalize_tv(tv.base) - if (tv.base.is_derived): base_df = tv.base.derived_func @@ -393,8 +396,9 @@ def normalize_tv(tv): TypeVar.derived(tv.base.base, df), base_df)) # Cancelling: HALFWIDTH, DOUBLEWIDTH and HALFVECTOR, DOUBLEVECTOR - # cancel each other. TODO: Does this cancellation hide type - # overflow/underflow? + # cancel each other. Note: This doesn't hide any over/underflows, + # since we 1) assert the safety of each TV in the chain upon its + # creation, and 2) the base typeset is only allowed to shrink. if (df, base_df) in \ [(TypeVar.HALFVECTOR, TypeVar.DOUBLEVECTOR), diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 1ea9831b91..b28502f61b 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -350,9 +350,7 @@ class TypeSet(object): """ Return the image of self across the derived function func """ - if (func == TypeVar.SAMEAS): - return self - elif (func == TypeVar.LANEOF): + if (func == TypeVar.LANEOF): return self.lane_of() elif (func == TypeVar.ASBOOL): return self.as_bool() @@ -376,9 +374,7 @@ class TypeSet(object): if (self.size() == 0): return self - if (func == TypeVar.SAMEAS): - return self - elif (func == TypeVar.LANEOF): + if (func == TypeVar.LANEOF): new = self.copy() new.lanes = set([2**i for i in range(0, int_log2(MAX_LANES)+1)]) return new @@ -388,6 +384,9 @@ class TypeSet(object): if 1 not in self.bools: new.ints = self.bools.difference(set([1])) new.floats = self.bools.intersection(set([32, 64])) + # If b1 is not in our typeset, than lanes=1 cannot be in the + # pre-image, as as_bool() of scalars is always b1. + new.lanes = self.lanes.difference(set([1])) else: new.ints = set([2**x for x in range(3, 7)]) new.floats = set([32, 64]) @@ -553,7 +552,6 @@ class TypeVar(object): # The names here must match the method names on `ir::types::Type`. # The camel_case of the names must match `enum OperandConstraint` in # `instructions.rs`. - SAMEAS = 'same_as' LANEOF = 'lane_of' ASBOOL = 'as_bool' HALFWIDTH = 'half_width' @@ -565,7 +563,6 @@ class TypeVar(object): def is_bijection(func): # type: (str) -> bool return func in [ - TypeVar.SAMEAS, TypeVar.HALFWIDTH, TypeVar.DOUBLEWIDTH, TypeVar.HALFVECTOR, @@ -575,7 +572,6 @@ class TypeVar(object): def inverse_func(func): # type: (str) -> str return { - TypeVar.SAMEAS: TypeVar.SAMEAS, TypeVar.HALFWIDTH: TypeVar.DOUBLEWIDTH, TypeVar.DOUBLEWIDTH: TypeVar.HALFWIDTH, TypeVar.HALFVECTOR: TypeVar.DOUBLEVECTOR, @@ -586,6 +582,31 @@ class TypeVar(object): def derived(base, derived_func): # type: (TypeVar, str) -> TypeVar """Create a type variable that is a function of another.""" + + # Safety checks to avoid over/underflows. + ts = base.get_typeset() + + if derived_func == TypeVar.HALFWIDTH: + if len(ts.ints) > 0: + assert min(ts.ints) > 8, "Can't halve all integer types" + if len(ts.floats) > 0: + assert min(ts.floats) > 32, "Can't halve all float types" + if len(ts.bools) > 0: + assert min(ts.bools) > 8, "Can't halve all boolean types" + elif derived_func == TypeVar.DOUBLEWIDTH: + if len(ts.ints) > 0: + assert max(ts.ints) < MAX_BITS,\ + "Can't double all integer types." + if len(ts.floats) > 0: + assert max(ts.floats) < MAX_BITS,\ + "Can't double all float types." + if len(ts.bools) > 0: + assert max(ts.bools) < MAX_BITS, "Can't double all bool types." + elif derived_func == TypeVar.HALFVECTOR: + assert min(ts.lanes) > 1, "Can't halve a scalar type" + elif derived_func == TypeVar.DOUBLEVECTOR: + assert max(ts.lanes) < MAX_LANES, "Can't double 256 lanes." + return TypeVar(None, None, base=base, derived_func=derived_func) @staticmethod @@ -596,27 +617,6 @@ class TypeVar(object): tv.type_set = ts return tv - def change_to_derived(self, base, derived_func): - # type: (TypeVar, str) -> None - """Change this type variable into a derived one.""" - self.type_set = None - self.is_derived = True - self.base = base - self.derived_func = derived_func - - def strip_sameas(self): - # type: () -> TypeVar - """ - Strip any `SAMEAS` functions from this typevar. - - Also rewrite any `SAMEAS` functions nested under this typevar. - """ - if self.is_derived: - self.base = self.base.strip_sameas() - if self.derived_func == self.SAMEAS: - return self.base - return self - def lane_of(self): # type: () -> TypeVar """ @@ -642,14 +642,6 @@ class TypeVar(object): Return a derived type variable that has the same number of vector lanes as this one, but the lanes are half the width. """ - ts = self.get_typeset() - if len(ts.ints) > 0: - assert min(ts.ints) > 8, "Can't halve all integer types" - if len(ts.floats) > 0: - assert min(ts.floats) > 32, "Can't halve all float types" - if len(ts.bools) > 0: - assert min(ts.bools) > 8, "Can't halve all boolean types" - return TypeVar.derived(self, self.HALFWIDTH) def double_width(self): @@ -658,14 +650,6 @@ class TypeVar(object): Return a derived type variable that has the same number of vector lanes as this one, but the lanes are double the width. """ - ts = self.get_typeset() - if len(ts.ints) > 0: - assert max(ts.ints) < MAX_BITS, "Can't double all integer types." - if len(ts.floats) > 0: - assert max(ts.floats) < MAX_BITS, "Can't double all float types." - if len(ts.bools) > 0: - assert max(ts.bools) < MAX_BITS, "Can't double all bool types." - return TypeVar.derived(self, self.DOUBLEWIDTH) def half_vector(self): @@ -674,9 +658,6 @@ class TypeVar(object): Return a derived type variable that has half the number of vector lanes as this one, with the same lane type. """ - ts = self.get_typeset() - assert min(ts.lanes) > 1, "Can't halve a scalar type" - return TypeVar.derived(self, self.HALFVECTOR) def double_vector(self): @@ -685,9 +666,6 @@ class TypeVar(object): Return a derived type variable that has twice the number of vector lanes as this one, with the same lane type. """ - ts = self.get_typeset() - assert max(ts.lanes) < MAX_LANES, "Can't double 256 lanes." - return TypeVar.derived(self, self.DOUBLEVECTOR) def singleton_type(self): @@ -744,15 +722,11 @@ class TypeVar(object): """ Constrain the range of types this variable can assume to a subset of those `other` can assume. - - If this is a SAMEAS-derived type variable, constrain the base instead. """ - a = self.strip_sameas() - b = other.strip_sameas() - if a is b: + if self is other: return - a.constrain_types_by_ts(b.get_typeset()) + self.constrain_types_by_ts(other.get_typeset()) def get_typeset(self): # type: () -> TypeSet diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index a8fbb7f66e..fd3c1dff1f 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -106,14 +106,14 @@ class XForm(object): # Sanity: The set of inferred free typevars should be a subset of the # TVs corresponding to Vars appearing in src - self.free_typevars = self.ti.free_typevars() + free_typevars = set(self.ti.free_typevars()) src_vars = set(self.inputs).union( [x for x in self.defs if not x.is_temp()]) src_tvs = set([v.get_typevar() for v in src_vars]) - if (not self.free_typevars.issubset(src_tvs)): + if (not free_typevars.issubset(src_tvs)): raise AssertionError( "Some free vars don't appear in src - {}" - .format(self.free_typevars.difference(src_tvs))) + .format(free_typevars.difference(src_tvs))) # Update the type vars for each Var to their inferred values for v in self.inputs + self.defs: From 9662f102e5c2ebdf58cb1debeb7864179696f8b6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 5 Jul 2017 15:12:35 -0700 Subject: [PATCH 0846/3084] Intel 32-bit encodings for copy.i32. --- cranelift/filetests/isa/intel/binary32.cton | 5 +++++ lib/cretonne/meta/isa/intel/encodings.py | 2 ++ lib/cretonne/meta/isa/intel/recipes.py | 8 ++++++-- lib/cretonne/src/isa/intel/binemit.rs | 10 ++++++++++ 4 files changed, 23 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 2e42d01fcc..14f050c2e7 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -56,6 +56,11 @@ ebb0: ; asm: sarl %cl, %ecx [-,%rcx] v25 = sshr v1, v1 ; bin: d3 f9 + ; asm: movl %esi, %ecx + [-,%rcx] v26 = copy v2 ; bin: 89 f1 + ; asm: movl %ecx, %esi + [-,%rsi] v27 = copy v1 ; bin: 89 ce + ; Integer Register - Immediate 8-bit operations. ; The 8-bit immediate is sign-extended. diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index c05308177a..0b59b32da3 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -13,6 +13,8 @@ I32.enc(base.band.i32, *r.rr(0x21)) I32.enc(base.bor.i32, *r.rr(0x09)) I32.enc(base.bxor.i32, *r.rr(0x31)) +I32.enc(base.copy.i32, *r.ur(0x89)) + # Immediate instructions with sign-extended 8-bit and 32-bit immediate. for inst, rrr in [ (base.iadd_imm.i32, 0), diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 4e41c9b990..3700ff60b5 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -4,8 +4,8 @@ Intel Encoding recipes. from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt, IsEqual -from base.formats import UnaryImm, Binary, BinaryImm, Store, Load -from base.formats import MultiAry, Call, IndirectCall +from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry +from base.formats import Call, IndirectCall, Store, Load from .registers import GPR, ABCD try: @@ -143,6 +143,10 @@ class TailRecipe: # XX /r rr = TailRecipe('rr', Binary, size=1, ins=(GPR, GPR), outs=0) +# XX /r, but for a unary operator with separate input/output register, like +# copies. +ur = TailRecipe('ur', Unary, size=1, ins=GPR, outs=GPR) + # XX /n with one arg in %rcx, for shifts. rc = TailRecipe('rc', Binary, size=1, ins=(GPR, GPR.rcx), outs=0) diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 001a41e009..10bd6288a7 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -111,6 +111,16 @@ fn recipe_op1rr(func: &Function, inst: Inst, sink: &mut C } } +fn recipe_op1ur(func: &Function, inst: Inst, sink: &mut CS) { + if let InstructionData::Unary { arg, .. } = func.dfg[inst] { + put_op1(func.encodings[inst].bits(), sink); + let res = func.locations[func.dfg.first_result(inst)].unwrap_reg(); + modrm_rr(res, func.locations[arg].unwrap_reg(), sink); + } else { + panic!("Expected Unary format: {:?}", func.dfg[inst]); + } +} + fn recipe_op1rc(func: &Function, inst: Inst, sink: &mut CS) { if let InstructionData::Binary { args, .. } = func.dfg[inst] { let bits = func.encodings[inst].bits(); From 60efc6893197cdf1c39e9f39686d4ee3c0686ef2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 5 Jul 2017 14:22:34 -0700 Subject: [PATCH 0847/3084] Only print pressure for toprcs containing registers. Many ISAs don't need 4 top-level register classes, so don't print them. --- lib/cretonne/src/regalloc/pressure.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index 3bf1e68f54..fd9cf7c363 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -238,7 +238,9 @@ impl fmt::Display for Pressure { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "Pressure[")?; for rc in &self.toprc { - write!(f, " {}+{}/{}", rc.base_count, rc.transient_count, rc.limit)?; + if rc.limit > 0 { + write!(f, " {}+{}/{}", rc.base_count, rc.transient_count, rc.limit)?; + } } write!(f, " ]") } From 64f6a98abe215cfd2f59d5f0fa73ec734c09dc0a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 5 Jul 2017 12:50:20 -0700 Subject: [PATCH 0848/3084] Test a tied operand following a fixed register operand. The redefined tied value lives in the diverted register. --- cranelift/filetests/regalloc/constraints.cton | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/cranelift/filetests/regalloc/constraints.cton b/cranelift/filetests/regalloc/constraints.cton index 28758a3bdd..f405e67db9 100644 --- a/cranelift/filetests/regalloc/constraints.cton +++ b/cranelift/filetests/regalloc/constraints.cton @@ -44,3 +44,26 @@ ebb0: return v3 } + +; Tied use of a diverted register. +function %fixed_op_twice() -> i32 { +ebb0: + ; check: ,%rax] + ; sameln: $v0 = iconst.i32 12 + v0 = iconst.i32 12 + v1 = iconst.i32 13 + ; The dynamic shift amount must be in %rcx + ; check: regmove $v0, %rax -> %rcx + ; check: $v2 = ishl $v1, $v0 + v2 = ishl v1, v0 + + ; Now v0 is globally allocated to %rax, but diverted to %rcx. + ; Check that the tied def gets the diverted register. + v3 = isub v0, v2 + ; not: regmove + ; check: ,%rcx] + ; sameln: isub + ; Move it into place for the return value. + ; check: regmove $v3, %rcx -> %rax + return v3 +} From f0abff36111040607d4d9ab0146770b41f6eba1b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 5 Jul 2017 14:23:58 -0700 Subject: [PATCH 0849/3084] Handle tied operands that are not killed by their use. Any tied register uses are interesting enough to be added to the reguses list if their value is not killed. A copy needs to be inserted in that case. --- cranelift/filetests/regalloc/constraints.cton | 13 ++++++++++++ lib/cretonne/src/regalloc/spilling.rs | 20 ++++++++++++++++++- 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/regalloc/constraints.cton b/cranelift/filetests/regalloc/constraints.cton index f405e67db9..720a886263 100644 --- a/cranelift/filetests/regalloc/constraints.cton +++ b/cranelift/filetests/regalloc/constraints.cton @@ -15,6 +15,19 @@ ebb0: return v2 } +; Tied operand is live after instruction. +function %tied_alive() -> i32 { +ebb0: + v0 = iconst.i32 12 + v1 = iconst.i32 13 + ; check: $(v0c=$V) = copy $v0 + ; check: $v2 = isub $v0c, $v1 + v2 = isub v0, v1 + ; check: $v3 = iadd $v2, $v0 + v3 = iadd v2, v0 + return v3 +} + ; Fixed register constraint. function %fixed_op() -> i32 { ebb0: diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 37ff870a26..465efa7856 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -26,6 +26,7 @@ use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; use regalloc::pressure::Pressure; use regalloc::virtregs::VirtRegs; +use std::fmt; use topo_order::TopoOrder; /// Persistent data structures for the spilling pass. @@ -338,7 +339,8 @@ impl<'a> Context<'a> { } // Only collect the interesting register uses. - if reguse.fixed || reguse.spilled { + if reguse.fixed || reguse.tied || reguse.spilled { + dbg!(" reguse: {}", reguse); self.reg_uses.push(reguse); } } @@ -578,3 +580,19 @@ impl RegUse { } } } + +impl fmt::Display for RegUse { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{}@op{}", self.value, self.opidx)?; + if self.fixed { + write!(f, "/fixed")?; + } + if self.spilled { + write!(f, "/spilled")?; + } + if self.tied { + write!(f, "/tied")?; + } + Ok(()) + } +} From 27d272ade04125bb711aac508f8d8b6ca034234f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 7 Jul 2017 14:32:13 -0700 Subject: [PATCH 0850/3084] Add a fmt.multi_line() method to srcgen.Formatter. Write out multiple code lines from a single string after stripping a common indentation. Also use this for doc_comment(). --- lib/cretonne/meta/srcgen.py | 43 ++++++++++++++++++++++++++++++++++--- 1 file changed, 40 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/meta/srcgen.py b/lib/cretonne/meta/srcgen.py index d91ecc4ab2..f0f7931ea1 100644 --- a/lib/cretonne/meta/srcgen.py +++ b/lib/cretonne/meta/srcgen.py @@ -8,7 +8,6 @@ source code. from __future__ import absolute_import import sys import os -import re try: from typing import Any, List # noqa @@ -130,6 +129,12 @@ class Formatter(object): # type: (str, *Any) -> None self.line(fmt.format(*args)) + def multi_line(self, s): + # type: (str) -> None + """Add one or more lines after stripping common indentation.""" + for l in parse_multiline(s): + self.line(l) + def comment(self, s): # type: (str) -> None """Add a comment line.""" @@ -138,5 +143,37 @@ class Formatter(object): def doc_comment(self, s): # type: (str) -> None """Add a (multi-line) documentation comment.""" - s = re.sub('^', self.indent + '/// ', s, flags=re.M) + '\n' - self.lines.append(s) + for l in parse_multiline(s): + self.line('/// ' + l if l else '///') + + +def _indent(s): + # type: (str) -> int + """ + Compute the indentation of s, or None of an empty line. + + Example: + >>> _indent("foo") + 0 + >>> _indent(" bar") + 4 + >>> _indent(" ") + >>> _indent("") + """ + t = s.lstrip() + return len(s) - len(t) if t else None + + +def parse_multiline(s): + # type: (str) -> List[str] + """ + Given a multi-line string, split it into a sequence of lines after + stripping a common indentation. This is useful for strings defined with doc + strings: + >>> parse_multiline('\\n hello\\n world\\n') + [None, 'hello', 'world'] + """ + lines = s.splitlines() + indents = list(i for i in (_indent(l) for l in lines) if i) + indent = min(indents) if indents else 0 + return list(l[indent:] if len(l) > indent else None for l in lines) From 528e6ff3f5a019223fc05c9a1cdfaa4da32f13e2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 7 Jul 2017 11:33:38 -0700 Subject: [PATCH 0851/3084] Generate instruction unwrapping code for binemit recipes. Generate code to: - Unwrap the instruction and generate an error if the instruction format doesn't match the recipe. - Look up the value locations of register and stack arguments. The recipe_* functions in the ISA binemit modules now take these unwrapped items as arguments. Also add an optional `emit` argument to the EncRecipe constructor which makes it possible to provide inline Rust code snippets for code emission. This requires a lot less boilerplate than recipe_* functions. --- lib/cretonne/meta/cdsl/isa.py | 5 +- lib/cretonne/meta/gen_binemit.py | 105 +++++- lib/cretonne/meta/isa/riscv/recipes.py | 91 ++++- lib/cretonne/src/ir/valueloc.rs | 8 + lib/cretonne/src/isa/intel/binemit.rs | 465 +++++++++++++------------ lib/cretonne/src/isa/riscv/binemit.rs | 180 ---------- 6 files changed, 435 insertions(+), 419 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index bc85f7fb8f..0562df10dc 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -202,6 +202,7 @@ class EncRecipe(object): :param: branch_range `(origin, bits)` range for branches. :param: instp Instruction predicate. :param: isap ISA predicate. + :param: emit Rust code for binary emission. """ def __init__( @@ -213,7 +214,8 @@ class EncRecipe(object): outs, # type: ConstraintSeq branch_range=None, # type: BranchRange instp=None, # type: PredNode - isap=None # type: PredNode + isap=None, # type: PredNode + emit=None # type: str ): # type: (...) -> None self.name = name @@ -223,6 +225,7 @@ class EncRecipe(object): self.branch_range = branch_range self.instp = instp self.isap = isap + self.emit = emit if instp: assert instp.predicate_context() == format self.number = None # type: int diff --git a/lib/cretonne/meta/gen_binemit.py b/lib/cretonne/meta/gen_binemit.py index 8d1954ce55..858fc36b39 100644 --- a/lib/cretonne/meta/gen_binemit.py +++ b/lib/cretonne/meta/gen_binemit.py @@ -3,15 +3,108 @@ Generate binary emission code for each ISA. """ from __future__ import absolute_import +from cdsl.registers import RegClass, Stack import srcgen try: from typing import Sequence, List # noqa - from cdsl.isa import TargetISA # noqa + from cdsl.isa import TargetISA, EncRecipe # noqa except ImportError: pass +def gen_recipe(recipe, fmt): + # type: (EncRecipe, srcgen.Formatter) -> None + """ + Generate code to handle a single recipe. + + - Unpack the instruction data, knowing the format. + - Determine register locations for operands with register constraints. + - Determine stack slot locations for operands with stack constraints. + - Call hand-written code for the actual emission. + """ + iform = recipe.format + nvops = iform.num_value_operands + want_args = any(isinstance(i, RegClass) or isinstance(i, Stack) + for i in recipe.ins) + assert not want_args or nvops > 0 + want_outs = any(isinstance(o, RegClass) or isinstance(o, Stack) + for o in recipe.outs) + + # First unpack the instruction. + with fmt.indented( + 'if let InstructionData::{} {{'.format(iform.name), + '}'): + for f in iform.imm_fields: + fmt.line('{},'.format(f.member)) + if want_args: + if iform.has_value_list or nvops > 1: + fmt.line('ref args,') + else: + fmt.line('arg,') + fmt.line('..') + fmt.outdented_line('} = func.dfg[inst] {') + + # Normalize to an `args` array. + if want_args: + if iform.has_value_list: + fmt.line('let args = args.as_slice(&func.dfg.value_lists);') + elif nvops == 1: + fmt.line('let args = [arg];') + + # Unwrap interesting input arguments. + # Don't bother with fixed registers. + args = '' + for i, arg in enumerate(recipe.ins): + if isinstance(arg, RegClass): + v = 'in_reg{}'.format(i) + args += ', ' + v + fmt.line( + 'let {} = func.locations[args[{}]].unwrap_reg();' + .format(v, i)) + elif isinstance(arg, Stack): + v = 'in_ss{}'.format(i) + args += ', ' + v + fmt.line( + 'let {} = func.locations[args[{}]].unwrap_stack();' + .format(v, i)) + + # Pass arguments in this order: inputs, imm_fields, outputs. + for f in iform.imm_fields: + args += ', ' + f.member + + # Unwrap interesting output arguments. + if want_outs: + if len(recipe.outs) == 1: + fmt.line('let results = [func.dfg.first_result(inst)];') + else: + fmt.line('let results = func.dfg.inst_results(inst);') + for i, res in enumerate(recipe.outs): + if isinstance(res, RegClass): + v = 'out_reg{}'.format(i) + args += ', ' + v + fmt.line( + 'let {} = func.locations[results[{}]].unwrap_reg();' + .format(v, i)) + elif isinstance(res, Stack): + v = 'out_ss{}'.format(i) + args += ', ' + v + fmt.line( + 'let {} = func.locations[results[{}]].unwrap_stack();' + .format(v, i)) + + # Call hand-written code. If the recipe contains a code snippet, use + # that. Otherwise cal a recipe function in the target ISA's binemit + # module. + if recipe.emit is None: + fmt.format( + 'return recipe_{}(func, inst, sink, bits{});', + recipe.name.lower(), args) + else: + fmt.multi_line(recipe.emit) + fmt.line('return;') + + def gen_isa(isa, fmt): # type: (TargetISA, srcgen.Formatter) -> None """ @@ -28,14 +121,18 @@ def gen_isa(isa, fmt): '(func: &Function, inst: Inst, _sink: &mut CS) {', '}'): fmt.line('bad_encoding(func, inst)') else: + fmt.line('#[allow(unused_variables, unreachable_code)]') with fmt.indented( 'pub fn emit_inst' '(func: &Function, inst: Inst, sink: &mut CS) {', '}'): + fmt.line('let bits = func.encodings[inst].bits();') with fmt.indented('match func.encodings[inst].recipe() {', '}'): for i, recipe in enumerate(isa.all_recipes): - fmt.line('{} => recipe_{}(func, inst, sink),'.format( - i, recipe.name.lower())) - fmt.line('_ => bad_encoding(func, inst),') + fmt.comment(recipe.name) + with fmt.indented('{} => {{'.format(i), '}'): + gen_recipe(recipe, fmt) + fmt.line('_ => {}') + fmt.line('bad_encoding(func, inst);') def generate(isas, out_dir): diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 2ea7f35597..510e13c860 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -92,68 +92,123 @@ def LUI(): # R-type 32-bit instructions: These are mostly binary arithmetic instructions. # The encbits are `opcode[6:2] | (funct3 << 5) | (funct7 << 8) -R = EncRecipe('R', Binary, size=4, ins=(GPR, GPR), outs=GPR) +R = EncRecipe( + 'R', Binary, size=4, ins=(GPR, GPR), outs=GPR, + emit='put_r(bits, in_reg0, in_reg1, out_reg0, sink);') # R-type with an immediate shift amount instead of rs2. -Rshamt = EncRecipe('Rshamt', BinaryImm, size=4, ins=GPR, outs=GPR) +Rshamt = EncRecipe( + 'Rshamt', BinaryImm, size=4, ins=GPR, outs=GPR, + emit='put_rshamt(bits, in_reg0, imm.into(), out_reg0, sink);') # R-type encoding of an integer comparison. -Ricmp = EncRecipe('Ricmp', IntCompare, size=4, ins=(GPR, GPR), outs=GPR) +Ricmp = EncRecipe( + 'Ricmp', IntCompare, size=4, ins=(GPR, GPR), outs=GPR, + emit='put_r(bits, in_reg0, in_reg1, out_reg0, sink);') I = EncRecipe( 'I', BinaryImm, size=4, ins=GPR, outs=GPR, - instp=IsSignedInt(BinaryImm.imm, 12)) + instp=IsSignedInt(BinaryImm.imm, 12), + emit='put_i(bits, in_reg0, imm.into(), out_reg0, sink);') # I-type instruction with a hardcoded %x0 rs1. Iz = EncRecipe( 'Iz', UnaryImm, size=4, ins=(), outs=GPR, - instp=IsSignedInt(UnaryImm.imm, 12)) + instp=IsSignedInt(UnaryImm.imm, 12), + emit='put_i(bits, 0, imm.into(), out_reg0, sink);') # I-type encoding of an integer comparison. Iicmp = EncRecipe( 'Iicmp', IntCompareImm, size=4, ins=GPR, outs=GPR, - instp=IsSignedInt(IntCompareImm.imm, 12)) + instp=IsSignedInt(IntCompareImm.imm, 12), + emit='put_i(bits, in_reg0, imm.into(), out_reg0, sink);') # I-type encoding for `jalr` as a return instruction. We won't use the # immediate offset. # The variable return values are not encoded. -Iret = EncRecipe('Iret', MultiAry, size=4, ins=(), outs=()) +Iret = EncRecipe( + 'Iret', MultiAry, size=4, ins=(), outs=(), + emit=''' + // Return instructions are always a jalr to %x1. + // The return address is provided as a special-purpose link argument. + put_i(bits, + 1, // rs1 = %x1 + 0, // no offset. + 0, // rd = %x0: no address written. + sink); + ''') # I-type encoding for `jalr` as an indirect call. -Icall = EncRecipe('Icall', IndirectCall, size=4, ins=GPR, outs=()) +Icall = EncRecipe( + 'Icall', IndirectCall, size=4, ins=GPR, outs=(), + emit=''' + // Indirect instructions are jalr with rd=%x1. + put_i(bits, + in_reg0, + 0, // no offset. + 1, // rd = %x1: link register. + sink); + ''') + # Copy of a GPR is implemented as addi x, 0. -Icopy = EncRecipe('Icopy', Unary, size=4, ins=GPR, outs=GPR) +Icopy = EncRecipe( + 'Icopy', Unary, size=4, ins=GPR, outs=GPR, + emit='put_i(bits, in_reg0, 0, out_reg0, sink);') # U-type instructions have a 20-bit immediate that targets bits 12-31. U = EncRecipe( 'U', UnaryImm, size=4, ins=(), outs=GPR, - instp=IsSignedInt(UnaryImm.imm, 32, 12)) + instp=IsSignedInt(UnaryImm.imm, 32, 12), + emit='put_u(bits, imm.into(), out_reg0, sink);') # UJ-type unconditional branch instructions. -UJ = EncRecipe('UJ', Jump, size=4, ins=(), outs=(), branch_range=(0, 21)) -UJcall = EncRecipe('UJcall', Call, size=4, ins=(), outs=()) +UJ = EncRecipe( + 'UJ', Jump, size=4, ins=(), outs=(), branch_range=(0, 21), + emit=''' + let dest = func.offsets[destination] as i64; + let disp = dest - sink.offset() as i64; + put_uj(bits, disp, 0, sink); + ''') + +UJcall = EncRecipe( + 'UJcall', Call, size=4, ins=(), outs=(), + emit=''' + sink.reloc_func(RelocKind::Call.into(), func_ref); + // rd=%x1 is the standard link register. + put_uj(bits, 0, 1, sink); + ''') # SB-type branch instructions. -# TODO: These instructions have a +/- 4 KB branch range. How to encode that -# constraint? SB = EncRecipe( 'SB', BranchIcmp, size=4, ins=(GPR, GPR), outs=(), - branch_range=(0, 13)) + branch_range=(0, 13), + emit=''' + let dest = func.offsets[destination] as i64; + let disp = dest - sink.offset() as i64; + put_sb(bits, disp, in_reg0, in_reg1, sink); + ''') # SB-type branch instruction with rs2 fixed to zero. SBzero = EncRecipe( 'SBzero', Branch, size=4, ins=(GPR), outs=(), - branch_range=(0, 13)) + branch_range=(0, 13), + emit=''' + let dest = func.offsets[destination] as i64; + let disp = dest - sink.offset() as i64; + put_sb(bits, disp, in_reg0, 0, sink); + ''') # Spill of a GPR. GPsp = EncRecipe( 'GPsp', Unary, size=4, - ins=GPR, outs=Stack(GPR)) + ins=GPR, outs=Stack(GPR), + emit='unimplemented!();') # Fill of a GPR. GPfi = EncRecipe( 'GPfi', Unary, size=4, - ins=Stack(GPR), outs=GPR) + ins=Stack(GPR), outs=GPR, + emit='unimplemented!();') diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/cretonne/src/ir/valueloc.rs index cae3042170..a6a358e14b 100644 --- a/lib/cretonne/src/ir/valueloc.rs +++ b/lib/cretonne/src/ir/valueloc.rs @@ -33,6 +33,14 @@ impl ValueLoc { } } + /// Get the stack slot of this location, or panic. + pub fn unwrap_stack(self) -> StackSlot { + match self { + ValueLoc::Stack(ss) => ss, + _ => panic!("Expected stack slot: {:?}", self), + } + } + /// Return an object that can display this value location, using the register info from the /// target ISA. pub fn display<'a, R: Into>>(self, regs: R) -> DisplayValueLoc<'a> { diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 10bd6288a7..831ef4c756 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -1,7 +1,8 @@ //! Emitting binary Intel machine code. use binemit::{CodeSink, Reloc, bad_encoding}; -use ir::{Function, Inst, InstructionData}; +use ir::{self, Function, Inst, InstructionData, MemFlags}; +use ir::immediates::{Imm64, Offset32}; use isa::RegUnit; include!(concat!(env!("OUT_DIR"), "/binemit-intel.rs")); @@ -100,257 +101,289 @@ fn modrm_disp32(rm: RegUnit, reg: RegUnit, sink: &mut CS) sink.put1(b); } -fn recipe_op1rr(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Binary { args, .. } = func.dfg[inst] { - put_op1(func.encodings[inst].bits(), sink); - modrm_rr(func.locations[args[0]].unwrap_reg(), - func.locations[args[1]].unwrap_reg(), - sink); - } else { - panic!("Expected Binary format: {:?}", func.dfg[inst]); - } +fn recipe_op1rr(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + in_reg1: RegUnit) { + put_op1(bits, sink); + modrm_rr(in_reg0, in_reg1, sink); } -fn recipe_op1ur(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Unary { arg, .. } = func.dfg[inst] { - put_op1(func.encodings[inst].bits(), sink); - let res = func.locations[func.dfg.first_result(inst)].unwrap_reg(); - modrm_rr(res, func.locations[arg].unwrap_reg(), sink); - } else { - panic!("Expected Unary format: {:?}", func.dfg[inst]); - } +fn recipe_op1ur(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + out_reg0: RegUnit) { + put_op1(bits, sink); + modrm_rr(out_reg0, in_reg0, sink); } -fn recipe_op1rc(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Binary { args, .. } = func.dfg[inst] { - let bits = func.encodings[inst].bits(); - put_op1(bits, sink); - modrm_r_bits(func.locations[args[0]].unwrap_reg(), bits, sink); - } else { - panic!("Expected Binary format: {:?}", func.dfg[inst]); - } +fn recipe_op1rc(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit) { + put_op1(bits, sink); + modrm_r_bits(in_reg0, bits, sink); } -fn recipe_op1rib(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::BinaryImm { arg, imm, .. } = func.dfg[inst] { - let bits = func.encodings[inst].bits(); - put_op1(bits, sink); - modrm_r_bits(func.locations[arg].unwrap_reg(), bits, sink); - let imm: i64 = imm.into(); - sink.put1(imm as u8); - } else { - panic!("Expected BinaryImm format: {:?}", func.dfg[inst]); - } +fn recipe_op1rib(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + imm: Imm64) { + put_op1(bits, sink); + modrm_r_bits(in_reg0, bits, sink); + let imm: i64 = imm.into(); + sink.put1(imm as u8); } -fn recipe_op1rid(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::BinaryImm { arg, imm, .. } = func.dfg[inst] { - let bits = func.encodings[inst].bits(); - put_op1(bits, sink); - modrm_r_bits(func.locations[arg].unwrap_reg(), bits, sink); - let imm: i64 = imm.into(); - sink.put4(imm as u32); - } else { - panic!("Expected BinaryImm format: {:?}", func.dfg[inst]); - } +fn recipe_op1rid(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + imm: Imm64) { + put_op1(bits, sink); + modrm_r_bits(in_reg0, bits, sink); + let imm: i64 = imm.into(); + sink.put4(imm as u32); } -fn recipe_op1uid(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::UnaryImm { imm, .. } = func.dfg[inst] { - let bits = func.encodings[inst].bits(); - let reg = func.locations[func.dfg.first_result(inst)].unwrap_reg(); - // The destination register is encoded in the low bits of the opcode. No ModR/M - put_op1(bits | (reg & 7), sink); - let imm: i64 = imm.into(); - sink.put4(imm as u32); - } else { - panic!("Expected UnaryImm format: {:?}", func.dfg[inst]); - } +fn recipe_op1uid(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + imm: Imm64, + out_reg0: RegUnit) { + // The destination register is encoded in the low bits of the opcode. No ModR/M + put_op1(bits | (out_reg0 & 7), sink); + let imm: i64 = imm.into(); + sink.put4(imm as u32); } // Store recipes. -fn recipe_op1st(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Store { args, .. } = func.dfg[inst] { - put_op1(func.encodings[inst].bits(), sink); - modrm_rm(func.locations[args[1]].unwrap_reg(), - func.locations[args[0]].unwrap_reg(), - sink); - } else { - panic!("Expected Store format: {:?}", func.dfg[inst]); - } +fn recipe_op1st(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + in_reg1: RegUnit, + _flags: MemFlags, + _offset: Offset32) { + put_op1(bits, sink); + modrm_rm(in_reg1, in_reg0, sink); } // This is just a tighter register class constraint. -fn recipe_op1st_abcd(func: &Function, inst: Inst, sink: &mut CS) { - recipe_op1st(func, inst, sink) +fn recipe_op1st_abcd(func: &Function, + inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + in_reg1: RegUnit, + flags: MemFlags, + offset: Offset32) { + recipe_op1st(func, inst, sink, bits, in_reg0, in_reg1, flags, offset) } -fn recipe_mp1st(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Store { args, .. } = func.dfg[inst] { - put_mp1(func.encodings[inst].bits(), sink); - modrm_rm(func.locations[args[1]].unwrap_reg(), - func.locations[args[0]].unwrap_reg(), - sink); - } else { - panic!("Expected Store format: {:?}", func.dfg[inst]); - } +fn recipe_mp1st(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + in_reg1: RegUnit, + _flags: MemFlags, + _offset: Offset32) { + put_mp1(bits, sink); + modrm_rm(in_reg1, in_reg0, sink); } -fn recipe_op1stdisp8(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Store { args, offset, .. } = func.dfg[inst] { - put_op1(func.encodings[inst].bits(), sink); - modrm_disp8(func.locations[args[1]].unwrap_reg(), - func.locations[args[0]].unwrap_reg(), - sink); - let offset: i32 = offset.into(); - sink.put1(offset as u8); - } else { - panic!("Expected Store format: {:?}", func.dfg[inst]); - } +fn recipe_op1stdisp8(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + in_reg1: RegUnit, + _flags: MemFlags, + offset: Offset32) { + put_op1(bits, sink); + modrm_disp8(in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); } -fn recipe_op1stdisp8_abcd(func: &Function, inst: Inst, sink: &mut CS) { - recipe_op1stdisp8(func, inst, sink) +fn recipe_op1stdisp8_abcd(func: &Function, + inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + in_reg1: RegUnit, + flags: MemFlags, + offset: Offset32) { + recipe_op1stdisp8(func, inst, sink, bits, in_reg0, in_reg1, flags, offset) } -fn recipe_mp1stdisp8(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Store { args, offset, .. } = func.dfg[inst] { - put_mp1(func.encodings[inst].bits(), sink); - modrm_disp8(func.locations[args[1]].unwrap_reg(), - func.locations[args[0]].unwrap_reg(), - sink); - let offset: i32 = offset.into(); - sink.put1(offset as u8); - } else { - panic!("Expected Store format: {:?}", func.dfg[inst]); - } +fn recipe_mp1stdisp8(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + in_reg1: RegUnit, + _flags: MemFlags, + offset: Offset32) { + put_mp1(bits, sink); + modrm_disp8(in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); } -fn recipe_op1stdisp32(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Store { args, offset, .. } = func.dfg[inst] { - put_op1(func.encodings[inst].bits(), sink); - modrm_disp32(func.locations[args[1]].unwrap_reg(), - func.locations[args[0]].unwrap_reg(), - sink); - let offset: i32 = offset.into(); - sink.put4(offset as u32); - } else { - panic!("Expected Store format: {:?}", func.dfg[inst]); - } +fn recipe_op1stdisp32(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + in_reg1: RegUnit, + _flags: MemFlags, + offset: Offset32) { + put_op1(bits, sink); + modrm_disp32(in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); } -fn recipe_op1stdisp32_abcd(func: &Function, inst: Inst, sink: &mut CS) { - recipe_op1stdisp32(func, inst, sink) +fn recipe_op1stdisp32_abcd(func: &Function, + inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + in_reg1: RegUnit, + flags: MemFlags, + offset: Offset32) { + recipe_op1stdisp32(func, inst, sink, bits, in_reg0, in_reg1, flags, offset) } -fn recipe_mp1stdisp32(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Store { args, offset, .. } = func.dfg[inst] { - put_mp1(func.encodings[inst].bits(), sink); - modrm_disp32(func.locations[args[1]].unwrap_reg(), - func.locations[args[0]].unwrap_reg(), - sink); - let offset: i32 = offset.into(); - sink.put4(offset as u32); - } else { - panic!("Expected Store format: {:?}", func.dfg[inst]); - } +fn recipe_mp1stdisp32(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + in_reg1: RegUnit, + _flags: MemFlags, + offset: Offset32) { + put_mp1(bits, sink); + modrm_disp32(in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); } // Load recipes -fn recipe_op1ld(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Load { arg, .. } = func.dfg[inst] { - put_op1(func.encodings[inst].bits(), sink); - modrm_rm(func.locations[arg].unwrap_reg(), - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - } else { - panic!("Expected Load format: {:?}", func.dfg[inst]); - } -} - -fn recipe_op1lddisp8(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Load { arg, offset, .. } = func.dfg[inst] { - put_op1(func.encodings[inst].bits(), sink); - modrm_disp8(func.locations[arg].unwrap_reg(), - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - let offset: i32 = offset.into(); - sink.put1(offset as u8); - } else { - panic!("Expected Load format: {:?}", func.dfg[inst]); - } -} - -fn recipe_op1lddisp32(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Load { arg, offset, .. } = func.dfg[inst] { - put_op1(func.encodings[inst].bits(), sink); - modrm_disp32(func.locations[arg].unwrap_reg(), - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - let offset: i32 = offset.into(); - sink.put4(offset as u32); - } else { - panic!("Expected Load format: {:?}", func.dfg[inst]); - } -} - -fn recipe_op2ld(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Load { arg, .. } = func.dfg[inst] { - put_op2(func.encodings[inst].bits(), sink); - modrm_rm(func.locations[arg].unwrap_reg(), - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - } else { - panic!("Expected Load format: {:?}", func.dfg[inst]); - } -} - -fn recipe_op2lddisp8(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Load { arg, offset, .. } = func.dfg[inst] { - put_op2(func.encodings[inst].bits(), sink); - modrm_disp8(func.locations[arg].unwrap_reg(), - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - let offset: i32 = offset.into(); - sink.put1(offset as u8); - } else { - panic!("Expected Load format: {:?}", func.dfg[inst]); - } -} - -fn recipe_op2lddisp32(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Load { arg, offset, .. } = func.dfg[inst] { - put_op2(func.encodings[inst].bits(), sink); - modrm_disp32(func.locations[arg].unwrap_reg(), - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - let offset: i32 = offset.into(); - sink.put4(offset as u32); - } else { - panic!("Expected Load format: {:?}", func.dfg[inst]); - } -} - -fn recipe_op1call_id(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Call { func_ref, .. } = func.dfg[inst] { - put_op1(func.encodings[inst].bits(), sink); - sink.reloc_func(RelocKind::PCRel4.into(), func_ref); - sink.put4(0); - } else { - panic!("Expected Call format: {:?}", func.dfg[inst]); - } -} - -fn recipe_op1call_r(func: &Function, inst: Inst, sink: &mut CS) { - let bits = func.encodings[inst].bits(); +fn recipe_op1ld(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + _flags: MemFlags, + _offset: Offset32, + out_reg0: RegUnit) { put_op1(bits, sink); - modrm_r_bits(func.locations[func.dfg.inst_args(inst)[0]].unwrap_reg(), - bits, - sink); + modrm_rm(in_reg0, out_reg0, sink); } -fn recipe_op1ret(func: &Function, inst: Inst, sink: &mut CS) { - put_op1(func.encodings[inst].bits(), sink); +fn recipe_op1lddisp8(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + _flags: MemFlags, + offset: Offset32, + out_reg0: RegUnit) { + put_op1(bits, sink); + modrm_disp8(in_reg0, out_reg0, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); +} + +fn recipe_op1lddisp32(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + _flags: MemFlags, + offset: Offset32, + out_reg0: RegUnit) { + put_op1(bits, sink); + modrm_disp32(in_reg0, out_reg0, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); +} + +fn recipe_op2ld(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + _flags: MemFlags, + _offset: Offset32, + out_reg0: RegUnit) { + put_op2(bits, sink); + modrm_rm(in_reg0, out_reg0, sink); +} + +fn recipe_op2lddisp8(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + _flags: MemFlags, + offset: Offset32, + out_reg0: RegUnit) { + put_op2(bits, sink); + modrm_disp8(in_reg0, out_reg0, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); +} + +fn recipe_op2lddisp32(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + _flags: MemFlags, + offset: Offset32, + out_reg0: RegUnit) { + put_op2(bits, sink); + modrm_disp32(in_reg0, out_reg0, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); +} + +fn recipe_op1call_id(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + func_ref: ir::FuncRef) { + put_op1(bits, sink); + sink.reloc_func(RelocKind::PCRel4.into(), func_ref); + sink.put4(0); +} + +fn recipe_op1call_r(_func: &Function, + _inst: Inst, + sink: &mut CS, + bits: u16, + in_reg0: RegUnit, + _sig_ref: ir::SigRef) { + put_op1(bits, sink); + modrm_r_bits(in_reg0, bits, sink); +} + +fn recipe_op1ret(_func: &Function, _inst: Inst, sink: &mut CS, bits: u16) { + put_op1(bits, sink); } diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 21638980b2..d193981887 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -87,42 +87,6 @@ fn put_rshamt(bits: u16, sink.put4(i); } -fn recipe_r(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Binary { args, .. } = func.dfg[inst] { - put_r(func.encodings[inst].bits(), - func.locations[args[0]].unwrap_reg(), - func.locations[args[1]].unwrap_reg(), - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - } else { - panic!("Expected Binary format: {:?}", func.dfg[inst]); - } -} - -fn recipe_ricmp(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::IntCompare { args, .. } = func.dfg[inst] { - put_r(func.encodings[inst].bits(), - func.locations[args[0]].unwrap_reg(), - func.locations[args[1]].unwrap_reg(), - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - } else { - panic!("Expected IntCompare format: {:?}", func.dfg[inst]); - } -} - -fn recipe_rshamt(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::BinaryImm { arg, imm, .. } = func.dfg[inst] { - put_rshamt(func.encodings[inst].bits(), - func.locations[arg].unwrap_reg(), - imm.into(), - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - } else { - panic!("Expected BinaryImm format: {:?}", func.dfg[inst]); - } -} - /// I-type instructions. /// /// 31 19 14 11 6 @@ -148,73 +112,6 @@ fn put_i(bits: u16, rs1: RegUnit, imm: i64, rd: RegUnit, sink.put4(i); } -fn recipe_i(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::BinaryImm { arg, imm, .. } = func.dfg[inst] { - put_i(func.encodings[inst].bits(), - func.locations[arg].unwrap_reg(), - imm.into(), - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - } else { - panic!("Expected BinaryImm format: {:?}", func.dfg[inst]); - } -} - -fn recipe_iz(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::UnaryImm { imm, .. } = func.dfg[inst] { - put_i(func.encodings[inst].bits(), - 0, - imm.into(), - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - } else { - panic!("Expected UnaryImm format: {:?}", func.dfg[inst]); - } -} - -fn recipe_iicmp(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::IntCompareImm { arg, imm, .. } = func.dfg[inst] { - put_i(func.encodings[inst].bits(), - func.locations[arg].unwrap_reg(), - imm.into(), - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - } else { - panic!("Expected IntCompareImm format: {:?}", func.dfg[inst]); - } -} - -fn recipe_iret(func: &Function, inst: Inst, sink: &mut CS) { - // Return instructions are always a jalr to %x1. - // The return address is provided as a special-purpose link argument. - put_i(func.encodings[inst].bits(), - 1, // rs1 = %x1 - 0, // no offset. - 0, // rd = %x0: no address written. - sink); -} - -fn recipe_icall(func: &Function, inst: Inst, sink: &mut CS) { - // Indirect instructions are jalr with rd=%x1. - put_i(func.encodings[inst].bits(), - func.locations[func.dfg.inst_args(inst)[0]].unwrap_reg(), - 0, // no offset. - 1, // rd = %x1: link register. - sink); -} - -fn recipe_icopy(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Unary { arg, .. } = func.dfg[inst] { - put_i(func.encodings[inst].bits(), - func.locations[arg].unwrap_reg(), - 0, - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - } else { - panic!("Expected Unary format: {:?}", func.dfg[inst]); - } -} - /// U-type instructions. /// /// 31 11 6 @@ -236,17 +133,6 @@ fn put_u(bits: u16, imm: i64, rd: RegUnit, sink: &mut CS) sink.put4(i); } -fn recipe_u(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::UnaryImm { imm, .. } = func.dfg[inst] { - put_u(func.encodings[inst].bits(), - imm.into(), - func.locations[func.dfg.first_result(inst)].unwrap_reg(), - sink); - } else { - panic!("Expected UnaryImm format: {:?}", func.dfg[inst]); - } -} - /// SB-type branch instructions. /// /// 31 24 19 14 11 6 @@ -280,44 +166,6 @@ fn put_sb(bits: u16, imm: i64, rs1: RegUnit, rs2: RegUnit sink.put4(i); } -fn recipe_sb(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::BranchIcmp { - destination, - ref args, - .. - } = func.dfg[inst] { - let dest = func.offsets[destination] as i64; - let disp = dest - sink.offset() as i64; - let args = &args.as_slice(&func.dfg.value_lists)[0..2]; - put_sb(func.encodings[inst].bits(), - disp, - func.locations[args[0]].unwrap_reg(), - func.locations[args[1]].unwrap_reg(), - sink); - } else { - panic!("Expected BranchIcmp format: {:?}", func.dfg[inst]); - } -} - -fn recipe_sbzero(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Branch { - destination, - ref args, - .. - } = func.dfg[inst] { - let dest = func.offsets[destination] as i64; - let disp = dest - sink.offset() as i64; - let args = &args.as_slice(&func.dfg.value_lists)[0..1]; - put_sb(func.encodings[inst].bits(), - disp, - func.locations[args[0]].unwrap_reg(), - 0, - sink); - } else { - panic!("Expected Branch format: {:?}", func.dfg[inst]); - } -} - /// UJ-type jump instructions. /// /// 31 11 6 @@ -346,31 +194,3 @@ fn put_uj(bits: u16, imm: i64, rd: RegUnit, sink: &mut CS sink.put4(i); } - -fn recipe_uj(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Jump { destination, .. } = func.dfg[inst] { - let dest = func.offsets[destination] as i64; - let disp = dest - sink.offset() as i64; - put_uj(func.encodings[inst].bits(), disp, 0, sink); - } else { - panic!("Expected Jump format: {:?}", func.dfg[inst]); - } -} - -fn recipe_ujcall(func: &Function, inst: Inst, sink: &mut CS) { - if let InstructionData::Call { func_ref, .. } = func.dfg[inst] { - sink.reloc_func(RelocKind::Call.into(), func_ref); - // rd=%x1 is the standard link register. - put_uj(func.encodings[inst].bits(), 0, 1, sink); - } else { - panic!("Expected Call format: {:?}", func.dfg[inst]); - } -} - -fn recipe_gpsp(_func: &Function, _inst: Inst, _sink: &mut CS) { - unimplemented!(); -} - -fn recipe_gpfi(_func: &Function, _inst: Inst, _sink: &mut CS) { - unimplemented!(); -} From fc11ae7b729c6d43fcb58a8f1c810d24a4c3e0df Mon Sep 17 00:00:00 2001 From: d1m0 Date: Mon, 10 Jul 2017 15:28:32 -0700 Subject: [PATCH 0852/3084] Emit runtime type checks in legalizer.rs (#112) * Emit runtime type checks in legalizer.rs --- lib/cretonne/meta/cdsl/ast.py | 39 +++++- lib/cretonne/meta/cdsl/test_ti.py | 26 +--- lib/cretonne/meta/cdsl/ti.py | 175 ++++++++++++++++++++---- lib/cretonne/meta/cdsl/xform.py | 8 ++ lib/cretonne/meta/gen_instr.py | 31 +++-- lib/cretonne/meta/gen_legalizer.py | 125 +++++++++++++++-- lib/cretonne/meta/test_gen_legalizer.py | 145 ++++++++++++++++++++ lib/cretonne/src/ir/instructions.rs | 12 +- lib/cretonne/src/legalizer/mod.rs | 2 + 9 files changed, 494 insertions(+), 69 deletions(-) create mode 100644 lib/cretonne/meta/test_gen_legalizer.py diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index 6efc492cf5..0d87bf8914 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -10,7 +10,7 @@ from .typevar import TypeVar from .predicates import IsEqual, And try: - from typing import Union, Tuple, Sequence, TYPE_CHECKING # noqa + from typing import Union, Tuple, Sequence, TYPE_CHECKING, Dict, List # noqa if TYPE_CHECKING: from .operands import ImmediateKind # noqa from .predicates import PredNode # noqa @@ -18,6 +18,19 @@ except ImportError: pass +def replace_var(arg, m): + # type: (Expr, Dict[Var, Var]) -> Expr + """ + Given a var v return either m[v] or a new variable v' (and remember + m[v]=v'). Otherwise return the argument unchanged + """ + if isinstance(arg, Var): + new_arg = m.get(arg, Var(arg.name)) # type: Var + m[arg] = new_arg + return new_arg + return arg + + class Def(object): """ An AST definition associates a set of variables with the values produced by @@ -60,6 +73,21 @@ class Def(object): return "({}) << {!s}".format( ', '.join(map(str, self.defs)), self.expr) + def copy(self, m): + # type: (Dict[Var, Var]) -> Def + """ + Return a copy of this Def with vars replaced with fresh variables, + in accordance with the map m. Update m as neccessary. + """ + new_expr = self.expr.copy(m) + new_defs = [] # type: List[Var] + for v in self.defs: + new_v = replace_var(v, m) + assert(isinstance(new_v, Var)) + new_defs.append(new_v) + + return Def(tuple(new_defs), new_expr) + class Expr(object): """ @@ -303,6 +331,15 @@ class Apply(Expr): return pred + def copy(self, m): + # type: (Dict[Var, Var]) -> Apply + """ + Return a copy of this Expr with vars replaced with fresh variables, + in accordance with the map m. Update m as neccessary. + """ + return Apply(self.inst, tuple(map(lambda e: replace_var(e, m), + self.args))) + class Enumerator(Expr): """ diff --git a/lib/cretonne/meta/cdsl/test_ti.py b/lib/cretonne/meta/cdsl/test_ti.py index 396d5ef844..71f439864e 100644 --- a/lib/cretonne/meta/cdsl/test_ti.py +++ b/lib/cretonne/meta/cdsl/test_ti.py @@ -6,30 +6,17 @@ from base.immediates import intcc from .typevar import TypeVar from .ast import Var, Def from .xform import Rtl, XForm -from .ti import ti_rtl, subst, TypeEnv, get_type_env +from .ti import ti_rtl, subst, TypeEnv, get_type_env, ConstrainTVsEqual from unittest import TestCase from functools import reduce try: from .ti import TypeMap, ConstraintList, VarMap, TypingOrError # noqa - from .ti import Constraint from typing import List, Dict, Tuple, TYPE_CHECKING, cast # noqa except ImportError: TYPE_CHECKING = False -def sort_constr(c): - # type: (Constraint) -> Constraint - """ - Sort the 2 typevars in a constraint by name for comparison - """ - r = tuple(sorted(c, key=lambda y: y.name)) - if TYPE_CHECKING: - return cast(Constraint, r) - else: - return r - - def agree(me, other): # type: (TypeEnv, TypeEnv) -> bool """ @@ -63,13 +50,10 @@ def agree(me, other): return False # Translate our constraints using m, and sort - me_equiv_constr = [(subst(a, m), subst(b, m)) for (a, b) in me.constraints] - me_equiv_constr = sorted([sort_constr(x) for x in me_equiv_constr]) - + me_equiv_constr = sorted([constr.translate(m) + for constr in me.constraints]) # Sort other's constraints - other_equiv_constr = sorted([sort_constr(x) for x in other.constraints], - key=lambda y: y[0].name) - + other_equiv_constr = sorted(other.constraints) return me_equiv_constr == other_equiv_constr @@ -224,7 +208,7 @@ class TestRTL(TypeCheckingBaseTest): self.v3: txn, self.v4: txn, self.v5: txn, - }, [(ixn.as_bool(), txn.as_bool())])) + }, [ConstrainTVsEqual(ixn.as_bool(), txn.as_bool())])) def test_vselect_vsplits(self): # type: () -> None diff --git a/lib/cretonne/meta/cdsl/ti.py b/lib/cretonne/meta/cdsl/ti.py index ce2f05c750..fe602e1342 100644 --- a/lib/cretonne/meta/cdsl/ti.py +++ b/lib/cretonne/meta/cdsl/ti.py @@ -8,13 +8,12 @@ from itertools import product try: from typing import Dict, TYPE_CHECKING, Union, Tuple, Optional, Set # noqa - from typing import Iterable # noqa - from typing import cast, List + from typing import Iterable, List # noqa + from typing import cast from .xform import Rtl, XForm # noqa from .ast import Expr # noqa + from .typevar import TypeSet # noqa if TYPE_CHECKING: - Constraint = Tuple[TypeVar, TypeVar] - ConstraintList = List[Constraint] TypeMap = Dict[TypeVar, TypeVar] VarMap = Dict[Var, TypeVar] except ImportError: @@ -22,6 +21,122 @@ except ImportError: pass +class TypeConstraint(object): + """ + Base class for all runtime-emittable type constraints. + """ + + +class ConstrainTVsEqual(TypeConstraint): + """ + Constraint specifying that two derived type vars must have the same runtime + type. + """ + def __init__(self, tv1, tv2): + # type: (TypeVar, TypeVar) -> None + assert tv1.is_derived and tv2.is_derived + (self.tv1, self.tv2) = sorted([tv1, tv2], key=repr) + + def is_trivial(self): + # type: () -> bool + """ + Return true if this constrain is statically decidable. + """ + return self.tv1 == self.tv2 or \ + (self.tv1.singleton_type() is not None and + self.tv2.singleton_type() is not None) + + def translate(self, m): + # type: (Union[TypeEnv, TypeMap]) -> ConstrainTVsEqual + """ + Translate any TypeVars in the constraint according to the map m + """ + if isinstance(m, TypeEnv): + return ConstrainTVsEqual(m[self.tv1], m[self.tv2]) + else: + return ConstrainTVsEqual(subst(self.tv1, m), subst(self.tv2, m)) + + def __eq__(self, other): + # type: (object) -> bool + if (not isinstance(other, ConstrainTVsEqual)): + return False + + return (self.tv1, self.tv2) == (other.tv1, other.tv2) + + def __hash__(self): + # type: () -> int + return hash((self.tv1, self.tv2)) + + def eval(self): + # type: () -> bool + """ + Evaluate this constraint. Should only be called when the constraint has + been translated to concrete types. + """ + assert self.tv1.singleton_type() is not None and \ + self.tv2.singleton_type() is not None + return self.tv1.singleton_type() == self.tv2.singleton_type() + + +class ConstrainTVInTypeset(TypeConstraint): + """ + Constraint specifying that a type var must belong to some typeset. + """ + def __init__(self, tv, ts): + # type: (TypeVar, TypeSet) -> None + assert not tv.is_derived and tv.name.startswith("typeof_") + self.tv = tv + self.ts = ts + + def is_trivial(self): + # type: () -> bool + """ + Return true if this constrain is statically decidable. + """ + tv_ts = self.tv.get_typeset().copy() + + # Trivially True + if (tv_ts.issubset(self.ts)): + return True + + # Trivially false + tv_ts &= self.ts + if (tv_ts.size() == 0): + return True + + return False + + def translate(self, m): + # type: (Union[TypeEnv, TypeMap]) -> ConstrainTVInTypeset + """ + Translate any TypeVars in the constraint according to the map m + """ + if isinstance(m, TypeEnv): + return ConstrainTVInTypeset(m[self.tv], self.ts) + else: + return ConstrainTVInTypeset(subst(self.tv, m), self.ts) + + def __eq__(self, other): + # type: (object) -> bool + if (not isinstance(other, ConstrainTVInTypeset)): + return False + + return (self.tv, self.ts) == (other.tv, other.ts) + + def __hash__(self): + # type: () -> int + return hash((self.tv, self.ts)) + + def eval(self): + # type: () -> bool + """ + Evaluate this constraint. Should only be called when the constraint has + been translated to concrete types. + """ + assert self.tv.singleton_type() is not None + return self.tv.get_typeset().issubset(self.ts) + + class TypeEnv(object): """ Class encapsulating the neccessary book keeping for type inference. @@ -43,13 +158,13 @@ class TypeEnv(object): RANK_INTERNAL = 0 def __init__(self, arg=None): - # type: (Optional[Tuple[TypeMap, ConstraintList]]) -> None + # type: (Optional[Tuple[TypeMap, List[TypeConstraint]]]) -> None self.ranks = {} # type: Dict[TypeVar, int] self.vars = set() # type: Set[Var] if arg is None: self.type_map = {} # type: TypeMap - self.constraints = [] # type: ConstraintList + self.constraints = [] # type: List[TypeConstraint] else: self.type_map, self.constraints = arg @@ -94,7 +209,9 @@ class TypeEnv(object): """ Add a new equivalence constraint between tv1 and tv2 """ - self.constraints.append((tv1, tv2)) + constr = ConstrainTVsEqual(tv1, tv2) + if (constr not in self.constraints): + self.constraints.append(constr) def get_uid(self): # type: () -> str @@ -206,15 +323,24 @@ class TypeEnv(object): """ vars_tvs = set([v.get_typevar() for v in self.vars]) new_type_map = {tv: self[tv] for tv in vars_tvs if tv != self[tv]} - new_constraints = [(self[tv1], self[tv2]) - for (tv1, tv2) in self.constraints] - # Sanity: new constraints and the new type_map should only contain - # tvs associated with real vars - for (a, b) in new_constraints: - assert a.free_typevar() in vars_tvs and\ - b.free_typevar() in vars_tvs + new_constraints = [] # type: List[TypeConstraint] + for constr in self.constraints: + # Currently typeinference only generates ConstrainTVsEqual + # constraints + assert isinstance(constr, ConstrainTVsEqual) + constr = constr.translate(self) + if constr.is_trivial() or constr in new_constraints: + continue + + # Sanity: translated constraints should refer to only real vars + assert constr.tv1.free_typevar() in vars_tvs and\ + constr.tv2.free_typevar() in vars_tvs + + new_constraints.append(constr) + + # Sanity: translated typemap should refer to only real vars for (k, v) in new_type_map.items(): assert k in vars_tvs assert v.free_typevar() is None or v.free_typevar() in vars_tvs @@ -245,13 +371,13 @@ class TypeEnv(object): # Check if constraints are satisfied for this typing failed = None - for (tv1, tv2) in self.constraints: - tv1 = subst(tv1, m) - tv2 = subst(tv2, m) - assert tv1.get_typeset().size() == 1 and\ - tv2.get_typeset().size() == 1 - if (tv1.get_typeset() != tv2.get_typeset()): - failed = (tv1, tv2) + for constr in self.constraints: + # Currently typeinference only generates ConstrainTVsEqual + # constraints + assert isinstance(constr, ConstrainTVsEqual) + concrete_constr = constr.translate(m) + if not concrete_constr.eval(): + failed = concrete_constr break if (failed is not None): @@ -287,9 +413,10 @@ class TypeEnv(object): edges.add((v, v.base, "solid", v.derived_func)) v = v.base - for (a, b) in self.constraints: - assert a in nodes and b in nodes - edges.add((a, b, "dashed", None)) + for constr in self.constraints: + assert isinstance(constr, ConstrainTVsEqual) + assert constr.tv1 in nodes and constr.tv2 in nodes + edges.add((constr.tv1, constr.tv2, "dashed", None)) root_nodes = set([x for x in nodes if x not in self.type_map and not x.is_derived]) diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index fd3c1dff1f..f809a91ce6 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -41,6 +41,14 @@ class Rtl(object): # type: (*DefApply) -> None self.rtl = tuple(map(canonicalize_defapply, args)) + def copy(self, m): + # type: (Dict[Var, Var]) -> Rtl + """ + Return a copy of this rtl with all Vars substituted with copies or + according to m. Update m as neccessary. + """ + return Rtl(*[d.copy(m) for d in self.rtl]) + class XForm(object): """ diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 2d67311ae2..242e5152d3 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -336,6 +336,25 @@ def get_constraint(op, ctrl_typevar, type_sets): return 'Same' +# TypeSet indexes are encoded in 8 bits, with `0xff` reserved. +typeset_limit = 0xff + + +def gen_typesets_table(fmt, type_sets): + # type: (srcgen.Formatter, UniqueTable) -> None + """ + Generate the table of ValueTypeSets described by type_sets. + """ + fmt.comment('Table of value type sets.') + assert len(type_sets.table) <= typeset_limit, "Too many type sets" + with fmt.indented( + 'const TYPE_SETS : [ValueTypeSet; {}] = [' + .format(len(type_sets.table)), '];'): + for ts in type_sets.table: + with fmt.indented('ValueTypeSet {', '},'): + ts.emit_fields(fmt) + + def gen_type_constraints(fmt, instrs): # type: (srcgen.Formatter, Sequence[Instruction]) -> None """ @@ -360,9 +379,6 @@ def gen_type_constraints(fmt, instrs): # Preload table with constraints for typical binops. operand_seqs.add(['Same'] * 3) - # TypeSet indexes are encoded in 8 bits, with `0xff` reserved. - typeset_limit = 0xff - fmt.comment('Table of opcode constraints.') with fmt.indented( 'const OPCODE_CONSTRAINTS : [OpcodeConstraints; {}] = [' @@ -418,14 +434,7 @@ def gen_type_constraints(fmt, instrs): fmt.line('typeset_offset: {},'.format(ctrl_typeset)) fmt.line('constraint_offset: {},'.format(offset)) - fmt.comment('Table of value type sets.') - assert len(type_sets.table) <= typeset_limit, "Too many type sets" - with fmt.indented( - 'const TYPE_SETS : [ValueTypeSet; {}] = [' - .format(len(type_sets.table)), '];'): - for ts in type_sets.table: - with fmt.indented('ValueTypeSet {', '},'): - ts.emit_fields(fmt) + gen_typesets_table(fmt, type_sets) fmt.comment('Table of operand constraint sequences.') with fmt.indented( diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 1376e3bc21..ce2f559ef2 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -11,16 +11,116 @@ from __future__ import absolute_import from srcgen import Formatter from base import legalize, instructions from cdsl.ast import Var +from cdsl.ti import ti_rtl, TypeEnv, get_type_env, ConstrainTVsEqual,\ + ConstrainTVInTypeset +from unique_table import UniqueTable +from gen_instr import gen_typesets_table +from cdsl.typevar import TypeVar try: - from typing import Sequence # noqa + from typing import Sequence, List, Dict # noqa from cdsl.isa import TargetISA # noqa from cdsl.ast import Def # noqa from cdsl.xform import XForm, XFormGroup # noqa + from cdsl.typevar import TypeSet # noqa + from cdsl.ti import TypeConstraint # noqa except ImportError: pass +def get_runtime_typechecks(xform): + # type: (XForm) -> List[TypeConstraint] + """ + Given a XForm build a list of runtime type checks neccessary to determine + if it applies. We have 2 types of runtime checks: + 1) typevar tv belongs to typeset T - needed for free tvs whose + typeset is constrainted by their use in the dst pattern + + 2) tv1 == tv2 where tv1 and tv2 are derived TVs - caused by unification + of non-bijective functions + """ + check_l = [] # type: List[TypeConstraint] + + # 1) Perform ti only on the source RTL. Accumulate any free tvs that have a + # different inferred type in src, compared to the type inferred for both + # src and dst. + symtab = {} # type: Dict[Var, Var] + src_copy = xform.src.copy(symtab) + src_typenv = get_type_env(ti_rtl(src_copy, TypeEnv())) + + for v in xform.ti.vars: + if not v.has_free_typevar(): + continue + + # In rust the local variable containing a free TV associated with var v + # has name typeof_v. We rely on the python TVs having the same name. + assert "typeof_{}".format(v) == xform.ti[v].name + + if v not in symtab: + # We can have singleton vars defined only on dst. Ignore them + assert v.get_typevar().singleton_type() is not None + continue + + src_ts = src_typenv[symtab[v]].get_typeset() + xform_ts = xform.ti[v].get_typeset() + + assert xform_ts.issubset(src_ts) + if src_ts != xform_ts: + check_l.append(ConstrainTVInTypeset(xform.ti[v], xform_ts)) + + # 2,3) Add any constraints that appear in xform.ti + check_l.extend(xform.ti.constraints) + + return check_l + + +def emit_runtime_typecheck(check, fmt, type_sets): + # type: (TypeConstraint, Formatter, UniqueTable) -> None + """ + Emit rust code for the given check. + """ + def build_derived_expr(tv): + # type: (TypeVar) -> str + if not tv.is_derived: + assert tv.name.startswith('typeof_') + return "Some({})".format(tv.name) + + base_exp = build_derived_expr(tv.base) + if (tv.derived_func == TypeVar.LANEOF): + return "{}.map(|t: Type| -> t.lane_type())".format(base_exp) + elif (tv.derived_func == TypeVar.ASBOOL): + return "{}.map(|t: Type| -> t.as_bool())".format(base_exp) + elif (tv.derived_func == TypeVar.HALFWIDTH): + return "{}.and_then(|t: Type| -> t.half_width())".format(base_exp) + elif (tv.derived_func == TypeVar.DOUBLEWIDTH): + return "{}.and_then(|t: Type| -> t.double_width())"\ + .format(base_exp) + elif (tv.derived_func == TypeVar.HALFVECTOR): + return "{}.and_then(|t: Type| -> t.half_vector())".format(base_exp) + elif (tv.derived_func == TypeVar.DOUBLEVECTOR): + return "{}.and_then(|t: Type| -> t.by(2))".format(base_exp) + else: + assert False, "Unknown derived function {}".format(tv.derived_func) + + if (isinstance(check, ConstrainTVInTypeset)): + tv = check.tv.name + if check.ts not in type_sets.index: + type_sets.add(check.ts) + ts = type_sets.index[check.ts] + + fmt.comment("{} must belong to {}".format(tv, check.ts)) + with fmt.indented('if !TYPE_SETS[{}].contains({}) {{'.format(ts, tv), + '};'): + fmt.line('return false;') + elif (isinstance(check, ConstrainTVsEqual)): + tv1 = build_derived_expr(check.tv1) + tv2 = build_derived_expr(check.tv2) + with fmt.indented('if {} != {} {{'.format(tv1, tv2), '};'): + fmt.line('return false;') + else: + assert False, "Unknown check {}".format(check) + + def unwrap_inst(iref, node, fmt): # type: (str, Def, Formatter) -> bool """ @@ -183,8 +283,8 @@ def emit_dst_inst(node, fmt): fmt.line('pos.next_inst();') -def gen_xform(xform, fmt): - # type: (XForm, Formatter) -> None +def gen_xform(xform, fmt, type_sets): + # type: (XForm, Formatter, UniqueTable) -> None """ Emit code for `xform`, assuming the the opcode of xform's root instruction has already been matched. @@ -203,6 +303,10 @@ def gen_xform(xform, fmt): instp = xform.src.rtl[0].expr.inst_predicate() assert instp is None, "Instruction predicates not supported in legalizer" + # Emit any runtime checks. + for check in get_runtime_typechecks(xform): + emit_runtime_typecheck(check, fmt, type_sets) + # Emit the destination pattern. for dst in xform.dst.rtl: emit_dst_inst(dst, fmt) @@ -213,8 +317,8 @@ def gen_xform(xform, fmt): fmt.line('assert_eq!(pos.remove_inst(), inst);') -def gen_xform_group(xgrp, fmt): - # type: (XFormGroup, Formatter) -> None +def gen_xform_group(xgrp, fmt, type_sets): + # type: (XFormGroup, Formatter, UniqueTable) -> None fmt.doc_comment("Legalize the instruction pointed to by `pos`.") fmt.line('#[allow(unused_variables,unused_assignments)]') with fmt.indented( @@ -231,7 +335,7 @@ def gen_xform_group(xgrp, fmt): inst = xform.src.rtl[0].expr.inst with fmt.indented( 'Opcode::{} => {{'.format(inst.camel_name), '}'): - gen_xform(xform, fmt) + gen_xform(xform, fmt, type_sets) # We'll assume there are uncovered opcodes. fmt.line('_ => return false,') fmt.line('true') @@ -240,6 +344,11 @@ def gen_xform_group(xgrp, fmt): def generate(isas, out_dir): # type: (Sequence[TargetISA], str) -> None fmt = Formatter() - gen_xform_group(legalize.narrow, fmt) - gen_xform_group(legalize.expand, fmt) + # Table of TypeSet instances + type_sets = UniqueTable() + + gen_xform_group(legalize.narrow, fmt, type_sets) + gen_xform_group(legalize.expand, fmt, type_sets) + + gen_typesets_table(fmt, type_sets) fmt.update_file('legalizer.rs', out_dir) diff --git a/lib/cretonne/meta/test_gen_legalizer.py b/lib/cretonne/meta/test_gen_legalizer.py new file mode 100644 index 0000000000..38f26959f4 --- /dev/null +++ b/lib/cretonne/meta/test_gen_legalizer.py @@ -0,0 +1,145 @@ +import doctest +import gen_legalizer +from unittest import TestCase +from srcgen import Formatter +from gen_legalizer import get_runtime_typechecks, emit_runtime_typecheck +from base.instructions import vselect, vsplit, isplit, iconcat, vconcat, \ + iconst, b1, icmp, copy # noqa +from base.legalize import narrow, expand # noqa +from base.immediates import intcc # noqa +from cdsl.typevar import TypeVar, TypeSet +from cdsl.ast import Var, Def # noqa +from cdsl.xform import Rtl, XForm # noqa +from cdsl.ti import ti_rtl, subst, TypeEnv, get_type_env # noqa +from unique_table import UniqueTable +from functools import reduce + +try: + from typing import Callable, TYPE_CHECKING, Iterable, Any # noqa + if TYPE_CHECKING: + CheckProducer = Callable[[UniqueTable], str] +except ImportError: + TYPE_CHECKING = False + + +def load_tests(loader, tests, ignore): + # type: (Any, Any, Any) -> Any + tests.addTests(doctest.DocTestSuite(gen_legalizer)) + return tests + + +def format_check(typesets, s, *args): + # type: (...) -> str + def transform(x): + # type: (Any) -> str + if isinstance(x, TypeSet): + return str(typesets.index[x]) + elif isinstance(x, TypeVar): + assert not x.is_derived + return x.name + else: + return str(x) + + dummy_s = s # type: str + args = tuple(map(lambda x: transform(x), args)) + return dummy_s.format(*args) + + +def typeset_check(v, ts): + # type: (Var, TypeSet) -> CheckProducer + return lambda typesets: format_check( + typesets, + 'if !TYPE_SETS[{}].contains(typeof_{}) ' + + '{{\n return false;\n}};\n', ts, v) + + +def equiv_check(tv1, tv2): + # type: (TypeVar, TypeVar) -> CheckProducer + return lambda typesets: format_check( + typesets, + 'if Some({}).map(|t: Type| -> t.as_bool()) != ' + + 'Some({}).map(|t: Type| -> t.as_bool()) ' + + '{{\n return false;\n}};\n', tv1, tv2) + + +def sequence(*args): + # type: (...) -> CheckProducer + dummy = args # type: Iterable[CheckProducer] + + def sequenceF(typesets): + # type: (UniqueTable) -> str + def strconcat(acc, el): + # type: (str, CheckProducer) -> str + return acc + el(typesets) + + return reduce(strconcat, dummy, "") + return sequenceF + + +class TestRuntimeChecks(TestCase): + + def setUp(self): + # type: () -> None + self.v0 = Var("v0") + self.v1 = Var("v1") + self.v2 = Var("v2") + self.v3 = Var("v3") + self.v4 = Var("v4") + self.v5 = Var("v5") + self.v6 = Var("v6") + self.v7 = Var("v7") + self.v8 = Var("v8") + self.v9 = Var("v9") + self.imm0 = Var("imm0") + self.IxN_nonscalar = TypeVar("IxN_nonscalar", "", ints=True, + scalars=False, simd=True) + self.TxN = TypeVar("TxN", "", ints=True, bools=True, floats=True, + scalars=False, simd=True) + self.b1 = TypeVar.singleton(b1) + + def check_yo_check(self, xform, expected_f): + # type: (XForm, CheckProducer) -> None + fmt = Formatter() + type_sets = UniqueTable() + for check in get_runtime_typechecks(xform): + emit_runtime_typecheck(check, fmt, type_sets) + + # Remove comments + got = "".join([l for l in fmt.lines if not l.strip().startswith("//")]) + expected = expected_f(type_sets) + self.assertEqual(got, expected) + + def test_width_check(self): + # type: () -> None + x = XForm(Rtl(self.v0 << copy(self.v1)), + Rtl((self.v2, self.v3) << isplit(self.v1), + self.v0 << iconcat(self.v2, self.v3))) + + WideInt = TypeSet(lanes=(1, 256), ints=(16, 64)) + self.check_yo_check(x, typeset_check(self.v1, WideInt)) + + def test_lanes_check(self): + # type: () -> None + x = XForm(Rtl(self.v0 << copy(self.v1)), + Rtl((self.v2, self.v3) << vsplit(self.v1), + self.v0 << vconcat(self.v2, self.v3))) + + WideVec = TypeSet(lanes=(2, 256), ints=(8, 64), floats=(32, 64), + bools=(1, 64)) + self.check_yo_check(x, typeset_check(self.v1, WideVec)) + + def test_vselect_imm(self): + # type: () -> None + ts = TypeSet(lanes=(2, 256), ints=(8, 64), + floats=(32, 64), bools=(8, 64)) + r = Rtl( + self.v0 << iconst(self.imm0), + self.v1 << icmp(intcc.eq, self.v2, self.v0), + self.v5 << vselect(self.v1, self.v3, self.v4), + ) + x = XForm(r, r) + + self.check_yo_check( + x, sequence(typeset_check(self.v3, ts), + equiv_check(self.v2.get_typevar(), + self.v3.get_typevar()))) diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 35a200de08..cf914b853a 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -506,10 +506,14 @@ type BitSet16 = BitSet; /// A value type set describes the permitted set of types for a type variable. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct ValueTypeSet { - lanes: BitSet16, - ints: BitSet8, - floats: BitSet8, - bools: BitSet8, + /// Allowed lane sizes + pub lanes: BitSet16, + /// Allowed int widths + pub ints: BitSet8, + /// Allowed float widths + pub floats: BitSet8, + /// Allowed bool widths + pub bools: BitSet8, } impl ValueTypeSet { diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 803f5808c2..f21f7e1d91 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -18,6 +18,8 @@ use flowgraph::ControlFlowGraph; use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, InstBuilder}; use ir::condcodes::IntCC; use isa::{TargetIsa, Legalize}; +use bitset::BitSet; +use ir::instructions::ValueTypeSet; mod boundary; mod split; From f837dcf4b7fe9987de2a24531196bbdc70482d0c Mon Sep 17 00:00:00 2001 From: d1m0 Date: Tue, 11 Jul 2017 08:39:22 -0700 Subject: [PATCH 0853/3084] Handle bound instructions in pattern type inference (#113) --- lib/cretonne/meta/cdsl/test_ti.py | 106 +++++++++++++++++++++++++++++- lib/cretonne/meta/cdsl/ti.py | 17 +++-- 2 files changed, 115 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/meta/cdsl/test_ti.py b/lib/cretonne/meta/cdsl/test_ti.py index 71f439864e..e89cedc16c 100644 --- a/lib/cretonne/meta/cdsl/test_ti.py +++ b/lib/cretonne/meta/cdsl/test_ti.py @@ -1,8 +1,9 @@ from __future__ import absolute_import from base.instructions import vselect, vsplit, vconcat, iconst, iadd, bint,\ - b1, icmp, iadd_cout, iadd_cin + b1, icmp, iadd_cout, iadd_cin, uextend, ireduce from base.legalize import narrow, expand from base.immediates import intcc +from base.types import i32, i8 from .typevar import TypeVar from .ast import Var, Def from .xform import Rtl, XForm @@ -171,7 +172,7 @@ class TestRTL(TypeCheckingBaseTest): ) ti = TypeEnv() self.assertEqual(ti_rtl(r, ti), - "On line 1: fail ti on `typeof_v2` <: `2`: " + + "On line 1: fail ti on `typeof_v2` <: `1`: " + "Error: empty type created when unifying " + "`typeof_v2` and `half_vector(typeof_v2)`") @@ -303,6 +304,21 @@ class TestRTL(TypeCheckingBaseTest): self.v0: itype, }, [])) + def test_fully_bound_inst_inference_bad(self): + # Incompatible bound instructions fail accordingly + r = Rtl( + self.v3 << uextend.i32(self.v1), + self.v4 << uextend.i16(self.v2), + self.v5 << iadd(self.v3, self.v4), + ) + ti = TypeEnv() + typing = ti_rtl(r, ti) + + self.assertEqual(typing, + "On line 2: fail ti on `typeof_v4` <: `4`: " + + "Error: empty type created when unifying " + + "`typeof_v4` and `typeof_v5`") + class TestXForm(TypeCheckingBaseTest): def test_iadd_cout(self): @@ -414,3 +430,89 @@ class TestXForm(TypeCheckingBaseTest): # xform for concrete_typing in concrete_typings_list: check_concrete_typing_xform(concrete_typing, xform) + + def test_bound_inst_inference(self): + # First example from issue #26 + x = XForm( + Rtl( + self.v0 << iadd(self.v1, self.v2), + ), + Rtl( + self.v3 << uextend.i32(self.v1), + self.v4 << uextend.i32(self.v2), + self.v5 << iadd(self.v3, self.v4), + self.v0 << ireduce(self.v5) + )) + itype = TypeVar("t", "", ints=True, simd=True) + i32t = TypeVar.singleton(i32) + + check_typing(x.ti, ({ + self.v0: itype, + self.v1: itype, + self.v2: itype, + self.v3: i32t, + self.v4: i32t, + self.v5: i32t, + }, []), x.symtab) + + def test_bound_inst_inference1(self): + # Second example taken from issue #26 + x = XForm( + Rtl( + self.v0 << iadd(self.v1, self.v2), + ), + Rtl( + self.v3 << uextend(self.v1), + self.v4 << uextend(self.v2), + self.v5 << iadd.i32(self.v3, self.v4), + self.v0 << ireduce(self.v5) + )) + itype = TypeVar("t", "", ints=True, simd=True) + i32t = TypeVar.singleton(i32) + + check_typing(x.ti, ({ + self.v0: itype, + self.v1: itype, + self.v2: itype, + self.v3: i32t, + self.v4: i32t, + self.v5: i32t, + }, []), x.symtab) + + def test_fully_bound_inst_inference(self): + # Second example taken from issue #26 with complete bounds + x = XForm( + Rtl( + self.v0 << iadd(self.v1, self.v2), + ), + Rtl( + self.v3 << uextend.i32.i8(self.v1), + self.v4 << uextend.i32.i8(self.v2), + self.v5 << iadd(self.v3, self.v4), + self.v0 << ireduce(self.v5) + )) + i8t = TypeVar.singleton(i8) + i32t = TypeVar.singleton(i32) + + check_typing(x.ti, ({ + self.v0: i8t, + self.v1: i8t, + self.v2: i8t, + self.v3: i32t, + self.v4: i32t, + self.v5: i32t, + }, []), x.symtab) + + def test_fully_bound_inst_inference_bad(self): + # Can't force a mistyped XForm using bound instructions + with self.assertRaises(AssertionError): + XForm( + Rtl( + self.v0 << iadd(self.v1, self.v2), + ), + Rtl( + self.v3 << uextend.i32.i8(self.v1), + self.v4 << uextend.i32.i16(self.v2), + self.v5 << iadd(self.v3, self.v4), + self.v0 << ireduce(self.v5) + )) diff --git a/lib/cretonne/meta/cdsl/ti.py b/lib/cretonne/meta/cdsl/ti.py index fe602e1342..85bd92507f 100644 --- a/lib/cretonne/meta/cdsl/ti.py +++ b/lib/cretonne/meta/cdsl/ti.py @@ -606,14 +606,19 @@ def ti_def(definition, typ): expr = definition.expr inst = expr.inst - # Create a map m mapping each free typevar in the signature of definition - # to a fresh copy of itself - all_formal_tvs = \ - [inst.outs[i].typevar for i in inst.value_results] +\ - [inst.ins[i].typevar for i in inst.value_opnums] - free_formal_tvs = [tv for tv in all_formal_tvs if not tv.is_derived] + # Create a dict m mapping each free typevar in the signature of definition + # to a fresh copy of itself. + if inst.is_polymorphic: + free_formal_tvs = [inst.ctrl_typevar] + inst.other_typevars + else: + free_formal_tvs = [] + m = {tv: tv.get_fresh_copy(str(typ.get_uid())) for tv in free_formal_tvs} + # Update m with any explicitly bound type vars + for (idx, bound_typ) in enumerate(expr.typevars): + m[free_formal_tvs[idx]] = TypeVar.singleton(bound_typ) + # Get fresh copies for each typevar in the signature (both free and # derived) fresh_formal_tvs = \ From fb227cb389bb357265d13cb4ccbd502bca62f1a0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 10 Jul 2017 14:06:08 -0700 Subject: [PATCH 0854/3084] Move Intel recipe_* bodies into intel/recipes.py. Use a PUT_OP macro in the TailRecipe Python class to replace the code snippet that emits the prefixes + opcode part of the instruction encoding. Prepare for the addition of REX prefixes by giving the PUT_OP functions a third argument representing the REX prefix. For the non-REX encodings, verify that no REX bits wold be needed. --- lib/cretonne/meta/isa/intel/recipes.py | 163 ++++++++++-- lib/cretonne/src/isa/intel/binemit.rs | 332 +++---------------------- 2 files changed, 176 insertions(+), 319 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 3700ff60b5..ebeacc5968 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -43,7 +43,9 @@ OPCODE_PREFIX = { (0xf2, 0x0f, 0x3a): ('Mp3', 0b1111) } -# VEX/XOP and EVEX prefixes are not yet supported. +# The table above does not include the REX prefix which goes after the +# mandatory prefix. VEX/XOP and EVEX prefixes are not yet supported. Encodings +# using any of these prefixes are represented by separate recipes. # # The encoding bits are: # @@ -79,6 +81,18 @@ def decode_ops(ops, rrr=0, w=0): return (name, op | (mmpp << 8) | (rrr << 12) | (w << 15)) +def replace_put_op(emit, prefix): + # type: (str, str) -> str + """ + Given a snippet of Rust code (or None), replace the `PUT_OP` macro with the + corresponding `put_*` function from the `binemit.rs` module. + """ + if emit is None: + return None + else: + return emit.replace('PUT_OP', 'put_' + prefix.lower()) + + class TailRecipe: """ Generate encoding recipes on demand. @@ -92,6 +106,10 @@ class TailRecipe: The arguments are the same as for an `EncRecipe`, except for `size` which does not include the size of the opcode. + + The `emit` parameter contains Rust code to actually emit an encoding, like + `EncRecipe` does it. Additionally, the text `PUT_OP` is substituted with + the proper `put_*` function from the `intel/binemit.rs` module. """ def __init__( @@ -103,7 +121,8 @@ class TailRecipe: outs, # type: ConstraintSeq branch_range=None, # type: BranchRange instp=None, # type: PredNode - isap=None # type: PredNode + isap=None, # type: PredNode + emit=None # type: str ): # type: (...) -> None self.name = name @@ -114,6 +133,7 @@ class TailRecipe: self.branch_range = branch_range self.instp = instp self.isap = isap + self.emit = emit # Cached recipes, keyed by name prefix. self.recipes = dict() # type: Dict[str, EncRecipe] @@ -136,34 +156,69 @@ class TailRecipe: outs=self.outs, branch_range=self.branch_range, instp=self.instp, - isap=self.isap) + isap=self.isap, + emit=replace_put_op(self.emit, name)) return (self.recipes[name], bits) # XX /r -rr = TailRecipe('rr', Binary, size=1, ins=(GPR, GPR), outs=0) +rr = TailRecipe( + 'rr', Binary, size=1, ins=(GPR, GPR), outs=0, + emit=''' + PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + modrm_rr(in_reg0, in_reg1, sink); + ''') # XX /r, but for a unary operator with separate input/output register, like # copies. -ur = TailRecipe('ur', Unary, size=1, ins=GPR, outs=GPR) +ur = TailRecipe( + 'ur', Unary, size=1, ins=GPR, outs=GPR, + emit=''' + PUT_OP(bits, rex2(out_reg0, in_reg0), sink); + modrm_rr(out_reg0, in_reg0, sink); + ''') # XX /n with one arg in %rcx, for shifts. -rc = TailRecipe('rc', Binary, size=1, ins=(GPR, GPR.rcx), outs=0) +rc = TailRecipe( + 'rc', Binary, size=1, ins=(GPR, GPR.rcx), outs=0, + emit=''' + PUT_OP(bits, rex1(in_reg0), sink); + modrm_r_bits(in_reg0, bits, sink); + ''') # XX /n ib with 8-bit immediate sign-extended. rib = TailRecipe( 'rib', BinaryImm, size=2, ins=GPR, outs=0, - instp=IsSignedInt(BinaryImm.imm, 8)) + instp=IsSignedInt(BinaryImm.imm, 8), + emit=''' + PUT_OP(bits, rex1(in_reg0), sink); + modrm_r_bits(in_reg0, bits, sink); + let imm: i64 = imm.into(); + sink.put1(imm as u8); + ''') # XX /n id with 32-bit immediate sign-extended. rid = TailRecipe( 'rid', BinaryImm, size=5, ins=GPR, outs=0, - instp=IsSignedInt(BinaryImm.imm, 32)) + instp=IsSignedInt(BinaryImm.imm, 32), + emit=''' + PUT_OP(bits, rex1(in_reg0), sink); + modrm_r_bits(in_reg0, bits, sink); + let imm: i64 = imm.into(); + sink.put4(imm as u32); + ''') # XX+rd id unary with 32-bit immediate. uid = TailRecipe( 'uid', UnaryImm, size=4, ins=(), outs=GPR, - instp=IsSignedInt(UnaryImm.imm, 32)) + instp=IsSignedInt(UnaryImm.imm, 32), + emit=''' + // The destination register is encoded in the low bits of the opcode. + // No ModR/M. + PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); + let imm: i64 = imm.into(); + sink.put4(imm as u32); + ''') # # Store recipes. @@ -172,26 +227,59 @@ uid = TailRecipe( # XX /r register-indirect store with no offset. st = TailRecipe( 'st', Store, size=1, ins=(GPR, GPR), outs=(), - instp=IsEqual(Store.offset, 0)) + instp=IsEqual(Store.offset, 0), + emit=''' + PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + modrm_rm(in_reg1, in_reg0, sink); + ''') # XX /r register-indirect store with no offset. # Only ABCD allowed for stored value. This is for byte stores. st_abcd = TailRecipe( 'st_abcd', Store, size=1, ins=(ABCD, GPR), outs=(), - instp=IsEqual(Store.offset, 0)) + instp=IsEqual(Store.offset, 0), + emit=''' + PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + modrm_rm(in_reg1, in_reg0, sink); + ''') # XX /r register-indirect store with 8-bit offset. stDisp8 = TailRecipe( 'stDisp8', Store, size=2, ins=(GPR, GPR), outs=(), - instp=IsSignedInt(Store.offset, 8)) + instp=IsSignedInt(Store.offset, 8), + emit=''' + PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + modrm_disp8(in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + ''') stDisp8_abcd = TailRecipe( 'stDisp8_abcd', Store, size=2, ins=(ABCD, GPR), outs=(), - instp=IsSignedInt(Store.offset, 8)) + instp=IsSignedInt(Store.offset, 8), + emit=''' + PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + modrm_disp8(in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + ''') # XX /r register-indirect store with 32-bit offset. -stDisp32 = TailRecipe('stDisp32', Store, size=5, ins=(GPR, GPR), outs=()) +stDisp32 = TailRecipe( + 'stDisp32', Store, size=5, ins=(GPR, GPR), outs=(), + emit=''' + PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + modrm_disp32(in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + ''') stDisp32_abcd = TailRecipe( - 'stDisp32_abcd', Store, size=5, ins=(ABCD, GPR), outs=()) + 'stDisp32_abcd', Store, size=5, ins=(ABCD, GPR), outs=(), + emit=''' + PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + modrm_disp32(in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + ''') # # Load recipes @@ -200,21 +288,54 @@ stDisp32_abcd = TailRecipe( # XX /r load with no offset. ld = TailRecipe( 'ld', Load, size=1, ins=(GPR), outs=(GPR), - instp=IsEqual(Load.offset, 0)) + instp=IsEqual(Load.offset, 0), + emit=''' + PUT_OP(bits, rex2(out_reg0, in_reg0), sink); + modrm_rm(in_reg0, out_reg0, sink); + ''') # XX /r load with 8-bit offset. ldDisp8 = TailRecipe( 'ldDisp8', Load, size=2, ins=(GPR), outs=(GPR), - instp=IsSignedInt(Load.offset, 8)) + instp=IsSignedInt(Load.offset, 8), + emit=''' + PUT_OP(bits, rex2(out_reg0, in_reg0), sink); + modrm_disp8(in_reg0, out_reg0, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + ''') # XX /r load with 32-bit offset. ldDisp32 = TailRecipe( 'ldDisp32', Load, size=5, ins=(GPR), outs=(GPR), - instp=IsSignedInt(Load.offset, 32)) + instp=IsSignedInt(Load.offset, 32), + emit=''' + PUT_OP(bits, rex2(out_reg0, in_reg0), sink); + modrm_disp32(in_reg0, out_reg0, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + ''') # # Call/return # -call_id = TailRecipe('call_id', Call, size=4, ins=(), outs=()) -call_r = TailRecipe('call_r', IndirectCall, size=1, ins=GPR, outs=()) -ret = TailRecipe('ret', MultiAry, size=0, ins=(), outs=()) +call_id = TailRecipe( + 'call_id', Call, size=4, ins=(), outs=(), + emit=''' + PUT_OP(bits, BASE_REX, sink); + sink.reloc_func(RelocKind::PCRel4.into(), func_ref); + sink.put4(0); + ''') + +call_r = TailRecipe( + 'call_r', IndirectCall, size=1, ins=GPR, outs=(), + emit=''' + PUT_OP(bits, rex1(in_reg0), sink); + modrm_r_bits(in_reg0, bits, sink); + ''') + +ret = TailRecipe( + 'ret', MultiAry, size=0, ins=(), outs=(), + emit=''' + PUT_OP(bits, BASE_REX, sink); + ''') diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 831ef4c756..e2cb5396bc 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -1,8 +1,7 @@ //! Emitting binary Intel machine code. use binemit::{CodeSink, Reloc, bad_encoding}; -use ir::{self, Function, Inst, InstructionData, MemFlags}; -use ir::immediates::{Imm64, Offset32}; +use ir::{Function, Inst, InstructionData}; use isa::RegUnit; include!(concat!(env!("OUT_DIR"), "/binemit-intel.rs")); @@ -21,27 +20,51 @@ impl Into for RelocKind { } } -// Emit single-byte opcode. -fn put_op1(bits: u16, sink: &mut CS) { - debug_assert_eq!(bits & 0x0f00, 0, "Invalid encoding bits for Op1*"); +// Mandatory prefix bytes for Mp* opcodes. +const PREFIX: [u8; 3] = [0x66, 0xf3, 0xf2]; + +// A REX prefix with no bits set: 0b0100WRXB. +const BASE_REX: u8 = 0b0100_0000; + +// Create a single-register REX prefix, setting the B bit to bit 3 of the register. +// This is used for instructions that encode a register in the low 3 bits of the opcode and for +// instructions that use the ModR/M `reg` field for something else. +fn rex1(reg_b: RegUnit) -> u8 { + let b = ((reg_b >> 3) & 1) as u8; + BASE_REX | b +} + +// Create a dual-register REX prefix, setting: +// +// REX.B = bit 3 of r/m register. +// REX.R = bit 3 of reg register. +fn rex2(rm: RegUnit, reg: RegUnit) -> u8 { + let b = ((rm >> 3) & 1) as u8; + let r = ((reg >> 3) & 1) as u8; + BASE_REX | b | (r << 2) +} + +// Emit a single-byte opcode with no REX prefix. +fn put_op1(bits: u16, rex: u8, sink: &mut CS) { + debug_assert_eq!(bits & 0x8f00, 0, "Invalid encoding bits for Op1*"); + debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less encoding"); sink.put1(bits as u8); } // Emit two-byte opcode: 0F XX -fn put_op2(bits: u16, sink: &mut CS) { - debug_assert_eq!(bits & 0x0f00, 0x0400, "Invalid encoding bits for Op2*"); +fn put_op2(bits: u16, rex: u8, sink: &mut CS) { + debug_assert_eq!(bits & 0x8f00, 0x0400, "Invalid encoding bits for Op2*"); + debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less encoding"); sink.put1(0x0f); sink.put1(bits as u8); } -// Mandatory prefix bytes for Mp* opcodes. -const PREFIX: [u8; 3] = [0x66, 0xf3, 0xf2]; - // Emit single-byte opcode with mandatory prefix. -fn put_mp1(bits: u16, sink: &mut CS) { +fn put_mp1(bits: u16, rex: u8, sink: &mut CS) { debug_assert_eq!(bits & 0x0c00, 0, "Invalid encoding bits for Mp1*"); let pp = (bits >> 8) & 3; sink.put1(PREFIX[(pp - 1) as usize]); + debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less encoding"); sink.put1(bits as u8); } @@ -100,290 +123,3 @@ fn modrm_disp32(rm: RegUnit, reg: RegUnit, sink: &mut CS) b |= rm; sink.put1(b); } - -fn recipe_op1rr(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - in_reg1: RegUnit) { - put_op1(bits, sink); - modrm_rr(in_reg0, in_reg1, sink); -} - -fn recipe_op1ur(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - out_reg0: RegUnit) { - put_op1(bits, sink); - modrm_rr(out_reg0, in_reg0, sink); -} - -fn recipe_op1rc(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit) { - put_op1(bits, sink); - modrm_r_bits(in_reg0, bits, sink); -} - -fn recipe_op1rib(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - imm: Imm64) { - put_op1(bits, sink); - modrm_r_bits(in_reg0, bits, sink); - let imm: i64 = imm.into(); - sink.put1(imm as u8); -} - -fn recipe_op1rid(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - imm: Imm64) { - put_op1(bits, sink); - modrm_r_bits(in_reg0, bits, sink); - let imm: i64 = imm.into(); - sink.put4(imm as u32); -} - -fn recipe_op1uid(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - imm: Imm64, - out_reg0: RegUnit) { - // The destination register is encoded in the low bits of the opcode. No ModR/M - put_op1(bits | (out_reg0 & 7), sink); - let imm: i64 = imm.into(); - sink.put4(imm as u32); -} - -// Store recipes. - -fn recipe_op1st(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - in_reg1: RegUnit, - _flags: MemFlags, - _offset: Offset32) { - put_op1(bits, sink); - modrm_rm(in_reg1, in_reg0, sink); -} - -// This is just a tighter register class constraint. -fn recipe_op1st_abcd(func: &Function, - inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - in_reg1: RegUnit, - flags: MemFlags, - offset: Offset32) { - recipe_op1st(func, inst, sink, bits, in_reg0, in_reg1, flags, offset) -} - -fn recipe_mp1st(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - in_reg1: RegUnit, - _flags: MemFlags, - _offset: Offset32) { - put_mp1(bits, sink); - modrm_rm(in_reg1, in_reg0, sink); -} - -fn recipe_op1stdisp8(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - in_reg1: RegUnit, - _flags: MemFlags, - offset: Offset32) { - put_op1(bits, sink); - modrm_disp8(in_reg1, in_reg0, sink); - let offset: i32 = offset.into(); - sink.put1(offset as u8); -} - -fn recipe_op1stdisp8_abcd(func: &Function, - inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - in_reg1: RegUnit, - flags: MemFlags, - offset: Offset32) { - recipe_op1stdisp8(func, inst, sink, bits, in_reg0, in_reg1, flags, offset) -} - -fn recipe_mp1stdisp8(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - in_reg1: RegUnit, - _flags: MemFlags, - offset: Offset32) { - put_mp1(bits, sink); - modrm_disp8(in_reg1, in_reg0, sink); - let offset: i32 = offset.into(); - sink.put1(offset as u8); -} - -fn recipe_op1stdisp32(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - in_reg1: RegUnit, - _flags: MemFlags, - offset: Offset32) { - put_op1(bits, sink); - modrm_disp32(in_reg1, in_reg0, sink); - let offset: i32 = offset.into(); - sink.put4(offset as u32); -} - -fn recipe_op1stdisp32_abcd(func: &Function, - inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - in_reg1: RegUnit, - flags: MemFlags, - offset: Offset32) { - recipe_op1stdisp32(func, inst, sink, bits, in_reg0, in_reg1, flags, offset) -} - -fn recipe_mp1stdisp32(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - in_reg1: RegUnit, - _flags: MemFlags, - offset: Offset32) { - put_mp1(bits, sink); - modrm_disp32(in_reg1, in_reg0, sink); - let offset: i32 = offset.into(); - sink.put4(offset as u32); -} - -// Load recipes - -fn recipe_op1ld(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - _flags: MemFlags, - _offset: Offset32, - out_reg0: RegUnit) { - put_op1(bits, sink); - modrm_rm(in_reg0, out_reg0, sink); -} - -fn recipe_op1lddisp8(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - _flags: MemFlags, - offset: Offset32, - out_reg0: RegUnit) { - put_op1(bits, sink); - modrm_disp8(in_reg0, out_reg0, sink); - let offset: i32 = offset.into(); - sink.put1(offset as u8); -} - -fn recipe_op1lddisp32(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - _flags: MemFlags, - offset: Offset32, - out_reg0: RegUnit) { - put_op1(bits, sink); - modrm_disp32(in_reg0, out_reg0, sink); - let offset: i32 = offset.into(); - sink.put4(offset as u32); -} - -fn recipe_op2ld(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - _flags: MemFlags, - _offset: Offset32, - out_reg0: RegUnit) { - put_op2(bits, sink); - modrm_rm(in_reg0, out_reg0, sink); -} - -fn recipe_op2lddisp8(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - _flags: MemFlags, - offset: Offset32, - out_reg0: RegUnit) { - put_op2(bits, sink); - modrm_disp8(in_reg0, out_reg0, sink); - let offset: i32 = offset.into(); - sink.put1(offset as u8); -} - -fn recipe_op2lddisp32(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - _flags: MemFlags, - offset: Offset32, - out_reg0: RegUnit) { - put_op2(bits, sink); - modrm_disp32(in_reg0, out_reg0, sink); - let offset: i32 = offset.into(); - sink.put4(offset as u32); -} - -fn recipe_op1call_id(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - func_ref: ir::FuncRef) { - put_op1(bits, sink); - sink.reloc_func(RelocKind::PCRel4.into(), func_ref); - sink.put4(0); -} - -fn recipe_op1call_r(_func: &Function, - _inst: Inst, - sink: &mut CS, - bits: u16, - in_reg0: RegUnit, - _sig_ref: ir::SigRef) { - put_op1(bits, sink); - modrm_r_bits(in_reg0, bits, sink); -} - -fn recipe_op1ret(_func: &Function, _inst: Inst, sink: &mut CS, bits: u16) { - put_op1(bits, sink); -} From a4a8c83aab303d4c7da7d219bcdbeb5fd60c2bdc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 10 Jul 2017 16:16:22 -0700 Subject: [PATCH 0855/3084] Start adding Intel 64-bit encodings. Add a TailRecipe.rex() method which creates an encoding recipe with a REX prefix. Define I64 encodings with REX.W for i64 operations and with/without REX for i32 ops. Only test the with-REX encodings for now. We don't yet have an instruction shrinking pass that can select the non-REX encodings. --- cranelift/filetests/isa/intel/binary64.cton | 248 ++++++++++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 68 ++++-- lib/cretonne/meta/isa/intel/recipes.py | 50 +++- lib/cretonne/src/isa/intel/binemit.rs | 17 ++ 4 files changed, 365 insertions(+), 18 deletions(-) create mode 100644 cranelift/filetests/isa/intel/binary64.cton diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton new file mode 100644 index 0000000000..48e328b455 --- /dev/null +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -0,0 +1,248 @@ +; binary emission of 64-bit code. +test binemit +set is_64bit +isa intel + +; The binary encodings can be verified with the command: +; +; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/binary64.cton | llvm-mc -show-encoding -triple=x86_64 +; + +; Tests for i64 instructions. +function %I64() { + fn0 = function %foo() + sig0 = signature() + +ebb0: + + ; Integer Constants. + + ; asm: movq $0x01020304f1f2f3f4, %rcx + [-,%rcx] v1 = iconst.i64 0x0102_0304_f1f2_f3f4 ; bin: 48 b9 01020304f1f2f3f4 + ; asm: movq $0x11020304f1f2f3f4, %rsi + [-,%rsi] v2 = iconst.i64 0x1102_0304_f1f2_f3f4 ; bin: 48 be 11020304f1f2f3f4 + ; asm: movq $0x21020304f1f2f3f4, %r10 + [-,%r10] v3 = iconst.i64 0x2102_0304_f1f2_f3f4 ; bin: 49 ba 21020304f1f2f3f4 + ; asm: movl $0xff001122, %r8d # 32-bit zero-extended constant. + [-,%r8] v4 = iconst.i64 0xff00_1122 ; bin: 41 b8 ff001122 + ; asm: movq $0xffffffff88001122, %r14 # 32-bit sign-extended constant. + [-,%r14] v5 = iconst.i64 0xffff_ffff_8800_1122 ; bin: 49 c7 c6 88001122 + + ; Integer Register-Register Operations. + + ; asm: addq %rsi, %rcx + [-,%rcx] v10 = iadd v1, v2 ; bin: 48 01 f1 + ; asm: addq %r10, %rsi + [-,%rsi] v11 = iadd v2, v3 ; bin: 4c 01 d6 + ; asm: addq %rcx, %r10 + [-,%r10] v12 = iadd v3, v1 ; bin: 49 01 ca + + ; asm: subq %rsi, %rcx + [-,%rcx] v20 = isub v1, v2 ; bin: 48 29 f1 + ; asm: subq %r10, %rsi + [-,%rsi] v21 = isub v2, v3 ; bin: 4c 29 d6 + ; asm: subq %rcx, %r10 + [-,%r10] v22 = isub v3, v1 ; bin: 49 29 ca + + ; asm: andq %rsi, %rcx + [-,%rcx] v30 = band v1, v2 ; bin: 48 21 f1 + ; asm: andq %r10, %rsi + [-,%rsi] v31 = band v2, v3 ; bin: 4c 21 d6 + ; asm: andq %rcx, %r10 + [-,%r10] v32 = band v3, v1 ; bin: 49 21 ca + + ; asm: orq %rsi, %rcx + [-,%rcx] v40 = bor v1, v2 ; bin: 48 09 f1 + ; asm: orq %r10, %rsi + [-,%rsi] v41 = bor v2, v3 ; bin: 4c 09 d6 + ; asm: orq %rcx, %r10 + [-,%r10] v42 = bor v3, v1 ; bin: 49 09 ca + + ; asm: xorq %rsi, %rcx + [-,%rcx] v50 = bxor v1, v2 ; bin: 48 31 f1 + ; asm: xorq %r10, %rsi + [-,%rsi] v51 = bxor v2, v3 ; bin: 4c 31 d6 + ; asm: xorq %rcx, %r10 + [-,%r10] v52 = bxor v3, v1 ; bin: 49 31 ca + + ; asm: movq %rsi, %rcx + [-,%rcx] v60 = copy v2 ; bin: 48 89 f1 + ; asm: movq %r10, %rsi + [-,%rsi] v61 = copy v3 ; bin: 4c 89 d6 + ; asm: movq %rcx, %r10 + [-,%r10] v62 = copy v1 ; bin: 49 89 ca + + + ; Integer Register-Immediate Operations. + ; These 64-bit ops all use a 32-bit immediate that is sign-extended to 64 bits. + ; Some take 8-bit immediates that are sign-extended to 64 bits. + + ; asm: addq $-100000, %rcx + [-,%rcx] v70 = iadd_imm v1, -100000 ; bin: 48 81 c1 fffe7960 + ; asm: addq $100000, %rsi + [-,%rsi] v71 = iadd_imm v2, 100000 ; bin: 48 81 c6 000186a0 + ; asm: addq $0x7fffffff, %r10 + [-,%r10] v72 = iadd_imm v3, 0x7fff_ffff ; bin: 49 81 c2 7fffffff + ; asm: addq $100, %r8 + [-,%r8] v73 = iadd_imm v4, 100 ; bin: 49 83 c0 64 + ; asm: addq $-100, %r14 + [-,%r14] v74 = iadd_imm v5, -100 ; bin: 49 83 c6 9c + + ; asm: andq $-100000, %rcx + [-,%rcx] v80 = band_imm v1, -100000 ; bin: 48 81 e1 fffe7960 + ; asm: andq $100000, %rsi + [-,%rsi] v81 = band_imm v2, 100000 ; bin: 48 81 e6 000186a0 + ; asm: andq $0x7fffffff, %r10 + [-,%r10] v82 = band_imm v3, 0x7fff_ffff ; bin: 49 81 e2 7fffffff + ; asm: andq $100, %r8 + [-,%r8] v83 = band_imm v4, 100 ; bin: 49 83 e0 64 + ; asm: andq $-100, %r14 + [-,%r14] v84 = band_imm v5, -100 ; bin: 49 83 e6 9c + + ; asm: orq $-100000, %rcx + [-,%rcx] v90 = bor_imm v1, -100000 ; bin: 48 81 c9 fffe7960 + ; asm: orq $100000, %rsi + [-,%rsi] v91 = bor_imm v2, 100000 ; bin: 48 81 ce 000186a0 + ; asm: orq $0x7fffffff, %r10 + [-,%r10] v92 = bor_imm v3, 0x7fff_ffff ; bin: 49 81 ca 7fffffff + ; asm: orq $100, %r8 + [-,%r8] v93 = bor_imm v4, 100 ; bin: 49 83 c8 64 + ; asm: orq $-100, %r14 + [-,%r14] v94 = bor_imm v5, -100 ; bin: 49 83 ce 9c + ; asm: ret + + ; asm: xorq $-100000, %rcx + [-,%rcx] v100 = bxor_imm v1, -100000 ; bin: 48 81 f1 fffe7960 + ; asm: xorq $100000, %rsi + [-,%rsi] v101 = bxor_imm v2, 100000 ; bin: 48 81 f6 000186a0 + ; asm: xorq $0x7fffffff, %r10 + [-,%r10] v102 = bxor_imm v3, 0x7fff_ffff ; bin: 49 81 f2 7fffffff + ; asm: xorq $100, %r8 + [-,%r8] v103 = bxor_imm v4, 100 ; bin: 49 83 f0 64 + ; asm: xorq $-100, %r14 + [-,%r14] v104 = bxor_imm v5, -100 ; bin: 49 83 f6 9c + + return ; bin: c3 +} + +; Tests for i32 instructions in 64-bit mode. +; +; Note that many i32 instructions can be encoded both with and without a REX +; prefix if they only use the low 8 registers. Here, we are testing the REX +; encodings which are chosen by default. Switching to non-REX encodings should +; be done by an instruction shrinking pass. +function %I32() { + fn0 = function %foo() + sig0 = signature() + +ebb0: + + ; Integer Constants. + + ; asm: movl $0x01020304, %ecx + [-,%rcx] v1 = iconst.i32 0x0102_0304 ; bin: 40 b9 01020304 + ; asm: movl $0x11020304, %esi + [-,%rsi] v2 = iconst.i32 0x1102_0304 ; bin: 40 be 11020304 + ; asm: movl $0x21020304, %r10d + [-,%r10] v3 = iconst.i32 0x2102_0304 ; bin: 41 ba 21020304 + ; asm: movl $0xff001122, %r8d + [-,%r8] v4 = iconst.i32 0xff00_1122 ; bin: 41 b8 ff001122 + ; asm: movl $0x88001122, %r14d + [-,%r14] v5 = iconst.i32 0xffff_ffff_8800_1122 ; bin: 41 be 88001122 + + ; Integer Register-Register Operations. + + ; asm: addl %esi, %ecx + [-,%rcx] v10 = iadd v1, v2 ; bin: 40 01 f1 + ; asm: addl %r10d, %esi + [-,%rsi] v11 = iadd v2, v3 ; bin: 44 01 d6 + ; asm: addl %ecx, %r10d + [-,%r10] v12 = iadd v3, v1 ; bin: 41 01 ca + + ; asm: subl %esi, %ecx + [-,%rcx] v20 = isub v1, v2 ; bin: 40 29 f1 + ; asm: subl %r10d, %esi + [-,%rsi] v21 = isub v2, v3 ; bin: 44 29 d6 + ; asm: subl %ecx, %r10d + [-,%r10] v22 = isub v3, v1 ; bin: 41 29 ca + + ; asm: andl %esi, %ecx + [-,%rcx] v30 = band v1, v2 ; bin: 40 21 f1 + ; asm: andl %r10d, %esi + [-,%rsi] v31 = band v2, v3 ; bin: 44 21 d6 + ; asm: andl %ecx, %r10d + [-,%r10] v32 = band v3, v1 ; bin: 41 21 ca + + ; asm: orl %esi, %ecx + [-,%rcx] v40 = bor v1, v2 ; bin: 40 09 f1 + ; asm: orl %r10d, %esi + [-,%rsi] v41 = bor v2, v3 ; bin: 44 09 d6 + ; asm: orl %ecx, %r10d + [-,%r10] v42 = bor v3, v1 ; bin: 41 09 ca + + ; asm: xorl %esi, %ecx + [-,%rcx] v50 = bxor v1, v2 ; bin: 40 31 f1 + ; asm: xorl %r10d, %esi + [-,%rsi] v51 = bxor v2, v3 ; bin: 44 31 d6 + ; asm: xorl %ecx, %r10d + [-,%r10] v52 = bxor v3, v1 ; bin: 41 31 ca + + ; asm: movl %esi, %ecx + [-,%rcx] v60 = copy v2 ; bin: 40 89 f1 + ; asm: movl %r10d, %esi + [-,%rsi] v61 = copy v3 ; bin: 44 89 d6 + ; asm: movl %ecx, %r10d + [-,%r10] v62 = copy v1 ; bin: 41 89 ca + + + ; Integer Register-Immediate Operations. + ; These 64-bit ops all use a 32-bit immediate that is sign-extended to 64 bits. + ; Some take 8-bit immediates that are sign-extended to 64 bits. + + ; asm: addl $-100000, %ecx + [-,%rcx] v70 = iadd_imm v1, -100000 ; bin: 40 81 c1 fffe7960 + ; asm: addl $100000, %esi + [-,%rsi] v71 = iadd_imm v2, 100000 ; bin: 40 81 c6 000186a0 + ; asm: addl $0x7fffffff, %r10d + [-,%r10] v72 = iadd_imm v3, 0x7fff_ffff ; bin: 41 81 c2 7fffffff + ; asm: addl $100, %r8d + [-,%r8] v73 = iadd_imm v4, 100 ; bin: 41 83 c0 64 + ; asm: addl $-100, %r14d + [-,%r14] v74 = iadd_imm v5, -100 ; bin: 41 83 c6 9c + + ; asm: andl $-100000, %ecx + [-,%rcx] v80 = band_imm v1, -100000 ; bin: 40 81 e1 fffe7960 + ; asm: andl $100000, %esi + [-,%rsi] v81 = band_imm v2, 100000 ; bin: 40 81 e6 000186a0 + ; asm: andl $0x7fffffff, %r10d + [-,%r10] v82 = band_imm v3, 0x7fff_ffff ; bin: 41 81 e2 7fffffff + ; asm: andl $100, %r8d + [-,%r8] v83 = band_imm v4, 100 ; bin: 41 83 e0 64 + ; asm: andl $-100, %r14d + [-,%r14] v84 = band_imm v5, -100 ; bin: 41 83 e6 9c + + ; asm: orl $-100000, %ecx + [-,%rcx] v90 = bor_imm v1, -100000 ; bin: 40 81 c9 fffe7960 + ; asm: orl $100000, %esi + [-,%rsi] v91 = bor_imm v2, 100000 ; bin: 40 81 ce 000186a0 + ; asm: orl $0x7fffffff, %r10d + [-,%r10] v92 = bor_imm v3, 0x7fff_ffff ; bin: 41 81 ca 7fffffff + ; asm: orl $100, %r8d + [-,%r8] v93 = bor_imm v4, 100 ; bin: 41 83 c8 64 + ; asm: orl $-100, %r14d + [-,%r14] v94 = bor_imm v5, -100 ; bin: 41 83 ce 9c + ; asm: ret + + ; asm: xorl $-100000, %ecx + [-,%rcx] v100 = bxor_imm v1, -100000 ; bin: 40 81 f1 fffe7960 + ; asm: xorl $100000, %esi + [-,%rsi] v101 = bxor_imm v2, 100000 ; bin: 40 81 f6 000186a0 + ; asm: xorl $0x7fffffff, %r10d + [-,%r10] v102 = bxor_imm v3, 0x7fff_ffff ; bin: 41 81 f2 7fffffff + ; asm: xorl $100, %r8d + [-,%r8] v103 = bxor_imm v4, 100 ; bin: 41 83 f0 64 + ; asm: xorl $-100, %r14d + [-,%r14] v104 = bxor_imm v5, -100 ; bin: 41 83 f6 9c + + return ; bin: c3 +} diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 0b59b32da3..74dd5b5999 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -2,30 +2,65 @@ Intel Encodings. """ from __future__ import absolute_import +from cdsl.predicates import IsUnsignedInt from base import instructions as base -from .defs import I32 +from base.formats import UnaryImm +from .defs import I32, I64 from . import recipes as r -I32.enc(base.iadd.i32, *r.rr(0x01)) -I32.enc(base.isub.i32, *r.rr(0x29)) +for inst, opc in [ + (base.iadd, 0x01), + (base.isub, 0x29), + (base.band, 0x21), + (base.bor, 0x09), + (base.bxor, 0x31)]: + I32.enc(inst.i32, *r.rr(opc)) -I32.enc(base.band.i32, *r.rr(0x21)) -I32.enc(base.bor.i32, *r.rr(0x09)) -I32.enc(base.bxor.i32, *r.rr(0x31)) + I64.enc(inst.i64, *r.rr.rex(opc, w=1)) + I64.enc(inst.i32, *r.rr.rex(opc)) + # REX-less encoding must come after REX encoding so we don't use it by + # default. Otherwise reg-alloc would never use r8 and up. + I64.enc(inst.i32, *r.rr(opc)) I32.enc(base.copy.i32, *r.ur(0x89)) -# Immediate instructions with sign-extended 8-bit and 32-bit immediate. -for inst, rrr in [ - (base.iadd_imm.i32, 0), - (base.band_imm.i32, 4), - (base.bor_imm.i32, 1), - (base.bxor_imm.i32, 6)]: - I32.enc(inst, *r.rib(0x83, rrr=rrr)) - I32.enc(inst, *r.rid(0x81, rrr=rrr)) +I64.enc(base.copy.i64, *r.ur.rex(0x89, w=1)) +I64.enc(base.copy.i32, *r.ur.rex(0x89)) +I64.enc(base.copy.i32, *r.ur(0x89)) -# Immediate constant. -I32.enc(base.iconst.i32, *r.uid(0xb8)) +# Immediate instructions with sign-extended 8-bit and 32-bit immediate. +for inst, rrr in [ + (base.iadd_imm, 0), + (base.band_imm, 4), + (base.bor_imm, 1), + (base.bxor_imm, 6)]: + I32.enc(inst.i32, *r.rib(0x83, rrr=rrr)) + I32.enc(inst.i32, *r.rid(0x81, rrr=rrr)) + + I64.enc(inst.i64, *r.rib.rex(0x83, rrr=rrr, w=1)) + I64.enc(inst.i64, *r.rid.rex(0x81, rrr=rrr, w=1)) + I64.enc(inst.i32, *r.rib.rex(0x83, rrr=rrr)) + I64.enc(inst.i32, *r.rid.rex(0x81, rrr=rrr)) + I64.enc(inst.i32, *r.rib(0x83, rrr=rrr)) + I64.enc(inst.i32, *r.rid(0x81, rrr=rrr)) + +# TODO: band_imm.i64 with an unsigned 32-bit immediate can be encoded as +# band_imm.i32. Can even use the single-byte immediate for 0xffff_ffXX masks. + +# Immediate constants. +I32.enc(base.iconst.i32, *r.puid(0xb8)) + +I64.enc(base.iconst.i32, *r.puid.rex(0xb8)) +I64.enc(base.iconst.i32, *r.puid(0xb8)) +# The 32-bit immediate movl also zero-extends to 64 bits. +I64.enc(base.iconst.i64, *r.puid.rex(0xb8), + instp=IsUnsignedInt(UnaryImm.imm, 32)) +I64.enc(base.iconst.i64, *r.puid(0xb8), + instp=IsUnsignedInt(UnaryImm.imm, 32)) +# Sign-extended 32-bit immediate. +I64.enc(base.iconst.i64, *r.uid.rex(0xc7, rrr=0, w=1)) +# Finally, the 0xb8 opcode takes an 8-byte immediate with a REX.W prefix. +I64.enc(base.iconst.i64, *r.puiq.rex(0xb8, w=1)) # 32-bit shifts and rotates. # Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit @@ -73,3 +108,4 @@ I32.enc(base.sload8.i32.i32, *r.ldDisp32(0x0f, 0xbe)) I32.enc(base.call, *r.call_id(0xe8)) I32.enc(base.call_indirect.i32, *r.call_r(0xff, rrr=2)) I32.enc(base.x_return, *r.ret(0xc3)) +I64.enc(base.x_return, *r.ret(0xc3)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index ebeacc5968..2d1fc62d80 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -160,6 +160,33 @@ class TailRecipe: emit=replace_put_op(self.emit, name)) return (self.recipes[name], bits) + def rex(self, *ops, **kwargs): + # type: (*int, **int) -> Tuple[EncRecipe, int] + """ + Create a REX encoding recipe and encoding bits for the opcode bytes in + `ops`. + + The recipe will always generate a REX prefix, whether it is required or + not. For instructions that don't require a REX prefix, two encodings + should be added: One with REX and one without. + """ + rrr = kwargs.get('rrr', 0) + w = kwargs.get('w', 0) + name, bits = decode_ops(ops, rrr, w) + name = 'Rex' + name + if name not in self.recipes: + self.recipes[name] = EncRecipe( + name + self.name, + self.format, + 1 + len(ops) + self.size, + ins=self.ins, + outs=self.outs, + branch_range=self.branch_range, + instp=self.instp, + isap=self.isap, + emit=replace_put_op(self.emit, name)) + return (self.recipes[name], bits) + # XX /r rr = TailRecipe( @@ -208,11 +235,21 @@ rid = TailRecipe( sink.put4(imm as u32); ''') -# XX+rd id unary with 32-bit immediate. +# XX /n id with 32-bit immediate sign-extended. UnaryImm version. uid = TailRecipe( - 'uid', UnaryImm, size=4, ins=(), outs=GPR, + 'uid', UnaryImm, size=5, ins=(), outs=GPR, instp=IsSignedInt(UnaryImm.imm, 32), emit=''' + PUT_OP(bits, rex1(out_reg0), sink); + modrm_r_bits(out_reg0, bits, sink); + let imm: i64 = imm.into(); + sink.put4(imm as u32); + ''') + +# XX+rd id unary with 32-bit immediate. Note no recipe predicate. +puid = TailRecipe( + 'uid', UnaryImm, size=4, ins=(), outs=GPR, + emit=''' // The destination register is encoded in the low bits of the opcode. // No ModR/M. PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); @@ -220,6 +257,15 @@ uid = TailRecipe( sink.put4(imm as u32); ''') +# XX+rd iq unary with 64-bit immediate. +puiq = TailRecipe( + 'uiq', UnaryImm, size=8, ins=(), outs=GPR, + emit=''' + PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); + let imm: i64 = imm.into(); + sink.put8(imm as u64); + ''') + # # Store recipes. # diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index e2cb5396bc..9db0ee1234 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -44,6 +44,16 @@ fn rex2(rm: RegUnit, reg: RegUnit) -> u8 { BASE_REX | b | (r << 2) } +// Emit a REX prefix. +// +// The R, X, and B bits are computed from registers using the functions above. The W bit is +// extracted from `bits`. +fn rex_prefix(bits: u16, rex: u8, sink: &mut CS) { + debug_assert_eq!(rex & 0xf8, BASE_REX); + let w = ((bits >> 15) & 1) as u8; + sink.put1(rex | (w << 3)); +} + // Emit a single-byte opcode with no REX prefix. fn put_op1(bits: u16, rex: u8, sink: &mut CS) { debug_assert_eq!(bits & 0x8f00, 0, "Invalid encoding bits for Op1*"); @@ -51,6 +61,13 @@ fn put_op1(bits: u16, rex: u8, sink: &mut CS) { sink.put1(bits as u8); } +// Emit a single-byte opcode with REX prefix. +fn put_rexop1(bits: u16, rex: u8, sink: &mut CS) { + debug_assert_eq!(bits & 0x0f00, 0, "Invalid encoding bits for Op1*"); + rex_prefix(bits, rex, sink); + sink.put1(bits as u8); +} + // Emit two-byte opcode: 0F XX fn put_op2(bits: u16, rex: u8, sink: &mut CS) { debug_assert_eq!(bits & 0x8f00, 0x0400, "Invalid encoding bits for Op2*"); From 962c945a3c1ff375e48391e6ceeb786689fbe1bc Mon Sep 17 00:00:00 2001 From: Denis Merigoux Date: Tue, 11 Jul 2017 15:08:57 -0700 Subject: [PATCH 0856/3084] Cretonne IL frontend: ILBuilder (#97) * API and data structures proposal for the SSA construction module * Polished API and implemented trivial functions * API more explicit, Variable now struct parameter * Sample test written to see how the API could be used * Implemented local value numbering for SSABuilder * Implemented SSA within a single Ebb * Unfinished unoptimized implementation for recursive use and seal * Working global value numbering The SSABuilder now create ebb args and modifies jump instructions accordingly * Updated doc and improved branch argument modifying. Removed instructions::branch_arguments and instructions::branch_argument_mut * SSA building: bugfix, asserts and new test case Missing a key optimization to remove cycles of Phi * SSA Building: small changes after code review Created helper function for seal_block (which now contains sanity checks) * Optimization: removed useless phis (ebb arguments) Using pessimistic assumption that when using a non-def variable in an unsealed block we create an ebb argument which is removed when sealing if we detect it as useless Using aliases to avoid rewriting variables * Changed the semantics of remove_ebb_arg and turned it into a proper API method * Adapted ssa branch to changes in the DFG API * Abandonned SparseMaps for EntityMaps, added named structure for headr block data. * Created skeletton for a Cretonne IL builder frontend * Frontend IL builder: first draft of implementation with example of instruction methods * Working basic implementation of the frontend Missing handling of function arguments and return values * Interaction with function signature, sample test, more checks * Test with function verifier, seal and fill sanity check * Implemented python script to generate ILBuilder methods * Added support for jump tables and stack slot * Major API overhaul * No longer generating rust through Python but implements InstBuilder * No longer parametrized by user's blocks but use regular `Ebb` * Reuse of allocated memory via distinction between ILBuilder and FunctionBuilder * Integrate changes from StackSlot * Improved error message * Added support for jump arguments supplied by the user * Added an ebb_args proxy method needed * Adapted to Entity_ref splitted into a new module * Better error messages and fixed tests * Added method to change jump destination * We whould be able to add unreachable code * Added inst_result proxy to frontend * Import support * Added optimization for SSA construction: If multiple predecessors but agree on value don't create EBB argument * Move unsafe and not write-only funcs apart, improved doc * Added proxy function for append_ebb_arg * Support for unreachable code and better layout of the Ebbs * Fixed a bug yielding an infinite loop in SSA construction * SSA predecessors lookup code refactoring * Fixed bug in unreachable definition * New sanity check and display debug function * Fixed bug in verifier and added is_pristine ;ethod for frontend * Extended set of characters printable in function names To be able to print names of functions in test suite * Fixes and improvements of SSA construction after code review * Bugfixes for frontend code simplification * On-the-fly critical edge splitting in case of br_table with jump arguments * No more dangling undefined values, now attached as EBB args * Bugfix: only split corresponding edges on demand, not all br_table edges * Added signature retrieval method * Bugfix for critical edge splitting not sealing the ebbs it created * Proper handling of SSA side effects by the frontend * Code refactoring: moving frontend and SSA to new crate * Frontend: small changes and bugfixes after code review --- cranelift/Cargo.toml | 1 + cranelift/test-all.sh | 2 +- lib/cretonne/src/ir/builder.rs | 2 + lib/cretonne/src/ir/dfg.rs | 31 + lib/cretonne/src/ir/mod.rs | 1 + lib/cretonne/src/lib.rs | 2 +- lib/frontend/Cargo.toml | 15 + lib/frontend/src/frontend.rs | 682 +++++++++++++++++ lib/frontend/src/lib.rs | 152 ++++ lib/frontend/src/ssa.rs | 1276 ++++++++++++++++++++++++++++++++ 10 files changed, 2162 insertions(+), 2 deletions(-) create mode 100644 lib/frontend/Cargo.toml create mode 100644 lib/frontend/src/frontend.rs create mode 100644 lib/frontend/src/lib.rs create mode 100644 lib/frontend/src/ssa.rs diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 06eed706e9..6df15bd86c 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -15,6 +15,7 @@ path = "src/cton-util.rs" [dependencies] cretonne = { path = "lib/cretonne" } cretonne-reader = { path = "lib/reader" } +cretonne-frontend = { path ="lib/frontend" } filecheck = { path = "lib/filecheck" } docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 07d6fa2bdd..72b07f8856 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -40,7 +40,7 @@ if [ -n "$needcheck" ]; then touch $tsfile || echo no target directory fi -PKGS="cretonne cretonne-reader cretonne-tools filecheck" +PKGS="cretonne cretonne-reader cretonne-tools cretonne-frontend filecheck" cd "$topdir" for PKG in $PKGS do diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 256845bcfb..332ce43b03 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -23,6 +23,8 @@ pub trait InstBuilderBase<'f>: Sized { /// Get an immutable reference to the data flow graph that will hold the constructed /// instructions. fn data_flow_graph(&self) -> &DataFlowGraph; + /// Get a mutable reference to the data flow graph that will hold the constructed + /// instructions. fn data_flow_graph_mut(&mut self) -> &mut DataFlowGraph; /// Insert an instruction and return a reference to it, consuming the builder. diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 51ada16775..85e971a9fc 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -728,6 +728,37 @@ impl DataFlowGraph { num as usize } + /// Removes `val` from `ebb`'s arguments by a standard linear time list removal which preserves + /// ordering. Also updates the values' data. + pub fn remove_ebb_arg(&mut self, val: Value) { + let (ebb, num) = if let ValueData::Arg { num, ebb, .. } = self.values[val] { + (ebb, num) + } else { + panic!("{} must be an EBB argument", val); + }; + self.ebbs[ebb] + .args + .remove(num as usize, &mut self.value_lists); + for index in num..(self.ebb_args(ebb).len() as u16) { + match self.values[self.ebbs[ebb] + .args + .get(index as usize, &self.value_lists) + .unwrap()] { + ValueData::Arg { ref mut num, .. } => { + *num -= 1; + } + _ => { + panic!("{} must be an EBB argument", + self.ebbs[ebb] + .args + .get(index as usize, &self.value_lists) + .unwrap()) + } + } + } + } + + /// Append an existing argument value to `ebb`. /// /// The appended value can't already be attached to something else. diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index c79edfa4c6..ebe39a3fcb 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -31,6 +31,7 @@ pub use ir::function::Function; pub use ir::builder::InstBuilder; pub use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; pub use ir::memflags::MemFlags; +pub use ir::builder::InstBuilderBase; use binemit; use entity_map::EntityMap; diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 08c1c4354b..0b09a8f10d 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -24,6 +24,7 @@ pub mod flowgraph; pub mod ir; pub mod isa; pub mod loop_analysis; +pub mod packed_option; pub mod regalloc; pub mod result; pub mod settings; @@ -36,7 +37,6 @@ mod context; mod iterators; mod legalizer; mod licm; -mod packed_option; mod partition_slice; mod predicates; mod ref_slice; diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml new file mode 100644 index 0000000000..7ff44a1f6f --- /dev/null +++ b/lib/frontend/Cargo.toml @@ -0,0 +1,15 @@ +[package] +authors = ["The Cretonne Project Developers"] +name = "cretonne-frontend" +version = "0.0.0" +description = "Cretonne IL builder helper" +license = "Apache-2.0" +documentation = "https://cretonne.readthedocs.io/" +repository = "https://github.com/stoklund/cretonne" +publish = false + +[lib] +name = "cton_frontend" + +[dependencies] +cretonne = { path = "../cretonne" } diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs new file mode 100644 index 0000000000..7f9fb9edb6 --- /dev/null +++ b/lib/frontend/src/frontend.rs @@ -0,0 +1,682 @@ +//! A frontend for building Cretonne IL from other languages. +use cretonne::ir::{Ebb, Type, Value, Function, Inst, JumpTable, StackSlot, JumpTableData, + StackSlotData, DataFlowGraph, InstructionData, ExtFuncData, FuncRef, SigRef, + Signature, InstBuilderBase}; +use cretonne::ir::instructions::BranchInfo; +use cretonne::ir::function::DisplayFunction; +use cretonne::isa::TargetIsa; +use ssa::{SSABuilder, SideEffects, Block}; +use cretonne::entity_map::{EntityMap, PrimaryEntityData}; +use cretonne::entity_ref::EntityRef; +use std::hash::Hash; + +/// Permanent structure used for translating into Cretonne IL. +pub struct ILBuilder + where Variable: EntityRef + Hash + Default +{ + ssa: SSABuilder, + ebbs: EntityMap, + types: EntityMap, + function_args_values: Vec, +} + + +/// Temporary object used to build a Cretonne IL `Function`. +pub struct FunctionBuilder<'a, Variable: 'a> + where Variable: EntityRef + Hash + Default +{ + func: &'a mut Function, + builder: &'a mut ILBuilder, + position: Position, + pristine: bool, +} + +#[derive(Clone, Default)] +struct EbbData { + filled: bool, + pristine: bool, + user_arg_count: usize, +} + +impl PrimaryEntityData for EbbData {} + +struct Position { + ebb: Ebb, + basic_block: Block, +} + +impl ILBuilder + where Variable: EntityRef + Hash + Default +{ + /// Creates a ILBuilder structure. The structure is automatically cleared each time it is + /// passed to a [`FunctionBuilder`](struct.FunctionBuilder.html) for creation. + pub fn new() -> ILBuilder { + ILBuilder { + ssa: SSABuilder::new(), + ebbs: EntityMap::new(), + types: EntityMap::new(), + function_args_values: Vec::new(), + } + } + + fn clear(&mut self) { + self.ssa.clear(); + self.ebbs.clear(); + self.types.clear(); + self.function_args_values.clear(); + } +} + +/// Implementation of the [`InstBuilder`](../cretonne/ir/builder/trait.InstBuilder.html) that has +/// one convenience method per Cretonne IL instruction. +pub struct FuncInstBuilder<'short, 'long: 'short, Variable: 'long> + where Variable: EntityRef + Hash + Default +{ + builder: &'short mut FunctionBuilder<'long, Variable>, + ebb: Ebb, +} + +impl<'short, 'long, Variable> FuncInstBuilder<'short, 'long, Variable> + where Variable: EntityRef + Hash + Default +{ + fn new<'s, 'l>(builder: &'s mut FunctionBuilder<'l, Variable>, + ebb: Ebb) + -> FuncInstBuilder<'s, 'l, Variable> { + FuncInstBuilder { builder, ebb } + } +} + +impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long, Variable> + where Variable: EntityRef + Hash + Default +{ + fn data_flow_graph(&self) -> &DataFlowGraph { + &self.builder.func.dfg + } + + fn data_flow_graph_mut(&mut self) -> &mut DataFlowGraph { + &mut self.builder.func.dfg + } + + // This implementation is richer than `InsertBuilder` because we use the data of the + // instruction being inserted to add related info to the DFG and the SSA building system, + // and perform debug sanity checks. + fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'short mut DataFlowGraph) { + if data.opcode().is_return() { + self.builder + .check_return_args(data.arguments(&self.builder.func.dfg.value_lists)) + } + // We only insert the Ebb in the layout when an instruction is added to it + if self.builder.builder.ebbs[self.builder.position.ebb].pristine { + if !self.builder + .func + .layout + .is_ebb_inserted(self.builder.position.ebb) { + self.builder + .func + .layout + .append_ebb(self.builder.position.ebb); + } + self.builder.builder.ebbs[self.builder.position.ebb].pristine = false; + } else { + debug_assert!(!self.builder.builder.ebbs[self.builder.position.ebb].filled, + "you cannot add an instruction to a block already filled"); + } + let inst = self.builder.func.dfg.make_inst(data.clone()); + self.builder.func.dfg.make_inst_results(inst, ctrl_typevar); + self.builder.func.layout.append_inst(inst, self.ebb); + if data.opcode().is_branch() { + match data.branch_destination() { + Some(dest_ebb) => { + // If the user has supplied jump arguments we must adapt the arguments of + // the destination ebb + // TODO: find a way not to allocate a vector + let args_types: Vec = + match data.analyze_branch(&self.builder.func.dfg.value_lists) { + BranchInfo::SingleDest(_, args) => { + args.iter() + .map(|arg| self.builder.func.dfg.value_type(arg.clone())) + .collect() + } + _ => panic!("should not happen"), + }; + self.builder + .ebb_args_adjustement(dest_ebb, args_types.as_slice()); + self.builder.declare_successor(dest_ebb, inst); + } + None => { + // branch_destination() doesn't detect jump_tables + match data { + // If jump table we declare all entries successor + // TODO: not collect with vector? + InstructionData::BranchTable { table, .. } => { + for dest_ebb in self.builder + .func + .jump_tables + .get(table) + .expect("you are referencing an undeclared jump table") + .entries() + .map(|(_, ebb)| ebb) + .collect::>() { + self.builder.declare_successor(dest_ebb, inst) + } + } + // If not we do nothing + _ => {} + } + } + } + } + if data.opcode().is_terminator() { + self.builder.fill_current_block() + } else if data.opcode().is_branch() { + self.builder.move_to_next_basic_block() + } + (inst, &mut self.builder.func.dfg) + } +} + +/// This module allows you to create a function in Cretonne IL in a straightforward way, hiding +/// all the complexity of its internal representation. +/// +/// The module is parametrized by one type which is the representation of variables in your +/// origin language. It offers a way to conveniently append instruction to your program flow. +/// You are responsible to split your instruction flow into extended blocks (declared with +/// `create_ebb`) whose properties are: +/// +/// - branch and jump instructions can only point at the top of extended blocks; +/// - the last instruction of each block is a terminator instruction which has no natural sucessor, +/// and those instructions can only appear at the end of extended blocks. +/// +/// The parameters of Cretonne IL instructions are Cretonne IL values, which can only be created +/// as results of other Cretonne IL instructions. To be able to create variables redefined multiple +/// times in your program, use the `def_var` and `use_var` command, that will maintain the +/// correspondance between your variables and Cretonne IL SSA values. +/// +/// The first block for which you call `switch_to_block` will be assumed to be the beginning of +/// the function. +/// +/// At creation, a `FunctionBuilder` instance borrows an already allocated `Function` which it +/// modifies with the information stored in the mutable borrowed +/// [`ILBuilder`](struct.ILBuilder.html). The function passed in argument should be newly created +/// with [`Function::with_name_signature()`](../function/struct.Function.html), whereas the +/// `ILBuilder` can be kept as is between two function translations. +/// +/// # Errors +/// +/// The functions below will panic in debug mode whenever you try to modify the Cretonne IL +/// function in a way that violate the coherence of the code. For instance: switching to a new +/// `Ebb` when you haven't filled the current one with a terminator instruction, inserting a +/// return instruction with arguments that don't match the function's signature. +impl<'a, Variable> FunctionBuilder<'a, Variable> + where Variable: EntityRef + Hash + Default +{ + /// Creates a new FunctionBuilder structure that will operate on a `Function` using a + /// `IlBuilder`. + pub fn new(func: &'a mut Function, + builder: &'a mut ILBuilder) + -> FunctionBuilder<'a, Variable> { + builder.clear(); + FunctionBuilder { + func: func, + builder: builder, + position: Position { + ebb: Ebb::new(0), + basic_block: Block::new(0), + }, + pristine: true, + } + } + + /// Creates a new `Ebb` for the function and returns its reference. + pub fn create_ebb(&mut self) -> Ebb { + let ebb = self.func.dfg.make_ebb(); + self.builder.ssa.declare_ebb_header_block(ebb); + *self.builder.ebbs.ensure(ebb) = EbbData { + filled: false, + pristine: true, + user_arg_count: 0, + }; + ebb + } + + /// After the call to this function, new instructions will be inserted into the designated + /// block, in the order they are declared. You must declare the types of the Ebb arguments + /// you will use here. + /// + /// When inserting the terminator instruction (which doesn't have a falltrough to its immediate + /// successor), the block will be declared filled and it will not be possible to append + /// instructions to it. + pub fn switch_to_block(&mut self, ebb: Ebb, jump_args: &[Type]) -> &[Value] { + if self.pristine { + self.fill_function_args_values(ebb); + } + if !self.builder.ebbs[self.position.ebb].pristine { + // First we check that the previous block has been filled. + debug_assert!(self.is_unreachable() || self.builder.ebbs[self.position.ebb].filled, + "you have to fill your block before switching"); + } + // We cannot switch to a filled block + debug_assert!(!self.builder.ebbs[ebb].filled, + "you cannot switch to a block which is already filled"); + + let basic_block = self.builder.ssa.header_block(ebb); + // Then we change the cursor position. + self.position = Position { + ebb: ebb, + basic_block: basic_block, + }; + self.ebb_args_adjustement(ebb, jump_args); + self.func.dfg.ebb_args(ebb) + } + + /// Declares that all the predecessors of this block are known. + /// + /// Function to call with `ebb` as soon as the last branch instruction to `ebb` has been + /// created. Forgetting to call this method on every block will cause inconsistences in the + /// produced functions. + pub fn seal_block(&mut self, ebb: Ebb) { + let side_effects = self.builder + .ssa + .seal_ebb_header_block(ebb, + &mut self.func.dfg, + &mut self.func.layout, + &mut self.func.jump_tables); + self.handle_ssa_side_effects(side_effects); + } + + /// In order to use a variable in a `use_var`, you need to declare its type with this method. + pub fn declare_var(&mut self, var: Variable, ty: Type) { + *self.builder.types.ensure(var) = ty; + } + + /// Returns the Cretonne IL value corresponding to the utilization at the current program + /// position of a previously defined user variable. + pub fn use_var(&mut self, var: Variable) -> Value { + let ty = *self.builder + .types + .get(var) + .expect("this variable is used but its type has not been declared"); + let (val, side_effects) = self.builder + .ssa + .use_var(&mut self.func.dfg, + &mut self.func.layout, + &mut self.func.jump_tables, + var, + ty, + self.position.basic_block); + self.handle_ssa_side_effects(side_effects); + val + } + + /// Register a new definition of a user variable. Panics if the type of the value is not the + /// same as the type registered for the variable. + pub fn def_var(&mut self, var: Variable, val: Value) { + debug_assert!(self.func.dfg.value_type(val) == self.builder.types[var], + "the type of the value is not the type registered for the variable"); + self.builder + .ssa + .def_var(var, val, self.position.basic_block); + } + + /// Returns the value corresponding to the `i`-th argument of the function as defined by + /// the function signature. Panics if `i` is out of bounds or if called before the first call + /// to `switch_to_block`. + pub fn arg_value(&self, i: usize) -> Value { + debug_assert!(!self.pristine, "you have to call switch_to_block first."); + self.builder.function_args_values[i] + } + + /// Creates a jump table in the function, to be used by `br_table` instructions. + pub fn create_jump_table(&mut self) -> JumpTable { + self.func.jump_tables.push(JumpTableData::new()) + } + + /// Inserts an entry in a previously declared jump table. + pub fn insert_jump_table_entry(&mut self, jt: JumpTable, index: usize, ebb: Ebb) { + self.func.jump_tables[jt].set_entry(index, ebb); + } + + /// Creates a stack slot in the function, to be used by `stack_load`, `stack_store` and + /// `stack_addr` instructions. + pub fn create_stack_slot(&mut self, data: StackSlotData) -> StackSlot { + self.func.stack_slots.push(data) + } + + /// Adds a signature which can later be used to declare an external function import. + pub fn import_signature(&mut self, signature: Signature) -> SigRef { + self.func.dfg.signatures.push(signature) + } + + /// Declare an external function import. + pub fn import_function(&mut self, data: ExtFuncData) -> FuncRef { + self.func.dfg.ext_funcs.push(data) + } + + /// Returns an object with the [`InstBuilder`](../cretonne/ir/builder/trait.InstBuilder.html) + /// trait that allows to conveniently append an instruction to the current `Ebb` being built. + pub fn ins<'short>(&'short mut self) -> FuncInstBuilder<'short, 'a, Variable> { + let ebb = self.position.ebb; + FuncInstBuilder::new(self, ebb) + } +} + +/// All the functions documented in the previous block are write-only and help you build a valid +/// Cretonne IL functions via multiple debug asserts. However, you might need to improve the +/// performance of your translation perform more complex transformations to your Cretonne IL +/// function. The functions below help you inspect the function you're creating and modify it +/// in ways that can be unsafe if used incorrectly. +impl<'a, Variable> FunctionBuilder<'a, Variable> + where Variable: EntityRef + Hash + Default +{ + /// Retrieves all the arguments for an `Ebb` currently infered from the jump instructions + /// inserted that target it and the SSA construction. + pub fn ebb_args(&self, ebb: Ebb) -> &[Value] { + self.func.dfg.ebb_args(ebb) + } + + /// Retrieves the signature with reference `sigref` previously added with `import_signature`. + pub fn signature(&self, sigref: SigRef) -> Option<&Signature> { + self.func.dfg.signatures.get(sigref) + } + + /// Creates a argument for a specific `Ebb` by appending it to the list of already existing + /// arguments. + /// + /// **Note:** this function has to be called at the creation of the `Ebb` before adding + /// instructions to it, otherwise this could interfere with SSA construction. + pub fn append_ebb_arg(&mut self, ebb: Ebb, ty: Type) -> Value { + debug_assert!(self.builder.ebbs[ebb].pristine); + self.func.dfg.append_ebb_arg(ebb, ty) + } + + /// Returns the result values of an instruction. + pub fn inst_results(&self, inst: Inst) -> &[Value] { + self.func.dfg.inst_results(inst) + } + + /// Changes the destination of a jump instruction after creation. + /// + /// **Note:** You are responsible for maintaining the coherence with the arguments of + /// other jump instructions. + pub fn change_jump_destination(&mut self, inst: Inst, new_dest: Ebb) { + let old_dest = + self.func.dfg[inst] + .branch_destination_mut() + .expect("you want to change the jump destination of a non-jump instruction"); + let pred = self.builder.ssa.remove_ebb_predecessor(*old_dest, inst); + *old_dest = new_dest; + self.builder + .ssa + .declare_ebb_predecessor(new_dest, pred, inst); + } + + /// Returns `true` if and only if the current `Ebb` is sealed and has no predecessors declared. + /// + /// The entry block of a function is never unreachable. + pub fn is_unreachable(&self) -> bool { + let is_entry = match self.func.layout.entry_block() { + None => false, + Some(entry) => self.position.ebb == entry, + }; + (!is_entry && self.builder.ssa.is_sealed(self.position.ebb) && + self.builder.ssa.predecessors(self.position.ebb).is_empty()) + } + + /// Returns `true` if and only if no instructions have been added since the last call to + /// `switch_to_block`. + pub fn is_pristine(&self) -> bool { + self.builder.ebbs[self.position.ebb].pristine + } + + /// Returns `true` if and only if a terminator instruction has been inserted since the + /// last call to `switch_to_block`. + pub fn is_filled(&self) -> bool { + self.builder.ebbs[self.position.ebb].filled + } + + /// Returns a displayable object for the function as it is. + /// + /// Useful for debug purposes. Use it with `None` for standard printing. + pub fn display<'b, I: Into>>(&'b self, isa: I) -> DisplayFunction { + self.func.display(isa) + } +} + +impl<'a, Variable> Drop for FunctionBuilder<'a, Variable> + where Variable: EntityRef + Hash + Default +{ + /// When a `FunctionBuilder` goes out of scope, it means that the function is fully built. + /// We then proceed to check if all the `Ebb`s are filled and sealed + fn drop(&mut self) { + debug_assert!(self.builder + .ebbs + .keys() + .all(|ebb| { + self.builder.ebbs[ebb].pristine || + (self.builder.ssa.is_sealed(ebb) && + self.builder.ebbs[ebb].filled) + }), + "all blocks should be filled and sealed before dropping a FunctionBuilder") + } +} + +// Helper functions +impl<'a, Variable> FunctionBuilder<'a, Variable> + where Variable: EntityRef + Hash + Default +{ + fn move_to_next_basic_block(&mut self) { + self.position.basic_block = self.builder + .ssa + .declare_ebb_body_block(self.position.basic_block); + } + + fn fill_current_block(&mut self) { + self.builder.ebbs[self.position.ebb].filled = true; + } + + fn declare_successor(&mut self, dest_ebb: Ebb, jump_inst: Inst) { + self.builder + .ssa + .declare_ebb_predecessor(dest_ebb, self.position.basic_block, jump_inst); + } + + fn check_return_args(&self, args: &[Value]) { + debug_assert_eq!(args.len(), + self.func.signature.return_types.len(), + "the number of returned values doesn't match the function signature "); + for (i, arg) in args.iter().enumerate() { + let valty = self.func.dfg.value_type(*arg); + debug_assert_eq!(valty, + self.func.signature.return_types[i].value_type, + "the types of the values returned don't match the \ + function signature"); + } + } + + fn fill_function_args_values(&mut self, ebb: Ebb) { + debug_assert!(self.pristine); + for argtyp in self.func.signature.argument_types.iter() { + self.builder + .function_args_values + .push(self.func.dfg.append_ebb_arg(ebb, argtyp.value_type)); + } + self.pristine = false; + } + + + fn ebb_args_adjustement(&mut self, dest_ebb: Ebb, jump_args: &[Type]) { + let ty_to_append: Option> = + if self.builder.ssa.predecessors(dest_ebb).len() == 0 || + self.builder.ebbs[dest_ebb].pristine { + // This is the first jump instruction targeting this Ebb + // so the jump arguments supplied here are this Ebb' arguments + // However some of the arguments might already be there + // in the Ebb so we have to check they're consistent + let dest_ebb_args = self.func.dfg.ebb_args(dest_ebb); + debug_assert!(dest_ebb_args + .iter() + .zip(jump_args.iter().take(dest_ebb_args.len())) + .all(|(dest_arg, jump_arg)| { + *jump_arg == self.func.dfg.value_type(*dest_arg) + }), + "the jump argument supplied has not the \ + same type as the corresponding dest ebb argument"); + self.builder.ebbs[dest_ebb].user_arg_count = jump_args.len(); + Some(jump_args + .iter() + .skip(dest_ebb_args.len()) + .cloned() + .collect()) + } else { + let dest_ebb_args = self.func.dfg.ebb_args(dest_ebb); + // The Ebb already has predecessors + // We check that the arguments supplied match those supplied + // previously. + debug_assert!(jump_args.len() == self.builder.ebbs[dest_ebb].user_arg_count, + "the jump instruction doesn't have the same \ + number of arguments as its destination Ebb \ + ({} vs {}).", + jump_args.len(), + dest_ebb_args.len()); + debug_assert!(jump_args + .iter() + .zip(dest_ebb_args + .iter() + .take(self.builder.ebbs[dest_ebb].user_arg_count) + ) + .all(|(jump_arg, dest_arg)| { + *jump_arg == self.func.dfg.value_type(*dest_arg) + }), + "the jump argument supplied has not the \ + same type as the corresponding dest ebb argument"); + None + }; + if let Some(ty_args) = ty_to_append { + for ty in ty_args { + self.func.dfg.append_ebb_arg(dest_ebb, ty); + } + } + } + + fn handle_ssa_side_effects(&mut self, side_effects: SideEffects) { + for split_ebb in side_effects.split_ebbs_created { + self.builder.ebbs.ensure(split_ebb).filled = true + } + for modified_ebb in side_effects.instructions_added_to_ebbs { + self.builder.ebbs[modified_ebb].pristine = false + } + } +} + +#[cfg(test)] +mod tests { + + use cretonne::entity_ref::EntityRef; + use cretonne::ir::{FunctionName, Function, Signature, ArgumentType, InstBuilder}; + use cretonne::ir::types::*; + use frontend::{ILBuilder, FunctionBuilder}; + use cretonne::verifier::verify_function; + + use std::u32; + + // An opaque reference to variable. + #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] + pub struct Variable(u32); + impl EntityRef for Variable { + fn new(index: usize) -> Self { + assert!(index < (u32::MAX as usize)); + Variable(index as u32) + } + + fn index(self) -> usize { + self.0 as usize + } + } + impl Default for Variable { + fn default() -> Variable { + Variable(u32::MAX) + } + } + + #[test] + fn sample_function() { + let mut sig = Signature::new(); + sig.return_types.push(ArgumentType::new(I32)); + sig.argument_types.push(ArgumentType::new(I32)); + + let mut il_builder = ILBuilder::::new(); + let mut func = Function::with_name_signature(FunctionName::new("sample_function"), sig); + { + let mut builder = FunctionBuilder::::new(&mut func, &mut il_builder); + + let block0 = builder.create_ebb(); + let block1 = builder.create_ebb(); + let block2 = builder.create_ebb(); + let x = Variable(0); + let y = Variable(1); + let z = Variable(2); + builder.declare_var(x, I32); + builder.declare_var(y, I32); + builder.declare_var(z, I32); + + builder.switch_to_block(block0, &[]); + builder.seal_block(block0); + { + let tmp = builder.arg_value(0); + builder.def_var(x, tmp); + } + { + let tmp = builder.ins().iconst(I32, 2); + builder.def_var(y, tmp); + } + { + let arg1 = builder.use_var(x); + let arg2 = builder.use_var(y); + let tmp = builder.ins().iadd(arg1, arg2); + builder.def_var(z, tmp); + } + builder.ins().jump(block1, &[]); + + builder.switch_to_block(block1, &[]); + { + let arg1 = builder.use_var(y); + let arg2 = builder.use_var(z); + let tmp = builder.ins().iadd(arg1, arg2); + builder.def_var(z, tmp); + } + { + let arg = builder.use_var(y); + builder.ins().brnz(arg, block2, &[]); + } + { + let arg1 = builder.use_var(z); + let arg2 = builder.use_var(x); + let tmp = builder.ins().isub(arg1, arg2); + builder.def_var(z, tmp); + } + { + let arg = builder.use_var(y); + builder.ins().return_(&[arg]); + } + + builder.switch_to_block(block2, &[]); + builder.seal_block(block2); + + { + let arg1 = builder.use_var(y); + let arg2 = builder.use_var(x); + let tmp = builder.ins().isub(arg1, arg2); + builder.def_var(y, tmp); + } + builder.ins().jump(block1, &[]); + builder.seal_block(block1); + } + + let res = verify_function(&func, None); + // println!("{}", func.display(None)); + match res { + Ok(_) => {} + Err(err) => panic!("{}{}", func.display(None), err), + } + } +} diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs new file mode 100644 index 0000000000..2cbbc30459 --- /dev/null +++ b/lib/frontend/src/lib.rs @@ -0,0 +1,152 @@ +//! Cretonne IL builder library. +//! +//! Provides a straightforward way to create a Cretonne IL function and fill it with instructions +//! translated from another language. Contains a SSA construction module that lets you translate +//! your non-SSA variables into SSA Cretonne IL values via `use_var` and `def_var` calls. +//! +//! To get started, create an [`IlBuilder`](struct.ILBuilder.html) and pass it as an argument +//! to a [`FunctionBuilder`](struct.FunctionBuilder.html). +//! +//! # Example +//! +//! Here is a pseudo-program we want to transform into Cretonne IL: +//! +//! ```cton +//! function(x) { +//! x, y, z : i32 +//! block0: +//! y = 2; +//! z = x + y; +//! jump block1 +//! block1: +//! z = z + y; +//! brnz y, block2; +//! z = z - x; +//! return y +//! block2: +//! y = y - x +//! jump block1 +//! } +//! ``` +//! +//! Here is how you build the corresponding Cretonne IL function using `ILBuilder`: +//! +//! ```rust +//! extern crate cretonne; +//! extern crate cton_frontend; +//! +//! use cretonne::entity_ref::EntityRef; +//! use cretonne::ir::{FunctionName, Function, Signature, ArgumentType, InstBuilder}; +//! use cretonne::ir::types::*; +//! use cton_frontend::{ILBuilder, FunctionBuilder}; +//! use cretonne::verifier::verify_function; +//! use std::u32; +//! +//! // An opaque reference to variable. +//! #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +//! pub struct Variable(u32); +//! impl EntityRef for Variable { +//! fn new(index: usize) -> Self { +//! assert!(index < (u32::MAX as usize)); +//! Variable(index as u32) +//! } +//! +//! fn index(self) -> usize { +//! self.0 as usize +//! } +//! } +//! impl Default for Variable { +//! fn default() -> Variable { +//! Variable(u32::MAX) +//! } +//! } +//! +//! fn main() { +//! let mut sig = Signature::new(); +//! sig.return_types.push(ArgumentType::new(I32)); +//! sig.argument_types.push(ArgumentType::new(I32)); +//! let mut il_builder = ILBuilder::::new(); +//! let mut func = Function::with_name_signature(FunctionName::new("sample_function"), sig); +//! { +//! let mut builder = FunctionBuilder::::new(&mut func, &mut il_builder); +//! +//! let block0 = builder.create_ebb(); +//! let block1 = builder.create_ebb(); +//! let block2 = builder.create_ebb(); +//! let x = Variable(0); +//! let y = Variable(1); +//! let z = Variable(2); +//! builder.declare_var(x, I32); +//! builder.declare_var(y, I32); +//! builder.declare_var(z, I32); +//! +//! builder.switch_to_block(block0, &[]); +//! builder.seal_block(block0); +//! { +//! let tmp = builder.arg_value(0); +//! builder.def_var(x, tmp); +//! } +//! { +//! let tmp = builder.ins().iconst(I32, 2); +//! builder.def_var(y, tmp); +//! } +//! { +//! let arg1 = builder.use_var(x); +//! let arg2 = builder.use_var(y); +//! let tmp = builder.ins().iadd(arg1, arg2); +//! builder.def_var(z, tmp); +//! } +//! builder.ins().jump(block1, &[]); +//! +//! builder.switch_to_block(block1, &[]); +//! { +//! let arg1 = builder.use_var(y); +//! let arg2 = builder.use_var(z); +//! let tmp = builder.ins().iadd(arg1, arg2); +//! builder.def_var(z, tmp); +//! } +//! { +//! let arg = builder.use_var(y); +//! builder.ins().brnz(arg, block2, &[]); +//! } +//! { +//! let arg1 = builder.use_var(z); +//! let arg2 = builder.use_var(x); +//! let tmp = builder.ins().isub(arg1, arg2); +//! builder.def_var(z, tmp); +//! } +//! { +//! let arg = builder.use_var(y); +//! builder.ins().return_(&[arg]); +//! } +//! +//! builder.switch_to_block(block2, &[]); +//! builder.seal_block(block2); +//! +//! { +//! let arg1 = builder.use_var(y); +//! let arg2 = builder.use_var(x); +//! let tmp = builder.ins().isub(arg1, arg2); +//! builder.def_var(y, tmp); +//! } +//! builder.ins().jump(block1, &[]); +//! builder.seal_block(block1); +//! } +//! +//! let res = verify_function(&func, None); +//! println!("{}", func.display(None)); +//! match res { +//! Ok(_) => {} +//! Err(err) => panic!("{}", err), +//! } +//! } +//! ``` + +#![deny(missing_docs)] + +extern crate cretonne; + +pub use frontend::{ILBuilder, FunctionBuilder}; + +mod frontend; +mod ssa; diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs new file mode 100644 index 0000000000..46b44d5251 --- /dev/null +++ b/lib/frontend/src/ssa.rs @@ -0,0 +1,1276 @@ +//! A SSA-building API that handles incomplete CFGs. +//! +//! The algorithm is based upon Braun M., Buchwald S., Hack S., Leißa R., Mallon C., +//! Zwinkau A. (2013) Simple and Efficient Construction of Static Single Assignment Form. +//! In: Jhala R., De Bosschere K. (eds) Compiler Construction. CC 2013. +//! Lecture Notes in Computer Science, vol 7791. Springer, Berlin, Heidelberg + +use cretonne::ir::{Ebb, Value, Inst, Type, DataFlowGraph, JumpTables, Layout, Cursor, InstBuilder}; +use cretonne::ir::instructions::BranchInfo; +use std::hash::Hash; +use cretonne::entity_map::{EntityMap, PrimaryEntityData}; +use cretonne::entity_ref::EntityRef; +use cretonne::packed_option::PackedOption; +use cretonne::packed_option::ReservedValue; +use std::u32; +use cretonne::ir::types::{F32, F64}; +use cretonne::ir::immediates::{Ieee32, Ieee64}; +use std::collections::HashMap; + +/// Structure containing the data relevant the construction of SSA for a given function. +/// +/// The parameter struct `Variable` corresponds to the way variables are represented in the +/// non-SSA language you're translating from. +/// +/// The SSA building relies on information about the variables used and defined, as well as +/// their position relative to basic blocks which are stricter than extended basic blocks since +/// they don't allow branching in the middle of them. +/// +/// This SSA building module allows you to def and use variables on the fly while you are +/// constructing the CFG, no need for a separate SSA pass after the CFG is completed. +/// +/// A basic block is said _filled_ if all the instruction that it contains have been translated, +/// and it is said _sealed_ if all of its predecessors have been declared. Only filled predecessors +/// can be declared. +pub struct SSABuilder + where Variable: EntityRef + Default +{ + // Records for every variable and for every revelant block, the last definition of + // the variable in the block. + variables: EntityMap>, + // Records the position of the basic blocks and the list of values used but not defined in the + // block. + blocks: EntityMap>, + // Records the basic blocks at the beginning of the `Ebb`s. + ebb_headers: EntityMap>, +} + +/// Side effects of a `use_var` or a `seal_ebb_header_block` method call. +pub struct SideEffects { + /// When we want to append jump arguments to a `br_table` instruction, the critical edge is + /// splitted and the newly created `Ebb`s are signaled here. + pub split_ebbs_created: Vec, + /// When a variable is used but has never been defined before (this happens in the case of + /// unreachable code), a placeholder `iconst` or `fconst` value is added to the right `Ebb`. + /// This field signals if it is the case and return the `Ebb` to which the initialization has + /// been added. + pub instructions_added_to_ebbs: Vec, +} + +// Describes the current position of a basic block in the control flow graph. +enum BlockData { + // A block at the top of an `Ebb`. + EbbHeader(EbbHeaderBlockData), + // A block inside an `Ebb` with an unique other block as its predecessor. + // The block is implicitely sealed at creation. + EbbBody { predecessor: Block }, +} +impl PrimaryEntityData for BlockData {} + +impl BlockData { + fn add_predecessor(&mut self, pred: Block, inst: Inst) { + match self { + &mut BlockData::EbbBody { .. } => panic!("you can't add a predecessor to a body block"), + &mut BlockData::EbbHeader(ref mut data) => { + data.predecessors.insert(pred, inst); + () + } + } + } + fn remove_predecessor(&mut self, inst: Inst) -> Block { + match self { + &mut BlockData::EbbBody { .. } => panic!("should not happen"), + &mut BlockData::EbbHeader(ref mut data) => { + // This a linear complexity operation but the number of predecessors is low + // in all non-pathological cases + let pred: Block = match data.predecessors + .iter() + .find(|&(_, &jump_inst)| jump_inst == inst) { + None => panic!("the predecessor you are trying to remove is not declared"), + Some((&b, _)) => b.clone(), + }; + data.predecessors.remove(&pred); + pred + } + } + } +} + +struct EbbHeaderBlockData { + // The predecessors of the Ebb header block, with the block and branch instruction. + predecessors: HashMap, + // A ebb header block is sealed if all of its predecessors have been declared. + sealed: bool, + // The ebb which this block is part of. + ebb: Ebb, + // List of current Ebb arguments for which a earlier def has not been found yet. + undef_variables: Vec<(Variable, Value)>, +} + +/// A opaque reference to a basic block. +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +pub struct Block(u32); +impl EntityRef for Block { + fn new(index: usize) -> Self { + assert!(index < (u32::MAX as usize)); + Block(index as u32) + } + + fn index(self) -> usize { + self.0 as usize + } +} + +impl ReservedValue for Block { + fn reserved_value() -> Block { + Block(u32::MAX) + } +} + +impl SSABuilder + where Variable: EntityRef + Default +{ + /// Allocate a new blank SSA builder struct. Use the API function to interact with the struct. + pub fn new() -> SSABuilder { + SSABuilder { + variables: EntityMap::new(), + blocks: EntityMap::new(), + ebb_headers: EntityMap::new(), + } + } + + /// Clears a `SSABuilder` from all its data, letting it in a pristine state without + /// deallocating memory. + pub fn clear(&mut self) { + self.variables.clear(); + self.blocks.clear(); + self.ebb_headers.clear(); + } +} + +// Small enum used for clarity in some functions. +#[derive(Debug)] +enum ZeroOneOrMore { + Zero(), + One(T), + More(), +} + +/// TODO: use entity list instead of vec +#[derive(Debug)] +enum UseVarCases { + Unsealed(Value), + SealedOnePredecessor(Block), + SealedMultiplePredecessors(Vec<(Block, Inst)>, Value, Ebb), +} + +/// The following methods are the API of the SSA builder. Here is how it should be used when +/// translating to Cretonne IL: +/// +/// - for each sequence of contiguous instructions (with no branches), create a corresponding +/// basic block with `declare_ebb_body_block` or `declare_ebb_header_block` depending on the +/// position of the basic block; +/// +/// - while traversing a basic block and translating instruction, use `def_var` and `use_var` +/// to record definitions and uses of variables, these methods will give you the corresponding +/// SSA values; +/// +/// - when all the instructions in a basic block have translated, the block is said _filled_ and +/// only then you can add it as a predecessor to other blocks with `declare_ebb_predecessor`; +/// +/// - when you have constructed all the predecessor to a basic block at the beginning of an `Ebb`, +/// call `seal_ebb_header_block` on it with the `Function` that you are building. +/// +/// This API will give you the correct SSA values to use as arguments of your instructions, +/// as well as modify the jump instruction and `Ebb` headers arguments to account for the SSA +/// Phi functions. +/// +impl SSABuilder + where Variable: EntityRef + Hash + Default +{ + /// Declares a new definition of a variable in a given basic block. + /// The SSA value is passed as an argument because it should be created with + /// `ir::DataFlowGraph::append_result`. + pub fn def_var(&mut self, var: Variable, val: Value, block: Block) { + self.variables.ensure(var).insert(block, val); + } + + /// Declares a use of a variable in a given basic block. Returns the SSA value corresponding + /// to the current SSA definition of this variable and a list of newly created Ebbs that + /// are the results of critical edge splitting for `br_table` with arguments. + /// + /// If the variable has never been defined in this blocks or recursively in its predecessors, + /// this method will silently create an initializer with `iconst` or `fconst`. You are + /// responsible for making sure that you initialize your variables. + pub fn use_var(&mut self, + dfg: &mut DataFlowGraph, + layout: &mut Layout, + jts: &mut JumpTables, + var: Variable, + ty: Type, + block: Block) + -> (Value, SideEffects) { + // First we lookup for the current definition of the variable in this block + if let Some(var_defs) = self.variables.get(var) { + if let Some(val) = var_defs.get(&block) { + return (*val, + SideEffects { + split_ebbs_created: Vec::new(), + instructions_added_to_ebbs: Vec::new(), + }); + } + }; + // At this point if we haven't returned it means that we have to search in the + // predecessors. + let case = match self.blocks[block] { + BlockData::EbbHeader(ref mut data) => { + // The block has multiple predecessors so we append an Ebb argument that + // will serve as a value. + if data.sealed { + if data.predecessors.len() == 1 { + // Only one predecessor, straightforward case + UseVarCases::SealedOnePredecessor(*data.predecessors.keys().next().unwrap()) + } else { + let val = dfg.append_ebb_arg(data.ebb, ty); + let preds = data.predecessors + .iter() + .map(|(&pred, &inst)| (pred, inst)) + .collect(); + UseVarCases::SealedMultiplePredecessors(preds, val, data.ebb) + } + } else { + let val = dfg.append_ebb_arg(data.ebb, ty); + data.undef_variables.push((var, val)); + UseVarCases::Unsealed(val) + } + } + BlockData::EbbBody { predecessor: pred, .. } => UseVarCases::SealedOnePredecessor(pred), + }; + // TODO: avoid recursion for the calls to use_var and predecessors_lookup. + match case { + // The block has a single predecessor or multiple predecessor with + // the same value, we look into it. + UseVarCases::SealedOnePredecessor(pred) => { + let (val, mids) = self.use_var(dfg, layout, jts, var, ty, pred); + self.def_var(var, val, block); + return (val, mids); + } + // The block has multiple predecessors, we register the ebb argument as the current + // definition for the variable. + UseVarCases::Unsealed(val) => { + self.def_var(var, val, block); + return (val, + SideEffects { + split_ebbs_created: Vec::new(), + instructions_added_to_ebbs: Vec::new(), + }); + } + UseVarCases::SealedMultiplePredecessors(preds, val, ebb) => { + // If multiple predecessor we look up a use_var in each of them: + // if they all yield the same value no need for an Ebb argument + self.def_var(var, val, block); + return self.predecessors_lookup(dfg, layout, jts, val, var, ebb, &preds); + } + }; + + } + + /// Declares a new basic block belonging to the body of a certain `Ebb` and having `pred` + /// as a predecessor. `pred` is the only predecessor of the block and the block is sealed + /// at creation. + /// + /// To declare a `Ebb` header block, see `declare_ebb_header_block`. + pub fn declare_ebb_body_block(&mut self, pred: Block) -> Block { + self.blocks.push(BlockData::EbbBody { predecessor: pred }) + } + + /// Declares a new basic block at the beginning of an `Ebb`. No predecessors are declared + /// here and the block is not sealed. + /// Predecessors have to be added with `declare_ebb_predecessor`. + pub fn declare_ebb_header_block(&mut self, ebb: Ebb) -> Block { + let block = self.blocks + .push(BlockData::EbbHeader(EbbHeaderBlockData { + predecessors: HashMap::new(), + sealed: false, + ebb: ebb, + undef_variables: Vec::new(), + })); + *self.ebb_headers.ensure(ebb) = block.into(); + block + } + /// Gets the header block corresponding to an Ebb, panics if the Ebb or the header block + /// isn't declared. + pub fn header_block(&self, ebb: Ebb) -> Block { + match self.ebb_headers.get(ebb) { + Some(&header) => { + match header.expand() { + Some(header) => header, + None => panic!("the header block has not been defined"), + } + } + None => panic!("the ebb has not been declared"), + } + } + + /// Declares a new predecessor for an `Ebb` header block and record the branch instruction + /// of the predecessor that leads to it. + /// + /// Note that the predecessor is a `Block` and not an `Ebb`. This `Block` must be filled + /// before added as predecessor. Note that you must provide no jump arguments to the branch + /// instruction when you create it since `SSABuilder` will fill them for you. + pub fn declare_ebb_predecessor(&mut self, ebb: Ebb, pred: Block, inst: Inst) { + let header_block = match self.blocks[self.header_block(ebb)] { + BlockData::EbbBody { .. } => panic!("you can't add predecessors to an Ebb body block"), + BlockData::EbbHeader(ref data) => { + assert!(!data.sealed); + self.header_block(ebb) + } + }; + self.blocks[header_block].add_predecessor(pred, inst) + } + + /// Remove a previously declared Ebb predecessor by giving a reference to the jump + /// instruction. Returns the basic block containing the instruction. + /// + /// Note: use only when you know what you are doing, this might break the SSA bbuilding problem + pub fn remove_ebb_predecessor(&mut self, ebb: Ebb, inst: Inst) -> Block { + let header_block = match self.blocks[self.header_block(ebb)] { + BlockData::EbbBody { .. } => panic!("you can't add predecessors to an Ebb body block"), + BlockData::EbbHeader(ref data) => { + assert!(!data.sealed); + self.header_block(ebb) + } + }; + self.blocks[header_block].remove_predecessor(inst) + } + + /// Completes the global value numbering for an `Ebb`, all of its predecessors having been + /// already sealed. + /// + /// This method modifies the function's `Layout` by adding arguments to the `Ebb`s to + /// take into account the Phi function placed by the SSA algorithm. + pub fn seal_ebb_header_block(&mut self, + ebb: Ebb, + dfg: &mut DataFlowGraph, + layout: &mut Layout, + jts: &mut JumpTables) + -> SideEffects { + let block = self.header_block(ebb); + + // Sanity check + match self.blocks[block] { + BlockData::EbbBody { .. } => panic!("you can't seal an Ebb body block"), + BlockData::EbbHeader(ref data) => { + assert!(!data.sealed); + } + } + + // Recurse over the predecessors to find good definitions. + let side_effects = self.resolve_undef_vars(block, dfg, layout, jts); + + // Then we mark the block as sealed. + match self.blocks[block] { + BlockData::EbbBody { .. } => panic!("this should not happen"), + BlockData::EbbHeader(ref mut data) => data.sealed = true, + }; + side_effects + } + + // For each undef_var in an Ebb header block, lookup in the predecessors to append the right + // jump argument to the branch instruction. + // Panics if called with a non-header block. + // Returns the list of newly created ebbs for critical edge splitting. + fn resolve_undef_vars(&mut self, + block: Block, + dfg: &mut DataFlowGraph, + layout: &mut Layout, + jts: &mut JumpTables) + -> SideEffects { + // TODO: find a way to not allocate vectors + let (predecessors, undef_vars, ebb): (Vec<(Block, Inst)>, + Vec<(Variable, Value)>, + Ebb) = match self.blocks[block] { + BlockData::EbbBody { .. } => panic!("this should not happen"), + BlockData::EbbHeader(ref mut data) => { + (data.predecessors.iter().map(|(&x, &y)| (x, y)).collect(), + data.undef_variables.clone(), + data.ebb) + } + }; + + let mut side_effects = SideEffects { + split_ebbs_created: Vec::new(), + instructions_added_to_ebbs: Vec::new(), + }; + // For each undef var we look up values in the predecessors and create an Ebb argument + // only if necessary. + for &(var, val) in undef_vars.iter() { + let (_, mut local_side_effects) = + self.predecessors_lookup(dfg, layout, jts, val, var, ebb, &predecessors); + side_effects + .split_ebbs_created + .append(&mut local_side_effects.split_ebbs_created); + side_effects + .instructions_added_to_ebbs + .append(&mut local_side_effects.instructions_added_to_ebbs); + } + + // Then we clear the undef_vars and mark the block as sealed. + match self.blocks[block] { + BlockData::EbbBody { .. } => panic!("this should not happen"), + BlockData::EbbHeader(ref mut data) => { + data.undef_variables.clear(); + } + }; + side_effects + } + + /// Look up in the predecessors of an Ebb the def for a value an decides wether or not + /// to keep the eeb arg, and act accordingly. Returns the chosen value and optionnaly a + /// list of Ebb that are the middle of newly created critical edges splits. + fn predecessors_lookup(&mut self, + dfg: &mut DataFlowGraph, + layout: &mut Layout, + jts: &mut JumpTables, + temp_arg_val: Value, + temp_arg_var: Variable, + dest_ebb: Ebb, + preds: &Vec<(Block, Inst)>) + -> (Value, SideEffects) { + let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero(); + // TODO: find a way not not allocate a vector + let mut jump_args_to_append: Vec<(Block, Inst, Value)> = Vec::new(); + let ty = dfg.value_type(temp_arg_val); + let mut side_effects = SideEffects { + split_ebbs_created: Vec::new(), + instructions_added_to_ebbs: Vec::new(), + }; + for &(pred, last_inst) in preds.iter() { + // For undef value and each predecessor we query what is the local SSA value + // corresponding to var and we put it as an argument of the branch instruction. + let (pred_val, mut local_side_effects) = + self.use_var(dfg, layout, jts, temp_arg_var, ty, pred); + pred_values = match pred_values { + ZeroOneOrMore::Zero() => { + if pred_val == temp_arg_val { + ZeroOneOrMore::Zero() + } else { + ZeroOneOrMore::One(pred_val) + } + } + ZeroOneOrMore::One(old_val) => { + if pred_val == temp_arg_val || pred_val == old_val { + ZeroOneOrMore::One(old_val) + } else { + ZeroOneOrMore::More() + } + } + ZeroOneOrMore::More() => ZeroOneOrMore::More(), + }; + jump_args_to_append.push((pred, last_inst, pred_val)); + side_effects + .split_ebbs_created + .append(&mut local_side_effects.split_ebbs_created); + side_effects + .instructions_added_to_ebbs + .append(&mut local_side_effects.instructions_added_to_ebbs); + } + match pred_values { + ZeroOneOrMore::Zero() => { + // The variable is used but never defined before. This is an irregularity in the + // code, but rather than throwing an error we silently initialize the variable to + // 0. This will have no effect since this situation happens in unreachable code. + if !layout.is_ebb_inserted(dest_ebb) { + layout.append_ebb(dest_ebb) + }; + let mut cur = Cursor::new(layout); + cur.goto_top(dest_ebb); + cur.next_inst(); + let ty = dfg.value_type(temp_arg_val); + let val = if ty.is_int() { + dfg.ins(&mut cur).iconst(ty, 0) + } else if ty == F32 { + dfg.ins(&mut cur).f32const(Ieee32::new(0.0)) + } else if ty == F64 { + dfg.ins(&mut cur).f64const(Ieee64::new(0.0)) + } else { + panic!("value used but never declared and initialization not supported") + }; + side_effects.instructions_added_to_ebbs.push(dest_ebb); + (val, side_effects) + } + ZeroOneOrMore::One(pred_val) => { + // Here all the predecessors use a single value to represent our variable + // so we don't need to have it as an ebb argument. + // We need to replace all the occurences of val with pred_val but since + // we can't afford a re-writing pass right now we just declare an alias. + dfg.remove_ebb_arg(temp_arg_val); + dfg.change_to_alias(temp_arg_val, pred_val); + (pred_val, side_effects) + } + ZeroOneOrMore::More() => { + // There is disagreement in the predecessors on which value to use so we have + // to keep the ebb argument. + for (pred_block, last_inst, pred_val) in jump_args_to_append { + match self.append_jump_argument(dfg, + layout, + last_inst, + pred_block, + dest_ebb, + pred_val, + temp_arg_var, + jts) { + None => (), + Some(middle_ebb) => side_effects.split_ebbs_created.push(middle_ebb), + }; + } + (temp_arg_val, side_effects) + } + } + } + + /// Appends a jump argument to a jump instruction, returns ebb created in case of + /// critical edge splitting. + fn append_jump_argument(&mut self, + dfg: &mut DataFlowGraph, + layout: &mut Layout, + jump_inst: Inst, + jump_inst_block: Block, + dest_ebb: Ebb, + val: Value, + var: Variable, + jts: &mut JumpTables) + -> Option { + match dfg[jump_inst].analyze_branch(&dfg.value_lists) { + BranchInfo::NotABranch => { + panic!("you have declared a non-branch instruction as a predecessor to an ebb"); + } + // For a single destination appending a jump argument to the instruction + // is sufficient. + BranchInfo::SingleDest(_, _) => { + dfg.append_inst_arg(jump_inst, val); + None + } + BranchInfo::Table(jt) => { + // In the case of a jump table, the situation is tricky because br_table doesn't + // support arguments. + // We have to split the critical edge + let indexes: Vec = jts[jt] + .entries() + .fold(Vec::new(), |mut acc, (index, dest)| if dest == dest_ebb { + acc.push(index); + acc + } else { + acc + }); + let middle_ebb = dfg.make_ebb(); + layout.append_ebb(middle_ebb); + let block = self.declare_ebb_header_block(middle_ebb); + self.blocks[block].add_predecessor(jump_inst_block, jump_inst); + self.seal_ebb_header_block(middle_ebb, dfg, layout, jts); + for index in indexes { + jts[jt].set_entry(index, middle_ebb) + } + let mut cur = Cursor::new(layout); + cur.goto_bottom(middle_ebb); + let middle_jump_inst = dfg.ins(&mut cur).jump(dest_ebb, &[val]); + let dest_header_block = self.header_block(dest_ebb); + self.blocks[dest_header_block].add_predecessor(block, middle_jump_inst); + self.blocks[dest_header_block].remove_predecessor(jump_inst); + self.def_var(var, val, block); + Some(middle_ebb) + } + } + } + + /// Returns the list of `Ebb`s that have been declared as predecessors of the argument. + pub fn predecessors(&self, ebb: Ebb) -> &HashMap { + let block = match self.ebb_headers[ebb].expand() { + Some(block) => block, + None => panic!("the ebb has not been declared yet"), + }; + match self.blocks[block] { + BlockData::EbbBody { .. } => panic!("should not happen"), + BlockData::EbbHeader(ref data) => &data.predecessors, + } + } + + /// Returns `true` if and only if `seal_ebb_header_block` has been called on the argument. + pub fn is_sealed(&self, ebb: Ebb) -> bool { + match self.blocks[self.header_block(ebb)] { + BlockData::EbbBody { .. } => panic!("should not happen"), + BlockData::EbbHeader(ref data) => data.sealed, + } + } +} + +#[cfg(test)] +mod tests { + use cretonne::entity_ref::EntityRef; + use cretonne::ir::{Function, InstBuilder, Cursor, Inst, JumpTableData}; + use cretonne::ir::types::*; + use cretonne::verify_function; + use cretonne::ir::instructions::BranchInfo; + use ssa::SSABuilder; + use std::u32; + + /// An opaque reference to variable. + #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] + pub struct Variable(u32); + impl EntityRef for Variable { + fn new(index: usize) -> Self { + assert!(index < (u32::MAX as usize)); + Variable(index as u32) + } + + fn index(self) -> usize { + self.0 as usize + } + } + impl Default for Variable { + fn default() -> Variable { + Variable(u32::MAX) + } + } + + #[test] + fn simple_block() { + let mut func = Function::new(); + let mut ssa: SSABuilder = SSABuilder::new(); + let ebb0 = func.dfg.make_ebb(); + // Here is the pseudo-program we want to translate: + // x = 1; + // y = 2; + // z = x + y; + // z = x + z; + + let block = ssa.declare_ebb_header_block(ebb0); + let x_var = Variable(0); + let x_ssa = { + let cur = &mut Cursor::new(&mut func.layout); + cur.insert_ebb(ebb0); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iconst(I32, 1) + }; + ssa.def_var(x_var, x_ssa, block); + let y_var = Variable(1); + let y_ssa = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iconst(I32, 2) + }; + ssa.def_var(y_var, y_ssa, block); + + assert_eq!(ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block) + .0, + x_ssa); + assert_eq!(ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block) + .0, + y_ssa); + let z_var = Variable(2); + let x_use1 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block) + .0; + let y_use1 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block) + .0; + let z1_ssa = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iadd(x_use1, y_use1) + }; + ssa.def_var(z_var, z1_ssa, block); + assert_eq!(ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block) + .0, + z1_ssa); + let x_use2 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block) + .0; + let z_use1 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block) + .0; + let z2_ssa = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iadd(x_use2, z_use1) + }; + ssa.def_var(z_var, z2_ssa, block); + assert_eq!(ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block) + .0, + z2_ssa); + } + + #[test] + fn sequence_of_blocks() { + let mut func = Function::new(); + let mut ssa: SSABuilder = SSABuilder::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + // Here is the pseudo-program we want to translate: + // ebb0: + // x = 1; + // y = 2; + // z = x + y; + // brnz y, ebb1; + // z = x + z; + // ebb1: + // y = x + y; + + let block0 = ssa.declare_ebb_header_block(ebb0); + let x_var = Variable(0); + let x_ssa = { + let cur = &mut Cursor::new(&mut func.layout); + cur.insert_ebb(ebb0); + cur.insert_ebb(ebb1); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iconst(I32, 1) + }; + ssa.def_var(x_var, x_ssa, block0); + let y_var = Variable(1); + let y_ssa = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iconst(I32, 2) + }; + ssa.def_var(y_var, y_ssa, block0); + assert_eq!(ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block0) + .0, + x_ssa); + assert_eq!(ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block0) + .0, + y_ssa); + let z_var = Variable(2); + let x_use1 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block0) + .0; + let y_use1 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block0) + .0; + let z1_ssa = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iadd(x_use1, y_use1) + }; + ssa.def_var(z_var, z1_ssa, block0); + assert_eq!(ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block0) + .0, + z1_ssa); + let y_use2 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block0) + .0; + let jump_inst: Inst = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).brnz(y_use2, ebb1, &[]) + }; + let block1 = ssa.declare_ebb_body_block(block0); + let x_use2 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block1) + .0; + assert_eq!(x_use2, x_ssa); + let z_use1 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block1) + .0; + assert_eq!(z_use1, z1_ssa); + let z2_ssa = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iadd(x_use2, z_use1) + }; + ssa.def_var(z_var, z2_ssa, block1); + assert_eq!(ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block1) + .0, + z2_ssa); + ssa.seal_ebb_header_block(ebb0, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + let block2 = ssa.declare_ebb_header_block(ebb1); + ssa.declare_ebb_predecessor(ebb1, block0, jump_inst); + ssa.seal_ebb_header_block(ebb1, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + let x_use3 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block2) + .0; + assert_eq!(x_ssa, x_use3); + let y_use3 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block2) + .0; + assert_eq!(y_ssa, y_use3); + let y2_ssa = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb1); + func.dfg.ins(cur).iadd(x_use3, y_use3) + }; + ssa.def_var(y_var, y2_ssa, block2); + match func.dfg[jump_inst].analyze_branch(&func.dfg.value_lists) { + BranchInfo::SingleDest(dest, jump_args) => { + assert_eq!(dest, ebb1); + assert_eq!(jump_args.len(), 0); + } + _ => assert!(false), + }; + } + + #[test] + fn program_with_loop() { + let mut func = Function::new(); + let mut ssa: SSABuilder = SSABuilder::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + let ebb2 = func.dfg.make_ebb(); + // Here is the pseudo-program we want to translate: + // ebb0: + // x = 1; + // y = 2; + // z = x + y; + // jump ebb1 + // ebb1: + // z = z + y; + // brnz y, ebb1; + // z = z - x; + // return y + // ebb2: + // y = y - x + // jump ebb1 + + let block0 = ssa.declare_ebb_header_block(ebb0); + ssa.seal_ebb_header_block(ebb0, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + let x_var = Variable(0); + let x1 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.insert_ebb(ebb0); + cur.insert_ebb(ebb1); + cur.insert_ebb(ebb2); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iconst(I32, 1) + }; + ssa.def_var(x_var, x1, block0); + assert_eq!(ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block0) + .0, + x1); + let y_var = Variable(1); + let y1 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iconst(I32, 2) + }; + ssa.def_var(y_var, y1, block0); + assert_eq!(ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block0) + .0, + y1); + let z_var = Variable(2); + let x2 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block0) + .0; + assert_eq!(x2, x1); + let y2 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block0) + .0; + assert_eq!(y2, y1); + let z1 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iadd(x2, y2) + }; + ssa.def_var(z_var, z1, block0); + let jump_ebb0_ebb1 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).jump(ebb1, &[]) + }; + let block1 = ssa.declare_ebb_header_block(ebb1); + ssa.declare_ebb_predecessor(ebb1, block0, jump_ebb0_ebb1); + let z2 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block1) + .0; + let y3 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block1) + .0; + let z3 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb1); + func.dfg.ins(cur).iadd(z2, y3) + }; + ssa.def_var(z_var, z3, block1); + let y4 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block1) + .0; + assert_eq!(y4, y3); + let jump_ebb1_ebb2 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb1); + func.dfg.ins(cur).brnz(y4, ebb2, &[]) + }; + let block2 = ssa.declare_ebb_body_block(block1); + let z4 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block2) + .0; + assert_eq!(z4, z3); + let x3 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block2) + .0; + let z5 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb1); + func.dfg.ins(cur).isub(z4, x3) + }; + ssa.def_var(z_var, z5, block2); + let y5 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block2) + .0; + assert_eq!(y5, y3); + { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb1); + func.dfg.ins(cur).return_(&[y5]) + }; + + let block3 = ssa.declare_ebb_header_block(ebb2); + ssa.declare_ebb_predecessor(ebb2, block1, jump_ebb1_ebb2); + ssa.seal_ebb_header_block(ebb2, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + let y6 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block3) + .0; + assert_eq!(y6, y3); + let x4 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block3) + .0; + assert_eq!(x4, x3); + let y7 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb2); + func.dfg.ins(cur).isub(y6, x4) + }; + ssa.def_var(y_var, y7, block3); + let jump_ebb2_ebb1 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb2); + func.dfg.ins(cur).jump(ebb1, &[]) + }; + + ssa.declare_ebb_predecessor(ebb1, block3, jump_ebb2_ebb1); + ssa.seal_ebb_header_block(ebb1, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + assert_eq!(func.dfg.ebb_args(ebb1)[0], z2); + assert_eq!(func.dfg.ebb_args(ebb1)[1], y3); + assert_eq!(func.dfg.resolve_aliases(x3), x1); + + } + + #[test] + fn br_table_with_args() { + // This tests the on-demand splitting of critical edges for br_table with jump arguments + let mut func = Function::new(); + let mut ssa: SSABuilder = SSABuilder::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + // Here is the pseudo-program we want to translate: + // ebb0: + // x = 0; + // br_table x ebb1 + // x = 1 + // jump ebb1 + // ebb1: + // x = x + 1 + // return + // + let block0 = ssa.declare_ebb_header_block(ebb0); + ssa.seal_ebb_header_block(ebb0, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + let x_var = Variable(0); + let x1 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.insert_ebb(ebb0); + cur.insert_ebb(ebb1); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iconst(I32, 1) + }; + ssa.def_var(x_var, x1, block0); + let mut jt_data = JumpTableData::new(); + jt_data.set_entry(0, ebb1); + let jt = func.jump_tables.push(jt_data); + ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block0) + .0; + let br_table = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).br_table(x1, jt) + }; + let block1 = ssa.declare_ebb_body_block(block0); + let x3 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iconst(I32, 2) + }; + ssa.def_var(x_var, x3, block1); + let jump_inst = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).jump(ebb1, &[]) + }; + let block2 = ssa.declare_ebb_header_block(ebb1); + ssa.declare_ebb_predecessor(ebb1, block1, jump_inst); + ssa.declare_ebb_predecessor(ebb1, block0, br_table); + ssa.seal_ebb_header_block(ebb1, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + let x4 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block2) + .0; + { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb1); + func.dfg.ins(cur).iadd_imm(x4, 1) + }; + { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb1); + func.dfg.ins(cur).return_(&[]) + }; + match verify_function(&func, None) { + Ok(()) => {} + Err(err) => panic!(err.message), + } + } + + #[test] + fn undef_values_reordering() { + let mut func = Function::new(); + let mut ssa: SSABuilder = SSABuilder::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + // Here is the pseudo-program we want to translate: + // ebb0: + // x = 0 + // y = 1 + // z = 2 + // jump ebb1 + // ebb1: + // x = z + x + // y = y - x + // jump ebb1 + // + let block0 = ssa.declare_ebb_header_block(ebb0); + let x_var = Variable(0); + let y_var = Variable(1); + let z_var = Variable(2); + ssa.seal_ebb_header_block(ebb0, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + let x1 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.insert_ebb(ebb0); + cur.insert_ebb(ebb1); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iconst(I32, 0) + }; + ssa.def_var(x_var, x1, block0); + let y1 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iconst(I32, 1) + }; + ssa.def_var(y_var, y1, block0); + let z1 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).iconst(I32, 2) + }; + ssa.def_var(z_var, z1, block0); + let jump_inst = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).jump(ebb1, &[]) + }; + let block1 = ssa.declare_ebb_header_block(ebb1); + ssa.declare_ebb_predecessor(ebb1, block0, jump_inst); + let z2 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block1) + .0; + assert_eq!(func.dfg.ebb_args(ebb1)[0], z2); + let x2 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block1) + .0; + assert_eq!(func.dfg.ebb_args(ebb1)[1], x2); + let x3 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb1); + func.dfg.ins(cur).iadd(x2, z2) + }; + ssa.def_var(x_var, x3, block1); + let x4 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block1) + .0; + let y3 = ssa.use_var(&mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block1) + .0; + assert_eq!(func.dfg.ebb_args(ebb1)[2], y3); + let y4 = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb1); + func.dfg.ins(cur).isub(y3, x4) + }; + ssa.def_var(y_var, y4, block1); + let jump_inst = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb1); + func.dfg.ins(cur).jump(ebb1, &[]) + }; + ssa.declare_ebb_predecessor(ebb1, block1, jump_inst); + ssa.seal_ebb_header_block(ebb1, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + // At sealing the "z" argument disappear but the remaining "x" and "y" args have to be + // in the right order. + assert_eq!(func.dfg.ebb_args(ebb1)[1], y3); + assert_eq!(func.dfg.ebb_args(ebb1)[0], x2); + } +} From 7c438f866c9fee1cc444bdac710834fbccd89601 Mon Sep 17 00:00:00 2001 From: d1m0 Date: Wed, 12 Jul 2017 08:51:55 -0700 Subject: [PATCH 0857/3084] Add fix for #114 (#115) * Reduce code duplication in TypeConstraint subclasses; Add ConstrainWiderOrEqual to ti and to ireduce,{s,u}extend and f{promote,demote}; Fix bug in emitting constraint edges in TypeEnv.dot(); Modify runtime constraint checks to reject match when they encounter overflow * Rename Constrain types to something shorter; Move lane_bits/lane_counts in subclasses of ValueType; Add wider_or_eq function in rust and python; --- lib/cretonne/meta/base/instructions.py | 27 +-- lib/cretonne/meta/cdsl/instructions.py | 25 ++- lib/cretonne/meta/cdsl/test_ti.py | 103 ++++++++- lib/cretonne/meta/cdsl/ti.py | 271 ++++++++++++++++-------- lib/cretonne/meta/cdsl/types.py | 50 +++++ lib/cretonne/meta/gen_legalizer.py | 44 +++- lib/cretonne/meta/test_gen_legalizer.py | 76 ++++++- lib/cretonne/src/ir/types.rs | 7 + 8 files changed, 471 insertions(+), 132 deletions(-) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index dc9accf1cc..109bc73f9e 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -12,6 +12,7 @@ from base.types import i8, f32, f64, b1 from base.immediates import imm64, uimm8, ieee32, ieee64, offset32, uoffset32 from base.immediates import intcc, floatcc, memflags, regunit from base import entities +from cdsl.ti import WiderOrEq import base.formats # noqa GROUP = InstructionGroup("base", "Shared base instruction set") @@ -1405,7 +1406,7 @@ ireduce = Instruction( and each lane must not have more bits that the input lanes. If the input and output types are the same, this is a no-op. """, - ins=x, outs=a) + ins=x, outs=a, constraints=WiderOrEq(Int, IntTo)) IntTo = TypeVar( @@ -1427,7 +1428,7 @@ uextend = Instruction( and each lane must not have fewer bits that the input lanes. If the input and output types are the same, this is a no-op. """, - ins=x, outs=a) + ins=x, outs=a, constraints=WiderOrEq(IntTo, Int)) sextend = Instruction( 'sextend', r""" @@ -1441,7 +1442,7 @@ sextend = Instruction( and each lane must not have fewer bits that the input lanes. If the input and output types are the same, this is a no-op. """, - ins=x, outs=a) + ins=x, outs=a, constraints=WiderOrEq(IntTo, Int)) FloatTo = TypeVar( 'FloatTo', 'A scalar or vector floating point number', @@ -1457,14 +1458,14 @@ fpromote = Instruction( Each lane in `x` is converted to the destination floating point format. This is an exact operation. - Since Cretonne currently only supports two floating point formats, this - instruction always converts :type:`f32` to :type:`f64`. This may change - in the future. + Cretonne currently only supports two floating point formats + - :type:`f32` and :type:`f64`. This may change in the future. The result type must have the same number of vector lanes as the input, - and the result lanes must be larger than the input lanes. + and the result lanes must not have fewer bits than the input lanes. If + the input and output types are the same, this is a no-op. """, - ins=x, outs=a) + ins=x, outs=a, constraints=WiderOrEq(FloatTo, Float)) fdemote = Instruction( 'fdemote', r""" @@ -1473,14 +1474,14 @@ fdemote = Instruction( Each lane in `x` is converted to the destination floating point format by rounding to nearest, ties to even. - Since Cretonne currently only supports two floating point formats, this - instruction always converts :type:`f64` to :type:`f32`. This may change - in the future. + Cretonne currently only supports two floating point formats + - :type:`f32` and :type:`f64`. This may change in the future. The result type must have the same number of vector lanes as the input, - and the result lanes must be smaller than the input lanes. + and the result lanes must not have more bits than the input lanes. If + the input and output types are the same, this is a no-op. """, - ins=x, outs=a) + ins=x, outs=a, constraints=WiderOrEq(Float, FloatTo)) x = Operand('x', Float) a = Operand('a', IntTo) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 22c989bd65..511459fdbe 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -10,8 +10,10 @@ try: if TYPE_CHECKING: from .ast import Expr, Apply # noqa from .typevar import TypeVar # noqa + from .ti import TypeConstraint # noqa # List of operands for ins/outs: OpList = Union[Sequence[Operand], Operand] + ConstrList = Union[Sequence[TypeConstraint], TypeConstraint] MaybeBoundInst = Union['Instruction', 'BoundInstruction'] except ImportError: pass @@ -80,6 +82,7 @@ class Instruction(object): operands and other operand kinds. :param outs: Tuple of output operands. The output operands must be SSA values or `variable_args`. + :param constraints: Tuple of instruction-specific TypeConstraints. :param is_terminator: This is a terminator instruction. :param is_branch: This is a branch instruction. :param is_call: This is a call instruction. @@ -102,13 +105,14 @@ class Instruction(object): 'can_trap': 'Can this instruction cause a trap?', } - def __init__(self, name, doc, ins=(), outs=(), **kwargs): - # type: (str, str, OpList, OpList, **Any) -> None # noqa + def __init__(self, name, doc, ins=(), outs=(), constraints=(), **kwargs): + # type: (str, str, OpList, OpList, ConstrList, **Any) -> None self.name = name self.camel_name = camel_case(name) self.__doc__ = doc self.ins = self._to_operand_tuple(ins) self.outs = self._to_operand_tuple(outs) + self.constraints = self._to_constraint_tuple(constraints) self.format = InstructionFormat.lookup(self.ins, self.outs) # Opcode number, assigned by gen_instr.py. @@ -268,6 +272,23 @@ class Instruction(object): assert isinstance(op, Operand) return x + @staticmethod + def _to_constraint_tuple(x): + # type: (ConstrList) -> Tuple[TypeConstraint, ...] + """ + Allow a single TypeConstraint instance instead of the awkward singleton + tuple syntax. + """ + # import placed here to avoid circular dependency + from .ti import TypeConstraint # noqa + if isinstance(x, TypeConstraint): + x = (x,) + else: + x = tuple(x) + for op in x: + assert isinstance(op, TypeConstraint) + return x + def bind(self, *args): # type: (*ValueType) -> BoundInstruction """ diff --git a/lib/cretonne/meta/cdsl/test_ti.py b/lib/cretonne/meta/cdsl/test_ti.py index e89cedc16c..f60a9222f5 100644 --- a/lib/cretonne/meta/cdsl/test_ti.py +++ b/lib/cretonne/meta/cdsl/test_ti.py @@ -1,13 +1,14 @@ from __future__ import absolute_import from base.instructions import vselect, vsplit, vconcat, iconst, iadd, bint,\ - b1, icmp, iadd_cout, iadd_cin, uextend, ireduce + b1, icmp, iadd_cout, iadd_cin, uextend, sextend, ireduce, fpromote, \ + fdemote from base.legalize import narrow, expand from base.immediates import intcc from base.types import i32, i8 from .typevar import TypeVar from .ast import Var, Def from .xform import Rtl, XForm -from .ti import ti_rtl, subst, TypeEnv, get_type_env, ConstrainTVsEqual +from .ti import ti_rtl, subst, TypeEnv, get_type_env, TypesEqual, WiderOrEq from unittest import TestCase from functools import reduce @@ -52,9 +53,10 @@ def agree(me, other): # Translate our constraints using m, and sort me_equiv_constr = sorted([constr.translate(m) - for constr in me.constraints]) + for constr in me.constraints], key=repr) # Sort other's constraints - other_equiv_constr = sorted(other.constraints) + other_equiv_constr = sorted([constr.translate(other) + for constr in other.constraints], key=repr) return me_equiv_constr == other_equiv_constr @@ -78,7 +80,7 @@ def check_typing(got_or_err, expected, symtab=None): tv_m = {subst(k.get_typevar(), subst_m): v for (k, v) in m.items()} # Rewrite the TVs in the input constraints to their XForm internal # versions - c = [(subst(a, subst_m), subst(b, subst_m)) for (a, b) in c] + c = [constr.translate(subst_m) for constr in c] else: # If no symtab, just convert m from Var->TypeVar map to a # TypeVar->TypeVar map @@ -209,7 +211,7 @@ class TestRTL(TypeCheckingBaseTest): self.v3: txn, self.v4: txn, self.v5: txn, - }, [ConstrainTVsEqual(ixn.as_bool(), txn.as_bool())])) + }, [TypesEqual(ixn.as_bool(), txn.as_bool())])) def test_vselect_vsplits(self): # type: () -> None @@ -319,6 +321,90 @@ class TestRTL(TypeCheckingBaseTest): "Error: empty type created when unifying " + "`typeof_v4` and `typeof_v5`") + def test_extend_reduce(self): + # type: () -> None + r = Rtl( + self.v1 << uextend(self.v0), + self.v2 << ireduce(self.v1), + self.v3 << sextend(self.v2), + ) + ti = TypeEnv() + typing = ti_rtl(r, ti) + typing = typing.extract() + + itype0 = TypeVar("t", "", ints=True, simd=(1, 256)) + itype1 = TypeVar("t1", "", ints=True, simd=(1, 256)) + itype2 = TypeVar("t2", "", ints=True, simd=(1, 256)) + itype3 = TypeVar("t3", "", ints=True, simd=(1, 256)) + + check_typing(typing, ({ + self.v0: itype0, + self.v1: itype1, + self.v2: itype2, + self.v3: itype3, + }, [WiderOrEq(itype1, itype0), + WiderOrEq(itype1, itype2), + WiderOrEq(itype3, itype2)])) + + def test_extend_reduce_enumeration(self): + # type: () -> None + for op in (uextend, sextend, ireduce): + r = Rtl( + self.v1 << op(self.v0), + ) + ti = TypeEnv() + typing = ti_rtl(r, ti).extract() + + # The number of possible typings is 9 * (3+ 2*2 + 3) = 90 + l = [(t[self.v0], t[self.v1]) for t in typing.concrete_typings()] + assert (len(l) == len(set(l)) and len(l) == 90) + for (tv0, tv1) in l: + typ0, typ1 = (tv0.singleton_type(), tv1.singleton_type()) + if (op == ireduce): + assert typ0.wider_or_equal(typ1) + else: + assert typ1.wider_or_equal(typ0) + + def test_fpromote_fdemote(self): + # type: () -> None + r = Rtl( + self.v1 << fpromote(self.v0), + self.v2 << fdemote(self.v1), + ) + ti = TypeEnv() + typing = ti_rtl(r, ti) + typing = typing.extract() + + ftype0 = TypeVar("t", "", floats=True, simd=(1, 256)) + ftype1 = TypeVar("t1", "", floats=True, simd=(1, 256)) + ftype2 = TypeVar("t2", "", floats=True, simd=(1, 256)) + + check_typing(typing, ({ + self.v0: ftype0, + self.v1: ftype1, + self.v2: ftype2, + }, [WiderOrEq(ftype1, ftype0), + WiderOrEq(ftype1, ftype2)])) + + def test_fpromote_fdemote_enumeration(self): + # type: () -> None + for op in (fpromote, fdemote): + r = Rtl( + self.v1 << op(self.v0), + ) + ti = TypeEnv() + typing = ti_rtl(r, ti).extract() + + # The number of possible typings is 9*(2 + 1) = 27 + l = [(t[self.v0], t[self.v1]) for t in typing.concrete_typings()] + assert (len(l) == len(set(l)) and len(l) == 27) + for (tv0, tv1) in l: + (typ0, typ1) = (tv0.singleton_type(), tv1.singleton_type()) + if (op == fdemote): + assert typ0.wider_or_equal(typ1) + else: + assert typ1.wider_or_equal(typ0) + class TestXForm(TypeCheckingBaseTest): def test_iadd_cout(self): @@ -453,7 +539,7 @@ class TestXForm(TypeCheckingBaseTest): self.v3: i32t, self.v4: i32t, self.v5: i32t, - }, []), x.symtab) + }, [WiderOrEq(i32t, itype)]), x.symtab) def test_bound_inst_inference1(self): # Second example taken from issue #26 @@ -477,7 +563,7 @@ class TestXForm(TypeCheckingBaseTest): self.v3: i32t, self.v4: i32t, self.v5: i32t, - }, []), x.symtab) + }, [WiderOrEq(i32t, itype)]), x.symtab) def test_fully_bound_inst_inference(self): # Second example taken from issue #26 with complete bounds @@ -494,6 +580,7 @@ class TestXForm(TypeCheckingBaseTest): i8t = TypeVar.singleton(i8) i32t = TypeVar.singleton(i32) + # Note no constraints here since they are all trivial check_typing(x.ti, ({ self.v0: i8t, self.v1: i8t, diff --git a/lib/cretonne/meta/cdsl/ti.py b/lib/cretonne/meta/cdsl/ti.py index 85bd92507f..79023b8b34 100644 --- a/lib/cretonne/meta/cdsl/ti.py +++ b/lib/cretonne/meta/cdsl/ti.py @@ -8,7 +8,7 @@ from itertools import product try: from typing import Dict, TYPE_CHECKING, Union, Tuple, Optional, Set # noqa - from typing import Iterable, List # noqa + from typing import Iterable, List, Any # noqa from typing import cast from .xform import Rtl, XForm # noqa from .ast import Expr # noqa @@ -25,9 +25,72 @@ class TypeConstraint(object): """ Base class for all runtime-emittable type constraints. """ + def translate(self, m): + # type: (Union[TypeEnv, TypeMap]) -> TypeConstraint + """ + Translate any TypeVars in the constraint according to the map or + TypeEnv m + """ + def translate_one(a): + # type: (Any) -> Any + if (isinstance(a, TypeVar)): + return m[a] if isinstance(m, TypeEnv) else subst(a, m) + return a + + res = None # type: TypeConstraint + res = self.__class__(*tuple(map(translate_one, self._args()))) + return res + + def __eq__(self, other): + # type: (object) -> bool + if (not isinstance(other, self.__class__)): + return False + + assert isinstance(other, TypeConstraint) # help MyPy figure out other + return self._args() == other._args() + + def is_concrete(self): + # type: () -> bool + """ + Return true iff all typevars in the constraint are singletons. + """ + tvs = filter(lambda x: isinstance(x, TypeVar), self._args()) + return [] == list(filter(lambda x: x.singleton_type() is None, tvs)) + + def __hash__(self): + # type: () -> int + return hash(self._args()) + + def _args(self): + # type: () -> Tuple[Any,...] + """ + Return a tuple with the exact arguments passed to __init__ to create + this object. + """ + assert False, "Abstract" + + def is_trivial(self): + # type: () -> bool + """ + Return true if this constrain is statically decidable. + """ + assert False, "Abstract" + + def eval(self): + # type: () -> bool + """ + Evaluate this constraint. Should only be called when the constraint has + been translated to concrete types. + """ + assert False, "Abstract" + + def __repr__(self): + # type: () -> str + return (self.__class__.__name__ + '(' + + ', '.join(map(str, self._args())) + ')') -class ConstrainTVsEqual(TypeConstraint): +class TypesEqual(TypeConstraint): """ Constraint specifying that two derived type vars must have the same runtime type. @@ -37,48 +100,24 @@ class ConstrainTVsEqual(TypeConstraint): assert tv1.is_derived and tv2.is_derived (self.tv1, self.tv2) = sorted([tv1, tv2], key=repr) + def _args(self): + # type: () -> Tuple[Any,...] + """ See TypeConstraint._args() """ + return (self.tv1, self.tv2) + def is_trivial(self): # type: () -> bool - """ - Return true if this constrain is statically decidable. - """ - return self.tv1 == self.tv2 or \ - (self.tv1.singleton_type() is not None and - self.tv2.singleton_type() is not None) - - def translate(self, m): - # type: (Union[TypeEnv, TypeMap]) -> ConstrainTVsEqual - """ - Translate any TypeVars in the constraint according to the map m - """ - if isinstance(m, TypeEnv): - return ConstrainTVsEqual(m[self.tv1], m[self.tv2]) - else: - return ConstrainTVsEqual(subst(self.tv1, m), subst(self.tv2, m)) - - def __eq__(self, other): - # type: (object) -> bool - if (not isinstance(other, ConstrainTVsEqual)): - return False - - return (self.tv1, self.tv2) == (other.tv1, other.tv2) - - def __hash__(self): - # type: () -> int - return hash((self.tv1, self.tv2)) + """ See TypeConstraint.is_trivial() """ + return self.tv1 == self.tv2 or self.is_concrete() def eval(self): # type: () -> bool - """ - Evaluate this constraint. Should only be called when the constraint has - been translated to concrete types. - """ - assert self.tv1.singleton_type() is not None and \ - self.tv2.singleton_type() is not None + """ See TypeConstraint.eval() """ + assert self.is_concrete() return self.tv1.singleton_type() == self.tv2.singleton_type() -class ConstrainTVInTypeset(TypeConstraint): +class InTypeset(TypeConstraint): """ Constraint specifying that a type var must belong to some typeset. """ @@ -88,11 +127,14 @@ class ConstrainTVInTypeset(TypeConstraint): self.tv = tv self.ts = ts + def _args(self): + # type: () -> Tuple[Any,...] + """ See TypeConstraint._args() """ + return (self.tv, self.ts) + def is_trivial(self): # type: () -> bool - """ - Return true if this constrain is statically decidable. - """ + """ See TypeConstraint.is_trivial() """ tv_ts = self.tv.get_typeset().copy() # Trivially True @@ -104,39 +146,78 @@ class ConstrainTVInTypeset(TypeConstraint): if (tv_ts.size() == 0): return True - return False - - def translate(self, m): - # type: (Union[TypeEnv, TypeMap]) -> ConstrainTVInTypeset - """ - Translate any TypeVars in the constraint according to the map m - """ - if isinstance(m, TypeEnv): - return ConstrainTVInTypeset(m[self.tv], self.ts) - else: - return ConstrainTVInTypeset(subst(self.tv, m), self.ts) - - def __eq__(self, other): - # type: (object) -> bool - if (not isinstance(other, ConstrainTVInTypeset)): - return False - - return (self.tv, self.ts) == (other.tv, other.ts) - - def __hash__(self): - # type: () -> int - return hash((self.tv, self.ts)) + return self.is_concrete() def eval(self): # type: () -> bool - """ - Evaluate this constraint. Should only be called when the constraint has - been translated to concrete types. - """ - assert self.tv.singleton_type() is not None + """ See TypeConstraint.eval() """ + assert self.is_concrete() return self.tv.get_typeset().issubset(self.ts) +class WiderOrEq(TypeConstraint): + """ + Constraint specifying that a type var tv1 must be wider than or equal to + type var tv2 at runtime. This requires that: + 1) They have the same number of lanes + 2) In a lane tv1 has at least as many bits as tv2. + """ + def __init__(self, tv1, tv2): + # type: (TypeVar, TypeVar) -> None + self.tv1 = tv1 + self.tv2 = tv2 + + def _args(self): + # type: () -> Tuple[Any,...] + """ See TypeConstraint._args() """ + return (self.tv1, self.tv2) + + def is_trivial(self): + # type: () -> bool + """ See TypeConstraint.is_trivial() """ + # Trivially true + if (self.tv1 == self.tv2): + return True + + ts1 = self.tv1.get_typeset() + ts2 = self.tv2.get_typeset() + + def set_wider_or_equal(s1, s2): + # type: (Set[int], Set[int]) -> bool + return len(s1) > 0 and len(s2) > 0 and min(s1) >= max(s2) + + # Trivially True + if set_wider_or_equal(ts1.ints, ts2.ints) and\ + set_wider_or_equal(ts1.floats, ts2.floats) and\ + set_wider_or_equal(ts1.bools, ts2.bools): + return True + + def set_narrower(s1, s2): + # type: (Set[int], Set[int]) -> bool + return len(s1) > 0 and len(s2) > 0 and min(s1) < max(s2) + + # Trivially False + if set_narrower(ts1.ints, ts2.ints) and\ + set_narrower(ts1.floats, ts2.floats) and\ + set_narrower(ts1.bools, ts2.bools): + return True + + # Trivially False + if len(ts1.lanes.intersection(ts2.lanes)) == 0: + return True + + return self.is_concrete() + + def eval(self): + # type: () -> bool + """ See TypeConstraint.eval() """ + assert self.is_concrete() + typ1 = self.tv1.singleton_type() + typ2 = self.tv2.singleton_type() + + return typ1.wider_or_equal(typ2) + + class TypeEnv(object): """ Class encapsulating the neccessary book keeping for type inference. @@ -204,12 +285,11 @@ class TypeEnv(object): self.type_map[tv1] = tv2 - def add_constraint(self, tv1, tv2): - # type: (TypeVar, TypeVar) -> None + def add_constraint(self, constr): + # type: (TypeConstraint) -> None """ - Add a new equivalence constraint between tv1 and tv2 + Add a new constraint """ - constr = ConstrainTVsEqual(tv1, tv2) if (constr not in self.constraints): self.constraints.append(constr) @@ -261,6 +341,7 @@ class TypeEnv(object): Get the free typevars in the current type env. """ tvs = set([self[tv].free_typevar() for tv in self.type_map.keys()]) + tvs = tvs.union(set([self[v].free_typevar() for v in self.vars])) # Filter out None here due to singleton type vars return sorted(filter(lambda x: x is not None, tvs), key=lambda x: x.name) @@ -326,17 +407,18 @@ class TypeEnv(object): new_constraints = [] # type: List[TypeConstraint] for constr in self.constraints: - # Currently typeinference only generates ConstrainTVsEqual - # constraints - assert isinstance(constr, ConstrainTVsEqual) constr = constr.translate(self) if constr.is_trivial() or constr in new_constraints: continue # Sanity: translated constraints should refer to only real vars - assert constr.tv1.free_typevar() in vars_tvs and\ - constr.tv2.free_typevar() in vars_tvs + for arg in constr._args(): + if (not isinstance(arg, TypeVar)): + continue + + arg_free_tv = arg.free_typevar() + assert arg_free_tv is None or arg_free_tv in vars_tvs new_constraints.append(constr) @@ -372,9 +454,6 @@ class TypeEnv(object): # Check if constraints are satisfied for this typing failed = None for constr in self.constraints: - # Currently typeinference only generates ConstrainTVsEqual - # constraints - assert isinstance(constr, ConstrainTVsEqual) concrete_constr = constr.translate(m) if not concrete_constr.eval(): failed = concrete_constr @@ -401,22 +480,27 @@ class TypeEnv(object): # Add all registered TVs (as some of them may be singleton nodes not # appearing in the graph nodes = set([v.get_typevar() for v in self.vars]) # type: Set[TypeVar] - edges = set() # type: Set[Tuple[TypeVar, TypeVar, str, Optional[str]]] + edges = set() # type: Set[Tuple[TypeVar, TypeVar, str, str, Optional[str]]] # noqa for (k, v) in self.type_map.items(): # Add all intermediate TVs appearing in edges nodes.add(k) nodes.add(v) - edges.add((k, v, "dotted", None)) + edges.add((k, v, "dotted", "forward", None)) while (v.is_derived): nodes.add(v.base) - edges.add((v, v.base, "solid", v.derived_func)) + edges.add((v, v.base, "solid", "forward", v.derived_func)) v = v.base for constr in self.constraints: - assert isinstance(constr, ConstrainTVsEqual) - assert constr.tv1 in nodes and constr.tv2 in nodes - edges.add((constr.tv1, constr.tv2, "dashed", None)) + if isinstance(constr, TypesEqual): + assert constr.tv1 in nodes and constr.tv2 in nodes + edges.add((constr.tv1, constr.tv2, "dashed", "none", "equal")) + elif isinstance(constr, WiderOrEq): + assert constr.tv1 in nodes and constr.tv2 in nodes + edges.add((constr.tv1, constr.tv2, "dashed", "forward", ">=")) + else: + assert False, "Can't display constraint {}".format(constr) root_nodes = set([x for x in nodes if x not in self.type_map and not x.is_derived]) @@ -428,17 +512,12 @@ class TypeEnv(object): r += "[xlabel=\"{}\"]".format(self[n].get_typeset()) r += ";\n" - for (n1, n2, style, elabel) in edges: - e = label(n1) - if style == "dashed": - e += '--' - else: - e += '->' - e += label(n2) - e += "[style={}".format(style) + for (n1, n2, style, direction, elabel) in edges: + e = label(n1) + "->" + label(n2) + e += "[style={},dir={}".format(style, direction) if elabel is not None: - e += ",label={}".format(elabel) + e += ",label=\"{}\"".format(elabel) e += "];\n" r += e @@ -589,7 +668,7 @@ def unify(tv1, tv2, typ): inv_f = TypeVar.inverse_func(tv1.derived_func) return unify(tv1.base, normalize_tv(TypeVar.derived(tv2, inv_f)), typ) - typ.add_constraint(tv1, tv2) + typ.add_constraint(TypesEqual(tv1, tv2)) return typ @@ -648,6 +727,10 @@ def ti_def(definition, typ): typ = get_type_env(typ_or_err) + # Add any instruction specific constraints + for constr in inst.constraints: + typ.add_constraint(constr.translate(m)) + return typ diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py index eeb5325ae8..cebe472fbf 100644 --- a/lib/cretonne/meta/cdsl/types.py +++ b/lib/cretonne/meta/cdsl/types.py @@ -49,6 +49,26 @@ class ValueType(object): else: raise AttributeError("No type named '{}'".format(name)) + def lane_bits(self): + # type: () -> int + """Return the number of bits in a lane.""" + assert False, "Abstract" + + def lane_count(self): + # type: () -> int + """Return the number of lanes.""" + assert False, "Abstract" + + def wider_or_equal(self, other): + # type: (ValueType) -> bool + """ + Return true iff: + 1. self and other have equal number of lanes + 2. each lane in self has at least as many bits as a lane in other + """ + return (self.lane_count() == other.lane_count() and + self.lane_bits() >= other.lane_bits()) + class ScalarType(ValueType): """ @@ -85,6 +105,11 @@ class ScalarType(ValueType): self._vectors[lanes] = v return v + def lane_count(self): + # type: () -> int + """Return the number of lanes.""" + return 1 + class VectorType(ValueType): """ @@ -112,6 +137,16 @@ class VectorType(ValueType): return ('VectorType(base={}, lanes={})' .format(self.base.name, self.lanes)) + def lane_count(self): + # type: () -> int + """Return the number of lanes.""" + return self.lanes + + def lane_bits(self): + # type: () -> int + """Return the number of bits in a lane.""" + return self.base.lane_bits() + class IntType(ScalarType): """A concrete scalar integer type.""" @@ -138,6 +173,11 @@ class IntType(ScalarType): else: return typ + def lane_bits(self): + # type: () -> int + """Return the number of bits in a lane.""" + return self.bits + class FloatType(ScalarType): """A concrete scalar floating point type.""" @@ -164,6 +204,11 @@ class FloatType(ScalarType): else: return typ + def lane_bits(self): + # type: () -> int + """Return the number of bits in a lane.""" + return self.bits + class BoolType(ScalarType): """A concrete scalar boolean type.""" @@ -189,3 +234,8 @@ class BoolType(ScalarType): return cast(BoolType, typ) else: return typ + + def lane_bits(self): + # type: () -> int + """Return the number of bits in a lane.""" + return self.bits diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index ce2f559ef2..eeb567722e 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -11,8 +11,8 @@ from __future__ import absolute_import from srcgen import Formatter from base import legalize, instructions from cdsl.ast import Var -from cdsl.ti import ti_rtl, TypeEnv, get_type_env, ConstrainTVsEqual,\ - ConstrainTVInTypeset +from cdsl.ti import ti_rtl, TypeEnv, get_type_env, TypesEqual,\ + InTypeset, WiderOrEq from unique_table import UniqueTable from gen_instr import gen_typesets_table from cdsl.typevar import TypeVar @@ -66,7 +66,7 @@ def get_runtime_typechecks(xform): assert xform_ts.issubset(src_ts) if src_ts != xform_ts: - check_l.append(ConstrainTVInTypeset(xform.ti[v], xform_ts)) + check_l.append(InTypeset(xform.ti[v], xform_ts)) # 2,3) Add any constraints that appear in xform.ti check_l.extend(xform.ti.constraints) @@ -81,6 +81,14 @@ def emit_runtime_typecheck(check, fmt, type_sets): """ def build_derived_expr(tv): # type: (TypeVar) -> str + """ + Build an expression of type Option corresponding to a concrete + type transformed by the sequence of derivation functions in tv. + + We are using Option, as some constraints may cause an + over/underflow on patterns that do not match them. We want to capture + this without panicking at runtime. + """ if not tv.is_derived: assert tv.name.startswith('typeof_') return "Some({})".format(tv.name) @@ -102,7 +110,8 @@ def emit_runtime_typecheck(check, fmt, type_sets): else: assert False, "Unknown derived function {}".format(tv.derived_func) - if (isinstance(check, ConstrainTVInTypeset)): + if (isinstance(check, InTypeset)): + assert not check.tv.is_derived tv = check.tv.name if check.ts not in type_sets.index: type_sets.add(check.ts) @@ -112,11 +121,28 @@ def emit_runtime_typecheck(check, fmt, type_sets): with fmt.indented('if !TYPE_SETS[{}].contains({}) {{'.format(ts, tv), '};'): fmt.line('return false;') - elif (isinstance(check, ConstrainTVsEqual)): - tv1 = build_derived_expr(check.tv1) - tv2 = build_derived_expr(check.tv2) - with fmt.indented('if {} != {} {{'.format(tv1, tv2), '};'): - fmt.line('return false;') + elif (isinstance(check, TypesEqual)): + with fmt.indented('{', '};'): + fmt.line('let a = {};'.format(build_derived_expr(check.tv1))) + fmt.line('let b = {};'.format(build_derived_expr(check.tv2))) + + fmt.comment('On overflow constraint doesn\'t appply') + with fmt.indented('if a.is_none() || b.is_none() {', '};'): + fmt.line('return false;') + + with fmt.indented('if a != b {', '};'): + fmt.line('return false;') + elif (isinstance(check, WiderOrEq)): + with fmt.indented('{', '};'): + fmt.line('let a = {};'.format(build_derived_expr(check.tv1))) + fmt.line('let b = {};'.format(build_derived_expr(check.tv2))) + + fmt.comment('On overflow constraint doesn\'t appply') + with fmt.indented('if a.is_none() || b.is_none() {', '};'): + fmt.line('return false;') + + with fmt.indented('if !a.wider_or_equal(b) {', '};'): + fmt.line('return false;') else: assert False, "Unknown check {}".format(check) diff --git a/lib/cretonne/meta/test_gen_legalizer.py b/lib/cretonne/meta/test_gen_legalizer.py index 38f26959f4..538ff69b63 100644 --- a/lib/cretonne/meta/test_gen_legalizer.py +++ b/lib/cretonne/meta/test_gen_legalizer.py @@ -4,7 +4,7 @@ from unittest import TestCase from srcgen import Formatter from gen_legalizer import get_runtime_typechecks, emit_runtime_typecheck from base.instructions import vselect, vsplit, isplit, iconcat, vconcat, \ - iconst, b1, icmp, copy # noqa + iconst, b1, icmp, copy, sextend, uextend, ireduce, fdemote, fpromote # noqa from base.legalize import narrow, expand # noqa from base.immediates import intcc # noqa from cdsl.typevar import TypeVar, TypeSet @@ -57,9 +57,32 @@ def equiv_check(tv1, tv2): # type: (TypeVar, TypeVar) -> CheckProducer return lambda typesets: format_check( typesets, - 'if Some({}).map(|t: Type| -> t.as_bool()) != ' + - 'Some({}).map(|t: Type| -> t.as_bool()) ' + - '{{\n return false;\n}};\n', tv1, tv2) + '{{\n' + + ' let a = {};\n' + + ' let b = {};\n' + + ' if a.is_none() || b.is_none() {{\n' + + ' return false;\n' + + ' }};\n' + + ' if a != b {{\n' + + ' return false;\n' + + ' }};\n' + + '}};\n', tv1, tv2) + + +def wider_check(tv1, tv2): + # type: (TypeVar, TypeVar) -> CheckProducer + return lambda typesets: format_check( + typesets, + '{{\n' + + ' let a = {};\n' + + ' let b = {};\n' + + ' if a.is_none() || b.is_none() {{\n' + + ' return false;\n' + + ' }};\n' + + ' if !a.wider_or_equal(b) {{\n' + + ' return false;\n' + + ' }};\n' + + '}};\n', tv1, tv2) def sequence(*args): @@ -138,8 +161,49 @@ class TestRuntimeChecks(TestCase): self.v5 << vselect(self.v1, self.v3, self.v4), ) x = XForm(r, r) + tv2_exp = 'Some({}).map(|t: Type| -> t.as_bool())'\ + .format(self.v2.get_typevar().name) + tv3_exp = 'Some({}).map(|t: Type| -> t.as_bool())'\ + .format(self.v3.get_typevar().name) self.check_yo_check( x, sequence(typeset_check(self.v3, ts), - equiv_check(self.v2.get_typevar(), - self.v3.get_typevar()))) + equiv_check(tv2_exp, tv3_exp))) + + def test_reduce_extend(self): + # type: () -> None + r = Rtl( + self.v1 << uextend(self.v0), + self.v2 << ireduce(self.v1), + self.v3 << sextend(self.v2), + ) + x = XForm(r, r) + + tv0_exp = 'Some({})'.format(self.v0.get_typevar().name) + tv1_exp = 'Some({})'.format(self.v1.get_typevar().name) + tv2_exp = 'Some({})'.format(self.v2.get_typevar().name) + tv3_exp = 'Some({})'.format(self.v3.get_typevar().name) + + self.check_yo_check( + x, sequence(wider_check(tv1_exp, tv0_exp), + wider_check(tv1_exp, tv2_exp), + wider_check(tv3_exp, tv2_exp))) + + def test_demote_promote(self): + # type: () -> None + r = Rtl( + self.v1 << fpromote(self.v0), + self.v2 << fdemote(self.v1), + self.v3 << fpromote(self.v2), + ) + x = XForm(r, r) + + tv0_exp = 'Some({})'.format(self.v0.get_typevar().name) + tv1_exp = 'Some({})'.format(self.v1.get_typevar().name) + tv2_exp = 'Some({})'.format(self.v2.get_typevar().name) + tv3_exp = 'Some({})'.format(self.v3.get_typevar().name) + + self.check_yo_check( + x, sequence(wider_check(tv1_exp, tv0_exp), + wider_check(tv1_exp, tv2_exp), + wider_check(tv3_exp, tv2_exp))) diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index e6379be2e6..0007154025 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -236,6 +236,13 @@ impl Type { pub fn index(self) -> usize { self.0 as usize } + + /// True iff: + /// 1) self.lane_count() == other.lane_count() and + /// 2) self.lane_bits() >= other.lane_bits() + pub fn wider_or_equal(self, other: Type) -> bool { + self.lane_count() == other.lane_count() && self.lane_bits() >= other.lane_bits() + } } impl Display for Type { From fc94ce7e8cc83d2137eea2830aabd6135f01979c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Jul 2017 09:15:40 -0700 Subject: [PATCH 0858/3084] Add an other_side_effects instruction flag. This is used to indicate instructions that have some side effect that is not modelled by the more specific instruction flags. --- lib/cretonne/meta/cdsl/instructions.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 511459fdbe..7b8cc55b31 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -90,6 +90,7 @@ class Instruction(object): :param can_trap: This instruction can trap. :param can_load: This instruction can load from memory. :param can_store: This instruction can store to memory. + :param other_side_effects: Instruction has other side effects. """ # Boolean instruction attributes that can be passed as keyword arguments to @@ -103,6 +104,8 @@ class Instruction(object): 'can_load': 'Can this instruction read from memory?', 'can_store': 'Can this instruction write to memory?', 'can_trap': 'Can this instruction cause a trap?', + 'other_side_effects': + 'Does this instruction have other side effects besides can_*', } def __init__(self, name, doc, ins=(), outs=(), constraints=(), **kwargs): From 924c4649cc0c8e967116245104d58903d86fd1b7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Jul 2017 09:29:48 -0700 Subject: [PATCH 0859/3084] Enforce encodings for instructions with side effects. We allow ghost instructions to exist if they have no side effects. Instructions that affect control flow or that have other side effects must be encoded. Teach the IL verifier to enforce this. Once any instruction has an encoding, all instructions with side effects must have an encoding. --- cranelift/filetests/regalloc/coalesce.cton | 2 +- lib/cretonne/src/verifier/mod.rs | 80 ++++++++++++++++------ 2 files changed, 61 insertions(+), 21 deletions(-) diff --git a/cranelift/filetests/regalloc/coalesce.cton b/cranelift/filetests/regalloc/coalesce.cton index 41fc6e9771..80cd38e62e 100644 --- a/cranelift/filetests/regalloc/coalesce.cton +++ b/cranelift/filetests/regalloc/coalesce.cton @@ -59,7 +59,7 @@ ebb0(v0: i32): brnz v0, ebb1(v0) v1 = iadd_imm v0, 7 ; v1 and v0 interfere here: - trapnz v0 + v2 = iadd_imm v0, 8 ; check: $(cp1=$V) = copy $v1 ; not: copy ; check: jump $ebb1($cp1) diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 2f817d1a48..12b190e3da 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -666,30 +666,70 @@ impl<'a> Verifier<'a> { /// If the verifier has been set up with an ISA, make sure that the recorded encoding for the /// instruction (if any) matches how the ISA would encode it. fn verify_encoding(&self, inst: Inst) -> Result { - if let Some(isa) = self.isa { - let encoding = self.func.encodings.get_or_default(inst); - if encoding.is_legal() { - let verify_encoding = - isa.encode(&self.func.dfg, - &self.func.dfg[inst], - self.func.dfg.ctrl_typevar(inst)); - match verify_encoding { - Ok(verify_encoding) => { - if verify_encoding != encoding { - return err!(inst, - "Instruction re-encoding {} doesn't match {}", - isa.encoding_info().display(verify_encoding), - isa.encoding_info().display(encoding)); - } - } - Err(e) => { + // When the encodings table is empty, we don't require any instructions to be encoded. + // + // Once some instructions are encoded, we require all side-effecting instructions to have a + // legal encoding. + if self.func.encodings.is_empty() { + return Ok(()); + } + + let isa = match self.isa { + Some(isa) => isa, + None => return Ok(()), + }; + + let encoding = self.func.encodings.get_or_default(inst); + if encoding.is_legal() { + let verify_encoding = + isa.encode(&self.func.dfg, + &self.func.dfg[inst], + self.func.dfg.ctrl_typevar(inst)); + match verify_encoding { + Ok(verify_encoding) => { + if verify_encoding != encoding { return err!(inst, - "Instruction failed to re-encode {}: {:?}", - isa.encoding_info().display(encoding), - e) + "Instruction re-encoding {} doesn't match {}", + isa.encoding_info().display(verify_encoding), + isa.encoding_info().display(encoding)); } } + Err(e) => { + return err!(inst, + "Instruction failed to re-encode {}: {:?}", + isa.encoding_info().display(encoding), + e) + } } + return Ok(()); + } + + // Instruction is not encoded, so it is a ghost instruction. + // Instructions with side effects are not allowed to be ghost instructions. + let opcode = self.func.dfg[inst].opcode(); + + if opcode.is_branch() { + return err!(inst, "Branch must have an encoding"); + } + + if opcode.is_call() { + return err!(inst, "Call must have an encoding"); + } + + if opcode.is_return() { + return err!(inst, "Return must have an encoding"); + } + + if opcode.can_store() { + return err!(inst, "Store must have an encoding"); + } + + if opcode.can_trap() { + return err!(inst, "Trapping instruction must have an encoding"); + } + + if opcode.other_side_effects() { + return err!(inst, "Instruction with side effects must have an encoding"); } Ok(()) From e4da2e1f22707cb542af0dbb5a84696ec6533766 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Jul 2017 09:59:00 -0700 Subject: [PATCH 0860/3084] Include ISA-specific information in verifier errors. When the test driver reports a verifier error, make sure to include the TargetIsa when printing the failing function. --- cranelift/src/filetest/legalizer.rs | 2 +- cranelift/src/filetest/licm.rs | 2 +- cranelift/src/filetest/regalloc.rs | 4 ++-- cranelift/src/filetest/runone.rs | 2 +- cranelift/src/filetest/simple_gvn.rs | 2 +- cranelift/src/utils.rs | 12 ++++++++---- 6 files changed, 14 insertions(+), 10 deletions(-) diff --git a/cranelift/src/filetest/legalizer.rs b/cranelift/src/filetest/legalizer.rs index 806730b4ed..0c5ea04b77 100644 --- a/cranelift/src/filetest/legalizer.rs +++ b/cranelift/src/filetest/legalizer.rs @@ -43,7 +43,7 @@ impl SubTest for TestLegalizer { comp_ctx.flowgraph(); comp_ctx .legalize(isa) - .map_err(|e| pretty_error(&comp_ctx.func, e))?; + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; let mut text = String::new(); write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))) diff --git a/cranelift/src/filetest/licm.rs b/cranelift/src/filetest/licm.rs index dcde7dd7be..548818e758 100644 --- a/cranelift/src/filetest/licm.rs +++ b/cranelift/src/filetest/licm.rs @@ -41,7 +41,7 @@ impl SubTest for TestLICM { comp_ctx.flowgraph(); comp_ctx .licm() - .map_err(|e| pretty_error(&comp_ctx.func, e))?; + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; let mut text = String::new(); write!(&mut text, "{}", &comp_ctx.func) diff --git a/cranelift/src/filetest/regalloc.rs b/cranelift/src/filetest/regalloc.rs index 1699d384a8..e90012973b 100644 --- a/cranelift/src/filetest/regalloc.rs +++ b/cranelift/src/filetest/regalloc.rs @@ -48,10 +48,10 @@ impl SubTest for TestRegalloc { // TODO: Should we have an option to skip legalization? comp_ctx .legalize(isa) - .map_err(|e| pretty_error(&comp_ctx.func, e))?; + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; comp_ctx .regalloc(isa) - .map_err(|e| pretty_error(&comp_ctx.func, e))?; + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; let mut text = String::new(); write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))) diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index b43d926f0e..3b23463fef 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -118,7 +118,7 @@ fn run_one_test<'a>(tuple: (&'a SubTest, &'a Flags, Option<&'a TargetIsa>), // Should we run the verifier before this test? if !context.verified && test.needs_verifier() { verify_function(&func, isa) - .map_err(|e| pretty_verifier_error(&func, e))?; + .map_err(|e| pretty_verifier_error(&func, isa, e))?; context.verified = true; } diff --git a/cranelift/src/filetest/simple_gvn.rs b/cranelift/src/filetest/simple_gvn.rs index f220cbdde7..a522c17bad 100644 --- a/cranelift/src/filetest/simple_gvn.rs +++ b/cranelift/src/filetest/simple_gvn.rs @@ -41,7 +41,7 @@ impl SubTest for TestSimpleGVN { comp_ctx.flowgraph(); comp_ctx .simple_gvn() - .map_err(|e| pretty_error(&comp_ctx.func, e))?; + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; let mut text = String::new(); write!(&mut text, "{}", &comp_ctx.func) diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index b637f7ca49..f620f5f316 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -3,6 +3,7 @@ use cretonne::ir::entities::AnyEntity; use cretonne::{ir, verifier}; use cretonne::result::CtonError; +use cretonne::isa::TargetIsa; use std::fmt::Write; use std::fs::File; use std::io::{Result, Read}; @@ -34,7 +35,10 @@ pub fn match_directive<'a>(comment: &'a str, directive: &str) -> Option<&'a str> } /// Pretty-print a verifier error. -pub fn pretty_verifier_error(func: &ir::Function, err: verifier::Error) -> String { +pub fn pretty_verifier_error(func: &ir::Function, + isa: Option<&TargetIsa>, + err: verifier::Error) + -> String { let mut msg = err.to_string(); match err.location { AnyEntity::Inst(inst) => { @@ -42,14 +46,14 @@ pub fn pretty_verifier_error(func: &ir::Function, err: verifier::Error) -> Strin } _ => msg.push('\n'), } - write!(msg, "{}", func).unwrap(); + write!(msg, "{}", func.display(isa)).unwrap(); msg } /// Pretty-print a Cretonne error. -pub fn pretty_error(func: &ir::Function, err: CtonError) -> String { +pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CtonError) -> String { if let CtonError::Verifier(e) = err { - pretty_verifier_error(func, e) + pretty_verifier_error(func, isa, e) } else { err.to_string() } From 69f974ba5d320c0dbe2e6186d6a3b1de410df844 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Jul 2017 10:12:20 -0700 Subject: [PATCH 0861/3084] Add an ISA argument to dfg.display_inst(). Include ISA-specific annotations in tracing and error messages. --- cranelift/src/filetest/binemit.rs | 8 +++---- cranelift/src/utils.rs | 2 +- lib/cretonne/src/binemit/mod.rs | 2 +- lib/cretonne/src/binemit/relaxation.rs | 2 +- lib/cretonne/src/ir/dfg.rs | 31 ++++++++++++++----------- lib/cretonne/src/legalizer/boundary.rs | 6 ++--- lib/cretonne/src/legalizer/split.rs | 2 +- lib/cretonne/src/regalloc/coalescing.rs | 8 +++---- lib/cretonne/src/regalloc/coloring.rs | 6 +++-- lib/cretonne/src/regalloc/spilling.rs | 8 ++++--- 10 files changed, 42 insertions(+), 33 deletions(-) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 2f695bff1e..15f5b0a982 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -127,7 +127,7 @@ impl SubTest for TestBinEmit { AnyEntity::Inst(inst) => { if let Some(prev) = bins.insert(inst, want) { return Err(format!("multiple 'bin:' directives on {}: '{}' and '{}'", - func.dfg.display_inst(inst), + func.dfg.display_inst(inst, isa), prev, want)); } @@ -166,7 +166,7 @@ impl SubTest for TestBinEmit { encinfo.bytes(enc), "Inconsistent size for [{}] {}", encinfo.display(enc), - func.dfg.display_inst(inst)); + func.dfg.display_inst(inst, isa)); } // Check against bin: directives. @@ -174,13 +174,13 @@ impl SubTest for TestBinEmit { if !enc.is_legal() { return Err(format!("{} can't be encoded: {}", inst, - func.dfg.display_inst(inst))); + func.dfg.display_inst(inst, isa))); } let have = sink.text.trim(); if have != want { return Err(format!("Bad machine code for {}: {}\nWant: {}\nGot: {}", inst, - func.dfg.display_inst(inst), + func.dfg.display_inst(inst, isa), want, have)); } diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index f620f5f316..3bc8e7b1df 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -42,7 +42,7 @@ pub fn pretty_verifier_error(func: &ir::Function, let mut msg = err.to_string(); match err.location { AnyEntity::Inst(inst) => { - write!(msg, "\n{}: {}\n\n", inst, func.dfg.display_inst(inst)).unwrap() + write!(msg, "\n{}: {}\n\n", inst, func.dfg.display_inst(inst, isa)).unwrap() } _ => msg.push('\n'), } diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 33cce8d2b0..eaa721d510 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -53,5 +53,5 @@ pub trait CodeSink { pub fn bad_encoding(func: &Function, inst: Inst) -> ! { panic!("Bad encoding {} for {}", func.encodings[inst], - func.dfg.display_inst(inst)); + func.dfg.display_inst(inst, None)); } diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index 53b7fd0112..d80d17d7bb 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -135,7 +135,7 @@ fn relax_branch(dfg: &mut DataFlowGraph, let inst = pos.current_inst().unwrap(); dbg!("Relaxing [{}] {} for {:#x}-{:#x} range", encinfo.display(encodings[inst]), - dfg.display_inst(inst), + dfg.display_inst(inst, None), offset, dest_offset); unimplemented!(); diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 85e971a9fc..ad67f7722b 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -1,6 +1,7 @@ //! Data flow graph tracking Instructions, Values, and EBBs. use entity_map::{EntityMap, PrimaryEntityData}; +use isa::TargetIsa; use ir::builder::{InsertBuilder, ReplaceBuilder}; use ir::extfunc::ExtFuncData; use ir::instructions::{Opcode, InstructionData, CallInfo}; @@ -162,7 +163,7 @@ impl DataFlowGraph { self.results[inst].get(num as usize, &self.value_lists), "Dangling result value {}: {}", v, - self.display_inst(inst)); + self.display_inst(inst, None)); ValueDef::Res(inst, num as usize) } ValueData::Arg { ebb, num, .. } => { @@ -376,8 +377,11 @@ impl DataFlowGraph { } /// Returns an object that displays `inst`. - pub fn display_inst(&self, inst: Inst) -> DisplayInst { - DisplayInst(self, inst) + pub fn display_inst<'a, I: Into>>(&'a self, + inst: Inst, + isa: I) + -> DisplayInst<'a> { + DisplayInst(self, isa.into(), inst) } /// Get all value arguments on `inst` as a slice. @@ -552,7 +556,7 @@ impl DataFlowGraph { old_value, "{} wasn't detached from {}", old_value, - self.display_inst(inst)); + self.display_inst(inst, None)); new_value } @@ -830,14 +834,15 @@ impl EbbData { } /// Object that can display an instruction. -pub struct DisplayInst<'a>(&'a DataFlowGraph, Inst); +pub struct DisplayInst<'a>(&'a DataFlowGraph, Option<&'a TargetIsa>, Inst); impl<'a> fmt::Display for DisplayInst<'a> { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { let dfg = self.0; - let inst = &dfg[self.1]; + let isa = self.1; + let inst = self.2; - if let Some((first, rest)) = dfg.inst_results(self.1).split_first() { + if let Some((first, rest)) = dfg.inst_results(inst).split_first() { write!(f, "{}", first)?; for v in rest { write!(f, ", {}", v)?; @@ -846,13 +851,13 @@ impl<'a> fmt::Display for DisplayInst<'a> { } - let typevar = dfg.ctrl_typevar(self.1); + let typevar = dfg.ctrl_typevar(inst); if typevar.is_void() { - write!(f, "{}", inst.opcode())?; + write!(f, "{}", dfg[inst].opcode())?; } else { - write!(f, "{}.{}", inst.opcode(), typevar)?; + write!(f, "{}.{}", dfg[inst].opcode(), typevar)?; } - write_operands(f, dfg, None, self.1) + write_operands(f, dfg, isa, inst) } } @@ -870,7 +875,7 @@ mod tests { let inst = dfg.make_inst(idata); dfg.make_inst_results(inst, types::I32); assert_eq!(inst.to_string(), "inst0"); - assert_eq!(dfg.display_inst(inst).to_string(), "v0 = iconst.i32"); + assert_eq!(dfg.display_inst(inst, None).to_string(), "v0 = iconst.i32"); // Immutable reference resolution. { @@ -902,7 +907,7 @@ mod tests { let idata = InstructionData::Nullary { opcode: Opcode::Trap }; let inst = dfg.make_inst(idata); - assert_eq!(dfg.display_inst(inst).to_string(), "trap"); + assert_eq!(dfg.display_inst(inst, None).to_string(), "trap"); // Result slice should be empty. assert_eq!(dfg.inst_results(inst), &[]); diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index b65eea0984..3f22694eb8 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -474,7 +474,7 @@ pub fn handle_call_abi(dfg: &mut DataFlowGraph, cfg: &ControlFlowGraph, pos: &mu debug_assert!(check_call_signature(dfg, inst).is_ok(), "Signature still wrong: {}, {}{}", - dfg.display_inst(inst), + dfg.display_inst(inst, None), sig_ref, dfg.signatures[sig_ref]); @@ -523,7 +523,7 @@ pub fn handle_return_abi(dfg: &mut DataFlowGraph, if special_args > 0 { dbg!("Adding {} special-purpose arguments to {}", special_args, - dfg.display_inst(inst)); + dfg.display_inst(inst, None)); let mut vlist = dfg[inst].take_value_list().unwrap(); for arg in &sig.return_types[abi_args..] { match arg.purpose { @@ -550,7 +550,7 @@ pub fn handle_return_abi(dfg: &mut DataFlowGraph, debug_assert!(check_return_signature(dfg, inst, sig), "Signature still wrong: {} / signature {}", - dfg.display_inst(inst), + dfg.display_inst(inst, None), sig); // Yes, we changed stuff. diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index cbca25c9c2..3e3a36f619 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -123,7 +123,7 @@ fn split_any(dfg: &mut DataFlowGraph, let branch_opc = dfg[inst].opcode(); assert!(branch_opc.is_branch(), "Predecessor not a branch: {}", - dfg.display_inst(inst)); + dfg.display_inst(inst, None)); let fixed_args = branch_opc.constraints().fixed_value_arguments(); let mut args = dfg[inst] .take_value_list() diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 634f0c0a32..08ecf12589 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -399,7 +399,7 @@ impl<'a> Context<'a> { dbg!("Checking {}: {}: {}", pred_val, pred_ebb, - self.func.dfg.display_inst(pred_inst)); + self.func.dfg.display_inst(pred_inst, self.isa)); // Never coalesce incoming function arguments on the stack. These arguments are // pre-spilled, and the rest of the virtual register would be forced to spill to the @@ -474,9 +474,9 @@ impl<'a> Context<'a> { let ty = self.func.dfg.value_type(copy); dbg!("Inserted {}, before {}: {}", - self.func.dfg.display_inst(inst), + self.func.dfg.display_inst(inst, self.isa), pred_ebb, - self.func.dfg.display_inst(pred_inst)); + self.func.dfg.display_inst(pred_inst, self.isa)); // Give it an encoding. let encoding = self.isa @@ -519,7 +519,7 @@ impl<'a> Context<'a> { self.liveness.move_def_locally(succ_val, inst); dbg!("Inserted {}, following {}({}: {})", - self.func.dfg.display_inst(inst), + self.func.dfg.display_inst(inst, self.isa), ebb, new_val, ty); diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index f61e88e9b7..85911efd0b 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -73,6 +73,7 @@ pub struct Coloring { /// Immutable context information and mutable references that don't need to be borrowed across /// method calls should go in this struct. struct Context<'a> { + isa: &'a TargetIsa, // Cached ISA information. // We save it here to avoid frequent virtual function calls on the `TargetIsa` trait object. reginfo: RegInfo, @@ -111,6 +112,7 @@ impl Coloring { tracker: &mut LiveValueTracker) { dbg!("Coloring for:\n{}", func.display(isa)); let mut ctx = Context { + isa, reginfo: isa.register_info(), encinfo: isa.encoding_info(), domtree, @@ -279,7 +281,7 @@ impl<'a> Context<'a> { locations: &mut ValueLocations, func_signature: &Signature) { dbg!("Coloring {}\n {}", - dfg.display_inst(inst), + dfg.display_inst(inst, self.isa), regs.display(&self.reginfo)); // EBB whose arguments should be colored to match the current branch instruction's @@ -308,7 +310,7 @@ impl<'a> Context<'a> { assert_eq!(dfg.inst_variable_args(inst).len(), 0, "Can't handle EBB arguments: {}", - dfg.display_inst(inst)); + dfg.display_inst(inst, self.isa)); self.undivert_regs(|lr| !lr.is_local()); } } diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 465efa7856..5733f6e80f 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -246,7 +246,9 @@ impl<'a> Context<'a> { pos: &mut Cursor, dfg: &mut DataFlowGraph, tracker: &mut LiveValueTracker) { - dbg!("Inst {}, {}", dfg.display_inst(inst), self.pressure); + dbg!("Inst {}, {}", + dfg.display_inst(inst, self.isa), + self.pressure); // We may need to resolve register constraints if there are any noteworthy uses. assert!(self.reg_uses.is_empty()); @@ -292,7 +294,7 @@ impl<'a> Context<'a> { None => { panic!("Ran out of {} registers for {}", op.regclass, - dfg.display_inst(inst)) + dfg.display_inst(inst, self.isa)) } } } @@ -429,7 +431,7 @@ impl<'a> Context<'a> { None => { panic!("Ran out of {} registers when inserting copy before {}", rc, - dfg.display_inst(inst)) + dfg.display_inst(inst, self.isa)) } } } From ad76f801271af81b4953c1bcd8af9e04a7ab4757 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Jul 2017 10:22:43 -0700 Subject: [PATCH 0862/3084] Add Intel regmove encodings. Same as a register copy, but different arguments. --- lib/cretonne/meta/isa/intel/encodings.py | 6 +++++- lib/cretonne/meta/isa/intel/recipes.py | 9 +++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 74dd5b5999..ab32e77303 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -23,11 +23,15 @@ for inst, opc in [ I64.enc(inst.i32, *r.rr(opc)) I32.enc(base.copy.i32, *r.ur(0x89)) - I64.enc(base.copy.i64, *r.ur.rex(0x89, w=1)) I64.enc(base.copy.i32, *r.ur.rex(0x89)) I64.enc(base.copy.i32, *r.ur(0x89)) +I32.enc(base.regmove.i32, *r.rmov(0x89)) +I64.enc(base.regmove.i64, *r.rmov.rex(0x89, w=1)) +I64.enc(base.regmove.i32, *r.rmov.rex(0x89)) +I64.enc(base.regmove.i32, *r.rmov(0x89)) + # Immediate instructions with sign-extended 8-bit and 32-bit immediate. for inst, rrr in [ (base.iadd_imm, 0), diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 2d1fc62d80..c6c7b3f2af 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -6,6 +6,7 @@ from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt, IsEqual from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry from base.formats import Call, IndirectCall, Store, Load +from base.formats import RegMove from .registers import GPR, ABCD try: @@ -205,6 +206,14 @@ ur = TailRecipe( modrm_rr(out_reg0, in_reg0, sink); ''') +# XX /r, for regmove instructions. +rmov = TailRecipe( + 'ur', RegMove, size=1, ins=GPR, outs=(), + emit=''' + PUT_OP(bits, rex2(dst, src), sink); + modrm_rr(dst, src, sink); + ''') + # XX /n with one arg in %rcx, for shifts. rc = TailRecipe( 'rc', Binary, size=1, ins=(GPR, GPR.rcx), outs=0, From ca99bd1641e89dc87fd3c382ff9e048c8e7f57b9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Jul 2017 10:43:13 -0700 Subject: [PATCH 0863/3084] Add RISC-V regmove encodings. --- lib/cretonne/meta/isa/riscv/encodings.py | 6 +++++- lib/cretonne/meta/isa/riscv/recipes.py | 7 ++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index d8afbab2cf..9ec6b34fc0 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -8,7 +8,7 @@ from .defs import RV32, RV64 from .recipes import OPIMM, OPIMM32, OP, OP32, LUI, BRANCH, JALR, JAL from .recipes import LOAD, STORE from .recipes import R, Rshamt, Ricmp, I, Iz, Iicmp, Iret, Icall, Icopy -from .recipes import U, UJ, UJcall, SB, SBzero, GPsp, GPfi +from .recipes import U, UJ, UJcall, SB, SBzero, GPsp, GPfi, Irmov from .settings import use_m from cdsl.ast import Var @@ -135,3 +135,7 @@ RV64.enc(base.fill.i64, GPfi, LOAD(0b011)) RV32.enc(base.copy.i32, Icopy, OPIMM(0b000)) RV64.enc(base.copy.i64, Icopy, OPIMM(0b000)) RV64.enc(base.copy.i32, Icopy, OPIMM32(0b000)) + +RV32.enc(base.regmove.i32, Irmov, OPIMM(0b000)) +RV64.enc(base.regmove.i64, Irmov, OPIMM(0b000)) +RV64.enc(base.regmove.i32, Irmov, OPIMM32(0b000)) diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 510e13c860..afffb2c0aa 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -14,7 +14,7 @@ from cdsl.predicates import IsSignedInt from cdsl.registers import Stack from base.formats import Binary, BinaryImm, MultiAry, IntCompare, IntCompareImm from base.formats import Unary, UnaryImm, BranchIcmp, Branch, Jump -from base.formats import Call, IndirectCall +from base.formats import Call, IndirectCall, RegMove from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit @@ -156,6 +156,11 @@ Icopy = EncRecipe( 'Icopy', Unary, size=4, ins=GPR, outs=GPR, emit='put_i(bits, in_reg0, 0, out_reg0, sink);') +# Same for a GPR regmove. +Irmov = EncRecipe( + 'Irmov', RegMove, size=4, ins=GPR, outs=(), + emit='put_i(bits, src, 0, dst, sink);') + # U-type instructions have a 20-bit immediate that targets bits 12-31. U = EncRecipe( 'U', UnaryImm, size=4, ins=(), outs=GPR, From 0917cfe7f469fe7a278262543114b56fd8cc2e55 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Jul 2017 10:37:50 -0700 Subject: [PATCH 0864/3084] Attach encodings to regmove instructions generated during coloring. All emitted regmove instructions must be materialized as real move instructions. --- lib/cretonne/src/regalloc/coloring.rs | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 85911efd0b..21a4453bb2 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -43,7 +43,8 @@ //! The exception is the entry block whose arguments are colored from the ABI requirements. use dominator_tree::DominatorTree; -use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph, Layout, ValueLocations}; +use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph, Layout}; +use ir::{InstEncodings, ValueLocations}; use ir::{InstBuilder, Signature, ArgumentType, ArgumentLoc}; use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; use isa::{TargetIsa, EncInfo, RecipeConstraints, OperandConstraint, ConstraintKind}; @@ -156,6 +157,7 @@ impl<'a> Context<'a> { tracker, &mut regs, &mut func.locations, + &mut func.encodings, &func.signature); } else { let (_throughs, kills) = tracker.process_ghost(inst); @@ -279,6 +281,7 @@ impl<'a> Context<'a> { tracker: &mut LiveValueTracker, regs: &mut AllocatableSet, locations: &mut ValueLocations, + encodings: &mut InstEncodings, func_signature: &Signature) { dbg!("Coloring {}\n {}", dfg.display_inst(inst, self.isa), @@ -354,7 +357,7 @@ impl<'a> Context<'a> { // The solution and/or fixed input constraints may require us to shuffle the set of live // registers around. - self.shuffle_inputs(pos, dfg, regs); + self.shuffle_inputs(pos, dfg, regs, encodings); // If this is the first time we branch to `dest`, color its arguments to match the current // register state. @@ -695,12 +698,18 @@ impl<'a> Context<'a> { fn shuffle_inputs(&mut self, pos: &mut Cursor, dfg: &mut DataFlowGraph, - regs: &mut AllocatableSet) { + regs: &mut AllocatableSet, + encodings: &mut InstEncodings) { self.solver.schedule_moves(regs); for m in self.solver.moves() { + let ty = dfg.value_type(m.value); self.divert.regmove(m.value, m.from, m.to); - dfg.ins(pos).regmove(m.value, m.from, m.to); + let inst = dfg.ins(pos).regmove(m.value, m.from, m.to); + match self.isa.encode(dfg, &dfg[inst], ty) { + Ok(encoding) => *encodings.ensure(inst) = encoding, + _ => panic!("Can't encode {} {}", m.rc, dfg.display_inst(inst, self.isa)), + } } } From d56d4d171e2c1a0c0ce78e4fc118f22d9e3332ac Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Jul 2017 09:49:02 -0700 Subject: [PATCH 0865/3084] Tag the regmove instruction with other_side_effects. This instruction moves a value between registers. This counts as a side effect that is not tracked by the SSA data flow graph. --- lib/cretonne/meta/base/instructions.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 109bc73f9e..e1b38086e4 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -484,7 +484,8 @@ regmove = Instruction( before the value leaves the EBB. At the entry to a new EBB, all live values must be in their originally assigned registers. """, - ins=(x, src, dst)) + ins=(x, src, dst), + other_side_effects=True) # # Vector operations From ae5e440094fed412dd01e77be074d5a837bca6cf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Jul 2017 11:17:59 -0700 Subject: [PATCH 0866/3084] Fix Vim syntax highlighting of numbers. Cretonne allows '_' in number constants. --- misc/vim/syntax/cton.vim | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/misc/vim/syntax/cton.vim b/misc/vim/syntax/cton.vim index 003029e1bb..d345742154 100644 --- a/misc/vim/syntax/cton.vim +++ b/misc/vim/syntax/cton.vim @@ -22,8 +22,8 @@ syn match ctonEntity /\<\(v\|ss\|jt\|fn\|sig\)\d\+\>/ syn match ctonLabel /\/ syn match ctonName /%\w\+\>/ -syn match ctonNumber /-\?\<\d\+\>/ -syn match ctonNumber /-\?\<0x\x\+\(\.\x*\)\(p[+-]\?\d\+\)\?\>/ +syn match ctonNumber /-\?\<[0-9_]\+\>/ +syn match ctonNumber /-\?\<0x[0-9a-fA-F_]\+\(\.[0-9a-fA-F_]*\)\?\(p[+-]\?\d\+\)\?\>/ syn match ctonHexSeq /#\x\+\>/ syn region ctonCommentLine start=";" end="$" contains=ctonFilecheck From 9e3b6a6eba83e0690dd5ee097e7eca5d5c4b3a81 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 11 Jul 2017 16:26:16 -0700 Subject: [PATCH 0867/3084] Add a Context::compile() function which runs all compiler passes. This is the main entry point to the code generator. It returns the computed size of the functions code. Also add a 'test compile' command which runs the whole code generation pipeline. --- cranelift/src/filetest/binemit.rs | 9 ++- cranelift/src/filetest/compile.rs | 98 ++++++++++++++++++++++++++ cranelift/src/filetest/mod.rs | 8 ++- lib/cretonne/src/binemit/relaxation.rs | 9 ++- lib/cretonne/src/context.rs | 27 ++++++- 5 files changed, 143 insertions(+), 8 deletions(-) create mode 100644 cranelift/src/filetest/compile.rs diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 15f5b0a982..d359cea794 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -12,7 +12,7 @@ use cretonne::ir::entities::AnyEntity; use cretonne::isa::TargetIsa; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result}; -use utils::match_directive; +use utils::{match_directive, pretty_error}; struct TestBinEmit; @@ -117,7 +117,8 @@ impl SubTest for TestBinEmit { } // Relax branches and compute EBB offsets based on the encodings. - binemit::relax_branches(&mut func, isa); + let code_size = binemit::relax_branches(&mut func, isa) + .map_err(|e| pretty_error(&func, context.isa, e))?; // Collect all of the 'bin:' directives on instructions. let mut bins = HashMap::new(); @@ -188,6 +189,10 @@ impl SubTest for TestBinEmit { } } + if sink.offset != code_size { + return Err(format!("Expected code size {}, got {}", code_size, sink.offset)); + } + Ok(()) } } diff --git a/cranelift/src/filetest/compile.rs b/cranelift/src/filetest/compile.rs new file mode 100644 index 0000000000..15db119efd --- /dev/null +++ b/cranelift/src/filetest/compile.rs @@ -0,0 +1,98 @@ +//! Test command for testing the code generator pipeline +//! +//! The `compile` test command runs each function through the full code generator pipeline + +use cretonne::binemit; +use cretonne::ir; +use cretonne; +use cton_reader::TestCommand; +use filetest::subtest::{SubTest, Context, Result}; +use std::borrow::Cow; +use utils::pretty_error; + +struct TestCompile; + +pub fn subtest(parsed: &TestCommand) -> Result> { + assert_eq!(parsed.command, "compile"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestCompile)) + } +} + +impl SubTest for TestCompile { + fn name(&self) -> Cow { + Cow::from("compile") + } + + fn is_mutating(&self) -> bool { + true + } + + fn needs_isa(&self) -> bool { + true + } + + fn run(&self, func: Cow, context: &Context) -> Result<()> { + let isa = context.isa.expect("compile needs an ISA"); + + // Create a compilation context, and drop in the function. + let mut comp_ctx = cretonne::Context::new(); + comp_ctx.func = func.into_owned(); + + let code_size = comp_ctx + .compile(isa) + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; + + dbg!("Generated {} bytes of code:\n{}", + code_size, + comp_ctx.func.display(isa)); + + // Finally verify that the returned code size matches the emitted bytes. + let mut sink = SizeSink { offset: 0 }; + for ebb in comp_ctx.func.layout.ebbs() { + assert_eq!(sink.offset, comp_ctx.func.offsets[ebb]); + for inst in comp_ctx.func.layout.ebb_insts(ebb) { + isa.emit_inst(&comp_ctx.func, inst, &mut sink); + } + } + + if sink.offset != code_size { + return Err(format!("Expected code size {}, got {}", code_size, sink.offset)); + } + + Ok(()) + } +} + +// Code sink that simply counts bytes. +struct SizeSink { + offset: binemit::CodeOffset, +} + +impl binemit::CodeSink for SizeSink { + fn offset(&self) -> binemit::CodeOffset { + self.offset + } + + fn put1(&mut self, _: u8) { + self.offset += 1; + } + + fn put2(&mut self, _: u16) { + self.offset += 2; + } + + fn put4(&mut self, _: u32) { + self.offset += 4; + } + + fn put8(&mut self, _: u64) { + self.offset += 8; + } + + fn reloc_ebb(&mut self, _reloc: binemit::Reloc, _ebb: ir::Ebb) {} + fn reloc_func(&mut self, _reloc: binemit::Reloc, _fref: ir::FuncRef) {} + fn reloc_jt(&mut self, _reloc: binemit::Reloc, _jt: ir::JumpTable) {} +} diff --git a/cranelift/src/filetest/mod.rs b/cranelift/src/filetest/mod.rs index 9c03cc6d4a..2d2a9c6cac 100644 --- a/cranelift/src/filetest/mod.rs +++ b/cranelift/src/filetest/mod.rs @@ -14,6 +14,7 @@ use filetest::runner::TestRunner; pub mod subtest; mod binemit; +mod compile; mod concurrent; mod domtree; mod legalizer; @@ -57,15 +58,16 @@ pub fn run(verbose: bool, files: Vec) -> CommandResult { /// a `.cton` test file. fn new_subtest(parsed: &TestCommand) -> subtest::Result> { match parsed.command { + "binemit" => binemit::subtest(parsed), "cat" => cat::subtest(parsed), - "print-cfg" => print_cfg::subtest(parsed), + "compile" => compile::subtest(parsed), "domtree" => domtree::subtest(parsed), - "verifier" => verifier::subtest(parsed), "legalizer" => legalizer::subtest(parsed), "licm" => licm::subtest(parsed), + "print-cfg" => print_cfg::subtest(parsed), "regalloc" => regalloc::subtest(parsed), - "binemit" => binemit::subtest(parsed), "simple-gvn" => simple_gvn::subtest(parsed), + "verifier" => verifier::subtest(parsed), _ => Err(format!("unknown test command '{}'", parsed.command)), } } diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index d80d17d7bb..3420134d1b 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -31,11 +31,12 @@ use binemit::CodeOffset; use ir::{Function, DataFlowGraph, Cursor, InstructionData, Opcode, InstEncodings}; use isa::{TargetIsa, EncInfo}; use iterators::IteratorExtras; +use result::CtonError; /// Relax branches and compute the final layout of EBB headers in `func`. /// /// Fill in the `func.offsets` table so the function is ready for binary emission. -pub fn relax_branches(func: &mut Function, isa: &TargetIsa) { +pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> Result { let encinfo = isa.encoding_info(); // Clear all offsets so we can recognize EBBs that haven't been visited yet. @@ -45,13 +46,15 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) { // Start by inserting fall through instructions. fallthroughs(func); + let mut offset = 0; + // The relaxation algorithm iterates to convergence. let mut go_again = true; while go_again { go_again = false; + offset = 0; // Visit all instructions in layout order - let mut offset = 0; let mut pos = Cursor::new(&mut func.layout); while let Some(ebb) = pos.next_ebb() { // Record the offset for `ebb` and make sure we iterate until offsets are stable. @@ -90,6 +93,8 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) { } } } + + Ok(offset) } /// Convert `jump` instructions to `fallthrough` instructions where possible and verify that any diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 5e23ed3efc..a87e35cd32 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -9,6 +9,7 @@ //! contexts concurrently. Typically, you would have one context per compilation thread and only a //! single ISA instance. +use binemit::{CodeOffset, relax_branches}; use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir::Function; @@ -16,7 +17,7 @@ use loop_analysis::LoopAnalysis; use isa::TargetIsa; use legalize_function; use regalloc; -use result::CtonResult; +use result::{CtonError, CtonResult}; use verifier; use simple_gvn::do_simple_gvn; use licm::do_licm; @@ -54,6 +55,22 @@ impl Context { } } + /// Compile the function. + /// + /// Run the function through all the passes necessary to generate code for the target ISA + /// represented by `isa`. This does not include the final step of emitting machine code into a + /// code sink. + /// + /// Returns the size of the function's code. + pub fn compile(&mut self, isa: &TargetIsa) -> Result { + self.flowgraph(); + self.verify_if(isa)?; + + self.legalize(isa)?; + self.regalloc(isa)?; + self.relax_branches(isa) + } + /// Run the verifier on the function. /// /// Also check that the dominator tree and control flow graph are consistent with the function. @@ -107,4 +124,12 @@ impl Context { self.regalloc .run(isa, &mut self.func, &self.cfg, &self.domtree) } + + /// Run the branch relaxation pass and return the final code size. + pub fn relax_branches(&mut self, isa: &TargetIsa) -> Result { + let code_size = relax_branches(&mut self.func, isa)?; + self.verify_if(isa)?; + + Ok(code_size) + } } From 1f52415b17fcc7fcc27315d8ee02d0f70deac943 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 11 Jul 2017 17:42:21 -0700 Subject: [PATCH 0868/3084] Add a WebAssembly filetests directory. Start adding little 'test compile' test cases which check that the full compilation pipeline works for each WebAssembly instruction. --- cranelift/filetests/wasm/i32-arith.cton | 62 +++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 cranelift/filetests/wasm/i32-arith.cton diff --git a/cranelift/filetests/wasm/i32-arith.cton b/cranelift/filetests/wasm/i32-arith.cton new file mode 100644 index 0000000000..1096d70ddb --- /dev/null +++ b/cranelift/filetests/wasm/i32-arith.cton @@ -0,0 +1,62 @@ +; Test basic code generation for i32 arithmetic WebAssembly instructions. +test compile + +set is_64bit +isa intel + +; Constants. + +function %i32_const() -> i32 { +ebb0: + v0 = iconst.i32 0x8765_4321 + return v0 +} + +; Unary operations. + +; function %i32_clz(i32) -> i32 +; function %i32_ctz(i32) -> i32 +; function %i32_popcnt(i32) -> i32 + +; Binary operations. + +function %i32_add(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = iadd v0, v1 + return v2 +} + +function %i32_sub(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = isub v0, v1 + return v2 +} + +; function %i32_mul(i32, i32) -> i32 +; function %i32_div(i32, i32) -> i32 +; function %i32_rem_s(i32, i32) -> i32 +; function %i32_rem_u(i32, i32) -> i32 + +function %i32_and(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = band v0, v1 + return v2 +} + +function %i32_or(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = bor v0, v1 + return v2 +} + +function %i32_xor(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = bxor v0, v1 + return v2 +} + +; function %i32_shl(i32, i32) -> i32 +; function %i32_shr_s(i32, i32) -> i32 +; function %i32_shr_u(i32, i32) -> i32 +; function %i32_rotl(i32, i32) -> i32 +; function %i32_rotr(i32, i32) -> i32 From f57c666d8a0063c532a74c7128f13dbb8f54d69e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Jul 2017 12:53:41 -0700 Subject: [PATCH 0869/3084] Add Intel encodings for shift and rotate instructions. --- cranelift/filetests/isa/intel/binary32.cton | 20 ++++-- cranelift/filetests/isa/intel/binary64.cton | 70 +++++++++++++++++---- cranelift/filetests/wasm/i32-arith.cton | 39 ++++++++++-- lib/cretonne/meta/isa/intel/encodings.py | 15 +++-- 4 files changed, 117 insertions(+), 27 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 14f050c2e7..4cc14b6905 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -55,11 +55,14 @@ ebb0: [-,%rsi] v24 = sshr v2, v1 ; bin: d3 fe ; asm: sarl %cl, %ecx [-,%rcx] v25 = sshr v1, v1 ; bin: d3 f9 - - ; asm: movl %esi, %ecx - [-,%rcx] v26 = copy v2 ; bin: 89 f1 - ; asm: movl %ecx, %esi - [-,%rsi] v27 = copy v1 ; bin: 89 ce + ; asm: roll %cl, %esi + [-,%rsi] v26 = rotl v2, v1 ; bin: d3 c6 + ; asm: roll %cl, %ecx + [-,%rcx] v27 = rotl v1, v1 ; bin: d3 c1 + ; asm: rorl %cl, %esi + [-,%rsi] v28 = rotr v2, v1 ; bin: d3 ce + ; asm: rorl %cl, %ecx + [-,%rcx] v29 = rotr v1, v1 ; bin: d3 c9 ; Integer Register - Immediate 8-bit operations. ; The 8-bit immediate is sign-extended. @@ -102,6 +105,13 @@ ebb0: ; asm: xorl $1000000, %esi [-,%rsi] v47 = bxor_imm v2, 1000000 ; bin: 81 f6 000f4240 + ; Register copies. + + ; asm: movl %esi, %ecx + [-,%rcx] v80 = copy v2 ; bin: 89 f1 + ; asm: movl %ecx, %esi + [-,%rsi] v81 = copy v1 ; bin: 89 ce + ; Load/Store instructions. ; Register indirect addressing with no displacement. diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 48e328b455..e9ee5d3941 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -65,13 +65,27 @@ ebb0: ; asm: xorq %rcx, %r10 [-,%r10] v52 = bxor v3, v1 ; bin: 49 31 ca - ; asm: movq %rsi, %rcx - [-,%rcx] v60 = copy v2 ; bin: 48 89 f1 - ; asm: movq %r10, %rsi - [-,%rsi] v61 = copy v3 ; bin: 4c 89 d6 - ; asm: movq %rcx, %r10 - [-,%r10] v62 = copy v1 ; bin: 49 89 ca + ; asm: shlq %cl, %rsi + [-,%rsi] v60 = ishl v2, v1 ; bin: 48 d3 e6 + ; asm: shlq %cl, %r10 + [-,%r10] v61 = ishl v3, v1 ; bin: 49 d3 e2 + ; asm: sarq %cl, %rsi + [-,%rsi] v62 = sshr v2, v1 ; bin: 48 d3 fe + ; asm: sarq %cl, %r10 + [-,%r10] v63 = sshr v3, v1 ; bin: 49 d3 fa + ; asm: shrq %cl, %rsi + [-,%rsi] v64 = ushr v2, v1 ; bin: 48 d3 ee + ; asm: shrq %cl, %r10 + [-,%r10] v65 = ushr v3, v1 ; bin: 49 d3 ea + ; asm: rolq %cl, %rsi + [-,%rsi] v66 = rotl v2, v1 ; bin: 48 d3 c6 + ; asm: rolq %cl, %r10 + [-,%r10] v67 = rotl v3, v1 ; bin: 49 d3 c2 + ; asm: rorq %cl, %rsi + [-,%rsi] v68 = rotr v2, v1 ; bin: 48 d3 ce + ; asm: rorq %cl, %r10 + [-,%r10] v69 = rotr v3, v1 ; bin: 49 d3 ca ; Integer Register-Immediate Operations. ; These 64-bit ops all use a 32-bit immediate that is sign-extended to 64 bits. @@ -122,6 +136,15 @@ ebb0: ; asm: xorq $-100, %r14 [-,%r14] v104 = bxor_imm v5, -100 ; bin: 49 83 f6 9c + ; Register copies. + + ; asm: movq %rsi, %rcx + [-,%rcx] v110 = copy v2 ; bin: 48 89 f1 + ; asm: movq %r10, %rsi + [-,%rsi] v111 = copy v3 ; bin: 4c 89 d6 + ; asm: movq %rcx, %r10 + [-,%r10] v112 = copy v1 ; bin: 49 89 ca + return ; bin: c3 } @@ -187,13 +210,27 @@ ebb0: ; asm: xorl %ecx, %r10d [-,%r10] v52 = bxor v3, v1 ; bin: 41 31 ca - ; asm: movl %esi, %ecx - [-,%rcx] v60 = copy v2 ; bin: 40 89 f1 - ; asm: movl %r10d, %esi - [-,%rsi] v61 = copy v3 ; bin: 44 89 d6 - ; asm: movl %ecx, %r10d - [-,%r10] v62 = copy v1 ; bin: 41 89 ca + ; asm: shll %cl, %esi + [-,%rsi] v60 = ishl v2, v1 ; bin: 40 d3 e6 + ; asm: shll %cl, %r10d + [-,%r10] v61 = ishl v3, v1 ; bin: 41 d3 e2 + ; asm: sarl %cl, %esi + [-,%rsi] v62 = sshr v2, v1 ; bin: 40 d3 fe + ; asm: sarl %cl, %r10d + [-,%r10] v63 = sshr v3, v1 ; bin: 41 d3 fa + ; asm: shrl %cl, %esi + [-,%rsi] v64 = ushr v2, v1 ; bin: 40 d3 ee + ; asm: shrl %cl, %r10d + [-,%r10] v65 = ushr v3, v1 ; bin: 41 d3 ea + ; asm: roll %cl, %esi + [-,%rsi] v66 = rotl v2, v1 ; bin: 40 d3 c6 + ; asm: roll %cl, %r10d + [-,%r10] v67 = rotl v3, v1 ; bin: 41 d3 c2 + ; asm: rorl %cl, %esi + [-,%rsi] v68 = rotr v2, v1 ; bin: 40 d3 ce + ; asm: rorl %cl, %r10d + [-,%r10] v69 = rotr v3, v1 ; bin: 41 d3 ca ; Integer Register-Immediate Operations. ; These 64-bit ops all use a 32-bit immediate that is sign-extended to 64 bits. @@ -244,5 +281,14 @@ ebb0: ; asm: xorl $-100, %r14d [-,%r14] v104 = bxor_imm v5, -100 ; bin: 41 83 f6 9c + ; Register copies. + + ; asm: movl %esi, %ecx + [-,%rcx] v110 = copy v2 ; bin: 40 89 f1 + ; asm: movl %r10d, %esi + [-,%rsi] v111 = copy v3 ; bin: 44 89 d6 + ; asm: movl %ecx, %r10d + [-,%r10] v112 = copy v1 ; bin: 41 89 ca + return ; bin: c3 } diff --git a/cranelift/filetests/wasm/i32-arith.cton b/cranelift/filetests/wasm/i32-arith.cton index 1096d70ddb..f2fafffee3 100644 --- a/cranelift/filetests/wasm/i32-arith.cton +++ b/cranelift/filetests/wasm/i32-arith.cton @@ -1,7 +1,10 @@ ; Test basic code generation for i32 arithmetic WebAssembly instructions. test compile -set is_64bit +set is_64bit=0 +isa intel + +set is_64bit=1 isa intel ; Constants. @@ -55,8 +58,32 @@ ebb0(v0: i32, v1: i32): return v2 } -; function %i32_shl(i32, i32) -> i32 -; function %i32_shr_s(i32, i32) -> i32 -; function %i32_shr_u(i32, i32) -> i32 -; function %i32_rotl(i32, i32) -> i32 -; function %i32_rotr(i32, i32) -> i32 +function %i32_shl(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = ishl v0, v1 + return v2 +} + +function %i32_shr_s(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = sshr v0, v1 + return v2 +} + +function %i32_shr_u(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = ushr v0, v1 + return v2 +} + +function %i32_rotl(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = rotl v0, v1 + return v2 +} + +function %i32_rotr(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = rotr v0, v1 + return v2 +} diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index ab32e77303..df6631fe4c 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -66,12 +66,19 @@ I64.enc(base.iconst.i64, *r.uid.rex(0xc7, rrr=0, w=1)) # Finally, the 0xb8 opcode takes an 8-byte immediate with a REX.W prefix. I64.enc(base.iconst.i64, *r.puiq.rex(0xb8, w=1)) -# 32-bit shifts and rotates. +# Shifts and rotates. # Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit # and 16-bit shifts would need explicit masking. -I32.enc(base.ishl.i32.i32, *r.rc(0xd3, rrr=4)) -I32.enc(base.ushr.i32.i32, *r.rc(0xd3, rrr=5)) -I32.enc(base.sshr.i32.i32, *r.rc(0xd3, rrr=7)) +for inst, rrr in [ + (base.rotl, 0), + (base.rotr, 1), + (base.ishl, 4), + (base.ushr, 5), + (base.sshr, 7)]: + I32.enc(inst.i32.i32, *r.rc(0xd3, rrr=rrr)) + I64.enc(inst.i64.i64, *r.rc.rex(0xd3, rrr=rrr, w=1)) + I64.enc(inst.i32.i32, *r.rc.rex(0xd3, rrr=rrr)) + I64.enc(inst.i32.i32, *r.rc(0xd3, rrr=rrr)) # Loads and stores. I32.enc(base.store.i32.i32, *r.st(0x89)) From 435a15b88dae7ce4f2a4634a50bff46a3558f41e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Jul 2017 14:14:08 -0700 Subject: [PATCH 0870/3084] Add Intel encodings for popcnt. Change the result type for the bit-counting instructions from a fixed i8 to the iB type variable which is the type of the input. This matches the convention in WebAssembly, and at least Intel's instructions will set a full register's worth of count result, even if it is always < 64. Duplicate the Intel 'ur' encoding recipe into 'umr' and 'urm' variants corresponding to the RM and MR encoding variants. The difference is which register is encoded as 'reg' and which is 'r/m' in the ModR/M byte. A 'mov' register copy uses the MR variant, a unary popcnt uses the RM variant. --- cranelift/filetests/isa/intel/binary32.cton | 7 ++++++ cranelift/filetests/isa/intel/binary64.cton | 18 +++++++++++++ cranelift/filetests/wasm/i32-arith.cton | 7 +++++- lib/cretonne/meta/base/instructions.py | 4 +-- lib/cretonne/meta/isa/intel/encodings.py | 14 ++++++++--- lib/cretonne/meta/isa/intel/recipes.py | 15 ++++++++--- lib/cretonne/src/isa/intel/binemit.rs | 28 ++++++++++++++++++--- 7 files changed, 79 insertions(+), 14 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 4cc14b6905..77e53ceeec 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -217,6 +217,13 @@ ebb0: ; asm: movsbl -50000(%esi), %edx [-,%rdx] v129 = sload8.i32 v2-50000 ; bin: 0f be 96 ffff3cb0 + ; Bit-counting instructions. + + ; asm: popcntl %esi, %ecx + [-,%rcx] v200 = popcnt v2 ; bin: f3 0f b8 ce + ; asm: popcntl %ecx, %esi + [-,%rsi] v201 = popcnt v1 ; bin: f3 0f b8 f1 + ; asm: call foo call fn0() ; bin: e8 PCRel4(fn0) 00000000 diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index e9ee5d3941..da350d1e45 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -145,6 +145,15 @@ ebb0: ; asm: movq %rcx, %r10 [-,%r10] v112 = copy v1 ; bin: 49 89 ca + ; Bit-counting instructions. + + ; asm: popcntq %rsi, %rcx + [-,%rcx] v200 = popcnt v2 ; bin: f3 48 0f b8 ce + ; asm: popcntq %r10, %rsi + [-,%rsi] v201 = popcnt v3 ; bin: f3 49 0f b8 f2 + ; asm: popcntq %rcx, %r10 + [-,%r10] v202 = popcnt v1 ; bin: f3 4c 0f b8 d1 + return ; bin: c3 } @@ -290,5 +299,14 @@ ebb0: ; asm: movl %ecx, %r10d [-,%r10] v112 = copy v1 ; bin: 41 89 ca + ; Bit-counting instructions. + + ; asm: popcntl %esi, %ecx + [-,%rcx] v200 = popcnt v2 ; bin: f3 40 0f b8 ce + ; asm: popcntl %r10d, %esi + [-,%rsi] v201 = popcnt v3 ; bin: f3 41 0f b8 f2 + ; asm: popcntl %ecx, %r10d + [-,%r10] v202 = popcnt v1 ; bin: f3 44 0f b8 d1 + return ; bin: c3 } diff --git a/cranelift/filetests/wasm/i32-arith.cton b/cranelift/filetests/wasm/i32-arith.cton index f2fafffee3..fc730cbe23 100644 --- a/cranelift/filetests/wasm/i32-arith.cton +++ b/cranelift/filetests/wasm/i32-arith.cton @@ -19,7 +19,12 @@ ebb0: ; function %i32_clz(i32) -> i32 ; function %i32_ctz(i32) -> i32 -; function %i32_popcnt(i32) -> i32 + +function %i32_popcnt(i32) -> i32 { +ebb0(v0: i32): + v1 = popcnt v0 + return v1 +} ; Binary operations. diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index e1b38086e4..2aa9027a45 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -8,7 +8,7 @@ from __future__ import absolute_import from cdsl.operands import Operand, VARIABLE_ARGS from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup -from base.types import i8, f32, f64, b1 +from base.types import f32, f64, b1 from base.immediates import imm64, uimm8, ieee32, ieee64, offset32, uoffset32 from base.immediates import intcc, floatcc, memflags, regunit from base import entities @@ -1050,7 +1050,7 @@ sshr_imm = Instruction( # x = Operand('x', iB) -a = Operand('a', i8) +a = Operand('a', iB) clz = Instruction( 'clz', r""" diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index df6631fe4c..d8dfe53140 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -22,10 +22,10 @@ for inst, opc in [ # default. Otherwise reg-alloc would never use r8 and up. I64.enc(inst.i32, *r.rr(opc)) -I32.enc(base.copy.i32, *r.ur(0x89)) -I64.enc(base.copy.i64, *r.ur.rex(0x89, w=1)) -I64.enc(base.copy.i32, *r.ur.rex(0x89)) -I64.enc(base.copy.i32, *r.ur(0x89)) +I32.enc(base.copy.i32, *r.umr(0x89)) +I64.enc(base.copy.i64, *r.umr.rex(0x89, w=1)) +I64.enc(base.copy.i32, *r.umr.rex(0x89)) +I64.enc(base.copy.i32, *r.umr(0x89)) I32.enc(base.regmove.i32, *r.rmov(0x89)) I64.enc(base.regmove.i64, *r.rmov.rex(0x89, w=1)) @@ -80,6 +80,12 @@ for inst, rrr in [ I64.enc(inst.i32.i32, *r.rc.rex(0xd3, rrr=rrr)) I64.enc(inst.i32.i32, *r.rc(0xd3, rrr=rrr)) +# Population count. +I32.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8)) +I64.enc(base.popcnt.i64, *r.urm.rex(0xf3, 0x0f, 0xb8, w=1)) +I64.enc(base.popcnt.i32, *r.urm.rex(0xf3, 0x0f, 0xb8)) +I64.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8)) + # Loads and stores. I32.enc(base.store.i32.i32, *r.st(0x89)) I32.enc(base.store.i32.i32, *r.stDisp8(0x89)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index c6c7b3f2af..b7e6aa5575 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -198,14 +198,23 @@ rr = TailRecipe( ''') # XX /r, but for a unary operator with separate input/output register, like -# copies. -ur = TailRecipe( - 'ur', Unary, size=1, ins=GPR, outs=GPR, +# copies. MR form. +umr = TailRecipe( + 'umr', Unary, size=1, ins=GPR, outs=GPR, emit=''' PUT_OP(bits, rex2(out_reg0, in_reg0), sink); modrm_rr(out_reg0, in_reg0, sink); ''') +# XX /r, but for a unary operator with separate input/output register. +# RM form. +urm = TailRecipe( + 'urm', Unary, size=1, ins=GPR, outs=GPR, + emit=''' + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); + modrm_rr(in_reg0, out_reg0, sink); + ''') + # XX /r, for regmove instructions. rmov = TailRecipe( 'ur', RegMove, size=1, ins=GPR, outs=(), diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 9db0ee1234..56f0c2f7f0 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -57,7 +57,7 @@ fn rex_prefix(bits: u16, rex: u8, sink: &mut CS) { // Emit a single-byte opcode with no REX prefix. fn put_op1(bits: u16, rex: u8, sink: &mut CS) { debug_assert_eq!(bits & 0x8f00, 0, "Invalid encoding bits for Op1*"); - debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less encoding"); + debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less Op1 encoding"); sink.put1(bits as u8); } @@ -71,17 +71,37 @@ fn put_rexop1(bits: u16, rex: u8, sink: &mut CS) { // Emit two-byte opcode: 0F XX fn put_op2(bits: u16, rex: u8, sink: &mut CS) { debug_assert_eq!(bits & 0x8f00, 0x0400, "Invalid encoding bits for Op2*"); - debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less encoding"); + debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less Op2 encoding"); sink.put1(0x0f); sink.put1(bits as u8); } // Emit single-byte opcode with mandatory prefix. fn put_mp1(bits: u16, rex: u8, sink: &mut CS) { - debug_assert_eq!(bits & 0x0c00, 0, "Invalid encoding bits for Mp1*"); + debug_assert_eq!(bits & 0x8c00, 0, "Invalid encoding bits for Mp1*"); let pp = (bits >> 8) & 3; sink.put1(PREFIX[(pp - 1) as usize]); - debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less encoding"); + debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less Mp1 encoding"); + sink.put1(bits as u8); +} + +// Emit two-byte opcode (0F XX) with mandatory prefix. +fn put_mp2(bits: u16, rex: u8, sink: &mut CS) { + debug_assert_eq!(bits & 0x8c00, 0x0400, "Invalid encoding bits for Mp2*"); + let pp = (bits >> 8) & 3; + sink.put1(PREFIX[(pp - 1) as usize]); + debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less Mp2 encoding"); + sink.put1(0x0f); + sink.put1(bits as u8); +} + +// Emit two-byte opcode (0F XX) with mandatory prefix and REX. +fn put_rexmp2(bits: u16, rex: u8, sink: &mut CS) { + debug_assert_eq!(bits & 0x0c00, 0x0400, "Invalid encoding bits for Mp2*"); + let pp = (bits >> 8) & 3; + sink.put1(PREFIX[(pp - 1) as usize]); + rex_prefix(bits, rex, sink); + sink.put1(0x0f); sink.put1(bits as u8); } From 5cbcd59cf0e4ef63efdc63903d38e3d50bf2c384 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Jul 2017 16:05:20 -0700 Subject: [PATCH 0871/3084] Add some ISA predicates for Intel CPUID features. Guard the popcnt instruction on the proper CPUID bits. --- cranelift/filetests/isa/intel/binary32.cton | 2 +- cranelift/filetests/isa/intel/binary64.cton | 2 +- cranelift/filetests/wasm/i32-arith.cton | 4 +-- lib/cretonne/meta/isa/intel/encodings.py | 10 ++++--- lib/cretonne/meta/isa/intel/settings.py | 32 ++++++++++++++++++++- 5 files changed, 41 insertions(+), 9 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 77e53ceeec..33df4794b3 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -1,6 +1,6 @@ ; binary emission of 32-bit code. test binemit -isa intel +isa intel has_sse42 has_popcnt ; The binary encodings can be verified with the command: ; diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index da350d1e45..908b42ac0e 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -1,7 +1,7 @@ ; binary emission of 64-bit code. test binemit set is_64bit -isa intel +isa intel has_sse42 has_popcnt ; The binary encodings can be verified with the command: ; diff --git a/cranelift/filetests/wasm/i32-arith.cton b/cranelift/filetests/wasm/i32-arith.cton index fc730cbe23..2992dcf5b8 100644 --- a/cranelift/filetests/wasm/i32-arith.cton +++ b/cranelift/filetests/wasm/i32-arith.cton @@ -2,10 +2,10 @@ test compile set is_64bit=0 -isa intel +isa intel has_sse42 has_popcnt set is_64bit=1 -isa intel +isa intel has_sse42 has_popcnt ; Constants. diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index d8dfe53140..dc54819a8b 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -7,6 +7,7 @@ from base import instructions as base from base.formats import UnaryImm from .defs import I32, I64 from . import recipes as r +from . import settings as cfg for inst, opc in [ (base.iadd, 0x01), @@ -81,10 +82,11 @@ for inst, rrr in [ I64.enc(inst.i32.i32, *r.rc(0xd3, rrr=rrr)) # Population count. -I32.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8)) -I64.enc(base.popcnt.i64, *r.urm.rex(0xf3, 0x0f, 0xb8, w=1)) -I64.enc(base.popcnt.i32, *r.urm.rex(0xf3, 0x0f, 0xb8)) -I64.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8)) +I32.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) +I64.enc(base.popcnt.i64, *r.urm.rex(0xf3, 0x0f, 0xb8, w=1), + isap=cfg.use_popcnt) +I64.enc(base.popcnt.i32, *r.urm.rex(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) +I64.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) # Loads and stores. I32.enc(base.store.i32.i32, *r.st(0x89)) diff --git a/lib/cretonne/meta/isa/intel/settings.py b/lib/cretonne/meta/isa/intel/settings.py index 42e1a0ab99..9251f73c5d 100644 --- a/lib/cretonne/meta/isa/intel/settings.py +++ b/lib/cretonne/meta/isa/intel/settings.py @@ -2,10 +2,40 @@ Intel settings. """ from __future__ import absolute_import -from cdsl.settings import SettingGroup +from cdsl.settings import SettingGroup, BoolSetting +from cdsl.predicates import And import base.settings as shared from .defs import ISA ISA.settings = SettingGroup('intel', parent=shared.group) +# The has_* settings here correspond to CPUID bits. + +# CPUID.01H:EDX +has_sse2 = BoolSetting("SSE2: CPUID.01H:EDX.SSE2[bit 26]") + +# CPUID.01H:ECX +has_sse3 = BoolSetting("SSE3: CPUID.01H:ECX.SSE3[bit 0]") +has_ssse3 = BoolSetting("SSSE3: CPUID.01H:ECX.SSSE3[bit 9]") +has_sse41 = BoolSetting("SSE4.1: CPUID.01H:ECX.SSE4_1[bit 19]") +has_sse42 = BoolSetting("SSE4.2: CPUID.01H:ECX.SSE4_2[bit 20]") +has_popcnt = BoolSetting("POPCNT: CPUID.01H:ECX.POPCNT[bit 23]") +has_avx = BoolSetting("AVX: CPUID.01H:ECX.AVX[bit 28]") + +# CPUID.(EAX=07H, ECX=0H):EBX +has_bmi1 = BoolSetting("BMI1: CPUID.(EAX=07H, ECX=0H):EBX.BMI1[bit 3]") +has_bmi2 = BoolSetting("BMI2: CPUID.(EAX=07H, ECX=0H):EBX.BMI2[bit 8]") + +# CPUID.EAX=80000001H:ECX +has_lzcnt = BoolSetting("LZCNT: CPUID.EAX=80000001H:ECX.LZCNT[bit 5]") + + +# The use_* settings here are used to determine if a feature can be used. + +use_sse41 = And(has_sse41) +use_sse42 = And(has_sse42, use_sse41) +use_popcnt = And(has_popcnt, has_sse42) +use_bmi1 = And(has_bmi1) +use_lzcnt = And(has_lzcnt) + ISA.settings.close(globals()) From 5a4aa1127403223d2832ace902232ede27ae80ab Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 13 Jul 2017 10:12:25 -0700 Subject: [PATCH 0872/3084] Add a bconst instruction. (#116) * Add a bconst instruction. --- cranelift/docs/langref.rst | 1 + cranelift/filetests/parser/tiny.cton | 17 +++++++++++++++++ lib/cretonne/meta/base/formats.py | 3 ++- lib/cretonne/meta/base/immediates.py | 7 +++++++ lib/cretonne/meta/base/instructions.py | 15 ++++++++++++++- lib/cretonne/src/ir/instructions.rs | 1 + lib/cretonne/src/verifier/mod.rs | 1 + lib/cretonne/src/write.rs | 1 + lib/reader/src/parser.rs | 20 ++++++++++++++++++++ 9 files changed, 64 insertions(+), 2 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 7f8d3e038f..78953ed8b0 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -562,6 +562,7 @@ Constant materialization .. autoinst:: iconst .. autoinst:: f32const .. autoinst:: f64const +.. autoinst:: bconst Live range splitting -------------------- diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index ff3d386947..fff7ccd4c1 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -25,6 +25,23 @@ ebb0: ; nextln: $v2 = ishl $v0, $v1 ; nextln: } +; Create and use values. +; Polymorphic instructions with type suffix. +function %bvalues() { +ebb0: + v0 = bconst.b32 true + v1 = bconst.b8 false + v2 = bextend.b32 v1 + v3 = bxor v0, v2 +} +; sameln: function %bvalues() { +; nextln: ebb0: +; nextln: $v0 = bconst.b32 true +; nextln: $v1 = bconst.b8 false +; nextln: $v2 = bextend.b32 v1 +; nextln: $v3 = bxor v0, v2 +; nextln: } + ; Polymorphic istruction controlled by second operand. function %select() { ebb0(v90: i32, v91: i32, v92: b1): diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 368fb13078..16732c1153 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -9,7 +9,7 @@ from __future__ import absolute_import from cdsl.formats import InstructionFormat from cdsl.operands import VALUE, VARIABLE_ARGS from .immediates import imm64, uimm8, ieee32, ieee64, offset32, uoffset32 -from .immediates import intcc, floatcc, memflags, regunit +from .immediates import boolean, intcc, floatcc, memflags, regunit from .entities import ebb, sig_ref, func_ref, jump_table, stack_slot Nullary = InstructionFormat() @@ -18,6 +18,7 @@ Unary = InstructionFormat(VALUE) UnaryImm = InstructionFormat(imm64) UnaryIeee32 = InstructionFormat(ieee32) UnaryIeee64 = InstructionFormat(ieee64) +UnaryBool = InstructionFormat(boolean) Binary = InstructionFormat(VALUE, VALUE) BinaryImm = InstructionFormat(VALUE, imm64) diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index 2458b76e70..5f64e73b80 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -45,6 +45,13 @@ ieee32 = ImmediateKind('ieee32', 'A 32-bit immediate floating point number.') #: IEEE 754-2008 binary64 interchange format. ieee64 = ImmediateKind('ieee64', 'A 64-bit immediate floating point number.') +#: An immediate boolean operand. +#: +#: This type of immediate boolean can interact with SSA values with any +#: :py:class:`cretonne.BoolType` type. +boolean = ImmediateKind('boolean', 'An immediate boolean.', + rust_type='bool') + #: A condition code for comparing integer values. #: #: This enumerated operand kind is used for the :cton:inst:`icmp` instruction diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 2aa9027a45..8638e19368 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -10,7 +10,7 @@ from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup from base.types import f32, f64, b1 from base.immediates import imm64, uimm8, ieee32, ieee64, offset32, uoffset32 -from base.immediates import intcc, floatcc, memflags, regunit +from base.immediates import boolean, intcc, floatcc, memflags, regunit from base import entities from cdsl.ti import WiderOrEq import base.formats # noqa @@ -18,6 +18,8 @@ import base.formats # noqa GROUP = InstructionGroup("base", "Shared base instruction set") Int = TypeVar('Int', 'A scalar or vector integer type', ints=True, simd=True) +Bool = TypeVar('Bool', 'A scalar or vector boolean type', + bools=True, simd=True) iB = TypeVar('iB', 'A scalar integer type', ints=True) iAddr = TypeVar('iAddr', 'An integer address type', ints=(32, 64)) Testable = TypeVar( @@ -417,6 +419,17 @@ f64const = Instruction( """, ins=N, outs=a) +N = Operand('N', boolean) +a = Operand('a', Bool, doc='A constant boolean scalar or vector value') +bconst = Instruction( + 'bconst', r""" + Boolean constant. + + Create a scalar boolean SSA value with an immediate constant value, or + a boolean vector where all the lanes have the same value. + """, + ins=N, outs=a) + # # Generics. # diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index cf914b853a..808b7cc01a 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -107,6 +107,7 @@ pub enum InstructionData { UnaryImm { opcode: Opcode, imm: Imm64 }, UnaryIeee32 { opcode: Opcode, imm: Ieee32 }, UnaryIeee64 { opcode: Opcode, imm: Ieee64 }, + UnaryBool { opcode: Opcode, imm: bool }, Binary { opcode: Opcode, args: [Value; 2] }, BinaryImm { opcode: Opcode, diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 12b190e3da..164fa70289 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -273,6 +273,7 @@ impl<'a> Verifier<'a> { UnaryImm { .. } | UnaryIeee32 { .. } | UnaryIeee64 { .. } | + UnaryBool { .. } | Binary { .. } | BinaryImm { .. } | Ternary { .. } | diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index f015b7d6df..56a06a0c04 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -243,6 +243,7 @@ pub fn write_operands(w: &mut Write, UnaryImm { imm, .. } => write!(w, " {}", imm), UnaryIeee32 { imm, .. } => write!(w, " {}", imm), UnaryIeee64 { imm, .. } => write!(w, " {}", imm), + UnaryBool { imm, .. } => write!(w, " {}", imm), Binary { args, .. } => write!(w, " {}, {}", args[0], args[1]), BinaryImm { arg, imm, .. } => write!(w, " {}, {}", arg, imm), Ternary { args, .. } => write!(w, " {}, {}, {}", args[0], args[1], args[2]), diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index f698b1f54f..620d02ca2b 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -498,6 +498,20 @@ impl<'a> Parser<'a> { } } + // Match and consume a boolean immediate. + fn match_bool(&mut self, err_msg: &str) -> Result { + if let Some(Token::Identifier(text)) = self.token() { + self.consume(); + match text { + "true" => Ok(true), + "false" => Ok(false), + _ => err!(self.loc, err_msg), + } + } else { + err!(self.loc, err_msg) + } + } + // Match and consume an enumerated immediate, like one of the condition codes. fn match_enum(&mut self, err_msg: &str) -> Result { if let Some(Token::Identifier(text)) = self.token() { @@ -1483,6 +1497,12 @@ impl<'a> Parser<'a> { imm: self.match_ieee64("expected immediate 64-bit float operand")?, } } + InstructionFormat::UnaryBool => { + InstructionData::UnaryBool { + opcode, + imm: self.match_bool("expected immediate boolean operand")?, + } + } InstructionFormat::Binary => { let lhs = self.match_value("expected SSA value first operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; From 6d6035b91859e76993f33f945b16e7bbea71e757 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 13 Jul 2017 13:18:18 -0700 Subject: [PATCH 0873/3084] CSSA verifier. During register allocation, the code must be kept in conventional SSA form. Add a verifier that checks this property. --- lib/cretonne/src/regalloc/context.rs | 6 +- lib/cretonne/src/regalloc/mod.rs | 2 +- lib/cretonne/src/regalloc/virtregs.rs | 8 +- lib/cretonne/src/verifier/cssa.rs | 122 ++++++++++++++++++++++++++ lib/cretonne/src/verifier/mod.rs | 2 + 5 files changed, 137 insertions(+), 3 deletions(-) create mode 100644 lib/cretonne/src/verifier/cssa.rs diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 1961add95c..0523984604 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -17,7 +17,7 @@ use regalloc::spilling::Spilling; use regalloc::virtregs::VirtRegs; use result::CtonResult; use topo_order::TopoOrder; -use verifier::{verify_context, verify_liveness}; +use verifier::{verify_context, verify_liveness, verify_cssa}; /// Persistent memory allocations for register allocation. pub struct Context { @@ -85,6 +85,7 @@ impl Context { if isa.flags().enable_verifier() { verify_context(func, cfg, domtree, Some(isa))?; verify_liveness(isa, func, cfg, &self.liveness)?; + verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?; } @@ -101,6 +102,7 @@ impl Context { if isa.flags().enable_verifier() { verify_context(func, cfg, domtree, Some(isa))?; verify_liveness(isa, func, cfg, &self.liveness)?; + verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?; } // Pass: Reload. @@ -115,6 +117,7 @@ impl Context { if isa.flags().enable_verifier() { verify_context(func, cfg, domtree, Some(isa))?; verify_liveness(isa, func, cfg, &self.liveness)?; + verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?; } // Pass: Coloring. @@ -124,6 +127,7 @@ impl Context { if isa.flags().enable_verifier() { verify_context(func, cfg, domtree, Some(isa))?; verify_liveness(isa, func, cfg, &self.liveness)?; + verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?; } Ok(()) } diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index f689503c03..6ae93c67b6 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -7,6 +7,7 @@ pub mod liveness; pub mod allocatable_set; pub mod live_value_tracker; pub mod coloring; +pub mod virtregs; mod affinity; mod coalescing; @@ -16,7 +17,6 @@ mod pressure; mod reload; mod solver; mod spilling; -mod virtregs; pub use self::allocatable_set::AllocatableSet; pub use self::context::Context; diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index 6efe196805..17fa641fcd 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -12,11 +12,12 @@ //! memory-to-memory copies when a spilled value is passed as an EBB argument. use entity_list::{EntityList, ListPool}; -use entity_map::{EntityMap, PrimaryEntityData}; +use entity_map::{EntityMap, PrimaryEntityData, Keys}; use ir::Value; use packed_option::PackedOption; use ref_slice::ref_slice; +/// A virtual register reference. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] pub struct VirtReg(u32); entity_impl!(VirtReg, "vreg"); @@ -71,6 +72,11 @@ impl VirtRegs { self.vregs[vreg].as_slice(&self.pool) } + /// Get an iterator over all virtual registers. + pub fn all_virtregs(&self) -> Keys { + self.vregs.keys() + } + /// Get the congruence class of `value`. /// /// If `value` belongs to a virtual register, the congruence class is the values of the virtual diff --git a/lib/cretonne/src/verifier/cssa.rs b/lib/cretonne/src/verifier/cssa.rs new file mode 100644 index 0000000000..fe9a2fe18e --- /dev/null +++ b/lib/cretonne/src/verifier/cssa.rs @@ -0,0 +1,122 @@ +//! Verify conventional SSA form. + +use dominator_tree::DominatorTree; +use flowgraph::ControlFlowGraph; +use ir::Function; +use regalloc::liveness::Liveness; +use regalloc::virtregs::VirtRegs; +use std::cmp::Ordering; +use verifier::Result; + +/// Verify conventional SSA form for `func`. +/// +/// Conventional SSA form is represented in Cretonne with the help of virtual registers: +/// +/// - Two values are said to be *PHI-related* if one is an EBB argument and the other is passed as +/// a branch argument in a location that matches the first value. +/// - PHI-related values must belong to the same virtual register. +/// - Two values in the same virtual register must not have overlapping live ranges. +/// +/// Additionally, we verify this property of virtual registers: +/// +/// - The values in a virtual register are ordered according to the dominator tree's `rpo_cmp()`. +/// +/// We don't verify that virtual registers are minimal. Minimal CSSA is not required. +pub fn verify_cssa(func: &Function, + cfg: &ControlFlowGraph, + domtree: &DominatorTree, + liveness: &Liveness, + virtregs: &VirtRegs) + -> Result { + let verifier = CssaVerifier { + func, + cfg, + domtree, + virtregs, + liveness, + }; + verifier.check_virtregs()?; + verifier.check_cssa()?; + Ok(()) +} + +struct CssaVerifier<'a> { + func: &'a Function, + cfg: &'a ControlFlowGraph, + domtree: &'a DominatorTree, + virtregs: &'a VirtRegs, + liveness: &'a Liveness, +} + +impl<'a> CssaVerifier<'a> { + fn check_virtregs(&self) -> Result { + for vreg in self.virtregs.all_virtregs() { + let values = self.virtregs.values(vreg); + + for (idx, &val) in values.iter().enumerate() { + if !self.func.dfg.value_is_valid(val) { + return err!(val, "Invalid value in {}", vreg); + } + if !self.func.dfg.value_is_attached(val) { + return err!(val, "Detached value in {}", vreg); + } + if self.liveness.get(val).is_none() { + return err!(val, "Value in {} has no live range", vreg); + }; + + // Check RPO ordering with the previous values in the virtual register. + let def = self.func.dfg.value_def(val).into(); + let def_ebb = self.func.layout.pp_ebb(def); + for &prev_val in &values[0..idx] { + let prev_def = self.func.dfg.value_def(prev_val); + + // Enforce RPO of defs in the virtual register. + match self.domtree.rpo_cmp(prev_def, def, &self.func.layout) { + Ordering::Less => {} + Ordering::Equal => { + return err!(val, "Value in {} has same def as {}", vreg, prev_val); + } + Ordering::Greater => { + return err!(val, + "Value in {} in wrong order relative to {}", + vreg, + prev_val); + } + } + + // Knowing that values are in RPO order, we can check for interference this + // way. + if self.liveness[prev_val].overlaps_def(def, def_ebb, &self.func.layout) { + return err!(val, "Value def in {} interferes with {}", vreg, prev_val); + } + } + } + } + + Ok(()) + } + + fn check_cssa(&self) -> Result { + for ebb in self.func.layout.ebbs() { + let ebb_args = self.func.dfg.ebb_args(ebb); + for &(_, pred) in self.cfg.get_predecessors(ebb) { + let pred_args = self.func.dfg.inst_variable_args(pred); + // This should have been caught by an earlier verifier pass. + assert_eq!(ebb_args.len(), + pred_args.len(), + "Wrong arguments on branch."); + + for (&ebb_arg, &pred_arg) in ebb_args.iter().zip(pred_args) { + if !self.virtregs.same_class(ebb_arg, pred_arg) { + return err!(pred, + "{} and {} must be in the same virtual register", + ebb_arg, + pred_arg); + } + } + } + } + + Ok(()) + } +} diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 164fa70289..2652579020 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -65,6 +65,7 @@ use std::result; use std::collections::BTreeSet; pub use self::liveness::verify_liveness; +pub use self::cssa::verify_cssa; // Create an `Err` variant of `Result` from a location and `format!` arguments. macro_rules! err { @@ -83,6 +84,7 @@ macro_rules! err { }; } +mod cssa; mod liveness; /// A verifier error. From 130b7fa2fa8eb175605b7581fe8ef7eeb9c86899 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 13 Jul 2017 16:23:41 -0700 Subject: [PATCH 0874/3084] Add documentation for immediates with type bool. This makes the documentation for the new bconst instruction more complete. --- cranelift/docs/langref.rst | 7 +++++++ lib/cretonne/meta/base/immediates.py | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 78953ed8b0..49593e3f09 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -265,6 +265,13 @@ indicate the different kinds of immediate operands on an instruction. A 64-bit immediate floating point number in the IEEE 754-2008 binary64 interchange format. All bit patterns are allowed. +.. type:: bool + + A boolean immediate value, either false or true. + + In the textual format, :type:`bool` immediates appear as 'false' + and 'true'. + .. type:: intcc An integer condition code. See the :inst:`icmp` instruction for details. diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index 5f64e73b80..1defe46f6d 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -49,7 +49,7 @@ ieee64 = ImmediateKind('ieee64', 'A 64-bit immediate floating point number.') #: #: This type of immediate boolean can interact with SSA values with any #: :py:class:`cretonne.BoolType` type. -boolean = ImmediateKind('boolean', 'An immediate boolean.', +boolean = ImmediateKind('bool', 'An immediate boolean.', rust_type='bool') #: A condition code for comparing integer values. From f91d747bda84faafe00afa2e31a2c7c9ba405a5f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 13 Jul 2017 14:49:17 -0700 Subject: [PATCH 0875/3084] Add support for setting presets. Fixes #11. Presets are groups of settings and values applied at once. This is used as a shorthand in test files, so for example "isa intel nehalem" enables all of the CPUID bits that the Nehalem micro-architecture provides. --- lib/cretonne/meta/cdsl/settings.py | 110 +++++++++++++++++++++++- lib/cretonne/meta/gen_settings.py | 32 +++++-- lib/cretonne/meta/isa/intel/settings.py | 8 +- lib/cretonne/src/isa/intel/settings.rs | 24 ++++++ lib/cretonne/src/isa/mod.rs | 4 +- lib/cretonne/src/isa/riscv/mod.rs | 8 +- lib/cretonne/src/isa/riscv/settings.rs | 10 +-- lib/cretonne/src/settings.rs | 46 +++++++--- lib/reader/src/isaspec.rs | 2 +- lib/reader/src/parser.rs | 2 +- 10 files changed, 213 insertions(+), 33 deletions(-) diff --git a/lib/cretonne/meta/cdsl/settings.py b/lib/cretonne/meta/cdsl/settings.py index d9404060dd..aca3ac296f 100644 --- a/lib/cretonne/meta/cdsl/settings.py +++ b/lib/cretonne/meta/cdsl/settings.py @@ -4,7 +4,8 @@ from collections import OrderedDict from .predicates import Predicate try: - from typing import Set, List, Dict, Any, TYPE_CHECKING # noqa + from typing import Tuple, Set, List, Dict, Any, Union, TYPE_CHECKING # noqa + BoolOrPresetOrDict = Union['BoolSetting', 'Preset', Dict['Setting', Any]] if TYPE_CHECKING: from .predicates import PredLeaf, PredNode # noqa except ImportError: @@ -47,6 +48,17 @@ class Setting(object): # type: () -> int raise NotImplementedError("default_byte is an abstract method") + def byte_for_value(self, value): + # type: (Any) -> int + """Get the setting byte value that corresponds to `value`""" + raise NotImplementedError("byte_for_value is an abstract method") + + def byte_mask(self): + # type: () -> int + """Get a mask of bits in our byte that are relevant to this setting.""" + # Only BoolSetting has a different mask. + return 0xff + class BoolSetting(Setting): """ @@ -73,6 +85,17 @@ class BoolSetting(Setting): else: return 0 + def byte_for_value(self, value): + # type: (Any) -> int + if value: + return 1 << self.bit_offset + else: + return 0 + + def byte_mask(self): + # type: () -> int + return 1 << self.bit_offset + def predicate_leafs(self, leafs): # type: (Set[PredLeaf]) -> None leafs.add(self) @@ -107,6 +130,12 @@ class NumSetting(Setting): # type: () -> int return self.default + def byte_for_value(self, value): + # type: (Any) -> int + assert isinstance(value, int), "NumSetting must be set to an int" + assert value >= 0 and value <= 255 + return value + class EnumSetting(Setting): """ @@ -129,6 +158,10 @@ class EnumSetting(Setting): # type: () -> int return 0 + def byte_for_value(self, value): + # type: (Any) -> int + return self.values.index(value) + class SettingGroup(object): """ @@ -160,6 +193,7 @@ class SettingGroup(object): # - Added parent predicates that are replicated in this group. # Maps predicate -> number. self.predicate_number = OrderedDict() # type: OrderedDict[PredNode, int] # noqa + self.presets = [] # type: List[Preset] # Fully qualified Rust module name. See gen_settings.py. self.qual_mod = None # type: str @@ -199,6 +233,10 @@ class SettingGroup(object): assert obj.name is None obj.name = name self.named_predicates.append(obj) + if isinstance(obj, Preset): + assert obj.name is None, obj.name + obj.name = name + self.layout() @staticmethod @@ -209,6 +247,14 @@ class SettingGroup(object): g.settings.append(setting) return g + @staticmethod + def append_preset(preset): + # type: (Preset) -> SettingGroup + g = SettingGroup._current + assert g, "Open a setting group before defining presets." + g.presets.append(preset) + return g + def number_predicate(self, pred): # type: (PredNode) -> int """ @@ -295,3 +341,65 @@ class SettingGroup(object): predcate bits rounded up to a whole number of bytes. """ return self.boolean_offset + (len(self.predicate_number) + 7) // 8 + + +class Preset(object): + """ + A collection of setting values that are applied at once. + + A `Preset` represents a shorthand notation for applying a number of + settings at once. Example: + + nehalem = Preset(has_sse41, has_cmov, has_avx=0) + + Enabling the `nehalem` setting is equivalent to enabling `has_sse41` and + `has_cmov` while disabling the `has_avx` setting. + """ + + def __init__(self, *args): + # type: (*BoolOrPresetOrDict) -> None + self.name = None # type: str # Assigned later by `SettingGroup`. + # Each tuple provides the value for a setting. + self.values = list() # type: List[Tuple[Setting, Any]] + + for arg in args: + if isinstance(arg, Preset): + # Any presets in args are immediately expanded. + self.values.extend(arg.values) + elif isinstance(arg, dict): + # A dictionary of key: value pairs. + self.values.extend(arg.items()) + else: + # A BoolSetting to enable. + assert isinstance(arg, BoolSetting) + self.values.append((arg, True)) + + self.group = SettingGroup.append_preset(self) + # Index into the generated DESCRIPTORS table. + self.descriptor_index = None # type: int + + def layout(self): + # type: () -> List[Tuple[int, int]] + """ + Compute a list of (mask, byte) pairs that incorporate all values in + this preset. + + The list will have an entry for each setting byte in the settings + group. + """ + l = [(0, 0)] * self.group.settings_size + + # Apply setting values in order. + for s, v in self.values: + ofs = s.byte_offset + s_mask = s.byte_mask() + s_val = s.byte_for_value(v) + assert (s_val & ~s_mask) == 0 + l_mask, l_val = l[ofs] + # Accumulated mask of modified bits. + l_mask |= s_mask + # Overwrite the relevant bits with the new value. + l_val = (l_val & ~s_mask) | s_val + l[ofs] = (l_mask, l_val) + + return l diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index ae16b337c0..c56cc4df41 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -10,10 +10,10 @@ from cdsl.settings import BoolSetting, NumSetting, EnumSetting from base import settings try: - from typing import Sequence, Set, Tuple, List, TYPE_CHECKING # noqa + from typing import Sequence, Set, Tuple, List, Union, TYPE_CHECKING # noqa if TYPE_CHECKING: from cdsl.isa import TargetISA # noqa - from cdsl.settings import Setting, SettingGroup # noqa + from cdsl.settings import Setting, Preset, SettingGroup # noqa from cdsl.predicates import Predicate, PredContext # noqa except ImportError: pass @@ -106,14 +106,14 @@ def gen_getters(sgrp, fmt): def gen_descriptors(sgrp, fmt): # type: (SettingGroup, srcgen.Formatter) -> None """ - Generate the DESCRIPTORS and ENUMERATORS tables. + Generate the DESCRIPTORS, ENUMERATORS, and PRESETS tables. """ enums = UniqueSeqTable() with fmt.indented( 'static DESCRIPTORS: [detail::Descriptor; {}] = [' - .format(len(sgrp.settings)), + .format(len(sgrp.settings) + len(sgrp.presets)), '];'): for idx, setting in enumerate(sgrp.settings): setting.descriptor_index = idx @@ -135,6 +135,13 @@ def gen_descriptors(sgrp, fmt): else: raise AssertionError("Unknown setting kind") + for idx, preset in enumerate(sgrp.presets): + preset.descriptor_index = len(sgrp.settings) + idx + with fmt.indented('detail::Descriptor {', '},'): + fmt.line('name: "{}",'.format(preset.name)) + fmt.line('offset: {},'.format(idx * sgrp.settings_size)) + fmt.line('detail: detail::Detail::Preset,') + with fmt.indented( 'static ENUMERATORS: [&str; {}] = [' .format(len(enums.table)), @@ -143,10 +150,13 @@ def gen_descriptors(sgrp, fmt): fmt.line('"{}",'.format(txt)) def hash_setting(s): - # type: (Setting) -> int + # type: (Union[Setting, Preset]) -> int return constant_hash.simple_hash(s.name) - hash_table = constant_hash.compute_quadratic(sgrp.settings, hash_setting) + hash_elems = [] # type: List[Union[Setting, Preset]] + hash_elems.extend(sgrp.settings) + hash_elems.extend(sgrp.presets) + hash_table = constant_hash.compute_quadratic(hash_elems, hash_setting) with fmt.indented( 'static HASH_TABLE: [u16; {}] = [' .format(len(hash_table)), @@ -157,6 +167,15 @@ def gen_descriptors(sgrp, fmt): else: fmt.line('{},'.format(h.descriptor_index)) + with fmt.indented( + 'static PRESETS: [(u8, u8); {}] = [' + .format(len(sgrp.presets) * sgrp.settings_size), + '];'): + for preset in sgrp.presets: + fmt.comment(preset.name) + for mask, value in preset.layout(): + fmt.format('(0b{:08b}, 0b{:08b}),', mask, value) + def gen_template(sgrp, fmt): # type: (SettingGroup, srcgen.Formatter) -> None @@ -175,6 +194,7 @@ def gen_template(sgrp, fmt): fmt.line('hash_table: &HASH_TABLE,') vs = ', '.join('{:#04x}'.format(x) for x in v) fmt.line('defaults: &[ {} ],'.format(vs)) + fmt.line('presets: &PRESETS,') fmt.doc_comment( 'Create a `settings::Builder` for the {} settings group.' diff --git a/lib/cretonne/meta/isa/intel/settings.py b/lib/cretonne/meta/isa/intel/settings.py index 9251f73c5d..a16303a3b5 100644 --- a/lib/cretonne/meta/isa/intel/settings.py +++ b/lib/cretonne/meta/isa/intel/settings.py @@ -2,7 +2,7 @@ Intel settings. """ from __future__ import absolute_import -from cdsl.settings import SettingGroup, BoolSetting +from cdsl.settings import SettingGroup, BoolSetting, Preset from cdsl.predicates import And import base.settings as shared from .defs import ISA @@ -38,4 +38,10 @@ use_popcnt = And(has_popcnt, has_sse42) use_bmi1 = And(has_bmi1) use_lzcnt = And(has_lzcnt) +# Presets corresponding to Intel CPUs. + +nehalem = Preset( + has_sse2, has_sse3, has_ssse3, has_sse41, has_sse42, has_popcnt) +haswell = Preset(nehalem, has_bmi1, has_lzcnt) + ISA.settings.close(globals()) diff --git a/lib/cretonne/src/isa/intel/settings.rs b/lib/cretonne/src/isa/intel/settings.rs index 341eb2dcc9..7ca4886588 100644 --- a/lib/cretonne/src/isa/intel/settings.rs +++ b/lib/cretonne/src/isa/intel/settings.rs @@ -7,3 +7,27 @@ use std::fmt; // `Flags` struct with an impl for all of the settings defined in // `lib/cretonne/meta/cretonne/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings-intel.rs")); + +#[cfg(test)] +mod tests { + use super::{builder, Flags}; + use settings::{self, Configurable}; + + #[test] + fn presets() { + let shared = settings::Flags::new(&settings::builder()); + + // Nehalem has SSE4.1 but not BMI1. + let mut b1 = builder(); + b1.enable("nehalem").unwrap(); + let f1 = Flags::new(&shared, &b1); + assert_eq!(f1.has_sse41(), true); + assert_eq!(f1.has_bmi1(), false); + + let mut b2 = builder(); + b2.enable("haswell").unwrap(); + let f2 = Flags::new(&shared, &b2); + assert_eq!(f2.has_sse41(), true); + assert_eq!(f2.has_bmi1(), true); + } +} diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 1ec80e790c..edd72b27af 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -108,8 +108,8 @@ impl settings::Configurable for Builder { self.setup.set(name, value) } - fn set_bool(&mut self, name: &str, value: bool) -> settings::Result<()> { - self.setup.set_bool(name, value) + fn enable(&mut self, name: &str) -> settings::Result<()> { + self.setup.enable(name) } } diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 5f9cd771a2..246ec13bd1 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -114,7 +114,7 @@ mod tests { #[test] fn test_64bitenc() { let mut shared_builder = settings::builder(); - shared_builder.set_bool("is_64bit", true).unwrap(); + shared_builder.enable("is_64bit").unwrap(); let shared_flags = settings::Flags::new(&shared_builder); let isa = isa::lookup("riscv").unwrap().finish(shared_flags); @@ -161,7 +161,7 @@ mod tests { #[test] fn test_32bitenc() { let mut shared_builder = settings::builder(); - shared_builder.set_bool("is_64bit", false).unwrap(); + shared_builder.set("is_64bit", "false").unwrap(); let shared_flags = settings::Flags::new(&shared_builder); let isa = isa::lookup("riscv").unwrap().finish(shared_flags); @@ -216,13 +216,13 @@ mod tests { #[test] fn test_rv32m() { let mut shared_builder = settings::builder(); - shared_builder.set_bool("is_64bit", false).unwrap(); + shared_builder.set("is_64bit", "false").unwrap(); let shared_flags = settings::Flags::new(&shared_builder); // Set the supports_m stting which in turn enables the use_m predicate that unlocks // encodings for imul. let mut isa_builder = isa::lookup("riscv").unwrap(); - isa_builder.set_bool("supports_m", true).unwrap(); + isa_builder.enable("supports_m").unwrap(); let isa = isa_builder.finish(shared_flags); diff --git a/lib/cretonne/src/isa/riscv/settings.rs b/lib/cretonne/src/isa/riscv/settings.rs index aa66d75fd9..69a47f8717 100644 --- a/lib/cretonne/src/isa/riscv/settings.rs +++ b/lib/cretonne/src/isa/riscv/settings.rs @@ -34,17 +34,17 @@ mod tests { fn predicates() { let shared = settings::Flags::new(&settings::builder()); let mut b = builder(); - b.set_bool("supports_f", true).unwrap(); - b.set_bool("supports_d", true).unwrap(); + b.enable("supports_f").unwrap(); + b.enable("supports_d").unwrap(); let f = Flags::new(&shared, &b); assert_eq!(f.full_float(), true); let mut sb = settings::builder(); - sb.set_bool("enable_simd", false).unwrap(); + sb.set("enable_simd", "false").unwrap(); let shared = settings::Flags::new(&sb); let mut b = builder(); - b.set_bool("supports_f", true).unwrap(); - b.set_bool("supports_d", true).unwrap(); + b.enable("supports_f").unwrap(); + b.enable("supports_d").unwrap(); let f = Flags::new(&shared, &b); assert_eq!(f.full_float(), false); } diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 204588c7c1..19b9fafc21 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -35,10 +35,10 @@ pub trait Configurable { /// This can set any type of setting whether it is numeric, boolean, or enumerated. fn set(&mut self, name: &str, value: &str) -> Result<()>; - /// Set the value of a boolean setting by name. + /// Enable a boolean setting or apply a preset. /// - /// If the identified setting isn't a boolean, a `BadType` error is returned. - fn set_bool(&mut self, name: &str, value: bool) -> Result<()>; + /// If the identified setting isn't a boolean or a preset, a `BadType` error is returned. + fn enable(&mut self, name: &str) -> Result<()>; } /// Collect settings values based on a template. @@ -73,6 +73,13 @@ impl Builder { } } + /// Apply a preset. The argument is a slice of (mask, value) bytes. + fn apply_preset(&mut self, values: &[(u8, u8)]) { + for (byte, &(mask, value)) in self.bytes.iter_mut().zip(values) { + *byte = (*byte & !mask) | value; + } + } + /// Look up a descriptor by name. fn lookup(&self, name: &str) -> Result<(usize, detail::Detail)> { match probe(self.template, name, simple_hash(name)) { @@ -101,14 +108,19 @@ fn parse_enum_value(value: &str, choices: &[&str]) -> Result { } impl Configurable for Builder { - fn set_bool(&mut self, name: &str, value: bool) -> Result<()> { + fn enable(&mut self, name: &str) -> Result<()> { use self::detail::Detail; let (offset, detail) = self.lookup(name)?; - if let Detail::Bool { bit } = detail { - self.set_bit(offset, bit, value); - Ok(()) - } else { - Err(Error::BadType) + match detail { + Detail::Bool { bit } => { + self.set_bit(offset, bit, true); + Ok(()) + } + Detail::Preset => { + self.apply_preset(&self.template.presets[offset..]); + Ok(()) + } + _ => Err(Error::BadType), } } @@ -128,6 +140,7 @@ impl Configurable for Builder { self.bytes[offset] = parse_enum_value(value, self.template.enums(last, enumerators))?; } + Detail::Preset => return Err(Error::BadName), } Ok(()) } @@ -169,6 +182,8 @@ pub mod detail { pub hash_table: &'static [u16], /// Default values. pub defaults: &'static [u8], + /// Pairs of (mask, value) for presets. + pub presets: &'static [(u8, u8)], } impl Template { @@ -197,6 +212,8 @@ pub mod detail { write!(f, "{}", byte) } } + // Presets aren't printed. They are reflected in the other settings. + Detail::Preset { .. } => Ok(()), } } } @@ -251,6 +268,11 @@ pub mod detail { /// First enumerator in the ENUMERATORS table. enumerators: u16, }, + + /// A preset is not an individual setting, it is a collection of settings applied at once. + /// + /// The `Descriptor::offset` field refers to the `PRESETS` table. + Preset, } } @@ -284,9 +306,9 @@ mod tests { #[test] fn modify_bool() { let mut b = builder(); - assert_eq!(b.set_bool("not_there", true), Err(BadName)); - assert_eq!(b.set_bool("enable_simd", true), Ok(())); - assert_eq!(b.set_bool("enable_simd", false), Ok(())); + assert_eq!(b.enable("not_there"), Err(BadName)); + assert_eq!(b.enable("enable_simd"), Ok(())); + assert_eq!(b.set("enable_simd", "false"), Ok(())); let f = Flags::new(&b); assert_eq!(f.enable_simd(), false); diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs index 3715f4ed6b..51748e284a 100644 --- a/lib/reader/src/isaspec.rs +++ b/lib/reader/src/isaspec.rs @@ -41,7 +41,7 @@ pub fn parse_options<'a, I>(iter: I, config: &mut Configurable, loc: &Location) for opt in iter.map(TestOption::new) { match opt { TestOption::Flag(name) => { - match config.set_bool(name, true) { + match config.enable(name) { Ok(_) => {} Err(SetError::BadName) => return err!(loc, "unknown flag '{}'", opt), Err(_) => return err!(loc, "not a boolean flag: '{}'", opt), diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 620d02ca2b..fb659a6b8e 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -609,7 +609,7 @@ impl<'a> Parser<'a> { // would slow down normal compilation, but when we're reading IL from a text file we're // either testing or debugging Cretonne, and verification makes sense. flag_builder - .set_bool("enable_verifier", true) + .enable("enable_verifier") .expect("Missing enable_verifier setting"); while let Some(Token::Identifier(command)) = self.token() { From 9dc92eb8b3ac3e033b6d08377dbe482426d37261 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 12 Jul 2017 16:28:33 -0700 Subject: [PATCH 0876/3084] Add Intel BMI1 ctz and clz encodings. --- cranelift/filetests/isa/intel/binary32.cton | 12 ++++++- cranelift/filetests/isa/intel/binary64.cton | 36 ++++++++++++++++++--- cranelift/filetests/wasm/i32-arith.cton | 17 +++++++--- lib/cretonne/meta/isa/intel/encodings.py | 14 ++++++++ 4 files changed, 70 insertions(+), 9 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 33df4794b3..bd3ff06289 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -1,6 +1,6 @@ ; binary emission of 32-bit code. test binemit -isa intel has_sse42 has_popcnt +isa intel haswell ; The binary encodings can be verified with the command: ; @@ -224,6 +224,16 @@ ebb0: ; asm: popcntl %ecx, %esi [-,%rsi] v201 = popcnt v1 ; bin: f3 0f b8 f1 + ; asm: lzcntl %esi, %ecx + [-,%rcx] v202 = clz v2 ; bin: f3 0f bd ce + ; asm: lzcntl %ecx, %esi + [-,%rsi] v203 = clz v1 ; bin: f3 0f bd f1 + + ; asm: tzcntl %esi, %ecx + [-,%rcx] v204 = ctz v2 ; bin: f3 0f bc ce + ; asm: tzcntl %ecx, %esi + [-,%rsi] v205 = ctz v1 ; bin: f3 0f bc f1 + ; asm: call foo call fn0() ; bin: e8 PCRel4(fn0) 00000000 diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 908b42ac0e..ecb4b8a40d 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -1,7 +1,7 @@ ; binary emission of 64-bit code. test binemit set is_64bit -isa intel has_sse42 has_popcnt +isa intel haswell ; The binary encodings can be verified with the command: ; @@ -154,6 +154,20 @@ ebb0: ; asm: popcntq %rcx, %r10 [-,%r10] v202 = popcnt v1 ; bin: f3 4c 0f b8 d1 + ; asm: lzcntq %rsi, %rcx + [-,%rcx] v203 = clz v2 ; bin: f3 48 0f bd ce + ; asm: lzcntq %r10, %rsi + [-,%rsi] v204 = clz v3 ; bin: f3 49 0f bd f2 + ; asm: lzcntq %rcx, %r10 + [-,%r10] v205 = clz v1 ; bin: f3 4c 0f bd d1 + + ; asm: tzcntq %rsi, %rcx + [-,%rcx] v206 = ctz v2 ; bin: f3 48 0f bc ce + ; asm: tzcntq %r10, %rsi + [-,%rsi] v207 = ctz v3 ; bin: f3 49 0f bc f2 + ; asm: tzcntq %rcx, %r10 + [-,%r10] v208 = ctz v1 ; bin: f3 4c 0f bc d1 + return ; bin: c3 } @@ -302,11 +316,25 @@ ebb0: ; Bit-counting instructions. ; asm: popcntl %esi, %ecx - [-,%rcx] v200 = popcnt v2 ; bin: f3 40 0f b8 ce + [-,%rcx] v200 = popcnt v2 ; bin: f3 40 0f b8 ce ; asm: popcntl %r10d, %esi - [-,%rsi] v201 = popcnt v3 ; bin: f3 41 0f b8 f2 + [-,%rsi] v201 = popcnt v3 ; bin: f3 41 0f b8 f2 ; asm: popcntl %ecx, %r10d - [-,%r10] v202 = popcnt v1 ; bin: f3 44 0f b8 d1 + [-,%r10] v202 = popcnt v1 ; bin: f3 44 0f b8 d1 + + ; asm: lzcntl %esi, %ecx + [-,%rcx] v203 = clz v2 ; bin: f3 40 0f bd ce + ; asm: lzcntl %r10d, %esi + [-,%rsi] v204 = clz v3 ; bin: f3 41 0f bd f2 + ; asm: lzcntl %ecx, %r10d + [-,%r10] v205 = clz v1 ; bin: f3 44 0f bd d1 + + ; asm: tzcntl %esi, %ecx + [-,%rcx] v206 = ctz v2 ; bin: f3 40 0f bc ce + ; asm: tzcntl %r10d, %esi + [-,%rsi] v207 = ctz v3 ; bin: f3 41 0f bc f2 + ; asm: tzcntl %ecx, %r10d + [-,%r10] v208 = ctz v1 ; bin: f3 44 0f bc d1 return ; bin: c3 } diff --git a/cranelift/filetests/wasm/i32-arith.cton b/cranelift/filetests/wasm/i32-arith.cton index 2992dcf5b8..bddec1b52b 100644 --- a/cranelift/filetests/wasm/i32-arith.cton +++ b/cranelift/filetests/wasm/i32-arith.cton @@ -2,10 +2,10 @@ test compile set is_64bit=0 -isa intel has_sse42 has_popcnt +isa intel haswell set is_64bit=1 -isa intel has_sse42 has_popcnt +isa intel haswell ; Constants. @@ -17,8 +17,17 @@ ebb0: ; Unary operations. -; function %i32_clz(i32) -> i32 -; function %i32_ctz(i32) -> i32 +function %i32_clz(i32) -> i32 { +ebb0(v0: i32): + v1 = clz v0 + return v1 +} + +function %i32_ctz(i32) -> i32 { +ebb0(v0: i32): + v1 = ctz v0 + return v1 +} function %i32_popcnt(i32) -> i32 { ebb0(v0: i32): diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index dc54819a8b..c2c361ea86 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -88,6 +88,20 @@ I64.enc(base.popcnt.i64, *r.urm.rex(0xf3, 0x0f, 0xb8, w=1), I64.enc(base.popcnt.i32, *r.urm.rex(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) I64.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) +# Count leading zero bits. +I32.enc(base.clz.i32, *r.urm(0xf3, 0x0f, 0xbd), isap=cfg.use_lzcnt) +I64.enc(base.clz.i64, *r.urm.rex(0xf3, 0x0f, 0xbd, w=1), + isap=cfg.use_lzcnt) +I64.enc(base.clz.i32, *r.urm.rex(0xf3, 0x0f, 0xbd), isap=cfg.use_lzcnt) +I64.enc(base.clz.i32, *r.urm(0xf3, 0x0f, 0xbd), isap=cfg.use_lzcnt) + +# Count trailing zero bits. +I32.enc(base.ctz.i32, *r.urm(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) +I64.enc(base.ctz.i64, *r.urm.rex(0xf3, 0x0f, 0xbc, w=1), + isap=cfg.use_bmi1) +I64.enc(base.ctz.i32, *r.urm.rex(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) +I64.enc(base.ctz.i32, *r.urm(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) + # Loads and stores. I32.enc(base.store.i32.i32, *r.st(0x89)) I32.enc(base.store.i32.i32, *r.stDisp8(0x89)) From 2f7057b96f41ef4969fa1472348e303731bc68e1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 17 Jul 2017 18:13:05 -0700 Subject: [PATCH 0877/3084] Add a Context::emit_to_memory function. This function will emit the binary machine code into contiguous raw memory while sending relocations to a RelocSink. Add a MemoryCodeSink for generating machine code directly into memory efficiently. Allow the TargetIsa to provide emit_function implementations that are specialized to the MemoryCodeSink type to avoid needless small virtual callbacks to put1() et etc. --- cranelift/src/filetest/compile.rs | 9 +-- lib/cretonne/src/binemit/memorysink.rs | 108 +++++++++++++++++++++++++ lib/cretonne/src/binemit/mod.rs | 19 +++++ lib/cretonne/src/context.rs | 12 ++- lib/cretonne/src/isa/arm32/mod.rs | 6 +- lib/cretonne/src/isa/arm64/mod.rs | 6 +- lib/cretonne/src/isa/intel/mod.rs | 6 +- lib/cretonne/src/isa/mod.rs | 9 ++- lib/cretonne/src/isa/riscv/mod.rs | 6 +- 9 files changed, 168 insertions(+), 13 deletions(-) create mode 100644 lib/cretonne/src/binemit/memorysink.rs diff --git a/cranelift/src/filetest/compile.rs b/cranelift/src/filetest/compile.rs index 15db119efd..f3766f001f 100644 --- a/cranelift/src/filetest/compile.rs +++ b/cranelift/src/filetest/compile.rs @@ -51,12 +51,9 @@ impl SubTest for TestCompile { // Finally verify that the returned code size matches the emitted bytes. let mut sink = SizeSink { offset: 0 }; - for ebb in comp_ctx.func.layout.ebbs() { - assert_eq!(sink.offset, comp_ctx.func.offsets[ebb]); - for inst in comp_ctx.func.layout.ebb_insts(ebb) { - isa.emit_inst(&comp_ctx.func, inst, &mut sink); - } - } + binemit::emit_function(&comp_ctx.func, + |func, inst, sink| isa.emit_inst(func, inst, sink), + &mut sink); if sink.offset != code_size { return Err(format!("Expected code size {}, got {}", code_size, sink.offset)); diff --git a/lib/cretonne/src/binemit/memorysink.rs b/lib/cretonne/src/binemit/memorysink.rs new file mode 100644 index 0000000000..f3fd5cebba --- /dev/null +++ b/lib/cretonne/src/binemit/memorysink.rs @@ -0,0 +1,108 @@ +//! Code sink that writes binary machine code into contiguous memory. +//! +//! The `CodeSink` trait is the most general way of extracting binary machine code from Cretonne, +//! and it is implemented by things like the `test binemit` file test driver to generate +//! hexadecimal machine code. The `CodeSink` has some undesirable performance properties because of +//! the dual abstraction: `TargetIsa` is a trait object implemented by each supported ISA, so it +//! can't have any generic functions that could be specialized for each `CodeSink` implementation. +//! This results in many virtual function callbacks (one per `put*` call) when +//! `TargetIsa::emit_inst()` is used. +//! +//! The `MemoryCodeSink` type fixes the performance problem because it is a type known to +//! `TargetIsa` so it can specialize its machine code generation for the type. The trade-off is +//! that a `MemoryCodeSink` will always write binary machine code to raw memory. It forwards any +//! relocations to a `RelocSink` trait object. Relocations are less frequent than the +//! `CodeSink::put*` methods, so the performance impact of the virtual callbacks is less severe. + +use ir::{Ebb, FuncRef, JumpTable}; +use super::{CodeSink, CodeOffset, Reloc}; +use std::ptr::write_unaligned; + +/// A `CodeSink` that writes binary machine code directly into memory. +/// +/// A `MemoryCodeSink` object should be used when emitting a Cretonne IL function into executable +/// memory. It writes machine code directly to a raw pointer without any bounds checking, so make +/// sure to allocate enough memory for the whole function. The number of bytes required is returned +/// by the `Context::compile()` function. +/// +/// Any relocations in the function are forwarded to the `RelocSink` trait object. +/// +/// Note that `MemoryCodeSink` writes multi-byte values in the native byte order of the host. This +/// is not the right thing to do for cross compilation. +pub struct MemoryCodeSink<'a> { + data: *mut u8, + offset: isize, + relocs: &'a mut RelocSink, +} + +impl<'a> MemoryCodeSink<'a> { + /// Create a new memory code sink that writes a function to the memory pointed to by `data`. + pub fn new(data: *mut u8, relocs: &mut RelocSink) -> MemoryCodeSink { + MemoryCodeSink { + data, + offset: 0, + relocs, + } + } +} + +/// A trait for receiving relocations for code that is emitted directly into memory. +pub trait RelocSink { + /// Add a relocation referencing an EBB at the current offset. + fn reloc_ebb(&mut self, CodeOffset, Reloc, Ebb); + + /// Add a relocation referencing an external function at the current offset. + fn reloc_func(&mut self, CodeOffset, Reloc, FuncRef); + + /// Add a relocation referencing a jump table. + fn reloc_jt(&mut self, CodeOffset, Reloc, JumpTable); +} + +impl<'a> CodeSink for MemoryCodeSink<'a> { + fn offset(&self) -> CodeOffset { + self.offset as CodeOffset + } + + fn put1(&mut self, x: u8) { + unsafe { + write_unaligned(self.data.offset(self.offset), x); + } + self.offset += 1; + } + + fn put2(&mut self, x: u16) { + unsafe { + write_unaligned(self.data.offset(self.offset) as *mut u16, x); + } + self.offset += 2; + } + + fn put4(&mut self, x: u32) { + unsafe { + write_unaligned(self.data.offset(self.offset) as *mut u32, x); + } + self.offset += 4; + } + + fn put8(&mut self, x: u64) { + unsafe { + write_unaligned(self.data.offset(self.offset) as *mut u64, x); + } + self.offset += 8; + } + + fn reloc_ebb(&mut self, rel: Reloc, ebb: Ebb) { + let ofs = self.offset(); + self.relocs.reloc_ebb(ofs, rel, ebb); + } + + fn reloc_func(&mut self, rel: Reloc, func: FuncRef) { + let ofs = self.offset(); + self.relocs.reloc_func(ofs, rel, func); + } + + fn reloc_jt(&mut self, rel: Reloc, jt: JumpTable) { + let ofs = self.offset(); + self.relocs.reloc_jt(ofs, rel, jt); + } +} diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index eaa721d510..02076270d0 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -4,8 +4,10 @@ //! binary machine code. mod relaxation; +mod memorysink; pub use self::relaxation::relax_branches; +pub use self::memorysink::{MemoryCodeSink, RelocSink}; use ir::{Ebb, FuncRef, JumpTable, Function, Inst}; @@ -55,3 +57,20 @@ pub fn bad_encoding(func: &Function, inst: Inst) -> ! { func.encodings[inst], func.dfg.display_inst(inst, None)); } + +/// Emit a function to `sink`, given an instruction emitter function. +/// +/// This function is called from the `TargetIsa::emit_function()` implementations with the +/// appropriate instruction emitter. +pub fn emit_function(func: &Function, emit_inst: EI, sink: &mut CS) + where CS: CodeSink, + EI: Fn(&Function, Inst, &mut CS) +{ + for ebb in func.layout.ebbs() { + assert_eq!(func.offsets[ebb], sink.offset()); + for inst in func.layout.ebb_insts(ebb) { + emit_inst(func, inst, sink); + } + + } +} diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index a87e35cd32..d0288789f6 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -9,7 +9,7 @@ //! contexts concurrently. Typically, you would have one context per compilation thread and only a //! single ISA instance. -use binemit::{CodeOffset, relax_branches}; +use binemit::{CodeOffset, relax_branches, MemoryCodeSink, RelocSink}; use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir::Function; @@ -71,6 +71,16 @@ impl Context { self.relax_branches(isa) } + /// Emit machine code directly into raw memory. + /// + /// Write all of the function's machine code to the memory at `mem`. The size of the machine + /// code is returned by `compile` above. + /// + /// The machine code is not relocated. Instead, any relocations are emitted into `relocs`. + pub fn emit_to_memory(&self, mem: *mut u8, relocs: &mut RelocSink, isa: &TargetIsa) { + isa.emit_function(&self.func, &mut MemoryCodeSink::new(mem, relocs)); + } + /// Run the verifier on the function. /// /// Also check that the dominator tree and control flow graph are consistent with the function. diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index d986b9e0e2..5591a2aece 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -6,7 +6,7 @@ mod binemit; mod enc_tables; mod registers; -use binemit::CodeSink; +use binemit::{CodeSink, MemoryCodeSink, emit_function}; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; @@ -95,6 +95,10 @@ impl TargetIsa for Isa { binemit::emit_inst(func, inst, sink) } + fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { + emit_function(func, binemit::emit_inst, sink) + } + fn reloc_names(&self) -> &'static [&'static str] { &binemit::RELOC_NAMES } diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index c8c2de3cf8..316d1957f5 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -6,7 +6,7 @@ mod binemit; mod enc_tables; mod registers; -use binemit::CodeSink; +use binemit::{CodeSink, MemoryCodeSink, emit_function}; use super::super::settings as shared_settings; use isa::enc_tables::{lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; @@ -88,6 +88,10 @@ impl TargetIsa for Isa { binemit::emit_inst(func, inst, sink) } + fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { + emit_function(func, binemit::emit_inst, sink) + } + fn reloc_names(&self) -> &'static [&'static str] { &binemit::RELOC_NAMES } diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 88df62c8ac..81e3127811 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -6,7 +6,7 @@ mod binemit; mod enc_tables; mod registers; -use binemit::CodeSink; +use binemit::{CodeSink, MemoryCodeSink, emit_function}; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; @@ -95,6 +95,10 @@ impl TargetIsa for Isa { binemit::emit_inst(func, inst, sink) } + fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { + emit_function(func, binemit::emit_inst, sink) + } + fn reloc_names(&self) -> &'static [&'static str] { &binemit::RELOC_NAMES } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index edd72b27af..0604c0f271 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -44,7 +44,7 @@ pub use isa::constraints::{RecipeConstraints, OperandConstraint, ConstraintKind, pub use isa::encoding::{Encoding, EncInfo}; pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex, regs_overlap}; -use binemit::CodeSink; +use binemit; use settings; use ir; use regalloc; @@ -215,7 +215,12 @@ pub trait TargetIsa { /// /// Note that this will call `put*` methods on the trait object via its vtable which is not the /// fastest way of emitting code. - fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink); + fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut binemit::CodeSink); + + /// Emit a whole function into memory. + /// + /// This is more performant than calling `emit_inst` for each instruction. + fn emit_function(&self, func: &ir::Function, sink: &mut binemit::MemoryCodeSink); /// Get a static array of names associated with relocations in this ISA. /// diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 246ec13bd1..6be2432a35 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -7,7 +7,7 @@ mod enc_tables; mod registers; use super::super::settings as shared_settings; -use binemit::CodeSink; +use binemit::{CodeSink, MemoryCodeSink, emit_function}; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Legalize}; @@ -95,6 +95,10 @@ impl TargetIsa for Isa { binemit::emit_inst(func, inst, sink) } + fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { + emit_function(func, binemit::emit_inst, sink) + } + fn reloc_names(&self) -> &'static [&'static str] { &binemit::RELOC_NAMES } From cf876e492adc8dfd1e40425789904cbd132a6710 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 18 Jul 2017 09:22:50 -0700 Subject: [PATCH 0878/3084] Add Intel encodings for imul. --- cranelift/filetests/isa/intel/binary32.cton | 7 +++++++ cranelift/filetests/isa/intel/binary64.cton | 18 ++++++++++++++++++ cranelift/filetests/wasm/i32-arith.cton | 7 ++++++- lib/cretonne/meta/isa/intel/encodings.py | 5 +++++ lib/cretonne/meta/isa/intel/recipes.py | 8 ++++++++ lib/cretonne/src/isa/intel/binemit.rs | 8 ++++++++ 6 files changed, 52 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index bd3ff06289..0eb136b5d6 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -105,6 +105,13 @@ ebb0: ; asm: xorl $1000000, %esi [-,%rsi] v47 = bxor_imm v2, 1000000 ; bin: 81 f6 000f4240 + ; More arithmetic. + + ; asm: imull %esi, %ecx + [-,%rcx] v50 = imul v1, v2 ; bin: 0f af ce + ; asm: imull %ecx, %esi + [-,%rsi] v51 = imul v2, v1 ; bin: 0f af f1 + ; Register copies. ; asm: movl %esi, %ecx diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index ecb4b8a40d..5e3507d473 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -145,6 +145,15 @@ ebb0: ; asm: movq %rcx, %r10 [-,%r10] v112 = copy v1 ; bin: 49 89 ca + ; More arithmetic. + + ; asm: imulq %rsi, %rcx + [-,%rcx] v120 = imul v1, v2 ; bin: 48 0f af ce + ; asm: imulq %r10, %rsi + [-,%rsi] v121 = imul v2, v3 ; bin: 49 0f af f2 + ; asm: imulq %rcx, %r10 + [-,%r10] v122 = imul v3, v1 ; bin: 4c 0f af d1 + ; Bit-counting instructions. ; asm: popcntq %rsi, %rcx @@ -313,6 +322,15 @@ ebb0: ; asm: movl %ecx, %r10d [-,%r10] v112 = copy v1 ; bin: 41 89 ca + ; More arithmetic. + + ; asm: imull %esi, %ecx + [-,%rcx] v120 = imul v1, v2 ; bin: 40 0f af ce + ; asm: imull %r10d, %esi + [-,%rsi] v121 = imul v2, v3 ; bin: 41 0f af f2 + ; asm: imull %ecx, %r10d + [-,%r10] v122 = imul v3, v1 ; bin: 44 0f af d1 + ; Bit-counting instructions. ; asm: popcntl %esi, %ecx diff --git a/cranelift/filetests/wasm/i32-arith.cton b/cranelift/filetests/wasm/i32-arith.cton index bddec1b52b..b345847eb9 100644 --- a/cranelift/filetests/wasm/i32-arith.cton +++ b/cranelift/filetests/wasm/i32-arith.cton @@ -49,7 +49,12 @@ ebb0(v0: i32, v1: i32): return v2 } -; function %i32_mul(i32, i32) -> i32 +function %i32_mul(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = imul v0, v1 + return v2 +} + ; function %i32_div(i32, i32) -> i32 ; function %i32_rem_s(i32, i32) -> i32 ; function %i32_rem_u(i32, i32) -> i32 diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index c2c361ea86..f6f003bb1e 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -23,6 +23,11 @@ for inst, opc in [ # default. Otherwise reg-alloc would never use r8 and up. I64.enc(inst.i32, *r.rr(opc)) +I32.enc(base.imul.i32, *r.rrx(0x0f, 0xaf)) +I64.enc(base.imul.i64, *r.rrx.rex(0x0f, 0xaf, w=1)) +I64.enc(base.imul.i32, *r.rrx.rex(0x0f, 0xaf)) +I64.enc(base.imul.i32, *r.rrx(0x0f, 0xaf)) + I32.enc(base.copy.i32, *r.umr(0x89)) I64.enc(base.copy.i64, *r.umr.rex(0x89, w=1)) I64.enc(base.copy.i32, *r.umr.rex(0x89)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index b7e6aa5575..e521b205b1 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -197,6 +197,14 @@ rr = TailRecipe( modrm_rr(in_reg0, in_reg1, sink); ''') +# XX /r with operands swapped. (RM form). +rrx = TailRecipe( + 'rrx', Binary, size=1, ins=(GPR, GPR), outs=0, + emit=''' + PUT_OP(bits, rex2(in_reg1, in_reg0), sink); + modrm_rr(in_reg1, in_reg0, sink); + ''') + # XX /r, but for a unary operator with separate input/output register, like # copies. MR form. umr = TailRecipe( diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 56f0c2f7f0..d3c9716680 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -76,6 +76,14 @@ fn put_op2(bits: u16, rex: u8, sink: &mut CS) { sink.put1(bits as u8); } +// Emit two-byte opcode: 0F XX with REX prefix. +fn put_rexop2(bits: u16, rex: u8, sink: &mut CS) { + debug_assert_eq!(bits & 0x0f00, 0x0400, "Invalid encoding bits for RexOp2*"); + rex_prefix(bits, rex, sink); + sink.put1(0x0f); + sink.put1(bits as u8); +} + // Emit single-byte opcode with mandatory prefix. fn put_mp1(bits: u16, rex: u8, sink: &mut CS) { debug_assert_eq!(bits & 0x8c00, 0, "Invalid encoding bits for Mp1*"); From c4db4c124b2fcecdcf714de68326c97193f5d3a6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 18 Jul 2017 10:09:02 -0700 Subject: [PATCH 0879/3084] Begin an Intel-specific instruction group. Add instructions representing Intel's division instructions which use a numerator that is twice as wide as the denominator and produce both the quotient and remainder. Add encodings for the x86_[su]divmodx instructions. --- cranelift/docs/cton_domain.py | 5 ++- cranelift/docs/langref.rst | 22 ++++++++-- cranelift/filetests/isa/intel/binary32.cton | 13 ++++++ cranelift/filetests/isa/intel/binary64.cton | 30 ++++++++++++++ lib/cretonne/meta/isa/intel/defs.py | 3 +- lib/cretonne/meta/isa/intel/encodings.py | 9 +++++ lib/cretonne/meta/isa/intel/instructions.py | 45 +++++++++++++++++++++ lib/cretonne/meta/isa/intel/recipes.py | 11 ++++- 8 files changed, 132 insertions(+), 6 deletions(-) create mode 100644 lib/cretonne/meta/isa/intel/instructions.py diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index 3dba386d90..a03acf1160 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -271,7 +271,10 @@ class InstDocumenter(sphinx.ext.autodoc.Documenter): return False def resolve_name(self, modname, parents, path, base): - return 'base.instructions', [base] + if path: + return path.rstrip('.'), [base] + else: + return 'base.instructions', [base] def format_signature(self): inst = self.object diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 49593e3f09..dde7865486 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -775,15 +775,31 @@ the target ISA. .. autoinst:: isplit .. autoinst:: iconcat -Base instruction group -====================== +ISA-specific instructions +========================= + +Target ISAs can define supplemental instructions that do not make sense to +support generally. + +Intel +----- + +Instructions that can only be used by the Intel target ISA. + +.. autoinst:: isa.intel.instructions.sdivmodx +.. autoinst:: isa.intel.instructions.udivmodx + +Instruction groups +================== All of the shared instructions are part of the :instgroup:`base` instruction group. .. autoinstgroup:: base.instructions.GROUP -Target ISAs may define further instructions in their own instruction groups. +Target ISAs may define further instructions in their own instruction groups: + +.. autoinstgroup:: isa.intel.instructions.GROUP Implementation limits ===================== diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 0eb136b5d6..67f37db11b 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -112,6 +112,19 @@ ebb0: ; asm: imull %ecx, %esi [-,%rsi] v51 = imul v2, v1 ; bin: 0f af f1 + ; asm: movl $1, %eax + [-,%rax] v52 = iconst.i32 1 ; bin: b8 00000001 + ; asm: movl $2, %edx + [-,%rdx] v53 = iconst.i32 2 ; bin: ba 00000002 + ; asm: idivl %ecx + [-,%rax,%rdx] v54, v55 = x86_sdivmodx v52, v53, v1 ; bin: f7 f9 + ; asm: idivl %esi + [-,%rax,%rdx] v56, v57 = x86_sdivmodx v52, v53, v2 ; bin: f7 fe + ; asm: divl %ecx + [-,%rax,%rdx] v58, v59 = x86_udivmodx v52, v53, v1 ; bin: f7 f1 + ; asm: divl %esi + [-,%rax,%rdx] v60, v61 = x86_udivmodx v52, v53, v2 ; bin: f7 f6 + ; Register copies. ; asm: movl %esi, %ecx diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 5e3507d473..bed934d5ab 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -154,6 +154,21 @@ ebb0: ; asm: imulq %rcx, %r10 [-,%r10] v122 = imul v3, v1 ; bin: 4c 0f af d1 + [-,%rax] v130 = iconst.i64 1 + [-,%rdx] v131 = iconst.i64 2 + ; asm: idivq %rcx + [-,%rax,%rdx] v132, v133 = x86_sdivmodx v130, v131, v1 ; bin: 48 f7 f9 + ; asm: idivq %rsi + [-,%rax,%rdx] v134, v135 = x86_sdivmodx v130, v131, v2 ; bin: 48 f7 fe + ; asm: idivq %r10 + [-,%rax,%rdx] v136, v137 = x86_sdivmodx v130, v131, v3 ; bin: 49 f7 fa + ; asm: divq %rcx + [-,%rax,%rdx] v138, v139 = x86_udivmodx v130, v131, v1 ; bin: 48 f7 f1 + ; asm: divq %rsi + [-,%rax,%rdx] v140, v141 = x86_udivmodx v130, v131, v2 ; bin: 48 f7 f6 + ; asm: divq %r10 + [-,%rax,%rdx] v142, v143 = x86_udivmodx v130, v131, v3 ; bin: 49 f7 f2 + ; Bit-counting instructions. ; asm: popcntq %rsi, %rcx @@ -331,6 +346,21 @@ ebb0: ; asm: imull %ecx, %r10d [-,%r10] v122 = imul v3, v1 ; bin: 44 0f af d1 + [-,%rax] v130 = iconst.i32 1 + [-,%rdx] v131 = iconst.i32 2 + ; asm: idivl %rcx + [-,%rax,%rdx] v132, v133 = x86_sdivmodx v130, v131, v1 ; bin: 40 f7 f9 + ; asm: idivl %rsi + [-,%rax,%rdx] v134, v135 = x86_sdivmodx v130, v131, v2 ; bin: 40 f7 fe + ; asm: idivl %r10d + [-,%rax,%rdx] v136, v137 = x86_sdivmodx v130, v131, v3 ; bin: 41 f7 fa + ; asm: divl %rcx + [-,%rax,%rdx] v138, v139 = x86_udivmodx v130, v131, v1 ; bin: 40 f7 f1 + ; asm: divl %rsi + [-,%rax,%rdx] v140, v141 = x86_udivmodx v130, v131, v2 ; bin: 40 f7 f6 + ; asm: divl %r10d + [-,%rax,%rdx] v142, v143 = x86_udivmodx v130, v131, v3 ; bin: 41 f7 f2 + ; Bit-counting instructions. ; asm: popcntl %esi, %ecx diff --git a/lib/cretonne/meta/isa/intel/defs.py b/lib/cretonne/meta/isa/intel/defs.py index 50251b44af..5078bcec7e 100644 --- a/lib/cretonne/meta/isa/intel/defs.py +++ b/lib/cretonne/meta/isa/intel/defs.py @@ -6,8 +6,9 @@ Commonly used definitions. from __future__ import absolute_import from cdsl.isa import TargetISA, CPUMode import base.instructions +from . import instructions as x86 -ISA = TargetISA('intel', [base.instructions.GROUP]) +ISA = TargetISA('intel', [base.instructions.GROUP, x86.GROUP]) # CPU modes for 32-bit and 64-bit operation. I32 = CPUMode('I32', ISA) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index f6f003bb1e..25421be999 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -8,6 +8,7 @@ from base.formats import UnaryImm from .defs import I32, I64 from . import recipes as r from . import settings as cfg +from . import instructions as x86 for inst, opc in [ (base.iadd, 0x01), @@ -28,6 +29,14 @@ I64.enc(base.imul.i64, *r.rrx.rex(0x0f, 0xaf, w=1)) I64.enc(base.imul.i32, *r.rrx.rex(0x0f, 0xaf)) I64.enc(base.imul.i32, *r.rrx(0x0f, 0xaf)) +for inst, rrr in [ + (x86.sdivmodx, 7), + (x86.udivmodx, 6)]: + I32.enc(inst.i32, *r.div(0xf7, rrr=rrr)) + I64.enc(inst.i64, *r.div.rex(0xf7, rrr=rrr, w=1)) + I64.enc(inst.i32, *r.div.rex(0xf7, rrr=rrr)) + I64.enc(inst.i32, *r.div(0xf7, rrr=rrr)) + I32.enc(base.copy.i32, *r.umr(0x89)) I64.enc(base.copy.i64, *r.umr.rex(0x89, w=1)) I64.enc(base.copy.i32, *r.umr.rex(0x89)) diff --git a/lib/cretonne/meta/isa/intel/instructions.py b/lib/cretonne/meta/isa/intel/instructions.py new file mode 100644 index 0000000000..a5c2bdafb0 --- /dev/null +++ b/lib/cretonne/meta/isa/intel/instructions.py @@ -0,0 +1,45 @@ +""" +Supplementary instruction definitions for Intel. + +This module defines additional instructions that are useful only to the Intel +target ISA. +""" + +from cdsl.operands import Operand +from cdsl.instructions import Instruction, InstructionGroup +from base.instructions import iB + + +GROUP = InstructionGroup("x86", "Intel-specific instruction set") + +nlo = Operand('nlo', iB, doc='Low part of numerator') +nhi = Operand('nhi', iB, doc='High part of numerator') +d = Operand('d', iB, doc='Denominator') +q = Operand('q', iB, doc='Quotient') +r = Operand('r', iB, doc='Remainder') + +udivmodx = Instruction( + 'x86_udivmodx', r""" + Extended unsigned division. + + Concatenate the bits in `nhi` and `nlo` to form the numerator. + Interpret the bits as an unsigned number and divide by the unsigned + denominator `d`. Trap when `d` is zero or if the quotient is larger + than the range of the output. + + Return both quotient and remainder. + """, + ins=(nlo, nhi, d), outs=(q, r), can_trap=True) + +sdivmodx = Instruction( + 'x86_sdivmodx', r""" + Extended signed division. + + Concatenate the bits in `nhi` and `nlo` to form the numerator. + Interpret the bits as a signed number and divide by the signed + denominator `d`. Trap when `d` is zero or if the quotient is outside + the range of the output. + + Return both quotient and remainder. + """, + ins=(nlo, nhi, d), outs=(q, r), can_trap=True) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index e521b205b1..9c48a529d2 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -6,7 +6,7 @@ from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt, IsEqual from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry from base.formats import Call, IndirectCall, Store, Load -from base.formats import RegMove +from base.formats import RegMove, Ternary from .registers import GPR, ABCD try: @@ -239,6 +239,15 @@ rc = TailRecipe( modrm_r_bits(in_reg0, bits, sink); ''') +# XX /n for division: inputs in %rax, %rdx, r. Outputs in %rax, %rdx. +div = TailRecipe( + 'div', Ternary, size=1, + ins=(GPR.rax, GPR.rdx, GPR), outs=(GPR.rax, GPR.rdx), + emit=''' + PUT_OP(bits, rex1(in_reg2), sink); + modrm_r_bits(in_reg2, bits, sink); + ''') + # XX /n ib with 8-bit immediate sign-extended. rib = TailRecipe( 'rib', BinaryImm, size=2, ins=GPR, outs=0, From 53d9232d39daea3a9776e2db5e928239e4718b7e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 18 Jul 2017 12:52:53 -0700 Subject: [PATCH 0880/3084] Track regmove instruction during binemit. Register locations can change throughout an EBB. Make sure the emit_inst() function considers this when encoding instructions and update the register diversion tracker. --- cranelift/filetests/isa/riscv/regmove.cton | 15 +++++++++++++++ cranelift/src/filetest/binemit.rs | 5 ++++- cranelift/src/filetest/compile.rs | 2 +- lib/cretonne/meta/gen_binemit.py | 20 +++++++++++++++----- lib/cretonne/src/binemit/mod.rs | 8 +++++--- lib/cretonne/src/isa/arm32/binemit.rs | 1 + lib/cretonne/src/isa/arm32/mod.rs | 8 ++++++-- lib/cretonne/src/isa/arm64/binemit.rs | 1 + lib/cretonne/src/isa/arm64/mod.rs | 8 ++++++-- lib/cretonne/src/isa/intel/binemit.rs | 1 + lib/cretonne/src/isa/intel/mod.rs | 8 ++++++-- lib/cretonne/src/isa/mod.rs | 6 +++++- lib/cretonne/src/isa/riscv/binemit.rs | 1 + lib/cretonne/src/isa/riscv/mod.rs | 8 ++++++-- 14 files changed, 73 insertions(+), 19 deletions(-) create mode 100644 cranelift/filetests/isa/riscv/regmove.cton diff --git a/cranelift/filetests/isa/riscv/regmove.cton b/cranelift/filetests/isa/riscv/regmove.cton new file mode 100644 index 0000000000..c316f74f21 --- /dev/null +++ b/cranelift/filetests/isa/riscv/regmove.cton @@ -0,0 +1,15 @@ +; Test tracking of register moves. +test binemit +isa riscv + +function %regmoves(i32 link [%x1]) -> i32 link [%x1] { +ebb0(v9999: i32): + [-,%x10] v1 = iconst.i32 1 + [-,%x7] v2 = iadd_imm v1, 1000 ; bin: 3e850393 + regmove v1, %x10 -> %x11 ; bin: 00050593 + [-,%x7] v3 = iadd_imm v1, 1000 ; bin: 3e858393 + regmove v1, %x11 -> %x10 ; bin: 00058513 + [-,%x7] v4 = iadd_imm v1, 1000 ; bin: 3e850393 + + return v9999 +} diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index d359cea794..09a6955b5e 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -10,6 +10,7 @@ use cretonne::binemit; use cretonne::ir; use cretonne::ir::entities::AnyEntity; use cretonne::isa::TargetIsa; +use cretonne::regalloc::RegDiversions; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result}; use utils::{match_directive, pretty_error}; @@ -147,7 +148,9 @@ impl SubTest for TestBinEmit { // Now emit all instructions. let mut sink = TextSink::new(isa); + let mut divert = RegDiversions::new(); for ebb in func.layout.ebbs() { + divert.clear(); // Correct header offsets should have been computed by `relax_branches()`. assert_eq!(sink.offset, func.offsets[ebb], @@ -160,7 +163,7 @@ impl SubTest for TestBinEmit { // Send legal encodings into the emitter. if enc.is_legal() { let before = sink.offset; - isa.emit_inst(&func, inst, &mut sink); + isa.emit_inst(&func, inst, &mut divert, &mut sink); let emitted = sink.offset - before; // Verify the encoding recipe sizes against the ISAs emit_inst implementation. assert_eq!(emitted, diff --git a/cranelift/src/filetest/compile.rs b/cranelift/src/filetest/compile.rs index f3766f001f..5d00ee63c2 100644 --- a/cranelift/src/filetest/compile.rs +++ b/cranelift/src/filetest/compile.rs @@ -52,7 +52,7 @@ impl SubTest for TestCompile { // Finally verify that the returned code size matches the emitted bytes. let mut sink = SizeSink { offset: 0 }; binemit::emit_function(&comp_ctx.func, - |func, inst, sink| isa.emit_inst(func, inst, sink), + |func, inst, div, sink| isa.emit_inst(func, inst, div, sink), &mut sink); if sink.offset != code_size { diff --git a/lib/cretonne/meta/gen_binemit.py b/lib/cretonne/meta/gen_binemit.py index 858fc36b39..8e734c5ff1 100644 --- a/lib/cretonne/meta/gen_binemit.py +++ b/lib/cretonne/meta/gen_binemit.py @@ -31,6 +31,9 @@ def gen_recipe(recipe, fmt): want_outs = any(isinstance(o, RegClass) or isinstance(o, Stack) for o in recipe.outs) + # Regmove instructions get special treatment. + is_regmove = (recipe.format.name == 'RegMove') + # First unpack the instruction. with fmt.indented( 'if let InstructionData::{} {{'.format(iform.name), @@ -46,7 +49,7 @@ def gen_recipe(recipe, fmt): fmt.outdented_line('} = func.dfg[inst] {') # Normalize to an `args` array. - if want_args: + if want_args and not is_regmove: if iform.has_value_list: fmt.line('let args = args.as_slice(&func.dfg.value_lists);') elif nvops == 1: @@ -56,11 +59,11 @@ def gen_recipe(recipe, fmt): # Don't bother with fixed registers. args = '' for i, arg in enumerate(recipe.ins): - if isinstance(arg, RegClass): + if isinstance(arg, RegClass) and not is_regmove: v = 'in_reg{}'.format(i) args += ', ' + v fmt.line( - 'let {} = func.locations[args[{}]].unwrap_reg();' + 'let {} = divert.reg(args[{}], &func.locations);' .format(v, i)) elif isinstance(arg, Stack): v = 'in_ss{}'.format(i) @@ -93,6 +96,11 @@ def gen_recipe(recipe, fmt): 'let {} = func.locations[results[{}]].unwrap_stack();' .format(v, i)) + # Special handling for regmove instructions. Update the register + # diversion tracker. + if recipe.format.name == 'RegMove': + fmt.line('divert.regmove(arg, src, dst);') + # Call hand-written code. If the recipe contains a code snippet, use # that. Otherwise cal a recipe function in the target ISA's binemit # module. @@ -118,13 +126,15 @@ def gen_isa(isa, fmt): # No encoding recipes: Emit a stub. with fmt.indented( 'pub fn emit_inst' - '(func: &Function, inst: Inst, _sink: &mut CS) {', '}'): + '(func: &Function, inst: Inst, ' + '_divert: &mut RegDiversions, _sink: &mut CS) {', '}'): fmt.line('bad_encoding(func, inst)') else: fmt.line('#[allow(unused_variables, unreachable_code)]') with fmt.indented( 'pub fn emit_inst' - '(func: &Function, inst: Inst, sink: &mut CS) {', '}'): + '(func: &Function, inst: Inst, ' + 'divert: &mut RegDiversions, sink: &mut CS) {', '}'): fmt.line('let bits = func.encodings[inst].bits();') with fmt.indented('match func.encodings[inst].recipe() {', '}'): for i, recipe in enumerate(isa.all_recipes): diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 02076270d0..a4a62f869f 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -10,6 +10,7 @@ pub use self::relaxation::relax_branches; pub use self::memorysink::{MemoryCodeSink, RelocSink}; use ir::{Ebb, FuncRef, JumpTable, Function, Inst}; +use regalloc::RegDiversions; /// Offset in bytes from the beginning of the function. /// @@ -64,13 +65,14 @@ pub fn bad_encoding(func: &Function, inst: Inst) -> ! { /// appropriate instruction emitter. pub fn emit_function(func: &Function, emit_inst: EI, sink: &mut CS) where CS: CodeSink, - EI: Fn(&Function, Inst, &mut CS) + EI: Fn(&Function, Inst, &mut RegDiversions, &mut CS) { + let mut divert = RegDiversions::new(); for ebb in func.layout.ebbs() { + divert.clear(); assert_eq!(func.offsets[ebb], sink.offset()); for inst in func.layout.ebb_insts(ebb) { - emit_inst(func, inst, sink); + emit_inst(func, inst, &mut divert, sink); } - } } diff --git a/lib/cretonne/src/isa/arm32/binemit.rs b/lib/cretonne/src/isa/arm32/binemit.rs index cf12bdbde2..bbae03432c 100644 --- a/lib/cretonne/src/isa/arm32/binemit.rs +++ b/lib/cretonne/src/isa/arm32/binemit.rs @@ -2,6 +2,7 @@ use binemit::{CodeSink, bad_encoding}; use ir::{Function, Inst}; +use regalloc::RegDiversions; include!(concat!(env!("OUT_DIR"), "/binemit-arm32.rs")); diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 5591a2aece..769adc376d 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -91,8 +91,12 @@ impl TargetIsa for Isa { abi::allocatable_registers(func) } - fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { - binemit::emit_inst(func, inst, sink) + fn emit_inst(&self, + func: &ir::Function, + inst: ir::Inst, + divert: &mut regalloc::RegDiversions, + sink: &mut CodeSink) { + binemit::emit_inst(func, inst, divert, sink) } fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { diff --git a/lib/cretonne/src/isa/arm64/binemit.rs b/lib/cretonne/src/isa/arm64/binemit.rs index 120115c0d8..ecff1662bc 100644 --- a/lib/cretonne/src/isa/arm64/binemit.rs +++ b/lib/cretonne/src/isa/arm64/binemit.rs @@ -2,6 +2,7 @@ use binemit::{CodeSink, bad_encoding}; use ir::{Function, Inst}; +use regalloc::RegDiversions; include!(concat!(env!("OUT_DIR"), "/binemit-arm64.rs")); diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 316d1957f5..a1b0ac2478 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -84,8 +84,12 @@ impl TargetIsa for Isa { abi::allocatable_registers(func) } - fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { - binemit::emit_inst(func, inst, sink) + fn emit_inst(&self, + func: &ir::Function, + inst: ir::Inst, + divert: &mut regalloc::RegDiversions, + sink: &mut CodeSink) { + binemit::emit_inst(func, inst, divert, sink) } fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index d3c9716680..61c6f512a9 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -3,6 +3,7 @@ use binemit::{CodeSink, Reloc, bad_encoding}; use ir::{Function, Inst, InstructionData}; use isa::RegUnit; +use regalloc::RegDiversions; include!(concat!(env!("OUT_DIR"), "/binemit-intel.rs")); diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 81e3127811..93a2ed2c09 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -91,8 +91,12 @@ impl TargetIsa for Isa { abi::allocatable_registers(func, &self.shared_flags) } - fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { - binemit::emit_inst(func, inst, sink) + fn emit_inst(&self, + func: &ir::Function, + inst: ir::Inst, + divert: &mut regalloc::RegDiversions, + sink: &mut CodeSink) { + binemit::emit_inst(func, inst, divert, sink) } fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 0604c0f271..6665de5f42 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -215,7 +215,11 @@ pub trait TargetIsa { /// /// Note that this will call `put*` methods on the trait object via its vtable which is not the /// fastest way of emitting code. - fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut binemit::CodeSink); + fn emit_inst(&self, + func: &ir::Function, + inst: ir::Inst, + divert: &mut regalloc::RegDiversions, + sink: &mut binemit::CodeSink); /// Emit a whole function into memory. /// diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index d193981887..86cbb6b82c 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -4,6 +4,7 @@ use binemit::{CodeSink, Reloc, bad_encoding}; use ir::{Function, Inst, InstructionData}; use isa::RegUnit; use predicates::is_signed_int; +use regalloc::RegDiversions; include!(concat!(env!("OUT_DIR"), "/binemit-riscv.rs")); diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 6be2432a35..b70f6185b8 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -91,8 +91,12 @@ impl TargetIsa for Isa { abi::allocatable_registers(func, &self.isa_flags) } - fn emit_inst(&self, func: &ir::Function, inst: ir::Inst, sink: &mut CodeSink) { - binemit::emit_inst(func, inst, sink) + fn emit_inst(&self, + func: &ir::Function, + inst: ir::Inst, + divert: &mut regalloc::RegDiversions, + sink: &mut CodeSink) { + binemit::emit_inst(func, inst, divert, sink) } fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { From efdbf0d73567d2d046b048dd4be31ac9e48edee8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 18 Jul 2017 14:54:34 -0700 Subject: [PATCH 0881/3084] Add Intel encodings for jump and branch instructions. Just implement jump, brz, and brnz as needed for WebAssembly. --- cranelift/filetests/isa/intel/binary32.cton | 22 ++++++++ cranelift/filetests/isa/intel/binary64.cton | 56 +++++++++++++++++++++ cranelift/filetests/wasm/control.cton | 34 +++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 18 +++++++ lib/cretonne/meta/isa/intel/recipes.py | 46 ++++++++++++++++- lib/cretonne/src/isa/intel/binemit.rs | 14 +++++- 6 files changed, 188 insertions(+), 2 deletions(-) create mode 100644 cranelift/filetests/wasm/control.cton diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 67f37db11b..7b43ed7aa2 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -262,6 +262,28 @@ ebb0: ; asm: call *%esi call_indirect sig0, v2() ; bin: ff d6 + ; asm: testl %ecx, %ecx + ; asm: je ebb1 + brz v1, ebb1 ; bin: 85 c9 74 0e + ; asm: testl %esi, %esi + ; asm: je ebb1 + brz v2, ebb1 ; bin: 85 f6 74 0a + ; asm: testl %ecx, %ecx + ; asm: jne ebb1 + brnz v1, ebb1 ; bin: 85 c9 75 06 + ; asm: testl %esi, %esi + ; asm: jne ebb1 + brnz v2, ebb1 ; bin: 85 f6 75 02 + + ; asm: jmp ebb2 + jump ebb2 ; bin: eb 01 + + ; asm: ebb1: +ebb1: ; asm: ret return ; bin: c3 + + ; asm: ebb2: +ebb2: + jump ebb1 ; bin: eb fd } diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index bed934d5ab..1d88b69d50 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -192,7 +192,35 @@ ebb0: ; asm: tzcntq %rcx, %r10 [-,%r10] v208 = ctz v1 ; bin: f3 4c 0f bc d1 + ; asm: testq %rcx, %ecx + ; asm: je ebb1 + brz v1, ebb1 ; bin: 48 85 c9 74 1b + ; asm: testq %rsi, %esi + ; asm: je ebb1 + brz v2, ebb1 ; bin: 48 85 f6 74 16 + ; asm: testq %r10, %r10d + ; asm: je ebb1 + brz v3, ebb1 ; bin: 4d 85 d2 74 11 + ; asm: testq %rcx, %ecx + ; asm: jne ebb1 + brnz v1, ebb1 ; bin: 48 85 c9 75 0c + ; asm: test %rsi, %esi + ; asm: jne ebb1 + brnz v2, ebb1 ; bin: 48 85 f6 75 07 + ; asm: testq %r10, %r10d + ; asm: jne ebb1 + brnz v3, ebb1 ; bin: 4d 85 d2 75 02 + + ; asm: jmp ebb2 + jump ebb2 ; bin: eb 01 + + ; asm: ebb1: +ebb1: return ; bin: c3 + + ; asm: ebb2: +ebb2: + jump ebb1 ; bin: eb fd } ; Tests for i32 instructions in 64-bit mode. @@ -384,5 +412,33 @@ ebb0: ; asm: tzcntl %ecx, %r10d [-,%r10] v208 = ctz v1 ; bin: f3 44 0f bc d1 + ; asm: testl %ecx, %ecx + ; asm: je ebb1 + brz v1, ebb1 ; bin: 40 85 c9 74 1b + ; asm: testl %esi, %esi + ; asm: je ebb1 + brz v2, ebb1 ; bin: 40 85 f6 74 16 + ; asm: testl %r10d, %r10d + ; asm: je ebb1 + brz v3, ebb1 ; bin: 45 85 d2 74 11 + ; asm: testl %ecx, %ecx + ; asm: jne ebb1 + brnz v1, ebb1 ; bin: 40 85 c9 75 0c + ; asm: test %esi, %esi + ; asm: jne ebb1 + brnz v2, ebb1 ; bin: 40 85 f6 75 07 + ; asm: testl %r10d, %r10d + ; asm: jne ebb1 + brnz v3, ebb1 ; bin: 45 85 d2 75 02 + + ; asm: jmp ebb2 + jump ebb2 ; bin: eb 01 + + ; asm: ebb1: +ebb1: return ; bin: c3 + + ; asm: ebb2: +ebb2: + jump ebb1 ; bin: eb fd } diff --git a/cranelift/filetests/wasm/control.cton b/cranelift/filetests/wasm/control.cton new file mode 100644 index 0000000000..f29c19b5c5 --- /dev/null +++ b/cranelift/filetests/wasm/control.cton @@ -0,0 +1,34 @@ +; Test basic code generation for control flow WebAssembly instructions. +test compile + +set is_64bit=0 +isa intel haswell + +set is_64bit=1 +isa intel haswell + +function %br_if(i32) -> i32 { +ebb0(v0: i32): + v1 = iconst.i32 1 + brz v0, ebb1(v1) + jump ebb2 + +ebb1(v2: i32): + return v2 + +ebb2: + jump ebb1(v0) +} + +function %br_if_not(i32) -> i32 { +ebb0(v0: i32): + v1 = iconst.i32 1 + brnz v0, ebb1(v0) + jump ebb2 + +ebb1(v2: i32): + return v2 + +ebb2: + jump ebb1(v0) +} diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 25421be999..f5e8aabd50 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -156,3 +156,21 @@ I32.enc(base.call, *r.call_id(0xe8)) I32.enc(base.call_indirect.i32, *r.call_r(0xff, rrr=2)) I32.enc(base.x_return, *r.ret(0xc3)) I64.enc(base.x_return, *r.ret(0xc3)) + +# +# Branches +# +I32.enc(base.jump, *r.jmpb(0xeb)) +I32.enc(base.jump, *r.jmpd(0xe9)) +I64.enc(base.jump, *r.jmpb(0xeb)) +I64.enc(base.jump, *r.jmpd(0xe9)) + +I32.enc(base.brz.i32, *r.tjccb(0x74)) +I64.enc(base.brz.i64, *r.tjccb.rex(0x74, w=1)) +I64.enc(base.brz.i32, *r.tjccb.rex(0x74)) +I64.enc(base.brz.i32, *r.tjccb(0x74)) + +I32.enc(base.brnz.i32, *r.tjccb(0x75)) +I64.enc(base.brnz.i64, *r.tjccb.rex(0x75, w=1)) +I64.enc(base.brnz.i32, *r.tjccb.rex(0x75)) +I64.enc(base.brnz.i32, *r.tjccb(0x75)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 9c48a529d2..f68a8344b3 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -6,7 +6,7 @@ from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt, IsEqual from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry from base.formats import Call, IndirectCall, Store, Load -from base.formats import RegMove, Ternary +from base.formats import RegMove, Ternary, Jump, Branch from .registers import GPR, ABCD try: @@ -420,3 +420,47 @@ ret = TailRecipe( emit=''' PUT_OP(bits, BASE_REX, sink); ''') + +# +# Branches +# +jmpb = TailRecipe( + 'jmpb', Jump, size=1, ins=(), outs=(), + branch_range=(2, 8), + emit=''' + PUT_OP(bits, BASE_REX, sink); + disp1(destination, func, sink); + ''') + +jmpd = TailRecipe( + 'jmpd', Jump, size=4, ins=(), outs=(), + branch_range=(5, 32), + emit=''' + PUT_OP(bits, BASE_REX, sink); + disp4(destination, func, sink); + ''') + +# Test-and-branch. +# +# This recipe represents the macro fusion of a test and a conditional branch. +# This serves two purposes: +# +# 1. Guarantee that the test and branch get scheduled next to each other so +# macro fusion is guaranteed to be possible. +# 2. Hide the status flags from Cretonne which doesn't currently model flags. +# +# The encoding bits affect both the test and the branch instruction: +# +# Bits 0-7 are the Jcc opcode. +# Bits 8-15 control the test instruction which always has opcode byte 0x85. +tjccb = TailRecipe( + 'tjcc', Branch, size=1 + 2, ins=GPR, outs=(), + branch_range=(2, 8), + emit=''' + // test r, r. + PUT_OP((bits & 0xff00) | 0x85, rex2(in_reg0, in_reg0), sink); + modrm_rr(in_reg0, in_reg0, sink); + // Jcc instruction. + sink.put1(bits as u8); + disp1(destination, func, sink); + ''') diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 61c6f512a9..0f3b1b76c6 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -1,7 +1,7 @@ //! Emitting binary Intel machine code. use binemit::{CodeSink, Reloc, bad_encoding}; -use ir::{Function, Inst, InstructionData}; +use ir::{Function, Inst, Ebb, InstructionData}; use isa::RegUnit; use regalloc::RegDiversions; @@ -169,3 +169,15 @@ fn modrm_disp32(rm: RegUnit, reg: RegUnit, sink: &mut CS) b |= rm; sink.put1(b); } + +/// Emit a single-byte branch displacement to `destination`. +fn disp1(destination: Ebb, func: &Function, sink: &mut CS) { + let delta = func.offsets[destination].wrapping_sub(sink.offset() + 1); + sink.put1(delta as u8); +} + +/// Emit a single-byte branch displacement to `destination`. +fn disp4(destination: Ebb, func: &Function, sink: &mut CS) { + let delta = func.offsets[destination].wrapping_sub(sink.offset() + 4); + sink.put4(delta); +} From c9bbc1e86edb4364d22aa9671f0e6cd8942eeaba Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Jul 2017 09:30:04 -0700 Subject: [PATCH 0882/3084] Don't require that the fallthrough instruction has an encoding. A fallthrough jump is actually represented as 0 bytes, so no encoding is needed. Also allow for unencoded instructions in the generated emit_inst implementations. The verifier has stricter rules for when this is allowed. --- cranelift/filetests/wasm/control.cton | 11 +++++++++++ lib/cretonne/meta/gen_binemit.py | 8 ++++++-- lib/cretonne/src/verifier/mod.rs | 8 +++++++- 3 files changed, 24 insertions(+), 3 deletions(-) diff --git a/cranelift/filetests/wasm/control.cton b/cranelift/filetests/wasm/control.cton index f29c19b5c5..0e82b05245 100644 --- a/cranelift/filetests/wasm/control.cton +++ b/cranelift/filetests/wasm/control.cton @@ -32,3 +32,14 @@ ebb1(v2: i32): ebb2: jump ebb1(v0) } + +function %br_if_fallthrough(i32) -> i32 { +ebb0(v0: i32): + v1 = iconst.i32 1 + brz v0, ebb1(v1) + ; This jump gets converted to a fallthrough. + jump ebb1(v0) + +ebb1(v2: i32): + return v2 +} diff --git a/lib/cretonne/meta/gen_binemit.py b/lib/cretonne/meta/gen_binemit.py index 8e734c5ff1..fddadf64e5 100644 --- a/lib/cretonne/meta/gen_binemit.py +++ b/lib/cretonne/meta/gen_binemit.py @@ -135,14 +135,18 @@ def gen_isa(isa, fmt): 'pub fn emit_inst' '(func: &Function, inst: Inst, ' 'divert: &mut RegDiversions, sink: &mut CS) {', '}'): - fmt.line('let bits = func.encodings[inst].bits();') + fmt.line('let encoding = func.encodings[inst];') + fmt.line('let bits = encoding.bits();') with fmt.indented('match func.encodings[inst].recipe() {', '}'): for i, recipe in enumerate(isa.all_recipes): fmt.comment(recipe.name) with fmt.indented('{} => {{'.format(i), '}'): gen_recipe(recipe, fmt) fmt.line('_ => {}') - fmt.line('bad_encoding(func, inst);') + # Allow for un-encoded ghost instructions. + # Verifier checks the details. + with fmt.indented('if encoding.is_legal() {', '}'): + fmt.line('bad_encoding(func, inst);') def generate(isas, out_dir): diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 2652579020..518c9a703f 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -57,7 +57,7 @@ use flowgraph::ControlFlowGraph; use ir::entities::AnyEntity; use ir::instructions::{InstructionFormat, BranchInfo, ResolvedConstraint, CallInfo}; use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, StackSlot, - Value, Type}; + Value, Type, Opcode}; use isa::TargetIsa; use std::error as std_error; use std::fmt::{self, Display, Formatter}; @@ -711,6 +711,12 @@ impl<'a> Verifier<'a> { // Instructions with side effects are not allowed to be ghost instructions. let opcode = self.func.dfg[inst].opcode(); + // The `fallthrough` instruction is marked as a terminator and a branch, but it is not + // required to have an encoding. + if opcode == Opcode::Fallthrough { + return Ok(()); + } + if opcode.is_branch() { return err!(inst, "Branch must have an encoding"); } From 421a88123dca7ecfcbc3c51824944907e4f373f3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Jul 2017 10:47:51 -0700 Subject: [PATCH 0883/3084] Add Intel encodings for the icmp instruction. This instruction returns a `b1` value which is represented as the output of a setCC instruction which is the low 8 bits of a GPR register. Use a cmp+setCC macro recipe to encode this. That is not ideal, but we can't represent CPU flags yet. --- cranelift/filetests/isa/intel/binary32.cton | 72 ++++++++ cranelift/filetests/isa/intel/binary64.cton | 184 +++++++++++++++++--- lib/cretonne/meta/base/legalize.py | 13 +- lib/cretonne/meta/isa/intel/encodings.py | 8 + lib/cretonne/meta/isa/intel/recipes.py | 42 +++++ 5 files changed, 297 insertions(+), 22 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 7b43ed7aa2..2ee9396bf0 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -254,6 +254,78 @@ ebb0: ; asm: tzcntl %ecx, %esi [-,%rsi] v205 = ctz v1 ; bin: f3 0f bc f1 + ; Integer comparisons. + + ; asm: cmpl %esi, %ecx + ; asm: sete %bl + [-,%rbx] v300 = icmp eq v1, v2 ; bin: 39 f1 0f 94 c3 + ; asm: cmpl %ecx, %esi + ; asm: sete %dl + [-,%rdx] v301 = icmp eq v2, v1 ; bin: 39 ce 0f 94 c2 + + ; asm: cmpl %esi, %ecx + ; asm: setne %bl + [-,%rbx] v302 = icmp ne v1, v2 ; bin: 39 f1 0f 95 c3 + ; asm: cmpl %ecx, %esi + ; asm: setne %dl + [-,%rdx] v303 = icmp ne v2, v1 ; bin: 39 ce 0f 95 c2 + + ; asm: cmpl %esi, %ecx + ; asm: setl %bl + [-,%rbx] v304 = icmp slt v1, v2 ; bin: 39 f1 0f 9c c3 + ; asm: cmpl %ecx, %esi + ; asm: setl %dl + [-,%rdx] v305 = icmp slt v2, v1 ; bin: 39 ce 0f 9c c2 + + ; asm: cmpl %esi, %ecx + ; asm: setge %bl + [-,%rbx] v306 = icmp sge v1, v2 ; bin: 39 f1 0f 9d c3 + ; asm: cmpl %ecx, %esi + ; asm: setge %dl + [-,%rdx] v307 = icmp sge v2, v1 ; bin: 39 ce 0f 9d c2 + + ; asm: cmpl %esi, %ecx + ; asm: setg %bl + [-,%rbx] v308 = icmp sgt v1, v2 ; bin: 39 f1 0f 9f c3 + ; asm: cmpl %ecx, %esi + ; asm: setg %dl + [-,%rdx] v309 = icmp sgt v2, v1 ; bin: 39 ce 0f 9f c2 + + ; asm: cmpl %esi, %ecx + ; asm: setle %bl + [-,%rbx] v310 = icmp sle v1, v2 ; bin: 39 f1 0f 9e c3 + ; asm: cmpl %ecx, %esi + ; asm: setle %dl + [-,%rdx] v311 = icmp sle v2, v1 ; bin: 39 ce 0f 9e c2 + + ; asm: cmpl %esi, %ecx + ; asm: setb %bl + [-,%rbx] v312 = icmp ult v1, v2 ; bin: 39 f1 0f 92 c3 + ; asm: cmpl %ecx, %esi + ; asm: setb %dl + [-,%rdx] v313 = icmp ult v2, v1 ; bin: 39 ce 0f 92 c2 + + ; asm: cmpl %esi, %ecx + ; asm: setae %bl + [-,%rbx] v314 = icmp uge v1, v2 ; bin: 39 f1 0f 93 c3 + ; asm: cmpl %ecx, %esi + ; asm: setae %dl + [-,%rdx] v315 = icmp uge v2, v1 ; bin: 39 ce 0f 93 c2 + + ; asm: cmpl %esi, %ecx + ; asm: seta %bl + [-,%rbx] v316 = icmp ugt v1, v2 ; bin: 39 f1 0f 97 c3 + ; asm: cmpl %ecx, %esi + ; asm: seta %dl + [-,%rdx] v317 = icmp ugt v2, v1 ; bin: 39 ce 0f 97 c2 + + ; asm: cmpl %esi, %ecx + ; asm: setbe %bl + [-,%rbx] v318 = icmp ule v1, v2 ; bin: 39 f1 0f 96 c3 + ; asm: cmpl %ecx, %esi + ; asm: setbe %dl + [-,%rdx] v319 = icmp ule v2, v1 ; bin: 39 ce 0f 96 c2 + ; asm: call foo call fn0() ; bin: e8 PCRel4(fn0) 00000000 diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 1d88b69d50..00524afc4e 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -192,22 +192,94 @@ ebb0: ; asm: tzcntq %rcx, %r10 [-,%r10] v208 = ctz v1 ; bin: f3 4c 0f bc d1 - ; asm: testq %rcx, %ecx + ; Integer comparisons. + + ; asm: cmpq %rsi, %rcx + ; asm: sete %bl + [-,%rbx] v300 = icmp eq v1, v2 ; bin: 48 39 f1 0f 94 c3 + ; asm: cmpq %r10, %rsi + ; asm: sete %dl + [-,%rdx] v301 = icmp eq v2, v3 ; bin: 4c 39 d6 0f 94 c2 + + ; asm: cmpq %rsi, %rcx + ; asm: setne %bl + [-,%rbx] v302 = icmp ne v1, v2 ; bin: 48 39 f1 0f 95 c3 + ; asm: cmpq %r10, %rsi + ; asm: setne %dl + [-,%rdx] v303 = icmp ne v2, v3 ; bin: 4c 39 d6 0f 95 c2 + + ; asm: cmpq %rsi, %rcx + ; asm: setl %bl + [-,%rbx] v304 = icmp slt v1, v2 ; bin: 48 39 f1 0f 9c c3 + ; asm: cmpq %r10, %rsi + ; asm: setl %dl + [-,%rdx] v305 = icmp slt v2, v3 ; bin: 4c 39 d6 0f 9c c2 + + ; asm: cmpq %rsi, %rcx + ; asm: setge %bl + [-,%rbx] v306 = icmp sge v1, v2 ; bin: 48 39 f1 0f 9d c3 + ; asm: cmpq %r10, %rsi + ; asm: setge %dl + [-,%rdx] v307 = icmp sge v2, v3 ; bin: 4c 39 d6 0f 9d c2 + + ; asm: cmpq %rsi, %rcx + ; asm: setg %bl + [-,%rbx] v308 = icmp sgt v1, v2 ; bin: 48 39 f1 0f 9f c3 + ; asm: cmpq %r10, %rsi + ; asm: setg %dl + [-,%rdx] v309 = icmp sgt v2, v3 ; bin: 4c 39 d6 0f 9f c2 + + ; asm: cmpq %rsi, %rcx + ; asm: setle %bl + [-,%rbx] v310 = icmp sle v1, v2 ; bin: 48 39 f1 0f 9e c3 + ; asm: cmpq %r10, %rsi + ; asm: setle %dl + [-,%rdx] v311 = icmp sle v2, v3 ; bin: 4c 39 d6 0f 9e c2 + + ; asm: cmpq %rsi, %rcx + ; asm: setb %bl + [-,%rbx] v312 = icmp ult v1, v2 ; bin: 48 39 f1 0f 92 c3 + ; asm: cmpq %r10, %rsi + ; asm: setb %dl + [-,%rdx] v313 = icmp ult v2, v3 ; bin: 4c 39 d6 0f 92 c2 + + ; asm: cmpq %rsi, %rcx + ; asm: setae %bl + [-,%rbx] v314 = icmp uge v1, v2 ; bin: 48 39 f1 0f 93 c3 + ; asm: cmpq %r10, %rsi + ; asm: setae %dl + [-,%rdx] v315 = icmp uge v2, v3 ; bin: 4c 39 d6 0f 93 c2 + + ; asm: cmpq %rsi, %rcx + ; asm: seta %bl + [-,%rbx] v316 = icmp ugt v1, v2 ; bin: 48 39 f1 0f 97 c3 + ; asm: cmpq %r10, %rsi + ; asm: seta %dl + [-,%rdx] v317 = icmp ugt v2, v3 ; bin: 4c 39 d6 0f 97 c2 + + ; asm: cmpq %rsi, %rcx + ; asm: setbe %bl + [-,%rbx] v318 = icmp ule v1, v2 ; bin: 48 39 f1 0f 96 c3 + ; asm: cmpq %r10, %rsi + ; asm: setbe %dl + [-,%rdx] v319 = icmp ule v2, v3 ; bin: 4c 39 d6 0f 96 c2 + + ; asm: testq %rcx, %rcx ; asm: je ebb1 brz v1, ebb1 ; bin: 48 85 c9 74 1b - ; asm: testq %rsi, %esi + ; asm: testq %rsi, %rsi ; asm: je ebb1 brz v2, ebb1 ; bin: 48 85 f6 74 16 - ; asm: testq %r10, %r10d + ; asm: testq %r10, %r10 ; asm: je ebb1 brz v3, ebb1 ; bin: 4d 85 d2 74 11 - ; asm: testq %rcx, %ecx + ; asm: testq %rcx, %rcx ; asm: jne ebb1 brnz v1, ebb1 ; bin: 48 85 c9 75 0c - ; asm: test %rsi, %esi + ; asm: testq %rsi, %rsi ; asm: jne ebb1 brnz v2, ebb1 ; bin: 48 85 f6 75 07 - ; asm: testq %r10, %r10d + ; asm: testq %r10, %r10 ; asm: jne ebb1 brnz v3, ebb1 ; bin: 4d 85 d2 75 02 @@ -376,15 +448,15 @@ ebb0: [-,%rax] v130 = iconst.i32 1 [-,%rdx] v131 = iconst.i32 2 - ; asm: idivl %rcx + ; asm: idivl %ecx [-,%rax,%rdx] v132, v133 = x86_sdivmodx v130, v131, v1 ; bin: 40 f7 f9 - ; asm: idivl %rsi + ; asm: idivl %esi [-,%rax,%rdx] v134, v135 = x86_sdivmodx v130, v131, v2 ; bin: 40 f7 fe ; asm: idivl %r10d [-,%rax,%rdx] v136, v137 = x86_sdivmodx v130, v131, v3 ; bin: 41 f7 fa - ; asm: divl %rcx + ; asm: divl %ecx [-,%rax,%rdx] v138, v139 = x86_udivmodx v130, v131, v1 ; bin: 40 f7 f1 - ; asm: divl %rsi + ; asm: divl %esi [-,%rax,%rdx] v140, v141 = x86_udivmodx v130, v131, v2 ; bin: 40 f7 f6 ; asm: divl %r10d [-,%rax,%rdx] v142, v143 = x86_udivmodx v130, v131, v3 ; bin: 41 f7 f2 @@ -412,33 +484,105 @@ ebb0: ; asm: tzcntl %ecx, %r10d [-,%r10] v208 = ctz v1 ; bin: f3 44 0f bc d1 + ; Integer comparisons. + + ; asm: cmpl %esi, %ecx + ; asm: sete %bl + [-,%rbx] v300 = icmp eq v1, v2 ; bin: 40 39 f1 0f 94 c3 + ; asm: cmpl %r10d, %esi + ; asm: sete %dl + [-,%rdx] v301 = icmp eq v2, v3 ; bin: 44 39 d6 0f 94 c2 + + ; asm: cmpl %esi, %ecx + ; asm: setne %bl + [-,%rbx] v302 = icmp ne v1, v2 ; bin: 40 39 f1 0f 95 c3 + ; asm: cmpl %r10d, %esi + ; asm: setne %dl + [-,%rdx] v303 = icmp ne v2, v3 ; bin: 44 39 d6 0f 95 c2 + + ; asm: cmpl %esi, %ecx + ; asm: setl %bl + [-,%rbx] v304 = icmp slt v1, v2 ; bin: 40 39 f1 0f 9c c3 + ; asm: cmpl %r10d, %esi + ; asm: setl %dl + [-,%rdx] v305 = icmp slt v2, v3 ; bin: 44 39 d6 0f 9c c2 + + ; asm: cmpl %esi, %ecx + ; asm: setge %bl + [-,%rbx] v306 = icmp sge v1, v2 ; bin: 40 39 f1 0f 9d c3 + ; asm: cmpl %r10d, %esi + ; asm: setge %dl + [-,%rdx] v307 = icmp sge v2, v3 ; bin: 44 39 d6 0f 9d c2 + + ; asm: cmpl %esi, %ecx + ; asm: setg %bl + [-,%rbx] v308 = icmp sgt v1, v2 ; bin: 40 39 f1 0f 9f c3 + ; asm: cmpl %r10d, %esi + ; asm: setg %dl + [-,%rdx] v309 = icmp sgt v2, v3 ; bin: 44 39 d6 0f 9f c2 + + ; asm: cmpl %esi, %ecx + ; asm: setle %bl + [-,%rbx] v310 = icmp sle v1, v2 ; bin: 40 39 f1 0f 9e c3 + ; asm: cmpl %r10d, %esi + ; asm: setle %dl + [-,%rdx] v311 = icmp sle v2, v3 ; bin: 44 39 d6 0f 9e c2 + + ; asm: cmpl %esi, %ecx + ; asm: setb %bl + [-,%rbx] v312 = icmp ult v1, v2 ; bin: 40 39 f1 0f 92 c3 + ; asm: cmpl %r10d, %esi + ; asm: setb %dl + [-,%rdx] v313 = icmp ult v2, v3 ; bin: 44 39 d6 0f 92 c2 + + ; asm: cmpl %esi, %ecx + ; asm: setae %bl + [-,%rbx] v314 = icmp uge v1, v2 ; bin: 40 39 f1 0f 93 c3 + ; asm: cmpl %r10d, %esi + ; asm: setae %dl + [-,%rdx] v315 = icmp uge v2, v3 ; bin: 44 39 d6 0f 93 c2 + + ; asm: cmpl %esi, %ecx + ; asm: seta %bl + [-,%rbx] v316 = icmp ugt v1, v2 ; bin: 40 39 f1 0f 97 c3 + ; asm: cmpl %r10d, %esi + ; asm: seta %dl + [-,%rdx] v317 = icmp ugt v2, v3 ; bin: 44 39 d6 0f 97 c2 + + ; asm: cmpl %esi, %ecx + ; asm: setbe %bl + [-,%rbx] v318 = icmp ule v1, v2 ; bin: 40 39 f1 0f 96 c3 + ; asm: cmpl %r10d, %esi + ; asm: setbe %dl + [-,%rdx] v319 = icmp ule v2, v3 ; bin: 44 39 d6 0f 96 c2 + ; asm: testl %ecx, %ecx - ; asm: je ebb1 + ; asm: je ebb1x brz v1, ebb1 ; bin: 40 85 c9 74 1b ; asm: testl %esi, %esi - ; asm: je ebb1 + ; asm: je ebb1x brz v2, ebb1 ; bin: 40 85 f6 74 16 ; asm: testl %r10d, %r10d - ; asm: je ebb1 + ; asm: je ebb1x brz v3, ebb1 ; bin: 45 85 d2 74 11 ; asm: testl %ecx, %ecx - ; asm: jne ebb1 + ; asm: jne ebb1x brnz v1, ebb1 ; bin: 40 85 c9 75 0c - ; asm: test %esi, %esi - ; asm: jne ebb1 + ; asm: testl %esi, %esi + ; asm: jne ebb1x brnz v2, ebb1 ; bin: 40 85 f6 75 07 ; asm: testl %r10d, %r10d - ; asm: jne ebb1 + ; asm: jne ebb1x brnz v3, ebb1 ; bin: 45 85 d2 75 02 - ; asm: jmp ebb2 + ; asm: jmp ebb2x jump ebb2 ; bin: eb 01 - ; asm: ebb1: + ; asm: ebb1x: ebb1: return ; bin: c3 - ; asm: ebb2: + ; asm: ebb2x: ebb2: jump ebb1 ; bin: eb fd } diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index 8211a0039f..058c87f062 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -11,7 +11,8 @@ from .immediates import intcc from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm from .instructions import isub, isub_bin, isub_bout, isub_borrow from .instructions import band, bor, bxor, isplit, iconcat -from .instructions import icmp, iconst, bint +from .instructions import icmp, icmp_imm +from .instructions import iconst, bint from cdsl.ast import Var from cdsl.xform import Rtl, XFormGroup @@ -53,6 +54,7 @@ yl = Var('yl') yh = Var('yh') al = Var('al') ah = Var('ah') +cc = Var('cc') narrow.legalize( a << iadd(x, y), @@ -135,10 +137,17 @@ expand.legalize( b << bor(b1, b2) )) -# Expansions for immediates that are too large. +# Expansions for immediate operands that are out of range. expand.legalize( a << iadd_imm(x, y), Rtl( a1 << iconst(y), a << iadd(x, a1) )) + +expand.legalize( + a << icmp_imm(cc, x, y), + Rtl( + a1 << iconst(y), + a << icmp(cc, x, a1) + )) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index f5e8aabd50..9af8d6484d 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -174,3 +174,11 @@ I32.enc(base.brnz.i32, *r.tjccb(0x75)) I64.enc(base.brnz.i64, *r.tjccb.rex(0x75, w=1)) I64.enc(base.brnz.i32, *r.tjccb.rex(0x75)) I64.enc(base.brnz.i32, *r.tjccb(0x75)) + +# +# Comparisons +# +I32.enc(base.icmp.i32, *r.icscc(0x39)) +I64.enc(base.icmp.i64, *r.icscc.rex(0x39, w=1)) +I64.enc(base.icmp.i32, *r.icscc.rex(0x39)) +I64.enc(base.icmp.i32, *r.icscc(0x39)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index f68a8344b3..7fa7ca1bf3 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -6,6 +6,7 @@ from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt, IsEqual from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry from base.formats import Call, IndirectCall, Store, Load +from base.formats import IntCompare from base.formats import RegMove, Ternary, Jump, Branch from .registers import GPR, ABCD @@ -464,3 +465,44 @@ tjccb = TailRecipe( sink.put1(bits as u8); disp1(destination, func, sink); ''') + +# Comparison that produces a `b1` result in a GPR. +# +# This is a macro of a `cmp` instruction followed by a `setCC` instruction. +# This is not a great solution because: +# +# - The cmp+setcc combination is not recognized by CPU's macro fusion. +# - The 64-bit encoding has issues with REX prefixes. The `cmp` and `setCC` +# instructions may need a REX independently. +# - Modeling CPU flags in the type system would be better. +# +# Since the `setCC` instructions only write an 8-bit register, we use that as +# our `b1` representation: A `b1` value is represented as a GPR where the low 8 +# bits are known to be 0 or 1. The high bits are undefined. +# +# This bandaid macro doesn't support a REX prefix for the final `setCC` +# instruction, so it is limited to the `ABCD` register class for booleans. +icscc = TailRecipe( + 'cscc', IntCompare, size=1 + 3, ins=(GPR, GPR), outs=ABCD, + emit=''' + // Comparison instruction. + PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + modrm_rr(in_reg0, in_reg1, sink); + // `setCC` instruction, no REX. + use ir::condcodes::IntCC::*; + let setcc = match cond { + Equal => 0x94, + NotEqual => 0x95, + SignedLessThan => 0x9c, + SignedGreaterThanOrEqual => 0x9d, + SignedGreaterThan => 0x9f, + SignedLessThanOrEqual => 0x9e, + UnsignedLessThan => 0x92, + UnsignedGreaterThanOrEqual => 0x93, + UnsignedGreaterThan => 0x97, + UnsignedLessThanOrEqual => 0x96, + }; + sink.put1(0x0f); + sink.put1(setcc); + modrm_rr(out_reg0, 0, sink); + ''') From 1a662575a58a4efe809c9c174d01e53fb2bb8f43 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Jul 2017 12:01:28 -0700 Subject: [PATCH 0884/3084] Add Intel encodings for the bint instructions. Convert b1 to i32 or i64 by zero-extending the byte. --- cranelift/filetests/isa/intel/binary32.cton | 7 +++++++ cranelift/filetests/isa/intel/binary64.cton | 14 ++++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 11 +++++++++++ lib/cretonne/meta/isa/intel/recipes.py | 8 ++++++++ 4 files changed, 40 insertions(+) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 2ee9396bf0..7e95f84818 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -326,6 +326,13 @@ ebb0: ; asm: setbe %dl [-,%rdx] v319 = icmp ule v2, v1 ; bin: 39 ce 0f 96 c2 + ; Bool-to-int conversions. + + ; asm: movzbl %bl, %ecx + [-,%rcx] v350 = bint.i32 v300 ; bin: 0f b6 cb + ; asm: movzbl %dl, %esi + [-,%rsi] v351 = bint.i32 v301 ; bin: 0f b6 f2 + ; asm: call foo call fn0() ; bin: e8 PCRel4(fn0) 00000000 diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 00524afc4e..6222a1e7be 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -264,6 +264,13 @@ ebb0: ; asm: setbe %dl [-,%rdx] v319 = icmp ule v2, v3 ; bin: 4c 39 d6 0f 96 c2 + ; Bool-to-int conversions. + + ; asm: movzbq %bl, %rcx + [-,%rcx] v350 = bint.i64 v300 ; bin: 48 0f b6 cb + ; asm: movzbq %dl, %rsi + [-,%rsi] v351 = bint.i64 v301 ; bin: 48 0f b6 f2 + ; asm: testq %rcx, %rcx ; asm: je ebb1 brz v1, ebb1 ; bin: 48 85 c9 74 1b @@ -556,6 +563,13 @@ ebb0: ; asm: setbe %dl [-,%rdx] v319 = icmp ule v2, v3 ; bin: 44 39 d6 0f 96 c2 + ; Bool-to-int conversions. + + ; asm: movzbl %bl, %ecx + [-,%rcx] v350 = bint.i32 v300 ; bin: 40 0f b6 cb + ; asm: movzbl %dl, %esi + [-,%rsi] v351 = bint.i32 v301 ; bin: 40 0f b6 f2 + ; asm: testl %ecx, %ecx ; asm: je ebb1x brz v1, ebb1 ; bin: 40 85 c9 74 1b diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 9af8d6484d..6b04ae110d 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -182,3 +182,14 @@ I32.enc(base.icmp.i32, *r.icscc(0x39)) I64.enc(base.icmp.i64, *r.icscc.rex(0x39, w=1)) I64.enc(base.icmp.i32, *r.icscc.rex(0x39)) I64.enc(base.icmp.i32, *r.icscc(0x39)) + +# +# Convert bool to int. +# +# This assumes that b1 is represented as an 8-bit low register with the value 0 +# or 1. +I32.enc(base.bint.i32.b1, *r.urm_abcd(0x0f, 0xb6)) +I64.enc(base.bint.i64.b1, *r.urm.rex(0x0f, 0xb6, w=1)) +I64.enc(base.bint.i64.b1, *r.urm_abcd(0x0f, 0xb6)) # zext to i64 implicit. +I64.enc(base.bint.i32.b1, *r.urm.rex(0x0f, 0xb6)) +I64.enc(base.bint.i32.b1, *r.urm_abcd(0x0f, 0xb6)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 7fa7ca1bf3..8ae65f4723 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -224,6 +224,14 @@ urm = TailRecipe( modrm_rr(in_reg0, out_reg0, sink); ''') +# XX /r. Same as urm, but input limited to ABCD. +urm_abcd = TailRecipe( + 'urm_abcd', Unary, size=1, ins=ABCD, outs=GPR, + emit=''' + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); + modrm_rr(in_reg0, out_reg0, sink); + ''') + # XX /r, for regmove instructions. rmov = TailRecipe( 'ur', RegMove, size=1, ins=GPR, outs=(), From cfcbf4476493ab14ec0bfc5c174a6742c3f19562 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Jul 2017 12:36:36 -0700 Subject: [PATCH 0885/3084] Add tests for WebAssembly i32 comparisons. One function for each comparison operator. --- cranelift/filetests/wasm/i32-compares.cton | 85 ++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 cranelift/filetests/wasm/i32-compares.cton diff --git a/cranelift/filetests/wasm/i32-compares.cton b/cranelift/filetests/wasm/i32-compares.cton new file mode 100644 index 0000000000..228258d279 --- /dev/null +++ b/cranelift/filetests/wasm/i32-compares.cton @@ -0,0 +1,85 @@ +; Test code generation for WebAssembly i32 comparison operators. +test compile + +set is_64bit=0 +isa intel haswell + +set is_64bit=1 +isa intel haswell + +function %i32_eqz(i32) -> i32 { +ebb0(v0: i32): + v1 = icmp_imm eq v0, 0 + v2 = bint.i32 v1 + return v2 +} + +function %i32_eq(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = icmp eq v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i32_ne(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = icmp ne v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i32_lt_s(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = icmp slt v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i32_lt_u(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = icmp ult v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i32_gt_s(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = icmp sgt v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i32_gt_u(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = icmp ugt v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i32_le_s(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = icmp sle v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i32_le_u(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = icmp ule v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i32_ge_s(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = icmp sge v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i32_ge_u(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = icmp uge v0, v1 + v3 = bint.i32 v2 + return v3 +} From f03f1e1898fa05202ba37b374c673e6e54419b73 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Jul 2017 12:56:54 -0700 Subject: [PATCH 0886/3084] Add tests for WebAssembly i64 operators. This only works on 64-bit haswell for now. We need more legalization patterns for 32-bit ISAs. --- cranelift/filetests/wasm/i64-arith.cton | 105 +++++++++++++++++++++ cranelift/filetests/wasm/i64-compares.cton | 82 ++++++++++++++++ 2 files changed, 187 insertions(+) create mode 100644 cranelift/filetests/wasm/i64-arith.cton create mode 100644 cranelift/filetests/wasm/i64-compares.cton diff --git a/cranelift/filetests/wasm/i64-arith.cton b/cranelift/filetests/wasm/i64-arith.cton new file mode 100644 index 0000000000..a7cce18d3c --- /dev/null +++ b/cranelift/filetests/wasm/i64-arith.cton @@ -0,0 +1,105 @@ +; Test basic code generation for i64 arithmetic WebAssembly instructions. +test compile + +set is_64bit=1 +isa intel haswell + +; Constants. + +function %i64_const() -> i64 { +ebb0: + v0 = iconst.i64 0x8765_4321 + return v0 +} + +; Unary operations. + +function %i64_clz(i64) -> i64 { +ebb0(v0: i64): + v1 = clz v0 + return v1 +} + +function %i64_ctz(i64) -> i64 { +ebb0(v0: i64): + v1 = ctz v0 + return v1 +} + +function %i64_popcnt(i64) -> i64 { +ebb0(v0: i64): + v1 = popcnt v0 + return v1 +} + +; Binary operations. + +function %i64_add(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + v2 = iadd v0, v1 + return v2 +} + +function %i64_sub(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + v2 = isub v0, v1 + return v2 +} + +function %i64_mul(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + v2 = imul v0, v1 + return v2 +} + +; function %i64_div(i64, i64) -> i64 +; function %i64_rem_s(i64, i64) -> i64 +; function %i64_rem_u(i64, i64) -> i64 + +function %i64_and(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + v2 = band v0, v1 + return v2 +} + +function %i64_or(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + v2 = bor v0, v1 + return v2 +} + +function %i64_xor(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + v2 = bxor v0, v1 + return v2 +} + +function %i64_shl(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + v2 = ishl v0, v1 + return v2 +} + +function %i64_shr_s(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + v2 = sshr v0, v1 + return v2 +} + +function %i64_shr_u(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + v2 = ushr v0, v1 + return v2 +} + +function %i64_rotl(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + v2 = rotl v0, v1 + return v2 +} + +function %i64_rotr(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + v2 = rotr v0, v1 + return v2 +} diff --git a/cranelift/filetests/wasm/i64-compares.cton b/cranelift/filetests/wasm/i64-compares.cton new file mode 100644 index 0000000000..3406463f0d --- /dev/null +++ b/cranelift/filetests/wasm/i64-compares.cton @@ -0,0 +1,82 @@ +; Test code generation for WebAssembly i64 comparison operators. +test compile + +set is_64bit=1 +isa intel haswell + +function %i64_eqz(i64) -> i32 { +ebb0(v0: i64): + v1 = icmp_imm eq v0, 0 + v2 = bint.i32 v1 + return v2 +} + +function %i64_eq(i64, i64) -> i32 { +ebb0(v0: i64, v1: i64): + v2 = icmp eq v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i64_ne(i64, i64) -> i32 { +ebb0(v0: i64, v1: i64): + v2 = icmp ne v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i64_lt_s(i64, i64) -> i32 { +ebb0(v0: i64, v1: i64): + v2 = icmp slt v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i64_lt_u(i64, i64) -> i32 { +ebb0(v0: i64, v1: i64): + v2 = icmp ult v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i64_gt_s(i64, i64) -> i32 { +ebb0(v0: i64, v1: i64): + v2 = icmp sgt v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i64_gt_u(i64, i64) -> i32 { +ebb0(v0: i64, v1: i64): + v2 = icmp ugt v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i64_le_s(i64, i64) -> i32 { +ebb0(v0: i64, v1: i64): + v2 = icmp sle v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i64_le_u(i64, i64) -> i32 { +ebb0(v0: i64, v1: i64): + v2 = icmp ule v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i64_ge_s(i64, i64) -> i32 { +ebb0(v0: i64, v1: i64): + v2 = icmp sge v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %i64_ge_u(i64, i64) -> i32 { +ebb0(v0: i64, v1: i64): + v2 = icmp uge v0, v1 + v3 = bint.i32 v2 + return v3 +} From 9f105145af68cfd9dbe8970a13fe97aeb2d25ce7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Jul 2017 13:11:11 -0700 Subject: [PATCH 0887/3084] Add a null encoding for ireduce.i32.i64. This conversion doesn't require any code, we're just looking at the bits differently. --- cranelift/filetests/wasm/conversions.cton | 11 +++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 5 +++++ lib/cretonne/meta/isa/intel/recipes.py | 4 ++++ 3 files changed, 20 insertions(+) create mode 100644 cranelift/filetests/wasm/conversions.cton diff --git a/cranelift/filetests/wasm/conversions.cton b/cranelift/filetests/wasm/conversions.cton new file mode 100644 index 0000000000..459e20e3bf --- /dev/null +++ b/cranelift/filetests/wasm/conversions.cton @@ -0,0 +1,11 @@ +; Test code generation for WebAssembly type conversion operators. +test compile + +set is_64bit=1 +isa intel haswell + +function %i32_wrap_i64(i64) -> i32 { +ebb0(v0: i64): + v1 = ireduce.i32 v0 + return v1 +} diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 6b04ae110d..db039f36a1 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -193,3 +193,8 @@ I64.enc(base.bint.i64.b1, *r.urm.rex(0x0f, 0xb6, w=1)) I64.enc(base.bint.i64.b1, *r.urm_abcd(0x0f, 0xb6)) # zext to i64 implicit. I64.enc(base.bint.i32.b1, *r.urm.rex(0x0f, 0xb6)) I64.enc(base.bint.i32.b1, *r.urm_abcd(0x0f, 0xb6)) + +# Numerical conversions. + +# Converting i64 to i32 is a no-op in 64-bit mode. +I64.enc(base.ireduce.i32.i64, r.null, 0) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 8ae65f4723..5ff1e8b225 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -190,6 +190,10 @@ class TailRecipe: return (self.recipes[name], bits) +# A null unary instruction that takes a GPR register. Can be used for identity +# copies and no-op conversions. +null = EncRecipe('null', Unary, size=0, ins=GPR, outs=0, emit='') + # XX /r rr = TailRecipe( 'rr', Binary, size=1, ins=(GPR, GPR), outs=0, From b59b348a1e1e8eeb0a15e628adff1581240346e4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Jul 2017 13:40:11 -0700 Subject: [PATCH 0888/3084] Add Intel encodings for sextend and uextend. --- cranelift/filetests/isa/intel/binary64.cton | 28 +++++++++++++++++++++ cranelift/filetests/wasm/conversions.cton | 12 +++++++++ lib/cretonne/meta/isa/intel/encodings.py | 4 +++ 3 files changed, 44 insertions(+) diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 6222a1e7be..3e332e2175 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -600,3 +600,31 @@ ebb1: ebb2: jump ebb1 ; bin: eb fd } + +; Tests for i64/i32 conversion instructions. +function %I64_I32() { +ebb0: + [-,%rcx] v1 = iconst.i64 1 + [-,%rsi] v2 = iconst.i64 2 + [-,%r10] v3 = iconst.i64 3 + + [-,%rcx] v11 = ireduce.i32 v1 ; bin: + [-,%rsi] v12 = ireduce.i32 v2 ; bin: + [-,%r10] v13 = ireduce.i32 v3 ; bin: + + ; asm: movslq %ecx, %rsi + [-,%rsi] v20 = sextend.i64 v11 ; bin: 48 63 f1 + ; asm: movslq %esi, %r10 + [-,%r10] v21 = sextend.i64 v12 ; bin: 4c 63 d6 + ; asm: movslq %r10d, %rcx + [-,%rcx] v22 = sextend.i64 v13 ; bin: 49 63 ca + + ; asm: movl %ecx, %esi + [-,%rsi] v30 = uextend.i64 v11 ; bin: 40 89 ce + ; asm: movl %esi, %r10d + [-,%r10] v31 = uextend.i64 v12 ; bin: 41 89 f2 + ; asm: movl %r10d, %ecx + [-,%rcx] v32 = uextend.i64 v13 ; bin: 44 89 d1 + + return +} diff --git a/cranelift/filetests/wasm/conversions.cton b/cranelift/filetests/wasm/conversions.cton index 459e20e3bf..a30cf97226 100644 --- a/cranelift/filetests/wasm/conversions.cton +++ b/cranelift/filetests/wasm/conversions.cton @@ -9,3 +9,15 @@ ebb0(v0: i64): v1 = ireduce.i32 v0 return v1 } + +function %i64_extend_s_i32(i32) -> i64 { +ebb0(v0: i32): + v1 = sextend.i64 v0 + return v1 +} + +function %i64_extend_u_i32(i32) -> i64 { +ebb0(v0: i32): + v1 = uextend.i64 v0 + return v1 +} diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index db039f36a1..16074cd849 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -198,3 +198,7 @@ I64.enc(base.bint.i32.b1, *r.urm_abcd(0x0f, 0xb6)) # Converting i64 to i32 is a no-op in 64-bit mode. I64.enc(base.ireduce.i32.i64, r.null, 0) +I64.enc(base.sextend.i64.i32, *r.urm.rex(0x63, w=1)) +# A 32-bit register copy clears the high 32 bits. +I64.enc(base.uextend.i64.i32, *r.umr.rex(0x89)) +I64.enc(base.uextend.i64.i32, *r.umr(0x89)) From 87c5f27ff7c24b3b30495c940572e2d68a956252 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Jul 2017 15:01:32 -0700 Subject: [PATCH 0889/3084] Intel encodings for trap. Use a ud2 instruction which generates an undefined instruction exception. --- cranelift/filetests/isa/intel/binary32.cton | 2 +- cranelift/filetests/isa/intel/binary64.cton | 2 +- cranelift/filetests/wasm/control.cton | 5 +++++ lib/cretonne/meta/isa/intel/encodings.py | 6 ++++++ lib/cretonne/meta/isa/intel/recipes.py | 7 ++++++- 5 files changed, 19 insertions(+), 3 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 7e95f84818..456623a235 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -364,5 +364,5 @@ ebb1: ; asm: ebb2: ebb2: - jump ebb1 ; bin: eb fd + trap ; bin: 0f 0b } diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 3e332e2175..3d93ba86ec 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -626,5 +626,5 @@ ebb0: ; asm: movl %r10d, %ecx [-,%rcx] v32 = uextend.i64 v13 ; bin: 44 89 d1 - return + trap ; bin: 0f 0b } diff --git a/cranelift/filetests/wasm/control.cton b/cranelift/filetests/wasm/control.cton index 0e82b05245..c8e37a536f 100644 --- a/cranelift/filetests/wasm/control.cton +++ b/cranelift/filetests/wasm/control.cton @@ -43,3 +43,8 @@ ebb0(v0: i32): ebb1(v2: i32): return v2 } + +function %undefined() { +ebb0: + trap +} diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 16074cd849..fcf114ef3e 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -175,6 +175,12 @@ I64.enc(base.brnz.i64, *r.tjccb.rex(0x75, w=1)) I64.enc(base.brnz.i32, *r.tjccb.rex(0x75)) I64.enc(base.brnz.i32, *r.tjccb(0x75)) +# +# Trap as ud2 +# +I32.enc(base.trap, *r.noop(0x0f, 0x0b)) +I64.enc(base.trap, *r.noop(0x0f, 0x0b)) + # # Comparisons # diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 5ff1e8b225..efffc47319 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -5,7 +5,7 @@ from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt, IsEqual from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry -from base.formats import Call, IndirectCall, Store, Load +from base.formats import Nullary, Call, IndirectCall, Store, Load from base.formats import IntCompare from base.formats import RegMove, Ternary, Jump, Branch from .registers import GPR, ABCD @@ -194,6 +194,11 @@ class TailRecipe: # copies and no-op conversions. null = EncRecipe('null', Unary, size=0, ins=GPR, outs=0, emit='') +# XX opcode, no ModR/M. +noop = TailRecipe( + 'noop', Nullary, size=0, ins=(), outs=(), + emit='PUT_OP(bits, 0, sink);') + # XX /r rr = TailRecipe( 'rr', Binary, size=1, ins=(GPR, GPR), outs=0, From e3f6755264d6aed25a2a83495e6b02215879ceed Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 19 Jul 2017 14:44:01 -0700 Subject: [PATCH 0890/3084] Add some signed int to float conversions. These map to single Intel instructions. The i64 to float conversions are not tested yet. The encoding tables can't yet differentiate instructions on a secondary type variable alone. --- .../filetests/isa/intel/binary32-float.cton | 34 +++++++++++++ .../filetests/isa/intel/binary64-float.cton | 49 +++++++++++++++++++ cranelift/filetests/wasm/conversions.cton | 26 ++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 14 ++++++ lib/cretonne/meta/isa/intel/recipes.py | 10 +++- 5 files changed, 132 insertions(+), 1 deletion(-) create mode 100644 cranelift/filetests/isa/intel/binary32-float.cton create mode 100644 cranelift/filetests/isa/intel/binary64-float.cton diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton new file mode 100644 index 0000000000..c59bf10217 --- /dev/null +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -0,0 +1,34 @@ +; Binary emission of 32-bit floating point code. +test binemit +isa intel has_sse2 + +; The binary encodings can be verified with the command: +; +; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/binary32-float.cton | llvm-mc -show-encoding -triple=i386 +; + +function %F32() { +ebb0: + [-,%rcx] v0 = iconst.i32 1 + [-,%rsi] v1 = iconst.i32 2 + + ; asm: cvtsi2ss %ecx, %xmm5 + [-,%xmm5] v10 = fcvt_from_sint.f32 v0 ; bin: f3 0f 2a e9 + ; asm: cvtsi2ss %esi, %xmm2 + [-,%xmm2] v11 = fcvt_from_sint.f32 v1 ; bin: f3 0f 2a d6 + + return +} + +function %F64() { +ebb0: + [-,%rcx] v0 = iconst.i32 1 + [-,%rsi] v1 = iconst.i32 2 + + ; asm: cvtsi2sd %ecx, %xmm5 + [-,%xmm5] v10 = fcvt_from_sint.f64 v0 ; bin: f2 0f 2a e9 + ; asm: cvtsi2sd %esi, %xmm2 + [-,%xmm2] v11 = fcvt_from_sint.f64 v1 ; bin: f2 0f 2a d6 + + return +} diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton new file mode 100644 index 0000000000..cf523b0e5b --- /dev/null +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -0,0 +1,49 @@ +; Binary emission of 64-bit floating point code. +test binemit +set is_64bit +isa intel has_sse2 + +; The binary encodings can be verified with the command: +; +; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/binary64-float.cton | llvm-mc -show-encoding -triple=x86_64 +; + +function %F32() { +ebb0: + [-,%r11] v0 = iconst.i32 1 + [-,%rsi] v1 = iconst.i32 2 + [-,%rax] v2 = iconst.i64 11 + [-,%r14] v3 = iconst.i64 12 + + ; asm: cvtsi2ssl %r11d, %xmm5 + [-,%xmm5] v10 = fcvt_from_sint.f32 v0 ; bin: f3 41 0f 2a eb + ; asm: cvtsi2ssl %esi, %xmm10 + [-,%xmm10] v11 = fcvt_from_sint.f32 v1 ; bin: f3 44 0f 2a d6 + + ; asm: cvtsi2ssq %rax, %xmm5 + [-,%xmm5] v12 = fcvt_from_sint.f32 v2 ; TODO: f3 48 0f 2a e8 + ; asm: cvtsi2ssq %r14, %xmm10 + [-,%xmm10] v13 = fcvt_from_sint.f32 v3 ; TODO: f3 4d 0f 2a d6 + + return +} + +function %F64() { +ebb0: + [-,%r11] v0 = iconst.i32 1 + [-,%rsi] v1 = iconst.i32 2 + [-,%rax] v2 = iconst.i64 11 + [-,%r14] v3 = iconst.i64 12 + + ; asm: cvtsi2sdl %r11d, %xmm5 + [-,%xmm5] v10 = fcvt_from_sint.f64 v0 ; bin: f2 41 0f 2a eb + ; asm: cvtsi2sdl %esi, %xmm10 + [-,%xmm10] v11 = fcvt_from_sint.f64 v1 ; bin: f2 44 0f 2a d6 + + ; asm: cvtsi2sdq %rax, %xmm5 + [-,%xmm5] v12 = fcvt_from_sint.f64 v2 ; TODO: f2 48 0f 2a e8 + ; asm: cvtsi2sdq %r14, %xmm10 + [-,%xmm10] v13 = fcvt_from_sint.f64 v3 ; TODO: f2 4d 0f 2a d6 + + return +} diff --git a/cranelift/filetests/wasm/conversions.cton b/cranelift/filetests/wasm/conversions.cton index a30cf97226..6f2354d624 100644 --- a/cranelift/filetests/wasm/conversions.cton +++ b/cranelift/filetests/wasm/conversions.cton @@ -21,3 +21,29 @@ ebb0(v0: i32): v1 = uextend.i64 v0 return v1 } + +function %f32_convert_s_i32(i32) -> f32 { +ebb0(v0: i32): + v1 = fcvt_from_sint.f32 v0 + return v1 +} + +function %f64_convert_s_i32(i32) -> f64 { +ebb0(v0: i32): + v1 = fcvt_from_sint.f64 v0 + return v1 +} + +function %f32_convert_s_i64(i64) -> f32 { +ebb0(v0: i64): + v1 = fcvt_from_sint.f32 v0 + return v1 +} + +function %f64_convert_s_i64(i64) -> f64 { +ebb0(v0: i64): + v1 = fcvt_from_sint.f64 v0 + return v1 +} + +; TODO: f*_convert_u_i* (Don't exist on Intel). diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index fcf114ef3e..264c07000e 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -208,3 +208,17 @@ I64.enc(base.sextend.i64.i32, *r.urm.rex(0x63, w=1)) # A 32-bit register copy clears the high 32 bits. I64.enc(base.uextend.i64.i32, *r.umr.rex(0x89)) I64.enc(base.uextend.i64.i32, *r.umr(0x89)) + +# +# Floating point +# + +# cvtsi2ss +I32.enc(base.fcvt_from_sint.f32.i32, *r.furm(0xf3, 0x0f, 0x2A)) +I64.enc(base.fcvt_from_sint.f32.i32, *r.furm.rex(0xf3, 0x0f, 0x2A)) +I64.enc(base.fcvt_from_sint.f32.i32, *r.furm(0xf3, 0x0f, 0x2A)) + +# cvtsi2sd +I32.enc(base.fcvt_from_sint.f64.i32, *r.furm(0xf2, 0x0f, 0x2A)) +I64.enc(base.fcvt_from_sint.f64.i32, *r.furm.rex(0xf2, 0x0f, 0x2A)) +I64.enc(base.fcvt_from_sint.f64.i32, *r.furm(0xf2, 0x0f, 0x2A)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index efffc47319..642e335f6a 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -8,7 +8,7 @@ from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry from base.formats import Nullary, Call, IndirectCall, Store, Load from base.formats import IntCompare from base.formats import RegMove, Ternary, Jump, Branch -from .registers import GPR, ABCD +from .registers import GPR, ABCD, FPR try: from typing import Tuple, Dict # noqa @@ -241,6 +241,14 @@ urm_abcd = TailRecipe( modrm_rr(in_reg0, out_reg0, sink); ''') +# XX /r, RM form, GPR -> FPR. +furm = TailRecipe( + 'furm', Unary, size=1, ins=GPR, outs=FPR, + emit=''' + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); + modrm_rr(in_reg0, out_reg0, sink); + ''') + # XX /r, for regmove instructions. rmov = TailRecipe( 'ur', RegMove, size=1, ins=GPR, outs=(), From 43e190ad2008a6b3a485573be81d864c4f18ff61 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 20 Jul 2017 10:08:09 -0700 Subject: [PATCH 0891/3084] Intel encodings for fadd, fsub, fmul, fdiv. --- .../filetests/isa/intel/binary32-float.cton | 44 ++++++++++++++++ .../filetests/isa/intel/binary64-float.cton | 44 ++++++++++++++++ cranelift/filetests/wasm/f32-arith.cton | 52 +++++++++++++++++++ cranelift/filetests/wasm/f64-arith.cton | 52 +++++++++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 14 +++++ lib/cretonne/meta/isa/intel/recipes.py | 10 +++- 6 files changed, 215 insertions(+), 1 deletion(-) create mode 100644 cranelift/filetests/wasm/f32-arith.cton create mode 100644 cranelift/filetests/wasm/f64-arith.cton diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index c59bf10217..a815ad3a52 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -17,6 +17,28 @@ ebb0: ; asm: cvtsi2ss %esi, %xmm2 [-,%xmm2] v11 = fcvt_from_sint.f32 v1 ; bin: f3 0f 2a d6 + ; Binary arithmetic. + + ; asm: addss %xmm2, %xmm5 + [-,%xmm5] v20 = fadd v10, v11 ; bin: f3 0f 58 ea + ; asm: addss %xmm5, %xmm2 + [-,%xmm2] v21 = fadd v11, v10 ; bin: f3 0f 58 d5 + + ; asm: subss %xmm2, %xmm5 + [-,%xmm5] v22 = fsub v10, v11 ; bin: f3 0f 5c ea + ; asm: subss %xmm5, %xmm2 + [-,%xmm2] v23 = fsub v11, v10 ; bin: f3 0f 5c d5 + + ; asm: mulss %xmm2, %xmm5 + [-,%xmm5] v24 = fmul v10, v11 ; bin: f3 0f 59 ea + ; asm: mulss %xmm5, %xmm2 + [-,%xmm2] v25 = fmul v11, v10 ; bin: f3 0f 59 d5 + + ; asm: divss %xmm2, %xmm5 + [-,%xmm5] v26 = fdiv v10, v11 ; bin: f3 0f 5e ea + ; asm: divss %xmm5, %xmm2 + [-,%xmm2] v27 = fdiv v11, v10 ; bin: f3 0f 5e d5 + return } @@ -25,10 +47,32 @@ ebb0: [-,%rcx] v0 = iconst.i32 1 [-,%rsi] v1 = iconst.i32 2 + ; Binary arithmetic. + ; asm: cvtsi2sd %ecx, %xmm5 [-,%xmm5] v10 = fcvt_from_sint.f64 v0 ; bin: f2 0f 2a e9 ; asm: cvtsi2sd %esi, %xmm2 [-,%xmm2] v11 = fcvt_from_sint.f64 v1 ; bin: f2 0f 2a d6 + ; asm: addsd %xmm2, %xmm5 + [-,%xmm5] v20 = fadd v10, v11 ; bin: f2 0f 58 ea + ; asm: addsd %xmm5, %xmm2 + [-,%xmm2] v21 = fadd v11, v10 ; bin: f2 0f 58 d5 + + ; asm: subsd %xmm2, %xmm5 + [-,%xmm5] v22 = fsub v10, v11 ; bin: f2 0f 5c ea + ; asm: subsd %xmm5, %xmm2 + [-,%xmm2] v23 = fsub v11, v10 ; bin: f2 0f 5c d5 + + ; asm: mulsd %xmm2, %xmm5 + [-,%xmm5] v24 = fmul v10, v11 ; bin: f2 0f 59 ea + ; asm: mulsd %xmm5, %xmm2 + [-,%xmm2] v25 = fmul v11, v10 ; bin: f2 0f 59 d5 + + ; asm: divsd %xmm2, %xmm5 + [-,%xmm5] v26 = fdiv v10, v11 ; bin: f2 0f 5e ea + ; asm: divsd %xmm5, %xmm2 + [-,%xmm2] v27 = fdiv v11, v10 ; bin: f2 0f 5e d5 + return } diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index cf523b0e5b..fcf78c71f1 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -25,6 +25,28 @@ ebb0: ; asm: cvtsi2ssq %r14, %xmm10 [-,%xmm10] v13 = fcvt_from_sint.f32 v3 ; TODO: f3 4d 0f 2a d6 + ; Binary arithmetic. + + ; asm: addss %xmm10, %xmm5 + [-,%xmm5] v20 = fadd v10, v11 ; bin: f3 41 0f 58 ea + ; asm: addss %xmm5, %xmm10 + [-,%xmm10] v21 = fadd v11, v10 ; bin: f3 44 0f 58 d5 + + ; asm: subss %xmm10, %xmm5 + [-,%xmm5] v22 = fsub v10, v11 ; bin: f3 41 0f 5c ea + ; asm: subss %xmm5, %xmm10 + [-,%xmm10] v23 = fsub v11, v10 ; bin: f3 44 0f 5c d5 + + ; asm: mulss %xmm10, %xmm5 + [-,%xmm5] v24 = fmul v10, v11 ; bin: f3 41 0f 59 ea + ; asm: mulss %xmm5, %xmm10 + [-,%xmm10] v25 = fmul v11, v10 ; bin: f3 44 0f 59 d5 + + ; asm: divss %xmm10, %xmm5 + [-,%xmm5] v26 = fdiv v10, v11 ; bin: f3 41 0f 5e ea + ; asm: divss %xmm5, %xmm10 + [-,%xmm10] v27 = fdiv v11, v10 ; bin: f3 44 0f 5e d5 + return } @@ -45,5 +67,27 @@ ebb0: ; asm: cvtsi2sdq %r14, %xmm10 [-,%xmm10] v13 = fcvt_from_sint.f64 v3 ; TODO: f2 4d 0f 2a d6 + ; Binary arithmetic. + + ; asm: addsd %xmm10, %xmm5 + [-,%xmm5] v20 = fadd v10, v11 ; bin: f2 41 0f 58 ea + ; asm: addsd %xmm5, %xmm10 + [-,%xmm10] v21 = fadd v11, v10 ; bin: f2 44 0f 58 d5 + + ; asm: subsd %xmm10, %xmm5 + [-,%xmm5] v22 = fsub v10, v11 ; bin: f2 41 0f 5c ea + ; asm: subsd %xmm5, %xmm10 + [-,%xmm10] v23 = fsub v11, v10 ; bin: f2 44 0f 5c d5 + + ; asm: mulsd %xmm10, %xmm5 + [-,%xmm5] v24 = fmul v10, v11 ; bin: f2 41 0f 59 ea + ; asm: mulsd %xmm5, %xmm10 + [-,%xmm10] v25 = fmul v11, v10 ; bin: f2 44 0f 59 d5 + + ; asm: divsd %xmm10, %xmm5 + [-,%xmm5] v26 = fdiv v10, v11 ; bin: f2 41 0f 5e ea + ; asm: divsd %xmm5, %xmm10 + [-,%xmm10] v27 = fdiv v11, v10 ; bin: f2 44 0f 5e d5 + return } diff --git a/cranelift/filetests/wasm/f32-arith.cton b/cranelift/filetests/wasm/f32-arith.cton new file mode 100644 index 0000000000..ad48b401b8 --- /dev/null +++ b/cranelift/filetests/wasm/f32-arith.cton @@ -0,0 +1,52 @@ +; Test basic code generation for f32 arithmetic WebAssembly instructions. +test compile + +set is_64bit=0 +isa intel haswell + +set is_64bit=1 +isa intel haswell + +; Constants. + +; function %f32_const() -> f32 + +; Unary operations + +; function %f32_abs(f32) -> f32 +; function %f32_neg(f32) -> f32 +; function %f32_sqrt(f32) -> f32 +; function %f32_ceil(f32) -> f32 +; function %f32_floor(f32) -> f32 +; function %f32_trunc(f32) -> f32 +; function %f32_nearest (f32) -> f32 + +; Binary Operations + +function %f32_add(f32, f32) -> f32 { +ebb0(v0: f32, v1: f32): + v2 = fadd v0, v1 + return v2 +} + +function %f32_sub(f32, f32) -> f32 { +ebb0(v0: f32, v1: f32): + v2 = fsub v0, v1 + return v2 +} + +function %f32_mul(f32, f32) -> f32 { +ebb0(v0: f32, v1: f32): + v2 = fmul v0, v1 + return v2 +} + +function %f32_div(f32, f32) -> f32 { +ebb0(v0: f32, v1: f32): + v2 = fdiv v0, v1 + return v2 +} + +; function %f32_min(f32, f32) -> f32 +; function %f32_max(f32, f32) -> f32 +; function %f32_copysign(f32, f32) -> f32 diff --git a/cranelift/filetests/wasm/f64-arith.cton b/cranelift/filetests/wasm/f64-arith.cton new file mode 100644 index 0000000000..bc81d69c3e --- /dev/null +++ b/cranelift/filetests/wasm/f64-arith.cton @@ -0,0 +1,52 @@ +; Test basic code generation for f64 arithmetic WebAssembly instructions. +test compile + +set is_64bit=0 +isa intel haswell + +set is_64bit=1 +isa intel haswell + +; Constants. + +; function %f64_const() -> f64 + +; Unary operations + +; function %f64_abs(f64) -> f64 +; function %f64_neg(f64) -> f64 +; function %f64_sqrt(f64) -> f64 +; function %f64_ceil(f64) -> f64 +; function %f64_floor(f64) -> f64 +; function %f64_trunc(f64) -> f64 +; function %f64_nearest (f64) -> f64 + +; Binary Operations + +function %f64_add(f64, f64) -> f64 { +ebb0(v0: f64, v1: f64): + v2 = fadd v0, v1 + return v2 +} + +function %f64_sub(f64, f64) -> f64 { +ebb0(v0: f64, v1: f64): + v2 = fsub v0, v1 + return v2 +} + +function %f64_mul(f64, f64) -> f64 { +ebb0(v0: f64, v1: f64): + v2 = fmul v0, v1 + return v2 +} + +function %f64_div(f64, f64) -> f64 { +ebb0(v0: f64, v1: f64): + v2 = fdiv v0, v1 + return v2 +} + +; function %f64_min(f64, f64) -> f64 +; function %f64_max(f64, f64) -> f64 +; function %f64_copysign(f64, f64) -> f64 diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 264c07000e..42d54a539f 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -222,3 +222,17 @@ I64.enc(base.fcvt_from_sint.f32.i32, *r.furm(0xf3, 0x0f, 0x2A)) I32.enc(base.fcvt_from_sint.f64.i32, *r.furm(0xf2, 0x0f, 0x2A)) I64.enc(base.fcvt_from_sint.f64.i32, *r.furm.rex(0xf2, 0x0f, 0x2A)) I64.enc(base.fcvt_from_sint.f64.i32, *r.furm(0xf2, 0x0f, 0x2A)) + +# Binary arithmetic ops. +for inst, opc in [ + (base.fadd, 0x58), + (base.fsub, 0x5c), + (base.fmul, 0x59), + (base.fdiv, 0x5e)]: + I32.enc(inst.f32, *r.frm(0xf3, 0x0f, opc)) + I64.enc(inst.f32, *r.frm.rex(0xf3, 0x0f, opc)) + I64.enc(inst.f32, *r.frm(0xf3, 0x0f, opc)) + + I32.enc(inst.f64, *r.frm(0xf2, 0x0f, opc)) + I64.enc(inst.f64, *r.frm.rex(0xf2, 0x0f, opc)) + I64.enc(inst.f64, *r.frm(0xf2, 0x0f, opc)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 642e335f6a..3eb3d51671 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -197,7 +197,7 @@ null = EncRecipe('null', Unary, size=0, ins=GPR, outs=0, emit='') # XX opcode, no ModR/M. noop = TailRecipe( 'noop', Nullary, size=0, ins=(), outs=(), - emit='PUT_OP(bits, 0, sink);') + emit='PUT_OP(bits, BASE_REX, sink);') # XX /r rr = TailRecipe( @@ -215,6 +215,14 @@ rrx = TailRecipe( modrm_rr(in_reg1, in_reg0, sink); ''') +# XX /r with FPR ins and outs. RM form. +frm = TailRecipe( + 'frr', Binary, size=1, ins=(FPR, FPR), outs=0, + emit=''' + PUT_OP(bits, rex2(in_reg1, in_reg0), sink); + modrm_rr(in_reg1, in_reg0, sink); + ''') + # XX /r, but for a unary operator with separate input/output register, like # copies. MR form. umr = TailRecipe( From a42eaa77b4517ef5a5b215c1806d0c51378dd4a4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 20 Jul 2017 11:14:11 -0700 Subject: [PATCH 0892/3084] Add bitwise ops that invert the second operand. ARM has all of these as scalar integer instructions. Intel has band_not in SSE and as a scalar in BMI1. Add the trivial legalization patterns that use a bnot instruction. --- cranelift/docs/langref.rst | 8 +++---- cranelift/filetests/isa/riscv/expand-i32.cton | 8 +++++++ lib/cretonne/meta/base/instructions.py | 24 +++++++++++++++++++ lib/cretonne/meta/base/legalize.py | 13 ++++++++++ 4 files changed, 48 insertions(+), 5 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index dde7865486..e7b0cad3d7 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -667,11 +667,9 @@ operating on boolean values, the bitwise operations work as logical operators. .. autoinst:: bxor .. autoinst:: bxor_imm .. autoinst:: bnot - -.. todo:: Redundant bitwise operators. - - ARM has instructions like ``bic(x,y) = x & ~y``, ``orn(x,y) = x | ~y``, and - ``eon(x,y) = x ^ ~y``. +.. autoinst:: band_not +.. autoinst:: bor_not +.. autoinst:: bxor_not The shift and rotate operations only work on integer types (scalar and vector). The shift amount does not have to be the same type as the value being shifted. diff --git a/cranelift/filetests/isa/riscv/expand-i32.cton b/cranelift/filetests/isa/riscv/expand-i32.cton index 2bffad234c..885d97d309 100644 --- a/cranelift/filetests/isa/riscv/expand-i32.cton +++ b/cranelift/filetests/isa/riscv/expand-i32.cton @@ -28,3 +28,11 @@ ebb0(v0: i32): ; check: $(cst=$V) = iconst.i32 0x3b9a_ca00 ; check: $v1 = iadd $v0, $cst ; check: return $v1 + +function %bitclear(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = band_not v0, v1 + ; check: bnot + ; check: band + return v2 +} diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 8638e19368..763550a5ee 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -920,6 +920,30 @@ bnot = Instruction( """, ins=x, outs=a) +band_not = Instruction( + 'band_not', """ + Bitwise and not. + + Computes `x & ~y`. + """, + ins=(x, y), outs=a) + +bor_not = Instruction( + 'bor_not', """ + Bitwise or not. + + Computes `x | ~y`. + """, + ins=(x, y), outs=a) + +bxor_not = Instruction( + 'bxor_not', """ + Bitwise xor not. + + Computes `x ^ ~y`. + """, + ins=(x, y), outs=a) + # Bitwise binary ops with immediate arg. x = Operand('x', iB) Y = Operand('Y', imm64) diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index 058c87f062..91e1e1114a 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -11,6 +11,7 @@ from .immediates import intcc from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm from .instructions import isub, isub_bin, isub_bout, isub_borrow from .instructions import band, bor, bxor, isplit, iconcat +from .instructions import bnot, band_not, bor_not, bxor_not from .instructions import icmp, icmp_imm from .instructions import iconst, bint from cdsl.ast import Var @@ -151,3 +152,15 @@ expand.legalize( a1 << iconst(y), a << icmp(cc, x, a1) )) + +# Expansions for *_not variants of bitwise ops. +for inst_not, inst in [ + (band_not, band), + (bor_not, bor), + (bxor_not, bxor)]: + expand.legalize( + a << inst_not(x, y), + Rtl( + a1 << bnot(y), + a << inst(x, a1) + )) From 35cbe68a70c72a9d385e92c58d4fc875d8e59d3e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 20 Jul 2017 11:45:06 -0700 Subject: [PATCH 0893/3084] Intel encodings for floating point bitwise ops. band, bor, bxor, band_not are all available on XMM registers. --- .../filetests/isa/intel/binary32-float.cton | 46 ++++++++++++++++ .../filetests/isa/intel/binary64-float.cton | 54 +++++++++++++++++-- lib/cretonne/meta/isa/intel/encodings.py | 16 +++++- 3 files changed, 111 insertions(+), 5 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index a815ad3a52..4a4361707c 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -39,6 +39,29 @@ ebb0: ; asm: divss %xmm5, %xmm2 [-,%xmm2] v27 = fdiv v11, v10 ; bin: f3 0f 5e d5 + ; Bitwise ops. + ; We use the *ps SSE instructions for everything because they are smaller. + + ; asm: andps %xmm2, %xmm5 + [-,%xmm5] v30 = band v10, v11 ; bin: 0f 54 ea + ; asm: andps %xmm5, %xmm2 + [-,%xmm2] v31 = band v11, v10 ; bin: 0f 54 d5 + + ; asm: andnps %xmm2, %xmm5 + [-,%xmm5] v32 = band_not v10, v11 ; bin: 0f 55 ea + ; asm: andnps %xmm5, %xmm2 + [-,%xmm2] v33 = band_not v11, v10 ; bin: 0f 55 d5 + + ; asm: orps %xmm2, %xmm5 + [-,%xmm5] v34 = bor v10, v11 ; bin: 0f 56 ea + ; asm: orps %xmm5, %xmm2 + [-,%xmm2] v35 = bor v11, v10 ; bin: 0f 56 d5 + + ; asm: xorps %xmm2, %xmm5 + [-,%xmm5] v36 = bxor v10, v11 ; bin: 0f 57 ea + ; asm: xorps %xmm5, %xmm2 + [-,%xmm2] v37 = bxor v11, v10 ; bin: 0f 57 d5 + return } @@ -74,5 +97,28 @@ ebb0: ; asm: divsd %xmm5, %xmm2 [-,%xmm2] v27 = fdiv v11, v10 ; bin: f2 0f 5e d5 + ; Bitwise ops. + ; We use the *ps SSE instructions for everything because they are smaller. + + ; asm: andps %xmm2, %xmm5 + [-,%xmm5] v30 = band v10, v11 ; bin: 0f 54 ea + ; asm: andps %xmm5, %xmm2 + [-,%xmm2] v31 = band v11, v10 ; bin: 0f 54 d5 + + ; asm: andnps %xmm2, %xmm5 + [-,%xmm5] v32 = band_not v10, v11 ; bin: 0f 55 ea + ; asm: andnps %xmm5, %xmm2 + [-,%xmm2] v33 = band_not v11, v10 ; bin: 0f 55 d5 + + ; asm: orps %xmm2, %xmm5 + [-,%xmm5] v34 = bor v10, v11 ; bin: 0f 56 ea + ; asm: orps %xmm5, %xmm2 + [-,%xmm2] v35 = bor v11, v10 ; bin: 0f 56 d5 + + ; asm: xorps %xmm2, %xmm5 + [-,%xmm5] v36 = bxor v10, v11 ; bin: 0f 57 ea + ; asm: xorps %xmm5, %xmm2 + [-,%xmm2] v37 = bxor v11, v10 ; bin: 0f 57 d5 + return } diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index fcf78c71f1..ba604ad43c 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -47,6 +47,29 @@ ebb0: ; asm: divss %xmm5, %xmm10 [-,%xmm10] v27 = fdiv v11, v10 ; bin: f3 44 0f 5e d5 + ; Bitwise ops. + ; We use the *ps SSE instructions for everything because they are smaller. + + ; asm: andps %xmm10, %xmm5 + [-,%xmm5] v30 = band v10, v11 ; bin: 41 0f 54 ea + ; asm: andps %xmm5, %xmm10 + [-,%xmm10] v31 = band v11, v10 ; bin: 44 0f 54 d5 + + ; asm: andnps %xmm10, %xmm5 + [-,%xmm5] v32 = band_not v10, v11 ; bin: 41 0f 55 ea + ; asm: andnps %xmm5, %xmm10 + [-,%xmm10] v33 = band_not v11, v10 ; bin: 44 0f 55 d5 + + ; asm: orps %xmm10, %xmm5 + [-,%xmm5] v34 = bor v10, v11 ; bin: 41 0f 56 ea + ; asm: orps %xmm5, %xmm10 + [-,%xmm10] v35 = bor v11, v10 ; bin: 44 0f 56 d5 + + ; asm: xorps %xmm10, %xmm5 + [-,%xmm5] v36 = bxor v10, v11 ; bin: 41 0f 57 ea + ; asm: xorps %xmm5, %xmm10 + [-,%xmm10] v37 = bxor v11, v10 ; bin: 44 0f 57 d5 + return } @@ -72,22 +95,45 @@ ebb0: ; asm: addsd %xmm10, %xmm5 [-,%xmm5] v20 = fadd v10, v11 ; bin: f2 41 0f 58 ea ; asm: addsd %xmm5, %xmm10 - [-,%xmm10] v21 = fadd v11, v10 ; bin: f2 44 0f 58 d5 + [-,%xmm10] v21 = fadd v11, v10 ; bin: f2 44 0f 58 d5 ; asm: subsd %xmm10, %xmm5 [-,%xmm5] v22 = fsub v10, v11 ; bin: f2 41 0f 5c ea ; asm: subsd %xmm5, %xmm10 - [-,%xmm10] v23 = fsub v11, v10 ; bin: f2 44 0f 5c d5 + [-,%xmm10] v23 = fsub v11, v10 ; bin: f2 44 0f 5c d5 ; asm: mulsd %xmm10, %xmm5 [-,%xmm5] v24 = fmul v10, v11 ; bin: f2 41 0f 59 ea ; asm: mulsd %xmm5, %xmm10 - [-,%xmm10] v25 = fmul v11, v10 ; bin: f2 44 0f 59 d5 + [-,%xmm10] v25 = fmul v11, v10 ; bin: f2 44 0f 59 d5 ; asm: divsd %xmm10, %xmm5 [-,%xmm5] v26 = fdiv v10, v11 ; bin: f2 41 0f 5e ea ; asm: divsd %xmm5, %xmm10 - [-,%xmm10] v27 = fdiv v11, v10 ; bin: f2 44 0f 5e d5 + [-,%xmm10] v27 = fdiv v11, v10 ; bin: f2 44 0f 5e d5 + + ; Bitwise ops. + ; We use the *ps SSE instructions for everything because they are smaller. + + ; asm: andps %xmm10, %xmm5 + [-,%xmm5] v30 = band v10, v11 ; bin: 41 0f 54 ea + ; asm: andps %xmm5, %xmm10 + [-,%xmm10] v31 = band v11, v10 ; bin: 44 0f 54 d5 + + ; asm: andnps %xmm10, %xmm5 + [-,%xmm5] v32 = band_not v10, v11 ; bin: 41 0f 55 ea + ; asm: andnps %xmm5, %xmm10 + [-,%xmm10] v33 = band_not v11, v10 ; bin: 44 0f 55 d5 + + ; asm: orps %xmm10, %xmm5 + [-,%xmm5] v34 = bor v10, v11 ; bin: 41 0f 56 ea + ; asm: orps %xmm5, %xmm10 + [-,%xmm10] v35 = bor v11, v10 ; bin: 44 0f 56 d5 + + ; asm: xorps %xmm10, %xmm5 + [-,%xmm5] v36 = bxor v10, v11 ; bin: 41 0f 57 ea + ; asm: xorps %xmm5, %xmm10 + [-,%xmm10] v37 = bxor v11, v10 ; bin: 44 0f 57 d5 return } diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 42d54a539f..f155795b6f 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -224,7 +224,7 @@ I64.enc(base.fcvt_from_sint.f64.i32, *r.furm.rex(0xf2, 0x0f, 0x2A)) I64.enc(base.fcvt_from_sint.f64.i32, *r.furm(0xf2, 0x0f, 0x2A)) # Binary arithmetic ops. -for inst, opc in [ +for inst, opc in [ (base.fadd, 0x58), (base.fsub, 0x5c), (base.fmul, 0x59), @@ -236,3 +236,17 @@ for inst, opc in [ I32.enc(inst.f64, *r.frm(0xf2, 0x0f, opc)) I64.enc(inst.f64, *r.frm.rex(0xf2, 0x0f, opc)) I64.enc(inst.f64, *r.frm(0xf2, 0x0f, opc)) + +# Binary bitwise ops. +for inst, opc in [ + (base.band, 0x54), + (base.band_not, 0x55), + (base.bor, 0x56), + (base.bxor, 0x57)]: + I32.enc(inst.f32, *r.frm(0x0f, opc)) + I64.enc(inst.f32, *r.frm.rex(0x0f, opc)) + I64.enc(inst.f32, *r.frm(0x0f, opc)) + + I32.enc(inst.f64, *r.frm(0x0f, opc)) + I64.enc(inst.f64, *r.frm.rex(0x0f, opc)) + I64.enc(inst.f64, *r.frm(0x0f, opc)) From 4142a9ca9cf64009057675f9dd4daf21f213390b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Jul 2017 21:10:32 -0700 Subject: [PATCH 0894/3084] Return a Result from constant_hash::probe. When a hash table probe fails, return the index of the failed entry. This can be used to store default values in the sentinel entries. --- lib/cretonne/src/constant_hash.rs | 12 ++++++--- lib/cretonne/src/ir/instructions.rs | 4 +-- lib/cretonne/src/isa/enc_tables.rs | 40 +++++++++++++++++++---------- lib/cretonne/src/settings.rs | 4 +-- 4 files changed, 38 insertions(+), 22 deletions(-) diff --git a/lib/cretonne/src/constant_hash.rs b/lib/cretonne/src/constant_hash.rs index 08cc2944c8..a165654dce 100644 --- a/lib/cretonne/src/constant_hash.rs +++ b/lib/cretonne/src/constant_hash.rs @@ -24,8 +24,12 @@ pub trait Table { /// The provided `hash` value must have been computed from `key` using the same hash function that /// was used to construct the table. /// -/// Returns the table index containing the found entry, or `None` if no entry could be found. -pub fn probe + ?Sized>(table: &T, key: K, hash: usize) -> Option { +/// Returns `Ok(idx)` with the table index containing the found entry, or `Err(idx)` with the empty +/// sentinel entry if no entry could be found. +pub fn probe + ?Sized>(table: &T, + key: K, + hash: usize) + -> Result { debug_assert!(table.len().is_power_of_two()); let mask = table.len() - 1; @@ -36,8 +40,8 @@ pub fn probe + ?Sized>(table: &T, key: K, hash: usize) idx &= mask; match table.key(idx) { - None => return None, - Some(k) if k == key => return Some(idx), + None => return Err(idx), + Some(k) if k == key => return Ok(idx), _ => {} } diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 808b7cc01a..279a7a37a9 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -85,10 +85,10 @@ impl FromStr for Opcode { } match probe::<&str, [Option]>(&OPCODE_HASH_TABLE, s, simple_hash(s)) { - None => Err("Unknown opcode"), + Err(_) => Err("Unknown opcode"), // We unwrap here because probe() should have ensured that the entry // at this index is not None. - Some(i) => Ok(OPCODE_HASH_TABLE[i].unwrap()), + Ok(i) => Ok(OPCODE_HASH_TABLE[i].unwrap()), } } } diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index 3467a2bdfd..b53e80036e 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -5,6 +5,7 @@ use ir::{Type, Opcode, InstructionData}; use isa::{Encoding, Legalize}; use constant_hash::{Table, probe}; +use std::ops::Range; /// Level 1 hash table entry. /// @@ -27,6 +28,14 @@ pub struct Level1Entry + Copy> { pub offset: OffT, } +impl + Copy> Level1Entry { + /// Get the level 2 table range indicated by this entry. + fn range(&self) -> Range { + let b = self.offset.into() as usize; + b..b + (1 << self.log2len) + } +} + impl + Copy> Table for [Level1Entry] { fn len(&self) -> usize { self.len() @@ -83,20 +92,23 @@ pub fn lookup_enclist(ctrl_typevar: Type, OffT2: Into + Copy { // TODO: The choice of legalization actions here is naive. This needs to be configurable. - probe(level1_table, ctrl_typevar, ctrl_typevar.index()) - .ok_or_else(|| if ctrl_typevar.lane_type().bits() > 32 { - Legalize::Narrow - } else { - Legalize::Expand - }) - .and_then(|l1idx| { - let l1ent = &level1_table[l1idx]; - let l2off = l1ent.offset.into() as usize; - let l2tab = &level2_table[l2off..l2off + (1 << l1ent.log2len)]; - probe(l2tab, opcode, opcode as usize) - .map(|l2idx| l2tab[l2idx].offset.into() as usize) - .ok_or(Legalize::Expand) - }) + match probe(level1_table, ctrl_typevar, ctrl_typevar.index()) { + Err(_) => { + // No level 1 entry for the type. + Err(if ctrl_typevar.lane_type().bits() > 32 { + Legalize::Narrow + } else { + Legalize::Expand + }) + } + Ok(l1idx) => { + let l1ent = &level1_table[l1idx]; + let l2tab = &level2_table[l1ent.range()]; + probe(l2tab, opcode, opcode as usize) + .map(|l2idx| l2tab[l2idx].offset.into() as usize) + .map_err(|_| Legalize::Expand) + } + } } /// Encoding list entry. diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 19b9fafc21..be5140ef34 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -83,8 +83,8 @@ impl Builder { /// Look up a descriptor by name. fn lookup(&self, name: &str) -> Result<(usize, detail::Detail)> { match probe(self.template, name, simple_hash(name)) { - None => Err(Error::BadName), - Some(entry) => { + Err(_) => Err(Error::BadName), + Ok(entry) => { let d = &self.template.descriptors[self.template.hash_table[entry] as usize]; Ok((d.offset as usize, d.detail)) } From f651ec4f787d17d9dd71d992655760263edcaaac Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Jul 2017 13:48:18 -0700 Subject: [PATCH 0895/3084] Add a PredicateView type to abstract the predicate bit vector a bit. The encoding tables contain references to numbered ISA predicates. - Give the ISA Flags types a predicate_view() method which returns a PredicateView. - Delete the old predicate_bytes() method which returned a raw &[u8]. - Use a 'static lifetime for the encoding list slice in the Encodings iterator, and a single 'a lifetime for everything else. --- lib/cretonne/meta/gen_settings.py | 22 +++++++++++++--------- lib/cretonne/src/isa/arm32/mod.rs | 12 ++++++------ lib/cretonne/src/isa/arm64/mod.rs | 12 ++++++------ lib/cretonne/src/isa/enc_tables.rs | 30 ++++++++++++++---------------- lib/cretonne/src/isa/intel/mod.rs | 12 ++++++------ lib/cretonne/src/isa/mod.rs | 10 +++++----- lib/cretonne/src/isa/riscv/mod.rs | 12 ++++++------ lib/cretonne/src/settings.rs | 22 ++++++++++++++++++++++ 8 files changed, 78 insertions(+), 54 deletions(-) diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index c56cc4df41..1a79026c36 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -87,16 +87,20 @@ def gen_getters(sgrp, fmt): """ fmt.doc_comment("User-defined settings.") with fmt.indented('impl Flags {', '}'): - fmt.doc_comment('Returns inner slice of bytes.') - fmt.doc_comment('The byte-sized settings are not included.') - with fmt.indented('pub fn predicate_bytes(&self) -> &[u8] {', '}'): - fmt.line('&self.bytes[{}..]'.format(sgrp.boolean_offset)) - fmt.doc_comment('Dynamic numbered predicate getter.') + fmt.doc_comment('Get a view of the boolean predicates.') with fmt.indented( - 'pub fn numbered_predicate(&self, p: usize) -> bool {', '}'): - fmt.line( - 'self.bytes[{} + p/8] & (1 << (p%8)) != 0' - .format(sgrp.boolean_offset)) + 'pub fn predicate_view(&self) -> ::settings::PredicateView {', + '}'): + fmt.format( + '::settings::PredicateView::new(&self.bytes[{}..])', + sgrp.boolean_offset) + if sgrp.settings: + fmt.doc_comment('Dynamic numbered predicate getter.') + with fmt.indented( + 'fn numbered_predicate(&self, p: usize) -> bool {', '}'): + fmt.line( + 'self.bytes[{} + p / 8] & (1 << (p % 8)) != 0' + .format(sgrp.boolean_offset)) for setting in sgrp.settings: gen_getter(setting, sgrp, fmt) for pred in sgrp.named_predicates: diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 769adc376d..5675d19d7a 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -61,11 +61,11 @@ impl TargetIsa for Isa { enc_tables::INFO.clone() } - fn legal_encodings<'a, 'b>(&'a self, - _dfg: &'b ir::DataFlowGraph, - inst: &'b ir::InstructionData, - ctrl_typevar: ir::Type) - -> Result, Legalize> { + fn legal_encodings<'a>(&'a self, + _dfg: &'a ir::DataFlowGraph, + inst: &'a ir::InstructionData, + ctrl_typevar: ir::Type) + -> Result, Legalize> { lookup_enclist(ctrl_typevar, inst.opcode(), self.cpumode, @@ -75,7 +75,7 @@ impl TargetIsa for Isa { &enc_tables::ENCLISTS[..], inst, enc_tables::check_instp, - self.isa_flags.predicate_bytes())) + self.isa_flags.predicate_view())) }) } diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index a1b0ac2478..e9aea5aacf 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -54,11 +54,11 @@ impl TargetIsa for Isa { enc_tables::INFO.clone() } - fn legal_encodings<'a, 'b>(&'a self, - _dfg: &'b ir::DataFlowGraph, - inst: &'b ir::InstructionData, - ctrl_typevar: ir::Type) - -> Result, Legalize> { + fn legal_encodings<'a>(&'a self, + _dfg: &'a ir::DataFlowGraph, + inst: &'a ir::InstructionData, + ctrl_typevar: ir::Type) + -> Result, Legalize> { lookup_enclist(ctrl_typevar, inst.opcode(), &enc_tables::LEVEL1_A64[..], @@ -68,7 +68,7 @@ impl TargetIsa for Isa { &enc_tables::ENCLISTS[..], inst, enc_tables::check_instp, - self.isa_flags.predicate_bytes())) + self.isa_flags.predicate_view())) }) } diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index b53e80036e..1c6645002e 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -2,9 +2,11 @@ //! //! This module contains types and functions for working with the encoding tables generated by //! `lib/cretonne/meta/gen_encoding.py`. + +use constant_hash::{Table, probe}; use ir::{Type, Opcode, InstructionData}; use isa::{Encoding, Legalize}; -use constant_hash::{Table, probe}; +use settings::PredicateView; use std::ops::Range; /// Level 1 hash table entry. @@ -127,15 +129,15 @@ const CODE_ALWAYS: EncListEntry = PRED_MASK; const CODE_FAIL: EncListEntry = 0xffff; /// An iterator over legal encodings for the instruction. -pub struct Encodings<'a, 'b> { +pub struct Encodings<'a> { offset: usize, - enclist: &'b [EncListEntry], - inst: &'b InstructionData, + enclist: &'static [EncListEntry], + inst: &'a InstructionData, instp: fn(&InstructionData, EncListEntry) -> bool, - isa_predicate_bytes: &'a [u8], + isa_predicates: PredicateView<'a>, } -impl<'a, 'b> Encodings<'a, 'b> { +impl<'a> Encodings<'a> { /// Creates a new instance of `Encodings`. /// /// # Parameters @@ -151,29 +153,25 @@ impl<'a, 'b> Encodings<'a, 'b> { /// encoding lists are laid out such that first call to `next` returns valid entry in the list /// or `None`. pub fn new(offset: usize, - enclist: &'b [EncListEntry], - inst: &'b InstructionData, + enclist: &'static [EncListEntry], + inst: &'a InstructionData, instp: fn(&InstructionData, EncListEntry) -> bool, - isa_predicate_bytes: &'a [u8]) + isa_predicates: PredicateView<'a>) -> Self { Encodings { offset, enclist, inst, instp, - isa_predicate_bytes, + isa_predicates, } } } -impl<'a, 'b> Iterator for Encodings<'a, 'b> { +impl<'a> Iterator for Encodings<'a> { type Item = Encoding; fn next(&mut self) -> Option { - fn numbered_predicate(bytes: &[u8], p: usize) -> bool { - bytes[p / 8] & (1 << (p % 8)) != 0 - } - while self.enclist[self.offset] != CODE_FAIL { let pred = self.enclist[self.offset]; if pred <= CODE_ALWAYS { @@ -189,7 +187,7 @@ impl<'a, 'b> Iterator for Encodings<'a, 'b> { } else { // This is an ISA predicate entry. self.offset += 1; - if !numbered_predicate(self.isa_predicate_bytes, (pred & PRED_MASK) as usize) { + if !self.isa_predicates.test((pred & PRED_MASK) as usize) { // ISA predicate failed, skip the next N entries. self.offset += 3 * (pred >> PRED_BITS) as usize; } diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 93a2ed2c09..35d339a740 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -61,11 +61,11 @@ impl TargetIsa for Isa { enc_tables::INFO.clone() } - fn legal_encodings<'a, 'b>(&'a self, - _dfg: &'b ir::DataFlowGraph, - inst: &'b ir::InstructionData, - ctrl_typevar: ir::Type) - -> Result, Legalize> { + fn legal_encodings<'a>(&'a self, + _dfg: &'a ir::DataFlowGraph, + inst: &'a ir::InstructionData, + ctrl_typevar: ir::Type) + -> Result, Legalize> { lookup_enclist(ctrl_typevar, inst.opcode(), self.cpumode, @@ -75,7 +75,7 @@ impl TargetIsa for Isa { &enc_tables::ENCLISTS[..], inst, enc_tables::check_instp, - self.isa_flags.predicate_bytes())) + self.isa_flags.predicate_view())) }) } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 6665de5f42..100c901ec0 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -138,11 +138,11 @@ pub trait TargetIsa { fn register_info(&self) -> RegInfo; /// Returns an iterartor over legal encodings for the instruction. - fn legal_encodings<'a, 'b>(&'a self, - dfg: &'b ir::DataFlowGraph, - inst: &'b ir::InstructionData, - ctrl_typevar: ir::Type) - -> Result, Legalize>; + fn legal_encodings<'a>(&'a self, + dfg: &'a ir::DataFlowGraph, + inst: &'a ir::InstructionData, + ctrl_typevar: ir::Type) + -> Result, Legalize>; /// Encode an instruction after determining it is legal. /// diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index b70f6185b8..869ea4c9cb 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -61,11 +61,11 @@ impl TargetIsa for Isa { enc_tables::INFO.clone() } - fn legal_encodings<'a, 'b>(&'a self, - _dfg: &'b ir::DataFlowGraph, - inst: &'b ir::InstructionData, - ctrl_typevar: ir::Type) - -> Result, Legalize> { + fn legal_encodings<'a>(&'a self, + _dfg: &'a ir::DataFlowGraph, + inst: &'a ir::InstructionData, + ctrl_typevar: ir::Type) + -> Result, Legalize> { lookup_enclist(ctrl_typevar, inst.opcode(), self.cpumode, @@ -75,7 +75,7 @@ impl TargetIsa for Isa { &enc_tables::ENCLISTS[..], inst, enc_tables::check_instp, - self.isa_flags.predicate_bytes())) + self.isa_flags.predicate_view())) }) } diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index be5140ef34..660c833a4a 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -162,6 +162,28 @@ pub enum Error { /// A result returned when changing a setting. pub type Result = result::Result; +/// A reference to just the boolean predicates of a settings object. +/// +/// The settings objects themselves are generated and appear in the `isa/*/settings.rs` modules. +/// Each settings object provides a `predicate_view()` method that makes it possible to query +/// ISA predicates by number. +#[derive(Clone, Copy)] +pub struct PredicateView<'a>(&'a [u8]); + +impl<'a> PredicateView<'a> { + /// Create a new view of a precomputed predicate vector. + /// + /// See the `predicate_view()` method on the various `Flags` types defined for each ISA. + pub fn new(bits: &'a [u8]) -> PredicateView { + PredicateView(bits) + } + + /// Check a numbered predicate. + pub fn test(self, p: usize) -> bool { + self.0[p / 8] & (1 << (p % 8)) != 0 + } +} + /// Implementation details for generated code. /// /// This module holds definitions that need to be public so the can be instantiated by generated From db28e733ec871cd862391e332cdd0121488e1f28 Mon Sep 17 00:00:00 2001 From: Dimo Date: Mon, 24 Jul 2017 11:56:00 -0700 Subject: [PATCH 0896/3084] test-all.sh should print the versions for both python2 and python3 its using --- cranelift/test-all.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 72b07f8856..33e47bc358 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -35,7 +35,7 @@ else needcheck=yes fi if [ -n "$needcheck" ]; then - banner $(python --version 2>&1) + banner "$(python --version 2>&1), $(python3 --version 2>&1)" $topdir/lib/cretonne/meta/check.sh touch $tsfile || echo no target directory fi From 716cd26fbfec61e1f64d19d5c58f69d83517e765 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 24 Jul 2017 11:21:12 -0700 Subject: [PATCH 0897/3084] Make legalization actions configurable. When an instruction doesn't have a valid encoding for the target ISA, it needs to be legalized. Different legalization strategies can be expressed as separate XFormGroup objects. Make the choice of XFormGroup configurable per CPU mode, rather than depending on a hard-coded default. Add a CPUMode.legalize_type() method which assigns an XFormGroup to controlling type variables and lets you set a default. Add a `legalize` field to Level1Entry so the first-level hash table lookup gives us the configured default legalization action for the instruction's controlling type variable. --- lib/cretonne/meta/base/legalize.py | 7 ++ lib/cretonne/meta/cdsl/isa.py | 63 ++++++++++++++- lib/cretonne/meta/cdsl/xform.py | 4 + lib/cretonne/meta/gen_encoding.py | 97 ++++++++++++++++++------ lib/cretonne/meta/isa/arm32/defs.py | 5 ++ lib/cretonne/meta/isa/arm64/defs.py | 4 + lib/cretonne/meta/isa/intel/encodings.py | 14 ++++ lib/cretonne/meta/isa/riscv/encodings.py | 14 ++++ lib/cretonne/src/isa/enc_tables.rs | 48 +++++++----- lib/cretonne/src/isa/mod.rs | 14 ++++ 10 files changed, 229 insertions(+), 41 deletions(-) diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index 91e1e1114a..1afefcda89 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -27,6 +27,13 @@ narrow = XFormGroup('narrow', """ operations are expressed in terms of smaller integer types. """) +widen = XFormGroup('widen', """ + Legalize instructions by widening. + + The transformations in the 'widen' group work by expressing + instructions in terms of larger types. + """) + expand = XFormGroup('expand', """ Legalize instructions by expansion. diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 0562df10dc..ea10c64640 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -1,8 +1,10 @@ """Defining instruction set architectures.""" from __future__ import absolute_import +from collections import OrderedDict from .predicates import And from .registers import RegClass, Register, Stack from .ast import Apply +from .types import ValueType # The typing module is only required by mypy, and we don't use these imports # outside type comments. @@ -12,8 +14,8 @@ try: from .instructions import MaybeBoundInst, InstructionGroup, InstructionFormat # noqa from .predicates import PredNode # noqa from .settings import SettingGroup # noqa - from .types import ValueType # noqa from .registers import RegBank # noqa + from .xform import XFormGroup # noqa OperandConstraint = Union[RegClass, Register, int, Stack] ConstraintSeq = Union[OperandConstraint, Tuple[OperandConstraint, ...]] # Instruction specification for encodings. Allows for predicated @@ -43,6 +45,11 @@ class TargetISA(object): self.cpumodes = list() # type: List[CPUMode] self.regbanks = list() # type: List[RegBank] self.regclasses = list() # type: List[RegClass] + self.legalize_codes = OrderedDict() # type: OrderedDict[XFormGroup, int] # noqa + + def __str__(self): + # type: () -> str + return self.name def finish(self): # type: () -> TargetISA @@ -138,6 +145,25 @@ class TargetISA(object): # `isa/registers.rs`. assert len(self.regclasses) <= 32, "Too many register classes" + def legalize_code(self, xgrp): + # type: (XFormGroup) -> int + """ + Get the legalization code for the transform group `xgrp`. Assign one if + necessary. + + Each target ISA has its own list of legalization actions with + associated legalize codes that appear in the encoding tables. + + This method is used to maintain the registry of legalization actions + and their table codes. + """ + if xgrp in self.legalize_codes: + code = self.legalize_codes[xgrp] + else: + code = len(self.legalize_codes) + self.legalize_codes[xgrp] = code + return code + class CPUMode(object): """ @@ -157,6 +183,11 @@ class CPUMode(object): self.encodings = [] # type: List[Encoding] isa.cpumodes.append(self) + # Tables for configuring legalization actions when no valid encoding + # exists for an instruction. + self.default_legalize = None # type: XFormGroup + self.type_legalize = dict() # type: Dict[ValueType, XFormGroup] + def __str__(self): # type: () -> str return self.name @@ -171,6 +202,36 @@ class CPUMode(object): """ self.encodings.append(Encoding(self, *args, **kwargs)) + def legalize_type(self, default=None, **kwargs): + # type: (XFormGroup, **XFormGroup) -> None + """ + Configure the legalization action per controlling type variable. + + Instructions that have a controlling type variable mentioned in one of + the arguments will be legalized according to the action specified here + instead of using the `legalize_default` action. + + The keyword arguments are value type names: + + mode.legalize_type(i8=widen, i16=widen, i32=expand) + + The `default` argument specifies the action to take for controlling + type variables that don't have an explicitly configured action. + """ + if default is not None: + self.default_legalize = default + + for name, xgrp in kwargs.items(): + ty = ValueType.by_name(name) + self.type_legalize[ty] = xgrp + + def get_legalize_action(self, ty): + # type: (ValueType) -> XFormGroup + """ + Get the legalization action to use for `ty`. + """ + return self.type_legalize.get(ty, self.default_legalize) + class EncRecipe(object): """ diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index f809a91ce6..6729267f7b 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -241,6 +241,10 @@ class XFormGroup(object): self.name = name self.__doc__ = doc + def __str__(self): + # type: () -> str + return self.name + def legalize(self, src, dst): # type: (Union[Def, Apply], Rtl) -> None """ diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 135940dd23..de8b138cf6 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -66,6 +66,7 @@ try: from cdsl.predicates import PredNode, PredLeaf # noqa from cdsl.types import ValueType # noqa from cdsl.instructions import Instruction # noqa + from cdsl.xform import XFormGroup # noqa except ImportError: pass @@ -261,12 +262,17 @@ class Level2Table(object): """ Level 2 table mapping instruction opcodes to `EncList` objects. + A level 2 table can be completely empty if it only holds a custom + legalization action for `ty`. + :param ty: Controlling type variable of all entries, or `None`. + :param legalize: Default legalize action for `ty`. """ - def __init__(self, ty): - # type: (ValueType) -> None + def __init__(self, ty, legalize): + # type: (ValueType, XFormGroup) -> None self.ty = ty + self.legalize = legalize # Maps inst -> EncList self.lists = OrderedDict() # type: OrderedDict[Instruction, EncList] @@ -278,6 +284,16 @@ class Level2Table(object): self.lists[inst] = ls return ls + def is_empty(self): + # type: () -> bool + """ + Check if this level 2 table is completely empty. + + This can happen if the associated type simply has an overridden + legalize action. + """ + return len(self.lists) == 0 + def enclists(self): # type: () -> Iterable[EncList] return iter(self.lists.values()) @@ -310,21 +326,32 @@ class Level1Table(object): Level 1 table mapping types to `Level2` objects. """ - def __init__(self): - # type: () -> None + def __init__(self, cpumode): + # type: (CPUMode) -> None + self.cpumode = cpumode self.tables = OrderedDict() # type: OrderedDict[ValueType, Level2Table] # noqa + if cpumode.default_legalize is None: + raise AssertionError( + 'CPU mode {}.{} needs a default legalize action' + .format(cpumode.isa, cpumode)) + self.legalize_code = cpumode.isa.legalize_code( + cpumode.default_legalize) + def __getitem__(self, ty): # type: (ValueType) -> Level2Table tbl = self.tables.get(ty) if not tbl: - tbl = Level2Table(ty) + legalize = self.cpumode.get_legalize_action(ty) + # Allocate a legalization code in a predictable order. + self.cpumode.isa.legalize_code(legalize) + tbl = Level2Table(ty, legalize) self.tables[ty] = tbl return tbl def l2tables(self): # type: () -> Iterable[Level2Table] - return iter(self.tables.values()) + return (l2 for l2 in self.tables.values() if not l2.is_empty()) def make_tables(cpumode): @@ -332,11 +359,17 @@ def make_tables(cpumode): """ Generate tables for `cpumode` as described above. """ - table = Level1Table() + table = Level1Table(cpumode) for enc in cpumode.encodings: ty = enc.ctrl_typevar() inst = enc.inst table[ty][inst].encodings.append(enc) + + # Ensure there are level 1 table entries for all types with a custom + # legalize action. Try to be stable relative to dict ordering. + for ty in sorted(cpumode.type_legalize.keys(), key=str): + table[ty] + return table @@ -412,22 +445,42 @@ def emit_level1_hashtable(cpumode, level1, offt, fmt): 'pub static LEVEL1_{}: [Level1Entry<{}>; {}] = [' .format(cpumode.name.upper(), offt, len(hash_table)), '];'): for level2 in hash_table: - if level2: - l2l = int(math.log(level2.hash_table_len, 2)) - assert l2l > 0, "Hash table too small" - tyname = level2.ty.name if level2.ty is not None else 'void' - fmt.line( - 'Level1Entry ' + - '{{ ty: types::{}, log2len: {}, offset: {:#08x} }},' - .format( - tyname.upper(), - l2l, - level2.hash_table_offset)) + # Empty hash table entry. Include the default legalization action. + if not level2: + fmt.format( + 'Level1Entry {{ ty: types::VOID, log2len: !0, ' + 'offset: 0, legalize: {} }},', + level1.legalize_code) + continue + + if level2.ty is not None: + tyname = level2.ty.rust_name() else: - # Empty entry. - fmt.line( - 'Level1Entry ' + - '{ ty: types::VOID, log2len: 0, offset: 0 },') + tyname = 'types::VOID' + + lcode = cpumode.isa.legalize_code(level2.legalize) + + # Empty level 2 table: Only a specialized legalization action, no + # actual table. + # Set an offset that is out of bounds, but make sure it doesn't + # overflow its type when adding `1< 0, "Level2 hash table too small" + fmt.format( + 'Level1Entry {{ ' + 'ty: {}, log2len: {}, offset: {:#08x}, ' + 'legalize: {} }}, // {}', + tyname, l2l, level2.hash_table_offset, + lcode, level2.legalize) def offset_type(length): diff --git a/lib/cretonne/meta/isa/arm32/defs.py b/lib/cretonne/meta/isa/arm32/defs.py index f90abc2001..6bba598d27 100644 --- a/lib/cretonne/meta/isa/arm32/defs.py +++ b/lib/cretonne/meta/isa/arm32/defs.py @@ -6,9 +6,14 @@ Commonly used definitions. from __future__ import absolute_import from cdsl.isa import TargetISA, CPUMode import base.instructions +from base.legalize import narrow ISA = TargetISA('arm32', [base.instructions.GROUP]) # CPU modes for 32-bit ARM and Thumb2. A32 = CPUMode('A32', ISA) T32 = CPUMode('T32', ISA) + +# TODO: Refine these. +A32.legalize_type(narrow) +T32.legalize_type(narrow) diff --git a/lib/cretonne/meta/isa/arm64/defs.py b/lib/cretonne/meta/isa/arm64/defs.py index e493b4424a..b1ed79b5d6 100644 --- a/lib/cretonne/meta/isa/arm64/defs.py +++ b/lib/cretonne/meta/isa/arm64/defs.py @@ -6,6 +6,10 @@ Commonly used definitions. from __future__ import absolute_import from cdsl.isa import TargetISA, CPUMode import base.instructions +from base.legalize import narrow ISA = TargetISA('arm64', [base.instructions.GROUP]) A64 = CPUMode('A64', ISA) + +# TODO: Refine these +A64.legalize_type(narrow) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index f155795b6f..df1a542101 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -9,6 +9,20 @@ from .defs import I32, I64 from . import recipes as r from . import settings as cfg from . import instructions as x86 +from base.legalize import narrow, expand + +I32.legalize_type( + default=narrow, + i32=expand, + f32=expand, + f64=expand) + +I64.legalize_type( + default=narrow, + i32=expand, + i64=expand, + f32=expand, + f64=expand) for inst, opc in [ (base.iadd, 0x01), diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index 9ec6b34fc0..b9f1f8245d 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -11,6 +11,20 @@ from .recipes import R, Rshamt, Ricmp, I, Iz, Iicmp, Iret, Icall, Icopy from .recipes import U, UJ, UJcall, SB, SBzero, GPsp, GPfi, Irmov from .settings import use_m from cdsl.ast import Var +from base.legalize import narrow, expand + +RV32.legalize_type( + default=narrow, + i32=expand, + f32=expand, + f64=expand) + +RV64.legalize_type( + default=narrow, + i32=expand, + i64=expand, + f32=expand, + f64=expand) # Dummies for instruction predicates. x = Var('x') diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index 1c6645002e..33ec38d3c1 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -9,6 +9,11 @@ use isa::{Encoding, Legalize}; use settings::PredicateView; use std::ops::Range; +/// Legalization action to perform when no encoding can be found for an instruction. +/// +/// This is an index into an ISA-specific table of legalization actions. +pub type LegalizeCode = u8; + /// Level 1 hash table entry. /// /// One level 1 hash table is generated per CPU mode. This table is keyed by the controlling type @@ -19,14 +24,15 @@ use std::ops::Range; /// have a power-of-two size. /// /// Entries are generic over the offset type. It will typically be `u32` or `u16`, depending on the -/// size of the `LEVEL2` table. A `u16` offset allows entries to shrink to 32 bits each, but some -/// ISAs may have tables so large that `u32` offsets are needed. +/// size of the `LEVEL2` table. /// -/// Empty entries are encoded with a 0 `log2len`. This is on the assumption that no level 2 tables -/// have only a single entry. +/// Empty entries are encoded with a `!0` value for `log2len` which will always be out of range. +/// Entries that have a `legalize` value but no level 2 table have an `offset` field that is out f +/// bounds. pub struct Level1Entry + Copy> { pub ty: Type, pub log2len: u8, + pub legalize: LegalizeCode, pub offset: OffT, } @@ -44,7 +50,7 @@ impl + Copy> Table for [Level1Entry] { } fn key(&self, idx: usize) -> Option { - if self[idx].log2len != 0 { + if self[idx].log2len != !0 { Some(self[idx].ty) } else { None @@ -55,7 +61,7 @@ impl + Copy> Table for [Level1Entry] { /// Level 2 hash table entry. /// /// The second level hash tables are keyed by `Opcode`, and contain an offset into the `ENCLISTS` -/// table where the encoding recipes for the instrution are stored. +/// table where the encoding recipes for the instruction are stored. /// /// Entries are generic over the offset type which depends on the size of `ENCLISTS`. A `u16` /// offset allows the entries to be only 32 bits each. There is no benefit to dropping down to `u8` @@ -93,22 +99,28 @@ pub fn lookup_enclist(ctrl_typevar: Type, where OffT1: Into + Copy, OffT2: Into + Copy { - // TODO: The choice of legalization actions here is naive. This needs to be configurable. match probe(level1_table, ctrl_typevar, ctrl_typevar.index()) { - Err(_) => { - // No level 1 entry for the type. - Err(if ctrl_typevar.lane_type().bits() > 32 { - Legalize::Narrow - } else { - Legalize::Expand - }) + Err(l1idx) => { + // No level 1 entry found for the type. + // We have a sentinel entry with the default legalization code. + let l1ent = &level1_table[l1idx]; + Err(l1ent.legalize.into()) } Ok(l1idx) => { + // We have a valid level 1 entry for this type. let l1ent = &level1_table[l1idx]; - let l2tab = &level2_table[l1ent.range()]; - probe(l2tab, opcode, opcode as usize) - .map(|l2idx| l2tab[l2idx].offset.into() as usize) - .map_err(|_| Legalize::Expand) + match level2_table.get(l1ent.range()) { + Some(l2tab) => { + probe(l2tab, opcode, opcode as usize) + .map(|l2idx| l2tab[l2idx].offset.into() as usize) + .map_err(|_| l1ent.legalize.into()) + } + None => { + // The l1ent range is invalid. This means that we just have a customized + // legalization code for this type. The level 2 table is empty. + Err(l1ent.legalize.into()) + } + } } } } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 100c901ec0..7b25aec8ec 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -126,6 +126,20 @@ pub enum Legalize { Expand, } +/// Translate a legalization code into a `Legalize` enum. +/// +/// This mapping is going away soon. It depends on matching the `TargetISA.legalize_code()` +/// mapping. +impl From for Legalize { + fn from(x: u8) -> Legalize { + match x { + 0 => Legalize::Narrow, + 1 => Legalize::Expand, + _ => panic!("Unknown legalization code {}"), + } + } +} + /// Methods that are specialized to a target ISA. pub trait TargetIsa { /// Get the name of this ISA. From 605886a2774a5c5bf82f32925bcf7043b8d6eff9 Mon Sep 17 00:00:00 2001 From: Dimo Date: Thu, 20 Jul 2017 16:32:07 -0700 Subject: [PATCH 0898/3084] Rename Dict[Var, TypeVar] to VarTyping; Add VarMap (Dict[Var,Var]). Add {Ast, Def, Rtl}.{vars(), substitution()} and Def.uses(), Def.definitions() - these enable checking structural equivalence between Rtls and doing variable substitutions between compatible Rtls; Add TypeEnv.permits() routine - allows checking if a given TypeEnv allows a given concrete typing without enumerating all typings (will be useful for determing which semantic transform applies to a given concrete typing). --- lib/cretonne/meta/cdsl/ast.py | 79 +++++++++++++++++++++++++- lib/cretonne/meta/cdsl/instructions.py | 4 +- lib/cretonne/meta/cdsl/test_ti.py | 8 +-- lib/cretonne/meta/cdsl/ti.py | 46 +++++++++++++-- lib/cretonne/meta/cdsl/xform.py | 38 ++++++++++++- 5 files changed, 161 insertions(+), 14 deletions(-) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index 0d87bf8914..fa9c359a62 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -11,15 +11,17 @@ from .predicates import IsEqual, And try: from typing import Union, Tuple, Sequence, TYPE_CHECKING, Dict, List # noqa + from typing import Optional, Set # noqa if TYPE_CHECKING: from .operands import ImmediateKind # noqa from .predicates import PredNode # noqa + VarMap = Dict["Var", "Var"] except ImportError: pass def replace_var(arg, m): - # type: (Expr, Dict[Var, Var]) -> Expr + # type: (Expr, VarMap) -> Expr """ Given a var v return either m[v] or a new variable v' (and remember m[v]=v'). Otherwise return the argument unchanged @@ -74,7 +76,7 @@ class Def(object): ', '.join(map(str, self.defs)), self.expr) def copy(self, m): - # type: (Dict[Var, Var]) -> Def + # type: (VarMap) -> Def """ Return a copy of this Def with vars replaced with fresh variables, in accordance with the map m. Update m as neccessary. @@ -88,6 +90,40 @@ class Def(object): return Def(tuple(new_defs), new_expr) + def definitions(self): + # type: () -> Set[Var] + """ Return the set of all Vars that are defined by self""" + return set(self.defs) + + def uses(self): + # type: () -> Set[Var] + """ Return the set of all Vars that are used(read) by self""" + return set(self.expr.vars()) + + def vars(self): + # type: () -> Set[Var] + """ Return the set of all Vars that appear in self""" + return self.definitions().union(self.uses()) + + def substitution(self, other, s): + # type: (Def, VarMap) -> Optional[VarMap] + """ + If the Defs self and other agree structurally, return a variable + substitution to transform self ot other. Two Defs agree structurally + if the contained Apply's agree structurally. + """ + s = self.expr.substitution(other.expr, s) + + if (s is None): + return s + + assert len(self.defs) == len(other.defs) + for (self_d, other_d) in zip(self.defs, other.defs): + assert self_d not in s # Guaranteed by SSA form + s[self_d] = other_d + + return s + class Expr(object): """ @@ -332,7 +368,7 @@ class Apply(Expr): return pred def copy(self, m): - # type: (Dict[Var, Var]) -> Apply + # type: (VarMap) -> Apply """ Return a copy of this Expr with vars replaced with fresh variables, in accordance with the map m. Update m as neccessary. @@ -340,6 +376,43 @@ class Apply(Expr): return Apply(self.inst, tuple(map(lambda e: replace_var(e, m), self.args))) + def vars(self): + # type: () -> Set[Var] + """ Return the set of all Vars that appear in self""" + res = set() + for i in self.inst.value_opnums: + arg = self.args[i] + assert isinstance(arg, Var) + res.add(arg) + return res + + def substitution(self, other, s): + # type: (Apply, VarMap) -> Optional[VarMap] + """ + If the application self and other agree structurally, return a variable + substitution to transform self ot other. Two applications agree + structurally if: + 1) They are over the same instruction + 2) Every Var v in self, maps to a single Var w in other. I.e for + each use of v in self, w is used in the corresponding place in + other. + """ + if self.inst != other.inst: + return None + + # TODO: Should we check imm/cond codes here as well? + for i in self.inst.value_opnums: + self_a = self.args[i] + other_a = other.args[i] + + assert isinstance(self_a, Var) and isinstance(other_a, Var) + if (self_a not in s): + s[self_a] = other_a + else: + if (s[self_a] != other_a): + return None + return s + class Enumerator(Expr): """ diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 7b8cc55b31..0bbaccb353 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -7,14 +7,16 @@ from .formats import InstructionFormat try: from typing import Union, Sequence, List, Tuple, Any, TYPE_CHECKING # noqa + from typing import Dict # noqa if TYPE_CHECKING: - from .ast import Expr, Apply # noqa + from .ast import Expr, Apply, Var # noqa from .typevar import TypeVar # noqa from .ti import TypeConstraint # noqa # List of operands for ins/outs: OpList = Union[Sequence[Operand], Operand] ConstrList = Union[Sequence[TypeConstraint], TypeConstraint] MaybeBoundInst = Union['Instruction', 'BoundInstruction'] + VarTyping = Dict[Var, TypeVar] except ImportError: pass diff --git a/lib/cretonne/meta/cdsl/test_ti.py b/lib/cretonne/meta/cdsl/test_ti.py index f60a9222f5..9351d9c3ea 100644 --- a/lib/cretonne/meta/cdsl/test_ti.py +++ b/lib/cretonne/meta/cdsl/test_ti.py @@ -13,7 +13,7 @@ from unittest import TestCase from functools import reduce try: - from .ti import TypeMap, ConstraintList, VarMap, TypingOrError # noqa + from .ti import TypeMap, ConstraintList, VarTyping, TypingOrError # noqa from typing import List, Dict, Tuple, TYPE_CHECKING, cast # noqa except ImportError: TYPE_CHECKING = False @@ -61,7 +61,7 @@ def agree(me, other): def check_typing(got_or_err, expected, symtab=None): - # type: (TypingOrError, Tuple[VarMap, ConstraintList], Dict[str, Var]) -> None # noqa + # type: (TypingOrError, Tuple[VarTyping, ConstraintList], Dict[str, Var]) -> None # noqa """ Check that a the typing we received (got_or_err) complies with the expected typing (expected). If symtab is specified, substitute the Vars in @@ -93,7 +93,7 @@ def check_typing(got_or_err, expected, symtab=None): def check_concrete_typing_rtl(var_types, rtl): - # type: (VarMap, Rtl) -> None + # type: (VarTyping, Rtl) -> None """ Check that a concrete type assignment var_types (Dict[Var, TypeVar]) is valid for an Rtl rtl. Specifically check that: @@ -136,7 +136,7 @@ def check_concrete_typing_rtl(var_types, rtl): def check_concrete_typing_xform(var_types, xform): - # type: (VarMap, XForm) -> None + # type: (VarTyping, XForm) -> None """ Check a concrete type assignment var_types for an XForm xform """ diff --git a/lib/cretonne/meta/cdsl/ti.py b/lib/cretonne/meta/cdsl/ti.py index 79023b8b34..ab4ae12f19 100644 --- a/lib/cretonne/meta/cdsl/ti.py +++ b/lib/cretonne/meta/cdsl/ti.py @@ -15,7 +15,7 @@ try: from .typevar import TypeSet # noqa if TYPE_CHECKING: TypeMap = Dict[TypeVar, TypeVar] - VarMap = Dict[Var, TypeVar] + VarTyping = Dict[Var, TypeVar] except ImportError: TYPE_CHECKING = False pass @@ -257,6 +257,7 @@ class TypeEnv(object): Lookup the canonical representative for a Var/TypeVar. """ if (isinstance(arg, Var)): + assert arg in self.vars tv = arg.get_typevar() else: assert (isinstance(arg, TypeVar)) @@ -290,8 +291,16 @@ class TypeEnv(object): """ Add a new constraint """ - if (constr not in self.constraints): - self.constraints.append(constr) + if (constr in self.constraints): + return + + # InTypeset constraints can be expressed by constraining the typeset of + # a variable. No need to add them to self.constraints + if (isinstance(constr, InTypeset)): + self[constr.tv].constrain_types_by_ts(constr.ts) + return + + self.constraints.append(constr) def get_uid(self): # type: () -> str @@ -436,7 +445,7 @@ class TypeEnv(object): return t def concrete_typings(self): - # type: () -> Iterable[VarMap] + # type: () -> Iterable[VarTyping] """ Return an iterable over all possible concrete typings permitted by this TypeEnv. @@ -464,6 +473,35 @@ class TypeEnv(object): yield concrete_var_map + def permits(self, concrete_typing): + # type: (VarTyping) -> bool + """ + Return true iff this TypeEnv permits the (possibly partial) concrete + variable type mapping concrete_typing. + """ + # Each variable has a concrete type, that is a subset of its inferred + # typeset. + for (v, typ) in concrete_typing.items(): + assert typ.singleton_type() is not None + if not typ.get_typeset().issubset(self[v].get_typeset()): + return False + + m = {self[v]: typ for (v, typ) in concrete_typing.items()} + + # Constraints involving vars in concrete_typing are satisfied + for constr in self.constraints: + try: + # If the constraint includes only vars in concrete_typing, we + # can translate it using m. Otherwise we encounter a KeyError + # and ignore it + constr = constr.translate(m) + if not constr.eval(): + return False + except KeyError: + pass + + return True + def dot(self): # type: () -> str """ diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index 6729267f7b..a43846a372 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -4,10 +4,12 @@ Instruction transformations. from __future__ import absolute_import from .ast import Def, Var, Apply from .ti import ti_xform, TypeEnv, get_type_env +from functools import reduce try: from typing import Union, Iterator, Sequence, Iterable, List, Dict # noqa - from .ast import Expr # noqa + from typing import Optional, Set # noqa + from .ast import Expr, VarMap # noqa DefApply = Union[Def, Apply] except ImportError: pass @@ -42,13 +44,45 @@ class Rtl(object): self.rtl = tuple(map(canonicalize_defapply, args)) def copy(self, m): - # type: (Dict[Var, Var]) -> Rtl + # type: (VarMap) -> Rtl """ Return a copy of this rtl with all Vars substituted with copies or according to m. Update m as neccessary. """ return Rtl(*[d.copy(m) for d in self.rtl]) + def vars(self): + # type: () -> Set[Var] + """ Return the set of all Vars that appear in self""" + return reduce(lambda x, y: x.union(y), + [d.vars() for d in self.rtl], + set([])) + + def definitions(self): + # type: () -> Set[Var] + """ Return the set of all Vars defined in self""" + return reduce(lambda x, y: x.union(y), + [d.definitions() for d in self.rtl], + set([])) + + def substitution(self, other, s): + # type: (Rtl, VarMap) -> Optional[VarMap] + """ + If the Rtl self agrees structurally with the Rtl other, return a + substitution to transform self to other. Two Rtls agree structurally if + they have the same sequence of Defs, that agree structurally. + """ + if len(self.rtl) != len(other.rtl): + return None + + for i in range(len(self.rtl)): + s = self.rtl[i].substitution(other.rtl[i], s) + + if s is None: + return None + + return s + class XForm(object): """ From a12fa86e60f5752f5ea75ade0c0cbb441c2ccda6 Mon Sep 17 00:00:00 2001 From: Dimo Date: Thu, 20 Jul 2017 17:20:23 -0700 Subject: [PATCH 0899/3084] Add the BVType; Add suport for bitvectors in TypeVar and TypeSet. --- lib/cretonne/meta/cdsl/ast.py | 2 +- lib/cretonne/meta/cdsl/types.py | 36 +++++++++- lib/cretonne/meta/cdsl/typevar.py | 110 +++++++++++++++++++++++++++--- 3 files changed, 136 insertions(+), 12 deletions(-) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index fa9c359a62..c5e6d58def 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -239,7 +239,7 @@ class Var(Expr): 'typeof_{}'.format(self), 'Type of the pattern variable `{}`'.format(self), ints=True, floats=True, bools=True, - scalars=True, simd=True) + scalars=True, simd=True, bitvecs=True) self.original_typevar = tv self.typevar = tv return self.typevar diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py index cebe472fbf..80b096cbc1 100644 --- a/lib/cretonne/meta/cdsl/types.py +++ b/lib/cretonne/meta/cdsl/types.py @@ -84,7 +84,10 @@ class ScalarType(ValueType): self._vectors = dict() # type: Dict[int, VectorType] # Assign numbers starting from 1. (0 is VOID). ValueType.all_scalars.append(self) - self.number = len(ValueType.all_scalars) + # Numbers are only valid for Cretone types that get emitted to Rust. + # This excludes BVTypes + self.number = len([x for x in ValueType.all_scalars + if not isinstance(x, BVType)]) assert self.number < 16, 'Too many scalar types' def __repr__(self): @@ -239,3 +242,34 @@ class BoolType(ScalarType): # type: () -> int """Return the number of bits in a lane.""" return self.bits + + +class BVType(ScalarType): + """A flat bitvector type. Used for semantics description only.""" + + def __init__(self, bits): + # type: (int) -> None + assert bits > 0, 'Must have positive number of bits' + super(BVType, self).__init__( + name='bv{:d}'.format(bits), + membytes=bits // 8, + doc="A bitvector type with {} bits.".format(bits)) + self.bits = bits + + def __repr__(self): + # type: () -> str + return 'BVType(bits={})'.format(self.bits) + + @staticmethod + def with_bits(bits): + # type: (int) -> BVType + typ = ValueType.by_name('bv{:d}'.format(bits)) + if TYPE_CHECKING: + return cast(BVType, typ) + else: + return typ + + def lane_bits(self): + # type: () -> int + """Return the number of bits in a lane.""" + return self.bits diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index b28502f61b..54a5c92cc0 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -22,6 +22,7 @@ except ImportError: MAX_LANES = 256 MAX_BITS = 64 +MAX_BITVEC = MAX_BITS * MAX_LANES def int_log2(x): @@ -169,15 +170,20 @@ class TypeSet(object): point widths. :param bools: `(min, max)` inclusive range of permitted scalar boolean widths. + :param bitvecs : `(min, max)` inclusive range of permitted bitvector + widths. """ - def __init__(self, lanes=None, ints=None, floats=None, bools=None): - # type: (BoolInterval, BoolInterval, BoolInterval, BoolInterval) -> None # noqa + def __init__(self, lanes=None, ints=None, floats=None, bools=None, + bitvecs=None): + # type: (BoolInterval, BoolInterval, BoolInterval, BoolInterval, BoolInterval) -> None # noqa self.lanes = interval_to_set(decode_interval(lanes, (1, MAX_LANES), 1)) self.ints = interval_to_set(decode_interval(ints, (8, MAX_BITS))) self.floats = interval_to_set(decode_interval(floats, (32, 64))) self.bools = interval_to_set(decode_interval(bools, (1, MAX_BITS))) self.bools = set(filter(legal_bool, self.bools)) + self.bitvecs = interval_to_set(decode_interval(bitvecs, + (1, MAX_BITVEC))) def copy(self): # type: (TypeSet) -> TypeSet @@ -188,12 +194,13 @@ class TypeSet(object): return deepcopy(self) def typeset_key(self): - # type: () -> Tuple[Tuple, Tuple, Tuple, Tuple] + # type: () -> Tuple[Tuple, Tuple, Tuple, Tuple, Tuple] """Key tuple used for hashing and equality.""" return (tuple(sorted(list(self.lanes))), tuple(sorted(list(self.ints))), tuple(sorted(list(self.floats))), - tuple(sorted(list(self.bools)))) + tuple(sorted(list(self.bools))), + tuple(sorted(list(self.bitvecs)))) def __hash__(self): # type: () -> int @@ -222,11 +229,14 @@ class TypeSet(object): s += ', floats={}'.format(pp_set(self.floats)) if len(self.bools) > 0: s += ', bools={}'.format(pp_set(self.bools)) + if len(self.bitvecs) > 0: + s += ', bitvecs={}'.format(pp_set(self.bitvecs)) return s + ')' def emit_fields(self, fmt): # type: (Formatter) -> None """Emit field initializers for this typeset.""" + assert len(self.bitvecs) == 0, "Bitvector types are not emitable." fmt.comment(repr(self)) fields = (('lanes', 16), @@ -262,6 +272,7 @@ class TypeSet(object): self.ints.intersection_update(other.ints) self.floats.intersection_update(other.floats) self.bools.intersection_update(other.bools) + self.bitvecs.intersection_update(other.bitvecs) return self @@ -273,7 +284,8 @@ class TypeSet(object): return self.lanes.issubset(other.lanes) and \ self.ints.issubset(other.ints) and \ self.floats.issubset(other.floats) and \ - self.bools.issubset(other.bools) + self.bools.issubset(other.bools) and \ + self.bitvecs.issubset(other.bitvecs) def lane_of(self): # type: () -> TypeSet @@ -282,6 +294,7 @@ class TypeSet(object): """ new = self.copy() new.lanes = set([1]) + new.bitvecs = set() return new def as_bool(self): @@ -292,6 +305,7 @@ class TypeSet(object): new = self.copy() new.ints = set() new.floats = set() + new.bitvecs = set() if len(self.lanes.difference(set([1]))) > 0: new.bools = self.ints.union(self.floats).union(self.bools) @@ -309,6 +323,7 @@ class TypeSet(object): new.ints = set([x//2 for x in self.ints if x > 8]) new.floats = set([x//2 for x in self.floats if x > 32]) new.bools = set([x//2 for x in self.bools if x > 8]) + new.bitvecs = set([x//2 for x in self.bitvecs if x > 1]) return new @@ -322,6 +337,7 @@ class TypeSet(object): new.floats = set([x*2 for x in self.floats if x < MAX_BITS]) new.bools = set(filter(legal_bool, set([x*2 for x in self.bools if x < MAX_BITS]))) + new.bitvecs = set([x*2 for x in self.bitvecs if x < MAX_BITVEC]) return new @@ -331,6 +347,7 @@ class TypeSet(object): Return a TypeSet describing the image of self across halfvector """ new = self.copy() + new.bitvecs = set() new.lanes = set([x//2 for x in self.lanes if x > 1]) return new @@ -341,10 +358,29 @@ class TypeSet(object): Return a TypeSet describing the image of self across doublevector """ new = self.copy() + new.bitvecs = set() new.lanes = set([x*2 for x in self.lanes if x < MAX_LANES]) return new + def to_bitvec(self): + # type: () -> TypeSet + """ + Return a TypeSet describing the image of self across to_bitvec + """ + assert len(self.bitvecs) == 0 + all_scalars = self.ints.union(self.floats.union(self.bools)) + + new = self.copy() + new.lanes = set([1]) + new.ints = set() + new.bools = set() + new.floats = set() + new.bitvecs = set([lane_w * nlanes for lane_w in all_scalars + for nlanes in self.lanes]) + + return new + def image(self, func): # type: (str) -> TypeSet """ @@ -362,6 +398,8 @@ class TypeSet(object): return self.half_vector() elif (func == TypeVar.DOUBLEVECTOR): return self.double_vector() + elif (func == TypeVar.TOBITVEC): + return self.to_bitvec() else: assert False, "Unknown derived function: " + func @@ -376,10 +414,12 @@ class TypeSet(object): if (func == TypeVar.LANEOF): new = self.copy() + new.bitvecs = set() new.lanes = set([2**i for i in range(0, int_log2(MAX_LANES)+1)]) return new elif (func == TypeVar.ASBOOL): new = self.copy() + new.bitvecs = set() if 1 not in self.bools: new.ints = self.bools.difference(set([1])) @@ -400,6 +440,39 @@ class TypeSet(object): return self.double_vector() elif (func == TypeVar.DOUBLEVECTOR): return self.half_vector() + elif (func == TypeVar.TOBITVEC): + new = TypeSet() + + # Start with all possible lanes/ints/floats/bools + lanes = interval_to_set(decode_interval(True, (1, MAX_LANES), 1)) + ints = interval_to_set(decode_interval(True, (8, MAX_BITS))) + floats = interval_to_set(decode_interval(True, (32, 64))) + bools = interval_to_set(decode_interval(True, (1, MAX_BITS))) + + # See which combinations have a size that appears in self.bitvecs + has_t = set() # type: Set[Tuple[str, int, int]] + for l in lanes: + for i in ints: + if i * l in self.bitvecs: + has_t.add(('i', i, l)) + for i in bools: + if i * l in self.bitvecs: + has_t.add(('b', i, l)) + for i in floats: + if i * l in self.bitvecs: + has_t.add(('f', i, l)) + + for (t, width, lane) in has_t: + new.lanes.add(lane) + if (t == 'i'): + new.ints.add(width) + elif (t == 'b'): + new.bools.add(width) + else: + assert t == 'f' + new.floats.add(width) + + return new else: assert False, "Unknown derived function: " + func @@ -409,7 +482,7 @@ class TypeSet(object): Return the number of concrete types represented by this typeset """ return len(self.lanes) * (len(self.ints) + len(self.floats) + - len(self.bools)) + len(self.bools) + len(self.bitvecs)) def concrete_types(self): # type: () -> Iterable[types.ValueType] @@ -427,6 +500,8 @@ class TypeSet(object): yield by(types.FloatType.with_bits(bits), nlanes) for bits in self.bools: yield by(types.BoolType.with_bits(bits), nlanes) + for bits in self.bitvecs: + yield by(types.BVType.with_bits(bits), nlanes) def get_singleton(self): # type: () -> types.ValueType @@ -458,14 +533,15 @@ class TypeVar(object): :param scalars: Allow type variable to assume scalar types. :param simd: Allow type variable to assume vector types, or `(min, max)` lane count range. + :param bitvecs: Allow all BitVec base types, or `(min, max)` bit-range. """ def __init__( self, name, doc, ints=False, floats=False, bools=False, - scalars=True, simd=False, + scalars=True, simd=False, bitvecs=False, base=None, derived_func=None): - # type: (str, str, BoolInterval, BoolInterval, BoolInterval, bool, BoolInterval, TypeVar, str) -> None # noqa + # type: (str, str, BoolInterval, BoolInterval, BoolInterval, bool, BoolInterval, BoolInterval, TypeVar, str) -> None # noqa self.name = name self.__doc__ = doc self.is_derived = isinstance(base, TypeVar) @@ -482,7 +558,8 @@ class TypeVar(object): lanes=lanes, ints=ints, floats=floats, - bools=bools) + bools=bools, + bitvecs=bitvecs) @staticmethod def singleton(typ): @@ -498,6 +575,7 @@ class TypeVar(object): ints = None floats = None bools = None + bitvecs = None if isinstance(scalar, types.IntType): ints = (scalar.bits, scalar.bits) @@ -505,10 +583,13 @@ class TypeVar(object): floats = (scalar.bits, scalar.bits) elif isinstance(scalar, types.BoolType): bools = (scalar.bits, scalar.bits) + elif isinstance(scalar, types.BVType): + bitvecs = (scalar.bits, scalar.bits) tv = TypeVar( typ.name, typ.__doc__, - ints, floats, bools, simd=lanes) + ints=ints, floats=floats, bools=bools, + bitvecs=bitvecs, simd=lanes) return tv def __str__(self): @@ -558,6 +639,7 @@ class TypeVar(object): DOUBLEWIDTH = 'double_width' HALFVECTOR = 'half_vector' DOUBLEVECTOR = 'double_vector' + TOBITVEC = 'to_bitvec' @staticmethod def is_bijection(func): @@ -668,6 +750,14 @@ class TypeVar(object): """ return TypeVar.derived(self, self.DOUBLEVECTOR) + def to_bitvec(self): + # type: () -> TypeVar + """ + Return a derived type variable that represent a flat bitvector with + the same size as self + """ + return TypeVar.derived(self, self.TOBITVEC) + def singleton_type(self): # type: () -> ValueType """ From a5fe64440fd13303e66b4181ee5125e2b2cbf990 Mon Sep 17 00:00:00 2001 From: Dimo Date: Thu, 20 Jul 2017 18:21:55 -0700 Subject: [PATCH 0900/3084] Add insturction semantics. Add semantics for vsplit,vconcat,iadd. Add initial tests --- lib/cretonne/meta/base/semantics.py | 57 +++ lib/cretonne/meta/cdsl/instructions.py | 15 +- lib/cretonne/meta/cdsl/ti.py | 56 ++- lib/cretonne/meta/cdsl/types.py | 5 + lib/cretonne/meta/cdsl/typevar.py | 7 + lib/cretonne/meta/cdsl/xform.py | 21 +- lib/cretonne/meta/semantics/__init__.py | 55 +++ lib/cretonne/meta/semantics/elaborate.py | 170 +++++++++ lib/cretonne/meta/semantics/primitives.py | 71 ++++ lib/cretonne/meta/semantics/test_elaborate.py | 337 ++++++++++++++++++ lib/cretonne/meta/semantics/types.py | 9 + 11 files changed, 797 insertions(+), 6 deletions(-) create mode 100644 lib/cretonne/meta/base/semantics.py create mode 100644 lib/cretonne/meta/semantics/__init__.py create mode 100644 lib/cretonne/meta/semantics/elaborate.py create mode 100644 lib/cretonne/meta/semantics/primitives.py create mode 100644 lib/cretonne/meta/semantics/test_elaborate.py create mode 100644 lib/cretonne/meta/semantics/types.py diff --git a/lib/cretonne/meta/base/semantics.py b/lib/cretonne/meta/base/semantics.py new file mode 100644 index 0000000000..d85dad0682 --- /dev/null +++ b/lib/cretonne/meta/base/semantics.py @@ -0,0 +1,57 @@ +from __future__ import absolute_import +from semantics.primitives import prim_to_bv, prim_from_bv, bvsplit, bvconcat,\ + bvadd +from .instructions import vsplit, vconcat, iadd +from cdsl.xform import XForm, Rtl +from cdsl.ast import Var +from cdsl.typevar import TypeSet +from cdsl.ti import InTypeset +import semantics.types # noqa + +x = Var('x') +y = Var('y') +a = Var('a') +xhi = Var('xhi') +yhi = Var('yhi') +ahi = Var('ahi') +xlo = Var('xlo') +ylo = Var('ylo') +alo = Var('alo') +lo = Var('lo') +hi = Var('hi') +bvx = Var('bvx') +bvy = Var('bvy') +bva = Var('bva') +bvlo = Var('bvlo') +bvhi = Var('bvhi') + +ScalarTS = TypeSet(lanes=(1, 1), ints=True, floats=True, bools=True) + +vsplit.set_semantics( + XForm(Rtl((lo, hi) << vsplit(x)), + Rtl(bvx << prim_to_bv(x), + (bvlo, bvhi) << bvsplit(bvx), + lo << prim_from_bv(bvlo), + hi << prim_from_bv(bvhi)))) + +vconcat.set_semantics( + XForm(Rtl(x << vconcat(lo, hi)), + Rtl(bvlo << prim_to_bv(lo), + bvhi << prim_to_bv(hi), + bvx << bvconcat(bvlo, bvhi), + x << prim_from_bv(bvx)))) + +iadd.set_semantics([ + XForm(Rtl(a << iadd(x, y)), + Rtl(bvx << prim_to_bv(x), + bvy << prim_to_bv(y), + bva << bvadd(bvx, bvy), + a << prim_from_bv(bva)), + constraints=[InTypeset(x.get_typevar(), ScalarTS)]), + XForm(Rtl(a << iadd(x, y)), + Rtl((xlo, xhi) << vsplit(x), + (ylo, yhi) << vsplit(y), + alo << iadd(xlo, ylo), + ahi << iadd(xhi, yhi), + a << vconcat(alo, ahi))) +]) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 0bbaccb353..ee9bf15e5e 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -12,11 +12,12 @@ try: from .ast import Expr, Apply, Var # noqa from .typevar import TypeVar # noqa from .ti import TypeConstraint # noqa + from .xform import XForm # List of operands for ins/outs: OpList = Union[Sequence[Operand], Operand] ConstrList = Union[Sequence[TypeConstraint], TypeConstraint] MaybeBoundInst = Union['Instruction', 'BoundInstruction'] - VarTyping = Dict[Var, TypeVar] + InstructionSemantics = List[XForm] except ImportError: pass @@ -119,6 +120,7 @@ class Instruction(object): self.outs = self._to_operand_tuple(outs) self.constraints = self._to_constraint_tuple(constraints) self.format = InstructionFormat.lookup(self.ins, self.outs) + self.semantics = None # type: InstructionSemantics # Opcode number, assigned by gen_instr.py. self.number = None # type: int @@ -334,6 +336,17 @@ class Instruction(object): from .ast import Apply # noqa return Apply(self, args) + def set_semantics(self, sem): + # type: (Union[XForm, InstructionSemantics]) -> None + """Set our semantics.""" + from semantics import verify_semantics + + if not isinstance(sem, list): + sem = [sem] + + verify_semantics(self, sem) + self.semantics = sem + class BoundInstruction(object): """ diff --git a/lib/cretonne/meta/cdsl/ti.py b/lib/cretonne/meta/cdsl/ti.py index ab4ae12f19..b80396b428 100644 --- a/lib/cretonne/meta/cdsl/ti.py +++ b/lib/cretonne/meta/cdsl/ti.py @@ -54,8 +54,8 @@ class TypeConstraint(object): """ Return true iff all typevars in the constraint are singletons. """ - tvs = filter(lambda x: isinstance(x, TypeVar), self._args()) - return [] == list(filter(lambda x: x.singleton_type() is None, tvs)) + return [] == list(filter(lambda x: x.singleton_type() is None, + self.tvs())) def __hash__(self): # type: () -> int @@ -69,6 +69,13 @@ class TypeConstraint(object): """ assert False, "Abstract" + def tvs(self): + # type: () -> Iterable[TypeVar] + """ + Return the typevars contained in this constraint. + """ + return filter(lambda x: isinstance(x, TypeVar), self._args()) + def is_trivial(self): # type: () -> bool """ @@ -218,6 +225,47 @@ class WiderOrEq(TypeConstraint): return typ1.wider_or_equal(typ2) +class SameWidth(TypeConstraint): + """ + Constraint specifying that two types have the same width. E.g. i32x2 has + the same width as i64x1, i16x4, f32x2, f64, b1x64 etc. + """ + def __init__(self, tv1, tv2): + # type: (TypeVar, TypeVar) -> None + self.tv1 = tv1 + self.tv2 = tv2 + + def _args(self): + # type: () -> Tuple[Any,...] + """ See TypeConstraint._args() """ + return (self.tv1, self.tv2) + + def is_trivial(self): + # type: () -> bool + """ See TypeConstraint.is_trivial() """ + # Trivially true + if (self.tv1 == self.tv2): + return True + + ts1 = self.tv1.get_typeset() + ts2 = self.tv2.get_typeset() + + # Trivially False + if len(ts1.widths().intersection(ts2.widths())) == 0: + return True + + return self.is_concrete() + + def eval(self): + # type: () -> bool + """ See TypeConstraint.eval() """ + assert self.is_concrete() + typ1 = self.tv1.singleton_type() + typ2 = self.tv2.singleton_type() + + return (typ1.width() == typ2.width()) + + class TypeEnv(object): """ Class encapsulating the neccessary book keeping for type inference. @@ -537,6 +585,10 @@ class TypeEnv(object): elif isinstance(constr, WiderOrEq): assert constr.tv1 in nodes and constr.tv2 in nodes edges.add((constr.tv1, constr.tv2, "dashed", "forward", ">=")) + elif isinstance(constr, SameWidth): + assert constr.tv1 in nodes and constr.tv2 in nodes + edges.add((constr.tv1, constr.tv2, "dashed", "none", + "same_width")) else: assert False, "Can't display constraint {}".format(constr) diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py index 80b096cbc1..846cc442c9 100644 --- a/lib/cretonne/meta/cdsl/types.py +++ b/lib/cretonne/meta/cdsl/types.py @@ -59,6 +59,11 @@ class ValueType(object): """Return the number of lanes.""" assert False, "Abstract" + def width(self): + # type: () -> int + """Return the total number of bits of an instance of this type.""" + return self.lane_count() * self.lane_bits() + def wider_or_equal(self, other): # type: (ValueType) -> bool """ diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 54a5c92cc0..2e7d4cb5cc 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -513,6 +513,13 @@ class TypeSet(object): assert len(types) == 1 return types[0] + def widths(self): + # type: () -> Set[int] + """ Return a set of the widths of all possible types in self""" + scalar_w = self.ints.union(self.floats.union(self.bools)) + scalar_w = scalar_w.union(self.bitvecs) + return set(w * l for l in self.lanes for w in scalar_w) + class TypeVar(object): """ diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index a43846a372..42655272b8 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -10,6 +10,8 @@ try: from typing import Union, Iterator, Sequence, Iterable, List, Dict # noqa from typing import Optional, Set # noqa from .ast import Expr, VarMap # noqa + from .ti import TypeConstraint # noqa + from .typevar import TypeVar # noqa DefApply = Union[Def, Apply] except ImportError: pass @@ -89,7 +91,8 @@ class XForm(object): An instruction transformation consists of a source and destination pattern. Patterns are expressed in *register transfer language* as tuples of - `ast.Def` or `ast.Expr` nodes. + `ast.Def` or `ast.Expr` nodes. A pattern may optionally have a sequence of + TypeConstraints, that additionally limit the set of cases when it applies. A legalization pattern must have a source pattern containing only a single instruction. @@ -111,8 +114,8 @@ class XForm(object): ) """ - def __init__(self, src, dst): - # type: (Rtl, Rtl) -> None + def __init__(self, src, dst, constraints=None): + # type: (Rtl, Rtl, Optional[Sequence[TypeConstraint]]) -> None self.src = src self.dst = dst # Variables that are inputs to the source pattern. @@ -146,6 +149,18 @@ class XForm(object): raw_ti.normalize() self.ti = raw_ti.extract() + def interp_tv(tv): + # type: (TypeVar) -> TypeVar + """ Convert typevars according to symtab """ + if not tv.name.startswith("typeof_"): + return tv + return symtab[tv.name[len("typeof_"):]].get_typevar() + + if constraints is not None: + for c in constraints: + type_m = {tv: interp_tv(tv) for tv in c.tvs()} + self.ti.add_constraint(c.translate(type_m)) + # Sanity: The set of inferred free typevars should be a subset of the # TVs corresponding to Vars appearing in src free_typevars = set(self.ti.free_typevars()) diff --git a/lib/cretonne/meta/semantics/__init__.py b/lib/cretonne/meta/semantics/__init__.py new file mode 100644 index 0000000000..d2c327c988 --- /dev/null +++ b/lib/cretonne/meta/semantics/__init__.py @@ -0,0 +1,55 @@ +"""Definitions for the semantics segment of the Cretonne language.""" +from cdsl.ti import TypeEnv, ti_rtl, get_type_env + +try: + from typing import List, Dict, Tuple # noqa + from cdsl.ast import Var # noqa + from cdsl.xform import XForm # noqa + from cdsl.ti import VarTyping # noqa + from cdsl.instructions import Instruction, InstructionSemantics # noqa +except ImportError: + pass + + +def verify_semantics(inst, sem): + # type: (Instruction, InstructionSemantics) -> None + """ + Verify that the semantics sem correctly describes the instruction inst. + This involves checking that: + 1) For all XForms x \in sem, x.src consists of a single instance of + inst + 2) For any possible concrete typing of inst there is exactly 1 XForm x + in sem that applies. + """ + # 1) The source rtl is always a single instance of inst. + for xform in sem: + assert len(xform.src.rtl) == 1 and\ + xform.src.rtl[0].expr.inst == inst,\ + "XForm {} doesn't describe instruction {}."\ + .format(xform, inst) + + # 2) Any possible typing for the instruction should be covered by + # exactly ONE semantic XForm + inst_rtl = sem[0].src + typenv = get_type_env(ti_rtl(inst_rtl, TypeEnv())) + + # This bit is awkward. Concrete typing is defined in terms of the vars + # of one Rtl. We arbitrarily picked that Rtl to be sem[0].src. For any + # other XForms in sem, we must build a substitution form + # sem[0].src->sem[N].src, before we can check if sem[N] permits one of + # the concrete typings of our Rtl. + # TODO: Can this be made cleaner? + subst = [inst_rtl.substitution(x.src, {}) for x in sem] + assert not any(x is None for x in subst) + sub_sem = list(zip(subst, sem)) # type: List[Tuple[Dict[Var, Var], XForm]] # noqa + + def subst_typing(typing, sub): + # type: (VarTyping, Dict[Var, Var]) -> VarTyping + return {sub[v]: tv for (v, tv) in typing.items()} + + for t in typenv.concrete_typings(): + matching_xforms = [x for (s, x) in sub_sem + if x.ti.permits(subst_typing(t, s))] + assert len(matching_xforms) == 1,\ + ("Possible typing {} of {} not matched by exactly one case " + + ": {}").format(t, inst, matching_xforms) diff --git a/lib/cretonne/meta/semantics/elaborate.py b/lib/cretonne/meta/semantics/elaborate.py new file mode 100644 index 0000000000..e6be0180e1 --- /dev/null +++ b/lib/cretonne/meta/semantics/elaborate.py @@ -0,0 +1,170 @@ +""" +Tools to elaborate a given Rtl with concrete types into its semantically +equivalent primitive version. Its elaborated primitive version contains only +primitive cretonne instructions, which map well to SMTLIB functions. +""" +from .primitives import GROUP as PRIMITIVES, prim_to_bv, prim_from_bv +from cdsl.ti import ti_rtl, TypeEnv, get_type_env +from cdsl.typevar import TypeVar + +try: + from typing import TYPE_CHECKING, Dict, Union, List, Set, Tuple # noqa + from cdsl.xform import Rtl, XForm # noqa + from cdsl.ast import Var, Def, VarMap # noqa + from cdsl.ti import VarTyping # noqa +except ImportError: + TYPE_CHECKING = False + + +def is_rtl_concrete(r): + # type: (Rtl) -> bool + """Return True iff every Var in the Rtl r has a single type.""" + return all(v.get_typevar().singleton_type() is not None for v in r.vars()) + + +def cleanup_concrete_rtl(r): + # type: (Rtl) -> Rtl + """ + Given an Rtl r + 1) assert that there is only 1 possible concrete typing T for r + 2) Assign a singleton TV with the single type t \in T for each Var v \in r + """ + # 1) Infer the types of any of the remaining vars in res + typenv = get_type_env(ti_rtl(r, TypeEnv())) + typenv.normalize() + typenv = typenv.extract() + + # 2) Make sure there is only one possible type assignment + typings = list(typenv.concrete_typings()) + assert len(typings) == 1 + typing = typings[0] + + # 3) Assign the only possible type to each variable. + for v in typenv.vars: + if v.get_typevar().singleton_type() is not None: + continue + + v.set_typevar(TypeVar.singleton(typing[v].singleton_type())) + + return r + + +def apply(r, x, suffix=None): + # type: (Rtl, XForm, str) -> Rtl + """ + Given a concrete Rtl r and XForm x, s.t. r matches x.src, return the + corresponding concrete x.dst. If suffix is provided, any temporary defs are + renamed with '.suffix' appended to their old name. + """ + assert is_rtl_concrete(r) + s = x.src.substitution(r, {}) # type: VarMap + assert s is not None + + if (suffix is not None): + for v in x.dst.vars(): + if v.is_temp(): + assert v not in s + s[v] = Var(v.name + '.' + suffix) + + dst = x.dst.copy(s) + return cleanup_concrete_rtl(dst) + + +def find_matching_xform(d): + # type: (Def) -> XForm + """ + Given a concrete Def d, find the unique semantic XForm x in + d.expr.inst.semantics that applies to it. + """ + res = [] # type: List[XForm] + typing = {v: v.get_typevar() for v in d.vars()} # type: VarTyping + + for x in d.expr.inst.semantics: + subst = d.substitution(x.src.rtl[0], {}) + + if x.ti.permits({subst[v]: tv for (v, tv) in typing.items()}): + res.append(x) + + assert len(res) == 1 + return res[0] + + +def elaborate(r): + # type: (Rtl) -> Rtl + """ + Given an Rtl r, return a semantically equivalent Rtl r1 consisting only + primitive instructions. + """ + fp = False + primitives = set(PRIMITIVES.instructions) + idx = 0 + + while not fp: + assert is_rtl_concrete(r) + new_defs = [] # type: List[Def] + fp = True + + for d in r.rtl: + inst = d.expr.inst + + if (inst not in primitives): + transformed = apply(Rtl(d), find_matching_xform(d), str(idx)) + idx += 1 + new_defs.extend(transformed.rtl) + fp = False + else: + new_defs.append(d) + + r.rtl = tuple(new_defs) + + return r + + +def cleanup_semantics(r, outputs): + # type: (Rtl, Set[Var]) -> Rtl + """ + The elaboration process creates a lot of redundant instruction pairs of the + shape: + + a.0 << prim_from_bv(bva.0) + ... + bva.1 << prim_to_bv(a.0) + ... + + Contract these to ease manual inspection. + """ + new_defs = [] # type: List[Def] + subst_m = {v: v for v in r.vars()} # type: VarMap + definition = {} # type: Dict[Var, Def] + + # Pass 1: Remove redundant prim_to_bv + for d in r.rtl: + inst = d.expr.inst + + if (inst == prim_to_bv): + if d.expr.args[0] in definition: + assert isinstance(d.expr.args[0], Var) + def_loc = definition[d.expr.args[0]] + + if def_loc.expr.inst == prim_from_bv: + assert isinstance(def_loc.expr.args[0], Var) + subst_m[d.defs[0]] = def_loc.expr.args[0] + continue + + new_def = d.copy(subst_m) + + for v in new_def.defs: + assert v not in definition # Guaranteed by SSA + definition[v] = new_def + + new_defs.append(new_def) + + # Pass 2: Remove dead prim_from_bv + live = set(outputs) # type: Set[Var] + for d in new_defs: + live = live.union(d.uses()) + + new_defs = [d for d in new_defs if not (d.expr.inst == prim_from_bv and + d.defs[0] not in live)] + + return Rtl(*new_defs) diff --git a/lib/cretonne/meta/semantics/primitives.py b/lib/cretonne/meta/semantics/primitives.py new file mode 100644 index 0000000000..d5b6b35d8b --- /dev/null +++ b/lib/cretonne/meta/semantics/primitives.py @@ -0,0 +1,71 @@ +""" +Cretonne primitive instruction set. + +This module defines a primitive instruction set, in terms of which the base set +is described. Most instructions in this set correspond 1-1 with an SMTLIB +bitvector function. +""" +from __future__ import absolute_import +from cdsl.operands import Operand +from cdsl.typevar import TypeVar +from cdsl.instructions import Instruction, InstructionGroup +from cdsl.ti import SameWidth +import base.formats # noqa + +GROUP = InstructionGroup("primitive", "Primitive instruction set") + +BV = TypeVar('BV', 'A bitvector type.', bitvecs=True) +Real = TypeVar('Real', 'Any real type.', ints=True, floats=True, + bools=True, simd=True) + +x = Operand('x', BV, doc="A semantic value X") +y = Operand('x', BV, doc="A semantic value Y (same width as X)") +a = Operand('a', BV, doc="A semantic value A (same width as X)") + +real = Operand('real', Real, doc="A real cretonne value") +fromReal = Operand('fromReal', Real.to_bitvec(), + doc="A real cretonne value converted to a BV") + +prim_to_bv = Instruction( + 'prim_to_bv', r""" + Convert an SSA Value to a flat bitvector + """, + ins=(real), outs=(fromReal)) + +# Note that when converting from BV->real values, we use a constraint and not a +# derived function. This reflects that fact that to_bitvec() is not a +# bijection. +prim_from_bv = Instruction( + 'prim_from_bv', r""" + Convert a flat bitvector to a real SSA Value. + """, + ins=(x), outs=(real), + constraints=SameWidth(BV, Real)) + +xh = Operand('xh', BV.half_width(), + doc="A semantic value representing the upper half of X") +xl = Operand('xl', BV.half_width(), + doc="A semantic value representing the lower half of X") +bvsplit = Instruction( + 'bvsplit', r""" + """, + ins=(x), outs=(xh, xl)) + +xy = Operand('xy', BV.double_width(), + doc="A semantic value representing the concatenation of X and Y") +bvconcat = Instruction( + 'bvconcat', r""" + """, + ins=(x, y), outs=xy) + +bvadd = Instruction( + 'bvadd', r""" + Standard 2's complement addition. Equivalent to wrapping integer + addition: :math:`a := x + y \pmod{2^B}`. + + This instruction does not depend on the signed/unsigned interpretation + of the operands. + """, + ins=(x, y), outs=a) + +GROUP.close() diff --git a/lib/cretonne/meta/semantics/test_elaborate.py b/lib/cretonne/meta/semantics/test_elaborate.py new file mode 100644 index 0000000000..15d4178ae8 --- /dev/null +++ b/lib/cretonne/meta/semantics/test_elaborate.py @@ -0,0 +1,337 @@ +from __future__ import absolute_import +from base.instructions import vselect, vsplit, vconcat, iconst, iadd, bint +from base.instructions import b1, icmp, ireduce +from base.immediates import intcc +from base.types import i64, i8, b32, i32, i16, f32 +from cdsl.typevar import TypeVar +from cdsl.ast import Var +from cdsl.xform import Rtl +from unittest import TestCase +from .elaborate import cleanup_concrete_rtl, elaborate, is_rtl_concrete,\ + cleanup_semantics +from .primitives import prim_to_bv, bvsplit, prim_from_bv, bvconcat, bvadd +import base.semantics # noqa + + +def concrete_rtls_eq(r1, r2): + # type: (Rtl, Rtl) -> bool + """ + Check whether 2 concrete Rtls are equivalent. That is: + 1) They are structurally the same (i.e. there is a substitution between + them) + 2) Corresponding Vars between them have the same singleton type. + """ + assert is_rtl_concrete(r1) + assert is_rtl_concrete(r2) + + s = r1.substitution(r2, {}) + + if s is None: + return False + + for (v, v1) in s.items(): + if v.get_typevar().singleton_type() !=\ + v1.get_typevar().singleton_type(): + return False + + return True + + +class TestCleanupConcreteRtl(TestCase): + """ + Test cleanup_concrete_rtl(). cleanup_concrete_rtl() should take Rtls for + which we can infer a single concrete typing, and update the TypeVars + in-place to singleton TVs. + """ + def test_cleanup_concrete_rtl(self): + # type: () -> None + typ = i64.by(4) + x = Var('x') + lo = Var('lo') + hi = Var('hi') + + x.set_typevar(TypeVar.singleton(typ)) + r = Rtl( + (lo, hi) << vsplit(x), + ) + r1 = cleanup_concrete_rtl(r) + + s = r.substitution(r1, {}) + assert s is not None + assert s[x].get_typevar().singleton_type() == typ + assert s[lo].get_typevar().singleton_type() == i64.by(2) + assert s[hi].get_typevar().singleton_type() == i64.by(2) + + def test_cleanup_concrete_rtl_fail(self): + # type: () -> None + x = Var('x') + lo = Var('lo') + hi = Var('hi') + r = Rtl( + (lo, hi) << vsplit(x), + ) + + with self.assertRaises(AssertionError): + cleanup_concrete_rtl(r) + + def test_cleanup_concrete_rtl_ireduce(self): + # type: () -> None + x = Var('x') + y = Var('y') + x.set_typevar(TypeVar.singleton(i8.by(2))) + r = Rtl( + y << ireduce(x), + ) + + r1 = cleanup_concrete_rtl(r) + + s = r.substitution(r1, {}) + assert s is not None + assert s[x].get_typevar().singleton_type() == i8.by(2) + assert s[y].get_typevar().singleton_type() == i8.by(2) + + def test_cleanup_concrete_rtl_ireduce_bad(self): + # type: () -> None + x = Var('x') + y = Var('y') + x.set_typevar(TypeVar.singleton(i16.by(1))) + r = Rtl( + y << ireduce(x), + ) + + with self.assertRaises(AssertionError): + cleanup_concrete_rtl(r) + + def test_vselect_icmpimm(self): + # type: () -> None + x = Var('x') + y = Var('y') + z = Var('z') + w = Var('w') + v = Var('v') + zeroes = Var('zeroes') + imm0 = Var("imm0") + + zeroes.set_typevar(TypeVar.singleton(i32.by(4))) + z.set_typevar(TypeVar.singleton(f32.by(4))) + + r = Rtl( + zeroes << iconst(imm0), + y << icmp(intcc.eq, x, zeroes), + v << vselect(y, z, w), + ) + + r1 = cleanup_concrete_rtl(r) + + s = r.substitution(r1, {}) + assert s is not None + assert s[zeroes].get_typevar().singleton_type() == i32.by(4) + assert s[x].get_typevar().singleton_type() == i32.by(4) + assert s[y].get_typevar().singleton_type() == b32.by(4) + assert s[z].get_typevar().singleton_type() == f32.by(4) + assert s[w].get_typevar().singleton_type() == f32.by(4) + assert s[v].get_typevar().singleton_type() == f32.by(4) + + def test_bint(self): + # type: () -> None + x = Var('x') + y = Var('y') + z = Var('z') + w = Var('w') + v = Var('v') + u = Var('u') + + x.set_typevar(TypeVar.singleton(i32.by(8))) + z.set_typevar(TypeVar.singleton(i32.by(8))) + # TODO: Relax this to simd=True + v.set_typevar(TypeVar('v', '', bools=(1, 1), simd=(8, 8))) + + r = Rtl( + z << iadd(x, y), + w << bint(v), + u << iadd(z, w) + ) + + r1 = cleanup_concrete_rtl(r) + + s = r.substitution(r1, {}) + assert s is not None + assert s[x].get_typevar().singleton_type() == i32.by(8) + assert s[y].get_typevar().singleton_type() == i32.by(8) + assert s[z].get_typevar().singleton_type() == i32.by(8) + assert s[w].get_typevar().singleton_type() == i32.by(8) + assert s[u].get_typevar().singleton_type() == i32.by(8) + assert s[v].get_typevar().singleton_type() == b1.by(8) + + +class TestElaborate(TestCase): + """ + Test semantics elaboration. + """ + def setUp(self): + # type: () -> None + self.v0 = Var("v0") + self.v1 = Var("v1") + self.v2 = Var("v2") + self.v3 = Var("v3") + self.v4 = Var("v4") + self.v5 = Var("v5") + self.v6 = Var("v6") + self.v7 = Var("v7") + self.v8 = Var("v8") + self.v9 = Var("v9") + self.imm0 = Var("imm0") + self.IxN_nonscalar = TypeVar("IxN_nonscalar", "", ints=True, + scalars=False, simd=True) + self.TxN = TypeVar("TxN", "", ints=True, bools=True, floats=True, + scalars=False, simd=True) + self.b1 = TypeVar.singleton(b1) + + def test_elaborate_vsplit(self): + # type: () -> None + i32.by(4) # Make sure i32x4 exists. + i32.by(2) # Make sure i32x2 exists. + r = Rtl( + (self.v0, self.v1) << vsplit.i32x4(self.v2), + ) + sem = elaborate(cleanup_concrete_rtl(r)) + bvx = Var('bvx') + bvlo = Var('bvlo') + bvhi = Var('bvhi') + x = Var('x') + lo = Var('lo') + hi = Var('hi') + + assert concrete_rtls_eq(sem, cleanup_concrete_rtl(Rtl( + bvx << prim_to_bv.i32x4(x), + (bvlo, bvhi) << bvsplit.bv128(bvx), + lo << prim_from_bv.i32x2.bv64(bvlo), + hi << prim_from_bv.i32x2.bv64(bvhi)))) + + def test_elaborate_vconcat(self): + # type: () -> None + i32.by(4) # Make sure i32x4 exists. + i32.by(2) # Make sure i32x2 exists. + r = Rtl( + self.v0 << vconcat.i32x2(self.v1, self.v2), + ) + sem = elaborate(cleanup_concrete_rtl(r)) + bvx = Var('bvx') + bvlo = Var('bvlo') + bvhi = Var('bvhi') + x = Var('x') + lo = Var('lo') + hi = Var('hi') + + assert concrete_rtls_eq(sem, cleanup_concrete_rtl(Rtl( + bvlo << prim_to_bv.i32x2(lo), + bvhi << prim_to_bv.i32x2(hi), + bvx << bvconcat.bv64(bvlo, bvhi), + x << prim_from_bv.i32x4.bv128(bvx)))) + + def test_elaborate_iadd_simple(self): + # type: () -> None + i32.by(2) # Make sure i32x2 exists. + r = Rtl( + self.v0 << iadd.i32(self.v1, self.v2), + ) + sem = elaborate(cleanup_concrete_rtl(r)) + x = Var('x') + y = Var('y') + a = Var('a') + bvx = Var('bvx') + bvy = Var('bvy') + bva = Var('bva') + + assert concrete_rtls_eq(sem, cleanup_concrete_rtl(Rtl( + bvx << prim_to_bv.i32(x), + bvy << prim_to_bv.i32(y), + bva << bvadd.bv32(bvx, bvy), + a << prim_from_bv.i32.bv32(bva)))) + + def test_elaborate_iadd_elaborate_1(self): + # type: () -> None + i32.by(2) # Make sure i32x2 exists. + r = Rtl( + self.v0 << iadd.i32x2(self.v1, self.v2), + ) + sem = cleanup_semantics(elaborate(cleanup_concrete_rtl(r)), + set([self.v0])) + x = Var('x') + y = Var('y') + a = Var('a') + bvx_1 = Var('bvx_1') + bvx_2 = Var('bvx_2') + bvx_5 = Var('bvx_5') + bvlo_1 = Var('bvlo_1') + bvlo_2 = Var('bvlo_2') + bvhi_1 = Var('bvhi_1') + bvhi_2 = Var('bvhi_2') + + bva_3 = Var('bva_3') + bva_4 = Var('bva_4') + + assert concrete_rtls_eq(sem, cleanup_concrete_rtl(Rtl( + bvx_1 << prim_to_bv.i32x2(x), + (bvlo_1, bvhi_1) << bvsplit.bv64(bvx_1), + bvx_2 << prim_to_bv.i32x2(y), + (bvlo_2, bvhi_2) << bvsplit.bv64(bvx_2), + bva_3 << bvadd.bv32(bvlo_1, bvlo_2), + bva_4 << bvadd.bv32(bvhi_1, bvhi_2), + bvx_5 << bvconcat.bv32(bva_3, bva_4), + a << prim_from_bv.i32x2.bv64(bvx_5)))) + + def test_elaborate_iadd_elaborate_2(self): + # type: () -> None + i8.by(4) # Make sure i32x2 exists. + r = Rtl( + self.v0 << iadd.i8x4(self.v1, self.v2), + ) + + sem = cleanup_semantics(elaborate(cleanup_concrete_rtl(r)), + set([self.v0])) + x = Var('x') + y = Var('y') + a = Var('a') + bvx_1 = Var('bvx_1') + bvx_2 = Var('bvx_2') + bvx_5 = Var('bvx_5') + bvx_10 = Var('bvx_10') + bvx_15 = Var('bvx_15') + + bvlo_1 = Var('bvlo_1') + bvlo_2 = Var('bvlo_2') + bvlo_6 = Var('bvlo_6') + bvlo_7 = Var('bvlo_7') + bvlo_11 = Var('bvlo_11') + bvlo_12 = Var('bvlo_12') + + bvhi_1 = Var('bvhi_1') + bvhi_2 = Var('bvhi_2') + bvhi_6 = Var('bvhi_6') + bvhi_7 = Var('bvhi_7') + bvhi_11 = Var('bvhi_11') + bvhi_12 = Var('bvhi_12') + + bva_8 = Var('bva_8') + bva_9 = Var('bva_9') + bva_13 = Var('bva_13') + bva_14 = Var('bva_14') + + assert concrete_rtls_eq(sem, cleanup_concrete_rtl(Rtl( + bvx_1 << prim_to_bv.i8x4(x), + (bvlo_1, bvhi_1) << bvsplit.bv32(bvx_1), + bvx_2 << prim_to_bv.i8x4(y), + (bvlo_2, bvhi_2) << bvsplit.bv32(bvx_2), + (bvlo_6, bvhi_6) << bvsplit.bv16(bvlo_1), + (bvlo_7, bvhi_7) << bvsplit.bv16(bvlo_2), + bva_8 << bvadd.bv8(bvlo_6, bvlo_7), + bva_9 << bvadd.bv8(bvhi_6, bvhi_7), + bvx_10 << bvconcat.bv8(bva_8, bva_9), + (bvlo_11, bvhi_11) << bvsplit.bv16(bvhi_1), + (bvlo_12, bvhi_12) << bvsplit.bv16(bvhi_2), + bva_13 << bvadd.bv8(bvlo_11, bvlo_12), + bva_14 << bvadd.bv8(bvhi_11, bvhi_12), + bvx_15 << bvconcat.bv8(bva_13, bva_14), + bvx_5 << bvconcat.bv16(bvx_10, bvx_15), + a << prim_from_bv.i8x4.bv32(bvx_5)))) diff --git a/lib/cretonne/meta/semantics/types.py b/lib/cretonne/meta/semantics/types.py new file mode 100644 index 0000000000..1221804fbd --- /dev/null +++ b/lib/cretonne/meta/semantics/types.py @@ -0,0 +1,9 @@ +""" +The semantics.types module predefines all the Cretone primitive bitvector +types. +""" +from cdsl.types import BVType +from cdsl.typevar import MAX_BITVEC, int_log2 + +for width in range(0, int_log2(MAX_BITVEC)+1): + BVType(2**width) From 74f72a3b4369ce88f4b349b3da06bbb82e2641f8 Mon Sep 17 00:00:00 2001 From: Dimo Date: Fri, 21 Jul 2017 16:46:20 -0700 Subject: [PATCH 0901/3084] Documentation nits; Sematnics syntax cleanup --- lib/cretonne/meta/base/semantics.py | 53 +++++++++-------- lib/cretonne/meta/cdsl/ast.py | 14 +++-- lib/cretonne/meta/cdsl/instructions.py | 25 +++++--- lib/cretonne/meta/cdsl/types.py | 18 ++++-- lib/cretonne/meta/cdsl/typevar.py | 8 ++- lib/cretonne/meta/cdsl/xform.py | 2 +- lib/cretonne/meta/semantics/__init__.py | 59 ++++++++----------- lib/cretonne/meta/semantics/elaborate.py | 1 + lib/cretonne/meta/semantics/test_elaborate.py | 8 +-- lib/cretonne/meta/semantics/types.py | 9 --- 10 files changed, 103 insertions(+), 94 deletions(-) delete mode 100644 lib/cretonne/meta/semantics/types.py diff --git a/lib/cretonne/meta/base/semantics.py b/lib/cretonne/meta/base/semantics.py index d85dad0682..b4315d8255 100644 --- a/lib/cretonne/meta/base/semantics.py +++ b/lib/cretonne/meta/base/semantics.py @@ -2,11 +2,10 @@ from __future__ import absolute_import from semantics.primitives import prim_to_bv, prim_from_bv, bvsplit, bvconcat,\ bvadd from .instructions import vsplit, vconcat, iadd -from cdsl.xform import XForm, Rtl +from cdsl.xform import Rtl from cdsl.ast import Var from cdsl.typevar import TypeSet from cdsl.ti import InTypeset -import semantics.types # noqa x = Var('x') y = Var('y') @@ -28,30 +27,32 @@ bvhi = Var('bvhi') ScalarTS = TypeSet(lanes=(1, 1), ints=True, floats=True, bools=True) vsplit.set_semantics( - XForm(Rtl((lo, hi) << vsplit(x)), - Rtl(bvx << prim_to_bv(x), - (bvlo, bvhi) << bvsplit(bvx), - lo << prim_from_bv(bvlo), - hi << prim_from_bv(bvhi)))) + (lo, hi) << vsplit(x), + Rtl( + bvx << prim_to_bv(x), + (bvlo, bvhi) << bvsplit(bvx), + lo << prim_from_bv(bvlo), + hi << prim_from_bv(bvhi) + )) vconcat.set_semantics( - XForm(Rtl(x << vconcat(lo, hi)), - Rtl(bvlo << prim_to_bv(lo), - bvhi << prim_to_bv(hi), - bvx << bvconcat(bvlo, bvhi), - x << prim_from_bv(bvx)))) + x << vconcat(lo, hi), + Rtl( + bvlo << prim_to_bv(lo), + bvhi << prim_to_bv(hi), + bvx << bvconcat(bvlo, bvhi), + x << prim_from_bv(bvx) + )) -iadd.set_semantics([ - XForm(Rtl(a << iadd(x, y)), - Rtl(bvx << prim_to_bv(x), - bvy << prim_to_bv(y), - bva << bvadd(bvx, bvy), - a << prim_from_bv(bva)), - constraints=[InTypeset(x.get_typevar(), ScalarTS)]), - XForm(Rtl(a << iadd(x, y)), - Rtl((xlo, xhi) << vsplit(x), - (ylo, yhi) << vsplit(y), - alo << iadd(xlo, ylo), - ahi << iadd(xhi, yhi), - a << vconcat(alo, ahi))) -]) +iadd.set_semantics( + a << iadd(x, y), + (Rtl(bvx << prim_to_bv(x), + bvy << prim_to_bv(y), + bva << bvadd(bvx, bvy), + a << prim_from_bv(bva)), + [InTypeset(x.get_typevar(), ScalarTS)]), + Rtl((xlo, xhi) << vsplit(x), + (ylo, yhi) << vsplit(y), + alo << iadd(xlo, ylo), + ahi << iadd(xhi, yhi), + a << vconcat(alo, ahi))) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index c5e6d58def..86e2805497 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -102,15 +102,17 @@ class Def(object): def vars(self): # type: () -> Set[Var] - """ Return the set of all Vars that appear in self""" + """Return the set of all Vars in self that correspond to SSA values""" return self.definitions().union(self.uses()) def substitution(self, other, s): # type: (Def, VarMap) -> Optional[VarMap] """ If the Defs self and other agree structurally, return a variable - substitution to transform self ot other. Two Defs agree structurally - if the contained Apply's agree structurally. + substitution to transform self to other. Otherwise return None. Two + Defs agree structurally if there exists a Var substitution, that can + transform one into the other. See Apply.substitution() for more + details. """ s = self.expr.substitution(other.expr, s) @@ -378,7 +380,7 @@ class Apply(Expr): def vars(self): # type: () -> Set[Var] - """ Return the set of all Vars that appear in self""" + """Return the set of all Vars in self that correspond to SSA values""" res = set() for i in self.inst.value_opnums: arg = self.args[i] @@ -390,8 +392,8 @@ class Apply(Expr): # type: (Apply, VarMap) -> Optional[VarMap] """ If the application self and other agree structurally, return a variable - substitution to transform self ot other. Two applications agree - structurally if: + substitution to transform self to other. Otherwise return None. Two + applications agree structurally if: 1) They are over the same instruction 2) Every Var v in self, maps to a single Var w in other. I.e for each use of v in self, w is used in the corresponding place in diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index ee9bf15e5e..9d23dea89e 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -9,15 +9,16 @@ try: from typing import Union, Sequence, List, Tuple, Any, TYPE_CHECKING # noqa from typing import Dict # noqa if TYPE_CHECKING: - from .ast import Expr, Apply, Var # noqa + from .ast import Expr, Apply, Var, Def # noqa from .typevar import TypeVar # noqa from .ti import TypeConstraint # noqa - from .xform import XForm + from .xform import XForm, Rtl # List of operands for ins/outs: OpList = Union[Sequence[Operand], Operand] ConstrList = Union[Sequence[TypeConstraint], TypeConstraint] MaybeBoundInst = Union['Instruction', 'BoundInstruction'] - InstructionSemantics = List[XForm] + InstructionSemantics = Sequence[XForm] + RtlCase = Union[Rtl, Tuple[Rtl, Sequence[TypeConstraint]]] except ImportError: pass @@ -336,15 +337,23 @@ class Instruction(object): from .ast import Apply # noqa return Apply(self, args) - def set_semantics(self, sem): - # type: (Union[XForm, InstructionSemantics]) -> None + def set_semantics(self, src, *dsts): + # type: (Union[Def, Apply], *RtlCase) -> None """Set our semantics.""" from semantics import verify_semantics + from .xform import XForm, Rtl - if not isinstance(sem, list): - sem = [sem] + sem = [] # type: List[XForm] + for dst in dsts: + if isinstance(dst, Rtl): + sem.append(XForm(Rtl(src).copy({}), dst)) + else: + assert isinstance(dst, tuple) + sem.append(XForm(Rtl(src).copy({}), dst[0], + constraints=dst[1])) + + verify_semantics(self, Rtl(src), sem) - verify_semantics(self, sem) self.semantics = sem diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py index 846cc442c9..53a1dd79ad 100644 --- a/lib/cretonne/meta/cdsl/types.py +++ b/lib/cretonne/meta/cdsl/types.py @@ -89,10 +89,7 @@ class ScalarType(ValueType): self._vectors = dict() # type: Dict[int, VectorType] # Assign numbers starting from 1. (0 is VOID). ValueType.all_scalars.append(self) - # Numbers are only valid for Cretone types that get emitted to Rust. - # This excludes BVTypes - self.number = len([x for x in ValueType.all_scalars - if not isinstance(x, BVType)]) + self.number = len(ValueType.all_scalars) assert self.number < 16, 'Too many scalar types' def __repr__(self): @@ -249,7 +246,7 @@ class BoolType(ScalarType): return self.bits -class BVType(ScalarType): +class BVType(ValueType): """A flat bitvector type. Used for semantics description only.""" def __init__(self, bits): @@ -268,7 +265,11 @@ class BVType(ScalarType): @staticmethod def with_bits(bits): # type: (int) -> BVType - typ = ValueType.by_name('bv{:d}'.format(bits)) + name = 'bv{:d}'.format(bits) + if name not in ValueType._registry: + return BVType(bits) + + typ = ValueType.by_name(name) if TYPE_CHECKING: return cast(BVType, typ) else: @@ -278,3 +279,8 @@ class BVType(ScalarType): # type: () -> int """Return the number of bits in a lane.""" return self.bits + + def lane_count(self): + # type: () -> int + """Return the number of lane. For BVtypes always 1.""" + return 1 diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 2e7d4cb5cc..988d34aaf8 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -501,7 +501,8 @@ class TypeSet(object): for bits in self.bools: yield by(types.BoolType.with_bits(bits), nlanes) for bits in self.bitvecs: - yield by(types.BVType.with_bits(bits), nlanes) + assert nlanes == 1 + yield types.BVType.with_bits(bits) def get_singleton(self): # type: () -> types.ValueType @@ -572,12 +573,17 @@ class TypeVar(object): def singleton(typ): # type: (types.ValueType) -> TypeVar """Create a type variable that can only assume a single type.""" + scalar = None # type: ValueType if isinstance(typ, types.VectorType): scalar = typ.base lanes = (typ.lanes, typ.lanes) elif isinstance(typ, types.ScalarType): scalar = typ lanes = (1, 1) + else: + assert isinstance(typ, types.BVType) + scalar = typ + lanes = (1, 1) ints = None floats = None diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index 42655272b8..3d3f25a3c7 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -55,7 +55,7 @@ class Rtl(object): def vars(self): # type: () -> Set[Var] - """ Return the set of all Vars that appear in self""" + """Return the set of all Vars in self that correspond to SSA values""" return reduce(lambda x, y: x.union(y), [d.vars() for d in self.rtl], set([])) diff --git a/lib/cretonne/meta/semantics/__init__.py b/lib/cretonne/meta/semantics/__init__.py index d2c327c988..8caf792259 100644 --- a/lib/cretonne/meta/semantics/__init__.py +++ b/lib/cretonne/meta/semantics/__init__.py @@ -4,52 +4,45 @@ from cdsl.ti import TypeEnv, ti_rtl, get_type_env try: from typing import List, Dict, Tuple # noqa from cdsl.ast import Var # noqa - from cdsl.xform import XForm # noqa + from cdsl.xform import XForm, Rtl # noqa from cdsl.ti import VarTyping # noqa from cdsl.instructions import Instruction, InstructionSemantics # noqa except ImportError: pass -def verify_semantics(inst, sem): - # type: (Instruction, InstructionSemantics) -> None +def verify_semantics(inst, src, xforms): + # type: (Instruction, Rtl, InstructionSemantics) -> None """ - Verify that the semantics sem correctly describes the instruction inst. - This involves checking that: - 1) For all XForms x \in sem, x.src consists of a single instance of - inst - 2) For any possible concrete typing of inst there is exactly 1 XForm x - in sem that applies. + Verify that the semantics transforms in xforms correctly describe the + instruction described by the src Rtl. This involves checking that: + 1) For all XForms x \in xforms, there is a Var substitution form src to + x.src + 2) For any possible concrete typing of src there is exactly 1 XForm x + in xforms that applies. """ - # 1) The source rtl is always a single instance of inst. - for xform in sem: - assert len(xform.src.rtl) == 1 and\ - xform.src.rtl[0].expr.inst == inst,\ - "XForm {} doesn't describe instruction {}."\ - .format(xform, inst) + # 0) The source rtl is always a single instruction + assert len(src.rtl) == 1 + + # 1) For all XForms x, x.src is structurally equivalent to src + for x in xforms: + assert src.substitution(x.src, {}) is not None,\ + "XForm {} doesn't describe instruction {}.".format(x, src) # 2) Any possible typing for the instruction should be covered by # exactly ONE semantic XForm - inst_rtl = sem[0].src - typenv = get_type_env(ti_rtl(inst_rtl, TypeEnv())) - - # This bit is awkward. Concrete typing is defined in terms of the vars - # of one Rtl. We arbitrarily picked that Rtl to be sem[0].src. For any - # other XForms in sem, we must build a substitution form - # sem[0].src->sem[N].src, before we can check if sem[N] permits one of - # the concrete typings of our Rtl. - # TODO: Can this be made cleaner? - subst = [inst_rtl.substitution(x.src, {}) for x in sem] - assert not any(x is None for x in subst) - sub_sem = list(zip(subst, sem)) # type: List[Tuple[Dict[Var, Var], XForm]] # noqa - - def subst_typing(typing, sub): - # type: (VarTyping, Dict[Var, Var]) -> VarTyping - return {sub[v]: tv for (v, tv) in typing.items()} + typenv = get_type_env(ti_rtl(src, TypeEnv())) + typenv.normalize() + typenv = typenv.extract() for t in typenv.concrete_typings(): - matching_xforms = [x for (s, x) in sub_sem - if x.ti.permits(subst_typing(t, s))] + matching_xforms = [] # type: List[XForm] + for x in xforms: + # Translate t using x.symtab + t = {x.symtab[str(v)]: tv for (v, tv) in t.items()} + if (x.ti.permits(t)): + matching_xforms.append(x) + assert len(matching_xforms) == 1,\ ("Possible typing {} of {} not matched by exactly one case " + ": {}").format(t, inst, matching_xforms) diff --git a/lib/cretonne/meta/semantics/elaborate.py b/lib/cretonne/meta/semantics/elaborate.py index e6be0180e1..9cd92c1f06 100644 --- a/lib/cretonne/meta/semantics/elaborate.py +++ b/lib/cretonne/meta/semantics/elaborate.py @@ -81,6 +81,7 @@ def find_matching_xform(d): for x in d.expr.inst.semantics: subst = d.substitution(x.src.rtl[0], {}) + assert subst is not None if x.ti.permits({subst[v]: tv for (v, tv) in typing.items()}): res.append(x) diff --git a/lib/cretonne/meta/semantics/test_elaborate.py b/lib/cretonne/meta/semantics/test_elaborate.py index 15d4178ae8..4cbfe07bf9 100644 --- a/lib/cretonne/meta/semantics/test_elaborate.py +++ b/lib/cretonne/meta/semantics/test_elaborate.py @@ -232,16 +232,16 @@ class TestElaborate(TestCase): def test_elaborate_iadd_simple(self): # type: () -> None i32.by(2) # Make sure i32x2 exists. - r = Rtl( - self.v0 << iadd.i32(self.v1, self.v2), - ) - sem = elaborate(cleanup_concrete_rtl(r)) x = Var('x') y = Var('y') a = Var('a') bvx = Var('bvx') bvy = Var('bvy') bva = Var('bva') + r = Rtl( + a << iadd.i32(x, y), + ) + sem = elaborate(cleanup_concrete_rtl(r)) assert concrete_rtls_eq(sem, cleanup_concrete_rtl(Rtl( bvx << prim_to_bv.i32(x), diff --git a/lib/cretonne/meta/semantics/types.py b/lib/cretonne/meta/semantics/types.py deleted file mode 100644 index 1221804fbd..0000000000 --- a/lib/cretonne/meta/semantics/types.py +++ /dev/null @@ -1,9 +0,0 @@ -""" -The semantics.types module predefines all the Cretone primitive bitvector -types. -""" -from cdsl.types import BVType -from cdsl.typevar import MAX_BITVEC, int_log2 - -for width in range(0, int_log2(MAX_BITVEC)+1): - BVType(2**width) From dfb5a524b9e0bf9991e921e53a9f441458caa801 Mon Sep 17 00:00:00 2001 From: Dimo Date: Fri, 21 Jul 2017 16:56:15 -0700 Subject: [PATCH 0902/3084] TI failure due to misplaced import --- lib/cretonne/meta/semantics/elaborate.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/semantics/elaborate.py b/lib/cretonne/meta/semantics/elaborate.py index 9cd92c1f06..0350c5c7dd 100644 --- a/lib/cretonne/meta/semantics/elaborate.py +++ b/lib/cretonne/meta/semantics/elaborate.py @@ -6,10 +6,11 @@ primitive cretonne instructions, which map well to SMTLIB functions. from .primitives import GROUP as PRIMITIVES, prim_to_bv, prim_from_bv from cdsl.ti import ti_rtl, TypeEnv, get_type_env from cdsl.typevar import TypeVar +from cdsl.xform import Rtl try: from typing import TYPE_CHECKING, Dict, Union, List, Set, Tuple # noqa - from cdsl.xform import Rtl, XForm # noqa + from cdsl.xform import XForm # noqa from cdsl.ast import Var, Def, VarMap # noqa from cdsl.ti import VarTyping # noqa except ImportError: From 7caaf7fea1a1302cb738db5e595d7fe272ebb275 Mon Sep 17 00:00:00 2001 From: Dimo Date: Mon, 24 Jul 2017 10:58:57 -0700 Subject: [PATCH 0903/3084] Fix CI: Var was only imported when mypy was present. --- lib/cretonne/meta/semantics/elaborate.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/semantics/elaborate.py b/lib/cretonne/meta/semantics/elaborate.py index 0350c5c7dd..03ca3195c4 100644 --- a/lib/cretonne/meta/semantics/elaborate.py +++ b/lib/cretonne/meta/semantics/elaborate.py @@ -7,11 +7,12 @@ from .primitives import GROUP as PRIMITIVES, prim_to_bv, prim_from_bv from cdsl.ti import ti_rtl, TypeEnv, get_type_env from cdsl.typevar import TypeVar from cdsl.xform import Rtl +from cdsl.ast import Var try: from typing import TYPE_CHECKING, Dict, Union, List, Set, Tuple # noqa from cdsl.xform import XForm # noqa - from cdsl.ast import Var, Def, VarMap # noqa + from cdsl.ast import Def, VarMap # noqa from cdsl.ti import VarTyping # noqa except ImportError: TYPE_CHECKING = False From b448574a49d6a1bc56fa8f9b5397720cf2d4392e Mon Sep 17 00:00:00 2001 From: Dimo Date: Mon, 24 Jul 2017 13:41:04 -0700 Subject: [PATCH 0904/3084] Assert all InstructionGroups are closed in TargetIsa.__init__(); Close x86 group --- lib/cretonne/meta/cdsl/isa.py | 5 +++++ lib/cretonne/meta/isa/intel/instructions.py | 2 ++ 2 files changed, 7 insertions(+) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index ea10c64640..29a3698304 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -5,6 +5,7 @@ from .predicates import And from .registers import RegClass, Register, Stack from .ast import Apply from .types import ValueType +from .instructions import InstructionGroup # The typing module is only required by mypy, and we don't use these imports # outside type comments. @@ -47,6 +48,10 @@ class TargetISA(object): self.regclasses = list() # type: List[RegClass] self.legalize_codes = OrderedDict() # type: OrderedDict[XFormGroup, int] # noqa + assert InstructionGroup._current is None,\ + "InstructionGroup {} is still open!"\ + .format(InstructionGroup._current.name) + def __str__(self): # type: () -> str return self.name diff --git a/lib/cretonne/meta/isa/intel/instructions.py b/lib/cretonne/meta/isa/intel/instructions.py index a5c2bdafb0..c9d87c43f3 100644 --- a/lib/cretonne/meta/isa/intel/instructions.py +++ b/lib/cretonne/meta/isa/intel/instructions.py @@ -43,3 +43,5 @@ sdivmodx = Instruction( Return both quotient and remainder. """, ins=(nlo, nhi, d), outs=(q, r), can_trap=True) + +GROUP.close() From f39d75fa58ac0c193f641a779d66f1a23a2a0fcf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Jul 2017 15:33:35 -0700 Subject: [PATCH 0905/3084] Generate a RECIPE_PREDICATES table for each ISA. It turns out that most encoding predicates are expressed as recipe predicates. This means that the encoding tables can be more compact since we can check the recipe predicate separately from individual instruction predicates, and the recipe number is already present in the table. - Don't combine recipe and encoding-specific predicates when creating an Encoding. Keep them separate. - Generate a table of recipe predicates with function pointers. Many of these are null. - Check any recipe predicate before accepting a recipe+bits pair. This has the effect of making almost all instruction predicates CODE_ALWAYS. --- lib/cretonne/meta/cdsl/isa.py | 27 ++++++-- lib/cretonne/meta/gen_encoding.py | 85 +++++++++++++++++++----- lib/cretonne/src/isa/arm32/enc_tables.rs | 2 +- lib/cretonne/src/isa/arm32/mod.rs | 1 + lib/cretonne/src/isa/arm64/enc_tables.rs | 2 +- lib/cretonne/src/isa/arm64/mod.rs | 1 + lib/cretonne/src/isa/enc_tables.rs | 33 +++++++-- lib/cretonne/src/isa/intel/enc_tables.rs | 2 +- lib/cretonne/src/isa/intel/mod.rs | 1 + lib/cretonne/src/isa/riscv/enc_tables.rs | 2 +- lib/cretonne/src/isa/riscv/mod.rs | 1 + 11 files changed, 126 insertions(+), 31 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 29a3698304..f65c2ad1fc 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -23,6 +23,9 @@ try: # instructions. InstSpec = Union[MaybeBoundInst, Apply] BranchRange = Sequence[int] + # A recipe predicate consisting of an ISA predicate and an instruction + # predicate. + RecipePred = Tuple[PredNode, PredNode] except ImportError: pass @@ -62,7 +65,7 @@ class TargetISA(object): Finish the definition of a target ISA after adding all CPU modes and settings. - This computes some derived properties that are used in multilple + This computes some derived properties that are used in multiple places. :returns self: @@ -86,6 +89,9 @@ class TargetISA(object): recipe.number = len(rcps) rcps.add(recipe) self.all_recipes.append(recipe) + # Make sure ISA predicates are registered. + if recipe.isap: + self.settings.number_predicate(recipe.isap) def _collect_predicates(self): # type: () -> None @@ -336,6 +342,19 @@ class EncRecipe(object): o2i[o] = i return (i2o, o2i) + def recipe_pred(self): + # type: () -> RecipePred + """ + Get the combined recipe predicate which includes both the ISA predicate + and the instruction predicate. + + Return `None` if this recipe has neither predicate. + """ + if self.isap is None and self.instp is None: + return None + else: + return (self.isap, self.instp) + class Encoding(object): """ @@ -386,9 +405,9 @@ class Encoding(object): self.recipe = recipe self.encbits = encbits - # Combine recipe predicates with the manually specified ones. - self.instp = And.combine(recipe.instp, instp) - self.isap = And.combine(recipe.isap, isap) + # Record specific predicates. Note that the recipe also has predicates. + self.instp = instp + self.isap = isap def __str__(self): # type: () -> str diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index de8b138cf6..5f49cb42c6 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -11,11 +11,12 @@ are allocated. This is the information available to us: -- The instruction to be encoded as an `Inst` reference. -- The data-flow graph containing the instruction, giving us access to the - `InstructionData` representation and the types of all values involved. -- A target ISA instance with shared and ISA-specific settings for evaluating - ISA predicates. +- The instruction to be encoded as an `InstructionData` reference. +- The controlling type variable. +- The data-flow graph giving us access to the types of all values involved. + This is needed for testing any secondary type variables. +- A `PredicateView` reference for the ISA-specific settings for evaluating ISA + predicates. - The currently active CPU mode is determined by the ISA. ## Level 1 table lookup @@ -62,7 +63,7 @@ from cdsl.predicates import FieldPredicate try: from typing import Sequence, Set, Tuple, List, Dict, Iterable, DefaultDict, TYPE_CHECKING # noqa if TYPE_CHECKING: - from cdsl.isa import TargetISA, OperandConstraint, Encoding, CPUMode, EncRecipe # noqa + from cdsl.isa import TargetISA, OperandConstraint, Encoding, CPUMode, EncRecipe, RecipePred # noqa from cdsl.predicates import PredNode, PredLeaf # noqa from cdsl.types import ValueType # noqa from cdsl.instructions import Instruction # noqa @@ -77,8 +78,8 @@ def emit_instp(instp, fmt): Emit code for matching an instruction predicate against an `InstructionData` reference called `inst`. - The generated code is a pattern match that falls through if the instruction - has an unexpected format. This should lead to a panic. + The generated code is an `if let` pattern match that falls through if the + instruction has an unexpected format. This should lead to a panic. """ iform = instp.predicate_context() @@ -94,11 +95,10 @@ def emit_instp(instp, fmt): fnames.add(p.field.rust_name()) fields = ', '.join(sorted(fnames)) - with fmt.indented('{} => {{'.format(instp.number), '}'): - with fmt.indented( - 'if let InstructionData::{} {{ {}, .. }} = *inst {{' - .format(iform.name, fields), '}'): - fmt.line('return {};'.format(instp.rust_predicate(0))) + with fmt.indented( + 'if let InstructionData::{} {{ {}, .. }} = *inst {{' + .format(iform.name, fields), '}'): + fmt.line('return {};'.format(instp.rust_predicate(0))) def emit_instps(instps, fmt): @@ -122,7 +122,8 @@ def emit_instps(instps, fmt): fmt.line('use ir::instructions::InstructionFormat;') with fmt.indented('match instp_idx {', '}'): for instp in instps: - emit_instp(instp, fmt) + with fmt.indented('{} => {{'.format(instp.number), '}'): + emit_instp(instp, fmt) fmt.line('_ => panic!("Invalid instruction predicate")') # The match cases will fall through if the instruction format is wrong. @@ -132,6 +133,55 @@ def emit_instps(instps, fmt): fmt.line(' instp_idx);') +def emit_recipe_predicates(recipes, fmt): + # type: (Sequence[EncRecipe], srcgen.Formatter) -> None + """ + Emit private functions for checking recipe predicates as well as a static + `RECIPE_PREDICATES` array indexed by recipe number. + + A recipe predicate is a combination of an ISA predicate and an instruction + predicates. Many recipes have identical predicates. + """ + # Table for uniquing recipe predicates. Maps predicate to generated + # function name. + pname = dict() # type: Dict[RecipePred, str] + + # Generate unique recipe predicates. + for rcp in recipes: + p = rcp.recipe_pred() + if p is None or p in pname: + continue + name = 'recipe_predicate_{}'.format(rcp.name.lower()) + pname[p] = name + isap, instp = p + + # Generate the predicate function. + with fmt.indented( + 'fn {}({}: ::settings::PredicateView, ' + 'inst: &InstructionData) -> bool {{' + .format( + name, + 'isap' if isap else '_'), '}'): + if isap: + with fmt.indented( + 'if isap.test({})'.format(isap.number), + '}'): + fmt.line('return false;') + emit_instp(instp, fmt) + fmt.line('unreachable!();') + + # Generate the static table. + with fmt.indented( + 'pub static RECIPE_PREDICATES: [RecipePredicate; {}] = [' + .format(len(recipes)), '];'): + for rcp in recipes: + p = rcp.recipe_pred() + if p is None: + fmt.line('None,') + else: + fmt.format('Some({}),', pname[p]) + + # Encoding lists are represented as u16 arrays. CODE_BITS = 16 PRED_BITS = 12 @@ -604,8 +654,11 @@ def emit_recipe_sizing(isa, fmt): def gen_isa(isa, fmt): # type: (TargetISA, srcgen.Formatter) -> None - # First assign numbers to relevant instruction predicates and generate the - # check_instp() function.. + + # Make the `RECIPE_PREDICATES` table. + emit_recipe_predicates(isa.all_recipes, fmt) + + # Generate the check_instp() function.. emit_instps(isa.all_instps, fmt) # Level1 tables, one per CPU mode diff --git a/lib/cretonne/src/isa/arm32/enc_tables.rs b/lib/cretonne/src/isa/arm32/enc_tables.rs index 9fdfbcda95..e33d47f12d 100644 --- a/lib/cretonne/src/isa/arm32/enc_tables.rs +++ b/lib/cretonne/src/isa/arm32/enc_tables.rs @@ -4,7 +4,7 @@ use ir::InstructionData; use ir::types; use isa::EncInfo; use isa::constraints::*; -use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::enc_tables::*; use isa::encoding::RecipeSizing; include!(concat!(env!("OUT_DIR"), "/encoding-arm32.rs")); diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 5675d19d7a..06852e4ad2 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -73,6 +73,7 @@ impl TargetIsa for Isa { .and_then(|enclist_offset| { Ok(Encodings::new(enclist_offset, &enc_tables::ENCLISTS[..], + &enc_tables::RECIPE_PREDICATES[..], inst, enc_tables::check_instp, self.isa_flags.predicate_view())) diff --git a/lib/cretonne/src/isa/arm64/enc_tables.rs b/lib/cretonne/src/isa/arm64/enc_tables.rs index cdfc255748..889a5a7dbe 100644 --- a/lib/cretonne/src/isa/arm64/enc_tables.rs +++ b/lib/cretonne/src/isa/arm64/enc_tables.rs @@ -4,7 +4,7 @@ use ir::InstructionData; use ir::types; use isa::EncInfo; use isa::constraints::*; -use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::enc_tables::*; use isa::encoding::RecipeSizing; include!(concat!(env!("OUT_DIR"), "/encoding-arm64.rs")); diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index e9aea5aacf..26e4ea9dfe 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -66,6 +66,7 @@ impl TargetIsa for Isa { .and_then(|enclist_offset| { Ok(Encodings::new(enclist_offset, &enc_tables::ENCLISTS[..], + &enc_tables::RECIPE_PREDICATES[..], inst, enc_tables::check_instp, self.isa_flags.predicate_view())) diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index 33ec38d3c1..843cbf6a1d 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -9,6 +9,13 @@ use isa::{Encoding, Legalize}; use settings::PredicateView; use std::ops::Range; +/// A recipe predicate. +/// +/// This is a predicate function capable of testing ISA and instruction predicates simultaneously. +/// +/// A None predicate is always satisfied. +pub type RecipePredicate = Option bool>; + /// Legalization action to perform when no encoding can be found for an instruction. /// /// This is an index into an ISA-specific table of legalization actions. @@ -147,6 +154,7 @@ pub struct Encodings<'a> { inst: &'a InstructionData, instp: fn(&InstructionData, EncListEntry) -> bool, isa_predicates: PredicateView<'a>, + recipe_predicates: &'static [RecipePredicate], } impl<'a> Encodings<'a> { @@ -155,8 +163,9 @@ impl<'a> Encodings<'a> { /// # Parameters /// /// - `offset` an offset into encoding list returned by `lookup_enclist` function. - /// - `inst` the current instruction. /// - `enclist` a list of encoding entries. + /// - `recipe_predicates` is a slice of recipe predicate functions. + /// - `inst` the current instruction. /// - `instp` an instruction predicate number to be evaluated on the current instruction. /// - `isa_predicate_bytes` an ISA flags as a slice of bytes to evaluate an ISA predicate number /// on the current instruction. @@ -166,6 +175,7 @@ impl<'a> Encodings<'a> { /// or `None`. pub fn new(offset: usize, enclist: &'static [EncListEntry], + recipe_predicates: &'static [RecipePredicate], inst: &'a InstructionData, instp: fn(&InstructionData, EncListEntry) -> bool, isa_predicates: PredicateView<'a>) @@ -176,6 +186,15 @@ impl<'a> Encodings<'a> { inst, instp, isa_predicates, + recipe_predicates, + } + } + + /// Check if the predicate for `recipe` is satisfied. + fn check_recipe(&self, recipe: u16) -> bool { + match self.recipe_predicates[recipe as usize] { + Some(p) => p(self.isa_predicates, self.inst), + None => true, } } } @@ -188,13 +207,13 @@ impl<'a> Iterator for Encodings<'a> { let pred = self.enclist[self.offset]; if pred <= CODE_ALWAYS { // This is an instruction predicate followed by recipe and encbits entries. + self.offset += 3; if pred == CODE_ALWAYS || (self.instp)(self.inst, pred) { - let encoding = Encoding::new(self.enclist[self.offset + 1], - self.enclist[self.offset + 2]); - self.offset += 3; - return Some(encoding); - } else { - self.offset += 3; + let recipe = self.enclist[self.offset - 2]; + if self.check_recipe(recipe) { + let encoding = Encoding::new(recipe, self.enclist[self.offset - 1]); + return Some(encoding); + } } } else { // This is an ISA predicate entry. diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index e72da36b1e..47ea5dbe11 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -4,7 +4,7 @@ use ir::types; use ir::{Opcode, InstructionData}; use isa::EncInfo; use isa::constraints::*; -use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::enc_tables::*; use isa::encoding::RecipeSizing; use predicates; use super::registers::*; diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 35d339a740..c459b49bd7 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -73,6 +73,7 @@ impl TargetIsa for Isa { .and_then(|enclist_offset| { Ok(Encodings::new(enclist_offset, &enc_tables::ENCLISTS[..], + &enc_tables::RECIPE_PREDICATES[..], inst, enc_tables::check_instp, self.isa_flags.predicate_view())) diff --git a/lib/cretonne/src/isa/riscv/enc_tables.rs b/lib/cretonne/src/isa/riscv/enc_tables.rs index 8ced0c681f..0dddebdbac 100644 --- a/lib/cretonne/src/isa/riscv/enc_tables.rs +++ b/lib/cretonne/src/isa/riscv/enc_tables.rs @@ -5,7 +5,7 @@ use ir::types; use ir::{Opcode, InstructionData}; use isa::EncInfo; use isa::constraints::*; -use isa::enc_tables::{Level1Entry, Level2Entry}; +use isa::enc_tables::*; use isa::encoding::RecipeSizing; use predicates; use super::registers::*; diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 869ea4c9cb..d09bfbd028 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -73,6 +73,7 @@ impl TargetIsa for Isa { .and_then(|enclist_offset| { Ok(Encodings::new(enclist_offset, &enc_tables::ENCLISTS[..], + &enc_tables::RECIPE_PREDICATES[..], inst, enc_tables::check_instp, self.isa_flags.predicate_view())) From 776af93ef2f47125ca5fce070e2715a102ca4207 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 24 Jul 2017 14:13:35 -0700 Subject: [PATCH 0906/3084] Generate an INST_PREDICATES table for each ISA. Instead of generating a single `check_instp()` function, create an array of individual function pointers for checking instruction predicates. This makes explicit the jump table in the old check_instp() method and it gives us a way of determining the number of instruction predicates that exists. --- lib/cretonne/meta/gen_encoding.py | 43 ++++++++++-------------- lib/cretonne/src/isa/arm32/enc_tables.rs | 1 - lib/cretonne/src/isa/arm32/mod.rs | 2 +- lib/cretonne/src/isa/arm64/enc_tables.rs | 1 - lib/cretonne/src/isa/arm64/mod.rs | 2 +- lib/cretonne/src/isa/enc_tables.rs | 18 +++++++--- lib/cretonne/src/isa/intel/mod.rs | 2 +- lib/cretonne/src/isa/riscv/mod.rs | 2 +- 8 files changed, 35 insertions(+), 36 deletions(-) diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 5f49cb42c6..43f02d6416 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -101,36 +101,27 @@ def emit_instp(instp, fmt): fmt.line('return {};'.format(instp.rust_predicate(0))) -def emit_instps(instps, fmt): +def emit_inst_predicates(instps, fmt): # type: (Sequence[PredNode], srcgen.Formatter) -> None """ - Emit a function for matching instruction predicates. + Emit private functions for matching instruction predicates as well as a + static `INST_PREDICATES` array indexed by predicate number. """ - - if not instps: - # If the ISA has no predicates, just emit a stub. + for instp in instps: + name = 'inst_predicate_{}'.format(instp.number) with fmt.indented( - 'pub fn check_instp(_: &InstructionData, _: u16) ' + - '-> bool {', '}'): - fmt.line('unimplemented!()') - return + 'fn {}(inst: &InstructionData) -> bool {{' + .format(name), + '}'): + emit_instp(instp, fmt) + fmt.line('unreachable!();') + # Generate the static table. with fmt.indented( - 'pub fn check_instp(inst: &InstructionData, instp_idx: u16) ' + - '-> bool {', '}'): - # The matches emitted by `emit_instp` need this. - fmt.line('use ir::instructions::InstructionFormat;') - with fmt.indented('match instp_idx {', '}'): - for instp in instps: - with fmt.indented('{} => {{'.format(instp.number), '}'): - emit_instp(instp, fmt) - fmt.line('_ => panic!("Invalid instruction predicate")') - - # The match cases will fall through if the instruction format is wrong. - fmt.line('panic!("Bad format {:?}/{} for instp {}",') - fmt.line(' InstructionFormat::from(inst),') - fmt.line(' inst.opcode(),') - fmt.line(' instp_idx);') + 'pub static INST_PREDICATES: [InstPredicate; {}] = [' + .format(len(instps)), '];'): + for instp in instps: + fmt.format('inst_predicate_{},', instp.number) def emit_recipe_predicates(recipes, fmt): @@ -658,8 +649,8 @@ def gen_isa(isa, fmt): # Make the `RECIPE_PREDICATES` table. emit_recipe_predicates(isa.all_recipes, fmt) - # Generate the check_instp() function.. - emit_instps(isa.all_instps, fmt) + # Make the `INST_PREDICATES` table. + emit_inst_predicates(isa.all_instps, fmt) # Level1 tables, one per CPU mode level1_tables = dict() diff --git a/lib/cretonne/src/isa/arm32/enc_tables.rs b/lib/cretonne/src/isa/arm32/enc_tables.rs index e33d47f12d..adcc2fd915 100644 --- a/lib/cretonne/src/isa/arm32/enc_tables.rs +++ b/lib/cretonne/src/isa/arm32/enc_tables.rs @@ -1,6 +1,5 @@ //! Encoding tables for ARM32 ISA. -use ir::InstructionData; use ir::types; use isa::EncInfo; use isa::constraints::*; diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 06852e4ad2..1e5cde38db 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -74,8 +74,8 @@ impl TargetIsa for Isa { Ok(Encodings::new(enclist_offset, &enc_tables::ENCLISTS[..], &enc_tables::RECIPE_PREDICATES[..], + &enc_tables::INST_PREDICATES[..], inst, - enc_tables::check_instp, self.isa_flags.predicate_view())) }) } diff --git a/lib/cretonne/src/isa/arm64/enc_tables.rs b/lib/cretonne/src/isa/arm64/enc_tables.rs index 889a5a7dbe..e57f3cc98c 100644 --- a/lib/cretonne/src/isa/arm64/enc_tables.rs +++ b/lib/cretonne/src/isa/arm64/enc_tables.rs @@ -1,6 +1,5 @@ //! Encoding tables for ARM64 ISA. -use ir::InstructionData; use ir::types; use isa::EncInfo; use isa::constraints::*; diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 26e4ea9dfe..4dd873e17b 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -67,8 +67,8 @@ impl TargetIsa for Isa { Ok(Encodings::new(enclist_offset, &enc_tables::ENCLISTS[..], &enc_tables::RECIPE_PREDICATES[..], + &enc_tables::INST_PREDICATES[..], inst, - enc_tables::check_instp, self.isa_flags.predicate_view())) }) } diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index 843cbf6a1d..d040dd9ca4 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -16,6 +16,12 @@ use std::ops::Range; /// A None predicate is always satisfied. pub type RecipePredicate = Option bool>; +/// An instruction predicate. +/// +/// This is a predicate function that needs to be tested in addition to the recipe predicate. It +/// can't depend on ISA settings. +pub type InstPredicate = fn(&InstructionData) -> bool; + /// Legalization action to perform when no encoding can be found for an instruction. /// /// This is an index into an ISA-specific table of legalization actions. @@ -152,9 +158,9 @@ pub struct Encodings<'a> { offset: usize, enclist: &'static [EncListEntry], inst: &'a InstructionData, - instp: fn(&InstructionData, EncListEntry) -> bool, isa_predicates: PredicateView<'a>, recipe_predicates: &'static [RecipePredicate], + inst_predicates: &'static [InstPredicate], } impl<'a> Encodings<'a> { @@ -176,17 +182,17 @@ impl<'a> Encodings<'a> { pub fn new(offset: usize, enclist: &'static [EncListEntry], recipe_predicates: &'static [RecipePredicate], + inst_predicates: &'static [InstPredicate], inst: &'a InstructionData, - instp: fn(&InstructionData, EncListEntry) -> bool, isa_predicates: PredicateView<'a>) -> Self { Encodings { offset, enclist, inst, - instp, isa_predicates, recipe_predicates, + inst_predicates, } } @@ -208,7 +214,11 @@ impl<'a> Iterator for Encodings<'a> { if pred <= CODE_ALWAYS { // This is an instruction predicate followed by recipe and encbits entries. self.offset += 3; - if pred == CODE_ALWAYS || (self.instp)(self.inst, pred) { + let satisfied = match self.inst_predicates.get(pred as usize) { + Some(p) => p(self.inst), + None => true, + }; + if satisfied { let recipe = self.enclist[self.offset - 2]; if self.check_recipe(recipe) { let encoding = Encoding::new(recipe, self.enclist[self.offset - 1]); diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index c459b49bd7..85445ac701 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -74,8 +74,8 @@ impl TargetIsa for Isa { Ok(Encodings::new(enclist_offset, &enc_tables::ENCLISTS[..], &enc_tables::RECIPE_PREDICATES[..], + &enc_tables::INST_PREDICATES[..], inst, - enc_tables::check_instp, self.isa_flags.predicate_view())) }) } diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index d09bfbd028..3fe675b7a4 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -74,8 +74,8 @@ impl TargetIsa for Isa { Ok(Encodings::new(enclist_offset, &enc_tables::ENCLISTS[..], &enc_tables::RECIPE_PREDICATES[..], + &enc_tables::INST_PREDICATES[..], inst, - enc_tables::check_instp, self.isa_flags.predicate_view())) }) } From 20d96a1ac441348b73b42a1524b3fb10baddfad2 Mon Sep 17 00:00:00 2001 From: Dimo Date: Mon, 24 Jul 2017 15:15:51 -0700 Subject: [PATCH 0907/3084] Handle non-ssa Vars and Enumerator constants in Rtl substitutions --- lib/cretonne/meta/cdsl/ast.py | 30 ++++++++++++++++------ lib/cretonne/meta/cdsl/test_xform.py | 37 +++++++++++++++++++++++++++- 2 files changed, 58 insertions(+), 9 deletions(-) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index 86e2805497..e954ab01e9 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -402,16 +402,30 @@ class Apply(Expr): if self.inst != other.inst: return None - # TODO: Should we check imm/cond codes here as well? - for i in self.inst.value_opnums: - self_a = self.args[i] - other_a = other.args[i] + # Guaranteed by self.inst == other.inst + assert (len(self.args) == len(other.args)) - assert isinstance(self_a, Var) and isinstance(other_a, Var) - if (self_a not in s): - s[self_a] = other_a + for (self_a, other_a) in zip(self.args, other.args): + if (isinstance(self_a, Var)): + if not isinstance(other_a, Var): + return None + + if (self_a not in s): + s[self_a] = other_a + else: + if (s[self_a] != other_a): + return None else: - if (s[self_a] != other_a): + assert isinstance(self_a, Enumerator) + + if not isinstance(other_a, Enumerator): + # Currently don't support substitutions Var->Enumerator + return None + + # Guaranteed by self.inst == other.inst + assert self_a.kind == other_a.kind + + if (self_a.value != other_a.value): return None return s diff --git a/lib/cretonne/meta/cdsl/test_xform.py b/lib/cretonne/meta/cdsl/test_xform.py index 1609bb6c5f..952d8c90cb 100644 --- a/lib/cretonne/meta/cdsl/test_xform.py +++ b/lib/cretonne/meta/cdsl/test_xform.py @@ -1,7 +1,8 @@ from __future__ import absolute_import from unittest import TestCase from doctest import DocTestSuite -from base.instructions import iadd, iadd_imm, iconst +from base.instructions import iadd, iadd_imm, iconst, icmp +from base.immediates import intcc from . import xform from .ast import Var from .xform import Rtl, XForm @@ -14,9 +15,15 @@ def load_tests(loader, tests, ignore): x = Var('x') y = Var('y') +z = Var('z') +u = Var('u') a = Var('a') +b = Var('b') c = Var('c') +CC1 = Var('CC1') +CC2 = Var('CC2') + class TestXForm(TestCase): def test_macro_pattern(self): @@ -57,3 +64,31 @@ class TestXForm(TestCase): dst = Rtl(a << iadd(x, y)) with self.assertRaisesRegexp(AssertionError, "'a' multiply defined"): XForm(src, dst) + + def test_subst_imm(self): + src = Rtl(a << iconst(x)) + dst = Rtl(c << iconst(y)) + assert src.substitution(dst, {}) == {a: c, x: y} + + def test_subst_enum_var(self): + src = Rtl(a << icmp(CC1, x, y)) + dst = Rtl(b << icmp(CC2, z, u)) + assert src.substitution(dst, {}) == {a: b, CC1: CC2, x: z, y: u} + + def test_subst_enum_const(self): + src = Rtl(a << icmp(intcc.eq, x, y)) + dst = Rtl(b << icmp(intcc.eq, z, u)) + assert src.substitution(dst, {}) == {a: b, x: z, y: u} + + def test_subst_enum_bad(self): + src = Rtl(a << icmp(CC1, x, y)) + dst = Rtl(b << icmp(intcc.eq, z, u)) + assert src.substitution(dst, {}) is None + + src = Rtl(a << icmp(intcc.eq, x, y)) + dst = Rtl(b << icmp(CC1, z, u)) + assert src.substitution(dst, {}) is None + + src = Rtl(a << icmp(intcc.eq, x, y)) + dst = Rtl(b << icmp(intcc.sge, z, u)) + assert src.substitution(dst, {}) is None From e41ddf2a0d66d221c710d93422fcf2edbd434c1e Mon Sep 17 00:00:00 2001 From: Dimo Date: Tue, 25 Jul 2017 15:09:22 -0700 Subject: [PATCH 0908/3084] Change TV ranking to select src vars as a representative during unification; Nit: cleanup dot() emitting code; Nit: fix small bug in verify_semantics() - make an internal copy of src rtl to avoid clobbering of typevars re-used in multiple definitions --- lib/cretonne/meta/cdsl/test_ti.py | 13 ++--- lib/cretonne/meta/cdsl/ti.py | 57 ++++++++++++------- lib/cretonne/meta/semantics/__init__.py | 1 + lib/cretonne/meta/semantics/primitives.py | 4 +- lib/cretonne/meta/semantics/test_elaborate.py | 12 ++-- lib/cretonne/meta/test_gen_legalizer.py | 5 +- 6 files changed, 51 insertions(+), 41 deletions(-) diff --git a/lib/cretonne/meta/cdsl/test_ti.py b/lib/cretonne/meta/cdsl/test_ti.py index 9351d9c3ea..31d9a349a8 100644 --- a/lib/cretonne/meta/cdsl/test_ti.py +++ b/lib/cretonne/meta/cdsl/test_ti.py @@ -158,8 +158,7 @@ class TypeCheckingBaseTest(TestCase): self.v8 = Var("v8") self.v9 = Var("v9") self.imm0 = Var("imm0") - self.IxN_nonscalar = TypeVar("IxN_nonscalar", "", ints=True, - scalars=False, simd=True) + self.IxN = TypeVar("IxN", "", ints=True, scalars=True, simd=True) self.TxN = TypeVar("TxN", "", ints=True, bools=True, floats=True, scalars=False, simd=True) self.b1 = TypeVar.singleton(b1) @@ -176,7 +175,7 @@ class TestRTL(TypeCheckingBaseTest): self.assertEqual(ti_rtl(r, ti), "On line 1: fail ti on `typeof_v2` <: `1`: " + "Error: empty type created when unifying " + - "`typeof_v2` and `half_vector(typeof_v2)`") + "`typeof_v3` and `half_vector(typeof_v3)`") def test_vselect(self): # type: () -> None @@ -202,11 +201,11 @@ class TestRTL(TypeCheckingBaseTest): ) ti = TypeEnv() typing = ti_rtl(r, ti) - ixn = self.IxN_nonscalar.get_fresh_copy("IxN1") + ixn = self.IxN.get_fresh_copy("IxN1") txn = self.TxN.get_fresh_copy("TxN1") check_typing(typing, ({ self.v0: ixn, - self.v1: ixn.as_bool(), + self.v1: txn.as_bool(), self.v2: ixn, self.v3: txn, self.v4: txn, @@ -319,7 +318,7 @@ class TestRTL(TypeCheckingBaseTest): self.assertEqual(typing, "On line 2: fail ti on `typeof_v4` <: `4`: " + "Error: empty type created when unifying " + - "`typeof_v4` and `typeof_v5`") + "`i16` and `i32`") def test_extend_reduce(self): # type: () -> None @@ -471,7 +470,7 @@ class TestXForm(TypeCheckingBaseTest): assert var_m[v0] == var_m[v2] and \ var_m[v3] == var_m[v4] and\ var_m[v5] == var_m[v3] and\ - var_m[v1] == var_m[v2].as_bool() and\ + var_m[v1] == var_m[v5].as_bool() and\ var_m[v1].get_typeset() == var_m[v3].as_bool().get_typeset() check_concrete_typing_xform(var_m, xform) diff --git a/lib/cretonne/meta/cdsl/ti.py b/lib/cretonne/meta/cdsl/ti.py index b80396b428..7a95daf425 100644 --- a/lib/cretonne/meta/cdsl/ti.py +++ b/lib/cretonne/meta/cdsl/ti.py @@ -104,7 +104,6 @@ class TypesEqual(TypeConstraint): """ def __init__(self, tv1, tv2): # type: (TypeVar, TypeVar) -> None - assert tv1.is_derived and tv2.is_derived (self.tv1, self.tv2) = sorted([tv1, tv2], key=repr) def _args(self): @@ -279,7 +278,7 @@ class TypeEnv(object): :attribute idx: counter used to get fresh ids """ - RANK_DERIVED = 5 + RANK_SINGLETON = 5 RANK_INPUT = 4 RANK_INTERMEDIATE = 3 RANK_OUTPUT = 2 @@ -364,12 +363,18 @@ class TypeEnv(object): # type: (TypeVar) -> int """ Get the rank of tv in the partial order. TVs directly associated with a - Var get their rank from the Var (see register()). - Internally generated non-derived TVs implicitly get the lowest rank (0) - Derived variables get the highest rank. + Var get their rank from the Var (see register()). Internally generated + non-derived TVs implicitly get the lowest rank (0). Derived variables + get their rank from their free typevar. Singletons have the highest + rank. TVs associated with vars in a source pattern have a higher rank + than TVs associted with temporary vars. """ - default_rank = TypeEnv.RANK_DERIVED if tv.is_derived else\ - TypeEnv.RANK_INTERNAL + default_rank = TypeEnv.RANK_INTERNAL if tv.singleton_type() is None \ + else TypeEnv.RANK_SINGLETON + + if tv.is_derived: + tv = tv.free_typevar() + return self.ranks.get(tv, default_rank) def register(self, v): @@ -565,28 +570,36 @@ class TypeEnv(object): # Add all registered TVs (as some of them may be singleton nodes not # appearing in the graph - nodes = set([v.get_typevar() for v in self.vars]) # type: Set[TypeVar] + nodes = set() # type: Set[TypeVar] edges = set() # type: Set[Tuple[TypeVar, TypeVar, str, str, Optional[str]]] # noqa - for (k, v) in self.type_map.items(): + def add_nodes(*args): + # type: (*TypeVar) -> None + for tv in args: + nodes.add(tv) + while (tv.is_derived): + nodes.add(tv.base) + edges.add((tv, tv.base, "solid", "forward", + tv.derived_func)) + tv = tv.base + + for v in self.vars: + add_nodes(v.get_typevar()) + + for (tv1, tv2) in self.type_map.items(): # Add all intermediate TVs appearing in edges - nodes.add(k) - nodes.add(v) - edges.add((k, v, "dotted", "forward", None)) - while (v.is_derived): - nodes.add(v.base) - edges.add((v, v.base, "solid", "forward", v.derived_func)) - v = v.base + add_nodes(tv1, tv2) + edges.add((tv1, tv2, "dotted", "forward", None)) for constr in self.constraints: if isinstance(constr, TypesEqual): - assert constr.tv1 in nodes and constr.tv2 in nodes + add_nodes(constr.tv1, constr.tv2) edges.add((constr.tv1, constr.tv2, "dashed", "none", "equal")) elif isinstance(constr, WiderOrEq): - assert constr.tv1 in nodes and constr.tv2 in nodes + add_nodes(constr.tv1, constr.tv2) edges.add((constr.tv1, constr.tv2, "dashed", "forward", ">=")) elif isinstance(constr, SameWidth): - assert constr.tv1 in nodes and constr.tv2 in nodes + add_nodes(constr.tv1, constr.tv2) edges.add((constr.tv1, constr.tv2, "dashed", "none", "same_width")) else: @@ -640,7 +653,9 @@ def get_type_env(typing_or_err): """ Helper function to appease mypy when checking the result of typing. """ - assert isinstance(typing_or_err, TypeEnv) + assert isinstance(typing_or_err, TypeEnv), \ + "Unexpected error: {}".format(typing_or_err) + if (TYPE_CHECKING): return cast(TypeEnv, typing_or_err) else: @@ -752,8 +767,6 @@ def unify(tv1, tv2, typ): typ.equivalent(tv1, tv2) return typ - assert tv2.is_derived, "Ordering gives us !tv1.is_derived==>tv2.is_derived" - if (tv1.is_derived and TypeVar.is_bijection(tv1.derived_func)): inv_f = TypeVar.inverse_func(tv1.derived_func) return unify(tv1.base, normalize_tv(TypeVar.derived(tv2, inv_f)), typ) diff --git a/lib/cretonne/meta/semantics/__init__.py b/lib/cretonne/meta/semantics/__init__.py index 8caf792259..1c1fee9b9f 100644 --- a/lib/cretonne/meta/semantics/__init__.py +++ b/lib/cretonne/meta/semantics/__init__.py @@ -31,6 +31,7 @@ def verify_semantics(inst, src, xforms): # 2) Any possible typing for the instruction should be covered by # exactly ONE semantic XForm + src = src.copy({}) typenv = get_type_env(ti_rtl(src, TypeEnv())) typenv.normalize() typenv = typenv.extract() diff --git a/lib/cretonne/meta/semantics/primitives.py b/lib/cretonne/meta/semantics/primitives.py index d5b6b35d8b..70fc196d5d 100644 --- a/lib/cretonne/meta/semantics/primitives.py +++ b/lib/cretonne/meta/semantics/primitives.py @@ -9,7 +9,6 @@ from __future__ import absolute_import from cdsl.operands import Operand from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup -from cdsl.ti import SameWidth import base.formats # noqa GROUP = InstructionGroup("primitive", "Primitive instruction set") @@ -39,8 +38,7 @@ prim_from_bv = Instruction( 'prim_from_bv', r""" Convert a flat bitvector to a real SSA Value. """, - ins=(x), outs=(real), - constraints=SameWidth(BV, Real)) + ins=(fromReal), outs=(real)) xh = Operand('xh', BV.half_width(), doc="A semantic value representing the upper half of X") diff --git a/lib/cretonne/meta/semantics/test_elaborate.py b/lib/cretonne/meta/semantics/test_elaborate.py index 4cbfe07bf9..8eafcb9e00 100644 --- a/lib/cretonne/meta/semantics/test_elaborate.py +++ b/lib/cretonne/meta/semantics/test_elaborate.py @@ -205,8 +205,8 @@ class TestElaborate(TestCase): assert concrete_rtls_eq(sem, cleanup_concrete_rtl(Rtl( bvx << prim_to_bv.i32x4(x), (bvlo, bvhi) << bvsplit.bv128(bvx), - lo << prim_from_bv.i32x2.bv64(bvlo), - hi << prim_from_bv.i32x2.bv64(bvhi)))) + lo << prim_from_bv.i32x2(bvlo), + hi << prim_from_bv.i32x2(bvhi)))) def test_elaborate_vconcat(self): # type: () -> None @@ -227,7 +227,7 @@ class TestElaborate(TestCase): bvlo << prim_to_bv.i32x2(lo), bvhi << prim_to_bv.i32x2(hi), bvx << bvconcat.bv64(bvlo, bvhi), - x << prim_from_bv.i32x4.bv128(bvx)))) + x << prim_from_bv.i32x4(bvx)))) def test_elaborate_iadd_simple(self): # type: () -> None @@ -247,7 +247,7 @@ class TestElaborate(TestCase): bvx << prim_to_bv.i32(x), bvy << prim_to_bv.i32(y), bva << bvadd.bv32(bvx, bvy), - a << prim_from_bv.i32.bv32(bva)))) + a << prim_from_bv.i32(bva)))) def test_elaborate_iadd_elaborate_1(self): # type: () -> None @@ -279,7 +279,7 @@ class TestElaborate(TestCase): bva_3 << bvadd.bv32(bvlo_1, bvlo_2), bva_4 << bvadd.bv32(bvhi_1, bvhi_2), bvx_5 << bvconcat.bv32(bva_3, bva_4), - a << prim_from_bv.i32x2.bv64(bvx_5)))) + a << prim_from_bv.i32x2(bvx_5)))) def test_elaborate_iadd_elaborate_2(self): # type: () -> None @@ -334,4 +334,4 @@ class TestElaborate(TestCase): bva_14 << bvadd.bv8(bvhi_11, bvhi_12), bvx_15 << bvconcat.bv8(bva_13, bva_14), bvx_5 << bvconcat.bv16(bvx_10, bvx_15), - a << prim_from_bv.i8x4.bv32(bvx_5)))) + a << prim_from_bv.i8x4(bvx_5)))) diff --git a/lib/cretonne/meta/test_gen_legalizer.py b/lib/cretonne/meta/test_gen_legalizer.py index 538ff69b63..da4683413d 100644 --- a/lib/cretonne/meta/test_gen_legalizer.py +++ b/lib/cretonne/meta/test_gen_legalizer.py @@ -153,8 +153,7 @@ class TestRuntimeChecks(TestCase): def test_vselect_imm(self): # type: () -> None - ts = TypeSet(lanes=(2, 256), ints=(8, 64), - floats=(32, 64), bools=(8, 64)) + ts = TypeSet(lanes=(2, 256), ints=(8, 64)) r = Rtl( self.v0 << iconst(self.imm0), self.v1 << icmp(intcc.eq, self.v2, self.v0), @@ -167,7 +166,7 @@ class TestRuntimeChecks(TestCase): .format(self.v3.get_typevar().name) self.check_yo_check( - x, sequence(typeset_check(self.v3, ts), + x, sequence(typeset_check(self.v2, ts), equiv_check(tv2_exp, tv3_exp))) def test_reduce_extend(self): From 2ccb261e8d854aa2fe789b52b1ead44a6a7f0e50 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 25 Jul 2017 09:37:10 -0700 Subject: [PATCH 0909/3084] Use a more compact encoding list representation. Encodings has a 16-bit "recipe" field, but even Intel only has 57 recipes currently, so it is unlikely that we will ever need to full range. Use this to represent encoding lists more compactly. Change the encoding list to a format that: - Doesn't need a predicate entry before every encoding entry. - Doesn't need a terminator after the list for each instruction. - Supports multiple "stop codes" for configurable guidance of the legalizer. The encoding scheme has these limits: - 2*NR + NS <= 0x1000 - INSTP + ISAP <= 0x1000 Where: - NR is the number of recipes in an ISA, - NS is the number of stop codes (legalization actions). - INSTP is the number of instruction predicates. - ISAP is the number of discrete ISA predicates. --- lib/cretonne/meta/gen_encoding.py | 323 ++++++++++++++++++++++------- lib/cretonne/src/isa/enc_tables.rs | 94 +++++---- 2 files changed, 298 insertions(+), 119 deletions(-) diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 43f02d6416..b8915367fc 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -39,13 +39,13 @@ types, so many of the level 2 tables will be cold. An encoding list is a non-empty sequence of list entries. Each entry has one of these forms: -1. Instruction predicate, encoding recipe, and encoding bits. If the - instruction predicate is true, use this recipe and bits. -2. ISA predicate and skip-count. If the ISA predicate is false, skip the next - *skip-count* entries in the list. If the skip count is zero, stop - completely. -3. Stop. End of list marker. If this is reached, the instruction does not have - a legal encoding. +1. Recipe + bits. Use this encoding if the recipe predicate is satisfied. +2. Recipe + bits, final entry. Use this encoding if the recipe predicate is + satisfied. Otherwise, stop with the default legalization code. +3. Stop with legalization code. +4. Predicate + skip count. Test predicate and skip N entries if it is false. +4. Predicate + stop. Test predicate and stop with the default legalization code + if it is false. The instruction predicate is also used to distinguish between polymorphic instructions with different types for secondary type variables. @@ -56,9 +56,10 @@ from constant_hash import compute_quadratic from unique_table import UniqueSeqTable from collections import OrderedDict, defaultdict import math -import itertools +from itertools import groupby from cdsl.registers import RegClass, Register, Stack from cdsl.predicates import FieldPredicate +from cdsl.settings import SettingGroup try: from typing import Sequence, Set, Tuple, List, Dict, Iterable, DefaultDict, TYPE_CHECKING # noqa @@ -173,42 +174,228 @@ def emit_recipe_predicates(recipes, fmt): fmt.format('Some({}),', pname[p]) -# Encoding lists are represented as u16 arrays. -CODE_BITS = 16 -PRED_BITS = 12 -PRED_MASK = (1 << PRED_BITS) - 1 - -# 0..CODE_ALWAYS means: Check instruction predicate and use the next two -# entries as a (recipe, encbits) pair if true. CODE_ALWAYS is the always-true -# predicate, smaller numbers refer to instruction predicates. -CODE_ALWAYS = PRED_MASK - -# Codes above CODE_ALWAYS indicate an ISA predicate to be tested. -# `x & PRED_MASK` is the ISA predicate number to test. -# `(x >> PRED_BITS)*3` is the number of u16 table entries to skip if the ISA -# predicate is false. (The factor of three corresponds to the (inst-pred, -# recipe, encbits) triples. +# The u16 values in an encoding list entry are interpreted as follows: # -# Finally, CODE_FAIL indicates the end of the list. -CODE_FAIL = (1 << CODE_BITS) - 1 +# NR = len(all_recipes) +# +# entry < 2*NR +# Try Encoding(entry/2, next_entry) if the recipe predicate is satisfied. +# If bit 0 is set, stop with the default legalization code. +# If bit 0 is clear, keep going down the list. +# entry < PRED_START +# Stop with legalization code `entry - 2*NR`. +# +# Remaining entries are interpreted as (skip, pred) pairs, where: +# +# skip = (entry - PRED_START) >> PRED_BITS +# pred = (entry - PRED_START) & PRED_MASK +# +# If the predicate is satisfied, keep going. Otherwise skip over the next +# `skip` entries. If skip == 0, stop with the default legalization code. +# +# The `pred` predicate number is interpreted as an instruction predicate if it +# is in range, otherwise an ISA predicate. -def seq_doc(enc): - # type: (Encoding) -> Tuple[Tuple[int, int, int], str] +class Encoder: """ - Return a tuple containing u16 representations of the instruction predicate - an recipe / encbits. + Encoder for the list format above. - Also return a doc string. + Two parameters are needed: + + :param NR: Number of recipes. + :param NI: Number of instruction predicates. """ - if enc.instp: - p = enc.instp.number - doc = '--> {} when {}'.format(enc, enc.instp) - else: - p = CODE_ALWAYS + + def __init__(self, isa): + # type: (TargetISA) -> None + self.isa = isa + self.NR = len(isa.all_recipes) + self.NI = len(isa.all_instps) + # u16 encoding list words. + self.words = list() # type: List[int] + # Documentation comments: Index into `words` + comment. + self.docs = list() # type: List[Tuple[int, str]] + + # Encoding lists are represented as u16 arrays. + CODE_BITS = 16 + + # Beginning of the predicate code words. + PRED_START = 0x1000 + + # Number of bits used to hold a predicate number (instruction + ISA + # predicates. + PRED_BITS = 12 + + # Mask for extracting the predicate number. + PRED_MASK = (1 << PRED_BITS) - 1 + + def max_skip(self): + # type: () -> int + """The maximum number of entries that a predicate can skip.""" + return (1 << (self.CODE_BITS - self.PRED_BITS)) - 1 + + def recipe(self, enc, final): + # type: (Encoding, bool) -> None + """Add a recipe+bits entry to the list.""" + offset = len(self.words) + code = 2 * enc.recipe.number doc = '--> {}'.format(enc) - assert p <= CODE_ALWAYS - return ((p, enc.recipe.number, enc.encbits), doc) + if final: + code += 1 + doc += ' and stop' + + assert(code < self.PRED_START) + self.words.extend((code, enc.encbits)) + self.docs.append((offset, doc)) + + def _pred(self, pred, skip, n): + # type: (PredNode, int, int) -> None + """Add a predicate entry.""" + assert n <= self.PRED_MASK + code = n | (skip << self.PRED_BITS) + code += self.PRED_START + assert code < (1 << self.CODE_BITS) + + if skip == 0: + doc = 'stop' + else: + doc = 'skip ' + str(skip) + doc = '{} unless {}'.format(doc, pred) + + self.docs.append((len(self.words), doc)) + self.words.append(code) + + def instp(self, pred, skip): + # type: (PredNode, int) -> None + """Add an instruction predicate entry.""" + self._pred(pred, skip, pred.number) + + def isap(self, pred, skip): + # type: (PredNode, int) -> None + """Add an ISA predicate entry.""" + n = self.isa.settings.predicate_number[pred] + # ISA predicates follow the instruction predicates. + self._pred(pred, skip, self.NI + n) + + +class EncNode(object): + """ + An abstract node in the encoder tree for an instruction. + + This tree is used to simplify the predicates guarding recipe+bits entries. + """ + + def size(self): + # type: () -> int + """Get the number of list entries needed to encode this tree.""" + raise NotImplementedError('EncNode.size() is abstract') + + def encode(self, encoder, final): + # type: (Encoder, bool) -> None + """Encode this tree.""" + raise NotImplementedError('EncNode.encode() is abstract') + + def optimize(self): + # type: () -> EncNode + """Transform this encoder tree into something simpler.""" + return self + + def predicate(self): + # type: () -> PredNode + """Get the predicate guarding this tree, or `None` for always""" + return None + + +class EncPred(EncNode): + """ + An encoder tree node which asserts a predicate on its child nodes. + + A `None` predicate is always satisfied. + """ + + def __init__(self, pred, children): + # type: (PredNode, List[EncNode]) -> None + self.pred = pred + self.children = children + + def size(self): + # type: () -> int + s = 1 if self.pred else 0 + s += sum(c.size() for c in self.children) + return s + + def encode(self, encoder, final): + # type: (Encoder, bool) -> None + if self.pred: + skip = 0 if final else self.size() - 1 + ctx = self.pred.predicate_context() + if isinstance(ctx, SettingGroup): + encoder.isap(self.pred, skip) + else: + encoder.instp(self.pred, skip) + + final_idx = len(self.children) - 1 if final else -1 + for idx, node in enumerate(self.children): + node.encode(encoder, idx == final_idx) + + def predicate(self): + # type: () -> PredNode + return self.pred + + def optimize(self): + # type: () -> EncNode + """ + Optimize a predicate node in the tree by combining child nodes that + have identical predicates. + """ + cnodes = list() # type: List[EncNode] + for pred, niter in groupby( + map(lambda c: c.optimize(), self.children), + key=lambda c: c.predicate()): + nodes = list(niter) + if pred is None or len(nodes) <= 1: + cnodes.extend(nodes) + continue + + # We have multiple children with identical predicates. + # Group them all into `n0`. + n0 = nodes[0] + assert isinstance(n0, EncPred) + for n in nodes[1:]: + assert isinstance(n, EncPred) + n0.children.extend(n.children) + + cnodes.append(n0) + + # Finally strip a redundant grouping node. + if self.pred is None and len(cnodes) == 1: + return cnodes[0] + else: + self.children = cnodes + return self + + +class EncLeaf(EncNode): + """ + A leaf in the encoder tree. + + This represents a single `Encoding`, without its predicates (they are + represented in the tree by parent nodes. + """ + + def __init__(self, encoding): + # type: (Encoding) -> None + self.encoding = encoding + + def size(self): + # type: () -> int + # recipe + bits. + return 2 + + def encode(self, encoder, final): + # type: (Encoder, bool) -> None + encoder.recipe(self.encoding, final) class EncList(object): @@ -239,25 +426,23 @@ class EncList(object): name += ' ({})'.format(self.encodings[0].cpumode) return name - def by_isap(self): - # type: () -> Iterable[Tuple[PredNode, Tuple[Encoding, ...]]] + def encoder_tree(self): + # type: () -> EncNode """ - Group the encodings by ISA predicate without reordering them. + Generate an optimized encoder tree for this list. The tree represents + all of the encodings with parent nodes for the predicates that need + checking. + """ + forest = list() # type: List[EncNode] + for enc in self.encodings: + n = EncLeaf(enc) # type: EncNode + if enc.instp: + n = EncPred(enc.instp, [n]) + if enc.isap: + n = EncPred(enc.isap, [n]) + forest.append(n) - Yield a sequence of `(isap, (encs...))` tuples where `isap` is the ISA - predicate or `None`, and `(encs...)` is a tuple of encodings that all - have the same ISA predicate. - """ - maxlen = CODE_FAIL >> PRED_BITS - for isap, groupi in itertools.groupby( - self.encodings, lambda enc: enc.isap): - group = tuple(groupi) - # This probably never happens, but we can't express more than - # maxlen encodings per isap. - while len(group) > maxlen: - yield (isap, group[0:maxlen]) - group = group[maxlen:] - yield (isap, group) + return EncPred(None, forest).optimize() def encode(self, seq_table, doc_table, isa): # type: (UniqueSeqTable, DefaultDict[int, List[str]], TargetISA) -> None # noqa @@ -269,34 +454,20 @@ class EncList(object): Adds comment lines to `doc_table` keyed by seq_table offsets. """ - words = list() # type: List[int] - docs = list() # type: List[Tuple[int, str]] + # Use an encoder object to hold the parameters. + encoder = Encoder(isa) + tree = self.encoder_tree() + tree.encode(encoder, True) - # Group our encodings by isap. - for isap, group in self.by_isap(): - if isap: - # We have an ISA predicate covering `glen` encodings. - pnum = isa.settings.predicate_number[isap] - glen = len(group) - doc = 'skip {}x3 unless {}'.format(glen, isap) - docs.append((len(words), doc)) - words.append((glen << PRED_BITS) | pnum) - - for enc in group: - seq, doc = seq_doc(enc) - docs.append((len(words), doc)) - words.extend(seq) - - # Terminate the list. - words.append(CODE_FAIL) - - self.offset = seq_table.add(words) + self.offset = seq_table.add(encoder.words) # Add doc comments. doc_table[self.offset].append( '{:06x}: {}'.format(self.offset, self.name())) - for pos, doc in docs: + for pos, doc in encoder.docs: doc_table[self.offset + pos].append(doc) + doc_table[self.offset + len(encoder.words)].insert( + 0, 'end of: {}'.format(self.name())) class Level2Table(object): diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index d040dd9ca4..95e79b2982 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -143,22 +143,19 @@ pub fn lookup_enclist(ctrl_typevar: Type, /// Encoding lists are represented as sequences of u16 words. pub type EncListEntry = u16; -/// Number of bits used to represent a predicate. c.f. `meta.gen_encoding.py`. +/// Number of bits used to represent a predicate. c.f. `meta/gen_encoding.py`. const PRED_BITS: u8 = 12; -const PRED_MASK: EncListEntry = (1 << PRED_BITS) - 1; - -/// The match-always instruction predicate. c.f. `meta.gen_encoding.py`. -const CODE_ALWAYS: EncListEntry = PRED_MASK; - -/// The encoding list terminator. -const CODE_FAIL: EncListEntry = 0xffff; +const PRED_MASK: usize = (1 << PRED_BITS) - 1; +/// First code word representing a predicate check. c.f. `meta/gen_encoding.py`. +const PRED_START: usize = 0x1000; /// An iterator over legal encodings for the instruction. pub struct Encodings<'a> { + // Current offset into `enclist`, or out of bounds after we've reached the end. offset: usize, - enclist: &'static [EncListEntry], inst: &'a InstructionData, isa_predicates: PredicateView<'a>, + enclist: &'static [EncListEntry], recipe_predicates: &'static [RecipePredicate], inst_predicates: &'static [InstPredicate], } @@ -166,16 +163,6 @@ pub struct Encodings<'a> { impl<'a> Encodings<'a> { /// Creates a new instance of `Encodings`. /// - /// # Parameters - /// - /// - `offset` an offset into encoding list returned by `lookup_enclist` function. - /// - `enclist` a list of encoding entries. - /// - `recipe_predicates` is a slice of recipe predicate functions. - /// - `inst` the current instruction. - /// - `instp` an instruction predicate number to be evaluated on the current instruction. - /// - `isa_predicate_bytes` an ISA flags as a slice of bytes to evaluate an ISA predicate number - /// on the current instruction. - /// /// This iterator provides search for encodings that applies to the given instruction. The /// encoding lists are laid out such that first call to `next` returns valid entry in the list /// or `None`. @@ -196,42 +183,63 @@ impl<'a> Encodings<'a> { } } - /// Check if the predicate for `recipe` is satisfied. - fn check_recipe(&self, recipe: u16) -> bool { - match self.recipe_predicates[recipe as usize] { + /// Check if the `rpred` recipe predicate s satisfied. + fn check_recipe(&self, rpred: RecipePredicate) -> bool { + match rpred { Some(p) => p(self.isa_predicates, self.inst), None => true, } } + + /// Check an instruction or isa predicate. + fn check_pred(&self, pred: usize) -> bool { + if let Some(&p) = self.inst_predicates.get(pred) { + p(self.inst) + } else { + let pred = pred - self.inst_predicates.len(); + self.isa_predicates.test(pred) + } + } } impl<'a> Iterator for Encodings<'a> { type Item = Encoding; fn next(&mut self) -> Option { - while self.enclist[self.offset] != CODE_FAIL { - let pred = self.enclist[self.offset]; - if pred <= CODE_ALWAYS { - // This is an instruction predicate followed by recipe and encbits entries. - self.offset += 3; - let satisfied = match self.inst_predicates.get(pred as usize) { - Some(p) => p(self.inst), - None => true, - }; - if satisfied { - let recipe = self.enclist[self.offset - 2]; - if self.check_recipe(recipe) { - let encoding = Encoding::new(recipe, self.enclist[self.offset - 1]); - return Some(encoding); - } + while let Some(entryref) = self.enclist.get(self.offset) { + let entry = *entryref as usize; + + // Check for "recipe+bits". + let recipe = entry >> 1; + if let Some(&rpred) = self.recipe_predicates.get(recipe) { + let bits = self.offset + 1; + if entry & 1 == 0 { + self.offset += 2; // Next entry. + } else { + self.offset = !0; // Stop. } - } else { - // This is an ISA predicate entry. + if self.check_recipe(rpred) { + return Some(Encoding::new(recipe as u16, self.enclist[bits])); + } + continue; + } + + // Check for "stop with legalize". + if entry < PRED_START { + unimplemented!(); + } + + // Finally, this must be a predicate entry. + let pred_entry = entry - PRED_START; + let skip = pred_entry >> PRED_BITS; + let pred = pred_entry & PRED_MASK; + + if self.check_pred(pred) { self.offset += 1; - if !self.isa_predicates.test((pred & PRED_MASK) as usize) { - // ISA predicate failed, skip the next N entries. - self.offset += 3 * (pred >> PRED_BITS) as usize; - } + } else if skip == 0 { + self.offset = !0 // This means stop. + } else { + self.offset += 1 + skip; } } None From 5a2bb8ba32607a64aa20b33bd0019fafa1fc6a20 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 21 Jul 2017 13:04:18 -0700 Subject: [PATCH 0910/3084] Define I64 before I32 for better encoding table compression. The encoding list compression algorithm is not the sharpest knife in the drawer. It can reuse subsets of I64 encoding lists for I32 instructions, but only when the I64 lists are defined first. With this change and the previous change to the encoding list format, we get the following table sizes for the Intel ISA: ENCLISTS: 1478 B -> 662 B LEVEL2: 1072 B (unchanged) LEVEL1: 32 B -> 48 B Total: 2582 B -> 1782 B (-31%) --- lib/cretonne/meta/isa/intel/defs.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/meta/isa/intel/defs.py b/lib/cretonne/meta/isa/intel/defs.py index 5078bcec7e..ad13741ebc 100644 --- a/lib/cretonne/meta/isa/intel/defs.py +++ b/lib/cretonne/meta/isa/intel/defs.py @@ -11,5 +11,5 @@ from . import instructions as x86 ISA = TargetISA('intel', [base.instructions.GROUP, x86.GROUP]) # CPU modes for 32-bit and 64-bit operation. -I32 = CPUMode('I32', ISA) I64 = CPUMode('I64', ISA) +I32 = CPUMode('I32', ISA) From 637966dc7fb890dec6e8e38d0944bedb1d4393ab Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 25 Jul 2017 16:33:35 -0700 Subject: [PATCH 0911/3084] Add support for legalization codes in the encoding tables. The new encoding format allows entries that mean "stop with this legalization code" which makes it possible to configure legalization actions per instruction, instead of only per controlling type variable. This patch adds the Rust side of the legalization codes: - Add an `Encodings::legalize()` method on the encoding iterator which can be called after the iterator has returned `None`. The returned code is either the default legalization action for the type, or a specific code encountered in the encoding list. - Change `lookup_enclist` to return a full iterator instead of just an offset. The two-phase lookup can bail at multiple points, each time with a default legalization code from the level 1 table. This default legalization code is stored in the returned iterator. - Change all the implementations of legal_encodings() in the ISA implementations. This change means that we don't need to return a Result any longer. The `Encodings` iterator can be empty with an associated legalization code. --- lib/cretonne/src/isa/arm32/mod.rs | 23 +++--- lib/cretonne/src/isa/arm64/mod.rs | 23 +++--- lib/cretonne/src/isa/enc_tables.rs | 115 ++++++++++++++++++----------- lib/cretonne/src/isa/intel/mod.rs | 23 +++--- lib/cretonne/src/isa/mod.rs | 6 +- lib/cretonne/src/isa/riscv/mod.rs | 23 +++--- 6 files changed, 116 insertions(+), 97 deletions(-) diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 1e5cde38db..f9a44db258 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -10,7 +10,7 @@ use binemit::{CodeSink, MemoryCodeSink, emit_function}; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Legalize}; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; @@ -62,22 +62,19 @@ impl TargetIsa for Isa { } fn legal_encodings<'a>(&'a self, - _dfg: &'a ir::DataFlowGraph, + dfg: &'a ir::DataFlowGraph, inst: &'a ir::InstructionData, ctrl_typevar: ir::Type) - -> Result, Legalize> { + -> Encodings<'a> { lookup_enclist(ctrl_typevar, - inst.opcode(), + inst, + dfg, self.cpumode, - &enc_tables::LEVEL2[..]) - .and_then(|enclist_offset| { - Ok(Encodings::new(enclist_offset, - &enc_tables::ENCLISTS[..], - &enc_tables::RECIPE_PREDICATES[..], - &enc_tables::INST_PREDICATES[..], - inst, - self.isa_flags.predicate_view())) - }) + &enc_tables::LEVEL2[..], + &enc_tables::ENCLISTS[..], + &enc_tables::RECIPE_PREDICATES[..], + &enc_tables::INST_PREDICATES[..], + self.isa_flags.predicate_view()) } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 4dd873e17b..f530d4b0b2 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -10,7 +10,7 @@ use binemit::{CodeSink, MemoryCodeSink, emit_function}; use super::super::settings as shared_settings; use isa::enc_tables::{lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Legalize}; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; @@ -55,22 +55,19 @@ impl TargetIsa for Isa { } fn legal_encodings<'a>(&'a self, - _dfg: &'a ir::DataFlowGraph, + dfg: &'a ir::DataFlowGraph, inst: &'a ir::InstructionData, ctrl_typevar: ir::Type) - -> Result, Legalize> { + -> Encodings<'a> { lookup_enclist(ctrl_typevar, - inst.opcode(), + inst, + dfg, &enc_tables::LEVEL1_A64[..], - &enc_tables::LEVEL2[..]) - .and_then(|enclist_offset| { - Ok(Encodings::new(enclist_offset, - &enc_tables::ENCLISTS[..], - &enc_tables::RECIPE_PREDICATES[..], - &enc_tables::INST_PREDICATES[..], - inst, - self.isa_flags.predicate_view())) - }) + &enc_tables::LEVEL2[..], + &enc_tables::ENCLISTS[..], + &enc_tables::RECIPE_PREDICATES[..], + &enc_tables::INST_PREDICATES[..], + self.isa_flags.predicate_view()) } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index 95e79b2982..402bf8f06d 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -4,8 +4,8 @@ //! `lib/cretonne/meta/gen_encoding.py`. use constant_hash::{Table, probe}; -use ir::{Type, Opcode, InstructionData}; -use isa::{Encoding, Legalize}; +use ir::{Type, Opcode, DataFlowGraph, InstructionData}; +use isa::Encoding; use settings::PredicateView; use std::ops::Range; @@ -97,45 +97,59 @@ impl + Copy> Table for [Level2Entry] { } } -/// Two-level hash table lookup. +/// Two-level hash table lookup and iterator construction. /// /// Given the controlling type variable and instruction opcode, find the corresponding encoding /// list. /// -/// Returns an offset into the ISA's `ENCLIST` table, or `None` if the opcode/type combination is -/// not legal. -pub fn lookup_enclist(ctrl_typevar: Type, - opcode: Opcode, - level1_table: &[Level1Entry], - level2_table: &[Level2Entry]) - -> Result +/// Returns an iterator that produces legal encodings for `inst`. +pub fn lookup_enclist<'a, OffT1, OffT2>(ctrl_typevar: Type, + inst: &'a InstructionData, + _dfg: &'a DataFlowGraph, + level1_table: &'static [Level1Entry], + level2_table: &'static [Level2Entry], + enclist: &'static [EncListEntry], + recipe_preds: &'static [RecipePredicate], + inst_preds: &'static [InstPredicate], + isa_preds: PredicateView<'a>) + -> Encodings<'a> where OffT1: Into + Copy, OffT2: Into + Copy { - match probe(level1_table, ctrl_typevar, ctrl_typevar.index()) { + let (offset, legalize) = match probe(level1_table, ctrl_typevar, ctrl_typevar.index()) { Err(l1idx) => { // No level 1 entry found for the type. // We have a sentinel entry with the default legalization code. - let l1ent = &level1_table[l1idx]; - Err(l1ent.legalize.into()) + (!0, level1_table[l1idx].legalize) } Ok(l1idx) => { // We have a valid level 1 entry for this type. let l1ent = &level1_table[l1idx]; - match level2_table.get(l1ent.range()) { + let offset = match level2_table.get(l1ent.range()) { Some(l2tab) => { - probe(l2tab, opcode, opcode as usize) - .map(|l2idx| l2tab[l2idx].offset.into() as usize) - .map_err(|_| l1ent.legalize.into()) + let opcode = inst.opcode(); + match probe(l2tab, opcode, opcode as usize) { + Ok(l2idx) => l2tab[l2idx].offset.into() as usize, + Err(_) => !0, + } } - None => { - // The l1ent range is invalid. This means that we just have a customized - // legalization code for this type. The level 2 table is empty. - Err(l1ent.legalize.into()) - } - } + // The l1ent range is invalid. This means that we just have a customized + // legalization code for this type. The level 2 table is empty. + None => !0, + }; + (offset, l1ent.legalize) } - } + }; + + // Now we have an offset into `enclist` that is `!0` when no encoding list could be found. + // The default legalization code is always valid. + Encodings::new(offset, + legalize, + inst, + enclist, + recipe_preds, + inst_preds, + isa_preds) } /// Encoding list entry. @@ -153,11 +167,13 @@ const PRED_START: usize = 0x1000; pub struct Encodings<'a> { // Current offset into `enclist`, or out of bounds after we've reached the end. offset: usize, + // Legalization code to use of no encoding is found. + legalize: LegalizeCode, inst: &'a InstructionData, - isa_predicates: PredicateView<'a>, enclist: &'static [EncListEntry], - recipe_predicates: &'static [RecipePredicate], - inst_predicates: &'static [InstPredicate], + recipe_preds: &'static [RecipePredicate], + inst_preds: &'static [InstPredicate], + isa_preds: PredicateView<'a>, } impl<'a> Encodings<'a> { @@ -167,37 +183,49 @@ impl<'a> Encodings<'a> { /// encoding lists are laid out such that first call to `next` returns valid entry in the list /// or `None`. pub fn new(offset: usize, - enclist: &'static [EncListEntry], - recipe_predicates: &'static [RecipePredicate], - inst_predicates: &'static [InstPredicate], + legalize: LegalizeCode, inst: &'a InstructionData, - isa_predicates: PredicateView<'a>) + enclist: &'static [EncListEntry], + recipe_preds: &'static [RecipePredicate], + inst_preds: &'static [InstPredicate], + isa_preds: PredicateView<'a>) -> Self { Encodings { offset, - enclist, inst, - isa_predicates, - recipe_predicates, - inst_predicates, + legalize, + isa_preds, + recipe_preds, + inst_preds, + enclist, } } + /// Get the legalization action that caused the enumeration of encodings to stop. + /// This can be the default legalization action for the type or a custom code for the + /// instruction. + /// + /// This method must only be called after the iterator returns `None`. + pub fn legalize(&self) -> LegalizeCode { + debug_assert_eq!(self.offset, !0, "Premature Encodings::legalize()"); + self.legalize + } + /// Check if the `rpred` recipe predicate s satisfied. fn check_recipe(&self, rpred: RecipePredicate) -> bool { match rpred { - Some(p) => p(self.isa_predicates, self.inst), + Some(p) => p(self.isa_preds, self.inst), None => true, } } /// Check an instruction or isa predicate. fn check_pred(&self, pred: usize) -> bool { - if let Some(&p) = self.inst_predicates.get(pred) { + if let Some(&p) = self.inst_preds.get(pred) { p(self.inst) } else { - let pred = pred - self.inst_predicates.len(); - self.isa_predicates.test(pred) + let pred = pred - self.inst_preds.len(); + self.isa_preds.test(pred) } } } @@ -211,7 +239,7 @@ impl<'a> Iterator for Encodings<'a> { // Check for "recipe+bits". let recipe = entry >> 1; - if let Some(&rpred) = self.recipe_predicates.get(recipe) { + if let Some(&rpred) = self.recipe_preds.get(recipe) { let bits = self.offset + 1; if entry & 1 == 0 { self.offset += 2; // Next entry. @@ -226,7 +254,9 @@ impl<'a> Iterator for Encodings<'a> { // Check for "stop with legalize". if entry < PRED_START { - unimplemented!(); + self.legalize = (entry - 2 * self.recipe_preds.len()) as LegalizeCode; + self.offset = !0; // Stop. + return None; } // Finally, this must be a predicate entry. @@ -237,7 +267,8 @@ impl<'a> Iterator for Encodings<'a> { if self.check_pred(pred) { self.offset += 1; } else if skip == 0 { - self.offset = !0 // This means stop. + self.offset = !0; // Stop. + return None; } else { self.offset += 1 + skip; } diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 85445ac701..9d31727b14 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -10,7 +10,7 @@ use binemit::{CodeSink, MemoryCodeSink, emit_function}; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Legalize}; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; @@ -62,22 +62,19 @@ impl TargetIsa for Isa { } fn legal_encodings<'a>(&'a self, - _dfg: &'a ir::DataFlowGraph, + dfg: &'a ir::DataFlowGraph, inst: &'a ir::InstructionData, ctrl_typevar: ir::Type) - -> Result, Legalize> { + -> Encodings<'a> { lookup_enclist(ctrl_typevar, - inst.opcode(), + inst, + dfg, self.cpumode, - &enc_tables::LEVEL2[..]) - .and_then(|enclist_offset| { - Ok(Encodings::new(enclist_offset, - &enc_tables::ENCLISTS[..], - &enc_tables::RECIPE_PREDICATES[..], - &enc_tables::INST_PREDICATES[..], - inst, - self.isa_flags.predicate_view())) - }) + &enc_tables::LEVEL2[..], + &enc_tables::ENCLISTS[..], + &enc_tables::RECIPE_PREDICATES[..], + &enc_tables::INST_PREDICATES[..], + self.isa_flags.predicate_view()) } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 7b25aec8ec..6c43040825 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -156,7 +156,7 @@ pub trait TargetIsa { dfg: &'a ir::DataFlowGraph, inst: &'a ir::InstructionData, ctrl_typevar: ir::Type) - -> Result, Legalize>; + -> Encodings<'a>; /// Encode an instruction after determining it is legal. /// @@ -169,8 +169,8 @@ pub trait TargetIsa { inst: &ir::InstructionData, ctrl_typevar: ir::Type) -> Result { - self.legal_encodings(dfg, inst, ctrl_typevar) - .and_then(|mut iter| iter.next().ok_or(Legalize::Expand)) + let mut iter = self.legal_encodings(dfg, inst, ctrl_typevar); + iter.next().ok_or(iter.legalize().into()) } /// Get a data structure describing the instruction encodings in this ISA. diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 3fe675b7a4..18f0172f86 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -10,7 +10,7 @@ use super::super::settings as shared_settings; use binemit::{CodeSink, MemoryCodeSink, emit_function}; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo, Legalize}; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; @@ -62,22 +62,19 @@ impl TargetIsa for Isa { } fn legal_encodings<'a>(&'a self, - _dfg: &'a ir::DataFlowGraph, + dfg: &'a ir::DataFlowGraph, inst: &'a ir::InstructionData, ctrl_typevar: ir::Type) - -> Result, Legalize> { + -> Encodings<'a> { lookup_enclist(ctrl_typevar, - inst.opcode(), + inst, + dfg, self.cpumode, - &enc_tables::LEVEL2[..]) - .and_then(|enclist_offset| { - Ok(Encodings::new(enclist_offset, - &enc_tables::ENCLISTS[..], - &enc_tables::RECIPE_PREDICATES[..], - &enc_tables::INST_PREDICATES[..], - inst, - self.isa_flags.predicate_view())) - }) + &enc_tables::LEVEL2[..], + &enc_tables::ENCLISTS[..], + &enc_tables::RECIPE_PREDICATES[..], + &enc_tables::INST_PREDICATES[..], + self.isa_flags.predicate_view()) } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { From 84aeb3eb56fd852fcdd3aac80e6923d542ab3feb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Jul 2017 08:12:16 -0700 Subject: [PATCH 0912/3084] Generate type check predicates for secondary type variables. The encoding tables are keyed by the controlling type variable only. We need to distinguish different encodings for instructions with multiple type variables. Add a TypePredicate instruction predicate which can check the type of an instruction value operand. Combine type checks into the instruction predicate for instructions with more than one type variable. Add Intel encodings for fcvt_from_sint.f32.i64 which can now be distinguished from fcvt_from_sint.f32.i32. --- .../filetests/isa/intel/binary64-float.cton | 8 +- lib/cretonne/meta/cdsl/formats.py | 24 ++++++ lib/cretonne/meta/cdsl/isa.py | 9 ++- lib/cretonne/meta/cdsl/predicates.py | 79 ++++++++++++++++++- lib/cretonne/meta/gen_encoding.py | 50 ++++++++---- lib/cretonne/meta/isa/intel/encodings.py | 2 + lib/cretonne/src/isa/enc_tables.rs | 10 ++- lib/cretonne/src/isa/intel/enc_tables.rs | 3 +- lib/cretonne/src/isa/riscv/enc_tables.rs | 3 +- 9 files changed, 156 insertions(+), 32 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index ba604ad43c..e8a7c574c1 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -21,9 +21,9 @@ ebb0: [-,%xmm10] v11 = fcvt_from_sint.f32 v1 ; bin: f3 44 0f 2a d6 ; asm: cvtsi2ssq %rax, %xmm5 - [-,%xmm5] v12 = fcvt_from_sint.f32 v2 ; TODO: f3 48 0f 2a e8 + [-,%xmm5] v12 = fcvt_from_sint.f32 v2 ; bin: f3 48 0f 2a e8 ; asm: cvtsi2ssq %r14, %xmm10 - [-,%xmm10] v13 = fcvt_from_sint.f32 v3 ; TODO: f3 4d 0f 2a d6 + [-,%xmm10] v13 = fcvt_from_sint.f32 v3 ; bin: f3 4d 0f 2a d6 ; Binary arithmetic. @@ -86,9 +86,9 @@ ebb0: [-,%xmm10] v11 = fcvt_from_sint.f64 v1 ; bin: f2 44 0f 2a d6 ; asm: cvtsi2sdq %rax, %xmm5 - [-,%xmm5] v12 = fcvt_from_sint.f64 v2 ; TODO: f2 48 0f 2a e8 + [-,%xmm5] v12 = fcvt_from_sint.f64 v2 ; bin: f2 48 0f 2a e8 ; asm: cvtsi2sdq %r14, %xmm10 - [-,%xmm10] v13 = fcvt_from_sint.f64 v3 ; TODO: f2 4d 0f 2a d6 + [-,%xmm10] v13 = fcvt_from_sint.f64 v3 ; bin: f2 4d 0f 2a d6 ; Binary arithmetic. diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/cretonne/meta/cdsl/formats.py index 0b7a72fe0a..aba83ed7a2 100644 --- a/lib/cretonne/meta/cdsl/formats.py +++ b/lib/cretonne/meta/cdsl/formats.py @@ -11,6 +11,29 @@ except ImportError: pass +class InstructionContext(object): + """ + Most instruction predicates refer to immediate fields of a specific + instruction format, so their `predicate_context()` method returns the + specific instruction format. + + Predicates that only care about the types of SSA values are independent of + the instruction format. They can be evaluated in the context of any + instruction. + + The singleton `InstructionContext` class serves as the predicate context + for these predicates. + """ + + def __init__(self): + # type: () -> None + self.name = 'inst' + + +# Singleton instance. +instruction_context = InstructionContext() + + class InstructionFormat(object): """ Every instruction opcode has a corresponding instruction format which @@ -48,6 +71,7 @@ class InstructionFormat(object): def __init__(self, *kinds, **kwargs): # type: (*Union[OperandKind, Tuple[str, OperandKind]], **Any) -> None # noqa self.name = kwargs.get('name', None) # type: str + self.parent = instruction_context # The number of value operands stored in the format, or `None` when # `has_value_list` is set. diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index f65c2ad1fc..563cf91e92 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -1,7 +1,7 @@ """Defining instruction set architectures.""" from __future__ import absolute_import from collections import OrderedDict -from .predicates import And +from .predicates import And, TypePredicate from .registers import RegClass, Register, Stack from .ast import Apply from .types import ValueType @@ -405,6 +405,13 @@ class Encoding(object): self.recipe = recipe self.encbits = encbits + + # Add secondary type variables to the instruction predicate. + if len(self.typevars) > 1: + for tv, vt in zip(self.inst.other_typevars, self.typevars[1:]): + typred = TypePredicate.typevar_check(self.inst, tv, vt) + instp = And.combine(instp, typred) + # Record specific predicates. Note that the recipe also has predicates. self.instp = instp self.isap = isap diff --git a/lib/cretonne/meta/cdsl/predicates.py b/lib/cretonne/meta/cdsl/predicates.py index 8a1cc147f9..62779c2d15 100644 --- a/lib/cretonne/meta/cdsl/predicates.py +++ b/lib/cretonne/meta/cdsl/predicates.py @@ -23,14 +23,19 @@ predicate, the context is the instruction format. """ from __future__ import absolute_import from functools import reduce +from .formats import instruction_context try: from typing import Sequence, Tuple, Set, Any, Union, TYPE_CHECKING # noqa if TYPE_CHECKING: - from .formats import InstructionFormat, FormatField # noqa + from .formats import InstructionFormat, InstructionContext, FormatField # noqa + from .instructions import Instruction # noqa from .settings import BoolSetting, SettingGroup # noqa - PredContext = Union[SettingGroup, InstructionFormat] - PredLeaf = Union[BoolSetting, 'FieldPredicate'] + from .types import ValueType # noqa + from .typevar import TypeVar # noqa + PredContext = Union[SettingGroup, InstructionFormat, + InstructionContext] + PredLeaf = Union[BoolSetting, 'FieldPredicate', 'TypePredicate'] PredNode = Union[PredLeaf, 'Predicate'] except ImportError: pass @@ -52,7 +57,7 @@ def _descendant(a, b): If a is a parent of b or b is a parent of a, return the descendant of the two. - If neiher is a parent of the other, return None. + If neither is a parent of the other, return None. """ if _is_parent(a, b): return b @@ -293,3 +298,69 @@ class IsUnsignedInt(FieldPredicate): self.scale = scale assert width >= 0 and width <= 64 assert scale >= 0 and scale < width + + +class TypePredicate(object): + """ + An instruction predicate that checks the type of an SSA argument value. + + Type predicates are used to implement encodings for instructions with + multiple type variables. The encoding tables are keyed by the controlling + type variable, type predicates check any secondary type variables. + + A type predicate is not bound to any specific instruction format. + + :param value_arg: Index of the value argument to type check. + :param value_type: The required value type. + """ + + def __init__(self, value_arg, value_type): + # type: (int, ValueType) -> None + assert value_arg >= 0 + assert value_type is not None + self.value_arg = value_arg + self.value_type = value_type + self.number = None # type: int + # All PredNode members must have a name field. This will never be set. + self.name = None # type: str + + def __str__(self): + # type: () -> str + return 'args[{}]:{}'.format(self.value_arg, self.value_type) + + def predicate_context(self): + # type: () -> PredContext + return instruction_context + + def predicate_leafs(self, leafs): + # type: (Set[PredLeaf]) -> None + leafs.add(self) + + @staticmethod + def typevar_check(inst, typevar, value_type): + # type: (Instruction, TypeVar, ValueType) -> TypePredicate + """ + Return a type check predicate for the given type variable in `inst`. + + The type variable must appear directly as the type of one of the + operands to `inst`, so this is only guaranteed to work for secondary + type variables. + + Find an `inst` value operand whose type is determined by `typevar` and + create a `TypePredicate` that checks that the type variable has the + value `value_type`. + """ + # Find the first value operand whose type is `typevar`. + value_arg = next(i for i, opnum in enumerate(inst.value_opnums) + if inst.ins[opnum].typevar == typevar) + return TypePredicate(value_arg, value_type) + + def rust_predicate(self, prec): + # type: (int) -> str + """ + Return Rust code for evaluating this predicate. + + It is assumed that the context has `dfg` and `args` variables. + """ + return 'dfg.value_type(args[{}]) == {}'.format( + self.value_arg, self.value_type.rust_name()) diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index b8915367fc..6f88aa70fb 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -58,8 +58,9 @@ from collections import OrderedDict, defaultdict import math from itertools import groupby from cdsl.registers import RegClass, Register, Stack -from cdsl.predicates import FieldPredicate +from cdsl.predicates import FieldPredicate, TypePredicate from cdsl.settings import SettingGroup +from cdsl.formats import instruction_context, InstructionFormat try: from typing import Sequence, Set, Tuple, List, Dict, Iterable, DefaultDict, TYPE_CHECKING # noqa @@ -73,8 +74,8 @@ except ImportError: pass -def emit_instp(instp, fmt): - # type: (PredNode, srcgen.Formatter) -> None +def emit_instp(instp, fmt, has_dfg=False): + # type: (PredNode, srcgen.Formatter, bool) -> None """ Emit code for matching an instruction predicate against an `InstructionData` reference called `inst`. @@ -84,22 +85,42 @@ def emit_instp(instp, fmt): """ iform = instp.predicate_context() + # Deal with pure type check predicates which apply to any instruction. + if iform == instruction_context: + fmt.line('let args = inst.arguments(&dfg.value_lists);') + fmt.format('return {};', instp.rust_predicate(0)) + return + + assert isinstance(iform, InstructionFormat) + # Which fields do we need in the InstructionData pattern match? + has_type_check = False # Collect the leaf predicates. leafs = set() # type: Set[PredLeaf] instp.predicate_leafs(leafs) - # All the leafs are FieldPredicate instances. Here we just care about - # the field names. + # All the leafs are FieldPredicate or TypePredicate instances. Here we just + # care about the field names. fnames = set() # type: Set[str] for p in leafs: - assert isinstance(p, FieldPredicate) - fnames.add(p.field.rust_name()) + if isinstance(p, FieldPredicate): + fnames.add(p.field.rust_name()) + else: + assert isinstance(p, TypePredicate) + has_type_check = True fields = ', '.join(sorted(fnames)) with fmt.indented( - 'if let InstructionData::{} {{ {}, .. }} = *inst {{' + 'if let ir::InstructionData::{} {{ {}, .. }} = *inst {{' .format(iform.name, fields), '}'): - fmt.line('return {};'.format(instp.rust_predicate(0))) + if has_type_check: + # We could implement this if we need to. + assert has_dfg, "Recipe predicates can't check type variables." + fmt.line('let args = inst.arguments(&dfg.value_lists);') + elif has_dfg: + # Silence dead argument warning. + fmt.line('let _ = dfg;') + fmt.format('return {};', instp.rust_predicate(0)) + fmt.line('unreachable!();') def emit_inst_predicates(instps, fmt): @@ -111,11 +132,9 @@ def emit_inst_predicates(instps, fmt): for instp in instps: name = 'inst_predicate_{}'.format(instp.number) with fmt.indented( - 'fn {}(inst: &InstructionData) -> bool {{' - .format(name), - '}'): - emit_instp(instp, fmt) - fmt.line('unreachable!();') + 'fn {}(dfg: &ir::DataFlowGraph, inst: &ir::InstructionData)' + '-> bool {{'.format(name), '}'): + emit_instp(instp, fmt, has_dfg=True) # Generate the static table. with fmt.indented( @@ -150,7 +169,7 @@ def emit_recipe_predicates(recipes, fmt): # Generate the predicate function. with fmt.indented( 'fn {}({}: ::settings::PredicateView, ' - 'inst: &InstructionData) -> bool {{' + 'inst: &ir::InstructionData) -> bool {{' .format( name, 'isap' if isap else '_'), '}'): @@ -160,7 +179,6 @@ def emit_recipe_predicates(recipes, fmt): '}'): fmt.line('return false;') emit_instp(instp, fmt) - fmt.line('unreachable!();') # Generate the static table. with fmt.indented( diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index df1a542101..b59dd8d9af 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -229,11 +229,13 @@ I64.enc(base.uextend.i64.i32, *r.umr(0x89)) # cvtsi2ss I32.enc(base.fcvt_from_sint.f32.i32, *r.furm(0xf3, 0x0f, 0x2A)) +I64.enc(base.fcvt_from_sint.f32.i64, *r.furm.rex(0xf3, 0x0f, 0x2A, w=1)) I64.enc(base.fcvt_from_sint.f32.i32, *r.furm.rex(0xf3, 0x0f, 0x2A)) I64.enc(base.fcvt_from_sint.f32.i32, *r.furm(0xf3, 0x0f, 0x2A)) # cvtsi2sd I32.enc(base.fcvt_from_sint.f64.i32, *r.furm(0xf2, 0x0f, 0x2A)) +I64.enc(base.fcvt_from_sint.f64.i64, *r.furm.rex(0xf2, 0x0f, 0x2A, w=1)) I64.enc(base.fcvt_from_sint.f64.i32, *r.furm.rex(0xf2, 0x0f, 0x2A)) I64.enc(base.fcvt_from_sint.f64.i32, *r.furm(0xf2, 0x0f, 0x2A)) diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index 402bf8f06d..d8bd9756d6 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -20,7 +20,7 @@ pub type RecipePredicate = Option bool>; /// /// This is a predicate function that needs to be tested in addition to the recipe predicate. It /// can't depend on ISA settings. -pub type InstPredicate = fn(&InstructionData) -> bool; +pub type InstPredicate = fn(&DataFlowGraph, &InstructionData) -> bool; /// Legalization action to perform when no encoding can be found for an instruction. /// @@ -105,7 +105,7 @@ impl + Copy> Table for [Level2Entry] { /// Returns an iterator that produces legal encodings for `inst`. pub fn lookup_enclist<'a, OffT1, OffT2>(ctrl_typevar: Type, inst: &'a InstructionData, - _dfg: &'a DataFlowGraph, + dfg: &'a DataFlowGraph, level1_table: &'static [Level1Entry], level2_table: &'static [Level2Entry], enclist: &'static [EncListEntry], @@ -146,6 +146,7 @@ pub fn lookup_enclist<'a, OffT1, OffT2>(ctrl_typevar: Type, Encodings::new(offset, legalize, inst, + dfg, enclist, recipe_preds, inst_preds, @@ -170,6 +171,7 @@ pub struct Encodings<'a> { // Legalization code to use of no encoding is found. legalize: LegalizeCode, inst: &'a InstructionData, + dfg: &'a DataFlowGraph, enclist: &'static [EncListEntry], recipe_preds: &'static [RecipePredicate], inst_preds: &'static [InstPredicate], @@ -185,6 +187,7 @@ impl<'a> Encodings<'a> { pub fn new(offset: usize, legalize: LegalizeCode, inst: &'a InstructionData, + dfg: &'a DataFlowGraph, enclist: &'static [EncListEntry], recipe_preds: &'static [RecipePredicate], inst_preds: &'static [InstPredicate], @@ -193,6 +196,7 @@ impl<'a> Encodings<'a> { Encodings { offset, inst, + dfg, legalize, isa_preds, recipe_preds, @@ -222,7 +226,7 @@ impl<'a> Encodings<'a> { /// Check an instruction or isa predicate. fn check_pred(&self, pred: usize) -> bool { if let Some(&p) = self.inst_preds.get(pred) { - p(self.inst) + p(self.dfg, self.inst) } else { let pred = pred - self.inst_preds.len(); self.isa_preds.test(pred) diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 47ea5dbe11..b3144f11e3 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -1,7 +1,6 @@ //! Encoding tables for Intel ISAs. -use ir::types; -use ir::{Opcode, InstructionData}; +use ir::{self, types, Opcode}; use isa::EncInfo; use isa::constraints::*; use isa::enc_tables::*; diff --git a/lib/cretonne/src/isa/riscv/enc_tables.rs b/lib/cretonne/src/isa/riscv/enc_tables.rs index 0dddebdbac..7e05879a3b 100644 --- a/lib/cretonne/src/isa/riscv/enc_tables.rs +++ b/lib/cretonne/src/isa/riscv/enc_tables.rs @@ -1,8 +1,7 @@ //! Encoding tables for RISC-V. use ir::condcodes::IntCC; -use ir::types; -use ir::{Opcode, InstructionData}; +use ir::{self, types, Opcode}; use isa::EncInfo; use isa::constraints::*; use isa::enc_tables::*; From 9ff785fabc1ec9edc956f071fc53a50f332d051f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Jul 2017 09:30:29 -0700 Subject: [PATCH 0913/3084] Add a predicate_key() method to all predicates. This enables interning of predicates to avoid duplicates. Add a predicate registry to TargetIsa for interning predicates per ISA. --- lib/cretonne/meta/cdsl/isa.py | 27 ++++++++++++++++++++++++--- lib/cretonne/meta/cdsl/predicates.py | 21 +++++++++++++++++++++ lib/cretonne/meta/cdsl/settings.py | 23 ++++++++++++++--------- 3 files changed, 59 insertions(+), 12 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 563cf91e92..cdb1923a05 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -13,7 +13,7 @@ try: from typing import Tuple, Union, Any, Iterable, Sequence, List, Set, Dict, TYPE_CHECKING # noqa if TYPE_CHECKING: from .instructions import MaybeBoundInst, InstructionGroup, InstructionFormat # noqa - from .predicates import PredNode # noqa + from .predicates import PredNode, PredKey # noqa from .settings import SettingGroup # noqa from .registers import RegBank # noqa from .xform import XFormGroup # noqa @@ -50,6 +50,8 @@ class TargetISA(object): self.regbanks = list() # type: List[RegBank] self.regclasses = list() # type: List[RegClass] self.legalize_codes = OrderedDict() # type: OrderedDict[XFormGroup, int] # noqa + # Unique copies of all predicates. + self._predicates = dict() # type: Dict[PredKey, PredNode] assert InstructionGroup._current is None,\ "InstructionGroup {} is still open!"\ @@ -86,12 +88,15 @@ class TargetISA(object): for enc in cpumode.encodings: recipe = enc.recipe if recipe not in rcps: + assert recipe.number is None recipe.number = len(rcps) rcps.add(recipe) self.all_recipes.append(recipe) # Make sure ISA predicates are registered. if recipe.isap: + recipe.isap = self.unique_pred(recipe.isap) self.settings.number_predicate(recipe.isap) + recipe.instp = self.unique_pred(recipe.instp) def _collect_predicates(self): # type: () -> None @@ -111,6 +116,7 @@ class TargetISA(object): instp = enc.instp if instp and instp not in instps: # assign predicate number starting from 0. + assert instp.number is None instp.number = len(instps) instps.add(instp) self.all_instps.append(instp) @@ -175,6 +181,21 @@ class TargetISA(object): self.legalize_codes[xgrp] = code return code + def unique_pred(self, pred): + # type: (PredNode) -> PredNode + """ + Get a unique predicate that is equivalent to `pred`. + """ + if pred is None: + return pred + # TODO: We could actually perform some algebraic simplifications. It's + # not clear if it is worthwhile. + k = pred.predicate_key() + if k in self._predicates: + return self._predicates[k] + self._predicates[k] = pred + return pred + class CPUMode(object): """ @@ -413,8 +434,8 @@ class Encoding(object): instp = And.combine(instp, typred) # Record specific predicates. Note that the recipe also has predicates. - self.instp = instp - self.isap = isap + self.instp = self.cpumode.isa.unique_pred(instp) + self.isap = self.cpumode.isa.unique_pred(isap) def __str__(self): # type: () -> str diff --git a/lib/cretonne/meta/cdsl/predicates.py b/lib/cretonne/meta/cdsl/predicates.py index 62779c2d15..ca5d87aa9a 100644 --- a/lib/cretonne/meta/cdsl/predicates.py +++ b/lib/cretonne/meta/cdsl/predicates.py @@ -37,6 +37,9 @@ try: InstructionContext] PredLeaf = Union[BoolSetting, 'FieldPredicate', 'TypePredicate'] PredNode = Union[PredLeaf, 'Predicate'] + # A predicate key is a (recursive) tuple of primitive types that + # uniquely describes a predicate. It is used for interning. + PredKey = Tuple[Any, ...] except ImportError: pass @@ -84,6 +87,7 @@ class Predicate(object): _descendant, (p.predicate_context() for p in parts)) assert self.context, "Incompatible predicate parts" + self.predkey = None # type: PredKey def __str__(self): # type: () -> str @@ -110,6 +114,14 @@ class Predicate(object): # type: (int) -> str raise NotImplementedError("rust_predicate is an abstract method") + def predicate_key(self): + # type: () -> PredKey + """Tuple uniquely identifying a predicate.""" + if not self.predkey: + p = tuple(p.predicate_key() for p in self.parts) # type: PredKey + self.predkey = (type(self).__name__,) + p + return self.predkey + class And(Predicate): """ @@ -224,6 +236,11 @@ class FieldPredicate(object): iform = self.field.format # type: InstructionFormat return iform + def predicate_key(self): + # type: () -> PredKey + a = tuple(map(str, self.args)) + return (self.function, str(self.field)) + a + def predicate_leafs(self, leafs): # type: (Set[PredLeaf]) -> None leafs.add(self) @@ -332,6 +349,10 @@ class TypePredicate(object): # type: () -> PredContext return instruction_context + def predicate_key(self): + # type: () -> PredKey + return ('typecheck', self.value_arg, self.value_type.name) + def predicate_leafs(self, leafs): # type: (Set[PredLeaf]) -> None leafs.add(self) diff --git a/lib/cretonne/meta/cdsl/settings.py b/lib/cretonne/meta/cdsl/settings.py index aca3ac296f..bf3d37dc00 100644 --- a/lib/cretonne/meta/cdsl/settings.py +++ b/lib/cretonne/meta/cdsl/settings.py @@ -7,7 +7,7 @@ try: from typing import Tuple, Set, List, Dict, Any, Union, TYPE_CHECKING # noqa BoolOrPresetOrDict = Union['BoolSetting', 'Preset', Dict['Setting', Any]] if TYPE_CHECKING: - from .predicates import PredLeaf, PredNode # noqa + from .predicates import PredLeaf, PredNode, PredKey # noqa except ImportError: pass @@ -36,14 +36,6 @@ class Setting(object): # type: () -> str return '{}.{}'.format(self.group.name, self.name) - def predicate_context(self): - # type: () -> SettingGroup - """ - Return the context where this setting can be evaluated as a (leaf) - predicate. - """ - return self.group - def default_byte(self): # type: () -> int raise NotImplementedError("default_byte is an abstract method") @@ -96,6 +88,19 @@ class BoolSetting(Setting): # type: () -> int return 1 << self.bit_offset + def predicate_context(self): + # type: () -> SettingGroup + """ + Return the context where this setting can be evaluated as a (leaf) + predicate. + """ + return self.group + + def predicate_key(self): + # type: () -> PredKey + assert self.name, "Can't compute key before setting is named" + return ('setting', self.group.name, self.name) + def predicate_leafs(self, leafs): # type: (Set[PredLeaf]) -> None leafs.add(self) From 98f0a8b8b4810f28643a5977e61e177142b4ac3c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Jul 2017 10:14:26 -0700 Subject: [PATCH 0914/3084] Remove the name field from the PredNode union type. The name of a predicate was only ever used for named settings that are computed as a boolean expression of other settings. - Record the names of these settings in named_predicates instead. - Remove the name field from all predicates. Named predicates does not interact well with the interning of predicates through isa.unique_pred(). --- lib/cretonne/meta/cdsl/predicates.py | 13 ++----------- lib/cretonne/meta/cdsl/settings.py | 10 ++++------ lib/cretonne/meta/gen_settings.py | 18 +++++++----------- 3 files changed, 13 insertions(+), 28 deletions(-) diff --git a/lib/cretonne/meta/cdsl/predicates.py b/lib/cretonne/meta/cdsl/predicates.py index ca5d87aa9a..3b375549c7 100644 --- a/lib/cretonne/meta/cdsl/predicates.py +++ b/lib/cretonne/meta/cdsl/predicates.py @@ -80,7 +80,6 @@ class Predicate(object): def __init__(self, parts): # type: (Sequence[PredNode]) -> None - self.name = None # type: str self.number = None # type: int self.parts = parts self.context = reduce( @@ -91,12 +90,8 @@ class Predicate(object): def __str__(self): # type: () -> str - if self.name: - return '{}.{}'.format(self.context.name, self.name) - else: - return '{}({})'.format( - type(self).__name__, - ', '.join(map(str, self.parts))) + return '{}({})'.format(type(self).__name__, + ', '.join(map(str, self.parts))) def predicate_context(self): # type: () -> PredContext @@ -219,8 +214,6 @@ class FieldPredicate(object): self.field = field self.function = function self.args = args - # All PredNode members must have a name field. This will never be set. - self.name = None # type: str def __str__(self): # type: () -> str @@ -338,8 +331,6 @@ class TypePredicate(object): self.value_arg = value_arg self.value_type = value_type self.number = None # type: int - # All PredNode members must have a name field. This will never be set. - self.name = None # type: str def __str__(self): # type: () -> str diff --git a/lib/cretonne/meta/cdsl/settings.py b/lib/cretonne/meta/cdsl/settings.py index bf3d37dc00..331696ccf0 100644 --- a/lib/cretonne/meta/cdsl/settings.py +++ b/lib/cretonne/meta/cdsl/settings.py @@ -190,7 +190,7 @@ class SettingGroup(object): self.settings = [] # type: List[Setting] # Named predicates computed from settings in this group or its # parents. - self.named_predicates = [] # type: List[Predicate] + self.named_predicates = OrderedDict() # type: OrderedDict[str, Predicate] # noqa # All boolean predicates that can be accessed by number. This includes: # - All boolean settings in this group. # - All named predicates. @@ -235,9 +235,7 @@ class SettingGroup(object): assert obj.name is None, obj.name obj.name = name if isinstance(obj, Predicate): - assert obj.name is None - obj.name = name - self.named_predicates.append(obj) + self.named_predicates[name] = obj if isinstance(obj, Preset): assert obj.name is None, obj.name obj.name = name @@ -333,8 +331,8 @@ class SettingGroup(object): self.settings_size = self.byte_size() # Now assign numbers to all our named predicates. - for p in self.named_predicates: - self.number_predicate(p) + for name, pred in self.named_predicates.items(): + self.number_predicate(pred) def byte_size(self): # type: () -> int diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index 1a79026c36..223e33f478 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -67,13 +67,13 @@ def gen_getter(setting, sgrp, fmt): raise AssertionError("Unknown setting kind") -def gen_pred_getter(pred, sgrp, fmt): - # type: (Predicate, SettingGroup, srcgen.Formatter) -> None +def gen_pred_getter(name, pred, sgrp, fmt): + # type: (str, Predicate, SettingGroup, srcgen.Formatter) -> None """ - Emit a getter for a pre-computed predicate. + Emit a getter for a named pre-computed predicate. """ fmt.doc_comment('Computed predicate `{}`.'.format(pred.rust_predicate(0))) - proto = 'pub fn {}(&self) -> bool'.format(pred.name) + proto = 'pub fn {}(&self) -> bool'.format(name) with fmt.indented(proto + ' {', '}'): fmt.line( 'self.numbered_predicate({})' @@ -103,8 +103,8 @@ def gen_getters(sgrp, fmt): .format(sgrp.boolean_offset)) for setting in sgrp.settings: gen_getter(setting, sgrp, fmt) - for pred in sgrp.named_predicates: - gen_pred_getter(pred, sgrp, fmt) + for name, pred in sgrp.named_predicates.items(): + gen_pred_getter(name, pred, sgrp, fmt) def gen_descriptors(sgrp, fmt): @@ -262,11 +262,7 @@ def gen_constructor(sgrp, parent, fmt): # Don't compute our own settings. if number < sgrp.boolean_settings: continue - if pred.name: - fmt.comment( - 'Precompute #{} ({}).'.format(number, pred.name)) - else: - fmt.comment('Precompute #{}.'.format(number)) + fmt.comment('Precompute #{}.'.format(number)) with fmt.indented( 'if {} {{'.format(pred.rust_predicate(0)), '}'): From ef812408f45900daf24d0b4ed683fc33a9622f04 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Jul 2017 11:01:47 -0700 Subject: [PATCH 0915/3084] Remove the number field from the PredNode union type. Predicate numbers are available in the maps isa.settings.predicate_number and isa.instp_number instead. Like the name field, predicate numbers don't interact well with unique_pred(). --- lib/cretonne/meta/cdsl/isa.py | 14 ++++-------- lib/cretonne/meta/cdsl/predicates.py | 3 --- lib/cretonne/meta/cdsl/settings.py | 1 - lib/cretonne/meta/gen_encoding.py | 34 ++++++++++++++-------------- 4 files changed, 21 insertions(+), 31 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index cdb1923a05..1e08ca0e4f 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -103,23 +103,17 @@ class TargetISA(object): """ Collect and number all predicates in use. - Sets `instp.number` for all used instruction predicates and places them - in `self.all_instps` in numerical order. - Ensures that all ISA predicates have an assigned bit number in `self.settings`. """ - self.all_instps = list() # type: List[PredNode] - instps = set() # type: Set[PredNode] + self.instp_number = OrderedDict() # type: OrderedDict[PredNode, int] for cpumode in self.cpumodes: for enc in cpumode.encodings: instp = enc.instp - if instp and instp not in instps: + if instp and instp not in self.instp_number: # assign predicate number starting from 0. - assert instp.number is None - instp.number = len(instps) - instps.add(instp) - self.all_instps.append(instp) + n = len(self.instp_number) + self.instp_number[instp] = n # All referenced ISA predicates must have a number in # `self.settings`. This may cause some parent predicates to be diff --git a/lib/cretonne/meta/cdsl/predicates.py b/lib/cretonne/meta/cdsl/predicates.py index 3b375549c7..f1f5232272 100644 --- a/lib/cretonne/meta/cdsl/predicates.py +++ b/lib/cretonne/meta/cdsl/predicates.py @@ -80,7 +80,6 @@ class Predicate(object): def __init__(self, parts): # type: (Sequence[PredNode]) -> None - self.number = None # type: int self.parts = parts self.context = reduce( _descendant, @@ -210,7 +209,6 @@ class FieldPredicate(object): def __init__(self, field, function, args): # type: (FormatField, str, Sequence[Any]) -> None - self.number = None # type: int self.field = field self.function = function self.args = args @@ -330,7 +328,6 @@ class TypePredicate(object): assert value_type is not None self.value_arg = value_arg self.value_type = value_type - self.number = None # type: int def __str__(self): # type: () -> str diff --git a/lib/cretonne/meta/cdsl/settings.py b/lib/cretonne/meta/cdsl/settings.py index 331696ccf0..cea8ec7eee 100644 --- a/lib/cretonne/meta/cdsl/settings.py +++ b/lib/cretonne/meta/cdsl/settings.py @@ -23,7 +23,6 @@ class Setting(object): def __init__(self, doc): # type: (str) -> None self.name = None # type: str # Assigned later by `extract_names()`. - self.number = None # type: int self.__doc__ = doc # Offset of byte in settings vector containing this setting. self.byte_offset = None # type: int diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 6f88aa70fb..a7f60f488f 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -124,13 +124,13 @@ def emit_instp(instp, fmt, has_dfg=False): def emit_inst_predicates(instps, fmt): - # type: (Sequence[PredNode], srcgen.Formatter) -> None + # type: (OrderedDict[PredNode, int], srcgen.Formatter) -> None """ Emit private functions for matching instruction predicates as well as a static `INST_PREDICATES` array indexed by predicate number. """ - for instp in instps: - name = 'inst_predicate_{}'.format(instp.number) + for instp, number in instps.items(): + name = 'inst_predicate_{}'.format(number) with fmt.indented( 'fn {}(dfg: &ir::DataFlowGraph, inst: &ir::InstructionData)' '-> bool {{'.format(name), '}'): @@ -140,12 +140,12 @@ def emit_inst_predicates(instps, fmt): with fmt.indented( 'pub static INST_PREDICATES: [InstPredicate; {}] = [' .format(len(instps)), '];'): - for instp in instps: - fmt.format('inst_predicate_{},', instp.number) + for instp, number in instps.items(): + fmt.format('inst_predicate_{},', number) -def emit_recipe_predicates(recipes, fmt): - # type: (Sequence[EncRecipe], srcgen.Formatter) -> None +def emit_recipe_predicates(isa, fmt): + # type: (TargetISA, srcgen.Formatter) -> None """ Emit private functions for checking recipe predicates as well as a static `RECIPE_PREDICATES` array indexed by recipe number. @@ -158,7 +158,7 @@ def emit_recipe_predicates(recipes, fmt): pname = dict() # type: Dict[RecipePred, str] # Generate unique recipe predicates. - for rcp in recipes: + for rcp in isa.all_recipes: p = rcp.recipe_pred() if p is None or p in pname: continue @@ -174,17 +174,16 @@ def emit_recipe_predicates(recipes, fmt): name, 'isap' if isap else '_'), '}'): if isap: - with fmt.indented( - 'if isap.test({})'.format(isap.number), - '}'): + n = isa.settings.predicate_number[isap] + with fmt.indented('if isap.test({})'.format(n), '}'): fmt.line('return false;') emit_instp(instp, fmt) # Generate the static table. with fmt.indented( 'pub static RECIPE_PREDICATES: [RecipePredicate; {}] = [' - .format(len(recipes)), '];'): - for rcp in recipes: + .format(len(isa.all_recipes)), '];'): + for rcp in isa.all_recipes: p = rcp.recipe_pred() if p is None: fmt.line('None,') @@ -229,7 +228,7 @@ class Encoder: # type: (TargetISA) -> None self.isa = isa self.NR = len(isa.all_recipes) - self.NI = len(isa.all_instps) + self.NI = len(isa.instp_number) # u16 encoding list words. self.words = list() # type: List[int] # Documentation comments: Index into `words` + comment. @@ -287,7 +286,8 @@ class Encoder: def instp(self, pred, skip): # type: (PredNode, int) -> None """Add an instruction predicate entry.""" - self._pred(pred, skip, pred.number) + number = self.isa.instp_number[pred] + self._pred(pred, skip, number) def isap(self, pred, skip): # type: (PredNode, int) -> None @@ -836,10 +836,10 @@ def gen_isa(isa, fmt): # type: (TargetISA, srcgen.Formatter) -> None # Make the `RECIPE_PREDICATES` table. - emit_recipe_predicates(isa.all_recipes, fmt) + emit_recipe_predicates(isa, fmt) # Make the `INST_PREDICATES` table. - emit_inst_predicates(isa.all_instps, fmt) + emit_inst_predicates(isa.instp_number, fmt) # Level1 tables, one per CPU mode level1_tables = dict() From 4cffb7fe539d2bcc59fd425852b55a1424f36330 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 26 Jul 2017 14:55:26 -0700 Subject: [PATCH 0916/3084] Add support for type variable wildcards in bound instructions. Instructions will multiple type variables can now use `any` to indicate encodings that don't care about the value of a secondary type variable: ishl.i32.any instead of ishl.i32.i32 This is only allowed for secondary type variables (which are converted to instruction predicates). The controlling type variable must still be fully specified because it is used to key the encoding tables. --- lib/cretonne/meta/cdsl/instructions.py | 5 +++ lib/cretonne/meta/cdsl/isa.py | 3 ++ lib/cretonne/meta/isa/intel/encodings.py | 56 ++++++++++++------------ 3 files changed, 36 insertions(+), 28 deletions(-) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 9d23dea89e..6dec0b7dff 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -314,6 +314,7 @@ class Instruction(object): >>> iadd.i32 """ + assert name != 'any', 'Wildcard not allowed for ctrl_typevar' return self.bind(ValueType.by_name(name)) def fully_bound(self): @@ -386,6 +387,10 @@ class BoundInstruction(object): >>> uext.i32.i8 """ + if name == 'any': + # This is a wild card bind represented as a None type variable. + return self.bind(None) + return self.bind(ValueType.by_name(name)) def fully_bound(self): diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 1e08ca0e4f..487001d9e1 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -424,6 +424,9 @@ class Encoding(object): # Add secondary type variables to the instruction predicate. if len(self.typevars) > 1: for tv, vt in zip(self.inst.other_typevars, self.typevars[1:]): + # A None tv is an 'any' wild card: `ishl.i32.any`. + if vt is None: + continue typred = TypePredicate.typevar_check(self.inst, tv, vt) instp = And.combine(instp, typred) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index b59dd8d9af..df95cf7f48 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -104,10 +104,10 @@ for inst, rrr in [ (base.ishl, 4), (base.ushr, 5), (base.sshr, 7)]: - I32.enc(inst.i32.i32, *r.rc(0xd3, rrr=rrr)) - I64.enc(inst.i64.i64, *r.rc.rex(0xd3, rrr=rrr, w=1)) - I64.enc(inst.i32.i32, *r.rc.rex(0xd3, rrr=rrr)) - I64.enc(inst.i32.i32, *r.rc(0xd3, rrr=rrr)) + I32.enc(inst.i32.any, *r.rc(0xd3, rrr=rrr)) + I64.enc(inst.i64.any, *r.rc.rex(0xd3, rrr=rrr, w=1)) + I64.enc(inst.i32.any, *r.rc.rex(0xd3, rrr=rrr)) + I64.enc(inst.i32.any, *r.rc(0xd3, rrr=rrr)) # Population count. I32.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) @@ -131,37 +131,37 @@ I64.enc(base.ctz.i32, *r.urm.rex(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) I64.enc(base.ctz.i32, *r.urm(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) # Loads and stores. -I32.enc(base.store.i32.i32, *r.st(0x89)) -I32.enc(base.store.i32.i32, *r.stDisp8(0x89)) -I32.enc(base.store.i32.i32, *r.stDisp32(0x89)) +I32.enc(base.store.i32.any, *r.st(0x89)) +I32.enc(base.store.i32.any, *r.stDisp8(0x89)) +I32.enc(base.store.i32.any, *r.stDisp32(0x89)) -I32.enc(base.istore16.i32.i32, *r.st(0x66, 0x89)) -I32.enc(base.istore16.i32.i32, *r.stDisp8(0x66, 0x89)) -I32.enc(base.istore16.i32.i32, *r.stDisp32(0x66, 0x89)) +I32.enc(base.istore16.i32.any, *r.st(0x66, 0x89)) +I32.enc(base.istore16.i32.any, *r.stDisp8(0x66, 0x89)) +I32.enc(base.istore16.i32.any, *r.stDisp32(0x66, 0x89)) -I32.enc(base.istore8.i32.i32, *r.st_abcd(0x88)) -I32.enc(base.istore8.i32.i32, *r.stDisp8_abcd(0x88)) -I32.enc(base.istore8.i32.i32, *r.stDisp32_abcd(0x88)) +I32.enc(base.istore8.i32.any, *r.st_abcd(0x88)) +I32.enc(base.istore8.i32.any, *r.stDisp8_abcd(0x88)) +I32.enc(base.istore8.i32.any, *r.stDisp32_abcd(0x88)) -I32.enc(base.load.i32.i32, *r.ld(0x8b)) -I32.enc(base.load.i32.i32, *r.ldDisp8(0x8b)) -I32.enc(base.load.i32.i32, *r.ldDisp32(0x8b)) +I32.enc(base.load.i32.any, *r.ld(0x8b)) +I32.enc(base.load.i32.any, *r.ldDisp8(0x8b)) +I32.enc(base.load.i32.any, *r.ldDisp32(0x8b)) -I32.enc(base.uload16.i32.i32, *r.ld(0x0f, 0xb7)) -I32.enc(base.uload16.i32.i32, *r.ldDisp8(0x0f, 0xb7)) -I32.enc(base.uload16.i32.i32, *r.ldDisp32(0x0f, 0xb7)) +I32.enc(base.uload16.i32.any, *r.ld(0x0f, 0xb7)) +I32.enc(base.uload16.i32.any, *r.ldDisp8(0x0f, 0xb7)) +I32.enc(base.uload16.i32.any, *r.ldDisp32(0x0f, 0xb7)) -I32.enc(base.sload16.i32.i32, *r.ld(0x0f, 0xbf)) -I32.enc(base.sload16.i32.i32, *r.ldDisp8(0x0f, 0xbf)) -I32.enc(base.sload16.i32.i32, *r.ldDisp32(0x0f, 0xbf)) +I32.enc(base.sload16.i32.any, *r.ld(0x0f, 0xbf)) +I32.enc(base.sload16.i32.any, *r.ldDisp8(0x0f, 0xbf)) +I32.enc(base.sload16.i32.any, *r.ldDisp32(0x0f, 0xbf)) -I32.enc(base.uload8.i32.i32, *r.ld(0x0f, 0xb6)) -I32.enc(base.uload8.i32.i32, *r.ldDisp8(0x0f, 0xb6)) -I32.enc(base.uload8.i32.i32, *r.ldDisp32(0x0f, 0xb6)) +I32.enc(base.uload8.i32.any, *r.ld(0x0f, 0xb6)) +I32.enc(base.uload8.i32.any, *r.ldDisp8(0x0f, 0xb6)) +I32.enc(base.uload8.i32.any, *r.ldDisp32(0x0f, 0xb6)) -I32.enc(base.sload8.i32.i32, *r.ld(0x0f, 0xbe)) -I32.enc(base.sload8.i32.i32, *r.ldDisp8(0x0f, 0xbe)) -I32.enc(base.sload8.i32.i32, *r.ldDisp32(0x0f, 0xbe)) +I32.enc(base.sload8.i32.any, *r.ld(0x0f, 0xbe)) +I32.enc(base.sload8.i32.any, *r.ldDisp8(0x0f, 0xbe)) +I32.enc(base.sload8.i32.any, *r.ldDisp32(0x0f, 0xbe)) # # Call/return From 051aaed43e01fdcc62c32a8f220464ef88758a4f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 27 Jul 2017 10:58:00 -0700 Subject: [PATCH 0917/3084] Add Intel encodings for more conversion instructions. The following instructions have simple encodings: - bitcast.f32.i32 - bitcast.i32.f32 - bitcast.f64.i64 - bitcast.i64.f64 - fpromote.f64.f32 - fdemote.f32.f64 Also add helper functions enc_flt() and enc_i32_i64 to intel.encodings.py for generating the common set of encodings for an instruction: I32, I64 w/REX, I64 w/o REX. --- .../filetests/isa/intel/binary32-float.cton | 26 +++- .../filetests/isa/intel/binary64-float.cton | 40 ++++- cranelift/filetests/wasm/conversions.cton | 45 ++++++ lib/cretonne/meta/isa/intel/encodings.py | 145 +++++++++--------- lib/cretonne/meta/isa/intel/recipes.py | 20 ++- 5 files changed, 195 insertions(+), 81 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 4a4361707c..5c0dc43b18 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -17,6 +17,21 @@ ebb0: ; asm: cvtsi2ss %esi, %xmm2 [-,%xmm2] v11 = fcvt_from_sint.f32 v1 ; bin: f3 0f 2a d6 + ; asm: cvtss2sd %xmm2, %xmm5 + [-,%xmm5] v12 = fpromote.f64 v11 ; bin: f3 0f 5a ea + ; asm: cvtss2sd %xmm5, %xmm2 + [-,%xmm2] v13 = fpromote.f64 v10 ; bin: f3 0f 5a d5 + + ; asm: movd %ecx, %xmm5 + [-,%xmm5] v14 = bitcast.f32 v0 ; bin: 66 0f 6e e9 + ; asm: movd %esi, %xmm2 + [-,%xmm2] v15 = bitcast.f32 v1 ; bin: 66 0f 6e d6 + + ; asm: movd %xmm5, %ecx + [-,%rcx] v16 = bitcast.i32 v10 ; bin: 66 0f 7e e9 + ; asm: movd %xmm2, %esi + [-,%rsi] v17 = bitcast.i32 v11 ; bin: 66 0f 7e d6 + ; Binary arithmetic. ; asm: addss %xmm2, %xmm5 @@ -70,13 +85,20 @@ ebb0: [-,%rcx] v0 = iconst.i32 1 [-,%rsi] v1 = iconst.i32 2 - ; Binary arithmetic. - ; asm: cvtsi2sd %ecx, %xmm5 [-,%xmm5] v10 = fcvt_from_sint.f64 v0 ; bin: f2 0f 2a e9 ; asm: cvtsi2sd %esi, %xmm2 [-,%xmm2] v11 = fcvt_from_sint.f64 v1 ; bin: f2 0f 2a d6 + ; asm: cvtsd2ss %xmm2, %xmm5 + [-,%xmm5] v12 = fdemote.f32 v11 ; bin: f2 0f 5a ea + ; asm: cvtsd2ss %xmm5, %xmm2 + [-,%xmm2] v13 = fdemote.f32 v10 ; bin: f2 0f 5a d5 + + ; No i64 <-> f64 bitcasts in 32-bit mode. + + ; Binary arithmetic. + ; asm: addsd %xmm2, %xmm5 [-,%xmm5] v20 = fadd v10, v11 ; bin: f2 0f 58 ea ; asm: addsd %xmm5, %xmm2 diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index e8a7c574c1..64dd1ebd05 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -25,27 +25,42 @@ ebb0: ; asm: cvtsi2ssq %r14, %xmm10 [-,%xmm10] v13 = fcvt_from_sint.f32 v3 ; bin: f3 4d 0f 2a d6 + ; asm: cvtss2sd %xmm10, %xmm5 + [-,%xmm5] v14 = fpromote.f64 v11 ; bin: f3 41 0f 5a ea + ; asm: cvtss2sd %xmm5, %xmm10 + [-,%xmm10] v15 = fpromote.f64 v10 ; bin: f3 44 0f 5a d5 + + ; asm: movd %r11d, %xmm5 + [-,%xmm5] v16 = bitcast.f32 v0 ; bin: 66 41 0f 6e eb + ; asm: movd %esi, %xmm10 + [-,%xmm10] v17 = bitcast.f32 v1 ; bin: 66 44 0f 6e d6 + + ; asm: movd %xmm5, %ecx + [-,%rcx] v18 = bitcast.i32 v10 ; bin: 66 40 0f 7e e9 + ; asm: movd %xmm10, %esi + [-,%rsi] v19 = bitcast.i32 v11 ; bin: 66 44 0f 7e d6 + ; Binary arithmetic. ; asm: addss %xmm10, %xmm5 [-,%xmm5] v20 = fadd v10, v11 ; bin: f3 41 0f 58 ea ; asm: addss %xmm5, %xmm10 - [-,%xmm10] v21 = fadd v11, v10 ; bin: f3 44 0f 58 d5 + [-,%xmm10] v21 = fadd v11, v10 ; bin: f3 44 0f 58 d5 ; asm: subss %xmm10, %xmm5 [-,%xmm5] v22 = fsub v10, v11 ; bin: f3 41 0f 5c ea ; asm: subss %xmm5, %xmm10 - [-,%xmm10] v23 = fsub v11, v10 ; bin: f3 44 0f 5c d5 + [-,%xmm10] v23 = fsub v11, v10 ; bin: f3 44 0f 5c d5 ; asm: mulss %xmm10, %xmm5 [-,%xmm5] v24 = fmul v10, v11 ; bin: f3 41 0f 59 ea ; asm: mulss %xmm5, %xmm10 - [-,%xmm10] v25 = fmul v11, v10 ; bin: f3 44 0f 59 d5 + [-,%xmm10] v25 = fmul v11, v10 ; bin: f3 44 0f 59 d5 ; asm: divss %xmm10, %xmm5 [-,%xmm5] v26 = fdiv v10, v11 ; bin: f3 41 0f 5e ea ; asm: divss %xmm5, %xmm10 - [-,%xmm10] v27 = fdiv v11, v10 ; bin: f3 44 0f 5e d5 + [-,%xmm10] v27 = fdiv v11, v10 ; bin: f3 44 0f 5e d5 ; Bitwise ops. ; We use the *ps SSE instructions for everything because they are smaller. @@ -90,12 +105,27 @@ ebb0: ; asm: cvtsi2sdq %r14, %xmm10 [-,%xmm10] v13 = fcvt_from_sint.f64 v3 ; bin: f2 4d 0f 2a d6 + ; asm: cvtsd2ss %xmm10, %xmm5 + [-,%xmm5] v14 = fdemote.f32 v11 ; bin: f2 41 0f 5a ea + ; asm: cvtsd2ss %xmm5, %xmm10 + [-,%xmm10] v15 = fdemote.f32 v10 ; bin: f2 44 0f 5a d5 + + ; asm: movq %rax, %xmm5 + [-,%xmm5] v16 = bitcast.f64 v2 ; bin: 66 48 0f 6e e8 + ; asm: movq %r14, %xmm10 + [-,%xmm10] v17 = bitcast.f64 v3 ; bin: 66 4d 0f 6e d6 + + ; asm: movq %xmm5, %rcx + [-,%rcx] v18 = bitcast.i64 v10 ; bin: 66 48 0f 7e e9 + ; asm: movq %xmm10, %rsi + [-,%rsi] v19 = bitcast.i64 v11 ; bin: 66 4c 0f 7e d6 + ; Binary arithmetic. ; asm: addsd %xmm10, %xmm5 [-,%xmm5] v20 = fadd v10, v11 ; bin: f2 41 0f 58 ea ; asm: addsd %xmm5, %xmm10 - [-,%xmm10] v21 = fadd v11, v10 ; bin: f2 44 0f 58 d5 + [-,%xmm10] v21 = fadd v11, v10 ; bin: f2 44 0f 58 d5 ; asm: subsd %xmm10, %xmm5 [-,%xmm5] v22 = fsub v10, v11 ; bin: f2 41 0f 5c ea diff --git a/cranelift/filetests/wasm/conversions.cton b/cranelift/filetests/wasm/conversions.cton index 6f2354d624..e742ebba27 100644 --- a/cranelift/filetests/wasm/conversions.cton +++ b/cranelift/filetests/wasm/conversions.cton @@ -22,6 +22,27 @@ ebb0(v0: i32): return v1 } +; function %i32_trunc_s_f32(f32) -> i32 +; function %i32_trunc_u_f32(f32) -> i32 +; function %i32_trunc_s_f64(f64) -> i32 +; function %i32_trunc_u_f64(f64) -> i32 +; function %i64_trunc_s_f32(f32) -> i64 +; function %i64_trunc_u_f32(f32) -> i64 +; function %i64_trunc_s_f64(f64) -> i64 +; function %i64_trunc_u_f64(f64) -> i64 + +function %f32_trunc_f64(f64) -> f32 { +ebb0(v0: f64): + v1 = fdemote.f32 v0 + return v1 +} + +function %f64_promote_f32(f32) -> f64 { +ebb0(v0: f32): + v1 = fpromote.f64 v0 + return v1 +} + function %f32_convert_s_i32(i32) -> f32 { ebb0(v0: i32): v1 = fcvt_from_sint.f32 v0 @@ -47,3 +68,27 @@ ebb0(v0: i64): } ; TODO: f*_convert_u_i* (Don't exist on Intel). + +function %i32_reinterpret_f32(f32) -> i32 { +ebb0(v0: f32): + v1 = bitcast.i32 v0 + return v1 +} + +function %f32_reinterpret_i32(i32) -> f32 { +ebb0(v0: i32): + v1 = bitcast.f32 v0 + return v1 +} + +function %i64_reinterpret_f64(f64) -> i64 { +ebb0(v0: f64): + v1 = bitcast.i64 v0 + return v1 +} + +function %f64_reinterpret_i64(i64) -> f64 { +ebb0(v0: i64): + v1 = bitcast.f64 v0 + return v1 +} diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index df95cf7f48..44f8c16677 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -11,6 +11,14 @@ from . import settings as cfg from . import instructions as x86 from base.legalize import narrow, expand +try: + from typing import TYPE_CHECKING + if TYPE_CHECKING: + from cdsl.instructions import MaybeBoundInst # noqa +except ImportError: + pass + + I32.legalize_type( default=narrow, i32=expand, @@ -24,42 +32,52 @@ I64.legalize_type( f32=expand, f64=expand) + +# +# Helper functions for generating encodings. +# + +def enc_i32_i64(inst, recipe, *args, **kwargs): + # type: (MaybeBoundInst, r.TailRecipe, *int, **int) -> None + """ + Add encodings for `inst.i32` to I32. + Add encodings for `inst.i32` to I64 with and without REX. + Add encodings for `inst.i64` to I64 with a REX.W prefix. + """ + I32.enc(inst.i32, *recipe(*args, **kwargs)) + + # REX-less encoding must come after REX encoding so we don't use it by + # default. Otherwise reg-alloc would never use r8 and up. + I64.enc(inst.i32, *recipe.rex(*args, **kwargs)) + I64.enc(inst.i32, *recipe(*args, **kwargs)) + + I64.enc(inst.i64, *recipe.rex(*args, w=1, **kwargs)) + + +def enc_flt(inst, recipe, *args, **kwargs): + # type: (MaybeBoundInst, r.TailRecipe, *int, **int) -> None + """ + Add encodings for floating point instruction `inst` to both I32 and I64. + """ + I32.enc(inst, *recipe(*args, **kwargs)) + I64.enc(inst, *recipe.rex(*args, **kwargs)) + I64.enc(inst, *recipe(*args, **kwargs)) + + for inst, opc in [ (base.iadd, 0x01), (base.isub, 0x29), (base.band, 0x21), (base.bor, 0x09), (base.bxor, 0x31)]: - I32.enc(inst.i32, *r.rr(opc)) + enc_i32_i64(inst, r.rr, opc) - I64.enc(inst.i64, *r.rr.rex(opc, w=1)) - I64.enc(inst.i32, *r.rr.rex(opc)) - # REX-less encoding must come after REX encoding so we don't use it by - # default. Otherwise reg-alloc would never use r8 and up. - I64.enc(inst.i32, *r.rr(opc)) +enc_i32_i64(base.imul, r.rrx, 0x0f, 0xaf) +enc_i32_i64(x86.sdivmodx, r.div, 0xf7, rrr=7) +enc_i32_i64(x86.udivmodx, r.div, 0xf7, rrr=6) -I32.enc(base.imul.i32, *r.rrx(0x0f, 0xaf)) -I64.enc(base.imul.i64, *r.rrx.rex(0x0f, 0xaf, w=1)) -I64.enc(base.imul.i32, *r.rrx.rex(0x0f, 0xaf)) -I64.enc(base.imul.i32, *r.rrx(0x0f, 0xaf)) - -for inst, rrr in [ - (x86.sdivmodx, 7), - (x86.udivmodx, 6)]: - I32.enc(inst.i32, *r.div(0xf7, rrr=rrr)) - I64.enc(inst.i64, *r.div.rex(0xf7, rrr=rrr, w=1)) - I64.enc(inst.i32, *r.div.rex(0xf7, rrr=rrr)) - I64.enc(inst.i32, *r.div(0xf7, rrr=rrr)) - -I32.enc(base.copy.i32, *r.umr(0x89)) -I64.enc(base.copy.i64, *r.umr.rex(0x89, w=1)) -I64.enc(base.copy.i32, *r.umr.rex(0x89)) -I64.enc(base.copy.i32, *r.umr(0x89)) - -I32.enc(base.regmove.i32, *r.rmov(0x89)) -I64.enc(base.regmove.i64, *r.rmov.rex(0x89, w=1)) -I64.enc(base.regmove.i32, *r.rmov.rex(0x89)) -I64.enc(base.regmove.i32, *r.rmov(0x89)) +enc_i32_i64(base.copy, r.umr, 0x89) +enc_i32_i64(base.regmove, r.rmov, 0x89) # Immediate instructions with sign-extended 8-bit and 32-bit immediate. for inst, rrr in [ @@ -67,15 +85,8 @@ for inst, rrr in [ (base.band_imm, 4), (base.bor_imm, 1), (base.bxor_imm, 6)]: - I32.enc(inst.i32, *r.rib(0x83, rrr=rrr)) - I32.enc(inst.i32, *r.rid(0x81, rrr=rrr)) - - I64.enc(inst.i64, *r.rib.rex(0x83, rrr=rrr, w=1)) - I64.enc(inst.i64, *r.rid.rex(0x81, rrr=rrr, w=1)) - I64.enc(inst.i32, *r.rib.rex(0x83, rrr=rrr)) - I64.enc(inst.i32, *r.rid.rex(0x81, rrr=rrr)) - I64.enc(inst.i32, *r.rib(0x83, rrr=rrr)) - I64.enc(inst.i32, *r.rid(0x81, rrr=rrr)) + enc_i32_i64(inst, r.rib, 0x83, rrr=rrr) + enc_i32_i64(inst, r.rid, 0x81, rrr=rrr) # TODO: band_imm.i64 with an unsigned 32-bit immediate can be encoded as # band_imm.i32. Can even use the single-byte immediate for 0xffff_ffXX masks. @@ -179,15 +190,8 @@ I32.enc(base.jump, *r.jmpd(0xe9)) I64.enc(base.jump, *r.jmpb(0xeb)) I64.enc(base.jump, *r.jmpd(0xe9)) -I32.enc(base.brz.i32, *r.tjccb(0x74)) -I64.enc(base.brz.i64, *r.tjccb.rex(0x74, w=1)) -I64.enc(base.brz.i32, *r.tjccb.rex(0x74)) -I64.enc(base.brz.i32, *r.tjccb(0x74)) - -I32.enc(base.brnz.i32, *r.tjccb(0x75)) -I64.enc(base.brnz.i64, *r.tjccb.rex(0x75, w=1)) -I64.enc(base.brnz.i32, *r.tjccb.rex(0x75)) -I64.enc(base.brnz.i32, *r.tjccb(0x75)) +enc_i32_i64(base.brz, r.tjccb, 0x74) +enc_i32_i64(base.brnz, r.tjccb, 0x75) # # Trap as ud2 @@ -198,10 +202,7 @@ I64.enc(base.trap, *r.noop(0x0f, 0x0b)) # # Comparisons # -I32.enc(base.icmp.i32, *r.icscc(0x39)) -I64.enc(base.icmp.i64, *r.icscc.rex(0x39, w=1)) -I64.enc(base.icmp.i32, *r.icscc.rex(0x39)) -I64.enc(base.icmp.i32, *r.icscc(0x39)) +enc_i32_i64(base.icmp, r.icscc, 0x39) # # Convert bool to int. @@ -223,21 +224,31 @@ I64.enc(base.sextend.i64.i32, *r.urm.rex(0x63, w=1)) I64.enc(base.uextend.i64.i32, *r.umr.rex(0x89)) I64.enc(base.uextend.i64.i32, *r.umr(0x89)) + # # Floating point # +# movd +enc_flt(base.bitcast.f32.i32, r.frurm, 0x66, 0x0f, 0x6e) +enc_flt(base.bitcast.i32.f32, r.rfumr, 0x66, 0x0f, 0x7e) + +# movq +I64.enc(base.bitcast.f64.i64, *r.frurm.rex(0x66, 0x0f, 0x6e, w=1)) +I64.enc(base.bitcast.i64.f64, *r.rfumr.rex(0x66, 0x0f, 0x7e, w=1)) + # cvtsi2ss -I32.enc(base.fcvt_from_sint.f32.i32, *r.furm(0xf3, 0x0f, 0x2A)) -I64.enc(base.fcvt_from_sint.f32.i64, *r.furm.rex(0xf3, 0x0f, 0x2A, w=1)) -I64.enc(base.fcvt_from_sint.f32.i32, *r.furm.rex(0xf3, 0x0f, 0x2A)) -I64.enc(base.fcvt_from_sint.f32.i32, *r.furm(0xf3, 0x0f, 0x2A)) +enc_i32_i64(base.fcvt_from_sint.f32, r.frurm, 0xf3, 0x0f, 0x2a) # cvtsi2sd -I32.enc(base.fcvt_from_sint.f64.i32, *r.furm(0xf2, 0x0f, 0x2A)) -I64.enc(base.fcvt_from_sint.f64.i64, *r.furm.rex(0xf2, 0x0f, 0x2A, w=1)) -I64.enc(base.fcvt_from_sint.f64.i32, *r.furm.rex(0xf2, 0x0f, 0x2A)) -I64.enc(base.fcvt_from_sint.f64.i32, *r.furm(0xf2, 0x0f, 0x2A)) +enc_i32_i64(base.fcvt_from_sint.f64, r.frurm, 0xf2, 0x0f, 0x2a) + +# cvtss2sd +enc_flt(base.fpromote.f64.f32, r.furm, 0xf3, 0x0f, 0x5a) + +# cvtsd2ss +enc_flt(base.fdemote.f32.f64, r.furm, 0xf2, 0x0f, 0x5a) + # Binary arithmetic ops. for inst, opc in [ @@ -245,13 +256,8 @@ for inst, opc in [ (base.fsub, 0x5c), (base.fmul, 0x59), (base.fdiv, 0x5e)]: - I32.enc(inst.f32, *r.frm(0xf3, 0x0f, opc)) - I64.enc(inst.f32, *r.frm.rex(0xf3, 0x0f, opc)) - I64.enc(inst.f32, *r.frm(0xf3, 0x0f, opc)) - - I32.enc(inst.f64, *r.frm(0xf2, 0x0f, opc)) - I64.enc(inst.f64, *r.frm.rex(0xf2, 0x0f, opc)) - I64.enc(inst.f64, *r.frm(0xf2, 0x0f, opc)) + enc_flt(inst.f32, r.frm, 0xf3, 0x0f, opc) + enc_flt(inst.f64, r.frm, 0xf2, 0x0f, opc) # Binary bitwise ops. for inst, opc in [ @@ -259,10 +265,5 @@ for inst, opc in [ (base.band_not, 0x55), (base.bor, 0x56), (base.bxor, 0x57)]: - I32.enc(inst.f32, *r.frm(0x0f, opc)) - I64.enc(inst.f32, *r.frm.rex(0x0f, opc)) - I64.enc(inst.f32, *r.frm(0x0f, opc)) - - I32.enc(inst.f64, *r.frm(0x0f, opc)) - I64.enc(inst.f64, *r.frm.rex(0x0f, opc)) - I64.enc(inst.f64, *r.frm(0x0f, opc)) + enc_flt(inst.f32, r.frm, 0x0f, opc) + enc_flt(inst.f64, r.frm, 0x0f, opc) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 3eb3d51671..3e038e1208 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -232,6 +232,14 @@ umr = TailRecipe( modrm_rr(out_reg0, in_reg0, sink); ''') +# Same as umr, but with FPR -> GPR registers. +rfumr = TailRecipe( + 'rfumr', Unary, size=1, ins=FPR, outs=GPR, + emit=''' + PUT_OP(bits, rex2(out_reg0, in_reg0), sink); + modrm_rr(out_reg0, in_reg0, sink); + ''') + # XX /r, but for a unary operator with separate input/output register. # RM form. urm = TailRecipe( @@ -249,9 +257,17 @@ urm_abcd = TailRecipe( modrm_rr(in_reg0, out_reg0, sink); ''') -# XX /r, RM form, GPR -> FPR. +# XX /r, RM form, FPR -> FPR. furm = TailRecipe( - 'furm', Unary, size=1, ins=GPR, outs=FPR, + 'furm', Unary, size=1, ins=FPR, outs=FPR, + emit=''' + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); + modrm_rr(in_reg0, out_reg0, sink); + ''') + +# XX /r, RM form, GPR -> FPR. +frurm = TailRecipe( + 'frurm', Unary, size=1, ins=GPR, outs=FPR, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_rr(in_reg0, out_reg0, sink); From b547b786837e3f0e42fe85c24d34ca3b35d505d3 Mon Sep 17 00:00:00 2001 From: Denis Merigoux Date: Thu, 27 Jul 2017 16:30:48 -0700 Subject: [PATCH 0918/3084] Bugfix: encode function wasn't calling legalize function properly --- lib/cretonne/src/isa/mod.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 6c43040825..f85c98df53 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -170,7 +170,7 @@ pub trait TargetIsa { ctrl_typevar: ir::Type) -> Result { let mut iter = self.legal_encodings(dfg, inst, ctrl_typevar); - iter.next().ok_or(iter.legalize().into()) + iter.next().ok_or_else(|| iter.legalize().into()) } /// Get a data structure describing the instruction encodings in this ISA. From d1353bba0520c8e5845a887d2fef127d5884e651 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 27 Jul 2017 13:33:45 -0700 Subject: [PATCH 0919/3084] Assign legalization codes early. Make sure legalization codes are assigned by TargetIsa::finish() such that they can be accessed by multiple gen_* drivers. --- lib/cretonne/meta/cdsl/isa.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 487001d9e1..3e903408ef 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -75,6 +75,7 @@ class TargetISA(object): self._collect_encoding_recipes() self._collect_predicates() self._collect_regclasses() + self._collect_legalize_codes() return self def _collect_encoding_recipes(self): @@ -156,6 +157,17 @@ class TargetISA(object): # `isa/registers.rs`. assert len(self.regclasses) <= 32, "Too many register classes" + def _collect_legalize_codes(self): + # type: () -> None + """ + Make sure all legalization transforms have been assigned a code. + """ + for cpumode in self.cpumodes: + self.legalize_code(cpumode.default_legalize) + for x in sorted(cpumode.type_legalize.values(), + key=lambda x: x.name): + self.legalize_code(x) + def legalize_code(self, xgrp): # type: (XFormGroup) -> int """ From 2aca35a9aac192efe61b85f0aa34c53d1445054b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 27 Jul 2017 14:46:56 -0700 Subject: [PATCH 0920/3084] Return a function pointer from TargetIsa::encode(). Replace the isa::Legalize enumeration with a function pointer. This allows an ISA to define its own specific legalization actions instead of relying on the default two. Generate a LEGALIZE_ACTIONS table for each ISA which contains legalization function pointers indexed by the legalization codes that are already in the encoding tables. Include this table in isa/*/enc_tables.rs. Give the `Encodings` iterator a reference to the action table and change its `legalize()` method to return a function pointer instead of an ISA-specific code. The Result<> returned from TargetIsa::encode() no longer implements Debug, so eliminate uses of unwrap and expect on that type. --- lib/cretonne/meta/cdsl/xform.py | 27 +++++++++-- lib/cretonne/meta/gen_encoding.py | 2 +- lib/cretonne/meta/gen_instr.py | 2 + lib/cretonne/meta/gen_legalizer.py | 60 +++++++++++++++++++----- lib/cretonne/src/isa/arm32/enc_tables.rs | 3 +- lib/cretonne/src/isa/arm32/mod.rs | 1 + lib/cretonne/src/isa/arm64/enc_tables.rs | 3 +- lib/cretonne/src/isa/arm64/mod.rs | 1 + lib/cretonne/src/isa/enc_tables.rs | 11 +++-- lib/cretonne/src/isa/intel/enc_tables.rs | 3 +- lib/cretonne/src/isa/intel/mod.rs | 1 + lib/cretonne/src/isa/mod.rs | 29 +++--------- lib/cretonne/src/isa/riscv/enc_tables.rs | 3 +- lib/cretonne/src/isa/riscv/mod.rs | 32 ++++++------- lib/cretonne/src/legalizer/mod.rs | 21 ++------- lib/cretonne/src/regalloc/coalescing.rs | 14 +++--- lib/cretonne/src/regalloc/reload.rs | 14 +++--- lib/cretonne/src/regalloc/spilling.rs | 8 ++-- lib/cretonne/src/verifier/mod.rs | 7 ++- 19 files changed, 140 insertions(+), 102 deletions(-) diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index 3d3f25a3c7..2cae1c1236 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -10,6 +10,7 @@ try: from typing import Union, Iterator, Sequence, Iterable, List, Dict # noqa from typing import Optional, Set # noqa from .ast import Expr, VarMap # noqa + from .isa import TargetISA # noqa from .ti import TypeConstraint # noqa from .typevar import TypeVar # noqa DefApply = Union[Def, Apply] @@ -282,17 +283,37 @@ class XForm(object): class XFormGroup(object): """ A group of related transformations. + + :param isa: A target ISA whose instructions are allowed. + :param chain: A next level group to try if this one doesn't match. """ - def __init__(self, name, doc): - # type: (str, str) -> None + def __init__(self, name, doc, isa=None, chain=None): + # type: (str, str, TargetISA, XFormGroup) -> None self.xforms = list() # type: List[XForm] self.name = name self.__doc__ = doc + self.isa = isa + self.chain = chain def __str__(self): # type: () -> str - return self.name + if self.isa: + return '{}.{}'.format(self.isa.name, self.name) + else: + return self.name + + def rust_name(self): + # type: () -> str + """ + Get the Rust name of this function implementing this transform. + """ + if self.isa: + # This is a function in the same module as the LEGALIZE_ACTION + # table referring to it. + return self.name + else: + return '::legalizer::{}'.format(self.name) def legalize(self, src, dst): # type: (Union[Def, Apply], Rtl) -> None diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index a7f60f488f..c3b3ddb87c 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -875,7 +875,7 @@ def gen_isa(isa, fmt): emit_recipe_sizing(isa, fmt) # Finally, tie it all together in an `EncInfo`. - with fmt.indented('pub static INFO: EncInfo = EncInfo {', '};'): + with fmt.indented('pub static INFO: isa::EncInfo = isa::EncInfo {', '};'): fmt.line('constraints: &RECIPE_CONSTRAINTS,') fmt.line('sizing: &RECIPE_SIZING,') fmt.line('names: &RECIPE_NAMES,') diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 242e5152d3..b0c9943806 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -345,6 +345,8 @@ def gen_typesets_table(fmt, type_sets): """ Generate the table of ValueTypeSets described by type_sets. """ + if len(type_sets.table) == 0: + return fmt.comment('Table of value type sets.') assert len(type_sets.table) <= typeset_limit, "Too many type sets" with fmt.indented( diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index eeb567722e..10c5bcf3ac 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -9,7 +9,7 @@ the input instruction. """ from __future__ import absolute_import from srcgen import Formatter -from base import legalize, instructions +from base import instructions from cdsl.ast import Var from cdsl.ti import ti_rtl, TypeEnv, get_type_env, TypesEqual,\ InTypeset, WiderOrEq @@ -18,7 +18,7 @@ from gen_instr import gen_typesets_table from cdsl.typevar import TypeVar try: - from typing import Sequence, List, Dict # noqa + from typing import Sequence, List, Dict, Set # noqa from cdsl.isa import TargetISA # noqa from cdsl.ast import Def # noqa from cdsl.xform import XForm, XFormGroup # noqa @@ -167,7 +167,7 @@ def unwrap_inst(iref, node, fmt): # The tuple of locals we're extracting is `expr.args`. with fmt.indented( - 'let ({}) = if let InstructionData::{} {{' + 'let ({}) = if let ir::InstructionData::{} {{' .format(', '.join(map(str, expr.args)), iform.name), '};'): # Fields are encoded directly. for f in iform.imm_fields: @@ -348,9 +348,11 @@ def gen_xform_group(xgrp, fmt, type_sets): fmt.doc_comment("Legalize the instruction pointed to by `pos`.") fmt.line('#[allow(unused_variables,unused_assignments)]') with fmt.indented( - 'fn {}(dfg: &mut DataFlowGraph, ' - 'cfg: &mut ControlFlowGraph, pos: &mut Cursor) -> ' + 'pub fn {}(dfg: &mut ir::DataFlowGraph, ' + 'cfg: &mut ::flowgraph::ControlFlowGraph, ' + 'pos: &mut ir::Cursor) -> ' 'bool {{'.format(xgrp.name), '}'): + fmt.line('use ir::InstBuilder;') # Gen the instruction to be legalized. The cursor we're passed must be # pointing at an instruction. @@ -360,21 +362,55 @@ def gen_xform_group(xgrp, fmt, type_sets): for xform in xgrp.xforms: inst = xform.src.rtl[0].expr.inst with fmt.indented( - 'Opcode::{} => {{'.format(inst.camel_name), '}'): + 'ir::Opcode::{} => {{'.format(inst.camel_name), '}'): gen_xform(xform, fmt, type_sets) # We'll assume there are uncovered opcodes. - fmt.line('_ => return false,') + if xgrp.chain: + fmt.format('_ => return {}(dfg, cfg, pos),', + xgrp.chain.rust_name()) + else: + fmt.line('_ => return false,') fmt.line('true') +def gen_isa(isa, fmt, shared_groups): + # type: (TargetISA, Formatter, Set[XFormGroup]) -> None + """ + Generate legalization functions for `isa` and add any shared `XFormGroup`s + encountered to `shared_groups`. + + Generate `TYPE_SETS` and `LEGALIZE_ACTION` tables. + """ + type_sets = UniqueTable() + for xgrp in isa.legalize_codes.keys(): + if xgrp.isa is None: + shared_groups.add(xgrp) + else: + assert xgrp.isa == isa + gen_xform_group(xgrp, fmt, type_sets) + + gen_typesets_table(fmt, type_sets) + + with fmt.indented( + 'pub static LEGALIZE_ACTIONS: [isa::Legalize; {}] = [' + .format(len(isa.legalize_codes)), '];'): + for xgrp in isa.legalize_codes.keys(): + fmt.format('{},', xgrp.rust_name()) + + def generate(isas, out_dir): # type: (Sequence[TargetISA], str) -> None + shared_groups = set() # type: Set[XFormGroup] + + for isa in isas: + fmt = Formatter() + gen_isa(isa, fmt, shared_groups) + fmt.update_file('legalize-{}.rs'.format(isa.name), out_dir) + + # Shared xform groups. fmt = Formatter() - # Table of TypeSet instances type_sets = UniqueTable() - - gen_xform_group(legalize.narrow, fmt, type_sets) - gen_xform_group(legalize.expand, fmt, type_sets) - + for xgrp in sorted(shared_groups, key=lambda g: g.name): + gen_xform_group(xgrp, fmt, type_sets) gen_typesets_table(fmt, type_sets) fmt.update_file('legalizer.rs', out_dir) diff --git a/lib/cretonne/src/isa/arm32/enc_tables.rs b/lib/cretonne/src/isa/arm32/enc_tables.rs index adcc2fd915..eeb8466669 100644 --- a/lib/cretonne/src/isa/arm32/enc_tables.rs +++ b/lib/cretonne/src/isa/arm32/enc_tables.rs @@ -1,9 +1,10 @@ //! Encoding tables for ARM32 ISA. use ir::types; -use isa::EncInfo; +use isa; use isa::constraints::*; use isa::enc_tables::*; use isa::encoding::RecipeSizing; include!(concat!(env!("OUT_DIR"), "/encoding-arm32.rs")); +include!(concat!(env!("OUT_DIR"), "/legalize-arm32.rs")); diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index f9a44db258..fafd5d9871 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -72,6 +72,7 @@ impl TargetIsa for Isa { self.cpumode, &enc_tables::LEVEL2[..], &enc_tables::ENCLISTS[..], + &enc_tables::LEGALIZE_ACTIONS[..], &enc_tables::RECIPE_PREDICATES[..], &enc_tables::INST_PREDICATES[..], self.isa_flags.predicate_view()) diff --git a/lib/cretonne/src/isa/arm64/enc_tables.rs b/lib/cretonne/src/isa/arm64/enc_tables.rs index e57f3cc98c..c2c087e039 100644 --- a/lib/cretonne/src/isa/arm64/enc_tables.rs +++ b/lib/cretonne/src/isa/arm64/enc_tables.rs @@ -1,9 +1,10 @@ //! Encoding tables for ARM64 ISA. use ir::types; -use isa::EncInfo; +use isa; use isa::constraints::*; use isa::enc_tables::*; use isa::encoding::RecipeSizing; include!(concat!(env!("OUT_DIR"), "/encoding-arm64.rs")); +include!(concat!(env!("OUT_DIR"), "/legalize-arm64.rs")); diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index f530d4b0b2..f4b21bf42d 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -65,6 +65,7 @@ impl TargetIsa for Isa { &enc_tables::LEVEL1_A64[..], &enc_tables::LEVEL2[..], &enc_tables::ENCLISTS[..], + &enc_tables::LEGALIZE_ACTIONS[..], &enc_tables::RECIPE_PREDICATES[..], &enc_tables::INST_PREDICATES[..], self.isa_flags.predicate_view()) diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index d8bd9756d6..b9fa09dd65 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -5,7 +5,7 @@ use constant_hash::{Table, probe}; use ir::{Type, Opcode, DataFlowGraph, InstructionData}; -use isa::Encoding; +use isa::{Encoding, Legalize}; use settings::PredicateView; use std::ops::Range; @@ -109,6 +109,7 @@ pub fn lookup_enclist<'a, OffT1, OffT2>(ctrl_typevar: Type, level1_table: &'static [Level1Entry], level2_table: &'static [Level2Entry], enclist: &'static [EncListEntry], + legalize_actions: &'static [Legalize], recipe_preds: &'static [RecipePredicate], inst_preds: &'static [InstPredicate], isa_preds: PredicateView<'a>) @@ -148,6 +149,7 @@ pub fn lookup_enclist<'a, OffT1, OffT2>(ctrl_typevar: Type, inst, dfg, enclist, + legalize_actions, recipe_preds, inst_preds, isa_preds) @@ -173,6 +175,7 @@ pub struct Encodings<'a> { inst: &'a InstructionData, dfg: &'a DataFlowGraph, enclist: &'static [EncListEntry], + legalize_actions: &'static [Legalize], recipe_preds: &'static [RecipePredicate], inst_preds: &'static [InstPredicate], isa_preds: PredicateView<'a>, @@ -189,6 +192,7 @@ impl<'a> Encodings<'a> { inst: &'a InstructionData, dfg: &'a DataFlowGraph, enclist: &'static [EncListEntry], + legalize_actions: &'static [Legalize], recipe_preds: &'static [RecipePredicate], inst_preds: &'static [InstPredicate], isa_preds: PredicateView<'a>) @@ -202,6 +206,7 @@ impl<'a> Encodings<'a> { recipe_preds, inst_preds, enclist, + legalize_actions, } } @@ -210,9 +215,9 @@ impl<'a> Encodings<'a> { /// instruction. /// /// This method must only be called after the iterator returns `None`. - pub fn legalize(&self) -> LegalizeCode { + pub fn legalize(&self) -> Legalize { debug_assert_eq!(self.offset, !0, "Premature Encodings::legalize()"); - self.legalize + self.legalize_actions[self.legalize as usize] } /// Check if the `rpred` recipe predicate s satisfied. diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index b3144f11e3..62ef7b3cd6 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -1,7 +1,7 @@ //! Encoding tables for Intel ISAs. use ir::{self, types, Opcode}; -use isa::EncInfo; +use isa; use isa::constraints::*; use isa::enc_tables::*; use isa::encoding::RecipeSizing; @@ -9,3 +9,4 @@ use predicates; use super::registers::*; include!(concat!(env!("OUT_DIR"), "/encoding-intel.rs")); +include!(concat!(env!("OUT_DIR"), "/legalize-intel.rs")); diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 9d31727b14..1cce45b12e 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -72,6 +72,7 @@ impl TargetIsa for Isa { self.cpumode, &enc_tables::LEVEL2[..], &enc_tables::ENCLISTS[..], + &enc_tables::LEGALIZE_ACTIONS[..], &enc_tables::RECIPE_PREDICATES[..], &enc_tables::INST_PREDICATES[..], self.isa_flags.predicate_view()) diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index f85c98df53..b808b88746 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -45,6 +45,7 @@ pub use isa::encoding::{Encoding, EncInfo}; pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex, regs_overlap}; use binemit; +use flowgraph; use settings; use ir; use regalloc; @@ -116,29 +117,11 @@ impl settings::Configurable for Builder { /// After determining that an instruction doesn't have an encoding, how should we proceed to /// legalize it? /// -/// These actions correspond to the transformation groups defined in `meta/cretonne/legalize.py`. -#[derive(Clone, Copy, PartialEq, Eq, Debug)] -pub enum Legalize { - /// Legalize in terms of narrower types. - Narrow, - - /// Expanding in terms of other instructions using the same types. - Expand, -} - -/// Translate a legalization code into a `Legalize` enum. -/// -/// This mapping is going away soon. It depends on matching the `TargetISA.legalize_code()` -/// mapping. -impl From for Legalize { - fn from(x: u8) -> Legalize { - match x { - 0 => Legalize::Narrow, - 1 => Legalize::Expand, - _ => panic!("Unknown legalization code {}"), - } - } -} +/// The `Encodings` iterator returns a legalization function to call. +pub type Legalize = fn(&mut ir::DataFlowGraph, + &mut flowgraph::ControlFlowGraph, + &mut ir::Cursor) + -> bool; /// Methods that are specialized to a target ISA. pub trait TargetIsa { diff --git a/lib/cretonne/src/isa/riscv/enc_tables.rs b/lib/cretonne/src/isa/riscv/enc_tables.rs index 7e05879a3b..0d11d006c5 100644 --- a/lib/cretonne/src/isa/riscv/enc_tables.rs +++ b/lib/cretonne/src/isa/riscv/enc_tables.rs @@ -2,7 +2,7 @@ use ir::condcodes::IntCC; use ir::{self, types, Opcode}; -use isa::EncInfo; +use isa; use isa::constraints::*; use isa::enc_tables::*; use isa::encoding::RecipeSizing; @@ -16,3 +16,4 @@ use super::registers::*; // - `ENCLIST` // - `INFO` include!(concat!(env!("OUT_DIR"), "/encoding-riscv.rs")); +include!(concat!(env!("OUT_DIR"), "/legalize-riscv.rs")); diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 18f0172f86..03482b920c 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -72,6 +72,7 @@ impl TargetIsa for Isa { self.cpumode, &enc_tables::LEVEL2[..], &enc_tables::ENCLISTS[..], + &enc_tables::LEGALIZE_ACTIONS[..], &enc_tables::RECIPE_PREDICATES[..], &enc_tables::INST_PREDICATES[..], self.isa_flags.predicate_view()) @@ -113,8 +114,11 @@ mod tests { use ir::{DataFlowGraph, InstructionData, Opcode}; use ir::{types, immediates}; - fn encstr(isa: &isa::TargetIsa, enc: isa::Encoding) -> String { - isa.encoding_info().display(enc).to_string() + fn encstr(isa: &isa::TargetIsa, enc: Result) -> String { + match enc { + Ok(e) => isa.encoding_info().display(e).to_string(), + Err(_) => "no encoding".to_string(), + } } #[test] @@ -137,8 +141,7 @@ mod tests { }; // ADDI is I/0b00100 - assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst64, types::I64).unwrap()), - "I#04"); + assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst64, types::I64)), "I#04"); // Try to encode iadd_imm.i64 v1, -10000. let inst64_large = InstructionData::BinaryImm { @@ -148,8 +151,7 @@ mod tests { }; // Immediate is out of range for ADDI. - assert_eq!(isa.encode(&dfg, &inst64_large, types::I64), - Err(isa::Legalize::Expand)); + assert!(isa.encode(&dfg, &inst64_large, types::I64).is_err()); // Create an iadd_imm.i32 which is encodable in RV64. let inst32 = InstructionData::BinaryImm { @@ -159,8 +161,7 @@ mod tests { }; // ADDIW is I/0b00110 - assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32, types::I32).unwrap()), - "I#06"); + assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32, types::I32)), "I#06"); } // Same as above, but for RV32. @@ -184,8 +185,7 @@ mod tests { }; // In 32-bit mode, an i64 bit add should be narrowed. - assert_eq!(isa.encode(&dfg, &inst64, types::I64), - Err(isa::Legalize::Narrow)); + assert!(isa.encode(&dfg, &inst64, types::I64).is_err()); // Try to encode iadd_imm.i64 v1, -10000. let inst64_large = InstructionData::BinaryImm { @@ -195,8 +195,7 @@ mod tests { }; // In 32-bit mode, an i64 bit add should be narrowed. - assert_eq!(isa.encode(&dfg, &inst64_large, types::I64), - Err(isa::Legalize::Narrow)); + assert!(isa.encode(&dfg, &inst64_large, types::I64).is_err()); // Create an iadd_imm.i32 which is encodable in RV32. let inst32 = InstructionData::BinaryImm { @@ -206,8 +205,7 @@ mod tests { }; // ADDI is I/0b00100 - assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32, types::I32).unwrap()), - "I#04"); + assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32, types::I32)), "I#04"); // Create an imul.i32 which is encodable in RV32, but only when use_m is true. let mul32 = InstructionData::Binary { @@ -215,8 +213,7 @@ mod tests { args: [arg32, arg32], }; - assert_eq!(isa.encode(&dfg, &mul32, types::I32), - Err(isa::Legalize::Expand)); + assert!(isa.encode(&dfg, &mul32, types::I32).is_err()); } #[test] @@ -241,7 +238,6 @@ mod tests { opcode: Opcode::Imul, args: [arg32, arg32], }; - assert_eq!(encstr(&*isa, isa.encode(&dfg, &mul32, types::I32).unwrap()), - "R#10c"); + assert_eq!(encstr(&*isa, isa.encode(&dfg, &mul32, types::I32)), "R#10c"); } } diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index f21f7e1d91..c5178a97ee 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -15,9 +15,9 @@ use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; -use ir::{Function, Cursor, DataFlowGraph, InstructionData, Opcode, InstBuilder}; +use ir::{self, Function, Cursor}; use ir::condcodes::IntCC; -use isa::{TargetIsa, Legalize}; +use isa::TargetIsa; use bitset::BitSet; use ir::instructions::ValueTypeSet; @@ -73,22 +73,7 @@ pub fn legalize_function(func: &mut Function, Ok(encoding) => *func.encodings.ensure(inst) = encoding, Err(action) => { // We should transform the instruction into legal equivalents. - // Possible strategies are: - // 1. Legalize::Expand: Expand instruction into sequence of legal instructions. - // Possibly iteratively. () - // 2. Legalize::Narrow: Split the controlling type variable into high and low - // parts. This applies both to SIMD vector types which can be halved and to - // integer types such as `i64` used on a 32-bit ISA. (). - // 3. TODO: Promote the controlling type variable to a larger type. This - // typically means expressing `i8` and `i16` arithmetic in terms if `i32` - // operations on RISC targets. (It may or may not be beneficial to promote - // small vector types versus splitting them.) - // 4. TODO: Convert to library calls. For example, floating point operations on - // an ISA with no IEEE 754 support. - let changed = match action { - Legalize::Expand => expand(&mut func.dfg, cfg, &mut pos), - Legalize::Narrow => narrow(&mut func.dfg, cfg, &mut pos), - }; + let changed = action(&mut func.dfg, cfg, &mut pos); // If the current instruction was replaced, we need to double back and revisit // the expanded sequence. This is both to assign encodings and possible to // expand further. diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 08ecf12589..964644fad3 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -479,9 +479,10 @@ impl<'a> Context<'a> { self.func.dfg.display_inst(pred_inst, self.isa)); // Give it an encoding. - let encoding = self.isa - .encode(&self.func.dfg, &self.func.dfg[inst], ty) - .expect("Can't encode copy"); + let encoding = match self.isa.encode(&self.func.dfg, &self.func.dfg[inst], ty) { + Ok(e) => e, + Err(_) => panic!("Can't encode copy.{}", ty), + }; *self.func.encodings.ensure(inst) = encoding; // Create a live range for the new value. @@ -525,9 +526,10 @@ impl<'a> Context<'a> { ty); // Give it an encoding. - let encoding = self.isa - .encode(&self.func.dfg, &self.func.dfg[inst], ty) - .expect("Can't encode copy"); + let encoding = match self.isa.encode(&self.func.dfg, &self.func.dfg[inst], ty) { + Ok(e) => e, + Err(_) => panic!("Can't encode copy.{}", ty), + }; *self.func.encodings.ensure(inst) = encoding; // Create a live range for the new value. diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 93860071d3..e22978938b 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -220,9 +220,10 @@ impl<'a> Context<'a> { let reg = dfg.ins(pos).fill(cand.value); let fill = dfg.value_def(reg).unwrap_inst(); - *encodings.ensure(fill) = self.isa - .encode(dfg, &dfg[fill], dfg.value_type(reg)) - .expect("Can't encode fill"); + match self.isa.encode(dfg, &dfg[fill], dfg.value_type(reg)) { + Ok(e) => *encodings.ensure(fill) = e, + Err(_) => panic!("Can't encode fill {}", cand.value), + } self.reloads .insert(ReloadedValue { @@ -351,9 +352,10 @@ impl<'a> Context<'a> { .Unary(Opcode::Spill, ty, reg); // Give it an encoding. - *encodings.ensure(inst) = self.isa - .encode(dfg, &dfg[inst], ty) - .expect("Can't encode spill"); + match self.isa.encode(dfg, &dfg[inst], ty) { + Ok(e) => *encodings.ensure(inst) = e, + Err(_) => panic!("Can't encode spill.{}", ty), + } // Update live ranges. self.liveness.move_def_locally(stack, inst); diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 5733f6e80f..49f6d67d09 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -533,10 +533,10 @@ impl<'a> Context<'a> { let ty = dfg.value_type(copy); // Give it an encoding. - let encoding = self.isa - .encode(dfg, &dfg[inst], ty) - .expect("Can't encode copy"); - *self.encodings.ensure(inst) = encoding; + match self.isa.encode(dfg, &dfg[inst], ty) { + Ok(e) => *self.encodings.ensure(inst) = e, + Err(_) => panic!("Can't encode {}", dfg.display_inst(inst, self.isa)), + } // Update live ranges. self.liveness.create_dead(copy, inst, Affinity::Reg(rci)); diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 518c9a703f..a1e04fbb79 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -697,11 +697,10 @@ impl<'a> Verifier<'a> { isa.encoding_info().display(encoding)); } } - Err(e) => { + Err(_) => { return err!(inst, - "Instruction failed to re-encode {}: {:?}", - isa.encoding_info().display(encoding), - e) + "Instruction failed to re-encode {}", + isa.encoding_info().display(encoding)) } } return Ok(()); From d5ca31a6fd3a01c6c4b4a4be2d749b1468162ea0 Mon Sep 17 00:00:00 2001 From: Dimo Date: Wed, 26 Jul 2017 14:41:59 -0700 Subject: [PATCH 0921/3084] bextend/breduce need constraints --- lib/cretonne/meta/base/instructions.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 763550a5ee..a26d791719 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -1382,7 +1382,7 @@ breduce = Instruction( The result type must have the same number of vector lanes as the input, and each lane must not have more bits that the input lanes. If the input and output types are the same, this is a no-op. - """, ins=x, outs=a) + """, ins=x, outs=a, constraints=WiderOrEq(Bool, BoolTo)) BoolTo = TypeVar( 'BoolTo', @@ -1399,7 +1399,7 @@ bextend = Instruction( The result type must have the same number of vector lanes as the input, and each lane must not have fewer bits that the input lanes. If the input and output types are the same, this is a no-op. - """, ins=x, outs=a) + """, ins=x, outs=a, constraints=WiderOrEq(BoolTo, Bool)) IntTo = TypeVar( 'IntTo', 'An integer type with the same number of lanes', From 100fbe94dde7c688ec6adb5620c8a692b6383b55 Mon Sep 17 00:00:00 2001 From: Dimo Date: Wed, 26 Jul 2017 14:47:57 -0700 Subject: [PATCH 0922/3084] When doing ti on a polymorphic definition first unify the control variable, then the rest. --- lib/cretonne/meta/cdsl/ti.py | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/cdsl/ti.py b/lib/cretonne/meta/cdsl/ti.py index 7a95daf425..028579857d 100644 --- a/lib/cretonne/meta/cdsl/ti.py +++ b/lib/cretonne/meta/cdsl/ti.py @@ -8,12 +8,13 @@ from itertools import product try: from typing import Dict, TYPE_CHECKING, Union, Tuple, Optional, Set # noqa - from typing import Iterable, List, Any # noqa + from typing import Iterable, List, Any, TypeVar as MTypeVar # noqa from typing import cast from .xform import Rtl, XForm # noqa from .ast import Expr # noqa from .typevar import TypeSet # noqa if TYPE_CHECKING: + T = MTypeVar('T') TypeMap = Dict[TypeVar, TypeVar] VarTyping = Dict[Var, TypeVar] except ImportError: @@ -775,6 +776,11 @@ def unify(tv1, tv2, typ): return typ +def move_first(l, i): + # type: (List[T], int) -> List[T] + return [l[i]] + l[:i] + l[i+1:] + + def ti_def(definition, typ): # type: (Def, TypeEnv) -> TypingOrError """ @@ -821,6 +827,12 @@ def ti_def(definition, typ): typ.register(v) actual_tvs.append(v.get_typevar()) + # Make sure we unify the control typevar first. + if inst.is_polymorphic: + idx = fresh_formal_tvs.index(m[inst.ctrl_typevar]) + fresh_formal_tvs = move_first(fresh_formal_tvs, idx) + actual_tvs = move_first(actual_tvs, idx) + # Unify each actual typevar with the correpsonding fresh formal tv for (actual_tv, formal_tv) in zip(actual_tvs, fresh_formal_tvs): typ_or_err = unify(actual_tv, formal_tv, typ) From 5f77369053cca1ce357980127684f0ef409f3b06 Mon Sep 17 00:00:00 2001 From: Dimo Date: Wed, 26 Jul 2017 15:26:59 -0700 Subject: [PATCH 0923/3084] Fix up a couple of test changed by unifying control tv first --- lib/cretonne/meta/cdsl/test_ti.py | 11 ++++++----- lib/cretonne/meta/test_gen_legalizer.py | 4 ++-- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/lib/cretonne/meta/cdsl/test_ti.py b/lib/cretonne/meta/cdsl/test_ti.py index 31d9a349a8..bffbbd527d 100644 --- a/lib/cretonne/meta/cdsl/test_ti.py +++ b/lib/cretonne/meta/cdsl/test_ti.py @@ -158,7 +158,8 @@ class TypeCheckingBaseTest(TestCase): self.v8 = Var("v8") self.v9 = Var("v9") self.imm0 = Var("imm0") - self.IxN = TypeVar("IxN", "", ints=True, scalars=True, simd=True) + self.IxN_nonscalar = TypeVar("IxN", "", ints=True, scalars=False, + simd=True) self.TxN = TypeVar("TxN", "", ints=True, bools=True, floats=True, scalars=False, simd=True) self.b1 = TypeVar.singleton(b1) @@ -175,7 +176,7 @@ class TestRTL(TypeCheckingBaseTest): self.assertEqual(ti_rtl(r, ti), "On line 1: fail ti on `typeof_v2` <: `1`: " + "Error: empty type created when unifying " + - "`typeof_v3` and `half_vector(typeof_v3)`") + "`typeof_v2` and `half_vector(typeof_v2)`") def test_vselect(self): # type: () -> None @@ -201,11 +202,11 @@ class TestRTL(TypeCheckingBaseTest): ) ti = TypeEnv() typing = ti_rtl(r, ti) - ixn = self.IxN.get_fresh_copy("IxN1") + ixn = self.IxN_nonscalar.get_fresh_copy("IxN1") txn = self.TxN.get_fresh_copy("TxN1") check_typing(typing, ({ self.v0: ixn, - self.v1: txn.as_bool(), + self.v1: ixn.as_bool(), self.v2: ixn, self.v3: txn, self.v4: txn, @@ -470,7 +471,7 @@ class TestXForm(TypeCheckingBaseTest): assert var_m[v0] == var_m[v2] and \ var_m[v3] == var_m[v4] and\ var_m[v5] == var_m[v3] and\ - var_m[v1] == var_m[v5].as_bool() and\ + var_m[v1] == var_m[v2].as_bool() and\ var_m[v1].get_typeset() == var_m[v3].as_bool().get_typeset() check_concrete_typing_xform(var_m, xform) diff --git a/lib/cretonne/meta/test_gen_legalizer.py b/lib/cretonne/meta/test_gen_legalizer.py index da4683413d..6a317d887b 100644 --- a/lib/cretonne/meta/test_gen_legalizer.py +++ b/lib/cretonne/meta/test_gen_legalizer.py @@ -153,7 +153,7 @@ class TestRuntimeChecks(TestCase): def test_vselect_imm(self): # type: () -> None - ts = TypeSet(lanes=(2, 256), ints=(8, 64)) + ts = TypeSet(lanes=(2, 256), ints=True, floats=True, bools=(8, 64)) r = Rtl( self.v0 << iconst(self.imm0), self.v1 << icmp(intcc.eq, self.v2, self.v0), @@ -166,7 +166,7 @@ class TestRuntimeChecks(TestCase): .format(self.v3.get_typevar().name) self.check_yo_check( - x, sequence(typeset_check(self.v2, ts), + x, sequence(typeset_check(self.v3, ts), equiv_check(tv2_exp, tv3_exp))) def test_reduce_extend(self): From 71f16466750d0e428da9cd01a60dea1f52544938 Mon Sep 17 00:00:00 2001 From: Dimo Date: Wed, 26 Jul 2017 15:28:52 -0700 Subject: [PATCH 0924/3084] With multiple semantic transforms mentioning Enumerators, it may be possible for there not to be a substitution from the concrete rtl to some of the transforms. This is not an error - just a case where a given semantic transform doesnt apply. (e.g. icmp being described by different transforms with concrete intcc condition codes) --- lib/cretonne/meta/semantics/elaborate.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/semantics/elaborate.py b/lib/cretonne/meta/semantics/elaborate.py index 03ca3195c4..0f2d10641d 100644 --- a/lib/cretonne/meta/semantics/elaborate.py +++ b/lib/cretonne/meta/semantics/elaborate.py @@ -83,7 +83,12 @@ def find_matching_xform(d): for x in d.expr.inst.semantics: subst = d.substitution(x.src.rtl[0], {}) - assert subst is not None + + # There may not be a substitution if there are concrete Enumerator + # values in the src pattern. (e.g. specifying the semantics of icmp.eq, + # icmp.ge... as separate transforms) + if (subst is None): + continue if x.ti.permits({subst[v]: tv for (v, tv) in typing.items()}): res.append(x) From ef4ca676a4f7b6a7aeafd6c81194288972b6bd63 Mon Sep 17 00:00:00 2001 From: Dimo Date: Wed, 26 Jul 2017 16:30:31 -0700 Subject: [PATCH 0925/3084] Move apply() -> Xform.apply(); is_concrete_rtl() -> Rtl.is_concrete(); cleanup_concrete_rtl() -> Rtl.cleanup_concrete_rtl(). Documnetation nits in semantics.elaborate --- lib/cretonne/meta/cdsl/xform.py | 54 +++++++++- lib/cretonne/meta/semantics/elaborate.py | 123 +++++++---------------- 2 files changed, 88 insertions(+), 89 deletions(-) diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index 2cae1c1236..f25068480c 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -5,6 +5,7 @@ from __future__ import absolute_import from .ast import Def, Var, Apply from .ti import ti_xform, TypeEnv, get_type_env from functools import reduce +from .typevar import TypeVar try: from typing import Union, Iterator, Sequence, Iterable, List, Dict # noqa @@ -12,7 +13,6 @@ try: from .ast import Expr, VarMap # noqa from .isa import TargetISA # noqa from .ti import TypeConstraint # noqa - from .typevar import TypeVar # noqa DefApply = Union[Def, Apply] except ImportError: pass @@ -86,6 +86,37 @@ class Rtl(object): return s + def is_concrete(self): + # type: (Rtl) -> bool + """Return True iff every Var in the self has a singleton type.""" + return all(v.get_typevar().singleton_type() is not None + for v in self.vars()) + + def cleanup_concrete_rtl(self): + # type: (Rtl) -> None + """ + Given that there is only 1 possible concrete typing T for self, assign + a singleton TV with the single type t=T[v] for each Var v \in self. + Its an error to call this on an Rtl with more than 1 possible typing. + """ + from .ti import ti_rtl, TypeEnv + # 1) Infer the types of all vars in res + typenv = get_type_env(ti_rtl(self, TypeEnv())) + typenv.normalize() + typenv = typenv.extract() + + # 2) Make sure there is only one possible type assignment + typings = list(typenv.concrete_typings()) + assert len(typings) == 1 + typing = typings[0] + + # 3) Assign the only possible type to each variable. + for v in typenv.vars: + if v.get_typevar().singleton_type() is not None: + continue + + v.set_typevar(TypeVar.singleton(typing[v].singleton_type())) + class XForm(object): """ @@ -279,6 +310,27 @@ class XForm(object): raise AssertionError( '{} not defined in dest pattern'.format(d)) + def apply(self, r, suffix=None): + # type: (Rtl, str) -> Rtl + """ + Given a concrete Rtl r s.t. r matches self.src, return the + corresponding concrete self.dst. If suffix is provided, any temporary + defs are renamed with '.suffix' appended to their old name. + """ + assert r.is_concrete() + s = self.src.substitution(r, {}) # type: VarMap + assert s is not None + + if (suffix is not None): + for v in self.dst.vars(): + if v.is_temp(): + assert v not in s + s[v] = Var(v.name + '.' + suffix) + + dst = self.dst.copy(s) + dst.cleanup_concrete_rtl() + return dst + class XFormGroup(object): """ diff --git a/lib/cretonne/meta/semantics/elaborate.py b/lib/cretonne/meta/semantics/elaborate.py index 0f2d10641d..4b7dc14ae3 100644 --- a/lib/cretonne/meta/semantics/elaborate.py +++ b/lib/cretonne/meta/semantics/elaborate.py @@ -4,8 +4,6 @@ equivalent primitive version. Its elaborated primitive version contains only primitive cretonne instructions, which map well to SMTLIB functions. """ from .primitives import GROUP as PRIMITIVES, prim_to_bv, prim_from_bv -from cdsl.ti import ti_rtl, TypeEnv, get_type_env -from cdsl.typevar import TypeVar from cdsl.xform import Rtl from cdsl.ast import Var @@ -18,60 +16,6 @@ except ImportError: TYPE_CHECKING = False -def is_rtl_concrete(r): - # type: (Rtl) -> bool - """Return True iff every Var in the Rtl r has a single type.""" - return all(v.get_typevar().singleton_type() is not None for v in r.vars()) - - -def cleanup_concrete_rtl(r): - # type: (Rtl) -> Rtl - """ - Given an Rtl r - 1) assert that there is only 1 possible concrete typing T for r - 2) Assign a singleton TV with the single type t \in T for each Var v \in r - """ - # 1) Infer the types of any of the remaining vars in res - typenv = get_type_env(ti_rtl(r, TypeEnv())) - typenv.normalize() - typenv = typenv.extract() - - # 2) Make sure there is only one possible type assignment - typings = list(typenv.concrete_typings()) - assert len(typings) == 1 - typing = typings[0] - - # 3) Assign the only possible type to each variable. - for v in typenv.vars: - if v.get_typevar().singleton_type() is not None: - continue - - v.set_typevar(TypeVar.singleton(typing[v].singleton_type())) - - return r - - -def apply(r, x, suffix=None): - # type: (Rtl, XForm, str) -> Rtl - """ - Given a concrete Rtl r and XForm x, s.t. r matches x.src, return the - corresponding concrete x.dst. If suffix is provided, any temporary defs are - renamed with '.suffix' appended to their old name. - """ - assert is_rtl_concrete(r) - s = x.src.substitution(r, {}) # type: VarMap - assert s is not None - - if (suffix is not None): - for v in x.dst.vars(): - if v.is_temp(): - assert v not in s - s[v] = Var(v.name + '.' + suffix) - - dst = x.dst.copy(s) - return cleanup_concrete_rtl(dst) - - def find_matching_xform(d): # type: (Def) -> XForm """ @@ -93,41 +37,10 @@ def find_matching_xform(d): if x.ti.permits({subst[v]: tv for (v, tv) in typing.items()}): res.append(x) - assert len(res) == 1 + assert len(res) == 1, "Couldn't find semantic transform for {}".format(d) return res[0] -def elaborate(r): - # type: (Rtl) -> Rtl - """ - Given an Rtl r, return a semantically equivalent Rtl r1 consisting only - primitive instructions. - """ - fp = False - primitives = set(PRIMITIVES.instructions) - idx = 0 - - while not fp: - assert is_rtl_concrete(r) - new_defs = [] # type: List[Def] - fp = True - - for d in r.rtl: - inst = d.expr.inst - - if (inst not in primitives): - transformed = apply(Rtl(d), find_matching_xform(d), str(idx)) - idx += 1 - new_defs.extend(transformed.rtl) - fp = False - else: - new_defs.append(d) - - r.rtl = tuple(new_defs) - - return r - - def cleanup_semantics(r, outputs): # type: (Rtl, Set[Var]) -> Rtl """ @@ -176,3 +89,37 @@ def cleanup_semantics(r, outputs): d.defs[0] not in live)] return Rtl(*new_defs) + + +def elaborate(r): + # type: (Rtl) -> Rtl + """ + Given a concrete Rtl r, return a semantically equivalent Rtl r1 containing + only primitive instructions. + """ + fp = False + primitives = set(PRIMITIVES.instructions) + idx = 0 + + outputs = r.definitions() + + while not fp: + assert r.is_concrete() + new_defs = [] # type: List[Def] + fp = True + + for d in r.rtl: + inst = d.expr.inst + + if (inst not in primitives): + t = find_matching_xform(d) + transformed = t.apply(Rtl(d), str(idx)) + idx += 1 + new_defs.extend(transformed.rtl) + fp = False + else: + new_defs.append(d) + + r.rtl = tuple(new_defs) + + return cleanup_semantics(r, outputs) From 11014ec5443de2f56492604dde7b7bf13d86a7ae Mon Sep 17 00:00:00 2001 From: Dimo Date: Wed, 26 Jul 2017 16:49:12 -0700 Subject: [PATCH 0926/3084] Nit: Make elaborate return a new Rtl instead of modifying the existing rtl inplace --- lib/cretonne/meta/semantics/elaborate.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/meta/semantics/elaborate.py b/lib/cretonne/meta/semantics/elaborate.py index 4b7dc14ae3..de59de35e5 100644 --- a/lib/cretonne/meta/semantics/elaborate.py +++ b/lib/cretonne/meta/semantics/elaborate.py @@ -101,14 +101,15 @@ def elaborate(r): primitives = set(PRIMITIVES.instructions) idx = 0 - outputs = r.definitions() + res = Rtl(*r.rtl) + outputs = res.definitions() while not fp: - assert r.is_concrete() + assert res.is_concrete() new_defs = [] # type: List[Def] fp = True - for d in r.rtl: + for d in res.rtl: inst = d.expr.inst if (inst not in primitives): @@ -120,6 +121,6 @@ def elaborate(r): else: new_defs.append(d) - r.rtl = tuple(new_defs) + res.rtl = tuple(new_defs) - return cleanup_semantics(r, outputs) + return cleanup_semantics(res, outputs) From 1ddee38895d8c8a05bf49271695ed0b7bc65b362 Mon Sep 17 00:00:00 2001 From: Dimo Date: Wed, 26 Jul 2017 19:10:14 -0700 Subject: [PATCH 0927/3084] cleanup_semantics() should remove repeated prim_from_bv(x) --- lib/cretonne/meta/semantics/elaborate.py | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/meta/semantics/elaborate.py b/lib/cretonne/meta/semantics/elaborate.py index de59de35e5..335c4ebb86 100644 --- a/lib/cretonne/meta/semantics/elaborate.py +++ b/lib/cretonne/meta/semantics/elaborate.py @@ -57,21 +57,30 @@ def cleanup_semantics(r, outputs): new_defs = [] # type: List[Def] subst_m = {v: v for v in r.vars()} # type: VarMap definition = {} # type: Dict[Var, Def] + prim_to_bv_map = {} # type: Dict[Var, Def] # Pass 1: Remove redundant prim_to_bv for d in r.rtl: inst = d.expr.inst if (inst == prim_to_bv): - if d.expr.args[0] in definition: - assert isinstance(d.expr.args[0], Var) - def_loc = definition[d.expr.args[0]] + arg = d.expr.args[0] + df = d.defs[0] + assert isinstance(arg, Var) + if arg in definition: + def_loc = definition[arg] if def_loc.expr.inst == prim_from_bv: assert isinstance(def_loc.expr.args[0], Var) - subst_m[d.defs[0]] = def_loc.expr.args[0] + subst_m[df] = def_loc.expr.args[0] continue + if arg in prim_to_bv_map: + subst_m[df] = prim_to_bv_map[arg].defs[0] + continue + + prim_to_bv_map[arg] = d + new_def = d.copy(subst_m) for v in new_def.defs: From 51ee47f01e51fbd152a5e3931310915a68833a77 Mon Sep 17 00:00:00 2001 From: Dimo Date: Wed, 26 Jul 2017 19:10:39 -0700 Subject: [PATCH 0928/3084] Add Rtl.free_vars() --- lib/cretonne/meta/cdsl/xform.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index f25068480c..1d74eaccf9 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -68,6 +68,17 @@ class Rtl(object): [d.definitions() for d in self.rtl], set([])) + def free_vars(self): + # type: () -> Set[Var] + """ Return the set of free Vars used in self""" + def flow_f(s, d): + # type: (Set[Var], Def): Set[Var] + """Compute the change in the set of free vars across a Def""" + s = s.difference(set(d.defs)) + return s.union(set(a for a in d.expr.args if isinstance(a, Var))) + + return reduce(flow_f, reversed(self.rtl), set([])) + def substitution(self, other, s): # type: (Rtl, VarMap) -> Optional[VarMap] """ From 42e0476cf403ecf943463d8849bddf7661ff4102 Mon Sep 17 00:00:00 2001 From: Dimo Date: Wed, 26 Jul 2017 19:12:11 -0700 Subject: [PATCH 0929/3084] Add primitive bvult, bvzeroext; Add semantics for bextend, icmp (partial - only for <) iadd_cout --- lib/cretonne/meta/base/semantics.py | 48 ++++++++++++++++++++++- lib/cretonne/meta/semantics/primitives.py | 16 ++++++++ 2 files changed, 62 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/meta/base/semantics.py b/lib/cretonne/meta/base/semantics.py index b4315d8255..cdf071f7d9 100644 --- a/lib/cretonne/meta/base/semantics.py +++ b/lib/cretonne/meta/base/semantics.py @@ -1,7 +1,8 @@ from __future__ import absolute_import from semantics.primitives import prim_to_bv, prim_from_bv, bvsplit, bvconcat,\ - bvadd -from .instructions import vsplit, vconcat, iadd + bvadd, bvult, bvzeroext +from .instructions import vsplit, vconcat, iadd, iadd_cout, icmp, bextend +from .immediates import intcc from cdsl.xform import Rtl from cdsl.ast import Var from cdsl.typevar import TypeSet @@ -10,6 +11,9 @@ from cdsl.ti import InTypeset x = Var('x') y = Var('y') a = Var('a') +b = Var('b') +c_out = Var('c_out') +bvc_out = Var('bvc_out') xhi = Var('xhi') yhi = Var('yhi') ahi = Var('ahi') @@ -21,6 +25,7 @@ hi = Var('hi') bvx = Var('bvx') bvy = Var('bvy') bva = Var('bva') +bva_wide = Var('bva_wide') bvlo = Var('bvlo') bvhi = Var('bvhi') @@ -56,3 +61,42 @@ iadd.set_semantics( alo << iadd(xlo, ylo), ahi << iadd(xhi, yhi), a << vconcat(alo, ahi))) + +iadd_cout.set_semantics( + (a, c_out) << iadd_cout(x, y), + Rtl( + bvx << prim_to_bv(x), + bvy << prim_to_bv(y), + bva << bvadd(bvx, bvy), + bvc_out << bvult(bva, bvx), + a << prim_from_bv(bva), + c_out << prim_from_bv(bvc_out) + )) + +bextend.set_semantics( + a << bextend(x), + (Rtl( + bvx << prim_to_bv(x), + bvy << bvzeroext(bvx), + a << prim_from_bv(bvy) + ), [InTypeset(x.get_typevar(), ScalarTS)]), + Rtl((xlo, xhi) << vsplit(x), + alo << bextend(xlo), + ahi << bextend(xhi), + a << vconcat(alo, ahi))) + +icmp.set_semantics( + a << icmp(intcc.ult, x, y), + (Rtl( + bvx << prim_to_bv(x), + bvy << prim_to_bv(y), + bva << bvult(bvx, bvy), + bva_wide << bvzeroext(bva), + a << prim_from_bv(bva_wide), + ), [InTypeset(x.get_typevar(), ScalarTS)]), + Rtl((xlo, xhi) << vsplit(x), + (ylo, yhi) << vsplit(y), + alo << icmp(intcc.ult, xlo, ylo), + ahi << icmp(intcc.ult, xhi, yhi), + b << vconcat(alo, ahi), + a << bextend(b))) diff --git a/lib/cretonne/meta/semantics/primitives.py b/lib/cretonne/meta/semantics/primitives.py index 70fc196d5d..62d936bc31 100644 --- a/lib/cretonne/meta/semantics/primitives.py +++ b/lib/cretonne/meta/semantics/primitives.py @@ -9,11 +9,13 @@ from __future__ import absolute_import from cdsl.operands import Operand from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup +from cdsl.ti import WiderOrEq import base.formats # noqa GROUP = InstructionGroup("primitive", "Primitive instruction set") BV = TypeVar('BV', 'A bitvector type.', bitvecs=True) +BV1 = TypeVar('BV1', 'A single bit bitvector.', bitvecs=(1, 1)) Real = TypeVar('Real', 'Any real type.', ints=True, floats=True, bools=True, simd=True) @@ -66,4 +68,18 @@ bvadd = Instruction( """, ins=(x, y), outs=a) +# Bitvector comparisons +cmp_res = Operand('cmp_res', BV1, doc="Single bit boolean") +bvult = Instruction( + 'bvult', r"""Unsigned bitvector comparison""", + ins=(x, y), outs=cmp_res) + +# Extensions +ToBV = TypeVar('ToBV', 'A bitvector type.', bitvecs=True) +x1 = Operand('x1', ToBV, doc="") + +bvzeroext = Instruction( + 'bvzeroext', r"""Unsigned bitvector extension""", + ins=x, outs=x1, constraints=WiderOrEq(ToBV, BV)) + GROUP.close() From eadb4cd39aefefc59a6eeb00b6aeff7fa4faa5f3 Mon Sep 17 00:00:00 2001 From: Dimo Date: Wed, 26 Jul 2017 19:12:41 -0700 Subject: [PATCH 0930/3084] Fix broken test_elaborate tests after the moving of is_concrete/cleanup_concrete_rtl --- lib/cretonne/meta/semantics/test_elaborate.py | 142 ++++++++++++------ 1 file changed, 96 insertions(+), 46 deletions(-) diff --git a/lib/cretonne/meta/semantics/test_elaborate.py b/lib/cretonne/meta/semantics/test_elaborate.py index 8eafcb9e00..cb798295b9 100644 --- a/lib/cretonne/meta/semantics/test_elaborate.py +++ b/lib/cretonne/meta/semantics/test_elaborate.py @@ -1,15 +1,15 @@ from __future__ import absolute_import from base.instructions import vselect, vsplit, vconcat, iconst, iadd, bint -from base.instructions import b1, icmp, ireduce +from base.instructions import b1, icmp, ireduce, iadd_cout from base.immediates import intcc from base.types import i64, i8, b32, i32, i16, f32 from cdsl.typevar import TypeVar from cdsl.ast import Var from cdsl.xform import Rtl from unittest import TestCase -from .elaborate import cleanup_concrete_rtl, elaborate, is_rtl_concrete,\ - cleanup_semantics -from .primitives import prim_to_bv, bvsplit, prim_from_bv, bvconcat, bvadd +from .elaborate import elaborate +from .primitives import prim_to_bv, bvsplit, prim_from_bv, bvconcat, bvadd, \ + bvult import base.semantics # noqa @@ -21,8 +21,8 @@ def concrete_rtls_eq(r1, r2): them) 2) Corresponding Vars between them have the same singleton type. """ - assert is_rtl_concrete(r1) - assert is_rtl_concrete(r2) + assert r1.is_concrete() + assert r2.is_concrete() s = r1.substitution(r2, {}) @@ -50,13 +50,14 @@ class TestCleanupConcreteRtl(TestCase): lo = Var('lo') hi = Var('hi') - x.set_typevar(TypeVar.singleton(typ)) r = Rtl( (lo, hi) << vsplit(x), ) - r1 = cleanup_concrete_rtl(r) - + r1 = r.copy({}) s = r.substitution(r1, {}) + + s[x].set_typevar(TypeVar.singleton(typ)) + r1.cleanup_concrete_rtl() assert s is not None assert s[x].get_typevar().singleton_type() == typ assert s[lo].get_typevar().singleton_type() == i64.by(2) @@ -72,20 +73,20 @@ class TestCleanupConcreteRtl(TestCase): ) with self.assertRaises(AssertionError): - cleanup_concrete_rtl(r) + r.cleanup_concrete_rtl() def test_cleanup_concrete_rtl_ireduce(self): # type: () -> None x = Var('x') y = Var('y') - x.set_typevar(TypeVar.singleton(i8.by(2))) r = Rtl( y << ireduce(x), ) - - r1 = cleanup_concrete_rtl(r) - + r1 = r.copy({}) s = r.substitution(r1, {}) + s[x].set_typevar(TypeVar.singleton(i8.by(2))) + r1.cleanup_concrete_rtl() + assert s is not None assert s[x].get_typevar().singleton_type() == i8.by(2) assert s[y].get_typevar().singleton_type() == i8.by(2) @@ -100,7 +101,7 @@ class TestCleanupConcreteRtl(TestCase): ) with self.assertRaises(AssertionError): - cleanup_concrete_rtl(r) + r.cleanup_concrete_rtl() def test_vselect_icmpimm(self): # type: () -> None @@ -112,18 +113,19 @@ class TestCleanupConcreteRtl(TestCase): zeroes = Var('zeroes') imm0 = Var("imm0") - zeroes.set_typevar(TypeVar.singleton(i32.by(4))) - z.set_typevar(TypeVar.singleton(f32.by(4))) - r = Rtl( zeroes << iconst(imm0), y << icmp(intcc.eq, x, zeroes), v << vselect(y, z, w), ) - - r1 = cleanup_concrete_rtl(r) + r1 = r.copy({}) s = r.substitution(r1, {}) + s[zeroes].set_typevar(TypeVar.singleton(i32.by(4))) + s[z].set_typevar(TypeVar.singleton(f32.by(4))) + + r1.cleanup_concrete_rtl() + assert s is not None assert s[zeroes].get_typevar().singleton_type() == i32.by(4) assert s[x].get_typevar().singleton_type() == i32.by(4) @@ -141,20 +143,20 @@ class TestCleanupConcreteRtl(TestCase): v = Var('v') u = Var('u') - x.set_typevar(TypeVar.singleton(i32.by(8))) - z.set_typevar(TypeVar.singleton(i32.by(8))) - # TODO: Relax this to simd=True - v.set_typevar(TypeVar('v', '', bools=(1, 1), simd=(8, 8))) - r = Rtl( z << iadd(x, y), w << bint(v), u << iadd(z, w) ) - - r1 = cleanup_concrete_rtl(r) - + r1 = r.copy({}) s = r.substitution(r1, {}) + + s[x].set_typevar(TypeVar.singleton(i32.by(8))) + s[z].set_typevar(TypeVar.singleton(i32.by(8))) + # TODO: Relax this to simd=True + s[v].set_typevar(TypeVar('v', '', bools=(1, 1), simd=(8, 8))) + r1.cleanup_concrete_rtl() + assert s is not None assert s[x].get_typevar().singleton_type() == i32.by(8) assert s[y].get_typevar().singleton_type() == i32.by(8) @@ -194,7 +196,8 @@ class TestElaborate(TestCase): r = Rtl( (self.v0, self.v1) << vsplit.i32x4(self.v2), ) - sem = elaborate(cleanup_concrete_rtl(r)) + r.cleanup_concrete_rtl() + sem = elaborate(r) bvx = Var('bvx') bvlo = Var('bvlo') bvhi = Var('bvhi') @@ -202,11 +205,15 @@ class TestElaborate(TestCase): lo = Var('lo') hi = Var('hi') - assert concrete_rtls_eq(sem, cleanup_concrete_rtl(Rtl( + exp = Rtl( bvx << prim_to_bv.i32x4(x), (bvlo, bvhi) << bvsplit.bv128(bvx), lo << prim_from_bv.i32x2(bvlo), - hi << prim_from_bv.i32x2(bvhi)))) + hi << prim_from_bv.i32x2(bvhi) + ) + exp.cleanup_concrete_rtl() + + assert concrete_rtls_eq(sem, exp) def test_elaborate_vconcat(self): # type: () -> None @@ -215,7 +222,8 @@ class TestElaborate(TestCase): r = Rtl( self.v0 << vconcat.i32x2(self.v1, self.v2), ) - sem = elaborate(cleanup_concrete_rtl(r)) + r.cleanup_concrete_rtl() + sem = elaborate(r) bvx = Var('bvx') bvlo = Var('bvlo') bvhi = Var('bvhi') @@ -223,11 +231,15 @@ class TestElaborate(TestCase): lo = Var('lo') hi = Var('hi') - assert concrete_rtls_eq(sem, cleanup_concrete_rtl(Rtl( + exp = Rtl( bvlo << prim_to_bv.i32x2(lo), bvhi << prim_to_bv.i32x2(hi), bvx << bvconcat.bv64(bvlo, bvhi), - x << prim_from_bv.i32x4(bvx)))) + x << prim_from_bv.i32x4(bvx) + ) + exp.cleanup_concrete_rtl() + + assert concrete_rtls_eq(sem, exp) def test_elaborate_iadd_simple(self): # type: () -> None @@ -241,13 +253,17 @@ class TestElaborate(TestCase): r = Rtl( a << iadd.i32(x, y), ) - sem = elaborate(cleanup_concrete_rtl(r)) - - assert concrete_rtls_eq(sem, cleanup_concrete_rtl(Rtl( + r.cleanup_concrete_rtl() + sem = elaborate(r) + exp = Rtl( bvx << prim_to_bv.i32(x), bvy << prim_to_bv.i32(y), bva << bvadd.bv32(bvx, bvy), - a << prim_from_bv.i32(bva)))) + a << prim_from_bv.i32(bva) + ) + exp.cleanup_concrete_rtl() + + assert concrete_rtls_eq(sem, exp) def test_elaborate_iadd_elaborate_1(self): # type: () -> None @@ -255,8 +271,8 @@ class TestElaborate(TestCase): r = Rtl( self.v0 << iadd.i32x2(self.v1, self.v2), ) - sem = cleanup_semantics(elaborate(cleanup_concrete_rtl(r)), - set([self.v0])) + r.cleanup_concrete_rtl() + sem = elaborate(r) x = Var('x') y = Var('y') a = Var('a') @@ -271,7 +287,7 @@ class TestElaborate(TestCase): bva_3 = Var('bva_3') bva_4 = Var('bva_4') - assert concrete_rtls_eq(sem, cleanup_concrete_rtl(Rtl( + exp = Rtl( bvx_1 << prim_to_bv.i32x2(x), (bvlo_1, bvhi_1) << bvsplit.bv64(bvx_1), bvx_2 << prim_to_bv.i32x2(y), @@ -279,7 +295,11 @@ class TestElaborate(TestCase): bva_3 << bvadd.bv32(bvlo_1, bvlo_2), bva_4 << bvadd.bv32(bvhi_1, bvhi_2), bvx_5 << bvconcat.bv32(bva_3, bva_4), - a << prim_from_bv.i32x2(bvx_5)))) + a << prim_from_bv.i32x2(bvx_5) + ) + exp.cleanup_concrete_rtl() + + assert concrete_rtls_eq(sem, exp) def test_elaborate_iadd_elaborate_2(self): # type: () -> None @@ -287,9 +307,9 @@ class TestElaborate(TestCase): r = Rtl( self.v0 << iadd.i8x4(self.v1, self.v2), ) + r.cleanup_concrete_rtl() - sem = cleanup_semantics(elaborate(cleanup_concrete_rtl(r)), - set([self.v0])) + sem = elaborate(r) x = Var('x') y = Var('y') a = Var('a') @@ -318,7 +338,7 @@ class TestElaborate(TestCase): bva_13 = Var('bva_13') bva_14 = Var('bva_14') - assert concrete_rtls_eq(sem, cleanup_concrete_rtl(Rtl( + exp = Rtl( bvx_1 << prim_to_bv.i8x4(x), (bvlo_1, bvhi_1) << bvsplit.bv32(bvx_1), bvx_2 << prim_to_bv.i8x4(y), @@ -334,4 +354,34 @@ class TestElaborate(TestCase): bva_14 << bvadd.bv8(bvhi_11, bvhi_12), bvx_15 << bvconcat.bv8(bva_13, bva_14), bvx_5 << bvconcat.bv16(bvx_10, bvx_15), - a << prim_from_bv.i8x4(bvx_5)))) + a << prim_from_bv.i8x4(bvx_5) + ) + exp.cleanup_concrete_rtl() + assert concrete_rtls_eq(sem, exp) + + def test_elaborate_iadd_cout_simple(self): + # type: () -> None + x = Var('x') + y = Var('y') + a = Var('a') + c_out = Var('c_out') + bvc_out = Var('bvc_out') + bvx = Var('bvx') + bvy = Var('bvy') + bva = Var('bva') + r = Rtl( + (a, c_out) << iadd_cout.i32(x, y), + ) + r.cleanup_concrete_rtl() + sem = elaborate(r) + exp = Rtl( + bvx << prim_to_bv.i32(x), + bvy << prim_to_bv.i32(y), + bva << bvadd.bv32(bvx, bvy), + bvc_out << bvult.bv32(bva, bvx), + a << prim_from_bv.i32(bva), + c_out << prim_from_bv.b1(bvc_out) + ) + exp.cleanup_concrete_rtl() + + assert concrete_rtls_eq(sem, exp) From b5e1e4d4545fecbfb6d63b7b7e4b68efceecdb84 Mon Sep 17 00:00:00 2001 From: Dimo Date: Wed, 26 Jul 2017 19:25:24 -0700 Subject: [PATCH 0931/3084] Add smtlib.py --- lib/cretonne/meta/cdsl/ast.py | 8 +- lib/cretonne/meta/cdsl/xform.py | 2 +- lib/cretonne/meta/semantics/smtlib.py | 150 ++++++++++++++++++++++++++ 3 files changed, 155 insertions(+), 5 deletions(-) create mode 100644 lib/cretonne/meta/semantics/smtlib.py diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index e954ab01e9..4207253684 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -156,19 +156,19 @@ class Var(Expr): Values that are defined only in the destination pattern. """ - def __init__(self, name): - # type: (str) -> None + def __init__(self, name, typevar=None): + # type: (str, TypeVar) -> None self.name = name # The `Def` defining this variable in a source pattern. self.src_def = None # type: Def # The `Def` defining this variable in a destination pattern. self.dst_def = None # type: Def # TypeVar representing the type of this variable. - self.typevar = None # type: TypeVar + self.typevar = typevar # type: TypeVar # The original 'typeof(x)' type variable that was created for this Var. # This one doesn't change. `self.typevar` above may be changed to # another typevar by type inference. - self.original_typevar = None # type: TypeVar + self.original_typevar = self.typevar # type: TypeVar def __str__(self): # type: () -> str diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index 1d74eaccf9..ecbb144e16 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -72,7 +72,7 @@ class Rtl(object): # type: () -> Set[Var] """ Return the set of free Vars used in self""" def flow_f(s, d): - # type: (Set[Var], Def): Set[Var] + # type: (Set[Var], Def) -> Set[Var] """Compute the change in the set of free vars across a Def""" s = s.difference(set(d.defs)) return s.union(set(a for a in d.expr.args if isinstance(a, Var))) diff --git a/lib/cretonne/meta/semantics/smtlib.py b/lib/cretonne/meta/semantics/smtlib.py new file mode 100644 index 0000000000..53dac682e5 --- /dev/null +++ b/lib/cretonne/meta/semantics/smtlib.py @@ -0,0 +1,150 @@ +""" +Tools to emit SMTLIB bitvector queries encoding concrete RTLs containing only +primitive instructions. +""" +from .primitives import GROUP as PRIMITIVES, prim_from_bv, prim_to_bv, bvadd,\ + bvult, bvzeroext +from cdsl.ast import Var +from cdsl.types import BVType + +try: + from typing import TYPE_CHECKING, Tuple # noqa + from cdsl.xform import Rtl # noqa + from cdsl.ast import VarMap # noqa +except ImportError: + TYPE_CHECKING = False + + +def bvtype_to_sort(typ): + # type: (BVType) -> str + """Return the BitVec sort corresponding to a BVType""" + return "(_ BitVec {})".format(typ.bits) + + +def to_smt(r): + # type: (Rtl) -> Tuple[str, VarMap] + """ + Encode a concrete primitive Rtl r sa SMTLIB 2.0 query. + Returns a tuple (query, var_m) where: + - query is the resulting query. + - var_m is a map from Vars v with non-BVType to their Vars v' with + BVType s.t. v' holds the flattend bitvector value of v. + """ + assert r.is_concrete() + # Should contain only primitives + primitives = set(PRIMITIVES.instructions) + assert all(d.expr.inst in primitives for d in r.rtl) + + q = "" + m = {} # type: VarMap + for v in r.vars(): + typ = v.get_typevar().singleton_type() + if not isinstance(typ, BVType): + continue + + q += "(declare-fun {} () {})\n".format(v.name, bvtype_to_sort(typ)) + + for d in r.rtl: + inst = d.expr.inst + + if inst == prim_to_bv: + assert isinstance(d.expr.args[0], Var) + m[d.expr.args[0]] = d.defs[0] + continue + + if inst == prim_from_bv: + assert isinstance(d.expr.args[0], Var) + m[d.defs[0]] = d.expr.args[0] + continue + + if inst in [bvadd, bvult]: # Binary instructions + assert len(d.expr.args) == 2 and len(d.defs) == 1 + lhs = d.expr.args[0] + rhs = d.expr.args[1] + df = d.defs[0] + assert isinstance(lhs, Var) and isinstance(rhs, Var) + + if inst in [bvadd]: # Normal binary - output type same as args + exp = "(= {} ({} {} {}))".format(df, inst.name, lhs, rhs) + else: + # Comparison binary - need to convert bool to BitVec 1 + exp = "(= {} (ite ({} {} {}) #b1 #b0))"\ + .format(df, inst.name, lhs, rhs) + elif inst == bvzeroext: + arg = d.expr.args[0] + df = d.defs[0] + assert isinstance(arg, Var) + fromW = arg.get_typevar().singleton_type().width() + toW = df.get_typevar().singleton_type().width() + + exp = "(= {} ((_ zero_extend {}) {}))"\ + .format(df, toW-fromW, arg, df) + else: + assert False, "Unknown primitive instruction {}".format(inst) + + q += "(assert {})\n".format(exp) + + return (q, m) + + +def equivalent(r1, r2, m): + # type: (Rtl, Rtl, VarMap) -> str + """ + Given concrete primitive Rtls r1 and r2, and a VarMap m, mapping all + non-primitive vars in r1 onto r2, return a query checking that the + two Rtls are semantically equivalent. + + If the returned query is unsatisfiable, then r1 and r2 are equivalent. + Otherwise, the satisfying example for the query gives us values + for which the two Rtls disagree. + """ + # Rename the vars in r1 and r2 to avoid conflicts + src_m = {v: Var(v.name + ".a", v.get_typevar()) for v in r1.vars()} + dst_m = {v: Var(v.name + ".b", v.get_typevar()) for v in r2.vars()} + m = {src_m[k]: dst_m[v] for (k, v) in m.items()} + + r1 = r1.copy(src_m) + r2 = r2.copy(dst_m) + + r1_nonprim_vars = set( + [v for v in r1.vars() + if not isinstance(v.get_typevar().singleton_type(), BVType)]) + + r2_nonprim_vars = set( + [v for v in r2.vars() + if not isinstance(v.get_typevar().singleton_type(), BVType)]) + + # Check that the map m maps all non real Cretone Vars from r1 onto r2 + assert r1_nonprim_vars == set(m.keys()) + assert r2_nonprim_vars == set(m.values()) + + (q1, m1) = to_smt(r1) + (q2, m2) = to_smt(r2) + + # Build an expression for the equality of real Cretone inputs + args_eq_exp = "(and \n" + + for v in r1.free_vars(): + assert v in r1_nonprim_vars + args_eq_exp += "(= {} {})\n".format(m1[v], m2[m[v]]) + args_eq_exp += ")" + + # Build an expression for the equality of real Cretone defs + results_eq_exp = "(and \n" + for v in r1.definitions(): + if (v not in r1_nonprim_vars): + continue + + results_eq_exp += "(= {} {})\n".format(m1[v], m2[m[v]]) + results_eq_exp += ")" + + q = '; Rtl 1 declarations and assertions\n' + q1 + q += '; Rtl 2 declarations and assertions\n' + q2 + + q += '; Assert that the inputs of Rtl1 and Rtl2 are equal\n' + \ + '(assert {})\n'.format(args_eq_exp) + + q += '; Assert that the outputs of Rtl1 and Rtl2 are not equal\n' + \ + '(assert (not {}))\n'.format(results_eq_exp) + + return q From 1bbe6440801721bbef558fb266e8a59fe4003df7 Mon Sep 17 00:00:00 2001 From: Dimo Date: Thu, 27 Jul 2017 17:16:26 -0700 Subject: [PATCH 0932/3084] Add semantics for several more iadd with carry; Add xform_correct() and doc cleanup --- lib/cretonne/meta/base/semantics.py | 103 +++++++++++++++++++++--- lib/cretonne/meta/semantics/smtlib.py | 110 +++++++++++++++++++------- 2 files changed, 171 insertions(+), 42 deletions(-) diff --git a/lib/cretonne/meta/base/semantics.py b/lib/cretonne/meta/base/semantics.py index cdf071f7d9..582fdc0889 100644 --- a/lib/cretonne/meta/base/semantics.py +++ b/lib/cretonne/meta/base/semantics.py @@ -1,7 +1,8 @@ from __future__ import absolute_import from semantics.primitives import prim_to_bv, prim_from_bv, bvsplit, bvconcat,\ bvadd, bvult, bvzeroext -from .instructions import vsplit, vconcat, iadd, iadd_cout, icmp, bextend +from .instructions import vsplit, vconcat, iadd, iadd_cout, icmp, bextend, \ + isplit, iconcat, iadd_cin, iadd_carry from .immediates import intcc from cdsl.xform import Rtl from cdsl.ast import Var @@ -13,18 +14,24 @@ y = Var('y') a = Var('a') b = Var('b') c_out = Var('c_out') +c_in = Var('c_in') bvc_out = Var('bvc_out') +bvc_in = Var('bvc_in') xhi = Var('xhi') yhi = Var('yhi') ahi = Var('ahi') +bhi = Var('bhi') xlo = Var('xlo') ylo = Var('ylo') alo = Var('alo') +blo = Var('blo') lo = Var('lo') hi = Var('hi') bvx = Var('bvx') bvy = Var('bvy') bva = Var('bva') +bvt = Var('bvt') +bvs = Var('bvs') bva_wide = Var('bva_wide') bvlo = Var('bvlo') bvhi = Var('bvhi') @@ -51,16 +58,34 @@ vconcat.set_semantics( iadd.set_semantics( a << iadd(x, y), - (Rtl(bvx << prim_to_bv(x), - bvy << prim_to_bv(y), - bva << bvadd(bvx, bvy), - a << prim_from_bv(bva)), - [InTypeset(x.get_typevar(), ScalarTS)]), - Rtl((xlo, xhi) << vsplit(x), + (Rtl( + bvx << prim_to_bv(x), + bvy << prim_to_bv(y), + bva << bvadd(bvx, bvy), + a << prim_from_bv(bva) + ), [InTypeset(x.get_typevar(), ScalarTS)]), + Rtl( + (xlo, xhi) << vsplit(x), (ylo, yhi) << vsplit(y), alo << iadd(xlo, ylo), ahi << iadd(xhi, yhi), - a << vconcat(alo, ahi))) + a << vconcat(alo, ahi) + )) + +# +# Integer arithmetic with carry and/or borrow. +# +iadd_cin.set_semantics( + a << iadd_cin(x, y, c_in), + Rtl( + bvx << prim_to_bv(x), + bvy << prim_to_bv(y), + bvc_in << prim_to_bv(c_in), + bvs << bvzeroext(bvc_in), + bvt << bvadd(bvx, bvy), + bva << bvadd(bvt, bvs), + a << prim_from_bv(bva) + )) iadd_cout.set_semantics( (a, c_out) << iadd_cout(x, y), @@ -73,6 +98,20 @@ iadd_cout.set_semantics( c_out << prim_from_bv(bvc_out) )) +iadd_carry.set_semantics( + (a, c_out) << iadd_carry(x, y, c_in), + Rtl( + bvx << prim_to_bv(x), + bvy << prim_to_bv(y), + bvc_in << prim_to_bv(c_in), + bvs << bvzeroext(bvc_in), + bvt << bvadd(bvx, bvy), + bva << bvadd(bvt, bvs), + bvc_out << bvult(bva, bvx), + a << prim_from_bv(bva), + c_out << prim_from_bv(bvc_out) + )) + bextend.set_semantics( a << bextend(x), (Rtl( @@ -80,10 +119,12 @@ bextend.set_semantics( bvy << bvzeroext(bvx), a << prim_from_bv(bvy) ), [InTypeset(x.get_typevar(), ScalarTS)]), - Rtl((xlo, xhi) << vsplit(x), + Rtl( + (xlo, xhi) << vsplit(x), alo << bextend(xlo), ahi << bextend(xhi), - a << vconcat(alo, ahi))) + a << vconcat(alo, ahi) + )) icmp.set_semantics( a << icmp(intcc.ult, x, y), @@ -94,9 +135,47 @@ icmp.set_semantics( bva_wide << bvzeroext(bva), a << prim_from_bv(bva_wide), ), [InTypeset(x.get_typevar(), ScalarTS)]), - Rtl((xlo, xhi) << vsplit(x), + Rtl( + (xlo, xhi) << vsplit(x), (ylo, yhi) << vsplit(y), alo << icmp(intcc.ult, xlo, ylo), ahi << icmp(intcc.ult, xhi, yhi), b << vconcat(alo, ahi), - a << bextend(b))) + a << bextend(b) + )) + +# +# Legalization helper instructions. +# + +isplit.set_semantics( + (xlo, xhi) << isplit(x), + (Rtl( + bvx << prim_to_bv(x), + (bvlo, bvhi) << bvsplit(bvx), + xlo << prim_from_bv(bvlo), + xhi << prim_from_bv(bvhi) + ), [InTypeset(x.get_typevar(), ScalarTS)]), + Rtl( + (a, b) << vsplit(x), + (alo, ahi) << isplit(a), + (blo, bhi) << isplit(b), + xlo << vconcat(alo, blo), + xhi << vconcat(bhi, bhi) + )) + +iconcat.set_semantics( + x << iconcat(xlo, xhi), + (Rtl( + bvlo << prim_to_bv(xlo), + bvhi << prim_to_bv(xhi), + bvx << bvconcat(bvlo, bvhi), + x << prim_from_bv(bvx) + ), [InTypeset(x.get_typevar(), ScalarTS)]), + Rtl( + (alo, ahi) << vsplit(xlo), + (blo, bhi) << vsplit(xhi), + a << iconcat(alo, blo), + b << iconcat(ahi, bhi), + x << vconcat(a, b), + )) diff --git a/lib/cretonne/meta/semantics/smtlib.py b/lib/cretonne/meta/semantics/smtlib.py index 53dac682e5..2bf94515ab 100644 --- a/lib/cretonne/meta/semantics/smtlib.py +++ b/lib/cretonne/meta/semantics/smtlib.py @@ -3,9 +3,10 @@ Tools to emit SMTLIB bitvector queries encoding concrete RTLs containing only primitive instructions. """ from .primitives import GROUP as PRIMITIVES, prim_from_bv, prim_to_bv, bvadd,\ - bvult, bvzeroext + bvult, bvzeroext, bvsplit, bvconcat from cdsl.ast import Var from cdsl.types import BVType +from .elaborate import elaborate try: from typing import TYPE_CHECKING, Tuple # noqa @@ -78,7 +79,32 @@ def to_smt(r): toW = df.get_typevar().singleton_type().width() exp = "(= {} ((_ zero_extend {}) {}))"\ - .format(df, toW-fromW, arg, df) + .format(df, toW-fromW, arg) + elif inst == bvsplit: + arg = d.expr.args[0] + arg_typ = arg.get_typevar().singleton_type() + width = arg_typ.width() + assert (width % 2 == 0) + + lo = d.defs[0] + hi = d.defs[1] + assert isinstance(arg, Var) + + exp = "(and " + exp += "(= {} ((_ extract {} {}) {})) "\ + .format(lo, width//2-1, 0, arg) + exp += "(= {} ((_ extract {} {}) {}))"\ + .format(hi, width-1, width//2, arg) + exp += ")" + elif inst == bvconcat: + lo = d.expr.args[0] + hi = d.expr.args[1] + assert isinstance(lo, Var) and isinstance(hi, Var) + df = d.defs[0] + + # Z3 Concat expects hi bits first, then lo bits + exp = "(= {} (concat {} {}))"\ + .format(df, hi, lo) else: assert False, "Unknown primitive instruction {}".format(inst) @@ -87,57 +113,49 @@ def to_smt(r): return (q, m) -def equivalent(r1, r2, m): - # type: (Rtl, Rtl, VarMap) -> str +def equivalent(r1, r2, inp_m, out_m): + # type: (Rtl, Rtl, VarMap, VarMap) -> str """ - Given concrete primitive Rtls r1 and r2, and a VarMap m, mapping all - non-primitive vars in r1 onto r2, return a query checking that the - two Rtls are semantically equivalent. + Given: + - concrete source Rtl r1 + - concrete dest Rtl r2 + - VarMap inp_m mapping r1's non-bitvector inputs to r2 + - VarMap out_m mapping r1's non-bitvector outputs to r2 + Build a query checking whether r1 and r2 are semantically equivalent. If the returned query is unsatisfiable, then r1 and r2 are equivalent. Otherwise, the satisfying example for the query gives us values for which the two Rtls disagree. """ - # Rename the vars in r1 and r2 to avoid conflicts + # Rename the vars in r1 and r2 with unique suffixes to avoid conflicts src_m = {v: Var(v.name + ".a", v.get_typevar()) for v in r1.vars()} dst_m = {v: Var(v.name + ".b", v.get_typevar()) for v in r2.vars()} - m = {src_m[k]: dst_m[v] for (k, v) in m.items()} - r1 = r1.copy(src_m) r2 = r2.copy(dst_m) - r1_nonprim_vars = set( - [v for v in r1.vars() - if not isinstance(v.get_typevar().singleton_type(), BVType)]) - - r2_nonprim_vars = set( - [v for v in r2.vars() - if not isinstance(v.get_typevar().singleton_type(), BVType)]) - - # Check that the map m maps all non real Cretone Vars from r1 onto r2 - assert r1_nonprim_vars == set(m.keys()) - assert r2_nonprim_vars == set(m.values()) + # Convert inp_m, out_m in terms of variables with the .a/.b suffixes + inp_m = {src_m[k]: dst_m[v] for (k, v) in inp_m.items()} + out_m = {src_m[k]: dst_m[v] for (k, v) in out_m.items()} + # Encode r1 and r2 as SMT queries (q1, m1) = to_smt(r1) (q2, m2) = to_smt(r2) - # Build an expression for the equality of real Cretone inputs + # Build an expression for the equality of real Cretone inputs of r1 and r2 args_eq_exp = "(and \n" for v in r1.free_vars(): - assert v in r1_nonprim_vars - args_eq_exp += "(= {} {})\n".format(m1[v], m2[m[v]]) + assert v in inp_m + args_eq_exp += "(= {} {})\n".format(m1[v], m2[inp_m[v]]) args_eq_exp += ")" - # Build an expression for the equality of real Cretone defs + # Build an expression for the equality of real Cretone outputs of r1 and r2 results_eq_exp = "(and \n" - for v in r1.definitions(): - if (v not in r1_nonprim_vars): - continue - - results_eq_exp += "(= {} {})\n".format(m1[v], m2[m[v]]) + for (v1, v2) in out_m.items(): + results_eq_exp += "(= {} {})\n".format(m1[v1], m2[v2]) results_eq_exp += ")" + # Put the whole query toghether q = '; Rtl 1 declarations and assertions\n' + q1 q += '; Rtl 2 declarations and assertions\n' + q2 @@ -148,3 +166,35 @@ def equivalent(r1, r2, m): '(assert (not {}))\n'.format(results_eq_exp) return q + + +def xform_correct(x, typing): + # type: (XForm, VarTyping) -> str + """ + Given an XForm x and a concrete variable typing for x typing, build the + smtlib query asserting that x is correct for the given typing. + """ + assert x.ti.permits(typing) + + # Create copies of the x.src and x.dst with the concrete types in typing. + src_m = {v: Var(v.name, typing[v]) for v in x.src.vars()} + src = x.src.copy(src_m) + dst = x.apply(src) + dst_m = x.dst.substitution(dst, {}) + + # Build maps for the inputs/outputs for src->dst + inp_m = {} + out_m = {} + + for v in x.src.vars(): + if v.is_input(): + inp_m[src_m[v]] = dst_m[v] + elif v.is_output(): + out_m[src_m[v]] = dst_m[v] + else: + assert False, "Haven't decided what to do with intermediates yet" + + # Get the primitive semantic Rtls for src and dst + prim_src = elaborate(src) + prim_dst = elaborate(dst) + return equivalent(prim_src, prim_dst, inp_m, out_m) From ede02e0f97f646288180b3643c67287ce84e4abb Mon Sep 17 00:00:00 2001 From: Dimo Date: Thu, 27 Jul 2017 18:05:38 -0700 Subject: [PATCH 0933/3084] Cleanup, typechecking and documentation nits --- lib/cretonne/meta/cdsl/xform.py | 22 ++++++++++------- lib/cretonne/meta/semantics/elaborate.py | 13 +++++++--- lib/cretonne/meta/semantics/smtlib.py | 31 ++++++++++++++++-------- 3 files changed, 43 insertions(+), 23 deletions(-) diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index ecbb144e16..991e429b18 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -5,7 +5,6 @@ from __future__ import absolute_import from .ast import Def, Var, Apply from .ti import ti_xform, TypeEnv, get_type_env from functools import reduce -from .typevar import TypeVar try: from typing import Union, Iterator, Sequence, Iterable, List, Dict # noqa @@ -13,6 +12,7 @@ try: from .ast import Expr, VarMap # noqa from .isa import TargetISA # noqa from .ti import TypeConstraint # noqa + from .typevar import TypeVar # noqa DefApply = Union[Def, Apply] except ImportError: pass @@ -70,12 +70,17 @@ class Rtl(object): def free_vars(self): # type: () -> Set[Var] - """ Return the set of free Vars used in self""" + """Return the set of free Vars corresp. to SSA vals used in self""" def flow_f(s, d): # type: (Set[Var], Def) -> Set[Var] """Compute the change in the set of free vars across a Def""" s = s.difference(set(d.defs)) - return s.union(set(a for a in d.expr.args if isinstance(a, Var))) + uses = set(d.expr.args[i] for i in d.expr.inst.value_opnums) + for v in uses: + assert isinstance(v, Var) + s.add(v) + + return s return reduce(flow_f, reversed(self.rtl), set([])) @@ -107,8 +112,9 @@ class Rtl(object): # type: (Rtl) -> None """ Given that there is only 1 possible concrete typing T for self, assign - a singleton TV with the single type t=T[v] for each Var v \in self. - Its an error to call this on an Rtl with more than 1 possible typing. + a singleton TV with type t=T[v] for each Var v \in self. Its an error + to call this on an Rtl with more than 1 possible typing. This modifies + the Rtl in-place. """ from .ti import ti_rtl, TypeEnv # 1) Infer the types of all vars in res @@ -123,10 +129,8 @@ class Rtl(object): # 3) Assign the only possible type to each variable. for v in typenv.vars: - if v.get_typevar().singleton_type() is not None: - continue - - v.set_typevar(TypeVar.singleton(typing[v].singleton_type())) + assert typing[v].singleton_type() is not None + v.set_typevar(typing[v]) class XForm(object): diff --git a/lib/cretonne/meta/semantics/elaborate.py b/lib/cretonne/meta/semantics/elaborate.py index 335c4ebb86..fc0ca98cc4 100644 --- a/lib/cretonne/meta/semantics/elaborate.py +++ b/lib/cretonne/meta/semantics/elaborate.py @@ -44,15 +44,20 @@ def find_matching_xform(d): def cleanup_semantics(r, outputs): # type: (Rtl, Set[Var]) -> Rtl """ - The elaboration process creates a lot of redundant instruction pairs of the - shape: + The elaboration process creates a lot of redundant prim_to_bv conversions. + Cleanup the following cases: + 1) prim_to_bv/prim_from_bv pair: a.0 << prim_from_bv(bva.0) ... - bva.1 << prim_to_bv(a.0) + bva.1 << prim_to_bv(a.0) <-- redundant, replace by bva.0 ... - Contract these to ease manual inspection. + 2) prim_to_bv/prim_to-bv pair: + bva.0 << prim_to_bv(a) + ... + bva.1 << prim_to_bv(a) <-- redundant, replace by bva.0 + ... """ new_defs = [] # type: List[Def] subst_m = {v: v for v in r.vars()} # type: VarMap diff --git a/lib/cretonne/meta/semantics/smtlib.py b/lib/cretonne/meta/semantics/smtlib.py index 2bf94515ab..f84176dc3c 100644 --- a/lib/cretonne/meta/semantics/smtlib.py +++ b/lib/cretonne/meta/semantics/smtlib.py @@ -10,8 +10,9 @@ from .elaborate import elaborate try: from typing import TYPE_CHECKING, Tuple # noqa - from cdsl.xform import Rtl # noqa + from cdsl.xform import Rtl, XForm # noqa from cdsl.ast import VarMap # noqa + from cdsl.ti import VarTyping # noqa except ImportError: TYPE_CHECKING = False @@ -34,10 +35,12 @@ def to_smt(r): assert r.is_concrete() # Should contain only primitives primitives = set(PRIMITIVES.instructions) - assert all(d.expr.inst in primitives for d in r.rtl) + assert set(d.expr.inst for d in r.rtl).issubset(primitives) q = "" m = {} # type: VarMap + + # Build declarations for any bitvector Vars for v in r.vars(): typ = v.get_typevar().singleton_type() if not isinstance(typ, BVType): @@ -45,9 +48,11 @@ def to_smt(r): q += "(declare-fun {} () {})\n".format(v.name, bvtype_to_sort(typ)) + # Encode each instruction as a equality assertion for d in r.rtl: inst = d.expr.inst + # For prim_to_bv/prim_from_bv just update var_m. No assertion needed if inst == prim_to_bv: assert isinstance(d.expr.args[0], Var) m[d.expr.args[0]] = d.defs[0] @@ -82,13 +87,13 @@ def to_smt(r): .format(df, toW-fromW, arg) elif inst == bvsplit: arg = d.expr.args[0] + assert isinstance(arg, Var) arg_typ = arg.get_typevar().singleton_type() width = arg_typ.width() assert (width % 2 == 0) lo = d.defs[0] hi = d.defs[1] - assert isinstance(arg, Var) exp = "(and " exp += "(= {} ((_ extract {} {}) {})) "\ @@ -97,9 +102,10 @@ def to_smt(r): .format(hi, width-1, width//2, arg) exp += ")" elif inst == bvconcat: + assert isinstance(d.expr.args[0], Var) and \ + isinstance(d.expr.args[1], Var) lo = d.expr.args[0] hi = d.expr.args[1] - assert isinstance(lo, Var) and isinstance(hi, Var) df = d.defs[0] # Z3 Concat expects hi bits first, then lo bits @@ -127,6 +133,14 @@ def equivalent(r1, r2, inp_m, out_m): Otherwise, the satisfying example for the query gives us values for which the two Rtls disagree. """ + # Sanity - inp_m is a bijection from the set of inputs of r1 to the set of + # inputs of r2 + assert set(r1.free_vars()) == set(inp_m.keys()) + assert set(r2.free_vars()) == set(inp_m.values()) + + # Note that the same rule is not expected to hold for out_m due to + # temporaries/intermediates. + # Rename the vars in r1 and r2 with unique suffixes to avoid conflicts src_m = {v: Var(v.name + ".a", v.get_typevar()) for v in r1.vars()} dst_m = {v: Var(v.name + ".b", v.get_typevar()) for v in r2.vars()} @@ -145,7 +159,6 @@ def equivalent(r1, r2, inp_m, out_m): args_eq_exp = "(and \n" for v in r1.free_vars(): - assert v in inp_m args_eq_exp += "(= {} {})\n".format(m1[v], m2[inp_m[v]]) args_eq_exp += ")" @@ -171,12 +184,12 @@ def equivalent(r1, r2, inp_m, out_m): def xform_correct(x, typing): # type: (XForm, VarTyping) -> str """ - Given an XForm x and a concrete variable typing for x typing, build the - smtlib query asserting that x is correct for the given typing. + Given an XForm x and a concrete variable typing for x build the smtlib + query asserting that x is correct for the given typing. """ assert x.ti.permits(typing) - # Create copies of the x.src and x.dst with the concrete types in typing. + # Create copies of the x.src and x.dst with their concrete types src_m = {v: Var(v.name, typing[v]) for v in x.src.vars()} src = x.src.copy(src_m) dst = x.apply(src) @@ -191,8 +204,6 @@ def xform_correct(x, typing): inp_m[src_m[v]] = dst_m[v] elif v.is_output(): out_m[src_m[v]] = dst_m[v] - else: - assert False, "Haven't decided what to do with intermediates yet" # Get the primitive semantic Rtls for src and dst prim_src = elaborate(src) From b5076108c19b6af5aa24b1da83bbfcb7fc57f364 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 28 Jul 2017 08:57:32 -0700 Subject: [PATCH 0934/3084] Support constant integers in AST expressions. Make it possible to write AST nodes: iconst.i32(imm64(0)). --- lib/cretonne/meta/cdsl/ast.py | 36 ++++++++++++++++++++++++++++-- lib/cretonne/meta/cdsl/operands.py | 16 ++++++++++++- 2 files changed, 49 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index 4207253684..0a9b5eece4 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -350,8 +350,8 @@ class Apply(Expr): on this instruction. Immediate operands in a source pattern can be either free variables or - constants like `Enumerator`. We don't currently support constraints on - free variables, but we may in the future. + constants like `ConstantInt` and `Enumerator`. We don't currently + support constraints on free variables, but we may in the future. """ pred = None # type: PredNode iform = self.inst.format @@ -415,6 +415,12 @@ class Apply(Expr): else: if (s[self_a] != other_a): return None + elif isinstance(self_a, ConstantInt): + if not isinstance(other_a, ConstantInt): + return None + assert self_a.kind == other_a.kind + if (self_a.value != other_a.value): + return None else: assert isinstance(self_a, Enumerator) @@ -430,6 +436,32 @@ class Apply(Expr): return s +class ConstantInt(Expr): + """ + A value of an integer immediate operand. + + Immediate operands like `imm64` or `offset32` can be specified in AST + expressions using the call syntax: `imm64(5)` which greates a `ConstantInt` + node. + """ + + def __init__(self, kind, value): + # type: (ImmediateKind, int) -> None + self.kind = kind + self.value = value + + def __str__(self): + # type: () -> str + """ + Get the Rust expression form of this constant. + """ + return str(self.value) + + def __repr__(self): + # type: () -> str + return '{}({})'.format(self.kind, self.value) + + class Enumerator(Expr): """ A value of an enumerated immediate operand. diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index 44bae5d500..abf409a8c4 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -8,7 +8,7 @@ try: from typing import Union, Dict, TYPE_CHECKING # noqa OperandSpec = Union['OperandKind', ValueType, TypeVar] if TYPE_CHECKING: - from .ast import Enumerator # noqa + from .ast import Enumerator, ConstantInt # noqa except ImportError: pass @@ -107,6 +107,20 @@ class ImmediateKind(OperandKind): n=self.name, a=value)) return Enumerator(self, value) + def __call__(self, value): + # type: (int) -> ConstantInt + """ + Create an AST node representing a constant integer: + + iconst(imm64(0)) + """ + from .ast import ConstantInt # noqa + if self.values: + raise AssertionError( + "{}({}): Can't make a constant numeric value for an enum" + .format(self.name, value)) + return ConstantInt(self, value) + def rust_enumerator(self, value): # type: (str) -> str """ From 1f02d1f8804945c7db351e90b49d11e89038dfc7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 28 Jul 2017 11:06:09 -0700 Subject: [PATCH 0935/3084] Allow for multiple legalization patterns for the same opcode. Each input pattern can have a predicate in addition to an opcode being matched. When an opcode has multiple patterns, execute the first pattern with a true predicate. The predicates can be type checks or instruction predicates checking immediate fields. --- lib/cretonne/meta/gen_legalizer.py | 112 +++++++++++++----------- lib/cretonne/meta/test_gen_legalizer.py | 32 +++---- 2 files changed, 73 insertions(+), 71 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 10c5bcf3ac..bee51a928a 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -9,6 +9,7 @@ the input instruction. """ from __future__ import absolute_import from srcgen import Formatter +from collections import defaultdict from base import instructions from cdsl.ast import Var from cdsl.ti import ti_rtl, TypeEnv, get_type_env, TypesEqual,\ @@ -18,7 +19,7 @@ from gen_instr import gen_typesets_table from cdsl.typevar import TypeVar try: - from typing import Sequence, List, Dict, Set # noqa + from typing import Sequence, List, Dict, Set, DefaultDict # noqa from cdsl.isa import TargetISA # noqa from cdsl.ast import Def # noqa from cdsl.xform import XForm, XFormGroup # noqa @@ -78,6 +79,11 @@ def emit_runtime_typecheck(check, fmt, type_sets): # type: (TypeConstraint, Formatter, UniqueTable) -> None """ Emit rust code for the given check. + + The emitted code is a statement redefining the `predicate` variable like + this: + + let predicate = predicate && ... """ def build_derived_expr(tv): # type: (TypeVar) -> str @@ -116,33 +122,26 @@ def emit_runtime_typecheck(check, fmt, type_sets): if check.ts not in type_sets.index: type_sets.add(check.ts) ts = type_sets.index[check.ts] - fmt.comment("{} must belong to {}".format(tv, check.ts)) - with fmt.indented('if !TYPE_SETS[{}].contains({}) {{'.format(ts, tv), - '};'): - fmt.line('return false;') + fmt.format( + 'let predicate = predicate && TYPE_SETS[{}].contains({});', + ts, tv) elif (isinstance(check, TypesEqual)): - with fmt.indented('{', '};'): - fmt.line('let a = {};'.format(build_derived_expr(check.tv1))) - fmt.line('let b = {};'.format(build_derived_expr(check.tv2))) - - fmt.comment('On overflow constraint doesn\'t appply') - with fmt.indented('if a.is_none() || b.is_none() {', '};'): - fmt.line('return false;') - - with fmt.indented('if a != b {', '};'): - fmt.line('return false;') + with fmt.indented( + 'let predicate = predicate && match ({}, {}) {{' + .format(build_derived_expr(check.tv1), + build_derived_expr(check.tv2)), '};'): + fmt.line('(Some(a), Some(b)) => a == b,') + fmt.comment('On overflow, constraint doesn\'t appply') + fmt.line('_ => false,') elif (isinstance(check, WiderOrEq)): - with fmt.indented('{', '};'): - fmt.line('let a = {};'.format(build_derived_expr(check.tv1))) - fmt.line('let b = {};'.format(build_derived_expr(check.tv2))) - - fmt.comment('On overflow constraint doesn\'t appply') - with fmt.indented('if a.is_none() || b.is_none() {', '};'): - fmt.line('return false;') - - with fmt.indented('if !a.wider_or_equal(b) {', '};'): - fmt.line('return false;') + with fmt.indented( + 'let predicate = predicate && match ({}, {}) {{' + .format(build_derived_expr(check.tv1), + build_derived_expr(check.tv2)), '};'): + fmt.line('(Some(a), Some(b)) => a.wider_or_equal(b),') + fmt.comment('On overflow, constraint doesn\'t appply') + fmt.line('_ => false,') else: assert False, "Unknown check {}".format(check) @@ -216,14 +215,12 @@ def unwrap_inst(iref, node, fmt): replace_inst = True else: # Boring case: Detach the result values, capture them in locals. - fmt.comment('Detaching results.') for d in node.defs: fmt.line('let {};'.format(d)) with fmt.indented('{', '}'): fmt.line('let r = dfg.inst_results(inst);') for i in range(len(node.defs)): fmt.line('{} = r[{}];'.format(node.defs[i], i)) - fmt.line('dfg.clear_results(inst);') for d in node.defs: if d.has_free_typevar(): fmt.line( @@ -312,7 +309,7 @@ def emit_dst_inst(node, fmt): def gen_xform(xform, fmt, type_sets): # type: (XForm, Formatter, UniqueTable) -> None """ - Emit code for `xform`, assuming the the opcode of xform's root instruction + Emit code for `xform`, assuming that the opcode of xform's root instruction has already been matched. `inst: Inst` is the variable to be replaced. It is pointed to by `pos: @@ -323,24 +320,33 @@ def gen_xform(xform, fmt, type_sets): # variables. replace_inst = unwrap_inst('inst', xform.src.rtl[0], fmt) - # We could support instruction predicates, but not yet. Should we just - # return false if it fails? What about multiple patterns with different - # predicates for the same opcode? + # Check instruction predicate and emit type checks. instp = xform.src.rtl[0].expr.inst_predicate() - assert instp is None, "Instruction predicates not supported in legalizer" + # TODO: The instruction predicate should be evaluated with all the inst + # immediate fields available. Probably by unwrap_inst(). + fmt.format('let predicate = {};', + instp.rust_predicate(0) if instp else 'true') # Emit any runtime checks. for check in get_runtime_typechecks(xform): emit_runtime_typecheck(check, fmt, type_sets) - # Emit the destination pattern. - for dst in xform.dst.rtl: - emit_dst_inst(dst, fmt) + # Guard the actual expansion by `predicate`. + with fmt.indented('if predicate {', '}'): + # If we're going to delete `inst`, we need to detach its results first + # so they can be reattached during pattern expansion. + if not replace_inst: + fmt.line('dfg.clear_results(inst);') - # Delete the original instruction if we didn't have an opportunity to - # replace it. - if not replace_inst: - fmt.line('assert_eq!(pos.remove_inst(), inst);') + # Emit the destination pattern. + for dst in xform.dst.rtl: + emit_dst_inst(dst, fmt) + + # Delete the original instruction if we didn't have an opportunity to + # replace it. + if not replace_inst: + fmt.line('assert_eq!(pos.remove_inst(), inst);') + fmt.line('return true;') def gen_xform_group(xgrp, fmt, type_sets): @@ -358,19 +364,27 @@ def gen_xform_group(xgrp, fmt, type_sets): # pointing at an instruction. fmt.line('let inst = pos.current_inst().expect("need instruction");') + # Group the xforms by opcode so we can generate a big switch. + # Preserve ordering. + xforms = defaultdict(list) # type: DefaultDict[str, List[XForm]] + for xform in xgrp.xforms: + inst = xform.src.rtl[0].expr.inst + xforms[inst.camel_name].append(xform) + with fmt.indented('match dfg[inst].opcode() {', '}'): - for xform in xgrp.xforms: - inst = xform.src.rtl[0].expr.inst + for camel_name in sorted(xforms.keys()): with fmt.indented( - 'ir::Opcode::{} => {{'.format(inst.camel_name), '}'): - gen_xform(xform, fmt, type_sets) + 'ir::Opcode::{} => {{'.format(camel_name), '}'): + for xform in xforms[camel_name]: + gen_xform(xform, fmt, type_sets) # We'll assume there are uncovered opcodes. - if xgrp.chain: - fmt.format('_ => return {}(dfg, cfg, pos),', - xgrp.chain.rust_name()) - else: - fmt.line('_ => return false,') - fmt.line('true') + fmt.line('_ => {},') + + # If we fall through, nothing was expanded. Call the chain if any. + if xgrp.chain: + fmt.format('{}(dfg, cfg, pos)', xgrp.chain.rust_name()) + else: + fmt.line('false') def gen_isa(isa, fmt, shared_groups): diff --git a/lib/cretonne/meta/test_gen_legalizer.py b/lib/cretonne/meta/test_gen_legalizer.py index 6a317d887b..793555a42c 100644 --- a/lib/cretonne/meta/test_gen_legalizer.py +++ b/lib/cretonne/meta/test_gen_legalizer.py @@ -49,39 +49,27 @@ def typeset_check(v, ts): # type: (Var, TypeSet) -> CheckProducer return lambda typesets: format_check( typesets, - 'if !TYPE_SETS[{}].contains(typeof_{}) ' + - '{{\n return false;\n}};\n', ts, v) + 'let predicate = predicate && TYPE_SETS[{}].contains(typeof_{});\n', + ts, v) def equiv_check(tv1, tv2): - # type: (TypeVar, TypeVar) -> CheckProducer + # type: (str, str) -> CheckProducer return lambda typesets: format_check( typesets, - '{{\n' + - ' let a = {};\n' + - ' let b = {};\n' + - ' if a.is_none() || b.is_none() {{\n' + - ' return false;\n' + - ' }};\n' + - ' if a != b {{\n' + - ' return false;\n' + - ' }};\n' + + 'let predicate = predicate && match ({}, {}) {{\n' + ' (Some(a), Some(b)) => a == b,\n' + ' _ => false,\n' '}};\n', tv1, tv2) def wider_check(tv1, tv2): - # type: (TypeVar, TypeVar) -> CheckProducer + # type: (str, str) -> CheckProducer return lambda typesets: format_check( typesets, - '{{\n' + - ' let a = {};\n' + - ' let b = {};\n' + - ' if a.is_none() || b.is_none() {{\n' + - ' return false;\n' + - ' }};\n' + - ' if !a.wider_or_equal(b) {{\n' + - ' return false;\n' + - ' }};\n' + + 'let predicate = predicate && match ({}, {}) {{\n' + ' (Some(a), Some(b)) => a.wider_or_equal(b),\n' + ' _ => false,\n' '}};\n', tv1, tv2) From 939f188e27403dd359fcb5fa573726e61284a7df Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 28 Jul 2017 14:45:56 -0700 Subject: [PATCH 0936/3084] Add an inst.all_typevars() method. Get all type variables controlling an instruction, whether it is polymorphic or not. --- lib/cretonne/meta/cdsl/instructions.py | 10 ++++++++++ lib/cretonne/meta/cdsl/ti.py | 6 +----- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 6dec0b7dff..30c27c6649 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -267,6 +267,16 @@ class Instruction(object): return other_tvs + def all_typevars(self): + # type: () -> List[TypeVar] + """ + Get a list of all type variables in the instruction. + """ + if self.is_polymorphic: + return [self.ctrl_typevar] + self.other_typevars + else: + return [] + @staticmethod def _to_operand_tuple(x): # type: (Union[Sequence[Operand], Operand]) -> Tuple[Operand, ...] diff --git a/lib/cretonne/meta/cdsl/ti.py b/lib/cretonne/meta/cdsl/ti.py index 028579857d..b2673366be 100644 --- a/lib/cretonne/meta/cdsl/ti.py +++ b/lib/cretonne/meta/cdsl/ti.py @@ -796,11 +796,7 @@ def ti_def(definition, typ): # Create a dict m mapping each free typevar in the signature of definition # to a fresh copy of itself. - if inst.is_polymorphic: - free_formal_tvs = [inst.ctrl_typevar] + inst.other_typevars - else: - free_formal_tvs = [] - + free_formal_tvs = inst.all_typevars() m = {tv: tv.get_fresh_copy(str(typ.get_uid())) for tv in free_formal_tvs} # Update m with any explicitly bound type vars From b1508d858805f3231314e446c96173608b7eb817 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 28 Jul 2017 14:52:24 -0700 Subject: [PATCH 0937/3084] Include bound typevars in the instruction predicate. We already do this for the encoding tables, but the instruction predicates computed by Apply.inst_predicate() did not include them. Make sure we don't duplicate the type check in the Encoding constructor when passed an Apply AST node. --- lib/cretonne/meta/cdsl/ast.py | 9 ++++++++- lib/cretonne/meta/cdsl/isa.py | 19 ++++++++++--------- 2 files changed, 18 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index 0a9b5eece4..aa53e7b9b5 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -7,7 +7,7 @@ for patern matching an rewriting of cretonne instructions. from __future__ import absolute_import from . import instructions from .typevar import TypeVar -from .predicates import IsEqual, And +from .predicates import IsEqual, And, TypePredicate try: from typing import Union, Tuple, Sequence, TYPE_CHECKING, Dict, List # noqa @@ -367,6 +367,13 @@ class Apply(Expr): pred = And.combine(pred, IsEqual(ffield, arg)) + # Add checks for any bound type variables. + for bound_ty, tv in zip(self.typevars, self.inst.all_typevars()): + if bound_ty is None: + continue + type_chk = TypePredicate.typevar_check(self.inst, tv, bound_ty) + pred = And.combine(pred, type_chk) + return pred def copy(self, m): diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 3e903408ef..3e971e7a5a 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -420,6 +420,16 @@ class Encoding(object): else: self.inst, self.typevars = inst.fully_bound() + # Add secondary type variables to the instruction predicate. + # This is already included by Apply.inst_predicate() above. + if len(self.typevars) > 1: + for tv, vt in zip(self.inst.other_typevars, self.typevars[1:]): + # A None tv is an 'any' wild card: `ishl.i32.any`. + if vt is None: + continue + typred = TypePredicate.typevar_check(self.inst, tv, vt) + instp = And.combine(instp, typred) + self.cpumode = cpumode assert self.inst.format == recipe.format, ( "Format {} must match recipe: {}".format( @@ -433,15 +443,6 @@ class Encoding(object): self.recipe = recipe self.encbits = encbits - # Add secondary type variables to the instruction predicate. - if len(self.typevars) > 1: - for tv, vt in zip(self.inst.other_typevars, self.typevars[1:]): - # A None tv is an 'any' wild card: `ishl.i32.any`. - if vt is None: - continue - typred = TypePredicate.typevar_check(self.inst, tv, vt) - instp = And.combine(instp, typred) - # Record specific predicates. Note that the recipe also has predicates. self.instp = self.cpumode.isa.unique_pred(instp) self.isap = self.cpumode.isa.unique_pred(isap) From 542bad0758a912dc16a5aab91a0c849ea722307b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 28 Jul 2017 15:21:10 -0700 Subject: [PATCH 0938/3084] Evaluate instruction predicates during legalization. The generated legalization code needs to evaluate any instruction patterns on the input pattern being matched. Emit predicate checking code inside the InstructionFormat pattern match where all the instruction's immediate fields are available to the predicate code. Also make sure an `args` array is available for any type predicates to evaluate correctly. --- lib/cretonne/meta/gen_legalizer.py | 37 ++++++++++++++---------------- 1 file changed, 17 insertions(+), 20 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index bee51a928a..0a75fca8af 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -154,6 +154,9 @@ def unwrap_inst(iref, node, fmt): Create local variables named after the `Var` instances in `node`. + Also create a local variable named `predicate` with the value of the + evaluated instruction predicate, or `true` if the node has no predicate. + :param iref: Name of the `Inst` reference to unwrap. :param node: `Def` node providing variable names. :returns: True if the instruction arguments were not detached, expecting a @@ -166,7 +169,7 @@ def unwrap_inst(iref, node, fmt): # The tuple of locals we're extracting is `expr.args`. with fmt.indented( - 'let ({}) = if let ir::InstructionData::{} {{' + 'let ({}, predicate) = if let ir::InstructionData::{} {{' .format(', '.join(map(str, expr.args)), iform.name), '};'): # Fields are encoded directly. for f in iform.imm_fields: @@ -179,20 +182,20 @@ def unwrap_inst(iref, node, fmt): fmt.outdented_line('} = dfg[inst] {') if iform.has_value_list: fmt.line('let args = args.as_slice(&dfg.value_lists);') + elif nvops == 1: + fmt.line('let args = [arg];') # Generate the values for the tuple. - outs = list() - for opnum, op in enumerate(expr.inst.ins): - if op.is_immediate(): - n = expr.inst.imm_opnums.index(opnum) - outs.append(iform.imm_fields[n].member) - elif op.is_value(): - if nvops == 1: - arg = 'arg' - else: + with fmt.indented('(', ')'): + for opnum, op in enumerate(expr.inst.ins): + if op.is_immediate(): + n = expr.inst.imm_opnums.index(opnum) + fmt.format('{},', iform.imm_fields[n].member) + elif op.is_value(): n = expr.inst.value_opnums.index(opnum) - arg = 'args[{}]'.format(n) - outs.append('dfg.resolve_aliases({})'.format(arg)) - fmt.line('({})'.format(', '.join(outs))) + fmt.format('dfg.resolve_aliases(args[{}]),', n) + # Evaluate the instruction predicate, if any. + instp = expr.inst_predicate() + fmt.line(instp.rust_predicate(0) if instp else 'true') fmt.outdented_line('} else {') fmt.line('unreachable!("bad instruction format")') @@ -320,14 +323,8 @@ def gen_xform(xform, fmt, type_sets): # variables. replace_inst = unwrap_inst('inst', xform.src.rtl[0], fmt) - # Check instruction predicate and emit type checks. - instp = xform.src.rtl[0].expr.inst_predicate() - # TODO: The instruction predicate should be evaluated with all the inst - # immediate fields available. Probably by unwrap_inst(). - fmt.format('let predicate = {};', - instp.rust_predicate(0) if instp else 'true') - # Emit any runtime checks. + # These will rebind `predicate` emitted by unwrap_inst(). for check in get_runtime_typechecks(xform): emit_runtime_typecheck(check, fmt, type_sets) From 99d34cbbd7e9924507b7469978658e70ba7ff346 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 28 Jul 2017 16:13:51 -0700 Subject: [PATCH 0939/3084] Try to depend only on the `ir` module being in scope. Generated code should used qualified names assuming that `ir` is in scope, not everything else. --- lib/cretonne/meta/cdsl/types.py | 2 +- lib/cretonne/meta/gen_encoding.py | 6 +++--- lib/cretonne/meta/gen_instr.py | 4 ++-- lib/cretonne/src/ir/instructions.rs | 1 + lib/cretonne/src/isa/arm32/enc_tables.rs | 2 +- lib/cretonne/src/isa/arm64/enc_tables.rs | 2 +- lib/cretonne/src/isa/intel/enc_tables.rs | 2 +- lib/cretonne/src/isa/riscv/enc_tables.rs | 2 +- lib/cretonne/src/legalizer/mod.rs | 1 - 9 files changed, 11 insertions(+), 11 deletions(-) diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py index 53a1dd79ad..8ae66ee359 100644 --- a/lib/cretonne/meta/cdsl/types.py +++ b/lib/cretonne/meta/cdsl/types.py @@ -39,7 +39,7 @@ class ValueType(object): def rust_name(self): # type: () -> str - return 'types::' + self.name.upper() + return 'ir::types::' + self.name.upper() @staticmethod def by_name(name): diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index c3b3ddb87c..0516b5c4c8 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -653,7 +653,7 @@ def emit_level2_hashtables(level2_hashtables, offt, level2_doc, fmt): if entry: fmt.line( 'Level2Entry ' + - '{{ opcode: Some(Opcode::{}), offset: {:#08x} }},' + '{{ opcode: Some(ir::Opcode::{}), offset: {:#08x} }},' .format(entry.inst.camel_name, entry.offset)) else: fmt.line( @@ -678,7 +678,7 @@ def emit_level1_hashtable(cpumode, level1, offt, fmt): # Empty hash table entry. Include the default legalization action. if not level2: fmt.format( - 'Level1Entry {{ ty: types::VOID, log2len: !0, ' + 'Level1Entry {{ ty: ir::types::VOID, log2len: !0, ' 'offset: 0, legalize: {} }},', level1.legalize_code) continue @@ -686,7 +686,7 @@ def emit_level1_hashtable(cpumode, level1, offt, fmt): if level2.ty is not None: tyname = level2.ty.rust_name() else: - tyname = 'types::VOID' + tyname = 'ir::types::VOID' lcode = cpumode.isa.legalize_code(level2.legalize) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index b0c9943806..ade66e643c 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -350,10 +350,10 @@ def gen_typesets_table(fmt, type_sets): fmt.comment('Table of value type sets.') assert len(type_sets.table) <= typeset_limit, "Too many type sets" with fmt.indented( - 'const TYPE_SETS : [ValueTypeSet; {}] = [' + 'const TYPE_SETS : [ir::instructions::ValueTypeSet; {}] = [' .format(len(type_sets.table)), '];'): for ts in type_sets.table: - with fmt.indented('ValueTypeSet {', '},'): + with fmt.indented('ir::instructions::ValueTypeSet {', '},'): ts.emit_fields(fmt) diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 279a7a37a9..900dd56e88 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -10,6 +10,7 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::ops::{Deref, DerefMut}; +use ir; use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef, StackSlot, MemFlags}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32, Uoffset32}; use ir::condcodes::*; diff --git a/lib/cretonne/src/isa/arm32/enc_tables.rs b/lib/cretonne/src/isa/arm32/enc_tables.rs index eeb8466669..f71dd33f87 100644 --- a/lib/cretonne/src/isa/arm32/enc_tables.rs +++ b/lib/cretonne/src/isa/arm32/enc_tables.rs @@ -1,6 +1,6 @@ //! Encoding tables for ARM32 ISA. -use ir::types; +use ir; use isa; use isa::constraints::*; use isa::enc_tables::*; diff --git a/lib/cretonne/src/isa/arm64/enc_tables.rs b/lib/cretonne/src/isa/arm64/enc_tables.rs index c2c087e039..6007450cb5 100644 --- a/lib/cretonne/src/isa/arm64/enc_tables.rs +++ b/lib/cretonne/src/isa/arm64/enc_tables.rs @@ -1,6 +1,6 @@ //! Encoding tables for ARM64 ISA. -use ir::types; +use ir; use isa; use isa::constraints::*; use isa::enc_tables::*; diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 62ef7b3cd6..ad66cb1f7f 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -1,6 +1,6 @@ //! Encoding tables for Intel ISAs. -use ir::{self, types, Opcode}; +use ir; use isa; use isa::constraints::*; use isa::enc_tables::*; diff --git a/lib/cretonne/src/isa/riscv/enc_tables.rs b/lib/cretonne/src/isa/riscv/enc_tables.rs index 0d11d006c5..f095e67243 100644 --- a/lib/cretonne/src/isa/riscv/enc_tables.rs +++ b/lib/cretonne/src/isa/riscv/enc_tables.rs @@ -1,7 +1,7 @@ //! Encoding tables for RISC-V. use ir::condcodes::IntCC; -use ir::{self, types, Opcode}; +use ir; use isa; use isa::constraints::*; use isa::enc_tables::*; diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index c5178a97ee..c6a844c3c1 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -19,7 +19,6 @@ use ir::{self, Function, Cursor}; use ir::condcodes::IntCC; use isa::TargetIsa; use bitset::BitSet; -use ir::instructions::ValueTypeSet; mod boundary; mod split; From 54534e2147bbd55308619e39caede2ff643909be Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 28 Jul 2017 08:46:45 -0700 Subject: [PATCH 0940/3084] Add Intel legalization for division and multiplication. These operations need custom legalization in order to use Intel's div and idiv instructions. --- cranelift/filetests/wasm/i32-arith.cton | 26 +++++++-- cranelift/filetests/wasm/i64-arith.cton | 26 +++++++-- lib/cretonne/meta/base/legalize.py | 16 ++++++ lib/cretonne/meta/isa/intel/encodings.py | 7 +-- lib/cretonne/meta/isa/intel/instructions.py | 14 ++--- lib/cretonne/meta/isa/intel/legalize.py | 58 +++++++++++++++++++++ lib/cretonne/src/isa/intel/enc_tables.rs | 3 +- 7 files changed, 134 insertions(+), 16 deletions(-) create mode 100644 lib/cretonne/meta/isa/intel/legalize.py diff --git a/cranelift/filetests/wasm/i32-arith.cton b/cranelift/filetests/wasm/i32-arith.cton index b345847eb9..fe9cf19883 100644 --- a/cranelift/filetests/wasm/i32-arith.cton +++ b/cranelift/filetests/wasm/i32-arith.cton @@ -55,9 +55,29 @@ ebb0(v0: i32, v1: i32): return v2 } -; function %i32_div(i32, i32) -> i32 -; function %i32_rem_s(i32, i32) -> i32 -; function %i32_rem_u(i32, i32) -> i32 +function %i32_div_s(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = sdiv v0, v1 + return v2 +} + +function %i32_div_u(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = udiv v0, v1 + return v2 +} + +function %i32_rem_s(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = srem v0, v1 + return v2 +} + +function %i32_rem_u(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = urem v0, v1 + return v2 +} function %i32_and(i32, i32) -> i32 { ebb0(v0: i32, v1: i32): diff --git a/cranelift/filetests/wasm/i64-arith.cton b/cranelift/filetests/wasm/i64-arith.cton index a7cce18d3c..4e8cdc06df 100644 --- a/cranelift/filetests/wasm/i64-arith.cton +++ b/cranelift/filetests/wasm/i64-arith.cton @@ -52,9 +52,29 @@ ebb0(v0: i64, v1: i64): return v2 } -; function %i64_div(i64, i64) -> i64 -; function %i64_rem_s(i64, i64) -> i64 -; function %i64_rem_u(i64, i64) -> i64 +function %i32_div_s(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = sdiv v0, v1 + return v2 +} + +function %i32_div_u(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = udiv v0, v1 + return v2 +} + +function %i32_rem_s(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = srem v0, v1 + return v2 +} + +function %i32_rem_u(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = urem v0, v1 + return v2 +} function %i64_and(i64, i64) -> i64 { ebb0(v0: i64, v1: i64): diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index 1afefcda89..c250d53ce2 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -14,6 +14,8 @@ from .instructions import band, bor, bxor, isplit, iconcat from .instructions import bnot, band_not, bor_not, bxor_not from .instructions import icmp, icmp_imm from .instructions import iconst, bint +from .instructions import ishl, ishl_imm, sshr, sshr_imm, ushr, ushr_imm +from .instructions import rotl, rotl_imm, rotr, rotr_imm from cdsl.ast import Var from cdsl.xform import Rtl, XFormGroup @@ -153,6 +155,20 @@ expand.legalize( a << iadd(x, a1) )) +# Rotates and shifts. +for inst_imm, inst in [ + (rotl_imm, rotl), + (rotr_imm, rotr), + (ishl_imm, ishl), + (sshr_imm, sshr), + (ushr_imm, ushr)]: + expand.legalize( + a << inst_imm(x, y), + Rtl( + a1 << iconst.i32(y), + a << inst(x, a1) + )) + expand.legalize( a << icmp_imm(cc, x, y), Rtl( diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 44f8c16677..448252f43a 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -9,6 +9,7 @@ from .defs import I32, I64 from . import recipes as r from . import settings as cfg from . import instructions as x86 +from .legalize import intel_expand from base.legalize import narrow, expand try: @@ -21,14 +22,14 @@ except ImportError: I32.legalize_type( default=narrow, - i32=expand, + i32=intel_expand, f32=expand, f64=expand) I64.legalize_type( default=narrow, - i32=expand, - i64=expand, + i32=intel_expand, + i64=intel_expand, f32=expand, f64=expand) diff --git a/lib/cretonne/meta/isa/intel/instructions.py b/lib/cretonne/meta/isa/intel/instructions.py index c9d87c43f3..7921b2f431 100644 --- a/lib/cretonne/meta/isa/intel/instructions.py +++ b/lib/cretonne/meta/isa/intel/instructions.py @@ -6,17 +6,19 @@ target ISA. """ from cdsl.operands import Operand +from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup -from base.instructions import iB GROUP = InstructionGroup("x86", "Intel-specific instruction set") -nlo = Operand('nlo', iB, doc='Low part of numerator') -nhi = Operand('nhi', iB, doc='High part of numerator') -d = Operand('d', iB, doc='Denominator') -q = Operand('q', iB, doc='Quotient') -r = Operand('r', iB, doc='Remainder') +iWord = TypeVar('iWord', 'A scalar integer machine word', ints=(32, 64)) + +nlo = Operand('nlo', iWord, doc='Low part of numerator') +nhi = Operand('nhi', iWord, doc='High part of numerator') +d = Operand('d', iWord, doc='Denominator') +q = Operand('q', iWord, doc='Quotient') +r = Operand('r', iWord, doc='Remainder') udivmodx = Instruction( 'x86_udivmodx', r""" diff --git a/lib/cretonne/meta/isa/intel/legalize.py b/lib/cretonne/meta/isa/intel/legalize.py new file mode 100644 index 0000000000..cc46846d81 --- /dev/null +++ b/lib/cretonne/meta/isa/intel/legalize.py @@ -0,0 +1,58 @@ +""" +Custom legalization patterns for Intel. +""" +from __future__ import absolute_import +from cdsl.ast import Var +from cdsl.xform import Rtl, XFormGroup +from base.immediates import imm64 +from base.types import i32, i64 +from base import legalize as shared +from base import instructions as insts +from . import instructions as x86 +from .defs import ISA + +intel_expand = XFormGroup( + 'intel_expand', + """ + Legalize instructions by expansion. + + Use Intel-specific instructions if needed. + """, + isa=ISA, chain=shared.expand) + +a = Var('a') +dead = Var('dead') +x = Var('x') +xhi = Var('xhi') +y = Var('y') + +# +# Division and remainder. +# +intel_expand.legalize( + a << insts.udiv(x, y), + Rtl( + xhi << insts.iconst(imm64(0)), + (a, dead) << x86.udivmodx(x, xhi, y) + )) + +intel_expand.legalize( + a << insts.urem(x, y), + Rtl( + xhi << insts.iconst(imm64(0)), + (dead, a) << x86.udivmodx(x, xhi, y) + )) + +for ty in [i32, i64]: + intel_expand.legalize( + a << insts.sdiv.bind(ty)(x, y), + Rtl( + xhi << insts.sshr_imm(x, imm64(ty.lane_bits() - 1)), + (a, dead) << x86.sdivmodx(x, xhi, y) + )) + intel_expand.legalize( + a << insts.srem.bind(ty)(x, y), + Rtl( + xhi << insts.sshr_imm(x, imm64(ty.lane_bits() - 1)), + (dead, a) << x86.sdivmodx(x, xhi, y) + )) diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index ad66cb1f7f..35ca9a9d76 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -1,10 +1,11 @@ //! Encoding tables for Intel ISAs. +use bitset::BitSet; use ir; -use isa; use isa::constraints::*; use isa::enc_tables::*; use isa::encoding::RecipeSizing; +use isa; use predicates; use super::registers::*; From 07e1f682d08379628d72f8606692a79dfc1e9a0a Mon Sep 17 00:00:00 2001 From: Denis Merigoux Date: Mon, 31 Jul 2017 14:52:39 -0700 Subject: [PATCH 0941/3084] Added Intel x86-64 encodings for 64bit loads and store instructions (#127) * Added Intel x86-64 encodings for 64bit loads and store instructions * Using GPR registers instead of ABCD for istore8 with REX prefix Fixed testing of 64bit intel encoding * Emit REX and REX-less encodings for optional REX prefix Value renumbering in binary64.cton --- cranelift/filetests/isa/intel/binary64.cton | 376 ++++++++++++++++---- lib/cretonne/meta/base/instructions.py | 4 +- lib/cretonne/meta/isa/intel/encodings.py | 86 +++-- lib/cretonne/src/isa/intel/binemit.rs | 9 + 4 files changed, 373 insertions(+), 102 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 3d93ba86ec..bc19e039d3 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -145,52 +145,199 @@ ebb0: ; asm: movq %rcx, %r10 [-,%r10] v112 = copy v1 ; bin: 49 89 ca + ; Load/Store instructions. + + ; Register indirect addressing with no displacement. + + ; asm: movq %rcx, (%rsi) + store v1, v2 ; bin: 48 89 0e + ; asm: movq %rsi, (%rcx) + store v2, v1 ; bin: 48 89 31 + ; asm: movl %ecx, (%rsi) + istore32 v1, v2 ; bin: 40 89 0e + ; asm: movl %esi, (%rcx) + istore32 v2, v1 ; bin: 40 89 31 + ; asm: movw %cx, (%rsi) + istore16 v1, v2 ; bin: 66 40 89 0e + ; asm: movw %si, (%rcx) + istore16 v2, v1 ; bin: 66 40 89 31 + ; asm: movb %cl, (%rsi) + istore8 v1, v2 ; bin: 40 88 0e + ; asm: movb %sil, (%rcx) + istore8 v2, v1 ; bin: 40 88 31 + + ; asm: movq (%rcx), %rdi + [-,%rdi] v120 = load.i64 v1 ; bin: 48 8b 39 + ; asm: movq (%rsi), %rdx + [-,%rdx] v121 = load.i64 v2 ; bin: 48 8b 16 + ; asm: movl (%rcx), %edi + [-,%rdi] v122 = uload32.i64 v1 ; bin: 40 8b 39 + ; asm: movl (%rsi), %edx + [-,%rdx] v123 = uload32.i64 v2 ; bin: 40 8b 16 + ; asm: movslq (%rcx), %rdi + [-,%rdi] v124 = sload32.i64 v1 ; bin: 48 63 39 + ; asm: movslq (%rsi), %rdx + [-,%rdx] v125 = sload32.i64 v2 ; bin: 48 63 16 + ; asm: movzwq (%rcx), %rdi + [-,%rdi] v126 = uload16.i64 v1 ; bin: 48 0f b7 39 + ; asm: movzwq (%rsi), %rdx + [-,%rdx] v127 = uload16.i64 v2 ; bin: 48 0f b7 16 + ; asm: movswq (%rcx), %rdi + [-,%rdi] v128 = sload16.i64 v1 ; bin: 48 0f bf 39 + ; asm: movswq (%rsi), %rdx + [-,%rdx] v129 = sload16.i64 v2 ; bin: 48 0f bf 16 + ; asm: movzbq (%rcx), %rdi + [-,%rdi] v130 = uload8.i64 v1 ; bin: 48 0f b6 39 + ; asm: movzbq (%rsi), %rdx + [-,%rdx] v131 = uload8.i64 v2 ; bin: 48 0f b6 16 + ; asm: movsbq (%rcx), %rdi + [-,%rdi] v132 = sload8.i64 v1 ; bin: 48 0f be 39 + ; asm: movsbq (%rsi), %rdx + [-,%rdx] v133 = sload8.i64 v2 ; bin: 48 0f be 16 + + ; Register-indirect with 8-bit signed displacement. + + ; asm: movq %rcx, 100(%rsi) + store v1, v2+100 ; bin: 48 89 4e 64 + ; asm: movq %rsi, -100(%rcx) + store v2, v1-100 ; bin: 48 89 71 9c + ; asm: movl %ecx, 100(%rsi) + istore32 v1, v2+100 ; bin: 40 89 4e 64 + ; asm: movl %esi, -100(%rcx) + istore32 v2, v1-100 ; bin: 40 89 71 9c + ; asm: movw %cx, 100(%rsi) + istore16 v1, v2+100 ; bin: 66 40 89 4e 64 + ; asm: movw %si, -100(%rcx) + istore16 v2, v1-100 ; bin: 66 40 89 71 9c + ; asm: movb %cl, 100(%rsi) + istore8 v1, v2+100 ; bin: 40 88 4e 64 + ; asm: movb %sil, 100(%rcx) + istore8 v2, v1+100 ; bin: 40 88 71 64 + + ; asm: movq 50(%rcx), %rdi + [-,%rdi] v140 = load.i64 v1+50 ; bin: 48 8b 79 32 + ; asm: movq -50(%rsi), %rdx + [-,%rdx] v141 = load.i64 v2-50 ; bin: 48 8b 56 ce + ; asm: movl 50(%rcx), %edi + [-,%rdi] v142 = uload32.i64 v1+50 ; bin: 40 8b 79 32 + ; asm: movl -50(%rsi), %edx + [-,%rdx] v143 = uload32.i64 v2-50 ; bin: 40 8b 56 ce + ; asm: movslq 50(%rcx), %rdi + [-,%rdi] v144 = sload32.i64 v1+50 ; bin: 48 63 79 32 + ; asm: movslq -50(%rsi), %rdx + [-,%rdx] v145 = sload32.i64 v2-50 ; bin: 48 63 56 ce + ; asm: movzwq 50(%rcx), %rdi + [-,%rdi] v146 = uload16.i64 v1+50 ; bin: 48 0f b7 79 32 + ; asm: movzwq -50(%rsi), %rdx + [-,%rdx] v147 = uload16.i64 v2-50 ; bin: 48 0f b7 56 ce + ; asm: movswq 50(%rcx), %rdi + [-,%rdi] v148 = sload16.i64 v1+50 ; bin: 48 0f bf 79 32 + ; asm: movswq -50(%rsi), %rdx + [-,%rdx] v149 = sload16.i64 v2-50 ; bin: 48 0f bf 56 ce + ; asm: movzbq 50(%rcx), %rdi + [-,%rdi] v150 = uload8.i64 v1+50 ; bin: 48 0f b6 79 32 + ; asm: movzbq -50(%rsi), %rdx + [-,%rdx] v151 = uload8.i64 v2-50 ; bin: 48 0f b6 56 ce + ; asm: movsbq 50(%rcx), %rdi + [-,%rdi] v152 = sload8.i64 v1+50 ; bin: 48 0f be 79 32 + ; asm: movsbq -50(%rsi), %rdx + [-,%rdx] v153 = sload8.i64 v2-50 ; bin: 48 0f be 56 ce + + ; Register-indirect with 32-bit signed displacement. + + ; asm: movq %rcx, 10000(%rsi) + store v1, v2+10000 ; bin: 48 89 8e 00002710 + ; asm: movq %rsi, -10000(%rcx) + store v2, v1-10000 ; bin: 48 89 b1 ffffd8f0 + ; asm: movl %ecx, 10000(%rsi) + istore32 v1, v2+10000 ; bin: 40 89 8e 00002710 + ; asm: movl %esi, -10000(%rcx) + istore32 v2, v1-10000 ; bin: 40 89 b1 ffffd8f0 + ; asm: movw %cx, 10000(%rsi) + istore16 v1, v2+10000 ; bin: 66 40 89 8e 00002710 + ; asm: movw %si, -10000(%rcx) + istore16 v2, v1-10000 ; bin: 66 40 89 b1 ffffd8f0 + ; asm: movb %cl, 10000(%rsi) + istore8 v1, v2+10000 ; bin: 40 88 8e 00002710 + ; asm: movb %sil, 10000(%rcx) + istore8 v2, v1+10000 ; bin: 40 88 b1 00002710 + + ; asm: movq 50000(%rcx), %rdi + [-,%rdi] v160 = load.i64 v1+50000 ; bin: 48 8b b9 0000c350 + ; asm: movq -50000(%rsi), %rdx + [-,%rdx] v161 = load.i64 v2-50000 ; bin: 48 8b 96 ffff3cb0 + ; asm: movl 50000(%rcx), %edi + [-,%rdi] v162 = uload32.i64 v1+50000 ; bin: 40 8b b9 0000c350 + ; asm: movl -50000(%rsi), %edx + [-,%rdx] v163 = uload32.i64 v2-50000 ; bin: 40 8b 96 ffff3cb0 + ; asm: movslq 50000(%rcx), %rdi + [-,%rdi] v164 = sload32.i64 v1+50000 ; bin: 48 63 b9 0000c350 + ; asm: movslq -50000(%rsi), %rdx + [-,%rdx] v165 = sload32.i64 v2-50000 ; bin: 48 63 96 ffff3cb0 + ; asm: movzwq 50000(%rcx), %rdi + [-,%rdi] v166 = uload16.i64 v1+50000 ; bin: 48 0f b7 b9 0000c350 + ; asm: movzwq -50000(%rsi), %rdx + [-,%rdx] v167 = uload16.i64 v2-50000 ; bin: 48 0f b7 96 ffff3cb0 + ; asm: movswq 50000(%rcx), %rdi + [-,%rdi] v168 = sload16.i64 v1+50000 ; bin: 48 0f bf b9 0000c350 + ; asm: movswq -50000(%rsi), %rdx + [-,%rdx] v169 = sload16.i64 v2-50000 ; bin: 48 0f bf 96 ffff3cb0 + ; asm: movzbq 50000(%rcx), %rdi + [-,%rdi] v170 = uload8.i64 v1+50000 ; bin: 48 0f b6 b9 0000c350 + ; asm: movzbq -50000(%rsi), %rdx + [-,%rdx] v171 = uload8.i64 v2-50000 ; bin: 48 0f b6 96 ffff3cb0 + ; asm: movsbq 50000(%rcx), %rdi + [-,%rdi] v172 = sload8.i64 v1+50000 ; bin: 48 0f be b9 0000c350 + ; asm: movsbq -50000(%rsi), %rdx + [-,%rdx] v173 = sload8.i64 v2-50000 ; bin: 48 0f be 96 ffff3cb0 + + ; More arithmetic. ; asm: imulq %rsi, %rcx - [-,%rcx] v120 = imul v1, v2 ; bin: 48 0f af ce + [-,%rcx] v180 = imul v1, v2 ; bin: 48 0f af ce ; asm: imulq %r10, %rsi - [-,%rsi] v121 = imul v2, v3 ; bin: 49 0f af f2 + [-,%rsi] v181 = imul v2, v3 ; bin: 49 0f af f2 ; asm: imulq %rcx, %r10 - [-,%r10] v122 = imul v3, v1 ; bin: 4c 0f af d1 + [-,%r10] v182 = imul v3, v1 ; bin: 4c 0f af d1 - [-,%rax] v130 = iconst.i64 1 - [-,%rdx] v131 = iconst.i64 2 + [-,%rax] v190 = iconst.i64 1 + [-,%rdx] v191 = iconst.i64 2 ; asm: idivq %rcx - [-,%rax,%rdx] v132, v133 = x86_sdivmodx v130, v131, v1 ; bin: 48 f7 f9 + [-,%rax,%rdx] v192, v193 = x86_sdivmodx v130, v131, v1 ; bin: 48 f7 f9 ; asm: idivq %rsi - [-,%rax,%rdx] v134, v135 = x86_sdivmodx v130, v131, v2 ; bin: 48 f7 fe + [-,%rax,%rdx] v194, v195 = x86_sdivmodx v130, v131, v2 ; bin: 48 f7 fe ; asm: idivq %r10 - [-,%rax,%rdx] v136, v137 = x86_sdivmodx v130, v131, v3 ; bin: 49 f7 fa + [-,%rax,%rdx] v196, v197 = x86_sdivmodx v130, v131, v3 ; bin: 49 f7 fa ; asm: divq %rcx - [-,%rax,%rdx] v138, v139 = x86_udivmodx v130, v131, v1 ; bin: 48 f7 f1 + [-,%rax,%rdx] v198, v199 = x86_udivmodx v130, v131, v1 ; bin: 48 f7 f1 ; asm: divq %rsi - [-,%rax,%rdx] v140, v141 = x86_udivmodx v130, v131, v2 ; bin: 48 f7 f6 + [-,%rax,%rdx] v200, v201 = x86_udivmodx v130, v131, v2 ; bin: 48 f7 f6 ; asm: divq %r10 - [-,%rax,%rdx] v142, v143 = x86_udivmodx v130, v131, v3 ; bin: 49 f7 f2 + [-,%rax,%rdx] v202, v203 = x86_udivmodx v130, v131, v3 ; bin: 49 f7 f2 ; Bit-counting instructions. ; asm: popcntq %rsi, %rcx - [-,%rcx] v200 = popcnt v2 ; bin: f3 48 0f b8 ce + [-,%rcx] v210 = popcnt v2 ; bin: f3 48 0f b8 ce ; asm: popcntq %r10, %rsi - [-,%rsi] v201 = popcnt v3 ; bin: f3 49 0f b8 f2 + [-,%rsi] v211 = popcnt v3 ; bin: f3 49 0f b8 f2 ; asm: popcntq %rcx, %r10 - [-,%r10] v202 = popcnt v1 ; bin: f3 4c 0f b8 d1 + [-,%r10] v212 = popcnt v1 ; bin: f3 4c 0f b8 d1 ; asm: lzcntq %rsi, %rcx - [-,%rcx] v203 = clz v2 ; bin: f3 48 0f bd ce + [-,%rcx] v213 = clz v2 ; bin: f3 48 0f bd ce ; asm: lzcntq %r10, %rsi - [-,%rsi] v204 = clz v3 ; bin: f3 49 0f bd f2 + [-,%rsi] v214 = clz v3 ; bin: f3 49 0f bd f2 ; asm: lzcntq %rcx, %r10 - [-,%r10] v205 = clz v1 ; bin: f3 4c 0f bd d1 + [-,%r10] v215 = clz v1 ; bin: f3 4c 0f bd d1 ; asm: tzcntq %rsi, %rcx - [-,%rcx] v206 = ctz v2 ; bin: f3 48 0f bc ce + [-,%rcx] v216 = ctz v2 ; bin: f3 48 0f bc ce ; asm: tzcntq %r10, %rsi - [-,%rsi] v207 = ctz v3 ; bin: f3 49 0f bc f2 + [-,%rsi] v217 = ctz v3 ; bin: f3 49 0f bc f2 ; asm: tzcntq %rcx, %r10 - [-,%r10] v208 = ctz v1 ; bin: f3 4c 0f bc d1 + [-,%r10] v218 = ctz v1 ; bin: f3 4c 0f bc d1 ; Integer comparisons. @@ -327,146 +474,217 @@ ebb0: ; asm: movl $0x88001122, %r14d [-,%r14] v5 = iconst.i32 0xffff_ffff_8800_1122 ; bin: 41 be 88001122 + ; Load/Store instructions. + + ; Register indirect addressing with no displacement. + + ; asm: movl (%rcx), %edi + [-,%rdi] v10 = load.i32 v1 ; bin: 40 8b 39 + ; asm: movl (%rsi), %edx + [-,%rdx] v11 = load.i32 v2 ; bin: 40 8b 16 + ; asm: movzwl (%rcx), %edi + [-,%rdi] v12 = uload16.i32 v1 ; bin: 40 0f b7 39 + ; asm: movzwl (%rsi), %edx + [-,%rdx] v13 = uload16.i32 v2 ; bin: 40 0f b7 16 + ; asm: movswl (%rcx), %edi + [-,%rdi] v14 = sload16.i32 v1 ; bin: 40 0f bf 39 + ; asm: movswl (%rsi), %edx + [-,%rdx] v15 = sload16.i32 v2 ; bin: 40 0f bf 16 + ; asm: movzbl (%rcx), %edi + [-,%rdi] v16 = uload8.i32 v1 ; bin: 40 0f b6 39 + ; asm: movzbl (%rsi), %edx + [-,%rdx] v17 = uload8.i32 v2 ; bin: 40 0f b6 16 + ; asm: movsbl (%rcx), %edi + [-,%rdi] v18 = sload8.i32 v1 ; bin: 40 0f be 39 + ; asm: movsbl (%rsi), %edx + [-,%rdx] v19 = sload8.i32 v2 ; bin: 40 0f be 16 + + ; Register-indirect with 8-bit signed displacement. + + ; asm: movl 50(%rcx), %edi + [-,%rdi] v20 = load.i32 v1+50 ; bin: 40 8b 79 32 + ; asm: movl -50(%rsi), %edx + [-,%rdx] v21 = load.i32 v2-50 ; bin: 40 8b 56 ce + ; asm: movzwl 50(%rcx), %edi + [-,%rdi] v22 = uload16.i32 v1+50 ; bin: 40 0f b7 79 32 + ; asm: movzwl -50(%rsi), %edx + [-,%rdx] v23 = uload16.i32 v2-50 ; bin: 40 0f b7 56 ce + ; asm: movswl 50(%rcx), %edi + [-,%rdi] v24 = sload16.i32 v1+50 ; bin: 40 0f bf 79 32 + ; asm: movswl -50(%rsi), %edx + [-,%rdx] v25 = sload16.i32 v2-50 ; bin: 40 0f bf 56 ce + ; asm: movzbl 50(%rcx), %edi + [-,%rdi] v26 = uload8.i32 v1+50 ; bin: 40 0f b6 79 32 + ; asm: movzbl -50(%rsi), %edx + [-,%rdx] v27 = uload8.i32 v2-50 ; bin: 40 0f b6 56 ce + ; asm: movsbl 50(%rcx), %edi + [-,%rdi] v28 = sload8.i32 v1+50 ; bin: 40 0f be 79 32 + ; asm: movsbl -50(%rsi), %edx + [-,%rdx] v29 = sload8.i32 v2-50 ; bin: 40 0f be 56 ce + + ; Register-indirect with 32-bit signed displacement. + + ; asm: movl 50000(%rcx), %edi + [-,%rdi] v30 = load.i32 v1+50000 ; bin: 40 8b b9 0000c350 + ; asm: movl -50000(%rsi), %edx + [-,%rdx] v31 = load.i32 v2-50000 ; bin: 40 8b 96 ffff3cb0 + ; asm: movzwl 50000(%rcx), %edi + [-,%rdi] v32 = uload16.i32 v1+50000 ; bin: 40 0f b7 b9 0000c350 + ; asm: movzwl -50000(%rsi), %edx + [-,%rdx] v33 = uload16.i32 v2-50000 ; bin: 40 0f b7 96 ffff3cb0 + ; asm: movswl 50000(%rcx), %edi + [-,%rdi] v34 = sload16.i32 v1+50000 ; bin: 40 0f bf b9 0000c350 + ; asm: movswl -50000(%rsi), %edx + [-,%rdx] v35 = sload16.i32 v2-50000 ; bin: 40 0f bf 96 ffff3cb0 + ; asm: movzbl 50000(%rcx), %edi + [-,%rdi] v36 = uload8.i32 v1+50000 ; bin: 40 0f b6 b9 0000c350 + ; asm: movzbl -50000(%rsi), %edx + [-,%rdx] v37 = uload8.i32 v2-50000 ; bin: 40 0f b6 96 ffff3cb0 + ; asm: movsbl 50000(%rcx), %edi + [-,%rdi] v38 = sload8.i32 v1+50000 ; bin: 40 0f be b9 0000c350 + ; asm: movsbl -50000(%rsi), %edx + [-,%rdx] v39 = sload8.i32 v2-50000 ; bin: 40 0f be 96 ffff3cb0 + ; Integer Register-Register Operations. ; asm: addl %esi, %ecx - [-,%rcx] v10 = iadd v1, v2 ; bin: 40 01 f1 + [-,%rcx] v40 = iadd v1, v2 ; bin: 40 01 f1 ; asm: addl %r10d, %esi - [-,%rsi] v11 = iadd v2, v3 ; bin: 44 01 d6 + [-,%rsi] v41 = iadd v2, v3 ; bin: 44 01 d6 ; asm: addl %ecx, %r10d - [-,%r10] v12 = iadd v3, v1 ; bin: 41 01 ca + [-,%r10] v42 = iadd v3, v1 ; bin: 41 01 ca ; asm: subl %esi, %ecx - [-,%rcx] v20 = isub v1, v2 ; bin: 40 29 f1 + [-,%rcx] v50 = isub v1, v2 ; bin: 40 29 f1 ; asm: subl %r10d, %esi - [-,%rsi] v21 = isub v2, v3 ; bin: 44 29 d6 + [-,%rsi] v51 = isub v2, v3 ; bin: 44 29 d6 ; asm: subl %ecx, %r10d - [-,%r10] v22 = isub v3, v1 ; bin: 41 29 ca + [-,%r10] v52 = isub v3, v1 ; bin: 41 29 ca ; asm: andl %esi, %ecx - [-,%rcx] v30 = band v1, v2 ; bin: 40 21 f1 + [-,%rcx] v60 = band v1, v2 ; bin: 40 21 f1 ; asm: andl %r10d, %esi - [-,%rsi] v31 = band v2, v3 ; bin: 44 21 d6 + [-,%rsi] v61 = band v2, v3 ; bin: 44 21 d6 ; asm: andl %ecx, %r10d - [-,%r10] v32 = band v3, v1 ; bin: 41 21 ca + [-,%r10] v62 = band v3, v1 ; bin: 41 21 ca ; asm: orl %esi, %ecx - [-,%rcx] v40 = bor v1, v2 ; bin: 40 09 f1 + [-,%rcx] v70 = bor v1, v2 ; bin: 40 09 f1 ; asm: orl %r10d, %esi - [-,%rsi] v41 = bor v2, v3 ; bin: 44 09 d6 + [-,%rsi] v71 = bor v2, v3 ; bin: 44 09 d6 ; asm: orl %ecx, %r10d - [-,%r10] v42 = bor v3, v1 ; bin: 41 09 ca + [-,%r10] v72 = bor v3, v1 ; bin: 41 09 ca ; asm: xorl %esi, %ecx - [-,%rcx] v50 = bxor v1, v2 ; bin: 40 31 f1 + [-,%rcx] v80 = bxor v1, v2 ; bin: 40 31 f1 ; asm: xorl %r10d, %esi - [-,%rsi] v51 = bxor v2, v3 ; bin: 44 31 d6 + [-,%rsi] v81 = bxor v2, v3 ; bin: 44 31 d6 ; asm: xorl %ecx, %r10d - [-,%r10] v52 = bxor v3, v1 ; bin: 41 31 ca + [-,%r10] v82 = bxor v3, v1 ; bin: 41 31 ca ; asm: shll %cl, %esi - [-,%rsi] v60 = ishl v2, v1 ; bin: 40 d3 e6 + [-,%rsi] v90 = ishl v2, v1 ; bin: 40 d3 e6 ; asm: shll %cl, %r10d - [-,%r10] v61 = ishl v3, v1 ; bin: 41 d3 e2 + [-,%r10] v91 = ishl v3, v1 ; bin: 41 d3 e2 ; asm: sarl %cl, %esi - [-,%rsi] v62 = sshr v2, v1 ; bin: 40 d3 fe + [-,%rsi] v92 = sshr v2, v1 ; bin: 40 d3 fe ; asm: sarl %cl, %r10d - [-,%r10] v63 = sshr v3, v1 ; bin: 41 d3 fa + [-,%r10] v93 = sshr v3, v1 ; bin: 41 d3 fa ; asm: shrl %cl, %esi - [-,%rsi] v64 = ushr v2, v1 ; bin: 40 d3 ee + [-,%rsi] v94 = ushr v2, v1 ; bin: 40 d3 ee ; asm: shrl %cl, %r10d - [-,%r10] v65 = ushr v3, v1 ; bin: 41 d3 ea + [-,%r10] v95 = ushr v3, v1 ; bin: 41 d3 ea ; asm: roll %cl, %esi - [-,%rsi] v66 = rotl v2, v1 ; bin: 40 d3 c6 + [-,%rsi] v96 = rotl v2, v1 ; bin: 40 d3 c6 ; asm: roll %cl, %r10d - [-,%r10] v67 = rotl v3, v1 ; bin: 41 d3 c2 + [-,%r10] v97 = rotl v3, v1 ; bin: 41 d3 c2 ; asm: rorl %cl, %esi - [-,%rsi] v68 = rotr v2, v1 ; bin: 40 d3 ce + [-,%rsi] v98 = rotr v2, v1 ; bin: 40 d3 ce ; asm: rorl %cl, %r10d - [-,%r10] v69 = rotr v3, v1 ; bin: 41 d3 ca + [-,%r10] v99 = rotr v3, v1 ; bin: 41 d3 ca ; Integer Register-Immediate Operations. ; These 64-bit ops all use a 32-bit immediate that is sign-extended to 64 bits. ; Some take 8-bit immediates that are sign-extended to 64 bits. ; asm: addl $-100000, %ecx - [-,%rcx] v70 = iadd_imm v1, -100000 ; bin: 40 81 c1 fffe7960 + [-,%rcx] v100 = iadd_imm v1, -100000 ; bin: 40 81 c1 fffe7960 ; asm: addl $100000, %esi - [-,%rsi] v71 = iadd_imm v2, 100000 ; bin: 40 81 c6 000186a0 + [-,%rsi] v101 = iadd_imm v2, 100000 ; bin: 40 81 c6 000186a0 ; asm: addl $0x7fffffff, %r10d - [-,%r10] v72 = iadd_imm v3, 0x7fff_ffff ; bin: 41 81 c2 7fffffff + [-,%r10] v102 = iadd_imm v3, 0x7fff_ffff ; bin: 41 81 c2 7fffffff ; asm: addl $100, %r8d - [-,%r8] v73 = iadd_imm v4, 100 ; bin: 41 83 c0 64 + [-,%r8] v103 = iadd_imm v4, 100 ; bin: 41 83 c0 64 ; asm: addl $-100, %r14d - [-,%r14] v74 = iadd_imm v5, -100 ; bin: 41 83 c6 9c + [-,%r14] v104 = iadd_imm v5, -100 ; bin: 41 83 c6 9c ; asm: andl $-100000, %ecx - [-,%rcx] v80 = band_imm v1, -100000 ; bin: 40 81 e1 fffe7960 + [-,%rcx] v110 = band_imm v1, -100000 ; bin: 40 81 e1 fffe7960 ; asm: andl $100000, %esi - [-,%rsi] v81 = band_imm v2, 100000 ; bin: 40 81 e6 000186a0 + [-,%rsi] v111 = band_imm v2, 100000 ; bin: 40 81 e6 000186a0 ; asm: andl $0x7fffffff, %r10d - [-,%r10] v82 = band_imm v3, 0x7fff_ffff ; bin: 41 81 e2 7fffffff + [-,%r10] v112 = band_imm v3, 0x7fff_ffff ; bin: 41 81 e2 7fffffff ; asm: andl $100, %r8d - [-,%r8] v83 = band_imm v4, 100 ; bin: 41 83 e0 64 + [-,%r8] v113 = band_imm v4, 100 ; bin: 41 83 e0 64 ; asm: andl $-100, %r14d - [-,%r14] v84 = band_imm v5, -100 ; bin: 41 83 e6 9c + [-,%r14] v114 = band_imm v5, -100 ; bin: 41 83 e6 9c ; asm: orl $-100000, %ecx - [-,%rcx] v90 = bor_imm v1, -100000 ; bin: 40 81 c9 fffe7960 + [-,%rcx] v120 = bor_imm v1, -100000 ; bin: 40 81 c9 fffe7960 ; asm: orl $100000, %esi - [-,%rsi] v91 = bor_imm v2, 100000 ; bin: 40 81 ce 000186a0 + [-,%rsi] v121 = bor_imm v2, 100000 ; bin: 40 81 ce 000186a0 ; asm: orl $0x7fffffff, %r10d - [-,%r10] v92 = bor_imm v3, 0x7fff_ffff ; bin: 41 81 ca 7fffffff + [-,%r10] v122 = bor_imm v3, 0x7fff_ffff ; bin: 41 81 ca 7fffffff ; asm: orl $100, %r8d - [-,%r8] v93 = bor_imm v4, 100 ; bin: 41 83 c8 64 + [-,%r8] v123 = bor_imm v4, 100 ; bin: 41 83 c8 64 ; asm: orl $-100, %r14d - [-,%r14] v94 = bor_imm v5, -100 ; bin: 41 83 ce 9c + [-,%r14] v124 = bor_imm v5, -100 ; bin: 41 83 ce 9c ; asm: ret ; asm: xorl $-100000, %ecx - [-,%rcx] v100 = bxor_imm v1, -100000 ; bin: 40 81 f1 fffe7960 + [-,%rcx] v130 = bxor_imm v1, -100000 ; bin: 40 81 f1 fffe7960 ; asm: xorl $100000, %esi - [-,%rsi] v101 = bxor_imm v2, 100000 ; bin: 40 81 f6 000186a0 + [-,%rsi] v131 = bxor_imm v2, 100000 ; bin: 40 81 f6 000186a0 ; asm: xorl $0x7fffffff, %r10d - [-,%r10] v102 = bxor_imm v3, 0x7fff_ffff ; bin: 41 81 f2 7fffffff + [-,%r10] v132 = bxor_imm v3, 0x7fff_ffff ; bin: 41 81 f2 7fffffff ; asm: xorl $100, %r8d - [-,%r8] v103 = bxor_imm v4, 100 ; bin: 41 83 f0 64 + [-,%r8] v133 = bxor_imm v4, 100 ; bin: 41 83 f0 64 ; asm: xorl $-100, %r14d - [-,%r14] v104 = bxor_imm v5, -100 ; bin: 41 83 f6 9c + [-,%r14] v134 = bxor_imm v5, -100 ; bin: 41 83 f6 9c ; Register copies. ; asm: movl %esi, %ecx - [-,%rcx] v110 = copy v2 ; bin: 40 89 f1 + [-,%rcx] v140 = copy v2 ; bin: 40 89 f1 ; asm: movl %r10d, %esi - [-,%rsi] v111 = copy v3 ; bin: 44 89 d6 + [-,%rsi] v141 = copy v3 ; bin: 44 89 d6 ; asm: movl %ecx, %r10d - [-,%r10] v112 = copy v1 ; bin: 41 89 ca + [-,%r10] v142 = copy v1 ; bin: 41 89 ca ; More arithmetic. ; asm: imull %esi, %ecx - [-,%rcx] v120 = imul v1, v2 ; bin: 40 0f af ce + [-,%rcx] v150 = imul v1, v2 ; bin: 40 0f af ce ; asm: imull %r10d, %esi - [-,%rsi] v121 = imul v2, v3 ; bin: 41 0f af f2 + [-,%rsi] v151 = imul v2, v3 ; bin: 41 0f af f2 ; asm: imull %ecx, %r10d - [-,%r10] v122 = imul v3, v1 ; bin: 44 0f af d1 + [-,%r10] v152 = imul v3, v1 ; bin: 44 0f af d1 - [-,%rax] v130 = iconst.i32 1 - [-,%rdx] v131 = iconst.i32 2 + [-,%rax] v160 = iconst.i32 1 + [-,%rdx] v161 = iconst.i32 2 ; asm: idivl %ecx - [-,%rax,%rdx] v132, v133 = x86_sdivmodx v130, v131, v1 ; bin: 40 f7 f9 + [-,%rax,%rdx] v162, v163 = x86_sdivmodx v130, v131, v1 ; bin: 40 f7 f9 ; asm: idivl %esi - [-,%rax,%rdx] v134, v135 = x86_sdivmodx v130, v131, v2 ; bin: 40 f7 fe + [-,%rax,%rdx] v164, v165 = x86_sdivmodx v130, v131, v2 ; bin: 40 f7 fe ; asm: idivl %r10d - [-,%rax,%rdx] v136, v137 = x86_sdivmodx v130, v131, v3 ; bin: 41 f7 fa + [-,%rax,%rdx] v166, v167 = x86_sdivmodx v130, v131, v3 ; bin: 41 f7 fa ; asm: divl %ecx - [-,%rax,%rdx] v138, v139 = x86_udivmodx v130, v131, v1 ; bin: 40 f7 f1 + [-,%rax,%rdx] v168, v169 = x86_udivmodx v130, v131, v1 ; bin: 40 f7 f1 ; asm: divl %esi - [-,%rax,%rdx] v140, v141 = x86_udivmodx v130, v131, v2 ; bin: 40 f7 f6 + [-,%rax,%rdx] v170, v171 = x86_udivmodx v130, v131, v2 ; bin: 40 f7 f6 ; asm: divl %r10d - [-,%rax,%rdx] v142, v143 = x86_udivmodx v130, v131, v3 ; bin: 41 f7 f2 + [-,%rax,%rdx] v172, v173 = x86_udivmodx v130, v131, v3 ; bin: 41 f7 f2 ; Bit-counting instructions. diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index a26d791719..3a4ce25cda 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -271,7 +271,7 @@ istore16 = Instruction( 'istore16', r""" Store the low 16 bits of ``x`` to memory at ``p + Offset``. - This is equivalent to ``ireduce.i16`` followed by ``store.i8``. + This is equivalent to ``ireduce.i16`` followed by ``store.i16``. """, ins=(Flags, x, p, Offset), can_store=True) @@ -301,7 +301,7 @@ istore32 = Instruction( 'istore32', r""" Store the low 32 bits of ``x`` to memory at ``p + Offset``. - This is equivalent to ``ireduce.i32`` followed by ``store.i8``. + This is equivalent to ``ireduce.i32`` followed by ``store.i32``. """, ins=(Flags, x, p, Offset), can_store=True) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 448252f43a..39b6812058 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -55,6 +55,28 @@ def enc_i32_i64(inst, recipe, *args, **kwargs): I64.enc(inst.i64, *recipe.rex(*args, w=1, **kwargs)) +def enc_i32_i64_ld_st(inst, w_bit, recipe, *args, **kwargs): + # type: (MaybeBoundInst, bool, r.TailRecipe, *int, **int) -> None + """ + Add encodings for `inst.i32` to I32. + Add encodings for `inst.i32` to I64 with and without REX. + Add encodings for `inst.i64` to I64 with a REX prefix, using the `w_bit` + argument to determine wheter or not to set the REX.W bit. + """ + I32.enc(inst.i32.any, *recipe(*args, **kwargs)) + + # REX-less encoding must come after REX encoding so we don't use it by + # default. Otherwise reg-alloc would never use r8 and up. + I64.enc(inst.i32.any, *recipe.rex(*args, **kwargs)) + I64.enc(inst.i32.any, *recipe(*args, **kwargs)) + + if w_bit: + I64.enc(inst.i64.any, *recipe.rex(*args, w=1, **kwargs)) + else: + I64.enc(inst.i64.any, *recipe.rex(*args, **kwargs)) + I64.enc(inst.i64.any, *recipe(*args, **kwargs)) + + def enc_flt(inst, recipe, *args, **kwargs): # type: (MaybeBoundInst, r.TailRecipe, *int, **int) -> None """ @@ -142,38 +164,60 @@ I64.enc(base.ctz.i64, *r.urm.rex(0xf3, 0x0f, 0xbc, w=1), I64.enc(base.ctz.i32, *r.urm.rex(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) I64.enc(base.ctz.i32, *r.urm(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) +# # Loads and stores. -I32.enc(base.store.i32.any, *r.st(0x89)) -I32.enc(base.store.i32.any, *r.stDisp8(0x89)) -I32.enc(base.store.i32.any, *r.stDisp32(0x89)) +# +enc_i32_i64_ld_st(base.store, True, r.st, 0x89) +enc_i32_i64_ld_st(base.store, True, r.stDisp8, 0x89) +enc_i32_i64_ld_st(base.store, True, r.stDisp32, 0x89) -I32.enc(base.istore16.i32.any, *r.st(0x66, 0x89)) -I32.enc(base.istore16.i32.any, *r.stDisp8(0x66, 0x89)) -I32.enc(base.istore16.i32.any, *r.stDisp32(0x66, 0x89)) +I64.enc(base.istore32.i64.any, *r.st.rex(0x89)) +I64.enc(base.istore32.i64.any, *r.stDisp8.rex(0x89)) +I64.enc(base.istore32.i64.any, *r.stDisp32.rex(0x89)) +enc_i32_i64_ld_st(base.istore16, False, r.st, 0x66, 0x89) +enc_i32_i64_ld_st(base.istore16, False, r.stDisp8, 0x66, 0x89) +enc_i32_i64_ld_st(base.istore16, False, r.stDisp32, 0x66, 0x89) + +# Byte stores are more complicated because the registers they can address +# depends of the presence of a REX prefix I32.enc(base.istore8.i32.any, *r.st_abcd(0x88)) +I64.enc(base.istore8.i32.any, *r.st_abcd(0x88)) +I64.enc(base.istore8.i64.any, *r.st.rex(0x88)) I32.enc(base.istore8.i32.any, *r.stDisp8_abcd(0x88)) +I64.enc(base.istore8.i32.any, *r.stDisp8_abcd(0x88)) +I64.enc(base.istore8.i64.any, *r.stDisp8.rex(0x88)) I32.enc(base.istore8.i32.any, *r.stDisp32_abcd(0x88)) +I64.enc(base.istore8.i32.any, *r.stDisp32_abcd(0x88)) +I64.enc(base.istore8.i64.any, *r.stDisp32.rex(0x88)) -I32.enc(base.load.i32.any, *r.ld(0x8b)) -I32.enc(base.load.i32.any, *r.ldDisp8(0x8b)) -I32.enc(base.load.i32.any, *r.ldDisp32(0x8b)) +enc_i32_i64_ld_st(base.load, True, r.ld, 0x8b) +enc_i32_i64_ld_st(base.load, True, r.ldDisp8, 0x8b) +enc_i32_i64_ld_st(base.load, True, r.ldDisp32, 0x8b) -I32.enc(base.uload16.i32.any, *r.ld(0x0f, 0xb7)) -I32.enc(base.uload16.i32.any, *r.ldDisp8(0x0f, 0xb7)) -I32.enc(base.uload16.i32.any, *r.ldDisp32(0x0f, 0xb7)) +I64.enc(base.uload32.i64, *r.ld.rex(0x8b)) +I64.enc(base.uload32.i64, *r.ldDisp8.rex(0x8b)) +I64.enc(base.uload32.i64, *r.ldDisp32.rex(0x8b)) -I32.enc(base.sload16.i32.any, *r.ld(0x0f, 0xbf)) -I32.enc(base.sload16.i32.any, *r.ldDisp8(0x0f, 0xbf)) -I32.enc(base.sload16.i32.any, *r.ldDisp32(0x0f, 0xbf)) +I64.enc(base.sload32.i64, *r.ld.rex(0x63, w=1)) +I64.enc(base.sload32.i64, *r.ldDisp8.rex(0x63, w=1)) +I64.enc(base.sload32.i64, *r.ldDisp32.rex(0x63, w=1)) -I32.enc(base.uload8.i32.any, *r.ld(0x0f, 0xb6)) -I32.enc(base.uload8.i32.any, *r.ldDisp8(0x0f, 0xb6)) -I32.enc(base.uload8.i32.any, *r.ldDisp32(0x0f, 0xb6)) +enc_i32_i64_ld_st(base.uload16, True, r.ld, 0x0f, 0xb7) +enc_i32_i64_ld_st(base.uload16, True, r.ldDisp8, 0x0f, 0xb7) +enc_i32_i64_ld_st(base.uload16, True, r.ldDisp32, 0x0f, 0xb7) -I32.enc(base.sload8.i32.any, *r.ld(0x0f, 0xbe)) -I32.enc(base.sload8.i32.any, *r.ldDisp8(0x0f, 0xbe)) -I32.enc(base.sload8.i32.any, *r.ldDisp32(0x0f, 0xbe)) +enc_i32_i64_ld_st(base.sload16, True, r.ld, 0x0f, 0xbf) +enc_i32_i64_ld_st(base.sload16, True, r.ldDisp8, 0x0f, 0xbf) +enc_i32_i64_ld_st(base.sload16, True, r.ldDisp32, 0x0f, 0xbf) + +enc_i32_i64_ld_st(base.uload8, True, r.ld, 0x0f, 0xb6) +enc_i32_i64_ld_st(base.uload8, True, r.ldDisp8, 0x0f, 0xb6) +enc_i32_i64_ld_st(base.uload8, True, r.ldDisp32, 0x0f, 0xb6) + +enc_i32_i64_ld_st(base.sload8, True, r.ld, 0x0f, 0xbe) +enc_i32_i64_ld_st(base.sload8, True, r.ldDisp8, 0x0f, 0xbe) +enc_i32_i64_ld_st(base.sload8, True, r.ldDisp32, 0x0f, 0xbe) # # Call/return diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 0f3b1b76c6..0eb6199e66 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -114,6 +114,15 @@ fn put_rexmp2(bits: u16, rex: u8, sink: &mut CS) { sink.put1(bits as u8); } +// Emit single-byte opcode with mandatory prefix and REX. +fn put_rexmp1(bits: u16, rex: u8, sink: &mut CS) { + debug_assert_eq!(bits & 0x0c00, 0, "Invalid encoding bits for Mp1*"); + let pp = (bits >> 8) & 3; + sink.put1(PREFIX[(pp - 1) as usize]); + rex_prefix(bits, rex, sink); + sink.put1(bits as u8); +} + /// Emit a ModR/M byte for reg-reg operands. fn modrm_rr(rm: RegUnit, reg: RegUnit, sink: &mut CS) { let reg = reg as u8 & 7; From 3a4a1d4faf8c23acd9e1829e797c2e710ab3a245 Mon Sep 17 00:00:00 2001 From: d1m0 Date: Mon, 31 Jul 2017 16:02:27 -0700 Subject: [PATCH 0942/3084] Cleanup for PR #123 (#129) * Fix bextend semantics; Change smtlib.py to use z3 python bindings for query building instead of raw strings * Forgot the mypy stubs for z3 --- lib/cretonne/meta/base/semantics.py | 4 +- lib/cretonne/meta/semantics/primitives.py | 4 + lib/cretonne/meta/semantics/smtlib.py | 114 ++++++++-------- lib/cretonne/meta/stubs/z3/__init__.pyi | 151 ++++++++++++++++++++++ lib/cretonne/meta/stubs/z3/z3core.pyi | 3 + lib/cretonne/meta/stubs/z3/z3types.pyi | 12 ++ 6 files changed, 234 insertions(+), 54 deletions(-) create mode 100644 lib/cretonne/meta/stubs/z3/__init__.pyi create mode 100644 lib/cretonne/meta/stubs/z3/z3core.pyi create mode 100644 lib/cretonne/meta/stubs/z3/z3types.pyi diff --git a/lib/cretonne/meta/base/semantics.py b/lib/cretonne/meta/base/semantics.py index 582fdc0889..edf4c5f82e 100644 --- a/lib/cretonne/meta/base/semantics.py +++ b/lib/cretonne/meta/base/semantics.py @@ -1,6 +1,6 @@ from __future__ import absolute_import from semantics.primitives import prim_to_bv, prim_from_bv, bvsplit, bvconcat,\ - bvadd, bvult, bvzeroext + bvadd, bvult, bvzeroext, bvsignext from .instructions import vsplit, vconcat, iadd, iadd_cout, icmp, bextend, \ isplit, iconcat, iadd_cin, iadd_carry from .immediates import intcc @@ -116,7 +116,7 @@ bextend.set_semantics( a << bextend(x), (Rtl( bvx << prim_to_bv(x), - bvy << bvzeroext(bvx), + bvy << bvsignext(bvx), a << prim_from_bv(bvy) ), [InTypeset(x.get_typevar(), ScalarTS)]), Rtl( diff --git a/lib/cretonne/meta/semantics/primitives.py b/lib/cretonne/meta/semantics/primitives.py index 62d936bc31..0a727c1cf9 100644 --- a/lib/cretonne/meta/semantics/primitives.py +++ b/lib/cretonne/meta/semantics/primitives.py @@ -82,4 +82,8 @@ bvzeroext = Instruction( 'bvzeroext', r"""Unsigned bitvector extension""", ins=x, outs=x1, constraints=WiderOrEq(ToBV, BV)) +bvsignext = Instruction( + 'bvsignext', r"""Signed bitvector extension""", + ins=x, outs=x1, constraints=WiderOrEq(ToBV, BV)) + GROUP.close() diff --git a/lib/cretonne/meta/semantics/smtlib.py b/lib/cretonne/meta/semantics/smtlib.py index f84176dc3c..c1b2526832 100644 --- a/lib/cretonne/meta/semantics/smtlib.py +++ b/lib/cretonne/meta/semantics/smtlib.py @@ -3,64 +3,74 @@ Tools to emit SMTLIB bitvector queries encoding concrete RTLs containing only primitive instructions. """ from .primitives import GROUP as PRIMITIVES, prim_from_bv, prim_to_bv, bvadd,\ - bvult, bvzeroext, bvsplit, bvconcat + bvult, bvzeroext, bvsplit, bvconcat, bvsignext from cdsl.ast import Var from cdsl.types import BVType from .elaborate import elaborate +from z3 import BitVec, ZeroExt, SignExt, And, Extract, Concat, Not, Solver,\ + unsat, BoolRef, BitVecVal, If +from z3.z3core import Z3_mk_eq try: - from typing import TYPE_CHECKING, Tuple # noqa + from typing import TYPE_CHECKING, Tuple, Dict, List # noqa from cdsl.xform import Rtl, XForm # noqa from cdsl.ast import VarMap # noqa from cdsl.ti import VarTyping # noqa + if TYPE_CHECKING: + from z3 import ExprRef, BitVecRef # noqa + Z3VarMap = Dict[Var, BitVecRef] except ImportError: TYPE_CHECKING = False -def bvtype_to_sort(typ): - # type: (BVType) -> str - """Return the BitVec sort corresponding to a BVType""" - return "(_ BitVec {})".format(typ.bits) +# Use this for constructing a == b instead of == since MyPy doesn't +# accept overloading of __eq__ that doesn't return bool +def mk_eq(e1, e2): + # type: (ExprRef, ExprRef) -> ExprRef + """Return a z3 expression equivalent to e1 == e2""" + return BoolRef(Z3_mk_eq(e1.ctx_ref(), e1.as_ast(), e2.as_ast()), e1.ctx) def to_smt(r): - # type: (Rtl) -> Tuple[str, VarMap] + # type: (Rtl) -> Tuple[List[ExprRef], Z3VarMap] """ - Encode a concrete primitive Rtl r sa SMTLIB 2.0 query. + Encode a concrete primitive Rtl r sa z3 query. Returns a tuple (query, var_m) where: - - query is the resulting query. - - var_m is a map from Vars v with non-BVType to their Vars v' with - BVType s.t. v' holds the flattend bitvector value of v. + - query is a list of z3 expressions + - var_m is a map from Vars v with non-BVType to their correspodning z3 + bitvector variable. """ assert r.is_concrete() # Should contain only primitives primitives = set(PRIMITIVES.instructions) assert set(d.expr.inst for d in r.rtl).issubset(primitives) - q = "" - m = {} # type: VarMap + q = [] # type: List[ExprRef] + m = {} # type: Z3VarMap # Build declarations for any bitvector Vars + var_to_bv = {} # type: Z3VarMap for v in r.vars(): typ = v.get_typevar().singleton_type() if not isinstance(typ, BVType): continue - q += "(declare-fun {} () {})\n".format(v.name, bvtype_to_sort(typ)) + var_to_bv[v] = BitVec(v.name, typ.bits) # Encode each instruction as a equality assertion for d in r.rtl: inst = d.expr.inst + exp = None # type: ExprRef # For prim_to_bv/prim_from_bv just update var_m. No assertion needed if inst == prim_to_bv: assert isinstance(d.expr.args[0], Var) - m[d.expr.args[0]] = d.defs[0] + m[d.expr.args[0]] = var_to_bv[d.defs[0]] continue if inst == prim_from_bv: assert isinstance(d.expr.args[0], Var) - m[d.defs[0]] = d.expr.args[0] + m[d.defs[0]] = var_to_bv[d.expr.args[0]] continue if inst in [bvadd, bvult]: # Binary instructions @@ -70,12 +80,15 @@ def to_smt(r): df = d.defs[0] assert isinstance(lhs, Var) and isinstance(rhs, Var) - if inst in [bvadd]: # Normal binary - output type same as args - exp = "(= {} ({} {} {}))".format(df, inst.name, lhs, rhs) + if inst == bvadd: # Normal binary - output type same as args + exp = (var_to_bv[lhs] + var_to_bv[rhs]) else: + assert inst == bvult + exp = (var_to_bv[lhs] < var_to_bv[rhs]) # Comparison binary - need to convert bool to BitVec 1 - exp = "(= {} (ite ({} {} {}) #b1 #b0))"\ - .format(df, inst.name, lhs, rhs) + exp = If(exp, BitVecVal(1, 1), BitVecVal(0, 1)) + + exp = mk_eq(var_to_bv[df], exp) elif inst == bvzeroext: arg = d.expr.args[0] df = d.defs[0] @@ -83,8 +96,15 @@ def to_smt(r): fromW = arg.get_typevar().singleton_type().width() toW = df.get_typevar().singleton_type().width() - exp = "(= {} ((_ zero_extend {}) {}))"\ - .format(df, toW-fromW, arg) + exp = mk_eq(var_to_bv[df], ZeroExt(toW-fromW, var_to_bv[arg])) + elif inst == bvsignext: + arg = d.expr.args[0] + df = d.defs[0] + assert isinstance(arg, Var) + fromW = arg.get_typevar().singleton_type().width() + toW = df.get_typevar().singleton_type().width() + + exp = mk_eq(var_to_bv[df], SignExt(toW-fromW, var_to_bv[arg])) elif inst == bvsplit: arg = d.expr.args[0] assert isinstance(arg, Var) @@ -95,12 +115,10 @@ def to_smt(r): lo = d.defs[0] hi = d.defs[1] - exp = "(and " - exp += "(= {} ((_ extract {} {}) {})) "\ - .format(lo, width//2-1, 0, arg) - exp += "(= {} ((_ extract {} {}) {}))"\ - .format(hi, width-1, width//2, arg) - exp += ")" + exp = And(mk_eq(var_to_bv[lo], + Extract(width//2-1, 0, var_to_bv[arg])), + mk_eq(var_to_bv[hi], + Extract(width-1, width//2, var_to_bv[arg]))) elif inst == bvconcat: assert isinstance(d.expr.args[0], Var) and \ isinstance(d.expr.args[1], Var) @@ -109,18 +127,17 @@ def to_smt(r): df = d.defs[0] # Z3 Concat expects hi bits first, then lo bits - exp = "(= {} (concat {} {}))"\ - .format(df, hi, lo) + exp = mk_eq(var_to_bv[df], Concat(var_to_bv[hi], var_to_bv[lo])) else: assert False, "Unknown primitive instruction {}".format(inst) - q += "(assert {})\n".format(exp) + q.append(exp) return (q, m) def equivalent(r1, r2, inp_m, out_m): - # type: (Rtl, Rtl, VarMap, VarMap) -> str + # type: (Rtl, Rtl, VarMap, VarMap) -> List[ExprRef] """ Given: - concrete source Rtl r1 @@ -156,36 +173,25 @@ def equivalent(r1, r2, inp_m, out_m): (q2, m2) = to_smt(r2) # Build an expression for the equality of real Cretone inputs of r1 and r2 - args_eq_exp = "(and \n" + args_eq_exp = [] # type: List[ExprRef] for v in r1.free_vars(): - args_eq_exp += "(= {} {})\n".format(m1[v], m2[inp_m[v]]) - args_eq_exp += ")" + args_eq_exp.append(mk_eq(m1[v], m2[inp_m[v]])) # Build an expression for the equality of real Cretone outputs of r1 and r2 - results_eq_exp = "(and \n" + results_eq_exp = [] # type: List[ExprRef] for (v1, v2) in out_m.items(): - results_eq_exp += "(= {} {})\n".format(m1[v1], m2[v2]) - results_eq_exp += ")" + results_eq_exp.append(mk_eq(m1[v1], m2[v2])) # Put the whole query toghether - q = '; Rtl 1 declarations and assertions\n' + q1 - q += '; Rtl 2 declarations and assertions\n' + q2 - - q += '; Assert that the inputs of Rtl1 and Rtl2 are equal\n' + \ - '(assert {})\n'.format(args_eq_exp) - - q += '; Assert that the outputs of Rtl1 and Rtl2 are not equal\n' + \ - '(assert (not {}))\n'.format(results_eq_exp) - - return q + return q1 + q2 + args_eq_exp + [Not(And(*results_eq_exp))] def xform_correct(x, typing): - # type: (XForm, VarTyping) -> str + # type: (XForm, VarTyping) -> bool """ - Given an XForm x and a concrete variable typing for x build the smtlib - query asserting that x is correct for the given typing. + Given an XForm x and a concrete variable typing for x check whether x is + semantically preserving for the concrete typing. """ assert x.ti.permits(typing) @@ -208,4 +214,8 @@ def xform_correct(x, typing): # Get the primitive semantic Rtls for src and dst prim_src = elaborate(src) prim_dst = elaborate(dst) - return equivalent(prim_src, prim_dst, inp_m, out_m) + asserts = equivalent(prim_src, prim_dst, inp_m, out_m) + + s = Solver() + s.add(*asserts) + return s.check() == unsat diff --git a/lib/cretonne/meta/stubs/z3/__init__.pyi b/lib/cretonne/meta/stubs/z3/__init__.pyi new file mode 100644 index 0000000000..2fd6c8341f --- /dev/null +++ b/lib/cretonne/meta/stubs/z3/__init__.pyi @@ -0,0 +1,151 @@ +from typing import overload, Tuple, Any, List, Iterable, Union, TypeVar +from .z3types import Ast, ContextObj + +TExprRef = TypeVar("TExprRef", bound="ExprRef") + +class Context: + ... + +class Z3PPObject: + ... + +class AstRef(Z3PPObject): + @overload + def __init__(self, ast: Ast, ctx: Context) -> None: + self.ast: Ast = ... + self.ctx: Context= ... + + @overload + def __init__(self, ast: Ast) -> None: + self.ast: Ast = ... + self.ctx: Context= ... + def ctx_ref(self) -> ContextObj: ... + def as_ast(self) -> Ast: ... + def children(self) -> List[AstRef]: ... + +class SortRef(AstRef): + ... + +class FuncDeclRef(AstRef): + def arity(self) -> int: ... + def name(self) -> str: ... + +class ExprRef(AstRef): + def eq(self, other: ExprRef) -> ExprRef: ... + def sort(self) -> SortRef: ... + def decl(self) -> FuncDeclRef: ... + +class BoolSortRef(SortRef): + ... + +class BoolRef(ExprRef): + ... + + +def is_true(a: BoolRef) -> bool: ... +def is_false(a: BoolRef) -> bool: ... +def is_int_value(a: AstRef) -> bool: ... +def substitute(a: AstRef, *m: Tuple[AstRef, AstRef]) -> AstRef: ... + + +class ArithSortRef(SortRef): + ... + +class ArithRef(ExprRef): + def __neg__(self) -> ExprRef: ... + def __le__(self, other: ArithRef) -> ArithRef: ... + def __lt__(self, other: ArithRef) -> ArithRef: ... + def __ge__(self, other: ArithRef) -> ArithRef: ... + def __gt__(self, other: ArithRef) -> ArithRef: ... + def __add__(self, other: ArithRef) -> ArithRef: ... + def __sub__(self, other: ArithRef) -> ArithRef: ... + def __mul__(self, other: ArithRef) -> ArithRef: ... + def __div__(self, other: ArithRef) -> ArithRef: ... + def __mod__(self, other: ArithRef) -> ArithRef: ... + +class IntNumRef(ArithRef): + def as_long(self) -> int: ... + +class BitVecRef(ExprRef): + def __neg__(self) -> ExprRef: ... + def __le__(self, other: BitVecRef) -> ExprRef: ... + def __lt__(self, other: BitVecRef) -> ExprRef: ... + def __ge__(self, other: BitVecRef) -> ExprRef: ... + def __gt__(self, other: BitVecRef) -> ExprRef: ... + def __add__(self, other: BitVecRef) -> BitVecRef: ... + def __sub__(self, other: BitVecRef) -> BitVecRef: ... + def __mul__(self, other: BitVecRef) -> BitVecRef: ... + def __div__(self, other: BitVecRef) -> BitVecRef: ... + def __mod__(self, other: BitVecRef) -> BitVecRef: ... + +class BitVecNumRef(BitVecRef): + def as_long(self) -> int: ... + +class CheckSatResult: ... + +class ModelRef(Z3PPObject): + def __getitem__(self, k: FuncDeclRef) -> IntNumRef: ... + def decls(self) -> Iterable[FuncDeclRef]: ... + +class Solver(Z3PPObject): + @overload + def __init__(self) -> None: + self.ctx: Context = ... + @overload + def __init__(self, ctx:Context) -> None: + self.ctx: Context = ... + + def add(self, e:ExprRef) -> None: ... + def to_smt2(self) -> str: ... + def check(self) -> CheckSatResult: ... + def push(self) -> None: ... + def pop(self) -> None: ... + def model(self) -> ModelRef: ... + +sat: CheckSatResult = ... +unsat: CheckSatResult = ... + +@overload +def Int(name: str) -> ArithRef: ... +@overload +def Int(name: str, ctx: Context) -> ArithRef: ... + +@overload +def Bool(name: str) -> BoolRef: ... +@overload +def Bool(name: str, ctx: Context) -> BoolRef: ... + +def BitVec(name: str, width: int) -> BitVecRef: ... + +@overload +def parse_smt2_string(s: str) -> ExprRef: ... +@overload +def parse_smt2_string(s: str, ctx: Context) -> ExprRef: ... + +# Can't give more precise types here since func signature is +# a vararg list of ExprRef optionally followed by a Context +def Or(*args: Union[ExprRef, Context]) -> ExprRef: ... +def And(*args: Union[ExprRef, Context]) -> ExprRef: ... +@overload +def Not(p: ExprRef) -> ExprRef: ... +@overload +def Not(p: ExprRef, ctx: Context) -> ExprRef: ... +def Implies(a: ExprRef, b: ExprRef, ctx:Context) -> ExprRef: ... +def If(a: ExprRef, b:TExprRef, c:TExprRef) -> TExprRef: ... + +def ZeroExt(width: int, expr: BitVecRef) -> BitVecRef: ... +def SignExt(width: int, expr: BitVecRef) -> BitVecRef: ... +def Extract(hi: int, lo: int, expr: BitVecRef) -> BitVecRef: ... +def Concat(expr1: BitVecRef, expr2: BitVecRef) -> BitVecRef: ... + +def Function(name: str, *sig: Tuple[SortRef,...]) -> FuncDeclRef: ... + +def IntVal(val: int, ctx: Context) -> IntNumRef: ... +@overload +def BoolVal(val: bool, ctx: Context) -> BoolRef: ... +@overload +def BoolVal(val: bool) -> BoolRef: ... +@overload +def BitVecVal(val: int, bits: int, ctx: Context) -> BitVecNumRef: ... +@overload +def BitVecVal(val: int, bits: int) -> BitVecNumRef: ... diff --git a/lib/cretonne/meta/stubs/z3/z3core.pyi b/lib/cretonne/meta/stubs/z3/z3core.pyi new file mode 100644 index 0000000000..36f1f88792 --- /dev/null +++ b/lib/cretonne/meta/stubs/z3/z3core.pyi @@ -0,0 +1,3 @@ +from .z3types import Ast, ContextObj +def Z3_mk_eq(ctx: ContextObj, a: Ast, b: Ast) -> Ast: ... +def Z3_mk_div(ctx: ContextObj, a: Ast, b: Ast) -> Ast: ... diff --git a/lib/cretonne/meta/stubs/z3/z3types.pyi b/lib/cretonne/meta/stubs/z3/z3types.pyi new file mode 100644 index 0000000000..fa8fc446d1 --- /dev/null +++ b/lib/cretonne/meta/stubs/z3/z3types.pyi @@ -0,0 +1,12 @@ +from typing import Any + +class Z3Exception(Exception): + def __init__(self, a: Any) -> None: + self.value = a + ... + +class ContextObj: + ... + +class Ast: + ... From 7d9edc2d5ee20bf3fda084873bb5fe389466d96f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 31 Jul 2017 12:58:20 -0700 Subject: [PATCH 0943/3084] Keep track of OutgoingArg stack slots. Stack slots for outgoing arguments can be reused between function calls. Add a list of outgoing argument stack slots allocated so far, and provide a `get_outgoing_arg()` method which will reuse any outgoing stack slots with matching size and offset. --- lib/cretonne/src/ir/stackslot.rs | 63 +++++++++++++++++++++++++++++++- 1 file changed, 62 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index c6fd64ce5c..56dd037dc5 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -75,6 +75,9 @@ pub struct StackSlotData { /// On Intel ISAs, the base address is the stack pointer *before* the return address was /// pushed. On RISC ISAs, the base address is the value of the stack pointer on entry to the /// function. + /// + /// For `OutgoingArg` stack slots, the offset is relative to the current function's stack + /// pointer immediately before the call. pub offset: i32, } @@ -106,19 +109,27 @@ impl PrimaryEntityData for StackSlotData {} /// Keep track of all the stack slots used by a function. #[derive(Clone, Debug)] pub struct StackSlots { + /// All allocated stack slots. slots: EntityMap, + + /// All the outgoing stack slots, ordered by offset. + outgoing: Vec, } /// Stack slot manager functions that behave mostly like an entity map. impl StackSlots { /// Create an empty stack slot manager. pub fn new() -> StackSlots { - StackSlots { slots: EntityMap::new() } + StackSlots { + slots: EntityMap::new(), + outgoing: Vec::new(), + } } /// Clear out everything. pub fn clear(&mut self) { self.slots.clear(); + self.outgoing.clear(); } /// Allocate a new stack slot. @@ -161,6 +172,33 @@ impl StackSlots { data.offset = offset; self.push(data) } + + /// Get a stack slot representing an outgoing argument. + /// + /// This may create a new stack slot, or reuse an existing outgoing stack slot with the + /// requested offset and size. + /// + /// The requested offset is relative to this function's stack pointer immediately before making + /// the call. + pub fn get_outgoing_arg(&mut self, ty: Type, offset: i32) -> StackSlot { + let size = ty.bytes(); + + // Look for an existing outgoing stack slot with the same offset and size. + let inspos = match self.outgoing + .binary_search_by_key(&(offset, size), + |&ss| (self[ss].offset, self[ss].size)) { + Ok(idx) => return self.outgoing[idx], + Err(idx) => idx, + }; + + // No existing slot found. Make one and insert it into `outgoing`. + let mut data = StackSlotData::new(StackSlotKind::OutgoingArg, size); + assert!(offset <= i32::max_value() - size as i32); + data.offset = offset; + let ss = self.slots.push(data); + self.outgoing.insert(inspos, ss); + ss + } } impl Index for StackSlots { @@ -174,6 +212,7 @@ impl Index for StackSlots { #[cfg(test)] mod tests { use ir::Function; + use ir::types; use super::*; #[test] @@ -193,4 +232,26 @@ mod tests { assert_eq!(func.stack_slots[ss0].to_string(), "incoming_arg 4"); assert_eq!(func.stack_slots[ss1].to_string(), "spill_slot 8"); } + + #[test] + fn outgoing() { + let mut sss = StackSlots::new(); + + let ss0 = sss.get_outgoing_arg(types::I32, 8); + let ss1 = sss.get_outgoing_arg(types::I32, 4); + let ss2 = sss.get_outgoing_arg(types::I64, 8); + + assert_eq!(sss[ss0].offset, 8); + assert_eq!(sss[ss0].size, 4); + + assert_eq!(sss[ss1].offset, 4); + assert_eq!(sss[ss1].size, 4); + + assert_eq!(sss[ss2].offset, 8); + assert_eq!(sss[ss2].size, 8); + + assert_eq!(sss.get_outgoing_arg(types::I32, 8), ss0); + assert_eq!(sss.get_outgoing_arg(types::I32, 4), ss1); + assert_eq!(sss.get_outgoing_arg(types::I64, 8), ss2); + } } From ce9e6fe53ea81c16ac61121d030dd5605d113f06 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 31 Jul 2017 15:04:41 -0700 Subject: [PATCH 0944/3084] Verify the location of outgoing call arguments. Once a signature has been legalized, the arguments to any call using that signature must be assigned to the proper stack locations. Outgoing arguments that are passed on the stack must be assigned to matching OutgoingArg stack slot locations. Outgoing arguments that are passed in registers don't need to appear in the correct registers until after register allocation. --- lib/cretonne/src/verifier/mod.rs | 62 +++++++++++++++++++++++++++++++- 1 file changed, 61 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index a1e04fbb79..751f6d5f6f 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -57,7 +57,7 @@ use flowgraph::ControlFlowGraph; use ir::entities::AnyEntity; use ir::instructions::{InstructionFormat, BranchInfo, ResolvedConstraint, CallInfo}; use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, StackSlot, - Value, Type, Opcode}; + StackSlotKind, Value, Type, Opcode, ValueLoc, ArgumentLoc}; use isa::TargetIsa; use std::error as std_error; use std::fmt::{self, Display, Formatter}; @@ -552,6 +552,7 @@ impl<'a> Verifier<'a> { .iter() .map(|a| a.value_type); self.typecheck_variable_args_iterator(inst, arg_types)?; + self.check_outgoing_args(inst, sig_ref)?; } CallInfo::Indirect(sig_ref, _) => { let arg_types = self.func.dfg.signatures[sig_ref] @@ -559,6 +560,7 @@ impl<'a> Verifier<'a> { .iter() .map(|a| a.value_type); self.typecheck_variable_args_iterator(inst, arg_types)?; + self.check_outgoing_args(inst, sig_ref)?; } CallInfo::NotACall => {} } @@ -599,6 +601,64 @@ impl<'a> Verifier<'a> { Ok(()) } + /// Check the locations assigned to outgoing call arguments. + /// + /// When a signature has been legalized, all values passed as outgoing arguments on the stack + /// must be assigned to a matching `OutgoingArg` stack slot. + fn check_outgoing_args(&self, inst: Inst, sig_ref: SigRef) -> Result { + let sig = &self.func.dfg.signatures[sig_ref]; + + // Before legalization, there's nothing to check. + if sig.argument_bytes.is_none() { + return Ok(()); + } + + let args = self.func.dfg.inst_variable_args(inst); + let expected_args = &sig.argument_types[..]; + + for (&arg, &abi) in args.iter().zip(expected_args) { + // Value types have already been checked by `typecheck_variable_args_iterator()`. + if let ArgumentLoc::Stack(offset) = abi.location { + let arg_loc = self.func.locations.get_or_default(arg); + if let ValueLoc::Stack(ss) = arg_loc { + // Argument value is assigned to a stack slot as expected. + self.verify_stack_slot(inst, ss)?; + let slot = &self.func.stack_slots[ss]; + if slot.kind != StackSlotKind::OutgoingArg { + return err!(inst, + "Outgoing stack argument {} in wrong stack slot: {} = {}", + arg, + ss, + slot); + } + if slot.offset != offset { + return err!(inst, + "Outgoing stack argument {} should have offset {}: {} = {}", + arg, + offset, + ss, + slot); + } + if slot.size != abi.value_type.bytes() { + return err!(inst, + "Outgoing stack argument {} wrong size for {}: {} = {}", + arg, + abi.value_type, + ss, + slot); + } + } else { + let reginfo = self.isa.map(|i| i.register_info()); + return err!(inst, + "Outgoing stack argument {} in wrong location: {}", + arg, + arg_loc.display(reginfo.as_ref())); + } + } + } + Ok(()) + } + fn typecheck_return(&self, inst: Inst) -> Result { if self.func.dfg[inst].opcode().is_return() { let args = self.func.dfg.inst_variable_args(inst); From f03f32ac9318458b37211f829eb497aafe0f02a8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 1 Aug 2017 13:32:30 -0700 Subject: [PATCH 0945/3084] Assign call arguments to stack slots. When making an outgoing call, some arguments may have to be passed on the stack. Allocate OutgoingArg stack slots for these arguments and write them immediately before the outgoing call instruction. Do the same for incoming function arguments on the stack, but use IncomingArg stack slots instead. This was previously done in the spiller, but we move it to the legalizer so it is done at the same time as outgoing stack arguments. These stack slot assignments are done in the legalizer before live range analysis because the outgoing arguments usually are in different SSSA values with their own short live ranges. --- .../filetests/isa/riscv/legalize-abi.cton | 14 +++ lib/cretonne/src/ir/extfunc.rs | 2 +- lib/cretonne/src/legalizer/boundary.rs | 99 ++++++++++++++++++- lib/cretonne/src/legalizer/mod.rs | 7 +- lib/cretonne/src/regalloc/spilling.rs | 29 +----- 5 files changed, 118 insertions(+), 33 deletions(-) diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index 4d6f4e6e7f..cb94836ab4 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -3,6 +3,8 @@ test legalizer isa riscv ; regex: V=v\d+ +; regex: SS=ss\d+ +; regex: WS=\s+ function %int_split_args(i64) -> i64 { ebb0(v0: i64): @@ -118,3 +120,15 @@ ebb0(v0: i32, v1: f32x2): ; check: call_indirect $sig1, $v0($V, $V) return } + +; Call a function that takes arguments on the stack. +function %stack_args(i32) { + ; check: $(ss0=$SS) = outgoing_arg 4 + fn1 = function %foo(i64, i64, i64, i64, i32) +ebb0(v0: i32): + v1 = iconst.i64 1 + call fn1(v1, v1, v1, v1, v0) + ; check: [GPsp#48,$ss0]$WS $(v0s=$V) = spill $v0 + ; check: call $fn1($(=.*), $v0s) + return +} diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index e46ef39a54..4cf5b9d61f 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -51,7 +51,7 @@ impl Signature { let bytes = self.argument_types .iter() .filter_map(|arg| match arg.location { - ArgumentLoc::Stack(offset) if offset > 0 => { + ArgumentLoc::Stack(offset) if offset >= 0 => { Some(offset as u32 + arg.value_type.bytes()) } _ => None, diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 3f22694eb8..caed486f3a 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -20,7 +20,8 @@ use abi::{legalize_abi_value, ValueConversion}; use flowgraph::ControlFlowGraph; use ir::{Function, Cursor, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, Signature, SigRef, - ArgumentType, ArgumentPurpose}; + ArgumentType, ArgumentPurpose, ArgumentLoc, ValueLoc, ValueLocations, StackSlots, + StackSlotKind}; use ir::instructions::CallInfo; use isa::TargetIsa; use legalizer::split::{isplit, vsplit}; @@ -32,12 +33,15 @@ use legalizer::split::{isplit, vsplit}; /// in a state with type discrepancies. pub fn legalize_signatures(func: &mut Function, isa: &TargetIsa) { isa.legalize_signature(&mut func.signature, true); + func.signature.compute_argument_bytes(); for sig in func.dfg.signatures.keys() { isa.legalize_signature(&mut func.dfg.signatures[sig], false); + func.dfg.signatures[sig].compute_argument_bytes(); } if let Some(entry) = func.layout.entry_block() { legalize_entry_arguments(func, entry); + spill_entry_arguments(func, entry); } } @@ -448,13 +452,18 @@ fn legalize_inst_arguments(dfg: &mut DataFlowGraph, /// original return values. The call's result values will be adapted to match the new signature. /// /// Returns `true` if any instructions were inserted. -pub fn handle_call_abi(dfg: &mut DataFlowGraph, cfg: &ControlFlowGraph, pos: &mut Cursor) -> bool { +pub fn handle_call_abi(dfg: &mut DataFlowGraph, + locations: &mut ValueLocations, + stack_slots: &mut StackSlots, + cfg: &ControlFlowGraph, + pos: &mut Cursor) + -> bool { let mut inst = pos.current_inst() .expect("Cursor must point to a call instruction"); // Start by checking if the argument types already match the signature. let sig_ref = match check_call_signature(dfg, inst) { - Ok(_) => return false, + Ok(_) => return spill_call_arguments(dfg, locations, stack_slots, pos), Err(s) => s, }; @@ -478,6 +487,10 @@ pub fn handle_call_abi(dfg: &mut DataFlowGraph, cfg: &ControlFlowGraph, pos: &mu sig_ref, dfg.signatures[sig_ref]); + // Go back and insert spills for any stack arguments. + pos.goto_inst(inst); + spill_call_arguments(dfg, locations, stack_slots, pos); + // Yes, we changed stuff. true } @@ -556,3 +569,83 @@ pub fn handle_return_abi(dfg: &mut DataFlowGraph, // Yes, we changed stuff. true } + +/// Assign stack slots to incoming function arguments on the stack. +/// +/// Values that are passed into the function on the stack must be assigned to an `IncomingArg` +/// stack slot already during legalization. +fn spill_entry_arguments(func: &mut Function, entry: Ebb) { + for (abi, &arg) in func.signature + .argument_types + .iter() + .zip(func.dfg.ebb_args(entry)) { + if let ArgumentLoc::Stack(offset) = abi.location { + let ss = func.stack_slots.make_incoming_arg(abi.value_type, offset); + *func.locations.ensure(arg) = ValueLoc::Stack(ss); + } + } +} + +/// Assign stack slots to outgoing function arguments on the stack. +/// +/// Values that are passed to a called function on the stack must be assigned to a matching +/// `OutgoingArg` stack slot. The assignment must happen immediately before the call. +/// +/// TODO: The outgoing stack slots can be written a bit earlier, as long as there are no branches +/// or calls between writing the stack slots and the call instruction. Writing the slots earlier +/// could help reduce register pressure before the call. +fn spill_call_arguments(dfg: &mut DataFlowGraph, + locations: &mut ValueLocations, + stack_slots: &mut StackSlots, + pos: &mut Cursor) + -> bool { + let inst = pos.current_inst() + .expect("Cursor must point to a call instruction"); + let sig_ref = dfg.call_signature(inst) + .expect("Call instruction expected."); + + // Start by building a list of stack slots and arguments to be replaced. + // This requires borrowing `dfg`, so we can't change anything. + let arglist = dfg.inst_variable_args(inst) + .iter() + .zip(&dfg.signatures[sig_ref].argument_types) + .enumerate() + .filter_map(|(idx, (&arg, abi))| { + match abi.location { + ArgumentLoc::Stack(offset) => { + // Is `arg` already in the right kind of stack slot? + match locations.get(arg) { + Some(&ValueLoc::Stack(ss)) => { + // We won't reassign `arg` to a different stack slot. Assert out of + // the stack slot is wrong. + assert_eq!(stack_slots[ss].kind, StackSlotKind::OutgoingArg); + assert_eq!(stack_slots[ss].offset, offset); + assert_eq!(stack_slots[ss].size, abi.value_type.bytes()); + None + } + _ => { + // Assign `arg` to a new stack slot. + let ss = stack_slots.get_outgoing_arg(abi.value_type, offset); + Some((idx, arg, ss)) + } + } + } + _ => None, + } + }) + .collect::>(); + + if arglist.is_empty() { + return false; + } + + // Insert the spill instructions and rewrite call arguments. + for (idx, arg, ss) in arglist { + let stack_val = dfg.ins(pos).spill(arg); + *locations.ensure(stack_val) = ValueLoc::Stack(ss); + dfg.inst_variable_args_mut(inst)[idx] = stack_val; + } + + // We changed stuff. + true +} diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index c6a844c3c1..4494eb60e9 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -51,7 +51,12 @@ pub fn legalize_function(func: &mut Function, let opcode = func.dfg[inst].opcode(); // Check for ABI boundaries that need to be converted to the legalized signature. - if opcode.is_call() && boundary::handle_call_abi(&mut func.dfg, cfg, &mut pos) { + if opcode.is_call() && + boundary::handle_call_abi(&mut func.dfg, + &mut func.locations, + &mut func.stack_slots, + cfg, + &mut pos) { // Go back and legalize the inserted argument conversion instructions. pos.set_position(prev_pos); continue; diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 49f6d67d09..8b3e8b7837 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -17,7 +17,7 @@ use dominator_tree::DominatorTree; use ir::{DataFlowGraph, Layout, Cursor, InstBuilder}; -use ir::{Function, Ebb, Inst, Value, ValueLoc, ArgumentLoc, Signature, SigRef}; +use ir::{Function, Ebb, Inst, Value, ValueLoc, SigRef}; use ir::{InstEncodings, StackSlots, ValueLocations}; use isa::registers::{RegClassMask, RegClassIndex}; use isa::{TargetIsa, RegInfo, EncInfo, RecipeConstraints, ConstraintKind}; @@ -46,7 +46,6 @@ struct Context<'a> { encodings: &'a mut InstEncodings, stack_slots: &'a mut StackSlots, locations: &'a mut ValueLocations, - func_signature: &'a Signature, // References to contextual data structures we need. domtree: &'a DominatorTree, @@ -94,7 +93,6 @@ impl Spilling { encodings: &mut func.encodings, stack_slots: &mut func.stack_slots, locations: &mut func.locations, - func_signature: &func.signature, domtree, liveness, virtregs, @@ -112,37 +110,12 @@ impl<'a> Context<'a> { layout: &mut Layout, dfg: &mut DataFlowGraph, tracker: &mut LiveValueTracker) { - if let Some(entry) = layout.entry_block() { - self.spill_entry_arguments(entry, dfg); - } - self.topo.reset(layout.ebbs()); while let Some(ebb) = self.topo.next(layout, self.domtree) { self.visit_ebb(ebb, layout, dfg, tracker); } } - /// Assign stack slots to incoming function arguments on the stack. - fn spill_entry_arguments(&mut self, entry: Ebb, dfg: &DataFlowGraph) { - for (abi, &arg) in self.func_signature - .argument_types - .iter() - .zip(dfg.ebb_args(entry)) { - if let ArgumentLoc::Stack(offset) = abi.location { - // Function arguments passed on the stack can't be part of a virtual register. We - // would need to write other values to the stack slot, but it belongs to the - // caller. (Not that the caller would care, nobody depends on stack arguments being - // preserved across calls). - assert_eq!(self.virtregs.get(arg), - None, - "Stack argument {} can't be part of a virtual register", - arg); - let ss = self.stack_slots.make_incoming_arg(abi.value_type, offset); - *self.locations.ensure(arg) = ValueLoc::Stack(ss); - } - } - } - fn visit_ebb(&mut self, ebb: Ebb, layout: &mut Layout, From 406d82f62a9602e75d1d3f5ebea49af04f1afdf4 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 2 Aug 2017 11:05:49 -0700 Subject: [PATCH 0946/3084] Avoid floating-point types in Ieee32::new and Ieee64::new. (#130) * Avoid floating-point types in Ieee32::new and Ieee64::new. This eliminates the need for unsafe code in code that uses Cretonne, a few instances of unsafe code in Cretonne itself, and eliminates the only instance of floating point in Cretonne. * Rename new to with_bits, and new_from_float to with_float. --- lib/cretonne/src/ir/immediates.rs | 83 +++++++++++++++++++------------ lib/frontend/src/ssa.rs | 4 +- 2 files changed, 52 insertions(+), 35 deletions(-) diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index fa6185cb8b..c525305a19 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -7,9 +7,11 @@ use std::fmt::{self, Display, Formatter}; use std::{i32, u32}; -use std::mem; use std::str::FromStr; +#[cfg(test)] +use std::mem; + /// 64-bit immediate integer operand. /// /// An `Imm64` operand can also be used to represent immediate values of smaller integer types by @@ -531,8 +533,14 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { } impl Ieee32 { + /// Create a new `Ieee32` containing the bits of `x`. + pub fn with_bits(x: u32) -> Ieee32 { + Ieee32(x) + } + /// Create a new `Ieee32` representing the number `x`. - pub fn new(x: f32) -> Ieee32 { + #[cfg(test)] + pub fn with_float(x: f32) -> Ieee32 { Ieee32(unsafe { mem::transmute(x) }) } } @@ -556,8 +564,14 @@ impl FromStr for Ieee32 { } impl Ieee64 { + /// Create a new `Ieee64` containing the bits of `x`. + pub fn with_bits(x: u64) -> Ieee64 { + Ieee64(x) + } + /// Create a new `Ieee64` representing the number `x`. - pub fn new(x: f64) -> Ieee64 { + #[cfg(test)] + pub fn with_float(x: f64) -> Ieee64 { Ieee64(unsafe { mem::transmute(x) }) } } @@ -714,26 +728,27 @@ mod tests { #[test] fn format_ieee32() { - assert_eq!(Ieee32::new(0.0).to_string(), "0.0"); - assert_eq!(Ieee32::new(-0.0).to_string(), "-0.0"); - assert_eq!(Ieee32::new(1.0).to_string(), "0x1.000000p0"); - assert_eq!(Ieee32::new(1.5).to_string(), "0x1.800000p0"); - assert_eq!(Ieee32::new(0.5).to_string(), "0x1.000000p-1"); - assert_eq!(Ieee32::new(f32::EPSILON).to_string(), "0x1.000000p-23"); - assert_eq!(Ieee32::new(f32::MIN).to_string(), "-0x1.fffffep127"); - assert_eq!(Ieee32::new(f32::MAX).to_string(), "0x1.fffffep127"); + assert_eq!(Ieee32::with_float(0.0).to_string(), "0.0"); + assert_eq!(Ieee32::with_float(-0.0).to_string(), "-0.0"); + assert_eq!(Ieee32::with_float(1.0).to_string(), "0x1.000000p0"); + assert_eq!(Ieee32::with_float(1.5).to_string(), "0x1.800000p0"); + assert_eq!(Ieee32::with_float(0.5).to_string(), "0x1.000000p-1"); + assert_eq!(Ieee32::with_float(f32::EPSILON).to_string(), + "0x1.000000p-23"); + assert_eq!(Ieee32::with_float(f32::MIN).to_string(), "-0x1.fffffep127"); + assert_eq!(Ieee32::with_float(f32::MAX).to_string(), "0x1.fffffep127"); // Smallest positive normal number. - assert_eq!(Ieee32::new(f32::MIN_POSITIVE).to_string(), + assert_eq!(Ieee32::with_float(f32::MIN_POSITIVE).to_string(), "0x1.000000p-126"); // Subnormals. - assert_eq!(Ieee32::new(f32::MIN_POSITIVE / 2.0).to_string(), + assert_eq!(Ieee32::with_float(f32::MIN_POSITIVE / 2.0).to_string(), "0x0.800000p-126"); - assert_eq!(Ieee32::new(f32::MIN_POSITIVE * f32::EPSILON).to_string(), + assert_eq!(Ieee32::with_float(f32::MIN_POSITIVE * f32::EPSILON).to_string(), "0x0.000002p-126"); - assert_eq!(Ieee32::new(f32::INFINITY).to_string(), "+Inf"); - assert_eq!(Ieee32::new(f32::NEG_INFINITY).to_string(), "-Inf"); - assert_eq!(Ieee32::new(f32::NAN).to_string(), "+NaN"); - assert_eq!(Ieee32::new(-f32::NAN).to_string(), "-NaN"); + assert_eq!(Ieee32::with_float(f32::INFINITY).to_string(), "+Inf"); + assert_eq!(Ieee32::with_float(f32::NEG_INFINITY).to_string(), "-Inf"); + assert_eq!(Ieee32::with_float(f32::NAN).to_string(), "+NaN"); + assert_eq!(Ieee32::with_float(-f32::NAN).to_string(), "-NaN"); // Construct some qNaNs with payloads. assert_eq!(Ieee32(0x7fc00001).to_string(), "+NaN:0x1"); assert_eq!(Ieee32(0x7ff00001).to_string(), "+NaN:0x300001"); @@ -815,27 +830,29 @@ mod tests { #[test] fn format_ieee64() { - assert_eq!(Ieee64::new(0.0).to_string(), "0.0"); - assert_eq!(Ieee64::new(-0.0).to_string(), "-0.0"); - assert_eq!(Ieee64::new(1.0).to_string(), "0x1.0000000000000p0"); - assert_eq!(Ieee64::new(1.5).to_string(), "0x1.8000000000000p0"); - assert_eq!(Ieee64::new(0.5).to_string(), "0x1.0000000000000p-1"); - assert_eq!(Ieee64::new(f64::EPSILON).to_string(), + assert_eq!(Ieee64::with_float(0.0).to_string(), "0.0"); + assert_eq!(Ieee64::with_float(-0.0).to_string(), "-0.0"); + assert_eq!(Ieee64::with_float(1.0).to_string(), "0x1.0000000000000p0"); + assert_eq!(Ieee64::with_float(1.5).to_string(), "0x1.8000000000000p0"); + assert_eq!(Ieee64::with_float(0.5).to_string(), "0x1.0000000000000p-1"); + assert_eq!(Ieee64::with_float(f64::EPSILON).to_string(), "0x1.0000000000000p-52"); - assert_eq!(Ieee64::new(f64::MIN).to_string(), "-0x1.fffffffffffffp1023"); - assert_eq!(Ieee64::new(f64::MAX).to_string(), "0x1.fffffffffffffp1023"); + assert_eq!(Ieee64::with_float(f64::MIN).to_string(), + "-0x1.fffffffffffffp1023"); + assert_eq!(Ieee64::with_float(f64::MAX).to_string(), + "0x1.fffffffffffffp1023"); // Smallest positive normal number. - assert_eq!(Ieee64::new(f64::MIN_POSITIVE).to_string(), + assert_eq!(Ieee64::with_float(f64::MIN_POSITIVE).to_string(), "0x1.0000000000000p-1022"); // Subnormals. - assert_eq!(Ieee64::new(f64::MIN_POSITIVE / 2.0).to_string(), + assert_eq!(Ieee64::with_float(f64::MIN_POSITIVE / 2.0).to_string(), "0x0.8000000000000p-1022"); - assert_eq!(Ieee64::new(f64::MIN_POSITIVE * f64::EPSILON).to_string(), + assert_eq!(Ieee64::with_float(f64::MIN_POSITIVE * f64::EPSILON).to_string(), "0x0.0000000000001p-1022"); - assert_eq!(Ieee64::new(f64::INFINITY).to_string(), "+Inf"); - assert_eq!(Ieee64::new(f64::NEG_INFINITY).to_string(), "-Inf"); - assert_eq!(Ieee64::new(f64::NAN).to_string(), "+NaN"); - assert_eq!(Ieee64::new(-f64::NAN).to_string(), "-NaN"); + assert_eq!(Ieee64::with_float(f64::INFINITY).to_string(), "+Inf"); + assert_eq!(Ieee64::with_float(f64::NEG_INFINITY).to_string(), "-Inf"); + assert_eq!(Ieee64::with_float(f64::NAN).to_string(), "+NaN"); + assert_eq!(Ieee64::with_float(-f64::NAN).to_string(), "-NaN"); // Construct some qNaNs with payloads. assert_eq!(Ieee64(0x7ff8000000000001).to_string(), "+NaN:0x1"); assert_eq!(Ieee64(0x7ffc000000000001).to_string(), diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 46b44d5251..3794d332b1 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -490,9 +490,9 @@ impl SSABuilder let val = if ty.is_int() { dfg.ins(&mut cur).iconst(ty, 0) } else if ty == F32 { - dfg.ins(&mut cur).f32const(Ieee32::new(0.0)) + dfg.ins(&mut cur).f32const(Ieee32::with_bits(0)) } else if ty == F64 { - dfg.ins(&mut cur).f64const(Ieee64::new(0.0)) + dfg.ins(&mut cur).f64const(Ieee64::with_bits(0)) } else { panic!("value used but never declared and initialization not supported") }; From 587f37ec6cfa1297824c621c2724fca3c943b192 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 2 Aug 2017 11:19:16 -0700 Subject: [PATCH 0947/3084] Don't use documentation-style comments inside a function body. (#131) --- lib/cretonne/src/regalloc/coalescing.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 964644fad3..29051bf426 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -383,8 +383,8 @@ impl<'a> Context<'a> { succ_val: Value, preds: &[BasicBlock]) -> Option { - /// Initialize the value list with the split values. These are guaranteed to be - /// interference free, and anything that interferes with them must be split away. + // Initialize the value list with the split values. These are guaranteed to be + // interference free, and anything that interferes with them must be split away. self.reset_values(); dbg!("Trying {} with split values: {:?}", succ_val, self.values); From 5edf48b5488911d6127f1c3cbc5762094b0b2203 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 2 Aug 2017 12:58:53 -0700 Subject: [PATCH 0948/3084] Compute the stack frame layout. Add a StackSlots::layout() method which computes the total stack frame size and assigns offsets to all spill slots and local variables so they don't interfere with each other or with incoming or outgoing function arguments. Stack slots are given an ad hoc alignment that is the natural alignment for power-of-two sized spill slots, up to the stack pointer alignment. It is possible we need explicit stack slot alignment in the future, but at least for spill slots, this scheme is likely to work for most ISAs. --- lib/cretonne/src/ir/stackslot.rs | 242 +++++++++++++++++++++++++++++-- 1 file changed, 230 insertions(+), 12 deletions(-) diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index 56dd037dc5..ea28eeae1a 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -5,10 +5,23 @@ use entity_map::{EntityMap, PrimaryEntityData, Keys}; use ir::{Type, StackSlot}; +use std::cmp::{min, max}; use std::fmt; use std::ops::Index; use std::str::FromStr; +/// The size of an object on the stack, or the size of a stack frame. +/// +/// We don't use `usize` to represent object sizes on the target platform because Cretonne supports +/// cross-compilation, and `usize` is a type that depends on the host platform, not the target +/// platform. +type StackSize = u32; + +/// A stack offset. +/// +/// The location of a stack offset relative to a stack pointer or frame pointer. +type StackOffset = i32; + /// The kind of a stack slot. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum StackSlotKind { @@ -68,7 +81,7 @@ pub struct StackSlotData { pub kind: StackSlotKind, /// Size of stack slot in bytes. - pub size: u32, + pub size: StackSize, /// Offset of stack slot relative to the stack pointer in the caller. /// @@ -78,18 +91,28 @@ pub struct StackSlotData { /// /// For `OutgoingArg` stack slots, the offset is relative to the current function's stack /// pointer immediately before the call. - pub offset: i32, + pub offset: StackOffset, } impl StackSlotData { /// Create a stack slot with the specified byte size. - pub fn new(kind: StackSlotKind, size: u32) -> StackSlotData { + pub fn new(kind: StackSlotKind, size: StackSize) -> StackSlotData { StackSlotData { kind, size, offset: 0, } } + + /// Get the alignment in bytes of this stack slot given the stack pointer alignment. + pub fn alignment(&self, max_align: StackSize) -> StackSize { + debug_assert!(max_align.is_power_of_two()); + // We want to find the largest power of two that divides both `self.size` and `max_align`. + // That is the same as isolating the rightmost bit in `x`. + let x = self.size | max_align; + // C.f. Hacker's delight. + x & x.wrapping_neg() + } } impl fmt::Display for StackSlotData { @@ -114,6 +137,15 @@ pub struct StackSlots { /// All the outgoing stack slots, ordered by offset. outgoing: Vec, + + /// The total size of the stack frame. + /// + /// This is the distance from the stack pointer in the current function to the stack pointer in + /// the calling function, so it includes a pushed return address as well as space for outgoing + /// call arguments. + /// + /// This is computed by the `layout()` method. + pub frame_size: Option, } /// Stack slot manager functions that behave mostly like an entity map. @@ -123,6 +155,7 @@ impl StackSlots { StackSlots { slots: EntityMap::new(), outgoing: Vec::new(), + frame_size: None, } } @@ -130,6 +163,7 @@ impl StackSlots { pub fn clear(&mut self) { self.slots.clear(); self.outgoing.clear(); + self.frame_size = None; } /// Allocate a new stack slot. @@ -158,6 +192,14 @@ impl StackSlots { } } +impl Index for StackSlots { + type Output = StackSlotData; + + fn index(&self, ss: StackSlot) -> &StackSlotData { + &self.slots[ss] + } +} + /// Higher-level stack frame manipulation functions. impl StackSlots { /// Create a new spill slot for spilling values of type `ty`. @@ -166,9 +208,9 @@ impl StackSlots { } /// Create a stack slot representing an incoming function argument. - pub fn make_incoming_arg(&mut self, ty: Type, offset: i32) -> StackSlot { + pub fn make_incoming_arg(&mut self, ty: Type, offset: StackOffset) -> StackSlot { let mut data = StackSlotData::new(StackSlotKind::IncomingArg, ty.bytes()); - assert!(offset <= i32::max_value() - data.size as i32); + assert!(offset <= StackOffset::max_value() - data.size as StackOffset); data.offset = offset; self.push(data) } @@ -180,7 +222,7 @@ impl StackSlots { /// /// The requested offset is relative to this function's stack pointer immediately before making /// the call. - pub fn get_outgoing_arg(&mut self, ty: Type, offset: i32) -> StackSlot { + pub fn get_outgoing_arg(&mut self, ty: Type, offset: StackOffset) -> StackSlot { let size = ty.bytes(); // Look for an existing outgoing stack slot with the same offset and size. @@ -193,19 +235,108 @@ impl StackSlots { // No existing slot found. Make one and insert it into `outgoing`. let mut data = StackSlotData::new(StackSlotKind::OutgoingArg, size); - assert!(offset <= i32::max_value() - size as i32); + assert!(offset <= StackOffset::max_value() - size as StackOffset); data.offset = offset; let ss = self.slots.push(data); self.outgoing.insert(inspos, ss); ss } -} -impl Index for StackSlots { - type Output = StackSlotData; + /// Compute the stack frame layout. + /// + /// Determine the total size of this function's stack frame and assign offsets to all `Spill` + /// and `Local` stack slots. + /// + /// The total frame size will be a multiple of `alignment` which must be a power of two. + /// + /// Returns the total stack frame size which is also saved in `self.frame_size`. + pub fn layout(&mut self, alignment: StackSize) -> StackSize { + assert!(alignment.is_power_of_two() && alignment <= StackOffset::max_value() as StackSize, + "Invalid stack alignment {}", + alignment); - fn index(&self, ss: StackSlot) -> &StackSlotData { - &self.slots[ss] + // We assume a stack that grows toward lower addresses as implemented by modern ISAs. The + // stack layout from high to low addresses will be: + // + // 1. incoming arguments. + // 2. spills + locals. + // 3. outgoing arguments. + // + // The incoming arguments can have both positive and negative offsets. A negative offset + // incoming arguments is usually the x86 return address pushed by the call instruction, but + // it can also be fixed stack slots pushed by an externally generated prologue. + // + // Both incoming and outgoing argument slots have fixed offsets that are treated as + // reserved zones by the layout algorithm. + + let mut incoming_min = 0; + let mut outgoing_max = 0; + let mut min_align = alignment; + + for ss in self.keys() { + let slot = &self[ss]; + assert!(slot.size <= StackOffset::max_value() as StackSize); + match slot.kind { + StackSlotKind::IncomingArg => { + incoming_min = min(incoming_min, slot.offset); + } + StackSlotKind::OutgoingArg => { + let offset = slot.offset + .checked_add(slot.size as StackOffset) + .expect("Outgoing call argument overflows stack"); + outgoing_max = max(outgoing_max, offset); + } + StackSlotKind::SpillSlot | StackSlotKind::Local => { + // Determine the smallest alignment of any local or spill slot. + min_align = slot.alignment(min_align); + } + } + } + + // Lay out spill slots and locals below the incoming arguments. + // The offset is negative, growing downwards. + // Start with the smallest alignments for better packing. + let mut offset = incoming_min; + assert!(min_align.is_power_of_two()); + while min_align <= alignment { + for ss in self.keys() { + let slot = &mut self.slots[ss]; + + // Pick out locals and spill slots with exact alignment `min_align`. + match slot.kind { + StackSlotKind::SpillSlot | StackSlotKind::Local => { + if slot.alignment(alignment) != min_align { + continue; + } + } + _ => continue, + } + + // These limits should never be exceeded by spill slots, but locals can be + // arbitrarily large. + assert!(slot.size <= StackOffset::max_value() as StackSize); + offset = offset + .checked_sub(slot.size as StackOffset) + .expect("Stack frame larger than 2 GB"); + + // Aligning the negative offset can never cause overflow. We're only clearing bits. + offset &= -(min_align as StackOffset); + slot.offset = offset; + } + + // Move on to the next higher alignment. + min_align *= 2; + } + + // Finally, make room for the outgoing arguments. + offset = offset + .checked_sub(outgoing_max) + .expect("Stack frame larger than 2 GB"); + offset &= -(alignment as StackOffset); + + let frame_size = (offset as StackSize).wrapping_neg(); + self.frame_size = Some(frame_size); + frame_size } } @@ -254,4 +385,91 @@ mod tests { assert_eq!(sss.get_outgoing_arg(types::I32, 4), ss1); assert_eq!(sss.get_outgoing_arg(types::I64, 8), ss2); } + + #[test] + fn alignment() { + let slot = StackSlotData::new(StackSlotKind::SpillSlot, 8); + + assert_eq!(slot.alignment(4), 4); + assert_eq!(slot.alignment(8), 8); + assert_eq!(slot.alignment(16), 8); + + let slot2 = StackSlotData::new(StackSlotKind::Local, 24); + + assert_eq!(slot2.alignment(4), 4); + assert_eq!(slot2.alignment(8), 8); + assert_eq!(slot2.alignment(16), 8); + assert_eq!(slot2.alignment(32), 8); + } + + #[test] + fn layout() { + let mut sss = StackSlots::new(); + + // An empty layout should have 0-sized stack frame. + assert_eq!(sss.layout(1), 0); + assert_eq!(sss.layout(16), 0); + + // Same for incoming arguments with non-negative offsets. + let in0 = sss.make_incoming_arg(types::I64, 0); + let in1 = sss.make_incoming_arg(types::I64, 8); + + assert_eq!(sss.layout(1), 0); + assert_eq!(sss.layout(16), 0); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + + // Add some spill slots. + let ss0 = sss.make_spill_slot(types::I64); + let ss1 = sss.make_spill_slot(types::I32); + + assert_eq!(sss.layout(1), 12); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + assert_eq!(sss[ss0].offset, -8); + assert_eq!(sss[ss1].offset, -12); + + assert_eq!(sss.layout(16), 16); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + assert_eq!(sss[ss0].offset, -16); + assert_eq!(sss[ss1].offset, -4); + + // An incoming argument with negative offset counts towards the total frame size, but it + // should still pack nicely with the spill slots. + let in2 = sss.make_incoming_arg(types::I32, -4); + + assert_eq!(sss.layout(1), 16); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + assert_eq!(sss[in2].offset, -4); + assert_eq!(sss[ss0].offset, -12); + assert_eq!(sss[ss1].offset, -16); + + assert_eq!(sss.layout(16), 16); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + assert_eq!(sss[in2].offset, -4); + assert_eq!(sss[ss0].offset, -16); + assert_eq!(sss[ss1].offset, -8); + + // Finally, make sure there is room for the outgoing args. + let out0 = sss.get_outgoing_arg(types::I32, 0); + + assert_eq!(sss.layout(1), 20); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + assert_eq!(sss[in2].offset, -4); + assert_eq!(sss[ss0].offset, -12); + assert_eq!(sss[ss1].offset, -16); + assert_eq!(sss[out0].offset, 0); + + assert_eq!(sss.layout(16), 32); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + assert_eq!(sss[in2].offset, -4); + assert_eq!(sss[ss0].offset, -16); + assert_eq!(sss[ss1].offset, -8); + assert_eq!(sss[out0].offset, 0); + } } From c057e005ce9a5c177088ead714aaebd476b8048f Mon Sep 17 00:00:00 2001 From: Aleksey Kuznetsov Date: Sat, 29 Jul 2017 16:21:41 +0500 Subject: [PATCH 0949/3084] Implement conditional compilation configuration in build.rs --- lib/cretonne/build.rs | 104 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) diff --git a/lib/cretonne/build.rs b/lib/cretonne/build.rs index 4766a032e0..2653bf54a0 100644 --- a/lib/cretonne/build.rs +++ b/lib/cretonne/build.rs @@ -8,6 +8,13 @@ // OUT_DIR // Directory where generated files should be placed. // +// TARGET +// Target triple provided by Cargo. +// +// CRETONNE_TARGETS (Optional) +// A setting for conditional compilation of isa targets. Possible values can be "native" or +// known isa targets separated by ','. +// // The build script expects to be run from the directory where this build.rs file lives. The // current directory is used to find the sources. @@ -17,6 +24,22 @@ use std::process; fn main() { let out_dir = env::var("OUT_DIR").expect("The OUT_DIR environment variable must be set"); + let target_triple = env::var("TARGET").expect("The TARGET environment variable must be set"); + let cretonne_targets = env::var("CRETONNE_TARGETS").ok(); + let cretonne_targets = cretonne_targets.as_ref().map(|s| s.as_ref()); + + // Configure isa targets cfg. + match isa_targets(cretonne_targets, &target_triple) { + Ok(isa_targets) => { + for isa in &isa_targets { + println!("cargo:rustc-cfg=build_{}", isa.name()); + } + } + Err(err) => { + eprintln!("Error: {}", err); + process::exit(1); + } + } println!("Build script generating files in {}", out_dir); @@ -45,3 +68,84 @@ fn main() { process::exit(status.code().unwrap()); } } + +/// Represents known ISA target. +#[derive(Copy, Clone)] +enum Isa { + Riscv, + Intel, + Arm32, + Arm64, +} + +impl Isa { + /// Creates isa target using name. + fn new(name: &str) -> Option { + Isa::all() + .iter() + .cloned() + .filter(|isa| isa.name() == name) + .next() + } + + /// Creates isa target from arch. + fn from_arch(arch: &str) -> Option { + Isa::all() + .iter() + .cloned() + .filter(|isa| isa.is_arch_applicable(arch)) + .next() + } + + /// Returns all supported isa targets. + fn all() -> [Isa; 4] { + [Isa::Riscv, Isa::Intel, Isa::Arm32, Isa::Arm64] + } + + /// Returns name of the isa target. + fn name(&self) -> &'static str { + match *self { + Isa::Riscv => "riscv", + Isa::Intel => "intel", + Isa::Arm32 => "arm32", + Isa::Arm64 => "arm64", + } + } + + /// Checks if arch is applicable for the isa target. + fn is_arch_applicable(&self, arch: &str) -> bool { + match *self { + Isa::Riscv => arch == "riscv", + Isa::Intel => ["x86_64", "i386", "i586", "i686"].contains(&arch), + Isa::Arm32 => arch.starts_with("arm") || arch.starts_with("thumb"), + Isa::Arm64 => arch == "aarch64", + } + } +} + +/// Returns isa targets to configure conditional compilation. +fn isa_targets(cretonne_targets: Option<&str>, target_triple: &str) -> Result, String> { + match cretonne_targets { + Some("native") => { + Isa::from_arch(target_triple.split('-').next().unwrap()) + .map(|isa| vec![isa]) + .ok_or_else(|| { + format!("no supported isa found for target triple `{}`", + target_triple) + }) + } + Some(targets) => { + let unknown_isa_targets = targets + .split(',') + .filter(|target| Isa::new(target).is_none()) + .collect::>(); + let isa_targets = targets.split(',').flat_map(Isa::new).collect::>(); + match (unknown_isa_targets.is_empty(), isa_targets.is_empty()) { + (true, true) => Ok(Isa::all().to_vec()), + (true, _) => Ok(isa_targets), + (_, _) => Err(format!("unknown isa targets: `{}`", unknown_isa_targets.join(", "))), + } + } + None => Ok(Isa::all().to_vec()), + } +} From 5fa991e325de8e71d27b8016fbe2bff60ba8a378 Mon Sep 17 00:00:00 2001 From: Aleksey Kuznetsov Date: Sat, 29 Jul 2017 20:05:45 +0500 Subject: [PATCH 0950/3084] Apply conditional compilation of isa targets --- lib/cretonne/src/isa/mod.rs | 63 +++++++++++++++++---------- lib/cretonne/src/regalloc/pressure.rs | 3 +- lib/cretonne/src/regalloc/solver.rs | 3 +- lib/reader/src/parser.rs | 14 ++++-- 4 files changed, 55 insertions(+), 28 deletions(-) diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index b808b88746..7caac93a67 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -27,10 +27,10 @@ //! let shared_flags = settings::Flags::new(&shared_builder); //! //! match isa::lookup("riscv") { -//! None => { +//! Err(_) => { //! // The RISC-V target ISA is not available. //! } -//! Some(mut isa_builder) => { +//! Ok(mut isa_builder) => { //! isa_builder.set("supports_m", "on"); //! let isa = isa_builder.finish(shared_flags); //! } @@ -51,42 +51,61 @@ use ir; use regalloc; use isa::enc_tables::Encodings; +#[cfg(build_riscv)] pub mod riscv; + +#[cfg(build_intel)] pub mod intel; + +#[cfg(build_arm32)] pub mod arm32; + +#[cfg(build_arm64)] pub mod arm64; + pub mod registers; mod encoding; mod enc_tables; mod constraints; +/// Returns a builder that can create a corresponding `TargetIsa` +/// or `Err(LookupError::Unsupported)` if not enabled. +macro_rules! isa_builder { + ($module:ident, $name:ident) => { + { + #[cfg($name)] + fn $name() -> Result { + Ok($module::isa_builder()) + }; + #[cfg(not($name))] + fn $name() -> Result { + Err(LookupError::Unsupported) + } + $name() + } + }; +} + /// Look for a supported ISA with the given `name`. /// Return a builder that can create a corresponding `TargetIsa`. -pub fn lookup(name: &str) -> Option { +pub fn lookup(name: &str) -> Result { match name { - "riscv" => riscv_builder(), - "intel" => intel_builder(), - "arm32" => arm32_builder(), - "arm64" => arm64_builder(), - _ => None, + "riscv" => isa_builder!(riscv, build_riscv), + "intel" => isa_builder!(intel, build_intel), + "arm32" => isa_builder!(arm32, build_arm32), + "arm64" => isa_builder!(arm64, build_arm64), + _ => Err(LookupError::Unknown), } } -// Make a builder for RISC-V. -fn riscv_builder() -> Option { - Some(riscv::isa_builder()) -} +/// Describes reason for target lookup failure +#[derive(PartialEq, Eq, Copy, Clone, Debug)] +pub enum LookupError { + /// Unknown Target + Unknown, -fn intel_builder() -> Option { - Some(intel::isa_builder()) -} - -fn arm32_builder() -> Option { - Some(arm32::isa_builder()) -} - -fn arm64_builder() -> Option { - Some(arm64::isa_builder()) + /// Target known but not built and thus not supported + Unsupported, } /// Builder for a `TargetIsa`. diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index fd9cf7c363..a9ab6bbbcb 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -247,6 +247,7 @@ impl fmt::Display for Pressure { } #[cfg(test)] +#[cfg(build_arm32)] mod tests { use isa::{TargetIsa, RegClass}; use regalloc::AllocatableSet; @@ -261,7 +262,7 @@ mod tests { let shared_builder = settings::builder(); let shared_flags = settings::Flags::new(&shared_builder); - isa::lookup("arm32").map(|b| b.finish(shared_flags)) + isa::lookup("arm32").ok().map(|b| b.finish(shared_flags)) } // Get a register class by name. diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index b135f051f3..ea028f551b 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -700,6 +700,7 @@ impl Solver { } #[cfg(test)] +#[cfg(build_arm32)] mod tests { use entity_ref::EntityRef; use ir::Value; @@ -716,7 +717,7 @@ mod tests { let shared_builder = settings::builder(); let shared_flags = settings::Flags::new(&shared_builder); - isa::lookup("arm32").map(|b| b.finish(shared_flags)) + isa::lookup("arm32").ok().map(|b| b.finish(shared_flags)) } // Get a register class by name. diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index fb659a6b8e..865cdcbd0d 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -621,8 +621,6 @@ impl<'a> Parser<'a> { &self.loc)?; } "isa" => { - last_set_loc = None; - seen_isa = true; let loc = self.loc; // Grab the whole line so the lexer won't go looking for tokens on the // following lines. @@ -633,9 +631,16 @@ impl<'a> Parser<'a> { Some(w) => w, }; let mut isa_builder = match isa::lookup(isa_name) { - None => return err!(loc, "unknown ISA '{}'", isa_name), - Some(b) => b, + Err(isa::LookupError::Unknown) => { + return err!(loc, "unknown ISA '{}'", isa_name) + } + Err(isa::LookupError::Unsupported) => { + continue; + } + Ok(b) => b, }; + last_set_loc = None; + seen_isa = true; // Apply the ISA-specific settings to `isa_builder`. isaspec::parse_options(words, &mut isa_builder, &self.loc)?; @@ -1939,6 +1944,7 @@ mod tests { } #[test] + #[cfg(build_riscv)] fn isa_spec() { assert!(parse_test("isa function %foo() {}") From c96d4daa203aad1344e8cb4e4c65b564fd461de2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 2 Aug 2017 16:40:35 -0700 Subject: [PATCH 0951/3084] Add a calling convention to all function signatures. A CallConv enum on every function signature makes it possible to generate calls to functions with different calling conventions within the same ISA / within a single function. The calling conventions also serve as a way of customizing Cretonne's behavior when embedded inside a VM. As an example, the SpiderWASM calling convention is used to compile WebAssembly functions that run inside the SpiderMonkey virtual machine. All function signatures must have a calling convention at the end, so this changes the textual IL syntax. Before: sig1 = signature(i32, f64) -> f64 After sig1 = (i32, f64) -> f64 native sig2 = (i32) spiderwasm When printing functions, the signature goes after the return types: function %r1() -> i32, f32 spiderwasm { ebb1: ... } In the parser, this calling convention is optional and defaults to "native". This is mostly to avoid updating all the existing test cases under filetests/. When printing a function, the calling convention is always included, including for "native" functions. --- cranelift/docs/example.cton | 2 +- cranelift/docs/langref.rst | 7 -- cranelift/filetests/isa/intel/abi64.cton | 12 +-- cranelift/filetests/isa/intel/binary32.cton | 2 +- cranelift/filetests/isa/intel/binary64.cton | 4 +- cranelift/filetests/isa/riscv/abi-e.cton | 4 +- cranelift/filetests/isa/riscv/abi.cton | 24 +++--- cranelift/filetests/isa/riscv/binary32.cton | 2 +- .../filetests/isa/riscv/legalize-abi.cton | 4 +- .../filetests/isa/riscv/parse-encoding.cton | 30 ++++---- cranelift/filetests/licm/basic.cton | 2 +- cranelift/filetests/licm/complex.cton | 2 +- cranelift/filetests/parser/branch.cton | 12 +-- cranelift/filetests/parser/call.cton | 26 +++---- .../parser/instruction_encoding.cton | 2 +- cranelift/filetests/parser/keywords.cton | 2 +- cranelift/filetests/parser/rewrite.cton | 4 +- cranelift/filetests/parser/tiny.cton | 24 +++--- cranelift/filetests/regalloc/spill.cton | 4 +- cranelift/tests/cfg_traversal.rs | 10 +-- lib/cretonne/src/ir/extfunc.rs | 70 +++++++++++++++--- lib/cretonne/src/ir/function.rs | 6 +- lib/cretonne/src/ir/mod.rs | 3 +- lib/cretonne/src/write.rs | 15 ++-- lib/frontend/src/frontend.rs | 4 +- lib/frontend/src/lib.rs | 4 +- lib/reader/src/parser.rs | 74 ++++++++++++------- 27 files changed, 211 insertions(+), 144 deletions(-) diff --git a/cranelift/docs/example.cton b/cranelift/docs/example.cton index 493794da50..0cbd77a6cd 100644 --- a/cranelift/docs/example.cton +++ b/cranelift/docs/example.cton @@ -1,6 +1,6 @@ test verifier -function %average(i32, i32) -> f32 { +function %average(i32, i32) -> f32 native { ss1 = local 8 ; Stack slot for ``sum``. ebb1(v1: i32, v2: i32): diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index e7b0cad3d7..f20b5b6e84 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -410,13 +410,6 @@ This simple example illustrates direct function calls and signatures:: Indirect function calls use a signature declared in the preamble. -.. inst:: SIG = signature signature - - Declare a function signature for use with indirect calls. - - :arg signature: Function signature. See :token:`signature`. - :result SIG: A signature identifier. - .. autoinst:: call_indirect .. todo:: Define safe indirect function calls. diff --git a/cranelift/filetests/isa/intel/abi64.cton b/cranelift/filetests/isa/intel/abi64.cton index ecd3d29fd6..59f3107560 100644 --- a/cranelift/filetests/isa/intel/abi64.cton +++ b/cranelift/filetests/isa/intel/abi64.cton @@ -6,14 +6,14 @@ isa intel ; regex: V=v\d+ function %f() { - sig0 = signature(i32) -> i32 - ; check: sig0 = signature(i32 [%rdi]) -> i32 [%rax] + sig0 = (i32) -> i32 native + ; check: sig0 = (i32 [%rdi]) -> i32 [%rax] native - sig1 = signature(i64) -> b1 - ; check: sig1 = signature(i64 [%rdi]) -> b1 [%rax] + sig1 = (i64) -> b1 native + ; check: sig1 = (i64 [%rdi]) -> b1 [%rax] native - sig2 = signature(f32, i64) -> f64 - ; check: sig2 = signature(f32 [%xmm0], i64 [%rdi]) -> f64 [%xmm0] + sig2 = (f32, i64) -> f64 native + ; check: sig2 = (f32 [%xmm0], i64 [%rdi]) -> f64 [%xmm0] native ebb0: return diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 456623a235..af1bf73a9b 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -9,7 +9,7 @@ isa intel haswell function %I32() { fn0 = function %foo() - sig0 = signature() + sig0 = () ebb0: ; asm: movl $1, %ecx diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index bc19e039d3..fb6f62d928 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -11,7 +11,7 @@ isa intel haswell ; Tests for i64 instructions. function %I64() { fn0 = function %foo() - sig0 = signature() + sig0 = () ebb0: @@ -457,7 +457,7 @@ ebb2: ; be done by an instruction shrinking pass. function %I32() { fn0 = function %foo() - sig0 = signature() + sig0 = () ebb0: diff --git a/cranelift/filetests/isa/riscv/abi-e.cton b/cranelift/filetests/isa/riscv/abi-e.cton index 543b55079f..df06402283 100644 --- a/cranelift/filetests/isa/riscv/abi-e.cton +++ b/cranelift/filetests/isa/riscv/abi-e.cton @@ -7,8 +7,8 @@ isa riscv enable_e function %f() { ; Spilling into the stack args after %x15 since %16 and up are not ; available in RV32E. - sig0 = signature(i64, i64, i64, i64) -> i64 - ; check: sig0 = signature(i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [0], i32 [4]) -> i32 [%x10], i32 [%x11] + sig0 = (i64, i64, i64, i64) -> i64 native + ; check: sig0 = (i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [0], i32 [4]) -> i32 [%x10], i32 [%x11] native ebb0: return } diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.cton index f26c7686cf..c57c09fd97 100644 --- a/cranelift/filetests/isa/riscv/abi.cton +++ b/cranelift/filetests/isa/riscv/abi.cton @@ -5,27 +5,27 @@ isa riscv ; regex: V=v\d+ function %f() { - sig0 = signature(i32) -> i32 - ; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] + sig0 = (i32) -> i32 native + ; check: sig0 = (i32 [%x10]) -> i32 [%x10] native - sig1 = signature(i64) -> b1 - ; check: sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] + sig1 = (i64) -> b1 native + ; check: sig1 = (i32 [%x10], i32 [%x11]) -> b1 [%x10] native ; The i64 argument must go in an even-odd register pair. - sig2 = signature(f32, i64) -> f64 - ; check: sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] + sig2 = (f32, i64) -> f64 native + ; check: sig2 = (f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] native ; Spilling into the stack args. - sig3 = signature(f64, f64, f64, f64, f64, f64, f64, i64) -> f64 - ; check: sig3 = signature(f64 [%f10], f64 [%f11], f64 [%f12], f64 [%f13], f64 [%f14], f64 [%f15], f64 [%f16], i32 [0], i32 [4]) -> f64 [%f10] + sig3 = (f64, f64, f64, f64, f64, f64, f64, i64) -> f64 native + ; check: sig3 = (f64 [%f10], f64 [%f11], f64 [%f12], f64 [%f13], f64 [%f14], f64 [%f15], f64 [%f16], i32 [0], i32 [4]) -> f64 [%f10] native ; Splitting vectors. - sig4 = signature(i32x4) - ; check: sig4 = signature(i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13]) + sig4 = (i32x4) native + ; check: sig4 = (i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13]) native ; Splitting vectors, then splitting ints. - sig5 = signature(i64x4) - ; check: sig5 = signature(i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [%x16], i32 [%x17]) + sig5 = (i64x4) native + ; check: sig5 = (i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [%x16], i32 [%x17]) native ebb0: return diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 6575609267..1ed2fcabea 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -4,7 +4,7 @@ isa riscv function %RV32I(i32 link [%x1]) -> i32 link [%x1] { fn0 = function %foo() - sig0 = signature() + sig0 = () ebb0(v9999: i32): [-,%x10] v1 = iconst.i32 1 diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index cb94836ab4..f80494cc1a 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -106,7 +106,7 @@ ebb0(v0: i64x4): } function %indirect(i32) { - sig1 = signature() + sig1 = () native ebb0(v0: i32): call_indirect sig1, v0() return @@ -114,7 +114,7 @@ ebb0(v0: i32): ; The first argument to call_indirect doesn't get altered. function %indirect_arg(i32, f32x2) { - sig1 = signature(f32x2) + sig1 = (f32x2) native ebb0(v0: i32, v1: f32x2): call_indirect sig1, v0(v1) ; check: call_indirect $sig1, $v0($V, $V) diff --git a/cranelift/filetests/isa/riscv/parse-encoding.cton b/cranelift/filetests/isa/riscv/parse-encoding.cton index cd02a3ca47..3fdb8f62d6 100644 --- a/cranelift/filetests/isa/riscv/parse-encoding.cton +++ b/cranelift/filetests/isa/riscv/parse-encoding.cton @@ -3,32 +3,32 @@ test legalizer isa riscv function %parse_encoding(i32 [%x5]) -> i32 [%x10] { - ; check: function %parse_encoding(i32 [%x5], i32 link [%x1]) -> i32 [%x10], i32 link [%x1] { + ; check: function %parse_encoding(i32 [%x5], i32 link [%x1]) -> i32 [%x10], i32 link [%x1] native { - sig0 = signature(i32 [%x10]) -> i32 [%x10] - ; check: sig0 = signature(i32 [%x10]) -> i32 [%x10] + sig0 = (i32 [%x10]) -> i32 [%x10] native + ; check: sig0 = (i32 [%x10]) -> i32 [%x10] native - sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] - ; check: sig1 = signature(i32 [%x10], i32 [%x11]) -> b1 [%x10] + sig1 = (i32 [%x10], i32 [%x11]) -> b1 [%x10] native + ; check: sig1 = (i32 [%x10], i32 [%x11]) -> b1 [%x10] native - sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] - ; check: sig2 = signature(f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] + sig2 = (f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] native + ; check: sig2 = (f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] native ; Arguments on stack where not necessary - sig3 = signature(f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] - ; check: sig3 = signature(f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] + sig3 = (f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] native + ; check: sig3 = (f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] native ; Stack argument before register argument - sig4 = signature(f32 [72], i32 [%x10]) - ; check: sig4 = signature(f32 [72], i32 [%x10]) + sig4 = (f32 [72], i32 [%x10]) native + ; check: sig4 = (f32 [72], i32 [%x10]) native ; Return value on stack - sig5 = signature() -> f32 [0] - ; check: sig5 = signature() -> f32 [0] + sig5 = () -> f32 [0] native + ; check: sig5 = () -> f32 [0] native ; function + signature - fn15 = function %bar(i32 [%x10]) -> b1 [%x10] - ; check: sig6 = signature(i32 [%x10]) -> b1 [%x10] + fn15 = function %bar(i32 [%x10]) -> b1 [%x10] native + ; check: sig6 = (i32 [%x10]) -> b1 [%x10] native ; nextln: fn0 = sig6 %bar ebb0(v0: i32): diff --git a/cranelift/filetests/licm/basic.cton b/cranelift/filetests/licm/basic.cton index 36b7864cfe..37dda60d2a 100644 --- a/cranelift/filetests/licm/basic.cton +++ b/cranelift/filetests/licm/basic.cton @@ -14,7 +14,7 @@ ebb2(v5: i32): return v5 } -; sameln: function %simple_loop(i32) -> i32 { +; sameln: function %simple_loop ; nextln: ebb2(v6: i32): ; nextln: v1 = iconst.i32 1 ; nextln: v2 = iconst.i32 2 diff --git a/cranelift/filetests/licm/complex.cton b/cranelift/filetests/licm/complex.cton index b5063d807f..07efb9ff5f 100644 --- a/cranelift/filetests/licm/complex.cton +++ b/cranelift/filetests/licm/complex.cton @@ -39,7 +39,7 @@ ebb5(v16: i32): return v17 } -; sameln: function %complex(i32) -> i32 { +; sameln: function %complex ; nextln: ebb6(v20: i32): ; nextln: v1 = iconst.i32 1 ; nextln: v2 = iconst.i32 4 diff --git a/cranelift/filetests/parser/branch.cton b/cranelift/filetests/parser/branch.cton index 4adb4b5d27..283dd9a03a 100644 --- a/cranelift/filetests/parser/branch.cton +++ b/cranelift/filetests/parser/branch.cton @@ -9,7 +9,7 @@ ebb0: ebb1: jump ebb0() } -; sameln: function %minimal() { +; sameln: function %minimal() native { ; nextln: ebb0: ; nextln: jump ebb1 ; nextln: @@ -25,7 +25,7 @@ ebb0(v90: i32): ebb1(v91: i32): jump ebb0(v91) } -; sameln: function %onearg(i32) { +; sameln: function %onearg(i32) native { ; nextln: ebb0($v90: i32): ; nextln: jump ebb1($v90) ; nextln: @@ -41,7 +41,7 @@ ebb0(v90: i32, v91: f32): ebb1(v92: i32, v93: f32): jump ebb0(v92, v93) } -; sameln: function %twoargs(i32, f32) { +; sameln: function %twoargs(i32, f32) native { ; nextln: ebb0($v90: i32, $v91: f32): ; nextln: jump ebb1($v90, $v91) ; nextln: @@ -57,7 +57,7 @@ ebb0(v90: i32): ebb1: brnz v90, ebb1() } -; sameln: function %minimal(i32) { +; sameln: function %minimal(i32) native { ; nextln: ebb0($v90: i32): ; nextln: brz $v90, ebb1 ; nextln: @@ -72,7 +72,7 @@ ebb0(v90: i32, v91: f32): ebb1(v92: i32, v93: f32): brnz v90, ebb0(v92, v93) } -; sameln: function %twoargs(i32, f32) { +; sameln: function %twoargs(i32, f32) native { ; nextln: ebb0($v90: i32, $v91: f32): ; nextln: brz $v90, ebb1($v90, $v91) ; nextln: @@ -94,7 +94,7 @@ ebb30: ebb40: trap } -; sameln: function %jumptable(i32) { +; sameln: function %jumptable(i32) native { ; nextln: jt0 = jump_table 0 ; nextln: jt1 = jump_table 0, 0, ebb0, ebb3, ebb1, ebb2 ; nextln: diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton index 9d7c2c3d16..662925db7f 100644 --- a/cranelift/filetests/parser/call.cton +++ b/cranelift/filetests/parser/call.cton @@ -5,18 +5,18 @@ function %mini() { ebb1: return } -; sameln: function %mini() { +; sameln: function %mini() native { ; nextln: ebb0: ; nextln: return ; nextln: } -function %r1() -> i32, f32 { +function %r1() -> i32, f32 spiderwasm { ebb1: v1 = iconst.i32 3 v2 = f32const 0.0 return v1, v2 } -; sameln: function %r1() -> i32, f32 { +; sameln: function %r1() -> i32, f32 spiderwasm { ; nextln: ebb0: ; nextln: $v1 = iconst.i32 3 ; nextln: $v2 = f32const 0.0 @@ -24,15 +24,15 @@ ebb1: ; nextln: } function %signatures() { - sig10 = signature() - sig11 = signature(i32, f64) -> i32, b1 + sig10 = () + sig11 = (i32, f64) -> i32, b1 spiderwasm fn5 = sig11 %foo fn8 = function %bar(i32) -> b1 } -; sameln: function %signatures() { -; nextln: $sig10 = signature() -; nextln: $sig11 = signature(i32, f64) -> i32, b1 -; nextln: sig2 = signature(i32) -> b1 +; sameln: function %signatures() native { +; nextln: $sig10 = () native +; nextln: $sig11 = (i32, f64) -> i32, b1 spiderwasm +; nextln: sig2 = (i32) -> b1 native ; nextln: $fn5 = $sig11 %foo ; nextln: $fn8 = sig2 %bar ; nextln: } @@ -54,9 +54,9 @@ ebb0: ; check: return function %indirect(i64) { - sig0 = signature(i64) - sig1 = signature() -> i32 - sig2 = signature() -> i32, f32 + sig0 = (i64) + sig1 = () -> i32 + sig2 = () -> i32, f32 ebb0(v0: i64): v1 = call_indirect sig1, v0() @@ -74,7 +74,7 @@ function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 ebb0(v1: i32, v2: i32, v3: i32, v4: i32): return v4, v2, v3, v1 } -; check: function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret { +; check: function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret native { ; check: ebb0($v1: i32, $v2: i32, $v3: i32, $v4: i32): ; check: return $v4, $v2, $v3, $v1 ; check: } diff --git a/cranelift/filetests/parser/instruction_encoding.cton b/cranelift/filetests/parser/instruction_encoding.cton index bc2e1dd239..c808892701 100644 --- a/cranelift/filetests/parser/instruction_encoding.cton +++ b/cranelift/filetests/parser/instruction_encoding.cton @@ -13,7 +13,7 @@ ebb1(v0: i32, v1: i32): v9 = iadd v8, v7 [Iret#5] return v0, v8 } -; sameln: function %foo(i32, i32) { +; sameln: function %foo(i32, i32) native { ; nextln: $ebb1($v0: i32, $v1: i32): ; nextln: [-,-]$WS $v2 = iadd $v0, $v1 ; nextln: [-]$WS trap diff --git a/cranelift/filetests/parser/keywords.cton b/cranelift/filetests/parser/keywords.cton index 37d0390a58..a4b894574e 100644 --- a/cranelift/filetests/parser/keywords.cton +++ b/cranelift/filetests/parser/keywords.cton @@ -2,4 +2,4 @@ test cat ; 'function' is not a keyword, and can be used as the name of a function too. function %function() {} -; check: function %function() +; check: function %function() native diff --git a/cranelift/filetests/parser/rewrite.cton b/cranelift/filetests/parser/rewrite.cton index f7ebfa3876..c6a04a9e9c 100644 --- a/cranelift/filetests/parser/rewrite.cton +++ b/cranelift/filetests/parser/rewrite.cton @@ -15,7 +15,7 @@ ebb100(v20: i32): v9200 = f64const 0x4.0p0 trap } -; sameln: function %defs() { +; sameln: function %defs() native { ; nextln: $ebb100($v20: i32): ; nextln: $v1000 = iconst.i32x8 5 ; nextln: $v9200 = f64const 0x1.0000000000000p2 @@ -29,7 +29,7 @@ ebb100(v20: i32): v200 = iadd v20, v1000 jump ebb100(v1000) } -; sameln: function %use_value() { +; sameln: function %use_value() native { ; nextln: ebb0($v20: i32): ; nextln: $v1000 = iadd_imm $v20, 5 ; nextln: $v200 = iadd $v20, $v1000 diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index fff7ccd4c1..ecd2525ba2 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -5,7 +5,7 @@ function %minimal() { ebb0: trap } -; sameln: function %minimal() { +; sameln: function %minimal() native { ; nextln: ebb0: ; nextln: trap ; nextln: } @@ -18,7 +18,7 @@ ebb0: v1 = iconst.i8 6 v2 = ishl v0, v1 } -; sameln: function %ivalues() { +; sameln: function %ivalues() native { ; nextln: ebb0: ; nextln: $v0 = iconst.i32 2 ; nextln: $v1 = iconst.i8 6 @@ -34,7 +34,7 @@ ebb0: v2 = bextend.b32 v1 v3 = bxor v0, v2 } -; sameln: function %bvalues() { +; sameln: function %bvalues() native { ; nextln: ebb0: ; nextln: $v0 = bconst.b32 true ; nextln: $v1 = bconst.b8 false @@ -47,7 +47,7 @@ function %select() { ebb0(v90: i32, v91: i32, v92: b1): v0 = select v92, v90, v91 } -; sameln: function %select() { +; sameln: function %select() native { ; nextln: ebb0($v90: i32, $v91: i32, $v92: b1): ; nextln: $v0 = select $v92, $v90, $v91 ; nextln: } @@ -59,7 +59,7 @@ ebb0: v1 = extractlane v0, 3 v2 = insertlane v0, 1, v1 } -; sameln: function %lanes() { +; sameln: function %lanes() native { ; nextln: ebb0: ; nextln: $v0 = iconst.i32x4 2 ; nextln: $v1 = extractlane $v0, 3 @@ -75,7 +75,7 @@ ebb0(v90: i32, v91: i32): v3 = irsub_imm v91, 45 br_icmp eq v90, v91, ebb0(v91, v90) } -; sameln: function %icmp(i32, i32) { +; sameln: function %icmp(i32, i32) native { ; nextln: ebb0($v90: i32, $v91: i32): ; nextln: $v0 = icmp eq $v90, $v91 ; nextln: $v1 = icmp ult $v90, $v91 @@ -91,7 +91,7 @@ ebb0(v90: f32, v91: f32): v1 = fcmp uno v90, v91 v2 = fcmp lt v90, v91 } -; sameln: function %fcmp(f32, f32) { +; sameln: function %fcmp(f32, f32) native { ; nextln: ebb0($v90: f32, $v91: f32): ; nextln: $v0 = fcmp eq $v90, $v91 ; nextln: $v1 = fcmp uno $v90, $v91 @@ -105,7 +105,7 @@ ebb0(v90: i32, v91: f32): v0 = bitcast.i8x4 v90 v1 = bitcast.i32 v91 } -; sameln: function %bitcast(i32, f32) { +; sameln: function %bitcast(i32, f32) native { ; nextln: ebb0($v90: i32, $v91: f32): ; nextln: $v0 = bitcast.i8x4 $v90 ; nextln: $v1 = bitcast.i32 $v91 @@ -124,7 +124,7 @@ ebb0: stack_store v1, ss10+2 stack_store v2, ss2 } -; sameln: function %stack() { +; sameln: function %stack() native { ; nextln: $ss10 = spill_slot 8 ; nextln: $ss2 = local 4 ; nextln: $ss3 = incoming_arg 4, offset 8 @@ -144,7 +144,7 @@ ebb0(v1: i32): v3 = heap_load.f32 v1+12 heap_store v3, v1 } -; sameln: function %heap(i32) { +; sameln: function %heap(i32) native { ; nextln: ebb0($v1: i32): ; nextln: $v2 = heap_load.f32 $v1 ; nextln: $v3 = heap_load.f32 $v1+12 @@ -164,7 +164,7 @@ ebb0(v1: i32): store aligned v3, v1+12 store notrap aligned v3, v1-12 } -; sameln: function %memory(i32) { +; sameln: function %memory(i32) native { ; nextln: ebb0($v1: i32): ; nextln: $v2 = load.i64 $v1 ; nextln: $v3 = load.i64 aligned $v1 @@ -185,7 +185,7 @@ ebb0(v1: i32): regmove v1, %20 -> %10 return } -; sameln: function %diversion(i32) { +; sameln: function %diversion(i32) native { ; nextln: ebb0($v1: i32): ; nextln: regmove $v1, %10 -> %20 ; nextln: regmove $v1, %20 -> %10 diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index 4c470a73e6..901509a8d4 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -93,7 +93,7 @@ ebb0(v0: i32): ; The same value used as indirect callee and argument. function %doubleuse_icall1(i32) { - sig0 = signature(i32) + sig0 = (i32) native ebb0(v0: i32): ; not:copy call_indirect sig0, v0(v0) @@ -102,7 +102,7 @@ ebb0(v0: i32): ; The same value used as indirect callee and two arguments. function %doubleuse_icall2(i32) { - sig0 = signature(i32, i32) + sig0 = (i32, i32) native ebb0(v0: i32): ; check: $(c=$V) = copy $v0 call_indirect sig0, v0(v0, v0) diff --git a/cranelift/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs index a0c9a64d90..37d568e0e4 100644 --- a/cranelift/tests/cfg_traversal.rs +++ b/cranelift/tests/cfg_traversal.rs @@ -27,7 +27,7 @@ fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) #[test] fn simple_traversal() { test_reverse_postorder_traversal(" - function %test(i32) { + function %test(i32) native { ebb0(v0: i32): brz v0, ebb1 jump ebb2 @@ -56,7 +56,7 @@ fn simple_traversal() { #[test] fn loops_one() { test_reverse_postorder_traversal(" - function %test(i32) { + function %test(i32) native { ebb0(v0: i32): jump ebb1 ebb1: @@ -74,7 +74,7 @@ fn loops_one() { #[test] fn loops_two() { test_reverse_postorder_traversal(" - function %test(i32) { + function %test(i32) native { ebb0(v0: i32): brz v0, ebb1 jump ebb2 @@ -99,7 +99,7 @@ fn loops_two() { #[test] fn loops_three() { test_reverse_postorder_traversal(" - function %test(i32) { + function %test(i32) native { ebb0(v0: i32): brz v0, ebb1 jump ebb2 @@ -129,7 +129,7 @@ fn loops_three() { #[test] fn back_edge_one() { test_reverse_postorder_traversal(" - function %test(i32) { + function %test(i32) native { ebb0(v0: i32): brz v0, ebb1 jump ebb2 diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 4cf5b9d61f..16211a4d83 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -25,6 +25,9 @@ pub struct Signature { /// Types returned from the function. pub return_types: Vec, + /// Calling convention. + pub call_conv: CallConv, + /// When the signature has been legalized to a specific ISA, this holds the size of the /// argument array on the stack. Before legalization, this is `None`. /// @@ -35,10 +38,11 @@ pub struct Signature { impl Signature { /// Create a new blank signature. - pub fn new() -> Signature { + pub fn new(call_conv: CallConv) -> Signature { Signature { argument_types: Vec::new(), return_types: Vec::new(), + call_conv, argument_bytes: None, } } @@ -94,7 +98,7 @@ impl<'a> fmt::Display for DisplaySignature<'a> { write!(f, " -> ")?; write_list(f, &self.0.return_types, self.1)?; } - Ok(()) + write!(f, " {}", self.0.call_conv) } } @@ -278,6 +282,46 @@ impl fmt::Display for ExtFuncData { } } +/// A Calling convention. +/// +/// A function's calling convention determines exactly how arguments and return values are passed, +/// and how stack frames are managed. Since all of these details depend on both the instruction set +/// architecture and possibly the operating system, a function's calling convention is only fully +/// determined by a `(TargetIsa, CallConv)` tuple. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum CallConv { + /// The C calling convention. + /// + /// This is the native calling convention that a C compiler would use on the platform. + Native, + + /// A JIT-compiled WebAssembly function in the SpiderMonkey VM. + SpiderWASM, +} + +impl fmt::Display for CallConv { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + use self::CallConv::*; + f.write_str(match *self { + Native => "native", + SpiderWASM => "spiderwasm", + }) + } +} + +impl FromStr for CallConv { + type Err = (); + + fn from_str(s: &str) -> Result { + use self::CallConv::*; + match s { + "native" => Ok(Native), + "spiderwasm" => Ok(SpiderWASM), + _ => Err(()), + } + } +} + #[cfg(test)] mod tests { use super::*; @@ -306,19 +350,26 @@ mod tests { } } + #[test] + fn call_conv() { + for &cc in &[CallConv::Native, CallConv::SpiderWASM] { + assert_eq!(Ok(cc), cc.to_string().parse()) + } + } + #[test] fn signatures() { - let mut sig = Signature::new(); - assert_eq!(sig.to_string(), "()"); + let mut sig = Signature::new(CallConv::SpiderWASM); + assert_eq!(sig.to_string(), "() spiderwasm"); sig.argument_types.push(ArgumentType::new(I32)); - assert_eq!(sig.to_string(), "(i32)"); + assert_eq!(sig.to_string(), "(i32) spiderwasm"); sig.return_types.push(ArgumentType::new(F32)); - assert_eq!(sig.to_string(), "(i32) -> f32"); + assert_eq!(sig.to_string(), "(i32) -> f32 spiderwasm"); sig.argument_types .push(ArgumentType::new(I32.by(4).unwrap())); - assert_eq!(sig.to_string(), "(i32, i32x4) -> f32"); + assert_eq!(sig.to_string(), "(i32, i32x4) -> f32 spiderwasm"); sig.return_types.push(ArgumentType::new(B8)); - assert_eq!(sig.to_string(), "(i32, i32x4) -> f32, b8"); + assert_eq!(sig.to_string(), "(i32, i32x4) -> f32, b8 spiderwasm"); // Test the offset computation algorithm. assert_eq!(sig.argument_bytes, None); @@ -332,6 +383,7 @@ mod tests { assert_eq!(sig.argument_bytes, Some(28)); // Writing ABI-annotated signatures. - assert_eq!(sig.to_string(), "(i32 [24], i32x4 [8]) -> f32, b8"); + assert_eq!(sig.to_string(), + "(i32 [24], i32x4 [8]) -> f32, b8 spiderwasm"); } } diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 59213aff13..c2b280dc6c 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -4,7 +4,7 @@ //! instructions. use entity_map::{EntityMap, PrimaryEntityData}; -use ir::{FunctionName, Signature, JumpTableData, DataFlowGraph, Layout}; +use ir::{FunctionName, CallConv, Signature, JumpTableData, DataFlowGraph, Layout}; use ir::{JumpTables, InstEncodings, ValueLocations, StackSlots, EbbOffsets}; use isa::TargetIsa; use std::fmt; @@ -67,9 +67,9 @@ impl Function { } } - /// Create a new empty, anonymous function. + /// Create a new empty, anonymous function with a native calling convention. pub fn new() -> Function { - Self::with_name_signature(FunctionName::default(), Signature::new()) + Self::with_name_signature(FunctionName::default(), Signature::new(CallConv::Native)) } /// Return an object that can display this function with correct ISA-specific annotations. diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index ebe39a3fcb..76f37efa3c 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -18,7 +18,8 @@ mod progpoint; mod valueloc; pub use ir::funcname::FunctionName; -pub use ir::extfunc::{Signature, ArgumentType, ArgumentExtension, ArgumentPurpose, ExtFuncData}; +pub use ir::extfunc::{Signature, CallConv, ArgumentType, ArgumentExtension, ArgumentPurpose, + ExtFuncData}; pub use ir::types::Type; pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable, FuncRef, SigRef}; pub use ir::instructions::{Opcode, InstructionData, VariableArgs, ValueList, ValueListPool}; diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 56a06a0c04..c342c4f273 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -54,7 +54,7 @@ fn write_preamble(w: &mut Write, for sig in func.dfg.signatures.keys() { any = true; writeln!(w, - " {} = signature{}", + " {} = {}", sig, func.dfg.signatures[sig].display(regs))?; } @@ -366,26 +366,27 @@ mod tests { #[test] fn basic() { let mut f = Function::new(); - assert_eq!(f.to_string(), "function %() {\n}\n"); + assert_eq!(f.to_string(), "function %() native {\n}\n"); f.name = FunctionName::new("foo"); - assert_eq!(f.to_string(), "function %foo() {\n}\n"); + assert_eq!(f.to_string(), "function %foo() native {\n}\n"); f.stack_slots .push(StackSlotData::new(StackSlotKind::Local, 4)); - assert_eq!(f.to_string(), "function %foo() {\n ss0 = local 4\n}\n"); + assert_eq!(f.to_string(), + "function %foo() native {\n ss0 = local 4\n}\n"); let ebb = f.dfg.make_ebb(); f.layout.append_ebb(ebb); assert_eq!(f.to_string(), - "function %foo() {\n ss0 = local 4\n\nebb0:\n}\n"); + "function %foo() native {\n ss0 = local 4\n\nebb0:\n}\n"); f.dfg.append_ebb_arg(ebb, types::I8); assert_eq!(f.to_string(), - "function %foo() {\n ss0 = local 4\n\nebb0(v0: i8):\n}\n"); + "function %foo() native {\n ss0 = local 4\n\nebb0(v0: i8):\n}\n"); f.dfg.append_ebb_arg(ebb, types::F32.by(4).unwrap()); assert_eq!(f.to_string(), - "function %foo() {\n ss0 = local 4\n\nebb0(v0: i8, v1: f32x4):\n}\n"); + "function %foo() native {\n ss0 = local 4\n\nebb0(v0: i8, v1: f32x4):\n}\n"); } } diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 7f9fb9edb6..d81cee4bc7 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -572,7 +572,7 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> mod tests { use cretonne::entity_ref::EntityRef; - use cretonne::ir::{FunctionName, Function, Signature, ArgumentType, InstBuilder}; + use cretonne::ir::{FunctionName, Function, CallConv, Signature, ArgumentType, InstBuilder}; use cretonne::ir::types::*; use frontend::{ILBuilder, FunctionBuilder}; use cretonne::verifier::verify_function; @@ -600,7 +600,7 @@ mod tests { #[test] fn sample_function() { - let mut sig = Signature::new(); + let mut sig = Signature::new(CallConv::Native); sig.return_types.push(ArgumentType::new(I32)); sig.argument_types.push(ArgumentType::new(I32)); diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 2cbbc30459..10a06b23cc 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -36,7 +36,7 @@ //! extern crate cton_frontend; //! //! use cretonne::entity_ref::EntityRef; -//! use cretonne::ir::{FunctionName, Function, Signature, ArgumentType, InstBuilder}; +//! use cretonne::ir::{FunctionName, CallConv, Function, Signature, ArgumentType, InstBuilder}; //! use cretonne::ir::types::*; //! use cton_frontend::{ILBuilder, FunctionBuilder}; //! use cretonne::verifier::verify_function; @@ -62,7 +62,7 @@ //! } //! //! fn main() { -//! let mut sig = Signature::new(); +//! let mut sig = Signature::new(CallConv::Native); //! sig.return_types.push(ArgumentType::new(I32)); //! sig.argument_types.push(ArgumentType::new(I32)); //! let mut il_builder = ILBuilder::::new(); diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 865cdcbd0d..fe2aec3e38 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -9,9 +9,9 @@ use std::collections::HashMap; use std::str::FromStr; use std::{u16, u32}; use std::mem; -use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, StackSlotData, JumpTable, - JumpTableData, Signature, ArgumentType, ArgumentExtension, ExtFuncData, SigRef, - FuncRef, StackSlot, ValueLoc, ArgumentLoc, MemFlags}; +use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, CallConv, StackSlotData, + JumpTable, JumpTableData, Signature, ArgumentType, ArgumentExtension, + ExtFuncData, SigRef, FuncRef, StackSlot, ValueLoc, ArgumentLoc, MemFlags}; use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Offset32, Uoffset32, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; @@ -767,13 +767,14 @@ impl<'a> Parser<'a> { // Parse a function signature. // - // signature ::= * "(" [arglist] ")" ["->" retlist] [call_conv] + // signature ::= * "(" [arglist] ")" ["->" retlist] [callconv] // fn parse_signature(&mut self, unique_isa: Option<&TargetIsa>) -> Result { - let mut sig = Signature::new(); + // Calling convention defaults to `native`, but can be changed. + let mut sig = Signature::new(CallConv::Native); self.match_token(Token::LPar, "expected function signature: ( args... )")?; - // signature ::= "(" * [arglist] ")" ["->" retlist] [call_conv] + // signature ::= "(" * [arglist] ")" ["->" retlist] [callconv] if self.token() != Some(Token::RPar) { sig.argument_types = self.parse_argument_list(unique_isa)?; } @@ -782,12 +783,21 @@ impl<'a> Parser<'a> { sig.return_types = self.parse_argument_list(unique_isa)?; } + // The calling convention is optional. + if let Some(Token::Identifier(text)) = self.token() { + match text.parse() { + Ok(cc) => { + self.consume(); + sig.call_conv = cc; + } + _ => return err!(self.loc, "unknown calling convention: {}", text), + } + } + if sig.argument_types.iter().all(|a| a.location.is_assigned()) { sig.compute_argument_bytes(); } - // TBD: calling convention. - Ok(sig) } @@ -951,12 +961,11 @@ impl<'a> Parser<'a> { // Parse a signature decl. // - // signature-decl ::= SigRef(sigref) "=" "signature" signature + // signature-decl ::= SigRef(sigref) "=" signature // fn parse_signature_decl(&mut self, unique_isa: Option<&TargetIsa>) -> Result<(u32, Signature)> { let number = self.match_sig("expected signature number: sig«n»")?; self.match_token(Token::Equal, "expected '=' in signature decl")?; - self.match_identifier("signature", "expected 'signature'")?; let data = self.parse_signature(unique_isa)?; Ok((number, data)) } @@ -1755,7 +1764,7 @@ impl<'a> Parser<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne::ir::{ArgumentExtension, ArgumentPurpose}; + use cretonne::ir::{CallConv, ArgumentExtension, ArgumentPurpose}; use cretonne::ir::types; use cretonne::ir::StackSlotKind; use cretonne::ir::entities::AnyEntity; @@ -1777,7 +1786,7 @@ mod tests { #[test] fn aliases() { - let (func, details) = Parser::new("function %qux() { + let (func, details) = Parser::new("function %qux() native { ebb0: v4 = iconst.i8 6 v3 -> v4 @@ -1801,15 +1810,26 @@ mod tests { #[test] fn signature() { - let sig = Parser::new("()").parse_signature(None).unwrap(); + let sig = Parser::new("()native").parse_signature(None).unwrap(); assert_eq!(sig.argument_types.len(), 0); assert_eq!(sig.return_types.len(), 0); + assert_eq!(sig.call_conv, CallConv::Native); - let sig2 = Parser::new("(i8 uext, f32, f64, i32 sret) -> i32 sext, f64") + let sig2 = Parser::new("(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 spiderwasm") .parse_signature(None) .unwrap(); assert_eq!(sig2.to_string(), - "(i8 uext, f32, f64, i32 sret) -> i32 sext, f64"); + "(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 spiderwasm"); + assert_eq!(sig2.call_conv, CallConv::SpiderWASM); + + // Old-style signature without a calling convention. + assert_eq!(Parser::new("()").parse_signature(None).unwrap().to_string(), + "() native"); + assert_eq!(Parser::new("() notacc") + .parse_signature(None) + .unwrap_err() + .to_string(), + "1: unknown calling convention: notacc"); // `void` is not recognized as a type by the lexer. It should not appear in files. assert_eq!(Parser::new("() -> void") @@ -1831,7 +1851,7 @@ mod tests { #[test] fn stack_slot_decl() { - let (func, _) = Parser::new("function %foo() { + let (func, _) = Parser::new("function %foo() native { ss3 = incoming_arg 13 ss1 = spill_slot 1 }") @@ -1850,7 +1870,7 @@ mod tests { assert_eq!(iter.next(), None); // Catch duplicate definitions. - assert_eq!(Parser::new("function %bar() { + assert_eq!(Parser::new("function %bar() native { ss1 = spill_slot 13 ss1 = spill_slot 1 }") @@ -1862,7 +1882,7 @@ mod tests { #[test] fn ebb_header() { - let (func, _) = Parser::new("function %ebbs() { + let (func, _) = Parser::new("function %ebbs() native { ebb0: ebb4(v3: i32): }") @@ -1884,7 +1904,7 @@ mod tests { #[test] fn comments() { let (func, Details { comments, .. }) = Parser::new("; before - function %comment() { ; decl + function %comment() native { ; decl ss10 = outgoing_arg 13 ; stackslot. ; Still stackslot. jt10 = jump_table ebb0 @@ -1924,7 +1944,7 @@ mod tests { test verify set enable_float=false ; still preamble - function %comment() {}") + function %comment() native {}") .unwrap(); assert_eq!(tf.commands.len(), 2); assert_eq!(tf.commands[0].command, "cfg"); @@ -1947,17 +1967,17 @@ mod tests { #[cfg(build_riscv)] fn isa_spec() { assert!(parse_test("isa - function %foo() {}") + function %foo() native {}") .is_err()); assert!(parse_test("isa riscv set enable_float=false - function %foo() {}") + function %foo() native {}") .is_err()); match parse_test("set enable_float=false isa riscv - function %foo() {}") + function %foo() native {}") .unwrap() .isa_spec { IsaSpec::None(_) => panic!("Expected some ISA"), @@ -1971,7 +1991,7 @@ mod tests { #[test] fn binary_function_name() { // Valid characters in the name. - let func = Parser::new("function #1234567890AbCdEf() { + let func = Parser::new("function #1234567890AbCdEf() native { ebb0: trap }") @@ -1981,21 +2001,21 @@ mod tests { assert_eq!(func.name.to_string(), "#1234567890abcdef"); // Invalid characters in the name. - let mut parser = Parser::new("function #12ww() { + let mut parser = Parser::new("function #12ww() native { ebb0: trap }"); assert!(parser.parse_function(None).is_err()); // The length of binary function name should be multiple of two. - let mut parser = Parser::new("function #1() { + let mut parser = Parser::new("function #1() native { ebb0: trap }"); assert!(parser.parse_function(None).is_err()); // Empty binary function name should be valid. - let func = Parser::new("function #() { + let func = Parser::new("function #() native { ebb0: trap }") From 39cc7efc2dd92edf5c076abed636efaa86a23be3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 3 Aug 2017 13:31:58 -0700 Subject: [PATCH 0952/3084] Move the stack layout computation into its own module. This is trying to keep algorithms out if the ir module which deals with the intermediate representation. Also give the layout_stack() function a Result return value so it can report a soft error when the stack frame is too large instead of asserting. Since local variables can be arbitrarily large, it is easy enough to overflow the stack with even a small function. --- lib/cretonne/src/ir/stackslot.rs | 178 ++--------------------------- lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/result.rs | 2 +- lib/cretonne/src/stack_layout.rs | 185 +++++++++++++++++++++++++++++++ 4 files changed, 194 insertions(+), 172 deletions(-) create mode 100644 lib/cretonne/src/stack_layout.rs diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index ea28eeae1a..6f3de96cd2 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -5,7 +5,6 @@ use entity_map::{EntityMap, PrimaryEntityData, Keys}; use ir::{Type, StackSlot}; -use std::cmp::{min, max}; use std::fmt; use std::ops::Index; use std::str::FromStr; @@ -15,12 +14,12 @@ use std::str::FromStr; /// We don't use `usize` to represent object sizes on the target platform because Cretonne supports /// cross-compilation, and `usize` is a type that depends on the host platform, not the target /// platform. -type StackSize = u32; +pub type StackSize = u32; /// A stack offset. /// /// The location of a stack offset relative to a stack pointer or frame pointer. -type StackOffset = i32; +pub type StackOffset = i32; /// The kind of a stack slot. #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -179,6 +178,11 @@ impl StackSlots { self.slots.is_valid(ss) } + /// Set the offset of a stack slot. + pub fn set_offset(&mut self, ss: StackSlot, offset: StackOffset) { + self.slots[ss].offset = offset; + } + /// Get an iterator over all the stack slot keys. pub fn keys(&self) -> Keys { self.slots.keys() @@ -241,103 +245,6 @@ impl StackSlots { self.outgoing.insert(inspos, ss); ss } - - /// Compute the stack frame layout. - /// - /// Determine the total size of this function's stack frame and assign offsets to all `Spill` - /// and `Local` stack slots. - /// - /// The total frame size will be a multiple of `alignment` which must be a power of two. - /// - /// Returns the total stack frame size which is also saved in `self.frame_size`. - pub fn layout(&mut self, alignment: StackSize) -> StackSize { - assert!(alignment.is_power_of_two() && alignment <= StackOffset::max_value() as StackSize, - "Invalid stack alignment {}", - alignment); - - // We assume a stack that grows toward lower addresses as implemented by modern ISAs. The - // stack layout from high to low addresses will be: - // - // 1. incoming arguments. - // 2. spills + locals. - // 3. outgoing arguments. - // - // The incoming arguments can have both positive and negative offsets. A negative offset - // incoming arguments is usually the x86 return address pushed by the call instruction, but - // it can also be fixed stack slots pushed by an externally generated prologue. - // - // Both incoming and outgoing argument slots have fixed offsets that are treated as - // reserved zones by the layout algorithm. - - let mut incoming_min = 0; - let mut outgoing_max = 0; - let mut min_align = alignment; - - for ss in self.keys() { - let slot = &self[ss]; - assert!(slot.size <= StackOffset::max_value() as StackSize); - match slot.kind { - StackSlotKind::IncomingArg => { - incoming_min = min(incoming_min, slot.offset); - } - StackSlotKind::OutgoingArg => { - let offset = slot.offset - .checked_add(slot.size as StackOffset) - .expect("Outgoing call argument overflows stack"); - outgoing_max = max(outgoing_max, offset); - } - StackSlotKind::SpillSlot | StackSlotKind::Local => { - // Determine the smallest alignment of any local or spill slot. - min_align = slot.alignment(min_align); - } - } - } - - // Lay out spill slots and locals below the incoming arguments. - // The offset is negative, growing downwards. - // Start with the smallest alignments for better packing. - let mut offset = incoming_min; - assert!(min_align.is_power_of_two()); - while min_align <= alignment { - for ss in self.keys() { - let slot = &mut self.slots[ss]; - - // Pick out locals and spill slots with exact alignment `min_align`. - match slot.kind { - StackSlotKind::SpillSlot | StackSlotKind::Local => { - if slot.alignment(alignment) != min_align { - continue; - } - } - _ => continue, - } - - // These limits should never be exceeded by spill slots, but locals can be - // arbitrarily large. - assert!(slot.size <= StackOffset::max_value() as StackSize); - offset = offset - .checked_sub(slot.size as StackOffset) - .expect("Stack frame larger than 2 GB"); - - // Aligning the negative offset can never cause overflow. We're only clearing bits. - offset &= -(min_align as StackOffset); - slot.offset = offset; - } - - // Move on to the next higher alignment. - min_align *= 2; - } - - // Finally, make room for the outgoing arguments. - offset = offset - .checked_sub(outgoing_max) - .expect("Stack frame larger than 2 GB"); - offset &= -(alignment as StackOffset); - - let frame_size = (offset as StackSize).wrapping_neg(); - self.frame_size = Some(frame_size); - frame_size - } } #[cfg(test)] @@ -401,75 +308,4 @@ mod tests { assert_eq!(slot2.alignment(16), 8); assert_eq!(slot2.alignment(32), 8); } - - #[test] - fn layout() { - let mut sss = StackSlots::new(); - - // An empty layout should have 0-sized stack frame. - assert_eq!(sss.layout(1), 0); - assert_eq!(sss.layout(16), 0); - - // Same for incoming arguments with non-negative offsets. - let in0 = sss.make_incoming_arg(types::I64, 0); - let in1 = sss.make_incoming_arg(types::I64, 8); - - assert_eq!(sss.layout(1), 0); - assert_eq!(sss.layout(16), 0); - assert_eq!(sss[in0].offset, 0); - assert_eq!(sss[in1].offset, 8); - - // Add some spill slots. - let ss0 = sss.make_spill_slot(types::I64); - let ss1 = sss.make_spill_slot(types::I32); - - assert_eq!(sss.layout(1), 12); - assert_eq!(sss[in0].offset, 0); - assert_eq!(sss[in1].offset, 8); - assert_eq!(sss[ss0].offset, -8); - assert_eq!(sss[ss1].offset, -12); - - assert_eq!(sss.layout(16), 16); - assert_eq!(sss[in0].offset, 0); - assert_eq!(sss[in1].offset, 8); - assert_eq!(sss[ss0].offset, -16); - assert_eq!(sss[ss1].offset, -4); - - // An incoming argument with negative offset counts towards the total frame size, but it - // should still pack nicely with the spill slots. - let in2 = sss.make_incoming_arg(types::I32, -4); - - assert_eq!(sss.layout(1), 16); - assert_eq!(sss[in0].offset, 0); - assert_eq!(sss[in1].offset, 8); - assert_eq!(sss[in2].offset, -4); - assert_eq!(sss[ss0].offset, -12); - assert_eq!(sss[ss1].offset, -16); - - assert_eq!(sss.layout(16), 16); - assert_eq!(sss[in0].offset, 0); - assert_eq!(sss[in1].offset, 8); - assert_eq!(sss[in2].offset, -4); - assert_eq!(sss[ss0].offset, -16); - assert_eq!(sss[ss1].offset, -8); - - // Finally, make sure there is room for the outgoing args. - let out0 = sss.get_outgoing_arg(types::I32, 0); - - assert_eq!(sss.layout(1), 20); - assert_eq!(sss[in0].offset, 0); - assert_eq!(sss[in1].offset, 8); - assert_eq!(sss[in2].offset, -4); - assert_eq!(sss[ss0].offset, -12); - assert_eq!(sss[ss1].offset, -16); - assert_eq!(sss[out0].offset, 0); - - assert_eq!(sss.layout(16), 32); - assert_eq!(sss[in0].offset, 0); - assert_eq!(sss[in1].offset, 8); - assert_eq!(sss[in2].offset, -4); - assert_eq!(sss[ss0].offset, -16); - assert_eq!(sss[ss1].offset, -8); - assert_eq!(sss[out0].offset, 0); - } } diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 0b09a8f10d..44f469ab4e 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -41,5 +41,6 @@ mod partition_slice; mod predicates; mod ref_slice; mod simple_gvn; +mod stack_layout; mod topo_order; mod write; diff --git a/lib/cretonne/src/result.rs b/lib/cretonne/src/result.rs index 960bd021d4..f4d7316fb5 100644 --- a/lib/cretonne/src/result.rs +++ b/lib/cretonne/src/result.rs @@ -7,7 +7,7 @@ use std::fmt; /// A compilation error. /// /// When Cretonne fails to compile a function, it will return one of these error codes. -#[derive(Debug)] +#[derive(Debug, PartialEq, Eq)] pub enum CtonError { /// An IL verifier error. /// diff --git a/lib/cretonne/src/stack_layout.rs b/lib/cretonne/src/stack_layout.rs new file mode 100644 index 0000000000..f968e56a07 --- /dev/null +++ b/lib/cretonne/src/stack_layout.rs @@ -0,0 +1,185 @@ +//! Computing stack layout. + +use ir::StackSlots; +use ir::stackslot::{StackSize, StackOffset, StackSlotKind}; +use result::CtonError; +use std::cmp::{min, max}; + +/// Compute the stack frame layout. +/// +/// Determine the total size of this stack frame and assign offsets to all `Spill` and `Local` +/// stack slots. +/// +/// The total frame size will be a multiple of `alignment` which must be a power of two. +/// +/// Returns the total stack frame size which is also saved in `frame.frame_size`. +/// +/// If the stack frame is too big, returns an `ImplLimitExceeded` error. +pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result { + // Each object and the whole stack frame must fit in 2 GB such that any relative offset within + // the frame fits in a `StackOffset`. + let max_size = StackOffset::max_value() as StackSize; + assert!(alignment.is_power_of_two() && alignment <= max_size); + + // We assume a stack that grows toward lower addresses as implemented by modern ISAs. The + // stack layout from high to low addresses will be: + // + // 1. incoming arguments. + // 2. spills + locals. + // 3. outgoing arguments. + // + // The incoming arguments can have both positive and negative offsets. A negative offset + // incoming arguments is usually the x86 return address pushed by the call instruction, but + // it can also be fixed stack slots pushed by an externally generated prologue. + // + // Both incoming and outgoing argument slots have fixed offsets that are treated as + // reserved zones by the layout algorithm. + + let mut incoming_min = 0; + let mut outgoing_max = 0; + let mut min_align = alignment; + + for ss in frame.keys() { + let slot = &frame[ss]; + + if slot.size > max_size { + return Err(CtonError::ImplLimitExceeded); + } + + match slot.kind { + StackSlotKind::IncomingArg => { + incoming_min = min(incoming_min, slot.offset); + } + StackSlotKind::OutgoingArg => { + let offset = slot.offset + .checked_add(slot.size as StackOffset) + .ok_or(CtonError::ImplLimitExceeded)?; + outgoing_max = max(outgoing_max, offset); + } + StackSlotKind::SpillSlot | StackSlotKind::Local => { + // Determine the smallest alignment of any local or spill slot. + min_align = slot.alignment(min_align); + } + } + } + + // Lay out spill slots and locals below the incoming arguments. + // The offset is negative, growing downwards. + // Start with the smallest alignments for better packing. + let mut offset = incoming_min; + assert!(min_align.is_power_of_two()); + while min_align <= alignment { + for ss in frame.keys() { + let slot = frame[ss].clone(); + + // Pick out locals and spill slots with exact alignment `min_align`. + match slot.kind { + StackSlotKind::SpillSlot | StackSlotKind::Local => { + if slot.alignment(alignment) != min_align { + continue; + } + } + _ => continue, + } + + offset = offset + .checked_sub(slot.size as StackOffset) + .ok_or(CtonError::ImplLimitExceeded)?; + + // Aligning the negative offset can never cause overflow. We're only clearing bits. + offset &= -(min_align as StackOffset); + frame.set_offset(ss, offset); + } + + // Move on to the next higher alignment. + min_align *= 2; + } + + // Finally, make room for the outgoing arguments. + offset = offset + .checked_sub(outgoing_max) + .ok_or(CtonError::ImplLimitExceeded)?; + offset &= -(alignment as StackOffset); + + let frame_size = (offset as StackSize).wrapping_neg(); + frame.frame_size = Some(frame_size); + Ok(frame_size) +} + +#[cfg(test)] +mod tests { + use ir::StackSlots; + use ir::types; + use super::layout_stack; + + #[test] + fn layout() { + let sss = &mut StackSlots::new(); + + // An empty layout should have 0-sized stack frame. + assert_eq!(layout_stack(sss, 1), Ok(0)); + assert_eq!(layout_stack(sss, 16), Ok(0)); + + // Same for incoming arguments with non-negative offsets. + let in0 = sss.make_incoming_arg(types::I64, 0); + let in1 = sss.make_incoming_arg(types::I64, 8); + + assert_eq!(layout_stack(sss, 1), Ok(0)); + assert_eq!(layout_stack(sss, 16), Ok(0)); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + + // Add some spill slots. + let ss0 = sss.make_spill_slot(types::I64); + let ss1 = sss.make_spill_slot(types::I32); + + assert_eq!(layout_stack(sss, 1), Ok(12)); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + assert_eq!(sss[ss0].offset, -8); + assert_eq!(sss[ss1].offset, -12); + + assert_eq!(layout_stack(sss, 16), Ok(16)); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + assert_eq!(sss[ss0].offset, -16); + assert_eq!(sss[ss1].offset, -4); + + // An incoming argument with negative offset counts towards the total frame size, but it + // should still pack nicely with the spill slots. + let in2 = sss.make_incoming_arg(types::I32, -4); + + assert_eq!(layout_stack(sss, 1), Ok(16)); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + assert_eq!(sss[in2].offset, -4); + assert_eq!(sss[ss0].offset, -12); + assert_eq!(sss[ss1].offset, -16); + + assert_eq!(layout_stack(sss, 16), Ok(16)); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + assert_eq!(sss[in2].offset, -4); + assert_eq!(sss[ss0].offset, -16); + assert_eq!(sss[ss1].offset, -8); + + // Finally, make sure there is room for the outgoing args. + let out0 = sss.get_outgoing_arg(types::I32, 0); + + assert_eq!(layout_stack(sss, 1), Ok(20)); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + assert_eq!(sss[in2].offset, -4); + assert_eq!(sss[ss0].offset, -12); + assert_eq!(sss[ss1].offset, -16); + assert_eq!(sss[out0].offset, 0); + + assert_eq!(layout_stack(sss, 16), Ok(32)); + assert_eq!(sss[in0].offset, 0); + assert_eq!(sss[in1].offset, 8); + assert_eq!(sss[in2].offset, -4); + assert_eq!(sss[ss0].offset, -16); + assert_eq!(sss[ss1].offset, -8); + assert_eq!(sss[out0].offset, 0); + } +} From 92392c6041ba02370be515978bb4a6a59878ea04 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 3 Aug 2017 13:48:30 -0700 Subject: [PATCH 0953/3084] Add a prologue_epilogue() hook to TargetIsa. This will compute the stack frame layout as appropriate for the function's calling convention and insert prologue and epilogue code. The default implementation is not useful, each target ISA will need to override this function. --- lib/cretonne/src/context.rs | 7 +++++++ lib/cretonne/src/isa/mod.rs | 13 +++++++++++++ 2 files changed, 20 insertions(+) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index d0288789f6..9967f1e11e 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -68,6 +68,7 @@ impl Context { self.legalize(isa)?; self.regalloc(isa)?; + self.prologue_epilogue(isa)?; self.relax_branches(isa) } @@ -135,6 +136,12 @@ impl Context { .run(isa, &mut self.func, &self.cfg, &self.domtree) } + /// Insert prologue and epilogues after computing the stack frame layout. + pub fn prologue_epilogue(&mut self, isa: &TargetIsa) -> CtonResult { + isa.prologue_epilogue(&mut self.func)?; + self.verify_if(isa) + } + /// Run the branch relaxation pass and return the final code size. pub fn relax_branches(&mut self, isa: &TargetIsa) -> Result { let code_size = relax_branches(&mut self.func, isa)?; diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 7caac93a67..08c45dd8b8 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -49,6 +49,7 @@ use flowgraph; use settings; use ir; use regalloc; +use result; use isa::enc_tables::Encodings; #[cfg(build_riscv)] @@ -227,6 +228,18 @@ pub trait TargetIsa { /// registers. fn allocatable_registers(&self, func: &ir::Function) -> regalloc::AllocatableSet; + /// Compute the stack layout and insert prologue and epilogue code into `func`. + /// + /// Return an error if the stack frame is too large. + fn prologue_epilogue(&self, func: &mut ir::Function) -> result::CtonResult { + // This default implementation is unlikely to be good enough. + use stack_layout::layout_stack; + + let align = if self.flags().is_64bit() { 8 } else { 4 }; + layout_stack(&mut func.stack_slots, align)?; + Ok(()) + } + /// Emit binary machine code for a single instruction into the `sink` trait object. /// /// Note that this will call `put*` methods on the trait object via its vtable which is not the From 05ba8dcab652d306b24c12ef1a547c087057ff6c Mon Sep 17 00:00:00 2001 From: Denis Merigoux Date: Thu, 3 Aug 2017 18:26:41 -0700 Subject: [PATCH 0954/3084] Added partial recompute of dominator tree in case of Ebb splitting (#135) * Partial recompute for dominator tree in case of ebb splitting Implemented via striding in RPO numbers --- lib/cretonne/src/dominator_tree.rs | 138 ++++++++++++++++++++++++++++- lib/cretonne/src/verifier/mod.rs | 30 +++++++ 2 files changed, 164 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 14f08e4263..6e61eeaa9d 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -7,10 +7,16 @@ use packed_option::PackedOption; use std::cmp::Ordering; +// RPO numbers are not first assigned in a contiguous way but as multiples of STRIDE, to leave +// room for modifications of the dominator tree. +const STRIDE: u32 = 4; + // Dominator tree node. We keep one of these per EBB. #[derive(Clone, Default)] struct DomNode { // Number of this node in a reverse post-order traversal of the CFG, starting from 1. + // This number is monotonic in the reverse postorder but not contiguous, since we leave + // holes for later localized modifications of the dominator tree. // Unreachable nodes get number 0, all others are positive. rpo_number: u32, @@ -137,7 +143,6 @@ impl DominatorTree { ebb_b = layout.inst_ebb(idom).expect("Dominator got removed."); inst_b = Some(idom); } - if a == ebb_b { inst_b } else { None } } @@ -266,10 +271,11 @@ impl DominatorTree { debug_assert_eq!(Some(entry_block), func.layout.entry_block()); // Do a first pass where we assign RPO numbers to all reachable nodes. - self.nodes[entry_block].rpo_number = 2; + self.nodes[entry_block].rpo_number = 2 * STRIDE; for (rpo_idx, &ebb) in postorder.iter().rev().enumerate() { // Update the current node and give it an RPO number. - // The entry block got 2, the rest start at 3. + // The entry block got 2, the rest start at 3 by multiples of STRIDE to leave + // room for future dominator tree modifications. // // Since `compute_idom` will only look at nodes with an assigned RPO number, the // function will never see an uninitialized predecessor. @@ -278,7 +284,7 @@ impl DominatorTree { // least one predecessor that has previously been visited during this RPO. self.nodes[ebb] = DomNode { idom: self.compute_idom(ebb, cfg, &func.layout).into(), - rpo_number: rpo_idx as u32 + 3, + rpo_number: (rpo_idx as u32 + 3) * STRIDE, } } @@ -323,11 +329,69 @@ impl DominatorTree { } } +impl DominatorTree { + /// When splitting an `Ebb` using `Layout::split_ebb`, you can use this method to update + /// the dominator tree locally rather than recomputing it. + /// + /// `old_ebb` is the `Ebb` before splitting, and `new_ebb` is the `Ebb` which now contains + /// the second half of `old_ebb`. `split_jump_inst` is the terminator jump instruction of + /// `old_ebb` that points to `new_ebb`. + pub fn recompute_split_ebb(&mut self, old_ebb: Ebb, new_ebb: Ebb, split_jump_inst: Inst) { + if !self.is_reachable(old_ebb) { + // old_ebb is unreachable, it stays so and new_ebb is unreachable too + *self.nodes.ensure(new_ebb) = Default::default(); + return; + } + // We use the RPO comparison on the postorder list so we invert the operands of the + // comparison + let old_ebb_postorder_index = + self.postorder + .as_slice() + .binary_search_by(|probe| self.rpo_cmp_ebb(old_ebb, *probe)) + .expect("the old ebb is not declared to the dominator tree"); + let new_ebb_rpo = self.insert_after_rpo(old_ebb, old_ebb_postorder_index, new_ebb); + *self.nodes.ensure(new_ebb) = DomNode { + rpo_number: new_ebb_rpo, + idom: Some(split_jump_inst).into(), + }; + + } + + // Insert new_ebb just after ebb in the RPO. This function checks + // if there is a gap in rpo numbers; if yes it returns the number in the gap and if + // not it renumbers. + fn insert_after_rpo(&mut self, ebb: Ebb, ebb_postorder_index: usize, new_ebb: Ebb) -> u32 { + let ebb_rpo_number = self.nodes[ebb].rpo_number; + let inserted_rpo_number = ebb_rpo_number + 1; + // If there is no gaps in RPo numbers to insert this new number, we iterate + // forward in RPO numbers and backwards in the postorder list of EBBs, renumbering the Ebbs + // until we find a gap + for (¤t_ebb, current_rpo) in + self.postorder[0..ebb_postorder_index] + .iter() + .rev() + .zip(inserted_rpo_number + 1..) { + if self.nodes[current_ebb].rpo_number < current_rpo { + // There is no gap, we renumber + self.nodes[current_ebb].rpo_number = current_rpo; + } else { + // There is a gap, we stop the renumbering and exit + break; + } + } + // TODO: insert in constant time? + self.postorder.insert(ebb_postorder_index, new_ebb); + inserted_rpo_number + } +} + #[cfg(test)] mod test { use flowgraph::ControlFlowGraph; use ir::{Function, InstBuilder, Cursor, types}; use super::*; + use ir::types::*; + use verifier::verify_context; #[test] fn empty() { @@ -465,4 +529,70 @@ mod test { assert!(!dt.dominates(jmp21, ebb2, &func.layout)); assert!(dt.dominates(jmp21, jmp21, &func.layout)); } + + #[test] + fn renumbering() { + let mut func = Function::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb100 = func.dfg.make_ebb(); + + let inst2; + let inst3; + let inst4; + let inst5; + { + let dfg = &mut func.dfg; + let cur = &mut Cursor::new(&mut func.layout); + + cur.insert_ebb(ebb0); + let cond = dfg.ins(cur).iconst(I32, 0); + inst2 = dfg.ins(cur).brz(cond, ebb0, &[]); + inst3 = dfg.ins(cur).brz(cond, ebb0, &[]); + inst4 = dfg.ins(cur).brz(cond, ebb0, &[]); + inst5 = dfg.ins(cur).brz(cond, ebb0, &[]); + dfg.ins(cur).jump(ebb100, &[]); + cur.insert_ebb(ebb100); + dfg.ins(cur).return_(&[]); + } + let mut cfg = ControlFlowGraph::with_function(&func); + let mut dt = DominatorTree::with_function(&func, &cfg); + + let ebb1 = func.dfg.make_ebb(); + func.layout.split_ebb(ebb1, inst2); + let middle_jump_inst = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb0); + func.dfg.ins(cur).jump(ebb1, &[]) + }; + dt.recompute_split_ebb(ebb0, ebb1, middle_jump_inst); + + let ebb2 = func.dfg.make_ebb(); + func.layout.split_ebb(ebb2, inst3); + let middle_jump_inst = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb1); + func.dfg.ins(cur).jump(ebb2, &[]) + }; + dt.recompute_split_ebb(ebb1, ebb2, middle_jump_inst); + + let ebb3 = func.dfg.make_ebb(); + func.layout.split_ebb(ebb3, inst4); + let middle_jump_inst = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb2); + func.dfg.ins(cur).jump(ebb3, &[]) + }; + dt.recompute_split_ebb(ebb2, ebb3, middle_jump_inst); + + let ebb4 = func.dfg.make_ebb(); + func.layout.split_ebb(ebb4, inst5); + let middle_jump_inst = { + let cur = &mut Cursor::new(&mut func.layout); + cur.goto_bottom(ebb3); + func.dfg.ins(cur).jump(ebb4, &[]) + }; + dt.recompute_split_ebb(ebb3, ebb4, middle_jump_inst); + cfg.compute(&func); + verify_context(&func, &cfg, &dt, None).unwrap(); + } } diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 751f6d5f6f..d3f05ef4c8 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -63,6 +63,8 @@ use std::error as std_error; use std::fmt::{self, Display, Formatter}; use std::result; use std::collections::BTreeSet; +use std::cmp::Ordering; +use iterators::IteratorExtras; pub use self::liveness::verify_liveness; pub use self::cssa::verify_cssa; @@ -409,6 +411,34 @@ impl<'a> Verifier<'a> { got); } } + // We also verify if the postorder defined by `DominatorTree` is sane + if self.domtree.cfg_postorder().len() != domtree.cfg_postorder().len() { + return err!(AnyEntity::Function, + "incorrect number of Ebbs in postorder traversal"); + } + for (index, (&true_ebb, &test_ebb)) in + self.domtree + .cfg_postorder() + .iter() + .zip(domtree.cfg_postorder().iter()) + .enumerate() { + if true_ebb != test_ebb { + return err!(test_ebb, + "invalid domtree, postorder ebb number {} should be {}, got {}", + index, + true_ebb, + test_ebb); + } + } + // We verify rpo_cmp on pairs of adjacent ebbs in the postorder + for (&prev_ebb, &next_ebb) in self.domtree.cfg_postorder().iter().adjacent_pairs() { + if domtree.rpo_cmp(prev_ebb, next_ebb, &self.func.layout) != Ordering::Greater { + return err!(next_ebb, + "invalid domtree, rpo_cmp does not says {} is greater than {}", + prev_ebb, + next_ebb); + } + } Ok(()) } From f1758820304c8643e6052e34019d7dad286f9ddd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Aug 2017 08:44:17 -0700 Subject: [PATCH 0955/3084] Avoid evaluating dbg!() arguments in a closure. The dbg!() macro should behave like a function call in how it evaluates its arguments, and captures by Rust closures are not fully compatible with path-specific borrowing. Specifically: let borrow = &mut obj.foo; dbg!("{}", obj.bar); would fail because the closure inside dbg!() would borrow all of obj instead of just obj.foo. Fix this by using the format_args!() macro to evaluate the dbg!() arguments and produce an fmt::Arguments object which can be safely passed to the thread-local closure for printing. The arguments are still evaluated inside an if { .. } which constant-folds away in release builds. --- lib/cretonne/src/dbg.rs | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/src/dbg.rs b/lib/cretonne/src/dbg.rs index dc7793f3d9..bf1b92ef75 100644 --- a/lib/cretonne/src/dbg.rs +++ b/lib/cretonne/src/dbg.rs @@ -15,7 +15,7 @@ use std::env; use std::ffi::OsStr; use std::fmt; use std::fs::File; -use std::io::{Write, BufWriter}; +use std::io::{self, Write}; use std::sync::atomic; use std::thread; @@ -56,20 +56,18 @@ fn initialize() -> bool { } thread_local! { - static WRITER : RefCell> = RefCell::new(open_file()); + static WRITER : RefCell> = RefCell::new(open_file()); } -/// Execute a closure with mutable access to the tracing file writer. +/// Write a line with the given format arguments. /// /// This is for use by the `dbg!` macro. -pub fn with_writer(f: F) -> R - where F: FnOnce(&mut Write) -> R -{ - WRITER.with(|rc| f(&mut *rc.borrow_mut())) +pub fn writeln_with_format_args(args: fmt::Arguments) -> io::Result<()> { + WRITER.with(|rc| writeln!(*rc.borrow_mut(), "{}", args)) } /// Open the tracing file for the current thread. -fn open_file() -> BufWriter { +fn open_file() -> io::BufWriter { let file = match thread::current().name() { None => File::create("cretonne.dbg"), Some(name) => { @@ -83,7 +81,7 @@ fn open_file() -> BufWriter { } } .expect("Can't open tracing file"); - BufWriter::new(file) + io::BufWriter::new(file) } /// Write a line to the debug trace file if tracing is enabled. @@ -95,7 +93,7 @@ macro_rules! dbg { if $crate::dbg::enabled() { // Drop the error result so we don't get compiler errors for ignoring it. // What are you going to do, log the error? - $crate::dbg::with_writer(|w| { writeln!(w, $($arg)+).ok(); }) + $crate::dbg::writeln_with_format_args(format_args!($($arg)+)).ok(); } } } From dba0df787c84d8ada72ec4b7742471672fe2165c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 3 Aug 2017 16:43:47 -0700 Subject: [PATCH 0956/3084] Move most Cursor methods into a CursorBase trait. The Cursor navigation methods all just depend on the cursor's position and layout reference. Make a CursorBase trait that provides access to this information with methods and implement the navigation methods on top of that. This makes it possible to have multiple types implement the cursor interface. --- lib/cretonne/meta/gen_legalizer.py | 2 +- lib/cretonne/src/binemit/relaxation.rs | 2 +- lib/cretonne/src/dominator_tree.rs | 2 +- lib/cretonne/src/flowgraph.rs | 2 +- lib/cretonne/src/ir/builder.rs | 4 +- lib/cretonne/src/ir/dfg.rs | 2 +- lib/cretonne/src/ir/layout.rs | 197 +++++++++++++----------- lib/cretonne/src/ir/mod.rs | 2 +- lib/cretonne/src/legalizer/boundary.rs | 6 +- lib/cretonne/src/legalizer/mod.rs | 2 +- lib/cretonne/src/legalizer/split.rs | 4 +- lib/cretonne/src/licm.rs | 2 +- lib/cretonne/src/loop_analysis.rs | 2 +- lib/cretonne/src/regalloc/coalescing.rs | 2 +- lib/cretonne/src/regalloc/coloring.rs | 2 +- lib/cretonne/src/regalloc/reload.rs | 2 +- lib/cretonne/src/regalloc/spilling.rs | 2 +- lib/cretonne/src/simple_gvn.rs | 2 +- lib/cretonne/src/topo_order.rs | 2 +- lib/frontend/src/ssa.rs | 5 +- 20 files changed, 136 insertions(+), 110 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 0a75fca8af..8a76d15e98 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -355,7 +355,7 @@ def gen_xform_group(xgrp, fmt, type_sets): 'cfg: &mut ::flowgraph::ControlFlowGraph, ' 'pos: &mut ir::Cursor) -> ' 'bool {{'.format(xgrp.name), '}'): - fmt.line('use ir::InstBuilder;') + fmt.line('use ir::{InstBuilder, CursorBase};') # Gen the instruction to be legalized. The cursor we're passed must be # pointing at an instruction. diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index 3420134d1b..0d0c07df35 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -28,7 +28,7 @@ //! ``` use binemit::CodeOffset; -use ir::{Function, DataFlowGraph, Cursor, InstructionData, Opcode, InstEncodings}; +use ir::{Function, DataFlowGraph, Cursor, CursorBase, InstructionData, Opcode, InstEncodings}; use isa::{TargetIsa, EncInfo}; use iterators::IteratorExtras; use result::CtonError; diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 6e61eeaa9d..9d47cc4bd7 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -388,7 +388,7 @@ impl DominatorTree { #[cfg(test)] mod test { use flowgraph::ControlFlowGraph; - use ir::{Function, InstBuilder, Cursor, types}; + use ir::{Function, InstBuilder, Cursor, CursorBase, types}; use super::*; use ir::types::*; use verifier::verify_context; diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index 45596d41ec..f7cb0c9729 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -136,7 +136,7 @@ impl ControlFlowGraph { #[cfg(test)] mod tests { use super::*; - use ir::{Function, InstBuilder, Cursor, types}; + use ir::{Function, InstBuilder, Cursor, CursorBase, types}; #[test] fn empty() { diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 332ce43b03..59de1f68b5 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -4,7 +4,7 @@ //! function. Many of its methods are generated from the meta language instruction definitions. use ir::types; -use ir::{InstructionData, DataFlowGraph, Cursor}; +use ir::{InstructionData, DataFlowGraph, Cursor, CursorBase}; use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, StackSlot, ValueList, MemFlags}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32, Uoffset32}; @@ -185,7 +185,7 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { #[cfg(test)] mod tests { - use ir::{Function, Cursor, InstBuilder, ValueDef}; + use ir::{Function, Cursor, CursorBase, InstBuilder, ValueDef}; use ir::types::*; use ir::condcodes::*; diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index ad67f7722b..49d4d13d6a 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -865,7 +865,7 @@ impl<'a> fmt::Display for DisplayInst<'a> { mod tests { use super::*; use ir::types; - use ir::{Function, Cursor, Opcode, InstructionData}; + use ir::{Function, Cursor, CursorBase, Opcode, InstructionData}; #[test] fn make_inst() { diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 0fa48edc6f..b1e068ff5c 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -658,40 +658,34 @@ pub enum CursorPosition { After(Ebb), } -impl<'f> Cursor<'f> { - /// Create a new `Cursor` for `layout`. - /// The cursor holds a mutable reference to `layout` for its entire lifetime. - pub fn new(layout: &'f mut Layout) -> Cursor { - Cursor { - layout, - pos: CursorPosition::Nowhere, - } - } +/// All cursor types implement the `CursorBase` which provides common navigation operations. +pub trait CursorBase { + /// Get the current cursor position. + fn position(&self) -> CursorPosition; - /// Get the current position. - pub fn position(&self) -> CursorPosition { - self.pos - } + /// Set the current position. + fn set_position(&mut self, pos: CursorPosition); - /// Move the cursor to a new position. - pub fn set_position(&mut self, pos: CursorPosition) { - self.pos = pos; - } + /// Borrow a reference to the function layout that this cursor is navigating. + fn layout(&self) -> &Layout; + + /// Borrow a mutable reference to the function layout that this cursor is navigating. + fn layout_mut(&mut self) -> &mut Layout; /// Get the EBB corresponding to the current position. - pub fn current_ebb(&self) -> Option { + fn current_ebb(&self) -> Option { use self::CursorPosition::*; - match self.pos { + match self.position() { Nowhere => None, - At(inst) => self.layout.inst_ebb(inst), + At(inst) => self.layout().inst_ebb(inst), Before(ebb) | After(ebb) => Some(ebb), } } /// Get the instruction corresponding to the current position, if any. - pub fn current_inst(&self) -> Option { + fn current_inst(&self) -> Option { use self::CursorPosition::*; - match self.pos { + match self.position() { At(inst) => Some(inst), _ => None, } @@ -699,24 +693,24 @@ impl<'f> Cursor<'f> { /// Go to a specific instruction which must be inserted in the layout. /// New instructions will be inserted before `inst`. - pub fn goto_inst(&mut self, inst: Inst) { - assert!(self.layout.inst_ebb(inst).is_some()); - self.pos = CursorPosition::At(inst); + fn goto_inst(&mut self, inst: Inst) { + assert!(self.layout().inst_ebb(inst).is_some()); + self.set_position(CursorPosition::At(inst)); } /// Go to the top of `ebb` which must be inserted into the layout. /// At this position, instructions cannot be inserted, but `next_inst()` will move to the first /// instruction in `ebb`. - pub fn goto_top(&mut self, ebb: Ebb) { - assert!(self.layout.is_ebb_inserted(ebb)); - self.pos = CursorPosition::Before(ebb); + fn goto_top(&mut self, ebb: Ebb) { + assert!(self.layout().is_ebb_inserted(ebb)); + self.set_position(CursorPosition::Before(ebb)); } /// Go to the bottom of `ebb` which must be inserted into the layout. /// At this position, inserted instructions will be appended to `ebb`. - pub fn goto_bottom(&mut self, ebb: Ebb) { - assert!(self.layout.is_ebb_inserted(ebb)); - self.pos = CursorPosition::After(ebb); + fn goto_bottom(&mut self, ebb: Ebb) { + assert!(self.layout().is_ebb_inserted(ebb)); + self.set_position(CursorPosition::After(ebb)); } /// Go to the top of the next EBB in layout order and return it. @@ -731,7 +725,7 @@ impl<'f> Cursor<'f> { /// /// ``` /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::ir::layout::Cursor; + /// # use cretonne::ir::layout::{Cursor, CursorBase}; /// fn edit_func(func: &mut Function) { /// let mut cursor = Cursor::new(&mut func.layout); /// while let Some(ebb) = cursor.next_ebb() { @@ -739,16 +733,16 @@ impl<'f> Cursor<'f> { /// } /// } /// ``` - pub fn next_ebb(&mut self) -> Option { + fn next_ebb(&mut self) -> Option { let next = if let Some(ebb) = self.current_ebb() { - self.layout.ebbs[ebb].next.expand() + self.layout().ebbs[ebb].next.expand() } else { - self.layout.first_ebb - }; - self.pos = match next { - Some(ebb) => CursorPosition::Before(ebb), - None => CursorPosition::Nowhere, + self.layout().first_ebb }; + self.set_position(match next { + Some(ebb) => CursorPosition::Before(ebb), + None => CursorPosition::Nowhere, + }); next } @@ -764,7 +758,7 @@ impl<'f> Cursor<'f> { /// /// ``` /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::ir::layout::Cursor; + /// # use cretonne::ir::layout::{Cursor, CursorBase}; /// fn edit_func(func: &mut Function) { /// let mut cursor = Cursor::new(&mut func.layout); /// while let Some(ebb) = cursor.prev_ebb() { @@ -772,16 +766,16 @@ impl<'f> Cursor<'f> { /// } /// } /// ``` - pub fn prev_ebb(&mut self) -> Option { + fn prev_ebb(&mut self) -> Option { let prev = if let Some(ebb) = self.current_ebb() { - self.layout.ebbs[ebb].prev.expand() + self.layout().ebbs[ebb].prev.expand() } else { - self.layout.last_ebb - }; - self.pos = match prev { - Some(ebb) => CursorPosition::After(ebb), - None => CursorPosition::Nowhere, + self.layout().last_ebb }; + self.set_position(match prev { + Some(ebb) => CursorPosition::After(ebb), + None => CursorPosition::Nowhere, + }); prev } @@ -801,7 +795,7 @@ impl<'f> Cursor<'f> { /// /// ``` /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::ir::layout::Cursor; + /// # use cretonne::ir::layout::{Cursor, CursorBase}; /// fn edit_ebb(func: &mut Function, ebb: Ebb) { /// let mut cursor = Cursor::new(&mut func.layout); /// cursor.goto_top(ebb); @@ -816,7 +810,7 @@ impl<'f> Cursor<'f> { /// /// ``` /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::ir::layout::Cursor; + /// # use cretonne::ir::layout::{Cursor, CursorBase}; /// fn edit_func(func: &mut Function) { /// let mut cursor = Cursor::new(&mut func.layout); /// while let Some(ebb) = cursor.next_ebb() { @@ -826,27 +820,28 @@ impl<'f> Cursor<'f> { /// } /// } /// ``` - pub fn next_inst(&mut self) -> Option { + fn next_inst(&mut self) -> Option { use self::CursorPosition::*; - match self.pos { + match self.position() { Nowhere | After(..) => None, At(inst) => { - if let Some(next) = self.layout.insts[inst].next.expand() { - self.pos = At(next); + if let Some(next) = self.layout().insts[inst].next.expand() { + self.set_position(At(next)); Some(next) } else { - self.pos = After(self.layout - .inst_ebb(inst) - .expect("current instruction removed?")); + let pos = After(self.layout() + .inst_ebb(inst) + .expect("current instruction removed?")); + self.set_position(pos); None } } Before(ebb) => { - if let Some(next) = self.layout.ebbs[ebb].first_inst.expand() { - self.pos = At(next); + if let Some(next) = self.layout().ebbs[ebb].first_inst.expand() { + self.set_position(At(next)); Some(next) } else { - self.pos = After(ebb); + self.set_position(After(ebb)); None } } @@ -869,7 +864,7 @@ impl<'f> Cursor<'f> { /// /// ``` /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::ir::layout::Cursor; + /// # use cretonne::ir::layout::{Cursor, CursorBase}; /// fn edit_ebb(func: &mut Function, ebb: Ebb) { /// let mut cursor = Cursor::new(&mut func.layout); /// cursor.goto_bottom(ebb); @@ -878,27 +873,28 @@ impl<'f> Cursor<'f> { /// } /// } /// ``` - pub fn prev_inst(&mut self) -> Option { + fn prev_inst(&mut self) -> Option { use self::CursorPosition::*; - match self.pos { + match self.position() { Nowhere | Before(..) => None, At(inst) => { - if let Some(prev) = self.layout.insts[inst].prev.expand() { - self.pos = At(prev); + if let Some(prev) = self.layout().insts[inst].prev.expand() { + self.set_position(At(prev)); Some(prev) } else { - self.pos = Before(self.layout - .inst_ebb(inst) - .expect("current instruction removed?")); + let pos = Before(self.layout() + .inst_ebb(inst) + .expect("current instruction removed?")); + self.set_position(pos); None } } After(ebb) => { - if let Some(prev) = self.layout.ebbs[ebb].last_inst.expand() { - self.pos = At(prev); + if let Some(prev) = self.layout().ebbs[ebb].last_inst.expand() { + self.set_position(At(prev)); Some(prev) } else { - self.pos = Before(ebb); + self.set_position(Before(ebb)); None } } @@ -914,12 +910,12 @@ impl<'f> Cursor<'f> { /// /// In either case, the cursor is not moved, such that repeated calls to `insert_inst()` causes /// instructions to appear in insertion order in the EBB. - pub fn insert_inst(&mut self, inst: Inst) { + fn insert_inst(&mut self, inst: Inst) { use self::CursorPosition::*; - match self.pos { + match self.position() { Nowhere | Before(..) => panic!("Invalid insert_inst position"), - At(cur) => self.layout.insert_inst(inst, cur), - After(ebb) => self.layout.append_inst(inst, ebb), + At(cur) => self.layout_mut().insert_inst(inst, cur), + After(ebb) => self.layout_mut().append_inst(inst, ebb), } } @@ -928,10 +924,10 @@ impl<'f> Cursor<'f> { /// The cursor is left pointing at the position following the current instruction. /// /// Return the instruction that was removed. - pub fn remove_inst(&mut self) -> Inst { + fn remove_inst(&mut self) -> Inst { let inst = self.current_inst().expect("No instruction to remove"); self.next_inst(); - self.layout.remove_inst(inst); + self.layout_mut().remove_inst(inst); inst } @@ -940,10 +936,10 @@ impl<'f> Cursor<'f> { /// The cursor is left pointing at the position preceding the current instruction. /// /// Return the instruction that was removed. - pub fn remove_inst_and_step_back(&mut self) -> Inst { + fn remove_inst_and_step_back(&mut self) -> Inst { let inst = self.current_inst().expect("No instruction to remove"); self.prev_inst(); - self.layout.remove_inst(inst); + self.layout_mut().remove_inst(inst); inst } @@ -961,27 +957,56 @@ impl<'f> Cursor<'f> { /// /// This means that is is always valid to call this method, and it always leaves the cursor in /// a state that will insert instructions into the new EBB. - pub fn insert_ebb(&mut self, new_ebb: Ebb) { + fn insert_ebb(&mut self, new_ebb: Ebb) { use self::CursorPosition::*; - match self.pos { + match self.position() { At(inst) => { - self.layout.split_ebb(new_ebb, inst); + self.layout_mut().split_ebb(new_ebb, inst); // All other cases move to `After(ebb)`, but in this case we we'll stay `At(inst)`. return; } - Nowhere => self.layout.append_ebb(new_ebb), - Before(ebb) => self.layout.insert_ebb(new_ebb, ebb), - After(ebb) => self.layout.insert_ebb_after(new_ebb, ebb), + Nowhere => self.layout_mut().append_ebb(new_ebb), + Before(ebb) => self.layout_mut().insert_ebb(new_ebb, ebb), + After(ebb) => self.layout_mut().insert_ebb_after(new_ebb, ebb), } // For everything but `At(inst)` we end up appending to the new EBB. - self.pos = After(new_ebb); + self.set_position(After(new_ebb)); + } +} + +impl<'f> CursorBase for Cursor<'f> { + fn position(&self) -> CursorPosition { + self.pos + } + + fn set_position(&mut self, pos: CursorPosition) { + self.pos = pos; + } + + fn layout(&self) -> &Layout { + self.layout + } + + fn layout_mut(&mut self) -> &mut Layout { + self.layout + } +} + +impl<'f> Cursor<'f> { + /// Create a new `Cursor` for `layout`. + /// The cursor holds a mutable reference to `layout` for its entire lifetime. + pub fn new(layout: &'f mut Layout) -> Cursor { + Cursor { + layout, + pos: CursorPosition::Nowhere, + } } } #[cfg(test)] mod tests { - use super::{Layout, Cursor, CursorPosition}; + use super::{Layout, Cursor, CursorBase, CursorPosition}; use entity_ref::EntityRef; use ir::{Ebb, Inst, ProgramOrder}; use std::cmp::Ordering; diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 76f37efa3c..dfea1048b5 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -27,7 +27,7 @@ pub use ir::stackslot::{StackSlots, StackSlotKind, StackSlotData}; pub use ir::jumptable::JumpTableData; pub use ir::valueloc::{ValueLoc, ArgumentLoc}; pub use ir::dfg::{DataFlowGraph, ValueDef}; -pub use ir::layout::{Layout, Cursor}; +pub use ir::layout::{Layout, CursorBase, Cursor}; pub use ir::function::Function; pub use ir::builder::InstBuilder; pub use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index caed486f3a..7d248c0cfa 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -19,9 +19,9 @@ use abi::{legalize_abi_value, ValueConversion}; use flowgraph::ControlFlowGraph; -use ir::{Function, Cursor, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, Signature, SigRef, - ArgumentType, ArgumentPurpose, ArgumentLoc, ValueLoc, ValueLocations, StackSlots, - StackSlotKind}; +use ir::{Function, Cursor, CursorBase, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, + Signature, SigRef, ArgumentType, ArgumentPurpose, ArgumentLoc, ValueLoc, ValueLocations, + StackSlots, StackSlotKind}; use ir::instructions::CallInfo; use isa::TargetIsa; use legalizer::split::{isplit, vsplit}; diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 4494eb60e9..9cc37dd896 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -15,7 +15,7 @@ use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; -use ir::{self, Function, Cursor}; +use ir::{self, Function, Cursor, CursorBase}; use ir::condcodes::IntCC; use isa::TargetIsa; use bitset::BitSet; diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index 3e3a36f619..da61303e05 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -65,8 +65,8 @@ //! instructions. These loops will remain in the program. use flowgraph::ControlFlowGraph; -use ir::{DataFlowGraph, Ebb, Inst, Cursor, Value, Type, Opcode, ValueDef, InstructionData, - InstBuilder}; +use ir::{DataFlowGraph, Ebb, Inst, Cursor, CursorBase, Value, Type, Opcode, ValueDef, + InstructionData, InstBuilder}; use std::iter; /// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 10e0558999..8ed88d47dd 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -1,6 +1,6 @@ //! A Loop Invariant Code Motion optimization pass -use ir::{Function, Ebb, Inst, Value, Cursor, Type, InstBuilder, Layout}; +use ir::{Function, Ebb, Inst, Value, Cursor, CursorBase, Type, InstBuilder, Layout}; use flowgraph::ControlFlowGraph; use std::collections::HashSet; use dominator_tree::DominatorTree; diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index d0a9d505bd..5a4607ba79 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -199,7 +199,7 @@ impl LoopAnalysis { #[cfg(test)] mod test { - use ir::{Function, InstBuilder, Cursor, types}; + use ir::{Function, InstBuilder, Cursor, CursorBase, types}; use loop_analysis::{Loop, LoopAnalysis}; use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 29051bf426..4b4ffff34e 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -8,7 +8,7 @@ use dbg::DisplayList; use dominator_tree::DominatorTree; use flowgraph::{ControlFlowGraph, BasicBlock}; -use ir::{DataFlowGraph, Layout, Cursor, InstBuilder, ValueDef}; +use ir::{DataFlowGraph, Layout, Cursor, CursorBase, InstBuilder, ValueDef}; use ir::{Function, Ebb, Inst, Value, ExpandedProgramPoint}; use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 21a4453bb2..02e4d47f81 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -43,7 +43,7 @@ //! The exception is the entry block whose arguments are colored from the ABI requirements. use dominator_tree::DominatorTree; -use ir::{Ebb, Inst, Value, Function, Cursor, ValueLoc, DataFlowGraph, Layout}; +use ir::{Ebb, Inst, Value, Function, Cursor, CursorBase, ValueLoc, DataFlowGraph, Layout}; use ir::{InstEncodings, ValueLocations}; use ir::{InstBuilder, Signature, ArgumentType, ArgumentLoc}; use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index e22978938b..18e834179b 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -11,7 +11,7 @@ use dominator_tree::DominatorTree; use ir::{Ebb, Inst, Value, Function, Signature, DataFlowGraph, InstEncodings}; -use ir::layout::{Cursor, CursorPosition}; +use ir::layout::{Cursor, CursorBase, CursorPosition}; use ir::{InstBuilder, Opcode, ArgumentType, ArgumentLoc}; use isa::RegClass; use isa::{TargetIsa, Encoding, EncInfo, RecipeConstraints, ConstraintKind}; diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 8b3e8b7837..e1e610c3f0 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -16,7 +16,7 @@ //! operands. use dominator_tree::DominatorTree; -use ir::{DataFlowGraph, Layout, Cursor, InstBuilder}; +use ir::{DataFlowGraph, Layout, Cursor, CursorBase, InstBuilder}; use ir::{Function, Ebb, Inst, Value, ValueLoc, SigRef}; use ir::{InstEncodings, StackSlots, ValueLocations}; use isa::registers::{RegClassMask, RegClassIndex}; diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index bb339fbd4d..078bca2857 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -2,7 +2,7 @@ use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; -use ir::{Cursor, InstructionData, Function, Inst, Opcode}; +use ir::{Cursor, CursorBase, InstructionData, Function, Inst, Opcode}; use std::collections::HashMap; /// Test whether the given opcode is unsafe to even consider for GVN. diff --git a/lib/cretonne/src/topo_order.rs b/lib/cretonne/src/topo_order.rs index 323df99ef0..938e4a876a 100644 --- a/lib/cretonne/src/topo_order.rs +++ b/lib/cretonne/src/topo_order.rs @@ -81,7 +81,7 @@ impl TopoOrder { mod test { use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; - use ir::{Function, InstBuilder, Cursor}; + use ir::{Function, InstBuilder, Cursor, CursorBase}; use std::iter; use super::*; diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 3794d332b1..fc5c85f1dd 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -5,7 +5,8 @@ //! In: Jhala R., De Bosschere K. (eds) Compiler Construction. CC 2013. //! Lecture Notes in Computer Science, vol 7791. Springer, Berlin, Heidelberg -use cretonne::ir::{Ebb, Value, Inst, Type, DataFlowGraph, JumpTables, Layout, Cursor, InstBuilder}; +use cretonne::ir::{Ebb, Value, Inst, Type, DataFlowGraph, JumpTables, Layout, Cursor, CursorBase, + InstBuilder}; use cretonne::ir::instructions::BranchInfo; use std::hash::Hash; use cretonne::entity_map::{EntityMap, PrimaryEntityData}; @@ -607,7 +608,7 @@ impl SSABuilder #[cfg(test)] mod tests { use cretonne::entity_ref::EntityRef; - use cretonne::ir::{Function, InstBuilder, Cursor, Inst, JumpTableData}; + use cretonne::ir::{Function, InstBuilder, Cursor, CursorBase, Inst, JumpTableData}; use cretonne::ir::types::*; use cretonne::verify_function; use cretonne::ir::instructions::BranchInfo; From aa0c37235a4ad93ce9cef9701d9c9fffdf75415e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 3 Aug 2017 18:03:58 -0700 Subject: [PATCH 0957/3084] Add CursorBase builder methods. The CursorBase::at_* are convenience methods for building a cursor that points to a specific instruction. --- lib/cretonne/src/ir/layout.rs | 41 +++++++++++++++++++++++++ lib/cretonne/src/regalloc/coalescing.rs | 7 ++--- 2 files changed, 43 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index b1e068ff5c..081a289053 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -672,6 +672,47 @@ pub trait CursorBase { /// Borrow a mutable reference to the function layout that this cursor is navigating. fn layout_mut(&mut self) -> &mut Layout; + /// Rebuild this cursor positioned at `inst`. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// fn edit_func(func: &mut Function, inst: Inst) { + /// let mut pos = Cursor::new(&mut func.layout).at_inst(inst); + /// + /// // Use `pos`... + /// } + /// ``` + fn at_inst(mut self, inst: Inst) -> Self + where Self: Sized + { + self.goto_inst(inst); + self + } + + /// Rebuild this cursor positioned at the first instruction in `ebb`. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// fn edit_func(func: &mut Function, ebb: Ebb) { + /// let mut pos = Cursor::new(&mut func.layout).at_first_inst(ebb); + /// + /// // Use `pos`... + /// } + /// ``` + fn at_first_inst(mut self, ebb: Ebb) -> Self + where Self: Sized + { + let inst = self.layout().ebbs[ebb].first_inst.expect("Empty EBB"); + self.goto_inst(inst); + self + } + /// Get the EBB corresponding to the current position. fn current_ebb(&self) -> Option { use self::CursorPosition::*; diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 4b4ffff34e..b21175ad87 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -466,8 +466,7 @@ impl<'a> Context<'a> { -> Value { let copy; { - let mut pos = Cursor::new(&mut self.func.layout); - pos.goto_inst(pred_inst); + let mut pos = Cursor::new(&mut self.func.layout).at_inst(pred_inst); copy = self.func.dfg.ins(&mut pos).copy(pred_val); } let inst = self.func.dfg.value_def(copy).unwrap_inst(); @@ -507,9 +506,7 @@ impl<'a> Context<'a> { // Insert a copy instruction at the top of ebb. { - let mut pos = Cursor::new(&mut self.func.layout); - pos.goto_top(ebb); - pos.next_inst(); + let mut pos = Cursor::new(&mut self.func.layout).at_first_inst(ebb); self.func .dfg .ins(&mut pos) From 621abb50269b9f9924a843183307d617defba3e4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Aug 2017 11:23:37 -0700 Subject: [PATCH 0958/3084] Reorganize the instruction builder traits. Leave the primary InstBuilderBase trait alone, but add an alternative InstInserterBase trait that can be implemented instead by builders that always allocate new instructions with dfg.make_inst(). Any implementation of InstInserterBase can be used as an instruction builder by wrapping it in an InsertBuilder. The InsertBuilder type adds additional functionality via the with_results() method which makes it possible to override the result values on the instruction that is built. The motivation for this shuffle is that the with_result() functionality can now be reused by different kinds of instruction builders, as long as they insert new instructions. So ReplaceBuilder doesn't get with_results(). --- lib/cretonne/src/ir/builder.rs | 109 +++++++++++++++++++++------------ lib/cretonne/src/ir/dfg.rs | 6 +- lib/cretonne/src/ir/layout.rs | 35 ++++++++++- lib/cretonne/src/ir/mod.rs | 23 ++++--- 4 files changed, 117 insertions(+), 56 deletions(-) diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 59de1f68b5..697acfebd5 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -4,7 +4,7 @@ //! function. Many of its methods are generated from the meta language instruction definitions. use ir::types; -use ir::{InstructionData, DataFlowGraph, Cursor, CursorBase}; +use ir::{InstructionData, DataFlowGraph}; use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, StackSlot, ValueList, MemFlags}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32, Uoffset32}; @@ -43,23 +43,44 @@ include!(concat!(env!("OUT_DIR"), "/builder.rs")); /// Any type implementing `InstBuilderBase` gets all the `InstBuilder` methods for free. impl<'f, T: InstBuilderBase<'f>> InstBuilder<'f> for T {} -/// Builder that inserts an instruction at the current cursor position. +/// Base trait for instruction inserters. /// -/// An `InsertBuilder` holds mutable references to a data flow graph and a layout cursor. It -/// provides convenience methods for creating and inserting instructions at the current cursor -/// position. -pub struct InsertBuilder<'c, 'fc: 'c, 'fd> { - pos: &'c mut Cursor<'fc>, - dfg: &'fd mut DataFlowGraph, +/// This is an alternative base trait for an instruction builder to implement. +/// +/// An instruction inserter can be adapted into an instruction builder by wrapping it in an +/// `InsertBuilder`. This provides some common functionality for instruction builders that insert +/// new instructions, as opposed to the `ReplaceBuilder` which overwrites existing instructions. +pub trait InstInserterBase<'f>: Sized { + /// Get an immutable reference to the data flow graph. + fn data_flow_graph(&self) -> &DataFlowGraph; + + /// Get a mutable reference to the data flow graph. + fn data_flow_graph_mut(&mut self) -> &mut DataFlowGraph; + + /// Insert a new instruction which belongs to the DFG. + fn insert_built_inst(self, inst: Inst, ctrl_typevar: Type) -> &'f mut DataFlowGraph; } -impl<'c, 'fc, 'fd> InsertBuilder<'c, 'fc, 'fd> { +use std::marker::PhantomData; + +/// Builder that inserts an instruction at the current position. +/// +/// An `InsertBuilder` is a wrapper for an `InstInserterBase` that turns it into an instruction +/// builder with some additional facilities for creating instructions that reuse existing values as +/// their results. +pub struct InsertBuilder<'f, IIB: InstInserterBase<'f>> { + inserter: IIB, + unused: PhantomData<&'f u32>, +} + +impl<'f, IIB: InstInserterBase<'f>> InsertBuilder<'f, IIB> { /// Create a new builder which inserts instructions at `pos`. /// The `dfg` and `pos.layout` references should be from the same `Function`. - pub fn new(dfg: &'fd mut DataFlowGraph, - pos: &'c mut Cursor<'fc>) - -> InsertBuilder<'c, 'fc, 'fd> { - InsertBuilder { dfg, pos } + pub fn new(inserter: IIB) -> InsertBuilder<'f, IIB> { + InsertBuilder { + inserter, + unused: PhantomData, + } } /// Reuse result values in `reuse`. @@ -69,13 +90,13 @@ impl<'c, 'fc, 'fd> InsertBuilder<'c, 'fc, 'fd> { /// missing result values will be allocated as normal. /// /// The `reuse` argument is expected to be an array of `Option`. - pub fn with_results(self, reuse: Array) -> InsertReuseBuilder<'c, 'fc, 'fd, Array> + pub fn with_results(self, reuse: Array) -> InsertReuseBuilder<'f, IIB, Array> where Array: AsRef<[Option]> { InsertReuseBuilder { - dfg: self.dfg, - pos: self.pos, + inserter: self.inserter, reuse, + unused: PhantomData, } } @@ -86,57 +107,65 @@ impl<'c, 'fc, 'fd> InsertBuilder<'c, 'fc, 'fd> { /// /// This method should only be used when building an instruction with exactly one result. Use /// `with_results()` for the more general case. - pub fn with_result(self, v: Value) -> InsertReuseBuilder<'c, 'fc, 'fd, [Option; 1]> { + pub fn with_result(self, v: Value) -> InsertReuseBuilder<'f, IIB, [Option; 1]> { // TODO: Specialize this to return a different builder that just attaches `v` instead of // calling `make_inst_results_reusing()`. self.with_results([Some(v)]) } } -impl<'c, 'fc, 'fd> InstBuilderBase<'fd> for InsertBuilder<'c, 'fc, 'fd> { +impl<'f, IIB: InstInserterBase<'f>> InstBuilderBase<'f> for InsertBuilder<'f, IIB> { fn data_flow_graph(&self) -> &DataFlowGraph { - self.dfg + self.inserter.data_flow_graph() } fn data_flow_graph_mut(&mut self) -> &mut DataFlowGraph { - self.dfg + self.inserter.data_flow_graph_mut() } - fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'fd mut DataFlowGraph) { - let inst = self.dfg.make_inst(data); - self.dfg.make_inst_results(inst, ctrl_typevar); - self.pos.insert_inst(inst); - (inst, self.dfg) + fn build(mut self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'f mut DataFlowGraph) { + let inst; + { + let dfg = self.inserter.data_flow_graph_mut(); + inst = dfg.make_inst(data); + dfg.make_inst_results(inst, ctrl_typevar); + } + (inst, self.inserter.insert_built_inst(inst, ctrl_typevar)) } } /// Builder that inserts a new instruction like `InsertBuilder`, but reusing result values. -pub struct InsertReuseBuilder<'c, 'fc: 'c, 'fd, Array> - where Array: AsRef<[Option]> +pub struct InsertReuseBuilder<'f, IIB, Array> + where IIB: InstInserterBase<'f>, + Array: AsRef<[Option]> { - pos: &'c mut Cursor<'fc>, - dfg: &'fd mut DataFlowGraph, + inserter: IIB, reuse: Array, + unused: PhantomData<&'f u32>, } -impl<'c, 'fc, 'fd, Array> InstBuilderBase<'fd> for InsertReuseBuilder<'c, 'fc, 'fd, Array> - where Array: AsRef<[Option]> +impl<'f, IIB, Array> InstBuilderBase<'f> for InsertReuseBuilder<'f, IIB, Array> + where IIB: InstInserterBase<'f>, + Array: AsRef<[Option]> { fn data_flow_graph(&self) -> &DataFlowGraph { - self.dfg + self.inserter.data_flow_graph() } fn data_flow_graph_mut(&mut self) -> &mut DataFlowGraph { - self.dfg + self.inserter.data_flow_graph_mut() } - fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'fd mut DataFlowGraph) { - let inst = self.dfg.make_inst(data); - // Make an `Interator>`. - let ru = self.reuse.as_ref().iter().cloned(); - self.dfg.make_inst_results_reusing(inst, ctrl_typevar, ru); - self.pos.insert_inst(inst); - (inst, self.dfg) + fn build(mut self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'f mut DataFlowGraph) { + let inst; + { + let dfg = self.inserter.data_flow_graph_mut(); + inst = dfg.make_inst(data); + // Make an `Interator>`. + let ru = self.reuse.as_ref().iter().cloned(); + dfg.make_inst_results_reusing(inst, ctrl_typevar, ru); + } + (inst, self.inserter.insert_built_inst(inst, ctrl_typevar)) } } diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 49d4d13d6a..959b29c8e8 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -5,7 +5,7 @@ use isa::TargetIsa; use ir::builder::{InsertBuilder, ReplaceBuilder}; use ir::extfunc::ExtFuncData; use ir::instructions::{Opcode, InstructionData, CallInfo}; -use ir::layout::Cursor; +use ir::layout::{Cursor, LayoutCursorInserter}; use ir::types; use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueList, ValueListPool}; use write::write_operands; @@ -486,8 +486,8 @@ impl DataFlowGraph { /// Create an `InsertBuilder` that will insert an instruction at the cursor's current position. pub fn ins<'c, 'fc: 'c, 'fd>(&'fd mut self, at: &'c mut Cursor<'fc>) - -> InsertBuilder<'c, 'fc, 'fd> { - InsertBuilder::new(self, at) + -> InsertBuilder<'fd, LayoutCursorInserter<'c, 'fc, 'fd>> { + InsertBuilder::new(LayoutCursorInserter::new(at, self)) } /// Create a `ReplaceBuilder` that will replace `inst` with a new instruction in place. diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 081a289053..b219a13bb2 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -7,7 +7,8 @@ use std::cmp; use std::iter::{Iterator, IntoIterator}; use entity_map::EntityMap; use packed_option::PackedOption; -use ir::entities::{Ebb, Inst}; +use ir::{Ebb, Inst, Type, DataFlowGraph}; +use ir::builder::InstInserterBase; use ir::progpoint::{ProgramOrder, ExpandedProgramPoint}; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not @@ -1044,6 +1045,38 @@ impl<'f> Cursor<'f> { } } +/// An instruction inserter which can be used to build and insert instructions at a cursor +/// position. +/// +/// This is used by `dfg.ins()`. +pub struct LayoutCursorInserter<'c, 'fc: 'c, 'fd> { + pos: &'c mut Cursor<'fc>, + dfg: &'fd mut DataFlowGraph, +} + +impl<'c, 'fc: 'c, 'fd> LayoutCursorInserter<'c, 'fc, 'fd> { + /// Create a new inserter. Don't use this, use `dfg.ins(pos)`. + pub fn new(pos: &'c mut Cursor<'fc>, + dfg: &'fd mut DataFlowGraph) + -> LayoutCursorInserter<'c, 'fc, 'fd> { + LayoutCursorInserter { pos, dfg } + } +} + +impl<'c, 'fc: 'c, 'fd> InstInserterBase<'fd> for LayoutCursorInserter<'c, 'fc, 'fd> { + fn data_flow_graph(&self) -> &DataFlowGraph { + self.dfg + } + + fn data_flow_graph_mut(&mut self) -> &mut DataFlowGraph { + self.dfg + } + + fn insert_built_inst(self, inst: Inst, _ctrl_typevar: Type) -> &'fd mut DataFlowGraph { + self.pos.insert_inst(inst); + self.dfg + } +} #[cfg(test)] mod tests { diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index dfea1048b5..443d70bd1c 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -17,22 +17,21 @@ mod memflags; mod progpoint; mod valueloc; -pub use ir::funcname::FunctionName; +pub use ir::builder::{InstBuilder, InstBuilderBase, InstInserterBase, InsertBuilder}; +pub use ir::dfg::{DataFlowGraph, ValueDef}; +pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable, FuncRef, SigRef}; pub use ir::extfunc::{Signature, CallConv, ArgumentType, ArgumentExtension, ArgumentPurpose, ExtFuncData}; -pub use ir::types::Type; -pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable, FuncRef, SigRef}; -pub use ir::instructions::{Opcode, InstructionData, VariableArgs, ValueList, ValueListPool}; -pub use ir::stackslot::{StackSlots, StackSlotKind, StackSlotData}; -pub use ir::jumptable::JumpTableData; -pub use ir::valueloc::{ValueLoc, ArgumentLoc}; -pub use ir::dfg::{DataFlowGraph, ValueDef}; -pub use ir::layout::{Layout, CursorBase, Cursor}; +pub use ir::funcname::FunctionName; pub use ir::function::Function; -pub use ir::builder::InstBuilder; -pub use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; +pub use ir::instructions::{Opcode, InstructionData, VariableArgs, ValueList, ValueListPool}; +pub use ir::jumptable::JumpTableData; +pub use ir::layout::{Layout, CursorBase, Cursor}; pub use ir::memflags::MemFlags; -pub use ir::builder::InstBuilderBase; +pub use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; +pub use ir::stackslot::{StackSlots, StackSlotKind, StackSlotData}; +pub use ir::types::Type; +pub use ir::valueloc::{ValueLoc, ArgumentLoc}; use binemit; use entity_map::EntityMap; From 87cee86acbf8e6e496d67595e780911256774e01 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 3 Aug 2017 21:08:36 -0700 Subject: [PATCH 0959/3084] Cursor library. Add a new cursor module and define an EncCursor data type in it. An EncCursor is a cursor that inserts instructions with a valid encoding for the ISA. This is useful for passes generating code after legalization. Implement a builder interface via the new InstInserterBase trait such that the EncCursor builders support with_result(). Use EncCursor in coalescing.rs instead of the layout cursor as a proof of concept. --- lib/cretonne/src/cursor.rs | 105 ++++++++++++++++++++++++ lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/regalloc/coalescing.rs | 55 ++++--------- 3 files changed, 122 insertions(+), 39 deletions(-) create mode 100644 lib/cretonne/src/cursor.rs diff --git a/lib/cretonne/src/cursor.rs b/lib/cretonne/src/cursor.rs new file mode 100644 index 0000000000..b706531eed --- /dev/null +++ b/lib/cretonne/src/cursor.rs @@ -0,0 +1,105 @@ +//! Cursor library. +//! +//! This module defines cursor data types that can be used for inserting instructions. + +use ir; +use isa::TargetIsa; + +// Re-export these types, anticipating their being moved here. +pub use ir::layout::CursorBase as Cursor; +pub use ir::layout::CursorPosition; +pub use ir::layout::Cursor as LayoutCursor; + +/// Encoding cursor. +/// +/// An `EncCursor` can be used to insert instructions that are immediately assigned an encoding. +/// The cursor holds a mutable reference to the whole function which can be re-borrowed from the +/// public `pos.func` member. +pub struct EncCursor<'f> { + pos: CursorPosition, + built_inst: Option, + pub func: &'f mut ir::Function, + pub isa: &'f TargetIsa, +} + +impl<'f> EncCursor<'f> { + /// Create a new `EncCursor` pointing nowhere. + pub fn new(func: &'f mut ir::Function, isa: &'f TargetIsa) -> EncCursor<'f> { + EncCursor { + pos: CursorPosition::Nowhere, + built_inst: None, + func, + isa, + } + } + + /// Create an instruction builder that will insert an encoded instruction at the current + /// position. + /// + /// The builder will panic if it is used to insert an instruction that can't be encoded for + /// `self.isa`. + pub fn ins(&mut self) -> ir::InsertBuilder<&mut EncCursor<'f>> { + ir::InsertBuilder::new(self) + } + + /// Get the last built instruction. + /// + /// This returns the last instruction that was built using the `ins()` method on this cursor. + /// Panics if no instruction was built. + pub fn built_inst(&self) -> ir::Inst { + self.built_inst.expect("No instruction was inserted") + } + + /// Return an object that can display `inst`. + /// + /// This is a convenience wrapper for the DFG equivalent. + pub fn display_inst(&self, inst: ir::Inst) -> ir::dfg::DisplayInst { + self.func.dfg.display_inst(inst, self.isa) + } +} + +impl<'f> Cursor for EncCursor<'f> { + fn position(&self) -> CursorPosition { + self.pos + } + + fn set_position(&mut self, pos: CursorPosition) { + self.pos = pos + } + + fn layout(&self) -> &ir::Layout { + &self.func.layout + } + + fn layout_mut(&mut self) -> &mut ir::Layout { + &mut self.func.layout + } +} + +impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut EncCursor<'f> { + fn data_flow_graph(&self) -> &ir::DataFlowGraph { + &self.func.dfg + } + + fn data_flow_graph_mut(&mut self) -> &mut ir::DataFlowGraph { + &mut self.func.dfg + } + + fn insert_built_inst(self, + inst: ir::Inst, + ctrl_typevar: ir::Type) + -> &'c mut ir::DataFlowGraph { + // Insert the instruction and remember the reference. + self.insert_inst(inst); + self.built_inst = Some(inst); + + // Assign an encoding. + match self.isa + .encode(&self.func.dfg, &self.func.dfg[inst], ctrl_typevar) { + Ok(e) => *self.func.encodings.ensure(inst) = e, + Err(_) => panic!("can't encode {}", self.display_inst(inst)), + } + + &mut self.func.dfg + } +} diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 44f469ab4e..b021bf55b2 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -34,6 +34,7 @@ pub mod verifier; mod abi; mod constant_hash; mod context; +mod cursor; mod iterators; mod legalizer; mod licm; diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index b21175ad87..ac11df3498 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -5,10 +5,11 @@ //! and inserting copies where necessary such that all values passed to an EBB argument will belong //! to the same virtual register as the EBB argument value itself. +use cursor::{Cursor, EncCursor}; use dbg::DisplayList; use dominator_tree::DominatorTree; use flowgraph::{ControlFlowGraph, BasicBlock}; -use ir::{DataFlowGraph, Layout, Cursor, CursorBase, InstBuilder, ValueDef}; +use ir::{DataFlowGraph, Layout, InstBuilder, ValueDef}; use ir::{Function, Ebb, Inst, Value, ExpandedProgramPoint}; use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; @@ -464,37 +465,26 @@ impl<'a> Context<'a> { argnum: usize, pred_val: Value) -> Value { - let copy; - { - let mut pos = Cursor::new(&mut self.func.layout).at_inst(pred_inst); - copy = self.func.dfg.ins(&mut pos).copy(pred_val); - } - let inst = self.func.dfg.value_def(copy).unwrap_inst(); - let ty = self.func.dfg.value_type(copy); + let mut pos = EncCursor::new(self.func, self.isa).at_inst(pred_inst); + let copy = pos.ins().copy(pred_val); + let inst = pos.built_inst(); dbg!("Inserted {}, before {}: {}", - self.func.dfg.display_inst(inst, self.isa), + pos.display_inst(inst), pred_ebb, - self.func.dfg.display_inst(pred_inst, self.isa)); - - // Give it an encoding. - let encoding = match self.isa.encode(&self.func.dfg, &self.func.dfg[inst], ty) { - Ok(e) => e, - Err(_) => panic!("Can't encode copy.{}", ty), - }; - *self.func.encodings.ensure(inst) = encoding; + pos.display_inst(pred_inst)); // Create a live range for the new value. let affinity = Affinity::new(&self.encinfo - .operand_constraints(encoding) + .operand_constraints(pos.func.encodings[inst]) .expect("Bad copy encoding") .outs [0]); self.liveness.create_dead(copy, inst, affinity); self.liveness - .extend_locally(copy, pred_ebb, pred_inst, &self.func.layout); + .extend_locally(copy, pred_ebb, pred_inst, &pos.func.layout); - self.func.dfg.inst_variable_args_mut(pred_inst)[argnum] = copy; + pos.func.dfg.inst_variable_args_mut(pred_inst)[argnum] = copy; self.split_values.push(copy); copy } @@ -505,39 +495,26 @@ impl<'a> Context<'a> { let new_val = self.func.dfg.replace_ebb_arg(succ_val, ty); // Insert a copy instruction at the top of ebb. - { - let mut pos = Cursor::new(&mut self.func.layout).at_first_inst(ebb); - self.func - .dfg - .ins(&mut pos) - .with_result(succ_val) - .copy(new_val); - } - let inst = self.func.dfg.value_def(succ_val).unwrap_inst(); + let mut pos = EncCursor::new(self.func, self.isa).at_first_inst(ebb); + pos.ins().with_result(succ_val).copy(new_val); + let inst = pos.built_inst(); self.liveness.move_def_locally(succ_val, inst); dbg!("Inserted {}, following {}({}: {})", - self.func.dfg.display_inst(inst, self.isa), + pos.display_inst(inst), ebb, new_val, ty); - // Give it an encoding. - let encoding = match self.isa.encode(&self.func.dfg, &self.func.dfg[inst], ty) { - Ok(e) => e, - Err(_) => panic!("Can't encode copy.{}", ty), - }; - *self.func.encodings.ensure(inst) = encoding; - // Create a live range for the new value. let affinity = Affinity::new(&self.encinfo - .operand_constraints(encoding) + .operand_constraints(pos.func.encodings[inst]) .expect("Bad copy encoding") .outs [0]); self.liveness.create_dead(new_val, ebb, affinity); self.liveness - .extend_locally(new_val, ebb, inst, &self.func.layout); + .extend_locally(new_val, ebb, inst, &pos.func.layout); self.split_values.push(new_val); new_val From f15abe715401da187d9103ea05834a2d03fb0649 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Aug 2017 14:13:56 -0700 Subject: [PATCH 0960/3084] Use EncCursor in regalloc/spilling.rs Use an EncCursor instead of a layout cursor to keep track of the current position in the function. Since the EncCursor holds a reference to the whole IR function insteadof just the layout, we can rework how IR borrowing works. The Context data structure that's live during the spilling pass now owns an EncCursor which in turn holds references to the function and ISA. This means that we no longer need to pass around references to parts of the ir::Function. We can no longer borrow any part of the IR function across a context method call, but that turns out to be not necessary. --- lib/cretonne/src/regalloc/spilling.rs | 194 +++++++++++--------------- 1 file changed, 81 insertions(+), 113 deletions(-) diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index e1e610c3f0..12a938c81c 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -15,10 +15,9 @@ //! be compatible. Otherwise, the value must be copied into a new register for some of the //! operands. +use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; -use ir::{DataFlowGraph, Layout, Cursor, CursorBase, InstBuilder}; -use ir::{Function, Ebb, Inst, Value, ValueLoc, SigRef}; -use ir::{InstEncodings, StackSlots, ValueLocations}; +use ir::{InstBuilder, Function, Ebb, Inst, Value, ValueLoc, SigRef}; use isa::registers::{RegClassMask, RegClassIndex}; use isa::{TargetIsa, RegInfo, EncInfo, RecipeConstraints, ConstraintKind}; use regalloc::affinity::Affinity; @@ -37,16 +36,13 @@ pub struct Spilling { /// Context data structure that gets instantiated once per pass. struct Context<'a> { - isa: &'a TargetIsa, + // Current instruction as well as reference to function and ISA. + cur: EncCursor<'a>, + // Cached ISA information. reginfo: RegInfo, encinfo: EncInfo, - // References to parts of the current function. - encodings: &'a mut InstEncodings, - stack_slots: &'a mut StackSlots, - locations: &'a mut ValueLocations, - // References to contextual data structures we need. domtree: &'a DominatorTree, liveness: &'a mut Liveness, @@ -87,12 +83,9 @@ impl Spilling { let reginfo = isa.register_info(); let usable_regs = isa.allocatable_registers(func); let mut ctx = Context { - isa, + cur: EncCursor::new(func, isa), reginfo: isa.register_info(), encinfo: isa.encoding_info(), - encodings: &mut func.encodings, - stack_slots: &mut func.stack_slots, - locations: &mut func.locations, domtree, liveness, virtregs, @@ -101,35 +94,29 @@ impl Spilling { spills: &mut self.spills, reg_uses: &mut self.reg_uses, }; - ctx.run(&mut func.layout, &mut func.dfg, tracker) + ctx.run(tracker) } } impl<'a> Context<'a> { - fn run(&mut self, - layout: &mut Layout, - dfg: &mut DataFlowGraph, - tracker: &mut LiveValueTracker) { - self.topo.reset(layout.ebbs()); - while let Some(ebb) = self.topo.next(layout, self.domtree) { - self.visit_ebb(ebb, layout, dfg, tracker); + fn run(&mut self, tracker: &mut LiveValueTracker) { + self.topo.reset(self.cur.func.layout.ebbs()); + while let Some(ebb) = self.topo.next(&self.cur.func.layout, self.domtree) { + self.visit_ebb(ebb, tracker); } } - fn visit_ebb(&mut self, - ebb: Ebb, - layout: &mut Layout, - dfg: &mut DataFlowGraph, - tracker: &mut LiveValueTracker) { + fn visit_ebb(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) { dbg!("Spilling {}:", ebb); - self.visit_ebb_header(ebb, layout, dfg, tracker); + self.cur.goto_top(ebb); + self.visit_ebb_header(ebb, tracker); tracker.drop_dead_args(); - let mut pos = Cursor::new(layout); - pos.goto_top(ebb); - while let Some(inst) = pos.next_inst() { - if let Some(constraints) = self.encinfo.operand_constraints(self.encodings[inst]) { - self.visit_inst(inst, ebb, constraints, &mut pos, dfg, tracker); + while let Some(inst) = self.cur.next_inst() { + if let Some(constraints) = + self.encinfo + .operand_constraints(self.cur.func.encodings[inst]) { + self.visit_inst(inst, ebb, constraints, tracker); } else { let (_throughs, kills) = tracker.process_ghost(inst); self.free_regs(kills); @@ -162,12 +149,12 @@ impl<'a> Context<'a> { } } - fn visit_ebb_header(&mut self, - ebb: Ebb, - layout: &mut Layout, - dfg: &mut DataFlowGraph, - tracker: &mut LiveValueTracker) { - let (liveins, args) = tracker.ebb_top(ebb, dfg, self.liveness, layout, self.domtree); + fn visit_ebb_header(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) { + let (liveins, args) = tracker.ebb_top(ebb, + &self.cur.func.dfg, + self.liveness, + &self.cur.func.layout, + self.domtree); // Count the live-in registers. These should already fit in registers; they did at the // dominator. @@ -184,13 +171,13 @@ impl<'a> Context<'a> { rc, lv.value, liveins.len()); - match self.spill_candidate(mask, liveins, dfg, layout) { + match self.spill_candidate(mask, liveins) { Some(cand) => { dbg!("Spilling live-in {} to make room for {} EBB argument {}", cand, rc, lv.value); - self.spill_reg(cand, dfg); + self.spill_reg(cand); } None => { // We can't spill any of the live-in registers, so we have to spill an @@ -200,7 +187,7 @@ impl<'a> Context<'a> { // Since `spill_reg` will free a register, add the current one here. self.pressure.take(rc); - self.spill_reg(lv.value, dfg); + self.spill_reg(lv.value); break 'try_take; } } @@ -216,29 +203,27 @@ impl<'a> Context<'a> { inst: Inst, ebb: Ebb, constraints: &RecipeConstraints, - pos: &mut Cursor, - dfg: &mut DataFlowGraph, tracker: &mut LiveValueTracker) { - dbg!("Inst {}, {}", - dfg.display_inst(inst, self.isa), - self.pressure); + dbg!("Inst {}, {}", self.cur.display_inst(inst), self.pressure); + debug_assert_eq!(self.cur.current_inst(), Some(inst)); + debug_assert_eq!(self.cur.current_ebb(), Some(ebb)); // We may need to resolve register constraints if there are any noteworthy uses. assert!(self.reg_uses.is_empty()); - self.collect_reg_uses(inst, ebb, constraints, dfg, pos.layout); + self.collect_reg_uses(inst, ebb, constraints); // Calls usually have fixed register uses. - let call_sig = dfg.call_signature(inst); + let call_sig = self.cur.func.dfg.call_signature(inst); if let Some(sig) = call_sig { - self.collect_abi_reg_uses(inst, sig, dfg); + self.collect_abi_reg_uses(inst, sig); } if !self.reg_uses.is_empty() { - self.process_reg_uses(inst, pos, dfg, tracker); + self.process_reg_uses(inst, tracker); } // Update the live value tracker with this instruction. - let (throughs, kills, defs) = tracker.process_inst(inst, dfg, self.liveness); + let (throughs, kills, defs) = tracker.process_inst(inst, &self.cur.func.dfg, self.liveness); // Remove kills from the pressure tracker. self.free_regs(kills); @@ -249,7 +234,7 @@ impl<'a> Context<'a> { if call_sig.is_some() { for lv in throughs { if lv.affinity.is_reg() && !self.spills.contains(&lv.value) { - self.spill_reg(lv.value, dfg); + self.spill_reg(lv.value); } } } @@ -262,12 +247,12 @@ impl<'a> Context<'a> { // Add register def to pressure, spill if needed. while let Err(mask) = self.pressure.take_transient(op.regclass) { dbg!("Need {} reg from {} throughs", op.regclass, throughs.len()); - match self.spill_candidate(mask, throughs, dfg, pos.layout) { - Some(cand) => self.spill_reg(cand, dfg), + match self.spill_candidate(mask, throughs) { + Some(cand) => self.spill_reg(cand), None => { panic!("Ran out of {} registers for {}", op.regclass, - dfg.display_inst(inst, self.isa)) + self.cur.display_inst(inst)) } } } @@ -290,13 +275,8 @@ impl<'a> Context<'a> { // We are assuming here that if a value is used both by a fixed register operand and a register // class operand, they two are compatible. We are also assuming that two register class // operands are always compatible. - fn collect_reg_uses(&mut self, - inst: Inst, - ebb: Ebb, - constraints: &RecipeConstraints, - dfg: &DataFlowGraph, - layout: &Layout) { - let args = dfg.inst_args(inst); + fn collect_reg_uses(&mut self, inst: Inst, ebb: Ebb, constraints: &RecipeConstraints) { + let args = self.cur.func.dfg.inst_args(inst); for (idx, (op, &arg)) in constraints.ins.iter().zip(args).enumerate() { let mut reguse = RegUse::new(arg, idx, op.regclass.into()); let lr = &self.liveness[arg]; @@ -305,7 +285,7 @@ impl<'a> Context<'a> { ConstraintKind::FixedReg(_) => reguse.fixed = true, ConstraintKind::Tied(_) => { // A tied operand must kill the used value. - reguse.tied = !lr.killed_at(inst, ebb, layout); + reguse.tied = !lr.killed_at(inst, ebb, &self.cur.func.layout); } ConstraintKind::Reg => {} } @@ -322,11 +302,14 @@ impl<'a> Context<'a> { } // Collect register uses from the ABI input constraints. - fn collect_abi_reg_uses(&mut self, inst: Inst, sig: SigRef, dfg: &DataFlowGraph) { - let fixed_args = dfg[inst].opcode().constraints().fixed_value_arguments(); - let args = dfg.inst_variable_args(inst); + fn collect_abi_reg_uses(&mut self, inst: Inst, sig: SigRef) { + let fixed_args = self.cur.func.dfg[inst] + .opcode() + .constraints() + .fixed_value_arguments(); + let args = self.cur.func.dfg.inst_variable_args(inst); for (idx, (abi, &arg)) in - dfg.signatures[sig] + self.cur.func.dfg.signatures[sig] .argument_types .iter() .zip(args) @@ -335,7 +318,7 @@ impl<'a> Context<'a> { let (rci, spilled) = match self.liveness[arg].affinity { Affinity::Reg(rci) => (rci, false), Affinity::Stack => { - (self.isa.regclass_for_abi_type(abi.value_type).into(), true) + (self.cur.isa.regclass_for_abi_type(abi.value_type).into(), true) } Affinity::None => panic!("Missing affinity for {}", arg), }; @@ -353,11 +336,7 @@ impl<'a> Context<'a> { // Trigger spilling if any of the temporaries cause the register pressure to become too high. // // Leave `self.reg_uses` empty. - fn process_reg_uses(&mut self, - inst: Inst, - pos: &mut Cursor, - dfg: &mut DataFlowGraph, - tracker: &LiveValueTracker) { + fn process_reg_uses(&mut self, inst: Inst, tracker: &LiveValueTracker) { // We're looking for multiple uses of the same value, so start by sorting by value. The // secondary `opidx` key makes it possible to use an unstable sort once that is available // outside nightly Rust. @@ -381,8 +360,8 @@ impl<'a> Context<'a> { }; if need_copy { - let copy = self.insert_copy(ru.value, ru.rci, pos, dfg); - dfg.inst_args_mut(inst)[ru.opidx as usize] = copy; + let copy = self.insert_copy(ru.value, ru.rci); + self.cur.func.dfg.inst_args_mut(inst)[ru.opidx as usize] = copy; } // Even if we don't insert a copy, we may need to account for register pressure for the @@ -393,18 +372,18 @@ impl<'a> Context<'a> { dbg!("Copy of {} reg causes spill", rc); // Spill a live register that is *not* used by the current instruction. // Spilling a use wouldn't help. - let args = dfg.inst_args(inst); - match self.spill_candidate(mask, - tracker.live().iter().filter(|lv| { - !args.contains(&lv.value) - }), - dfg, - &pos.layout) { - Some(cand) => self.spill_reg(cand, dfg), + match { + let args = self.cur.func.dfg.inst_args(inst); + self.spill_candidate(mask, + tracker.live().iter().filter(|lv| { + !args.contains(&lv.value) + })) + } { + Some(cand) => self.spill_reg(cand), None => { panic!("Ran out of {} registers when inserting copy before {}", rc, - dfg.display_inst(inst, self.isa)) + self.cur.display_inst(inst)) } } } @@ -415,12 +394,7 @@ impl<'a> Context<'a> { } // Find a spill candidate from `candidates` whose top-level register class is in `mask`. - fn spill_candidate<'ii, II>(&self, - mask: RegClassMask, - candidates: II, - dfg: &DataFlowGraph, - layout: &Layout) - -> Option + fn spill_candidate<'ii, II>(&self, mask: RegClassMask, candidates: II) -> Option where II: IntoIterator { // Find the best viable spill candidate. @@ -448,7 +422,9 @@ impl<'a> Context<'a> { .min_by(|&a, &b| { // Find the minimum candidate according to the RPO of their defs. self.domtree - .rpo_cmp(dfg.value_def(a), dfg.value_def(b), layout) + .rpo_cmp(self.cur.func.dfg.value_def(a), + self.cur.func.dfg.value_def(b), + &self.cur.func.layout) }) } @@ -460,7 +436,7 @@ impl<'a> Context<'a> { /// /// Note that this does not update the cached affinity in the live value tracker. Call /// `process_spills` to do that. - fn spill_reg(&mut self, value: Value, dfg: &DataFlowGraph) { + fn spill_reg(&mut self, value: Value) { if let Affinity::Reg(rci) = self.liveness.spill(value) { let rc = self.reginfo.rc(rci); self.pressure.free(rc); @@ -471,10 +447,13 @@ impl<'a> Context<'a> { } // Assign a spill slot for the whole virtual register. - let ss = self.stack_slots.make_spill_slot(dfg.value_type(value)); + let ss = self.cur + .func + .stack_slots + .make_spill_slot(self.cur.func.dfg.value_type(value)); for &v in self.virtregs.congruence_class(&value) { self.liveness.spill(v); - *self.locations.ensure(v) = ValueLoc::Stack(ss); + *self.cur.func.locations.ensure(v) = ValueLoc::Stack(ss); } } @@ -492,32 +471,21 @@ impl<'a> Context<'a> { } } - /// Insert a `copy value` before `pos` and give it a live range extending to `pos`. + /// Insert a `copy value` before the current instruction and give it a live range extending to + /// the current instruction. /// /// Returns the new local value created. - fn insert_copy(&mut self, - value: Value, - rci: RegClassIndex, - pos: &mut Cursor, - dfg: &mut DataFlowGraph) - -> Value { - let copy = dfg.ins(pos).copy(value); - let inst = dfg.value_def(copy).unwrap_inst(); - let ty = dfg.value_type(copy); - - // Give it an encoding. - match self.isa.encode(dfg, &dfg[inst], ty) { - Ok(e) => *self.encodings.ensure(inst) = e, - Err(_) => panic!("Can't encode {}", dfg.display_inst(inst, self.isa)), - } + fn insert_copy(&mut self, value: Value, rci: RegClassIndex) -> Value { + let copy = self.cur.ins().copy(value); + let inst = self.cur.built_inst(); // Update live ranges. self.liveness.create_dead(copy, inst, Affinity::Reg(rci)); self.liveness .extend_locally(copy, - pos.layout.pp_ebb(inst), - pos.current_inst().expect("must be at an instruction"), - pos.layout); + self.cur.func.layout.pp_ebb(inst), + self.cur.current_inst().expect("must be at an instruction"), + &self.cur.func.layout); copy } From e904bff420c5577899e0678ce2d8ece60915f9ee Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Aug 2017 15:02:46 -0700 Subject: [PATCH 0961/3084] Use EncCursor for reload.rs. Same deal as for spilling. Place an EncCursor in the context and use that to reference into the IR function when necessary. --- lib/cretonne/src/ir/layout.rs | 9 +- lib/cretonne/src/regalloc/reload.rs | 222 ++++++++++++---------------- 2 files changed, 98 insertions(+), 133 deletions(-) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index b219a13bb2..3f31d34f23 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -709,8 +709,7 @@ pub trait CursorBase { fn at_first_inst(mut self, ebb: Ebb) -> Self where Self: Sized { - let inst = self.layout().ebbs[ebb].first_inst.expect("Empty EBB"); - self.goto_inst(inst); + self.goto_first_inst(ebb); self } @@ -740,6 +739,12 @@ pub trait CursorBase { self.set_position(CursorPosition::At(inst)); } + /// Go to the first instruction in `ebb`. + fn goto_first_inst(&mut self, ebb: Ebb) { + let inst = self.layout().ebbs[ebb].first_inst.expect("Empty EBB"); + self.set_position(CursorPosition::At(inst)); + } + /// Go to the top of `ebb` which must be inserted into the layout. /// At this position, instructions cannot be inserted, but `next_inst()` will move to the first /// instruction in `ebb`. diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 18e834179b..837dbd40d0 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -9,10 +9,10 @@ //! possible to minimize the number of `fill` instructions needed. This must not cause the register //! pressure limits to be exceeded. +use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; -use ir::{Ebb, Inst, Value, Function, Signature, DataFlowGraph, InstEncodings}; -use ir::layout::{Cursor, CursorBase, CursorPosition}; -use ir::{InstBuilder, Opcode, ArgumentType, ArgumentLoc}; +use ir::{Ebb, Inst, Value, Function}; +use ir::{InstBuilder, ArgumentType, ArgumentLoc}; use isa::RegClass; use isa::{TargetIsa, Encoding, EncInfo, RecipeConstraints, ConstraintKind}; use regalloc::affinity::Affinity; @@ -29,7 +29,8 @@ pub struct Reload { /// Context data structure that gets instantiated once per pass. struct Context<'a> { - isa: &'a TargetIsa, + cur: EncCursor<'a>, + // Cached ISA information. // We save it here to avoid frequent virtual function calls on the `TargetIsa` trait object. encinfo: EncInfo, @@ -62,7 +63,7 @@ impl Reload { tracker: &mut LiveValueTracker) { dbg!("Reload for:\n{}", func.display(isa)); let mut ctx = Context { - isa, + cur: EncCursor::new(func, isa), encinfo: isa.encoding_info(), domtree, liveness, @@ -70,7 +71,7 @@ impl Reload { candidates: &mut self.candidates, reloads: &mut self.reloads, }; - ctx.run(func, tracker) + ctx.run(tracker) } } @@ -98,82 +99,63 @@ impl SparseMapValue for ReloadedValue { } impl<'a> Context<'a> { - fn run(&mut self, func: &mut Function, tracker: &mut LiveValueTracker) { - self.topo.reset(func.layout.ebbs()); - while let Some(ebb) = self.topo.next(&func.layout, self.domtree) { - self.visit_ebb(ebb, func, tracker); + fn run(&mut self, tracker: &mut LiveValueTracker) { + self.topo.reset(self.cur.func.layout.ebbs()); + while let Some(ebb) = self.topo.next(&self.cur.func.layout, self.domtree) { + self.visit_ebb(ebb, tracker); } } - fn visit_ebb(&mut self, ebb: Ebb, func: &mut Function, tracker: &mut LiveValueTracker) { + fn visit_ebb(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) { dbg!("Reloading {}:", ebb); - let start_from = self.visit_ebb_header(ebb, func, tracker); + self.visit_ebb_header(ebb, tracker); tracker.drop_dead_args(); - let mut pos = Cursor::new(&mut func.layout); - pos.set_position(start_from); - while let Some(inst) = pos.current_inst() { - let encoding = func.encodings[inst]; + // visit_ebb_header() places us at the first interesting instruction in the EBB. + while let Some(inst) = self.cur.current_inst() { + let encoding = self.cur.func.encodings[inst]; if encoding.is_legal() { - self.visit_inst(ebb, - inst, - encoding, - &mut pos, - &mut func.dfg, - &mut func.encodings, - &func.signature, - tracker); + self.visit_inst(ebb, inst, encoding, tracker); tracker.drop_dead(inst); } else { - pos.next_inst(); + self.cur.next_inst(); } } } - /// Process the EBB parameters. Return the next instruction in the EBB to be processed - fn visit_ebb_header(&mut self, - ebb: Ebb, - func: &mut Function, - tracker: &mut LiveValueTracker) - -> CursorPosition { - let (liveins, args) = - tracker.ebb_top(ebb, &func.dfg, self.liveness, &func.layout, self.domtree); + /// Process the EBB parameters. Move to the next instruction in the EBB to be processed + fn visit_ebb_header(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) { + let (liveins, args) = tracker.ebb_top(ebb, + &self.cur.func.dfg, + self.liveness, + &self.cur.func.layout, + self.domtree); - if func.layout.entry_block() == Some(ebb) { + if self.cur.func.layout.entry_block() == Some(ebb) { assert_eq!(liveins.len(), 0); - self.visit_entry_args(ebb, func, args) + self.visit_entry_args(ebb, args); } else { - self.visit_ebb_args(ebb, func, args) + self.visit_ebb_args(ebb, args); } } /// Visit the arguments to the entry block. /// These values have ABI constraints from the function signature. - fn visit_entry_args(&mut self, - ebb: Ebb, - func: &mut Function, - args: &[LiveValue]) - -> CursorPosition { - assert_eq!(func.signature.argument_types.len(), args.len()); - let mut pos = Cursor::new(&mut func.layout); - pos.goto_top(ebb); - pos.next_inst(); + fn visit_entry_args(&mut self, ebb: Ebb, args: &[LiveValue]) { + assert_eq!(self.cur.func.signature.argument_types.len(), args.len()); + self.cur.goto_first_inst(ebb); - for (abi, arg) in func.signature.argument_types.iter().zip(args) { + for (arg_idx, arg) in args.iter().enumerate() { + let abi = self.cur.func.signature.argument_types[arg_idx]; match abi.location { ArgumentLoc::Reg(_) => { if arg.affinity.is_stack() { // An incoming register parameter was spilled. Replace the parameter value // with a temporary register value that is immediately spilled. - let reg = func.dfg.replace_ebb_arg(arg.value, abi.value_type); - let affinity = Affinity::abi(abi, self.isa); + let reg = self.cur.func.dfg.replace_ebb_arg(arg.value, abi.value_type); + let affinity = Affinity::abi(&abi, self.cur.isa); self.liveness.create_dead(reg, ebb, affinity); - self.insert_spill(ebb, - arg.value, - reg, - &mut pos, - &mut func.encodings, - &mut func.dfg); + self.insert_spill(ebb, arg.value, reg); } } ArgumentLoc::Stack(_) => { @@ -182,14 +164,10 @@ impl<'a> Context<'a> { ArgumentLoc::Unassigned => panic!("Unexpected ABI location"), } } - pos.position() } - fn visit_ebb_args(&self, ebb: Ebb, func: &mut Function, _args: &[LiveValue]) -> CursorPosition { - let mut pos = Cursor::new(&mut func.layout); - pos.goto_top(ebb); - pos.next_inst(); - pos.position() + fn visit_ebb_args(&mut self, ebb: Ebb, _args: &[LiveValue]) { + self.cur.goto_first_inst(ebb); } /// Process the instruction pointed to by `pos`, and advance the cursor to the next instruction @@ -198,10 +176,6 @@ impl<'a> Context<'a> { ebb: Ebb, inst: Inst, encoding: Encoding, - pos: &mut Cursor, - dfg: &mut DataFlowGraph, - encodings: &mut InstEncodings, - func_signature: &Signature, tracker: &mut LiveValueTracker) { // Get the operand constraints for `inst` that we are trying to satisfy. let constraints = self.encinfo @@ -210,7 +184,7 @@ impl<'a> Context<'a> { // Identify reload candidates. assert!(self.candidates.is_empty()); - self.find_candidates(inst, constraints, func_signature, dfg); + self.find_candidates(inst, constraints); // Insert fill instructions before `inst`. while let Some(cand) = self.candidates.pop() { @@ -218,12 +192,8 @@ impl<'a> Context<'a> { continue; } - let reg = dfg.ins(pos).fill(cand.value); - let fill = dfg.value_def(reg).unwrap_inst(); - match self.isa.encode(dfg, &dfg[fill], dfg.value_type(reg)) { - Ok(e) => *encodings.ensure(fill) = e, - Err(_) => panic!("Can't encode fill {}", cand.value), - } + let reg = self.cur.ins().fill(cand.value); + let fill = self.cur.built_inst(); self.reloads .insert(ReloadedValue { @@ -233,12 +203,13 @@ impl<'a> Context<'a> { // Create a live range for the new reload. let affinity = Affinity::Reg(cand.regclass.into()); - self.liveness.create_dead(reg, dfg.value_def(reg), affinity); - self.liveness.extend_locally(reg, ebb, inst, pos.layout); + self.liveness.create_dead(reg, fill, affinity); + self.liveness + .extend_locally(reg, ebb, inst, &self.cur.func.layout); } // Rewrite arguments. - for arg in dfg.inst_args_mut(inst) { + for arg in self.cur.func.dfg.inst_args_mut(inst) { if let Some(reload) = self.reloads.get(*arg) { *arg = reload.reg; } @@ -247,10 +218,11 @@ impl<'a> Context<'a> { // TODO: Reuse reloads for future instructions. self.reloads.clear(); - let (_throughs, _kills, defs) = tracker.process_inst(inst, dfg, self.liveness); + let (_throughs, _kills, defs) = tracker + .process_inst(inst, &self.cur.func.dfg, self.liveness); // Advance to the next instruction so we can insert any spills after the instruction. - pos.next_inst(); + self.cur.next_inst(); // Rewrite register defs that need to be spilled. // @@ -266,10 +238,10 @@ impl<'a> Context<'a> { // That way, we don't need to rewrite all future uses of v2. for (lv, op) in defs.iter().zip(constraints.outs) { if lv.affinity.is_stack() && op.kind != ConstraintKind::Stack { - let value_type = dfg.value_type(lv.value); - let reg = dfg.replace_result(lv.value, value_type); + let value_type = self.cur.func.dfg.value_type(lv.value); + let reg = self.cur.func.dfg.replace_result(lv.value, value_type); self.liveness.create_dead(reg, inst, Affinity::new(op)); - self.insert_spill(ebb, lv.value, reg, pos, encodings, dfg); + self.insert_spill(ebb, lv.value, reg); } } } @@ -277,12 +249,8 @@ impl<'a> Context<'a> { // Find reload candidates for `inst` and add them to `self.condidates`. // // These are uses of spilled values where the operand constraint requires a register. - fn find_candidates(&mut self, - inst: Inst, - constraints: &RecipeConstraints, - func_signature: &Signature, - dfg: &DataFlowGraph) { - let args = dfg.inst_args(inst); + fn find_candidates(&mut self, inst: Inst, constraints: &RecipeConstraints) { + let args = self.cur.func.dfg.inst_args(inst); for (op, &arg) in constraints.ins.iter().zip(args) { if op.kind != ConstraintKind::Stack { @@ -303,30 +271,18 @@ impl<'a> Context<'a> { let var_args = &args[constraints.ins.len()..]; // Handle ABI arguments. - if let Some(sig) = dfg.call_signature(inst) { - self.handle_abi_args(&dfg.signatures[sig].argument_types, var_args); - } else if dfg[inst].opcode().is_return() { - self.handle_abi_args(&func_signature.return_types, var_args); - } - } - - /// Find reload candidates in the instruction's ABI variable arguments. This handles both - /// return values and call arguments. - fn handle_abi_args(&mut self, abi_types: &[ArgumentType], var_args: &[Value]) { - assert_eq!(abi_types.len(), var_args.len()); - for (abi, &arg) in abi_types.iter().zip(var_args) { - if abi.location.is_reg() { - let lv = self.liveness - .get(arg) - .expect("Missing live range for ABI arg"); - if lv.affinity.is_stack() { - self.candidates - .push(ReloadCandidate { - value: arg, - regclass: self.isa.regclass_for_abi_type(abi.value_type), - }); - } - } + if let Some(sig) = self.cur.func.dfg.call_signature(inst) { + handle_abi_args(self.candidates, + &self.cur.func.dfg.signatures[sig].argument_types, + var_args, + self.cur.isa, + self.liveness); + } else if self.cur.func.dfg[inst].opcode().is_return() { + handle_abi_args(self.candidates, + &self.cur.func.signature.return_types, + var_args, + self.cur.isa, + self.liveness); } } @@ -335,30 +291,34 @@ impl<'a> Context<'a> { /// - Insert `stack = spill reg` at `pos`, and assign an encoding. /// - Move the `stack` live range starting point to the new instruction. /// - Extend the `reg` live range to reach the new instruction. - fn insert_spill(&mut self, - ebb: Ebb, - stack: Value, - reg: Value, - pos: &mut Cursor, - encodings: &mut InstEncodings, - dfg: &mut DataFlowGraph) { - let ty = dfg.value_type(reg); - - // Insert spill instruction. Use the low-level `Unary` constructor because it returns an - // instruction reference directly rather than a result value (which we know is equal to - // `stack`). - let (inst, _) = dfg.ins(pos) - .with_result(stack) - .Unary(Opcode::Spill, ty, reg); - - // Give it an encoding. - match self.isa.encode(dfg, &dfg[inst], ty) { - Ok(e) => *encodings.ensure(inst) = e, - Err(_) => panic!("Can't encode spill.{}", ty), - } + fn insert_spill(&mut self, ebb: Ebb, stack: Value, reg: Value) { + self.cur.ins().with_result(stack).spill(reg); + let inst = self.cur.built_inst(); // Update live ranges. self.liveness.move_def_locally(stack, inst); - self.liveness.extend_locally(reg, ebb, inst, pos.layout); + self.liveness + .extend_locally(reg, ebb, inst, &self.cur.func.layout); + } +} + +/// Find reload candidates in the instruction's ABI variable arguments. This handles both +/// return values and call arguments. +fn handle_abi_args(candidates: &mut Vec, + abi_types: &[ArgumentType], + var_args: &[Value], + isa: &TargetIsa, + liveness: &Liveness) { + assert_eq!(abi_types.len(), var_args.len()); + for (abi, &arg) in abi_types.iter().zip(var_args) { + if abi.location.is_reg() { + let lv = liveness.get(arg).expect("Missing live range for ABI arg"); + if lv.affinity.is_stack() { + candidates.push(ReloadCandidate { + value: arg, + regclass: isa.regclass_for_abi_type(abi.value_type), + }); + } + } } } From 92c3d8f87b13108e014deb6d56a01babc9056a00 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Aug 2017 15:31:08 -0700 Subject: [PATCH 0962/3084] Add a FuncCursor type to the cursor library. A FuncCursor works a like a layout cursor, but it holds a reference to the entire function and lets you re-borrow the function reference. Rewrite the dominator tree unit tests with a FuncCursor instead of a layout cursor to demonstrate the difference. It avoids the constrained lifetimes of the layout cursor in the tests. --- lib/cretonne/src/cursor.rs | 61 ++++++++ lib/cretonne/src/dominator_tree.rs | 228 +++++++++++++---------------- 2 files changed, 161 insertions(+), 128 deletions(-) diff --git a/lib/cretonne/src/cursor.rs b/lib/cretonne/src/cursor.rs index b706531eed..3719aa1d34 100644 --- a/lib/cretonne/src/cursor.rs +++ b/lib/cretonne/src/cursor.rs @@ -10,6 +10,67 @@ pub use ir::layout::CursorBase as Cursor; pub use ir::layout::CursorPosition; pub use ir::layout::Cursor as LayoutCursor; +/// Function cursor. +/// +/// A `FuncCursor` holds a mutable reference to a whole `ir::Function` while keeping a position +/// too. The function can be re-borrowed by accessing the public `cur.func` member. +/// +/// This cursor is for use before legalization. The inserted instructions are not given an +/// encoding. +pub struct FuncCursor<'f> { + pos: CursorPosition, + pub func: &'f mut ir::Function, +} + +impl<'f> FuncCursor<'f> { + /// Create a new `FuncCursor` pointing nowhere. + pub fn new(func: &'f mut ir::Function) -> FuncCursor<'f> { + FuncCursor { + pos: CursorPosition::Nowhere, + func, + } + } + + /// Create an instruction builder that inserts an instruction at the current position. + pub fn ins(&mut self) -> ir::InsertBuilder<&mut FuncCursor<'f>> { + ir::InsertBuilder::new(self) + } +} + +impl<'f> Cursor for FuncCursor<'f> { + fn position(&self) -> CursorPosition { + self.pos + } + + fn set_position(&mut self, pos: CursorPosition) { + self.pos = pos + } + + fn layout(&self) -> &ir::Layout { + &self.func.layout + } + + fn layout_mut(&mut self) -> &mut ir::Layout { + &mut self.func.layout + } +} + +impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut FuncCursor<'f> { + fn data_flow_graph(&self) -> &ir::DataFlowGraph { + &self.func.dfg + } + + fn data_flow_graph_mut(&mut self) -> &mut ir::DataFlowGraph { + &mut self.func.dfg + } + + fn insert_built_inst(self, inst: ir::Inst, _: ir::Type) -> &'c mut ir::DataFlowGraph { + self.insert_inst(inst); + &mut self.func.dfg + } +} + + /// Encoding cursor. /// /// An `EncCursor` can be used to insert instructions that are immediately assigned an encoding. diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 9d47cc4bd7..d88adbd4a0 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -387,8 +387,9 @@ impl DominatorTree { #[cfg(test)] mod test { + use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; - use ir::{Function, InstBuilder, Cursor, CursorBase, types}; + use ir::{Function, InstBuilder, types}; use super::*; use ir::types::*; use verifier::verify_context; @@ -411,45 +412,38 @@ mod test { let ebb2 = func.dfg.make_ebb(); let ebb0 = func.dfg.make_ebb(); - let jmp_ebb3_ebb1; - let br_ebb1_ebb0; - let jmp_ebb1_ebb2; + let mut cur = FuncCursor::new(&mut func); - { - let dfg = &mut func.dfg; - let cur = &mut Cursor::new(&mut func.layout); + cur.insert_ebb(ebb3); + let jmp_ebb3_ebb1 = cur.ins().jump(ebb1, &[]); - cur.insert_ebb(ebb3); - jmp_ebb3_ebb1 = dfg.ins(cur).jump(ebb1, &[]); + cur.insert_ebb(ebb1); + let br_ebb1_ebb0 = cur.ins().brnz(cond, ebb0, &[]); + let jmp_ebb1_ebb2 = cur.ins().jump(ebb2, &[]); - cur.insert_ebb(ebb1); - br_ebb1_ebb0 = dfg.ins(cur).brnz(cond, ebb0, &[]); - jmp_ebb1_ebb2 = dfg.ins(cur).jump(ebb2, &[]); + cur.insert_ebb(ebb2); + cur.ins().jump(ebb0, &[]); - cur.insert_ebb(ebb2); - dfg.ins(cur).jump(ebb0, &[]); + cur.insert_ebb(ebb0); - cur.insert_ebb(ebb0); - } + let cfg = ControlFlowGraph::with_function(cur.func); + let dt = DominatorTree::with_function(cur.func, &cfg); - let cfg = ControlFlowGraph::with_function(&func); - let dt = DominatorTree::with_function(&func, &cfg); - - assert_eq!(func.layout.entry_block().unwrap(), ebb3); + assert_eq!(cur.func.layout.entry_block().unwrap(), ebb3); assert_eq!(dt.idom(ebb3), None); assert_eq!(dt.idom(ebb1).unwrap(), jmp_ebb3_ebb1); assert_eq!(dt.idom(ebb2).unwrap(), jmp_ebb1_ebb2); assert_eq!(dt.idom(ebb0).unwrap(), br_ebb1_ebb0); - assert!(dt.dominates(br_ebb1_ebb0, br_ebb1_ebb0, &func.layout)); - assert!(!dt.dominates(br_ebb1_ebb0, jmp_ebb3_ebb1, &func.layout)); - assert!(dt.dominates(jmp_ebb3_ebb1, br_ebb1_ebb0, &func.layout)); + assert!(dt.dominates(br_ebb1_ebb0, br_ebb1_ebb0, &cur.func.layout)); + assert!(!dt.dominates(br_ebb1_ebb0, jmp_ebb3_ebb1, &cur.func.layout)); + assert!(dt.dominates(jmp_ebb3_ebb1, br_ebb1_ebb0, &cur.func.layout)); - assert_eq!(dt.rpo_cmp(ebb3, ebb3, &func.layout), Ordering::Equal); - assert_eq!(dt.rpo_cmp(ebb3, ebb1, &func.layout), Ordering::Less); - assert_eq!(dt.rpo_cmp(ebb3, jmp_ebb3_ebb1, &func.layout), + assert_eq!(dt.rpo_cmp(ebb3, ebb3, &cur.func.layout), Ordering::Equal); + assert_eq!(dt.rpo_cmp(ebb3, ebb1, &cur.func.layout), Ordering::Less); + assert_eq!(dt.rpo_cmp(ebb3, jmp_ebb3_ebb1, &cur.func.layout), Ordering::Less); - assert_eq!(dt.rpo_cmp(jmp_ebb3_ebb1, jmp_ebb1_ebb2, &func.layout), + assert_eq!(dt.rpo_cmp(jmp_ebb3_ebb1, jmp_ebb1_ebb2, &cur.func.layout), Ordering::Less); assert_eq!(dt.cfg_postorder(), &[ebb2, ebb0, ebb1, ebb3]); @@ -462,72 +456,66 @@ mod test { let ebb1 = func.dfg.make_ebb(); let ebb2 = func.dfg.make_ebb(); - let jmp02; - let jmp21; - let trap; - { - let dfg = &mut func.dfg; - let cur = &mut Cursor::new(&mut func.layout); + let mut cur = FuncCursor::new(&mut func); - cur.insert_ebb(ebb0); - jmp02 = dfg.ins(cur).jump(ebb2, &[]); + cur.insert_ebb(ebb0); + let jmp02 = cur.ins().jump(ebb2, &[]); - cur.insert_ebb(ebb1); - trap = dfg.ins(cur).trap(); + cur.insert_ebb(ebb1); + let trap = cur.ins().trap(); - cur.insert_ebb(ebb2); - jmp21 = dfg.ins(cur).jump(ebb1, &[]); - } + cur.insert_ebb(ebb2); + let jmp21 = cur.ins().jump(ebb1, &[]); - let cfg = ControlFlowGraph::with_function(&func); - let dt = DominatorTree::with_function(&func, &cfg); + let cfg = ControlFlowGraph::with_function(cur.func); + let dt = DominatorTree::with_function(cur.func, &cfg); - assert_eq!(func.layout.entry_block(), Some(ebb0)); + assert_eq!(cur.func.layout.entry_block(), Some(ebb0)); assert_eq!(dt.idom(ebb0), None); assert_eq!(dt.idom(ebb1), Some(jmp21)); assert_eq!(dt.idom(ebb2), Some(jmp02)); - assert!(dt.dominates(ebb0, ebb0, &func.layout)); - assert!(dt.dominates(ebb0, jmp02, &func.layout)); - assert!(dt.dominates(ebb0, ebb1, &func.layout)); - assert!(dt.dominates(ebb0, trap, &func.layout)); - assert!(dt.dominates(ebb0, ebb2, &func.layout)); - assert!(dt.dominates(ebb0, jmp21, &func.layout)); + assert!(dt.dominates(ebb0, ebb0, &cur.func.layout)); + assert!(dt.dominates(ebb0, jmp02, &cur.func.layout)); + assert!(dt.dominates(ebb0, ebb1, &cur.func.layout)); + assert!(dt.dominates(ebb0, trap, &cur.func.layout)); + assert!(dt.dominates(ebb0, ebb2, &cur.func.layout)); + assert!(dt.dominates(ebb0, jmp21, &cur.func.layout)); - assert!(!dt.dominates(jmp02, ebb0, &func.layout)); - assert!(dt.dominates(jmp02, jmp02, &func.layout)); - assert!(dt.dominates(jmp02, ebb1, &func.layout)); - assert!(dt.dominates(jmp02, trap, &func.layout)); - assert!(dt.dominates(jmp02, ebb2, &func.layout)); - assert!(dt.dominates(jmp02, jmp21, &func.layout)); + assert!(!dt.dominates(jmp02, ebb0, &cur.func.layout)); + assert!(dt.dominates(jmp02, jmp02, &cur.func.layout)); + assert!(dt.dominates(jmp02, ebb1, &cur.func.layout)); + assert!(dt.dominates(jmp02, trap, &cur.func.layout)); + assert!(dt.dominates(jmp02, ebb2, &cur.func.layout)); + assert!(dt.dominates(jmp02, jmp21, &cur.func.layout)); - assert!(!dt.dominates(ebb1, ebb0, &func.layout)); - assert!(!dt.dominates(ebb1, jmp02, &func.layout)); - assert!(dt.dominates(ebb1, ebb1, &func.layout)); - assert!(dt.dominates(ebb1, trap, &func.layout)); - assert!(!dt.dominates(ebb1, ebb2, &func.layout)); - assert!(!dt.dominates(ebb1, jmp21, &func.layout)); + assert!(!dt.dominates(ebb1, ebb0, &cur.func.layout)); + assert!(!dt.dominates(ebb1, jmp02, &cur.func.layout)); + assert!(dt.dominates(ebb1, ebb1, &cur.func.layout)); + assert!(dt.dominates(ebb1, trap, &cur.func.layout)); + assert!(!dt.dominates(ebb1, ebb2, &cur.func.layout)); + assert!(!dt.dominates(ebb1, jmp21, &cur.func.layout)); - assert!(!dt.dominates(trap, ebb0, &func.layout)); - assert!(!dt.dominates(trap, jmp02, &func.layout)); - assert!(!dt.dominates(trap, ebb1, &func.layout)); - assert!(dt.dominates(trap, trap, &func.layout)); - assert!(!dt.dominates(trap, ebb2, &func.layout)); - assert!(!dt.dominates(trap, jmp21, &func.layout)); + assert!(!dt.dominates(trap, ebb0, &cur.func.layout)); + assert!(!dt.dominates(trap, jmp02, &cur.func.layout)); + assert!(!dt.dominates(trap, ebb1, &cur.func.layout)); + assert!(dt.dominates(trap, trap, &cur.func.layout)); + assert!(!dt.dominates(trap, ebb2, &cur.func.layout)); + assert!(!dt.dominates(trap, jmp21, &cur.func.layout)); - assert!(!dt.dominates(ebb2, ebb0, &func.layout)); - assert!(!dt.dominates(ebb2, jmp02, &func.layout)); - assert!(dt.dominates(ebb2, ebb1, &func.layout)); - assert!(dt.dominates(ebb2, trap, &func.layout)); - assert!(dt.dominates(ebb2, ebb2, &func.layout)); - assert!(dt.dominates(ebb2, jmp21, &func.layout)); + assert!(!dt.dominates(ebb2, ebb0, &cur.func.layout)); + assert!(!dt.dominates(ebb2, jmp02, &cur.func.layout)); + assert!(dt.dominates(ebb2, ebb1, &cur.func.layout)); + assert!(dt.dominates(ebb2, trap, &cur.func.layout)); + assert!(dt.dominates(ebb2, ebb2, &cur.func.layout)); + assert!(dt.dominates(ebb2, jmp21, &cur.func.layout)); - assert!(!dt.dominates(jmp21, ebb0, &func.layout)); - assert!(!dt.dominates(jmp21, jmp02, &func.layout)); - assert!(dt.dominates(jmp21, ebb1, &func.layout)); - assert!(dt.dominates(jmp21, trap, &func.layout)); - assert!(!dt.dominates(jmp21, ebb2, &func.layout)); - assert!(dt.dominates(jmp21, jmp21, &func.layout)); + assert!(!dt.dominates(jmp21, ebb0, &cur.func.layout)); + assert!(!dt.dominates(jmp21, jmp02, &cur.func.layout)); + assert!(dt.dominates(jmp21, ebb1, &cur.func.layout)); + assert!(dt.dominates(jmp21, trap, &cur.func.layout)); + assert!(!dt.dominates(jmp21, ebb2, &cur.func.layout)); + assert!(dt.dominates(jmp21, jmp21, &cur.func.layout)); } #[test] @@ -536,63 +524,47 @@ mod test { let ebb0 = func.dfg.make_ebb(); let ebb100 = func.dfg.make_ebb(); - let inst2; - let inst3; - let inst4; - let inst5; - { - let dfg = &mut func.dfg; - let cur = &mut Cursor::new(&mut func.layout); + let mut cur = FuncCursor::new(&mut func); - cur.insert_ebb(ebb0); - let cond = dfg.ins(cur).iconst(I32, 0); - inst2 = dfg.ins(cur).brz(cond, ebb0, &[]); - inst3 = dfg.ins(cur).brz(cond, ebb0, &[]); - inst4 = dfg.ins(cur).brz(cond, ebb0, &[]); - inst5 = dfg.ins(cur).brz(cond, ebb0, &[]); - dfg.ins(cur).jump(ebb100, &[]); - cur.insert_ebb(ebb100); - dfg.ins(cur).return_(&[]); - } - let mut cfg = ControlFlowGraph::with_function(&func); - let mut dt = DominatorTree::with_function(&func, &cfg); + cur.insert_ebb(ebb0); + let cond = cur.ins().iconst(I32, 0); + let inst2 = cur.ins().brz(cond, ebb0, &[]); + let inst3 = cur.ins().brz(cond, ebb0, &[]); + let inst4 = cur.ins().brz(cond, ebb0, &[]); + let inst5 = cur.ins().brz(cond, ebb0, &[]); + cur.ins().jump(ebb100, &[]); + cur.insert_ebb(ebb100); + cur.ins().return_(&[]); + + let mut cfg = ControlFlowGraph::with_function(cur.func); + let mut dt = DominatorTree::with_function(cur.func, &cfg); + + let ebb1 = cur.func.dfg.make_ebb(); + cur.func.layout.split_ebb(ebb1, inst2); + cur.goto_bottom(ebb0); + let middle_jump_inst = cur.ins().jump(ebb1, &[]); - let ebb1 = func.dfg.make_ebb(); - func.layout.split_ebb(ebb1, inst2); - let middle_jump_inst = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); - func.dfg.ins(cur).jump(ebb1, &[]) - }; dt.recompute_split_ebb(ebb0, ebb1, middle_jump_inst); - let ebb2 = func.dfg.make_ebb(); - func.layout.split_ebb(ebb2, inst3); - let middle_jump_inst = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb1); - func.dfg.ins(cur).jump(ebb2, &[]) - }; + let ebb2 = cur.func.dfg.make_ebb(); + cur.func.layout.split_ebb(ebb2, inst3); + cur.goto_bottom(ebb1); + let middle_jump_inst = cur.ins().jump(ebb2, &[]); dt.recompute_split_ebb(ebb1, ebb2, middle_jump_inst); - let ebb3 = func.dfg.make_ebb(); - func.layout.split_ebb(ebb3, inst4); - let middle_jump_inst = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb2); - func.dfg.ins(cur).jump(ebb3, &[]) - }; + let ebb3 = cur.func.dfg.make_ebb(); + cur.func.layout.split_ebb(ebb3, inst4); + cur.goto_bottom(ebb2); + let middle_jump_inst = cur.ins().jump(ebb3, &[]); dt.recompute_split_ebb(ebb2, ebb3, middle_jump_inst); - let ebb4 = func.dfg.make_ebb(); - func.layout.split_ebb(ebb4, inst5); - let middle_jump_inst = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb3); - func.dfg.ins(cur).jump(ebb4, &[]) - }; + let ebb4 = cur.func.dfg.make_ebb(); + cur.func.layout.split_ebb(ebb4, inst5); + cur.goto_bottom(ebb3); + let middle_jump_inst = cur.ins().jump(ebb4, &[]); dt.recompute_split_ebb(ebb3, ebb4, middle_jump_inst); - cfg.compute(&func); - verify_context(&func, &cfg, &dt, None).unwrap(); + + cfg.compute(cur.func); + verify_context(cur.func, &cfg, &dt, None).unwrap(); } } From ffae39377c83f2c8bd26cdcbabce6ec75c9d6ac6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 4 Aug 2017 16:00:48 -0700 Subject: [PATCH 0963/3084] Switch branch relaxation to a FuncCursor. --- lib/cretonne/src/binemit/relaxation.rs | 44 +++++++++++--------------- 1 file changed, 19 insertions(+), 25 deletions(-) diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index 0d0c07df35..e49b534624 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -28,7 +28,8 @@ //! ``` use binemit::CodeOffset; -use ir::{Function, DataFlowGraph, Cursor, CursorBase, InstructionData, Opcode, InstEncodings}; +use cursor::{Cursor, FuncCursor}; +use ir::{Function, InstructionData, Opcode}; use isa::{TargetIsa, EncInfo}; use iterators::IteratorExtras; use result::CtonError; @@ -55,34 +56,29 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> Result CodeOffset { - let inst = pos.current_inst().unwrap(); + let inst = cur.current_inst().unwrap(); dbg!("Relaxing [{}] {} for {:#x}-{:#x} range", - encinfo.display(encodings[inst]), - dfg.display_inst(inst, None), + encinfo.display(cur.func.encodings[inst]), + cur.func.dfg.display_inst(inst, None), offset, dest_offset); unimplemented!(); From 8546cabc8fa9c900695e870547b6c565e4557c8c Mon Sep 17 00:00:00 2001 From: Denis Merigoux Date: Tue, 8 Aug 2017 13:43:20 -0700 Subject: [PATCH 0964/3084] Verifier now checks it branch target ebbs are inserted in the layout --- lib/cretonne/src/verifier/mod.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index d3f05ef4c8..be95456212 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -297,7 +297,7 @@ impl<'a> Verifier<'a> { } fn verify_ebb(&self, inst: Inst, e: Ebb) -> Result { - if !self.func.dfg.ebb_is_valid(e) { + if !self.func.dfg.ebb_is_valid(e) || !self.func.layout.is_ebb_inserted(e) { err!(inst, "invalid ebb reference {}", e) } else { Ok(()) From 79d8aa366dcd61f0914363f7fed8b5cbe0da4ecf Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 8 Aug 2017 14:14:59 -0700 Subject: [PATCH 0965/3084] Don't perform gvn on instructions with other_side_effects(). --- cranelift/filetests/simple_gvn/reject.cton | 11 +++++++++++ lib/cretonne/src/simple_gvn.rs | 2 +- 2 files changed, 12 insertions(+), 1 deletion(-) create mode 100644 cranelift/filetests/simple_gvn/reject.cton diff --git a/cranelift/filetests/simple_gvn/reject.cton b/cranelift/filetests/simple_gvn/reject.cton new file mode 100644 index 0000000000..755891ad5d --- /dev/null +++ b/cranelift/filetests/simple_gvn/reject.cton @@ -0,0 +1,11 @@ +test simple-gvn + +function %other_side_effects(i32) -> i32 { +ebb0(v0: i32): + regmove v0, %10 -> %20 + regmove v0, %10 -> %20 + regmove v0, %20 -> %10 +; check: regmove v0, %10 -> %20 +; check: regmove v0, %10 -> %20 + return v0 +} diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index 078bca2857..2d1c5a7e61 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -8,7 +8,7 @@ use std::collections::HashMap; /// Test whether the given opcode is unsafe to even consider for GVN. fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || opcode.is_return() || - opcode.can_trap() + opcode.can_trap() || opcode.other_side_effects() } /// Perform simple GVN on `func`. From 591f6c1632b1a38de127a56b981bc9007ffce54f Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Thu, 30 Mar 2017 23:38:31 +0100 Subject: [PATCH 0966/3084] Added tests for verifier type checking. Closes PR #71. --- cranelift/filetests/verifier/type_check.cton | 97 ++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 cranelift/filetests/verifier/type_check.cton diff --git a/cranelift/filetests/verifier/type_check.cton b/cranelift/filetests/verifier/type_check.cton new file mode 100644 index 0000000000..b28f52930f --- /dev/null +++ b/cranelift/filetests/verifier/type_check.cton @@ -0,0 +1,97 @@ +test verifier + +function %entry_block_signature_mismatch(i32) { + ebb0: ; error: entry block arguments must match function signature + return +} + +function %entry_block_arg_type(i32) { + ebb0(v0: f32): ; error: entry block argument 0 expected to have type i32, got f32 + return +} + +function %incorrect_arg_type(i32, b1) -> i32 { + ebb0(v0: i32, v1: b1): + v2 = iadd v0, v1 ; error: arg 1 (v1) has type b1, expected i32 + return v2 +} + +function %incorrect_return_type() -> f32 { + ebb0: + v0 = iconst.i32 1 + return v0 ; error: arg 0 (v0) has type i32, must match function signature of f32 +} + +function %too_many_return_values() { + ebb0: + v0 = iconst.i32 1 + return v0 ; error: arguments of return must match function signature +} + +function %too_few_return_values() -> f32, i64 { + ebb0: + return ; error: arguments of return must match function signature +} + +function %type_mismatch_controlling_variable() { + ebb0: + v0 = iconst.i32 5 + v1 = iconst.i64 6 + v2 = iadd v0, v1 ; error: arg 1 (v1) has type i64, expected i32 + return +} + +function %fn_call_too_few_args() { + fn2 = function %great_fn(i32, f32) + ebb0: + call fn2() ; error: mismatched argument count, got 0, expected 2 + return +} + +function %fn_call_too_many_args() { + fn5 = function %best_fn() + ebb0: + v0 = iconst.i64 56 + v1 = f32const 0.0 + call fn5(v0, v1) ; error: mismatched argument count, got 2, expected 0 + return +} + +function %fn_call_incorrect_arg_type(i64) { + sig9 = (f32) + ebb0(v0: i64): + v1 = iconst.i32 56 + call_indirect sig9, v0(v1) ; error: arg 0 (v1) has type i32, expected f32 + return +} + +; TODO: Should we instead just verify that jump tables contain no EBBs that take arguments? This +; error doesn't occur if no instruction uses the jump table. +function %jump_table_args() { + jt1 = jump_table ebb1 + ebb0: + v0 = iconst.i32 0 + br_table v0, jt1 ; error: takes no arguments, but had target ebb1 with 1 arguments + return + ebb1(v5: i32): + return +} + +function %jump_args() { + ebb0: + v0 = iconst.i16 10 + v3 = iconst.i64 20 + jump ebb1(v0, v3) ; error: arg 0 (v0) has type i16, expected i64 + ebb1(v10: i64, v11: i16): + return +} + +function %jump_args2() { + ebb0: + v0 = iconst.i16 10 + v3 = iconst.i64 20 + brz v0, ebb1(v0, v3) ; error: arg 0 (v0) has type i16, expected i64 + jump ebb1(v3, v0) + ebb1(v10: i64, v11: i16): + return +} From 66da171050abeea351a394035fa3451b84b0c772 Mon Sep 17 00:00:00 2001 From: d1m0 Date: Mon, 14 Aug 2017 20:19:47 -0700 Subject: [PATCH 0967/3084] Fix for #141 (#142) * Add Atom and Literal base classes to CDSL Ast. Change substitution() and copy() on Def/Apply/Rtl to support substituting Var->Union[Var, Literal]. Check in Apply() constructor kinds of passed in Literals respect instruction signature * Change verify_semantics to check all possible instantiations of enumerated immediates (needed to descrive icmp). Add all bitvector comparison primitives and bvite; Change set_semantics to optionally accept XForms; Add semantics for icmp; Fix typing errors in semantics/{smtlib, elaborate, __init__}.py after the change of VarMap->VarAtomMap * Forgot macros.py * Nit obscured by testing with mypy enabled present. * Typo --- lib/cretonne/meta/base/semantics.py | 67 +++++++--- lib/cretonne/meta/cdsl/ast.py | 120 +++++++++++------- lib/cretonne/meta/cdsl/instructions.py | 11 +- lib/cretonne/meta/cdsl/operands.py | 15 ++- lib/cretonne/meta/cdsl/test_xform.py | 43 ++++++- lib/cretonne/meta/cdsl/xform.py | 30 +++-- lib/cretonne/meta/gen_legalizer.py | 8 +- lib/cretonne/meta/semantics/__init__.py | 80 ++++++++---- lib/cretonne/meta/semantics/elaborate.py | 12 +- lib/cretonne/meta/semantics/macros.py | 45 +++++++ lib/cretonne/meta/semantics/primitives.py | 58 ++++++++- lib/cretonne/meta/semantics/smtlib.py | 50 +++++--- lib/cretonne/meta/semantics/test_elaborate.py | 13 +- 13 files changed, 415 insertions(+), 137 deletions(-) create mode 100644 lib/cretonne/meta/semantics/macros.py diff --git a/lib/cretonne/meta/base/semantics.py b/lib/cretonne/meta/base/semantics.py index edf4c5f82e..ec1852133e 100644 --- a/lib/cretonne/meta/base/semantics.py +++ b/lib/cretonne/meta/base/semantics.py @@ -1,20 +1,33 @@ from __future__ import absolute_import from semantics.primitives import prim_to_bv, prim_from_bv, bvsplit, bvconcat,\ - bvadd, bvult, bvzeroext, bvsignext + bvadd, bvzeroext, bvsignext +from semantics.primitives import bveq, bvne, bvsge, bvsgt, bvsle, bvslt,\ + bvuge, bvugt, bvule, bvult +from semantics.macros import bool2bv from .instructions import vsplit, vconcat, iadd, iadd_cout, icmp, bextend, \ isplit, iconcat, iadd_cin, iadd_carry from .immediates import intcc -from cdsl.xform import Rtl +from cdsl.xform import Rtl, XForm from cdsl.ast import Var from cdsl.typevar import TypeSet from cdsl.ti import InTypeset +try: + from typing import TYPE_CHECKING # noqa + if TYPE_CHECKING: + from cdsl.ast import Enumerator # noqa + from cdsl.instructions import Instruction # noqa +except ImportError: + TYPE_CHECKING = False + x = Var('x') y = Var('y') a = Var('a') b = Var('b') c_out = Var('c_out') c_in = Var('c_in') +CC = Var('CC') +bc_out = Var('bc_out') bvc_out = Var('bvc_out') bvc_in = Var('bvc_in') xhi = Var('xhi') @@ -93,7 +106,8 @@ iadd_cout.set_semantics( bvx << prim_to_bv(x), bvy << prim_to_bv(y), bva << bvadd(bvx, bvy), - bvc_out << bvult(bva, bvx), + bc_out << bvult(bva, bvx), + bvc_out << bool2bv(bc_out), a << prim_from_bv(bva), c_out << prim_from_bv(bvc_out) )) @@ -107,7 +121,8 @@ iadd_carry.set_semantics( bvs << bvzeroext(bvc_in), bvt << bvadd(bvx, bvy), bva << bvadd(bvt, bvs), - bvc_out << bvult(bva, bvx), + bc_out << bvult(bva, bvx), + bvc_out << bool2bv(bc_out), a << prim_from_bv(bva), c_out << prim_from_bv(bvc_out) )) @@ -126,23 +141,45 @@ bextend.set_semantics( a << vconcat(alo, ahi) )) + +def create_comp_xform(cc, bvcmp_func): + # type: (Enumerator, Instruction) -> XForm + ba = Var('ba') + return XForm( + Rtl( + a << icmp(cc, x, y) + ), + Rtl( + bvx << prim_to_bv(x), + bvy << prim_to_bv(y), + ba << bvcmp_func(bvx, bvy), + bva << bool2bv(ba), + bva_wide << bvzeroext(bva), + a << prim_from_bv(bva_wide), + ), + constraints=InTypeset(x.get_typevar(), ScalarTS)) + + icmp.set_semantics( - a << icmp(intcc.ult, x, y), - (Rtl( - bvx << prim_to_bv(x), - bvy << prim_to_bv(y), - bva << bvult(bvx, bvy), - bva_wide << bvzeroext(bva), - a << prim_from_bv(bva_wide), - ), [InTypeset(x.get_typevar(), ScalarTS)]), + a << icmp(CC, x, y), Rtl( (xlo, xhi) << vsplit(x), (ylo, yhi) << vsplit(y), - alo << icmp(intcc.ult, xlo, ylo), - ahi << icmp(intcc.ult, xhi, yhi), + alo << icmp(CC, xlo, ylo), + ahi << icmp(CC, xhi, yhi), b << vconcat(alo, ahi), a << bextend(b) - )) + ), + create_comp_xform(intcc.eq, bveq), + create_comp_xform(intcc.ne, bvne), + create_comp_xform(intcc.sge, bvsge), + create_comp_xform(intcc.sgt, bvsgt), + create_comp_xform(intcc.sle, bvsle), + create_comp_xform(intcc.slt, bvslt), + create_comp_xform(intcc.uge, bvuge), + create_comp_xform(intcc.ugt, bvugt), + create_comp_xform(intcc.ule, bvule), + create_comp_xform(intcc.ult, bvult)) # # Legalization helper instructions. diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index aa53e7b9b5..251fee3641 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -11,23 +11,23 @@ from .predicates import IsEqual, And, TypePredicate try: from typing import Union, Tuple, Sequence, TYPE_CHECKING, Dict, List # noqa - from typing import Optional, Set # noqa + from typing import Optional, Set, Any # noqa if TYPE_CHECKING: from .operands import ImmediateKind # noqa from .predicates import PredNode # noqa - VarMap = Dict["Var", "Var"] + VarAtomMap = Dict["Var", "Atom"] except ImportError: pass def replace_var(arg, m): - # type: (Expr, VarMap) -> Expr + # type: (Expr, VarAtomMap) -> Expr """ Given a var v return either m[v] or a new variable v' (and remember m[v]=v'). Otherwise return the argument unchanged """ if isinstance(arg, Var): - new_arg = m.get(arg, Var(arg.name)) # type: Var + new_arg = m.get(arg, Var(arg.name)) # type: Atom m[arg] = new_arg return new_arg return arg @@ -76,7 +76,7 @@ class Def(object): ', '.join(map(str, self.defs)), self.expr) def copy(self, m): - # type: (VarMap) -> Def + # type: (VarAtomMap) -> Def """ Return a copy of this Def with vars replaced with fresh variables, in accordance with the map m. Update m as neccessary. @@ -106,7 +106,7 @@ class Def(object): return self.definitions().union(self.uses()) def substitution(self, other, s): - # type: (Def, VarMap) -> Optional[VarMap] + # type: (Def, VarAtomMap) -> Optional[VarAtomMap] """ If the Defs self and other agree structurally, return a variable substitution to transform self to other. Otherwise return None. Two @@ -133,7 +133,13 @@ class Expr(object): """ -class Var(Expr): +class Atom(Expr): + """ + An Atom in the DSL is either a literal or a Var + """ + + +class Var(Atom): """ A free variable. @@ -304,6 +310,16 @@ class Apply(Expr): self.args = args assert len(self.inst.ins) == len(args) + # Check that the kinds of Literals arguments match the expected Operand + for op_idx in self.inst.imm_opnums: + arg = self.args[op_idx] + op = self.inst.ins[op_idx] + + if isinstance(arg, Literal): + assert arg.kind == op.kind, \ + "Passing literal {} to field of wrong kind {}."\ + .format(arg, op.kind) + def __rlshift__(self, other): # type: (Union[Var, Tuple[Var, ...]]) -> Def """ @@ -377,7 +393,7 @@ class Apply(Expr): return pred def copy(self, m): - # type: (VarMap) -> Apply + # type: (VarAtomMap) -> Apply """ Return a copy of this Expr with vars replaced with fresh variables, in accordance with the map m. Update m as neccessary. @@ -396,15 +412,12 @@ class Apply(Expr): return res def substitution(self, other, s): - # type: (Apply, VarMap) -> Optional[VarMap] + # type: (Apply, VarAtomMap) -> Optional[VarAtomMap] """ - If the application self and other agree structurally, return a variable - substitution to transform self to other. Otherwise return None. Two - applications agree structurally if: - 1) They are over the same instruction - 2) Every Var v in self, maps to a single Var w in other. I.e for - each use of v in self, w is used in the corresponding place in - other. + If there is a substituion from Var->Atom that converts self to other, + return it, otherwise return None. Note that this is strictly weaker + than unification (see TestXForm.test_subst_enum_bad_var_const for + example). """ if self.inst != other.inst: return None @@ -413,37 +426,62 @@ class Apply(Expr): assert (len(self.args) == len(other.args)) for (self_a, other_a) in zip(self.args, other.args): - if (isinstance(self_a, Var)): - if not isinstance(other_a, Var): - return None + assert isinstance(self_a, Atom) and isinstance(other_a, Atom) + if (isinstance(self_a, Var)): if (self_a not in s): s[self_a] = other_a else: if (s[self_a] != other_a): return None - elif isinstance(self_a, ConstantInt): - if not isinstance(other_a, ConstantInt): - return None - assert self_a.kind == other_a.kind - if (self_a.value != other_a.value): - return None + elif isinstance(other_a, Var): + assert isinstance(self_a, Literal) + if (other_a not in s): + s[other_a] = self_a + else: + if s[other_a] != self_a: + return None else: - assert isinstance(self_a, Enumerator) - - if not isinstance(other_a, Enumerator): - # Currently don't support substitutions Var->Enumerator - return None - + assert (isinstance(self_a, Literal) and + isinstance(other_a, Literal)) # Guaranteed by self.inst == other.inst assert self_a.kind == other_a.kind - if (self_a.value != other_a.value): return None + return s -class ConstantInt(Expr): +class Literal(Atom): + """ + Base Class for all literal expressions in the DSL. + """ + def __init__(self, kind, value): + # type: (ImmediateKind, Any) -> None + self.kind = kind + self.value = value + + def __eq__(self, other): + # type: (Any) -> bool + if not isinstance(other, Literal): + return False + + if self.kind != other.kind: + return False + + # Can't just compare value here, as comparison Any <> Any returns Any + return repr(self) == repr(other) + + def __ne__(self, other): + # type: (Any) -> bool + return not self.__eq__(other) + + def __repr__(self): + # type: () -> str + return '{}.{}'.format(self.kind, self.value) + + +class ConstantInt(Literal): """ A value of an integer immediate operand. @@ -454,8 +492,7 @@ class ConstantInt(Expr): def __init__(self, kind, value): # type: (ImmediateKind, int) -> None - self.kind = kind - self.value = value + super(ConstantInt, self).__init__(kind, value) def __str__(self): # type: () -> str @@ -464,12 +501,8 @@ class ConstantInt(Expr): """ return str(self.value) - def __repr__(self): - # type: () -> str - return '{}({})'.format(self.kind, self.value) - -class Enumerator(Expr): +class Enumerator(Literal): """ A value of an enumerated immediate operand. @@ -486,8 +519,7 @@ class Enumerator(Expr): def __init__(self, kind, value): # type: (ImmediateKind, str) -> None - self.kind = kind - self.value = value + super(Enumerator, self).__init__(kind, value) def __str__(self): # type: () -> str @@ -495,7 +527,3 @@ class Enumerator(Expr): Get the Rust expression form of this enumerator. """ return self.kind.rust_enumerator(self.value) - - def __repr__(self): - # type: () -> str - return '{}.{}'.format(self.kind, self.value) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 30c27c6649..d295f59b41 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -9,7 +9,7 @@ try: from typing import Union, Sequence, List, Tuple, Any, TYPE_CHECKING # noqa from typing import Dict # noqa if TYPE_CHECKING: - from .ast import Expr, Apply, Var, Def # noqa + from .ast import Expr, Apply, Var, Def, VarAtomMap # noqa from .typevar import TypeVar # noqa from .ti import TypeConstraint # noqa from .xform import XForm, Rtl @@ -18,7 +18,7 @@ try: ConstrList = Union[Sequence[TypeConstraint], TypeConstraint] MaybeBoundInst = Union['Instruction', 'BoundInstruction'] InstructionSemantics = Sequence[XForm] - RtlCase = Union[Rtl, Tuple[Rtl, Sequence[TypeConstraint]]] + SemDefCase = Union[Rtl, Tuple[Rtl, Sequence[TypeConstraint]], XForm] except ImportError: pass @@ -349,7 +349,7 @@ class Instruction(object): return Apply(self, args) def set_semantics(self, src, *dsts): - # type: (Union[Def, Apply], *RtlCase) -> None + # type: (Union[Def, Apply], *SemDefCase) -> None """Set our semantics.""" from semantics import verify_semantics from .xform import XForm, Rtl @@ -358,6 +358,11 @@ class Instruction(object): for dst in dsts: if isinstance(dst, Rtl): sem.append(XForm(Rtl(src).copy({}), dst)) + elif isinstance(dst, XForm): + sem.append(XForm( + dst.src.copy({}), + dst.dst.copy({}), + dst.constraints)) else: assert isinstance(dst, tuple) sem.append(XForm(Rtl(src).copy({}), dst[0], diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index abf409a8c4..2ceb94b0fa 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -5,10 +5,10 @@ from .types import ValueType from .typevar import TypeVar try: - from typing import Union, Dict, TYPE_CHECKING # noqa + from typing import Union, Dict, TYPE_CHECKING, Iterable # noqa OperandSpec = Union['OperandKind', ValueType, TypeVar] if TYPE_CHECKING: - from .ast import Enumerator, ConstantInt # noqa + from .ast import Enumerator, ConstantInt, Literal # noqa except ImportError: pass @@ -128,6 +128,17 @@ class ImmediateKind(OperandKind): """ return '{}::{}'.format(self.rust_type, self.values[value]) + def is_enumerable(self): + # type: () -> bool + return self.values is not None + + def possible_values(self): + # type: () -> Iterable[Literal] + from cdsl.ast import Enumerator # noqa + assert self.is_enumerable() + for v in self.values.keys(): + yield Enumerator(self, v) + # Instances of entity reference operand types are provided in the # `cretonne.entities` module. diff --git a/lib/cretonne/meta/cdsl/test_xform.py b/lib/cretonne/meta/cdsl/test_xform.py index 952d8c90cb..424a7c824d 100644 --- a/lib/cretonne/meta/cdsl/test_xform.py +++ b/lib/cretonne/meta/cdsl/test_xform.py @@ -80,15 +80,52 @@ class TestXForm(TestCase): dst = Rtl(b << icmp(intcc.eq, z, u)) assert src.substitution(dst, {}) == {a: b, x: z, y: u} - def test_subst_enum_bad(self): + def test_subst_enum_var_const(self): src = Rtl(a << icmp(CC1, x, y)) dst = Rtl(b << icmp(intcc.eq, z, u)) - assert src.substitution(dst, {}) is None + assert src.substitution(dst, {}) == {CC1: intcc.eq, x: z, y: u, a: b},\ + "{} != {}".format(src.substitution(dst, {}), + {CC1: intcc.eq, x: z, y: u, a: b}) src = Rtl(a << icmp(intcc.eq, x, y)) dst = Rtl(b << icmp(CC1, z, u)) - assert src.substitution(dst, {}) is None + assert src.substitution(dst, {}) == {CC1: intcc.eq, x: z, y: u, a: b} + def test_subst_enum_bad(self): src = Rtl(a << icmp(intcc.eq, x, y)) dst = Rtl(b << icmp(intcc.sge, z, u)) assert src.substitution(dst, {}) is None + + def test_subst_enum_bad_var_const(self): + a1 = Var('a1') + x1 = Var('x1') + y1 = Var('y1') + + b1 = Var('b1') + z1 = Var('z1') + u1 = Var('u1') + + # Var mapping to 2 different constants + src = Rtl(a << icmp(CC1, x, y), + a1 << icmp(CC1, x1, y1)) + dst = Rtl(b << icmp(intcc.eq, z, u), + b1 << icmp(intcc.sge, z1, u1)) + + assert src.substitution(dst, {}) is None + + # 2 different constants mapping to the same var + src = Rtl(a << icmp(intcc.eq, x, y), + a1 << icmp(intcc.sge, x1, y1)) + dst = Rtl(b << icmp(CC1, z, u), + b1 << icmp(CC1, z1, u1)) + + assert src.substitution(dst, {}) is None + + # Var mapping to var and constant - note that full unification would + # have allowed this. + src = Rtl(a << icmp(CC1, x, y), + a1 << icmp(CC1, x1, y1)) + dst = Rtl(b << icmp(CC2, z, u), + b1 << icmp(intcc.sge, z1, u1)) + + assert src.substitution(dst, {}) is None diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index 991e429b18..261a70a4af 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -3,16 +3,16 @@ Instruction transformations. """ from __future__ import absolute_import from .ast import Def, Var, Apply -from .ti import ti_xform, TypeEnv, get_type_env +from .ti import ti_xform, TypeEnv, get_type_env, TypeConstraint from functools import reduce try: from typing import Union, Iterator, Sequence, Iterable, List, Dict # noqa from typing import Optional, Set # noqa - from .ast import Expr, VarMap # noqa + from .ast import Expr, VarAtomMap # noqa from .isa import TargetISA # noqa - from .ti import TypeConstraint # noqa from .typevar import TypeVar # noqa + from .instructions import ConstrList # noqa DefApply = Union[Def, Apply] except ImportError: pass @@ -47,7 +47,7 @@ class Rtl(object): self.rtl = tuple(map(canonicalize_defapply, args)) def copy(self, m): - # type: (VarMap) -> Rtl + # type: (VarAtomMap) -> Rtl """ Return a copy of this rtl with all Vars substituted with copies or according to m. Update m as neccessary. @@ -85,7 +85,7 @@ class Rtl(object): return reduce(flow_f, reversed(self.rtl), set([])) def substitution(self, other, s): - # type: (Rtl, VarMap) -> Optional[VarMap] + # type: (Rtl, VarAtomMap) -> Optional[VarAtomMap] """ If the Rtl self agrees structurally with the Rtl other, return a substitution to transform self to other. Two Rtls agree structurally if @@ -132,6 +132,10 @@ class Rtl(object): assert typing[v].singleton_type() is not None v.set_typevar(typing[v]) + def __str__(self): + # type: () -> str + return "\n".join(map(str, self.rtl)) + class XForm(object): """ @@ -162,7 +166,7 @@ class XForm(object): """ def __init__(self, src, dst, constraints=None): - # type: (Rtl, Rtl, Optional[Sequence[TypeConstraint]]) -> None + # type: (Rtl, Rtl, Optional[ConstrList]) -> None self.src = src self.dst = dst # Variables that are inputs to the source pattern. @@ -203,10 +207,18 @@ class XForm(object): return tv return symtab[tv.name[len("typeof_"):]].get_typevar() + self.constraints = [] # type: List[TypeConstraint] if constraints is not None: - for c in constraints: + if isinstance(constraints, TypeConstraint): + constr_list = [constraints] # type: Sequence[TypeConstraint] + else: + constr_list = constraints + + for c in constr_list: type_m = {tv: interp_tv(tv) for tv in c.tvs()} - self.ti.add_constraint(c.translate(type_m)) + inner_c = c.translate(type_m) + self.constraints.append(inner_c) + self.ti.add_constraint(inner_c) # Sanity: The set of inferred free typevars should be a subset of the # TVs corresponding to Vars appearing in src @@ -333,7 +345,7 @@ class XForm(object): defs are renamed with '.suffix' appended to their old name. """ assert r.is_concrete() - s = self.src.substitution(r, {}) # type: VarMap + s = self.src.substitution(r, {}) # type: VarAtomMap assert s is not None if (suffix is not None): diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 8a76d15e98..7189695a71 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -21,7 +21,7 @@ from cdsl.typevar import TypeVar try: from typing import Sequence, List, Dict, Set, DefaultDict # noqa from cdsl.isa import TargetISA # noqa - from cdsl.ast import Def # noqa + from cdsl.ast import Def, VarAtomMap # noqa from cdsl.xform import XForm, XFormGroup # noqa from cdsl.typevar import TypeSet # noqa from cdsl.ti import TypeConstraint # noqa @@ -45,7 +45,7 @@ def get_runtime_typechecks(xform): # 1) Perform ti only on the source RTL. Accumulate any free tvs that have a # different inferred type in src, compared to the type inferred for both # src and dst. - symtab = {} # type: Dict[Var, Var] + symtab = {} # type: VarAtomMap src_copy = xform.src.copy(symtab) src_typenv = get_type_env(ti_rtl(src_copy, TypeEnv())) @@ -62,7 +62,9 @@ def get_runtime_typechecks(xform): assert v.get_typevar().singleton_type() is not None continue - src_ts = src_typenv[symtab[v]].get_typeset() + inner_v = symtab[v] + assert isinstance(inner_v, Var) + src_ts = src_typenv[inner_v].get_typeset() xform_ts = xform.ti[v].get_typeset() assert xform_ts.issubset(src_ts) diff --git a/lib/cretonne/meta/semantics/__init__.py b/lib/cretonne/meta/semantics/__init__.py index 1c1fee9b9f..94e32b652c 100644 --- a/lib/cretonne/meta/semantics/__init__.py +++ b/lib/cretonne/meta/semantics/__init__.py @@ -1,9 +1,11 @@ """Definitions for the semantics segment of the Cretonne language.""" from cdsl.ti import TypeEnv, ti_rtl, get_type_env +from cdsl.operands import ImmediateKind +from cdsl.ast import Var try: from typing import List, Dict, Tuple # noqa - from cdsl.ast import Var # noqa + from cdsl.ast import VarAtomMap # noqa from cdsl.xform import XForm, Rtl # noqa from cdsl.ti import VarTyping # noqa from cdsl.instructions import Instruction, InstructionSemantics # noqa @@ -16,34 +18,60 @@ def verify_semantics(inst, src, xforms): """ Verify that the semantics transforms in xforms correctly describe the instruction described by the src Rtl. This involves checking that: - 1) For all XForms x \in xforms, there is a Var substitution form src to - x.src - 2) For any possible concrete typing of src there is exactly 1 XForm x - in xforms that applies. + 0) src is a single instance of inst + 1) For all x\in xforms x.src is a single instance of inst + 2) For any concrete values V of Literals in inst: + For all concrete typing T of inst: + Exists single x \in xforms that applies to src conretazied to V + and T """ - # 0) The source rtl is always a single instruction - assert len(src.rtl) == 1 + # 0) The source rtl is always a single instance of inst + assert len(src.rtl) == 1 and src.rtl[0].expr.inst == inst - # 1) For all XForms x, x.src is structurally equivalent to src + # 1) For all XForms x, x.src is a single instance of inst for x in xforms: - assert src.substitution(x.src, {}) is not None,\ - "XForm {} doesn't describe instruction {}.".format(x, src) + assert len(x.src.rtl) == 1 and x.src.rtl[0].expr.inst == inst - # 2) Any possible typing for the instruction should be covered by - # exactly ONE semantic XForm - src = src.copy({}) - typenv = get_type_env(ti_rtl(src, TypeEnv())) - typenv.normalize() - typenv = typenv.extract() + variants = [src] # type: List[Rtl] - for t in typenv.concrete_typings(): - matching_xforms = [] # type: List[XForm] - for x in xforms: - # Translate t using x.symtab - t = {x.symtab[str(v)]: tv for (v, tv) in t.items()} - if (x.ti.permits(t)): - matching_xforms.append(x) + # 2) For all enumerated immediates, compute all the possible + # versions of src with the concrete value filled in. + for i in inst.imm_opnums: + op = inst.ins[i] + if not (isinstance(op.kind, ImmediateKind) and + op.kind.is_enumerable()): + continue - assert len(matching_xforms) == 1,\ - ("Possible typing {} of {} not matched by exactly one case " + - ": {}").format(t, inst, matching_xforms) + new_variants = [] # type: List[Rtl] + for rtl_var in variants: + s = {v: v for v in rtl_var.vars()} # type: VarAtomMap + arg = rtl_var.rtl[0].expr.args[i] + assert isinstance(arg, Var) + for val in op.kind.possible_values(): + s[arg] = val + new_variants.append(rtl_var.copy(s)) + variants = new_variants + + # For any possible version of the src with concrete enumerated immediates + for src in variants: + # 2) Any possible typing should be covered by exactly ONE semantic + # XForm + src = src.copy({}) + typenv = get_type_env(ti_rtl(src, TypeEnv())) + typenv.normalize() + typenv = typenv.extract() + + for t in typenv.concrete_typings(): + matching_xforms = [] # type: List[XForm] + for x in xforms: + if src.substitution(x.src, {}) is None: + continue + + # Translate t using x.symtab + t = {x.symtab[str(v)]: tv for (v, tv) in t.items()} + if (x.ti.permits(t)): + matching_xforms.append(x) + + assert len(matching_xforms) == 1,\ + ("Possible typing {} of {} not matched by exactly one case " + + ": {}").format(t, src.rtl[0], matching_xforms) diff --git a/lib/cretonne/meta/semantics/elaborate.py b/lib/cretonne/meta/semantics/elaborate.py index fc0ca98cc4..8d2ecd7fd6 100644 --- a/lib/cretonne/meta/semantics/elaborate.py +++ b/lib/cretonne/meta/semantics/elaborate.py @@ -10,7 +10,7 @@ from cdsl.ast import Var try: from typing import TYPE_CHECKING, Dict, Union, List, Set, Tuple # noqa from cdsl.xform import XForm # noqa - from cdsl.ast import Def, VarMap # noqa + from cdsl.ast import Def, VarAtomMap # noqa from cdsl.ti import VarTyping # noqa except ImportError: TYPE_CHECKING = False @@ -34,7 +34,13 @@ def find_matching_xform(d): if (subst is None): continue - if x.ti.permits({subst[v]: tv for (v, tv) in typing.items()}): + inner_typing = {} # type: VarTyping + for (v, tv) in typing.items(): + inner_v = subst[v] + assert isinstance(inner_v, Var) + inner_typing[inner_v] = tv + + if x.ti.permits(inner_typing): res.append(x) assert len(res) == 1, "Couldn't find semantic transform for {}".format(d) @@ -60,7 +66,7 @@ def cleanup_semantics(r, outputs): ... """ new_defs = [] # type: List[Def] - subst_m = {v: v for v in r.vars()} # type: VarMap + subst_m = {v: v for v in r.vars()} # type: VarAtomMap definition = {} # type: Dict[Var, Def] prim_to_bv_map = {} # type: Dict[Var, Def] diff --git a/lib/cretonne/meta/semantics/macros.py b/lib/cretonne/meta/semantics/macros.py new file mode 100644 index 0000000000..566bf92eae --- /dev/null +++ b/lib/cretonne/meta/semantics/macros.py @@ -0,0 +1,45 @@ +""" +Useful semantics "macro" instructions built on top of +the primitives. +""" +from __future__ import absolute_import +from cdsl.operands import Operand +from cdsl.typevar import TypeVar +from cdsl.instructions import Instruction, InstructionGroup +from base.types import b1 +from base.immediates import imm64 +from cdsl.ast import Var +from cdsl.xform import Rtl +from semantics.primitives import bv_from_imm64, bvite +import base.formats # noqa + +GROUP = InstructionGroup("primitive_macros", "Semantic macros instruction set") +AnyBV = TypeVar('AnyBV', bitvecs=True, doc="") +x = Var('x') +y = Var('y') +imm = Var('imm') +a = Var('a') + +# +# Bool-to-bv1 +# +BV1 = TypeVar("BV1", bitvecs=(1, 1), doc="") +bv1_op = Operand('bv1_op', BV1, doc="") +cond_op = Operand("cond", b1, doc="") +bool2bv = Instruction( + 'bool2bv', r"""Convert a b1 value to a 1-bit BV""", + ins=cond_op, outs=bv1_op) + +v1 = Var('v1') +v2 = Var('v2') +bvone = Var('bvone') +bvzero = Var('bvzero') +bool2bv.set_semantics( + v1 << bool2bv(v2), + Rtl( + bvone << bv_from_imm64(imm64(1)), + bvzero << bv_from_imm64(imm64(0)), + v1 << bvite(v2, bvone, bvzero) + )) + +GROUP.close() diff --git a/lib/cretonne/meta/semantics/primitives.py b/lib/cretonne/meta/semantics/primitives.py index 0a727c1cf9..656db41538 100644 --- a/lib/cretonne/meta/semantics/primitives.py +++ b/lib/cretonne/meta/semantics/primitives.py @@ -10,6 +10,8 @@ from cdsl.operands import Operand from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup from cdsl.ti import WiderOrEq +from base.types import b1 +from base.immediates import imm64 import base.formats # noqa GROUP = InstructionGroup("primitive", "Primitive instruction set") @@ -22,26 +24,40 @@ Real = TypeVar('Real', 'Any real type.', ints=True, floats=True, x = Operand('x', BV, doc="A semantic value X") y = Operand('x', BV, doc="A semantic value Y (same width as X)") a = Operand('a', BV, doc="A semantic value A (same width as X)") +cond = Operand('b', TypeVar.singleton(b1), doc='A b1 value') real = Operand('real', Real, doc="A real cretonne value") fromReal = Operand('fromReal', Real.to_bitvec(), doc="A real cretonne value converted to a BV") +# +# BV Conversion/Materialization +# prim_to_bv = Instruction( 'prim_to_bv', r""" Convert an SSA Value to a flat bitvector """, ins=(real), outs=(fromReal)) -# Note that when converting from BV->real values, we use a constraint and not a -# derived function. This reflects that fact that to_bitvec() is not a -# bijection. prim_from_bv = Instruction( 'prim_from_bv', r""" Convert a flat bitvector to a real SSA Value. """, ins=(fromReal), outs=(real)) +N = Operand('N', imm64) +bv_from_imm64 = Instruction( + 'bv_from_imm64', r"""Materialize an imm64 as a bitvector.""", + ins=(N), outs=a) + +# +# Generics +# +bvite = Instruction( + 'bvite', r"""Bitvector ternary operator""", + ins=(cond, x, y), outs=a) + + xh = Operand('xh', BV.half_width(), doc="A semantic value representing the upper half of X") xl = Operand('xl', BV.half_width(), @@ -67,12 +83,40 @@ bvadd = Instruction( of the operands. """, ins=(x, y), outs=a) - +# # Bitvector comparisons -cmp_res = Operand('cmp_res', BV1, doc="Single bit boolean") +# + +bveq = Instruction( + 'bveq', r"""Unsigned bitvector equality""", + ins=(x, y), outs=cond) +bvne = Instruction( + 'bveq', r"""Unsigned bitvector inequality""", + ins=(x, y), outs=cond) +bvsge = Instruction( + 'bvsge', r"""Signed bitvector greater or equal""", + ins=(x, y), outs=cond) +bvsgt = Instruction( + 'bvsgt', r"""Signed bitvector greater than""", + ins=(x, y), outs=cond) +bvsle = Instruction( + 'bvsle', r"""Signed bitvector less than or equal""", + ins=(x, y), outs=cond) +bvslt = Instruction( + 'bvslt', r"""Signed bitvector less than""", + ins=(x, y), outs=cond) +bvuge = Instruction( + 'bvuge', r"""Unsigned bitvector greater or equal""", + ins=(x, y), outs=cond) +bvugt = Instruction( + 'bvugt', r"""Unsigned bitvector greater than""", + ins=(x, y), outs=cond) +bvule = Instruction( + 'bvule', r"""Unsigned bitvector less than or equal""", + ins=(x, y), outs=cond) bvult = Instruction( - 'bvult', r"""Unsigned bitvector comparison""", - ins=(x, y), outs=cmp_res) + 'bvult', r"""Unsigned bitvector less than""", + ins=(x, y), outs=cond) # Extensions ToBV = TypeVar('ToBV', 'A bitvector type.', bitvecs=True) diff --git a/lib/cretonne/meta/semantics/smtlib.py b/lib/cretonne/meta/semantics/smtlib.py index c1b2526832..3a2c819153 100644 --- a/lib/cretonne/meta/semantics/smtlib.py +++ b/lib/cretonne/meta/semantics/smtlib.py @@ -14,7 +14,7 @@ from z3.z3core import Z3_mk_eq try: from typing import TYPE_CHECKING, Tuple, Dict, List # noqa from cdsl.xform import Rtl, XForm # noqa - from cdsl.ast import VarMap # noqa + from cdsl.ast import VarAtomMap, Atom # noqa from cdsl.ti import VarTyping # noqa if TYPE_CHECKING: from z3 import ExprRef, BitVecRef # noqa @@ -137,13 +137,13 @@ def to_smt(r): def equivalent(r1, r2, inp_m, out_m): - # type: (Rtl, Rtl, VarMap, VarMap) -> List[ExprRef] + # type: (Rtl, Rtl, VarAtomMap, VarAtomMap) -> List[ExprRef] """ Given: - concrete source Rtl r1 - concrete dest Rtl r2 - - VarMap inp_m mapping r1's non-bitvector inputs to r2 - - VarMap out_m mapping r1's non-bitvector outputs to r2 + - VarAtomMap inp_m mapping r1's non-bitvector inputs to r2 + - VarAtomMap out_m mapping r1's non-bitvector outputs to r2 Build a query checking whether r1 and r2 are semantically equivalent. If the returned query is unsatisfiable, then r1 and r2 are equivalent. @@ -156,17 +156,31 @@ def equivalent(r1, r2, inp_m, out_m): assert set(r2.free_vars()) == set(inp_m.values()) # Note that the same rule is not expected to hold for out_m due to - # temporaries/intermediates. + # temporaries/intermediates. out_m specified which values are enough for + # equivalence. # Rename the vars in r1 and r2 with unique suffixes to avoid conflicts - src_m = {v: Var(v.name + ".a", v.get_typevar()) for v in r1.vars()} - dst_m = {v: Var(v.name + ".b", v.get_typevar()) for v in r2.vars()} + src_m = {v: Var(v.name + ".a", v.get_typevar()) for v in r1.vars()} # type: VarAtomMap # noqa + dst_m = {v: Var(v.name + ".b", v.get_typevar()) for v in r2.vars()} # type: VarAtomMap # noqa r1 = r1.copy(src_m) r2 = r2.copy(dst_m) + def _translate(m, k_m, v_m): + # type: (VarAtomMap, VarAtomMap, VarAtomMap) -> VarAtomMap + """Obtain a new map from m, by mapping m's keys with k_m and m's values + with v_m""" + res = {} # type: VarAtomMap + for (k, v) in m1.items(): + new_k = k_m[k] + new_v = v_m[v] + assert isinstance(new_k, Var) + res[new_k] = new_v + + return res + # Convert inp_m, out_m in terms of variables with the .a/.b suffixes - inp_m = {src_m[k]: dst_m[v] for (k, v) in inp_m.items()} - out_m = {src_m[k]: dst_m[v] for (k, v) in out_m.items()} + inp_m = _translate(inp_m, src_m, dst_m) + out_m = _translate(out_m, src_m, dst_m) # Encode r1 and r2 as SMT queries (q1, m1) = to_smt(r1) @@ -175,12 +189,14 @@ def equivalent(r1, r2, inp_m, out_m): # Build an expression for the equality of real Cretone inputs of r1 and r2 args_eq_exp = [] # type: List[ExprRef] - for v in r1.free_vars(): - args_eq_exp.append(mk_eq(m1[v], m2[inp_m[v]])) + for (v1, v2) in inp_m.items(): + assert isinstance(v2, Var) + args_eq_exp.append(mk_eq(m1[v1], m2[v2])) # Build an expression for the equality of real Cretone outputs of r1 and r2 results_eq_exp = [] # type: List[ExprRef] for (v1, v2) in out_m.items(): + assert isinstance(v2, Var) results_eq_exp.append(mk_eq(m1[v1], m2[v2])) # Put the whole query toghether @@ -196,20 +212,22 @@ def xform_correct(x, typing): assert x.ti.permits(typing) # Create copies of the x.src and x.dst with their concrete types - src_m = {v: Var(v.name, typing[v]) for v in x.src.vars()} + src_m = {v: Var(v.name, typing[v]) for v in x.src.vars()} # type: VarAtomMap # noqa src = x.src.copy(src_m) dst = x.apply(src) dst_m = x.dst.substitution(dst, {}) # Build maps for the inputs/outputs for src->dst - inp_m = {} - out_m = {} + inp_m = {} # type: VarAtomMap + out_m = {} # type: VarAtomMap for v in x.src.vars(): + src_v = src_m[v] + assert isinstance(src_v, Var) if v.is_input(): - inp_m[src_m[v]] = dst_m[v] + inp_m[src_v] = dst_m[v] elif v.is_output(): - out_m[src_m[v]] = dst_m[v] + out_m[src_v] = dst_m[v] # Get the primitive semantic Rtls for src and dst prim_src = elaborate(src) diff --git a/lib/cretonne/meta/semantics/test_elaborate.py b/lib/cretonne/meta/semantics/test_elaborate.py index cb798295b9..9ca938bc1f 100644 --- a/lib/cretonne/meta/semantics/test_elaborate.py +++ b/lib/cretonne/meta/semantics/test_elaborate.py @@ -1,7 +1,7 @@ from __future__ import absolute_import from base.instructions import vselect, vsplit, vconcat, iconst, iadd, bint from base.instructions import b1, icmp, ireduce, iadd_cout -from base.immediates import intcc +from base.immediates import intcc, imm64 from base.types import i64, i8, b32, i32, i16, f32 from cdsl.typevar import TypeVar from cdsl.ast import Var @@ -9,7 +9,7 @@ from cdsl.xform import Rtl from unittest import TestCase from .elaborate import elaborate from .primitives import prim_to_bv, bvsplit, prim_from_bv, bvconcat, bvadd, \ - bvult + bvult, bv_from_imm64, bvite import base.semantics # noqa @@ -366,9 +366,12 @@ class TestElaborate(TestCase): a = Var('a') c_out = Var('c_out') bvc_out = Var('bvc_out') + bc_out = Var('bc_out') bvx = Var('bvx') bvy = Var('bvy') bva = Var('bva') + bvone = Var('bvone') + bvzero = Var('bvzero') r = Rtl( (a, c_out) << iadd_cout.i32(x, y), ) @@ -378,10 +381,12 @@ class TestElaborate(TestCase): bvx << prim_to_bv.i32(x), bvy << prim_to_bv.i32(y), bva << bvadd.bv32(bvx, bvy), - bvc_out << bvult.bv32(bva, bvx), + bc_out << bvult.bv32(bva, bvx), + bvone << bv_from_imm64(imm64(1)), + bvzero << bv_from_imm64(imm64(0)), + bvc_out << bvite(bc_out, bvone, bvzero), a << prim_from_bv.i32(bva), c_out << prim_from_bv.b1(bvc_out) ) exp.cleanup_concrete_rtl() - assert concrete_rtls_eq(sem, exp) From 7e402a6104851d18f87687c4c9082ede08f10b20 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 17 Aug 2017 10:40:35 -0700 Subject: [PATCH 0968/3084] Document memory operation flags. Also move the extending loads and truncating stores into the bulkier "Operations" section to improve the flow of the "Memory" section in the language reference. --- cranelift/docs/langref.rst | 77 ++++++++++++++++---------- lib/cretonne/meta/base/instructions.py | 3 +- 2 files changed, 50 insertions(+), 30 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index f20b5b6e84..5ad4d89247 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -424,38 +424,36 @@ Memory ====== Cretonne provides fully general :inst:`load` and :inst:`store` instructions for -accessing memory. However, it can be very complicated to verify the safety of -general loads and stores when compiling code for a sandboxed environment, so -Cretonne also provides more restricted memory operations that are always safe. +accessing memory, as well as :ref:`extending loads and truncating stores +`. There are also more restricted operations for accessing +specific types of memory objects. .. autoinst:: load .. autoinst:: store +Memory operation flags +---------------------- + +Loads and stores can have flags that loosen their semantics in order to enable +optimizations. + +======= ========================================= +Flag Description +======= ========================================= +notrap Trapping is not required. +aligned Trapping allowed for misaligned accesses. +======= ========================================= + +Trapping is part of the semantics of memory accesses. The operating system may +have configured parts of the address space to cause a trap when read and/or +written, and Cretonne's memory instructions respect that. When the ``notrap`` +flat is set, the trapping behavior is optional. This allows the optimizer to +delete loads whose results are not used. + Loads and stores are *misaligned* if the resultant address is not a multiple of -the expected alignment. Depending on the target architecture, misaligned memory -accesses may trap, or they may work. Sometimes, operating systems catch -alignment traps and emulate the misaligned memory access. - - -Extending loads and truncating stores -------------------------------------- - -Most ISAs provide instructions that load an integer value smaller than a register -and extends it to the width of the register. Similarly, store instructions that -only write the low bits of an integer register are common. - -Cretonne provides extending loads and truncation stores for 8, 16, and 32-bit -memory accesses. - -.. autoinst:: uload8 -.. autoinst:: sload8 -.. autoinst:: istore8 -.. autoinst:: uload16 -.. autoinst:: sload16 -.. autoinst:: istore16 -.. autoinst:: uload32 -.. autoinst:: sload32 -.. autoinst:: istore32 +the expected alignment. By default, misaligned loads and stores are allowed, +but when the ``aligned`` flag is set, a misaligned memory access is allowed to +trap. Local variables --------------- @@ -548,8 +546,6 @@ depends on the runtime environment. Operations ========== -The remaining instruction set is mostly arithmetic. - A few instructions have variants that take immediate operands (e.g., :inst:`band` / :inst:`band_imm`), but in general an instruction is required to load a constant into an SSA value. @@ -766,6 +762,29 @@ the target ISA. .. autoinst:: isplit .. autoinst:: iconcat +.. _extload-truncstore: + +Extending loads and truncating stores +------------------------------------- + +Most ISAs provide instructions that load an integer value smaller than a register +and extends it to the width of the register. Similarly, store instructions that +only write the low bits of an integer register are common. + +In addition to the normal :inst:`load` and :inst:`store` instructions, Cretonne +provides extending loads and truncation stores for 8, 16, and 32-bit memory +accesses. + +.. autoinst:: uload8 +.. autoinst:: sload8 +.. autoinst:: istore8 +.. autoinst:: uload16 +.. autoinst:: sload16 +.. autoinst:: istore16 +.. autoinst:: uload32 +.. autoinst:: sload32 +.. autoinst:: istore32 + ISA-specific instructions ========================= diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 3a4ce25cda..a303045331 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -190,7 +190,7 @@ call_indirect = Instruction( # SS = Operand('SS', entities.stack_slot) -Offset = Operand('Offset', offset32, 'In-bounds offset into stack slot') +Offset = Operand('Offset', offset32, 'Byte offset from base address') x = Operand('x', Mem, doc='Value to be stored') a = Operand('a', Mem, doc='Value loaded') p = Operand('p', iAddr) @@ -307,6 +307,7 @@ istore32 = Instruction( x = Operand('x', Mem, doc='Value to be stored') a = Operand('a', Mem, doc='Value loaded') +Offset = Operand('Offset', offset32, 'In-bounds offset into stack slot') stack_load = Instruction( 'stack_load', r""" From bf4ae3bb2e295720bc4d3a5e46ef6fb91369049c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 17 Aug 2017 11:30:00 -0700 Subject: [PATCH 0969/3084] Add global variables to Cretonne IL. See #144 for discussion. - Add a new GlobalVar entity type both in Python and Rust. - Define a UnaryGlobalVar instruction format containing a GlobalVar reference. - Add a globalvar.rs module defining the GlobalVarData with support for 'vmctx' and 'deref' global variable kinds. Langref: Add a section about global variables and the global_addr instruction. Parser: Add support for the UnaryGlobalVar instruction format as well as global variable declarations in the preamble. --- cranelift/docs/Makefile | 2 +- cranelift/docs/langref.rst | 56 +++++++++++++++++ cranelift/filetests/parser/memory.cton | 37 +++++++++++ lib/cretonne/meta/base/entities.py | 3 + lib/cretonne/meta/base/formats.py | 6 +- lib/cretonne/meta/base/instructions.py | 12 ++++ lib/cretonne/meta/cdsl/operands.py | 2 +- lib/cretonne/src/ir/builder.rs | 4 +- lib/cretonne/src/ir/entities.rs | 27 ++++++++ lib/cretonne/src/ir/function.rs | 9 ++- lib/cretonne/src/ir/globalvar.rs | 37 +++++++++++ lib/cretonne/src/ir/instructions.rs | 4 ++ lib/cretonne/src/ir/mod.rs | 7 ++- lib/cretonne/src/verifier/mod.rs | 13 +++- lib/cretonne/src/write.rs | 7 ++- lib/reader/src/lexer.rs | 2 + lib/reader/src/parser.rs | 85 +++++++++++++++++++++++++- lib/reader/src/sourcemap.rs | 37 ++++++++++- 18 files changed, 336 insertions(+), 14 deletions(-) create mode 100644 cranelift/filetests/parser/memory.cton create mode 100644 lib/cretonne/src/ir/globalvar.rs diff --git a/cranelift/docs/Makefile b/cranelift/docs/Makefile index 8cec8c36e6..635aafde1c 100644 --- a/cranelift/docs/Makefile +++ b/cranelift/docs/Makefile @@ -58,7 +58,7 @@ html: @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." autohtml: html - $(SPHINXABUILD) -z ../lib/cretonne/meta --ignore '.*.sw?' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html + $(SPHINXABUILD) -z ../lib/cretonne/meta --ignore '.*' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 5ad4d89247..d1d353186a 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -495,6 +495,62 @@ instructions before instruction selection:: v9 = stack_addr ss3, 16 v1 = load.f64 v9 +Global variables +---------------- + +A *global variable* is an object in memory whose address is not known at +compile time. The address is computed at runtime by :inst:`global_addr`, +possibly using information provided by the linker via relocations. There are +multiple kinds of global variables using different methods for determining +their address. Cretonne does not track the type or even the size of global +variables, they are just pointers to non-stack memory. + +When Cretonne is generating code for a virtual machine environment, globals can +be used to access data structures in the VM's runtime. This requires functions +to have access to a *VM context pointer* which is used as the base address. +Typically, the VM context pointer is passed as a hidden function argument to +Cretonne functions. + +.. inst:: GV = vmctx+Offset + + Declare a global variable in the VM context struct. + + This declares a global variable whose address is a constant offset from the + VM context pointer which is passed as a hidden argument to all functions + JIT-compiled for the VM. + + Typically, the VM context is a C struct, and the declared global variable + is a member of the struct. + + :arg Offset: Byte offset from the VM context pointer to the global + variable. + :result GV: Global variable. + +The address of a global variable can also be derived by treating another global +variable as a struct pointer. This makes it possible to chase pointers into VM +runtime data structures. + +.. inst:: GV = deref(BaseGV)+Offset + + Declare a global variable in a struct pointed to by BaseGV. + + The address of GV can be computed by first loading a pointer from BaseGV + and adding Offset to it. + + It is assumed the BaseGV resides in readable memory with the apropriate + alignment for storing a pointer. + + Chains of ``deref`` global variables are possible, but cycles are not + allowed. They will be caught by the IL verifier. + + :arg BaseGV: Global variable containing the base pointer. + :arg Offset: Byte offset from the loaded base pointer to the global + variable. + :result GV: Global variable. + +.. autoinst:: global_addr + + Heaps ----- diff --git a/cranelift/filetests/parser/memory.cton b/cranelift/filetests/parser/memory.cton new file mode 100644 index 0000000000..5f300394b3 --- /dev/null +++ b/cranelift/filetests/parser/memory.cton @@ -0,0 +1,37 @@ +test cat +test verifier + +function %vmglobal() -> i32 { + gv3 = vmctx+16 + ; check: $gv3 = vmctx+16 + gv4 = vmctx+0 + ; check: $gv4 = vmctx + ; not: +0 + gv5 = vmctx -256 + ; check: $gv5 = vmctx-256 +ebb0: + v1 = global_addr.i32 gv3 + ; check: $v1 = global_addr.i32 $gv3 + return v1 +} + +function %deref() -> i32 { + gv3 = vmctx+16 + gv4 = deref(gv3)-32 + ; check: $gv4 = deref($gv3)-32 +ebb0: + v1 = global_addr.i32 gv4 + ; check: $v1 = global_addr.i32 $gv4 + return v1 +} + +; Refer to a global variable before it's been declared. +function %backref() -> i32 { + gv1 = deref(gv2)-32 + ; check: $gv1 = deref($gv2)-32 + gv2 = vmctx+16 + ; check: $gv2 = vmctx+16 +ebb0: + v1 = global_addr.i32 gv1 + return v1 +} diff --git a/lib/cretonne/meta/base/entities.py b/lib/cretonne/meta/base/entities.py index 327d45e8bc..6a9c1ff5d2 100644 --- a/lib/cretonne/meta/base/entities.py +++ b/lib/cretonne/meta/base/entities.py @@ -16,6 +16,9 @@ ebb = EntityRefKind( #: A reference to a stack slot declared in the function preamble. stack_slot = EntityRefKind('stack_slot', 'A stack slot.') +#: A reference to a global variable. +global_var = EntityRefKind('global_var', 'A global variable.') + #: A reference to a function sugnature declared in the function preamble. #: Tbis is used to provide the call signature in an indirect call instruction. sig_ref = EntityRefKind('sig_ref', 'A function signature.') diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 16732c1153..62572f5a02 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -10,7 +10,8 @@ from cdsl.formats import InstructionFormat from cdsl.operands import VALUE, VARIABLE_ARGS from .immediates import imm64, uimm8, ieee32, ieee64, offset32, uoffset32 from .immediates import boolean, intcc, floatcc, memflags, regunit -from .entities import ebb, sig_ref, func_ref, jump_table, stack_slot +from . import entities +from .entities import ebb, sig_ref, func_ref, stack_slot Nullary = InstructionFormat() @@ -19,6 +20,7 @@ UnaryImm = InstructionFormat(imm64) UnaryIeee32 = InstructionFormat(ieee32) UnaryIeee64 = InstructionFormat(ieee64) UnaryBool = InstructionFormat(boolean) +UnaryGlobalVar = InstructionFormat(entities.global_var) Binary = InstructionFormat(VALUE, VALUE) BinaryImm = InstructionFormat(VALUE, imm64) @@ -42,7 +44,7 @@ FloatCompare = InstructionFormat(floatcc, VALUE, VALUE) Jump = InstructionFormat(ebb, VARIABLE_ARGS) Branch = InstructionFormat(VALUE, ebb, VARIABLE_ARGS) BranchIcmp = InstructionFormat(intcc, VALUE, VALUE, ebb, VARIABLE_ARGS) -BranchTable = InstructionFormat(VALUE, jump_table) +BranchTable = InstructionFormat(VALUE, entities.jump_table) Call = InstructionFormat(func_ref, VARIABLE_ARGS) IndirectCall = InstructionFormat(sig_ref, VALUE, VARIABLE_ARGS) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index a303045331..4e80641d76 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -345,6 +345,18 @@ stack_addr = Instruction( """, ins=(SS, Offset), outs=addr) +# +# Global variables. +# + +GV = Operand('GV', entities.global_var) + +global_addr = Instruction( + 'global_addr', r""" + Compute the address of global variable GV. + """, + ins=GV, outs=addr) + # # WebAssembly bounds-checked heap accesses. # diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index 2ceb94b0fa..c80632f22a 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -59,7 +59,7 @@ VARIABLE_ARGS = OperandKind( 'variable_args', """ A variable size list of `value` operands. - Use this to represent arguemtns passed to a function call, arguments + Use this to represent arguments passed to a function call, arguments passed to an extended basic block, or a variable number of results returned from an instruction. """, diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 697acfebd5..3673788543 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -5,8 +5,8 @@ use ir::types; use ir::{InstructionData, DataFlowGraph}; -use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, StackSlot, ValueList, - MemFlags}; +use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, StackSlot, GlobalVar, + ValueList, MemFlags}; use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32, Uoffset32}; use ir::condcodes::{IntCC, FloatCC}; use isa::RegUnit; diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 98a29c7e61..628a442fdc 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -65,6 +65,24 @@ entity_impl!(Inst, "inst"); pub struct StackSlot(u32); entity_impl!(StackSlot, "ss"); +/// An opaque reference to a global variable. +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +pub struct GlobalVar(u32); +entity_impl!(GlobalVar, "gv"); + +impl GlobalVar { + /// Create a new global variable reference from its number. + /// + /// This method is for use by the parser. + pub fn with_number(n: u32) -> Option { + if n < u32::MAX { + Some(GlobalVar(n)) + } else { + None + } + } +} + /// An opaque reference to a jump table. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub struct JumpTable(u32); @@ -93,6 +111,8 @@ pub enum AnyEntity { Value(Value), /// A stack slot. StackSlot(StackSlot), + /// A Global variable. + GlobalVar(GlobalVar), /// A jump table. JumpTable(JumpTable), /// An external function. @@ -109,6 +129,7 @@ impl fmt::Display for AnyEntity { AnyEntity::Inst(r) => r.fmt(f), AnyEntity::Value(r) => r.fmt(f), AnyEntity::StackSlot(r) => r.fmt(f), + AnyEntity::GlobalVar(r) => r.fmt(f), AnyEntity::JumpTable(r) => r.fmt(f), AnyEntity::FuncRef(r) => r.fmt(f), AnyEntity::SigRef(r) => r.fmt(f), @@ -140,6 +161,12 @@ impl From for AnyEntity { } } +impl From for AnyEntity { + fn from(r: GlobalVar) -> AnyEntity { + AnyEntity::GlobalVar(r) + } +} + impl From for AnyEntity { fn from(r: JumpTable) -> AnyEntity { AnyEntity::JumpTable(r) diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index c2b280dc6c..028c376479 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -4,8 +4,8 @@ //! instructions. use entity_map::{EntityMap, PrimaryEntityData}; -use ir::{FunctionName, CallConv, Signature, JumpTableData, DataFlowGraph, Layout}; -use ir::{JumpTables, InstEncodings, ValueLocations, StackSlots, EbbOffsets}; +use ir::{FunctionName, CallConv, Signature, JumpTableData, GlobalVarData, DataFlowGraph, Layout}; +use ir::{JumpTables, InstEncodings, ValueLocations, StackSlots, GlobalVars, EbbOffsets}; use isa::TargetIsa; use std::fmt; use write::write_function; @@ -25,6 +25,9 @@ pub struct Function { /// Stack slots allocated in this function. pub stack_slots: StackSlots, + /// Global variables referenced. + pub global_vars: GlobalVars, + /// Jump tables used in this function. pub jump_tables: JumpTables, @@ -50,6 +53,7 @@ pub struct Function { } impl PrimaryEntityData for JumpTableData {} +impl PrimaryEntityData for GlobalVarData {} impl Function { /// Create a function with the given name and signature. @@ -58,6 +62,7 @@ impl Function { name, signature: sig, stack_slots: StackSlots::new(), + global_vars: GlobalVars::new(), jump_tables: EntityMap::new(), dfg: DataFlowGraph::new(), layout: Layout::new(), diff --git a/lib/cretonne/src/ir/globalvar.rs b/lib/cretonne/src/ir/globalvar.rs new file mode 100644 index 0000000000..ad209b0eca --- /dev/null +++ b/lib/cretonne/src/ir/globalvar.rs @@ -0,0 +1,37 @@ +//! Global variables. + +use ir::GlobalVar; +use ir::immediates::Offset32; +use std::fmt; + +/// Information about a global variable declaration. +#[derive(Clone)] +pub enum GlobalVarData { + /// Variable is part of the VM context struct, it's address is a constant offset from the VM + /// context pointer. + VmCtx { + /// Offset from the `vmctx` pointer to this global. + offset: Offset32, + }, + + /// Variable is part of a struct pointed to by another global variable. + /// + /// The `base` global variable is assumed to contain a pointer to a struct. This global + /// variable lives at an offset into the struct. + Deref { + /// The base pointer global variable. + base: GlobalVar, + + /// Byte offset to be added to the pointer loaded from `base`. + offset: Offset32, + }, +} + +impl fmt::Display for GlobalVarData { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self { + &GlobalVarData::VmCtx { offset } => write!(f, "vmctx{}", offset), + &GlobalVarData::Deref { base, offset } => write!(f, "deref({}){}", base, offset), + } + } +} diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 900dd56e88..b7c5d898e5 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -109,6 +109,10 @@ pub enum InstructionData { UnaryIeee32 { opcode: Opcode, imm: Ieee32 }, UnaryIeee64 { opcode: Opcode, imm: Ieee64 }, UnaryBool { opcode: Opcode, imm: bool }, + UnaryGlobalVar { + opcode: Opcode, + global_var: ir::GlobalVar, + }, Binary { opcode: Opcode, args: [Value; 2] }, BinaryImm { opcode: Opcode, diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 443d70bd1c..1e31e34010 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -13,17 +13,19 @@ pub mod function; mod builder; mod extfunc; mod funcname; +mod globalvar; mod memflags; mod progpoint; mod valueloc; pub use ir::builder::{InstBuilder, InstBuilderBase, InstInserterBase, InsertBuilder}; pub use ir::dfg::{DataFlowGraph, ValueDef}; -pub use ir::entities::{Ebb, Inst, Value, StackSlot, JumpTable, FuncRef, SigRef}; +pub use ir::entities::{Ebb, Inst, Value, StackSlot, GlobalVar, JumpTable, FuncRef, SigRef}; pub use ir::extfunc::{Signature, CallConv, ArgumentType, ArgumentExtension, ArgumentPurpose, ExtFuncData}; pub use ir::funcname::FunctionName; pub use ir::function::Function; +pub use ir::globalvar::GlobalVarData; pub use ir::instructions::{Opcode, InstructionData, VariableArgs, ValueList, ValueListPool}; pub use ir::jumptable::JumpTableData; pub use ir::layout::{Layout, CursorBase, Cursor}; @@ -48,3 +50,6 @@ pub type InstEncodings = EntityMap; /// Code offsets for EBBs. pub type EbbOffsets = EntityMap; + +/// Map of global variables. +pub type GlobalVars = EntityMap; diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index be95456212..af8bcd0c7e 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -57,7 +57,7 @@ use flowgraph::ControlFlowGraph; use ir::entities::AnyEntity; use ir::instructions::{InstructionFormat, BranchInfo, ResolvedConstraint, CallInfo}; use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, StackSlot, - StackSlotKind, Value, Type, Opcode, ValueLoc, ArgumentLoc}; + StackSlotKind, GlobalVar, Value, Type, Opcode, ValueLoc, ArgumentLoc}; use isa::TargetIsa; use std::error as std_error; use std::fmt::{self, Display, Formatter}; @@ -270,6 +270,9 @@ impl<'a> Verifier<'a> { StackStore { stack_slot, .. } => { self.verify_stack_slot(inst, stack_slot)?; } + UnaryGlobalVar { global_var, .. } => { + self.verify_global_var(inst, global_var)?; + } // Exhaustive list so we can't forget to add new formats Nullary { .. } | @@ -328,6 +331,14 @@ impl<'a> Verifier<'a> { } } + fn verify_global_var(&self, inst: Inst, gv: GlobalVar) -> Result { + if !self.func.global_vars.is_valid(gv) { + err!(inst, "invalid global variable {}", gv) + } else { + Ok(()) + } + } + fn verify_value_list(&self, inst: Inst, l: &ValueList) -> Result { if !l.is_valid(&self.func.dfg.value_lists) { err!(inst, "invalid value list reference {:?}", l) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index c342c4f273..d0166a1063 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -49,6 +49,11 @@ fn write_preamble(w: &mut Write, writeln!(w, " {} = {}", ss, func.stack_slots[ss])?; } + for gv in func.global_vars.keys() { + any = true; + writeln!(w, " {} = {}", gv, func.global_vars[gv])?; + } + // Write out all signatures before functions since function declarations can refer to // signatures. for sig in func.dfg.signatures.keys() { @@ -244,6 +249,7 @@ pub fn write_operands(w: &mut Write, UnaryIeee32 { imm, .. } => write!(w, " {}", imm), UnaryIeee64 { imm, .. } => write!(w, " {}", imm), UnaryBool { imm, .. } => write!(w, " {}", imm), + UnaryGlobalVar { global_var, .. } => write!(w, " {}", global_var), Binary { args, .. } => write!(w, " {}, {}", args[0], args[1]), BinaryImm { arg, imm, .. } => write!(w, " {}, {}", arg, imm), Ternary { args, .. } => write!(w, " {}, {}, {}", args[0], args[1], args[2]), @@ -338,7 +344,6 @@ pub fn write_operands(w: &mut Write, write!(w, " {}, %{} -> %{}", arg, src, dst) } } - } } diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index d4392a5bd9..7e5fc4f827 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -37,6 +37,7 @@ pub enum Token<'a> { Value(Value), // v12, v7 Ebb(Ebb), // ebb3 StackSlot(u32), // ss3 + GlobalVar(u32), // gv3 JumpTable(u32), // jt2 FuncRef(u32), // fn2 SigRef(u32), // sig2 @@ -308,6 +309,7 @@ impl<'a> Lexer<'a> { "v" => Value::with_number(number).map(Token::Value), "ebb" => Ebb::with_number(number).map(Token::Ebb), "ss" => Some(Token::StackSlot(number)), + "gv" => Some(Token::GlobalVar(number)), "jt" => Some(Token::JumpTable(number)), "fn" => Some(Token::FuncRef(number)), "sig" => Some(Token::SigRef(number)), diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index fe2aec3e38..0d96861b51 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -11,7 +11,8 @@ use std::{u16, u32}; use std::mem; use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, CallConv, StackSlotData, JumpTable, JumpTableData, Signature, ArgumentType, ArgumentExtension, - ExtFuncData, SigRef, FuncRef, StackSlot, ValueLoc, ArgumentLoc, MemFlags}; + ExtFuncData, SigRef, FuncRef, StackSlot, ValueLoc, ArgumentLoc, MemFlags, + GlobalVar, GlobalVarData}; use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Offset32, Uoffset32, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; @@ -127,6 +128,20 @@ impl<'a> Context<'a> { } } + // Allocate a global variable slot and add a mapping number -> GlobalVar. + fn add_gv(&mut self, number: u32, data: GlobalVarData, loc: &Location) -> Result<()> { + self.map + .def_gv(number, self.function.global_vars.push(data), loc) + } + + // Resolve a reference to a global variable. + fn get_gv(&self, number: u32, loc: &Location) -> Result { + match self.map.get_gv(number) { + Some(gv) => Ok(gv), + None => err!(loc, "undefined stack slot ss{}", number), + } + } + // Allocate a new signature and add a mapping number -> SigRef. fn add_sig(&mut self, number: u32, data: Signature, loc: &Location) -> Result<()> { self.map @@ -228,6 +243,18 @@ impl<'a> Context<'a> { } } + // Rewrite base references in `deref` globals. Other `GlobalVar` references are already + // resolved. + for gv in self.function.global_vars.keys() { + let loc = gv.into(); + match self.function.global_vars[gv] { + GlobalVarData::Deref { ref mut base, .. } => { + self.map.rewrite_gv(base, loc)?; + } + _ => {} + } + } + Ok(()) } } @@ -344,6 +371,16 @@ impl<'a> Parser<'a> { } } + // Match and consume a global variable reference. + fn match_gv(&mut self, err_msg: &str) -> Result { + if let Some(Token::GlobalVar(gv)) = self.token() { + self.consume(); + Ok(gv) + } else { + err!(self.loc, err_msg) + } + } + // Match and consume a function reference. fn match_fn(&mut self, err_msg: &str) -> Result { if let Some(Token::FuncRef(fnref)) = self.token() { @@ -903,6 +940,11 @@ impl<'a> Parser<'a> { self.parse_stack_slot_decl() .and_then(|(num, dat)| ctx.add_ss(num, dat, &loc)) } + Some(Token::GlobalVar(..)) => { + self.gather_comments(ctx.function.global_vars.next_key()); + self.parse_global_var_decl() + .and_then(|(num, dat)| ctx.add_gv(num, dat, &self.loc)) + } Some(Token::SigRef(..)) => { self.gather_comments(ctx.function.dfg.signatures.next_key()); self.parse_signature_decl(ctx.unique_isa) @@ -959,6 +1001,40 @@ impl<'a> Parser<'a> { Ok((number, data)) } + // Parse a global variable decl. + // + // global-var-decl ::= * GlobalVar(gv) "=" global-var-desc + // global-var-desc ::= "vmctx" offset32 + // | "deref" "(" GlobalVar(base) ")" offset32 + // + fn parse_global_var_decl(&mut self) -> Result<(u32, GlobalVarData)> { + let number = self.match_gv("expected global variable number: gv«n»")?; + self.match_token(Token::Equal, "expected '=' in global variable declaration")?; + + let data = match self.match_any_identifier("expected global variable kind")? { + "vmctx" => { + let offset = self.optional_offset32()?; + GlobalVarData::VmCtx { offset } + } + "deref" => { + self.match_token(Token::LPar, "expected '(' in 'deref' global variable decl")?; + let base_num = self.match_gv("expected global variable: gv«n»")?; + // The base global variable may not have been declared yet, so create a fake + // reference using the source number. We'll rewrite these later. + let base = match GlobalVar::with_number(base_num) { + Some(gv) => gv, + None => return err!(self.loc, "Invalid global variable number for deref base"), + }; + self.match_token(Token::RPar, "expected ')' in 'deref' global variable decl")?; + let offset = self.optional_offset32()?; + GlobalVarData::Deref { base, offset } + } + other => return err!(self.loc, "Unknown global variable kind '{}'", other), + }; + + Ok((number, data)) + } + // Parse a signature decl. // // signature-decl ::= SigRef(sigref) "=" signature @@ -1517,6 +1593,13 @@ impl<'a> Parser<'a> { imm: self.match_bool("expected immediate boolean operand")?, } } + InstructionFormat::UnaryGlobalVar => { + InstructionData::UnaryGlobalVar { + opcode, + global_var: self.match_gv("expected global variable") + .and_then(|num| ctx.get_gv(num, &self.loc))?, + } + } InstructionFormat::Binary => { let lhs = self.match_value("expected SSA value first operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index d590a5220f..b085a78c1b 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -7,11 +7,12 @@ //! The `SourceMap` struct defined in this module makes the same mapping available to parser //! clients. -use std::collections::HashMap; -use cretonne::ir::{StackSlot, JumpTable, Ebb, Value, SigRef, FuncRef}; +use cretonne::entity_ref::EntityRef; use cretonne::ir::entities::AnyEntity; +use cretonne::ir::{StackSlot, GlobalVar, JumpTable, Ebb, Value, SigRef, FuncRef}; use error::{Result, Location}; use lexer::split_entity_name; +use std::collections::HashMap; /// Mapping from source entity names to entity references that are valid in the parsed function. #[derive(Debug)] @@ -19,6 +20,7 @@ pub struct SourceMap { values: HashMap, // vNN ebbs: HashMap, // ebbNN stack_slots: HashMap, // ssNN + global_vars: HashMap, // gvNN signatures: HashMap, // sigNN functions: HashMap, // fnNN jump_tables: HashMap, // jtNN @@ -44,6 +46,11 @@ impl SourceMap { self.stack_slots.get(&src_num).cloned() } + /// Look up a global variable entity by its source number. + pub fn get_gv(&self, src_num: u32) -> Option { + self.global_vars.get(&src_num).cloned() + } + /// Look up a signature entity by its source number. pub fn get_sig(&self, src_num: u32) -> Option { self.signatures.get(&src_num).cloned() @@ -74,6 +81,7 @@ impl SourceMap { .map(AnyEntity::Ebb) } "ss" => self.get_ss(num).map(AnyEntity::StackSlot), + "gv" => self.get_gv(num).map(AnyEntity::GlobalVar), "sig" => self.get_sig(num).map(AnyEntity::SigRef), "fn" => self.get_fn(num).map(AnyEntity::FuncRef), "jt" => self.get_jt(num).map(AnyEntity::JumpTable), @@ -124,6 +132,21 @@ impl SourceMap { } Ok(()) } + + /// Rewrite a `GlobalVar` reference. + pub fn rewrite_gv(&self, gv: &mut GlobalVar, loc: AnyEntity) -> Result<()> { + match self.get_gv(gv.index() as u32) { + Some(new) => { + *gv = new; + Ok(()) + } + None => { + err!(self.location(loc).unwrap_or_default(), + "undefined reference: {}", + gv) + } + } + } } @@ -137,6 +160,7 @@ pub trait MutableSourceMap { fn def_value(&mut self, src: Value, entity: Value, loc: &Location) -> Result<()>; fn def_ebb(&mut self, src: Ebb, entity: Ebb, loc: &Location) -> Result<()>; fn def_ss(&mut self, src_num: u32, entity: StackSlot, loc: &Location) -> Result<()>; + fn def_gv(&mut self, src_num: u32, entity: GlobalVar, loc: &Location) -> Result<()>; fn def_sig(&mut self, src_num: u32, entity: SigRef, loc: &Location) -> Result<()>; fn def_fn(&mut self, src_num: u32, entity: FuncRef, loc: &Location) -> Result<()>; fn def_jt(&mut self, src_num: u32, entity: JumpTable, loc: &Location) -> Result<()>; @@ -152,6 +176,7 @@ impl MutableSourceMap for SourceMap { values: HashMap::new(), ebbs: HashMap::new(), stack_slots: HashMap::new(), + global_vars: HashMap::new(), signatures: HashMap::new(), functions: HashMap::new(), jump_tables: HashMap::new(), @@ -183,6 +208,14 @@ impl MutableSourceMap for SourceMap { } } + fn def_gv(&mut self, src_num: u32, entity: GlobalVar, loc: &Location) -> Result<()> { + if self.global_vars.insert(src_num, entity).is_some() { + err!(loc, "duplicate global variable: gv{}", src_num) + } else { + self.def_entity(entity.into(), loc) + } + } + fn def_sig(&mut self, src_num: u32, entity: SigRef, loc: &Location) -> Result<()> { if self.signatures.insert(src_num, entity).is_some() { err!(loc, "duplicate signature: sig{}", src_num) From 5566c99dba285e2a6691f05452eea3261dbff75e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 17 Aug 2017 15:11:35 -0700 Subject: [PATCH 0970/3084] Check for global variable deref cycles in the verifier. --- cranelift/filetests/verifier/memory.cton | 16 +++++++++++++ lib/cretonne/src/sparse_map.rs | 5 ++++ lib/cretonne/src/verifier/mod.rs | 29 ++++++++++++++++++++++++ 3 files changed, 50 insertions(+) create mode 100644 cranelift/filetests/verifier/memory.cton diff --git a/cranelift/filetests/verifier/memory.cton b/cranelift/filetests/verifier/memory.cton new file mode 100644 index 0000000000..46bf18d636 --- /dev/null +++ b/cranelift/filetests/verifier/memory.cton @@ -0,0 +1,16 @@ +test verifier + +function %deref_cycle() { + gv1 = deref(gv2)-32 ; error: deref cycle: [gv0, gv1] + gv2 = deref(gv1) + +ebb1: + return +} + +function %self_cycle() { + gv0 = deref(gv0)-32 ; error: deref cycle: [gv0] + +ebb1: + return +} diff --git a/lib/cretonne/src/sparse_map.rs b/lib/cretonne/src/sparse_map.rs index b1e844b618..c1a8bf5480 100644 --- a/lib/cretonne/src/sparse_map.rs +++ b/lib/cretonne/src/sparse_map.rs @@ -184,6 +184,11 @@ impl SparseMap pub fn values(&self) -> slice::Iter { self.dense.iter() } + + /// Get the values as a slice. + pub fn as_slice(&self) -> &[V] { + self.dense.as_slice() + } } /// Iterating over the elements of a set. diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index af8bcd0c7e..0995449362 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -40,6 +40,10 @@ //! - All return instructions must have return value operands matching the current //! function signature. //! +//! Global variables +//! +//! - Detect cycles in deref(base) declarations. +//! //! TODO: //! Ad hoc checking //! @@ -52,13 +56,16 @@ //! - Swizzle and shuffle instructions take a variable number of lane arguments. The number //! of arguments must match the destination type, and the lane indexes must be in range. +use dbg::DisplayList; use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; +use ir; use ir::entities::AnyEntity; use ir::instructions::{InstructionFormat, BranchInfo, ResolvedConstraint, CallInfo}; use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, StackSlot, StackSlotKind, GlobalVar, Value, Type, Opcode, ValueLoc, ArgumentLoc}; use isa::TargetIsa; +use sparse_map::SparseSet; use std::error as std_error; use std::fmt::{self, Display, Formatter}; use std::result; @@ -150,6 +157,27 @@ impl<'a> Verifier<'a> { } } + // Check for cycles in the global variable declarations. + fn verify_global_vars(&self) -> Result { + let mut seen = SparseSet::new(); + + for gv in self.func.global_vars.keys() { + seen.clear(); + seen.insert(gv); + + let mut cur = gv; + while let &ir::GlobalVarData::Deref { base, .. } = &self.func.global_vars[cur] { + if seen.insert(base).is_some() { + return err!(gv, "deref cycle: {}", DisplayList(seen.as_slice())); + } + + cur = base; + } + } + + Ok(()) + } + fn ebb_integrity(&self, ebb: Ebb, inst: Inst) -> Result { let is_terminator = self.func.dfg[inst].opcode().is_terminator(); @@ -845,6 +873,7 @@ impl<'a> Verifier<'a> { } pub fn run(&self) -> Result { + self.verify_global_vars()?; self.typecheck_entry_block_arguments()?; for ebb in self.func.layout.ebbs() { for inst in self.func.layout.ebb_insts(ebb) { From 4b94ea21ed6d299bbcdc288ecc08180a1eba5877 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 17 Aug 2017 15:45:21 -0700 Subject: [PATCH 0971/3084] Switch to a FuncCursor in the top-level legalizer loop. Stop passing Cursor references to legalizer functions. Give them the whole &mut Function instead. Given the whole Function reference, these functions can create their own cursors. This lets legalizer actions access other Function data structures like the global variables. --- lib/cretonne/meta/gen_legalizer.py | 33 +++++++++++++------------- lib/cretonne/src/isa/mod.rs | 6 ++--- lib/cretonne/src/legalizer/boundary.rs | 28 ++++++++-------------- lib/cretonne/src/legalizer/mod.rs | 30 ++++++++++------------- 4 files changed, 43 insertions(+), 54 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 7189695a71..b9c125681b 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -353,16 +353,12 @@ def gen_xform_group(xgrp, fmt, type_sets): fmt.doc_comment("Legalize the instruction pointed to by `pos`.") fmt.line('#[allow(unused_variables,unused_assignments)]') with fmt.indented( - 'pub fn {}(dfg: &mut ir::DataFlowGraph, ' - 'cfg: &mut ::flowgraph::ControlFlowGraph, ' - 'pos: &mut ir::Cursor) -> ' + 'pub fn {}(inst: ir::Inst, ' + 'func: &mut ir::Function, ' + 'cfg: &mut ::flowgraph::ControlFlowGraph) -> ' 'bool {{'.format(xgrp.name), '}'): fmt.line('use ir::{InstBuilder, CursorBase};') - # Gen the instruction to be legalized. The cursor we're passed must be - # pointing at an instruction. - fmt.line('let inst = pos.current_inst().expect("need instruction");') - # Group the xforms by opcode so we can generate a big switch. # Preserve ordering. xforms = defaultdict(list) # type: DefaultDict[str, List[XForm]] @@ -370,18 +366,23 @@ def gen_xform_group(xgrp, fmt, type_sets): inst = xform.src.rtl[0].expr.inst xforms[inst.camel_name].append(xform) - with fmt.indented('match dfg[inst].opcode() {', '}'): - for camel_name in sorted(xforms.keys()): - with fmt.indented( - 'ir::Opcode::{} => {{'.format(camel_name), '}'): - for xform in xforms[camel_name]: - gen_xform(xform, fmt, type_sets) - # We'll assume there are uncovered opcodes. - fmt.line('_ => {},') + with fmt.indented('{', '}'): + fmt.line( + 'let pos = &mut ir::Cursor::new(&mut func.layout)' + '.at_inst(inst);') + fmt.line('let dfg = &mut func.dfg;') + with fmt.indented('match dfg[inst].opcode() {', '}'): + for camel_name in sorted(xforms.keys()): + with fmt.indented( + 'ir::Opcode::{} => {{'.format(camel_name), '}'): + for xform in xforms[camel_name]: + gen_xform(xform, fmt, type_sets) + # We'll assume there are uncovered opcodes. + fmt.line('_ => {},') # If we fall through, nothing was expanded. Call the chain if any. if xgrp.chain: - fmt.format('{}(dfg, cfg, pos)', xgrp.chain.rust_name()) + fmt.format('{}(inst, func, cfg)', xgrp.chain.rust_name()) else: fmt.line('false') diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 08c45dd8b8..28a7c9d615 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -138,9 +138,9 @@ impl settings::Configurable for Builder { /// legalize it? /// /// The `Encodings` iterator returns a legalization function to call. -pub type Legalize = fn(&mut ir::DataFlowGraph, - &mut flowgraph::ControlFlowGraph, - &mut ir::Cursor) +pub type Legalize = fn(ir::Inst, + &mut ir::Function, + &mut flowgraph::ControlFlowGraph) -> bool; /// Methods that are specialized to a target ISA. diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 7d248c0cfa..17ff37c8f7 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -452,18 +452,13 @@ fn legalize_inst_arguments(dfg: &mut DataFlowGraph, /// original return values. The call's result values will be adapted to match the new signature. /// /// Returns `true` if any instructions were inserted. -pub fn handle_call_abi(dfg: &mut DataFlowGraph, - locations: &mut ValueLocations, - stack_slots: &mut StackSlots, - cfg: &ControlFlowGraph, - pos: &mut Cursor) - -> bool { - let mut inst = pos.current_inst() - .expect("Cursor must point to a call instruction"); +pub fn handle_call_abi(mut inst: Inst, func: &mut Function, cfg: &ControlFlowGraph) -> bool { + let dfg = &mut func.dfg; + let pos = &mut Cursor::new(&mut func.layout).at_inst(inst); // Start by checking if the argument types already match the signature. let sig_ref = match check_call_signature(dfg, inst) { - Ok(_) => return spill_call_arguments(dfg, locations, stack_slots, pos), + Ok(_) => return spill_call_arguments(dfg, &mut func.locations, &mut func.stack_slots, pos), Err(s) => s, }; @@ -489,22 +484,19 @@ pub fn handle_call_abi(dfg: &mut DataFlowGraph, // Go back and insert spills for any stack arguments. pos.goto_inst(inst); - spill_call_arguments(dfg, locations, stack_slots, pos); + spill_call_arguments(dfg, &mut func.locations, &mut func.stack_slots, pos); // Yes, we changed stuff. true } -/// Insert ABI conversion code before and after the return instruction at `pos`. +/// Insert ABI conversion code before and after the return instruction at `inst`. /// /// Return `true` if any instructions were inserted. -pub fn handle_return_abi(dfg: &mut DataFlowGraph, - cfg: &ControlFlowGraph, - pos: &mut Cursor, - sig: &Signature) - -> bool { - let inst = pos.current_inst() - .expect("Cursor must point to a return instruction"); +pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph) -> bool { + let dfg = &mut func.dfg; + let sig = &mut func.signature; + let pos = &mut Cursor::new(&mut func.layout).at_inst(inst); // Check if the returned types already match the signature. if check_return_signature(dfg, inst, sig) { diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 9cc37dd896..f4db515c6b 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -13,9 +13,10 @@ //! The legalizer does not deal with register allocation constraints. These constraints are derived //! from the encoding recipes, and solved later by the register allocator. +use cursor::{Cursor, FuncCursor}; use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; -use ir::{self, Function, Cursor, CursorBase}; +use ir; use ir::condcodes::IntCC; use isa::TargetIsa; use bitset::BitSet; @@ -28,7 +29,7 @@ mod split; /// - Transform any instructions that don't have a legal representation in `isa`. /// - Fill out `func.encodings`. /// -pub fn legalize_function(func: &mut Function, +pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, domtree: &DominatorTree, isa: &TargetIsa) { @@ -36,7 +37,7 @@ pub fn legalize_function(func: &mut Function, func.encodings.resize(func.dfg.num_insts()); - let mut pos = Cursor::new(&mut func.layout); + let mut pos = FuncCursor::new(func); // Process EBBs in a reverse post-order. This minimizes the number of split instructions we // need. @@ -48,36 +49,32 @@ pub fn legalize_function(func: &mut Function, let mut prev_pos = pos.position(); while let Some(inst) = pos.next_inst() { - let opcode = func.dfg[inst].opcode(); + let opcode = pos.func.dfg[inst].opcode(); // Check for ABI boundaries that need to be converted to the legalized signature. - if opcode.is_call() && - boundary::handle_call_abi(&mut func.dfg, - &mut func.locations, - &mut func.stack_slots, - cfg, - &mut pos) { + if opcode.is_call() && boundary::handle_call_abi(inst, pos.func, cfg) { // Go back and legalize the inserted argument conversion instructions. pos.set_position(prev_pos); continue; } - if opcode.is_return() && - boundary::handle_return_abi(&mut func.dfg, cfg, &mut pos, &func.signature) { + if opcode.is_return() && boundary::handle_return_abi(inst, pos.func, cfg) { // Go back and legalize the inserted return value conversion instructions. pos.set_position(prev_pos); continue; } if opcode.is_branch() { - split::simplify_branch_arguments(&mut func.dfg, inst); + split::simplify_branch_arguments(&mut pos.func.dfg, inst); } - match isa.encode(&func.dfg, &func.dfg[inst], func.dfg.ctrl_typevar(inst)) { - Ok(encoding) => *func.encodings.ensure(inst) = encoding, + match isa.encode(&pos.func.dfg, + &pos.func.dfg[inst], + pos.func.dfg.ctrl_typevar(inst)) { + Ok(encoding) => *pos.func.encodings.ensure(inst) = encoding, Err(action) => { // We should transform the instruction into legal equivalents. - let changed = action(&mut func.dfg, cfg, &mut pos); + let changed = action(inst, pos.func, cfg); // If the current instruction was replaced, we need to double back and revisit // the expanded sequence. This is both to assign encodings and possible to // expand further. @@ -94,7 +91,6 @@ pub fn legalize_function(func: &mut Function, prev_pos = pos.position(); } } - func.encodings.resize(func.dfg.num_insts()); } // Include legalization patterns that were generated by `gen_legalizer.py` from the `XForms` in From 5dba00b7618b78a71a9a7f875c842881fff963ea Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 17 Aug 2017 18:33:53 -0700 Subject: [PATCH 0972/3084] Add support for custom legalization actions. The custom_legalize() method on XFormGroup can be used to call a custom function to legalize specific opcodes. This will be used shortly to expand global_addr which has an expansion that depends on the details of the global variable being referenced. --- lib/cretonne/meta/cdsl/xform.py | 18 +++++++++++++++++- lib/cretonne/meta/gen_legalizer.py | 21 ++++++++++++++++----- 2 files changed, 33 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index 261a70a4af..bad15e0245 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -4,6 +4,7 @@ Instruction transformations. from __future__ import absolute_import from .ast import Def, Var, Apply from .ti import ti_xform, TypeEnv, get_type_env, TypeConstraint +from collections import OrderedDict from functools import reduce try: @@ -12,7 +13,7 @@ try: from .ast import Expr, VarAtomMap # noqa from .isa import TargetISA # noqa from .typevar import TypeVar # noqa - from .instructions import ConstrList # noqa + from .instructions import ConstrList, Instruction # noqa DefApply = Union[Def, Apply] except ImportError: pass @@ -370,6 +371,7 @@ class XFormGroup(object): def __init__(self, name, doc, isa=None, chain=None): # type: (str, str, TargetISA, XFormGroup) -> None self.xforms = list() # type: List[XForm] + self.custom = OrderedDict() # type: OrderedDict[Instruction, str] self.name = name self.__doc__ = doc self.isa = isa @@ -405,3 +407,17 @@ class XFormGroup(object): xform = XForm(Rtl(src), dst) xform.verify_legalize() self.xforms.append(xform) + + def custom_legalize(self, inst, funcname): + # type: (Instruction, str) -> None + """ + Add a custom legalization action for `inst`. + + The `funcname` parameter is the fully qualified name of a Rust function + which takes the same arguments as the `isa::Legalize` actions. + + The custom function will be called to legalize `inst` and any return + value is ignored. + """ + assert inst not in self.custom, "Duplicate custom_legalize" + self.custom[inst] = funcname diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index b9c125681b..f3e5067420 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -367,16 +367,27 @@ def gen_xform_group(xgrp, fmt, type_sets): xforms[inst.camel_name].append(xform) with fmt.indented('{', '}'): - fmt.line( - 'let pos = &mut ir::Cursor::new(&mut func.layout)' - '.at_inst(inst);') - fmt.line('let dfg = &mut func.dfg;') - with fmt.indented('match dfg[inst].opcode() {', '}'): + with fmt.indented('match func.dfg[inst].opcode() {', '}'): for camel_name in sorted(xforms.keys()): with fmt.indented( 'ir::Opcode::{} => {{'.format(camel_name), '}'): + fmt.line( + 'let pos = &mut ' + 'ir::Cursor::new(&mut func.layout)' + '.at_inst(inst);') + fmt.line('let dfg = &mut func.dfg;') for xform in xforms[camel_name]: gen_xform(xform, fmt, type_sets) + + # Emit the custom transforms. The Rust compiler will complain + # about any overlap with the normal xforms. + for inst, funcname in xgrp.custom.items(): + with fmt.indented( + 'ir::Opcode::{} => {{' + .format(inst.camel_name), '}'): + fmt.format('{}(inst, func, cfg);', funcname) + fmt.line('return true;') + # We'll assume there are uncovered opcodes. fmt.line('_ => {},') From f2ebabaf5fd748feab38023b45ee75b334f8b137 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 18 Aug 2017 09:08:41 -0700 Subject: [PATCH 0973/3084] Custom legalization for global_addr. The code to compute the address of a global variable depends on the kind of variable, so custom legalization is required. - Add a legalizer::globalvar module which exposes an expand_global_addr() function. This module is likely to grow as we add more types of global variables. - Add a ArgumentPurpose::VMContext enumerator. This is used to represent special 'vmctx' arguments that are used as base pointers for vmctx globals. --- cranelift/docs/langref.rst | 2 +- .../filetests/isa/intel/legalize-memory.cton | 29 +++++++++ lib/cretonne/meta/base/legalize.py | 4 ++ lib/cretonne/src/ir/extfunc.rs | 12 +++- lib/cretonne/src/legalizer/boundary.rs | 30 +++++++--- lib/cretonne/src/legalizer/globalvar.rs | 59 +++++++++++++++++++ lib/cretonne/src/legalizer/mod.rs | 3 + 7 files changed, 127 insertions(+), 12 deletions(-) create mode 100644 cranelift/filetests/isa/intel/legalize-memory.cton create mode 100644 lib/cretonne/src/legalizer/globalvar.rs diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index d1d353186a..218e1a5e48 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -373,7 +373,7 @@ calling convention: retlist : arglist arg : type [argext] [argspecial] argext : "uext" | "sext" - argspecial: "sret" | "link" | "fp" | "csr" + argspecial: "sret" | "link" | "fp" | "csr" | "vmctx" callconv : `string` Arguments and return values have flags whose meaning is mostly target diff --git a/cranelift/filetests/isa/intel/legalize-memory.cton b/cranelift/filetests/isa/intel/legalize-memory.cton new file mode 100644 index 0000000000..ade217c84d --- /dev/null +++ b/cranelift/filetests/isa/intel/legalize-memory.cton @@ -0,0 +1,29 @@ +; Test the legalization of memory objects. +test legalizer +set is_64bit +isa intel + +; regex: V=v\d+ + +function %vmctx(i64 vmctx) -> i64 { + gv1 = vmctx-16 + +ebb1(v1: i64): + v2 = global_addr.i64 gv1 + ; check: $v2 = iadd_imm $v1, -16 + return v2 + ; check: return $v2 +} + +function %deref(i64 vmctx) -> i64 { + gv1 = vmctx-16 + gv2 = deref(gv1)+32 + +ebb1(v1: i64): + v2 = global_addr.i64 gv2 + ; check: $(a1=$V) = iadd_imm $v1, -16 + ; check: $(p1=$V) = load.i64 $a1 + ; check: $v2 = iadd_imm $p1, 32 + return v2 + ; check: return $v2 +} diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index c250d53ce2..8c98468518 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -8,6 +8,7 @@ instructions that are legal. """ from __future__ import absolute_import from .immediates import intcc +from . import instructions as insts from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm from .instructions import isub, isub_bin, isub_bout, isub_borrow from .instructions import band, bor, bxor, isplit, iconcat @@ -43,6 +44,9 @@ expand = XFormGroup('expand', """ operating on the same types as the original instructions. """) +# Custom expansions for memory objects. +expand.custom_legalize(insts.global_addr, 'expand_global_addr') + x = Var('x') y = Var('y') a = Var('a') diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 16211a4d83..9aad88ef13 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -240,10 +240,16 @@ pub enum ArgumentPurpose { /// Some calling conventions have registers that must be saved by the callee. These registers /// are represented as `CalleeSaved` arguments and return values. CalleeSaved, + + /// A VM context pointer. + /// + /// This is a pointer to a context struct containing details about the current sandbox. It is + /// used as a base pointer for `vmctx` global variables. + VMContext, } /// Text format names of the `ArgumentPurpose` variants. -static PURPOSE_NAMES: [&str; 5] = ["normal", "sret", "link", "fp", "csr"]; +static PURPOSE_NAMES: [&str; 6] = ["normal", "sret", "link", "fp", "csr", "vmctx"]; impl fmt::Display for ArgumentPurpose { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { @@ -260,6 +266,7 @@ impl FromStr for ArgumentPurpose { "link" => Ok(ArgumentPurpose::Link), "fp" => Ok(ArgumentPurpose::FramePointer), "csr" => Ok(ArgumentPurpose::CalleeSaved), + "vmctx" => Ok(ArgumentPurpose::VMContext), _ => Err(()), } } @@ -343,7 +350,8 @@ mod tests { ArgumentPurpose::StructReturn, ArgumentPurpose::Link, ArgumentPurpose::FramePointer, - ArgumentPurpose::CalleeSaved]; + ArgumentPurpose::CalleeSaved, + ArgumentPurpose::VMContext]; for (&e, &n) in all_purpose.iter().zip(PURPOSE_NAMES.iter()) { assert_eq!(e.to_string(), n); assert_eq!(Ok(e), n.parse()); diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 17ff37c8f7..6977c9e7bc 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -56,6 +56,7 @@ pub fn legalize_signatures(func: &mut Function, isa: &TargetIsa) { fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { let mut has_sret = false; let mut has_link = false; + let mut has_vmctx = false; // Insert position for argument conversion code. // We want to insert instructions before the first instruction in the entry block. @@ -86,6 +87,10 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { assert!(!has_sret, "Multiple sret arguments found"); has_sret = true; } + ArgumentPurpose::VMContext => { + assert!(!has_vmctx, "Multiple vmctx arguments found"); + has_vmctx = true; + } _ => panic!("Unexpected special-purpose arg {}", abi_types[abi_arg]), } abi_arg += 1; @@ -134,7 +139,12 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { assert!(!has_sret, "Multiple sret arguments found"); has_sret = true; } + ArgumentPurpose::VMContext => { + assert!(!has_vmctx, "Multiple vmctx arguments found"); + has_vmctx = true; + } } + // Just create entry block values to match here. We will use them in `handle_return_abi()` // below. func.dfg.append_ebb_arg(entry, arg.value_type); @@ -503,14 +513,15 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph return false; } - // Count the special-purpose return values (`link` and `sret`) that were appended to the - // legalized signature. + // Count the special-purpose return values (`link`, `sret`, and `vmctx`) that were appended to + // the legalized signature. let special_args = sig.return_types .iter() .rev() .take_while(|&rt| { rt.purpose == ArgumentPurpose::Link || - rt.purpose == ArgumentPurpose::StructReturn + rt.purpose == ArgumentPurpose::StructReturn || + rt.purpose == ArgumentPurpose::VMContext }) .count(); @@ -522,8 +533,8 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph |_, abi_arg| sig.return_types[abi_arg]); assert_eq!(dfg.inst_variable_args(inst).len(), abi_args); - // Append special return arguments for any `sret` and `link` return values added to the - // legalized signature. These values should simply be propagated from the entry block + // Append special return arguments for any `sret`, `link`, and `vmctx` return values added to + // the legalized signature. These values should simply be propagated from the entry block // arguments. if special_args > 0 { dbg!("Adding {} special-purpose arguments to {}", @@ -533,13 +544,14 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph for arg in &sig.return_types[abi_args..] { match arg.purpose { ArgumentPurpose::Link | - ArgumentPurpose::StructReturn => {} + ArgumentPurpose::StructReturn | + ArgumentPurpose::VMContext => {} ArgumentPurpose::Normal => panic!("unexpected return value {}", arg), _ => panic!("Unsupported special purpose return value {}", arg), } - // A `link` or `sret` return value can only appear in a signature that has a unique - // matching argument. They are appended at the end, so search the signature from the - // end. + // A `link`/`sret`/`vmctx` return value can only appear in a signature that has a + // unique matching argument. They are appended at the end, so search the signature from + // the end. let idx = sig.argument_types .iter() .rposition(|t| t.purpose == arg.purpose) diff --git a/lib/cretonne/src/legalizer/globalvar.rs b/lib/cretonne/src/legalizer/globalvar.rs new file mode 100644 index 0000000000..308c8df9e5 --- /dev/null +++ b/lib/cretonne/src/legalizer/globalvar.rs @@ -0,0 +1,59 @@ +//! Legalization of global variables. +//! +//! This module exports the `expand_global_addr` function which transforms a `global_addr` +//! instruction into code that depends on the kind of global variable referenced. + +use cursor::{Cursor, FuncCursor}; +use flowgraph::ControlFlowGraph; +use ir::{self, InstBuilder}; + +/// Expand a `global_addr` instruction according to the definition of the global variable. +pub fn expand_global_addr(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlowGraph) { + // Unpack the instruction. + let gv = match &func.dfg[inst] { + &ir::InstructionData::UnaryGlobalVar { opcode, global_var } => { + assert_eq!(opcode, ir::Opcode::GlobalAddr); + global_var + } + _ => panic!("Wanted global_addr: {}", func.dfg.display_inst(inst, None)), + }; + + match func.global_vars[gv] { + ir::GlobalVarData::VmCtx { offset } => vmctx_addr(inst, func, offset.into()), + ir::GlobalVarData::Deref { base, offset } => deref_addr(inst, func, base, offset.into()), + } +} + +/// Expand a `global_addr` instruction for a vmctx global. +fn vmctx_addr(inst: ir::Inst, func: &mut ir::Function, offset: i64) { + // Find the incoming `vmctx` function argument. Start searching from the back since the special + // arguments are appended by signature legalization. + // + // This argument must exist; `vmctx` global variables can not be used in functions with calling + // conventions that don't add a `vmctx` argument. + let argidx = func.signature + .argument_types + .iter() + .rposition(|abi| abi.purpose == ir::ArgumentPurpose::VMContext) + .expect("Need vmctx argument for vmctx global"); + + // Get the value representing the `vmctx` argument. + let vmctx = func.dfg.ebb_args(func.layout.entry_block().unwrap())[argidx]; + + // Simply replace the `global_addr` instruction with an `iadd_imm`, reusing the result value. + func.dfg.replace(inst).iadd_imm(vmctx, offset); +} + +/// Expand a `global_addr` instruction for a deref global. +fn deref_addr(inst: ir::Inst, func: &mut ir::Function, base: ir::GlobalVar, offset: i64) { + // We need to load a pointer from the `base` global variable, so insert a new `global_addr` + // instruction. This depends on the iterative legalization loop. Note that the IL verifier + // detects any cycles in the `deref` globals. + let ptr_ty = func.dfg.value_type(func.dfg.first_result(inst)); + let mut pos = FuncCursor::new(func).at_inst(inst); + + let base_addr = pos.ins().global_addr(ptr_ty, base); + // TODO: We could probably set both `notrap` and `aligned` on this load instruction. + let base_ptr = pos.ins().load(ptr_ty, ir::MemFlags::new(), base_addr, 0); + pos.func.dfg.replace(inst).iadd_imm(base_ptr, offset); +} diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index f4db515c6b..9e96c61e5d 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -22,8 +22,11 @@ use isa::TargetIsa; use bitset::BitSet; mod boundary; +mod globalvar; mod split; +use self::globalvar::expand_global_addr; + /// Legalize `func` for `isa`. /// /// - Transform any instructions that don't have a legal representation in `isa`. From 6bcb24b3a6f2a36db5feb3153e2c1a5fc37c1cab Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 18 Aug 2017 13:06:37 -0700 Subject: [PATCH 0974/3084] Add a uimm32 operand kind. We already have a uoffset32 operand kind, but that prints as an offset with a permanent sign. --- lib/cretonne/meta/base/immediates.py | 3 ++ lib/cretonne/src/ir/immediates.rs | 48 ++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index 1defe46f6d..2cb73476e5 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -17,6 +17,9 @@ imm64 = ImmediateKind('imm64', 'A 64-bit immediate integer.') #: immediate bit counts on shift instructions. uimm8 = ImmediateKind('uimm8', 'An 8-bit immediate unsigned integer.') +#: An unsigned 32-bit immediate integer operand. +uimm32 = ImmediateKind('uimm32', 'A 32-bit immediate unsigned integer.') + #: A 32-bit immediate signed offset. #: #: This is used to represent an immediate address offset in load/store diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index c525305a19..8800fb2229 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -151,6 +151,54 @@ impl FromStr for Imm64 { /// This is used to indicate lane indexes typically. pub type Uimm8 = u8; +/// A 32-bit unsigned integer immediate operand. +/// +/// This is used to represent sizes of memory objects. +#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)] +pub struct Uimm32(u32); + +impl Into for Uimm32 { + fn into(self) -> u32 { + self.0 + } +} + +impl Into for Uimm32 { + fn into(self) -> i64 { + self.0 as i64 + } +} + +impl From for Uimm32 { + fn from(x: u32) -> Self { + Uimm32(x) + } +} + +impl Display for Uimm32 { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + if self.0 < 10_000 { + write!(f, "{}", self.0) + } else { + write_hex(self.0 as i64, f) + } + + } +} + +impl FromStr for Uimm32 { + type Err = &'static str; + + // Parse a decimal or hexadecimal `Uimm32`, formatted as above. + fn from_str(s: &str) -> Result { + parse_i64(s).and_then(|x| if 0 <= x && x <= u32::MAX as i64 { + Ok(Uimm32(x as u32)) + } else { + Err("Uimm32 out of range") + }) + } +} + /// 32-bit signed immediate offset. /// /// This is used to encode an immediate offset for load/store instructions. All supported ISAs have From c7b9bc1abfc900c357186524b7b0a29002d62d3f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 18 Aug 2017 14:14:23 -0700 Subject: [PATCH 0975/3084] Prefer to use qualified type names in generated code. Emit type names like ir::Foo instead of just Foo to avoid very long manual use declarations in files including generated code. --- lib/cretonne/meta/base/immediates.py | 10 ++++++---- lib/cretonne/meta/cdsl/operands.py | 4 +++- lib/cretonne/meta/gen_instr.py | 25 +++++++++++++----------- lib/cretonne/src/ir/builder.rs | 8 +++----- lib/cretonne/src/isa/riscv/enc_tables.rs | 1 - lib/cretonne/src/legalizer/mod.rs | 1 - 6 files changed, 26 insertions(+), 23 deletions(-) diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index 2cb73476e5..048a82dd12 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -62,7 +62,8 @@ boolean = ImmediateKind('bool', 'An immediate boolean.', intcc = ImmediateKind( 'intcc', 'An integer comparison condition code.', - default_member='cond', rust_type='IntCC', + default_member='cond', + rust_type='ir::condcodes::IntCC', values={ 'eq': 'Equal', 'ne': 'NotEqual', @@ -83,7 +84,8 @@ intcc = ImmediateKind( floatcc = ImmediateKind( 'floatcc', 'A floating point comparison condition code.', - default_member='cond', rust_type='FloatCC', + default_member='cond', + rust_type='ir::condcodes::FloatCC', values={ 'ord': 'Ordered', 'uno': 'Unordered', @@ -105,10 +107,10 @@ floatcc = ImmediateKind( memflags = ImmediateKind( 'memflags', 'Memory operation flags', - default_member='flags', rust_type='MemFlags') + default_member='flags', rust_type='ir::MemFlags') #: A register unit in the current target ISA. regunit = ImmediateKind( 'regunit', 'A register unit in the target ISA', - rust_type='RegUnit') + rust_type='isa::RegUnit') diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index c80632f22a..ea59de5b3e 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -32,7 +32,7 @@ class OperandKind(object): self.default_member = default_member # The camel-cased name of an operand kind is also the Rust type used to # represent it. - self.rust_type = rust_type or camel_case(name) + self.rust_type = rust_type or ('ir::' + camel_case(name)) def __str__(self): # type: () -> str @@ -82,6 +82,8 @@ class ImmediateKind(OperandKind): rust_type=None, values=None): # type: (str, str, str, str, Dict[str, str]) -> None + if rust_type is None: + rust_type = 'ir::immediates::' + camel_case(name) super(ImmediateKind, self).__init__( name, doc, default_member, rust_type) self.values = values diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index ade66e643c..104f779e2d 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -68,7 +68,8 @@ def gen_arguments_method(fmt, is_mut): as_slice = 'as_mut_slice' with fmt.indented( - 'pub fn {f}<\'a>(&\'a {m}self, pool: &\'a {m}ValueListPool) -> ' + 'pub fn {f}<\'a>(&\'a {m}self, ' + 'pool: &\'a {m}ir::ValueListPool) -> ' '&{m}[Value] {{' .format(f=method, m=mut), '}'): with fmt.indented('match *self {', '}'): @@ -111,8 +112,8 @@ def gen_instruction_data_impl(fmt): - `pub fn opcode(&self) -> Opcode` - `pub fn arguments(&self, &pool) -> &[Value]` - `pub fn arguments_mut(&mut self, &pool) -> &mut [Value]` - - `pub fn take_value_list(&mut self) -> Option` - - `pub fn put_value_list(&mut self, args: ValueList>` + - `pub fn take_value_list(&mut self) -> Option` + - `pub fn put_value_list(&mut self, args: ir::ValueList>` """ # The `opcode` method simply reads the `opcode` members. This is really a @@ -128,7 +129,7 @@ def gen_instruction_data_impl(fmt): fmt.doc_comment('Get the controlling type variable operand.') with fmt.indented( - 'pub fn typevar_operand(&self, pool: &ValueListPool) -> ' + 'pub fn typevar_operand(&self, pool: &ir::ValueListPool) -> ' 'Option {', '}'): with fmt.indented('match *self {', '}'): for f in InstructionFormat.all_formats: @@ -174,7 +175,7 @@ def gen_instruction_data_impl(fmt): `put_value_list` to put the value list back. """) with fmt.indented( - 'pub fn take_value_list(&mut self) -> Option {', + 'pub fn take_value_list(&mut self) -> Option {', '}'): with fmt.indented('match *self {', '}'): for f in InstructionFormat.all_formats: @@ -194,7 +195,8 @@ def gen_instruction_data_impl(fmt): list is empty. This avoids leaking list pool memory. """) with fmt.indented( - 'pub fn put_value_list(&mut self, vlist: ValueList) {', '}'): + 'pub fn put_value_list(&mut self, vlist: ir::ValueList) {', + '}'): with fmt.indented('let args = match *self {', '};'): for f in InstructionFormat.all_formats: n = 'InstructionData::' + f.name @@ -466,21 +468,22 @@ def gen_format_constructor(iform, fmt): if iform.has_value_list: # Take all value arguments as a finished value list. The value lists # are created by the individual instruction constructors. - args.append('args: ValueList') + args.append('args: ir::ValueList') else: # Take a fixed number of value operands. for i in range(iform.num_value_operands): args.append('arg{}: Value'.format(i)) proto = '{}({})'.format(iform.name, ', '.join(args)) - proto += " -> (Inst, &'f mut DataFlowGraph)" + proto += " -> (Inst, &'f mut ir::DataFlowGraph)" fmt.doc_comment(str(iform)) fmt.line('#[allow(non_snake_case)]') with fmt.indented('fn {} {{'.format(proto), '}'): # Generate the instruction data. with fmt.indented( - 'let data = InstructionData::{} {{'.format(iform.name), '};'): + 'let data = ir::InstructionData::{} {{'.format(iform.name), + '};'): fmt.line('opcode,') gen_member_inits(iform, fmt) @@ -527,7 +530,7 @@ def gen_inst_builder(inst, fmt): # The controlling type variable will be inferred from the input values if # possible. Otherwise, it is the first method argument. if inst.is_polymorphic and not inst.use_typevar_operand: - args.append('{}: Type'.format(inst.ctrl_typevar.name)) + args.append('{}: ir::Type'.format(inst.ctrl_typevar.name)) tmpl_types = list() # type: List[str] into_args = list() # type: List[str] @@ -590,7 +593,7 @@ def gen_inst_builder(inst, fmt): # Finally, the value operands. if inst.format.has_value_list: # We need to build a value list with all the arguments. - fmt.line('let mut vlist = ValueList::default();') + fmt.line('let mut vlist = ir::ValueList::default();') args.append('vlist') with fmt.indented('{', '}'): fmt.line( diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 3673788543..c4d2000bf6 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -3,13 +3,11 @@ //! A `Builder` provides a convenient interface for inserting instructions into a Cretonne //! function. Many of its methods are generated from the meta language instruction definitions. +use ir; use ir::types; use ir::{InstructionData, DataFlowGraph}; -use ir::{Opcode, Type, Inst, Value, Ebb, JumpTable, SigRef, FuncRef, StackSlot, GlobalVar, - ValueList, MemFlags}; -use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32, Uoffset32}; -use ir::condcodes::{IntCC, FloatCC}; -use isa::RegUnit; +use ir::{Opcode, Type, Inst, Value}; +use isa; /// Base trait for instruction builders. /// diff --git a/lib/cretonne/src/isa/riscv/enc_tables.rs b/lib/cretonne/src/isa/riscv/enc_tables.rs index f095e67243..1f0f2e132d 100644 --- a/lib/cretonne/src/isa/riscv/enc_tables.rs +++ b/lib/cretonne/src/isa/riscv/enc_tables.rs @@ -1,6 +1,5 @@ //! Encoding tables for RISC-V. -use ir::condcodes::IntCC; use ir; use isa; use isa::constraints::*; diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 9e96c61e5d..faed731c95 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -17,7 +17,6 @@ use cursor::{Cursor, FuncCursor}; use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir; -use ir::condcodes::IntCC; use isa::TargetIsa; use bitset::BitSet; From 859937209854ec1332dca68c66a1775d8f7402a5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 18 Aug 2017 14:37:03 -0700 Subject: [PATCH 0976/3084] Rename the entity_ref module to 'entity'. This is in preparation for moving a number of data structures into sub-modules of the 'entity' module: - EntityList - EntityMap - SparseMap --- lib/cretonne/src/{entity_ref.rs => entity/mod.rs} | 4 +++- lib/cretonne/src/entity_list.rs | 4 ++-- lib/cretonne/src/entity_map.rs | 2 +- lib/cretonne/src/ir/jumptable.rs | 2 +- lib/cretonne/src/ir/layout.rs | 2 +- lib/cretonne/src/ir/progpoint.rs | 4 ++-- lib/cretonne/src/isa/registers.rs | 2 +- lib/cretonne/src/lib.rs | 2 +- lib/cretonne/src/regalloc/diversion.rs | 2 +- lib/cretonne/src/regalloc/liverange.rs | 2 +- lib/cretonne/src/regalloc/solver.rs | 2 +- lib/cretonne/src/sparse_map.rs | 4 ++-- lib/frontend/src/frontend.rs | 4 ++-- lib/frontend/src/lib.rs | 2 +- lib/frontend/src/ssa.rs | 4 ++-- lib/reader/src/sourcemap.rs | 2 +- 16 files changed, 23 insertions(+), 21 deletions(-) rename lib/cretonne/src/{entity_ref.rs => entity/mod.rs} (91%) diff --git a/lib/cretonne/src/entity_ref.rs b/lib/cretonne/src/entity/mod.rs similarity index 91% rename from lib/cretonne/src/entity_ref.rs rename to lib/cretonne/src/entity/mod.rs index 9990b521f0..26c1e54353 100644 --- a/lib/cretonne/src/entity_ref.rs +++ b/lib/cretonne/src/entity/mod.rs @@ -2,6 +2,8 @@ //! //! This module defines an `EntityRef` trait that should be implemented by reference types wrapping //! a small integer index. +//! +//! Various data structures based on the entity references are defined in sub-modules. /// A type wrapping a small integer index should implement `EntityRef` so it can be used as the key /// of an `EntityMap` or `SparseMap`. @@ -19,7 +21,7 @@ pub trait EntityRef: Copy + Eq { macro_rules! entity_impl { // Basic traits. ($entity:ident) => { - impl $crate::entity_ref::EntityRef for $entity { + impl $crate::entity::EntityRef for $entity { fn new(index: usize) -> Self { assert!(index < (::std::u32::MAX as usize)); $entity(index as u32) diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity_list.rs index 4b535e3f56..3685f00eab 100644 --- a/lib/cretonne/src/entity_list.rs +++ b/lib/cretonne/src/entity_list.rs @@ -46,7 +46,7 @@ //! The index stored in an `EntityList` points to part 2, the list elements. The value 0 is //! reserved for the empty list which isn't allocated in the vector. -use entity_ref::EntityRef; +use entity::EntityRef; use std::hash::{Hash, Hasher}; use std::marker::PhantomData; use std::mem; @@ -483,7 +483,7 @@ mod tests { use super::*; use super::{sclass_size, sclass_for_length}; use ir::Inst; - use entity_ref::EntityRef; + use entity::EntityRef; #[test] fn size_classes() { diff --git a/lib/cretonne/src/entity_map.rs b/lib/cretonne/src/entity_map.rs index be73b790e1..7afb863ed0 100644 --- a/lib/cretonne/src/entity_map.rs +++ b/lib/cretonne/src/entity_map.rs @@ -9,7 +9,7 @@ //! - A *secondary* `EntityMap` contains additional data about entities kept in a primary map. The //! values need to implement `Clone + Default` traits so the map can be grown with `ensure`. -use entity_ref::EntityRef; +use entity::EntityRef; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index aaaa694a50..c0f97afea3 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -122,7 +122,7 @@ impl Display for JumpTableData { mod tests { use super::JumpTableData; use ir::Ebb; - use entity_ref::EntityRef; + use entity::EntityRef; #[test] fn empty() { diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 3f31d34f23..9e03daa8a9 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -1086,7 +1086,7 @@ impl<'c, 'fc: 'c, 'fd> InstInserterBase<'fd> for LayoutCursorInserter<'c, 'fc, ' #[cfg(test)] mod tests { use super::{Layout, Cursor, CursorBase, CursorPosition}; - use entity_ref::EntityRef; + use entity::EntityRef; use ir::{Ebb, Inst, ProgramOrder}; use std::cmp::Ordering; diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index 7adeb43c5c..7d6e9197bd 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -1,6 +1,6 @@ //! Program points. -use entity_ref::EntityRef; +use entity::EntityRef; use ir::{Ebb, Inst, ValueDef}; use std::fmt; use std::u32; @@ -135,7 +135,7 @@ pub trait ProgramOrder { #[cfg(test)] mod tests { use super::*; - use entity_ref::EntityRef; + use entity::EntityRef; use ir::{Inst, Ebb}; #[test] diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 2823fbea37..eacce024fb 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -1,6 +1,6 @@ //! Data structures describing the registers in an ISA. -use entity_ref::EntityRef; +use entity::EntityRef; use std::fmt; /// Register units are the smallest units of register allocation. diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index b021bf55b2..324698c53c 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -13,7 +13,7 @@ pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); #[macro_use] pub mod dbg; #[macro_use] -pub mod entity_ref; +pub mod entity; pub mod binemit; pub mod bitset; diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index 4f17d3685d..bbc5c067ae 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -104,7 +104,7 @@ impl RegDiversions { mod tests { use super::*; use ir::Value; - use entity_ref::EntityRef; + use entity::EntityRef; #[test] fn inserts() { diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 9174e0cdba..6b5353910e 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -435,7 +435,7 @@ impl SparseMapValue for LiveRange { mod tests { use super::LiveRange; use ir::{Inst, Ebb, Value}; - use entity_ref::EntityRef; + use entity::EntityRef; use ir::{ProgramOrder, ExpandedProgramPoint}; use std::cmp::Ordering; diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index ea028f551b..4cec792afe 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -702,7 +702,7 @@ impl Solver { #[cfg(test)] #[cfg(build_arm32)] mod tests { - use entity_ref::EntityRef; + use entity::EntityRef; use ir::Value; use isa::{TargetIsa, RegClass, RegUnit}; use regalloc::AllocatableSet; diff --git a/lib/cretonne/src/sparse_map.rs b/lib/cretonne/src/sparse_map.rs index c1a8bf5480..84c17f7b5f 100644 --- a/lib/cretonne/src/sparse_map.rs +++ b/lib/cretonne/src/sparse_map.rs @@ -36,7 +36,7 @@ //! contain their own key. use entity_map::EntityMap; -use entity_ref::EntityRef; +use entity::EntityRef; use std::mem; use std::slice; use std::u32; @@ -221,7 +221,7 @@ pub type SparseSet = SparseMap; #[cfg(test)] mod tests { use super::*; - use entity_ref::EntityRef; + use entity::EntityRef; use ir::Inst; // Mock key-value object for testing. diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index d81cee4bc7..935251e89e 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -7,7 +7,7 @@ use cretonne::ir::function::DisplayFunction; use cretonne::isa::TargetIsa; use ssa::{SSABuilder, SideEffects, Block}; use cretonne::entity_map::{EntityMap, PrimaryEntityData}; -use cretonne::entity_ref::EntityRef; +use cretonne::entity::EntityRef; use std::hash::Hash; /// Permanent structure used for translating into Cretonne IL. @@ -571,7 +571,7 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> #[cfg(test)] mod tests { - use cretonne::entity_ref::EntityRef; + use cretonne::entity::EntityRef; use cretonne::ir::{FunctionName, Function, CallConv, Signature, ArgumentType, InstBuilder}; use cretonne::ir::types::*; use frontend::{ILBuilder, FunctionBuilder}; diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 10a06b23cc..69723ae50c 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -35,7 +35,7 @@ //! extern crate cretonne; //! extern crate cton_frontend; //! -//! use cretonne::entity_ref::EntityRef; +//! use cretonne::entity::EntityRef; //! use cretonne::ir::{FunctionName, CallConv, Function, Signature, ArgumentType, InstBuilder}; //! use cretonne::ir::types::*; //! use cton_frontend::{ILBuilder, FunctionBuilder}; diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index fc5c85f1dd..bad49c6264 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -10,7 +10,7 @@ use cretonne::ir::{Ebb, Value, Inst, Type, DataFlowGraph, JumpTables, Layout, Cu use cretonne::ir::instructions::BranchInfo; use std::hash::Hash; use cretonne::entity_map::{EntityMap, PrimaryEntityData}; -use cretonne::entity_ref::EntityRef; +use cretonne::entity::EntityRef; use cretonne::packed_option::PackedOption; use cretonne::packed_option::ReservedValue; use std::u32; @@ -607,7 +607,7 @@ impl SSABuilder #[cfg(test)] mod tests { - use cretonne::entity_ref::EntityRef; + use cretonne::entity::EntityRef; use cretonne::ir::{Function, InstBuilder, Cursor, CursorBase, Inst, JumpTableData}; use cretonne::ir::types::*; use cretonne::verify_function; diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index b085a78c1b..cfd7e85cec 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -7,7 +7,7 @@ //! The `SourceMap` struct defined in this module makes the same mapping available to parser //! clients. -use cretonne::entity_ref::EntityRef; +use cretonne::entity::EntityRef; use cretonne::ir::entities::AnyEntity; use cretonne::ir::{StackSlot, GlobalVar, JumpTable, Ebb, Value, SigRef, FuncRef}; use error::{Result, Location}; From 7e08b14cf6d967502fa923773a269cfd88568540 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 18 Aug 2017 15:04:10 -0700 Subject: [PATCH 0977/3084] Split EntityMap into entity::PrimaryMap and entity::EntityMap. The new PrimaryMap replaces the primary EntityMap and the PrimaryEntityData marker trait which was causing some confusion. We now have a clear division between the two types of maps: - PrimaryMap is used to assign entity numbers to the primary data for an entity. - EntityMap is a secondary mapping adding additional info. The split also means that the secondary EntityMap can now behave as if all keys have a default value. This means that we can get rid of the annoying ensure() and get_or_default() methods ther were used everywhere instead of indexing. Just use normal indexing now; non-existent keys will return the default value. --- cranelift/src/filetest/binemit.rs | 6 +- lib/cretonne/src/binemit/relaxation.rs | 2 +- lib/cretonne/src/cursor.rs | 2 +- lib/cretonne/src/dominator_tree.rs | 6 +- lib/cretonne/src/entity/keys.rs | 55 ++++++ lib/cretonne/src/entity/map.rs | 135 +++++++++++++ lib/cretonne/src/entity/mod.rs | 8 + lib/cretonne/src/entity/primary.rs | 141 +++++++++++++ lib/cretonne/src/entity_map.rs | 263 ------------------------- lib/cretonne/src/flowgraph.rs | 2 +- lib/cretonne/src/ir/dfg.rs | 30 ++- lib/cretonne/src/ir/function.rs | 16 +- lib/cretonne/src/ir/layout.rs | 22 +-- lib/cretonne/src/ir/mod.rs | 7 +- lib/cretonne/src/ir/stackslot.rs | 8 +- lib/cretonne/src/legalizer/boundary.rs | 4 +- lib/cretonne/src/legalizer/mod.rs | 2 +- lib/cretonne/src/lib.rs | 1 - lib/cretonne/src/loop_analysis.rs | 9 +- lib/cretonne/src/regalloc/coloring.rs | 2 +- lib/cretonne/src/regalloc/diversion.rs | 2 +- lib/cretonne/src/regalloc/spilling.rs | 2 +- lib/cretonne/src/regalloc/virtregs.rs | 11 +- lib/cretonne/src/sparse_map.rs | 5 +- lib/cretonne/src/verifier/liveness.rs | 2 +- lib/cretonne/src/verifier/mod.rs | 4 +- lib/cretonne/src/write.rs | 2 +- lib/frontend/src/frontend.rs | 11 +- lib/frontend/src/ssa.rs | 12 +- lib/reader/src/parser.rs | 4 +- 30 files changed, 413 insertions(+), 363 deletions(-) create mode 100644 lib/cretonne/src/entity/keys.rs create mode 100644 lib/cretonne/src/entity/map.rs create mode 100644 lib/cretonne/src/entity/primary.rs delete mode 100644 lib/cretonne/src/entity_map.rs diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 09a6955b5e..af6e72f21e 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -107,11 +107,11 @@ impl SubTest for TestBinEmit { // Give an encoding to any instruction that doesn't already have one. for ebb in func.layout.ebbs() { for inst in func.layout.ebb_insts(ebb) { - if !func.encodings.get_or_default(inst).is_legal() { + if !func.encodings[inst].is_legal() { if let Ok(enc) = isa.encode(&func.dfg, &func.dfg[inst], func.dfg.ctrl_typevar(inst)) { - *func.encodings.ensure(inst) = enc; + func.encodings[inst] = enc; } } } @@ -158,7 +158,7 @@ impl SubTest for TestBinEmit { ebb); for inst in func.layout.ebb_insts(ebb) { sink.text.clear(); - let enc = func.encodings.get_or_default(inst); + let enc = func.encodings[inst]; // Send legal encodings into the emitter. if enc.is_legal() { diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index e49b534624..51d288703c 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -67,7 +67,7 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> Result ir::InstInserterBase<'c> for &'c mut EncCursor<'f> { // Assign an encoding. match self.isa .encode(&self.func.dfg, &self.func.dfg[inst], ctrl_typevar) { - Ok(e) => *self.func.encodings.ensure(inst) = e, + Ok(e) => self.func.encodings[inst] = e, Err(_) => panic!("can't encode {}", self.display_inst(inst)), } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index d88adbd4a0..bb80bd3687 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -1,6 +1,6 @@ //! A Dominator Tree represented as mappings of Ebbs to their immediate dominator. -use entity_map::EntityMap; +use entity::EntityMap; use flowgraph::{ControlFlowGraph, BasicBlock}; use ir::{Ebb, Inst, Function, Layout, ProgramOrder, ExpandedProgramPoint}; use packed_option::PackedOption; @@ -339,7 +339,7 @@ impl DominatorTree { pub fn recompute_split_ebb(&mut self, old_ebb: Ebb, new_ebb: Ebb, split_jump_inst: Inst) { if !self.is_reachable(old_ebb) { // old_ebb is unreachable, it stays so and new_ebb is unreachable too - *self.nodes.ensure(new_ebb) = Default::default(); + self.nodes[new_ebb] = Default::default(); return; } // We use the RPO comparison on the postorder list so we invert the operands of the @@ -350,7 +350,7 @@ impl DominatorTree { .binary_search_by(|probe| self.rpo_cmp_ebb(old_ebb, *probe)) .expect("the old ebb is not declared to the dominator tree"); let new_ebb_rpo = self.insert_after_rpo(old_ebb, old_ebb_postorder_index, new_ebb); - *self.nodes.ensure(new_ebb) = DomNode { + self.nodes[new_ebb] = DomNode { rpo_number: new_ebb_rpo, idom: Some(split_jump_inst).into(), }; diff --git a/lib/cretonne/src/entity/keys.rs b/lib/cretonne/src/entity/keys.rs new file mode 100644 index 0000000000..795e896a00 --- /dev/null +++ b/lib/cretonne/src/entity/keys.rs @@ -0,0 +1,55 @@ +//! A double-ended iterator over entity references. + +use entity::EntityRef; +use std::marker::PhantomData; + +/// Iterate over all keys in order. +pub struct Keys { + pos: usize, + rev_pos: usize, + unused: PhantomData, +} + +impl Keys { + /// Create a `Keys` iterator that visits `count` entities starting from 0. + pub fn new(count: usize) -> Keys { + Keys { + pos: 0, + rev_pos: count, + unused: PhantomData, + } + } +} + +impl Iterator for Keys { + type Item = K; + + fn next(&mut self) -> Option { + if self.pos < self.rev_pos { + let k = K::new(self.pos); + self.pos += 1; + Some(k) + } else { + None + } + } + + fn size_hint(&self) -> (usize, Option) { + let size = self.rev_pos - self.pos; + (size, Some(size)) + } +} + +impl DoubleEndedIterator for Keys { + fn next_back(&mut self) -> Option { + if self.rev_pos > self.pos { + let k = K::new(self.rev_pos - 1); + self.rev_pos -= 1; + Some(k) + } else { + None + } + } +} + +impl ExactSizeIterator for Keys {} diff --git a/lib/cretonne/src/entity/map.rs b/lib/cretonne/src/entity/map.rs new file mode 100644 index 0000000000..ca6f1ff7d9 --- /dev/null +++ b/lib/cretonne/src/entity/map.rs @@ -0,0 +1,135 @@ +//! Densely numbered entity references as mapping keys. +//! +//! The `EntityMap` data structure uses the dense index space to implement a map with a vector. +//! Unlike `PrimaryMap`, and `EntityMap` can't be used to allocate entity references. It is used to +//! associate secondary information with entities. + +use entity::{EntityRef, Keys}; +use std::marker::PhantomData; +use std::ops::{Index, IndexMut}; + +/// A mapping `K -> V` for densely indexed entity references. +#[derive(Debug, Clone)] +pub struct EntityMap + where K: EntityRef, + V: Clone +{ + elems: Vec, + default: V, + unused: PhantomData, +} + +/// Shared `EntityMap` implementation for all value types. +impl EntityMap + where K: EntityRef, + V: Clone +{ + /// Create a new empty map. + pub fn new() -> Self + where V: Default + { + EntityMap { + elems: Vec::new(), + default: Default::default(), + unused: PhantomData, + } + } + + /// Get the element at `k` if it exists. + pub fn get(&self, k: K) -> Option<&V> { + self.elems.get(k.index()) + } + + /// Is this map completely empty? + pub fn is_empty(&self) -> bool { + self.elems.is_empty() + } + + /// Remove all entries from this map. + pub fn clear(&mut self) { + self.elems.clear() + } + + /// Iterate over all the keys in this map. + pub fn keys(&self) -> Keys { + Keys::new(self.elems.len()) + } + + /// Resize the map to have `n` entries by adding default entries as needed. + pub fn resize(&mut self, n: usize) { + self.elems.resize(n, self.default.clone()); + } +} + +/// Immutable indexing into an `EntityMap`. +/// +/// All keys are permitted. Untouched entries have the default value. +impl Index for EntityMap + where K: EntityRef, + V: Clone +{ + type Output = V; + + fn index(&self, k: K) -> &V { + self.get(k).unwrap_or(&self.default) + } +} + +/// Mutable indexing into an `EntityMap`. +/// +/// The map grows as needed to accommodate new keys. +impl IndexMut for EntityMap + where K: EntityRef, + V: Clone +{ + fn index_mut(&mut self, k: K) -> &mut V { + let i = k.index(); + if i >= self.elems.len() { + self.resize(i + 1); + } + &mut self.elems[i] + } +} + +#[cfg(test)] +mod tests { + use super::*; + + // `EntityRef` impl for testing. + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + struct E(u32); + + impl EntityRef for E { + fn new(i: usize) -> Self { + E(i as u32) + } + fn index(self) -> usize { + self.0 as usize + } + } + + #[test] + fn basic() { + let r0 = E(0); + let r1 = E(1); + let r2 = E(2); + let mut m = EntityMap::new(); + + let v: Vec = m.keys().collect(); + assert_eq!(v, []); + + m[r2] = 3; + m[r1] = 5; + + assert_eq!(m[r1], 5); + assert_eq!(m[r2], 3); + + let v: Vec = m.keys().collect(); + assert_eq!(v, [r0, r1, r2]); + + let shared = &m; + assert_eq!(shared[r0], 0); + assert_eq!(shared[r1], 5); + assert_eq!(shared[r2], 3); + } +} diff --git a/lib/cretonne/src/entity/mod.rs b/lib/cretonne/src/entity/mod.rs index 26c1e54353..58f900abc4 100644 --- a/lib/cretonne/src/entity/mod.rs +++ b/lib/cretonne/src/entity/mod.rs @@ -5,6 +5,14 @@ //! //! Various data structures based on the entity references are defined in sub-modules. +mod keys; +mod map; +mod primary; + +pub use self::keys::Keys; +pub use self::map::EntityMap; +pub use self::primary::PrimaryMap; + /// A type wrapping a small integer index should implement `EntityRef` so it can be used as the key /// of an `EntityMap` or `SparseMap`. pub trait EntityRef: Copy + Eq { diff --git a/lib/cretonne/src/entity/primary.rs b/lib/cretonne/src/entity/primary.rs new file mode 100644 index 0000000000..e98698f117 --- /dev/null +++ b/lib/cretonne/src/entity/primary.rs @@ -0,0 +1,141 @@ +//! Densely numbered entity references as mapping keys. +//! +//! The `PrimaryMap` data structure uses the dense index space to implement a map with a vector. +//! +//! A primary map contains the main definition of an entity, and it can be used to allocate new +//! entity references with the `push` method. +//! +//! There should only be a single `PrimaryMap` instance for a given `EntityRef` type, otherwise +//! conflicting references will be created. + +use entity::{EntityRef, Keys}; +use std::marker::PhantomData; +use std::ops::{Index, IndexMut}; + +/// A mapping `K -> V` for densely indexed entity references. +#[derive(Debug, Clone)] +pub struct PrimaryMap + where K: EntityRef +{ + elems: Vec, + unused: PhantomData, +} + +impl PrimaryMap + where K: EntityRef +{ + /// Create a new empty map. + pub fn new() -> Self { + PrimaryMap { + elems: Vec::new(), + unused: PhantomData, + } + } + + /// Check if `k` is a valid key in the map. + pub fn is_valid(&self, k: K) -> bool { + k.index() < self.elems.len() + } + + /// Get the element at `k` if it exists. + pub fn get(&self, k: K) -> Option<&V> { + self.elems.get(k.index()) + } + + /// Is this map completely empty? + pub fn is_empty(&self) -> bool { + self.elems.is_empty() + } + + /// Get the total number of entity references created. + pub fn len(&self) -> usize { + self.elems.len() + } + + /// Iterate over all the keys in this map. + pub fn keys(&self) -> Keys { + Keys::new(self.elems.len()) + } + + /// Remove all entries from this map. + pub fn clear(&mut self) { + self.elems.clear() + } + + /// Get the key that will be assigned to the next pushed value. + pub fn next_key(&self) -> K { + K::new(self.elems.len()) + } + + /// Append `v` to the mapping, assigning a new key which is returned. + pub fn push(&mut self, v: V) -> K { + let k = self.next_key(); + self.elems.push(v); + k + } +} + +/// Immutable indexing into an `PrimaryMap`. +/// The indexed value must be in the map. +impl Index for PrimaryMap + where K: EntityRef +{ + type Output = V; + + fn index(&self, k: K) -> &V { + &self.elems[k.index()] + } +} + +/// Mutable indexing into an `PrimaryMap`. +impl IndexMut for PrimaryMap + where K: EntityRef +{ + fn index_mut(&mut self, k: K) -> &mut V { + &mut self.elems[k.index()] + } +} + +#[cfg(test)] +mod tests { + use super::*; + + // `EntityRef` impl for testing. + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + struct E(u32); + + impl EntityRef for E { + fn new(i: usize) -> Self { + E(i as u32) + } + fn index(self) -> usize { + self.0 as usize + } + } + + #[test] + fn basic() { + let r0 = E(0); + let r1 = E(1); + let m = PrimaryMap::::new(); + + let v: Vec = m.keys().collect(); + assert_eq!(v, []); + + assert!(!m.is_valid(r0)); + assert!(!m.is_valid(r1)); + } + + #[test] + fn push() { + let mut m = PrimaryMap::new(); + let k1: E = m.push(12); + let k2 = m.push(33); + + assert_eq!(m[k1], 12); + assert_eq!(m[k2], 33); + + let v: Vec = m.keys().collect(); + assert_eq!(v, [k1, k2]); + } +} diff --git a/lib/cretonne/src/entity_map.rs b/lib/cretonne/src/entity_map.rs deleted file mode 100644 index 7afb863ed0..0000000000 --- a/lib/cretonne/src/entity_map.rs +++ /dev/null @@ -1,263 +0,0 @@ -//! Densely numbered entity references as mapping keys. -//! -//! The `EntityMap` data structure uses the dense index space to implement a map with a vector. -//! There are primary and secondary entity maps: -//! -//! - A *primary* `EntityMap` contains the main definition of an entity, and it can be used to -//! allocate new entity references with the `push` method. The values stores in a primary map -//! must implement the `PrimaryEntityData` marker trait. -//! - A *secondary* `EntityMap` contains additional data about entities kept in a primary map. The -//! values need to implement `Clone + Default` traits so the map can be grown with `ensure`. - -use entity::EntityRef; -use std::marker::PhantomData; -use std::ops::{Index, IndexMut}; - -/// A mapping `K -> V` for densely indexed entity references. -#[derive(Debug, Clone)] -pub struct EntityMap - where K: EntityRef -{ - elems: Vec, - unused: PhantomData, -} - -/// Shared `EntityMap` implementation for all value types. -impl EntityMap - where K: EntityRef -{ - /// Create a new empty map. - pub fn new() -> Self { - EntityMap { - elems: Vec::new(), - unused: PhantomData, - } - } - - /// Check if `k` is a valid key in the map. - pub fn is_valid(&self, k: K) -> bool { - k.index() < self.elems.len() - } - - /// Get the element at `k` if it exists. - pub fn get(&self, k: K) -> Option<&V> { - self.elems.get(k.index()) - } - - /// Is this map completely empty? - pub fn is_empty(&self) -> bool { - self.elems.is_empty() - } - - /// Remove all entries from this map. - pub fn clear(&mut self) { - self.elems.clear() - } - - /// Iterate over all the keys in this map. - pub fn keys(&self) -> Keys { - Keys { - pos: 0, - rev_pos: self.elems.len(), - unused: PhantomData, - } - } -} - -/// A marker trait for data stored in primary entity maps. -/// -/// A primary entity map can be used to allocate new entity references with the `push` method. It -/// is important that entity references can't be created anywhere else, so the data stored in a -/// primary entity map must be tagged as `PrimaryEntityData` to unlock the `push` method. -pub trait PrimaryEntityData {} - -/// Additional methods for primary entry maps only. -/// -/// These are identified by the `PrimaryEntityData` marker trait. -impl EntityMap - where K: EntityRef, - V: PrimaryEntityData -{ - /// Get the key that will be assigned to the next pushed value. - pub fn next_key(&self) -> K { - K::new(self.elems.len()) - } - - /// Append `v` to the mapping, assigning a new key which is returned. - pub fn push(&mut self, v: V) -> K { - let k = self.next_key(); - self.elems.push(v); - k - } - - /// Get the total number of entity references created. - pub fn len(&self) -> usize { - self.elems.len() - } -} - -/// Additional methods for value types that implement `Clone` and `Default`. -/// -/// When the value type implements these additional traits, the `EntityMap` can be resized -/// explicitly with the `ensure` method. -/// -/// Use this for secondary maps that are mapping keys created by another primary map. -impl EntityMap - where K: EntityRef, - V: Clone + Default -{ - /// Create a new secondary `EntityMap` that is prepared to hold `n` elements. - /// - /// Use this when the length of the primary map is known: - /// ``` - /// let secondary_map = EntityMap::with_capacity(primary_map.len()); - /// ``` - pub fn with_capacity(n: usize) -> Self { - let mut map = EntityMap { - elems: Vec::with_capacity(n), - unused: PhantomData, - }; - map.elems.resize(n, V::default()); - map - } - - /// Resize the map to have `n` entries by adding default entries as needed. - pub fn resize(&mut self, n: usize) { - self.elems.resize(n, V::default()); - } - - /// Ensure that `k` is a valid key but adding default entries if necessary. - /// - /// Return a mutable reference to the corresponding entry. - pub fn ensure(&mut self, k: K) -> &mut V { - if !self.is_valid(k) { - self.resize(k.index() + 1) - } - &mut self.elems[k.index()] - } - - /// Get the element at `k` or the default value if `k` is out of range. - pub fn get_or_default(&self, k: K) -> V { - self.elems.get(k.index()).cloned().unwrap_or_default() - } -} - -/// Immutable indexing into an `EntityMap`. -/// The indexed value must be in the map, either because it was created by `push`, or the key was -/// passed to `ensure`. -impl Index for EntityMap - where K: EntityRef -{ - type Output = V; - - fn index(&self, k: K) -> &V { - &self.elems[k.index()] - } -} - -/// Mutable indexing into an `EntityMap`. -/// Use `ensure` instead if the key is not known to be valid. -impl IndexMut for EntityMap - where K: EntityRef -{ - fn index_mut(&mut self, k: K) -> &mut V { - &mut self.elems[k.index()] - } -} - -/// Iterate over all keys in order. -pub struct Keys - where K: EntityRef -{ - pos: usize, - rev_pos: usize, - unused: PhantomData, -} - -impl Iterator for Keys - where K: EntityRef -{ - type Item = K; - - fn next(&mut self) -> Option { - if self.pos < self.rev_pos { - let k = K::new(self.pos); - self.pos += 1; - Some(k) - } else { - None - } - } -} - -impl DoubleEndedIterator for Keys - where K: EntityRef -{ - fn next_back(&mut self) -> Option { - if self.rev_pos > self.pos { - let k = K::new(self.rev_pos - 1); - self.rev_pos -= 1; - Some(k) - } else { - None - } - } -} - -#[cfg(test)] -mod tests { - use super::*; - - // `EntityRef` impl for testing. - #[derive(Clone, Copy, Debug, PartialEq, Eq)] - struct E(u32); - - impl EntityRef for E { - fn new(i: usize) -> Self { - E(i as u32) - } - fn index(self) -> usize { - self.0 as usize - } - } - - impl PrimaryEntityData for isize {} - - #[test] - fn basic() { - let r0 = E(0); - let r1 = E(1); - let r2 = E(2); - let mut m = EntityMap::new(); - - let v: Vec = m.keys().collect(); - assert_eq!(v, []); - - assert!(!m.is_valid(r0)); - m.ensure(r2); - m[r2] = 3; - assert!(m.is_valid(r1)); - m[r1] = 5; - - assert_eq!(m[r1], 5); - assert_eq!(m[r2], 3); - - let v: Vec = m.keys().collect(); - assert_eq!(v, [r0, r1, r2]); - - let shared = &m; - assert_eq!(shared[r0], 0); - assert_eq!(shared[r1], 5); - assert_eq!(shared[r2], 3); - } - - #[test] - fn push() { - let mut m = EntityMap::new(); - let k1: E = m.push(12); - let k2 = m.push(33); - - assert_eq!(m[k1], 12); - assert_eq!(m[k2], 33); - } -} diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index f7cb0c9729..504d90fb9d 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -25,7 +25,7 @@ use ir::{Function, Inst, Ebb}; use ir::instructions::BranchInfo; -use entity_map::EntityMap; +use entity::EntityMap; use std::mem; /// A basic block denoted by its enclosing Ebb and last instruction. diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 959b29c8e8..219557aeb8 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -1,6 +1,6 @@ //! Data flow graph tracking Instructions, Values, and EBBs. -use entity_map::{EntityMap, PrimaryEntityData}; +use entity::{PrimaryMap, EntityMap}; use isa::TargetIsa; use ir::builder::{InsertBuilder, ReplaceBuilder}; use ir::extfunc::ExtFuncData; @@ -27,7 +27,7 @@ pub struct DataFlowGraph { /// Data about all of the instructions in the function, including opcodes and operands. /// The instructions in this map are not in program order. That is tracked by `Layout`, along /// with the EBB containing each instruction. - insts: EntityMap, + insts: PrimaryMap, /// List of result values for each instruction. /// @@ -38,7 +38,7 @@ pub struct DataFlowGraph { /// Extended basic blocks in the function and their arguments. /// This map is not in program order. That is handled by `Layout`, and so is the sequence of /// instructions contained in each EBB. - ebbs: EntityMap, + ebbs: PrimaryMap, /// Memory pool of value lists. /// @@ -50,33 +50,27 @@ pub struct DataFlowGraph { pub value_lists: ValueListPool, /// Primary value table with entries for all values. - values: EntityMap, + values: PrimaryMap, /// Function signature table. These signatures are referenced by indirect call instructions as /// well as the external function references. - pub signatures: EntityMap, + pub signatures: PrimaryMap, /// External function references. These are functions that can be called directly. - pub ext_funcs: EntityMap, + pub ext_funcs: PrimaryMap, } -impl PrimaryEntityData for InstructionData {} -impl PrimaryEntityData for EbbData {} -impl PrimaryEntityData for ValueData {} -impl PrimaryEntityData for Signature {} -impl PrimaryEntityData for ExtFuncData {} - impl DataFlowGraph { /// Create a new empty `DataFlowGraph`. pub fn new() -> DataFlowGraph { DataFlowGraph { - insts: EntityMap::new(), + insts: PrimaryMap::new(), results: EntityMap::new(), - ebbs: EntityMap::new(), + ebbs: PrimaryMap::new(), value_lists: ValueListPool::new(), - values: EntityMap::new(), - signatures: EntityMap::new(), - ext_funcs: EntityMap::new(), + values: PrimaryMap::new(), + signatures: PrimaryMap::new(), + ext_funcs: PrimaryMap::new(), } } @@ -115,7 +109,7 @@ impl DataFlowGraph { /// Resolve value aliases. /// /// Find the original SSA value that `value` aliases. -fn resolve_aliases(values: &EntityMap, value: Value) -> Value { +fn resolve_aliases(values: &PrimaryMap, value: Value) -> Value { let mut v = value; // Note that values may be empty here. diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 028c376479..c9d0031deb 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -3,9 +3,10 @@ //! The `Function` struct defined in this module owns all of its extended basic blocks and //! instructions. -use entity_map::{EntityMap, PrimaryEntityData}; -use ir::{FunctionName, CallConv, Signature, JumpTableData, GlobalVarData, DataFlowGraph, Layout}; -use ir::{JumpTables, InstEncodings, ValueLocations, StackSlots, GlobalVars, EbbOffsets}; +use entity::{PrimaryMap, EntityMap}; +use ir; +use ir::{FunctionName, CallConv, Signature, DataFlowGraph, Layout}; +use ir::{InstEncodings, ValueLocations, JumpTables, StackSlots, EbbOffsets}; use isa::TargetIsa; use std::fmt; use write::write_function; @@ -26,7 +27,7 @@ pub struct Function { pub stack_slots: StackSlots, /// Global variables referenced. - pub global_vars: GlobalVars, + pub global_vars: PrimaryMap, /// Jump tables used in this function. pub jump_tables: JumpTables, @@ -52,9 +53,6 @@ pub struct Function { pub offsets: EbbOffsets, } -impl PrimaryEntityData for JumpTableData {} -impl PrimaryEntityData for GlobalVarData {} - impl Function { /// Create a function with the given name and signature. pub fn with_name_signature(name: FunctionName, sig: Signature) -> Function { @@ -62,8 +60,8 @@ impl Function { name, signature: sig, stack_slots: StackSlots::new(), - global_vars: GlobalVars::new(), - jump_tables: EntityMap::new(), + global_vars: PrimaryMap::new(), + jump_tables: PrimaryMap::new(), dfg: DataFlowGraph::new(), layout: Layout::new(), encodings: EntityMap::new(), diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 9e03daa8a9..ca54cefdb3 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -5,7 +5,7 @@ use std::cmp; use std::iter::{Iterator, IntoIterator}; -use entity_map::EntityMap; +use entity::EntityMap; use packed_option::PackedOption; use ir::{Ebb, Inst, Type, DataFlowGraph}; use ir::builder::InstInserterBase; @@ -278,7 +278,7 @@ impl Layout { impl Layout { /// Is `ebb` currently part of the layout? pub fn is_ebb_inserted(&self, ebb: Ebb) -> bool { - Some(ebb) == self.first_ebb || (self.ebbs.is_valid(ebb) && self.ebbs[ebb].prev.is_some()) + Some(ebb) == self.first_ebb || self.ebbs[ebb].prev.is_some() } /// Insert `ebb` as the last EBB in the layout. @@ -286,7 +286,7 @@ impl Layout { assert!(!self.is_ebb_inserted(ebb), "Cannot append EBB that is already in the layout"); { - let node = self.ebbs.ensure(ebb); + let node = &mut self.ebbs[ebb]; assert!(node.first_inst.is_none() && node.last_inst.is_none()); node.prev = self.last_ebb.into(); node.next = None.into(); @@ -308,7 +308,7 @@ impl Layout { "EBB Insertion point not in the layout"); let after = self.ebbs[before].prev; { - let node = self.ebbs.ensure(ebb); + let node = &mut self.ebbs[ebb]; node.next = before.into(); node.prev = after; } @@ -328,7 +328,7 @@ impl Layout { "EBB Insertion point not in the layout"); let before = self.ebbs[after].next; { - let node = self.ebbs.ensure(ebb); + let node = &mut self.ebbs[ebb]; node.next = before; node.prev = after.into(); } @@ -406,11 +406,7 @@ impl<'f> IntoIterator for &'f Layout { impl Layout { /// Get the EBB containing `inst`, or `None` if `inst` is not inserted in the layout. pub fn inst_ebb(&self, inst: Inst) -> Option { - if self.insts.is_valid(inst) { - self.insts[inst].ebb.into() - } else { - None - } + self.insts[inst].ebb.into() } /// Get the EBB containing the program point `pp`. Panic if `pp` is not in the layout. @@ -433,7 +429,7 @@ impl Layout { { let ebb_node = &mut self.ebbs[ebb]; { - let inst_node = self.insts.ensure(inst); + let inst_node = &mut self.insts[inst]; inst_node.ebb = ebb.into(); inst_node.prev = ebb_node.last_inst; assert!(inst_node.next.is_none()); @@ -465,7 +461,7 @@ impl Layout { .expect("Instruction before insertion point not in the layout"); let after = self.insts[before].prev; { - let inst_node = self.insts.ensure(inst); + let inst_node = &mut self.insts[inst]; inst_node.ebb = ebb.into(); inst_node.next = before.into(); inst_node.prev = after; @@ -543,7 +539,7 @@ impl Layout { let next_ebb = self.ebbs[old_ebb].next; let last_inst = self.ebbs[old_ebb].last_inst; { - let node = self.ebbs.ensure(new_ebb); + let node = &mut self.ebbs[new_ebb]; node.prev = old_ebb.into(); node.next = next_ebb; node.first_inst = before.into(); diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 1e31e34010..75e68d110f 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -36,20 +36,17 @@ pub use ir::types::Type; pub use ir::valueloc::{ValueLoc, ArgumentLoc}; use binemit; -use entity_map::EntityMap; +use entity::{PrimaryMap, EntityMap}; use isa; /// Map of value locations. pub type ValueLocations = EntityMap; /// Map of jump tables. -pub type JumpTables = EntityMap; +pub type JumpTables = PrimaryMap; /// Map of instruction encodings. pub type InstEncodings = EntityMap; /// Code offsets for EBBs. pub type EbbOffsets = EntityMap; - -/// Map of global variables. -pub type GlobalVars = EntityMap; diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index 6f3de96cd2..528f6b22f2 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -3,7 +3,7 @@ //! The `StackSlotData` struct keeps track of a single stack slot in a function. //! -use entity_map::{EntityMap, PrimaryEntityData, Keys}; +use entity::{PrimaryMap, Keys}; use ir::{Type, StackSlot}; use std::fmt; use std::ops::Index; @@ -124,15 +124,13 @@ impl fmt::Display for StackSlotData { } } -impl PrimaryEntityData for StackSlotData {} - /// Stack frame manager. /// /// Keep track of all the stack slots used by a function. #[derive(Clone, Debug)] pub struct StackSlots { /// All allocated stack slots. - slots: EntityMap, + slots: PrimaryMap, /// All the outgoing stack slots, ordered by offset. outgoing: Vec, @@ -152,7 +150,7 @@ impl StackSlots { /// Create an empty stack slot manager. pub fn new() -> StackSlots { StackSlots { - slots: EntityMap::new(), + slots: PrimaryMap::new(), outgoing: Vec::new(), frame_size: None, } diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 6977c9e7bc..71c79ef46b 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -585,7 +585,7 @@ fn spill_entry_arguments(func: &mut Function, entry: Ebb) { .zip(func.dfg.ebb_args(entry)) { if let ArgumentLoc::Stack(offset) = abi.location { let ss = func.stack_slots.make_incoming_arg(abi.value_type, offset); - *func.locations.ensure(arg) = ValueLoc::Stack(ss); + func.locations[arg] = ValueLoc::Stack(ss); } } } @@ -646,7 +646,7 @@ fn spill_call_arguments(dfg: &mut DataFlowGraph, // Insert the spill instructions and rewrite call arguments. for (idx, arg, ss) in arglist { let stack_val = dfg.ins(pos).spill(arg); - *locations.ensure(stack_val) = ValueLoc::Stack(ss); + locations[stack_val] = ValueLoc::Stack(ss); dfg.inst_variable_args_mut(inst)[idx] = stack_val; } diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index faed731c95..230c7adc7e 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -73,7 +73,7 @@ pub fn legalize_function(func: &mut ir::Function, match isa.encode(&pos.func.dfg, &pos.func.dfg[inst], pos.func.dfg.ctrl_typevar(inst)) { - Ok(encoding) => *pos.func.encodings.ensure(inst) = encoding, + Ok(encoding) => pos.func.encodings[inst] = encoding, Err(action) => { // We should transform the instruction into legal equivalents. let changed = action(inst, pos.func, cfg); diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 324698c53c..6cb6687138 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -19,7 +19,6 @@ pub mod binemit; pub mod bitset; pub mod dominator_tree; pub mod entity_list; -pub mod entity_map; pub mod flowgraph; pub mod ir; pub mod isa; diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index 5a4607ba79..55ddf1f4df 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -2,7 +2,8 @@ //! and parent in the loop tree. use dominator_tree::DominatorTree; -use entity_map::{EntityMap, PrimaryEntityData, Keys}; +use entity::{PrimaryMap, Keys}; +use entity::EntityMap; use flowgraph::ControlFlowGraph; use ir::{Function, Ebb, Layout}; use packed_option::PackedOption; @@ -17,7 +18,7 @@ entity_impl!(Loop, "loop"); /// Loops are referenced by the Loop object, and for each loop you can access its header EBB, /// its eventual parent in the loop tree and all the EBB belonging to the loop. pub struct LoopAnalysis { - loops: EntityMap, + loops: PrimaryMap, ebb_loop_map: EntityMap>, } @@ -26,8 +27,6 @@ struct LoopData { parent: PackedOption, } -impl PrimaryEntityData for LoopData {} - impl LoopData { /// Creates a `LoopData` object with the loop header and its eventual parent in the loop tree. pub fn new(header: Ebb, parent: Option) -> LoopData { @@ -44,7 +43,7 @@ impl LoopAnalysis { /// a function. pub fn new() -> LoopAnalysis { LoopAnalysis { - loops: EntityMap::new(), + loops: PrimaryMap::new(), ebb_loop_map: EntityMap::new(), } } diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 02e4d47f81..49f26fbd37 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -707,7 +707,7 @@ impl<'a> Context<'a> { self.divert.regmove(m.value, m.from, m.to); let inst = dfg.ins(pos).regmove(m.value, m.from, m.to); match self.isa.encode(dfg, &dfg[inst], ty) { - Ok(encoding) => *encodings.ensure(inst) = encoding, + Ok(encoding) => encodings[inst] = encoding, _ => panic!("Can't encode {} {}", m.rc, dfg.display_inst(inst, self.isa)), } } diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index bbc5c067ae..c7c9595972 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -7,7 +7,7 @@ //! These register diversions are local to an EBB. No values can be diverted when entering a new //! EBB. -use entity_map::EntityMap; +use entity::EntityMap; use ir::{Value, ValueLoc}; use isa::RegUnit; diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 12a938c81c..35ef526993 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -453,7 +453,7 @@ impl<'a> Context<'a> { .make_spill_slot(self.cur.func.dfg.value_type(value)); for &v in self.virtregs.congruence_class(&value) { self.liveness.spill(v); - *self.cur.func.locations.ensure(v) = ValueLoc::Stack(ss); + self.cur.func.locations[v] = ValueLoc::Stack(ss); } } diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index 17fa641fcd..722de5a1f4 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -12,7 +12,7 @@ //! memory-to-memory copies when a spilled value is passed as an EBB argument. use entity_list::{EntityList, ListPool}; -use entity_map::{EntityMap, PrimaryEntityData, Keys}; +use entity::{PrimaryMap, EntityMap, Keys}; use ir::Value; use packed_option::PackedOption; use ref_slice::ref_slice; @@ -23,7 +23,6 @@ pub struct VirtReg(u32); entity_impl!(VirtReg, "vreg"); type ValueList = EntityList; -impl PrimaryEntityData for ValueList {} /// Collection of virtual registers. /// @@ -37,7 +36,7 @@ pub struct VirtRegs { /// /// The list of values ion a virtual register is kept sorted according to the dominator tree's /// RPO of the value defs. - vregs: EntityMap, + vregs: PrimaryMap, /// Each value belongs to at most one virtual register. value_vregs: EntityMap>, @@ -49,7 +48,7 @@ impl VirtRegs { pub fn new() -> VirtRegs { VirtRegs { pool: ListPool::new(), - vregs: EntityMap::new(), + vregs: PrimaryMap::new(), value_vregs: EntityMap::new(), } } @@ -63,7 +62,7 @@ impl VirtRegs { /// Get the virtual register containing `value`, if any. pub fn get(&self, value: Value) -> Option { - self.value_vregs.get_or_default(value).into() + self.value_vregs[value].into() } /// Get the list of values in `vreg`. The values are ordered according to `DomTree::rpo_cmp` of @@ -133,7 +132,7 @@ impl VirtRegs { self.vregs[vreg].extend(values.iter().cloned(), &mut self.pool); for &v in values { - *self.value_vregs.ensure(v) = vreg.into(); + self.value_vregs[v] = vreg.into(); } vreg diff --git a/lib/cretonne/src/sparse_map.rs b/lib/cretonne/src/sparse_map.rs index 84c17f7b5f..4918675d27 100644 --- a/lib/cretonne/src/sparse_map.rs +++ b/lib/cretonne/src/sparse_map.rs @@ -35,8 +35,7 @@ //! - `SparseMap` requires the values to implement `SparseMapValue` which means that they must //! contain their own key. -use entity_map::EntityMap; -use entity::EntityRef; +use entity::{EntityRef, EntityMap}; use std::mem; use std::slice; use std::u32; @@ -151,7 +150,7 @@ impl SparseMap let idx = self.dense.len(); assert!(idx <= u32::MAX as usize, "SparseMap overflow"); self.dense.push(value); - *self.sparse.ensure(key) = idx as u32; + self.sparse[key] = idx as u32; None } diff --git a/lib/cretonne/src/verifier/liveness.rs b/lib/cretonne/src/verifier/liveness.rs index 2dff4a6e3e..8b1ab47f4c 100644 --- a/lib/cretonne/src/verifier/liveness.rs +++ b/lib/cretonne/src/verifier/liveness.rs @@ -63,7 +63,7 @@ impl<'a> LivenessVerifier<'a> { fn check_insts(&self) -> Result { for ebb in self.func.layout.ebbs() { for inst in self.func.layout.ebb_insts(ebb) { - let encoding = self.func.encodings.get_or_default(inst); + let encoding = self.func.encodings[inst]; // Check the defs. for &val in self.func.dfg.inst_results(inst) { diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 0995449362..15b5219334 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -688,7 +688,7 @@ impl<'a> Verifier<'a> { for (&arg, &abi) in args.iter().zip(expected_args) { // Value types have already been checked by `typecheck_variable_args_iterator()`. if let ArgumentLoc::Stack(offset) = abi.location { - let arg_loc = self.func.locations.get_or_default(arg); + let arg_loc = self.func.locations[arg]; if let ValueLoc::Stack(ss) = arg_loc { // Argument value is assigned to a stack slot as expected. self.verify_stack_slot(inst, ss)?; @@ -811,7 +811,7 @@ impl<'a> Verifier<'a> { None => return Ok(()), }; - let encoding = self.func.encodings.get_or_default(inst); + let encoding = self.func.encodings[inst]; if encoding.is_legal() { let verify_encoding = isa.encode(&self.func.dfg, diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index d0166a1063..9004546f32 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -194,7 +194,7 @@ fn write_instruction(w: &mut Write, if !func.locations.is_empty() { let regs = isa.register_info(); for &r in func.dfg.inst_results(inst) { - write!(s, ",{}", func.locations.get_or_default(r).display(®s))? + write!(s, ",{}", func.locations[r].display(®s))? } } write!(s, "]")?; diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 935251e89e..57c0aaa9e0 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -6,8 +6,7 @@ use cretonne::ir::instructions::BranchInfo; use cretonne::ir::function::DisplayFunction; use cretonne::isa::TargetIsa; use ssa::{SSABuilder, SideEffects, Block}; -use cretonne::entity_map::{EntityMap, PrimaryEntityData}; -use cretonne::entity::EntityRef; +use cretonne::entity::{EntityRef, EntityMap}; use std::hash::Hash; /// Permanent structure used for translating into Cretonne IL. @@ -38,8 +37,6 @@ struct EbbData { user_arg_count: usize, } -impl PrimaryEntityData for EbbData {} - struct Position { ebb: Ebb, basic_block: Block, @@ -231,7 +228,7 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> pub fn create_ebb(&mut self) -> Ebb { let ebb = self.func.dfg.make_ebb(); self.builder.ssa.declare_ebb_header_block(ebb); - *self.builder.ebbs.ensure(ebb) = EbbData { + self.builder.ebbs[ebb] = EbbData { filled: false, pristine: true, user_arg_count: 0, @@ -286,7 +283,7 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> /// In order to use a variable in a `use_var`, you need to declare its type with this method. pub fn declare_var(&mut self, var: Variable, ty: Type) { - *self.builder.types.ensure(var) = ty; + self.builder.types[var] = ty; } /// Returns the Cretonne IL value corresponding to the utilization at the current program @@ -560,7 +557,7 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> fn handle_ssa_side_effects(&mut self, side_effects: SideEffects) { for split_ebb in side_effects.split_ebbs_created { - self.builder.ebbs.ensure(split_ebb).filled = true + self.builder.ebbs[split_ebb].filled = true } for modified_ebb in side_effects.instructions_added_to_ebbs { self.builder.ebbs[modified_ebb].pristine = false diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index bad49c6264..54485629d7 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -9,8 +9,7 @@ use cretonne::ir::{Ebb, Value, Inst, Type, DataFlowGraph, JumpTables, Layout, Cu InstBuilder}; use cretonne::ir::instructions::BranchInfo; use std::hash::Hash; -use cretonne::entity_map::{EntityMap, PrimaryEntityData}; -use cretonne::entity::EntityRef; +use cretonne::entity::{EntityRef, PrimaryMap, EntityMap}; use cretonne::packed_option::PackedOption; use cretonne::packed_option::ReservedValue; use std::u32; @@ -41,7 +40,7 @@ pub struct SSABuilder variables: EntityMap>, // Records the position of the basic blocks and the list of values used but not defined in the // block. - blocks: EntityMap>, + blocks: PrimaryMap>, // Records the basic blocks at the beginning of the `Ebb`s. ebb_headers: EntityMap>, } @@ -66,7 +65,6 @@ enum BlockData { // The block is implicitely sealed at creation. EbbBody { predecessor: Block }, } -impl PrimaryEntityData for BlockData {} impl BlockData { fn add_predecessor(&mut self, pred: Block, inst: Inst) { @@ -135,7 +133,7 @@ impl SSABuilder pub fn new() -> SSABuilder { SSABuilder { variables: EntityMap::new(), - blocks: EntityMap::new(), + blocks: PrimaryMap::new(), ebb_headers: EntityMap::new(), } } @@ -193,7 +191,7 @@ impl SSABuilder /// The SSA value is passed as an argument because it should be created with /// `ir::DataFlowGraph::append_result`. pub fn def_var(&mut self, var: Variable, val: Value, block: Block) { - self.variables.ensure(var).insert(block, val); + self.variables[var].insert(block, val); } /// Declares a use of a variable in a given basic block. Returns the SSA value corresponding @@ -296,7 +294,7 @@ impl SSABuilder ebb: ebb, undef_variables: Vec::new(), })); - *self.ebb_headers.ensure(ebb) = block.into(); + self.ebb_headers[ebb] = block.into(); block } /// Gets the header block corresponding to an Ebb, panics if the Ebb or the header block diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 0d96861b51..9ec7547c43 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1388,7 +1388,7 @@ impl<'a> Parser<'a> { .expect("duplicate inst references created"); if let Some(encoding) = encoding { - *ctx.function.encodings.ensure(inst) = encoding; + ctx.function.encodings[inst] = encoding; } if results.len() != num_results { @@ -1421,7 +1421,7 @@ impl<'a> Parser<'a> { .inst_results(inst) .iter() .zip(result_locations) { - *ctx.function.locations.ensure(value) = loc; + ctx.function.locations[value] = loc; } } From 9cb0529be4e221369e2250034d7680ba5c72a4a4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 18 Aug 2017 16:09:13 -0700 Subject: [PATCH 0978/3084] Move EntityList and SparseMap into the entity module. These data structures are dependent on EntityRef and EntityMap, so it makes sense to keep them in the same module. --- lib/cretonne/src/{entity_list.rs => entity/list.rs} | 0 lib/cretonne/src/entity/mod.rs | 4 ++++ lib/cretonne/src/{sparse_map.rs => entity/sparse.rs} | 0 lib/cretonne/src/ir/instructions.rs | 6 +++--- lib/cretonne/src/lib.rs | 2 -- lib/cretonne/src/licm.rs | 2 +- lib/cretonne/src/regalloc/live_value_tracker.rs | 2 +- lib/cretonne/src/regalloc/liveness.rs | 2 +- lib/cretonne/src/regalloc/liverange.rs | 4 ++-- lib/cretonne/src/regalloc/reload.rs | 2 +- lib/cretonne/src/regalloc/solver.rs | 2 +- lib/cretonne/src/regalloc/virtregs.rs | 2 +- lib/cretonne/src/topo_order.rs | 2 +- lib/cretonne/src/verifier/mod.rs | 2 +- 14 files changed, 17 insertions(+), 15 deletions(-) rename lib/cretonne/src/{entity_list.rs => entity/list.rs} (100%) rename lib/cretonne/src/{sparse_map.rs => entity/sparse.rs} (100%) diff --git a/lib/cretonne/src/entity_list.rs b/lib/cretonne/src/entity/list.rs similarity index 100% rename from lib/cretonne/src/entity_list.rs rename to lib/cretonne/src/entity/list.rs diff --git a/lib/cretonne/src/entity/mod.rs b/lib/cretonne/src/entity/mod.rs index 58f900abc4..9a9cc5d72c 100644 --- a/lib/cretonne/src/entity/mod.rs +++ b/lib/cretonne/src/entity/mod.rs @@ -6,12 +6,16 @@ //! Various data structures based on the entity references are defined in sub-modules. mod keys; +mod list; mod map; mod primary; +mod sparse; pub use self::keys::Keys; +pub use self::list::{EntityList, ListPool}; pub use self::map::EntityMap; pub use self::primary::PrimaryMap; +pub use self::sparse::{SparseSet, SparseMap, SparseMapValue}; /// A type wrapping a small integer index should implement `EntityRef` so it can be used as the key /// of an `EntityMap` or `SparseMap`. diff --git a/lib/cretonne/src/sparse_map.rs b/lib/cretonne/src/entity/sparse.rs similarity index 100% rename from lib/cretonne/src/sparse_map.rs rename to lib/cretonne/src/entity/sparse.rs diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index b7c5d898e5..2cd83cf67d 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -17,17 +17,17 @@ use ir::condcodes::*; use ir::types; use isa::RegUnit; -use entity_list; +use entity; use bitset::BitSet; use ref_slice::{ref_slice, ref_slice_mut}; /// Some instructions use an external list of argument values because there is not enough space in /// the 16-byte `InstructionData` struct. These value lists are stored in a memory pool in /// `dfg.value_lists`. -pub type ValueList = entity_list::EntityList; +pub type ValueList = entity::EntityList; /// Memory pool for holding value lists. See `ValueList`. -pub type ValueListPool = entity_list::ListPool; +pub type ValueListPool = entity::ListPool; // Include code generated by `lib/cretonne/meta/gen_instr.py`. This file contains: // diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 6cb6687138..a63b00b3ae 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -18,7 +18,6 @@ pub mod entity; pub mod binemit; pub mod bitset; pub mod dominator_tree; -pub mod entity_list; pub mod flowgraph; pub mod ir; pub mod isa; @@ -27,7 +26,6 @@ pub mod packed_option; pub mod regalloc; pub mod result; pub mod settings; -pub mod sparse_map; pub mod verifier; mod abi; diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 8ed88d47dd..5cd7b22280 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -4,7 +4,7 @@ use ir::{Function, Ebb, Inst, Value, Cursor, CursorBase, Type, InstBuilder, Layo use flowgraph::ControlFlowGraph; use std::collections::HashSet; use dominator_tree::DominatorTree; -use entity_list::{EntityList, ListPool}; +use entity::{EntityList, ListPool}; use loop_analysis::{Loop, LoopAnalysis}; /// Performs the LICM pass by detecting loops within the CFG and moving diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index bee25aa287..09c0dfea07 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -5,7 +5,7 @@ //! instruction, starting at the EBB header. use dominator_tree::DominatorTree; -use entity_list::{EntityList, ListPool}; +use entity::{EntityList, ListPool}; use ir::instructions::BranchInfo; use ir::{Inst, Ebb, Value, DataFlowGraph, Layout, ExpandedProgramPoint}; use partition_slice::partition_slice; diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 973e278f62..124252b4fc 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -175,13 +175,13 @@ //! //! There is some room for improvement. +use entity::SparseMap; use flowgraph::ControlFlowGraph; use ir::dfg::ValueDef; use ir::{Function, Value, Inst, Ebb, Layout, ProgramPoint}; use isa::{TargetIsa, EncInfo}; use regalloc::affinity::Affinity; use regalloc::liverange::LiveRange; -use sparse_map::SparseMap; use std::mem; use std::ops::Index; diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 6b5353910e..e7ba997f6e 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -107,10 +107,10 @@ //! of coalescing, so we would need to roll our own. //! -use std::cmp::Ordering; +use entity::SparseMapValue; use ir::{Inst, Ebb, Value, ProgramPoint, ExpandedProgramPoint, ProgramOrder}; use regalloc::affinity::Affinity; -use sparse_map::SparseMapValue; +use std::cmp::Ordering; /// Global live range of a single SSA value. /// diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 837dbd40d0..81c6c3661b 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -11,6 +11,7 @@ use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; +use entity::{SparseMap, SparseMapValue}; use ir::{Ebb, Inst, Value, Function}; use ir::{InstBuilder, ArgumentType, ArgumentLoc}; use isa::RegClass; @@ -18,7 +19,6 @@ use isa::{TargetIsa, Encoding, EncInfo, RecipeConstraints, ConstraintKind}; use regalloc::affinity::Affinity; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; -use sparse_map::{SparseMap, SparseMapValue}; use topo_order::TopoOrder; /// Reusable data structures for the reload pass. diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 4cec792afe..c443180be2 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -105,10 +105,10 @@ //! appropriate candidate among the set of live register values, add it as a variable and start //! over. +use entity::{SparseMap, SparseMapValue}; use ir::Value; use isa::{RegInfo, RegClass, RegUnit}; use regalloc::allocatable_set::RegSetIter; -use sparse_map::{SparseMap, SparseMapValue}; use std::fmt; use super::AllocatableSet; diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index 722de5a1f4..7c98fd83e0 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -11,7 +11,7 @@ //! If any values in a virtual register are spilled, they will use the same stack slot. This avoids //! memory-to-memory copies when a spilled value is passed as an EBB argument. -use entity_list::{EntityList, ListPool}; +use entity::{EntityList, ListPool}; use entity::{PrimaryMap, EntityMap, Keys}; use ir::Value; use packed_option::PackedOption; diff --git a/lib/cretonne/src/topo_order.rs b/lib/cretonne/src/topo_order.rs index 938e4a876a..0e9210e239 100644 --- a/lib/cretonne/src/topo_order.rs +++ b/lib/cretonne/src/topo_order.rs @@ -1,8 +1,8 @@ //! Topological order of EBBs, according to the dominator tree. +use entity::SparseSet; use dominator_tree::DominatorTree; use ir::{Ebb, Layout}; -use sparse_map::SparseSet; /// Present EBBs in a topological order such that all dominating EBBs are guaranteed to be visited /// before the current EBB. diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 15b5219334..714ad2532d 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -58,6 +58,7 @@ use dbg::DisplayList; use dominator_tree::DominatorTree; +use entity::SparseSet; use flowgraph::ControlFlowGraph; use ir; use ir::entities::AnyEntity; @@ -65,7 +66,6 @@ use ir::instructions::{InstructionFormat, BranchInfo, ResolvedConstraint, CallIn use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, StackSlot, StackSlotKind, GlobalVar, Value, Type, Opcode, ValueLoc, ArgumentLoc}; use isa::TargetIsa; -use sparse_map::SparseSet; use std::error as std_error; use std::fmt::{self, Display, Formatter}; use std::result; From a9238eda7a3e03c95994f63df6efe85e6e38f90c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 18 Aug 2017 17:14:31 -0700 Subject: [PATCH 0979/3084] Clean up the documentation for the entity module. --- lib/cretonne/src/entity/list.rs | 94 ++++++++++++++---------------- lib/cretonne/src/entity/map.rs | 11 ++-- lib/cretonne/src/entity/mod.rs | 29 +++++++-- lib/cretonne/src/entity/primary.rs | 19 +++--- lib/cretonne/src/entity/sparse.rs | 55 +++++++++-------- 5 files changed, 113 insertions(+), 95 deletions(-) diff --git a/lib/cretonne/src/entity/list.rs b/lib/cretonne/src/entity/list.rs index 3685f00eab..e70909bf60 100644 --- a/lib/cretonne/src/entity/list.rs +++ b/lib/cretonne/src/entity/list.rs @@ -1,51 +1,4 @@ //! Small lists of entity references. -//! -//! This module defines an `EntityList` type which provides similar functionality to `Vec`, -//! but with some important differences in the implementation: -//! -//! 1. Memory is allocated from a `ListPool` instead of the global heap. -//! 2. The footprint of an entity list is 4 bytes, compared with the 24 bytes for `Vec`. -//! 3. An entity list doesn't implement `Drop`, leaving it to the pool to manage memory. -//! -//! The list pool is intended to be used as a LIFO allocator. After building up a larger data -//! structure with many list references, the whole thing can be discarded quickly by clearing the -//! pool. -//! -//! # Safety -//! -//! Entity lists are not as safe to use as `Vec`, but they never jeopardize Rust's memory safety -//! guarantees. These are the problems to be aware of: -//! -//! - If you lose track of an entity list, its memory won't be recycled until the pool is cleared. -//! This can cause the pool to grow very large with leaked lists. -//! - If entity lists are used after their pool is cleared, they may contain garbage data, and -//! modifying them may corrupt other lists in the pool. -//! - If an entity list is used with two different pool instances, both pools are likely to become -//! corrupted. -//! -//! # Implementation -//! -//! The `EntityList` itself is designed to have the smallest possible footprint. This is important -//! because it is used inside very compact data structures like `InstructionData`. The list -//! contains only a 32-bit index into the pool's memory vector, pointing to the first element of -//! the list. -//! -//! The pool is just a single `Vec` containing all of the allocated lists. Each list is -//! represented as three contiguous parts: -//! -//! 1. The number of elements in the list. -//! 2. The list elements. -//! 3. Excess capacity elements. -//! -//! The total size of the three parts is always a power of two, and the excess capacity is always -//! as small as possible. This means that shrinking a list may cause the excess capacity to shrink -//! if a smaller power-of-two size becomes available. -//! -//! Both growing and shrinking a list may cause it to be reallocated in the pool vector. -//! -//! The index stored in an `EntityList` points to part 2, the list elements. The value 0 is -//! reserved for the empty list which isn't allocated in the vector. - use entity::EntityRef; use std::hash::{Hash, Hasher}; use std::marker::PhantomData; @@ -53,8 +6,28 @@ use std::mem; /// A small list of entity references allocated from a pool. /// -/// All of the list methods that take a pool reference must be given the same pool reference every -/// time they are called. Otherwise data structures will be corrupted. +/// An `EntityList` type provides similar functionality to `Vec`, but with some important +/// differences in the implementation: +/// +/// 1. Memory is allocated from a `ListPool` instead of the global heap. +/// 2. The footprint of an entity list is 4 bytes, compared with the 24 bytes for `Vec`. +/// 3. An entity list doesn't implement `Drop`, leaving it to the pool to manage memory. +/// +/// The list pool is intended to be used as a LIFO allocator. After building up a larger data +/// structure with many list references, the whole thing can be discarded quickly by clearing the +/// pool. +/// +/// # Safety +/// +/// Entity lists are not as safe to use as `Vec`, but they never jeopardize Rust's memory safety +/// guarantees. These are the problems to be aware of: +/// +/// - If you lose track of an entity list, its memory won't be recycled until the pool is cleared. +/// This can cause the pool to grow very large with leaked lists. +/// - If entity lists are used after their pool is cleared, they may contain garbage data, and +/// modifying them may corrupt other lists in the pool. +/// - If an entity list is used with two different pool instances, both pools are likely to become +/// corrupted. /// /// Entity lists can be cloned, but that operation should only be used as part of cloning the whole /// function they belong to. *Cloning an entity list does not allocate new memory for the clone*. @@ -63,6 +36,29 @@ use std::mem; /// Entity lists can also be hashed and compared for equality, but those operations just panic if, /// they're ever actually called, because it's not possible to compare the contents of the list /// without the pool reference. +/// +/// # Implementation +/// +/// The `EntityList` itself is designed to have the smallest possible footprint. This is important +/// because it is used inside very compact data structures like `InstructionData`. The list +/// contains only a 32-bit index into the pool's memory vector, pointing to the first element of +/// the list. +/// +/// The pool is just a single `Vec` containing all of the allocated lists. Each list is +/// represented as three contiguous parts: +/// +/// 1. The number of elements in the list. +/// 2. The list elements. +/// 3. Excess capacity elements. +/// +/// The total size of the three parts is always a power of two, and the excess capacity is always +/// as small as possible. This means that shrinking a list may cause the excess capacity to shrink +/// if a smaller power-of-two size becomes available. +/// +/// Both growing and shrinking a list may cause it to be reallocated in the pool vector. +/// +/// The index stored in an `EntityList` points to part 2, the list elements. The value 0 is +/// reserved for the empty list which isn't allocated in the vector. #[derive(Clone, Debug)] pub struct EntityList { index: u32, diff --git a/lib/cretonne/src/entity/map.rs b/lib/cretonne/src/entity/map.rs index ca6f1ff7d9..9afc5ff356 100644 --- a/lib/cretonne/src/entity/map.rs +++ b/lib/cretonne/src/entity/map.rs @@ -1,14 +1,17 @@ //! Densely numbered entity references as mapping keys. -//! -//! The `EntityMap` data structure uses the dense index space to implement a map with a vector. -//! Unlike `PrimaryMap`, and `EntityMap` can't be used to allocate entity references. It is used to -//! associate secondary information with entities. use entity::{EntityRef, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; /// A mapping `K -> V` for densely indexed entity references. +/// +/// The `EntityMap` data structure uses the dense index space to implement a map with a vector. +/// Unlike `PrimaryMap`, an `EntityMap` can't be used to allocate entity references. It is used to +/// associate secondary information with entities. +/// +/// The map does not track if an entry for a key has been inserted or not. Instead it behaves as if +/// all keys have a default entry from the beginning. #[derive(Debug, Clone)] pub struct EntityMap where K: EntityRef, diff --git a/lib/cretonne/src/entity/mod.rs b/lib/cretonne/src/entity/mod.rs index 9a9cc5d72c..6702c43dab 100644 --- a/lib/cretonne/src/entity/mod.rs +++ b/lib/cretonne/src/entity/mod.rs @@ -1,9 +1,30 @@ -//! Densely numbered entity references as mapping keys. +//! Array-based data structures using densely numbered entity references as mapping keys. //! -//! This module defines an `EntityRef` trait that should be implemented by reference types wrapping -//! a small integer index. +//! This module defines a number of data structures based on arrays. The arrays are not indexed by +//! `usize` as usual, but by *entity references* which are integers wrapped in new-types. This has +//! a couple advantages: //! -//! Various data structures based on the entity references are defined in sub-modules. +//! - Improved type safety. The various map types accept a specific key type, so there is no +//! confusion about the meaning of an array index, as there is with plain arrays. +//! - Smaller indexes. The normal `usize` index is often 64 bits which is way too large for most +//! purposes. The entity reference types can be smaller, allowing for more compact data +//! structures. +//! +//! The `EntityRef` trait should be implemented by types to be used as indexed. The `entity_impl!` +//! macro provides convenient defaults for types wrapping `u32` which is common. +//! +//! - [`PrimaryMap`](struct.PrimaryMap.html) is used to keep track of a vector of entities, +//! assigning a unique entity reference to each. +//! - [`EntityMap`](struct.EntityMap.html) is used to associate secondary information to an entity. +//! The map is implemented as a simple vector, so it does not keep track of which entities have +//! been inserted. Instead, any unknown entities map to the default value. +//! - [`SparseMap`](struct.SparseMap.html) is used to associate secondary information to a small +//! number of entities. It tracks accurately which entities have been inserted. This is a +//! specialized data structure which can use a lot of memory, so read the documentation before +//! using it. +//! - [`EntityList`](struct.EntityList.html) is a compact representation of lists of entity +//! references allocated from an associated memory pool. It has a much smaller footprint than +//! `Vec`. mod keys; mod list; diff --git a/lib/cretonne/src/entity/primary.rs b/lib/cretonne/src/entity/primary.rs index e98698f117..39af48ff3f 100644 --- a/lib/cretonne/src/entity/primary.rs +++ b/lib/cretonne/src/entity/primary.rs @@ -1,18 +1,17 @@ //! Densely numbered entity references as mapping keys. -//! -//! The `PrimaryMap` data structure uses the dense index space to implement a map with a vector. -//! -//! A primary map contains the main definition of an entity, and it can be used to allocate new -//! entity references with the `push` method. -//! -//! There should only be a single `PrimaryMap` instance for a given `EntityRef` type, otherwise -//! conflicting references will be created. - use entity::{EntityRef, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; -/// A mapping `K -> V` for densely indexed entity references. +/// A primary mapping `K -> V` allocating dense entity references. +/// +/// The `PrimaryMap` data structure uses the dense index space to implement a map with a vector. +/// +/// A primary map contains the main definition of an entity, and it can be used to allocate new +/// entity references with the `push` method. +/// +/// There should only be a single `PrimaryMap` instance for a given `EntityRef` type, otherwise +/// conflicting references will be created. Using unknown keys for indexing will cause a panic. #[derive(Debug, Clone)] pub struct PrimaryMap where K: EntityRef diff --git a/lib/cretonne/src/entity/sparse.rs b/lib/cretonne/src/entity/sparse.rs index 4918675d27..e22df2786a 100644 --- a/lib/cretonne/src/entity/sparse.rs +++ b/lib/cretonne/src/entity/sparse.rs @@ -6,34 +6,6 @@ //! //! > Briggs, Torczon, *An efficient representation for sparse sets*, //! ACM Letters on Programming Languages and Systems, Volume 2, Issue 1-4, March-Dec. 1993. -//! -//! A `SparseMap` map provides: -//! -//! - Memory usage equivalent to `EntityMap` + `Vec`, so much smaller than -//! `EntityMap` for sparse mappings of larger `V` types. -//! - Constant time lookup, slightly slower than `EntityMap`. -//! - A very fast, constant time `clear()` operation. -//! - Fast insert and erase operations. -//! - Stable iteration that is as fast as a `Vec`. -//! -//! # Compared to `EntityMap` -//! -//! When should we use a `SparseMap` instead of a secondary `EntityMap`? First of all, `SparseMap` -//! does not provide the functionality of a primary `EntityMap` which can allocate and assign -//! entity references to objects as they are pushed onto the map. It is only the secondary -//! entity maps that can be replaced with a `SparseMap`. -//! -//! - A secondary entity map requires its values to implement `Default`, and it is a bit loose -//! about creating new mappings to the default value. It doesn't distinguish clearly between an -//! unmapped key and one that maps to the default value. `SparseMap` does not require `Default` -//! values, and it tracks accurately if a key has been mapped or not. -//! - Iterating over the contents of an `EntityMap` is linear in the size of the *key space*, while -//! iterating over a `SparseMap` is linear in the number of elements in the mapping. This is an -//! advantage precisely when the mapping is sparse. -//! - `SparseMap::clear()` is constant time and super-fast. `EntityMap::clear()` is linear in the -//! size of the key space. (Or, rather the required `resize()` call following the `clear()` is). -//! - `SparseMap` requires the values to implement `SparseMapValue` which means that they must -//! contain their own key. use entity::{EntityRef, EntityMap}; use std::mem; @@ -51,6 +23,33 @@ pub trait SparseMapValue { } /// A sparse mapping of entity references. +/// +/// A `SparseMap` map provides: +/// +/// - Memory usage equivalent to `EntityMap` + `Vec`, so much smaller than +/// `EntityMap` for sparse mappings of larger `V` types. +/// - Constant time lookup, slightly slower than `EntityMap`. +/// - A very fast, constant time `clear()` operation. +/// - Fast insert and erase operations. +/// - Stable iteration that is as fast as a `Vec`. +/// +/// # Compared to `EntityMap` +/// +/// When should we use a `SparseMap` instead of a secondary `EntityMap`? First of all, `SparseMap` +/// does not provide the functionality of a `PrimaryMap` which can allocate and assign entity +/// references to objects as they are pushed onto the map. It is only the secondary entity maps +/// that can be replaced with a `SparseMap`. +/// +/// - A secondary entity map assigns a default mapping to all keys. It doesn't distinguish between +/// an unmapped key and one that maps to the default value. `SparseMap` does not require +/// `Default` values, and it tracks accurately if a key has been mapped or not. +/// - Iterating over the contents of an `EntityMap` is linear in the size of the *key space*, while +/// iterating over a `SparseMap` is linear in the number of elements in the mapping. This is an +/// advantage precisely when the mapping is sparse. +/// - `SparseMap::clear()` is constant time and super-fast. `EntityMap::clear()` is linear in the +/// size of the key space. (Or, rather the required `resize()` call following the `clear()` is). +/// - `SparseMap` requires the values to implement `SparseMapValue` which means that they must +/// contain their own key. pub struct SparseMap where K: EntityRef, V: SparseMapValue From 3b71a276328a52976a1c1675e2f7a9650e418f8d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 18 Aug 2017 12:51:54 -0700 Subject: [PATCH 0980/3084] Add heaps to the Cretonne IL. Add preamble syntax for declaring static and dynamic heaps, and update the langref section on heaps. Add IR support for heap references. Remove the heap_load and heap_store as discussed in #144. We will use heap_addr along with native load and store instructions in their place. Add the heap_addr instruction and document its bounds checking semantics. --- cranelift/docs/example.cton | 2 +- cranelift/docs/langref.rst | 92 ++++++++++---- cranelift/filetests/parser/memory.cton | 29 +++++ cranelift/filetests/parser/tiny.cton | 14 --- lib/cretonne/meta/base/entities.py | 3 + lib/cretonne/meta/base/formats.py | 6 +- lib/cretonne/meta/base/instructions.py | 38 ++---- lib/cretonne/src/ir/entities.rs | 14 +++ lib/cretonne/src/ir/function.rs | 4 + lib/cretonne/src/ir/heap.rs | 70 +++++++++++ lib/cretonne/src/ir/instructions.rs | 8 +- lib/cretonne/src/ir/mod.rs | 4 +- lib/cretonne/src/verifier/mod.rs | 11 ++ lib/cretonne/src/write.rs | 6 + lib/reader/src/lexer.rs | 2 + lib/reader/src/parser.rs | 166 +++++++++++++++++++++++-- lib/reader/src/sourcemap.rs | 19 ++- 17 files changed, 405 insertions(+), 83 deletions(-) create mode 100644 lib/cretonne/src/ir/heap.rs diff --git a/cranelift/docs/example.cton b/cranelift/docs/example.cton index 0cbd77a6cd..1a3117c521 100644 --- a/cranelift/docs/example.cton +++ b/cranelift/docs/example.cton @@ -13,7 +13,7 @@ ebb1(v1: i32, v2: i32): ebb2(v5: i32): v6 = imul_imm v5, 4 v7 = iadd v1, v6 - v8 = heap_load.f32 v7 ; array[i] + v8 = load.f32 v7 ; array[i] v9 = fpromote.f64 v8 v10 = stack_load.f64 ss1 v11 = fadd v9, v10 diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 218e1a5e48..e31d8574a9 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -559,44 +559,82 @@ all process memory. Instead, it is given a small set of memory areas to work in, and all accesses are bounds checked. Cretonne models this through the concept of *heaps*. -A heap is declared in the function preamble and can be accessed with restricted -instructions that trap on out-of-bounds accesses. Heap addresses can be smaller -than the native pointer size, for example unsigned :type:`i32` offsets on a -64-bit architecture. +A heap is declared in the function preamble and can be accessed with the +:inst:`heap_addr` instruction that traps on out-of-bounds accesses or returns a +pointer that is guaranteed to trap. Heap addresses can be smaller than the +native pointer size, for example unsigned :type:`i32` offsets on a 64-bit +architecture. -.. inst:: H = heap Name +.. digraph:: static + :align: center + :caption: Heap address space layout - Declare a heap in the function preamble. + node [ + shape=record, + fontsize=10, + fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans" + ] + "static" [label="mapped\npages|unmapped\npages|guard\npages"] - This doesn't allocate memory, it just retrieves a handle to a sandbox from - the runtime environment. +A heap appears as three consecutive ranges of address space: - :arg Name: String identifying the heap in the runtime environment. - :result H: Heap identifier. +1. The *mapped pages* are the usable memory range in the heap. Loads and stores + to this range won't trap. A heap may have a minimum guaranteed size which + means that some mapped pages are always present. +2. The *unmapped pages* is a possibly empty range of address space that may be + mapped in the future when the heap is grown. +3. The *guard pages* is a range of address space that is guaranteed to cause a + trap when accessed. It is used to optimize bounds checking for heap accesses + with a shared base pointer. -.. autoinst:: heap_load -.. autoinst:: heap_store - -When optimizing heap accesses, Cretonne may separate the heap bounds checking -and address computations from the memory accesses. +The *heap bound* is the total size of the mapped and unmapped pages. This is +the bound that :inst:`heap_addr` checks against. Memory accesses inside the +heap bounds can trap if they hit an unmapped page. .. autoinst:: heap_addr -A small example using heaps:: +Two styles of heaps are supported, *static* and *dynamic*. They behave +differently when resized. - function %vdup(i32, i32) { - h1 = heap "main" +Static heaps +~~~~~~~~~~~~ - ebb1(v1: i32, v2: i32): - v3 = heap_load.i32x4 h1, v1, 0 - v4 = heap_addr h1, v2, 32 ; Shared range check for two stores. - store v3, v4, 0 - store v3, v4, 16 - return - } +A *static heap* starts out with all the address space it will ever need, so it +never moves to a different address. At the base address is a number of mapped +pages corresponding to the heap's current size. Then follows a number of +unmapped pages where the heap can grow up to its maximum size. After the +unmapped pages follow the guard pages which are also guaranteed to generate a +trap when accessed. -The final expansion of the :inst:`heap_addr` range check and address conversion -depends on the runtime environment. +.. inst:: H = static Base, min MinBytes, bound BoundBytes, guard GuardBytes + + Declare a static heap in the preamble. + + :arg Base: Global variable holding the heap's base address or + ``reserved_reg``. + :arg MinBytes: Guaranteed minimum heap size in bytes. Accesses below this + size will never trap. + :arg BoundBytes: Fixed heap bound in bytes. This defines the amount of + address space reserved for the heap, not including the guard pages. + :arg GuardBytes: Size of the guard pages in bytes. + +Dynamic heaps +~~~~~~~~~~~~~ + +A *dynamic heap* can be relocated to a different base address when it is +resized, and its bound can move dynamically. The guard pages move when the heap +is resized. The bound of a dynamic heap is stored in a global variable. + +.. inst:: H = dynamic Base, min MinBytes, bound BoundGV, guard GuardBytes + + Declare a dynamic heap in the preamble. + + :arg Base: Global variable holding the heap's base address or + ``reserved_reg``. + :arg MinBytes: Guaranteed minimum heap size in bytes. Accesses below this + size will never trap. + :arg BoundGV: Global variable containing the current heap bound in bytes. + :arg GuardBytes: Size of the guard pages in bytes. Operations diff --git a/cranelift/filetests/parser/memory.cton b/cranelift/filetests/parser/memory.cton index 5f300394b3..fc26169961 100644 --- a/cranelift/filetests/parser/memory.cton +++ b/cranelift/filetests/parser/memory.cton @@ -35,3 +35,32 @@ ebb0: v1 = global_addr.i32 gv1 return v1 } + +; Declare static heaps. +function %sheap(i32) -> i64 { + heap1 = static reserved_reg, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000 + heap2 = static gv5, guard 0x1000, bound 0x1_0000 + gv5 = vmctx+64 + + ; check: $heap1 = static reserved_reg, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + ; check: $heap2 = static $gv5, min 0, bound 0x0001_0000, guard 4096 +ebb0(v1: i32): + v2 = heap_addr.i64 heap1, v1, 0 + ; check: $v2 = heap_addr.i64 $heap1, $v1, 0 + return v2 +} + +; Declare dynamic heaps. +function %dheap(i32) -> i64 { + heap1 = dynamic reserved_reg, min 0x1_0000, bound gv6, guard 0x8000_0000 + heap2 = dynamic gv5, bound gv6, guard 0x1000 + gv5 = vmctx+64 + gv6 = vmctx+72 + + ; check: $heap1 = dynamic reserved_reg, min 0x0001_0000, bound $gv6, guard 0x8000_0000 + ; check: $heap2 = dynamic $gv5, min 0, bound $gv6, guard 4096 +ebb0(v1: i32): + v2 = heap_addr.i64 heap2, v1, 0 + ; check: $v2 = heap_addr.i64 $heap2, $v1, 0 + return v2 +} diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index ecd2525ba2..56dfc53973 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -136,20 +136,6 @@ ebb0: ; nextln: stack_store $v1, $ss10+2 ; nextln: stack_store $v2, $ss2 -; Heap access instructions. -function %heap(i32) { - ; TODO: heap0 = heap %foo -ebb0(v1: i32): - v2 = heap_load.f32 v1 - v3 = heap_load.f32 v1+12 - heap_store v3, v1 -} -; sameln: function %heap(i32) native { -; nextln: ebb0($v1: i32): -; nextln: $v2 = heap_load.f32 $v1 -; nextln: $v3 = heap_load.f32 $v1+12 -; nextln: heap_store $v3, $v1 - ; Memory access instructions. function %memory(i32) { ebb0(v1: i32): diff --git a/lib/cretonne/meta/base/entities.py b/lib/cretonne/meta/base/entities.py index 6a9c1ff5d2..614b4d6284 100644 --- a/lib/cretonne/meta/base/entities.py +++ b/lib/cretonne/meta/base/entities.py @@ -30,3 +30,6 @@ func_ref = EntityRefKind('func_ref', 'An external function.') #: A reference to a jump table declared in the function preamble. jump_table = EntityRefKind( 'jump_table', 'A jump table.', default_member='table') + +#: A reference to a heap declared in the function preamble. +heap = EntityRefKind('heap', 'A heap.') diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 62572f5a02..64a9adb456 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -8,10 +8,11 @@ in this module. from __future__ import absolute_import from cdsl.formats import InstructionFormat from cdsl.operands import VALUE, VARIABLE_ARGS -from .immediates import imm64, uimm8, ieee32, ieee64, offset32, uoffset32 +from .immediates import imm64, uimm8, uimm32, ieee32, ieee64 +from .immediates import offset32, uoffset32 from .immediates import boolean, intcc, floatcc, memflags, regunit from . import entities -from .entities import ebb, sig_ref, func_ref, stack_slot +from .entities import ebb, sig_ref, func_ref, stack_slot, heap Nullary = InstructionFormat() @@ -59,6 +60,7 @@ StackStore = InstructionFormat(VALUE, stack_slot, offset32) # TODO: Add a reference to a `heap` declared in the preamble. HeapLoad = InstructionFormat(VALUE, uoffset32) HeapStore = InstructionFormat(VALUE, VALUE, uoffset32) +HeapAddr = InstructionFormat(heap, VALUE, uimm32) RegMove = InstructionFormat(VALUE, ('src', regunit), ('dst', regunit)) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 4e80641d76..482cb2a12d 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -9,7 +9,7 @@ from cdsl.operands import Operand, VARIABLE_ARGS from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup from base.types import f32, f64, b1 -from base.immediates import imm64, uimm8, ieee32, ieee64, offset32, uoffset32 +from base.immediates import imm64, uimm8, uimm32, ieee32, ieee64, offset32 from base.immediates import boolean, intcc, floatcc, memflags, regunit from base import entities from cdsl.ti import WiderOrEq @@ -360,40 +360,26 @@ global_addr = Instruction( # # WebAssembly bounds-checked heap accesses. # -# TODO: Add a `heap` operand that selects between multiple heaps. -# TODO: Should the immediate offset be a `u32`? -# TODO: Distinguish between `iAddr` for a heap and for a target address? i.e., -# 32-bit WebAssembly on a 64-bit target has two different types. -Offset = Operand('Offset', uoffset32, 'Unsigned offset to effective address') +HeapOffset = TypeVar('HeapOffset', 'An unsigned heap offset', ints=(32, 64)) -heap_load = Instruction( - 'heap_load', r""" - Load a value at the address :math:`p + Offset` in the heap H. - - Trap if the heap access would be out of bounds. - """, - ins=(p, Offset), outs=a, can_load=True) - -heap_store = Instruction( - 'heap_store', r""" - Store a value at the address :math:`p + Offset` in the heap H. - - Trap if the heap access would be out of bounds. - """, - ins=(x, p, Offset), can_store=True) +H = Operand('H', entities.heap) +p = Operand('p', HeapOffset) +Size = Operand('Size', uimm32, 'Size in bytes') heap_addr = Instruction( 'heap_addr', r""" Bounds check and compute absolute address of heap memory. - Verify that the address range ``p .. p + Size - 1`` is valid in the - heap H, and trap if not. + Verify that the offset range ``p .. p + Size - 1`` is in bounds for the + heap H, and generate an absolute address that is safe to dereference. - Convert the heap-relative address in ``p`` to a real absolute address - and return it. + 1. If ``p + Size`` is not greater than the heap bound, return an + absolute address corresponding to a byte offset of ``p`` from the + heap's base address. + 2. If ``p + Size`` is greater than the heap bound, generate a trap. """, - ins=(p, Offset), outs=addr) + ins=(H, p, Size), outs=addr) # # Materializing constants. diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 628a442fdc..434da750e9 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -98,6 +98,11 @@ entity_impl!(FuncRef, "fn"); pub struct SigRef(u32); entity_impl!(SigRef, "sig"); +/// A reference to a heap. +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +pub struct Heap(u32); +entity_impl!(Heap, "heap"); + /// A reference to any of the entities defined in this module. #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] pub enum AnyEntity { @@ -119,6 +124,8 @@ pub enum AnyEntity { FuncRef(FuncRef), /// A function call signature. SigRef(SigRef), + /// A heap. + Heap(Heap), } impl fmt::Display for AnyEntity { @@ -133,6 +140,7 @@ impl fmt::Display for AnyEntity { AnyEntity::JumpTable(r) => r.fmt(f), AnyEntity::FuncRef(r) => r.fmt(f), AnyEntity::SigRef(r) => r.fmt(f), + AnyEntity::Heap(r) => r.fmt(f), } } } @@ -185,6 +193,12 @@ impl From for AnyEntity { } } +impl From for AnyEntity { + fn from(r: Heap) -> AnyEntity { + AnyEntity::Heap(r) + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index c9d0031deb..0f1599d6bd 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -29,6 +29,9 @@ pub struct Function { /// Global variables referenced. pub global_vars: PrimaryMap, + /// Heaps referenced. + pub heaps: PrimaryMap, + /// Jump tables used in this function. pub jump_tables: JumpTables, @@ -61,6 +64,7 @@ impl Function { signature: sig, stack_slots: StackSlots::new(), global_vars: PrimaryMap::new(), + heaps: PrimaryMap::new(), jump_tables: PrimaryMap::new(), dfg: DataFlowGraph::new(), layout: Layout::new(), diff --git a/lib/cretonne/src/ir/heap.rs b/lib/cretonne/src/ir/heap.rs new file mode 100644 index 0000000000..27a970e36c --- /dev/null +++ b/lib/cretonne/src/ir/heap.rs @@ -0,0 +1,70 @@ +//! Heaps. + +use ir::immediates::Imm64; +use ir::GlobalVar; +use std::fmt; + +/// Information about a heap declaration. +#[derive(Clone)] +pub struct HeapData { + /// Method for determining the heap base address. + pub base: HeapBase, + + /// Guaranteed minimum heap size in bytes. Heap accesses before `min_size` don't need bounds + /// checking. + pub min_size: Imm64, + + /// Size in bytes of the guard pages following the heap. + pub guard_size: Imm64, + + /// Heap style, with additional style-specific info. + pub style: HeapStyle, +} + +/// Method for determining the base address of a heap. +#[derive(Clone)] +pub enum HeapBase { + /// The heap base lives in a reserved register. + ReservedReg, + + /// The heap base is in a global variable. + GlobalVar(GlobalVar), +} + +/// Style of heap including style-specific information. +#[derive(Clone)] +pub enum HeapStyle { + /// A dynamic heap can be relocated to a different base address when it is grown. + Dynamic { + /// Global variable holding the current bound of the heap in bytes. + bound_gv: GlobalVar, + }, + + /// A static heap has a fixed base address and a number of not-yet-allocated pages before the + /// guard pages. + Static { + /// Heap bound in bytes. The guard pages are allocated after the bound. + bound: Imm64, + }, +} + +impl fmt::Display for HeapData { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + f.write_str(match self.style { + HeapStyle::Dynamic { .. } => "dynamic", + HeapStyle::Static { .. } => "static", + })?; + + match self.base { + HeapBase::ReservedReg => write!(f, " reserved_reg")?, + HeapBase::GlobalVar(gv) => write!(f, " {}", gv)?, + } + + write!(f, ", min {}", self.min_size)?; + match self.style { + HeapStyle::Dynamic { bound_gv } => write!(f, ", bound {}", bound_gv)?, + HeapStyle::Static { bound } => write!(f, ", bound {}", bound)?, + } + write!(f, ", guard {}", self.guard_size) + } +} diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 2cd83cf67d..ce646150ee 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -12,7 +12,7 @@ use std::ops::{Deref, DerefMut}; use ir; use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef, StackSlot, MemFlags}; -use ir::immediates::{Imm64, Uimm8, Ieee32, Ieee64, Offset32, Uoffset32}; +use ir::immediates::{Imm64, Uimm8, Uimm32, Ieee32, Ieee64, Offset32, Uoffset32}; use ir::condcodes::*; use ir::types; use isa::RegUnit; @@ -199,6 +199,12 @@ pub enum InstructionData { args: [Value; 2], offset: Uoffset32, }, + HeapAddr { + opcode: Opcode, + heap: ir::Heap, + arg: Value, + imm: Uimm32, + }, Load { opcode: Opcode, flags: MemFlags, diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 75e68d110f..b65174245f 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -14,18 +14,20 @@ mod builder; mod extfunc; mod funcname; mod globalvar; +mod heap; mod memflags; mod progpoint; mod valueloc; pub use ir::builder::{InstBuilder, InstBuilderBase, InstInserterBase, InsertBuilder}; pub use ir::dfg::{DataFlowGraph, ValueDef}; -pub use ir::entities::{Ebb, Inst, Value, StackSlot, GlobalVar, JumpTable, FuncRef, SigRef}; +pub use ir::entities::{Ebb, Inst, Value, StackSlot, GlobalVar, JumpTable, FuncRef, SigRef, Heap}; pub use ir::extfunc::{Signature, CallConv, ArgumentType, ArgumentExtension, ArgumentPurpose, ExtFuncData}; pub use ir::funcname::FunctionName; pub use ir::function::Function; pub use ir::globalvar::GlobalVarData; +pub use ir::heap::{HeapData, HeapStyle, HeapBase}; pub use ir::instructions::{Opcode, InstructionData, VariableArgs, ValueList, ValueListPool}; pub use ir::jumptable::JumpTableData; pub use ir::layout::{Layout, CursorBase, Cursor}; diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 714ad2532d..4c8fcd04a5 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -301,6 +301,9 @@ impl<'a> Verifier<'a> { UnaryGlobalVar { global_var, .. } => { self.verify_global_var(inst, global_var)?; } + HeapAddr { heap, .. } => { + self.verify_heap(inst, heap)?; + } // Exhaustive list so we can't forget to add new formats Nullary { .. } | @@ -367,6 +370,14 @@ impl<'a> Verifier<'a> { } } + fn verify_heap(&self, inst: Inst, heap: ir::Heap) -> Result { + if !self.func.heaps.is_valid(heap) { + err!(inst, "invalid heap {}", heap) + } else { + Ok(()) + } + } + fn verify_value_list(&self, inst: Inst, l: &ValueList) -> Result { if !l.is_valid(&self.func.dfg.value_lists) { err!(inst, "invalid value list reference {:?}", l) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 9004546f32..3be1bc3cd1 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -54,6 +54,11 @@ fn write_preamble(w: &mut Write, writeln!(w, " {} = {}", gv, func.global_vars[gv])?; } + for heap in func.heaps.keys() { + any = true; + writeln!(w, " {} = {}", heap, func.heaps[heap])?; + } + // Write out all signatures before functions since function declarations can refer to // signatures. for sig in func.dfg.signatures.keys() { @@ -325,6 +330,7 @@ pub fn write_operands(w: &mut Write, } => write!(w, " {}, {}{}", arg, stack_slot, offset), HeapLoad { arg, offset, .. } => write!(w, " {}{}", arg, offset), HeapStore { args, offset, .. } => write!(w, " {}, {}{}", args[0], args[1], offset), + HeapAddr { heap, arg, imm, .. } => write!(w, " {}, {}, {}", heap, arg, imm), Load { flags, arg, offset, .. } => write!(w, "{} {}{}", flags, arg, offset), Store { flags, diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 7e5fc4f827..5be93676d1 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -38,6 +38,7 @@ pub enum Token<'a> { Ebb(Ebb), // ebb3 StackSlot(u32), // ss3 GlobalVar(u32), // gv3 + Heap(u32), // heap2 JumpTable(u32), // jt2 FuncRef(u32), // fn2 SigRef(u32), // sig2 @@ -310,6 +311,7 @@ impl<'a> Lexer<'a> { "ebb" => Ebb::with_number(number).map(Token::Ebb), "ss" => Some(Token::StackSlot(number)), "gv" => Some(Token::GlobalVar(number)), + "heap" => Some(Token::Heap(number)), "jt" => Some(Token::JumpTable(number)), "fn" => Some(Token::FuncRef(number)), "sig" => Some(Token::SigRef(number)), diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 9ec7547c43..b5e2e12d6b 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -12,9 +12,9 @@ use std::mem; use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, CallConv, StackSlotData, JumpTable, JumpTableData, Signature, ArgumentType, ArgumentExtension, ExtFuncData, SigRef, FuncRef, StackSlot, ValueLoc, ArgumentLoc, MemFlags, - GlobalVar, GlobalVarData}; + GlobalVar, GlobalVarData, Heap, HeapData, HeapStyle, HeapBase}; use cretonne::ir::types::VOID; -use cretonne::ir::immediates::{Imm64, Offset32, Uoffset32, Ieee32, Ieee64}; +use cretonne::ir::immediates::{Imm64, Uimm32, Offset32, Uoffset32, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs}; use cretonne::isa::{self, TargetIsa, Encoding, RegUnit}; @@ -138,7 +138,21 @@ impl<'a> Context<'a> { fn get_gv(&self, number: u32, loc: &Location) -> Result { match self.map.get_gv(number) { Some(gv) => Ok(gv), - None => err!(loc, "undefined stack slot ss{}", number), + None => err!(loc, "undefined global variable gv{}", number), + } + } + + // Allocate a heap slot and add a mapping number -> Heap. + fn add_heap(&mut self, number: u32, data: HeapData, loc: &Location) -> Result<()> { + self.map + .def_heap(number, self.function.heaps.push(data), loc) + } + + // Resolve a reference to a heap. + fn get_heap(&self, number: u32, loc: &Location) -> Result { + match self.map.get_heap(number) { + Some(heap) => Ok(heap), + None => err!(loc, "undefined heap heap{}", number), } } @@ -255,6 +269,23 @@ impl<'a> Context<'a> { } } + // Rewrite references to global variables in heaps. + for heap in self.function.heaps.keys() { + let loc = heap.into(); + match self.function.heaps[heap].base { + HeapBase::GlobalVar(ref mut base) => { + self.map.rewrite_gv(base, loc)?; + } + _ => {} + } + match self.function.heaps[heap].style { + HeapStyle::Dynamic { ref mut bound_gv } => { + self.map.rewrite_gv(bound_gv, loc)?; + } + _ => {} + } + } + Ok(()) } } @@ -381,6 +412,17 @@ impl<'a> Parser<'a> { } } + // Match and consume a global variable reference in the preamble where it can't be rewritten. + // + // Any global variable references appearing in the preamble need to be rewritten after parsing + // the whole preamble. + fn match_gv_preamble(&mut self, err_msg: &str) -> Result { + match GlobalVar::with_number(self.match_gv(err_msg)?) { + Some(gv) => Ok(gv), + None => err!(self.loc, "Invalid global variable number"), + } + } + // Match and consume a function reference. fn match_fn(&mut self, err_msg: &str) -> Result { if let Some(Token::FuncRef(fnref)) = self.token() { @@ -401,6 +443,16 @@ impl<'a> Parser<'a> { } } + // Match and consume a heap reference. + fn match_heap(&mut self, err_msg: &str) -> Result { + if let Some(Token::Heap(heap)) = self.token() { + self.consume(); + Ok(heap) + } else { + err!(self.loc, err_msg) + } + } + // Match and consume a jump table reference. fn match_jt(&mut self) -> Result { if let Some(Token::JumpTable(jt)) = self.token() { @@ -451,6 +503,18 @@ impl<'a> Parser<'a> { } } + // Match and consume a Uimm32 immediate. + fn match_uimm32(&mut self, err_msg: &str) -> Result { + if let Some(Token::Integer(text)) = self.token() { + self.consume(); + // Lexer just gives us raw text that looks like an integer. + // Parse it as an Uimm32 to check for overflow and other issues. + text.parse().map_err(|e| self.error(e)) + } else { + err!(self.loc, err_msg) + } + } + // Match and consume a u8 immediate. // This is used for lane numbers in SIMD vectors. fn match_uimm8(&mut self, err_msg: &str) -> Result { @@ -945,6 +1009,11 @@ impl<'a> Parser<'a> { self.parse_global_var_decl() .and_then(|(num, dat)| ctx.add_gv(num, dat, &self.loc)) } + Some(Token::Heap(..)) => { + self.gather_comments(ctx.function.heaps.next_key()); + self.parse_heap_decl() + .and_then(|(num, dat)| ctx.add_heap(num, dat, &self.loc)) + } Some(Token::SigRef(..)) => { self.gather_comments(ctx.function.dfg.signatures.next_key()); self.parse_signature_decl(ctx.unique_isa) @@ -1018,13 +1087,7 @@ impl<'a> Parser<'a> { } "deref" => { self.match_token(Token::LPar, "expected '(' in 'deref' global variable decl")?; - let base_num = self.match_gv("expected global variable: gv«n»")?; - // The base global variable may not have been declared yet, so create a fake - // reference using the source number. We'll rewrite these later. - let base = match GlobalVar::with_number(base_num) { - Some(gv) => gv, - None => return err!(self.loc, "Invalid global variable number for deref base"), - }; + let base = self.match_gv_preamble("expected global variable: gv«n»")?; self.match_token(Token::RPar, "expected ')' in 'deref' global variable decl")?; let offset = self.optional_offset32()?; GlobalVarData::Deref { base, offset } @@ -1035,6 +1098,75 @@ impl<'a> Parser<'a> { Ok((number, data)) } + // Parse a heap decl. + // + // heap-decl ::= * Heap(heap) "=" heap-desc + // heap-desc ::= heap-style heap-base { "," heap-attr } + // heap-style ::= "static" | "dynamic" + // heap-base ::= "reserved_reg" + // | GlobalVar(base) + // heap-attr ::= "min" Imm64(bytes) + // | "max" Imm64(bytes) + // | "guard" Imm64(bytes) + // + fn parse_heap_decl(&mut self) -> Result<(u32, HeapData)> { + let number = self.match_heap("expected heap number: heap«n»")?; + self.match_token(Token::Equal, "expected '=' in heap declaration")?; + + let style_name = self.match_any_identifier("expected 'static' or 'dynamic'")?; + + // heap-desc ::= heap-style * heap-base { "," heap-attr } + // heap-base ::= * "reserved_reg" + // | * GlobalVar(base) + let base = match self.token() { + Some(Token::Identifier("reserved_reg")) => HeapBase::ReservedReg, + Some(Token::GlobalVar(base_num)) => { + let base_gv = match GlobalVar::with_number(base_num) { + Some(gv) => gv, + None => return err!(self.loc, "invalid global variable number for heap base"), + }; + HeapBase::GlobalVar(base_gv) + } + _ => return err!(self.loc, "expected heap base"), + }; + self.consume(); + + let mut data = HeapData { + base, + min_size: 0.into(), + guard_size: 0.into(), + style: HeapStyle::Static { bound: 0.into() }, + }; + + // heap-desc ::= heap-style heap-base * { "," heap-attr } + while self.optional(Token::Comma) { + match self.match_any_identifier("expected heap attribute name")? { + "min" => { + data.min_size = self.match_imm64("expected integer min size")?; + } + "bound" => { + data.style = match style_name { + "dynamic" => { + HeapStyle::Dynamic { + bound_gv: self.match_gv_preamble("expected gv bound")?, + } + } + "static" => { + HeapStyle::Static { bound: self.match_imm64("expected integer bound")? } + } + t => return err!(self.loc, "unknown heap style '{}'", t), + }; + } + "guard" => { + data.guard_size = self.match_imm64("expected integer guard size")?; + } + t => return err!(self.loc, "unknown heap attribute '{}'", t), + } + } + + Ok((number, data)) + } + // Parse a signature decl. // // signature-decl ::= SigRef(sigref) "=" signature @@ -1802,6 +1934,20 @@ impl<'a> Parser<'a> { offset, } } + InstructionFormat::HeapAddr => { + let heap = self.match_heap("expected heap identifier") + .and_then(|h| ctx.get_heap(h, &self.loc))?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let arg = self.match_value("expected SSA value heap address")?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let imm = self.match_uimm32("expected 32-bit integer size")?; + InstructionData::HeapAddr { + opcode, + heap, + arg, + imm, + } + } InstructionFormat::Load => { let flags = self.optional_memflags(); let addr = self.match_value("expected SSA value address")?; diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index cfd7e85cec..daa6422543 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -9,7 +9,7 @@ use cretonne::entity::EntityRef; use cretonne::ir::entities::AnyEntity; -use cretonne::ir::{StackSlot, GlobalVar, JumpTable, Ebb, Value, SigRef, FuncRef}; +use cretonne::ir::{StackSlot, GlobalVar, Heap, JumpTable, Ebb, Value, SigRef, FuncRef}; use error::{Result, Location}; use lexer::split_entity_name; use std::collections::HashMap; @@ -21,6 +21,7 @@ pub struct SourceMap { ebbs: HashMap, // ebbNN stack_slots: HashMap, // ssNN global_vars: HashMap, // gvNN + heaps: HashMap, // heapNN signatures: HashMap, // sigNN functions: HashMap, // fnNN jump_tables: HashMap, // jtNN @@ -51,6 +52,11 @@ impl SourceMap { self.global_vars.get(&src_num).cloned() } + /// Look up a heap entity by its source number. + pub fn get_heap(&self, src_num: u32) -> Option { + self.heaps.get(&src_num).cloned() + } + /// Look up a signature entity by its source number. pub fn get_sig(&self, src_num: u32) -> Option { self.signatures.get(&src_num).cloned() @@ -82,6 +88,7 @@ impl SourceMap { } "ss" => self.get_ss(num).map(AnyEntity::StackSlot), "gv" => self.get_gv(num).map(AnyEntity::GlobalVar), + "heap" => self.get_heap(num).map(AnyEntity::Heap), "sig" => self.get_sig(num).map(AnyEntity::SigRef), "fn" => self.get_fn(num).map(AnyEntity::FuncRef), "jt" => self.get_jt(num).map(AnyEntity::JumpTable), @@ -161,6 +168,7 @@ pub trait MutableSourceMap { fn def_ebb(&mut self, src: Ebb, entity: Ebb, loc: &Location) -> Result<()>; fn def_ss(&mut self, src_num: u32, entity: StackSlot, loc: &Location) -> Result<()>; fn def_gv(&mut self, src_num: u32, entity: GlobalVar, loc: &Location) -> Result<()>; + fn def_heap(&mut self, src_num: u32, entity: Heap, loc: &Location) -> Result<()>; fn def_sig(&mut self, src_num: u32, entity: SigRef, loc: &Location) -> Result<()>; fn def_fn(&mut self, src_num: u32, entity: FuncRef, loc: &Location) -> Result<()>; fn def_jt(&mut self, src_num: u32, entity: JumpTable, loc: &Location) -> Result<()>; @@ -177,6 +185,7 @@ impl MutableSourceMap for SourceMap { ebbs: HashMap::new(), stack_slots: HashMap::new(), global_vars: HashMap::new(), + heaps: HashMap::new(), signatures: HashMap::new(), functions: HashMap::new(), jump_tables: HashMap::new(), @@ -216,6 +225,14 @@ impl MutableSourceMap for SourceMap { } } + fn def_heap(&mut self, src_num: u32, entity: Heap, loc: &Location) -> Result<()> { + if self.heaps.insert(src_num, entity).is_some() { + err!(loc, "duplicate heap: heap{}", src_num) + } else { + self.def_entity(entity.into(), loc) + } + } + fn def_sig(&mut self, src_num: u32, entity: SigRef, loc: &Location) -> Result<()> { if self.signatures.insert(src_num, entity).is_some() { err!(loc, "duplicate signature: sig{}", src_num) From aae946128b711c630e2815db71f89e7db25fbbb5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 24 Aug 2017 14:04:35 -0700 Subject: [PATCH 0981/3084] Add heap_addr custom legalization. The expansion of a heap_addr instruction depends on the type of heap and its configuration, so this is handled by custom code. Add a couple examples of heap access code to the language reference manual. --- cranelift/docs/cton_lexer.py | 17 +-- cranelift/docs/heapex-dyn.cton | 15 ++ cranelift/docs/heapex-sm32.cton | 14 ++ cranelift/docs/heapex-sm64.cton | 13 ++ cranelift/docs/langref.rst | 31 ++++ .../filetests/isa/intel/legalize-memory.cton | 23 +++ lib/cretonne/meta/base/legalize.py | 1 + lib/cretonne/src/legalizer/heap.rs | 144 ++++++++++++++++++ lib/cretonne/src/legalizer/mod.rs | 2 + 9 files changed, 250 insertions(+), 10 deletions(-) create mode 100644 cranelift/docs/heapex-dyn.cton create mode 100644 cranelift/docs/heapex-sm32.cton create mode 100644 cranelift/docs/heapex-sm64.cton create mode 100644 lib/cretonne/src/legalizer/heap.rs diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/cton_lexer.py index 1024765cbe..bd1a47758f 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/cton_lexer.py @@ -31,16 +31,13 @@ class CretonneLexer(RegexLexer): bygroups(Comment.Single, Comment.Special, Comment.Single)), # Plain comments. (r';.*?$', Comment.Single), - # Strings are in double quotes, support \xx escapes only. - (r'"([^"\\]+|\\[0-9a-fA-F]{2})*"', String), - # A naked function name following 'function' is also a string. - (r'\b(function)([ \t]+)(\w+)\b', - bygroups(Keyword, Whitespace, String.Symbol)), + # Strings are prefixed by % or # with hex. + (r'%\w+|#[0-9a-fA-F]*', String), # Numbers. - (r'[-+]?0[xX][0-9a-fA-F]+', Number.Hex), - (r'[-+]?0[xX][0-9a-fA-F]*\.[0-9a-fA-F]*([pP]\d+)?', Number.Hex), - (r'[-+]?(\d+\.\d+([eE]\d+)?|s?NaN|Inf)', Number.Float), - (r'[-+]?\d+', Number.Integer), + (r'[-+]?0[xX][0-9a-fA-F_]+', Number.Hex), + (r'[-+]?0[xX][0-9a-fA-F_]*\.[0-9a-fA-F_]*([pP]\d+)?', Number.Hex), + (r'[-+]?([0-9_]+\.[0-9_]+([eE]\d+)?|s?NaN|Inf)', Number.Float), + (r'[-+]?[0-9_]+', Number.Integer), # Known attributes. (keywords('uext', 'sext'), Name.Attribute), # Well known value types. @@ -48,7 +45,7 @@ class CretonneLexer(RegexLexer): # v = value # ss = stack slot # jt = jump table - (r'(v|ss|jt)\d+', Name.Variable), + (r'(v|ss|gv|jt|fn|sig|heap)\d+', Name.Variable), # ebb = extended basic block (r'(ebb)\d+', Name.Label), # Match instruction names in context. diff --git a/cranelift/docs/heapex-dyn.cton b/cranelift/docs/heapex-dyn.cton new file mode 100644 index 0000000000..2be801da86 --- /dev/null +++ b/cranelift/docs/heapex-dyn.cton @@ -0,0 +1,15 @@ +test verifier + +function %add_members(i32) -> f32 spiderwasm { + gv0 = vmctx+64 + gv1 = vmctx+72 + heap0 = dynamic gv0, min 0x1000, bound gv1, guard 0 + +ebb0(v0: i32): + v1 = heap_addr.i64 heap0, v0, 20 + v2 = load.f32 v1+16 + v3 = heap_addr.i64 heap0, v0, 24 + v4 = load.f32 v3+20 + v5 = fadd v2, v4 + return v5 +} diff --git a/cranelift/docs/heapex-sm32.cton b/cranelift/docs/heapex-sm32.cton new file mode 100644 index 0000000000..51b0c4cbcb --- /dev/null +++ b/cranelift/docs/heapex-sm32.cton @@ -0,0 +1,14 @@ +test verifier + +function %add_members(i32) -> f32 spiderwasm { + gv0 = vmctx+64 + heap0 = static gv0, min 0x1000, bound 0x10_0000, guard 0x1000 + +ebb0(v0: i32): + v1 = heap_addr.i32 heap0, v0, 1 + v2 = load.f32 v1+16 + v3 = load.f32 v1+20 + v4 = fadd v2, v3 + return v4 +} + diff --git a/cranelift/docs/heapex-sm64.cton b/cranelift/docs/heapex-sm64.cton new file mode 100644 index 0000000000..e29fa9caf5 --- /dev/null +++ b/cranelift/docs/heapex-sm64.cton @@ -0,0 +1,13 @@ +test verifier + +function %add_members(i32) -> f32 spiderwasm { + gv0 = vmctx+64 + heap0 = static gv0, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32): + v1 = heap_addr.i64 heap0, v0, 1 + v2 = load.f32 v1+16 + v3 = load.f32 v1+20 + v4 = fadd v2, v3 + return v4 +} diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index e31d8574a9..c8c7ede4fd 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -636,6 +636,37 @@ is resized. The bound of a dynamic heap is stored in a global variable. :arg BoundGV: Global variable containing the current heap bound in bytes. :arg GuardBytes: Size of the guard pages in bytes. +Heap examples +~~~~~~~~~~~~~ + +The SpiderMonkey VM prefers to use fixed heaps with a 4 GB bound and 2 GB of +guard pages when running WebAssembly code on 64-bit CPUs. The combination of a +4 GB fixed bound and 1-byte bounds checks means that no code needs to be +generated for bounds checks at all: + +.. literalinclude:: heapex-sm64.cton + :language: cton + :lines: 2- + +A static heap can also be used for 32-bit code when the WebAssembly module +declares a small upper bound on its memory. A 1 MB static bound with a single 4 +KB guard page still has opportunities for sharing bounds checking code: + +.. literalinclude:: heapex-sm32.cton + :language: cton + :lines: 2- + +If the upper bound on the heap size is too large, a dynamic heap is required +instead. + +Finally, a runtime environment that simply allocates a heap with +:c:func:`malloc()` may not have any guard pages at all. In that case, full +bounds checking is required for each access: + +.. literalinclude:: heapex-dyn.cton + :language: cton + :lines: 2- + Operations ========== diff --git a/cranelift/filetests/isa/intel/legalize-memory.cton b/cranelift/filetests/isa/intel/legalize-memory.cton index ade217c84d..1aaf044f9f 100644 --- a/cranelift/filetests/isa/intel/legalize-memory.cton +++ b/cranelift/filetests/isa/intel/legalize-memory.cton @@ -27,3 +27,26 @@ ebb1(v1: i64): return v2 ; check: return $v2 } + +; SpiderMonkey VM-style static 4+2 GB heap. +; This eliminates bounds checks completely for offsets < 2GB. +function %staticheap_sm64(i32, i64 vmctx) -> f32 spiderwasm { + gv0 = vmctx+64 + heap0 = static gv0, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v999: i64): + ; check: $ebb0( + v1 = heap_addr.i64 heap0, v0, 1 + ; Boundscheck should be eliminated. + ; Checks here are assuming that no pipehole opts fold the load offsets. + ; nextln: $(xoff=$V) = uextend.i64 $v0 + ; nextln: $(haddr=$V) = iadd_imm $v999, 64 + ; nextln: $(hbase=$V) = load.i64 $haddr + ; nextln: $v1 = iadd $hbase, $xoff + v2 = load.f32 v1+16 + ; nextln: $v2 = load.f32 $v1+16 + v3 = load.f32 v1+20 + ; nextln: $v3 = load.f32 $v1+20 + v4 = fadd v2, v3 + return v4 +} diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index 8c98468518..5db7e6b7c8 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -46,6 +46,7 @@ expand = XFormGroup('expand', """ # Custom expansions for memory objects. expand.custom_legalize(insts.global_addr, 'expand_global_addr') +expand.custom_legalize(insts.heap_addr, 'expand_heap_addr') x = Var('x') y = Var('y') diff --git a/lib/cretonne/src/legalizer/heap.rs b/lib/cretonne/src/legalizer/heap.rs new file mode 100644 index 0000000000..30cd0917c7 --- /dev/null +++ b/lib/cretonne/src/legalizer/heap.rs @@ -0,0 +1,144 @@ +//! Legalization of heaps. +//! +//! This module exports the `expand_heap_addr` function which transforms a `heap_addr` +//! instruction into code that depends on the kind of heap referenced. + +use cursor::{Cursor, FuncCursor}; +use flowgraph::ControlFlowGraph; +use ir::{self, InstBuilder, MemFlags}; +use ir::condcodes::IntCC; + +/// Expand a `heap_addr` instruction according to the definition of the heap. +pub fn expand_heap_addr(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlowGraph) { + // Unpack the instruction. + let (heap, offset, size) = match &func.dfg[inst] { + &ir::InstructionData::HeapAddr { + opcode, + heap, + arg, + imm, + } => { + assert_eq!(opcode, ir::Opcode::HeapAddr); + (heap, arg, imm.into()) + } + _ => panic!("Wanted heap_addr: {}", func.dfg.display_inst(inst, None)), + }; + + match func.heaps[heap].style { + ir::HeapStyle::Dynamic { bound_gv } => { + dynamic_addr(inst, heap, offset, size, bound_gv, func) + } + ir::HeapStyle::Static { bound } => { + static_addr(inst, heap, offset, size, bound.into(), func) + } + } +} + +/// Expand a `heap_addr` for a dynamic heap. +fn dynamic_addr(inst: ir::Inst, + heap: ir::Heap, + offset: ir::Value, + size: u32, + bound_gv: ir::GlobalVar, + func: &mut ir::Function) { + let size = size as i64; + let offset_ty = func.dfg.value_type(offset); + let addr_ty = func.dfg.value_type(func.dfg.first_result(inst)); + let min_size = func.heaps[heap].min_size.into(); + let mut pos = FuncCursor::new(func).at_inst(inst); + + // Start with the bounds check. Trap if `offset + size > bound`. + let bound_addr = pos.ins().global_addr(addr_ty, bound_gv); + let bound = pos.ins().load(offset_ty, MemFlags::new(), bound_addr, 0); + + let oob; + if size == 1 { + // `offset > bound - 1` is the same as `offset >= bound`. + oob = pos.ins() + .icmp(IntCC::UnsignedGreaterThanOrEqual, offset, bound); + } else if size <= min_size { + // We know that bound >= min_size, so here we can compare `offset > bound - size` without + // wrapping. + let adj_bound = pos.ins().iadd_imm(bound, -size); + oob = pos.ins() + .icmp(IntCC::UnsignedGreaterThan, offset, adj_bound); + } else { + // We need an overflow check for the adjusted offset. + let size_val = pos.ins().iconst(offset_ty, size); + let (adj_offset, overflow) = pos.ins().iadd_cout(offset, size_val); + pos.ins().trapnz(overflow); + oob = pos.ins() + .icmp(IntCC::UnsignedGreaterThan, adj_offset, bound); + } + pos.ins().trapnz(oob); + + offset_addr(inst, heap, addr_ty, offset, offset_ty, pos.func); +} + +/// Expand a `heap_addr` for a static heap. +fn static_addr(inst: ir::Inst, + heap: ir::Heap, + offset: ir::Value, + size: u32, + bound: i64, + func: &mut ir::Function) { + let size = size as i64; + let offset_ty = func.dfg.value_type(offset); + let addr_ty = func.dfg.value_type(func.dfg.first_result(inst)); + let mut pos = FuncCursor::new(func).at_inst(inst); + + // Start with the bounds check. Trap if `offset + size > bound`. + if size > bound { + // This will simply always trap since `offset >= 0`. + pos.ins().trap(); + pos.func.dfg.replace(inst).iconst(addr_ty, 0); + return; + } + + // Check `offset > limit` which is now known non-negative. + let limit = bound - size; + + // We may be able to omit the check entirely for 32-bit offsets if the heap bound is 4 GB or + // more. + if offset_ty != ir::types::I32 || limit < 0xffff_ffff { + let oob = if limit & 1 == 1 { + // Prefer testing `offset >= limit - 1` when limit is odd because an even number is + // likely to be a convenient constant on ARM and other RISC architectures. + pos.ins() + .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, offset, limit - 1) + } else { + pos.ins() + .icmp_imm(IntCC::UnsignedGreaterThan, offset, limit) + }; + pos.ins().trapnz(oob); + } + + offset_addr(inst, heap, addr_ty, offset, offset_ty, pos.func); +} + +/// Emit code for the base address computation of a `heap_addr` instruction. +/// +/// +fn offset_addr(inst: ir::Inst, + heap: ir::Heap, + addr_ty: ir::Type, + mut offset: ir::Value, + offset_ty: ir::Type, + func: &mut ir::Function) { + let mut pos = FuncCursor::new(func).at_inst(inst); + + // Convert `offset` to `addr_ty`. + if offset_ty != addr_ty { + offset = pos.ins().uextend(addr_ty, offset); + } + + // Add the heap base address base + match pos.func.heaps[heap].base { + ir::HeapBase::ReservedReg => unimplemented!(), + ir::HeapBase::GlobalVar(base_gv) => { + let base_addr = pos.ins().global_addr(addr_ty, base_gv); + let base = pos.ins().load(addr_ty, MemFlags::new(), base_addr, 0); + pos.func.dfg.replace(inst).iadd(base, offset); + } + } +} diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 230c7adc7e..65365018d2 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -22,9 +22,11 @@ use bitset::BitSet; mod boundary; mod globalvar; +mod heap; mod split; use self::globalvar::expand_global_addr; +use self::heap::expand_heap_addr; /// Legalize `func` for `isa`. /// From fecbcbb7b48e72da9034b1d1906482a080f4fabe Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 25 Aug 2017 10:36:25 -0700 Subject: [PATCH 0982/3084] Drop the domtree argument to legalize_function(). Future legalization patterns will have the ability to mutate the flowgraph, so the domtree's list of RPO blocks is not a good guide for iteration. Use the layout order instead. This will pick up any new EBBs inserted. --- lib/cretonne/src/context.rs | 2 +- lib/cretonne/src/legalizer/mod.rs | 14 ++++---------- 2 files changed, 5 insertions(+), 11 deletions(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 9967f1e11e..7fec7ea0fb 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -103,7 +103,7 @@ impl Context { /// Run the legalizer for `isa` on the function. pub fn legalize(&mut self, isa: &TargetIsa) -> CtonResult { - legalize_function(&mut self.func, &mut self.cfg, &self.domtree, isa); + legalize_function(&mut self.func, &mut self.cfg, isa); self.verify_if(isa) } diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 65365018d2..8909366e01 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -14,7 +14,6 @@ //! from the encoding recipes, and solved later by the register allocator. use cursor::{Cursor, FuncCursor}; -use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir; use isa::TargetIsa; @@ -33,21 +32,16 @@ use self::heap::expand_heap_addr; /// - Transform any instructions that don't have a legal representation in `isa`. /// - Fill out `func.encodings`. /// -pub fn legalize_function(func: &mut ir::Function, - cfg: &mut ControlFlowGraph, - domtree: &DominatorTree, - isa: &TargetIsa) { +pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, isa: &TargetIsa) { boundary::legalize_signatures(func, isa); func.encodings.resize(func.dfg.num_insts()); let mut pos = FuncCursor::new(func); - // Process EBBs in a reverse post-order. This minimizes the number of split instructions we - // need. - for &ebb in domtree.cfg_postorder().iter().rev() { - pos.goto_top(ebb); - + // Process EBBs in layout order. Some legalization actions may split the current EBB or append + // new ones to the end. We need to make sure we visit those new EBBs too. + while let Some(_ebb) = pos.next_ebb() { // Keep track of the cursor position before the instruction being processed, so we can // double back when replacing instructions. let mut prev_pos = pos.position(); From 6d9198d55f84c7bf265af826a9fb94c0af47486e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 28 Aug 2017 11:13:53 -0700 Subject: [PATCH 0983/3084] Recompute the dominator tree on demand. The legalizer can invalidate the dominator tree, but we don't actually need a dominator tree during legalization, so defer the construction of the domtree. - Add an "invalid" state to the dominator tree along with clear() and is_valid() methods to test it. - Invalidate the dominator tree as part of legalization. - Ensure that a valid dominator tree exists before the passes that need it. Together these features add up to a manual invalidation mechanism for the dominator tree. --- lib/cretonne/src/context.rs | 13 ++++++++++++- lib/cretonne/src/dominator_tree.rs | 22 +++++++++++++++++++--- lib/cretonne/src/verifier/mod.rs | 4 +++- 3 files changed, 34 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 7fec7ea0fb..6f80f16056 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -63,7 +63,7 @@ impl Context { /// /// Returns the size of the function's code. pub fn compile(&mut self, isa: &TargetIsa) -> Result { - self.flowgraph(); + self.cfg.compute(&self.func); self.verify_if(isa)?; self.legalize(isa)?; @@ -103,6 +103,8 @@ impl Context { /// Run the legalizer for `isa` on the function. pub fn legalize(&mut self, isa: &TargetIsa) -> CtonResult { + // Legalization invalidates the domtree by mutating the CFG. + self.domtree.clear(); legalize_function(&mut self.func, &mut self.cfg, isa); self.verify_if(isa) } @@ -113,6 +115,13 @@ impl Context { self.domtree.compute(&self.func, &self.cfg); } + /// Ensure that a valid domtree exists. + pub fn ensure_domtree(&mut self) { + if !self.domtree.is_valid() { + self.domtree.compute(&self.func, &self.cfg); + } + } + /// Perform simple GVN on the function. pub fn simple_gvn(&mut self) -> CtonResult { do_simple_gvn(&mut self.func, &mut self.cfg); @@ -123,6 +132,7 @@ impl Context { /// Perform LICM on the function. pub fn licm(&mut self) -> CtonResult { + self.ensure_domtree(); do_licm(&mut self.func, &mut self.cfg, &mut self.domtree, @@ -132,6 +142,7 @@ impl Context { /// Run the register allocator. pub fn regalloc(&mut self, isa: &TargetIsa) -> CtonResult { + self.ensure_domtree(); self.regalloc .run(isa, &mut self.func, &self.cfg, &self.domtree) } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index bb80bd3687..067142b0f2 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -205,14 +205,30 @@ impl DominatorTree { self.compute_domtree(func, cfg); } + /// Clear the data structures used to represent the dominator tree. This will leave the tree in + /// a state where `is_valid()` returns false. + pub fn clear(&mut self) { + self.nodes.clear(); + self.postorder.clear(); + assert!(self.stack.is_empty()); + } + + /// Check if the dominator tree is in a valid state. + /// + /// Note that this doesn't perform any kind of validity checks. It simply checks if the + /// `compute()` method has been called since the last `clear()`. It does not check that the + /// dominator tree is consistent + /// with the CFG> + pub fn is_valid(&self) -> bool { + !self.nodes.is_empty() + } + /// Reset all internal data structures and compute a post-order for `cfg`. /// /// This leaves `rpo_number == 1` for all reachable EBBs, 0 for unreachable ones. fn compute_postorder(&mut self, func: &Function, cfg: &ControlFlowGraph) { - self.nodes.clear(); + self.clear(); self.nodes.resize(func.dfg.num_ebbs()); - self.postorder.clear(); - assert!(self.stack.is_empty()); // During this algorithm only, use `rpo_number` to hold the following state: // diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 4c8fcd04a5..ca1f2a8f09 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -134,7 +134,9 @@ pub fn verify_context(func: &Function, -> Result { let verifier = Verifier::new(func, isa); verifier.cfg_integrity(cfg)?; - verifier.domtree_integrity(domtree)?; + if domtree.is_valid() { + verifier.domtree_integrity(domtree)?; + } verifier.run() } From 217434b474b76f855b237dee921d2841d2bea8cb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 25 Aug 2017 12:40:43 -0700 Subject: [PATCH 0984/3084] Add custom legalization for conditional traps. The expansion of these instructions requires the CFG to be modified, something the Python XForms can't yet do. --- .../filetests/isa/intel/legalize-custom.cton | 30 ++++++++++++ lib/cretonne/meta/base/legalize.py | 5 ++ lib/cretonne/src/legalizer/mod.rs | 48 ++++++++++++++++++- 3 files changed, 82 insertions(+), 1 deletion(-) create mode 100644 cranelift/filetests/isa/intel/legalize-custom.cton diff --git a/cranelift/filetests/isa/intel/legalize-custom.cton b/cranelift/filetests/isa/intel/legalize-custom.cton new file mode 100644 index 0000000000..ae63a56e0a --- /dev/null +++ b/cranelift/filetests/isa/intel/legalize-custom.cton @@ -0,0 +1,30 @@ +; Test the custom legalizations. +test legalizer +isa intel +set is_64bit +isa intel + +; regex: V=v\d+ +; regex: EBB=ebb\d+ + +function %cond_trap(i32) { +ebb0(v1: i32): + trapz v1 + return + ; check: $ebb0($v1: i32): + ; nextln: brnz $v1, $(new=$EBB) + ; nextln: trap + ; check: $new: + ; nextln: return +} + +function %cond_trap2(i32) { +ebb0(v1: i32): + trapnz v1 + return + ; check: $ebb0($v1: i32): + ; nextln: brz $v1, $(new=$EBB) + ; nextln: trap + ; check: $new: + ; nextln: return +} diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index 5db7e6b7c8..2bbdd7247a 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -48,6 +48,11 @@ expand = XFormGroup('expand', """ expand.custom_legalize(insts.global_addr, 'expand_global_addr') expand.custom_legalize(insts.heap_addr, 'expand_heap_addr') +# Custom expansions that need to change the CFG. +# TODO: Add sufficient XForm syntax that we don't need to hand-code these. +expand.custom_legalize(insts.trapz, 'expand_cond_trap') +expand.custom_legalize(insts.trapnz, 'expand_cond_trap') + x = Var('x') y = Var('y') a = Var('a') diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 8909366e01..da4a2b93fa 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -15,7 +15,7 @@ use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; -use ir; +use ir::{self, InstBuilder}; use isa::TargetIsa; use bitset::BitSet; @@ -96,3 +96,49 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is // // Concretely, this defines private functions `narrow()`, and `expand()`. include!(concat!(env!("OUT_DIR"), "/legalizer.rs")); + +/// Custom expansion for conditional trap instructions. +/// TODO: Add CFG support to the Python patterns so we won't have to do this. +fn expand_cond_trap(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { + // Parse the instruction. + let trapz; + let arg = match func.dfg[inst] { + ir::InstructionData::Unary { opcode, arg } => { + // We want to branch *over* an unconditional trap. + trapz = match opcode { + ir::Opcode::Trapz => true, + ir::Opcode::Trapnz => false, + _ => panic!("Expected cond trap: {}", func.dfg.display_inst(inst, None)), + }; + arg + } + _ => panic!("Expected cond trap: {}", func.dfg.display_inst(inst, None)), + }; + + // Split the EBB after `inst`: + // + // trapnz arg + // + // Becomes: + // + // brz arg, new_ebb + // trap + // new_ebb: + // + let old_ebb = func.layout.pp_ebb(inst); + let new_ebb = func.dfg.make_ebb(); + if trapz { + func.dfg.replace(inst).brnz(arg, new_ebb, &[]); + } else { + func.dfg.replace(inst).brz(arg, new_ebb, &[]); + } + + let mut pos = FuncCursor::new(func).at_inst(inst); + pos.next_inst(); + pos.ins().trap(); + pos.insert_ebb(new_ebb); + + // Finally update the CFG. + cfg.recompute_ebb(pos.func, old_ebb); + cfg.recompute_ebb(pos.func, new_ebb); +} From 2201e6249e40fb7b167b9609592f0b01887c9638 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 28 Aug 2017 14:54:35 -0700 Subject: [PATCH 0985/3084] Add Intel encodings for brz.b1 and brnz.b1. Use these encodings to test trapz.b1 and trapnz.b1. When a b1 value is stored in a register, only the low 8 bits are valid. This is so we can use the various setCC instructions to generate the b1 registers. --- .../filetests/isa/intel/legalize-custom.cton | 24 +++++++++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 12 ++++++++++ lib/cretonne/meta/isa/intel/recipes.py | 16 +++++++++++++ 3 files changed, 52 insertions(+) diff --git a/cranelift/filetests/isa/intel/legalize-custom.cton b/cranelift/filetests/isa/intel/legalize-custom.cton index ae63a56e0a..e8af47a460 100644 --- a/cranelift/filetests/isa/intel/legalize-custom.cton +++ b/cranelift/filetests/isa/intel/legalize-custom.cton @@ -28,3 +28,27 @@ ebb0(v1: i32): ; check: $new: ; nextln: return } + +function %cond_trap_b1(i32) { +ebb0(v1: i32): + v2 = icmp_imm eq v1, 6 + trapz v2 + return + ; check: $ebb0($v1: i32): + ; check: brnz $v2, $(new=$EBB) + ; nextln: trap + ; check: $new: + ; nextln: return +} + +function %cond_trap2_b1(i32) { +ebb0(v1: i32): + v2 = icmp_imm eq v1, 6 + trapnz v2 + return + ; check: $ebb0($v1: i32): + ; check: brz $v2, $(new=$EBB) + ; nextln: trap + ; check: $new: + ; nextln: return +} diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 39b6812058..3b2a91a1f2 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -22,12 +22,14 @@ except ImportError: I32.legalize_type( default=narrow, + b1=expand, i32=intel_expand, f32=expand, f64=expand) I64.legalize_type( default=narrow, + b1=expand, i32=intel_expand, i64=intel_expand, f32=expand, @@ -238,6 +240,16 @@ I64.enc(base.jump, *r.jmpd(0xe9)) enc_i32_i64(base.brz, r.tjccb, 0x74) enc_i32_i64(base.brnz, r.tjccb, 0x75) +# Branch on a b1 value in a register only looks at the low 8 bits. See also +# bint encodings below. +I32.enc(base.brz.b1, *r.t8jccb_abcd(0x74)) +I64.enc(base.brz.b1, *r.t8jccb_abcd.rex(0x74)) +I64.enc(base.brz.b1, *r.t8jccb_abcd(0x74)) +I32.enc(base.brnz.b1, *r.t8jccb_abcd(0x75)) +I64.enc(base.brnz.b1, *r.t8jccb_abcd.rex(0x75)) +I64.enc(base.brnz.b1, *r.t8jccb_abcd(0x75)) + + # # Trap as ud2 # diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 3e038e1208..1302350f7c 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -515,6 +515,22 @@ tjccb = TailRecipe( disp1(destination, func, sink); ''') +# 8-bit test-and-branch. +# +# Same as tjccb, but only looks at the low 8 bits of the register, for b1 +# types. +t8jccb_abcd = TailRecipe( + 't8jccb_abcd', Branch, size=1 + 2, ins=ABCD, outs=(), + branch_range=(2, 8), + emit=''' + // test8 r, r. + PUT_OP(0x84, rex2(in_reg0, in_reg0), sink); + modrm_rr(in_reg0, in_reg0, sink); + // Jcc instruction. + sink.put1(bits as u8); + disp1(destination, func, sink); + ''') + # Comparison that produces a `b1` result in a GPR. # # This is a macro of a `cmp` instruction followed by a `setCC` instruction. From e8276ed965266530b6a1ab3503e19a6c1c923b6f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 28 Aug 2017 14:34:46 -0700 Subject: [PATCH 0986/3084] Add more heap expansion tests. --- .../filetests/isa/intel/legalize-memory.cton | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/cranelift/filetests/isa/intel/legalize-memory.cton b/cranelift/filetests/isa/intel/legalize-memory.cton index 1aaf044f9f..5db903aa52 100644 --- a/cranelift/filetests/isa/intel/legalize-memory.cton +++ b/cranelift/filetests/isa/intel/legalize-memory.cton @@ -4,6 +4,7 @@ set is_64bit isa intel ; regex: V=v\d+ +; regex: EBB=ebb\d+ function %vmctx(i64 vmctx) -> i64 { gv1 = vmctx-16 @@ -50,3 +51,27 @@ ebb0(v0: i32, v999: i64): v4 = fadd v2, v3 return v4 } + +; SpiderMonkey VM-style static 4+2 GB heap. +; Offsets >= 2 GB do require a boundscheck. +function %staticheap_sm64(i32, i64 vmctx) -> f32 spiderwasm { + gv0 = vmctx+64 + heap0 = static gv0, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v999: i64): + ; check: $ebb0( + v1 = heap_addr.i64 heap0, v0, 0x8000_0000 + ; Boundscheck code + ; check: $(oob=$V) = icmp + ; nextln: brz $oob, $(ok=$EBB) + ; nextln: trap + ; check: $ok: + ; Checks here are assuming that no pipehole opts fold the load offsets. + ; nextln: $(xoff=$V) = uextend.i64 $v0 + ; nextln: $(haddr=$V) = iadd_imm.i64 $v999, 64 + ; nextln: $(hbase=$V) = load.i64 $haddr + ; nextln: $v1 = iadd $hbase, $xoff + v2 = load.f32 v1+0x7fff_ffff + ; nextln: $v2 = load.f32 $v1+0x7fff_ffff + return v2 +} From ee9989c4b9efb9bf572a783c7fcaa00fca2a25f9 Mon Sep 17 00:00:00 2001 From: Denis Merigoux Date: Thu, 10 Aug 2017 16:05:04 -0700 Subject: [PATCH 0987/3084] Dumped code from the wasm2cretonne repo. Integrated wasm test suite translation as cretonne test Fixes #146. Fixes #143. --- cranelift/Cargo.toml | 6 +- cranelift/src/cton-util.rs | 22 +- cranelift/src/wasm.rs | 227 +++++ cranelift/test-all.sh | 3 +- cranelift/wasmtests/arith.wast | 13 + cranelift/wasmtests/call.wast | 10 + cranelift/wasmtests/fibonacci.wast | 22 + cranelift/wasmtests/globals.wast | 8 + cranelift/wasmtests/memory.wast | 11 + lib/wasm/.gitignore | 3 + lib/wasm/Cargo.toml | 13 + lib/wasm/src/code_translator.rs | 1385 +++++++++++++++++++++++++++ lib/wasm/src/lib.rs | 27 + lib/wasm/src/module_translator.rs | 288 ++++++ lib/wasm/src/runtime/dummy.rs | 93 ++ lib/wasm/src/runtime/mod.rs | 5 + lib/wasm/src/runtime/spec.rs | 61 ++ lib/wasm/src/sections_translator.rs | 372 +++++++ lib/wasm/src/translation_utils.rs | 138 +++ lib/wasm/tests/testsuite.rs | 102 ++ 20 files changed, 2804 insertions(+), 5 deletions(-) create mode 100644 cranelift/src/wasm.rs create mode 100644 cranelift/wasmtests/arith.wast create mode 100644 cranelift/wasmtests/call.wast create mode 100644 cranelift/wasmtests/fibonacci.wast create mode 100644 cranelift/wasmtests/globals.wast create mode 100644 cranelift/wasmtests/memory.wast create mode 100644 lib/wasm/.gitignore create mode 100644 lib/wasm/Cargo.toml create mode 100644 lib/wasm/src/code_translator.rs create mode 100644 lib/wasm/src/lib.rs create mode 100644 lib/wasm/src/module_translator.rs create mode 100644 lib/wasm/src/runtime/dummy.rs create mode 100644 lib/wasm/src/runtime/mod.rs create mode 100644 lib/wasm/src/runtime/spec.rs create mode 100644 lib/wasm/src/sections_translator.rs create mode 100644 lib/wasm/src/translation_utils.rs create mode 100644 lib/wasm/tests/testsuite.rs diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 6df15bd86c..3d5080974c 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -15,11 +15,15 @@ path = "src/cton-util.rs" [dependencies] cretonne = { path = "lib/cretonne" } cretonne-reader = { path = "lib/reader" } -cretonne-frontend = { path ="lib/frontend" } +cretonne-frontend = { path = "lib/frontend" } +cretonne-wasm = { path = "lib/wasm" } filecheck = { path = "lib/filecheck" } +wasmparser = "0.6.1" docopt = "0.8.0" serde = "1.0.8" serde_derive = "1.0.8" num_cpus = "1.5.1" +tempdir="0.3.5" +term = "0.4.6" [workspace] diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 28b4b90de0..e85eb138c7 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -1,12 +1,16 @@ #[macro_use(dbg)] extern crate cretonne; extern crate cton_reader; +extern crate cretonne_wasm; +extern crate wasmparser; extern crate docopt; extern crate serde; #[macro_use] extern crate serde_derive; extern crate filecheck; extern crate num_cpus; +extern crate tempdir; +extern crate term; use cretonne::VERSION; use docopt::Docopt; @@ -18,6 +22,7 @@ mod filetest; mod cat; mod print_cfg; mod rsfilecheck; +mod wasm; const USAGE: &str = " Cretonne code generator utility @@ -27,12 +32,15 @@ Usage: cton-util cat ... cton-util filecheck [-v] cton-util print-cfg ... + cton-util wasm [-cvo] ... cton-util --help | --version Options: - -v, --verbose be more verbose - -h, --help print this help message - --version print the Cretonne version + -v, --verbose be more verbose + -c, --check checks the correctness of Cretonne IL translated from WebAssembly + -o, --optimize runs otpimization passes on translated WebAssembly functions + -h, --help print this help message + --version print the Cretonne version "; @@ -42,7 +50,10 @@ struct Args { cmd_cat: bool, cmd_filecheck: bool, cmd_print_cfg: bool, + cmd_wasm: bool, arg_file: Vec, + flag_check: bool, + flag_optimize: bool, flag_verbose: bool, } @@ -69,6 +80,11 @@ fn cton_util() -> CommandResult { rsfilecheck::run(args.arg_file, args.flag_verbose) } else if args.cmd_print_cfg { print_cfg::run(args.arg_file) + } else if args.cmd_wasm { + wasm::run(args.arg_file, + args.flag_verbose, + args.flag_optimize, + args.flag_check) } else { // Debugging / shouldn't happen with proper command line handling above. Err(format!("Unhandled args: {:?}", args)) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs new file mode 100644 index 0000000000..953bba50c5 --- /dev/null +++ b/cranelift/src/wasm.rs @@ -0,0 +1,227 @@ +//! CLI tool to use the functions provided by crates [wasm2cretonne](../wasm2cretonne/index.html) +//! and [wasmstandalone](../wasmstandalone/index.html). +//! +//! Reads Wasm binary files (one Wasm module per file), translates the functions' code to Cretonne +//! IL. Can also executes the `start` function of the module by laying out the memories, globals +//! and tables, then emitting the translated code with hardcoded addresses to memory. + +use cretonne_wasm::{translate_module, FunctionTranslation, DummyRuntime, WasmRuntime}; +use std::path::PathBuf; +use cretonne::loop_analysis::LoopAnalysis; +use cretonne::flowgraph::ControlFlowGraph; +use cretonne::dominator_tree::DominatorTree; +use cretonne::Context; +use cretonne::result::CtonError; +use cretonne::ir; +use cretonne::ir::entities::AnyEntity; +use cretonne::isa::TargetIsa; +use cretonne::verifier; +use std::fs::File; +use std::error::Error; +use std::io; +use std::io::BufReader; +use std::io::prelude::*; +use std::path::Path; +use std::process::Command; +use tempdir::TempDir; +use term; + +macro_rules! vprintln { + ($x: expr, $($tts:tt)*) => { + if $x { + println!($($tts)*); + } + } +} + +macro_rules! vprint { + ($x: expr, $($tts:tt)*) => { + if $x { + print!($($tts)*); + } + } +} + +fn read_wasm_file(path: PathBuf) -> Result, io::Error> { + let mut buf: Vec = Vec::new(); + let file = File::open(path)?; + let mut buf_reader = BufReader::new(file); + buf_reader.read_to_end(&mut buf)?; + Ok(buf) +} + + +pub fn run(files: Vec, + flag_verbose: bool, + flag_optimize: bool, + flag_check: bool) + -> Result<(), String> { + for filename in files.iter() { + let path = Path::new(&filename); + let name = String::from(path.as_os_str().to_string_lossy()); + match handle_module(flag_verbose, + flag_optimize, + flag_check, + path.to_path_buf(), + name) { + Ok(()) => {} + Err(message) => return Err(message), + } + } + Ok(()) +} + +fn handle_module(flag_verbose: bool, + flag_optimize: bool, + flag_check: bool, + path: PathBuf, + name: String) + -> Result<(), String> { + let mut terminal = term::stdout().unwrap(); + terminal.fg(term::color::YELLOW).unwrap(); + vprint!(flag_verbose, "Handling: "); + terminal.reset().unwrap(); + vprintln!(flag_verbose, "\"{}\"", name); + terminal.fg(term::color::MAGENTA).unwrap(); + vprint!(flag_verbose, "Translating..."); + terminal.reset().unwrap(); + let data = match path.extension() { + None => { + return Err(String::from("the file extension is not wasm or wast")); + } + Some(ext) => { + match ext.to_str() { + Some("wasm") => { + match read_wasm_file(path.clone()) { + Ok(data) => data, + Err(err) => { + return Err(String::from(err.description())); + } + } + } + Some("wast") => { + let tmp_dir = TempDir::new("wasm2cretonne").unwrap(); + let file_path = tmp_dir.path().join("module.wasm"); + File::create(file_path.clone()).unwrap(); + Command::new("wast2wasm") + .arg(path.clone()) + .arg("-o") + .arg(file_path.to_str().unwrap()) + .output() + .or_else(|e| if let io::ErrorKind::NotFound = e.kind() { + return Err(String::from("wast2wasm not found")); + } else { + return Err(String::from(e.description())); + }) + .unwrap(); + match read_wasm_file(file_path) { + Ok(data) => data, + Err(err) => { + return Err(String::from(err.description())); + } + } + } + None | Some(&_) => { + return Err(String::from("the file extension is not wasm or wast")); + } + } + } + }; + let mut dummy_runtime = DummyRuntime::new(); + let translation = { + let mut runtime: &mut WasmRuntime = &mut dummy_runtime; + match translate_module(&data, runtime) { + Ok(x) => x, + Err(string) => { + return Err(string); + } + } + }; + terminal.fg(term::color::GREEN).unwrap(); + vprintln!(flag_verbose, " ok"); + terminal.reset().unwrap(); + if flag_check { + terminal.fg(term::color::MAGENTA).unwrap(); + vprint!(flag_verbose, "Checking... "); + terminal.reset().unwrap(); + for func in translation.functions.iter() { + let il = match func { + &FunctionTranslation::Import() => continue, + &FunctionTranslation::Code { ref il, .. } => il.clone(), + }; + match verifier::verify_function(&il, None) { + Ok(()) => (), + Err(err) => return Err(pretty_verifier_error(&il, None, err)), + } + } + terminal.fg(term::color::GREEN).unwrap(); + vprintln!(flag_verbose, " ok"); + terminal.reset().unwrap(); + } + if flag_optimize { + terminal.fg(term::color::MAGENTA).unwrap(); + vprint!(flag_verbose, "Optimizing... "); + terminal.reset().unwrap(); + for func in translation.functions.iter() { + let mut il = match func { + &FunctionTranslation::Import() => continue, + &FunctionTranslation::Code { ref il, .. } => il.clone(), + }; + let mut loop_analysis = LoopAnalysis::new(); + let mut cfg = ControlFlowGraph::new(); + cfg.compute(&il); + let mut domtree = DominatorTree::new(); + domtree.compute(&mut il, &cfg); + loop_analysis.compute(&mut il, &mut cfg, &mut domtree); + let mut context = Context::new(); + context.func = il; + context.cfg = cfg; + context.domtree = domtree; + context.loop_analysis = loop_analysis; + match verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) { + Ok(()) => (), + Err(err) => { + return Err(pretty_verifier_error(&context.func, None, err)); + } + }; + match context.licm() { + Ok(())=> (), + Err(error) => { + match error { + CtonError::Verifier(err) => { + return Err(pretty_verifier_error(&context.func, None, err)); + } + CtonError::ImplLimitExceeded | + CtonError::CodeTooLarge => return Err(String::from(error.description())), + } + } + }; + match verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) { + Ok(()) => (), + Err(err) => return Err(pretty_verifier_error(&context.func, None, err)), + } + } + terminal.fg(term::color::GREEN).unwrap(); + vprintln!(flag_verbose, " ok"); + terminal.reset().unwrap(); + } + Ok(()) +} + +/// Pretty-print a verifier error. +pub fn pretty_verifier_error(func: &ir::Function, + isa: Option<&TargetIsa>, + err: verifier::Error) + -> String { + let msg = err.to_string(); + let str1 = match err.location { + AnyEntity::Inst(inst) => { + format!("{}\n{}: {}\n\n", + msg, + inst, + func.dfg.display_inst(inst, isa)) + } + _ => String::from(format!("{}\n", msg)), + }; + format!("{}{}", str1, func.display(isa)) +} diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 33e47bc358..41b2b96f50 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -40,7 +40,8 @@ if [ -n "$needcheck" ]; then touch $tsfile || echo no target directory fi -PKGS="cretonne cretonne-reader cretonne-tools cretonne-frontend filecheck" +PKGS="cretonne cretonne-reader cretonne-tools cretonne-frontend cretonne-wasm \ + filecheck " cd "$topdir" for PKG in $PKGS do diff --git a/cranelift/wasmtests/arith.wast b/cranelift/wasmtests/arith.wast new file mode 100644 index 0000000000..fa7115696b --- /dev/null +++ b/cranelift/wasmtests/arith.wast @@ -0,0 +1,13 @@ +(module + (memory 1) + (func $main (local i32) + (set_local 0 (i32.sub (i32.const 4) (i32.const 4))) + (if + (get_local 0) + (then unreachable) + (else (drop (i32.mul (i32.const 6) (get_local 0)))) + ) + ) + (start $main) + (data (i32.const 0) "abcdefgh") +) diff --git a/cranelift/wasmtests/call.wast b/cranelift/wasmtests/call.wast new file mode 100644 index 0000000000..e8640d2342 --- /dev/null +++ b/cranelift/wasmtests/call.wast @@ -0,0 +1,10 @@ +(module + (func $main (local i32) + (set_local 0 (i32.const 0)) + (drop (call $inc)) + ) + (func $inc (result i32) + (i32.const 1) + ) + (start $main) +) diff --git a/cranelift/wasmtests/fibonacci.wast b/cranelift/wasmtests/fibonacci.wast new file mode 100644 index 0000000000..1788a467ca --- /dev/null +++ b/cranelift/wasmtests/fibonacci.wast @@ -0,0 +1,22 @@ +(module + (memory 1) + (func $main (local i32 i32 i32 i32) + (set_local 0 (i32.const 0)) + (set_local 1 (i32.const 1)) + (set_local 2 (i32.const 1)) + (set_local 3 (i32.const 0)) + (block + (loop + (br_if 1 (i32.gt_s (get_local 0) (i32.const 5))) + (set_local 3 (get_local 2)) + (set_local 2 (i32.add (get_local 2) (get_local 1))) + (set_local 1 (get_local 3)) + (set_local 0 (i32.add (get_local 0) (i32.const 1))) + (br 0) + ) + ) + (i32.store (i32.const 0) (get_local 2)) + ) + (start $main) + (data (i32.const 0) "0000") +) diff --git a/cranelift/wasmtests/globals.wast b/cranelift/wasmtests/globals.wast new file mode 100644 index 0000000000..646e5f0f45 --- /dev/null +++ b/cranelift/wasmtests/globals.wast @@ -0,0 +1,8 @@ +(module + (global $x (mut i32) (i32.const 4)) + (memory 1) + (func $main (local i32) + (i32.store (i32.const 0) (get_global $x)) + ) + (start $main) +) diff --git a/cranelift/wasmtests/memory.wast b/cranelift/wasmtests/memory.wast new file mode 100644 index 0000000000..0c81bad174 --- /dev/null +++ b/cranelift/wasmtests/memory.wast @@ -0,0 +1,11 @@ +(module + (memory 1) + (func $main (local i32) + (i32.store (i32.const 0) (i32.const 0x0)) + (if (i32.load (i32.const 0)) + (then (i32.store (i32.const 0) (i32.const 0xa))) + (else (i32.store (i32.const 0) (i32.const 0xb)))) + ) + (start $main) + (data (i32.const 0) "0000") +) diff --git a/lib/wasm/.gitignore b/lib/wasm/.gitignore new file mode 100644 index 0000000000..4308d82204 --- /dev/null +++ b/lib/wasm/.gitignore @@ -0,0 +1,3 @@ +target/ +**/*.rs.bk +Cargo.lock diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml new file mode 100644 index 0000000000..d03f178cd5 --- /dev/null +++ b/lib/wasm/Cargo.toml @@ -0,0 +1,13 @@ +[package] +name = "cretonne-wasm" +version = "0.0.0" +authors = ["The Cretonne Project Developers"] +publish = false +description = "Translator from WebAssembly to Cretonne IL" +repository = "https://github.com/stoklund/cretonne" +license = "Apache-2.0" + +[dependencies] +wasmparser = "0.6.1" +cretonne = { path = "../cretonne" } +cretonne-frontend = { path = "../frontend" } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs new file mode 100644 index 0000000000..b1647a9aee --- /dev/null +++ b/lib/wasm/src/code_translator.rs @@ -0,0 +1,1385 @@ +//! This module contains the bulk of the interesting code performing the translation between +//! WebAssembly and Cretonne IL. +//! +//! The translation is done in one pass, opcode by opcode. Two main data structures are used during +//! code translations: the value stack and the control stack. The value stack mimics the execution +//! of the WebAssembly stack machine: each instruction result is pushed onto the stack and +//! instruction arguments are popped off the stack. Similarly, when encountering a control flow +//! block, it is pushed onto the control stack and popped off when encountering the corresponding +//! `End`. +//! +//! Another data structure, the translation state, records information concerning unreachable code +//! status and about if inserting a return at the end of the function is necessary. +//! +//! Some of the WebAssembly instructions need information about the runtime to be translated: +//! +//! - the loads and stores need the memory base address; +//! - the `get_global` et `set_global` instructions depends on how the globals are implemented; +//! - `current_memory` and `grow_memory` are runtime functions; +//! - `call_indirect` has to translate the function index into the address of where this +//! is; +//! +//! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as +//! argument. +use cretonne::ir::{Function, Signature, Value, Type, InstBuilder, FunctionName, Ebb, FuncRef, + SigRef, ExtFuncData, Inst, MemFlags}; +use cretonne::ir::types::*; +use cretonne::ir::immediates::{Ieee32, Ieee64, Offset32}; +use cretonne::ir::condcodes::{IntCC, FloatCC}; +use cton_frontend::{ILBuilder, FunctionBuilder}; +use wasmparser::{Parser, ParserState, Operator, WasmDecoder, MemoryImmediate}; +use translation_utils::{f32_translation, f64_translation, type_to_type, translate_type, Local, + GlobalIndex, FunctionIndex, SignatureIndex}; +use std::collections::HashMap; +use runtime::WasmRuntime; +use std::u32; + + +/// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following +/// fields: +/// +/// - `destination`: reference to the `Ebb` that will hold the code after the control block; +/// - `return_values`: types of the values returned by the control block; +/// - `original_stack_size`: size of the value stack at the beginning of the control block. +/// +/// Moreover, the `if` frame has the `branch_inst` field that points to the `brz` instruction +/// separating the `true` and `false` branch. The `loop` frame has a `header` field that references +/// the `Ebb` that contains the beginning of the body of the loop. +#[derive(Debug)] +enum ControlStackFrame { + If { + destination: Ebb, + branch_inst: Inst, + return_values: Vec, + original_stack_size: usize, + reachable: bool, + }, + Block { + destination: Ebb, + return_values: Vec, + original_stack_size: usize, + reachable: bool, + }, + Loop { + destination: Ebb, + header: Ebb, + return_values: Vec, + original_stack_size: usize, + reachable: bool, + }, +} + +/// Helper methods for the control stack objects. +impl ControlStackFrame { + fn return_values(&self) -> &[Type] { + match self { + &ControlStackFrame::If { ref return_values, .. } | + &ControlStackFrame::Block { ref return_values, .. } | + &ControlStackFrame::Loop { ref return_values, .. } => return_values.as_slice(), + } + } + fn following_code(&self) -> Ebb { + match self { + &ControlStackFrame::If { destination, .. } | + &ControlStackFrame::Block { destination, .. } | + &ControlStackFrame::Loop { destination, .. } => destination, + } + } + fn br_destination(&self) -> Ebb { + match self { + &ControlStackFrame::If { destination, .. } | + &ControlStackFrame::Block { destination, .. } => destination, + &ControlStackFrame::Loop { header, .. } => header, + } + } + fn original_stack_size(&self) -> usize { + match self { + &ControlStackFrame::If { original_stack_size, .. } | + &ControlStackFrame::Block { original_stack_size, .. } | + &ControlStackFrame::Loop { original_stack_size, .. } => original_stack_size, + } + } + fn is_loop(&self) -> bool { + match self { + &ControlStackFrame::If { .. } | + &ControlStackFrame::Block { .. } => false, + &ControlStackFrame::Loop { .. } => true, + } + } + + fn is_reachable(&self) -> bool { + match self { + &ControlStackFrame::If { reachable, .. } | + &ControlStackFrame::Block { reachable, .. } | + &ControlStackFrame::Loop { reachable, .. } => reachable, + } + } + + fn set_reachable(&mut self) { + match self { + &mut ControlStackFrame::If { ref mut reachable, .. } | + &mut ControlStackFrame::Block { ref mut reachable, .. } | + &mut ControlStackFrame::Loop { ref mut reachable, .. } => *reachable = true, + } + } +} + +/// Contains information passed along during the translation and that records: +/// +/// - if the last instruction added was a `return`; +/// - the depth of the two unreachable control blocks stacks, that are manipulated when translating +/// unreachable code; +/// - all the `Ebb`s referenced by `br_table` instructions, because those are always reachable even +/// if they are at a point of the code that would have been unreachable otherwise. +struct TranslationState { + last_inst_return: bool, + phantom_unreachable_stack_depth: usize, + real_unreachable_stack_depth: usize, +} + +/// Holds mappings between the function and signatures indexes in the Wasm module and their +/// references as imports of the Cretonne IL function. +#[derive(Clone,Debug)] +pub struct FunctionImports { + /// Mappings index in function index space -> index in function local imports + pub functions: HashMap, + /// Mappings index in signature index space -> index in signature local imports + pub signatures: HashMap, +} + +impl FunctionImports { + fn new() -> FunctionImports { + FunctionImports { + functions: HashMap::new(), + signatures: HashMap::new(), + } + } +} + +/// Returns a well-formed Cretonne IL function from a wasm function body and a signature. +pub fn translate_function_body(parser: &mut Parser, + function_index: FunctionIndex, + sig: Signature, + locals: &Vec<(usize, Type)>, + exports: &Option>, + signatures: &Vec, + functions: &Vec, + il_builder: &mut ILBuilder, + runtime: &mut WasmRuntime) + -> Result<(Function, FunctionImports), String> { + runtime.next_function(); + // First we build the Function object with its name and signature + let mut func = Function::new(); + let args_num: usize = sig.argument_types.len(); + let args_types: Vec = sig.argument_types + .iter() + .map(|arg| arg.value_type) + .collect(); + func.signature = sig.clone(); + match exports { + &None => (), + &Some(ref exports) => { + match exports.get(&function_index) { + None => (), + Some(name) => func.name = FunctionName::new(name.clone()), + } + } + } + let mut func_imports = FunctionImports::new(); + let mut stack: Vec = Vec::new(); + let mut control_stack: Vec = Vec::new(); + // We introduce a arbitrary scope for the FunctionBuilder object + { + let mut builder = FunctionBuilder::new(&mut func, il_builder); + let first_ebb = builder.create_ebb(); + builder.switch_to_block(first_ebb, &[]); + builder.seal_block(first_ebb); + for i in 0..args_num { + // First we declare the function arguments' as non-SSA vars because they will be + // accessed by get_local + let arg_value = builder.arg_value(i as usize); + builder.declare_var(Local(i as u32), args_types[i]); + builder.def_var(Local(i as u32), arg_value); + } + // We also declare and initialize to 0 the local variables + let mut local_index = args_num; + for &(loc_count, ty) in locals { + let val = match ty { + I32 => builder.ins().iconst(ty, 0), + I64 => builder.ins().iconst(ty, 0), + F32 => builder.ins().f32const(Ieee32::with_bits(0)), + F64 => builder.ins().f64const(Ieee64::with_bits(0)), + _ => panic!("should not happen"), + }; + for _ in 0..loc_count { + builder.declare_var(Local(local_index as u32), ty); + builder.def_var(Local(local_index as u32), val); + local_index += 1; + } + } + let mut state = TranslationState { + last_inst_return: false, + phantom_unreachable_stack_depth: 0, + real_unreachable_stack_depth: 0, + }; + // We initialize the control stack with the implicit function block + let end_ebb = builder.create_ebb(); + control_stack.push(ControlStackFrame::Block { + destination: end_ebb, + original_stack_size: 0, + return_values: sig.return_types + .iter() + .map(|argty| argty.value_type) + .collect(), + reachable: false, + }); + // Now the main loop that reads every wasm instruction and translates it + loop { + let parser_state = parser.read(); + match *parser_state { + ParserState::CodeOperator(ref op) => { + if state.phantom_unreachable_stack_depth + + state.real_unreachable_stack_depth > 0 { + translate_unreachable_operator(op, + &mut builder, + &mut stack, + &mut control_stack, + &mut state) + } else { + translate_operator(op, + &mut builder, + runtime, + &mut stack, + &mut control_stack, + &mut state, + &sig, + &functions, + &signatures, + &exports, + &mut func_imports) + } + } + + ParserState::EndFunctionBody => break, + _ => return Err(String::from("wrong content in function body")), + } + } + // In WebAssembly, the final return instruction is implicit so we need to build it + // explicitely in Cretonne IL. + if !state.last_inst_return && !builder.is_filled() && + (!builder.is_unreachable() || !builder.is_pristine()) { + let cut_index = stack.len() - sig.return_types.len(); + let return_vals = stack.split_off(cut_index); + builder.ins().return_(return_vals.as_slice()); + } + // Because the function has an implicit block as body, we need to explicitely close it. + let frame = control_stack.pop().unwrap(); + builder.switch_to_block(frame.following_code(), frame.return_values()); + builder.seal_block(frame.following_code()); + // If the block is reachable we also have to include a return instruction in it. + if !builder.is_unreachable() { + stack.truncate(frame.original_stack_size()); + stack.extend_from_slice(builder.ebb_args(frame.following_code())); + let cut_index = stack.len() - sig.return_types.len(); + let return_vals = stack.split_off(cut_index); + builder.ins().return_(return_vals.as_slice()); + } + } + Ok((func, func_imports)) +} + +/// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted +/// a return. +fn translate_operator(op: &Operator, + builder: &mut FunctionBuilder, + runtime: &mut WasmRuntime, + stack: &mut Vec, + control_stack: &mut Vec, + state: &mut TranslationState, + sig: &Signature, + functions: &Vec, + signatures: &Vec, + exports: &Option>, + func_imports: &mut FunctionImports) { + state.last_inst_return = false; + // This big match treats all Wasm code operators. + match *op { + /********************************** Locals **************************************** + * `get_local` and `set_local` are treated as non-SSA variables and will completely + * diseappear in the Cretonne Code + ***********************************************************************************/ + Operator::GetLocal { local_index } => stack.push(builder.use_var(Local(local_index))), + Operator::SetLocal { local_index } => { + let val = stack.pop().unwrap(); + builder.def_var(Local(local_index), val); + } + Operator::TeeLocal { local_index } => { + let val = stack.last().unwrap(); + builder.def_var(Local(local_index), *val); + } + /********************************** Globals **************************************** + * `get_global` and `set_global` are handled by the runtime. + ***********************************************************************************/ + Operator::GetGlobal { global_index } => { + let val = runtime.translate_get_global(builder, global_index as GlobalIndex); + stack.push(val); + } + Operator::SetGlobal { global_index } => { + let val = stack.pop().unwrap(); + runtime.translate_set_global(builder, global_index as GlobalIndex, val); + } + /********************************* Stack misc *************************************** + * `drop`, `nop`, `unreachable` and `select`. + ***********************************************************************************/ + Operator::Drop => { + stack.pop(); + } + Operator::Select => { + let cond = stack.pop().unwrap(); + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().select(cond, arg2, arg1)); + } + Operator::Nop => { + // We do nothing + } + Operator::Unreachable => { + builder.ins().trap(); + state.real_unreachable_stack_depth = 1; + } + /***************************** Control flow blocks ********************************** + * When starting a control flow block, we create a new `Ebb` that will hold the code + * after the block, and we push a frame on the control stack. Depending on the type + * of block, we create a new `Ebb` for the body of the block with an associated + * jump instruction. + * + * The `End` instruction pops the last control frame from the control stack, seals + * the destination block (since `br` instructions targeting it only appear inside the + * block and have already been translated) and modify the value stack to use the + * possible `Ebb`'s arguments values. + ***********************************************************************************/ + Operator::Block { ty } => { + let next = builder.create_ebb(); + match type_to_type(&ty) { + Ok(ty_cre) => { + builder.append_ebb_arg(next, ty_cre); + } + Err(_) => {} + } + control_stack.push(ControlStackFrame::Block { + destination: next, + return_values: translate_type(ty).unwrap(), + original_stack_size: stack.len(), + reachable: false, + }); + } + Operator::Loop { ty } => { + let loop_body = builder.create_ebb(); + let next = builder.create_ebb(); + match type_to_type(&ty) { + Ok(ty_cre) => { + builder.append_ebb_arg(next, ty_cre); + } + Err(_) => {} + } + builder.ins().jump(loop_body, &[]); + control_stack.push(ControlStackFrame::Loop { + destination: next, + header: loop_body, + return_values: translate_type(ty).unwrap(), + original_stack_size: stack.len(), + reachable: false, + }); + builder.switch_to_block(loop_body, &[]); + } + Operator::If { ty } => { + let val = stack.pop().unwrap(); + let if_not = builder.create_ebb(); + let jump_inst = builder.ins().brz(val, if_not, &[]); + // Here we append an argument to an Ebb targeted by an argumentless jump instruction + // But in fact there are two cases: + // - either the If does not have a Else clause, in that case ty = EmptyBlock + // and we add nothing; + // - either the If have an Else clause, in that case the destination of this jump + // instruction will be changed later when we translate the Else operator. + match type_to_type(&ty) { + Ok(ty_cre) => { + builder.append_ebb_arg(if_not, ty_cre); + } + Err(_) => {} + } + control_stack.push(ControlStackFrame::If { + destination: if_not, + branch_inst: jump_inst, + return_values: translate_type(ty).unwrap(), + original_stack_size: stack.len(), + reachable: false, + }); + } + Operator::Else => { + // We take the control frame pushed by the if, use its ebb as the else body + // and push a new control frame with a new ebb for the code after the if/then/else + // At the end of the then clause we jump to the destination + let i = control_stack.len() - 1; + let (destination, return_values, branch_inst) = match &control_stack[i] { + &ControlStackFrame::If { + destination, + ref return_values, + branch_inst, + .. + } => (destination, return_values, branch_inst), + _ => panic!("should not happen"), + }; + let cut_index = stack.len() - return_values.len(); + let jump_args = stack.split_off(cut_index); + builder.ins().jump(destination, jump_args.as_slice()); + // We change the target of the branch instruction + let else_ebb = builder.create_ebb(); + builder.change_jump_destination(branch_inst, else_ebb); + builder.seal_block(else_ebb); + builder.switch_to_block(else_ebb, &[]); + } + Operator::End => { + let frame = control_stack.pop().unwrap(); + if !builder.is_unreachable() || !builder.is_pristine() { + let cut_index = stack.len() - frame.return_values().len(); + let jump_args = stack.split_off(cut_index); + builder + .ins() + .jump(frame.following_code(), jump_args.as_slice()); + } + builder.switch_to_block(frame.following_code(), frame.return_values()); + builder.seal_block(frame.following_code()); + // If it is a loop we also have to seal the body loop block + match frame { + ControlStackFrame::Loop { header, .. } => builder.seal_block(header), + _ => {} + } + stack.truncate(frame.original_stack_size()); + stack.extend_from_slice(builder.ebb_args(frame.following_code())); + } + /**************************** Branch instructions ********************************* + * The branch instructions all have as arguments a target nesting level, which + * corresponds to how many control stack frames do we have to pop to get the + * destination `Ebb`. + * + * Once the destination `Ebb` is found, we sometimes have to declare a certain depth + * of the stack unreachable, because some branch instructions are terminator. + * + * The `br_table` case is much more complicated because Cretonne's `br_table` instruction + * does not support jump arguments like all the other branch instructions. That is why, in + * the case where we would use jump arguments for every other branch instructions, we + * need to split the critical edges leaving the `br_tables` by creating one `Ebb` per + * table destination; the `br_table` will point to these newly created `Ebbs` and these + * `Ebb`s contain only a jump instruction pointing to the final destination, this time with + * jump arguments. + * + * This system is also implemented in Cretonne's SSA construction algorithm, because + * `use_var` located in a destination `Ebb` of a `br_table` might trigger the addition + * of jump arguments in each predecessor branch instruction, one of which might be a + * `br_table`. + ***********************************************************************************/ + Operator::Br { relative_depth } => { + let i = control_stack.len() - 1 - (relative_depth as usize); + let frame = &mut control_stack[i]; + let jump_args = if frame.is_loop() { + Vec::new() + } else { + let cut_index = stack.len() - frame.return_values().len(); + stack.split_off(cut_index) + }; + builder + .ins() + .jump(frame.br_destination(), jump_args.as_slice()); + // We signal that all the code that follows until the next End is unreachable + frame.set_reachable(); + state.real_unreachable_stack_depth = 1 + relative_depth as usize; + } + Operator::BrIf { relative_depth } => { + let val = stack.pop().unwrap(); + let i = control_stack.len() - 1 - (relative_depth as usize); + let frame = &mut control_stack[i]; + let jump_args = if frame.is_loop() { + Vec::new() + } else { + let cut_index = stack.len() - frame.return_values().len(); + stack.split_off(cut_index) + }; + builder + .ins() + .brnz(val, frame.br_destination(), jump_args.as_slice()); + // The values returned by the branch are still available for the reachable + // code that comes after it + frame.set_reachable(); + stack.extend(jump_args); + } + Operator::BrTable { ref table } => { + let (depths, default) = table.read_table(); + let mut min_depth = default; + for depth in depths.iter() { + if *depth < min_depth { + min_depth = *depth; + } + } + let jump_args_count = { + let i = control_stack.len() - 1 - (min_depth as usize); + let min_depth_frame = &control_stack[i]; + if min_depth_frame.is_loop() { + 0 + } else { + min_depth_frame.return_values().len() + } + }; + if jump_args_count == 0 { + // No jump arguments + let val = stack.pop().unwrap(); + if depths.len() > 0 { + let jt = builder.create_jump_table(); + for (index, depth) in depths.iter().enumerate() { + let i = control_stack.len() - 1 - (*depth as usize); + let frame = &mut control_stack[i]; + let ebb = frame.br_destination(); + builder.insert_jump_table_entry(jt, index, ebb); + frame.set_reachable(); + } + builder.ins().br_table(val, jt); + } + let i = control_stack.len() - 1 - (default as usize); + let frame = &mut control_stack[i]; + let ebb = frame.br_destination(); + builder.ins().jump(ebb, &[]); + state.real_unreachable_stack_depth = 1 + min_depth as usize; + frame.set_reachable(); + } else { + // Here we have jump arguments, but Cretonne's br_table doesn't support them + // We then proceed to split the edges going out of the br_table + let val = stack.pop().unwrap(); + let cut_index = stack.len() - jump_args_count; + let jump_args = stack.split_off(cut_index); + if depths.len() > 0 { + let jt = builder.create_jump_table(); + let dest_ebbs: HashMap = depths + .iter() + .enumerate() + .fold(HashMap::new(), |mut acc, (index, &depth)| { + if acc.get(&(depth as usize)).is_none() { + let branch_ebb = builder.create_ebb(); + builder.insert_jump_table_entry(jt, index, branch_ebb); + acc.insert(depth as usize, branch_ebb); + return acc; + }; + let branch_ebb = acc.get(&(depth as usize)).unwrap().clone(); + builder.insert_jump_table_entry(jt, index, branch_ebb); + acc + }); + builder.ins().br_table(val, jt); + let default_ebb = control_stack[control_stack.len() - 1 - (default as usize)] + .br_destination(); + builder.ins().jump(default_ebb, jump_args.as_slice()); + stack.extend(jump_args.clone()); + for (depth, dest_ebb) in dest_ebbs { + builder.switch_to_block(dest_ebb, &[]); + builder.seal_block(dest_ebb); + let i = control_stack.len() - 1 - (depth as usize); + let frame = &mut control_stack[i]; + let real_dest_ebb = frame.br_destination(); + builder.ins().jump(real_dest_ebb, jump_args.as_slice()); + frame.set_reachable(); + } + state.real_unreachable_stack_depth = 1 + min_depth as usize; + } else { + let ebb = control_stack[control_stack.len() - 1 - (default as usize)] + .br_destination(); + builder.ins().jump(ebb, jump_args.as_slice()); + stack.extend(jump_args); + state.real_unreachable_stack_depth = 1 + min_depth as usize; + } + } + } + Operator::Return => { + let return_count = sig.return_types.len(); + let cut_index = stack.len() - return_count; + let return_args = stack.split_off(cut_index); + builder.ins().return_(return_args.as_slice()); + state.last_inst_return = true; + state.real_unreachable_stack_depth = 1; + } + /************************************ Calls **************************************** + * The call instructions pop off their arguments from the stack and append their + * return values to it. `call_indirect` needs runtime support because there is an + * argument referring to an index in the external functions table of the module. + ************************************************************************************/ + Operator::Call { function_index } => { + let args_num = args_count(function_index as usize, functions, signatures); + let cut_index = stack.len() - args_num; + let call_args = stack.split_off(cut_index); + let internal_function_index = find_function_import(function_index as usize, + builder, + func_imports, + functions, + exports, + signatures); + let call_inst = builder + .ins() + .call(internal_function_index, call_args.as_slice()); + let ret_values = builder.inst_results(call_inst); + for val in ret_values { + stack.push(*val); + } + } + Operator::CallIndirect { + index, + table_index: _, + } => { + // index is the index of the function's signature and table_index is the index + // of the table to search the function in + // TODO: have runtime support for tables + let sigref = find_signature_import(index as usize, builder, func_imports, signatures); + let args_num = builder.signature(sigref).unwrap().argument_types.len(); + let index_val = stack.pop().unwrap(); + let cut_index = stack.len() - args_num; + let call_args = stack.split_off(cut_index); + let ret_values = + runtime.translate_call_indirect(builder, sigref, index_val, call_args.as_slice()); + for val in ret_values { + stack.push(*val); + } + } + /******************************* Memory management *********************************** + * Memory management is handled by runtime. It is usually translated into calls to + * special functions. + ************************************************************************************/ + Operator::GrowMemory { reserved: _ } => { + let val = stack.pop().unwrap(); + stack.push(runtime.translate_grow_memory(builder, val)); + } + Operator::CurrentMemory { reserved: _ } => { + stack.push(runtime.translate_current_memory(builder)); + } + /******************************* Load instructions *********************************** + * Wasm specifies an integer alignment flag but we drop it in Cretonne. + * The memory base address is provided by the runtime. + * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not + ************************************************************************************/ + Operator::I32Load8U { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().uload8(I32, memflags, addr, memoffset)) + } + Operator::I32Load16U { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().uload8(I32, memflags, addr, memoffset)) + } + Operator::I32Load8S { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().sload8(I32, memflags, addr, memoffset)) + } + Operator::I32Load16S { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().sload8(I32, memflags, addr, memoffset)) + } + Operator::I64Load8U { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().uload8(I64, memflags, addr, memoffset)) + } + Operator::I64Load16U { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().uload16(I64, memflags, addr, memoffset)) + } + Operator::I64Load8S { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().sload8(I64, memflags, addr, memoffset)) + } + Operator::I64Load16S { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().sload16(I64, memflags, addr, memoffset)) + } + Operator::I64Load32S { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().sload32(memflags, addr, memoffset)) + } + Operator::I64Load32U { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().uload32(memflags, addr, memoffset)) + } + Operator::I32Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().load(I32, memflags, addr, memoffset)) + } + Operator::F32Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().load(F32, memflags, addr, memoffset)) + } + Operator::I64Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().load(I64, memflags, addr, memoffset)) + } + Operator::F64Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + stack.push(builder.ins().load(F64, memflags, addr, memoffset)) + } + /****************************** Store instructions *********************************** + * Wasm specifies an integer alignment flag but we drop it in Cretonne. + * The memory base address is provided by the runtime. + * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not + ************************************************************************************/ + Operator::I32Store { memory_immediate: MemoryImmediate { flags: _, offset } } | + Operator::I64Store { memory_immediate: MemoryImmediate { flags: _, offset } } | + Operator::F32Store { memory_immediate: MemoryImmediate { flags: _, offset } } | + Operator::F64Store { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let val = stack.pop().unwrap(); + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + builder.ins().store(memflags, val, addr, memoffset); + } + Operator::I32Store8 { memory_immediate: MemoryImmediate { flags: _, offset } } | + Operator::I64Store8 { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let val = stack.pop().unwrap(); + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + builder.ins().istore8(memflags, val, addr, memoffset); + } + Operator::I32Store16 { memory_immediate: MemoryImmediate { flags: _, offset } } | + Operator::I64Store16 { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let val = stack.pop().unwrap(); + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + builder.ins().istore16(memflags, val, addr, memoffset); + } + Operator::I64Store32 { memory_immediate: MemoryImmediate { flags: _, offset } } => { + let val = stack.pop().unwrap(); + let address_i32 = stack.pop().unwrap(); + let base = runtime.translate_memory_base_address(builder, 0); + let address_i64 = builder.ins().uextend(I64, address_i32); + let addr = builder.ins().iadd(base, address_i64); + let memflags = MemFlags::new(); + let memoffset = Offset32::new(offset as i32); + builder.ins().istore32(memflags, val, addr, memoffset); + } + /****************************** Nullary Operators ************************************/ + Operator::I32Const { value } => stack.push(builder.ins().iconst(I32, value as i64)), + Operator::I64Const { value } => stack.push(builder.ins().iconst(I64, value)), + Operator::F32Const { value } => { + stack.push(builder.ins().f32const(f32_translation(value))); + } + Operator::F64Const { value } => { + stack.push(builder.ins().f64const(f64_translation(value))); + } + /******************************* Unary Operators *************************************/ + Operator::I32Clz => { + let arg = stack.pop().unwrap(); + let val = builder.ins().clz(arg); + stack.push(builder.ins().sextend(I32, val)); + } + Operator::I64Clz => { + let arg = stack.pop().unwrap(); + let val = builder.ins().clz(arg); + stack.push(builder.ins().sextend(I64, val)); + } + Operator::I32Ctz => { + let val = stack.pop().unwrap(); + let short_res = builder.ins().ctz(val); + stack.push(builder.ins().sextend(I32, short_res)); + } + Operator::I64Ctz => { + let val = stack.pop().unwrap(); + let short_res = builder.ins().ctz(val); + stack.push(builder.ins().sextend(I64, short_res)); + } + Operator::I32Popcnt => { + let arg = stack.pop().unwrap(); + let val = builder.ins().popcnt(arg); + stack.push(builder.ins().sextend(I32, val)); + } + Operator::I64Popcnt => { + let arg = stack.pop().unwrap(); + let val = builder.ins().popcnt(arg); + stack.push(builder.ins().sextend(I64, val)); + } + Operator::I64ExtendSI32 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().sextend(I64, val)); + } + Operator::I64ExtendUI32 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().uextend(I64, val)); + } + Operator::I32WrapI64 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().ireduce(I32, val)); + } + Operator::F32Sqrt | + Operator::F64Sqrt => { + let arg = stack.pop().unwrap(); + stack.push(builder.ins().sqrt(arg)); + } + Operator::F32Ceil | + Operator::F64Ceil => { + let arg = stack.pop().unwrap(); + stack.push(builder.ins().ceil(arg)); + } + Operator::F32Floor | + Operator::F64Floor => { + let arg = stack.pop().unwrap(); + stack.push(builder.ins().floor(arg)); + } + Operator::F32Trunc | + Operator::F64Trunc => { + let arg = stack.pop().unwrap(); + stack.push(builder.ins().trunc(arg)); + } + Operator::F32Nearest | + Operator::F64Nearest => { + let arg = stack.pop().unwrap(); + stack.push(builder.ins().nearest(arg)); + } + Operator::F32Abs | Operator::F64Abs => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().fabs(val)); + } + Operator::F32Neg | Operator::F64Neg => { + let arg = stack.pop().unwrap(); + stack.push(builder.ins().fneg(arg)); + } + Operator::F64ConvertUI64 | + Operator::F64ConvertUI32 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().fcvt_from_uint(F64, val)); + } + Operator::F64ConvertSI64 | + Operator::F64ConvertSI32 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().fcvt_from_sint(F64, val)); + } + Operator::F32ConvertSI64 | + Operator::F32ConvertSI32 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().fcvt_from_sint(F32, val)); + } + Operator::F32ConvertUI64 | + Operator::F32ConvertUI32 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().fcvt_from_uint(F32, val)); + } + Operator::F64PromoteF32 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().fpromote(F64, val)); + } + Operator::F32DemoteF64 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().fdemote(F32, val)); + } + Operator::I64TruncSF64 | + Operator::I64TruncSF32 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().fcvt_to_sint(I64, val)); + } + Operator::I32TruncSF64 | + Operator::I32TruncSF32 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().fcvt_to_sint(I32, val)); + } + Operator::I64TruncUF64 | + Operator::I64TruncUF32 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().fcvt_to_uint(I64, val)); + } + Operator::I32TruncUF64 | + Operator::I32TruncUF32 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().fcvt_to_uint(I32, val)); + } + Operator::F32ReinterpretI32 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().bitcast(F32, val)); + } + Operator::F64ReinterpretI64 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().bitcast(F64, val)); + } + Operator::I32ReinterpretF32 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().bitcast(I32, val)); + } + Operator::I64ReinterpretF64 => { + let val = stack.pop().unwrap(); + stack.push(builder.ins().bitcast(I64, val)); + } + /****************************** Binary Operators ************************************/ + Operator::I32Add | Operator::I64Add => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().iadd(arg1, arg2)); + } + Operator::I32And | Operator::I64And => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().band(arg1, arg2)); + } + Operator::I32Or | Operator::I64Or => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().bor(arg1, arg2)); + } + Operator::I32Xor | Operator::I64Xor => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().bxor(arg1, arg2)); + } + Operator::I32Shl | Operator::I64Shl => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().ishl(arg1, arg2)); + } + Operator::I32ShrS | + Operator::I64ShrS => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().sshr(arg1, arg2)); + } + Operator::I32ShrU | + Operator::I64ShrU => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().ushr(arg1, arg2)); + } + Operator::I32Rotl | + Operator::I64Rotl => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().rotl(arg1, arg2)); + } + Operator::I32Rotr | + Operator::I64Rotr => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().rotr(arg1, arg2)); + } + Operator::F32Add | Operator::F64Add => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().fadd(arg1, arg2)); + } + Operator::I32Sub | Operator::I64Sub => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().isub(arg1, arg2)); + } + Operator::F32Sub | Operator::F64Sub => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().fsub(arg1, arg2)); + } + Operator::I32Mul | Operator::I64Mul => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().imul(arg1, arg2)); + } + Operator::F32Mul | Operator::F64Mul => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().fmul(arg1, arg2)); + } + Operator::F32Div | Operator::F64Div => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().fdiv(arg1, arg2)); + } + Operator::I32DivS | + Operator::I64DivS => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().sdiv(arg1, arg2)); + } + Operator::I32DivU | + Operator::I64DivU => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().udiv(arg1, arg2)); + } + Operator::I32RemS | + Operator::I64RemS => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().srem(arg1, arg2)); + } + Operator::I32RemU | + Operator::I64RemU => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().urem(arg1, arg2)); + } + Operator::F32Min | Operator::F64Min => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().fmin(arg1, arg2)); + } + Operator::F32Max | Operator::F64Max => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().fmax(arg1, arg2)); + } + Operator::F32Copysign | + Operator::F64Copysign => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + stack.push(builder.ins().fcopysign(arg1, arg2)); + } + /**************************** Comparison Operators **********************************/ + Operator::I32LtS | Operator::I64LtS => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder.ins().icmp(IntCC::SignedLessThan, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::I32LtU | Operator::I64LtU => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder.ins().icmp(IntCC::UnsignedLessThan, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::I32LeS | Operator::I64LeS => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder.ins().icmp(IntCC::SignedLessThanOrEqual, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::I32LeU | Operator::I64LeU => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder + .ins() + .icmp(IntCC::UnsignedLessThanOrEqual, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::I32GtS | Operator::I64GtS => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder.ins().icmp(IntCC::SignedGreaterThan, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::I32GtU | Operator::I64GtU => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder.ins().icmp(IntCC::UnsignedGreaterThan, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::I32GeS | Operator::I64GeS => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder + .ins() + .icmp(IntCC::SignedGreaterThanOrEqual, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::I32GeU | Operator::I64GeU => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder + .ins() + .icmp(IntCC::UnsignedGreaterThanOrEqual, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::I32Eqz | Operator::I64Eqz => { + let arg = stack.pop().unwrap(); + let val = builder.ins().icmp_imm(IntCC::Equal, arg, 0); + stack.push(builder.ins().bint(I32, val)); + } + Operator::I32Eq | Operator::I64Eq => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder.ins().icmp(IntCC::Equal, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::F32Eq | Operator::F64Eq => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder.ins().fcmp(FloatCC::Equal, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::I32Ne | Operator::I64Ne => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder.ins().icmp(IntCC::NotEqual, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::F32Ne | Operator::F64Ne => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder.ins().fcmp(FloatCC::NotEqual, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::F32Gt | Operator::F64Gt => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder.ins().fcmp(FloatCC::GreaterThan, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::F32Ge | Operator::F64Ge => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder.ins().fcmp(FloatCC::GreaterThanOrEqual, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::F32Lt | Operator::F64Lt => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder.ins().fcmp(FloatCC::LessThan, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + Operator::F32Le | Operator::F64Le => { + let arg2 = stack.pop().unwrap(); + let arg1 = stack.pop().unwrap(); + let val = builder.ins().fcmp(FloatCC::LessThanOrEqual, arg1, arg2); + stack.push(builder.ins().bint(I32, val)); + } + } +} + +/// Deals with a Wasm instruction located in an unreachable portion of the code. Most of them +/// are dropped but special ones like `End` or `Else` signal the potential end of the unreachable +/// portion so the translation state muts be updated accordingly. +fn translate_unreachable_operator(op: &Operator, + builder: &mut FunctionBuilder, + stack: &mut Vec, + control_stack: &mut Vec, + state: &mut TranslationState) { + // We don't translate because the code is unreachable + // Nevertheless we have to record a phantom stack for this code + // to know when the unreachable code ends + match *op { + Operator::If { ty: _ } | + Operator::Loop { ty: _ } | + Operator::Block { ty: _ } => { + state.phantom_unreachable_stack_depth += 1; + } + Operator::End => { + if state.phantom_unreachable_stack_depth > 0 { + state.phantom_unreachable_stack_depth -= 1; + } else { + // This End corresponds to a real control stack frame + // We switch to the destination block but we don't insert + // a jump instruction since the code is still unreachable + let frame = control_stack.pop().unwrap(); + + builder.switch_to_block(frame.following_code(), &[]); + builder.seal_block(frame.following_code()); + match frame { + // If it is a loop we also have to seal the body loop block + ControlStackFrame::Loop { header, .. } => builder.seal_block(header), + // If it is a if then the code after is reachable again + ControlStackFrame::If { .. } => { + state.real_unreachable_stack_depth = 1; + } + _ => {} + } + if frame.is_reachable() { + state.real_unreachable_stack_depth = 1; + } + // Now we have to split off the stack the values not used + // by unreachable code that hasn't been translated + stack.truncate(frame.original_stack_size()); + // And add the return values of the block but only if the next block is reachble + // (which corresponds to testing if the stack depth is 1) + if state.real_unreachable_stack_depth == 1 { + stack.extend_from_slice(builder.ebb_args(frame.following_code())); + } + state.real_unreachable_stack_depth -= 1; + state.last_inst_return = false; + } + } + Operator::Else => { + if state.phantom_unreachable_stack_depth > 0 { + // This is part of a phantom if-then-else, we do nothing + } else { + // Encountering an real else means that the code in the else + // clause is reachable again + let (branch_inst, original_stack_size) = match &control_stack[control_stack.len() - + 1] { + &ControlStackFrame::If { + branch_inst, + original_stack_size, + .. + } => (branch_inst, original_stack_size), + _ => panic!("should not happen"), + }; + // We change the target of the branch instruction + let else_ebb = builder.create_ebb(); + builder.change_jump_destination(branch_inst, else_ebb); + builder.seal_block(else_ebb); + builder.switch_to_block(else_ebb, &[]); + // Now we have to split off the stack the values not used + // by unreachable code that hasn't been translated + stack.truncate(original_stack_size); + state.real_unreachable_stack_depth = 0; + state.last_inst_return = false; + } + } + _ => { + // We don't translate because this is unreachable code + } + } +} + +fn args_count(index: FunctionIndex, + functions: &Vec, + signatures: &Vec) + -> usize { + signatures[functions[index] as usize].argument_types.len() +} + +// Given a index in the function index space, search for it in the function imports and if it is +// not there add it to the function imports. +fn find_function_import(index: FunctionIndex, + builder: &mut FunctionBuilder, + func_imports: &mut FunctionImports, + functions: &Vec, + exports: &Option>, + signatures: &Vec) + -> FuncRef { + match func_imports.functions.get(&index) { + Some(local_index) => return *local_index, + None => {} + } + // We have to import the function + let sig_index = functions[index]; + match func_imports.signatures.get(&(sig_index as usize)) { + Some(local_sig_index) => { + let local_func_index = + builder.import_function(ExtFuncData { + name: match exports { + &None => FunctionName::new(""), + &Some(ref exports) => { + match exports.get(&index) { + None => FunctionName::new(""), + Some(name) => { + FunctionName::new(name.clone()) + } + } + } + }, + signature: *local_sig_index, + }); + func_imports.functions.insert(index, local_func_index); + return local_func_index; + } + None => {} + }; + // We have to import the signature + let sig_local_index = builder.import_signature(signatures[sig_index as usize].clone()); + func_imports + .signatures + .insert(sig_index as usize, sig_local_index); + let local_func_index = + builder.import_function(ExtFuncData { + name: match exports { + &None => FunctionName::new(""), + &Some(ref exports) => { + match exports.get(&index) { + None => FunctionName::new(""), + Some(name) => FunctionName::new(name.clone()), + } + } + }, + signature: sig_local_index, + }); + func_imports.functions.insert(index, local_func_index); + local_func_index +} + +fn find_signature_import(sig_index: SignatureIndex, + builder: &mut FunctionBuilder, + func_imports: &mut FunctionImports, + signatures: &Vec) + -> SigRef { + match func_imports.signatures.get(&(sig_index as usize)) { + Some(local_sig_index) => return *local_sig_index, + None => {} + } + let sig_local_index = builder.import_signature(signatures[sig_index as usize].clone()); + func_imports + .signatures + .insert(sig_index as usize, sig_local_index); + sig_local_index +} diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs new file mode 100644 index 0000000000..5006e5c9aa --- /dev/null +++ b/lib/wasm/src/lib.rs @@ -0,0 +1,27 @@ +//! Performs the translation from a wasm module in binary format to the in-memory representation +//! of the Cretonne IL. More particularly, it translates the code of all the functions bodies and +//! interacts with a runtime implementing the [`WasmRuntime`](trait.WasmRuntime.html) trait to +//! deal with tables, globals and linear memory. +//! +//! The crate provides a `DummyRuntime` trait that will allow to translate the code of the +//! functions but will fail at execution. You should use +//! [`wasmstandalone::StandaloneRuntime`](../wasmstandalone/struct.StandaloneRuntime.html) to be +//! able to execute the translated code. +//! +//! The main function of this module is [`translate_module`](fn.translate_module.html). + +extern crate wasmparser; +extern crate cton_frontend; +extern crate cretonne; + +mod module_translator; +mod translation_utils; +mod code_translator; +mod runtime; +mod sections_translator; + +pub use module_translator::{translate_module, TranslationResult, FunctionTranslation, + ImportMappings}; +pub use runtime::{WasmRuntime, DummyRuntime}; +pub use translation_utils::{Local, FunctionIndex, GlobalIndex, TableIndex, MemoryIndex, RawByte, + MemoryAddress, SignatureIndex, Global, GlobalInit, Table, Memory}; diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs new file mode 100644 index 0000000000..1c22655300 --- /dev/null +++ b/lib/wasm/src/module_translator.rs @@ -0,0 +1,288 @@ +//! Translation skeletton that traverses the whole WebAssembly module and call helper functions +//! to deal with each part of it. +use wasmparser::{ParserState, SectionCode, ParserInput, Parser, WasmDecoder}; +use sections_translator::{SectionParsingError, parse_function_signatures, parse_import_section, + parse_function_section, parse_export_section, parse_memory_section, + parse_global_section, parse_table_section, parse_elements_section, + parse_data_section}; +use translation_utils::{type_to_type, Import, SignatureIndex, FunctionIndex, invert_hashmaps}; +use cretonne::ir::{Function, Type, FuncRef, SigRef}; +use code_translator::translate_function_body; +use cton_frontend::ILBuilder; +use std::collections::HashMap; +use runtime::WasmRuntime; + +/// Output of the [`translate_module`](fn.translate_module.html) function. Contains the translated +/// functions and when present the index of the function defined as `start` of the module. +pub struct TranslationResult { + pub functions: Vec, + pub start_index: Option, +} + +/// A function in a WebAssembly module can be either imported, or defined inside it. If it is +/// defined inside it, then the translation in Cretonne IL is available as well as the mappings +/// between Cretonne imports and indexes in the function index space. +#[derive(Clone)] +pub enum FunctionTranslation { + Code { + il: Function, + imports: ImportMappings, + }, + Import(), +} + +#[derive(Clone,Debug)] +/// Mappings describing the relations between imports of the Cretonne IL functions and the +/// functions in the WebAssembly module. +pub struct ImportMappings { + /// Find the index of a function in the WebAssembly module thanks to a `FuncRef`. + pub functions: HashMap, + /// Find the index of a signature in the WebAssembly module thanks to a `SigRef`. + pub signatures: HashMap, +} + +impl ImportMappings { + pub fn new() -> ImportMappings { + ImportMappings { + functions: HashMap::new(), + signatures: HashMap::new(), + } + } +} + +/// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IL +/// [`Function`](../cretonne/ir/function/struct.Function.html). +/// Returns the functions and also the mappings for imported functions and signature between the +/// indexes in the wasm module and the indexes inside each functions. +pub fn translate_module(data: &Vec, + runtime: &mut WasmRuntime) + -> Result { + let mut parser = Parser::new(data.as_slice()); + match *parser.read() { + ParserState::BeginWasm { .. } => {} + ref s @ _ => panic!("modules should begin properly: {:?}", s), + } + let mut signatures = None; + let mut functions: Option> = None; + let mut globals = Vec::new(); + let mut exports: Option> = None; + let mut next_input = ParserInput::Default; + let mut function_index: FunctionIndex = 0; + let mut function_imports_count = 0; + let mut start_index: Option = None; + loop { + match *parser.read_with_input(next_input) { + ParserState::BeginSection { code: SectionCode::Type, .. } => { + match parse_function_signatures(&mut parser) { + Ok(sigs) => signatures = Some(sigs), + Err(SectionParsingError::WrongSectionContent(s)) => { + return Err(format!("wrong content in the type section: {}", s)) + } + }; + next_input = ParserInput::Default; + } + ParserState::BeginSection { code: SectionCode::Import, .. } => { + match parse_import_section(&mut parser) { + Ok(imps) => { + for import in imps { + match import { + Import::Function { sig_index } => { + functions = match functions { + None => Some(vec![sig_index as SignatureIndex]), + Some(mut funcs) => { + funcs.push(sig_index as SignatureIndex); + Some(funcs) + } + }; + function_index += 1; + } + Import::Memory(mem) => { + runtime.declare_memory(mem); + } + Import::Global(glob) => { + runtime.declare_global(glob.clone()); + globals.push(glob); + } + Import::Table(tab) => { + runtime.declare_table(tab); + } + } + } + } + Err(SectionParsingError::WrongSectionContent(s)) => { + return Err(format!("wrong content in the import section: {}", s)) + } + } + function_imports_count = function_index; + next_input = ParserInput::Default; + } + ParserState::BeginSection { code: SectionCode::Function, .. } => { + match parse_function_section(&mut parser) { + Ok(funcs) => { + match functions { + None => functions = Some(funcs), + Some(ref mut imps) => imps.extend(funcs), + } + } + Err(SectionParsingError::WrongSectionContent(s)) => { + return Err(format!("wrong content in the function section: {}", s)) + } + } + next_input = ParserInput::Default; + } + ParserState::BeginSection { code: SectionCode::Table, .. } => { + match parse_table_section(&mut parser, runtime) { + Ok(()) => (), + Err(SectionParsingError::WrongSectionContent(s)) => { + return Err(format!("wrong content in the table section: {}", s)) + } + } + } + ParserState::BeginSection { code: SectionCode::Memory, .. } => { + match parse_memory_section(&mut parser) { + Ok(mems) => { + for mem in mems { + runtime.declare_memory(mem); + } + } + Err(SectionParsingError::WrongSectionContent(s)) => { + return Err(format!("wrong content in the memory section: {}", s)) + } + } + next_input = ParserInput::Default; + } + ParserState::BeginSection { code: SectionCode::Global, .. } => { + match parse_global_section(&mut parser, runtime) { + Ok(mut globs) => globals.append(&mut globs), + Err(SectionParsingError::WrongSectionContent(s)) => { + return Err(format!("wrong content in the global section: {}", s)) + } + } + next_input = ParserInput::Default; + } + ParserState::BeginSection { code: SectionCode::Export, .. } => { + match parse_export_section(&mut parser) { + Ok(exps) => exports = Some(exps), + Err(SectionParsingError::WrongSectionContent(s)) => { + return Err(format!("wrong content in the export section: {}", s)) + } + } + next_input = ParserInput::Default; + } + ParserState::BeginSection { code: SectionCode::Start, .. } => { + match *parser.read() { + ParserState::StartSectionEntry(index) => { + start_index = Some(index as FunctionIndex) + } + _ => return Err(String::from("wrong content in the start section")), + } + match *parser.read() { + ParserState::EndSection => {} + _ => return Err(String::from("wrong content in the start section")), + } + next_input = ParserInput::Default; + } + ParserState::BeginSection { code: SectionCode::Element, .. } => { + match parse_elements_section(&mut parser, runtime, &globals) { + Ok(()) => (), + Err(SectionParsingError::WrongSectionContent(s)) => { + return Err(format!("wrong content in the element section: {}", s)) + } + } + next_input = ParserInput::Default; + } + ParserState::BeginSection { code: SectionCode::Code, .. } => { + // The code section begins + break; + } + ParserState::EndSection => { + next_input = ParserInput::Default; + } + ParserState::EndWasm => { + return Ok(TranslationResult { + functions: Vec::new(), + start_index: None, + }) + } + ParserState::BeginSection { code: SectionCode::Data, .. } => { + match parse_data_section(&mut parser, runtime, &globals) { + Ok(()) => (), + Err(SectionParsingError::WrongSectionContent(s)) => { + return Err(format!("wrong content in the data section: {}", s)) + } + } + } + _ => return Err(String::from("wrong content in the preamble")), + }; + } + // At this point we've entered the code section + // First we check that we have all that is necessary to translate a function. + let signatures = match signatures { + None => Vec::new(), + Some(sigs) => sigs, + }; + let functions = match functions { + None => return Err(String::from("missing a function section")), + Some(functions) => functions, + }; + let mut il_functions: Vec = Vec::new(); + il_functions.resize(function_imports_count, FunctionTranslation::Import()); + let mut il_builder = ILBuilder::new(); + runtime.begin_translation(); + loop { + let locals: Vec<(usize, Type)> = match *parser.read() { + ParserState::BeginFunctionBody { ref locals, .. } => { + locals + .iter() + .map(|&(index, ref ty)| { + (index as usize, + match type_to_type(ty) { + Ok(ty) => ty, + Err(()) => panic!("unsupported type for local variable"), + }) + }) + .collect() + } + ParserState::EndSection => break, + _ => return Err(String::from(format!("wrong content in code section"))), + }; + let signature = signatures[functions[function_index as usize] as usize].clone(); + match translate_function_body(&mut parser, + function_index, + signature, + &locals, + &exports, + &signatures, + &functions, + &mut il_builder, + runtime) { + Ok((il_func, imports)) => { + il_functions.push(FunctionTranslation::Code { + il: il_func, + imports: invert_hashmaps(imports), + }) + } + Err(s) => return Err(s), + } + function_index += 1; + } + loop { + match *parser.read() { + ParserState::BeginSection { code: SectionCode::Data, .. } => { + match parse_data_section(&mut parser, runtime, &globals) { + Ok(()) => (), + Err(SectionParsingError::WrongSectionContent(s)) => { + return Err(format!("wrong content in the data section: {}", s)) + } + } + } + ParserState::EndWasm => { + return Ok(TranslationResult { + functions: il_functions, + start_index, + }) + } + _ => (), + } + } +} diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs new file mode 100644 index 0000000000..5969c0ee8f --- /dev/null +++ b/lib/wasm/src/runtime/dummy.rs @@ -0,0 +1,93 @@ +use runtime::WasmRuntime; +use translation_utils::{Local, Global, Memory, Table, GlobalIndex, TableIndex, FunctionIndex, + MemoryIndex}; +use cton_frontend::FunctionBuilder; +use cretonne::ir::{Value, InstBuilder, SigRef}; +use cretonne::ir::immediates::{Ieee32, Ieee64}; +use cretonne::ir::types::*; + +/// This runtime implementation is a "naïve" one, doing essentially nothing and emitting +/// placeholders when forced to. Don't try to execute code translated with this runtime, it is +/// essentially here for translation debug purposes. +pub struct DummyRuntime { + globals: Vec, +} + +impl DummyRuntime { + /// Allocates the runtime data structures. + pub fn new() -> DummyRuntime { + DummyRuntime { globals: Vec::new() } + } +} + +impl WasmRuntime for DummyRuntime { + fn translate_get_global(&self, + builder: &mut FunctionBuilder, + global_index: GlobalIndex) + -> Value { + let ref glob = self.globals.get(global_index as usize).unwrap(); + match glob.ty { + I32 => builder.ins().iconst(glob.ty, -1), + I64 => builder.ins().iconst(glob.ty, -1), + F32 => builder.ins().f32const(Ieee32::with_bits(0xbf800000)), // -1.0 + F64 => { + builder + .ins() + .f64const(Ieee64::with_bits(0xbff0000000000000)) + } // -1.0 + _ => panic!("should not happen"), + } + } + + fn translate_set_global(&self, _: &mut FunctionBuilder, _: GlobalIndex, _: Value) { + // We do nothing + } + fn translate_grow_memory(&mut self, builder: &mut FunctionBuilder, _: Value) -> Value { + builder.ins().iconst(I32, -1) + } + fn translate_current_memory(&mut self, builder: &mut FunctionBuilder) -> Value { + builder.ins().iconst(I32, -1) + } + fn translate_call_indirect<'a>(&self, + builder: &'a mut FunctionBuilder, + sig_ref: SigRef, + index_val: Value, + call_args: &[Value]) + -> &'a [Value] { + let call_inst = builder.ins().call_indirect(sig_ref, index_val, call_args); + builder.inst_results(call_inst) + } + fn translate_memory_base_address(&self, + builder: &mut FunctionBuilder, + _: MemoryIndex) + -> Value { + builder.ins().iconst(I64, 0) + } + fn declare_global(&mut self, global: Global) { + self.globals.push(global); + } + fn declare_table(&mut self, _: Table) { + //We do nothing + } + fn declare_table_elements(&mut self, _: TableIndex, _: usize, _: &[FunctionIndex]) { + //We do nothing + } + fn declare_memory(&mut self, _: Memory) { + //We do nothing + } + fn declare_data_initialization(&mut self, + _: MemoryIndex, + _: usize, + _: &[u8]) + -> Result<(), String> { + // We do nothing + Ok(()) + } + + fn begin_translation(&mut self) { + // We do nothing + } + fn next_function(&mut self) { + // We do nothing + } +} diff --git a/lib/wasm/src/runtime/mod.rs b/lib/wasm/src/runtime/mod.rs new file mode 100644 index 0000000000..9f2a41d4f9 --- /dev/null +++ b/lib/wasm/src/runtime/mod.rs @@ -0,0 +1,5 @@ +mod spec; +mod dummy; + +pub use runtime::spec::WasmRuntime; +pub use runtime::dummy::DummyRuntime; diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs new file mode 100644 index 0000000000..24e98b0d58 --- /dev/null +++ b/lib/wasm/src/runtime/spec.rs @@ -0,0 +1,61 @@ +//! All the runtime support necessary for the wasm to cretonne translation is formalized by the +//! trait `WasmRuntime`. +use cton_frontend::FunctionBuilder; +use cretonne::ir::{Value, SigRef}; +use translation_utils::{Local, FunctionIndex, TableIndex, GlobalIndex, MemoryIndex, Global, Table, + Memory}; + +/// An object satisfyng the `WasmRuntime` trait can be passed as argument to the +/// [`translate_module`](fn.translate_module.html) function. These methods should not be called +/// by the user, they are only for the `wasm2cretonne` internal use. +pub trait WasmRuntime { + /// Declares a global to the runtime. + fn declare_global(&mut self, global: Global); + /// Declares a table to the runtime. + fn declare_table(&mut self, table: Table); + /// Fills a declared table with references to functions in the module. + fn declare_table_elements(&mut self, + table_index: TableIndex, + offset: usize, + elements: &[FunctionIndex]); + /// Declares a memory to the runtime + fn declare_memory(&mut self, memory: Memory); + /// Fills a declared memory with bytes at module instantiation. + fn declare_data_initialization(&mut self, + memory_index: MemoryIndex, + offset: usize, + data: &[u8]) + -> Result<(), String>; + /// Call this function after having declared all the runtime elements but prior to the + /// function body translation. + fn begin_translation(&mut self); + /// Call this function between each function body translation. + fn next_function(&mut self); + /// Translates a `get_global` wasm instruction. + fn translate_get_global(&self, + builder: &mut FunctionBuilder, + global_index: GlobalIndex) + -> Value; + /// Translates a `set_global` wasm instruction. + fn translate_set_global(&self, + builder: &mut FunctionBuilder, + global_index: GlobalIndex, + val: Value); + /// Translates a `grow_memory` wasm instruction. Returns the old size (in pages) of the memory. + fn translate_grow_memory(&mut self, builder: &mut FunctionBuilder, val: Value) -> Value; + /// Translates a `current_memory` wasm instruction. Returns the size in pages of the memory. + fn translate_current_memory(&mut self, builder: &mut FunctionBuilder) -> Value; + /// Returns the base address of a wasm memory as a Cretonne `Value`. + fn translate_memory_base_address(&self, + builder: &mut FunctionBuilder, + index: MemoryIndex) + -> Value; + /// Translates a `call_indirect` wasm instruction. It involves looking up the value contained + /// it the table at location `index_val` and calling the corresponding function. + fn translate_call_indirect<'a>(&self, + builder: &'a mut FunctionBuilder, + sig_ref: SigRef, + index_val: Value, + call_args: &[Value]) + -> &'a [Value]; +} diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs new file mode 100644 index 0000000000..c9954e00c6 --- /dev/null +++ b/lib/wasm/src/sections_translator.rs @@ -0,0 +1,372 @@ +//! Helper functions to gather information for each of the non-function sections of a +//! WebAssembly module. +//! +//! The code of theses helper function is straightforward since it is only about reading metadata +//! about linear memories, tables, globals, etc. and storing them for later use. +//! +//! The special case of the initialize expressions for table elements offsets or global variables +//! is handled, according to the semantics of WebAssembly, to only specific expressions that are +//! interpreted on the fly. +use translation_utils::{type_to_type, Import, TableIndex, FunctionIndex, GlobalIndex, + SignatureIndex, MemoryIndex, Global, GlobalInit, Table, TableElementType, + Memory}; +use cretonne::ir::{Signature, ArgumentType, CallConv}; +use cretonne; +use wasmparser::{Parser, ParserState, FuncType, ImportSectionEntryType, ExternalKind, WasmDecoder, + MemoryType, Operator}; +use wasmparser; +use std::collections::HashMap; +use std::str::from_utf8; +use runtime::WasmRuntime; + +pub enum SectionParsingError { + WrongSectionContent(String), +} + +/// Reads the Type Section of the wasm module and returns the corresponding function signatures. +pub fn parse_function_signatures(parser: &mut Parser) + -> Result, SectionParsingError> { + let mut signatures: Vec = Vec::new(); + loop { + match *parser.read() { + ParserState::EndSection => break, + ParserState::TypeSectionEntry(FuncType { + form: wasmparser::Type::Func, + ref params, + ref returns, + }) => { + let mut sig = Signature::new(CallConv::Native); + sig.argument_types + .extend(params + .iter() + .map(|ty| { + let cret_arg: cretonne::ir::Type = match type_to_type(ty) { + Ok(ty) => ty, + Err(()) => panic!("only numeric types are supported in\ + function signatures"), + }; + ArgumentType::new(cret_arg) + })); + sig.return_types + .extend(returns + .iter() + .map(|ty| { + let cret_arg: cretonne::ir::Type = match type_to_type(ty) { + Ok(ty) => ty, + Err(()) => panic!("only numeric types are supported in\ + function signatures"), + }; + ArgumentType::new(cret_arg) + })); + signatures.push(sig); + } + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + } + } + Ok(signatures) +} + +/// Retrieves the imports from the imports section of the binary. +pub fn parse_import_section(parser: &mut Parser) -> Result, SectionParsingError> { + let mut imports = Vec::new(); + loop { + match *parser.read() { + ParserState::ImportSectionEntry { + ty: ImportSectionEntryType::Function(sig), .. + } => imports.push(Import::Function { sig_index: sig }), + ParserState::ImportSectionEntry { + ty: ImportSectionEntryType::Memory(MemoryType { limits: ref memlimits }), .. + } => { + imports.push(Import::Memory(Memory { + pages_count: memlimits.initial as usize, + maximum: memlimits.maximum.map(|x| x as usize), + })) + } + ParserState::ImportSectionEntry { + ty: ImportSectionEntryType::Global(ref ty), .. + } => { + imports.push(Import::Global(Global { + ty: type_to_type(&ty.content_type).unwrap(), + mutability: ty.mutability != 0, + initializer: GlobalInit::Import(), + })); + } + ParserState::ImportSectionEntry { + ty: ImportSectionEntryType::Table(ref tab), .. + } => { + imports.push(Import::Table(Table { + ty: match type_to_type(&tab.element_type) { + Ok(t) => TableElementType::Val(t), + Err(()) => TableElementType::Func(), + }, + size: tab.limits.initial as usize, + maximum: tab.limits.maximum.map(|x| x as usize), + })); + } + ParserState::EndSection => break, + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + } + Ok(imports) +} + +/// Retrieves the correspondances between functions and signatures from the function section +pub fn parse_function_section(parser: &mut Parser) + -> Result, SectionParsingError> { + let mut funcs = Vec::new(); + loop { + match *parser.read() { + ParserState::FunctionSectionEntry(sigindex) => funcs.push(sigindex as SignatureIndex), + ParserState::EndSection => break, + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + } + Ok(funcs) +} + +/// Retrieves the names of the functions from the export section +pub fn parse_export_section(parser: &mut Parser) + -> Result, SectionParsingError> { + let mut exports: HashMap = HashMap::new(); + loop { + match *parser.read() { + ParserState::ExportSectionEntry { + field, + ref kind, + index, + } => { + match kind { + &ExternalKind::Function => { + exports.insert(index as FunctionIndex, + String::from(from_utf8(field).unwrap())); + () + } + _ => (),//TODO: deal with other kind of exports + } + } + ParserState::EndSection => break, + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + } + Ok(exports) +} + +/// Retrieves the size and maximum fields of memories from the memory section +pub fn parse_memory_section(parser: &mut Parser) -> Result, SectionParsingError> { + let mut memories: Vec = Vec::new(); + loop { + match *parser.read() { + ParserState::MemorySectionEntry(ref ty) => { + memories.push(Memory { + pages_count: ty.limits.initial as usize, + maximum: ty.limits.maximum.map(|x| x as usize), + }) + } + ParserState::EndSection => break, + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + } + Ok(memories) +} + +/// Retrieves the size and maximum fields of memories from the memory section +pub fn parse_global_section(parser: &mut Parser, + runtime: &mut WasmRuntime) + -> Result, SectionParsingError> { + let mut globals = Vec::new(); + loop { + let (content_type, mutability) = match *parser.read() { + ParserState::BeginGlobalSectionEntry(ref ty) => (ty.content_type, ty.mutability), + ParserState::EndSection => break, + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + match *parser.read() { + ParserState::BeginInitExpressionBody => (), + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + } + let initializer = match *parser.read() { + ParserState::InitExpressionOperator(Operator::I32Const { value }) => { + GlobalInit::I32Const(value) + } + ParserState::InitExpressionOperator(Operator::I64Const { value }) => { + GlobalInit::I64Const(value) + } + ParserState::InitExpressionOperator(Operator::F32Const { value }) => { + GlobalInit::F32Const(value.bits()) + } + ParserState::InitExpressionOperator(Operator::F64Const { value }) => { + GlobalInit::F64Const(value.bits()) + } + ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => { + GlobalInit::GlobalRef(global_index as GlobalIndex) + } + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + + }; + match *parser.read() { + ParserState::EndInitExpressionBody => (), + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + } + let global = Global { + ty: type_to_type(&content_type).unwrap(), + mutability: mutability != 0, + initializer: initializer, + }; + runtime.declare_global(global.clone()); + globals.push(global); + match *parser.read() { + ParserState::EndGlobalSectionEntry => (), + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + } + } + Ok(globals) +} + +pub fn parse_data_section(parser: &mut Parser, + runtime: &mut WasmRuntime, + globals: &Vec) + -> Result<(), SectionParsingError> { + loop { + let memory_index = match *parser.read() { + ParserState::BeginDataSectionEntry(memory_index) => memory_index, + ParserState::EndSection => break, + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + match *parser.read() { + ParserState::BeginInitExpressionBody => (), + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + let offset = match *parser.read() { + ParserState::InitExpressionOperator(Operator::I32Const { value }) => { + if value < 0 { + return Err(SectionParsingError::WrongSectionContent(String::from("negative \ + offset value"))); + } else { + value as usize + } + } + ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => { + match globals[global_index as usize].initializer { + GlobalInit::I32Const(value) => { + if value < 0 { + return Err(SectionParsingError::WrongSectionContent(String::from("\ + negative offset value"))); + } else { + value as usize + } + } + GlobalInit::Import() => { + return Err(SectionParsingError::WrongSectionContent(String::from("\ + imported globals not supported"))) + } // TODO: add runtime support + _ => panic!("should not happen"), + } + } + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + match *parser.read() { + ParserState::EndInitExpressionBody => (), + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + { + let data = match *parser.read() { + ParserState::DataSectionEntryBody(data) => data, + ref s @ _ => { + return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))) + } + }; + match runtime.declare_data_initialization(memory_index as MemoryIndex, offset, data) { + Ok(()) => (), + Err(s) => return Err(SectionParsingError::WrongSectionContent(format!("{}", s))), + }; + } + match *parser.read() { + ParserState::EndDataSectionEntry => (), + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + } + Ok(()) +} + +/// Retrieves the tables from the table section +pub fn parse_table_section(parser: &mut Parser, + runtime: &mut WasmRuntime) + -> Result<(), SectionParsingError> { + loop { + match *parser.read() { + ParserState::TableSectionEntry(ref table) => { + runtime.declare_table(Table { + ty: match type_to_type(&table.element_type) { + Ok(t) => TableElementType::Val(t), + Err(()) => TableElementType::Func(), + }, + size: table.limits.initial as usize, + maximum: table.limits.maximum.map(|x| x as usize), + }) + } + ParserState::EndSection => break, + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + } + Ok(()) +} + +/// Retrieves the tables from the table section +pub fn parse_elements_section(parser: &mut Parser, + runtime: &mut WasmRuntime, + globals: &Vec) + -> Result<(), SectionParsingError> { + loop { + let table_index = match *parser.read() { + ParserState::BeginElementSectionEntry(ref table_index) => *table_index as TableIndex, + ParserState::EndSection => break, + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + match *parser.read() { + ParserState::BeginInitExpressionBody => (), + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + let offset = match *parser.read() { + ParserState::InitExpressionOperator(Operator::I32Const { value }) => { + if value < 0 { + return Err(SectionParsingError::WrongSectionContent(String::from("negative \ + offset value"))); + } else { + value as usize + } + } + ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => { + match globals[global_index as usize].initializer { + GlobalInit::I32Const(value) => { + if value < 0 { + return Err(SectionParsingError::WrongSectionContent(String::from("\ + negative offset value"))); + } else { + value as usize + } + } + GlobalInit::Import() => 0, // TODO: add runtime support + _ => panic!("should not happen"), + } + } + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + match *parser.read() { + ParserState::EndInitExpressionBody => (), + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + match *parser.read() { + ParserState::ElementSectionEntryBody(ref elements) => { + let elems: Vec = + elements.iter().map(|&x| x as FunctionIndex).collect(); + runtime.declare_table_elements(table_index, offset, elems.as_slice()) + } + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + match *parser.read() { + ParserState::EndElementSectionEntry => (), + ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + } + Ok(()) +} diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs new file mode 100644 index 0000000000..04921d254f --- /dev/null +++ b/lib/wasm/src/translation_utils.rs @@ -0,0 +1,138 @@ +///! Helper functions and structures for the translation. +use wasmparser; +use cretonne; +use std::u32; +use code_translator; +use module_translator; + +/// Index of a function (imported or defined) inside the WebAssembly module. +pub type FunctionIndex = usize; +/// Index of a table (imported or defined) inside the WebAssembly module. +pub type TableIndex = usize; +/// Index of a global variable (imported or defined) inside the WebAssembly module. +pub type GlobalIndex = usize; +/// Index of a linear memory (imported or defined) inside the WebAssembly module. +pub type MemoryIndex = usize; +/// Index of a signature (imported or defined) inside the WebAssembly module. +pub type SignatureIndex = usize; +/// Raw byte read from memory. +pub type RawByte = u8; +/// Pointer referring to a memory address. +pub type MemoryAddress = usize; + +/// WebAssembly import. +#[derive(Debug,Clone,Copy)] +pub enum Import { + Function { sig_index: u32 }, + Memory(Memory), + Global(Global), + Table(Table), +} + +/// WebAssembly global. +#[derive(Debug,Clone,Copy)] +pub struct Global { + pub ty: cretonne::ir::Type, + pub mutability: bool, + pub initializer: GlobalInit, +} + +/// Globals are initialized via the four `const` operators or by referring to another import. +#[derive(Debug,Clone,Copy)] +pub enum GlobalInit { + I32Const(i32), + I64Const(i64), + F32Const(u32), + F64Const(u64), + Import(), + GlobalRef(GlobalIndex), +} + +/// WebAssembly table. +#[derive(Debug,Clone,Copy)] +pub struct Table { + pub ty: TableElementType, + pub size: usize, + pub maximum: Option, +} + +/// WebAssembly table element. Can be a function or a scalar type. +#[derive(Debug,Clone,Copy)] +pub enum TableElementType { + Val(cretonne::ir::Type), + Func(), +} + +/// WebAssembly linear memory. +#[derive(Debug,Clone,Copy)] +pub struct Memory { + pub pages_count: usize, + pub maximum: Option, +} + +/// Wrapper to a `get_local` and `set_local` index. They are WebAssembly's non-SSA variables. +#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +pub struct Local(pub u32); +impl cretonne::entity::EntityRef for Local { + fn new(index: usize) -> Self { + assert!(index < (u32::MAX as usize)); + Local(index as u32) + } + + fn index(self) -> usize { + self.0 as usize + } +} +impl Default for Local { + fn default() -> Local { + Local(u32::MAX) + } +} + +/// Helper function translating wasmparser types to Cretonne types when possible. +pub fn type_to_type(ty: &wasmparser::Type) -> Result { + match *ty { + wasmparser::Type::I32 => Ok(cretonne::ir::types::I32), + wasmparser::Type::I64 => Ok(cretonne::ir::types::I64), + wasmparser::Type::F32 => Ok(cretonne::ir::types::F32), + wasmparser::Type::F64 => Ok(cretonne::ir::types::F64), + _ => Err(()), + } +} + +/// Turns a `wasmparser` `f32` into a `Cretonne` one. +pub fn f32_translation(x: wasmparser::Ieee32) -> cretonne::ir::immediates::Ieee32 { + cretonne::ir::immediates::Ieee32::with_bits(x.bits()) +} + +/// Turns a `wasmparser` `f64` into a `Cretonne` one. +pub fn f64_translation(x: wasmparser::Ieee64) -> cretonne::ir::immediates::Ieee64 { + cretonne::ir::immediates::Ieee64::with_bits(x.bits()) +} + +/// Translate a `wasmparser` type into its `Cretonne` equivalent, when possible +pub fn translate_type(ty: wasmparser::Type) -> Result, ()> { + match ty { + wasmparser::Type::EmptyBlockType => Ok(Vec::new()), + wasmparser::Type::I32 => Ok(vec![cretonne::ir::types::I32]), + wasmparser::Type::F32 => Ok(vec![cretonne::ir::types::F32]), + wasmparser::Type::I64 => Ok(vec![cretonne::ir::types::I64]), + wasmparser::Type::F64 => Ok(vec![cretonne::ir::types::F64]), + _ => panic!("unsupported return value type"), + } +} + +/// Inverts the key-value relation in the imports hashmap. Indeed, these hashmaps are built by +/// feeding the function indexes in the module but are used by the runtime with the `FuncRef` as +/// keys. +pub fn invert_hashmaps(imports: code_translator::FunctionImports) + -> module_translator::ImportMappings { + let mut new_imports = module_translator::ImportMappings::new(); + for (func_index, func_ref) in imports.functions.iter() { + new_imports.functions.insert(*func_ref, *func_index); + } + for (sig_index, sig_ref) in imports.signatures.iter() { + new_imports.signatures.insert(*sig_ref, *sig_index); + } + new_imports +} diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/testsuite.rs new file mode 100644 index 0000000000..ed3f339426 --- /dev/null +++ b/lib/wasm/tests/testsuite.rs @@ -0,0 +1,102 @@ +extern crate cretonne_wasm; +extern crate cretonne; + +use cretonne_wasm::{translate_module, FunctionTranslation, DummyRuntime, WasmRuntime}; +use std::path::PathBuf; +use std::fs::File; +use std::error::Error; +use std::io; +use std::io::BufReader; +use std::io::prelude::*; +use std::fs; +use cretonne::ir; +use cretonne::ir::entities::AnyEntity; +use cretonne::isa::TargetIsa; +use cretonne::verifier; + +#[test] +fn testsuite() { + let mut paths: Vec<_> = fs::read_dir("../../wasmtests") + .unwrap() + .map(|r| r.unwrap()) + .collect(); + paths.sort_by_key(|dir| dir.path()); + for path in paths { + let path = path.path(); + match handle_module(path) { + Ok(()) => (), + Err(message) => println!("{}", message), + }; + } +} + +fn read_wasm_file(path: PathBuf) -> Result, io::Error> { + let mut buf: Vec = Vec::new(); + let file = File::open(path)?; + let mut buf_reader = BufReader::new(file); + buf_reader.read_to_end(&mut buf)?; + Ok(buf) +} + +fn handle_module(path: PathBuf) -> Result<(), String> { + let data = match path.extension() { + None => { + return Err(String::from("the file extension is not wasm or wast")); + } + Some(ext) => { + match ext.to_str() { + Some("wasm") => { + match read_wasm_file(path.clone()) { + Ok(data) => data, + Err(err) => { + return Err(String::from(err.description())); + } + } + } + None | Some(&_) => { + return Err(String::from("the file extension is not wasm or wast")); + } + } + } + }; + let mut dummy_runtime = DummyRuntime::new(); + let translation = { + let mut runtime: &mut WasmRuntime = &mut dummy_runtime; + match translate_module(&data, runtime) { + Ok(x) => x, + Err(string) => { + return Err(string); + } + } + }; + for func in translation.functions.iter() { + let il = match func { + &FunctionTranslation::Import() => continue, + &FunctionTranslation::Code { ref il, .. } => il.clone(), + }; + match verifier::verify_function(&il, None) { + Ok(()) => (), + Err(err) => return Err(pretty_verifier_error(&il, None, err)), + } + } + Ok(()) +} + + +/// Pretty-print a verifier error. +pub fn pretty_verifier_error(func: &ir::Function, + isa: Option<&TargetIsa>, + err: verifier::Error) + -> String { + let msg = err.to_string(); + let str1 = match err.location { + AnyEntity::Inst(inst) => { + format!("{}\n{}: {}\n\n", + msg, + inst, + func.dfg.display_inst(inst, isa)) + } + _ => String::from(format!("{}\n", msg)), + }; + format!("{}{}", str1, func.display(isa)) +} From f905dc914b321d2bce979724d177c5653431f1bf Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 28 Aug 2017 17:07:28 -0700 Subject: [PATCH 0988/3084] Enable missing_docs errors in the wasm crate. This adds `#![deny(missing_docs)]` to the wasm crate, and adds documentation to several struct and enum fields, as needed. --- lib/wasm/src/lib.rs | 2 ++ lib/wasm/src/module_translator.rs | 14 +++++++++----- lib/wasm/src/translation_utils.rs | 16 +++++++++++++++- 3 files changed, 26 insertions(+), 6 deletions(-) diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 5006e5c9aa..bb342bf7c6 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -10,6 +10,8 @@ //! //! The main function of this module is [`translate_module`](fn.translate_module.html). +#![deny(missing_docs)] + extern crate wasmparser; extern crate cton_frontend; extern crate cretonne; diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 1c22655300..10dbbbc43b 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -12,22 +12,25 @@ use cton_frontend::ILBuilder; use std::collections::HashMap; use runtime::WasmRuntime; -/// Output of the [`translate_module`](fn.translate_module.html) function. Contains the translated -/// functions and when present the index of the function defined as `start` of the module. +/// Output of the [`translate_module`](fn.translate_module.html) function. pub struct TranslationResult { + /// The translated functions. pub functions: Vec, + /// When present, the index of the function defined as `start` of the module. pub start_index: Option, } -/// A function in a WebAssembly module can be either imported, or defined inside it. If it is -/// defined inside it, then the translation in Cretonne IL is available as well as the mappings -/// between Cretonne imports and indexes in the function index space. +/// A function in a WebAssembly module can be either imported, or defined inside it. #[derive(Clone)] pub enum FunctionTranslation { + /// A function defined inside the WebAssembly module. Code { + /// The translation in Cretonne IL. il: Function, + /// The mappings between Cretonne imports and indexes in the function index space. imports: ImportMappings, }, + /// An imported function. Import(), } @@ -42,6 +45,7 @@ pub struct ImportMappings { } impl ImportMappings { + /// Create a new empty `ImportMappings`. pub fn new() -> ImportMappings { ImportMappings { functions: HashMap::new(), diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 04921d254f..609b4a3dfc 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -32,27 +32,39 @@ pub enum Import { /// WebAssembly global. #[derive(Debug,Clone,Copy)] pub struct Global { + /// The type of the value stored in the global. pub ty: cretonne::ir::Type, + /// A flag indicating whether the value may change at runtime. pub mutability: bool, + /// The source of the initial value. pub initializer: GlobalInit, } /// Globals are initialized via the four `const` operators or by referring to another import. #[derive(Debug,Clone,Copy)] pub enum GlobalInit { + /// An `i32.const`. I32Const(i32), + /// An `i64.const`. I64Const(i64), + /// An `f32.const`. F32Const(u32), + /// An `f64.const`. F64Const(u64), - Import(), + /// A `get_global` of another global. GlobalRef(GlobalIndex), + ///< The global is imported from, and thus initialized by, a different module. + Import(), } /// WebAssembly table. #[derive(Debug,Clone,Copy)] pub struct Table { + /// The type of data stored in elements of the table. pub ty: TableElementType, + /// The minimum number of elements in the table. pub size: usize, + /// The maximum number of elements in the table. pub maximum: Option, } @@ -66,7 +78,9 @@ pub enum TableElementType { /// WebAssembly linear memory. #[derive(Debug,Clone,Copy)] pub struct Memory { + /// The minimum number of pages in the memory. pub pages_count: usize, + /// The maximum number of pages in the memory. pub maximum: Option, } From 253f602a4c942b42d7dcc1e4b510cadc59ee93e6 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 28 Aug 2017 17:17:41 -0700 Subject: [PATCH 0989/3084] Fix a few warnings. --- cranelift/src/cton-util.rs | 1 - cranelift/src/wasm.rs | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index e85eb138c7..5b7991bace 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -2,7 +2,6 @@ extern crate cretonne; extern crate cton_reader; extern crate cretonne_wasm; -extern crate wasmparser; extern crate docopt; extern crate serde; #[macro_use] diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 953bba50c5..c83df807fc 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -129,7 +129,7 @@ fn handle_module(flag_verbose: bool, }; let mut dummy_runtime = DummyRuntime::new(); let translation = { - let mut runtime: &mut WasmRuntime = &mut dummy_runtime; + let runtime: &mut WasmRuntime = &mut dummy_runtime; match translate_module(&data, runtime) { Ok(x) => x, Err(string) => { From c71d4fc01b0650abd909c0af54a4fc057c6173e3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 Aug 2017 03:30:22 -0700 Subject: [PATCH 0990/3084] Remove a stale comment. --- lib/wasm/src/code_translator.rs | 2 -- 1 file changed, 2 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index b1647a9aee..748a151e45 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -129,8 +129,6 @@ impl ControlStackFrame { /// - if the last instruction added was a `return`; /// - the depth of the two unreachable control blocks stacks, that are manipulated when translating /// unreachable code; -/// - all the `Ebb`s referenced by `br_table` instructions, because those are always reachable even -/// if they are at a point of the code that would have been unreachable otherwise. struct TranslationState { last_inst_return: bool, phantom_unreachable_stack_depth: usize, From ea1d05383125dbb2a2b5eeb866957a15ab966546 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 Aug 2017 03:36:38 -0700 Subject: [PATCH 0991/3084] Simplify the "am I in unreachable code" predicate. The phantom unreachable stack is only used when the real unreachable stack is active, so it's sufficient to check whether the real unreachable stack is empty. --- lib/wasm/src/code_translator.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 748a151e45..696d91b304 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -236,8 +236,9 @@ pub fn translate_function_body(parser: &mut Parser, let parser_state = parser.read(); match *parser_state { ParserState::CodeOperator(ref op) => { - if state.phantom_unreachable_stack_depth + - state.real_unreachable_stack_depth > 0 { + debug_assert!(state.phantom_unreachable_stack_depth == 0 || + state.real_unreachable_stack_depth > 0); + if state.real_unreachable_stack_depth > 0 { translate_unreachable_operator(op, &mut builder, &mut stack, From 4afa5df3b6b488c37c75dde00ef4c7a585ed5a37 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 Aug 2017 04:40:48 -0700 Subject: [PATCH 0992/3084] Remove the last_inst_return field. It's not necessary to explicitly track whether the last instruction is a return; if the builder for the last block isn't filled by the time we reach the end, it needs a return to fill it. --- lib/wasm/src/code_translator.rs | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 696d91b304..f2521fcddb 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -126,11 +126,9 @@ impl ControlStackFrame { /// Contains information passed along during the translation and that records: /// -/// - if the last instruction added was a `return`; /// - the depth of the two unreachable control blocks stacks, that are manipulated when translating /// unreachable code; struct TranslationState { - last_inst_return: bool, phantom_unreachable_stack_depth: usize, real_unreachable_stack_depth: usize, } @@ -216,7 +214,6 @@ pub fn translate_function_body(parser: &mut Parser, } } let mut state = TranslationState { - last_inst_return: false, phantom_unreachable_stack_depth: 0, real_unreachable_stack_depth: 0, }; @@ -265,8 +262,7 @@ pub fn translate_function_body(parser: &mut Parser, } // In WebAssembly, the final return instruction is implicit so we need to build it // explicitely in Cretonne IL. - if !state.last_inst_return && !builder.is_filled() && - (!builder.is_unreachable() || !builder.is_pristine()) { + if !builder.is_filled() && (!builder.is_unreachable() || !builder.is_pristine()) { let cut_index = stack.len() - sig.return_types.len(); let return_vals = stack.split_off(cut_index); builder.ins().return_(return_vals.as_slice()); @@ -300,7 +296,6 @@ fn translate_operator(op: &Operator, signatures: &Vec, exports: &Option>, func_imports: &mut FunctionImports) { - state.last_inst_return = false; // This big match treats all Wasm code operators. match *op { /********************************** Locals **************************************** @@ -600,7 +595,6 @@ fn translate_operator(op: &Operator, let cut_index = stack.len() - return_count; let return_args = stack.split_off(cut_index); builder.ins().return_(return_args.as_slice()); - state.last_inst_return = true; state.real_unreachable_stack_depth = 1; } /************************************ Calls **************************************** @@ -1265,7 +1259,6 @@ fn translate_unreachable_operator(op: &Operator, stack.extend_from_slice(builder.ebb_args(frame.following_code())); } state.real_unreachable_stack_depth -= 1; - state.last_inst_return = false; } } Operator::Else => { @@ -1292,7 +1285,6 @@ fn translate_unreachable_operator(op: &Operator, // by unreachable code that hasn't been translated stack.truncate(original_stack_size); state.real_unreachable_stack_depth = 0; - state.last_inst_return = false; } } _ => { From c380df1d04fcda885b9017524a98b7f0ca6361a5 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 Aug 2017 05:44:40 -0700 Subject: [PATCH 0993/3084] Eliminate a heap allocation. --- lib/wasm/src/code_translator.rs | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index f2521fcddb..efd54617f4 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -167,10 +167,6 @@ pub fn translate_function_body(parser: &mut Parser, // First we build the Function object with its name and signature let mut func = Function::new(); let args_num: usize = sig.argument_types.len(); - let args_types: Vec = sig.argument_types - .iter() - .map(|arg| arg.value_type) - .collect(); func.signature = sig.clone(); match exports { &None => (), @@ -190,11 +186,11 @@ pub fn translate_function_body(parser: &mut Parser, let first_ebb = builder.create_ebb(); builder.switch_to_block(first_ebb, &[]); builder.seal_block(first_ebb); - for i in 0..args_num { + for (i, arg_type) in sig.argument_types.iter().enumerate() { // First we declare the function arguments' as non-SSA vars because they will be // accessed by get_local let arg_value = builder.arg_value(i as usize); - builder.declare_var(Local(i as u32), args_types[i]); + builder.declare_var(Local(i as u32), arg_type.value_type); builder.def_var(Local(i as u32), arg_value); } // We also declare and initialize to 0 the local variables From 0deaa616a3eed4464740a1c1281f70ed8288f281 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 29 Aug 2017 10:24:30 -0700 Subject: [PATCH 0994/3084] Record identity assignments in regalloc constraint solver. Fixes #147. The Solver::reassign_in() method would previously not record fixed register assignments for values that are already in the correct register. The register would simply be marked as unavailable for the solver. This did have the effect of tripping up the sanity checks in Solver::add_var() when that method was called with such a "reassigned" value. The function can be called for a value that already has a fixed assignment, but the sanity checks want to make sure the variable constraints are compatible with the existing fixed assignment. When no such assignment could be found, the method panicked. To fix this, make sure that even identity reassignments are recorded in the assignments vector. Instead, filter the identity assignments out before scheduling a move sequence for the assignments. Also add some debug tracing to the regalloc solver. --- .../filetests/regalloc/intel-regres.cton | 40 ++++++++ lib/cretonne/src/regalloc/solver.rs | 92 ++++++++++++++----- 2 files changed, 110 insertions(+), 22 deletions(-) create mode 100644 cranelift/filetests/regalloc/intel-regres.cton diff --git a/cranelift/filetests/regalloc/intel-regres.cton b/cranelift/filetests/regalloc/intel-regres.cton new file mode 100644 index 0000000000..24d13f5216 --- /dev/null +++ b/cranelift/filetests/regalloc/intel-regres.cton @@ -0,0 +1,40 @@ +test regalloc + +isa intel + +; regex: V=v\d+ + +; The value v9 appears both as the branch control and one of the EBB arguments +; in the brnz instruction in ebb2. It also happens that v7 and v9 are assigned +; to the same register, so v9 doesn't need to be moved before the brnz. +; +; This ended up confusong the constraint solver which had not made a record of +; the fixed register assignment for v9 since it was already in the correct +; register. +function %pr147(i32) -> i32 native { +ebb0(v0: i32): + v1 = iconst.i32 0 + v2 = iconst.i32 1 + v3 = iconst.i32 0 + jump ebb2(v3, v2, v0) + +ebb2(v4: i32, v5: i32, v7: i32): + ; check: $ebb2 + v6 = iadd v4, v5 + v8 = iconst.i32 -1 + ; v7 is killed here and v9 gets the same register. + v9 = iadd v7, v8 + ; check: $v9 = iadd $v7, $v8 + ; Here v9 the brnz control appears to interfere with v9 the EBB argument, + ; so divert_fixed_input_conflicts() calls add_var(v9), which is ok. The + ; add_var sanity checks got confused when no fixed assignment could be + ; found for v9. + ; + ; We should be able to handle this situation without making copies of v9. + brnz v9, ebb2(v5, v6, v9) + ; check: brnz $v9, $ebb2($V, $V, $v9) + jump ebb3 + +ebb3: + return v5 +} diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index c443180be2..2847a77b84 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -18,13 +18,6 @@ //! assignments. Some are used by the instruction, some are not. //! - A subset of the live register values that are killed by the instruction. //! - A set of new register values that are defined by the instruction. -//! The constraint solver addresses the problem of satisfying the constraints of a single -//! instruction. We have: -//! -//! - A set of values that are live in registers before the instruction, with current register -//! assignments. Some are used by the instruction, some are not. -//! - A subset of the live register values that are killed by the instruction. -//! - A set of new register values that are defined by the instruction. //! //! We are not concerned with stack values at all. The reload pass ensures that all values required //! to be in a register by the instruction are already in a register. @@ -32,7 +25,7 @@ //! A solution to the register coloring problem consists of: //! //! - Register reassignment prescriptions for a subset of the live register values. -//! - Register assignments for the defined values. +//! - Register assignments for the instruction's defined values. //! //! The solution ensures that when live registers are reassigned as prescribed before the //! instruction, all its operand constraints are satisfied, and the definition assignments won't @@ -105,6 +98,7 @@ //! appropriate candidate among the set of live register values, add it as a variable and start //! over. +use dbg::DisplayList; use entity::{SparseMap, SparseMapValue}; use ir::Value; use isa::{RegInfo, RegClass, RegUnit}; @@ -235,6 +229,17 @@ impl SparseMapValue for Assignment { } } +impl fmt::Display for Assignment { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, + "{}:{}(%{} -> %{})", + self.value, + self.rc, + self.from, + self.to) + } +} + #[cfg(test)] impl PartialEq for Assignment { fn eq(&self, other: &Assignment) -> bool { @@ -263,10 +268,12 @@ impl PartialEq for Assignment { /// pub struct Solver { /// Register reassignments that are required or decided as part of a full solution. + /// This includes identity assignments for values that are already in the correct fixed + /// register. assignments: SparseMap, /// Variables are the values that should be reassigned as part of a solution. - /// Values with a fixed register constraints are not considered variables. They are represented + /// Values with fixed register constraints are not considered variables. They are represented /// in the `assignments` vector if necessary. vars: Vec, @@ -346,13 +353,14 @@ impl Solver { /// In either case, `to` will not be available for variables on the input side of the /// instruction. pub fn reassign_in(&mut self, value: Value, rc: RegClass, from: RegUnit, to: RegUnit) { + dbg!("reassign_in({}:{}, %{} -> %{})", value, rc, from, to); debug_assert!(!self.inputs_done); if self.regs_in.is_avail(rc, from) { // It looks like `value` was already removed from the register set. It must have been // added as a variable previously. A fixed constraint beats a variable, so convert it. if let Some(idx) = self.vars.iter().position(|v| v.value == value) { let v = self.vars.remove(idx); - dbg!("Converting variable {} to a fixed constraint", v); + dbg!("-> converting variable {} to a fixed constraint", v); // The spiller is responsible for ensuring that all constraints on the uses of a // value are compatible. assert!(v.constraint.contains(to), @@ -364,15 +372,13 @@ impl Solver { } self.regs_in.free(rc, from); self.regs_out.take(rc, to); - if from != to { - self.assignments - .insert(Assignment { - value, - rc, - from, - to, - }); - } + self.assignments + .insert(Assignment { + value, + rc, + from, + to, + }); } /// Add a variable representing an input side value with an existing register assignment. @@ -389,8 +395,16 @@ impl Solver { reginfo: &RegInfo) { // Check for existing entries for this value. if self.regs_in.is_avail(constraint, from) { - // There cold be an existing variable entry. + dbg!("add_var({}:{}, from={}/%{}) for existing entry", + value, + constraint, + reginfo.display_regunit(from), + from); + + // There could be an existing variable entry. if let Some(v) = self.vars.iter_mut().find(|v| v.value == value) { + dbg!("-> combining constraint with {}", v); + // We have an existing variable entry for `value`. Combine the constraints. if let Some(rci) = v.constraint.intersect(constraint) { v.constraint = reginfo.rc(rci); @@ -404,19 +418,30 @@ impl Solver { // No variable, then it must be a fixed reassignment. if let Some(a) = self.assignments.get(value) { + dbg!("-> already fixed assignment {}", a); assert!(constraint.contains(a.to), "Incompatible constraints for {}", value); return; } + dbg!("{}", self); panic!("Wrong from register for {}", value); } + + let new_var = Variable::new_live(value, constraint, from); + dbg!("add_var({}:{}, from={}/%{}) new entry: {}", + value, + constraint, + reginfo.display_regunit(from), + from, + new_var); + self.regs_in.free(constraint, from); if self.inputs_done { self.regs_out.free(constraint, from); } - self.vars.push(Variable::new_live(value, constraint, from)); + self.vars.push(new_var); } /// Check for conflicts between fixed input assignments and existing live values. @@ -608,7 +633,15 @@ impl Solver { } } - self.moves.extend(self.assignments.values().cloned()); + // Convert all of the fixed register assignments into moves, but omit the ones that are + // already in the right register. + self.moves + .extend(self.assignments + .values() + .cloned() + .filter(|v| v.from != v.to)); + + dbg!("collect_moves: {}", DisplayList(self.moves.as_slice())); } /// Try to schedule a sequence of `regmove` instructions that will shuffle registers into @@ -636,6 +669,7 @@ impl Solver { let m = &self.moves[i]; avail.take(m.rc, m.to); avail.free(m.rc, m.from); + dbg!("move #{}: {}", i, m); i += 1; continue; } @@ -666,6 +700,8 @@ impl Solver { let m = self.moves[i].clone(); if let Some(reg) = avail.iter(m.rc).next() { + dbg!("breaking cycle at {} with available register %{}", m, reg); + // Alter the move so it is guaranteed to be picked up when we loop. It is important // that this move is scheduled immediately, otherwise we would have multiple moves // of the same value, and they would not be commutable. @@ -699,6 +735,18 @@ impl Solver { } } +impl fmt::Display for Solver { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + writeln!(f, "Solver {{ inputs_done: {},", self.inputs_done)?; + writeln!(f, + " assignments: {}", + DisplayList(self.assignments.as_slice()))?; + writeln!(f, " vars: {}", DisplayList(self.vars.as_slice()))?; + writeln!(f, " moves: {}", DisplayList(self.moves.as_slice()))?; + writeln!(f, "}}") + } +} + #[cfg(test)] #[cfg(build_arm32)] mod tests { From 5303e7708bd2edc05c609252abcc5baae9e7ca33 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 Aug 2017 06:16:22 -0700 Subject: [PATCH 0995/3084] Handle wasmparser errors gracefully. --- lib/wasm/src/module_translator.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 10dbbbc43b..11acbb1b4a 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -1,6 +1,6 @@ //! Translation skeletton that traverses the whole WebAssembly module and call helper functions //! to deal with each part of it. -use wasmparser::{ParserState, SectionCode, ParserInput, Parser, WasmDecoder}; +use wasmparser::{ParserState, SectionCode, ParserInput, Parser, WasmDecoder, BinaryReaderError}; use sections_translator::{SectionParsingError, parse_function_signatures, parse_import_section, parse_function_section, parse_export_section, parse_memory_section, parse_global_section, parse_table_section, parse_elements_section, @@ -64,6 +64,9 @@ pub fn translate_module(data: &Vec, let mut parser = Parser::new(data.as_slice()); match *parser.read() { ParserState::BeginWasm { .. } => {} + ParserState::Error(BinaryReaderError { message, offset }) => { + return Err(format!("at offset {}: {}", offset, message)); + } ref s @ _ => panic!("modules should begin properly: {:?}", s), } let mut signatures = None; From b2fcb1ad177fb5a467f9b9355af6ad798328dd93 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 Aug 2017 06:21:28 -0700 Subject: [PATCH 0996/3084] Change translate_module to use a slice rather than a borrowed Vec. This makes it more convenient to call from a cargo-fuzz fuzzer. --- lib/wasm/src/module_translator.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 11acbb1b4a..99071b1bc8 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -58,10 +58,10 @@ impl ImportMappings { /// [`Function`](../cretonne/ir/function/struct.Function.html). /// Returns the functions and also the mappings for imported functions and signature between the /// indexes in the wasm module and the indexes inside each functions. -pub fn translate_module(data: &Vec, +pub fn translate_module(data: &[u8], runtime: &mut WasmRuntime) -> Result { - let mut parser = Parser::new(data.as_slice()); + let mut parser = Parser::new(data); match *parser.read() { ParserState::BeginWasm { .. } => {} ParserState::Error(BinaryReaderError { message, offset }) => { From dcb65b59c1edcd8f3bf53f0e2fb23b4ff0305da1 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 Aug 2017 06:25:19 -0700 Subject: [PATCH 0997/3084] Remove an unnecessary `mut`, fixing a compiler warning. --- lib/wasm/tests/testsuite.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/testsuite.rs index ed3f339426..3853ba1703 100644 --- a/lib/wasm/tests/testsuite.rs +++ b/lib/wasm/tests/testsuite.rs @@ -61,7 +61,7 @@ fn handle_module(path: PathBuf) -> Result<(), String> { }; let mut dummy_runtime = DummyRuntime::new(); let translation = { - let mut runtime: &mut WasmRuntime = &mut dummy_runtime; + let runtime: &mut WasmRuntime = &mut dummy_runtime; match translate_module(&data, runtime) { Ok(x) => x, Err(string) => { From 9ea5226b891c502f3195732d3e8705a11849ff3d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 30 Aug 2017 14:32:42 -0700 Subject: [PATCH 0998/3084] Add simple_gvn to the optimization pipeline for WebAssembly functions. --- cranelift/src/wasm.rs | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index c83df807fc..49292dbe29 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -196,6 +196,18 @@ fn handle_module(flag_verbose: bool, } } }; + match context.simple_gvn() { + Ok(())=> (), + Err(error) => { + match error { + CtonError::Verifier(err) => { + return Err(pretty_verifier_error(&context.func, None, err)); + } + CtonError::ImplLimitExceeded | + CtonError::CodeTooLarge => return Err(String::from(error.description())), + } + } + }; match verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) { Ok(()) => (), Err(err) => return Err(pretty_verifier_error(&context.func, None, err)), From 3532c3533a00b553d1ffb28345f903a1501e1cd7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 30 Aug 2017 14:33:54 -0700 Subject: [PATCH 0999/3084] Teach simple_gvn that iconst.i32 is not congruent to iconst.i64. --- cranelift/filetests/simple_gvn/reject.cton | 14 ++++++++++++++ lib/cretonne/src/ir/types.rs | 2 +- lib/cretonne/src/simple_gvn.rs | 7 ++++--- 3 files changed, 19 insertions(+), 4 deletions(-) diff --git a/cranelift/filetests/simple_gvn/reject.cton b/cranelift/filetests/simple_gvn/reject.cton index 755891ad5d..1bb6ad16fa 100644 --- a/cranelift/filetests/simple_gvn/reject.cton +++ b/cranelift/filetests/simple_gvn/reject.cton @@ -9,3 +9,17 @@ ebb0(v0: i32): ; check: regmove v0, %10 -> %20 return v0 } + +function %differing_typevars() -> i64 { +ebb0: + v0 = iconst.i32 7 + v1 = iconst.i64 7 + v2 = iconst.i64 8 +; check: v0 = iconst.i32 7 +; check: v1 = iconst.i64 7 +; check: v2 = iconst.i64 8 + v3 = uextend.i64 v0 + v4 = iadd v2, v1 + v5 = iadd v4, v3 + return v5 +} diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 0007154025..275ec70c9b 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -23,7 +23,7 @@ use std::fmt::{self, Display, Debug, Formatter}; /// /// SIMD vector types have power-of-two lanes, up to 256. Lanes can be any int/float/bool type. /// -#[derive(Copy, Clone, PartialEq, Eq)] +#[derive(Copy, Clone, PartialEq, Eq, Hash)] pub struct Type(u8); /// No type. Used for functions without a return value. Can't be loaded or stored. Can't be part of diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index 2d1c5a7e61..8a8022ed2d 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -2,7 +2,7 @@ use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; -use ir::{Cursor, CursorBase, InstructionData, Function, Inst, Opcode}; +use ir::{Cursor, CursorBase, InstructionData, Function, Inst, Opcode, Type}; use std::collections::HashMap; /// Test whether the given opcode is unsafe to even consider for GVN. @@ -14,7 +14,7 @@ fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { /// Perform simple GVN on `func`. /// pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph) { - let mut visible_values: HashMap = HashMap::new(); + let mut visible_values: HashMap<(InstructionData, Type), Inst> = HashMap::new(); let domtree = DominatorTree::with_function(func, cfg); @@ -26,6 +26,7 @@ pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph) { while let Some(inst) = pos.next_inst() { let opcode = func.dfg[inst].opcode(); + let ctrl_typevar = func.dfg.ctrl_typevar(inst); // Resolve aliases, particularly aliases we created earlier. func.dfg.resolve_aliases_in_arguments(inst); @@ -42,7 +43,7 @@ pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph) { continue; } - let key = func.dfg[inst].clone(); + let key = (func.dfg[inst].clone(), ctrl_typevar); let entry = visible_values.entry(key); use std::collections::hash_map::Entry::*; match entry { From 01744d6f65b7e99be1671f72ab08dd4017a41641 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 30 Aug 2017 12:40:44 -0700 Subject: [PATCH 1000/3084] Avoid unneeded return keywords. --- lib/cretonne/src/bitset.rs | 2 +- lib/frontend/src/ssa.rs | 17 ++++++++--------- 2 files changed, 9 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/src/bitset.rs b/lib/cretonne/src/bitset.rs index cfca8371fb..58e26a933c 100644 --- a/lib/cretonne/src/bitset.rs +++ b/lib/cretonne/src/bitset.rs @@ -31,7 +31,7 @@ impl BitSet pub fn contains(&self, num: u8) -> bool { assert!((num as usize) < Self::bits()); assert!((num as usize) < Self::max_bits()); - return self.0.into() & (1 << num) != 0; + self.0.into() & (1 << num) != 0 } /// Return the smallest number contained in the bitset or None if empty diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 54485629d7..9e4969ebb6 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -252,26 +252,25 @@ impl SSABuilder UseVarCases::SealedOnePredecessor(pred) => { let (val, mids) = self.use_var(dfg, layout, jts, var, ty, pred); self.def_var(var, val, block); - return (val, mids); + (val, mids) } // The block has multiple predecessors, we register the ebb argument as the current // definition for the variable. UseVarCases::Unsealed(val) => { self.def_var(var, val, block); - return (val, - SideEffects { - split_ebbs_created: Vec::new(), - instructions_added_to_ebbs: Vec::new(), - }); + (val, + SideEffects { + split_ebbs_created: Vec::new(), + instructions_added_to_ebbs: Vec::new(), + }) } UseVarCases::SealedMultiplePredecessors(preds, val, ebb) => { // If multiple predecessor we look up a use_var in each of them: // if they all yield the same value no need for an Ebb argument self.def_var(var, val, block); - return self.predecessors_lookup(dfg, layout, jts, val, var, ebb, &preds); + self.predecessors_lookup(dfg, layout, jts, val, var, ebb, &preds) } - }; - + } } /// Declares a new basic block belonging to the body of a certain `Ebb` and having `pred` From 03698f6bc8560e77e3f78dad43286870e85fa667 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 10 Aug 2017 14:18:27 -0700 Subject: [PATCH 1001/3084] Use slices rather than Vec borrows. https://github.com/rust-lang-nursery/rust-clippy/wiki#ptr_arg --- lib/frontend/src/ssa.rs | 2 +- lib/wasm/src/code_translator.rs | 20 ++++++++++---------- lib/wasm/src/sections_translator.rs | 4 ++-- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 9e4969ebb6..d5d2b51bc4 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -433,7 +433,7 @@ impl SSABuilder temp_arg_val: Value, temp_arg_var: Variable, dest_ebb: Ebb, - preds: &Vec<(Block, Inst)>) + preds: &[(Block, Inst)]) -> (Value, SideEffects) { let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero(); // TODO: find a way not not allocate a vector diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index efd54617f4..d720ea34fc 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -156,10 +156,10 @@ impl FunctionImports { pub fn translate_function_body(parser: &mut Parser, function_index: FunctionIndex, sig: Signature, - locals: &Vec<(usize, Type)>, + locals: &[(usize, Type)], exports: &Option>, - signatures: &Vec, - functions: &Vec, + signatures: &[Signature], + functions: &[SignatureIndex], il_builder: &mut ILBuilder, runtime: &mut WasmRuntime) -> Result<(Function, FunctionImports), String> { @@ -288,8 +288,8 @@ fn translate_operator(op: &Operator, control_stack: &mut Vec, state: &mut TranslationState, sig: &Signature, - functions: &Vec, - signatures: &Vec, + functions: &[SignatureIndex], + signatures: &[Signature], exports: &Option>, func_imports: &mut FunctionImports) { // This big match treats all Wasm code operators. @@ -1290,8 +1290,8 @@ fn translate_unreachable_operator(op: &Operator, } fn args_count(index: FunctionIndex, - functions: &Vec, - signatures: &Vec) + functions: &[SignatureIndex], + signatures: &[Signature]) -> usize { signatures[functions[index] as usize].argument_types.len() } @@ -1301,9 +1301,9 @@ fn args_count(index: FunctionIndex, fn find_function_import(index: FunctionIndex, builder: &mut FunctionBuilder, func_imports: &mut FunctionImports, - functions: &Vec, + functions: &[SignatureIndex], exports: &Option>, - signatures: &Vec) + signatures: &[Signature]) -> FuncRef { match func_imports.functions.get(&index) { Some(local_index) => return *local_index, @@ -1358,7 +1358,7 @@ fn find_function_import(index: FunctionIndex, fn find_signature_import(sig_index: SignatureIndex, builder: &mut FunctionBuilder, func_imports: &mut FunctionImports, - signatures: &Vec) + signatures: &[Signature]) -> SigRef { match func_imports.signatures.get(&(sig_index as usize)) { Some(local_sig_index) => return *local_sig_index, diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index c9954e00c6..574c606063 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -224,7 +224,7 @@ pub fn parse_global_section(parser: &mut Parser, pub fn parse_data_section(parser: &mut Parser, runtime: &mut WasmRuntime, - globals: &Vec) + globals: &[Global]) -> Result<(), SectionParsingError> { loop { let memory_index = match *parser.read() { @@ -314,7 +314,7 @@ pub fn parse_table_section(parser: &mut Parser, /// Retrieves the tables from the table section pub fn parse_elements_section(parser: &mut Parser, runtime: &mut WasmRuntime, - globals: &Vec) + globals: &[Global]) -> Result<(), SectionParsingError> { loop { let table_index = match *parser.read() { From 8647b10135a54dbbc56137d159d4d268eb108c98 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 10 Aug 2017 13:59:46 -0700 Subject: [PATCH 1002/3084] Loop over references to containers instead of using explicit iteration methods. https://github.com/rust-lang-nursery/rust-clippy/wiki#explicit_iter_loop --- lib/frontend/src/frontend.rs | 2 +- lib/frontend/src/ssa.rs | 2 +- lib/wasm/src/code_translator.rs | 2 +- lib/wasm/src/translation_utils.rs | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 57c0aaa9e0..1b69acab00 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -492,7 +492,7 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> fn fill_function_args_values(&mut self, ebb: Ebb) { debug_assert!(self.pristine); - for argtyp in self.func.signature.argument_types.iter() { + for argtyp in &self.func.signature.argument_types { self.builder .function_args_values .push(self.func.dfg.append_ebb_arg(ebb, argtyp.value_type)); diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index d5d2b51bc4..310434e54d 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -402,7 +402,7 @@ impl SSABuilder }; // For each undef var we look up values in the predecessors and create an Ebb argument // only if necessary. - for &(var, val) in undef_vars.iter() { + for &(var, val) in &undef_vars { let (_, mut local_side_effects) = self.predecessors_lookup(dfg, layout, jts, val, var, ebb, &predecessors); side_effects diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index d720ea34fc..dcab3f3699 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -506,7 +506,7 @@ fn translate_operator(op: &Operator, Operator::BrTable { ref table } => { let (depths, default) = table.read_table(); let mut min_depth = default; - for depth in depths.iter() { + for depth in &depths { if *depth < min_depth { min_depth = *depth; } diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 609b4a3dfc..88686d4221 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -142,10 +142,10 @@ pub fn translate_type(ty: wasmparser::Type) -> Result, ( pub fn invert_hashmaps(imports: code_translator::FunctionImports) -> module_translator::ImportMappings { let mut new_imports = module_translator::ImportMappings::new(); - for (func_index, func_ref) in imports.functions.iter() { + for (func_index, func_ref) in &imports.functions { new_imports.functions.insert(*func_ref, *func_index); } - for (sig_index, sig_ref) in imports.signatures.iter() { + for (sig_index, sig_ref) in &imports.signatures { new_imports.signatures.insert(*sig_ref, *sig_index); } new_imports From 0cc8bd06cd1755d841d90db619d1cfb33cd3c854 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 30 Aug 2017 09:10:50 -0700 Subject: [PATCH 1003/3084] Remove unnecessary `()` values. --- lib/frontend/src/ssa.rs | 1 - lib/wasm/src/sections_translator.rs | 1 - 2 files changed, 2 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 310434e54d..9de49b95b5 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -72,7 +72,6 @@ impl BlockData { &mut BlockData::EbbBody { .. } => panic!("you can't add a predecessor to a body block"), &mut BlockData::EbbHeader(ref mut data) => { data.predecessors.insert(pred, inst); - () } } } diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 574c606063..fc110abf3c 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -139,7 +139,6 @@ pub fn parse_export_section(parser: &mut Parser) &ExternalKind::Function => { exports.insert(index as FunctionIndex, String::from(from_utf8(field).unwrap())); - () } _ => (),//TODO: deal with other kind of exports } From 5e99a4fb6a954e78703b44473072f96d6a9c4980 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 30 Aug 2017 11:12:05 -0700 Subject: [PATCH 1004/3084] Simplify a pattern-match. --- lib/frontend/src/ssa.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 9de49b95b5..3d20e9d86a 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -242,7 +242,7 @@ impl SSABuilder UseVarCases::Unsealed(val) } } - BlockData::EbbBody { predecessor: pred, .. } => UseVarCases::SealedOnePredecessor(pred), + BlockData::EbbBody { predecessor: pred } => UseVarCases::SealedOnePredecessor(pred), }; // TODO: avoid recursion for the calls to use_var and predecessors_lookup. match case { From 110697cfddbb2b8698ad45eb643d2a01ad30c10d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 30 Aug 2017 11:17:25 -0700 Subject: [PATCH 1005/3084] Replace match blocks with .expect calls. --- lib/frontend/src/ssa.rs | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 3d20e9d86a..f4b73c280a 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -298,15 +298,11 @@ impl SSABuilder /// Gets the header block corresponding to an Ebb, panics if the Ebb or the header block /// isn't declared. pub fn header_block(&self, ebb: Ebb) -> Block { - match self.ebb_headers.get(ebb) { - Some(&header) => { - match header.expand() { - Some(header) => header, - None => panic!("the header block has not been defined"), - } - } - None => panic!("the ebb has not been declared"), - } + self.ebb_headers + .get(ebb) + .expect("the ebb has not been declared") + .expand() + .expect("the header block has not been defined") } /// Declares a new predecessor for an `Ebb` header block and record the branch instruction From 4ccd21ba94148505b2d64e99675a7703b535f938 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 30 Aug 2017 11:21:02 -0700 Subject: [PATCH 1006/3084] Simplify iteration. --- lib/frontend/src/ssa.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index f4b73c280a..01baf27e2e 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -397,7 +397,7 @@ impl SSABuilder }; // For each undef var we look up values in the predecessors and create an Ebb argument // only if necessary. - for &(var, val) in &undef_vars { + for (var, val) in undef_vars { let (_, mut local_side_effects) = self.predecessors_lookup(dfg, layout, jts, val, var, ebb, &predecessors); side_effects From a0a3401ef1874ae680761e554604a37bf8a4e56d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 30 Aug 2017 09:21:58 -0700 Subject: [PATCH 1007/3084] Don't special-case br_table with an empty table. Empty br_table tables are very uncommon (they're easy for wasm producers to optimize away), so we don't need to special-case them. --- lib/wasm/src/code_translator.rs | 82 +++++++++++++++------------------ 1 file changed, 36 insertions(+), 46 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index dcab3f3699..708bdba6fa 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -523,17 +523,15 @@ fn translate_operator(op: &Operator, if jump_args_count == 0 { // No jump arguments let val = stack.pop().unwrap(); - if depths.len() > 0 { - let jt = builder.create_jump_table(); - for (index, depth) in depths.iter().enumerate() { - let i = control_stack.len() - 1 - (*depth as usize); - let frame = &mut control_stack[i]; - let ebb = frame.br_destination(); - builder.insert_jump_table_entry(jt, index, ebb); - frame.set_reachable(); - } - builder.ins().br_table(val, jt); + let jt = builder.create_jump_table(); + for (index, depth) in depths.iter().enumerate() { + let i = control_stack.len() - 1 - (*depth as usize); + let frame = &mut control_stack[i]; + let ebb = frame.br_destination(); + builder.insert_jump_table_entry(jt, index, ebb); + frame.set_reachable(); } + builder.ins().br_table(val, jt); let i = control_stack.len() - 1 - (default as usize); let frame = &mut control_stack[i]; let ebb = frame.br_destination(); @@ -546,44 +544,36 @@ fn translate_operator(op: &Operator, let val = stack.pop().unwrap(); let cut_index = stack.len() - jump_args_count; let jump_args = stack.split_off(cut_index); - if depths.len() > 0 { - let jt = builder.create_jump_table(); - let dest_ebbs: HashMap = depths - .iter() - .enumerate() - .fold(HashMap::new(), |mut acc, (index, &depth)| { - if acc.get(&(depth as usize)).is_none() { - let branch_ebb = builder.create_ebb(); - builder.insert_jump_table_entry(jt, index, branch_ebb); - acc.insert(depth as usize, branch_ebb); - return acc; - }; - let branch_ebb = acc.get(&(depth as usize)).unwrap().clone(); + let jt = builder.create_jump_table(); + let dest_ebbs: HashMap = depths + .iter() + .enumerate() + .fold(HashMap::new(), |mut acc, (index, &depth)| { + if acc.get(&(depth as usize)).is_none() { + let branch_ebb = builder.create_ebb(); builder.insert_jump_table_entry(jt, index, branch_ebb); - acc - }); - builder.ins().br_table(val, jt); - let default_ebb = control_stack[control_stack.len() - 1 - (default as usize)] - .br_destination(); - builder.ins().jump(default_ebb, jump_args.as_slice()); - stack.extend(jump_args.clone()); - for (depth, dest_ebb) in dest_ebbs { - builder.switch_to_block(dest_ebb, &[]); - builder.seal_block(dest_ebb); - let i = control_stack.len() - 1 - (depth as usize); - let frame = &mut control_stack[i]; - let real_dest_ebb = frame.br_destination(); - builder.ins().jump(real_dest_ebb, jump_args.as_slice()); - frame.set_reachable(); - } - state.real_unreachable_stack_depth = 1 + min_depth as usize; - } else { - let ebb = control_stack[control_stack.len() - 1 - (default as usize)] - .br_destination(); - builder.ins().jump(ebb, jump_args.as_slice()); - stack.extend(jump_args); - state.real_unreachable_stack_depth = 1 + min_depth as usize; + acc.insert(depth as usize, branch_ebb); + return acc; + }; + let branch_ebb = acc.get(&(depth as usize)).unwrap().clone(); + builder.insert_jump_table_entry(jt, index, branch_ebb); + acc + }); + builder.ins().br_table(val, jt); + let default_ebb = control_stack[control_stack.len() - 1 - (default as usize)] + .br_destination(); + builder.ins().jump(default_ebb, jump_args.as_slice()); + stack.extend(jump_args.clone()); + for (depth, dest_ebb) in dest_ebbs { + builder.switch_to_block(dest_ebb, &[]); + builder.seal_block(dest_ebb); + let i = control_stack.len() - 1 - (depth as usize); + let frame = &mut control_stack[i]; + let real_dest_ebb = frame.br_destination(); + builder.ins().jump(real_dest_ebb, jump_args.as_slice()); + frame.set_reachable(); } + state.real_unreachable_stack_depth = 1 + min_depth as usize; } } Operator::Return => { From 6702221e94d1f3176fad24edd4f5fe65f6754f75 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 08:54:34 -0700 Subject: [PATCH 1008/3084] Change EbbHeaderBlockData's predecessors list from a HashMap to a Vec. (#148) In addition to efficiency, this change also eliminates some nondeterminsm resulting from HashMap key ordering. --- lib/frontend/src/frontend.rs | 6 ++++++ lib/frontend/src/ssa.rs | 36 ++++++++++++++++-------------------- 2 files changed, 22 insertions(+), 20 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 1b69acab00..5535feebc6 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -7,6 +7,7 @@ use cretonne::ir::function::DisplayFunction; use cretonne::isa::TargetIsa; use ssa::{SSABuilder, SideEffects, Block}; use cretonne::entity::{EntityRef, EntityMap}; +use std::collections::HashSet; use std::hash::Hash; /// Permanent structure used for translating into Cretonne IL. @@ -146,6 +147,10 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short // If jump table we declare all entries successor // TODO: not collect with vector? InstructionData::BranchTable { table, .. } => { + // Unlike all other jumps/branches, jump tables are + // capable of having the same successor appear + // multiple times. Use a HashSet to deduplicate. + let mut unique = HashSet::new(); for dest_ebb in self.builder .func .jump_tables @@ -153,6 +158,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short .expect("you are referencing an undeclared jump table") .entries() .map(|(_, ebb)| ebb) + .filter(|dest_ebb| unique.insert(*dest_ebb)) .collect::>() { self.builder.declare_successor(dest_ebb, inst) } diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 01baf27e2e..0e4bcfcb9e 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -71,7 +71,7 @@ impl BlockData { match self { &mut BlockData::EbbBody { .. } => panic!("you can't add a predecessor to a body block"), &mut BlockData::EbbHeader(ref mut data) => { - data.predecessors.insert(pred, inst); + data.predecessors.push((pred, inst)); } } } @@ -81,14 +81,12 @@ impl BlockData { &mut BlockData::EbbHeader(ref mut data) => { // This a linear complexity operation but the number of predecessors is low // in all non-pathological cases - let pred: Block = match data.predecessors - .iter() - .find(|&(_, &jump_inst)| jump_inst == inst) { - None => panic!("the predecessor you are trying to remove is not declared"), - Some((&b, _)) => b.clone(), - }; - data.predecessors.remove(&pred); - pred + let pred: usize = + data.predecessors + .iter() + .position(|pair| pair.1 == inst) + .expect("the predecessor you are trying to remove is not declared"); + data.predecessors.swap_remove(pred).0 } } } @@ -96,7 +94,7 @@ impl BlockData { struct EbbHeaderBlockData { // The predecessors of the Ebb header block, with the block and branch instruction. - predecessors: HashMap, + predecessors: Vec<(Block, Inst)>, // A ebb header block is sealed if all of its predecessors have been declared. sealed: bool, // The ebb which this block is part of. @@ -227,13 +225,10 @@ impl SSABuilder if data.sealed { if data.predecessors.len() == 1 { // Only one predecessor, straightforward case - UseVarCases::SealedOnePredecessor(*data.predecessors.keys().next().unwrap()) + UseVarCases::SealedOnePredecessor(data.predecessors[0].0) } else { let val = dfg.append_ebb_arg(data.ebb, ty); - let preds = data.predecessors - .iter() - .map(|(&pred, &inst)| (pred, inst)) - .collect(); + let preds = data.predecessors.clone(); UseVarCases::SealedMultiplePredecessors(preds, val, data.ebb) } } else { @@ -287,7 +282,7 @@ impl SSABuilder pub fn declare_ebb_header_block(&mut self, ebb: Ebb) -> Block { let block = self.blocks .push(BlockData::EbbHeader(EbbHeaderBlockData { - predecessors: HashMap::new(), + predecessors: Vec::new(), sealed: false, ebb: ebb, undef_variables: Vec::new(), @@ -311,6 +306,9 @@ impl SSABuilder /// Note that the predecessor is a `Block` and not an `Ebb`. This `Block` must be filled /// before added as predecessor. Note that you must provide no jump arguments to the branch /// instruction when you create it since `SSABuilder` will fill them for you. + /// + /// Callers are expected to avoid adding the same predecessor more than once in the case + /// of a jump table. pub fn declare_ebb_predecessor(&mut self, ebb: Ebb, pred: Block, inst: Inst) { let header_block = match self.blocks[self.header_block(ebb)] { BlockData::EbbBody { .. } => panic!("you can't add predecessors to an Ebb body block"), @@ -385,9 +383,7 @@ impl SSABuilder Ebb) = match self.blocks[block] { BlockData::EbbBody { .. } => panic!("this should not happen"), BlockData::EbbHeader(ref mut data) => { - (data.predecessors.iter().map(|(&x, &y)| (x, y)).collect(), - data.undef_variables.clone(), - data.ebb) + (data.predecessors.clone(), data.undef_variables.clone(), data.ebb) } }; @@ -577,7 +573,7 @@ impl SSABuilder } /// Returns the list of `Ebb`s that have been declared as predecessors of the argument. - pub fn predecessors(&self, ebb: Ebb) -> &HashMap { + pub fn predecessors(&self, ebb: Ebb) -> &[(Block, Inst)] { let block = match self.ebb_headers[ebb].expand() { Some(block) => block, None => panic!("the ebb has not been declared yet"), From 4c3ac6053f4493a3f6306c67ae067bd195a0e41b Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 07:45:29 -0700 Subject: [PATCH 1009/3084] Test the error case of condcode parsing. --- lib/cretonne/src/ir/condcodes.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/cretonne/src/ir/condcodes.rs b/lib/cretonne/src/ir/condcodes.rs index f63bd60938..26d81ede6e 100644 --- a/lib/cretonne/src/ir/condcodes.rs +++ b/lib/cretonne/src/ir/condcodes.rs @@ -303,6 +303,7 @@ mod tests { let cc = *r; assert_eq!(cc.to_string().parse(), Ok(cc)); } + assert_eq!("bogus".parse::(), Err(())); } static FLOAT_ALL: [FloatCC; 14] = [FloatCC::Ordered, @@ -345,5 +346,6 @@ mod tests { let cc = *r; assert_eq!(cc.to_string().parse(), Ok(cc)); } + assert_eq!("bogus".parse::(), Err(())); } } From bd0590604d1da3c9bccbdcadd09f2b0b3997ac47 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 07:45:35 -0700 Subject: [PATCH 1010/3084] Test dfg's next_inst(). --- lib/cretonne/src/ir/dfg.rs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 219557aeb8..8a1447f858 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -866,7 +866,10 @@ mod tests { let mut dfg = DataFlowGraph::new(); let idata = InstructionData::Nullary { opcode: Opcode::Iconst }; + let next = dfg.next_inst(); let inst = dfg.make_inst(idata); + assert_eq!(next, inst); + dfg.make_inst_results(inst, types::I32); assert_eq!(inst.to_string(), "inst0"); assert_eq!(dfg.display_inst(inst, None).to_string(), "v0 = iconst.i32"); From fc374b6c068760f317696c8a72f2a893e7f4c516 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 07:45:39 -0700 Subject: [PATCH 1011/3084] Test that stack layout with an unsupported offset is rejected cleanly. --- lib/cretonne/src/stack_layout.rs | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/cretonne/src/stack_layout.rs b/lib/cretonne/src/stack_layout.rs index f968e56a07..c367849096 100644 --- a/lib/cretonne/src/stack_layout.rs +++ b/lib/cretonne/src/stack_layout.rs @@ -111,6 +111,8 @@ mod tests { use ir::StackSlots; use ir::types; use super::layout_stack; + use ir::stackslot::StackOffset; + use result::CtonError; #[test] fn layout() { @@ -181,5 +183,9 @@ mod tests { assert_eq!(sss[ss0].offset, -16); assert_eq!(sss[ss1].offset, -8); assert_eq!(sss[out0].offset, 0); + + // Also test that an unsupported offset is rejected. + sss.get_outgoing_arg(types::I8, StackOffset::max_value() - 1); + assert_eq!(layout_stack(sss, 1), Err(CtonError::ImplLimitExceeded)); } } From 52186b83908a20e481740bea3d1b69090449769d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 09:33:07 -0700 Subject: [PATCH 1012/3084] Refactor code for obtaining the header block. --- lib/frontend/src/ssa.rs | 23 +++++------------------ 1 file changed, 5 insertions(+), 18 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 0e4bcfcb9e..dcc2a092b8 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -310,13 +310,8 @@ impl SSABuilder /// Callers are expected to avoid adding the same predecessor more than once in the case /// of a jump table. pub fn declare_ebb_predecessor(&mut self, ebb: Ebb, pred: Block, inst: Inst) { - let header_block = match self.blocks[self.header_block(ebb)] { - BlockData::EbbBody { .. } => panic!("you can't add predecessors to an Ebb body block"), - BlockData::EbbHeader(ref data) => { - assert!(!data.sealed); - self.header_block(ebb) - } - }; + debug_assert!(!self.is_sealed(ebb)); + let header_block = self.header_block(ebb); self.blocks[header_block].add_predecessor(pred, inst) } @@ -325,13 +320,8 @@ impl SSABuilder /// /// Note: use only when you know what you are doing, this might break the SSA bbuilding problem pub fn remove_ebb_predecessor(&mut self, ebb: Ebb, inst: Inst) -> Block { - let header_block = match self.blocks[self.header_block(ebb)] { - BlockData::EbbBody { .. } => panic!("you can't add predecessors to an Ebb body block"), - BlockData::EbbHeader(ref data) => { - assert!(!data.sealed); - self.header_block(ebb) - } - }; + debug_assert!(!self.is_sealed(ebb)); + let header_block = self.header_block(ebb); self.blocks[header_block].remove_predecessor(inst) } @@ -574,10 +564,7 @@ impl SSABuilder /// Returns the list of `Ebb`s that have been declared as predecessors of the argument. pub fn predecessors(&self, ebb: Ebb) -> &[(Block, Inst)] { - let block = match self.ebb_headers[ebb].expand() { - Some(block) => block, - None => panic!("the ebb has not been declared yet"), - }; + let block = self.header_block(ebb); match self.blocks[block] { BlockData::EbbBody { .. } => panic!("should not happen"), BlockData::EbbHeader(ref data) => &data.predecessors, From 46fb64cbb442b7c3750a6f70755b6008c69e026a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 7 Aug 2017 13:59:26 -0700 Subject: [PATCH 1013/3084] Add SideEffects::new(). --- lib/frontend/src/ssa.rs | 31 +++++++++++++------------------ 1 file changed, 13 insertions(+), 18 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index dcc2a092b8..f25b3eaa7b 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -57,6 +57,15 @@ pub struct SideEffects { pub instructions_added_to_ebbs: Vec, } +impl SideEffects { + fn new() -> SideEffects { + SideEffects { + split_ebbs_created: Vec::new(), + instructions_added_to_ebbs: Vec::new(), + } + } +} + // Describes the current position of a basic block in the control flow graph. enum BlockData { // A block at the top of an `Ebb`. @@ -209,11 +218,7 @@ impl SSABuilder // First we lookup for the current definition of the variable in this block if let Some(var_defs) = self.variables.get(var) { if let Some(val) = var_defs.get(&block) { - return (*val, - SideEffects { - split_ebbs_created: Vec::new(), - instructions_added_to_ebbs: Vec::new(), - }); + return (*val, SideEffects::new()); } }; // At this point if we haven't returned it means that we have to search in the @@ -252,11 +257,7 @@ impl SSABuilder // definition for the variable. UseVarCases::Unsealed(val) => { self.def_var(var, val, block); - (val, - SideEffects { - split_ebbs_created: Vec::new(), - instructions_added_to_ebbs: Vec::new(), - }) + (val, SideEffects::new()) } UseVarCases::SealedMultiplePredecessors(preds, val, ebb) => { // If multiple predecessor we look up a use_var in each of them: @@ -377,10 +378,7 @@ impl SSABuilder } }; - let mut side_effects = SideEffects { - split_ebbs_created: Vec::new(), - instructions_added_to_ebbs: Vec::new(), - }; + let mut side_effects = SideEffects::new(); // For each undef var we look up values in the predecessors and create an Ebb argument // only if necessary. for (var, val) in undef_vars { @@ -420,10 +418,7 @@ impl SSABuilder // TODO: find a way not not allocate a vector let mut jump_args_to_append: Vec<(Block, Inst, Value)> = Vec::new(); let ty = dfg.value_type(temp_arg_val); - let mut side_effects = SideEffects { - split_ebbs_created: Vec::new(), - instructions_added_to_ebbs: Vec::new(), - }; + let mut side_effects = SideEffects::new(); for &(pred, last_inst) in preds.iter() { // For undef value and each predecessor we query what is the local SSA value // corresponding to var and we put it as an argument of the branch instruction. From 2efdc0ed37ed7ff0652fb2ea1c4b3180a8c20867 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 10:44:59 -0700 Subject: [PATCH 1014/3084] Update rustfmt to 0.9.0. --- check-rustfmt.sh | 2 +- cranelift/src/cat.rs | 10 +- cranelift/src/cton-util.rs | 18 +- cranelift/src/filetest/binemit.rs | 80 +- cranelift/src/filetest/compile.rs | 28 +- cranelift/src/filetest/concurrent.rs | 17 +- cranelift/src/filetest/domtree.rs | 42 +- cranelift/src/filetest/legalizer.rs | 6 +- cranelift/src/filetest/licm.rs | 11 +- cranelift/src/filetest/regalloc.rs | 12 +- cranelift/src/filetest/runner.rs | 54 +- cranelift/src/filetest/runone.rs | 28 +- cranelift/src/filetest/simple_gvn.rs | 11 +- cranelift/src/filetest/subtest.rs | 32 +- cranelift/src/filetest/verifier.rs | 8 +- cranelift/src/print_cfg.rs | 10 +- cranelift/src/rsfilecheck.rs | 36 +- cranelift/src/utils.rs | 15 +- cranelift/src/wasm.rs | 63 +- cranelift/tests/cfg_traversal.rs | 30 +- lib/cretonne/build.rs | 19 +- lib/cretonne/src/abi.rs | 42 +- lib/cretonne/src/binemit/mod.rs | 13 +- lib/cretonne/src/binemit/relaxation.rs | 38 +- lib/cretonne/src/bitset.rs | 46 +- lib/cretonne/src/btree.rs | 3 +- lib/cretonne/src/constant_hash.rs | 9 +- lib/cretonne/src/context.rs | 18 +- lib/cretonne/src/cursor.rs | 16 +- lib/cretonne/src/dbg.rs | 24 +- lib/cretonne/src/dominator_tree.rs | 80 +- lib/cretonne/src/entity/list.rs | 22 +- lib/cretonne/src/entity/map.rs | 23 +- lib/cretonne/src/entity/primary.rs | 12 +- lib/cretonne/src/entity/sparse.rs | 24 +- lib/cretonne/src/ir/builder.rs | 8 +- lib/cretonne/src/ir/condcodes.rs | 104 ++- lib/cretonne/src/ir/dfg.rs | 250 ++--- lib/cretonne/src/ir/entities.rs | 6 +- lib/cretonne/src/ir/extfunc.rs | 50 +- lib/cretonne/src/ir/funcname.rs | 27 +- lib/cretonne/src/ir/heap.rs | 6 +- lib/cretonne/src/ir/immediates.rs | 122 ++- lib/cretonne/src/ir/instructions.rs | 33 +- lib/cretonne/src/ir/jumptable.rs | 12 +- lib/cretonne/src/ir/layout.rs | 96 +- lib/cretonne/src/ir/progpoint.rs | 5 +- lib/cretonne/src/ir/stackslot.rs | 28 +- lib/cretonne/src/isa/arm32/abi.rs | 8 +- lib/cretonne/src/isa/arm32/mod.rs | 60 +- lib/cretonne/src/isa/arm64/abi.rs | 8 +- lib/cretonne/src/isa/arm64/mod.rs | 58 +- lib/cretonne/src/isa/enc_tables.rs | 69 +- lib/cretonne/src/isa/encoding.rs | 10 +- lib/cretonne/src/isa/intel/abi.rs | 11 +- lib/cretonne/src/isa/intel/mod.rs | 60 +- lib/cretonne/src/isa/mod.rs | 34 +- lib/cretonne/src/isa/registers.rs | 31 +- lib/cretonne/src/isa/riscv/abi.rs | 10 +- lib/cretonne/src/isa/riscv/binemit.rs | 18 +- lib/cretonne/src/isa/riscv/mod.rs | 60 +- lib/cretonne/src/isa/riscv/settings.rs | 8 +- lib/cretonne/src/iterators.rs | 99 +- lib/cretonne/src/legalizer/boundary.rs | 187 ++-- lib/cretonne/src/legalizer/heap.rs | 77 +- lib/cretonne/src/legalizer/mod.rs | 8 +- lib/cretonne/src/legalizer/split.rs | 112 ++- lib/cretonne/src/licm.rs | 58 +- lib/cretonne/src/loop_analysis.rs | 20 +- lib/cretonne/src/packed_option.rs | 12 +- lib/cretonne/src/partition_slice.rs | 3 +- lib/cretonne/src/regalloc/affinity.rs | 3 +- lib/cretonne/src/regalloc/allocatable_set.rs | 7 +- lib/cretonne/src/regalloc/coalescing.rs | 234 +++-- lib/cretonne/src/regalloc/coloring.rs | 313 ++++--- lib/cretonne/src/regalloc/context.rs | 69 +- lib/cretonne/src/regalloc/diversion.rs | 23 +- .../src/regalloc/live_value_tracker.rs | 97 +- lib/cretonne/src/regalloc/liveness.rs | 54 +- lib/cretonne/src/regalloc/liverange.rs | 76 +- lib/cretonne/src/regalloc/pressure.rs | 8 +- lib/cretonne/src/regalloc/reload.rs | 126 +-- lib/cretonne/src/regalloc/solver.rs | 186 ++-- lib/cretonne/src/regalloc/spilling.rs | 138 +-- lib/cretonne/src/regalloc/virtregs.rs | 17 +- lib/cretonne/src/settings.rs | 23 +- lib/cretonne/src/simple_gvn.rs | 2 +- lib/cretonne/src/stack_layout.rs | 18 +- lib/cretonne/src/topo_order.rs | 3 +- lib/cretonne/src/verifier/cssa.rs | 41 +- lib/cretonne/src/verifier/liveness.rs | 66 +- lib/cretonne/src/verifier/mod.rs | 331 ++++--- lib/cretonne/src/write.rs | 110 ++- lib/filecheck/src/checker.rs | 102 +- lib/filecheck/src/explain.rs | 92 +- lib/filecheck/src/pattern.rs | 273 ++++-- lib/filecheck/tests/basic.rs | 186 ++-- lib/frontend/src/frontend.rs | 251 ++--- lib/frontend/src/ssa.rs | 761 ++++++++------- lib/reader/src/isaspec.rs | 3 +- lib/reader/src/lexer.rs | 124 +-- lib/reader/src/parser.rs | 875 ++++++++++++------ lib/reader/src/sourcemap.rs | 67 +- lib/reader/src/testcommand.rs | 6 +- lib/wasm/src/code_translator.rs | 344 +++---- lib/wasm/src/module_translator.rs | 61 +- lib/wasm/src/runtime/dummy.rs | 48 +- lib/wasm/src/runtime/spec.rs | 62 +- lib/wasm/src/sections_translator.rs | 165 ++-- lib/wasm/src/translation_utils.rs | 17 +- lib/wasm/tests/testsuite.rs | 19 +- 111 files changed, 4692 insertions(+), 3379 deletions(-) diff --git a/check-rustfmt.sh b/check-rustfmt.sh index 6e8563900f..0439f35ff7 100755 --- a/check-rustfmt.sh +++ b/check-rustfmt.sh @@ -15,7 +15,7 @@ # With the --install option, also tries to install the right version. # This version should always be bumped to the newest version available. -VERS="0.8.4" +VERS="0.9.0" if cargo install --list | grep -q "^rustfmt v$VERS"; then exit 0 diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs index a53d3b4294..e8519b10d9 100644 --- a/cranelift/src/cat.rs +++ b/cranelift/src/cat.rs @@ -21,10 +21,12 @@ pub fn run(files: Vec) -> CommandResult { } fn cat_one(filename: String) -> CommandResult { - let buffer = read_to_string(&filename) - .map_err(|e| format!("{}: {}", filename, e))?; - let items = parse_functions(&buffer) - .map_err(|e| format!("{}: {}", filename, e))?; + let buffer = read_to_string(&filename).map_err( + |e| format!("{}: {}", filename, e), + )?; + let items = parse_functions(&buffer).map_err( + |e| format!("{}: {}", filename, e), + )?; for (idx, func) in items.into_iter().enumerate() { if idx != 0 { diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 5b7991bace..4e18001a0b 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -64,10 +64,10 @@ fn cton_util() -> CommandResult { // Parse command line arguments. let args: Args = Docopt::new(USAGE) .and_then(|d| { - d.help(true) - .version(Some(format!("Cretonne {}", VERSION))) - .deserialize() - }) + d.help(true) + .version(Some(format!("Cretonne {}", VERSION))) + .deserialize() + }) .unwrap_or_else(|e| e.exit()); // Find the sub-command to execute. @@ -80,10 +80,12 @@ fn cton_util() -> CommandResult { } else if args.cmd_print_cfg { print_cfg::run(args.arg_file) } else if args.cmd_wasm { - wasm::run(args.arg_file, - args.flag_verbose, - args.flag_optimize, - args.flag_check) + wasm::run( + args.arg_file, + args.flag_verbose, + args.flag_optimize, + args.flag_check, + ) } else { // Debugging / shouldn't happen with proper command line handling above. Err(format!("Unhandled args: {:?}", args)) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index af6e72f21e..6fd4ed1fcf 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -108,9 +108,12 @@ impl SubTest for TestBinEmit { for ebb in func.layout.ebbs() { for inst in func.layout.ebb_insts(ebb) { if !func.encodings[inst].is_legal() { - if let Ok(enc) = isa.encode(&func.dfg, - &func.dfg[inst], - func.dfg.ctrl_typevar(inst)) { + if let Ok(enc) = isa.encode( + &func.dfg, + &func.dfg[inst], + func.dfg.ctrl_typevar(inst), + ) + { func.encodings[inst] = enc; } } @@ -118,8 +121,9 @@ impl SubTest for TestBinEmit { } // Relax branches and compute EBB offsets based on the encodings. - let code_size = binemit::relax_branches(&mut func, isa) - .map_err(|e| pretty_error(&func, context.isa, e))?; + let code_size = binemit::relax_branches(&mut func, isa).map_err(|e| { + pretty_error(&func, context.isa, e) + })?; // Collect all of the 'bin:' directives on instructions. let mut bins = HashMap::new(); @@ -128,16 +132,20 @@ impl SubTest for TestBinEmit { match comment.entity { AnyEntity::Inst(inst) => { if let Some(prev) = bins.insert(inst, want) { - return Err(format!("multiple 'bin:' directives on {}: '{}' and '{}'", - func.dfg.display_inst(inst, isa), - prev, - want)); + return Err(format!( + "multiple 'bin:' directives on {}: '{}' and '{}'", + func.dfg.display_inst(inst, isa), + prev, + want + )); } } _ => { - return Err(format!("'bin:' directive on non-inst {}: {}", - comment.entity, - comment.text)) + return Err(format!( + "'bin:' directive on non-inst {}: {}", + comment.entity, + comment.text + )) } } } @@ -152,10 +160,12 @@ impl SubTest for TestBinEmit { for ebb in func.layout.ebbs() { divert.clear(); // Correct header offsets should have been computed by `relax_branches()`. - assert_eq!(sink.offset, - func.offsets[ebb], - "Inconsistent {} header offset", - ebb); + assert_eq!( + sink.offset, + func.offsets[ebb], + "Inconsistent {} header offset", + ebb + ); for inst in func.layout.ebb_insts(ebb) { sink.text.clear(); let enc = func.encodings[inst]; @@ -166,34 +176,44 @@ impl SubTest for TestBinEmit { isa.emit_inst(&func, inst, &mut divert, &mut sink); let emitted = sink.offset - before; // Verify the encoding recipe sizes against the ISAs emit_inst implementation. - assert_eq!(emitted, - encinfo.bytes(enc), - "Inconsistent size for [{}] {}", - encinfo.display(enc), - func.dfg.display_inst(inst, isa)); + assert_eq!( + emitted, + encinfo.bytes(enc), + "Inconsistent size for [{}] {}", + encinfo.display(enc), + func.dfg.display_inst(inst, isa) + ); } // Check against bin: directives. if let Some(want) = bins.remove(&inst) { if !enc.is_legal() { - return Err(format!("{} can't be encoded: {}", - inst, - func.dfg.display_inst(inst, isa))); + return Err(format!( + "{} can't be encoded: {}", + inst, + func.dfg.display_inst(inst, isa) + )); } let have = sink.text.trim(); if have != want { - return Err(format!("Bad machine code for {}: {}\nWant: {}\nGot: {}", - inst, - func.dfg.display_inst(inst, isa), - want, - have)); + return Err(format!( + "Bad machine code for {}: {}\nWant: {}\nGot: {}", + inst, + func.dfg.display_inst(inst, isa), + want, + have + )); } } } } if sink.offset != code_size { - return Err(format!("Expected code size {}, got {}", code_size, sink.offset)); + return Err(format!( + "Expected code size {}, got {}", + code_size, + sink.offset + )); } Ok(()) diff --git a/cranelift/src/filetest/compile.rs b/cranelift/src/filetest/compile.rs index 5d00ee63c2..799bcc0a0a 100644 --- a/cranelift/src/filetest/compile.rs +++ b/cranelift/src/filetest/compile.rs @@ -41,22 +41,30 @@ impl SubTest for TestCompile { let mut comp_ctx = cretonne::Context::new(); comp_ctx.func = func.into_owned(); - let code_size = comp_ctx - .compile(isa) - .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; + let code_size = comp_ctx.compile(isa).map_err(|e| { + pretty_error(&comp_ctx.func, context.isa, e) + })?; - dbg!("Generated {} bytes of code:\n{}", - code_size, - comp_ctx.func.display(isa)); + dbg!( + "Generated {} bytes of code:\n{}", + code_size, + comp_ctx.func.display(isa) + ); // Finally verify that the returned code size matches the emitted bytes. let mut sink = SizeSink { offset: 0 }; - binemit::emit_function(&comp_ctx.func, - |func, inst, div, sink| isa.emit_inst(func, inst, div, sink), - &mut sink); + binemit::emit_function( + &comp_ctx.func, + |func, inst, div, sink| isa.emit_inst(func, inst, div, sink), + &mut sink, + ); if sink.offset != code_size { - return Err(format!("Expected code size {}, got {}", code_size, sink.offset)); + return Err(format!( + "Expected code size {}, got {}", + code_size, + sink.offset + )); } Ok(()) diff --git a/cranelift/src/filetest/concurrent.rs b/cranelift/src/filetest/concurrent.rs index 7dcbe1b4fc..1d7de213f3 100644 --- a/cranelift/src/filetest/concurrent.rs +++ b/cranelift/src/filetest/concurrent.rs @@ -46,7 +46,9 @@ impl ConcurrentRunner { heartbeat_thread(reply_tx.clone()); let handles = (0..num_cpus::get()) - .map(|num| worker_thread(num, request_mutex.clone(), reply_tx.clone())) + .map(|num| { + worker_thread(num, request_mutex.clone(), reply_tx.clone()) + }) .collect(); ConcurrentRunner { @@ -97,16 +99,17 @@ fn heartbeat_thread(replies: Sender) -> thread::JoinHandle<()> { thread::Builder::new() .name("heartbeat".to_string()) .spawn(move || while replies.send(Reply::Tick).is_ok() { - thread::sleep(Duration::from_secs(1)); - }) + thread::sleep(Duration::from_secs(1)); + }) .unwrap() } /// Spawn a worker thread running tests. -fn worker_thread(thread_num: usize, - requests: Arc>>, - replies: Sender) - -> thread::JoinHandle<()> { +fn worker_thread( + thread_num: usize, + requests: Arc>>, + replies: Sender, +) -> thread::JoinHandle<()> { thread::Builder::new() .name(format!("worker #{}", thread_num)) .spawn(move || { diff --git a/cranelift/src/filetest/domtree.rs b/cranelift/src/filetest/domtree.rs index 29bde6d13f..46665ec428 100644 --- a/cranelift/src/filetest/domtree.rs +++ b/cranelift/src/filetest/domtree.rs @@ -50,9 +50,11 @@ impl SubTest for TestDomtree { let inst = match comment.entity { AnyEntity::Inst(inst) => inst, _ => { - return Err(format!("annotation on non-inst {}: {}", - comment.entity, - comment.text)) + return Err(format!( + "annotation on non-inst {}: {}", + comment.entity, + comment.text + )) } }; for src_ebb in tail.split_whitespace() { @@ -69,17 +71,21 @@ impl SubTest for TestDomtree { // Compare to computed domtree. match domtree.idom(ebb) { Some(got_inst) if got_inst != inst => { - return Err(format!("mismatching idoms for {}:\n\ + return Err(format!( + "mismatching idoms for {}:\n\ want: {}, got: {}", - src_ebb, - inst, - got_inst)); + src_ebb, + inst, + got_inst + )); } None => { - return Err(format!("mismatching idoms for {}:\n\ + return Err(format!( + "mismatching idoms for {}:\n\ want: {}, got: unreachable", - src_ebb, - inst)); + src_ebb, + inst + )); } _ => {} } @@ -89,15 +95,17 @@ impl SubTest for TestDomtree { // Now we know that everything in `expected` is consistent with `domtree`. // All other EBB's should be either unreachable or the entry block. - for ebb in func.layout - .ebbs() - .skip(1) - .filter(|ebb| !expected.contains_key(&ebb)) { + for ebb in func.layout.ebbs().skip(1).filter( + |ebb| !expected.contains_key(&ebb), + ) + { if let Some(got_inst) = domtree.idom(ebb) { - return Err(format!("mismatching idoms for renumbered {}:\n\ + return Err(format!( + "mismatching idoms for renumbered {}:\n\ want: unrechable, got: {}", - ebb, - got_inst)); + ebb, + got_inst + )); } } diff --git a/cranelift/src/filetest/legalizer.rs b/cranelift/src/filetest/legalizer.rs index 0c5ea04b77..08155289ce 100644 --- a/cranelift/src/filetest/legalizer.rs +++ b/cranelift/src/filetest/legalizer.rs @@ -41,9 +41,9 @@ impl SubTest for TestLegalizer { let isa = context.isa.expect("legalizer needs an ISA"); comp_ctx.flowgraph(); - comp_ctx - .legalize(isa) - .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; + comp_ctx.legalize(isa).map_err(|e| { + pretty_error(&comp_ctx.func, context.isa, e) + })?; let mut text = String::new(); write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))) diff --git a/cranelift/src/filetest/licm.rs b/cranelift/src/filetest/licm.rs index 548818e758..db817530a9 100644 --- a/cranelift/src/filetest/licm.rs +++ b/cranelift/src/filetest/licm.rs @@ -39,13 +39,14 @@ impl SubTest for TestLICM { comp_ctx.func = func.into_owned(); comp_ctx.flowgraph(); - comp_ctx - .licm() - .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; + comp_ctx.licm().map_err(|e| { + pretty_error(&comp_ctx.func, context.isa, e) + })?; let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func) - .map_err(|e| e.to_string())?; + write!(&mut text, "{}", &comp_ctx.func).map_err( + |e| e.to_string(), + )?; run_filecheck(&text, context) } } diff --git a/cranelift/src/filetest/regalloc.rs b/cranelift/src/filetest/regalloc.rs index e90012973b..3974854416 100644 --- a/cranelift/src/filetest/regalloc.rs +++ b/cranelift/src/filetest/regalloc.rs @@ -46,12 +46,12 @@ impl SubTest for TestRegalloc { comp_ctx.flowgraph(); // TODO: Should we have an option to skip legalization? - comp_ctx - .legalize(isa) - .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; - comp_ctx - .regalloc(isa) - .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; + comp_ctx.legalize(isa).map_err(|e| { + pretty_error(&comp_ctx.func, context.isa, e) + })?; + comp_ctx.regalloc(isa).map_err(|e| { + pretty_error(&comp_ctx.func, context.isa, e) + })?; let mut text = String::new(); write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))) diff --git a/cranelift/src/filetest/runner.rs b/cranelift/src/filetest/runner.rs index 9188ee00ad..6425482084 100644 --- a/cranelift/src/filetest/runner.rs +++ b/cranelift/src/filetest/runner.rs @@ -41,11 +41,13 @@ impl Display for QueueEntry { let p = self.path.to_string_lossy(); match self.state { State::Done(Ok(dur)) => { - write!(f, - "{}.{:03} {}", - dur.as_secs(), - dur.subsec_nanos() / 1000000, - p) + write!( + f, + "{}.{:03} {}", + dur.as_secs(), + dur.subsec_nanos() / 1000000, + p + ) } State::Done(Err(ref e)) => write!(f, "FAIL {}: {}", p, e), _ => write!(f, "{}", p), @@ -104,11 +106,10 @@ impl TestRunner { /// /// Any problems reading `file` as a test case file will be reported as a test failure. pub fn push_test>(&mut self, file: P) { - self.tests - .push(QueueEntry { - path: file.into(), - state: State::New, - }); + self.tests.push(QueueEntry { + path: file.into(), + state: State::New, + }); } /// Begin running tests concurrently. @@ -240,10 +241,12 @@ impl TestRunner { Reply::Tick => { self.ticks_since_progress += 1; if self.ticks_since_progress == TIMEOUT_SLOW { - println!("STALLED for {} seconds with {}/{} tests finished", - self.ticks_since_progress, - self.reported_tests, - self.tests.len()); + println!( + "STALLED for {} seconds with {}/{} tests finished", + self.ticks_since_progress, + self.reported_tests, + self.tests.len() + ); for jobid in self.reported_tests..self.tests.len() { if self.tests[jobid].state == State::Running { println!("slow: {}", self.tests[jobid]); @@ -251,8 +254,10 @@ impl TestRunner { } } if self.ticks_since_progress >= TIMEOUT_PANIC { - panic!("worker threads stalled for {} seconds.", - self.ticks_since_progress); + panic!( + "worker threads stalled for {} seconds.", + self.ticks_since_progress + ); } } } @@ -278,9 +283,9 @@ impl TestRunner { let mut times = self.tests .iter() .filter_map(|entry| match *entry { - QueueEntry { state: State::Done(Ok(dur)), .. } => Some(dur), - _ => None, - }) + QueueEntry { state: State::Done(Ok(dur)), .. } => Some(dur), + _ => None, + }) .collect::>(); // Get me some real data, kid. @@ -304,12 +309,11 @@ impl TestRunner { return; } - for t in self.tests - .iter() - .filter(|entry| match **entry { - QueueEntry { state: State::Done(Ok(dur)), .. } => dur > cut, - _ => false, - }) { + for t in self.tests.iter().filter(|entry| match **entry { + QueueEntry { state: State::Done(Ok(dur)), .. } => dur > cut, + _ => false, + }) + { println!("slow: {}", t) } diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index 3b23463fef..dad7b89e16 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -76,10 +76,11 @@ pub fn run(path: &Path) -> TestResult { } // Given a slice of tests, generate a vector of (test, flags, isa) tuples. -fn test_tuples<'a>(tests: &'a [Box], - isa_spec: &'a IsaSpec, - no_isa_flags: &'a Flags) - -> Result)>> { +fn test_tuples<'a>( + tests: &'a [Box], + isa_spec: &'a IsaSpec, + no_isa_flags: &'a Flags, +) -> Result)>> { let mut out = Vec::new(); for test in tests { if test.needs_isa() { @@ -104,10 +105,11 @@ fn test_tuples<'a>(tests: &'a [Box], Ok(out) } -fn run_one_test<'a>(tuple: (&'a SubTest, &'a Flags, Option<&'a TargetIsa>), - func: Cow, - context: &mut Context<'a>) - -> Result<()> { +fn run_one_test<'a>( + tuple: (&'a SubTest, &'a Flags, Option<&'a TargetIsa>), + func: Cow, + context: &mut Context<'a>, +) -> Result<()> { let (test, flags, isa) = tuple; let name = format!("{}({})", test.name(), func.name); dbg!("Test: {} {}", name, isa.map(TargetIsa::name).unwrap_or("-")); @@ -117,11 +119,13 @@ fn run_one_test<'a>(tuple: (&'a SubTest, &'a Flags, Option<&'a TargetIsa>), // Should we run the verifier before this test? if !context.verified && test.needs_verifier() { - verify_function(&func, isa) - .map_err(|e| pretty_verifier_error(&func, isa, e))?; + verify_function(&func, isa).map_err(|e| { + pretty_verifier_error(&func, isa, e) + })?; context.verified = true; } - test.run(func, context) - .map_err(|e| format!("{}: {}", name, e)) + test.run(func, context).map_err( + |e| format!("{}: {}", name, e), + ) } diff --git a/cranelift/src/filetest/simple_gvn.rs b/cranelift/src/filetest/simple_gvn.rs index a522c17bad..e8eb377b5e 100644 --- a/cranelift/src/filetest/simple_gvn.rs +++ b/cranelift/src/filetest/simple_gvn.rs @@ -39,13 +39,14 @@ impl SubTest for TestSimpleGVN { comp_ctx.func = func.into_owned(); comp_ctx.flowgraph(); - comp_ctx - .simple_gvn() - .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; + comp_ctx.simple_gvn().map_err(|e| { + pretty_error(&comp_ctx.func, context.isa, e) + })?; let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func) - .map_err(|e| e.to_string())?; + write!(&mut text, "{}", &comp_ctx.func).map_err( + |e| e.to_string(), + )?; run_filecheck(&text, context) } } diff --git a/cranelift/src/filetest/subtest.rs b/cranelift/src/filetest/subtest.rs index 9c7159dbb5..1129617db2 100644 --- a/cranelift/src/filetest/subtest.rs +++ b/cranelift/src/filetest/subtest.rs @@ -66,25 +66,25 @@ pub trait SubTest { /// match 'inst10'. impl<'a> filecheck::VariableMap for Context<'a> { fn lookup(&self, varname: &str) -> Option { - self.details - .map - .lookup_str(varname) - .map(|e| FCValue::Regex(format!(r"\b{}\b", e).into())) + self.details.map.lookup_str(varname).map(|e| { + FCValue::Regex(format!(r"\b{}\b", e).into()) + }) } } /// Run filecheck on `text`, using directives extracted from `context`. pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { let checker = build_filechecker(context)?; - if checker - .check(&text, context) - .map_err(|e| format!("filecheck: {}", e))? { + if checker.check(&text, context).map_err( + |e| format!("filecheck: {}", e), + )? + { Ok(()) } else { // Filecheck mismatch. Emit an explanation as output. - let (_, explain) = checker - .explain(&text, context) - .map_err(|e| format!("explain: {}", e))?; + let (_, explain) = checker.explain(&text, context).map_err( + |e| format!("explain: {}", e), + )?; Err(format!("filecheck failed:\n{}{}", checker, explain)) } } @@ -94,14 +94,14 @@ pub fn build_filechecker(context: &Context) -> Result { let mut builder = CheckerBuilder::new(); // Preamble comments apply to all functions. for comment in context.preamble_comments { - builder - .directive(comment.text) - .map_err(|e| format!("filecheck: {}", e))?; + builder.directive(comment.text).map_err(|e| { + format!("filecheck: {}", e) + })?; } for comment in &context.details.comments { - builder - .directive(comment.text) - .map_err(|e| format!("filecheck: {}", e))?; + builder.directive(comment.text).map_err(|e| { + format!("filecheck: {}", e) + })?; } let checker = builder.finish(); if checker.is_empty() { diff --git a/cranelift/src/filetest/verifier.rs b/cranelift/src/filetest/verifier.rs index 1dd5cc88b4..7fa94f7b5e 100644 --- a/cranelift/src/filetest/verifier.rs +++ b/cranelift/src/filetest/verifier.rs @@ -65,9 +65,11 @@ impl SubTest for TestVerifier { if want_loc == got.location { Ok(()) } else { - Err(format!("correct error reported on {}, but wanted {}", - got.location, - want_loc)) + Err(format!( + "correct error reported on {}, but wanted {}", + got.location, + want_loc + )) } } Some(_) => Err(format!("mismatching error: {}", got)), diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index ec479e3c92..bc3fc3dd9b 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -91,10 +91,12 @@ impl<'a> Display for CFGPrinter<'a> { } fn print_cfg(filename: String) -> CommandResult { - let buffer = read_to_string(&filename) - .map_err(|e| format!("{}: {}", filename, e))?; - let items = parse_functions(&buffer) - .map_err(|e| format!("{}: {}", filename, e))?; + let buffer = read_to_string(&filename).map_err( + |e| format!("{}: {}", filename, e), + )?; + let items = parse_functions(&buffer).map_err( + |e| format!("{}: {}", filename, e), + )?; for (idx, func) in items.into_iter().enumerate() { if idx != 0 { diff --git a/cranelift/src/rsfilecheck.rs b/cranelift/src/rsfilecheck.rs index 5ee464d20a..46605bb368 100644 --- a/cranelift/src/rsfilecheck.rs +++ b/cranelift/src/rsfilecheck.rs @@ -18,14 +18,14 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { } let mut buffer = String::new(); - io::stdin() - .read_to_string(&mut buffer) - .map_err(|e| format!("stdin: {}", e))?; + io::stdin().read_to_string(&mut buffer).map_err(|e| { + format!("stdin: {}", e) + })?; if verbose { - let (success, explain) = checker - .explain(&buffer, NO_VARIABLES) - .map_err(|e| e.to_string())?; + let (success, explain) = checker.explain(&buffer, NO_VARIABLES).map_err( + |e| e.to_string(), + )?; print!("{}", explain); if success { println!("OK"); @@ -33,25 +33,27 @@ pub fn run(files: Vec, verbose: bool) -> CommandResult { } else { Err("Check failed".to_string()) } - } else if checker - .check(&buffer, NO_VARIABLES) - .map_err(|e| e.to_string())? { + } else if checker.check(&buffer, NO_VARIABLES).map_err( + |e| e.to_string(), + )? + { Ok(()) } else { - let (_, explain) = checker - .explain(&buffer, NO_VARIABLES) - .map_err(|e| e.to_string())?; + let (_, explain) = checker.explain(&buffer, NO_VARIABLES).map_err( + |e| e.to_string(), + )?; print!("{}", explain); Err("Check failed".to_string()) } } fn read_checkfile(filename: &str) -> Result { - let buffer = read_to_string(&filename) - .map_err(|e| format!("{}: {}", filename, e))?; + let buffer = read_to_string(&filename).map_err( + |e| format!("{}: {}", filename, e), + )?; let mut builder = CheckerBuilder::new(); - builder - .text(&buffer) - .map_err(|e| format!("{}: {}", filename, e))?; + builder.text(&buffer).map_err( + |e| format!("{}: {}", filename, e), + )?; Ok(builder.finish()) } diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index 3bc8e7b1df..a5f9940eb4 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -24,8 +24,10 @@ pub fn read_to_string>(path: P) -> Result { /// /// Return the comment text following the directive. pub fn match_directive<'a>(comment: &'a str, directive: &str) -> Option<&'a str> { - assert!(directive.ends_with(':'), - "Directive must include trailing colon"); + assert!( + directive.ends_with(':'), + "Directive must include trailing colon" + ); let text = comment.trim_left_matches(';').trim_left(); if text.starts_with(directive) { Some(text[directive.len()..].trim()) @@ -35,10 +37,11 @@ pub fn match_directive<'a>(comment: &'a str, directive: &str) -> Option<&'a str> } /// Pretty-print a verifier error. -pub fn pretty_verifier_error(func: &ir::Function, - isa: Option<&TargetIsa>, - err: verifier::Error) - -> String { +pub fn pretty_verifier_error( + func: &ir::Function, + isa: Option<&TargetIsa>, + err: verifier::Error, +) -> String { let mut msg = err.to_string(); match err.location { AnyEntity::Inst(inst) => { diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 49292dbe29..7dc85b8aa4 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -51,19 +51,22 @@ fn read_wasm_file(path: PathBuf) -> Result, io::Error> { } -pub fn run(files: Vec, - flag_verbose: bool, - flag_optimize: bool, - flag_check: bool) - -> Result<(), String> { +pub fn run( + files: Vec, + flag_verbose: bool, + flag_optimize: bool, + flag_check: bool, +) -> Result<(), String> { for filename in files.iter() { let path = Path::new(&filename); let name = String::from(path.as_os_str().to_string_lossy()); - match handle_module(flag_verbose, - flag_optimize, - flag_check, - path.to_path_buf(), - name) { + match handle_module( + flag_verbose, + flag_optimize, + flag_check, + path.to_path_buf(), + name, + ) { Ok(()) => {} Err(message) => return Err(message), } @@ -71,12 +74,13 @@ pub fn run(files: Vec, Ok(()) } -fn handle_module(flag_verbose: bool, - flag_optimize: bool, - flag_check: bool, - path: PathBuf, - name: String) - -> Result<(), String> { +fn handle_module( + flag_verbose: bool, + flag_optimize: bool, + flag_check: bool, + path: PathBuf, + name: String, +) -> Result<(), String> { let mut terminal = term::stdout().unwrap(); terminal.fg(term::color::YELLOW).unwrap(); vprint!(flag_verbose, "Handling: "); @@ -109,10 +113,10 @@ fn handle_module(flag_verbose: bool, .arg(file_path.to_str().unwrap()) .output() .or_else(|e| if let io::ErrorKind::NotFound = e.kind() { - return Err(String::from("wast2wasm not found")); - } else { - return Err(String::from(e.description())); - }) + return Err(String::from("wast2wasm not found")); + } else { + return Err(String::from(e.description())); + }) .unwrap(); match read_wasm_file(file_path) { Ok(data) => data, @@ -221,17 +225,20 @@ fn handle_module(flag_verbose: bool, } /// Pretty-print a verifier error. -pub fn pretty_verifier_error(func: &ir::Function, - isa: Option<&TargetIsa>, - err: verifier::Error) - -> String { +pub fn pretty_verifier_error( + func: &ir::Function, + isa: Option<&TargetIsa>, + err: verifier::Error, +) -> String { let msg = err.to_string(); let str1 = match err.location { AnyEntity::Inst(inst) => { - format!("{}\n{}: {}\n\n", - msg, - inst, - func.dfg.display_inst(inst, isa)) + format!( + "{}\n{}: {}\n\n", + msg, + inst, + func.dfg.display_inst(inst, isa) + ) } _ => String::from(format!("{}\n", msg)), }; diff --git a/cranelift/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs index 37d568e0e4..9154349bc7 100644 --- a/cranelift/tests/cfg_traversal.rs +++ b/cranelift/tests/cfg_traversal.rs @@ -26,7 +26,8 @@ fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) #[test] fn simple_traversal() { - test_reverse_postorder_traversal(" + test_reverse_postorder_traversal( + " function %test(i32) native { ebb0(v0: i32): brz v0, ebb1 @@ -50,12 +51,14 @@ fn simple_traversal() { trap } ", - vec![0, 1, 3, 2, 4, 5]); + vec![0, 1, 3, 2, 4, 5], + ); } #[test] fn loops_one() { - test_reverse_postorder_traversal(" + test_reverse_postorder_traversal( + " function %test(i32) native { ebb0(v0: i32): jump ebb1 @@ -68,12 +71,14 @@ fn loops_one() { return } ", - vec![0, 1, 3, 2]); + vec![0, 1, 3, 2], + ); } #[test] fn loops_two() { - test_reverse_postorder_traversal(" + test_reverse_postorder_traversal( + " function %test(i32) native { ebb0(v0: i32): brz v0, ebb1 @@ -93,12 +98,14 @@ fn loops_two() { return } ", - vec![0, 1, 2, 4, 3, 5]); + vec![0, 1, 2, 4, 3, 5], + ); } #[test] fn loops_three() { - test_reverse_postorder_traversal(" + test_reverse_postorder_traversal( + " function %test(i32) native { ebb0(v0: i32): brz v0, ebb1 @@ -123,12 +130,14 @@ fn loops_three() { return } ", - vec![0, 1, 2, 4, 3, 6, 7, 5]); + vec![0, 1, 2, 4, 3, 6, 7, 5], + ); } #[test] fn back_edge_one() { - test_reverse_postorder_traversal(" + test_reverse_postorder_traversal( + " function %test(i32) native { ebb0(v0: i32): brz v0, ebb1 @@ -146,5 +155,6 @@ fn back_edge_one() { trap } ", - vec![0, 1, 3, 2, 4]); + vec![0, 1, 3, 2, 4], + ); } diff --git a/lib/cretonne/build.rs b/lib/cretonne/build.rs index 2653bf54a0..4d1b0edf72 100644 --- a/lib/cretonne/build.rs +++ b/lib/cretonne/build.rs @@ -49,8 +49,10 @@ fn main() { // Make sure we rebuild is this build script changes. // I guess that won't happen if you have non-UTF8 bytes in your path names. // The `build.py` script prints out its own dependencies. - println!("cargo:rerun-if-changed={}", - crate_dir.join("build.rs").to_string_lossy()); + println!( + "cargo:rerun-if-changed={}", + crate_dir.join("build.rs").to_string_lossy() + ); // Scripts are in `$crate_dir/meta`. let meta_dir = crate_dir.join("meta"); @@ -130,9 +132,11 @@ fn isa_targets(cretonne_targets: Option<&str>, target_triple: &str) -> Result { let unknown_isa_targets = targets @@ -143,7 +147,10 @@ fn isa_targets(cretonne_targets: Option<&str>, target_triple: &str) -> Result Ok(Isa::all().to_vec()), (true, _) => Ok(isa_targets), - (_, _) => Err(format!("unknown isa targets: `{}`", unknown_isa_targets.join(", "))), + (_, _) => Err(format!( + "unknown isa targets: `{}`", + unknown_isa_targets.join(", ") + )), } } None => Ok(Isa::all().to_vec()), diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index 32210fecfb..49620bcfde 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -150,8 +150,10 @@ pub fn legalize_abi_value(have: Type, arg: &ArgumentType) -> ValueConversion { match have_bits.cmp(&arg_bits) { // We have fewer bits than the ABI argument. Ordering::Less => { - assert!(have.is_int() && arg.value_type.is_int(), - "Can only extend integer values"); + assert!( + have.is_int() && arg.value_type.is_int(), + "Can only extend integer values" + ); match arg.extension { ArgumentExtension::Uext => ValueConversion::Uext(arg.value_type), ArgumentExtension::Sext => ValueConversion::Sext(arg.value_type), @@ -192,22 +194,34 @@ mod tests { fn legalize() { let mut arg = ArgumentType::new(types::I32); - assert_eq!(legalize_abi_value(types::I64X2, &arg), - ValueConversion::VectorSplit); - assert_eq!(legalize_abi_value(types::I64, &arg), - ValueConversion::IntSplit); + assert_eq!( + legalize_abi_value(types::I64X2, &arg), + ValueConversion::VectorSplit + ); + assert_eq!( + legalize_abi_value(types::I64, &arg), + ValueConversion::IntSplit + ); // Vector of integers is broken down, then sign-extended. arg.extension = ArgumentExtension::Sext; - assert_eq!(legalize_abi_value(types::I16X4, &arg), - ValueConversion::VectorSplit); - assert_eq!(legalize_abi_value(types::I16.by(2).unwrap(), &arg), - ValueConversion::VectorSplit); - assert_eq!(legalize_abi_value(types::I16, &arg), - ValueConversion::Sext(types::I32)); + assert_eq!( + legalize_abi_value(types::I16X4, &arg), + ValueConversion::VectorSplit + ); + assert_eq!( + legalize_abi_value(types::I16.by(2).unwrap(), &arg), + ValueConversion::VectorSplit + ); + assert_eq!( + legalize_abi_value(types::I16, &arg), + ValueConversion::Sext(types::I32) + ); // 64-bit float is split as an integer. - assert_eq!(legalize_abi_value(types::F64, &arg), - ValueConversion::IntBits); + assert_eq!( + legalize_abi_value(types::F64, &arg), + ValueConversion::IntBits + ); } } diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index a4a62f869f..5715b1a86d 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -54,9 +54,11 @@ pub trait CodeSink { /// Report a bad encoding error. #[inline(never)] pub fn bad_encoding(func: &Function, inst: Inst) -> ! { - panic!("Bad encoding {} for {}", - func.encodings[inst], - func.dfg.display_inst(inst, None)); + panic!( + "Bad encoding {} for {}", + func.encodings[inst], + func.dfg.display_inst(inst, None) + ); } /// Emit a function to `sink`, given an instruction emitter function. @@ -64,8 +66,9 @@ pub fn bad_encoding(func: &Function, inst: Inst) -> ! { /// This function is called from the `TargetIsa::emit_function()` implementations with the /// appropriate instruction emitter. pub fn emit_function(func: &Function, emit_inst: EI, sink: &mut CS) - where CS: CodeSink, - EI: Fn(&Function, Inst, &mut RegDiversions, &mut CS) +where + CS: CodeSink, + EI: Fn(&Function, Inst, &mut RegDiversions, &mut CS), { let mut divert = RegDiversions::new(); for ebb in func.layout.ebbs() { diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index 51d288703c..6ae9b4e4f7 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -60,8 +60,10 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> Result { // Somebody used a fall-through instruction before the branch relaxation pass. @@ -126,16 +129,19 @@ fn fallthroughs(func: &mut Function) { /// /// Return the size of the replacement instructions up to and including the location where `pos` is /// left. -fn relax_branch(cur: &mut FuncCursor, - offset: CodeOffset, - dest_offset: CodeOffset, - encinfo: &EncInfo) - -> CodeOffset { +fn relax_branch( + cur: &mut FuncCursor, + offset: CodeOffset, + dest_offset: CodeOffset, + encinfo: &EncInfo, +) -> CodeOffset { let inst = cur.current_inst().unwrap(); - dbg!("Relaxing [{}] {} for {:#x}-{:#x} range", - encinfo.display(cur.func.encodings[inst]), - cur.func.dfg.display_inst(inst, None), - offset, - dest_offset); + dbg!( + "Relaxing [{}] {} for {:#x}-{:#x} range", + encinfo.display(cur.func.encodings[inst]), + cur.func.dfg.display_inst(inst, None), + offset, + dest_offset + ); unimplemented!(); } diff --git a/lib/cretonne/src/bitset.rs b/lib/cretonne/src/bitset.rs index 58e26a933c..a9c41ce78a 100644 --- a/lib/cretonne/src/bitset.rs +++ b/lib/cretonne/src/bitset.rs @@ -14,27 +14,34 @@ use std::convert::{Into, From}; pub struct BitSet(pub T); impl BitSet - where T: Into + From + BitOr + Shl + Sub + - Add + PartialEq + Copy +where + T: Into + + From + + BitOr + + Shl + + Sub + + Add + + PartialEq + + Copy, { -/// Maximum number of bits supported by this BitSet instance + /// Maximum number of bits supported by this BitSet instance pub fn bits() -> usize { size_of::() * 8 } -/// Maximum number of bits supported by any bitset instance atm. + /// Maximum number of bits supported by any bitset instance atm. pub fn max_bits() -> usize { size_of::() * 8 } -/// Check if this BitSet contains the number num + /// Check if this BitSet contains the number num pub fn contains(&self, num: u8) -> bool { assert!((num as usize) < Self::bits()); assert!((num as usize) < Self::max_bits()); self.0.into() & (1 << num) != 0 } -/// Return the smallest number contained in the bitset or None if empty + /// Return the smallest number contained in the bitset or None if empty pub fn min(&self) -> Option { if self.0.into() == 0 { None @@ -43,7 +50,7 @@ impl BitSet } } -/// Return the largest number contained in the bitset or None if empty + /// Return the largest number contained in the bitset or None if empty pub fn max(&self) -> Option { if self.0.into() == 0 { None @@ -53,17 +60,17 @@ impl BitSet } } -/// Construct a BitSet with the half-open range [lo,hi) filled in + /// Construct a BitSet with the half-open range [lo,hi) filled in pub fn from_range(lo: u8, hi: u8) -> BitSet { assert!(lo <= hi); assert!((hi as usize) <= Self::bits()); - let one : T = T::from(1); -// I can't just do (one << hi) - one here as the shift may overflow + let one: T = T::from(1); + // I can't just do (one << hi) - one here as the shift may overflow let hi_rng = if hi >= 1 { - (one << (hi-1)) + ((one << (hi-1)) - one) - } else { - T::from(0) - }; + (one << (hi - 1)) + ((one << (hi - 1)) - one) + } else { + T::from(0) + }; let lo_rng = (one << lo) - one; @@ -94,14 +101,15 @@ mod tests { assert!(!s2.contains(7)); let s3 = BitSet::(2 | 4 | 64); - assert!(!s3.contains(0) && !s3.contains(3) && !s3.contains(4) && !s3.contains(5) && - !s3.contains(7)); + assert!(!s3.contains(0) && !s3.contains(3) && !s3.contains(4)); + assert!(!s3.contains(5) && !s3.contains(7)); assert!(s3.contains(1) && s3.contains(2) && s3.contains(6)); let s4 = BitSet::(4 | 8 | 256 | 1024); - assert!(!s4.contains(0) && !s4.contains(1) && !s4.contains(4) && !s4.contains(5) && - !s4.contains(6) && !s4.contains(7) && - !s4.contains(9) && !s4.contains(11)); + assert!( + !s4.contains(0) && !s4.contains(1) && !s4.contains(4) && !s4.contains(5) && + !s4.contains(6) && !s4.contains(7) && !s4.contains(9) && !s4.contains(11) + ); assert!(s4.contains(2) && s4.contains(3) && s4.contains(8) && s4.contains(10)); } diff --git a/lib/cretonne/src/btree.rs b/lib/cretonne/src/btree.rs index b6732788f7..e625b6108e 100644 --- a/lib/cretonne/src/btree.rs +++ b/lib/cretonne/src/btree.rs @@ -63,7 +63,8 @@ impl NodePool { } impl BTree { - /// Search for `key` and return a `Cursor` that either points at `key` or the position where it would be inserted. + /// Search for `key` and return a `Cursor` that either points at `key` or the position + /// where it would be inserted. pub fn search(&mut self, key: K) -> Cursor { unimplemented!() } diff --git a/lib/cretonne/src/constant_hash.rs b/lib/cretonne/src/constant_hash.rs index a165654dce..d049d8e752 100644 --- a/lib/cretonne/src/constant_hash.rs +++ b/lib/cretonne/src/constant_hash.rs @@ -26,10 +26,11 @@ pub trait Table { /// /// Returns `Ok(idx)` with the table index containing the found entry, or `Err(idx)` with the empty /// sentinel entry if no entry could be found. -pub fn probe + ?Sized>(table: &T, - key: K, - hash: usize) - -> Result { +pub fn probe + ?Sized>( + table: &T, + key: K, + hash: usize, +) -> Result { debug_assert!(table.len().is_power_of_two()); let mask = table.len() - 1; diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 6f80f16056..ece812a5de 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -133,18 +133,24 @@ impl Context { /// Perform LICM on the function. pub fn licm(&mut self) -> CtonResult { self.ensure_domtree(); - do_licm(&mut self.func, - &mut self.cfg, - &mut self.domtree, - &mut self.loop_analysis); + do_licm( + &mut self.func, + &mut self.cfg, + &mut self.domtree, + &mut self.loop_analysis, + ); self.verify(None).map_err(Into::into) } /// Run the register allocator. pub fn regalloc(&mut self, isa: &TargetIsa) -> CtonResult { self.ensure_domtree(); - self.regalloc - .run(isa, &mut self.func, &self.cfg, &self.domtree) + self.regalloc.run( + isa, + &mut self.func, + &self.cfg, + &self.domtree, + ) } /// Insert prologue and epilogues after computing the stack frame layout. diff --git a/lib/cretonne/src/cursor.rs b/lib/cretonne/src/cursor.rs index 66e87ccb9f..8de400a1f1 100644 --- a/lib/cretonne/src/cursor.rs +++ b/lib/cretonne/src/cursor.rs @@ -146,17 +146,21 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut EncCursor<'f> { &mut self.func.dfg } - fn insert_built_inst(self, - inst: ir::Inst, - ctrl_typevar: ir::Type) - -> &'c mut ir::DataFlowGraph { + fn insert_built_inst( + self, + inst: ir::Inst, + ctrl_typevar: ir::Type, + ) -> &'c mut ir::DataFlowGraph { // Insert the instruction and remember the reference. self.insert_inst(inst); self.built_inst = Some(inst); // Assign an encoding. - match self.isa - .encode(&self.func.dfg, &self.func.dfg[inst], ctrl_typevar) { + match self.isa.encode( + &self.func.dfg, + &self.func.dfg[inst], + ctrl_typevar, + ) { Ok(e) => self.func.encodings[inst] = e, Err(_) => panic!("can't encode {}", self.display_inst(inst)), } diff --git a/lib/cretonne/src/dbg.rs b/lib/cretonne/src/dbg.rs index bf1b92ef75..0ac03f4c4f 100644 --- a/lib/cretonne/src/dbg.rs +++ b/lib/cretonne/src/dbg.rs @@ -69,18 +69,17 @@ pub fn writeln_with_format_args(args: fmt::Arguments) -> io::Result<()> { /// Open the tracing file for the current thread. fn open_file() -> io::BufWriter { let file = match thread::current().name() { - None => File::create("cretonne.dbg"), - Some(name) => { - let mut path = "cretonne.dbg.".to_owned(); - for ch in name.chars() { - if ch.is_ascii() && ch.is_alphanumeric() { - path.push(ch); - } + None => File::create("cretonne.dbg"), + Some(name) => { + let mut path = "cretonne.dbg.".to_owned(); + for ch in name.chars() { + if ch.is_ascii() && ch.is_alphanumeric() { + path.push(ch); } - File::create(path) } + File::create(path) } - .expect("Can't open tracing file"); + }.expect("Can't open tracing file"); io::BufWriter::new(file) } @@ -99,10 +98,13 @@ macro_rules! dbg { } /// Helper for printing lists. -pub struct DisplayList<'a, T>(pub &'a [T]) where T: 'a + fmt::Display; +pub struct DisplayList<'a, T>(pub &'a [T]) +where + T: 'a + fmt::Display; impl<'a, T> fmt::Display for DisplayList<'a, T> - where T: 'a + fmt::Display +where + T: 'a + fmt::Display, { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match self.0.split_first() { diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 067142b0f2..2e9a6dc38d 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -85,13 +85,15 @@ impl DominatorTree { /// /// If `a` and `b` belong to the same EBB, compare their relative position in the EBB. pub fn rpo_cmp(&self, a: A, b: B, layout: &Layout) -> Ordering - where A: Into, - B: Into + where + A: Into, + B: Into, { let a = a.into(); let b = b.into(); - self.rpo_cmp_ebb(layout.pp_ebb(a), layout.pp_ebb(b)) - .then(layout.cmp(a, b)) + self.rpo_cmp_ebb(layout.pp_ebb(a), layout.pp_ebb(b)).then( + layout.cmp(a, b), + ) } /// Returns `true` if `a` dominates `b`. @@ -104,8 +106,9 @@ impl DominatorTree { /// /// An instruction is considered to dominate itself. pub fn dominates(&self, a: A, b: B, layout: &Layout) -> bool - where A: Into, - B: Into + where + A: Into, + B: Into, { let a = a.into(); let b = b.into(); @@ -126,12 +129,16 @@ impl DominatorTree { /// Find the last instruction in `a` that dominates `b`. /// If no instructions in `a` dominate `b`, return `None`. fn last_dominator(&self, a: Ebb, b: B, layout: &Layout) -> Option - where B: Into + where + B: Into, { let (mut ebb_b, mut inst_b) = match b.into() { ExpandedProgramPoint::Ebb(ebb) => (ebb, None), ExpandedProgramPoint::Inst(inst) => { - (layout.inst_ebb(inst).expect("Instruction not in layout."), Some(inst)) + ( + layout.inst_ebb(inst).expect("Instruction not in layout."), + Some(inst), + ) } }; let rpo_a = self.nodes[a].rpo_number; @@ -149,22 +156,29 @@ impl DominatorTree { /// Compute the common dominator of two basic blocks. /// /// Both basic blocks are assumed to be reachable. - pub fn common_dominator(&self, - mut a: BasicBlock, - mut b: BasicBlock, - layout: &Layout) - -> BasicBlock { + pub fn common_dominator( + &self, + mut a: BasicBlock, + mut b: BasicBlock, + layout: &Layout, + ) -> BasicBlock { loop { match self.rpo_cmp_ebb(a.0, b.0) { Ordering::Less => { // `a` comes before `b` in the RPO. Move `b` up. let idom = self.nodes[b.0].idom.expect("Unreachable basic block?"); - b = (layout.inst_ebb(idom).expect("Dangling idom instruction"), idom); + b = ( + layout.inst_ebb(idom).expect("Dangling idom instruction"), + idom, + ); } Ordering::Greater => { // `b` comes before `a` in the RPO. Move `a` up. let idom = self.nodes[a.0].idom.expect("Unreachable basic block?"); - a = (layout.inst_ebb(idom).expect("Dangling idom instruction"), idom); + a = ( + layout.inst_ebb(idom).expect("Dangling idom instruction"), + idom, + ); } Ordering::Equal => break, } @@ -327,15 +341,16 @@ impl DominatorTree { // Get an iterator with just the reachable, already visited predecessors to `ebb`. // Note that during the first pass, `rpo_number` is 1 for reachable blocks that haven't // been visited yet, 0 for unreachable blocks. - let mut reachable_preds = cfg.get_predecessors(ebb) - .iter() - .cloned() - .filter(|&(pred, _)| self.nodes[pred].rpo_number > 1); + let mut reachable_preds = cfg.get_predecessors(ebb).iter().cloned().filter( + |&(pred, _)| { + self.nodes[pred].rpo_number > 1 + }, + ); // The RPO must visit at least one predecessor before this node. - let mut idom = reachable_preds - .next() - .expect("EBB node must have one reachable predecessor"); + let mut idom = reachable_preds.next().expect( + "EBB node must have one reachable predecessor", + ); for pred in reachable_preds { idom = self.common_dominator(idom, pred, layout); @@ -383,10 +398,11 @@ impl DominatorTree { // forward in RPO numbers and backwards in the postorder list of EBBs, renumbering the Ebbs // until we find a gap for (¤t_ebb, current_rpo) in - self.postorder[0..ebb_postorder_index] - .iter() - .rev() - .zip(inserted_rpo_number + 1..) { + self.postorder[0..ebb_postorder_index].iter().rev().zip( + inserted_rpo_number + + 1.., + ) + { if self.nodes[current_ebb].rpo_number < current_rpo { // There is no gap, we renumber self.nodes[current_ebb].rpo_number = current_rpo; @@ -457,10 +473,14 @@ mod test { assert_eq!(dt.rpo_cmp(ebb3, ebb3, &cur.func.layout), Ordering::Equal); assert_eq!(dt.rpo_cmp(ebb3, ebb1, &cur.func.layout), Ordering::Less); - assert_eq!(dt.rpo_cmp(ebb3, jmp_ebb3_ebb1, &cur.func.layout), - Ordering::Less); - assert_eq!(dt.rpo_cmp(jmp_ebb3_ebb1, jmp_ebb1_ebb2, &cur.func.layout), - Ordering::Less); + assert_eq!( + dt.rpo_cmp(ebb3, jmp_ebb3_ebb1, &cur.func.layout), + Ordering::Less + ); + assert_eq!( + dt.rpo_cmp(jmp_ebb3_ebb1, jmp_ebb1_ebb2, &cur.func.layout), + Ordering::Less + ); assert_eq!(dt.cfg_postorder(), &[ebb2, ebb0, ebb1, ebb3]); } diff --git a/lib/cretonne/src/entity/list.rs b/lib/cretonne/src/entity/list.rs index e70909bf60..3f5eace692 100644 --- a/lib/cretonne/src/entity/list.rs +++ b/lib/cretonne/src/entity/list.rs @@ -212,12 +212,13 @@ impl ListPool { /// Reallocate a block to a different size class. /// /// Copy `elems_to_copy` elements from the old to the new block. - fn realloc(&mut self, - block: usize, - from_sclass: SizeClass, - to_sclass: SizeClass, - elems_to_copy: usize) - -> usize { + fn realloc( + &mut self, + block: usize, + from_sclass: SizeClass, + to_sclass: SizeClass, + elems_to_copy: usize, + ) -> usize { assert!(elems_to_copy <= sclass_size(from_sclass)); assert!(elems_to_copy <= sclass_size(to_sclass)); let new_block = self.alloc(to_sclass); @@ -384,7 +385,8 @@ impl EntityList { /// Appends multiple elements to the back of the list. pub fn extend(&mut self, elements: I, pool: &mut ListPool) - where I: IntoIterator + where + I: IntoIterator, { // TODO: use `size_hint()` to reduce reallocations. for x in elements { @@ -597,8 +599,10 @@ mod tests { list.extend([i1, i1, i2, i2, i3, i3, i4, i4].iter().cloned(), pool); assert_eq!(list.len(pool), 12); - assert_eq!(list.as_slice(pool), - &[i1, i2, i3, i4, i1, i1, i2, i2, i3, i3, i4, i4]); + assert_eq!( + list.as_slice(pool), + &[i1, i2, i3, i4, i1, i1, i2, i2, i3, i3, i4, i4] + ); } #[test] diff --git a/lib/cretonne/src/entity/map.rs b/lib/cretonne/src/entity/map.rs index 9afc5ff356..5db8522382 100644 --- a/lib/cretonne/src/entity/map.rs +++ b/lib/cretonne/src/entity/map.rs @@ -14,8 +14,9 @@ use std::ops::{Index, IndexMut}; /// all keys have a default entry from the beginning. #[derive(Debug, Clone)] pub struct EntityMap - where K: EntityRef, - V: Clone +where + K: EntityRef, + V: Clone, { elems: Vec, default: V, @@ -24,12 +25,14 @@ pub struct EntityMap /// Shared `EntityMap` implementation for all value types. impl EntityMap - where K: EntityRef, - V: Clone +where + K: EntityRef, + V: Clone, { /// Create a new empty map. pub fn new() -> Self - where V: Default + where + V: Default, { EntityMap { elems: Vec::new(), @@ -68,8 +71,9 @@ impl EntityMap /// /// All keys are permitted. Untouched entries have the default value. impl Index for EntityMap - where K: EntityRef, - V: Clone +where + K: EntityRef, + V: Clone, { type Output = V; @@ -82,8 +86,9 @@ impl Index for EntityMap /// /// The map grows as needed to accommodate new keys. impl IndexMut for EntityMap - where K: EntityRef, - V: Clone +where + K: EntityRef, + V: Clone, { fn index_mut(&mut self, k: K) -> &mut V { let i = k.index(); diff --git a/lib/cretonne/src/entity/primary.rs b/lib/cretonne/src/entity/primary.rs index 39af48ff3f..c0347304aa 100644 --- a/lib/cretonne/src/entity/primary.rs +++ b/lib/cretonne/src/entity/primary.rs @@ -14,14 +14,16 @@ use std::ops::{Index, IndexMut}; /// conflicting references will be created. Using unknown keys for indexing will cause a panic. #[derive(Debug, Clone)] pub struct PrimaryMap - where K: EntityRef +where + K: EntityRef, { elems: Vec, unused: PhantomData, } impl PrimaryMap - where K: EntityRef +where + K: EntityRef, { /// Create a new empty map. pub fn new() -> Self { @@ -77,7 +79,8 @@ impl PrimaryMap /// Immutable indexing into an `PrimaryMap`. /// The indexed value must be in the map. impl Index for PrimaryMap - where K: EntityRef +where + K: EntityRef, { type Output = V; @@ -88,7 +91,8 @@ impl Index for PrimaryMap /// Mutable indexing into an `PrimaryMap`. impl IndexMut for PrimaryMap - where K: EntityRef +where + K: EntityRef, { fn index_mut(&mut self, k: K) -> &mut V { &mut self.elems[k.index()] diff --git a/lib/cretonne/src/entity/sparse.rs b/lib/cretonne/src/entity/sparse.rs index e22df2786a..1476092121 100644 --- a/lib/cretonne/src/entity/sparse.rs +++ b/lib/cretonne/src/entity/sparse.rs @@ -51,16 +51,18 @@ pub trait SparseMapValue { /// - `SparseMap` requires the values to implement `SparseMapValue` which means that they must /// contain their own key. pub struct SparseMap - where K: EntityRef, - V: SparseMapValue +where + K: EntityRef, + V: SparseMapValue, { sparse: EntityMap, dense: Vec, } impl SparseMap - where K: EntityRef, - V: SparseMapValue +where + K: EntityRef, + V: SparseMapValue, { /// Create a new empty mapping. pub fn new() -> Self { @@ -191,8 +193,9 @@ impl SparseMap /// Iterating over the elements of a set. impl<'a, K, V> IntoIterator for &'a SparseMap - where K: EntityRef, - V: SparseMapValue +where + K: EntityRef, + V: SparseMapValue, { type Item = &'a V; type IntoIter = slice::Iter<'a, V>; @@ -204,7 +207,8 @@ impl<'a, K, V> IntoIterator for &'a SparseMap /// Any `EntityRef` can be used as a sparse map value representing itself. impl SparseMapValue for T - where T: EntityRef +where + T: EntityRef, { fn key(&self) -> T { *self @@ -290,8 +294,10 @@ mod tests { assert_eq!(map.insert(Obj(i0, "baz")), None); // Iteration order = insertion order when nothing has been removed yet. - assert_eq!(map.values().map(|obj| obj.1).collect::>(), - ["foo", "bar", "baz"]); + assert_eq!( + map.values().map(|obj| obj.1).collect::>(), + ["foo", "bar", "baz"] + ); assert_eq!(map.len(), 3); assert_eq!(map.get(i0), Some(&Obj(i0, "baz"))); diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index c4d2000bf6..fa0b5c69c8 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -89,7 +89,8 @@ impl<'f, IIB: InstInserterBase<'f>> InsertBuilder<'f, IIB> { /// /// The `reuse` argument is expected to be an array of `Option`. pub fn with_results(self, reuse: Array) -> InsertReuseBuilder<'f, IIB, Array> - where Array: AsRef<[Option]> + where + Array: AsRef<[Option]>, { InsertReuseBuilder { inserter: self.inserter, @@ -134,8 +135,9 @@ impl<'f, IIB: InstInserterBase<'f>> InstBuilderBase<'f> for InsertBuilder<'f, II /// Builder that inserts a new instruction like `InsertBuilder`, but reusing result values. pub struct InsertReuseBuilder<'f, IIB, Array> - where IIB: InstInserterBase<'f>, - Array: AsRef<[Option]> +where + IIB: InstInserterBase<'f>, + Array: AsRef<[Option]>, { inserter: IIB, reuse: Array, diff --git a/lib/cretonne/src/ir/condcodes.rs b/lib/cretonne/src/ir/condcodes.rs index 26d81ede6e..4bbf82cb6f 100644 --- a/lib/cretonne/src/ir/condcodes.rs +++ b/lib/cretonne/src/ir/condcodes.rs @@ -89,17 +89,17 @@ impl Display for IntCC { fn fmt(&self, f: &mut Formatter) -> fmt::Result { use self::IntCC::*; f.write_str(match *self { - Equal => "eq", - NotEqual => "ne", - SignedGreaterThan => "sgt", - SignedGreaterThanOrEqual => "sge", - SignedLessThan => "slt", - SignedLessThanOrEqual => "sle", - UnsignedGreaterThan => "ugt", - UnsignedGreaterThanOrEqual => "uge", - UnsignedLessThan => "ult", - UnsignedLessThanOrEqual => "ule", - }) + Equal => "eq", + NotEqual => "ne", + SignedGreaterThan => "sgt", + SignedGreaterThanOrEqual => "sge", + SignedLessThan => "slt", + SignedLessThanOrEqual => "sle", + UnsignedGreaterThan => "ugt", + UnsignedGreaterThanOrEqual => "uge", + UnsignedLessThan => "ult", + UnsignedLessThanOrEqual => "ule", + }) } } @@ -220,21 +220,21 @@ impl Display for FloatCC { fn fmt(&self, f: &mut Formatter) -> fmt::Result { use self::FloatCC::*; f.write_str(match *self { - Ordered => "ord", - Unordered => "uno", - Equal => "eq", - NotEqual => "ne", - OrderedNotEqual => "one", - UnorderedOrEqual => "ueq", - LessThan => "lt", - LessThanOrEqual => "le", - GreaterThan => "gt", - GreaterThanOrEqual => "ge", - UnorderedOrLessThan => "ult", - UnorderedOrLessThanOrEqual => "ule", - UnorderedOrGreaterThan => "ugt", - UnorderedOrGreaterThanOrEqual => "uge", - }) + Ordered => "ord", + Unordered => "uno", + Equal => "eq", + NotEqual => "ne", + OrderedNotEqual => "one", + UnorderedOrEqual => "ueq", + LessThan => "lt", + LessThanOrEqual => "le", + GreaterThan => "gt", + GreaterThanOrEqual => "ge", + UnorderedOrLessThan => "ult", + UnorderedOrLessThanOrEqual => "ule", + UnorderedOrGreaterThan => "ugt", + UnorderedOrGreaterThanOrEqual => "uge", + }) } } @@ -267,16 +267,18 @@ impl FromStr for FloatCC { mod tests { use super::*; - static INT_ALL: [IntCC; 10] = [IntCC::Equal, - IntCC::NotEqual, - IntCC::SignedLessThan, - IntCC::SignedGreaterThanOrEqual, - IntCC::SignedGreaterThan, - IntCC::SignedLessThanOrEqual, - IntCC::UnsignedLessThan, - IntCC::UnsignedGreaterThanOrEqual, - IntCC::UnsignedGreaterThan, - IntCC::UnsignedLessThanOrEqual]; + static INT_ALL: [IntCC; 10] = [ + IntCC::Equal, + IntCC::NotEqual, + IntCC::SignedLessThan, + IntCC::SignedGreaterThanOrEqual, + IntCC::SignedGreaterThan, + IntCC::SignedLessThanOrEqual, + IntCC::UnsignedLessThan, + IntCC::UnsignedGreaterThanOrEqual, + IntCC::UnsignedGreaterThan, + IntCC::UnsignedLessThanOrEqual, + ]; #[test] fn int_inverse() { @@ -306,20 +308,22 @@ mod tests { assert_eq!("bogus".parse::(), Err(())); } - static FLOAT_ALL: [FloatCC; 14] = [FloatCC::Ordered, - FloatCC::Unordered, - FloatCC::Equal, - FloatCC::NotEqual, - FloatCC::OrderedNotEqual, - FloatCC::UnorderedOrEqual, - FloatCC::LessThan, - FloatCC::LessThanOrEqual, - FloatCC::GreaterThan, - FloatCC::GreaterThanOrEqual, - FloatCC::UnorderedOrLessThan, - FloatCC::UnorderedOrLessThanOrEqual, - FloatCC::UnorderedOrGreaterThan, - FloatCC::UnorderedOrGreaterThanOrEqual]; + static FLOAT_ALL: [FloatCC; 14] = [ + FloatCC::Ordered, + FloatCC::Unordered, + FloatCC::Equal, + FloatCC::NotEqual, + FloatCC::OrderedNotEqual, + FloatCC::UnorderedOrEqual, + FloatCC::LessThan, + FloatCC::LessThanOrEqual, + FloatCC::GreaterThan, + FloatCC::GreaterThanOrEqual, + FloatCC::UnorderedOrLessThan, + FloatCC::UnorderedOrLessThanOrEqual, + FloatCC::UnorderedOrGreaterThan, + FloatCC::UnorderedOrGreaterThanOrEqual, + ]; #[test] fn float_inverse() { diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 8a1447f858..240c5377c0 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -153,17 +153,21 @@ impl DataFlowGraph { pub fn value_def(&self, v: Value) -> ValueDef { match self.values[v] { ValueData::Inst { inst, num, .. } => { - assert_eq!(Some(v), - self.results[inst].get(num as usize, &self.value_lists), - "Dangling result value {}: {}", - v, - self.display_inst(inst, None)); + assert_eq!( + Some(v), + self.results[inst].get(num as usize, &self.value_lists), + "Dangling result value {}: {}", + v, + self.display_inst(inst, None) + ); ValueDef::Res(inst, num as usize) } ValueData::Arg { ebb, num, .. } => { - assert_eq!(Some(v), - self.ebbs[ebb].args.get(num as usize, &self.value_lists), - "Dangling EBB argument value"); + assert_eq!( + Some(v), + self.ebbs[ebb].args.get(num as usize, &self.value_lists), + "Dangling EBB argument value" + ); ValueDef::Arg(ebb, num as usize) } ValueData::Alias { original, .. } => { @@ -247,19 +251,23 @@ impl DataFlowGraph { // Try to create short alias chains by finding the original source value. // This also avoids the creation of loops. let original = self.resolve_aliases(src); - assert_ne!(dest, - original, - "Aliasing {} to {} would create a loop", - dest, - src); + assert_ne!( + dest, + original, + "Aliasing {} to {} would create a loop", + dest, + src + ); let ty = self.value_type(original); - assert_eq!(self.value_type(dest), - ty, - "Aliasing {} to {} would change its type {} to {}", - dest, - src, - self.value_type(dest), - ty); + assert_eq!( + self.value_type(dest), + ty, + "Aliasing {} to {} would change its type {} to {}", + dest, + src, + self.value_type(dest), + ty + ); self.values[dest] = ValueData::Alias { ty, original }; } @@ -274,29 +282,36 @@ impl DataFlowGraph { /// cleared, so it likely needs to be removed from the graph. /// pub fn replace_with_aliases(&mut self, dest_inst: Inst, src_inst: Inst) { - debug_assert_ne!(dest_inst, - src_inst, - "Replacing {} with itself would create a loop", - dest_inst); - debug_assert_eq!(self.results[dest_inst].len(&self.value_lists), - self.results[src_inst].len(&self.value_lists), - "Replacing {} with {} would produce a different number of results.", - dest_inst, - src_inst); + debug_assert_ne!( + dest_inst, + src_inst, + "Replacing {} with itself would create a loop", + dest_inst + ); + debug_assert_eq!( + self.results[dest_inst].len(&self.value_lists), + self.results[src_inst].len(&self.value_lists), + "Replacing {} with {} would produce a different number of results.", + dest_inst, + src_inst + ); for (&dest, &src) in self.results[dest_inst] - .as_slice(&self.value_lists) - .iter() - .zip(self.results[src_inst].as_slice(&self.value_lists)) { + .as_slice(&self.value_lists) + .iter() + .zip(self.results[src_inst].as_slice(&self.value_lists)) + { let original = src; let ty = self.value_type(original); - assert_eq!(self.value_type(dest), - ty, - "Aliasing {} to {} would change its type {} to {}", - dest, - src, - self.value_type(dest), - ty); + assert_eq!( + self.value_type(dest), + ty, + "Aliasing {} to {} would change its type {} to {}", + dest, + src, + self.value_type(dest), + ty + ); self.values[dest] = ValueData::Alias { ty, original }; } @@ -371,10 +386,11 @@ impl DataFlowGraph { } /// Returns an object that displays `inst`. - pub fn display_inst<'a, I: Into>>(&'a self, - inst: Inst, - isa: I) - -> DisplayInst<'a> { + pub fn display_inst<'a, I: Into>>( + &'a self, + inst: Inst, + isa: I, + ) -> DisplayInst<'a> { DisplayInst(self, isa.into(), inst) } @@ -433,12 +449,14 @@ impl DataFlowGraph { /// Create a new set of result values for `inst` using `ctrl_typevar` to determine the result /// types. Any values provided by `reuse` will be reused. When `reuse` is exhausted or when it /// produces `None`, a new value is created. - pub fn make_inst_results_reusing(&mut self, - inst: Inst, - ctrl_typevar: Type, - reuse: I) - -> usize - where I: Iterator> + pub fn make_inst_results_reusing( + &mut self, + inst: Inst, + ctrl_typevar: Type, + reuse: I, + ) -> usize + where + I: Iterator>, { let mut reuse = reuse.fuse(); let constraints = self.insts[inst].opcode().constraints(); @@ -478,9 +496,10 @@ impl DataFlowGraph { } /// Create an `InsertBuilder` that will insert an instruction at the cursor's current position. - pub fn ins<'c, 'fc: 'c, 'fd>(&'fd mut self, - at: &'c mut Cursor<'fc>) - -> InsertBuilder<'fd, LayoutCursorInserter<'c, 'fc, 'fd>> { + pub fn ins<'c, 'fc: 'c, 'fd>( + &'fd mut self, + at: &'c mut Cursor<'fc>, + ) -> InsertBuilder<'fd, LayoutCursorInserter<'c, 'fc, 'fd>> { InsertBuilder::new(LayoutCursorInserter::new(at, self)) } @@ -537,20 +556,24 @@ impl DataFlowGraph { _ => panic!("{} is not an instruction result value", old_value), }; let new_value = self.make_value(ValueData::Inst { - ty: new_type, - num, - inst, - }); + ty: new_type, + num, + inst, + }); let num = num as usize; - let attached = mem::replace(self.results[inst] - .get_mut(num, &mut self.value_lists) - .expect("Replacing detached result"), - new_value); - assert_eq!(attached, - old_value, - "{} wasn't detached from {}", - old_value, - self.display_inst(inst, None)); + let attached = mem::replace( + self.results[inst] + .get_mut(num, &mut self.value_lists) + .expect("Replacing detached result"), + new_value, + ); + assert_eq!( + attached, + old_value, + "{} wasn't detached from {}", + old_value, + self.display_inst(inst, None) + ); new_value } @@ -560,19 +583,19 @@ impl DataFlowGraph { let num = self.results[inst].push(res, &mut self.value_lists); assert!(num <= u16::MAX as usize, "Too many result values"); self.make_value(ValueData::Inst { - ty, - inst, - num: num as u16, - }) + ty, + inst, + num: num as u16, + }) } /// Append a new value argument to an instruction. /// /// Panics if the instruction doesn't support arguments. pub fn append_inst_arg(&mut self, inst: Inst, new_arg: Value) { - let mut branch_values = self.insts[inst] - .take_value_list() - .expect("the instruction doesn't have value arguments"); + let mut branch_values = self.insts[inst].take_value_list().expect( + "the instruction doesn't have value arguments", + ); branch_values.push(new_arg, &mut self.value_lists); self.insts[inst].put_value_list(branch_values) } @@ -581,9 +604,9 @@ impl DataFlowGraph { /// /// This function panics if the instruction doesn't have any result. pub fn first_result(&self, inst: Inst) -> Value { - self.results[inst] - .first(&self.value_lists) - .expect("Instruction has no results") + self.results[inst].first(&self.value_lists).expect( + "Instruction has no results", + ) } /// Test if `inst` has any result values currently. @@ -613,11 +636,12 @@ impl DataFlowGraph { /// called first. /// /// Returns `None` if asked about a result index that is too large. - pub fn compute_result_type(&self, - inst: Inst, - result_idx: usize, - ctrl_typevar: Type) - -> Option { + pub fn compute_result_type( + &self, + inst: Inst, + result_idx: usize, + ctrl_typevar: Type, + ) -> Option { let constraints = self.insts[inst].opcode().constraints(); let fixed_results = constraints.fixed_results(); @@ -626,13 +650,12 @@ impl DataFlowGraph { } // Not a fixed result, try to extract a return type from the call signature. - self.call_signature(inst) - .and_then(|sigref| { - self.signatures[sigref] - .return_types - .get(result_idx - fixed_results) - .map(|&arg| arg.value_type) - }) + self.call_signature(inst).and_then(|sigref| { + self.signatures[sigref] + .return_types + .get(result_idx - fixed_results) + .map(|&arg| arg.value_type) + }) } /// Get the controlling type variable, or `VOID` if `inst` isn't polymorphic. @@ -644,8 +667,9 @@ impl DataFlowGraph { } else if constraints.requires_typevar_operand() { // Not all instruction formats have a designated operand, but in that case // `requires_typevar_operand()` should never be true. - self.value_type(self[inst].typevar_operand(&self.value_lists) - .expect("Instruction format doesn't have a designated operand, bad opcode.")) + self.value_type(self[inst].typevar_operand(&self.value_lists).expect( + "Instruction format doesn't have a designated operand, bad opcode.", + )) } else { self.value_type(self.first_result(inst)) } @@ -691,10 +715,10 @@ impl DataFlowGraph { let num = self.ebbs[ebb].args.push(arg, &mut self.value_lists); assert!(num <= u16::MAX as usize, "Too many arguments to EBB"); self.make_value(ValueData::Arg { - ty, - num: num as u16, - ebb, - }) + ty, + num: num as u16, + ebb, + }) } /// Removes `val` from `ebb`'s argument by swapping it with the last argument of `ebb`. @@ -712,9 +736,10 @@ impl DataFlowGraph { } else { panic!("{} must be an EBB argument", val); }; - self.ebbs[ebb] - .args - .swap_remove(num as usize, &mut self.value_lists); + self.ebbs[ebb].args.swap_remove( + num as usize, + &mut self.value_lists, + ); if let Some(last_arg_val) = self.ebbs[ebb].args.get(num as usize, &self.value_lists) { // We update the position of the old last arg. if let ValueData::Arg { num: ref mut old_num, .. } = self.values[last_arg_val] { @@ -734,23 +759,26 @@ impl DataFlowGraph { } else { panic!("{} must be an EBB argument", val); }; - self.ebbs[ebb] - .args - .remove(num as usize, &mut self.value_lists); + self.ebbs[ebb].args.remove( + num as usize, + &mut self.value_lists, + ); for index in num..(self.ebb_args(ebb).len() as u16) { match self.values[self.ebbs[ebb] - .args - .get(index as usize, &self.value_lists) - .unwrap()] { + .args + .get(index as usize, &self.value_lists) + .unwrap()] { ValueData::Arg { ref mut num, .. } => { *num -= 1; } _ => { - panic!("{} must be an EBB argument", - self.ebbs[ebb] - .args - .get(index as usize, &self.value_lists) - .unwrap()) + panic!( + "{} must be an EBB argument", + self.ebbs[ebb] + .args + .get(index as usize, &self.value_lists) + .unwrap() + ) } } } @@ -791,10 +819,10 @@ impl DataFlowGraph { panic!("{} must be an EBB argument", old_arg); }; let new_arg = self.make_value(ValueData::Arg { - ty: new_type, - num, - ebb, - }); + ty: new_type, + num, + ebb, + }); self.ebbs[ebb].args.as_mut_slice(&mut self.value_lists)[num as usize] = new_arg; new_arg diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 434da750e9..4060e6ff37 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -218,7 +218,9 @@ mod tests { use std::mem; use packed_option::PackedOption; // This is the whole point of `PackedOption`. - assert_eq!(mem::size_of::(), - mem::size_of::>()); + assert_eq!( + mem::size_of::(), + mem::size_of::>() + ); } } diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 9aad88ef13..2606ebe19e 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -55,11 +55,11 @@ impl Signature { let bytes = self.argument_types .iter() .filter_map(|arg| match arg.location { - ArgumentLoc::Stack(offset) if offset >= 0 => { - Some(offset as u32 + arg.value_type.bytes()) - } - _ => None, - }) + ArgumentLoc::Stack(offset) if offset >= 0 => { + Some(offset as u32 + arg.value_type.bytes()) + } + _ => None, + }) .fold(0, cmp::max); self.argument_bytes = Some(bytes); } @@ -73,10 +73,11 @@ impl Signature { /// Wrapper type capable of displaying a `Signature` with correct register names. pub struct DisplaySignature<'a>(&'a Signature, Option<&'a RegInfo>); -fn write_list(f: &mut fmt::Formatter, - args: &[ArgumentType], - regs: Option<&RegInfo>) - -> fmt::Result { +fn write_list( + f: &mut fmt::Formatter, + args: &[ArgumentType], + regs: Option<&RegInfo>, +) -> fmt::Result { match args.split_first() { None => {} Some((first, rest)) => { @@ -310,9 +311,9 @@ impl fmt::Display for CallConv { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { use self::CallConv::*; f.write_str(match *self { - Native => "native", - SpiderWASM => "spiderwasm", - }) + Native => "native", + SpiderWASM => "spiderwasm", + }) } } @@ -346,12 +347,14 @@ mod tests { #[test] fn argument_purpose() { - let all_purpose = [ArgumentPurpose::Normal, - ArgumentPurpose::StructReturn, - ArgumentPurpose::Link, - ArgumentPurpose::FramePointer, - ArgumentPurpose::CalleeSaved, - ArgumentPurpose::VMContext]; + let all_purpose = [ + ArgumentPurpose::Normal, + ArgumentPurpose::StructReturn, + ArgumentPurpose::Link, + ArgumentPurpose::FramePointer, + ArgumentPurpose::CalleeSaved, + ArgumentPurpose::VMContext, + ]; for (&e, &n) in all_purpose.iter().zip(PURPOSE_NAMES.iter()) { assert_eq!(e.to_string(), n); assert_eq!(Ok(e), n.parse()); @@ -373,8 +376,9 @@ mod tests { assert_eq!(sig.to_string(), "(i32) spiderwasm"); sig.return_types.push(ArgumentType::new(F32)); assert_eq!(sig.to_string(), "(i32) -> f32 spiderwasm"); - sig.argument_types - .push(ArgumentType::new(I32.by(4).unwrap())); + sig.argument_types.push( + ArgumentType::new(I32.by(4).unwrap()), + ); assert_eq!(sig.to_string(), "(i32, i32x4) -> f32 spiderwasm"); sig.return_types.push(ArgumentType::new(B8)); assert_eq!(sig.to_string(), "(i32, i32x4) -> f32, b8 spiderwasm"); @@ -391,7 +395,9 @@ mod tests { assert_eq!(sig.argument_bytes, Some(28)); // Writing ABI-annotated signatures. - assert_eq!(sig.to_string(), - "(i32 [24], i32x4 [8]) -> f32, b8 spiderwasm"); + assert_eq!( + sig.to_string(), + "(i32 [24], i32x4 [8]) -> f32, b8 spiderwasm" + ); } } diff --git a/lib/cretonne/src/ir/funcname.rs b/lib/cretonne/src/ir/funcname.rs index 4ab76d2001..2081f519e5 100644 --- a/lib/cretonne/src/ir/funcname.rs +++ b/lib/cretonne/src/ir/funcname.rs @@ -30,7 +30,8 @@ impl FunctionName { /// assert_eq!(name.to_string(), "#0a0908"); /// ``` pub fn new(v: T) -> FunctionName - where T: Into> + where + T: Into>, { let vec = v.into(); if vec.len() <= NAME_LENGTH_THRESHOLD { @@ -39,9 +40,9 @@ impl FunctionName { bytes[i] = byte; } FunctionName(NameRepr::Short { - length: vec.len() as u8, - bytes: bytes, - }) + length: vec.len() as u8, + bytes: bytes, + }) } else { FunctionName(NameRepr::Long(vec)) } @@ -114,11 +115,17 @@ mod tests { assert_eq!(FunctionName::new("x").to_string(), "%x"); assert_eq!(FunctionName::new("x_1").to_string(), "%x_1"); assert_eq!(FunctionName::new(" ").to_string(), "#20"); - assert_eq!(FunctionName::new("кретон").to_string(), - "#d0bad180d0b5d182d0bed0bd"); - assert_eq!(FunctionName::new("印花棉布").to_string(), - "#e58db0e88ab1e6a389e5b883"); - assert_eq!(FunctionName::new(vec![0, 1, 2, 3, 4, 5]).to_string(), - "#000102030405"); + assert_eq!( + FunctionName::new("кретон").to_string(), + "#d0bad180d0b5d182d0bed0bd" + ); + assert_eq!( + FunctionName::new("印花棉布").to_string(), + "#e58db0e88ab1e6a389e5b883" + ); + assert_eq!( + FunctionName::new(vec![0, 1, 2, 3, 4, 5]).to_string(), + "#000102030405" + ); } } diff --git a/lib/cretonne/src/ir/heap.rs b/lib/cretonne/src/ir/heap.rs index 27a970e36c..7920ec344a 100644 --- a/lib/cretonne/src/ir/heap.rs +++ b/lib/cretonne/src/ir/heap.rs @@ -51,9 +51,9 @@ pub enum HeapStyle { impl fmt::Display for HeapData { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { f.write_str(match self.style { - HeapStyle::Dynamic { .. } => "dynamic", - HeapStyle::Static { .. } => "static", - })?; + HeapStyle::Dynamic { .. } => "dynamic", + HeapStyle::Static { .. } => "static", + })?; match self.base { HeapBase::ReservedReg => write!(f, " reserved_reg")?, diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 8800fb2229..72763c558e 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -192,10 +192,10 @@ impl FromStr for Uimm32 { // Parse a decimal or hexadecimal `Uimm32`, formatted as above. fn from_str(s: &str) -> Result { parse_i64(s).and_then(|x| if 0 <= x && x <= u32::MAX as i64 { - Ok(Uimm32(x as u32)) - } else { - Err("Uimm32 out of range") - }) + Ok(Uimm32(x as u32)) + } else { + Err("Uimm32 out of range") + }) } } @@ -260,10 +260,10 @@ impl FromStr for Offset32 { return Err("Offset must begin with sign"); } parse_i64(s).and_then(|x| if i32::MIN as i64 <= x && x <= i32::MAX as i64 { - Ok(Offset32::new(x as i32)) - } else { - Err("Offset out of range") - }) + Ok(Offset32::new(x as i32)) + } else { + Err("Offset out of range") + }) } } @@ -325,10 +325,10 @@ impl FromStr for Uoffset32 { return Err("Unsigned offset must begin with '+' sign"); } parse_i64(s).and_then(|x| if 0 <= x && x <= u32::MAX as i64 { - Ok(Uoffset32::new(x as u32)) - } else { - Err("Offset out of range") - }) + Ok(Uoffset32::new(x as u32)) + } else { + Err("Offset out of range") + }) } } @@ -458,20 +458,20 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { if s2.starts_with("NaN:0x") { // Quiet NaN with payload. return match u64::from_str_radix(&s2[6..], 16) { - Ok(payload) if payload < quiet_bit => { - Ok(sign_bit | max_e_bits | quiet_bit | payload) - } - _ => Err("Invalid NaN payload"), - }; + Ok(payload) if payload < quiet_bit => { + Ok(sign_bit | max_e_bits | quiet_bit | payload) + } + _ => Err("Invalid NaN payload"), + }; } if s2.starts_with("sNaN:0x") { // Signaling NaN with payload. return match u64::from_str_radix(&s2[7..], 16) { - Ok(payload) if 0 < payload && payload < quiet_bit => { - Ok(sign_bit | max_e_bits | payload) - } - _ => Err("Invalid sNaN payload"), - }; + Ok(payload) if 0 < payload && payload < quiet_bit => { + Ok(sign_bit | max_e_bits | payload) + } + _ => Err("Invalid sNaN payload"), + }; } return Err("Float must be hexadecimal"); @@ -662,7 +662,8 @@ mod tests { // Verify that `text` can be parsed as a `T` into a value that displays as `want`. fn parse_ok(text: &str, want: &str) - where ::Err: Display + where + ::Err: Display, { match text.parse::() { Err(s) => panic!("\"{}\".parse() error: {}", text, s), @@ -672,7 +673,8 @@ mod tests { // Verify that `text` fails to parse as `T` with the error `msg`. fn parse_err(text: &str, msg: &str) - where ::Err: Display + where + ::Err: Display, { match text.parse::() { Err(s) => assert_eq!(s.to_string(), msg), @@ -781,18 +783,26 @@ mod tests { assert_eq!(Ieee32::with_float(1.0).to_string(), "0x1.000000p0"); assert_eq!(Ieee32::with_float(1.5).to_string(), "0x1.800000p0"); assert_eq!(Ieee32::with_float(0.5).to_string(), "0x1.000000p-1"); - assert_eq!(Ieee32::with_float(f32::EPSILON).to_string(), - "0x1.000000p-23"); + assert_eq!( + Ieee32::with_float(f32::EPSILON).to_string(), + "0x1.000000p-23" + ); assert_eq!(Ieee32::with_float(f32::MIN).to_string(), "-0x1.fffffep127"); assert_eq!(Ieee32::with_float(f32::MAX).to_string(), "0x1.fffffep127"); // Smallest positive normal number. - assert_eq!(Ieee32::with_float(f32::MIN_POSITIVE).to_string(), - "0x1.000000p-126"); + assert_eq!( + Ieee32::with_float(f32::MIN_POSITIVE).to_string(), + "0x1.000000p-126" + ); // Subnormals. - assert_eq!(Ieee32::with_float(f32::MIN_POSITIVE / 2.0).to_string(), - "0x0.800000p-126"); - assert_eq!(Ieee32::with_float(f32::MIN_POSITIVE * f32::EPSILON).to_string(), - "0x0.000002p-126"); + assert_eq!( + Ieee32::with_float(f32::MIN_POSITIVE / 2.0).to_string(), + "0x0.800000p-126" + ); + assert_eq!( + Ieee32::with_float(f32::MIN_POSITIVE * f32::EPSILON).to_string(), + "0x0.000002p-126" + ); assert_eq!(Ieee32::with_float(f32::INFINITY).to_string(), "+Inf"); assert_eq!(Ieee32::with_float(f32::NEG_INFINITY).to_string(), "-Inf"); assert_eq!(Ieee32::with_float(f32::NAN).to_string(), "+NaN"); @@ -883,32 +893,48 @@ mod tests { assert_eq!(Ieee64::with_float(1.0).to_string(), "0x1.0000000000000p0"); assert_eq!(Ieee64::with_float(1.5).to_string(), "0x1.8000000000000p0"); assert_eq!(Ieee64::with_float(0.5).to_string(), "0x1.0000000000000p-1"); - assert_eq!(Ieee64::with_float(f64::EPSILON).to_string(), - "0x1.0000000000000p-52"); - assert_eq!(Ieee64::with_float(f64::MIN).to_string(), - "-0x1.fffffffffffffp1023"); - assert_eq!(Ieee64::with_float(f64::MAX).to_string(), - "0x1.fffffffffffffp1023"); + assert_eq!( + Ieee64::with_float(f64::EPSILON).to_string(), + "0x1.0000000000000p-52" + ); + assert_eq!( + Ieee64::with_float(f64::MIN).to_string(), + "-0x1.fffffffffffffp1023" + ); + assert_eq!( + Ieee64::with_float(f64::MAX).to_string(), + "0x1.fffffffffffffp1023" + ); // Smallest positive normal number. - assert_eq!(Ieee64::with_float(f64::MIN_POSITIVE).to_string(), - "0x1.0000000000000p-1022"); + assert_eq!( + Ieee64::with_float(f64::MIN_POSITIVE).to_string(), + "0x1.0000000000000p-1022" + ); // Subnormals. - assert_eq!(Ieee64::with_float(f64::MIN_POSITIVE / 2.0).to_string(), - "0x0.8000000000000p-1022"); - assert_eq!(Ieee64::with_float(f64::MIN_POSITIVE * f64::EPSILON).to_string(), - "0x0.0000000000001p-1022"); + assert_eq!( + Ieee64::with_float(f64::MIN_POSITIVE / 2.0).to_string(), + "0x0.8000000000000p-1022" + ); + assert_eq!( + Ieee64::with_float(f64::MIN_POSITIVE * f64::EPSILON).to_string(), + "0x0.0000000000001p-1022" + ); assert_eq!(Ieee64::with_float(f64::INFINITY).to_string(), "+Inf"); assert_eq!(Ieee64::with_float(f64::NEG_INFINITY).to_string(), "-Inf"); assert_eq!(Ieee64::with_float(f64::NAN).to_string(), "+NaN"); assert_eq!(Ieee64::with_float(-f64::NAN).to_string(), "-NaN"); // Construct some qNaNs with payloads. assert_eq!(Ieee64(0x7ff8000000000001).to_string(), "+NaN:0x1"); - assert_eq!(Ieee64(0x7ffc000000000001).to_string(), - "+NaN:0x4000000000001"); + assert_eq!( + Ieee64(0x7ffc000000000001).to_string(), + "+NaN:0x4000000000001" + ); // Signaling NaNs. assert_eq!(Ieee64(0x7ff0000000000001).to_string(), "+sNaN:0x1"); - assert_eq!(Ieee64(0x7ff4000000000001).to_string(), - "+sNaN:0x4000000000001"); + assert_eq!( + Ieee64(0x7ff4000000000001).to_string(), + "+sNaN:0x4000000000001" + ); } #[test] diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index ce646150ee..19ef8efa29 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -481,7 +481,8 @@ impl OpcodeConstraints { pub fn result_type(self, n: usize, ctrl_type: Type) -> Type { assert!(n < self.fixed_results(), "Invalid result index"); if let ResolvedConstraint::Bound(t) = - OPERAND_CONSTRAINTS[self.constraint_offset() + n].resolve(ctrl_type) { + OPERAND_CONSTRAINTS[self.constraint_offset() + n].resolve(ctrl_type) + { t } else { panic!("Result constraints can't be free"); @@ -494,8 +495,10 @@ impl OpcodeConstraints { /// Unlike results, it is possible for some input values to vary freely within a specific /// `ValueTypeSet`. This is represented with the `ArgumentConstraint::Free` variant. pub fn value_argument_constraint(self, n: usize, ctrl_type: Type) -> ResolvedConstraint { - assert!(n < self.fixed_value_arguments(), - "Invalid value argument index"); + assert!( + n < self.fixed_value_arguments(), + "Invalid value argument index" + ); let offset = self.constraint_offset() + self.fixed_results(); OPERAND_CONSTRAINTS[offset + n].resolve(ctrl_type) } @@ -613,14 +616,14 @@ impl OperandConstraint { AsBool => Bound(ctrl_type.as_bool()), HalfWidth => Bound(ctrl_type.half_width().expect("invalid type for half_width")), DoubleWidth => { - Bound(ctrl_type - .double_width() - .expect("invalid type for double_width")) + Bound(ctrl_type.double_width().expect( + "invalid type for double_width", + )) } HalfVector => { - Bound(ctrl_type - .half_vector() - .expect("invalid type for half_vector")) + Bound(ctrl_type.half_vector().expect( + "invalid type for half_vector", + )) } DoubleVector => Bound(ctrl_type.by(2).expect("invalid type for double_vector")), } @@ -688,10 +691,14 @@ mod tests { assert_eq!(a.fixed_value_arguments(), 2); assert_eq!(a.result_type(0, types::I32), types::I32); assert_eq!(a.result_type(0, types::I8), types::I8); - assert_eq!(a.value_argument_constraint(0, types::I32), - ResolvedConstraint::Bound(types::I32)); - assert_eq!(a.value_argument_constraint(1, types::I32), - ResolvedConstraint::Bound(types::I32)); + assert_eq!( + a.value_argument_constraint(0, types::I32), + ResolvedConstraint::Bound(types::I32) + ); + assert_eq!( + a.value_argument_constraint(1, types::I32), + ResolvedConstraint::Bound(types::I32) + ); let b = Opcode::Bitcast.constraints(); assert!(!b.use_typevar_operand()); diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index c0f97afea3..5e34d07c4d 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -71,9 +71,9 @@ impl JumpTableData { /// Checks if any of the entries branch to `ebb`. pub fn branches_to(&self, ebb: Ebb) -> bool { - self.table - .iter() - .any(|target_ebb| target_ebb.expand() == Some(ebb)) + self.table.iter().any(|target_ebb| { + target_ebb.expand() == Some(ebb) + }) } /// Access the whole table as a mutable slice. @@ -148,8 +148,10 @@ mod tests { jt.set_entry(0, e2); jt.set_entry(10, e1); - assert_eq!(jt.to_string(), - "jump_table ebb2, 0, 0, 0, 0, 0, 0, 0, 0, 0, ebb1"); + assert_eq!( + jt.to_string(), + "jump_table ebb2, 0, 0, 0, 0, 0, 0, 0, 0, 0, ebb1" + ); let v: Vec<(usize, Ebb)> = jt.entries().collect(); assert_eq!(v, [(0, e2), (10, e1)]); diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index ca54cefdb3..c1cc62b110 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -96,8 +96,9 @@ fn test_midpoint() { impl ProgramOrder for Layout { fn cmp(&self, a: A, b: B) -> cmp::Ordering - where A: Into, - B: Into + where + A: Into, + B: Into, { let a_seq = self.seq(a); let b_seq = self.seq(b); @@ -166,8 +167,9 @@ impl Layout { /// Assign a valid sequence number to `inst` such that the numbers are still monotonic. This may /// require renumbering. fn assign_inst_seq(&mut self, inst: Inst) { - let ebb = self.inst_ebb(inst) - .expect("inst must be inserted before assigning an seq"); + let ebb = self.inst_ebb(inst).expect( + "inst must be inserted before assigning an seq", + ); // Get the sequence number immediately before `inst`. let prev_seq = match self.insts[inst].prev.expand() { @@ -283,8 +285,10 @@ impl Layout { /// Insert `ebb` as the last EBB in the layout. pub fn append_ebb(&mut self, ebb: Ebb) { - assert!(!self.is_ebb_inserted(ebb), - "Cannot append EBB that is already in the layout"); + assert!( + !self.is_ebb_inserted(ebb), + "Cannot append EBB that is already in the layout" + ); { let node = &mut self.ebbs[ebb]; assert!(node.first_inst.is_none() && node.last_inst.is_none()); @@ -302,10 +306,14 @@ impl Layout { /// Insert `ebb` in the layout before the existing EBB `before`. pub fn insert_ebb(&mut self, ebb: Ebb, before: Ebb) { - assert!(!self.is_ebb_inserted(ebb), - "Cannot insert EBB that is already in the layout"); - assert!(self.is_ebb_inserted(before), - "EBB Insertion point not in the layout"); + assert!( + !self.is_ebb_inserted(ebb), + "Cannot insert EBB that is already in the layout" + ); + assert!( + self.is_ebb_inserted(before), + "EBB Insertion point not in the layout" + ); let after = self.ebbs[before].prev; { let node = &mut self.ebbs[ebb]; @@ -322,10 +330,14 @@ impl Layout { /// Insert `ebb` in the layout *after* the existing EBB `after`. pub fn insert_ebb_after(&mut self, ebb: Ebb, after: Ebb) { - assert!(!self.is_ebb_inserted(ebb), - "Cannot insert EBB that is already in the layout"); - assert!(self.is_ebb_inserted(after), - "EBB Insertion point not in the layout"); + assert!( + !self.is_ebb_inserted(ebb), + "Cannot insert EBB that is already in the layout" + ); + assert!( + self.is_ebb_inserted(after), + "EBB Insertion point not in the layout" + ); let before = self.ebbs[after].next; { let node = &mut self.ebbs[ebb]; @@ -411,7 +423,8 @@ impl Layout { /// Get the EBB containing the program point `pp`. Panic if `pp` is not in the layout. pub fn pp_ebb(&self, pp: PP) -> Ebb - where PP: Into + where + PP: Into, { match pp.into() { ExpandedProgramPoint::Ebb(ebb) => ebb, @@ -424,8 +437,10 @@ impl Layout { /// Append `inst` to the end of `ebb`. pub fn append_inst(&mut self, inst: Inst, ebb: Ebb) { assert_eq!(self.inst_ebb(inst), None); - assert!(self.is_ebb_inserted(ebb), - "Cannot append instructions to EBB not in layout"); + assert!( + self.is_ebb_inserted(ebb), + "Cannot append instructions to EBB not in layout" + ); { let ebb_node = &mut self.ebbs[ebb]; { @@ -457,8 +472,9 @@ impl Layout { /// Insert `inst` before the instruction `before` in the same EBB. pub fn insert_inst(&mut self, inst: Inst, before: Inst) { assert_eq!(self.inst_ebb(inst), None); - let ebb = self.inst_ebb(before) - .expect("Instruction before insertion point not in the layout"); + let ebb = self.inst_ebb(before).expect( + "Instruction before insertion point not in the layout", + ); let after = self.insts[before].prev; { let inst_node = &mut self.insts[inst]; @@ -531,8 +547,9 @@ impl Layout { /// i4 /// ``` pub fn split_ebb(&mut self, new_ebb: Ebb, before: Inst) { - let old_ebb = self.inst_ebb(before) - .expect("The `before` instruction must be in the layout"); + let old_ebb = self.inst_ebb(before).expect( + "The `before` instruction must be in the layout", + ); assert!(!self.is_ebb_inserted(new_ebb)); // Insert new_ebb after old_ebb. @@ -683,7 +700,8 @@ pub trait CursorBase { /// } /// ``` fn at_inst(mut self, inst: Inst) -> Self - where Self: Sized + where + Self: Sized, { self.goto_inst(inst); self @@ -703,7 +721,8 @@ pub trait CursorBase { /// } /// ``` fn at_first_inst(mut self, ebb: Ebb) -> Self - where Self: Sized + where + Self: Sized, { self.goto_first_inst(ebb); self @@ -783,9 +802,9 @@ pub trait CursorBase { self.layout().first_ebb }; self.set_position(match next { - Some(ebb) => CursorPosition::Before(ebb), - None => CursorPosition::Nowhere, - }); + Some(ebb) => CursorPosition::Before(ebb), + None => CursorPosition::Nowhere, + }); next } @@ -816,9 +835,9 @@ pub trait CursorBase { self.layout().last_ebb }; self.set_position(match prev { - Some(ebb) => CursorPosition::After(ebb), - None => CursorPosition::Nowhere, - }); + Some(ebb) => CursorPosition::After(ebb), + None => CursorPosition::Nowhere, + }); prev } @@ -872,9 +891,9 @@ pub trait CursorBase { self.set_position(At(next)); Some(next) } else { - let pos = After(self.layout() - .inst_ebb(inst) - .expect("current instruction removed?")); + let pos = After(self.layout().inst_ebb(inst).expect( + "current instruction removed?", + )); self.set_position(pos); None } @@ -925,9 +944,9 @@ pub trait CursorBase { self.set_position(At(prev)); Some(prev) } else { - let pos = Before(self.layout() - .inst_ebb(inst) - .expect("current instruction removed?")); + let pos = Before(self.layout().inst_ebb(inst).expect( + "current instruction removed?", + )); self.set_position(pos); None } @@ -1057,9 +1076,10 @@ pub struct LayoutCursorInserter<'c, 'fc: 'c, 'fd> { impl<'c, 'fc: 'c, 'fd> LayoutCursorInserter<'c, 'fc, 'fd> { /// Create a new inserter. Don't use this, use `dfg.ins(pos)`. - pub fn new(pos: &'c mut Cursor<'fc>, - dfg: &'fd mut DataFlowGraph) - -> LayoutCursorInserter<'c, 'fc, 'fd> { + pub fn new( + pos: &'c mut Cursor<'fc>, + dfg: &'fd mut DataFlowGraph, + ) -> LayoutCursorInserter<'c, 'fc, 'fd> { LayoutCursorInserter { pos, dfg } } } diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index 7d6e9197bd..d61d6755f6 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -123,8 +123,9 @@ pub trait ProgramOrder { /// directly. Depending on the implementation, there is a good chance performance will be /// improved for those cases where the type of either argument is known statically. fn cmp(&self, a: A, b: B) -> cmp::Ordering - where A: Into, - B: Into; + where + A: Into, + B: Into; /// Is the range from `inst` to `ebb` just the gap between consecutive EBBs? /// diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index 528f6b22f2..effa65a25e 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -65,11 +65,11 @@ impl fmt::Display for StackSlotKind { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { use self::StackSlotKind::*; f.write_str(match *self { - Local => "local", - SpillSlot => "spill_slot", - IncomingArg => "incoming_arg", - OutgoingArg => "outgoing_arg", - }) + Local => "local", + SpillSlot => "spill_slot", + IncomingArg => "incoming_arg", + OutgoingArg => "outgoing_arg", + }) } } @@ -228,9 +228,9 @@ impl StackSlots { let size = ty.bytes(); // Look for an existing outgoing stack slot with the same offset and size. - let inspos = match self.outgoing - .binary_search_by_key(&(offset, size), - |&ss| (self[ss].offset, self[ss].size)) { + let inspos = match self.outgoing.binary_search_by_key(&(offset, size), |&ss| { + (self[ss].offset, self[ss].size) + }) { Ok(idx) => return self.outgoing[idx], Err(idx) => idx, }; @@ -255,10 +255,14 @@ mod tests { fn stack_slot() { let mut func = Function::new(); - let ss0 = func.stack_slots - .push(StackSlotData::new(StackSlotKind::IncomingArg, 4)); - let ss1 = func.stack_slots - .push(StackSlotData::new(StackSlotKind::SpillSlot, 8)); + let ss0 = func.stack_slots.push(StackSlotData::new( + StackSlotKind::IncomingArg, + 4, + )); + let ss1 = func.stack_slots.push(StackSlotData::new( + StackSlotKind::SpillSlot, + 8, + )); assert_eq!(ss0.to_string(), "ss0"); assert_eq!(ss1.to_string(), "ss1"); diff --git a/lib/cretonne/src/isa/arm32/abi.rs b/lib/cretonne/src/isa/arm32/abi.rs index 93ae0e7e61..47efbc97bf 100644 --- a/lib/cretonne/src/isa/arm32/abi.rs +++ b/lib/cretonne/src/isa/arm32/abi.rs @@ -7,9 +7,11 @@ use settings as shared_settings; use super::registers::{S, D, Q, GPR}; /// Legalize `sig`. -pub fn legalize_signature(_sig: &mut ir::Signature, - _flags: &shared_settings::Flags, - _current: bool) { +pub fn legalize_signature( + _sig: &mut ir::Signature, + _flags: &shared_settings::Flags, + _current: bool, +) { unimplemented!() } diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index fafd5d9871..552e159ca1 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -29,19 +29,20 @@ pub fn isa_builder() -> IsaBuilder { } } -fn isa_constructor(shared_flags: shared_settings::Flags, - builder: &shared_settings::Builder) - -> Box { +fn isa_constructor( + shared_flags: shared_settings::Flags, + builder: &shared_settings::Builder, +) -> Box { let level1 = if shared_flags.is_compressed() { &enc_tables::LEVEL1_T32[..] } else { &enc_tables::LEVEL1_A32[..] }; Box::new(Isa { - isa_flags: settings::Flags::new(&shared_flags, builder), - shared_flags, - cpumode: level1, - }) + isa_flags: settings::Flags::new(&shared_flags, builder), + shared_flags, + cpumode: level1, + }) } impl TargetIsa for Isa { @@ -61,21 +62,24 @@ impl TargetIsa for Isa { enc_tables::INFO.clone() } - fn legal_encodings<'a>(&'a self, - dfg: &'a ir::DataFlowGraph, - inst: &'a ir::InstructionData, - ctrl_typevar: ir::Type) - -> Encodings<'a> { - lookup_enclist(ctrl_typevar, - inst, - dfg, - self.cpumode, - &enc_tables::LEVEL2[..], - &enc_tables::ENCLISTS[..], - &enc_tables::LEGALIZE_ACTIONS[..], - &enc_tables::RECIPE_PREDICATES[..], - &enc_tables::INST_PREDICATES[..], - self.isa_flags.predicate_view()) + fn legal_encodings<'a>( + &'a self, + dfg: &'a ir::DataFlowGraph, + inst: &'a ir::InstructionData, + ctrl_typevar: ir::Type, + ) -> Encodings<'a> { + lookup_enclist( + ctrl_typevar, + inst, + dfg, + self.cpumode, + &enc_tables::LEVEL2[..], + &enc_tables::ENCLISTS[..], + &enc_tables::LEGALIZE_ACTIONS[..], + &enc_tables::RECIPE_PREDICATES[..], + &enc_tables::INST_PREDICATES[..], + self.isa_flags.predicate_view(), + ) } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { @@ -90,11 +94,13 @@ impl TargetIsa for Isa { abi::allocatable_registers(func) } - fn emit_inst(&self, - func: &ir::Function, - inst: ir::Inst, - divert: &mut regalloc::RegDiversions, - sink: &mut CodeSink) { + fn emit_inst( + &self, + func: &ir::Function, + inst: ir::Inst, + divert: &mut regalloc::RegDiversions, + sink: &mut CodeSink, + ) { binemit::emit_inst(func, inst, divert, sink) } diff --git a/lib/cretonne/src/isa/arm64/abi.rs b/lib/cretonne/src/isa/arm64/abi.rs index f5a2dc91c7..14af59ab18 100644 --- a/lib/cretonne/src/isa/arm64/abi.rs +++ b/lib/cretonne/src/isa/arm64/abi.rs @@ -7,9 +7,11 @@ use settings as shared_settings; use super::registers::{GPR, FPR}; /// Legalize `sig`. -pub fn legalize_signature(_sig: &mut ir::Signature, - _flags: &shared_settings::Flags, - _current: bool) { +pub fn legalize_signature( + _sig: &mut ir::Signature, + _flags: &shared_settings::Flags, + _current: bool, +) { unimplemented!() } diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index f4b21bf42d..88086b233e 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -28,13 +28,14 @@ pub fn isa_builder() -> IsaBuilder { } } -fn isa_constructor(shared_flags: shared_settings::Flags, - builder: &shared_settings::Builder) - -> Box { +fn isa_constructor( + shared_flags: shared_settings::Flags, + builder: &shared_settings::Builder, +) -> Box { Box::new(Isa { - isa_flags: settings::Flags::new(&shared_flags, builder), - shared_flags, - }) + isa_flags: settings::Flags::new(&shared_flags, builder), + shared_flags, + }) } impl TargetIsa for Isa { @@ -54,21 +55,24 @@ impl TargetIsa for Isa { enc_tables::INFO.clone() } - fn legal_encodings<'a>(&'a self, - dfg: &'a ir::DataFlowGraph, - inst: &'a ir::InstructionData, - ctrl_typevar: ir::Type) - -> Encodings<'a> { - lookup_enclist(ctrl_typevar, - inst, - dfg, - &enc_tables::LEVEL1_A64[..], - &enc_tables::LEVEL2[..], - &enc_tables::ENCLISTS[..], - &enc_tables::LEGALIZE_ACTIONS[..], - &enc_tables::RECIPE_PREDICATES[..], - &enc_tables::INST_PREDICATES[..], - self.isa_flags.predicate_view()) + fn legal_encodings<'a>( + &'a self, + dfg: &'a ir::DataFlowGraph, + inst: &'a ir::InstructionData, + ctrl_typevar: ir::Type, + ) -> Encodings<'a> { + lookup_enclist( + ctrl_typevar, + inst, + dfg, + &enc_tables::LEVEL1_A64[..], + &enc_tables::LEVEL2[..], + &enc_tables::ENCLISTS[..], + &enc_tables::LEGALIZE_ACTIONS[..], + &enc_tables::RECIPE_PREDICATES[..], + &enc_tables::INST_PREDICATES[..], + self.isa_flags.predicate_view(), + ) } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { @@ -83,11 +87,13 @@ impl TargetIsa for Isa { abi::allocatable_registers(func) } - fn emit_inst(&self, - func: &ir::Function, - inst: ir::Inst, - divert: &mut regalloc::RegDiversions, - sink: &mut CodeSink) { + fn emit_inst( + &self, + func: &ir::Function, + inst: ir::Inst, + divert: &mut regalloc::RegDiversions, + sink: &mut CodeSink, + ) { binemit::emit_inst(func, inst, divert, sink) } diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index b9fa09dd65..5154138c1b 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -103,19 +103,21 @@ impl + Copy> Table for [Level2Entry] { /// list. /// /// Returns an iterator that produces legal encodings for `inst`. -pub fn lookup_enclist<'a, OffT1, OffT2>(ctrl_typevar: Type, - inst: &'a InstructionData, - dfg: &'a DataFlowGraph, - level1_table: &'static [Level1Entry], - level2_table: &'static [Level2Entry], - enclist: &'static [EncListEntry], - legalize_actions: &'static [Legalize], - recipe_preds: &'static [RecipePredicate], - inst_preds: &'static [InstPredicate], - isa_preds: PredicateView<'a>) - -> Encodings<'a> - where OffT1: Into + Copy, - OffT2: Into + Copy +pub fn lookup_enclist<'a, OffT1, OffT2>( + ctrl_typevar: Type, + inst: &'a InstructionData, + dfg: &'a DataFlowGraph, + level1_table: &'static [Level1Entry], + level2_table: &'static [Level2Entry], + enclist: &'static [EncListEntry], + legalize_actions: &'static [Legalize], + recipe_preds: &'static [RecipePredicate], + inst_preds: &'static [InstPredicate], + isa_preds: PredicateView<'a>, +) -> Encodings<'a> +where + OffT1: Into + Copy, + OffT2: Into + Copy, { let (offset, legalize) = match probe(level1_table, ctrl_typevar, ctrl_typevar.index()) { Err(l1idx) => { @@ -144,15 +146,17 @@ pub fn lookup_enclist<'a, OffT1, OffT2>(ctrl_typevar: Type, // Now we have an offset into `enclist` that is `!0` when no encoding list could be found. // The default legalization code is always valid. - Encodings::new(offset, - legalize, - inst, - dfg, - enclist, - legalize_actions, - recipe_preds, - inst_preds, - isa_preds) + Encodings::new( + offset, + legalize, + inst, + dfg, + enclist, + legalize_actions, + recipe_preds, + inst_preds, + isa_preds, + ) } /// Encoding list entry. @@ -187,16 +191,17 @@ impl<'a> Encodings<'a> { /// This iterator provides search for encodings that applies to the given instruction. The /// encoding lists are laid out such that first call to `next` returns valid entry in the list /// or `None`. - pub fn new(offset: usize, - legalize: LegalizeCode, - inst: &'a InstructionData, - dfg: &'a DataFlowGraph, - enclist: &'static [EncListEntry], - legalize_actions: &'static [Legalize], - recipe_preds: &'static [RecipePredicate], - inst_preds: &'static [InstPredicate], - isa_preds: PredicateView<'a>) - -> Self { + pub fn new( + offset: usize, + legalize: LegalizeCode, + inst: &'a InstructionData, + dfg: &'a DataFlowGraph, + enclist: &'static [EncListEntry], + legalize_actions: &'static [Legalize], + recipe_preds: &'static [RecipePredicate], + inst_preds: &'static [InstPredicate], + isa_preds: PredicateView<'a>, + ) -> Self { Encodings { offset, inst, diff --git a/lib/cretonne/src/isa/encoding.rs b/lib/cretonne/src/isa/encoding.rs index 9bc78d9ef0..cfb429ba1e 100644 --- a/lib/cretonne/src/isa/encoding.rs +++ b/lib/cretonne/src/isa/encoding.rs @@ -66,10 +66,12 @@ pub struct DisplayEncoding { impl fmt::Display for DisplayEncoding { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { if self.encoding.is_legal() { - write!(f, - "{}#{:02x}", - self.recipe_names[self.encoding.recipe()], - self.encoding.bits) + write!( + f, + "{}#{:02x}", + self.recipe_names[self.encoding.recipe()], + self.encoding.bits + ) } else { write!(f, "-") } diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 240e9d327d..58ca657dc4 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -87,9 +87,7 @@ impl ArgAssigner for Args { } /// Legalize `sig`. -pub fn legalize_signature(sig: &mut ir::Signature, - flags: &shared_settings::Flags, - _current: bool) { +pub fn legalize_signature(sig: &mut ir::Signature, flags: &shared_settings::Flags, _current: bool) { let bits = if flags.is_64bit() { 64 } else { 32 }; let mut args = Args::new(bits, &ARG_GPRS, 8); @@ -105,9 +103,10 @@ pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { } /// Get the set of allocatable registers for `func`. -pub fn allocatable_registers(_func: &ir::Function, - flags: &shared_settings::Flags) - -> AllocatableSet { +pub fn allocatable_registers( + _func: &ir::Function, + flags: &shared_settings::Flags, +) -> AllocatableSet { let mut regs = AllocatableSet::new(); regs.take(GPR, RU::rsp as RegUnit); regs.take(GPR, RU::rbp as RegUnit); diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 1cce45b12e..a8b6314521 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -29,19 +29,20 @@ pub fn isa_builder() -> IsaBuilder { } } -fn isa_constructor(shared_flags: shared_settings::Flags, - builder: &shared_settings::Builder) - -> Box { +fn isa_constructor( + shared_flags: shared_settings::Flags, + builder: &shared_settings::Builder, +) -> Box { let level1 = if shared_flags.is_64bit() { &enc_tables::LEVEL1_I64[..] } else { &enc_tables::LEVEL1_I32[..] }; Box::new(Isa { - isa_flags: settings::Flags::new(&shared_flags, builder), - shared_flags, - cpumode: level1, - }) + isa_flags: settings::Flags::new(&shared_flags, builder), + shared_flags, + cpumode: level1, + }) } impl TargetIsa for Isa { @@ -61,21 +62,24 @@ impl TargetIsa for Isa { enc_tables::INFO.clone() } - fn legal_encodings<'a>(&'a self, - dfg: &'a ir::DataFlowGraph, - inst: &'a ir::InstructionData, - ctrl_typevar: ir::Type) - -> Encodings<'a> { - lookup_enclist(ctrl_typevar, - inst, - dfg, - self.cpumode, - &enc_tables::LEVEL2[..], - &enc_tables::ENCLISTS[..], - &enc_tables::LEGALIZE_ACTIONS[..], - &enc_tables::RECIPE_PREDICATES[..], - &enc_tables::INST_PREDICATES[..], - self.isa_flags.predicate_view()) + fn legal_encodings<'a>( + &'a self, + dfg: &'a ir::DataFlowGraph, + inst: &'a ir::InstructionData, + ctrl_typevar: ir::Type, + ) -> Encodings<'a> { + lookup_enclist( + ctrl_typevar, + inst, + dfg, + self.cpumode, + &enc_tables::LEVEL2[..], + &enc_tables::ENCLISTS[..], + &enc_tables::LEGALIZE_ACTIONS[..], + &enc_tables::RECIPE_PREDICATES[..], + &enc_tables::INST_PREDICATES[..], + self.isa_flags.predicate_view(), + ) } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { @@ -90,11 +94,13 @@ impl TargetIsa for Isa { abi::allocatable_registers(func, &self.shared_flags) } - fn emit_inst(&self, - func: &ir::Function, - inst: ir::Inst, - divert: &mut regalloc::RegDiversions, - sink: &mut CodeSink) { + fn emit_inst( + &self, + func: &ir::Function, + inst: ir::Inst, + divert: &mut regalloc::RegDiversions, + sink: &mut CodeSink, + ) { binemit::emit_inst(func, inst, divert, sink) } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 28a7c9d615..b68cd17041 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -155,11 +155,12 @@ pub trait TargetIsa { fn register_info(&self) -> RegInfo; /// Returns an iterartor over legal encodings for the instruction. - fn legal_encodings<'a>(&'a self, - dfg: &'a ir::DataFlowGraph, - inst: &'a ir::InstructionData, - ctrl_typevar: ir::Type) - -> Encodings<'a>; + fn legal_encodings<'a>( + &'a self, + dfg: &'a ir::DataFlowGraph, + inst: &'a ir::InstructionData, + ctrl_typevar: ir::Type, + ) -> Encodings<'a>; /// Encode an instruction after determining it is legal. /// @@ -167,11 +168,12 @@ pub trait TargetIsa { /// Otherwise, return `Legalize` action. /// /// This is also the main entry point for determining if an instruction is legal. - fn encode(&self, - dfg: &ir::DataFlowGraph, - inst: &ir::InstructionData, - ctrl_typevar: ir::Type) - -> Result { + fn encode( + &self, + dfg: &ir::DataFlowGraph, + inst: &ir::InstructionData, + ctrl_typevar: ir::Type, + ) -> Result { let mut iter = self.legal_encodings(dfg, inst, ctrl_typevar); iter.next().ok_or_else(|| iter.legalize().into()) } @@ -244,11 +246,13 @@ pub trait TargetIsa { /// /// Note that this will call `put*` methods on the trait object via its vtable which is not the /// fastest way of emitting code. - fn emit_inst(&self, - func: &ir::Function, - inst: ir::Inst, - divert: &mut regalloc::RegDiversions, - sink: &mut binemit::CodeSink); + fn emit_inst( + &self, + func: &ir::Function, + inst: ir::Inst, + divert: &mut regalloc::RegDiversions, + sink: &mut binemit::CodeSink, + ); /// Emit a whole function into memory. /// diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index eacce024fb..e156de331e 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -74,24 +74,23 @@ impl RegBank { /// Try to parse a regunit name. The name is not expected to begin with `%`. fn parse_regunit(&self, name: &str) -> Option { match self.names.iter().position(|&x| x == name) { - Some(offset) => { - // This is one of the special-cased names. - Some(offset as RegUnit) - } - None => { - // Try a regular prefixed name. - if name.starts_with(self.prefix) { - name[self.prefix.len()..].parse().ok() - } else { - None - } + Some(offset) => { + // This is one of the special-cased names. + Some(offset as RegUnit) + } + None => { + // Try a regular prefixed name. + if name.starts_with(self.prefix) { + name[self.prefix.len()..].parse().ok() + } else { + None } } - .and_then(|offset| if offset < self.units { - Some(offset + self.first_unit) - } else { - None - }) + }.and_then(|offset| if offset < self.units { + Some(offset + self.first_unit) + } else { + None + }) } /// Write `regunit` to `w`, assuming that it belongs to this bank. diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index 66d4c2e6b7..f4ab75fde9 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -86,10 +86,12 @@ impl ArgAssigner for Args { } /// Legalize `sig` for RISC-V. -pub fn legalize_signature(sig: &mut ir::Signature, - flags: &shared_settings::Flags, - isa_flags: &settings::Flags, - current: bool) { +pub fn legalize_signature( + sig: &mut ir::Signature, + flags: &shared_settings::Flags, + isa_flags: &settings::Flags, + current: bool, +) { let bits = if flags.is_64bit() { 64 } else { 32 }; let mut args = Args::new(bits, isa_flags.enable_e()); diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 86cbb6b82c..7a658267b9 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -29,11 +29,7 @@ impl Into for RelocKind { /// 25 20 15 12 7 0 /// /// Encoding bits: `opcode[6:2] | (funct3 << 5) | (funct7 << 8)`. -fn put_r(bits: u16, - rs1: RegUnit, - rs2: RegUnit, - rd: RegUnit, - sink: &mut CS) { +fn put_r(bits: u16, rs1: RegUnit, rs2: RegUnit, rd: RegUnit, sink: &mut CS) { let bits = bits as u32; let opcode5 = bits & 0x1f; let funct3 = (bits >> 5) & 0x7; @@ -63,11 +59,13 @@ fn put_r(bits: u16, /// Both funct7 and shamt contribute to bit 25. In RV64, shamt uses it for shifts > 31. /// /// Encoding bits: `opcode[6:2] | (funct3 << 5) | (funct7 << 8)`. -fn put_rshamt(bits: u16, - rs1: RegUnit, - shamt: i64, - rd: RegUnit, - sink: &mut CS) { +fn put_rshamt( + bits: u16, + rs1: RegUnit, + shamt: i64, + rd: RegUnit, + sink: &mut CS, +) { let bits = bits as u32; let opcode5 = bits & 0x1f; let funct3 = (bits >> 5) & 0x7; diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 03482b920c..e6841b7299 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -29,19 +29,20 @@ pub fn isa_builder() -> IsaBuilder { } } -fn isa_constructor(shared_flags: shared_settings::Flags, - builder: &shared_settings::Builder) - -> Box { +fn isa_constructor( + shared_flags: shared_settings::Flags, + builder: &shared_settings::Builder, +) -> Box { let level1 = if shared_flags.is_64bit() { &enc_tables::LEVEL1_RV64[..] } else { &enc_tables::LEVEL1_RV32[..] }; Box::new(Isa { - isa_flags: settings::Flags::new(&shared_flags, builder), - shared_flags, - cpumode: level1, - }) + isa_flags: settings::Flags::new(&shared_flags, builder), + shared_flags, + cpumode: level1, + }) } impl TargetIsa for Isa { @@ -61,21 +62,24 @@ impl TargetIsa for Isa { enc_tables::INFO.clone() } - fn legal_encodings<'a>(&'a self, - dfg: &'a ir::DataFlowGraph, - inst: &'a ir::InstructionData, - ctrl_typevar: ir::Type) - -> Encodings<'a> { - lookup_enclist(ctrl_typevar, - inst, - dfg, - self.cpumode, - &enc_tables::LEVEL2[..], - &enc_tables::ENCLISTS[..], - &enc_tables::LEGALIZE_ACTIONS[..], - &enc_tables::RECIPE_PREDICATES[..], - &enc_tables::INST_PREDICATES[..], - self.isa_flags.predicate_view()) + fn legal_encodings<'a>( + &'a self, + dfg: &'a ir::DataFlowGraph, + inst: &'a ir::InstructionData, + ctrl_typevar: ir::Type, + ) -> Encodings<'a> { + lookup_enclist( + ctrl_typevar, + inst, + dfg, + self.cpumode, + &enc_tables::LEVEL2[..], + &enc_tables::ENCLISTS[..], + &enc_tables::LEGALIZE_ACTIONS[..], + &enc_tables::RECIPE_PREDICATES[..], + &enc_tables::INST_PREDICATES[..], + self.isa_flags.predicate_view(), + ) } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { @@ -90,11 +94,13 @@ impl TargetIsa for Isa { abi::allocatable_registers(func, &self.isa_flags) } - fn emit_inst(&self, - func: &ir::Function, - inst: ir::Inst, - divert: &mut regalloc::RegDiversions, - sink: &mut CodeSink) { + fn emit_inst( + &self, + func: &ir::Function, + inst: ir::Inst, + divert: &mut regalloc::RegDiversions, + sink: &mut CodeSink, + ) { binemit::emit_inst(func, inst, divert, sink) } diff --git a/lib/cretonne/src/isa/riscv/settings.rs b/lib/cretonne/src/isa/riscv/settings.rs index 69a47f8717..8cb376e5da 100644 --- a/lib/cretonne/src/isa/riscv/settings.rs +++ b/lib/cretonne/src/isa/riscv/settings.rs @@ -18,14 +18,16 @@ mod tests { let shared = settings::Flags::new(&settings::builder()); let b = builder(); let f = Flags::new(&shared, &b); - assert_eq!(f.to_string(), - "[riscv]\n\ + assert_eq!( + f.to_string(), + "[riscv]\n\ supports_m = false\n\ supports_a = false\n\ supports_f = false\n\ supports_d = false\n\ enable_m = true\n\ - enable_e = false\n"); + enable_e = false\n" + ); // Predicates are not part of the Display output. assert_eq!(f.full_float(), false); } diff --git a/lib/cretonne/src/iterators.rs b/lib/cretonne/src/iterators.rs index 70d1165ed7..08866717e8 100644 --- a/lib/cretonne/src/iterators.rs +++ b/lib/cretonne/src/iterators.rs @@ -4,40 +4,45 @@ pub trait IteratorExtras: Iterator { /// Create an iterator that produces adjacent pairs of elements from the iterator. fn adjacent_pairs(mut self) -> AdjacentPairs - where Self: Sized, - Self::Item: Clone + where + Self: Sized, + Self::Item: Clone, { let elem = self.next(); AdjacentPairs { iter: self, elem } } } -impl IteratorExtras for T where T: Iterator {} +impl IteratorExtras for T +where + T: Iterator, +{ +} /// Adjacent pairs iterator returned by `adjacent_pairs()`. /// /// This wraps another iterator and produces a sequence of adjacent pairs of elements. pub struct AdjacentPairs - where I: Iterator, - I::Item: Clone +where + I: Iterator, + I::Item: Clone, { iter: I, elem: Option, } impl Iterator for AdjacentPairs - where I: Iterator, - I::Item: Clone +where + I: Iterator, + I::Item: Clone, { type Item = (I::Item, I::Item); fn next(&mut self) -> Option { - self.elem - .take() - .and_then(|e| { - self.elem = self.iter.next(); - self.elem.clone().map(|n| (e, n)) - }) + self.elem.take().and_then(|e| { + self.elem = self.iter.next(); + self.elem.clone().map(|n| (e, n)) + }) } } @@ -47,33 +52,45 @@ mod tests { fn adjpairs() { use super::IteratorExtras; - assert_eq!([1, 2, 3, 4] - .iter() - .cloned() - .adjacent_pairs() - .collect::>(), - vec![(1, 2), (2, 3), (3, 4)]); - assert_eq!([2, 3, 4] - .iter() - .cloned() - .adjacent_pairs() - .collect::>(), - vec![(2, 3), (3, 4)]); - assert_eq!([2, 3, 4] - .iter() - .cloned() - .adjacent_pairs() - .collect::>(), - vec![(2, 3), (3, 4)]); - assert_eq!([3, 4].iter().cloned().adjacent_pairs().collect::>(), - vec![(3, 4)]); - assert_eq!([4].iter().cloned().adjacent_pairs().collect::>(), - vec![]); - assert_eq!([] - .iter() - .cloned() - .adjacent_pairs() - .collect::>(), - vec![]); + assert_eq!( + [1, 2, 3, 4] + .iter() + .cloned() + .adjacent_pairs() + .collect::>(), + vec![(1, 2), (2, 3), (3, 4)] + ); + assert_eq!( + [2, 3, 4] + .iter() + .cloned() + .adjacent_pairs() + .collect::>(), + vec![(2, 3), (3, 4)] + ); + assert_eq!( + [2, 3, 4] + .iter() + .cloned() + .adjacent_pairs() + .collect::>(), + vec![(2, 3), (3, 4)] + ); + assert_eq!( + [3, 4].iter().cloned().adjacent_pairs().collect::>(), + vec![(3, 4)] + ); + assert_eq!( + [4].iter().cloned().adjacent_pairs().collect::>(), + vec![] + ); + assert_eq!( + [] + .iter() + .cloned() + .adjacent_pairs() + .collect::>(), + vec![] + ); } } diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 71c79ef46b..a5442ba8be 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -98,9 +98,11 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { // Compute the value we want for `arg` from the legalized ABI arguments. let mut get_arg = |dfg: &mut DataFlowGraph, ty| { let abi_type = abi_types[abi_arg]; - assert_eq!(abi_type.purpose, - ArgumentPurpose::Normal, - "Can't legalize special-purpose argument"); + assert_eq!( + abi_type.purpose, + ArgumentPurpose::Normal, + "Can't legalize special-purpose argument" + ); if ty == abi_type.value_type { abi_arg += 1; Ok(dfg.append_ebb_arg(entry, ty)) @@ -159,14 +161,17 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { /// This function is very similar to the `legalize_entry_arguments` function above. /// /// Returns the possibly new instruction representing the call. -fn legalize_inst_results(dfg: &mut DataFlowGraph, - pos: &mut Cursor, - mut get_abi_type: ResType) - -> Inst - where ResType: FnMut(&DataFlowGraph, usize) -> ArgumentType +fn legalize_inst_results( + dfg: &mut DataFlowGraph, + pos: &mut Cursor, + mut get_abi_type: ResType, +) -> Inst +where + ResType: FnMut(&DataFlowGraph, usize) -> ArgumentType, { - let call = pos.current_inst() - .expect("Cursor must point to a call instruction"); + let call = pos.current_inst().expect( + "Cursor must point to a call instruction", + ); // We theoretically allow for call instructions that return a number of fixed results before // the call return values. In practice, it doesn't happen. @@ -216,13 +221,15 @@ fn legalize_inst_results(dfg: &mut DataFlowGraph, /// - `Err(arg_type)` if further conversions are needed from the ABI argument `arg_type`. /// /// If the `into_result` value is provided, the converted result will be written into that value. -fn convert_from_abi(dfg: &mut DataFlowGraph, - pos: &mut Cursor, - ty: Type, - into_result: Option, - get_arg: &mut GetArg) - -> Value - where GetArg: FnMut(&mut DataFlowGraph, Type) -> Result +fn convert_from_abi( + dfg: &mut DataFlowGraph, + pos: &mut Cursor, + ty: Type, + into_result: Option, + get_arg: &mut GetArg, +) -> Value +where + GetArg: FnMut(&mut DataFlowGraph, Type) -> Result, { // Terminate the recursion when we get the desired type. let arg_type = match get_arg(dfg, ty) { @@ -246,11 +253,13 @@ fn convert_from_abi(dfg: &mut DataFlowGraph, let abi_ty = ty.half_width().expect("Invalid type for conversion"); let lo = convert_from_abi(dfg, pos, abi_ty, None, get_arg); let hi = convert_from_abi(dfg, pos, abi_ty, None, get_arg); - dbg!("intsplit {}: {}, {}: {}", - lo, - dfg.value_type(lo), - hi, - dfg.value_type(hi)); + dbg!( + "intsplit {}: {}, {}: {}", + lo, + dfg.value_type(lo), + hi, + dfg.value_type(hi) + ); dfg.ins(pos).with_results([into_result]).iconcat(lo, hi) } // Construct a `ty` by concatenating two halves of a vector. @@ -296,12 +305,14 @@ fn convert_from_abi(dfg: &mut DataFlowGraph, /// 2. If the suggested argument doesn't have the right value type, don't change anything, but /// return the `Err(ArgumentType)` that is needed. /// -fn convert_to_abi(dfg: &mut DataFlowGraph, - cfg: &ControlFlowGraph, - pos: &mut Cursor, - value: Value, - put_arg: &mut PutArg) - where PutArg: FnMut(&mut DataFlowGraph, Value) -> Result<(), ArgumentType> +fn convert_to_abi( + dfg: &mut DataFlowGraph, + cfg: &ControlFlowGraph, + pos: &mut Cursor, + value: Value, + put_arg: &mut PutArg, +) where + PutArg: FnMut(&mut DataFlowGraph, Value) -> Result<(), ArgumentType>, { // Start by invoking the closure to either terminate the recursion or get the argument type // we're trying to match. @@ -360,7 +371,8 @@ fn check_call_signature(dfg: &DataFlowGraph, inst: Inst) -> Result<(), SigRef> { let sig = &dfg.signatures[sig_ref]; if check_arg_types(dfg, args, &sig.argument_types[..]) && - check_arg_types(dfg, dfg.inst_results(inst), &sig.return_types[..]) { + check_arg_types(dfg, dfg.inst_results(inst), &sig.return_types[..]) + { // All types check out. Ok(()) } else { @@ -380,20 +392,23 @@ fn check_return_signature(dfg: &DataFlowGraph, inst: Inst, sig: &Signature) -> b /// - `get_abi_type` is a closure that can provide the desired `ArgumentType` for a given ABI /// argument number in `0..abi_args`. /// -fn legalize_inst_arguments(dfg: &mut DataFlowGraph, - cfg: &ControlFlowGraph, - pos: &mut Cursor, - abi_args: usize, - mut get_abi_type: ArgType) - where ArgType: FnMut(&DataFlowGraph, usize) -> ArgumentType +fn legalize_inst_arguments( + dfg: &mut DataFlowGraph, + cfg: &ControlFlowGraph, + pos: &mut Cursor, + abi_args: usize, + mut get_abi_type: ArgType, +) where + ArgType: FnMut(&DataFlowGraph, usize) -> ArgumentType, { - let inst = pos.current_inst() - .expect("Cursor must point to a call instruction"); + let inst = pos.current_inst().expect( + "Cursor must point to a call instruction", + ); // Lift the value list out of the call instruction so we modify it. - let mut vlist = dfg[inst] - .take_value_list() - .expect("Call must have a value list"); + let mut vlist = dfg[inst].take_value_list().expect( + "Call must have a value list", + ); // The value list contains all arguments to the instruction, including the callee on an // indirect call which isn't part of the call arguments that must match the ABI signature. @@ -474,23 +489,23 @@ pub fn handle_call_abi(mut inst: Inst, func: &mut Function, cfg: &ControlFlowGra // OK, we need to fix the call arguments to match the ABI signature. let abi_args = dfg.signatures[sig_ref].argument_types.len(); - legalize_inst_arguments(dfg, - cfg, - pos, - abi_args, - |dfg, abi_arg| dfg.signatures[sig_ref].argument_types[abi_arg]); + legalize_inst_arguments(dfg, cfg, pos, abi_args, |dfg, abi_arg| { + dfg.signatures[sig_ref].argument_types[abi_arg] + }); if !dfg.signatures[sig_ref].return_types.is_empty() { - inst = legalize_inst_results(dfg, - pos, - |dfg, abi_res| dfg.signatures[sig_ref].return_types[abi_res]); + inst = legalize_inst_results(dfg, pos, |dfg, abi_res| { + dfg.signatures[sig_ref].return_types[abi_res] + }); } - debug_assert!(check_call_signature(dfg, inst).is_ok(), - "Signature still wrong: {}, {}{}", - dfg.display_inst(inst, None), - sig_ref, - dfg.signatures[sig_ref]); + debug_assert!( + check_call_signature(dfg, inst).is_ok(), + "Signature still wrong: {}, {}{}", + dfg.display_inst(inst, None), + sig_ref, + dfg.signatures[sig_ref] + ); // Go back and insert spills for any stack arguments. pos.goto_inst(inst); @@ -519,27 +534,30 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph .iter() .rev() .take_while(|&rt| { - rt.purpose == ArgumentPurpose::Link || - rt.purpose == ArgumentPurpose::StructReturn || - rt.purpose == ArgumentPurpose::VMContext - }) + rt.purpose == ArgumentPurpose::Link || rt.purpose == ArgumentPurpose::StructReturn || + rt.purpose == ArgumentPurpose::VMContext + }) .count(); let abi_args = sig.return_types.len() - special_args; - legalize_inst_arguments(dfg, - cfg, - pos, - abi_args, - |_, abi_arg| sig.return_types[abi_arg]); + legalize_inst_arguments( + dfg, + cfg, + pos, + abi_args, + |_, abi_arg| sig.return_types[abi_arg], + ); assert_eq!(dfg.inst_variable_args(inst).len(), abi_args); // Append special return arguments for any `sret`, `link`, and `vmctx` return values added to // the legalized signature. These values should simply be propagated from the entry block // arguments. if special_args > 0 { - dbg!("Adding {} special-purpose arguments to {}", - special_args, - dfg.display_inst(inst, None)); + dbg!( + "Adding {} special-purpose arguments to {}", + special_args, + dfg.display_inst(inst, None) + ); let mut vlist = dfg[inst].take_value_list().unwrap(); for arg in &sig.return_types[abi_args..] { match arg.purpose { @@ -565,10 +583,12 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph dfg[inst].put_value_list(vlist); } - debug_assert!(check_return_signature(dfg, inst, sig), - "Signature still wrong: {} / signature {}", - dfg.display_inst(inst, None), - sig); + debug_assert!( + check_return_signature(dfg, inst, sig), + "Signature still wrong: {} / signature {}", + dfg.display_inst(inst, None), + sig + ); // Yes, we changed stuff. true @@ -579,10 +599,10 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph /// Values that are passed into the function on the stack must be assigned to an `IncomingArg` /// stack slot already during legalization. fn spill_entry_arguments(func: &mut Function, entry: Ebb) { - for (abi, &arg) in func.signature - .argument_types - .iter() - .zip(func.dfg.ebb_args(entry)) { + for (abi, &arg) in func.signature.argument_types.iter().zip( + func.dfg.ebb_args(entry), + ) + { if let ArgumentLoc::Stack(offset) = abi.location { let ss = func.stack_slots.make_incoming_arg(abi.value_type, offset); func.locations[arg] = ValueLoc::Stack(ss); @@ -598,15 +618,18 @@ fn spill_entry_arguments(func: &mut Function, entry: Ebb) { /// TODO: The outgoing stack slots can be written a bit earlier, as long as there are no branches /// or calls between writing the stack slots and the call instruction. Writing the slots earlier /// could help reduce register pressure before the call. -fn spill_call_arguments(dfg: &mut DataFlowGraph, - locations: &mut ValueLocations, - stack_slots: &mut StackSlots, - pos: &mut Cursor) - -> bool { - let inst = pos.current_inst() - .expect("Cursor must point to a call instruction"); - let sig_ref = dfg.call_signature(inst) - .expect("Call instruction expected."); +fn spill_call_arguments( + dfg: &mut DataFlowGraph, + locations: &mut ValueLocations, + stack_slots: &mut StackSlots, + pos: &mut Cursor, +) -> bool { + let inst = pos.current_inst().expect( + "Cursor must point to a call instruction", + ); + let sig_ref = dfg.call_signature(inst).expect( + "Call instruction expected.", + ); // Start by building a list of stack slots and arguments to be replaced. // This requires borrowing `dfg`, so we can't change anything. diff --git a/lib/cretonne/src/legalizer/heap.rs b/lib/cretonne/src/legalizer/heap.rs index 30cd0917c7..ac68044a4a 100644 --- a/lib/cretonne/src/legalizer/heap.rs +++ b/lib/cretonne/src/legalizer/heap.rs @@ -35,12 +35,14 @@ pub fn expand_heap_addr(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut Cont } /// Expand a `heap_addr` for a dynamic heap. -fn dynamic_addr(inst: ir::Inst, - heap: ir::Heap, - offset: ir::Value, - size: u32, - bound_gv: ir::GlobalVar, - func: &mut ir::Function) { +fn dynamic_addr( + inst: ir::Inst, + heap: ir::Heap, + offset: ir::Value, + size: u32, + bound_gv: ir::GlobalVar, + func: &mut ir::Function, +) { let size = size as i64; let offset_ty = func.dfg.value_type(offset); let addr_ty = func.dfg.value_type(func.dfg.first_result(inst)); @@ -54,21 +56,30 @@ fn dynamic_addr(inst: ir::Inst, let oob; if size == 1 { // `offset > bound - 1` is the same as `offset >= bound`. - oob = pos.ins() - .icmp(IntCC::UnsignedGreaterThanOrEqual, offset, bound); + oob = pos.ins().icmp( + IntCC::UnsignedGreaterThanOrEqual, + offset, + bound, + ); } else if size <= min_size { // We know that bound >= min_size, so here we can compare `offset > bound - size` without // wrapping. let adj_bound = pos.ins().iadd_imm(bound, -size); - oob = pos.ins() - .icmp(IntCC::UnsignedGreaterThan, offset, adj_bound); + oob = pos.ins().icmp( + IntCC::UnsignedGreaterThan, + offset, + adj_bound, + ); } else { // We need an overflow check for the adjusted offset. let size_val = pos.ins().iconst(offset_ty, size); let (adj_offset, overflow) = pos.ins().iadd_cout(offset, size_val); pos.ins().trapnz(overflow); - oob = pos.ins() - .icmp(IntCC::UnsignedGreaterThan, adj_offset, bound); + oob = pos.ins().icmp( + IntCC::UnsignedGreaterThan, + adj_offset, + bound, + ); } pos.ins().trapnz(oob); @@ -76,12 +87,14 @@ fn dynamic_addr(inst: ir::Inst, } /// Expand a `heap_addr` for a static heap. -fn static_addr(inst: ir::Inst, - heap: ir::Heap, - offset: ir::Value, - size: u32, - bound: i64, - func: &mut ir::Function) { +fn static_addr( + inst: ir::Inst, + heap: ir::Heap, + offset: ir::Value, + size: u32, + bound: i64, + func: &mut ir::Function, +) { let size = size as i64; let offset_ty = func.dfg.value_type(offset); let addr_ty = func.dfg.value_type(func.dfg.first_result(inst)); @@ -104,11 +117,17 @@ fn static_addr(inst: ir::Inst, let oob = if limit & 1 == 1 { // Prefer testing `offset >= limit - 1` when limit is odd because an even number is // likely to be a convenient constant on ARM and other RISC architectures. - pos.ins() - .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, offset, limit - 1) + pos.ins().icmp_imm( + IntCC::UnsignedGreaterThanOrEqual, + offset, + limit - 1, + ) } else { - pos.ins() - .icmp_imm(IntCC::UnsignedGreaterThan, offset, limit) + pos.ins().icmp_imm( + IntCC::UnsignedGreaterThan, + offset, + limit, + ) }; pos.ins().trapnz(oob); } @@ -119,12 +138,14 @@ fn static_addr(inst: ir::Inst, /// Emit code for the base address computation of a `heap_addr` instruction. /// /// -fn offset_addr(inst: ir::Inst, - heap: ir::Heap, - addr_ty: ir::Type, - mut offset: ir::Value, - offset_ty: ir::Type, - func: &mut ir::Function) { +fn offset_addr( + inst: ir::Inst, + heap: ir::Heap, + addr_ty: ir::Type, + mut offset: ir::Value, + offset_ty: ir::Type, + func: &mut ir::Function, +) { let mut pos = FuncCursor::new(func).at_inst(inst); // Convert `offset` to `addr_ty`. diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index da4a2b93fa..179c0d3884 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -66,9 +66,11 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is split::simplify_branch_arguments(&mut pos.func.dfg, inst); } - match isa.encode(&pos.func.dfg, - &pos.func.dfg[inst], - pos.func.dfg.ctrl_typevar(inst)) { + match isa.encode( + &pos.func.dfg, + &pos.func.dfg[inst], + pos.func.dfg.ctrl_typevar(inst), + ) { Ok(encoding) => pos.func.encodings[inst] = encoding, Err(action) => { // We should transform the instruction into legal equivalents. diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index da61303e05..b225586842 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -71,21 +71,23 @@ use std::iter; /// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values /// if possible. -pub fn isplit(dfg: &mut DataFlowGraph, - cfg: &ControlFlowGraph, - pos: &mut Cursor, - value: Value) - -> (Value, Value) { +pub fn isplit( + dfg: &mut DataFlowGraph, + cfg: &ControlFlowGraph, + pos: &mut Cursor, + value: Value, +) -> (Value, Value) { split_any(dfg, cfg, pos, value, Opcode::Iconcat) } /// Split `value` into halves using the `vsplit` semantics. Do this by reusing existing values if /// possible. -pub fn vsplit(dfg: &mut DataFlowGraph, - cfg: &ControlFlowGraph, - pos: &mut Cursor, - value: Value) - -> (Value, Value) { +pub fn vsplit( + dfg: &mut DataFlowGraph, + cfg: &ControlFlowGraph, + pos: &mut Cursor, + value: Value, +) -> (Value, Value) { split_any(dfg, cfg, pos, value, Opcode::Vconcat) } @@ -107,12 +109,13 @@ struct Repair { } /// Generic version of `isplit` and `vsplit` controlled by the `concat` opcode. -fn split_any(dfg: &mut DataFlowGraph, - cfg: &ControlFlowGraph, - pos: &mut Cursor, - value: Value, - concat: Opcode) - -> (Value, Value) { +fn split_any( + dfg: &mut DataFlowGraph, + cfg: &ControlFlowGraph, + pos: &mut Cursor, + value: Value, + concat: Opcode, +) -> (Value, Value) { let saved_pos = pos.position(); let mut repairs = Vec::new(); let result = split_value(dfg, pos, value, concat, &mut repairs); @@ -121,17 +124,20 @@ fn split_any(dfg: &mut DataFlowGraph, while let Some(repair) = repairs.pop() { for &(_, inst) in cfg.get_predecessors(repair.ebb) { let branch_opc = dfg[inst].opcode(); - assert!(branch_opc.is_branch(), - "Predecessor not a branch: {}", - dfg.display_inst(inst, None)); + assert!( + branch_opc.is_branch(), + "Predecessor not a branch: {}", + dfg.display_inst(inst, None) + ); let fixed_args = branch_opc.constraints().fixed_value_arguments(); - let mut args = dfg[inst] - .take_value_list() - .expect("Branches must have value lists."); + let mut args = dfg[inst].take_value_list().expect( + "Branches must have value lists.", + ); let num_args = args.len(&dfg.value_lists); // Get the old value passed to the EBB argument we're repairing. - let old_arg = args.get(fixed_args + repair.num, &dfg.value_lists) - .expect("Too few branch arguments"); + let old_arg = args.get(fixed_args + repair.num, &dfg.value_lists).expect( + "Too few branch arguments", + ); // It's possible that the CFG's predecessor list has duplicates. Detect them here. if dfg.value_type(old_arg) == repair.split_type { @@ -145,19 +151,21 @@ fn split_any(dfg: &mut DataFlowGraph, // The `lo` part replaces the original argument. *args.get_mut(fixed_args + repair.num, &mut dfg.value_lists) - .unwrap() = lo; + .unwrap() = lo; // The `hi` part goes at the end. Since multiple repairs may have been scheduled to the // same EBB, there could be multiple arguments missing. if num_args > fixed_args + repair.hi_num { *args.get_mut(fixed_args + repair.hi_num, &mut dfg.value_lists) - .unwrap() = hi; + .unwrap() = hi; } else { // We need to append one or more arguments. If we're adding more than one argument, // there must be pending repairs on the stack that will fill in the correct values // instead of `hi`. - args.extend(iter::repeat(hi).take(1 + fixed_args + repair.hi_num - num_args), - &mut dfg.value_lists); + args.extend( + iter::repeat(hi).take(1 + fixed_args + repair.hi_num - num_args), + &mut dfg.value_lists, + ); } // Put the value list back after manipulating it. @@ -175,12 +183,13 @@ fn split_any(dfg: &mut DataFlowGraph, /// instruction. /// /// Return the two new values representing the parts of `value`. -fn split_value(dfg: &mut DataFlowGraph, - pos: &mut Cursor, - value: Value, - concat: Opcode, - repairs: &mut Vec) - -> (Value, Value) { +fn split_value( + dfg: &mut DataFlowGraph, + pos: &mut Cursor, + value: Value, + concat: Opcode, + repairs: &mut Vec, +) -> (Value, Value) { let value = dfg.resolve_copies(value); let mut reuse = None; @@ -228,9 +237,12 @@ fn split_value(dfg: &mut DataFlowGraph, // need to insert a split instruction before returning. pos.goto_top(ebb); pos.next_inst(); - dfg.ins(pos) - .with_result(value) - .Binary(concat, split_type, lo, hi); + dfg.ins(pos).with_result(value).Binary( + concat, + split_type, + lo, + hi, + ); // Finally, splitting the EBB argument is not enough. We also have to repair all // of the predecessor instructions that branch here. @@ -254,19 +266,21 @@ fn split_value(dfg: &mut DataFlowGraph, } // Add a repair entry to the work list. -fn add_repair(concat: Opcode, - split_type: Type, - ebb: Ebb, - num: usize, - hi_num: usize, - repairs: &mut Vec) { +fn add_repair( + concat: Opcode, + split_type: Type, + ebb: Ebb, + num: usize, + hi_num: usize, + repairs: &mut Vec, +) { repairs.push(Repair { - concat, - split_type, - ebb, - num, - hi_num, - }); + concat, + split_type, + ebb, + num, + hi_num, + }); } /// Strip concat-split chains. Return a simpler way of computing the same value. diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 5cd7b22280..689067c188 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -10,10 +10,12 @@ use loop_analysis::{Loop, LoopAnalysis}; /// Performs the LICM pass by detecting loops within the CFG and moving /// loop-invariant instructions out of them. /// Changes the CFG and domtree in-place during the operation. -pub fn do_licm(func: &mut Function, - cfg: &mut ControlFlowGraph, - domtree: &mut DominatorTree, - loop_analysis: &mut LoopAnalysis) { +pub fn do_licm( + func: &mut Function, + cfg: &mut ControlFlowGraph, + domtree: &mut DominatorTree, + loop_analysis: &mut LoopAnalysis, +) { loop_analysis.compute(func, cfg, domtree); for lp in loop_analysis.loops() { // For each loop that we want to optimize we determine the set of loop-invariant @@ -53,11 +55,12 @@ pub fn do_licm(func: &mut Function, // Insert a pre-header before the header, modifying the function layout and CFG to reflect it. // A jump instruction to the header is placed at the end of the pre-header. -fn create_pre_header(header: Ebb, - func: &mut Function, - cfg: &mut ControlFlowGraph, - domtree: &DominatorTree) - -> Ebb { +fn create_pre_header( + header: Ebb, + func: &mut Function, + cfg: &mut ControlFlowGraph, + domtree: &DominatorTree, +) -> Ebb { let pool = &mut ListPool::::new(); let header_args_values: Vec = func.dfg.ebb_args(header).into_iter().cloned().collect(); let header_args_types: Vec = header_args_values @@ -82,9 +85,10 @@ fn create_pre_header(header: Ebb, // Inserts the pre-header at the right place in the layout. pos.insert_ebb(pre_header); pos.next_inst(); - func.dfg - .ins(&mut pos) - .jump(header, pre_header_args_value.as_slice(pool)); + func.dfg.ins(&mut pos).jump( + header, + pre_header_args_value.as_slice(pool), + ); } pre_header } @@ -94,11 +98,12 @@ fn create_pre_header(header: Ebb, // A loop header has a pre-header if there is only one predecessor that the header doesn't // dominate. // Returns the pre-header Ebb and the instruction jumping to the header. -fn has_pre_header(layout: &Layout, - cfg: &ControlFlowGraph, - domtree: &DominatorTree, - header: Ebb) - -> Option<(Ebb, Inst)> { +fn has_pre_header( + layout: &Layout, + cfg: &ControlFlowGraph, + domtree: &DominatorTree, + header: Ebb, +) -> Option<(Ebb, Inst)> { let mut result = None; let mut found = false; for &(pred_ebb, last_inst) in cfg.get_predecessors(header) { @@ -129,11 +134,12 @@ fn change_branch_jump_destination(inst: Inst, new_ebb: Ebb, func: &mut Function) // Traverses a loop in reverse post-order from a header EBB and identify loop-invariant // instructions. These loop-invariant instructions are then removed from the code and returned // (in reverse post-order) for later use. -fn remove_loop_invariant_instructions(lp: Loop, - func: &mut Function, - cfg: &ControlFlowGraph, - loop_analysis: &LoopAnalysis) - -> Vec { +fn remove_loop_invariant_instructions( + lp: Loop, + func: &mut Function, + cfg: &ControlFlowGraph, + loop_analysis: &LoopAnalysis, +) -> Vec { let mut loop_values: HashSet = HashSet::new(); let mut invariant_inst: Vec = Vec::new(); let mut pos = Cursor::new(&mut func.layout); @@ -146,10 +152,10 @@ fn remove_loop_invariant_instructions(lp: Loop, pos.goto_top(*ebb); while let Some(inst) = pos.next_inst() { if func.dfg.has_results(inst) && - func.dfg - .inst_args(inst) - .into_iter() - .all(|arg| !loop_values.contains(arg)) { + func.dfg.inst_args(inst).into_iter().all(|arg| { + !loop_values.contains(arg) + }) + { // If all the instruction's argument are defined outside the loop // then this instruction is loop-invariant invariant_inst.push(inst); diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index 55ddf1f4df..4c87e773a7 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -105,10 +105,12 @@ impl LoopAnalysis { // Traverses the CFG in reverse postorder and create a loop object for every EBB having a // back edge. - fn find_loop_headers(&mut self, - cfg: &ControlFlowGraph, - domtree: &DominatorTree, - layout: &Layout) { + fn find_loop_headers( + &mut self, + cfg: &ControlFlowGraph, + domtree: &DominatorTree, + layout: &Layout, + ) { // We traverse the CFG in reverse postorder for &ebb in domtree.cfg_postorder().iter().rev() { for &(_, pred_inst) in cfg.get_predecessors(ebb) { @@ -127,10 +129,12 @@ impl LoopAnalysis { // Intended to be called after `find_loop_headers`. For each detected loop header, // discovers all the ebb belonging to the loop and its inner loops. After a call to this // function, the loop tree is fully constructed. - fn discover_loop_blocks(&mut self, - cfg: &ControlFlowGraph, - domtree: &DominatorTree, - layout: &Layout) { + fn discover_loop_blocks( + &mut self, + cfg: &ControlFlowGraph, + domtree: &DominatorTree, + layout: &Layout, + ) { let mut stack: Vec = Vec::new(); // We handle each loop header in reverse order, corresponding to a pesudo postorder // traversal of the graph. diff --git a/lib/cretonne/src/packed_option.rs b/lib/cretonne/src/packed_option.rs index 9082a4846f..f92e821f72 100644 --- a/lib/cretonne/src/packed_option.rs +++ b/lib/cretonne/src/packed_option.rs @@ -38,7 +38,8 @@ impl PackedOption { /// Maps a `PackedOption` to `Option` by applying a function to a contained value. pub fn map(self, f: F) -> Option - where F: FnOnce(T) -> U + where + F: FnOnce(T) -> U, { self.expand().map(f) } @@ -69,8 +70,10 @@ impl Default for PackedOption { impl From for PackedOption { /// Convert `t` into a packed `Some(x)`. fn from(t: T) -> PackedOption { - debug_assert!(t != T::reserved_value(), - "Can't make a PackedOption from the reserved value."); + debug_assert!( + t != T::reserved_value(), + "Can't make a PackedOption from the reserved value." + ); PackedOption(t) } } @@ -92,7 +95,8 @@ impl Into> for PackedOption { } impl fmt::Debug for PackedOption - where T: ReservedValue + fmt::Debug +where + T: ReservedValue + fmt::Debug, { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { if self.is_none() { diff --git a/lib/cretonne/src/partition_slice.rs b/lib/cretonne/src/partition_slice.rs index 9626b5fd37..6f106e5bfc 100644 --- a/lib/cretonne/src/partition_slice.rs +++ b/lib/cretonne/src/partition_slice.rs @@ -7,7 +7,8 @@ /// /// Returns the number of elements where `p(t)` is true. pub fn partition_slice<'a, T: 'a, F>(s: &'a mut [T], mut p: F) -> usize - where F: FnMut(&T) -> bool +where + F: FnMut(&T) -> bool, { // Count the length of the prefix where `p` returns true. let mut count = match s.iter().position(|t| !p(t)) { diff --git a/lib/cretonne/src/regalloc/affinity.rs b/lib/cretonne/src/regalloc/affinity.rs index a7c95df99f..d7dede67d2 100644 --- a/lib/cretonne/src/regalloc/affinity.rs +++ b/lib/cretonne/src/regalloc/affinity.rs @@ -91,7 +91,8 @@ impl Affinity { // If the preferred register class is a subclass of the constraint, there's no need // to change anything. if constraint.kind != ConstraintKind::Stack && - !constraint.regclass.has_subclass(rc) { + !constraint.regclass.has_subclass(rc) + { // If the register classes don't overlap, `intersect` returns `None`, and we // just keep our previous affinity. if let Some(subclass) = constraint.regclass.intersect(reg_info.rc(rc)) { diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index 063c8e63ac..c765d73b1a 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -86,10 +86,9 @@ impl AllocatableSet { /// /// This assumes that unused bits are 1. pub fn interferes_with(&self, other: &AllocatableSet) -> bool { - self.avail - .iter() - .zip(&other.avail) - .any(|(&x, &y)| (x | y) != !0) + self.avail.iter().zip(&other.avail).any( + |(&x, &y)| (x | y) != !0, + ) } /// Intersect this set of allocatable registers with `other`. This has the effect of removing diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index ac11df3498..35957f90c1 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -132,14 +132,15 @@ impl DomForest { /// /// If the merge succeeds, returns `Ok(())`. The merged sequence can be extracted with /// `swap()`. - pub fn try_merge(&mut self, - va: &[Value], - vb: &[Value], - dfg: &DataFlowGraph, - layout: &Layout, - domtree: &DominatorTree, - liveness: &Liveness) - -> Result<(), (Value, Value)> { + pub fn try_merge( + &mut self, + va: &[Value], + vb: &[Value], + dfg: &DataFlowGraph, + layout: &Layout, + domtree: &DominatorTree, + liveness: &Liveness, + ) -> Result<(), (Value, Value)> { self.stack.clear(); self.values.clear(); self.values.reserve(va.len() + vb.len()); @@ -154,16 +155,16 @@ impl DomForest { for node in merged { if let Some(parent) = self.push_node(node, layout, domtree) { // Check if `parent` live range contains `node.def`. - let lr = liveness - .get(parent) - .expect("No live range for parent value"); + let lr = liveness.get(parent).expect( + "No live range for parent value", + ); if lr.overlaps_def(node.def, layout.pp_ebb(node.def), layout) { // Interference detected. Get the `(a, b)` order right in the error. return Err(if node.set == 0 { - (node.value, parent) - } else { - (parent, node.value) - }); + (node.value, parent) + } else { + (parent, node.value) + }); } } } @@ -177,8 +178,9 @@ impl DomForest { /// Given two ordered sequences of nodes, yield an ordered sequence containing all of them. /// Duplicates are removed. struct MergedNodes<'a, IA, IB> - where IA: Iterator, - IB: Iterator +where + IA: Iterator, + IB: Iterator, { a: Peekable, b: Peekable, @@ -187,8 +189,9 @@ struct MergedNodes<'a, IA, IB> } impl<'a, IA, IB> Iterator for MergedNodes<'a, IA, IB> - where IA: Iterator, - IB: Iterator +where + IA: Iterator, + IB: Iterator, { type Item = Node; @@ -198,9 +201,12 @@ impl<'a, IA, IB> Iterator for MergedNodes<'a, IA, IB> // If the two values are defined at the same point, compare value numbers instead // this is going to cause an interference conflict unless its actually the same // value appearing in both streams. - self.domtree - .rpo_cmp(a.def, b.def, self.layout) - .then(Ord::cmp(&a.value, &b.value)) + self.domtree.rpo_cmp(a.def, b.def, self.layout).then( + Ord::cmp( + &a.value, + &b.value, + ), + ) } (Some(_), None) => Ordering::Less, (None, Some(_)) => Ordering::Greater, @@ -256,13 +262,15 @@ impl Coalescing { } /// Convert `func` to conventional SSA form and build virtual registers in the process. - pub fn conventional_ssa(&mut self, - isa: &TargetIsa, - func: &mut Function, - cfg: &ControlFlowGraph, - domtree: &DominatorTree, - liveness: &mut Liveness, - virtregs: &mut VirtRegs) { + pub fn conventional_ssa( + &mut self, + isa: &TargetIsa, + func: &mut Function, + cfg: &ControlFlowGraph, + domtree: &DominatorTree, + liveness: &mut Liveness, + virtregs: &mut VirtRegs, + ) { dbg!("Coalescing for:\n{}", func.display(isa)); let mut context = Context { isa, @@ -329,9 +337,11 @@ impl<'a> Context<'a> { // // Try to catch infinite splitting loops. The values created by splitting should never // have irreconcilable interferences. - assert!(!self.split_values.contains(&bad_value), - "{} was already isolated", - bad_value); + assert!( + !self.split_values.contains(&bad_value), + "{} was already isolated", + bad_value + ); let split_len = self.split_values.len(); // The bad value can be both the successor value and a predecessor value at the same @@ -349,18 +359,22 @@ impl<'a> Context<'a> { } // Second loop check. - assert_ne!(split_len, - self.split_values.len(), - "Couldn't isolate {}", - bad_value); + assert_ne!( + split_len, + self.split_values.len(), + "Couldn't isolate {}", + bad_value + ); } let vreg = self.virtregs.unify(self.values); - dbg!("Coalesced {} arg {} into {} = {}", - ebb, - argnum, - vreg, - DisplayList(self.virtregs.values(vreg))); + dbg!( + "Coalesced {} arg {} into {} = {}", + ebb, + argnum, + vreg, + DisplayList(self.virtregs.values(vreg)) + ); } /// Reset `self.values` to just the set of split values. @@ -369,21 +383,21 @@ impl<'a> Context<'a> { self.values.extend_from_slice(self.split_values); let domtree = &self.domtree; let func = &self.func; - self.values - .sort_by(|&a, &b| { - domtree.rpo_cmp(func.dfg.value_def(a), func.dfg.value_def(b), &func.layout) - }); + self.values.sort_by(|&a, &b| { + domtree.rpo_cmp(func.dfg.value_def(a), func.dfg.value_def(b), &func.layout) + }); } /// Try coalescing predecessors with `succ_val`. /// /// Returns a value from a congruence class that needs to be split before starting over, or /// `None` if everything was successfully coalesced into `self.values`. - fn try_coalesce(&mut self, - argnum: usize, - succ_val: Value, - preds: &[BasicBlock]) - -> Option { + fn try_coalesce( + &mut self, + argnum: usize, + succ_val: Value, + preds: &[BasicBlock], + ) -> Option { // Initialize the value list with the split values. These are guaranteed to be // interference free, and anything that interferes with them must be split away. self.reset_values(); @@ -397,19 +411,22 @@ impl<'a> Context<'a> { for &(pred_ebb, pred_inst) in preds { let pred_val = self.func.dfg.inst_variable_args(pred_inst)[argnum]; - dbg!("Checking {}: {}: {}", - pred_val, - pred_ebb, - self.func.dfg.display_inst(pred_inst, self.isa)); + dbg!( + "Checking {}: {}: {}", + pred_val, + pred_ebb, + self.func.dfg.display_inst(pred_inst, self.isa) + ); // Never coalesce incoming function arguments on the stack. These arguments are // pre-spilled, and the rest of the virtual register would be forced to spill to the // `incoming_arg` stack slot too. if let ValueDef::Arg(def_ebb, def_num) = self.func.dfg.value_def(pred_val) { if Some(def_ebb) == self.func.layout.entry_block() && - self.func.signature.argument_types[def_num] - .location - .is_stack() { + self.func.signature.argument_types[def_num] + .location + .is_stack() + { dbg!("Isolating incoming stack parameter {}", pred_val); let new_val = self.split_pred(pred_inst, pred_ebb, argnum, pred_val); assert!(self.add_class(new_val).is_ok()); @@ -424,9 +441,10 @@ impl<'a> Context<'a> { // // Check if the `a` live range is fundamentally incompatible with `pred_inst`. if self.liveness - .get(a) - .expect("No live range for interfering value") - .reaches_use(pred_inst, pred_ebb, &self.func.layout) { + .get(a) + .expect("No live range for interfering value") + .reaches_use(pred_inst, pred_ebb, &self.func.layout) + { // Splitting at `pred_inst` wouldn't resolve the interference, so we need to // start over. return Some(a); @@ -435,8 +453,10 @@ impl<'a> Context<'a> { // The local conflict could be avoided by splitting at this predecessor, so try // that. This split is not necessarily required, but it allows us to make progress. let new_val = self.split_pred(pred_inst, pred_ebb, argnum, pred_val); - assert!(self.add_class(new_val).is_ok(), - "Splitting didn't resolve conflict."); + assert!( + self.add_class(new_val).is_ok(), + "Splitting didn't resolve conflict." + ); } } @@ -447,42 +467,52 @@ impl<'a> Context<'a> { /// /// Leave `self.values` unchanged on failure. fn add_class(&mut self, value: Value) -> Result<(), (Value, Value)> { - self.forest - .try_merge(&self.values, - self.virtregs.congruence_class(&value), - &self.func.dfg, - &self.func.layout, - self.domtree, - self.liveness)?; + self.forest.try_merge( + &self.values, + self.virtregs.congruence_class(&value), + &self.func.dfg, + &self.func.layout, + self.domtree, + self.liveness, + )?; self.forest.swap(&mut self.values); Ok(()) } /// Split the congruence class for the `argnum` argument to `pred_inst` by inserting a copy. - fn split_pred(&mut self, - pred_inst: Inst, - pred_ebb: Ebb, - argnum: usize, - pred_val: Value) - -> Value { + fn split_pred( + &mut self, + pred_inst: Inst, + pred_ebb: Ebb, + argnum: usize, + pred_val: Value, + ) -> Value { let mut pos = EncCursor::new(self.func, self.isa).at_inst(pred_inst); let copy = pos.ins().copy(pred_val); let inst = pos.built_inst(); - dbg!("Inserted {}, before {}: {}", - pos.display_inst(inst), - pred_ebb, - pos.display_inst(pred_inst)); + dbg!( + "Inserted {}, before {}: {}", + pos.display_inst(inst), + pred_ebb, + pos.display_inst(pred_inst) + ); // Create a live range for the new value. - let affinity = Affinity::new(&self.encinfo - .operand_constraints(pos.func.encodings[inst]) - .expect("Bad copy encoding") - .outs - [0]); + let affinity = Affinity::new( + &self.encinfo + .operand_constraints(pos.func.encodings[inst]) + .expect("Bad copy encoding") + .outs + [0], + ); self.liveness.create_dead(copy, inst, affinity); - self.liveness - .extend_locally(copy, pred_ebb, pred_inst, &pos.func.layout); + self.liveness.extend_locally( + copy, + pred_ebb, + pred_inst, + &pos.func.layout, + ); pos.func.dfg.inst_variable_args_mut(pred_inst)[argnum] = copy; self.split_values.push(copy); @@ -500,21 +530,29 @@ impl<'a> Context<'a> { let inst = pos.built_inst(); self.liveness.move_def_locally(succ_val, inst); - dbg!("Inserted {}, following {}({}: {})", - pos.display_inst(inst), - ebb, - new_val, - ty); + dbg!( + "Inserted {}, following {}({}: {})", + pos.display_inst(inst), + ebb, + new_val, + ty + ); // Create a live range for the new value. - let affinity = Affinity::new(&self.encinfo - .operand_constraints(pos.func.encodings[inst]) - .expect("Bad copy encoding") - .outs - [0]); + let affinity = Affinity::new( + &self.encinfo + .operand_constraints(pos.func.encodings[inst]) + .expect("Bad copy encoding") + .outs + [0], + ); self.liveness.create_dead(new_val, ebb, affinity); - self.liveness - .extend_locally(new_val, ebb, inst, &pos.func.layout); + self.liveness.extend_locally( + new_val, + ebb, + inst, + &pos.func.layout, + ); self.split_values.push(new_val); new_val diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 49f26fbd37..31fe44f03c 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -105,12 +105,14 @@ impl Coloring { } /// Run the coloring algorithm over `func`. - pub fn run(&mut self, - isa: &TargetIsa, - func: &mut Function, - domtree: &DominatorTree, - liveness: &mut Liveness, - tracker: &mut LiveValueTracker) { + pub fn run( + &mut self, + isa: &TargetIsa, + func: &mut Function, + domtree: &DominatorTree, + liveness: &mut Liveness, + tracker: &mut LiveValueTracker, + ) { dbg!("Coloring for:\n{}", func.display(isa)); let mut ctx = Context { isa, @@ -150,15 +152,17 @@ impl<'a> Context<'a> { pos.goto_top(ebb); while let Some(inst) = pos.next_inst() { if let Some(constraints) = self.encinfo.operand_constraints(func.encodings[inst]) { - self.visit_inst(inst, - constraints, - &mut pos, - &mut func.dfg, - tracker, - &mut regs, - &mut func.locations, - &mut func.encodings, - &func.signature); + self.visit_inst( + inst, + constraints, + &mut pos, + &mut func.dfg, + tracker, + &mut regs, + &mut func.locations, + &mut func.encodings, + &func.signature, + ); } else { let (_throughs, kills) = tracker.process_ghost(inst); self.process_ghost_kills(kills, &mut regs, &func.locations); @@ -170,11 +174,12 @@ impl<'a> Context<'a> { /// Visit the `ebb` header. /// /// Initialize the set of live registers and color the arguments to `ebb`. - fn visit_ebb_header(&self, - ebb: Ebb, - func: &mut Function, - tracker: &mut LiveValueTracker) - -> AllocatableSet { + fn visit_ebb_header( + &self, + ebb: Ebb, + func: &mut Function, + tracker: &mut LiveValueTracker, + ) -> AllocatableSet { // Reposition the live value tracker and deal with the EBB arguments. tracker.ebb_top(ebb, &func.dfg, self.liveness, &func.layout, self.domtree); @@ -204,10 +209,12 @@ impl<'a> Context<'a> { .get(value) .expect("No live range for live-in") .affinity; - dbg!("Live-in: {}:{} in {}", - value, - affinity.display(&self.reginfo), - func.locations[value].display(&self.reginfo)); + dbg!( + "Live-in: {}:{} in {}", + value, + affinity.display(&self.reginfo), + func.locations[value].display(&self.reginfo) + ); if let Affinity::Reg(rci) = affinity { let rc = self.reginfo.rc(rci); let loc = func.locations[value]; @@ -230,11 +237,12 @@ impl<'a> Context<'a> { /// function signature. /// /// Return the set of remaining allocatable registers after filtering out the dead arguments. - fn color_entry_args(&self, - sig: &Signature, - args: &[LiveValue], - locations: &mut ValueLocations) - -> AllocatableSet { + fn color_entry_args( + &self, + sig: &Signature, + args: &[LiveValue], + locations: &mut ValueLocations, + ) -> AllocatableSet { assert_eq!(sig.argument_types.len(), args.len()); let mut regs = self.usable_regs.clone(); @@ -250,10 +258,12 @@ impl<'a> Context<'a> { locations[lv.value] = ValueLoc::Reg(reg); } else { // This should have been fixed by the reload pass. - panic!("Entry arg {} has {} affinity, but ABI {}", - lv.value, - lv.affinity.display(&self.reginfo), - abi.display(&self.reginfo)); + panic!( + "Entry arg {} has {} affinity, but ABI {}", + lv.value, + lv.affinity.display(&self.reginfo), + abi.display(&self.reginfo) + ); } } @@ -273,19 +283,23 @@ impl<'a> Context<'a> { /// /// Update `regs` to reflect the allocated registers after `inst`, including removing any dead /// or killed values from the set. - fn visit_inst(&mut self, - inst: Inst, - constraints: &RecipeConstraints, - pos: &mut Cursor, - dfg: &mut DataFlowGraph, - tracker: &mut LiveValueTracker, - regs: &mut AllocatableSet, - locations: &mut ValueLocations, - encodings: &mut InstEncodings, - func_signature: &Signature) { - dbg!("Coloring {}\n {}", - dfg.display_inst(inst, self.isa), - regs.display(&self.reginfo)); + fn visit_inst( + &mut self, + inst: Inst, + constraints: &RecipeConstraints, + pos: &mut Cursor, + dfg: &mut DataFlowGraph, + tracker: &mut LiveValueTracker, + regs: &mut AllocatableSet, + locations: &mut ValueLocations, + encodings: &mut InstEncodings, + func_signature: &Signature, + ) { + dbg!( + "Coloring {}\n {}", + dfg.display_inst(inst, self.isa), + regs.display(&self.reginfo) + ); // EBB whose arguments should be colored to match the current branch instruction's // arguments. @@ -310,10 +324,12 @@ impl<'a> Context<'a> { } else { // This is a multi-way branch like `br_table`. We only support arguments on // single-destination branches. - assert_eq!(dfg.inst_variable_args(inst).len(), - 0, - "Can't handle EBB arguments: {}", - dfg.display_inst(inst, self.isa)); + assert_eq!( + dfg.inst_variable_args(inst).len(), + 0, + "Can't handle EBB arguments: {}", + dfg.display_inst(inst, self.isa) + ); self.undivert_regs(|lr| !lr.is_local()); } } @@ -329,10 +345,11 @@ impl<'a> Context<'a> { // Get rid of the killed values. for lv in kills { if let Affinity::Reg(rci) = lv.affinity { - self.solver - .add_kill(lv.value, - self.reginfo.rc(rci), - self.divert.reg(lv.value, locations)); + self.solver.add_kill( + lv.value, + self.reginfo.rc(rci), + self.divert.reg(lv.value, locations), + ); } } @@ -350,9 +367,9 @@ impl<'a> Context<'a> { // Finally, we've fully programmed the constraint solver. // We expect a quick solution in most cases. - let mut output_regs = self.solver - .quick_solve() - .unwrap_or_else(|_| self.iterate_solution()); + let mut output_regs = self.solver.quick_solve().unwrap_or_else( + |_| self.iterate_solution(), + ); // The solution and/or fixed input constraints may require us to shuffle the set of live @@ -399,30 +416,42 @@ impl<'a> Context<'a> { } /// Program the input-side constraints for `inst` into the constraint solver. - fn program_input_constraints(&mut self, - inst: Inst, - constraints: &[OperandConstraint], - dfg: &DataFlowGraph, - locations: &ValueLocations) { - for (op, &value) in constraints - .iter() - .zip(dfg.inst_args(inst)) - .filter(|&(op, _)| op.kind != ConstraintKind::Stack) { + fn program_input_constraints( + &mut self, + inst: Inst, + constraints: &[OperandConstraint], + dfg: &DataFlowGraph, + locations: &ValueLocations, + ) { + for (op, &value) in constraints.iter().zip(dfg.inst_args(inst)).filter( + |&(op, _)| { + op.kind != ConstraintKind::Stack + }, + ) + { // Reload pass is supposed to ensure that all arguments to register operands are // already in a register. let cur_reg = self.divert.reg(value, locations); match op.kind { ConstraintKind::FixedReg(regunit) => { if regunit != cur_reg { - self.solver - .reassign_in(value, op.regclass, cur_reg, regunit); + self.solver.reassign_in( + value, + op.regclass, + cur_reg, + regunit, + ); } } ConstraintKind::Reg | ConstraintKind::Tied(_) => { if !op.regclass.contains(cur_reg) { - self.solver - .add_var(value, op.regclass, cur_reg, &self.reginfo); + self.solver.add_var( + value, + op.regclass, + cur_reg, + &self.reginfo, + ); } } ConstraintKind::Stack => unreachable!(), @@ -433,18 +462,21 @@ impl<'a> Context<'a> { /// Program the input-side ABI constraints for `inst` into the constraint solver. /// /// ABI constraints are the fixed register assignments used for calls and returns. - fn program_input_abi(&mut self, - inst: Inst, - abi_types: &[ArgumentType], - dfg: &DataFlowGraph, - locations: &ValueLocations) { + fn program_input_abi( + &mut self, + inst: Inst, + abi_types: &[ArgumentType], + dfg: &DataFlowGraph, + locations: &ValueLocations, + ) { for (abi, &value) in abi_types.iter().zip(dfg.inst_variable_args(inst)) { if let ArgumentLoc::Reg(reg) = abi.location { if let Affinity::Reg(rci) = self.liveness .get(value) .expect("ABI register must have live range") - .affinity { + .affinity + { let rc = self.reginfo.rc(rci); let cur_reg = self.divert.reg(value, locations); self.solver.reassign_in(value, rc, cur_reg, reg); @@ -464,13 +496,14 @@ impl<'a> Context<'a> { /// /// Returns true if this is the first time a branch to `dest` is seen, so the `dest` argument /// values should be colored after `shuffle_inputs`. - fn program_ebb_arguments(&mut self, - inst: Inst, - dest: Ebb, - dfg: &DataFlowGraph, - layout: &Layout, - locations: &ValueLocations) - -> bool { + fn program_ebb_arguments( + &mut self, + inst: Inst, + dest: Ebb, + dfg: &DataFlowGraph, + layout: &Layout, + locations: &ValueLocations, + ) -> bool { // Find diverted registers that are live-in to `dest` and reassign them to their global // home. // @@ -523,11 +556,13 @@ impl<'a> Context<'a> { /// register state. /// /// This function is only called when `program_ebb_arguments()` returned `true`. - fn color_ebb_arguments(&mut self, - inst: Inst, - dest: Ebb, - dfg: &DataFlowGraph, - locations: &mut ValueLocations) { + fn color_ebb_arguments( + &mut self, + inst: Inst, + dest: Ebb, + dfg: &DataFlowGraph, + locations: &mut ValueLocations, + ) { let br_args = dfg.inst_variable_args(inst); let dest_args = dfg.ebb_args(dest); assert_eq!(br_args.len(), dest_args.len()); @@ -549,20 +584,23 @@ impl<'a> Context<'a> { /// Find all diverted registers where `pred` returns `true` and undo their diversion so they /// are reallocated to their global register assignments. fn undivert_regs(&mut self, mut pred: Pred) - where Pred: FnMut(&LiveRange) -> bool + where + Pred: FnMut(&LiveRange) -> bool, { for rdiv in self.divert.all() { - let lr = self.liveness - .get(rdiv.value) - .expect("Missing live range for diverted register"); + let lr = self.liveness.get(rdiv.value).expect( + "Missing live range for diverted register", + ); if pred(lr) { if let Affinity::Reg(rci) = lr.affinity { let rc = self.reginfo.rc(rci); self.solver.reassign_in(rdiv.value, rc, rdiv.to, rdiv.from); } else { - panic!("Diverted register {} with {} affinity", - rdiv.value, - lr.affinity.display(&self.reginfo)); + panic!( + "Diverted register {} with {} affinity", + rdiv.value, + lr.affinity.display(&self.reginfo) + ); } } } @@ -570,9 +608,7 @@ impl<'a> Context<'a> { // Find existing live values that conflict with the fixed input register constraints programmed // into the constraint solver. Convert them to solver variables so they can be diverted. - fn divert_fixed_input_conflicts(&mut self, - live: &[LiveValue], - locations: &mut ValueLocations) { + fn divert_fixed_input_conflicts(&mut self, live: &[LiveValue], locations: &mut ValueLocations) { for lv in live { if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); @@ -587,11 +623,13 @@ impl<'a> Context<'a> { /// Program any fixed-register output constraints into the solver. This may also detect /// conflicts between live-through registers and fixed output registers. These live-through /// values need to be turned into solver variables so they can be reassigned. - fn program_fixed_outputs(&mut self, - constraints: &[OperandConstraint], - defs: &[LiveValue], - throughs: &[LiveValue], - locations: &mut ValueLocations) { + fn program_fixed_outputs( + &mut self, + constraints: &[OperandConstraint], + defs: &[LiveValue], + throughs: &[LiveValue], + locations: &mut ValueLocations, + ) { for (op, lv) in constraints.iter().zip(defs) { if let ConstraintKind::FixedReg(reg) = op.kind { self.add_fixed_output(lv.value, op.regclass, reg, throughs, locations); @@ -602,11 +640,13 @@ impl<'a> Context<'a> { /// Program the output-side ABI constraints for `inst` into the constraint solver. /// /// That means return values for a call instruction. - fn program_output_abi(&mut self, - abi_types: &[ArgumentType], - defs: &[LiveValue], - throughs: &[LiveValue], - locations: &mut ValueLocations) { + fn program_output_abi( + &mut self, + abi_types: &[ArgumentType], + defs: &[LiveValue], + throughs: &[LiveValue], + locations: &mut ValueLocations, + ) { // It's technically possible for a call instruction to have fixed results before the // variable list of results, but we have no known instances of that. // Just assume all results are variable return values. @@ -624,12 +664,14 @@ impl<'a> Context<'a> { } /// Add a single fixed output value to the solver. - fn add_fixed_output(&mut self, - value: Value, - rc: RegClass, - reg: RegUnit, - throughs: &[LiveValue], - locations: &mut ValueLocations) { + fn add_fixed_output( + &mut self, + value: Value, + rc: RegClass, + reg: RegUnit, + throughs: &[LiveValue], + locations: &mut ValueLocations, + ) { if !self.solver.add_fixed_output(rc, reg) { // The fixed output conflicts with some of the live-through registers. for lv in throughs { @@ -656,12 +698,14 @@ impl<'a> Context<'a> { /// Program the output-side constraints for `inst` into the constraint solver. /// /// It is assumed that all fixed outputs have already been handled. - fn program_output_constraints(&mut self, - inst: Inst, - constraints: &[OperandConstraint], - defs: &[LiveValue], - dfg: &mut DataFlowGraph, - locations: &mut ValueLocations) { + fn program_output_constraints( + &mut self, + inst: Inst, + constraints: &[OperandConstraint], + defs: &[LiveValue], + dfg: &mut DataFlowGraph, + locations: &mut ValueLocations, + ) { for (op, lv) in constraints.iter().zip(defs) { match op.kind { ConstraintKind::FixedReg(_) | @@ -673,8 +717,11 @@ impl<'a> Context<'a> { // Find the input operand we're tied to. // The solver doesn't care about the output value. let arg = dfg.inst_args(inst)[num as usize]; - self.solver - .add_tied_input(arg, op.regclass, self.divert.reg(arg, locations)); + self.solver.add_tied_input( + arg, + op.regclass, + self.divert.reg(arg, locations), + ); } } } @@ -695,11 +742,13 @@ impl<'a> Context<'a> { /// before. /// /// The solver needs to be reminded of the available registers before any moves are inserted. - fn shuffle_inputs(&mut self, - pos: &mut Cursor, - dfg: &mut DataFlowGraph, - regs: &mut AllocatableSet, - encodings: &mut InstEncodings) { + fn shuffle_inputs( + &mut self, + pos: &mut Cursor, + dfg: &mut DataFlowGraph, + regs: &mut AllocatableSet, + encodings: &mut InstEncodings, + ) { self.solver.schedule_moves(regs); for m in self.solver.moves() { @@ -729,10 +778,12 @@ impl<'a> Context<'a> { /// Process kills on a ghost instruction. /// - Forget diversions. /// - Free killed registers. - fn process_ghost_kills(&mut self, - kills: &[LiveValue], - regs: &mut AllocatableSet, - locations: &ValueLocations) { + fn process_ghost_kills( + &mut self, + kills: &[LiveValue], + regs: &mut AllocatableSet, + locations: &ValueLocations, + ) { for lv in kills { if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 0523984604..2e11536596 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -53,12 +53,13 @@ impl Context { /// /// After register allocation, all values in `func` have been assigned to a register or stack /// location that is consistent with instruction encoding constraints. - pub fn run(&mut self, - isa: &TargetIsa, - func: &mut Function, - cfg: &ControlFlowGraph, - domtree: &DominatorTree) - -> CtonResult { + pub fn run( + &mut self, + isa: &TargetIsa, + func: &mut Function, + cfg: &ControlFlowGraph, + domtree: &DominatorTree, + ) -> CtonResult { // `Liveness` and `Coloring` are self-clearing. self.virtregs.clear(); @@ -74,13 +75,14 @@ impl Context { } // Pass: Coalesce and create conventional SSA form. - self.coalescing - .conventional_ssa(isa, - func, - cfg, - domtree, - &mut self.liveness, - &mut self.virtregs); + self.coalescing.conventional_ssa( + isa, + func, + cfg, + domtree, + &mut self.liveness, + &mut self.virtregs, + ); if isa.flags().enable_verifier() { verify_context(func, cfg, domtree, Some(isa))?; @@ -90,14 +92,15 @@ impl Context { // Pass: Spilling. - self.spilling - .run(isa, - func, - domtree, - &mut self.liveness, - &self.virtregs, - &mut self.topo, - &mut self.tracker); + self.spilling.run( + isa, + func, + domtree, + &mut self.liveness, + &self.virtregs, + &mut self.topo, + &mut self.tracker, + ); if isa.flags().enable_verifier() { verify_context(func, cfg, domtree, Some(isa))?; @@ -106,13 +109,14 @@ impl Context { } // Pass: Reload. - self.reload - .run(isa, - func, - domtree, - &mut self.liveness, - &mut self.topo, - &mut self.tracker); + self.reload.run( + isa, + func, + domtree, + &mut self.liveness, + &mut self.topo, + &mut self.tracker, + ); if isa.flags().enable_verifier() { verify_context(func, cfg, domtree, Some(isa))?; @@ -121,8 +125,13 @@ impl Context { } // Pass: Coloring. - self.coloring - .run(isa, func, domtree, &mut self.liveness, &mut self.tracker); + self.coloring.run( + isa, + func, + domtree, + &mut self.liveness, + &mut self.tracker, + ); if isa.flags().enable_verifier() { verify_context(func, cfg, domtree, Some(isa))?; diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index c7c9595972..103c624d5e 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -93,10 +93,11 @@ impl RegDiversions { /// /// Returns the `to` register of the removed diversion. pub fn remove(&mut self, value: Value) -> Option { - self.current - .iter() - .position(|d| d.value == value) - .map(|i| self.current.swap_remove(i).to) + self.current.iter().position(|d| d.value == value).map( + |i| { + self.current.swap_remove(i).to + }, + ) } } @@ -113,12 +114,14 @@ mod tests { let v2 = Value::new(2); divs.regmove(v1, 10, 12); - assert_eq!(divs.diversion(v1), - Some(&Diversion { - value: v1, - from: 10, - to: 12, - })); + assert_eq!( + divs.diversion(v1), + Some(&Diversion { + value: v1, + from: 10, + to: 12, + }) + ); assert_eq!(divs.diversion(v2), None); divs.regmove(v1, 12, 11); diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 09c0dfea07..26d47d4b90 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -74,14 +74,13 @@ impl LiveValueVec { /// Add a new live value to `values`. Copy some properties from `lr`. fn push(&mut self, value: Value, endpoint: Inst, lr: &LiveRange) { - self.values - .push(LiveValue { - value, - endpoint, - affinity: lr.affinity, - is_local: lr.is_local(), - is_dead: lr.is_dead(), - }); + self.values.push(LiveValue { + value, + endpoint, + affinity: lr.affinity, + is_local: lr.is_local(), + is_dead: lr.is_dead(), + }); } /// Remove all elements. @@ -157,13 +156,14 @@ impl LiveValueTracker { /// from the immediate dominator. The second slice is the set of `ebb` arguments that are live. /// /// Dead arguments with no uses are included in `args`. Call `drop_dead_args()` to remove them. - pub fn ebb_top(&mut self, - ebb: Ebb, - dfg: &DataFlowGraph, - liveness: &Liveness, - layout: &Layout, - domtree: &DominatorTree) - -> (&[LiveValue], &[LiveValue]) { + pub fn ebb_top( + &mut self, + ebb: Ebb, + dfg: &DataFlowGraph, + liveness: &Liveness, + layout: &Layout, + domtree: &DominatorTree, + ) -> (&[LiveValue], &[LiveValue]) { // Start over, compute the set of live values at the top of the EBB from two sources: // // 1. Values that were live before `ebb`'s immediate dominator, filtered for those that are @@ -179,14 +179,14 @@ impl LiveValueTracker { // If the immediate dominator exits, we must have a stored list for it. This is a // requirement to the order EBBs are visited: All dominators must have been processed // before the current EBB. - let idom_live_list = self.idom_sets - .get(&idom) - .expect("No stored live set for dominator"); + let idom_live_list = self.idom_sets.get(&idom).expect( + "No stored live set for dominator", + ); // Get just the values that are live-in to `ebb`. for &value in idom_live_list.as_slice(&self.idom_pool) { - let lr = liveness - .get(value) - .expect("Immediate dominator value has no live range"); + let lr = liveness.get(value).expect( + "Immediate dominator value has no live range", + ); // Check if this value is live-in here. if let Some(endpoint) = lr.livein_local_end(ebb, layout) { @@ -198,9 +198,9 @@ impl LiveValueTracker { // Now add all the live arguments to `ebb`. let first_arg = self.live.values.len(); for &value in dfg.ebb_args(ebb) { - let lr = liveness - .get(value) - .expect("EBB argument value has no live range"); + let lr = liveness.get(value).expect( + "EBB argument value has no live range", + ); assert_eq!(lr.def(), ebb.into()); match lr.def_local_end().into() { ExpandedProgramPoint::Inst(endpoint) => { @@ -209,13 +209,18 @@ impl LiveValueTracker { ExpandedProgramPoint::Ebb(local_ebb) => { // This is a dead EBB argument which is not even live into the first // instruction in the EBB. - assert_eq!(local_ebb, - ebb, - "EBB argument live range ends at wrong EBB header"); + assert_eq!( + local_ebb, + ebb, + "EBB argument live range ends at wrong EBB header" + ); // Give this value a fake endpoint that is the first instruction in the EBB. // We expect it to be removed by calling `drop_dead_args()`. - self.live - .push(value, layout.first_inst(ebb).expect("Empty EBB"), lr); + self.live.push( + value, + layout.first_inst(ebb).expect("Empty EBB"), + lr, + ); } } } @@ -241,11 +246,12 @@ impl LiveValueTracker { /// /// The `drop_dead()` method must be called next to actually remove the dead values from the /// tracked set after the two returned slices are no longer needed. - pub fn process_inst(&mut self, - inst: Inst, - dfg: &DataFlowGraph, - liveness: &Liveness) - -> (&[LiveValue], &[LiveValue], &[LiveValue]) { + pub fn process_inst( + &mut self, + inst: Inst, + dfg: &DataFlowGraph, + liveness: &Liveness, + ) -> (&[LiveValue], &[LiveValue], &[LiveValue]) { // Save a copy of the live values before any branches or jumps that could be somebody's // immediate dominator. match dfg[inst].analyze_branch(&dfg.value_lists) { @@ -272,9 +278,11 @@ impl LiveValueTracker { } } - (&self.live.values[0..first_kill], - &self.live.values[first_kill..first_def], - &self.live.values[first_def..]) + ( + &self.live.values[0..first_kill], + &self.live.values[first_kill..first_def], + &self.live.values[first_def..], + ) } /// Prepare to move past a ghost instruction. @@ -310,7 +318,8 @@ impl LiveValueTracker { /// Any values where `f` returns true are spilled and will be treated as if their affinity was /// `Stack`. pub fn process_spills(&mut self, mut f: F) - where F: FnMut(Value) -> bool + where + F: FnMut(Value) -> bool, { for lv in &mut self.live.values { if f(lv.value) { @@ -324,12 +333,10 @@ impl LiveValueTracker { let values = self.live.values.iter().map(|lv| lv.value); let pool = &mut self.idom_pool; // If there already is a set saved for `idom`, just keep it. - self.idom_sets - .entry(idom) - .or_insert_with(|| { - let mut list = ValueList::default(); - list.extend(values, pool); - list - }); + self.idom_sets.entry(idom).or_insert_with(|| { + let mut list = ValueList::default(); + list.extend(values, pool); + list + }); } } diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 124252b4fc..c47a703f3c 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -190,12 +190,13 @@ type LiveRangeSet = SparseMap; /// Get a mutable reference to the live range for `value`. /// Create it if necessary. -fn get_or_create<'a>(lrset: &'a mut LiveRangeSet, - value: Value, - isa: &TargetIsa, - func: &Function, - enc_info: &EncInfo) - -> &'a mut LiveRange { +fn get_or_create<'a>( + lrset: &'a mut LiveRangeSet, + value: Value, + isa: &TargetIsa, + func: &Function, + enc_info: &EncInfo, +) -> &'a mut LiveRange { // It would be better to use `get_mut()` here, but that leads to borrow checker fighting // which can probably only be resolved by non-lexical lifetimes. // https://github.com/rust-lang/rfcs/issues/811 @@ -233,12 +234,14 @@ fn get_or_create<'a>(lrset: &'a mut LiveRangeSet, } /// Extend the live range for `value` so it reaches `to` which must live in `ebb`. -fn extend_to_use(lr: &mut LiveRange, - ebb: Ebb, - to: Inst, - worklist: &mut Vec, - func: &Function, - cfg: &ControlFlowGraph) { +fn extend_to_use( + lr: &mut LiveRange, + ebb: Ebb, + to: Inst, + worklist: &mut Vec, + func: &Function, + cfg: &ControlFlowGraph, +) { // This is our scratch working space, and we'll leave it empty when we return. assert!(worklist.is_empty()); @@ -309,10 +312,12 @@ impl Liveness { /// /// This asserts that `value` does not have an existing live range. pub fn create_dead(&mut self, value: Value, def: PP, affinity: Affinity) - where PP: Into + where + PP: Into, { - let old = self.ranges - .insert(LiveRange::new(value, def.into(), affinity)); + let old = self.ranges.insert( + LiveRange::new(value, def.into(), affinity), + ); assert!(old.is_none(), "{} already has a live range", value); } @@ -320,7 +325,8 @@ impl Liveness { /// /// The old and new def points must be in the same EBB, and before the end of the live range. pub fn move_def_locally(&mut self, value: Value, def: PP) - where PP: Into + where + PP: Into, { let mut lr = self.ranges.get_mut(value).expect("Value has no live range"); lr.move_def_locally(def.into()); @@ -331,12 +337,13 @@ impl Liveness { /// It is assumed the `value` is already live before `user` in `ebb`. /// /// Returns a mutable reference to the value's affinity in case that also needs to be updated. - pub fn extend_locally(&mut self, - value: Value, - ebb: Ebb, - user: Inst, - layout: &Layout) - -> &mut Affinity { + pub fn extend_locally( + &mut self, + value: Value, + ebb: Ebb, + user: Inst, + layout: &Layout, + ) -> &mut Affinity { debug_assert_eq!(Some(ebb), layout.inst_ebb(user)); let mut lr = self.ranges.get_mut(value).expect("Value has no live range"); let livein = lr.extend_in_ebb(ebb, user, layout); @@ -401,7 +408,8 @@ impl Liveness { if let Some(constraint) = operand_constraints.next() { lr.affinity.merge(constraint, ®_info); } else if lr.affinity.is_none() && encoding.is_legal() && - !func.dfg[inst].opcode().is_branch() { + !func.dfg[inst].opcode().is_branch() + { // This is a real encoded instruction using a value that doesn't yet have a // concrete affinity. Most likely a call argument or a return value. Give // the value a register affinity matching the ABI type. diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index e7ba997f6e..3da125ef3a 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -224,13 +224,13 @@ impl LiveRange { self.liveins .binary_search_by(|intv| order.cmp(intv.begin, ebb)) .or_else(|n| { - // The interval at `n-1` may cover `ebb`. - if n > 0 && order.cmp(self.liveins[n - 1].end, ebb) == Ordering::Greater { - Ok(n - 1) - } else { - Err(n) - } - }) + // The interval at `n-1` may cover `ebb`. + if n > 0 && order.cmp(self.liveins[n - 1].end, ebb) == Ordering::Greater { + Ok(n - 1) + } else { + Err(n) + } + }) } /// Extend the local interval for `ebb` so it reaches `to` which must belong to `ebb`. @@ -250,11 +250,14 @@ impl LiveRange { // We're assuming here that `to` never precedes `def_begin` in the same EBB, but we can't // check it without a method for getting `to`'s EBB. if order.cmp(ebb, self.def_end) != Ordering::Greater && - order.cmp(to, self.def_begin) != Ordering::Less { + order.cmp(to, self.def_begin) != Ordering::Less + { let to_pp = to.into(); - assert_ne!(to_pp, - self.def_begin, - "Can't use value in the defining instruction."); + assert_ne!( + to_pp, + self.def_begin, + "Can't use value in the defining instruction." + ); if order.cmp(to, self.def_end) == Ordering::Greater { self.def_end = to_pp; } @@ -288,8 +291,10 @@ impl LiveRange { let prev = n.checked_sub(1).and_then(|i| self.liveins.get(i)); let next = self.liveins.get(n); - (prev.map_or(false, |prev| order.is_ebb_gap(prev.end, ebb)), - next.map_or(false, |next| order.is_ebb_gap(to, next.begin))) + ( + prev.map_or(false, |prev| order.is_ebb_gap(prev.end, ebb)), + next.map_or(false, |next| order.is_ebb_gap(to, next.begin)), + ) }; match (coalesce_prev, coalesce_next) { @@ -309,12 +314,13 @@ impl LiveRange { } // Cannot coalesce; insert new interval (false, false) => { - self.liveins - .insert(n, - Interval { - begin: ebb, - end: to, - }); + self.liveins.insert( + n, + Interval { + begin: ebb, + end: to, + }, + ); } } @@ -372,9 +378,9 @@ impl LiveRange { /// answer, but it is also possible that an even later program point is returned. So don't /// depend on the returned `Inst` to belong to `ebb`. pub fn livein_local_end(&self, ebb: Ebb, order: &PO) -> Option { - self.find_ebb_interval(ebb, order) - .ok() - .map(|n| self.liveins[n].end) + self.find_ebb_interval(ebb, order).ok().map(|n| { + self.liveins[n].end + }) } /// Get all the live-in intervals. @@ -384,11 +390,13 @@ impl LiveRange { /// Check if this live range overlaps a definition in `ebb`. pub fn overlaps_def(&self, def: ExpandedProgramPoint, ebb: Ebb, order: &PO) -> bool - where PO: ProgramOrder + where + PO: ProgramOrder, { // Check for an overlap with the local range. if order.cmp(def, self.def_begin) != Ordering::Less && - order.cmp(def, self.def_end) == Ordering::Less { + order.cmp(def, self.def_end) == Ordering::Less + { return true; } @@ -401,11 +409,13 @@ impl LiveRange { /// Check if this live range reaches a use at `user` in `ebb`. pub fn reaches_use(&self, user: Inst, ebb: Ebb, order: &PO) -> bool - where PO: ProgramOrder + where + PO: ProgramOrder, { // Check for an overlap with the local range. if order.cmp(user, self.def_begin) == Ordering::Greater && - order.cmp(user, self.def_end) != Ordering::Greater { + order.cmp(user, self.def_end) != Ordering::Greater + { return true; } @@ -418,7 +428,8 @@ impl LiveRange { /// Check if this live range is killed at `user` in `ebb`. pub fn killed_at(&self, user: Inst, ebb: Ebb, order: &PO) -> bool - where PO: ProgramOrder + where + PO: ProgramOrder, { self.def_local_end() == user.into() || self.livein_local_end(ebb, order) == Some(user) } @@ -447,8 +458,9 @@ mod tests { impl ProgramOrder for ProgOrder { fn cmp(&self, a: A, b: B) -> Ordering - where A: Into, - B: Into + where + A: Into, + B: Into, { fn idx(pp: ExpandedProgramPoint) -> usize { match pp { @@ -505,9 +517,11 @@ mod tests { assert_eq!(self.cmp(e, li.begin), Ordering::Less); } - assert!(self.cmp(lr.def_end, li.begin) == Ordering::Less || + assert!( + self.cmp(lr.def_end, li.begin) == Ordering::Less || self.cmp(lr.def_begin, li.end) == Ordering::Greater, - "Interval can't overlap the def EBB"); + "Interval can't overlap the def EBB" + ); // Save for next round. prev_end = Some(li.end); diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index a9ab6bbbcb..7a4f1f715b 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -103,10 +103,10 @@ impl Pressure { } // Compute per-class limits from `usable`. - for (toprc, rc) in p.toprc - .iter_mut() - .take_while(|t| t.num_toprcs > 0) - .zip(reginfo.classes) { + for (toprc, rc) in p.toprc.iter_mut().take_while(|t| t.num_toprcs > 0).zip( + reginfo.classes, + ) + { toprc.limit = usable.iter(rc).len() as u32; toprc.width = rc.width; } diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 81c6c3661b..51aded85e0 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -54,13 +54,15 @@ impl Reload { } /// Run the reload algorithm over `func`. - pub fn run(&mut self, - isa: &TargetIsa, - func: &mut Function, - domtree: &DominatorTree, - liveness: &mut Liveness, - topo: &mut TopoOrder, - tracker: &mut LiveValueTracker) { + pub fn run( + &mut self, + isa: &TargetIsa, + func: &mut Function, + domtree: &DominatorTree, + liveness: &mut Liveness, + topo: &mut TopoOrder, + tracker: &mut LiveValueTracker, + ) { dbg!("Reload for:\n{}", func.display(isa)); let mut ctx = Context { cur: EncCursor::new(func, isa), @@ -125,11 +127,13 @@ impl<'a> Context<'a> { /// Process the EBB parameters. Move to the next instruction in the EBB to be processed fn visit_ebb_header(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) { - let (liveins, args) = tracker.ebb_top(ebb, - &self.cur.func.dfg, - self.liveness, - &self.cur.func.layout, - self.domtree); + let (liveins, args) = tracker.ebb_top( + ebb, + &self.cur.func.dfg, + self.liveness, + &self.cur.func.layout, + self.domtree, + ); if self.cur.func.layout.entry_block() == Some(ebb) { assert_eq!(liveins.len(), 0); @@ -172,15 +176,17 @@ impl<'a> Context<'a> { /// Process the instruction pointed to by `pos`, and advance the cursor to the next instruction /// that needs processing. - fn visit_inst(&mut self, - ebb: Ebb, - inst: Inst, - encoding: Encoding, - tracker: &mut LiveValueTracker) { + fn visit_inst( + &mut self, + ebb: Ebb, + inst: Inst, + encoding: Encoding, + tracker: &mut LiveValueTracker, + ) { // Get the operand constraints for `inst` that we are trying to satisfy. - let constraints = self.encinfo - .operand_constraints(encoding) - .expect("Missing instruction encoding"); + let constraints = self.encinfo.operand_constraints(encoding).expect( + "Missing instruction encoding", + ); // Identify reload candidates. assert!(self.candidates.is_empty()); @@ -195,17 +201,20 @@ impl<'a> Context<'a> { let reg = self.cur.ins().fill(cand.value); let fill = self.cur.built_inst(); - self.reloads - .insert(ReloadedValue { - stack: cand.value, - reg: reg, - }); + self.reloads.insert(ReloadedValue { + stack: cand.value, + reg: reg, + }); // Create a live range for the new reload. let affinity = Affinity::Reg(cand.regclass.into()); self.liveness.create_dead(reg, fill, affinity); - self.liveness - .extend_locally(reg, ebb, inst, &self.cur.func.layout); + self.liveness.extend_locally( + reg, + ebb, + inst, + &self.cur.func.layout, + ); } // Rewrite arguments. @@ -218,8 +227,8 @@ impl<'a> Context<'a> { // TODO: Reuse reloads for future instructions. self.reloads.clear(); - let (_throughs, _kills, defs) = tracker - .process_inst(inst, &self.cur.func.dfg, self.liveness); + let (_throughs, _kills, defs) = + tracker.process_inst(inst, &self.cur.func.dfg, self.liveness); // Advance to the next instruction so we can insert any spills after the instruction. self.cur.next_inst(); @@ -255,11 +264,10 @@ impl<'a> Context<'a> { for (op, &arg) in constraints.ins.iter().zip(args) { if op.kind != ConstraintKind::Stack { if self.liveness[arg].affinity.is_stack() { - self.candidates - .push(ReloadCandidate { - value: arg, - regclass: op.regclass, - }) + self.candidates.push(ReloadCandidate { + value: arg, + regclass: op.regclass, + }) } } } @@ -272,17 +280,21 @@ impl<'a> Context<'a> { // Handle ABI arguments. if let Some(sig) = self.cur.func.dfg.call_signature(inst) { - handle_abi_args(self.candidates, - &self.cur.func.dfg.signatures[sig].argument_types, - var_args, - self.cur.isa, - self.liveness); + handle_abi_args( + self.candidates, + &self.cur.func.dfg.signatures[sig].argument_types, + var_args, + self.cur.isa, + self.liveness, + ); } else if self.cur.func.dfg[inst].opcode().is_return() { - handle_abi_args(self.candidates, - &self.cur.func.signature.return_types, - var_args, - self.cur.isa, - self.liveness); + handle_abi_args( + self.candidates, + &self.cur.func.signature.return_types, + var_args, + self.cur.isa, + self.liveness, + ); } } @@ -297,27 +309,33 @@ impl<'a> Context<'a> { // Update live ranges. self.liveness.move_def_locally(stack, inst); - self.liveness - .extend_locally(reg, ebb, inst, &self.cur.func.layout); + self.liveness.extend_locally( + reg, + ebb, + inst, + &self.cur.func.layout, + ); } } /// Find reload candidates in the instruction's ABI variable arguments. This handles both /// return values and call arguments. -fn handle_abi_args(candidates: &mut Vec, - abi_types: &[ArgumentType], - var_args: &[Value], - isa: &TargetIsa, - liveness: &Liveness) { +fn handle_abi_args( + candidates: &mut Vec, + abi_types: &[ArgumentType], + var_args: &[Value], + isa: &TargetIsa, + liveness: &Liveness, +) { assert_eq!(abi_types.len(), var_args.len()); for (abi, &arg) in abi_types.iter().zip(var_args) { if abi.location.is_reg() { let lv = liveness.get(arg).expect("Missing live range for ABI arg"); if lv.affinity.is_stack() { candidates.push(ReloadCandidate { - value: arg, - regclass: isa.regclass_for_abi_type(abi.value_type), - }); + value: arg, + regclass: isa.regclass_for_abi_type(abi.value_type), + }); } } } diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 2847a77b84..b99da23d43 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -231,12 +231,14 @@ impl SparseMapValue for Assignment { impl fmt::Display for Assignment { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - write!(f, - "{}:{}(%{} -> %{})", - self.value, - self.rc, - self.from, - self.to) + write!( + f, + "{}:{}(%{} -> %{})", + self.value, + self.rc, + self.from, + self.to + ) } } @@ -244,7 +246,7 @@ impl fmt::Display for Assignment { impl PartialEq for Assignment { fn eq(&self, other: &Assignment) -> bool { self.value == other.value && self.from == other.from && self.to == other.to && - self.rc.index == other.rc.index + self.rc.index == other.rc.index } } @@ -363,22 +365,23 @@ impl Solver { dbg!("-> converting variable {} to a fixed constraint", v); // The spiller is responsible for ensuring that all constraints on the uses of a // value are compatible. - assert!(v.constraint.contains(to), - "Incompatible constraints for {}", - value); + assert!( + v.constraint.contains(to), + "Incompatible constraints for {}", + value + ); } else { panic!("Invalid from register for fixed {} constraint", value); } } self.regs_in.free(rc, from); self.regs_out.take(rc, to); - self.assignments - .insert(Assignment { - value, - rc, - from, - to, - }); + self.assignments.insert(Assignment { + value, + rc, + from, + to, + }); } /// Add a variable representing an input side value with an existing register assignment. @@ -388,18 +391,22 @@ impl Solver { /// /// It is assumed initially that the value is also live on the output side of the instruction. /// This can be changed by calling to `add_kill()`. - pub fn add_var(&mut self, - value: Value, - constraint: RegClass, - from: RegUnit, - reginfo: &RegInfo) { + pub fn add_var( + &mut self, + value: Value, + constraint: RegClass, + from: RegUnit, + reginfo: &RegInfo, + ) { // Check for existing entries for this value. if self.regs_in.is_avail(constraint, from) { - dbg!("add_var({}:{}, from={}/%{}) for existing entry", - value, - constraint, - reginfo.display_regunit(from), - from); + dbg!( + "add_var({}:{}, from={}/%{}) for existing entry", + value, + constraint, + reginfo.display_regunit(from), + from + ); // There could be an existing variable entry. if let Some(v) = self.vars.iter_mut().find(|v| v.value == value) { @@ -419,9 +426,11 @@ impl Solver { // No variable, then it must be a fixed reassignment. if let Some(a) = self.assignments.get(value) { dbg!("-> already fixed assignment {}", a); - assert!(constraint.contains(a.to), - "Incompatible constraints for {}", - value); + assert!( + constraint.contains(a.to), + "Incompatible constraints for {}", + value + ); return; } @@ -430,12 +439,14 @@ impl Solver { } let new_var = Variable::new_live(value, constraint, from); - dbg!("add_var({}:{}, from={}/%{}) new entry: {}", - value, - constraint, - reginfo.display_regunit(from), - from, - new_var); + dbg!( + "add_var({}:{}, from={}/%{}) new entry: {}", + value, + constraint, + reginfo.display_regunit(from), + from, + new_var + ); self.regs_in.free(constraint, from); if self.inputs_done { @@ -623,23 +634,20 @@ impl Solver { // Collect moves from the chosen solution for all non-define variables. for v in &self.vars { if let Some(from) = v.from { - self.moves - .push(Assignment { - value: v.value, - from, - to: v.solution, - rc: v.constraint, - }); + self.moves.push(Assignment { + value: v.value, + from, + to: v.solution, + rc: v.constraint, + }); } } // Convert all of the fixed register assignments into moves, but omit the ones that are // already in the right register. - self.moves - .extend(self.assignments - .values() - .cloned() - .filter(|v| v.from != v.to)); + self.moves.extend(self.assignments.values().cloned().filter( + |v| v.from != v.to, + )); dbg!("collect_moves: {}", DisplayList(self.moves.as_slice())); } @@ -661,9 +669,10 @@ impl Solver { let mut i = 0; while i < self.moves.len() { // Find the first move that can be executed now. - if let Some(j) = self.moves[i..] - .iter() - .position(|m| avail.is_avail(m.rc, m.to)) { + if let Some(j) = self.moves[i..].iter().position( + |m| avail.is_avail(m.rc, m.to), + ) + { // This move can be executed now. self.moves.swap(i, i + j); let m = &self.moves[i]; @@ -709,17 +718,16 @@ impl Solver { // Append a fixup move so we end up in the right place. This move will be scheduled // later. That's ok because it is the single remaining move of `m.value` after the // next iteration. - self.moves - .push(Assignment { - value: m.value, - rc: m.rc, - from: reg, - to: m.to, - }); - // TODO: What if allocating an extra register is not enough to break a cycle? This - // can happen when there are registers of different widths in a cycle. For ARM, we - // may have to move two S-registers out of the way before we can resolve a cycle - // involving a D-register. + self.moves.push(Assignment { + value: m.value, + rc: m.rc, + from: reg, + to: m.to, + }); + // TODO: What if allocating an extra register is not enough to break a cycle? This + // can happen when there are registers of different widths in a cycle. For ARM, we + // may have to move two S-registers out of the way before we can resolve a cycle + // involving a D-register. } else { panic!("Not enough registers in {} to schedule moves", m.rc); } @@ -738,9 +746,11 @@ impl Solver { impl fmt::Display for Solver { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { writeln!(f, "Solver {{ inputs_done: {},", self.inputs_done)?; - writeln!(f, - " assignments: {}", - DisplayList(self.assignments.as_slice()))?; + writeln!( + f, + " assignments: {}", + DisplayList(self.assignments.as_slice()) + )?; writeln!(f, " vars: {}", DisplayList(self.vars.as_slice()))?; writeln!(f, " moves: {}", DisplayList(self.moves.as_slice()))?; writeln!(f, "}}") @@ -817,8 +827,10 @@ mod tests { solver.inputs_done(); assert!(solver.quick_solve().is_ok()); assert_eq!(solver.schedule_moves(®s), 0); - assert_eq!(solver.moves(), - &[mov(v11, gpr, r1, r2), mov(v10, gpr, r0, r1)]); + assert_eq!( + solver.moves(), + &[mov(v11, gpr, r1, r2), mov(v10, gpr, r0, r1)] + ); // Swap r0 and r1 in three moves using r2 as a scratch. solver.reset(®s); @@ -827,10 +839,14 @@ mod tests { solver.inputs_done(); assert!(solver.quick_solve().is_ok()); assert_eq!(solver.schedule_moves(®s), 0); - assert_eq!(solver.moves(), - &[mov(v10, gpr, r0, r2), - mov(v11, gpr, r1, r0), - mov(v10, gpr, r2, r1)]); + assert_eq!( + solver.moves(), + &[ + mov(v10, gpr, r0, r2), + mov(v11, gpr, r1, r0), + mov(v10, gpr, r2, r1), + ] + ); } #[test] @@ -862,11 +878,15 @@ mod tests { solver.inputs_done(); assert!(solver.quick_solve().is_ok()); assert_eq!(solver.schedule_moves(®s), 0); - assert_eq!(solver.moves(), - &[mov(v10, d, d0, d2), - mov(v11, s, s2, s0), - mov(v12, s, s3, s1), - mov(v10, d, d2, d1)]); + assert_eq!( + solver.moves(), + &[ + mov(v10, d, d0, d2), + mov(v11, s, s2, s0), + mov(v12, s, s3, s1), + mov(v10, d, d2, d1), + ] + ); // Same problem in the other direction: Swap (s0, s1) <-> d1. // @@ -879,10 +899,14 @@ mod tests { solver.inputs_done(); assert!(solver.quick_solve().is_ok()); assert_eq!(solver.schedule_moves(®s), 0); - assert_eq!(solver.moves(), - &[mov(v10, d, d1, d2), - mov(v12, s, s1, s3), - mov(v11, s, s0, s2), - mov(v10, d, d2, d0)]); + assert_eq!( + solver.moves(), + &[ + mov(v10, d, d1, d2), + mov(v12, s, s1, s3), + mov(v11, s, s0, s2), + mov(v10, d, d2, d0), + ] + ); } } diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 35ef526993..4fe14a9528 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -71,14 +71,16 @@ impl Spilling { } /// Run the spilling algorithm over `func`. - pub fn run(&mut self, - isa: &TargetIsa, - func: &mut Function, - domtree: &DominatorTree, - liveness: &mut Liveness, - virtregs: &VirtRegs, - topo: &mut TopoOrder, - tracker: &mut LiveValueTracker) { + pub fn run( + &mut self, + isa: &TargetIsa, + func: &mut Function, + domtree: &DominatorTree, + liveness: &mut Liveness, + virtregs: &VirtRegs, + topo: &mut TopoOrder, + tracker: &mut LiveValueTracker, + ) { dbg!("Spilling for:\n{}", func.display(isa)); let reginfo = isa.register_info(); let usable_regs = isa.allocatable_registers(func); @@ -114,8 +116,10 @@ impl<'a> Context<'a> { while let Some(inst) = self.cur.next_inst() { if let Some(constraints) = - self.encinfo - .operand_constraints(self.cur.func.encodings[inst]) { + self.encinfo.operand_constraints( + self.cur.func.encodings[inst], + ) + { self.visit_inst(inst, ebb, constraints, tracker); } else { let (_throughs, kills) = tracker.process_ghost(inst); @@ -150,11 +154,13 @@ impl<'a> Context<'a> { } fn visit_ebb_header(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) { - let (liveins, args) = tracker.ebb_top(ebb, - &self.cur.func.dfg, - self.liveness, - &self.cur.func.layout, - self.domtree); + let (liveins, args) = tracker.ebb_top( + ebb, + &self.cur.func.dfg, + self.liveness, + &self.cur.func.layout, + self.domtree, + ); // Count the live-in registers. These should already fit in registers; they did at the // dominator. @@ -167,16 +173,20 @@ impl<'a> Context<'a> { if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); 'try_take: while let Err(mask) = self.pressure.take_transient(rc) { - dbg!("Need {} reg for EBB argument {} from {} live-ins", - rc, - lv.value, - liveins.len()); + dbg!( + "Need {} reg for EBB argument {} from {} live-ins", + rc, + lv.value, + liveins.len() + ); match self.spill_candidate(mask, liveins) { Some(cand) => { - dbg!("Spilling live-in {} to make room for {} EBB argument {}", - cand, - rc, - lv.value); + dbg!( + "Spilling live-in {} to make room for {} EBB argument {}", + cand, + rc, + lv.value + ); self.spill_reg(cand); } None => { @@ -199,11 +209,13 @@ impl<'a> Context<'a> { self.pressure.preserve_transient(); } - fn visit_inst(&mut self, - inst: Inst, - ebb: Ebb, - constraints: &RecipeConstraints, - tracker: &mut LiveValueTracker) { + fn visit_inst( + &mut self, + inst: Inst, + ebb: Ebb, + constraints: &RecipeConstraints, + tracker: &mut LiveValueTracker, + ) { dbg!("Inst {}, {}", self.cur.display_inst(inst), self.pressure); debug_assert_eq!(self.cur.current_inst(), Some(inst)); debug_assert_eq!(self.cur.current_ebb(), Some(ebb)); @@ -250,9 +262,11 @@ impl<'a> Context<'a> { match self.spill_candidate(mask, throughs) { Some(cand) => self.spill_reg(cand), None => { - panic!("Ran out of {} registers for {}", - op.regclass, - self.cur.display_inst(inst)) + panic!( + "Ran out of {} registers for {}", + op.regclass, + self.cur.display_inst(inst) + ) } } } @@ -313,12 +327,16 @@ impl<'a> Context<'a> { .argument_types .iter() .zip(args) - .enumerate() { + .enumerate() + { if abi.location.is_reg() { let (rci, spilled) = match self.liveness[arg].affinity { Affinity::Reg(rci) => (rci, false), Affinity::Stack => { - (self.cur.isa.regclass_for_abi_type(abi.value_type).into(), true) + ( + self.cur.isa.regclass_for_abi_type(abi.value_type).into(), + true, + ) } Affinity::None => panic!("Missing affinity for {}", arg), }; @@ -373,17 +391,19 @@ impl<'a> Context<'a> { // Spill a live register that is *not* used by the current instruction. // Spilling a use wouldn't help. match { - let args = self.cur.func.dfg.inst_args(inst); - self.spill_candidate(mask, - tracker.live().iter().filter(|lv| { - !args.contains(&lv.value) - })) - } { + let args = self.cur.func.dfg.inst_args(inst); + self.spill_candidate( + mask, + tracker.live().iter().filter(|lv| !args.contains(&lv.value)), + ) + } { Some(cand) => self.spill_reg(cand), None => { - panic!("Ran out of {} registers when inserting copy before {}", - rc, - self.cur.display_inst(inst)) + panic!( + "Ran out of {} registers when inserting copy before {}", + rc, + self.cur.display_inst(inst) + ) } } } @@ -395,7 +415,8 @@ impl<'a> Context<'a> { // Find a spill candidate from `candidates` whose top-level register class is in `mask`. fn spill_candidate<'ii, II>(&self, mask: RegClassMask, candidates: II) -> Option - where II: IntoIterator + where + II: IntoIterator, { // Find the best viable spill candidate. // @@ -420,12 +441,13 @@ impl<'a> Context<'a> { None }) .min_by(|&a, &b| { - // Find the minimum candidate according to the RPO of their defs. - self.domtree - .rpo_cmp(self.cur.func.dfg.value_def(a), - self.cur.func.dfg.value_def(b), - &self.cur.func.layout) - }) + // Find the minimum candidate according to the RPO of their defs. + self.domtree.rpo_cmp( + self.cur.func.dfg.value_def(a), + self.cur.func.dfg.value_def(b), + &self.cur.func.layout, + ) + }) } /// Spill `value` immediately by @@ -447,10 +469,9 @@ impl<'a> Context<'a> { } // Assign a spill slot for the whole virtual register. - let ss = self.cur - .func - .stack_slots - .make_spill_slot(self.cur.func.dfg.value_type(value)); + let ss = self.cur.func.stack_slots.make_spill_slot( + self.cur.func.dfg.value_type(value), + ); for &v in self.virtregs.congruence_class(&value) { self.liveness.spill(v); self.cur.func.locations[v] = ValueLoc::Stack(ss); @@ -481,11 +502,12 @@ impl<'a> Context<'a> { // Update live ranges. self.liveness.create_dead(copy, inst, Affinity::Reg(rci)); - self.liveness - .extend_locally(copy, - self.cur.func.layout.pp_ebb(inst), - self.cur.current_inst().expect("must be at an instruction"), - &self.cur.func.layout); + self.liveness.extend_locally( + copy, + self.cur.func.layout.pp_ebb(inst), + self.cur.current_inst().expect("must be at an instruction"), + &self.cur.func.layout, + ); copy } diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index 7c98fd83e0..21a2a917d5 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -81,11 +81,12 @@ impl VirtRegs { /// If `value` belongs to a virtual register, the congruence class is the values of the virtual /// register. Otherwise it is just the value itself. pub fn congruence_class<'a, 'b>(&'a self, value: &'b Value) -> &'b [Value] - where 'a: 'b + where + 'a: 'b, { - self.get(*value) - .map(|vr| self.values(vr)) - .unwrap_or(ref_slice(value)) + self.get(*value).map(|vr| self.values(vr)).unwrap_or( + ref_slice(value), + ) } /// Check if `a` and `b` belong to the same congruence class. @@ -126,9 +127,11 @@ impl VirtRegs { .min() .unwrap_or_else(|| self.vregs.push(Default::default())); - assert_eq!(values.len(), - singletons + cleared, - "Can't unify partial virtual registers"); + assert_eq!( + values.len(), + singletons + cleared, + "Can't unify partial virtual registers" + ); self.vregs[vreg].extend(values.iter().cloned(), &mut self.pool); for &v in values { diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 660c833a4a..9738c422af 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -137,8 +137,8 @@ impl Configurable for Builder { self.bytes[offset] = value.parse().map_err(|_| Error::BadValue)?; } Detail::Enum { last, enumerators } => { - self.bytes[offset] = parse_enum_value(value, - self.template.enums(last, enumerators))?; + self.bytes[offset] = + parse_enum_value(value, self.template.enums(last, enumerators))?; } Detail::Preset => return Err(Error::BadName), } @@ -218,11 +218,12 @@ pub mod detail { /// Format a setting value as a TOML string. This is mostly for use by the generated /// `Display` implementation. - pub fn format_toml_value(&self, - detail: Detail, - byte: u8, - f: &mut fmt::Formatter) - -> fmt::Result { + pub fn format_toml_value( + &self, + detail: Detail, + byte: u8, + f: &mut fmt::Formatter, + ) -> fmt::Result { match detail { Detail::Bool { bit } => write!(f, "{}", (byte & (1 << bit)) != 0), Detail::Num => write!(f, "{}", byte), @@ -312,15 +313,17 @@ mod tests { fn display_default() { let b = builder(); let f = Flags::new(&b); - assert_eq!(f.to_string(), - "[shared]\n\ + assert_eq!( + f.to_string(), + "[shared]\n\ opt_level = \"default\"\n\ enable_verifier = false\n\ is_64bit = false\n\ is_compressed = false\n\ enable_float = true\n\ enable_simd = true\n\ - enable_atomics = true\n"); + enable_atomics = true\n" + ); assert_eq!(f.opt_level(), super::OptLevel::Default); assert_eq!(f.enable_simd(), true); } diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index 8a8022ed2d..308d23e083 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -8,7 +8,7 @@ use std::collections::HashMap; /// Test whether the given opcode is unsafe to even consider for GVN. fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || opcode.is_return() || - opcode.can_trap() || opcode.other_side_effects() + opcode.can_trap() || opcode.other_side_effects() } /// Perform simple GVN on `func`. diff --git a/lib/cretonne/src/stack_layout.rs b/lib/cretonne/src/stack_layout.rs index c367849096..7486cee8a6 100644 --- a/lib/cretonne/src/stack_layout.rs +++ b/lib/cretonne/src/stack_layout.rs @@ -51,9 +51,9 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result { - let offset = slot.offset - .checked_add(slot.size as StackOffset) - .ok_or(CtonError::ImplLimitExceeded)?; + let offset = slot.offset.checked_add(slot.size as StackOffset).ok_or( + CtonError::ImplLimitExceeded, + )?; outgoing_max = max(outgoing_max, offset); } StackSlotKind::SpillSlot | StackSlotKind::Local => { @@ -82,9 +82,9 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result continue, } - offset = offset - .checked_sub(slot.size as StackOffset) - .ok_or(CtonError::ImplLimitExceeded)?; + offset = offset.checked_sub(slot.size as StackOffset).ok_or( + CtonError::ImplLimitExceeded, + )?; // Aligning the negative offset can never cause overflow. We're only clearing bits. offset &= -(min_align as StackOffset); @@ -96,9 +96,9 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result(&mut self, preferred: Ebbs) - where Ebbs: IntoIterator + where + Ebbs: IntoIterator, { self.preferred.clear(); self.preferred.extend(preferred); diff --git a/lib/cretonne/src/verifier/cssa.rs b/lib/cretonne/src/verifier/cssa.rs index fe9a2fe18e..e87c3bb39e 100644 --- a/lib/cretonne/src/verifier/cssa.rs +++ b/lib/cretonne/src/verifier/cssa.rs @@ -22,12 +22,13 @@ use verifier::Result; /// - The values in a virtual register are ordered according to the dominator tree's `rpo_cmp()`. /// /// We don't verify that virtual registers are minimal. Minimal CSSA is not required. -pub fn verify_cssa(func: &Function, - cfg: &ControlFlowGraph, - domtree: &DominatorTree, - liveness: &Liveness, - virtregs: &VirtRegs) - -> Result { +pub fn verify_cssa( + func: &Function, + cfg: &ControlFlowGraph, + domtree: &DominatorTree, + liveness: &Liveness, + virtregs: &VirtRegs, +) -> Result { let verifier = CssaVerifier { func, cfg, @@ -77,10 +78,12 @@ impl<'a> CssaVerifier<'a> { return err!(val, "Value in {} has same def as {}", vreg, prev_val); } Ordering::Greater => { - return err!(val, - "Value in {} in wrong order relative to {}", - vreg, - prev_val); + return err!( + val, + "Value in {} in wrong order relative to {}", + vreg, + prev_val + ); } } @@ -102,16 +105,20 @@ impl<'a> CssaVerifier<'a> { for &(_, pred) in self.cfg.get_predecessors(ebb) { let pred_args = self.func.dfg.inst_variable_args(pred); // This should have been caught by an earlier verifier pass. - assert_eq!(ebb_args.len(), - pred_args.len(), - "Wrong arguments on branch."); + assert_eq!( + ebb_args.len(), + pred_args.len(), + "Wrong arguments on branch." + ); for (&ebb_arg, &pred_arg) in ebb_args.iter().zip(pred_args) { if !self.virtregs.same_class(ebb_arg, pred_arg) { - return err!(pred, - "{} and {} must be in the same virtual register", - ebb_arg, - pred_arg); + return err!( + pred, + "{} and {} must be in the same virtual register", + ebb_arg, + pred_arg + ); } } } diff --git a/lib/cretonne/src/verifier/liveness.rs b/lib/cretonne/src/verifier/liveness.rs index 8b1ab47f4c..fa7b23ec13 100644 --- a/lib/cretonne/src/verifier/liveness.rs +++ b/lib/cretonne/src/verifier/liveness.rs @@ -21,11 +21,12 @@ use verifier::Result; /// /// We don't verify that live ranges are minimal. This would require recomputing live ranges for /// all values. -pub fn verify_liveness(isa: &TargetIsa, - func: &Function, - cfg: &ControlFlowGraph, - liveness: &Liveness) - -> Result { +pub fn verify_liveness( + isa: &TargetIsa, + func: &Function, + cfg: &ControlFlowGraph, + liveness: &Liveness, +) -> Result { let verifier = LivenessVerifier { isa, func, @@ -76,18 +77,22 @@ impl<'a> LivenessVerifier<'a> { if encoding.is_legal() { // A legal instruction is not allowed to define ghost values. if lr.affinity.is_none() { - return err!(inst, - "{} is a ghost value defined by a real [{}] instruction", - val, - self.isa.encoding_info().display(encoding)); + return err!( + inst, + "{} is a ghost value defined by a real [{}] instruction", + val, + self.isa.encoding_info().display(encoding) + ); } } else { // A non-encoded instruction can only define ghost values. if !lr.affinity.is_none() { - return err!(inst, - "{} is a real {} value defined by a ghost instruction", - val, - lr.affinity.display(&self.isa.register_info())); + return err!( + inst, + "{} is a real {} value defined by a ghost instruction", + val, + lr.affinity.display(&self.isa.register_info()) + ); } } } @@ -108,10 +113,12 @@ impl<'a> LivenessVerifier<'a> { // A branch argument can be a ghost value if the corresponding destination // EBB argument is a ghost value. if lr.affinity.is_none() && !self.is_ghost_branch_argument(inst, idx) { - return err!(inst, - "{} is a ghost value used by a real [{}] instruction", - val, - self.isa.encoding_info().display(encoding)); + return err!( + inst, + "{} is a ghost value used by a real [{}] instruction", + val, + self.isa.encoding_info().display(encoding) + ); } } } @@ -126,7 +133,8 @@ impl<'a> LivenessVerifier<'a> { // Check if `inst` is in the def range, not including the def itself. if l.cmp(lr.def(), inst) == Ordering::Less && - l.cmp(inst, lr.def_local_end()) != Ordering::Greater { + l.cmp(inst, lr.def_local_end()) != Ordering::Greater + { return true; } @@ -205,11 +213,13 @@ impl<'a> LivenessVerifier<'a> { let end_ebb = match l.inst_ebb(livein.end) { Some(e) => e, None => { - return err!(loc, - "{} livein for {} ends at {} which is not in the layout", - val, - ebb, - livein.end) + return err!( + loc, + "{} livein for {} ends at {} which is not in the layout", + val, + ebb, + livein.end + ) } }; @@ -218,10 +228,12 @@ impl<'a> LivenessVerifier<'a> { // If `val` is live-in at `ebb`, it must be live at all the predecessors. for &(_, pred) in self.cfg.get_predecessors(ebb) { if !self.live_at_use(lr, pred) { - return err!(pred, - "{} is live in to {} but not live at predecessor", - val, - ebb); + return err!( + pred, + "{} is live in to {} but not live at predecessor", + val, + ebb + ); } } diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index ca1f2a8f09..47a419dafd 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -127,11 +127,12 @@ pub fn verify_function(func: &Function, isa: Option<&TargetIsa>) -> Result { /// Verify `func` after checking the integrity of associated context data structures `cfg` and /// `domtree`. -pub fn verify_context(func: &Function, - cfg: &ControlFlowGraph, - domtree: &DominatorTree, - isa: Option<&TargetIsa>) - -> Result { +pub fn verify_context( + func: &Function, + cfg: &ControlFlowGraph, + domtree: &DominatorTree, + isa: Option<&TargetIsa>, +) -> Result { let verifier = Verifier::new(func, isa); verifier.cfg_integrity(cfg)?; if domtree.is_valid() { @@ -187,9 +188,11 @@ impl<'a> Verifier<'a> { if is_terminator && !is_last_inst { // Terminating instructions only occur at the end of blocks. - return err!(inst, - "a terminator instruction was encountered before the end of {}", - ebb); + return err!( + inst, + "a terminator instruction was encountered before the end of {}", + ebb + ); } if is_last_inst && !is_terminator { return err!(ebb, "block does not end in a terminator instruction!"); @@ -237,10 +240,12 @@ impl<'a> Verifier<'a> { // All result values for multi-valued instructions are created let got_results = dfg.inst_results(inst).len(); if got_results != total_results { - return err!(inst, - "expected {} result values, found {}", - total_results, - got_results); + return err!( + inst, + "expected {} result values, found {}", + total_results, + got_results + ); } self.verify_entity_references(inst) @@ -407,22 +412,30 @@ impl<'a> Verifier<'a> { ValueDef::Res(def_inst, _) => { // Value is defined by an instruction that exists. if !dfg.inst_is_valid(def_inst) { - return err!(loc_inst, - "{} is defined by invalid instruction {}", - v, - def_inst); + return err!( + loc_inst, + "{} is defined by invalid instruction {}", + v, + def_inst + ); } // Defining instruction is inserted in an EBB. if self.func.layout.inst_ebb(def_inst) == None { - return err!(loc_inst, - "{} is defined by {} which has no EBB", - v, - def_inst); + return err!( + loc_inst, + "{} is defined by {} which has no EBB", + v, + def_inst + ); } // Defining instruction dominates the instruction that uses the value. if self.domtree.is_reachable(self.func.layout.pp_ebb(loc_inst)) && - !self.domtree - .dominates(def_inst, loc_inst, &self.func.layout) { + !self.domtree.dominates( + def_inst, + loc_inst, + &self.func.layout, + ) + { return err!(loc_inst, "uses value from non-dominating {}", def_inst); } } @@ -433,14 +446,17 @@ impl<'a> Verifier<'a> { } // Defining EBB is inserted in the layout if !self.func.layout.is_ebb_inserted(ebb) { - return err!(loc_inst, - "{} is defined by {} which is not in the layout", - v, - ebb); + return err!( + loc_inst, + "{} is defined by {} which is not in the layout", + v, + ebb + ); } // The defining EBB dominates the instruction using this value. if self.domtree.is_reachable(ebb) && - !self.domtree.dominates(ebb, loc_inst, &self.func.layout) { + !self.domtree.dominates(ebb, loc_inst, &self.func.layout) + { return err!(loc_inst, "uses value arg from non-dominating {}", ebb); } } @@ -456,39 +472,48 @@ impl<'a> Verifier<'a> { let expected = domtree.idom(ebb); let got = self.domtree.idom(ebb); if got != expected { - return err!(ebb, - "invalid domtree, expected idom({}) = {:?}, got {:?}", - ebb, - expected, - got); + return err!( + ebb, + "invalid domtree, expected idom({}) = {:?}, got {:?}", + ebb, + expected, + got + ); } } // We also verify if the postorder defined by `DominatorTree` is sane if self.domtree.cfg_postorder().len() != domtree.cfg_postorder().len() { - return err!(AnyEntity::Function, - "incorrect number of Ebbs in postorder traversal"); + return err!( + AnyEntity::Function, + "incorrect number of Ebbs in postorder traversal" + ); } for (index, (&true_ebb, &test_ebb)) in self.domtree .cfg_postorder() .iter() .zip(domtree.cfg_postorder().iter()) - .enumerate() { + .enumerate() + { if true_ebb != test_ebb { - return err!(test_ebb, - "invalid domtree, postorder ebb number {} should be {}, got {}", - index, - true_ebb, - test_ebb); + return err!( + test_ebb, + "invalid domtree, postorder ebb number {} should be {}, got {}", + index, + true_ebb, + test_ebb + ); } } // We verify rpo_cmp on pairs of adjacent ebbs in the postorder for (&prev_ebb, &next_ebb) in self.domtree.cfg_postorder().iter().adjacent_pairs() { if domtree.rpo_cmp(prev_ebb, next_ebb, &self.func.layout) != Ordering::Greater { - return err!(next_ebb, - "invalid domtree, rpo_cmp does not says {} is greater than {}", - prev_ebb, - next_ebb); + return err!( + next_ebb, + "invalid domtree, rpo_cmp does not says {} is greater than {}", + prev_ebb, + next_ebb + ); } } Ok(()) @@ -506,11 +531,13 @@ impl<'a> Verifier<'a> { for (i, &arg) in self.func.dfg.ebb_args(ebb).iter().enumerate() { let arg_type = self.func.dfg.value_type(arg); if arg_type != expected_types[i].value_type { - return err!(ebb, - "entry block argument {} expected to have type {}, got {}", - i, - expected_types[i], - arg_type); + return err!( + ebb, + "entry block argument {} expected to have type {}, got {}", + i, + expected_types[i], + arg_type + ); } } } @@ -551,12 +578,14 @@ impl<'a> Verifier<'a> { let expected_type = self.func.dfg.compute_result_type(inst, i, ctrl_type); if let Some(expected_type) = expected_type { if result_type != expected_type { - return err!(inst, - "expected result {} ({}) to have type {}, found {}", - i, - result, - expected_type, - result_type); + return err!( + inst, + "expected result {} ({}) to have type {}, found {}", + i, + result, + expected_type, + result_type + ); } } else { return err!(inst, "has more result values than expected"); @@ -579,22 +608,26 @@ impl<'a> Verifier<'a> { match constraints.value_argument_constraint(i, ctrl_type) { ResolvedConstraint::Bound(expected_type) => { if arg_type != expected_type { - return err!(inst, - "arg {} ({}) has type {}, expected {}", - i, - arg, - arg_type, - expected_type); + return err!( + inst, + "arg {} ({}) has type {}, expected {}", + i, + arg, + arg_type, + expected_type + ); } } ResolvedConstraint::Free(type_set) => { if !type_set.contains(arg_type) { - return err!(inst, - "arg {} ({}) with type {} failed to satisfy type set {:?}", - i, - arg, - arg_type, - type_set); + return err!( + inst, + "arg {} ({}) with type {} failed to satisfy type set {:?}", + i, + arg, + arg_type, + type_set + ); } } } @@ -605,21 +638,21 @@ impl<'a> Verifier<'a> { fn typecheck_variable_args(&self, inst: Inst) -> Result { match self.func.dfg[inst].analyze_branch(&self.func.dfg.value_lists) { BranchInfo::SingleDest(ebb, _) => { - let iter = self.func - .dfg - .ebb_args(ebb) - .iter() - .map(|&v| self.func.dfg.value_type(v)); + let iter = self.func.dfg.ebb_args(ebb).iter().map(|&v| { + self.func.dfg.value_type(v) + }); self.typecheck_variable_args_iterator(inst, iter)?; } BranchInfo::Table(table) => { for (_, ebb) in self.func.jump_tables[table].entries() { let arg_count = self.func.dfg.num_ebb_args(ebb); if arg_count != 0 { - return err!(inst, - "takes no arguments, but had target {} with {} arguments", - ebb, - arg_count); + return err!( + inst, + "takes no arguments, but had target {} with {} arguments", + ebb, + arg_count + ); } } } @@ -649,10 +682,11 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck_variable_args_iterator>(&self, - inst: Inst, - iter: I) - -> Result { + fn typecheck_variable_args_iterator>( + &self, + inst: Inst, + iter: I, + ) -> Result { let variable_args = self.func.dfg.inst_variable_args(inst); let mut i = 0; @@ -665,20 +699,24 @@ impl<'a> Verifier<'a> { let arg = variable_args[i]; let arg_type = self.func.dfg.value_type(arg); if expected_type != arg_type { - return err!(inst, - "arg {} ({}) has type {}, expected {}", - i, - variable_args[i], - arg_type, - expected_type); + return err!( + inst, + "arg {} ({}) has type {}, expected {}", + i, + variable_args[i], + arg_type, + expected_type + ); } i += 1; } if i != variable_args.len() { - return err!(inst, - "mismatched argument count, got {}, expected {}", - variable_args.len(), - i); + return err!( + inst, + "mismatched argument count, got {}, expected {}", + variable_args.len(), + i + ); } Ok(()) } @@ -707,34 +745,42 @@ impl<'a> Verifier<'a> { self.verify_stack_slot(inst, ss)?; let slot = &self.func.stack_slots[ss]; if slot.kind != StackSlotKind::OutgoingArg { - return err!(inst, - "Outgoing stack argument {} in wrong stack slot: {} = {}", - arg, - ss, - slot); + return err!( + inst, + "Outgoing stack argument {} in wrong stack slot: {} = {}", + arg, + ss, + slot + ); } if slot.offset != offset { - return err!(inst, - "Outgoing stack argument {} should have offset {}: {} = {}", - arg, - offset, - ss, - slot); + return err!( + inst, + "Outgoing stack argument {} should have offset {}: {} = {}", + arg, + offset, + ss, + slot + ); } if slot.size != abi.value_type.bytes() { - return err!(inst, - "Outgoing stack argument {} wrong size for {}: {} = {}", - arg, - abi.value_type, - ss, - slot); + return err!( + inst, + "Outgoing stack argument {} wrong size for {}: {} = {}", + arg, + abi.value_type, + ss, + slot + ); } } else { let reginfo = self.isa.map(|i| i.register_info()); - return err!(inst, - "Outgoing stack argument {} in wrong location: {}", - arg, - arg_loc.display(reginfo.as_ref())); + return err!( + inst, + "Outgoing stack argument {} in wrong location: {}", + arg, + arg_loc.display(reginfo.as_ref()) + ); } } } @@ -751,12 +797,14 @@ impl<'a> Verifier<'a> { for (i, (&arg, &expected_type)) in args.iter().zip(expected_types).enumerate() { let arg_type = self.func.dfg.value_type(arg); if arg_type != expected_type.value_type { - return err!(inst, - "arg {} ({}) has type {}, must match function signature of {}", - i, - arg, - arg_type, - expected_type); + return err!( + inst, + "arg {} ({}) has type {}, must match function signature of {}", + i, + arg, + arg_type, + expected_type + ); } } } @@ -775,9 +823,11 @@ impl<'a> Verifier<'a> { let missing_succs: Vec = expected_succs.difference(&got_succs).cloned().collect(); if !missing_succs.is_empty() { - return err!(ebb, - "cfg lacked the following successor(s) {:?}", - missing_succs); + return err!( + ebb, + "cfg lacked the following successor(s) {:?}", + missing_succs + ); } let excess_succs: Vec = got_succs.difference(&expected_succs).cloned().collect(); @@ -790,9 +840,11 @@ impl<'a> Verifier<'a> { let missing_preds: Vec = expected_preds.difference(&got_preds).cloned().collect(); if !missing_preds.is_empty() { - return err!(ebb, - "cfg lacked the following predecessor(s) {:?}", - missing_preds); + return err!( + ebb, + "cfg lacked the following predecessor(s) {:?}", + missing_preds + ); } let excess_preds: Vec = got_preds.difference(&expected_preds).cloned().collect(); @@ -826,23 +878,28 @@ impl<'a> Verifier<'a> { let encoding = self.func.encodings[inst]; if encoding.is_legal() { - let verify_encoding = - isa.encode(&self.func.dfg, - &self.func.dfg[inst], - self.func.dfg.ctrl_typevar(inst)); + let verify_encoding = isa.encode( + &self.func.dfg, + &self.func.dfg[inst], + self.func.dfg.ctrl_typevar(inst), + ); match verify_encoding { Ok(verify_encoding) => { if verify_encoding != encoding { - return err!(inst, - "Instruction re-encoding {} doesn't match {}", - isa.encoding_info().display(verify_encoding), - isa.encoding_info().display(encoding)); + return err!( + inst, + "Instruction re-encoding {} doesn't match {}", + isa.encoding_info().display(verify_encoding), + isa.encoding_info().display(encoding) + ); } } Err(_) => { - return err!(inst, - "Instruction failed to re-encode {}", - isa.encoding_info().display(encoding)) + return err!( + inst, + "Instruction failed to re-encode {}", + isa.encoding_info().display(encoding) + ) } } return Ok(()); @@ -932,9 +989,9 @@ mod tests { let mut func = Function::new(); let ebb0 = func.dfg.make_ebb(); func.layout.append_ebb(ebb0); - let nullary_with_bad_opcode = - func.dfg - .make_inst(InstructionData::Nullary { opcode: Opcode::Jump }); + let nullary_with_bad_opcode = func.dfg.make_inst( + InstructionData::Nullary { opcode: Opcode::Jump }, + ); func.layout.append_inst(nullary_with_bad_opcode, ebb0); let verifier = Verifier::new(&func, None); assert_err_with_msg!(verifier.run(), "instruction format"); diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 3be1bc3cd1..f69586b2ea 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -38,10 +38,11 @@ fn write_spec(w: &mut Write, func: &Function, regs: Option<&RegInfo>) -> Result write!(w, "function {}{}", func.name, func.signature.display(regs)) } -fn write_preamble(w: &mut Write, - func: &Function, - regs: Option<&RegInfo>) - -> result::Result { +fn write_preamble( + w: &mut Write, + func: &Function, + regs: Option<&RegInfo>, +) -> result::Result { let mut any = false; for ss in func.stack_slots.keys() { @@ -63,10 +64,12 @@ fn write_preamble(w: &mut Write, // signatures. for sig in func.dfg.signatures.keys() { any = true; - writeln!(w, - " {} = {}", - sig, - func.dfg.signatures[sig].display(regs))?; + writeln!( + w, + " {} = {}", + sig, + func.dfg.signatures[sig].display(regs) + )?; } for fnref in func.dfg.ext_funcs.keys() { @@ -163,8 +166,10 @@ fn type_suffix(func: &Function, inst: Inst) -> Option { } let rtype = func.dfg.ctrl_typevar(inst); - assert!(!rtype.is_void(), - "Polymorphic instruction must produce a result"); + assert!( + !rtype.is_void(), + "Polymorphic instruction must produce a result" + ); Some(rtype) } @@ -179,11 +184,12 @@ fn write_value_aliases(w: &mut Write, func: &Function, inst: Inst, indent: usize Ok(()) } -fn write_instruction(w: &mut Write, - func: &Function, - isa: Option<&TargetIsa>, - inst: Inst) - -> Result { +fn write_instruction( + w: &mut Write, + func: &Function, + isa: Option<&TargetIsa>, + inst: Inst, +) -> Result { // Indent all instructions to col 24 if any encodings are present. let indent = if func.encodings.is_empty() { 4 } else { 24 }; @@ -240,11 +246,12 @@ fn write_instruction(w: &mut Write, } /// Write the operands of `inst` to `w` with a prepended space. -pub fn write_operands(w: &mut Write, - dfg: &DataFlowGraph, - isa: Option<&TargetIsa>, - inst: Inst) - -> Result { +pub fn write_operands( + w: &mut Write, + dfg: &DataFlowGraph, + isa: Option<&TargetIsa>, + inst: Inst, +) -> Result { let pool = &dfg.value_lists; use ir::instructions::InstructionData::*; match dfg[inst] { @@ -278,10 +285,12 @@ pub fn write_operands(w: &mut Write, if args.is_empty() { write!(w, " {}", destination) } else { - write!(w, - " {}({})", - destination, - DisplayValues(args.as_slice(pool))) + write!( + w, + " {}({})", + destination, + DisplayValues(args.as_slice(pool)) + ) } } Branch { @@ -315,11 +324,13 @@ pub fn write_operands(w: &mut Write, } IndirectCall { sig_ref, ref args, .. } => { let args = args.as_slice(pool); - write!(w, - " {}, {}({})", - sig_ref, - args[0], - DisplayValues(&args[1..])) + write!( + w, + " {}, {}({})", + sig_ref, + args[0], + DisplayValues(&args[1..]) + ) } StackLoad { stack_slot, offset, .. } => write!(w, " {}{}", stack_slot, offset), StackStore { @@ -341,11 +352,13 @@ pub fn write_operands(w: &mut Write, RegMove { arg, src, dst, .. } => { if let Some(isa) = isa { let regs = isa.register_info(); - write!(w, - " {}, {} -> {}", - arg, - regs.display_regunit(src), - regs.display_regunit(dst)) + write!( + w, + " {}, {} -> {}", + arg, + regs.display_regunit(src), + regs.display_regunit(dst) + ) } else { write!(w, " {}, %{} -> %{}", arg, src, dst) } @@ -382,22 +395,31 @@ mod tests { f.name = FunctionName::new("foo"); assert_eq!(f.to_string(), "function %foo() native {\n}\n"); - f.stack_slots - .push(StackSlotData::new(StackSlotKind::Local, 4)); - assert_eq!(f.to_string(), - "function %foo() native {\n ss0 = local 4\n}\n"); + f.stack_slots.push( + StackSlotData::new(StackSlotKind::Local, 4), + ); + assert_eq!( + f.to_string(), + "function %foo() native {\n ss0 = local 4\n}\n" + ); let ebb = f.dfg.make_ebb(); f.layout.append_ebb(ebb); - assert_eq!(f.to_string(), - "function %foo() native {\n ss0 = local 4\n\nebb0:\n}\n"); + assert_eq!( + f.to_string(), + "function %foo() native {\n ss0 = local 4\n\nebb0:\n}\n" + ); f.dfg.append_ebb_arg(ebb, types::I8); - assert_eq!(f.to_string(), - "function %foo() native {\n ss0 = local 4\n\nebb0(v0: i8):\n}\n"); + assert_eq!( + f.to_string(), + "function %foo() native {\n ss0 = local 4\n\nebb0(v0: i8):\n}\n" + ); f.dfg.append_ebb_arg(ebb, types::F32.by(4).unwrap()); - assert_eq!(f.to_string(), - "function %foo() native {\n ss0 = local 4\n\nebb0(v0: i8, v1: f32x4):\n}\n"); + assert_eq!( + f.to_string(), + "function %foo() native {\n ss0 = local 4\n\nebb0(v0: i8, v1: f32x4):\n}\n" + ); } } diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs index 21d9049fee..19d8ae8092 100644 --- a/lib/filecheck/src/checker.rs +++ b/lib/filecheck/src/checker.rs @@ -47,9 +47,11 @@ impl Directive { "unordered" => Ok(Directive::Unordered(pat)), "not" => { if !pat.defs().is_empty() { - let msg = format!("can't define variables '$({}=...' in not: {}", - pat.defs()[0], - rest); + let msg = format!( + "can't define variables '$({}=...' in not: {}", + pat.defs()[0], + rest + ); Err(Error::DuplicateDef(msg)) } else { Ok(Directive::Not(pat)) @@ -63,16 +65,23 @@ impl Directive { fn regex(rest: &str) -> Result { let varlen = varname_prefix(rest); if varlen == 0 { - return Err(Error::Syntax(format!("invalid variable name in regex: {}", rest))); + return Err(Error::Syntax( + format!("invalid variable name in regex: {}", rest), + )); } let var = rest[0..varlen].to_string(); if !rest[varlen..].starts_with('=') { - return Err(Error::Syntax(format!("expected '=' after variable '{}' in regex: {}", - var, - rest))); + return Err(Error::Syntax(format!( + "expected '=' after variable '{}' in regex: {}", + var, + rest + ))); } // Ignore trailing white space in the regex, including CR. - Ok(Directive::Regex(var, rest[varlen + 1..].trim_right().to_string())) + Ok(Directive::Regex( + var, + rest[varlen + 1..].trim_right().to_string(), + )) } } @@ -183,13 +192,13 @@ impl Checker { continue; } Directive::Regex(ref var, ref rx) => { - state - .vars - .insert(var.clone(), - VarDef { - value: Value::Regex(Cow::Borrowed(rx)), - offset: 0, - }); + state.vars.insert( + var.clone(), + VarDef { + value: Value::Regex(Cow::Borrowed(rx)), + offset: 0, + }, + ); continue; } }; @@ -210,15 +219,16 @@ impl Checker { state.recorder.directive(not_idx); if let Some(mat) = rx.find(&text[not_begin..match_begin]) { // Matched `not:` pattern. - state - .recorder - .matched_not(rx.as_str(), - (not_begin + mat.start(), not_begin + mat.end())); + state.recorder.matched_not(rx.as_str(), ( + not_begin + mat.start(), + not_begin + mat.end(), + )); return Ok(false); } else { - state - .recorder - .missed_not(rx.as_str(), (not_begin, match_begin)); + state.recorder.missed_not( + rx.as_str(), + (not_begin, match_begin), + ); } } } @@ -354,13 +364,13 @@ impl<'a> State<'a> { }) }; Ok(if let Some(mat) = matched_range { - let r = (range.0 + mat.start(), range.0 + mat.end()); - self.recorder.matched_check(rx.as_str(), r); - Some(r) - } else { - self.recorder.missed_check(rx.as_str(), range); - None - }) + let r = (range.0 + mat.start(), range.0 + mat.end()); + self.recorder.matched_check(rx.as_str(), r); + Some(r) + } else { + self.recorder.missed_check(rx.as_str(), range); + None + }) } } @@ -413,20 +423,32 @@ mod tests { let mut b = CheckerBuilder::new(); assert_eq!(b.directive("not here: more text").map_err(e2s), Ok(false)); - assert_eq!(b.directive("not here: regex: X=more text").map_err(e2s), - Ok(true)); - assert_eq!(b.directive("regex: X = tommy").map_err(e2s), - Err("expected '=' after variable 'X' in regex: X = tommy".to_string())); - assert_eq!(b.directive("[arm]not: patt $x $(y) here").map_err(e2s), - Ok(true)); - assert_eq!(b.directive("[x86]sameln: $x $(y=[^]]*) there").map_err(e2s), - Ok(true)); + assert_eq!( + b.directive("not here: regex: X=more text").map_err(e2s), + Ok(true) + ); + assert_eq!( + b.directive("regex: X = tommy").map_err(e2s), + Err( + "expected '=' after variable 'X' in regex: X = tommy".to_string(), + ) + ); + assert_eq!( + b.directive("[arm]not: patt $x $(y) here").map_err(e2s), + Ok(true) + ); + assert_eq!( + b.directive("[x86]sameln: $x $(y=[^]]*) there").map_err(e2s), + Ok(true) + ); // Windows line ending sneaking in. assert_eq!(b.directive("regex: Y=foo\r").map_err(e2s), Ok(true)); let c = b.finish(); - assert_eq!(c.to_string(), - "#0 regex: X=more text\n#1 not: patt $(x) $(y) here\n#2 sameln: $(x) \ - $(y=[^]]*) there\n#3 regex: Y=foo\n"); + assert_eq!( + c.to_string(), + "#0 regex: X=more text\n#1 not: patt $(x) $(y) here\n#2 sameln: $(x) \ + $(y=[^]]*) there\n#3 regex: Y=foo\n" + ); } } diff --git a/lib/filecheck/src/explain.rs b/lib/filecheck/src/explain.rs index f268620d11..137d360f89 100644 --- a/lib/filecheck/src/explain.rs +++ b/lib/filecheck/src/explain.rs @@ -111,16 +111,21 @@ impl<'a> Display for Explainer<'a> { } // Emit the match message itself. - writeln!(f, - "{} #{}{}: {}", - if m.is_match { "Matched" } else { "Missed" }, - m.directive, - if m.is_not { " not" } else { "" }, - m.regex)?; + writeln!( + f, + "{} #{}{}: {}", + if m.is_match { "Matched" } else { "Missed" }, + m.directive, + if m.is_not { " not" } else { "" }, + m.regex + )?; // Emit any variable definitions. - if let Ok(found) = self.vardefs - .binary_search_by_key(&m.directive, |v| v.directive) { + if let Ok(found) = self.vardefs.binary_search_by_key( + &m.directive, + |v| v.directive, + ) + { let mut first = found; while first > 0 && self.vardefs[first - 1].directive == m.directive { first -= 1; @@ -148,55 +153,50 @@ impl<'a> Recorder for Explainer<'a> { } fn matched_check(&mut self, regex: &str, matched: MatchRange) { - self.matches - .push(Match { - directive: self.directive, - is_match: true, - is_not: false, - regex: regex.to_owned(), - range: matched, - }); + self.matches.push(Match { + directive: self.directive, + is_match: true, + is_not: false, + regex: regex.to_owned(), + range: matched, + }); } fn matched_not(&mut self, regex: &str, matched: MatchRange) { - self.matches - .push(Match { - directive: self.directive, - is_match: true, - is_not: true, - regex: regex.to_owned(), - range: matched, - }); + self.matches.push(Match { + directive: self.directive, + is_match: true, + is_not: true, + regex: regex.to_owned(), + range: matched, + }); } fn missed_check(&mut self, regex: &str, searched: MatchRange) { - self.matches - .push(Match { - directive: self.directive, - is_match: false, - is_not: false, - regex: regex.to_owned(), - range: searched, - }); + self.matches.push(Match { + directive: self.directive, + is_match: false, + is_not: false, + regex: regex.to_owned(), + range: searched, + }); } fn missed_not(&mut self, regex: &str, searched: MatchRange) { - self.matches - .push(Match { - directive: self.directive, - is_match: false, - is_not: true, - regex: regex.to_owned(), - range: searched, - }); + self.matches.push(Match { + directive: self.directive, + is_match: false, + is_not: true, + regex: regex.to_owned(), + range: searched, + }); } fn defined_var(&mut self, varname: &str, value: &str) { - self.vardefs - .push(VarDef { - directive: self.directive, - varname: varname.to_owned(), - value: value.to_owned(), - }); + self.vardefs.push(VarDef { + directive: self.directive, + varname: varname.to_owned(), + value: value.to_owned(), + }); } } diff --git a/lib/filecheck/src/pattern.rs b/lib/filecheck/src/pattern.rs index 7eb3c8fa30..0d1da57f12 100644 --- a/lib/filecheck/src/pattern.rs +++ b/lib/filecheck/src/pattern.rs @@ -70,7 +70,9 @@ impl Pattern { /// Return the allocated def number. fn add_def(&mut self, v: &str) -> Result { if self.defines_var(v) { - Err(Error::DuplicateDef(format!("duplicate definition of ${} in same pattern", v))) + Err(Error::DuplicateDef( + format!("duplicate definition of ${} in same pattern", v), + )) } else { let idx = self.defs.len(); self.defs.push(v.to_string()); @@ -111,8 +113,10 @@ impl Pattern { // All remaining possibilities start with `$(`. if s.len() < 2 || !s.starts_with("$(") { - return Err(Error::Syntax("pattern syntax error, use $$ to match a single $" - .to_string())); + return Err(Error::Syntax( + "pattern syntax error, use $$ to match a single $" + .to_string(), + )); } // Match the variable name, allowing for an empty varname in `$()`, or `$(=...)`. @@ -137,7 +141,9 @@ impl Pattern { // Variable definition. Fall through. } Some(ch) => { - return Err(Error::Syntax(format!("syntax error in $({}... '{}'", varname, ch))); + return Err(Error::Syntax( + format!("syntax error in $({}... '{}'", varname, ch), + )); } } @@ -155,23 +161,31 @@ impl Pattern { let refname_begin = varname_end + 2; let refname_end = refname_begin + varname_prefix(&s[refname_begin..]); if refname_begin == refname_end { - return Err(Error::Syntax(format!("expected variable name in $({}=$...", varname))); + return Err(Error::Syntax( + format!("expected variable name in $({}=$...", varname), + )); } if !s[refname_end..].starts_with(')') { - return Err(Error::Syntax(format!("expected ')' after $({}=${}...", - varname, - &s[refname_begin..refname_end]))); + return Err(Error::Syntax(format!( + "expected ')' after $({}=${}...", + varname, + &s[refname_begin..refname_end] + ))); } let refname = s[refname_begin..refname_end].to_string(); return if let Some(defidx) = def { - Ok((Part::DefVar { - def: defidx, - var: refname, - }, - refname_end + 1)) - } else { - Err(Error::Syntax(format!("expected variable name in $(=${})", refname))) - }; + Ok(( + Part::DefVar { + def: defidx, + var: refname, + }, + refname_end + 1, + )) + } else { + Err(Error::Syntax( + format!("expected variable name in $(=${})", refname), + )) + }; } // Last case: `$(var=...)` where `...` is a regular expression, possibly containing matched @@ -193,9 +207,11 @@ impl Pattern { }; Ok((part, rx_end + 1)) } else { - Err(Error::Syntax(format!("missing ')' after regex in $({}={}", - varname, - &s[rx_begin..rx_end]))) + Err(Error::Syntax(format!( + "missing ')' after regex in $({}={}", + varname, + &s[rx_begin..rx_end] + ))) } } } @@ -273,9 +289,11 @@ impl FromStr for Pattern { let (part, len) = pat.parse_part(&s[pos..])?; if let Some(v) = part.ref_var() { if pat.defines_var(v) { - return Err(Error::Backref(format!("unsupported back-reference to '${}' \ + return Err(Error::Backref(format!( + "unsupported back-reference to '${}' \ defined in same pattern", - v))); + v + ))); } } pat.parts.push(part); @@ -410,49 +428,87 @@ mod tests { // This is dubious, should we panic instead? assert_eq!(pat.parse_part("").unwrap(), (Part::Text("".to_string()), 0)); - assert_eq!(pat.parse_part("x").unwrap(), - (Part::Text("x".to_string()), 1)); - assert_eq!(pat.parse_part("x2").unwrap(), - (Part::Text("x2".to_string()), 2)); - assert_eq!(pat.parse_part("x$").unwrap(), - (Part::Text("x".to_string()), 1)); - assert_eq!(pat.parse_part("x$$").unwrap(), - (Part::Text("x".to_string()), 1)); + assert_eq!( + pat.parse_part("x").unwrap(), + (Part::Text("x".to_string()), 1) + ); + assert_eq!(pat.parse_part("x2").unwrap(), ( + Part::Text("x2".to_string()), + 2, + )); + assert_eq!(pat.parse_part("x$").unwrap(), ( + Part::Text("x".to_string()), + 1, + )); + assert_eq!(pat.parse_part("x$$").unwrap(), ( + Part::Text("x".to_string()), + 1, + )); - assert_eq!(pat.parse_part("$").unwrap_err().to_string(), - "pattern syntax error, use $$ to match a single $"); + assert_eq!( + pat.parse_part("$").unwrap_err().to_string(), + "pattern syntax error, use $$ to match a single $" + ); - assert_eq!(pat.parse_part("$$").unwrap(), - (Part::Text("$".to_string()), 2)); - assert_eq!(pat.parse_part("$$ ").unwrap(), - (Part::Text("$".to_string()), 2)); + assert_eq!(pat.parse_part("$$").unwrap(), ( + Part::Text("$".to_string()), + 2, + )); + assert_eq!(pat.parse_part("$$ ").unwrap(), ( + Part::Text("$".to_string()), + 2, + )); - assert_eq!(pat.parse_part("$0").unwrap(), - (Part::Var("0".to_string()), 2)); - assert_eq!(pat.parse_part("$xx=").unwrap(), - (Part::Var("xx".to_string()), 3)); - assert_eq!(pat.parse_part("$xx$").unwrap(), - (Part::Var("xx".to_string()), 3)); + assert_eq!( + pat.parse_part("$0").unwrap(), + (Part::Var("0".to_string()), 2) + ); + assert_eq!(pat.parse_part("$xx=").unwrap(), ( + Part::Var("xx".to_string()), + 3, + )); + assert_eq!(pat.parse_part("$xx$").unwrap(), ( + Part::Var("xx".to_string()), + 3, + )); - assert_eq!(pat.parse_part("$(0)").unwrap(), - (Part::Var("0".to_string()), 4)); - assert_eq!(pat.parse_part("$()").unwrap(), - (Part::Text("".to_string()), 3)); + assert_eq!(pat.parse_part("$(0)").unwrap(), ( + Part::Var("0".to_string()), + 4, + )); + assert_eq!(pat.parse_part("$()").unwrap(), ( + Part::Text("".to_string()), + 3, + )); - assert_eq!(pat.parse_part("$(0").unwrap_err().to_string(), - ("unterminated $(0...")); - assert_eq!(pat.parse_part("$(foo:").unwrap_err().to_string(), - ("syntax error in $(foo... ':'")); - assert_eq!(pat.parse_part("$(foo =").unwrap_err().to_string(), - ("syntax error in $(foo... ' '")); - assert_eq!(pat.parse_part("$(eo0=$bar").unwrap_err().to_string(), - ("expected ')' after $(eo0=$bar...")); - assert_eq!(pat.parse_part("$(eo1=$bar}").unwrap_err().to_string(), - ("expected ')' after $(eo1=$bar...")); - assert_eq!(pat.parse_part("$(eo2=$)").unwrap_err().to_string(), - ("expected variable name in $(eo2=$...")); - assert_eq!(pat.parse_part("$(eo3=$-)").unwrap_err().to_string(), - ("expected variable name in $(eo3=$...")); + assert_eq!( + pat.parse_part("$(0").unwrap_err().to_string(), + ("unterminated $(0...") + ); + assert_eq!( + pat.parse_part("$(foo:").unwrap_err().to_string(), + ("syntax error in $(foo... ':'") + ); + assert_eq!( + pat.parse_part("$(foo =").unwrap_err().to_string(), + ("syntax error in $(foo... ' '") + ); + assert_eq!( + pat.parse_part("$(eo0=$bar").unwrap_err().to_string(), + ("expected ')' after $(eo0=$bar...") + ); + assert_eq!( + pat.parse_part("$(eo1=$bar}").unwrap_err().to_string(), + ("expected ')' after $(eo1=$bar...") + ); + assert_eq!( + pat.parse_part("$(eo2=$)").unwrap_err().to_string(), + ("expected variable name in $(eo2=$...") + ); + assert_eq!( + pat.parse_part("$(eo3=$-)").unwrap_err().to_string(), + ("expected variable name in $(eo3=$...") + ); } #[test] @@ -460,48 +516,65 @@ mod tests { use super::{Pattern, Part}; let mut pat = Pattern::new(); - assert_eq!(pat.parse_part("$(foo=$bar)").unwrap(), - (Part::DefVar { - def: 0, - var: "bar".to_string(), - }, - 11)); - assert_eq!(pat.parse_part("$(foo=$bar)").unwrap_err().to_string(), - "duplicate definition of $foo in same pattern"); + assert_eq!(pat.parse_part("$(foo=$bar)").unwrap(), ( + Part::DefVar { + def: 0, + var: "bar".to_string(), + }, + 11, + )); + assert_eq!( + pat.parse_part("$(foo=$bar)").unwrap_err().to_string(), + "duplicate definition of $foo in same pattern" + ); - assert_eq!(pat.parse_part("$(fxo=$bar)x").unwrap(), - (Part::DefVar { - def: 1, - var: "bar".to_string(), - }, - 11)); + assert_eq!(pat.parse_part("$(fxo=$bar)x").unwrap(), ( + Part::DefVar { + def: 1, + var: "bar".to_string(), + }, + 11, + )); - assert_eq!(pat.parse_part("$(fo2=[a-z])").unwrap(), - (Part::DefLit { - def: 2, - regex: "(?P[a-z])".to_string(), - }, - 12)); - assert_eq!(pat.parse_part("$(fo3=[a-)])").unwrap(), - (Part::DefLit { - def: 3, - regex: "(?P[a-)])".to_string(), - }, - 12)); - assert_eq!(pat.parse_part("$(fo4=)").unwrap(), - (Part::DefLit { - def: 4, - regex: "(?P)".to_string(), - }, - 7)); + assert_eq!(pat.parse_part("$(fo2=[a-z])").unwrap(), ( + Part::DefLit { + def: 2, + regex: "(?P[a-z])".to_string(), + }, + 12, + )); + assert_eq!(pat.parse_part("$(fo3=[a-)])").unwrap(), ( + Part::DefLit { + def: 3, + regex: "(?P[a-)])".to_string(), + }, + 12, + )); + assert_eq!(pat.parse_part("$(fo4=)").unwrap(), ( + Part::DefLit { + def: 4, + regex: "(?P)".to_string(), + }, + 7, + )); - assert_eq!(pat.parse_part("$(=.*)").unwrap(), - (Part::Regex("(?:.*)".to_string()), 6)); + assert_eq!(pat.parse_part("$(=.*)").unwrap(), ( + Part::Regex( + "(?:.*)".to_string(), + ), + 6, + )); - assert_eq!(pat.parse_part("$(=)").unwrap(), - (Part::Regex("(?:)".to_string()), 4)); - assert_eq!(pat.parse_part("$()").unwrap(), - (Part::Text("".to_string()), 3)); + assert_eq!(pat.parse_part("$(=)").unwrap(), ( + Part::Regex( + "(?:)".to_string(), + ), + 4, + )); + assert_eq!(pat.parse_part("$()").unwrap(), ( + Part::Text("".to_string()), + 3, + )); } #[test] @@ -512,7 +585,9 @@ mod tests { assert_eq!(format!("{:?}", p.parts), "[Text(\"Hello world!\")]"); let p: Pattern = " $foo=$(bar) ".parse().unwrap(); - assert_eq!(format!("{:?}", p.parts), - "[Var(\"foo\"), Text(\"=\"), Var(\"bar\")]"); + assert_eq!( + format!("{:?}", p.parts), + "[Var(\"foo\"), Text(\"=\"), Var(\"bar\")]" + ); } } diff --git a/lib/filecheck/tests/basic.rs b/lib/filecheck/tests/basic.rs index 15e74788d7..debced2a1b 100644 --- a/lib/filecheck/tests/basic.rs +++ b/lib/filecheck/tests/basic.rs @@ -42,10 +42,12 @@ fn no_matches() { #[test] fn simple() { let c = CheckerBuilder::new() - .text(" + .text( + " check: one check: two - ") + ", + ) .unwrap() .finish(); @@ -71,10 +73,12 @@ fn simple() { #[test] fn sameln() { let c = CheckerBuilder::new() - .text(" + .text( + " check: one sameln: two - ") + ", + ) .unwrap() .finish(); @@ -106,10 +110,12 @@ fn sameln() { #[test] fn nextln() { let c = CheckerBuilder::new() - .text(" + .text( + " check: one nextln: two - ") + ", + ) .unwrap() .finish(); @@ -149,10 +155,12 @@ fn leading_nextln() { // A leading nextln directive should match from line 2. // This is somewhat arbitrary, but consistent with a preceeding 'check: $()' directive. let c = CheckerBuilder::new() - .text(" + .text( + " nextln: one nextln: two - ") + ", + ) .unwrap() .finish(); @@ -174,10 +182,12 @@ fn leading_nextln() { fn leading_sameln() { // A leading sameln directive should match from line 1. let c = CheckerBuilder::new() - .text(" + .text( + " sameln: one sameln: two - ") + ", + ) .unwrap() .finish(); @@ -197,11 +207,13 @@ fn leading_sameln() { #[test] fn not() { let c = CheckerBuilder::new() - .text(" + .text( + " check: one$() not: $()eat$() check: $()two - ") + ", + ) .unwrap() .finish(); @@ -221,12 +233,14 @@ fn not() { #[test] fn notnot() { let c = CheckerBuilder::new() - .text(" + .text( + " check: one$() not: $()eat$() not: half check: $()two - ") + ", + ) .unwrap() .finish(); @@ -254,87 +268,135 @@ fn notnot() { #[test] fn unordered() { let c = CheckerBuilder::new() - .text(" + .text( + " check: one unordered: two unordered: three check: four - ") + ", + ) .unwrap() .finish(); - assert_eq!(c.check("one two three four", NO_VARIABLES).map_err(e2s), - Ok(true)); - assert_eq!(c.check("one three two four", NO_VARIABLES).map_err(e2s), - Ok(true)); + assert_eq!( + c.check("one two three four", NO_VARIABLES).map_err(e2s), + Ok(true) + ); + assert_eq!( + c.check("one three two four", NO_VARIABLES).map_err(e2s), + Ok(true) + ); - assert_eq!(c.check("one two four three four", NO_VARIABLES) - .map_err(e2s), - Ok(true)); - assert_eq!(c.check("one three four two four", NO_VARIABLES) - .map_err(e2s), - Ok(true)); + assert_eq!( + c.check("one two four three four", NO_VARIABLES).map_err( + e2s, + ), + Ok(true) + ); + assert_eq!( + c.check("one three four two four", NO_VARIABLES).map_err( + e2s, + ), + Ok(true) + ); - assert_eq!(c.check("one two four three", NO_VARIABLES).map_err(e2s), - Ok(false)); - assert_eq!(c.check("one three four two", NO_VARIABLES).map_err(e2s), - Ok(false)); + assert_eq!( + c.check("one two four three", NO_VARIABLES).map_err(e2s), + Ok(false) + ); + assert_eq!( + c.check("one three four two", NO_VARIABLES).map_err(e2s), + Ok(false) + ); } #[test] fn leading_unordered() { let c = CheckerBuilder::new() - .text(" + .text( + " unordered: two unordered: three check: four - ") + ", + ) .unwrap() .finish(); - assert_eq!(c.check("one two three four", NO_VARIABLES).map_err(e2s), - Ok(true)); - assert_eq!(c.check("one three two four", NO_VARIABLES).map_err(e2s), - Ok(true)); + assert_eq!( + c.check("one two three four", NO_VARIABLES).map_err(e2s), + Ok(true) + ); + assert_eq!( + c.check("one three two four", NO_VARIABLES).map_err(e2s), + Ok(true) + ); - assert_eq!(c.check("one two four three four", NO_VARIABLES) - .map_err(e2s), - Ok(true)); - assert_eq!(c.check("one three four two four", NO_VARIABLES) - .map_err(e2s), - Ok(true)); + assert_eq!( + c.check("one two four three four", NO_VARIABLES).map_err( + e2s, + ), + Ok(true) + ); + assert_eq!( + c.check("one three four two four", NO_VARIABLES).map_err( + e2s, + ), + Ok(true) + ); - assert_eq!(c.check("one two four three", NO_VARIABLES).map_err(e2s), - Ok(false)); - assert_eq!(c.check("one three four two", NO_VARIABLES).map_err(e2s), - Ok(false)); + assert_eq!( + c.check("one two four three", NO_VARIABLES).map_err(e2s), + Ok(false) + ); + assert_eq!( + c.check("one three four two", NO_VARIABLES).map_err(e2s), + Ok(false) + ); } #[test] fn trailing_unordered() { let c = CheckerBuilder::new() - .text(" + .text( + " check: one unordered: two unordered: three - ") + ", + ) .unwrap() .finish(); - assert_eq!(c.check("one two three four", NO_VARIABLES).map_err(e2s), - Ok(true)); - assert_eq!(c.check("one three two four", NO_VARIABLES).map_err(e2s), - Ok(true)); + assert_eq!( + c.check("one two three four", NO_VARIABLES).map_err(e2s), + Ok(true) + ); + assert_eq!( + c.check("one three two four", NO_VARIABLES).map_err(e2s), + Ok(true) + ); - assert_eq!(c.check("one two four three four", NO_VARIABLES) - .map_err(e2s), - Ok(true)); - assert_eq!(c.check("one three four two four", NO_VARIABLES) - .map_err(e2s), - Ok(true)); + assert_eq!( + c.check("one two four three four", NO_VARIABLES).map_err( + e2s, + ), + Ok(true) + ); + assert_eq!( + c.check("one three four two four", NO_VARIABLES).map_err( + e2s, + ), + Ok(true) + ); - assert_eq!(c.check("one two four three", NO_VARIABLES).map_err(e2s), - Ok(true)); - assert_eq!(c.check("one three four two", NO_VARIABLES).map_err(e2s), - Ok(true)); + assert_eq!( + c.check("one two four three", NO_VARIABLES).map_err(e2s), + Ok(true) + ); + assert_eq!( + c.check("one three four two", NO_VARIABLES).map_err(e2s), + Ok(true) + ); } diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 5535feebc6..60b427b715 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -12,7 +12,8 @@ use std::hash::Hash; /// Permanent structure used for translating into Cretonne IL. pub struct ILBuilder - where Variable: EntityRef + Hash + Default +where + Variable: EntityRef + Hash + Default, { ssa: SSABuilder, ebbs: EntityMap, @@ -23,7 +24,8 @@ pub struct ILBuilder /// Temporary object used to build a Cretonne IL `Function`. pub struct FunctionBuilder<'a, Variable: 'a> - where Variable: EntityRef + Hash + Default +where + Variable: EntityRef + Hash + Default, { func: &'a mut Function, builder: &'a mut ILBuilder, @@ -44,7 +46,8 @@ struct Position { } impl ILBuilder - where Variable: EntityRef + Hash + Default +where + Variable: EntityRef + Hash + Default, { /// Creates a ILBuilder structure. The structure is automatically cleared each time it is /// passed to a [`FunctionBuilder`](struct.FunctionBuilder.html) for creation. @@ -68,7 +71,8 @@ impl ILBuilder /// Implementation of the [`InstBuilder`](../cretonne/ir/builder/trait.InstBuilder.html) that has /// one convenience method per Cretonne IL instruction. pub struct FuncInstBuilder<'short, 'long: 'short, Variable: 'long> - where Variable: EntityRef + Hash + Default +where + Variable: EntityRef + Hash + Default, { builder: &'short mut FunctionBuilder<'long, Variable>, ebb: Ebb, @@ -103,7 +107,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short self.builder .check_return_args(data.arguments(&self.builder.func.dfg.value_lists)) } - // We only insert the Ebb in the layout when an instruction is added to it +// We only insert the Ebb in the layout when an instruction is added to it if self.builder.builder.ebbs[self.builder.position.ebb].pristine { if !self.builder .func @@ -125,9 +129,9 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short if data.opcode().is_branch() { match data.branch_destination() { Some(dest_ebb) => { - // If the user has supplied jump arguments we must adapt the arguments of - // the destination ebb - // TODO: find a way not to allocate a vector +// If the user has supplied jump arguments we must adapt the arguments of +// the destination ebb +// TODO: find a way not to allocate a vector let args_types: Vec = match data.analyze_branch(&self.builder.func.dfg.value_lists) { BranchInfo::SingleDest(_, args) => { @@ -142,14 +146,14 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short self.builder.declare_successor(dest_ebb, inst); } None => { - // branch_destination() doesn't detect jump_tables +// branch_destination() doesn't detect jump_tables match data { - // If jump table we declare all entries successor - // TODO: not collect with vector? +// If jump table we declare all entries successor +// TODO: not collect with vector? InstructionData::BranchTable { table, .. } => { - // Unlike all other jumps/branches, jump tables are - // capable of having the same successor appear - // multiple times. Use a HashSet to deduplicate. +// Unlike all other jumps/branches, jump tables are +// capable of having the same successor appear +// multiple times. Use a HashSet to deduplicate. let mut unique = HashSet::new(); for dest_ebb in self.builder .func @@ -163,7 +167,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short self.builder.declare_successor(dest_ebb, inst) } } - // If not we do nothing +// If not we do nothing _ => {} } } @@ -211,13 +215,15 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short /// `Ebb` when you haven't filled the current one with a terminator instruction, inserting a /// return instruction with arguments that don't match the function's signature. impl<'a, Variable> FunctionBuilder<'a, Variable> - where Variable: EntityRef + Hash + Default +where + Variable: EntityRef + Hash + Default, { /// Creates a new FunctionBuilder structure that will operate on a `Function` using a /// `IlBuilder`. - pub fn new(func: &'a mut Function, - builder: &'a mut ILBuilder) - -> FunctionBuilder<'a, Variable> { + pub fn new( + func: &'a mut Function, + builder: &'a mut ILBuilder, + ) -> FunctionBuilder<'a, Variable> { builder.clear(); FunctionBuilder { func: func, @@ -255,12 +261,16 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> } if !self.builder.ebbs[self.position.ebb].pristine { // First we check that the previous block has been filled. - debug_assert!(self.is_unreachable() || self.builder.ebbs[self.position.ebb].filled, - "you have to fill your block before switching"); + debug_assert!( + self.is_unreachable() || self.builder.ebbs[self.position.ebb].filled, + "you have to fill your block before switching" + ); } // We cannot switch to a filled block - debug_assert!(!self.builder.ebbs[ebb].filled, - "you cannot switch to a block which is already filled"); + debug_assert!( + !self.builder.ebbs[ebb].filled, + "you cannot switch to a block which is already filled" + ); let basic_block = self.builder.ssa.header_block(ebb); // Then we change the cursor position. @@ -278,12 +288,12 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> /// created. Forgetting to call this method on every block will cause inconsistences in the /// produced functions. pub fn seal_block(&mut self, ebb: Ebb) { - let side_effects = self.builder - .ssa - .seal_ebb_header_block(ebb, - &mut self.func.dfg, - &mut self.func.layout, - &mut self.func.jump_tables); + let side_effects = self.builder.ssa.seal_ebb_header_block( + ebb, + &mut self.func.dfg, + &mut self.func.layout, + &mut self.func.jump_tables, + ); self.handle_ssa_side_effects(side_effects); } @@ -295,18 +305,17 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> /// Returns the Cretonne IL value corresponding to the utilization at the current program /// position of a previously defined user variable. pub fn use_var(&mut self, var: Variable) -> Value { - let ty = *self.builder - .types - .get(var) - .expect("this variable is used but its type has not been declared"); - let (val, side_effects) = self.builder - .ssa - .use_var(&mut self.func.dfg, - &mut self.func.layout, - &mut self.func.jump_tables, - var, - ty, - self.position.basic_block); + let ty = *self.builder.types.get(var).expect( + "this variable is used but its type has not been declared", + ); + let (val, side_effects) = self.builder.ssa.use_var( + &mut self.func.dfg, + &mut self.func.layout, + &mut self.func.jump_tables, + var, + ty, + self.position.basic_block, + ); self.handle_ssa_side_effects(side_effects); val } @@ -314,11 +323,15 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> /// Register a new definition of a user variable. Panics if the type of the value is not the /// same as the type registered for the variable. pub fn def_var(&mut self, var: Variable, val: Value) { - debug_assert!(self.func.dfg.value_type(val) == self.builder.types[var], - "the type of the value is not the type registered for the variable"); - self.builder - .ssa - .def_var(var, val, self.position.basic_block); + debug_assert!( + self.func.dfg.value_type(val) == self.builder.types[var], + "the type of the value is not the type registered for the variable" + ); + self.builder.ssa.def_var( + var, + val, + self.position.basic_block, + ); } /// Returns the value corresponding to the `i`-th argument of the function as defined by @@ -369,7 +382,8 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> /// function. The functions below help you inspect the function you're creating and modify it /// in ways that can be unsafe if used incorrectly. impl<'a, Variable> FunctionBuilder<'a, Variable> - where Variable: EntityRef + Hash + Default +where + Variable: EntityRef + Hash + Default, { /// Retrieves all the arguments for an `Ebb` currently infered from the jump instructions /// inserted that target it and the SSA construction. @@ -402,15 +416,16 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> /// **Note:** You are responsible for maintaining the coherence with the arguments of /// other jump instructions. pub fn change_jump_destination(&mut self, inst: Inst, new_dest: Ebb) { - let old_dest = - self.func.dfg[inst] - .branch_destination_mut() - .expect("you want to change the jump destination of a non-jump instruction"); + let old_dest = self.func.dfg[inst].branch_destination_mut().expect( + "you want to change the jump destination of a non-jump instruction", + ); let pred = self.builder.ssa.remove_ebb_predecessor(*old_dest, inst); *old_dest = new_dest; - self.builder - .ssa - .declare_ebb_predecessor(new_dest, pred, inst); + self.builder.ssa.declare_ebb_predecessor( + new_dest, + pred, + inst, + ); } /// Returns `true` if and only if the current `Ebb` is sealed and has no predecessors declared. @@ -422,7 +437,7 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> Some(entry) => self.position.ebb == entry, }; (!is_entry && self.builder.ssa.is_sealed(self.position.ebb) && - self.builder.ssa.predecessors(self.position.ebb).is_empty()) + self.builder.ssa.predecessors(self.position.ebb).is_empty()) } /// Returns `true` if and only if no instructions have been added since the last call to @@ -446,31 +461,31 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> } impl<'a, Variable> Drop for FunctionBuilder<'a, Variable> - where Variable: EntityRef + Hash + Default +where + Variable: EntityRef + Hash + Default, { /// When a `FunctionBuilder` goes out of scope, it means that the function is fully built. /// We then proceed to check if all the `Ebb`s are filled and sealed fn drop(&mut self) { - debug_assert!(self.builder - .ebbs - .keys() - .all(|ebb| { - self.builder.ebbs[ebb].pristine || - (self.builder.ssa.is_sealed(ebb) && - self.builder.ebbs[ebb].filled) - }), - "all blocks should be filled and sealed before dropping a FunctionBuilder") + debug_assert!( + self.builder.ebbs.keys().all(|ebb| { + self.builder.ebbs[ebb].pristine || + (self.builder.ssa.is_sealed(ebb) && self.builder.ebbs[ebb].filled) + }), + "all blocks should be filled and sealed before dropping a FunctionBuilder" + ) } } // Helper functions impl<'a, Variable> FunctionBuilder<'a, Variable> - where Variable: EntityRef + Hash + Default +where + Variable: EntityRef + Hash + Default, { fn move_to_next_basic_block(&mut self) { - self.position.basic_block = self.builder - .ssa - .declare_ebb_body_block(self.position.basic_block); + self.position.basic_block = self.builder.ssa.declare_ebb_body_block( + self.position.basic_block, + ); } fn fill_current_block(&mut self) { @@ -478,30 +493,36 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> } fn declare_successor(&mut self, dest_ebb: Ebb, jump_inst: Inst) { - self.builder - .ssa - .declare_ebb_predecessor(dest_ebb, self.position.basic_block, jump_inst); + self.builder.ssa.declare_ebb_predecessor( + dest_ebb, + self.position.basic_block, + jump_inst, + ); } fn check_return_args(&self, args: &[Value]) { - debug_assert_eq!(args.len(), - self.func.signature.return_types.len(), - "the number of returned values doesn't match the function signature "); + debug_assert_eq!( + args.len(), + self.func.signature.return_types.len(), + "the number of returned values doesn't match the function signature " + ); for (i, arg) in args.iter().enumerate() { let valty = self.func.dfg.value_type(*arg); - debug_assert_eq!(valty, - self.func.signature.return_types[i].value_type, - "the types of the values returned don't match the \ - function signature"); + debug_assert_eq!( + valty, + self.func.signature.return_types[i].value_type, + "the types of the values returned don't match the \ + function signature" + ); } } fn fill_function_args_values(&mut self, ebb: Ebb) { debug_assert!(self.pristine); for argtyp in &self.func.signature.argument_types { - self.builder - .function_args_values - .push(self.func.dfg.append_ebb_arg(ebb, argtyp.value_type)); + self.builder.function_args_values.push( + self.func.dfg.append_ebb_arg(ebb, argtyp.value_type), + ); } self.pristine = false; } @@ -510,48 +531,56 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> fn ebb_args_adjustement(&mut self, dest_ebb: Ebb, jump_args: &[Type]) { let ty_to_append: Option> = if self.builder.ssa.predecessors(dest_ebb).len() == 0 || - self.builder.ebbs[dest_ebb].pristine { + self.builder.ebbs[dest_ebb].pristine + { // This is the first jump instruction targeting this Ebb // so the jump arguments supplied here are this Ebb' arguments // However some of the arguments might already be there // in the Ebb so we have to check they're consistent let dest_ebb_args = self.func.dfg.ebb_args(dest_ebb); - debug_assert!(dest_ebb_args - .iter() - .zip(jump_args.iter().take(dest_ebb_args.len())) - .all(|(dest_arg, jump_arg)| { - *jump_arg == self.func.dfg.value_type(*dest_arg) - }), - "the jump argument supplied has not the \ - same type as the corresponding dest ebb argument"); + debug_assert!( + dest_ebb_args + .iter() + .zip(jump_args.iter().take(dest_ebb_args.len())) + .all(|(dest_arg, jump_arg)| { + *jump_arg == self.func.dfg.value_type(*dest_arg) + }), + "the jump argument supplied has not the \ + same type as the corresponding dest ebb argument" + ); self.builder.ebbs[dest_ebb].user_arg_count = jump_args.len(); - Some(jump_args - .iter() - .skip(dest_ebb_args.len()) - .cloned() - .collect()) + Some( + jump_args + .iter() + .skip(dest_ebb_args.len()) + .cloned() + .collect(), + ) } else { let dest_ebb_args = self.func.dfg.ebb_args(dest_ebb); // The Ebb already has predecessors // We check that the arguments supplied match those supplied // previously. - debug_assert!(jump_args.len() == self.builder.ebbs[dest_ebb].user_arg_count, - "the jump instruction doesn't have the same \ + debug_assert!( + jump_args.len() == self.builder.ebbs[dest_ebb].user_arg_count, + "the jump instruction doesn't have the same \ number of arguments as its destination Ebb \ ({} vs {}).", - jump_args.len(), - dest_ebb_args.len()); - debug_assert!(jump_args - .iter() - .zip(dest_ebb_args - .iter() - .take(self.builder.ebbs[dest_ebb].user_arg_count) - ) - .all(|(jump_arg, dest_arg)| { - *jump_arg == self.func.dfg.value_type(*dest_arg) - }), - "the jump argument supplied has not the \ - same type as the corresponding dest ebb argument"); + jump_args.len(), + dest_ebb_args.len() + ); + debug_assert!( + jump_args + .iter() + .zip(dest_ebb_args.iter().take( + self.builder.ebbs[dest_ebb].user_arg_count, + )) + .all(|(jump_arg, dest_arg)| { + *jump_arg == self.func.dfg.value_type(*dest_arg) + }), + "the jump argument supplied has not the \ + same type as the corresponding dest ebb argument" + ); None }; if let Some(ty_args) = ty_to_append { diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index f25b3eaa7b..c00d77e685 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -33,7 +33,8 @@ use std::collections::HashMap; /// and it is said _sealed_ if all of its predecessors have been declared. Only filled predecessors /// can be declared. pub struct SSABuilder - where Variable: EntityRef + Default +where + Variable: EntityRef + Default, { // Records for every variable and for every revelant block, the last definition of // the variable in the block. @@ -133,7 +134,8 @@ impl ReservedValue for Block { } impl SSABuilder - where Variable: EntityRef + Default +where + Variable: EntityRef + Default, { /// Allocate a new blank SSA builder struct. Use the API function to interact with the struct. pub fn new() -> SSABuilder { @@ -191,7 +193,8 @@ enum UseVarCases { /// Phi functions. /// impl SSABuilder - where Variable: EntityRef + Hash + Default +where + Variable: EntityRef + Hash + Default, { /// Declares a new definition of a variable in a given basic block. /// The SSA value is passed as an argument because it should be created with @@ -207,14 +210,15 @@ impl SSABuilder /// If the variable has never been defined in this blocks or recursively in its predecessors, /// this method will silently create an initializer with `iconst` or `fconst`. You are /// responsible for making sure that you initialize your variables. - pub fn use_var(&mut self, - dfg: &mut DataFlowGraph, - layout: &mut Layout, - jts: &mut JumpTables, - var: Variable, - ty: Type, - block: Block) - -> (Value, SideEffects) { + pub fn use_var( + &mut self, + dfg: &mut DataFlowGraph, + layout: &mut Layout, + jts: &mut JumpTables, + var: Variable, + ty: Type, + block: Block, + ) -> (Value, SideEffects) { // First we lookup for the current definition of the variable in this block if let Some(var_defs) = self.variables.get(var) { if let Some(val) = var_defs.get(&block) { @@ -281,13 +285,12 @@ impl SSABuilder /// here and the block is not sealed. /// Predecessors have to be added with `declare_ebb_predecessor`. pub fn declare_ebb_header_block(&mut self, ebb: Ebb) -> Block { - let block = self.blocks - .push(BlockData::EbbHeader(EbbHeaderBlockData { - predecessors: Vec::new(), - sealed: false, - ebb: ebb, - undef_variables: Vec::new(), - })); + let block = self.blocks.push(BlockData::EbbHeader(EbbHeaderBlockData { + predecessors: Vec::new(), + sealed: false, + ebb: ebb, + undef_variables: Vec::new(), + })); self.ebb_headers[ebb] = block.into(); block } @@ -331,12 +334,13 @@ impl SSABuilder /// /// This method modifies the function's `Layout` by adding arguments to the `Ebb`s to /// take into account the Phi function placed by the SSA algorithm. - pub fn seal_ebb_header_block(&mut self, - ebb: Ebb, - dfg: &mut DataFlowGraph, - layout: &mut Layout, - jts: &mut JumpTables) - -> SideEffects { + pub fn seal_ebb_header_block( + &mut self, + ebb: Ebb, + dfg: &mut DataFlowGraph, + layout: &mut Layout, + jts: &mut JumpTables, + ) -> SideEffects { let block = self.header_block(ebb); // Sanity check @@ -362,19 +366,24 @@ impl SSABuilder // jump argument to the branch instruction. // Panics if called with a non-header block. // Returns the list of newly created ebbs for critical edge splitting. - fn resolve_undef_vars(&mut self, - block: Block, - dfg: &mut DataFlowGraph, - layout: &mut Layout, - jts: &mut JumpTables) - -> SideEffects { + fn resolve_undef_vars( + &mut self, + block: Block, + dfg: &mut DataFlowGraph, + layout: &mut Layout, + jts: &mut JumpTables, + ) -> SideEffects { // TODO: find a way to not allocate vectors let (predecessors, undef_vars, ebb): (Vec<(Block, Inst)>, Vec<(Variable, Value)>, Ebb) = match self.blocks[block] { BlockData::EbbBody { .. } => panic!("this should not happen"), BlockData::EbbHeader(ref mut data) => { - (data.predecessors.clone(), data.undef_variables.clone(), data.ebb) + ( + data.predecessors.clone(), + data.undef_variables.clone(), + data.ebb, + ) } }; @@ -384,12 +393,13 @@ impl SSABuilder for (var, val) in undef_vars { let (_, mut local_side_effects) = self.predecessors_lookup(dfg, layout, jts, val, var, ebb, &predecessors); - side_effects - .split_ebbs_created - .append(&mut local_side_effects.split_ebbs_created); - side_effects - .instructions_added_to_ebbs - .append(&mut local_side_effects.instructions_added_to_ebbs); + side_effects.split_ebbs_created.append( + &mut local_side_effects + .split_ebbs_created, + ); + side_effects.instructions_added_to_ebbs.append( + &mut local_side_effects.instructions_added_to_ebbs, + ); } // Then we clear the undef_vars and mark the block as sealed. @@ -405,15 +415,16 @@ impl SSABuilder /// Look up in the predecessors of an Ebb the def for a value an decides wether or not /// to keep the eeb arg, and act accordingly. Returns the chosen value and optionnaly a /// list of Ebb that are the middle of newly created critical edges splits. - fn predecessors_lookup(&mut self, - dfg: &mut DataFlowGraph, - layout: &mut Layout, - jts: &mut JumpTables, - temp_arg_val: Value, - temp_arg_var: Variable, - dest_ebb: Ebb, - preds: &[(Block, Inst)]) - -> (Value, SideEffects) { + fn predecessors_lookup( + &mut self, + dfg: &mut DataFlowGraph, + layout: &mut Layout, + jts: &mut JumpTables, + temp_arg_val: Value, + temp_arg_var: Variable, + dest_ebb: Ebb, + preds: &[(Block, Inst)], + ) -> (Value, SideEffects) { let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero(); // TODO: find a way not not allocate a vector let mut jump_args_to_append: Vec<(Block, Inst, Value)> = Vec::new(); @@ -442,12 +453,13 @@ impl SSABuilder ZeroOneOrMore::More() => ZeroOneOrMore::More(), }; jump_args_to_append.push((pred, last_inst, pred_val)); - side_effects - .split_ebbs_created - .append(&mut local_side_effects.split_ebbs_created); - side_effects - .instructions_added_to_ebbs - .append(&mut local_side_effects.instructions_added_to_ebbs); + side_effects.split_ebbs_created.append( + &mut local_side_effects + .split_ebbs_created, + ); + side_effects.instructions_added_to_ebbs.append( + &mut local_side_effects.instructions_added_to_ebbs, + ); } match pred_values { ZeroOneOrMore::Zero() => { @@ -486,14 +498,16 @@ impl SSABuilder // There is disagreement in the predecessors on which value to use so we have // to keep the ebb argument. for (pred_block, last_inst, pred_val) in jump_args_to_append { - match self.append_jump_argument(dfg, - layout, - last_inst, - pred_block, - dest_ebb, - pred_val, - temp_arg_var, - jts) { + match self.append_jump_argument( + dfg, + layout, + last_inst, + pred_block, + dest_ebb, + pred_val, + temp_arg_var, + jts, + ) { None => (), Some(middle_ebb) => side_effects.split_ebbs_created.push(middle_ebb), }; @@ -505,16 +519,17 @@ impl SSABuilder /// Appends a jump argument to a jump instruction, returns ebb created in case of /// critical edge splitting. - fn append_jump_argument(&mut self, - dfg: &mut DataFlowGraph, - layout: &mut Layout, - jump_inst: Inst, - jump_inst_block: Block, - dest_ebb: Ebb, - val: Value, - var: Variable, - jts: &mut JumpTables) - -> Option { + fn append_jump_argument( + &mut self, + dfg: &mut DataFlowGraph, + layout: &mut Layout, + jump_inst: Inst, + jump_inst_block: Block, + dest_ebb: Ebb, + val: Value, + var: Variable, + jts: &mut JumpTables, + ) -> Option { match dfg[jump_inst].analyze_branch(&dfg.value_lists) { BranchInfo::NotABranch => { panic!("you have declared a non-branch instruction as a predecessor to an ebb"); @@ -529,14 +544,17 @@ impl SSABuilder // In the case of a jump table, the situation is tricky because br_table doesn't // support arguments. // We have to split the critical edge - let indexes: Vec = jts[jt] - .entries() - .fold(Vec::new(), |mut acc, (index, dest)| if dest == dest_ebb { + let indexes: Vec = jts[jt].entries().fold( + Vec::new(), + |mut acc, (index, dest)| if dest == + dest_ebb + { acc.push(index); acc } else { acc - }); + }, + ); let middle_ebb = dfg.make_ebb(); layout.append_ebb(middle_ebb); let block = self.declare_ebb_header_block(middle_ebb); @@ -632,79 +650,95 @@ mod tests { }; ssa.def_var(y_var, y_ssa, block); - assert_eq!(ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block) - .0, - x_ssa); - assert_eq!(ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block) - .0, - y_ssa); + assert_eq!( + ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block, + ).0, + x_ssa + ); + assert_eq!( + ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block, + ).0, + y_ssa + ); let z_var = Variable(2); - let x_use1 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block) - .0; - let y_use1 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block) - .0; + let x_use1 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block, + ).0; + let y_use1 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block, + ).0; let z1_ssa = { let cur = &mut Cursor::new(&mut func.layout); cur.goto_bottom(ebb0); func.dfg.ins(cur).iadd(x_use1, y_use1) }; ssa.def_var(z_var, z1_ssa, block); - assert_eq!(ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block) - .0, - z1_ssa); - let x_use2 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block) - .0; - let z_use1 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block) - .0; + assert_eq!( + ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block, + ).0, + z1_ssa + ); + let x_use2 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block, + ).0; + let z_use1 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block, + ).0; let z2_ssa = { let cur = &mut Cursor::new(&mut func.layout); cur.goto_bottom(ebb0); func.dfg.ins(cur).iadd(x_use2, z_use1) }; ssa.def_var(z_var, z2_ssa, block); - assert_eq!(ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block) - .0, - z2_ssa); + assert_eq!( + ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block, + ).0, + z2_ssa + ); } #[test] @@ -740,79 +774,93 @@ mod tests { func.dfg.ins(cur).iconst(I32, 2) }; ssa.def_var(y_var, y_ssa, block0); - assert_eq!(ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block0) - .0, - x_ssa); - assert_eq!(ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block0) - .0, - y_ssa); + assert_eq!( + ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block0, + ).0, + x_ssa + ); + assert_eq!( + ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block0, + ).0, + y_ssa + ); let z_var = Variable(2); - let x_use1 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block0) - .0; - let y_use1 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block0) - .0; + let x_use1 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block0, + ).0; + let y_use1 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block0, + ).0; let z1_ssa = { let cur = &mut Cursor::new(&mut func.layout); cur.goto_bottom(ebb0); func.dfg.ins(cur).iadd(x_use1, y_use1) }; ssa.def_var(z_var, z1_ssa, block0); - assert_eq!(ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block0) - .0, - z1_ssa); - let y_use2 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block0) - .0; + assert_eq!( + ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block0, + ).0, + z1_ssa + ); + let y_use2 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block0, + ).0; let jump_inst: Inst = { let cur = &mut Cursor::new(&mut func.layout); cur.goto_bottom(ebb0); func.dfg.ins(cur).brnz(y_use2, ebb1, &[]) }; let block1 = ssa.declare_ebb_body_block(block0); - let x_use2 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block1) - .0; + let x_use2 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block1, + ).0; assert_eq!(x_use2, x_ssa); - let z_use1 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block1) - .0; + let z_use1 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block1, + ).0; assert_eq!(z_use1, z1_ssa); let z2_ssa = { let cur = &mut Cursor::new(&mut func.layout); @@ -820,33 +868,38 @@ mod tests { func.dfg.ins(cur).iadd(x_use2, z_use1) }; ssa.def_var(z_var, z2_ssa, block1); - assert_eq!(ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block1) - .0, - z2_ssa); + assert_eq!( + ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block1, + ).0, + z2_ssa + ); ssa.seal_ebb_header_block(ebb0, &mut func.dfg, &mut func.layout, &mut func.jump_tables); let block2 = ssa.declare_ebb_header_block(ebb1); ssa.declare_ebb_predecessor(ebb1, block0, jump_inst); ssa.seal_ebb_header_block(ebb1, &mut func.dfg, &mut func.layout, &mut func.jump_tables); - let x_use3 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block2) - .0; + let x_use3 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block2, + ).0; assert_eq!(x_ssa, x_use3); - let y_use3 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block2) - .0; + let y_use3 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block2, + ).0; assert_eq!(y_ssa, y_use3); let y2_ssa = { let cur = &mut Cursor::new(&mut func.layout); @@ -897,14 +950,17 @@ mod tests { func.dfg.ins(cur).iconst(I32, 1) }; ssa.def_var(x_var, x1, block0); - assert_eq!(ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block0) - .0, - x1); + assert_eq!( + ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block0, + ).0, + x1 + ); let y_var = Variable(1); let y1 = { let cur = &mut Cursor::new(&mut func.layout); @@ -912,30 +968,35 @@ mod tests { func.dfg.ins(cur).iconst(I32, 2) }; ssa.def_var(y_var, y1, block0); - assert_eq!(ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block0) - .0, - y1); + assert_eq!( + ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block0, + ).0, + y1 + ); let z_var = Variable(2); - let x2 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block0) - .0; + let x2 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block0, + ).0; assert_eq!(x2, x1); - let y2 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block0) - .0; + let y2 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block0, + ).0; assert_eq!(y2, y1); let z1 = { let cur = &mut Cursor::new(&mut func.layout); @@ -950,33 +1011,36 @@ mod tests { }; let block1 = ssa.declare_ebb_header_block(ebb1); ssa.declare_ebb_predecessor(ebb1, block0, jump_ebb0_ebb1); - let z2 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block1) - .0; - let y3 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block1) - .0; + let z2 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block1, + ).0; + let y3 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block1, + ).0; let z3 = { let cur = &mut Cursor::new(&mut func.layout); cur.goto_bottom(ebb1); func.dfg.ins(cur).iadd(z2, y3) }; ssa.def_var(z_var, z3, block1); - let y4 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block1) - .0; + let y4 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block1, + ).0; assert_eq!(y4, y3); let jump_ebb1_ebb2 = { let cur = &mut Cursor::new(&mut func.layout); @@ -984,34 +1048,37 @@ mod tests { func.dfg.ins(cur).brnz(y4, ebb2, &[]) }; let block2 = ssa.declare_ebb_body_block(block1); - let z4 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block2) - .0; + let z4 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block2, + ).0; assert_eq!(z4, z3); - let x3 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block2) - .0; + let x3 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block2, + ).0; let z5 = { let cur = &mut Cursor::new(&mut func.layout); cur.goto_bottom(ebb1); func.dfg.ins(cur).isub(z4, x3) }; ssa.def_var(z_var, z5, block2); - let y5 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block2) - .0; + let y5 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block2, + ).0; assert_eq!(y5, y3); { let cur = &mut Cursor::new(&mut func.layout); @@ -1022,21 +1089,23 @@ mod tests { let block3 = ssa.declare_ebb_header_block(ebb2); ssa.declare_ebb_predecessor(ebb2, block1, jump_ebb1_ebb2); ssa.seal_ebb_header_block(ebb2, &mut func.dfg, &mut func.layout, &mut func.jump_tables); - let y6 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block3) - .0; + let y6 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block3, + ).0; assert_eq!(y6, y3); - let x4 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block3) - .0; + let x4 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block3, + ).0; assert_eq!(x4, x3); let y7 = { let cur = &mut Cursor::new(&mut func.layout); @@ -1089,13 +1158,14 @@ mod tests { let mut jt_data = JumpTableData::new(); jt_data.set_entry(0, ebb1); let jt = func.jump_tables.push(jt_data); - ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block0) - .0; + ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block0, + ).0; let br_table = { let cur = &mut Cursor::new(&mut func.layout); cur.goto_bottom(ebb0); @@ -1117,13 +1187,14 @@ mod tests { ssa.declare_ebb_predecessor(ebb1, block1, jump_inst); ssa.declare_ebb_predecessor(ebb1, block0, br_table); ssa.seal_ebb_header_block(ebb1, &mut func.dfg, &mut func.layout, &mut func.jump_tables); - let x4 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block2) - .0; + let x4 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block2, + ).0; { let cur = &mut Cursor::new(&mut func.layout); cur.goto_bottom(ebb1); @@ -1189,21 +1260,23 @@ mod tests { }; let block1 = ssa.declare_ebb_header_block(ebb1); ssa.declare_ebb_predecessor(ebb1, block0, jump_inst); - let z2 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block1) - .0; + let z2 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + z_var, + I32, + block1, + ).0; assert_eq!(func.dfg.ebb_args(ebb1)[0], z2); - let x2 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block1) - .0; + let x2 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block1, + ).0; assert_eq!(func.dfg.ebb_args(ebb1)[1], x2); let x3 = { let cur = &mut Cursor::new(&mut func.layout); @@ -1211,20 +1284,22 @@ mod tests { func.dfg.ins(cur).iadd(x2, z2) }; ssa.def_var(x_var, x3, block1); - let x4 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block1) - .0; - let y3 = ssa.use_var(&mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block1) - .0; + let x4 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + x_var, + I32, + block1, + ).0; + let y3 = ssa.use_var( + &mut func.dfg, + &mut func.layout, + &mut func.jump_tables, + y_var, + I32, + block1, + ).0; assert_eq!(func.dfg.ebb_args(ebb1)[2], y3); let y4 = { let cur = &mut Cursor::new(&mut func.layout); diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs index 51748e284a..2b1e8fae6d 100644 --- a/lib/reader/src/isaspec.rs +++ b/lib/reader/src/isaspec.rs @@ -36,7 +36,8 @@ impl IsaSpec { /// Parse an iterator of command line options and apply them to `config`. pub fn parse_options<'a, I>(iter: I, config: &mut Configurable, loc: &Location) -> Result<()> - where I: Iterator +where + I: Iterator, { for opt in iter.map(TestOption::new) { match opt { diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 5be93676d1..1f78b27d92 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -57,9 +57,9 @@ pub struct LocatedToken<'a> { /// Wrap up a `Token` with the given location. fn token<'a>(token: Token<'a>, loc: Location) -> Result, LocatedError> { Ok(LocatedToken { - token, - location: loc, - }) + token, + location: loc, + }) } /// An error from the lexical analysis. @@ -78,9 +78,9 @@ pub struct LocatedError { /// Wrap up an `Error` with the given location. fn error<'a>(error: Error, loc: Location) -> Result, LocatedError> { Err(LocatedError { - error, - location: loc, - }) + error, + location: loc, + }) } /// Get the number of decimal digits at the end of `s`. @@ -180,10 +180,11 @@ impl<'a> Lexer<'a> { } // Scan a multi-char token. - fn scan_chars(&mut self, - count: usize, - tok: Token<'a>) - -> Result, LocatedError> { + fn scan_chars( + &mut self, + count: usize, + tok: Token<'a>, + ) -> Result, LocatedError> { let loc = self.loc(); for _ in 0..count { assert_ne!(self.lookahead, None); @@ -294,13 +295,16 @@ impl<'a> Lexer<'a> { let text = &self.source[begin..self.pos]; // Look for numbered well-known entities like ebb15, v45, ... - token(split_entity_name(text) - .and_then(|(prefix, number)| { - Self::numbered_entity(prefix, number) - .or_else(|| Self::value_type(text, prefix, number)) - }) - .unwrap_or(Token::Identifier(text)), - loc) + token( + split_entity_name(text) + .and_then(|(prefix, number)| { + Self::numbered_entity(prefix, number).or_else(|| { + Self::value_type(text, prefix, number) + }) + }) + .unwrap_or(Token::Identifier(text)), + loc, + ) } // If prefix is a well-known entity prefix and suffix is a valid entity number, return the @@ -391,40 +395,40 @@ impl<'a> Lexer<'a> { loop { let loc = self.loc(); return match self.lookahead { - None => None, - Some(';') => Some(self.scan_comment()), - Some('(') => Some(self.scan_char(Token::LPar)), - Some(')') => Some(self.scan_char(Token::RPar)), - Some('{') => Some(self.scan_char(Token::LBrace)), - Some('}') => Some(self.scan_char(Token::RBrace)), - Some('[') => Some(self.scan_char(Token::LBracket)), - Some(']') => Some(self.scan_char(Token::RBracket)), - Some(',') => Some(self.scan_char(Token::Comma)), - Some('.') => Some(self.scan_char(Token::Dot)), - Some(':') => Some(self.scan_char(Token::Colon)), - Some('=') => Some(self.scan_char(Token::Equal)), - Some('+') => Some(self.scan_number()), - Some('-') => { - if self.looking_at("->") { - Some(self.scan_chars(2, Token::Arrow)) - } else { - Some(self.scan_number()) - } - } - Some(ch) if ch.is_digit(10) => Some(self.scan_number()), - Some(ch) if ch.is_alphabetic() => Some(self.scan_word()), - Some('%') => Some(self.scan_name()), - Some('#') => Some(self.scan_hex_sequence()), - Some(ch) if ch.is_whitespace() => { - self.next_ch(); - continue; - } - _ => { - // Skip invalid char, return error. - self.next_ch(); - Some(error(Error::InvalidChar, loc)) - } - }; + None => None, + Some(';') => Some(self.scan_comment()), + Some('(') => Some(self.scan_char(Token::LPar)), + Some(')') => Some(self.scan_char(Token::RPar)), + Some('{') => Some(self.scan_char(Token::LBrace)), + Some('}') => Some(self.scan_char(Token::RBrace)), + Some('[') => Some(self.scan_char(Token::LBracket)), + Some(']') => Some(self.scan_char(Token::RBracket)), + Some(',') => Some(self.scan_char(Token::Comma)), + Some('.') => Some(self.scan_char(Token::Dot)), + Some(':') => Some(self.scan_char(Token::Colon)), + Some('=') => Some(self.scan_char(Token::Equal)), + Some('+') => Some(self.scan_number()), + Some('-') => { + if self.looking_at("->") { + Some(self.scan_chars(2, Token::Arrow)) + } else { + Some(self.scan_number()) + } + } + Some(ch) if ch.is_digit(10) => Some(self.scan_number()), + Some(ch) if ch.is_alphabetic() => Some(self.scan_word()), + Some('%') => Some(self.scan_name()), + Some('#') => Some(self.scan_hex_sequence()), + Some(ch) if ch.is_whitespace() => { + self.next_ch(); + continue; + } + _ => { + // Skip invalid char, return error. + self.next_ch(); + Some(error(Error::InvalidChar, loc)) + } + }; } } } @@ -530,14 +534,20 @@ mod tests { #[test] fn lex_identifiers() { - let mut lex = Lexer::new("v0 v00 vx01 ebb1234567890 ebb5234567890 v1x vx1 vxvx4 \ - function0 function b1 i32x4 f32x5"); - assert_eq!(lex.next(), - token(Token::Value(Value::with_number(0).unwrap()), 1)); + let mut lex = Lexer::new( + "v0 v00 vx01 ebb1234567890 ebb5234567890 v1x vx1 vxvx4 \ + function0 function b1 i32x4 f32x5", + ); + assert_eq!( + lex.next(), + token(Token::Value(Value::with_number(0).unwrap()), 1) + ); assert_eq!(lex.next(), token(Token::Identifier("v00"), 1)); assert_eq!(lex.next(), token(Token::Identifier("vx01"), 1)); - assert_eq!(lex.next(), - token(Token::Ebb(Ebb::with_number(1234567890).unwrap()), 1)); + assert_eq!( + lex.next(), + token(Token::Ebb(Ebb::with_number(1234567890).unwrap()), 1) + ); assert_eq!(lex.next(), token(Token::Identifier("ebb5234567890"), 1)); assert_eq!(lex.next(), token(Token::Identifier("v1x"), 1)); assert_eq!(lex.next(), token(Token::Identifier("vx1"), 1)); diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index b5e2e12d6b..39bac8af78 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -30,7 +30,9 @@ use sourcemap::{SourceMap, MutableSourceMap}; /// /// Any test commands or ISA declarations are ignored. pub fn parse_functions(text: &str) -> Result> { - parse_test(text).map(|file| file.functions.into_iter().map(|(func, _)| func).collect()) + parse_test(text).map(|file| { + file.functions.into_iter().map(|(func, _)| func).collect() + }) } /// Parse the entire `text` as a test case file. @@ -47,11 +49,11 @@ pub fn parse_test<'a>(text: &'a str) -> Result> { let functions = parser.parse_function_list(isa_spec.unique_isa())?; Ok(TestFile { - commands, - isa_spec, - preamble_comments, - functions, - }) + commands, + isa_spec, + preamble_comments, + functions, + }) } pub struct Parser<'a> { @@ -116,8 +118,11 @@ impl<'a> Context<'a> { // Allocate a new stack slot and add a mapping number -> StackSlot. fn add_ss(&mut self, number: u32, data: StackSlotData, loc: &Location) -> Result<()> { - self.map - .def_ss(number, self.function.stack_slots.push(data), loc) + self.map.def_ss( + number, + self.function.stack_slots.push(data), + loc, + ) } // Resolve a reference to a stack slot. @@ -130,8 +135,11 @@ impl<'a> Context<'a> { // Allocate a global variable slot and add a mapping number -> GlobalVar. fn add_gv(&mut self, number: u32, data: GlobalVarData, loc: &Location) -> Result<()> { - self.map - .def_gv(number, self.function.global_vars.push(data), loc) + self.map.def_gv( + number, + self.function.global_vars.push(data), + loc, + ) } // Resolve a reference to a global variable. @@ -144,8 +152,11 @@ impl<'a> Context<'a> { // Allocate a heap slot and add a mapping number -> Heap. fn add_heap(&mut self, number: u32, data: HeapData, loc: &Location) -> Result<()> { - self.map - .def_heap(number, self.function.heaps.push(data), loc) + self.map.def_heap( + number, + self.function.heaps.push(data), + loc, + ) } // Resolve a reference to a heap. @@ -158,8 +169,11 @@ impl<'a> Context<'a> { // Allocate a new signature and add a mapping number -> SigRef. fn add_sig(&mut self, number: u32, data: Signature, loc: &Location) -> Result<()> { - self.map - .def_sig(number, self.function.dfg.signatures.push(data), loc) + self.map.def_sig( + number, + self.function.dfg.signatures.push(data), + loc, + ) } // Resolve a reference to a signature. @@ -172,8 +186,11 @@ impl<'a> Context<'a> { // Allocate a new external function and add a mapping number -> FuncRef. fn add_fn(&mut self, number: u32, data: ExtFuncData, loc: &Location) -> Result<()> { - self.map - .def_fn(number, self.function.dfg.ext_funcs.push(data), loc) + self.map.def_fn( + number, + self.function.dfg.ext_funcs.push(data), + loc, + ) } // Resolve a reference to a function. @@ -186,8 +203,11 @@ impl<'a> Context<'a> { // Allocate a new jump table and add a mapping number -> JumpTable. fn add_jt(&mut self, number: u32, data: JumpTableData, loc: &Location) -> Result<()> { - self.map - .def_jt(number, self.function.jump_tables.push(data), loc) + self.map.def_jt( + number, + self.function.jump_tables.push(data), + loc, + ) } // Resolve a reference to a jump table. @@ -220,16 +240,18 @@ impl<'a> Context<'a> { let ir_to = match self.map.get_value(source_to) { Some(v) => v, None => { - return err!(source_loc, - "IR destination value alias not found for {}", - source_to); + return err!( + source_loc, + "IR destination value alias not found for {}", + source_to + ); } }; - let dest_loc = self.map - .location(AnyEntity::from(ir_to)) - .expect(&*format!("Error in looking up location of IR destination value alias \ + let dest_loc = self.map.location(AnyEntity::from(ir_to)).expect(&*format!( + "Error in looking up location of IR destination value alias \ for {}", - ir_to)); + ir_to + )); let ir_from = self.function.dfg.make_value_alias(ir_to); self.map.def_value(source_from, ir_from, &dest_loc)?; } @@ -237,8 +259,10 @@ impl<'a> Context<'a> { for ebb in self.function.layout.ebbs() { for inst in self.function.layout.ebb_insts(ebb) { let loc = inst.into(); - self.map - .rewrite_values(self.function.dfg.inst_args_mut(inst), loc)?; + self.map.rewrite_values( + self.function.dfg.inst_args_mut(inst), + loc, + )?; if let Some(dest) = self.function.dfg[inst].branch_destination_mut() { self.map.rewrite_ebb(dest, loc)?; } @@ -522,8 +546,9 @@ impl<'a> Parser<'a> { self.consume(); // Lexer just gives us raw text that looks like an integer. // Parse it as a u8 to check for overflow and other issues. - text.parse() - .map_err(|_| self.error("expected u8 decimal immediate")) + text.parse().map_err( + |_| self.error("expected u8 decimal immediate"), + ) } else { err!(self.loc, err_msg) } @@ -536,8 +561,9 @@ impl<'a> Parser<'a> { self.consume(); // Lexer just gives us raw text that looks like an integer. // Parse it as a i32 to check for overflow and other issues. - text.parse() - .map_err(|_| self.error("expected i32 decimal immediate")) + text.parse().map_err( + |_| self.error("expected i32 decimal immediate"), + ) } else { err!(self.loc, err_msg) } @@ -654,8 +680,9 @@ impl<'a> Parser<'a> { // The only error we anticipate from this parse is overflow, the lexer should // already have ensured that the string doesn't contain invalid characters, and // isn't empty or negative. - u16::from_str_radix(bits_str, 16) - .map_err(|_| self.error("the hex sequence given overflows the u16 type")) + u16::from_str_radix(bits_str, 16).map_err(|_| { + self.error("the hex sequence given overflows the u16 type") + }) } else { err!(self.loc, err_msg) } @@ -667,13 +694,14 @@ impl<'a> Parser<'a> { self.consume(); match isa { Some(isa) => { - isa.register_info() - .parse_regunit(name) - .ok_or_else(|| self.error("invalid register name")) + isa.register_info().parse_regunit(name).ok_or_else(|| { + self.error("invalid register name") + }) } None => { - name.parse() - .map_err(|_| self.error("invalid register number")) + name.parse().map_err( + |_| self.error("invalid register number"), + ) } } } else { @@ -709,17 +737,19 @@ impl<'a> Parser<'a> { // Change the default for `enable_verifier` to `true`. It defaults to `false` because it // would slow down normal compilation, but when we're reading IL from a text file we're // either testing or debugging Cretonne, and verification makes sense. - flag_builder - .enable("enable_verifier") - .expect("Missing enable_verifier setting"); + flag_builder.enable("enable_verifier").expect( + "Missing enable_verifier setting", + ); while let Some(Token::Identifier(command)) = self.token() { match command { "set" => { last_set_loc = Some(self.loc); - isaspec::parse_options(self.consume_line().trim().split_whitespace(), - &mut flag_builder, - &self.loc)?; + isaspec::parse_options( + self.consume_line().trim().split_whitespace(), + &mut flag_builder, + &self.loc, + )?; } "isa" => { let loc = self.loc; @@ -755,8 +785,10 @@ impl<'a> Parser<'a> { // No `isa` commands, but we allow for `set` commands. Ok(isaspec::IsaSpec::None(settings::Flags::new(&flag_builder))) } else if let Some(loc) = last_set_loc { - err!(loc, - "dangling 'set' command after ISA specification has no effect.") + err!( + loc, + "dangling 'set' command after ISA specification has no effect." + ) } else { Ok(isaspec::IsaSpec::Some(isas)) } @@ -765,9 +797,10 @@ impl<'a> Parser<'a> { /// Parse a list of function definitions. /// /// This is the top-level parse function matching the whole contents of a file. - pub fn parse_function_list(&mut self, - unique_isa: Option<&TargetIsa>) - -> Result)>> { + pub fn parse_function_list( + &mut self, + unique_isa: Option<&TargetIsa>, + ) -> Result)>> { let mut list = Vec::new(); while self.token().is_some() { list.push(self.parse_function(unique_isa)?); @@ -779,9 +812,10 @@ impl<'a> Parser<'a> { // // function ::= * function-spec "{" preamble function-body "}" // - fn parse_function(&mut self, - unique_isa: Option<&TargetIsa>) - -> Result<(Function, Details<'a>)> { + fn parse_function( + &mut self, + unique_isa: Option<&TargetIsa>, + ) -> Result<(Function, Details<'a>)> { // Begin gathering comments. // Make sure we don't include any comments before the `function` keyword. self.token(); @@ -792,13 +826,19 @@ impl<'a> Parser<'a> { let mut ctx = Context::new(Function::with_name_signature(name, sig), unique_isa); // function ::= function-spec * "{" preamble function-body "}" - self.match_token(Token::LBrace, "expected '{' before function body")?; + self.match_token( + Token::LBrace, + "expected '{' before function body", + )?; // function ::= function-spec "{" * preamble function-body "}" self.parse_preamble(&mut ctx)?; // function ::= function-spec "{" preamble * function-body "}" self.parse_function_body(&mut ctx)?; // function ::= function-spec "{" preamble function-body * "}" - self.match_token(Token::RBrace, "expected '}' after function body")?; + self.match_token( + Token::RBrace, + "expected '}' after function body", + )?; // Collect any comments following the end of the function, then stop gathering comments. self.gather_comments(AnyEntity::Function); @@ -822,9 +862,10 @@ impl<'a> Parser<'a> { // // function-spec ::= * "function" name signature // - fn parse_function_spec(&mut self, - unique_isa: Option<&TargetIsa>) - -> Result<(Location, FunctionName, Signature)> { + fn parse_function_spec( + &mut self, + unique_isa: Option<&TargetIsa>, + ) -> Result<(Location, FunctionName, Signature)> { self.match_identifier("function", "expected 'function'")?; let location = self.loc; @@ -849,8 +890,10 @@ impl<'a> Parser<'a> { } Some(Token::HexSequence(s)) => { if s.len() % 2 != 0 { - return err!(self.loc, - "expected binary function name to have length multiple of two"); + return err!( + self.loc, + "expected binary function name to have length multiple of two" + ); } let mut bin_name = Vec::with_capacity(s.len() / 2); let mut i = 0; @@ -874,12 +917,18 @@ impl<'a> Parser<'a> { // Calling convention defaults to `native`, but can be changed. let mut sig = Signature::new(CallConv::Native); - self.match_token(Token::LPar, "expected function signature: ( args... )")?; + self.match_token( + Token::LPar, + "expected function signature: ( args... )", + )?; // signature ::= "(" * [arglist] ")" ["->" retlist] [callconv] if self.token() != Some(Token::RPar) { sig.argument_types = self.parse_argument_list(unique_isa)?; } - self.match_token(Token::RPar, "expected ')' after function arguments")?; + self.match_token( + Token::RPar, + "expected ')' after function arguments", + )?; if self.optional(Token::Arrow) { sig.return_types = self.parse_argument_list(unique_isa)?; } @@ -977,8 +1026,10 @@ impl<'a> Parser<'a> { _ => err!(self.loc, "expected argument location"), }; - self.match_token(Token::RBracket, - "expected ']' to end argument location annotation")?; + self.match_token( + Token::RBracket, + "expected ']' to end argument location annotation", + )?; result } else { @@ -1001,33 +1052,41 @@ impl<'a> Parser<'a> { Some(Token::StackSlot(..)) => { self.gather_comments(ctx.function.stack_slots.next_key()); let loc = self.loc; - self.parse_stack_slot_decl() - .and_then(|(num, dat)| ctx.add_ss(num, dat, &loc)) + self.parse_stack_slot_decl().and_then(|(num, dat)| { + ctx.add_ss(num, dat, &loc) + }) } Some(Token::GlobalVar(..)) => { self.gather_comments(ctx.function.global_vars.next_key()); - self.parse_global_var_decl() - .and_then(|(num, dat)| ctx.add_gv(num, dat, &self.loc)) + self.parse_global_var_decl().and_then(|(num, dat)| { + ctx.add_gv(num, dat, &self.loc) + }) } Some(Token::Heap(..)) => { self.gather_comments(ctx.function.heaps.next_key()); - self.parse_heap_decl() - .and_then(|(num, dat)| ctx.add_heap(num, dat, &self.loc)) + self.parse_heap_decl().and_then(|(num, dat)| { + ctx.add_heap(num, dat, &self.loc) + }) } Some(Token::SigRef(..)) => { self.gather_comments(ctx.function.dfg.signatures.next_key()); - self.parse_signature_decl(ctx.unique_isa) - .and_then(|(num, dat)| ctx.add_sig(num, dat, &self.loc)) + self.parse_signature_decl(ctx.unique_isa).and_then( + |(num, dat)| { + ctx.add_sig(num, dat, &self.loc) + }, + ) } Some(Token::FuncRef(..)) => { self.gather_comments(ctx.function.dfg.ext_funcs.next_key()); - self.parse_function_decl(ctx) - .and_then(|(num, dat)| ctx.add_fn(num, dat, &self.loc)) + self.parse_function_decl(ctx).and_then(|(num, dat)| { + ctx.add_fn(num, dat, &self.loc) + }) } Some(Token::JumpTable(..)) => { self.gather_comments(ctx.function.jump_tables.next_key()); - self.parse_jump_table_decl() - .and_then(|(num, dat)| ctx.add_jt(num, dat, &self.loc)) + self.parse_jump_table_decl().and_then(|(num, dat)| { + ctx.add_jt(num, dat, &self.loc) + }) } // More to come.. _ => return Ok(()), @@ -1044,7 +1103,10 @@ impl<'a> Parser<'a> { // | "outgoing_arg" fn parse_stack_slot_decl(&mut self) -> Result<(u32, StackSlotData)> { let number = self.match_ss("expected stack slot number: ss«n»")?; - self.match_token(Token::Equal, "expected '=' in stack slot declaration")?; + self.match_token( + Token::Equal, + "expected '=' in stack slot declaration", + )?; let kind = self.match_enum("expected stack slot kind")?; // stack-slot-decl ::= StackSlot(ss) "=" stack-slot-kind * Bytes {"," stack-slot-flag} @@ -1078,7 +1140,10 @@ impl<'a> Parser<'a> { // fn parse_global_var_decl(&mut self) -> Result<(u32, GlobalVarData)> { let number = self.match_gv("expected global variable number: gv«n»")?; - self.match_token(Token::Equal, "expected '=' in global variable declaration")?; + self.match_token( + Token::Equal, + "expected '=' in global variable declaration", + )?; let data = match self.match_any_identifier("expected global variable kind")? { "vmctx" => { @@ -1086,9 +1151,15 @@ impl<'a> Parser<'a> { GlobalVarData::VmCtx { offset } } "deref" => { - self.match_token(Token::LPar, "expected '(' in 'deref' global variable decl")?; + self.match_token( + Token::LPar, + "expected '(' in 'deref' global variable decl", + )?; let base = self.match_gv_preamble("expected global variable: gv«n»")?; - self.match_token(Token::RPar, "expected ')' in 'deref' global variable decl")?; + self.match_token( + Token::RPar, + "expected ')' in 'deref' global variable decl", + )?; let offset = self.optional_offset32()?; GlobalVarData::Deref { base, offset } } @@ -1111,7 +1182,10 @@ impl<'a> Parser<'a> { // fn parse_heap_decl(&mut self) -> Result<(u32, HeapData)> { let number = self.match_heap("expected heap number: heap«n»")?; - self.match_token(Token::Equal, "expected '=' in heap declaration")?; + self.match_token( + Token::Equal, + "expected '=' in heap declaration", + )?; let style_name = self.match_any_identifier("expected 'static' or 'dynamic'")?; @@ -1173,7 +1247,10 @@ impl<'a> Parser<'a> { // fn parse_signature_decl(&mut self, unique_isa: Option<&TargetIsa>) -> Result<(u32, Signature)> { let number = self.match_sig("expected signature number: sig«n»")?; - self.match_token(Token::Equal, "expected '=' in signature decl")?; + self.match_token( + Token::Equal, + "expected '=' in signature decl", + )?; let data = self.parse_signature(unique_isa)?; Ok((number, data)) } @@ -1190,15 +1267,18 @@ impl<'a> Parser<'a> { // fn parse_function_decl(&mut self, ctx: &mut Context) -> Result<(u32, ExtFuncData)> { let number = self.match_fn("expected function number: fn«n»")?; - self.match_token(Token::Equal, "expected '=' in function decl")?; + self.match_token( + Token::Equal, + "expected '=' in function decl", + )?; let data = match self.token() { Some(Token::Identifier("function")) => { let (loc, name, sig) = self.parse_function_spec(ctx.unique_isa)?; let sigref = ctx.function.dfg.signatures.push(sig); - ctx.map - .def_entity(sigref.into(), &loc) - .expect("duplicate SigRef entities created"); + ctx.map.def_entity(sigref.into(), &loc).expect( + "duplicate SigRef entities created", + ); ExtFuncData { name, signature: sigref, @@ -1223,7 +1303,10 @@ impl<'a> Parser<'a> { // jump-table-decl ::= * JumpTable(jt) "=" "jump_table" jt-entry {"," jt-entry} fn parse_jump_table_decl(&mut self) -> Result<(u32, JumpTableData)> { let number = self.match_jt()?; - self.match_token(Token::Equal, "expected '=' in jump_table decl")?; + self.match_token( + Token::Equal, + "expected '=' in jump_table decl", + )?; self.match_identifier("jump_table", "expected 'jump_table'")?; let mut data = JumpTableData::new(); @@ -1284,16 +1367,20 @@ impl<'a> Parser<'a> { if !self.optional(Token::Colon) { // ebb-header ::= Ebb(ebb) [ * ebb-args ] ":" self.parse_ebb_args(ctx, ebb)?; - self.match_token(Token::Colon, "expected ':' after EBB arguments")?; + self.match_token( + Token::Colon, + "expected ':' after EBB arguments", + )?; } // extended-basic-block ::= ebb-header * { instruction } while match self.token() { - Some(Token::Value(_)) => true, - Some(Token::Identifier(_)) => true, - Some(Token::LBracket) => true, - _ => false, - } { + Some(Token::Value(_)) => true, + Some(Token::Identifier(_)) => true, + Some(Token::LBracket) => true, + _ => false, + } + { let (encoding, result_locations) = self.parse_instruction_encoding(ctx)?; // We need to parse instruction results here because they are shared @@ -1309,10 +1396,24 @@ impl<'a> Parser<'a> { } Some(Token::Equal) => { self.consume(); - self.parse_instruction(results, encoding, result_locations, ctx, ebb)?; + self.parse_instruction( + results, + encoding, + result_locations, + ctx, + ebb, + )?; } _ if !results.is_empty() => return err!(self.loc, "expected -> or ="), - _ => self.parse_instruction(results, encoding, result_locations, ctx, ebb)?, + _ => { + self.parse_instruction( + results, + encoding, + result_locations, + ctx, + ebb, + )? + } } } @@ -1325,7 +1426,10 @@ impl<'a> Parser<'a> { // ebb-args ::= * "(" ebb-arg { "," ebb-arg } ")" fn parse_ebb_args(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { // ebb-args ::= * "(" ebb-arg { "," ebb-arg } ")" - self.match_token(Token::LPar, "expected '(' before EBB arguments")?; + self.match_token( + Token::LPar, + "expected '(' before EBB arguments", + )?; // ebb-args ::= "(" * ebb-arg { "," ebb-arg } ")" self.parse_ebb_arg(ctx, ebb)?; @@ -1337,7 +1441,10 @@ impl<'a> Parser<'a> { } // ebb-args ::= "(" ebb-arg { "," ebb-arg } * ")" - self.match_token(Token::RPar, "expected ')' after EBB arguments")?; + self.match_token( + Token::RPar, + "expected ')' after EBB arguments", + )?; Ok(()) } @@ -1351,7 +1458,10 @@ impl<'a> Parser<'a> { let v = self.match_value("EBB argument must be a value")?; let v_location = self.loc; // ebb-arg ::= Value(v) * ":" Type(t) - self.match_token(Token::Colon, "expected ':' after EBB argument")?; + self.match_token( + Token::Colon, + "expected ':' after EBB argument", + )?; // ebb-arg ::= Value(v) ":" * Type(t) let t = self.match_type("expected EBB argument type")?; // Allocate the EBB argument and add the mapping. @@ -1366,9 +1476,11 @@ impl<'a> Parser<'a> { if let Some(ss) = ctx.map.get_ss(src_num) { Ok(ValueLoc::Stack(ss)) } else { - err!(self.loc, - "attempted to use undefined stack slot ss{}", - src_num) + err!( + self.loc, + "attempted to use undefined stack slot ss{}", + src_num + ) } } Some(Token::Name(name)) => { @@ -1391,16 +1503,19 @@ impl<'a> Parser<'a> { } } - fn parse_instruction_encoding(&mut self, - ctx: &Context) - -> Result<(Option, Option>)> { + fn parse_instruction_encoding( + &mut self, + ctx: &Context, + ) -> Result<(Option, Option>)> { let (mut encoding, mut result_locations) = (None, None); // encoding ::= "[" encoding_literal result_locations "]" if self.optional(Token::LBracket) { // encoding_literal ::= "-" | Identifier HexSequence if !self.optional(Token::Minus) { - let recipe = self.match_any_identifier("expected instruction encoding or '-'")?; + let recipe = self.match_any_identifier( + "expected instruction encoding or '-'", + )?; let bits = self.match_hex16("expected a hex sequence")?; if let Some(recipe_index) = ctx.find_recipe_index(recipe) { @@ -1426,8 +1541,10 @@ impl<'a> Parser<'a> { result_locations = Some(results); } - self.match_token(Token::RBracket, - "expected ']' to terminate instruction encoding")?; + self.match_token( + Token::RBracket, + "expected ']' to terminate instruction encoding", + )?; } Ok((encoding, result_locations)) @@ -1473,13 +1590,14 @@ impl<'a> Parser<'a> { // // instruction ::= [inst-results "="] Opcode(opc) ["." Type] ... // - fn parse_instruction(&mut self, - results: Vec, - encoding: Option, - result_locations: Option>, - ctx: &mut Context, - ebb: Ebb) - -> Result<()> { + fn parse_instruction( + &mut self, + results: Vec, + encoding: Option, + result_locations: Option>, + ctx: &mut Context, + ebb: Ebb, + ) -> Result<()> { // Collect comments for the next instruction to be allocated. self.gather_comments(ctx.function.dfg.next_inst()); @@ -1511,48 +1629,58 @@ impl<'a> Parser<'a> { // We still need to check that the number of result values in the source matches the opcode // or function call signature. We also need to create values with the right type for all // the instruction results. - let ctrl_typevar = self.infer_typevar(ctx, opcode, explicit_ctrl_type, &inst_data)?; + let ctrl_typevar = self.infer_typevar( + ctx, + opcode, + explicit_ctrl_type, + &inst_data, + )?; let inst = ctx.function.dfg.make_inst(inst_data); let num_results = ctx.function.dfg.make_inst_results(inst, ctrl_typevar); ctx.function.layout.append_inst(inst, ebb); - ctx.map - .def_entity(inst.into(), &opcode_loc) - .expect("duplicate inst references created"); + ctx.map.def_entity(inst.into(), &opcode_loc).expect( + "duplicate inst references created", + ); if let Some(encoding) = encoding { ctx.function.encodings[inst] = encoding; } if results.len() != num_results { - return err!(self.loc, - "instruction produces {} result values, {} given", - num_results, - results.len()); + return err!( + self.loc, + "instruction produces {} result values, {} given", + num_results, + results.len() + ); } if let Some(ref result_locations) = result_locations { if results.len() != result_locations.len() { - return err!(self.loc, - "instruction produces {} result values, but {} locations were \ + return err!( + self.loc, + "instruction produces {} result values, but {} locations were \ specified", - results.len(), - result_locations.len()); + results.len(), + result_locations.len() + ); } } // Now map the source result values to the just created instruction results. // Pass a reference to `ctx.values` instead of `ctx` itself since the `Values` iterator // holds a reference to `ctx.function`. - self.add_values(&mut ctx.map, - results.into_iter(), - ctx.function.dfg.inst_results(inst).iter().cloned())?; + self.add_values( + &mut ctx.map, + results.into_iter(), + ctx.function.dfg.inst_results(inst).iter().cloned(), + )?; if let Some(result_locations) = result_locations { - for (&value, loc) in ctx.function - .dfg - .inst_results(inst) - .iter() - .zip(result_locations) { + for (&value, loc) in ctx.function.dfg.inst_results(inst).iter().zip( + result_locations, + ) + { ctx.function.locations[value] = loc; } } @@ -1569,57 +1697,62 @@ impl<'a> Parser<'a> { // // Returns the controlling typevar for a polymorphic opcode, or `VOID` for a non-polymorphic // opcode. - fn infer_typevar(&self, - ctx: &Context, - opcode: Opcode, - explicit_ctrl_type: Option, - inst_data: &InstructionData) - -> Result { + fn infer_typevar( + &self, + ctx: &Context, + opcode: Opcode, + explicit_ctrl_type: Option, + inst_data: &InstructionData, + ) -> Result { let constraints = opcode.constraints(); - let ctrl_type = match explicit_ctrl_type { - Some(t) => t, - None => { - if constraints.use_typevar_operand() { - // This is an opcode that supports type inference, AND there was no - // explicit type specified. Look up `ctrl_value` to see if it was defined - // already. - // TBD: If it is defined in another block, the type should have been - // specified explicitly. It is unfortunate that the correctness of IL - // depends on the layout of the blocks. - let ctrl_src_value = inst_data - .typevar_operand(&ctx.function.dfg.value_lists) - .expect("Constraints <-> Format inconsistency"); - ctx.function - .dfg - .value_type(match ctx.map.get_value(ctrl_src_value) { - Some(v) => v, - None => { - if let Some(v) = ctx.aliases - .get(&ctrl_src_value) - .and_then(|&(aliased, _)| { - ctx.map.get_value(aliased) - }) { - v - } else { - return err!(self.loc, - "cannot determine type of operand {}", - ctrl_src_value); - } - } - }) - } else if constraints.is_polymorphic() { - // This opcode does not support type inference, so the explicit type - // variable is required. - return err!(self.loc, - "type variable required for polymorphic opcode, e.g. '{}.{}'", - opcode, - constraints.ctrl_typeset().unwrap().example()); - } else { - // This is a non-polymorphic opcode. No typevar needed. - VOID + let ctrl_type = + match explicit_ctrl_type { + Some(t) => t, + None => { + if constraints.use_typevar_operand() { + // This is an opcode that supports type inference, AND there was no + // explicit type specified. Look up `ctrl_value` to see if it was defined + // already. + // TBD: If it is defined in another block, the type should have been + // specified explicitly. It is unfortunate that the correctness of IL + // depends on the layout of the blocks. + let ctrl_src_value = inst_data + .typevar_operand(&ctx.function.dfg.value_lists) + .expect("Constraints <-> Format inconsistency"); + ctx.function.dfg.value_type( + match ctx.map.get_value(ctrl_src_value) { + Some(v) => v, + None => { + if let Some(v) = ctx.aliases.get(&ctrl_src_value).and_then( + |&(aliased, _)| ctx.map.get_value(aliased), + ) + { + v + } else { + return err!( + self.loc, + "cannot determine type of operand {}", + ctrl_src_value + ); + } + } + }, + ) + } else if constraints.is_polymorphic() { + // This opcode does not support type inference, so the explicit type + // variable is required. + return err!( + self.loc, + "type variable required for polymorphic opcode, e.g. '{}.{}'", + opcode, + constraints.ctrl_typeset().unwrap().example() + ); + } else { + // This is a non-polymorphic opcode. No typevar needed. + VOID + } } - } - }; + }; // Verify that `ctrl_type` is valid for the controlling type variable. We don't want to // attempt deriving types from an incorrect basis. @@ -1627,10 +1760,12 @@ impl<'a> Parser<'a> { if let Some(typeset) = constraints.ctrl_typeset() { // This is a polymorphic opcode. if !typeset.contains(ctrl_type) { - return err!(self.loc, - "{} is not a valid typevar for {}", - ctrl_type, - opcode); + return err!( + self.loc, + "{} is not a valid typevar for {}", + ctrl_type, + opcode + ); } } else { // Treat it as a syntax error to speficy a typevar on a non-polymorphic opcode. @@ -1644,8 +1779,9 @@ impl<'a> Parser<'a> { // Add mappings for a list of source values to their corresponding new values. fn add_values(&self, map: &mut SourceMap, results: S, new_results: V) -> Result<()> - where S: Iterator, - V: Iterator + where + S: Iterator, + V: Iterator, { for (src, val) in results.zip(new_results) { map.def_value(src, val, &self.loc)?; @@ -1682,17 +1818,21 @@ impl<'a> Parser<'a> { let args = self.parse_value_list()?; - self.match_token(Token::RPar, "expected ')' after arguments")?; + self.match_token( + Token::RPar, + "expected ')' after arguments", + )?; Ok(args) } // Parse the operands following the instruction opcode. // This depends on the format of the opcode. - fn parse_inst_operands(&mut self, - ctx: &mut Context, - opcode: Opcode) - -> Result { + fn parse_inst_operands( + &mut self, + ctx: &mut Context, + opcode: Opcode, + ) -> Result { let idata = match opcode.format() { InstructionFormat::Nullary => InstructionData::Nullary { opcode }, InstructionFormat::Unary => { @@ -1728,13 +1868,17 @@ impl<'a> Parser<'a> { InstructionFormat::UnaryGlobalVar => { InstructionData::UnaryGlobalVar { opcode, - global_var: self.match_gv("expected global variable") - .and_then(|num| ctx.get_gv(num, &self.loc))?, + global_var: self.match_gv("expected global variable").and_then(|num| { + ctx.get_gv(num, &self.loc) + })?, } } InstructionFormat::Binary => { let lhs = self.match_value("expected SSA value first operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let rhs = self.match_value("expected SSA value second operand")?; InstructionData::Binary { opcode, @@ -1743,8 +1887,13 @@ impl<'a> Parser<'a> { } InstructionFormat::BinaryImm => { let lhs = self.match_value("expected SSA value first operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; - let rhs = self.match_imm64("expected immediate integer second operand")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; + let rhs = self.match_imm64( + "expected immediate integer second operand", + )?; InstructionData::BinaryImm { opcode, arg: lhs, @@ -1755,9 +1904,15 @@ impl<'a> Parser<'a> { // Names here refer to the `select` instruction. // This format is also use by `fma`. let ctrl_arg = self.match_value("expected SSA value control operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let true_arg = self.match_value("expected SSA value true operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let false_arg = self.match_value("expected SSA value false operand")?; InstructionData::Ternary { opcode, @@ -1783,7 +1938,10 @@ impl<'a> Parser<'a> { } InstructionFormat::Branch => { let ctrl_arg = self.match_value("expected SSA value control operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let ebb_num = self.match_ebb("expected branch destination EBB")?; let args = self.parse_opt_value_list()?; InstructionData::Branch { @@ -1795,9 +1953,15 @@ impl<'a> Parser<'a> { InstructionFormat::BranchIcmp => { let cond = self.match_enum("expected intcc condition code")?; let lhs = self.match_value("expected SSA value first operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let rhs = self.match_value("expected SSA value second operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let ebb_num = self.match_ebb("expected branch destination EBB")?; let args = self.parse_opt_value_list()?; InstructionData::BranchIcmp { @@ -1809,9 +1973,15 @@ impl<'a> Parser<'a> { } InstructionFormat::InsertLane => { let lhs = self.match_value("expected SSA value first operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let lane = self.match_uimm8("expected lane number")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let rhs = self.match_value("expected SSA value last operand")?; InstructionData::InsertLane { opcode, @@ -1821,14 +1991,20 @@ impl<'a> Parser<'a> { } InstructionFormat::ExtractLane => { let arg = self.match_value("expected SSA value last operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let lane = self.match_uimm8("expected lane number")?; InstructionData::ExtractLane { opcode, lane, arg } } InstructionFormat::IntCompare => { let cond = self.match_enum("expected intcc condition code")?; let lhs = self.match_value("expected SSA value first operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let rhs = self.match_value("expected SSA value second operand")?; InstructionData::IntCompare { opcode, @@ -1839,7 +2015,10 @@ impl<'a> Parser<'a> { InstructionFormat::IntCompareImm => { let cond = self.match_enum("expected intcc condition code")?; let lhs = self.match_value("expected SSA value first operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let rhs = self.match_imm64("expected immediate second operand")?; InstructionData::IntCompareImm { opcode, @@ -1851,7 +2030,10 @@ impl<'a> Parser<'a> { InstructionFormat::FloatCompare => { let cond = self.match_enum("expected floatcc condition code")?; let lhs = self.match_value("expected SSA value first operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let rhs = self.match_value("expected SSA value second operand")?; InstructionData::FloatCompare { opcode, @@ -1860,11 +2042,20 @@ impl<'a> Parser<'a> { } } InstructionFormat::Call => { - let func_ref = self.match_fn("expected function reference") - .and_then(|num| ctx.get_fn(num, &self.loc))?; - self.match_token(Token::LPar, "expected '(' before arguments")?; + let func_ref = self.match_fn("expected function reference").and_then( + |num| { + ctx.get_fn(num, &self.loc) + }, + )?; + self.match_token( + Token::LPar, + "expected '(' before arguments", + )?; let args = self.parse_value_list()?; - self.match_token(Token::RPar, "expected ')' after arguments")?; + self.match_token( + Token::RPar, + "expected ')' after arguments", + )?; InstructionData::Call { opcode, func_ref, @@ -1872,13 +2063,25 @@ impl<'a> Parser<'a> { } } InstructionFormat::IndirectCall => { - let sig_ref = self.match_sig("expected signature reference") - .and_then(|num| ctx.get_sig(num, &self.loc))?; - self.match_token(Token::Comma, "expected ',' between operands")?; + let sig_ref = self.match_sig("expected signature reference").and_then( + |num| { + ctx.get_sig(num, &self.loc) + }, + )?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let callee = self.match_value("expected SSA value callee operand")?; - self.match_token(Token::LPar, "expected '(' before arguments")?; + self.match_token( + Token::LPar, + "expected '(' before arguments", + )?; let args = self.parse_value_list()?; - self.match_token(Token::RPar, "expected ')' after arguments")?; + self.match_token( + Token::RPar, + "expected ')' after arguments", + )?; InstructionData::IndirectCall { opcode, sig_ref, @@ -1887,7 +2090,10 @@ impl<'a> Parser<'a> { } InstructionFormat::BranchTable => { let arg = self.match_value("expected SSA value operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let table = self.match_jt().and_then(|num| ctx.get_jt(num, &self.loc))?; InstructionData::BranchTable { opcode, arg, table } } @@ -1903,7 +2109,10 @@ impl<'a> Parser<'a> { } InstructionFormat::StackStore => { let arg = self.match_value("expected SSA value operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let ss = self.match_ss("expected stack slot number: ss«n»") .and_then(|num| ctx.get_ss(num, &self.loc))?; let offset = self.optional_offset32()?; @@ -1925,7 +2134,10 @@ impl<'a> Parser<'a> { } InstructionFormat::HeapStore => { let arg = self.match_value("expected SSA value operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let addr = self.match_value("expected SSA value address")?; let offset = self.optional_uoffset32()?; InstructionData::HeapStore { @@ -1935,11 +2147,18 @@ impl<'a> Parser<'a> { } } InstructionFormat::HeapAddr => { - let heap = self.match_heap("expected heap identifier") - .and_then(|h| ctx.get_heap(h, &self.loc))?; - self.match_token(Token::Comma, "expected ',' between operands")?; + let heap = self.match_heap("expected heap identifier").and_then(|h| { + ctx.get_heap(h, &self.loc) + })?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let arg = self.match_value("expected SSA value heap address")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let imm = self.match_uimm32("expected 32-bit integer size")?; InstructionData::HeapAddr { opcode, @@ -1962,7 +2181,10 @@ impl<'a> Parser<'a> { InstructionFormat::Store => { let flags = self.optional_memflags(); let arg = self.match_value("expected SSA value operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let addr = self.match_value("expected SSA value address")?; let offset = self.optional_offset32()?; InstructionData::Store { @@ -1974,9 +2196,15 @@ impl<'a> Parser<'a> { } InstructionFormat::RegMove => { let arg = self.match_value("expected SSA value operand")?; - self.match_token(Token::Comma, "expected ',' between operands")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; let src = self.match_regunit(ctx.unique_isa)?; - self.match_token(Token::Arrow, "expected '->' between register units")?; + self.match_token( + Token::Arrow, + "expected '->' between register units", + )?; let dst = self.match_regunit(ctx.unique_isa)?; InstructionData::RegMove { opcode, @@ -2015,14 +2243,15 @@ mod tests { #[test] fn aliases() { - let (func, details) = Parser::new("function %qux() native { + let (func, details) = Parser::new( + "function %qux() native { ebb0: v4 = iconst.i8 6 v3 -> v4 v1 = iadd_imm v3, 17 - }") - .parse_function(None) - .unwrap(); + }", + ).parse_function(None) + .unwrap(); assert_eq!(func.name.to_string(), "%qux"); let v4 = details.map.lookup_str("v4").unwrap(); assert_eq!(v4.to_string(), "v0"); @@ -2047,45 +2276,58 @@ mod tests { let sig2 = Parser::new("(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 spiderwasm") .parse_signature(None) .unwrap(); - assert_eq!(sig2.to_string(), - "(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 spiderwasm"); + assert_eq!( + sig2.to_string(), + "(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 spiderwasm" + ); assert_eq!(sig2.call_conv, CallConv::SpiderWASM); // Old-style signature without a calling convention. - assert_eq!(Parser::new("()").parse_signature(None).unwrap().to_string(), - "() native"); - assert_eq!(Parser::new("() notacc") - .parse_signature(None) - .unwrap_err() - .to_string(), - "1: unknown calling convention: notacc"); + assert_eq!( + Parser::new("()").parse_signature(None).unwrap().to_string(), + "() native" + ); + assert_eq!( + Parser::new("() notacc") + .parse_signature(None) + .unwrap_err() + .to_string(), + "1: unknown calling convention: notacc" + ); // `void` is not recognized as a type by the lexer. It should not appear in files. - assert_eq!(Parser::new("() -> void") - .parse_signature(None) - .unwrap_err() - .to_string(), - "1: expected argument type"); - assert_eq!(Parser::new("i8 -> i8") - .parse_signature(None) - .unwrap_err() - .to_string(), - "1: expected function signature: ( args... )"); - assert_eq!(Parser::new("(i8 -> i8") - .parse_signature(None) - .unwrap_err() - .to_string(), - "1: expected ')' after function arguments"); + assert_eq!( + Parser::new("() -> void") + .parse_signature(None) + .unwrap_err() + .to_string(), + "1: expected argument type" + ); + assert_eq!( + Parser::new("i8 -> i8") + .parse_signature(None) + .unwrap_err() + .to_string(), + "1: expected function signature: ( args... )" + ); + assert_eq!( + Parser::new("(i8 -> i8") + .parse_signature(None) + .unwrap_err() + .to_string(), + "1: expected ')' after function arguments" + ); } #[test] fn stack_slot_decl() { - let (func, _) = Parser::new("function %foo() native { + let (func, _) = Parser::new( + "function %foo() native { ss3 = incoming_arg 13 ss1 = spill_slot 1 - }") - .parse_function(None) - .unwrap(); + }", + ).parse_function(None) + .unwrap(); assert_eq!(func.name.to_string(), "%foo"); let mut iter = func.stack_slots.keys(); let ss0 = iter.next().unwrap(); @@ -2099,24 +2341,28 @@ mod tests { assert_eq!(iter.next(), None); // Catch duplicate definitions. - assert_eq!(Parser::new("function %bar() native { + assert_eq!( + Parser::new( + "function %bar() native { ss1 = spill_slot 13 ss1 = spill_slot 1 - }") - .parse_function(None) - .unwrap_err() - .to_string(), - "3: duplicate stack slot: ss1"); + }", + ).parse_function(None) + .unwrap_err() + .to_string(), + "3: duplicate stack slot: ss1" + ); } #[test] fn ebb_header() { - let (func, _) = Parser::new("function %ebbs() native { + let (func, _) = Parser::new( + "function %ebbs() native { ebb0: ebb4(v3: i32): - }") - .parse_function(None) - .unwrap(); + }", + ).parse_function(None) + .unwrap(); assert_eq!(func.name.to_string(), "%ebbs"); let mut ebbs = func.layout.ebbs(); @@ -2132,7 +2378,8 @@ mod tests { #[test] fn comments() { - let (func, Details { comments, .. }) = Parser::new("; before + let (func, Details { comments, .. }) = Parser::new( + "; before function %comment() native { ; decl ss10 = outgoing_arg 13 ; stackslot. ; Still stackslot. @@ -2141,16 +2388,18 @@ mod tests { ebb0: ; Basic block trap ; Instruction } ; Trailing. - ; More trailing.") - .parse_function(None) - .unwrap(); + ; More trailing.", + ).parse_function(None) + .unwrap(); assert_eq!(func.name.to_string(), "%comment"); assert_eq!(comments.len(), 8); // no 'before' comment. - assert_eq!(comments[0], - Comment { - entity: AnyEntity::Function, - text: "; decl", - }); + assert_eq!( + comments[0], + Comment { + entity: AnyEntity::Function, + text: "; decl", + } + ); assert_eq!(comments[1].entity.to_string(), "ss0"); assert_eq!(comments[2].entity.to_string(), "ss0"); assert_eq!(comments[2].text, "; Still stackslot."); @@ -2168,13 +2417,14 @@ mod tests { #[test] fn test_file() { - let tf = parse_test("; before + let tf = parse_test( + "; before test cfg option=5 test verify set enable_float=false ; still preamble - function %comment() native {}") - .unwrap(); + function %comment() native {}", + ).unwrap(); assert_eq!(tf.commands.len(), 2); assert_eq!(tf.commands[0].command, "cfg"); assert_eq!(tf.commands[1].command, "verify"); @@ -2195,20 +2445,27 @@ mod tests { #[test] #[cfg(build_riscv)] fn isa_spec() { - assert!(parse_test("isa - function %foo() native {}") - .is_err()); + assert!( + parse_test( + "isa + function %foo() native {}", + ).is_err() + ); - assert!(parse_test("isa riscv + assert!( + parse_test( + "isa riscv set enable_float=false - function %foo() native {}") - .is_err()); + function %foo() native {}", + ).is_err() + ); - match parse_test("set enable_float=false + match parse_test( + "set enable_float=false isa riscv - function %foo() native {}") - .unwrap() - .isa_spec { + function %foo() native {}", + ).unwrap() + .isa_spec { IsaSpec::None(_) => panic!("Expected some ISA"), IsaSpec::Some(v) => { assert_eq!(v.len(), 1); @@ -2220,37 +2477,43 @@ mod tests { #[test] fn binary_function_name() { // Valid characters in the name. - let func = Parser::new("function #1234567890AbCdEf() native { + let func = Parser::new( + "function #1234567890AbCdEf() native { ebb0: trap - }") - .parse_function(None) - .unwrap() - .0; + }", + ).parse_function(None) + .unwrap() + .0; assert_eq!(func.name.to_string(), "#1234567890abcdef"); // Invalid characters in the name. - let mut parser = Parser::new("function #12ww() native { + let mut parser = Parser::new( + "function #12ww() native { ebb0: trap - }"); + }", + ); assert!(parser.parse_function(None).is_err()); // The length of binary function name should be multiple of two. - let mut parser = Parser::new("function #1() native { + let mut parser = Parser::new( + "function #1() native { ebb0: trap - }"); + }", + ); assert!(parser.parse_function(None).is_err()); // Empty binary function name should be valid. - let func = Parser::new("function #() native { + let func = Parser::new( + "function #() native { ebb0: trap - }") - .parse_function(None) - .unwrap() - .0; + }", + ).parse_function(None) + .unwrap() + .0; assert_eq!(func.name.to_string(), "%"); } } diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index daa6422543..2135e5e037 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -76,24 +76,24 @@ impl SourceMap { /// Returns the entity reference corresponding to `name`, if it exists. pub fn lookup_str(&self, name: &str) -> Option { split_entity_name(name).and_then(|(ent, num)| match ent { - "v" => { - Value::with_number(num) - .and_then(|v| self.get_value(v)) - .map(AnyEntity::Value) - } - "ebb" => { - Ebb::with_number(num) - .and_then(|e| self.get_ebb(e)) - .map(AnyEntity::Ebb) - } - "ss" => self.get_ss(num).map(AnyEntity::StackSlot), - "gv" => self.get_gv(num).map(AnyEntity::GlobalVar), - "heap" => self.get_heap(num).map(AnyEntity::Heap), - "sig" => self.get_sig(num).map(AnyEntity::SigRef), - "fn" => self.get_fn(num).map(AnyEntity::FuncRef), - "jt" => self.get_jt(num).map(AnyEntity::JumpTable), - _ => None, - }) + "v" => { + Value::with_number(num) + .and_then(|v| self.get_value(v)) + .map(AnyEntity::Value) + } + "ebb" => { + Ebb::with_number(num).and_then(|e| self.get_ebb(e)).map( + AnyEntity::Ebb, + ) + } + "ss" => self.get_ss(num).map(AnyEntity::StackSlot), + "gv" => self.get_gv(num).map(AnyEntity::GlobalVar), + "heap" => self.get_heap(num).map(AnyEntity::Heap), + "sig" => self.get_sig(num).map(AnyEntity::SigRef), + "fn" => self.get_fn(num).map(AnyEntity::FuncRef), + "jt" => self.get_jt(num).map(AnyEntity::JumpTable), + _ => None, + }) } /// Get the source location where an entity was defined. @@ -110,9 +110,11 @@ impl SourceMap { Ok(()) } None => { - err!(self.location(loc).unwrap_or_default(), - "undefined reference: {}", - ebb) + err!( + self.location(loc).unwrap_or_default(), + "undefined reference: {}", + ebb + ) } } } @@ -125,9 +127,11 @@ impl SourceMap { Ok(()) } None => { - err!(self.location(loc).unwrap_or_default(), - "undefined reference: {}", - val) + err!( + self.location(loc).unwrap_or_default(), + "undefined reference: {}", + val + ) } } } @@ -148,9 +152,11 @@ impl SourceMap { Ok(()) } None => { - err!(self.location(loc).unwrap_or_default(), - "undefined reference: {}", - gv) + err!( + self.location(loc).unwrap_or_default(), + "undefined reference: {}", + gv + ) } } } @@ -272,13 +278,14 @@ mod tests { #[test] fn details() { - let tf = parse_test("function %detail() { + let tf = parse_test( + "function %detail() { ss10 = incoming_arg 13 jt10 = jump_table ebb0 ebb0(v4: i32, v7: i32): v10 = iadd v4, v7 - }") - .unwrap(); + }", + ).unwrap(); let map = &tf.functions[0].1.map; assert_eq!(map.lookup_str("v0"), None); diff --git a/lib/reader/src/testcommand.rs b/lib/reader/src/testcommand.rs index a147487c77..ad856e9949 100644 --- a/lib/reader/src/testcommand.rs +++ b/lib/reader/src/testcommand.rs @@ -95,7 +95,9 @@ mod tests { assert_eq!(&TestCommand::new("cat").to_string(), "cat\n"); assert_eq!(&TestCommand::new("cat ").to_string(), "cat\n"); assert_eq!(&TestCommand::new("cat 1 ").to_string(), "cat 1\n"); - assert_eq!(&TestCommand::new("cat one=4 two t").to_string(), - "cat one=4 two t\n"); + assert_eq!( + &TestCommand::new("cat one=4 two t").to_string(), + "cat one=4 two t\n" + ); } } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 708bdba6fa..12f3829338 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -135,7 +135,7 @@ struct TranslationState { /// Holds mappings between the function and signatures indexes in the Wasm module and their /// references as imports of the Cretonne IL function. -#[derive(Clone,Debug)] +#[derive(Clone, Debug)] pub struct FunctionImports { /// Mappings index in function index space -> index in function local imports pub functions: HashMap, @@ -153,16 +153,17 @@ impl FunctionImports { } /// Returns a well-formed Cretonne IL function from a wasm function body and a signature. -pub fn translate_function_body(parser: &mut Parser, - function_index: FunctionIndex, - sig: Signature, - locals: &[(usize, Type)], - exports: &Option>, - signatures: &[Signature], - functions: &[SignatureIndex], - il_builder: &mut ILBuilder, - runtime: &mut WasmRuntime) - -> Result<(Function, FunctionImports), String> { +pub fn translate_function_body( + parser: &mut Parser, + function_index: FunctionIndex, + sig: Signature, + locals: &[(usize, Type)], + exports: &Option>, + signatures: &[Signature], + functions: &[SignatureIndex], + il_builder: &mut ILBuilder, + runtime: &mut WasmRuntime, +) -> Result<(Function, FunctionImports), String> { runtime.next_function(); // First we build the Function object with its name and signature let mut func = Function::new(); @@ -216,39 +217,45 @@ pub fn translate_function_body(parser: &mut Parser, // We initialize the control stack with the implicit function block let end_ebb = builder.create_ebb(); control_stack.push(ControlStackFrame::Block { - destination: end_ebb, - original_stack_size: 0, - return_values: sig.return_types - .iter() - .map(|argty| argty.value_type) - .collect(), - reachable: false, - }); + destination: end_ebb, + original_stack_size: 0, + return_values: sig.return_types + .iter() + .map(|argty| argty.value_type) + .collect(), + reachable: false, + }); // Now the main loop that reads every wasm instruction and translates it loop { let parser_state = parser.read(); match *parser_state { ParserState::CodeOperator(ref op) => { - debug_assert!(state.phantom_unreachable_stack_depth == 0 || - state.real_unreachable_stack_depth > 0); + debug_assert!( + state.phantom_unreachable_stack_depth == 0 || + state.real_unreachable_stack_depth > 0 + ); if state.real_unreachable_stack_depth > 0 { - translate_unreachable_operator(op, - &mut builder, - &mut stack, - &mut control_stack, - &mut state) + translate_unreachable_operator( + op, + &mut builder, + &mut stack, + &mut control_stack, + &mut state, + ) } else { - translate_operator(op, - &mut builder, - runtime, - &mut stack, - &mut control_stack, - &mut state, - &sig, - &functions, - &signatures, - &exports, - &mut func_imports) + translate_operator( + op, + &mut builder, + runtime, + &mut stack, + &mut control_stack, + &mut state, + &sig, + &functions, + &signatures, + &exports, + &mut func_imports, + ) } } @@ -281,17 +288,19 @@ pub fn translate_function_body(parser: &mut Parser, /// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted /// a return. -fn translate_operator(op: &Operator, - builder: &mut FunctionBuilder, - runtime: &mut WasmRuntime, - stack: &mut Vec, - control_stack: &mut Vec, - state: &mut TranslationState, - sig: &Signature, - functions: &[SignatureIndex], - signatures: &[Signature], - exports: &Option>, - func_imports: &mut FunctionImports) { +fn translate_operator( + op: &Operator, + builder: &mut FunctionBuilder, + runtime: &mut WasmRuntime, + stack: &mut Vec, + control_stack: &mut Vec, + state: &mut TranslationState, + sig: &Signature, + functions: &[SignatureIndex], + signatures: &[Signature], + exports: &Option>, + func_imports: &mut FunctionImports, +) { // This big match treats all Wasm code operators. match *op { /********************************** Locals **************************************** @@ -357,11 +366,11 @@ fn translate_operator(op: &Operator, Err(_) => {} } control_stack.push(ControlStackFrame::Block { - destination: next, - return_values: translate_type(ty).unwrap(), - original_stack_size: stack.len(), - reachable: false, - }); + destination: next, + return_values: translate_type(ty).unwrap(), + original_stack_size: stack.len(), + reachable: false, + }); } Operator::Loop { ty } => { let loop_body = builder.create_ebb(); @@ -374,12 +383,12 @@ fn translate_operator(op: &Operator, } builder.ins().jump(loop_body, &[]); control_stack.push(ControlStackFrame::Loop { - destination: next, - header: loop_body, - return_values: translate_type(ty).unwrap(), - original_stack_size: stack.len(), - reachable: false, - }); + destination: next, + header: loop_body, + return_values: translate_type(ty).unwrap(), + original_stack_size: stack.len(), + reachable: false, + }); builder.switch_to_block(loop_body, &[]); } Operator::If { ty } => { @@ -399,12 +408,12 @@ fn translate_operator(op: &Operator, Err(_) => {} } control_stack.push(ControlStackFrame::If { - destination: if_not, - branch_inst: jump_inst, - return_values: translate_type(ty).unwrap(), - original_stack_size: stack.len(), - reachable: false, - }); + destination: if_not, + branch_inst: jump_inst, + return_values: translate_type(ty).unwrap(), + original_stack_size: stack.len(), + reachable: false, + }); } Operator::Else => { // We take the control frame pushed by the if, use its ebb as the else body @@ -434,9 +443,10 @@ fn translate_operator(op: &Operator, if !builder.is_unreachable() || !builder.is_pristine() { let cut_index = stack.len() - frame.return_values().len(); let jump_args = stack.split_off(cut_index); - builder - .ins() - .jump(frame.following_code(), jump_args.as_slice()); + builder.ins().jump( + frame.following_code(), + jump_args.as_slice(), + ); } builder.switch_to_block(frame.following_code(), frame.return_values()); builder.seal_block(frame.following_code()); @@ -478,9 +488,10 @@ fn translate_operator(op: &Operator, let cut_index = stack.len() - frame.return_values().len(); stack.split_off(cut_index) }; - builder - .ins() - .jump(frame.br_destination(), jump_args.as_slice()); + builder.ins().jump( + frame.br_destination(), + jump_args.as_slice(), + ); // We signal that all the code that follows until the next End is unreachable frame.set_reachable(); state.real_unreachable_stack_depth = 1 + relative_depth as usize; @@ -495,9 +506,11 @@ fn translate_operator(op: &Operator, let cut_index = stack.len() - frame.return_values().len(); stack.split_off(cut_index) }; - builder - .ins() - .brnz(val, frame.br_destination(), jump_args.as_slice()); + builder.ins().brnz( + val, + frame.br_destination(), + jump_args.as_slice(), + ); // The values returned by the branch are still available for the reachable // code that comes after it frame.set_reachable(); @@ -545,10 +558,9 @@ fn translate_operator(op: &Operator, let cut_index = stack.len() - jump_args_count; let jump_args = stack.split_off(cut_index); let jt = builder.create_jump_table(); - let dest_ebbs: HashMap = depths - .iter() - .enumerate() - .fold(HashMap::new(), |mut acc, (index, &depth)| { + let dest_ebbs: HashMap = + depths.iter().enumerate().fold(HashMap::new(), |mut acc, + (index, &depth)| { if acc.get(&(depth as usize)).is_none() { let branch_ebb = builder.create_ebb(); builder.insert_jump_table_entry(jt, index, branch_ebb); @@ -592,15 +604,18 @@ fn translate_operator(op: &Operator, let args_num = args_count(function_index as usize, functions, signatures); let cut_index = stack.len() - args_num; let call_args = stack.split_off(cut_index); - let internal_function_index = find_function_import(function_index as usize, - builder, - func_imports, - functions, - exports, - signatures); - let call_inst = builder - .ins() - .call(internal_function_index, call_args.as_slice()); + let internal_function_index = find_function_import( + function_index as usize, + builder, + func_imports, + functions, + exports, + signatures, + ); + let call_inst = builder.ins().call( + internal_function_index, + call_args.as_slice(), + ); let ret_values = builder.inst_results(call_inst); for val in ret_values { stack.push(*val); @@ -1107,9 +1122,11 @@ fn translate_operator(op: &Operator, Operator::I32LeU | Operator::I64LeU => { let arg2 = stack.pop().unwrap(); let arg1 = stack.pop().unwrap(); - let val = builder - .ins() - .icmp(IntCC::UnsignedLessThanOrEqual, arg1, arg2); + let val = builder.ins().icmp( + IntCC::UnsignedLessThanOrEqual, + arg1, + arg2, + ); stack.push(builder.ins().bint(I32, val)); } Operator::I32GtS | Operator::I64GtS => { @@ -1127,17 +1144,21 @@ fn translate_operator(op: &Operator, Operator::I32GeS | Operator::I64GeS => { let arg2 = stack.pop().unwrap(); let arg1 = stack.pop().unwrap(); - let val = builder - .ins() - .icmp(IntCC::SignedGreaterThanOrEqual, arg1, arg2); + let val = builder.ins().icmp( + IntCC::SignedGreaterThanOrEqual, + arg1, + arg2, + ); stack.push(builder.ins().bint(I32, val)); } Operator::I32GeU | Operator::I64GeU => { let arg2 = stack.pop().unwrap(); let arg1 = stack.pop().unwrap(); - let val = builder - .ins() - .icmp(IntCC::UnsignedGreaterThanOrEqual, arg1, arg2); + let val = builder.ins().icmp( + IntCC::UnsignedGreaterThanOrEqual, + arg1, + arg2, + ); stack.push(builder.ins().bint(I32, val)); } Operator::I32Eqz | Operator::I64Eqz => { @@ -1199,11 +1220,13 @@ fn translate_operator(op: &Operator, /// Deals with a Wasm instruction located in an unreachable portion of the code. Most of them /// are dropped but special ones like `End` or `Else` signal the potential end of the unreachable /// portion so the translation state muts be updated accordingly. -fn translate_unreachable_operator(op: &Operator, - builder: &mut FunctionBuilder, - stack: &mut Vec, - control_stack: &mut Vec, - state: &mut TranslationState) { +fn translate_unreachable_operator( + op: &Operator, + builder: &mut FunctionBuilder, + stack: &mut Vec, + control_stack: &mut Vec, + state: &mut TranslationState, +) { // We don't translate because the code is unreachable // Nevertheless we have to record a phantom stack for this code // to know when the unreachable code ends @@ -1253,15 +1276,15 @@ fn translate_unreachable_operator(op: &Operator, } else { // Encountering an real else means that the code in the else // clause is reachable again - let (branch_inst, original_stack_size) = match &control_stack[control_stack.len() - - 1] { - &ControlStackFrame::If { - branch_inst, - original_stack_size, - .. - } => (branch_inst, original_stack_size), - _ => panic!("should not happen"), - }; + let (branch_inst, original_stack_size) = + match &control_stack[control_stack.len() - 1] { + &ControlStackFrame::If { + branch_inst, + original_stack_size, + .. + } => (branch_inst, original_stack_size), + _ => panic!("should not happen"), + }; // We change the target of the branch instruction let else_ebb = builder.create_ebb(); builder.change_jump_destination(branch_inst, else_ebb); @@ -1279,22 +1302,24 @@ fn translate_unreachable_operator(op: &Operator, } } -fn args_count(index: FunctionIndex, - functions: &[SignatureIndex], - signatures: &[Signature]) - -> usize { +fn args_count( + index: FunctionIndex, + functions: &[SignatureIndex], + signatures: &[Signature], +) -> usize { signatures[functions[index] as usize].argument_types.len() } // Given a index in the function index space, search for it in the function imports and if it is // not there add it to the function imports. -fn find_function_import(index: FunctionIndex, - builder: &mut FunctionBuilder, - func_imports: &mut FunctionImports, - functions: &[SignatureIndex], - exports: &Option>, - signatures: &[Signature]) - -> FuncRef { +fn find_function_import( + index: FunctionIndex, + builder: &mut FunctionBuilder, + func_imports: &mut FunctionImports, + functions: &[SignatureIndex], + exports: &Option>, + signatures: &[Signature], +) -> FuncRef { match func_imports.functions.get(&index) { Some(local_index) => return *local_index, None => {} @@ -1303,21 +1328,18 @@ fn find_function_import(index: FunctionIndex, let sig_index = functions[index]; match func_imports.signatures.get(&(sig_index as usize)) { Some(local_sig_index) => { - let local_func_index = - builder.import_function(ExtFuncData { - name: match exports { - &None => FunctionName::new(""), - &Some(ref exports) => { - match exports.get(&index) { - None => FunctionName::new(""), - Some(name) => { - FunctionName::new(name.clone()) - } - } - } - }, - signature: *local_sig_index, - }); + let local_func_index = builder.import_function(ExtFuncData { + name: match exports { + &None => FunctionName::new(""), + &Some(ref exports) => { + match exports.get(&index) { + None => FunctionName::new(""), + Some(name) => FunctionName::new(name.clone()), + } + } + }, + signature: *local_sig_index, + }); func_imports.functions.insert(index, local_func_index); return local_func_index; } @@ -1325,38 +1347,40 @@ fn find_function_import(index: FunctionIndex, }; // We have to import the signature let sig_local_index = builder.import_signature(signatures[sig_index as usize].clone()); - func_imports - .signatures - .insert(sig_index as usize, sig_local_index); - let local_func_index = - builder.import_function(ExtFuncData { - name: match exports { - &None => FunctionName::new(""), - &Some(ref exports) => { - match exports.get(&index) { - None => FunctionName::new(""), - Some(name) => FunctionName::new(name.clone()), - } - } - }, - signature: sig_local_index, - }); + func_imports.signatures.insert( + sig_index as usize, + sig_local_index, + ); + let local_func_index = builder.import_function(ExtFuncData { + name: match exports { + &None => FunctionName::new(""), + &Some(ref exports) => { + match exports.get(&index) { + None => FunctionName::new(""), + Some(name) => FunctionName::new(name.clone()), + } + } + }, + signature: sig_local_index, + }); func_imports.functions.insert(index, local_func_index); local_func_index } -fn find_signature_import(sig_index: SignatureIndex, - builder: &mut FunctionBuilder, - func_imports: &mut FunctionImports, - signatures: &[Signature]) - -> SigRef { +fn find_signature_import( + sig_index: SignatureIndex, + builder: &mut FunctionBuilder, + func_imports: &mut FunctionImports, + signatures: &[Signature], +) -> SigRef { match func_imports.signatures.get(&(sig_index as usize)) { Some(local_sig_index) => return *local_sig_index, None => {} } let sig_local_index = builder.import_signature(signatures[sig_index as usize].clone()); - func_imports - .signatures - .insert(sig_index as usize, sig_local_index); + func_imports.signatures.insert( + sig_index as usize, + sig_local_index, + ); sig_local_index } diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 99071b1bc8..0aaa35e697 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -34,7 +34,7 @@ pub enum FunctionTranslation { Import(), } -#[derive(Clone,Debug)] +#[derive(Clone, Debug)] /// Mappings describing the relations between imports of the Cretonne IL functions and the /// functions in the WebAssembly module. pub struct ImportMappings { @@ -58,9 +58,10 @@ impl ImportMappings { /// [`Function`](../cretonne/ir/function/struct.Function.html). /// Returns the functions and also the mappings for imported functions and signature between the /// indexes in the wasm module and the indexes inside each functions. -pub fn translate_module(data: &[u8], - runtime: &mut WasmRuntime) - -> Result { +pub fn translate_module( + data: &[u8], + runtime: &mut WasmRuntime, +) -> Result { let mut parser = Parser::new(data); match *parser.read() { ParserState::BeginWasm { .. } => {} @@ -207,9 +208,9 @@ pub fn translate_module(data: &[u8], } ParserState::EndWasm => { return Ok(TranslationResult { - functions: Vec::new(), - start_index: None, - }) + functions: Vec::new(), + start_index: None, + }) } ParserState::BeginSection { code: SectionCode::Data, .. } => { match parse_data_section(&mut parser, runtime, &globals) { @@ -242,32 +243,36 @@ pub fn translate_module(data: &[u8], locals .iter() .map(|&(index, ref ty)| { - (index as usize, - match type_to_type(ty) { - Ok(ty) => ty, - Err(()) => panic!("unsupported type for local variable"), - }) - }) + ( + index as usize, + match type_to_type(ty) { + Ok(ty) => ty, + Err(()) => panic!("unsupported type for local variable"), + }, + ) + }) .collect() } ParserState::EndSection => break, _ => return Err(String::from(format!("wrong content in code section"))), }; let signature = signatures[functions[function_index as usize] as usize].clone(); - match translate_function_body(&mut parser, - function_index, - signature, - &locals, - &exports, - &signatures, - &functions, - &mut il_builder, - runtime) { + match translate_function_body( + &mut parser, + function_index, + signature, + &locals, + &exports, + &signatures, + &functions, + &mut il_builder, + runtime, + ) { Ok((il_func, imports)) => { il_functions.push(FunctionTranslation::Code { - il: il_func, - imports: invert_hashmaps(imports), - }) + il: il_func, + imports: invert_hashmaps(imports), + }) } Err(s) => return Err(s), } @@ -285,9 +290,9 @@ pub fn translate_module(data: &[u8], } ParserState::EndWasm => { return Ok(TranslationResult { - functions: il_functions, - start_index, - }) + functions: il_functions, + start_index, + }) } _ => (), } diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 5969c0ee8f..f52b9e9d43 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -21,19 +21,20 @@ impl DummyRuntime { } impl WasmRuntime for DummyRuntime { - fn translate_get_global(&self, - builder: &mut FunctionBuilder, - global_index: GlobalIndex) - -> Value { + fn translate_get_global( + &self, + builder: &mut FunctionBuilder, + global_index: GlobalIndex, + ) -> Value { let ref glob = self.globals.get(global_index as usize).unwrap(); match glob.ty { I32 => builder.ins().iconst(glob.ty, -1), I64 => builder.ins().iconst(glob.ty, -1), F32 => builder.ins().f32const(Ieee32::with_bits(0xbf800000)), // -1.0 F64 => { - builder - .ins() - .f64const(Ieee64::with_bits(0xbff0000000000000)) + builder.ins().f64const( + Ieee64::with_bits(0xbff0000000000000), + ) } // -1.0 _ => panic!("should not happen"), } @@ -48,19 +49,21 @@ impl WasmRuntime for DummyRuntime { fn translate_current_memory(&mut self, builder: &mut FunctionBuilder) -> Value { builder.ins().iconst(I32, -1) } - fn translate_call_indirect<'a>(&self, - builder: &'a mut FunctionBuilder, - sig_ref: SigRef, - index_val: Value, - call_args: &[Value]) - -> &'a [Value] { + fn translate_call_indirect<'a>( + &self, + builder: &'a mut FunctionBuilder, + sig_ref: SigRef, + index_val: Value, + call_args: &[Value], + ) -> &'a [Value] { let call_inst = builder.ins().call_indirect(sig_ref, index_val, call_args); builder.inst_results(call_inst) } - fn translate_memory_base_address(&self, - builder: &mut FunctionBuilder, - _: MemoryIndex) - -> Value { + fn translate_memory_base_address( + &self, + builder: &mut FunctionBuilder, + _: MemoryIndex, + ) -> Value { builder.ins().iconst(I64, 0) } fn declare_global(&mut self, global: Global) { @@ -75,11 +78,12 @@ impl WasmRuntime for DummyRuntime { fn declare_memory(&mut self, _: Memory) { //We do nothing } - fn declare_data_initialization(&mut self, - _: MemoryIndex, - _: usize, - _: &[u8]) - -> Result<(), String> { + fn declare_data_initialization( + &mut self, + _: MemoryIndex, + _: usize, + _: &[u8], + ) -> Result<(), String> { // We do nothing Ok(()) } diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 24e98b0d58..9f346337ab 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -14,48 +14,56 @@ pub trait WasmRuntime { /// Declares a table to the runtime. fn declare_table(&mut self, table: Table); /// Fills a declared table with references to functions in the module. - fn declare_table_elements(&mut self, - table_index: TableIndex, - offset: usize, - elements: &[FunctionIndex]); + fn declare_table_elements( + &mut self, + table_index: TableIndex, + offset: usize, + elements: &[FunctionIndex], + ); /// Declares a memory to the runtime fn declare_memory(&mut self, memory: Memory); /// Fills a declared memory with bytes at module instantiation. - fn declare_data_initialization(&mut self, - memory_index: MemoryIndex, - offset: usize, - data: &[u8]) - -> Result<(), String>; + fn declare_data_initialization( + &mut self, + memory_index: MemoryIndex, + offset: usize, + data: &[u8], + ) -> Result<(), String>; /// Call this function after having declared all the runtime elements but prior to the /// function body translation. fn begin_translation(&mut self); /// Call this function between each function body translation. fn next_function(&mut self); /// Translates a `get_global` wasm instruction. - fn translate_get_global(&self, - builder: &mut FunctionBuilder, - global_index: GlobalIndex) - -> Value; + fn translate_get_global( + &self, + builder: &mut FunctionBuilder, + global_index: GlobalIndex, + ) -> Value; /// Translates a `set_global` wasm instruction. - fn translate_set_global(&self, - builder: &mut FunctionBuilder, - global_index: GlobalIndex, - val: Value); + fn translate_set_global( + &self, + builder: &mut FunctionBuilder, + global_index: GlobalIndex, + val: Value, + ); /// Translates a `grow_memory` wasm instruction. Returns the old size (in pages) of the memory. fn translate_grow_memory(&mut self, builder: &mut FunctionBuilder, val: Value) -> Value; /// Translates a `current_memory` wasm instruction. Returns the size in pages of the memory. fn translate_current_memory(&mut self, builder: &mut FunctionBuilder) -> Value; /// Returns the base address of a wasm memory as a Cretonne `Value`. - fn translate_memory_base_address(&self, - builder: &mut FunctionBuilder, - index: MemoryIndex) - -> Value; + fn translate_memory_base_address( + &self, + builder: &mut FunctionBuilder, + index: MemoryIndex, + ) -> Value; /// Translates a `call_indirect` wasm instruction. It involves looking up the value contained /// it the table at location `index_val` and calling the corresponding function. - fn translate_call_indirect<'a>(&self, - builder: &'a mut FunctionBuilder, - sig_ref: SigRef, - index_val: Value, - call_args: &[Value]) - -> &'a [Value]; + fn translate_call_indirect<'a>( + &self, + builder: &'a mut FunctionBuilder, + sig_ref: SigRef, + index_val: Value, + call_args: &[Value], + ) -> &'a [Value]; } diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index fc110abf3c..95fb673d17 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -24,8 +24,9 @@ pub enum SectionParsingError { } /// Reads the Type Section of the wasm module and returns the corresponding function signatures. -pub fn parse_function_signatures(parser: &mut Parser) - -> Result, SectionParsingError> { +pub fn parse_function_signatures( + parser: &mut Parser, +) -> Result, SectionParsingError> { let mut signatures: Vec = Vec::new(); loop { match *parser.read() { @@ -36,28 +37,22 @@ pub fn parse_function_signatures(parser: &mut Parser) ref returns, }) => { let mut sig = Signature::new(CallConv::Native); - sig.argument_types - .extend(params - .iter() - .map(|ty| { - let cret_arg: cretonne::ir::Type = match type_to_type(ty) { - Ok(ty) => ty, - Err(()) => panic!("only numeric types are supported in\ + sig.argument_types.extend(params.iter().map(|ty| { + let cret_arg: cretonne::ir::Type = match type_to_type(ty) { + Ok(ty) => ty, + Err(()) => panic!("only numeric types are supported in\ function signatures"), - }; - ArgumentType::new(cret_arg) - })); - sig.return_types - .extend(returns - .iter() - .map(|ty| { - let cret_arg: cretonne::ir::Type = match type_to_type(ty) { - Ok(ty) => ty, - Err(()) => panic!("only numeric types are supported in\ + }; + ArgumentType::new(cret_arg) + })); + sig.return_types.extend(returns.iter().map(|ty| { + let cret_arg: cretonne::ir::Type = match type_to_type(ty) { + Ok(ty) => ty, + Err(()) => panic!("only numeric types are supported in\ function signatures"), - }; - ArgumentType::new(cret_arg) - })); + }; + ArgumentType::new(cret_arg) + })); signatures.push(sig); } ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), @@ -78,30 +73,30 @@ pub fn parse_import_section(parser: &mut Parser) -> Result, SectionP ty: ImportSectionEntryType::Memory(MemoryType { limits: ref memlimits }), .. } => { imports.push(Import::Memory(Memory { - pages_count: memlimits.initial as usize, - maximum: memlimits.maximum.map(|x| x as usize), - })) + pages_count: memlimits.initial as usize, + maximum: memlimits.maximum.map(|x| x as usize), + })) } ParserState::ImportSectionEntry { ty: ImportSectionEntryType::Global(ref ty), .. } => { imports.push(Import::Global(Global { - ty: type_to_type(&ty.content_type).unwrap(), - mutability: ty.mutability != 0, - initializer: GlobalInit::Import(), - })); + ty: type_to_type(&ty.content_type).unwrap(), + mutability: ty.mutability != 0, + initializer: GlobalInit::Import(), + })); } ParserState::ImportSectionEntry { ty: ImportSectionEntryType::Table(ref tab), .. } => { imports.push(Import::Table(Table { - ty: match type_to_type(&tab.element_type) { - Ok(t) => TableElementType::Val(t), - Err(()) => TableElementType::Func(), - }, - size: tab.limits.initial as usize, - maximum: tab.limits.maximum.map(|x| x as usize), - })); + ty: match type_to_type(&tab.element_type) { + Ok(t) => TableElementType::Val(t), + Err(()) => TableElementType::Func(), + }, + size: tab.limits.initial as usize, + maximum: tab.limits.maximum.map(|x| x as usize), + })); } ParserState::EndSection => break, ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), @@ -111,8 +106,9 @@ pub fn parse_import_section(parser: &mut Parser) -> Result, SectionP } /// Retrieves the correspondances between functions and signatures from the function section -pub fn parse_function_section(parser: &mut Parser) - -> Result, SectionParsingError> { +pub fn parse_function_section( + parser: &mut Parser, +) -> Result, SectionParsingError> { let mut funcs = Vec::new(); loop { match *parser.read() { @@ -125,8 +121,9 @@ pub fn parse_function_section(parser: &mut Parser) } /// Retrieves the names of the functions from the export section -pub fn parse_export_section(parser: &mut Parser) - -> Result, SectionParsingError> { +pub fn parse_export_section( + parser: &mut Parser, +) -> Result, SectionParsingError> { let mut exports: HashMap = HashMap::new(); loop { match *parser.read() { @@ -137,8 +134,10 @@ pub fn parse_export_section(parser: &mut Parser) } => { match kind { &ExternalKind::Function => { - exports.insert(index as FunctionIndex, - String::from(from_utf8(field).unwrap())); + exports.insert( + index as FunctionIndex, + String::from(from_utf8(field).unwrap()), + ); } _ => (),//TODO: deal with other kind of exports } @@ -157,9 +156,9 @@ pub fn parse_memory_section(parser: &mut Parser) -> Result, SectionP match *parser.read() { ParserState::MemorySectionEntry(ref ty) => { memories.push(Memory { - pages_count: ty.limits.initial as usize, - maximum: ty.limits.maximum.map(|x| x as usize), - }) + pages_count: ty.limits.initial as usize, + maximum: ty.limits.maximum.map(|x| x as usize), + }) } ParserState::EndSection => break, ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), @@ -169,9 +168,10 @@ pub fn parse_memory_section(parser: &mut Parser) -> Result, SectionP } /// Retrieves the size and maximum fields of memories from the memory section -pub fn parse_global_section(parser: &mut Parser, - runtime: &mut WasmRuntime) - -> Result, SectionParsingError> { +pub fn parse_global_section( + parser: &mut Parser, + runtime: &mut WasmRuntime, +) -> Result, SectionParsingError> { let mut globals = Vec::new(); loop { let (content_type, mutability) = match *parser.read() { @@ -221,10 +221,11 @@ pub fn parse_global_section(parser: &mut Parser, Ok(globals) } -pub fn parse_data_section(parser: &mut Parser, - runtime: &mut WasmRuntime, - globals: &[Global]) - -> Result<(), SectionParsingError> { +pub fn parse_data_section( + parser: &mut Parser, + runtime: &mut WasmRuntime, + globals: &[Global], +) -> Result<(), SectionParsingError> { loop { let memory_index = match *parser.read() { ParserState::BeginDataSectionEntry(memory_index) => memory_index, @@ -238,8 +239,10 @@ pub fn parse_data_section(parser: &mut Parser, let offset = match *parser.read() { ParserState::InitExpressionOperator(Operator::I32Const { value }) => { if value < 0 { - return Err(SectionParsingError::WrongSectionContent(String::from("negative \ - offset value"))); + return Err(SectionParsingError::WrongSectionContent(String::from( + "negative \ + offset value", + ))); } else { value as usize } @@ -248,15 +251,19 @@ pub fn parse_data_section(parser: &mut Parser, match globals[global_index as usize].initializer { GlobalInit::I32Const(value) => { if value < 0 { - return Err(SectionParsingError::WrongSectionContent(String::from("\ - negative offset value"))); + return Err(SectionParsingError::WrongSectionContent(String::from( + "\ + negative offset value", + ))); } else { value as usize } } GlobalInit::Import() => { - return Err(SectionParsingError::WrongSectionContent(String::from("\ - imported globals not supported"))) + return Err(SectionParsingError::WrongSectionContent(String::from( + "\ + imported globals not supported", + ))) } // TODO: add runtime support _ => panic!("should not happen"), } @@ -288,20 +295,21 @@ pub fn parse_data_section(parser: &mut Parser, } /// Retrieves the tables from the table section -pub fn parse_table_section(parser: &mut Parser, - runtime: &mut WasmRuntime) - -> Result<(), SectionParsingError> { +pub fn parse_table_section( + parser: &mut Parser, + runtime: &mut WasmRuntime, +) -> Result<(), SectionParsingError> { loop { match *parser.read() { ParserState::TableSectionEntry(ref table) => { runtime.declare_table(Table { - ty: match type_to_type(&table.element_type) { - Ok(t) => TableElementType::Val(t), - Err(()) => TableElementType::Func(), - }, - size: table.limits.initial as usize, - maximum: table.limits.maximum.map(|x| x as usize), - }) + ty: match type_to_type(&table.element_type) { + Ok(t) => TableElementType::Val(t), + Err(()) => TableElementType::Func(), + }, + size: table.limits.initial as usize, + maximum: table.limits.maximum.map(|x| x as usize), + }) } ParserState::EndSection => break, ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), @@ -311,10 +319,11 @@ pub fn parse_table_section(parser: &mut Parser, } /// Retrieves the tables from the table section -pub fn parse_elements_section(parser: &mut Parser, - runtime: &mut WasmRuntime, - globals: &[Global]) - -> Result<(), SectionParsingError> { +pub fn parse_elements_section( + parser: &mut Parser, + runtime: &mut WasmRuntime, + globals: &[Global], +) -> Result<(), SectionParsingError> { loop { let table_index = match *parser.read() { ParserState::BeginElementSectionEntry(ref table_index) => *table_index as TableIndex, @@ -328,8 +337,10 @@ pub fn parse_elements_section(parser: &mut Parser, let offset = match *parser.read() { ParserState::InitExpressionOperator(Operator::I32Const { value }) => { if value < 0 { - return Err(SectionParsingError::WrongSectionContent(String::from("negative \ - offset value"))); + return Err(SectionParsingError::WrongSectionContent(String::from( + "negative \ + offset value", + ))); } else { value as usize } @@ -338,8 +349,10 @@ pub fn parse_elements_section(parser: &mut Parser, match globals[global_index as usize].initializer { GlobalInit::I32Const(value) => { if value < 0 { - return Err(SectionParsingError::WrongSectionContent(String::from("\ - negative offset value"))); + return Err(SectionParsingError::WrongSectionContent(String::from( + "\ + negative offset value", + ))); } else { value as usize } diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 88686d4221..645b236f5d 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -21,7 +21,7 @@ pub type RawByte = u8; pub type MemoryAddress = usize; /// WebAssembly import. -#[derive(Debug,Clone,Copy)] +#[derive(Debug, Clone, Copy)] pub enum Import { Function { sig_index: u32 }, Memory(Memory), @@ -30,7 +30,7 @@ pub enum Import { } /// WebAssembly global. -#[derive(Debug,Clone,Copy)] +#[derive(Debug, Clone, Copy)] pub struct Global { /// The type of the value stored in the global. pub ty: cretonne::ir::Type, @@ -41,7 +41,7 @@ pub struct Global { } /// Globals are initialized via the four `const` operators or by referring to another import. -#[derive(Debug,Clone,Copy)] +#[derive(Debug, Clone, Copy)] pub enum GlobalInit { /// An `i32.const`. I32Const(i32), @@ -58,7 +58,7 @@ pub enum GlobalInit { } /// WebAssembly table. -#[derive(Debug,Clone,Copy)] +#[derive(Debug, Clone, Copy)] pub struct Table { /// The type of data stored in elements of the table. pub ty: TableElementType, @@ -69,14 +69,14 @@ pub struct Table { } /// WebAssembly table element. Can be a function or a scalar type. -#[derive(Debug,Clone,Copy)] +#[derive(Debug, Clone, Copy)] pub enum TableElementType { Val(cretonne::ir::Type), Func(), } /// WebAssembly linear memory. -#[derive(Debug,Clone,Copy)] +#[derive(Debug, Clone, Copy)] pub struct Memory { /// The minimum number of pages in the memory. pub pages_count: usize, @@ -139,8 +139,9 @@ pub fn translate_type(ty: wasmparser::Type) -> Result, ( /// Inverts the key-value relation in the imports hashmap. Indeed, these hashmaps are built by /// feeding the function indexes in the module but are used by the runtime with the `FuncRef` as /// keys. -pub fn invert_hashmaps(imports: code_translator::FunctionImports) - -> module_translator::ImportMappings { +pub fn invert_hashmaps( + imports: code_translator::FunctionImports, +) -> module_translator::ImportMappings { let mut new_imports = module_translator::ImportMappings::new(); for (func_index, func_ref) in &imports.functions { new_imports.functions.insert(*func_ref, *func_index); diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/testsuite.rs index 3853ba1703..50bf08c344 100644 --- a/lib/wasm/tests/testsuite.rs +++ b/lib/wasm/tests/testsuite.rs @@ -84,17 +84,20 @@ fn handle_module(path: PathBuf) -> Result<(), String> { /// Pretty-print a verifier error. -pub fn pretty_verifier_error(func: &ir::Function, - isa: Option<&TargetIsa>, - err: verifier::Error) - -> String { +pub fn pretty_verifier_error( + func: &ir::Function, + isa: Option<&TargetIsa>, + err: verifier::Error, +) -> String { let msg = err.to_string(); let str1 = match err.location { AnyEntity::Inst(inst) => { - format!("{}\n{}: {}\n\n", - msg, - inst, - func.dfg.display_inst(inst, isa)) + format!( + "{}\n{}: {}\n\n", + msg, + inst, + func.dfg.display_inst(inst, isa) + ) } _ => String::from(format!("{}\n", msg)), }; From b411d01d6dcf791452c7fd56dee21a15eba42989 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 11:37:58 -0700 Subject: [PATCH 1015/3084] Use `[]` instead of `.get().unwrap()`. https://github.com/rust-lang-nursery/rust-clippy/wiki#get_unwrap --- lib/wasm/src/code_translator.rs | 2 +- lib/wasm/src/runtime/dummy.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 12f3829338..c500ba7c0c 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -567,7 +567,7 @@ fn translate_operator( acc.insert(depth as usize, branch_ebb); return acc; }; - let branch_ebb = acc.get(&(depth as usize)).unwrap().clone(); + let branch_ebb = acc[&(depth as usize)].clone(); builder.insert_jump_table_entry(jt, index, branch_ebb); acc }); diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index f52b9e9d43..0d6a7755e9 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -26,7 +26,7 @@ impl WasmRuntime for DummyRuntime { builder: &mut FunctionBuilder, global_index: GlobalIndex, ) -> Value { - let ref glob = self.globals.get(global_index as usize).unwrap(); + let ref glob = self.globals[global_index as usize]; match glob.ty { I32 => builder.ins().iconst(glob.ty, -1), I64 => builder.ins().iconst(glob.ty, -1), From adfdd773119492812a0d4f789d1281a039a5cd42 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 11:38:55 -0700 Subject: [PATCH 1016/3084] Avoid creating a reference to a reference. https://github.com/rust-lang-nursery/rust-clippy/wiki#needless_borrow --- lib/cretonne/src/regalloc/coalescing.rs | 2 +- lib/wasm/src/runtime/dummy.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 35957f90c1..509f5a45e7 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -468,7 +468,7 @@ impl<'a> Context<'a> { /// Leave `self.values` unchanged on failure. fn add_class(&mut self, value: Value) -> Result<(), (Value, Value)> { self.forest.try_merge( - &self.values, + self.values, self.virtregs.congruence_class(&value), &self.func.dfg, &self.func.layout, diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 0d6a7755e9..7e8dc02005 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -26,7 +26,7 @@ impl WasmRuntime for DummyRuntime { builder: &mut FunctionBuilder, global_index: GlobalIndex, ) -> Value { - let ref glob = self.globals[global_index as usize]; + let glob = self.globals[global_index as usize]; match glob.ty { I32 => builder.ins().iconst(glob.ty, -1), I64 => builder.ins().iconst(glob.ty, -1), From abbc6ddf240b0c868f18eaa43a310a3da858ced8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 11:40:18 -0700 Subject: [PATCH 1017/3084] Use is_empty() instead of comparing len() with 0. https://github.com/rust-lang-nursery/rust-clippy/wiki#len_zero --- lib/frontend/src/frontend.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 60b427b715..3c92e24bcc 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -530,7 +530,7 @@ where fn ebb_args_adjustement(&mut self, dest_ebb: Ebb, jump_args: &[Type]) { let ty_to_append: Option> = - if self.builder.ssa.predecessors(dest_ebb).len() == 0 || + if self.builder.ssa.predecessors(dest_ebb).is_empty() || self.builder.ebbs[dest_ebb].pristine { // This is the first jump instruction targeting this Ebb From acf4f1009bad36b2e5b3de5e11dc591edbf846aa Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 11:43:54 -0700 Subject: [PATCH 1018/3084] Eliminate redundant uses of format!. https://github.com/rust-lang-nursery/rust-clippy/wiki#useless_format --- lib/wasm/src/module_translator.rs | 2 +- lib/wasm/src/sections_translator.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 0aaa35e697..81b0baa654 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -254,7 +254,7 @@ pub fn translate_module( .collect() } ParserState::EndSection => break, - _ => return Err(String::from(format!("wrong content in code section"))), + _ => return Err(String::from("wrong content in code section")), }; let signature = signatures[functions[function_index as usize] as usize].clone(); match translate_function_body( diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 95fb673d17..af11570763 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -283,7 +283,7 @@ pub fn parse_data_section( }; match runtime.declare_data_initialization(memory_index as MemoryIndex, offset, data) { Ok(()) => (), - Err(s) => return Err(SectionParsingError::WrongSectionContent(format!("{}", s))), + Err(s) => return Err(SectionParsingError::WrongSectionContent(s)), }; } match *parser.read() { From 9a8f01b832ba972c162d7f089a9fcdbc57331e48 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 11:46:05 -0700 Subject: [PATCH 1019/3084] Avoid unnecessary '&' in matches. https://github.com/rust-lang-nursery/rust-clippy/wiki#match_ref_pats --- lib/cretonne/src/legalizer/globalvar.rs | 4 ++-- lib/cretonne/src/legalizer/heap.rs | 4 ++-- lib/cretonne/src/verifier/mod.rs | 2 +- lib/wasm/src/code_translator.rs | 22 +++++++++++----------- 4 files changed, 16 insertions(+), 16 deletions(-) diff --git a/lib/cretonne/src/legalizer/globalvar.rs b/lib/cretonne/src/legalizer/globalvar.rs index 308c8df9e5..42856edc9b 100644 --- a/lib/cretonne/src/legalizer/globalvar.rs +++ b/lib/cretonne/src/legalizer/globalvar.rs @@ -10,8 +10,8 @@ use ir::{self, InstBuilder}; /// Expand a `global_addr` instruction according to the definition of the global variable. pub fn expand_global_addr(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlowGraph) { // Unpack the instruction. - let gv = match &func.dfg[inst] { - &ir::InstructionData::UnaryGlobalVar { opcode, global_var } => { + let gv = match func.dfg[inst] { + ir::InstructionData::UnaryGlobalVar { opcode, global_var } => { assert_eq!(opcode, ir::Opcode::GlobalAddr); global_var } diff --git a/lib/cretonne/src/legalizer/heap.rs b/lib/cretonne/src/legalizer/heap.rs index ac68044a4a..0a14f5f8ee 100644 --- a/lib/cretonne/src/legalizer/heap.rs +++ b/lib/cretonne/src/legalizer/heap.rs @@ -11,8 +11,8 @@ use ir::condcodes::IntCC; /// Expand a `heap_addr` instruction according to the definition of the heap. pub fn expand_heap_addr(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlowGraph) { // Unpack the instruction. - let (heap, offset, size) = match &func.dfg[inst] { - &ir::InstructionData::HeapAddr { + let (heap, offset, size) = match func.dfg[inst] { + ir::InstructionData::HeapAddr { opcode, heap, arg, diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 47a419dafd..e6f9ea933e 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -169,7 +169,7 @@ impl<'a> Verifier<'a> { seen.insert(gv); let mut cur = gv; - while let &ir::GlobalVarData::Deref { base, .. } = &self.func.global_vars[cur] { + while let ir::GlobalVarData::Deref { base, .. } = self.func.global_vars[cur] { if seen.insert(base).is_some() { return err!(gv, "deref cycle: {}", DisplayList(seen.as_slice())); } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index c500ba7c0c..4729ddbaa8 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -420,8 +420,8 @@ fn translate_operator( // and push a new control frame with a new ebb for the code after the if/then/else // At the end of the then clause we jump to the destination let i = control_stack.len() - 1; - let (destination, return_values, branch_inst) = match &control_stack[i] { - &ControlStackFrame::If { + let (destination, return_values, branch_inst) = match control_stack[i] { + ControlStackFrame::If { destination, ref return_values, branch_inst, @@ -1276,15 +1276,15 @@ fn translate_unreachable_operator( } else { // Encountering an real else means that the code in the else // clause is reachable again - let (branch_inst, original_stack_size) = - match &control_stack[control_stack.len() - 1] { - &ControlStackFrame::If { - branch_inst, - original_stack_size, - .. - } => (branch_inst, original_stack_size), - _ => panic!("should not happen"), - }; + let (branch_inst, original_stack_size) = match control_stack[control_stack.len() - + 1] { + ControlStackFrame::If { + branch_inst, + original_stack_size, + .. + } => (branch_inst, original_stack_size), + _ => panic!("should not happen"), + }; // We change the target of the branch instruction let else_ebb = builder.create_ebb(); builder.change_jump_destination(branch_inst, else_ebb); From 105998944e3c633be0cc36eac2900d0abf03d086 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 11:46:45 -0700 Subject: [PATCH 1020/3084] Avoid redundant borrows. https://github.com/rust-lang-nursery/rust-clippy/wiki#needless_borrow --- lib/wasm/src/code_translator.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 4729ddbaa8..067834827b 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -250,10 +250,10 @@ pub fn translate_function_body( &mut stack, &mut control_stack, &mut state, - &sig, - &functions, - &signatures, - &exports, + sig, + functions, + signatures, + exports, &mut func_imports, ) } From b6641ff44387afb6863a6fb00530f37c52394bb8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 11:48:14 -0700 Subject: [PATCH 1021/3084] Avoid clone() on a `Copy` type. https://github.com/rust-lang-nursery/rust-clippy/wiki#clone_on_copy --- lib/frontend/src/frontend.rs | 2 +- lib/wasm/src/code_translator.rs | 2 +- lib/wasm/src/module_translator.rs | 2 +- lib/wasm/src/sections_translator.rs | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 3c92e24bcc..20c660a1f0 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -136,7 +136,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short match data.analyze_branch(&self.builder.func.dfg.value_lists) { BranchInfo::SingleDest(_, args) => { args.iter() - .map(|arg| self.builder.func.dfg.value_type(arg.clone())) + .map(|arg| self.builder.func.dfg.value_type(*arg)) .collect() } _ => panic!("should not happen"), diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 067834827b..f4547f5003 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -567,7 +567,7 @@ fn translate_operator( acc.insert(depth as usize, branch_ebb); return acc; }; - let branch_ebb = acc[&(depth as usize)].clone(); + let branch_ebb = acc[&(depth as usize)]; builder.insert_jump_table_entry(jt, index, branch_ebb); acc }); diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 81b0baa654..eeaf5f67e5 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -108,7 +108,7 @@ pub fn translate_module( runtime.declare_memory(mem); } Import::Global(glob) => { - runtime.declare_global(glob.clone()); + runtime.declare_global(glob); globals.push(glob); } Import::Table(tab) => { diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index af11570763..2605855e44 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -211,7 +211,7 @@ pub fn parse_global_section( mutability: mutability != 0, initializer: initializer, }; - runtime.declare_global(global.clone()); + runtime.declare_global(global); globals.push(global); match *parser.read() { ParserState::EndGlobalSectionEntry => (), From 574031e4d2779fc2804d92f85715205545434c17 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 11:49:33 -0700 Subject: [PATCH 1022/3084] Avoid unneeded passing by value. https://github.com/rust-lang-nursery/rust-clippy/wiki#needless_pass_by_value --- lib/wasm/src/code_translator.rs | 2 +- lib/wasm/src/module_translator.rs | 4 ++-- lib/wasm/src/translation_utils.rs | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index f4547f5003..a4ad9f7445 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -156,7 +156,7 @@ impl FunctionImports { pub fn translate_function_body( parser: &mut Parser, function_index: FunctionIndex, - sig: Signature, + sig: &Signature, locals: &[(usize, Type)], exports: &Option>, signatures: &[Signature], diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index eeaf5f67e5..ba99b5593b 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -260,7 +260,7 @@ pub fn translate_module( match translate_function_body( &mut parser, function_index, - signature, + &signature, &locals, &exports, &signatures, @@ -271,7 +271,7 @@ pub fn translate_module( Ok((il_func, imports)) => { il_functions.push(FunctionTranslation::Code { il: il_func, - imports: invert_hashmaps(imports), + imports: invert_hashmaps(&imports), }) } Err(s) => return Err(s), diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 645b236f5d..15099f475b 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -140,7 +140,7 @@ pub fn translate_type(ty: wasmparser::Type) -> Result, ( /// feeding the function indexes in the module but are used by the runtime with the `FuncRef` as /// keys. pub fn invert_hashmaps( - imports: code_translator::FunctionImports, + imports: &code_translator::FunctionImports, ) -> module_translator::ImportMappings { let mut new_imports = module_translator::ImportMappings::new(); for (func_index, func_ref) in &imports.functions { From 5a8d1a9fda56aeabc60e6ee45f5cb1700a8a6aa4 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 11:57:03 -0700 Subject: [PATCH 1023/3084] Use `if let` instead of `match`. https://github.com/rust-lang-nursery/rust-clippy/wiki#single_match --- lib/frontend/src/frontend.rs | 10 ++--- lib/wasm/src/code_translator.rs | 74 +++++++++++++-------------------- 2 files changed, 31 insertions(+), 53 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 20c660a1f0..008610f62e 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -147,15 +147,14 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short } None => { // branch_destination() doesn't detect jump_tables - match data { // If jump table we declare all entries successor // TODO: not collect with vector? - InstructionData::BranchTable { table, .. } => { + if let InstructionData::BranchTable { table, .. } = data { // Unlike all other jumps/branches, jump tables are // capable of having the same successor appear // multiple times. Use a HashSet to deduplicate. - let mut unique = HashSet::new(); - for dest_ebb in self.builder + let mut unique = HashSet::new(); + for dest_ebb in self.builder .func .jump_tables .get(table) @@ -166,9 +165,6 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short .collect::>() { self.builder.declare_successor(dest_ebb, inst) } - } -// If not we do nothing - _ => {} } } } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index a4ad9f7445..8fb90aa179 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -169,13 +169,9 @@ pub fn translate_function_body( let mut func = Function::new(); let args_num: usize = sig.argument_types.len(); func.signature = sig.clone(); - match exports { - &None => (), - &Some(ref exports) => { - match exports.get(&function_index) { - None => (), - Some(name) => func.name = FunctionName::new(name.clone()), - } + if let Some(ref exports) = *exports { + if let Some(name) = exports.get(&function_index) { + func.name = FunctionName::new(name.clone()); } } let mut func_imports = FunctionImports::new(); @@ -359,11 +355,8 @@ fn translate_operator( ***********************************************************************************/ Operator::Block { ty } => { let next = builder.create_ebb(); - match type_to_type(&ty) { - Ok(ty_cre) => { - builder.append_ebb_arg(next, ty_cre); - } - Err(_) => {} + if let Ok(ty_cre) = type_to_type(&ty) { + builder.append_ebb_arg(next, ty_cre); } control_stack.push(ControlStackFrame::Block { destination: next, @@ -375,11 +368,8 @@ fn translate_operator( Operator::Loop { ty } => { let loop_body = builder.create_ebb(); let next = builder.create_ebb(); - match type_to_type(&ty) { - Ok(ty_cre) => { - builder.append_ebb_arg(next, ty_cre); - } - Err(_) => {} + if let Ok(ty_cre) = type_to_type(&ty) { + builder.append_ebb_arg(next, ty_cre); } builder.ins().jump(loop_body, &[]); control_stack.push(ControlStackFrame::Loop { @@ -401,11 +391,8 @@ fn translate_operator( // and we add nothing; // - either the If have an Else clause, in that case the destination of this jump // instruction will be changed later when we translate the Else operator. - match type_to_type(&ty) { - Ok(ty_cre) => { - builder.append_ebb_arg(if_not, ty_cre); - } - Err(_) => {} + if let Ok(ty_cre) = type_to_type(&ty) { + builder.append_ebb_arg(if_not, ty_cre); } control_stack.push(ControlStackFrame::If { destination: if_not, @@ -1320,31 +1307,27 @@ fn find_function_import( exports: &Option>, signatures: &[Signature], ) -> FuncRef { - match func_imports.functions.get(&index) { - Some(local_index) => return *local_index, - None => {} + if let Some(local_index) = func_imports.functions.get(&index) { + return *local_index; } // We have to import the function let sig_index = functions[index]; - match func_imports.signatures.get(&(sig_index as usize)) { - Some(local_sig_index) => { - let local_func_index = builder.import_function(ExtFuncData { - name: match exports { - &None => FunctionName::new(""), - &Some(ref exports) => { - match exports.get(&index) { - None => FunctionName::new(""), - Some(name) => FunctionName::new(name.clone()), - } + if let Some(local_sig_index) = func_imports.signatures.get(&(sig_index as usize)) { + let local_func_index = builder.import_function(ExtFuncData { + name: match exports { + &None => FunctionName::new(""), + &Some(ref exports) => { + match exports.get(&index) { + None => FunctionName::new(""), + Some(name) => FunctionName::new(name.clone()), } - }, - signature: *local_sig_index, - }); - func_imports.functions.insert(index, local_func_index); - return local_func_index; - } - None => {} - }; + } + }, + signature: *local_sig_index, + }); + func_imports.functions.insert(index, local_func_index); + return local_func_index; + } // We have to import the signature let sig_local_index = builder.import_signature(signatures[sig_index as usize].clone()); func_imports.signatures.insert( @@ -1373,9 +1356,8 @@ fn find_signature_import( func_imports: &mut FunctionImports, signatures: &[Signature], ) -> SigRef { - match func_imports.signatures.get(&(sig_index as usize)) { - Some(local_sig_index) => return *local_sig_index, - None => {} + if let Some(local_sig_index) = func_imports.signatures.get(&(sig_index as usize)) { + return *local_sig_index; } let sig_local_index = builder.import_signature(signatures[sig_index as usize].clone()); func_imports.signatures.insert( From 9726bb7367e470a7fbb04459f080a5ca07af1e67 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 11:59:26 -0700 Subject: [PATCH 1024/3084] Avoid matching with reference patterns. https://github.com/rust-lang-nursery/rust-clippy/wiki#match_ref_pats --- cranelift/src/wasm.rs | 12 ++--- lib/cretonne/src/ir/globalvar.rs | 6 +-- lib/frontend/src/ssa.rs | 12 ++--- lib/wasm/src/code_translator.rs | 68 ++++++++++++++--------------- lib/wasm/src/sections_translator.rs | 4 +- lib/wasm/tests/testsuite.rs | 6 +-- 6 files changed, 54 insertions(+), 54 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 7dc85b8aa4..3a972e9716 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -149,9 +149,9 @@ fn handle_module( vprint!(flag_verbose, "Checking... "); terminal.reset().unwrap(); for func in translation.functions.iter() { - let il = match func { - &FunctionTranslation::Import() => continue, - &FunctionTranslation::Code { ref il, .. } => il.clone(), + let il = match *func { + FunctionTranslation::Import() => continue, + FunctionTranslation::Code { ref il, .. } => il.clone(), }; match verifier::verify_function(&il, None) { Ok(()) => (), @@ -167,9 +167,9 @@ fn handle_module( vprint!(flag_verbose, "Optimizing... "); terminal.reset().unwrap(); for func in translation.functions.iter() { - let mut il = match func { - &FunctionTranslation::Import() => continue, - &FunctionTranslation::Code { ref il, .. } => il.clone(), + let mut il = match *func { + FunctionTranslation::Import() => continue, + FunctionTranslation::Code { ref il, .. } => il.clone(), }; let mut loop_analysis = LoopAnalysis::new(); let mut cfg = ControlFlowGraph::new(); diff --git a/lib/cretonne/src/ir/globalvar.rs b/lib/cretonne/src/ir/globalvar.rs index ad209b0eca..110cf687ce 100644 --- a/lib/cretonne/src/ir/globalvar.rs +++ b/lib/cretonne/src/ir/globalvar.rs @@ -29,9 +29,9 @@ pub enum GlobalVarData { impl fmt::Display for GlobalVarData { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match self { - &GlobalVarData::VmCtx { offset } => write!(f, "vmctx{}", offset), - &GlobalVarData::Deref { base, offset } => write!(f, "deref({}){}", base, offset), + match *self { + GlobalVarData::VmCtx { offset } => write!(f, "vmctx{}", offset), + GlobalVarData::Deref { base, offset } => write!(f, "deref({}){}", base, offset), } } } diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index c00d77e685..509da5f592 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -78,17 +78,17 @@ enum BlockData { impl BlockData { fn add_predecessor(&mut self, pred: Block, inst: Inst) { - match self { - &mut BlockData::EbbBody { .. } => panic!("you can't add a predecessor to a body block"), - &mut BlockData::EbbHeader(ref mut data) => { + match *self { + BlockData::EbbBody { .. } => panic!("you can't add a predecessor to a body block"), + BlockData::EbbHeader(ref mut data) => { data.predecessors.push((pred, inst)); } } } fn remove_predecessor(&mut self, inst: Inst) -> Block { - match self { - &mut BlockData::EbbBody { .. } => panic!("should not happen"), - &mut BlockData::EbbHeader(ref mut data) => { + match *self { + BlockData::EbbBody { .. } => panic!("should not happen"), + BlockData::EbbHeader(ref mut data) => { // This a linear complexity operation but the number of predecessors is low // in all non-pathological cases let pred: usize = diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 8fb90aa179..d6fd219bd5 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -72,54 +72,54 @@ enum ControlStackFrame { /// Helper methods for the control stack objects. impl ControlStackFrame { fn return_values(&self) -> &[Type] { - match self { - &ControlStackFrame::If { ref return_values, .. } | - &ControlStackFrame::Block { ref return_values, .. } | - &ControlStackFrame::Loop { ref return_values, .. } => return_values.as_slice(), + match *self { + ControlStackFrame::If { ref return_values, .. } | + ControlStackFrame::Block { ref return_values, .. } | + ControlStackFrame::Loop { ref return_values, .. } => return_values.as_slice(), } } fn following_code(&self) -> Ebb { - match self { - &ControlStackFrame::If { destination, .. } | - &ControlStackFrame::Block { destination, .. } | - &ControlStackFrame::Loop { destination, .. } => destination, + match *self { + ControlStackFrame::If { destination, .. } | + ControlStackFrame::Block { destination, .. } | + ControlStackFrame::Loop { destination, .. } => destination, } } fn br_destination(&self) -> Ebb { - match self { - &ControlStackFrame::If { destination, .. } | - &ControlStackFrame::Block { destination, .. } => destination, - &ControlStackFrame::Loop { header, .. } => header, + match *self { + ControlStackFrame::If { destination, .. } | + ControlStackFrame::Block { destination, .. } => destination, + ControlStackFrame::Loop { header, .. } => header, } } fn original_stack_size(&self) -> usize { - match self { - &ControlStackFrame::If { original_stack_size, .. } | - &ControlStackFrame::Block { original_stack_size, .. } | - &ControlStackFrame::Loop { original_stack_size, .. } => original_stack_size, + match *self { + ControlStackFrame::If { original_stack_size, .. } | + ControlStackFrame::Block { original_stack_size, .. } | + ControlStackFrame::Loop { original_stack_size, .. } => original_stack_size, } } fn is_loop(&self) -> bool { - match self { - &ControlStackFrame::If { .. } | - &ControlStackFrame::Block { .. } => false, - &ControlStackFrame::Loop { .. } => true, + match *self { + ControlStackFrame::If { .. } | + ControlStackFrame::Block { .. } => false, + ControlStackFrame::Loop { .. } => true, } } fn is_reachable(&self) -> bool { - match self { - &ControlStackFrame::If { reachable, .. } | - &ControlStackFrame::Block { reachable, .. } | - &ControlStackFrame::Loop { reachable, .. } => reachable, + match *self { + ControlStackFrame::If { reachable, .. } | + ControlStackFrame::Block { reachable, .. } | + ControlStackFrame::Loop { reachable, .. } => reachable, } } fn set_reachable(&mut self) { - match self { - &mut ControlStackFrame::If { ref mut reachable, .. } | - &mut ControlStackFrame::Block { ref mut reachable, .. } | - &mut ControlStackFrame::Loop { ref mut reachable, .. } => *reachable = true, + match *self { + ControlStackFrame::If { ref mut reachable, .. } | + ControlStackFrame::Block { ref mut reachable, .. } | + ControlStackFrame::Loop { ref mut reachable, .. } => *reachable = true, } } } @@ -1314,9 +1314,9 @@ fn find_function_import( let sig_index = functions[index]; if let Some(local_sig_index) = func_imports.signatures.get(&(sig_index as usize)) { let local_func_index = builder.import_function(ExtFuncData { - name: match exports { - &None => FunctionName::new(""), - &Some(ref exports) => { + name: match *exports { + None => FunctionName::new(""), + Some(ref exports) => { match exports.get(&index) { None => FunctionName::new(""), Some(name) => FunctionName::new(name.clone()), @@ -1335,9 +1335,9 @@ fn find_function_import( sig_local_index, ); let local_func_index = builder.import_function(ExtFuncData { - name: match exports { - &None => FunctionName::new(""), - &Some(ref exports) => { + name: match *exports { + None => FunctionName::new(""), + Some(ref exports) => { match exports.get(&index) { None => FunctionName::new(""), Some(name) => FunctionName::new(name.clone()), diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 2605855e44..34d9c90604 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -132,8 +132,8 @@ pub fn parse_export_section( ref kind, index, } => { - match kind { - &ExternalKind::Function => { + match *kind { + ExternalKind::Function => { exports.insert( index as FunctionIndex, String::from(from_utf8(field).unwrap()), diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/testsuite.rs index 50bf08c344..d4c51c6a90 100644 --- a/lib/wasm/tests/testsuite.rs +++ b/lib/wasm/tests/testsuite.rs @@ -70,9 +70,9 @@ fn handle_module(path: PathBuf) -> Result<(), String> { } }; for func in translation.functions.iter() { - let il = match func { - &FunctionTranslation::Import() => continue, - &FunctionTranslation::Code { ref il, .. } => il.clone(), + let il = match *func { + FunctionTranslation::Import() => continue, + FunctionTranslation::Code { ref il, .. } => il.clone(), }; match verifier::verify_function(&il, None) { Ok(()) => (), From da2c2151b1bd5d8cd84ca5bf0aa9172a2e8a64a7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 12:07:54 -0700 Subject: [PATCH 1025/3084] Fix trivial_numeric_casts errors. --- lib/wasm/src/code_translator.rs | 24 +++++++++--------------- lib/wasm/src/module_translator.rs | 2 +- lib/wasm/src/runtime/dummy.rs | 2 +- 3 files changed, 11 insertions(+), 17 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index d6fd219bd5..a98c9ad4f5 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -186,7 +186,7 @@ pub fn translate_function_body( for (i, arg_type) in sig.argument_types.iter().enumerate() { // First we declare the function arguments' as non-SSA vars because they will be // accessed by get_local - let arg_value = builder.arg_value(i as usize); + let arg_value = builder.arg_value(i); builder.declare_var(Local(i as u32), arg_type.value_type); builder.def_var(Local(i as u32), arg_value); } @@ -566,7 +566,7 @@ fn translate_operator( for (depth, dest_ebb) in dest_ebbs { builder.switch_to_block(dest_ebb, &[]); builder.seal_block(dest_ebb); - let i = control_stack.len() - 1 - (depth as usize); + let i = control_stack.len() - 1 - depth; let frame = &mut control_stack[i]; let real_dest_ebb = frame.br_destination(); builder.ins().jump(real_dest_ebb, jump_args.as_slice()); @@ -1294,7 +1294,7 @@ fn args_count( functions: &[SignatureIndex], signatures: &[Signature], ) -> usize { - signatures[functions[index] as usize].argument_types.len() + signatures[functions[index]].argument_types.len() } // Given a index in the function index space, search for it in the function imports and if it is @@ -1312,7 +1312,7 @@ fn find_function_import( } // We have to import the function let sig_index = functions[index]; - if let Some(local_sig_index) = func_imports.signatures.get(&(sig_index as usize)) { + if let Some(local_sig_index) = func_imports.signatures.get(&sig_index) { let local_func_index = builder.import_function(ExtFuncData { name: match *exports { None => FunctionName::new(""), @@ -1329,11 +1329,8 @@ fn find_function_import( return local_func_index; } // We have to import the signature - let sig_local_index = builder.import_signature(signatures[sig_index as usize].clone()); - func_imports.signatures.insert( - sig_index as usize, - sig_local_index, - ); + let sig_local_index = builder.import_signature(signatures[sig_index].clone()); + func_imports.signatures.insert(sig_index, sig_local_index); let local_func_index = builder.import_function(ExtFuncData { name: match *exports { None => FunctionName::new(""), @@ -1356,13 +1353,10 @@ fn find_signature_import( func_imports: &mut FunctionImports, signatures: &[Signature], ) -> SigRef { - if let Some(local_sig_index) = func_imports.signatures.get(&(sig_index as usize)) { + if let Some(local_sig_index) = func_imports.signatures.get(&sig_index) { return *local_sig_index; } - let sig_local_index = builder.import_signature(signatures[sig_index as usize].clone()); - func_imports.signatures.insert( - sig_index as usize, - sig_local_index, - ); + let sig_local_index = builder.import_signature(signatures[sig_index].clone()); + func_imports.signatures.insert(sig_index, sig_local_index); sig_local_index } diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index ba99b5593b..22bf59d17e 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -256,7 +256,7 @@ pub fn translate_module( ParserState::EndSection => break, _ => return Err(String::from("wrong content in code section")), }; - let signature = signatures[functions[function_index as usize] as usize].clone(); + let signature = signatures[functions[function_index]].clone(); match translate_function_body( &mut parser, function_index, diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 7e8dc02005..c6e1741dc2 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -26,7 +26,7 @@ impl WasmRuntime for DummyRuntime { builder: &mut FunctionBuilder, global_index: GlobalIndex, ) -> Value { - let glob = self.globals[global_index as usize]; + let glob = self.globals[global_index]; match glob.ty { I32 => builder.ins().iconst(glob.ty, -1), I64 => builder.ins().iconst(glob.ty, -1), From 55ae51acfedfab17eda23f169b3e29d8b3f0b6bf Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 12:09:33 -0700 Subject: [PATCH 1026/3084] Add an underscore between a literal value and its suffix. https://github.com/rust-lang-nursery/rust-clippy/wiki#unseparated_literal_suffix --- lib/reader/src/parser.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 39bac8af78..f8e90de1aa 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1312,7 +1312,7 @@ impl<'a> Parser<'a> { let mut data = JumpTableData::new(); // jump-table-decl ::= JumpTable(jt) "=" "jump_table" * jt-entry {"," jt-entry} - for idx in 0usize.. { + for idx in 0_usize.. { if let Some(dest) = self.parse_jump_table_entry()? { data.set_entry(idx, dest); } From 99b361567abe9c49f9bb3260f5252229d0ca0832 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 12:09:42 -0700 Subject: [PATCH 1027/3084] Remove unneeded `mut` keywords. --- lib/cretonne/src/entity/list.rs | 2 +- lib/cretonne/src/regalloc/liveness.rs | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/src/entity/list.rs b/lib/cretonne/src/entity/list.rs index 3f5eace692..03c66a8ec0 100644 --- a/lib/cretonne/src/entity/list.rs +++ b/lib/cretonne/src/entity/list.rs @@ -467,7 +467,7 @@ impl EntityList { /// Since the memory comes from the pool, this will be either zero entity references or /// whatever where in a previously deallocated list. pub fn grow_at(&mut self, index: usize, count: usize, pool: &mut ListPool) { - let mut data = self.grow(count, pool); + let data = self.grow(count, pool); // Copy elements after `index` up. for i in (index + count..data.len()).rev() { diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index c47a703f3c..72b3814770 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -328,7 +328,7 @@ impl Liveness { where PP: Into, { - let mut lr = self.ranges.get_mut(value).expect("Value has no live range"); + let lr = self.ranges.get_mut(value).expect("Value has no live range"); lr.move_def_locally(def.into()); } @@ -345,7 +345,7 @@ impl Liveness { layout: &Layout, ) -> &mut Affinity { debug_assert_eq!(Some(ebb), layout.inst_ebb(user)); - let mut lr = self.ranges.get_mut(value).expect("Value has no live range"); + let lr = self.ranges.get_mut(value).expect("Value has no live range"); let livein = lr.extend_in_ebb(ebb, user, layout); assert!(!livein, "{} should already be live in {}", value, ebb); &mut lr.affinity @@ -353,7 +353,7 @@ impl Liveness { /// Change the affinity of `value` to `Stack` and return the previous affinity. pub fn spill(&mut self, value: Value) -> Affinity { - let mut lr = self.ranges.get_mut(value).expect("Value has no live range"); + let lr = self.ranges.get_mut(value).expect("Value has no live range"); mem::replace(&mut lr.affinity, Affinity::Stack) } From a7d629c3689e6ea978fff7bf767666b528964bd2 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 12:12:04 -0700 Subject: [PATCH 1028/3084] Use the `Self` keyword where applicable. https://github.com/rust-lang-nursery/rust-clippy/wiki#use_self --- lib/frontend/src/frontend.rs | 4 ++-- lib/frontend/src/ssa.rs | 6 +++--- lib/reader/src/sourcemap.rs | 4 ++-- lib/wasm/src/code_translator.rs | 4 ++-- lib/wasm/src/module_translator.rs | 4 ++-- lib/wasm/src/runtime/dummy.rs | 4 ++-- lib/wasm/src/translation_utils.rs | 2 +- 7 files changed, 14 insertions(+), 14 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 008610f62e..0210e43a56 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -51,8 +51,8 @@ where { /// Creates a ILBuilder structure. The structure is automatically cleared each time it is /// passed to a [`FunctionBuilder`](struct.FunctionBuilder.html) for creation. - pub fn new() -> ILBuilder { - ILBuilder { + pub fn new() -> Self { + Self { ssa: SSABuilder::new(), ebbs: EntityMap::new(), types: EntityMap::new(), diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 509da5f592..3fefb4e555 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -128,7 +128,7 @@ impl EntityRef for Block { } impl ReservedValue for Block { - fn reserved_value() -> Block { + fn reserved_value() -> Self { Block(u32::MAX) } } @@ -138,8 +138,8 @@ where Variable: EntityRef + Default, { /// Allocate a new blank SSA builder struct. Use the API function to interact with the struct. - pub fn new() -> SSABuilder { - SSABuilder { + pub fn new() -> Self { + Self { variables: EntityMap::new(), blocks: PrimaryMap::new(), ebb_headers: EntityMap::new(), diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 2135e5e037..05d7e3ca3e 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -185,8 +185,8 @@ pub trait MutableSourceMap { } impl MutableSourceMap for SourceMap { - fn new() -> SourceMap { - SourceMap { + fn new() -> Self { + Self { values: HashMap::new(), ebbs: HashMap::new(), stack_slots: HashMap::new(), diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index a98c9ad4f5..91dfb00281 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -144,8 +144,8 @@ pub struct FunctionImports { } impl FunctionImports { - fn new() -> FunctionImports { - FunctionImports { + fn new() -> Self { + Self { functions: HashMap::new(), signatures: HashMap::new(), } diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 22bf59d17e..a5949fef30 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -46,8 +46,8 @@ pub struct ImportMappings { impl ImportMappings { /// Create a new empty `ImportMappings`. - pub fn new() -> ImportMappings { - ImportMappings { + pub fn new() -> Self { + Self { functions: HashMap::new(), signatures: HashMap::new(), } diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index c6e1741dc2..05d7cbfaa3 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -15,8 +15,8 @@ pub struct DummyRuntime { impl DummyRuntime { /// Allocates the runtime data structures. - pub fn new() -> DummyRuntime { - DummyRuntime { globals: Vec::new() } + pub fn new() -> Self { + Self { globals: Vec::new() } } } diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 15099f475b..4a7f375bb8 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -98,7 +98,7 @@ impl cretonne::entity::EntityRef for Local { } } impl Default for Local { - fn default() -> Local { + fn default() -> Self { Local(u32::MAX) } } From 3c5755cfbd176400bc26cd84fd6755a7f9bd0868 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 12:15:33 -0700 Subject: [PATCH 1029/3084] Avoid unneeded return keywords. --- lib/cretonne/meta/gen_encoding.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 0516b5c4c8..a9bef26dba 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -88,7 +88,7 @@ def emit_instp(instp, fmt, has_dfg=False): # Deal with pure type check predicates which apply to any instruction. if iform == instruction_context: fmt.line('let args = inst.arguments(&dfg.value_lists);') - fmt.format('return {};', instp.rust_predicate(0)) + fmt.line(instp.rust_predicate(0)) return assert isinstance(iform, InstructionFormat) From ebbb836bb95a78c5a532574d988cb2f26f811614 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 12:16:44 -0700 Subject: [PATCH 1030/3084] Use debug_assert_eq rather than debug_assert with an ==. --- lib/frontend/src/frontend.rs | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 0210e43a56..a7d98004d1 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -319,8 +319,9 @@ where /// Register a new definition of a user variable. Panics if the type of the value is not the /// same as the type registered for the variable. pub fn def_var(&mut self, var: Variable, val: Value) { - debug_assert!( - self.func.dfg.value_type(val) == self.builder.types[var], + debug_assert_eq!( + self.func.dfg.value_type(val), + self.builder.types[var], "the type of the value is not the type registered for the variable" ); self.builder.ssa.def_var( @@ -557,8 +558,9 @@ where // The Ebb already has predecessors // We check that the arguments supplied match those supplied // previously. - debug_assert!( - jump_args.len() == self.builder.ebbs[dest_ebb].user_arg_count, + debug_assert_eq!( + jump_args.len(), + self.builder.ebbs[dest_ebb].user_arg_count, "the jump instruction doesn't have the same \ number of arguments as its destination Ebb \ ({} vs {}).", From bc528917fd2ec690268eec3029d061cb1142529b Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 12:18:03 -0700 Subject: [PATCH 1031/3084] Avoid redundant '@ _' in match patterns. https://github.com/rust-lang-nursery/rust-clippy/wiki#redundant_pattern --- lib/wasm/src/module_translator.rs | 2 +- lib/wasm/src/sections_translator.rs | 48 ++++++++++++++--------------- 2 files changed, 24 insertions(+), 26 deletions(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index a5949fef30..6a2e5f3963 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -68,7 +68,7 @@ pub fn translate_module( ParserState::Error(BinaryReaderError { message, offset }) => { return Err(format!("at offset {}: {}", offset, message)); } - ref s @ _ => panic!("modules should begin properly: {:?}", s), + ref s => panic!("modules should begin properly: {:?}", s), } let mut signatures = None; let mut functions: Option> = None; diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 34d9c90604..3abb6f232a 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -55,7 +55,7 @@ pub fn parse_function_signatures( })); signatures.push(sig); } - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), } } Ok(signatures) @@ -99,7 +99,7 @@ pub fn parse_import_section(parser: &mut Parser) -> Result, SectionP })); } ParserState::EndSection => break, - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; } Ok(imports) @@ -114,7 +114,7 @@ pub fn parse_function_section( match *parser.read() { ParserState::FunctionSectionEntry(sigindex) => funcs.push(sigindex as SignatureIndex), ParserState::EndSection => break, - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; } Ok(funcs) @@ -143,7 +143,7 @@ pub fn parse_export_section( } } ParserState::EndSection => break, - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; } Ok(exports) @@ -161,7 +161,7 @@ pub fn parse_memory_section(parser: &mut Parser) -> Result, SectionP }) } ParserState::EndSection => break, - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; } Ok(memories) @@ -177,11 +177,11 @@ pub fn parse_global_section( let (content_type, mutability) = match *parser.read() { ParserState::BeginGlobalSectionEntry(ref ty) => (ty.content_type, ty.mutability), ParserState::EndSection => break, - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; match *parser.read() { ParserState::BeginInitExpressionBody => (), - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), } let initializer = match *parser.read() { ParserState::InitExpressionOperator(Operator::I32Const { value }) => { @@ -199,12 +199,12 @@ pub fn parse_global_section( ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => { GlobalInit::GlobalRef(global_index as GlobalIndex) } - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; match *parser.read() { ParserState::EndInitExpressionBody => (), - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), } let global = Global { ty: type_to_type(&content_type).unwrap(), @@ -215,7 +215,7 @@ pub fn parse_global_section( globals.push(global); match *parser.read() { ParserState::EndGlobalSectionEntry => (), - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), } } Ok(globals) @@ -230,11 +230,11 @@ pub fn parse_data_section( let memory_index = match *parser.read() { ParserState::BeginDataSectionEntry(memory_index) => memory_index, ParserState::EndSection => break, - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; match *parser.read() { ParserState::BeginInitExpressionBody => (), - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; let offset = match *parser.read() { ParserState::InitExpressionOperator(Operator::I32Const { value }) => { @@ -268,18 +268,16 @@ pub fn parse_data_section( _ => panic!("should not happen"), } } - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; match *parser.read() { ParserState::EndInitExpressionBody => (), - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; { let data = match *parser.read() { ParserState::DataSectionEntryBody(data) => data, - ref s @ _ => { - return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))) - } + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; match runtime.declare_data_initialization(memory_index as MemoryIndex, offset, data) { Ok(()) => (), @@ -288,7 +286,7 @@ pub fn parse_data_section( } match *parser.read() { ParserState::EndDataSectionEntry => (), - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; } Ok(()) @@ -312,7 +310,7 @@ pub fn parse_table_section( }) } ParserState::EndSection => break, - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; } Ok(()) @@ -328,11 +326,11 @@ pub fn parse_elements_section( let table_index = match *parser.read() { ParserState::BeginElementSectionEntry(ref table_index) => *table_index as TableIndex, ParserState::EndSection => break, - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; match *parser.read() { ParserState::BeginInitExpressionBody => (), - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; let offset = match *parser.read() { ParserState::InitExpressionOperator(Operator::I32Const { value }) => { @@ -361,11 +359,11 @@ pub fn parse_elements_section( _ => panic!("should not happen"), } } - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; match *parser.read() { ParserState::EndInitExpressionBody => (), - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; match *parser.read() { ParserState::ElementSectionEntryBody(ref elements) => { @@ -373,11 +371,11 @@ pub fn parse_elements_section( elements.iter().map(|&x| x as FunctionIndex).collect(); runtime.declare_table_elements(table_index, offset, elems.as_slice()) } - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; match *parser.read() { ParserState::EndElementSectionEntry => (), - ref s @ _ => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; } Ok(()) From dc79d155ff9de491ecf637abb6dae4eac08bd7b9 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 12:18:15 -0700 Subject: [PATCH 1032/3084] Comment wording cleanups. --- lib/cretonne/meta/base/settings.py | 2 +- lib/cretonne/meta/gen_instr.py | 4 ++-- lib/cretonne/src/ir/layout.rs | 4 ++-- lib/filecheck/src/checker.rs | 4 ++-- lib/frontend/src/frontend.rs | 2 +- lib/frontend/src/ssa.rs | 4 ++-- lib/reader/src/parser.rs | 4 ++-- lib/wasm/src/code_translator.rs | 6 +++--- 8 files changed, 15 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/meta/base/settings.py b/lib/cretonne/meta/base/settings.py index 186c808524..46ff16d851 100644 --- a/lib/cretonne/meta/base/settings.py +++ b/lib/cretonne/meta/base/settings.py @@ -1,7 +1,7 @@ """ Cretonne shared settings. -This module defines settings are are relevant for all code generators. +This module defines settings relevant for all code generators. """ from __future__ import absolute_import from cdsl.settings import SettingGroup, BoolSetting, EnumSetting diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 104f779e2d..3b43ad368b 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -451,8 +451,8 @@ def gen_type_constraints(fmt, instrs): def gen_format_constructor(iform, fmt): # type: (InstructionFormat, srcgen.Formatter) -> None """ - Emit a method for creating and inserting inserting an `iform` instruction, - where `iform` is an instruction format. + Emit a method for creating and inserting an `iform` instruction, where + `iform` is an instruction format. All instruction formats take an `opcode` argument and a `ctrl_typevar` argument for deducing the result types. diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index c1cc62b110..7f3247c080 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -1017,14 +1017,14 @@ pub trait CursorBase { /// - If the cursor points to the top of an EBB, the new EBB is inserted above the current one. /// - If the cursor is not pointing at anything, the new EBB is placed last in the layout. /// - /// This means that is is always valid to call this method, and it always leaves the cursor in + /// This means that it is always valid to call this method, and it always leaves the cursor in /// a state that will insert instructions into the new EBB. fn insert_ebb(&mut self, new_ebb: Ebb) { use self::CursorPosition::*; match self.position() { At(inst) => { self.layout_mut().split_ebb(new_ebb, inst); - // All other cases move to `After(ebb)`, but in this case we we'll stay `At(inst)`. + // All other cases move to `After(ebb)`, but in this case we'll stay `At(inst)`. return; } Nowhere => self.layout_mut().append_ebb(new_ebb), diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs index 19d8ae8092..9c88df8eb4 100644 --- a/lib/filecheck/src/checker.rs +++ b/lib/filecheck/src/checker.rs @@ -103,7 +103,7 @@ impl CheckerBuilder { /// Add a potential directive line. /// - /// Returns true if this is a a directive with one of the known prefixes. + /// Returns true if this is a directive with one of the known prefixes. /// Returns false if no known directive was found. /// Returns an error if there is a problem with the directive. pub fn directive(&mut self, l: &str) -> Result { @@ -206,7 +206,7 @@ impl Checker { state.recorder.directive(dct_idx); if let Some((match_begin, match_end)) = state.match_positive(pat, range)? { if let Directive::Unordered(_) = *dct { - // This was an unordered unordered match. + // This was an unordered match. // Keep track of the largest matched position, but leave `last_ordered` alone. state.max_match = max(state.max_match, match_end); } else { diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index a7d98004d1..c896e13f8e 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -393,7 +393,7 @@ where self.func.dfg.signatures.get(sigref) } - /// Creates a argument for a specific `Ebb` by appending it to the list of already existing + /// Creates an argument for a specific `Ebb` by appending it to the list of already existing /// arguments. /// /// **Note:** this function has to be called at the creation of the `Ebb` before adding diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 3fefb4e555..279630ef00 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -109,7 +109,7 @@ struct EbbHeaderBlockData { sealed: bool, // The ebb which this block is part of. ebb: Ebb, - // List of current Ebb arguments for which a earlier def has not been found yet. + // List of current Ebb arguments for which an earlier def has not been found yet. undef_variables: Vec<(Variable, Value)>, } @@ -426,7 +426,7 @@ where preds: &[(Block, Inst)], ) -> (Value, SideEffects) { let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero(); - // TODO: find a way not not allocate a vector + // TODO: find a way to not allocate a vector let mut jump_args_to_append: Vec<(Block, Inst, Value)> = Vec::new(); let ty = dfg.value_type(temp_arg_val); let mut side_effects = SideEffects::new(); diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index f8e90de1aa..d35ee6bcb4 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -571,7 +571,7 @@ impl<'a> Parser<'a> { // Match and consume an optional offset32 immediate. // - // Note that that this will match an empty string as an empty offset, and that if an offset is + // Note that this will match an empty string as an empty offset, and that if an offset is // present, it must contain a sign. fn optional_offset32(&mut self) -> Result { if let Some(Token::Integer(text)) = self.token() { @@ -587,7 +587,7 @@ impl<'a> Parser<'a> { // Match and consume an optional uoffset32 immediate. // - // Note that that this will match an empty string as an empty offset, and that if an offset is + // Note that this will match an empty string as an empty offset, and that if an offset is // present, it must contain a `+` sign. fn optional_uoffset32(&mut self) -> Result { if let Some(Token::Integer(text)) = self.token() { diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 91dfb00281..62d5adcfe8 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -177,7 +177,7 @@ pub fn translate_function_body( let mut func_imports = FunctionImports::new(); let mut stack: Vec = Vec::new(); let mut control_stack: Vec = Vec::new(); - // We introduce a arbitrary scope for the FunctionBuilder object + // We introduce an arbitrary scope for the FunctionBuilder object { let mut builder = FunctionBuilder::new(&mut func, il_builder); let first_ebb = builder.create_ebb(); @@ -1237,7 +1237,7 @@ fn translate_unreachable_operator( match frame { // If it is a loop we also have to seal the body loop block ControlStackFrame::Loop { header, .. } => builder.seal_block(header), - // If it is a if then the code after is reachable again + // If it is an if then the code after is reachable again ControlStackFrame::If { .. } => { state.real_unreachable_stack_depth = 1; } @@ -1297,7 +1297,7 @@ fn args_count( signatures[functions[index]].argument_types.len() } -// Given a index in the function index space, search for it in the function imports and if it is +// Given an index in the function index space, search for it in the function imports and if it is // not there add it to the function imports. fn find_function_import( index: FunctionIndex, From af74cdf36439a38c6537e16b5c48fe0b753760c9 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 13:40:35 -0700 Subject: [PATCH 1033/3084] Cretonne's top-level tools don't need to directly depend on wasmparser. --- cranelift/Cargo.toml | 1 - 1 file changed, 1 deletion(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 3d5080974c..6523a9d2da 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -18,7 +18,6 @@ cretonne-reader = { path = "lib/reader" } cretonne-frontend = { path = "lib/frontend" } cretonne-wasm = { path = "lib/wasm" } filecheck = { path = "lib/filecheck" } -wasmparser = "0.6.1" docopt = "0.8.0" serde = "1.0.8" serde_derive = "1.0.8" From 0cacba15b937f306dbb8859c0ff36aea8b6911ec Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 13 Jul 2017 16:37:07 -0700 Subject: [PATCH 1034/3084] Replace `as` casts with type-conversion functions. https://github.com/rust-lang-nursery/rust-clippy/wiki#cast_lossless --- lib/cretonne/meta/isa/riscv/recipes.py | 12 +++++----- lib/cretonne/src/ir/immediates.rs | 32 ++++++++++++++++---------- lib/cretonne/src/ir/instructions.rs | 2 +- lib/cretonne/src/ir/types.rs | 8 +++---- lib/cretonne/src/isa/constraints.rs | 2 +- lib/cretonne/src/isa/encoding.rs | 2 +- lib/cretonne/src/isa/intel/abi.rs | 2 +- lib/cretonne/src/isa/registers.rs | 4 ++-- lib/cretonne/src/isa/riscv/abi.rs | 2 +- lib/cretonne/src/regalloc/pressure.rs | 4 ++-- lib/cretonne/src/settings.rs | 4 ++-- lib/reader/src/parser.rs | 2 +- 12 files changed, 42 insertions(+), 34 deletions(-) diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index afffb2c0aa..87dce643eb 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -171,8 +171,8 @@ U = EncRecipe( UJ = EncRecipe( 'UJ', Jump, size=4, ins=(), outs=(), branch_range=(0, 21), emit=''' - let dest = func.offsets[destination] as i64; - let disp = dest - sink.offset() as i64; + let dest = i64::from(func.offsets[destination]); + let disp = dest - i64::from(sink.offset()); put_uj(bits, disp, 0, sink); ''') @@ -190,8 +190,8 @@ SB = EncRecipe( ins=(GPR, GPR), outs=(), branch_range=(0, 13), emit=''' - let dest = func.offsets[destination] as i64; - let disp = dest - sink.offset() as i64; + let dest = i64::from(func.offsets[destination]); + let disp = dest - i64::from(sink.offset()); put_sb(bits, disp, in_reg0, in_reg1, sink); ''') @@ -201,8 +201,8 @@ SBzero = EncRecipe( ins=(GPR), outs=(), branch_range=(0, 13), emit=''' - let dest = func.offsets[destination] as i64; - let disp = dest - sink.offset() as i64; + let dest = i64::from(func.offsets[destination]); + let disp = dest - i64::from(sink.offset()); put_sb(bits, disp, in_reg0, 0, sink); ''') diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 72763c558e..2ab486f864 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -87,7 +87,7 @@ fn parse_i64(s: &str) -> Result { return Err("Too many hexadecimal digits"); } // This can't overflow given the digit limit. - value = (value << 4) | digit as u64; + value = (value << 4) | u64::from(digit); } None => { // Allow embedded underscores, but fail on anything else. @@ -107,7 +107,7 @@ fn parse_i64(s: &str) -> Result { None => return Err("Too large decimal number"), Some(v) => value = v, } - match value.checked_add(digit as u64) { + match value.checked_add(u64::from(digit)) { None => return Err("Too large decimal number"), Some(v) => value = v, } @@ -221,7 +221,7 @@ impl Into for Offset32 { impl Into for Offset32 { fn into(self) -> i64 { - self.0 as i64 + i64::from(self.0) } } @@ -241,7 +241,7 @@ impl Display for Offset32 { // Always include a sign. write!(f, "{}", if self.0 < 0 { '-' } else { '+' })?; - let val = (self.0 as i64).abs(); + let val = i64::from(self.0).abs(); if val < 10_000 { write!(f, "{}", val) } else { @@ -259,7 +259,9 @@ impl FromStr for Offset32 { if !(s.starts_with('-') || s.starts_with('+')) { return Err("Offset must begin with sign"); } - parse_i64(s).and_then(|x| if i32::MIN as i64 <= x && x <= i32::MAX as i64 { + parse_i64(s).and_then(|x| if i64::from(i32::MIN) as i64 <= x && + x <= i64::from(i32::MAX) + { Ok(Offset32::new(x as i32)) } else { Err("Offset out of range") @@ -288,7 +290,7 @@ impl Into for Uoffset32 { impl Into for Uoffset32 { fn into(self) -> i64 { - self.0 as i64 + i64::from(self.0) } } @@ -310,7 +312,7 @@ impl Display for Uoffset32 { write!(f, "+{}", self.0) } else { write!(f, "+")?; - write_hex(self.0 as i64, f) + write_hex(i64::from(self.0), f) } } @@ -324,7 +326,7 @@ impl FromStr for Uoffset32 { if !s.starts_with('+') { return Err("Unsigned offset must begin with '+' sign"); } - parse_i64(s).and_then(|x| if 0 <= x && x <= u32::MAX as i64 { + parse_i64(s).and_then(|x| if 0 <= x && x <= i64::from(u32::MAX) { Ok(Uoffset32::new(x as u32)) } else { Err("Offset out of range") @@ -386,7 +388,13 @@ fn format_float(bits: u64, w: u8, t: u8, f: &mut Formatter) -> fmt::Result { write!(f, "0.0") } else { // Subnormal. - write!(f, "0x0.{0:01$x}p{2}", left_t_bits, digits as usize, emin) + write!( + f, + "0x0.{0:01$x}p{2}", + left_t_bits, + usize::from(digits), + emin + ) } } else if e_bits == max_e_bits { // Always print a `+` or `-` sign for these special values. @@ -414,7 +422,7 @@ fn format_float(bits: u64, w: u8, t: u8, f: &mut Formatter) -> fmt::Result { } } else { // Normal number. - write!(f, "0x1.{0:01$x}p{2}", left_t_bits, digits as usize, e) + write!(f, "0x1.{0:01$x}p{2}", left_t_bits, usize::from(digits), e) } } @@ -511,7 +519,7 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { if digits > 16 { return Err("Too many digits"); } - significand = (significand << 4) | digit as u64; + significand = (significand << 4) | u64::from(digit); } None => return Err("Invalid character"), } @@ -596,7 +604,7 @@ impl Ieee32 { impl Display for Ieee32 { fn fmt(&self, f: &mut Formatter) -> fmt::Result { let bits: u32 = self.0; - format_float(bits as u64, 8, 23, f) + format_float(u64::from(bits), 8, 23, f) } } diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 19ef8efa29..b37646afbf 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -463,7 +463,7 @@ impl OpcodeConstraints { /// Get the offset into `TYPE_SETS` for the controlling type variable. /// Returns `None` if the instruction is not polymorphic. fn typeset_offset(self) -> Option { - let offset = self.typeset_offset as usize; + let offset = usize::from(self.typeset_offset); if offset < TYPE_SETS.len() { Some(offset) } else { diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 275ec70c9b..5b6f0fe7fb 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -194,12 +194,12 @@ impl Type { /// Get the total number of bits used to represent this type. pub fn bits(self) -> u16 { - self.lane_bits() as u16 * self.lane_count() + u16::from(self.lane_bits()) * self.lane_count() } /// Get the number of bytes used to store this type in memory. pub fn bytes(self) -> u32 { - (self.bits() as u32 + 7) / 8 + (u32::from(self.bits()) + 7) / 8 } /// Get a SIMD vector type with `n` times more lanes than this one. @@ -213,7 +213,7 @@ impl Type { return None; } let log2_lanes: u32 = n.trailing_zeros(); - let new_type = self.0 as u32 + (log2_lanes << 4); + let new_type = u32::from(self.0) + (log2_lanes << 4); if new_type < 0x90 { Some(Type(new_type as u8)) } else { @@ -234,7 +234,7 @@ impl Type { /// Index of this type, for use with hash tables etc. pub fn index(self) -> usize { - self.0 as usize + usize::from(self.0) } /// True iff: diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index e38b044535..c5f1d4bd16 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -105,7 +105,7 @@ impl BranchRange { /// /// This method does not detect if the range is larger than 2 GB. pub fn contains(self, branch: CodeOffset, dest: CodeOffset) -> bool { - let d = dest.wrapping_sub(branch + self.origin as CodeOffset) as i32; + let d = dest.wrapping_sub(branch + CodeOffset::from(self.origin)) as i32; let s = 32 - self.bits; d == d << s >> s } diff --git a/lib/cretonne/src/isa/encoding.rs b/lib/cretonne/src/isa/encoding.rs index cfb429ba1e..b82137a6b8 100644 --- a/lib/cretonne/src/isa/encoding.rs +++ b/lib/cretonne/src/isa/encoding.rs @@ -124,7 +124,7 @@ impl EncInfo { pub fn bytes(&self, enc: Encoding) -> CodeOffset { self.sizing .get(enc.recipe()) - .map(|s| s.bytes as CodeOffset) + .map(|s| CodeOffset::from(s.bytes)) .unwrap_or(0) } diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 58ca657dc4..707d42b8d9 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -28,7 +28,7 @@ struct Args { impl Args { fn new(bits: u16, gpr: &'static [RU], fpr_limit: usize) -> Args { Args { - pointer_bytes: bits as u32 / 8, + pointer_bytes: u32::from(bits) / 8, pointer_bits: bits, pointer_type: ir::Type::int(bits).unwrap(), gpr, diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index e156de331e..a43d5cfd62 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -174,7 +174,7 @@ impl RegClassData { /// Get a specific register unit in this class. pub fn unit(&self, offset: usize) -> RegUnit { - let uoffset = offset * self.width as usize; + let uoffset = offset * usize::from(self.width); self.first + uoffset as RegUnit } @@ -209,7 +209,7 @@ impl EntityRef for RegClassIndex { } fn index(self) -> usize { - self.0 as usize + usize::from(self.0) } } diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index f4ab75fde9..0859a53f31 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -26,7 +26,7 @@ impl Args { fn new(bits: u16, enable_e: bool) -> Args { Args { pointer_bits: bits, - pointer_bytes: bits as u32 / 8, + pointer_bytes: u32::from(bits) / 8, pointer_type: Type::int(bits).unwrap(), regs: 0, reg_limit: if enable_e { 6 } else { 8 }, diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index 7a4f1f715b..28cf15f48f 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -142,8 +142,8 @@ impl Pressure { /// /// This is the out-of-line slow path for `check_avail()`. fn check_avail_aliased(&self, entry: &TopRC) -> RegClassMask { - let first = entry.first_toprc as usize; - let num = entry.num_toprcs as usize; + let first = usize::from(entry.first_toprc); + let num = usize::from(entry.num_toprcs); let width = entry.width as u32; let ulimit = entry.limit * width; diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 9738c422af..999d305439 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -212,7 +212,7 @@ pub mod detail { /// Get enumerators corresponding to a `Details::Enum`. pub fn enums(&self, last: u8, enumerators: u16) -> &[&'static str] { let from = enumerators as usize; - let len = last as usize + 1; + let len = usize::from(last) + 1; &self.enumerators[from..from + len] } @@ -230,7 +230,7 @@ pub mod detail { Detail::Enum { last, enumerators } => { if byte <= last { let tags = self.enums(last, enumerators); - write!(f, "\"{}\"", tags[byte as usize]) + write!(f, "\"{}\"", tags[usize::from(byte)]) } else { write!(f, "{}", byte) } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index d35ee6bcb4..259d462298 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1115,7 +1115,7 @@ impl<'a> Parser<'a> { if bytes < 0 { return err!(self.loc, "negative stack slot size"); } - if bytes > u32::MAX as i64 { + if bytes > i64::from(u32::MAX) { return err!(self.loc, "stack slot too large"); } let mut data = StackSlotData::new(kind, bytes as u32); From 7be0abb442090dc3247e691d813450c529c36612 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 14:15:25 -0700 Subject: [PATCH 1035/3084] Remove an unneeded extern crate. --- cranelift/src/cton-util.rs | 1 - 1 file changed, 1 deletion(-) diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 4e18001a0b..b940947523 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -3,7 +3,6 @@ extern crate cretonne; extern crate cton_reader; extern crate cretonne_wasm; extern crate docopt; -extern crate serde; #[macro_use] extern crate serde_derive; extern crate filecheck; From 2d9b902d2e0851c825744a9592cc5cca1ebae8e3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 14:35:19 -0700 Subject: [PATCH 1036/3084] Clean up a redundant cast. --- lib/cretonne/src/ir/immediates.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 2ab486f864..f33e13eccf 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -259,7 +259,7 @@ impl FromStr for Offset32 { if !(s.starts_with('-') || s.starts_with('+')) { return Err("Offset must begin with sign"); } - parse_i64(s).and_then(|x| if i64::from(i32::MIN) as i64 <= x && + parse_i64(s).and_then(|x| if i64::from(i32::MIN) <= x && x <= i64::from(i32::MAX) { Ok(Offset32::new(x as i32)) From 79f257060f645313df41e930d92b0cb3f35484b3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 14:49:53 -0700 Subject: [PATCH 1037/3084] Inline resolve_undef_vars, and collapse the resulting redundancy. --- lib/frontend/src/ssa.rs | 35 +++++------------------------------ 1 file changed, 5 insertions(+), 30 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 279630ef00..603f81073d 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -334,6 +334,8 @@ where /// /// This method modifies the function's `Layout` by adding arguments to the `Ebb`s to /// take into account the Phi function placed by the SSA algorithm. + /// + /// Returns the list of newly created ebbs for critical edge splitting. pub fn seal_ebb_header_block( &mut self, ebb: Ebb, @@ -343,42 +345,13 @@ where ) -> SideEffects { let block = self.header_block(ebb); - // Sanity check - match self.blocks[block] { - BlockData::EbbBody { .. } => panic!("you can't seal an Ebb body block"), - BlockData::EbbHeader(ref data) => { - assert!(!data.sealed); - } - } - - // Recurse over the predecessors to find good definitions. - let side_effects = self.resolve_undef_vars(block, dfg, layout, jts); - - // Then we mark the block as sealed. - match self.blocks[block] { - BlockData::EbbBody { .. } => panic!("this should not happen"), - BlockData::EbbHeader(ref mut data) => data.sealed = true, - }; - side_effects - } - - // For each undef_var in an Ebb header block, lookup in the predecessors to append the right - // jump argument to the branch instruction. - // Panics if called with a non-header block. - // Returns the list of newly created ebbs for critical edge splitting. - fn resolve_undef_vars( - &mut self, - block: Block, - dfg: &mut DataFlowGraph, - layout: &mut Layout, - jts: &mut JumpTables, - ) -> SideEffects { // TODO: find a way to not allocate vectors let (predecessors, undef_vars, ebb): (Vec<(Block, Inst)>, Vec<(Variable, Value)>, Ebb) = match self.blocks[block] { BlockData::EbbBody { .. } => panic!("this should not happen"), BlockData::EbbHeader(ref mut data) => { + assert!(!data.sealed); ( data.predecessors.clone(), data.undef_variables.clone(), @@ -406,7 +379,9 @@ where match self.blocks[block] { BlockData::EbbBody { .. } => panic!("this should not happen"), BlockData::EbbHeader(ref mut data) => { + assert!(!data.sealed); data.undef_variables.clear(); + data.sealed = true; } }; side_effects From 539a5b33787d204b535ce5cee059c8566adf6df4 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 14:56:27 -0700 Subject: [PATCH 1038/3084] Avoid a heap allocation for a block's undef_variables. --- lib/frontend/src/ssa.rs | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 603f81073d..c9f763cf7f 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -16,6 +16,7 @@ use std::u32; use cretonne::ir::types::{F32, F64}; use cretonne::ir::immediates::{Ieee32, Ieee64}; use std::collections::HashMap; +use std::mem; /// Structure containing the data relevant the construction of SSA for a given function. /// @@ -352,11 +353,11 @@ where BlockData::EbbBody { .. } => panic!("this should not happen"), BlockData::EbbHeader(ref mut data) => { assert!(!data.sealed); - ( - data.predecessors.clone(), - data.undef_variables.clone(), - data.ebb, - ) + // Extract the undef_variables data from the block so that we + // can iterate over it without borrowing the whole builder. + let mut undef_variables = Vec::new(); + mem::swap(&mut data.undef_variables, &mut undef_variables); + (data.predecessors.clone(), undef_variables, data.ebb) } }; @@ -375,12 +376,12 @@ where ); } - // Then we clear the undef_vars and mark the block as sealed. + // Then we mark the block as sealed. match self.blocks[block] { BlockData::EbbBody { .. } => panic!("this should not happen"), BlockData::EbbHeader(ref mut data) => { assert!(!data.sealed); - data.undef_variables.clear(); + assert!(data.undef_variables.is_empty()); data.sealed = true; } }; From e3227e38ce25f371dcae2566e083ed39a9e06691 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 16:09:26 -0700 Subject: [PATCH 1039/3084] Avoid cloning precessor lists to the heap. --- lib/frontend/src/ssa.rs | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index c9f763cf7f..016d8f0ce7 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -169,7 +169,7 @@ enum ZeroOneOrMore { enum UseVarCases { Unsealed(Value), SealedOnePredecessor(Block), - SealedMultiplePredecessors(Vec<(Block, Inst)>, Value, Ebb), + SealedMultiplePredecessors(Value, Ebb), } /// The following methods are the API of the SSA builder. Here is how it should be used when @@ -238,8 +238,7 @@ where UseVarCases::SealedOnePredecessor(data.predecessors[0].0) } else { let val = dfg.append_ebb_arg(data.ebb, ty); - let preds = data.predecessors.clone(); - UseVarCases::SealedMultiplePredecessors(preds, val, data.ebb) + UseVarCases::SealedMultiplePredecessors(val, data.ebb) } } else { let val = dfg.append_ebb_arg(data.ebb, ty); @@ -264,11 +263,11 @@ where self.def_var(var, val, block); (val, SideEffects::new()) } - UseVarCases::SealedMultiplePredecessors(preds, val, ebb) => { + UseVarCases::SealedMultiplePredecessors(val, ebb) => { // If multiple predecessor we look up a use_var in each of them: // if they all yield the same value no need for an Ebb argument self.def_var(var, val, block); - self.predecessors_lookup(dfg, layout, jts, val, var, ebb, &preds) + self.predecessors_lookup(dfg, layout, jts, val, var, ebb) } } } @@ -346,10 +345,7 @@ where ) -> SideEffects { let block = self.header_block(ebb); - // TODO: find a way to not allocate vectors - let (predecessors, undef_vars, ebb): (Vec<(Block, Inst)>, - Vec<(Variable, Value)>, - Ebb) = match self.blocks[block] { + let (undef_vars, ebb): (Vec<(Variable, Value)>, Ebb) = match self.blocks[block] { BlockData::EbbBody { .. } => panic!("this should not happen"), BlockData::EbbHeader(ref mut data) => { assert!(!data.sealed); @@ -357,7 +353,7 @@ where // can iterate over it without borrowing the whole builder. let mut undef_variables = Vec::new(); mem::swap(&mut data.undef_variables, &mut undef_variables); - (data.predecessors.clone(), undef_variables, data.ebb) + (undef_variables, data.ebb) } }; @@ -366,7 +362,7 @@ where // only if necessary. for (var, val) in undef_vars { let (_, mut local_side_effects) = - self.predecessors_lookup(dfg, layout, jts, val, var, ebb, &predecessors); + self.predecessors_lookup(dfg, layout, jts, val, var, ebb); side_effects.split_ebbs_created.append( &mut local_side_effects .split_ebbs_created, @@ -399,13 +395,18 @@ where temp_arg_val: Value, temp_arg_var: Variable, dest_ebb: Ebb, - preds: &[(Block, Inst)], ) -> (Value, SideEffects) { let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero(); // TODO: find a way to not allocate a vector let mut jump_args_to_append: Vec<(Block, Inst, Value)> = Vec::new(); let ty = dfg.value_type(temp_arg_val); let mut side_effects = SideEffects::new(); + + // Iterate over the predecessors. To avoid borrowing `self` for the whole loop, + // temporarily detach the predecessors list and replace it with an empty list. + // `use_var`'s traversal won't revisit these predecesors. + let mut preds = Vec::new(); + mem::swap(&mut preds, &mut self.predecessors_mut(dest_ebb)); for &(pred, last_inst) in preds.iter() { // For undef value and each predecessor we query what is the local SSA value // corresponding to var and we put it as an argument of the branch instruction. @@ -437,6 +438,10 @@ where &mut local_side_effects.instructions_added_to_ebbs, ); } + // Now that we're done iterating, move the predecessors list back. + debug_assert!(self.predecessors(dest_ebb).is_empty()); + *self.predecessors_mut(dest_ebb) = preds; + match pred_values { ZeroOneOrMore::Zero() => { // The variable is used but never defined before. This is an irregularity in the @@ -560,6 +565,15 @@ where } } + /// Same as predecessors, but for &mut. + pub fn predecessors_mut(&mut self, ebb: Ebb) -> &mut Vec<(Block, Inst)> { + let block = self.header_block(ebb); + match self.blocks[block] { + BlockData::EbbBody { .. } => panic!("should not happen"), + BlockData::EbbHeader(ref mut data) => &mut data.predecessors, + } + } + /// Returns `true` if and only if `seal_ebb_header_block` has been called on the argument. pub fn is_sealed(&self, ebb: Ebb) -> bool { match self.blocks[self.header_block(ebb)] { From 583487bfac14a26481a761f6cec812141a8f6cbd Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 16:30:00 -0700 Subject: [PATCH 1040/3084] Refactor the non-local case of use_var into a separate function. --- lib/frontend/src/ssa.rs | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 016d8f0ce7..6ec44c5f7a 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -225,9 +225,23 @@ where if let Some(val) = var_defs.get(&block) { return (*val, SideEffects::new()); } - }; - // At this point if we haven't returned it means that we have to search in the - // predecessors. + } + + // Otherwise, we have to do a non-local lookup. + self.use_var_nonlocal(dfg, layout, jts, var, ty, block) + } + + // The non-local case of use_var. Query each predecessor for a value and add branch + // arguments as needed to satisfy the use. + fn use_var_nonlocal( + &mut self, + dfg: &mut DataFlowGraph, + layout: &mut Layout, + jts: &mut JumpTables, + var: Variable, + ty: Type, + block: Block, + ) -> (Value, SideEffects) { let case = match self.blocks[block] { BlockData::EbbHeader(ref mut data) => { // The block has multiple predecessors so we append an Ebb argument that From 1d03244e902bb0f02b4e366bcbee71e824f188b5 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 16:36:18 -0700 Subject: [PATCH 1041/3084] Use debug_assert instead of assert in code where performance is important. --- lib/frontend/src/ssa.rs | 8 ++++---- lib/wasm/src/translation_utils.rs | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 6ec44c5f7a..498b0b0ea3 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -119,7 +119,7 @@ struct EbbHeaderBlockData { pub struct Block(u32); impl EntityRef for Block { fn new(index: usize) -> Self { - assert!(index < (u32::MAX as usize)); + debug_assert!(index < (u32::MAX as usize)); Block(index as u32) } @@ -362,7 +362,7 @@ where let (undef_vars, ebb): (Vec<(Variable, Value)>, Ebb) = match self.blocks[block] { BlockData::EbbBody { .. } => panic!("this should not happen"), BlockData::EbbHeader(ref mut data) => { - assert!(!data.sealed); + debug_assert!(!data.sealed); // Extract the undef_variables data from the block so that we // can iterate over it without borrowing the whole builder. let mut undef_variables = Vec::new(); @@ -390,8 +390,8 @@ where match self.blocks[block] { BlockData::EbbBody { .. } => panic!("this should not happen"), BlockData::EbbHeader(ref mut data) => { - assert!(!data.sealed); - assert!(data.undef_variables.is_empty()); + debug_assert!(!data.sealed); + debug_assert!(data.undef_variables.is_empty()); data.sealed = true; } }; diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 4a7f375bb8..d378864e5c 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -89,7 +89,7 @@ pub struct Memory { pub struct Local(pub u32); impl cretonne::entity::EntityRef for Local { fn new(index: usize) -> Self { - assert!(index < (u32::MAX as usize)); + debug_assert!(index < (u32::MAX as usize)); Local(index as u32) } From de029761625b526d117c1b124552b1684115ce72 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 16:16:08 -0700 Subject: [PATCH 1042/3084] Minor comment cleanups. --- lib/cretonne/src/abi.rs | 2 +- lib/cretonne/src/licm.rs | 2 +- lib/frontend/src/ssa.rs | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index 49620bcfde..38bc6781d1 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -162,7 +162,7 @@ pub fn legalize_abi_value(have: Type, arg: &ArgumentType) -> ValueConversion { } // We have the same number of bits as the argument. Ordering::Equal => { - // This must be an integer vector that is split and then extended. + // This must be an integer vector that is split and then extended. assert!(arg.value_type.is_int()); assert!(!have.is_scalar()); ValueConversion::VectorSplit diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 689067c188..20dfca9f5d 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -97,7 +97,7 @@ fn create_pre_header( // // A loop header has a pre-header if there is only one predecessor that the header doesn't // dominate. -// Returns the pre-header Ebb and the instruction jumping to the header. +// Returns the pre-header Ebb and the instruction jumping to the header. fn has_pre_header( layout: &Layout, cfg: &ControlFlowGraph, diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 498b0b0ea3..250d679983 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -422,8 +422,8 @@ where let mut preds = Vec::new(); mem::swap(&mut preds, &mut self.predecessors_mut(dest_ebb)); for &(pred, last_inst) in preds.iter() { - // For undef value and each predecessor we query what is the local SSA value - // corresponding to var and we put it as an argument of the branch instruction. + // For each predecessor, we query what is the local SSA value corresponding + // to var and we put it as an argument of the branch instruction. let (pred_val, mut local_side_effects) = self.use_var(dfg, layout, jts, temp_arg_var, ty, pred); pred_values = match pred_values { @@ -527,7 +527,7 @@ where ) -> Option { match dfg[jump_inst].analyze_branch(&dfg.value_lists) { BranchInfo::NotABranch => { - panic!("you have declared a non-branch instruction as a predecessor to an ebb"); + panic!("you have declared a non-branch instruction as a predecessor to an ebb"); } // For a single destination appending a jump argument to the instruction // is sufficient. From 566c772e2018fcca0b840123eb471edd4cb39094 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 16:51:06 -0700 Subject: [PATCH 1043/3084] Eliminate more unnecessary calls to .iter(). --- cranelift/src/wasm.rs | 6 +++--- lib/cretonne/src/regalloc/liveness.rs | 2 +- lib/frontend/src/ssa.rs | 2 +- lib/wasm/tests/testsuite.rs | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 3a972e9716..c54cfa85d0 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -57,7 +57,7 @@ pub fn run( flag_optimize: bool, flag_check: bool, ) -> Result<(), String> { - for filename in files.iter() { + for filename in files { let path = Path::new(&filename); let name = String::from(path.as_os_str().to_string_lossy()); match handle_module( @@ -148,7 +148,7 @@ fn handle_module( terminal.fg(term::color::MAGENTA).unwrap(); vprint!(flag_verbose, "Checking... "); terminal.reset().unwrap(); - for func in translation.functions.iter() { + for func in &translation.functions { let il = match *func { FunctionTranslation::Import() => continue, FunctionTranslation::Code { ref il, .. } => il.clone(), @@ -166,7 +166,7 @@ fn handle_module( terminal.fg(term::color::MAGENTA).unwrap(); vprint!(flag_verbose, "Optimizing... "); terminal.reset().unwrap(); - for func in translation.functions.iter() { + for func in &translation.functions { let mut il = match *func { FunctionTranslation::Import() => continue, FunctionTranslation::Code { ref il, .. } => il.clone(), diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 72b3814770..2c3c430534 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -433,7 +433,7 @@ impl Liveness { assert!(self.ebb_args.is_empty()); for ebb in func.layout.ebbs() { - for &arg in func.dfg.ebb_args(ebb).iter() { + for &arg in func.dfg.ebb_args(ebb) { let affinity = self.ranges.get(arg).unwrap().affinity; if affinity.is_none() { continue; diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 250d679983..ecfe57b234 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -421,7 +421,7 @@ where // `use_var`'s traversal won't revisit these predecesors. let mut preds = Vec::new(); mem::swap(&mut preds, &mut self.predecessors_mut(dest_ebb)); - for &(pred, last_inst) in preds.iter() { + for &(pred, last_inst) in &preds { // For each predecessor, we query what is the local SSA value corresponding // to var and we put it as an argument of the branch instruction. let (pred_val, mut local_side_effects) = diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/testsuite.rs index d4c51c6a90..cb56779b00 100644 --- a/lib/wasm/tests/testsuite.rs +++ b/lib/wasm/tests/testsuite.rs @@ -69,8 +69,8 @@ fn handle_module(path: PathBuf) -> Result<(), String> { } } }; - for func in translation.functions.iter() { - let il = match *func { + for func in translation.functions { + let il = match func { FunctionTranslation::Import() => continue, FunctionTranslation::Code { ref il, .. } => il.clone(), }; From 52335c9e0f076b6b3e7ef8ef41bcb4f5d3062f0f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 17:06:14 -0700 Subject: [PATCH 1044/3084] Delete an obsolete TODO comment. --- lib/frontend/src/ssa.rs | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index ecfe57b234..e2e618d0f1 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -164,7 +164,6 @@ enum ZeroOneOrMore { More(), } -/// TODO: use entity list instead of vec #[derive(Debug)] enum UseVarCases { Unsealed(Value), From ec3972f515c6c30e52e043bdc2a8adda840a7f20 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 17:42:19 -0700 Subject: [PATCH 1045/3084] Avoid allocating a vector for predecessor variables in the single-value case. --- lib/frontend/src/ssa.rs | 37 ++++++++++++++++++------------------- 1 file changed, 18 insertions(+), 19 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index e2e618d0f1..51cbe1e253 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -161,7 +161,7 @@ where enum ZeroOneOrMore { Zero(), One(T), - More(), + More(Vec), } #[derive(Debug)] @@ -409,9 +409,7 @@ where temp_arg_var: Variable, dest_ebb: Ebb, ) -> (Value, SideEffects) { - let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero(); - // TODO: find a way to not allocate a vector - let mut jump_args_to_append: Vec<(Block, Inst, Value)> = Vec::new(); + let mut pred_values: ZeroOneOrMore<(Block, Inst, Value)> = ZeroOneOrMore::Zero(); let ty = dfg.value_type(temp_arg_val); let mut side_effects = SideEffects::new(); @@ -425,24 +423,25 @@ where // to var and we put it as an argument of the branch instruction. let (pred_val, mut local_side_effects) = self.use_var(dfg, layout, jts, temp_arg_var, ty, pred); - pred_values = match pred_values { + match pred_values { ZeroOneOrMore::Zero() => { - if pred_val == temp_arg_val { - ZeroOneOrMore::Zero() - } else { - ZeroOneOrMore::One(pred_val) + if pred_val != temp_arg_val { + pred_values = ZeroOneOrMore::One((pred, last_inst, pred_val)) } } - ZeroOneOrMore::One(old_val) => { - if pred_val == temp_arg_val || pred_val == old_val { - ZeroOneOrMore::One(old_val) - } else { - ZeroOneOrMore::More() + ZeroOneOrMore::One((old_pred, old_last_inst, old_val)) => { + if pred_val != temp_arg_val && pred_val != old_val { + // TODO: find a way to not allocate a vector + pred_values = ZeroOneOrMore::More(vec![ + (old_pred, old_last_inst, old_val), + (pred, last_inst, pred_val), + ]); } } - ZeroOneOrMore::More() => ZeroOneOrMore::More(), - }; - jump_args_to_append.push((pred, last_inst, pred_val)); + ZeroOneOrMore::More(ref mut jump_args_to_append) => { + jump_args_to_append.push((pred, last_inst, pred_val)); + } + } side_effects.split_ebbs_created.append( &mut local_side_effects .split_ebbs_created, @@ -479,7 +478,7 @@ where side_effects.instructions_added_to_ebbs.push(dest_ebb); (val, side_effects) } - ZeroOneOrMore::One(pred_val) => { + ZeroOneOrMore::One((_, _, pred_val)) => { // Here all the predecessors use a single value to represent our variable // so we don't need to have it as an ebb argument. // We need to replace all the occurences of val with pred_val but since @@ -488,7 +487,7 @@ where dfg.change_to_alias(temp_arg_val, pred_val); (pred_val, side_effects) } - ZeroOneOrMore::More() => { + ZeroOneOrMore::More(jump_args_to_append) => { // There is disagreement in the predecessors on which value to use so we have // to keep the ebb argument. for (pred_block, last_inst, pred_val) in jump_args_to_append { From 7049cc78ae11e987e35e7ce11636adf8bdd38056 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 Aug 2017 17:49:39 -0700 Subject: [PATCH 1046/3084] Refactor out an append helper function for SideEffects. --- lib/frontend/src/ssa.rs | 28 +++++++++++----------------- 1 file changed, 11 insertions(+), 17 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 51cbe1e253..eccdfc2dab 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -66,6 +66,13 @@ impl SideEffects { instructions_added_to_ebbs: Vec::new(), } } + + fn append(&mut self, mut more: SideEffects) { + self.split_ebbs_created.append(&mut more.split_ebbs_created); + self.instructions_added_to_ebbs.append( + &mut more.instructions_added_to_ebbs, + ); + } } // Describes the current position of a basic block in the control flow graph. @@ -374,15 +381,8 @@ where // For each undef var we look up values in the predecessors and create an Ebb argument // only if necessary. for (var, val) in undef_vars { - let (_, mut local_side_effects) = - self.predecessors_lookup(dfg, layout, jts, val, var, ebb); - side_effects.split_ebbs_created.append( - &mut local_side_effects - .split_ebbs_created, - ); - side_effects.instructions_added_to_ebbs.append( - &mut local_side_effects.instructions_added_to_ebbs, - ); + let (_, local_side_effects) = self.predecessors_lookup(dfg, layout, jts, val, var, ebb); + side_effects.append(local_side_effects); } // Then we mark the block as sealed. @@ -421,7 +421,7 @@ where for &(pred, last_inst) in &preds { // For each predecessor, we query what is the local SSA value corresponding // to var and we put it as an argument of the branch instruction. - let (pred_val, mut local_side_effects) = + let (pred_val, local_side_effects) = self.use_var(dfg, layout, jts, temp_arg_var, ty, pred); match pred_values { ZeroOneOrMore::Zero() => { @@ -442,13 +442,7 @@ where jump_args_to_append.push((pred, last_inst, pred_val)); } } - side_effects.split_ebbs_created.append( - &mut local_side_effects - .split_ebbs_created, - ); - side_effects.instructions_added_to_ebbs.append( - &mut local_side_effects.instructions_added_to_ebbs, - ); + side_effects.append(local_side_effects); } // Now that we're done iterating, move the predecessors list back. debug_assert!(self.predecessors(dest_ebb).is_empty()); From 9bc4264a33705b98a92c4a2e93178f0f6ad4cd60 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 1 Sep 2017 09:31:05 -0700 Subject: [PATCH 1047/3084] Update dependency to wasmparser v0.8.2. Soon we'll need the BinaryReader::read_local_decl() method which was added in that release. --- lib/wasm/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index d03f178cd5..28710bdc76 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -8,6 +8,6 @@ repository = "https://github.com/stoklund/cretonne" license = "Apache-2.0" [dependencies] -wasmparser = "0.6.1" +wasmparser = "0.8.2" cretonne = { path = "../cretonne" } cretonne-frontend = { path = "../frontend" } From 28779dc7e47ac435901644d373c5a0bf6ebfea01 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 1 Sep 2017 12:11:30 -0700 Subject: [PATCH 1048/3084] Avoid allocating jump arguments on the heap. Instead of allocating a vector to store jump arguments for processing later, just have the later code rescan through the predecessors to obtain the values, which are memoized. This also eliminates a linear search through the predecessor list when splitting a critical edge. --- lib/frontend/src/ssa.rs | 89 ++++++++++++++++++++++------------------- 1 file changed, 47 insertions(+), 42 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index eccdfc2dab..a1b9d56de9 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -168,7 +168,7 @@ where enum ZeroOneOrMore { Zero(), One(T), - More(Vec), + More(), } #[derive(Debug)] @@ -409,7 +409,7 @@ where temp_arg_var: Variable, dest_ebb: Ebb, ) -> (Value, SideEffects) { - let mut pred_values: ZeroOneOrMore<(Block, Inst, Value)> = ZeroOneOrMore::Zero(); + let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero(); let ty = dfg.value_type(temp_arg_val); let mut side_effects = SideEffects::new(); @@ -418,7 +418,7 @@ where // `use_var`'s traversal won't revisit these predecesors. let mut preds = Vec::new(); mem::swap(&mut preds, &mut self.predecessors_mut(dest_ebb)); - for &(pred, last_inst) in &preds { + for &(pred, _) in &preds { // For each predecessor, we query what is the local SSA value corresponding // to var and we put it as an argument of the branch instruction. let (pred_val, local_side_effects) = @@ -426,29 +426,21 @@ where match pred_values { ZeroOneOrMore::Zero() => { if pred_val != temp_arg_val { - pred_values = ZeroOneOrMore::One((pred, last_inst, pred_val)) + pred_values = ZeroOneOrMore::One(pred_val); } } - ZeroOneOrMore::One((old_pred, old_last_inst, old_val)) => { + ZeroOneOrMore::One(old_val) => { if pred_val != temp_arg_val && pred_val != old_val { - // TODO: find a way to not allocate a vector - pred_values = ZeroOneOrMore::More(vec![ - (old_pred, old_last_inst, old_val), - (pred, last_inst, pred_val), - ]); + pred_values = ZeroOneOrMore::More(); } } - ZeroOneOrMore::More(ref mut jump_args_to_append) => { - jump_args_to_append.push((pred, last_inst, pred_val)); - } - } + ZeroOneOrMore::More() => {} + }; side_effects.append(local_side_effects); } - // Now that we're done iterating, move the predecessors list back. debug_assert!(self.predecessors(dest_ebb).is_empty()); - *self.predecessors_mut(dest_ebb) = preds; - match pred_values { + let result_val = match pred_values { ZeroOneOrMore::Zero() => { // The variable is used but never defined before. This is an irregularity in the // code, but rather than throwing an error we silently initialize the variable to @@ -470,38 +462,54 @@ where panic!("value used but never declared and initialization not supported") }; side_effects.instructions_added_to_ebbs.push(dest_ebb); - (val, side_effects) + val } - ZeroOneOrMore::One((_, _, pred_val)) => { + ZeroOneOrMore::One(pred_val) => { // Here all the predecessors use a single value to represent our variable // so we don't need to have it as an ebb argument. // We need to replace all the occurences of val with pred_val but since // we can't afford a re-writing pass right now we just declare an alias. dfg.remove_ebb_arg(temp_arg_val); dfg.change_to_alias(temp_arg_val, pred_val); - (pred_val, side_effects) + pred_val } - ZeroOneOrMore::More(jump_args_to_append) => { + ZeroOneOrMore::More() => { // There is disagreement in the predecessors on which value to use so we have // to keep the ebb argument. - for (pred_block, last_inst, pred_val) in jump_args_to_append { - match self.append_jump_argument( - dfg, - layout, - last_inst, - pred_block, - dest_ebb, - pred_val, - temp_arg_var, - jts, - ) { - None => (), - Some(middle_ebb) => side_effects.split_ebbs_created.push(middle_ebb), - }; + for &mut (ref mut pred_block, ref mut last_inst) in &mut preds { + // We already did a full `use_var` above, so we can do just the fast path. + let pred_val = *self.variables + .get(temp_arg_var) + .unwrap() + .get(&pred_block) + .unwrap(); + if let Some((middle_ebb, middle_block, middle_jump_inst)) = + self.append_jump_argument( + dfg, + layout, + *last_inst, + *pred_block, + dest_ebb, + pred_val, + temp_arg_var, + jts, + ) + { + *pred_block = middle_block; + *last_inst = middle_jump_inst; + side_effects.split_ebbs_created.push(middle_ebb); + } } - (temp_arg_val, side_effects) + debug_assert!(self.predecessors(dest_ebb).is_empty()); + temp_arg_val } - } + }; + + // Now that we're done, move the predecessors list back. + debug_assert!(self.predecessors(dest_ebb).is_empty()); + *self.predecessors_mut(dest_ebb) = preds; + + (result_val, side_effects) } /// Appends a jump argument to a jump instruction, returns ebb created in case of @@ -516,7 +524,7 @@ where val: Value, var: Variable, jts: &mut JumpTables, - ) -> Option { + ) -> Option<(Ebb, Block, Inst)> { match dfg[jump_inst].analyze_branch(&dfg.value_lists) { BranchInfo::NotABranch => { panic!("you have declared a non-branch instruction as a predecessor to an ebb"); @@ -553,11 +561,8 @@ where let mut cur = Cursor::new(layout); cur.goto_bottom(middle_ebb); let middle_jump_inst = dfg.ins(&mut cur).jump(dest_ebb, &[val]); - let dest_header_block = self.header_block(dest_ebb); - self.blocks[dest_header_block].add_predecessor(block, middle_jump_inst); - self.blocks[dest_header_block].remove_predecessor(jump_inst); self.def_var(var, val, block); - Some(middle_ebb) + Some((middle_ebb, block, middle_jump_inst)) } } } From 9d2fbdae62b341ded21090730262885e00cd0086 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 1 Sep 2017 10:16:30 -0700 Subject: [PATCH 1049/3084] Avoid cloning a jump argument vector to the heap. --- lib/wasm/src/code_translator.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 62d5adcfe8..34c68bf813 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -562,7 +562,7 @@ fn translate_operator( let default_ebb = control_stack[control_stack.len() - 1 - (default as usize)] .br_destination(); builder.ins().jump(default_ebb, jump_args.as_slice()); - stack.extend(jump_args.clone()); + stack.extend_from_slice(&jump_args); for (depth, dest_ebb) in dest_ebbs { builder.switch_to_block(dest_ebb, &[]); builder.seal_block(dest_ebb); From fe12fe0e6367dd45293af25795ba85b924080e3c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 1 Sep 2017 10:12:23 -0700 Subject: [PATCH 1050/3084] Avoid unneeded calls to .as_slice(). --- lib/cretonne/src/regalloc/solver.rs | 6 ++--- lib/frontend/src/frontend.rs | 3 +-- lib/wasm/src/code_translator.rs | 37 ++++++++++------------------- lib/wasm/src/sections_translator.rs | 2 +- 4 files changed, 17 insertions(+), 31 deletions(-) diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index b99da23d43..c9215f4ebb 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -649,7 +649,7 @@ impl Solver { |v| v.from != v.to, )); - dbg!("collect_moves: {}", DisplayList(self.moves.as_slice())); + dbg!("collect_moves: {}", DisplayList(&self.moves)); } /// Try to schedule a sequence of `regmove` instructions that will shuffle registers into @@ -751,8 +751,8 @@ impl fmt::Display for Solver { " assignments: {}", DisplayList(self.assignments.as_slice()) )?; - writeln!(f, " vars: {}", DisplayList(self.vars.as_slice()))?; - writeln!(f, " moves: {}", DisplayList(self.moves.as_slice()))?; + writeln!(f, " vars: {}", DisplayList(&self.vars))?; + writeln!(f, " moves: {}", DisplayList(&self.moves))?; writeln!(f, "}}") } } diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index c896e13f8e..c32a824158 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -141,8 +141,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short } _ => panic!("should not happen"), }; - self.builder - .ebb_args_adjustement(dest_ebb, args_types.as_slice()); + self.builder.ebb_args_adjustement(dest_ebb, &args_types); self.builder.declare_successor(dest_ebb, inst); } None => { diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 34c68bf813..751f528f63 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -75,7 +75,7 @@ impl ControlStackFrame { match *self { ControlStackFrame::If { ref return_values, .. } | ControlStackFrame::Block { ref return_values, .. } | - ControlStackFrame::Loop { ref return_values, .. } => return_values.as_slice(), + ControlStackFrame::Loop { ref return_values, .. } => &return_values, } } fn following_code(&self) -> Ebb { @@ -264,7 +264,7 @@ pub fn translate_function_body( if !builder.is_filled() && (!builder.is_unreachable() || !builder.is_pristine()) { let cut_index = stack.len() - sig.return_types.len(); let return_vals = stack.split_off(cut_index); - builder.ins().return_(return_vals.as_slice()); + builder.ins().return_(&return_vals); } // Because the function has an implicit block as body, we need to explicitely close it. let frame = control_stack.pop().unwrap(); @@ -276,7 +276,7 @@ pub fn translate_function_body( stack.extend_from_slice(builder.ebb_args(frame.following_code())); let cut_index = stack.len() - sig.return_types.len(); let return_vals = stack.split_off(cut_index); - builder.ins().return_(return_vals.as_slice()); + builder.ins().return_(&return_vals); } } Ok((func, func_imports)) @@ -418,7 +418,7 @@ fn translate_operator( }; let cut_index = stack.len() - return_values.len(); let jump_args = stack.split_off(cut_index); - builder.ins().jump(destination, jump_args.as_slice()); + builder.ins().jump(destination, &jump_args); // We change the target of the branch instruction let else_ebb = builder.create_ebb(); builder.change_jump_destination(branch_inst, else_ebb); @@ -430,10 +430,7 @@ fn translate_operator( if !builder.is_unreachable() || !builder.is_pristine() { let cut_index = stack.len() - frame.return_values().len(); let jump_args = stack.split_off(cut_index); - builder.ins().jump( - frame.following_code(), - jump_args.as_slice(), - ); + builder.ins().jump(frame.following_code(), &jump_args); } builder.switch_to_block(frame.following_code(), frame.return_values()); builder.seal_block(frame.following_code()); @@ -475,10 +472,7 @@ fn translate_operator( let cut_index = stack.len() - frame.return_values().len(); stack.split_off(cut_index) }; - builder.ins().jump( - frame.br_destination(), - jump_args.as_slice(), - ); + builder.ins().jump(frame.br_destination(), &jump_args); // We signal that all the code that follows until the next End is unreachable frame.set_reachable(); state.real_unreachable_stack_depth = 1 + relative_depth as usize; @@ -493,11 +487,7 @@ fn translate_operator( let cut_index = stack.len() - frame.return_values().len(); stack.split_off(cut_index) }; - builder.ins().brnz( - val, - frame.br_destination(), - jump_args.as_slice(), - ); + builder.ins().brnz(val, frame.br_destination(), &jump_args); // The values returned by the branch are still available for the reachable // code that comes after it frame.set_reachable(); @@ -561,7 +551,7 @@ fn translate_operator( builder.ins().br_table(val, jt); let default_ebb = control_stack[control_stack.len() - 1 - (default as usize)] .br_destination(); - builder.ins().jump(default_ebb, jump_args.as_slice()); + builder.ins().jump(default_ebb, &jump_args); stack.extend_from_slice(&jump_args); for (depth, dest_ebb) in dest_ebbs { builder.switch_to_block(dest_ebb, &[]); @@ -569,7 +559,7 @@ fn translate_operator( let i = control_stack.len() - 1 - depth; let frame = &mut control_stack[i]; let real_dest_ebb = frame.br_destination(); - builder.ins().jump(real_dest_ebb, jump_args.as_slice()); + builder.ins().jump(real_dest_ebb, &jump_args); frame.set_reachable(); } state.real_unreachable_stack_depth = 1 + min_depth as usize; @@ -579,7 +569,7 @@ fn translate_operator( let return_count = sig.return_types.len(); let cut_index = stack.len() - return_count; let return_args = stack.split_off(cut_index); - builder.ins().return_(return_args.as_slice()); + builder.ins().return_(&return_args); state.real_unreachable_stack_depth = 1; } /************************************ Calls **************************************** @@ -599,10 +589,7 @@ fn translate_operator( exports, signatures, ); - let call_inst = builder.ins().call( - internal_function_index, - call_args.as_slice(), - ); + let call_inst = builder.ins().call(internal_function_index, &call_args); let ret_values = builder.inst_results(call_inst); for val in ret_values { stack.push(*val); @@ -621,7 +608,7 @@ fn translate_operator( let cut_index = stack.len() - args_num; let call_args = stack.split_off(cut_index); let ret_values = - runtime.translate_call_indirect(builder, sigref, index_val, call_args.as_slice()); + runtime.translate_call_indirect(builder, sigref, index_val, &call_args); for val in ret_values { stack.push(*val); } diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 3abb6f232a..6c97d928f7 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -369,7 +369,7 @@ pub fn parse_elements_section( ParserState::ElementSectionEntryBody(ref elements) => { let elems: Vec = elements.iter().map(|&x| x as FunctionIndex).collect(); - runtime.declare_table_elements(table_index, offset, elems.as_slice()) + runtime.declare_table_elements(table_index, offset, &elems) } ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; From d4c53935b435a863cb4b1a51d573b16082219d82 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 1 Sep 2017 10:13:41 -0700 Subject: [PATCH 1051/3084] Introduce EntitySet, and use it to replace the HashSet in the frontend. --- lib/cretonne/src/entity/mod.rs | 9 ++- lib/cretonne/src/entity/set.rs | 144 +++++++++++++++++++++++++++++++++ lib/frontend/src/frontend.rs | 7 +- 3 files changed, 154 insertions(+), 6 deletions(-) create mode 100644 lib/cretonne/src/entity/set.rs diff --git a/lib/cretonne/src/entity/mod.rs b/lib/cretonne/src/entity/mod.rs index 6702c43dab..4fc40ea56e 100644 --- a/lib/cretonne/src/entity/mod.rs +++ b/lib/cretonne/src/entity/mod.rs @@ -4,8 +4,8 @@ //! `usize` as usual, but by *entity references* which are integers wrapped in new-types. This has //! a couple advantages: //! -//! - Improved type safety. The various map types accept a specific key type, so there is no -//! confusion about the meaning of an array index, as there is with plain arrays. +//! - Improved type safety. The various map and set types accept a specific key type, so there is +//! no confusion about the meaning of an array index, as there is with plain arrays. //! - Smaller indexes. The normal `usize` index is often 64 bits which is way too large for most //! purposes. The entity reference types can be smaller, allowing for more compact data //! structures. @@ -22,6 +22,9 @@ //! number of entities. It tracks accurately which entities have been inserted. This is a //! specialized data structure which can use a lot of memory, so read the documentation before //! using it. +//! - [`EntitySet`](struct.EntitySet.html) is used to represent a secondary set of entities. +//! The set is implemented as a simple vector, so it does not keep track of which entities have +//! been inserted into the primary map. Instead, any unknown entities are not in the set. //! - [`EntityList`](struct.EntityList.html) is a compact representation of lists of entity //! references allocated from an associated memory pool. It has a much smaller footprint than //! `Vec`. @@ -31,11 +34,13 @@ mod list; mod map; mod primary; mod sparse; +mod set; pub use self::keys::Keys; pub use self::list::{EntityList, ListPool}; pub use self::map::EntityMap; pub use self::primary::PrimaryMap; +pub use self::set::EntitySet; pub use self::sparse::{SparseSet, SparseMap, SparseMapValue}; /// A type wrapping a small integer index should implement `EntityRef` so it can be used as the key diff --git a/lib/cretonne/src/entity/set.rs b/lib/cretonne/src/entity/set.rs new file mode 100644 index 0000000000..be9d76bb4e --- /dev/null +++ b/lib/cretonne/src/entity/set.rs @@ -0,0 +1,144 @@ +//! Densely numbered entity references as set keys. + +use entity::{EntityRef, Keys}; +use std::marker::PhantomData; + +/// A set of `K` for densely indexed entity references. +/// +/// The `EntitySet` data structure uses the dense index space to implement a set with a bitvector. +/// Like `EntityMap`, an `EntitySet` is used to associate secondary information with entities. +#[derive(Debug, Clone)] +pub struct EntitySet +where + K: EntityRef, +{ + elems: Vec, + len: usize, + unused: PhantomData, +} + +/// Shared `EntitySet` implementation for all value types. +impl EntitySet +where + K: EntityRef, +{ + /// Create a new empty set. + pub fn new() -> Self { + EntitySet { + elems: Vec::new(), + len: 0, + unused: PhantomData, + } + } + + /// Get the element at `k` if it exists. + pub fn contains(&self, k: K) -> bool { + let index = k.index(); + debug_assert!(index < self.len); + (self.elems[index / 8] & (1 << (index % 8))) != 0 + } + + /// Is this set completely empty? + pub fn is_empty(&self) -> bool { + self.len == 0 + } + + /// Remove all entries from this set. + pub fn clear(&mut self) { + self.len = 0; + self.elems.clear() + } + + /// Iterate over all the keys in this set. + pub fn keys(&self) -> Keys { + Keys::new(self.len) + } + + /// Resize the set to have `n` entries by adding default entries as needed. + pub fn resize(&mut self, n: usize) { + if n < self.len { + self.elems.truncate((n + 7) / 8) + } else { + // TODO: Is there a better way to grow/resize/etc.? + let additional = (n - self.len + 7) / 8; + self.elems.reserve(additional); + for _ in 0..additional { + self.elems.push(0) + } + } + self.len = n + } + + /// Insert the element at `k`. + pub fn insert(&mut self, k: K) -> bool { + let index = k.index(); + if index >= self.len { + self.resize(index + 1) + } + let result = !self.contains(k); + self.elems[index / 8] |= 1 << (index % 8); + result + } +} + +#[cfg(test)] +mod tests { + use super::*; + + // `EntityRef` impl for testing. + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + struct E(u32); + + impl EntityRef for E { + fn new(i: usize) -> Self { + E(i as u32) + } + fn index(self) -> usize { + self.0 as usize + } + } + + #[test] + fn basic() { + let r0 = E(0); + let r1 = E(1); + let r2 = E(2); + let mut m = EntitySet::new(); + + let v: Vec = m.keys().collect(); + assert_eq!(v, []); + assert!(m.is_empty()); + + m.insert(r2); + m.insert(r1); + + assert!(!m.contains(r0)); + assert!(m.contains(r1)); + assert!(m.contains(r2)); + assert!(!m.is_empty()); + + let v: Vec = m.keys().collect(); + assert_eq!(v, [r0, r1, r2]); + + m.resize(20); + assert!(!m.contains(E(3))); + assert!(!m.contains(E(4))); + assert!(!m.contains(E(8))); + assert!(!m.contains(E(15))); + assert!(!m.contains(E(19))); + + m.insert(E(8)); + m.insert(E(15)); + assert!(!m.contains(E(3))); + assert!(!m.contains(E(4))); + assert!(m.contains(E(8))); + assert!(!m.contains(E(9))); + assert!(!m.contains(E(14))); + assert!(m.contains(E(15))); + assert!(!m.contains(E(16))); + assert!(!m.contains(E(19))); + + m.clear(); + assert!(m.is_empty()); + } +} diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index c32a824158..beb5dfdcfe 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -6,8 +6,7 @@ use cretonne::ir::instructions::BranchInfo; use cretonne::ir::function::DisplayFunction; use cretonne::isa::TargetIsa; use ssa::{SSABuilder, SideEffects, Block}; -use cretonne::entity::{EntityRef, EntityMap}; -use std::collections::HashSet; +use cretonne::entity::{EntityRef, EntityMap, EntitySet}; use std::hash::Hash; /// Permanent structure used for translating into Cretonne IL. @@ -151,8 +150,8 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short if let InstructionData::BranchTable { table, .. } = data { // Unlike all other jumps/branches, jump tables are // capable of having the same successor appear -// multiple times. Use a HashSet to deduplicate. - let mut unique = HashSet::new(); +// multiple times, so we must deduplicate. + let mut unique = EntitySet::::new(); for dest_ebb in self.builder .func .jump_tables From 8237893113ac15fd99cc2e0ab95b7a4ffe81079f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 1 Sep 2017 12:50:51 -0700 Subject: [PATCH 1052/3084] Eliminate a temporary heap allocation when splitting a critical edge. --- lib/frontend/src/ssa.rs | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index a1b9d56de9..2cabe2cedf 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -539,24 +539,15 @@ where // In the case of a jump table, the situation is tricky because br_table doesn't // support arguments. // We have to split the critical edge - let indexes: Vec = jts[jt].entries().fold( - Vec::new(), - |mut acc, (index, dest)| if dest == - dest_ebb - { - acc.push(index); - acc - } else { - acc - }, - ); let middle_ebb = dfg.make_ebb(); layout.append_ebb(middle_ebb); let block = self.declare_ebb_header_block(middle_ebb); self.blocks[block].add_predecessor(jump_inst_block, jump_inst); self.seal_ebb_header_block(middle_ebb, dfg, layout, jts); - for index in indexes { - jts[jt].set_entry(index, middle_ebb) + for old_dest in jts[jt].as_mut_slice() { + if old_dest.unwrap() == dest_ebb { + *old_dest = PackedOption::from(middle_ebb); + } } let mut cur = Cursor::new(layout); cur.goto_bottom(middle_ebb); From 52cbbcd069d28fdd3d5879e6781727085f8c5c0b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 1 Sep 2017 13:06:05 -0700 Subject: [PATCH 1053/3084] Add a CtonError::InvalidInput variant. This error code will be used to complain when a WebAssembly binary code can't be parsed. --- cranelift/src/wasm.rs | 2 ++ lib/cretonne/src/result.rs | 9 +++++++++ 2 files changed, 11 insertions(+) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index c54cfa85d0..c418c3cc2c 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -195,6 +195,7 @@ fn handle_module( CtonError::Verifier(err) => { return Err(pretty_verifier_error(&context.func, None, err)); } + CtonError::InvalidInput | CtonError::ImplLimitExceeded | CtonError::CodeTooLarge => return Err(String::from(error.description())), } @@ -207,6 +208,7 @@ fn handle_module( CtonError::Verifier(err) => { return Err(pretty_verifier_error(&context.func, None, err)); } + CtonError::InvalidInput | CtonError::ImplLimitExceeded | CtonError::CodeTooLarge => return Err(String::from(error.description())), } diff --git a/lib/cretonne/src/result.rs b/lib/cretonne/src/result.rs index f4d7316fb5..b5e9861f95 100644 --- a/lib/cretonne/src/result.rs +++ b/lib/cretonne/src/result.rs @@ -9,6 +9,12 @@ use std::fmt; /// When Cretonne fails to compile a function, it will return one of these error codes. #[derive(Debug, PartialEq, Eq)] pub enum CtonError { + /// The input is invalid. + /// + /// This error code is used by a WebAssembly translator when it encounters invalid WebAssembly + /// code. This should never happen for validated WebAssembly code. + InvalidInput, + /// An IL verifier error. /// /// This always represents a bug, either in the code that generated IL for Cretonne, or a bug @@ -37,6 +43,7 @@ impl fmt::Display for CtonError { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match *self { CtonError::Verifier(ref e) => write!(f, "Verifier error: {}", e), + CtonError::InvalidInput | CtonError::ImplLimitExceeded | CtonError::CodeTooLarge => f.write_str(self.description()), } @@ -46,6 +53,7 @@ impl fmt::Display for CtonError { impl StdError for CtonError { fn description(&self) -> &str { match *self { + CtonError::InvalidInput => "Invalid input code", CtonError::Verifier(ref e) => &e.message, CtonError::ImplLimitExceeded => "Implementation limit exceeded", CtonError::CodeTooLarge => "Code for function is too large", @@ -54,6 +62,7 @@ impl StdError for CtonError { fn cause(&self) -> Option<&StdError> { match *self { CtonError::Verifier(ref e) => Some(e), + CtonError::InvalidInput | CtonError::ImplLimitExceeded | CtonError::CodeTooLarge => None, } From de92b2b9676e30dfea9eb1c9f7f1d2028d0d7788 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 1 Sep 2017 13:20:03 -0700 Subject: [PATCH 1054/3084] Use Vec::resize. --- lib/cretonne/src/entity/set.rs | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/lib/cretonne/src/entity/set.rs b/lib/cretonne/src/entity/set.rs index be9d76bb4e..6d8d88fadd 100644 --- a/lib/cretonne/src/entity/set.rs +++ b/lib/cretonne/src/entity/set.rs @@ -56,16 +56,7 @@ where /// Resize the set to have `n` entries by adding default entries as needed. pub fn resize(&mut self, n: usize) { - if n < self.len { - self.elems.truncate((n + 7) / 8) - } else { - // TODO: Is there a better way to grow/resize/etc.? - let additional = (n - self.len + 7) / 8; - self.elems.reserve(additional); - for _ in 0..additional { - self.elems.push(0) - } - } + self.elems.resize((n + 7) / 8, 0); self.len = n } From 0c2ab06e6601b0b2428e1384f15997015ef3ac4e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 1 Sep 2017 13:25:37 -0700 Subject: [PATCH 1055/3084] Make EntitySet::contains return false for out-of-bounds indices. This is consistent with EntityMap. --- lib/cretonne/src/entity/set.rs | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/entity/set.rs b/lib/cretonne/src/entity/set.rs index 6d8d88fadd..b1c351b650 100644 --- a/lib/cretonne/src/entity/set.rs +++ b/lib/cretonne/src/entity/set.rs @@ -34,8 +34,11 @@ where /// Get the element at `k` if it exists. pub fn contains(&self, k: K) -> bool { let index = k.index(); - debug_assert!(index < self.len); - (self.elems[index / 8] & (1 << (index % 8))) != 0 + if index < self.len { + (self.elems[index / 8] & (1 << (index % 8))) != 0 + } else { + false + } } /// Is this set completely empty? @@ -106,6 +109,7 @@ mod tests { assert!(!m.contains(r0)); assert!(m.contains(r1)); assert!(m.contains(r2)); + assert!(!m.contains(E(3))); assert!(!m.is_empty()); let v: Vec = m.keys().collect(); @@ -128,6 +132,8 @@ mod tests { assert!(m.contains(E(15))); assert!(!m.contains(E(16))); assert!(!m.contains(E(19))); + assert!(!m.contains(E(20))); + assert!(!m.contains(E(u32::max_value()))); m.clear(); assert!(m.is_empty()); From 67621948013d186fb5cb9ccb6b922e054425bafc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 1 Sep 2017 14:27:25 -0700 Subject: [PATCH 1056/3084] Move stack and control_stack into TranslationState. This reduces the number of function arguments passed around and it keeps related information together. --- lib/wasm/src/code_translator.rs | 63 ++++++++++++++++++--------------- 1 file changed, 35 insertions(+), 28 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 751f528f63..48e689dbef 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -126,13 +126,27 @@ impl ControlStackFrame { /// Contains information passed along during the translation and that records: /// -/// - the depth of the two unreachable control blocks stacks, that are manipulated when translating +/// - The current value and control stacks. +/// - The depth of the two unreachable control blocks stacks, that are manipulated when translating /// unreachable code; struct TranslationState { + stack: Vec, + control_stack: Vec, phantom_unreachable_stack_depth: usize, real_unreachable_stack_depth: usize, } +impl TranslationState { + fn new() -> TranslationState { + TranslationState { + stack: Vec::new(), + control_stack: Vec::new(), + phantom_unreachable_stack_depth: 0, + real_unreachable_stack_depth: 0, + } + } +} + /// Holds mappings between the function and signatures indexes in the Wasm module and their /// references as imports of the Cretonne IL function. #[derive(Clone, Debug)] @@ -175,8 +189,6 @@ pub fn translate_function_body( } } let mut func_imports = FunctionImports::new(); - let mut stack: Vec = Vec::new(); - let mut control_stack: Vec = Vec::new(); // We introduce an arbitrary scope for the FunctionBuilder object { let mut builder = FunctionBuilder::new(&mut func, il_builder); @@ -206,13 +218,12 @@ pub fn translate_function_body( local_index += 1; } } - let mut state = TranslationState { - phantom_unreachable_stack_depth: 0, - real_unreachable_stack_depth: 0, - }; + + let mut state = TranslationState::new(); + // We initialize the control stack with the implicit function block let end_ebb = builder.create_ebb(); - control_stack.push(ControlStackFrame::Block { + state.control_stack.push(ControlStackFrame::Block { destination: end_ebb, original_stack_size: 0, return_values: sig.return_types @@ -231,20 +242,12 @@ pub fn translate_function_body( state.real_unreachable_stack_depth > 0 ); if state.real_unreachable_stack_depth > 0 { - translate_unreachable_operator( - op, - &mut builder, - &mut stack, - &mut control_stack, - &mut state, - ) + translate_unreachable_operator(op, &mut builder, &mut state) } else { translate_operator( op, &mut builder, runtime, - &mut stack, - &mut control_stack, &mut state, sig, functions, @@ -262,20 +265,22 @@ pub fn translate_function_body( // In WebAssembly, the final return instruction is implicit so we need to build it // explicitely in Cretonne IL. if !builder.is_filled() && (!builder.is_unreachable() || !builder.is_pristine()) { - let cut_index = stack.len() - sig.return_types.len(); - let return_vals = stack.split_off(cut_index); + let cut_index = state.stack.len() - sig.return_types.len(); + let return_vals = state.stack.split_off(cut_index); builder.ins().return_(&return_vals); } // Because the function has an implicit block as body, we need to explicitely close it. - let frame = control_stack.pop().unwrap(); + let frame = state.control_stack.pop().unwrap(); builder.switch_to_block(frame.following_code(), frame.return_values()); builder.seal_block(frame.following_code()); // If the block is reachable we also have to include a return instruction in it. if !builder.is_unreachable() { - stack.truncate(frame.original_stack_size()); - stack.extend_from_slice(builder.ebb_args(frame.following_code())); - let cut_index = stack.len() - sig.return_types.len(); - let return_vals = stack.split_off(cut_index); + state.stack.truncate(frame.original_stack_size()); + state.stack.extend_from_slice( + builder.ebb_args(frame.following_code()), + ); + let cut_index = state.stack.len() - sig.return_types.len(); + let return_vals = state.stack.split_off(cut_index); builder.ins().return_(&return_vals); } } @@ -288,8 +293,6 @@ fn translate_operator( op: &Operator, builder: &mut FunctionBuilder, runtime: &mut WasmRuntime, - stack: &mut Vec, - control_stack: &mut Vec, state: &mut TranslationState, sig: &Signature, functions: &[SignatureIndex], @@ -297,6 +300,9 @@ fn translate_operator( exports: &Option>, func_imports: &mut FunctionImports, ) { + let stack = &mut state.stack; + let control_stack = &mut state.control_stack; + // This big match treats all Wasm code operators. match *op { /********************************** Locals **************************************** @@ -1197,10 +1203,11 @@ fn translate_operator( fn translate_unreachable_operator( op: &Operator, builder: &mut FunctionBuilder, - stack: &mut Vec, - control_stack: &mut Vec, state: &mut TranslationState, ) { + let stack = &mut state.stack; + let control_stack = &mut state.control_stack; + // We don't translate because the code is unreachable // Nevertheless we have to record a phantom stack for this code // to know when the unreachable code ends From 03939e4f9f735238cb154bbef395e2d9012fcb71 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 1 Sep 2017 15:06:27 -0700 Subject: [PATCH 1057/3084] Move TranslationState into its own module. Add some convenience methods for common state access patterns. --- lib/wasm/src/code_translator.rs | 662 ++++++++++++-------------------- lib/wasm/src/lib.rs | 5 +- lib/wasm/src/state.rs | 180 +++++++++ 3 files changed, 429 insertions(+), 418 deletions(-) create mode 100644 lib/wasm/src/state.rs diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 48e689dbef..83b123e8f6 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -21,8 +21,8 @@ //! //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as //! argument. -use cretonne::ir::{Function, Signature, Value, Type, InstBuilder, FunctionName, Ebb, FuncRef, - SigRef, ExtFuncData, Inst, MemFlags}; +use cretonne::ir::{Function, Signature, Type, InstBuilder, FunctionName, Ebb, FuncRef, SigRef, + ExtFuncData, MemFlags}; use cretonne::ir::types::*; use cretonne::ir::immediates::{Ieee32, Ieee64, Offset32}; use cretonne::ir::condcodes::{IntCC, FloatCC}; @@ -30,123 +30,12 @@ use cton_frontend::{ILBuilder, FunctionBuilder}; use wasmparser::{Parser, ParserState, Operator, WasmDecoder, MemoryImmediate}; use translation_utils::{f32_translation, f64_translation, type_to_type, translate_type, Local, GlobalIndex, FunctionIndex, SignatureIndex}; +use state::{TranslationState, ControlStackFrame}; use std::collections::HashMap; use runtime::WasmRuntime; use std::u32; -/// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following -/// fields: -/// -/// - `destination`: reference to the `Ebb` that will hold the code after the control block; -/// - `return_values`: types of the values returned by the control block; -/// - `original_stack_size`: size of the value stack at the beginning of the control block. -/// -/// Moreover, the `if` frame has the `branch_inst` field that points to the `brz` instruction -/// separating the `true` and `false` branch. The `loop` frame has a `header` field that references -/// the `Ebb` that contains the beginning of the body of the loop. -#[derive(Debug)] -enum ControlStackFrame { - If { - destination: Ebb, - branch_inst: Inst, - return_values: Vec, - original_stack_size: usize, - reachable: bool, - }, - Block { - destination: Ebb, - return_values: Vec, - original_stack_size: usize, - reachable: bool, - }, - Loop { - destination: Ebb, - header: Ebb, - return_values: Vec, - original_stack_size: usize, - reachable: bool, - }, -} - -/// Helper methods for the control stack objects. -impl ControlStackFrame { - fn return_values(&self) -> &[Type] { - match *self { - ControlStackFrame::If { ref return_values, .. } | - ControlStackFrame::Block { ref return_values, .. } | - ControlStackFrame::Loop { ref return_values, .. } => &return_values, - } - } - fn following_code(&self) -> Ebb { - match *self { - ControlStackFrame::If { destination, .. } | - ControlStackFrame::Block { destination, .. } | - ControlStackFrame::Loop { destination, .. } => destination, - } - } - fn br_destination(&self) -> Ebb { - match *self { - ControlStackFrame::If { destination, .. } | - ControlStackFrame::Block { destination, .. } => destination, - ControlStackFrame::Loop { header, .. } => header, - } - } - fn original_stack_size(&self) -> usize { - match *self { - ControlStackFrame::If { original_stack_size, .. } | - ControlStackFrame::Block { original_stack_size, .. } | - ControlStackFrame::Loop { original_stack_size, .. } => original_stack_size, - } - } - fn is_loop(&self) -> bool { - match *self { - ControlStackFrame::If { .. } | - ControlStackFrame::Block { .. } => false, - ControlStackFrame::Loop { .. } => true, - } - } - - fn is_reachable(&self) -> bool { - match *self { - ControlStackFrame::If { reachable, .. } | - ControlStackFrame::Block { reachable, .. } | - ControlStackFrame::Loop { reachable, .. } => reachable, - } - } - - fn set_reachable(&mut self) { - match *self { - ControlStackFrame::If { ref mut reachable, .. } | - ControlStackFrame::Block { ref mut reachable, .. } | - ControlStackFrame::Loop { ref mut reachable, .. } => *reachable = true, - } - } -} - -/// Contains information passed along during the translation and that records: -/// -/// - The current value and control stacks. -/// - The depth of the two unreachable control blocks stacks, that are manipulated when translating -/// unreachable code; -struct TranslationState { - stack: Vec, - control_stack: Vec, - phantom_unreachable_stack_depth: usize, - real_unreachable_stack_depth: usize, -} - -impl TranslationState { - fn new() -> TranslationState { - TranslationState { - stack: Vec::new(), - control_stack: Vec::new(), - phantom_unreachable_stack_depth: 0, - real_unreachable_stack_depth: 0, - } - } -} - /// Holds mappings between the function and signatures indexes in the Wasm module and their /// references as imports of the Cretonne IL function. #[derive(Clone, Debug)] @@ -223,15 +112,13 @@ pub fn translate_function_body( // We initialize the control stack with the implicit function block let end_ebb = builder.create_ebb(); - state.control_stack.push(ControlStackFrame::Block { - destination: end_ebb, - original_stack_size: 0, - return_values: sig.return_types + state.push_block( + end_ebb, + sig.return_types .iter() .map(|argty| argty.value_type) .collect(), - reachable: false, - }); + ); // Now the main loop that reads every wasm instruction and translates it loop { let parser_state = parser.read(); @@ -300,46 +187,41 @@ fn translate_operator( exports: &Option>, func_imports: &mut FunctionImports, ) { - let stack = &mut state.stack; - let control_stack = &mut state.control_stack; - // This big match treats all Wasm code operators. match *op { /********************************** Locals **************************************** * `get_local` and `set_local` are treated as non-SSA variables and will completely * diseappear in the Cretonne Code ***********************************************************************************/ - Operator::GetLocal { local_index } => stack.push(builder.use_var(Local(local_index))), + Operator::GetLocal { local_index } => state.push1(builder.use_var(Local(local_index))), Operator::SetLocal { local_index } => { - let val = stack.pop().unwrap(); + let val = state.pop1(); builder.def_var(Local(local_index), val); } Operator::TeeLocal { local_index } => { - let val = stack.last().unwrap(); - builder.def_var(Local(local_index), *val); + let val = state.peek1(); + builder.def_var(Local(local_index), val); } /********************************** Globals **************************************** * `get_global` and `set_global` are handled by the runtime. ***********************************************************************************/ Operator::GetGlobal { global_index } => { let val = runtime.translate_get_global(builder, global_index as GlobalIndex); - stack.push(val); + state.push1(val); } Operator::SetGlobal { global_index } => { - let val = stack.pop().unwrap(); + let val = state.pop1(); runtime.translate_set_global(builder, global_index as GlobalIndex, val); } /********************************* Stack misc *************************************** * `drop`, `nop`, `unreachable` and `select`. ***********************************************************************************/ Operator::Drop => { - stack.pop(); + state.pop1(); } Operator::Select => { - let cond = stack.pop().unwrap(); - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().select(cond, arg2, arg1)); + let (arg1, arg2, cond) = state.pop3(); + state.push1(builder.ins().select(cond, arg2, arg1)); } Operator::Nop => { // We do nothing @@ -364,12 +246,7 @@ fn translate_operator( if let Ok(ty_cre) = type_to_type(&ty) { builder.append_ebb_arg(next, ty_cre); } - control_stack.push(ControlStackFrame::Block { - destination: next, - return_values: translate_type(ty).unwrap(), - original_stack_size: stack.len(), - reachable: false, - }); + state.push_block(next, translate_type(ty).unwrap()); } Operator::Loop { ty } => { let loop_body = builder.create_ebb(); @@ -378,17 +255,11 @@ fn translate_operator( builder.append_ebb_arg(next, ty_cre); } builder.ins().jump(loop_body, &[]); - control_stack.push(ControlStackFrame::Loop { - destination: next, - header: loop_body, - return_values: translate_type(ty).unwrap(), - original_stack_size: stack.len(), - reachable: false, - }); + state.push_loop(loop_body, next, translate_type(ty).unwrap()); builder.switch_to_block(loop_body, &[]); } Operator::If { ty } => { - let val = stack.pop().unwrap(); + let val = state.pop1(); let if_not = builder.create_ebb(); let jump_inst = builder.ins().brz(val, if_not, &[]); // Here we append an argument to an Ebb targeted by an argumentless jump instruction @@ -400,20 +271,14 @@ fn translate_operator( if let Ok(ty_cre) = type_to_type(&ty) { builder.append_ebb_arg(if_not, ty_cre); } - control_stack.push(ControlStackFrame::If { - destination: if_not, - branch_inst: jump_inst, - return_values: translate_type(ty).unwrap(), - original_stack_size: stack.len(), - reachable: false, - }); + state.push_if(jump_inst, if_not, translate_type(ty).unwrap()); } Operator::Else => { // We take the control frame pushed by the if, use its ebb as the else body // and push a new control frame with a new ebb for the code after the if/then/else // At the end of the then clause we jump to the destination - let i = control_stack.len() - 1; - let (destination, return_values, branch_inst) = match control_stack[i] { + let i = state.control_stack.len() - 1; + let (destination, return_values, branch_inst) = match state.control_stack[i] { ControlStackFrame::If { destination, ref return_values, @@ -422,8 +287,8 @@ fn translate_operator( } => (destination, return_values, branch_inst), _ => panic!("should not happen"), }; - let cut_index = stack.len() - return_values.len(); - let jump_args = stack.split_off(cut_index); + let cut_index = state.stack.len() - return_values.len(); + let jump_args = state.stack.split_off(cut_index); builder.ins().jump(destination, &jump_args); // We change the target of the branch instruction let else_ebb = builder.create_ebb(); @@ -432,10 +297,10 @@ fn translate_operator( builder.switch_to_block(else_ebb, &[]); } Operator::End => { - let frame = control_stack.pop().unwrap(); + let frame = state.control_stack.pop().unwrap(); if !builder.is_unreachable() || !builder.is_pristine() { - let cut_index = stack.len() - frame.return_values().len(); - let jump_args = stack.split_off(cut_index); + let cut_index = state.stack.len() - frame.return_values().len(); + let jump_args = state.stack.split_off(cut_index); builder.ins().jump(frame.following_code(), &jump_args); } builder.switch_to_block(frame.following_code(), frame.return_values()); @@ -445,8 +310,10 @@ fn translate_operator( ControlStackFrame::Loop { header, .. } => builder.seal_block(header), _ => {} } - stack.truncate(frame.original_stack_size()); - stack.extend_from_slice(builder.ebb_args(frame.following_code())); + state.stack.truncate(frame.original_stack_size()); + state.stack.extend_from_slice( + builder.ebb_args(frame.following_code()), + ); } /**************************** Branch instructions ********************************* * The branch instructions all have as arguments a target nesting level, which @@ -470,13 +337,13 @@ fn translate_operator( * `br_table`. ***********************************************************************************/ Operator::Br { relative_depth } => { - let i = control_stack.len() - 1 - (relative_depth as usize); - let frame = &mut control_stack[i]; + let i = state.control_stack.len() - 1 - (relative_depth as usize); + let frame = &mut state.control_stack[i]; let jump_args = if frame.is_loop() { Vec::new() } else { - let cut_index = stack.len() - frame.return_values().len(); - stack.split_off(cut_index) + let cut_index = state.stack.len() - frame.return_values().len(); + state.stack.split_off(cut_index) }; builder.ins().jump(frame.br_destination(), &jump_args); // We signal that all the code that follows until the next End is unreachable @@ -484,20 +351,20 @@ fn translate_operator( state.real_unreachable_stack_depth = 1 + relative_depth as usize; } Operator::BrIf { relative_depth } => { - let val = stack.pop().unwrap(); - let i = control_stack.len() - 1 - (relative_depth as usize); - let frame = &mut control_stack[i]; + let val = state.pop1(); + let i = state.control_stack.len() - 1 - (relative_depth as usize); + let frame = &mut state.control_stack[i]; let jump_args = if frame.is_loop() { Vec::new() } else { - let cut_index = stack.len() - frame.return_values().len(); - stack.split_off(cut_index) + let cut_index = state.stack.len() - frame.return_values().len(); + state.stack.split_off(cut_index) }; builder.ins().brnz(val, frame.br_destination(), &jump_args); // The values returned by the branch are still available for the reachable // code that comes after it frame.set_reachable(); - stack.extend(jump_args); + state.stack.extend(jump_args); } Operator::BrTable { ref table } => { let (depths, default) = table.read_table(); @@ -508,8 +375,8 @@ fn translate_operator( } } let jump_args_count = { - let i = control_stack.len() - 1 - (min_depth as usize); - let min_depth_frame = &control_stack[i]; + let i = state.control_stack.len() - 1 - (min_depth as usize); + let min_depth_frame = &state.control_stack[i]; if min_depth_frame.is_loop() { 0 } else { @@ -518,18 +385,18 @@ fn translate_operator( }; if jump_args_count == 0 { // No jump arguments - let val = stack.pop().unwrap(); + let val = state.pop1(); let jt = builder.create_jump_table(); for (index, depth) in depths.iter().enumerate() { - let i = control_stack.len() - 1 - (*depth as usize); - let frame = &mut control_stack[i]; + let i = state.control_stack.len() - 1 - (*depth as usize); + let frame = &mut state.control_stack[i]; let ebb = frame.br_destination(); builder.insert_jump_table_entry(jt, index, ebb); frame.set_reachable(); } builder.ins().br_table(val, jt); - let i = control_stack.len() - 1 - (default as usize); - let frame = &mut control_stack[i]; + let i = state.control_stack.len() - 1 - (default as usize); + let frame = &mut state.control_stack[i]; let ebb = frame.br_destination(); builder.ins().jump(ebb, &[]); state.real_unreachable_stack_depth = 1 + min_depth as usize; @@ -537,9 +404,9 @@ fn translate_operator( } else { // Here we have jump arguments, but Cretonne's br_table doesn't support them // We then proceed to split the edges going out of the br_table - let val = stack.pop().unwrap(); - let cut_index = stack.len() - jump_args_count; - let jump_args = stack.split_off(cut_index); + let val = state.pop1(); + let cut_index = state.stack.len() - jump_args_count; + let jump_args = state.stack.split_off(cut_index); let jt = builder.create_jump_table(); let dest_ebbs: HashMap = depths.iter().enumerate().fold(HashMap::new(), |mut acc, @@ -555,15 +422,16 @@ fn translate_operator( acc }); builder.ins().br_table(val, jt); - let default_ebb = control_stack[control_stack.len() - 1 - (default as usize)] + let default_ebb = state.control_stack[state.control_stack.len() - 1 - + (default as usize)] .br_destination(); builder.ins().jump(default_ebb, &jump_args); - stack.extend_from_slice(&jump_args); + state.stack.extend_from_slice(&jump_args); for (depth, dest_ebb) in dest_ebbs { builder.switch_to_block(dest_ebb, &[]); builder.seal_block(dest_ebb); - let i = control_stack.len() - 1 - depth; - let frame = &mut control_stack[i]; + let i = state.control_stack.len() - 1 - depth; + let frame = &mut state.control_stack[i]; let real_dest_ebb = frame.br_destination(); builder.ins().jump(real_dest_ebb, &jump_args); frame.set_reachable(); @@ -573,8 +441,8 @@ fn translate_operator( } Operator::Return => { let return_count = sig.return_types.len(); - let cut_index = stack.len() - return_count; - let return_args = stack.split_off(cut_index); + let cut_index = state.stack.len() - return_count; + let return_args = state.stack.split_off(cut_index); builder.ins().return_(&return_args); state.real_unreachable_stack_depth = 1; } @@ -585,8 +453,8 @@ fn translate_operator( ************************************************************************************/ Operator::Call { function_index } => { let args_num = args_count(function_index as usize, functions, signatures); - let cut_index = stack.len() - args_num; - let call_args = stack.split_off(cut_index); + let cut_index = state.stack.len() - args_num; + let call_args = state.stack.split_off(cut_index); let internal_function_index = find_function_import( function_index as usize, builder, @@ -598,7 +466,7 @@ fn translate_operator( let call_inst = builder.ins().call(internal_function_index, &call_args); let ret_values = builder.inst_results(call_inst); for val in ret_values { - stack.push(*val); + state.push1(*val); } } Operator::CallIndirect { @@ -610,13 +478,13 @@ fn translate_operator( // TODO: have runtime support for tables let sigref = find_signature_import(index as usize, builder, func_imports, signatures); let args_num = builder.signature(sigref).unwrap().argument_types.len(); - let index_val = stack.pop().unwrap(); - let cut_index = stack.len() - args_num; - let call_args = stack.split_off(cut_index); + let index_val = state.pop1(); + let cut_index = state.stack.len() - args_num; + let call_args = state.stack.split_off(cut_index); let ret_values = runtime.translate_call_indirect(builder, sigref, index_val, &call_args); for val in ret_values { - stack.push(*val); + state.push1(*val); } } /******************************* Memory management *********************************** @@ -624,11 +492,11 @@ fn translate_operator( * special functions. ************************************************************************************/ Operator::GrowMemory { reserved: _ } => { - let val = stack.pop().unwrap(); - stack.push(runtime.translate_grow_memory(builder, val)); + let val = state.pop1(); + state.push1(runtime.translate_grow_memory(builder, val)); } Operator::CurrentMemory { reserved: _ } => { - stack.push(runtime.translate_current_memory(builder)); + state.push1(runtime.translate_current_memory(builder)); } /******************************* Load instructions *********************************** * Wasm specifies an integer alignment flag but we drop it in Cretonne. @@ -636,130 +504,130 @@ fn translate_operator( * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not ************************************************************************************/ Operator::I32Load8U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().uload8(I32, memflags, addr, memoffset)) + state.push1(builder.ins().uload8(I32, memflags, addr, memoffset)) } Operator::I32Load16U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().uload8(I32, memflags, addr, memoffset)) + state.push1(builder.ins().uload8(I32, memflags, addr, memoffset)) } Operator::I32Load8S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().sload8(I32, memflags, addr, memoffset)) + state.push1(builder.ins().sload8(I32, memflags, addr, memoffset)) } Operator::I32Load16S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().sload8(I32, memflags, addr, memoffset)) + state.push1(builder.ins().sload8(I32, memflags, addr, memoffset)) } Operator::I64Load8U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().uload8(I64, memflags, addr, memoffset)) + state.push1(builder.ins().uload8(I64, memflags, addr, memoffset)) } Operator::I64Load16U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().uload16(I64, memflags, addr, memoffset)) + state.push1(builder.ins().uload16(I64, memflags, addr, memoffset)) } Operator::I64Load8S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().sload8(I64, memflags, addr, memoffset)) + state.push1(builder.ins().sload8(I64, memflags, addr, memoffset)) } Operator::I64Load16S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().sload16(I64, memflags, addr, memoffset)) + state.push1(builder.ins().sload16(I64, memflags, addr, memoffset)) } Operator::I64Load32S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().sload32(memflags, addr, memoffset)) + state.push1(builder.ins().sload32(memflags, addr, memoffset)) } Operator::I64Load32U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().uload32(memflags, addr, memoffset)) + state.push1(builder.ins().uload32(memflags, addr, memoffset)) } Operator::I32Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().load(I32, memflags, addr, memoffset)) + state.push1(builder.ins().load(I32, memflags, addr, memoffset)) } Operator::F32Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().load(F32, memflags, addr, memoffset)) + state.push1(builder.ins().load(F32, memflags, addr, memoffset)) } Operator::I64Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().load(I64, memflags, addr, memoffset)) + state.push1(builder.ins().load(I64, memflags, addr, memoffset)) } Operator::F64Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = stack.pop().unwrap(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); let memflags = MemFlags::new(); let memoffset = Offset32::new(offset as i32); - stack.push(builder.ins().load(F64, memflags, addr, memoffset)) + state.push1(builder.ins().load(F64, memflags, addr, memoffset)) } /****************************** Store instructions *********************************** * Wasm specifies an integer alignment flag but we drop it in Cretonne. @@ -770,8 +638,8 @@ fn translate_operator( Operator::I64Store { memory_immediate: MemoryImmediate { flags: _, offset } } | Operator::F32Store { memory_immediate: MemoryImmediate { flags: _, offset } } | Operator::F64Store { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let val = stack.pop().unwrap(); - let address_i32 = stack.pop().unwrap(); + let val = state.pop1(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); @@ -781,8 +649,8 @@ fn translate_operator( } Operator::I32Store8 { memory_immediate: MemoryImmediate { flags: _, offset } } | Operator::I64Store8 { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let val = stack.pop().unwrap(); - let address_i32 = stack.pop().unwrap(); + let val = state.pop1(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); @@ -792,8 +660,8 @@ fn translate_operator( } Operator::I32Store16 { memory_immediate: MemoryImmediate { flags: _, offset } } | Operator::I64Store16 { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let val = stack.pop().unwrap(); - let address_i32 = stack.pop().unwrap(); + let val = state.pop1(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); @@ -802,8 +670,8 @@ fn translate_operator( builder.ins().istore16(memflags, val, addr, memoffset); } Operator::I64Store32 { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let val = stack.pop().unwrap(); - let address_i32 = stack.pop().unwrap(); + let val = state.pop1(); + let address_i32 = state.pop1(); let base = runtime.translate_memory_base_address(builder, 0); let address_i64 = builder.ins().uextend(I64, address_i32); let addr = builder.ins().iadd(base, address_i64); @@ -812,387 +680,349 @@ fn translate_operator( builder.ins().istore32(memflags, val, addr, memoffset); } /****************************** Nullary Operators ************************************/ - Operator::I32Const { value } => stack.push(builder.ins().iconst(I32, value as i64)), - Operator::I64Const { value } => stack.push(builder.ins().iconst(I64, value)), + Operator::I32Const { value } => state.push1(builder.ins().iconst(I32, value as i64)), + Operator::I64Const { value } => state.push1(builder.ins().iconst(I64, value)), Operator::F32Const { value } => { - stack.push(builder.ins().f32const(f32_translation(value))); + state.push1(builder.ins().f32const(f32_translation(value))); } Operator::F64Const { value } => { - stack.push(builder.ins().f64const(f64_translation(value))); + state.push1(builder.ins().f64const(f64_translation(value))); } /******************************* Unary Operators *************************************/ Operator::I32Clz => { - let arg = stack.pop().unwrap(); + let arg = state.pop1(); let val = builder.ins().clz(arg); - stack.push(builder.ins().sextend(I32, val)); + state.push1(builder.ins().sextend(I32, val)); } Operator::I64Clz => { - let arg = stack.pop().unwrap(); + let arg = state.pop1(); let val = builder.ins().clz(arg); - stack.push(builder.ins().sextend(I64, val)); + state.push1(builder.ins().sextend(I64, val)); } Operator::I32Ctz => { - let val = stack.pop().unwrap(); + let val = state.pop1(); let short_res = builder.ins().ctz(val); - stack.push(builder.ins().sextend(I32, short_res)); + state.push1(builder.ins().sextend(I32, short_res)); } Operator::I64Ctz => { - let val = stack.pop().unwrap(); + let val = state.pop1(); let short_res = builder.ins().ctz(val); - stack.push(builder.ins().sextend(I64, short_res)); + state.push1(builder.ins().sextend(I64, short_res)); } Operator::I32Popcnt => { - let arg = stack.pop().unwrap(); + let arg = state.pop1(); let val = builder.ins().popcnt(arg); - stack.push(builder.ins().sextend(I32, val)); + state.push1(builder.ins().sextend(I32, val)); } Operator::I64Popcnt => { - let arg = stack.pop().unwrap(); + let arg = state.pop1(); let val = builder.ins().popcnt(arg); - stack.push(builder.ins().sextend(I64, val)); + state.push1(builder.ins().sextend(I64, val)); } Operator::I64ExtendSI32 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().sextend(I64, val)); + let val = state.pop1(); + state.push1(builder.ins().sextend(I64, val)); } Operator::I64ExtendUI32 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().uextend(I64, val)); + let val = state.pop1(); + state.push1(builder.ins().uextend(I64, val)); } Operator::I32WrapI64 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().ireduce(I32, val)); + let val = state.pop1(); + state.push1(builder.ins().ireduce(I32, val)); } Operator::F32Sqrt | Operator::F64Sqrt => { - let arg = stack.pop().unwrap(); - stack.push(builder.ins().sqrt(arg)); + let arg = state.pop1(); + state.push1(builder.ins().sqrt(arg)); } Operator::F32Ceil | Operator::F64Ceil => { - let arg = stack.pop().unwrap(); - stack.push(builder.ins().ceil(arg)); + let arg = state.pop1(); + state.push1(builder.ins().ceil(arg)); } Operator::F32Floor | Operator::F64Floor => { - let arg = stack.pop().unwrap(); - stack.push(builder.ins().floor(arg)); + let arg = state.pop1(); + state.push1(builder.ins().floor(arg)); } Operator::F32Trunc | Operator::F64Trunc => { - let arg = stack.pop().unwrap(); - stack.push(builder.ins().trunc(arg)); + let arg = state.pop1(); + state.push1(builder.ins().trunc(arg)); } Operator::F32Nearest | Operator::F64Nearest => { - let arg = stack.pop().unwrap(); - stack.push(builder.ins().nearest(arg)); + let arg = state.pop1(); + state.push1(builder.ins().nearest(arg)); } Operator::F32Abs | Operator::F64Abs => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().fabs(val)); + let val = state.pop1(); + state.push1(builder.ins().fabs(val)); } Operator::F32Neg | Operator::F64Neg => { - let arg = stack.pop().unwrap(); - stack.push(builder.ins().fneg(arg)); + let arg = state.pop1(); + state.push1(builder.ins().fneg(arg)); } Operator::F64ConvertUI64 | Operator::F64ConvertUI32 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().fcvt_from_uint(F64, val)); + let val = state.pop1(); + state.push1(builder.ins().fcvt_from_uint(F64, val)); } Operator::F64ConvertSI64 | Operator::F64ConvertSI32 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().fcvt_from_sint(F64, val)); + let val = state.pop1(); + state.push1(builder.ins().fcvt_from_sint(F64, val)); } Operator::F32ConvertSI64 | Operator::F32ConvertSI32 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().fcvt_from_sint(F32, val)); + let val = state.pop1(); + state.push1(builder.ins().fcvt_from_sint(F32, val)); } Operator::F32ConvertUI64 | Operator::F32ConvertUI32 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().fcvt_from_uint(F32, val)); + let val = state.pop1(); + state.push1(builder.ins().fcvt_from_uint(F32, val)); } Operator::F64PromoteF32 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().fpromote(F64, val)); + let val = state.pop1(); + state.push1(builder.ins().fpromote(F64, val)); } Operator::F32DemoteF64 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().fdemote(F32, val)); + let val = state.pop1(); + state.push1(builder.ins().fdemote(F32, val)); } Operator::I64TruncSF64 | Operator::I64TruncSF32 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().fcvt_to_sint(I64, val)); + let val = state.pop1(); + state.push1(builder.ins().fcvt_to_sint(I64, val)); } Operator::I32TruncSF64 | Operator::I32TruncSF32 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().fcvt_to_sint(I32, val)); + let val = state.pop1(); + state.push1(builder.ins().fcvt_to_sint(I32, val)); } Operator::I64TruncUF64 | Operator::I64TruncUF32 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().fcvt_to_uint(I64, val)); + let val = state.pop1(); + state.push1(builder.ins().fcvt_to_uint(I64, val)); } Operator::I32TruncUF64 | Operator::I32TruncUF32 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().fcvt_to_uint(I32, val)); + let val = state.pop1(); + state.push1(builder.ins().fcvt_to_uint(I32, val)); } Operator::F32ReinterpretI32 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().bitcast(F32, val)); + let val = state.pop1(); + state.push1(builder.ins().bitcast(F32, val)); } Operator::F64ReinterpretI64 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().bitcast(F64, val)); + let val = state.pop1(); + state.push1(builder.ins().bitcast(F64, val)); } Operator::I32ReinterpretF32 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().bitcast(I32, val)); + let val = state.pop1(); + state.push1(builder.ins().bitcast(I32, val)); } Operator::I64ReinterpretF64 => { - let val = stack.pop().unwrap(); - stack.push(builder.ins().bitcast(I64, val)); + let val = state.pop1(); + state.push1(builder.ins().bitcast(I64, val)); } /****************************** Binary Operators ************************************/ Operator::I32Add | Operator::I64Add => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().iadd(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().iadd(arg1, arg2)); } Operator::I32And | Operator::I64And => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().band(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().band(arg1, arg2)); } Operator::I32Or | Operator::I64Or => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().bor(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().bor(arg1, arg2)); } Operator::I32Xor | Operator::I64Xor => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().bxor(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().bxor(arg1, arg2)); } Operator::I32Shl | Operator::I64Shl => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().ishl(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().ishl(arg1, arg2)); } Operator::I32ShrS | Operator::I64ShrS => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().sshr(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().sshr(arg1, arg2)); } Operator::I32ShrU | Operator::I64ShrU => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().ushr(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().ushr(arg1, arg2)); } Operator::I32Rotl | Operator::I64Rotl => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().rotl(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().rotl(arg1, arg2)); } Operator::I32Rotr | Operator::I64Rotr => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().rotr(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().rotr(arg1, arg2)); } Operator::F32Add | Operator::F64Add => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().fadd(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().fadd(arg1, arg2)); } Operator::I32Sub | Operator::I64Sub => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().isub(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().isub(arg1, arg2)); } Operator::F32Sub | Operator::F64Sub => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().fsub(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().fsub(arg1, arg2)); } Operator::I32Mul | Operator::I64Mul => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().imul(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().imul(arg1, arg2)); } Operator::F32Mul | Operator::F64Mul => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().fmul(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().fmul(arg1, arg2)); } Operator::F32Div | Operator::F64Div => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().fdiv(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().fdiv(arg1, arg2)); } Operator::I32DivS | Operator::I64DivS => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().sdiv(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().sdiv(arg1, arg2)); } Operator::I32DivU | Operator::I64DivU => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().udiv(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().udiv(arg1, arg2)); } Operator::I32RemS | Operator::I64RemS => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().srem(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().srem(arg1, arg2)); } Operator::I32RemU | Operator::I64RemU => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().urem(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().urem(arg1, arg2)); } Operator::F32Min | Operator::F64Min => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().fmin(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().fmin(arg1, arg2)); } Operator::F32Max | Operator::F64Max => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().fmax(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().fmax(arg1, arg2)); } Operator::F32Copysign | Operator::F64Copysign => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); - stack.push(builder.ins().fcopysign(arg1, arg2)); + let (arg1, arg2) = state.pop2(); + state.push1(builder.ins().fcopysign(arg1, arg2)); } /**************************** Comparison Operators **********************************/ Operator::I32LtS | Operator::I64LtS => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().icmp(IntCC::SignedLessThan, arg1, arg2); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::I32LtU | Operator::I64LtU => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().icmp(IntCC::UnsignedLessThan, arg1, arg2); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::I32LeS | Operator::I64LeS => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().icmp(IntCC::SignedLessThanOrEqual, arg1, arg2); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::I32LeU | Operator::I64LeU => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().icmp( IntCC::UnsignedLessThanOrEqual, arg1, arg2, ); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::I32GtS | Operator::I64GtS => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().icmp(IntCC::SignedGreaterThan, arg1, arg2); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::I32GtU | Operator::I64GtU => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().icmp(IntCC::UnsignedGreaterThan, arg1, arg2); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::I32GeS | Operator::I64GeS => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().icmp( IntCC::SignedGreaterThanOrEqual, arg1, arg2, ); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::I32GeU | Operator::I64GeU => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().icmp( IntCC::UnsignedGreaterThanOrEqual, arg1, arg2, ); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::I32Eqz | Operator::I64Eqz => { - let arg = stack.pop().unwrap(); + let arg = state.pop1(); let val = builder.ins().icmp_imm(IntCC::Equal, arg, 0); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::I32Eq | Operator::I64Eq => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().icmp(IntCC::Equal, arg1, arg2); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::F32Eq | Operator::F64Eq => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().fcmp(FloatCC::Equal, arg1, arg2); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::I32Ne | Operator::I64Ne => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().icmp(IntCC::NotEqual, arg1, arg2); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::F32Ne | Operator::F64Ne => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().fcmp(FloatCC::NotEqual, arg1, arg2); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::F32Gt | Operator::F64Gt => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().fcmp(FloatCC::GreaterThan, arg1, arg2); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::F32Ge | Operator::F64Ge => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().fcmp(FloatCC::GreaterThanOrEqual, arg1, arg2); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::F32Lt | Operator::F64Lt => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().fcmp(FloatCC::LessThan, arg1, arg2); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } Operator::F32Le | Operator::F64Le => { - let arg2 = stack.pop().unwrap(); - let arg1 = stack.pop().unwrap(); + let (arg1, arg2) = state.pop2(); let val = builder.ins().fcmp(FloatCC::LessThanOrEqual, arg1, arg2); - stack.push(builder.ins().bint(I32, val)); + state.push1(builder.ins().bint(I32, val)); } } } diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index bb342bf7c6..bb750572a1 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -16,11 +16,12 @@ extern crate wasmparser; extern crate cton_frontend; extern crate cretonne; -mod module_translator; -mod translation_utils; mod code_translator; +mod module_translator; mod runtime; mod sections_translator; +mod state; +mod translation_utils; pub use module_translator::{translate_module, TranslationResult, FunctionTranslation, ImportMappings}; diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs new file mode 100644 index 0000000000..99e9678562 --- /dev/null +++ b/lib/wasm/src/state.rs @@ -0,0 +1,180 @@ +//! WebAssembly function translation state. +//! +//! The `TranslationState` struct defined in this module is used to keep track of the WebAssembly +//! value and control stacks during the translation of a single function. + +use cretonne::ir::{Ebb, Inst, Type, Value}; + +/// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following +/// fields: +/// +/// - `destination`: reference to the `Ebb` that will hold the code after the control block; +/// - `return_values`: types of the values returned by the control block; +/// - `original_stack_size`: size of the value stack at the beginning of the control block. +/// +/// Moreover, the `if` frame has the `branch_inst` field that points to the `brz` instruction +/// separating the `true` and `false` branch. The `loop` frame has a `header` field that references +/// the `Ebb` that contains the beginning of the body of the loop. +#[derive(Debug)] +pub enum ControlStackFrame { + If { + destination: Ebb, + branch_inst: Inst, + return_values: Vec, + original_stack_size: usize, + reachable: bool, + }, + Block { + destination: Ebb, + return_values: Vec, + original_stack_size: usize, + reachable: bool, + }, + Loop { + destination: Ebb, + header: Ebb, + return_values: Vec, + original_stack_size: usize, + reachable: bool, + }, +} + +/// Helper methods for the control stack objects. +impl ControlStackFrame { + pub fn return_values(&self) -> &[Type] { + match *self { + ControlStackFrame::If { ref return_values, .. } | + ControlStackFrame::Block { ref return_values, .. } | + ControlStackFrame::Loop { ref return_values, .. } => &return_values, + } + } + pub fn following_code(&self) -> Ebb { + match *self { + ControlStackFrame::If { destination, .. } | + ControlStackFrame::Block { destination, .. } | + ControlStackFrame::Loop { destination, .. } => destination, + } + } + pub fn br_destination(&self) -> Ebb { + match *self { + ControlStackFrame::If { destination, .. } | + ControlStackFrame::Block { destination, .. } => destination, + ControlStackFrame::Loop { header, .. } => header, + } + } + pub fn original_stack_size(&self) -> usize { + match *self { + ControlStackFrame::If { original_stack_size, .. } | + ControlStackFrame::Block { original_stack_size, .. } | + ControlStackFrame::Loop { original_stack_size, .. } => original_stack_size, + } + } + pub fn is_loop(&self) -> bool { + match *self { + ControlStackFrame::If { .. } | + ControlStackFrame::Block { .. } => false, + ControlStackFrame::Loop { .. } => true, + } + } + + pub fn is_reachable(&self) -> bool { + match *self { + ControlStackFrame::If { reachable, .. } | + ControlStackFrame::Block { reachable, .. } | + ControlStackFrame::Loop { reachable, .. } => reachable, + } + } + + pub fn set_reachable(&mut self) { + match *self { + ControlStackFrame::If { ref mut reachable, .. } | + ControlStackFrame::Block { ref mut reachable, .. } | + ControlStackFrame::Loop { ref mut reachable, .. } => *reachable = true, + } + } +} + +/// Contains information passed along during the translation and that records: +/// +/// - The current value and control stacks. +/// - The depth of the two unreachable control blocks stacks, that are manipulated when translating +/// unreachable code; +pub struct TranslationState { + pub stack: Vec, + pub control_stack: Vec, + pub phantom_unreachable_stack_depth: usize, + pub real_unreachable_stack_depth: usize, +} + +impl TranslationState { + pub fn new() -> TranslationState { + TranslationState { + stack: Vec::new(), + control_stack: Vec::new(), + phantom_unreachable_stack_depth: 0, + real_unreachable_stack_depth: 0, + } + } + + /// Push a value. + pub fn push1(&mut self, val: Value) { + self.stack.push(val); + } + + /// Pop one value. + pub fn pop1(&mut self) -> Value { + self.stack.pop().unwrap() + } + + /// Peek at the top of the stack without popping it. + pub fn peek1(&self) -> Value { + *self.stack.last().unwrap() + } + + /// Pop two values. Return them in the order they were pushed. + pub fn pop2(&mut self) -> (Value, Value) { + let v2 = self.stack.pop().unwrap(); + let v1 = self.stack.pop().unwrap(); + (v1, v2) + } + + /// Pop three values. Return them in the order they were pushed. + pub fn pop3(&mut self) -> (Value, Value, Value) { + let v3 = self.stack.pop().unwrap(); + let v2 = self.stack.pop().unwrap(); + let v1 = self.stack.pop().unwrap(); + (v1, v2, v3) + } + + // Push a block on the control stack. + pub fn push_block(&mut self, following_code: Ebb, result_types: Vec) { + self.control_stack.push(ControlStackFrame::Block { + destination: following_code, + original_stack_size: self.stack.len(), + return_values: result_types, + reachable: false, + }); + } + + // Push a loop on the control stack. + pub fn push_loop(&mut self, header: Ebb, following_code: Ebb, result_types: Vec) { + self.control_stack.push(ControlStackFrame::Loop { + header, + destination: following_code, + original_stack_size: self.stack.len(), + return_values: result_types, + reachable: false, + }); + } + + // Push an if on the control stack. + pub fn push_if(&mut self, branch_inst: Inst, following_code: Ebb, result_types: Vec) { + self.control_stack.push(ControlStackFrame::If { + branch_inst, + destination: following_code, + original_stack_size: self.stack.len(), + return_values: result_types, + reachable: false, + }); + } +} From 9ea40ad44a9b035d0835d638f928c590d7e3c618 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sat, 2 Sep 2017 04:11:51 -0700 Subject: [PATCH 1058/3084] Replace SSABuilder's variables HashMaps with EntityMaps. (#150) * Replace SSABuilder's variables HashMaps with EntityMaps. Current measurements show that memory usage is approximately the same, and it's about 20% faster. * Add EntityMap::with_default and use it. * rustfmt * Use var_defs[block].expand(). --- lib/cretonne/src/entity/map.rs | 11 +++++++++++ lib/frontend/src/ssa.rs | 16 ++++++++-------- 2 files changed, 19 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/src/entity/map.rs b/lib/cretonne/src/entity/map.rs index 5db8522382..200aa464c3 100644 --- a/lib/cretonne/src/entity/map.rs +++ b/lib/cretonne/src/entity/map.rs @@ -41,6 +41,17 @@ where } } + /// Create a new empty map with a specified default value. + /// + /// This constructor does not require V to implement Default. + pub fn with_default(default: V) -> Self { + EntityMap { + elems: Vec::new(), + default: default, + unused: PhantomData, + } + } + /// Get the element at `k` if it exists. pub fn get(&self, k: K) -> Option<&V> { self.elems.get(k.index()) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 2cabe2cedf..13c3dbbfc3 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -15,7 +15,6 @@ use cretonne::packed_option::ReservedValue; use std::u32; use cretonne::ir::types::{F32, F64}; use cretonne::ir::immediates::{Ieee32, Ieee64}; -use std::collections::HashMap; use std::mem; /// Structure containing the data relevant the construction of SSA for a given function. @@ -39,7 +38,7 @@ where { // Records for every variable and for every revelant block, the last definition of // the variable in the block. - variables: EntityMap>, + variables: EntityMap>>, // Records the position of the basic blocks and the list of values used but not defined in the // block. blocks: PrimaryMap>, @@ -148,7 +147,7 @@ where /// Allocate a new blank SSA builder struct. Use the API function to interact with the struct. pub fn new() -> Self { Self { - variables: EntityMap::new(), + variables: EntityMap::with_default(EntityMap::new()), blocks: PrimaryMap::new(), ebb_headers: EntityMap::new(), } @@ -207,7 +206,7 @@ where /// The SSA value is passed as an argument because it should be created with /// `ir::DataFlowGraph::append_result`. pub fn def_var(&mut self, var: Variable, val: Value, block: Block) { - self.variables[var].insert(block, val); + self.variables[var][block] = PackedOption::from(val); } /// Declares a use of a variable in a given basic block. Returns the SSA value corresponding @@ -228,8 +227,8 @@ where ) -> (Value, SideEffects) { // First we lookup for the current definition of the variable in this block if let Some(var_defs) = self.variables.get(var) { - if let Some(val) = var_defs.get(&block) { - return (*val, SideEffects::new()); + if let Some(val) = var_defs[block].expand() { + return (val, SideEffects::new()); } } @@ -478,10 +477,11 @@ where // to keep the ebb argument. for &mut (ref mut pred_block, ref mut last_inst) in &mut preds { // We already did a full `use_var` above, so we can do just the fast path. - let pred_val = *self.variables + let pred_val = self.variables .get(temp_arg_var) .unwrap() - .get(&pred_block) + .get(*pred_block) + .unwrap() .unwrap(); if let Some((middle_ebb, middle_block, middle_jump_inst)) = self.append_jump_argument( From ef3ea724220aa37f80469de2b40471a7ff822f40 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 5 Sep 2017 09:15:29 -0700 Subject: [PATCH 1059/3084] Avoid calling Vec::split_off, avoiding more heap allocations. --- lib/filecheck/src/checker.rs | 5 +- lib/wasm/src/code_translator.rs | 84 ++++++++++++++++++++------------- 2 files changed, 54 insertions(+), 35 deletions(-) diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs index 9c88df8eb4..d48c0ba3e4 100644 --- a/lib/filecheck/src/checker.rs +++ b/lib/filecheck/src/checker.rs @@ -6,6 +6,7 @@ use std::borrow::Cow; use std::collections::HashMap; use std::cmp::max; use std::fmt::{self, Display, Formatter}; +use std::mem; use MatchRange; use explain::{Recorder, Explainer}; @@ -131,7 +132,9 @@ impl CheckerBuilder { pub fn finish(&mut self) -> Checker { // Move directives into the new checker, leaving `self.directives` empty and ready for // building a new checker. - Checker::new(self.directives.split_off(0)) + let mut new_directives = Vec::new(); + mem::swap(&mut new_directives, &mut self.directives); + Checker::new(new_directives) } } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 83b123e8f6..b057aa11c8 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -153,8 +153,8 @@ pub fn translate_function_body( // explicitely in Cretonne IL. if !builder.is_filled() && (!builder.is_unreachable() || !builder.is_pristine()) { let cut_index = state.stack.len() - sig.return_types.len(); - let return_vals = state.stack.split_off(cut_index); - builder.ins().return_(&return_vals); + builder.ins().return_(&state.stack[cut_index..]); + state.stack.truncate(cut_index); } // Because the function has an implicit block as body, we need to explicitely close it. let frame = state.control_stack.pop().unwrap(); @@ -167,8 +167,8 @@ pub fn translate_function_body( builder.ebb_args(frame.following_code()), ); let cut_index = state.stack.len() - sig.return_types.len(); - let return_vals = state.stack.split_off(cut_index); - builder.ins().return_(&return_vals); + builder.ins().return_(&state.stack[cut_index..]); + state.stack.truncate(cut_index); } } Ok((func, func_imports)) @@ -288,8 +288,8 @@ fn translate_operator( _ => panic!("should not happen"), }; let cut_index = state.stack.len() - return_values.len(); - let jump_args = state.stack.split_off(cut_index); - builder.ins().jump(destination, &jump_args); + builder.ins().jump(destination, &state.stack[cut_index..]); + state.stack.truncate(cut_index); // We change the target of the branch instruction let else_ebb = builder.create_ebb(); builder.change_jump_destination(branch_inst, else_ebb); @@ -300,8 +300,11 @@ fn translate_operator( let frame = state.control_stack.pop().unwrap(); if !builder.is_unreachable() || !builder.is_pristine() { let cut_index = state.stack.len() - frame.return_values().len(); - let jump_args = state.stack.split_off(cut_index); - builder.ins().jump(frame.following_code(), &jump_args); + builder.ins().jump( + frame.following_code(), + &state.stack[cut_index..], + ); + state.stack.truncate(cut_index); } builder.switch_to_block(frame.following_code(), frame.return_values()); builder.seal_block(frame.following_code()); @@ -339,13 +342,17 @@ fn translate_operator( Operator::Br { relative_depth } => { let i = state.control_stack.len() - 1 - (relative_depth as usize); let frame = &mut state.control_stack[i]; - let jump_args = if frame.is_loop() { - Vec::new() - } else { - let cut_index = state.stack.len() - frame.return_values().len(); - state.stack.split_off(cut_index) - }; - builder.ins().jump(frame.br_destination(), &jump_args); + let cut_index = state.stack.len() - + if frame.is_loop() { + 0 + } else { + frame.return_values().len() + }; + builder.ins().jump( + frame.br_destination(), + &state.stack[cut_index..], + ); + state.stack.truncate(cut_index); // We signal that all the code that follows until the next End is unreachable frame.set_reachable(); state.real_unreachable_stack_depth = 1 + relative_depth as usize; @@ -354,17 +361,20 @@ fn translate_operator( let val = state.pop1(); let i = state.control_stack.len() - 1 - (relative_depth as usize); let frame = &mut state.control_stack[i]; - let jump_args = if frame.is_loop() { - Vec::new() - } else { - let cut_index = state.stack.len() - frame.return_values().len(); - state.stack.split_off(cut_index) - }; - builder.ins().brnz(val, frame.br_destination(), &jump_args); + let cut_index = state.stack.len() - + if frame.is_loop() { + 0 + } else { + frame.return_values().len() + }; + builder.ins().brnz( + val, + frame.br_destination(), + &state.stack[cut_index..], + ); // The values returned by the branch are still available for the reachable // code that comes after it frame.set_reachable(); - state.stack.extend(jump_args); } Operator::BrTable { ref table } => { let (depths, default) = table.read_table(); @@ -406,7 +416,6 @@ fn translate_operator( // We then proceed to split the edges going out of the br_table let val = state.pop1(); let cut_index = state.stack.len() - jump_args_count; - let jump_args = state.stack.split_off(cut_index); let jt = builder.create_jump_table(); let dest_ebbs: HashMap = depths.iter().enumerate().fold(HashMap::new(), |mut acc, @@ -425,25 +434,25 @@ fn translate_operator( let default_ebb = state.control_stack[state.control_stack.len() - 1 - (default as usize)] .br_destination(); - builder.ins().jump(default_ebb, &jump_args); - state.stack.extend_from_slice(&jump_args); + builder.ins().jump(default_ebb, &state.stack[cut_index..]); for (depth, dest_ebb) in dest_ebbs { builder.switch_to_block(dest_ebb, &[]); builder.seal_block(dest_ebb); let i = state.control_stack.len() - 1 - depth; let frame = &mut state.control_stack[i]; let real_dest_ebb = frame.br_destination(); - builder.ins().jump(real_dest_ebb, &jump_args); + builder.ins().jump(real_dest_ebb, &state.stack[cut_index..]); frame.set_reachable(); } + state.stack.truncate(cut_index); state.real_unreachable_stack_depth = 1 + min_depth as usize; } } Operator::Return => { let return_count = sig.return_types.len(); let cut_index = state.stack.len() - return_count; - let return_args = state.stack.split_off(cut_index); - builder.ins().return_(&return_args); + builder.ins().return_(&state.stack[cut_index..]); + state.stack.truncate(cut_index); state.real_unreachable_stack_depth = 1; } /************************************ Calls **************************************** @@ -454,7 +463,6 @@ fn translate_operator( Operator::Call { function_index } => { let args_num = args_count(function_index as usize, functions, signatures); let cut_index = state.stack.len() - args_num; - let call_args = state.stack.split_off(cut_index); let internal_function_index = find_function_import( function_index as usize, builder, @@ -463,7 +471,11 @@ fn translate_operator( exports, signatures, ); - let call_inst = builder.ins().call(internal_function_index, &call_args); + let call_inst = builder.ins().call( + internal_function_index, + &state.stack[cut_index..], + ); + state.stack.truncate(cut_index); let ret_values = builder.inst_results(call_inst); for val in ret_values { state.push1(*val); @@ -480,9 +492,13 @@ fn translate_operator( let args_num = builder.signature(sigref).unwrap().argument_types.len(); let index_val = state.pop1(); let cut_index = state.stack.len() - args_num; - let call_args = state.stack.split_off(cut_index); - let ret_values = - runtime.translate_call_indirect(builder, sigref, index_val, &call_args); + let ret_values = runtime.translate_call_indirect( + builder, + sigref, + index_val, + &state.stack[cut_index..], + ); + state.stack.truncate(cut_index); for val in ret_values { state.push1(*val); } From 6f864f2926564c6a063e9950dc86b9ae63cf04f4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Sep 2017 10:52:31 -0700 Subject: [PATCH 1060/3084] Move translation state initialization into a method. --- lib/wasm/src/code_translator.rs | 12 +++--------- lib/wasm/src/state.rs | 25 ++++++++++++++++++++++++- 2 files changed, 27 insertions(+), 10 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index b057aa11c8..0ffa89806e 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -108,17 +108,11 @@ pub fn translate_function_body( } } - let mut state = TranslationState::new(); - // We initialize the control stack with the implicit function block + let mut state = TranslationState::new(); let end_ebb = builder.create_ebb(); - state.push_block( - end_ebb, - sig.return_types - .iter() - .map(|argty| argty.value_type) - .collect(), - ); + state.initialize(sig, end_ebb); + // Now the main loop that reads every wasm instruction and translates it loop { let parser_state = parser.read(); diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index 99e9678562..5e9d0186e5 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -3,7 +3,7 @@ //! The `TranslationState` struct defined in this module is used to keep track of the WebAssembly //! value and control stacks during the translation of a single function. -use cretonne::ir::{Ebb, Inst, Type, Value}; +use cretonne::ir::{self, Ebb, Inst, Type, Value}; /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following /// fields: @@ -116,6 +116,29 @@ impl TranslationState { } } + fn clear(&mut self) { + self.stack.clear(); + self.control_stack.clear(); + self.phantom_unreachable_stack_depth = 0; + self.real_unreachable_stack_depth = 0; + } + + /// Initialize the state for compiling a function with the given signature. + /// + /// This resets the state to containing only a single block representing the whole function. + /// The exit block is the last block in the function which will contain the return instruction. + pub fn initialize(&mut self, sig: &ir::Signature, exit_block: Ebb) { + self.clear(); + self.push_block( + exit_block, + sig.return_types + .iter() + .filter(|arg| arg.purpose == ir::ArgumentPurpose::Normal) + .map(|argty| argty.value_type) + .collect(), + ); + } + /// Push a value. pub fn push1(&mut self, val: Value) { self.stack.push(val); From 2671cbb092e51df410e60751438ee8fe11dfb718 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Sep 2017 11:46:08 -0700 Subject: [PATCH 1061/3084] Add TranslationState::in_unreachable_code(). Move an unreachable code test and sanity check into this method. --- lib/wasm/src/code_translator.rs | 6 +----- lib/wasm/src/state.rs | 10 ++++++++++ 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 0ffa89806e..0b540782f6 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -118,11 +118,7 @@ pub fn translate_function_body( let parser_state = parser.read(); match *parser_state { ParserState::CodeOperator(ref op) => { - debug_assert!( - state.phantom_unreachable_stack_depth == 0 || - state.real_unreachable_stack_depth > 0 - ); - if state.real_unreachable_stack_depth > 0 { + if state.in_unreachable_code() { translate_unreachable_operator(op, &mut builder, &mut state) } else { translate_operator( diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index 5e9d0186e5..d9fabb1194 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -200,4 +200,14 @@ impl TranslationState { reachable: false, }); } + + /// Test if the translation state is currently in unreachable code. + pub fn in_unreachable_code(&self) -> bool { + if self.real_unreachable_stack_depth > 0 { + true + } else { + debug_assert_eq!(self.phantom_unreachable_stack_depth, 0, "in reachable code"); + false + } + } } From 19c8ba5021fd6d046921f2b00b2fb7e99e49828b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Sep 2017 11:51:32 -0700 Subject: [PATCH 1062/3084] Eliminate the `sig` argument to translate_operator. The current function's return types are pushed as the first control stack frame. --- lib/wasm/src/code_translator.rs | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 0b540782f6..1e96b02c4d 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -126,7 +126,6 @@ pub fn translate_function_body( &mut builder, runtime, &mut state, - sig, functions, signatures, exports, @@ -171,7 +170,6 @@ fn translate_operator( builder: &mut FunctionBuilder, runtime: &mut WasmRuntime, state: &mut TranslationState, - sig: &Signature, functions: &[SignatureIndex], signatures: &[Signature], exports: &Option>, @@ -439,7 +437,7 @@ fn translate_operator( } } Operator::Return => { - let return_count = sig.return_types.len(); + let return_count = state.control_stack[0].return_values().len(); let cut_index = state.stack.len() - return_count; builder.ins().return_(&state.stack[cut_index..]); state.stack.truncate(cut_index); From 0ac1d0dd942fafb47b165f7ea63361d036905ce1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Sep 2017 13:11:35 -0700 Subject: [PATCH 1063/3084] Add FuncEnvironment trait. This trait is used to provide the environment necessary to translate a single WebAssembly function without having other global data structures for the WebAssembly module. The WasmRuntime trait extends the FuncEnvironment trait for those uses that want to parse a whole WebAssembly module. - Change the handling of WebAssembly globals to use the FuncEnvironment trait as well as the new GlobalVar infrastructure in Cretonne. The runtime is not consulted on the translation of each get_global/get_global instruction. Instead it gets to create the GlobalVar declaration in the function preamble the first time the global is used. - Change the handling of heap load/store instructions to use the new Heap infrastructure in Cretonne. The runtime is called to create the Heap declaration in the preamble. It is not involved in individual load/store instructions. --- lib/frontend/src/frontend.rs | 5 +- lib/wasm/src/code_translator.rs | 265 +++++++++++++++----------------- lib/wasm/src/runtime/dummy.rs | 53 +++---- lib/wasm/src/runtime/mod.rs | 2 +- lib/wasm/src/runtime/spec.rs | 60 +++++--- lib/wasm/src/state.rs | 45 ++++++ 6 files changed, 240 insertions(+), 190 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index beb5dfdcfe..8eb90dd1d6 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -26,7 +26,10 @@ pub struct FunctionBuilder<'a, Variable: 'a> where Variable: EntityRef + Hash + Default, { - func: &'a mut Function, + /// The function currently being built. + /// This field is public so the function can be re-borrowed. + pub func: &'a mut Function, + builder: &'a mut ILBuilder, position: Position, pristine: bool, diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 1e96b02c4d..d7e895cd9a 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -21,18 +21,18 @@ //! //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as //! argument. -use cretonne::ir::{Function, Signature, Type, InstBuilder, FunctionName, Ebb, FuncRef, SigRef, - ExtFuncData, MemFlags}; +use cretonne::ir::{self, Function, Signature, Type, InstBuilder, FunctionName, Ebb, FuncRef, + SigRef, ExtFuncData, MemFlags}; use cretonne::ir::types::*; -use cretonne::ir::immediates::{Ieee32, Ieee64, Offset32}; +use cretonne::ir::immediates::{Ieee32, Ieee64}; use cretonne::ir::condcodes::{IntCC, FloatCC}; use cton_frontend::{ILBuilder, FunctionBuilder}; use wasmparser::{Parser, ParserState, Operator, WasmDecoder, MemoryImmediate}; use translation_utils::{f32_translation, f64_translation, type_to_type, translate_type, Local, - GlobalIndex, FunctionIndex, SignatureIndex}; + FunctionIndex, SignatureIndex}; use state::{TranslationState, ControlStackFrame}; use std::collections::HashMap; -use runtime::WasmRuntime; +use runtime::{FuncEnvironment, GlobalValue, WasmRuntime}; use std::u32; @@ -194,12 +194,28 @@ fn translate_operator( * `get_global` and `set_global` are handled by the runtime. ***********************************************************************************/ Operator::GetGlobal { global_index } => { - let val = runtime.translate_get_global(builder, global_index as GlobalIndex); + let val = match state.get_global(builder.func, global_index, runtime) { + GlobalValue::Const(val) => val, + GlobalValue::Memory { gv, ty } => { + let addr = builder.ins().global_addr(runtime.native_pointer(), gv); + // TODO: It is likely safe to set `aligned notrap` flags on a global load. + let flags = ir::MemFlags::new(); + builder.ins().load(ty, flags, addr, 0) + } + }; state.push1(val); } Operator::SetGlobal { global_index } => { - let val = state.pop1(); - runtime.translate_set_global(builder, global_index as GlobalIndex, val); + match state.get_global(builder.func, global_index, runtime) { + GlobalValue::Const(_) => panic!("global #{} is a constant", global_index), + GlobalValue::Memory { gv, .. } => { + let addr = builder.ins().global_addr(runtime.native_pointer(), gv); + // TODO: It is likely safe to set `aligned notrap` flags on a global store. + let flags = ir::MemFlags::new(); + let val = state.pop1(); + builder.ins().store(flags, val, addr, 0); + } + } } /********************************* Stack misc *************************************** * `drop`, `nop`, `unreachable` and `select`. @@ -508,130 +524,46 @@ fn translate_operator( * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not ************************************************************************************/ Operator::I32Load8U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().uload8(I32, memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Uload8, I32, builder, state, runtime); } Operator::I32Load16U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().uload8(I32, memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Uload16, I32, builder, state, runtime); } Operator::I32Load8S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().sload8(I32, memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Sload8, I32, builder, state, runtime); } Operator::I32Load16S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().sload8(I32, memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Sload16, I32, builder, state, runtime); } Operator::I64Load8U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().uload8(I64, memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Uload8, I64, builder, state, runtime); } Operator::I64Load16U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().uload16(I64, memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Uload16, I64, builder, state, runtime); } Operator::I64Load8S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().sload8(I64, memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Sload8, I64, builder, state, runtime); } Operator::I64Load16S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().sload16(I64, memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Sload16, I64, builder, state, runtime); } Operator::I64Load32S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().sload32(memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Sload32, I64, builder, state, runtime); } Operator::I64Load32U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().uload32(memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Uload32, I64, builder, state, runtime); } Operator::I32Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().load(I32, memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Load, I32, builder, state, runtime); } Operator::F32Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().load(F32, memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Load, F32, builder, state, runtime); } Operator::I64Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().load(I64, memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Load, I64, builder, state, runtime); } Operator::F64Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - state.push1(builder.ins().load(F64, memflags, addr, memoffset)) + translate_load(offset, ir::Opcode::Load, F64, builder, state, runtime); } /****************************** Store instructions *********************************** * Wasm specifies an integer alignment flag but we drop it in Cretonne. @@ -642,46 +574,18 @@ fn translate_operator( Operator::I64Store { memory_immediate: MemoryImmediate { flags: _, offset } } | Operator::F32Store { memory_immediate: MemoryImmediate { flags: _, offset } } | Operator::F64Store { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let val = state.pop1(); - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - builder.ins().store(memflags, val, addr, memoffset); + translate_store(offset, ir::Opcode::Store, builder, state, runtime); } Operator::I32Store8 { memory_immediate: MemoryImmediate { flags: _, offset } } | Operator::I64Store8 { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let val = state.pop1(); - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - builder.ins().istore8(memflags, val, addr, memoffset); + translate_store(offset, ir::Opcode::Istore8, builder, state, runtime); } Operator::I32Store16 { memory_immediate: MemoryImmediate { flags: _, offset } } | Operator::I64Store16 { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let val = state.pop1(); - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - builder.ins().istore16(memflags, val, addr, memoffset); + translate_store(offset, ir::Opcode::Istore16, builder, state, runtime); } Operator::I64Store32 { memory_immediate: MemoryImmediate { flags: _, offset } } => { - let val = state.pop1(); - let address_i32 = state.pop1(); - let base = runtime.translate_memory_base_address(builder, 0); - let address_i64 = builder.ins().uextend(I64, address_i32); - let addr = builder.ins().iadd(base, address_i64); - let memflags = MemFlags::new(); - let memoffset = Offset32::new(offset as i32); - builder.ins().istore32(memflags, val, addr, memoffset); + translate_store(offset, ir::Opcode::Istore32, builder, state, runtime); } /****************************** Nullary Operators ************************************/ Operator::I32Const { value } => state.push1(builder.ins().iconst(I32, value as i64)), @@ -1117,6 +1021,91 @@ fn translate_unreachable_operator( } } +// Get the address+offset to use for a heap access. +fn get_heap_addr( + heap: ir::Heap, + addr32: ir::Value, + offset: u32, + addr_ty: ir::Type, + builder: &mut FunctionBuilder, +) -> (ir::Value, i32) { + use std::cmp::min; + + let guard_size: i64 = builder.func.heaps[heap].guard_size.into(); + assert!(guard_size > 0, "Heap guard pages currently required"); + + // Generate `heap_addr` instructions that are friendly to CSE by checking offsets that are + // multiples of the guard size. Add one to make sure that we check the pointer itself is in + // bounds. + // + // For accesses on the outer skirts of the guard pages, we expect that we get a trap + // even if the access goes beyond the guard pages. This is because the first byte pointed to is + // inside the guard pages. + let check_size = min( + u32::max_value() as i64, + 1 + (offset as i64 / guard_size) * guard_size, + ) as u32; + let base = builder.ins().heap_addr(addr_ty, heap, addr32, check_size); + + // Native load/store instructions take a signed `Offset32` immediate, so adjust the base + // pointer if necessary. + if offset > i32::max_value() as u32 { + // Offset doesn't fit in the load/store instruction. + let adj = builder.ins().iadd_imm(base, i32::max_value() as i64 + 1); + (adj, (offset - (i32::max_value() as u32 + 1)) as i32) + } else { + (base, offset as i32) + } +} + +// Translate a load instruction. +fn translate_load( + offset: u32, + opcode: ir::Opcode, + result_ty: ir::Type, + builder: &mut FunctionBuilder, + state: &mut TranslationState, + environ: &FE, +) { + let addr32 = state.pop1(); + // We don't yet support multiple linear memories. + let heap = state.get_heap(builder.func, 0, environ); + let (base, offset) = get_heap_addr(heap, addr32, offset, environ.native_pointer(), builder); + let flags = MemFlags::new(); + let (load, dfg) = builder.ins().Load( + opcode, + result_ty, + flags, + offset.into(), + base, + ); + state.push1(dfg.first_result(load)); +} + +// Translate a store instruction. +fn translate_store( + offset: u32, + opcode: ir::Opcode, + builder: &mut FunctionBuilder, + state: &mut TranslationState, + environ: &FE, +) { + let (addr32, val) = state.pop2(); + let val_ty = builder.func.dfg.value_type(val); + + // We don't yet support multiple linear memories. + let heap = state.get_heap(builder.func, 0, environ); + let (base, offset) = get_heap_addr(heap, addr32, offset, environ.native_pointer(), builder); + let flags = MemFlags::new(); + builder.ins().Store( + opcode, + val_ty, + flags, + offset.into(), + val, + base, + ); +} fn args_count( index: FunctionIndex, functions: &[SignatureIndex], diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 05d7cbfaa3..9b0df58174 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -1,9 +1,8 @@ -use runtime::WasmRuntime; +use runtime::{FuncEnvironment, GlobalValue, WasmRuntime}; use translation_utils::{Local, Global, Memory, Table, GlobalIndex, TableIndex, FunctionIndex, MemoryIndex}; use cton_frontend::FunctionBuilder; -use cretonne::ir::{Value, InstBuilder, SigRef}; -use cretonne::ir::immediates::{Ieee32, Ieee64}; +use cretonne::ir::{self, Value, InstBuilder, SigRef}; use cretonne::ir::types::*; /// This runtime implementation is a "naïve" one, doing essentially nothing and emitting @@ -20,29 +19,32 @@ impl DummyRuntime { } } -impl WasmRuntime for DummyRuntime { - fn translate_get_global( - &self, - builder: &mut FunctionBuilder, - global_index: GlobalIndex, - ) -> Value { - let glob = self.globals[global_index]; - match glob.ty { - I32 => builder.ins().iconst(glob.ty, -1), - I64 => builder.ins().iconst(glob.ty, -1), - F32 => builder.ins().f32const(Ieee32::with_bits(0xbf800000)), // -1.0 - F64 => { - builder.ins().f64const( - Ieee64::with_bits(0xbff0000000000000), - ) - } // -1.0 - _ => panic!("should not happen"), +impl FuncEnvironment for DummyRuntime { + fn native_pointer(&self) -> ir::Type { + ir::types::I64 + } + + fn make_global(&self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue { + // Just create a dummy `vmctx` global. + let offset = ((index * 8) as i32 + 8).into(); + let gv = func.global_vars.push(ir::GlobalVarData::VmCtx { offset }); + GlobalValue::Memory { + gv, + ty: self.globals[index].ty, } } - fn translate_set_global(&self, _: &mut FunctionBuilder, _: GlobalIndex, _: Value) { - // We do nothing + fn make_heap(&self, func: &mut ir::Function, _index: MemoryIndex) -> ir::Heap { + func.heaps.push(ir::HeapData { + base: ir::HeapBase::ReservedReg, + min_size: 0.into(), + guard_size: 0x8000_0000.into(), + style: ir::HeapStyle::Static { bound: 0x1_0000_0000.into() }, + }) } +} + +impl WasmRuntime for DummyRuntime { fn translate_grow_memory(&mut self, builder: &mut FunctionBuilder, _: Value) -> Value { builder.ins().iconst(I32, -1) } @@ -59,13 +61,6 @@ impl WasmRuntime for DummyRuntime { let call_inst = builder.ins().call_indirect(sig_ref, index_val, call_args); builder.inst_results(call_inst) } - fn translate_memory_base_address( - &self, - builder: &mut FunctionBuilder, - _: MemoryIndex, - ) -> Value { - builder.ins().iconst(I64, 0) - } fn declare_global(&mut self, global: Global) { self.globals.push(global); } diff --git a/lib/wasm/src/runtime/mod.rs b/lib/wasm/src/runtime/mod.rs index 9f2a41d4f9..548ad6bb6b 100644 --- a/lib/wasm/src/runtime/mod.rs +++ b/lib/wasm/src/runtime/mod.rs @@ -1,5 +1,5 @@ mod spec; mod dummy; -pub use runtime::spec::WasmRuntime; +pub use runtime::spec::{WasmRuntime, FuncEnvironment, GlobalValue}; pub use runtime::dummy::DummyRuntime; diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 9f346337ab..5333d84c95 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -1,14 +1,51 @@ //! All the runtime support necessary for the wasm to cretonne translation is formalized by the //! trait `WasmRuntime`. use cton_frontend::FunctionBuilder; -use cretonne::ir::{Value, SigRef}; +use cretonne::ir::{self, Value, SigRef}; use translation_utils::{Local, FunctionIndex, TableIndex, GlobalIndex, MemoryIndex, Global, Table, Memory}; +/// The value of a WebAssembly global variable. +#[derive(Clone, Copy)] +pub enum GlobalValue { + /// This is a constant global with a value known at compile time. + Const(ir::Value), + + /// This is a variable in memory that should be referenced as a `GlobalVar`. + Memory { gv: ir::GlobalVar, ty: ir::Type }, +} + +/// Environment affecting the translation of a single WebAssembly function. +/// +/// A `FuncEnvironment` trait object is required to translate a WebAssembly function to Cretonne +/// IL. The function environment provides information about the WebAssembly module as well as the +/// runtime environment. +pub trait FuncEnvironment { + /// Get the Cretonne integer type to use for native pointers. + /// + /// This should be `I64` for 64-bit architectures and `I32` for 32-bit architectures. + fn native_pointer(&self) -> ir::Type; + + /// Set up the necessary preamble definitions in `func` to access the global variable + /// identified by `index`. + /// + /// The index space covers both imported globals and globals defined by the module. + /// + /// Return the global variable reference that should be used to access the global and the + /// WebAssembly type of the global. + fn make_global(&self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue; + + /// Set up the necessary preamble definitions in `func` to access the linear memory identified + /// by `index`. + /// + /// The index space covers both imported and locally declared memories. + fn make_heap(&self, func: &mut ir::Function, index: MemoryIndex) -> ir::Heap; +} + /// An object satisfyng the `WasmRuntime` trait can be passed as argument to the /// [`translate_module`](fn.translate_module.html) function. These methods should not be called /// by the user, they are only for the `wasm2cretonne` internal use. -pub trait WasmRuntime { +pub trait WasmRuntime: FuncEnvironment { /// Declares a global to the runtime. fn declare_global(&mut self, global: Global); /// Declares a table to the runtime. @@ -34,29 +71,10 @@ pub trait WasmRuntime { fn begin_translation(&mut self); /// Call this function between each function body translation. fn next_function(&mut self); - /// Translates a `get_global` wasm instruction. - fn translate_get_global( - &self, - builder: &mut FunctionBuilder, - global_index: GlobalIndex, - ) -> Value; - /// Translates a `set_global` wasm instruction. - fn translate_set_global( - &self, - builder: &mut FunctionBuilder, - global_index: GlobalIndex, - val: Value, - ); /// Translates a `grow_memory` wasm instruction. Returns the old size (in pages) of the memory. fn translate_grow_memory(&mut self, builder: &mut FunctionBuilder, val: Value) -> Value; /// Translates a `current_memory` wasm instruction. Returns the size in pages of the memory. fn translate_current_memory(&mut self, builder: &mut FunctionBuilder) -> Value; - /// Returns the base address of a wasm memory as a Cretonne `Value`. - fn translate_memory_base_address( - &self, - builder: &mut FunctionBuilder, - index: MemoryIndex, - ) -> Value; /// Translates a `call_indirect` wasm instruction. It involves looking up the value contained /// it the table at location `index_val` and calling the corresponding function. fn translate_call_indirect<'a>( diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index d9fabb1194..8479b5cc8a 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -4,6 +4,9 @@ //! value and control stacks during the translation of a single function. use cretonne::ir::{self, Ebb, Inst, Type, Value}; +use runtime::{FuncEnvironment, GlobalValue}; +use std::collections::HashMap; +use translation_utils::{GlobalIndex, MemoryIndex}; /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following /// fields: @@ -104,6 +107,12 @@ pub struct TranslationState { pub control_stack: Vec, pub phantom_unreachable_stack_depth: usize, pub real_unreachable_stack_depth: usize, + + // Map of global variables that have already been created by `FuncEnvironment::make_global`. + globals: HashMap, + + // Map of heaps that have been created by `FuncEnvironment::make_heap`. + heaps: HashMap, } impl TranslationState { @@ -113,6 +122,8 @@ impl TranslationState { control_stack: Vec::new(), phantom_unreachable_stack_depth: 0, real_unreachable_stack_depth: 0, + globals: HashMap::new(), + heaps: HashMap::new(), } } @@ -121,6 +132,8 @@ impl TranslationState { self.control_stack.clear(); self.phantom_unreachable_stack_depth = 0; self.real_unreachable_stack_depth = 0; + self.globals.clear(); + self.heaps.clear(); } /// Initialize the state for compiling a function with the given signature. @@ -211,3 +224,35 @@ impl TranslationState { } } } + +/// Methods for handling entity references. +impl TranslationState { + /// Get the `GlobalVar` reference that should be used to access the global variable `index`. + /// Create the reference if necessary. + /// Also return the WebAssembly type of the global. + pub fn get_global( + &mut self, + func: &mut ir::Function, + index: u32, + environ: &FE, + ) -> GlobalValue { + let index = index as GlobalIndex; + *self.globals.entry(index).or_insert_with( + || environ.make_global(func, index), + ) + } + + /// Get the `Heap` reference that should be used to access linear memory `index`. + /// Create the reference if necessary. + pub fn get_heap( + &mut self, + func: &mut ir::Function, + index: u32, + environ: &FE, + ) -> ir::Heap { + let index = index as MemoryIndex; + *self.heaps.entry(index).or_insert_with( + || environ.make_heap(func, index), + ) + } +} From d3712575b523daea161d6e661c6fb6522e2e0245 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 5 Sep 2017 13:55:33 -0700 Subject: [PATCH 1064/3084] Use mem::replace instead of mem::swap when it's cleaner. --- lib/filecheck/src/checker.rs | 3 +-- lib/frontend/src/ssa.rs | 6 ++---- 2 files changed, 3 insertions(+), 6 deletions(-) diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs index d48c0ba3e4..bad8e4774e 100644 --- a/lib/filecheck/src/checker.rs +++ b/lib/filecheck/src/checker.rs @@ -132,8 +132,7 @@ impl CheckerBuilder { pub fn finish(&mut self) -> Checker { // Move directives into the new checker, leaving `self.directives` empty and ready for // building a new checker. - let mut new_directives = Vec::new(); - mem::swap(&mut new_directives, &mut self.directives); + let new_directives = mem::replace(&mut self.directives, Vec::new()); Checker::new(new_directives) } } diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 13c3dbbfc3..60c5663866 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -370,8 +370,7 @@ where debug_assert!(!data.sealed); // Extract the undef_variables data from the block so that we // can iterate over it without borrowing the whole builder. - let mut undef_variables = Vec::new(); - mem::swap(&mut data.undef_variables, &mut undef_variables); + let undef_variables = mem::replace(&mut data.undef_variables, Vec::new()); (undef_variables, data.ebb) } }; @@ -415,8 +414,7 @@ where // Iterate over the predecessors. To avoid borrowing `self` for the whole loop, // temporarily detach the predecessors list and replace it with an empty list. // `use_var`'s traversal won't revisit these predecesors. - let mut preds = Vec::new(); - mem::swap(&mut preds, &mut self.predecessors_mut(dest_ebb)); + let mut preds = mem::replace(self.predecessors_mut(dest_ebb), Vec::new()); for &(pred, _) in &preds { // For each predecessor, we query what is the local SSA value corresponding // to var and we put it as an argument of the branch instruction. From 320c88f3654b06a02a1b0f606d2257f3bdac10a1 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 5 Sep 2017 16:27:38 -0700 Subject: [PATCH 1065/3084] Rename cretonne_wasm to cton_wasm, for consistency with the other libraries. --- cranelift/src/cton-util.rs | 2 +- cranelift/src/wasm.rs | 2 +- lib/wasm/Cargo.toml | 3 +++ lib/wasm/tests/testsuite.rs | 4 ++-- 4 files changed, 7 insertions(+), 4 deletions(-) diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index b940947523..3d9b979e2f 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -1,7 +1,7 @@ #[macro_use(dbg)] extern crate cretonne; extern crate cton_reader; -extern crate cretonne_wasm; +extern crate cton_wasm; extern crate docopt; #[macro_use] extern crate serde_derive; diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index c418c3cc2c..87efd7d629 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -5,7 +5,7 @@ //! IL. Can also executes the `start` function of the module by laying out the memories, globals //! and tables, then emitting the translated code with hardcoded addresses to memory. -use cretonne_wasm::{translate_module, FunctionTranslation, DummyRuntime, WasmRuntime}; +use cton_wasm::{translate_module, FunctionTranslation, DummyRuntime, WasmRuntime}; use std::path::PathBuf; use cretonne::loop_analysis::LoopAnalysis; use cretonne::flowgraph::ControlFlowGraph; diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 28710bdc76..15dd4bb342 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -7,6 +7,9 @@ description = "Translator from WebAssembly to Cretonne IL" repository = "https://github.com/stoklund/cretonne" license = "Apache-2.0" +[lib] +name = "cton_wasm" + [dependencies] wasmparser = "0.8.2" cretonne = { path = "../cretonne" } diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/testsuite.rs index cb56779b00..91f514fde4 100644 --- a/lib/wasm/tests/testsuite.rs +++ b/lib/wasm/tests/testsuite.rs @@ -1,7 +1,7 @@ -extern crate cretonne_wasm; +extern crate cton_wasm; extern crate cretonne; -use cretonne_wasm::{translate_module, FunctionTranslation, DummyRuntime, WasmRuntime}; +use cton_wasm::{translate_module, FunctionTranslation, DummyRuntime, WasmRuntime}; use std::path::PathBuf; use std::fs::File; use std::error::Error; From 27e9e16077e456a8e6db131d211054b49f6275e1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 6 Sep 2017 10:28:11 -0700 Subject: [PATCH 1066/3084] Add a FuncEnvironment::make_indirect_sig() callback. The function environment is now expected to keep track of the function signatures in the module, and it is asked to generate Cretonne signatures to be used for indirect calls. The combination of make_indirect_sig() and translate_call_indirect() callbacks allow the runtime to insert additional function arguments for indirect calls such as vmctx pointers and CFI-style signature identifiers. --- cranelift/wasmtests/icall.wast | 7 ++++ lib/wasm/src/code_translator.rs | 38 +++++---------------- lib/wasm/src/module_translator.rs | 2 +- lib/wasm/src/runtime/dummy.rs | 22 +++++++++++-- lib/wasm/src/runtime/spec.rs | 18 ++++++++-- lib/wasm/src/sections_translator.rs | 2 ++ lib/wasm/src/state.rs | 51 ++++++++++++++++++++++++++++- 7 files changed, 103 insertions(+), 37 deletions(-) create mode 100644 cranelift/wasmtests/icall.wast diff --git a/cranelift/wasmtests/icall.wast b/cranelift/wasmtests/icall.wast new file mode 100644 index 0000000000..21894c55ca --- /dev/null +++ b/cranelift/wasmtests/icall.wast @@ -0,0 +1,7 @@ +(module + (type $ft (func (param f32) (result i32))) + (func $foo (export "foo") (param i32 f32) (result i32) + (call_indirect $ft (get_local 1) (get_local 0)) + ) + (table (;0;) 23 23 anyfunc) +) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index d7e895cd9a..ee3c6c6f57 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -489,23 +489,15 @@ fn translate_operator( index, table_index: _, } => { - // index is the index of the function's signature and table_index is the index - // of the table to search the function in - // TODO: have runtime support for tables - let sigref = find_signature_import(index as usize, builder, func_imports, signatures); - let args_num = builder.signature(sigref).unwrap().argument_types.len(); + // `index` is the index of the function's signature and `table_index` is the index of + // the table to search the function in. + // TODO: Have runtime support for tables. + let (sigref, num_args) = state.get_indirect_sig(builder.func, index, runtime); let index_val = state.pop1(); - let cut_index = state.stack.len() - args_num; - let ret_values = runtime.translate_call_indirect( - builder, - sigref, - index_val, - &state.stack[cut_index..], - ); - state.stack.truncate(cut_index); - for val in ret_values { - state.push1(*val); - } + let ret_values = + runtime.translate_call_indirect(builder, sigref, index_val, &state.peekn(num_args)); + state.popn(num_args); + state.pushn(ret_values); } /******************************* Memory management *********************************** * Memory management is handled by runtime. It is usually translated into calls to @@ -1163,17 +1155,3 @@ fn find_function_import( func_imports.functions.insert(index, local_func_index); local_func_index } - -fn find_signature_import( - sig_index: SignatureIndex, - builder: &mut FunctionBuilder, - func_imports: &mut FunctionImports, - signatures: &[Signature], -) -> SigRef { - if let Some(local_sig_index) = func_imports.signatures.get(&sig_index) { - return *local_sig_index; - } - let sig_local_index = builder.import_signature(signatures[sig_index].clone()); - func_imports.signatures.insert(sig_index, sig_local_index); - sig_local_index -} diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 6a2e5f3963..c2346e52b3 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -81,7 +81,7 @@ pub fn translate_module( loop { match *parser.read_with_input(next_input) { ParserState::BeginSection { code: SectionCode::Type, .. } => { - match parse_function_signatures(&mut parser) { + match parse_function_signatures(&mut parser, runtime) { Ok(sigs) => signatures = Some(sigs), Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the type section: {}", s)) diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 9b0df58174..7ff9747dc6 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -1,6 +1,6 @@ use runtime::{FuncEnvironment, GlobalValue, WasmRuntime}; -use translation_utils::{Local, Global, Memory, Table, GlobalIndex, TableIndex, FunctionIndex, - MemoryIndex}; +use translation_utils::{Local, Global, Memory, Table, GlobalIndex, TableIndex, SignatureIndex, + FunctionIndex, MemoryIndex}; use cton_frontend::FunctionBuilder; use cretonne::ir::{self, Value, InstBuilder, SigRef}; use cretonne::ir::types::*; @@ -9,13 +9,18 @@ use cretonne::ir::types::*; /// placeholders when forced to. Don't try to execute code translated with this runtime, it is /// essentially here for translation debug purposes. pub struct DummyRuntime { + // Unprocessed signatures exactly as provided by `declare_signature()`. + signatures: Vec, globals: Vec, } impl DummyRuntime { /// Allocates the runtime data structures. pub fn new() -> Self { - Self { globals: Vec::new() } + Self { + signatures: Vec::new(), + globals: Vec::new(), + } } } @@ -42,6 +47,12 @@ impl FuncEnvironment for DummyRuntime { style: ir::HeapStyle::Static { bound: 0x1_0000_0000.into() }, }) } + + fn make_indirect_sig(&self, func: &mut ir::Function, index: SignatureIndex) -> ir::SigRef { + // A real implementation would probably change the calling convention and add `vmctx` and + // signature index arguments. + func.dfg.signatures.push(self.signatures[index].clone()) + } } impl WasmRuntime for DummyRuntime { @@ -61,6 +72,11 @@ impl WasmRuntime for DummyRuntime { let call_inst = builder.ins().call_indirect(sig_ref, index_val, call_args); builder.inst_results(call_inst) } + + fn declare_signature(&mut self, sig: &ir::Signature) { + self.signatures.push(sig.clone()); + } + fn declare_global(&mut self, global: Global) { self.globals.push(global); } diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 5333d84c95..3dee0833bf 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -2,8 +2,8 @@ //! trait `WasmRuntime`. use cton_frontend::FunctionBuilder; use cretonne::ir::{self, Value, SigRef}; -use translation_utils::{Local, FunctionIndex, TableIndex, GlobalIndex, MemoryIndex, Global, Table, - Memory}; +use translation_utils::{Local, SignatureIndex, FunctionIndex, TableIndex, GlobalIndex, + MemoryIndex, Global, Table, Memory}; /// The value of a WebAssembly global variable. #[derive(Clone, Copy)] @@ -40,12 +40,26 @@ pub trait FuncEnvironment { /// /// The index space covers both imported and locally declared memories. fn make_heap(&self, func: &mut ir::Function, index: MemoryIndex) -> ir::Heap; + + /// Set up a signature definition in the preamble of `func` that can be used for an indirect + /// call with signature `index`. + /// + /// The signature may contain additional arguments needed for an indirect call, but the + /// arguments marked as `ArgumentPurpose::Normal` must correspond to the WebAssembly signature + /// arguments. + /// + /// The signature will only be used for indirect calls, even if the module has direct function + /// calls with the same WebAssembly type. + fn make_indirect_sig(&self, func: &mut ir::Function, index: SignatureIndex) -> ir::SigRef; } /// An object satisfyng the `WasmRuntime` trait can be passed as argument to the /// [`translate_module`](fn.translate_module.html) function. These methods should not be called /// by the user, they are only for the `wasm2cretonne` internal use. pub trait WasmRuntime: FuncEnvironment { + /// Declares a function signature to the runtime. + fn declare_signature(&mut self, sig: &ir::Signature); + /// Declares a global to the runtime. fn declare_global(&mut self, global: Global); /// Declares a table to the runtime. diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 6c97d928f7..591146c716 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -26,6 +26,7 @@ pub enum SectionParsingError { /// Reads the Type Section of the wasm module and returns the corresponding function signatures. pub fn parse_function_signatures( parser: &mut Parser, + runtime: &mut WasmRuntime, ) -> Result, SectionParsingError> { let mut signatures: Vec = Vec::new(); loop { @@ -53,6 +54,7 @@ pub fn parse_function_signatures( }; ArgumentType::new(cret_arg) })); + runtime.declare_signature(&sig); signatures.push(sig); } ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index 8479b5cc8a..5747ceda72 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -6,7 +6,7 @@ use cretonne::ir::{self, Ebb, Inst, Type, Value}; use runtime::{FuncEnvironment, GlobalValue}; use std::collections::HashMap; -use translation_utils::{GlobalIndex, MemoryIndex}; +use translation_utils::{GlobalIndex, MemoryIndex, SignatureIndex}; /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following /// fields: @@ -113,6 +113,11 @@ pub struct TranslationState { // Map of heaps that have been created by `FuncEnvironment::make_heap`. heaps: HashMap, + + // Map of indirect call signatures that have been created by + // `FuncEnvironment::make_indirect_sig()`. + // Stores both the signature reference and the number of WebAssembly arguments + signatures: HashMap, } impl TranslationState { @@ -124,6 +129,7 @@ impl TranslationState { real_unreachable_stack_depth: 0, globals: HashMap::new(), heaps: HashMap::new(), + signatures: HashMap::new(), } } @@ -134,6 +140,7 @@ impl TranslationState { self.real_unreachable_stack_depth = 0; self.globals.clear(); self.heaps.clear(); + self.signatures.clear(); } /// Initialize the state for compiling a function with the given signature. @@ -157,6 +164,11 @@ impl TranslationState { self.stack.push(val); } + /// Push multiple values. + pub fn pushn(&mut self, vals: &[Value]) { + self.stack.extend_from_slice(vals); + } + /// Pop one value. pub fn pop1(&mut self) -> Value { self.stack.pop().unwrap() @@ -182,6 +194,19 @@ impl TranslationState { (v1, v2, v3) } + /// Pop the top `n` values on the stack. + /// + /// The popped values are not returned. Use `peekn` to look at them before popping. + pub fn popn(&mut self, n: usize) { + let new_len = self.stack.len() - n; + self.stack.truncate(new_len); + } + + /// Peek at the top `n` values on the stack in the order they were pushed. + pub fn peekn(&self, n: usize) -> &[Value] { + &self.stack[self.stack.len() - n..] + } + // Push a block on the control stack. pub fn push_block(&mut self, following_code: Ebb, result_types: Vec) { self.control_stack.push(ControlStackFrame::Block { @@ -255,4 +280,28 @@ impl TranslationState { || environ.make_heap(func, index), ) } + + /// Get the `SigRef` reference that should be used to make an indirect call with signature + /// `index`. Also return the number of WebAssembly arguments in the signature. + /// + /// Create the signature if necessary. + pub fn get_indirect_sig( + &mut self, + func: &mut ir::Function, + index: u32, + environ: &FE, + ) -> (ir::SigRef, usize) { + let index = index as MemoryIndex; + *self.signatures.entry(index).or_insert_with(|| { + let sig = environ.make_indirect_sig(func, index); + // Count the number of normal arguments. + // The environment is allowed to add special purpose arguments to the signature. + let args = func.dfg.signatures[sig] + .argument_types + .iter() + .filter(|arg| arg.purpose == ir::ArgumentPurpose::Normal) + .count(); + (sig, args) + }) + } } From 45b093ea596a8397360e2bd0f2db06a0ee273a08 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 6 Sep 2017 08:50:57 -0700 Subject: [PATCH 1067/3084] Use `write_all()` to write to write an entire string rather than `write()`. https://github.com/rust-lang-nursery/rust-clippy/wiki#unused_io_amount --- cranelift/src/cton-util.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 3d9b979e2f..bd104e9d5f 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -96,7 +96,7 @@ fn main() { if !msg.ends_with('\n') { msg.push('\n'); } - io::stderr().write(msg.as_bytes()).unwrap(); + io::stderr().write_all(msg.as_bytes()).unwrap(); process::exit(1); } } From dc2bee9cef874eeca4779d072f00ccabb57ccbd9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 6 Sep 2017 11:44:52 -0700 Subject: [PATCH 1068/3084] Add a FuncEnvironment::make_direct_func() callback. This allows the environment to control the signatures used for direct function calls. The signature and calling convention may depend on whether the function is imported or local. Also add WasmRuntime::declare_func_{import,type} to notify the runtime about imported and local functions. This is necessary so the runtime knows what function indexes are referring to . Since imported and local functions are now declared to the runtime, it is no longer necessary to return hashes mapping between WebAssembly indexes and Cretonne entities. Also stop return null entries for the imported functions in the TranslationResult. Just return a vector of local functions. --- cranelift/src/wasm.rs | 15 +--- lib/wasm/src/code_translator.rs | 130 +++------------------------- lib/wasm/src/lib.rs | 5 +- lib/wasm/src/module_translator.rs | 61 +++---------- lib/wasm/src/runtime/dummy.rs | 50 +++++++++++ lib/wasm/src/runtime/spec.rs | 19 ++++ lib/wasm/src/sections_translator.rs | 20 ++++- lib/wasm/src/state.rs | 47 +++++++--- lib/wasm/src/translation_utils.rs | 18 ---- lib/wasm/tests/testsuite.rs | 12 +-- 10 files changed, 154 insertions(+), 223 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 87efd7d629..0dda98b524 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -5,7 +5,7 @@ //! IL. Can also executes the `start` function of the module by laying out the memories, globals //! and tables, then emitting the translated code with hardcoded addresses to memory. -use cton_wasm::{translate_module, FunctionTranslation, DummyRuntime, WasmRuntime}; +use cton_wasm::{translate_module, DummyRuntime, WasmRuntime}; use std::path::PathBuf; use cretonne::loop_analysis::LoopAnalysis; use cretonne::flowgraph::ControlFlowGraph; @@ -149,13 +149,9 @@ fn handle_module( vprint!(flag_verbose, "Checking... "); terminal.reset().unwrap(); for func in &translation.functions { - let il = match *func { - FunctionTranslation::Import() => continue, - FunctionTranslation::Code { ref il, .. } => il.clone(), - }; - match verifier::verify_function(&il, None) { + match verifier::verify_function(func, None) { Ok(()) => (), - Err(err) => return Err(pretty_verifier_error(&il, None, err)), + Err(err) => return Err(pretty_verifier_error(func, None, err)), } } terminal.fg(term::color::GREEN).unwrap(); @@ -167,10 +163,7 @@ fn handle_module( vprint!(flag_verbose, "Optimizing... "); terminal.reset().unwrap(); for func in &translation.functions { - let mut il = match *func { - FunctionTranslation::Import() => continue, - FunctionTranslation::Code { ref il, .. } => il.clone(), - }; + let mut il = func.clone(); let mut loop_analysis = LoopAnalysis::new(); let mut cfg = ControlFlowGraph::new(); cfg.compute(&il); diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index ee3c6c6f57..59bb306817 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -21,40 +21,19 @@ //! //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as //! argument. -use cretonne::ir::{self, Function, Signature, Type, InstBuilder, FunctionName, Ebb, FuncRef, - SigRef, ExtFuncData, MemFlags}; +use cretonne::ir::{self, Function, Signature, Type, InstBuilder, FunctionName, Ebb, MemFlags}; use cretonne::ir::types::*; use cretonne::ir::immediates::{Ieee32, Ieee64}; use cretonne::ir::condcodes::{IntCC, FloatCC}; use cton_frontend::{ILBuilder, FunctionBuilder}; use wasmparser::{Parser, ParserState, Operator, WasmDecoder, MemoryImmediate}; use translation_utils::{f32_translation, f64_translation, type_to_type, translate_type, Local, - FunctionIndex, SignatureIndex}; + FunctionIndex}; use state::{TranslationState, ControlStackFrame}; use std::collections::HashMap; use runtime::{FuncEnvironment, GlobalValue, WasmRuntime}; use std::u32; - -/// Holds mappings between the function and signatures indexes in the Wasm module and their -/// references as imports of the Cretonne IL function. -#[derive(Clone, Debug)] -pub struct FunctionImports { - /// Mappings index in function index space -> index in function local imports - pub functions: HashMap, - /// Mappings index in signature index space -> index in signature local imports - pub signatures: HashMap, -} - -impl FunctionImports { - fn new() -> Self { - Self { - functions: HashMap::new(), - signatures: HashMap::new(), - } - } -} - /// Returns a well-formed Cretonne IL function from a wasm function body and a signature. pub fn translate_function_body( parser: &mut Parser, @@ -62,11 +41,9 @@ pub fn translate_function_body( sig: &Signature, locals: &[(usize, Type)], exports: &Option>, - signatures: &[Signature], - functions: &[SignatureIndex], il_builder: &mut ILBuilder, runtime: &mut WasmRuntime, -) -> Result<(Function, FunctionImports), String> { +) -> Result { runtime.next_function(); // First we build the Function object with its name and signature let mut func = Function::new(); @@ -77,7 +54,6 @@ pub fn translate_function_body( func.name = FunctionName::new(name.clone()); } } - let mut func_imports = FunctionImports::new(); // We introduce an arbitrary scope for the FunctionBuilder object { let mut builder = FunctionBuilder::new(&mut func, il_builder); @@ -121,16 +97,7 @@ pub fn translate_function_body( if state.in_unreachable_code() { translate_unreachable_operator(op, &mut builder, &mut state) } else { - translate_operator( - op, - &mut builder, - runtime, - &mut state, - functions, - signatures, - exports, - &mut func_imports, - ) + translate_operator(op, &mut builder, &mut state, runtime) } } @@ -160,7 +127,7 @@ pub fn translate_function_body( state.stack.truncate(cut_index); } } - Ok((func, func_imports)) + Ok(func) } /// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted @@ -168,12 +135,8 @@ pub fn translate_function_body( fn translate_operator( op: &Operator, builder: &mut FunctionBuilder, - runtime: &mut WasmRuntime, state: &mut TranslationState, - functions: &[SignatureIndex], - signatures: &[Signature], - exports: &Option>, - func_imports: &mut FunctionImports, + runtime: &mut WasmRuntime, ) { // This big match treats all Wasm code operators. match *op { @@ -465,25 +428,13 @@ fn translate_operator( * argument referring to an index in the external functions table of the module. ************************************************************************************/ Operator::Call { function_index } => { - let args_num = args_count(function_index as usize, functions, signatures); - let cut_index = state.stack.len() - args_num; - let internal_function_index = find_function_import( - function_index as usize, - builder, - func_imports, - functions, - exports, - signatures, - ); - let call_inst = builder.ins().call( - internal_function_index, - &state.stack[cut_index..], - ); - state.stack.truncate(cut_index); + let (fref, num_args) = state.get_direct_func(builder.func, function_index, runtime); + // TODO: Let the function environment override the call instruction. It may want to add + // arguments. + let call_inst = builder.ins().call(fref, &state.peekn(num_args)); + state.popn(num_args); let ret_values = builder.inst_results(call_inst); - for val in ret_values { - state.push1(*val); - } + state.pushn(ret_values); } Operator::CallIndirect { index, @@ -1098,60 +1049,3 @@ fn translate_store( base, ); } -fn args_count( - index: FunctionIndex, - functions: &[SignatureIndex], - signatures: &[Signature], -) -> usize { - signatures[functions[index]].argument_types.len() -} - -// Given an index in the function index space, search for it in the function imports and if it is -// not there add it to the function imports. -fn find_function_import( - index: FunctionIndex, - builder: &mut FunctionBuilder, - func_imports: &mut FunctionImports, - functions: &[SignatureIndex], - exports: &Option>, - signatures: &[Signature], -) -> FuncRef { - if let Some(local_index) = func_imports.functions.get(&index) { - return *local_index; - } - // We have to import the function - let sig_index = functions[index]; - if let Some(local_sig_index) = func_imports.signatures.get(&sig_index) { - let local_func_index = builder.import_function(ExtFuncData { - name: match *exports { - None => FunctionName::new(""), - Some(ref exports) => { - match exports.get(&index) { - None => FunctionName::new(""), - Some(name) => FunctionName::new(name.clone()), - } - } - }, - signature: *local_sig_index, - }); - func_imports.functions.insert(index, local_func_index); - return local_func_index; - } - // We have to import the signature - let sig_local_index = builder.import_signature(signatures[sig_index].clone()); - func_imports.signatures.insert(sig_index, sig_local_index); - let local_func_index = builder.import_function(ExtFuncData { - name: match *exports { - None => FunctionName::new(""), - Some(ref exports) => { - match exports.get(&index) { - None => FunctionName::new(""), - Some(name) => FunctionName::new(name.clone()), - } - } - }, - signature: sig_local_index, - }); - func_imports.functions.insert(index, local_func_index); - local_func_index -} diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index bb750572a1..22fb1c9291 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -23,8 +23,7 @@ mod sections_translator; mod state; mod translation_utils; -pub use module_translator::{translate_module, TranslationResult, FunctionTranslation, - ImportMappings}; -pub use runtime::{WasmRuntime, DummyRuntime}; +pub use module_translator::{translate_module, TranslationResult}; +pub use runtime::{FuncEnvironment, WasmRuntime, DummyRuntime}; pub use translation_utils::{Local, FunctionIndex, GlobalIndex, TableIndex, MemoryIndex, RawByte, MemoryAddress, SignatureIndex, Global, GlobalInit, Table, Memory}; diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index c2346e52b3..113330d4cf 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -5,8 +5,8 @@ use sections_translator::{SectionParsingError, parse_function_signatures, parse_ parse_function_section, parse_export_section, parse_memory_section, parse_global_section, parse_table_section, parse_elements_section, parse_data_section}; -use translation_utils::{type_to_type, Import, SignatureIndex, FunctionIndex, invert_hashmaps}; -use cretonne::ir::{Function, Type, FuncRef, SigRef}; +use translation_utils::{type_to_type, Import, SignatureIndex, FunctionIndex}; +use cretonne::ir::{Function, Type}; use code_translator::translate_function_body; use cton_frontend::ILBuilder; use std::collections::HashMap; @@ -15,45 +15,14 @@ use runtime::WasmRuntime; /// Output of the [`translate_module`](fn.translate_module.html) function. pub struct TranslationResult { /// The translated functions. - pub functions: Vec, + pub functions: Vec, /// When present, the index of the function defined as `start` of the module. + /// + /// Note that this is a WebAssembly function index and not an index into the `functions` vector + /// above. The imported functions are numbered before the local functions. pub start_index: Option, } -/// A function in a WebAssembly module can be either imported, or defined inside it. -#[derive(Clone)] -pub enum FunctionTranslation { - /// A function defined inside the WebAssembly module. - Code { - /// The translation in Cretonne IL. - il: Function, - /// The mappings between Cretonne imports and indexes in the function index space. - imports: ImportMappings, - }, - /// An imported function. - Import(), -} - -#[derive(Clone, Debug)] -/// Mappings describing the relations between imports of the Cretonne IL functions and the -/// functions in the WebAssembly module. -pub struct ImportMappings { - /// Find the index of a function in the WebAssembly module thanks to a `FuncRef`. - pub functions: HashMap, - /// Find the index of a signature in the WebAssembly module thanks to a `SigRef`. - pub signatures: HashMap, -} - -impl ImportMappings { - /// Create a new empty `ImportMappings`. - pub fn new() -> Self { - Self { - functions: HashMap::new(), - signatures: HashMap::new(), - } - } -} - /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IL /// [`Function`](../cretonne/ir/function/struct.Function.html). /// Returns the functions and also the mappings for imported functions and signature between the @@ -76,7 +45,6 @@ pub fn translate_module( let mut exports: Option> = None; let mut next_input = ParserInput::Default; let mut function_index: FunctionIndex = 0; - let mut function_imports_count = 0; let mut start_index: Option = None; loop { match *parser.read_with_input(next_input) { @@ -90,7 +58,7 @@ pub fn translate_module( next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Import, .. } => { - match parse_import_section(&mut parser) { + match parse_import_section(&mut parser, runtime) { Ok(imps) => { for import in imps { match import { @@ -121,11 +89,10 @@ pub fn translate_module( return Err(format!("wrong content in the import section: {}", s)) } } - function_imports_count = function_index; next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Function, .. } => { - match parse_function_section(&mut parser) { + match parse_function_section(&mut parser, runtime) { Ok(funcs) => { match functions { None => functions = Some(funcs), @@ -233,8 +200,7 @@ pub fn translate_module( None => return Err(String::from("missing a function section")), Some(functions) => functions, }; - let mut il_functions: Vec = Vec::new(); - il_functions.resize(function_imports_count, FunctionTranslation::Import()); + let mut il_functions: Vec = Vec::new(); let mut il_builder = ILBuilder::new(); runtime.begin_translation(); loop { @@ -263,17 +229,10 @@ pub fn translate_module( &signature, &locals, &exports, - &signatures, - &functions, &mut il_builder, runtime, ) { - Ok((il_func, imports)) => { - il_functions.push(FunctionTranslation::Code { - il: il_func, - imports: invert_hashmaps(&imports), - }) - } + Ok(il_func) => il_functions.push(il_func), Err(s) => return Err(s), } function_index += 1; diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 7ff9747dc6..1bb48257c9 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -12,6 +12,12 @@ pub struct DummyRuntime { // Unprocessed signatures exactly as provided by `declare_signature()`. signatures: Vec, globals: Vec, + + // Types of functions, imported and local. + func_types: Vec, + + // Names of imported functions. + imported_funcs: Vec, } impl DummyRuntime { @@ -20,6 +26,8 @@ impl DummyRuntime { Self { signatures: Vec::new(), globals: Vec::new(), + func_types: Vec::new(), + imported_funcs: Vec::new(), } } } @@ -53,6 +61,20 @@ impl FuncEnvironment for DummyRuntime { // signature index arguments. func.dfg.signatures.push(self.signatures[index].clone()) } + + fn make_direct_func(&self, func: &mut ir::Function, index: FunctionIndex) -> ir::FuncRef { + let sigidx = self.func_types[index]; + // A real implementation would probably add a `vmctx` argument. + // And maybe attempt some signature de-duplication. + let signature = func.dfg.signatures.push(self.signatures[sigidx].clone()); + + let name = match self.imported_funcs.get(index) { + Some(name) => name.clone(), + None => ir::FunctionName::new(format!("localfunc{}", index)), + }; + + func.dfg.ext_funcs.push(ir::ExtFuncData { name, signature }) + } } impl WasmRuntime for DummyRuntime { @@ -77,6 +99,25 @@ impl WasmRuntime for DummyRuntime { self.signatures.push(sig.clone()); } + fn declare_func_import(&mut self, sig_index: SignatureIndex, module: &[u8], field: &[u8]) { + assert_eq!( + self.func_types.len(), + self.imported_funcs.len(), + "Imported functions must be declared first" + ); + self.func_types.push(sig_index); + + let mut name = Vec::new(); + name.extend(module.iter().cloned().map(name_fold)); + name.push(b'_'); + name.extend(field.iter().cloned().map(name_fold)); + self.imported_funcs.push(ir::FunctionName::new(name)); + } + + fn declare_func_type(&mut self, sig_index: SignatureIndex) { + self.func_types.push(sig_index); + } + fn declare_global(&mut self, global: Global) { self.globals.push(global); } @@ -106,3 +147,12 @@ impl WasmRuntime for DummyRuntime { // We do nothing } } + +// Generate characters suitable for printable `FuncName`s. +fn name_fold(c: u8) -> u8 { + if (c as char).is_alphanumeric() { + c + } else { + b'_' + } +} diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 3dee0833bf..b261181491 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -51,6 +51,19 @@ pub trait FuncEnvironment { /// The signature will only be used for indirect calls, even if the module has direct function /// calls with the same WebAssembly type. fn make_indirect_sig(&self, func: &mut ir::Function, index: SignatureIndex) -> ir::SigRef; + + /// Set up an external function definition in the preamble of `func` that can be used to + /// directly call the function `index`. + /// + /// The index space covers both imported functions and functions defined in the current module. + /// + /// The function's signature may contain additional arguments needed for a direct call, but the + /// arguments marked as `ArgumentPurpose::Normal` must correspond to the WebAssembly signature + /// arguments. + /// + /// The function's signature will only be used for direct calls, even if the module has + /// indirect calls with the same WebAssembly type. + fn make_direct_func(&self, func: &mut ir::Function, index: FunctionIndex) -> ir::FuncRef; } /// An object satisfyng the `WasmRuntime` trait can be passed as argument to the @@ -60,6 +73,12 @@ pub trait WasmRuntime: FuncEnvironment { /// Declares a function signature to the runtime. fn declare_signature(&mut self, sig: &ir::Signature); + /// Declares a function import to the runtime. + fn declare_func_import(&mut self, sig_index: SignatureIndex, module: &[u8], field: &[u8]); + + /// Declares the type (signature) of a local function in the module. + fn declare_func_type(&mut self, sig_index: SignatureIndex); + /// Declares a global to the runtime. fn declare_global(&mut self, global: Global); /// Declares a table to the runtime. diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 591146c716..8c8b7aac71 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -64,13 +64,21 @@ pub fn parse_function_signatures( } /// Retrieves the imports from the imports section of the binary. -pub fn parse_import_section(parser: &mut Parser) -> Result, SectionParsingError> { +pub fn parse_import_section( + parser: &mut Parser, + runtime: &mut WasmRuntime, +) -> Result, SectionParsingError> { let mut imports = Vec::new(); loop { match *parser.read() { ParserState::ImportSectionEntry { - ty: ImportSectionEntryType::Function(sig), .. - } => imports.push(Import::Function { sig_index: sig }), + ty: ImportSectionEntryType::Function(sig), + module, + field, + } => { + runtime.declare_func_import(sig as SignatureIndex, module, field); + imports.push(Import::Function { sig_index: sig }); + } ParserState::ImportSectionEntry { ty: ImportSectionEntryType::Memory(MemoryType { limits: ref memlimits }), .. } => { @@ -110,11 +118,15 @@ pub fn parse_import_section(parser: &mut Parser) -> Result, SectionP /// Retrieves the correspondances between functions and signatures from the function section pub fn parse_function_section( parser: &mut Parser, + runtime: &mut WasmRuntime, ) -> Result, SectionParsingError> { let mut funcs = Vec::new(); loop { match *parser.read() { - ParserState::FunctionSectionEntry(sigindex) => funcs.push(sigindex as SignatureIndex), + ParserState::FunctionSectionEntry(sigindex) => { + runtime.declare_func_type(sigindex as SignatureIndex); + funcs.push(sigindex as SignatureIndex); + } ParserState::EndSection => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index 5747ceda72..cfe06e081a 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -6,7 +6,7 @@ use cretonne::ir::{self, Ebb, Inst, Type, Value}; use runtime::{FuncEnvironment, GlobalValue}; use std::collections::HashMap; -use translation_utils::{GlobalIndex, MemoryIndex, SignatureIndex}; +use translation_utils::{GlobalIndex, MemoryIndex, SignatureIndex, FunctionIndex}; /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following /// fields: @@ -118,6 +118,11 @@ pub struct TranslationState { // `FuncEnvironment::make_indirect_sig()`. // Stores both the signature reference and the number of WebAssembly arguments signatures: HashMap, + + // Imported and local functions that have been created by + // `FuncEnvironment::make_direct_func()`. + // Stores both the function reference and the number of WebAssembly arguments + functions: HashMap, } impl TranslationState { @@ -130,6 +135,7 @@ impl TranslationState { globals: HashMap::new(), heaps: HashMap::new(), signatures: HashMap::new(), + functions: HashMap::new(), } } @@ -141,6 +147,7 @@ impl TranslationState { self.globals.clear(); self.heaps.clear(); self.signatures.clear(); + self.functions.clear(); } /// Initialize the state for compiling a function with the given signature. @@ -291,17 +298,37 @@ impl TranslationState { index: u32, environ: &FE, ) -> (ir::SigRef, usize) { - let index = index as MemoryIndex; + let index = index as SignatureIndex; *self.signatures.entry(index).or_insert_with(|| { let sig = environ.make_indirect_sig(func, index); - // Count the number of normal arguments. - // The environment is allowed to add special purpose arguments to the signature. - let args = func.dfg.signatures[sig] - .argument_types - .iter() - .filter(|arg| arg.purpose == ir::ArgumentPurpose::Normal) - .count(); - (sig, args) + (sig, normal_args(&func.dfg.signatures[sig])) + }) + } + + /// Get the `FuncRef` reference that should be used to make a direct call to function + /// `index`. Also return the number of WebAssembly arguments in the signature. + /// + /// Create the function reference if necessary. + pub fn get_direct_func( + &mut self, + func: &mut ir::Function, + index: u32, + environ: &FE, + ) -> (ir::FuncRef, usize) { + let index = index as FunctionIndex; + *self.functions.entry(index).or_insert_with(|| { + let fref = environ.make_direct_func(func, index); + let sig = func.dfg.ext_funcs[fref].signature; + (fref, normal_args(&func.dfg.signatures[sig])) }) } } + +/// Count the number of normal arguments in a signature. +/// Exclude special-purpose arguments that represent runtime stuff and not WebAssembly arguments. +fn normal_args(sig: &ir::Signature) -> usize { + sig.argument_types + .iter() + .filter(|arg| arg.purpose == ir::ArgumentPurpose::Normal) + .count() +} diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index d378864e5c..e95a624f83 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -2,8 +2,6 @@ use wasmparser; use cretonne; use std::u32; -use code_translator; -use module_translator; /// Index of a function (imported or defined) inside the WebAssembly module. pub type FunctionIndex = usize; @@ -135,19 +133,3 @@ pub fn translate_type(ty: wasmparser::Type) -> Result, ( _ => panic!("unsupported return value type"), } } - -/// Inverts the key-value relation in the imports hashmap. Indeed, these hashmaps are built by -/// feeding the function indexes in the module but are used by the runtime with the `FuncRef` as -/// keys. -pub fn invert_hashmaps( - imports: &code_translator::FunctionImports, -) -> module_translator::ImportMappings { - let mut new_imports = module_translator::ImportMappings::new(); - for (func_index, func_ref) in &imports.functions { - new_imports.functions.insert(*func_ref, *func_index); - } - for (sig_index, sig_ref) in &imports.signatures { - new_imports.signatures.insert(*sig_ref, *sig_index); - } - new_imports -} diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/testsuite.rs index 91f514fde4..9e59b2bab7 100644 --- a/lib/wasm/tests/testsuite.rs +++ b/lib/wasm/tests/testsuite.rs @@ -1,7 +1,7 @@ extern crate cton_wasm; extern crate cretonne; -use cton_wasm::{translate_module, FunctionTranslation, DummyRuntime, WasmRuntime}; +use cton_wasm::{translate_module, DummyRuntime, WasmRuntime}; use std::path::PathBuf; use std::fs::File; use std::error::Error; @@ -69,14 +69,10 @@ fn handle_module(path: PathBuf) -> Result<(), String> { } } }; - for func in translation.functions { - let il = match func { - FunctionTranslation::Import() => continue, - FunctionTranslation::Code { ref il, .. } => il.clone(), - }; - match verifier::verify_function(&il, None) { + for func in &translation.functions { + match verifier::verify_function(func, None) { Ok(()) => (), - Err(err) => return Err(pretty_verifier_error(&il, None, err)), + Err(err) => return Err(pretty_verifier_error(func, None, err)), } } Ok(()) From 26048c2eccb4a5921f52bd450debff524e1c63d1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 6 Sep 2017 15:06:28 -0700 Subject: [PATCH 1069/3084] Move WasmRuntime::translate_call_indirect() into FuncEnvironment. Add two new arguments: - table_index is the WebAssembly table referenced in the indirect call. - sig_index is the WebAssembly signature index. We still have the SigRef that was created by make_indirect_sig(), but the WebAssembly signature index may be needed for detecting type mismatches at runtime. Change the insertion location to a plain FuncCursor rather than a FunctionBuilder. The fact that cretonne-wasm uses FunctionBuilder should be an implementation detail, and the callbacks don't need to access WebAssembly locals, so they don't need the extended interface. Add a FunctionBuilder::cursor() method which creates a FuncCursor for inserting instructions in the current EBB. Also add a FuncEnvironment::translate_call() method which allows the environment to override direct calls the same way as indirect calls. --- lib/cretonne/src/cursor.rs | 6 +++++ lib/cretonne/src/ir/layout.rs | 21 +++++++++++++++ lib/cretonne/src/lib.rs | 2 +- lib/frontend/src/frontend.rs | 45 ++++++++++++++++++++----------- lib/wasm/src/code_translator.rs | 36 ++++++++++++++----------- lib/wasm/src/runtime/dummy.rs | 25 +++++++++-------- lib/wasm/src/runtime/spec.rs | 48 ++++++++++++++++++++++++++------- 7 files changed, 129 insertions(+), 54 deletions(-) diff --git a/lib/cretonne/src/cursor.rs b/lib/cretonne/src/cursor.rs index 8de400a1f1..442d820c09 100644 --- a/lib/cretonne/src/cursor.rs +++ b/lib/cretonne/src/cursor.rs @@ -19,6 +19,8 @@ pub use ir::layout::Cursor as LayoutCursor; /// encoding. pub struct FuncCursor<'f> { pos: CursorPosition, + + /// The referenced function. pub func: &'f mut ir::Function, } @@ -79,7 +81,11 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut FuncCursor<'f> { pub struct EncCursor<'f> { pos: CursorPosition, built_inst: Option, + + /// The referenced function. pub func: &'f mut ir::Function, + + /// The target ISA that will be used to encode instructions. pub isa: &'f TargetIsa, } diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 7f3247c080..33fbaf943d 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -728,6 +728,27 @@ pub trait CursorBase { self } + /// Rebuild this cursor positioned at the bottom of `ebb`. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// fn edit_func(func: &mut Function, ebb: Ebb) { + /// let mut pos = Cursor::new(&mut func.layout).at_bottom(ebb); + /// + /// // Use `pos`... + /// } + /// ``` + fn at_bottom(mut self, ebb: Ebb) -> Self + where + Self: Sized, + { + self.goto_bottom(ebb); + self + } + /// Get the EBB corresponding to the current position. fn current_ebb(&self) -> Option { use self::CursorPosition::*; diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index a63b00b3ae..6949f92f5f 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -17,6 +17,7 @@ pub mod entity; pub mod binemit; pub mod bitset; +pub mod cursor; pub mod dominator_tree; pub mod flowgraph; pub mod ir; @@ -31,7 +32,6 @@ pub mod verifier; mod abi; mod constant_hash; mod context; -mod cursor; mod iterators; mod legalizer; mod licm; diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 8eb90dd1d6..4f6d4ef640 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -1,4 +1,5 @@ //! A frontend for building Cretonne IL from other languages. +use cretonne::cursor::{Cursor, FuncCursor}; use cretonne::ir::{Ebb, Type, Value, Function, Inst, JumpTable, StackSlot, JumpTableData, StackSlotData, DataFlowGraph, InstructionData, ExtFuncData, FuncRef, SigRef, Signature, InstBuilderBase}; @@ -107,24 +108,11 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'short mut DataFlowGraph) { if data.opcode().is_return() { self.builder - .check_return_args(data.arguments(&self.builder.func.dfg.value_lists)) + .check_return_args(data.arguments(&self.builder.func.dfg.value_lists)); } // We only insert the Ebb in the layout when an instruction is added to it - if self.builder.builder.ebbs[self.builder.position.ebb].pristine { - if !self.builder - .func - .layout - .is_ebb_inserted(self.builder.position.ebb) { - self.builder - .func - .layout - .append_ebb(self.builder.position.ebb); - } - self.builder.builder.ebbs[self.builder.position.ebb].pristine = false; - } else { - debug_assert!(!self.builder.builder.ebbs[self.builder.position.ebb].filled, - "you cannot add an instruction to a block already filled"); - } + self.builder.ensure_inserted_ebb(); + let inst = self.builder.func.dfg.make_inst(data.clone()); self.builder.func.dfg.make_inst_results(inst, ctrl_typevar); self.builder.func.layout.append_inst(inst, self.ebb); @@ -372,6 +360,31 @@ where let ebb = self.position.ebb; FuncInstBuilder::new(self, ebb) } + + /// Make sure that the current EBB is inserted in the layout. + fn ensure_inserted_ebb(&mut self) { + let ebb = self.position.ebb; + if self.builder.ebbs[ebb].pristine { + if !self.func.layout.is_ebb_inserted(ebb) { + self.func.layout.append_ebb(ebb); + } + self.builder.ebbs[ebb].pristine = false; + } else { + debug_assert!( + !self.builder.ebbs[ebb].filled, + "you cannot add an instruction to a block already filled" + ); + } + } + + /// Returns a `FuncCursor` pointed at the current position ready for inserting instructions. + /// + /// This can be used to insert SSA code that doesn't need to access locals and that doesn't + /// need to know about `FunctionBuilder` at all. + pub fn cursor<'f>(&'f mut self) -> FuncCursor<'f> { + self.ensure_inserted_ebb(); + FuncCursor::new(self.func).at_bottom(self.position.ebb) + } } /// All the functions documented in the previous block are write-only and help you build a valid diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 59bb306817..02cd7e1b10 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -27,8 +27,8 @@ use cretonne::ir::immediates::{Ieee32, Ieee64}; use cretonne::ir::condcodes::{IntCC, FloatCC}; use cton_frontend::{ILBuilder, FunctionBuilder}; use wasmparser::{Parser, ParserState, Operator, WasmDecoder, MemoryImmediate}; -use translation_utils::{f32_translation, f64_translation, type_to_type, translate_type, Local, - FunctionIndex}; +use translation_utils::{f32_translation, f64_translation, type_to_type, translate_type, Local}; +use translation_utils::{TableIndex, SignatureIndex, FunctionIndex}; use state::{TranslationState, ControlStackFrame}; use std::collections::HashMap; use runtime::{FuncEnvironment, GlobalValue, WasmRuntime}; @@ -429,26 +429,30 @@ fn translate_operator( ************************************************************************************/ Operator::Call { function_index } => { let (fref, num_args) = state.get_direct_func(builder.func, function_index, runtime); - // TODO: Let the function environment override the call instruction. It may want to add - // arguments. - let call_inst = builder.ins().call(fref, &state.peekn(num_args)); + let call = runtime.translate_call( + builder.cursor(), + function_index as FunctionIndex, + fref, + state.peekn(num_args), + ); state.popn(num_args); - let ret_values = builder.inst_results(call_inst); - state.pushn(ret_values); + state.pushn(builder.func.dfg.inst_results(call)); } - Operator::CallIndirect { - index, - table_index: _, - } => { + Operator::CallIndirect { index, table_index } => { // `index` is the index of the function's signature and `table_index` is the index of // the table to search the function in. - // TODO: Have runtime support for tables. let (sigref, num_args) = state.get_indirect_sig(builder.func, index, runtime); - let index_val = state.pop1(); - let ret_values = - runtime.translate_call_indirect(builder, sigref, index_val, &state.peekn(num_args)); + let callee = state.pop1(); + let call = runtime.translate_call_indirect( + builder.cursor(), + table_index as TableIndex, + index as SignatureIndex, + sigref, + callee, + &state.peekn(num_args), + ); state.popn(num_args); - state.pushn(ret_values); + state.pushn(builder.func.dfg.inst_results(call)); } /******************************* Memory management *********************************** * Memory management is handled by runtime. It is usually translated into calls to diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 1bb48257c9..178d39c5d1 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -2,8 +2,9 @@ use runtime::{FuncEnvironment, GlobalValue, WasmRuntime}; use translation_utils::{Local, Global, Memory, Table, GlobalIndex, TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; use cton_frontend::FunctionBuilder; -use cretonne::ir::{self, Value, InstBuilder, SigRef}; +use cretonne::ir::{self, Value, InstBuilder}; use cretonne::ir::types::*; +use cretonne::cursor::FuncCursor; /// This runtime implementation is a "naïve" one, doing essentially nothing and emitting /// placeholders when forced to. Don't try to execute code translated with this runtime, it is @@ -75,6 +76,18 @@ impl FuncEnvironment for DummyRuntime { func.dfg.ext_funcs.push(ir::ExtFuncData { name, signature }) } + + fn translate_call_indirect( + &self, + mut pos: FuncCursor, + _table_index: TableIndex, + _sig_index: SignatureIndex, + sig_ref: ir::SigRef, + callee: ir::Value, + call_args: &[ir::Value], + ) -> ir::Inst { + pos.ins().call_indirect(sig_ref, callee, call_args) + } } impl WasmRuntime for DummyRuntime { @@ -84,16 +97,6 @@ impl WasmRuntime for DummyRuntime { fn translate_current_memory(&mut self, builder: &mut FunctionBuilder) -> Value { builder.ins().iconst(I32, -1) } - fn translate_call_indirect<'a>( - &self, - builder: &'a mut FunctionBuilder, - sig_ref: SigRef, - index_val: Value, - call_args: &[Value], - ) -> &'a [Value] { - let call_inst = builder.ins().call_indirect(sig_ref, index_val, call_args); - builder.inst_results(call_inst) - } fn declare_signature(&mut self, sig: &ir::Signature) { self.signatures.push(sig.clone()); diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index b261181491..5750fe5520 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -1,7 +1,8 @@ //! All the runtime support necessary for the wasm to cretonne translation is formalized by the //! trait `WasmRuntime`. use cton_frontend::FunctionBuilder; -use cretonne::ir::{self, Value, SigRef}; +use cretonne::ir::{self, Value, InstBuilder}; +use cretonne::cursor::FuncCursor; use translation_utils::{Local, SignatureIndex, FunctionIndex, TableIndex, GlobalIndex, MemoryIndex, Global, Table, Memory}; @@ -64,6 +65,42 @@ pub trait FuncEnvironment { /// The function's signature will only be used for direct calls, even if the module has /// indirect calls with the same WebAssembly type. fn make_direct_func(&self, func: &mut ir::Function, index: FunctionIndex) -> ir::FuncRef; + + /// Translate a `call_indirect` WebAssembly instruction at `pos`. + /// + /// Insert instructions at `pos` for an indirect call to the function `callee` in the table + /// `table_index` with WebAssembly signature `sig_index`. The `callee` value will have type + /// `i32`. + /// + /// The signature `sig_ref` was previously created by `make_indirect_sig()`. + /// + /// Return the call instruction whose results are the WebAssembly return values. + fn translate_call_indirect( + &self, + pos: FuncCursor, + table_index: TableIndex, + sig_index: SignatureIndex, + sig_ref: ir::SigRef, + callee: ir::Value, + call_args: &[ir::Value], + ) -> ir::Inst; + + /// Translate a `call` WebAssembly instruction at `pos`. + /// + /// Insert instructions at `pos` for a direct call to the function `callee_index`. + /// + /// The function reference `callee` was previously created by `make_direct_func()`. + /// + /// Return the call instruction whose results are the WebAssembly return values. + fn translate_call( + &self, + mut pos: FuncCursor, + _callee_index: FunctionIndex, + callee: ir::FuncRef, + call_args: &[ir::Value], + ) -> ir::Inst { + pos.ins().call(callee, call_args) + } } /// An object satisfyng the `WasmRuntime` trait can be passed as argument to the @@ -108,13 +145,4 @@ pub trait WasmRuntime: FuncEnvironment { fn translate_grow_memory(&mut self, builder: &mut FunctionBuilder, val: Value) -> Value; /// Translates a `current_memory` wasm instruction. Returns the size in pages of the memory. fn translate_current_memory(&mut self, builder: &mut FunctionBuilder) -> Value; - /// Translates a `call_indirect` wasm instruction. It involves looking up the value contained - /// it the table at location `index_val` and calling the corresponding function. - fn translate_call_indirect<'a>( - &self, - builder: &'a mut FunctionBuilder, - sig_ref: SigRef, - index_val: Value, - call_args: &[Value], - ) -> &'a [Value]; } From b10faca5340b6fa444194ccb7c3d1318f18e5081 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 6 Sep 2017 15:44:23 -0700 Subject: [PATCH 1070/3084] Move translate_grow_memory and translate_current_memory too. This moves the last instruction-level callbacks into FuncEnvironment such that the trait has all the information required to translate a whole function. Change the position argument to a FuncCursor. This eliminates all exposure of FunctionBuilder, so its use properly becomes an implementation detail. --- lib/wasm/src/code_translator.rs | 25 ++++++++++++++++----- lib/wasm/src/runtime/dummy.rs | 31 ++++++++++++++++--------- lib/wasm/src/runtime/spec.rs | 40 ++++++++++++++++++++++++++------- 3 files changed, 73 insertions(+), 23 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 02cd7e1b10..681eb3b7d0 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -28,7 +28,7 @@ use cretonne::ir::condcodes::{IntCC, FloatCC}; use cton_frontend::{ILBuilder, FunctionBuilder}; use wasmparser::{Parser, ParserState, Operator, WasmDecoder, MemoryImmediate}; use translation_utils::{f32_translation, f64_translation, type_to_type, translate_type, Local}; -use translation_utils::{TableIndex, SignatureIndex, FunctionIndex}; +use translation_utils::{TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; use state::{TranslationState, ControlStackFrame}; use std::collections::HashMap; use runtime::{FuncEnvironment, GlobalValue, WasmRuntime}; @@ -458,12 +458,27 @@ fn translate_operator( * Memory management is handled by runtime. It is usually translated into calls to * special functions. ************************************************************************************/ - Operator::GrowMemory { reserved: _ } => { + Operator::GrowMemory { reserved } => { + // The WebAssembly MVP only supports one linear memory, but we expect the reserved + // argument to be a memory index. + let heap_index = reserved as MemoryIndex; + let heap = state.get_heap(builder.func, reserved, runtime); let val = state.pop1(); - state.push1(runtime.translate_grow_memory(builder, val)); + state.push1(runtime.translate_grow_memory( + builder.cursor(), + heap_index, + heap, + val, + )) } - Operator::CurrentMemory { reserved: _ } => { - state.push1(runtime.translate_current_memory(builder)); + Operator::CurrentMemory { reserved } => { + let heap_index = reserved as MemoryIndex; + let heap = state.get_heap(builder.func, reserved, runtime); + state.push1(runtime.translate_current_memory( + builder.cursor(), + heap_index, + heap, + )); } /******************************* Load instructions *********************************** * Wasm specifies an integer alignment flag but we drop it in Cretonne. diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 178d39c5d1..703b5f164a 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -1,8 +1,7 @@ use runtime::{FuncEnvironment, GlobalValue, WasmRuntime}; -use translation_utils::{Local, Global, Memory, Table, GlobalIndex, TableIndex, SignatureIndex, +use translation_utils::{Global, Memory, Table, GlobalIndex, TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; -use cton_frontend::FunctionBuilder; -use cretonne::ir::{self, Value, InstBuilder}; +use cretonne::ir::{self, InstBuilder}; use cretonne::ir::types::*; use cretonne::cursor::FuncCursor; @@ -88,16 +87,28 @@ impl FuncEnvironment for DummyRuntime { ) -> ir::Inst { pos.ins().call_indirect(sig_ref, callee, call_args) } + + fn translate_grow_memory( + &mut self, + mut pos: FuncCursor, + _index: MemoryIndex, + _heap: ir::Heap, + _val: ir::Value, + ) -> ir::Value { + pos.ins().iconst(I32, -1) + } + + fn translate_current_memory( + &mut self, + mut pos: FuncCursor, + _index: MemoryIndex, + _heap: ir::Heap, + ) -> ir::Value { + pos.ins().iconst(I32, -1) + } } impl WasmRuntime for DummyRuntime { - fn translate_grow_memory(&mut self, builder: &mut FunctionBuilder, _: Value) -> Value { - builder.ins().iconst(I32, -1) - } - fn translate_current_memory(&mut self, builder: &mut FunctionBuilder) -> Value { - builder.ins().iconst(I32, -1) - } - fn declare_signature(&mut self, sig: &ir::Signature) { self.signatures.push(sig.clone()); } diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 5750fe5520..35f8d9ea37 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -1,10 +1,9 @@ //! All the runtime support necessary for the wasm to cretonne translation is formalized by the //! trait `WasmRuntime`. -use cton_frontend::FunctionBuilder; -use cretonne::ir::{self, Value, InstBuilder}; +use cretonne::ir::{self, InstBuilder}; use cretonne::cursor::FuncCursor; -use translation_utils::{Local, SignatureIndex, FunctionIndex, TableIndex, GlobalIndex, - MemoryIndex, Global, Table, Memory}; +use translation_utils::{SignatureIndex, FunctionIndex, TableIndex, GlobalIndex, MemoryIndex, + Global, Table, Memory}; /// The value of a WebAssembly global variable. #[derive(Clone, Copy)] @@ -101,6 +100,35 @@ pub trait FuncEnvironment { ) -> ir::Inst { pos.ins().call(callee, call_args) } + + /// Translate a `grow_memory` WebAssembly instruction. + /// + /// The `index` provided identifies the linear memory to grow, and `heap` is the heap reference + /// returned by `make_heap` for the same index. + /// + /// The `val` value is the requested memory size in pages. + /// + /// Returns the old size (in pages) of the memory. + fn translate_grow_memory( + &mut self, + pos: FuncCursor, + index: MemoryIndex, + heap: ir::Heap, + val: ir::Value, + ) -> ir::Value; + + /// Translates a `current_memory` WebAssembly instruction. + /// + /// The `index` provided identifies the linear memory to query, and `heap` is the heap reference + /// returned by `make_heap` for the same index. + /// + /// Returns the size in pages of the memory. + fn translate_current_memory( + &mut self, + pos: FuncCursor, + index: MemoryIndex, + heap: ir::Heap, + ) -> ir::Value; } /// An object satisfyng the `WasmRuntime` trait can be passed as argument to the @@ -141,8 +169,4 @@ pub trait WasmRuntime: FuncEnvironment { fn begin_translation(&mut self); /// Call this function between each function body translation. fn next_function(&mut self); - /// Translates a `grow_memory` wasm instruction. Returns the old size (in pages) of the memory. - fn translate_grow_memory(&mut self, builder: &mut FunctionBuilder, val: Value) -> Value; - /// Translates a `current_memory` wasm instruction. Returns the size in pages of the memory. - fn translate_current_memory(&mut self, builder: &mut FunctionBuilder) -> Value; } From 062dd41c937078d4661875408770672acf12b1ce Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 6 Sep 2017 15:56:39 -0700 Subject: [PATCH 1071/3084] Don't export the 'Local' type from cton_wasm. This type is not longer used in any public interface, it has become an internal implementation detail. Also remove some unused exported types from the crate. --- lib/wasm/src/lib.rs | 4 ++-- lib/wasm/src/translation_utils.rs | 4 ---- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 22fb1c9291..e334570cfe 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -25,5 +25,5 @@ mod translation_utils; pub use module_translator::{translate_module, TranslationResult}; pub use runtime::{FuncEnvironment, WasmRuntime, DummyRuntime}; -pub use translation_utils::{Local, FunctionIndex, GlobalIndex, TableIndex, MemoryIndex, RawByte, - MemoryAddress, SignatureIndex, Global, GlobalInit, Table, Memory}; +pub use translation_utils::{FunctionIndex, GlobalIndex, TableIndex, MemoryIndex, SignatureIndex, + Global, GlobalInit, Table, Memory}; diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index e95a624f83..2c915f1e43 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -13,10 +13,6 @@ pub type GlobalIndex = usize; pub type MemoryIndex = usize; /// Index of a signature (imported or defined) inside the WebAssembly module. pub type SignatureIndex = usize; -/// Raw byte read from memory. -pub type RawByte = u8; -/// Pointer referring to a memory address. -pub type MemoryAddress = usize; /// WebAssembly import. #[derive(Debug, Clone, Copy)] From d6f6af104b9b3a470fcc60ed39e91f5431e1c84b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 6 Sep 2017 16:24:55 -0700 Subject: [PATCH 1072/3084] Make translate_operator() generic on FuncEnvironment. This makes it clear that this function only uses the FuncEnvironment trait and not WasmRuntime. Also make the translate_{grow,current}_memory() methods take &self instead of &mut self. The &mut was left on there by accident. --- lib/wasm/src/code_translator.rs | 64 ++++++++++++++++----------------- lib/wasm/src/runtime/dummy.rs | 4 +-- lib/wasm/src/runtime/spec.rs | 4 +-- 3 files changed, 36 insertions(+), 36 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 681eb3b7d0..2cb61d3239 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -132,11 +132,11 @@ pub fn translate_function_body( /// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted /// a return. -fn translate_operator( +fn translate_operator( op: &Operator, builder: &mut FunctionBuilder, state: &mut TranslationState, - runtime: &mut WasmRuntime, + environ: &FE, ) { // This big match treats all Wasm code operators. match *op { @@ -157,10 +157,10 @@ fn translate_operator( * `get_global` and `set_global` are handled by the runtime. ***********************************************************************************/ Operator::GetGlobal { global_index } => { - let val = match state.get_global(builder.func, global_index, runtime) { + let val = match state.get_global(builder.func, global_index, environ) { GlobalValue::Const(val) => val, GlobalValue::Memory { gv, ty } => { - let addr = builder.ins().global_addr(runtime.native_pointer(), gv); + let addr = builder.ins().global_addr(environ.native_pointer(), gv); // TODO: It is likely safe to set `aligned notrap` flags on a global load. let flags = ir::MemFlags::new(); builder.ins().load(ty, flags, addr, 0) @@ -169,10 +169,10 @@ fn translate_operator( state.push1(val); } Operator::SetGlobal { global_index } => { - match state.get_global(builder.func, global_index, runtime) { + match state.get_global(builder.func, global_index, environ) { GlobalValue::Const(_) => panic!("global #{} is a constant", global_index), GlobalValue::Memory { gv, .. } => { - let addr = builder.ins().global_addr(runtime.native_pointer(), gv); + let addr = builder.ins().global_addr(environ.native_pointer(), gv); // TODO: It is likely safe to set `aligned notrap` flags on a global store. let flags = ir::MemFlags::new(); let val = state.pop1(); @@ -428,8 +428,8 @@ fn translate_operator( * argument referring to an index in the external functions table of the module. ************************************************************************************/ Operator::Call { function_index } => { - let (fref, num_args) = state.get_direct_func(builder.func, function_index, runtime); - let call = runtime.translate_call( + let (fref, num_args) = state.get_direct_func(builder.func, function_index, environ); + let call = environ.translate_call( builder.cursor(), function_index as FunctionIndex, fref, @@ -441,9 +441,9 @@ fn translate_operator( Operator::CallIndirect { index, table_index } => { // `index` is the index of the function's signature and `table_index` is the index of // the table to search the function in. - let (sigref, num_args) = state.get_indirect_sig(builder.func, index, runtime); + let (sigref, num_args) = state.get_indirect_sig(builder.func, index, environ); let callee = state.pop1(); - let call = runtime.translate_call_indirect( + let call = environ.translate_call_indirect( builder.cursor(), table_index as TableIndex, index as SignatureIndex, @@ -462,9 +462,9 @@ fn translate_operator( // The WebAssembly MVP only supports one linear memory, but we expect the reserved // argument to be a memory index. let heap_index = reserved as MemoryIndex; - let heap = state.get_heap(builder.func, reserved, runtime); + let heap = state.get_heap(builder.func, reserved, environ); let val = state.pop1(); - state.push1(runtime.translate_grow_memory( + state.push1(environ.translate_grow_memory( builder.cursor(), heap_index, heap, @@ -473,8 +473,8 @@ fn translate_operator( } Operator::CurrentMemory { reserved } => { let heap_index = reserved as MemoryIndex; - let heap = state.get_heap(builder.func, reserved, runtime); - state.push1(runtime.translate_current_memory( + let heap = state.get_heap(builder.func, reserved, environ); + state.push1(environ.translate_current_memory( builder.cursor(), heap_index, heap, @@ -486,46 +486,46 @@ fn translate_operator( * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not ************************************************************************************/ Operator::I32Load8U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Uload8, I32, builder, state, runtime); + translate_load(offset, ir::Opcode::Uload8, I32, builder, state, environ); } Operator::I32Load16U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Uload16, I32, builder, state, runtime); + translate_load(offset, ir::Opcode::Uload16, I32, builder, state, environ); } Operator::I32Load8S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Sload8, I32, builder, state, runtime); + translate_load(offset, ir::Opcode::Sload8, I32, builder, state, environ); } Operator::I32Load16S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Sload16, I32, builder, state, runtime); + translate_load(offset, ir::Opcode::Sload16, I32, builder, state, environ); } Operator::I64Load8U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Uload8, I64, builder, state, runtime); + translate_load(offset, ir::Opcode::Uload8, I64, builder, state, environ); } Operator::I64Load16U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Uload16, I64, builder, state, runtime); + translate_load(offset, ir::Opcode::Uload16, I64, builder, state, environ); } Operator::I64Load8S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Sload8, I64, builder, state, runtime); + translate_load(offset, ir::Opcode::Sload8, I64, builder, state, environ); } Operator::I64Load16S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Sload16, I64, builder, state, runtime); + translate_load(offset, ir::Opcode::Sload16, I64, builder, state, environ); } Operator::I64Load32S { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Sload32, I64, builder, state, runtime); + translate_load(offset, ir::Opcode::Sload32, I64, builder, state, environ); } Operator::I64Load32U { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Uload32, I64, builder, state, runtime); + translate_load(offset, ir::Opcode::Uload32, I64, builder, state, environ); } Operator::I32Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Load, I32, builder, state, runtime); + translate_load(offset, ir::Opcode::Load, I32, builder, state, environ); } Operator::F32Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Load, F32, builder, state, runtime); + translate_load(offset, ir::Opcode::Load, F32, builder, state, environ); } Operator::I64Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Load, I64, builder, state, runtime); + translate_load(offset, ir::Opcode::Load, I64, builder, state, environ); } Operator::F64Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_load(offset, ir::Opcode::Load, F64, builder, state, runtime); + translate_load(offset, ir::Opcode::Load, F64, builder, state, environ); } /****************************** Store instructions *********************************** * Wasm specifies an integer alignment flag but we drop it in Cretonne. @@ -536,18 +536,18 @@ fn translate_operator( Operator::I64Store { memory_immediate: MemoryImmediate { flags: _, offset } } | Operator::F32Store { memory_immediate: MemoryImmediate { flags: _, offset } } | Operator::F64Store { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_store(offset, ir::Opcode::Store, builder, state, runtime); + translate_store(offset, ir::Opcode::Store, builder, state, environ); } Operator::I32Store8 { memory_immediate: MemoryImmediate { flags: _, offset } } | Operator::I64Store8 { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_store(offset, ir::Opcode::Istore8, builder, state, runtime); + translate_store(offset, ir::Opcode::Istore8, builder, state, environ); } Operator::I32Store16 { memory_immediate: MemoryImmediate { flags: _, offset } } | Operator::I64Store16 { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_store(offset, ir::Opcode::Istore16, builder, state, runtime); + translate_store(offset, ir::Opcode::Istore16, builder, state, environ); } Operator::I64Store32 { memory_immediate: MemoryImmediate { flags: _, offset } } => { - translate_store(offset, ir::Opcode::Istore32, builder, state, runtime); + translate_store(offset, ir::Opcode::Istore32, builder, state, environ); } /****************************** Nullary Operators ************************************/ Operator::I32Const { value } => state.push1(builder.ins().iconst(I32, value as i64)), diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 703b5f164a..3517fabf84 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -89,7 +89,7 @@ impl FuncEnvironment for DummyRuntime { } fn translate_grow_memory( - &mut self, + &self, mut pos: FuncCursor, _index: MemoryIndex, _heap: ir::Heap, @@ -99,7 +99,7 @@ impl FuncEnvironment for DummyRuntime { } fn translate_current_memory( - &mut self, + &self, mut pos: FuncCursor, _index: MemoryIndex, _heap: ir::Heap, diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 35f8d9ea37..e7cbb1ec91 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -110,7 +110,7 @@ pub trait FuncEnvironment { /// /// Returns the old size (in pages) of the memory. fn translate_grow_memory( - &mut self, + &self, pos: FuncCursor, index: MemoryIndex, heap: ir::Heap, @@ -124,7 +124,7 @@ pub trait FuncEnvironment { /// /// Returns the size in pages of the memory. fn translate_current_memory( - &mut self, + &self, pos: FuncCursor, index: MemoryIndex, heap: ir::Heap, From 388a96421be1ae25f6f3049e4d8665e5a8ba490e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Sep 2017 04:40:06 -0700 Subject: [PATCH 1073/3084] Add documentation comments to GlobalValue's fields. --- lib/wasm/src/runtime/spec.rs | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index e7cbb1ec91..9f093e9a07 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -12,7 +12,12 @@ pub enum GlobalValue { Const(ir::Value), /// This is a variable in memory that should be referenced as a `GlobalVar`. - Memory { gv: ir::GlobalVar, ty: ir::Type }, + Memory { + /// Which global variable should be referenced. + gv: ir::GlobalVar, + /// The global variable's type. + ty: ir::Type, + }, } /// Environment affecting the translation of a single WebAssembly function. From c0f3eaafbc13dd6767f3110d6b5c003003e9b65d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Sep 2017 04:42:01 -0700 Subject: [PATCH 1074/3084] Publically declare GlobalValue. It's used in the publically declared FuncEnvironment trait, so library users should be able to name it. --- lib/wasm/src/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index e334570cfe..0559f74121 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -24,6 +24,6 @@ mod state; mod translation_utils; pub use module_translator::{translate_module, TranslationResult}; -pub use runtime::{FuncEnvironment, WasmRuntime, DummyRuntime}; +pub use runtime::{FuncEnvironment, WasmRuntime, DummyRuntime, GlobalValue}; pub use translation_utils::{FunctionIndex, GlobalIndex, TableIndex, MemoryIndex, SignatureIndex, Global, GlobalInit, Table, Memory}; From 439a40e5e42c90853dd1615b51de7d9cf5e9cd1b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 1 Sep 2017 13:19:31 -0700 Subject: [PATCH 1075/3084] Add a WebAssembly function translator. The new FuncTranslator type can be used to translate binary WebAssembly functions to Cretonne IL one at a time. It is independent of the module-level parser also present in the cretonne-wasm crate. --- lib/wasm/src/code_translator.rs | 18 +- lib/wasm/src/func_translator.rs | 315 ++++++++++++++++++++++++++++++++ lib/wasm/src/lib.rs | 3 + 3 files changed, 327 insertions(+), 9 deletions(-) create mode 100644 lib/wasm/src/func_translator.rs diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 2cb61d3239..04a3770ee8 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -94,11 +94,7 @@ pub fn translate_function_body( let parser_state = parser.read(); match *parser_state { ParserState::CodeOperator(ref op) => { - if state.in_unreachable_code() { - translate_unreachable_operator(op, &mut builder, &mut state) - } else { - translate_operator(op, &mut builder, &mut state, runtime) - } + translate_operator(op, &mut builder, &mut state, runtime) } ParserState::EndFunctionBody => break, @@ -132,12 +128,16 @@ pub fn translate_function_body( /// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted /// a return. -fn translate_operator( +pub fn translate_operator( op: &Operator, builder: &mut FunctionBuilder, state: &mut TranslationState, - environ: &FE, + environ: &mut FE, ) { + if state.in_unreachable_code() { + return translate_unreachable_operator(op, builder, state); + } + // This big match treats all Wasm code operators. match *op { /********************************** Locals **************************************** @@ -1027,7 +1027,7 @@ fn translate_load( result_ty: ir::Type, builder: &mut FunctionBuilder, state: &mut TranslationState, - environ: &FE, + environ: &mut FE, ) { let addr32 = state.pop1(); // We don't yet support multiple linear memories. @@ -1050,7 +1050,7 @@ fn translate_store( opcode: ir::Opcode, builder: &mut FunctionBuilder, state: &mut TranslationState, - environ: &FE, + environ: &mut FE, ) { let (addr32, val) = state.pop2(); let val_ty = builder.func.dfg.value_type(val); diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs new file mode 100644 index 0000000000..9eba880f32 --- /dev/null +++ b/lib/wasm/src/func_translator.rs @@ -0,0 +1,315 @@ +//! Stand-alone WebAssembly to Cretonne IL translator. +//! +//! This module defines the `FuncTranslator` type which can translate a single WebAssembly +//! function to Cretonne IL guided by a `FuncEnvironment` which provides information about the +//! WebAssembly module and the runtime environment. + +use code_translator::translate_operator; +use cretonne::entity::EntityRef; +use cretonne::ir::{self, InstBuilder}; +use cretonne::result::{CtonResult, CtonError}; +use cton_frontend::{ILBuilder, FunctionBuilder}; +use runtime::FuncEnvironment; +use state::TranslationState; +use translation_utils::Local; +use wasmparser::{self, BinaryReader}; + +/// Maximum number of local variables permitted in a function. The translation fails with a +/// `CtonError::ImplLimitExceeded` error if the limit is exceeded. +const MAX_LOCALS: usize = 50_000; + +/// WebAssembly to Cretonne IL function translator. +/// +/// A `FuncTranslator` is used to translate a binary WebAssembly function into Cretonne IL guided +/// by a `FuncEnvironment` object. A single translator instance can be reused to translate multiple +/// functions which will reduce heap allocation traffic. +pub struct FuncTranslator { + il_builder: ILBuilder, + state: TranslationState, +} + +impl FuncTranslator { + /// Create a new translator. + pub fn new() -> FuncTranslator { + FuncTranslator { + il_builder: ILBuilder::new(), + state: TranslationState::new(), + } + } + + /// Translate a binary WebAssembly function. + /// + /// The `code` slice contains the binary WebAssembly *function code* as it appears in the code + /// section of a WebAssembly module, not including the initial size of the function code. The + /// slice is expected to contain two parts: + /// + /// - The declaration of *locals*, and + /// - The function *body* as an expression. + /// + /// See [the WebAssembly specification] + /// (http://webassembly.github.io/spec/binary/modules.html#code-section). + /// + /// The Cretonne IR function `func` should be completely empty except for the `func.signature` + /// and `func.name` fields. The signature may contain special-purpose arguments which are not + /// regarded as WebAssembly local variables. Any signature arguments marked as + /// `ArgumentPurpose::Normal` are made accessible as WebAssembly local variables. + /// + pub fn translate( + &mut self, + code: &[u8], + func: &mut ir::Function, + environ: &mut FE, + ) -> CtonResult { + dbg!( + "translate({} bytes, {}{})", + code.len(), + func.name, + func.signature + ); + assert_eq!(func.dfg.num_ebbs(), 0, "Function must be empty"); + assert_eq!(func.dfg.num_insts(), 0, "Function must be empty"); + + // This clears the `ILBuilder`. + let builder = &mut FunctionBuilder::new(func, &mut self.il_builder); + let entry_block = builder.create_ebb(); + builder.switch_to_block(entry_block, &[]); // This also creates values for the arguments. + builder.seal_block(entry_block); + let num_args = declare_wasm_arguments(builder); + + // Set up the translation state with a single pushed control block representing the whole + // function and its return values. + let exit_block = builder.create_ebb(); + self.state.initialize(&builder.func.signature, exit_block); + + let reader = &mut BinaryReader::new(code); + parse_local_decls(reader, builder, num_args)?; + parse_function_body(reader, builder, &mut self.state, environ) + } +} + +/// Declare local variables for the signature arguments that correspond to WebAssembly locals. +/// +/// Return the number of local variables declared. +fn declare_wasm_arguments(builder: &mut FunctionBuilder) -> usize { + let sig_len = builder.func.signature.argument_types.len(); + let mut next_local = 0; + for i in 0..sig_len { + let arg_type = builder.func.signature.argument_types[i]; + // There may be additional special-purpose arguments following the normal WebAssembly + // signature arguments. For example, a `vmctx` pointer. + if arg_type.purpose == ir::ArgumentPurpose::Normal { + // This is a normal WebAssembly signature argument, so create a local for it. + let local = Local::new(next_local); + builder.declare_var(local, arg_type.value_type); + next_local += 1; + + let arg_value = builder.arg_value(i); + builder.def_var(local, arg_value); + } + } + + next_local +} + +/// Parse the local variable declarations that precede the function body. +/// +/// Declare local variables, starting from `num_args`. +fn parse_local_decls( + reader: &mut BinaryReader, + builder: &mut FunctionBuilder, + num_args: usize, +) -> CtonResult { + let mut next_local = num_args; + let local_count = reader.read_var_u32().map_err(|_| CtonError::InvalidInput)?; + + for _ in 0..local_count { + let (count, ty) = reader.read_local_decl().map_err( + |_| CtonError::InvalidInput, + )?; + declare_locals(builder, count, ty, &mut next_local)?; + } + + Ok(()) +} + +/// Declare `count` local variables of the same type, starting from `next_local`. +/// +/// Fail of too many locals are declared in the function, or if the type is not valid for a local. +fn declare_locals( + builder: &mut FunctionBuilder, + count: u32, + wasm_type: wasmparser::Type, + next_local: &mut usize, +) -> CtonResult { + // All locals are initialized to 0. + use wasmparser::Type::*; + let zeroval = match wasm_type { + I32 => builder.ins().iconst(ir::types::I32, 0), + I64 => builder.ins().iconst(ir::types::I64, 0), + F32 => builder.ins().f32const(ir::immediates::Ieee32::with_bits(0)), + F64 => builder.ins().f64const(ir::immediates::Ieee64::with_bits(0)), + _ => return Err(CtonError::InvalidInput), + }; + + let ty = builder.func.dfg.value_type(zeroval); + for _ in 0..count { + // This implementation limit is arbitrary, but it ensures that a small function can't blow + // up the compiler by declaring millions of locals. + if *next_local >= MAX_LOCALS { + return Err(CtonError::ImplLimitExceeded); + } + + let local = Local::new(*next_local); + builder.declare_var(local, ty); + builder.def_var(local, zeroval); + *next_local += 1; + } + + Ok(()) +} + +/// Parse the function body in `reader`. +/// +/// This assumes that the local variable declarations have already been parsed and function +/// arguments and locals are declared in the builder. +fn parse_function_body( + reader: &mut BinaryReader, + builder: &mut FunctionBuilder, + state: &mut TranslationState, + environ: &mut FE, +) -> CtonResult { + // The control stack is initialized with a single block representing the whole function. + assert_eq!(state.control_stack.len(), 1, "State not initialized"); + + // Keep going until the final `End` operator which pops the outermost block. + while !state.control_stack.is_empty() { + let op = reader.read_operator().map_err(|_| CtonError::InvalidInput)?; + translate_operator(&op, builder, state, environ); + } + + // The final `End` operator left us in the exit block where we need to manually add a return + // instruction. + // + // If the exit block is unreachable, it may not have the correct arguments, so we would + // generate a return instruction that doesn't match the signature. + if !builder.is_unreachable() { + builder.ins().return_(state.stack.as_slice()); + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use cretonne::{ir, Context}; + use cretonne::ir::types::I32; + use runtime::DummyRuntime; + use super::FuncTranslator; + + #[test] + fn small1() { + // Implicit return. + // + // (func $small1 (param i32) (result i32) + // (i32.add (get_local 0) (i32.const 1)) + // ) + const BODY: [u8; 7] = [ + 0x00, // local decl count + 0x20, 0x00, // get_local 0 + 0x41, 0x01, // i32.const 1 + 0x6a, // i32.add + 0x0b, // end + ]; + + let mut trans = FuncTranslator::new(); + let mut runtime = DummyRuntime::new(); + let mut ctx = Context::new(); + + ctx.func.name = ir::FunctionName::new("small1"); + ctx.func.signature.argument_types.push( + ir::ArgumentType::new(I32), + ); + ctx.func.signature.return_types.push( + ir::ArgumentType::new(I32), + ); + + trans.translate(&BODY, &mut ctx.func, &mut runtime).unwrap(); + dbg!("{}", ctx.func.display(None)); + ctx.flowgraph(); + ctx.verify(None).unwrap(); + } + + #[test] + fn small2() { + // Same as above, but with an explicit return instruction. + // + // (func $small2 (param i32) (result i32) + // (return (i32.add (get_local 0) (i32.const 1))) + // ) + const BODY: [u8; 8] = [ + 0x00, // local decl count + 0x20, 0x00, // get_local 0 + 0x41, 0x01, // i32.const 1 + 0x6a, // i32.add + 0x0f, // return + 0x0b, // end + ]; + + let mut trans = FuncTranslator::new(); + let mut runtime = DummyRuntime::new(); + let mut ctx = Context::new(); + + ctx.func.name = ir::FunctionName::new("small2"); + ctx.func.signature.argument_types.push( + ir::ArgumentType::new(I32), + ); + ctx.func.signature.return_types.push( + ir::ArgumentType::new(I32), + ); + + trans.translate(&BODY, &mut ctx.func, &mut runtime).unwrap(); + dbg!("{}", ctx.func.display(None)); + ctx.flowgraph(); + ctx.verify(None).unwrap(); + } + + #[test] + fn infloop() { + // An infinite loop, no return instructions. + // + // (func $infloop (result i32) + // (local i32) + // (loop (result i32) + // (i32.add (get_local 0) (i32.const 1)) + // (set_local 0) + // (br 0) + // ) + // ) + const BODY: [u8; 16] = [ + 0x01, // 1 local decl. + 0x01, 0x7f, // 1 i32 local. + 0x03, 0x7f, // loop i32 + 0x20, 0x00, // get_local 0 + 0x41, 0x01, // i32.const 0 + 0x6a, // i32.add + 0x21, 0x00, // set_local 0 + 0x0c, 0x00, // br 0 + 0x0b, // end + 0x0b, // end + ]; + + let mut trans = FuncTranslator::new(); + let mut runtime = DummyRuntime::new(); + let mut ctx = Context::new(); + + ctx.func.name = ir::FunctionName::new("infloop"); + ctx.func.signature.return_types.push( + ir::ArgumentType::new(I32), + ); + + trans.translate(&BODY, &mut ctx.func, &mut runtime).unwrap(); + dbg!("{}", ctx.func.display(None)); + ctx.flowgraph(); + ctx.verify(None).unwrap(); + } +} diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 0559f74121..9d1ad548bc 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -14,15 +14,18 @@ extern crate wasmparser; extern crate cton_frontend; +#[macro_use(dbg)] extern crate cretonne; mod code_translator; +mod func_translator; mod module_translator; mod runtime; mod sections_translator; mod state; mod translation_utils; +pub use func_translator::FuncTranslator; pub use module_translator::{translate_module, TranslationResult}; pub use runtime::{FuncEnvironment, WasmRuntime, DummyRuntime, GlobalValue}; pub use translation_utils::{FunctionIndex, GlobalIndex, TableIndex, MemoryIndex, SignatureIndex, From 0c16f13c6b8062bf0c00c5d294236741233f0c2d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Sep 2017 14:53:55 -0700 Subject: [PATCH 1076/3084] Add a Function::clear() method. This makes it possible to clear out a Function data structure so it can be reused for compiling multiple functions. Also add clear() methods to various sub-structures. --- lib/cretonne/src/ir/dfg.rs | 11 +++++++++++ lib/cretonne/src/ir/extfunc.rs | 8 ++++++++ lib/cretonne/src/ir/function.rs | 14 ++++++++++++++ lib/cretonne/src/ir/layout.rs | 8 ++++++++ 4 files changed, 41 insertions(+) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 240c5377c0..b81bbe96fc 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -74,6 +74,17 @@ impl DataFlowGraph { } } + /// Clear everything. + pub fn clear(&mut self) { + self.insts.clear(); + self.results.clear(); + self.ebbs.clear(); + self.value_lists.clear(); + self.values.clear(); + self.signatures.clear(); + self.ext_funcs.clear(); + } + /// Get the total number of instructions created in this function, whether they are currently /// inserted in the layout or not. /// diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 2606ebe19e..353d49256e 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -47,6 +47,14 @@ impl Signature { } } + /// Clear the signature so it is identical to a fresh one returned by `new()`. + pub fn clear(&mut self, call_conv: CallConv) { + self.argument_types.clear(); + self.return_types.clear(); + self.call_conv = call_conv; + self.argument_bytes = None; + } + /// Compute the size of the stack arguments and mark signature as legalized. /// /// Even if there are no stack arguments, this will set `argument_types` to `Some(0)` instead diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 0f1599d6bd..95b6458558 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -74,6 +74,20 @@ impl Function { } } + /// Clear all data structures in this function. + pub fn clear(&mut self) { + self.signature.clear(ir::CallConv::Native); + self.stack_slots.clear(); + self.global_vars.clear(); + self.heaps.clear(); + self.jump_tables.clear(); + self.dfg.clear(); + self.layout.clear(); + self.encodings.clear(); + self.locations.clear(); + self.offsets.clear(); + } + /// Create a new empty, anonymous function with a native calling convention. pub fn new() -> Function { Self::with_name_signature(FunctionName::default(), Signature::new(CallConv::Native)) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 33fbaf943d..6d16b004e1 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -51,6 +51,14 @@ impl Layout { last_ebb: None, } } + + /// Clear the layout. + pub fn clear(&mut self) { + self.ebbs.clear(); + self.insts.clear(); + self.first_ebb = None; + self.last_ebb = None; + } } // Sequence numbers. From 3775fa4867b8238cb7f982bc2576e34739bc9f57 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 8 Sep 2017 16:16:19 -0700 Subject: [PATCH 1077/3084] Update to wasmparser 0.9.3. wasmparser's API is changing in anticipation of streaming decoding, so it will now hand large data section initializers back in chunks rather than all at once. --- lib/wasm/Cargo.toml | 2 +- lib/wasm/src/sections_translator.rs | 16 +++++++++------- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 15dd4bb342..8cc3b40819 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -11,6 +11,6 @@ license = "Apache-2.0" name = "cton_wasm" [dependencies] -wasmparser = "0.8.2" +wasmparser = "0.9.3" cretonne = { path = "../cretonne" } cretonne-frontend = { path = "../frontend" } diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 8c8b7aac71..e2dae0a58c 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -250,7 +250,7 @@ pub fn parse_data_section( ParserState::BeginInitExpressionBody => (), ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; - let offset = match *parser.read() { + let mut offset = match *parser.read() { ParserState::InitExpressionOperator(Operator::I32Const { value }) => { if value < 0 { return Err(SectionParsingError::WrongSectionContent(String::from( @@ -288,20 +288,22 @@ pub fn parse_data_section( ParserState::EndInitExpressionBody => (), ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; - { + match *parser.read() { + ParserState::BeginDataSectionEntryBody(_) => (), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + loop { let data = match *parser.read() { - ParserState::DataSectionEntryBody(data) => data, + ParserState::DataSectionEntryBodyChunk(data) => data, + ParserState::EndDataSectionEntry => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; match runtime.declare_data_initialization(memory_index as MemoryIndex, offset, data) { Ok(()) => (), Err(s) => return Err(SectionParsingError::WrongSectionContent(s)), }; + offset += data.len(); } - match *parser.read() { - ParserState::EndDataSectionEntry => (), - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), - }; } Ok(()) } From 2fa0a7a3a46301b9d44cb18a0454e35d07a4bb4d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 8 Sep 2017 17:12:32 -0700 Subject: [PATCH 1078/3084] Fix a confusion between EndDataSectionEntry and EndDataSectionEntryBody. --- lib/wasm/src/sections_translator.rs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index e2dae0a58c..3df3e7e3d9 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -295,7 +295,7 @@ pub fn parse_data_section( loop { let data = match *parser.read() { ParserState::DataSectionEntryBodyChunk(data) => data, - ParserState::EndDataSectionEntry => break, + ParserState::EndDataSectionEntryBody => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; match runtime.declare_data_initialization(memory_index as MemoryIndex, offset, data) { @@ -304,6 +304,10 @@ pub fn parse_data_section( }; offset += data.len(); } + match *parser.read() { + ParserState::EndDataSectionEntry => (), + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; } Ok(()) } From 620f1f49e26d401803cc50353bb292b9ab099848 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Sep 2017 12:02:45 -0700 Subject: [PATCH 1079/3084] Move several functions from FunctionBuilder to Function. With FuncEnvironment using FuncCursors in place of full FunctionBuilders, it's useful to move several of these convenience functions from FunctionBuilder to Function. --- lib/cretonne/src/ir/function.rs | 38 ++++++++++++++++++++++++++++++ lib/cretonne/src/ir/jumptable.rs | 5 ++++ lib/cretonne/src/ir/stackslot.rs | 10 ++------ lib/cretonne/src/write.rs | 4 +--- lib/frontend/src/frontend.rs | 24 +++++++++++++------ lib/frontend/src/ssa.rs | 6 ++--- lib/reader/src/parser.rs | 14 +++++------ lib/wasm/src/code_translator.rs | 40 +++++++++++++++++--------------- lib/wasm/src/runtime/dummy.rs | 10 ++++---- 9 files changed, 99 insertions(+), 52 deletions(-) diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 95b6458558..0c86ac66e0 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -7,6 +7,8 @@ use entity::{PrimaryMap, EntityMap}; use ir; use ir::{FunctionName, CallConv, Signature, DataFlowGraph, Layout}; use ir::{InstEncodings, ValueLocations, JumpTables, StackSlots, EbbOffsets}; +use ir::{Ebb, JumpTableData, JumpTable, StackSlotData, StackSlot, SigRef, ExtFuncData, FuncRef, + GlobalVarData, GlobalVar, HeapData, Heap}; use isa::TargetIsa; use std::fmt; use write::write_function; @@ -93,6 +95,42 @@ impl Function { Self::with_name_signature(FunctionName::default(), Signature::new(CallConv::Native)) } + /// Creates a jump table in the function, to be used by `br_table` instructions. + pub fn create_jump_table(&mut self, data: JumpTableData) -> JumpTable { + self.jump_tables.push(data) + } + + /// Inserts an entry in a previously declared jump table. + pub fn insert_jump_table_entry(&mut self, jt: JumpTable, index: usize, ebb: Ebb) { + self.jump_tables[jt].set_entry(index, ebb); + } + + /// Creates a stack slot in the function, to be used by `stack_load`, `stack_store` and + /// `stack_addr` instructions. + pub fn create_stack_slot(&mut self, data: StackSlotData) -> StackSlot { + self.stack_slots.push(data) + } + + /// Adds a signature which can later be used to declare an external function import. + pub fn import_signature(&mut self, signature: Signature) -> SigRef { + self.dfg.signatures.push(signature) + } + + /// Declare an external function import. + pub fn import_function(&mut self, data: ExtFuncData) -> FuncRef { + self.dfg.ext_funcs.push(data) + } + + /// Declares a global variable accessible to the function. + pub fn create_global_var(&mut self, data: GlobalVarData) -> GlobalVar { + self.global_vars.push(data) + } + + /// Declares a heap accessible to the function. + pub fn create_heap(&mut self, data: HeapData) -> Heap { + self.heaps.push(data) + } + /// Return an object that can display this function with correct ISA-specific annotations. pub fn display<'a, I: Into>>(&'a self, isa: I) -> DisplayFunction<'a> { DisplayFunction(self, isa.into()) diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index 5e34d07c4d..6d97341458 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -46,6 +46,11 @@ impl JumpTableData { self.table[idx] = dest.into(); } + /// Append a table entry. + pub fn push_entry(&mut self, dest: Ebb) { + self.table.push(dest.into()) + } + /// Clear a table entry. /// /// The `br_table` instruction will fall through if given an index corresponding to a cleared diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index effa65a25e..e5a4b10161 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -255,14 +255,8 @@ mod tests { fn stack_slot() { let mut func = Function::new(); - let ss0 = func.stack_slots.push(StackSlotData::new( - StackSlotKind::IncomingArg, - 4, - )); - let ss1 = func.stack_slots.push(StackSlotData::new( - StackSlotKind::SpillSlot, - 8, - )); + let ss0 = func.create_stack_slot(StackSlotData::new(StackSlotKind::IncomingArg, 4)); + let ss1 = func.create_stack_slot(StackSlotData::new(StackSlotKind::SpillSlot, 8)); assert_eq!(ss0.to_string(), "ss0"); assert_eq!(ss1.to_string(), "ss1"); diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index f69586b2ea..77f0209da3 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -395,9 +395,7 @@ mod tests { f.name = FunctionName::new("foo"); assert_eq!(f.to_string(), "function %foo() native {\n}\n"); - f.stack_slots.push( - StackSlotData::new(StackSlotKind::Local, 4), - ); + f.create_stack_slot(StackSlotData::new(StackSlotKind::Local, 4)); assert_eq!( f.to_string(), "function %foo() native {\n ss0 = local 4\n}\n" diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 4f6d4ef640..00e6377897 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -2,7 +2,7 @@ use cretonne::cursor::{Cursor, FuncCursor}; use cretonne::ir::{Ebb, Type, Value, Function, Inst, JumpTable, StackSlot, JumpTableData, StackSlotData, DataFlowGraph, InstructionData, ExtFuncData, FuncRef, SigRef, - Signature, InstBuilderBase}; + Signature, InstBuilderBase, GlobalVarData, GlobalVar, HeapData, Heap}; use cretonne::ir::instructions::BranchInfo; use cretonne::ir::function::DisplayFunction; use cretonne::isa::TargetIsa; @@ -329,29 +329,39 @@ where } /// Creates a jump table in the function, to be used by `br_table` instructions. - pub fn create_jump_table(&mut self) -> JumpTable { - self.func.jump_tables.push(JumpTableData::new()) + pub fn create_jump_table(&mut self, data: JumpTableData) -> JumpTable { + self.func.create_jump_table(data) } /// Inserts an entry in a previously declared jump table. pub fn insert_jump_table_entry(&mut self, jt: JumpTable, index: usize, ebb: Ebb) { - self.func.jump_tables[jt].set_entry(index, ebb); + self.func.insert_jump_table_entry(jt, index, ebb) } /// Creates a stack slot in the function, to be used by `stack_load`, `stack_store` and /// `stack_addr` instructions. pub fn create_stack_slot(&mut self, data: StackSlotData) -> StackSlot { - self.func.stack_slots.push(data) + self.func.create_stack_slot(data) } /// Adds a signature which can later be used to declare an external function import. pub fn import_signature(&mut self, signature: Signature) -> SigRef { - self.func.dfg.signatures.push(signature) + self.func.import_signature(signature) } /// Declare an external function import. pub fn import_function(&mut self, data: ExtFuncData) -> FuncRef { - self.func.dfg.ext_funcs.push(data) + self.func.import_function(data) + } + + /// Declares a global variable accessible to the function. + pub fn create_global_var(&mut self, data: GlobalVarData) -> GlobalVar { + self.func.create_global_var(data) + } + + /// Declares a heap accessible to the function. + pub fn create_heap(&mut self, data: HeapData) -> Heap { + self.func.create_heap(data) } /// Returns an object with the [`InstBuilder`](../cretonne/ir/builder/trait.InstBuilder.html) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 60c5663866..a4c99885a0 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -1145,9 +1145,9 @@ mod tests { func.dfg.ins(cur).iconst(I32, 1) }; ssa.def_var(x_var, x1, block0); - let mut jt_data = JumpTableData::new(); - jt_data.set_entry(0, ebb1); - let jt = func.jump_tables.push(jt_data); + let mut data = JumpTableData::new(); + data.push_entry(ebb1); + let jt = func.create_jump_table(data); ssa.use_var( &mut func.dfg, &mut func.layout, diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 259d462298..2169ded9c9 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -120,7 +120,7 @@ impl<'a> Context<'a> { fn add_ss(&mut self, number: u32, data: StackSlotData, loc: &Location) -> Result<()> { self.map.def_ss( number, - self.function.stack_slots.push(data), + self.function.create_stack_slot(data), loc, ) } @@ -137,7 +137,7 @@ impl<'a> Context<'a> { fn add_gv(&mut self, number: u32, data: GlobalVarData, loc: &Location) -> Result<()> { self.map.def_gv( number, - self.function.global_vars.push(data), + self.function.create_global_var(data), loc, ) } @@ -154,7 +154,7 @@ impl<'a> Context<'a> { fn add_heap(&mut self, number: u32, data: HeapData, loc: &Location) -> Result<()> { self.map.def_heap( number, - self.function.heaps.push(data), + self.function.create_heap(data), loc, ) } @@ -171,7 +171,7 @@ impl<'a> Context<'a> { fn add_sig(&mut self, number: u32, data: Signature, loc: &Location) -> Result<()> { self.map.def_sig( number, - self.function.dfg.signatures.push(data), + self.function.import_signature(data), loc, ) } @@ -188,7 +188,7 @@ impl<'a> Context<'a> { fn add_fn(&mut self, number: u32, data: ExtFuncData, loc: &Location) -> Result<()> { self.map.def_fn( number, - self.function.dfg.ext_funcs.push(data), + self.function.import_function(data), loc, ) } @@ -205,7 +205,7 @@ impl<'a> Context<'a> { fn add_jt(&mut self, number: u32, data: JumpTableData, loc: &Location) -> Result<()> { self.map.def_jt( number, - self.function.jump_tables.push(data), + self.function.create_jump_table(data), loc, ) } @@ -1275,7 +1275,7 @@ impl<'a> Parser<'a> { let data = match self.token() { Some(Token::Identifier("function")) => { let (loc, name, sig) = self.parse_function_spec(ctx.unique_isa)?; - let sigref = ctx.function.dfg.signatures.push(sig); + let sigref = ctx.function.import_signature(sig); ctx.map.def_entity(sigref.into(), &loc).expect( "duplicate SigRef entities created", ); diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 04a3770ee8..096d7d2b4d 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -21,7 +21,8 @@ //! //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as //! argument. -use cretonne::ir::{self, Function, Signature, Type, InstBuilder, FunctionName, Ebb, MemFlags}; +use cretonne::ir::{self, Function, Signature, Type, InstBuilder, FunctionName, Ebb, MemFlags, + JumpTableData}; use cretonne::ir::types::*; use cretonne::ir::immediates::{Ieee32, Ieee64}; use cretonne::ir::condcodes::{IntCC, FloatCC}; @@ -363,14 +364,15 @@ pub fn translate_operator( if jump_args_count == 0 { // No jump arguments let val = state.pop1(); - let jt = builder.create_jump_table(); - for (index, depth) in depths.iter().enumerate() { - let i = state.control_stack.len() - 1 - (*depth as usize); + let mut data = JumpTableData::new(); + for depth in depths { + let i = state.control_stack.len() - 1 - (depth as usize); let frame = &mut state.control_stack[i]; let ebb = frame.br_destination(); - builder.insert_jump_table_entry(jt, index, ebb); + data.push_entry(ebb); frame.set_reachable(); } + let jt = builder.create_jump_table(data); builder.ins().br_table(val, jt); let i = state.control_stack.len() - 1 - (default as usize); let frame = &mut state.control_stack[i]; @@ -383,20 +385,20 @@ pub fn translate_operator( // We then proceed to split the edges going out of the br_table let val = state.pop1(); let cut_index = state.stack.len() - jump_args_count; - let jt = builder.create_jump_table(); - let dest_ebbs: HashMap = - depths.iter().enumerate().fold(HashMap::new(), |mut acc, - (index, &depth)| { - if acc.get(&(depth as usize)).is_none() { - let branch_ebb = builder.create_ebb(); - builder.insert_jump_table_entry(jt, index, branch_ebb); - acc.insert(depth as usize, branch_ebb); - return acc; - }; - let branch_ebb = acc[&(depth as usize)]; - builder.insert_jump_table_entry(jt, index, branch_ebb); - acc - }); + let mut data = JumpTableData::new(); + let dest_ebbs: HashMap = depths.iter().fold(HashMap::new(), |mut acc, + &depth| { + if acc.get(&(depth as usize)).is_none() { + let branch_ebb = builder.create_ebb(); + data.push_entry(branch_ebb); + acc.insert(depth as usize, branch_ebb); + return acc; + }; + let branch_ebb = acc[&(depth as usize)]; + data.push_entry(branch_ebb); + acc + }); + let jt = builder.create_jump_table(data); builder.ins().br_table(val, jt); let default_ebb = state.control_stack[state.control_stack.len() - 1 - (default as usize)] diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 3517fabf84..04762988f2 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -40,7 +40,7 @@ impl FuncEnvironment for DummyRuntime { fn make_global(&self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue { // Just create a dummy `vmctx` global. let offset = ((index * 8) as i32 + 8).into(); - let gv = func.global_vars.push(ir::GlobalVarData::VmCtx { offset }); + let gv = func.create_global_var(ir::GlobalVarData::VmCtx { offset }); GlobalValue::Memory { gv, ty: self.globals[index].ty, @@ -48,7 +48,7 @@ impl FuncEnvironment for DummyRuntime { } fn make_heap(&self, func: &mut ir::Function, _index: MemoryIndex) -> ir::Heap { - func.heaps.push(ir::HeapData { + func.create_heap(ir::HeapData { base: ir::HeapBase::ReservedReg, min_size: 0.into(), guard_size: 0x8000_0000.into(), @@ -59,21 +59,21 @@ impl FuncEnvironment for DummyRuntime { fn make_indirect_sig(&self, func: &mut ir::Function, index: SignatureIndex) -> ir::SigRef { // A real implementation would probably change the calling convention and add `vmctx` and // signature index arguments. - func.dfg.signatures.push(self.signatures[index].clone()) + func.import_signature(self.signatures[index].clone()) } fn make_direct_func(&self, func: &mut ir::Function, index: FunctionIndex) -> ir::FuncRef { let sigidx = self.func_types[index]; // A real implementation would probably add a `vmctx` argument. // And maybe attempt some signature de-duplication. - let signature = func.dfg.signatures.push(self.signatures[sigidx].clone()); + let signature = func.import_signature(self.signatures[sigidx].clone()); let name = match self.imported_funcs.get(index) { Some(name) => name.clone(), None => ir::FunctionName::new(format!("localfunc{}", index)), }; - func.dfg.ext_funcs.push(ir::ExtFuncData { name, signature }) + func.import_function(ir::ExtFuncData { name, signature }) } fn translate_call_indirect( From 47bc963ba5e7cba890d8b2091d14018bd5af943e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sun, 10 Sep 2017 16:17:03 -0700 Subject: [PATCH 1080/3084] Add a JumpTableData::with_capacity and use it. As with Vec::with_capacity, this helps reduce intermediate heap allocation. --- lib/cretonne/src/ir/jumptable.rs | 8 ++++++++ lib/wasm/src/code_translator.rs | 4 ++-- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index 6d97341458..319eeb1edf 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -31,6 +31,14 @@ impl JumpTableData { } } + /// Create a new empty jump table with the specified capacity. + pub fn with_capacity(capacity: usize) -> JumpTableData { + JumpTableData { + table: Vec::with_capacity(capacity), + holes: 0, + } + } + /// Set a table entry. /// /// The table will grow as needed to fit `idx`. diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 096d7d2b4d..ad099dc398 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -364,7 +364,7 @@ pub fn translate_operator( if jump_args_count == 0 { // No jump arguments let val = state.pop1(); - let mut data = JumpTableData::new(); + let mut data = JumpTableData::with_capacity(depths.len()); for depth in depths { let i = state.control_stack.len() - 1 - (depth as usize); let frame = &mut state.control_stack[i]; @@ -385,7 +385,7 @@ pub fn translate_operator( // We then proceed to split the edges going out of the br_table let val = state.pop1(); let cut_index = state.stack.len() - jump_args_count; - let mut data = JumpTableData::new(); + let mut data = JumpTableData::with_capacity(depths.len()); let dest_ebbs: HashMap = depths.iter().fold(HashMap::new(), |mut acc, &depth| { if acc.get(&(depth as usize)).is_none() { From a2542ed71d329f9482f53bc44ec15e6bb5ae2126 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 11 Sep 2017 08:31:35 -0700 Subject: [PATCH 1081/3084] Replace a match with a .unwrap_or_default(). --- lib/wasm/src/module_translator.rs | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 113330d4cf..b66e896380 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -192,10 +192,7 @@ pub fn translate_module( } // At this point we've entered the code section // First we check that we have all that is necessary to translate a function. - let signatures = match signatures { - None => Vec::new(), - Some(sigs) => sigs, - }; + let signatures = signatures.unwrap_or_default(); let functions = match functions { None => return Err(String::from("missing a function section")), Some(functions) => functions, From 7bf2747e1ee7655bb47c08e0a3d02c9deb44f2ee Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 11 Sep 2017 08:32:02 -0700 Subject: [PATCH 1082/3084] Replace a match with a `?`. --- lib/wasm/src/module_translator.rs | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index b66e896380..984f5ab78b 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -220,7 +220,7 @@ pub fn translate_module( _ => return Err(String::from("wrong content in code section")), }; let signature = signatures[functions[function_index]].clone(); - match translate_function_body( + let il_func = translate_function_body( &mut parser, function_index, &signature, @@ -228,10 +228,8 @@ pub fn translate_module( &exports, &mut il_builder, runtime, - ) { - Ok(il_func) => il_functions.push(il_func), - Err(s) => return Err(s), - } + )?; + il_functions.push(il_func); function_index += 1; } loop { From 25af6d380bed202e4597a8a3470508dd7fc9379b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 11 Sep 2017 11:04:38 -0700 Subject: [PATCH 1083/3084] Add a return_at_end setting. The flag guarantees that the generated function does not have any internal return instructions. If the function returns at all, the return must be the last instruction. For now just implement a verifier check for this property. When we get CFG simplifiers and block layout optimizations, they will need to heed the flag. --- .../filetests/verifier/return_at_end.cton | 26 +++++++++++++++++++ lib/cretonne/meta/base/settings.py | 10 +++++++ lib/cretonne/src/ir/layout.rs | 5 ++++ lib/cretonne/src/settings.rs | 1 + lib/cretonne/src/verifier/mod.rs | 19 ++++++++++++++ 5 files changed, 61 insertions(+) create mode 100644 cranelift/filetests/verifier/return_at_end.cton diff --git a/cranelift/filetests/verifier/return_at_end.cton b/cranelift/filetests/verifier/return_at_end.cton new file mode 100644 index 0000000000..b0fe05889a --- /dev/null +++ b/cranelift/filetests/verifier/return_at_end.cton @@ -0,0 +1,26 @@ +test verifier +set return_at_end + +; The verifier doesn't have an API for verifying with flags without also +; setting an ISA. +isa riscv + +function %ok(i32) { +ebb0(v0: i32): + brnz v0, ebb1 + trap + +ebb1: + trapz v0 + return +} + +function %bad(i32) { +ebb0(v0: i32): + brnz v0, ebb1 + return ; error: Internal return not allowed + +ebb1: + trapz v0 + return +} diff --git a/lib/cretonne/meta/base/settings.py b/lib/cretonne/meta/base/settings.py index 46ff16d851..2baf9927ce 100644 --- a/lib/cretonne/meta/base/settings.py +++ b/lib/cretonne/meta/base/settings.py @@ -28,6 +28,16 @@ enable_verifier = BoolSetting( is_64bit = BoolSetting("Enable 64-bit code generation") +return_at_end = BoolSetting( + """ + Generate functions with at most a single return instruction at the + end of the function. + + This guarantees that functions do not have any internal return + instructions. Either they never return, or they have a single return + instruction at the end. + """) + is_compressed = BoolSetting("Enable compressed instructions") enable_float = BoolSetting( diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 6d16b004e1..5c908812cf 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -374,6 +374,11 @@ impl Layout { self.first_ebb } + /// Get the last EBB in the layout. + pub fn last_ebb(&self) -> Option { + self.last_ebb + } + /// Get the block following `ebb` in the layout order. pub fn next_ebb(&self, ebb: Ebb) -> Option { self.ebbs[ebb].next.expand() diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 999d305439..e45dba395c 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -319,6 +319,7 @@ mod tests { opt_level = \"default\"\n\ enable_verifier = false\n\ is_64bit = false\n\ + return_at_end = false\n\ is_compressed = false\n\ enable_float = true\n\ enable_simd = true\n\ diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index e6f9ea933e..f3952606ca 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -942,6 +942,21 @@ impl<'a> Verifier<'a> { Ok(()) } + /// Verify the `return_at_end` property which requires that there are no internal return + /// instructions. + fn verify_return_at_end(&self) -> Result { + for ebb in self.func.layout.ebbs() { + let inst = self.func.layout.last_inst(ebb).unwrap(); + if self.func.dfg[inst].opcode().is_return() && + Some(ebb) != self.func.layout.last_ebb() + { + return err!(inst, "Internal return not allowed with return_at_end=1"); + } + } + + Ok(()) + } + pub fn run(&self) -> Result { self.verify_global_vars()?; self.typecheck_entry_block_arguments()?; @@ -954,6 +969,10 @@ impl<'a> Verifier<'a> { } } + if self.isa.map(|isa| isa.flags().return_at_end()) == Some(true) { + self.verify_return_at_end()?; + } + Ok(()) } } From f8f6878b2cf086dcadf6a90198b31316843b19be Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sat, 2 Sep 2017 04:14:11 -0700 Subject: [PATCH 1084/3084] Switch to wasmparser's create_binary_reader() API for reading function bodies. A subtle difference here is that the `end` at the end of a function body now gets handled by translate_operator/translate_unreachable_operator, so we no longer have to do as much special-case cleanup at the end of the function body. This is a preperatory step for converting module_translator.rs to use the new FuncTranslator mechanism. --- lib/wasm/src/code_translator.rs | 38 +++++++-------------------------- 1 file changed, 8 insertions(+), 30 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index ad099dc398..70fbf98c6d 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -27,7 +27,7 @@ use cretonne::ir::types::*; use cretonne::ir::immediates::{Ieee32, Ieee64}; use cretonne::ir::condcodes::{IntCC, FloatCC}; use cton_frontend::{ILBuilder, FunctionBuilder}; -use wasmparser::{Parser, ParserState, Operator, WasmDecoder, MemoryImmediate}; +use wasmparser::{Parser, Operator, WasmDecoder, MemoryImmediate}; use translation_utils::{f32_translation, f64_translation, type_to_type, translate_type, Local}; use translation_utils::{TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; use state::{TranslationState, ControlStackFrame}; @@ -91,37 +91,15 @@ pub fn translate_function_body( state.initialize(sig, end_ebb); // Now the main loop that reads every wasm instruction and translates it - loop { - let parser_state = parser.read(); - match *parser_state { - ParserState::CodeOperator(ref op) => { - translate_operator(op, &mut builder, &mut state, runtime) - } - - ParserState::EndFunctionBody => break, - _ => return Err(String::from("wrong content in function body")), - } + let mut reader = parser.create_binary_reader(); + while let Ok(ref op) = reader.read_operator() { + translate_operator(op, &mut builder, &mut state, runtime) } - // In WebAssembly, the final return instruction is implicit so we need to build it - // explicitely in Cretonne IL. - if !builder.is_filled() && (!builder.is_unreachable() || !builder.is_pristine()) { - let cut_index = state.stack.len() - sig.return_types.len(); - builder.ins().return_(&state.stack[cut_index..]); - state.stack.truncate(cut_index); - } - // Because the function has an implicit block as body, we need to explicitely close it. - let frame = state.control_stack.pop().unwrap(); - builder.switch_to_block(frame.following_code(), frame.return_values()); - builder.seal_block(frame.following_code()); - // If the block is reachable we also have to include a return instruction in it. + debug_assert!(state.control_stack.is_empty()); + debug_assert!(builder.is_pristine()); if !builder.is_unreachable() { - state.stack.truncate(frame.original_stack_size()); - state.stack.extend_from_slice( - builder.ebb_args(frame.following_code()), - ); - let cut_index = state.stack.len() - sig.return_types.len(); - builder.ins().return_(&state.stack[cut_index..]); - state.stack.truncate(cut_index); + debug_assert!(state.stack.len() == sig.return_types.len()); + builder.ins().return_(&state.stack); } } Ok(func) From 9bce21c17cee54c2d2b711e731f90f343933f151 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 11 Sep 2017 11:34:39 -0700 Subject: [PATCH 1085/3084] Update to wasmparser 0.9.4 for a bugfix in create_binary_reader. --- lib/wasm/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 8cc3b40819..9cabdc1b92 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -11,6 +11,6 @@ license = "Apache-2.0" name = "cton_wasm" [dependencies] -wasmparser = "0.9.3" +wasmparser = "0.9.4" cretonne = { path = "../cretonne" } cretonne-frontend = { path = "../frontend" } From 95e2566d6fec83a2b2f463251b1616beb6373bbb Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 11 Sep 2017 13:58:49 -0700 Subject: [PATCH 1086/3084] Use TranslationState's popn/peekn functions. (#151) This entailed reorganizing surrounding code to minimize the extent of mutable borrows of the control-flow stack. --- lib/wasm/src/code_translator.rs | 74 +++++++++++++++++---------------- 1 file changed, 39 insertions(+), 35 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 70fbf98c6d..8a1b7c4e14 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -224,18 +224,17 @@ pub fn translate_operator( // and push a new control frame with a new ebb for the code after the if/then/else // At the end of the then clause we jump to the destination let i = state.control_stack.len() - 1; - let (destination, return_values, branch_inst) = match state.control_stack[i] { + let (destination, return_count, branch_inst) = match state.control_stack[i] { ControlStackFrame::If { destination, ref return_values, branch_inst, .. - } => (destination, return_values, branch_inst), + } => (destination, return_values.len(), branch_inst), _ => panic!("should not happen"), }; - let cut_index = state.stack.len() - return_values.len(); - builder.ins().jump(destination, &state.stack[cut_index..]); - state.stack.truncate(cut_index); + builder.ins().jump(destination, state.peekn(return_count)); + state.popn(return_count); // We change the target of the branch instruction let else_ebb = builder.create_ebb(); builder.change_jump_destination(branch_inst, else_ebb); @@ -245,12 +244,12 @@ pub fn translate_operator( Operator::End => { let frame = state.control_stack.pop().unwrap(); if !builder.is_unreachable() || !builder.is_pristine() { - let cut_index = state.stack.len() - frame.return_values().len(); + let return_count = frame.return_values().len(); builder.ins().jump( frame.following_code(), - &state.stack[cut_index..], + state.peekn(return_count), ); - state.stack.truncate(cut_index); + state.popn(return_count); } builder.switch_to_block(frame.following_code(), frame.return_values()); builder.seal_block(frame.following_code()); @@ -287,40 +286,44 @@ pub fn translate_operator( ***********************************************************************************/ Operator::Br { relative_depth } => { let i = state.control_stack.len() - 1 - (relative_depth as usize); - let frame = &mut state.control_stack[i]; - let cut_index = state.stack.len() - - if frame.is_loop() { + let (return_count, br_destination) = { + let frame = &mut state.control_stack[i]; + // We signal that all the code that follows until the next End is unreachable + frame.set_reachable(); + let return_count = if frame.is_loop() { 0 } else { frame.return_values().len() }; + (return_count, frame.br_destination()) + }; builder.ins().jump( - frame.br_destination(), - &state.stack[cut_index..], + br_destination, + state.peekn(return_count), ); - state.stack.truncate(cut_index); - // We signal that all the code that follows until the next End is unreachable - frame.set_reachable(); + state.popn(return_count); state.real_unreachable_stack_depth = 1 + relative_depth as usize; } Operator::BrIf { relative_depth } => { let val = state.pop1(); let i = state.control_stack.len() - 1 - (relative_depth as usize); - let frame = &mut state.control_stack[i]; - let cut_index = state.stack.len() - - if frame.is_loop() { + let (return_count, br_destination) = { + let frame = &mut state.control_stack[i]; + // The values returned by the branch are still available for the reachable + // code that comes after it + frame.set_reachable(); + let return_count = if frame.is_loop() { 0 } else { frame.return_values().len() }; + (return_count, frame.br_destination()) + }; builder.ins().brnz( val, - frame.br_destination(), - &state.stack[cut_index..], + br_destination, + state.peekn(return_count), ); - // The values returned by the branch are still available for the reachable - // code that comes after it - frame.set_reachable(); } Operator::BrTable { ref table } => { let (depths, default) = table.read_table(); @@ -362,7 +365,7 @@ pub fn translate_operator( // Here we have jump arguments, but Cretonne's br_table doesn't support them // We then proceed to split the edges going out of the br_table let val = state.pop1(); - let cut_index = state.stack.len() - jump_args_count; + let return_count = jump_args_count; let mut data = JumpTableData::with_capacity(depths.len()); let dest_ebbs: HashMap = depths.iter().fold(HashMap::new(), |mut acc, &depth| { @@ -381,25 +384,26 @@ pub fn translate_operator( let default_ebb = state.control_stack[state.control_stack.len() - 1 - (default as usize)] .br_destination(); - builder.ins().jump(default_ebb, &state.stack[cut_index..]); + builder.ins().jump(default_ebb, state.peekn(return_count)); for (depth, dest_ebb) in dest_ebbs { builder.switch_to_block(dest_ebb, &[]); builder.seal_block(dest_ebb); let i = state.control_stack.len() - 1 - depth; - let frame = &mut state.control_stack[i]; - let real_dest_ebb = frame.br_destination(); - builder.ins().jump(real_dest_ebb, &state.stack[cut_index..]); - frame.set_reachable(); + let real_dest_ebb = { + let frame = &mut state.control_stack[i]; + frame.set_reachable(); + frame.br_destination() + }; + builder.ins().jump(real_dest_ebb, state.peekn(return_count)); } - state.stack.truncate(cut_index); + state.popn(return_count); state.real_unreachable_stack_depth = 1 + min_depth as usize; } } Operator::Return => { let return_count = state.control_stack[0].return_values().len(); - let cut_index = state.stack.len() - return_count; - builder.ins().return_(&state.stack[cut_index..]); - state.stack.truncate(cut_index); + builder.ins().return_(state.peekn(return_count)); + state.popn(return_count); state.real_unreachable_stack_depth = 1; } /************************************ Calls **************************************** @@ -429,7 +433,7 @@ pub fn translate_operator( index as SignatureIndex, sigref, callee, - &state.peekn(num_args), + state.peekn(num_args), ); state.popn(num_args); state.pushn(builder.func.dfg.inst_results(call)); From de27abcb2b7089dcf29e7cec1ecc6e7e5f6d0090 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 11 Sep 2017 16:37:50 -0700 Subject: [PATCH 1087/3084] Change translate_module to use FuncTranslator. --- lib/wasm/Cargo.toml | 2 +- lib/wasm/src/code_translator.rs | 80 ++----------------------------- lib/wasm/src/func_translator.rs | 33 +++++++++---- lib/wasm/src/module_translator.rs | 53 ++++++++------------ 4 files changed, 50 insertions(+), 118 deletions(-) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 9cabdc1b92..3de76671d0 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -11,6 +11,6 @@ license = "Apache-2.0" name = "cton_wasm" [dependencies] -wasmparser = "0.9.4" +wasmparser = "0.10.0" cretonne = { path = "../cretonne" } cretonne-frontend = { path = "../frontend" } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 8a1b7c4e14..625552232b 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -21,90 +21,18 @@ //! //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as //! argument. -use cretonne::ir::{self, Function, Signature, Type, InstBuilder, FunctionName, Ebb, MemFlags, - JumpTableData}; +use cretonne::ir::{self, InstBuilder, Ebb, MemFlags, JumpTableData}; use cretonne::ir::types::*; -use cretonne::ir::immediates::{Ieee32, Ieee64}; use cretonne::ir::condcodes::{IntCC, FloatCC}; -use cton_frontend::{ILBuilder, FunctionBuilder}; -use wasmparser::{Parser, Operator, WasmDecoder, MemoryImmediate}; +use cton_frontend::FunctionBuilder; +use wasmparser::{Operator, MemoryImmediate}; use translation_utils::{f32_translation, f64_translation, type_to_type, translate_type, Local}; use translation_utils::{TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; use state::{TranslationState, ControlStackFrame}; use std::collections::HashMap; -use runtime::{FuncEnvironment, GlobalValue, WasmRuntime}; +use runtime::{FuncEnvironment, GlobalValue}; use std::u32; -/// Returns a well-formed Cretonne IL function from a wasm function body and a signature. -pub fn translate_function_body( - parser: &mut Parser, - function_index: FunctionIndex, - sig: &Signature, - locals: &[(usize, Type)], - exports: &Option>, - il_builder: &mut ILBuilder, - runtime: &mut WasmRuntime, -) -> Result { - runtime.next_function(); - // First we build the Function object with its name and signature - let mut func = Function::new(); - let args_num: usize = sig.argument_types.len(); - func.signature = sig.clone(); - if let Some(ref exports) = *exports { - if let Some(name) = exports.get(&function_index) { - func.name = FunctionName::new(name.clone()); - } - } - // We introduce an arbitrary scope for the FunctionBuilder object - { - let mut builder = FunctionBuilder::new(&mut func, il_builder); - let first_ebb = builder.create_ebb(); - builder.switch_to_block(first_ebb, &[]); - builder.seal_block(first_ebb); - for (i, arg_type) in sig.argument_types.iter().enumerate() { - // First we declare the function arguments' as non-SSA vars because they will be - // accessed by get_local - let arg_value = builder.arg_value(i); - builder.declare_var(Local(i as u32), arg_type.value_type); - builder.def_var(Local(i as u32), arg_value); - } - // We also declare and initialize to 0 the local variables - let mut local_index = args_num; - for &(loc_count, ty) in locals { - let val = match ty { - I32 => builder.ins().iconst(ty, 0), - I64 => builder.ins().iconst(ty, 0), - F32 => builder.ins().f32const(Ieee32::with_bits(0)), - F64 => builder.ins().f64const(Ieee64::with_bits(0)), - _ => panic!("should not happen"), - }; - for _ in 0..loc_count { - builder.declare_var(Local(local_index as u32), ty); - builder.def_var(Local(local_index as u32), val); - local_index += 1; - } - } - - // We initialize the control stack with the implicit function block - let mut state = TranslationState::new(); - let end_ebb = builder.create_ebb(); - state.initialize(sig, end_ebb); - - // Now the main loop that reads every wasm instruction and translates it - let mut reader = parser.create_binary_reader(); - while let Ok(ref op) = reader.read_operator() { - translate_operator(op, &mut builder, &mut state, runtime) - } - debug_assert!(state.control_stack.is_empty()); - debug_assert!(builder.is_pristine()); - if !builder.is_unreachable() { - debug_assert!(state.stack.len() == sig.return_types.len()); - builder.ins().return_(&state.stack); - } - } - Ok(func) -} - /// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted /// a return. pub fn translate_operator( diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 9eba880f32..e2dede8b88 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -59,10 +59,20 @@ impl FuncTranslator { code: &[u8], func: &mut ir::Function, environ: &mut FE, + ) -> CtonResult { + self.translate_from_reader(BinaryReader::new(code), func, environ) + } + + /// Translate a binary WebAssembly function from a `BinaryReader`. + pub fn translate_from_reader( + &mut self, + mut reader: BinaryReader, + func: &mut ir::Function, + environ: &mut FE, ) -> CtonResult { dbg!( "translate({} bytes, {}{})", - code.len(), + reader.bytes_remaining(), func.name, func.signature ); @@ -81,8 +91,7 @@ impl FuncTranslator { let exit_block = builder.create_ebb(); self.state.initialize(&builder.func.signature, exit_block); - let reader = &mut BinaryReader::new(code); - parse_local_decls(reader, builder, num_args)?; + parse_local_decls(&mut reader, builder, num_args)?; parse_function_body(reader, builder, &mut self.state, environ) } } @@ -120,12 +129,15 @@ fn parse_local_decls( num_args: usize, ) -> CtonResult { let mut next_local = num_args; - let local_count = reader.read_var_u32().map_err(|_| CtonError::InvalidInput)?; + let local_count = reader.read_local_count().map_err( + |_| CtonError::InvalidInput, + )?; + let mut locals_total = 0; for _ in 0..local_count { - let (count, ty) = reader.read_local_decl().map_err( - |_| CtonError::InvalidInput, - )?; + let (count, ty) = reader.read_local_decl(&mut locals_total).map_err(|_| { + CtonError::InvalidInput + })?; declare_locals(builder, count, ty, &mut next_local)?; } @@ -173,7 +185,7 @@ fn declare_locals( /// This assumes that the local variable declarations have already been parsed and function /// arguments and locals are declared in the builder. fn parse_function_body( - reader: &mut BinaryReader, + mut reader: BinaryReader, builder: &mut FunctionBuilder, state: &mut TranslationState, environ: &mut FE, @@ -192,10 +204,13 @@ fn parse_function_body( // // If the exit block is unreachable, it may not have the correct arguments, so we would // generate a return instruction that doesn't match the signature. + debug_assert!(builder.is_pristine()); if !builder.is_unreachable() { - builder.ins().return_(state.stack.as_slice()); + builder.ins().return_(&state.stack); } + debug_assert!(reader.eof()); + Ok(()) } diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 984f5ab78b..b9df83f9fe 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -5,11 +5,11 @@ use sections_translator::{SectionParsingError, parse_function_signatures, parse_ parse_function_section, parse_export_section, parse_memory_section, parse_global_section, parse_table_section, parse_elements_section, parse_data_section}; -use translation_utils::{type_to_type, Import, SignatureIndex, FunctionIndex}; -use cretonne::ir::{Function, Type}; -use code_translator::translate_function_body; -use cton_frontend::ILBuilder; +use translation_utils::{Import, SignatureIndex, FunctionIndex}; +use cretonne::ir::{Function, FunctionName}; +use func_translator::FuncTranslator; use std::collections::HashMap; +use std::error::Error; use runtime::WasmRuntime; /// Output of the [`translate_module`](fn.translate_module.html) function. @@ -198,38 +198,27 @@ pub fn translate_module( Some(functions) => functions, }; let mut il_functions: Vec = Vec::new(); - let mut il_builder = ILBuilder::new(); + let mut trans = FuncTranslator::new(); runtime.begin_translation(); loop { - let locals: Vec<(usize, Type)> = match *parser.read() { - ParserState::BeginFunctionBody { ref locals, .. } => { - locals - .iter() - .map(|&(index, ref ty)| { - ( - index as usize, - match type_to_type(ty) { - Ok(ty) => ty, - Err(()) => panic!("unsupported type for local variable"), - }, - ) - }) - .collect() - } + match *parser.read() { + ParserState::BeginFunctionBody { .. } => {} ParserState::EndSection => break, _ => return Err(String::from("wrong content in code section")), - }; - let signature = signatures[functions[function_index]].clone(); - let il_func = translate_function_body( - &mut parser, - function_index, - &signature, - &locals, - &exports, - &mut il_builder, - runtime, - )?; - il_functions.push(il_func); + } + runtime.next_function(); + // First we build the Function object with its name and signature + let mut func = Function::new(); + func.signature = signatures[functions[function_index]].clone(); + if let Some(ref exports) = exports { + if let Some(name) = exports.get(&function_index) { + func.name = FunctionName::new(name.clone()); + } + } + trans + .translate_from_reader(parser.create_binary_reader(), &mut func, runtime) + .map_err(|e| String::from(e.description()))?; + il_functions.push(func); function_index += 1; } loop { From ddd398b790bff42a22edbc987c086db4b5543ba6 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Sep 2017 09:00:32 -0700 Subject: [PATCH 1088/3084] Remove a redundant copy of pretty_verifier_error. --- cranelift/src/wasm.rs | 25 +------------------------ 1 file changed, 1 insertion(+), 24 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 0dda98b524..c915379d7e 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -12,9 +12,6 @@ use cretonne::flowgraph::ControlFlowGraph; use cretonne::dominator_tree::DominatorTree; use cretonne::Context; use cretonne::result::CtonError; -use cretonne::ir; -use cretonne::ir::entities::AnyEntity; -use cretonne::isa::TargetIsa; use cretonne::verifier; use std::fs::File; use std::error::Error; @@ -25,6 +22,7 @@ use std::path::Path; use std::process::Command; use tempdir::TempDir; use term; +use utils::pretty_verifier_error; macro_rules! vprintln { ($x: expr, $($tts:tt)*) => { @@ -218,24 +216,3 @@ fn handle_module( } Ok(()) } - -/// Pretty-print a verifier error. -pub fn pretty_verifier_error( - func: &ir::Function, - isa: Option<&TargetIsa>, - err: verifier::Error, -) -> String { - let msg = err.to_string(); - let str1 = match err.location { - AnyEntity::Inst(inst) => { - format!( - "{}\n{}: {}\n\n", - msg, - inst, - func.dfg.display_inst(inst, isa) - ) - } - _ => String::from(format!("{}\n", msg)), - }; - format!("{}{}", str1, func.display(isa)) -} From cc35b5e7246672b0768a14d4e004c76026dcf84c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 12 Sep 2017 08:59:35 -0700 Subject: [PATCH 1089/3084] Flush the debug trace file after each dbg!(). When debugging code built with panic=about, the debug trace buffer is not flushed on a panic and important messages are lost. This does slow down debug tracing a bit, but this is still better than unbuffered or line buffered I/O since the dbg!() macro can write out many lines like a whole function. --- lib/cretonne/src/dbg.rs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/dbg.rs b/lib/cretonne/src/dbg.rs index 0ac03f4c4f..9bd5500e3c 100644 --- a/lib/cretonne/src/dbg.rs +++ b/lib/cretonne/src/dbg.rs @@ -63,7 +63,11 @@ thread_local! { /// /// This is for use by the `dbg!` macro. pub fn writeln_with_format_args(args: fmt::Arguments) -> io::Result<()> { - WRITER.with(|rc| writeln!(*rc.borrow_mut(), "{}", args)) + WRITER.with(|rc| { + let mut w = rc.borrow_mut(); + writeln!(*w, "{}", args)?; + w.flush() + }) } /// Open the tracing file for the current thread. From 9b5295f3e23cdec344a24f62a9956d9e96857d5a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 11 Sep 2017 13:32:55 -0700 Subject: [PATCH 1090/3084] Add a spiderwasm_prologue_words setting. This makes the details of the spiderwasm prologue configurable so it is easier to modify SpiderMonkey without having to change Cretonne. Create a stack object representing the SpiderMonkey prologue words before calculating the stack layout so they won't be overwritten by Cretonne's stack objects. --- lib/cretonne/meta/base/settings.py | 20 +++++++++++++++++++- lib/cretonne/src/isa/mod.rs | 14 ++++++++++++-- lib/cretonne/src/settings.rs | 4 +++- 3 files changed, 34 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/meta/base/settings.py b/lib/cretonne/meta/base/settings.py index 2baf9927ce..434bd28a52 100644 --- a/lib/cretonne/meta/base/settings.py +++ b/lib/cretonne/meta/base/settings.py @@ -4,7 +4,7 @@ Cretonne shared settings. This module defines settings relevant for all code generators. """ from __future__ import absolute_import -from cdsl.settings import SettingGroup, BoolSetting, EnumSetting +from cdsl.settings import SettingGroup, BoolSetting, EnumSetting, NumSetting group = SettingGroup('shared') @@ -52,4 +52,22 @@ enable_atomics = BoolSetting( """Enable the use of atomic instructions""", default=True) +# +# Settings specific to the `spiderwasm` calling convention. +# +spiderwasm_prologue_words = NumSetting( + """ + Number of pointer-sized words pushed by the spiderwasm prologue. + + Functions with the `spiderwasm` calling convention don't generate their + own prologue and epilogue. They depend on externally generated code + that pushes a fixed number of words in the prologue and restores them + in the epilogue. + + This setting configures the number of pointer-sized words pushed on the + stack when the Cretonne-generated code is entered. This includes the + pushed return address on Intel ISAs. + """) + + group.close(globals()) diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index b68cd17041..1ccacce8b1 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -236,9 +236,19 @@ pub trait TargetIsa { fn prologue_epilogue(&self, func: &mut ir::Function) -> result::CtonResult { // This default implementation is unlikely to be good enough. use stack_layout::layout_stack; + use ir::stackslot::{StackSize, StackOffset}; - let align = if self.flags().is_64bit() { 8 } else { 4 }; - layout_stack(&mut func.stack_slots, align)?; + let word_size = if self.flags().is_64bit() { 8 } else { 4 }; + + // Account for the SpiderMonkey standard prologue pushes. + if func.signature.call_conv == ir::CallConv::SpiderWASM { + let bytes = self.flags().spiderwasm_prologue_words() as StackSize * word_size; + let mut ss = ir::StackSlotData::new(ir::StackSlotKind::IncomingArg, bytes); + ss.offset = -(bytes as StackOffset); + func.stack_slots.push(ss); + } + + layout_stack(&mut func.stack_slots, word_size)?; Ok(()) } diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index e45dba395c..43d1639028 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -323,10 +323,12 @@ mod tests { is_compressed = false\n\ enable_float = true\n\ enable_simd = true\n\ - enable_atomics = true\n" + enable_atomics = true\n\ + spiderwasm_prologue_words = 0\n" ); assert_eq!(f.opt_level(), super::OptLevel::Default); assert_eq!(f.enable_simd(), true); + assert_eq!(f.spiderwasm_prologue_words(), 0); } #[test] From 2e046d68ce1c20c31e69d7fd479a48e232e01b7b Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Sep 2017 12:28:25 -0700 Subject: [PATCH 1091/3084] Remove obsolete derivations of Hash. --- lib/frontend/src/frontend.rs | 31 ++++++++++++++++--------------- lib/frontend/src/lib.rs | 2 +- lib/frontend/src/ssa.rs | 5 ++--- 3 files changed, 19 insertions(+), 19 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 00e6377897..696dad6556 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -8,12 +8,11 @@ use cretonne::ir::function::DisplayFunction; use cretonne::isa::TargetIsa; use ssa::{SSABuilder, SideEffects, Block}; use cretonne::entity::{EntityRef, EntityMap, EntitySet}; -use std::hash::Hash; /// Permanent structure used for translating into Cretonne IL. pub struct ILBuilder where - Variable: EntityRef + Hash + Default, + Variable: EntityRef + Default, { ssa: SSABuilder, ebbs: EntityMap, @@ -25,7 +24,7 @@ where /// Temporary object used to build a Cretonne IL `Function`. pub struct FunctionBuilder<'a, Variable: 'a> where - Variable: EntityRef + Hash + Default, + Variable: EntityRef + Default, { /// The function currently being built. /// This field is public so the function can be re-borrowed. @@ -50,7 +49,7 @@ struct Position { impl ILBuilder where - Variable: EntityRef + Hash + Default, + Variable: EntityRef + Default, { /// Creates a ILBuilder structure. The structure is automatically cleared each time it is /// passed to a [`FunctionBuilder`](struct.FunctionBuilder.html) for creation. @@ -75,24 +74,26 @@ where /// one convenience method per Cretonne IL instruction. pub struct FuncInstBuilder<'short, 'long: 'short, Variable: 'long> where - Variable: EntityRef + Hash + Default, + Variable: EntityRef + Default, { builder: &'short mut FunctionBuilder<'long, Variable>, ebb: Ebb, } impl<'short, 'long, Variable> FuncInstBuilder<'short, 'long, Variable> - where Variable: EntityRef + Hash + Default +where + Variable: EntityRef + Default, { - fn new<'s, 'l>(builder: &'s mut FunctionBuilder<'l, Variable>, - ebb: Ebb) - -> FuncInstBuilder<'s, 'l, Variable> { + fn new<'s, 'l>( + builder: &'s mut FunctionBuilder<'l, Variable>, + ebb: Ebb, + ) -> FuncInstBuilder<'s, 'l, Variable> { FuncInstBuilder { builder, ebb } } } impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long, Variable> - where Variable: EntityRef + Hash + Default + where Variable: EntityRef + Default { fn data_flow_graph(&self) -> &DataFlowGraph { &self.builder.func.dfg @@ -201,7 +202,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short /// return instruction with arguments that don't match the function's signature. impl<'a, Variable> FunctionBuilder<'a, Variable> where - Variable: EntityRef + Hash + Default, + Variable: EntityRef + Default, { /// Creates a new FunctionBuilder structure that will operate on a `Function` using a /// `IlBuilder`. @@ -404,7 +405,7 @@ where /// in ways that can be unsafe if used incorrectly. impl<'a, Variable> FunctionBuilder<'a, Variable> where - Variable: EntityRef + Hash + Default, + Variable: EntityRef + Default, { /// Retrieves all the arguments for an `Ebb` currently infered from the jump instructions /// inserted that target it and the SSA construction. @@ -483,7 +484,7 @@ where impl<'a, Variable> Drop for FunctionBuilder<'a, Variable> where - Variable: EntityRef + Hash + Default, + Variable: EntityRef + Default, { /// When a `FunctionBuilder` goes out of scope, it means that the function is fully built. /// We then proceed to check if all the `Ebb`s are filled and sealed @@ -501,7 +502,7 @@ where // Helper functions impl<'a, Variable> FunctionBuilder<'a, Variable> where - Variable: EntityRef + Hash + Default, + Variable: EntityRef + Default, { fn move_to_next_basic_block(&mut self) { self.position.basic_block = self.builder.ssa.declare_ebb_body_block( @@ -634,7 +635,7 @@ mod tests { use std::u32; // An opaque reference to variable. - #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] + #[derive(Copy, Clone, PartialEq, Eq, Debug)] pub struct Variable(u32); impl EntityRef for Variable { fn new(index: usize) -> Self { diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 69723ae50c..af31a2c723 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -43,7 +43,7 @@ //! use std::u32; //! //! // An opaque reference to variable. -//! #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +//! #[derive(Copy, Clone, PartialEq, Eq, Debug)] //! pub struct Variable(u32); //! impl EntityRef for Variable { //! fn new(index: usize) -> Self { diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index a4c99885a0..3dd0335009 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -8,7 +8,6 @@ use cretonne::ir::{Ebb, Value, Inst, Type, DataFlowGraph, JumpTables, Layout, Cursor, CursorBase, InstBuilder}; use cretonne::ir::instructions::BranchInfo; -use std::hash::Hash; use cretonne::entity::{EntityRef, PrimaryMap, EntityMap}; use cretonne::packed_option::PackedOption; use cretonne::packed_option::ReservedValue; @@ -200,7 +199,7 @@ enum UseVarCases { /// impl SSABuilder where - Variable: EntityRef + Hash + Default, + Variable: EntityRef + Default, { /// Declares a new definition of a variable in a given basic block. /// The SSA value is passed as an argument because it should be created with @@ -594,7 +593,7 @@ mod tests { use std::u32; /// An opaque reference to variable. - #[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] + #[derive(Copy, Clone, PartialEq, Eq, Debug)] pub struct Variable(u32); impl EntityRef for Variable { fn new(index: usize) -> Self { From 1ab207b93c9e0cef42cee415f1079b288c636265 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Sep 2017 13:27:36 -0700 Subject: [PATCH 1092/3084] Add support for emitting code with a single return at the end. (#153) This also enables testing of the wasmtests tests. This also updates for wabt updating to the official "wat" filename extension, as opposed to "wast". --- cranelift/src/cton-util.rs | 4 +- cranelift/src/wasm.rs | 29 ++++-- cranelift/wasmtests/{arith.wast => arith.wat} | 0 cranelift/wasmtests/{call.wast => call.wat} | 0 .../{fibonacci.wast => fibonacci.wat} | 0 .../wasmtests/{globals.wast => globals.wat} | 0 cranelift/wasmtests/{icall.wast => icall.wat} | 0 .../wasmtests/{memory.wast => memory.wat} | 0 cranelift/wasmtests/return_at_end.wat | 10 +++ lib/wasm/Cargo.toml | 3 + lib/wasm/src/code_translator.rs | 16 +++- lib/wasm/src/func_translator.rs | 6 +- lib/wasm/src/runtime/dummy.rs | 18 +++- lib/wasm/src/runtime/spec.rs | 16 +++- lib/wasm/tests/testsuite.rs | 90 +++++++++++++++---- 15 files changed, 154 insertions(+), 38 deletions(-) rename cranelift/wasmtests/{arith.wast => arith.wat} (100%) rename cranelift/wasmtests/{call.wast => call.wat} (100%) rename cranelift/wasmtests/{fibonacci.wast => fibonacci.wat} (100%) rename cranelift/wasmtests/{globals.wast => globals.wat} (100%) rename cranelift/wasmtests/{icall.wast => icall.wat} (100%) rename cranelift/wasmtests/{memory.wast => memory.wat} (100%) create mode 100644 cranelift/wasmtests/return_at_end.wat diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index bd104e9d5f..a7ad8a44b4 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -30,7 +30,7 @@ Usage: cton-util cat ... cton-util filecheck [-v] cton-util print-cfg ... - cton-util wasm [-cvo] ... + cton-util wasm [-cvo] [--enable=]... ... cton-util --help | --version Options: @@ -53,6 +53,7 @@ struct Args { flag_check: bool, flag_optimize: bool, flag_verbose: bool, + flag_enable: Vec, } /// A command either succeeds or fails with an error message. @@ -84,6 +85,7 @@ fn cton_util() -> CommandResult { args.flag_verbose, args.flag_optimize, args.flag_check, + args.flag_enable, ) } else { // Debugging / shouldn't happen with proper command line handling above. diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index c915379d7e..4dddddf65c 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -1,5 +1,4 @@ -//! CLI tool to use the functions provided by crates [wasm2cretonne](../wasm2cretonne/index.html) -//! and [wasmstandalone](../wasmstandalone/index.html). +//! CLI tool to use the functions provided by the [cretonne-wasm](../cton_wasm/index.html) crate. //! //! Reads Wasm binary files (one Wasm module per file), translates the functions' code to Cretonne //! IL. Can also executes the `start` function of the module by laying out the memories, globals @@ -13,6 +12,7 @@ use cretonne::dominator_tree::DominatorTree; use cretonne::Context; use cretonne::result::CtonError; use cretonne::verifier; +use cretonne::settings::{self, Configurable}; use std::fs::File; use std::error::Error; use std::io; @@ -54,7 +54,16 @@ pub fn run( flag_verbose: bool, flag_optimize: bool, flag_check: bool, + flag_enable: Vec, ) -> Result<(), String> { + let mut flag_builder = settings::builder(); + for enable in flag_enable { + flag_builder.enable(&enable).map_err(|_| { + format!("unrecognized flag: {}", enable) + })?; + } + let flags = settings::Flags::new(&flag_builder); + for filename in files { let path = Path::new(&filename); let name = String::from(path.as_os_str().to_string_lossy()); @@ -64,6 +73,7 @@ pub fn run( flag_check, path.to_path_buf(), name, + &flags, ) { Ok(()) => {} Err(message) => return Err(message), @@ -78,6 +88,7 @@ fn handle_module( flag_check: bool, path: PathBuf, name: String, + flags: &settings::Flags, ) -> Result<(), String> { let mut terminal = term::stdout().unwrap(); terminal.fg(term::color::YELLOW).unwrap(); @@ -89,7 +100,7 @@ fn handle_module( terminal.reset().unwrap(); let data = match path.extension() { None => { - return Err(String::from("the file extension is not wasm or wast")); + return Err(String::from("the file extension is not wasm or wat")); } Some(ext) => { match ext.to_str() { @@ -101,17 +112,17 @@ fn handle_module( } } } - Some("wast") => { - let tmp_dir = TempDir::new("wasm2cretonne").unwrap(); + Some("wat") => { + let tmp_dir = TempDir::new("cretonne-wasm").unwrap(); let file_path = tmp_dir.path().join("module.wasm"); File::create(file_path.clone()).unwrap(); - Command::new("wast2wasm") + Command::new("wat2wasm") .arg(path.clone()) .arg("-o") .arg(file_path.to_str().unwrap()) .output() .or_else(|e| if let io::ErrorKind::NotFound = e.kind() { - return Err(String::from("wast2wasm not found")); + return Err(String::from("wat2wasm not found")); } else { return Err(String::from(e.description())); }) @@ -124,12 +135,12 @@ fn handle_module( } } None | Some(&_) => { - return Err(String::from("the file extension is not wasm or wast")); + return Err(String::from("the file extension is not wasm or wat")); } } } }; - let mut dummy_runtime = DummyRuntime::new(); + let mut dummy_runtime = DummyRuntime::with_flags(flags.clone()); let translation = { let runtime: &mut WasmRuntime = &mut dummy_runtime; match translate_module(&data, runtime) { diff --git a/cranelift/wasmtests/arith.wast b/cranelift/wasmtests/arith.wat similarity index 100% rename from cranelift/wasmtests/arith.wast rename to cranelift/wasmtests/arith.wat diff --git a/cranelift/wasmtests/call.wast b/cranelift/wasmtests/call.wat similarity index 100% rename from cranelift/wasmtests/call.wast rename to cranelift/wasmtests/call.wat diff --git a/cranelift/wasmtests/fibonacci.wast b/cranelift/wasmtests/fibonacci.wat similarity index 100% rename from cranelift/wasmtests/fibonacci.wast rename to cranelift/wasmtests/fibonacci.wat diff --git a/cranelift/wasmtests/globals.wast b/cranelift/wasmtests/globals.wat similarity index 100% rename from cranelift/wasmtests/globals.wast rename to cranelift/wasmtests/globals.wat diff --git a/cranelift/wasmtests/icall.wast b/cranelift/wasmtests/icall.wat similarity index 100% rename from cranelift/wasmtests/icall.wast rename to cranelift/wasmtests/icall.wat diff --git a/cranelift/wasmtests/memory.wast b/cranelift/wasmtests/memory.wat similarity index 100% rename from cranelift/wasmtests/memory.wast rename to cranelift/wasmtests/memory.wat diff --git a/cranelift/wasmtests/return_at_end.wat b/cranelift/wasmtests/return_at_end.wat new file mode 100644 index 0000000000..44eab2b3f6 --- /dev/null +++ b/cranelift/wasmtests/return_at_end.wat @@ -0,0 +1,10 @@ +(module + (memory 1) + (func $main (param i32) + (if + (get_local 0) + (then (return)) + (else (unreachable)) + ) + ) +) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 3de76671d0..1bd44219c7 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -14,3 +14,6 @@ name = "cton_wasm" wasmparser = "0.10.0" cretonne = { path = "../cretonne" } cretonne-frontend = { path = "../frontend" } + +[dev-dependencies] +tempdir = "0.3.5" diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 625552232b..b3d202aa2d 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -329,8 +329,20 @@ pub fn translate_operator( } } Operator::Return => { - let return_count = state.control_stack[0].return_values().len(); - builder.ins().return_(state.peekn(return_count)); + let (return_count, br_destination) = { + let frame = &mut state.control_stack[0]; + frame.set_reachable(); + let return_count = frame.return_values().len(); + (return_count, frame.br_destination()) + }; + { + let args = state.peekn(return_count); + if environ.flags().return_at_end() { + builder.ins().jump(br_destination, args); + } else { + builder.ins().return_(args); + } + } state.popn(return_count); state.real_unreachable_stack_depth = 1; } diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index e2dede8b88..3383b0caf1 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -237,7 +237,7 @@ mod tests { ]; let mut trans = FuncTranslator::new(); - let mut runtime = DummyRuntime::new(); + let mut runtime = DummyRuntime::default(); let mut ctx = Context::new(); ctx.func.name = ir::FunctionName::new("small1"); @@ -271,7 +271,7 @@ mod tests { ]; let mut trans = FuncTranslator::new(); - let mut runtime = DummyRuntime::new(); + let mut runtime = DummyRuntime::default(); let mut ctx = Context::new(); ctx.func.name = ir::FunctionName::new("small2"); @@ -314,7 +314,7 @@ mod tests { ]; let mut trans = FuncTranslator::new(); - let mut runtime = DummyRuntime::new(); + let mut runtime = DummyRuntime::default(); let mut ctx = Context::new(); ctx.func.name = ir::FunctionName::new("infloop"); diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 04762988f2..b6d849d751 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -4,6 +4,7 @@ use translation_utils::{Global, Memory, Table, GlobalIndex, TableIndex, Signatur use cretonne::ir::{self, InstBuilder}; use cretonne::ir::types::*; use cretonne::cursor::FuncCursor; +use cretonne::settings; /// This runtime implementation is a "naïve" one, doing essentially nothing and emitting /// placeholders when forced to. Don't try to execute code translated with this runtime, it is @@ -18,23 +19,32 @@ pub struct DummyRuntime { // Names of imported functions. imported_funcs: Vec, + + // Compilation setting flags. + flags: settings::Flags, } impl DummyRuntime { - /// Allocates the runtime data structures. - pub fn new() -> Self { + /// Allocates the runtime data structures with default flags. + pub fn default() -> Self { + Self::with_flags(settings::Flags::new(&settings::builder())) + } + + /// Allocates the runtime data structures with the given flags. + pub fn with_flags(flags: settings::Flags) -> Self { Self { signatures: Vec::new(), globals: Vec::new(), func_types: Vec::new(), imported_funcs: Vec::new(), + flags, } } } impl FuncEnvironment for DummyRuntime { - fn native_pointer(&self) -> ir::Type { - ir::types::I64 + fn flags(&self) -> &settings::Flags { + &self.flags } fn make_global(&self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue { diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 9f093e9a07..cdd3b69920 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -2,6 +2,7 @@ //! trait `WasmRuntime`. use cretonne::ir::{self, InstBuilder}; use cretonne::cursor::FuncCursor; +use cretonne::settings::Flags; use translation_utils::{SignatureIndex, FunctionIndex, TableIndex, GlobalIndex, MemoryIndex, Global, Table, Memory}; @@ -26,10 +27,19 @@ pub enum GlobalValue { /// IL. The function environment provides information about the WebAssembly module as well as the /// runtime environment. pub trait FuncEnvironment { + /// Get the flags for the current compilation. + fn flags(&self) -> &Flags; + /// Get the Cretonne integer type to use for native pointers. /// - /// This should be `I64` for 64-bit architectures and `I32` for 32-bit architectures. - fn native_pointer(&self) -> ir::Type; + /// This returns `I64` for 64-bit architectures and `I32` for 32-bit architectures. + fn native_pointer(&self) -> ir::Type { + if self.flags().is_64bit() { + ir::types::I64 + } else { + ir::types::I32 + } + } /// Set up the necessary preamble definitions in `func` to access the global variable /// identified by `index`. @@ -138,7 +148,7 @@ pub trait FuncEnvironment { /// An object satisfyng the `WasmRuntime` trait can be passed as argument to the /// [`translate_module`](fn.translate_module.html) function. These methods should not be called -/// by the user, they are only for the `wasm2cretonne` internal use. +/// by the user, they are only for `cretonne-wasm` internal use. pub trait WasmRuntime: FuncEnvironment { /// Declares a function signature to the runtime. fn declare_signature(&mut self, sig: &ir::Signature); diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/testsuite.rs index 9e59b2bab7..7234541b66 100644 --- a/lib/wasm/tests/testsuite.rs +++ b/lib/wasm/tests/testsuite.rs @@ -1,18 +1,24 @@ extern crate cton_wasm; extern crate cretonne; +extern crate tempdir; use cton_wasm::{translate_module, DummyRuntime, WasmRuntime}; use std::path::PathBuf; +use std::borrow::Borrow; use std::fs::File; use std::error::Error; use std::io; +use std::str; use std::io::BufReader; use std::io::prelude::*; +use std::process::Command; use std::fs; use cretonne::ir; use cretonne::ir::entities::AnyEntity; -use cretonne::isa::TargetIsa; +use cretonne::isa::{self, TargetIsa}; +use cretonne::settings::{self, Configurable}; use cretonne::verifier; +use tempdir::TempDir; #[test] fn testsuite() { @@ -23,13 +29,29 @@ fn testsuite() { paths.sort_by_key(|dir| dir.path()); for path in paths { let path = path.path(); - match handle_module(path) { - Ok(()) => (), - Err(message) => println!("{}", message), - }; + handle_module(path, None); } } +#[test] +fn return_at_end() { + let mut flag_builder = settings::builder(); + flag_builder.enable("return_at_end").unwrap(); + let flags = settings::Flags::new(&flag_builder); + // We don't care about the target itself here, so just pick one arbitrarily. + let isa = match isa::lookup("riscv") { + Err(_) => { + println!("riscv target not found; disabled test return_at_end.wat"); + return; + } + Ok(isa_builder) => isa_builder.finish(flags), + }; + handle_module( + PathBuf::from("../../wasmtests/return_at_end.wat"), + Some(isa.borrow()), + ); +} + fn read_wasm_file(path: PathBuf) -> Result, io::Error> { let mut buf: Vec = Vec::new(); let file = File::open(path)?; @@ -38,44 +60,80 @@ fn read_wasm_file(path: PathBuf) -> Result, io::Error> { Ok(buf) } -fn handle_module(path: PathBuf) -> Result<(), String> { +fn handle_module(path: PathBuf, isa: Option<&TargetIsa>) { let data = match path.extension() { None => { - return Err(String::from("the file extension is not wasm or wast")); + panic!("the file extension is not wasm or wat"); } Some(ext) => { match ext.to_str() { Some("wasm") => { match read_wasm_file(path.clone()) { + Ok(data) => data, + Err(err) => panic!("error reading wasm file: {}", err.description()), + } + } + Some("wat") => { + let tmp_dir = TempDir::new("cretonne-wasm").unwrap(); + let file_path = tmp_dir.path().join("module.wasm"); + File::create(file_path.clone()).unwrap(); + let result_output = Command::new("wat2wasm") + .arg(path.clone()) + .arg("-o") + .arg(file_path.to_str().unwrap()) + .output(); + match result_output { + Err(e) => { + if e.kind() == io::ErrorKind::NotFound { + println!( + "wat2wasm not found; disabled test {}", + path.to_str().unwrap() + ); + return; + } + panic!("error convering wat file: {}", e.description()); + } + Ok(output) => { + if !output.status.success() { + panic!( + "error running wat2wasm: {}", + str::from_utf8(&output.stderr).expect( + "wat2wasm's error message should be valid UTF-8", + ) + ); + } + } + } + match read_wasm_file(file_path) { Ok(data) => data, Err(err) => { - return Err(String::from(err.description())); + panic!("error reading converted wasm file: {}", err.description()); } } } - None | Some(&_) => { - return Err(String::from("the file extension is not wasm or wast")); - } + None | Some(&_) => panic!("the file extension is not wasm or wat"), } } }; - let mut dummy_runtime = DummyRuntime::new(); + let mut dummy_runtime = match isa { + Some(isa) => DummyRuntime::with_flags(isa.flags().clone()), + None => DummyRuntime::default(), + }; let translation = { let runtime: &mut WasmRuntime = &mut dummy_runtime; match translate_module(&data, runtime) { Ok(x) => x, Err(string) => { - return Err(string); + panic!(string); } } }; for func in &translation.functions { - match verifier::verify_function(func, None) { + match verifier::verify_function(func, isa) { Ok(()) => (), - Err(err) => return Err(pretty_verifier_error(func, None, err)), + Err(err) => panic!(pretty_verifier_error(func, isa, err)), } } - Ok(()) } From ddf967962f964567a1e810a2c3e0d3a089f221fe Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Sep 2017 13:27:58 -0700 Subject: [PATCH 1093/3084] Use --all in some scripts. (#154) This avoids the need to list all the packages in test-all.sh. --- cranelift/format-all.sh | 6 +----- cranelift/test-all.sh | 9 ++------- 2 files changed, 3 insertions(+), 12 deletions(-) diff --git a/cranelift/format-all.sh b/cranelift/format-all.sh index d9e97f1026..6ccac9c289 100755 --- a/cranelift/format-all.sh +++ b/cranelift/format-all.sh @@ -6,12 +6,8 @@ set -e cd $(dirname "$0") -src=$(pwd) # Make sure we can find rustfmt. export PATH="$PATH:$HOME/.cargo/bin" -for crate in $(find "$src" -name Cargo.toml); do - cd $(dirname "$crate") - cargo fmt -- "$@" -done +exec cargo fmt --all -- "$@" diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 41b2b96f50..a5b2e92066 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -40,14 +40,9 @@ if [ -n "$needcheck" ]; then touch $tsfile || echo no target directory fi -PKGS="cretonne cretonne-reader cretonne-tools cretonne-frontend cretonne-wasm \ - filecheck " cd "$topdir" -for PKG in $PKGS -do - banner "Rust $PKG unit tests" - cargo test -p $PKG -done +banner "Rust unit tests" +cargo test --all # Build cton-util for parser testing. cd "$topdir" From c7b1bb5f9e19ae02abf09575c190a736eb865d65 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Sep 2017 12:51:34 -0700 Subject: [PATCH 1094/3084] Simplify using `map_err` and `expect`. --- cranelift/src/wasm.rs | 91 +++++++++-------------------- lib/wasm/src/sections_translator.rs | 16 ++--- lib/wasm/tests/testsuite.rs | 28 ++------- 3 files changed, 40 insertions(+), 95 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 4dddddf65c..e0b4d17772 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -67,17 +67,14 @@ pub fn run( for filename in files { let path = Path::new(&filename); let name = String::from(path.as_os_str().to_string_lossy()); - match handle_module( + handle_module( flag_verbose, flag_optimize, flag_check, path.to_path_buf(), name, &flags, - ) { - Ok(()) => {} - Err(message) => return Err(message), - } + )?; } Ok(()) } @@ -105,12 +102,9 @@ fn handle_module( Some(ext) => { match ext.to_str() { Some("wasm") => { - match read_wasm_file(path.clone()) { - Ok(data) => data, - Err(err) => { - return Err(String::from(err.description())); - } - } + read_wasm_file(path.clone()).map_err(|err| { + String::from(err.description()) + })? } Some("wat") => { let tmp_dir = TempDir::new("cretonne-wasm").unwrap(); @@ -127,12 +121,9 @@ fn handle_module( return Err(String::from(e.description())); }) .unwrap(); - match read_wasm_file(file_path) { - Ok(data) => data, - Err(err) => { - return Err(String::from(err.description())); - } - } + read_wasm_file(file_path).map_err(|err| { + String::from(err.description()) + })? } None | Some(&_) => { return Err(String::from("the file extension is not wasm or wat")); @@ -143,12 +134,7 @@ fn handle_module( let mut dummy_runtime = DummyRuntime::with_flags(flags.clone()); let translation = { let runtime: &mut WasmRuntime = &mut dummy_runtime; - match translate_module(&data, runtime) { - Ok(x) => x, - Err(string) => { - return Err(string); - } - } + translate_module(&data, runtime)? }; terminal.fg(term::color::GREEN).unwrap(); vprintln!(flag_verbose, " ok"); @@ -158,10 +144,9 @@ fn handle_module( vprint!(flag_verbose, "Checking... "); terminal.reset().unwrap(); for func in &translation.functions { - match verifier::verify_function(func, None) { - Ok(()) => (), - Err(err) => return Err(pretty_verifier_error(func, None, err)), - } + verifier::verify_function(func, None).map_err(|err| { + pretty_verifier_error(func, None, err) + })?; } terminal.fg(term::color::GREEN).unwrap(); vprintln!(flag_verbose, " ok"); @@ -184,42 +169,22 @@ fn handle_module( context.cfg = cfg; context.domtree = domtree; context.loop_analysis = loop_analysis; - match verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) { - Ok(()) => (), - Err(err) => { - return Err(pretty_verifier_error(&context.func, None, err)); - } - }; - match context.licm() { - Ok(())=> (), - Err(error) => { - match error { - CtonError::Verifier(err) => { - return Err(pretty_verifier_error(&context.func, None, err)); - } - CtonError::InvalidInput | - CtonError::ImplLimitExceeded | - CtonError::CodeTooLarge => return Err(String::from(error.description())), - } - } - }; - match context.simple_gvn() { - Ok(())=> (), - Err(error) => { - match error { - CtonError::Verifier(err) => { - return Err(pretty_verifier_error(&context.func, None, err)); - } - CtonError::InvalidInput | - CtonError::ImplLimitExceeded | - CtonError::CodeTooLarge => return Err(String::from(error.description())), - } - } - }; - match verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) { - Ok(()) => (), - Err(err) => return Err(pretty_verifier_error(&context.func, None, err)), - } + verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) + .map_err(|err| pretty_verifier_error(&context.func, None, err))?; + context.licm().map_err(|error| match error { + CtonError::Verifier(err) => pretty_verifier_error(&context.func, None, err), + CtonError::InvalidInput | + CtonError::ImplLimitExceeded | + CtonError::CodeTooLarge => String::from(error.description()), + })?; + context.simple_gvn().map_err(|error| match error { + CtonError::Verifier(err) => pretty_verifier_error(&context.func, None, err), + CtonError::InvalidInput | + CtonError::ImplLimitExceeded | + CtonError::CodeTooLarge => String::from(error.description()), + })?; + verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) + .map_err(|err| pretty_verifier_error(&context.func, None, err))?; } terminal.fg(term::color::GREEN).unwrap(); vprintln!(flag_verbose, " ok"); diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 3df3e7e3d9..08f05eca79 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -39,19 +39,15 @@ pub fn parse_function_signatures( }) => { let mut sig = Signature::new(CallConv::Native); sig.argument_types.extend(params.iter().map(|ty| { - let cret_arg: cretonne::ir::Type = match type_to_type(ty) { - Ok(ty) => ty, - Err(()) => panic!("only numeric types are supported in\ - function signatures"), - }; + let cret_arg: cretonne::ir::Type = type_to_type(ty).expect( + "only numeric types are supported in function signatures", + ); ArgumentType::new(cret_arg) })); sig.return_types.extend(returns.iter().map(|ty| { - let cret_arg: cretonne::ir::Type = match type_to_type(ty) { - Ok(ty) => ty, - Err(()) => panic!("only numeric types are supported in\ - function signatures"), - }; + let cret_arg: cretonne::ir::Type = type_to_type(ty).expect( + "only numeric types are supported in function signatures", + ); ArgumentType::new(cret_arg) })); runtime.declare_signature(&sig); diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/testsuite.rs index 7234541b66..96d1a645ef 100644 --- a/lib/wasm/tests/testsuite.rs +++ b/lib/wasm/tests/testsuite.rs @@ -67,12 +67,7 @@ fn handle_module(path: PathBuf, isa: Option<&TargetIsa>) { } Some(ext) => { match ext.to_str() { - Some("wasm") => { - match read_wasm_file(path.clone()) { - Ok(data) => data, - Err(err) => panic!("error reading wasm file: {}", err.description()), - } - } + Some("wasm") => read_wasm_file(path.clone()).expect("error reading wasm file"), Some("wat") => { let tmp_dir = TempDir::new("cretonne-wasm").unwrap(); let file_path = tmp_dir.path().join("module.wasm"); @@ -104,12 +99,7 @@ fn handle_module(path: PathBuf, isa: Option<&TargetIsa>) { } } } - match read_wasm_file(file_path) { - Ok(data) => data, - Err(err) => { - panic!("error reading converted wasm file: {}", err.description()); - } - } + read_wasm_file(file_path).expect("error reading converted wasm file") } None | Some(&_) => panic!("the file extension is not wasm or wat"), } @@ -121,18 +111,12 @@ fn handle_module(path: PathBuf, isa: Option<&TargetIsa>) { }; let translation = { let runtime: &mut WasmRuntime = &mut dummy_runtime; - match translate_module(&data, runtime) { - Ok(x) => x, - Err(string) => { - panic!(string); - } - } + translate_module(&data, runtime).unwrap() }; for func in &translation.functions { - match verifier::verify_function(func, isa) { - Ok(()) => (), - Err(err) => panic!(pretty_verifier_error(func, isa, err)), - } + verifier::verify_function(func, isa) + .map_err(|err| panic!(pretty_verifier_error(func, isa, err))) + .unwrap(); } } From 69f8174c03de322f7c39eb12cd778b1a18fa4453 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Sep 2017 13:47:17 -0700 Subject: [PATCH 1095/3084] Move `ensure_domtree` out of Context and into DominatorTree. This also moves the calls to it out of Context and into the passes that actually need it, so that Context's functions don't have any logic of their own. --- lib/cretonne/src/context.rs | 11 +---------- lib/cretonne/src/dominator_tree.rs | 8 ++++++++ lib/cretonne/src/licm.rs | 1 + lib/cretonne/src/regalloc/context.rs | 5 ++++- 4 files changed, 14 insertions(+), 11 deletions(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index ece812a5de..45914d8069 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -115,13 +115,6 @@ impl Context { self.domtree.compute(&self.func, &self.cfg); } - /// Ensure that a valid domtree exists. - pub fn ensure_domtree(&mut self) { - if !self.domtree.is_valid() { - self.domtree.compute(&self.func, &self.cfg); - } - } - /// Perform simple GVN on the function. pub fn simple_gvn(&mut self) -> CtonResult { do_simple_gvn(&mut self.func, &mut self.cfg); @@ -132,7 +125,6 @@ impl Context { /// Perform LICM on the function. pub fn licm(&mut self) -> CtonResult { - self.ensure_domtree(); do_licm( &mut self.func, &mut self.cfg, @@ -144,12 +136,11 @@ impl Context { /// Run the register allocator. pub fn regalloc(&mut self, isa: &TargetIsa) -> CtonResult { - self.ensure_domtree(); self.regalloc.run( isa, &mut self.func, &self.cfg, - &self.domtree, + &mut self.domtree, ) } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 2e9a6dc38d..18fd9e91b2 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -237,6 +237,14 @@ impl DominatorTree { !self.nodes.is_empty() } + /// Conveneince function to call `compute` if `compute` hasn't been called + /// since the last `clear()`. + pub fn ensure(&mut self, func: &Function, cfg: &ControlFlowGraph) { + if !self.is_valid() { + self.compute(func, cfg) + } + } + /// Reset all internal data structures and compute a post-order for `cfg`. /// /// This leaves `rpo_number == 1` for all reachable EBBs, 0 for unreachable ones. diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 20dfca9f5d..41aa498c5c 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -16,6 +16,7 @@ pub fn do_licm( domtree: &mut DominatorTree, loop_analysis: &mut LoopAnalysis, ) { + domtree.ensure(func, cfg); loop_analysis.compute(func, cfg, domtree); for lp in loop_analysis.loops() { // For each loop that we want to optimize we determine the set of loop-invariant diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 2e11536596..92be408919 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -58,8 +58,11 @@ impl Context { isa: &TargetIsa, func: &mut Function, cfg: &ControlFlowGraph, - domtree: &DominatorTree, + domtree: &mut DominatorTree, ) -> CtonResult { + // Ensure that a valid domtree exists. + domtree.ensure(func, cfg); + // `Liveness` and `Coloring` are self-clearing. self.virtregs.clear(); From 9d0d6b5a1b909355c0d1906f11bfa6bae49482e2 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Sep 2017 15:38:30 -0700 Subject: [PATCH 1096/3084] Move `verify` calls out of Context. Also, move flowgraph() calls out of filetest and into the passes that need them so that filetest doesn't have embedded knowledge of these dependencies. This resolves a TODO about the way Context was running the verifier, and it makes the Context functions and the filetest runners more transparent. This also fixes simple_gvn to use the existing dominator tree rather than computing its own. --- cranelift/src/filetest/licm.rs | 6 +++--- cranelift/src/filetest/simple_gvn.rs | 6 +++--- cranelift/src/wasm.rs | 17 ++++------------- lib/cretonne/src/context.rs | 12 ++++-------- lib/cretonne/src/dominator_tree.rs | 3 +-- lib/cretonne/src/flowgraph.rs | 17 +++++++++++++++++ lib/cretonne/src/licm.rs | 1 + lib/cretonne/src/simple_gvn.rs | 7 ++++--- 8 files changed, 37 insertions(+), 32 deletions(-) diff --git a/cranelift/src/filetest/licm.rs b/cranelift/src/filetest/licm.rs index db817530a9..79b785f8c3 100644 --- a/cranelift/src/filetest/licm.rs +++ b/cranelift/src/filetest/licm.rs @@ -38,9 +38,9 @@ impl SubTest for TestLICM { let mut comp_ctx = cretonne::Context::new(); comp_ctx.func = func.into_owned(); - comp_ctx.flowgraph(); - comp_ctx.licm().map_err(|e| { - pretty_error(&comp_ctx.func, context.isa, e) + comp_ctx.licm(); + comp_ctx.verify(context.isa).map_err(|e| { + pretty_error(&comp_ctx.func, context.isa, Into::into(e)) })?; let mut text = String::new(); diff --git a/cranelift/src/filetest/simple_gvn.rs b/cranelift/src/filetest/simple_gvn.rs index e8eb377b5e..1b04cd67b9 100644 --- a/cranelift/src/filetest/simple_gvn.rs +++ b/cranelift/src/filetest/simple_gvn.rs @@ -38,9 +38,9 @@ impl SubTest for TestSimpleGVN { let mut comp_ctx = cretonne::Context::new(); comp_ctx.func = func.into_owned(); - comp_ctx.flowgraph(); - comp_ctx.simple_gvn().map_err(|e| { - pretty_error(&comp_ctx.func, context.isa, e) + comp_ctx.simple_gvn(); + comp_ctx.verify(context.isa).map_err(|e| { + pretty_error(&comp_ctx.func, context.isa, Into::into(e)) })?; let mut text = String::new(); diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index e0b4d17772..350a15d51a 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -10,7 +10,6 @@ use cretonne::loop_analysis::LoopAnalysis; use cretonne::flowgraph::ControlFlowGraph; use cretonne::dominator_tree::DominatorTree; use cretonne::Context; -use cretonne::result::CtonError; use cretonne::verifier; use cretonne::settings::{self, Configurable}; use std::fs::File; @@ -171,18 +170,10 @@ fn handle_module( context.loop_analysis = loop_analysis; verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) .map_err(|err| pretty_verifier_error(&context.func, None, err))?; - context.licm().map_err(|error| match error { - CtonError::Verifier(err) => pretty_verifier_error(&context.func, None, err), - CtonError::InvalidInput | - CtonError::ImplLimitExceeded | - CtonError::CodeTooLarge => String::from(error.description()), - })?; - context.simple_gvn().map_err(|error| match error { - CtonError::Verifier(err) => pretty_verifier_error(&context.func, None, err), - CtonError::InvalidInput | - CtonError::ImplLimitExceeded | - CtonError::CodeTooLarge => String::from(error.description()), - })?; + context.licm(); + verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) + .map_err(|err| pretty_verifier_error(&context.func, None, err))?; + context.simple_gvn(); verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) .map_err(|err| pretty_verifier_error(&context.func, None, err))?; } diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 45914d8069..c7347a1079 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -116,22 +116,18 @@ impl Context { } /// Perform simple GVN on the function. - pub fn simple_gvn(&mut self) -> CtonResult { - do_simple_gvn(&mut self.func, &mut self.cfg); - // TODO: Factor things such that we can get a Flags and test - // enable_verifier(). - self.verify(None).map_err(Into::into) + pub fn simple_gvn(&mut self) { + do_simple_gvn(&mut self.func, &mut self.cfg, &mut self.domtree) } /// Perform LICM on the function. - pub fn licm(&mut self) -> CtonResult { + pub fn licm(&mut self) { do_licm( &mut self.func, &mut self.cfg, &mut self.domtree, &mut self.loop_analysis, - ); - self.verify(None).map_err(Into::into) + ) } /// Run the register allocator. diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 18fd9e91b2..143df57db6 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -231,8 +231,7 @@ impl DominatorTree { /// /// Note that this doesn't perform any kind of validity checks. It simply checks if the /// `compute()` method has been called since the last `clear()`. It does not check that the - /// dominator tree is consistent - /// with the CFG> + /// dominator tree is consistent with the CFG. pub fn is_valid(&self) -> bool { !self.nodes.is_empty() } diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index 504d90fb9d..1517ae5ee3 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -131,6 +131,23 @@ impl ControlFlowGraph { pub fn get_successors(&self, ebb: Ebb) -> &[Ebb] { &self.data[ebb].successors } + + /// Check if the CFG is in a valid state. + /// + /// Note that this doesn't perform any kind of validity checks. It simply checks if the + /// `compute()` method has been called since the last `clear()`. It does not check that the + /// CFG is consistent with the function. + pub fn is_valid(&self) -> bool { + self.entry_block.is_some() + } + + /// Conveneince function to call `compute` if `compute` hasn't been called + /// since the last `clear()`. + pub fn ensure(&mut self, func: &Function) { + if !self.is_valid() { + self.compute(func) + } + } } #[cfg(test)] diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 41aa498c5c..225cd4d524 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -16,6 +16,7 @@ pub fn do_licm( domtree: &mut DominatorTree, loop_analysis: &mut LoopAnalysis, ) { + cfg.ensure(func); domtree.ensure(func, cfg); loop_analysis.compute(func, cfg, domtree); for lp in loop_analysis.loops() { diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index 308d23e083..d3d8391991 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -13,10 +13,11 @@ fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { /// Perform simple GVN on `func`. /// -pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph) { - let mut visible_values: HashMap<(InstructionData, Type), Inst> = HashMap::new(); +pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph, domtree: &mut DominatorTree) { + cfg.ensure(func); + domtree.ensure(func, cfg); - let domtree = DominatorTree::with_function(func, cfg); + let mut visible_values: HashMap<(InstructionData, Type), Inst> = HashMap::new(); // Visit EBBs in a reverse post-order. let mut pos = Cursor::new(&mut func.layout); From e2f0fe58d128662a66624833619bd4dbee16b874 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Sep 2017 16:15:22 -0700 Subject: [PATCH 1097/3084] Move logic out of `cton-util wasm`. Give LoopAnalysis `is_valid` and `ensure` functions similar to DominatorTree and others, so that it can be computed on demand in the same way. This removes the last need for src/wasm.rs to have embedded knowledge of the dependencies of the passes it's running. --- cranelift/src/wasm.rs | 15 +-------------- lib/cretonne/src/licm.rs | 2 +- lib/cretonne/src/loop_analysis.rs | 22 +++++++++++++++++++++- 3 files changed, 23 insertions(+), 16 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 350a15d51a..4f48ae6d70 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -6,9 +6,6 @@ use cton_wasm::{translate_module, DummyRuntime, WasmRuntime}; use std::path::PathBuf; -use cretonne::loop_analysis::LoopAnalysis; -use cretonne::flowgraph::ControlFlowGraph; -use cretonne::dominator_tree::DominatorTree; use cretonne::Context; use cretonne::verifier; use cretonne::settings::{self, Configurable}; @@ -156,18 +153,8 @@ fn handle_module( vprint!(flag_verbose, "Optimizing... "); terminal.reset().unwrap(); for func in &translation.functions { - let mut il = func.clone(); - let mut loop_analysis = LoopAnalysis::new(); - let mut cfg = ControlFlowGraph::new(); - cfg.compute(&il); - let mut domtree = DominatorTree::new(); - domtree.compute(&mut il, &cfg); - loop_analysis.compute(&mut il, &mut cfg, &mut domtree); let mut context = Context::new(); - context.func = il; - context.cfg = cfg; - context.domtree = domtree; - context.loop_analysis = loop_analysis; + context.func = func.clone(); verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) .map_err(|err| pretty_verifier_error(&context.func, None, err))?; context.licm(); diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 225cd4d524..107f3f7766 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -18,7 +18,7 @@ pub fn do_licm( ) { cfg.ensure(func); domtree.ensure(func, cfg); - loop_analysis.compute(func, cfg, domtree); + loop_analysis.ensure(func, cfg, domtree); for lp in loop_analysis.loops() { // For each loop that we want to optimize we determine the set of loop-invariant // instructions diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index 4c87e773a7..9d5e466c16 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -18,6 +18,7 @@ entity_impl!(Loop, "loop"); /// Loops are referenced by the Loop object, and for each loop you can access its header EBB, /// its eventual parent in the loop tree and all the EBB belonging to the loop. pub struct LoopAnalysis { + valid: bool, loops: PrimaryMap, ebb_loop_map: EntityMap>, } @@ -43,6 +44,7 @@ impl LoopAnalysis { /// a function. pub fn new() -> LoopAnalysis { LoopAnalysis { + valid: false, loops: PrimaryMap::new(), ebb_loop_map: EntityMap::new(), } @@ -100,7 +102,25 @@ impl LoopAnalysis { self.ebb_loop_map.clear(); self.ebb_loop_map.resize(func.dfg.num_ebbs()); self.find_loop_headers(cfg, domtree, &func.layout); - self.discover_loop_blocks(cfg, domtree, &func.layout) + self.discover_loop_blocks(cfg, domtree, &func.layout); + self.valid = true; + } + + /// Check if the loop analysis is in a valid state. + /// + /// Note that this doesn't perform any kind of validity checks. It simply checks if the + /// `compute()` method has been called since the last `clear()`. It does not check that the + /// loop analysis is consistent with the CFG. + pub fn is_valid(&self) -> bool { + self.valid + } + + /// Conveneince function to call `compute` if `compute` hasn't been called + /// since the last `clear()`. + pub fn ensure(&mut self, func: &Function, cfg: &ControlFlowGraph, domtree: &DominatorTree) { + if !self.is_valid() { + self.compute(func, cfg, domtree) + } } // Traverses the CFG in reverse postorder and create a loop object for every EBB having a From c7bd842ee39ce9d13f313087791a5d706ed6fdf0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Sep 2017 16:25:15 -0700 Subject: [PATCH 1098/3084] Refactor code to use Context::verify instead of doing it manually. --- cranelift/src/wasm.rs | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 4f48ae6d70..6b22d66e31 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -155,14 +155,17 @@ fn handle_module( for func in &translation.functions { let mut context = Context::new(); context.func = func.clone(); - verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) - .map_err(|err| pretty_verifier_error(&context.func, None, err))?; + context.verify(None).map_err(|err| { + pretty_verifier_error(&context.func, None, err) + })?; context.licm(); - verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) - .map_err(|err| pretty_verifier_error(&context.func, None, err))?; + context.verify(None).map_err(|err| { + pretty_verifier_error(&context.func, None, err) + })?; context.simple_gvn(); - verifier::verify_context(&context.func, &context.cfg, &context.domtree, None) - .map_err(|err| pretty_verifier_error(&context.func, None, err))?; + context.verify(None).map_err(|err| { + pretty_verifier_error(&context.func, None, err) + })?; } terminal.fg(term::color::GREEN).unwrap(); vprintln!(flag_verbose, " ok"); From f5639a66e67c8900247c4c48ad9c7c31916e9ffd Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Sep 2017 16:35:45 -0700 Subject: [PATCH 1099/3084] Flush stdout before writing to stderr. In particular, terminal.reset() doesn't flush, so it ensures that stderr isn't unintentionally printed in the color of the preceding line. --- cranelift/src/cton-util.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index a7ad8a44b4..e702f57ae4 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -98,6 +98,7 @@ fn main() { if !msg.ends_with('\n') { msg.push('\n'); } + io::stdout().flush().expect("flushing stdout"); io::stderr().write_all(msg.as_bytes()).unwrap(); process::exit(1); } From 5845f56cda5096ee0bb43b128d6f94bd3ff85192 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 12 Sep 2017 13:32:04 -0700 Subject: [PATCH 1100/3084] Add x86-64 encodings for call instructions. --- cranelift/filetests/isa/intel/binary64.cton | 10 ++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 5 +++++ 2 files changed, 15 insertions(+) diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index fb6f62d928..47f3e72a3c 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -418,6 +418,16 @@ ebb0: ; asm: movzbq %dl, %rsi [-,%rsi] v351 = bint.i64 v301 ; bin: 48 0f b6 f2 + ; asm: call foo + call fn0() ; bin: e8 PCRel4(fn0) 00000000 + + ; asm: call *%rcx + call_indirect sig0, v1() ; bin: 40 ff d1 + ; asm: call *%rsi + call_indirect sig0, v2() ; bin: 40 ff d6 + ; asm: call *%r10 + call_indirect sig0, v3() ; bin: 41 ff d2 + ; asm: testq %rcx, %rcx ; asm: je ebb1 brz v1, ebb1 ; bin: 48 85 c9 74 1b diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 3b2a91a1f2..ed0bbb433d 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -225,7 +225,12 @@ enc_i32_i64_ld_st(base.sload8, True, r.ldDisp32, 0x0f, 0xbe) # Call/return # I32.enc(base.call, *r.call_id(0xe8)) +I64.enc(base.call, *r.call_id(0xe8)) + I32.enc(base.call_indirect.i32, *r.call_r(0xff, rrr=2)) +I64.enc(base.call_indirect.i64, *r.call_r.rex(0xff, rrr=2)) +I64.enc(base.call_indirect.i64, *r.call_r(0xff, rrr=2)) + I32.enc(base.x_return, *r.ret(0xc3)) I64.enc(base.x_return, *r.ret(0xc3)) From ef27c3daf0d2077e9ce664f32ea9b974e3dab969 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 13 Sep 2017 09:34:05 -0700 Subject: [PATCH 1101/3084] Add an AsRef implementation to FuncName so we can read its contents. --- lib/cretonne/src/ir/funcname.rs | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/cretonne/src/ir/funcname.rs b/lib/cretonne/src/ir/funcname.rs index 2081f519e5..142a846c22 100644 --- a/lib/cretonne/src/ir/funcname.rs +++ b/lib/cretonne/src/ir/funcname.rs @@ -82,6 +82,12 @@ impl AsRef<[u8]> for NameRepr { } } +impl AsRef<[u8]> for FunctionName { + fn as_ref(&self) -> &[u8] { + self.0.as_ref() + } +} + impl Default for NameRepr { fn default() -> Self { NameRepr::Short { From eb42a2547eb9877e4ae020aef1d548c5408d0438 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 13 Sep 2017 11:35:33 -0700 Subject: [PATCH 1102/3084] Add helper routines for special-purpose arguments. - ArgumentType::special() creates a new special-purpose argument without assigning it to a register location. - Signature::special_arg_index() funds a unique special-purpose argument. - Function::special_arg() finds a special-purpose argument by value. Also add a new "sigid" argument purpose which will be used for runtime signature checks in WebAssembly indirect calls. --- lib/cretonne/src/ir/extfunc.rs | 26 +++++++++++++++++++++++++- lib/cretonne/src/ir/function.rs | 10 ++++++++++ lib/cretonne/src/legalizer/boundary.rs | 9 +++++++++ 3 files changed, 44 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 353d49256e..5983182d87 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -76,6 +76,13 @@ impl Signature { pub fn display<'a, R: Into>>(&'a self, regs: R) -> DisplaySignature<'a> { DisplaySignature(self, regs.into()) } + + /// Find the index of a presumed unique special-purpose argument. + pub fn special_arg_index(&self, purpose: ArgumentPurpose) -> Option { + self.argument_types.iter().rposition( + |arg| arg.purpose == purpose, + ) + } } /// Wrapper type capable of displaying a `Signature` with correct register names. @@ -146,6 +153,16 @@ impl ArgumentType { } } + /// Create a special-purpose argument type that is not (yet) bound to a specific register. + pub fn special(vt: Type, purpose: ArgumentPurpose) -> ArgumentType { + ArgumentType { + value_type: vt, + extension: ArgumentExtension::None, + purpose, + location: Default::default(), + } + } + /// Create an argument type for a special-purpose register. pub fn special_reg(vt: Type, purpose: ArgumentPurpose, regunit: RegUnit) -> ArgumentType { ArgumentType { @@ -255,10 +272,16 @@ pub enum ArgumentPurpose { /// This is a pointer to a context struct containing details about the current sandbox. It is /// used as a base pointer for `vmctx` global variables. VMContext, + + /// A signature identifier. + /// + /// This is a special-purpose argument used to identify the calling convention expected by the + /// caller in an indirect call. The callee can verify that the expected signature ID matches. + SignatureId, } /// Text format names of the `ArgumentPurpose` variants. -static PURPOSE_NAMES: [&str; 6] = ["normal", "sret", "link", "fp", "csr", "vmctx"]; +static PURPOSE_NAMES: [&str; 7] = ["normal", "sret", "link", "fp", "csr", "vmctx", "sigid"]; impl fmt::Display for ArgumentPurpose { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { @@ -276,6 +299,7 @@ impl FromStr for ArgumentPurpose { "fp" => Ok(ArgumentPurpose::FramePointer), "csr" => Ok(ArgumentPurpose::CalleeSaved), "vmctx" => Ok(ArgumentPurpose::VMContext), + "sigid" => Ok(ArgumentPurpose::SignatureId), _ => Err(()), } } diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 0c86ac66e0..f7ba244ae0 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -135,6 +135,16 @@ impl Function { pub fn display<'a, I: Into>>(&'a self, isa: I) -> DisplayFunction<'a> { DisplayFunction(self, isa.into()) } + + /// Find a presumed unique special-purpose function argument value. + /// + /// Returns the value of the last `purpose` argument, or `None` if no such argument exists. + pub fn special_arg(&self, purpose: ir::ArgumentPurpose) -> Option { + let entry = self.layout.entry_block().expect("Function is empty"); + self.signature.special_arg_index(purpose).map(|i| { + self.dfg.ebb_args(entry)[i] + }) + } } /// Wrapper type capable of displaying a `Function` with correct ISA annotations. diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index a5442ba8be..2e16627efe 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -57,6 +57,7 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { let mut has_sret = false; let mut has_link = false; let mut has_vmctx = false; + let mut has_sigid = false; // Insert position for argument conversion code. // We want to insert instructions before the first instruction in the entry block. @@ -91,6 +92,10 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { assert!(!has_vmctx, "Multiple vmctx arguments found"); has_vmctx = true; } + ArgumentPurpose::SignatureId => { + assert!(!has_sigid, "Multiple sigid arguments found"); + has_sigid = true; + } _ => panic!("Unexpected special-purpose arg {}", abi_types[abi_arg]), } abi_arg += 1; @@ -145,6 +150,10 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { assert!(!has_vmctx, "Multiple vmctx arguments found"); has_vmctx = true; } + ArgumentPurpose::SignatureId => { + assert!(!has_sigid, "Multiple sigid arguments found"); + has_sigid = true; + } } // Just create entry block values to match here. We will use them in `handle_return_abi()` From 45f50120ef631ea2a2d716697190551932b87f16 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 13 Sep 2017 12:04:14 -0700 Subject: [PATCH 1103/3084] Handle special-purpose arguments in the Intel ABI. The VMContext and SignatureId arguments are passed in fixed registers for the spiderwasm calling convention. --- lib/cretonne/src/isa/intel/abi.rs | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 707d42b8d9..7a051905d4 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -6,7 +6,7 @@ use regalloc::AllocatableSet; use settings as shared_settings; use super::registers::{GPR, FPR, RU}; use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; -use ir::{ArgumentType, ArgumentLoc, ArgumentExtension}; +use ir::{ArgumentType, ArgumentPurpose, ArgumentLoc, ArgumentExtension}; /// Argument registers for x86-64 static ARG_GPRS: [RU; 6] = [RU::rdi, RU::rsi, RU::rdx, RU::rcx, RU::r8, RU::r9]; @@ -64,6 +64,24 @@ impl ArgAssigner for Args { } } + // Handle special-purpose arguments. + // TODO: The registers below are for `spiderwasm`. Should we check the calling convention? + if ty.is_int() { + match arg.purpose { + // This is SpiderMonkey's `WasmTlsReg`. + ArgumentPurpose::VMContext => { + return ArgumentLoc::Reg(if self.pointer_bits == 64 { + RU::r14 + } else { + RU::rsi + } as RegUnit).into() + } + // This is SpiderMonkey's `WasmTableCallSigReg`. + ArgumentPurpose::SignatureId => return ArgumentLoc::Reg(RU::rbx as RegUnit).into(), + _ => {} + } + } + // Try to use a GPR. if !ty.is_float() && self.gpr_used < self.gpr.len() { let reg = self.gpr[self.gpr_used] as RegUnit; From 78c39ce078dfd84b8fc583ea160c693fe3cb244a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 14 Sep 2017 09:26:37 -0700 Subject: [PATCH 1104/3084] Don't silently ignore argument or value locations. (#157) No current tests actually rely on this, and it's surprising. --- lib/reader/src/parser.rs | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 2169ded9c9..e5c59530fc 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1010,9 +1010,7 @@ impl<'a> Parser<'a> { .map(ArgumentLoc::Reg) .ok_or(self.error("invalid register name")) } else { - // We are unable to parse the register without a TargetISA, so we quietly - // ignore it. - Ok(ArgumentLoc::Unassigned) + err!(self.loc, "argument location requires exactly one isa") } } Some(Token::Integer(_)) => { @@ -1491,8 +1489,7 @@ impl<'a> Parser<'a> { .map(ValueLoc::Reg) .ok_or(self.error("invalid register value location")) } else { - // For convenience we ignore value locations when no unique ISA is specified - Ok(ValueLoc::Unassigned) + err!(self.loc, "value location requires exactly one isa") } } Some(Token::Minus) => { From 39992014e0bf1afba8df5c6c57d4b0d8a7916f7b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 14 Sep 2017 12:25:59 -0700 Subject: [PATCH 1105/3084] Make FuncEnvironment callbacks take an &mut self parameter. The WasmRuntime trait already does this, and nothing in the function translator requires the environment trait object to be non-mutable. --- lib/wasm/src/runtime/dummy.rs | 14 +++++++------- lib/wasm/src/runtime/spec.rs | 16 ++++++++-------- lib/wasm/src/state.rs | 8 ++++---- 3 files changed, 19 insertions(+), 19 deletions(-) diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index b6d849d751..de630d8810 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -47,7 +47,7 @@ impl FuncEnvironment for DummyRuntime { &self.flags } - fn make_global(&self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue { + fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue { // Just create a dummy `vmctx` global. let offset = ((index * 8) as i32 + 8).into(); let gv = func.create_global_var(ir::GlobalVarData::VmCtx { offset }); @@ -57,7 +57,7 @@ impl FuncEnvironment for DummyRuntime { } } - fn make_heap(&self, func: &mut ir::Function, _index: MemoryIndex) -> ir::Heap { + fn make_heap(&mut self, func: &mut ir::Function, _index: MemoryIndex) -> ir::Heap { func.create_heap(ir::HeapData { base: ir::HeapBase::ReservedReg, min_size: 0.into(), @@ -66,13 +66,13 @@ impl FuncEnvironment for DummyRuntime { }) } - fn make_indirect_sig(&self, func: &mut ir::Function, index: SignatureIndex) -> ir::SigRef { + fn make_indirect_sig(&mut self, func: &mut ir::Function, index: SignatureIndex) -> ir::SigRef { // A real implementation would probably change the calling convention and add `vmctx` and // signature index arguments. func.import_signature(self.signatures[index].clone()) } - fn make_direct_func(&self, func: &mut ir::Function, index: FunctionIndex) -> ir::FuncRef { + fn make_direct_func(&mut self, func: &mut ir::Function, index: FunctionIndex) -> ir::FuncRef { let sigidx = self.func_types[index]; // A real implementation would probably add a `vmctx` argument. // And maybe attempt some signature de-duplication. @@ -87,7 +87,7 @@ impl FuncEnvironment for DummyRuntime { } fn translate_call_indirect( - &self, + &mut self, mut pos: FuncCursor, _table_index: TableIndex, _sig_index: SignatureIndex, @@ -99,7 +99,7 @@ impl FuncEnvironment for DummyRuntime { } fn translate_grow_memory( - &self, + &mut self, mut pos: FuncCursor, _index: MemoryIndex, _heap: ir::Heap, @@ -109,7 +109,7 @@ impl FuncEnvironment for DummyRuntime { } fn translate_current_memory( - &self, + &mut self, mut pos: FuncCursor, _index: MemoryIndex, _heap: ir::Heap, diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index cdd3b69920..503e5079ef 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -48,13 +48,13 @@ pub trait FuncEnvironment { /// /// Return the global variable reference that should be used to access the global and the /// WebAssembly type of the global. - fn make_global(&self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue; + fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue; /// Set up the necessary preamble definitions in `func` to access the linear memory identified /// by `index`. /// /// The index space covers both imported and locally declared memories. - fn make_heap(&self, func: &mut ir::Function, index: MemoryIndex) -> ir::Heap; + fn make_heap(&mut self, func: &mut ir::Function, index: MemoryIndex) -> ir::Heap; /// Set up a signature definition in the preamble of `func` that can be used for an indirect /// call with signature `index`. @@ -65,7 +65,7 @@ pub trait FuncEnvironment { /// /// The signature will only be used for indirect calls, even if the module has direct function /// calls with the same WebAssembly type. - fn make_indirect_sig(&self, func: &mut ir::Function, index: SignatureIndex) -> ir::SigRef; + fn make_indirect_sig(&mut self, func: &mut ir::Function, index: SignatureIndex) -> ir::SigRef; /// Set up an external function definition in the preamble of `func` that can be used to /// directly call the function `index`. @@ -78,7 +78,7 @@ pub trait FuncEnvironment { /// /// The function's signature will only be used for direct calls, even if the module has /// indirect calls with the same WebAssembly type. - fn make_direct_func(&self, func: &mut ir::Function, index: FunctionIndex) -> ir::FuncRef; + fn make_direct_func(&mut self, func: &mut ir::Function, index: FunctionIndex) -> ir::FuncRef; /// Translate a `call_indirect` WebAssembly instruction at `pos`. /// @@ -90,7 +90,7 @@ pub trait FuncEnvironment { /// /// Return the call instruction whose results are the WebAssembly return values. fn translate_call_indirect( - &self, + &mut self, pos: FuncCursor, table_index: TableIndex, sig_index: SignatureIndex, @@ -107,7 +107,7 @@ pub trait FuncEnvironment { /// /// Return the call instruction whose results are the WebAssembly return values. fn translate_call( - &self, + &mut self, mut pos: FuncCursor, _callee_index: FunctionIndex, callee: ir::FuncRef, @@ -125,7 +125,7 @@ pub trait FuncEnvironment { /// /// Returns the old size (in pages) of the memory. fn translate_grow_memory( - &self, + &mut self, pos: FuncCursor, index: MemoryIndex, heap: ir::Heap, @@ -139,7 +139,7 @@ pub trait FuncEnvironment { /// /// Returns the size in pages of the memory. fn translate_current_memory( - &self, + &mut self, pos: FuncCursor, index: MemoryIndex, heap: ir::Heap, diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index cfe06e081a..8130b469af 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -266,7 +266,7 @@ impl TranslationState { &mut self, func: &mut ir::Function, index: u32, - environ: &FE, + environ: &mut FE, ) -> GlobalValue { let index = index as GlobalIndex; *self.globals.entry(index).or_insert_with( @@ -280,7 +280,7 @@ impl TranslationState { &mut self, func: &mut ir::Function, index: u32, - environ: &FE, + environ: &mut FE, ) -> ir::Heap { let index = index as MemoryIndex; *self.heaps.entry(index).or_insert_with( @@ -296,7 +296,7 @@ impl TranslationState { &mut self, func: &mut ir::Function, index: u32, - environ: &FE, + environ: &mut FE, ) -> (ir::SigRef, usize) { let index = index as SignatureIndex; *self.signatures.entry(index).or_insert_with(|| { @@ -313,7 +313,7 @@ impl TranslationState { &mut self, func: &mut ir::Function, index: u32, - environ: &FE, + environ: &mut FE, ) -> (ir::FuncRef, usize) { let index = index as FunctionIndex; *self.functions.entry(index).or_insert_with(|| { From bbe056bf9d5f15a4effde683b3d2816525616608 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 14 Sep 2017 14:38:53 -0700 Subject: [PATCH 1106/3084] Make passes assert their dependencies consistently. (#156) * Make passes assert their dependencies consistently. This avoids ambiguity about whose responsibility it is to run to compute cfg, domtree, and loop_analysis data. * Reset the `valid` flag in DominatorTree's `clear()`. * Remove the redundant assert from DominatorTree::with_function. * Remove the message strings from obvious asserts. This avoids having them spill out into multiple lines. * Refactor calls to `compute` on `Context` objects into helper functions. --- cranelift/src/filetest/legalizer.rs | 2 +- cranelift/src/filetest/licm.rs | 2 ++ cranelift/src/filetest/regalloc.rs | 3 ++- cranelift/src/filetest/simple_gvn.rs | 1 + cranelift/src/wasm.rs | 2 ++ lib/cretonne/src/context.rs | 32 +++++++++++++++++++++++----- lib/cretonne/src/dominator_tree.rs | 18 ++++++++-------- lib/cretonne/src/flowgraph.rs | 17 +++++++-------- lib/cretonne/src/legalizer/mod.rs | 2 ++ lib/cretonne/src/licm.rs | 7 +++--- lib/cretonne/src/loop_analysis.rs | 15 +++++++------ lib/cretonne/src/regalloc/context.rs | 3 +-- lib/cretonne/src/simple_gvn.rs | 4 ++-- lib/cretonne/src/verifier/mod.rs | 4 +++- lib/wasm/src/func_translator.rs | 3 --- 15 files changed, 72 insertions(+), 43 deletions(-) diff --git a/cranelift/src/filetest/legalizer.rs b/cranelift/src/filetest/legalizer.rs index 08155289ce..e79aada799 100644 --- a/cranelift/src/filetest/legalizer.rs +++ b/cranelift/src/filetest/legalizer.rs @@ -40,7 +40,7 @@ impl SubTest for TestLegalizer { comp_ctx.func = func.into_owned(); let isa = context.isa.expect("legalizer needs an ISA"); - comp_ctx.flowgraph(); + comp_ctx.compute_cfg(); comp_ctx.legalize(isa).map_err(|e| { pretty_error(&comp_ctx.func, context.isa, e) })?; diff --git a/cranelift/src/filetest/licm.rs b/cranelift/src/filetest/licm.rs index 79b785f8c3..5806776e84 100644 --- a/cranelift/src/filetest/licm.rs +++ b/cranelift/src/filetest/licm.rs @@ -38,6 +38,8 @@ impl SubTest for TestLICM { let mut comp_ctx = cretonne::Context::new(); comp_ctx.func = func.into_owned(); + comp_ctx.flowgraph(); + comp_ctx.compute_loop_analysis(); comp_ctx.licm(); comp_ctx.verify(context.isa).map_err(|e| { pretty_error(&comp_ctx.func, context.isa, Into::into(e)) diff --git a/cranelift/src/filetest/regalloc.rs b/cranelift/src/filetest/regalloc.rs index 3974854416..48160796a8 100644 --- a/cranelift/src/filetest/regalloc.rs +++ b/cranelift/src/filetest/regalloc.rs @@ -44,11 +44,12 @@ impl SubTest for TestRegalloc { let mut comp_ctx = cretonne::Context::new(); comp_ctx.func = func.into_owned(); - comp_ctx.flowgraph(); + comp_ctx.compute_cfg(); // TODO: Should we have an option to skip legalization? comp_ctx.legalize(isa).map_err(|e| { pretty_error(&comp_ctx.func, context.isa, e) })?; + comp_ctx.compute_domtree(); comp_ctx.regalloc(isa).map_err(|e| { pretty_error(&comp_ctx.func, context.isa, e) })?; diff --git a/cranelift/src/filetest/simple_gvn.rs b/cranelift/src/filetest/simple_gvn.rs index 1b04cd67b9..d25618eb1d 100644 --- a/cranelift/src/filetest/simple_gvn.rs +++ b/cranelift/src/filetest/simple_gvn.rs @@ -38,6 +38,7 @@ impl SubTest for TestSimpleGVN { let mut comp_ctx = cretonne::Context::new(); comp_ctx.func = func.into_owned(); + comp_ctx.flowgraph(); comp_ctx.simple_gvn(); comp_ctx.verify(context.isa).map_err(|e| { pretty_error(&comp_ctx.func, context.isa, Into::into(e)) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 6b22d66e31..73d27dac91 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -158,6 +158,8 @@ fn handle_module( context.verify(None).map_err(|err| { pretty_verifier_error(&context.func, None, err) })?; + context.flowgraph(); + context.compute_loop_analysis(); context.licm(); context.verify(None).map_err(|err| { pretty_verifier_error(&context.func, None, err) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index c7347a1079..398fc33c71 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -63,10 +63,11 @@ impl Context { /// /// Returns the size of the function's code. pub fn compile(&mut self, isa: &TargetIsa) -> Result { - self.cfg.compute(&self.func); self.verify_if(isa)?; + self.compute_cfg(); self.legalize(isa)?; + self.compute_domtree(); self.regalloc(isa)?; self.prologue_epilogue(isa)?; self.relax_branches(isa) @@ -103,16 +104,37 @@ impl Context { /// Run the legalizer for `isa` on the function. pub fn legalize(&mut self, isa: &TargetIsa) -> CtonResult { - // Legalization invalidates the domtree by mutating the CFG. + // Legalization invalidates the domtree and loop_analysis by mutating the CFG. + // TODO: Avoid doing this when legalization doesn't actually mutate the CFG. self.domtree.clear(); + self.loop_analysis.clear(); legalize_function(&mut self.func, &mut self.cfg, isa); self.verify_if(isa) } - /// Recompute the control flow graph and dominator tree. + /// Compute the control flow graph. + pub fn compute_cfg(&mut self) { + self.cfg.compute(&self.func) + } + + /// Compute dominator tree. + pub fn compute_domtree(&mut self) { + self.domtree.compute(&self.func, &self.cfg) + } + + /// Compute the loop analysis. + pub fn compute_loop_analysis(&mut self) { + self.loop_analysis.compute( + &self.func, + &self.cfg, + &self.domtree, + ) + } + + /// Compute the control flow graph and dominator tree. pub fn flowgraph(&mut self) { - self.cfg.compute(&self.func); - self.domtree.compute(&self.func, &self.cfg); + self.compute_cfg(); + self.compute_domtree() } /// Perform simple GVN on the function. diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 143df57db6..faf369fcf6 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -37,6 +37,8 @@ pub struct DominatorTree { // Scratch memory used by `compute_postorder()`. stack: Vec, + + valid: bool, } /// Methods for querying the dominator tree. @@ -51,6 +53,7 @@ impl DominatorTree { /// Note that this post-order is not updated automatically when the CFG is modified. It is /// computed from scratch and cached by `compute()`. pub fn cfg_postorder(&self) -> &[Ebb] { + debug_assert!(self.is_valid()); &self.postorder } @@ -203,6 +206,7 @@ impl DominatorTree { nodes: EntityMap::new(), postorder: Vec::new(), stack: Vec::new(), + valid: false, } } @@ -215,8 +219,10 @@ impl DominatorTree { /// Reset and compute a CFG post-order and dominator tree. pub fn compute(&mut self, func: &Function, cfg: &ControlFlowGraph) { + debug_assert!(cfg.is_valid()); self.compute_postorder(func, cfg); self.compute_domtree(func, cfg); + self.valid = true; } /// Clear the data structures used to represent the dominator tree. This will leave the tree in @@ -225,6 +231,7 @@ impl DominatorTree { self.nodes.clear(); self.postorder.clear(); assert!(self.stack.is_empty()); + self.valid = false; } /// Check if the dominator tree is in a valid state. @@ -233,15 +240,7 @@ impl DominatorTree { /// `compute()` method has been called since the last `clear()`. It does not check that the /// dominator tree is consistent with the CFG. pub fn is_valid(&self) -> bool { - !self.nodes.is_empty() - } - - /// Conveneince function to call `compute` if `compute` hasn't been called - /// since the last `clear()`. - pub fn ensure(&mut self, func: &Function, cfg: &ControlFlowGraph) { - if !self.is_valid() { - self.compute(func, cfg) - } + self.valid } /// Reset all internal data structures and compute a post-order for `cfg`. @@ -437,6 +436,7 @@ mod test { fn empty() { let func = Function::new(); let cfg = ControlFlowGraph::with_function(&func); + debug_assert!(cfg.is_valid()); let dtree = DominatorTree::with_function(&func, &cfg); assert_eq!(0, dtree.nodes.keys().count()); assert_eq!(dtree.cfg_postorder(), &[]); diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index 1517ae5ee3..d246a1fa3a 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -47,6 +47,7 @@ pub struct CFGNode { pub struct ControlFlowGraph { entry_block: Option, data: EntityMap, + valid: bool, } impl ControlFlowGraph { @@ -55,6 +56,7 @@ impl ControlFlowGraph { ControlFlowGraph { entry_block: None, data: EntityMap::new(), + valid: false, } } @@ -76,6 +78,8 @@ impl ControlFlowGraph { for ebb in &func.layout { self.compute_ebb(func, ebb); } + + self.valid = true; } fn compute_ebb(&mut self, func: &Function, ebb: Ebb) { @@ -113,6 +117,7 @@ impl ControlFlowGraph { /// more expensive `compute`, and should be used when we know we don't need to recompute the CFG /// from scratch, but rather that our changes have been restricted to specific EBBs. pub fn recompute_ebb(&mut self, func: &Function, ebb: Ebb) { + debug_assert!(self.is_valid()); self.invalidate_ebb_successors(ebb); self.compute_ebb(func, ebb); } @@ -124,11 +129,13 @@ impl ControlFlowGraph { /// Get the CFG predecessor basic blocks to `ebb`. pub fn get_predecessors(&self, ebb: Ebb) -> &[BasicBlock] { + debug_assert!(self.is_valid()); &self.data[ebb].predecessors } /// Get the CFG successors to `ebb`. pub fn get_successors(&self, ebb: Ebb) -> &[Ebb] { + debug_assert!(self.is_valid()); &self.data[ebb].successors } @@ -138,15 +145,7 @@ impl ControlFlowGraph { /// `compute()` method has been called since the last `clear()`. It does not check that the /// CFG is consistent with the function. pub fn is_valid(&self) -> bool { - self.entry_block.is_some() - } - - /// Conveneince function to call `compute` if `compute` hasn't been called - /// since the last `clear()`. - pub fn ensure(&mut self, func: &Function) { - if !self.is_valid() { - self.compute(func) - } + self.valid } } diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 179c0d3884..0367854681 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -33,6 +33,8 @@ use self::heap::expand_heap_addr; /// - Fill out `func.encodings`. /// pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, isa: &TargetIsa) { + debug_assert!(cfg.is_valid()); + boundary::legalize_signatures(func, isa); func.encodings.resize(func.dfg.num_insts()); diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 107f3f7766..58ea3733e9 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -16,9 +16,10 @@ pub fn do_licm( domtree: &mut DominatorTree, loop_analysis: &mut LoopAnalysis, ) { - cfg.ensure(func); - domtree.ensure(func, cfg); - loop_analysis.ensure(func, cfg, domtree); + debug_assert!(cfg.is_valid()); + debug_assert!(domtree.is_valid()); + debug_assert!(loop_analysis.is_valid()); + for lp in loop_analysis.loops() { // For each loop that we want to optimize we determine the set of loop-invariant // instructions diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index 9d5e466c16..4deedbd568 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -18,9 +18,9 @@ entity_impl!(Loop, "loop"); /// Loops are referenced by the Loop object, and for each loop you can access its header EBB, /// its eventual parent in the loop tree and all the EBB belonging to the loop. pub struct LoopAnalysis { - valid: bool, loops: PrimaryMap, ebb_loop_map: EntityMap>, + valid: bool, } struct LoopData { @@ -115,12 +115,13 @@ impl LoopAnalysis { self.valid } - /// Conveneince function to call `compute` if `compute` hasn't been called - /// since the last `clear()`. - pub fn ensure(&mut self, func: &Function, cfg: &ControlFlowGraph, domtree: &DominatorTree) { - if !self.is_valid() { - self.compute(func, cfg, domtree) - } + /// Clear all the data structures contanted in the loop analysis. This will leave the + /// analysis in a similar state to a context returned by `new()` except that allocated + /// memory be retained. + pub fn clear(&mut self) { + self.loops.clear(); + self.ebb_loop_map.clear(); + self.valid = false; } // Traverses the CFG in reverse postorder and create a loop object for every EBB having a diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 92be408919..1cb7d3f129 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -60,8 +60,7 @@ impl Context { cfg: &ControlFlowGraph, domtree: &mut DominatorTree, ) -> CtonResult { - // Ensure that a valid domtree exists. - domtree.ensure(func, cfg); + debug_assert!(domtree.is_valid()); // `Liveness` and `Coloring` are self-clearing. self.virtregs.clear(); diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index d3d8391991..40d0ecf527 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -14,8 +14,8 @@ fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { /// Perform simple GVN on `func`. /// pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph, domtree: &mut DominatorTree) { - cfg.ensure(func); - domtree.ensure(func, cfg); + debug_assert!(cfg.is_valid()); + debug_assert!(domtree.is_valid()); let mut visible_values: HashMap<(InstructionData, Type), Inst> = HashMap::new(); diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index f3952606ca..118192786b 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -134,7 +134,9 @@ pub fn verify_context( isa: Option<&TargetIsa>, ) -> Result { let verifier = Verifier::new(func, isa); - verifier.cfg_integrity(cfg)?; + if cfg.is_valid() { + verifier.cfg_integrity(cfg)?; + } if domtree.is_valid() { verifier.domtree_integrity(domtree)?; } diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 3383b0caf1..acbb43b36b 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -250,7 +250,6 @@ mod tests { trans.translate(&BODY, &mut ctx.func, &mut runtime).unwrap(); dbg!("{}", ctx.func.display(None)); - ctx.flowgraph(); ctx.verify(None).unwrap(); } @@ -284,7 +283,6 @@ mod tests { trans.translate(&BODY, &mut ctx.func, &mut runtime).unwrap(); dbg!("{}", ctx.func.display(None)); - ctx.flowgraph(); ctx.verify(None).unwrap(); } @@ -324,7 +322,6 @@ mod tests { trans.translate(&BODY, &mut ctx.func, &mut runtime).unwrap(); dbg!("{}", ctx.func.display(None)); - ctx.flowgraph(); ctx.verify(None).unwrap(); } } From 0737aa48f23504704f4929e16e57b20280dd9acf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 14 Sep 2017 16:05:42 -0700 Subject: [PATCH 1107/3084] Add legalization for imul_imm. This is handled just like iadd_imm. --- lib/cretonne/meta/base/legalize.py | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index 2bbdd7247a..7fc252bc0c 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -11,6 +11,7 @@ from .immediates import intcc from . import instructions as insts from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm from .instructions import isub, isub_bin, isub_bout, isub_borrow +from .instructions import imul, imul_imm from .instructions import band, bor, bxor, isplit, iconcat from .instructions import bnot, band_not, bor_not, bxor_not from .instructions import icmp, icmp_imm @@ -158,12 +159,15 @@ expand.legalize( )) # Expansions for immediate operands that are out of range. -expand.legalize( - a << iadd_imm(x, y), - Rtl( - a1 << iconst(y), - a << iadd(x, a1) - )) +for inst_imm, inst in [ + (iadd_imm, iadd), + (imul_imm, imul)]: + expand.legalize( + a << inst_imm(x, y), + Rtl( + a1 << iconst(y), + a << inst(x, a1) + )) # Rotates and shifts. for inst_imm, inst in [ From 9e77af25a36043086591a4ec2aab7daaf72fd1b9 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 14 Sep 2017 17:41:43 -0700 Subject: [PATCH 1108/3084] Add settings and isa command-line options to cton-util wasm. (#158) * Add settings and isa command-line options to cton-util wasm. * Use map_err to simplify error handling. * Use `&*` instead of `.borrow()`. --- cranelift/src/cton-util.rs | 10 +++++--- cranelift/src/wasm.rs | 47 ++++++++++++++++++++++++++------------ lib/reader/src/error.rs | 9 ++++++-- lib/reader/src/lib.rs | 2 +- 4 files changed, 47 insertions(+), 21 deletions(-) diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index e702f57ae4..fbb0603cf7 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -30,7 +30,7 @@ Usage: cton-util cat ... cton-util filecheck [-v] cton-util print-cfg ... - cton-util wasm [-cvo] [--enable=]... ... + cton-util wasm [-cvo] [--set ]... [--isa ] ... cton-util --help | --version Options: @@ -38,6 +38,8 @@ Options: -c, --check checks the correctness of Cretonne IL translated from WebAssembly -o, --optimize runs otpimization passes on translated WebAssembly functions -h, --help print this help message + --set= configure Cretonne settings + --isa= specify the Cretonne ISA --version print the Cretonne version "; @@ -53,7 +55,8 @@ struct Args { flag_check: bool, flag_optimize: bool, flag_verbose: bool, - flag_enable: Vec, + flag_set: Vec, + flag_isa: String, } /// A command either succeeds or fails with an error message. @@ -85,7 +88,8 @@ fn cton_util() -> CommandResult { args.flag_verbose, args.flag_optimize, args.flag_check, - args.flag_enable, + args.flag_set, + args.flag_isa, ) } else { // Debugging / shouldn't happen with proper command line handling above. diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 73d27dac91..7ed47a2a9d 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -5,10 +5,12 @@ //! and tables, then emitting the translated code with hardcoded addresses to memory. use cton_wasm::{translate_module, DummyRuntime, WasmRuntime}; +use cton_reader::{parse_options, Location}; use std::path::PathBuf; use cretonne::Context; use cretonne::verifier; -use cretonne::settings::{self, Configurable}; +use cretonne::settings; +use cretonne::isa::{self, TargetIsa}; use std::fs::File; use std::error::Error; use std::io; @@ -50,16 +52,29 @@ pub fn run( flag_verbose: bool, flag_optimize: bool, flag_check: bool, - flag_enable: Vec, + flag_set: Vec, + flag_isa: String, ) -> Result<(), String> { let mut flag_builder = settings::builder(); - for enable in flag_enable { - flag_builder.enable(&enable).map_err(|_| { - format!("unrecognized flag: {}", enable) - })?; - } + parse_options( + flag_set.iter().map(|x| x.as_str()), + &mut flag_builder, + &Location { line_number: 0 }, + ).map_err(|err| err.to_string())?; let flags = settings::Flags::new(&flag_builder); + let mut words = flag_isa.trim().split_whitespace(); + // Look for `isa foo`. + let isa_name = match words.next() { + None => return Err(String::from("expected ISA name")), + Some(w) => w, + }; + let isa_builder = isa::lookup(isa_name).map_err(|err| match err { + isa::LookupError::Unknown => format!("unknown ISA '{}'", isa_name), + isa::LookupError::Unsupported => format!("support for ISA '{}' not enabled", isa_name), + })?; + let isa = isa_builder.finish(settings::Flags::new(&flag_builder)); + for filename in files { let path = Path::new(&filename); let name = String::from(path.as_os_str().to_string_lossy()); @@ -70,6 +85,7 @@ pub fn run( path.to_path_buf(), name, &flags, + &*isa, )?; } Ok(()) @@ -82,6 +98,7 @@ fn handle_module( path: PathBuf, name: String, flags: &settings::Flags, + isa: &TargetIsa, ) -> Result<(), String> { let mut terminal = term::stdout().unwrap(); terminal.fg(term::color::YELLOW).unwrap(); @@ -140,8 +157,8 @@ fn handle_module( vprint!(flag_verbose, "Checking... "); terminal.reset().unwrap(); for func in &translation.functions { - verifier::verify_function(func, None).map_err(|err| { - pretty_verifier_error(func, None, err) + verifier::verify_function(func, Some(isa)).map_err(|err| { + pretty_verifier_error(func, Some(isa), err) })?; } terminal.fg(term::color::GREEN).unwrap(); @@ -155,18 +172,18 @@ fn handle_module( for func in &translation.functions { let mut context = Context::new(); context.func = func.clone(); - context.verify(None).map_err(|err| { - pretty_verifier_error(&context.func, None, err) + context.verify(Some(isa)).map_err(|err| { + pretty_verifier_error(&context.func, Some(isa), err) })?; context.flowgraph(); context.compute_loop_analysis(); context.licm(); - context.verify(None).map_err(|err| { - pretty_verifier_error(&context.func, None, err) + context.verify(Some(isa)).map_err(|err| { + pretty_verifier_error(&context.func, Some(isa), err) })?; context.simple_gvn(); - context.verify(None).map_err(|err| { - pretty_verifier_error(&context.func, None, err) + context.verify(Some(isa)).map_err(|err| { + pretty_verifier_error(&context.func, Some(isa), err) })?; } terminal.fg(term::color::GREEN).unwrap(); diff --git a/lib/reader/src/error.rs b/lib/reader/src/error.rs index c49e523b72..965eba3cb8 100644 --- a/lib/reader/src/error.rs +++ b/lib/reader/src/error.rs @@ -8,7 +8,8 @@ use std::result; /// The location of a `Token` or `Error`. #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] pub struct Location { - /// Line number, starting from 1. + /// Line number. Command-line arguments are line 0 and source file + /// lines start from 1. pub line_number: usize, } @@ -23,7 +24,11 @@ pub struct Error { impl fmt::Display for Error { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - write!(f, "{}: {}", self.location.line_number, self.message) + if self.location.line_number == 0 { + write!(f, "command-line arguments: {}", self.message) + } else { + write!(f, "{}: {}", self.location.line_number, self.message) + } } } diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index 61473afe38..06a03dd6fe 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -11,7 +11,7 @@ pub use error::{Location, Result, Error}; pub use parser::{parse_functions, parse_test}; pub use testcommand::{TestCommand, TestOption}; pub use testfile::{TestFile, Details, Comment}; -pub use isaspec::IsaSpec; +pub use isaspec::{IsaSpec, parse_options}; pub use sourcemap::SourceMap; mod error; From 1349a6bdbcc488043cbf9d2917252ed366519c52 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 14 Sep 2017 17:40:43 -0700 Subject: [PATCH 1109/3084] Always require a Flags reference for verifying functions. Add a settings::FlagsOrIsa struct which represents a flags reference and optionally the ISA it belongs to. Use this for passing flags/isa information to the verifier. The verify_function() and verify_context() functions are now generic so they accept either a &Flags or a &TargetISa argument. Fix the return_at_end verifier tests which no longer require an ISA specified. The signle "set return_at_end" flag setting now makes it to the verifier even when no ISA is present to carry it. --- .../filetests/verifier/return_at_end.cton | 4 --- cranelift/src/filetest/licm.rs | 2 +- cranelift/src/filetest/runone.rs | 8 +++-- cranelift/src/filetest/simple_gvn.rs | 2 +- cranelift/src/filetest/subtest.rs | 12 ++++++- cranelift/src/filetest/verifier.rs | 2 +- cranelift/src/wasm.rs | 21 +++++------- lib/cretonne/src/context.rs | 10 +++--- lib/cretonne/src/dominator_tree.rs | 9 +++-- lib/cretonne/src/regalloc/context.rs | 8 ++--- lib/cretonne/src/settings.rs | 32 +++++++++++++++-- lib/cretonne/src/verifier/mod.rs | 26 ++++++++------ lib/frontend/src/frontend.rs | 4 ++- lib/frontend/src/lib.rs | 4 ++- lib/frontend/src/ssa.rs | 4 ++- lib/wasm/src/func_translator.rs | 8 ++--- lib/wasm/tests/testsuite.rs | 34 ++++++------------- 17 files changed, 111 insertions(+), 79 deletions(-) diff --git a/cranelift/filetests/verifier/return_at_end.cton b/cranelift/filetests/verifier/return_at_end.cton index b0fe05889a..7acc8b8fad 100644 --- a/cranelift/filetests/verifier/return_at_end.cton +++ b/cranelift/filetests/verifier/return_at_end.cton @@ -1,10 +1,6 @@ test verifier set return_at_end -; The verifier doesn't have an API for verifying with flags without also -; setting an ISA. -isa riscv - function %ok(i32) { ebb0(v0: i32): brnz v0, ebb1 diff --git a/cranelift/src/filetest/licm.rs b/cranelift/src/filetest/licm.rs index 5806776e84..3f3bc6cecc 100644 --- a/cranelift/src/filetest/licm.rs +++ b/cranelift/src/filetest/licm.rs @@ -41,7 +41,7 @@ impl SubTest for TestLICM { comp_ctx.flowgraph(); comp_ctx.compute_loop_analysis(); comp_ctx.licm(); - comp_ctx.verify(context.isa).map_err(|e| { + comp_ctx.verify(context.flags_or_isa()).map_err(|e| { pretty_error(&comp_ctx.func, context.isa, Into::into(e)) })?; diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index dad7b89e16..67ea25b08d 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -119,9 +119,11 @@ fn run_one_test<'a>( // Should we run the verifier before this test? if !context.verified && test.needs_verifier() { - verify_function(&func, isa).map_err(|e| { - pretty_verifier_error(&func, isa, e) - })?; + verify_function(&func, context.flags_or_isa()).map_err( + |e| { + pretty_verifier_error(&func, isa, e) + }, + )?; context.verified = true; } diff --git a/cranelift/src/filetest/simple_gvn.rs b/cranelift/src/filetest/simple_gvn.rs index d25618eb1d..12a5554f74 100644 --- a/cranelift/src/filetest/simple_gvn.rs +++ b/cranelift/src/filetest/simple_gvn.rs @@ -40,7 +40,7 @@ impl SubTest for TestSimpleGVN { comp_ctx.flowgraph(); comp_ctx.simple_gvn(); - comp_ctx.verify(context.isa).map_err(|e| { + comp_ctx.verify(context.flags_or_isa()).map_err(|e| { pretty_error(&comp_ctx.func, context.isa, Into::into(e)) })?; diff --git a/cranelift/src/filetest/subtest.rs b/cranelift/src/filetest/subtest.rs index 1129617db2..dc7ac6eeee 100644 --- a/cranelift/src/filetest/subtest.rs +++ b/cranelift/src/filetest/subtest.rs @@ -4,7 +4,7 @@ use std::result; use std::borrow::Cow; use cretonne::ir::Function; use cretonne::isa::TargetIsa; -use cretonne::settings::Flags; +use cretonne::settings::{Flags, FlagsOrIsa}; use cton_reader::{Details, Comment}; use filecheck::{self, CheckerBuilder, Checker, Value as FCValue}; @@ -29,6 +29,16 @@ pub struct Context<'a> { pub isa: Option<&'a TargetIsa>, } +impl<'a> Context<'a> { + /// Get a `FlagsOrIsa` object for passing to the verifier. + pub fn flags_or_isa(&self) -> FlagsOrIsa<'a> { + FlagsOrIsa { + flags: self.flags, + isa: self.isa, + } + } +} + /// Common interface for implementations of test commands. /// /// Each `.cton` test file may contain multiple test commands, each represented by a `SubTest` diff --git a/cranelift/src/filetest/verifier.rs b/cranelift/src/filetest/verifier.rs index 7fa94f7b5e..7834fa778d 100644 --- a/cranelift/src/filetest/verifier.rs +++ b/cranelift/src/filetest/verifier.rs @@ -51,7 +51,7 @@ impl SubTest for TestVerifier { } } - match verify_function(func, context.isa) { + match verify_function(func, context.flags_or_isa()) { Ok(_) => { match expected { None => Ok(()), diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 7ed47a2a9d..e9f9f39c30 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -61,7 +61,6 @@ pub fn run( &mut flag_builder, &Location { line_number: 0 }, ).map_err(|err| err.to_string())?; - let flags = settings::Flags::new(&flag_builder); let mut words = flag_isa.trim().split_whitespace(); // Look for `isa foo`. @@ -84,7 +83,6 @@ pub fn run( flag_check, path.to_path_buf(), name, - &flags, &*isa, )?; } @@ -97,7 +95,6 @@ fn handle_module( flag_check: bool, path: PathBuf, name: String, - flags: &settings::Flags, isa: &TargetIsa, ) -> Result<(), String> { let mut terminal = term::stdout().unwrap(); @@ -144,7 +141,7 @@ fn handle_module( } } }; - let mut dummy_runtime = DummyRuntime::with_flags(flags.clone()); + let mut dummy_runtime = DummyRuntime::with_flags(isa.flags().clone()); let translation = { let runtime: &mut WasmRuntime = &mut dummy_runtime; translate_module(&data, runtime)? @@ -157,8 +154,8 @@ fn handle_module( vprint!(flag_verbose, "Checking... "); terminal.reset().unwrap(); for func in &translation.functions { - verifier::verify_function(func, Some(isa)).map_err(|err| { - pretty_verifier_error(func, Some(isa), err) + verifier::verify_function(func, isa).map_err(|err| { + pretty_verifier_error(func, None, err) })?; } terminal.fg(term::color::GREEN).unwrap(); @@ -172,18 +169,18 @@ fn handle_module( for func in &translation.functions { let mut context = Context::new(); context.func = func.clone(); - context.verify(Some(isa)).map_err(|err| { - pretty_verifier_error(&context.func, Some(isa), err) + context.verify(isa).map_err(|err| { + pretty_verifier_error(&context.func, None, err) })?; context.flowgraph(); context.compute_loop_analysis(); context.licm(); - context.verify(Some(isa)).map_err(|err| { - pretty_verifier_error(&context.func, Some(isa), err) + context.verify(isa).map_err(|err| { + pretty_verifier_error(&context.func, None, err) })?; context.simple_gvn(); - context.verify(Some(isa)).map_err(|err| { - pretty_verifier_error(&context.func, Some(isa), err) + context.verify(isa).map_err(|err| { + pretty_verifier_error(&context.func, None, err) })?; } terminal.fg(term::color::GREEN).unwrap(); diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 398fc33c71..30f67a811f 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -18,6 +18,7 @@ use isa::TargetIsa; use legalize_function; use regalloc; use result::{CtonError, CtonResult}; +use settings::FlagsOrIsa; use verifier; use simple_gvn::do_simple_gvn; use licm::do_licm; @@ -86,17 +87,14 @@ impl Context { /// Run the verifier on the function. /// /// Also check that the dominator tree and control flow graph are consistent with the function. - /// - /// The `isa` argument is currently unused, but the verifier will soon be able to also - /// check ISA-dependent constraints. - pub fn verify(&self, isa: Option<&TargetIsa>) -> verifier::Result { - verifier::verify_context(&self.func, &self.cfg, &self.domtree, isa) + pub fn verify<'a, FOI: Into>>(&self, fisa: FOI) -> verifier::Result { + verifier::verify_context(&self.func, &self.cfg, &self.domtree, fisa) } /// Run the verifier only if the `enable_verifier` setting is true. pub fn verify_if(&self, isa: &TargetIsa) -> CtonResult { if isa.flags().enable_verifier() { - self.verify(Some(isa)).map_err(Into::into) + self.verify(isa).map_err(Into::into) } else { Ok(()) } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index faf369fcf6..cf8ecc8cb9 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -427,9 +427,10 @@ impl DominatorTree { mod test { use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; - use ir::{Function, InstBuilder, types}; - use super::*; use ir::types::*; + use ir::{Function, InstBuilder, types}; + use settings; + use super::*; use verifier::verify_context; #[test] @@ -608,6 +609,8 @@ mod test { dt.recompute_split_ebb(ebb3, ebb4, middle_jump_inst); cfg.compute(cur.func); - verify_context(cur.func, &cfg, &dt, None).unwrap(); + + let flags = settings::Flags::new(&settings::builder()); + verify_context(cur.func, &cfg, &dt, &flags).unwrap(); } } diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 1cb7d3f129..5e5c17c113 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -87,7 +87,7 @@ impl Context { ); if isa.flags().enable_verifier() { - verify_context(func, cfg, domtree, Some(isa))?; + verify_context(func, cfg, domtree, isa)?; verify_liveness(isa, func, cfg, &self.liveness)?; verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?; } @@ -105,7 +105,7 @@ impl Context { ); if isa.flags().enable_verifier() { - verify_context(func, cfg, domtree, Some(isa))?; + verify_context(func, cfg, domtree, isa)?; verify_liveness(isa, func, cfg, &self.liveness)?; verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?; } @@ -121,7 +121,7 @@ impl Context { ); if isa.flags().enable_verifier() { - verify_context(func, cfg, domtree, Some(isa))?; + verify_context(func, cfg, domtree, isa)?; verify_liveness(isa, func, cfg, &self.liveness)?; verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?; } @@ -136,7 +136,7 @@ impl Context { ); if isa.flags().enable_verifier() { - verify_context(func, cfg, domtree, Some(isa))?; + verify_context(func, cfg, domtree, isa)?; verify_liveness(isa, func, cfg, &self.liveness)?; verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?; } diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 43d1639028..67ee68a214 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -20,11 +20,11 @@ //! assert_eq!(f.opt_level(), settings::OptLevel::Fastest); //! ``` +use constant_hash::{probe, simple_hash}; +use isa::TargetIsa; use std::fmt; use std::result; -use constant_hash::{probe, simple_hash}; - /// A string-based configurator for settings groups. /// /// The `Configurable` protocol allows settings to be modified by name before a finished `Flags` @@ -303,6 +303,34 @@ pub mod detail { // with an impl for all of the settings defined in `meta/cretonne/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings.rs")); +/// Wrapper containing flags and optionally a `TargetIsa` trait object. +/// +/// A few passes need to access the flags but only optionally a target ISA. The `FlagsOrIsa` +/// wrapper can be used to pass either, and extract the flags so they are always accessible. +#[derive(Clone, Copy)] +pub struct FlagsOrIsa<'a> { + /// Flags are always present. + pub flags: &'a Flags, + + /// The ISA may not be present. + pub isa: Option<&'a TargetIsa>, +} + +impl<'a> From<&'a Flags> for FlagsOrIsa<'a> { + fn from(flags: &'a Flags) -> FlagsOrIsa { + FlagsOrIsa { flags, isa: None } + } +} + +impl<'a> From<&'a TargetIsa> for FlagsOrIsa<'a> { + fn from(isa: &'a TargetIsa) -> FlagsOrIsa { + FlagsOrIsa { + flags: isa.flags(), + isa: Some(isa), + } + } +} + #[cfg(test)] mod tests { use super::{builder, Flags}; diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 118192786b..f7170bbd24 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -66,6 +66,7 @@ use ir::instructions::{InstructionFormat, BranchInfo, ResolvedConstraint, CallIn use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, StackSlot, StackSlotKind, GlobalVar, Value, Type, Opcode, ValueLoc, ArgumentLoc}; use isa::TargetIsa; +use settings::{Flags, FlagsOrIsa}; use std::error as std_error; use std::fmt::{self, Display, Formatter}; use std::result; @@ -121,19 +122,19 @@ impl std_error::Error for Error { pub type Result = result::Result<(), Error>; /// Verify `func`. -pub fn verify_function(func: &Function, isa: Option<&TargetIsa>) -> Result { - Verifier::new(func, isa).run() +pub fn verify_function<'a, FOI: Into>>(func: &Function, fisa: FOI) -> Result { + Verifier::new(func, fisa.into()).run() } /// Verify `func` after checking the integrity of associated context data structures `cfg` and /// `domtree`. -pub fn verify_context( +pub fn verify_context<'a, FOI: Into>>( func: &Function, cfg: &ControlFlowGraph, domtree: &DominatorTree, - isa: Option<&TargetIsa>, + fisa: FOI, ) -> Result { - let verifier = Verifier::new(func, isa); + let verifier = Verifier::new(func, fisa.into()); if cfg.is_valid() { verifier.cfg_integrity(cfg)?; } @@ -147,18 +148,20 @@ struct Verifier<'a> { func: &'a Function, cfg: ControlFlowGraph, domtree: DominatorTree, + flags: &'a Flags, isa: Option<&'a TargetIsa>, } impl<'a> Verifier<'a> { - pub fn new(func: &'a Function, isa: Option<&'a TargetIsa>) -> Verifier<'a> { + pub fn new(func: &'a Function, fisa: FlagsOrIsa<'a>) -> Verifier<'a> { let cfg = ControlFlowGraph::with_function(func); let domtree = DominatorTree::with_function(func, &cfg); Verifier { func, cfg, domtree, - isa, + flags: fisa.flags, + isa: fisa.isa, } } @@ -971,7 +974,7 @@ impl<'a> Verifier<'a> { } } - if self.isa.map(|isa| isa.flags().return_at_end()) == Some(true) { + if self.flags.return_at_end() { self.verify_return_at_end()?; } @@ -984,6 +987,7 @@ mod tests { use super::{Verifier, Error}; use ir::Function; use ir::instructions::{InstructionData, Opcode}; + use settings; macro_rules! assert_err_with_msg { ($e:expr, $msg:expr) => ( @@ -1001,7 +1005,8 @@ mod tests { #[test] fn empty() { let func = Function::new(); - let verifier = Verifier::new(&func, None); + let flags = &settings::Flags::new(&settings::builder()); + let verifier = Verifier::new(&func, flags.into()); assert_eq!(verifier.run(), Ok(())); } @@ -1014,7 +1019,8 @@ mod tests { InstructionData::Nullary { opcode: Opcode::Jump }, ); func.layout.append_inst(nullary_with_bad_opcode, ebb0); - let verifier = Verifier::new(&func, None); + let flags = &settings::Flags::new(&settings::builder()); + let verifier = Verifier::new(&func, flags.into()); assert_err_with_msg!(verifier.run(), "instruction format"); } } diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 696dad6556..29a4f3f644 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -631,6 +631,7 @@ mod tests { use cretonne::ir::types::*; use frontend::{ILBuilder, FunctionBuilder}; use cretonne::verifier::verify_function; + use cretonne::settings; use std::u32; @@ -727,7 +728,8 @@ mod tests { builder.seal_block(block1); } - let res = verify_function(&func, None); + let flags = settings::Flags::new(&settings::builder()); + let res = verify_function(&func, &flags); // println!("{}", func.display(None)); match res { Ok(_) => {} diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index af31a2c723..cdf2361489 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -38,6 +38,7 @@ //! use cretonne::entity::EntityRef; //! use cretonne::ir::{FunctionName, CallConv, Function, Signature, ArgumentType, InstBuilder}; //! use cretonne::ir::types::*; +//! use cretonne::settings; //! use cton_frontend::{ILBuilder, FunctionBuilder}; //! use cretonne::verifier::verify_function; //! use std::u32; @@ -133,7 +134,8 @@ //! builder.seal_block(block1); //! } //! -//! let res = verify_function(&func, None); +//! let flags = settings::Flags::new(&settings::builder()); +//! let res = verify_function(&func, &flags); //! println!("{}", func.display(None)); //! match res { //! Ok(_) => {} diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 3dd0335009..ed2847c93e 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -589,6 +589,7 @@ mod tests { use cretonne::ir::types::*; use cretonne::verify_function; use cretonne::ir::instructions::BranchInfo; + use cretonne::settings; use ssa::SSABuilder; use std::u32; @@ -1194,7 +1195,8 @@ mod tests { cur.goto_bottom(ebb1); func.dfg.ins(cur).return_(&[]) }; - match verify_function(&func, None) { + let flags = settings::Flags::new(&settings::builder()); + match verify_function(&func, &flags) { Ok(()) => {} Err(err) => panic!(err.message), } diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index acbb43b36b..c38c420bbb 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -218,7 +218,7 @@ fn parse_function_body( mod tests { use cretonne::{ir, Context}; use cretonne::ir::types::I32; - use runtime::DummyRuntime; + use runtime::{DummyRuntime, FuncEnvironment}; use super::FuncTranslator; #[test] @@ -250,7 +250,7 @@ mod tests { trans.translate(&BODY, &mut ctx.func, &mut runtime).unwrap(); dbg!("{}", ctx.func.display(None)); - ctx.verify(None).unwrap(); + ctx.verify(runtime.flags()).unwrap(); } #[test] @@ -283,7 +283,7 @@ mod tests { trans.translate(&BODY, &mut ctx.func, &mut runtime).unwrap(); dbg!("{}", ctx.func.display(None)); - ctx.verify(None).unwrap(); + ctx.verify(runtime.flags()).unwrap(); } #[test] @@ -322,6 +322,6 @@ mod tests { trans.translate(&BODY, &mut ctx.func, &mut runtime).unwrap(); dbg!("{}", ctx.func.display(None)); - ctx.verify(None).unwrap(); + ctx.verify(runtime.flags()).unwrap(); } } diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/testsuite.rs index 96d1a645ef..b5fe5997e8 100644 --- a/lib/wasm/tests/testsuite.rs +++ b/lib/wasm/tests/testsuite.rs @@ -4,7 +4,6 @@ extern crate tempdir; use cton_wasm::{translate_module, DummyRuntime, WasmRuntime}; use std::path::PathBuf; -use std::borrow::Borrow; use std::fs::File; use std::error::Error; use std::io; @@ -15,8 +14,8 @@ use std::process::Command; use std::fs; use cretonne::ir; use cretonne::ir::entities::AnyEntity; -use cretonne::isa::{self, TargetIsa}; -use cretonne::settings::{self, Configurable}; +use cretonne::isa::TargetIsa; +use cretonne::settings::{self, Configurable, Flags}; use cretonne::verifier; use tempdir::TempDir; @@ -27,9 +26,10 @@ fn testsuite() { .map(|r| r.unwrap()) .collect(); paths.sort_by_key(|dir| dir.path()); + let flags = Flags::new(&settings::builder()); for path in paths { let path = path.path(); - handle_module(path, None); + handle_module(path, &flags); } } @@ -37,19 +37,8 @@ fn testsuite() { fn return_at_end() { let mut flag_builder = settings::builder(); flag_builder.enable("return_at_end").unwrap(); - let flags = settings::Flags::new(&flag_builder); - // We don't care about the target itself here, so just pick one arbitrarily. - let isa = match isa::lookup("riscv") { - Err(_) => { - println!("riscv target not found; disabled test return_at_end.wat"); - return; - } - Ok(isa_builder) => isa_builder.finish(flags), - }; - handle_module( - PathBuf::from("../../wasmtests/return_at_end.wat"), - Some(isa.borrow()), - ); + let flags = Flags::new(&flag_builder); + handle_module(PathBuf::from("../../wasmtests/return_at_end.wat"), &flags); } fn read_wasm_file(path: PathBuf) -> Result, io::Error> { @@ -60,7 +49,7 @@ fn read_wasm_file(path: PathBuf) -> Result, io::Error> { Ok(buf) } -fn handle_module(path: PathBuf, isa: Option<&TargetIsa>) { +fn handle_module(path: PathBuf, flags: &Flags) { let data = match path.extension() { None => { panic!("the file extension is not wasm or wat"); @@ -105,17 +94,14 @@ fn handle_module(path: PathBuf, isa: Option<&TargetIsa>) { } } }; - let mut dummy_runtime = match isa { - Some(isa) => DummyRuntime::with_flags(isa.flags().clone()), - None => DummyRuntime::default(), - }; + let mut dummy_runtime = DummyRuntime::with_flags(flags.clone()); let translation = { let runtime: &mut WasmRuntime = &mut dummy_runtime; translate_module(&data, runtime).unwrap() }; for func in &translation.functions { - verifier::verify_function(func, isa) - .map_err(|err| panic!(pretty_verifier_error(func, isa, err))) + verifier::verify_function(func, flags) + .map_err(|err| panic!(pretty_verifier_error(func, None, err))) .unwrap(); } } From cba7a032751b3ed21d2b5aac3e215256bbdeb20e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 15 Sep 2017 09:32:56 -0700 Subject: [PATCH 1110/3084] Give better verifier errors for missing instruction encodings. When possible, provide the ISA's default encoding as a suggesting for the missing instruction encoding. --- lib/cretonne/src/verifier/mod.rs | 50 ++++++++++++++++++++------------ 1 file changed, 31 insertions(+), 19 deletions(-) diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index f7170bbd24..bfc45b7d0d 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -920,28 +920,40 @@ impl<'a> Verifier<'a> { return Ok(()); } + // Check if this opcode must be encoded. + let mut needs_enc = None; if opcode.is_branch() { - return err!(inst, "Branch must have an encoding"); + needs_enc = Some("Branch"); + } else if opcode.is_call() { + needs_enc = Some("Call"); + } else if opcode.is_return() { + needs_enc = Some("Return"); + } else if opcode.can_store() { + needs_enc = Some("Store"); + } else if opcode.can_trap() { + needs_enc = Some("Trapping instruction"); + } else if opcode.other_side_effects() { + needs_enc = Some("Instruction with side effects"); } - if opcode.is_call() { - return err!(inst, "Call must have an encoding"); - } - - if opcode.is_return() { - return err!(inst, "Return must have an encoding"); - } - - if opcode.can_store() { - return err!(inst, "Store must have an encoding"); - } - - if opcode.can_trap() { - return err!(inst, "Trapping instruction must have an encoding"); - } - - if opcode.other_side_effects() { - return err!(inst, "Instruction with side effects must have an encoding"); + if let Some(text) = needs_enc { + // This instruction needs an encoding, so generate an error. + // Provide the ISA default encoding as a hint. + match isa.encode( + &self.func.dfg, + &self.func.dfg[inst], + self.func.dfg.ctrl_typevar(inst), + ) { + Ok(enc) => { + return err!( + inst, + "{} must have an encoding (e.g., {})", + text, + isa.encoding_info().display(enc) + ) + } + Err(_) => return err!(inst, "{} must have an encoding", text), + } } Ok(()) From 24a5a02752bc164fe926317a2f73ff14fc9596be Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 15 Sep 2017 10:48:53 -0700 Subject: [PATCH 1111/3084] Add a RegDiversions::display() function. Display the current register diversions for debugging purposes. --- lib/cretonne/src/regalloc/diversion.rs | 32 +++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index 103c624d5e..769998de3a 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -9,7 +9,8 @@ use entity::EntityMap; use ir::{Value, ValueLoc}; -use isa::RegUnit; +use isa::{RegUnit, RegInfo}; +use std::fmt; /// A diversion of a value from its original register location to a new register. /// @@ -99,6 +100,35 @@ impl RegDiversions { }, ) } + + /// Return an object that can display the diversions. + pub fn display<'a, R: Into>>(&'a self, regs: R) -> DisplayDiversions<'a> { + DisplayDiversions(self, regs.into()) + } +} + +/// Object that displays register diversions. +pub struct DisplayDiversions<'a>(&'a RegDiversions, Option<&'a RegInfo>); + +impl<'a> fmt::Display for DisplayDiversions<'a> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{{")?; + for div in self.0.all() { + match self.1 { + Some(regs) => { + write!( + f, + " {}: {} -> {}", + div.value, + regs.display_regunit(div.from), + regs.display_regunit(div.to) + )? + } + None => write!(f, " {}: %{} -> %{}", div.value, div.from, div.to)?, + } + } + write!(f, " }}") + } } #[cfg(test)] From cc3707706cbefd47286a5a2e90ec41b3069ff857 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 15 Sep 2017 11:21:29 -0700 Subject: [PATCH 1112/3084] Write and parse value locations for EBB arguments Fixes #56. We now have complete support for value location annotations in the textual IL format. Values defined by instructions as well as EBB arguments are covered. --- .../parser/instruction_encoding.cton | 4 +-- cranelift/filetests/regalloc/coalesce.cton | 3 ++- cranelift/filetests/regalloc/spill.cton | 4 +-- lib/cretonne/src/ir/valueloc.rs | 8 ++++++ lib/cretonne/src/write.rs | 26 ++++++++++++++----- lib/reader/src/parser.rs | 23 ++++++++++++---- 6 files changed, 52 insertions(+), 16 deletions(-) diff --git a/cranelift/filetests/parser/instruction_encoding.cton b/cranelift/filetests/parser/instruction_encoding.cton index c808892701..c9c668dc3f 100644 --- a/cranelift/filetests/parser/instruction_encoding.cton +++ b/cranelift/filetests/parser/instruction_encoding.cton @@ -5,7 +5,7 @@ isa riscv ; regex: WS=[ \t]* function %foo(i32, i32) { -ebb1(v0: i32, v1: i32): +ebb1(v0: i32 [%x8], v1: i32): [-,-] v2 = iadd v0, v1 [-] trap [R#1234, %x5, %x11] v6, v7 = iadd_cout v2, v0 @@ -14,7 +14,7 @@ ebb1(v0: i32, v1: i32): [Iret#5] return v0, v8 } ; sameln: function %foo(i32, i32) native { -; nextln: $ebb1($v0: i32, $v1: i32): +; nextln: $ebb1($v0: i32 [%x8], $v1: i32): ; nextln: [-,-]$WS $v2 = iadd $v0, $v1 ; nextln: [-]$WS trap ; nextln: [R#1234,%x5,%x11]$WS $v6, $v7 = iadd_cout $v2, $v0 diff --git a/cranelift/filetests/regalloc/coalesce.cton b/cranelift/filetests/regalloc/coalesce.cton index 80cd38e62e..7624eaef7c 100644 --- a/cranelift/filetests/regalloc/coalesce.cton +++ b/cranelift/filetests/regalloc/coalesce.cton @@ -4,6 +4,7 @@ isa riscv ; Test the coalescer. ; regex: V=v\d+ ; regex: WS=\s+ +; regex: LOC=%\w+ ; This function is already CSSA, so no copies should be inserted. function %cssa(i32) -> i32 { @@ -81,7 +82,7 @@ ebb0(v0: i32): ebb1(v10: i32, v11: i32): ; v11 needs to be isolated because it interferes with v10. - ; check: $ebb1($v10: i32, $(nv11a=$V): i32) + ; check: $ebb1($v10: i32 [$LOC], $(nv11a=$V): i32 [$LOC]) ; check: $v11 = copy $nv11a v12 = iadd v10, v11 v13 = icmp ult v12, v0 diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index 901509a8d4..8f00c93184 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -25,7 +25,7 @@ function %pyramid(i32) -> i32 { ; check: ss2 = spill_slot 4 ; not: spill_slot ebb0(v1: i32): -; check: $ebb0($(rv1=$V): i32, $(rlink=$V): i32) +; check: $ebb0($(rv1=$V): i32 [%x10], $(rlink=$V): i32 [%x1]) ; check: ,ss0]$WS $v1 = spill $rv1 ; nextln: ,ss1]$WS $(link=$V) = spill $rlink ; not: spill @@ -155,7 +155,7 @@ function %use_spilled_value(i32) -> i32 { ; check: ss1 = spill_slot 4 ; check: ss2 = spill_slot 4 ebb0(v1: i32): -; check: $ebb0($(rv1=$V): i32, $(rlink=$V): i32) +; check: $ebb0($(rv1=$V): i32 [%x10], $(rlink=$V): i32 [%x1]) ; check: ,ss0]$WS $v1 = spill $rv1 ; nextln: ,ss1]$WS $(link=$V) = spill $rlink ; not: spill diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/cretonne/src/ir/valueloc.rs index a6a358e14b..1e3cf20f87 100644 --- a/lib/cretonne/src/ir/valueloc.rs +++ b/lib/cretonne/src/ir/valueloc.rs @@ -25,6 +25,14 @@ impl Default for ValueLoc { } impl ValueLoc { + /// Is this an assigned location? (That is, not `Unassigned`). + pub fn is_assigned(&self) -> bool { + match *self { + ValueLoc::Unassigned => false, + _ => true, + } + } + /// Get the register unit of this location, or panic. pub fn unwrap_reg(self) -> RegUnit { match self { diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 77f0209da3..3467f79adb 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -91,11 +91,22 @@ fn write_preamble( // // ====--------------------------------------------------------------------------------------====// -pub fn write_arg(w: &mut Write, func: &Function, arg: Value) -> Result { - write!(w, "{}: {}", arg, func.dfg.value_type(arg)) +pub fn write_arg(w: &mut Write, func: &Function, regs: Option<&RegInfo>, arg: Value) -> Result { + write!(w, "{}: {}", arg, func.dfg.value_type(arg))?; + let loc = func.locations[arg]; + if loc.is_assigned() { + write!(w, " [{}]", loc.display(regs))? + } + + Ok(()) } -pub fn write_ebb_header(w: &mut Write, func: &Function, ebb: Ebb) -> Result { +pub fn write_ebb_header( + w: &mut Write, + func: &Function, + isa: Option<&TargetIsa>, + ebb: Ebb, +) -> Result { // Write out the basic block header, outdented: // // ebb1: @@ -108,24 +119,27 @@ pub fn write_ebb_header(w: &mut Write, func: &Function, ebb: Ebb) -> Result { write!(w, " ")?; } + let regs = isa.map(TargetIsa::register_info); + let regs = regs.as_ref(); + let mut args = func.dfg.ebb_args(ebb).iter().cloned(); match args.next() { None => return writeln!(w, "{}:", ebb), Some(arg) => { write!(w, "{}(", ebb)?; - write_arg(w, func, arg)?; + write_arg(w, func, regs, arg)?; } } // Remaining arguments. for arg in args { write!(w, ", ")?; - write_arg(w, func, arg)?; + write_arg(w, func, regs, arg)?; } writeln!(w, "):") } pub fn write_ebb(w: &mut Write, func: &Function, isa: Option<&TargetIsa>, ebb: Ebb) -> Result { - write_ebb_header(w, func, ebb)?; + write_ebb_header(w, func, isa, ebb)?; for inst in func.layout.ebb_insts(ebb) { write_instruction(w, func, isa, inst)?; } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index e5c59530fc..f9b3f334e1 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1449,22 +1449,35 @@ impl<'a> Parser<'a> { // Parse a single EBB argument declaration, and append it to `ebb`. // - // ebb-arg ::= * Value(v) ":" Type(t) + // ebb-arg ::= * Value(v) ":" Type(t) arg-loc? + // arg-loc ::= "[" value-location "]" // fn parse_ebb_arg(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { - // ebb-arg ::= * Value(v) ":" Type(t) + // ebb-arg ::= * Value(v) ":" Type(t) arg-loc? let v = self.match_value("EBB argument must be a value")?; let v_location = self.loc; - // ebb-arg ::= Value(v) * ":" Type(t) + // ebb-arg ::= Value(v) * ":" Type(t) arg-loc? self.match_token( Token::Colon, "expected ':' after EBB argument", )?; - // ebb-arg ::= Value(v) ":" * Type(t) + // ebb-arg ::= Value(v) ":" * Type(t) arg-loc? let t = self.match_type("expected EBB argument type")?; // Allocate the EBB argument and add the mapping. let value = ctx.function.dfg.append_ebb_arg(ebb, t); - ctx.map.def_value(v, value, &v_location) + ctx.map.def_value(v, value, &v_location)?; + + // ebb-arg ::= Value(v) ":" Type(t) * arg-loc? + if self.optional(Token::LBracket) { + let loc = self.parse_value_location(ctx)?; + ctx.function.locations[value] = loc; + self.match_token( + Token::RBracket, + "expected ']' after value location", + )?; + } + + Ok(()) } fn parse_value_location(&mut self, ctx: &Context) -> Result { From 446fcdd7c5b65ab6735f3a3397769f782ae7cd2f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 15 Sep 2017 13:02:36 -0700 Subject: [PATCH 1113/3084] Fix the REX bits for load/store instruction encodings. The two registers were swapped in the REX encoding, and the tests didn't have any high bit set registers. --- cranelift/filetests/isa/intel/binary64.cton | 144 ++++++++++---------- lib/cretonne/meta/isa/intel/recipes.py | 18 +-- 2 files changed, 81 insertions(+), 81 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 47f3e72a3c..c654c014e8 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -149,75 +149,75 @@ ebb0: ; Register indirect addressing with no displacement. - ; asm: movq %rcx, (%rsi) - store v1, v2 ; bin: 48 89 0e - ; asm: movq %rsi, (%rcx) - store v2, v1 ; bin: 48 89 31 - ; asm: movl %ecx, (%rsi) - istore32 v1, v2 ; bin: 40 89 0e - ; asm: movl %esi, (%rcx) - istore32 v2, v1 ; bin: 40 89 31 - ; asm: movw %cx, (%rsi) - istore16 v1, v2 ; bin: 66 40 89 0e - ; asm: movw %si, (%rcx) - istore16 v2, v1 ; bin: 66 40 89 31 - ; asm: movb %cl, (%rsi) - istore8 v1, v2 ; bin: 40 88 0e - ; asm: movb %sil, (%rcx) - istore8 v2, v1 ; bin: 40 88 31 + ; asm: movq %rcx, (%r10) + store v1, v3 ; bin: 49 89 0a + ; asm: movq %r10, (%rcx) + store v3, v1 ; bin: 4c 89 11 + ; asm: movl %ecx, (%r10) + istore32 v1, v3 ; bin: 41 89 0a + ; asm: movl %r10d, (%rcx) + istore32 v3, v1 ; bin: 44 89 11 + ; asm: movw %cx, (%r10) + istore16 v1, v3 ; bin: 66 41 89 0a + ; asm: movw %r10w, (%rcx) + istore16 v3, v1 ; bin: 66 44 89 11 + ; asm: movb %cl, (%r10) + istore8 v1, v3 ; bin: 41 88 0a + ; asm: movb %r10b, (%rcx) + istore8 v3, v1 ; bin: 44 88 11 - ; asm: movq (%rcx), %rdi - [-,%rdi] v120 = load.i64 v1 ; bin: 48 8b 39 - ; asm: movq (%rsi), %rdx - [-,%rdx] v121 = load.i64 v2 ; bin: 48 8b 16 - ; asm: movl (%rcx), %edi - [-,%rdi] v122 = uload32.i64 v1 ; bin: 40 8b 39 - ; asm: movl (%rsi), %edx - [-,%rdx] v123 = uload32.i64 v2 ; bin: 40 8b 16 - ; asm: movslq (%rcx), %rdi - [-,%rdi] v124 = sload32.i64 v1 ; bin: 48 63 39 - ; asm: movslq (%rsi), %rdx - [-,%rdx] v125 = sload32.i64 v2 ; bin: 48 63 16 - ; asm: movzwq (%rcx), %rdi - [-,%rdi] v126 = uload16.i64 v1 ; bin: 48 0f b7 39 - ; asm: movzwq (%rsi), %rdx - [-,%rdx] v127 = uload16.i64 v2 ; bin: 48 0f b7 16 - ; asm: movswq (%rcx), %rdi - [-,%rdi] v128 = sload16.i64 v1 ; bin: 48 0f bf 39 - ; asm: movswq (%rsi), %rdx - [-,%rdx] v129 = sload16.i64 v2 ; bin: 48 0f bf 16 - ; asm: movzbq (%rcx), %rdi - [-,%rdi] v130 = uload8.i64 v1 ; bin: 48 0f b6 39 - ; asm: movzbq (%rsi), %rdx - [-,%rdx] v131 = uload8.i64 v2 ; bin: 48 0f b6 16 - ; asm: movsbq (%rcx), %rdi - [-,%rdi] v132 = sload8.i64 v1 ; bin: 48 0f be 39 - ; asm: movsbq (%rsi), %rdx - [-,%rdx] v133 = sload8.i64 v2 ; bin: 48 0f be 16 + ; asm: movq (%rcx), %r14 + [-,%r14] v120 = load.i64 v1 ; bin: 4c 8b 31 + ; asm: movq (%r10), %rdx + [-,%rdx] v121 = load.i64 v3 ; bin: 49 8b 12 + ; asm: movl (%rcx), %r14d + [-,%r14] v122 = uload32.i64 v1 ; bin: 44 8b 31 + ; asm: movl (%r10), %edx + [-,%rdx] v123 = uload32.i64 v3 ; bin: 41 8b 12 + ; asm: movslq (%rcx), %r14 + [-,%r14] v124 = sload32.i64 v1 ; bin: 4c 63 31 + ; asm: movslq (%r10), %rdx + [-,%rdx] v125 = sload32.i64 v3 ; bin: 49 63 12 + ; asm: movzwq (%rcx), %r14 + [-,%r14] v126 = uload16.i64 v1 ; bin: 4c 0f b7 31 + ; asm: movzwq (%r10), %rdx + [-,%rdx] v127 = uload16.i64 v3 ; bin: 49 0f b7 12 + ; asm: movswq (%rcx), %r14 + [-,%r14] v128 = sload16.i64 v1 ; bin: 4c 0f bf 31 + ; asm: movswq (%r10), %rdx + [-,%rdx] v129 = sload16.i64 v3 ; bin: 49 0f bf 12 + ; asm: movzbq (%rcx), %r14 + [-,%r14] v130 = uload8.i64 v1 ; bin: 4c 0f b6 31 + ; asm: movzbq (%r10), %rdx + [-,%rdx] v131 = uload8.i64 v3 ; bin: 49 0f b6 12 + ; asm: movsbq (%rcx), %r14 + [-,%r14] v132 = sload8.i64 v1 ; bin: 4c 0f be 31 + ; asm: movsbq (%r10), %rdx + [-,%rdx] v133 = sload8.i64 v3 ; bin: 49 0f be 12 ; Register-indirect with 8-bit signed displacement. - ; asm: movq %rcx, 100(%rsi) - store v1, v2+100 ; bin: 48 89 4e 64 - ; asm: movq %rsi, -100(%rcx) - store v2, v1-100 ; bin: 48 89 71 9c - ; asm: movl %ecx, 100(%rsi) - istore32 v1, v2+100 ; bin: 40 89 4e 64 - ; asm: movl %esi, -100(%rcx) - istore32 v2, v1-100 ; bin: 40 89 71 9c - ; asm: movw %cx, 100(%rsi) - istore16 v1, v2+100 ; bin: 66 40 89 4e 64 - ; asm: movw %si, -100(%rcx) - istore16 v2, v1-100 ; bin: 66 40 89 71 9c - ; asm: movb %cl, 100(%rsi) - istore8 v1, v2+100 ; bin: 40 88 4e 64 - ; asm: movb %sil, 100(%rcx) - istore8 v2, v1+100 ; bin: 40 88 71 64 + ; asm: movq %rcx, 100(%r10) + store v1, v3+100 ; bin: 49 89 4a 64 + ; asm: movq %r10, -100(%rcx) + store v3, v1-100 ; bin: 4c 89 51 9c + ; asm: movl %ecx, 100(%r10) + istore32 v1, v3+100 ; bin: 41 89 4a 64 + ; asm: movl %r10d, -100(%rcx) + istore32 v3, v1-100 ; bin: 44 89 51 9c + ; asm: movw %cx, 100(%r10) + istore16 v1, v3+100 ; bin: 66 41 89 4a 64 + ; asm: movw %r10w, -100(%rcx) + istore16 v3, v1-100 ; bin: 66 44 89 51 9c + ; asm: movb %cl, 100(%r10) + istore8 v1, v3+100 ; bin: 41 88 4a 64 + ; asm: movb %r10b, 100(%rcx) + istore8 v3, v1+100 ; bin: 44 88 51 64 - ; asm: movq 50(%rcx), %rdi - [-,%rdi] v140 = load.i64 v1+50 ; bin: 48 8b 79 32 - ; asm: movq -50(%rsi), %rdx - [-,%rdx] v141 = load.i64 v2-50 ; bin: 48 8b 56 ce + ; asm: movq 50(%rcx), %r10 + [-,%r10] v140 = load.i64 v1+50 ; bin: 4c 8b 51 32 + ; asm: movq -50(%r10), %rdx + [-,%rdx] v141 = load.i64 v3-50 ; bin: 49 8b 52 ce ; asm: movl 50(%rcx), %edi [-,%rdi] v142 = uload32.i64 v1+50 ; bin: 40 8b 79 32 ; asm: movl -50(%rsi), %edx @@ -245,10 +245,10 @@ ebb0: ; Register-indirect with 32-bit signed displacement. - ; asm: movq %rcx, 10000(%rsi) - store v1, v2+10000 ; bin: 48 89 8e 00002710 - ; asm: movq %rsi, -10000(%rcx) - store v2, v1-10000 ; bin: 48 89 b1 ffffd8f0 + ; asm: movq %rcx, 10000(%r10) + store v1, v3+10000 ; bin: 49 89 8a 00002710 + ; asm: movq %r10, -10000(%rcx) + store v3, v1-10000 ; bin: 4c 89 91 ffffd8f0 ; asm: movl %ecx, 10000(%rsi) istore32 v1, v2+10000 ; bin: 40 89 8e 00002710 ; asm: movl %esi, -10000(%rcx) @@ -262,10 +262,10 @@ ebb0: ; asm: movb %sil, 10000(%rcx) istore8 v2, v1+10000 ; bin: 40 88 b1 00002710 - ; asm: movq 50000(%rcx), %rdi - [-,%rdi] v160 = load.i64 v1+50000 ; bin: 48 8b b9 0000c350 - ; asm: movq -50000(%rsi), %rdx - [-,%rdx] v161 = load.i64 v2-50000 ; bin: 48 8b 96 ffff3cb0 + ; asm: movq 50000(%rcx), %r10 + [-,%r10] v160 = load.i64 v1+50000 ; bin: 4c 8b 91 0000c350 + ; asm: movq -50000(%r10), %rdx + [-,%rdx] v161 = load.i64 v3-50000 ; bin: 49 8b 92 ffff3cb0 ; asm: movl 50000(%rcx), %edi [-,%rdi] v162 = uload32.i64 v1+50000 ; bin: 40 8b b9 0000c350 ; asm: movl -50000(%rsi), %edx diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 1302350f7c..b45091d837 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -360,7 +360,7 @@ st = TailRecipe( 'st', Store, size=1, ins=(GPR, GPR), outs=(), instp=IsEqual(Store.offset, 0), emit=''' - PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_rm(in_reg1, in_reg0, sink); ''') @@ -370,7 +370,7 @@ st_abcd = TailRecipe( 'st_abcd', Store, size=1, ins=(ABCD, GPR), outs=(), instp=IsEqual(Store.offset, 0), emit=''' - PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_rm(in_reg1, in_reg0, sink); ''') @@ -379,7 +379,7 @@ stDisp8 = TailRecipe( 'stDisp8', Store, size=2, ins=(GPR, GPR), outs=(), instp=IsSignedInt(Store.offset, 8), emit=''' - PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp8(in_reg1, in_reg0, sink); let offset: i32 = offset.into(); sink.put1(offset as u8); @@ -388,7 +388,7 @@ stDisp8_abcd = TailRecipe( 'stDisp8_abcd', Store, size=2, ins=(ABCD, GPR), outs=(), instp=IsSignedInt(Store.offset, 8), emit=''' - PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp8(in_reg1, in_reg0, sink); let offset: i32 = offset.into(); sink.put1(offset as u8); @@ -398,7 +398,7 @@ stDisp8_abcd = TailRecipe( stDisp32 = TailRecipe( 'stDisp32', Store, size=5, ins=(GPR, GPR), outs=(), emit=''' - PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp32(in_reg1, in_reg0, sink); let offset: i32 = offset.into(); sink.put4(offset as u32); @@ -406,7 +406,7 @@ stDisp32 = TailRecipe( stDisp32_abcd = TailRecipe( 'stDisp32_abcd', Store, size=5, ins=(ABCD, GPR), outs=(), emit=''' - PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp32(in_reg1, in_reg0, sink); let offset: i32 = offset.into(); sink.put4(offset as u32); @@ -421,7 +421,7 @@ ld = TailRecipe( 'ld', Load, size=1, ins=(GPR), outs=(GPR), instp=IsEqual(Load.offset, 0), emit=''' - PUT_OP(bits, rex2(out_reg0, in_reg0), sink); + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_rm(in_reg0, out_reg0, sink); ''') @@ -430,7 +430,7 @@ ldDisp8 = TailRecipe( 'ldDisp8', Load, size=2, ins=(GPR), outs=(GPR), instp=IsSignedInt(Load.offset, 8), emit=''' - PUT_OP(bits, rex2(out_reg0, in_reg0), sink); + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_disp8(in_reg0, out_reg0, sink); let offset: i32 = offset.into(); sink.put1(offset as u8); @@ -441,7 +441,7 @@ ldDisp32 = TailRecipe( 'ldDisp32', Load, size=5, ins=(GPR), outs=(GPR), instp=IsSignedInt(Load.offset, 32), emit=''' - PUT_OP(bits, rex2(out_reg0, in_reg0), sink); + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_disp32(in_reg0, out_reg0, sink); let offset: i32 = offset.into(); sink.put4(offset as u32); From f5f37ac4a6db655bcfa3dfac4ec4f22604286059 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 15 Sep 2017 15:43:47 -0700 Subject: [PATCH 1114/3084] Use FlagsOrIsa to avoid requiring a --isa argument in cton-util wasm. --- cranelift/src/wasm.rs | 48 +++++++++++++++++++++++++------------------ 1 file changed, 28 insertions(+), 20 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index e9f9f39c30..7e9040c2c9 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -9,7 +9,7 @@ use cton_reader::{parse_options, Location}; use std::path::PathBuf; use cretonne::Context; use cretonne::verifier; -use cretonne::settings; +use cretonne::settings::{self, FlagsOrIsa}; use cretonne::isa::{self, TargetIsa}; use std::fs::File; use std::error::Error; @@ -46,6 +46,10 @@ fn read_wasm_file(path: PathBuf) -> Result, io::Error> { Ok(buf) } +enum OwnedFlagsOrIsa { + Flags(settings::Flags), + Isa(Box), +} pub fn run( files: Vec, @@ -64,15 +68,19 @@ pub fn run( let mut words = flag_isa.trim().split_whitespace(); // Look for `isa foo`. - let isa_name = match words.next() { - None => return Err(String::from("expected ISA name")), - Some(w) => w, + let owned_fisa = if let Some(isa_name) = words.next() { + let isa_builder = isa::lookup(isa_name).map_err(|err| match err { + isa::LookupError::Unknown => format!("unknown ISA '{}'", isa_name), + isa::LookupError::Unsupported => format!("support for ISA '{}' not enabled", isa_name), + })?; + OwnedFlagsOrIsa::Isa(isa_builder.finish(settings::Flags::new(&flag_builder))) + } else { + OwnedFlagsOrIsa::Flags(settings::Flags::new(&flag_builder)) + }; + let fisa = match owned_fisa { + OwnedFlagsOrIsa::Flags(ref flags) => FlagsOrIsa::from(flags), + OwnedFlagsOrIsa::Isa(ref isa) => FlagsOrIsa::from(&**isa), }; - let isa_builder = isa::lookup(isa_name).map_err(|err| match err { - isa::LookupError::Unknown => format!("unknown ISA '{}'", isa_name), - isa::LookupError::Unsupported => format!("support for ISA '{}' not enabled", isa_name), - })?; - let isa = isa_builder.finish(settings::Flags::new(&flag_builder)); for filename in files { let path = Path::new(&filename); @@ -83,7 +91,7 @@ pub fn run( flag_check, path.to_path_buf(), name, - &*isa, + &fisa, )?; } Ok(()) @@ -95,7 +103,7 @@ fn handle_module( flag_check: bool, path: PathBuf, name: String, - isa: &TargetIsa, + fisa: &FlagsOrIsa, ) -> Result<(), String> { let mut terminal = term::stdout().unwrap(); terminal.fg(term::color::YELLOW).unwrap(); @@ -141,7 +149,7 @@ fn handle_module( } } }; - let mut dummy_runtime = DummyRuntime::with_flags(isa.flags().clone()); + let mut dummy_runtime = DummyRuntime::with_flags(fisa.flags.clone()); let translation = { let runtime: &mut WasmRuntime = &mut dummy_runtime; translate_module(&data, runtime)? @@ -154,8 +162,8 @@ fn handle_module( vprint!(flag_verbose, "Checking... "); terminal.reset().unwrap(); for func in &translation.functions { - verifier::verify_function(func, isa).map_err(|err| { - pretty_verifier_error(func, None, err) + verifier::verify_function(func, *fisa).map_err(|err| { + pretty_verifier_error(func, fisa.isa, err) })?; } terminal.fg(term::color::GREEN).unwrap(); @@ -169,18 +177,18 @@ fn handle_module( for func in &translation.functions { let mut context = Context::new(); context.func = func.clone(); - context.verify(isa).map_err(|err| { - pretty_verifier_error(&context.func, None, err) + context.verify(*fisa).map_err(|err| { + pretty_verifier_error(&context.func, fisa.isa, err) })?; context.flowgraph(); context.compute_loop_analysis(); context.licm(); - context.verify(isa).map_err(|err| { - pretty_verifier_error(&context.func, None, err) + context.verify(*fisa).map_err(|err| { + pretty_verifier_error(&context.func, fisa.isa, err) })?; context.simple_gvn(); - context.verify(isa).map_err(|err| { - pretty_verifier_error(&context.func, None, err) + context.verify(*fisa).map_err(|err| { + pretty_verifier_error(&context.func, fisa.isa, err) })?; } terminal.fg(term::color::GREEN).unwrap(); From da4cde8117aa1c5d2d6bf5c777c114e609ed0658 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 18 Sep 2017 11:53:32 -0700 Subject: [PATCH 1115/3084] Always insert the entry EBB before translating any WASM. The FuncEnvironment callbacks (make_global in particular) may need to insert code in the entry EBB. We need to make sure the entry EBB has been inserted in the layout before making those callbacks. --- lib/frontend/src/frontend.rs | 2 +- lib/wasm/src/func_translator.rs | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 29a4f3f644..86a91ae977 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -373,7 +373,7 @@ where } /// Make sure that the current EBB is inserted in the layout. - fn ensure_inserted_ebb(&mut self) { + pub fn ensure_inserted_ebb(&mut self) { let ebb = self.position.ebb; if self.builder.ebbs[ebb].pristine { if !self.func.layout.is_ebb_inserted(ebb) { diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index c38c420bbb..9a43083461 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -84,6 +84,10 @@ impl FuncTranslator { let entry_block = builder.create_ebb(); builder.switch_to_block(entry_block, &[]); // This also creates values for the arguments. builder.seal_block(entry_block); + // Make sure the entry block is inserted in the layout before we make any callbacks to + // `environ`. The callback functions may need to insert things in the entry block. + builder.ensure_inserted_ebb(); + let num_args = declare_wasm_arguments(builder); // Set up the translation state with a single pushed control block representing the whole From d2273c73eaf5f5d3c2278258b10cdf86021c6a87 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Mon, 18 Sep 2017 20:41:01 +0100 Subject: [PATCH 1116/3084] Make the verifier accept any of the legal encodings of an instruction --- .../filetests/isa/riscv/verify-encoding.cton | 2 +- lib/cretonne/src/verifier/mod.rs | 56 ++++++++++++------- 2 files changed, 38 insertions(+), 20 deletions(-) diff --git a/cranelift/filetests/isa/riscv/verify-encoding.cton b/cranelift/filetests/isa/riscv/verify-encoding.cton index b88fdc6402..942179ba23 100644 --- a/cranelift/filetests/isa/riscv/verify-encoding.cton +++ b/cranelift/filetests/isa/riscv/verify-encoding.cton @@ -16,6 +16,6 @@ function %RV32I(i32 link [%x1]) -> i32 link [%x1] { ebb0(v9999: i32): v1 = iconst.i32 1 v2 = iconst.i32 2 - [R#0,-] v3 = iadd v1, v2 ; error: Instruction re-encoding + [R#0,-] v3 = iadd v1, v2 ; error: Instruction encoding R#00 doesn't match any possibilities return v9999 } diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index bfc45b7d0d..cded1ffc70 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -68,7 +68,7 @@ use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpT use isa::TargetIsa; use settings::{Flags, FlagsOrIsa}; use std::error as std_error; -use std::fmt::{self, Display, Formatter}; +use std::fmt::{self, Display, Formatter, Write}; use std::result; use std::collections::BTreeSet; use std::cmp::Ordering; @@ -883,29 +883,47 @@ impl<'a> Verifier<'a> { let encoding = self.func.encodings[inst]; if encoding.is_legal() { - let verify_encoding = isa.encode( + let mut encodings = isa.legal_encodings( &self.func.dfg, &self.func.dfg[inst], self.func.dfg.ctrl_typevar(inst), - ); - match verify_encoding { - Ok(verify_encoding) => { - if verify_encoding != encoding { - return err!( - inst, - "Instruction re-encoding {} doesn't match {}", - isa.encoding_info().display(verify_encoding), - isa.encoding_info().display(encoding) - ); + ).peekable(); + + if encodings.peek().is_none() { + return err!( + inst, + "Instruction failed to re-encode {}", + isa.encoding_info().display(encoding) + ); + } + + let has_valid_encoding = encodings + .position(|possible_enc| encoding == possible_enc) + .is_some(); + + if !has_valid_encoding { + let mut possible_encodings = String::new(); + + for enc in isa.legal_encodings( + &self.func.dfg, + &self.func.dfg[inst], + self.func.dfg.ctrl_typevar(inst), + ) + { + if possible_encodings.len() != 0 { + possible_encodings.push_str(", "); } + possible_encodings + .write_fmt(format_args!("{}", isa.encoding_info().display(enc))) + .unwrap(); } - Err(_) => { - return err!( - inst, - "Instruction failed to re-encode {}", - isa.encoding_info().display(encoding) - ) - } + + return err!( + inst, + "Instruction encoding {} doesn't match any possibilities: [{}]", + isa.encoding_info().display(encoding), + possible_encodings + ); } return Ok(()); } From 88348368a8be42071262f6c184b6dcdddbe43761 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 18 Sep 2017 13:27:25 -0700 Subject: [PATCH 1117/3084] Add custom legalization for floating point constants. Use the simplest expansion which materializes the bits of the floating point constant as an integer and then bit-casts to the floating point type. In the future, we may want to use constant pools instead. Either way, we need custom legalization. Also add a legalize_monomorphic() function to the Python targetISA class which permits the configuration of a default legalization action for monomorphic instructions, just like legalize_type() does for polymorphic instructions. --- .../filetests/isa/intel/legalize-custom.cton | 16 ++++++++++++++ lib/cretonne/meta/base/legalize.py | 5 +++++ lib/cretonne/meta/cdsl/isa.py | 10 +++++++++ lib/cretonne/meta/isa/intel/encodings.py | 2 ++ lib/cretonne/meta/isa/riscv/encodings.py | 2 ++ lib/cretonne/src/ir/immediates.rs | 10 +++++++++ lib/cretonne/src/legalizer/mod.rs | 22 +++++++++++++++++++ 7 files changed, 67 insertions(+) diff --git a/cranelift/filetests/isa/intel/legalize-custom.cton b/cranelift/filetests/isa/intel/legalize-custom.cton index e8af47a460..4c8488173b 100644 --- a/cranelift/filetests/isa/intel/legalize-custom.cton +++ b/cranelift/filetests/isa/intel/legalize-custom.cton @@ -52,3 +52,19 @@ ebb0(v1: i32): ; check: $new: ; nextln: return } + +function %f32const() -> f32 { +ebb0: + v1 = f32const 0x1.0p1 + ; check: $(tmp=$V) = iconst.i32 + ; check: $v1 = bitcast.f32 $tmp + return v1 +} + +function %f64const() -> f64 { +ebb0: + v1 = f64const 0x1.0p1 + ; check: $(tmp=$V) = iconst.i64 + ; check: $v1 = bitcast.f64 $tmp + return v1 +} diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index 7fc252bc0c..626295e165 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -54,6 +54,11 @@ expand.custom_legalize(insts.heap_addr, 'expand_heap_addr') expand.custom_legalize(insts.trapz, 'expand_cond_trap') expand.custom_legalize(insts.trapnz, 'expand_cond_trap') +# Custom expansions for floating point constants. +# These expansions require bit-casting or creating constant pool entries. +expand.custom_legalize(insts.f32const, 'expand_fconst') +expand.custom_legalize(insts.f64const, 'expand_fconst') + x = Var('x') y = Var('y') a = Var('a') diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 3e971e7a5a..ae517203e1 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -263,6 +263,16 @@ class CPUMode(object): ty = ValueType.by_name(name) self.type_legalize[ty] = xgrp + def legalize_monomorphic(self, xgrp): + # type: (XFormGroup) -> None + """ + Configure the legalization action to take for monomorphic instructions + which don't have a controlling type variable. + + See also `legalize_type()` for polymorphic instructions. + """ + self.type_legalize[None] = xgrp + def get_legalize_action(self, ty): # type: (ValueType) -> XFormGroup """ diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index ed0bbb433d..ffc6d28458 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -20,6 +20,7 @@ except ImportError: pass +I32.legalize_monomorphic(expand) I32.legalize_type( default=narrow, b1=expand, @@ -27,6 +28,7 @@ I32.legalize_type( f32=expand, f64=expand) +I64.legalize_monomorphic(expand) I64.legalize_type( default=narrow, b1=expand, diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index b9f1f8245d..990b5c4112 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -13,12 +13,14 @@ from .settings import use_m from cdsl.ast import Var from base.legalize import narrow, expand +RV32.legalize_monomorphic(expand) RV32.legalize_type( default=narrow, i32=expand, f32=expand, f64=expand) +RV64.legalize_monomorphic(expand) RV64.legalize_type( default=narrow, i32=expand, diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index f33e13eccf..51120344cd 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -599,6 +599,11 @@ impl Ieee32 { pub fn with_float(x: f32) -> Ieee32 { Ieee32(unsafe { mem::transmute(x) }) } + + /// Get the bitwise representation. + pub fn bits(self) -> u32 { + self.0 + } } impl Display for Ieee32 { @@ -630,6 +635,11 @@ impl Ieee64 { pub fn with_float(x: f64) -> Ieee64 { Ieee64(unsafe { mem::transmute(x) }) } + + /// Get the bitwise representation. + pub fn bits(self) -> u64 { + self.0 + } } impl Display for Ieee64 { diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 0367854681..f47dc87241 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -146,3 +146,25 @@ fn expand_cond_trap(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFl cfg.recompute_ebb(pos.func, old_ebb); cfg.recompute_ebb(pos.func, new_ebb); } + +/// Expand illegal `f32const` and `f64const` instructions. +fn expand_fconst(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlowGraph) { + let ty = func.dfg.value_type(func.dfg.first_result(inst)); + assert!(ty.is_scalar(), "Only scalar fconst supported: {}", ty); + + // In the future, we may want to generate constant pool entries for these constants, but for + // now use an `iconst` and a bit cast. + let mut pos = FuncCursor::new(func).at_inst(inst); + let ival = match pos.func.dfg[inst] { + ir::InstructionData::UnaryIeee32 { + opcode: ir::Opcode::F32const, + imm, + } => pos.ins().iconst(ir::types::I32, imm.bits() as i64), + ir::InstructionData::UnaryIeee64 { + opcode: ir::Opcode::F64const, + imm, + } => pos.ins().iconst(ir::types::I64, imm.bits() as i64), + _ => panic!("Expected fconst: {}", pos.func.dfg.display_inst(inst, None)), + }; + pos.func.dfg.replace(inst).bitcast(ty, ival); +} From 1fdeddd0d3d350bbae193eb63d19e7aaaefd6ecc Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 18 Sep 2017 18:23:53 -0700 Subject: [PATCH 1118/3084] Add Intel encodings for floating point load/store instructions. Include wasm/*-memory64.cton tests too. --- .../filetests/isa/intel/binary32-float.cton | 56 +++++++++ .../filetests/isa/intel/binary64-float.cton | 56 +++++++++ cranelift/filetests/wasm/f32-memory64.cton | 27 ++++ cranelift/filetests/wasm/f64-memory64.cton | 27 ++++ cranelift/filetests/wasm/i32-memory64.cton | 88 +++++++++++++ cranelift/filetests/wasm/i64-memory64.cton | 117 ++++++++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 20 +++ lib/cretonne/meta/isa/intel/recipes.py | 57 +++++++++ 8 files changed, 448 insertions(+) create mode 100644 cranelift/filetests/wasm/f32-memory64.cton create mode 100644 cranelift/filetests/wasm/f64-memory64.cton create mode 100644 cranelift/filetests/wasm/i32-memory64.cton create mode 100644 cranelift/filetests/wasm/i64-memory64.cton diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 5c0dc43b18..3756fbff1e 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -77,6 +77,34 @@ ebb0: ; asm: xorps %xmm5, %xmm2 [-,%xmm2] v37 = bxor v11, v10 ; bin: 0f 57 d5 + ; Load/Store + + ; asm: movd (%ecx), %xmm5 + [-,%xmm5] v100 = load.f32 v0 ; bin: 66 0f 6e 29 + ; asm: movd (%esi), %xmm2 + [-,%xmm2] v101 = load.f32 v1 ; bin: 66 0f 6e 16 + ; asm: movd 50(%ecx), %xmm5 + [-,%xmm5] v110 = load.f32 v0+50 ; bin: 66 0f 6e 69 32 + ; asm: movd -50(%esi), %xmm2 + [-,%xmm2] v111 = load.f32 v1-50 ; bin: 66 0f 6e 56 ce + ; asm: movd 10000(%ecx), %xmm5 + [-,%xmm5] v120 = load.f32 v0+10000 ; bin: 66 0f 6e a9 00002710 + ; asm: movd -10000(%esi), %xmm2 + [-,%xmm2] v121 = load.f32 v1-10000 ; bin: 66 0f 6e 96 ffffd8f0 + + ; asm: movd %xmm5, (%ecx) + [-] store.f32 v100, v0 ; bin: 66 0f 7e 29 + ; asm: movd %xmm2, (%esi) + [-] store.f32 v101, v1 ; bin: 66 0f 7e 16 + ; asm: movd %xmm5, 50(%ecx) + [-] store.f32 v100, v0+50 ; bin: 66 0f 7e 69 32 + ; asm: movd %xmm2, -50(%esi) + [-] store.f32 v101, v1-50 ; bin: 66 0f 7e 56 ce + ; asm: movd %xmm5, 10000(%ecx) + [-] store.f32 v100, v0+10000 ; bin: 66 0f 7e a9 00002710 + ; asm: movd %xmm2, -10000(%esi) + [-] store.f32 v101, v1-10000 ; bin: 66 0f 7e 96 ffffd8f0 + return } @@ -142,5 +170,33 @@ ebb0: ; asm: xorps %xmm5, %xmm2 [-,%xmm2] v37 = bxor v11, v10 ; bin: 0f 57 d5 + ; Load/Store + + ; asm: movq (%ecx), %xmm5 + [-,%xmm5] v100 = load.f64 v0 ; bin: f3 0f 7e 29 + ; asm: movq (%esi), %xmm2 + [-,%xmm2] v101 = load.f64 v1 ; bin: f3 0f 7e 16 + ; asm: movq 50(%ecx), %xmm5 + [-,%xmm5] v110 = load.f64 v0+50 ; bin: f3 0f 7e 69 32 + ; asm: movq -50(%esi), %xmm2 + [-,%xmm2] v111 = load.f64 v1-50 ; bin: f3 0f 7e 56 ce + ; asm: movq 10000(%ecx), %xmm5 + [-,%xmm5] v120 = load.f64 v0+10000 ; bin: f3 0f 7e a9 00002710 + ; asm: movq -10000(%esi), %xmm2 + [-,%xmm2] v121 = load.f64 v1-10000 ; bin: f3 0f 7e 96 ffffd8f0 + + ; asm: movq %xmm5, (%ecx) + [-] store.f64 v100, v0 ; bin: 66 0f d6 29 + ; asm: movq %xmm2, (%esi) + [-] store.f64 v101, v1 ; bin: 66 0f d6 16 + ; asm: movq %xmm5, 50(%ecx) + [-] store.f64 v100, v0+50 ; bin: 66 0f d6 69 32 + ; asm: movq %xmm2, -50(%esi) + [-] store.f64 v101, v1-50 ; bin: 66 0f d6 56 ce + ; asm: movq %xmm5, 10000(%ecx) + [-] store.f64 v100, v0+10000 ; bin: 66 0f d6 a9 00002710 + ; asm: movq %xmm2, -10000(%esi) + [-] store.f64 v101, v1-10000 ; bin: 66 0f d6 96 ffffd8f0 + return } diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index 64dd1ebd05..83aaf6d753 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -85,6 +85,34 @@ ebb0: ; asm: xorps %xmm5, %xmm10 [-,%xmm10] v37 = bxor v11, v10 ; bin: 44 0f 57 d5 + ; Load/Store + + ; asm: movd (%r14), %xmm5 + [-,%xmm5] v100 = load.f32 v3 ; bin: 66 41 0f 6e 2e + ; asm: movd (%rax), %xmm10 + [-,%xmm10] v101 = load.f32 v2 ; bin: 66 44 0f 6e 10 + ; asm: movd 50(%r14), %xmm5 + [-,%xmm5] v110 = load.f32 v3+50 ; bin: 66 41 0f 6e 6e 32 + ; asm: movd -50(%rax), %xmm10 + [-,%xmm10] v111 = load.f32 v2-50 ; bin: 66 44 0f 6e 50 ce + ; asm: movd 10000(%r14), %xmm5 + [-,%xmm5] v120 = load.f32 v3+10000 ; bin: 66 41 0f 6e ae 00002710 + ; asm: movd -10000(%rax), %xmm10 + [-,%xmm10] v121 = load.f32 v2-10000 ; bin: 66 44 0f 6e 90 ffffd8f0 + + ; asm: movd %xmm5, (%r14) + [-] store.f32 v100, v3 ; bin: 66 41 0f 7e 2e + ; asm: movd %xmm10, (%rax) + [-] store.f32 v101, v2 ; bin: 66 44 0f 7e 10 + ; asm: movd %xmm5, 50(%r14) + [-] store.f32 v100, v3+50 ; bin: 66 41 0f 7e 6e 32 + ; asm: movd %xmm10, -50(%rax) + [-] store.f32 v101, v2-50 ; bin: 66 44 0f 7e 50 ce + ; asm: movd %xmm5, 10000(%r14) + [-] store.f32 v100, v3+10000 ; bin: 66 41 0f 7e ae 00002710 + ; asm: movd %xmm10, -10000(%rax) + [-] store.f32 v101, v2-10000 ; bin: 66 44 0f 7e 90 ffffd8f0 + return } @@ -165,5 +193,33 @@ ebb0: ; asm: xorps %xmm5, %xmm10 [-,%xmm10] v37 = bxor v11, v10 ; bin: 44 0f 57 d5 + ; Load/Store + + ; asm: movq (%r14), %xmm5 + [-,%xmm5] v100 = load.f64 v3 ; bin: f3 41 0f 7e 2e + ; asm: movq (%rax), %xmm10 + [-,%xmm10] v101 = load.f64 v2 ; bin: f3 44 0f 7e 10 + ; asm: movq 50(%r14), %xmm5 + [-,%xmm5] v110 = load.f64 v3+50 ; bin: f3 41 0f 7e 6e 32 + ; asm: movq -50(%rax), %xmm10 + [-,%xmm10] v111 = load.f64 v2-50 ; bin: f3 44 0f 7e 50 ce + ; asm: movq 10000(%r14), %xmm5 + [-,%xmm5] v120 = load.f64 v3+10000 ; bin: f3 41 0f 7e ae 00002710 + ; asm: movq -10000(%rax), %xmm10 + [-,%xmm10] v121 = load.f64 v2-10000 ; bin: f3 44 0f 7e 90 ffffd8f0 + + ; asm: movq %xmm5, (%r14) + [-] store.f64 v100, v3 ; bin: 66 41 0f d6 2e + ; asm: movq %xmm10, (%rax) + [-] store.f64 v101, v2 ; bin: 66 44 0f d6 10 + ; asm: movq %xmm5, 50(%r14) + [-] store.f64 v100, v3+50 ; bin: 66 41 0f d6 6e 32 + ; asm: movq %xmm10, -50(%rax) + [-] store.f64 v101, v2-50 ; bin: 66 44 0f d6 50 ce + ; asm: movq %xmm5, 10000(%r14) + [-] store.f64 v100, v3+10000 ; bin: 66 41 0f d6 ae 00002710 + ; asm: movq %xmm10, -10000(%rax) + [-] store.f64 v101, v2-10000 ; bin: 66 44 0f d6 90 ffffd8f0 + return } diff --git a/cranelift/filetests/wasm/f32-memory64.cton b/cranelift/filetests/wasm/f32-memory64.cton new file mode 100644 index 0000000000..7125e66d3b --- /dev/null +++ b/cranelift/filetests/wasm/f32-memory64.cton @@ -0,0 +1,27 @@ +; Test basic code generation for f32 memory WebAssembly instructions. +test compile + +; We only test on 64-bit since the heap_addr instructions and vmctx parameters +; explicitly mention the pointer width. +set is_64bit=1 +isa intel haswell + +function %f32_load(i32, i64 vmctx) -> f32 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = load.f32 v2 + return v3 +} + +function %f32_store(f32, i32, i64 vmctx) { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: f32, v1: i32, v2: i64): + v3 = heap_addr.i64 heap0, v1, 1 + store v0, v3 + return +} diff --git a/cranelift/filetests/wasm/f64-memory64.cton b/cranelift/filetests/wasm/f64-memory64.cton new file mode 100644 index 0000000000..1f61749e51 --- /dev/null +++ b/cranelift/filetests/wasm/f64-memory64.cton @@ -0,0 +1,27 @@ +; Test basic code generation for f64 memory WebAssembly instructions. +test compile + +; We only test on 64-bit since the heap_addr instructions and vmctx parameters +; explicitly mention the pointer width. +set is_64bit=1 +isa intel haswell + +function %f64_load(i32, i64 vmctx) -> f64 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = load.f64 v2 + return v3 +} + +function %f64_store(f64, i32, i64 vmctx) { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: f64, v1: i32, v2: i64): + v3 = heap_addr.i64 heap0, v1, 1 + store v0, v3 + return +} diff --git a/cranelift/filetests/wasm/i32-memory64.cton b/cranelift/filetests/wasm/i32-memory64.cton new file mode 100644 index 0000000000..0fbffa4fb8 --- /dev/null +++ b/cranelift/filetests/wasm/i32-memory64.cton @@ -0,0 +1,88 @@ +; Test basic code generation for i32 memory WebAssembly instructions. +test compile + +; We only test on 64-bit since the heap_addr instructions and vmctx parameters +; explicitly mention the pointer width. +set is_64bit=1 +isa intel haswell + +function %i32_load(i32, i64 vmctx) -> i32 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = load.i32 v2 + return v3 +} + +function %i32_store(i32, i32, i64 vmctx) { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i32, v2: i64): + v3 = heap_addr.i64 heap0, v1, 1 + store v0, v3 + return +} + +function %i32_load8_s(i32, i64 vmctx) -> i32 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = sload8.i32 v2 + return v3 +} + +function %i32_load8_u(i32, i64 vmctx) -> i32 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = uload8.i32 v2 + return v3 +} + +function %i32_store8(i32, i32, i64 vmctx) { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i32, v2: i64): + v3 = heap_addr.i64 heap0, v1, 1 + istore8 v0, v3 + return +} + +function %i32_load16_s(i32, i64 vmctx) -> i32 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = sload16.i32 v2 + return v3 +} + +function %i32_load16_u(i32, i64 vmctx) -> i32 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = uload16.i32 v2 + return v3 +} + +function %i32_store16(i32, i32, i64 vmctx) { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i32, v2: i64): + v3 = heap_addr.i64 heap0, v1, 1 + istore16 v0, v3 + return +} + diff --git a/cranelift/filetests/wasm/i64-memory64.cton b/cranelift/filetests/wasm/i64-memory64.cton new file mode 100644 index 0000000000..bc44a2bbea --- /dev/null +++ b/cranelift/filetests/wasm/i64-memory64.cton @@ -0,0 +1,117 @@ +; Test basic code generation for i32 memory WebAssembly instructions. +test compile + +; We only test on 64-bit since the heap_addr instructions and vmctx parameters +; explicitly mention the pointer width. +set is_64bit=1 +isa intel haswell + +function %i64_load(i32, i64 vmctx) -> i64 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = load.i64 v2 + return v3 +} + +function %i64_store(i64, i32, i64 vmctx) { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i64, v1: i32, v2: i64): + v3 = heap_addr.i64 heap0, v1, 1 + store v0, v3 + return +} + +function %i64_load8_s(i32, i64 vmctx) -> i64 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = sload8.i64 v2 + return v3 +} + +function %i64_load8_u(i32, i64 vmctx) -> i64 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = uload8.i64 v2 + return v3 +} + +function %i64_store8(i64, i32, i64 vmctx) { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i64, v1: i32, v2: i64): + v3 = heap_addr.i64 heap0, v1, 1 + istore8 v0, v3 + return +} + +function %i64_load16_s(i32, i64 vmctx) -> i64 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = sload16.i64 v2 + return v3 +} + +function %i64_load16_u(i32, i64 vmctx) -> i64 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = uload16.i64 v2 + return v3 +} + +function %i64_store16(i64, i32, i64 vmctx) { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i64, v1: i32, v2: i64): + v3 = heap_addr.i64 heap0, v1, 1 + istore16 v0, v3 + return +} + +function %i64_load32_s(i32, i64 vmctx) -> i64 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = sload32.i64 v2 + return v3 +} + +function %i64_load32_u(i32, i64 vmctx) -> i64 { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: i64): + v2 = heap_addr.i64 heap0, v0, 1 + v3 = uload32.i64 v2 + return v3 +} + +function %i64_store32(i64, i32, i64 vmctx) { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i64, v1: i32, v2: i64): + v3 = heap_addr.i64 heap0, v1, 1 + istore32 v0, v3 + return +} diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index ffc6d28458..7ac5f80d49 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -223,6 +223,26 @@ enc_i32_i64_ld_st(base.sload8, True, r.ld, 0x0f, 0xbe) enc_i32_i64_ld_st(base.sload8, True, r.ldDisp8, 0x0f, 0xbe) enc_i32_i64_ld_st(base.sload8, True, r.ldDisp32, 0x0f, 0xbe) +# +# Float loads and stores. +# + +enc_flt(base.load.f32.any, r.fld, 0x66, 0x0f, 0x6e) +enc_flt(base.load.f32.any, r.fldDisp8, 0x66, 0x0f, 0x6e) +enc_flt(base.load.f32.any, r.fldDisp32, 0x66, 0x0f, 0x6e) + +enc_flt(base.load.f64.any, r.fld, 0xf3, 0x0f, 0x7e) +enc_flt(base.load.f64.any, r.fldDisp8, 0xf3, 0x0f, 0x7e) +enc_flt(base.load.f64.any, r.fldDisp32, 0xf3, 0x0f, 0x7e) + +enc_flt(base.store.f32.any, r.fst, 0x66, 0x0f, 0x7e) +enc_flt(base.store.f32.any, r.fstDisp8, 0x66, 0x0f, 0x7e) +enc_flt(base.store.f32.any, r.fstDisp32, 0x66, 0x0f, 0x7e) + +enc_flt(base.store.f64.any, r.fst, 0x66, 0x0f, 0xd6) +enc_flt(base.store.f64.any, r.fstDisp8, 0x66, 0x0f, 0xd6) +enc_flt(base.store.f64.any, r.fstDisp32, 0x66, 0x0f, 0xd6) + # # Call/return # diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index b45091d837..8736ca5371 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -374,6 +374,15 @@ st_abcd = TailRecipe( modrm_rm(in_reg1, in_reg0, sink); ''') +# XX /r register-indirect store of FPR with no offset. +fst = TailRecipe( + 'fst', Store, size=1, ins=(FPR, GPR), outs=(), + instp=IsEqual(Store.offset, 0), + emit=''' + PUT_OP(bits, rex2(in_reg1, in_reg0), sink); + modrm_rm(in_reg1, in_reg0, sink); + ''') + # XX /r register-indirect store with 8-bit offset. stDisp8 = TailRecipe( 'stDisp8', Store, size=2, ins=(GPR, GPR), outs=(), @@ -393,6 +402,15 @@ stDisp8_abcd = TailRecipe( let offset: i32 = offset.into(); sink.put1(offset as u8); ''') +fstDisp8 = TailRecipe( + 'fstDisp8', Store, size=2, ins=(FPR, GPR), outs=(), + instp=IsSignedInt(Store.offset, 8), + emit=''' + PUT_OP(bits, rex2(in_reg1, in_reg0), sink); + modrm_disp8(in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + ''') # XX /r register-indirect store with 32-bit offset. stDisp32 = TailRecipe( @@ -411,6 +429,14 @@ stDisp32_abcd = TailRecipe( let offset: i32 = offset.into(); sink.put4(offset as u32); ''') +fstDisp32 = TailRecipe( + 'fstDisp32', Store, size=5, ins=(FPR, GPR), outs=(), + emit=''' + PUT_OP(bits, rex2(in_reg1, in_reg0), sink); + modrm_disp32(in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + ''') # # Load recipes @@ -425,6 +451,15 @@ ld = TailRecipe( modrm_rm(in_reg0, out_reg0, sink); ''') +# XX /r float load with no offset. +fld = TailRecipe( + 'fld', Load, size=1, ins=(GPR), outs=(FPR), + instp=IsEqual(Load.offset, 0), + emit=''' + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); + modrm_rm(in_reg0, out_reg0, sink); + ''') + # XX /r load with 8-bit offset. ldDisp8 = TailRecipe( 'ldDisp8', Load, size=2, ins=(GPR), outs=(GPR), @@ -436,6 +471,17 @@ ldDisp8 = TailRecipe( sink.put1(offset as u8); ''') +# XX /r float load with 8-bit offset. +fldDisp8 = TailRecipe( + 'fldDisp8', Load, size=2, ins=(GPR), outs=(FPR), + instp=IsSignedInt(Load.offset, 8), + emit=''' + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); + modrm_disp8(in_reg0, out_reg0, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + ''') + # XX /r load with 32-bit offset. ldDisp32 = TailRecipe( 'ldDisp32', Load, size=5, ins=(GPR), outs=(GPR), @@ -447,6 +493,17 @@ ldDisp32 = TailRecipe( sink.put4(offset as u32); ''') +# XX /r float load with 32-bit offset. +fldDisp32 = TailRecipe( + 'fldDisp32', Load, size=5, ins=(GPR), outs=(FPR), + instp=IsSignedInt(Load.offset, 32), + emit=''' + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); + modrm_disp32(in_reg0, out_reg0, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + ''') + # # Call/return # From 0cfea8858a0dae2480e0cf9849f3b7ba995dbe20 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 19 Sep 2017 12:59:07 -0700 Subject: [PATCH 1119/3084] Add uext() and sext() builder methods to ArgumentType. This makes it simpler to construct arguments like: ArgumentType::new(I32).uext() --- lib/cretonne/src/ir/extfunc.rs | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 5983182d87..8b8cc9dca0 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -173,6 +173,24 @@ impl ArgumentType { } } + /// Convert `self` to an argument type with the `uext` flag set. + pub fn uext(self) -> ArgumentType { + debug_assert!(self.value_type.is_int(), "uext on {} arg", self.value_type); + ArgumentType { + extension: ArgumentExtension::Uext, + ..self + } + } + + /// Convert `self` to an argument type with the `sext` flag set. + pub fn sext(self) -> ArgumentType { + debug_assert!(self.value_type.is_int(), "sext on {} arg", self.value_type); + ArgumentType { + extension: ArgumentExtension::Sext, + ..self + } + } + /// Return an object that can display `self` with correct register names. pub fn display<'a, R: Into>>(&'a self, regs: R) -> DisplayArgumentType<'a> { DisplayArgumentType(self, regs.into()) @@ -369,10 +387,11 @@ mod tests { #[test] fn argument_type() { - let mut t = ArgumentType::new(I32); + let t = ArgumentType::new(I32); assert_eq!(t.to_string(), "i32"); - t.extension = ArgumentExtension::Uext; + let mut t = t.uext(); assert_eq!(t.to_string(), "i32 uext"); + assert_eq!(t.sext().to_string(), "i32 sext"); t.purpose = ArgumentPurpose::StructReturn; assert_eq!(t.to_string(), "i32 uext sret"); } From d92686d1cdbda4cf753dba6dca5b54a57426f7a6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 19 Sep 2017 15:54:02 -0700 Subject: [PATCH 1120/3084] Add a func_addr instruction. Get the callable address of a function. Use for long distance calls and for creating arguments to call_indirect in general. --- cranelift/docs/langref.rst | 8 +------- cranelift/filetests/parser/call.cton | 13 +++++++++++++ lib/cretonne/meta/base/formats.py | 1 + lib/cretonne/meta/base/instructions.py | 15 ++++++++++++++- lib/cretonne/src/ir/instructions.rs | 1 + lib/cretonne/src/verifier/mod.rs | 3 +++ lib/cretonne/src/write.rs | 1 + lib/reader/src/parser.rs | 8 ++++++++ 8 files changed, 42 insertions(+), 8 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index c8c7ede4fd..1e148190e6 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -411,13 +411,7 @@ This simple example illustrates direct function calls and signatures:: Indirect function calls use a signature declared in the preamble. .. autoinst:: call_indirect - -.. todo:: Define safe indirect function calls. - - The :inst:`call_indirect` instruction is dangerous to use in a sandboxed - environment since it is not easy to verify the callee address. - We need a table-driven indirect call instruction, similar to - :inst:`br_table`. +.. autoinst:: func_addr Memory diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton index 662925db7f..f4ed00c222 100644 --- a/cranelift/filetests/parser/call.cton +++ b/cranelift/filetests/parser/call.cton @@ -69,6 +69,19 @@ ebb0(v0: i64): ; check: $v3, $v4 = call_indirect $sig2, $v1() ; check: return +function %long_call() { + sig0 = () + fn0 = sig0 %none + +ebb0: + v0 = func_addr.i32 fn0 + call_indirect sig0, v0() + return +} +; check: $v0 = func_addr.i32 $fn0 +; check: call_indirect $sig0, $v0() +; check: return + ; Special purpose function arguments function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret { ebb0(v1: i32, v2: i32, v3: i32, v4: i32): diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 64a9adb456..783539e105 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -49,6 +49,7 @@ BranchTable = InstructionFormat(VALUE, entities.jump_table) Call = InstructionFormat(func_ref, VARIABLE_ARGS) IndirectCall = InstructionFormat(sig_ref, VALUE, VARIABLE_ARGS) +FuncAddr = InstructionFormat(func_ref) Load = InstructionFormat(memflags, VALUE, offset32) Store = InstructionFormat(memflags, VALUE, VALUE, offset32) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 482cb2a12d..735ec742bc 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -38,6 +38,8 @@ MemTo = TypeVar( 'MemTo', 'Any type that can be stored in memory', ints=True, floats=True, simd=True) +addr = Operand('addr', iAddr) + # # Control flow # @@ -185,6 +187,18 @@ call_indirect = Instruction( """, ins=(SIG, callee, args), outs=rvals, is_call=True) +func_addr = Instruction( + 'func_addr', r""" + Get the address of a function. + + Compute the absolute address of a function declared in the preamble. + The returned address can be used as a ``callee`` argument to + :inst:`call_indirect`. This is also a method for calling functions that + are too far away to be addressable by a direct :inst:`call` + instruction. + """, + ins=FN, outs=addr) + # # Memory operations # @@ -194,7 +208,6 @@ Offset = Operand('Offset', offset32, 'Byte offset from base address') x = Operand('x', Mem, doc='Value to be stored') a = Operand('a', Mem, doc='Value loaded') p = Operand('p', iAddr) -addr = Operand('addr', iAddr) Flags = Operand('Flags', memflags) load = Instruction( diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index b37646afbf..9a9301e7db 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -178,6 +178,7 @@ pub enum InstructionData { sig_ref: SigRef, args: ValueList, }, + FuncAddr { opcode: Opcode, func_ref: FuncRef }, StackLoad { opcode: Opcode, stack_slot: StackSlot, diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index cded1ffc70..7fcb4b508f 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -306,6 +306,9 @@ impl<'a> Verifier<'a> { self.verify_sig_ref(inst, sig_ref)?; self.verify_value_list(inst, args)?; } + FuncAddr { func_ref, .. } => { + self.verify_func_ref(inst, func_ref)?; + } StackLoad { stack_slot, .. } | StackStore { stack_slot, .. } => { self.verify_stack_slot(inst, stack_slot)?; diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 3467f79adb..56d1eb4652 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -346,6 +346,7 @@ pub fn write_operands( DisplayValues(&args[1..]) ) } + FuncAddr { func_ref, .. } => write!(w, " {}", func_ref), StackLoad { stack_slot, offset, .. } => write!(w, " {}{}", stack_slot, offset), StackStore { arg, diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index f9b3f334e1..d9dd334964 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -2098,6 +2098,14 @@ impl<'a> Parser<'a> { args: args.into_value_list(&[callee], &mut ctx.function.dfg.value_lists), } } + InstructionFormat::FuncAddr => { + let func_ref = self.match_fn("expected function reference").and_then( + |num| { + ctx.get_fn(num, &self.loc) + }, + )?; + InstructionData::FuncAddr { opcode, func_ref } + } InstructionFormat::BranchTable => { let arg = self.match_value("expected SSA value operand")?; self.match_token( From fb827a2d4bb1c00027fd93e9a9117dbb9d89ca43 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 19 Sep 2017 16:33:38 -0700 Subject: [PATCH 1121/3084] Add func_addr encodings for Intel. --- cranelift/filetests/isa/intel/binary32.cton | 9 +++++++-- cranelift/filetests/isa/intel/binary64.cton | 13 +++++++++--- lib/cretonne/meta/isa/intel/encodings.py | 7 +++++++ lib/cretonne/meta/isa/intel/recipes.py | 22 ++++++++++++++++++++- lib/cretonne/src/isa/intel/binemit.rs | 8 +++++++- 5 files changed, 52 insertions(+), 7 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index af1bf73a9b..b531dfdfd7 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -336,10 +336,15 @@ ebb0: ; asm: call foo call fn0() ; bin: e8 PCRel4(fn0) 00000000 + ; asm: movl $-1, %ecx + [-,%rcx] v400 = func_addr.i32 fn0 ; bin: b9 Abs4(fn0) ffffffff + ; asm: movl $-1, %esi + [-,%rsi] v401 = func_addr.i32 fn0 ; bin: be Abs4(fn0) ffffffff + ; asm: call *%ecx - call_indirect sig0, v1() ; bin: ff d1 + call_indirect sig0, v400() ; bin: ff d1 ; asm: call *%esi - call_indirect sig0, v2() ; bin: ff d6 + call_indirect sig0, v401() ; bin: ff d6 ; asm: testl %ecx, %ecx ; asm: je ebb1 diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index c654c014e8..680da4273e 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -421,12 +421,19 @@ ebb0: ; asm: call foo call fn0() ; bin: e8 PCRel4(fn0) 00000000 + ; asm: movabsq $-1, %rcx + [-,%rcx] v400 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(fn0) ffffffffffffffff + ; asm: movabsq $-1, %rsi + [-,%rsi] v401 = func_addr.i64 fn0 ; bin: 48 be Abs8(fn0) ffffffffffffffff + ; asm: movabsq $-1, %r10 + [-,%r10] v402 = func_addr.i64 fn0 ; bin: 49 ba Abs8(fn0) ffffffffffffffff + ; asm: call *%rcx - call_indirect sig0, v1() ; bin: 40 ff d1 + call_indirect sig0, v400() ; bin: 40 ff d1 ; asm: call *%rsi - call_indirect sig0, v2() ; bin: 40 ff d6 + call_indirect sig0, v401() ; bin: 40 ff d6 ; asm: call *%r10 - call_indirect sig0, v3() ; bin: 41 ff d2 + call_indirect sig0, v402() ; bin: 41 ff d2 ; asm: testq %rcx, %rcx ; asm: je ebb1 diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 7ac5f80d49..bf1b91c0dc 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -243,6 +243,13 @@ enc_flt(base.store.f64.any, r.fst, 0x66, 0x0f, 0xd6) enc_flt(base.store.f64.any, r.fstDisp8, 0x66, 0x0f, 0xd6) enc_flt(base.store.f64.any, r.fstDisp32, 0x66, 0x0f, 0xd6) +# +# Function addresses. +# + +I32.enc(base.func_addr.i32, *r.fnaddr4(0xb8)) +I64.enc(base.func_addr.i64, *r.fnaddr8.rex(0xb8, w=1)) + # # Call/return # diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 8736ca5371..06aea4b01f 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -7,7 +7,7 @@ from cdsl.predicates import IsSignedInt, IsEqual from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry from base.formats import Nullary, Call, IndirectCall, Store, Load from base.formats import IntCompare -from base.formats import RegMove, Ternary, Jump, Branch +from base.formats import RegMove, Ternary, Jump, Branch, FuncAddr from .registers import GPR, ABCD, FPR try: @@ -351,6 +351,26 @@ puiq = TailRecipe( sink.put8(imm as u64); ''') +# XX+rd id with Abs4 function relocation. +fnaddr4 = TailRecipe( + 'fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, + emit=''' + PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); + sink.reloc_func(RelocKind::Abs4.into(), func_ref); + // Write the immediate as `!0` for the benefit of BaldrMonkey. + sink.put4(!0); + ''') + +# XX+rd iq with Abs8 function relocation. +fnaddr8 = TailRecipe( + 'fnaddr8', FuncAddr, size=8, ins=(), outs=GPR, + emit=''' + PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); + sink.reloc_func(RelocKind::Abs8.into(), func_ref); + // Write the immediate as `!0` for the benefit of BaldrMonkey. + sink.put8(!0); + ''') + # # Store recipes. # diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 0eb6199e66..6f75833be6 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -11,9 +11,15 @@ include!(concat!(env!("OUT_DIR"), "/binemit-intel.rs")); pub enum RelocKind { /// A 4-byte relative function reference. Based from relocation + 4 bytes. PCRel4, + + /// A 4-byte absolute function reference. + Abs4, + + /// An 8-byte absolute function reference. + Abs8, } -pub static RELOC_NAMES: [&'static str; 1] = ["PCRel4"]; +pub static RELOC_NAMES: [&'static str; 3] = ["PCRel4", "Abs4", "Abs8"]; impl Into for RelocKind { fn into(self) -> Reloc { From 5194298d9b3ccd7b61d0f29d72d5efc00a0af802 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 19 Sep 2017 14:03:54 -0700 Subject: [PATCH 1122/3084] Fix a typo in a comment. --- lib/cretonne/src/loop_analysis.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index 4deedbd568..05ce1952f2 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -157,7 +157,7 @@ impl LoopAnalysis { layout: &Layout, ) { let mut stack: Vec = Vec::new(); - // We handle each loop header in reverse order, corresponding to a pesudo postorder + // We handle each loop header in reverse order, corresponding to a pseudo postorder // traversal of the graph. for lp in self.loops().rev() { for &(pred, pred_inst) in cfg.get_predecessors(self.loops[lp].header) { From 144d39a53dec9b0dbfc145e65caf3bccaf230512 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 19 Sep 2017 14:04:03 -0700 Subject: [PATCH 1123/3084] Minor code simplification. --- lib/frontend/src/frontend.rs | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 86a91ae977..98ef98ac62 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -260,10 +260,7 @@ where let basic_block = self.builder.ssa.header_block(ebb); // Then we change the cursor position. - self.position = Position { - ebb: ebb, - basic_block: basic_block, - }; + self.position = Position { ebb, basic_block }; self.ebb_args_adjustement(ebb, jump_args); self.func.dfg.ebb_args(ebb) } From b888894fbb1721990f4b006ece790dd770ccca5b Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 20 Sep 2017 06:58:45 -0700 Subject: [PATCH 1124/3084] Use `CursorBase` utility functions to reduce repositioning clutter. --- lib/cretonne/src/ir/layout.rs | 19 ++++++++++++++++++- lib/cretonne/src/legalizer/boundary.rs | 3 +-- lib/cretonne/src/legalizer/split.rs | 3 +-- lib/cretonne/src/licm.rs | 3 +-- lib/frontend/src/ssa.rs | 3 +-- 5 files changed, 22 insertions(+), 9 deletions(-) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 5c908812cf..b92a6d7336 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -788,10 +788,27 @@ pub trait CursorBase { self.set_position(CursorPosition::At(inst)); } + /// Go to the position for inserting instructions at the beginning of `ebb`, + /// which unlike `goto_first_inst` doesn't assume that any instructions have + /// been inserted into `ebb` yet. + fn goto_first_insertion_point(&mut self, ebb: Ebb) { + if let Some(inst) = self.layout().ebbs[ebb].first_inst.expand() { + self.goto_inst(inst); + } else { + self.goto_bottom(ebb); + } + } + /// Go to the first instruction in `ebb`. fn goto_first_inst(&mut self, ebb: Ebb) { let inst = self.layout().ebbs[ebb].first_inst.expect("Empty EBB"); - self.set_position(CursorPosition::At(inst)); + self.goto_inst(inst); + } + + /// Go to the last instruction in `ebb`. + fn goto_last_inst(&mut self, ebb: Ebb) { + let inst = self.layout().ebbs[ebb].last_inst.expect("Empty EBB"); + self.goto_inst(inst); } /// Go to the top of `ebb` which must be inserted into the layout. diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 2e16627efe..a95ec2cfba 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -63,8 +63,7 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { // We want to insert instructions before the first instruction in the entry block. // If the entry block is empty, append instructions to it instead. let mut pos = Cursor::new(&mut func.layout); - pos.goto_top(entry); - pos.next_inst(); + pos.goto_first_inst(entry); // Keep track of the argument types in the ABI-legalized signature. let abi_types = &func.signature.argument_types; diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index b225586842..23c0e02bcd 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -235,8 +235,7 @@ fn split_value( // // Note that it is safe to move `pos` here since `reuse` was set above, so we don't // need to insert a split instruction before returning. - pos.goto_top(ebb); - pos.next_inst(); + pos.goto_first_inst(ebb); dfg.ins(pos).with_result(value).Binary( concat, split_type, diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 58ea3733e9..0a739f8f2d 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -34,8 +34,7 @@ pub fn do_licm( let pre_header = create_pre_header(loop_analysis.loop_header(lp), func, cfg, domtree); pos = Cursor::new(&mut func.layout); - pos.goto_bottom(pre_header); - pos.prev_inst(); + pos.goto_last_inst(pre_header); } // If there is a natural pre-header we insert new instructions just before the // related jumping instruction (which is not necessarily at the end). diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index ed2847c93e..73d10e0494 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -445,8 +445,7 @@ where layout.append_ebb(dest_ebb) }; let mut cur = Cursor::new(layout); - cur.goto_top(dest_ebb); - cur.next_inst(); + cur.goto_first_insertion_point(dest_ebb); let ty = dfg.value_type(temp_arg_val); let val = if ty.is_int() { dfg.ins(&mut cur).iconst(ty, 0) From 482439c94b4f2ea806e04b81136eb24f5b2228d9 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 20 Sep 2017 09:55:01 -0700 Subject: [PATCH 1125/3084] Minor code cleanups in simple_gvn's main loop. Redundant load/store elimination isn't critical for the use case of optimizing wasm code which has already been optimized, so remove the TODO for that for now. --- lib/cretonne/src/simple_gvn.rs | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index 40d0ecf527..20e2298440 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -7,8 +7,9 @@ use std::collections::HashMap; /// Test whether the given opcode is unsafe to even consider for GVN. fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { - opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || opcode.is_return() || - opcode.can_trap() || opcode.other_side_effects() + opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || + opcode.is_return() || opcode.can_trap() || opcode.other_side_effects() || + opcode.can_store() || opcode.can_load() } /// Perform simple GVN on `func`. @@ -26,24 +27,15 @@ pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph, domtree: & pos.goto_top(ebb); while let Some(inst) = pos.next_inst() { - let opcode = func.dfg[inst].opcode(); - let ctrl_typevar = func.dfg.ctrl_typevar(inst); - // Resolve aliases, particularly aliases we created earlier. func.dfg.resolve_aliases_in_arguments(inst); + let opcode = func.dfg[inst].opcode(); if trivially_unsafe_for_gvn(opcode) { continue; } - // TODO: Implement simple redundant-load elimination. - if opcode.can_store() { - continue; - } - if opcode.can_load() { - continue; - } - + let ctrl_typevar = func.dfg.ctrl_typevar(inst); let key = (func.dfg[inst].clone(), ctrl_typevar); let entry = visible_values.entry(key); use std::collections::hash_map::Entry::*; From 74845fed6c409cf37a07df4090d090c1c2ceaf3f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 20 Sep 2017 11:19:35 -0700 Subject: [PATCH 1126/3084] Use more helper functions on `CursorBase` to reduce cursor repositioning. --- lib/cretonne/src/ir/layout.rs | 109 +++++++++++++++++++++++-- lib/cretonne/src/legalizer/boundary.rs | 3 +- lib/cretonne/src/legalizer/mod.rs | 3 +- lib/cretonne/src/licm.rs | 9 +- lib/cretonne/src/regalloc/coloring.rs | 3 +- lib/frontend/src/ssa.rs | 91 +++++++-------------- 6 files changed, 139 insertions(+), 79 deletions(-) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index b92a6d7336..c2c8f5ea7a 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -720,6 +720,29 @@ pub trait CursorBase { self } + /// Rebuild this cursor positioned at the first insertion point for `ebb`. + /// This differs from `at_first_inst` in that it doesn't assume that any + /// instructions have been inserted into `ebb` yet. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// fn edit_func(func: &mut Function, ebb: Ebb) { + /// let mut pos = Cursor::new(&mut func.layout).at_first_insertion_point(ebb); + /// + /// // Use `pos`... + /// } + /// ``` + fn at_first_insertion_point(mut self, ebb: Ebb) -> Self + where + Self: Sized, + { + self.goto_first_insertion_point(ebb); + self + } + /// Rebuild this cursor positioned at the first instruction in `ebb`. /// /// This is intended to be used as a builder method: @@ -741,6 +764,69 @@ pub trait CursorBase { self } + /// Rebuild this cursor positioned at the last instruction in `ebb`. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// fn edit_func(func: &mut Function, ebb: Ebb) { + /// let mut pos = Cursor::new(&mut func.layout).at_last_inst(ebb); + /// + /// // Use `pos`... + /// } + /// ``` + fn at_last_inst(mut self, ebb: Ebb) -> Self + where + Self: Sized, + { + self.goto_last_inst(ebb); + self + } + + /// Rebuild this cursor positioned after `inst`. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// fn edit_func(func: &mut Function, inst: Inst) { + /// let mut pos = Cursor::new(&mut func.layout).after_inst(inst); + /// + /// // Use `pos`... + /// } + /// ``` + fn after_inst(mut self, inst: Inst) -> Self + where + Self: Sized, + { + self.goto_after_inst(inst); + self + } + + /// Rebuild this cursor positioned at the top of `ebb`. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// fn edit_func(func: &mut Function, ebb: Ebb) { + /// let mut pos = Cursor::new(&mut func.layout).at_top(ebb); + /// + /// // Use `pos`... + /// } + /// ``` + fn at_top(mut self, ebb: Ebb) -> Self + where + Self: Sized, + { + self.goto_top(ebb); + self + } + /// Rebuild this cursor positioned at the bottom of `ebb`. /// /// This is intended to be used as a builder method: @@ -781,6 +867,20 @@ pub trait CursorBase { } } + /// Go to the position after a specific instruction, which must be inserted + /// in the layout. New instructions will be inserted after `inst`. + fn goto_after_inst(&mut self, inst: Inst) { + debug_assert!(self.layout().inst_ebb(inst).is_some()); + let new_pos = if let Some(next) = self.layout().insts[inst].next.expand() { + CursorPosition::At(next) + } else { + CursorPosition::After(self.layout().inst_ebb(inst).expect( + "current instruction removed?", + )) + }; + self.set_position(new_pos); + } + /// Go to a specific instruction which must be inserted in the layout. /// New instructions will be inserted before `inst`. fn goto_inst(&mut self, inst: Inst) { @@ -910,8 +1010,7 @@ pub trait CursorBase { /// # use cretonne::ir::{Function, Ebb}; /// # use cretonne::ir::layout::{Cursor, CursorBase}; /// fn edit_ebb(func: &mut Function, ebb: Ebb) { - /// let mut cursor = Cursor::new(&mut func.layout); - /// cursor.goto_top(ebb); + /// let mut cursor = Cursor::new(&mut func.layout).at_top(ebb); /// while let Some(inst) = cursor.next_inst() { /// // Edit instructions... /// } @@ -979,8 +1078,7 @@ pub trait CursorBase { /// # use cretonne::ir::{Function, Ebb}; /// # use cretonne::ir::layout::{Cursor, CursorBase}; /// fn edit_ebb(func: &mut Function, ebb: Ebb) { - /// let mut cursor = Cursor::new(&mut func.layout); - /// cursor.goto_bottom(ebb); + /// let mut cursor = Cursor::new(&mut func.layout).at_bottom(ebb); /// while let Some(inst) = cursor.prev_inst() { /// // Edit instructions... /// } @@ -1357,8 +1455,7 @@ mod tests { verify(&mut layout, &[(e1, &[i1, i2, i0])]); // Test cursor positioning. - let mut cur = Cursor::new(&mut layout); - cur.goto_top(e1); + let mut cur = Cursor::new(&mut layout).at_top(e1); assert_eq!(cur.position(), CursorPosition::Before(e1)); assert_eq!(cur.prev_inst(), None); assert_eq!(cur.position(), CursorPosition::Before(e1)); diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index a95ec2cfba..b3e7b64658 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -62,8 +62,7 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { // Insert position for argument conversion code. // We want to insert instructions before the first instruction in the entry block. // If the entry block is empty, append instructions to it instead. - let mut pos = Cursor::new(&mut func.layout); - pos.goto_first_inst(entry); + let mut pos = Cursor::new(&mut func.layout).at_first_inst(entry); // Keep track of the argument types in the ABI-legalized signature. let abi_types = &func.signature.argument_types; diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index f47dc87241..6dbeb15d04 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -137,8 +137,7 @@ fn expand_cond_trap(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFl func.dfg.replace(inst).brz(arg, new_ebb, &[]); } - let mut pos = FuncCursor::new(func).at_inst(inst); - pos.next_inst(); + let mut pos = FuncCursor::new(func).after_inst(inst); pos.ins().trap(); pos.insert_ebb(new_ebb); diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 0a739f8f2d..c7b4ff72f6 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -33,14 +33,12 @@ pub fn do_licm( None => { let pre_header = create_pre_header(loop_analysis.loop_header(lp), func, cfg, domtree); - pos = Cursor::new(&mut func.layout); - pos.goto_last_inst(pre_header); + pos = Cursor::new(&mut func.layout).at_last_inst(pre_header); } // If there is a natural pre-header we insert new instructions just before the // related jumping instruction (which is not necessarily at the end). Some((_, last_inst)) => { - pos = Cursor::new(&mut func.layout); - pos.goto_inst(last_inst); + pos = Cursor::new(&mut func.layout).at_inst(last_inst); } }; // The last instruction of the pre-header is the termination instruction (usually @@ -82,8 +80,7 @@ fn create_pre_header( } } { - let mut pos = Cursor::new(&mut func.layout); - pos.goto_top(header); + let mut pos = Cursor::new(&mut func.layout).at_top(header); // Inserts the pre-header at the right place in the layout. pos.insert_ebb(pre_header); pos.next_inst(); diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 31fe44f03c..9a9122ba2f 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -148,8 +148,7 @@ impl<'a> Context<'a> { self.divert.clear(); // Now go through the instructions in `ebb` and color the values they define. - let mut pos = Cursor::new(&mut func.layout); - pos.goto_top(ebb); + let mut pos = Cursor::new(&mut func.layout).at_top(ebb); while let Some(inst) = pos.next_inst() { if let Some(constraints) = self.encinfo.operand_constraints(func.encodings[inst]) { self.visit_inst( diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 73d10e0494..e054ee511c 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -444,8 +444,7 @@ where if !layout.is_ebb_inserted(dest_ebb) { layout.append_ebb(dest_ebb) }; - let mut cur = Cursor::new(layout); - cur.goto_first_insertion_point(dest_ebb); + let mut cur = Cursor::new(layout).at_first_insertion_point(dest_ebb); let ty = dfg.value_type(temp_arg_val); let val = if ty.is_int() { dfg.ins(&mut cur).iconst(ty, 0) @@ -545,8 +544,7 @@ where *old_dest = PackedOption::from(middle_ebb); } } - let mut cur = Cursor::new(layout); - cur.goto_bottom(middle_ebb); + let mut cur = Cursor::new(layout).at_bottom(middle_ebb); let middle_jump_inst = dfg.ins(&mut cur).jump(dest_ebb, &[val]); self.def_var(var, val, block); Some((middle_ebb, block, middle_jump_inst)) @@ -627,14 +625,12 @@ mod tests { let x_ssa = { let cur = &mut Cursor::new(&mut func.layout); cur.insert_ebb(ebb0); - cur.goto_bottom(ebb0); func.dfg.ins(cur).iconst(I32, 1) }; ssa.def_var(x_var, x_ssa, block); let y_var = Variable(1); let y_ssa = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).iconst(I32, 2) }; ssa.def_var(y_var, y_ssa, block); @@ -679,8 +675,7 @@ mod tests { block, ).0; let z1_ssa = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).iadd(x_use1, y_use1) }; ssa.def_var(z_var, z1_ssa, block); @@ -712,8 +707,7 @@ mod tests { block, ).0; let z2_ssa = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).iadd(x_use2, z_use1) }; ssa.def_var(z_var, z2_ssa, block); @@ -758,8 +752,7 @@ mod tests { ssa.def_var(x_var, x_ssa, block0); let y_var = Variable(1); let y_ssa = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).iconst(I32, 2) }; ssa.def_var(y_var, y_ssa, block0); @@ -803,8 +796,7 @@ mod tests { block0, ).0; let z1_ssa = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).iadd(x_use1, y_use1) }; ssa.def_var(z_var, z1_ssa, block0); @@ -828,8 +820,7 @@ mod tests { block0, ).0; let jump_inst: Inst = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).brnz(y_use2, ebb1, &[]) }; let block1 = ssa.declare_ebb_body_block(block0); @@ -852,8 +843,7 @@ mod tests { ).0; assert_eq!(z_use1, z1_ssa); let z2_ssa = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).iadd(x_use2, z_use1) }; ssa.def_var(z_var, z2_ssa, block1); @@ -891,8 +881,7 @@ mod tests { ).0; assert_eq!(y_ssa, y_use3); let y2_ssa = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb1); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); func.dfg.ins(cur).iadd(x_use3, y_use3) }; ssa.def_var(y_var, y2_ssa, block2); @@ -952,8 +941,7 @@ mod tests { ); let y_var = Variable(1); let y1 = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).iconst(I32, 2) }; ssa.def_var(y_var, y1, block0); @@ -988,14 +976,12 @@ mod tests { ).0; assert_eq!(y2, y1); let z1 = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).iadd(x2, y2) }; ssa.def_var(z_var, z1, block0); let jump_ebb0_ebb1 = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).jump(ebb1, &[]) }; let block1 = ssa.declare_ebb_header_block(ebb1); @@ -1017,8 +1003,7 @@ mod tests { block1, ).0; let z3 = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb1); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); func.dfg.ins(cur).iadd(z2, y3) }; ssa.def_var(z_var, z3, block1); @@ -1032,8 +1017,7 @@ mod tests { ).0; assert_eq!(y4, y3); let jump_ebb1_ebb2 = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb1); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); func.dfg.ins(cur).brnz(y4, ebb2, &[]) }; let block2 = ssa.declare_ebb_body_block(block1); @@ -1055,8 +1039,7 @@ mod tests { block2, ).0; let z5 = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb1); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); func.dfg.ins(cur).isub(z4, x3) }; ssa.def_var(z_var, z5, block2); @@ -1070,8 +1053,7 @@ mod tests { ).0; assert_eq!(y5, y3); { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb1); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); func.dfg.ins(cur).return_(&[y5]) }; @@ -1097,14 +1079,12 @@ mod tests { ).0; assert_eq!(x4, x3); let y7 = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb2); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb2); func.dfg.ins(cur).isub(y6, x4) }; ssa.def_var(y_var, y7, block3); let jump_ebb2_ebb1 = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb2); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb2); func.dfg.ins(cur).jump(ebb1, &[]) }; @@ -1156,20 +1136,17 @@ mod tests { block0, ).0; let br_table = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).br_table(x1, jt) }; let block1 = ssa.declare_ebb_body_block(block0); let x3 = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).iconst(I32, 2) }; ssa.def_var(x_var, x3, block1); let jump_inst = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).jump(ebb1, &[]) }; let block2 = ssa.declare_ebb_header_block(ebb1); @@ -1185,13 +1162,11 @@ mod tests { block2, ).0; { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb1); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); func.dfg.ins(cur).iadd_imm(x4, 1) }; { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb1); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); func.dfg.ins(cur).return_(&[]) }; let flags = settings::Flags::new(&settings::builder()); @@ -1232,20 +1207,17 @@ mod tests { }; ssa.def_var(x_var, x1, block0); let y1 = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).iconst(I32, 1) }; ssa.def_var(y_var, y1, block0); let z1 = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).iconst(I32, 2) }; ssa.def_var(z_var, z1, block0); let jump_inst = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb0); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); func.dfg.ins(cur).jump(ebb1, &[]) }; let block1 = ssa.declare_ebb_header_block(ebb1); @@ -1269,8 +1241,7 @@ mod tests { ).0; assert_eq!(func.dfg.ebb_args(ebb1)[1], x2); let x3 = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb1); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); func.dfg.ins(cur).iadd(x2, z2) }; ssa.def_var(x_var, x3, block1); @@ -1292,14 +1263,12 @@ mod tests { ).0; assert_eq!(func.dfg.ebb_args(ebb1)[2], y3); let y4 = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb1); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); func.dfg.ins(cur).isub(y3, x4) }; ssa.def_var(y_var, y4, block1); let jump_inst = { - let cur = &mut Cursor::new(&mut func.layout); - cur.goto_bottom(ebb1); + let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); func.dfg.ins(cur).jump(ebb1, &[]) }; ssa.declare_ebb_predecessor(ebb1, block1, jump_inst); From 42af6d59bf248d5257af831dde8ad42a6264b9c0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 19 Sep 2017 13:25:16 -0700 Subject: [PATCH 1127/3084] Add a ScopedHashMap class, for use in a future GVN implementation. --- lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/scoped_hash_map.rs | 222 ++++++++++++++++++++++++++++ 2 files changed, 223 insertions(+) create mode 100644 lib/cretonne/src/scoped_hash_map.rs diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 6949f92f5f..8fb1c393d9 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -38,6 +38,7 @@ mod licm; mod partition_slice; mod predicates; mod ref_slice; +mod scoped_hash_map; mod simple_gvn; mod stack_layout; mod topo_order; diff --git a/lib/cretonne/src/scoped_hash_map.rs b/lib/cretonne/src/scoped_hash_map.rs new file mode 100644 index 0000000000..d5acf2ecb6 --- /dev/null +++ b/lib/cretonne/src/scoped_hash_map.rs @@ -0,0 +1,222 @@ +//! ScopedHashMap +//! +//! This module defines a struct `ScopedHashMap` which defines a `HashMap`-like +//! container that has a concept of scopes that can be entered and exited, such that +//! values inserted while inside a scope aren't visible outside the scope. + +use std::collections::{HashMap, hash_map}; +use std::hash::Hash; +use std::mem; + +struct Val { + value: V, + next_key: Option, + depth: usize, +} + +/// A view into an occupied entry in a `ScopedHashMap`. It is part of the `Entry` enum. +pub struct OccupiedEntry<'a, K: 'a, V: 'a> { + entry: hash_map::OccupiedEntry<'a, K, Val>, +} + +impl<'a, K, V> OccupiedEntry<'a, K, V> { + /// Gets a reference to the value in the entry. + pub fn get(&self) -> &V { + &self.entry.get().value + } +} + +/// A view into a vacant entry in a `ScopedHashMap`. It is part of the `Entry` enum. +pub struct VacantEntry<'a, K: 'a, V: 'a> { + entry: hash_map::VacantEntry<'a, K, Val>, + next_key: Option, + depth: usize, +} + +impl<'a, K, V> VacantEntry<'a, K, V> { + /// Sets the value of the entry with the `VacantEntry`'s key. + pub fn insert(self, value: V) { + self.entry.insert(Val { + value, + next_key: self.next_key, + depth: self.depth, + }); + } +} + +/// A view into a single entry in a map, which may either be vacant or occupied. +/// +/// This enum is constructed from the `entry` method on `ScopedHashMap`. +pub enum Entry<'a, K: 'a, V: 'a> { + Occupied(OccupiedEntry<'a, K, V>), + Vacant(VacantEntry<'a, K, V>), +} + +/// A wrapper around a `HashMap` which adds the concept of scopes. Items inserted +/// within a scope are removed when the scope is exited. +/// +/// Shadowing, where one scope has entries with the same keys as a containing scope, +/// is not supported in this implementation. +pub struct ScopedHashMap { + map: HashMap>, + last_insert: Option, + current_depth: usize, +} + +impl ScopedHashMap +where + K: PartialEq + Eq + Hash + Clone, +{ + /// Creates an empty `ScopedHashMap`. + pub fn new() -> Self { + Self { + map: HashMap::new(), + last_insert: None, + current_depth: 0, + } + } + + /// Similar to `HashMap::entry`, gets the given key's corresponding entry in the map for + /// in-place manipulation. + pub fn entry(&mut self, key: K) -> Entry { + use self::hash_map::Entry::*; + match self.map.entry(key) { + Occupied(entry) => Entry::Occupied(OccupiedEntry { entry }), + Vacant(entry) => { + let clone_key = entry.key().clone(); + Entry::Vacant(VacantEntry { + entry, + next_key: mem::replace(&mut self.last_insert, Some(clone_key)), + depth: self.current_depth, + }) + } + } + } + + /// Enter a new scope. + pub fn increment_depth(&mut self) { + // Increment the depth. + self.current_depth = self.current_depth.checked_add(1).unwrap(); + } + + /// Exit the current scope. + pub fn decrement_depth(&mut self) { + // Remove all elements inserted at the current depth. + while let Some(key) = self.last_insert.clone() { + use self::hash_map::Entry::*; + match self.map.entry(key) { + Occupied(entry) => { + if entry.get().depth != self.current_depth { + break; + } + self.last_insert = entry.remove_entry().1.next_key; + } + Vacant(_) => panic!(), + } + } + + // Decrement the depth. + self.current_depth = self.current_depth.checked_sub(1).unwrap(); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn basic() { + let mut map: ScopedHashMap = ScopedHashMap::new(); + + match map.entry(0) { + Entry::Occupied(_entry) => panic!(), + Entry::Vacant(entry) => entry.insert(1), + } + match map.entry(2) { + Entry::Occupied(_entry) => panic!(), + Entry::Vacant(entry) => entry.insert(8), + } + match map.entry(2) { + Entry::Occupied(entry) => assert!(*entry.get() == 8), + Entry::Vacant(_entry) => panic!(), + } + map.increment_depth(); + match map.entry(2) { + Entry::Occupied(entry) => assert!(*entry.get() == 8), + Entry::Vacant(_entry) => panic!(), + } + match map.entry(1) { + Entry::Occupied(_entry) => panic!(), + Entry::Vacant(entry) => entry.insert(3), + } + match map.entry(1) { + Entry::Occupied(entry) => assert!(*entry.get() == 3), + Entry::Vacant(_entry) => panic!(), + } + match map.entry(0) { + Entry::Occupied(entry) => assert!(*entry.get() == 1), + Entry::Vacant(_entry) => panic!(), + } + match map.entry(2) { + Entry::Occupied(entry) => assert!(*entry.get() == 8), + Entry::Vacant(_entry) => panic!(), + } + map.decrement_depth(); + match map.entry(0) { + Entry::Occupied(entry) => assert!(*entry.get() == 1), + Entry::Vacant(_entry) => panic!(), + } + match map.entry(2) { + Entry::Occupied(entry) => assert!(*entry.get() == 8), + Entry::Vacant(_entry) => panic!(), + } + map.increment_depth(); + match map.entry(2) { + Entry::Occupied(entry) => assert!(*entry.get() == 8), + Entry::Vacant(_entry) => panic!(), + } + match map.entry(1) { + Entry::Occupied(_entry) => panic!(), + Entry::Vacant(entry) => entry.insert(4), + } + match map.entry(1) { + Entry::Occupied(entry) => assert!(*entry.get() == 4), + Entry::Vacant(_entry) => panic!(), + } + match map.entry(2) { + Entry::Occupied(entry) => assert!(*entry.get() == 8), + Entry::Vacant(_entry) => panic!(), + } + map.decrement_depth(); + map.increment_depth(); + map.increment_depth(); + map.increment_depth(); + match map.entry(2) { + Entry::Occupied(entry) => assert!(*entry.get() == 8), + Entry::Vacant(_entry) => panic!(), + } + match map.entry(1) { + Entry::Occupied(_entry) => panic!(), + Entry::Vacant(entry) => entry.insert(5), + } + match map.entry(1) { + Entry::Occupied(entry) => assert!(*entry.get() == 5), + Entry::Vacant(_entry) => panic!(), + } + match map.entry(2) { + Entry::Occupied(entry) => assert!(*entry.get() == 8), + Entry::Vacant(_entry) => panic!(), + } + map.decrement_depth(); + map.decrement_depth(); + map.decrement_depth(); + match map.entry(2) { + Entry::Occupied(entry) => assert!(*entry.get() == 8), + Entry::Vacant(_entry) => panic!(), + } + match map.entry(1) { + Entry::Occupied(_entry) => panic!(), + Entry::Vacant(entry) => entry.insert(3), + } + } +} From 6710216e96a78bd06b049cbcd0a64d5a110af726 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 20 Sep 2017 12:38:13 -0700 Subject: [PATCH 1128/3084] Add next_inst and prev_inst helper functions to Layout. --- lib/cretonne/src/ir/layout.rs | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index c2c8f5ea7a..2139848382 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -482,6 +482,16 @@ impl Layout { self.ebbs[ebb].last_inst.into() } + /// Fetch the instruction following `inst`. + pub fn next_inst(&self, inst: Inst) -> Option { + self.insts[inst].next.expand() + } + + /// Fetch the instruction preceding `inst`. + pub fn prev_inst(&self, inst: Inst) -> Option { + self.insts[inst].prev.expand() + } + /// Insert `inst` before the instruction `before` in the same EBB. pub fn insert_inst(&mut self, inst: Inst, before: Inst) { assert_eq!(self.inst_ebb(inst), None); From ce94a3fa3918373f742e23030ba1232cc252f573 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 19 Sep 2017 21:05:53 -0700 Subject: [PATCH 1129/3084] Use ScopedHashMap in simple_gvn. This avoids effectively ending up with most of a function body stored in the hash map at once by removing elements promptly when they go out of scope. --- cranelift/filetests/simple_gvn/scopes.cton | 76 ++++++++++++++++++++++ lib/cretonne/src/simple_gvn.rs | 49 ++++++++++---- 2 files changed, 111 insertions(+), 14 deletions(-) create mode 100644 cranelift/filetests/simple_gvn/scopes.cton diff --git a/cranelift/filetests/simple_gvn/scopes.cton b/cranelift/filetests/simple_gvn/scopes.cton new file mode 100644 index 0000000000..85ea5583c2 --- /dev/null +++ b/cranelift/filetests/simple_gvn/scopes.cton @@ -0,0 +1,76 @@ +test simple-gvn + +function %two_diamonds(i32, i32, i32, i32, i32) { +ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32): + v5 = iconst.i32 16 + ; check: v5 = iconst.i32 16 + brz v0, ebb1 + v6 = iconst.i32 17 + ; check: v6 = iconst.i32 17 + v7 = iconst.i32 16 + ; not: v7 = iconst.i32 16 + jump ebb2 + +ebb1: + v8 = iconst.i32 18 + ; check: v8 = iconst.i32 18 + v9 = iconst.i32 17 + ; check: v9 = iconst.i32 17 + v10 = iconst.i32 16 + ; not: v10 = iconst.i32 16 + jump ebb2 + +ebb2: + v11 = iconst.i32 19 + ; check: v11 = iconst.i32 19 + v12 = iconst.i32 18 + ; check: v12 = iconst.i32 18 + v13 = iconst.i32 17 + ; check: v13 = iconst.i32 17 + v14 = iconst.i32 16 + ; not: v14 = iconst.i32 16 + brz v1, ebb3 + v15 = iconst.i32 20 + ; check: v15 = iconst.i32 20 + v16 = iconst.i32 19 + ; not: v16 = iconst.i32 19 + v17 = iconst.i32 18 + ; not: v17 = iconst.i32 18 + v18 = iconst.i32 17 + ; not: v18 = iconst.i32 17 + v19 = iconst.i32 16 + ; not: v19 = iconst.i32 16 + jump ebb4 + +ebb3: + v20 = iconst.i32 21 + ; check: v20 = iconst.i32 21 + v21 = iconst.i32 20 + ; check: v21 = iconst.i32 20 + v22 = iconst.i32 19 + ; not: v22 = iconst.i32 19 + v23 = iconst.i32 18 + ; not: v23 = iconst.i32 18 + v24 = iconst.i32 17 + ; not: v24 = iconst.i32 17 + v25 = iconst.i32 16 + ; not: v25 = iconst.i32 16 + jump ebb4 + +ebb4: + v26 = iconst.i32 22 + ; check: v26 = iconst.i32 22 + v27 = iconst.i32 21 + ; check: v27 = iconst.i32 21 + v28 = iconst.i32 20 + ; check: v28 = iconst.i32 20 + v29 = iconst.i32 19 + ; not: v29 = iconst.i32 19 + v30 = iconst.i32 18 + ; not: v30 = iconst.i32 18 + v31 = iconst.i32 17 + ; not: v31 = iconst.i32 17 + v32 = iconst.i32 16 + ; not: v32 = iconst.i32 16 + return +} diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index 20e2298440..c50f70a3fa 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -3,7 +3,7 @@ use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; use ir::{Cursor, CursorBase, InstructionData, Function, Inst, Opcode, Type}; -use std::collections::HashMap; +use scoped_hash_map::ScopedHashMap; /// Test whether the given opcode is unsafe to even consider for GVN. fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { @@ -18,19 +18,40 @@ pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph, domtree: & debug_assert!(cfg.is_valid()); debug_assert!(domtree.is_valid()); - let mut visible_values: HashMap<(InstructionData, Type), Inst> = HashMap::new(); + let mut visible_values: ScopedHashMap<(InstructionData, Type), Inst> = ScopedHashMap::new(); + let mut scope_stack: Vec = Vec::new(); // Visit EBBs in a reverse post-order. let mut pos = Cursor::new(&mut func.layout); for &ebb in domtree.cfg_postorder().iter().rev() { - pos.goto_top(ebb); + // Pop any scopes that we just exited. + loop { + if let Some(current) = scope_stack.last() { + if domtree.dominates(*current, ebb, &pos.layout) { + break; + } + } else { + break; + } + scope_stack.pop(); + visible_values.decrement_depth(); + } + // Push a scope for the current block. + scope_stack.push(pos.layout.first_inst(ebb).unwrap()); + visible_values.increment_depth(); + + pos.goto_top(ebb); while let Some(inst) = pos.next_inst() { // Resolve aliases, particularly aliases we created earlier. func.dfg.resolve_aliases_in_arguments(inst); let opcode = func.dfg[inst].opcode(); + if opcode.is_branch() && !opcode.is_terminator() { + scope_stack.push(pos.layout.next_inst(inst).unwrap()); + visible_values.increment_depth(); + } if trivially_unsafe_for_gvn(opcode) { continue; } @@ -38,19 +59,19 @@ pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph, domtree: & let ctrl_typevar = func.dfg.ctrl_typevar(inst); let key = (func.dfg[inst].clone(), ctrl_typevar); let entry = visible_values.entry(key); - use std::collections::hash_map::Entry::*; + use scoped_hash_map::Entry::*; match entry { - Occupied(mut entry) => { - if domtree.dominates(*entry.get(), inst, pos.layout) { - func.dfg.replace_with_aliases(inst, *entry.get()); - pos.remove_inst_and_step_back(); - } else { - // The prior instruction doesn't dominate inst, so it - // won't dominate any subsequent instructions we'll - // visit, so just replace it. - *entry.get_mut() = inst; - continue; + Occupied(entry) => { + debug_assert!(domtree.dominates(*entry.get(), inst, pos.layout)); + // If the redundant instruction is representing the current + // scope, pick a new representative. + let old = scope_stack.last_mut().unwrap(); + if *old == inst { + *old = pos.layout.next_inst(inst).unwrap(); } + // Replace the redundant instruction and remove it. + func.dfg.replace_with_aliases(inst, *entry.get()); + pos.remove_inst_and_step_back(); } Vacant(entry) => { entry.insert(inst); From d3f3fdf5afb4338eca413bdadfc0bdf9d9b7c298 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 20 Sep 2017 15:35:26 -0700 Subject: [PATCH 1130/3084] Remove a redundant .gitignore file. --- lib/wasm/.gitignore | 3 --- 1 file changed, 3 deletions(-) delete mode 100644 lib/wasm/.gitignore diff --git a/lib/wasm/.gitignore b/lib/wasm/.gitignore deleted file mode 100644 index 4308d82204..0000000000 --- a/lib/wasm/.gitignore +++ /dev/null @@ -1,3 +0,0 @@ -target/ -**/*.rs.bk -Cargo.lock From 0f21fd342af6f0595179dc6bc0adca18221e9fb4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 20 Sep 2017 14:21:41 -0700 Subject: [PATCH 1131/3084] Remove the HeapLoad/HeapStore instruction formats. These formats are not used any longer after the heap_load and heap_store instructions were replaced by heap_addr. Also drop the Uoffset32 immediate operand type which isn't used either. --- lib/cretonne/meta/base/formats.py | 6 +- lib/cretonne/meta/base/immediates.py | 9 --- lib/cretonne/src/ir/immediates.rs | 91 +--------------------------- lib/cretonne/src/ir/instructions.rs | 12 +--- lib/cretonne/src/verifier/mod.rs | 2 - lib/cretonne/src/write.rs | 2 - lib/reader/src/parser.rs | 41 +------------ 7 files changed, 5 insertions(+), 158 deletions(-) diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 783539e105..9d6315ddcf 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -8,8 +8,7 @@ in this module. from __future__ import absolute_import from cdsl.formats import InstructionFormat from cdsl.operands import VALUE, VARIABLE_ARGS -from .immediates import imm64, uimm8, uimm32, ieee32, ieee64 -from .immediates import offset32, uoffset32 +from .immediates import imm64, uimm8, uimm32, ieee32, ieee64, offset32 from .immediates import boolean, intcc, floatcc, memflags, regunit from . import entities from .entities import ebb, sig_ref, func_ref, stack_slot, heap @@ -58,9 +57,6 @@ StackLoad = InstructionFormat(stack_slot, offset32) StackStore = InstructionFormat(VALUE, stack_slot, offset32) # Accessing a WebAssembly heap. -# TODO: Add a reference to a `heap` declared in the preamble. -HeapLoad = InstructionFormat(VALUE, uoffset32) -HeapStore = InstructionFormat(VALUE, VALUE, uoffset32) HeapAddr = InstructionFormat(heap, VALUE, uimm32) RegMove = InstructionFormat(VALUE, ('src', regunit), ('dst', regunit)) diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index 048a82dd12..dd4a5c7760 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -29,15 +29,6 @@ offset32 = ImmediateKind( 'A 32-bit immediate signed offset.', default_member='offset') -#: A 32-bit immediate unsigned offset. -#: -#: This is used to represent an immediate address offset in WebAssembly memory -#: instructions. -uoffset32 = ImmediateKind( - 'uoffset32', - 'A 32-bit immediate unsigned offset.', - default_member='offset') - #: A 32-bit immediate floating point operand. #: #: IEEE 754-2008 binary32 interchange format. diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 51120344cd..bf9d4bea72 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -269,80 +269,15 @@ impl FromStr for Offset32 { } } -/// 32-bit unsigned immediate offset. -/// -/// This is used to encode an immediate offset for WebAssembly heap_load/heap_store instructions. -#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)] -pub struct Uoffset32(u32); - -impl Uoffset32 { - /// Create a new `Uoffset32` representing the number `x`. - pub fn new(x: u32) -> Uoffset32 { - Uoffset32(x) - } -} - -impl Into for Uoffset32 { - fn into(self) -> u32 { - self.0 - } -} - -impl Into for Uoffset32 { - fn into(self) -> i64 { - i64::from(self.0) - } -} - -impl From for Uoffset32 { - fn from(x: u32) -> Self { - Uoffset32(x) - } -} - -impl Display for Uoffset32 { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - // 0 displays as an empty offset. - if self.0 == 0 { - return Ok(()); - } - - // Always include a sign. - if self.0 < 10_000 { - write!(f, "+{}", self.0) - } else { - write!(f, "+")?; - write_hex(i64::from(self.0), f) - } - - } -} - -impl FromStr for Uoffset32 { - type Err = &'static str; - - // Parse a decimal or hexadecimal `Uoffset32`, formatted as above. - fn from_str(s: &str) -> Result { - if !s.starts_with('+') { - return Err("Unsigned offset must begin with '+' sign"); - } - parse_i64(s).and_then(|x| if 0 <= x && x <= i64::from(u32::MAX) { - Ok(Uoffset32::new(x as u32)) - } else { - Err("Offset out of range") - }) - } -} - /// An IEEE binary32 immediate floating point value, represented as a u32 -/// containing the bitpattern. +/// containing the bit pattern. /// /// All bit patterns are allowed. #[derive(Copy, Clone, Debug, Eq, PartialEq, Hash)] pub struct Ieee32(u32); /// An IEEE binary64 immediate floating point value, represented as a u64 -/// containing the bitpattern. +/// containing the bit pattern. /// /// All bit patterns are allowed. #[derive(Copy, Clone, Debug, Eq, PartialEq, Hash)] @@ -772,28 +707,6 @@ mod tests { parse_err::("+0x8000_0000", "Offset out of range"); } - #[test] - fn format_uoffset32() { - assert_eq!(Uoffset32(0).to_string(), ""); - assert_eq!(Uoffset32(1).to_string(), "+1"); - assert_eq!(Uoffset32(9999).to_string(), "+9999"); - assert_eq!(Uoffset32(10000).to_string(), "+0x2710"); - assert_eq!(Uoffset32(0xffff).to_string(), "+0xffff"); - assert_eq!(Uoffset32(0x10000).to_string(), "+0x0001_0000"); - } - - #[test] - fn parse_uoffset32() { - parse_ok::("+0", ""); - parse_ok::("+1", "+1"); - parse_ok::("+0x0", ""); - parse_ok::("+0xf", "+15"); - parse_ok::("+0x8000_0000", "+0x8000_0000"); - parse_ok::("+0xffff_ffff", "+0xffff_ffff"); - - parse_err::("+0x1_0000_0000", "Offset out of range"); - } - #[test] fn format_ieee32() { assert_eq!(Ieee32::with_float(0.0).to_string(), "0.0"); diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 9a9301e7db..6dd5eafde7 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -12,7 +12,7 @@ use std::ops::{Deref, DerefMut}; use ir; use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef, StackSlot, MemFlags}; -use ir::immediates::{Imm64, Uimm8, Uimm32, Ieee32, Ieee64, Offset32, Uoffset32}; +use ir::immediates::{Imm64, Uimm8, Uimm32, Ieee32, Ieee64, Offset32}; use ir::condcodes::*; use ir::types; use isa::RegUnit; @@ -190,16 +190,6 @@ pub enum InstructionData { stack_slot: StackSlot, offset: Offset32, }, - HeapLoad { - opcode: Opcode, - arg: Value, - offset: Uoffset32, - }, - HeapStore { - opcode: Opcode, - args: [Value; 2], - offset: Uoffset32, - }, HeapAddr { opcode: Opcode, heap: ir::Heap, diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 7fcb4b508f..f387c0d295 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -335,8 +335,6 @@ impl<'a> Verifier<'a> { IntCompare { .. } | IntCompareImm { .. } | FloatCompare { .. } | - HeapLoad { .. } | - HeapStore { .. } | Load { .. } | Store { .. } | RegMove { .. } => {} diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 56d1eb4652..f225e0aa7d 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -354,8 +354,6 @@ pub fn write_operands( offset, .. } => write!(w, " {}, {}{}", arg, stack_slot, offset), - HeapLoad { arg, offset, .. } => write!(w, " {}{}", arg, offset), - HeapStore { args, offset, .. } => write!(w, " {}, {}{}", args[0], args[1], offset), HeapAddr { heap, arg, imm, .. } => write!(w, " {}, {}, {}", heap, arg, imm), Load { flags, arg, offset, .. } => write!(w, "{} {}{}", flags, arg, offset), Store { diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index d9dd334964..94cf785fb4 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -14,7 +14,7 @@ use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, CallConv, S ExtFuncData, SigRef, FuncRef, StackSlot, ValueLoc, ArgumentLoc, MemFlags, GlobalVar, GlobalVarData, Heap, HeapData, HeapStyle, HeapBase}; use cretonne::ir::types::VOID; -use cretonne::ir::immediates::{Imm64, Uimm32, Offset32, Uoffset32, Ieee32, Ieee64}; +use cretonne::ir::immediates::{Imm64, Uimm32, Offset32, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs}; use cretonne::isa::{self, TargetIsa, Encoding, RegUnit}; @@ -585,22 +585,6 @@ impl<'a> Parser<'a> { } } - // Match and consume an optional uoffset32 immediate. - // - // Note that this will match an empty string as an empty offset, and that if an offset is - // present, it must contain a `+` sign. - fn optional_uoffset32(&mut self) -> Result { - if let Some(Token::Integer(text)) = self.token() { - self.consume(); - // Lexer just gives us raw text that looks like an integer. - // Parse it as a `Uoffset32` to check for overflow and other issues. - text.parse().map_err(|e| self.error(e)) - } else { - // A uoffset32 operand can be absent. - Ok(Uoffset32::new(0)) - } - } - // Match and consume an Ieee32 immediate. fn match_ieee32(&mut self, err_msg: &str) -> Result { if let Some(Token::Float(text)) = self.token() { @@ -2141,29 +2125,6 @@ impl<'a> Parser<'a> { offset, } } - InstructionFormat::HeapLoad => { - let addr = self.match_value("expected SSA value address")?; - let offset = self.optional_uoffset32()?; - InstructionData::HeapLoad { - opcode, - arg: addr, - offset, - } - } - InstructionFormat::HeapStore => { - let arg = self.match_value("expected SSA value operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; - let addr = self.match_value("expected SSA value address")?; - let offset = self.optional_uoffset32()?; - InstructionData::HeapStore { - opcode, - args: [arg, addr], - offset, - } - } InstructionFormat::HeapAddr => { let heap = self.match_heap("expected heap identifier").and_then(|h| { ctx.get_heap(h, &self.loc) From e8723be33f9367718498395c1710b1c953e1782c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 20 Sep 2017 13:58:57 -0700 Subject: [PATCH 1132/3084] Add trap codes to the Cretonne IL. The trap and trapz/trapnz instructions now take a trap code immediate operand which indicates the reason for trapping. --- cranelift/filetests/cfg/traps_early.cton | 2 +- cranelift/filetests/cfg/unused_node.cton | 2 +- cranelift/filetests/isa/intel/binary32.cton | 2 +- cranelift/filetests/isa/intel/binary64.cton | 2 +- .../filetests/isa/intel/legalize-custom.cton | 16 ++-- .../filetests/isa/intel/legalize-memory.cton | 2 +- cranelift/filetests/parser/branch.cton | 16 ++-- .../parser/instruction_encoding.cton | 4 +- cranelift/filetests/parser/rewrite.cton | 4 +- cranelift/filetests/parser/tiny.cton | 4 +- .../filetests/verifier/return_at_end.cton | 6 +- cranelift/filetests/wasm/control.cton | 2 +- cranelift/tests/cfg_traversal.rs | 10 +-- lib/cretonne/meta/base/formats.py | 5 +- lib/cretonne/meta/base/immediates.py | 16 ++++ lib/cretonne/meta/base/instructions.py | 8 +- lib/cretonne/meta/isa/intel/encodings.py | 4 +- lib/cretonne/meta/isa/intel/recipes.py | 6 +- lib/cretonne/src/dominator_tree.rs | 4 +- lib/cretonne/src/ir/instructions.rs | 6 ++ lib/cretonne/src/ir/mod.rs | 2 + lib/cretonne/src/ir/trapcode.rs | 89 +++++++++++++++++++ lib/cretonne/src/legalizer/heap.rs | 8 +- lib/cretonne/src/legalizer/mod.rs | 8 +- lib/cretonne/src/verifier/mod.rs | 4 +- lib/cretonne/src/write.rs | 2 + lib/reader/src/parser.rs | 23 +++-- lib/wasm/src/code_translator.rs | 4 +- 28 files changed, 199 insertions(+), 62 deletions(-) create mode 100644 lib/cretonne/src/ir/trapcode.rs diff --git a/cranelift/filetests/cfg/traps_early.cton b/cranelift/filetests/cfg/traps_early.cton index 814e251f51..e12c3c65ee 100644 --- a/cranelift/filetests/cfg/traps_early.cton +++ b/cranelift/filetests/cfg/traps_early.cton @@ -7,7 +7,7 @@ function %nonsense(i32) { ; check: digraph %nonsense { ebb0(v1: i32): - trap ; error: terminator instruction was encountered before the end + trap user0 ; error: terminator instruction was encountered before the end brnz v1, ebb2 ; unordered: ebb0:inst1 -> ebb2 jump ebb1 ; unordered: ebb0:inst2 -> ebb1 diff --git a/cranelift/filetests/cfg/unused_node.cton b/cranelift/filetests/cfg/unused_node.cton index cbde9757bc..d339e5ea83 100644 --- a/cranelift/filetests/cfg/unused_node.cton +++ b/cranelift/filetests/cfg/unused_node.cton @@ -9,7 +9,7 @@ function %not_reached(i32) -> i32 { ebb0(v0: i32): brnz v0, ebb2 ; unordered: ebb0:inst0 -> ebb2 - trap + trap user0 ebb1: v1 = iconst.i32 1 diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index b531dfdfd7..16f8131cef 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -369,5 +369,5 @@ ebb1: ; asm: ebb2: ebb2: - trap ; bin: 0f 0b + trap user0 ; bin: 0f 0b } diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 680da4273e..e67f630753 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -861,5 +861,5 @@ ebb0: ; asm: movl %r10d, %ecx [-,%rcx] v32 = uextend.i64 v13 ; bin: 44 89 d1 - trap ; bin: 0f 0b + trap user0 ; bin: 0f 0b } diff --git a/cranelift/filetests/isa/intel/legalize-custom.cton b/cranelift/filetests/isa/intel/legalize-custom.cton index 4c8488173b..39b92c6306 100644 --- a/cranelift/filetests/isa/intel/legalize-custom.cton +++ b/cranelift/filetests/isa/intel/legalize-custom.cton @@ -9,22 +9,22 @@ isa intel function %cond_trap(i32) { ebb0(v1: i32): - trapz v1 + trapz v1, user67 return ; check: $ebb0($v1: i32): ; nextln: brnz $v1, $(new=$EBB) - ; nextln: trap + ; nextln: trap user67 ; check: $new: ; nextln: return } function %cond_trap2(i32) { ebb0(v1: i32): - trapnz v1 + trapnz v1, int_ovf return ; check: $ebb0($v1: i32): ; nextln: brz $v1, $(new=$EBB) - ; nextln: trap + ; nextln: trap int_ovf ; check: $new: ; nextln: return } @@ -32,11 +32,11 @@ ebb0(v1: i32): function %cond_trap_b1(i32) { ebb0(v1: i32): v2 = icmp_imm eq v1, 6 - trapz v2 + trapz v2, user7 return ; check: $ebb0($v1: i32): ; check: brnz $v2, $(new=$EBB) - ; nextln: trap + ; nextln: trap user7 ; check: $new: ; nextln: return } @@ -44,11 +44,11 @@ ebb0(v1: i32): function %cond_trap2_b1(i32) { ebb0(v1: i32): v2 = icmp_imm eq v1, 6 - trapnz v2 + trapnz v2, user9 return ; check: $ebb0($v1: i32): ; check: brz $v2, $(new=$EBB) - ; nextln: trap + ; nextln: trap user9 ; check: $new: ; nextln: return } diff --git a/cranelift/filetests/isa/intel/legalize-memory.cton b/cranelift/filetests/isa/intel/legalize-memory.cton index 5db903aa52..c7d5ed3617 100644 --- a/cranelift/filetests/isa/intel/legalize-memory.cton +++ b/cranelift/filetests/isa/intel/legalize-memory.cton @@ -64,7 +64,7 @@ ebb0(v0: i32, v999: i64): ; Boundscheck code ; check: $(oob=$V) = icmp ; nextln: brz $oob, $(ok=$EBB) - ; nextln: trap + ; nextln: trap heap_oob ; check: $ok: ; Checks here are assuming that no pipehole opts fold the load offsets. ; nextln: $(xoff=$V) = uextend.i64 $v0 diff --git a/cranelift/filetests/parser/branch.cton b/cranelift/filetests/parser/branch.cton index 283dd9a03a..7b3ca787b6 100644 --- a/cranelift/filetests/parser/branch.cton +++ b/cranelift/filetests/parser/branch.cton @@ -86,13 +86,13 @@ function %jumptable(i32) { ebb10(v3: i32): br_table v3, jt2 - trap + trap user1 ebb20: - trap + trap user2 ebb30: - trap + trap user3 ebb40: - trap + trap user4 } ; sameln: function %jumptable(i32) native { ; nextln: jt0 = jump_table 0 @@ -100,14 +100,14 @@ ebb40: ; nextln: ; nextln: ebb0($v3: i32): ; nextln: br_table $v3, jt1 -; nextln: trap +; nextln: trap user1 ; nextln: ; nextln: ebb1: -; nextln: trap +; nextln: trap user2 ; nextln: ; nextln: ebb2: -; nextln: trap +; nextln: trap user3 ; nextln: ; nextln: ebb3: -; nextln: trap +; nextln: trap user4 ; nextln: } diff --git a/cranelift/filetests/parser/instruction_encoding.cton b/cranelift/filetests/parser/instruction_encoding.cton index c9c668dc3f..d6dda30da5 100644 --- a/cranelift/filetests/parser/instruction_encoding.cton +++ b/cranelift/filetests/parser/instruction_encoding.cton @@ -7,7 +7,7 @@ isa riscv function %foo(i32, i32) { ebb1(v0: i32 [%x8], v1: i32): [-,-] v2 = iadd v0, v1 - [-] trap + [-] trap heap_oob [R#1234, %x5, %x11] v6, v7 = iadd_cout v2, v0 [Rshamt#beef, %x25] v8 = ishl_imm v6, 2 v9 = iadd v8, v7 @@ -16,7 +16,7 @@ ebb1(v0: i32 [%x8], v1: i32): ; sameln: function %foo(i32, i32) native { ; nextln: $ebb1($v0: i32 [%x8], $v1: i32): ; nextln: [-,-]$WS $v2 = iadd $v0, $v1 -; nextln: [-]$WS trap +; nextln: [-]$WS trap heap_oob ; nextln: [R#1234,%x5,%x11]$WS $v6, $v7 = iadd_cout $v2, $v0 ; nextln: [Rshamt#beef,%x25]$WS $v8 = ishl_imm $v6, 2 ; nextln: [-,-]$WS $v9 = iadd $v8, $v7 diff --git a/cranelift/filetests/parser/rewrite.cton b/cranelift/filetests/parser/rewrite.cton index c6a04a9e9c..1601002b27 100644 --- a/cranelift/filetests/parser/rewrite.cton +++ b/cranelift/filetests/parser/rewrite.cton @@ -13,13 +13,13 @@ function %defs() { ebb100(v20: i32): v1000 = iconst.i32x8 5 v9200 = f64const 0x4.0p0 - trap + trap user4 } ; sameln: function %defs() native { ; nextln: $ebb100($v20: i32): ; nextln: $v1000 = iconst.i32x8 5 ; nextln: $v9200 = f64const 0x1.0000000000000p2 -; nextln: trap +; nextln: trap user4 ; nextln: } ; Using values. diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 56dfc53973..e538b5acaa 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -3,11 +3,11 @@ test cat ; The smallest possible function. function %minimal() { ebb0: - trap + trap user0 } ; sameln: function %minimal() native { ; nextln: ebb0: -; nextln: trap +; nextln: trap user0 ; nextln: } ; Create and use values. diff --git a/cranelift/filetests/verifier/return_at_end.cton b/cranelift/filetests/verifier/return_at_end.cton index 7acc8b8fad..fcbae57070 100644 --- a/cranelift/filetests/verifier/return_at_end.cton +++ b/cranelift/filetests/verifier/return_at_end.cton @@ -4,10 +4,10 @@ set return_at_end function %ok(i32) { ebb0(v0: i32): brnz v0, ebb1 - trap + trap int_divz ebb1: - trapz v0 + trapz v0, user5 return } @@ -17,6 +17,6 @@ ebb0(v0: i32): return ; error: Internal return not allowed ebb1: - trapz v0 + trapz v0, user6 return } diff --git a/cranelift/filetests/wasm/control.cton b/cranelift/filetests/wasm/control.cton index c8e37a536f..9d46a3577b 100644 --- a/cranelift/filetests/wasm/control.cton +++ b/cranelift/filetests/wasm/control.cton @@ -46,5 +46,5 @@ ebb1(v2: i32): function %undefined() { ebb0: - trap + trap user0 } diff --git a/cranelift/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs index 9154349bc7..90fb900dfe 100644 --- a/cranelift/tests/cfg_traversal.rs +++ b/cranelift/tests/cfg_traversal.rs @@ -44,11 +44,11 @@ fn simple_traversal() { brz v4, ebb4 jump ebb5 ebb3: - trap + trap user0 ebb4: - trap + trap user0 ebb5: - trap + trap user0 } ", vec![0, 1, 3, 2, 4, 5], @@ -123,7 +123,7 @@ fn loops_three() { jump ebb6 ebb5: brz v0, ebb4 - trap + trap user0 ebb6: jump ebb7 ebb7: @@ -152,7 +152,7 @@ fn back_edge_one() { brnz v0, ebb0 return ebb4: - trap + trap user0 } ", vec![0, 1, 3, 2, 4], diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 9d6315ddcf..1746217958 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -9,7 +9,7 @@ from __future__ import absolute_import from cdsl.formats import InstructionFormat from cdsl.operands import VALUE, VARIABLE_ARGS from .immediates import imm64, uimm8, uimm32, ieee32, ieee64, offset32 -from .immediates import boolean, intcc, floatcc, memflags, regunit +from .immediates import boolean, intcc, floatcc, memflags, regunit, trapcode from . import entities from .entities import ebb, sig_ref, func_ref, stack_slot, heap @@ -61,5 +61,8 @@ HeapAddr = InstructionFormat(heap, VALUE, uimm32) RegMove = InstructionFormat(VALUE, ('src', regunit), ('dst', regunit)) +Trap = InstructionFormat(trapcode) +CondTrap = InstructionFormat(VALUE, trapcode) + # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) diff --git a/lib/cretonne/meta/base/immediates.py b/lib/cretonne/meta/base/immediates.py index dd4a5c7760..53293acc77 100644 --- a/lib/cretonne/meta/base/immediates.py +++ b/lib/cretonne/meta/base/immediates.py @@ -105,3 +105,19 @@ regunit = ImmediateKind( 'regunit', 'A register unit in the target ISA', rust_type='isa::RegUnit') + +#: A trap code indicating the reason for trapping. +#: +#: The Rust enum type also has a `User(u16)` variant for user-provided trap +#: codes. +trapcode = ImmediateKind( + 'trapcode', + 'A trap reason code.', + default_member='code', + rust_type='ir::TrapCode', + values={ + "stk_ovf": 'StackOverflow', + "heap_oob": 'HeapOutOfBounds', + "int_ovf": 'IntegerOverflow', + "int_divz": 'IntegerDivisionByZero', + }) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 735ec742bc..f49a532a3f 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -11,6 +11,7 @@ from cdsl.instructions import Instruction, InstructionGroup from base.types import f32, f64, b1 from base.immediates import imm64, uimm8, uimm32, ieee32, ieee64, offset32 from base.immediates import boolean, intcc, floatcc, memflags, regunit +from base.immediates import trapcode from base import entities from cdsl.ti import WiderOrEq import base.formats # noqa @@ -126,11 +127,12 @@ br_table = Instruction( """, ins=(x, JT), is_branch=True) +code = Operand('code', trapcode) trap = Instruction( 'trap', r""" Terminate execution unconditionally. """, - is_terminator=True, can_trap=True) + ins=code, is_terminator=True, can_trap=True) trapz = Instruction( 'trapz', r""" @@ -138,7 +140,7 @@ trapz = Instruction( if ``c`` is non-zero, execution continues at the following instruction. """, - ins=c, can_trap=True) + ins=(c, code), can_trap=True) trapnz = Instruction( 'trapnz', r""" @@ -146,7 +148,7 @@ trapnz = Instruction( if ``c`` is zero, execution continues at the following instruction. """, - ins=c, can_trap=True) + ins=(c, code), can_trap=True) rvals = Operand('rvals', VARIABLE_ARGS, doc='return values') diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index bf1b91c0dc..355c755e7c 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -287,8 +287,8 @@ I64.enc(base.brnz.b1, *r.t8jccb_abcd(0x75)) # # Trap as ud2 # -I32.enc(base.trap, *r.noop(0x0f, 0x0b)) -I64.enc(base.trap, *r.noop(0x0f, 0x0b)) +I32.enc(base.trap, *r.trap(0x0f, 0x0b)) +I64.enc(base.trap, *r.trap(0x0f, 0x0b)) # # Comparisons diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 06aea4b01f..8d07f06de1 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -5,7 +5,7 @@ from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt, IsEqual from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry -from base.formats import Nullary, Call, IndirectCall, Store, Load +from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare from base.formats import RegMove, Ternary, Jump, Branch, FuncAddr from .registers import GPR, ABCD, FPR @@ -195,8 +195,8 @@ class TailRecipe: null = EncRecipe('null', Unary, size=0, ins=GPR, outs=0, emit='') # XX opcode, no ModR/M. -noop = TailRecipe( - 'noop', Nullary, size=0, ins=(), outs=(), +trap = TailRecipe( + 'trap', Trap, size=0, ins=(), outs=(), emit='PUT_OP(bits, BASE_REX, sink);') # XX /r diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index cf8ecc8cb9..5f342baf89 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -428,7 +428,7 @@ mod test { use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::types::*; - use ir::{Function, InstBuilder, types}; + use ir::{Function, InstBuilder, types, TrapCode}; use settings; use super::*; use verifier::verify_context; @@ -506,7 +506,7 @@ mod test { let jmp02 = cur.ins().jump(ebb2, &[]); cur.insert_ebb(ebb1); - let trap = cur.ins().trap(); + let trap = cur.ins().trap(TrapCode::User(5)); cur.insert_ebb(ebb2); let jmp21 = cur.ins().jump(ebb1, &[]); diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 6dd5eafde7..0b3c671cc6 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -214,6 +214,12 @@ pub enum InstructionData { src: RegUnit, dst: RegUnit, }, + Trap { opcode: Opcode, code: ir::TrapCode }, + CondTrap { + opcode: Opcode, + arg: Value, + code: ir::TrapCode, + }, } /// A variable list of `Value` operands used for function call arguments and passing arguments to diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index b65174245f..077db6537b 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -17,6 +17,7 @@ mod globalvar; mod heap; mod memflags; mod progpoint; +mod trapcode; mod valueloc; pub use ir::builder::{InstBuilder, InstBuilderBase, InstInserterBase, InsertBuilder}; @@ -34,6 +35,7 @@ pub use ir::layout::{Layout, CursorBase, Cursor}; pub use ir::memflags::MemFlags; pub use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; pub use ir::stackslot::{StackSlots, StackSlotKind, StackSlotData}; +pub use ir::trapcode::TrapCode; pub use ir::types::Type; pub use ir::valueloc::{ValueLoc, ArgumentLoc}; diff --git a/lib/cretonne/src/ir/trapcode.rs b/lib/cretonne/src/ir/trapcode.rs new file mode 100644 index 0000000000..c9312cdff0 --- /dev/null +++ b/lib/cretonne/src/ir/trapcode.rs @@ -0,0 +1,89 @@ +//! Trap codes describing the reason for a trap. + +use std::fmt::{self, Display, Formatter}; +use std::str::FromStr; + +/// A trap code describing the reason for a trap. +/// +/// All trap instructions have an explicit trap code. +#[derive(Clone, Copy, PartialEq, Eq, Debug, Hash)] +pub enum TrapCode { + /// The current stack space was exhausted. + /// + /// On some platforms, a stack overflow may also be indicated by a segmentation fault from the + /// stack guard page. + StackOverflow, + + /// A `heap_addr` instruction detected an out-of-bounds error. + /// + /// Some out-of-bounds heap accesses are detected by a segmentation fault on the heap guard + /// pages. + HeapOutOfBounds, + + /// An integer arithmetic operation caused an overflow. + IntegerOverflow, + + /// An integer division by zero. + IntegerDivisionByZero, + + /// A user-defined trap code. + User(u16), +} + +impl Display for TrapCode { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + use self::TrapCode::*; + let identifier = match *self { + StackOverflow => "stk_ovf", + HeapOutOfBounds => "heap_oob", + IntegerOverflow => "int_ovf", + IntegerDivisionByZero => "int_divz", + User(x) => return write!(f, "user{}", x), + }; + f.write_str(identifier) + } +} + +impl FromStr for TrapCode { + type Err = (); + + fn from_str(s: &str) -> Result { + use self::TrapCode::*; + match s { + "stk_ovf" => Ok(StackOverflow), + "heap_oob" => Ok(HeapOutOfBounds), + "int_ovf" => Ok(IntegerOverflow), + "int_divz" => Ok(IntegerDivisionByZero), + _ if s.starts_with("user") => s[4..].parse().map(User).map_err(|_| ()), + _ => Err(()), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + // Everything but user-defined codes. + const CODES: [TrapCode; 4] = [ + TrapCode::StackOverflow, + TrapCode::HeapOutOfBounds, + TrapCode::IntegerOverflow, + TrapCode::IntegerDivisionByZero, + ]; + + #[test] + fn display() { + for r in &CODES { + let tc = *r; + assert_eq!(tc.to_string().parse(), Ok(tc)); + } + assert_eq!("bogus".parse::(), Err(())); + + assert_eq!(TrapCode::User(17).to_string(), "user17"); + assert_eq!("user22".parse(), Ok(TrapCode::User(22))); + assert_eq!("user".parse::(), Err(())); + assert_eq!("user-1".parse::(), Err(())); + assert_eq!("users".parse::(), Err(())); + } +} diff --git a/lib/cretonne/src/legalizer/heap.rs b/lib/cretonne/src/legalizer/heap.rs index 0a14f5f8ee..ab2ce444af 100644 --- a/lib/cretonne/src/legalizer/heap.rs +++ b/lib/cretonne/src/legalizer/heap.rs @@ -74,14 +74,14 @@ fn dynamic_addr( // We need an overflow check for the adjusted offset. let size_val = pos.ins().iconst(offset_ty, size); let (adj_offset, overflow) = pos.ins().iadd_cout(offset, size_val); - pos.ins().trapnz(overflow); + pos.ins().trapnz(overflow, ir::TrapCode::HeapOutOfBounds); oob = pos.ins().icmp( IntCC::UnsignedGreaterThan, adj_offset, bound, ); } - pos.ins().trapnz(oob); + pos.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds); offset_addr(inst, heap, addr_ty, offset, offset_ty, pos.func); } @@ -103,7 +103,7 @@ fn static_addr( // Start with the bounds check. Trap if `offset + size > bound`. if size > bound { // This will simply always trap since `offset >= 0`. - pos.ins().trap(); + pos.ins().trap(ir::TrapCode::HeapOutOfBounds); pos.func.dfg.replace(inst).iconst(addr_ty, 0); return; } @@ -129,7 +129,7 @@ fn static_addr( limit, ) }; - pos.ins().trapnz(oob); + pos.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds); } offset_addr(inst, heap, addr_ty, offset, offset_ty, pos.func); diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 6dbeb15d04..5cb1c27819 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -106,15 +106,15 @@ include!(concat!(env!("OUT_DIR"), "/legalizer.rs")); fn expand_cond_trap(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { // Parse the instruction. let trapz; - let arg = match func.dfg[inst] { - ir::InstructionData::Unary { opcode, arg } => { + let (arg, code) = match func.dfg[inst] { + ir::InstructionData::CondTrap { opcode, arg, code } => { // We want to branch *over* an unconditional trap. trapz = match opcode { ir::Opcode::Trapz => true, ir::Opcode::Trapnz => false, _ => panic!("Expected cond trap: {}", func.dfg.display_inst(inst, None)), }; - arg + (arg, code) } _ => panic!("Expected cond trap: {}", func.dfg.display_inst(inst, None)), }; @@ -138,7 +138,7 @@ fn expand_cond_trap(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFl } let mut pos = FuncCursor::new(func).after_inst(inst); - pos.ins().trap(); + pos.ins().trap(code); pos.insert_ebb(new_ebb); // Finally update the CFG. diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index f387c0d295..3fd6685133 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -337,7 +337,9 @@ impl<'a> Verifier<'a> { FloatCompare { .. } | Load { .. } | Store { .. } | - RegMove { .. } => {} + RegMove { .. } | + Trap { .. } | + CondTrap { .. } => {} } Ok(()) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index f225e0aa7d..6dd6f333df 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -376,6 +376,8 @@ pub fn write_operands( write!(w, " {}, %{} -> %{}", arg, src, dst) } } + Trap { code, .. } => write!(w, " {}", code), + CondTrap { arg, code, .. } => write!(w, " {}, {}", arg, code), } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 94cf785fb4..8ce55d4445 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -2192,6 +2192,19 @@ impl<'a> Parser<'a> { dst, } } + InstructionFormat::Trap => { + let code = self.match_enum("expected trap code")?; + InstructionData::Trap { opcode, code } + } + InstructionFormat::CondTrap => { + let arg = self.match_value("expected SSA value operand")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; + let code = self.match_enum("expected trap code")?; + InstructionData::CondTrap { opcode, arg, code } + } }; Ok(idata) } @@ -2365,7 +2378,7 @@ mod tests { jt10 = jump_table ebb0 ; Jumptable ebb0: ; Basic block - trap ; Instruction + trap user42; Instruction } ; Trailing. ; More trailing.", ).parse_function(None) @@ -2459,7 +2472,7 @@ mod tests { let func = Parser::new( "function #1234567890AbCdEf() native { ebb0: - trap + trap int_divz }", ).parse_function(None) .unwrap() @@ -2470,7 +2483,7 @@ mod tests { let mut parser = Parser::new( "function #12ww() native { ebb0: - trap + trap stk_ovf }", ); assert!(parser.parse_function(None).is_err()); @@ -2479,7 +2492,7 @@ mod tests { let mut parser = Parser::new( "function #1() native { ebb0: - trap + trap user0 }", ); assert!(parser.parse_function(None).is_err()); @@ -2488,7 +2501,7 @@ mod tests { let func = Parser::new( "function #() native { ebb0: - trap + trap int_ovf }", ).parse_function(None) .unwrap() diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index b3d202aa2d..c66f9cea79 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -101,7 +101,9 @@ pub fn translate_operator( // We do nothing } Operator::Unreachable => { - builder.ins().trap(); + // We use `trap user0` to indicate a user-generated trap. + // We could make the trap code configurable if need be. + builder.ins().trap(ir::TrapCode::User(0)); state.real_unreachable_stack_depth = 1; } /***************************** Control flow blocks ********************************** From 8def91b4ad1deb2075b777666e5d86c0735c4fc9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 20 Sep 2017 15:59:13 -0700 Subject: [PATCH 1133/3084] Remove the unused Nullary instruction format. This format was only used by the trap instruction which has its own format now. --- lib/cretonne/meta/base/formats.py | 2 -- lib/cretonne/src/ir/dfg.rs | 19 ++++++++++++++----- lib/cretonne/src/ir/instructions.rs | 1 - lib/cretonne/src/verifier/mod.rs | 8 ++++---- lib/cretonne/src/write.rs | 1 - lib/reader/src/parser.rs | 1 - 6 files changed, 18 insertions(+), 14 deletions(-) diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 1746217958..7fd93de1f9 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -13,8 +13,6 @@ from .immediates import boolean, intcc, floatcc, memflags, regunit, trapcode from . import entities from .entities import ebb, sig_ref, func_ref, stack_slot, heap -Nullary = InstructionFormat() - Unary = InstructionFormat(VALUE) UnaryImm = InstructionFormat(imm64) UnaryIeee32 = InstructionFormat(ieee32) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index b81bbe96fc..47c53100cf 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -898,20 +898,26 @@ impl<'a> fmt::Display for DisplayInst<'a> { mod tests { use super::*; use ir::types; - use ir::{Function, Cursor, CursorBase, Opcode, InstructionData}; + use ir::{Function, Cursor, CursorBase, Opcode, InstructionData, TrapCode}; #[test] fn make_inst() { let mut dfg = DataFlowGraph::new(); - let idata = InstructionData::Nullary { opcode: Opcode::Iconst }; + let idata = InstructionData::UnaryImm { + opcode: Opcode::Iconst, + imm: 0.into(), + }; let next = dfg.next_inst(); let inst = dfg.make_inst(idata); assert_eq!(next, inst); dfg.make_inst_results(inst, types::I32); assert_eq!(inst.to_string(), "inst0"); - assert_eq!(dfg.display_inst(inst, None).to_string(), "v0 = iconst.i32"); + assert_eq!( + dfg.display_inst(inst, None).to_string(), + "v0 = iconst.i32 0" + ); // Immutable reference resolution. { @@ -941,9 +947,12 @@ mod tests { fn no_results() { let mut dfg = DataFlowGraph::new(); - let idata = InstructionData::Nullary { opcode: Opcode::Trap }; + let idata = InstructionData::Trap { + opcode: Opcode::Trap, + code: TrapCode::User(0), + }; let inst = dfg.make_inst(idata); - assert_eq!(dfg.display_inst(inst, None).to_string(), "trap"); + assert_eq!(dfg.display_inst(inst, None).to_string(), "trap user0"); // Result slice should be empty. assert_eq!(dfg.inst_results(inst), &[]); diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 0b3c671cc6..14bd3de726 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -103,7 +103,6 @@ impl FromStr for Opcode { #[derive(Clone, Debug, Hash, PartialEq, Eq)] #[allow(missing_docs)] pub enum InstructionData { - Nullary { opcode: Opcode }, Unary { opcode: Opcode, arg: Value }, UnaryImm { opcode: Opcode, imm: Imm64 }, UnaryIeee32 { opcode: Opcode, imm: Ieee32 }, diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 3fd6685133..82ba075f39 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -321,7 +321,6 @@ impl<'a> Verifier<'a> { } // Exhaustive list so we can't forget to add new formats - Nullary { .. } | Unary { .. } | UnaryImm { .. } | UnaryIeee32 { .. } | @@ -1048,9 +1047,10 @@ mod tests { let mut func = Function::new(); let ebb0 = func.dfg.make_ebb(); func.layout.append_ebb(ebb0); - let nullary_with_bad_opcode = func.dfg.make_inst( - InstructionData::Nullary { opcode: Opcode::Jump }, - ); + let nullary_with_bad_opcode = func.dfg.make_inst(InstructionData::UnaryImm { + opcode: Opcode::Jump, + imm: 0.into(), + }); func.layout.append_inst(nullary_with_bad_opcode, ebb0); let flags = &settings::Flags::new(&settings::builder()); let verifier = Verifier::new(&func, flags.into()); diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 6dd6f333df..2cbc8766ab 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -269,7 +269,6 @@ pub fn write_operands( let pool = &dfg.value_lists; use ir::instructions::InstructionData::*; match dfg[inst] { - Nullary { .. } => write!(w, ""), Unary { arg, .. } => write!(w, " {}", arg), UnaryImm { imm, .. } => write!(w, " {}", imm), UnaryIeee32 { imm, .. } => write!(w, " {}", imm), diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 8ce55d4445..331ca5eb92 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1828,7 +1828,6 @@ impl<'a> Parser<'a> { opcode: Opcode, ) -> Result { let idata = match opcode.format() { - InstructionFormat::Nullary => InstructionData::Nullary { opcode }, InstructionFormat::Unary => { InstructionData::Unary { opcode, From f80ee7af3995520ef7bdb10b7c7f647ba3d1fea6 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 20 Sep 2017 16:14:26 -0700 Subject: [PATCH 1134/3084] Add a "host" crate for autodetecting the host. (#159) * Add a "native" crate for autodetecting the host. * Remove the redundant .gitignore. * Use the proper builder for enabling subtarget flags. --- cranelift/Cargo.toml | 1 + lib/native/Cargo.toml | 17 +++++++++ lib/native/src/lib.rs | 88 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 106 insertions(+) create mode 100644 lib/native/Cargo.toml create mode 100644 lib/native/src/lib.rs diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 6523a9d2da..c576d3990e 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -17,6 +17,7 @@ cretonne = { path = "lib/cretonne" } cretonne-reader = { path = "lib/reader" } cretonne-frontend = { path = "lib/frontend" } cretonne-wasm = { path = "lib/wasm" } +cretonne-native = { path = "lib/native" } filecheck = { path = "lib/filecheck" } docopt = "0.8.0" serde = "1.0.8" diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml new file mode 100644 index 0000000000..0f66bc6037 --- /dev/null +++ b/lib/native/Cargo.toml @@ -0,0 +1,17 @@ +[package] +name = "cretonne-native" +version = "0.0.0" +authors = ["The Cretonne Project Developers"] +publish = false +description = "Support for targetting the host with Cretonne" +repository = "https://github.com/stoklund/cretonne" +license = "Apache-2.0" + +[lib] +name = "cton_native" + +[dependencies] +cretonne = { path = "../cretonne" } + +[target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] +raw-cpuid = "3.0.0" diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs new file mode 100644 index 0000000000..49cb173383 --- /dev/null +++ b/lib/native/src/lib.rs @@ -0,0 +1,88 @@ +//! Performs autodetection of the host for the purposes of running +//! Cretonne to generate code to run on the same machine. + +#![deny(missing_docs)] + +extern crate cretonne; + +#[cfg(any(target_arch = "x86", target_arch = "x86_64"))] +extern crate raw_cpuid; + +use cretonne::isa; +use cretonne::settings::{self, Configurable}; + +#[cfg(any(target_arch = "x86", target_arch = "x86_64"))] +use raw_cpuid::CpuId; + +/// Return `settings` and `isa` builders configured for the current host +/// machine, or `Err(())` if the host machine is not supported +/// in the current configuration. +pub fn builders() -> Result<(settings::Builder, isa::Builder), ()> { + let mut flag_builder = settings::builder(); + + // TODO: Add RISC-V support once Rust supports it. + + if cfg!(target_pointer_width = "64") { + flag_builder.enable("is_64bit").unwrap(); + } + + let name = if cfg!(any(target_arch = "x86", target_arch = "x86_64")) { + "intel" + } else if cfg!(target_arch = "arm") { + "arm32" + } else if cfg!(target_arch = "aarch64") { + "arm64" + } else { + return Err(()); + }; + + let mut isa_builder = isa::lookup(name).map_err(|err| match err { + isa::LookupError::Unknown => panic!(), + isa::LookupError::Unsupported => (()), + })?; + + if cfg!(any(target_arch = "x86", target_arch = "x86_64")) { + parse_x86_cpuid(&mut isa_builder); + } + + Ok((flag_builder, isa_builder)) +} + +#[cfg(any(target_arch = "x86", target_arch = "x86_64"))] +fn parse_x86_cpuid(isa_builder: &mut isa::Builder) { + let cpuid = CpuId::new(); + + if let Some(info) = cpuid.get_feature_info() { + if info.has_sse2() { + isa_builder.enable("has_sse2").unwrap(); + } + if info.has_sse3() { + isa_builder.enable("has_sse3").unwrap(); + } + if info.has_sse41() { + isa_builder.enable("has_sse41").unwrap(); + } + if info.has_sse42() { + isa_builder.enable("has_sse42").unwrap(); + } + if info.has_popcnt() { + isa_builder.enable("has_popcnt").unwrap(); + } + if info.has_avx() { + isa_builder.enable("has_avx").unwrap(); + } + } + if let Some(info) = cpuid.get_extended_feature_info() { + if info.has_bmi1() { + isa_builder.enable("has_bmi1").unwrap(); + } + if info.has_bmi2() { + isa_builder.enable("has_bmi2").unwrap(); + } + } + if let Some(info) = cpuid.get_extended_function_info() { + if info.has_lzcnt() { + isa_builder.enable("has_lzcnt").unwrap(); + } + } +} From ed6630dc02b80a3c038dadfa15f0d6a38701eebf Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 20 Sep 2017 16:37:55 -0700 Subject: [PATCH 1135/3084] Move `verify` calls back into Context, using FlagsOrIsa. With FlagsOrIsa, we can pass around the information we need to run the verifier from the Context even when a TargetIsa is not available. --- cranelift/src/filetest/licm.rs | 3 +-- cranelift/src/filetest/simple_gvn.rs | 3 +-- cranelift/src/wasm.rs | 12 +++++------- lib/cretonne/src/context.rs | 17 ++++++++++------- 4 files changed, 17 insertions(+), 18 deletions(-) diff --git a/cranelift/src/filetest/licm.rs b/cranelift/src/filetest/licm.rs index 3f3bc6cecc..32be450476 100644 --- a/cranelift/src/filetest/licm.rs +++ b/cranelift/src/filetest/licm.rs @@ -40,8 +40,7 @@ impl SubTest for TestLICM { comp_ctx.flowgraph(); comp_ctx.compute_loop_analysis(); - comp_ctx.licm(); - comp_ctx.verify(context.flags_or_isa()).map_err(|e| { + comp_ctx.licm(context.flags_or_isa()).map_err(|e| { pretty_error(&comp_ctx.func, context.isa, Into::into(e)) })?; diff --git a/cranelift/src/filetest/simple_gvn.rs b/cranelift/src/filetest/simple_gvn.rs index 12a5554f74..a1da0e42a4 100644 --- a/cranelift/src/filetest/simple_gvn.rs +++ b/cranelift/src/filetest/simple_gvn.rs @@ -39,8 +39,7 @@ impl SubTest for TestSimpleGVN { comp_ctx.func = func.into_owned(); comp_ctx.flowgraph(); - comp_ctx.simple_gvn(); - comp_ctx.verify(context.flags_or_isa()).map_err(|e| { + comp_ctx.simple_gvn(context.flags_or_isa()).map_err(|e| { pretty_error(&comp_ctx.func, context.isa, Into::into(e)) })?; diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 7e9040c2c9..e665822e60 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -20,7 +20,7 @@ use std::path::Path; use std::process::Command; use tempdir::TempDir; use term; -use utils::pretty_verifier_error; +use utils::{pretty_verifier_error, pretty_error}; macro_rules! vprintln { ($x: expr, $($tts:tt)*) => { @@ -182,13 +182,11 @@ fn handle_module( })?; context.flowgraph(); context.compute_loop_analysis(); - context.licm(); - context.verify(*fisa).map_err(|err| { - pretty_verifier_error(&context.func, fisa.isa, err) + context.licm(*fisa).map_err(|err| { + pretty_error(&context.func, fisa.isa, err) })?; - context.simple_gvn(); - context.verify(*fisa).map_err(|err| { - pretty_verifier_error(&context.func, fisa.isa, err) + context.simple_gvn(*fisa).map_err(|err| { + pretty_error(&context.func, fisa.isa, err) })?; } terminal.fg(term::color::GREEN).unwrap(); diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 30f67a811f..cd402873e8 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -92,9 +92,10 @@ impl Context { } /// Run the verifier only if the `enable_verifier` setting is true. - pub fn verify_if(&self, isa: &TargetIsa) -> CtonResult { - if isa.flags().enable_verifier() { - self.verify(isa).map_err(Into::into) + pub fn verify_if<'a, FOI: Into>>(&self, fisa: FOI) -> CtonResult { + let fisa = fisa.into(); + if fisa.flags.enable_verifier() { + self.verify(fisa).map_err(Into::into) } else { Ok(()) } @@ -136,18 +137,20 @@ impl Context { } /// Perform simple GVN on the function. - pub fn simple_gvn(&mut self) { - do_simple_gvn(&mut self.func, &mut self.cfg, &mut self.domtree) + pub fn simple_gvn<'a, FOI: Into>>(&mut self, fisa: FOI) -> CtonResult { + do_simple_gvn(&mut self.func, &mut self.cfg, &mut self.domtree); + self.verify_if(fisa) } /// Perform LICM on the function. - pub fn licm(&mut self) { + pub fn licm<'a, FOI: Into>>(&mut self, fisa: FOI) -> CtonResult { do_licm( &mut self.func, &mut self.cfg, &mut self.domtree, &mut self.loop_analysis, - ) + ); + self.verify_if(fisa) } /// Run the register allocator. From 03dee5e44266627a97a6baf256c0d0120a883c41 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 21 Sep 2017 11:21:23 -0700 Subject: [PATCH 1136/3084] Replace some uses of layout::Cursor with FuncCursor. The layout::Cursor is unfortunate because it doesn't reference the whole function. --- lib/cretonne/src/flowgraph.rs | 14 +++++++------- lib/cretonne/src/licm.rs | 24 +++++++++++------------- lib/cretonne/src/simple_gvn.rs | 25 +++++++++++++------------ 3 files changed, 31 insertions(+), 32 deletions(-) diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index d246a1fa3a..d4ec4ca717 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -152,7 +152,8 @@ impl ControlFlowGraph { #[cfg(test)] mod tests { use super::*; - use ir::{Function, InstBuilder, Cursor, CursorBase, types}; + use cursor::{Cursor, FuncCursor}; + use ir::{Function, InstBuilder, types}; #[test] fn empty() { @@ -194,16 +195,15 @@ mod tests { let jmp_ebb1_ebb2; { - let dfg = &mut func.dfg; - let cur = &mut Cursor::new(&mut func.layout); + let cur = &mut FuncCursor::new(&mut func); cur.insert_ebb(ebb0); - br_ebb0_ebb2 = dfg.ins(cur).brnz(cond, ebb2, &[]); - jmp_ebb0_ebb1 = dfg.ins(cur).jump(ebb1, &[]); + br_ebb0_ebb2 = cur.ins().brnz(cond, ebb2, &[]); + jmp_ebb0_ebb1 = cur.ins().jump(ebb1, &[]); cur.insert_ebb(ebb1); - br_ebb1_ebb1 = dfg.ins(cur).brnz(cond, ebb1, &[]); - jmp_ebb1_ebb2 = dfg.ins(cur).jump(ebb2, &[]); + br_ebb1_ebb1 = cur.ins().brnz(cond, ebb1, &[]); + jmp_ebb1_ebb2 = cur.ins().jump(ebb2, &[]); cur.insert_ebb(ebb2); } diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index c7b4ff72f6..490e0d8335 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -1,6 +1,7 @@ //! A Loop Invariant Code Motion optimization pass -use ir::{Function, Ebb, Inst, Value, Cursor, CursorBase, Type, InstBuilder, Layout}; +use cursor::{Cursor, FuncCursor}; +use ir::{Function, Ebb, Inst, Value, Type, InstBuilder, Layout}; use flowgraph::ControlFlowGraph; use std::collections::HashSet; use dominator_tree::DominatorTree; @@ -33,12 +34,12 @@ pub fn do_licm( None => { let pre_header = create_pre_header(loop_analysis.loop_header(lp), func, cfg, domtree); - pos = Cursor::new(&mut func.layout).at_last_inst(pre_header); + pos = FuncCursor::new(func).at_last_inst(pre_header); } // If there is a natural pre-header we insert new instructions just before the // related jumping instruction (which is not necessarily at the end). Some((_, last_inst)) => { - pos = Cursor::new(&mut func.layout).at_inst(last_inst); + pos = FuncCursor::new(func).at_inst(last_inst); } }; // The last instruction of the pre-header is the termination instruction (usually @@ -80,14 +81,11 @@ fn create_pre_header( } } { - let mut pos = Cursor::new(&mut func.layout).at_top(header); + let mut pos = FuncCursor::new(func).at_top(header); // Inserts the pre-header at the right place in the layout. pos.insert_ebb(pre_header); pos.next_inst(); - func.dfg.ins(&mut pos).jump( - header, - pre_header_args_value.as_slice(pool), - ); + pos.ins().jump(header, pre_header_args_value.as_slice(pool)); } pre_header } @@ -141,17 +139,17 @@ fn remove_loop_invariant_instructions( ) -> Vec { let mut loop_values: HashSet = HashSet::new(); let mut invariant_inst: Vec = Vec::new(); - let mut pos = Cursor::new(&mut func.layout); + let mut pos = FuncCursor::new(func); // We traverse the loop EBB in reverse post-order. for ebb in postorder_ebbs_loop(loop_analysis, cfg, lp).iter().rev() { // Arguments of the EBB are loop values - for val in func.dfg.ebb_args(*ebb) { + for val in pos.func.dfg.ebb_args(*ebb) { loop_values.insert(*val); } pos.goto_top(*ebb); while let Some(inst) = pos.next_inst() { - if func.dfg.has_results(inst) && - func.dfg.inst_args(inst).into_iter().all(|arg| { + if pos.func.dfg.has_results(inst) && + pos.func.dfg.inst_args(inst).into_iter().all(|arg| { !loop_values.contains(arg) }) { @@ -163,7 +161,7 @@ fn remove_loop_invariant_instructions( } else { // If the instruction is not loop-invariant we push its results in the set of // loop values - for out in func.dfg.inst_results(inst) { + for out in pos.func.dfg.inst_results(inst) { loop_values.insert(*out); } } diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index c50f70a3fa..b59f6bafd8 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -1,8 +1,9 @@ //! A simple GVN pass. +use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; -use ir::{Cursor, CursorBase, InstructionData, Function, Inst, Opcode, Type}; +use ir::{InstructionData, Function, Inst, Opcode, Type}; use scoped_hash_map::ScopedHashMap; /// Test whether the given opcode is unsafe to even consider for GVN. @@ -22,13 +23,13 @@ pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph, domtree: & let mut scope_stack: Vec = Vec::new(); // Visit EBBs in a reverse post-order. - let mut pos = Cursor::new(&mut func.layout); + let mut pos = FuncCursor::new(func); for &ebb in domtree.cfg_postorder().iter().rev() { // Pop any scopes that we just exited. loop { if let Some(current) = scope_stack.last() { - if domtree.dominates(*current, ebb, &pos.layout) { + if domtree.dominates(*current, ebb, &pos.func.layout) { break; } } else { @@ -39,38 +40,38 @@ pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph, domtree: & } // Push a scope for the current block. - scope_stack.push(pos.layout.first_inst(ebb).unwrap()); + scope_stack.push(pos.func.layout.first_inst(ebb).unwrap()); visible_values.increment_depth(); pos.goto_top(ebb); while let Some(inst) = pos.next_inst() { // Resolve aliases, particularly aliases we created earlier. - func.dfg.resolve_aliases_in_arguments(inst); + pos.func.dfg.resolve_aliases_in_arguments(inst); - let opcode = func.dfg[inst].opcode(); + let opcode = pos.func.dfg[inst].opcode(); if opcode.is_branch() && !opcode.is_terminator() { - scope_stack.push(pos.layout.next_inst(inst).unwrap()); + scope_stack.push(pos.func.layout.next_inst(inst).unwrap()); visible_values.increment_depth(); } if trivially_unsafe_for_gvn(opcode) { continue; } - let ctrl_typevar = func.dfg.ctrl_typevar(inst); - let key = (func.dfg[inst].clone(), ctrl_typevar); + let ctrl_typevar = pos.func.dfg.ctrl_typevar(inst); + let key = (pos.func.dfg[inst].clone(), ctrl_typevar); let entry = visible_values.entry(key); use scoped_hash_map::Entry::*; match entry { Occupied(entry) => { - debug_assert!(domtree.dominates(*entry.get(), inst, pos.layout)); + debug_assert!(domtree.dominates(*entry.get(), inst, &pos.func.layout)); // If the redundant instruction is representing the current // scope, pick a new representative. let old = scope_stack.last_mut().unwrap(); if *old == inst { - *old = pos.layout.next_inst(inst).unwrap(); + *old = pos.func.layout.next_inst(inst).unwrap(); } // Replace the redundant instruction and remove it. - func.dfg.replace_with_aliases(inst, *entry.get()); + pos.func.dfg.replace_with_aliases(inst, *entry.get()); pos.remove_inst_and_step_back(); } Vacant(entry) => { From 1cd91b6f3086d7ca68d361319063510ea2a8afd0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 21 Sep 2017 12:16:32 -0700 Subject: [PATCH 1137/3084] Eliminate layout::Cursor from cton_frontend. Replace all uses with a FuncCursor. Avoid the anti-pattern of passing parts of a function around as independent references. --- lib/cretonne/src/loop_analysis.rs | 37 +- lib/cretonne/src/topo_order.rs | 10 +- lib/frontend/src/frontend.rs | 11 +- lib/frontend/src/ssa.rs | 599 ++++++++---------------------- 4 files changed, 172 insertions(+), 485 deletions(-) diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index 05ce1952f2..21c612356e 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -223,7 +223,8 @@ impl LoopAnalysis { #[cfg(test)] mod test { - use ir::{Function, InstBuilder, Cursor, CursorBase, types}; + use cursor::{Cursor, FuncCursor}; + use ir::{Function, InstBuilder, types}; use loop_analysis::{Loop, LoopAnalysis}; use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; @@ -238,21 +239,20 @@ mod test { let cond = func.dfg.append_ebb_arg(ebb0, types::I32); { - let dfg = &mut func.dfg; - let cur = &mut Cursor::new(&mut func.layout); + let mut cur = FuncCursor::new(&mut func); cur.insert_ebb(ebb0); - dfg.ins(cur).jump(ebb1, &[]); + cur.ins().jump(ebb1, &[]); cur.insert_ebb(ebb1); - dfg.ins(cur).jump(ebb2, &[]); + cur.ins().jump(ebb2, &[]); cur.insert_ebb(ebb2); - dfg.ins(cur).brnz(cond, ebb1, &[]); - dfg.ins(cur).jump(ebb3, &[]); + cur.ins().brnz(cond, ebb1, &[]); + cur.ins().jump(ebb3, &[]); cur.insert_ebb(ebb3); - dfg.ins(cur).brnz(cond, ebb0, &[]); + cur.ins().brnz(cond, ebb0, &[]); } @@ -291,29 +291,28 @@ mod test { let cond = func.dfg.append_ebb_arg(ebb0, types::I32); { - let dfg = &mut func.dfg; - let cur = &mut Cursor::new(&mut func.layout); + let mut cur = FuncCursor::new(&mut func); cur.insert_ebb(ebb0); - dfg.ins(cur).brnz(cond, ebb1, &[]); - dfg.ins(cur).jump(ebb3, &[]); + cur.ins().brnz(cond, ebb1, &[]); + cur.ins().jump(ebb3, &[]); cur.insert_ebb(ebb1); - dfg.ins(cur).jump(ebb2, &[]); + cur.ins().jump(ebb2, &[]); cur.insert_ebb(ebb2); - dfg.ins(cur).brnz(cond, ebb1, &[]); - dfg.ins(cur).jump(ebb5, &[]); + cur.ins().brnz(cond, ebb1, &[]); + cur.ins().jump(ebb5, &[]); cur.insert_ebb(ebb3); - dfg.ins(cur).jump(ebb4, &[]); + cur.ins().jump(ebb4, &[]); cur.insert_ebb(ebb4); - dfg.ins(cur).brnz(cond, ebb3, &[]); - dfg.ins(cur).jump(ebb5, &[]); + cur.ins().brnz(cond, ebb3, &[]); + cur.ins().jump(ebb5, &[]); cur.insert_ebb(ebb5); - dfg.ins(cur).brnz(cond, ebb0, &[]); + cur.ins().brnz(cond, ebb0, &[]); } diff --git a/lib/cretonne/src/topo_order.rs b/lib/cretonne/src/topo_order.rs index 2224b339a1..db566ae7c8 100644 --- a/lib/cretonne/src/topo_order.rs +++ b/lib/cretonne/src/topo_order.rs @@ -80,9 +80,10 @@ impl TopoOrder { #[cfg(test)] mod test { + use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; - use ir::{Function, InstBuilder, Cursor, CursorBase}; + use ir::{Function, InstBuilder}; use std::iter; use super::*; @@ -105,13 +106,12 @@ mod test { let ebb1 = func.dfg.make_ebb(); { - let dfg = &mut func.dfg; - let cur = &mut Cursor::new(&mut func.layout); + let mut cur = FuncCursor::new(&mut func); cur.insert_ebb(ebb0); - dfg.ins(cur).jump(ebb1, &[]); + cur.ins().jump(ebb1, &[]); cur.insert_ebb(ebb1); - dfg.ins(cur).jump(ebb1, &[]); + cur.ins().jump(ebb1, &[]); } let cfg = ControlFlowGraph::with_function(&func); diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 98ef98ac62..75c7dfe481 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -271,12 +271,7 @@ where /// created. Forgetting to call this method on every block will cause inconsistences in the /// produced functions. pub fn seal_block(&mut self, ebb: Ebb) { - let side_effects = self.builder.ssa.seal_ebb_header_block( - ebb, - &mut self.func.dfg, - &mut self.func.layout, - &mut self.func.jump_tables, - ); + let side_effects = self.builder.ssa.seal_ebb_header_block(ebb, self.func); self.handle_ssa_side_effects(side_effects); } @@ -292,9 +287,7 @@ where "this variable is used but its type has not been declared", ); let (val, side_effects) = self.builder.ssa.use_var( - &mut self.func.dfg, - &mut self.func.layout, - &mut self.func.jump_tables, + self.func, var, ty, self.position.basic_block, diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index e054ee511c..0d3e00c5a9 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -5,8 +5,8 @@ //! In: Jhala R., De Bosschere K. (eds) Compiler Construction. CC 2013. //! Lecture Notes in Computer Science, vol 7791. Springer, Berlin, Heidelberg -use cretonne::ir::{Ebb, Value, Inst, Type, DataFlowGraph, JumpTables, Layout, Cursor, CursorBase, - InstBuilder}; +use cretonne::cursor::{Cursor, FuncCursor}; +use cretonne::ir::{Ebb, Value, Inst, Type, Function, InstBuilder}; use cretonne::ir::instructions::BranchInfo; use cretonne::entity::{EntityRef, PrimaryMap, EntityMap}; use cretonne::packed_option::PackedOption; @@ -217,9 +217,7 @@ where /// responsible for making sure that you initialize your variables. pub fn use_var( &mut self, - dfg: &mut DataFlowGraph, - layout: &mut Layout, - jts: &mut JumpTables, + func: &mut Function, var: Variable, ty: Type, block: Block, @@ -232,16 +230,14 @@ where } // Otherwise, we have to do a non-local lookup. - self.use_var_nonlocal(dfg, layout, jts, var, ty, block) + self.use_var_nonlocal(func, var, ty, block) } // The non-local case of use_var. Query each predecessor for a value and add branch // arguments as needed to satisfy the use. fn use_var_nonlocal( &mut self, - dfg: &mut DataFlowGraph, - layout: &mut Layout, - jts: &mut JumpTables, + func: &mut Function, var: Variable, ty: Type, block: Block, @@ -255,11 +251,11 @@ where // Only one predecessor, straightforward case UseVarCases::SealedOnePredecessor(data.predecessors[0].0) } else { - let val = dfg.append_ebb_arg(data.ebb, ty); + let val = func.dfg.append_ebb_arg(data.ebb, ty); UseVarCases::SealedMultiplePredecessors(val, data.ebb) } } else { - let val = dfg.append_ebb_arg(data.ebb, ty); + let val = func.dfg.append_ebb_arg(data.ebb, ty); data.undef_variables.push((var, val)); UseVarCases::Unsealed(val) } @@ -271,7 +267,7 @@ where // The block has a single predecessor or multiple predecessor with // the same value, we look into it. UseVarCases::SealedOnePredecessor(pred) => { - let (val, mids) = self.use_var(dfg, layout, jts, var, ty, pred); + let (val, mids) = self.use_var(func, var, ty, pred); self.def_var(var, val, block); (val, mids) } @@ -285,7 +281,7 @@ where // If multiple predecessor we look up a use_var in each of them: // if they all yield the same value no need for an Ebb argument self.def_var(var, val, block); - self.predecessors_lookup(dfg, layout, jts, val, var, ebb) + self.predecessors_lookup(func, val, var, ebb) } } } @@ -354,13 +350,7 @@ where /// take into account the Phi function placed by the SSA algorithm. /// /// Returns the list of newly created ebbs for critical edge splitting. - pub fn seal_ebb_header_block( - &mut self, - ebb: Ebb, - dfg: &mut DataFlowGraph, - layout: &mut Layout, - jts: &mut JumpTables, - ) -> SideEffects { + pub fn seal_ebb_header_block(&mut self, ebb: Ebb, func: &mut Function) -> SideEffects { let block = self.header_block(ebb); let (undef_vars, ebb): (Vec<(Variable, Value)>, Ebb) = match self.blocks[block] { @@ -378,7 +368,7 @@ where // For each undef var we look up values in the predecessors and create an Ebb argument // only if necessary. for (var, val) in undef_vars { - let (_, local_side_effects) = self.predecessors_lookup(dfg, layout, jts, val, var, ebb); + let (_, local_side_effects) = self.predecessors_lookup(func, val, var, ebb); side_effects.append(local_side_effects); } @@ -399,15 +389,13 @@ where /// list of Ebb that are the middle of newly created critical edges splits. fn predecessors_lookup( &mut self, - dfg: &mut DataFlowGraph, - layout: &mut Layout, - jts: &mut JumpTables, + func: &mut Function, temp_arg_val: Value, temp_arg_var: Variable, dest_ebb: Ebb, ) -> (Value, SideEffects) { let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero(); - let ty = dfg.value_type(temp_arg_val); + let ty = func.dfg.value_type(temp_arg_val); let mut side_effects = SideEffects::new(); // Iterate over the predecessors. To avoid borrowing `self` for the whole loop, @@ -417,8 +405,7 @@ where for &(pred, _) in &preds { // For each predecessor, we query what is the local SSA value corresponding // to var and we put it as an argument of the branch instruction. - let (pred_val, local_side_effects) = - self.use_var(dfg, layout, jts, temp_arg_var, ty, pred); + let (pred_val, local_side_effects) = self.use_var(func, temp_arg_var, ty, pred); match pred_values { ZeroOneOrMore::Zero() => { if pred_val != temp_arg_val { @@ -441,17 +428,17 @@ where // The variable is used but never defined before. This is an irregularity in the // code, but rather than throwing an error we silently initialize the variable to // 0. This will have no effect since this situation happens in unreachable code. - if !layout.is_ebb_inserted(dest_ebb) { - layout.append_ebb(dest_ebb) + if !func.layout.is_ebb_inserted(dest_ebb) { + func.layout.append_ebb(dest_ebb) }; - let mut cur = Cursor::new(layout).at_first_insertion_point(dest_ebb); - let ty = dfg.value_type(temp_arg_val); + let ty = func.dfg.value_type(temp_arg_val); + let mut cur = FuncCursor::new(func).at_first_insertion_point(dest_ebb); let val = if ty.is_int() { - dfg.ins(&mut cur).iconst(ty, 0) + cur.ins().iconst(ty, 0) } else if ty == F32 { - dfg.ins(&mut cur).f32const(Ieee32::with_bits(0)) + cur.ins().f32const(Ieee32::with_bits(0)) } else if ty == F64 { - dfg.ins(&mut cur).f64const(Ieee64::with_bits(0)) + cur.ins().f64const(Ieee64::with_bits(0)) } else { panic!("value used but never declared and initialization not supported") }; @@ -463,8 +450,8 @@ where // so we don't need to have it as an ebb argument. // We need to replace all the occurences of val with pred_val but since // we can't afford a re-writing pass right now we just declare an alias. - dfg.remove_ebb_arg(temp_arg_val); - dfg.change_to_alias(temp_arg_val, pred_val); + func.dfg.remove_ebb_arg(temp_arg_val); + func.dfg.change_to_alias(temp_arg_val, pred_val); pred_val } ZeroOneOrMore::More() => { @@ -480,14 +467,12 @@ where .unwrap(); if let Some((middle_ebb, middle_block, middle_jump_inst)) = self.append_jump_argument( - dfg, - layout, + func, *last_inst, *pred_block, dest_ebb, pred_val, temp_arg_var, - jts, ) { *pred_block = middle_block; @@ -511,41 +496,39 @@ where /// critical edge splitting. fn append_jump_argument( &mut self, - dfg: &mut DataFlowGraph, - layout: &mut Layout, + func: &mut Function, jump_inst: Inst, jump_inst_block: Block, dest_ebb: Ebb, val: Value, var: Variable, - jts: &mut JumpTables, ) -> Option<(Ebb, Block, Inst)> { - match dfg[jump_inst].analyze_branch(&dfg.value_lists) { + match func.dfg[jump_inst].analyze_branch(&func.dfg.value_lists) { BranchInfo::NotABranch => { panic!("you have declared a non-branch instruction as a predecessor to an ebb"); } // For a single destination appending a jump argument to the instruction // is sufficient. BranchInfo::SingleDest(_, _) => { - dfg.append_inst_arg(jump_inst, val); + func.dfg.append_inst_arg(jump_inst, val); None } BranchInfo::Table(jt) => { // In the case of a jump table, the situation is tricky because br_table doesn't // support arguments. // We have to split the critical edge - let middle_ebb = dfg.make_ebb(); - layout.append_ebb(middle_ebb); + let middle_ebb = func.dfg.make_ebb(); + func.layout.append_ebb(middle_ebb); let block = self.declare_ebb_header_block(middle_ebb); self.blocks[block].add_predecessor(jump_inst_block, jump_inst); - self.seal_ebb_header_block(middle_ebb, dfg, layout, jts); - for old_dest in jts[jt].as_mut_slice() { + self.seal_ebb_header_block(middle_ebb, func); + for old_dest in func.jump_tables[jt].as_mut_slice() { if old_dest.unwrap() == dest_ebb { *old_dest = PackedOption::from(middle_ebb); } } - let mut cur = Cursor::new(layout).at_bottom(middle_ebb); - let middle_jump_inst = dfg.ins(&mut cur).jump(dest_ebb, &[val]); + let mut cur = FuncCursor::new(func).at_bottom(middle_ebb); + let middle_jump_inst = cur.ins().jump(dest_ebb, &[val]); self.def_var(var, val, block); Some((middle_ebb, block, middle_jump_inst)) } @@ -581,8 +564,9 @@ where #[cfg(test)] mod tests { + use cretonne::cursor::{Cursor, FuncCursor}; use cretonne::entity::EntityRef; - use cretonne::ir::{Function, InstBuilder, Cursor, CursorBase, Inst, JumpTableData}; + use cretonne::ir::{Function, InstBuilder, Inst, JumpTableData}; use cretonne::ir::types::*; use cretonne::verify_function; use cretonne::ir::instructions::BranchInfo; @@ -623,105 +607,37 @@ mod tests { let block = ssa.declare_ebb_header_block(ebb0); let x_var = Variable(0); let x_ssa = { - let cur = &mut Cursor::new(&mut func.layout); + let mut cur = FuncCursor::new(&mut func); cur.insert_ebb(ebb0); - func.dfg.ins(cur).iconst(I32, 1) + cur.ins().iconst(I32, 1) }; ssa.def_var(x_var, x_ssa, block); let y_var = Variable(1); let y_ssa = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).iconst(I32, 2) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().iconst(I32, 2) }; ssa.def_var(y_var, y_ssa, block); - assert_eq!( - ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block, - ).0, - x_ssa - ); - assert_eq!( - ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block, - ).0, - y_ssa - ); + assert_eq!(ssa.use_var(&mut func, x_var, I32, block).0, x_ssa); + assert_eq!(ssa.use_var(&mut func, y_var, I32, block).0, y_ssa); let z_var = Variable(2); - let x_use1 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block, - ).0; - let y_use1 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block, - ).0; + let x_use1 = ssa.use_var(&mut func, x_var, I32, block).0; + let y_use1 = ssa.use_var(&mut func, y_var, I32, block).0; let z1_ssa = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).iadd(x_use1, y_use1) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().iadd(x_use1, y_use1) }; ssa.def_var(z_var, z1_ssa, block); - assert_eq!( - ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block, - ).0, - z1_ssa - ); - let x_use2 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block, - ).0; - let z_use1 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block, - ).0; + assert_eq!(ssa.use_var(&mut func, z_var, I32, block).0, z1_ssa); + let x_use2 = ssa.use_var(&mut func, x_var, I32, block).0; + let z_use1 = ssa.use_var(&mut func, z_var, I32, block).0; let z2_ssa = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).iadd(x_use2, z_use1) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().iadd(x_use2, z_use1) }; ssa.def_var(z_var, z2_ssa, block); - assert_eq!( - ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block, - ).0, - z2_ssa - ); + assert_eq!(ssa.use_var(&mut func, z_var, I32, block).0, z2_ssa); } #[test] @@ -743,146 +659,57 @@ mod tests { let block0 = ssa.declare_ebb_header_block(ebb0); let x_var = Variable(0); let x_ssa = { - let cur = &mut Cursor::new(&mut func.layout); + let mut cur = FuncCursor::new(&mut func); cur.insert_ebb(ebb0); cur.insert_ebb(ebb1); cur.goto_bottom(ebb0); - func.dfg.ins(cur).iconst(I32, 1) + cur.ins().iconst(I32, 1) }; ssa.def_var(x_var, x_ssa, block0); let y_var = Variable(1); let y_ssa = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).iconst(I32, 2) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().iconst(I32, 2) }; ssa.def_var(y_var, y_ssa, block0); - assert_eq!( - ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block0, - ).0, - x_ssa - ); - assert_eq!( - ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block0, - ).0, - y_ssa - ); + assert_eq!(ssa.use_var(&mut func, x_var, I32, block0).0, x_ssa); + assert_eq!(ssa.use_var(&mut func, y_var, I32, block0).0, y_ssa); let z_var = Variable(2); - let x_use1 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block0, - ).0; - let y_use1 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block0, - ).0; + let x_use1 = ssa.use_var(&mut func, x_var, I32, block0).0; + let y_use1 = ssa.use_var(&mut func, y_var, I32, block0).0; let z1_ssa = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).iadd(x_use1, y_use1) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().iadd(x_use1, y_use1) }; ssa.def_var(z_var, z1_ssa, block0); - assert_eq!( - ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block0, - ).0, - z1_ssa - ); - let y_use2 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block0, - ).0; + assert_eq!(ssa.use_var(&mut func, z_var, I32, block0).0, z1_ssa); + let y_use2 = ssa.use_var(&mut func, y_var, I32, block0).0; let jump_inst: Inst = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).brnz(y_use2, ebb1, &[]) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().brnz(y_use2, ebb1, &[]) }; let block1 = ssa.declare_ebb_body_block(block0); - let x_use2 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block1, - ).0; + let x_use2 = ssa.use_var(&mut func, x_var, I32, block1).0; assert_eq!(x_use2, x_ssa); - let z_use1 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block1, - ).0; + let z_use1 = ssa.use_var(&mut func, z_var, I32, block1).0; assert_eq!(z_use1, z1_ssa); let z2_ssa = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).iadd(x_use2, z_use1) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().iadd(x_use2, z_use1) }; ssa.def_var(z_var, z2_ssa, block1); - assert_eq!( - ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block1, - ).0, - z2_ssa - ); - ssa.seal_ebb_header_block(ebb0, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + assert_eq!(ssa.use_var(&mut func, z_var, I32, block1).0, z2_ssa); + ssa.seal_ebb_header_block(ebb0, &mut func); let block2 = ssa.declare_ebb_header_block(ebb1); ssa.declare_ebb_predecessor(ebb1, block0, jump_inst); - ssa.seal_ebb_header_block(ebb1, &mut func.dfg, &mut func.layout, &mut func.jump_tables); - let x_use3 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block2, - ).0; + ssa.seal_ebb_header_block(ebb1, &mut func); + let x_use3 = ssa.use_var(&mut func, x_var, I32, block2).0; assert_eq!(x_ssa, x_use3); - let y_use3 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block2, - ).0; + let y_use3 = ssa.use_var(&mut func, y_var, I32, block2).0; assert_eq!(y_ssa, y_use3); let y2_ssa = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); - func.dfg.ins(cur).iadd(x_use3, y_use3) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().iadd(x_use3, y_use3) }; ssa.def_var(y_var, y2_ssa, block2); match func.dfg[jump_inst].analyze_branch(&func.dfg.value_lists) { @@ -917,179 +744,89 @@ mod tests { // jump ebb1 let block0 = ssa.declare_ebb_header_block(ebb0); - ssa.seal_ebb_header_block(ebb0, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + ssa.seal_ebb_header_block(ebb0, &mut func); let x_var = Variable(0); let x1 = { - let cur = &mut Cursor::new(&mut func.layout); + let mut cur = FuncCursor::new(&mut func); cur.insert_ebb(ebb0); cur.insert_ebb(ebb1); cur.insert_ebb(ebb2); cur.goto_bottom(ebb0); - func.dfg.ins(cur).iconst(I32, 1) + cur.ins().iconst(I32, 1) }; ssa.def_var(x_var, x1, block0); - assert_eq!( - ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block0, - ).0, - x1 - ); + assert_eq!(ssa.use_var(&mut func, x_var, I32, block0).0, x1); let y_var = Variable(1); let y1 = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).iconst(I32, 2) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().iconst(I32, 2) }; ssa.def_var(y_var, y1, block0); - assert_eq!( - ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block0, - ).0, - y1 - ); + assert_eq!(ssa.use_var(&mut func, y_var, I32, block0).0, y1); let z_var = Variable(2); - let x2 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block0, - ).0; + let x2 = ssa.use_var(&mut func, x_var, I32, block0).0; assert_eq!(x2, x1); - let y2 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block0, - ).0; + let y2 = ssa.use_var(&mut func, y_var, I32, block0).0; assert_eq!(y2, y1); let z1 = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).iadd(x2, y2) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().iadd(x2, y2) }; ssa.def_var(z_var, z1, block0); let jump_ebb0_ebb1 = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).jump(ebb1, &[]) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().jump(ebb1, &[]) }; let block1 = ssa.declare_ebb_header_block(ebb1); ssa.declare_ebb_predecessor(ebb1, block0, jump_ebb0_ebb1); - let z2 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block1, - ).0; - let y3 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block1, - ).0; + let z2 = ssa.use_var(&mut func, z_var, I32, block1).0; + let y3 = ssa.use_var(&mut func, y_var, I32, block1).0; let z3 = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); - func.dfg.ins(cur).iadd(z2, y3) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1); + cur.ins().iadd(z2, y3) }; ssa.def_var(z_var, z3, block1); - let y4 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block1, - ).0; + let y4 = ssa.use_var(&mut func, y_var, I32, block1).0; assert_eq!(y4, y3); let jump_ebb1_ebb2 = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); - func.dfg.ins(cur).brnz(y4, ebb2, &[]) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1); + cur.ins().brnz(y4, ebb2, &[]) }; let block2 = ssa.declare_ebb_body_block(block1); - let z4 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block2, - ).0; + let z4 = ssa.use_var(&mut func, z_var, I32, block2).0; assert_eq!(z4, z3); - let x3 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block2, - ).0; + let x3 = ssa.use_var(&mut func, x_var, I32, block2).0; let z5 = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); - func.dfg.ins(cur).isub(z4, x3) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1); + cur.ins().isub(z4, x3) }; ssa.def_var(z_var, z5, block2); - let y5 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block2, - ).0; + let y5 = ssa.use_var(&mut func, y_var, I32, block2).0; assert_eq!(y5, y3); { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); - func.dfg.ins(cur).return_(&[y5]) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1); + cur.ins().return_(&[y5]) }; let block3 = ssa.declare_ebb_header_block(ebb2); ssa.declare_ebb_predecessor(ebb2, block1, jump_ebb1_ebb2); - ssa.seal_ebb_header_block(ebb2, &mut func.dfg, &mut func.layout, &mut func.jump_tables); - let y6 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block3, - ).0; + ssa.seal_ebb_header_block(ebb2, &mut func); + let y6 = ssa.use_var(&mut func, y_var, I32, block3).0; assert_eq!(y6, y3); - let x4 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block3, - ).0; + let x4 = ssa.use_var(&mut func, x_var, I32, block3).0; assert_eq!(x4, x3); let y7 = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb2); - func.dfg.ins(cur).isub(y6, x4) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb2); + cur.ins().isub(y6, x4) }; ssa.def_var(y_var, y7, block3); let jump_ebb2_ebb1 = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb2); - func.dfg.ins(cur).jump(ebb1, &[]) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb2); + cur.ins().jump(ebb1, &[]) }; ssa.declare_ebb_predecessor(ebb1, block3, jump_ebb2_ebb1); - ssa.seal_ebb_header_block(ebb1, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + ssa.seal_ebb_header_block(ebb1, &mut func); assert_eq!(func.dfg.ebb_args(ebb1)[0], z2); assert_eq!(func.dfg.ebb_args(ebb1)[1], y3); assert_eq!(func.dfg.resolve_aliases(x3), x1); @@ -1114,60 +851,46 @@ mod tests { // return // let block0 = ssa.declare_ebb_header_block(ebb0); - ssa.seal_ebb_header_block(ebb0, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + ssa.seal_ebb_header_block(ebb0, &mut func); let x_var = Variable(0); let x1 = { - let cur = &mut Cursor::new(&mut func.layout); + let mut cur = FuncCursor::new(&mut func); cur.insert_ebb(ebb0); cur.insert_ebb(ebb1); cur.goto_bottom(ebb0); - func.dfg.ins(cur).iconst(I32, 1) + cur.ins().iconst(I32, 1) }; ssa.def_var(x_var, x1, block0); let mut data = JumpTableData::new(); data.push_entry(ebb1); let jt = func.create_jump_table(data); - ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block0, - ).0; + ssa.use_var(&mut func, x_var, I32, block0).0; let br_table = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).br_table(x1, jt) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().br_table(x1, jt) }; let block1 = ssa.declare_ebb_body_block(block0); let x3 = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).iconst(I32, 2) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().iconst(I32, 2) }; ssa.def_var(x_var, x3, block1); let jump_inst = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).jump(ebb1, &[]) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().jump(ebb1, &[]) }; let block2 = ssa.declare_ebb_header_block(ebb1); ssa.declare_ebb_predecessor(ebb1, block1, jump_inst); ssa.declare_ebb_predecessor(ebb1, block0, br_table); - ssa.seal_ebb_header_block(ebb1, &mut func.dfg, &mut func.layout, &mut func.jump_tables); - let x4 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block2, - ).0; + ssa.seal_ebb_header_block(ebb1, &mut func); + let x4 = ssa.use_var(&mut func, x_var, I32, block2).0; { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); - func.dfg.ins(cur).iadd_imm(x4, 1) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1); + cur.ins().iadd_imm(x4, 1) }; { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); - func.dfg.ins(cur).return_(&[]) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1); + cur.ins().return_(&[]) }; let flags = settings::Flags::new(&settings::builder()); match verify_function(&func, &flags) { @@ -1197,82 +920,54 @@ mod tests { let x_var = Variable(0); let y_var = Variable(1); let z_var = Variable(2); - ssa.seal_ebb_header_block(ebb0, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + ssa.seal_ebb_header_block(ebb0, &mut func); let x1 = { - let cur = &mut Cursor::new(&mut func.layout); + let mut cur = FuncCursor::new(&mut func); cur.insert_ebb(ebb0); cur.insert_ebb(ebb1); cur.goto_bottom(ebb0); - func.dfg.ins(cur).iconst(I32, 0) + cur.ins().iconst(I32, 0) }; ssa.def_var(x_var, x1, block0); let y1 = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).iconst(I32, 1) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().iconst(I32, 1) }; ssa.def_var(y_var, y1, block0); let z1 = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).iconst(I32, 2) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().iconst(I32, 2) }; ssa.def_var(z_var, z1, block0); let jump_inst = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb0); - func.dfg.ins(cur).jump(ebb1, &[]) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); + cur.ins().jump(ebb1, &[]) }; let block1 = ssa.declare_ebb_header_block(ebb1); ssa.declare_ebb_predecessor(ebb1, block0, jump_inst); - let z2 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - z_var, - I32, - block1, - ).0; + let z2 = ssa.use_var(&mut func, z_var, I32, block1).0; assert_eq!(func.dfg.ebb_args(ebb1)[0], z2); - let x2 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block1, - ).0; + let x2 = ssa.use_var(&mut func, x_var, I32, block1).0; assert_eq!(func.dfg.ebb_args(ebb1)[1], x2); let x3 = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); - func.dfg.ins(cur).iadd(x2, z2) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1); + cur.ins().iadd(x2, z2) }; ssa.def_var(x_var, x3, block1); - let x4 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - x_var, - I32, - block1, - ).0; - let y3 = ssa.use_var( - &mut func.dfg, - &mut func.layout, - &mut func.jump_tables, - y_var, - I32, - block1, - ).0; + let x4 = ssa.use_var(&mut func, x_var, I32, block1).0; + let y3 = ssa.use_var(&mut func, y_var, I32, block1).0; assert_eq!(func.dfg.ebb_args(ebb1)[2], y3); let y4 = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); - func.dfg.ins(cur).isub(y3, x4) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1); + cur.ins().isub(y3, x4) }; ssa.def_var(y_var, y4, block1); let jump_inst = { - let cur = &mut Cursor::new(&mut func.layout).at_bottom(ebb1); - func.dfg.ins(cur).jump(ebb1, &[]) + let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1); + cur.ins().jump(ebb1, &[]) }; ssa.declare_ebb_predecessor(ebb1, block1, jump_inst); - ssa.seal_ebb_header_block(ebb1, &mut func.dfg, &mut func.layout, &mut func.jump_tables); + ssa.seal_ebb_header_block(ebb1, &mut func); // At sealing the "z" argument disappear but the remaining "x" and "y" args have to be // in the right order. assert_eq!(func.dfg.ebb_args(ebb1)[1], y3); From 8e93aa7ce71f747ab1a0c7a0c820f4666f05d832 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 21 Sep 2017 13:06:28 -0700 Subject: [PATCH 1138/3084] Remove layout::Cursor from the legalizer::split API. We still use the cursor internally, but don't require callers to use it too. --- lib/cretonne/meta/gen_legalizer.py | 12 ++++---- lib/cretonne/src/ir/layout.rs | 9 ++++++ lib/cretonne/src/legalizer/boundary.rs | 6 ++-- lib/cretonne/src/legalizer/split.rs | 38 ++++++++++++++------------ 4 files changed, 39 insertions(+), 26 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index f3e5067420..a79b05da76 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -264,12 +264,12 @@ def emit_dst_inst(node, fmt): # Split instructions are not emitted with the builder, but by calling # special functions in the `legalizer::split` module. These functions # will eliminate concat-split patterns. - fmt.line( - 'let {} = split::{}(dfg, cfg, pos, {});' - .format( - wrap_tup(node.defs), - node.expr.inst.snake_name(), - node.expr.args[0])) + fmt.line('let curpos = pos.position();') + fmt.format( + 'let {} = split::{}(dfg, pos.layout, cfg, curpos, {});', + wrap_tup(node.defs), + node.expr.inst.snake_name(), + node.expr.args[0]) else: if len(node.defs) == 0: # This node doesn't define any values, so just insert the new diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 2139848382..877701a1a1 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -709,6 +709,15 @@ pub trait CursorBase { /// Borrow a mutable reference to the function layout that this cursor is navigating. fn layout_mut(&mut self) -> &mut Layout; + /// Rebuild this cursor positioned at `pos`. + fn at_position(mut self, pos: CursorPosition) -> Self + where + Self: Sized, + { + self.set_position(pos); + self + } + /// Rebuild this cursor positioned at `inst`. /// /// This is intended to be used as a builder method: diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index b3e7b64658..dfeb88b777 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -331,12 +331,14 @@ fn convert_to_abi( let ty = dfg.value_type(value); match legalize_abi_value(ty, &arg_type) { ValueConversion::IntSplit => { - let (lo, hi) = isplit(dfg, cfg, pos, value); + let curpos = pos.position(); + let (lo, hi) = isplit(dfg, pos.layout, cfg, curpos, value); convert_to_abi(dfg, cfg, pos, lo, put_arg); convert_to_abi(dfg, cfg, pos, hi, put_arg); } ValueConversion::VectorSplit => { - let (lo, hi) = vsplit(dfg, cfg, pos, value); + let curpos = pos.position(); + let (lo, hi) = vsplit(dfg, pos.layout, cfg, curpos, value); convert_to_abi(dfg, cfg, pos, lo, put_arg); convert_to_abi(dfg, cfg, pos, hi, put_arg); } diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index 23c0e02bcd..f3df6b7b82 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -64,31 +64,33 @@ //! It is possible to have circular dependencies of EBB arguments that are never used by any real //! instructions. These loops will remain in the program. +use cursor::{Cursor, CursorPosition}; use flowgraph::ControlFlowGraph; -use ir::{DataFlowGraph, Ebb, Inst, Cursor, CursorBase, Value, Type, Opcode, ValueDef, - InstructionData, InstBuilder}; +use ir::{self, Ebb, Inst, Value, Type, Opcode, ValueDef, InstructionData, InstBuilder}; use std::iter; /// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values /// if possible. pub fn isplit( - dfg: &mut DataFlowGraph, + dfg: &mut ir::DataFlowGraph, + layout: &mut ir::Layout, cfg: &ControlFlowGraph, - pos: &mut Cursor, + pos: CursorPosition, value: Value, ) -> (Value, Value) { - split_any(dfg, cfg, pos, value, Opcode::Iconcat) + split_any(dfg, layout, cfg, pos, value, Opcode::Iconcat) } /// Split `value` into halves using the `vsplit` semantics. Do this by reusing existing values if /// possible. pub fn vsplit( - dfg: &mut DataFlowGraph, + dfg: &mut ir::DataFlowGraph, + layout: &mut ir::Layout, cfg: &ControlFlowGraph, - pos: &mut Cursor, + pos: CursorPosition, value: Value, ) -> (Value, Value) { - split_any(dfg, cfg, pos, value, Opcode::Vconcat) + split_any(dfg, layout, cfg, pos, value, Opcode::Vconcat) } /// After splitting an EBB argument, we need to go back and fix up all of the predecessor @@ -110,15 +112,16 @@ struct Repair { /// Generic version of `isplit` and `vsplit` controlled by the `concat` opcode. fn split_any( - dfg: &mut DataFlowGraph, + dfg: &mut ir::DataFlowGraph, + layout: &mut ir::Layout, cfg: &ControlFlowGraph, - pos: &mut Cursor, + pos: CursorPosition, value: Value, concat: Opcode, ) -> (Value, Value) { - let saved_pos = pos.position(); let mut repairs = Vec::new(); - let result = split_value(dfg, pos, value, concat, &mut repairs); + let mut pos = ir::Cursor::new(layout).at_position(pos); + let result = split_value(dfg, &mut pos, value, concat, &mut repairs); // We have split the value requested, and now we may need to fix some EBB predecessors. while let Some(repair) = repairs.pop() { @@ -147,7 +150,7 @@ fn split_any( // Split the old argument, possibly causing more repairs to be scheduled. pos.goto_inst(inst); - let (lo, hi) = split_value(dfg, pos, old_arg, repair.concat, &mut repairs); + let (lo, hi) = split_value(dfg, &mut pos, old_arg, repair.concat, &mut repairs); // The `lo` part replaces the original argument. *args.get_mut(fixed_args + repair.num, &mut dfg.value_lists) @@ -173,7 +176,6 @@ fn split_any( } } - pos.set_position(saved_pos); result } @@ -184,8 +186,8 @@ fn split_any( /// /// Return the two new values representing the parts of `value`. fn split_value( - dfg: &mut DataFlowGraph, - pos: &mut Cursor, + dfg: &mut ir::DataFlowGraph, + pos: &mut ir::Cursor, value: Value, concat: Opcode, repairs: &mut Vec, @@ -292,7 +294,7 @@ fn add_repair( /// ``` /// /// This function resolves `v11` to `v1` and `v12` to `v2`. -fn resolve_splits(dfg: &DataFlowGraph, value: Value) -> Value { +fn resolve_splits(dfg: &ir::DataFlowGraph, value: Value) -> Value { let value = dfg.resolve_copies(value); // Deconstruct a split instruction. @@ -330,7 +332,7 @@ fn resolve_splits(dfg: &DataFlowGraph, value: Value) -> Value { /// /// After legalizing the instructions computing the value that was split, it is likely that we can /// avoid depending on the split instruction. Its input probably comes from a concatenation. -pub fn simplify_branch_arguments(dfg: &mut DataFlowGraph, branch: Inst) { +pub fn simplify_branch_arguments(dfg: &mut ir::DataFlowGraph, branch: Inst) { let mut new_args = Vec::new(); for &arg in dfg.inst_args(branch) { From 218959ff654446cc3c84e0867ae33912fe91f3bd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 21 Sep 2017 13:29:45 -0700 Subject: [PATCH 1139/3084] Convert more tests to FuncCursor. --- lib/cretonne/src/ir/builder.rs | 41 +++++++++++++++++----------------- lib/cretonne/src/ir/dfg.rs | 38 +++++++++++++++---------------- 2 files changed, 39 insertions(+), 40 deletions(-) diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index fa0b5c69c8..4a662d9d14 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -214,53 +214,52 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { #[cfg(test)] mod tests { - use ir::{Function, Cursor, CursorBase, InstBuilder, ValueDef}; + use cursor::{Cursor, FuncCursor}; + use ir::{Function, InstBuilder, ValueDef}; use ir::types::*; use ir::condcodes::*; #[test] fn types() { let mut func = Function::new(); - let dfg = &mut func.dfg; - let ebb0 = dfg.make_ebb(); - let arg0 = dfg.append_ebb_arg(ebb0, I32); - let pos = &mut Cursor::new(&mut func.layout); + let ebb0 = func.dfg.make_ebb(); + let arg0 = func.dfg.append_ebb_arg(ebb0, I32); + let mut pos = FuncCursor::new(&mut func); pos.insert_ebb(ebb0); // Explicit types. - let v0 = dfg.ins(pos).iconst(I32, 3); - assert_eq!(dfg.value_type(v0), I32); + let v0 = pos.ins().iconst(I32, 3); + assert_eq!(pos.func.dfg.value_type(v0), I32); // Inferred from inputs. - let v1 = dfg.ins(pos).iadd(arg0, v0); - assert_eq!(dfg.value_type(v1), I32); + let v1 = pos.ins().iadd(arg0, v0); + assert_eq!(pos.func.dfg.value_type(v1), I32); // Formula. - let cmp = dfg.ins(pos).icmp(IntCC::Equal, arg0, v0); - assert_eq!(dfg.value_type(cmp), B1); + let cmp = pos.ins().icmp(IntCC::Equal, arg0, v0); + assert_eq!(pos.func.dfg.value_type(cmp), B1); } #[test] fn reuse_results() { let mut func = Function::new(); - let dfg = &mut func.dfg; - let ebb0 = dfg.make_ebb(); - let arg0 = dfg.append_ebb_arg(ebb0, I32); - let pos = &mut Cursor::new(&mut func.layout); + let ebb0 = func.dfg.make_ebb(); + let arg0 = func.dfg.append_ebb_arg(ebb0, I32); + let mut pos = FuncCursor::new(&mut func); pos.insert_ebb(ebb0); - let v0 = dfg.ins(pos).iadd_imm(arg0, 17); - assert_eq!(dfg.value_type(v0), I32); + let v0 = pos.ins().iadd_imm(arg0, 17); + assert_eq!(pos.func.dfg.value_type(v0), I32); let iadd = pos.prev_inst().unwrap(); - assert_eq!(dfg.value_def(v0), ValueDef::Res(iadd, 0)); + assert_eq!(pos.func.dfg.value_def(v0), ValueDef::Res(iadd, 0)); // Detach v0 and reuse it for a different instruction. - dfg.clear_results(iadd); - let v0b = dfg.ins(pos).with_result(v0).iconst(I32, 3); + pos.func.dfg.clear_results(iadd); + let v0b = pos.ins().with_result(v0).iconst(I32, 3); assert_eq!(v0, v0b); assert_eq!(pos.current_inst(), Some(iadd)); let iconst = pos.prev_inst().unwrap(); assert!(iadd != iconst); - assert_eq!(dfg.value_def(v0), ValueDef::Res(iconst, 0)); + assert_eq!(pos.func.dfg.value_def(v0), ValueDef::Res(iconst, 0)); } } diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 47c53100cf..25859535bc 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -897,8 +897,9 @@ impl<'a> fmt::Display for DisplayInst<'a> { #[cfg(test)] mod tests { use super::*; + use cursor::{Cursor, FuncCursor}; use ir::types; - use ir::{Function, Cursor, CursorBase, Opcode, InstructionData, TrapCode}; + use ir::{Function, Opcode, InstructionData, TrapCode}; #[test] fn make_inst() { @@ -1056,41 +1057,40 @@ mod tests { use ir::condcodes::IntCC; let mut func = Function::new(); - let dfg = &mut func.dfg; - let ebb0 = dfg.make_ebb(); - let pos = &mut Cursor::new(&mut func.layout); + let ebb0 = func.dfg.make_ebb(); + let mut pos = FuncCursor::new(&mut func); pos.insert_ebb(ebb0); // Build a little test program. - let v1 = dfg.ins(pos).iconst(types::I32, 42); + let v1 = pos.ins().iconst(types::I32, 42); // Make sure we can resolve value aliases even when values is empty. - assert_eq!(dfg.resolve_aliases(v1), v1); + assert_eq!(pos.func.dfg.resolve_aliases(v1), v1); - let arg0 = dfg.append_ebb_arg(ebb0, types::I32); - let (s, c) = dfg.ins(pos).iadd_cout(v1, arg0); - let iadd = match dfg.value_def(s) { + let arg0 = pos.func.dfg.append_ebb_arg(ebb0, types::I32); + let (s, c) = pos.ins().iadd_cout(v1, arg0); + let iadd = match pos.func.dfg.value_def(s) { ValueDef::Res(i, 0) => i, _ => panic!(), }; // Remove `c` from the result list. - dfg.clear_results(iadd); - dfg.attach_result(iadd, s); + pos.func.dfg.clear_results(iadd); + pos.func.dfg.attach_result(iadd, s); // Replace `iadd_cout` with a normal `iadd` and an `icmp`. - dfg.replace(iadd).iadd(v1, arg0); - let c2 = dfg.ins(pos).icmp(IntCC::UnsignedLessThan, s, v1); - dfg.change_to_alias(c, c2); + pos.func.dfg.replace(iadd).iadd(v1, arg0); + let c2 = pos.ins().icmp(IntCC::UnsignedLessThan, s, v1); + pos.func.dfg.change_to_alias(c, c2); - assert_eq!(dfg.resolve_aliases(c2), c2); - assert_eq!(dfg.resolve_aliases(c), c2); + assert_eq!(pos.func.dfg.resolve_aliases(c2), c2); + assert_eq!(pos.func.dfg.resolve_aliases(c), c2); // Make a copy of the alias. - let c3 = dfg.ins(pos).copy(c); + let c3 = pos.ins().copy(c); // This does not see through copies. - assert_eq!(dfg.resolve_aliases(c3), c3); + assert_eq!(pos.func.dfg.resolve_aliases(c3), c3); // But this goes through both copies and aliases. - assert_eq!(dfg.resolve_copies(c3), c2); + assert_eq!(pos.func.dfg.resolve_copies(c3), c2); } } From f524977920322a89d02bf1e3d73b3c030d80a764 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 20 Sep 2017 17:09:54 -0700 Subject: [PATCH 1140/3084] Factor some logic out of seal_ebb_header_block into parts. This allows append_jump_argument to call just the part that it needs instead of the whole of seal_ebb_header_block, which is a nice cleanup on its own, and it also eliminates a cycle from the static call graph. --- lib/frontend/src/ssa.rs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 0d3e00c5a9..cd0a6cf7f2 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -371,7 +371,12 @@ where let (_, local_side_effects) = self.predecessors_lookup(func, val, var, ebb); side_effects.append(local_side_effects); } + self.mark_ebb_header_block_sealed(block); + side_effects + } + /// Set the `sealed` flag for `block`. + fn mark_ebb_header_block_sealed(&mut self, block: Block) { // Then we mark the block as sealed. match self.blocks[block] { BlockData::EbbBody { .. } => panic!("this should not happen"), @@ -381,7 +386,6 @@ where data.sealed = true; } }; - side_effects } /// Look up in the predecessors of an Ebb the def for a value an decides wether or not @@ -521,7 +525,7 @@ where func.layout.append_ebb(middle_ebb); let block = self.declare_ebb_header_block(middle_ebb); self.blocks[block].add_predecessor(jump_inst_block, jump_inst); - self.seal_ebb_header_block(middle_ebb, func); + self.mark_ebb_header_block_sealed(block); for old_dest in func.jump_tables[jt].as_mut_slice() { if old_dest.unwrap() == dest_ebb { *old_dest = PackedOption::from(middle_ebb); From df5ba8bbbaaa745fb4a1f9acc233b448b369c20c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 20 Sep 2017 22:48:55 -0700 Subject: [PATCH 1141/3084] Correct a comment. --- lib/frontend/src/ssa.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index cd0a6cf7f2..33470e965e 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -264,8 +264,7 @@ where }; // TODO: avoid recursion for the calls to use_var and predecessors_lookup. match case { - // The block has a single predecessor or multiple predecessor with - // the same value, we look into it. + // The block has a single predecessor, we look into it. UseVarCases::SealedOnePredecessor(pred) => { let (val, mids) = self.use_var(func, var, ty, pred); self.def_var(var, val, block); From 9cda4eacdeab7063f918f92d2dd7fb33dcc30eb7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 20 Sep 2017 17:32:12 -0700 Subject: [PATCH 1142/3084] Minor refactor. --- lib/frontend/src/ssa.rs | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 33470e965e..5abd827e75 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -280,7 +280,7 @@ where // If multiple predecessor we look up a use_var in each of them: // if they all yield the same value no need for an Ebb argument self.def_var(var, val, block); - self.predecessors_lookup(func, val, var, ebb) + self.predecessors_lookup(func, val, var, ty, ebb) } } } @@ -367,7 +367,8 @@ where // For each undef var we look up values in the predecessors and create an Ebb argument // only if necessary. for (var, val) in undef_vars { - let (_, local_side_effects) = self.predecessors_lookup(func, val, var, ebb); + let ty = func.dfg.value_type(val); + let (_, local_side_effects) = self.predecessors_lookup(func, val, var, ty, ebb); side_effects.append(local_side_effects); } self.mark_ebb_header_block_sealed(block); @@ -395,10 +396,10 @@ where func: &mut Function, temp_arg_val: Value, temp_arg_var: Variable, + ty: Type, dest_ebb: Ebb, ) -> (Value, SideEffects) { let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero(); - let ty = func.dfg.value_type(temp_arg_val); let mut side_effects = SideEffects::new(); // Iterate over the predecessors. To avoid borrowing `self` for the whole loop, From 22b769b7160d1fce3000738fdd9853648b4e3a6d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 20 Sep 2017 21:35:34 -0700 Subject: [PATCH 1143/3084] Convert use_var and predecessors_lookup into a state machine to avoid recursion. --- lib/frontend/src/ssa.rs | 167 +++++++++++++++++++++++++++++----------- 1 file changed, 122 insertions(+), 45 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 5abd827e75..ee4aaf3a65 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -43,6 +43,12 @@ where blocks: PrimaryMap>, // Records the basic blocks at the beginning of the `Ebb`s. ebb_headers: EntityMap>, + + // Call and result stacks for use in the `use_var`/`predecessors_lookup` state machine. + calls: Vec, + results: Vec, + // Side effects accumulated in the `use_var`/`predecessors_lookup` state machine. + side_effects: SideEffects, } /// Side effects of a `use_var` or a `seal_ebb_header_block` method call. @@ -65,11 +71,8 @@ impl SideEffects { } } - fn append(&mut self, mut more: SideEffects) { - self.split_ebbs_created.append(&mut more.split_ebbs_created); - self.instructions_added_to_ebbs.append( - &mut more.instructions_added_to_ebbs, - ); + fn is_empty(&self) -> bool { + self.split_ebbs_created.is_empty() && self.instructions_added_to_ebbs.is_empty() } } @@ -149,6 +152,9 @@ where variables: EntityMap::with_default(EntityMap::new()), blocks: PrimaryMap::new(), ebb_headers: EntityMap::new(), + calls: Vec::new(), + results: Vec::new(), + side_effects: SideEffects::new(), } } @@ -158,6 +164,9 @@ where self.variables.clear(); self.blocks.clear(); self.ebb_headers.clear(); + debug_assert!(self.calls.is_empty()); + debug_assert!(self.results.is_empty()); + debug_assert!(self.side_effects.is_empty()); } } @@ -176,6 +185,13 @@ enum UseVarCases { SealedMultiplePredecessors(Value, Ebb), } +// States for the `use_var`/`predecessors_lookup` state machine. +enum Call { + UseVar(Block), + FinishSealedOnePredecessor(Block), + FinishPredecessorsLookup(Value, Ebb), +} + /// The following methods are the API of the SSA builder. Here is how it should be used when /// translating to Cretonne IL: /// @@ -230,18 +246,19 @@ where } // Otherwise, we have to do a non-local lookup. - self.use_var_nonlocal(func, var, ty, block) + debug_assert!(self.calls.is_empty()); + debug_assert!(self.results.is_empty()); + debug_assert!(self.side_effects.is_empty()); + self.use_var_nonlocal(func, var, ty, block); + ( + self.run_state_machine(func, var, ty), + mem::replace(&mut self.side_effects, SideEffects::new()), + ) } - // The non-local case of use_var. Query each predecessor for a value and add branch - // arguments as needed to satisfy the use. - fn use_var_nonlocal( - &mut self, - func: &mut Function, - var: Variable, - ty: Type, - block: Block, - ) -> (Value, SideEffects) { + /// Resolve a use of `var` in `block` in the case where there's no prior def + /// in `block`. + fn use_var_nonlocal(&mut self, func: &mut Function, var: Variable, ty: Type, block: Block) { let case = match self.blocks[block] { BlockData::EbbHeader(ref mut data) => { // The block has multiple predecessors so we append an Ebb argument that @@ -262,29 +279,34 @@ where } BlockData::EbbBody { predecessor: pred } => UseVarCases::SealedOnePredecessor(pred), }; - // TODO: avoid recursion for the calls to use_var and predecessors_lookup. match case { // The block has a single predecessor, we look into it. UseVarCases::SealedOnePredecessor(pred) => { - let (val, mids) = self.use_var(func, var, ty, pred); - self.def_var(var, val, block); - (val, mids) + self.calls.push(Call::FinishSealedOnePredecessor(block)); + self.calls.push(Call::UseVar(pred)); } // The block has multiple predecessors, we register the ebb argument as the current // definition for the variable. UseVarCases::Unsealed(val) => { self.def_var(var, val, block); - (val, SideEffects::new()) + self.results.push(val); } UseVarCases::SealedMultiplePredecessors(val, ebb) => { // If multiple predecessor we look up a use_var in each of them: // if they all yield the same value no need for an Ebb argument self.def_var(var, val, block); - self.predecessors_lookup(func, val, var, ty, ebb) + self.begin_predecessors_lookup(val, ebb); } } } + /// For blocks with a single predecessor, once we've determined the value, + /// record a local def for it for future queries to find. + fn finish_sealed_one_predecessor(&mut self, var: Variable, block: Block) { + let val = *self.results.last().unwrap(); + self.def_var(var, val, block); + } + /// Declares a new basic block belonging to the body of a certain `Ebb` and having `pred` /// as a predecessor. `pred` is the only predecessor of the block and the block is sealed /// at creation. @@ -363,16 +385,14 @@ where } }; - let mut side_effects = SideEffects::new(); // For each undef var we look up values in the predecessors and create an Ebb argument // only if necessary. for (var, val) in undef_vars { let ty = func.dfg.value_type(val); - let (_, local_side_effects) = self.predecessors_lookup(func, val, var, ty, ebb); - side_effects.append(local_side_effects); + self.predecessors_lookup(func, val, var, ty, ebb); } self.mark_ebb_header_block_sealed(block); - side_effects + mem::replace(&mut self.side_effects, SideEffects::new()) } /// Set the `sealed` flag for `block`. @@ -398,18 +418,46 @@ where temp_arg_var: Variable, ty: Type, dest_ebb: Ebb, - ) -> (Value, SideEffects) { - let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero(); - let mut side_effects = SideEffects::new(); + ) -> Value { + debug_assert!(self.calls.is_empty()); + debug_assert!(self.results.is_empty()); + // self.side_effects may be non-empty here so that callers can + // accumulate side effects over multiple calls. + self.begin_predecessors_lookup(temp_arg_val, dest_ebb); + self.run_state_machine(func, temp_arg_var, ty) + } - // Iterate over the predecessors. To avoid borrowing `self` for the whole loop, - // temporarily detach the predecessors list and replace it with an empty list. - // `use_var`'s traversal won't revisit these predecesors. - let mut preds = mem::replace(self.predecessors_mut(dest_ebb), Vec::new()); - for &(pred, _) in &preds { + /// Initiate use lookups in all predecessors of `dest_ebb`, and arrange for a call + /// to `finish_predecessors_lookup` once they complete. + fn begin_predecessors_lookup(&mut self, temp_arg_val: Value, dest_ebb: Ebb) { + self.calls.push(Call::FinishPredecessorsLookup( + temp_arg_val, + dest_ebb, + )); + // Iterate over the predecessors. + let mut calls = mem::replace(&mut self.calls, Vec::new()); + calls.extend(self.predecessors(dest_ebb).iter().rev().map(|&(pred, _)| { + Call::UseVar(pred) + })); + self.calls = calls; + } + + /// Examine the values from the predecessors and compute a result value, creating + /// block arguments as needed. + fn finish_predecessors_lookup( + &mut self, + func: &mut Function, + temp_arg_val: Value, + temp_arg_var: Variable, + dest_ebb: Ebb, + ) { + let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero(); + + // Iterate over the predecessors. + for _ in 0..self.predecessors(dest_ebb).len() { // For each predecessor, we query what is the local SSA value corresponding // to var and we put it as an argument of the branch instruction. - let (pred_val, local_side_effects) = self.use_var(func, temp_arg_var, ty, pred); + let pred_val = self.results.pop().unwrap(); match pred_values { ZeroOneOrMore::Zero() => { if pred_val != temp_arg_val { @@ -423,10 +471,7 @@ where } ZeroOneOrMore::More() => {} }; - side_effects.append(local_side_effects); } - debug_assert!(self.predecessors(dest_ebb).is_empty()); - let result_val = match pred_values { ZeroOneOrMore::Zero() => { // The variable is used but never defined before. This is an irregularity in the @@ -446,7 +491,7 @@ where } else { panic!("value used but never declared and initialization not supported") }; - side_effects.instructions_added_to_ebbs.push(dest_ebb); + self.side_effects.instructions_added_to_ebbs.push(dest_ebb); val } ZeroOneOrMore::One(pred_val) => { @@ -460,7 +505,9 @@ where } ZeroOneOrMore::More() => { // There is disagreement in the predecessors on which value to use so we have - // to keep the ebb argument. + // to keep the ebb argument. To avoid borrowing `self` for the whole loop, + // temporarily detach the predecessors list and replace it with an empty list. + let mut preds = mem::replace(self.predecessors_mut(dest_ebb), Vec::new()); for &mut (ref mut pred_block, ref mut last_inst) in &mut preds { // We already did a full `use_var` above, so we can do just the fast path. let pred_val = self.variables @@ -481,19 +528,18 @@ where { *pred_block = middle_block; *last_inst = middle_jump_inst; - side_effects.split_ebbs_created.push(middle_ebb); + self.side_effects.split_ebbs_created.push(middle_ebb); } } + // Now that we're done, move the predecessors list back. debug_assert!(self.predecessors(dest_ebb).is_empty()); + *self.predecessors_mut(dest_ebb) = preds; + temp_arg_val } }; - // Now that we're done, move the predecessors list back. - debug_assert!(self.predecessors(dest_ebb).is_empty()); - *self.predecessors_mut(dest_ebb) = preds; - - (result_val, side_effects) + self.results.push(result_val); } /// Appends a jump argument to a jump instruction, returns ebb created in case of @@ -564,6 +610,37 @@ where BlockData::EbbHeader(ref data) => data.sealed, } } + + /// The main algorithm is naturally recursive: when there's a `use_var` in a + /// block with no correspondin local defs, it recurses and performs a + /// `use_var` in each predecessor. To avoid risking running out of callstack + /// space, we keep an explicit stack and use a small state machine rather + /// than literal recursion. + fn run_state_machine(&mut self, func: &mut Function, var: Variable, ty: Type) -> Value { + // Process the calls scheduled in `self.calls` until it is empty. + while let Some(call) = self.calls.pop() { + match call { + Call::UseVar(block) => { + // First we lookup for the current definition of the variable in this block + if let Some(var_defs) = self.variables.get(var) { + if let Some(val) = var_defs[block].expand() { + self.results.push(val); + continue; + } + } + self.use_var_nonlocal(func, var, ty, block); + } + Call::FinishSealedOnePredecessor(block) => { + self.finish_sealed_one_predecessor(var, block); + } + Call::FinishPredecessorsLookup(temp_arg_val, dest_ebb) => { + self.finish_predecessors_lookup(func, temp_arg_val, var, dest_ebb); + } + } + } + debug_assert_eq!(self.results.len(), 1); + self.results.pop().unwrap() + } } #[cfg(test)] From ff18772d0e7b69989cda187d60f93d31d0635c0a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 21 Sep 2017 09:24:18 -0700 Subject: [PATCH 1144/3084] Avoid a heap allocation. --- lib/frontend/src/frontend.rs | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 75c7dfe481..82ae583e1e 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -138,7 +138,6 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short None => { // branch_destination() doesn't detect jump_tables // If jump table we declare all entries successor -// TODO: not collect with vector? if let InstructionData::BranchTable { table, .. } = data { // Unlike all other jumps/branches, jump tables are // capable of having the same successor appear @@ -151,10 +150,13 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short .expect("you are referencing an undeclared jump table") .entries() .map(|(_, ebb)| ebb) - .filter(|dest_ebb| unique.insert(*dest_ebb)) - .collect::>() { - self.builder.declare_successor(dest_ebb, inst) - } + .filter(|dest_ebb| unique.insert(*dest_ebb)) { + self.builder.builder.ssa.declare_ebb_predecessor( + dest_ebb, + self.builder.position.basic_block, + inst, + ) + } } } } From dda82936686fa4ac884c9ce9da4eaf2fe23e0b7d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 21 Sep 2017 09:30:33 -0700 Subject: [PATCH 1145/3084] Restructure code to avoid a heap allocation. --- lib/frontend/src/frontend.rs | 84 ++++++++++++++++-------------------- 1 file changed, 38 insertions(+), 46 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 82ae583e1e..e1cdbdf53a 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -543,14 +543,14 @@ where fn ebb_args_adjustement(&mut self, dest_ebb: Ebb, jump_args: &[Type]) { - let ty_to_append: Option> = - if self.builder.ssa.predecessors(dest_ebb).is_empty() || - self.builder.ebbs[dest_ebb].pristine - { - // This is the first jump instruction targeting this Ebb - // so the jump arguments supplied here are this Ebb' arguments - // However some of the arguments might already be there - // in the Ebb so we have to check they're consistent + if self.builder.ssa.predecessors(dest_ebb).is_empty() || + self.builder.ebbs[dest_ebb].pristine + { + // This is the first jump instruction targeting this Ebb + // so the jump arguments supplied here are this Ebb' arguments + // However some of the arguments might already be there + // in the Ebb so we have to check they're consistent + let dest_ebb_args_len = { let dest_ebb_args = self.func.dfg.ebb_args(dest_ebb); debug_assert!( dest_ebb_args @@ -560,48 +560,40 @@ where *jump_arg == self.func.dfg.value_type(*dest_arg) }), "the jump argument supplied has not the \ - same type as the corresponding dest ebb argument" - ); - self.builder.ebbs[dest_ebb].user_arg_count = jump_args.len(); - Some( - jump_args - .iter() - .skip(dest_ebb_args.len()) - .cloned() - .collect(), - ) - } else { - let dest_ebb_args = self.func.dfg.ebb_args(dest_ebb); - // The Ebb already has predecessors - // We check that the arguments supplied match those supplied - // previously. - debug_assert_eq!( - jump_args.len(), - self.builder.ebbs[dest_ebb].user_arg_count, - "the jump instruction doesn't have the same \ - number of arguments as its destination Ebb \ - ({} vs {}).", - jump_args.len(), - dest_ebb_args.len() - ); - debug_assert!( - jump_args - .iter() - .zip(dest_ebb_args.iter().take( - self.builder.ebbs[dest_ebb].user_arg_count, - )) - .all(|(jump_arg, dest_arg)| { - *jump_arg == self.func.dfg.value_type(*dest_arg) - }), - "the jump argument supplied has not the \ same type as the corresponding dest ebb argument" ); - None + dest_ebb_args.len() }; - if let Some(ty_args) = ty_to_append { - for ty in ty_args { - self.func.dfg.append_ebb_arg(dest_ebb, ty); + self.builder.ebbs[dest_ebb].user_arg_count = jump_args.len(); + for ty in jump_args.iter().skip(dest_ebb_args_len) { + self.func.dfg.append_ebb_arg(dest_ebb, *ty); } + } else { + let dest_ebb_args = self.func.dfg.ebb_args(dest_ebb); + // The Ebb already has predecessors + // We check that the arguments supplied match those supplied + // previously. + debug_assert_eq!( + jump_args.len(), + self.builder.ebbs[dest_ebb].user_arg_count, + "the jump instruction doesn't have the same \ + number of arguments as its destination Ebb \ + ({} vs {}).", + jump_args.len(), + dest_ebb_args.len() + ); + debug_assert!( + jump_args + .iter() + .zip(dest_ebb_args.iter().take( + self.builder.ebbs[dest_ebb].user_arg_count, + )) + .all(|(jump_arg, dest_arg)| { + *jump_arg == self.func.dfg.value_type(*dest_arg) + }), + "the jump argument supplied has not the \ + same type as the corresponding dest ebb argument" + ); } } From e75558b8005c8450b9e77f21f8e2ffa1316599b0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 21 Sep 2017 09:31:50 -0700 Subject: [PATCH 1146/3084] Fix spelling of "adjustment". --- lib/frontend/src/frontend.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index e1cdbdf53a..f2ae3b692b 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -132,7 +132,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short } _ => panic!("should not happen"), }; - self.builder.ebb_args_adjustement(dest_ebb, &args_types); + self.builder.ebb_args_adjustment(dest_ebb, &args_types); self.builder.declare_successor(dest_ebb, inst); } None => { @@ -263,7 +263,7 @@ where let basic_block = self.builder.ssa.header_block(ebb); // Then we change the cursor position. self.position = Position { ebb, basic_block }; - self.ebb_args_adjustement(ebb, jump_args); + self.ebb_args_adjustment(ebb, jump_args); self.func.dfg.ebb_args(ebb) } @@ -542,7 +542,7 @@ where } - fn ebb_args_adjustement(&mut self, dest_ebb: Ebb, jump_args: &[Type]) { + fn ebb_args_adjustment(&mut self, dest_ebb: Ebb, jump_args: &[Type]) { if self.builder.ssa.predecessors(dest_ebb).is_empty() || self.builder.ebbs[dest_ebb].pristine { From e5e4b59683c055e35328208a0954413c7268d4d5 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 21 Sep 2017 09:39:31 -0700 Subject: [PATCH 1147/3084] Tidy up some asserts. --- lib/frontend/src/frontend.rs | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index f2ae3b692b..91c2b4c4e6 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -569,7 +569,6 @@ where self.func.dfg.append_ebb_arg(dest_ebb, *ty); } } else { - let dest_ebb_args = self.func.dfg.ebb_args(dest_ebb); // The Ebb already has predecessors // We check that the arguments supplied match those supplied // previously. @@ -577,15 +576,12 @@ where jump_args.len(), self.builder.ebbs[dest_ebb].user_arg_count, "the jump instruction doesn't have the same \ - number of arguments as its destination Ebb \ - ({} vs {}).", - jump_args.len(), - dest_ebb_args.len() + number of arguments as its destination Ebb.", ); debug_assert!( jump_args .iter() - .zip(dest_ebb_args.iter().take( + .zip(self.func.dfg.ebb_args(dest_ebb).iter().take( self.builder.ebbs[dest_ebb].user_arg_count, )) .all(|(jump_arg, dest_arg)| { From 16eb689dd1f0dbbba52c947db6bb6329752b5142 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 21 Sep 2017 13:28:40 -0700 Subject: [PATCH 1148/3084] Use `Self` instead of repeating the type name. --- lib/frontend/src/ssa.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index ee4aaf3a65..90090d6d0e 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -64,8 +64,8 @@ pub struct SideEffects { } impl SideEffects { - fn new() -> SideEffects { - SideEffects { + fn new() -> Self { + Self { split_ebbs_created: Vec::new(), instructions_added_to_ebbs: Vec::new(), } From b2a314a2290cb476971378ca0cde4047d3933939 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 20 Sep 2017 16:42:30 -0700 Subject: [PATCH 1149/3084] Add per-instruction source locations to the Cretonne IR. Source locations are opaque 32-bit entities that can be used to represent WebAssembly byte-code positions or some other source identifier. --- .../parser/instruction_encoding.cton | 8 +-- lib/cretonne/src/ir/function.rs | 10 ++- lib/cretonne/src/ir/mod.rs | 5 ++ lib/cretonne/src/ir/sourceloc.rs | 62 +++++++++++++++++++ lib/cretonne/src/write.rs | 48 ++++++++------ lib/reader/src/lexer.rs | 18 ++++++ lib/reader/src/parser.rs | 27 ++++++++ misc/vim/syntax/cton.vim | 2 + 8 files changed, 156 insertions(+), 24 deletions(-) create mode 100644 lib/cretonne/src/ir/sourceloc.rs diff --git a/cranelift/filetests/parser/instruction_encoding.cton b/cranelift/filetests/parser/instruction_encoding.cton index d6dda30da5..6bc7dcd6ce 100644 --- a/cranelift/filetests/parser/instruction_encoding.cton +++ b/cranelift/filetests/parser/instruction_encoding.cton @@ -10,8 +10,8 @@ ebb1(v0: i32 [%x8], v1: i32): [-] trap heap_oob [R#1234, %x5, %x11] v6, v7 = iadd_cout v2, v0 [Rshamt#beef, %x25] v8 = ishl_imm v6, 2 - v9 = iadd v8, v7 - [Iret#5] return v0, v8 +@55 v9 = iadd v8, v7 +@a5 [Iret#5] return v0, v8 } ; sameln: function %foo(i32, i32) native { ; nextln: $ebb1($v0: i32 [%x8], $v1: i32): @@ -19,6 +19,6 @@ ebb1(v0: i32 [%x8], v1: i32): ; nextln: [-]$WS trap heap_oob ; nextln: [R#1234,%x5,%x11]$WS $v6, $v7 = iadd_cout $v2, $v0 ; nextln: [Rshamt#beef,%x25]$WS $v8 = ishl_imm $v6, 2 -; nextln: [-,-]$WS $v9 = iadd $v8, $v7 -; nextln: [Iret#05]$WS return $v0, $v8 +; nextln: @0055 [-,-]$WS $v9 = iadd $v8, $v7 +; nextln: @00a5 [Iret#05]$WS return $v0, $v8 ; nextln: } diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index f7ba244ae0..f8134b34d9 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -6,7 +6,7 @@ use entity::{PrimaryMap, EntityMap}; use ir; use ir::{FunctionName, CallConv, Signature, DataFlowGraph, Layout}; -use ir::{InstEncodings, ValueLocations, JumpTables, StackSlots, EbbOffsets}; +use ir::{InstEncodings, ValueLocations, JumpTables, StackSlots, EbbOffsets, SourceLocs}; use ir::{Ebb, JumpTableData, JumpTable, StackSlotData, StackSlot, SigRef, ExtFuncData, FuncRef, GlobalVarData, GlobalVar, HeapData, Heap}; use isa::TargetIsa; @@ -56,6 +56,12 @@ pub struct Function { /// computes it, and it can easily be recomputed by calling that function. It is not included /// in the textual IL format. pub offsets: EbbOffsets, + + /// Source locations. + /// + /// Track the original source location for each instruction. The source locations are not + /// interpreted by Cretonne, only preserved. + pub srclocs: SourceLocs, } impl Function { @@ -73,6 +79,7 @@ impl Function { encodings: EntityMap::new(), locations: EntityMap::new(), offsets: EntityMap::new(), + srclocs: EntityMap::new(), } } @@ -88,6 +95,7 @@ impl Function { self.encodings.clear(); self.locations.clear(); self.offsets.clear(); + self.srclocs.clear(); } /// Create a new empty, anonymous function with a native calling convention. diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 077db6537b..9fafd99b9a 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -17,6 +17,7 @@ mod globalvar; mod heap; mod memflags; mod progpoint; +mod sourceloc; mod trapcode; mod valueloc; @@ -34,6 +35,7 @@ pub use ir::jumptable::JumpTableData; pub use ir::layout::{Layout, CursorBase, Cursor}; pub use ir::memflags::MemFlags; pub use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; +pub use ir::sourceloc::SourceLoc; pub use ir::stackslot::{StackSlots, StackSlotKind, StackSlotData}; pub use ir::trapcode::TrapCode; pub use ir::types::Type; @@ -54,3 +56,6 @@ pub type InstEncodings = EntityMap; /// Code offsets for EBBs. pub type EbbOffsets = EntityMap; + +/// Source locations for instructions. +pub type SourceLocs = EntityMap; diff --git a/lib/cretonne/src/ir/sourceloc.rs b/lib/cretonne/src/ir/sourceloc.rs new file mode 100644 index 0000000000..104bc541b6 --- /dev/null +++ b/lib/cretonne/src/ir/sourceloc.rs @@ -0,0 +1,62 @@ +//! Source locations. +//! +//! Cretonne tracks the original source location of each instruction, and preserves the source +//! location when instructions are transformed. + +use std::fmt; + +/// A source location. +/// +/// This is an opaque 32-bit number attached to each Cretonne IL instruction. Cretonne does not +/// interpret source locations in any way, they are simply preserved from the input to the output. +/// +/// The default source location uses the all-ones bit pattern `!0`. It is used for instructions +/// that can't be given a real source location. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct SourceLoc(u32); + +impl SourceLoc { + /// Create a new source location with the given bits. + pub fn new(bits: u32) -> SourceLoc { + SourceLoc(bits) + } + + /// Is this the default source location? + pub fn is_default(self) -> bool { + self == Default::default() + } + + /// Read the bits of this source location. + pub fn bits(self) -> u32 { + self.0 + } +} + +impl Default for SourceLoc { + fn default() -> SourceLoc { + SourceLoc(!0) + } +} + +impl fmt::Display for SourceLoc { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + if self.is_default() { + write!(f, "@-") + } else { + write!(f, "@{:04x}", self.0) + } + } +} + +#[cfg(test)] +mod tests { + use ir::SourceLoc; + + #[test] + fn display() { + assert_eq!(SourceLoc::default().to_string(), "@-"); + assert_eq!(SourceLoc::new(0).to_string(), "@0000"); + assert_eq!(SourceLoc::new(16).to_string(), "@0010"); + assert_eq!(SourceLoc::new(0xabcdef).to_string(), "@abcdef"); + } +} diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 2cbc8766ab..d2f903cc7a 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -106,6 +106,7 @@ pub fn write_ebb_header( func: &Function, isa: Option<&TargetIsa>, ebb: Ebb, + indent: usize, ) -> Result { // Write out the basic block header, outdented: // @@ -114,19 +115,17 @@ pub fn write_ebb_header( // ebb10(v4: f64, v5: b1): // - // If we're writing encoding annotations, shift by 20. - if !func.encodings.is_empty() { - write!(w, " ")?; - } + // The `indent` is the instruction indentation. EBB headers are 4 spaces out from that. + write!(w, "{1:0$}{2}", indent - 4, "", ebb)?; let regs = isa.map(TargetIsa::register_info); let regs = regs.as_ref(); let mut args = func.dfg.ebb_args(ebb).iter().cloned(); match args.next() { - None => return writeln!(w, "{}:", ebb), + None => return writeln!(w, ":"), Some(arg) => { - write!(w, "{}(", ebb)?; + write!(w, "(")?; write_arg(w, func, regs, arg)?; } } @@ -139,9 +138,16 @@ pub fn write_ebb_header( } pub fn write_ebb(w: &mut Write, func: &Function, isa: Option<&TargetIsa>, ebb: Ebb) -> Result { - write_ebb_header(w, func, isa, ebb)?; + // Indent all instructions if any encodings are present. + let indent = if func.encodings.is_empty() && func.srclocs.is_empty() { + 4 + } else { + 36 + }; + + write_ebb_header(w, func, isa, ebb, indent)?; for inst in func.layout.ebb_insts(ebb) { - write_instruction(w, func, isa, inst)?; + write_instruction(w, func, isa, inst, indent)?; } Ok(()) } @@ -203,16 +209,22 @@ fn write_instruction( func: &Function, isa: Option<&TargetIsa>, inst: Inst, + indent: usize, ) -> Result { - // Indent all instructions to col 24 if any encodings are present. - let indent = if func.encodings.is_empty() { 4 } else { 24 }; - // Value aliases come out on lines before the instruction using them. write_value_aliases(w, func, inst, indent)?; + // Prefix containing source location, encoding, and value locations. + let mut s = String::with_capacity(16); + + // Source location goes first. + let srcloc = func.srclocs[inst]; + if !srcloc.is_default() { + write!(s, "{} ", srcloc)?; + } + // Write out encoding info. if let Some(enc) = func.encodings.get(inst).cloned() { - let mut s = String::with_capacity(16); if let Some(isa) = isa { write!(s, "[{}", isa.encoding_info().display(enc))?; // Write value locations, if we have them. @@ -222,17 +234,15 @@ fn write_instruction( write!(s, ",{}", func.locations[r].display(®s))? } } - write!(s, "]")?; + write!(s, "] ")?; } else { - write!(s, "[{}]", enc)?; + write!(s, "[{}] ", enc)?; } - // Align instruction following ISA annotation to col 24. - write!(w, "{:23} ", s)?; - } else { - // No annotations, simply indent. - write!(w, "{1:0$}", indent, "")?; } + // Write out prefix and indent the instruction. + write!(w, "{1:0$}", indent, s)?; + // Write out the result values, if any. let mut has_results = false; for r in func.dfg.inst_results(inst) { diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 1f78b27d92..357880ef4a 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -45,6 +45,7 @@ pub enum Token<'a> { Name(&'a str), // %9arbitrary_alphanum, %x3, %0, %function ... HexSequence(&'a str), // #89AF Identifier(&'a str), // Unrecognized identifier (opcode, enumerator, ...) + SourceLoc(&'a str), // @00c7 } /// A `Token` with an associated location. @@ -388,6 +389,22 @@ impl<'a> Lexer<'a> { token(Token::HexSequence(&self.source[begin..end]), loc) } + fn scan_srcloc(&mut self) -> Result, LocatedError> { + let loc = self.loc(); + let begin = self.pos + 1; + + assert_eq!(self.lookahead, Some('@')); + + while let Some(c) = self.next_ch() { + if !char::is_digit(c, 16) { + break; + } + } + + let end = self.pos; + token(Token::SourceLoc(&self.source[begin..end]), loc) + } + /// Get the next token or a lexical error. /// /// Return None when the end of the source is encountered. @@ -419,6 +436,7 @@ impl<'a> Lexer<'a> { Some(ch) if ch.is_alphabetic() => Some(self.scan_word()), Some('%') => Some(self.scan_name()), Some('#') => Some(self.scan_hex_sequence()), + Some('@') => Some(self.scan_srcloc()), Some(ch) if ch.is_whitespace() => { self.next_ch(); continue; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 331ca5eb92..be86302645 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -13,6 +13,7 @@ use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, CallConv, S JumpTable, JumpTableData, Signature, ArgumentType, ArgumentExtension, ExtFuncData, SigRef, FuncRef, StackSlot, ValueLoc, ArgumentLoc, MemFlags, GlobalVar, GlobalVarData, Heap, HeapData, HeapStyle, HeapBase}; +use cretonne::ir; use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Uimm32, Offset32, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; @@ -696,6 +697,23 @@ impl<'a> Parser<'a> { } } + /// Parse an optional source location. + /// + /// Return an optional source location if no real location is present. + fn optional_srcloc(&mut self) -> Result { + if let Some(Token::SourceLoc(text)) = self.token() { + match u32::from_str_radix(text, 16) { + Ok(num) => { + self.consume(); + Ok(ir::SourceLoc::new(num)) + } + Err(_) => return err!(self.loc, "invalid source location: {}", text), + } + } else { + Ok(Default::default()) + } + } + /// Parse a list of test commands. pub fn parse_test_commands(&mut self) -> Vec> { let mut list = Vec::new(); @@ -1360,9 +1378,11 @@ impl<'a> Parser<'a> { Some(Token::Value(_)) => true, Some(Token::Identifier(_)) => true, Some(Token::LBracket) => true, + Some(Token::SourceLoc(_)) => true, _ => false, } { + let srcloc = self.optional_srcloc()?; let (encoding, result_locations) = self.parse_instruction_encoding(ctx)?; // We need to parse instruction results here because they are shared @@ -1380,6 +1400,7 @@ impl<'a> Parser<'a> { self.consume(); self.parse_instruction( results, + srcloc, encoding, result_locations, ctx, @@ -1390,6 +1411,7 @@ impl<'a> Parser<'a> { _ => { self.parse_instruction( results, + srcloc, encoding, result_locations, ctx, @@ -1587,6 +1609,7 @@ impl<'a> Parser<'a> { fn parse_instruction( &mut self, results: Vec, + srcloc: ir::SourceLoc, encoding: Option, result_locations: Option>, ctx: &mut Context, @@ -1636,6 +1659,10 @@ impl<'a> Parser<'a> { "duplicate inst references created", ); + if !srcloc.is_default() { + ctx.function.srclocs[inst] = srcloc; + } + if let Some(encoding) = encoding { ctx.function.encodings[inst] = encoding; } diff --git a/misc/vim/syntax/cton.vim b/misc/vim/syntax/cton.vim index d345742154..3bc6a4a5da 100644 --- a/misc/vim/syntax/cton.vim +++ b/misc/vim/syntax/cton.vim @@ -25,6 +25,7 @@ syn match ctonName /%\w\+\>/ syn match ctonNumber /-\?\<[0-9_]\+\>/ syn match ctonNumber /-\?\<0x[0-9a-fA-F_]\+\(\.[0-9a-fA-F_]*\)\?\(p[+-]\?\d\+\)\?\>/ syn match ctonHexSeq /#\x\+\>/ +syn match ctonSourceLoc /@[0-9a-f]\+\>/ syn region ctonCommentLine start=";" end="$" contains=ctonFilecheck @@ -38,5 +39,6 @@ hi def link ctonNumber Number hi def link ctonHexSeq Number hi def link ctonCommentLine Comment hi def link ctonFilecheck SpecialComment +hi def link ctonSourceLoc LineNr let b:current_syntax = "cton" From 4d4da2dc60ae2066480dff439b00ed534fc8fd16 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 21 Sep 2017 10:38:11 -0700 Subject: [PATCH 1150/3084] Add source location support to FuncCursor and EncCursor. A cursor now also remembers a current source location which will be assigned to all new instructions created with the cursor. The old layout::Cursor can't support source locations because it doesn't have a reference to the full ir::Function. --- lib/cretonne/meta/gen_legalizer.py | 6 +- lib/cretonne/src/cursor.rs | 37 ++++++++ lib/cretonne/src/ir/layout.rs | 119 ++++++++++++++++++------ lib/cretonne/src/legalizer/boundary.rs | 8 +- lib/cretonne/src/legalizer/globalvar.rs | 1 + lib/cretonne/src/legalizer/heap.rs | 3 + lib/cretonne/src/legalizer/mod.rs | 2 + lib/cretonne/src/legalizer/split.rs | 2 +- lib/cretonne/src/regalloc/coloring.rs | 3 +- 9 files changed, 143 insertions(+), 38 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index a79b05da76..41b64ede5d 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -358,6 +358,7 @@ def gen_xform_group(xgrp, fmt, type_sets): 'cfg: &mut ::flowgraph::ControlFlowGraph) -> ' 'bool {{'.format(xgrp.name), '}'): fmt.line('use ir::{InstBuilder, CursorBase};') + fmt.line('let srcloc = func.srclocs[inst];') # Group the xforms by opcode so we can generate a big switch. # Preserve ordering. @@ -372,8 +373,9 @@ def gen_xform_group(xgrp, fmt, type_sets): with fmt.indented( 'ir::Opcode::{} => {{'.format(camel_name), '}'): fmt.line( - 'let pos = &mut ' - 'ir::Cursor::new(&mut func.layout)' + 'let pos = &mut ir::Cursor::new' + '(&mut func.layout, &mut func.srclocs)' + '.with_srcloc(srcloc)' '.at_inst(inst);') fmt.line('let dfg = &mut func.dfg;') for xform in xforms[camel_name]: diff --git a/lib/cretonne/src/cursor.rs b/lib/cretonne/src/cursor.rs index 442d820c09..5e3ea866f5 100644 --- a/lib/cretonne/src/cursor.rs +++ b/lib/cretonne/src/cursor.rs @@ -19,6 +19,7 @@ pub use ir::layout::Cursor as LayoutCursor; /// encoding. pub struct FuncCursor<'f> { pos: CursorPosition, + srcloc: ir::SourceLoc, /// The referenced function. pub func: &'f mut ir::Function, @@ -29,10 +30,16 @@ impl<'f> FuncCursor<'f> { pub fn new(func: &'f mut ir::Function) -> FuncCursor<'f> { FuncCursor { pos: CursorPosition::Nowhere, + srcloc: Default::default(), func, } } + /// Use the source location of `inst` for future instructions. + pub fn use_srcloc(&mut self, inst: ir::Inst) { + self.srcloc = self.func.srclocs[inst]; + } + /// Create an instruction builder that inserts an instruction at the current position. pub fn ins(&mut self) -> ir::InsertBuilder<&mut FuncCursor<'f>> { ir::InsertBuilder::new(self) @@ -48,6 +55,14 @@ impl<'f> Cursor for FuncCursor<'f> { self.pos = pos } + fn srcloc(&self) -> ir::SourceLoc { + self.srcloc + } + + fn set_srcloc(&mut self, srcloc: ir::SourceLoc) { + self.srcloc = srcloc; + } + fn layout(&self) -> &ir::Layout { &self.func.layout } @@ -68,6 +83,9 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut FuncCursor<'f> { fn insert_built_inst(self, inst: ir::Inst, _: ir::Type) -> &'c mut ir::DataFlowGraph { self.insert_inst(inst); + if !self.srcloc.is_default() { + self.func.srclocs[inst] = self.srcloc; + } &mut self.func.dfg } } @@ -80,6 +98,7 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut FuncCursor<'f> { /// public `pos.func` member. pub struct EncCursor<'f> { pos: CursorPosition, + srcloc: ir::SourceLoc, built_inst: Option, /// The referenced function. @@ -94,12 +113,18 @@ impl<'f> EncCursor<'f> { pub fn new(func: &'f mut ir::Function, isa: &'f TargetIsa) -> EncCursor<'f> { EncCursor { pos: CursorPosition::Nowhere, + srcloc: Default::default(), built_inst: None, func, isa, } } + /// Use the source location of `inst` for future instructions. + pub fn use_srcloc(&mut self, inst: ir::Inst) { + self.srcloc = self.func.srclocs[inst]; + } + /// Create an instruction builder that will insert an encoded instruction at the current /// position. /// @@ -134,6 +159,14 @@ impl<'f> Cursor for EncCursor<'f> { self.pos = pos } + fn srcloc(&self) -> ir::SourceLoc { + self.srcloc + } + + fn set_srcloc(&mut self, srcloc: ir::SourceLoc) { + self.srcloc = srcloc; + } + fn layout(&self) -> &ir::Layout { &self.func.layout } @@ -161,6 +194,10 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut EncCursor<'f> { self.insert_inst(inst); self.built_inst = Some(inst); + if !self.srcloc.is_default() { + self.func.srclocs[inst] = self.srcloc; + } + // Assign an encoding. match self.isa.encode( &self.func.dfg, diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 877701a1a1..2b8b5b6099 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -7,7 +7,7 @@ use std::cmp; use std::iter::{Iterator, IntoIterator}; use entity::EntityMap; use packed_option::PackedOption; -use ir::{Ebb, Inst, Type, DataFlowGraph}; +use ir::{Ebb, Inst, Type, DataFlowGraph, SourceLoc, SourceLocs}; use ir::builder::InstInserterBase; use ir::progpoint::{ProgramOrder, ExpandedProgramPoint}; @@ -676,7 +676,10 @@ impl<'f> DoubleEndedIterator for Insts<'f> { pub struct Cursor<'f> { /// Borrowed function layout. Public so it can be re-borrowed from this cursor. pub layout: &'f mut Layout, + /// Borrowed source locations. + pub srclocs: Option<&'f mut SourceLocs>, pos: CursorPosition, + srcloc: SourceLoc, } /// The possible positions of a cursor. @@ -703,12 +706,39 @@ pub trait CursorBase { /// Set the current position. fn set_position(&mut self, pos: CursorPosition); + /// Get the source location that should be assigned to new instructions. + fn srcloc(&self) -> SourceLoc; + + /// Set the source location that should be assigned to new instructions. + fn set_srcloc(&mut self, srcloc: SourceLoc); + /// Borrow a reference to the function layout that this cursor is navigating. fn layout(&self) -> &Layout; /// Borrow a mutable reference to the function layout that this cursor is navigating. fn layout_mut(&mut self) -> &mut Layout; + /// Exchange this cursor for one with a set source location. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, SourceLoc}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_func(func: &mut Function, srcloc: SourceLoc) { + /// let mut pos = FuncCursor::new(func).with_srcloc(srcloc); + /// + /// // Use `pos`... + /// } + /// ``` + fn with_srcloc(mut self, srcloc: SourceLoc) -> Self + where + Self: Sized, + { + self.set_srcloc(srcloc); + self + } + /// Rebuild this cursor positioned at `pos`. fn at_position(mut self, pos: CursorPosition) -> Self where @@ -724,9 +754,9 @@ pub trait CursorBase { /// /// ``` /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, inst: Inst) { - /// let mut pos = Cursor::new(&mut func.layout).at_inst(inst); + /// let mut pos = FuncCursor::new(func).at_inst(inst); /// /// // Use `pos`... /// } @@ -747,9 +777,9 @@ pub trait CursorBase { /// /// ``` /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { - /// let mut pos = Cursor::new(&mut func.layout).at_first_insertion_point(ebb); + /// let mut pos = FuncCursor::new(func).at_first_insertion_point(ebb); /// /// // Use `pos`... /// } @@ -768,9 +798,9 @@ pub trait CursorBase { /// /// ``` /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { - /// let mut pos = Cursor::new(&mut func.layout).at_first_inst(ebb); + /// let mut pos = FuncCursor::new(func).at_first_inst(ebb); /// /// // Use `pos`... /// } @@ -789,9 +819,9 @@ pub trait CursorBase { /// /// ``` /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { - /// let mut pos = Cursor::new(&mut func.layout).at_last_inst(ebb); + /// let mut pos = FuncCursor::new(func).at_last_inst(ebb); /// /// // Use `pos`... /// } @@ -810,9 +840,9 @@ pub trait CursorBase { /// /// ``` /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, inst: Inst) { - /// let mut pos = Cursor::new(&mut func.layout).after_inst(inst); + /// let mut pos = FuncCursor::new(func).after_inst(inst); /// /// // Use `pos`... /// } @@ -831,9 +861,9 @@ pub trait CursorBase { /// /// ``` /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { - /// let mut pos = Cursor::new(&mut func.layout).at_top(ebb); + /// let mut pos = FuncCursor::new(func).at_top(ebb); /// /// // Use `pos`... /// } @@ -852,9 +882,9 @@ pub trait CursorBase { /// /// ``` /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { - /// let mut pos = Cursor::new(&mut func.layout).at_bottom(ebb); + /// let mut pos = FuncCursor::new(func).at_bottom(ebb); /// /// // Use `pos`... /// } @@ -957,9 +987,9 @@ pub trait CursorBase { /// /// ``` /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function) { - /// let mut cursor = Cursor::new(&mut func.layout); + /// let mut cursor = FuncCursor::new(func); /// while let Some(ebb) = cursor.next_ebb() { /// // Edit ebb. /// } @@ -990,9 +1020,9 @@ pub trait CursorBase { /// /// ``` /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function) { - /// let mut cursor = Cursor::new(&mut func.layout); + /// let mut cursor = FuncCursor::new(func); /// while let Some(ebb) = cursor.prev_ebb() { /// // Edit ebb. /// } @@ -1027,9 +1057,9 @@ pub trait CursorBase { /// /// ``` /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; /// fn edit_ebb(func: &mut Function, ebb: Ebb) { - /// let mut cursor = Cursor::new(&mut func.layout).at_top(ebb); + /// let mut cursor = FuncCursor::new(func).at_top(ebb); /// while let Some(inst) = cursor.next_inst() { /// // Edit instructions... /// } @@ -1041,9 +1071,9 @@ pub trait CursorBase { /// /// ``` /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function) { - /// let mut cursor = Cursor::new(&mut func.layout); + /// let mut cursor = FuncCursor::new(func); /// while let Some(ebb) = cursor.next_ebb() { /// while let Some(inst) = cursor.next_inst() { /// // Edit instructions... @@ -1095,9 +1125,9 @@ pub trait CursorBase { /// /// ``` /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::ir::layout::{Cursor, CursorBase}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; /// fn edit_ebb(func: &mut Function, ebb: Ebb) { - /// let mut cursor = Cursor::new(&mut func.layout).at_bottom(ebb); + /// let mut cursor = FuncCursor::new(func).at_bottom(ebb); /// while let Some(inst) = cursor.prev_inst() { /// // Edit instructions... /// } @@ -1213,6 +1243,14 @@ impl<'f> CursorBase for Cursor<'f> { self.pos = pos; } + fn srcloc(&self) -> SourceLoc { + self.srcloc + } + + fn set_srcloc(&mut self, srcloc: SourceLoc) { + self.srcloc = srcloc + } + fn layout(&self) -> &Layout { self.layout } @@ -1225,10 +1263,22 @@ impl<'f> CursorBase for Cursor<'f> { impl<'f> Cursor<'f> { /// Create a new `Cursor` for `layout`. /// The cursor holds a mutable reference to `layout` for its entire lifetime. - pub fn new(layout: &'f mut Layout) -> Cursor { + pub fn new>>( + layout: &'f mut Layout, + srclocs: SL, + ) -> Cursor<'f> { Cursor { layout, + srclocs: srclocs.into(), pos: CursorPosition::Nowhere, + srcloc: Default::default(), + } + } + + /// Use the source location of `inst` for future instructions. + pub fn use_srcloc(&mut self, inst: Inst) { + if let Some(ref mut ss) = self.srclocs { + self.srcloc = ss[inst]; } } } @@ -1263,6 +1313,13 @@ impl<'c, 'fc: 'c, 'fd> InstInserterBase<'fd> for LayoutCursorInserter<'c, 'fc, ' fn insert_built_inst(self, inst: Inst, _ctrl_typevar: Type) -> &'fd mut DataFlowGraph { self.pos.insert_inst(inst); + if !self.pos.srcloc.is_default() { + if let Some(ref mut ss) = self.pos.srclocs { + ss[inst] = self.pos.srcloc; + } else { + panic!("layout::Cursor missing a SourceLocs reference"); + } + } self.dfg } } @@ -1300,7 +1357,7 @@ mod tests { } // Check backwards linkage with a cursor. - let mut cur = Cursor::new(layout); + let mut cur = Cursor::new(layout, None); for &(ebb, insts) in ebbs.into_iter().rev() { assert_eq!(cur.prev_ebb(), Some(ebb)); for &inst in insts.into_iter().rev() { @@ -1356,7 +1413,7 @@ mod tests { } // Test cursor positioning. - let mut cur = Cursor::new(&mut layout); + let mut cur = Cursor::new(&mut layout, None); assert_eq!(cur.position(), CursorPosition::Nowhere); assert_eq!(cur.next_inst(), None); assert_eq!(cur.position(), CursorPosition::Nowhere); @@ -1474,7 +1531,7 @@ mod tests { verify(&mut layout, &[(e1, &[i1, i2, i0])]); // Test cursor positioning. - let mut cur = Cursor::new(&mut layout).at_top(e1); + let mut cur = Cursor::new(&mut layout, None).at_top(e1); assert_eq!(cur.position(), CursorPosition::Before(e1)); assert_eq!(cur.prev_inst(), None); assert_eq!(cur.position(), CursorPosition::Before(e1)); @@ -1594,7 +1651,7 @@ mod tests { assert_eq!(layout.inst_ebb(i0), Some(e1)); { - let mut cur = Cursor::new(&mut layout); + let mut cur = Cursor::new(&mut layout, None); assert_eq!(cur.next_ebb(), Some(e0)); assert_eq!(cur.next_inst(), None); assert_eq!(cur.next_ebb(), Some(e1)); @@ -1622,7 +1679,7 @@ mod tests { assert_eq!(layout.inst_ebb(i3), Some(e2)); { - let mut cur = Cursor::new(&mut layout); + let mut cur = Cursor::new(&mut layout, None); assert_eq!(cur.next_ebb(), Some(e0)); assert_eq!(cur.next_inst(), Some(i1)); assert_eq!(cur.next_inst(), None); diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index dfeb88b777..fe76d21d6d 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -62,7 +62,7 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { // Insert position for argument conversion code. // We want to insert instructions before the first instruction in the entry block. // If the entry block is empty, append instructions to it instead. - let mut pos = Cursor::new(&mut func.layout).at_first_inst(entry); + let mut pos = Cursor::new(&mut func.layout, &mut func.srclocs).at_first_inst(entry); // Keep track of the argument types in the ABI-legalized signature. let abi_types = &func.signature.argument_types; @@ -488,7 +488,8 @@ fn legalize_inst_arguments( /// Returns `true` if any instructions were inserted. pub fn handle_call_abi(mut inst: Inst, func: &mut Function, cfg: &ControlFlowGraph) -> bool { let dfg = &mut func.dfg; - let pos = &mut Cursor::new(&mut func.layout).at_inst(inst); + let pos = &mut Cursor::new(&mut func.layout, &mut func.srclocs).at_inst(inst); + pos.use_srcloc(inst); // Start by checking if the argument types already match the signature. let sig_ref = match check_call_signature(dfg, inst) { @@ -530,7 +531,8 @@ pub fn handle_call_abi(mut inst: Inst, func: &mut Function, cfg: &ControlFlowGra pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph) -> bool { let dfg = &mut func.dfg; let sig = &mut func.signature; - let pos = &mut Cursor::new(&mut func.layout).at_inst(inst); + let pos = &mut Cursor::new(&mut func.layout, &mut func.srclocs).at_inst(inst); + pos.use_srcloc(inst); // Check if the returned types already match the signature. if check_return_signature(dfg, inst, sig) { diff --git a/lib/cretonne/src/legalizer/globalvar.rs b/lib/cretonne/src/legalizer/globalvar.rs index 42856edc9b..b4b28d3032 100644 --- a/lib/cretonne/src/legalizer/globalvar.rs +++ b/lib/cretonne/src/legalizer/globalvar.rs @@ -51,6 +51,7 @@ fn deref_addr(inst: ir::Inst, func: &mut ir::Function, base: ir::GlobalVar, offs // detects any cycles in the `deref` globals. let ptr_ty = func.dfg.value_type(func.dfg.first_result(inst)); let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); let base_addr = pos.ins().global_addr(ptr_ty, base); // TODO: We could probably set both `notrap` and `aligned` on this load instruction. diff --git a/lib/cretonne/src/legalizer/heap.rs b/lib/cretonne/src/legalizer/heap.rs index ab2ce444af..775b745725 100644 --- a/lib/cretonne/src/legalizer/heap.rs +++ b/lib/cretonne/src/legalizer/heap.rs @@ -48,6 +48,7 @@ fn dynamic_addr( let addr_ty = func.dfg.value_type(func.dfg.first_result(inst)); let min_size = func.heaps[heap].min_size.into(); let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); // Start with the bounds check. Trap if `offset + size > bound`. let bound_addr = pos.ins().global_addr(addr_ty, bound_gv); @@ -99,6 +100,7 @@ fn static_addr( let offset_ty = func.dfg.value_type(offset); let addr_ty = func.dfg.value_type(func.dfg.first_result(inst)); let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); // Start with the bounds check. Trap if `offset + size > bound`. if size > bound { @@ -147,6 +149,7 @@ fn offset_addr( func: &mut ir::Function, ) { let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); // Convert `offset` to `addr_ty`. if offset_ty != addr_ty { diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 5cb1c27819..920bebde81 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -138,6 +138,7 @@ fn expand_cond_trap(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFl } let mut pos = FuncCursor::new(func).after_inst(inst); + pos.use_srcloc(inst); pos.ins().trap(code); pos.insert_ebb(new_ebb); @@ -154,6 +155,7 @@ fn expand_fconst(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlow // In the future, we may want to generate constant pool entries for these constants, but for // now use an `iconst` and a bit cast. let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); let ival = match pos.func.dfg[inst] { ir::InstructionData::UnaryIeee32 { opcode: ir::Opcode::F32const, diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index f3df6b7b82..bc73db6817 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -120,7 +120,7 @@ fn split_any( concat: Opcode, ) -> (Value, Value) { let mut repairs = Vec::new(); - let mut pos = ir::Cursor::new(layout).at_position(pos); + let mut pos = ir::Cursor::new(layout, None).at_position(pos); let result = split_value(dfg, &mut pos, value, concat, &mut repairs); // We have split the value requested, and now we may need to fix some EBB predecessors. diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 9a9122ba2f..e46c95c101 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -148,8 +148,9 @@ impl<'a> Context<'a> { self.divert.clear(); // Now go through the instructions in `ebb` and color the values they define. - let mut pos = Cursor::new(&mut func.layout).at_top(ebb); + let mut pos = Cursor::new(&mut func.layout, &mut func.srclocs).at_top(ebb); while let Some(inst) = pos.next_inst() { + pos.use_srcloc(inst); if let Some(constraints) = self.encinfo.operand_constraints(func.encodings[inst]) { self.visit_inst( inst, From 85e4e9f511ca070832f6a7dd8fda13a139e3754f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 21 Sep 2017 08:12:53 -0700 Subject: [PATCH 1151/3084] Assign source locations when translating WebAssembly to Cretonne. The source locations are byte code offsets relative to the beginning of the function. --- lib/frontend/src/frontend.rs | 22 ++++++++++++++++++---- lib/wasm/src/func_translator.rs | 11 +++++++++++ 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 91c2b4c4e6..376f798dd2 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -1,5 +1,6 @@ //! A frontend for building Cretonne IL from other languages. use cretonne::cursor::{Cursor, FuncCursor}; +use cretonne::ir; use cretonne::ir::{Ebb, Type, Value, Function, Inst, JumpTable, StackSlot, JumpTableData, StackSlotData, DataFlowGraph, InstructionData, ExtFuncData, FuncRef, SigRef, Signature, InstBuilderBase, GlobalVarData, GlobalVar, HeapData, Heap}; @@ -30,6 +31,9 @@ where /// This field is public so the function can be re-borrowed. pub func: &'a mut Function, + /// Source location to assign to all new instructions. + srcloc: ir::SourceLoc, + builder: &'a mut ILBuilder, position: Position, pristine: bool, @@ -117,6 +121,8 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short let inst = self.builder.func.dfg.make_inst(data.clone()); self.builder.func.dfg.make_inst_results(inst, ctrl_typevar); self.builder.func.layout.append_inst(inst, self.ebb); + self.builder.func.srclocs[inst] = self.builder.srcloc; + if data.opcode().is_branch() { match data.branch_destination() { Some(dest_ebb) => { @@ -179,13 +185,13 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short /// `create_ebb`) whose properties are: /// /// - branch and jump instructions can only point at the top of extended blocks; -/// - the last instruction of each block is a terminator instruction which has no natural sucessor, +/// - the last instruction of each block is a terminator instruction which has no natural successor, /// and those instructions can only appear at the end of extended blocks. /// /// The parameters of Cretonne IL instructions are Cretonne IL values, which can only be created /// as results of other Cretonne IL instructions. To be able to create variables redefined multiple /// times in your program, use the `def_var` and `use_var` command, that will maintain the -/// correspondance between your variables and Cretonne IL SSA values. +/// correspondence between your variables and Cretonne IL SSA values. /// /// The first block for which you call `switch_to_block` will be assumed to be the beginning of /// the function. @@ -215,6 +221,7 @@ where builder.clear(); FunctionBuilder { func: func, + srcloc: Default::default(), builder: builder, position: Position { ebb: Ebb::new(0), @@ -224,6 +231,11 @@ where } } + /// Set the source location that should be assigned to all new instructions. + pub fn set_srcloc(&mut self, srcloc: ir::SourceLoc) { + self.srcloc = srcloc; + } + /// Creates a new `Ebb` for the function and returns its reference. pub fn create_ebb(&mut self) -> Ebb { let ebb = self.func.dfg.make_ebb(); @@ -386,7 +398,9 @@ where /// need to know about `FunctionBuilder` at all. pub fn cursor<'f>(&'f mut self) -> FuncCursor<'f> { self.ensure_inserted_ebb(); - FuncCursor::new(self.func).at_bottom(self.position.ebb) + FuncCursor::new(self.func) + .with_srcloc(self.srcloc) + .at_bottom(self.position.ebb) } } @@ -399,7 +413,7 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> where Variable: EntityRef + Default, { - /// Retrieves all the arguments for an `Ebb` currently infered from the jump instructions + /// Retrieves all the arguments for an `Ebb` currently inferred from the jump instructions /// inserted that target it and the SSA construction. pub fn ebb_args(&self, ebb: Ebb) -> &[Value] { self.func.dfg.ebb_args(ebb) diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 9a43083461..a5abdf382c 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -139,6 +139,7 @@ fn parse_local_decls( let mut locals_total = 0; for _ in 0..local_count { + builder.set_srcloc(cur_srcloc(reader)); let (count, ty) = reader.read_local_decl(&mut locals_total).map_err(|_| { CtonError::InvalidInput })?; @@ -199,6 +200,7 @@ fn parse_function_body( // Keep going until the final `End` operator which pops the outermost block. while !state.control_stack.is_empty() { + builder.set_srcloc(cur_srcloc(&reader)); let op = reader.read_operator().map_err(|_| CtonError::InvalidInput)?; translate_operator(&op, builder, state, environ); } @@ -218,6 +220,15 @@ fn parse_function_body( Ok(()) } +/// Get the current source location from a reader. +fn cur_srcloc(reader: &BinaryReader) -> ir::SourceLoc { + // We record source locations as byte code offsets relative to the beginning of the function. + // This will wrap around of a single function's byte code is larger than 4 GB, but a) the + // WebAssembly format doesn't allow for that, and b) that would hit other Cretonne + // implementation limits anyway. + ir::SourceLoc::new(reader.current_position() as u32) +} + #[cfg(test)] mod tests { use cretonne::{ir, Context}; From a9acbd1afd8574d63cadf124c5565403ebf50d7d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 21 Sep 2017 16:43:47 -0700 Subject: [PATCH 1152/3084] Convert the legalizer/boundary module to FuncCursor. This is a larger refactoring because all the changes need to be done together. Either you pass a Function reference around, or you pass around references to the parts. There is no in between. --- lib/cretonne/src/legalizer/boundary.rs | 327 ++++++++++++------------- 1 file changed, 163 insertions(+), 164 deletions(-) diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index fe76d21d6d..c7b4627af1 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -18,10 +18,10 @@ //! intermediate state doesn't type check. use abi::{legalize_abi_value, ValueConversion}; +use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; -use ir::{Function, Cursor, CursorBase, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, - Signature, SigRef, ArgumentType, ArgumentPurpose, ArgumentLoc, ValueLoc, ValueLocations, - StackSlots, StackSlotKind}; +use ir::{Function, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, Signature, SigRef, + ArgumentType, ArgumentPurpose, ArgumentLoc, ValueLoc, StackSlotKind}; use ir::instructions::CallInfo; use isa::TargetIsa; use legalizer::split::{isplit, vsplit}; @@ -62,25 +62,25 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { // Insert position for argument conversion code. // We want to insert instructions before the first instruction in the entry block. // If the entry block is empty, append instructions to it instead. - let mut pos = Cursor::new(&mut func.layout, &mut func.srclocs).at_first_inst(entry); + let mut pos = FuncCursor::new(func).at_first_inst(entry); // Keep track of the argument types in the ABI-legalized signature. - let abi_types = &func.signature.argument_types; let mut abi_arg = 0; // Process the EBB arguments one at a time, possibly replacing one argument with multiple new // ones. We do this by detaching the entry EBB arguments first. - let ebb_args = func.dfg.detach_ebb_args(entry); + let ebb_args = pos.func.dfg.detach_ebb_args(entry); let mut old_arg = 0; - while let Some(arg) = ebb_args.get(old_arg, &func.dfg.value_lists) { + while let Some(arg) = ebb_args.get(old_arg, &pos.func.dfg.value_lists) { old_arg += 1; - let arg_type = func.dfg.value_type(arg); - if arg_type == abi_types[abi_arg].value_type { + let abi_type = pos.func.signature.argument_types[abi_arg]; + let arg_type = pos.func.dfg.value_type(arg); + if arg_type == abi_type.value_type { // No value translation is necessary, this argument matches the ABI type. // Just use the original EBB argument value. This is the most common case. - func.dfg.attach_ebb_arg(entry, arg); - match abi_types[abi_arg].purpose { + pos.func.dfg.attach_ebb_arg(entry, arg); + match abi_type.purpose { ArgumentPurpose::Normal => {} ArgumentPurpose::StructReturn => { assert!(!has_sret, "Multiple sret arguments found"); @@ -94,13 +94,13 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { assert!(!has_sigid, "Multiple sigid arguments found"); has_sigid = true; } - _ => panic!("Unexpected special-purpose arg {}", abi_types[abi_arg]), + _ => panic!("Unexpected special-purpose arg {}", abi_type), } abi_arg += 1; } else { // Compute the value we want for `arg` from the legalized ABI arguments. - let mut get_arg = |dfg: &mut DataFlowGraph, ty| { - let abi_type = abi_types[abi_arg]; + let mut get_arg = |func: &mut Function, ty| { + let abi_type = func.signature.argument_types[abi_arg]; assert_eq!( abi_type.purpose, ArgumentPurpose::Normal, @@ -108,22 +108,21 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { ); if ty == abi_type.value_type { abi_arg += 1; - Ok(dfg.append_ebb_arg(entry, ty)) + Ok(func.dfg.append_ebb_arg(entry, ty)) } else { Err(abi_type) } }; - let converted = - convert_from_abi(&mut func.dfg, &mut pos, arg_type, Some(arg), &mut get_arg); + let converted = convert_from_abi(&mut pos, arg_type, Some(arg), &mut get_arg); // The old `arg` is no longer an attached EBB argument, but there are probably still // uses of the value. - assert_eq!(func.dfg.resolve_aliases(arg), converted); + assert_eq!(pos.func.dfg.resolve_aliases(arg), converted); } } // The legalized signature may contain additional arguments representing special-purpose // registers. - for &arg in &abi_types[abi_arg..] { + for &arg in &pos.func.signature.argument_types[abi_arg..] { match arg.purpose { // Any normal arguments should have been processed above. ArgumentPurpose::Normal => { @@ -156,7 +155,7 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { // Just create entry block values to match here. We will use them in `handle_return_abi()` // below. - func.dfg.append_ebb_arg(entry, arg.value_type); + pos.func.dfg.append_ebb_arg(entry, arg.value_type); } } @@ -168,13 +167,9 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { /// This function is very similar to the `legalize_entry_arguments` function above. /// /// Returns the possibly new instruction representing the call. -fn legalize_inst_results( - dfg: &mut DataFlowGraph, - pos: &mut Cursor, - mut get_abi_type: ResType, -) -> Inst +fn legalize_inst_results(pos: &mut FuncCursor, mut get_abi_type: ResType) -> Inst where - ResType: FnMut(&DataFlowGraph, usize) -> ArgumentType, + ResType: FnMut(&Function, usize) -> ArgumentType, { let call = pos.current_inst().expect( "Cursor must point to a call instruction", @@ -182,37 +177,37 @@ where // We theoretically allow for call instructions that return a number of fixed results before // the call return values. In practice, it doesn't happen. - let fixed_results = dfg[call].opcode().constraints().fixed_results(); + let fixed_results = pos.func.dfg[call].opcode().constraints().fixed_results(); assert_eq!(fixed_results, 0, "Fixed results on calls not supported"); - let results = dfg.detach_results(call); + let results = pos.func.dfg.detach_results(call); let mut next_res = 0; let mut abi_res = 0; // Point immediately after the call. pos.next_inst(); - while let Some(res) = results.get(next_res, &dfg.value_lists) { + while let Some(res) = results.get(next_res, &pos.func.dfg.value_lists) { next_res += 1; - let res_type = dfg.value_type(res); - if res_type == get_abi_type(dfg, abi_res).value_type { + let res_type = pos.func.dfg.value_type(res); + if res_type == get_abi_type(pos.func, abi_res).value_type { // No value translation is necessary, this result matches the ABI type. - dfg.attach_result(call, res); + pos.func.dfg.attach_result(call, res); abi_res += 1; } else { - let mut get_res = |dfg: &mut DataFlowGraph, ty| { - let abi_type = get_abi_type(dfg, abi_res); + let mut get_res = |func: &mut Function, ty| { + let abi_type = get_abi_type(func, abi_res); if ty == abi_type.value_type { - let last_res = dfg.append_result(call, ty); + let last_res = func.dfg.append_result(call, ty); abi_res += 1; Ok(last_res) } else { Err(abi_type) } }; - let v = convert_from_abi(dfg, pos, res_type, Some(res), &mut get_res); - assert_eq!(dfg.resolve_aliases(res), v); + let v = convert_from_abi(pos, res_type, Some(res), &mut get_res); + assert_eq!(pos.func.dfg.resolve_aliases(res), v); } } @@ -229,19 +224,18 @@ where /// /// If the `into_result` value is provided, the converted result will be written into that value. fn convert_from_abi( - dfg: &mut DataFlowGraph, - pos: &mut Cursor, + pos: &mut FuncCursor, ty: Type, into_result: Option, get_arg: &mut GetArg, ) -> Value where - GetArg: FnMut(&mut DataFlowGraph, Type) -> Result, + GetArg: FnMut(&mut Function, Type) -> Result, { // Terminate the recursion when we get the desired type. - let arg_type = match get_arg(dfg, ty) { + let arg_type = match get_arg(pos.func, ty) { Ok(v) => { - debug_assert_eq!(dfg.value_type(v), ty); + debug_assert_eq!(pos.func.dfg.value_type(v), ty); assert_eq!(into_result, None); return v; } @@ -258,45 +252,45 @@ where // Construct a `ty` by concatenating two ABI integers. ValueConversion::IntSplit => { let abi_ty = ty.half_width().expect("Invalid type for conversion"); - let lo = convert_from_abi(dfg, pos, abi_ty, None, get_arg); - let hi = convert_from_abi(dfg, pos, abi_ty, None, get_arg); + let lo = convert_from_abi(pos, abi_ty, None, get_arg); + let hi = convert_from_abi(pos, abi_ty, None, get_arg); dbg!( "intsplit {}: {}, {}: {}", lo, - dfg.value_type(lo), + pos.func.dfg.value_type(lo), hi, - dfg.value_type(hi) + pos.func.dfg.value_type(hi) ); - dfg.ins(pos).with_results([into_result]).iconcat(lo, hi) + pos.ins().with_results([into_result]).iconcat(lo, hi) } // Construct a `ty` by concatenating two halves of a vector. ValueConversion::VectorSplit => { let abi_ty = ty.half_vector().expect("Invalid type for conversion"); - let lo = convert_from_abi(dfg, pos, abi_ty, None, get_arg); - let hi = convert_from_abi(dfg, pos, abi_ty, None, get_arg); - dfg.ins(pos).with_results([into_result]).vconcat(lo, hi) + let lo = convert_from_abi(pos, abi_ty, None, get_arg); + let hi = convert_from_abi(pos, abi_ty, None, get_arg); + pos.ins().with_results([into_result]).vconcat(lo, hi) } // Construct a `ty` by bit-casting from an integer type. ValueConversion::IntBits => { assert!(!ty.is_int()); let abi_ty = Type::int(ty.bits()).expect("Invalid type for conversion"); - let arg = convert_from_abi(dfg, pos, abi_ty, None, get_arg); - dfg.ins(pos).with_results([into_result]).bitcast(ty, arg) + let arg = convert_from_abi(pos, abi_ty, None, get_arg); + pos.ins().with_results([into_result]).bitcast(ty, arg) } // ABI argument is a sign-extended version of the value we want. ValueConversion::Sext(abi_ty) => { - let arg = convert_from_abi(dfg, pos, abi_ty, None, get_arg); + let arg = convert_from_abi(pos, abi_ty, None, get_arg); // TODO: Currently, we don't take advantage of the ABI argument being sign-extended. // We could insert an `assert_sreduce` which would fold with a following `sextend` of // this value. - dfg.ins(pos).with_results([into_result]).ireduce(ty, arg) + pos.ins().with_results([into_result]).ireduce(ty, arg) } ValueConversion::Uext(abi_ty) => { - let arg = convert_from_abi(dfg, pos, abi_ty, None, get_arg); + let arg = convert_from_abi(pos, abi_ty, None, get_arg); // TODO: Currently, we don't take advantage of the ABI argument being sign-extended. // We could insert an `assert_ureduce` which would fold with a following `uextend` of // this value. - dfg.ins(pos).with_results([into_result]).ireduce(ty, arg) + pos.ins().with_results([into_result]).ireduce(ty, arg) } } } @@ -313,48 +307,47 @@ where /// return the `Err(ArgumentType)` that is needed. /// fn convert_to_abi( - dfg: &mut DataFlowGraph, + pos: &mut FuncCursor, cfg: &ControlFlowGraph, - pos: &mut Cursor, value: Value, put_arg: &mut PutArg, ) where - PutArg: FnMut(&mut DataFlowGraph, Value) -> Result<(), ArgumentType>, + PutArg: FnMut(&mut Function, Value) -> Result<(), ArgumentType>, { // Start by invoking the closure to either terminate the recursion or get the argument type // we're trying to match. - let arg_type = match put_arg(dfg, value) { + let arg_type = match put_arg(pos.func, value) { Ok(_) => return, Err(t) => t, }; - let ty = dfg.value_type(value); + let ty = pos.func.dfg.value_type(value); match legalize_abi_value(ty, &arg_type) { ValueConversion::IntSplit => { let curpos = pos.position(); - let (lo, hi) = isplit(dfg, pos.layout, cfg, curpos, value); - convert_to_abi(dfg, cfg, pos, lo, put_arg); - convert_to_abi(dfg, cfg, pos, hi, put_arg); + let (lo, hi) = isplit(&mut pos.func.dfg, &mut pos.func.layout, cfg, curpos, value); + convert_to_abi(pos, cfg, lo, put_arg); + convert_to_abi(pos, cfg, hi, put_arg); } ValueConversion::VectorSplit => { let curpos = pos.position(); - let (lo, hi) = vsplit(dfg, pos.layout, cfg, curpos, value); - convert_to_abi(dfg, cfg, pos, lo, put_arg); - convert_to_abi(dfg, cfg, pos, hi, put_arg); + let (lo, hi) = vsplit(&mut pos.func.dfg, &mut pos.func.layout, cfg, curpos, value); + convert_to_abi(pos, cfg, lo, put_arg); + convert_to_abi(pos, cfg, hi, put_arg); } ValueConversion::IntBits => { assert!(!ty.is_int()); let abi_ty = Type::int(ty.bits()).expect("Invalid type for conversion"); - let arg = dfg.ins(pos).bitcast(abi_ty, value); - convert_to_abi(dfg, cfg, pos, arg, put_arg); + let arg = pos.ins().bitcast(abi_ty, value); + convert_to_abi(pos, cfg, arg, put_arg); } ValueConversion::Sext(abi_ty) => { - let arg = dfg.ins(pos).sextend(abi_ty, value); - convert_to_abi(dfg, cfg, pos, arg, put_arg); + let arg = pos.ins().sextend(abi_ty, value); + convert_to_abi(pos, cfg, arg, put_arg); } ValueConversion::Uext(abi_ty) => { - let arg = dfg.ins(pos).uextend(abi_ty, value); - convert_to_abi(dfg, cfg, pos, arg, put_arg); + let arg = pos.ins().uextend(abi_ty, value); + convert_to_abi(pos, cfg, arg, put_arg); } } } @@ -402,28 +395,30 @@ fn check_return_signature(dfg: &DataFlowGraph, inst: Inst, sig: &Signature) -> b /// argument number in `0..abi_args`. /// fn legalize_inst_arguments( - dfg: &mut DataFlowGraph, + pos: &mut FuncCursor, cfg: &ControlFlowGraph, - pos: &mut Cursor, abi_args: usize, mut get_abi_type: ArgType, ) where - ArgType: FnMut(&DataFlowGraph, usize) -> ArgumentType, + ArgType: FnMut(&Function, usize) -> ArgumentType, { let inst = pos.current_inst().expect( "Cursor must point to a call instruction", ); // Lift the value list out of the call instruction so we modify it. - let mut vlist = dfg[inst].take_value_list().expect( + let mut vlist = pos.func.dfg[inst].take_value_list().expect( "Call must have a value list", ); // The value list contains all arguments to the instruction, including the callee on an // indirect call which isn't part of the call arguments that must match the ABI signature. // Figure out how many fixed values are at the front of the list. We won't touch those. - let fixed_values = dfg[inst].opcode().constraints().fixed_value_arguments(); - let have_args = vlist.len(&dfg.value_lists) - fixed_values; + let fixed_values = pos.func.dfg[inst] + .opcode() + .constraints() + .fixed_value_arguments(); + let have_args = vlist.len(&pos.func.dfg.value_lists) - fixed_values; // Grow the value list to the right size and shift all the existing arguments to the right. // This lets us write the new argument values into the list without overwriting the old @@ -450,30 +445,34 @@ fn legalize_inst_arguments( // <------------------> abi_args // [FFFFNNNNNNNNNNNNNNNNNNNN] // - vlist.grow_at(fixed_values, abi_args - have_args, &mut dfg.value_lists); + vlist.grow_at( + fixed_values, + abi_args - have_args, + &mut pos.func.dfg.value_lists, + ); let old_arg_offset = fixed_values + abi_args - have_args; let mut abi_arg = 0; for old_arg in 0..have_args { let old_value = vlist - .get(old_arg_offset + old_arg, &dfg.value_lists) + .get(old_arg_offset + old_arg, &pos.func.dfg.value_lists) .unwrap(); - let mut put_arg = |dfg: &mut DataFlowGraph, arg| { - let abi_type = get_abi_type(dfg, abi_arg); - if dfg.value_type(arg) == abi_type.value_type { + let mut put_arg = |func: &mut Function, arg| { + let abi_type = get_abi_type(func, abi_arg); + if func.dfg.value_type(arg) == abi_type.value_type { // This is the argument type we need. - vlist.as_mut_slice(&mut dfg.value_lists)[fixed_values + abi_arg] = arg; + vlist.as_mut_slice(&mut func.dfg.value_lists)[fixed_values + abi_arg] = arg; abi_arg += 1; Ok(()) } else { Err(abi_type) } }; - convert_to_abi(dfg, cfg, pos, old_value, &mut put_arg); + convert_to_abi(pos, cfg, old_value, &mut put_arg); } // Put the modified value list back. - dfg[inst].put_value_list(vlist); + pos.func.dfg[inst].put_value_list(vlist); } /// Insert ABI conversion code before and after the call instruction at `pos`. @@ -487,39 +486,38 @@ fn legalize_inst_arguments( /// /// Returns `true` if any instructions were inserted. pub fn handle_call_abi(mut inst: Inst, func: &mut Function, cfg: &ControlFlowGraph) -> bool { - let dfg = &mut func.dfg; - let pos = &mut Cursor::new(&mut func.layout, &mut func.srclocs).at_inst(inst); + let pos = &mut FuncCursor::new(func).at_inst(inst); pos.use_srcloc(inst); // Start by checking if the argument types already match the signature. - let sig_ref = match check_call_signature(dfg, inst) { - Ok(_) => return spill_call_arguments(dfg, &mut func.locations, &mut func.stack_slots, pos), + let sig_ref = match check_call_signature(&pos.func.dfg, inst) { + Ok(_) => return spill_call_arguments(pos), Err(s) => s, }; // OK, we need to fix the call arguments to match the ABI signature. - let abi_args = dfg.signatures[sig_ref].argument_types.len(); - legalize_inst_arguments(dfg, cfg, pos, abi_args, |dfg, abi_arg| { - dfg.signatures[sig_ref].argument_types[abi_arg] + let abi_args = pos.func.dfg.signatures[sig_ref].argument_types.len(); + legalize_inst_arguments(pos, cfg, abi_args, |func, abi_arg| { + func.dfg.signatures[sig_ref].argument_types[abi_arg] }); - if !dfg.signatures[sig_ref].return_types.is_empty() { - inst = legalize_inst_results(dfg, pos, |dfg, abi_res| { - dfg.signatures[sig_ref].return_types[abi_res] + if !pos.func.dfg.signatures[sig_ref].return_types.is_empty() { + inst = legalize_inst_results(pos, |func, abi_res| { + func.dfg.signatures[sig_ref].return_types[abi_res] }); } debug_assert!( - check_call_signature(dfg, inst).is_ok(), + check_call_signature(&pos.func.dfg, inst).is_ok(), "Signature still wrong: {}, {}{}", - dfg.display_inst(inst, None), + pos.func.dfg.display_inst(inst, None), sig_ref, - dfg.signatures[sig_ref] + pos.func.dfg.signatures[sig_ref] ); // Go back and insert spills for any stack arguments. pos.goto_inst(inst); - spill_call_arguments(dfg, &mut func.locations, &mut func.stack_slots, pos); + spill_call_arguments(pos); // Yes, we changed stuff. true @@ -529,19 +527,15 @@ pub fn handle_call_abi(mut inst: Inst, func: &mut Function, cfg: &ControlFlowGra /// /// Return `true` if any instructions were inserted. pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph) -> bool { - let dfg = &mut func.dfg; - let sig = &mut func.signature; - let pos = &mut Cursor::new(&mut func.layout, &mut func.srclocs).at_inst(inst); - pos.use_srcloc(inst); - // Check if the returned types already match the signature. - if check_return_signature(dfg, inst, sig) { + if check_return_signature(&func.dfg, inst, &func.signature) { return false; } // Count the special-purpose return values (`link`, `sret`, and `vmctx`) that were appended to // the legalized signature. - let special_args = sig.return_types + let special_args = func.signature + .return_types .iter() .rev() .take_while(|&rt| { @@ -549,16 +543,15 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph rt.purpose == ArgumentPurpose::VMContext }) .count(); + let abi_args = func.signature.return_types.len() - special_args; - let abi_args = sig.return_types.len() - special_args; - legalize_inst_arguments( - dfg, - cfg, - pos, - abi_args, - |_, abi_arg| sig.return_types[abi_arg], - ); - assert_eq!(dfg.inst_variable_args(inst).len(), abi_args); + let pos = &mut FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); + + legalize_inst_arguments(pos, cfg, abi_args, |func, abi_arg| { + func.signature.return_types[abi_arg] + }); + assert_eq!(pos.func.dfg.inst_variable_args(inst).len(), abi_args); // Append special return arguments for any `sret`, `link`, and `vmctx` return values added to // the legalized signature. These values should simply be propagated from the entry block @@ -567,10 +560,10 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph dbg!( "Adding {} special-purpose arguments to {}", special_args, - dfg.display_inst(inst, None) + pos.func.dfg.display_inst(inst, None) ); - let mut vlist = dfg[inst].take_value_list().unwrap(); - for arg in &sig.return_types[abi_args..] { + let mut vlist = pos.func.dfg[inst].take_value_list().unwrap(); + for arg in &pos.func.signature.return_types[abi_args..] { match arg.purpose { ArgumentPurpose::Link | ArgumentPurpose::StructReturn | @@ -581,24 +574,29 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph // A `link`/`sret`/`vmctx` return value can only appear in a signature that has a // unique matching argument. They are appended at the end, so search the signature from // the end. - let idx = sig.argument_types + let idx = pos.func + .signature + .argument_types .iter() .rposition(|t| t.purpose == arg.purpose) .expect("No matching special purpose argument."); // Get the corresponding entry block value and add it to the return instruction's // arguments. - let val = dfg.ebb_args(pos.layout.entry_block().unwrap())[idx]; - debug_assert_eq!(dfg.value_type(val), arg.value_type); - vlist.push(val, &mut dfg.value_lists); + let val = pos.func.dfg.ebb_args( + pos.func.layout.entry_block().unwrap(), + ) + [idx]; + debug_assert_eq!(pos.func.dfg.value_type(val), arg.value_type); + vlist.push(val, &mut pos.func.dfg.value_lists); } - dfg[inst].put_value_list(vlist); + pos.func.dfg[inst].put_value_list(vlist); } debug_assert!( - check_return_signature(dfg, inst, sig), + check_return_signature(&pos.func.dfg, inst, &pos.func.signature), "Signature still wrong: {} / signature {}", - dfg.display_inst(inst, None), - sig + pos.func.dfg.display_inst(inst, None), + pos.func.signature ); // Yes, we changed stuff. @@ -629,49 +627,50 @@ fn spill_entry_arguments(func: &mut Function, entry: Ebb) { /// TODO: The outgoing stack slots can be written a bit earlier, as long as there are no branches /// or calls between writing the stack slots and the call instruction. Writing the slots earlier /// could help reduce register pressure before the call. -fn spill_call_arguments( - dfg: &mut DataFlowGraph, - locations: &mut ValueLocations, - stack_slots: &mut StackSlots, - pos: &mut Cursor, -) -> bool { +fn spill_call_arguments(pos: &mut FuncCursor) -> bool { let inst = pos.current_inst().expect( "Cursor must point to a call instruction", ); - let sig_ref = dfg.call_signature(inst).expect( + let sig_ref = pos.func.dfg.call_signature(inst).expect( "Call instruction expected.", ); // Start by building a list of stack slots and arguments to be replaced. - // This requires borrowing `dfg`, so we can't change anything. - let arglist = dfg.inst_variable_args(inst) - .iter() - .zip(&dfg.signatures[sig_ref].argument_types) - .enumerate() - .filter_map(|(idx, (&arg, abi))| { - match abi.location { - ArgumentLoc::Stack(offset) => { - // Is `arg` already in the right kind of stack slot? - match locations.get(arg) { - Some(&ValueLoc::Stack(ss)) => { - // We won't reassign `arg` to a different stack slot. Assert out of - // the stack slot is wrong. - assert_eq!(stack_slots[ss].kind, StackSlotKind::OutgoingArg); - assert_eq!(stack_slots[ss].offset, offset); - assert_eq!(stack_slots[ss].size, abi.value_type.bytes()); - None - } - _ => { - // Assign `arg` to a new stack slot. - let ss = stack_slots.get_outgoing_arg(abi.value_type, offset); - Some((idx, arg, ss)) + // This requires borrowing `pos.func.dfg`, so we can't change anything. + let arglist = { + let locations = &pos.func.locations; + let stack_slots = &mut pos.func.stack_slots; + pos.func + .dfg + .inst_variable_args(inst) + .iter() + .zip(&pos.func.dfg.signatures[sig_ref].argument_types) + .enumerate() + .filter_map(|(idx, (&arg, abi))| { + match abi.location { + ArgumentLoc::Stack(offset) => { + // Is `arg` already in the right kind of stack slot? + match locations.get(arg) { + Some(&ValueLoc::Stack(ss)) => { + // We won't reassign `arg` to a different stack slot. Assert out of + // the stack slot is wrong. + assert_eq!(stack_slots[ss].kind, StackSlotKind::OutgoingArg); + assert_eq!(stack_slots[ss].offset, offset); + assert_eq!(stack_slots[ss].size, abi.value_type.bytes()); + None + } + _ => { + // Assign `arg` to a new stack slot. + let ss = stack_slots.get_outgoing_arg(abi.value_type, offset); + Some((idx, arg, ss)) + } } } + _ => None, } - _ => None, - } - }) - .collect::>(); + }) + .collect::>() + }; if arglist.is_empty() { return false; @@ -679,9 +678,9 @@ fn spill_call_arguments( // Insert the spill instructions and rewrite call arguments. for (idx, arg, ss) in arglist { - let stack_val = dfg.ins(pos).spill(arg); - locations[stack_val] = ValueLoc::Stack(ss); - dfg.inst_variable_args_mut(inst)[idx] = stack_val; + let stack_val = pos.ins().spill(arg); + pos.func.locations[stack_val] = ValueLoc::Stack(ss); + pos.func.dfg.inst_variable_args_mut(inst)[idx] = stack_val; } // We changed stuff. From 2d4c8601875047211e482f7e3e4af92fb17329df Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 21 Sep 2017 17:05:51 -0700 Subject: [PATCH 1153/3084] Convert legalizer::split and generated legalization code to FuncCursor. --- lib/cretonne/meta/gen_legalizer.py | 40 +++++++------- lib/cretonne/src/legalizer/boundary.rs | 6 ++- lib/cretonne/src/legalizer/split.rs | 74 +++++++++++++------------- 3 files changed, 59 insertions(+), 61 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 41b64ede5d..0d8407bb30 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -152,7 +152,7 @@ def unwrap_inst(iref, node, fmt): # type: (str, Def, Formatter) -> bool """ Given a `Def` node, emit code that extracts all the instruction fields from - `dfg[iref]`. + `pos.func.dfg[iref]`. Create local variables named after the `Var` instances in `node`. @@ -181,7 +181,8 @@ def unwrap_inst(iref, node, fmt): elif iform.has_value_list or nvops > 1: fmt.line('ref args,') fmt.line('..') - fmt.outdented_line('} = dfg[inst] {') + fmt.outdented_line('} = pos.func.dfg[inst] {') + fmt.line('let dfg = &pos.func.dfg;') if iform.has_value_list: fmt.line('let args = args.as_slice(&dfg.value_lists);') elif nvops == 1: @@ -205,7 +206,7 @@ def unwrap_inst(iref, node, fmt): for opnum in expr.inst.value_opnums: v = expr.args[opnum] if isinstance(v, Var) and v.has_free_typevar(): - fmt.line('let typeof_{0} = dfg.value_type({0});'.format(v)) + fmt.format('let typeof_{0} = pos.func.dfg.value_type({0});', v) # If the node has results, detach the values. # Place the values in locals. @@ -223,13 +224,13 @@ def unwrap_inst(iref, node, fmt): for d in node.defs: fmt.line('let {};'.format(d)) with fmt.indented('{', '}'): - fmt.line('let r = dfg.inst_results(inst);') + fmt.line('let r = pos.func.dfg.inst_results(inst);') for i in range(len(node.defs)): fmt.line('{} = r[{}];'.format(node.defs[i], i)) for d in node.defs: if d.has_free_typevar(): fmt.line( - 'let typeof_{0} = dfg.value_type({0});' + 'let typeof_{0} = pos.func.dfg.value_type({0});' .format(d)) return replace_inst @@ -265,8 +266,9 @@ def emit_dst_inst(node, fmt): # special functions in the `legalizer::split` module. These functions # will eliminate concat-split patterns. fmt.line('let curpos = pos.position();') + fmt.line('let srcloc = pos.srcloc();') fmt.format( - 'let {} = split::{}(dfg, pos.layout, cfg, curpos, {});', + 'let {} = split::{}(pos.func, cfg, curpos, srcloc, {});', wrap_tup(node.defs), node.expr.inst.snake_name(), node.expr.args[0]) @@ -274,7 +276,7 @@ def emit_dst_inst(node, fmt): if len(node.defs) == 0: # This node doesn't define any values, so just insert the new # instruction. - builder = 'dfg.ins(pos)' + builder = 'pos.ins()' else: src_def0 = node.defs[0].src_def if src_def0 and node.defs == src_def0.defs: @@ -282,12 +284,12 @@ def emit_dst_inst(node, fmt): # the source pattern. Unwrapping would have left the results # intact. # Replace the whole instruction. - builder = 'let {} = dfg.replace(inst)'.format( + builder = 'let {} = pos.func.dfg.replace(inst)'.format( wrap_tup(node.defs)) replaced_inst = 'inst' else: # Insert a new instruction. - builder = 'let {} = dfg.ins(pos)'.format(wrap_tup(node.defs)) + builder = 'let {} = pos.ins()'.format(wrap_tup(node.defs)) # We may want to reuse some of the detached output values. if len(node.defs) == 1 and node.defs[0].is_output(): # Reuse the single source result value. @@ -335,7 +337,7 @@ def gen_xform(xform, fmt, type_sets): # If we're going to delete `inst`, we need to detach its results first # so they can be reattached during pattern expansion. if not replace_inst: - fmt.line('dfg.clear_results(inst);') + fmt.line('pos.func.dfg.clear_results(inst);') # Emit the destination pattern. for dst in xform.dst.rtl: @@ -357,8 +359,10 @@ def gen_xform_group(xgrp, fmt, type_sets): 'func: &mut ir::Function, ' 'cfg: &mut ::flowgraph::ControlFlowGraph) -> ' 'bool {{'.format(xgrp.name), '}'): - fmt.line('use ir::{InstBuilder, CursorBase};') - fmt.line('let srcloc = func.srclocs[inst];') + fmt.line('use ir::{InstBuilder};') + fmt.line('use cursor::{Cursor, FuncCursor};') + fmt.line('let pos = &mut FuncCursor::new(func).at_inst(inst);') + fmt.line('pos.use_srcloc(inst);') # Group the xforms by opcode so we can generate a big switch. # Preserve ordering. @@ -368,16 +372,10 @@ def gen_xform_group(xgrp, fmt, type_sets): xforms[inst.camel_name].append(xform) with fmt.indented('{', '}'): - with fmt.indented('match func.dfg[inst].opcode() {', '}'): + with fmt.indented('match pos.func.dfg[inst].opcode() {', '}'): for camel_name in sorted(xforms.keys()): with fmt.indented( 'ir::Opcode::{} => {{'.format(camel_name), '}'): - fmt.line( - 'let pos = &mut ir::Cursor::new' - '(&mut func.layout, &mut func.srclocs)' - '.with_srcloc(srcloc)' - '.at_inst(inst);') - fmt.line('let dfg = &mut func.dfg;') for xform in xforms[camel_name]: gen_xform(xform, fmt, type_sets) @@ -387,7 +385,7 @@ def gen_xform_group(xgrp, fmt, type_sets): with fmt.indented( 'ir::Opcode::{} => {{' .format(inst.camel_name), '}'): - fmt.format('{}(inst, func, cfg);', funcname) + fmt.format('{}(inst, pos.func, cfg);', funcname) fmt.line('return true;') # We'll assume there are uncovered opcodes. @@ -395,7 +393,7 @@ def gen_xform_group(xgrp, fmt, type_sets): # If we fall through, nothing was expanded. Call the chain if any. if xgrp.chain: - fmt.format('{}(inst, func, cfg)', xgrp.chain.rust_name()) + fmt.format('{}(inst, pos.func, cfg)', xgrp.chain.rust_name()) else: fmt.line('false') diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index c7b4627af1..5522db921f 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -325,13 +325,15 @@ fn convert_to_abi( match legalize_abi_value(ty, &arg_type) { ValueConversion::IntSplit => { let curpos = pos.position(); - let (lo, hi) = isplit(&mut pos.func.dfg, &mut pos.func.layout, cfg, curpos, value); + let srcloc = pos.srcloc(); + let (lo, hi) = isplit(&mut pos.func, cfg, curpos, srcloc, value); convert_to_abi(pos, cfg, lo, put_arg); convert_to_abi(pos, cfg, hi, put_arg); } ValueConversion::VectorSplit => { let curpos = pos.position(); - let (lo, hi) = vsplit(&mut pos.func.dfg, &mut pos.func.layout, cfg, curpos, value); + let srcloc = pos.srcloc(); + let (lo, hi) = vsplit(&mut pos.func, cfg, curpos, srcloc, value); convert_to_abi(pos, cfg, lo, put_arg); convert_to_abi(pos, cfg, hi, put_arg); } diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index bc73db6817..62e5391474 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -64,7 +64,7 @@ //! It is possible to have circular dependencies of EBB arguments that are never used by any real //! instructions. These loops will remain in the program. -use cursor::{Cursor, CursorPosition}; +use cursor::{Cursor, CursorPosition, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::{self, Ebb, Inst, Value, Type, Opcode, ValueDef, InstructionData, InstBuilder}; use std::iter; @@ -72,25 +72,25 @@ use std::iter; /// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values /// if possible. pub fn isplit( - dfg: &mut ir::DataFlowGraph, - layout: &mut ir::Layout, + func: &mut ir::Function, cfg: &ControlFlowGraph, pos: CursorPosition, + srcloc: ir::SourceLoc, value: Value, ) -> (Value, Value) { - split_any(dfg, layout, cfg, pos, value, Opcode::Iconcat) + split_any(func, cfg, pos, srcloc, value, Opcode::Iconcat) } /// Split `value` into halves using the `vsplit` semantics. Do this by reusing existing values if /// possible. pub fn vsplit( - dfg: &mut ir::DataFlowGraph, - layout: &mut ir::Layout, + func: &mut ir::Function, cfg: &ControlFlowGraph, pos: CursorPosition, + srcloc: ir::SourceLoc, value: Value, ) -> (Value, Value) { - split_any(dfg, layout, cfg, pos, value, Opcode::Vconcat) + split_any(func, cfg, pos, srcloc, value, Opcode::Vconcat) } /// After splitting an EBB argument, we need to go back and fix up all of the predecessor @@ -112,54 +112,53 @@ struct Repair { /// Generic version of `isplit` and `vsplit` controlled by the `concat` opcode. fn split_any( - dfg: &mut ir::DataFlowGraph, - layout: &mut ir::Layout, + func: &mut ir::Function, cfg: &ControlFlowGraph, pos: CursorPosition, + srcloc: ir::SourceLoc, value: Value, concat: Opcode, ) -> (Value, Value) { let mut repairs = Vec::new(); - let mut pos = ir::Cursor::new(layout, None).at_position(pos); - let result = split_value(dfg, &mut pos, value, concat, &mut repairs); + let pos = &mut FuncCursor::new(func).at_position(pos).with_srcloc(srcloc); + let result = split_value(pos, value, concat, &mut repairs); // We have split the value requested, and now we may need to fix some EBB predecessors. while let Some(repair) = repairs.pop() { for &(_, inst) in cfg.get_predecessors(repair.ebb) { - let branch_opc = dfg[inst].opcode(); + let branch_opc = pos.func.dfg[inst].opcode(); assert!( branch_opc.is_branch(), "Predecessor not a branch: {}", - dfg.display_inst(inst, None) + pos.func.dfg.display_inst(inst, None) ); let fixed_args = branch_opc.constraints().fixed_value_arguments(); - let mut args = dfg[inst].take_value_list().expect( + let mut args = pos.func.dfg[inst].take_value_list().expect( "Branches must have value lists.", ); - let num_args = args.len(&dfg.value_lists); + let num_args = args.len(&pos.func.dfg.value_lists); // Get the old value passed to the EBB argument we're repairing. - let old_arg = args.get(fixed_args + repair.num, &dfg.value_lists).expect( - "Too few branch arguments", - ); + let old_arg = args.get(fixed_args + repair.num, &pos.func.dfg.value_lists) + .expect("Too few branch arguments"); // It's possible that the CFG's predecessor list has duplicates. Detect them here. - if dfg.value_type(old_arg) == repair.split_type { - dfg[inst].put_value_list(args); + if pos.func.dfg.value_type(old_arg) == repair.split_type { + pos.func.dfg[inst].put_value_list(args); continue; } // Split the old argument, possibly causing more repairs to be scheduled. pos.goto_inst(inst); - let (lo, hi) = split_value(dfg, &mut pos, old_arg, repair.concat, &mut repairs); + let (lo, hi) = split_value(pos, old_arg, repair.concat, &mut repairs); // The `lo` part replaces the original argument. - *args.get_mut(fixed_args + repair.num, &mut dfg.value_lists) + *args.get_mut(fixed_args + repair.num, &mut pos.func.dfg.value_lists) .unwrap() = lo; // The `hi` part goes at the end. Since multiple repairs may have been scheduled to the // same EBB, there could be multiple arguments missing. if num_args > fixed_args + repair.hi_num { - *args.get_mut(fixed_args + repair.hi_num, &mut dfg.value_lists) + *args.get_mut(fixed_args + repair.hi_num, &mut pos.func.dfg.value_lists) .unwrap() = hi; } else { // We need to append one or more arguments. If we're adding more than one argument, @@ -167,12 +166,12 @@ fn split_any( // instead of `hi`. args.extend( iter::repeat(hi).take(1 + fixed_args + repair.hi_num - num_args), - &mut dfg.value_lists, + &mut pos.func.dfg.value_lists, ); } // Put the value list back after manipulating it. - dfg[inst].put_value_list(args); + pos.func.dfg[inst].put_value_list(args); } } @@ -186,20 +185,19 @@ fn split_any( /// /// Return the two new values representing the parts of `value`. fn split_value( - dfg: &mut ir::DataFlowGraph, - pos: &mut ir::Cursor, + pos: &mut FuncCursor, value: Value, concat: Opcode, repairs: &mut Vec, ) -> (Value, Value) { - let value = dfg.resolve_copies(value); + let value = pos.func.dfg.resolve_copies(value); let mut reuse = None; - match dfg.value_def(value) { + match pos.func.dfg.value_def(value) { ValueDef::Res(inst, num) => { // This is an instruction result. See if the value was created by a `concat` // instruction. - if let InstructionData::Binary { opcode, args, .. } = dfg[inst] { + if let InstructionData::Binary { opcode, args, .. } = pos.func.dfg[inst] { assert_eq!(num, 0); if opcode == concat { reuse = Some((args[0], args[1])); @@ -209,10 +207,10 @@ fn split_value( ValueDef::Arg(ebb, num) => { // This is an EBB argument. We can split the argument value unless this is the entry // block. - if pos.layout.entry_block() != Some(ebb) { + if pos.func.layout.entry_block() != Some(ebb) { // We are going to replace the argument at `num` with two new arguments. // Determine the new value types. - let ty = dfg.value_type(value); + let ty = pos.func.dfg.value_type(value); let split_type = match concat { Opcode::Iconcat => ty.half_width().expect("Invalid type for isplit"), Opcode::Vconcat => ty.half_vector().expect("Invalid type for vsplit"), @@ -225,9 +223,9 @@ fn split_value( // // Replace the original `value` with the low part, and append the high part at the // end of the argument list. - let lo = dfg.replace_ebb_arg(value, split_type); - let hi_num = dfg.num_ebb_args(ebb); - let hi = dfg.append_ebb_arg(ebb, split_type); + let lo = pos.func.dfg.replace_ebb_arg(value, split_type); + let hi_num = pos.func.dfg.num_ebb_args(ebb); + let hi = pos.func.dfg.append_ebb_arg(ebb, split_type); reuse = Some((lo, hi)); @@ -238,7 +236,7 @@ fn split_value( // Note that it is safe to move `pos` here since `reuse` was set above, so we don't // need to insert a split instruction before returning. pos.goto_first_inst(ebb); - dfg.ins(pos).with_result(value).Binary( + pos.ins().with_result(value).Binary( concat, split_type, lo, @@ -259,8 +257,8 @@ fn split_value( // No, we'll just have to insert the requested split instruction at `pos`. Note that `pos` // has not been moved by the EBB argument code above when `reuse` is `None`. match concat { - Opcode::Iconcat => dfg.ins(pos).isplit(value), - Opcode::Vconcat => dfg.ins(pos).vsplit(value), + Opcode::Iconcat => pos.ins().isplit(value), + Opcode::Vconcat => pos.ins().vsplit(value), _ => panic!("Unhandled concat opcode: {}", concat), } } From 3b66c0be4037b7de5bb75df1a1ae6f8436293ad7 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Fri, 22 Sep 2017 00:40:43 +0100 Subject: [PATCH 1154/3084] Emit compressed instruction encodings for instructions where constraints allow --- cranelift/filetests/isa/intel/binary64.cton | 1 + cranelift/src/filetest/binemit.rs | 22 ++++++-- lib/cretonne/meta/isa/intel/recipes.py | 2 +- lib/cretonne/src/isa/constraints.rs | 59 +++++++++++++++++++++ 4 files changed, 80 insertions(+), 4 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index e67f630753..a70c9fdf08 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -1,6 +1,7 @@ ; binary emission of 64-bit code. test binemit set is_64bit +set is_compressed isa intel haswell ; The binary encodings can be verified with the command: diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 6fd4ed1fcf..0ea634548a 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -104,16 +104,32 @@ impl SubTest for TestBinEmit { // value locations. The current error reporting is just crashing... let mut func = func.into_owned(); + let is_compressed = isa.flags().is_compressed(); + // Give an encoding to any instruction that doesn't already have one. for ebb in func.layout.ebbs() { for inst in func.layout.ebb_insts(ebb) { if !func.encodings[inst].is_legal() { - if let Ok(enc) = isa.encode( + let mut legal_encodings = isa.legal_encodings( &func.dfg, &func.dfg[inst], func.dfg.ctrl_typevar(inst), - ) - { + ); + + let enc = if is_compressed { + // Get the smallest legal encoding + legal_encodings + .filter(|e| { + let recipe_constraints = &encinfo.constraints[e.recipe()]; + recipe_constraints.satisfied(inst, &func) + }) + .min_by_key(|&e| encinfo.bytes(e)) + } else { + // If not using compressed, just use the first encoding. + legal_encodings.next() + }; + + if let Some(enc) = enc { func.encodings[inst] = enc; } } diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 8d07f06de1..c271ab1e34 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -13,7 +13,7 @@ from .registers import GPR, ABCD, FPR try: from typing import Tuple, Dict # noqa from cdsl.instructions import InstructionFormat # noqa - from cdsl.isa import ConstraintSeq, BranchRange, PredNode # noqa + from cdsl.isa import ConstraintSeq, BranchRange, PredNode, OperandConstraint # noqa except ImportError: pass diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index c5f1d4bd16..d9fc3aa8ea 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -9,6 +9,7 @@ use binemit::CodeOffset; use isa::{RegClass, RegUnit}; +use ir::{Function, ValueLoc, Inst}; /// Register constraint for a single value operand or instruction result. pub struct OperandConstraint { @@ -21,6 +22,34 @@ pub struct OperandConstraint { pub regclass: RegClass, } +impl OperandConstraint { + /// Check if this operand constraint is satisfied by the given value location. + /// For tied constraints, this only checks the register class, not that the + /// counterpart operand has the same value location. + pub fn satisfied(&self, loc: ValueLoc) -> bool { + match self.kind { + ConstraintKind::Reg | + ConstraintKind::Tied(_) => { + if let ValueLoc::Reg(reg) = loc { + self.regclass.contains(reg) + } else { + false + } + } + ConstraintKind::FixedReg(reg) => { + loc == ValueLoc::Reg(reg) && self.regclass.contains(reg) + } + ConstraintKind::Stack => { + if let ValueLoc::Stack(_) = loc { + true + } else { + false + } + } + } + } +} + /// The different kinds of operand constraints. #[derive(Clone, Copy, PartialEq, Eq)] pub enum ConstraintKind { @@ -78,6 +107,36 @@ pub struct RecipeConstraints { pub tied_ops: bool, } +impl RecipeConstraints { + /// Check that these constraints are satisfied by the operands on `inst`. + pub fn satisfied(&self, inst: Inst, func: &Function) -> bool { + for (&arg, constraint) in func.dfg.inst_args(inst).iter().zip(self.ins) { + let loc = func.locations[arg]; + + if let ConstraintKind::Tied(out_index) = constraint.kind { + let out_val = func.dfg.inst_results(inst)[out_index as usize]; + let out_loc = func.locations[out_val]; + if loc != out_loc { + return false; + } + } + + if !constraint.satisfied(loc) { + return false; + } + } + + for (&arg, constraint) in func.dfg.inst_results(inst).iter().zip(self.outs) { + let loc = func.locations[arg]; + if !constraint.satisfied(loc) { + return false; + } + } + + true + } +} + /// Constraints on the range of a branch instruction. /// /// A branch instruction usually encodes its destination as a signed n-bit offset from an origin. From b003605132b1d3794cdfdcd80896ca3fa7440065 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Fri, 22 Sep 2017 00:49:21 +0100 Subject: [PATCH 1155/3084] Adapt intel to be able to correctly choose compressed instruction encodings: create a register class to identify the lower 8 registers, omit unnecessary REX prefixes, and fix the tests --- cranelift/filetests/isa/intel/binary64.cton | 196 ++++++++++---------- lib/cretonne/meta/check.sh | 2 +- lib/cretonne/meta/isa/intel/encodings.py | 23 ++- lib/cretonne/meta/isa/intel/recipes.py | 22 ++- lib/cretonne/meta/isa/intel/registers.py | 2 + 5 files changed, 136 insertions(+), 109 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index a70c9fdf08..8b22173fe4 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -220,9 +220,9 @@ ebb0: ; asm: movq -50(%r10), %rdx [-,%rdx] v141 = load.i64 v3-50 ; bin: 49 8b 52 ce ; asm: movl 50(%rcx), %edi - [-,%rdi] v142 = uload32.i64 v1+50 ; bin: 40 8b 79 32 + [-,%rdi] v142 = uload32.i64 v1+50 ; bin: 8b 79 32 ; asm: movl -50(%rsi), %edx - [-,%rdx] v143 = uload32.i64 v2-50 ; bin: 40 8b 56 ce + [-,%rdx] v143 = uload32.i64 v2-50 ; bin: 8b 56 ce ; asm: movslq 50(%rcx), %rdi [-,%rdi] v144 = sload32.i64 v1+50 ; bin: 48 63 79 32 ; asm: movslq -50(%rsi), %rdx @@ -251,15 +251,15 @@ ebb0: ; asm: movq %r10, -10000(%rcx) store v3, v1-10000 ; bin: 4c 89 91 ffffd8f0 ; asm: movl %ecx, 10000(%rsi) - istore32 v1, v2+10000 ; bin: 40 89 8e 00002710 + istore32 v1, v2+10000 ; bin: 89 8e 00002710 ; asm: movl %esi, -10000(%rcx) - istore32 v2, v1-10000 ; bin: 40 89 b1 ffffd8f0 + istore32 v2, v1-10000 ; bin: 89 b1 ffffd8f0 ; asm: movw %cx, 10000(%rsi) - istore16 v1, v2+10000 ; bin: 66 40 89 8e 00002710 + istore16 v1, v2+10000 ; bin: 66 89 8e 00002710 ; asm: movw %si, -10000(%rcx) - istore16 v2, v1-10000 ; bin: 66 40 89 b1 ffffd8f0 + istore16 v2, v1-10000 ; bin: 66 89 b1 ffffd8f0 ; asm: movb %cl, 10000(%rsi) - istore8 v1, v2+10000 ; bin: 40 88 8e 00002710 + istore8 v1, v2+10000 ; bin: 88 8e 00002710 ; asm: movb %sil, 10000(%rcx) istore8 v2, v1+10000 ; bin: 40 88 b1 00002710 @@ -268,9 +268,9 @@ ebb0: ; asm: movq -50000(%r10), %rdx [-,%rdx] v161 = load.i64 v3-50000 ; bin: 49 8b 92 ffff3cb0 ; asm: movl 50000(%rcx), %edi - [-,%rdi] v162 = uload32.i64 v1+50000 ; bin: 40 8b b9 0000c350 + [-,%rdi] v162 = uload32.i64 v1+50000 ; bin: 8b b9 0000c350 ; asm: movl -50000(%rsi), %edx - [-,%rdx] v163 = uload32.i64 v2-50000 ; bin: 40 8b 96 ffff3cb0 + [-,%rdx] v163 = uload32.i64 v2-50000 ; bin: 8b 96 ffff3cb0 ; asm: movslq 50000(%rcx), %rdi [-,%rdi] v164 = sload32.i64 v1+50000 ; bin: 48 63 b9 0000c350 ; asm: movslq -50000(%rsi), %rdx @@ -305,17 +305,17 @@ ebb0: [-,%rax] v190 = iconst.i64 1 [-,%rdx] v191 = iconst.i64 2 ; asm: idivq %rcx - [-,%rax,%rdx] v192, v193 = x86_sdivmodx v130, v131, v1 ; bin: 48 f7 f9 + [-,%rax,%rdx] v192, v193 = x86_sdivmodx v190, v191, v1 ; bin: 48 f7 f9 ; asm: idivq %rsi - [-,%rax,%rdx] v194, v195 = x86_sdivmodx v130, v131, v2 ; bin: 48 f7 fe + [-,%rax,%rdx] v194, v195 = x86_sdivmodx v190, v191, v2 ; bin: 48 f7 fe ; asm: idivq %r10 - [-,%rax,%rdx] v196, v197 = x86_sdivmodx v130, v131, v3 ; bin: 49 f7 fa + [-,%rax,%rdx] v196, v197 = x86_sdivmodx v190, v191, v3 ; bin: 49 f7 fa ; asm: divq %rcx - [-,%rax,%rdx] v198, v199 = x86_udivmodx v130, v131, v1 ; bin: 48 f7 f1 + [-,%rax,%rdx] v198, v199 = x86_udivmodx v190, v191, v1 ; bin: 48 f7 f1 ; asm: divq %rsi - [-,%rax,%rdx] v200, v201 = x86_udivmodx v130, v131, v2 ; bin: 48 f7 f6 + [-,%rax,%rdx] v200, v201 = x86_udivmodx v190, v191, v2 ; bin: 48 f7 f6 ; asm: divq %r10 - [-,%rax,%rdx] v202, v203 = x86_udivmodx v130, v131, v3 ; bin: 49 f7 f2 + [-,%rax,%rdx] v202, v203 = x86_udivmodx v190, v191, v3 ; bin: 49 f7 f2 ; Bit-counting instructions. @@ -415,9 +415,9 @@ ebb0: ; Bool-to-int conversions. ; asm: movzbq %bl, %rcx - [-,%rcx] v350 = bint.i64 v300 ; bin: 48 0f b6 cb + [-,%rcx] v350 = bint.i64 v300 ; bin: 0f b6 cb ; asm: movzbq %dl, %rsi - [-,%rsi] v351 = bint.i64 v301 ; bin: 48 0f b6 f2 + [-,%rsi] v351 = bint.i64 v301 ; bin: 0f b6 f2 ; asm: call foo call fn0() ; bin: e8 PCRel4(fn0) 00000000 @@ -430,9 +430,9 @@ ebb0: [-,%r10] v402 = func_addr.i64 fn0 ; bin: 49 ba Abs8(fn0) ffffffffffffffff ; asm: call *%rcx - call_indirect sig0, v400() ; bin: 40 ff d1 + call_indirect sig0, v400() ; bin: ff d1 ; asm: call *%rsi - call_indirect sig0, v401() ; bin: 40 ff d6 + call_indirect sig0, v401() ; bin: ff d6 ; asm: call *%r10 call_indirect sig0, v402() ; bin: 41 ff d2 @@ -482,9 +482,9 @@ ebb0: ; Integer Constants. ; asm: movl $0x01020304, %ecx - [-,%rcx] v1 = iconst.i32 0x0102_0304 ; bin: 40 b9 01020304 + [-,%rcx] v1 = iconst.i32 0x0102_0304 ; bin: b9 01020304 ; asm: movl $0x11020304, %esi - [-,%rsi] v2 = iconst.i32 0x1102_0304 ; bin: 40 be 11020304 + [-,%rsi] v2 = iconst.i32 0x1102_0304 ; bin: be 11020304 ; asm: movl $0x21020304, %r10d [-,%r10] v3 = iconst.i32 0x2102_0304 ; bin: 41 ba 21020304 ; asm: movl $0xff001122, %r8d @@ -497,128 +497,128 @@ ebb0: ; Register indirect addressing with no displacement. ; asm: movl (%rcx), %edi - [-,%rdi] v10 = load.i32 v1 ; bin: 40 8b 39 + [-,%rdi] v10 = load.i32 v1 ; bin: 8b 39 ; asm: movl (%rsi), %edx - [-,%rdx] v11 = load.i32 v2 ; bin: 40 8b 16 + [-,%rdx] v11 = load.i32 v2 ; bin: 8b 16 ; asm: movzwl (%rcx), %edi - [-,%rdi] v12 = uload16.i32 v1 ; bin: 40 0f b7 39 + [-,%rdi] v12 = uload16.i32 v1 ; bin: 0f b7 39 ; asm: movzwl (%rsi), %edx - [-,%rdx] v13 = uload16.i32 v2 ; bin: 40 0f b7 16 + [-,%rdx] v13 = uload16.i32 v2 ; bin: 0f b7 16 ; asm: movswl (%rcx), %edi - [-,%rdi] v14 = sload16.i32 v1 ; bin: 40 0f bf 39 + [-,%rdi] v14 = sload16.i32 v1 ; bin: 0f bf 39 ; asm: movswl (%rsi), %edx - [-,%rdx] v15 = sload16.i32 v2 ; bin: 40 0f bf 16 + [-,%rdx] v15 = sload16.i32 v2 ; bin: 0f bf 16 ; asm: movzbl (%rcx), %edi - [-,%rdi] v16 = uload8.i32 v1 ; bin: 40 0f b6 39 + [-,%rdi] v16 = uload8.i32 v1 ; bin: 0f b6 39 ; asm: movzbl (%rsi), %edx - [-,%rdx] v17 = uload8.i32 v2 ; bin: 40 0f b6 16 + [-,%rdx] v17 = uload8.i32 v2 ; bin: 0f b6 16 ; asm: movsbl (%rcx), %edi - [-,%rdi] v18 = sload8.i32 v1 ; bin: 40 0f be 39 + [-,%rdi] v18 = sload8.i32 v1 ; bin: 0f be 39 ; asm: movsbl (%rsi), %edx - [-,%rdx] v19 = sload8.i32 v2 ; bin: 40 0f be 16 + [-,%rdx] v19 = sload8.i32 v2 ; bin: 0f be 16 ; Register-indirect with 8-bit signed displacement. ; asm: movl 50(%rcx), %edi - [-,%rdi] v20 = load.i32 v1+50 ; bin: 40 8b 79 32 + [-,%rdi] v20 = load.i32 v1+50 ; bin: 8b 79 32 ; asm: movl -50(%rsi), %edx - [-,%rdx] v21 = load.i32 v2-50 ; bin: 40 8b 56 ce + [-,%rdx] v21 = load.i32 v2-50 ; bin: 8b 56 ce ; asm: movzwl 50(%rcx), %edi - [-,%rdi] v22 = uload16.i32 v1+50 ; bin: 40 0f b7 79 32 + [-,%rdi] v22 = uload16.i32 v1+50 ; bin: 0f b7 79 32 ; asm: movzwl -50(%rsi), %edx - [-,%rdx] v23 = uload16.i32 v2-50 ; bin: 40 0f b7 56 ce + [-,%rdx] v23 = uload16.i32 v2-50 ; bin: 0f b7 56 ce ; asm: movswl 50(%rcx), %edi - [-,%rdi] v24 = sload16.i32 v1+50 ; bin: 40 0f bf 79 32 + [-,%rdi] v24 = sload16.i32 v1+50 ; bin: 0f bf 79 32 ; asm: movswl -50(%rsi), %edx - [-,%rdx] v25 = sload16.i32 v2-50 ; bin: 40 0f bf 56 ce + [-,%rdx] v25 = sload16.i32 v2-50 ; bin: 0f bf 56 ce ; asm: movzbl 50(%rcx), %edi - [-,%rdi] v26 = uload8.i32 v1+50 ; bin: 40 0f b6 79 32 + [-,%rdi] v26 = uload8.i32 v1+50 ; bin: 0f b6 79 32 ; asm: movzbl -50(%rsi), %edx - [-,%rdx] v27 = uload8.i32 v2-50 ; bin: 40 0f b6 56 ce + [-,%rdx] v27 = uload8.i32 v2-50 ; bin: 0f b6 56 ce ; asm: movsbl 50(%rcx), %edi - [-,%rdi] v28 = sload8.i32 v1+50 ; bin: 40 0f be 79 32 + [-,%rdi] v28 = sload8.i32 v1+50 ; bin: 0f be 79 32 ; asm: movsbl -50(%rsi), %edx - [-,%rdx] v29 = sload8.i32 v2-50 ; bin: 40 0f be 56 ce + [-,%rdx] v29 = sload8.i32 v2-50 ; bin: 0f be 56 ce ; Register-indirect with 32-bit signed displacement. ; asm: movl 50000(%rcx), %edi - [-,%rdi] v30 = load.i32 v1+50000 ; bin: 40 8b b9 0000c350 + [-,%rdi] v30 = load.i32 v1+50000 ; bin: 8b b9 0000c350 ; asm: movl -50000(%rsi), %edx - [-,%rdx] v31 = load.i32 v2-50000 ; bin: 40 8b 96 ffff3cb0 + [-,%rdx] v31 = load.i32 v2-50000 ; bin: 8b 96 ffff3cb0 ; asm: movzwl 50000(%rcx), %edi - [-,%rdi] v32 = uload16.i32 v1+50000 ; bin: 40 0f b7 b9 0000c350 + [-,%rdi] v32 = uload16.i32 v1+50000 ; bin: 0f b7 b9 0000c350 ; asm: movzwl -50000(%rsi), %edx - [-,%rdx] v33 = uload16.i32 v2-50000 ; bin: 40 0f b7 96 ffff3cb0 + [-,%rdx] v33 = uload16.i32 v2-50000 ; bin: 0f b7 96 ffff3cb0 ; asm: movswl 50000(%rcx), %edi - [-,%rdi] v34 = sload16.i32 v1+50000 ; bin: 40 0f bf b9 0000c350 + [-,%rdi] v34 = sload16.i32 v1+50000 ; bin: 0f bf b9 0000c350 ; asm: movswl -50000(%rsi), %edx - [-,%rdx] v35 = sload16.i32 v2-50000 ; bin: 40 0f bf 96 ffff3cb0 + [-,%rdx] v35 = sload16.i32 v2-50000 ; bin: 0f bf 96 ffff3cb0 ; asm: movzbl 50000(%rcx), %edi - [-,%rdi] v36 = uload8.i32 v1+50000 ; bin: 40 0f b6 b9 0000c350 + [-,%rdi] v36 = uload8.i32 v1+50000 ; bin: 0f b6 b9 0000c350 ; asm: movzbl -50000(%rsi), %edx - [-,%rdx] v37 = uload8.i32 v2-50000 ; bin: 40 0f b6 96 ffff3cb0 + [-,%rdx] v37 = uload8.i32 v2-50000 ; bin: 0f b6 96 ffff3cb0 ; asm: movsbl 50000(%rcx), %edi - [-,%rdi] v38 = sload8.i32 v1+50000 ; bin: 40 0f be b9 0000c350 + [-,%rdi] v38 = sload8.i32 v1+50000 ; bin: 0f be b9 0000c350 ; asm: movsbl -50000(%rsi), %edx - [-,%rdx] v39 = sload8.i32 v2-50000 ; bin: 40 0f be 96 ffff3cb0 + [-,%rdx] v39 = sload8.i32 v2-50000 ; bin: 0f be 96 ffff3cb0 ; Integer Register-Register Operations. ; asm: addl %esi, %ecx - [-,%rcx] v40 = iadd v1, v2 ; bin: 40 01 f1 + [-,%rcx] v40 = iadd v1, v2 ; bin: 01 f1 ; asm: addl %r10d, %esi [-,%rsi] v41 = iadd v2, v3 ; bin: 44 01 d6 ; asm: addl %ecx, %r10d [-,%r10] v42 = iadd v3, v1 ; bin: 41 01 ca ; asm: subl %esi, %ecx - [-,%rcx] v50 = isub v1, v2 ; bin: 40 29 f1 + [-,%rcx] v50 = isub v1, v2 ; bin: 29 f1 ; asm: subl %r10d, %esi [-,%rsi] v51 = isub v2, v3 ; bin: 44 29 d6 ; asm: subl %ecx, %r10d [-,%r10] v52 = isub v3, v1 ; bin: 41 29 ca ; asm: andl %esi, %ecx - [-,%rcx] v60 = band v1, v2 ; bin: 40 21 f1 + [-,%rcx] v60 = band v1, v2 ; bin: 21 f1 ; asm: andl %r10d, %esi [-,%rsi] v61 = band v2, v3 ; bin: 44 21 d6 ; asm: andl %ecx, %r10d [-,%r10] v62 = band v3, v1 ; bin: 41 21 ca ; asm: orl %esi, %ecx - [-,%rcx] v70 = bor v1, v2 ; bin: 40 09 f1 + [-,%rcx] v70 = bor v1, v2 ; bin: 09 f1 ; asm: orl %r10d, %esi [-,%rsi] v71 = bor v2, v3 ; bin: 44 09 d6 ; asm: orl %ecx, %r10d [-,%r10] v72 = bor v3, v1 ; bin: 41 09 ca ; asm: xorl %esi, %ecx - [-,%rcx] v80 = bxor v1, v2 ; bin: 40 31 f1 + [-,%rcx] v80 = bxor v1, v2 ; bin: 31 f1 ; asm: xorl %r10d, %esi [-,%rsi] v81 = bxor v2, v3 ; bin: 44 31 d6 ; asm: xorl %ecx, %r10d [-,%r10] v82 = bxor v3, v1 ; bin: 41 31 ca ; asm: shll %cl, %esi - [-,%rsi] v90 = ishl v2, v1 ; bin: 40 d3 e6 + [-,%rsi] v90 = ishl v2, v1 ; bin: d3 e6 ; asm: shll %cl, %r10d [-,%r10] v91 = ishl v3, v1 ; bin: 41 d3 e2 ; asm: sarl %cl, %esi - [-,%rsi] v92 = sshr v2, v1 ; bin: 40 d3 fe + [-,%rsi] v92 = sshr v2, v1 ; bin: d3 fe ; asm: sarl %cl, %r10d [-,%r10] v93 = sshr v3, v1 ; bin: 41 d3 fa ; asm: shrl %cl, %esi - [-,%rsi] v94 = ushr v2, v1 ; bin: 40 d3 ee + [-,%rsi] v94 = ushr v2, v1 ; bin: d3 ee ; asm: shrl %cl, %r10d [-,%r10] v95 = ushr v3, v1 ; bin: 41 d3 ea ; asm: roll %cl, %esi - [-,%rsi] v96 = rotl v2, v1 ; bin: 40 d3 c6 + [-,%rsi] v96 = rotl v2, v1 ; bin: d3 c6 ; asm: roll %cl, %r10d [-,%r10] v97 = rotl v3, v1 ; bin: 41 d3 c2 ; asm: rorl %cl, %esi - [-,%rsi] v98 = rotr v2, v1 ; bin: 40 d3 ce + [-,%rsi] v98 = rotr v2, v1 ; bin: d3 ce ; asm: rorl %cl, %r10d [-,%r10] v99 = rotr v3, v1 ; bin: 41 d3 ca @@ -627,9 +627,9 @@ ebb0: ; Some take 8-bit immediates that are sign-extended to 64 bits. ; asm: addl $-100000, %ecx - [-,%rcx] v100 = iadd_imm v1, -100000 ; bin: 40 81 c1 fffe7960 + [-,%rcx] v100 = iadd_imm v1, -100000 ; bin: 81 c1 fffe7960 ; asm: addl $100000, %esi - [-,%rsi] v101 = iadd_imm v2, 100000 ; bin: 40 81 c6 000186a0 + [-,%rsi] v101 = iadd_imm v2, 100000 ; bin: 81 c6 000186a0 ; asm: addl $0x7fffffff, %r10d [-,%r10] v102 = iadd_imm v3, 0x7fff_ffff ; bin: 41 81 c2 7fffffff ; asm: addl $100, %r8d @@ -638,9 +638,9 @@ ebb0: [-,%r14] v104 = iadd_imm v5, -100 ; bin: 41 83 c6 9c ; asm: andl $-100000, %ecx - [-,%rcx] v110 = band_imm v1, -100000 ; bin: 40 81 e1 fffe7960 + [-,%rcx] v110 = band_imm v1, -100000 ; bin: 81 e1 fffe7960 ; asm: andl $100000, %esi - [-,%rsi] v111 = band_imm v2, 100000 ; bin: 40 81 e6 000186a0 + [-,%rsi] v111 = band_imm v2, 100000 ; bin: 81 e6 000186a0 ; asm: andl $0x7fffffff, %r10d [-,%r10] v112 = band_imm v3, 0x7fff_ffff ; bin: 41 81 e2 7fffffff ; asm: andl $100, %r8d @@ -649,9 +649,9 @@ ebb0: [-,%r14] v114 = band_imm v5, -100 ; bin: 41 83 e6 9c ; asm: orl $-100000, %ecx - [-,%rcx] v120 = bor_imm v1, -100000 ; bin: 40 81 c9 fffe7960 + [-,%rcx] v120 = bor_imm v1, -100000 ; bin: 81 c9 fffe7960 ; asm: orl $100000, %esi - [-,%rsi] v121 = bor_imm v2, 100000 ; bin: 40 81 ce 000186a0 + [-,%rsi] v121 = bor_imm v2, 100000 ; bin: 81 ce 000186a0 ; asm: orl $0x7fffffff, %r10d [-,%r10] v122 = bor_imm v3, 0x7fff_ffff ; bin: 41 81 ca 7fffffff ; asm: orl $100, %r8d @@ -661,9 +661,9 @@ ebb0: ; asm: ret ; asm: xorl $-100000, %ecx - [-,%rcx] v130 = bxor_imm v1, -100000 ; bin: 40 81 f1 fffe7960 + [-,%rcx] v130 = bxor_imm v1, -100000 ; bin: 81 f1 fffe7960 ; asm: xorl $100000, %esi - [-,%rsi] v131 = bxor_imm v2, 100000 ; bin: 40 81 f6 000186a0 + [-,%rsi] v131 = bxor_imm v2, 100000 ; bin: 81 f6 000186a0 ; asm: xorl $0x7fffffff, %r10d [-,%r10] v132 = bxor_imm v3, 0x7fff_ffff ; bin: 41 81 f2 7fffffff ; asm: xorl $100, %r8d @@ -674,7 +674,7 @@ ebb0: ; Register copies. ; asm: movl %esi, %ecx - [-,%rcx] v140 = copy v2 ; bin: 40 89 f1 + [-,%rcx] v140 = copy v2 ; bin: 89 f1 ; asm: movl %r10d, %esi [-,%rsi] v141 = copy v3 ; bin: 44 89 d6 ; asm: movl %ecx, %r10d @@ -683,7 +683,7 @@ ebb0: ; More arithmetic. ; asm: imull %esi, %ecx - [-,%rcx] v150 = imul v1, v2 ; bin: 40 0f af ce + [-,%rcx] v150 = imul v1, v2 ; bin: 0f af ce ; asm: imull %r10d, %esi [-,%rsi] v151 = imul v2, v3 ; bin: 41 0f af f2 ; asm: imull %ecx, %r10d @@ -692,36 +692,36 @@ ebb0: [-,%rax] v160 = iconst.i32 1 [-,%rdx] v161 = iconst.i32 2 ; asm: idivl %ecx - [-,%rax,%rdx] v162, v163 = x86_sdivmodx v130, v131, v1 ; bin: 40 f7 f9 + [-,%rax,%rdx] v162, v163 = x86_sdivmodx v160, v161, v1 ; bin: f7 f9 ; asm: idivl %esi - [-,%rax,%rdx] v164, v165 = x86_sdivmodx v130, v131, v2 ; bin: 40 f7 fe + [-,%rax,%rdx] v164, v165 = x86_sdivmodx v160, v161, v2 ; bin: f7 fe ; asm: idivl %r10d - [-,%rax,%rdx] v166, v167 = x86_sdivmodx v130, v131, v3 ; bin: 41 f7 fa + [-,%rax,%rdx] v166, v167 = x86_sdivmodx v160, v161, v3 ; bin: 41 f7 fa ; asm: divl %ecx - [-,%rax,%rdx] v168, v169 = x86_udivmodx v130, v131, v1 ; bin: 40 f7 f1 + [-,%rax,%rdx] v168, v169 = x86_udivmodx v160, v161, v1 ; bin: f7 f1 ; asm: divl %esi - [-,%rax,%rdx] v170, v171 = x86_udivmodx v130, v131, v2 ; bin: 40 f7 f6 + [-,%rax,%rdx] v170, v171 = x86_udivmodx v160, v161, v2 ; bin: f7 f6 ; asm: divl %r10d - [-,%rax,%rdx] v172, v173 = x86_udivmodx v130, v131, v3 ; bin: 41 f7 f2 + [-,%rax,%rdx] v172, v173 = x86_udivmodx v160, v161, v3 ; bin: 41 f7 f2 ; Bit-counting instructions. ; asm: popcntl %esi, %ecx - [-,%rcx] v200 = popcnt v2 ; bin: f3 40 0f b8 ce + [-,%rcx] v200 = popcnt v2 ; bin: f3 0f b8 ce ; asm: popcntl %r10d, %esi [-,%rsi] v201 = popcnt v3 ; bin: f3 41 0f b8 f2 ; asm: popcntl %ecx, %r10d [-,%r10] v202 = popcnt v1 ; bin: f3 44 0f b8 d1 ; asm: lzcntl %esi, %ecx - [-,%rcx] v203 = clz v2 ; bin: f3 40 0f bd ce + [-,%rcx] v203 = clz v2 ; bin: f3 0f bd ce ; asm: lzcntl %r10d, %esi [-,%rsi] v204 = clz v3 ; bin: f3 41 0f bd f2 ; asm: lzcntl %ecx, %r10d [-,%r10] v205 = clz v1 ; bin: f3 44 0f bd d1 ; asm: tzcntl %esi, %ecx - [-,%rcx] v206 = ctz v2 ; bin: f3 40 0f bc ce + [-,%rcx] v206 = ctz v2 ; bin: f3 0f bc ce ; asm: tzcntl %r10d, %esi [-,%rsi] v207 = ctz v3 ; bin: f3 41 0f bc f2 ; asm: tzcntl %ecx, %r10d @@ -731,70 +731,70 @@ ebb0: ; asm: cmpl %esi, %ecx ; asm: sete %bl - [-,%rbx] v300 = icmp eq v1, v2 ; bin: 40 39 f1 0f 94 c3 + [-,%rbx] v300 = icmp eq v1, v2 ; bin: 39 f1 0f 94 c3 ; asm: cmpl %r10d, %esi ; asm: sete %dl [-,%rdx] v301 = icmp eq v2, v3 ; bin: 44 39 d6 0f 94 c2 ; asm: cmpl %esi, %ecx ; asm: setne %bl - [-,%rbx] v302 = icmp ne v1, v2 ; bin: 40 39 f1 0f 95 c3 + [-,%rbx] v302 = icmp ne v1, v2 ; bin: 39 f1 0f 95 c3 ; asm: cmpl %r10d, %esi ; asm: setne %dl [-,%rdx] v303 = icmp ne v2, v3 ; bin: 44 39 d6 0f 95 c2 ; asm: cmpl %esi, %ecx ; asm: setl %bl - [-,%rbx] v304 = icmp slt v1, v2 ; bin: 40 39 f1 0f 9c c3 + [-,%rbx] v304 = icmp slt v1, v2 ; bin: 39 f1 0f 9c c3 ; asm: cmpl %r10d, %esi ; asm: setl %dl [-,%rdx] v305 = icmp slt v2, v3 ; bin: 44 39 d6 0f 9c c2 ; asm: cmpl %esi, %ecx ; asm: setge %bl - [-,%rbx] v306 = icmp sge v1, v2 ; bin: 40 39 f1 0f 9d c3 + [-,%rbx] v306 = icmp sge v1, v2 ; bin: 39 f1 0f 9d c3 ; asm: cmpl %r10d, %esi ; asm: setge %dl [-,%rdx] v307 = icmp sge v2, v3 ; bin: 44 39 d6 0f 9d c2 ; asm: cmpl %esi, %ecx ; asm: setg %bl - [-,%rbx] v308 = icmp sgt v1, v2 ; bin: 40 39 f1 0f 9f c3 + [-,%rbx] v308 = icmp sgt v1, v2 ; bin: 39 f1 0f 9f c3 ; asm: cmpl %r10d, %esi ; asm: setg %dl [-,%rdx] v309 = icmp sgt v2, v3 ; bin: 44 39 d6 0f 9f c2 ; asm: cmpl %esi, %ecx ; asm: setle %bl - [-,%rbx] v310 = icmp sle v1, v2 ; bin: 40 39 f1 0f 9e c3 + [-,%rbx] v310 = icmp sle v1, v2 ; bin: 39 f1 0f 9e c3 ; asm: cmpl %r10d, %esi ; asm: setle %dl [-,%rdx] v311 = icmp sle v2, v3 ; bin: 44 39 d6 0f 9e c2 ; asm: cmpl %esi, %ecx ; asm: setb %bl - [-,%rbx] v312 = icmp ult v1, v2 ; bin: 40 39 f1 0f 92 c3 + [-,%rbx] v312 = icmp ult v1, v2 ; bin: 39 f1 0f 92 c3 ; asm: cmpl %r10d, %esi ; asm: setb %dl [-,%rdx] v313 = icmp ult v2, v3 ; bin: 44 39 d6 0f 92 c2 ; asm: cmpl %esi, %ecx ; asm: setae %bl - [-,%rbx] v314 = icmp uge v1, v2 ; bin: 40 39 f1 0f 93 c3 + [-,%rbx] v314 = icmp uge v1, v2 ; bin: 39 f1 0f 93 c3 ; asm: cmpl %r10d, %esi ; asm: setae %dl [-,%rdx] v315 = icmp uge v2, v3 ; bin: 44 39 d6 0f 93 c2 ; asm: cmpl %esi, %ecx ; asm: seta %bl - [-,%rbx] v316 = icmp ugt v1, v2 ; bin: 40 39 f1 0f 97 c3 + [-,%rbx] v316 = icmp ugt v1, v2 ; bin: 39 f1 0f 97 c3 ; asm: cmpl %r10d, %esi ; asm: seta %dl [-,%rdx] v317 = icmp ugt v2, v3 ; bin: 44 39 d6 0f 97 c2 ; asm: cmpl %esi, %ecx ; asm: setbe %bl - [-,%rbx] v318 = icmp ule v1, v2 ; bin: 40 39 f1 0f 96 c3 + [-,%rbx] v318 = icmp ule v1, v2 ; bin: 39 f1 0f 96 c3 ; asm: cmpl %r10d, %esi ; asm: setbe %dl [-,%rdx] v319 = icmp ule v2, v3 ; bin: 44 39 d6 0f 96 c2 @@ -802,25 +802,25 @@ ebb0: ; Bool-to-int conversions. ; asm: movzbl %bl, %ecx - [-,%rcx] v350 = bint.i32 v300 ; bin: 40 0f b6 cb + [-,%rcx] v350 = bint.i32 v300 ; bin: 0f b6 cb ; asm: movzbl %dl, %esi - [-,%rsi] v351 = bint.i32 v301 ; bin: 40 0f b6 f2 + [-,%rsi] v351 = bint.i32 v301 ; bin: 0f b6 f2 ; asm: testl %ecx, %ecx ; asm: je ebb1x - brz v1, ebb1 ; bin: 40 85 c9 74 1b + brz v1, ebb1 ; bin: 85 c9 74 18 ; asm: testl %esi, %esi ; asm: je ebb1x - brz v2, ebb1 ; bin: 40 85 f6 74 16 + brz v2, ebb1 ; bin: 85 f6 74 14 ; asm: testl %r10d, %r10d ; asm: je ebb1x - brz v3, ebb1 ; bin: 45 85 d2 74 11 + brz v3, ebb1 ; bin: 45 85 d2 74 0f ; asm: testl %ecx, %ecx ; asm: jne ebb1x - brnz v1, ebb1 ; bin: 40 85 c9 75 0c + brnz v1, ebb1 ; bin: 85 c9 75 0b ; asm: testl %esi, %esi ; asm: jne ebb1x - brnz v2, ebb1 ; bin: 40 85 f6 75 07 + brnz v2, ebb1 ; bin: 85 f6 75 07 ; asm: testl %r10d, %r10d ; asm: jne ebb1x brnz v3, ebb1 ; bin: 45 85 d2 75 02 @@ -856,7 +856,7 @@ ebb0: [-,%rcx] v22 = sextend.i64 v13 ; bin: 49 63 ca ; asm: movl %ecx, %esi - [-,%rsi] v30 = uextend.i64 v11 ; bin: 40 89 ce + [-,%rsi] v30 = uextend.i64 v11 ; bin: 89 ce ; asm: movl %esi, %r10d [-,%r10] v31 = uextend.i64 v12 ; bin: 41 89 f2 ; asm: movl %r10d, %ecx diff --git a/lib/cretonne/meta/check.sh b/lib/cretonne/meta/check.sh index 655092e6cb..c7fd511ec1 100755 --- a/lib/cretonne/meta/check.sh +++ b/lib/cretonne/meta/check.sh @@ -15,7 +15,7 @@ runif() { runif flake8 . # Type checking. -runif mypy --py2 build.py +runif python3 -m mypy --py2 build.py # Python unit tests. runif python -m unittest discover diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 355c755e7c..117940eb39 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -91,6 +91,12 @@ def enc_flt(inst, recipe, *args, **kwargs): I64.enc(inst, *recipe(*args, **kwargs)) +def enc_i64(inst, recipe, *args, **kwargs): + # type: (MaybeBoundInst, r.TailRecipe, *int, **int) -> None + I64.enc(inst, *recipe.rex(*args, **kwargs)) + I64.enc(inst, *recipe(*args, **kwargs)) + + for inst, opc in [ (base.iadd, 0x01), (base.isub, 0x29), @@ -175,9 +181,9 @@ enc_i32_i64_ld_st(base.store, True, r.st, 0x89) enc_i32_i64_ld_st(base.store, True, r.stDisp8, 0x89) enc_i32_i64_ld_st(base.store, True, r.stDisp32, 0x89) -I64.enc(base.istore32.i64.any, *r.st.rex(0x89)) -I64.enc(base.istore32.i64.any, *r.stDisp8.rex(0x89)) -I64.enc(base.istore32.i64.any, *r.stDisp32.rex(0x89)) +enc_i64(base.istore32.i64.any, r.st, 0x89) +enc_i64(base.istore32.i64.any, r.stDisp8, 0x89) +enc_i64(base.istore32.i64.any, r.stDisp32, 0x89) enc_i32_i64_ld_st(base.istore16, False, r.st, 0x66, 0x89) enc_i32_i64_ld_st(base.istore16, False, r.stDisp8, 0x66, 0x89) @@ -187,21 +193,24 @@ enc_i32_i64_ld_st(base.istore16, False, r.stDisp32, 0x66, 0x89) # depends of the presence of a REX prefix I32.enc(base.istore8.i32.any, *r.st_abcd(0x88)) I64.enc(base.istore8.i32.any, *r.st_abcd(0x88)) +I64.enc(base.istore8.i64.any, *r.st_abcd(0x88)) I64.enc(base.istore8.i64.any, *r.st.rex(0x88)) I32.enc(base.istore8.i32.any, *r.stDisp8_abcd(0x88)) I64.enc(base.istore8.i32.any, *r.stDisp8_abcd(0x88)) +I64.enc(base.istore8.i64.any, *r.stDisp8_abcd(0x88)) I64.enc(base.istore8.i64.any, *r.stDisp8.rex(0x88)) I32.enc(base.istore8.i32.any, *r.stDisp32_abcd(0x88)) I64.enc(base.istore8.i32.any, *r.stDisp32_abcd(0x88)) +I64.enc(base.istore8.i64.any, *r.stDisp32_abcd(0x88)) I64.enc(base.istore8.i64.any, *r.stDisp32.rex(0x88)) enc_i32_i64_ld_st(base.load, True, r.ld, 0x8b) enc_i32_i64_ld_st(base.load, True, r.ldDisp8, 0x8b) enc_i32_i64_ld_st(base.load, True, r.ldDisp32, 0x8b) -I64.enc(base.uload32.i64, *r.ld.rex(0x8b)) -I64.enc(base.uload32.i64, *r.ldDisp8.rex(0x8b)) -I64.enc(base.uload32.i64, *r.ldDisp32.rex(0x8b)) +enc_i64(base.uload32.i64, r.ld, 0x8b) +enc_i64(base.uload32.i64, r.ldDisp8, 0x8b) +enc_i64(base.uload32.i64, r.ldDisp32, 0x8b) I64.enc(base.sload32.i64, *r.ld.rex(0x63, w=1)) I64.enc(base.sload32.i64, *r.ldDisp8.rex(0x63, w=1)) @@ -301,7 +310,7 @@ enc_i32_i64(base.icmp, r.icscc, 0x39) # This assumes that b1 is represented as an 8-bit low register with the value 0 # or 1. I32.enc(base.bint.i32.b1, *r.urm_abcd(0x0f, 0xb6)) -I64.enc(base.bint.i64.b1, *r.urm.rex(0x0f, 0xb6, w=1)) +I64.enc(base.bint.i64.b1, *r.urm.rex(0x0f, 0xb6)) # zext to i64 implicit. I64.enc(base.bint.i64.b1, *r.urm_abcd(0x0f, 0xb6)) # zext to i64 implicit. I64.enc(base.bint.i32.b1, *r.urm.rex(0x0f, 0xb6)) I64.enc(base.bint.i32.b1, *r.urm_abcd(0x0f, 0xb6)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index c271ab1e34..89556a4049 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -8,10 +8,10 @@ from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare from base.formats import RegMove, Ternary, Jump, Branch, FuncAddr -from .registers import GPR, ABCD, FPR +from .registers import GPR, ABCD, FPR, GPR8, FPR8 try: - from typing import Tuple, Dict # noqa + from typing import Tuple, Dict, Sequence # noqa from cdsl.instructions import InstructionFormat # noqa from cdsl.isa import ConstraintSeq, BranchRange, PredNode, OperandConstraint # noqa except ImportError: @@ -95,6 +95,15 @@ def replace_put_op(emit, prefix): return emit.replace('PUT_OP', 'put_' + prefix.lower()) +def map_regs( + regs, # type: Sequence[OperandConstraint] + from_class, # type: OperandConstraint + to_class # type: OperandConstraint +): + # type: (...) -> Sequence[OperandConstraint] + return tuple(to_class if (reg is from_class) else reg for reg in regs) + + class TailRecipe: """ Generate encoding recipes on demand. @@ -150,7 +159,7 @@ class TailRecipe: w = kwargs.get('w', 0) name, bits = decode_ops(ops, rrr, w) if name not in self.recipes: - self.recipes[name] = EncRecipe( + recipe = EncRecipe( name + self.name, self.format, len(ops) + self.size, @@ -160,6 +169,13 @@ class TailRecipe: instp=self.instp, isap=self.isap, emit=replace_put_op(self.emit, name)) + + recipe.ins = map_regs(recipe.ins, GPR, GPR8) + recipe.ins = map_regs(recipe.ins, FPR, FPR8) + recipe.outs = map_regs(recipe.outs, GPR, GPR8) + recipe.outs = map_regs(recipe.outs, FPR, FPR8) + + self.recipes[name] = recipe return (self.recipes[name], bits) def rex(self, *ops, **kwargs): diff --git a/lib/cretonne/meta/isa/intel/registers.py b/lib/cretonne/meta/isa/intel/registers.py index 92e76191fa..62966aac3b 100644 --- a/lib/cretonne/meta/isa/intel/registers.py +++ b/lib/cretonne/meta/isa/intel/registers.py @@ -39,7 +39,9 @@ FloatRegs = RegBank( units=16, prefix='xmm') GPR = RegClass(IntRegs) +GPR8 = GPR[0:8] ABCD = GPR[0:4] FPR = RegClass(FloatRegs) +FPR8 = FPR[0:8] RegClass.extract_names(globals()) From 7fbf1cb810bb1174cf0fbc5c02442853fcf4e147 Mon Sep 17 00:00:00 2001 From: Angus Holder Date: Fri, 22 Sep 2017 15:44:15 +0100 Subject: [PATCH 1156/3084] Revert accidental change --- lib/cretonne/meta/check.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/meta/check.sh b/lib/cretonne/meta/check.sh index c7fd511ec1..655092e6cb 100755 --- a/lib/cretonne/meta/check.sh +++ b/lib/cretonne/meta/check.sh @@ -15,7 +15,7 @@ runif() { runif flake8 . # Type checking. -runif python3 -m mypy --py2 build.py +runif mypy --py2 build.py # Python unit tests. runif python -m unittest discover From 2946cc54d34435ce8ddf23659cd228bc614ed293 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Sep 2017 08:51:55 -0700 Subject: [PATCH 1157/3084] Add more trap codes. These are codes that come up naturally when translating WebAssembly and legalizing the Cretonne instruction set. --- lib/cretonne/src/ir/trapcode.rs | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/ir/trapcode.rs b/lib/cretonne/src/ir/trapcode.rs index c9312cdff0..2374c206f3 100644 --- a/lib/cretonne/src/ir/trapcode.rs +++ b/lib/cretonne/src/ir/trapcode.rs @@ -20,12 +20,24 @@ pub enum TrapCode { /// pages. HeapOutOfBounds, + /// Other bounds checking error. + OutOfBounds, + + /// Indirect call to a null table entry. + IndirectCallToNull, + + /// Signature mismatch on indirect call. + BadSignature, + /// An integer arithmetic operation caused an overflow. IntegerOverflow, /// An integer division by zero. IntegerDivisionByZero, + /// Failed float-to-int conversion. + BadConversionToInteger, + /// A user-defined trap code. User(u16), } @@ -36,8 +48,12 @@ impl Display for TrapCode { let identifier = match *self { StackOverflow => "stk_ovf", HeapOutOfBounds => "heap_oob", + OutOfBounds => "oob", + IndirectCallToNull => "icall_null", + BadSignature => "bad_sig", IntegerOverflow => "int_ovf", IntegerDivisionByZero => "int_divz", + BadConversionToInteger => "bad_toint", User(x) => return write!(f, "user{}", x), }; f.write_str(identifier) @@ -52,8 +68,12 @@ impl FromStr for TrapCode { match s { "stk_ovf" => Ok(StackOverflow), "heap_oob" => Ok(HeapOutOfBounds), + "oob" => Ok(OutOfBounds), + "icall_null" => Ok(IndirectCallToNull), + "bad_sig" => Ok(BadSignature), "int_ovf" => Ok(IntegerOverflow), "int_divz" => Ok(IntegerDivisionByZero), + "bad_toint" => Ok(BadConversionToInteger), _ if s.starts_with("user") => s[4..].parse().map(User).map_err(|_| ()), _ => Err(()), } @@ -65,11 +85,15 @@ mod tests { use super::*; // Everything but user-defined codes. - const CODES: [TrapCode; 4] = [ + const CODES: [TrapCode; 8] = [ TrapCode::StackOverflow, TrapCode::HeapOutOfBounds, + TrapCode::OutOfBounds, + TrapCode::IndirectCallToNull, + TrapCode::BadSignature, TrapCode::IntegerOverflow, TrapCode::IntegerDivisionByZero, + TrapCode::BadConversionToInteger, ]; #[test] From 76eb7df9f020ab933370a7387393406134d62447 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Sep 2017 11:16:07 -0700 Subject: [PATCH 1158/3084] Add an isa::StackRef type. This contains encoding details for a stack reference: The base register and offset to use in the specific instruction encoding. Generate StackRef objects called in_stk0 etc for the binemit recipe code. All binemit recipes need to compute base pointer offsets for stack references, so have the automatically generated code do it. --- cranelift/src/filetest/binemit.rs | 7 +++ lib/cretonne/meta/cdsl/registers.py | 10 ++++ lib/cretonne/meta/gen_binemit.py | 32 +++++++---- lib/cretonne/src/isa/mod.rs | 2 + lib/cretonne/src/isa/riscv/binemit.rs | 2 +- lib/cretonne/src/isa/stack.rs | 82 +++++++++++++++++++++++++++ 6 files changed, 123 insertions(+), 12 deletions(-) create mode 100644 lib/cretonne/src/isa/stack.rs diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 0ea634548a..f1a493c749 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -104,6 +104,13 @@ impl SubTest for TestBinEmit { // value locations. The current error reporting is just crashing... let mut func = func.into_owned(); + // Fix the stack frame layout so we can test spill/fill encodings. + let min_offset = func.stack_slots + .keys() + .map(|ss| func.stack_slots[ss].offset) + .min(); + func.stack_slots.frame_size = min_offset.map(|off| (-off) as u32); + let is_compressed = isa.flags().is_compressed(); // Give an encoding to any instruction that doesn't already have one. diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index 880f9096d4..89c6db0a28 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -352,3 +352,13 @@ class Stack(object): def __init__(self, rc): # type: (RegClass) -> None self.regclass = rc + + def stack_base_mask(self): + # type: () -> str + """ + Get the StackBaseMask to use for this operand. + + This is a mask of base registers that can be supported by this operand. + """ + # TODO: Make this configurable instead of just using the SP. + return 'StackBaseMask(1)' diff --git a/lib/cretonne/meta/gen_binemit.py b/lib/cretonne/meta/gen_binemit.py index fddadf64e5..39cab5ab20 100644 --- a/lib/cretonne/meta/gen_binemit.py +++ b/lib/cretonne/meta/gen_binemit.py @@ -66,11 +66,16 @@ def gen_recipe(recipe, fmt): 'let {} = divert.reg(args[{}], &func.locations);' .format(v, i)) elif isinstance(arg, Stack): - v = 'in_ss{}'.format(i) + v = 'in_stk{}'.format(i) args += ', ' + v - fmt.line( - 'let {} = func.locations[args[{}]].unwrap_stack();' - .format(v, i)) + with fmt.indented( + 'let {} = StackRef::masked('.format(v), + ').unwrap();'): + fmt.format( + 'func.locations[args[{}]].unwrap_stack(),', + i) + fmt.format('{},', arg.stack_base_mask()) + fmt.line('&func.stack_slots,') # Pass arguments in this order: inputs, imm_fields, outputs. for f in iform.imm_fields: @@ -86,15 +91,20 @@ def gen_recipe(recipe, fmt): if isinstance(res, RegClass): v = 'out_reg{}'.format(i) args += ', ' + v - fmt.line( - 'let {} = func.locations[results[{}]].unwrap_reg();' - .format(v, i)) + fmt.format( + 'let {} = func.locations[results[{}]].unwrap_reg();', + v, i) elif isinstance(res, Stack): - v = 'out_ss{}'.format(i) + v = 'out_stk{}'.format(i) args += ', ' + v - fmt.line( - 'let {} = func.locations[results[{}]].unwrap_stack();' - .format(v, i)) + with fmt.indented( + 'let {} = StackRef::masked('.format(v), + ').unwrap();'): + fmt.format( + 'func.locations[results[{}]].unwrap_stack(),', + i) + fmt.format('{},', res.stack_base_mask()) + fmt.line('&func.stack_slots,') # Special handling for regmove instructions. Update the register # diversion tracker. diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 1ccacce8b1..f6c95edec8 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -43,6 +43,7 @@ pub use isa::constraints::{RecipeConstraints, OperandConstraint, ConstraintKind, BranchRange}; pub use isa::encoding::{Encoding, EncInfo}; pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex, regs_overlap}; +pub use isa::stack::{StackBase, StackBaseMask, StackRef}; use binemit; use flowgraph; @@ -68,6 +69,7 @@ pub mod registers; mod encoding; mod enc_tables; mod constraints; +mod stack; /// Returns a builder that can create a corresponding `TargetIsa` /// or `Err(LookupError::Unsupported)` if not enabled. diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 7a658267b9..61a497000f 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -2,7 +2,7 @@ use binemit::{CodeSink, Reloc, bad_encoding}; use ir::{Function, Inst, InstructionData}; -use isa::RegUnit; +use isa::{RegUnit, StackRef, StackBaseMask}; use predicates::is_signed_int; use regalloc::RegDiversions; diff --git a/lib/cretonne/src/isa/stack.rs b/lib/cretonne/src/isa/stack.rs new file mode 100644 index 0000000000..cfb054dd66 --- /dev/null +++ b/lib/cretonne/src/isa/stack.rs @@ -0,0 +1,82 @@ +//! Low-level details of stack accesses. +//! +//! The `ir::StackSlots` type deals with stack slots and stack frame layout. The `StackRef` type +//! defined in this module expresses the low-level details of accessing a stack slot from an +//! encoded instruction. + +use ir::stackslot::{StackSlots, StackOffset}; +use ir::StackSlot; + +/// A method for referencing a stack slot in the current stack frame. +/// +/// Stack slots are addressed with a constant offset from a base register. The base can be the +/// stack pointer, the frame pointer, or (in the future) a zone register pointing to an inner zone +/// of a large stack frame. +#[derive(Clone, Copy, Debug)] +pub struct StackRef { + /// The base register to use for addressing. + pub base: StackBase, + + /// Immediate offset from the base register to the first byte of the stack slot. + pub offset: StackOffset, +} + +impl StackRef { + /// Get a reference to the stack slot `ss` using one of the base pointers in `mask`. + pub fn masked(ss: StackSlot, mask: StackBaseMask, frame: &StackSlots) -> Option { + let size = frame.frame_size.expect( + "Stack layout must be computed before referencing stack slots", + ); + // Offsets relative to the caller's stack pointer. + let offset = frame[ss].offset; + + // Try an SP-relative reference. + if mask.contains(StackBase::SP) { + // Offset where SP is pointing. (All ISAs have stacks growing downwards.) + let sp_offset = -(size as StackOffset); + return Some(StackRef { + base: StackBase::SP, + offset: offset - sp_offset, + }); + } + + // No reference possible with this mask. + None + } +} + +/// Generic base register for referencing stack slots. +/// +/// Most ISAs have a stack pointer and an optional frame pointer, so provide generic names for +/// those two base pointers. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum StackBase { + /// Use the stack pointer. + SP = 0, + + /// Use the frame pointer (if one is present). + FP = 1, + + /// Use an explicit zone pointer in a general-purpose register. + Zone = 2, +} + +/// Bit mask of supported stack bases. +/// +/// Many instruction encodings can use different base registers while others only work with the +/// stack pointer, say. A `StackBaseMask` is a bit mask of supported stack bases for a given +/// instruction encoding. +/// +/// This behaves like a set of `StackBase` variants. +/// +/// The internal representation as a `u8` is public because stack base masks are used in constant +/// tables generated from the Python encoding definitions. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct StackBaseMask(pub u8); + +impl StackBaseMask { + /// Check if this mask contains the `base` variant. + pub fn contains(self, base: StackBase) -> bool { + self.0 & (1 << base as usize) != 0 + } +} From 29dfcf5dfbf468e3ef3415e840b2666388c90324 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 22 Sep 2017 15:35:11 -0700 Subject: [PATCH 1159/3084] Add spill/fill encodings for Intel ISAs. To begin with, these are catch-all encodings with a SIB byte and a 32-bit displacement, so they can access any stack slot via both the stack pointer and the frame pointer. In the future, we will add encodings for 8-bit displacements as well as EBP-relative references without a SIB byte. --- .../filetests/isa/intel/binary32-float.cton | 34 ++++++++++++++ cranelift/filetests/isa/intel/binary32.cton | 17 +++++++ .../filetests/isa/intel/binary64-float.cton | 37 +++++++++++++++- cranelift/filetests/isa/intel/binary64.cton | 44 +++++++++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 12 ++++- lib/cretonne/meta/isa/intel/recipes.py | 42 +++++++++++++++++- lib/cretonne/meta/isa/intel/registers.py | 8 +++- lib/cretonne/src/isa/intel/binemit.rs | 29 +++++++++++- 8 files changed, 217 insertions(+), 6 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 3756fbff1e..c5a1b08a0e 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -8,6 +8,11 @@ isa intel has_sse2 ; function %F32() { + ss0 = incoming_arg 8, offset 0 + ss1 = incoming_arg 1024, offset -1024 + ss2 = incoming_arg 1024, offset -2048 + ss3 = incoming_arg 8, offset -2056 + ebb0: [-,%rcx] v0 = iconst.i32 1 [-,%rsi] v1 = iconst.i32 2 @@ -105,10 +110,27 @@ ebb0: ; asm: movd %xmm2, -10000(%esi) [-] store.f32 v101, v1-10000 ; bin: 66 0f 7e 96 ffffd8f0 + ; Spill / Fill. + + ; asm: movd %xmm5, 1032(%esp) + [-,ss1] v200 = spill v100 ; bin: 66 0f 7e ac 24 00000408 + ; asm: movd %xmm2, 1032(%esp) + [-,ss1] v201 = spill v101 ; bin: 66 0f 7e 94 24 00000408 + + ; asm: movd 1032(%esp), %xmm5 + [-,%xmm5] v210 = fill v200 ; bin: 66 0f 6e ac 24 00000408 + ; asm: movd 1032(%esp), %xmm2 + [-,%xmm2] v211 = fill v201 ; bin: 66 0f 6e 94 24 00000408 + return } function %F64() { + ss0 = incoming_arg 8, offset 0 + ss1 = incoming_arg 1024, offset -1024 + ss2 = incoming_arg 1024, offset -2048 + ss3 = incoming_arg 8, offset -2056 + ebb0: [-,%rcx] v0 = iconst.i32 1 [-,%rsi] v1 = iconst.i32 2 @@ -198,5 +220,17 @@ ebb0: ; asm: movq %xmm2, -10000(%esi) [-] store.f64 v101, v1-10000 ; bin: 66 0f d6 96 ffffd8f0 + ; Spill / Fill. + + ; asm: movq %xmm5, 1032(%esp) + [-,ss1] v200 = spill v100 ; bin: 66 0f d6 ac 24 00000408 + ; asm: movq %xmm2, 1032(%esp) + [-,ss1] v201 = spill v101 ; bin: 66 0f d6 94 24 00000408 + + ; asm: movq 1032(%esp), %xmm5 + [-,%xmm5] v210 = fill v200 ; bin: f3 0f 7e ac 24 00000408 + ; asm: movq 1032(%esp), %xmm2 + [-,%xmm2] v211 = fill v201 ; bin: f3 0f 7e 94 24 00000408 + return } diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 16f8131cef..ee8c79e24f 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -11,6 +11,11 @@ function %I32() { fn0 = function %foo() sig0 = () + ss0 = incoming_arg 8, offset 0 + ss1 = incoming_arg 1024, offset -1024 + ss2 = incoming_arg 1024, offset -2048 + ss3 = incoming_arg 8, offset -2056 + ebb0: ; asm: movl $1, %ecx [-,%rcx] v1 = iconst.i32 1 ; bin: b9 00000001 @@ -346,6 +351,18 @@ ebb0: ; asm: call *%esi call_indirect sig0, v401() ; bin: ff d6 + ; Spill / Fill. + + ; asm: movl %ecx, 1032(%esp) + [-,ss1] v500 = spill v1 ; bin: 89 8c 24 00000408 + ; asm: movl %esi, 1032(%esp) + [-,ss1] v501 = spill v2 ; bin: 89 b4 24 00000408 + + ; asm: movl 1032(%esp), %ecx + [-,%rcx] v510 = fill v500 ; bin: 8b 8c 24 00000408 + ; asm: movl 1032(%esp), %esi + [-,%rsi] v511 = fill v501 ; bin: 8b b4 24 00000408 + ; asm: testl %ecx, %ecx ; asm: je ebb1 brz v1, ebb1 ; bin: 85 c9 74 0e diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index 83aaf6d753..a5b4aaa163 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -1,6 +1,7 @@ ; Binary emission of 64-bit floating point code. test binemit set is_64bit +set is_compressed isa intel has_sse2 ; The binary encodings can be verified with the command: @@ -9,6 +10,11 @@ isa intel has_sse2 ; function %F32() { + ss0 = incoming_arg 8, offset 0 + ss1 = incoming_arg 1024, offset -1024 + ss2 = incoming_arg 1024, offset -2048 + ss3 = incoming_arg 8, offset -2056 + ebb0: [-,%r11] v0 = iconst.i32 1 [-,%rsi] v1 = iconst.i32 2 @@ -36,7 +42,7 @@ ebb0: [-,%xmm10] v17 = bitcast.f32 v1 ; bin: 66 44 0f 6e d6 ; asm: movd %xmm5, %ecx - [-,%rcx] v18 = bitcast.i32 v10 ; bin: 66 40 0f 7e e9 + [-,%rcx] v18 = bitcast.i32 v10 ; bin: 66 0f 7e e9 ; asm: movd %xmm10, %esi [-,%rsi] v19 = bitcast.i32 v11 ; bin: 66 44 0f 7e d6 @@ -113,10 +119,27 @@ ebb0: ; asm: movd %xmm10, -10000(%rax) [-] store.f32 v101, v2-10000 ; bin: 66 44 0f 7e 90 ffffd8f0 + ; Spill / Fill. + + ; asm: movd %xmm5, 1032(%rsp) + [-,ss1] v200 = spill v100 ; bin: 66 0f 7e ac 24 00000408 + ; asm: movd %xmm10, 1032(%rsp) + [-,ss1] v201 = spill v101 ; bin: 66 44 0f 7e 94 24 00000408 + + ; asm: movd 1032(%rsp), %xmm5 + [-,%xmm5] v210 = fill v200 ; bin: 66 0f 6e ac 24 00000408 + ; asm: movd 1032(%rsp), %xmm10 + [-,%xmm10] v211 = fill v201 ; bin: 66 44 0f 6e 94 24 00000408 + return } function %F64() { + ss0 = incoming_arg 8, offset 0 + ss1 = incoming_arg 1024, offset -1024 + ss2 = incoming_arg 1024, offset -2048 + ss3 = incoming_arg 8, offset -2056 + ebb0: [-,%r11] v0 = iconst.i32 1 [-,%rsi] v1 = iconst.i32 2 @@ -221,5 +244,17 @@ ebb0: ; asm: movq %xmm10, -10000(%rax) [-] store.f64 v101, v2-10000 ; bin: 66 44 0f d6 90 ffffd8f0 + ; Spill / Fill. + + ; asm: movq %xmm5, 1032(%rsp) + [-,ss1] v200 = spill v100 ; bin: 66 0f d6 ac 24 00000408 + ; asm: movq %xmm10, 1032(%rsp) + [-,ss1] v201 = spill v101 ; bin: 66 44 0f d6 94 24 00000408 + + ; asm: movq 1032(%rsp), %xmm5 + [-,%xmm5] v210 = fill v200 ; bin: f3 0f 7e ac 24 00000408 + ; asm: movq 1032(%rsp), %xmm10 + [-,%xmm10] v211 = fill v201 ; bin: f3 44 0f 7e 94 24 00000408 + return } diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 8b22173fe4..c9febfdab0 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -14,6 +14,13 @@ function %I64() { fn0 = function %foo() sig0 = () + ; Use incoming_arg stack slots because they won't be relocated by the frame + ; layout. + ss0 = incoming_arg 8, offset 0 + ss1 = incoming_arg 1024, offset -1024 + ss2 = incoming_arg 1024, offset -2048 + ss3 = incoming_arg 8, offset -2056 + ebb0: ; Integer Constants. @@ -436,6 +443,22 @@ ebb0: ; asm: call *%r10 call_indirect sig0, v402() ; bin: 41 ff d2 + ; Spill / Fill. + + ; asm: movq %rcx, 1032(%rsp) + [-,ss1] v500 = spill v1 ; bin: 48 89 8c 24 00000408 + ; asm: movq %rsi, 1032(%rsp) + [-,ss1] v501 = spill v2 ; bin: 48 89 b4 24 00000408 + ; asm: movq %r10, 1032(%rsp) + [-,ss1] v502 = spill v3 ; bin: 4c 89 94 24 00000408 + + ; asm: movq 1032(%rsp), %rcx + [-,%rcx] v510 = fill v500 ; bin: 48 8b 8c 24 00000408 + ; asm: movq 1032(%rsp), %rsi + [-,%rsi] v511 = fill v501 ; bin: 48 8b b4 24 00000408 + ; asm: movq 1032(%rsp), %r10 + [-,%r10] v512 = fill v502 ; bin: 4c 8b 94 24 00000408 + ; asm: testq %rcx, %rcx ; asm: je ebb1 brz v1, ebb1 ; bin: 48 85 c9 74 1b @@ -477,6 +500,11 @@ function %I32() { fn0 = function %foo() sig0 = () + ss0 = incoming_arg 8, offset 0 + ss1 = incoming_arg 1024, offset -1024 + ss2 = incoming_arg 1024, offset -2048 + ss3 = incoming_arg 8, offset -2056 + ebb0: ; Integer Constants. @@ -806,6 +834,22 @@ ebb0: ; asm: movzbl %dl, %esi [-,%rsi] v351 = bint.i32 v301 ; bin: 0f b6 f2 + ; Spill / Fill. + + ; asm: movl %ecx, 1032(%rsp) + [-,ss1] v500 = spill v1 ; bin: 89 8c 24 00000408 + ; asm: movl %esi, 1032(%rsp) + [-,ss1] v501 = spill v2 ; bin: 89 b4 24 00000408 + ; asm: movl %r10d, 1032(%rsp) + [-,ss1] v502 = spill v3 ; bin: 44 89 94 24 00000408 + + ; asm: movl 1032(%rsp), %ecx + [-,%rcx] v510 = fill v500 ; bin: 8b 8c 24 00000408 + ; asm: movl 1032(%rsp), %esi + [-,%rsi] v511 = fill v501 ; bin: 8b b4 24 00000408 + ; asm: movl 1032(%rsp), %r10d + [-,%r10] v512 = fill v502 ; bin: 44 8b 94 24 00000408 + ; asm: testl %ecx, %ecx ; asm: je ebb1x brz v1, ebb1 ; bin: 85 c9 74 18 diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 117940eb39..4b646658c1 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -65,7 +65,7 @@ def enc_i32_i64_ld_st(inst, w_bit, recipe, *args, **kwargs): Add encodings for `inst.i32` to I32. Add encodings for `inst.i32` to I64 with and without REX. Add encodings for `inst.i64` to I64 with a REX prefix, using the `w_bit` - argument to determine wheter or not to set the REX.W bit. + argument to determine whether or not to set the REX.W bit. """ I32.enc(inst.i32.any, *recipe(*args, **kwargs)) @@ -181,6 +181,8 @@ enc_i32_i64_ld_st(base.store, True, r.st, 0x89) enc_i32_i64_ld_st(base.store, True, r.stDisp8, 0x89) enc_i32_i64_ld_st(base.store, True, r.stDisp32, 0x89) +enc_i32_i64(base.spill, r.spSib32, 0x89) + enc_i64(base.istore32.i64.any, r.st, 0x89) enc_i64(base.istore32.i64.any, r.stDisp8, 0x89) enc_i64(base.istore32.i64.any, r.stDisp32, 0x89) @@ -208,6 +210,8 @@ enc_i32_i64_ld_st(base.load, True, r.ld, 0x8b) enc_i32_i64_ld_st(base.load, True, r.ldDisp8, 0x8b) enc_i32_i64_ld_st(base.load, True, r.ldDisp32, 0x8b) +enc_i32_i64(base.fill, r.fiSib32, 0x8b) + enc_i64(base.uload32.i64, r.ld, 0x8b) enc_i64(base.uload32.i64, r.ldDisp8, 0x8b) enc_i64(base.uload32.i64, r.ldDisp32, 0x8b) @@ -252,6 +256,12 @@ enc_flt(base.store.f64.any, r.fst, 0x66, 0x0f, 0xd6) enc_flt(base.store.f64.any, r.fstDisp8, 0x66, 0x0f, 0xd6) enc_flt(base.store.f64.any, r.fstDisp32, 0x66, 0x0f, 0xd6) +enc_flt(base.fill.f32, r.ffiSib32, 0x66, 0x0f, 0x6e) +enc_flt(base.fill.f64, r.ffiSib32, 0xf3, 0x0f, 0x7e) + +enc_flt(base.spill.f32, r.fspSib32, 0x66, 0x0f, 0x7e) +enc_flt(base.spill.f64, r.fspSib32, 0x66, 0x0f, 0xd6) + # # Function addresses. # diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 89556a4049..358f9cae72 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -8,7 +8,7 @@ from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare from base.formats import RegMove, Ternary, Jump, Branch, FuncAddr -from .registers import GPR, ABCD, FPR, GPR8, FPR8 +from .registers import GPR, ABCD, FPR, GPR8, FPR8, StackGPR32, StackFPR32 try: from typing import Tuple, Dict, Sequence # noqa @@ -474,6 +474,26 @@ fstDisp32 = TailRecipe( sink.put4(offset as u32); ''') +# Unary spill with SIB and 32-bit displacement. +spSib32 = TailRecipe( + 'spSib32', Unary, size=6, ins=GPR, outs=StackGPR32, + emit=''' + let base = stk_base(out_stk0.base); + PUT_OP(bits, rex2(base, in_reg0), sink); + modrm_sib_disp32(in_reg0, sink); + sib_noindex(base, sink); + sink.put4(out_stk0.offset as u32); + ''') +fspSib32 = TailRecipe( + 'fspSib32', Unary, size=6, ins=FPR, outs=StackFPR32, + emit=''' + let base = stk_base(out_stk0.base); + PUT_OP(bits, rex2(base, in_reg0), sink); + modrm_sib_disp32(in_reg0, sink); + sib_noindex(base, sink); + sink.put4(out_stk0.offset as u32); + ''') + # # Load recipes # @@ -540,6 +560,26 @@ fldDisp32 = TailRecipe( sink.put4(offset as u32); ''') +# Unary fill with SIB and 32-bit displacement. +fiSib32 = TailRecipe( + 'fiSib32', Unary, size=6, ins=StackGPR32, outs=GPR, + emit=''' + let base = stk_base(in_stk0.base); + PUT_OP(bits, rex2(base, out_reg0), sink); + modrm_sib_disp32(out_reg0, sink); + sib_noindex(base, sink); + sink.put4(in_stk0.offset as u32); + ''') +ffiSib32 = TailRecipe( + 'ffiSib32', Unary, size=6, ins=StackFPR32, outs=FPR, + emit=''' + let base = stk_base(in_stk0.base); + PUT_OP(bits, rex2(base, out_reg0), sink); + modrm_sib_disp32(out_reg0, sink); + sib_noindex(base, sink); + sink.put4(in_stk0.offset as u32); + ''') + # # Call/return # diff --git a/lib/cretonne/meta/isa/intel/registers.py b/lib/cretonne/meta/isa/intel/registers.py index 62966aac3b..886812d6ce 100644 --- a/lib/cretonne/meta/isa/intel/registers.py +++ b/lib/cretonne/meta/isa/intel/registers.py @@ -23,7 +23,7 @@ data types, and the H-registers even less so. Rather than trying to model the H-registers accurately, we'll avoid using them in both I32 and I64 modes. """ from __future__ import absolute_import -from cdsl.registers import RegBank, RegClass +from cdsl.registers import RegBank, RegClass, Stack from .defs import ISA @@ -44,4 +44,10 @@ ABCD = GPR[0:4] FPR = RegClass(FloatRegs) FPR8 = FPR[0:8] +# Constraints for stack operands. + +# Stack operand with a 32-bit signed displacement from either RBP or RSP. +StackGPR32 = Stack(GPR) +StackFPR32 = Stack(FPR) + RegClass.extract_names(globals()) diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 6f75833be6..888128caed 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -2,8 +2,9 @@ use binemit::{CodeSink, Reloc, bad_encoding}; use ir::{Function, Inst, Ebb, InstructionData}; -use isa::RegUnit; +use isa::{RegUnit, StackRef, StackBase, StackBaseMask}; use regalloc::RegDiversions; +use super::registers::RU; include!(concat!(env!("OUT_DIR"), "/binemit-intel.rs")); @@ -27,6 +28,16 @@ impl Into for RelocKind { } } +// Convert a stack base to the corresponding register. +fn stk_base(base: StackBase) -> RegUnit { + let ru = match base { + StackBase::SP => RU::rsp, + StackBase::FP => RU::rbp, + StackBase::Zone => unimplemented!(), + }; + ru as RegUnit +} + // Mandatory prefix bytes for Mp* opcodes. const PREFIX: [u8; 3] = [0x66, 0xf3, 0xf2]; @@ -43,7 +54,7 @@ fn rex1(reg_b: RegUnit) -> u8 { // Create a dual-register REX prefix, setting: // -// REX.B = bit 3 of r/m register. +// REX.B = bit 3 of r/m register, or SIB base register when a SIB byte is present. // REX.R = bit 3 of reg register. fn rex2(rm: RegUnit, reg: RegUnit) -> u8 { let b = ((rm >> 3) & 1) as u8; @@ -185,6 +196,20 @@ fn modrm_disp32(rm: RegUnit, reg: RegUnit, sink: &mut CS) sink.put1(b); } +/// Emit a mode 10 ModR/M byte indicating that a SIB byte is present. +fn modrm_sib_disp32(reg: RegUnit, sink: &mut CS) { + modrm_disp32(0b100, reg, sink); +} + +/// Emit a SIB byte with a base register and no scale+index. +fn sib_noindex(base: RegUnit, sink: &mut CS) { + let base = base as u8 & 7; + // SIB SS_III_BBB. + let mut b = 0b00_100_000; + b |= base; + sink.put1(b); +} + /// Emit a single-byte branch displacement to `destination`. fn disp1(destination: Ebb, func: &Function, sink: &mut CS) { let delta = func.offsets[destination].wrapping_sub(sink.offset() + 1); From fdb97da21b7877c83cf8f7fbbcfe3fc899d3dfc3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 25 Sep 2017 10:56:14 -0700 Subject: [PATCH 1160/3084] Implement a poor man's jump table. We will eventually support real jump tables, but for now just expand br_table into a sequence of conditional branches. --- cranelift/filetests/wasm/control.cton | 15 +++++++++++++ lib/cretonne/meta/base/legalize.py | 1 + lib/cretonne/src/ir/jumptable.rs | 5 +++++ lib/cretonne/src/legalizer/mod.rs | 32 +++++++++++++++++++++++++++ 4 files changed, 53 insertions(+) diff --git a/cranelift/filetests/wasm/control.cton b/cranelift/filetests/wasm/control.cton index 9d46a3577b..1d12aeff3d 100644 --- a/cranelift/filetests/wasm/control.cton +++ b/cranelift/filetests/wasm/control.cton @@ -48,3 +48,18 @@ function %undefined() { ebb0: trap user0 } + +function %br_table(i32) { +jt0 = jump_table ebb3, ebb1, 0, ebb2 + +ebb0(v0: i32): + br_table v0, jt0 + trap oob + +ebb1: + return +ebb2: + return +ebb3: + return +} diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index 626295e165..4f92827b24 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -53,6 +53,7 @@ expand.custom_legalize(insts.heap_addr, 'expand_heap_addr') # TODO: Add sufficient XForm syntax that we don't need to hand-code these. expand.custom_legalize(insts.trapz, 'expand_cond_trap') expand.custom_legalize(insts.trapnz, 'expand_cond_trap') +expand.custom_legalize(insts.br_table, 'expand_br_table') # Custom expansions for floating point constants. # These expansions require bit-casting or creating constant pool entries. diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index 319eeb1edf..a6cfad3346 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -39,6 +39,11 @@ impl JumpTableData { } } + /// Get the number of table entries. + pub fn len(&self) -> usize { + self.table.len() + } + /// Set a table entry. /// /// The table will grow as needed to fit `idx`. diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 920bebde81..61a433b6ed 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -147,6 +147,38 @@ fn expand_cond_trap(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFl cfg.recompute_ebb(pos.func, new_ebb); } +/// Jump tables. +fn expand_br_table(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { + use ir::condcodes::IntCC; + + let (arg, table) = match func.dfg[inst] { + ir::InstructionData::BranchTable { + opcode: ir::Opcode::BrTable, + arg, + table, + } => (arg, table), + _ => panic!("Expected br_table: {}", func.dfg.display_inst(inst, None)), + }; + + // This is a poor man's jump table using just a sequence of conditional branches. + // TODO: Lower into a jump table load and indirect branch. + let table_size = func.jump_tables[table].len(); + let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); + + for i in 0..table_size { + if let Some(dest) = pos.func.jump_tables[table].get_entry(i) { + let t = pos.ins().icmp_imm(IntCC::Equal, arg, i as i64); + pos.ins().brnz(t, dest, &[]); + } + } + + // `br_table` falls through when nothing matches. + let ebb = pos.current_ebb().unwrap(); + pos.remove_inst(); + cfg.recompute_ebb(pos.func, ebb); +} + /// Expand illegal `f32const` and `f64const` instructions. fn expand_fconst(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlowGraph) { let ty = func.dfg.value_type(func.dfg.first_result(inst)); From ba1c50d6c1b38dc75885a24e6e9ed84c5953b590 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 25 Sep 2017 11:06:18 -0700 Subject: [PATCH 1161/3084] Test WebAssembly floating point constants. f64.const does not yet work on 32-bit Intel. --- cranelift/filetests/wasm/f32-arith.cton | 6 +++++- cranelift/filetests/wasm/f64-arith.cton | 9 +++++---- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/cranelift/filetests/wasm/f32-arith.cton b/cranelift/filetests/wasm/f32-arith.cton index ad48b401b8..a72b2ae3ff 100644 --- a/cranelift/filetests/wasm/f32-arith.cton +++ b/cranelift/filetests/wasm/f32-arith.cton @@ -9,7 +9,11 @@ isa intel haswell ; Constants. -; function %f32_const() -> f32 +function %f32_const() -> f32 { +ebb0: + v1 = f32const 0x3.0 + return v1 +} ; Unary operations diff --git a/cranelift/filetests/wasm/f64-arith.cton b/cranelift/filetests/wasm/f64-arith.cton index bc81d69c3e..decfa6cba0 100644 --- a/cranelift/filetests/wasm/f64-arith.cton +++ b/cranelift/filetests/wasm/f64-arith.cton @@ -1,15 +1,16 @@ ; Test basic code generation for f64 arithmetic WebAssembly instructions. test compile -set is_64bit=0 -isa intel haswell - set is_64bit=1 isa intel haswell ; Constants. -; function %f64_const() -> f64 +function %f64_const() -> f64 { +ebb0: + v1 = f64const 0x3.0 + return v1 +} ; Unary operations From 8deca67968b531b4c85d46b232e86e354a7d40a3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 25 Sep 2017 12:12:47 -0700 Subject: [PATCH 1162/3084] Add legalization patterns for fabs and fneg. These sign bit manipulations need to use a -0.0 floating point constant which we didn't have a way of materializing previously. Add a ieee32.bits(0x...) syntax to the Python AST nodes that creates am f32 immediate value with the exact requested bitwise representation. --- cranelift/filetests/wasm/f32-arith.cton | 14 ++++++++++++-- cranelift/filetests/wasm/f64-arith.cton | 14 ++++++++++++-- lib/cretonne/meta/base/legalize.py | 21 ++++++++++++++++++++- lib/cretonne/meta/cdsl/ast.py | 21 +++++++++++++++++++++ lib/cretonne/meta/cdsl/operands.py | 11 ++++++++++- 5 files changed, 75 insertions(+), 6 deletions(-) diff --git a/cranelift/filetests/wasm/f32-arith.cton b/cranelift/filetests/wasm/f32-arith.cton index a72b2ae3ff..cb912dc3fc 100644 --- a/cranelift/filetests/wasm/f32-arith.cton +++ b/cranelift/filetests/wasm/f32-arith.cton @@ -17,8 +17,18 @@ ebb0: ; Unary operations -; function %f32_abs(f32) -> f32 -; function %f32_neg(f32) -> f32 +function %f32_abs(f32) -> f32 { +ebb0(v0: f32): + v1 = fabs v0 + return v1 +} + +function %f32_neg(f32) -> f32 { +ebb0(v0: f32): + v1 = fneg v0 + return v1 +} + ; function %f32_sqrt(f32) -> f32 ; function %f32_ceil(f32) -> f32 ; function %f32_floor(f32) -> f32 diff --git a/cranelift/filetests/wasm/f64-arith.cton b/cranelift/filetests/wasm/f64-arith.cton index decfa6cba0..1adc36e4f4 100644 --- a/cranelift/filetests/wasm/f64-arith.cton +++ b/cranelift/filetests/wasm/f64-arith.cton @@ -14,8 +14,18 @@ ebb0: ; Unary operations -; function %f64_abs(f64) -> f64 -; function %f64_neg(f64) -> f64 +function %f64_abs(f64) -> f64 { +ebb0(v0: f64): + v1 = fabs v0 + return v1 +} + +function %f64_neg(f64) -> f64 { +ebb0(v0: f64): + v1 = fneg v0 + return v1 +} + ; function %f64_sqrt(f64) -> f64 ; function %f64_ceil(f64) -> f64 ; function %f64_floor(f64) -> f64 diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index 4f92827b24..e535f62114 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -7,8 +7,9 @@ patterns that describe how base instructions can be transformed to other base instructions that are legal. """ from __future__ import absolute_import -from .immediates import intcc +from .immediates import intcc, ieee32, ieee64 from . import instructions as insts +from . import types from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm from .instructions import isub, isub_bin, isub_bout, isub_borrow from .instructions import imul, imul_imm @@ -18,6 +19,7 @@ from .instructions import icmp, icmp_imm from .instructions import iconst, bint from .instructions import ishl, ishl_imm, sshr, sshr_imm, ushr, ushr_imm from .instructions import rotl, rotl_imm, rotr, rotr_imm +from .instructions import f32const, f64const from cdsl.ast import Var from cdsl.xform import Rtl, XFormGroup @@ -207,3 +209,20 @@ for inst_not, inst in [ a1 << bnot(y), a << inst(x, a1) )) + +# Floating-point sign manipulations. +for ty, minus_zero in [ + (types.f32, f32const(ieee32.bits(0x80000000))), + (types.f64, f64const(ieee64.bits(0x8000000000000000)))]: + expand.legalize( + a << insts.fabs.bind(ty)(x), + Rtl( + b << minus_zero, + a << band_not(x, b), + )) + expand.legalize( + a << insts.fneg.bind(ty)(x), + Rtl( + b << minus_zero, + a << bxor(x, b), + )) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index 251fee3641..619c2d36a2 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -502,6 +502,27 @@ class ConstantInt(Literal): return str(self.value) +class ConstantBits(Literal): + """ + A bitwise value of an immediate operand. + + This is used to create bitwise exact floating point constants using + `ieee32.bits(0x80000000)`. + """ + + def __init__(self, kind, bits): + # type: (ImmediateKind, int) -> None + v = '{}::with_bits({:#x})'.format(kind.rust_type, bits) + super(ConstantBits, self).__init__(kind, v) + + def __str__(self): + # type: () -> str + """ + Get the Rust expression form of this constant. + """ + return str(self.value) + + class Enumerator(Literal): """ A value of an enumerated immediate operand. diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index ea59de5b3e..85d4adef99 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -8,7 +8,7 @@ try: from typing import Union, Dict, TYPE_CHECKING, Iterable # noqa OperandSpec = Union['OperandKind', ValueType, TypeVar] if TYPE_CHECKING: - from .ast import Enumerator, ConstantInt, Literal # noqa + from .ast import Enumerator, ConstantInt, ConstantBits, Literal # noqa except ImportError: pass @@ -123,6 +123,15 @@ class ImmediateKind(OperandKind): .format(self.name, value)) return ConstantInt(self, value) + def bits(self, bits): + # type: (int) -> ConstantBits + """ + Create an AST literal node for the given bitwise representation of this + immediate operand kind. + """ + from .ast import ConstantBits # noqa + return ConstantBits(self, bits) + def rust_enumerator(self, value): # type: (str) -> str """ From 14d6d1117de5fecff6271fe475fb0aa711e8f6fe Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 21 Sep 2017 14:50:28 -0700 Subject: [PATCH 1163/3084] Clean up unneeded '&'s. --- cranelift/src/filetest/concurrent.rs | 2 +- cranelift/src/filetest/domtree.rs | 2 +- cranelift/src/filetest/subtest.rs | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/cranelift/src/filetest/concurrent.rs b/cranelift/src/filetest/concurrent.rs index 1d7de213f3..430fb54142 100644 --- a/cranelift/src/filetest/concurrent.rs +++ b/cranelift/src/filetest/concurrent.rs @@ -136,7 +136,7 @@ fn worker_thread( } }); - if let &Err(ref msg) = &result { + if let Err(ref msg) = result { dbg!("FAIL: {}", msg); } diff --git a/cranelift/src/filetest/domtree.rs b/cranelift/src/filetest/domtree.rs index 46665ec428..622b01a891 100644 --- a/cranelift/src/filetest/domtree.rs +++ b/cranelift/src/filetest/domtree.rs @@ -96,7 +96,7 @@ impl SubTest for TestDomtree { // Now we know that everything in `expected` is consistent with `domtree`. // All other EBB's should be either unreachable or the entry block. for ebb in func.layout.ebbs().skip(1).filter( - |ebb| !expected.contains_key(&ebb), + |ebb| !expected.contains_key(ebb), ) { if let Some(got_inst) = domtree.idom(ebb) { diff --git a/cranelift/src/filetest/subtest.rs b/cranelift/src/filetest/subtest.rs index dc7ac6eeee..bbb8b666ca 100644 --- a/cranelift/src/filetest/subtest.rs +++ b/cranelift/src/filetest/subtest.rs @@ -85,14 +85,14 @@ impl<'a> filecheck::VariableMap for Context<'a> { /// Run filecheck on `text`, using directives extracted from `context`. pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { let checker = build_filechecker(context)?; - if checker.check(&text, context).map_err( + if checker.check(text, context).map_err( |e| format!("filecheck: {}", e), )? { Ok(()) } else { // Filecheck mismatch. Emit an explanation as output. - let (_, explain) = checker.explain(&text, context).map_err( + let (_, explain) = checker.explain(text, context).map_err( |e| format!("explain: {}", e), )?; Err(format!("filecheck failed:\n{}{}", checker, explain)) From 55e48ce7aa4c64db5f29f1df2ece053fcfe97435 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 25 Sep 2017 12:11:56 -0700 Subject: [PATCH 1164/3084] Simplify translate_type's return type. --- lib/wasm/src/code_translator.rs | 6 +++--- lib/wasm/src/translation_utils.rs | 12 ++++++------ 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index c66f9cea79..8af0127410 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -122,7 +122,7 @@ pub fn translate_operator( if let Ok(ty_cre) = type_to_type(&ty) { builder.append_ebb_arg(next, ty_cre); } - state.push_block(next, translate_type(ty).unwrap()); + state.push_block(next, translate_type(ty)); } Operator::Loop { ty } => { let loop_body = builder.create_ebb(); @@ -131,7 +131,7 @@ pub fn translate_operator( builder.append_ebb_arg(next, ty_cre); } builder.ins().jump(loop_body, &[]); - state.push_loop(loop_body, next, translate_type(ty).unwrap()); + state.push_loop(loop_body, next, translate_type(ty)); builder.switch_to_block(loop_body, &[]); } Operator::If { ty } => { @@ -147,7 +147,7 @@ pub fn translate_operator( if let Ok(ty_cre) = type_to_type(&ty) { builder.append_ebb_arg(if_not, ty_cre); } - state.push_if(jump_inst, if_not, translate_type(ty).unwrap()); + state.push_if(jump_inst, if_not, translate_type(ty)); } Operator::Else => { // We take the control frame pushed by the if, use its ebb as the else body diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 2c915f1e43..2d522a1ba8 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -119,13 +119,13 @@ pub fn f64_translation(x: wasmparser::Ieee64) -> cretonne::ir::immediates::Ieee6 } /// Translate a `wasmparser` type into its `Cretonne` equivalent, when possible -pub fn translate_type(ty: wasmparser::Type) -> Result, ()> { +pub fn translate_type(ty: wasmparser::Type) -> Vec { match ty { - wasmparser::Type::EmptyBlockType => Ok(Vec::new()), - wasmparser::Type::I32 => Ok(vec![cretonne::ir::types::I32]), - wasmparser::Type::F32 => Ok(vec![cretonne::ir::types::F32]), - wasmparser::Type::I64 => Ok(vec![cretonne::ir::types::I64]), - wasmparser::Type::F64 => Ok(vec![cretonne::ir::types::F64]), + wasmparser::Type::EmptyBlockType => Vec::new(), + wasmparser::Type::I32 => vec![cretonne::ir::types::I32], + wasmparser::Type::F32 => vec![cretonne::ir::types::F32], + wasmparser::Type::I64 => vec![cretonne::ir::types::I64], + wasmparser::Type::F64 => vec![cretonne::ir::types::F64], _ => panic!("unsupported return value type"), } } From 36585ddc4ff0a7b6baebf4d68a5666bd02d0bb47 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 25 Sep 2017 12:48:31 -0700 Subject: [PATCH 1165/3084] Wasm control stack entries only need the number of return types. This eliminates heap-allocated vectors which stored the actual types. --- lib/frontend/src/frontend.rs | 20 +++++++++---------- lib/wasm/src/code_translator.rs | 25 +++++++++++------------ lib/wasm/src/state.rs | 33 +++++++++++++++---------------- lib/wasm/src/translation_utils.rs | 12 +++++------ 4 files changed, 44 insertions(+), 46 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 376f798dd2..53b33334ea 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -129,12 +129,10 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short // If the user has supplied jump arguments we must adapt the arguments of // the destination ebb // TODO: find a way not to allocate a vector - let args_types: Vec = + let args_types: Vec = match data.analyze_branch(&self.builder.func.dfg.value_lists) { BranchInfo::SingleDest(_, args) => { - args.iter() - .map(|arg| self.builder.func.dfg.value_type(*arg)) - .collect() + args.to_vec() } _ => panic!("should not happen"), }; @@ -255,7 +253,7 @@ where /// When inserting the terminator instruction (which doesn't have a falltrough to its immediate /// successor), the block will be declared filled and it will not be possible to append /// instructions to it. - pub fn switch_to_block(&mut self, ebb: Ebb, jump_args: &[Type]) -> &[Value] { + pub fn switch_to_block(&mut self, ebb: Ebb, jump_args: &[Value]) -> &[Value] { if self.pristine { self.fill_function_args_values(ebb); } @@ -556,7 +554,7 @@ where } - fn ebb_args_adjustment(&mut self, dest_ebb: Ebb, jump_args: &[Type]) { + fn ebb_args_adjustment(&mut self, dest_ebb: Ebb, jump_args: &[Value]) { if self.builder.ssa.predecessors(dest_ebb).is_empty() || self.builder.ebbs[dest_ebb].pristine { @@ -571,7 +569,8 @@ where .iter() .zip(jump_args.iter().take(dest_ebb_args.len())) .all(|(dest_arg, jump_arg)| { - *jump_arg == self.func.dfg.value_type(*dest_arg) + self.func.dfg.value_type(*jump_arg) == + self.func.dfg.value_type(*dest_arg) }), "the jump argument supplied has not the \ same type as the corresponding dest ebb argument" @@ -579,8 +578,9 @@ where dest_ebb_args.len() }; self.builder.ebbs[dest_ebb].user_arg_count = jump_args.len(); - for ty in jump_args.iter().skip(dest_ebb_args_len) { - self.func.dfg.append_ebb_arg(dest_ebb, *ty); + for val in jump_args.iter().skip(dest_ebb_args_len) { + let ty = self.func.dfg.value_type(*val); + self.func.dfg.append_ebb_arg(dest_ebb, ty); } } else { // The Ebb already has predecessors @@ -599,7 +599,7 @@ where self.builder.ebbs[dest_ebb].user_arg_count, )) .all(|(jump_arg, dest_arg)| { - *jump_arg == self.func.dfg.value_type(*dest_arg) + self.func.dfg.value_type(*jump_arg) == self.func.dfg.value_type(*dest_arg) }), "the jump argument supplied has not the \ same type as the corresponding dest ebb argument" diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 8af0127410..d96deafb0a 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -26,7 +26,7 @@ use cretonne::ir::types::*; use cretonne::ir::condcodes::{IntCC, FloatCC}; use cton_frontend::FunctionBuilder; use wasmparser::{Operator, MemoryImmediate}; -use translation_utils::{f32_translation, f64_translation, type_to_type, translate_type, Local}; +use translation_utils::{f32_translation, f64_translation, type_to_type, num_return_values, Local}; use translation_utils::{TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; use state::{TranslationState, ControlStackFrame}; use std::collections::HashMap; @@ -122,7 +122,7 @@ pub fn translate_operator( if let Ok(ty_cre) = type_to_type(&ty) { builder.append_ebb_arg(next, ty_cre); } - state.push_block(next, translate_type(ty)); + state.push_block(next, num_return_values(ty)); } Operator::Loop { ty } => { let loop_body = builder.create_ebb(); @@ -131,7 +131,7 @@ pub fn translate_operator( builder.append_ebb_arg(next, ty_cre); } builder.ins().jump(loop_body, &[]); - state.push_loop(loop_body, next, translate_type(ty)); + state.push_loop(loop_body, next, num_return_values(ty)); builder.switch_to_block(loop_body, &[]); } Operator::If { ty } => { @@ -147,7 +147,7 @@ pub fn translate_operator( if let Ok(ty_cre) = type_to_type(&ty) { builder.append_ebb_arg(if_not, ty_cre); } - state.push_if(jump_inst, if_not, translate_type(ty)); + state.push_if(jump_inst, if_not, num_return_values(ty)); } Operator::Else => { // We take the control frame pushed by the if, use its ebb as the else body @@ -157,10 +157,10 @@ pub fn translate_operator( let (destination, return_count, branch_inst) = match state.control_stack[i] { ControlStackFrame::If { destination, - ref return_values, + num_return_values, branch_inst, .. - } => (destination, return_values.len(), branch_inst), + } => (destination, num_return_values, branch_inst), _ => panic!("should not happen"), }; builder.ins().jump(destination, state.peekn(return_count)); @@ -173,15 +173,14 @@ pub fn translate_operator( } Operator::End => { let frame = state.control_stack.pop().unwrap(); + let return_count = frame.num_return_values(); if !builder.is_unreachable() || !builder.is_pristine() { - let return_count = frame.return_values().len(); builder.ins().jump( frame.following_code(), state.peekn(return_count), ); - state.popn(return_count); } - builder.switch_to_block(frame.following_code(), frame.return_values()); + builder.switch_to_block(frame.following_code(), state.peekn(return_count)); builder.seal_block(frame.following_code()); // If it is a loop we also have to seal the body loop block match frame { @@ -223,7 +222,7 @@ pub fn translate_operator( let return_count = if frame.is_loop() { 0 } else { - frame.return_values().len() + frame.num_return_values() }; (return_count, frame.br_destination()) }; @@ -245,7 +244,7 @@ pub fn translate_operator( let return_count = if frame.is_loop() { 0 } else { - frame.return_values().len() + frame.num_return_values() }; (return_count, frame.br_destination()) }; @@ -269,7 +268,7 @@ pub fn translate_operator( if min_depth_frame.is_loop() { 0 } else { - min_depth_frame.return_values().len() + min_depth_frame.num_return_values() } }; if jump_args_count == 0 { @@ -334,7 +333,7 @@ pub fn translate_operator( let (return_count, br_destination) = { let frame = &mut state.control_stack[0]; frame.set_reachable(); - let return_count = frame.return_values().len(); + let return_count = frame.num_return_values(); (return_count, frame.br_destination()) }; { diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index 8130b469af..d862e8d9a0 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -3,7 +3,7 @@ //! The `TranslationState` struct defined in this module is used to keep track of the WebAssembly //! value and control stacks during the translation of a single function. -use cretonne::ir::{self, Ebb, Inst, Type, Value}; +use cretonne::ir::{self, Ebb, Inst, Value}; use runtime::{FuncEnvironment, GlobalValue}; use std::collections::HashMap; use translation_utils::{GlobalIndex, MemoryIndex, SignatureIndex, FunctionIndex}; @@ -12,7 +12,7 @@ use translation_utils::{GlobalIndex, MemoryIndex, SignatureIndex, FunctionIndex} /// fields: /// /// - `destination`: reference to the `Ebb` that will hold the code after the control block; -/// - `return_values`: types of the values returned by the control block; +/// - `num_return_values`: number of values returned by the control block; /// - `original_stack_size`: size of the value stack at the beginning of the control block. /// /// Moreover, the `if` frame has the `branch_inst` field that points to the `brz` instruction @@ -23,20 +23,20 @@ pub enum ControlStackFrame { If { destination: Ebb, branch_inst: Inst, - return_values: Vec, + num_return_values: usize, original_stack_size: usize, reachable: bool, }, Block { destination: Ebb, - return_values: Vec, + num_return_values: usize, original_stack_size: usize, reachable: bool, }, Loop { destination: Ebb, header: Ebb, - return_values: Vec, + num_return_values: usize, original_stack_size: usize, reachable: bool, }, @@ -44,11 +44,11 @@ pub enum ControlStackFrame { /// Helper methods for the control stack objects. impl ControlStackFrame { - pub fn return_values(&self) -> &[Type] { + pub fn num_return_values(&self) -> usize { match *self { - ControlStackFrame::If { ref return_values, .. } | - ControlStackFrame::Block { ref return_values, .. } | - ControlStackFrame::Loop { ref return_values, .. } => &return_values, + ControlStackFrame::If { num_return_values, .. } | + ControlStackFrame::Block { num_return_values, .. } | + ControlStackFrame::Loop { num_return_values, .. } => num_return_values, } } pub fn following_code(&self) -> Ebb { @@ -161,8 +161,7 @@ impl TranslationState { sig.return_types .iter() .filter(|arg| arg.purpose == ir::ArgumentPurpose::Normal) - .map(|argty| argty.value_type) - .collect(), + .count(), ); } @@ -215,33 +214,33 @@ impl TranslationState { } // Push a block on the control stack. - pub fn push_block(&mut self, following_code: Ebb, result_types: Vec) { + pub fn push_block(&mut self, following_code: Ebb, num_result_types: usize) { self.control_stack.push(ControlStackFrame::Block { destination: following_code, original_stack_size: self.stack.len(), - return_values: result_types, + num_return_values: num_result_types, reachable: false, }); } // Push a loop on the control stack. - pub fn push_loop(&mut self, header: Ebb, following_code: Ebb, result_types: Vec) { + pub fn push_loop(&mut self, header: Ebb, following_code: Ebb, num_result_types: usize) { self.control_stack.push(ControlStackFrame::Loop { header, destination: following_code, original_stack_size: self.stack.len(), - return_values: result_types, + num_return_values: num_result_types, reachable: false, }); } // Push an if on the control stack. - pub fn push_if(&mut self, branch_inst: Inst, following_code: Ebb, result_types: Vec) { + pub fn push_if(&mut self, branch_inst: Inst, following_code: Ebb, num_result_types: usize) { self.control_stack.push(ControlStackFrame::If { branch_inst, destination: following_code, original_stack_size: self.stack.len(), - return_values: result_types, + num_return_values: num_result_types, reachable: false, }); } diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 2d522a1ba8..c21b580b29 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -119,13 +119,13 @@ pub fn f64_translation(x: wasmparser::Ieee64) -> cretonne::ir::immediates::Ieee6 } /// Translate a `wasmparser` type into its `Cretonne` equivalent, when possible -pub fn translate_type(ty: wasmparser::Type) -> Vec { +pub fn num_return_values(ty: wasmparser::Type) -> usize { match ty { - wasmparser::Type::EmptyBlockType => Vec::new(), - wasmparser::Type::I32 => vec![cretonne::ir::types::I32], - wasmparser::Type::F32 => vec![cretonne::ir::types::F32], - wasmparser::Type::I64 => vec![cretonne::ir::types::I64], - wasmparser::Type::F64 => vec![cretonne::ir::types::F64], + wasmparser::Type::EmptyBlockType => 0, + wasmparser::Type::I32 | + wasmparser::Type::F32 | + wasmparser::Type::I64 | + wasmparser::Type::F64 => 1, _ => panic!("unsupported return value type"), } } From ac343ba92a56bf5f9abafd35b50dcd730af5be30 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 25 Sep 2017 13:14:55 -0700 Subject: [PATCH 1166/3084] Add encodings for square root instructions. --- .../filetests/isa/intel/binary32-float.cton | 16 ++++++++++++++++ .../filetests/isa/intel/binary64-float.cton | 14 ++++++++++++++ cranelift/filetests/wasm/f32-arith.cton | 7 ++++++- cranelift/filetests/wasm/f64-arith.cton | 7 ++++++- lib/cretonne/meta/isa/intel/encodings.py | 3 +++ 5 files changed, 45 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index c5a1b08a0e..85d613fc7a 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -82,6 +82,14 @@ ebb0: ; asm: xorps %xmm5, %xmm2 [-,%xmm2] v37 = bxor v11, v10 ; bin: 0f 57 d5 + ; Unary arithmetic. + + ; asm: sqrtss %xmm5, %xmm2 + [-,%xmm2] v50 = sqrt v10 ; bin: f3 0f 51 d5 + ; asm: sqrtss %xmm2, %xmm5 + [-,%xmm5] v51 = sqrt v11 ; bin: f3 0f 51 ea + + ; Load/Store ; asm: movd (%ecx), %xmm5 @@ -192,6 +200,14 @@ ebb0: ; asm: xorps %xmm5, %xmm2 [-,%xmm2] v37 = bxor v11, v10 ; bin: 0f 57 d5 + ; Unary arithmetic. + + ; asm: sqrtsd %xmm5, %xmm2 + [-,%xmm2] v50 = sqrt v10 ; bin: f2 0f 51 d5 + ; asm: sqrtsd %xmm2, %xmm5 + [-,%xmm5] v51 = sqrt v11 ; bin: f2 0f 51 ea + + ; Load/Store ; asm: movq (%ecx), %xmm5 diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index a5b4aaa163..e59ca0d1de 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -91,6 +91,13 @@ ebb0: ; asm: xorps %xmm5, %xmm10 [-,%xmm10] v37 = bxor v11, v10 ; bin: 44 0f 57 d5 + ; Unary arithmetic. + + ; asm: sqrtss %xmm5, %xmm10 + [-,%xmm10] v50 = sqrt v10 ; bin: f3 44 0f 51 d5 + ; asm: sqrtss %xmm10, %xmm5 + [-,%xmm5] v51 = sqrt v11 ; bin: f3 41 0f 51 ea + ; Load/Store ; asm: movd (%r14), %xmm5 @@ -216,6 +223,13 @@ ebb0: ; asm: xorps %xmm5, %xmm10 [-,%xmm10] v37 = bxor v11, v10 ; bin: 44 0f 57 d5 + ; Unary arithmetic. + + ; asm: sqrtsd %xmm5, %xmm10 + [-,%xmm10] v50 = sqrt v10 ; bin: f2 44 0f 51 d5 + ; asm: sqrtsd %xmm10, %xmm5 + [-,%xmm5] v51 = sqrt v11 ; bin: f2 41 0f 51 ea + ; Load/Store ; asm: movq (%r14), %xmm5 diff --git a/cranelift/filetests/wasm/f32-arith.cton b/cranelift/filetests/wasm/f32-arith.cton index cb912dc3fc..5a84579136 100644 --- a/cranelift/filetests/wasm/f32-arith.cton +++ b/cranelift/filetests/wasm/f32-arith.cton @@ -29,7 +29,12 @@ ebb0(v0: f32): return v1 } -; function %f32_sqrt(f32) -> f32 +function %f32_sqrt(f32) -> f32 { +ebb0(v0: f32): + v1 = sqrt v0 + return v1 +} + ; function %f32_ceil(f32) -> f32 ; function %f32_floor(f32) -> f32 ; function %f32_trunc(f32) -> f32 diff --git a/cranelift/filetests/wasm/f64-arith.cton b/cranelift/filetests/wasm/f64-arith.cton index 1adc36e4f4..3eaecc5a0b 100644 --- a/cranelift/filetests/wasm/f64-arith.cton +++ b/cranelift/filetests/wasm/f64-arith.cton @@ -26,7 +26,12 @@ ebb0(v0: f64): return v1 } -; function %f64_sqrt(f64) -> f64 +function %f64_sqrt(f64) -> f64 { +ebb0(v0: f64): + v1 = sqrt v0 + return v1 +} + ; function %f64_ceil(f64) -> f64 ; function %f64_floor(f64) -> f64 ; function %f64_trunc(f64) -> f64 diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 4b646658c1..37766c5207 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -359,6 +359,9 @@ enc_flt(base.fpromote.f64.f32, r.furm, 0xf3, 0x0f, 0x5a) # cvtsd2ss enc_flt(base.fdemote.f32.f64, r.furm, 0xf2, 0x0f, 0x5a) +# Exact square roots. +enc_flt(base.sqrt.f32, r.furm, 0xf3, 0x0f, 0x51) +enc_flt(base.sqrt.f64, r.furm, 0xf2, 0x0f, 0x51) # Binary arithmetic ops. for inst, opc in [ From 6bec5f85077c63fccf808eb5c2689b3caf27a1ae Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 25 Sep 2017 14:57:01 -0700 Subject: [PATCH 1167/3084] Intel encodings for nearest/floor/ceil/trunc. These floating point rounding operations all use the roundss/roundsd instructions that are available in SSE 4.1. --- .../filetests/isa/intel/binary32-float.cton | 54 ++++++++++++++++++ .../filetests/isa/intel/binary64-float.cton | 56 +++++++++++++++++++ cranelift/filetests/wasm/f32-arith.cton | 27 +++++++-- cranelift/filetests/wasm/f64-arith.cton | 27 +++++++-- lib/cretonne/meta/gen_binemit.py | 1 + lib/cretonne/meta/isa/intel/encodings.py | 15 ++++- lib/cretonne/meta/isa/intel/recipes.py | 15 +++++ lib/cretonne/src/isa/intel/binemit.rs | 35 ++++++++++-- 8 files changed, 216 insertions(+), 14 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 85d613fc7a..4fa083b3de 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -89,6 +89,33 @@ ebb0: ; asm: sqrtss %xmm2, %xmm5 [-,%xmm5] v51 = sqrt v11 ; bin: f3 0f 51 ea + ; asm: roundss $0, %xmm5, %xmm4 + [-,%xmm4] v52 = nearest v10 ; bin: 66 0f 3a 0a e5 00 + ; asm: roundss $0, %xmm2, %xmm5 + [-,%xmm5] v53 = nearest v11 ; bin: 66 0f 3a 0a ea 00 + ; asm: roundss $0, %xmm5, %xmm2 + [-,%xmm2] v54 = nearest v10 ; bin: 66 0f 3a 0a d5 00 + + ; asm: roundss $1, %xmm5, %xmm4 + [-,%xmm4] v55 = floor v10 ; bin: 66 0f 3a 0a e5 01 + ; asm: roundss $1, %xmm2, %xmm5 + [-,%xmm5] v56 = floor v11 ; bin: 66 0f 3a 0a ea 01 + ; asm: roundss $1, %xmm5, %xmm2 + [-,%xmm2] v57 = floor v10 ; bin: 66 0f 3a 0a d5 01 + + ; asm: roundss $2, %xmm5, %xmm4 + [-,%xmm4] v58 = ceil v10 ; bin: 66 0f 3a 0a e5 02 + ; asm: roundss $2, %xmm2, %xmm5 + [-,%xmm5] v59 = ceil v11 ; bin: 66 0f 3a 0a ea 02 + ; asm: roundss $2, %xmm5, %xmm2 + [-,%xmm2] v60 = ceil v10 ; bin: 66 0f 3a 0a d5 02 + + ; asm: roundss $3, %xmm5, %xmm4 + [-,%xmm4] v61 = trunc v10 ; bin: 66 0f 3a 0a e5 03 + ; asm: roundss $3, %xmm2, %xmm5 + [-,%xmm5] v62 = trunc v11 ; bin: 66 0f 3a 0a ea 03 + ; asm: roundss $3, %xmm5, %xmm2 + [-,%xmm2] v63 = trunc v10 ; bin: 66 0f 3a 0a d5 03 ; Load/Store @@ -207,6 +234,33 @@ ebb0: ; asm: sqrtsd %xmm2, %xmm5 [-,%xmm5] v51 = sqrt v11 ; bin: f2 0f 51 ea + ; asm: roundsd $0, %xmm5, %xmm4 + [-,%xmm4] v52 = nearest v10 ; bin: 66 0f 3a 0b e5 00 + ; asm: roundsd $0, %xmm2, %xmm5 + [-,%xmm5] v53 = nearest v11 ; bin: 66 0f 3a 0b ea 00 + ; asm: roundsd $0, %xmm5, %xmm2 + [-,%xmm2] v54 = nearest v10 ; bin: 66 0f 3a 0b d5 00 + + ; asm: roundsd $1, %xmm5, %xmm4 + [-,%xmm4] v55 = floor v10 ; bin: 66 0f 3a 0b e5 01 + ; asm: roundsd $1, %xmm2, %xmm5 + [-,%xmm5] v56 = floor v11 ; bin: 66 0f 3a 0b ea 01 + ; asm: roundsd $1, %xmm5, %xmm2 + [-,%xmm2] v57 = floor v10 ; bin: 66 0f 3a 0b d5 01 + + ; asm: roundsd $2, %xmm5, %xmm4 + [-,%xmm4] v58 = ceil v10 ; bin: 66 0f 3a 0b e5 02 + ; asm: roundsd $2, %xmm2, %xmm5 + [-,%xmm5] v59 = ceil v11 ; bin: 66 0f 3a 0b ea 02 + ; asm: roundsd $2, %xmm5, %xmm2 + [-,%xmm2] v60 = ceil v10 ; bin: 66 0f 3a 0b d5 02 + + ; asm: roundsd $3, %xmm5, %xmm4 + [-,%xmm4] v61 = trunc v10 ; bin: 66 0f 3a 0b e5 03 + ; asm: roundsd $3, %xmm2, %xmm5 + [-,%xmm5] v62 = trunc v11 ; bin: 66 0f 3a 0b ea 03 + ; asm: roundsd $3, %xmm5, %xmm2 + [-,%xmm2] v63 = trunc v10 ; bin: 66 0f 3a 0b d5 03 ; Load/Store diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index e59ca0d1de..c8d4df232f 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -98,6 +98,34 @@ ebb0: ; asm: sqrtss %xmm10, %xmm5 [-,%xmm5] v51 = sqrt v11 ; bin: f3 41 0f 51 ea + ; asm: roundss $0, %xmm5, %xmm10 + [-,%xmm10] v52 = nearest v10 ; bin: 66 44 0f 3a 0a d5 00 + ; asm: roundss $0, %xmm10, %xmm5 + [-,%xmm5] v53 = nearest v11 ; bin: 66 41 0f 3a 0a ea 00 + ; asm: roundss $0, %xmm5, %xmm2 + [-,%xmm2] v54 = nearest v10 ; bin: 66 0f 3a 0a d5 00 + + ; asm: roundss $1, %xmm5, %xmm10 + [-,%xmm10] v55 = floor v10 ; bin: 66 44 0f 3a 0a d5 01 + ; asm: roundss $1, %xmm10, %xmm5 + [-,%xmm5] v56 = floor v11 ; bin: 66 41 0f 3a 0a ea 01 + ; asm: roundss $1, %xmm5, %xmm2 + [-,%xmm2] v57 = floor v10 ; bin: 66 0f 3a 0a d5 01 + + ; asm: roundss $2, %xmm5, %xmm10 + [-,%xmm10] v58 = ceil v10 ; bin: 66 44 0f 3a 0a d5 02 + ; asm: roundss $2, %xmm10, %xmm5 + [-,%xmm5] v59 = ceil v11 ; bin: 66 41 0f 3a 0a ea 02 + ; asm: roundss $2, %xmm5, %xmm2 + [-,%xmm2] v60 = ceil v10 ; bin: 66 0f 3a 0a d5 02 + + ; asm: roundss $3, %xmm5, %xmm10 + [-,%xmm10] v61 = trunc v10 ; bin: 66 44 0f 3a 0a d5 03 + ; asm: roundss $3, %xmm10, %xmm5 + [-,%xmm5] v62 = trunc v11 ; bin: 66 41 0f 3a 0a ea 03 + ; asm: roundss $3, %xmm5, %xmm2 + [-,%xmm2] v63 = trunc v10 ; bin: 66 0f 3a 0a d5 03 + ; Load/Store ; asm: movd (%r14), %xmm5 @@ -230,6 +258,34 @@ ebb0: ; asm: sqrtsd %xmm10, %xmm5 [-,%xmm5] v51 = sqrt v11 ; bin: f2 41 0f 51 ea + ; asm: roundsd $0, %xmm5, %xmm10 + [-,%xmm10] v52 = nearest v10 ; bin: 66 44 0f 3a 0b d5 00 + ; asm: roundsd $0, %xmm10, %xmm5 + [-,%xmm5] v53 = nearest v11 ; bin: 66 41 0f 3a 0b ea 00 + ; asm: roundsd $0, %xmm5, %xmm2 + [-,%xmm2] v54 = nearest v10 ; bin: 66 0f 3a 0b d5 00 + + ; asm: roundsd $1, %xmm5, %xmm10 + [-,%xmm10] v55 = floor v10 ; bin: 66 44 0f 3a 0b d5 01 + ; asm: roundsd $1, %xmm10, %xmm5 + [-,%xmm5] v56 = floor v11 ; bin: 66 41 0f 3a 0b ea 01 + ; asm: roundsd $1, %xmm5, %xmm2 + [-,%xmm2] v57 = floor v10 ; bin: 66 0f 3a 0b d5 01 + + ; asm: roundsd $2, %xmm5, %xmm10 + [-,%xmm10] v58 = ceil v10 ; bin: 66 44 0f 3a 0b d5 02 + ; asm: roundsd $2, %xmm10, %xmm5 + [-,%xmm5] v59 = ceil v11 ; bin: 66 41 0f 3a 0b ea 02 + ; asm: roundsd $2, %xmm5, %xmm2 + [-,%xmm2] v60 = ceil v10 ; bin: 66 0f 3a 0b d5 02 + + ; asm: roundsd $3, %xmm5, %xmm10 + [-,%xmm10] v61 = trunc v10 ; bin: 66 44 0f 3a 0b d5 03 + ; asm: roundsd $3, %xmm10, %xmm5 + [-,%xmm5] v62 = trunc v11 ; bin: 66 41 0f 3a 0b ea 03 + ; asm: roundsd $3, %xmm5, %xmm2 + [-,%xmm2] v63 = trunc v10 ; bin: 66 0f 3a 0b d5 03 + ; Load/Store ; asm: movq (%r14), %xmm5 diff --git a/cranelift/filetests/wasm/f32-arith.cton b/cranelift/filetests/wasm/f32-arith.cton index 5a84579136..2e7281398d 100644 --- a/cranelift/filetests/wasm/f32-arith.cton +++ b/cranelift/filetests/wasm/f32-arith.cton @@ -35,10 +35,29 @@ ebb0(v0: f32): return v1 } -; function %f32_ceil(f32) -> f32 -; function %f32_floor(f32) -> f32 -; function %f32_trunc(f32) -> f32 -; function %f32_nearest (f32) -> f32 +function %f32_ceil(f32) -> f32 { +ebb0(v0: f32): + v1 = ceil v0 + return v1 +} + +function %f32_floor(f32) -> f32 { +ebb0(v0: f32): + v1 = floor v0 + return v1 +} + +function %f32_trunc(f32) -> f32 { +ebb0(v0: f32): + v1 = trunc v0 + return v1 +} + +function %f32_nearest (f32) -> f32 { +ebb0(v0: f32): + v1 = nearest v0 + return v1 +} ; Binary Operations diff --git a/cranelift/filetests/wasm/f64-arith.cton b/cranelift/filetests/wasm/f64-arith.cton index 3eaecc5a0b..cfed4f95a5 100644 --- a/cranelift/filetests/wasm/f64-arith.cton +++ b/cranelift/filetests/wasm/f64-arith.cton @@ -32,10 +32,29 @@ ebb0(v0: f64): return v1 } -; function %f64_ceil(f64) -> f64 -; function %f64_floor(f64) -> f64 -; function %f64_trunc(f64) -> f64 -; function %f64_nearest (f64) -> f64 +function %f64_ceil(f64) -> f64 { +ebb0(v0: f64): + v1 = ceil v0 + return v1 +} + +function %f64_floor(f64) -> f64 { +ebb0(v0: f64): + v1 = floor v0 + return v1 +} + +function %f64_trunc(f64) -> f64 { +ebb0(v0: f64): + v1 = trunc v0 + return v1 +} + +function %f64_nearest (f64) -> f64 { +ebb0(v0: f64): + v1 = nearest v0 + return v1 +} ; Binary Operations diff --git a/lib/cretonne/meta/gen_binemit.py b/lib/cretonne/meta/gen_binemit.py index 39cab5ab20..58a44473a3 100644 --- a/lib/cretonne/meta/gen_binemit.py +++ b/lib/cretonne/meta/gen_binemit.py @@ -38,6 +38,7 @@ def gen_recipe(recipe, fmt): with fmt.indented( 'if let InstructionData::{} {{'.format(iform.name), '}'): + fmt.line('opcode,') for f in iform.imm_fields: fmt.line('{},'.format(f.member)) if want_args: diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 37766c5207..6d46111d7f 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -11,9 +11,10 @@ from . import settings as cfg from . import instructions as x86 from .legalize import intel_expand from base.legalize import narrow, expand +from .settings import use_sse41 try: - from typing import TYPE_CHECKING + from typing import TYPE_CHECKING, Any # noqa if TYPE_CHECKING: from cdsl.instructions import MaybeBoundInst # noqa except ImportError: @@ -82,7 +83,7 @@ def enc_i32_i64_ld_st(inst, w_bit, recipe, *args, **kwargs): def enc_flt(inst, recipe, *args, **kwargs): - # type: (MaybeBoundInst, r.TailRecipe, *int, **int) -> None + # type: (MaybeBoundInst, r.TailRecipe, *int, **Any) -> None """ Add encodings for floating point instruction `inst` to both I32 and I64. """ @@ -363,6 +364,16 @@ enc_flt(base.fdemote.f32.f64, r.furm, 0xf2, 0x0f, 0x5a) enc_flt(base.sqrt.f32, r.furm, 0xf3, 0x0f, 0x51) enc_flt(base.sqrt.f64, r.furm, 0xf2, 0x0f, 0x51) +# Rounding. The recipe looks at the opcode to pick an immediate. +for inst in [ + base.nearest, + base.floor, + base.ceil, + base.trunc]: + enc_flt(inst.f32, r.furmi_rnd, 0x66, 0x0f, 0x3a, 0x0a, isap=use_sse41) + enc_flt(inst.f64, r.furmi_rnd, 0x66, 0x0f, 0x3a, 0x0b, isap=use_sse41) + + # Binary arithmetic ops. for inst, opc in [ (base.fadd, 0x58), diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 358f9cae72..04c7c91891 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -289,6 +289,21 @@ frurm = TailRecipe( modrm_rr(in_reg0, out_reg0, sink); ''') +# XX /r, RMI form for one of the roundXX SSE 4.1 instructions. +furmi_rnd = TailRecipe( + 'furmi_rnd', Unary, size=2, ins=FPR, outs=FPR, + emit=''' + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); + modrm_rr(in_reg0, out_reg0, sink); + sink.put1(match opcode { + Opcode::Nearest => 0b00, + Opcode::Floor => 0b01, + Opcode::Ceil => 0b10, + Opcode::Trunc => 0b11, + x => panic!("{} unexpected for furmi_rnd", opcode), + }); + ''') + # XX /r, for regmove instructions. rmov = TailRecipe( 'ur', RegMove, size=1, ins=GPR, outs=(), diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 888128caed..4358c9da15 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -1,7 +1,7 @@ //! Emitting binary Intel machine code. use binemit::{CodeSink, Reloc, bad_encoding}; -use ir::{Function, Inst, Ebb, InstructionData}; +use ir::{Function, Inst, Ebb, InstructionData, Opcode}; use isa::{RegUnit, StackRef, StackBase, StackBaseMask}; use regalloc::RegDiversions; use super::registers::RU; @@ -41,6 +41,9 @@ fn stk_base(base: StackBase) -> RegUnit { // Mandatory prefix bytes for Mp* opcodes. const PREFIX: [u8; 3] = [0x66, 0xf3, 0xf2]; +// Second byte for three-byte opcodes for mm=0b10 and mm=0b11. +const OP3_BYTE2: [u8; 2] = [0x38, 0x3a]; + // A REX prefix with no bits set: 0b0100WRXB. const BASE_REX: u8 = 0b0100_0000; @@ -111,6 +114,15 @@ fn put_mp1(bits: u16, rex: u8, sink: &mut CS) { sink.put1(bits as u8); } +// Emit single-byte opcode with mandatory prefix and REX. +fn put_rexmp1(bits: u16, rex: u8, sink: &mut CS) { + debug_assert_eq!(bits & 0x0c00, 0, "Invalid encoding bits for Mp1*"); + let pp = (bits >> 8) & 3; + sink.put1(PREFIX[(pp - 1) as usize]); + rex_prefix(bits, rex, sink); + sink.put1(bits as u8); +} + // Emit two-byte opcode (0F XX) with mandatory prefix. fn put_mp2(bits: u16, rex: u8, sink: &mut CS) { debug_assert_eq!(bits & 0x8c00, 0x0400, "Invalid encoding bits for Mp2*"); @@ -131,12 +143,27 @@ fn put_rexmp2(bits: u16, rex: u8, sink: &mut CS) { sink.put1(bits as u8); } -// Emit single-byte opcode with mandatory prefix and REX. -fn put_rexmp1(bits: u16, rex: u8, sink: &mut CS) { - debug_assert_eq!(bits & 0x0c00, 0, "Invalid encoding bits for Mp1*"); +// Emit three-byte opcode (0F 3[8A] XX) with mandatory prefix. +fn put_mp3(bits: u16, rex: u8, sink: &mut CS) { + debug_assert_eq!(bits & 0x8800, 0x0800, "Invalid encoding bits for Mp3*"); + let pp = (bits >> 8) & 3; + sink.put1(PREFIX[(pp - 1) as usize]); + debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less Mp3 encoding"); + let mm = (bits >> 10) & 3; + sink.put1(0x0f); + sink.put1(OP3_BYTE2[(mm - 2) as usize]); + sink.put1(bits as u8); +} + +// Emit three-byte opcode (0F 3[8A] XX) with mandatory prefix and REX +fn put_rexmp3(bits: u16, rex: u8, sink: &mut CS) { + debug_assert_eq!(bits & 0x0800, 0x0800, "Invalid encoding bits for Mp3*"); let pp = (bits >> 8) & 3; sink.put1(PREFIX[(pp - 1) as usize]); rex_prefix(bits, rex, sink); + let mm = (bits >> 10) & 3; + sink.put1(0x0f); + sink.put1(OP3_BYTE2[(mm - 2) as usize]); sink.put1(bits as u8); } From 79968a232536aef28924de41a3e69abcbb5dfd29 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 25 Sep 2017 15:17:32 -0700 Subject: [PATCH 1168/3084] Add standard expansions for fcopysign. This is also just a sign bit manipulation. --- cranelift/filetests/wasm/f32-arith.cton | 7 ++++++- cranelift/filetests/wasm/f64-arith.cton | 7 ++++++- lib/cretonne/meta/base/legalize.py | 8 ++++++++ 3 files changed, 20 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/wasm/f32-arith.cton b/cranelift/filetests/wasm/f32-arith.cton index 2e7281398d..ddb05138e9 100644 --- a/cranelift/filetests/wasm/f32-arith.cton +++ b/cranelift/filetests/wasm/f32-arith.cton @@ -87,4 +87,9 @@ ebb0(v0: f32, v1: f32): ; function %f32_min(f32, f32) -> f32 ; function %f32_max(f32, f32) -> f32 -; function %f32_copysign(f32, f32) -> f32 + +function %f32_copysign(f32, f32) -> f32 { +ebb0(v0: f32, v1: f32): + v2 = fcopysign v0, v1 + return v2 +} diff --git a/cranelift/filetests/wasm/f64-arith.cton b/cranelift/filetests/wasm/f64-arith.cton index cfed4f95a5..d67066e615 100644 --- a/cranelift/filetests/wasm/f64-arith.cton +++ b/cranelift/filetests/wasm/f64-arith.cton @@ -84,4 +84,9 @@ ebb0(v0: f64, v1: f64): ; function %f64_min(f64, f64) -> f64 ; function %f64_max(f64, f64) -> f64 -; function %f64_copysign(f64, f64) -> f64 + +function %f64_copysign(f64, f64) -> f64 { +ebb0(v0: f64, v1: f64): + v2 = fcopysign v0, v1 + return v2 +} diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index e535f62114..8608feb392 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -226,3 +226,11 @@ for ty, minus_zero in [ b << minus_zero, a << bxor(x, b), )) + expand.legalize( + a << insts.fcopysign.bind(ty)(x, y), + Rtl( + b << minus_zero, + a1 << band_not(x, b), + a2 << band(y, b), + a << bor(a1, a2) + )) From 7fb6159a85ac4ca0c424bc47850b5c1a5563d97e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 26 Sep 2017 09:54:54 -0700 Subject: [PATCH 1169/3084] Add Intel encodings for the fcmp instruction. Not all floating point condition codes are directly supported by the ucimiss/ucomisd instructions. Some inequalities need to be reversed and eq+ne require two separate tests. --- .../filetests/isa/intel/binary32-float.cton | 60 +++++++++++++++++++ .../filetests/isa/intel/binary64-float.cton | 60 +++++++++++++++++++ cranelift/filetests/wasm/f32-compares.cton | 50 ++++++++++++++++ cranelift/filetests/wasm/f64-compares.cton | 50 ++++++++++++++++ lib/cretonne/meta/gen_legalizer.py | 6 +- lib/cretonne/meta/isa/intel/defs.py | 13 ++++ lib/cretonne/meta/isa/intel/encodings.py | 22 +++++-- lib/cretonne/meta/isa/intel/legalize.py | 38 +++++++++++- lib/cretonne/meta/isa/intel/recipes.py | 53 +++++++++++++++- 9 files changed, 342 insertions(+), 10 deletions(-) create mode 100644 cranelift/filetests/wasm/f32-compares.cton create mode 100644 cranelift/filetests/wasm/f64-compares.cton diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 4fa083b3de..b0a2c83c8f 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -157,6 +157,36 @@ ebb0: ; asm: movd 1032(%esp), %xmm2 [-,%xmm2] v211 = fill v201 ; bin: 66 0f 6e 94 24 00000408 + ; Comparisons. + ; + ; Only `supported_floatccs` are tested here. Others are handled by + ; legalization paterns. + + ; asm: ucomiss %xmm2, %xmm5 + ; asm: setnp %bl + [-,%rbx] v300 = fcmp ord v10, v11 ; bin: 0f 2e ea 0f 9b c3 + ; asm: ucomiss %xmm5, %xmm2 + ; asm: setp %bl + [-,%rbx] v301 = fcmp uno v11, v10 ; bin: 0f 2e d5 0f 9a c3 + ; asm: ucomiss %xmm2, %xmm5 + ; asm: setne %dl + [-,%rdx] v302 = fcmp one v10, v11 ; bin: 0f 2e ea 0f 95 c2 + ; asm: ucomiss %xmm5, %xmm2 + ; asm: sete %dl + [-,%rdx] v303 = fcmp ueq v11, v10 ; bin: 0f 2e d5 0f 94 c2 + ; asm: ucomiss %xmm2, %xmm5 + ; asm: seta %bl + [-,%rbx] v304 = fcmp gt v10, v11 ; bin: 0f 2e ea 0f 97 c3 + ; asm: ucomiss %xmm5, %xmm2 + ; asm: setae %bl + [-,%rbx] v305 = fcmp ge v11, v10 ; bin: 0f 2e d5 0f 93 c3 + ; asm: ucomiss %xmm2, %xmm5 + ; asm: setb %dl + [-,%rdx] v306 = fcmp ult v10, v11 ; bin: 0f 2e ea 0f 92 c2 + ; asm: ucomiss %xmm5, %xmm2 + ; asm: setbe %dl + [-,%rdx] v307 = fcmp ule v11, v10 ; bin: 0f 2e d5 0f 96 c2 + return } @@ -302,5 +332,35 @@ ebb0: ; asm: movq 1032(%esp), %xmm2 [-,%xmm2] v211 = fill v201 ; bin: f3 0f 7e 94 24 00000408 + ; Comparisons. + ; + ; Only `supported_floatccs` are tested here. Others are handled by + ; legalization paterns. + + ; asm: ucomisd %xmm2, %xmm5 + ; asm: setnp %bl + [-,%rbx] v300 = fcmp ord v10, v11 ; bin: 66 0f 2e ea 0f 9b c3 + ; asm: ucomisd %xmm5, %xmm2 + ; asm: setp %bl + [-,%rbx] v301 = fcmp uno v11, v10 ; bin: 66 0f 2e d5 0f 9a c3 + ; asm: ucomisd %xmm2, %xmm5 + ; asm: setne %dl + [-,%rdx] v302 = fcmp one v10, v11 ; bin: 66 0f 2e ea 0f 95 c2 + ; asm: ucomisd %xmm5, %xmm2 + ; asm: sete %dl + [-,%rdx] v303 = fcmp ueq v11, v10 ; bin: 66 0f 2e d5 0f 94 c2 + ; asm: ucomisd %xmm2, %xmm5 + ; asm: seta %bl + [-,%rbx] v304 = fcmp gt v10, v11 ; bin: 66 0f 2e ea 0f 97 c3 + ; asm: ucomisd %xmm5, %xmm2 + ; asm: setae %bl + [-,%rbx] v305 = fcmp ge v11, v10 ; bin: 66 0f 2e d5 0f 93 c3 + ; asm: ucomisd %xmm2, %xmm5 + ; asm: setb %dl + [-,%rdx] v306 = fcmp ult v10, v11 ; bin: 66 0f 2e ea 0f 92 c2 + ; asm: ucomisd %xmm5, %xmm2 + ; asm: setbe %dl + [-,%rdx] v307 = fcmp ule v11, v10 ; bin: 66 0f 2e d5 0f 96 c2 + return } diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index c8d4df232f..542a712f26 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -166,6 +166,36 @@ ebb0: ; asm: movd 1032(%rsp), %xmm10 [-,%xmm10] v211 = fill v201 ; bin: 66 44 0f 6e 94 24 00000408 + ; Comparisons. + ; + ; Only `supported_floatccs` are tested here. Others are handled by + ; legalization paterns. + + ; asm: ucomiss %xmm10, %xmm5 + ; asm: setnp %bl + [-,%rbx] v300 = fcmp ord v10, v11 ; bin: 41 0f 2e ea 0f 9b c3 + ; asm: ucomiss %xmm5, %xmm10 + ; asm: setp %bl + [-,%rbx] v301 = fcmp uno v11, v10 ; bin: 44 0f 2e d5 0f 9a c3 + ; asm: ucomiss %xmm10, %xmm5 + ; asm: setne %dl + [-,%rdx] v302 = fcmp one v10, v11 ; bin: 41 0f 2e ea 0f 95 c2 + ; asm: ucomiss %xmm5, %xmm10 + ; asm: sete %dl + [-,%rdx] v303 = fcmp ueq v11, v10 ; bin: 44 0f 2e d5 0f 94 c2 + ; asm: ucomiss %xmm10, %xmm5 + ; asm: seta %bl + [-,%rbx] v304 = fcmp gt v10, v11 ; bin: 41 0f 2e ea 0f 97 c3 + ; asm: ucomiss %xmm5, %xmm10 + ; asm: setae %bl + [-,%rbx] v305 = fcmp ge v11, v10 ; bin: 44 0f 2e d5 0f 93 c3 + ; asm: ucomiss %xmm10, %xmm5 + ; asm: setb %dl + [-,%rdx] v306 = fcmp ult v10, v11 ; bin: 41 0f 2e ea 0f 92 c2 + ; asm: ucomiss %xmm5, %xmm10 + ; asm: setbe %dl + [-,%rdx] v307 = fcmp ule v11, v10 ; bin: 44 0f 2e d5 0f 96 c2 + return } @@ -326,5 +356,35 @@ ebb0: ; asm: movq 1032(%rsp), %xmm10 [-,%xmm10] v211 = fill v201 ; bin: f3 44 0f 7e 94 24 00000408 + ; Comparisons. + ; + ; Only `supported_floatccs` are tested here. Others are handled by + ; legalization paterns. + + ; asm: ucomisd %xmm10, %xmm5 + ; asm: setnp %bl + [-,%rbx] v300 = fcmp ord v10, v11 ; bin: 66 41 0f 2e ea 0f 9b c3 + ; asm: ucomisd %xmm5, %xmm10 + ; asm: setp %bl + [-,%rbx] v301 = fcmp uno v11, v10 ; bin: 66 44 0f 2e d5 0f 9a c3 + ; asm: ucomisd %xmm10, %xmm5 + ; asm: setne %dl + [-,%rdx] v302 = fcmp one v10, v11 ; bin: 66 41 0f 2e ea 0f 95 c2 + ; asm: ucomisd %xmm5, %xmm10 + ; asm: sete %dl + [-,%rdx] v303 = fcmp ueq v11, v10 ; bin: 66 44 0f 2e d5 0f 94 c2 + ; asm: ucomisd %xmm10, %xmm5 + ; asm: seta %bl + [-,%rbx] v304 = fcmp gt v10, v11 ; bin: 66 41 0f 2e ea 0f 97 c3 + ; asm: ucomisd %xmm5, %xmm10 + ; asm: setae %bl + [-,%rbx] v305 = fcmp ge v11, v10 ; bin: 66 44 0f 2e d5 0f 93 c3 + ; asm: ucomisd %xmm10, %xmm5 + ; asm: setb %dl + [-,%rdx] v306 = fcmp ult v10, v11 ; bin: 66 41 0f 2e ea 0f 92 c2 + ; asm: ucomisd %xmm5, %xmm10 + ; asm: setbe %dl + [-,%rdx] v307 = fcmp ule v11, v10 ; bin: 66 44 0f 2e d5 0f 96 c2 + return } diff --git a/cranelift/filetests/wasm/f32-compares.cton b/cranelift/filetests/wasm/f32-compares.cton new file mode 100644 index 0000000000..560b86ebcb --- /dev/null +++ b/cranelift/filetests/wasm/f32-compares.cton @@ -0,0 +1,50 @@ +; Test code generation for WebAssembly f32 comparison operators. +test compile + +set is_64bit=0 +isa intel haswell + +set is_64bit=1 +isa intel haswell + +function %f32_eq(f32, f32) -> i32 { +ebb0(v0: f32, v1: f32): + v2 = fcmp eq v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %f32_ne(f32, f32) -> i32 { +ebb0(v0: f32, v1: f32): + v2 = fcmp ne v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %f32_lt(f32, f32) -> i32 { +ebb0(v0: f32, v1: f32): + v2 = fcmp lt v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %f32_gt(f32, f32) -> i32 { +ebb0(v0: f32, v1: f32): + v2 = fcmp gt v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %f32_le(f32, f32) -> i32 { +ebb0(v0: f32, v1: f32): + v2 = fcmp le v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %f32_ge(f32, f32) -> i32 { +ebb0(v0: f32, v1: f32): + v2 = fcmp ge v0, v1 + v3 = bint.i32 v2 + return v3 +} diff --git a/cranelift/filetests/wasm/f64-compares.cton b/cranelift/filetests/wasm/f64-compares.cton new file mode 100644 index 0000000000..78a260ef27 --- /dev/null +++ b/cranelift/filetests/wasm/f64-compares.cton @@ -0,0 +1,50 @@ +; Test code generation for WebAssembly f64 comparison operators. +test compile + +set is_64bit=0 +isa intel haswell + +set is_64bit=1 +isa intel haswell + +function %f64_eq(f64, f64) -> i32 { +ebb0(v0: f64, v1: f64): + v2 = fcmp eq v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %f64_ne(f64, f64) -> i32 { +ebb0(v0: f64, v1: f64): + v2 = fcmp ne v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %f64_lt(f64, f64) -> i32 { +ebb0(v0: f64, v1: f64): + v2 = fcmp lt v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %f64_gt(f64, f64) -> i32 { +ebb0(v0: f64, v1: f64): + v2 = fcmp gt v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %f64_le(f64, f64) -> i32 { +ebb0(v0: f64, v1: f64): + v2 = fcmp le v0, v1 + v3 = bint.i32 v2 + return v3 +} + +function %f64_ge(f64, f64) -> i32 { +ebb0(v0: f64, v1: f64): + v2 = fcmp ge v0, v1 + v3 = bint.i32 v2 + return v3 +} diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 0d8407bb30..1b39918658 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -169,10 +169,12 @@ def unwrap_inst(iref, node, fmt): iform = expr.inst.format nvops = iform.num_value_operands - # The tuple of locals we're extracting is `expr.args`. + # The tuple of locals to extract is the `Var` instances in `expr.args`. + arg_names = tuple( + arg.name if isinstance(arg, Var) else '_' for arg in expr.args) with fmt.indented( 'let ({}, predicate) = if let ir::InstructionData::{} {{' - .format(', '.join(map(str, expr.args)), iform.name), '};'): + .format(', '.join(map(str, arg_names)), iform.name), '};'): # Fields are encoded directly. for f in iform.imm_fields: fmt.line('{},'.format(f.member)) diff --git a/lib/cretonne/meta/isa/intel/defs.py b/lib/cretonne/meta/isa/intel/defs.py index ad13741ebc..d5bb0b5a1f 100644 --- a/lib/cretonne/meta/isa/intel/defs.py +++ b/lib/cretonne/meta/isa/intel/defs.py @@ -7,9 +7,22 @@ from __future__ import absolute_import from cdsl.isa import TargetISA, CPUMode import base.instructions from . import instructions as x86 +from base.immediates import floatcc ISA = TargetISA('intel', [base.instructions.GROUP, x86.GROUP]) # CPU modes for 32-bit and 64-bit operation. I64 = CPUMode('I64', ISA) I32 = CPUMode('I32', ISA) + +# The set of floating point condition codes that are directly supported. +# Other condition codes need to be reversed or expressed as two tests. +supported_floatccs = [ + floatcc.ord, + floatcc.uno, + floatcc.one, + floatcc.ueq, + floatcc.gt, + floatcc.ge, + floatcc.ult, + floatcc.ule] diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 6d46111d7f..c7f9a7c0b6 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -26,8 +26,8 @@ I32.legalize_type( default=narrow, b1=expand, i32=intel_expand, - f32=expand, - f64=expand) + f32=intel_expand, + f64=intel_expand) I64.legalize_monomorphic(expand) I64.legalize_type( @@ -35,8 +35,8 @@ I64.legalize_type( b1=expand, i32=intel_expand, i64=intel_expand, - f32=expand, - f64=expand) + f32=intel_expand, + f64=intel_expand) # @@ -106,6 +106,13 @@ for inst, opc in [ (base.bxor, 0x31)]: enc_i32_i64(inst, r.rr, opc) +# Also add a `b1` encodings for the logic instructions. +# TODO: Should this be done with 8-bit instructions? It would improve +# partial register dependencies. +enc_flt(base.band.b1, r.rr, 0x21) +enc_flt(base.bor.b1, r.rr, 0x09) +enc_flt(base.bxor.b1, r.rr, 0x31) + enc_i32_i64(base.imul, r.rrx, 0x0f, 0xaf) enc_i32_i64(x86.sdivmodx, r.div, 0xf7, rrr=7) enc_i32_i64(x86.udivmodx, r.div, 0xf7, rrr=6) @@ -391,3 +398,10 @@ for inst, opc in [ (base.bxor, 0x57)]: enc_flt(inst.f32, r.frm, 0x0f, opc) enc_flt(inst.f64, r.frm, 0x0f, opc) + +# Comparisons. +# +# This only covers the condition codes in `supported_floatccs`, the rest are +# handled by legalization patterns. +enc_flt(base.fcmp.f32, r.fcscc, 0x0f, 0x2e) +enc_flt(base.fcmp.f64, r.fcscc, 0x66, 0x0f, 0x2e) diff --git a/lib/cretonne/meta/isa/intel/legalize.py b/lib/cretonne/meta/isa/intel/legalize.py index cc46846d81..6125dcd0e4 100644 --- a/lib/cretonne/meta/isa/intel/legalize.py +++ b/lib/cretonne/meta/isa/intel/legalize.py @@ -4,7 +4,7 @@ Custom legalization patterns for Intel. from __future__ import absolute_import from cdsl.ast import Var from cdsl.xform import Rtl, XFormGroup -from base.immediates import imm64 +from base.immediates import imm64, floatcc from base.types import i32, i64 from base import legalize as shared from base import instructions as insts @@ -25,6 +25,8 @@ dead = Var('dead') x = Var('x') xhi = Var('xhi') y = Var('y') +a1 = Var('a1') +a2 = Var('a2') # # Division and remainder. @@ -56,3 +58,37 @@ for ty in [i32, i64]: xhi << insts.sshr_imm(x, imm64(ty.lane_bits() - 1)), (dead, a) << x86.sdivmodx(x, xhi, y) )) + +# Floating point condition codes. +# +# The 8 condition codes in `supported_floatccs` are directly supported by a +# `ucomiss` or `ucomisd` instruction. The remaining codes need legalization +# patterns. + +# Equality needs an explicit `ord` test which checks the parity bit. +intel_expand.legalize( + a << insts.fcmp(floatcc.eq, x, y), + Rtl( + a1 << insts.fcmp(floatcc.ord, x, y), + a2 << insts.fcmp(floatcc.ueq, x, y), + a << insts.band(a1, a2) + )) +intel_expand.legalize( + a << insts.fcmp(floatcc.ne, x, y), + Rtl( + a1 << insts.fcmp(floatcc.uno, x, y), + a2 << insts.fcmp(floatcc.one, x, y), + a << insts.bor(a1, a2) + )) + +# Inequalities that need to be reversed. +for cc, rev_cc in [ + (floatcc.lt, floatcc.gt), + (floatcc.le, floatcc.ge), + (floatcc.ugt, floatcc.ult), + (floatcc.uge, floatcc.ule)]: + intel_expand.legalize( + a << insts.fcmp(cc, x, y), + Rtl( + a << insts.fcmp(rev_cc, y, x) + )) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 04c7c91891..74391b0746 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -3,12 +3,13 @@ Intel Encoding recipes. """ from __future__ import absolute_import from cdsl.isa import EncRecipe -from cdsl.predicates import IsSignedInt, IsEqual +from cdsl.predicates import IsSignedInt, IsEqual, Or from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry from base.formats import Trap, Call, IndirectCall, Store, Load -from base.formats import IntCompare +from base.formats import IntCompare, FloatCompare from base.formats import RegMove, Ternary, Jump, Branch, FuncAddr from .registers import GPR, ABCD, FPR, GPR8, FPR8, StackGPR32, StackFPR32 +from .defs import supported_floatccs try: from typing import Tuple, Dict, Sequence # noqa @@ -696,7 +697,7 @@ t8jccb_abcd = TailRecipe( # This bandaid macro doesn't support a REX prefix for the final `setCC` # instruction, so it is limited to the `ABCD` register class for booleans. icscc = TailRecipe( - 'cscc', IntCompare, size=1 + 3, ins=(GPR, GPR), outs=ABCD, + 'icscc', IntCompare, size=1 + 3, ins=(GPR, GPR), outs=ABCD, emit=''' // Comparison instruction. PUT_OP(bits, rex2(in_reg0, in_reg1), sink); @@ -719,3 +720,49 @@ icscc = TailRecipe( sink.put1(setcc); modrm_rr(out_reg0, 0, sink); ''') + + +# Make a FloatCompare instruction predicate with the supported condition codes. + +# Same thing for floating point. +# +# The ucomiss/ucomisd instructions set the EFLAGS bits CF/PF/CF like this: +# +# ZPC OSA +# UN 111 000 +# GT 000 000 +# LT 001 000 +# EQ 100 000 +# +# Not all floating point condition codes are supported. +fcscc = TailRecipe( + 'fcscc', FloatCompare, size=1 + 3, ins=(FPR, FPR), outs=ABCD, + instp=Or(*(IsEqual(FloatCompare.cond, cc) + for cc in supported_floatccs)), + emit=''' + // Comparison instruction. + PUT_OP(bits, rex2(in_reg1, in_reg0), sink); + modrm_rr(in_reg1, in_reg0, sink); + // `setCC` instruction, no REX. + use ir::condcodes::FloatCC::*; + let setcc = match cond { + Ordered => 0x9b, // EQ|LT|GT => setnp (P=0) + Unordered => 0x9a, // UN => setp (P=1) + OrderedNotEqual => 0x95, // LT|GT => setne (Z=0), + UnorderedOrEqual => 0x94, // UN|EQ => sete (Z=1) + GreaterThan => 0x97, // GT => seta (C=0&Z=0) + GreaterThanOrEqual => 0x93, // GT|EQ => setae (C=0) + UnorderedOrLessThan => 0x92, // UN|LT => setb (C=1) + UnorderedOrLessThanOrEqual => 0x96, // UN|LT|EQ => setbe (Z=1|C=1) + Equal | // EQ + NotEqual | // UN|LT|GT + LessThan | // LT + LessThanOrEqual | // LT|EQ + UnorderedOrGreaterThan | // UN|GT + UnorderedOrGreaterThanOrEqual // UN|GT|EQ + => panic!("{} not supported by fcscc", cond), + }; + sink.put1(0x0f); + sink.put1(setcc); + modrm_rr(out_reg0, 0, sink); + ''') From ce767be70333e9dd617205f91f32fb117146cafa Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 26 Sep 2017 13:54:19 -0700 Subject: [PATCH 1170/3084] Intel encodings for floating point copies. --- cranelift/filetests/isa/intel/binary32-float.cton | 10 ++++++++++ cranelift/filetests/isa/intel/binary64-float.cton | 10 ++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 4 ++++ 3 files changed, 24 insertions(+) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index b0a2c83c8f..0d823e2dae 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -37,6 +37,11 @@ ebb0: ; asm: movd %xmm2, %esi [-,%rsi] v17 = bitcast.i32 v11 ; bin: 66 0f 7e d6 + ; asm: movaps %xmm2, %xmm5 + [-,%xmm5] v18 = copy v11 ; bin: 0f 28 ea + ; asm: movaps %xmm5, %xmm2 + [-,%xmm2] v19 = copy v10 ; bin: 0f 28 d5 + ; Binary arithmetic. ; asm: addss %xmm2, %xmm5 @@ -212,6 +217,11 @@ ebb0: ; No i64 <-> f64 bitcasts in 32-bit mode. + ; asm: movaps %xmm2, %xmm5 + [-,%xmm5] v18 = copy v11 ; bin: 0f 28 ea + ; asm: movaps %xmm5, %xmm2 + [-,%xmm2] v19 = copy v10 ; bin: 0f 28 d5 + ; Binary arithmetic. ; asm: addsd %xmm2, %xmm5 diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index 542a712f26..12a26c937e 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -91,6 +91,11 @@ ebb0: ; asm: xorps %xmm5, %xmm10 [-,%xmm10] v37 = bxor v11, v10 ; bin: 44 0f 57 d5 + ; asm: movaps %xmm10, %xmm5 + [-,%xmm5] v38 = copy v11 ; bin: 41 0f 28 ea + ; asm: movaps %xmm5, %xmm10 + [-,%xmm10] v39 = copy v10 ; bin: 44 0f 28 d5 + ; Unary arithmetic. ; asm: sqrtss %xmm5, %xmm10 @@ -281,6 +286,11 @@ ebb0: ; asm: xorps %xmm5, %xmm10 [-,%xmm10] v37 = bxor v11, v10 ; bin: 44 0f 57 d5 + ; asm: movaps %xmm10, %xmm5 + [-,%xmm5] v38 = copy v11 ; bin: 41 0f 28 ea + ; asm: movaps %xmm5, %xmm10 + [-,%xmm10] v39 = copy v10 ; bin: 44 0f 28 d5 + ; Unary arithmetic. ; asm: sqrtsd %xmm5, %xmm10 diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index c7f9a7c0b6..5f295eddf2 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -355,6 +355,10 @@ enc_flt(base.bitcast.i32.f32, r.rfumr, 0x66, 0x0f, 0x7e) I64.enc(base.bitcast.f64.i64, *r.frurm.rex(0x66, 0x0f, 0x6e, w=1)) I64.enc(base.bitcast.i64.f64, *r.rfumr.rex(0x66, 0x0f, 0x7e, w=1)) +# movaps +enc_flt(base.copy.f32, r.furm, 0x0f, 0x28) +enc_flt(base.copy.f64, r.furm, 0x0f, 0x28) + # cvtsi2ss enc_i32_i64(base.fcvt_from_sint.f32, r.frurm, 0xf3, 0x0f, 0x2a) From 6ff681a90d0b49c7088e38c4aad0dc14e4719de7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 26 Sep 2017 13:38:06 -0700 Subject: [PATCH 1171/3084] Add general legalization for the select instruction. --- .../filetests/isa/intel/legalize-custom.cton | 10 ++++++ cranelift/filetests/wasm/select.cton | 32 +++++++++++++++++ lib/cretonne/meta/base/legalize.py | 13 ++++++- lib/cretonne/src/legalizer/mod.rs | 35 +++++++++++++++++++ 4 files changed, 89 insertions(+), 1 deletion(-) create mode 100644 cranelift/filetests/wasm/select.cton diff --git a/cranelift/filetests/isa/intel/legalize-custom.cton b/cranelift/filetests/isa/intel/legalize-custom.cton index 39b92c6306..8c9e898586 100644 --- a/cranelift/filetests/isa/intel/legalize-custom.cton +++ b/cranelift/filetests/isa/intel/legalize-custom.cton @@ -68,3 +68,13 @@ ebb0: ; check: $v1 = bitcast.f64 $tmp return v1 } + +function %select_f64(f64, f64, i32) -> f64 { +ebb0(v0: f64, v1: f64, v2: i32): + v3 = select v2, v0, v1 + ; check: brnz v2, $(new=$EBB)($v0) + ; nextln: jump $new($v1) + ; check: $new($v3: f64): + ; nextln: return $v3 + return v3 +} diff --git a/cranelift/filetests/wasm/select.cton b/cranelift/filetests/wasm/select.cton new file mode 100644 index 0000000000..fc08c92a8a --- /dev/null +++ b/cranelift/filetests/wasm/select.cton @@ -0,0 +1,32 @@ +; Test basic code generation for the select WebAssembly instruction. +test compile + +set is_64bit=0 +isa intel haswell + +set is_64bit=1 +isa intel haswell + +function %select_i32(i32, i32, i32) -> i32 { +ebb0(v0: i32, v1: i32, v2: i32): + v3 = select v2, v0, v1 + return v3 +} + +function %select_i64(i64, i64, i32) -> i64 { +ebb0(v0: i64, v1: i64, v2: i32): + v3 = select v2, v0, v1 + return v3 +} + +function %select_f32(f32, f32, i32) -> f32 { +ebb0(v0: f32, v1: f32, v2: i32): + v3 = select v2, v0, v1 + return v3 +} + +function %select_f64(f64, f64, i32) -> f64 { +ebb0(v0: f64, v1: f64, v2: i32): + v3 = select v2, v0, v1 + return v3 +} diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index 8608feb392..ebcda3fee1 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -16,7 +16,7 @@ from .instructions import imul, imul_imm from .instructions import band, bor, bxor, isplit, iconcat from .instructions import bnot, band_not, bor_not, bxor_not from .instructions import icmp, icmp_imm -from .instructions import iconst, bint +from .instructions import iconst, bint, select from .instructions import ishl, ishl_imm, sshr, sshr_imm, ushr, ushr_imm from .instructions import rotl, rotl_imm, rotr, rotr_imm from .instructions import f32const, f64const @@ -56,6 +56,7 @@ expand.custom_legalize(insts.heap_addr, 'expand_heap_addr') expand.custom_legalize(insts.trapz, 'expand_cond_trap') expand.custom_legalize(insts.trapnz, 'expand_cond_trap') expand.custom_legalize(insts.br_table, 'expand_br_table') +expand.custom_legalize(insts.select, 'expand_select') # Custom expansions for floating point constants. # These expansions require bit-casting or creating constant pool entries. @@ -116,6 +117,16 @@ for bitop in [band, bor, bxor]: a << iconcat(al, ah) )) +narrow.legalize( + a << select(c, x, y), + Rtl( + (xl, xh) << isplit(x), + (yl, yh) << isplit(y), + al << select(c, xl, yl), + ah << select(c, xh, yh), + a << iconcat(al, ah) + )) + # Expand integer operations with carry for RISC architectures that don't have # the flags. expand.legalize( diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 61a433b6ed..a56de36a3c 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -179,6 +179,41 @@ fn expand_br_table(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlo cfg.recompute_ebb(pos.func, ebb); } +/// Expand the select instruction. +/// +/// Conditional moves are available in some ISAs for some register classes. The remaining selects +/// are handled by a branch. +fn expand_select(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { + let (ctrl, tval, fval) = match func.dfg[inst] { + ir::InstructionData::Ternary { + opcode: ir::Opcode::Select, + args, + } => (args[0], args[1], args[2]), + _ => panic!("Expected select: {}", func.dfg.display_inst(inst, None)), + }; + + // Replace `result = select ctrl, tval, fval` with: + // + // brnz ctrl, new_ebb(tval) + // jump new_ebb(fval) + // new_ebb(result): + let old_ebb = func.layout.pp_ebb(inst); + let result = func.dfg.first_result(inst); + func.dfg.clear_results(inst); + let new_ebb = func.dfg.make_ebb(); + func.dfg.attach_ebb_arg(new_ebb, result); + + func.dfg.replace(inst).brnz(ctrl, new_ebb, &[tval]); + let mut pos = FuncCursor::new(func).after_inst(inst); + pos.use_srcloc(inst); + pos.ins().jump(new_ebb, &[fval]); + pos.insert_ebb(new_ebb); + + cfg.recompute_ebb(pos.func, new_ebb); + cfg.recompute_ebb(pos.func, old_ebb); +} + + /// Expand illegal `f32const` and `f64const` instructions. fn expand_fconst(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlowGraph) { let ty = func.dfg.value_type(func.dfg.first_result(inst)); From d13f29cfe4716f00dc57f0aa49733f7112fced21 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 26 Sep 2017 14:56:33 -0700 Subject: [PATCH 1172/3084] Update to wasmparser 0.11.2. --- lib/wasm/Cargo.toml | 2 +- lib/wasm/src/sections_translator.rs | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 1bd44219c7..97c63ff367 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -11,7 +11,7 @@ license = "Apache-2.0" name = "cton_wasm" [dependencies] -wasmparser = "0.10.0" +wasmparser = "0.11.2" cretonne = { path = "../cretonne" } cretonne-frontend = { path = "../frontend" } diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 08f05eca79..7c0f794b3e 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -88,7 +88,7 @@ pub fn parse_import_section( } => { imports.push(Import::Global(Global { ty: type_to_type(&ty.content_type).unwrap(), - mutability: ty.mutability != 0, + mutability: ty.mutable, initializer: GlobalInit::Import(), })); } @@ -185,7 +185,7 @@ pub fn parse_global_section( let mut globals = Vec::new(); loop { let (content_type, mutability) = match *parser.read() { - ParserState::BeginGlobalSectionEntry(ref ty) => (ty.content_type, ty.mutability), + ParserState::BeginGlobalSectionEntry(ref ty) => (ty.content_type, ty.mutable), ParserState::EndSection => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; @@ -218,7 +218,7 @@ pub fn parse_global_section( } let global = Global { ty: type_to_type(&content_type).unwrap(), - mutability: mutability != 0, + mutability: mutability, initializer: initializer, }; runtime.declare_global(global); From ac69f3bfdff12a686b860cffda965f98a3ede1df Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 26 Sep 2017 15:35:32 -0700 Subject: [PATCH 1173/3084] Add an Intel-specific x86_cvtt2si instruction. This is used to represent the non-trapping semantics of the cvttss2si and cvttsd2si instructions (and their vectorized counterparts). The overflow behavior of this instruction is specific to the Intel ISAs. There is no float-to-i64 instruction on the 32-bit Intel ISA. --- .../filetests/isa/intel/binary32-float.cton | 14 +++++++++++ .../filetests/isa/intel/binary64-float.cton | 24 +++++++++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 8 +++++++ lib/cretonne/meta/isa/intel/instructions.py | 23 ++++++++++++++++++ lib/cretonne/meta/isa/intel/recipes.py | 8 +++++++ 5 files changed, 77 insertions(+) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 0d823e2dae..9358fc338e 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -87,6 +87,13 @@ ebb0: ; asm: xorps %xmm5, %xmm2 [-,%xmm2] v37 = bxor v11, v10 ; bin: 0f 57 d5 + ; Convert float to int. (No i64 dest on i386). + + ; asm: cvttss2si %xmm5, %ecx + [-,%rcx] v40 = x86_cvtt2si.i32 v10 ; bin: f3 0f 2c cd + ; asm: cvttss2si %xmm2, %esi + [-,%rsi] v41 = x86_cvtt2si.i32 v11 ; bin: f3 0f 2c f2 + ; Unary arithmetic. ; asm: sqrtss %xmm5, %xmm2 @@ -267,6 +274,13 @@ ebb0: ; asm: xorps %xmm5, %xmm2 [-,%xmm2] v37 = bxor v11, v10 ; bin: 0f 57 d5 + ; Convert float to int. (No i64 dest on i386). + + ; asm: cvttsd2si %xmm5, %ecx + [-,%rcx] v40 = x86_cvtt2si.i32 v10 ; bin: f2 0f 2c cd + ; asm: cvttsd2si %xmm2, %esi + [-,%rsi] v41 = x86_cvtt2si.i32 v11 ; bin: f2 0f 2c f2 + ; Unary arithmetic. ; asm: sqrtsd %xmm5, %xmm2 diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index 12a26c937e..5763a5ab67 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -96,6 +96,18 @@ ebb0: ; asm: movaps %xmm5, %xmm10 [-,%xmm10] v39 = copy v10 ; bin: 44 0f 28 d5 + ; Convert float to int. + + ; asm: cvttss2si %xmm5, %ecx + [-,%rcx] v40 = x86_cvtt2si.i32 v10 ; bin: f3 0f 2c cd + ; asm: cvttss2si %xmm10, %esi + [-,%rsi] v41 = x86_cvtt2si.i32 v11 ; bin: f3 41 0f 2c f2 + + ; asm: cvttss2si %xmm5, %rcx + [-,%rcx] v42 = x86_cvtt2si.i64 v10 ; bin: f3 48 0f 2c cd + ; asm: cvttss2si %xmm10, %rsi + [-,%rsi] v43 = x86_cvtt2si.i64 v11 ; bin: f3 49 0f 2c f2 + ; Unary arithmetic. ; asm: sqrtss %xmm5, %xmm10 @@ -291,6 +303,18 @@ ebb0: ; asm: movaps %xmm5, %xmm10 [-,%xmm10] v39 = copy v10 ; bin: 44 0f 28 d5 + ; Convert float to int. + + ; asm: cvttsd2si %xmm5, %ecx + [-,%rcx] v40 = x86_cvtt2si.i32 v10 ; bin: f2 0f 2c cd + ; asm: cvttsd2si %xmm10, %esi + [-,%rsi] v41 = x86_cvtt2si.i32 v11 ; bin: f2 41 0f 2c f2 + + ; asm: cvttsd2si %xmm5, %rcx + [-,%rcx] v42 = x86_cvtt2si.i64 v10 ; bin: f2 48 0f 2c cd + ; asm: cvttsd2si %xmm10, %rsi + [-,%rsi] v43 = x86_cvtt2si.i64 v11 ; bin: f2 49 0f 2c f2 + ; Unary arithmetic. ; asm: sqrtsd %xmm5, %xmm10 diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 5f295eddf2..9eb832572d 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -371,6 +371,14 @@ enc_flt(base.fpromote.f64.f32, r.furm, 0xf3, 0x0f, 0x5a) # cvtsd2ss enc_flt(base.fdemote.f32.f64, r.furm, 0xf2, 0x0f, 0x5a) +# cvttss2si +enc_flt(x86.cvtt2si.i32.f32, r.rfurm, 0xf3, 0x0f, 0x2c) +I64.enc(x86.cvtt2si.i64.f32, *r.rfurm.rex(0xf3, 0x0f, 0x2c, w=1)) + +# cvttsd2si +enc_flt(x86.cvtt2si.i32.f64, r.rfurm, 0xf2, 0x0f, 0x2c) +I64.enc(x86.cvtt2si.i64.f64, *r.rfurm.rex(0xf2, 0x0f, 0x2c, w=1)) + # Exact square roots. enc_flt(base.sqrt.f32, r.furm, 0xf3, 0x0f, 0x51) enc_flt(base.sqrt.f64, r.furm, 0xf2, 0x0f, 0x51) diff --git a/lib/cretonne/meta/isa/intel/instructions.py b/lib/cretonne/meta/isa/intel/instructions.py index 7921b2f431..c9739ccecb 100644 --- a/lib/cretonne/meta/isa/intel/instructions.py +++ b/lib/cretonne/meta/isa/intel/instructions.py @@ -46,4 +46,27 @@ sdivmodx = Instruction( """, ins=(nlo, nhi, d), outs=(q, r), can_trap=True) + +Float = TypeVar( + 'Float', 'A scalar or vector floating point number', + floats=True, simd=True) +IntTo = TypeVar( + 'IntTo', 'An integer type with the same number of lanes', + ints=(32, 64), simd=True) + +x = Operand('x', Float) +a = Operand('a', IntTo) + +cvtt2si = Instruction( + 'x86_cvtt2si', r""" + Convert with truncation floating point to signed integer. + + The source floating point operand is converted to a signed integer by + rounding towards zero. If the result can't be represented in the output + type, returns the smallest signed value the output type can represent. + + This instruction does not trap. + """, + ins=x, outs=a) + GROUP.close() diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 74391b0746..a5ead7e6fd 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -290,6 +290,14 @@ frurm = TailRecipe( modrm_rr(in_reg0, out_reg0, sink); ''') +# XX /r, RM form, FPR -> GPR. +rfurm = TailRecipe( + 'rfurm', Unary, size=1, ins=FPR, outs=GPR, + emit=''' + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); + modrm_rr(in_reg0, out_reg0, sink); + ''') + # XX /r, RMI form for one of the roundXX SSE 4.1 instructions. furmi_rnd = TailRecipe( 'furmi_rnd', Unary, size=2, ins=FPR, outs=FPR, From 1fe789070046eac36c655c151c86992bd71e3ac5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Sep 2017 09:17:09 -0700 Subject: [PATCH 1174/3084] Add x86_fmin and x86_fmax instructions. These Intel-specific instructions represent the semantics of the minss / maxss Intel instructions which behave more like a C ternary operator than the WebAssembly fmin and fmax instructions. They will be used as building blocks for implementing the WebAssembly semantics. --- .../filetests/isa/intel/binary32-float.cton | 22 ++++++++++++++ .../filetests/isa/intel/binary64-float.cton | 22 ++++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 4 ++- lib/cretonne/meta/isa/intel/instructions.py | 30 +++++++++++++++++++ 4 files changed, 77 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 9358fc338e..bb1df58602 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -94,6 +94,17 @@ ebb0: ; asm: cvttss2si %xmm2, %esi [-,%rsi] v41 = x86_cvtt2si.i32 v11 ; bin: f3 0f 2c f2 + ; Min/max. + + ; asm: minss %xmm2, %xmm5 + [-,%xmm5] v42 = x86_fmin v10, v11 ; bin: f3 0f 5d ea + ; asm: minss %xmm5, %xmm2 + [-,%xmm2] v43 = x86_fmin v11, v10 ; bin: f3 0f 5d d5 + ; asm: maxss %xmm2, %xmm5 + [-,%xmm5] v44 = x86_fmax v10, v11 ; bin: f3 0f 5f ea + ; asm: maxss %xmm5, %xmm2 + [-,%xmm2] v45 = x86_fmax v11, v10 ; bin: f3 0f 5f d5 + ; Unary arithmetic. ; asm: sqrtss %xmm5, %xmm2 @@ -281,6 +292,17 @@ ebb0: ; asm: cvttsd2si %xmm2, %esi [-,%rsi] v41 = x86_cvtt2si.i32 v11 ; bin: f2 0f 2c f2 + ; Min/max. + + ; asm: minsd %xmm2, %xmm5 + [-,%xmm5] v42 = x86_fmin v10, v11 ; bin: f2 0f 5d ea + ; asm: minsd %xmm5, %xmm2 + [-,%xmm2] v43 = x86_fmin v11, v10 ; bin: f2 0f 5d d5 + ; asm: maxsd %xmm2, %xmm5 + [-,%xmm5] v44 = x86_fmax v10, v11 ; bin: f2 0f 5f ea + ; asm: maxsd %xmm5, %xmm2 + [-,%xmm2] v45 = x86_fmax v11, v10 ; bin: f2 0f 5f d5 + ; Unary arithmetic. ; asm: sqrtsd %xmm5, %xmm2 diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index 5763a5ab67..2156ebc5db 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -108,6 +108,17 @@ ebb0: ; asm: cvttss2si %xmm10, %rsi [-,%rsi] v43 = x86_cvtt2si.i64 v11 ; bin: f3 49 0f 2c f2 + ; Min/max. + + ; asm: minss %xmm10, %xmm5 + [-,%xmm5] v44 = x86_fmin v10, v11 ; bin: f3 41 0f 5d ea + ; asm: minss %xmm5, %xmm10 + [-,%xmm10] v45 = x86_fmin v11, v10 ; bin: f3 44 0f 5d d5 + ; asm: maxss %xmm10, %xmm5 + [-,%xmm5] v46 = x86_fmax v10, v11 ; bin: f3 41 0f 5f ea + ; asm: maxss %xmm5, %xmm10 + [-,%xmm10] v47 = x86_fmax v11, v10 ; bin: f3 44 0f 5f d5 + ; Unary arithmetic. ; asm: sqrtss %xmm5, %xmm10 @@ -315,6 +326,17 @@ ebb0: ; asm: cvttsd2si %xmm10, %rsi [-,%rsi] v43 = x86_cvtt2si.i64 v11 ; bin: f2 49 0f 2c f2 + ; Min/max. + + ; asm: minsd %xmm10, %xmm5 + [-,%xmm5] v44 = x86_fmin v10, v11 ; bin: f2 41 0f 5d ea + ; asm: minsd %xmm5, %xmm10 + [-,%xmm10] v45 = x86_fmin v11, v10 ; bin: f2 44 0f 5d d5 + ; asm: maxsd %xmm10, %xmm5 + [-,%xmm5] v46 = x86_fmax v10, v11 ; bin: f2 41 0f 5f ea + ; asm: maxsd %xmm5, %xmm10 + [-,%xmm10] v47 = x86_fmax v11, v10 ; bin: f2 44 0f 5f d5 + ; Unary arithmetic. ; asm: sqrtsd %xmm5, %xmm10 diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 9eb832572d..c647d5bf28 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -398,7 +398,9 @@ for inst, opc in [ (base.fadd, 0x58), (base.fsub, 0x5c), (base.fmul, 0x59), - (base.fdiv, 0x5e)]: + (base.fdiv, 0x5e), + (x86.fmin, 0x5d), + (x86.fmax, 0x5f)]: enc_flt(inst.f32, r.frm, 0xf3, 0x0f, opc) enc_flt(inst.f64, r.frm, 0xf2, 0x0f, opc) diff --git a/lib/cretonne/meta/isa/intel/instructions.py b/lib/cretonne/meta/isa/intel/instructions.py index c9739ccecb..27a0063557 100644 --- a/lib/cretonne/meta/isa/intel/instructions.py +++ b/lib/cretonne/meta/isa/intel/instructions.py @@ -69,4 +69,34 @@ cvtt2si = Instruction( """, ins=x, outs=a) +x = Operand('x', Float) +a = Operand('a', Float) +y = Operand('y', Float) + +fmin = Instruction( + 'x86_fmin', r""" + Floating point minimum with Intel semantics. + + This is equivalent to the C ternary operator `x < y ? x : y` which + differs from :inst:`fmin` when either operand is NaN or when comparing + +0.0 to -0.0. + + When the two operands don't compare as LT, `y` is returned unchanged, + even if it is a signalling NaN. + """, + ins=(x, y), outs=a) + +fmax = Instruction( + 'x86_fmax', r""" + Floating point maximum with Intel semantics. + + This is equivalent to the C ternary operator `x > y ? x : y` which + differs from :inst:`fmax` when either operand is NaN or when comparing + +0.0 to -0.0. + + When the two operands don't compare as GT, `y` is returned unchanged, + even if it is a signalling NaN. + """, + ins=(x, y), outs=a) + GROUP.close() From 44eab3e15855f66ad274e6699c6bfa4851c01487 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Sep 2017 12:36:27 -0700 Subject: [PATCH 1175/3084] Add Intel regmove encodings for floating point types. --- cranelift/filetests/isa/intel/binary32-float.cton | 10 ++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 2 ++ lib/cretonne/meta/isa/intel/recipes.py | 8 ++++++++ 3 files changed, 20 insertions(+) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index bb1df58602..4f38807c9e 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -42,6 +42,11 @@ ebb0: ; asm: movaps %xmm5, %xmm2 [-,%xmm2] v19 = copy v10 ; bin: 0f 28 d5 + ; asm: movaps %xmm2, %xmm5 + regmove v19, %xmm2 -> %xmm5 ; bin: 0f 28 ea + ; asm: movaps %xmm5, %xmm2 + regmove v19, %xmm5 -> %xmm2 ; bin: 0f 28 d5 + ; Binary arithmetic. ; asm: addss %xmm2, %xmm5 @@ -240,6 +245,11 @@ ebb0: ; asm: movaps %xmm5, %xmm2 [-,%xmm2] v19 = copy v10 ; bin: 0f 28 d5 + ; asm: movaps %xmm2, %xmm5 + regmove v19, %xmm2 -> %xmm5 ; bin: 0f 28 ea + ; asm: movaps %xmm5, %xmm2 + regmove v19, %xmm5 -> %xmm2 ; bin: 0f 28 d5 + ; Binary arithmetic. ; asm: addsd %xmm2, %xmm5 diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index c647d5bf28..f1d6a28900 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -358,6 +358,8 @@ I64.enc(base.bitcast.i64.f64, *r.rfumr.rex(0x66, 0x0f, 0x7e, w=1)) # movaps enc_flt(base.copy.f32, r.furm, 0x0f, 0x28) enc_flt(base.copy.f64, r.furm, 0x0f, 0x28) +enc_flt(base.regmove.f32, r.frmov, 0x0f, 0x28) +enc_flt(base.regmove.f64, r.frmov, 0x0f, 0x28) # cvtsi2ss enc_i32_i64(base.fcvt_from_sint.f32, r.frurm, 0xf3, 0x0f, 0x2a) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index a5ead7e6fd..74d7757474 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -321,6 +321,14 @@ rmov = TailRecipe( modrm_rr(dst, src, sink); ''') +# XX /r, for regmove instructions (FPR version, RM encoded). +frmov = TailRecipe( + 'frmov', RegMove, size=1, ins=FPR, outs=(), + emit=''' + PUT_OP(bits, rex2(src, dst), sink); + modrm_rr(src, dst, sink); + ''') + # XX /n with one arg in %rcx, for shifts. rc = TailRecipe( 'rc', Binary, size=1, ins=(GPR, GPR.rcx), outs=0, From 384b04b4117c7c0fb28de92d8d01cc8a70e8f18d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Sep 2017 12:48:48 -0700 Subject: [PATCH 1176/3084] Fix some misnamed TailRecipes and add a consistency check. --- lib/cretonne/meta/isa/intel/recipes.py | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 74d7757474..eebba58700 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -12,7 +12,7 @@ from .registers import GPR, ABCD, FPR, GPR8, FPR8, StackGPR32, StackFPR32 from .defs import supported_floatccs try: - from typing import Tuple, Dict, Sequence # noqa + from typing import Tuple, Dict, Sequence, Any # noqa from cdsl.instructions import InstructionFormat # noqa from cdsl.isa import ConstraintSeq, BranchRange, PredNode, OperandConstraint # noqa except ImportError: @@ -206,6 +206,13 @@ class TailRecipe: emit=replace_put_op(self.emit, name)) return (self.recipes[name], bits) + @staticmethod + def check_names(globs): + # type: (Dict[str, Any]) -> None + for name, obj in globs.items(): + if isinstance(obj, TailRecipe): + assert name == obj.name, "Mismatched TailRecipe name: " + name + # A null unary instruction that takes a GPR register. Can be used for identity # copies and no-op conversions. @@ -234,7 +241,7 @@ rrx = TailRecipe( # XX /r with FPR ins and outs. RM form. frm = TailRecipe( - 'frr', Binary, size=1, ins=(FPR, FPR), outs=0, + 'frm', Binary, size=1, ins=(FPR, FPR), outs=0, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_rr(in_reg1, in_reg0, sink); @@ -315,7 +322,7 @@ furmi_rnd = TailRecipe( # XX /r, for regmove instructions. rmov = TailRecipe( - 'ur', RegMove, size=1, ins=GPR, outs=(), + 'rmov', RegMove, size=1, ins=GPR, outs=(), emit=''' PUT_OP(bits, rex2(dst, src), sink); modrm_rr(dst, src, sink); @@ -381,7 +388,7 @@ uid = TailRecipe( # XX+rd id unary with 32-bit immediate. Note no recipe predicate. puid = TailRecipe( - 'uid', UnaryImm, size=4, ins=(), outs=GPR, + 'puid', UnaryImm, size=4, ins=(), outs=GPR, emit=''' // The destination register is encoded in the low bits of the opcode. // No ModR/M. @@ -392,7 +399,7 @@ puid = TailRecipe( # XX+rd iq unary with 64-bit immediate. puiq = TailRecipe( - 'uiq', UnaryImm, size=8, ins=(), outs=GPR, + 'puiq', UnaryImm, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); let imm: i64 = imm.into(); @@ -669,7 +676,7 @@ jmpd = TailRecipe( # Bits 0-7 are the Jcc opcode. # Bits 8-15 control the test instruction which always has opcode byte 0x85. tjccb = TailRecipe( - 'tjcc', Branch, size=1 + 2, ins=GPR, outs=(), + 'tjccb', Branch, size=1 + 2, ins=GPR, outs=(), branch_range=(2, 8), emit=''' // test r, r. @@ -782,3 +789,5 @@ fcscc = TailRecipe( sink.put1(setcc); modrm_rr(out_reg0, 0, sink); ''') + +TailRecipe.check_names(globals()) From b6b474a8c982f40adeb0e25016abb32ab58d465a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Sep 2017 11:02:57 -0700 Subject: [PATCH 1177/3084] Add Intel legalization for fmin and fmax. The native x86_fmin and x86_fmax instructions don't behave correctly for NaN inputs and when comparing +0.0 to -0.0, so we need separate branches for those cases. --- .../filetests/isa/intel/legalize-custom.cton | 19 ++++ cranelift/filetests/wasm/f32-arith.cton | 13 ++- cranelift/filetests/wasm/f64-arith.cton | 13 ++- lib/cretonne/meta/isa/intel/legalize.py | 4 + lib/cretonne/src/isa/intel/enc_tables.rs | 92 ++++++++++++++++++- 5 files changed, 136 insertions(+), 5 deletions(-) diff --git a/cranelift/filetests/isa/intel/legalize-custom.cton b/cranelift/filetests/isa/intel/legalize-custom.cton index 8c9e898586..94aa7ab1d2 100644 --- a/cranelift/filetests/isa/intel/legalize-custom.cton +++ b/cranelift/filetests/isa/intel/legalize-custom.cton @@ -78,3 +78,22 @@ ebb0(v0: f64, v1: f64, v2: i32): ; nextln: return $v3 return v3 } + +function %f32_min(f32, f32) -> f32 { +ebb0(v0: f32, v1: f32): + v2 = fmin v0, v1 + return v2 + ; check: $(vnat=$V) = x86_fmin $v0, $v1 + ; nextln: jump $(done=$EBB)($vnat) + + ; check: $(uno=$EBB): + ; nextln: $(vuno=$V) = fadd.f32 $v0, $v1 + ; nextln: jump $(done=$EBB)($vuno) + + ; check: $(ueq=$EBB): + ; check: $(veq=$V) = bor.f32 $v0, $v1 + ; nextln: jump $(done=$EBB)($veq) + + ; check: $done($v2: f32): + ; nextln: return $v2 +} diff --git a/cranelift/filetests/wasm/f32-arith.cton b/cranelift/filetests/wasm/f32-arith.cton index ddb05138e9..331f9c7fad 100644 --- a/cranelift/filetests/wasm/f32-arith.cton +++ b/cranelift/filetests/wasm/f32-arith.cton @@ -85,8 +85,17 @@ ebb0(v0: f32, v1: f32): return v2 } -; function %f32_min(f32, f32) -> f32 -; function %f32_max(f32, f32) -> f32 +function %f32_min(f32, f32) -> f32 { +ebb0(v0: f32, v1: f32): + v2 = fmin v0, v1 + return v2 +} + +function %f32_max(f32, f32) -> f32 { +ebb0(v0: f32, v1: f32): + v2 = fmax v0, v1 + return v2 +} function %f32_copysign(f32, f32) -> f32 { ebb0(v0: f32, v1: f32): diff --git a/cranelift/filetests/wasm/f64-arith.cton b/cranelift/filetests/wasm/f64-arith.cton index d67066e615..4f39fb6a5f 100644 --- a/cranelift/filetests/wasm/f64-arith.cton +++ b/cranelift/filetests/wasm/f64-arith.cton @@ -82,8 +82,17 @@ ebb0(v0: f64, v1: f64): return v2 } -; function %f64_min(f64, f64) -> f64 -; function %f64_max(f64, f64) -> f64 +function %f64_min(f64, f64) -> f64 { +ebb0(v0: f64, v1: f64): + v2 = fmin v0, v1 + return v2 +} + +function %f64_max(f64, f64) -> f64 { +ebb0(v0: f64, v1: f64): + v2 = fmax v0, v1 + return v2 +} function %f64_copysign(f64, f64) -> f64 { ebb0(v0: f64, v1: f64): diff --git a/lib/cretonne/meta/isa/intel/legalize.py b/lib/cretonne/meta/isa/intel/legalize.py index 6125dcd0e4..8d93337c69 100644 --- a/lib/cretonne/meta/isa/intel/legalize.py +++ b/lib/cretonne/meta/isa/intel/legalize.py @@ -92,3 +92,7 @@ for cc, rev_cc in [ Rtl( a << insts.fcmp(rev_cc, y, x) )) + +# We need to modify the CFG for min/max legalization. +intel_expand.custom_legalize(insts.fmin, 'expand_minmax') +intel_expand.custom_legalize(insts.fmax, 'expand_minmax') diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 35ca9a9d76..4c8c428bfc 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -1,7 +1,9 @@ //! Encoding tables for Intel ISAs. use bitset::BitSet; -use ir; +use cursor::{Cursor, FuncCursor}; +use flowgraph::ControlFlowGraph; +use ir::{self, InstBuilder}; use isa::constraints::*; use isa::enc_tables::*; use isa::encoding::RecipeSizing; @@ -11,3 +13,91 @@ use super::registers::*; include!(concat!(env!("OUT_DIR"), "/encoding-intel.rs")); include!(concat!(env!("OUT_DIR"), "/legalize-intel.rs")); + +/// Expand the `fmin` and `fmax` instructions using the Intel `x86_fmin` and `x86_fmax` +/// instructions. +fn expand_minmax(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { + use ir::condcodes::FloatCC; + + let (x, y, x86_opc, bitwise_opc) = match func.dfg[inst] { + ir::InstructionData::Binary { + opcode: ir::Opcode::Fmin, + args, + } => (args[0], args[1], ir::Opcode::X86Fmin, ir::Opcode::Bor), + ir::InstructionData::Binary { + opcode: ir::Opcode::Fmax, + args, + } => (args[0], args[1], ir::Opcode::X86Fmax, ir::Opcode::Band), + _ => panic!("Expected fmin/fmax: {}", func.dfg.display_inst(inst, None)), + }; + let old_ebb = func.layout.pp_ebb(inst); + + // We need to handle the following conditions, depending on how x and y compare: + // + // 1. LT or GT: The native `x86_opc` min/max instruction does what we need. + // 2. EQ: We need to use `bitwise_opc` to make sure that + // fmin(0.0, -0.0) -> -0.0 and fmax(0.0, -0.0) -> 0.0. + // 3. UN: We need to produce a quiet NaN that is canonical if the inputs are canonical. + + // EBB handling case 3) where one operand is NaN. + let uno_ebb = func.dfg.make_ebb(); + + // EBB that handles the unordered or equal cases 2) and 3). + let ueq_ebb = func.dfg.make_ebb(); + + // Final EBB with one argument representing the final result value. + let done = func.dfg.make_ebb(); + + // The basic blocks are laid out to minimize branching for the common cases: + // + // 1) One branch not taken, one jump. + // 2) One branch taken. + // 3) Two branches taken, one jump. + + // Move the `inst` result value onto the `done` EBB. + let result = func.dfg.first_result(inst); + let ty = func.dfg.value_type(result); + func.dfg.clear_results(inst); + func.dfg.attach_ebb_arg(done, result); + + // Test for case 1) ordered and not equal. + let mut pos = FuncCursor::new(func).at_inst(inst); + let cmp_ueq = pos.ins().fcmp(FloatCC::UnorderedOrEqual, x, y); + pos.ins().brnz(cmp_ueq, ueq_ebb, &[]); + + // Handle the common ordered, not equal (LT|GT) case. + let one_inst = pos.ins().Binary(x86_opc, ty, x, y).0; + let one_result = pos.func.dfg.first_result(one_inst); + pos.ins().jump(done, &[one_result]); + + // Case 3) Unordered. + // We know that at least one operand is a NaN that needs to be propagated. We simply use an + // `fadd` instruction which has the same NaN propagation semantics. + pos.insert_ebb(uno_ebb); + let uno_result = pos.ins().fadd(x, y); + pos.ins().jump(done, &[uno_result]); + + // Case 2) or 3). + pos.insert_ebb(ueq_ebb); + // Test for case 3) (UN) one value is NaN. + // TODO: When we get support for flag values, we can reuse the above comparison. + let cmp_uno = pos.ins().fcmp(FloatCC::Unordered, x, y); + pos.ins().brnz(cmp_uno, uno_ebb, &[]); + + // We are now in case 2) where x and y compare EQ. + // We need a bitwise operation to get the sign right. + let bw_inst = pos.ins().Binary(bitwise_opc, ty, x, y).0; + let bw_result = pos.func.dfg.first_result(bw_inst); + // This should become a fall-through for this second most common case. + // Recycle the original instruction as a jump. + pos.func.dfg.replace(inst).jump(done, &[bw_result]); + + // Finally insert a label for the completion. + pos.next_inst(); + pos.insert_ebb(done); + + cfg.recompute_ebb(pos.func, old_ebb); + cfg.recompute_ebb(pos.func, ueq_ebb); + cfg.recompute_ebb(pos.func, uno_ebb); + cfg.recompute_ebb(pos.func, done); +} From 84471a8431db79aa41d4b8a900c7a635d43e0011 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Sep 2017 11:59:50 -0700 Subject: [PATCH 1178/3084] Add some very basic support for the Intel32 ABI. In 32-bit mode, all function arguments are passed on the stack, not in registers. This ABI support is not complete or properly tested, but at least it doesn't try to pass arguments in r8. --- cranelift/filetests/isa/intel/abi32.cton | 20 +++++++++++++++++++ .../filetests/isa/intel/legalize-custom.cton | 8 ++++---- .../filetests/regalloc/intel-regres.cton | 6 ++++++ lib/cretonne/src/isa/intel/abi.rs | 12 +++++++++-- 4 files changed, 40 insertions(+), 6 deletions(-) create mode 100644 cranelift/filetests/isa/intel/abi32.cton diff --git a/cranelift/filetests/isa/intel/abi32.cton b/cranelift/filetests/isa/intel/abi32.cton new file mode 100644 index 0000000000..c7e399960c --- /dev/null +++ b/cranelift/filetests/isa/intel/abi32.cton @@ -0,0 +1,20 @@ +; Test the legalization of function signatures. +test legalizer +isa intel + +; regex: V=v\d+ + +function %f() { + sig0 = (i32) -> i32 native + ; check: sig0 = (i32 [0]) -> i32 [%rax] native + + sig1 = (i64) -> b1 native + ; check: sig1 = (i32 [0], i32 [4]) -> b1 [%rax] native + + sig2 = (f32, i64) -> f64 native + ; check: sig2 = (f32 [0], i32 [4], i32 [8]) -> f64 [%xmm0] native + +ebb0: + return +} + diff --git a/cranelift/filetests/isa/intel/legalize-custom.cton b/cranelift/filetests/isa/intel/legalize-custom.cton index 94aa7ab1d2..0327e0a317 100644 --- a/cranelift/filetests/isa/intel/legalize-custom.cton +++ b/cranelift/filetests/isa/intel/legalize-custom.cton @@ -11,7 +11,7 @@ function %cond_trap(i32) { ebb0(v1: i32): trapz v1, user67 return - ; check: $ebb0($v1: i32): + ; check: $ebb0($v1: i32 ; nextln: brnz $v1, $(new=$EBB) ; nextln: trap user67 ; check: $new: @@ -22,7 +22,7 @@ function %cond_trap2(i32) { ebb0(v1: i32): trapnz v1, int_ovf return - ; check: $ebb0($v1: i32): + ; check: $ebb0($v1: i32 ; nextln: brz $v1, $(new=$EBB) ; nextln: trap int_ovf ; check: $new: @@ -34,7 +34,7 @@ ebb0(v1: i32): v2 = icmp_imm eq v1, 6 trapz v2, user7 return - ; check: $ebb0($v1: i32): + ; check: $ebb0($v1: i32 ; check: brnz $v2, $(new=$EBB) ; nextln: trap user7 ; check: $new: @@ -46,7 +46,7 @@ ebb0(v1: i32): v2 = icmp_imm eq v1, 6 trapnz v2, user9 return - ; check: $ebb0($v1: i32): + ; check: $ebb0($v1: i32 ; check: brz $v2, $(new=$EBB) ; nextln: trap user9 ; check: $new: diff --git a/cranelift/filetests/regalloc/intel-regres.cton b/cranelift/filetests/regalloc/intel-regres.cton index 24d13f5216..0de1b620c5 100644 --- a/cranelift/filetests/regalloc/intel-regres.cton +++ b/cranelift/filetests/regalloc/intel-regres.cton @@ -38,3 +38,9 @@ ebb2(v4: i32, v5: i32, v7: i32): ebb3: return v5 } + +function %select_i64(i64, i64, i32) -> i64 { +ebb0(v0: i64, v1: i64, v2: i32): + v3 = select v2, v0, v1 + return v3 +} diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 7a051905d4..c6bbe7fb2a 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -106,9 +106,17 @@ impl ArgAssigner for Args { /// Legalize `sig`. pub fn legalize_signature(sig: &mut ir::Signature, flags: &shared_settings::Flags, _current: bool) { - let bits = if flags.is_64bit() { 64 } else { 32 }; + let bits; + let mut args; + + if flags.is_64bit() { + bits = 64; + args = Args::new(bits, &ARG_GPRS, 8); + } else { + bits = 32; + args = Args::new(bits, &[], 0); + } - let mut args = Args::new(bits, &ARG_GPRS, 8); legalize_args(&mut sig.argument_types, &mut args); let mut rets = Args::new(bits, &RET_GPRS, 2); From de9b1d1004bb127075e16579639ab5bb1c6bd306 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Sep 2017 11:20:45 -0700 Subject: [PATCH 1179/3084] Enable debug assertions for cretonne-tools. The cton-util executable is used in --release mode to run the file tests. We want assertions enabled for that. This doesn't affect the compiler flags used to build the Cretonne crate when it is used as a dependency in another project. --- cranelift/Cargo.toml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index c576d3990e..45b39ef0a3 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -27,3 +27,11 @@ tempdir="0.3.5" term = "0.4.6" [workspace] + +# Enable debug assertions and parallel compilation when building cretonne-tools +# since they are for testing and development mostly. This doesn't affect the +# flags used to build the Cretonne crate when used as a dependency. +[profile.release] +opt-level = 2 +debug-assertions = true +codegen-units = 4 From 1d481d78978a6b36c9612341dab2a04aaca42ca6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Sep 2017 16:27:49 -0700 Subject: [PATCH 1180/3084] Use the ThreadId to name cretonne.dbg in unnamed threads. Don't use a single cretonne.dbg log file when there are multiple unnamed threads logging. They will clobber each other. --- lib/cretonne/src/dbg.rs | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/lib/cretonne/src/dbg.rs b/lib/cretonne/src/dbg.rs index 9bd5500e3c..5acadd2098 100644 --- a/lib/cretonne/src/dbg.rs +++ b/lib/cretonne/src/dbg.rs @@ -9,7 +9,6 @@ /// The output will appear in files named `cretonne.dbg.*`, where the suffix is named after the /// thread doing the logging. -use std::ascii::AsciiExt; use std::cell::RefCell; use std::env; use std::ffi::OsStr; @@ -72,18 +71,20 @@ pub fn writeln_with_format_args(args: fmt::Arguments) -> io::Result<()> { /// Open the tracing file for the current thread. fn open_file() -> io::BufWriter { - let file = match thread::current().name() { - None => File::create("cretonne.dbg"), - Some(name) => { - let mut path = "cretonne.dbg.".to_owned(); - for ch in name.chars() { - if ch.is_ascii() && ch.is_alphanumeric() { - path.push(ch); - } + let curthread = thread::current(); + let tmpstr; + let mut path = "cretonne.dbg.".to_owned(); + path.extend( + match curthread.name() { + Some(name) => name.chars(), + // The thread is unnamed, so use the thread ID instead. + None => { + tmpstr = format!("{:?}", curthread.id()); + tmpstr.chars() } - File::create(path) - } - }.expect("Can't open tracing file"); + }.filter(|ch| ch.is_alphanumeric() || *ch == '-' || *ch == '_'), + ); + let file = File::create(path).expect("Can't open tracing file"); io::BufWriter::new(file) } From a274cdf275c5df5f05c76214121d5ce2c5e90ccf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 27 Sep 2017 18:14:13 -0700 Subject: [PATCH 1181/3084] Fix the Intel encoding of band_not. The andnps instruction inverts its first argument while band_not inverts is second argument. Use a swapped-operands "fax" encoding recipe. --- cranelift/filetests/isa/intel/binary32-float.cton | 8 ++++---- cranelift/filetests/isa/intel/binary64-float.cton | 8 ++++---- lib/cretonne/meta/isa/intel/encodings.py | 13 ++++++++----- lib/cretonne/meta/isa/intel/recipes.py | 14 +++++++++++--- 4 files changed, 27 insertions(+), 16 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 4f38807c9e..a880ed9a8a 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -78,9 +78,9 @@ ebb0: [-,%xmm2] v31 = band v11, v10 ; bin: 0f 54 d5 ; asm: andnps %xmm2, %xmm5 - [-,%xmm5] v32 = band_not v10, v11 ; bin: 0f 55 ea + [-,%xmm5] v32 = band_not v11, v10 ; bin: 0f 55 ea ; asm: andnps %xmm5, %xmm2 - [-,%xmm2] v33 = band_not v11, v10 ; bin: 0f 55 d5 + [-,%xmm2] v33 = band_not v10, v11 ; bin: 0f 55 d5 ; asm: orps %xmm2, %xmm5 [-,%xmm5] v34 = bor v10, v11 ; bin: 0f 56 ea @@ -281,9 +281,9 @@ ebb0: [-,%xmm2] v31 = band v11, v10 ; bin: 0f 54 d5 ; asm: andnps %xmm2, %xmm5 - [-,%xmm5] v32 = band_not v10, v11 ; bin: 0f 55 ea + [-,%xmm5] v32 = band_not v11, v10 ; bin: 0f 55 ea ; asm: andnps %xmm5, %xmm2 - [-,%xmm2] v33 = band_not v11, v10 ; bin: 0f 55 d5 + [-,%xmm2] v33 = band_not v10, v11 ; bin: 0f 55 d5 ; asm: orps %xmm2, %xmm5 [-,%xmm5] v34 = bor v10, v11 ; bin: 0f 56 ea diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index 2156ebc5db..18a24f9e7e 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -77,9 +77,9 @@ ebb0: [-,%xmm10] v31 = band v11, v10 ; bin: 44 0f 54 d5 ; asm: andnps %xmm10, %xmm5 - [-,%xmm5] v32 = band_not v10, v11 ; bin: 41 0f 55 ea + [-,%xmm5] v32 = band_not v11, v10 ; bin: 41 0f 55 ea ; asm: andnps %xmm5, %xmm10 - [-,%xmm10] v33 = band_not v11, v10 ; bin: 44 0f 55 d5 + [-,%xmm10] v33 = band_not v10, v11 ; bin: 44 0f 55 d5 ; asm: orps %xmm10, %xmm5 [-,%xmm5] v34 = bor v10, v11 ; bin: 41 0f 56 ea @@ -295,9 +295,9 @@ ebb0: [-,%xmm10] v31 = band v11, v10 ; bin: 44 0f 54 d5 ; asm: andnps %xmm10, %xmm5 - [-,%xmm5] v32 = band_not v10, v11 ; bin: 41 0f 55 ea + [-,%xmm5] v32 = band_not v11, v10 ; bin: 41 0f 55 ea ; asm: andnps %xmm5, %xmm10 - [-,%xmm10] v33 = band_not v11, v10 ; bin: 44 0f 55 d5 + [-,%xmm10] v33 = band_not v10, v11 ; bin: 44 0f 55 d5 ; asm: orps %xmm10, %xmm5 [-,%xmm5] v34 = bor v10, v11 ; bin: 41 0f 56 ea diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index f1d6a28900..671b743827 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -403,17 +403,20 @@ for inst, opc in [ (base.fdiv, 0x5e), (x86.fmin, 0x5d), (x86.fmax, 0x5f)]: - enc_flt(inst.f32, r.frm, 0xf3, 0x0f, opc) - enc_flt(inst.f64, r.frm, 0xf2, 0x0f, opc) + enc_flt(inst.f32, r.fa, 0xf3, 0x0f, opc) + enc_flt(inst.f64, r.fa, 0xf2, 0x0f, opc) # Binary bitwise ops. for inst, opc in [ (base.band, 0x54), - (base.band_not, 0x55), (base.bor, 0x56), (base.bxor, 0x57)]: - enc_flt(inst.f32, r.frm, 0x0f, opc) - enc_flt(inst.f64, r.frm, 0x0f, opc) + enc_flt(inst.f32, r.fa, 0x0f, opc) + enc_flt(inst.f64, r.fa, 0x0f, opc) + +# The `andnps(x,y)` instruction computes `~x&y`, while band_not(x,y)` is `x&~y. +enc_flt(base.band_not.f32, r.fax, 0x0f, 0x55) +enc_flt(base.band_not.f64, r.fax, 0x0f, 0x55) # Comparisons. # diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index eebba58700..fe29735efe 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -239,14 +239,22 @@ rrx = TailRecipe( modrm_rr(in_reg1, in_reg0, sink); ''') -# XX /r with FPR ins and outs. RM form. -frm = TailRecipe( - 'frm', Binary, size=1, ins=(FPR, FPR), outs=0, +# XX /r with FPR ins and outs. A form. +fa = TailRecipe( + 'fa', Binary, size=1, ins=(FPR, FPR), outs=0, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_rr(in_reg1, in_reg0, sink); ''') +# XX /r with FPR ins and outs. A form with input operands swapped. +fax = TailRecipe( + 'fax', Binary, size=1, ins=(FPR, FPR), outs=1, + emit=''' + PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + modrm_rr(in_reg0, in_reg1, sink); + ''') + # XX /r, but for a unary operator with separate input/output register, like # copies. MR form. umr = TailRecipe( From 979a22f548df7c624a269482b5a190bb4d6c7c94 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 28 Sep 2017 11:26:37 -0700 Subject: [PATCH 1182/3084] Add pow2() and neg() methods for the IEEE immediate types. These are convenient methods for creating common floating point constants. --- lib/cretonne/src/ir/immediates.rs | 56 +++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index bf9d4bea72..d13887e4b9 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -529,6 +529,23 @@ impl Ieee32 { Ieee32(x) } + /// Create an `Ieee32` number representing 2.0^n. + pub fn pow2>(n: I) -> Ieee32 { + let n = n.into(); + let w = 8; + let t = 23; + let bias = (1 << (w - 1)) - 1; + let exponent = (n + bias) as u32; + assert!(exponent > 0, "Underflow n={}", n); + assert!(exponent < (1 << w) + 1, "Overflow n={}", n); + Ieee32(exponent << t) + } + + /// Return self negated. + pub fn neg(self) -> Ieee32 { + Ieee32(self.0 ^ (1 << 31)) + } + /// Create a new `Ieee32` representing the number `x`. #[cfg(test)] pub fn with_float(x: f32) -> Ieee32 { @@ -565,6 +582,23 @@ impl Ieee64 { Ieee64(x) } + /// Create an `Ieee64` number representing 2.0^n. + pub fn pow2>(n: I) -> Ieee64 { + let n = n.into(); + let w = 11; + let t = 52; + let bias = (1 << (w - 1)) - 1; + let exponent = (n + bias) as u64; + assert!(exponent > 0, "Underflow n={}", n); + assert!(exponent < (1 << w) + 1, "Overflow n={}", n); + Ieee64(exponent << t) + } + + /// Return self negated. + pub fn neg(self) -> Ieee64 { + Ieee64(self.0 ^ (1 << 63)) + } + /// Create a new `Ieee64` representing the number `x`. #[cfg(test)] pub fn with_float(x: f64) -> Ieee64 { @@ -817,6 +851,17 @@ mod tests { parse_err::("sNaN:0x400001", "Invalid sNaN payload"); } + #[test] + fn pow2_ieee32() { + assert_eq!(Ieee32::pow2(0).to_string(), "0x1.000000p0"); + assert_eq!(Ieee32::pow2(1).to_string(), "0x1.000000p1"); + assert_eq!(Ieee32::pow2(-1).to_string(), "0x1.000000p-1"); + assert_eq!(Ieee32::pow2(127).to_string(), "0x1.000000p127"); + assert_eq!(Ieee32::pow2(-126).to_string(), "0x1.000000p-126"); + + assert_eq!(Ieee32::pow2(1).neg().to_string(), "-0x1.000000p1"); + } + #[test] fn format_ieee64() { assert_eq!(Ieee64::with_float(0.0).to_string(), "0.0"); @@ -934,4 +979,15 @@ mod tests { parse_ok::("sNaN:0x4000000000001", "+sNaN:0x4000000000001"); parse_err::("sNaN:0x8000000000001", "Invalid sNaN payload"); } + + #[test] + fn pow2_ieee64() { + assert_eq!(Ieee64::pow2(0).to_string(), "0x1.0000000000000p0"); + assert_eq!(Ieee64::pow2(1).to_string(), "0x1.0000000000000p1"); + assert_eq!(Ieee64::pow2(-1).to_string(), "0x1.0000000000000p-1"); + assert_eq!(Ieee64::pow2(1023).to_string(), "0x1.0000000000000p1023"); + assert_eq!(Ieee64::pow2(-1022).to_string(), "0x1.0000000000000p-1022"); + + assert_eq!(Ieee64::pow2(1).neg().to_string(), "-0x1.0000000000000p1"); + } } From 34146435e5763bf9ea92b156d3b67de3733efa19 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 28 Sep 2017 09:05:39 -0700 Subject: [PATCH 1183/3084] Legalize unsigned-to-float conversions for Intel 64. Also make sure we generate type checks for the controlling type variable in legalization patterns. This is not needed for encodings since the encoding tables are already keyed on the controlling type variable. --- cranelift/filetests/wasm/conversions.cton | 24 +++++++- lib/cretonne/meta/cdsl/ast.py | 40 ++++++++++--- lib/cretonne/meta/cdsl/predicates.py | 42 +++++++++++++- lib/cretonne/meta/gen_legalizer.py | 2 +- lib/cretonne/meta/isa/intel/legalize.py | 3 + lib/cretonne/src/isa/intel/enc_tables.rs | 71 +++++++++++++++++++++++ 6 files changed, 172 insertions(+), 10 deletions(-) diff --git a/cranelift/filetests/wasm/conversions.cton b/cranelift/filetests/wasm/conversions.cton index e742ebba27..a9bf625ca1 100644 --- a/cranelift/filetests/wasm/conversions.cton +++ b/cranelift/filetests/wasm/conversions.cton @@ -49,25 +49,47 @@ ebb0(v0: i32): return v1 } +function %f32_convert_u_i32(i32) -> f32 { +ebb0(v0: i32): + v1 = fcvt_from_uint.f32 v0 + return v1 +} + function %f64_convert_s_i32(i32) -> f64 { ebb0(v0: i32): v1 = fcvt_from_sint.f64 v0 return v1 } +function %f64_convert_u_i32(i32) -> f64 { +ebb0(v0: i32): + v1 = fcvt_from_uint.f64 v0 + return v1 +} + function %f32_convert_s_i64(i64) -> f32 { ebb0(v0: i64): v1 = fcvt_from_sint.f32 v0 return v1 } +function %f32_convert_u_i64(i64) -> f32 { +ebb0(v0: i64): + v1 = fcvt_from_uint.f32 v0 + return v1 +} + function %f64_convert_s_i64(i64) -> f64 { ebb0(v0: i64): v1 = fcvt_from_sint.f64 v0 return v1 } -; TODO: f*_convert_u_i* (Don't exist on Intel). +function %f64_convert_u_i64(i64) -> f64 { +ebb0(v0: i64): + v1 = fcvt_from_uint.f64 v0 + return v1 +} function %i32_reinterpret_f32(f32) -> i32 { ebb0(v0: f32): diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index 619c2d36a2..d6c0f42952 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -7,7 +7,7 @@ for patern matching an rewriting of cretonne instructions. from __future__ import absolute_import from . import instructions from .typevar import TypeVar -from .predicates import IsEqual, And, TypePredicate +from .predicates import IsEqual, And, TypePredicate, CtrlTypePredicate try: from typing import Union, Tuple, Sequence, TYPE_CHECKING, Dict, List # noqa @@ -383,12 +383,38 @@ class Apply(Expr): pred = And.combine(pred, IsEqual(ffield, arg)) - # Add checks for any bound type variables. - for bound_ty, tv in zip(self.typevars, self.inst.all_typevars()): - if bound_ty is None: - continue - type_chk = TypePredicate.typevar_check(self.inst, tv, bound_ty) - pred = And.combine(pred, type_chk) + # Add checks for any bound secondary type variables. + # We can't check the controlling type variable this way since it may + # not appear as the type of an operand. + if len(self.typevars) > 1: + for bound_ty, tv in zip(self.typevars[1:], + self.inst.other_typevars): + if bound_ty is None: + continue + type_chk = TypePredicate.typevar_check(self.inst, tv, bound_ty) + pred = And.combine(pred, type_chk) + + return pred + + def inst_predicate_with_ctrl_typevar(self): + # type: () -> PredNode + """ + Same as `inst_predicate()`, but also check the controlling type + variable. + """ + pred = self.inst_predicate() + + if len(self.typevars) > 0: + bound_ty = self.typevars[0] + type_chk = None # type: PredNode + if bound_ty is not None: + # Prefer to look at the types of input operands. + if self.inst.use_typevar_operand: + type_chk = TypePredicate.typevar_check( + self.inst, self.inst.ctrl_typevar, bound_ty) + else: + type_chk = CtrlTypePredicate(bound_ty) + pred = And.combine(pred, type_chk) return pred diff --git a/lib/cretonne/meta/cdsl/predicates.py b/lib/cretonne/meta/cdsl/predicates.py index f1f5232272..9a28fd9945 100644 --- a/lib/cretonne/meta/cdsl/predicates.py +++ b/lib/cretonne/meta/cdsl/predicates.py @@ -35,7 +35,8 @@ try: from .typevar import TypeVar # noqa PredContext = Union[SettingGroup, InstructionFormat, InstructionContext] - PredLeaf = Union[BoolSetting, 'FieldPredicate', 'TypePredicate'] + PredLeaf = Union[BoolSetting, 'FieldPredicate', 'TypePredicate', + 'CtrlTypePredicate'] PredNode = Union[PredLeaf, 'Predicate'] # A predicate key is a (recursive) tuple of primitive types that # uniquely describes a predicate. It is used for interning. @@ -373,3 +374,42 @@ class TypePredicate(object): """ return 'dfg.value_type(args[{}]) == {}'.format( self.value_arg, self.value_type.rust_name()) + + +class CtrlTypePredicate(object): + """ + An instruction predicate that checks the controlling type variable + + :param value_type: The required value type. + """ + + def __init__(self, value_type): + # type: (ValueType) -> None + assert value_type is not None + self.value_type = value_type + + def __str__(self): + # type: () -> str + return 'ctrl_typevar:{}'.format(self.value_type) + + def predicate_context(self): + # type: () -> PredContext + return instruction_context + + def predicate_key(self): + # type: () -> PredKey + return ('ctrltypecheck', self.value_type.name) + + def predicate_leafs(self, leafs): + # type: (Set[PredLeaf]) -> None + leafs.add(self) + + def rust_predicate(self, prec): + # type: (int) -> str + """ + Return Rust code for evaluating this predicate. + + It is assumed that the context has `dfg` and `inst` variables. + """ + return 'dfg.ctrl_typevar(inst) == {}'.format( + self.value_type.rust_name()) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 1b39918658..c27df484cf 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -199,7 +199,7 @@ def unwrap_inst(iref, node, fmt): n = expr.inst.value_opnums.index(opnum) fmt.format('dfg.resolve_aliases(args[{}]),', n) # Evaluate the instruction predicate, if any. - instp = expr.inst_predicate() + instp = expr.inst_predicate_with_ctrl_typevar() fmt.line(instp.rust_predicate(0) if instp else 'true') fmt.outdented_line('} else {') fmt.line('unreachable!("bad instruction format")') diff --git a/lib/cretonne/meta/isa/intel/legalize.py b/lib/cretonne/meta/isa/intel/legalize.py index 8d93337c69..d698208ccc 100644 --- a/lib/cretonne/meta/isa/intel/legalize.py +++ b/lib/cretonne/meta/isa/intel/legalize.py @@ -96,3 +96,6 @@ for cc, rev_cc in [ # We need to modify the CFG for min/max legalization. intel_expand.custom_legalize(insts.fmin, 'expand_minmax') intel_expand.custom_legalize(insts.fmax, 'expand_minmax') + +# Conversions from unsigned need special handling. +intel_expand.custom_legalize(insts.fcvt_from_uint, 'expand_fcvt_from_uint') diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 4c8c428bfc..56ccdd15b2 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -62,6 +62,7 @@ fn expand_minmax(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowG // Test for case 1) ordered and not equal. let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); let cmp_ueq = pos.ins().fcmp(FloatCC::UnorderedOrEqual, x, y); pos.ins().brnz(cmp_ueq, ueq_ebb, &[]); @@ -101,3 +102,73 @@ fn expand_minmax(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowG cfg.recompute_ebb(pos.func, uno_ebb); cfg.recompute_ebb(pos.func, done); } + +/// Intel has no unsigned-to-float conversions. We handle the easy case of zero-extending i32 to +/// i64 with a pattern, the rest needs more code. +fn expand_fcvt_from_uint(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { + use ir::condcodes::IntCC; + + let x; + match func.dfg[inst] { + ir::InstructionData::Unary { + opcode: ir::Opcode::FcvtFromUint, + arg, + } => x = arg, + _ => panic!("Need fcvt_from_uint: {}", func.dfg.display_inst(inst, None)), + } + let xty = func.dfg.value_type(x); + let result = func.dfg.first_result(inst); + let ty = func.dfg.value_type(result); + let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); + + // Conversion from unsigned 32-bit is easy on x86-64. + // TODO: This should be guarded by an ISA check. + if xty == ir::types::I32 { + let wide = pos.ins().uextend(ir::types::I64, x); + pos.func.dfg.replace(inst).fcvt_from_sint(ty, wide); + return; + } + + let old_ebb = pos.func.layout.pp_ebb(inst); + + // EBB handling the case where x < 0. + let neg_ebb = pos.func.dfg.make_ebb(); + + // Final EBB with one argument representing the final result value. + let done = pos.func.dfg.make_ebb(); + + // Move the `inst` result value onto the `done` EBB. + pos.func.dfg.clear_results(inst); + pos.func.dfg.attach_ebb_arg(done, result); + + // If x as a signed int is not negative, we can use the existing `fcvt_from_sint` instruction. + let is_neg = pos.ins().icmp_imm(IntCC::SignedLessThan, x, 0); + pos.ins().brnz(is_neg, neg_ebb, &[]); + + // Easy case: just use a signed conversion. + let posres = pos.ins().fcvt_from_sint(ty, x); + pos.ins().jump(done, &[posres]); + + // Now handle the negative case. + pos.insert_ebb(neg_ebb); + + // Divide x by two to get it in range for the signed conversion, keep the LSB, and scale it + // back up on the FP side. + let ihalf = pos.ins().ushr_imm(x, 1); + let lsb = pos.ins().band_imm(x, 1); + let ifinal = pos.ins().bor(ihalf, lsb); + let fhalf = pos.ins().fcvt_from_sint(ty, ifinal); + let negres = pos.ins().fadd(fhalf, fhalf); + + // Recycle the original instruction as a jump. + pos.func.dfg.replace(inst).jump(done, &[negres]); + + // Finally insert a label for the completion. + pos.next_inst(); + pos.insert_ebb(done); + + cfg.recompute_ebb(pos.func, old_ebb); + cfg.recompute_ebb(pos.func, neg_ebb); + cfg.recompute_ebb(pos.func, done); +} From 8abcdac5a169f61b457a3baf2d3a1e54460486a7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 28 Sep 2017 10:54:56 -0700 Subject: [PATCH 1184/3084] Legalize fcvt_to_sint and fcvt_to_uint for Intel64. We need to generate traps on NaN and overflow. --- cranelift/filetests/wasm/conversions.cton | 55 +++++++-- lib/cretonne/meta/isa/intel/legalize.py | 3 + lib/cretonne/src/isa/intel/enc_tables.rs | 132 ++++++++++++++++++++++ 3 files changed, 182 insertions(+), 8 deletions(-) diff --git a/cranelift/filetests/wasm/conversions.cton b/cranelift/filetests/wasm/conversions.cton index a9bf625ca1..4ce70ee583 100644 --- a/cranelift/filetests/wasm/conversions.cton +++ b/cranelift/filetests/wasm/conversions.cton @@ -22,14 +22,53 @@ ebb0(v0: i32): return v1 } -; function %i32_trunc_s_f32(f32) -> i32 -; function %i32_trunc_u_f32(f32) -> i32 -; function %i32_trunc_s_f64(f64) -> i32 -; function %i32_trunc_u_f64(f64) -> i32 -; function %i64_trunc_s_f32(f32) -> i64 -; function %i64_trunc_u_f32(f32) -> i64 -; function %i64_trunc_s_f64(f64) -> i64 -; function %i64_trunc_u_f64(f64) -> i64 +function %i32_trunc_s_f32(f32) -> i32 { +ebb0(v0: f32): + v1 = fcvt_to_sint.i32 v0 + return v1 +} + +function %i32_trunc_u_f32(f32) -> i32 { +ebb0(v0: f32): + v1 = fcvt_to_uint.i32 v0 + return v1 +} + +function %i32_trunc_s_f64(f64) -> i32 { +ebb0(v0: f64): + v1 = fcvt_to_sint.i32 v0 + return v1 +} + +function %i32_trunc_u_f64(f64) -> i32 { +ebb0(v0: f64): + v1 = fcvt_to_uint.i32 v0 + return v1 +} + +function %i64_trunc_s_f32(f32) -> i64 { +ebb0(v0: f32): + v1 = fcvt_to_sint.i64 v0 + return v1 +} + +function %i64_trunc_u_f32(f32) -> i64 { +ebb0(v0: f32): + v1 = fcvt_to_uint.i64 v0 + return v1 +} + +function %i64_trunc_s_f64(f64) -> i64 { +ebb0(v0: f64): + v1 = fcvt_to_sint.i64 v0 + return v1 +} + +function %i64_trunc_u_f64(f64) -> i64 { +ebb0(v0: f64): + v1 = fcvt_to_uint.i64 v0 + return v1 +} function %f32_trunc_f64(f64) -> f32 { ebb0(v0: f64): diff --git a/lib/cretonne/meta/isa/intel/legalize.py b/lib/cretonne/meta/isa/intel/legalize.py index d698208ccc..5987fd7221 100644 --- a/lib/cretonne/meta/isa/intel/legalize.py +++ b/lib/cretonne/meta/isa/intel/legalize.py @@ -99,3 +99,6 @@ intel_expand.custom_legalize(insts.fmax, 'expand_minmax') # Conversions from unsigned need special handling. intel_expand.custom_legalize(insts.fcvt_from_uint, 'expand_fcvt_from_uint') +# Conversions from float to int can trap. +intel_expand.custom_legalize(insts.fcvt_to_sint, 'expand_fcvt_to_sint') +intel_expand.custom_legalize(insts.fcvt_to_uint, 'expand_fcvt_to_uint') diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 56ccdd15b2..69667bd745 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -172,3 +172,135 @@ fn expand_fcvt_from_uint(inst: ir::Inst, func: &mut ir::Function, cfg: &mut Cont cfg.recompute_ebb(pos.func, neg_ebb); cfg.recompute_ebb(pos.func, done); } + +fn expand_fcvt_to_sint(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { + use ir::condcodes::{IntCC, FloatCC}; + use ir::immediates::{Ieee32, Ieee64}; + + let x; + match func.dfg[inst] { + ir::InstructionData::Unary { + opcode: ir::Opcode::FcvtToSint, + arg, + } => x = arg, + _ => panic!("Need fcvt_to_sint: {}", func.dfg.display_inst(inst, None)), + } + let old_ebb = func.layout.pp_ebb(inst); + let xty = func.dfg.value_type(x); + let result = func.dfg.first_result(inst); + let ty = func.dfg.value_type(result); + + // Final EBB after the bad value checks. + let done = func.dfg.make_ebb(); + + // The `x86_cvtt2si` performs the desired conversion, but it doesn't trap on NaN or overflow. + // It produces an INT_MIN result instead. + func.dfg.replace(inst).x86_cvtt2si(ty, x); + + let mut pos = FuncCursor::new(func).after_inst(inst); + pos.use_srcloc(inst); + + let is_done = pos.ins().icmp_imm( + IntCC::NotEqual, + result, + 1 << (ty.lane_bits() - 1), + ); + pos.ins().brnz(is_done, done, &[]); + + // We now have the following possibilities: + // + // 1. INT_MIN was actually the correct conversion result. + // 2. The input was NaN -> trap bad_toint + // 3. The input was out of range -> trap int_ovf + // + + // Check for NaN. + let is_nan = pos.ins().fcmp(FloatCC::Unordered, x, x); + pos.ins().trapnz( + is_nan, + ir::TrapCode::BadConversionToInteger, + ); + + // Check for case 1: INT_MIN is the correct result. + // We use a `ueq` condition here because that can be translated into a single branch, and we + // already know that we don't have a NaN. + let fintmin = match xty { + ir::types::F32 => pos.ins().f32const(Ieee32::pow2(ty.lane_bits() - 1).neg()), + ir::types::F64 => pos.ins().f64const(Ieee64::pow2(ty.lane_bits() - 1).neg()), + _ => panic!("Can't convert {}", xty), + }; + let in_range = pos.ins().fcmp(FloatCC::UnorderedOrEqual, x, fintmin); + pos.ins().trapz(in_range, ir::TrapCode::IntegerOverflow); + + pos.ins().jump(done, &[]); + pos.insert_ebb(done); + + cfg.recompute_ebb(pos.func, old_ebb); + cfg.recompute_ebb(pos.func, done); +} + +fn expand_fcvt_to_uint(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { + use ir::condcodes::{IntCC, FloatCC}; + use ir::immediates::{Ieee32, Ieee64}; + + let x; + match func.dfg[inst] { + ir::InstructionData::Unary { + opcode: ir::Opcode::FcvtToUint, + arg, + } => x = arg, + _ => panic!("Need fcvt_to_uint: {}", func.dfg.display_inst(inst, None)), + } + let old_ebb = func.layout.pp_ebb(inst); + let xty = func.dfg.value_type(x); + let result = func.dfg.first_result(inst); + let ty = func.dfg.value_type(result); + + // EBB handling numbers >= 2^(N-1). + let large = func.dfg.make_ebb(); + + // Final EBB after the bad value checks. + let done = func.dfg.make_ebb(); + + // Move the `inst` result value onto the `done` EBB. + func.dfg.clear_results(inst); + func.dfg.attach_ebb_arg(done, result); + + let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); + + // Start by materializing the floating point constant 2^(N-1) where N is the number of bits in + // the destination integer type. + let pow2nm1 = match xty { + ir::types::F32 => pos.ins().f32const(Ieee32::pow2(ty.lane_bits() - 1)), + ir::types::F64 => pos.ins().f64const(Ieee64::pow2(ty.lane_bits() - 1)), + _ => panic!("Can't convert {}", xty), + }; + let is_large = pos.ins().fcmp(FloatCC::GreaterThanOrEqual, x, pow2nm1); + pos.ins().brnz(is_large, large, &[]); + + // Now we know that x < 2^(N-1) or x is NaN. + let sres = pos.ins().x86_cvtt2si(ty, x); + let is_neg = pos.ins().icmp_imm(IntCC::SignedLessThan, sres, 0); + pos.ins().brz(is_neg, done, &[sres]); + pos.ins().trap(ir::TrapCode::BadConversionToInteger); + + // Handle the case where x >= 2^(N-1) and not NaN. + pos.insert_ebb(large); + let adjx = pos.ins().fsub(x, pow2nm1); + let lres = pos.ins().x86_cvtt2si(ty, adjx); + let is_neg = pos.ins().icmp_imm(IntCC::SignedLessThan, lres, 0); + pos.ins().trapnz(is_neg, ir::TrapCode::IntegerOverflow); + let lfinal = pos.ins().iadd_imm(lres, 1 << (ty.lane_bits() - 1)); + + // Recycle the original instruction as a jump. + pos.func.dfg.replace(inst).jump(done, &[lfinal]); + + // Finally insert a label for the completion. + pos.next_inst(); + pos.insert_ebb(done); + + cfg.recompute_ebb(pos.func, old_ebb); + cfg.recompute_ebb(pos.func, large); + cfg.recompute_ebb(pos.func, done); +} From 2888ff5bf35ecd9984a97cd2c6bf91f6092885d3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 28 Sep 2017 14:24:39 -0700 Subject: [PATCH 1185/3084] Fix a corner case in fcvt_to_sint.i32.f64 legalization. An f64 can represent multiple values in the range INT_MIN-1 < x <= INT_MIN which all truncate to INT_MIN, so comparing the input value against INT_MIN is not good enough. Instead, detect overflow on x <= INT_MIN-1 when INT_MIN-1 is an exact floating point value. --- lib/cretonne/src/ir/immediates.rs | 8 ++------ lib/cretonne/src/isa/intel/enc_tables.rs | 24 +++++++++++++++++------- 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index d13887e4b9..ecaa6d5e4d 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -6,11 +6,9 @@ //! module in the meta language. use std::fmt::{self, Display, Formatter}; -use std::{i32, u32}; -use std::str::FromStr; - -#[cfg(test)] use std::mem; +use std::str::FromStr; +use std::{i32, u32}; /// 64-bit immediate integer operand. /// @@ -547,7 +545,6 @@ impl Ieee32 { } /// Create a new `Ieee32` representing the number `x`. - #[cfg(test)] pub fn with_float(x: f32) -> Ieee32 { Ieee32(unsafe { mem::transmute(x) }) } @@ -600,7 +597,6 @@ impl Ieee64 { } /// Create a new `Ieee64` representing the number `x`. - #[cfg(test)] pub fn with_float(x: f64) -> Ieee64 { Ieee64(unsafe { mem::transmute(x) }) } diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 69667bd745..ad979a2e91 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -222,15 +222,25 @@ fn expand_fcvt_to_sint(inst: ir::Inst, func: &mut ir::Function, cfg: &mut Contro ); // Check for case 1: INT_MIN is the correct result. - // We use a `ueq` condition here because that can be translated into a single branch, and we - // already know that we don't have a NaN. - let fintmin = match xty { - ir::types::F32 => pos.ins().f32const(Ieee32::pow2(ty.lane_bits() - 1).neg()), - ir::types::F64 => pos.ins().f64const(Ieee64::pow2(ty.lane_bits() - 1).neg()), + // Determine the smallest floating point number that would convert to INT_MIN. + let mut overflow_cc = FloatCC::LessThan; + let output_bits = ty.lane_bits(); + let flimit = match xty { + ir::types::F32 => pos.ins().f32const(Ieee32::pow2(output_bits - 1).neg()), + ir::types::F64 => { + // An f64 can represent `i32::min_value() - 1` exactly with precision to spare, so + // there are values less than -2^(N-1) that convert correctly to INT_MIN. + pos.ins().f64const(if output_bits < 64 { + overflow_cc = FloatCC::LessThanOrEqual; + Ieee64::with_float(-((1u64 << (output_bits - 1)) as f64) - 1.0) + } else { + Ieee64::pow2(output_bits - 1).neg() + }) + } _ => panic!("Can't convert {}", xty), }; - let in_range = pos.ins().fcmp(FloatCC::UnorderedOrEqual, x, fintmin); - pos.ins().trapz(in_range, ir::TrapCode::IntegerOverflow); + let overflow = pos.ins().fcmp(overflow_cc, x, flimit); + pos.ins().trapnz(overflow, ir::TrapCode::IntegerOverflow); pos.ins().jump(done, &[]); pos.insert_ebb(done); From 53404a9387a29c738c7c8151ff172c9a4c54770c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 28 Sep 2017 16:22:04 -0700 Subject: [PATCH 1186/3084] Check for invalid special type constraints. The extend and reduce instructions have additional type constraints. Stop inserting sextend instructions after ctz, clz, and popcnt when translating from WebAssembly. The Cretonne instructions have the same signature as the WebAssembly equivalents. --- cranelift/filetests/verifier/type_check.cton | 14 +++++ lib/cretonne/src/verifier/mod.rs | 54 +++++++++++++++++++- lib/wasm/src/code_translator.rs | 22 +++----- 3 files changed, 74 insertions(+), 16 deletions(-) diff --git a/cranelift/filetests/verifier/type_check.cton b/cranelift/filetests/verifier/type_check.cton index b28f52930f..d8250389c4 100644 --- a/cranelift/filetests/verifier/type_check.cton +++ b/cranelift/filetests/verifier/type_check.cton @@ -95,3 +95,17 @@ function %jump_args2() { ebb1(v10: i64, v11: i16): return } + +function %bad_extend() { +ebb0: + v0 = iconst.i32 10 + v1 = uextend.i16 v0 ; error: input i32 must be smaller than output i16 + return +} + +function %bad_reduce() { +ebb0: + v0 = iconst.i32 10 + v1 = ireduce.i64 v0 ; error: input i32 must be larger than output i64 + return +} diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 82ba075f39..d76a5efbba 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -49,8 +49,6 @@ //! //! - Stack slot loads and stores must be in-bounds. //! - Immediate constraints for certain opcodes, like `udiv_imm v3, 0`. -//! - Extend / truncate instructions have more type constraints: Source type can't be -//! larger / smaller than result type. //! - `Insertlane` and `extractlane` instructions have immediate lane numbers that must be in //! range for their polymorphic type. //! - Swizzle and shuffle instructions take a variable number of lane arguments. The number @@ -574,6 +572,7 @@ impl<'a> Verifier<'a> { self.typecheck_fixed_args(inst, ctrl_type)?; self.typecheck_variable_args(inst)?; self.typecheck_return(inst)?; + self.typecheck_special(inst, ctrl_type)?; Ok(()) } @@ -818,6 +817,57 @@ impl<'a> Verifier<'a> { Ok(()) } + // Check special-purpose type constraints that can't be expressed in the normal opcode + // constraints. + fn typecheck_special(&self, inst: Inst, ctrl_type: Type) -> Result { + match self.func.dfg[inst] { + ir::InstructionData::Unary { opcode, arg } => { + let arg_type = self.func.dfg.value_type(arg); + match opcode { + Opcode::Bextend | Opcode::Uextend | Opcode::Sextend | Opcode::Fpromote => { + if arg_type.lane_count() != ctrl_type.lane_count() { + return err!( + inst, + "input {} and output {} must have same number of lanes", + arg_type, + ctrl_type + ); + } + if arg_type.lane_bits() >= ctrl_type.lane_bits() { + return err!( + inst, + "input {} must be smaller than output {}", + arg_type, + ctrl_type + ); + } + } + Opcode::Breduce | Opcode::Ireduce | Opcode::Fdemote => { + if arg_type.lane_count() != ctrl_type.lane_count() { + return err!( + inst, + "input {} and output {} must have same number of lanes", + arg_type, + ctrl_type + ); + } + if arg_type.lane_bits() <= ctrl_type.lane_bits() { + return err!( + inst, + "input {} must be larger than output {}", + arg_type, + ctrl_type + ); + } + } + _ => {} + } + } + _ => {} + } + Ok(()) + } + fn cfg_integrity(&self, cfg: &ControlFlowGraph) -> Result { let mut expected_succs = BTreeSet::::new(); let mut got_succs = BTreeSet::::new(); diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index d96deafb0a..39a1d27253 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -486,33 +486,27 @@ pub fn translate_operator( /******************************* Unary Operators *************************************/ Operator::I32Clz => { let arg = state.pop1(); - let val = builder.ins().clz(arg); - state.push1(builder.ins().sextend(I32, val)); + state.push1(builder.ins().clz(arg)); } Operator::I64Clz => { let arg = state.pop1(); - let val = builder.ins().clz(arg); - state.push1(builder.ins().sextend(I64, val)); + state.push1(builder.ins().clz(arg)); } Operator::I32Ctz => { - let val = state.pop1(); - let short_res = builder.ins().ctz(val); - state.push1(builder.ins().sextend(I32, short_res)); + let arg = state.pop1(); + state.push1(builder.ins().ctz(arg)); } Operator::I64Ctz => { - let val = state.pop1(); - let short_res = builder.ins().ctz(val); - state.push1(builder.ins().sextend(I64, short_res)); + let arg = state.pop1(); + state.push1(builder.ins().ctz(arg)); } Operator::I32Popcnt => { let arg = state.pop1(); - let val = builder.ins().popcnt(arg); - state.push1(builder.ins().sextend(I32, val)); + state.push1(builder.ins().popcnt(arg)); } Operator::I64Popcnt => { let arg = state.pop1(); - let val = builder.ins().popcnt(arg); - state.push1(builder.ins().sextend(I64, val)); + state.push1(builder.ins().popcnt(arg)); } Operator::I64ExtendSI32 => { let val = state.pop1(); From 711e5cd64403e359150bf2c71fd6711b3025d848 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Sep 2017 08:53:49 -0700 Subject: [PATCH 1187/3084] Handle srem INT_MIN, -1 correctly. The x86_divmodx traps on integer overflow, but the srem instruction is not supposed to trap with a -1 divisor. Generate a legalization expansion for srem that special-cases the -1 divisor to simply return 0. --- lib/cretonne/meta/isa/intel/legalize.py | 10 ++--- lib/cretonne/src/isa/intel/enc_tables.rs | 54 ++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/legalize.py b/lib/cretonne/meta/isa/intel/legalize.py index 5987fd7221..5c883baf09 100644 --- a/lib/cretonne/meta/isa/intel/legalize.py +++ b/lib/cretonne/meta/isa/intel/legalize.py @@ -52,12 +52,10 @@ for ty in [i32, i64]: xhi << insts.sshr_imm(x, imm64(ty.lane_bits() - 1)), (a, dead) << x86.sdivmodx(x, xhi, y) )) - intel_expand.legalize( - a << insts.srem.bind(ty)(x, y), - Rtl( - xhi << insts.sshr_imm(x, imm64(ty.lane_bits() - 1)), - (dead, a) << x86.sdivmodx(x, xhi, y) - )) + +# The srem expansion requires custom code because srem INT_MIN, -1 is not +# allowed to trap. +intel_expand.custom_legalize(insts.srem, 'expand_srem') # Floating point condition codes. # diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index ad979a2e91..23c73014d5 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -14,6 +14,60 @@ use super::registers::*; include!(concat!(env!("OUT_DIR"), "/encoding-intel.rs")); include!(concat!(env!("OUT_DIR"), "/legalize-intel.rs")); +/// Expand the `srem` instruction using `x86_sdivmodx`. +fn expand_srem(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { + use ir::condcodes::IntCC; + + let (x, y) = match func.dfg[inst] { + ir::InstructionData::Binary { + opcode: ir::Opcode::Srem, + args, + } => (args[0], args[1]), + _ => panic!("Need srem: {}", func.dfg.display_inst(inst, None)), + }; + let old_ebb = func.layout.pp_ebb(inst); + + // EBB handling the -1 divisor case. + let minus_one = func.dfg.make_ebb(); + + // Final EBB with one argument representing the final result value. + let done = func.dfg.make_ebb(); + + // Move the `inst` result value onto the `done` EBB. + let result = func.dfg.first_result(inst); + let ty = func.dfg.value_type(result); + func.dfg.clear_results(inst); + func.dfg.attach_ebb_arg(done, result); + + let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); + + // Start by checking for a -1 divisor which needs to be handled specially. + let is_m1 = pos.ins().icmp_imm(IntCC::Equal, y, -1); + pos.ins().brnz(is_m1, minus_one, &[]); + + // Now it is safe to execute the `x86_sdivmodx` instruction which will still trap on division + // by zero. + let xhi = pos.ins().sshr_imm(x, ty.lane_bits() as i64 - 1); + let (_qout, rem) = pos.ins().x86_sdivmodx(x, xhi, y); + pos.ins().jump(done, &[rem]); + + // Now deal with the -1 divisor which always yields a 0 remainder. + pos.insert_ebb(minus_one); + let zero = pos.ins().iconst(ty, 0); + + // Recycle the original instruction as a jump. + pos.func.dfg.replace(inst).jump(done, &[zero]); + + // Finally insert a label for the completion. + pos.next_inst(); + pos.insert_ebb(done); + + cfg.recompute_ebb(pos.func, old_ebb); + cfg.recompute_ebb(pos.func, minus_one); + cfg.recompute_ebb(pos.func, done); +} + /// Expand the `fmin` and `fmax` instructions using the Intel `x86_fmin` and `x86_fmax` /// instructions. fn expand_minmax(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { From 45888ab84e536b5380653571ff2415a096d6df95 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Sep 2017 11:09:21 -0700 Subject: [PATCH 1188/3084] Reload for spilled call return values. When the return value from a call has been spilled, the reload pass needs to insert a spill instruction right after the call instruction which returns its results in registers. --- cranelift/filetests/regalloc/reload.cton | 19 +++++++++++++++++++ lib/cretonne/src/regalloc/liveness.rs | 6 ++++++ lib/cretonne/src/regalloc/reload.rs | 23 +++++++++++++++++++++++ 3 files changed, 48 insertions(+) create mode 100644 cranelift/filetests/regalloc/reload.cton diff --git a/cranelift/filetests/regalloc/reload.cton b/cranelift/filetests/regalloc/reload.cton new file mode 100644 index 0000000000..3db769eea8 --- /dev/null +++ b/cranelift/filetests/regalloc/reload.cton @@ -0,0 +1,19 @@ +test regalloc +isa riscv enable_e + +; regex: V=v\d+ + +; Check that we can handle a function return value that got spilled. +function %spill_return() -> i32 { + fn0 = function %foo() -> i32 native + +ebb0: + v0 = call fn0() + ; check: $(reg=$V) = call $fn0 + ; check: $v0 = spill $reg + v2 = call fn0() + ; check: $v2 = call $fn0 + return v0 + ; check: $(reload=$V) = fill $v0 + ; check: return $reload +} diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 2c3c430534..8faa8131c6 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -213,6 +213,12 @@ fn get_or_create<'a>( .operand_constraints(func.encodings[inst]) .and_then(|rc| rc.outs.get(rnum)) .map(Affinity::new) + .or_else(|| { + // If this is a call, get the return value affinity. + func.dfg.call_signature(inst).map(|sig| { + Affinity::abi(&func.dfg.signatures[sig].return_types[rnum], isa) + }) + }) .unwrap_or_default(); } ValueDef::Arg(ebb, num) => { diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 51aded85e0..aed4f040ba 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -183,6 +183,8 @@ impl<'a> Context<'a> { encoding: Encoding, tracker: &mut LiveValueTracker, ) { + self.cur.use_srcloc(inst); + // Get the operand constraints for `inst` that we are trying to satisfy. let constraints = self.encinfo.operand_constraints(encoding).expect( "Missing instruction encoding", @@ -253,6 +255,27 @@ impl<'a> Context<'a> { self.insert_spill(ebb, lv.value, reg); } } + + // Same thing for spilled call return values. + let retvals = &defs[constraints.outs.len()..]; + if !retvals.is_empty() { + let sig = self.cur.func.dfg.call_signature(inst).expect( + "Extra results on non-call instruction", + ); + for (i, lv) in retvals.iter().enumerate() { + let abi = self.cur.func.dfg.signatures[sig].return_types[i]; + debug_assert!(abi.location.is_reg()); + if lv.affinity.is_stack() { + let reg = self.cur.func.dfg.replace_result(lv.value, abi.value_type); + self.liveness.create_dead( + reg, + inst, + Affinity::abi(&abi, self.cur.isa), + ); + self.insert_spill(ebb, lv.value, reg); + } + } + } } // Find reload candidates for `inst` and add them to `self.condidates`. From 50ccd000a94cc7d701ae148a4deab862226fa545 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Sep 2017 12:41:37 -0700 Subject: [PATCH 1189/3084] Implement branch relaxation. Now that we have the legal_encodings iterator, it is simpler to find an alternative branch encoding with a better range. --- lib/cretonne/src/binemit/relaxation.rs | 27 ++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index 6ae9b4e4f7..4e6d1a2408 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -80,7 +80,8 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> Result CodeOffset { let inst = cur.current_inst().unwrap(); dbg!( "Relaxing [{}] {} for {:#x}-{:#x} range", encinfo.display(cur.func.encodings[inst]), - cur.func.dfg.display_inst(inst, None), + cur.func.dfg.display_inst(inst, isa), offset, dest_offset ); + + // Pick the first encoding that can handle the branch range. + let dfg = &cur.func.dfg; + let ctrl_type = dfg.ctrl_typevar(inst); + if let Some(enc) = isa.legal_encodings(dfg, &dfg[inst], ctrl_type).find( + |&enc| { + let range = encinfo.branch_range(enc).expect("Branch with no range"); + let in_range = range.contains(offset, dest_offset); + dbg!( + " trying [{}]: {}", + encinfo.display(enc), + if in_range { "OK" } else { "out of range" } + ); + in_range + }, + ) + { + cur.func.encodings[inst] = enc; + return encinfo.bytes(enc); + } + unimplemented!(); } From 86e22e7de5bf29ab42a127ee5e12bab862270ef3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Sep 2017 13:18:29 -0700 Subject: [PATCH 1190/3084] Add long-range encodings for conditional branches. The brz and brnz instructions get support for 32-bit jump displacements for long range branches. Also change the way branch ranges are specified on tail recipes for the Intel instructions. All branch displacements are relative to the end of the instruction, so just compute the branch range origin as the instruction size instead of trying to specify it in the tail recipe definitions. --- lib/cretonne/meta/isa/intel/encodings.py | 14 +++--- lib/cretonne/meta/isa/intel/recipes.py | 58 ++++++++++++++++++++---- 2 files changed, 56 insertions(+), 16 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 671b743827..9dee014771 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -298,18 +298,18 @@ I32.enc(base.jump, *r.jmpd(0xe9)) I64.enc(base.jump, *r.jmpb(0xeb)) I64.enc(base.jump, *r.jmpd(0xe9)) +# Note that the tjccd opcode will be prefixed with 0x0f. enc_i32_i64(base.brz, r.tjccb, 0x74) +enc_i32_i64(base.brz, r.tjccd, 0x84) enc_i32_i64(base.brnz, r.tjccb, 0x75) +enc_i32_i64(base.brnz, r.tjccd, 0x85) # Branch on a b1 value in a register only looks at the low 8 bits. See also # bint encodings below. -I32.enc(base.brz.b1, *r.t8jccb_abcd(0x74)) -I64.enc(base.brz.b1, *r.t8jccb_abcd.rex(0x74)) -I64.enc(base.brz.b1, *r.t8jccb_abcd(0x74)) -I32.enc(base.brnz.b1, *r.t8jccb_abcd(0x75)) -I64.enc(base.brnz.b1, *r.t8jccb_abcd.rex(0x75)) -I64.enc(base.brnz.b1, *r.t8jccb_abcd(0x75)) - +enc_flt(base.brz.b1, r.t8jccb_abcd, 0x74) +enc_flt(base.brz.b1, r.t8jccd_abcd, 0x84) +enc_flt(base.brnz.b1, r.t8jccb_abcd, 0x75) +enc_flt(base.brnz.b1, r.t8jccd_abcd, 0x85) # # Trap as ud2 diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index fe29735efe..67248dc86c 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -131,7 +131,7 @@ class TailRecipe: size, # type: int ins, # type: ConstraintSeq outs, # type: ConstraintSeq - branch_range=None, # type: BranchRange + branch_range=None, # type: int instp=None, # type: PredNode isap=None, # type: PredNode emit=None # type: str @@ -159,14 +159,21 @@ class TailRecipe: rrr = kwargs.get('rrr', 0) w = kwargs.get('w', 0) name, bits = decode_ops(ops, rrr, w) + size = len(ops) + self.size + + # All branch ranges are relative to the end of the instruction. + branch_range = None # type BranchRange + if self.branch_range is not None: + branch_range = (size, self.branch_range) + if name not in self.recipes: recipe = EncRecipe( name + self.name, self.format, - len(ops) + self.size, + size, ins=self.ins, outs=self.outs, - branch_range=self.branch_range, + branch_range=branch_range, instp=self.instp, isap=self.isap, emit=replace_put_op(self.emit, name)) @@ -193,14 +200,21 @@ class TailRecipe: w = kwargs.get('w', 0) name, bits = decode_ops(ops, rrr, w) name = 'Rex' + name + size = 1 + len(ops) + self.size + + # All branch ranges are relative to the end of the instruction. + branch_range = None # type BranchRange + if self.branch_range is not None: + branch_range = (size, self.branch_range) + if name not in self.recipes: self.recipes[name] = EncRecipe( name + self.name, self.format, - 1 + len(ops) + self.size, + size, ins=self.ins, outs=self.outs, - branch_range=self.branch_range, + branch_range=branch_range, instp=self.instp, isap=self.isap, emit=replace_put_op(self.emit, name)) @@ -656,7 +670,7 @@ ret = TailRecipe( # jmpb = TailRecipe( 'jmpb', Jump, size=1, ins=(), outs=(), - branch_range=(2, 8), + branch_range=8, emit=''' PUT_OP(bits, BASE_REX, sink); disp1(destination, func, sink); @@ -664,7 +678,7 @@ jmpb = TailRecipe( jmpd = TailRecipe( 'jmpd', Jump, size=4, ins=(), outs=(), - branch_range=(5, 32), + branch_range=32, emit=''' PUT_OP(bits, BASE_REX, sink); disp4(destination, func, sink); @@ -685,7 +699,7 @@ jmpd = TailRecipe( # Bits 8-15 control the test instruction which always has opcode byte 0x85. tjccb = TailRecipe( 'tjccb', Branch, size=1 + 2, ins=GPR, outs=(), - branch_range=(2, 8), + branch_range=8, emit=''' // test r, r. PUT_OP((bits & 0xff00) | 0x85, rex2(in_reg0, in_reg0), sink); @@ -695,13 +709,26 @@ tjccb = TailRecipe( disp1(destination, func, sink); ''') +tjccd = TailRecipe( + 'tjccd', Branch, size=1 + 6, ins=GPR, outs=(), + branch_range=32, + emit=''' + // test r, r. + PUT_OP((bits & 0xff00) | 0x85, rex2(in_reg0, in_reg0), sink); + modrm_rr(in_reg0, in_reg0, sink); + // Jcc instruction. + sink.put1(0x0f); + sink.put1(bits as u8); + disp4(destination, func, sink); + ''') + # 8-bit test-and-branch. # # Same as tjccb, but only looks at the low 8 bits of the register, for b1 # types. t8jccb_abcd = TailRecipe( 't8jccb_abcd', Branch, size=1 + 2, ins=ABCD, outs=(), - branch_range=(2, 8), + branch_range=8, emit=''' // test8 r, r. PUT_OP(0x84, rex2(in_reg0, in_reg0), sink); @@ -711,6 +738,19 @@ t8jccb_abcd = TailRecipe( disp1(destination, func, sink); ''') +t8jccd_abcd = TailRecipe( + 't8jccd_abcd', Branch, size=1 + 6, ins=ABCD, outs=(), + branch_range=32, + emit=''' + // test8 r, r. + PUT_OP(0x84, rex2(in_reg0, in_reg0), sink); + modrm_rr(in_reg0, in_reg0, sink); + // Jcc instruction. + sink.put1(0x0f); + sink.put1(bits as u8); + disp4(destination, func, sink); + ''') + # Comparison that produces a `b1` result in a GPR. # # This is a macro of a `cmp` instruction followed by a `setCC` instruction. From 51a6901a7fbccf99065c901181c1c9e43bed7099 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Sep 2017 14:38:30 -0700 Subject: [PATCH 1191/3084] Implement coloring::iterate_solution(). It can happen that the currently live registers are blocking a smaller register class completely, so the only way of solving the allocation problem is to turn some of the live-through registers into solver variables. When the quick_solve attempt fails, try to free up registers in the critical register class by turning live-through values into solver variables. --- cranelift/filetests/regalloc/iterate.cton | 107 ++++++++++++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 1 + lib/cretonne/src/isa/registers.rs | 7 +- lib/cretonne/src/regalloc/coloring.rs | 55 ++++++++++- lib/cretonne/src/regalloc/solver.rs | 17 ++++ 5 files changed, 181 insertions(+), 6 deletions(-) create mode 100644 cranelift/filetests/regalloc/iterate.cton diff --git a/cranelift/filetests/regalloc/iterate.cton b/cranelift/filetests/regalloc/iterate.cton new file mode 100644 index 0000000000..cacd9a4318 --- /dev/null +++ b/cranelift/filetests/regalloc/iterate.cton @@ -0,0 +1,107 @@ +test compile +set is_64bit +isa intel haswell + +function #00000009(i64 [%rdi], f32 [%xmm0], f64 [%xmm1], i32 [%rsi], i32 [%rdx], i64 vmctx [%r14]) -> i64 [%rax] spiderwasm { +ebb0(v0: i64, v1: f32, v2: f64, v3: i32, v4: i32, v5: i64): + v32 = iconst.i32 0 + v6 = bitcast.f32 v32 + v7 = iconst.i64 0 + v33 = iconst.i64 0 + v8 = bitcast.f64 v33 + v34 = iconst.i32 0xbe99_999a + v9 = bitcast.f32 v34 + v10 = iconst.i32 40 + v11 = iconst.i32 -7 + v35 = iconst.i32 0x40b0_0000 + v12 = bitcast.f32 v35 + v13 = iconst.i64 6 + v36 = iconst.i64 0x4020_0000_0000_0000 + v14 = bitcast.f64 v36 + v44 = iconst.i64 0 + v37 = icmp slt v0, v44 + brnz v37, ebb2 + v38 = fcvt_from_sint.f64 v0 + jump ebb3(v38) + +ebb2: + v45 = iconst.i32 1 + v39 = ushr.i64 v0, v45 + v40 = band_imm.i64 v0, 1 + v41 = bor v39, v40 + v42 = fcvt_from_sint.f64 v41 + v43 = fadd v42, v42 + jump ebb3(v43) + +ebb3(v15: f64): + v16 = fpromote.f64 v9 + v46 = uextend.i64 v10 + v17 = fcvt_from_sint.f64 v46 + v18 = fcvt_from_sint.f64 v11 + v19 = fpromote.f64 v12 + v54 = iconst.i64 0 + v47 = icmp.i64 slt v13, v54 + brnz v47, ebb4 + v48 = fcvt_from_sint.f64 v13 + jump ebb5(v48) + +ebb4: + v55 = iconst.i32 1 + v49 = ushr.i64 v13, v55 + v50 = band_imm.i64 v13, 1 + v51 = bor v49, v50 + v52 = fcvt_from_sint.f64 v51 + v53 = fadd v52, v52 + jump ebb5(v53) + +ebb5(v20: f64): + v63 = iconst.i64 0 + v56 = icmp.i64 slt v7, v63 + brnz v56, ebb6 + v57 = fcvt_from_sint.f64 v7 + jump ebb7(v57) + +ebb6: + v64 = iconst.i32 1 + v58 = ushr.i64 v7, v64 + v59 = band_imm.i64 v7, 1 + v60 = bor v58, v59 + v61 = fcvt_from_sint.f64 v60 + v62 = fadd v61, v61 + jump ebb7(v62) + +ebb7(v21: f64): + v22 = fadd v21, v14 + v23 = fadd.f64 v20, v22 + v24 = fadd.f64 v19, v23 + v25 = fadd.f64 v18, v24 + v26 = fadd.f64 v17, v25 + v27 = fadd.f64 v2, v26 + v28 = fadd.f64 v16, v27 + v29 = fadd.f64 v15, v28 + v30 = x86_cvtt2si.i64 v29 + v69 = iconst.i64 0x8000_0000_0000_0000 + v65 = icmp ne v30, v69 + brnz v65, ebb8 + v66 = fcmp uno v29, v29 + brz v66, ebb9 + trap bad_toint + +ebb9: + v70 = iconst.i64 0xc3e0_0000_0000_0000 + v67 = bitcast.f64 v70 + v68 = fcmp gt v67, v29 + brz v68, ebb10 + trap int_ovf + +ebb10: + jump ebb8 + +ebb8: + jump ebb1(v30) + +ebb1(v31: i64): + return v31 +} + + diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 9dee014771..eb4cee439e 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -119,6 +119,7 @@ enc_i32_i64(x86.udivmodx, r.div, 0xf7, rrr=6) enc_i32_i64(base.copy, r.umr, 0x89) enc_i32_i64(base.regmove, r.rmov, 0x89) +enc_flt(base.regmove.b1, r.rmov, 0x89) # Immediate instructions with sign-extended 8-bit and 32-bit immediate. for inst, rrr in [ diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index a43d5cfd62..a1de76d97e 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -167,7 +167,7 @@ impl RegClassData { } /// Returns true if `other` is a subclass of this register class. - /// A register class is considerd to be a subclass of itself. + /// A register class is considered to be a subclass of itself. pub fn has_subclass>(&self, other: RCI) -> bool { self.subclasses & (1 << other.into().0) != 0 } @@ -276,6 +276,11 @@ impl RegInfo { pub fn rc(&self, idx: RegClassIndex) -> RegClass { &self.classes[idx.index()] } + + /// Get the top-level register class containing `rc`. + pub fn toprc(&self, rc: RegClass) -> RegClass { + &self.classes[rc.toprc as usize] + } } /// Temporary object that holds enough information to print a register unit. diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index e46c95c101..ce6e22f756 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -367,9 +367,10 @@ impl<'a> Context<'a> { // Finally, we've fully programmed the constraint solver. // We expect a quick solution in most cases. - let mut output_regs = self.solver.quick_solve().unwrap_or_else( - |_| self.iterate_solution(), - ); + let mut output_regs = self.solver.quick_solve().unwrap_or_else(|rc| { + dbg!("quick_solve needs more registers in {}", rc); + self.iterate_solution(throughs, locations) + }); // The solution and/or fixed input constraints may require us to shuffle the set of live @@ -731,8 +732,52 @@ impl<'a> Context<'a> { /// /// We may need to move more registers around before a solution is possible. Use an iterative /// algorithm that adds one more variable until a solution can be found. - fn iterate_solution(&self) -> AllocatableSet { - unimplemented!(); + fn iterate_solution( + &mut self, + throughs: &[LiveValue], + locations: &mut ValueLocations, + ) -> AllocatableSet { + loop { + dbg!("real_solve for {} variables", self.solver.vars().len()); + let rc = match self.solver.real_solve() { + Ok(regs) => return regs, + Err(rc) => rc, + }; + + // Do we have any live-through `rc` registers that are not already variables? + assert!( + self.try_add_var(rc, throughs, locations), + "Ran out of registers in {}", + rc + ); + } + } + + /// Try to add an `rc` variable to the solver from the `throughs` set. + fn try_add_var( + &mut self, + rc: RegClass, + throughs: &[LiveValue], + locations: &mut ValueLocations, + ) -> bool { + dbg!("Trying to add a {} reg from {} values", rc, throughs.len()); + + for lv in throughs { + if let Affinity::Reg(rci) = lv.affinity { + let rc2 = self.reginfo.rc(rci); + let reg2 = self.divert.reg(lv.value, locations); + if rc.contains(reg2) && self.solver.can_add_var(lv.value, rc2, reg2) { + // The new variable gets to roam the whole top-level register class because + // it is not actually constrained by the instruction. We just want it out + // of the way. + let toprc = self.reginfo.toprc(rc2); + self.solver.add_var(lv.value, toprc, reg2, &self.reginfo); + return true; + } + } + } + + false } /// Emit `regmove` instructions as needed to move the live registers into place before the diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index c9215f4ebb..bfa6d22eb3 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -590,6 +590,18 @@ impl Solver { self.find_solution() } + /// Try harder to find a solution. + /// + /// Call this method after `quick_solve()` fails. + /// + /// This may return an error with a register class that has run out of registers. If registers + /// can be freed up in the starving class, this method can be called again after adding + /// variables for the freed registers. + pub fn real_solve(&mut self) -> Result { + // TODO: Sort variables to assign smallest register classes first. + self.find_solution() + } + /// Search for a solution with the current list of variables. /// /// If a solution was found, returns `Ok(regs)` with the set of available registers on the @@ -623,6 +635,11 @@ impl Solver { pub fn vars(&self) -> &[Variable] { &self.vars } + + /// Check if `value` can be added as a variable to help find a solution. + pub fn can_add_var(&mut self, _value: Value, constraint: RegClass, from: RegUnit) -> bool { + !self.regs_in.is_avail(constraint, from) + } } /// Interface for working with parallel copies once a solution has been found. From c82e68efeaaf676bb2a4aeaeec4bc9ddc397086b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Sep 2017 15:29:25 -0700 Subject: [PATCH 1192/3084] Eliminate the ABCD register class constaint in REX encodings. Some REX-less encodings require an ABCD input because they are looking at 8-bit registers. This constraint doesn't apply with a REX prefix where the low 8 bits of all registers are addressable. --- cranelift/filetests/regalloc/iterate.cton | 33 ++++++++++++++++++++ lib/cretonne/meta/isa/intel/recipes.py | 37 ++++++++++++++++------- 2 files changed, 59 insertions(+), 11 deletions(-) diff --git a/cranelift/filetests/regalloc/iterate.cton b/cranelift/filetests/regalloc/iterate.cton index cacd9a4318..15e0145407 100644 --- a/cranelift/filetests/regalloc/iterate.cton +++ b/cranelift/filetests/regalloc/iterate.cton @@ -104,4 +104,37 @@ ebb1(v31: i64): return v31 } +function #0000001a(i64 vmctx [%r14]) -> i64 [%rax] spiderwasm { + gv0 = vmctx+48 + sig0 = (i32 [%rdi], i64 [%rsi], i64 vmctx [%r14], i64 sigid [%rbx]) -> i64 [%rax] spiderwasm +ebb0(v0: i64): + v1 = iconst.i32 32 + v2 = iconst.i64 64 + v3 = iconst.i32 9 + v4 = iconst.i64 1063 + v5 = iadd_imm v0, 48 + v6 = load.i32 v5 + v7 = icmp uge v3, v6 + ; If we're unlucky, there are no ABCD registers available for v7 at this branch. + brz v7, ebb2 + trap oob + +ebb2: + v8 = load.i64 v5+8 + v9 = uextend.i64 v3 + v16 = iconst.i64 16 + v10 = imul v9, v16 + v11 = iadd v8, v10 + v12 = load.i64 v11 + brnz v12, ebb3 + trap icall_null + +ebb3: + v13 = load.i64 v11+8 + v14 = call_indirect.i64 sig0, v12(v1, v2, v13, v4) + jump ebb1(v14) + +ebb1(v15: i64): + return v15 +} diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 67248dc86c..fb5104381f 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -4,6 +4,7 @@ Intel Encoding recipes. from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt, IsEqual, Or +from cdsl.registers import RegClass from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare, FloatCompare @@ -96,13 +97,26 @@ def replace_put_op(emit, prefix): return emit.replace('PUT_OP', 'put_' + prefix.lower()) +# Register class mapping for no-REX instructions. +NOREX_MAP = { + GPR: GPR8, + FPR: FPR8 + } + +# Register class mapping for REX instructions. The ABCD constraint no longer +# applies. +REX_MAP = { + ABCD: GPR + } + + def map_regs( regs, # type: Sequence[OperandConstraint] - from_class, # type: OperandConstraint - to_class # type: OperandConstraint + mapping # type: Dict[RegClass, RegClass] ): # type: (...) -> Sequence[OperandConstraint] - return tuple(to_class if (reg is from_class) else reg for reg in regs) + return tuple(mapping.get(rc, rc) if isinstance(rc, RegClass) else rc + for rc in regs) class TailRecipe: @@ -178,11 +192,8 @@ class TailRecipe: isap=self.isap, emit=replace_put_op(self.emit, name)) - recipe.ins = map_regs(recipe.ins, GPR, GPR8) - recipe.ins = map_regs(recipe.ins, FPR, FPR8) - recipe.outs = map_regs(recipe.outs, GPR, GPR8) - recipe.outs = map_regs(recipe.outs, FPR, FPR8) - + recipe.ins = map_regs(recipe.ins, NOREX_MAP) + recipe.outs = map_regs(recipe.outs, NOREX_MAP) self.recipes[name] = recipe return (self.recipes[name], bits) @@ -208,7 +219,7 @@ class TailRecipe: branch_range = (size, self.branch_range) if name not in self.recipes: - self.recipes[name] = EncRecipe( + recipe = EncRecipe( name + self.name, self.format, size, @@ -218,6 +229,10 @@ class TailRecipe: instp=self.instp, isap=self.isap, emit=replace_put_op(self.emit, name)) + recipe.ins = map_regs(recipe.ins, REX_MAP) + recipe.outs = map_regs(recipe.outs, REX_MAP) + self.recipes[name] = recipe + return (self.recipes[name], bits) @staticmethod @@ -731,7 +746,7 @@ t8jccb_abcd = TailRecipe( branch_range=8, emit=''' // test8 r, r. - PUT_OP(0x84, rex2(in_reg0, in_reg0), sink); + PUT_OP((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink); modrm_rr(in_reg0, in_reg0, sink); // Jcc instruction. sink.put1(bits as u8); @@ -743,7 +758,7 @@ t8jccd_abcd = TailRecipe( branch_range=32, emit=''' // test8 r, r. - PUT_OP(0x84, rex2(in_reg0, in_reg0), sink); + PUT_OP((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink); modrm_rr(in_reg0, in_reg0, sink); // Jcc instruction. sink.put1(0x0f); From 5f56f8125122353475fccb6da6f96ca63cea5298 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 29 Sep 2017 15:54:06 -0700 Subject: [PATCH 1193/3084] Resolve all value aliases when computing live ranges. Value aliases are only in the way during register allocation, so make sure they are all dead as we enter the register allocation passes. --- cranelift/filetests/regalloc/aliases.cton | 36 +++++++++++++++++++++++ lib/cretonne/src/regalloc/liveness.rs | 5 +++- 2 files changed, 40 insertions(+), 1 deletion(-) create mode 100644 cranelift/filetests/regalloc/aliases.cton diff --git a/cranelift/filetests/regalloc/aliases.cton b/cranelift/filetests/regalloc/aliases.cton new file mode 100644 index 0000000000..575fedd923 --- /dev/null +++ b/cranelift/filetests/regalloc/aliases.cton @@ -0,0 +1,36 @@ +test compile +set is_64bit +isa intel haswell + +function %value_aliases(i32, f32, i64 vmctx) spiderwasm { + gv0 = vmctx + heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v1: f32, v2: i64): + v3 = iconst.i32 0 + jump ebb3(v3) + +ebb3(v4: i32): + v5 = heap_addr.i64 heap0, v4, 1 + v6 = load.f32 v5 + v7 -> v1 + v8 = fdiv v6, v7 + v9 = heap_addr.i64 heap0, v4, 1 + store v8, v9 + v10 = iconst.i32 4 + v11 = iadd v4, v10 + v12 -> v0 + v13 = icmp ult v11, v12 + v14 = bint.i32 v13 + brnz v14, ebb3(v11) + jump ebb4 + +ebb4: + jump ebb2 + +ebb2: + jump ebb1 + +ebb1: + return +} diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 8faa8131c6..5dd6ce71bb 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -366,7 +366,7 @@ impl Liveness { /// Compute the live ranges of all SSA values used in `func`. /// This clears out any existing analysis stored in this data structure. - pub fn compute(&mut self, isa: &TargetIsa, func: &Function, cfg: &ControlFlowGraph) { + pub fn compute(&mut self, isa: &TargetIsa, func: &mut Function, cfg: &ControlFlowGraph) { self.ranges.clear(); // Get ISA data structures used for computing live range affinities. @@ -386,6 +386,9 @@ impl Liveness { } for inst in func.layout.ebb_insts(ebb) { + // Eliminate all value aliases, they would confuse the register allocator. + func.dfg.resolve_aliases_in_arguments(inst); + // Make sure we have created live ranges for dead defs. // TODO: When we implement DCE, we can use the absence of a live range to indicate // an unused value. From 3a34c35f953b5fe1bd6c4621072fb9c862748d6f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 2 Oct 2017 14:43:30 -0700 Subject: [PATCH 1194/3084] Don't swap the arguments to the select instruction. Both WebAssembly and Cretonne use true-value, false-value. --- lib/wasm/src/code_translator.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 39a1d27253..24eae598a2 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -95,7 +95,7 @@ pub fn translate_operator( } Operator::Select => { let (arg1, arg2, cond) = state.pop3(); - state.push1(builder.ins().select(cond, arg2, arg1)); + state.push1(builder.ins().select(cond, arg1, arg2)); } Operator::Nop => { // We do nothing From 1efa670f6040e1dc9b0496ff67b968e0149adde9 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 2 Oct 2017 16:57:14 -0700 Subject: [PATCH 1195/3084] Do a full compile in 'cton-util wasm'. This removes the `optimize` option, as one can do that with `--set`, eg. `--set opt_level=best`. And it adds an option to print the compilation output. --- cranelift/src/cton-util.rs | 19 +++++---- cranelift/src/wasm.rs | 79 +++++++++++++++++++------------------- 2 files changed, 52 insertions(+), 46 deletions(-) diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index fbb0603cf7..ac0a25c698 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -30,13 +30,16 @@ Usage: cton-util cat ... cton-util filecheck [-v] cton-util print-cfg ... - cton-util wasm [-cvo] [--set ]... [--isa ] ... + cton-util wasm [-ctvp] [--set ]... [--isa ] ... cton-util --help | --version Options: -v, --verbose be more verbose - -c, --check checks the correctness of Cretonne IL translated from WebAssembly - -o, --optimize runs otpimization passes on translated WebAssembly functions + -t, --just-decode + just decode WebAssembly to Cretonne IL + -c, --check-translation + just checks the correctness of Cretonne IL translated from WebAssembly + -p, --print print the resulting Cretonne IL -h, --help print this help message --set= configure Cretonne settings --isa= specify the Cretonne ISA @@ -52,8 +55,9 @@ struct Args { cmd_print_cfg: bool, cmd_wasm: bool, arg_file: Vec, - flag_check: bool, - flag_optimize: bool, + flag_just_decode: bool, + flag_check_translation: bool, + flag_print: bool, flag_verbose: bool, flag_set: Vec, flag_isa: String, @@ -86,8 +90,9 @@ fn cton_util() -> CommandResult { wasm::run( args.arg_file, args.flag_verbose, - args.flag_optimize, - args.flag_check, + args.flag_just_decode, + args.flag_check_translation, + args.flag_print, args.flag_set, args.flag_isa, ) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index e665822e60..7f18130d67 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -8,7 +8,6 @@ use cton_wasm::{translate_module, DummyRuntime, WasmRuntime}; use cton_reader::{parse_options, Location}; use std::path::PathBuf; use cretonne::Context; -use cretonne::verifier; use cretonne::settings::{self, FlagsOrIsa}; use cretonne::isa::{self, TargetIsa}; use std::fs::File; @@ -54,8 +53,9 @@ enum OwnedFlagsOrIsa { pub fn run( files: Vec, flag_verbose: bool, - flag_optimize: bool, - flag_check: bool, + flag_just_decode: bool, + flag_check_translation: bool, + flag_print: bool, flag_set: Vec, flag_isa: String, ) -> Result<(), String> { @@ -87,8 +87,9 @@ pub fn run( let name = String::from(path.as_os_str().to_string_lossy()); handle_module( flag_verbose, - flag_optimize, - flag_check, + flag_just_decode, + flag_check_translation, + flag_print, path.to_path_buf(), name, &fisa, @@ -99,8 +100,9 @@ pub fn run( fn handle_module( flag_verbose: bool, - flag_optimize: bool, - flag_check: bool, + flag_just_decode: bool, + flag_check_translation: bool, + flag_print: bool, path: PathBuf, name: String, fisa: &FlagsOrIsa, @@ -111,7 +113,7 @@ fn handle_module( terminal.reset().unwrap(); vprintln!(flag_verbose, "\"{}\"", name); terminal.fg(term::color::MAGENTA).unwrap(); - vprint!(flag_verbose, "Translating..."); + vprint!(flag_verbose, "Translating... "); terminal.reset().unwrap(); let data = match path.extension() { None => { @@ -155,43 +157,42 @@ fn handle_module( translate_module(&data, runtime)? }; terminal.fg(term::color::GREEN).unwrap(); - vprintln!(flag_verbose, " ok"); + vprintln!(flag_verbose, "ok"); terminal.reset().unwrap(); - if flag_check { - terminal.fg(term::color::MAGENTA).unwrap(); - vprint!(flag_verbose, "Checking... "); - terminal.reset().unwrap(); - for func in &translation.functions { - verifier::verify_function(func, *fisa).map_err(|err| { - pretty_verifier_error(func, fisa.isa, err) - })?; - } - terminal.fg(term::color::GREEN).unwrap(); - vprintln!(flag_verbose, " ok"); - terminal.reset().unwrap(); + if flag_just_decode { + return Ok(()); } - if flag_optimize { - terminal.fg(term::color::MAGENTA).unwrap(); - vprint!(flag_verbose, "Optimizing... "); - terminal.reset().unwrap(); - for func in &translation.functions { - let mut context = Context::new(); - context.func = func.clone(); + terminal.fg(term::color::MAGENTA).unwrap(); + if flag_check_translation { + vprint!(flag_verbose, "Checking... "); + } else { + vprint!(flag_verbose, "Compiling... "); + } + terminal.reset().unwrap(); + for func in &translation.functions { + let mut context = Context::new(); + context.func = func.clone(); + if flag_check_translation { context.verify(*fisa).map_err(|err| { pretty_verifier_error(&context.func, fisa.isa, err) })?; - context.flowgraph(); - context.compute_loop_analysis(); - context.licm(*fisa).map_err(|err| { - pretty_error(&context.func, fisa.isa, err) - })?; - context.simple_gvn(*fisa).map_err(|err| { - pretty_error(&context.func, fisa.isa, err) - })?; + continue; + } + if let Some(isa) = fisa.isa { + context.compile(isa).map_err(|err| { + pretty_error(&context.func, fisa.isa, err) + })?; + } else { + return Err(String::from("compilation requires a target isa")); + } + if flag_print { + vprintln!(flag_verbose, ""); + println!("{}", context.func.display(fisa.isa)); + vprintln!(flag_verbose, ""); } - terminal.fg(term::color::GREEN).unwrap(); - vprintln!(flag_verbose, " ok"); - terminal.reset().unwrap(); } + terminal.fg(term::color::GREEN).unwrap(); + vprintln!(flag_verbose, "ok"); + terminal.reset().unwrap(); Ok(()) } From eba55cb1da8044b33dbff273403f4fd5bf21997d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Oct 2017 09:31:42 -0700 Subject: [PATCH 1196/3084] Make the "wat2wasm" error message not panic. --- cranelift/src/wasm.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 7f18130d67..dc9c6ba188 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -139,8 +139,7 @@ fn handle_module( return Err(String::from("wat2wasm not found")); } else { return Err(String::from(e.description())); - }) - .unwrap(); + })?; read_wasm_file(file_path).map_err(|err| { String::from(err.description()) })? From 25b8b45a415196be0b4ebcc54c9d5c26dc13339e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Oct 2017 09:34:41 -0700 Subject: [PATCH 1197/3084] Avoid unnecessary BufReaders. --- cranelift/src/wasm.rs | 6 ++---- lib/wasm/tests/testsuite.rs | 6 ++---- 2 files changed, 4 insertions(+), 8 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index dc9c6ba188..81331faa40 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -13,7 +13,6 @@ use cretonne::isa::{self, TargetIsa}; use std::fs::File; use std::error::Error; use std::io; -use std::io::BufReader; use std::io::prelude::*; use std::path::Path; use std::process::Command; @@ -39,9 +38,8 @@ macro_rules! vprint { fn read_wasm_file(path: PathBuf) -> Result, io::Error> { let mut buf: Vec = Vec::new(); - let file = File::open(path)?; - let mut buf_reader = BufReader::new(file); - buf_reader.read_to_end(&mut buf)?; + let mut file = File::open(path)?; + file.read_to_end(&mut buf)?; Ok(buf) } diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/testsuite.rs index b5fe5997e8..d992de753e 100644 --- a/lib/wasm/tests/testsuite.rs +++ b/lib/wasm/tests/testsuite.rs @@ -8,7 +8,6 @@ use std::fs::File; use std::error::Error; use std::io; use std::str; -use std::io::BufReader; use std::io::prelude::*; use std::process::Command; use std::fs; @@ -43,9 +42,8 @@ fn return_at_end() { fn read_wasm_file(path: PathBuf) -> Result, io::Error> { let mut buf: Vec = Vec::new(); - let file = File::open(path)?; - let mut buf_reader = BufReader::new(file); - buf_reader.read_to_end(&mut buf)?; + let mut file = File::open(path)?; + file.read_to_end(&mut buf)?; Ok(buf) } From 7c7b1651d89f2f968f337e78b1aa14abdb9eb2fd Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 2 Oct 2017 16:57:14 -0700 Subject: [PATCH 1198/3084] Do a full compile in 'cton-util wasm'. This removes the `optimize` option, as one can do that with `--set`, eg. `--set opt_level=best`. And it adds an option to print the compilation output. And, it enables simple_gvn and licm for opt_level=best. --- lib/cretonne/src/context.rs | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index cd402873e8..4b62721fff 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -18,7 +18,7 @@ use isa::TargetIsa; use legalize_function; use regalloc; use result::{CtonError, CtonResult}; -use settings::FlagsOrIsa; +use settings::{FlagsOrIsa, OptLevel}; use verifier; use simple_gvn::do_simple_gvn; use licm::do_licm; @@ -68,6 +68,12 @@ impl Context { self.compute_cfg(); self.legalize(isa)?; + if isa.flags().opt_level() == OptLevel::Best { + self.compute_domtree(); + self.compute_loop_analysis(); + self.licm(isa)?; + self.simple_gvn(isa)?; + } self.compute_domtree(); self.regalloc(isa)?; self.prologue_epilogue(isa)?; From 12ab4cd9148a88b6e38466aaf901bb6187ebb15c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Oct 2017 06:46:38 -0700 Subject: [PATCH 1199/3084] Add a cton-util compile command. --- cranelift/src/compile.rs | 82 ++++++++++++++++++++++++++++++++++++++ cranelift/src/cton-util.rs | 5 +++ 2 files changed, 87 insertions(+) create mode 100644 cranelift/src/compile.rs diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs new file mode 100644 index 0000000000..2db5d21f85 --- /dev/null +++ b/cranelift/src/compile.rs @@ -0,0 +1,82 @@ +//! CLI tool to compile cretonne IL into native code. +//! +//! Reads IR files into Cretonne IL and compiles it. + +use cton_reader::{parse_options, Location, parse_functions}; +use std::path::PathBuf; +use cretonne::Context; +use cretonne::settings::{self, FlagsOrIsa}; +use cretonne::isa::{self, TargetIsa}; +use std::path::Path; +use utils::{pretty_error, read_to_string}; + +enum OwnedFlagsOrIsa { + Flags(settings::Flags), + Isa(Box), +} + +pub fn run( + files: Vec, + flag_print: bool, + flag_set: Vec, + flag_isa: String, +) -> Result<(), String> { + let mut flag_builder = settings::builder(); + parse_options( + flag_set.iter().map(|x| x.as_str()), + &mut flag_builder, + &Location { line_number: 0 }, + ).map_err(|err| err.to_string())?; + + let mut words = flag_isa.trim().split_whitespace(); + // Look for `isa foo`. + let owned_fisa = if let Some(isa_name) = words.next() { + let isa_builder = isa::lookup(isa_name).map_err(|err| match err { + isa::LookupError::Unknown => format!("unknown ISA '{}'", isa_name), + isa::LookupError::Unsupported => format!("support for ISA '{}' not enabled", isa_name), + })?; + OwnedFlagsOrIsa::Isa(isa_builder.finish(settings::Flags::new(&flag_builder))) + } else { + OwnedFlagsOrIsa::Flags(settings::Flags::new(&flag_builder)) + }; + let fisa = match owned_fisa { + OwnedFlagsOrIsa::Flags(ref flags) => FlagsOrIsa::from(flags), + OwnedFlagsOrIsa::Isa(ref isa) => FlagsOrIsa::from(&**isa), + }; + + for filename in files { + let path = Path::new(&filename); + let name = String::from(path.as_os_str().to_string_lossy()); + handle_module(flag_print, path.to_path_buf(), name, &fisa)?; + } + Ok(()) +} + +fn handle_module( + flag_print: bool, + path: PathBuf, + name: String, + fisa: &FlagsOrIsa, +) -> Result<(), String> { + let buffer = read_to_string(&path).map_err( + |e| format!("{}: {}", name, e), + )?; + let items = parse_functions(&buffer).map_err( + |e| format!("{}: {}", name, e), + )?; + for func in items.into_iter() { + let mut context = Context::new(); + context.func = func; + if let Some(isa) = fisa.isa { + context.compile(isa).map_err(|err| { + pretty_error(&context.func, fisa.isa, err) + })?; + } else { + return Err(String::from("compilation requires a target isa")); + } + if flag_print { + println!("{}", context.func.display(fisa.isa)); + } + } + Ok(()) +} diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index ac0a25c698..b08fa8cde3 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -21,6 +21,7 @@ mod cat; mod print_cfg; mod rsfilecheck; mod wasm; +mod compile; const USAGE: &str = " Cretonne code generator utility @@ -30,6 +31,7 @@ Usage: cton-util cat ... cton-util filecheck [-v] cton-util print-cfg ... + cton-util compile [-vp] [--set ]... [--isa ] ... cton-util wasm [-ctvp] [--set ]... [--isa ] ... cton-util --help | --version @@ -53,6 +55,7 @@ struct Args { cmd_cat: bool, cmd_filecheck: bool, cmd_print_cfg: bool, + cmd_compile: bool, cmd_wasm: bool, arg_file: Vec, flag_just_decode: bool, @@ -86,6 +89,8 @@ fn cton_util() -> CommandResult { rsfilecheck::run(args.arg_file, args.flag_verbose) } else if args.cmd_print_cfg { print_cfg::run(args.arg_file) + } else if args.cmd_compile { + compile::run(args.arg_file, args.flag_print, args.flag_set, args.flag_isa) } else if args.cmd_wasm { wasm::run( args.arg_file, From f0644186521618024ab04ffd78287f404fe1f50f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Oct 2017 09:05:23 -0700 Subject: [PATCH 1200/3084] Refactor set/isa parsing into a utility function. --- cranelift/src/compile.rs | 39 +++++-------------------------- cranelift/src/utils.rs | 50 ++++++++++++++++++++++++++++++++++++++-- cranelift/src/wasm.rs | 40 +++++--------------------------- 3 files changed, 60 insertions(+), 69 deletions(-) diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 2db5d21f85..f9f718685d 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -2,18 +2,12 @@ //! //! Reads IR files into Cretonne IL and compiles it. -use cton_reader::{parse_options, Location, parse_functions}; +use cton_reader::parse_functions; use std::path::PathBuf; use cretonne::Context; -use cretonne::settings::{self, FlagsOrIsa}; -use cretonne::isa::{self, TargetIsa}; +use cretonne::settings::FlagsOrIsa; use std::path::Path; -use utils::{pretty_error, read_to_string}; - -enum OwnedFlagsOrIsa { - Flags(settings::Flags), - Isa(Box), -} +use utils::{pretty_error, read_to_string, parse_sets_and_isa}; pub fn run( files: Vec, @@ -21,33 +15,12 @@ pub fn run( flag_set: Vec, flag_isa: String, ) -> Result<(), String> { - let mut flag_builder = settings::builder(); - parse_options( - flag_set.iter().map(|x| x.as_str()), - &mut flag_builder, - &Location { line_number: 0 }, - ).map_err(|err| err.to_string())?; - - let mut words = flag_isa.trim().split_whitespace(); - // Look for `isa foo`. - let owned_fisa = if let Some(isa_name) = words.next() { - let isa_builder = isa::lookup(isa_name).map_err(|err| match err { - isa::LookupError::Unknown => format!("unknown ISA '{}'", isa_name), - isa::LookupError::Unsupported => format!("support for ISA '{}' not enabled", isa_name), - })?; - OwnedFlagsOrIsa::Isa(isa_builder.finish(settings::Flags::new(&flag_builder))) - } else { - OwnedFlagsOrIsa::Flags(settings::Flags::new(&flag_builder)) - }; - let fisa = match owned_fisa { - OwnedFlagsOrIsa::Flags(ref flags) => FlagsOrIsa::from(flags), - OwnedFlagsOrIsa::Isa(ref isa) => FlagsOrIsa::from(&**isa), - }; + let parsed = parse_sets_and_isa(flag_set, flag_isa)?; for filename in files { let path = Path::new(&filename); let name = String::from(path.as_os_str().to_string_lossy()); - handle_module(flag_print, path.to_path_buf(), name, &fisa)?; + handle_module(flag_print, path.to_path_buf(), name, parsed.as_fisa())?; } Ok(()) } @@ -56,7 +29,7 @@ fn handle_module( flag_print: bool, path: PathBuf, name: String, - fisa: &FlagsOrIsa, + fisa: FlagsOrIsa, ) -> Result<(), String> { let buffer = read_to_string(&path).map_err( |e| format!("{}: {}", name, e), diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index a5f9940eb4..6a962d04f7 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -4,13 +4,16 @@ use cretonne::ir::entities::AnyEntity; use cretonne::{ir, verifier}; use cretonne::result::CtonError; use cretonne::isa::TargetIsa; +use cretonne::settings::{self, FlagsOrIsa}; +use cretonne::isa; +use cton_reader::{parse_options, Location}; use std::fmt::Write; use std::fs::File; -use std::io::{Result, Read}; +use std::io::{self, Read}; use std::path::Path; /// Read an entire file into a string. -pub fn read_to_string>(path: P) -> Result { +pub fn read_to_string>(path: P) -> io::Result { let mut file = File::open(path)?; let mut buffer = String::new(); file.read_to_string(&mut buffer)?; @@ -62,6 +65,49 @@ pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CtonError } } +/// Like FlagsOrIsa, but holds ownership. +pub enum OwnedFlagsOrIsa { + Flags(settings::Flags), + Isa(Box), +} + +impl OwnedFlagsOrIsa { + /// Produce a FlagsOrIsa reference. + pub fn as_fisa(&self) -> FlagsOrIsa { + match *self { + OwnedFlagsOrIsa::Flags(ref flags) => FlagsOrIsa::from(flags), + OwnedFlagsOrIsa::Isa(ref isa) => FlagsOrIsa::from(&**isa), + } + } +} + +/// Parse "set" and "isa" commands. +pub fn parse_sets_and_isa( + flag_set: Vec, + flag_isa: String, +) -> Result { + let mut flag_builder = settings::builder(); + parse_options( + flag_set.iter().map(|x| x.as_str()), + &mut flag_builder, + &Location { line_number: 0 }, + ).map_err(|err| err.to_string())?; + + let mut words = flag_isa.trim().split_whitespace(); + // Look for `isa foo`. + if let Some(isa_name) = words.next() { + let isa_builder = isa::lookup(isa_name).map_err(|err| match err { + isa::LookupError::Unknown => format!("unknown ISA '{}'", isa_name), + isa::LookupError::Unsupported => format!("support for ISA '{}' not enabled", isa_name), + })?; + Ok(OwnedFlagsOrIsa::Isa( + isa_builder.finish(settings::Flags::new(&flag_builder)), + )) + } else { + Ok(OwnedFlagsOrIsa::Flags(settings::Flags::new(&flag_builder))) + } +} + #[test] fn test_match_directive() { assert_eq!(match_directive("; foo: bar ", "foo:"), Some("bar")); diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 81331faa40..460d4280f5 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -5,11 +5,9 @@ //! and tables, then emitting the translated code with hardcoded addresses to memory. use cton_wasm::{translate_module, DummyRuntime, WasmRuntime}; -use cton_reader::{parse_options, Location}; use std::path::PathBuf; use cretonne::Context; -use cretonne::settings::{self, FlagsOrIsa}; -use cretonne::isa::{self, TargetIsa}; +use cretonne::settings::FlagsOrIsa; use std::fs::File; use std::error::Error; use std::io; @@ -18,7 +16,7 @@ use std::path::Path; use std::process::Command; use tempdir::TempDir; use term; -use utils::{pretty_verifier_error, pretty_error}; +use utils::{pretty_verifier_error, pretty_error, parse_sets_and_isa}; macro_rules! vprintln { ($x: expr, $($tts:tt)*) => { @@ -43,11 +41,6 @@ fn read_wasm_file(path: PathBuf) -> Result, io::Error> { Ok(buf) } -enum OwnedFlagsOrIsa { - Flags(settings::Flags), - Isa(Box), -} - pub fn run( files: Vec, flag_verbose: bool, @@ -57,28 +50,7 @@ pub fn run( flag_set: Vec, flag_isa: String, ) -> Result<(), String> { - let mut flag_builder = settings::builder(); - parse_options( - flag_set.iter().map(|x| x.as_str()), - &mut flag_builder, - &Location { line_number: 0 }, - ).map_err(|err| err.to_string())?; - - let mut words = flag_isa.trim().split_whitespace(); - // Look for `isa foo`. - let owned_fisa = if let Some(isa_name) = words.next() { - let isa_builder = isa::lookup(isa_name).map_err(|err| match err { - isa::LookupError::Unknown => format!("unknown ISA '{}'", isa_name), - isa::LookupError::Unsupported => format!("support for ISA '{}' not enabled", isa_name), - })?; - OwnedFlagsOrIsa::Isa(isa_builder.finish(settings::Flags::new(&flag_builder))) - } else { - OwnedFlagsOrIsa::Flags(settings::Flags::new(&flag_builder)) - }; - let fisa = match owned_fisa { - OwnedFlagsOrIsa::Flags(ref flags) => FlagsOrIsa::from(flags), - OwnedFlagsOrIsa::Isa(ref isa) => FlagsOrIsa::from(&**isa), - }; + let parsed = parse_sets_and_isa(flag_set, flag_isa)?; for filename in files { let path = Path::new(&filename); @@ -90,7 +62,7 @@ pub fn run( flag_print, path.to_path_buf(), name, - &fisa, + parsed.as_fisa(), )?; } Ok(()) @@ -103,7 +75,7 @@ fn handle_module( flag_print: bool, path: PathBuf, name: String, - fisa: &FlagsOrIsa, + fisa: FlagsOrIsa, ) -> Result<(), String> { let mut terminal = term::stdout().unwrap(); terminal.fg(term::color::YELLOW).unwrap(); @@ -170,7 +142,7 @@ fn handle_module( let mut context = Context::new(); context.func = func.clone(); if flag_check_translation { - context.verify(*fisa).map_err(|err| { + context.verify(fisa).map_err(|err| { pretty_verifier_error(&context.func, fisa.isa, err) })?; continue; From ba14499fe90e5da7599160333e7415286e3e3c02 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Oct 2017 09:09:48 -0700 Subject: [PATCH 1201/3084] Factor out the code for reading a file into a utility function. --- cranelift/src/utils.rs | 8 ++++++++ cranelift/src/wasm.rs | 17 +++++------------ 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index 6a962d04f7..c9cae4cd58 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -20,6 +20,14 @@ pub fn read_to_string>(path: P) -> io::Result { Ok(buffer) } +/// Read an entire file into a vector of bytes. +pub fn read_to_end>(path: P) -> io::Result> { + let mut file = File::open(path)?; + let mut buffer = Vec::new(); + file.read_to_end(&mut buffer)?; + Ok(buffer) +} + /// Look for a directive in a comment string. /// The directive is of the form "foo:" and should follow the leading `;` in the comment: /// diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 460d4280f5..d6bf98110b 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -16,7 +16,7 @@ use std::path::Path; use std::process::Command; use tempdir::TempDir; use term; -use utils::{pretty_verifier_error, pretty_error, parse_sets_and_isa}; +use utils::{pretty_verifier_error, pretty_error, parse_sets_and_isa, read_to_end}; macro_rules! vprintln { ($x: expr, $($tts:tt)*) => { @@ -34,13 +34,6 @@ macro_rules! vprint { } } -fn read_wasm_file(path: PathBuf) -> Result, io::Error> { - let mut buf: Vec = Vec::new(); - let mut file = File::open(path)?; - file.read_to_end(&mut buf)?; - Ok(buf) -} - pub fn run( files: Vec, flag_verbose: bool, @@ -92,7 +85,7 @@ fn handle_module( Some(ext) => { match ext.to_str() { Some("wasm") => { - read_wasm_file(path.clone()).map_err(|err| { + read_to_end(path.clone()).map_err(|err| { String::from(err.description()) })? } @@ -110,9 +103,9 @@ fn handle_module( } else { return Err(String::from(e.description())); })?; - read_wasm_file(file_path).map_err(|err| { - String::from(err.description()) - })? + read_to_end(file_path).map_err( + |err| String::from(err.description()), + )? } None | Some(&_) => { return Err(String::from("the file extension is not wasm or wat")); From d857aacec3ecf6ee9a20d449a8c4739044d82100 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Oct 2017 09:19:55 -0700 Subject: [PATCH 1202/3084] Sniff the wasm magic bytes, rather than relying on the filename extension. --- cranelift/src/wasm.rs | 57 ++++++++++++++++--------------------------- 1 file changed, 21 insertions(+), 36 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index d6bf98110b..840f108480 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -11,7 +11,6 @@ use cretonne::settings::FlagsOrIsa; use std::fs::File; use std::error::Error; use std::io; -use std::io::prelude::*; use std::path::Path; use std::process::Command; use tempdir::TempDir; @@ -78,41 +77,27 @@ fn handle_module( terminal.fg(term::color::MAGENTA).unwrap(); vprint!(flag_verbose, "Translating... "); terminal.reset().unwrap(); - let data = match path.extension() { - None => { - return Err(String::from("the file extension is not wasm or wat")); - } - Some(ext) => { - match ext.to_str() { - Some("wasm") => { - read_to_end(path.clone()).map_err(|err| { - String::from(err.description()) - })? - } - Some("wat") => { - let tmp_dir = TempDir::new("cretonne-wasm").unwrap(); - let file_path = tmp_dir.path().join("module.wasm"); - File::create(file_path.clone()).unwrap(); - Command::new("wat2wasm") - .arg(path.clone()) - .arg("-o") - .arg(file_path.to_str().unwrap()) - .output() - .or_else(|e| if let io::ErrorKind::NotFound = e.kind() { - return Err(String::from("wat2wasm not found")); - } else { - return Err(String::from(e.description())); - })?; - read_to_end(file_path).map_err( - |err| String::from(err.description()), - )? - } - None | Some(&_) => { - return Err(String::from("the file extension is not wasm or wat")); - } - } - } - }; + let mut data = read_to_end(path.clone()).map_err(|err| { + String::from(err.description()) + })?; + if !data.starts_with(&[b'\0', b'a', b's', b'm']) { + let tmp_dir = TempDir::new("cretonne-wasm").unwrap(); + let file_path = tmp_dir.path().join("module.wasm"); + File::create(file_path.clone()).unwrap(); + Command::new("wat2wasm") + .arg(path.clone()) + .arg("-o") + .arg(file_path.to_str().unwrap()) + .output() + .or_else(|e| if let io::ErrorKind::NotFound = e.kind() { + return Err(String::from("wat2wasm not found")); + } else { + return Err(String::from(e.description())); + })?; + data = read_to_end(file_path).map_err( + |err| String::from(err.description()), + )?; + } let mut dummy_runtime = DummyRuntime::with_flags(fisa.flags.clone()); let translation = { let runtime: &mut WasmRuntime = &mut dummy_runtime; From d585f094236de21b80c7f27e8d0e71995479754e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Oct 2017 10:05:46 -0700 Subject: [PATCH 1203/3084] Don't silently swallow lexer errors. --- lib/reader/src/parser.rs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index be86302645..e334319e91 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -807,6 +807,11 @@ impl<'a> Parser<'a> { while self.token().is_some() { list.push(self.parse_function(unique_isa)?); } + if let Some(err) = self.lex_error { + return match err { + lexer::Error::InvalidChar => err!(self.loc, "invalid character"), + }; + } Ok(list) } From 374ed3a07b0c1a91b3ef5a3e53e071cdb7d0115b Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Oct 2017 11:03:06 -0700 Subject: [PATCH 1204/3084] Fix dominator-tree queries on unreachable nodes. --- lib/cretonne/src/dominator_tree.rs | 34 +++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 5f342baf89..7454cc0849 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -149,7 +149,10 @@ impl DominatorTree { // Run a finger up the dominator tree from b until we see a. // Do nothing if b is unreachable. while rpo_a < self.nodes[ebb_b].rpo_number { - let idom = self.idom(ebb_b).expect("Shouldn't meet unreachable here."); + let idom = match self.idom(ebb_b) { + Some(idom) => idom, + None => return None, // a is unreachable, so we climbed past the entry + }; ebb_b = layout.inst_ebb(idom).expect("Dominator got removed."); inst_b = Some(idom); } @@ -443,6 +446,35 @@ mod test { assert_eq!(dtree.cfg_postorder(), &[]); } + #[test] + fn unreachable_node() { + let mut func = Function::new(); + let ebb0 = func.dfg.make_ebb(); + let v0 = func.dfg.append_ebb_arg(ebb0, types::I32); + let ebb1 = func.dfg.make_ebb(); + let ebb2 = func.dfg.make_ebb(); + + let mut cur = FuncCursor::new(&mut func); + + cur.insert_ebb(ebb0); + cur.ins().brnz(v0, ebb2, &[]); + cur.ins().trap(TrapCode::User(0)); + + cur.insert_ebb(ebb1); + let v1 = cur.ins().iconst(I32, 1); + let v2 = cur.ins().iadd(v0, v1); + cur.ins().jump(ebb0, &[v2]); + + cur.insert_ebb(ebb2); + cur.ins().return_(&[v0]); + + let cfg = ControlFlowGraph::with_function(cur.func); + let dt = DominatorTree::with_function(cur.func, &cfg); + let v2_def = cur.func.dfg.value_def(v2).unwrap_inst(); + assert!(!dt.dominates(v2_def, ebb0, &cur.func.layout)); + assert!(!dt.dominates(ebb0, v2_def, &cur.func.layout)); + } + #[test] fn non_zero_entry_block() { let mut func = Function::new(); From ef048b8899ebc35667c7e2de19702ca835ee0b04 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 2 Oct 2017 15:56:38 -0700 Subject: [PATCH 1205/3084] Allow for call args in incoming stack slots. A value passed as an argument to a function call may live in an incoming stack slot initially. Fix the call legalizer so it copies such an argument into the expected outgoing stack slot for the call. --- cranelift/filetests/isa/intel/abi64.cton | 12 ++++++++++++ lib/cretonne/src/legalizer/boundary.rs | 25 +++++++++--------------- 2 files changed, 21 insertions(+), 16 deletions(-) diff --git a/cranelift/filetests/isa/intel/abi64.cton b/cranelift/filetests/isa/intel/abi64.cton index 59f3107560..ec943203ee 100644 --- a/cranelift/filetests/isa/intel/abi64.cton +++ b/cranelift/filetests/isa/intel/abi64.cton @@ -18,3 +18,15 @@ function %f() { ebb0: return } + +function %pass_stack_int64(i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) spiderwasm { + sig0 = (i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) spiderwasm + fn0 = sig0 #00000000 + +ebb0(v0: i64, v1: i64, v2: i64, v3: i64, v4: i64, v5: i64, v6: i64, v7: i64, v8: i64, v9: i64, v10: i64, v11: i64, v12: i64, v13: i64, v14: i64, v15: i64, v16: i64, v17: i64, v18: i64, v19: i64, v20: i64): + call fn0(v0, v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16, v17, v18, v19, v20) + jump ebb1 + +ebb1: + return +} diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 5522db921f..d1d48b053b 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -21,7 +21,7 @@ use abi::{legalize_abi_value, ValueConversion}; use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::{Function, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, Signature, SigRef, - ArgumentType, ArgumentPurpose, ArgumentLoc, ValueLoc, StackSlotKind}; + ArgumentType, ArgumentPurpose, ArgumentLoc, ValueLoc}; use ir::instructions::CallInfo; use isa::TargetIsa; use legalizer::split::{isplit, vsplit}; @@ -651,21 +651,14 @@ fn spill_call_arguments(pos: &mut FuncCursor) -> bool { .filter_map(|(idx, (&arg, abi))| { match abi.location { ArgumentLoc::Stack(offset) => { - // Is `arg` already in the right kind of stack slot? - match locations.get(arg) { - Some(&ValueLoc::Stack(ss)) => { - // We won't reassign `arg` to a different stack slot. Assert out of - // the stack slot is wrong. - assert_eq!(stack_slots[ss].kind, StackSlotKind::OutgoingArg); - assert_eq!(stack_slots[ss].offset, offset); - assert_eq!(stack_slots[ss].size, abi.value_type.bytes()); - None - } - _ => { - // Assign `arg` to a new stack slot. - let ss = stack_slots.get_outgoing_arg(abi.value_type, offset); - Some((idx, arg, ss)) - } + // Assign `arg` to a new stack slot, unless it's already in the correct + // slot. The legalization needs to be idempotent, so we should see a + // correct outgoing slot on the second pass. + let ss = stack_slots.get_outgoing_arg(abi.value_type, offset); + if locations[arg] != ValueLoc::Stack(ss) { + Some((idx, arg, ss)) + } else { + None } } _ => None, From c091a695e64e6b2b84b169e232981495cc9d4edb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 2 Oct 2017 15:38:05 -0700 Subject: [PATCH 1206/3084] Fix coalescer bug exposed by the gvn-unremovable-phi test. When we detect interference between the values that have already been merged into the candidate virtual register and an EBB argument, we first try to resolve the conflict by splitting. We also check if the existing interfering value is fundamentally incompatible with the branch instruction so it needs to be removed from the virtual register, restarting the merge operation. However, this existing interfering value is not necessarily the only interference, so the split is not guaranteed to resolve the conflict. If it turns out that splitting didn't resolve the conflict, restart the merge after removing this second conflicting value. --- cranelift/filetests/regalloc/coalesce.cton | 12 ++++++++++++ lib/cretonne/src/regalloc/coalescing.rs | 20 ++++++++++++++------ 2 files changed, 26 insertions(+), 6 deletions(-) diff --git a/cranelift/filetests/regalloc/coalesce.cton b/cranelift/filetests/regalloc/coalesce.cton index 7624eaef7c..bdddf26ef4 100644 --- a/cranelift/filetests/regalloc/coalesce.cton +++ b/cranelift/filetests/regalloc/coalesce.cton @@ -110,3 +110,15 @@ ebb1(v10: i32): v11 = iadd_imm v10, 1 return v11 } + +function %gvn_unremovable_phi(i32) native { +ebb0(v0: i32): + v2 = iconst.i32 0 + jump ebb2(v2, v0) + +ebb2(v3: i32, v4: i32): + brnz v3, ebb2(v3, v4) + v5 = iconst.i32 1 + brnz v3, ebb2(v2, v5) + return +} diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 509f5a45e7..34a9b8bde0 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -450,13 +450,21 @@ impl<'a> Context<'a> { return Some(a); } - // The local conflict could be avoided by splitting at this predecessor, so try - // that. This split is not necessarily required, but it allows us to make progress. + // The local conflict could likely be avoided by splitting at this predecessor, so + // try that. This split is not necessarily required, but it allows us to make + // progress. let new_val = self.split_pred(pred_inst, pred_ebb, argnum, pred_val); - assert!( - self.add_class(new_val).is_ok(), - "Splitting didn't resolve conflict." - ); + + // If this tiny new live range can't be merged, there is something in the already + // merged values that is fundamentally incompatible with `pred_inst`, and we need + // to start over after removing that value. + // TODO: It is unfortunate that we discover this *after* splitting. It would have + // been better if we could detect and isolate `merged` before splitting. + if let Err((merged, _)) = self.add_class(new_val) { + dbg!("Splitting didn't help: {} interferes", merged); + // We need to start over, isolating the bad value. + return Some(merged); + } } } From e10b3117cb858d806fc495bbbd8b21962a15c883 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 3 Oct 2017 13:25:33 -0700 Subject: [PATCH 1207/3084] Rename enc_flt() to enc_both(). This encoding method is not only used for floating point instructions. --- lib/cretonne/meta/isa/intel/encodings.py | 126 ++++++++++++----------- 1 file changed, 64 insertions(+), 62 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index eb4cee439e..db5c9db4ce 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -43,6 +43,24 @@ I64.legalize_type( # Helper functions for generating encodings. # +def enc_i64(inst, recipe, *args, **kwargs): + # type: (MaybeBoundInst, r.TailRecipe, *int, **int) -> None + """ + Add encodings for `inst` to I64 with and without a REX prefix. + """ + I64.enc(inst, *recipe.rex(*args, **kwargs)) + I64.enc(inst, *recipe(*args, **kwargs)) + + +def enc_both(inst, recipe, *args, **kwargs): + # type: (MaybeBoundInst, r.TailRecipe, *int, **Any) -> None + """ + Add encodings for `inst` to both I32 and I64. + """ + I32.enc(inst, *recipe(*args, **kwargs)) + enc_i64(inst, recipe, *args, **kwargs) + + def enc_i32_i64(inst, recipe, *args, **kwargs): # type: (MaybeBoundInst, r.TailRecipe, *int, **int) -> None """ @@ -82,22 +100,6 @@ def enc_i32_i64_ld_st(inst, w_bit, recipe, *args, **kwargs): I64.enc(inst.i64.any, *recipe(*args, **kwargs)) -def enc_flt(inst, recipe, *args, **kwargs): - # type: (MaybeBoundInst, r.TailRecipe, *int, **Any) -> None - """ - Add encodings for floating point instruction `inst` to both I32 and I64. - """ - I32.enc(inst, *recipe(*args, **kwargs)) - I64.enc(inst, *recipe.rex(*args, **kwargs)) - I64.enc(inst, *recipe(*args, **kwargs)) - - -def enc_i64(inst, recipe, *args, **kwargs): - # type: (MaybeBoundInst, r.TailRecipe, *int, **int) -> None - I64.enc(inst, *recipe.rex(*args, **kwargs)) - I64.enc(inst, *recipe(*args, **kwargs)) - - for inst, opc in [ (base.iadd, 0x01), (base.isub, 0x29), @@ -109,9 +111,9 @@ for inst, opc in [ # Also add a `b1` encodings for the logic instructions. # TODO: Should this be done with 8-bit instructions? It would improve # partial register dependencies. -enc_flt(base.band.b1, r.rr, 0x21) -enc_flt(base.bor.b1, r.rr, 0x09) -enc_flt(base.bxor.b1, r.rr, 0x31) +enc_both(base.band.b1, r.rr, 0x21) +enc_both(base.bor.b1, r.rr, 0x09) +enc_both(base.bxor.b1, r.rr, 0x31) enc_i32_i64(base.imul, r.rrx, 0x0f, 0xaf) enc_i32_i64(x86.sdivmodx, r.div, 0xf7, rrr=7) @@ -119,7 +121,7 @@ enc_i32_i64(x86.udivmodx, r.div, 0xf7, rrr=6) enc_i32_i64(base.copy, r.umr, 0x89) enc_i32_i64(base.regmove, r.rmov, 0x89) -enc_flt(base.regmove.b1, r.rmov, 0x89) +enc_both(base.regmove.b1, r.rmov, 0x89) # Immediate instructions with sign-extended 8-bit and 32-bit immediate. for inst, rrr in [ @@ -249,27 +251,27 @@ enc_i32_i64_ld_st(base.sload8, True, r.ldDisp32, 0x0f, 0xbe) # Float loads and stores. # -enc_flt(base.load.f32.any, r.fld, 0x66, 0x0f, 0x6e) -enc_flt(base.load.f32.any, r.fldDisp8, 0x66, 0x0f, 0x6e) -enc_flt(base.load.f32.any, r.fldDisp32, 0x66, 0x0f, 0x6e) +enc_both(base.load.f32.any, r.fld, 0x66, 0x0f, 0x6e) +enc_both(base.load.f32.any, r.fldDisp8, 0x66, 0x0f, 0x6e) +enc_both(base.load.f32.any, r.fldDisp32, 0x66, 0x0f, 0x6e) -enc_flt(base.load.f64.any, r.fld, 0xf3, 0x0f, 0x7e) -enc_flt(base.load.f64.any, r.fldDisp8, 0xf3, 0x0f, 0x7e) -enc_flt(base.load.f64.any, r.fldDisp32, 0xf3, 0x0f, 0x7e) +enc_both(base.load.f64.any, r.fld, 0xf3, 0x0f, 0x7e) +enc_both(base.load.f64.any, r.fldDisp8, 0xf3, 0x0f, 0x7e) +enc_both(base.load.f64.any, r.fldDisp32, 0xf3, 0x0f, 0x7e) -enc_flt(base.store.f32.any, r.fst, 0x66, 0x0f, 0x7e) -enc_flt(base.store.f32.any, r.fstDisp8, 0x66, 0x0f, 0x7e) -enc_flt(base.store.f32.any, r.fstDisp32, 0x66, 0x0f, 0x7e) +enc_both(base.store.f32.any, r.fst, 0x66, 0x0f, 0x7e) +enc_both(base.store.f32.any, r.fstDisp8, 0x66, 0x0f, 0x7e) +enc_both(base.store.f32.any, r.fstDisp32, 0x66, 0x0f, 0x7e) -enc_flt(base.store.f64.any, r.fst, 0x66, 0x0f, 0xd6) -enc_flt(base.store.f64.any, r.fstDisp8, 0x66, 0x0f, 0xd6) -enc_flt(base.store.f64.any, r.fstDisp32, 0x66, 0x0f, 0xd6) +enc_both(base.store.f64.any, r.fst, 0x66, 0x0f, 0xd6) +enc_both(base.store.f64.any, r.fstDisp8, 0x66, 0x0f, 0xd6) +enc_both(base.store.f64.any, r.fstDisp32, 0x66, 0x0f, 0xd6) -enc_flt(base.fill.f32, r.ffiSib32, 0x66, 0x0f, 0x6e) -enc_flt(base.fill.f64, r.ffiSib32, 0xf3, 0x0f, 0x7e) +enc_both(base.fill.f32, r.ffiSib32, 0x66, 0x0f, 0x6e) +enc_both(base.fill.f64, r.ffiSib32, 0xf3, 0x0f, 0x7e) -enc_flt(base.spill.f32, r.fspSib32, 0x66, 0x0f, 0x7e) -enc_flt(base.spill.f64, r.fspSib32, 0x66, 0x0f, 0xd6) +enc_both(base.spill.f32, r.fspSib32, 0x66, 0x0f, 0x7e) +enc_both(base.spill.f64, r.fspSib32, 0x66, 0x0f, 0xd6) # # Function addresses. @@ -307,10 +309,10 @@ enc_i32_i64(base.brnz, r.tjccd, 0x85) # Branch on a b1 value in a register only looks at the low 8 bits. See also # bint encodings below. -enc_flt(base.brz.b1, r.t8jccb_abcd, 0x74) -enc_flt(base.brz.b1, r.t8jccd_abcd, 0x84) -enc_flt(base.brnz.b1, r.t8jccb_abcd, 0x75) -enc_flt(base.brnz.b1, r.t8jccd_abcd, 0x85) +enc_both(base.brz.b1, r.t8jccb_abcd, 0x74) +enc_both(base.brz.b1, r.t8jccd_abcd, 0x84) +enc_both(base.brnz.b1, r.t8jccb_abcd, 0x75) +enc_both(base.brnz.b1, r.t8jccd_abcd, 0x85) # # Trap as ud2 @@ -349,18 +351,18 @@ I64.enc(base.uextend.i64.i32, *r.umr(0x89)) # # movd -enc_flt(base.bitcast.f32.i32, r.frurm, 0x66, 0x0f, 0x6e) -enc_flt(base.bitcast.i32.f32, r.rfumr, 0x66, 0x0f, 0x7e) +enc_both(base.bitcast.f32.i32, r.frurm, 0x66, 0x0f, 0x6e) +enc_both(base.bitcast.i32.f32, r.rfumr, 0x66, 0x0f, 0x7e) # movq I64.enc(base.bitcast.f64.i64, *r.frurm.rex(0x66, 0x0f, 0x6e, w=1)) I64.enc(base.bitcast.i64.f64, *r.rfumr.rex(0x66, 0x0f, 0x7e, w=1)) # movaps -enc_flt(base.copy.f32, r.furm, 0x0f, 0x28) -enc_flt(base.copy.f64, r.furm, 0x0f, 0x28) -enc_flt(base.regmove.f32, r.frmov, 0x0f, 0x28) -enc_flt(base.regmove.f64, r.frmov, 0x0f, 0x28) +enc_both(base.copy.f32, r.furm, 0x0f, 0x28) +enc_both(base.copy.f64, r.furm, 0x0f, 0x28) +enc_both(base.regmove.f32, r.frmov, 0x0f, 0x28) +enc_both(base.regmove.f64, r.frmov, 0x0f, 0x28) # cvtsi2ss enc_i32_i64(base.fcvt_from_sint.f32, r.frurm, 0xf3, 0x0f, 0x2a) @@ -369,22 +371,22 @@ enc_i32_i64(base.fcvt_from_sint.f32, r.frurm, 0xf3, 0x0f, 0x2a) enc_i32_i64(base.fcvt_from_sint.f64, r.frurm, 0xf2, 0x0f, 0x2a) # cvtss2sd -enc_flt(base.fpromote.f64.f32, r.furm, 0xf3, 0x0f, 0x5a) +enc_both(base.fpromote.f64.f32, r.furm, 0xf3, 0x0f, 0x5a) # cvtsd2ss -enc_flt(base.fdemote.f32.f64, r.furm, 0xf2, 0x0f, 0x5a) +enc_both(base.fdemote.f32.f64, r.furm, 0xf2, 0x0f, 0x5a) # cvttss2si -enc_flt(x86.cvtt2si.i32.f32, r.rfurm, 0xf3, 0x0f, 0x2c) +enc_both(x86.cvtt2si.i32.f32, r.rfurm, 0xf3, 0x0f, 0x2c) I64.enc(x86.cvtt2si.i64.f32, *r.rfurm.rex(0xf3, 0x0f, 0x2c, w=1)) # cvttsd2si -enc_flt(x86.cvtt2si.i32.f64, r.rfurm, 0xf2, 0x0f, 0x2c) +enc_both(x86.cvtt2si.i32.f64, r.rfurm, 0xf2, 0x0f, 0x2c) I64.enc(x86.cvtt2si.i64.f64, *r.rfurm.rex(0xf2, 0x0f, 0x2c, w=1)) # Exact square roots. -enc_flt(base.sqrt.f32, r.furm, 0xf3, 0x0f, 0x51) -enc_flt(base.sqrt.f64, r.furm, 0xf2, 0x0f, 0x51) +enc_both(base.sqrt.f32, r.furm, 0xf3, 0x0f, 0x51) +enc_both(base.sqrt.f64, r.furm, 0xf2, 0x0f, 0x51) # Rounding. The recipe looks at the opcode to pick an immediate. for inst in [ @@ -392,8 +394,8 @@ for inst in [ base.floor, base.ceil, base.trunc]: - enc_flt(inst.f32, r.furmi_rnd, 0x66, 0x0f, 0x3a, 0x0a, isap=use_sse41) - enc_flt(inst.f64, r.furmi_rnd, 0x66, 0x0f, 0x3a, 0x0b, isap=use_sse41) + enc_both(inst.f32, r.furmi_rnd, 0x66, 0x0f, 0x3a, 0x0a, isap=use_sse41) + enc_both(inst.f64, r.furmi_rnd, 0x66, 0x0f, 0x3a, 0x0b, isap=use_sse41) # Binary arithmetic ops. @@ -404,24 +406,24 @@ for inst, opc in [ (base.fdiv, 0x5e), (x86.fmin, 0x5d), (x86.fmax, 0x5f)]: - enc_flt(inst.f32, r.fa, 0xf3, 0x0f, opc) - enc_flt(inst.f64, r.fa, 0xf2, 0x0f, opc) + enc_both(inst.f32, r.fa, 0xf3, 0x0f, opc) + enc_both(inst.f64, r.fa, 0xf2, 0x0f, opc) # Binary bitwise ops. for inst, opc in [ (base.band, 0x54), (base.bor, 0x56), (base.bxor, 0x57)]: - enc_flt(inst.f32, r.fa, 0x0f, opc) - enc_flt(inst.f64, r.fa, 0x0f, opc) + enc_both(inst.f32, r.fa, 0x0f, opc) + enc_both(inst.f64, r.fa, 0x0f, opc) # The `andnps(x,y)` instruction computes `~x&y`, while band_not(x,y)` is `x&~y. -enc_flt(base.band_not.f32, r.fax, 0x0f, 0x55) -enc_flt(base.band_not.f64, r.fax, 0x0f, 0x55) +enc_both(base.band_not.f32, r.fax, 0x0f, 0x55) +enc_both(base.band_not.f64, r.fax, 0x0f, 0x55) # Comparisons. # # This only covers the condition codes in `supported_floatccs`, the rest are # handled by legalization patterns. -enc_flt(base.fcmp.f32, r.fcscc, 0x0f, 0x2e) -enc_flt(base.fcmp.f64, r.fcscc, 0x66, 0x0f, 0x2e) +enc_both(base.fcmp.f32, r.fcscc, 0x0f, 0x2e) +enc_both(base.fcmp.f64, r.fcscc, 0x66, 0x0f, 0x2e) From 7c023b24306ea0081e5cdd9bf62c4d1f6827bdce Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 3 Oct 2017 13:37:18 -0700 Subject: [PATCH 1208/3084] Don't omit the controlling typevar for instructions without results. The controlling type variable passed to the format constructor in the InstBuilder trait is not just used to generate the result values. In an EncCursor, it is also used to encode the instruction, so VOID doesn't work. --- lib/cretonne/meta/gen_instr.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 3b43ad368b..b004aaadf1 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -572,7 +572,7 @@ def gen_inst_builder(inst, fmt): if inst.is_polymorphic and not inst.use_typevar_operand: # This was an explicit method argument. args.append(inst.ctrl_typevar.name) - elif len(inst.value_results) == 0 or not inst.is_polymorphic: + elif not inst.is_polymorphic: # No controlling type variable needed. args.append('types::VOID') else: From 739d414d18719983841e709afcc4414d264115bf Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 3 Oct 2017 13:14:07 -0700 Subject: [PATCH 1209/3084] Convert regalloc::coloring to use an EncCursor. No functional change intended, this is just a big fight with the borrow checker. --- lib/cretonne/src/regalloc/coloring.rs | 366 +++++++++++--------------- 1 file changed, 161 insertions(+), 205 deletions(-) diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index ce6e22f756..83ecddb0ce 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -42,10 +42,10 @@ //! //! The exception is the entry block whose arguments are colored from the ABI requirements. +use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; -use ir::{Ebb, Inst, Value, Function, Cursor, CursorBase, ValueLoc, DataFlowGraph, Layout}; -use ir::{InstEncodings, ValueLocations}; -use ir::{InstBuilder, Signature, ArgumentType, ArgumentLoc}; +use ir::{Ebb, Inst, Value, Function, ValueLoc, SigRef}; +use ir::{InstBuilder, ArgumentType, ArgumentLoc}; use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; use isa::{TargetIsa, EncInfo, RecipeConstraints, OperandConstraint, ConstraintKind}; use regalloc::RegDiversions; @@ -74,7 +74,9 @@ pub struct Coloring { /// Immutable context information and mutable references that don't need to be borrowed across /// method calls should go in this struct. struct Context<'a> { - isa: &'a TargetIsa, + // Current instruction as well as reference to function and ISA. + cur: EncCursor<'a>, + // Cached ISA information. // We save it here to avoid frequent virtual function calls on the `TargetIsa` trait object. reginfo: RegInfo, @@ -115,57 +117,53 @@ impl Coloring { ) { dbg!("Coloring for:\n{}", func.display(isa)); let mut ctx = Context { - isa, + usable_regs: isa.allocatable_registers(func), + cur: EncCursor::new(func, isa), reginfo: isa.register_info(), encinfo: isa.encoding_info(), domtree, liveness, divert: &mut self.divert, solver: &mut self.solver, - usable_regs: isa.allocatable_registers(func), }; - ctx.run(func, tracker) + ctx.run(tracker) } } impl<'a> Context<'a> { /// Run the coloring algorithm. - fn run(&mut self, func: &mut Function, tracker: &mut LiveValueTracker) { - func.locations.resize(func.dfg.num_values()); + fn run(&mut self, tracker: &mut LiveValueTracker) { + self.cur.func.locations.resize( + self.cur.func.dfg.num_values(), + ); // Visit blocks in reverse post-order. We need to ensure that at least one predecessor has // been visited before each EBB. That guarantees that the EBB arguments have been colored. for &ebb in self.domtree.cfg_postorder().iter().rev() { - self.visit_ebb(ebb, func, tracker); + self.visit_ebb(ebb, tracker); } } /// Visit `ebb`, assuming that the immediate dominator has already been visited. - fn visit_ebb(&mut self, ebb: Ebb, func: &mut Function, tracker: &mut LiveValueTracker) { + fn visit_ebb(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) { dbg!("Coloring {}:", ebb); - let mut regs = self.visit_ebb_header(ebb, func, tracker); + let mut regs = self.visit_ebb_header(ebb, tracker); tracker.drop_dead_args(); self.divert.clear(); // Now go through the instructions in `ebb` and color the values they define. - let mut pos = Cursor::new(&mut func.layout, &mut func.srclocs).at_top(ebb); - while let Some(inst) = pos.next_inst() { - pos.use_srcloc(inst); - if let Some(constraints) = self.encinfo.operand_constraints(func.encodings[inst]) { - self.visit_inst( - inst, - constraints, - &mut pos, - &mut func.dfg, - tracker, - &mut regs, - &mut func.locations, - &mut func.encodings, - &func.signature, - ); + self.cur.goto_top(ebb); + while let Some(inst) = self.cur.next_inst() { + self.cur.use_srcloc(inst); + if let Some(constraints) = + self.encinfo.operand_constraints( + self.cur.func.encodings[inst], + ) + { + self.visit_inst(inst, constraints, tracker, &mut regs); } else { let (_throughs, kills) = tracker.process_ghost(inst); - self.process_ghost_kills(kills, &mut regs, &func.locations); + self.process_ghost_kills(kills, &mut regs); } tracker.drop_dead(inst); } @@ -174,22 +172,23 @@ impl<'a> Context<'a> { /// Visit the `ebb` header. /// /// Initialize the set of live registers and color the arguments to `ebb`. - fn visit_ebb_header( - &self, - ebb: Ebb, - func: &mut Function, - tracker: &mut LiveValueTracker, - ) -> AllocatableSet { + fn visit_ebb_header(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) -> AllocatableSet { // Reposition the live value tracker and deal with the EBB arguments. - tracker.ebb_top(ebb, &func.dfg, self.liveness, &func.layout, self.domtree); + tracker.ebb_top( + ebb, + &self.cur.func.dfg, + self.liveness, + &self.cur.func.layout, + self.domtree, + ); - if func.layout.entry_block() == Some(ebb) { + if self.cur.func.layout.entry_block() == Some(ebb) { // Arguments to the entry block have ABI constraints. - self.color_entry_args(&func.signature, tracker.live(), &mut func.locations) + self.color_entry_args(tracker.live()) } else { // The live-ins and arguments to a non-entry EBB have already been assigned a register. // Reconstruct the allocatable set. - self.livein_regs(tracker.live(), func) + self.livein_regs(tracker.live()) } } @@ -198,7 +197,7 @@ impl<'a> Context<'a> { /// /// Also process the EBB arguments which were colored when the first predecessor branch was /// encountered. - fn livein_regs(&self, live: &[LiveValue], func: &Function) -> AllocatableSet { + fn livein_regs(&self, live: &[LiveValue]) -> AllocatableSet { // Start from the registers that are actually usable. We don't want to include any reserved // registers in the set. let mut regs = self.usable_regs.clone(); @@ -213,11 +212,11 @@ impl<'a> Context<'a> { "Live-in: {}:{} in {}", value, affinity.display(&self.reginfo), - func.locations[value].display(&self.reginfo) + self.cur.func.locations[value].display(&self.reginfo) ); if let Affinity::Reg(rci) = affinity { let rc = self.reginfo.rc(rci); - let loc = func.locations[value]; + let loc = self.cur.func.locations[value]; match loc { ValueLoc::Reg(reg) => regs.take(rc, reg), ValueLoc::Unassigned => panic!("Live-in {} wasn't assigned", value), @@ -237,12 +236,8 @@ impl<'a> Context<'a> { /// function signature. /// /// Return the set of remaining allocatable registers after filtering out the dead arguments. - fn color_entry_args( - &self, - sig: &Signature, - args: &[LiveValue], - locations: &mut ValueLocations, - ) -> AllocatableSet { + fn color_entry_args(&mut self, args: &[LiveValue]) -> AllocatableSet { + let sig = &self.cur.func.signature; assert_eq!(sig.argument_types.len(), args.len()); let mut regs = self.usable_regs.clone(); @@ -255,7 +250,7 @@ impl<'a> Context<'a> { if !lv.is_dead { regs.take(rc, reg); } - locations[lv.value] = ValueLoc::Reg(reg); + self.cur.func.locations[lv.value] = ValueLoc::Reg(reg); } else { // This should have been fixed by the reload pass. panic!( @@ -287,17 +282,12 @@ impl<'a> Context<'a> { &mut self, inst: Inst, constraints: &RecipeConstraints, - pos: &mut Cursor, - dfg: &mut DataFlowGraph, tracker: &mut LiveValueTracker, regs: &mut AllocatableSet, - locations: &mut ValueLocations, - encodings: &mut InstEncodings, - func_signature: &Signature, ) { dbg!( "Coloring {}\n {}", - dfg.display_inst(inst, self.isa), + self.cur.display_inst(inst), regs.display(&self.reginfo) ); @@ -307,40 +297,56 @@ impl<'a> Context<'a> { // Program the solver with register constraints for the input side. self.solver.reset(regs); - self.program_input_constraints(inst, constraints.ins, dfg, locations); - let call_sig = dfg.call_signature(inst); + self.program_input_constraints(inst, constraints.ins); + let call_sig = self.cur.func.dfg.call_signature(inst); if let Some(sig) = call_sig { - self.program_input_abi(inst, &dfg.signatures[sig].argument_types, dfg, locations); - } else if dfg[inst].opcode().is_return() { - self.program_input_abi(inst, &func_signature.return_types, dfg, locations); - } else if dfg[inst].opcode().is_branch() { + program_input_abi( + &mut self.solver, + inst, + &self.cur.func.dfg.signatures[sig].argument_types, + &self.cur.func, + &self.liveness, + &self.reginfo, + &self.divert, + ); + } else if self.cur.func.dfg[inst].opcode().is_return() { + program_input_abi( + &mut self.solver, + inst, + &self.cur.func.signature.return_types, + &self.cur.func, + &self.liveness, + &self.reginfo, + &self.divert, + ); + } else if self.cur.func.dfg[inst].opcode().is_branch() { // This is a branch, so we need to make sure that globally live values are in their // global registers. For EBBs that take arguments, we also need to place the argument // values in the expected registers. - if let Some(dest) = dfg[inst].branch_destination() { - if self.program_ebb_arguments(inst, dest, dfg, pos.layout, locations) { + if let Some(dest) = self.cur.func.dfg[inst].branch_destination() { + if self.program_ebb_arguments(inst, dest) { color_dest_args = Some(dest); } } else { // This is a multi-way branch like `br_table`. We only support arguments on // single-destination branches. assert_eq!( - dfg.inst_variable_args(inst).len(), + self.cur.func.dfg.inst_variable_args(inst).len(), 0, "Can't handle EBB arguments: {}", - dfg.display_inst(inst, self.isa) + self.cur.display_inst(inst) ); - self.undivert_regs(|lr| !lr.is_local()); + self.undivert_regs(|lr, _| !lr.is_local()); } } if self.solver.has_fixed_input_conflicts() { - self.divert_fixed_input_conflicts(tracker.live(), locations); + self.divert_fixed_input_conflicts(tracker.live()); } self.solver.inputs_done(); // Update the live value tracker with this instruction. - let (throughs, kills, defs) = tracker.process_inst(inst, dfg, self.liveness); + let (throughs, kills, defs) = tracker.process_inst(inst, &self.cur.func.dfg, self.liveness); // Get rid of the killed values. for lv in kills { @@ -348,7 +354,7 @@ impl<'a> Context<'a> { self.solver.add_kill( lv.value, self.reginfo.rc(rci), - self.divert.reg(lv.value, locations), + self.divert.reg(lv.value, &self.cur.func.locations), ); } } @@ -357,35 +363,34 @@ impl<'a> Context<'a> { // detect conflicts between fixed outputs and tied operands where the input value hasn't // been converted to a solver variable. if constraints.fixed_outs { - self.program_fixed_outputs(constraints.outs, defs, throughs, locations); + self.program_fixed_outputs(constraints.outs, defs, throughs); } if let Some(sig) = call_sig { - let abi = &dfg.signatures[sig].return_types; - self.program_output_abi(abi, defs, throughs, locations); + self.program_output_abi(sig, defs, throughs); } - self.program_output_constraints(inst, constraints.outs, defs, dfg, locations); + self.program_output_constraints(inst, constraints.outs, defs); // Finally, we've fully programmed the constraint solver. // We expect a quick solution in most cases. let mut output_regs = self.solver.quick_solve().unwrap_or_else(|rc| { dbg!("quick_solve needs more registers in {}", rc); - self.iterate_solution(throughs, locations) + self.iterate_solution(throughs) }); // The solution and/or fixed input constraints may require us to shuffle the set of live // registers around. - self.shuffle_inputs(pos, dfg, regs, encodings); + self.shuffle_inputs(regs); // If this is the first time we branch to `dest`, color its arguments to match the current // register state. if let Some(dest) = color_dest_args { - self.color_ebb_arguments(inst, dest, dfg, locations); + self.color_ebb_arguments(inst, dest); } // Apply the solution to the defs. for v in self.solver.vars().iter().filter(|&v| v.is_define()) { - locations[v.value] = ValueLoc::Reg(v.solution); + self.cur.func.locations[v.value] = ValueLoc::Reg(v.solution); } // Tied defs are not part of the solution above. @@ -393,9 +398,9 @@ impl<'a> Context<'a> { if constraints.tied_ops { for (op, lv) in constraints.outs.iter().zip(defs) { if let ConstraintKind::Tied(num) = op.kind { - let arg = dfg.inst_args(inst)[num as usize]; - let reg = self.divert.reg(arg, locations); - locations[lv.value] = ValueLoc::Reg(reg); + let arg = self.cur.func.dfg.inst_args(inst)[num as usize]; + let reg = self.divert.reg(arg, &self.cur.func.locations); + self.cur.func.locations[lv.value] = ValueLoc::Reg(reg); } } } @@ -405,7 +410,7 @@ impl<'a> Context<'a> { if lv.endpoint == inst { if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); - let reg = self.divert.reg(lv.value, locations); + let reg = self.divert.reg(lv.value, &self.cur.func.locations); output_regs.free(rc, reg); } } @@ -417,22 +422,15 @@ impl<'a> Context<'a> { } /// Program the input-side constraints for `inst` into the constraint solver. - fn program_input_constraints( - &mut self, - inst: Inst, - constraints: &[OperandConstraint], - dfg: &DataFlowGraph, - locations: &ValueLocations, - ) { - for (op, &value) in constraints.iter().zip(dfg.inst_args(inst)).filter( - |&(op, _)| { - op.kind != ConstraintKind::Stack - }, - ) + fn program_input_constraints(&mut self, inst: Inst, constraints: &[OperandConstraint]) { + for (op, &value) in constraints + .iter() + .zip(self.cur.func.dfg.inst_args(inst)) + .filter(|&(op, _)| op.kind != ConstraintKind::Stack) { // Reload pass is supposed to ensure that all arguments to register operands are // already in a register. - let cur_reg = self.divert.reg(value, locations); + let cur_reg = self.divert.reg(value, &self.cur.func.locations); match op.kind { ConstraintKind::FixedReg(regunit) => { if regunit != cur_reg { @@ -460,34 +458,6 @@ impl<'a> Context<'a> { } } - /// Program the input-side ABI constraints for `inst` into the constraint solver. - /// - /// ABI constraints are the fixed register assignments used for calls and returns. - fn program_input_abi( - &mut self, - inst: Inst, - abi_types: &[ArgumentType], - dfg: &DataFlowGraph, - locations: &ValueLocations, - ) { - for (abi, &value) in abi_types.iter().zip(dfg.inst_variable_args(inst)) { - if let ArgumentLoc::Reg(reg) = abi.location { - if let Affinity::Reg(rci) = - self.liveness - .get(value) - .expect("ABI register must have live range") - .affinity - { - let rc = self.reginfo.rc(rci); - let cur_reg = self.divert.reg(value, locations); - self.solver.reassign_in(value, rc, cur_reg, reg); - } else { - panic!("ABI argument {} should be in a register", value); - } - } - } - } - /// Prepare for a branch to `dest`. /// /// 1. Any values that are live-in to `dest` must be un-diverted so they live in their globally @@ -497,29 +467,22 @@ impl<'a> Context<'a> { /// /// Returns true if this is the first time a branch to `dest` is seen, so the `dest` argument /// values should be colored after `shuffle_inputs`. - fn program_ebb_arguments( - &mut self, - inst: Inst, - dest: Ebb, - dfg: &DataFlowGraph, - layout: &Layout, - locations: &ValueLocations, - ) -> bool { + fn program_ebb_arguments(&mut self, inst: Inst, dest: Ebb) -> bool { // Find diverted registers that are live-in to `dest` and reassign them to their global // home. // // Values with a global live range that are not live in to `dest` could appear as branch // arguments, so they can't always be un-diverted. - self.undivert_regs(|lr| lr.livein_local_end(dest, layout).is_some()); + self.undivert_regs(|lr, func| lr.livein_local_end(dest, &func.layout).is_some()); // Now handle the EBB arguments. - let br_args = dfg.inst_variable_args(inst); - let dest_args = dfg.ebb_args(dest); + let br_args = self.cur.func.dfg.inst_variable_args(inst); + let dest_args = self.cur.func.dfg.ebb_args(dest); assert_eq!(br_args.len(), dest_args.len()); for (&dest_arg, &br_arg) in dest_args.iter().zip(br_args) { // The first time we encounter a branch to `dest`, we get to pick the location. The // following times we see a branch to `dest`, we must follow suit. - match locations[dest_arg] { + match self.cur.func.locations[dest_arg] { ValueLoc::Unassigned => { // This is the first branch to `dest`, so we should color `dest_arg` instead of // `br_arg`. However, we don't know where `br_arg` will end up until @@ -536,7 +499,7 @@ impl<'a> Context<'a> { // registers by reassigning `br_arg`. if let Affinity::Reg(rci) = self.liveness[br_arg].affinity { let rc = self.reginfo.rc(rci); - let br_reg = self.divert.reg(br_arg, locations); + let br_reg = self.divert.reg(br_arg, &self.cur.func.locations); self.solver.reassign_in(br_arg, rc, br_reg, dest_reg); } else { panic!("Branch argument {} is not in a register", br_arg); @@ -544,7 +507,7 @@ impl<'a> Context<'a> { } ValueLoc::Stack(ss) => { // The spiller should already have given us identical stack slots. - debug_assert_eq!(ValueLoc::Stack(ss), locations[br_arg]); + debug_assert_eq!(ValueLoc::Stack(ss), self.cur.func.locations[br_arg]); } } } @@ -557,22 +520,16 @@ impl<'a> Context<'a> { /// register state. /// /// This function is only called when `program_ebb_arguments()` returned `true`. - fn color_ebb_arguments( - &mut self, - inst: Inst, - dest: Ebb, - dfg: &DataFlowGraph, - locations: &mut ValueLocations, - ) { - let br_args = dfg.inst_variable_args(inst); - let dest_args = dfg.ebb_args(dest); + fn color_ebb_arguments(&mut self, inst: Inst, dest: Ebb) { + let br_args = self.cur.func.dfg.inst_variable_args(inst); + let dest_args = self.cur.func.dfg.ebb_args(dest); assert_eq!(br_args.len(), dest_args.len()); for (&dest_arg, &br_arg) in dest_args.iter().zip(br_args) { - match locations[dest_arg] { + match self.cur.func.locations[dest_arg] { ValueLoc::Unassigned => { if self.liveness[dest_arg].affinity.is_reg() { - let br_reg = self.divert.reg(br_arg, locations); - locations[dest_arg] = ValueLoc::Reg(br_reg); + let br_reg = self.divert.reg(br_arg, &self.cur.func.locations); + self.cur.func.locations[dest_arg] = ValueLoc::Reg(br_reg); } } ValueLoc::Reg(_) => panic!("{} arg {} already colored", dest, dest_arg), @@ -586,13 +543,13 @@ impl<'a> Context<'a> { /// are reallocated to their global register assignments. fn undivert_regs(&mut self, mut pred: Pred) where - Pred: FnMut(&LiveRange) -> bool, + Pred: FnMut(&LiveRange, &Function) -> bool, { for rdiv in self.divert.all() { let lr = self.liveness.get(rdiv.value).expect( "Missing live range for diverted register", ); - if pred(lr) { + if pred(lr, &self.cur.func) { if let Affinity::Reg(rci) = lr.affinity { let rc = self.reginfo.rc(rci); self.solver.reassign_in(rdiv.value, rc, rdiv.to, rdiv.from); @@ -609,11 +566,11 @@ impl<'a> Context<'a> { // Find existing live values that conflict with the fixed input register constraints programmed // into the constraint solver. Convert them to solver variables so they can be diverted. - fn divert_fixed_input_conflicts(&mut self, live: &[LiveValue], locations: &mut ValueLocations) { + fn divert_fixed_input_conflicts(&mut self, live: &[LiveValue]) { for lv in live { if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); - let reg = self.divert.reg(lv.value, locations); + let reg = self.divert.reg(lv.value, &self.cur.func.locations); if self.solver.is_fixed_input_conflict(rc, reg) { self.solver.add_var(lv.value, rc, reg, &self.reginfo); } @@ -629,11 +586,10 @@ impl<'a> Context<'a> { constraints: &[OperandConstraint], defs: &[LiveValue], throughs: &[LiveValue], - locations: &mut ValueLocations, ) { for (op, lv) in constraints.iter().zip(defs) { if let ConstraintKind::FixedReg(reg) = op.kind { - self.add_fixed_output(lv.value, op.regclass, reg, throughs, locations); + self.add_fixed_output(lv.value, op.regclass, reg, throughs); } } } @@ -641,22 +597,20 @@ impl<'a> Context<'a> { /// Program the output-side ABI constraints for `inst` into the constraint solver. /// /// That means return values for a call instruction. - fn program_output_abi( - &mut self, - abi_types: &[ArgumentType], - defs: &[LiveValue], - throughs: &[LiveValue], - locations: &mut ValueLocations, - ) { + fn program_output_abi(&mut self, sig: SigRef, defs: &[LiveValue], throughs: &[LiveValue]) { // It's technically possible for a call instruction to have fixed results before the // variable list of results, but we have no known instances of that. // Just assume all results are variable return values. - assert_eq!(defs.len(), abi_types.len()); - for (abi, lv) in abi_types.iter().zip(defs) { + assert_eq!( + defs.len(), + self.cur.func.dfg.signatures[sig].return_types.len() + ); + for (i, lv) in defs.iter().enumerate() { + let abi = self.cur.func.dfg.signatures[sig].return_types[i]; if let ArgumentLoc::Reg(reg) = abi.location { if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); - self.add_fixed_output(lv.value, rc, reg, throughs, locations); + self.add_fixed_output(lv.value, rc, reg, throughs); } else { panic!("ABI argument {} should be in a register", lv.value); } @@ -671,14 +625,13 @@ impl<'a> Context<'a> { rc: RegClass, reg: RegUnit, throughs: &[LiveValue], - locations: &mut ValueLocations, ) { if !self.solver.add_fixed_output(rc, reg) { // The fixed output conflicts with some of the live-through registers. for lv in throughs { if let Affinity::Reg(rci) = lv.affinity { let rc2 = self.reginfo.rc(rci); - let reg2 = self.divert.reg(lv.value, locations); + let reg2 = self.divert.reg(lv.value, &self.cur.func.locations); if regs_overlap(rc, reg, rc2, reg2) { // This live-through value is interfering with the fixed output assignment. // Convert it to a solver variable. @@ -693,7 +646,7 @@ impl<'a> Context<'a> { let ok = self.solver.add_fixed_output(rc, reg); assert!(ok, "Couldn't clear fixed output interference for {}", value); } - locations[value] = ValueLoc::Reg(reg); + self.cur.func.locations[value] = ValueLoc::Reg(reg); } /// Program the output-side constraints for `inst` into the constraint solver. @@ -704,8 +657,6 @@ impl<'a> Context<'a> { inst: Inst, constraints: &[OperandConstraint], defs: &[LiveValue], - dfg: &mut DataFlowGraph, - locations: &mut ValueLocations, ) { for (op, lv) in constraints.iter().zip(defs) { match op.kind { @@ -717,11 +668,11 @@ impl<'a> Context<'a> { ConstraintKind::Tied(num) => { // Find the input operand we're tied to. // The solver doesn't care about the output value. - let arg = dfg.inst_args(inst)[num as usize]; + let arg = self.cur.func.dfg.inst_args(inst)[num as usize]; self.solver.add_tied_input( arg, op.regclass, - self.divert.reg(arg, locations), + self.divert.reg(arg, &self.cur.func.locations), ); } } @@ -732,11 +683,7 @@ impl<'a> Context<'a> { /// /// We may need to move more registers around before a solution is possible. Use an iterative /// algorithm that adds one more variable until a solution can be found. - fn iterate_solution( - &mut self, - throughs: &[LiveValue], - locations: &mut ValueLocations, - ) -> AllocatableSet { + fn iterate_solution(&mut self, throughs: &[LiveValue]) -> AllocatableSet { loop { dbg!("real_solve for {} variables", self.solver.vars().len()); let rc = match self.solver.real_solve() { @@ -746,7 +693,7 @@ impl<'a> Context<'a> { // Do we have any live-through `rc` registers that are not already variables? assert!( - self.try_add_var(rc, throughs, locations), + self.try_add_var(rc, throughs), "Ran out of registers in {}", rc ); @@ -754,18 +701,13 @@ impl<'a> Context<'a> { } /// Try to add an `rc` variable to the solver from the `throughs` set. - fn try_add_var( - &mut self, - rc: RegClass, - throughs: &[LiveValue], - locations: &mut ValueLocations, - ) -> bool { + fn try_add_var(&mut self, rc: RegClass, throughs: &[LiveValue]) -> bool { dbg!("Trying to add a {} reg from {} values", rc, throughs.len()); for lv in throughs { if let Affinity::Reg(rci) = lv.affinity { let rc2 = self.reginfo.rc(rci); - let reg2 = self.divert.reg(lv.value, locations); + let reg2 = self.divert.reg(lv.value, &self.cur.func.locations); if rc.contains(reg2) && self.solver.can_add_var(lv.value, rc2, reg2) { // The new variable gets to roam the whole top-level register class because // it is not actually constrained by the instruction. We just want it out @@ -783,27 +725,16 @@ impl<'a> Context<'a> { /// Emit `regmove` instructions as needed to move the live registers into place before the /// instruction. Also update `self.divert` accordingly. /// - /// The `pos` cursor is expected to point at the instruction. The register moves are inserted - /// before. + /// The `self.cur` cursor is expected to point at the instruction. The register moves are + /// inserted before. /// /// The solver needs to be reminded of the available registers before any moves are inserted. - fn shuffle_inputs( - &mut self, - pos: &mut Cursor, - dfg: &mut DataFlowGraph, - regs: &mut AllocatableSet, - encodings: &mut InstEncodings, - ) { + fn shuffle_inputs(&mut self, regs: &mut AllocatableSet) { self.solver.schedule_moves(regs); for m in self.solver.moves() { - let ty = dfg.value_type(m.value); self.divert.regmove(m.value, m.from, m.to); - let inst = dfg.ins(pos).regmove(m.value, m.from, m.to); - match self.isa.encode(dfg, &dfg[inst], ty) { - Ok(encoding) => encodings[inst] = encoding, - _ => panic!("Can't encode {} {}", m.rc, dfg.display_inst(inst, self.isa)), - } + self.cur.ins().regmove(m.value, m.from, m.to); } } @@ -823,21 +754,46 @@ impl<'a> Context<'a> { /// Process kills on a ghost instruction. /// - Forget diversions. /// - Free killed registers. - fn process_ghost_kills( - &mut self, - kills: &[LiveValue], - regs: &mut AllocatableSet, - locations: &ValueLocations, - ) { + fn process_ghost_kills(&mut self, kills: &[LiveValue], regs: &mut AllocatableSet) { for lv in kills { if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); let reg = match self.divert.remove(lv.value) { Some(r) => r, - None => locations[lv.value].unwrap_reg(), + None => self.cur.func.locations[lv.value].unwrap_reg(), }; regs.free(rc, reg); } } } } + +/// Program the input-side ABI constraints for `inst` into the constraint solver. +/// +/// ABI constraints are the fixed register assignments used for calls and returns. +fn program_input_abi( + solver: &mut Solver, + inst: Inst, + abi_types: &[ArgumentType], + func: &Function, + liveness: &Liveness, + reginfo: &RegInfo, + divert: &RegDiversions, +) { + for (abi, &value) in abi_types.iter().zip(func.dfg.inst_variable_args(inst)) { + if let ArgumentLoc::Reg(reg) = abi.location { + if let Affinity::Reg(rci) = + liveness + .get(value) + .expect("ABI register must have live range") + .affinity + { + let rc = reginfo.rc(rci); + let cur_reg = divert.reg(value, &func.locations); + solver.reassign_in(value, rc, cur_reg, reg); + } else { + panic!("ABI argument {} should be in a register", value); + } + } + } +} From fb0999ce333a2443e2b3b6bfc591287e1d1ef825 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 3 Oct 2017 12:18:55 -0700 Subject: [PATCH 1210/3084] Check the top-level register class for available registers. Fixes #165. The constraint solver's schedule_move() function sometimes need to use an extra available register when the moves to be scheduled contains cycles. The pending moves have associated register classes that come from the constraint programming. Since the moves have hard-coded to and from registers, these register classes are only meant to indicate the register sizes. In particular, we can use the whole top-level register class when scavenging for a spare register to break a cycle. --- .../filetests/regalloc/schedule-moves.cton | 19 +++++++ lib/cretonne/src/regalloc/coloring.rs | 2 +- lib/cretonne/src/regalloc/solver.rs | 51 ++++++++++--------- 3 files changed, 48 insertions(+), 24 deletions(-) create mode 100644 cranelift/filetests/regalloc/schedule-moves.cton diff --git a/cranelift/filetests/regalloc/schedule-moves.cton b/cranelift/filetests/regalloc/schedule-moves.cton new file mode 100644 index 0000000000..79536aecf6 --- /dev/null +++ b/cranelift/filetests/regalloc/schedule-moves.cton @@ -0,0 +1,19 @@ +test compile +set is_64bit=1 +isa intel haswell + +function %pr165() native { +ebb0: + v0 = iconst.i64 0x0102_0304_f1f2_f3f4 + v1 = iconst.i64 0x1102_0304_f1f2_f3f4 + v2 = iconst.i64 0x2102_0304_f1f2_f3f4 + v20 = ishl v1, v0 + v21 = ishl v2, v0 + v22 = sshr v1, v0 + v23 = sshr v2, v0 + v24 = ushr v1, v0 + v25 = ushr v2, v0 + istore8 v0, v1+0x2710 + istore8 v1, v0+0x2710 + return +} diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 83ecddb0ce..9aa5f48b2b 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -730,7 +730,7 @@ impl<'a> Context<'a> { /// /// The solver needs to be reminded of the available registers before any moves are inserted. fn shuffle_inputs(&mut self, regs: &mut AllocatableSet) { - self.solver.schedule_moves(regs); + self.solver.schedule_moves(regs, &self.reginfo); for m in self.solver.moves() { self.divert.regmove(m.value, m.from, m.to); diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index bfa6d22eb3..e3491ba8cc 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -679,7 +679,7 @@ impl Solver { /// a register. /// /// Returns the number of spills that had to be emitted. - pub fn schedule_moves(&mut self, regs: &AllocatableSet) -> usize { + pub fn schedule_moves(&mut self, regs: &AllocatableSet, reginfo: &RegInfo) -> usize { self.collect_moves(); let mut avail = regs.clone(); @@ -724,9 +724,17 @@ impl Solver { .0; self.moves.swap(i, i + j); + // Check the top-level register class for an available register. It is an axiom of the + // register allocator that we can move between all registers in the top-level RC. let m = self.moves[i].clone(); - if let Some(reg) = avail.iter(m.rc).next() { - dbg!("breaking cycle at {} with available register %{}", m, reg); + let toprc = reginfo.toprc(m.rc); + if let Some(reg) = avail.iter(toprc).next() { + dbg!( + "breaking cycle at {} with available {} register {}", + m, + toprc, + reginfo.display_regunit(reg) + ); // Alter the move so it is guaranteed to be picked up when we loop. It is important // that this move is scheduled immediately, otherwise we would have multiple moves @@ -737,7 +745,7 @@ impl Solver { // next iteration. self.moves.push(Assignment { value: m.value, - rc: m.rc, + rc: toprc, from: reg, to: m.to, }); @@ -746,7 +754,7 @@ impl Solver { // may have to move two S-registers out of the way before we can resolve a cycle // involving a D-register. } else { - panic!("Not enough registers in {} to schedule moves", m.rc); + panic!("Not enough registers in {} to schedule moves", toprc); } } @@ -779,9 +787,8 @@ impl fmt::Display for Solver { mod tests { use entity::EntityRef; use ir::Value; - use isa::{TargetIsa, RegClass, RegUnit}; + use isa::{TargetIsa, RegClass, RegUnit, RegInfo}; use regalloc::AllocatableSet; - use std::borrow::Borrow; use super::{Solver, Assignment}; // Make an arm32 `TargetIsa`, if possible. @@ -796,12 +803,10 @@ mod tests { } // Get a register class by name. - fn rc_by_name(isa: &TargetIsa, name: &str) -> RegClass { - isa.register_info() - .classes - .iter() - .find(|rc| rc.name == name) - .expect("Can't find named register class.") + fn rc_by_name(reginfo: &RegInfo, name: &str) -> RegClass { + reginfo.classes.iter().find(|rc| rc.name == name).expect( + "Can't find named register class.", + ) } // Construct a move. @@ -817,8 +822,8 @@ mod tests { #[test] fn simple_moves() { let isa = arm32().expect("This test requires arm32 support"); - let isa = isa.borrow(); - let gpr = rc_by_name(isa, "GPR"); + let reginfo = isa.register_info(); + let gpr = rc_by_name(®info, "GPR"); let r0 = gpr.unit(0); let r1 = gpr.unit(1); let r2 = gpr.unit(2); @@ -833,7 +838,7 @@ mod tests { solver.reassign_in(v10, gpr, r1, r0); solver.inputs_done(); assert!(solver.quick_solve().is_ok()); - assert_eq!(solver.schedule_moves(®s), 0); + assert_eq!(solver.schedule_moves(®s, ®info), 0); assert_eq!(solver.moves(), &[mov(v10, gpr, r1, r0)]); // A bit harder: r0, r1 need to go in r1, r2. @@ -843,7 +848,7 @@ mod tests { solver.reassign_in(v11, gpr, r1, r2); solver.inputs_done(); assert!(solver.quick_solve().is_ok()); - assert_eq!(solver.schedule_moves(®s), 0); + assert_eq!(solver.schedule_moves(®s, ®info), 0); assert_eq!( solver.moves(), &[mov(v11, gpr, r1, r2), mov(v10, gpr, r0, r1)] @@ -855,7 +860,7 @@ mod tests { solver.reassign_in(v11, gpr, r1, r0); solver.inputs_done(); assert!(solver.quick_solve().is_ok()); - assert_eq!(solver.schedule_moves(®s), 0); + assert_eq!(solver.schedule_moves(®s, ®info), 0); assert_eq!( solver.moves(), &[ @@ -869,9 +874,9 @@ mod tests { #[test] fn harder_move_cycles() { let isa = arm32().expect("This test requires arm32 support"); - let isa = isa.borrow(); - let s = rc_by_name(isa, "S"); - let d = rc_by_name(isa, "D"); + let reginfo = isa.register_info(); + let s = rc_by_name(®info, "S"); + let d = rc_by_name(®info, "D"); let d0 = d.unit(0); let d1 = d.unit(1); let d2 = d.unit(2); @@ -894,7 +899,7 @@ mod tests { solver.reassign_in(v12, s, s3, s1); solver.inputs_done(); assert!(solver.quick_solve().is_ok()); - assert_eq!(solver.schedule_moves(®s), 0); + assert_eq!(solver.schedule_moves(®s, ®info), 0); assert_eq!( solver.moves(), &[ @@ -915,7 +920,7 @@ mod tests { solver.reassign_in(v10, d, d1, d0); solver.inputs_done(); assert!(solver.quick_solve().is_ok()); - assert_eq!(solver.schedule_moves(®s), 0); + assert_eq!(solver.schedule_moves(®s, ®info), 0); assert_eq!( solver.moves(), &[ From 196795017bae358a0ae53fb9db3408a8c430657d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Oct 2017 11:46:51 -0700 Subject: [PATCH 1211/3084] Tidy up handling of the DummyRuntime. --- cranelift/src/wasm.rs | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 840f108480..1c853efafa 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -4,7 +4,7 @@ //! IL. Can also executes the `start` function of the module by laying out the memories, globals //! and tables, then emitting the translated code with hardcoded addresses to memory. -use cton_wasm::{translate_module, DummyRuntime, WasmRuntime}; +use cton_wasm::{translate_module, DummyRuntime}; use std::path::PathBuf; use cretonne::Context; use cretonne::settings::FlagsOrIsa; @@ -99,10 +99,7 @@ fn handle_module( )?; } let mut dummy_runtime = DummyRuntime::with_flags(fisa.flags.clone()); - let translation = { - let runtime: &mut WasmRuntime = &mut dummy_runtime; - translate_module(&data, runtime)? - }; + let translation = translate_module(&data, &mut dummy_runtime)?; terminal.fg(term::color::GREEN).unwrap(); vprintln!(flag_verbose, "ok"); terminal.reset().unwrap(); From 7410ddfe08e7c10d50360a33306a7fafe2ee9e94 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 4 Oct 2017 16:55:14 -0700 Subject: [PATCH 1212/3084] Use the WasmRuntime's signature list rather than keeping a separate list. This way, if the runtime modifies the signature, such as to add special arguments, they are reflected in the resulting function defintions. --- lib/wasm/src/module_translator.rs | 6 ++---- lib/wasm/src/runtime/dummy.rs | 4 ++++ lib/wasm/src/runtime/spec.rs | 3 +++ lib/wasm/src/sections_translator.rs | 6 ++---- 4 files changed, 11 insertions(+), 8 deletions(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index b9df83f9fe..7966bdd543 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -39,7 +39,6 @@ pub fn translate_module( } ref s => panic!("modules should begin properly: {:?}", s), } - let mut signatures = None; let mut functions: Option> = None; let mut globals = Vec::new(); let mut exports: Option> = None; @@ -50,7 +49,7 @@ pub fn translate_module( match *parser.read_with_input(next_input) { ParserState::BeginSection { code: SectionCode::Type, .. } => { match parse_function_signatures(&mut parser, runtime) { - Ok(sigs) => signatures = Some(sigs), + Ok(()) => (), Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the type section: {}", s)) } @@ -192,7 +191,6 @@ pub fn translate_module( } // At this point we've entered the code section // First we check that we have all that is necessary to translate a function. - let signatures = signatures.unwrap_or_default(); let functions = match functions { None => return Err(String::from("missing a function section")), Some(functions) => functions, @@ -209,7 +207,7 @@ pub fn translate_module( runtime.next_function(); // First we build the Function object with its name and signature let mut func = Function::new(); - func.signature = signatures[functions[function_index]].clone(); + func.signature = runtime.get_signature(functions[function_index]).clone(); if let Some(ref exports) = exports { if let Some(name) = exports.get(&function_index) { func.name = FunctionName::new(name.clone()); diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index de630d8810..a2f58fb709 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -123,6 +123,10 @@ impl WasmRuntime for DummyRuntime { self.signatures.push(sig.clone()); } + fn get_signature(&self, sig_index: SignatureIndex) -> &ir::Signature { + &self.signatures[sig_index] + } + fn declare_func_import(&mut self, sig_index: SignatureIndex, module: &[u8], field: &[u8]) { assert_eq!( self.func_types.len(), diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 503e5079ef..74bc27770a 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -153,6 +153,9 @@ pub trait WasmRuntime: FuncEnvironment { /// Declares a function signature to the runtime. fn declare_signature(&mut self, sig: &ir::Signature); + /// Return the signature with the given index. + fn get_signature(&self, sig_index: SignatureIndex) -> &ir::Signature; + /// Declares a function import to the runtime. fn declare_func_import(&mut self, sig_index: SignatureIndex, module: &[u8], field: &[u8]); diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 7c0f794b3e..8329220a21 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -27,8 +27,7 @@ pub enum SectionParsingError { pub fn parse_function_signatures( parser: &mut Parser, runtime: &mut WasmRuntime, -) -> Result, SectionParsingError> { - let mut signatures: Vec = Vec::new(); +) -> Result<(), SectionParsingError> { loop { match *parser.read() { ParserState::EndSection => break, @@ -51,12 +50,11 @@ pub fn parse_function_signatures( ArgumentType::new(cret_arg) })); runtime.declare_signature(&sig); - signatures.push(sig); } ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), } } - Ok(signatures) + Ok(()) } /// Retrieves the imports from the imports section of the binary. From ce4d723a7304322408b4614a4ba5a4c382dec5c6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 4 Oct 2017 10:10:07 -0700 Subject: [PATCH 1213/3084] Give RegClassData a reference to its parent RegInfo. This makes it possible to materialize new RegClass references without requiring a RegInfo reference to be passed around. - Move the RegInfo::toprc() method to RegClassData. - Rename RegClassData::intersect() to intersect_index() and provide a new intersect() which returns a register class. - Remove some &RegInfo parameters that are no longer needed. --- lib/cretonne/meta/gen_encoding.py | 8 ++--- lib/cretonne/meta/gen_registers.py | 25 +++++++------- lib/cretonne/src/isa/intel/registers.rs | 18 +++++------ lib/cretonne/src/isa/registers.rs | 26 +++++++++------ lib/cretonne/src/regalloc/affinity.rs | 2 +- lib/cretonne/src/regalloc/allocatable_set.rs | 8 +++++ lib/cretonne/src/regalloc/coloring.rs | 16 +++------ lib/cretonne/src/regalloc/solver.rs | 34 ++++++++------------ 8 files changed, 69 insertions(+), 68 deletions(-) diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index a9bef26dba..0228c33839 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -790,22 +790,22 @@ def emit_operand_constraints( fmt.format('kind: ConstraintKind::Tied({}),', tied[n]) else: fmt.line('kind: ConstraintKind::Reg,') - fmt.format('regclass: {},', cons) + fmt.format('regclass: &{}_DATA,', cons) elif isinstance(cons, Register): assert n not in tied, "Can't tie fixed register operand" fmt.format( 'kind: ConstraintKind::FixedReg({}),', cons.unit) - fmt.format('regclass: {},', cons.regclass) + fmt.format('regclass: &{}_DATA,', cons.regclass) elif isinstance(cons, int): # This is a tied output constraint. It should never happen # for input constraints. assert cons == tied[n], "Invalid tied constraint" fmt.format('kind: ConstraintKind::Tied({}),', cons) - fmt.format('regclass: {},', recipe.ins[cons]) + fmt.format('regclass: &{}_DATA,', recipe.ins[cons]) elif isinstance(cons, Stack): assert n not in tied, "Can't tie stack operand" fmt.line('kind: ConstraintKind::Stack,') - fmt.format('regclass: {},', cons.regclass) + fmt.format('regclass: &{}_DATA,', cons.regclass) else: raise AssertionError( 'Unsupported constraint {}'.format(cons)) diff --git a/lib/cretonne/meta/gen_registers.py b/lib/cretonne/meta/gen_registers.py index 6d41a40330..9056441cb1 100644 --- a/lib/cretonne/meta/gen_registers.py +++ b/lib/cretonne/meta/gen_registers.py @@ -48,7 +48,9 @@ def gen_regclass(rc, fmt): """ Emit a static data definition for a register class. """ - with fmt.indented('RegClassData {', '},'): + with fmt.indented( + 'pub static {}_DATA: RegClassData = RegClassData {{' + .format(rc.name), '};'): fmt.format('name: "{}",', rc.name) fmt.format('index: {},', rc.index) fmt.format('width: {},', rc.width) @@ -58,6 +60,10 @@ def gen_regclass(rc, fmt): fmt.format('subclasses: 0x{:x},', rc.subclass_mask()) mask = ', '.join('0x{:08x}'.format(x) for x in rc.mask()) fmt.format('mask: [{}],', mask) + fmt.line('info: &INFO,') + # Also emit a convenient reference for use by hand-written code. + fmt.line('#[allow(dead_code)]') + fmt.format('pub static {0}: RegClass = &{0}_DATA;', rc.name) def gen_isa(isa, fmt): @@ -73,22 +79,13 @@ def gen_isa(isa, fmt): with fmt.indented('banks: &[', '],'): for regbank in isa.regbanks: gen_regbank(regbank, fmt) - fmt.line('classes: &CLASSES,') + with fmt.indented('classes: &[', '],'): + for rc in isa.regclasses: + fmt.format('&{}_DATA,', rc.name) # Register class descriptors. - with fmt.indented( - 'const CLASSES: [RegClassData; {}] = [' - .format(len(isa.regclasses)), '];'): - for idx, rc in enumerate(isa.regclasses): - assert idx == rc.index - gen_regclass(rc, fmt) - - # Emit constants referencing the register classes. for rc in isa.regclasses: - fmt.line('#[allow(dead_code)]') - fmt.line( - 'pub const {}: RegClass = &CLASSES[{}];' - .format(rc.name, rc.index)) + gen_regclass(rc, fmt) # Emit constants for all the register units. fmt.line('#[allow(dead_code, non_camel_case_types)]') diff --git a/lib/cretonne/src/isa/intel/registers.rs b/lib/cretonne/src/isa/intel/registers.rs index 14b4050d94..fcb1cf4445 100644 --- a/lib/cretonne/src/isa/intel/registers.rs +++ b/lib/cretonne/src/isa/intel/registers.rs @@ -49,14 +49,14 @@ mod tests { #[test] fn regclasses() { - assert_eq!(GPR.intersect(GPR), Some(GPR.into())); - assert_eq!(GPR.intersect(ABCD), Some(ABCD.into())); - assert_eq!(GPR.intersect(FPR), None); - assert_eq!(ABCD.intersect(GPR), Some(ABCD.into())); - assert_eq!(ABCD.intersect(ABCD), Some(ABCD.into())); - assert_eq!(ABCD.intersect(FPR), None); - assert_eq!(FPR.intersect(FPR), Some(FPR.into())); - assert_eq!(FPR.intersect(GPR), None); - assert_eq!(FPR.intersect(ABCD), None); + assert_eq!(GPR.intersect_index(GPR), Some(GPR.into())); + assert_eq!(GPR.intersect_index(ABCD), Some(ABCD.into())); + assert_eq!(GPR.intersect_index(FPR), None); + assert_eq!(ABCD.intersect_index(GPR), Some(ABCD.into())); + assert_eq!(ABCD.intersect_index(ABCD), Some(ABCD.into())); + assert_eq!(ABCD.intersect_index(FPR), None); + assert_eq!(FPR.intersect_index(FPR), Some(FPR.into())); + assert_eq!(FPR.intersect_index(GPR), None); + assert_eq!(FPR.intersect_index(ABCD), None); } } diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index a1de76d97e..961aae9c83 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -145,14 +145,17 @@ pub struct RegClassData { /// Mask of register units in the class. If `width > 1`, the mask only has a bit set for the /// first register unit in each allocatable register. pub mask: RegUnitMask, + + /// The global `RegInfo` instance containing that this register class. + pub info: &'static RegInfo, } impl RegClassData { - /// Get the register class corresponding to the intersection of `self` and `other`. + /// Get the register class index corresponding to the intersection of `self` and `other`. /// /// This register class is guaranteed to exist if the register classes overlap. If the register /// classes don't overlap, returns `None`. - pub fn intersect(&self, other: RegClass) -> Option { + pub fn intersect_index(&self, other: RegClass) -> Option { // Compute the set of common subclasses. let mask = self.subclasses & other.subclasses; @@ -166,12 +169,22 @@ impl RegClassData { } } + /// Get the intersection of `self` and `other`. + pub fn intersect(&self, other: RegClass) -> Option { + self.intersect_index(other).map(|rci| self.info.rc(rci)) + } + /// Returns true if `other` is a subclass of this register class. /// A register class is considered to be a subclass of itself. pub fn has_subclass>(&self, other: RCI) -> bool { self.subclasses & (1 << other.into().0) != 0 } + /// Get the top-level register class containing this class. + pub fn toprc(&self) -> RegClass { + self.info.rc(RegClassIndex(self.toprc)) + } + /// Get a specific register unit in this class. pub fn unit(&self, offset: usize) -> RegUnit { let uoffset = offset * usize::from(self.width); @@ -246,7 +259,7 @@ pub struct RegInfo { pub banks: &'static [RegBank], /// All register classes ordered topologically so a sub-class always follows its parent. - pub classes: &'static [RegClassData], + pub classes: &'static [RegClass], } impl RegInfo { @@ -274,12 +287,7 @@ impl RegInfo { /// Get the register class corresponding to `idx`. pub fn rc(&self, idx: RegClassIndex) -> RegClass { - &self.classes[idx.index()] - } - - /// Get the top-level register class containing `rc`. - pub fn toprc(&self, rc: RegClass) -> RegClass { - &self.classes[rc.toprc as usize] + self.classes[idx.index()] } } diff --git a/lib/cretonne/src/regalloc/affinity.rs b/lib/cretonne/src/regalloc/affinity.rs index d7dede67d2..5cfd3ada97 100644 --- a/lib/cretonne/src/regalloc/affinity.rs +++ b/lib/cretonne/src/regalloc/affinity.rs @@ -95,7 +95,7 @@ impl Affinity { { // If the register classes don't overlap, `intersect` returns `None`, and we // just keep our previous affinity. - if let Some(subclass) = constraint.regclass.intersect(reg_info.rc(rc)) { + if let Some(subclass) = constraint.regclass.intersect_index(reg_info.rc(rc)) { // This constraint shrinks our preferred register class. *self = Affinity::Reg(subclass); } diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index c765d73b1a..c6f8c86a07 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -198,7 +198,9 @@ mod tests { first: 28, subclasses: 0, mask: [0xf0000000, 0x0000000f, 0], + info: &INFO, }; + const DPR: RegClass = &RegClassData { name: "DPR", index: 0, @@ -208,6 +210,12 @@ mod tests { first: 28, subclasses: 0, mask: [0x50000000, 0x0000000a, 0], + info: &INFO, + }; + + const INFO: RegInfo = RegInfo { + banks: &[], + classes: &[], }; #[test] diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 9aa5f48b2b..115fc2513b 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -445,12 +445,7 @@ impl<'a> Context<'a> { ConstraintKind::Reg | ConstraintKind::Tied(_) => { if !op.regclass.contains(cur_reg) { - self.solver.add_var( - value, - op.regclass, - cur_reg, - &self.reginfo, - ); + self.solver.add_var(value, op.regclass, cur_reg); } } ConstraintKind::Stack => unreachable!(), @@ -572,7 +567,7 @@ impl<'a> Context<'a> { let rc = self.reginfo.rc(rci); let reg = self.divert.reg(lv.value, &self.cur.func.locations); if self.solver.is_fixed_input_conflict(rc, reg) { - self.solver.add_var(lv.value, rc, reg, &self.reginfo); + self.solver.add_var(lv.value, rc, reg); } } } @@ -638,7 +633,7 @@ impl<'a> Context<'a> { // TODO: Use a looser constraint than the affinity hint. Any allocatable // register in the top-level register class would be OK. Maybe `add_var` // should take both a preferred class and a required constraint class. - self.solver.add_var(lv.value, rc2, reg2, &self.reginfo); + self.solver.add_var(lv.value, rc2, reg2); } } } @@ -712,8 +707,7 @@ impl<'a> Context<'a> { // The new variable gets to roam the whole top-level register class because // it is not actually constrained by the instruction. We just want it out // of the way. - let toprc = self.reginfo.toprc(rc2); - self.solver.add_var(lv.value, toprc, reg2, &self.reginfo); + self.solver.add_var(lv.value, rc2.toprc(), reg2); return true; } } @@ -730,7 +724,7 @@ impl<'a> Context<'a> { /// /// The solver needs to be reminded of the available registers before any moves are inserted. fn shuffle_inputs(&mut self, regs: &mut AllocatableSet) { - self.solver.schedule_moves(regs, &self.reginfo); + self.solver.schedule_moves(regs); for m in self.solver.moves() { self.divert.regmove(m.value, m.from, m.to); diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index e3491ba8cc..0256b0d0d1 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -101,7 +101,7 @@ use dbg::DisplayList; use entity::{SparseMap, SparseMapValue}; use ir::Value; -use isa::{RegInfo, RegClass, RegUnit}; +use isa::{RegClass, RegUnit}; use regalloc::allocatable_set::RegSetIter; use std::fmt; use super::AllocatableSet; @@ -391,20 +391,14 @@ impl Solver { /// /// It is assumed initially that the value is also live on the output side of the instruction. /// This can be changed by calling to `add_kill()`. - pub fn add_var( - &mut self, - value: Value, - constraint: RegClass, - from: RegUnit, - reginfo: &RegInfo, - ) { + pub fn add_var(&mut self, value: Value, constraint: RegClass, from: RegUnit) { // Check for existing entries for this value. if self.regs_in.is_avail(constraint, from) { dbg!( "add_var({}:{}, from={}/%{}) for existing entry", value, constraint, - reginfo.display_regunit(from), + constraint.info.display_regunit(from), from ); @@ -413,8 +407,8 @@ impl Solver { dbg!("-> combining constraint with {}", v); // We have an existing variable entry for `value`. Combine the constraints. - if let Some(rci) = v.constraint.intersect(constraint) { - v.constraint = reginfo.rc(rci); + if let Some(rc) = v.constraint.intersect(constraint) { + v.constraint = rc; return; } else { // The spiller should have made sure the same value is not used with disjoint @@ -443,7 +437,7 @@ impl Solver { "add_var({}:{}, from={}/%{}) new entry: {}", value, constraint, - reginfo.display_regunit(from), + constraint.info.display_regunit(from), from, new_var ); @@ -679,7 +673,7 @@ impl Solver { /// a register. /// /// Returns the number of spills that had to be emitted. - pub fn schedule_moves(&mut self, regs: &AllocatableSet, reginfo: &RegInfo) -> usize { + pub fn schedule_moves(&mut self, regs: &AllocatableSet) -> usize { self.collect_moves(); let mut avail = regs.clone(); @@ -727,13 +721,13 @@ impl Solver { // Check the top-level register class for an available register. It is an axiom of the // register allocator that we can move between all registers in the top-level RC. let m = self.moves[i].clone(); - let toprc = reginfo.toprc(m.rc); + let toprc = m.rc.toprc(); if let Some(reg) = avail.iter(toprc).next() { dbg!( "breaking cycle at {} with available {} register {}", m, toprc, - reginfo.display_regunit(reg) + toprc.info.display_regunit(reg) ); // Alter the move so it is guaranteed to be picked up when we loop. It is important @@ -838,7 +832,7 @@ mod tests { solver.reassign_in(v10, gpr, r1, r0); solver.inputs_done(); assert!(solver.quick_solve().is_ok()); - assert_eq!(solver.schedule_moves(®s, ®info), 0); + assert_eq!(solver.schedule_moves(®s), 0); assert_eq!(solver.moves(), &[mov(v10, gpr, r1, r0)]); // A bit harder: r0, r1 need to go in r1, r2. @@ -848,7 +842,7 @@ mod tests { solver.reassign_in(v11, gpr, r1, r2); solver.inputs_done(); assert!(solver.quick_solve().is_ok()); - assert_eq!(solver.schedule_moves(®s, ®info), 0); + assert_eq!(solver.schedule_moves(®s), 0); assert_eq!( solver.moves(), &[mov(v11, gpr, r1, r2), mov(v10, gpr, r0, r1)] @@ -860,7 +854,7 @@ mod tests { solver.reassign_in(v11, gpr, r1, r0); solver.inputs_done(); assert!(solver.quick_solve().is_ok()); - assert_eq!(solver.schedule_moves(®s, ®info), 0); + assert_eq!(solver.schedule_moves(®s), 0); assert_eq!( solver.moves(), &[ @@ -899,7 +893,7 @@ mod tests { solver.reassign_in(v12, s, s3, s1); solver.inputs_done(); assert!(solver.quick_solve().is_ok()); - assert_eq!(solver.schedule_moves(®s, ®info), 0); + assert_eq!(solver.schedule_moves(®s), 0); assert_eq!( solver.moves(), &[ @@ -920,7 +914,7 @@ mod tests { solver.reassign_in(v10, d, d1, d0); solver.inputs_done(); assert!(solver.quick_solve().is_ok()); - assert_eq!(solver.schedule_moves(®s, ®info), 0); + assert_eq!(solver.schedule_moves(®s), 0); assert_eq!( solver.moves(), &[ From e32aa8ab606590f4494619bf50f7580959a9e6d6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 3 Oct 2017 18:04:56 -0700 Subject: [PATCH 1214/3084] Emergency spilling for the solver's move scheduler. The register constraint solver schedules a set of move instructions to execute before the instruction getting colored. In extreme cases, this is not possible because there are no available registers to break cycles in the register assignments that must be scheduled. When that happens, we spill one register to an emergency slot so it becomes available for implementing the assignment cycle. Then the original register is restored. The coloring pass can't yet understand the spill and fill move types. This will be implemented next. --- lib/cretonne/src/isa/registers.rs | 7 + lib/cretonne/src/regalloc/coloring.rs | 12 +- lib/cretonne/src/regalloc/solver.rs | 369 +++++++++++++++++++++++--- 3 files changed, 341 insertions(+), 47 deletions(-) diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 961aae9c83..b8e7fe1148 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -209,6 +209,13 @@ impl fmt::Debug for RegClassData { } } +/// Within an ISA, register classes are uniquely identified by their index. +impl PartialEq for RegClassData { + fn eq(&self, other: &RegClassData) -> bool { + self.index == other.index + } +} + /// A small reference to a register class. /// /// Use this when storing register classes in compact data structures. The `RegInfo::rc()` method diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 115fc2513b..3bf8a34fc8 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -724,11 +724,17 @@ impl<'a> Context<'a> { /// /// The solver needs to be reminded of the available registers before any moves are inserted. fn shuffle_inputs(&mut self, regs: &mut AllocatableSet) { - self.solver.schedule_moves(regs); + use regalloc::solver::Move::*; + self.solver.schedule_moves(regs); for m in self.solver.moves() { - self.divert.regmove(m.value, m.from, m.to); - self.cur.ins().regmove(m.value, m.from, m.to); + match *m { + Reg { value, from, to, .. } => { + self.divert.regmove(value, from, to); + self.cur.ins().regmove(value, from, to); + } + Spill { .. } | Fill { .. } => unimplemented!(), + } } } diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 0256b0d0d1..5f6f27cbfd 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -104,6 +104,7 @@ use ir::Value; use isa::{RegClass, RegUnit}; use regalloc::allocatable_set::RegSetIter; use std::fmt; +use std::mem; use super::AllocatableSet; /// A variable in the constraint problem. @@ -196,6 +197,9 @@ impl Variable { impl fmt::Display for Variable { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "{}({}", self.value, self.constraint)?; + if let Some(reg) = self.from { + write!(f, ", from {}", self.constraint.info.display_regunit(reg))?; + } if self.is_input { write!(f, ", in")?; } @@ -231,22 +235,182 @@ impl SparseMapValue for Assignment { impl fmt::Display for Assignment { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + let ri = self.rc.info; write!( f, - "{}:{}(%{} -> %{})", + "{}:{}({} -> {})", self.value, self.rc, - self.from, - self.to + ri.display_regunit(self.from), + ri.display_regunit(self.to) ) } } -#[cfg(test)] -impl PartialEq for Assignment { - fn eq(&self, other: &Assignment) -> bool { - self.value == other.value && self.from == other.from && self.to == other.to && - self.rc.index == other.rc.index +/// A move operation between two registers or between a register and an emergency spill slot. +#[derive(Clone, PartialEq)] +pub enum Move { + Reg { + value: Value, + rc: RegClass, + from: RegUnit, + to: RegUnit, + }, + Spill { + value: Value, + rc: RegClass, + from: RegUnit, + to_slot: usize, + }, + Fill { + value: Value, + rc: RegClass, + from_slot: usize, + to: RegUnit, + }, +} + +impl Move { + /// Create a register move from an assignment, but not for identity assignments. + fn with_assignment(a: &Assignment) -> Option { + if a.from != a.to { + Some(Move::Reg { + value: a.value, + from: a.from, + to: a.to, + rc: a.rc, + }) + } else { + None + } + } + + /// Get the "from" register and register class, if possible. + fn from_reg(&self) -> Option<(RegClass, RegUnit)> { + match *self { + Move::Reg { rc, from, .. } | + Move::Spill { rc, from, .. } => Some((rc, from)), + Move::Fill { .. } => None, + } + } + + /// Get the "to" register and register class, if possible. + fn to_reg(&self) -> Option<(RegClass, RegUnit)> { + match *self { + Move::Reg { rc, to, .. } | + Move::Fill { rc, to, .. } => Some((rc, to)), + Move::Spill { .. } => None, + } + } + + /// Replace the "to" register with `new` and return the old value. + fn replace_to_reg(&mut self, new: RegUnit) -> RegUnit { + mem::replace( + match self { + &mut Move::Reg { ref mut to, .. } | + &mut Move::Fill { ref mut to, .. } => to, + &mut Move::Spill { .. } => panic!("No to register in a spill {}", self), + }, + new, + ) + } + + /// Convert this `Reg` move to a spill to `slot` and return the old "to" register. + fn change_to_spill(&mut self, slot: usize) -> RegUnit { + match self.clone() { + Move::Reg { + value, + rc, + from, + to, + } => { + *self = Move::Spill { + value, + rc, + from, + to_slot: slot, + }; + to + } + _ => panic!("Expected reg move: {}", self), + } + } + + + /// Get the value being moved. + fn value(&self) -> Value { + match *self { + Move::Reg { value, .. } | + Move::Fill { value, .. } | + Move::Spill { value, .. } => value, + } + } + + /// Get the associated register class. + fn rc(&self) -> RegClass { + match *self { + Move::Reg { rc, .. } | + Move::Fill { rc, .. } | + Move::Spill { rc, .. } => rc, + } + } +} + +impl fmt::Display for Move { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match *self { + Move::Reg { + value, + from, + to, + rc, + } => { + write!( + f, + "{}:{}({} -> {})", + value, + rc, + rc.info.display_regunit(from), + rc.info.display_regunit(to) + ) + } + Move::Spill { + value, + from, + to_slot, + rc, + } => { + write!( + f, + "{}:{}({} -> slot {})", + value, + rc, + rc.info.display_regunit(from), + to_slot + ) + } + Move::Fill { + value, + from_slot, + to, + rc, + } => { + write!( + f, + "{}:{}(slot {} -> {})", + value, + rc, + from_slot, + rc.info.display_regunit(to) + ) + } + } + } +} + +impl fmt::Debug for Move { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + (self as &fmt::Display).fmt(f) } } @@ -317,7 +481,10 @@ pub struct Solver { /// List of register moves scheduled to avoid conflicts. /// /// This is used as working space by the `schedule_moves()` function. - moves: Vec, + moves: Vec, + + /// List of pending fill moves. This is only used during `schedule_moves()`. + fills: Vec, } /// Interface for programming the constraints into the solver. @@ -331,6 +498,7 @@ impl Solver { regs_in: AllocatableSet::new(), regs_out: AllocatableSet::new(), moves: Vec::new(), + fills: Vec::new(), } } @@ -395,11 +563,10 @@ impl Solver { // Check for existing entries for this value. if self.regs_in.is_avail(constraint, from) { dbg!( - "add_var({}:{}, from={}/%{}) for existing entry", + "add_var({}:{}, from={}) for existing entry", value, constraint, constraint.info.display_regunit(from), - from ); // There could be an existing variable entry. @@ -434,11 +601,10 @@ impl Solver { let new_var = Variable::new_live(value, constraint, from); dbg!( - "add_var({}:{}, from={}/%{}) new entry: {}", + "add_var({}:{}, from={}) new entry: {}", value, constraint, constraint.info.display_regunit(from), - from, new_var ); @@ -645,7 +811,7 @@ impl Solver { // Collect moves from the chosen solution for all non-define variables. for v in &self.vars { if let Some(from) = v.from { - self.moves.push(Assignment { + self.moves.push(Move::Reg { value: v.value, from, to: v.solution, @@ -656,8 +822,8 @@ impl Solver { // Convert all of the fixed register assignments into moves, but omit the ones that are // already in the right register. - self.moves.extend(self.assignments.values().cloned().filter( - |v| v.from != v.to, + self.moves.extend(self.assignments.values().filter_map( + Move::with_assignment, )); dbg!("collect_moves: {}", DisplayList(&self.moves)); @@ -675,33 +841,46 @@ impl Solver { /// Returns the number of spills that had to be emitted. pub fn schedule_moves(&mut self, regs: &AllocatableSet) -> usize { self.collect_moves(); + assert!(self.fills.is_empty()); + let mut num_spill_slots = 0; let mut avail = regs.clone(); let mut i = 0; - while i < self.moves.len() { + while i < self.moves.len() + self.fills.len() { + // Don't even look at the fills until we've spent all the moves. Deferring these let's + // us potentially reuse the claimed registers to resolve multiple cycles. + if i >= self.moves.len() { + self.moves.append(&mut self.fills); + } + // Find the first move that can be executed now. - if let Some(j) = self.moves[i..].iter().position( - |m| avail.is_avail(m.rc, m.to), - ) + if let Some(j) = self.moves[i..].iter().position(|m| match m.to_reg() { + Some((rc, reg)) => avail.is_avail(rc, reg), + None => true, + }) { // This move can be executed now. self.moves.swap(i, i + j); let m = &self.moves[i]; - avail.take(m.rc, m.to); - avail.free(m.rc, m.from); + if let Some((rc, reg)) = m.to_reg() { + avail.take(rc, reg); + } + if let Some((rc, reg)) = m.from_reg() { + avail.free(rc, reg); + } dbg!("move #{}: {}", i, m); i += 1; continue; } - // When we get here, non of the `moves[i..]` can be executed. This means there are only - // cycles remaining. The cycles can be broken in a few ways: + // When we get here, none of the `moves[i..]` can be executed. This means there are + // only cycles remaining. The cycles can be broken in a few ways: // // 1. Grab an available register and use it to break a cycle. // 2. Move a value temporarily into a stack slot instead of a register. // 3. Use swap instructions. // - // TODO: So far we only implement 1. + // TODO: So far we only implement 1 and 2. // Pick an assignment with the largest possible width. This is more likely to break up // a cycle than an assignment with fewer register units. For example, it may be @@ -713,7 +892,7 @@ impl Solver { let j = self.moves[i..] .iter() .enumerate() - .min_by_key(|&(_, m)| !m.rc.width) + .min_by_key(|&(_, m)| !m.rc().width) .unwrap() .0; self.moves.swap(i, i + j); @@ -721,7 +900,7 @@ impl Solver { // Check the top-level register class for an available register. It is an axiom of the // register allocator that we can move between all registers in the top-level RC. let m = self.moves[i].clone(); - let toprc = m.rc.toprc(); + let toprc = m.rc().toprc(); if let Some(reg) = avail.iter(toprc).next() { dbg!( "breaking cycle at {} with available {} register {}", @@ -733,31 +912,42 @@ impl Solver { // Alter the move so it is guaranteed to be picked up when we loop. It is important // that this move is scheduled immediately, otherwise we would have multiple moves // of the same value, and they would not be commutable. - self.moves[i].to = reg; + let old_to_reg = self.moves[i].replace_to_reg(reg); // Append a fixup move so we end up in the right place. This move will be scheduled // later. That's ok because it is the single remaining move of `m.value` after the // next iteration. - self.moves.push(Assignment { - value: m.value, + self.moves.push(Move::Reg { + value: m.value(), rc: toprc, from: reg, - to: m.to, + to: old_to_reg, }); - // TODO: What if allocating an extra register is not enough to break a cycle? This - // can happen when there are registers of different widths in a cycle. For ARM, we - // may have to move two S-registers out of the way before we can resolve a cycle - // involving a D-register. - } else { - panic!("Not enough registers in {} to schedule moves", toprc); + // TODO: What if allocating an extra register is not enough to break a cycle? This + // can happen when there are registers of different widths in a cycle. For ARM, we + // may have to move two S-registers out of the way before we can resolve a cycle + // involving a D-register. + continue; } + + // It was impossible to free up a register in toprc, so use an emergency spill slot as + // a last resort. + let slot = num_spill_slots; + num_spill_slots += 1; + dbg!("breaking cycle at {} with slot {}", m, slot); + let old_to_reg = self.moves[i].change_to_spill(slot); + self.fills.push(Move::Fill { + value: m.value(), + rc: toprc, + from_slot: slot, + to: old_to_reg, + }); } - // Spilling not implemented yet. - 0 + num_spill_slots } /// Borrow the scheduled set of register moves that was computed by `schedule_moves()`. - pub fn moves(&self) -> &[Assignment] { + pub fn moves(&self) -> &[Move] { &self.moves } } @@ -783,7 +973,7 @@ mod tests { use ir::Value; use isa::{TargetIsa, RegClass, RegUnit, RegInfo}; use regalloc::AllocatableSet; - use super::{Solver, Assignment}; + use super::{Solver, Move}; // Make an arm32 `TargetIsa`, if possible. fn arm32() -> Option> { @@ -803,9 +993,9 @@ mod tests { ) } - // Construct a move. - fn mov(value: Value, rc: RegClass, from: RegUnit, to: RegUnit) -> Assignment { - Assignment { + // Construct a register move. + fn mov(value: Value, rc: RegClass, from: RegUnit, to: RegUnit) -> Move { + Move::Reg { value, rc, from, @@ -813,6 +1003,24 @@ mod tests { } } + fn spill(value: Value, rc: RegClass, from: RegUnit, to_slot: usize) -> Move { + Move::Spill { + value, + rc, + from, + to_slot, + } + } + + fn fill(value: Value, rc: RegClass, from_slot: usize, to: RegUnit) -> Move { + Move::Fill { + value, + rc, + from_slot, + to, + } + } + #[test] fn simple_moves() { let isa = arm32().expect("This test requires arm32 support"); @@ -925,4 +1133,77 @@ mod tests { ] ); } + + #[test] + fn emergency_spill() { + let isa = arm32().expect("This test requires arm32 support"); + let reginfo = isa.register_info(); + let gpr = rc_by_name(®info, "GPR"); + let r0 = gpr.unit(0); + let r1 = gpr.unit(1); + let r2 = gpr.unit(2); + let r3 = gpr.unit(3); + let r4 = gpr.unit(4); + let r5 = gpr.unit(5); + let mut regs = AllocatableSet::new(); + let mut solver = Solver::new(); + let v10 = Value::new(10); + let v11 = Value::new(11); + let v12 = Value::new(12); + let v13 = Value::new(13); + let v14 = Value::new(14); + let v15 = Value::new(15); + + // Claim r0--r2 and r3--r15 for other values. + for i in 0..16 { + regs.take(gpr, gpr.unit(i)); + } + + // Request a permutation cycle. + solver.reset(®s); + solver.reassign_in(v10, gpr, r0, r1); + solver.reassign_in(v11, gpr, r1, r2); + solver.reassign_in(v12, gpr, r2, r0); + solver.inputs_done(); + assert!(solver.quick_solve().is_ok()); + assert_eq!(solver.schedule_moves(®s), 1); + assert_eq!( + solver.moves(), + &[ + spill(v10, gpr, r0, 0), + mov(v12, gpr, r2, r0), + mov(v11, gpr, r1, r2), + fill(v10, gpr, 0, r1), + ] + ); + + // Two cycles should only require a single spill. + solver.reset(®s); + // Cycle 1. + solver.reassign_in(v10, gpr, r0, r1); + solver.reassign_in(v11, gpr, r1, r2); + solver.reassign_in(v12, gpr, r2, r0); + // Cycle 2. + solver.reassign_in(v13, gpr, r3, r4); + solver.reassign_in(v14, gpr, r4, r5); + solver.reassign_in(v15, gpr, r5, r3); + + solver.inputs_done(); + assert!(solver.quick_solve().is_ok()); + // We resolve two cycles with one spill. + assert_eq!(solver.schedule_moves(®s), 1); + assert_eq!( + solver.moves(), + &[ + spill(v10, gpr, r0, 0), + mov(v12, gpr, r2, r0), + mov(v11, gpr, r1, r2), + mov(v13, gpr, r3, r1), // Use available r1 to break cycle 2. + mov(v15, gpr, r5, r3), + mov(v14, gpr, r4, r5), + mov(v13, gpr, r1, r4), + fill(v10, gpr, 0, r1), // Finally complete cycle 1. + ] + ); + } } From d4aeec6ecee81f806dfea94e8f850a12cf116da1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 4 Oct 2017 12:13:25 -0700 Subject: [PATCH 1215/3084] Generalize RegDiversions to track stack locations too. For emergency spilling, we need to be able to temporarily divert an SSA value to a stack slot if there are no available registers. --- lib/cretonne/src/regalloc/coloring.rs | 17 +++-- lib/cretonne/src/regalloc/diversion.rs | 91 ++++++++++++++++---------- 2 files changed, 69 insertions(+), 39 deletions(-) diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 3bf8a34fc8..9a92df0270 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -547,7 +547,14 @@ impl<'a> Context<'a> { if pred(lr, &self.cur.func) { if let Affinity::Reg(rci) = lr.affinity { let rc = self.reginfo.rc(rci); - self.solver.reassign_in(rdiv.value, rc, rdiv.to, rdiv.from); + // Stack diversions should not be possible here. The only live transiently + // during `shuffle_inputs()`. + self.solver.reassign_in( + rdiv.value, + rc, + rdiv.to.unwrap_reg(), + rdiv.from.unwrap_reg(), + ); } else { panic!( "Diverted register {} with {} affinity", @@ -758,11 +765,11 @@ impl<'a> Context<'a> { for lv in kills { if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); - let reg = match self.divert.remove(lv.value) { - Some(r) => r, - None => self.cur.func.locations[lv.value].unwrap_reg(), + let loc = match self.divert.remove(lv.value) { + Some(loc) => loc, + None => self.cur.func.locations[lv.value], }; - regs.free(rc, reg); + regs.free(rc, loc.unwrap_reg()); } } } diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index 769998de3a..9d241ebcdd 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -7,12 +7,11 @@ //! These register diversions are local to an EBB. No values can be diverted when entering a new //! EBB. -use entity::EntityMap; -use ir::{Value, ValueLoc}; +use ir::{Value, ValueLoc, ValueLocations, StackSlot}; use isa::{RegUnit, RegInfo}; use std::fmt; -/// A diversion of a value from its original register location to a new register. +/// A diversion of a value from its original location to a new register or stack location. /// /// In IL, a diversion is represented by a `regmove` instruction, possibly a chain of them for the /// same value. @@ -23,20 +22,21 @@ use std::fmt; pub struct Diversion { /// The value that is diverted. pub value: Value, - /// The original register value location. - pub from: RegUnit, - /// The current register value location. - pub to: RegUnit, + /// The original value location. + pub from: ValueLoc, + /// The current value location. + pub to: ValueLoc, } impl Diversion { - /// Make a new register diversion. - pub fn new(value: Value, from: RegUnit, to: RegUnit) -> Diversion { + /// Make a new diversion. + pub fn new(value: Value, from: ValueLoc, to: ValueLoc) -> Diversion { + debug_assert!(from.is_assigned() && to.is_assigned()); Diversion { value, from, to } } } -/// Keep track of register diversions in an EBB. +/// Keep track of diversions in an EBB. pub struct RegDiversions { current: Vec, } @@ -67,17 +67,30 @@ impl RegDiversions { self.current.as_slice() } - /// Get the current register location for `value`. Fall back to the assignment map for - /// non-diverted values. - pub fn reg(&self, value: Value, locations: &EntityMap) -> RegUnit { + /// Get the current location for `value`. Fall back to the assignment map for non-diverted + /// values + pub fn get(&self, value: Value, locations: &ValueLocations) -> ValueLoc { match self.diversion(value) { Some(d) => d.to, - None => locations[value].unwrap_reg(), + None => locations[value], } } - /// Record a register move. - pub fn regmove(&mut self, value: Value, from: RegUnit, to: RegUnit) { + /// Get the current register location for `value`, or panic if `value` isn't in a register. + pub fn reg(&self, value: Value, locations: &ValueLocations) -> RegUnit { + self.get(value, locations).unwrap_reg() + } + + /// Get the current stack location for `value`, or panic if `value` isn't in a stack slot. + pub fn stack(&self, value: Value, locations: &ValueLocations) -> StackSlot { + self.get(value, locations).unwrap_stack() + } + + /// Record any kind of move. + /// + /// The `from` location must match an existing `to` location, if any. + pub fn divert(&mut self, value: Value, from: ValueLoc, to: ValueLoc) { + debug_assert!(from.is_assigned() && to.is_assigned()); if let Some(i) = self.current.iter().position(|d| d.value == value) { debug_assert_eq!(self.current[i].to, from, "Bad regmove chain for {}", value); if self.current[i].from != to { @@ -90,10 +103,25 @@ impl RegDiversions { } } - /// Drop any recorded register move for `value`. + /// Record a register -> register move. + pub fn regmove(&mut self, value: Value, from: RegUnit, to: RegUnit) { + self.divert(value, ValueLoc::Reg(from), ValueLoc::Reg(to)); + } + + /// Record a register -> stack move. + pub fn regspill(&mut self, value: Value, from: RegUnit, to: StackSlot) { + self.divert(value, ValueLoc::Reg(from), ValueLoc::Stack(to)); + } + + /// Record a stack -> register move. + pub fn regfill(&mut self, value: Value, from: StackSlot, to: RegUnit) { + self.divert(value, ValueLoc::Stack(from), ValueLoc::Reg(to)); + } + + /// Drop any recorded move for `value`. /// - /// Returns the `to` register of the removed diversion. - pub fn remove(&mut self, value: Value) -> Option { + /// Returns the `to` location of the removed diversion. + pub fn remove(&mut self, value: Value) -> Option { self.current.iter().position(|d| d.value == value).map( |i| { self.current.swap_remove(i).to @@ -114,18 +142,13 @@ impl<'a> fmt::Display for DisplayDiversions<'a> { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "{{")?; for div in self.0.all() { - match self.1 { - Some(regs) => { - write!( - f, - " {}: {} -> {}", - div.value, - regs.display_regunit(div.from), - regs.display_regunit(div.to) - )? - } - None => write!(f, " {}: %{} -> %{}", div.value, div.from, div.to)?, - } + write!( + f, + " {}: {} -> {}", + div.value, + div.from.display(self.1), + div.to.display(self.1) + )? } write!(f, " }}") } @@ -148,14 +171,14 @@ mod tests { divs.diversion(v1), Some(&Diversion { value: v1, - from: 10, - to: 12, + from: ValueLoc::Reg(10), + to: ValueLoc::Reg(12), }) ); assert_eq!(divs.diversion(v2), None); divs.regmove(v1, 12, 11); - assert_eq!(divs.diversion(v1).unwrap().to, 11); + assert_eq!(divs.diversion(v1).unwrap().to, ValueLoc::Reg(11)); divs.regmove(v1, 11, 10); assert_eq!(divs.diversion(v1), None); } From dda3efcbdd18e2588878db420adcf40f7083db92 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 4 Oct 2017 12:42:53 -0700 Subject: [PATCH 1216/3084] Add regspill and regfill instructions. These are parallels to the existing regmove instruction, but the divert the value to and from a stack slot. Like regmove diversions, this is a temporary diversion that must be local to the EBB. --- cranelift/docs/langref.rst | 5 +++- cranelift/filetests/parser/tiny.cton | 9 +++++- lib/cretonne/meta/base/formats.py | 4 +++ lib/cretonne/meta/base/instructions.py | 28 ++++++++++++++++++ lib/cretonne/meta/gen_binemit.py | 6 +++- lib/cretonne/src/ir/instructions.rs | 12 ++++++++ lib/cretonne/src/verifier/mod.rs | 6 ++++ lib/cretonne/src/write.rs | 16 +++++++++++ lib/reader/src/parser.rs | 40 ++++++++++++++++++++++++++ 9 files changed, 123 insertions(+), 3 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 1e148190e6..5d2b46cc73 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -700,9 +700,12 @@ allocation pass and beyond. .. autoinst:: fill Register values can be temporarily diverted to other registers by the -:inst:`regmove` instruction. +:inst:`regmove` instruction, and to and from stack slots by :inst:`regspill` +and :inst:`regfill`. .. autoinst:: regmove +.. autoinst:: regspill +.. autoinst:: regfill Vector operations ----------------- diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index e538b5acaa..9727e40177 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -166,14 +166,21 @@ ebb0(v1: i32): ; Register diversions. ; This test file has no ISA, so we can unly use register unit numbers. function %diversion(i32) { + ss0 = spill_slot 4 + ebb0(v1: i32): regmove v1, %10 -> %20 regmove v1, %20 -> %10 + regspill v1, %10 -> ss0 + regfill v1, ss0 -> %10 return } ; sameln: function %diversion(i32) native { -; nextln: ebb0($v1: i32): +; nextln: $ss0 = spill_slot 4 +; check: ebb0($v1: i32): ; nextln: regmove $v1, %10 -> %20 ; nextln: regmove $v1, %20 -> %10 +; nextln: regspill $v1, %10 -> $ss0 +; nextln: regfill $v1, $ss0 -> %10 ; nextln: return ; nextln: } diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 7fd93de1f9..0f31ff2c5f 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -58,6 +58,10 @@ StackStore = InstructionFormat(VALUE, stack_slot, offset32) HeapAddr = InstructionFormat(heap, VALUE, uimm32) RegMove = InstructionFormat(VALUE, ('src', regunit), ('dst', regunit)) +RegSpill = InstructionFormat( + VALUE, ('src', regunit), ('dst', entities.stack_slot)) +RegFill = InstructionFormat( + VALUE, ('src', entities.stack_slot), ('dst', regunit)) Trap = InstructionFormat(trapcode) CondTrap = InstructionFormat(VALUE, trapcode) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index f49a532a3f..6e94b4c553 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -514,6 +514,34 @@ regmove = Instruction( ins=(x, src, dst), other_side_effects=True) +regspill = Instruction( + 'regspill', r""" + Temporarily divert ``x`` from ``src`` to ``SS``. + + This instruction moves the location of a value from a register to a + stack slot without creating a new SSA value. It is used by the register + allocator to temporarily rearrange register assignments in order to + satisfy instruction constraints. + + See also :inst:`regmove`. + """, + ins=(x, src, SS), + other_side_effects=True) + + +regfill = Instruction( + 'regfill', r""" + Temporarily divert ``x`` from ``SS`` to ``dst``. + + This instruction moves the location of a value from a stack slot to a + register without creating a new SSA value. It is used by the register + allocator to temporarily rearrange register assignments in order to + satisfy instruction constraints. + + See also :inst:`regmove`. + """, + ins=(x, SS, dst), + other_side_effects=True) # # Vector operations # diff --git a/lib/cretonne/meta/gen_binemit.py b/lib/cretonne/meta/gen_binemit.py index 58a44473a3..9e2bac451f 100644 --- a/lib/cretonne/meta/gen_binemit.py +++ b/lib/cretonne/meta/gen_binemit.py @@ -32,7 +32,7 @@ def gen_recipe(recipe, fmt): for o in recipe.outs) # Regmove instructions get special treatment. - is_regmove = (recipe.format.name == 'RegMove') + is_regmove = (recipe.format.name in ('RegMove', 'RegSpill', 'RegFill')) # First unpack the instruction. with fmt.indented( @@ -111,6 +111,10 @@ def gen_recipe(recipe, fmt): # diversion tracker. if recipe.format.name == 'RegMove': fmt.line('divert.regmove(arg, src, dst);') + elif recipe.format.name == 'RegSpill': + fmt.line('divert.regspill(arg, src, dst);') + elif recipe.format.name == 'RegFill': + fmt.line('divert.regfill(arg, src, dst);') # Call hand-written code. If the recipe contains a code snippet, use # that. Otherwise cal a recipe function in the target ISA's binemit diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 14bd3de726..2174bb4c99 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -213,6 +213,18 @@ pub enum InstructionData { src: RegUnit, dst: RegUnit, }, + RegSpill { + opcode: Opcode, + arg: Value, + src: RegUnit, + dst: StackSlot, + }, + RegFill { + opcode: Opcode, + arg: Value, + src: StackSlot, + dst: RegUnit, + }, Trap { opcode: Opcode, code: ir::TrapCode }, CondTrap { opcode: Opcode, diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index d76a5efbba..735f78a754 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -317,6 +317,12 @@ impl<'a> Verifier<'a> { HeapAddr { heap, .. } => { self.verify_heap(inst, heap)?; } + RegSpill { dst, .. } => { + self.verify_stack_slot(inst, dst)?; + } + RegFill { src, .. } => { + self.verify_stack_slot(inst, src)?; + } // Exhaustive list so we can't forget to add new formats Unary { .. } | diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index d2f903cc7a..a1e195cd17 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -385,6 +385,22 @@ pub fn write_operands( write!(w, " {}, %{} -> %{}", arg, src, dst) } } + RegSpill { arg, src, dst, .. } => { + if let Some(isa) = isa { + let regs = isa.register_info(); + write!(w, " {}, {} -> {}", arg, regs.display_regunit(src), dst) + } else { + write!(w, " {}, %{} -> {}", arg, src, dst) + } + } + RegFill { arg, src, dst, .. } => { + if let Some(isa) = isa { + let regs = isa.register_info(); + write!(w, " {}, {} -> {}", arg, src, regs.display_regunit(dst)) + } else { + write!(w, " {}, {} -> %{}", arg, src, dst) + } + } Trap { code, .. } => write!(w, " {}", code), CondTrap { arg, code, .. } => write!(w, " {}, {}", arg, code), } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index e334319e91..354cb02a22 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -2223,6 +2223,46 @@ impl<'a> Parser<'a> { dst, } } + InstructionFormat::RegSpill => { + let arg = self.match_value("expected SSA value operand")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; + let src = self.match_regunit(ctx.unique_isa)?; + self.match_token( + Token::Arrow, + "expected '->' before destination stack slot", + )?; + let dst = self.match_ss("expected stack slot number: ss«n»") + .and_then(|num| ctx.get_ss(num, &self.loc))?; + InstructionData::RegSpill { + opcode, + arg, + src, + dst, + } + } + InstructionFormat::RegFill => { + let arg = self.match_value("expected SSA value operand")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; + let src = self.match_ss("expected stack slot number: ss«n»") + .and_then(|num| ctx.get_ss(num, &self.loc))?; + self.match_token( + Token::Arrow, + "expected '->' before destination register units", + )?; + let dst = self.match_regunit(ctx.unique_isa)?; + InstructionData::RegFill { + opcode, + arg, + src, + dst, + } + } InstructionFormat::Trap => { let code = self.match_enum("expected trap code")?; InstructionData::Trap { opcode, code } From 826d4062fb9044d2f2a82c7ac56a2a2531ec3a0d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 4 Oct 2017 16:33:50 -0700 Subject: [PATCH 1217/3084] Apply register diversions during binemit tests. When "binemit" tests encode instructions, keep track of the current set of register diversions, and use the diverted locations to check operand constraints. This matches how constraints are applied during a real binemit phase. --- cranelift/src/filetest/binemit.rs | 6 ++++-- lib/cretonne/src/isa/constraints.rs | 7 ++++--- lib/cretonne/src/regalloc/diversion.rs | 29 ++++++++++++++++++++++++++ 3 files changed, 37 insertions(+), 5 deletions(-) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index f1a493c749..da0e2012a0 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -114,6 +114,7 @@ impl SubTest for TestBinEmit { let is_compressed = isa.flags().is_compressed(); // Give an encoding to any instruction that doesn't already have one. + let mut divert = RegDiversions::new(); for ebb in func.layout.ebbs() { for inst in func.layout.ebb_insts(ebb) { if !func.encodings[inst].is_legal() { @@ -128,7 +129,7 @@ impl SubTest for TestBinEmit { legal_encodings .filter(|e| { let recipe_constraints = &encinfo.constraints[e.recipe()]; - recipe_constraints.satisfied(inst, &func) + recipe_constraints.satisfied(inst, &divert, &func) }) .min_by_key(|&e| encinfo.bytes(e)) } else { @@ -140,6 +141,7 @@ impl SubTest for TestBinEmit { func.encodings[inst] = enc; } } + divert.apply(&func.dfg[inst]); } } @@ -178,8 +180,8 @@ impl SubTest for TestBinEmit { } // Now emit all instructions. + divert.clear(); let mut sink = TextSink::new(isa); - let mut divert = RegDiversions::new(); for ebb in func.layout.ebbs() { divert.clear(); // Correct header offsets should have been computed by `relax_branches()`. diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index d9fc3aa8ea..9efcf7bb4f 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -10,6 +10,7 @@ use binemit::CodeOffset; use isa::{RegClass, RegUnit}; use ir::{Function, ValueLoc, Inst}; +use regalloc::RegDiversions; /// Register constraint for a single value operand or instruction result. pub struct OperandConstraint { @@ -109,9 +110,9 @@ pub struct RecipeConstraints { impl RecipeConstraints { /// Check that these constraints are satisfied by the operands on `inst`. - pub fn satisfied(&self, inst: Inst, func: &Function) -> bool { + pub fn satisfied(&self, inst: Inst, divert: &RegDiversions, func: &Function) -> bool { for (&arg, constraint) in func.dfg.inst_args(inst).iter().zip(self.ins) { - let loc = func.locations[arg]; + let loc = divert.get(arg, &func.locations); if let ConstraintKind::Tied(out_index) = constraint.kind { let out_val = func.dfg.inst_results(inst)[out_index as usize]; @@ -127,7 +128,7 @@ impl RecipeConstraints { } for (&arg, constraint) in func.dfg.inst_results(inst).iter().zip(self.outs) { - let loc = func.locations[arg]; + let loc = divert.get(arg, &func.locations); if !constraint.satisfied(loc) { return false; } diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index 9d241ebcdd..c6d25f543e 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -8,6 +8,7 @@ //! EBB. use ir::{Value, ValueLoc, ValueLocations, StackSlot}; +use ir::{InstructionData, Opcode}; use isa::{RegUnit, RegInfo}; use std::fmt; @@ -118,6 +119,34 @@ impl RegDiversions { self.divert(value, ValueLoc::Stack(from), ValueLoc::Reg(to)); } + /// Apply the effect of `inst`. + /// + /// If `inst` is a `regmove`, `regfill`, or `regspill` instruction, update the diversions to + /// match. + pub fn apply(&mut self, inst: &InstructionData) { + match *inst { + InstructionData::RegMove { + opcode: Opcode::Regmove, + arg, + src, + dst, + } => self.regmove(arg, src, dst), + InstructionData::RegSpill { + opcode: Opcode::Regspill, + arg, + src, + dst, + } => self.regspill(arg, src, dst), + InstructionData::RegFill { + opcode: Opcode::Regfill, + arg, + src, + dst, + } => self.regfill(arg, src, dst), + _ => {} + } + } + /// Drop any recorded move for `value`. /// /// Returns the `to` location of the removed diversion. From 73d4bb47c0542859dea8434a12690002c99a03ef Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 4 Oct 2017 17:01:40 -0700 Subject: [PATCH 1218/3084] Intel encodings for regspill and regfill. These are always SP-based. --- .../filetests/isa/intel/binary32-float.cton | 10 +++ cranelift/filetests/isa/intel/binary32.cton | 5 ++ .../filetests/isa/intel/binary64-float.cton | 10 +++ cranelift/filetests/isa/intel/binary64.cton | 10 +++ lib/cretonne/meta/gen_binemit.py | 78 +++++++++---------- lib/cretonne/meta/isa/intel/encodings.py | 6 ++ lib/cretonne/meta/isa/intel/recipes.py | 47 ++++++++++- lib/cretonne/src/isa/stack.rs | 27 ++++--- 8 files changed, 138 insertions(+), 55 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index a880ed9a8a..6ac14039de 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -185,6 +185,11 @@ ebb0: ; asm: movd 1032(%esp), %xmm2 [-,%xmm2] v211 = fill v201 ; bin: 66 0f 6e 94 24 00000408 + ; asm: movd %xmm5, 1032(%rsp) + regspill v100, %xmm5 -> ss1 ; bin: 66 0f 7e ac 24 00000408 + ; asm: movd 1032(%rsp), %xmm5 + regfill v100, ss1 -> %xmm5 ; bin: 66 0f 6e ac 24 00000408 + ; Comparisons. ; ; Only `supported_floatccs` are tested here. Others are handled by @@ -388,6 +393,11 @@ ebb0: ; asm: movq 1032(%esp), %xmm2 [-,%xmm2] v211 = fill v201 ; bin: f3 0f 7e 94 24 00000408 + ; asm: movq %xmm5, 1032(%rsp) + regspill v100, %xmm5 -> ss1 ; bin: 66 0f d6 ac 24 00000408 + ; asm: movq 1032(%rsp), %xmm5 + regfill v100, ss1 -> %xmm5 ; bin: f3 0f 7e ac 24 00000408 + ; Comparisons. ; ; Only `supported_floatccs` are tested here. Others are handled by diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index ee8c79e24f..e777cfb359 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -363,6 +363,11 @@ ebb0: ; asm: movl 1032(%esp), %esi [-,%rsi] v511 = fill v501 ; bin: 8b b4 24 00000408 + ; asm: movl %ecx, 1032(%esp) + regspill v1, %rcx -> ss1 ; bin: 89 8c 24 00000408 + ; asm: movl 1032(%esp), %ecx + regfill v1, ss1 -> %rcx ; bin: 8b 8c 24 00000408 + ; asm: testl %ecx, %ecx ; asm: je ebb1 brz v1, ebb1 ; bin: 85 c9 74 0e diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index 18a24f9e7e..4ef385bbb2 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -194,6 +194,11 @@ ebb0: ; asm: movd 1032(%rsp), %xmm10 [-,%xmm10] v211 = fill v201 ; bin: 66 44 0f 6e 94 24 00000408 + ; asm: movd %xmm5, 1032(%rsp) + regspill v100, %xmm5 -> ss1 ; bin: 66 0f 7e ac 24 00000408 + ; asm: movd 1032(%rsp), %xmm5 + regfill v100, ss1 -> %xmm5 ; bin: 66 0f 6e ac 24 00000408 + ; Comparisons. ; ; Only `supported_floatccs` are tested here. Others are handled by @@ -412,6 +417,11 @@ ebb0: ; asm: movq 1032(%rsp), %xmm10 [-,%xmm10] v211 = fill v201 ; bin: f3 44 0f 7e 94 24 00000408 + ; asm: movq %xmm5, 1032(%rsp) + regspill v100, %xmm5 -> ss1 ; bin: 66 0f d6 ac 24 00000408 + ; asm: movq 1032(%rsp), %xmm5 + regfill v100, ss1 -> %xmm5 ; bin: f3 0f 7e ac 24 00000408 + ; Comparisons. ; ; Only `supported_floatccs` are tested here. Others are handled by diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index c9febfdab0..03cb54e467 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -459,6 +459,11 @@ ebb0: ; asm: movq 1032(%rsp), %r10 [-,%r10] v512 = fill v502 ; bin: 4c 8b 94 24 00000408 + ; asm: movq %rcx, 1032(%rsp) + regspill v1, %rcx -> ss1 ; bin: 48 89 8c 24 00000408 + ; asm: movq 1032(%rsp), %rcx + regfill v1, ss1 -> %rcx ; bin: 48 8b 8c 24 00000408 + ; asm: testq %rcx, %rcx ; asm: je ebb1 brz v1, ebb1 ; bin: 48 85 c9 74 1b @@ -850,6 +855,11 @@ ebb0: ; asm: movl 1032(%rsp), %r10d [-,%r10] v512 = fill v502 ; bin: 44 8b 94 24 00000408 + ; asm: movl %ecx, 1032(%rsp) + regspill v1, %rcx -> ss1 ; bin: 89 8c 24 00000408 + ; asm: movl 1032(%rsp), %ecx + regfill v1, ss1 -> %rcx ; bin: 8b 8c 24 00000408 + ; asm: testl %ecx, %ecx ; asm: je ebb1x brz v1, ebb1 ; bin: 85 c9 74 18 diff --git a/lib/cretonne/meta/gen_binemit.py b/lib/cretonne/meta/gen_binemit.py index 9e2bac451f..8c591f0513 100644 --- a/lib/cretonne/meta/gen_binemit.py +++ b/lib/cretonne/meta/gen_binemit.py @@ -8,7 +8,7 @@ import srcgen try: from typing import Sequence, List # noqa - from cdsl.isa import TargetISA, EncRecipe # noqa + from cdsl.isa import TargetISA, EncRecipe, OperandConstraint # noqa except ImportError: pass @@ -49,36 +49,17 @@ def gen_recipe(recipe, fmt): fmt.line('..') fmt.outdented_line('} = func.dfg[inst] {') + # Pass recipe arguments in this order: inputs, imm_fields, outputs. + args = '' + # Normalize to an `args` array. if want_args and not is_regmove: if iform.has_value_list: fmt.line('let args = args.as_slice(&func.dfg.value_lists);') elif nvops == 1: fmt.line('let args = [arg];') + args += unwrap_values(recipe.ins, 'in', 'args', fmt) - # Unwrap interesting input arguments. - # Don't bother with fixed registers. - args = '' - for i, arg in enumerate(recipe.ins): - if isinstance(arg, RegClass) and not is_regmove: - v = 'in_reg{}'.format(i) - args += ', ' + v - fmt.line( - 'let {} = divert.reg(args[{}], &func.locations);' - .format(v, i)) - elif isinstance(arg, Stack): - v = 'in_stk{}'.format(i) - args += ', ' + v - with fmt.indented( - 'let {} = StackRef::masked('.format(v), - ').unwrap();'): - fmt.format( - 'func.locations[args[{}]].unwrap_stack(),', - i) - fmt.format('{},', arg.stack_base_mask()) - fmt.line('&func.stack_slots,') - - # Pass arguments in this order: inputs, imm_fields, outputs. for f in iform.imm_fields: args += ', ' + f.member @@ -88,24 +69,7 @@ def gen_recipe(recipe, fmt): fmt.line('let results = [func.dfg.first_result(inst)];') else: fmt.line('let results = func.dfg.inst_results(inst);') - for i, res in enumerate(recipe.outs): - if isinstance(res, RegClass): - v = 'out_reg{}'.format(i) - args += ', ' + v - fmt.format( - 'let {} = func.locations[results[{}]].unwrap_reg();', - v, i) - elif isinstance(res, Stack): - v = 'out_stk{}'.format(i) - args += ', ' + v - with fmt.indented( - 'let {} = StackRef::masked('.format(v), - ').unwrap();'): - fmt.format( - 'func.locations[results[{}]].unwrap_stack(),', - i) - fmt.format('{},', res.stack_base_mask()) - fmt.line('&func.stack_slots,') + args += unwrap_values(recipe.outs, 'out', 'results', fmt) # Special handling for regmove instructions. Update the register # diversion tracker. @@ -128,6 +92,36 @@ def gen_recipe(recipe, fmt): fmt.line('return;') +def unwrap_values(args, prefix, values, fmt): + # type: (Sequence[OperandConstraint], str, str, srcgen.Formatter) -> str # noqa + """ + Emit code that unwraps values living in registers or stack slots. + + :param args: Input or output constraints. + :param prefix: Prefix to be used for the generated local variables. + :param values: Name of slice containing the values to be unwrapped. + :returns: Comma separated list of the generated variables + """ + varlist = '' + for i, cst in enumerate(args): + if isinstance(cst, RegClass): + v = '{}_reg{}'.format(prefix, i) + varlist += ', ' + v + fmt.format( + 'let {} = divert.reg({}[{}], &func.locations);', + v, values, i) + elif isinstance(cst, Stack): + v = '{}_stk{}'.format(prefix, i) + varlist += ', ' + v + with fmt.indented( + 'let {} = StackRef::masked('.format(v), + ').unwrap();'): + fmt.format('divert.stack({}[{}], &func.locations),', values, i) + fmt.format('{},', cst.stack_base_mask()) + fmt.line('&func.stack_slots,') + return varlist + + def gen_isa(isa, fmt): # type: (TargetISA, srcgen.Formatter) -> None """ diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index db5c9db4ce..6d79ca726e 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -193,6 +193,7 @@ enc_i32_i64_ld_st(base.store, True, r.stDisp8, 0x89) enc_i32_i64_ld_st(base.store, True, r.stDisp32, 0x89) enc_i32_i64(base.spill, r.spSib32, 0x89) +enc_i32_i64(base.regspill, r.rsp32, 0x89) enc_i64(base.istore32.i64.any, r.st, 0x89) enc_i64(base.istore32.i64.any, r.stDisp8, 0x89) @@ -222,6 +223,7 @@ enc_i32_i64_ld_st(base.load, True, r.ldDisp8, 0x8b) enc_i32_i64_ld_st(base.load, True, r.ldDisp32, 0x8b) enc_i32_i64(base.fill, r.fiSib32, 0x8b) +enc_i32_i64(base.regfill, r.rfi32, 0x8b) enc_i64(base.uload32.i64, r.ld, 0x8b) enc_i64(base.uload32.i64, r.ldDisp8, 0x8b) @@ -268,10 +270,14 @@ enc_both(base.store.f64.any, r.fstDisp8, 0x66, 0x0f, 0xd6) enc_both(base.store.f64.any, r.fstDisp32, 0x66, 0x0f, 0xd6) enc_both(base.fill.f32, r.ffiSib32, 0x66, 0x0f, 0x6e) +enc_both(base.regfill.f32, r.frfi32, 0x66, 0x0f, 0x6e) enc_both(base.fill.f64, r.ffiSib32, 0xf3, 0x0f, 0x7e) +enc_both(base.regfill.f64, r.frfi32, 0xf3, 0x0f, 0x7e) enc_both(base.spill.f32, r.fspSib32, 0x66, 0x0f, 0x7e) +enc_both(base.regspill.f32, r.frsp32, 0x66, 0x0f, 0x7e) enc_both(base.spill.f64, r.fspSib32, 0x66, 0x0f, 0xd6) +enc_both(base.regspill.f64, r.frsp32, 0x66, 0x0f, 0xd6) # # Function addresses. diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index fb5104381f..9c4713500a 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -8,7 +8,8 @@ from cdsl.registers import RegClass from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare, FloatCompare -from base.formats import RegMove, Ternary, Jump, Branch, FuncAddr +from base.formats import Ternary, Jump, Branch, FuncAddr +from base.formats import RegMove, RegSpill, RegFill from .registers import GPR, ABCD, FPR, GPR8, FPR8, StackGPR32, StackFPR32 from .defs import supported_floatccs @@ -570,6 +571,28 @@ fspSib32 = TailRecipe( sink.put4(out_stk0.offset as u32); ''') +# Regspill using RSP-relative addressing. +rsp32 = TailRecipe( + 'rsp32', RegSpill, size=6, ins=GPR, outs=(), + emit=''' + let dst = StackRef::sp(dst, &func.stack_slots); + let base = stk_base(dst.base); + PUT_OP(bits, rex2(base, src), sink); + modrm_sib_disp32(src, sink); + sib_noindex(base, sink); + sink.put4(dst.offset as u32); + ''') +frsp32 = TailRecipe( + 'frsp32', RegSpill, size=6, ins=FPR, outs=(), + emit=''' + let dst = StackRef::sp(dst, &func.stack_slots); + let base = stk_base(dst.base); + PUT_OP(bits, rex2(base, src), sink); + modrm_sib_disp32(src, sink); + sib_noindex(base, sink); + sink.put4(dst.offset as u32); + ''') + # # Load recipes # @@ -656,6 +679,28 @@ ffiSib32 = TailRecipe( sink.put4(in_stk0.offset as u32); ''') +# Regfill with RSP-relative 32-bit displacement. +rfi32 = TailRecipe( + 'rfi32', RegFill, size=6, ins=StackGPR32, outs=(), + emit=''' + let src = StackRef::sp(src, &func.stack_slots); + let base = stk_base(src.base); + PUT_OP(bits, rex2(base, dst), sink); + modrm_sib_disp32(dst, sink); + sib_noindex(base, sink); + sink.put4(src.offset as u32); + ''') +frfi32 = TailRecipe( + 'frfi32', RegFill, size=6, ins=StackFPR32, outs=(), + emit=''' + let src = StackRef::sp(src, &func.stack_slots); + let base = stk_base(src.base); + PUT_OP(bits, rex2(base, dst), sink); + modrm_sib_disp32(dst, sink); + sib_noindex(base, sink); + sink.put4(src.offset as u32); + ''') + # # Call/return # diff --git a/lib/cretonne/src/isa/stack.rs b/lib/cretonne/src/isa/stack.rs index cfb054dd66..ea7ee37248 100644 --- a/lib/cretonne/src/isa/stack.rs +++ b/lib/cretonne/src/isa/stack.rs @@ -24,25 +24,28 @@ pub struct StackRef { impl StackRef { /// Get a reference to the stack slot `ss` using one of the base pointers in `mask`. pub fn masked(ss: StackSlot, mask: StackBaseMask, frame: &StackSlots) -> Option { - let size = frame.frame_size.expect( - "Stack layout must be computed before referencing stack slots", - ); - // Offsets relative to the caller's stack pointer. - let offset = frame[ss].offset; - // Try an SP-relative reference. if mask.contains(StackBase::SP) { - // Offset where SP is pointing. (All ISAs have stacks growing downwards.) - let sp_offset = -(size as StackOffset); - return Some(StackRef { - base: StackBase::SP, - offset: offset - sp_offset, - }); + return Some(StackRef::sp(ss, frame)); } // No reference possible with this mask. None } + + /// Get a reference to `ss` using the stack pointer as a base. + pub fn sp(ss: StackSlot, frame: &StackSlots) -> StackRef { + let size = frame.frame_size.expect( + "Stack layout must be computed before referencing stack slots", + ); + + // Offset where SP is pointing. (All ISAs have stacks growing downwards.) + let sp_offset = -(size as StackOffset); + return StackRef { + base: StackBase::SP, + offset: frame[ss].offset - sp_offset, + }; + } } /// Generic base register for referencing stack slots. From e4ef2cbf22155a6955fe9b6d17f4d1188f38daa0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 5 Oct 2017 10:05:53 -0700 Subject: [PATCH 1219/3084] Support ISA-specific settings in the `--isa` command-line option. --- cranelift/src/utils.rs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index c9cae4cd58..b243b8953b 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -104,10 +104,14 @@ pub fn parse_sets_and_isa( let mut words = flag_isa.trim().split_whitespace(); // Look for `isa foo`. if let Some(isa_name) = words.next() { - let isa_builder = isa::lookup(isa_name).map_err(|err| match err { + let mut isa_builder = isa::lookup(isa_name).map_err(|err| match err { isa::LookupError::Unknown => format!("unknown ISA '{}'", isa_name), isa::LookupError::Unsupported => format!("support for ISA '{}' not enabled", isa_name), })?; + // Apply the ISA-specific settings to `isa_builder`. + parse_options(words, &mut isa_builder, &Location { line_number: 0 }) + .map_err(|err| err.to_string())?; + Ok(OwnedFlagsOrIsa::Isa( isa_builder.finish(settings::Flags::new(&flag_builder)), )) From 30aeb5708342de0e913699eba3281e7eb4b7639d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 5 Oct 2017 12:42:19 -0700 Subject: [PATCH 1220/3084] Add a value location verifier. This is a verification pass that can be run after register allocation. It verifies that value locations are consistent with constraints on their uses, and that the register diversions are consistent. Make it clear that register diversions are local to an EBB only. This affects what branch relaxation is allowed to do. The verify_locations() takes an optional Liveness parameter which is used to check that no diverted values are live across CFG edges. --- cranelift/src/filetest/binemit.rs | 2 +- lib/cretonne/src/binemit/relaxation.rs | 21 +- lib/cretonne/src/regalloc/coloring.rs | 2 +- lib/cretonne/src/regalloc/context.rs | 3 +- lib/cretonne/src/regalloc/liverange.rs | 7 + lib/cretonne/src/verifier/locations.rs | 316 +++++++++++++++++++++++++ lib/cretonne/src/verifier/mod.rs | 4 +- 7 files changed, 350 insertions(+), 5 deletions(-) create mode 100644 lib/cretonne/src/verifier/locations.rs diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index da0e2012a0..ae1c741d13 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -116,6 +116,7 @@ impl SubTest for TestBinEmit { // Give an encoding to any instruction that doesn't already have one. let mut divert = RegDiversions::new(); for ebb in func.layout.ebbs() { + divert.clear(); for inst in func.layout.ebb_insts(ebb) { if !func.encodings[inst].is_legal() { let mut legal_encodings = isa.legal_encodings( @@ -180,7 +181,6 @@ impl SubTest for TestBinEmit { } // Now emit all instructions. - divert.clear(); let mut sink = TextSink::new(isa); for ebb in func.layout.ebbs() { divert.clear(); diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index 4e6d1a2408..94835b736e 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -166,5 +166,24 @@ fn relax_branch( return encinfo.bytes(enc); } - unimplemented!(); + // Note: On some RISC ISAs, conditional branches have shorter range than unconditional + // branches, so one way of extending the range of a conditional branch is to invert its + // condition and make it branch over an unconditional jump which has the larger range. + // + // Splitting the EBB is problematic this late because there may be register diversions in + // effect across the conditional branch, and they can't survive the control flow edge to a new + // EBB. We have two options for handling that: + // + // 1. Set a flag on the new EBB that indicates it wants the preserve the register diversions of + // its layout predecessor, or + // 2. Use an encoding macro for the branch-over-jump pattern so we don't need to split the EBB. + // + // It seems that 1. would allow us to share code among RISC ISAs that need this. + // + // We can't allow register diversions to survive from the layout predecessor because the layout + // predecessor could contain kill points for some values that are live in this EBB, and + // diversions are not automatically cancelled when the live range of a value ends. + + // This assumes solution 2. above: + panic!("No branch in range for {:#x}-{:#x}", offset, dest_offset); } diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 9a92df0270..69e1086416 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -468,7 +468,7 @@ impl<'a> Context<'a> { // // Values with a global live range that are not live in to `dest` could appear as branch // arguments, so they can't always be un-diverted. - self.undivert_regs(|lr, func| lr.livein_local_end(dest, &func.layout).is_some()); + self.undivert_regs(|lr, func| lr.is_livein(dest, &func.layout)); // Now handle the EBB arguments. let br_args = self.cur.func.dfg.inst_variable_args(inst); diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 5e5c17c113..06d27ff325 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -17,7 +17,7 @@ use regalloc::spilling::Spilling; use regalloc::virtregs::VirtRegs; use result::CtonResult; use topo_order::TopoOrder; -use verifier::{verify_context, verify_liveness, verify_cssa}; +use verifier::{verify_context, verify_liveness, verify_cssa, verify_locations}; /// Persistent memory allocations for register allocation. pub struct Context { @@ -138,6 +138,7 @@ impl Context { if isa.flags().enable_verifier() { verify_context(func, cfg, domtree, isa)?; verify_liveness(isa, func, cfg, &self.liveness)?; + verify_locations(isa, func, Some(&self.liveness))?; verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?; } Ok(()) diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 3da125ef3a..7b94fce280 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -383,6 +383,13 @@ impl LiveRange { }) } + /// Is this value live-in to `ebb`? + /// + /// An EBB argument is not considered to be live in. + pub fn is_livein(&self, ebb: Ebb, order: &PO) -> bool { + self.livein_local_end(ebb, order).is_some() + } + /// Get all the live-in intervals. pub fn liveins(&self) -> &[Interval] { &self.liveins diff --git a/lib/cretonne/src/verifier/locations.rs b/lib/cretonne/src/verifier/locations.rs new file mode 100644 index 0000000000..5f84f0403d --- /dev/null +++ b/lib/cretonne/src/verifier/locations.rs @@ -0,0 +1,316 @@ +//! Verify value locations. + +use ir; +use isa; +use regalloc::RegDiversions; +use regalloc::liveness::Liveness; +use verifier::Result; + +/// Verify value locations for `func`. +/// +/// After register allocation, every value must be assigned to a location - either a register or a +/// stack slot. These locations must be compatible with the constraints described by the +/// instruction encoding recipes. +/// +/// Values can be temporarily diverted to a different location by using the `regmove`, `regspill`, +/// and `regfill` instructions, but only inside an EBB. +/// +/// If a liveness analysis is provided, it is used to verify that there are no active register +/// diversions across control flow edges. +pub fn verify_locations( + isa: &isa::TargetIsa, + func: &ir::Function, + liveness: Option<&Liveness>, +) -> Result { + let verifier = LocationVerifier { + isa, + func, + reginfo: isa.register_info(), + encinfo: isa.encoding_info(), + liveness, + }; + verifier.check_constraints()?; + Ok(()) +} + +struct LocationVerifier<'a> { + isa: &'a isa::TargetIsa, + func: &'a ir::Function, + reginfo: isa::RegInfo, + encinfo: isa::EncInfo, + liveness: Option<&'a Liveness>, +} + +impl<'a> LocationVerifier<'a> { + /// Check that the assigned value locations match the operand constraints of their uses. + fn check_constraints(&self) -> Result { + let dfg = &self.func.dfg; + let mut divert = RegDiversions::new(); + + for ebb in self.func.layout.ebbs() { + // Diversions are reset at the top of each EBB. No diversions can exist across control + // flow edges. + divert.clear(); + for inst in self.func.layout.ebb_insts(ebb) { + let enc = self.func.encodings[inst]; + + if enc.is_legal() { + self.check_enc_constraints(inst, enc, &divert)? + } else { + self.check_ghost_results(inst)?; + } + + if let Some(sig) = dfg.call_signature(inst) { + self.check_call_abi(inst, sig, &divert)?; + } + + let opcode = dfg[inst].opcode(); + if opcode.is_return() { + self.check_return_abi(inst, &divert)?; + } + + if opcode.is_branch() && !divert.is_empty() { + self.check_cfg_edges(inst, &divert)?; + } + + self.update_diversions(inst, &mut divert)?; + } + } + + Ok(()) + } + + /// Check encoding constraints against the current value locations. + fn check_enc_constraints( + &self, + inst: ir::Inst, + enc: isa::Encoding, + divert: &RegDiversions, + ) -> Result { + let constraints = self.encinfo.operand_constraints(enc).expect( + "check_enc_constraints requires a legal encoding", + ); + + if constraints.satisfied(inst, divert, self.func) { + return Ok(()); + } + + // TODO: We could give a better error message here. + err!( + inst, + "{} constraints not satisfied", + self.encinfo.display(enc) + ) + } + + /// Check that the result values produced by a ghost instruction are not assigned a value + /// location. + fn check_ghost_results(&self, inst: ir::Inst) -> Result { + let results = self.func.dfg.inst_results(inst); + + for &res in results { + let loc = self.func.locations[res]; + if loc.is_assigned() { + return err!( + inst, + "ghost result {} value must not have a location ({}).", + res, + loc.display(&self.reginfo) + ); + } + } + + Ok(()) + } + + /// Check the ABI argument and result locations for a call. + fn check_call_abi(&self, inst: ir::Inst, sig: ir::SigRef, divert: &RegDiversions) -> Result { + let sig = &self.func.dfg.signatures[sig]; + let varargs = self.func.dfg.inst_variable_args(inst); + let results = self.func.dfg.inst_results(inst); + + for (abi, &value) in sig.argument_types.iter().zip(varargs) { + self.check_abi_location( + inst, + value, + abi, + divert.get(value, &self.func.locations), + ir::StackSlotKind::OutgoingArg, + )?; + } + + for (abi, &value) in sig.return_types.iter().zip(results) { + self.check_abi_location( + inst, + value, + abi, + self.func.locations[value], + ir::StackSlotKind::OutgoingArg, + )?; + } + + Ok(()) + } + + /// Check the ABI argument locations for a return. + fn check_return_abi(&self, inst: ir::Inst, divert: &RegDiversions) -> Result { + let sig = &self.func.signature; + let varargs = self.func.dfg.inst_variable_args(inst); + + for (abi, &value) in sig.return_types.iter().zip(varargs) { + self.check_abi_location( + inst, + value, + abi, + divert.get(value, &self.func.locations), + ir::StackSlotKind::IncomingArg, + )?; + } + + Ok(()) + } + + /// Check a single ABI location. + fn check_abi_location( + &self, + inst: ir::Inst, + value: ir::Value, + abi: &ir::ArgumentType, + loc: ir::ValueLoc, + want_kind: ir::StackSlotKind, + ) -> Result { + match abi.location { + ir::ArgumentLoc::Unassigned => {} + ir::ArgumentLoc::Reg(reg) => { + if loc != ir::ValueLoc::Reg(reg) { + return err!( + inst, + "ABI expects {} in {}, got {}", + value, + abi.location.display(&self.reginfo), + loc.display(&self.reginfo) + ); + } + } + ir::ArgumentLoc::Stack(offset) => { + if let ir::ValueLoc::Stack(ss) = loc { + let slot = &self.func.stack_slots[ss]; + if slot.kind != want_kind { + return err!( + inst, + "call argument {} should be in a {} slot, but {} is {}", + value, + want_kind, + ss, + slot.kind + ); + } + if slot.offset != offset { + return err!( + inst, + "ABI expects {} at stack offset {}, but {} is at {}", + value, + offset, + ss, + slot.offset + ); + } + } else { + return err!( + inst, + "ABI expects {} at stack offset {}, got {}", + value, + offset, + loc.display(&self.reginfo) + ); + } + } + } + + Ok(()) + } + + /// Update diversions to reflect the current instruction and check their consistency. + fn update_diversions(&self, inst: ir::Inst, divert: &mut RegDiversions) -> Result { + let (arg, src) = match self.func.dfg[inst] { + ir::InstructionData::RegMove { arg, src, .. } => (arg, ir::ValueLoc::Reg(src)), + ir::InstructionData::RegSpill { arg, src, .. } => (arg, ir::ValueLoc::Reg(src)), + ir::InstructionData::RegFill { arg, src, .. } => (arg, ir::ValueLoc::Stack(src)), + _ => return Ok(()), + }; + + if let Some(d) = divert.diversion(arg) { + if d.to != src { + return err!( + inst, + "inconsistent with current diversion to {}", + d.to.display(&self.reginfo) + ); + } + } else if self.func.locations[arg] != src { + return err!( + inst, + "inconsistent with global location {}", + self.func.locations[arg].display(&self.reginfo) + ); + } + + divert.apply(&self.func.dfg[inst]); + + Ok(()) + } + + /// We have active diversions before a branch. Make sure none of the diverted values are live + /// on the outgoing CFG edges. + fn check_cfg_edges(&self, inst: ir::Inst, divert: &RegDiversions) -> Result { + use ir::instructions::BranchInfo::*; + + // We can only check CFG edges if we have a liveness analysis. + let liveness = match self.liveness { + Some(l) => l, + None => return Ok(()), + }; + let dfg = &self.func.dfg; + + match dfg[inst].analyze_branch(&dfg.value_lists) { + NotABranch => { + panic!( + "No branch information for {}", + dfg.display_inst(inst, self.isa) + ) + } + SingleDest(ebb, _) => { + for d in divert.all() { + let lr = &liveness[d.value]; + if lr.is_livein(ebb, &self.func.layout) { + return err!( + inst, + "{} is diverted to {} and live in to {}", + d.value, + d.to.display(&self.reginfo), + ebb + ); + } + } + } + Table(jt) => { + for d in divert.all() { + let lr = &liveness[d.value]; + for (_, ebb) in self.func.jump_tables[jt].entries() { + if lr.is_livein(ebb, &self.func.layout) { + return err!( + inst, + "{} is diverted to {} and live in to {}", + d.value, + d.to.display(&self.reginfo), + ebb + ); + } + } + } + } + } + + Ok(()) + } +} diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 735f78a754..196054bec3 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -72,8 +72,9 @@ use std::collections::BTreeSet; use std::cmp::Ordering; use iterators::IteratorExtras; -pub use self::liveness::verify_liveness; pub use self::cssa::verify_cssa; +pub use self::liveness::verify_liveness; +pub use self::locations::verify_locations; // Create an `Err` variant of `Result` from a location and `format!` arguments. macro_rules! err { @@ -94,6 +95,7 @@ macro_rules! err { mod cssa; mod liveness; +mod locations; /// A verifier error. #[derive(Debug, PartialEq, Eq)] From b562fdcd5cb637c8c42aae7ba048b248cb3318d9 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 5 Oct 2017 14:46:34 -0700 Subject: [PATCH 1221/3084] Remove the dfg::resolve_copies() method. This method was important back when result values couldn't be moved between instructions. Now that results can be moved, value aliases do everything we need. Copy instructions are still used to break interferences in the register allocator's coalescing phase, but there isn't really any reason to use a copy instruction over a value alias anywhere else. After and during register allocation, copy instructions are significant, so we never want to "see through" them like the resolve_copies() function did. This is related to #166, but probably doesn't fix the underlying problem. --- lib/cretonne/src/ir/dfg.rs | 31 +---------------------------- lib/cretonne/src/legalizer/split.rs | 4 ++-- 2 files changed, 3 insertions(+), 32 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 25859535bc..dbb643e7f3 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -4,7 +4,7 @@ use entity::{PrimaryMap, EntityMap}; use isa::TargetIsa; use ir::builder::{InsertBuilder, ReplaceBuilder}; use ir::extfunc::ExtFuncData; -use ir::instructions::{Opcode, InstructionData, CallInfo}; +use ir::instructions::{InstructionData, CallInfo}; use ir::layout::{Cursor, LayoutCursorInserter}; use ir::types; use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueList, ValueListPool}; @@ -211,33 +211,6 @@ impl DataFlowGraph { resolve_aliases(&self.values, value) } - /// Resolve value copies. - /// - /// Find the original definition of a value, looking through value aliases as well as - /// copy/spill/fill instructions. - pub fn resolve_copies(&self, value: Value) -> Value { - let mut v = value; - - for _ in 0..self.insts.len() { - v = self.resolve_aliases(v); - v = match self.value_def(v) { - ValueDef::Res(inst, 0) => { - match self[inst] { - InstructionData::Unary { opcode, arg, .. } => { - match opcode { - Opcode::Copy | Opcode::Spill | Opcode::Fill => arg, - _ => return v, - } - } - _ => return v, - } - } - _ => return v, - }; - } - panic!("Copy loop detected for {}", value); - } - /// Resolve all aliases among inst's arguments. /// /// For each argument of inst which is defined by an alias, replace the @@ -1090,7 +1063,5 @@ mod tests { let c3 = pos.ins().copy(c); // This does not see through copies. assert_eq!(pos.func.dfg.resolve_aliases(c3), c3); - // But this goes through both copies and aliases. - assert_eq!(pos.func.dfg.resolve_copies(c3), c2); } } diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index 62e5391474..96e1b03145 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -190,7 +190,7 @@ fn split_value( concat: Opcode, repairs: &mut Vec, ) -> (Value, Value) { - let value = pos.func.dfg.resolve_copies(value); + let value = pos.func.dfg.resolve_aliases(value); let mut reuse = None; match pos.func.dfg.value_def(value) { @@ -293,7 +293,7 @@ fn add_repair( /// /// This function resolves `v11` to `v1` and `v12` to `v2`. fn resolve_splits(dfg: &ir::DataFlowGraph, value: Value) -> Value { - let value = dfg.resolve_copies(value); + let value = dfg.resolve_aliases(value); // Deconstruct a split instruction. let split_res; From d0b4c76262338624782cca4d6f664df1a335456e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 6 Oct 2017 09:20:39 -0700 Subject: [PATCH 1222/3084] Use a non-allocating sort algorithm. The sort_unstable* functions are available in stable Rust now. These functions never allocate memory. --- lib/cretonne/src/regalloc/spilling.rs | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 4fe14a9528..a4fb4ededd 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -356,9 +356,8 @@ impl<'a> Context<'a> { // Leave `self.reg_uses` empty. fn process_reg_uses(&mut self, inst: Inst, tracker: &LiveValueTracker) { // We're looking for multiple uses of the same value, so start by sorting by value. The - // secondary `opidx` key makes it possible to use an unstable sort once that is available - // outside nightly Rust. - self.reg_uses.sort_by_key(|u| (u.value, u.opidx)); + // secondary `opidx` key makes it possible to use an unstable (non-allocating) sort. + self.reg_uses.sort_unstable_by_key(|u| (u.value, u.opidx)); for i in 0..self.reg_uses.len() { let ru = self.reg_uses[i]; From b3fa47cacc651a259008b82f92e53c72a294d743 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 6 Oct 2017 09:11:51 -0700 Subject: [PATCH 1223/3084] Add support for emergency spill slots. - Create a new kind of stack slot: emergency_slot. - Add a get_emergency_slot() method which finds a suitable emergency slot given a list of slots already in use. - Use emergency spill slots when schedule_moves needs them. --- cranelift/filetests/parser/tiny.cton | 2 + .../filetests/regalloc/schedule-moves.cton | 28 ++++++- lib/cretonne/src/ir/stackslot.rs | 78 +++++++++++++++++++ lib/cretonne/src/regalloc/coloring.rs | 38 ++++++++- lib/cretonne/src/regalloc/solver.rs | 12 ++- lib/cretonne/src/stack_layout.rs | 4 +- misc/vim/syntax/cton.vim | 2 +- 7 files changed, 154 insertions(+), 10 deletions(-) diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 9727e40177..13513a2b6c 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -117,6 +117,7 @@ function %stack() { ss2 = local 4 ss3 = incoming_arg 4, offset 8 ss4 = outgoing_arg 4 + ss5 = emergency_slot 4 ebb0: v1 = stack_load.i32 ss10 @@ -129,6 +130,7 @@ ebb0: ; nextln: $ss2 = local 4 ; nextln: $ss3 = incoming_arg 4, offset 8 ; nextln: $ss4 = outgoing_arg 4 +; nextln: $ss5 = emergency_slot 4 ; check: ebb0: ; nextln: $v1 = stack_load.i32 $ss10 diff --git a/cranelift/filetests/regalloc/schedule-moves.cton b/cranelift/filetests/regalloc/schedule-moves.cton index 79536aecf6..e0d9cf4e6e 100644 --- a/cranelift/filetests/regalloc/schedule-moves.cton +++ b/cranelift/filetests/regalloc/schedule-moves.cton @@ -1,12 +1,11 @@ test compile -set is_64bit=1 isa intel haswell function %pr165() native { ebb0: - v0 = iconst.i64 0x0102_0304_f1f2_f3f4 - v1 = iconst.i64 0x1102_0304_f1f2_f3f4 - v2 = iconst.i64 0x2102_0304_f1f2_f3f4 + v0 = iconst.i32 0x0102_0304 + v1 = iconst.i32 0x1102_0304 + v2 = iconst.i32 0x2102_0304 v20 = ishl v1, v0 v21 = ishl v2, v0 v22 = sshr v1, v0 @@ -17,3 +16,24 @@ ebb0: istore8 v1, v0+0x2710 return } + +; Same as above, but use so many registers that spilling is required. +; Note: This is also a candidate for using xchg instructions. +function %emergency_spill() native { +ebb0: + v0 = iconst.i32 0x0102_0304 + v1 = iconst.i32 0x1102_0304 + v2 = iconst.i32 0x2102_0304 + v3 = iconst.i32 0x3102_0304 + v4 = iconst.i32 0x4102_0304 + v20 = ishl v1, v0 + v21 = ishl v2, v3 + v22 = sshr v1, v0 + v23 = sshr v2, v0 + v24 = ushr v1, v0 + v25 = ushr v2, v0 + istore8 v0, v1+0x2710 + istore8 v1, v0+0x2710 + istore8 v3, v4+0x2710 + return +} diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index e5a4b10161..c1ad330e22 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -5,6 +5,7 @@ use entity::{PrimaryMap, Keys}; use ir::{Type, StackSlot}; +use packed_option::PackedOption; use std::fmt; use std::ops::Index; use std::str::FromStr; @@ -44,6 +45,12 @@ pub enum StackSlotKind { /// stack slots are used to represent individual arguments in the outgoing call frame. These /// stack slots are only valid while setting up a call. OutgoingArg, + + /// An emergency spill slot. + /// + /// Emergency slots are allocated late when the register's constraint solver needs extra space + /// to shuffle registers around. The are only used briefly, and can be reused. + EmergencySlot, } impl FromStr for StackSlotKind { @@ -56,6 +63,7 @@ impl FromStr for StackSlotKind { "spill_slot" => Ok(SpillSlot), "incoming_arg" => Ok(IncomingArg), "outgoing_arg" => Ok(OutgoingArg), + "emergency_slot" => Ok(EmergencySlot), _ => Err(()), } } @@ -69,6 +77,7 @@ impl fmt::Display for StackSlotKind { SpillSlot => "spill_slot", IncomingArg => "incoming_arg", OutgoingArg => "outgoing_arg", + EmergencySlot => "emergency_slot", }) } } @@ -135,6 +144,9 @@ pub struct StackSlots { /// All the outgoing stack slots, ordered by offset. outgoing: Vec, + /// All the emergency slots. + emergency: Vec, + /// The total size of the stack frame. /// /// This is the distance from the stack pointer in the current function to the stack pointer in @@ -152,6 +164,7 @@ impl StackSlots { StackSlots { slots: PrimaryMap::new(), outgoing: Vec::new(), + emergency: Vec::new(), frame_size: None, } } @@ -160,6 +173,7 @@ impl StackSlots { pub fn clear(&mut self) { self.slots.clear(); self.outgoing.clear(); + self.emergency.clear(); self.frame_size = None; } @@ -243,6 +257,43 @@ impl StackSlots { self.outgoing.insert(inspos, ss); ss } + + /// Get an emergency spill slot that can be used to store a `ty` value. + /// + /// This may allocate a new slot, or it may reuse an existing emergency spill slot, excluding + /// any slots in the `in_use` list. + pub fn get_emergency_slot( + &mut self, + ty: Type, + in_use: &[PackedOption], + ) -> StackSlot { + let size = ty.bytes(); + + // Find the smallest existing slot that can fit the type. + if let Some(&ss) = self.emergency + .iter() + .filter(|&&ss| self[ss].size >= size && !in_use.contains(&ss.into())) + .min_by_key(|&&ss| self[ss].size) + { + return ss; + } + + // Alternatively, use the largest available slot and make it larger. + if let Some(&ss) = self.emergency + .iter() + .filter(|&&ss| !in_use.contains(&ss.into())) + .max_by_key(|&&ss| self[ss].size) + { + self.slots[ss].size = size; + return ss; + } + + // No existing slot found. Make one and insert it into `emergency`. + let data = StackSlotData::new(StackSlotKind::EmergencySlot, size); + let ss = self.slots.push(data); + self.emergency.push(ss); + ss + } } #[cfg(test)] @@ -304,4 +355,31 @@ mod tests { assert_eq!(slot2.alignment(16), 8); assert_eq!(slot2.alignment(32), 8); } + + #[test] + fn emergency() { + let mut sss = StackSlots::new(); + + let ss0 = sss.get_emergency_slot(types::I32, &[]); + assert_eq!(sss[ss0].size, 4); + + // When a smaller size is requested, we should simply get the same slot back. + assert_eq!(sss.get_emergency_slot(types::I8, &[]), ss0); + assert_eq!(sss[ss0].size, 4); + assert_eq!(sss.get_emergency_slot(types::F32, &[]), ss0); + assert_eq!(sss[ss0].size, 4); + + // Ask for a larger size and the slot should grow. + assert_eq!(sss.get_emergency_slot(types::F64, &[]), ss0); + assert_eq!(sss[ss0].size, 8); + + // When one slot is in use, we should get a new one. + let ss1 = sss.get_emergency_slot(types::I32, &[None.into(), ss0.into()]); + assert_eq!(sss[ss0].size, 8); + assert_eq!(sss[ss1].size, 4); + + // Now we should get the smallest fit of the two available slots. + assert_eq!(sss.get_emergency_slot(types::F32, &[]), ss1); + assert_eq!(sss.get_emergency_slot(types::F64, &[]), ss0); + } } diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 69e1086416..2aad6093a3 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -48,6 +48,7 @@ use ir::{Ebb, Inst, Value, Function, ValueLoc, SigRef}; use ir::{InstBuilder, ArgumentType, ArgumentLoc}; use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; use isa::{TargetIsa, EncInfo, RecipeConstraints, OperandConstraint, ConstraintKind}; +use packed_option::PackedOption; use regalloc::RegDiversions; use regalloc::affinity::Affinity; use regalloc::allocatable_set::AllocatableSet; @@ -733,14 +734,47 @@ impl<'a> Context<'a> { fn shuffle_inputs(&mut self, regs: &mut AllocatableSet) { use regalloc::solver::Move::*; - self.solver.schedule_moves(regs); + let spills = self.solver.schedule_moves(regs); + + // The move operations returned by `schedule_moves` refer to emergency spill slots by + // consecutive indexes starting from 0. Map these to real stack slots. + // It is very unlikely (impossible?) that we would need more than one spill per top-level + // register class, so avoid allocation by using a fixed array here. + let mut slot = [PackedOption::default(); 8]; + assert!(spills <= slot.len(), "Too many spills ({})", spills); + for m in self.solver.moves() { match *m { Reg { value, from, to, .. } => { self.divert.regmove(value, from, to); self.cur.ins().regmove(value, from, to); } - Spill { .. } | Fill { .. } => unimplemented!(), + Spill { + value, + from, + to_slot, + .. + } => { + debug_assert_eq!(slot[to_slot].expand(), None, "Overwriting slot in use"); + let ss = self.cur.func.stack_slots.get_emergency_slot( + self.cur.func.dfg.value_type(value), + &slot[0..spills], + ); + slot[to_slot] = ss.into(); + self.divert.regspill(value, from, ss); + self.cur.ins().regspill(value, from, ss); + } + Fill { + value, + from_slot, + to, + .. + } => { + // These slots are single use, so mark `ss` as available again. + let ss = slot[from_slot].take().expect("Using unallocated slot"); + self.divert.regfill(value, ss, to); + self.cur.ins().regfill(value, ss, to); + } } } } diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 5f6f27cbfd..ae5f23afb4 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -523,7 +523,13 @@ impl Solver { /// In either case, `to` will not be available for variables on the input side of the /// instruction. pub fn reassign_in(&mut self, value: Value, rc: RegClass, from: RegUnit, to: RegUnit) { - dbg!("reassign_in({}:{}, %{} -> %{})", value, rc, from, to); + dbg!( + "reassign_in({}:{}, {} -> {})", + value, + rc, + rc.info.display_regunit(from), + rc.info.display_regunit(to) + ); debug_assert!(!self.inputs_done); if self.regs_in.is_avail(rc, from) { // It looks like `value` was already removed from the register set. It must have been @@ -826,7 +832,9 @@ impl Solver { Move::with_assignment, )); - dbg!("collect_moves: {}", DisplayList(&self.moves)); + if !(self.moves.is_empty()) { + dbg!("collect_moves: {}", DisplayList(&self.moves)); + } } /// Try to schedule a sequence of `regmove` instructions that will shuffle registers into diff --git a/lib/cretonne/src/stack_layout.rs b/lib/cretonne/src/stack_layout.rs index 7486cee8a6..796850e059 100644 --- a/lib/cretonne/src/stack_layout.rs +++ b/lib/cretonne/src/stack_layout.rs @@ -56,7 +56,9 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result { + StackSlotKind::SpillSlot | + StackSlotKind::Local | + StackSlotKind::EmergencySlot => { // Determine the smallest alignment of any local or spill slot. min_align = slot.alignment(min_align); } diff --git a/misc/vim/syntax/cton.vim b/misc/vim/syntax/cton.vim index 3bc6a4a5da..56602e8ded 100644 --- a/misc/vim/syntax/cton.vim +++ b/misc/vim/syntax/cton.vim @@ -14,7 +14,7 @@ endif syn spell notoplevel syn keyword ctonHeader test isa set -syn keyword ctonDecl function jump_table incoming_arg outgoing_arg spill_slot local +syn keyword ctonDecl function jump_table incoming_arg outgoing_arg spill_slot local emergency_slot syn keyword ctonFilecheck check sameln nextln unordered not regex contained syn match ctonType /\<[bif]\d\+\(x\d\+\)\?\>/ From 56862c3da9e45f2df4faabd514bd4746e9281a94 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 6 Oct 2017 12:12:32 -0700 Subject: [PATCH 1224/3084] Roll back to mypy 0.521 to avoid a bug in 0.530. Filed here: https://github.com/python/mypy/issues/4069 We should go back to tracking the latest mypy releases as soon as possible. --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 03dd6f76e6..1309bfce63 100644 --- a/.travis.yml +++ b/.travis.yml @@ -9,7 +9,7 @@ addons: packages: - python3-pip install: - - pip3 install --user --upgrade mypy flake8 + - pip3 install --user --upgrade mypy==0.521 flake8 - travis_wait ./check-rustfmt.sh --install script: ./test-all.sh cache: From d1766f0ba46fb5318fc5f3929a6d6ee1ec184fdc Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 25 Sep 2017 16:22:36 -0700 Subject: [PATCH 1225/3084] Simplify code by removing unnecessary `Option`s. --- lib/wasm/src/module_translator.rs | 30 +++++++----------------------- 1 file changed, 7 insertions(+), 23 deletions(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 7966bdd543..001a85bd69 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -39,9 +39,9 @@ pub fn translate_module( } ref s => panic!("modules should begin properly: {:?}", s), } - let mut functions: Option> = None; + let mut functions: Vec = Vec::new(); let mut globals = Vec::new(); - let mut exports: Option> = None; + let mut exports: HashMap = HashMap::new(); let mut next_input = ParserInput::Default; let mut function_index: FunctionIndex = 0; let mut start_index: Option = None; @@ -62,13 +62,7 @@ pub fn translate_module( for import in imps { match import { Import::Function { sig_index } => { - functions = match functions { - None => Some(vec![sig_index as SignatureIndex]), - Some(mut funcs) => { - funcs.push(sig_index as SignatureIndex); - Some(funcs) - } - }; + functions.push(sig_index as SignatureIndex); function_index += 1; } Import::Memory(mem) => { @@ -93,10 +87,7 @@ pub fn translate_module( ParserState::BeginSection { code: SectionCode::Function, .. } => { match parse_function_section(&mut parser, runtime) { Ok(funcs) => { - match functions { - None => functions = Some(funcs), - Some(ref mut imps) => imps.extend(funcs), - } + functions.extend(funcs); } Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the function section: {}", s)) @@ -136,7 +127,7 @@ pub fn translate_module( } ParserState::BeginSection { code: SectionCode::Export, .. } => { match parse_export_section(&mut parser) { - Ok(exps) => exports = Some(exps), + Ok(exps) => exports = exps, Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the export section: {}", s)) } @@ -190,11 +181,6 @@ pub fn translate_module( }; } // At this point we've entered the code section - // First we check that we have all that is necessary to translate a function. - let functions = match functions { - None => return Err(String::from("missing a function section")), - Some(functions) => functions, - }; let mut il_functions: Vec = Vec::new(); let mut trans = FuncTranslator::new(); runtime.begin_translation(); @@ -208,10 +194,8 @@ pub fn translate_module( // First we build the Function object with its name and signature let mut func = Function::new(); func.signature = runtime.get_signature(functions[function_index]).clone(); - if let Some(ref exports) = exports { - if let Some(name) = exports.get(&function_index) { - func.name = FunctionName::new(name.clone()); - } + if let Some(name) = exports.get(&function_index) { + func.name = FunctionName::new(name.clone()); } trans .translate_from_reader(parser.create_binary_reader(), &mut func, runtime) From 0c4500897f792451333d77f75aef05efd4f1e8de Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 9 Oct 2017 13:50:03 -0700 Subject: [PATCH 1226/3084] Clarify FunctionName's role in its comment. --- lib/cretonne/src/ir/funcname.rs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/ir/funcname.rs b/lib/cretonne/src/ir/funcname.rs index 142a846c22..0cffae6d7c 100644 --- a/lib/cretonne/src/ir/funcname.rs +++ b/lib/cretonne/src/ir/funcname.rs @@ -8,8 +8,12 @@ use std::ascii::AsciiExt; /// The name of a function can be any sequence of bytes. /// -/// Function names are mostly a testing and debugging tool. -/// In particular, `.cton` files use function names to identify functions. +/// Function names are primarily used as keys by code using Cretonne to map +/// from a cretonne::ir::Function to additional associated data. +/// +/// Function names can also serve as a primitive testing and debugging tool. +/// In particular, many `.cton` test files use function names to identify +/// functions. #[derive(Debug, Clone, PartialEq, Eq, Default)] pub struct FunctionName(NameRepr); From 12a8d6cce1f98337f6c5c19dca859285b8524767 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 6 Oct 2017 14:50:33 -0700 Subject: [PATCH 1227/3084] Avoid diverting values that are live on an outgoing CFG edge. When try_add_var is looking for values that can be moved out of the way in order to satisfy constraints for the current instruction, avoid values that are live on a CFG edge originating at the current (branch) instruction. These values must be in their globally assigned location when entering the branch destination EBB. This is covered by the existing regalloc/iterate.cton test case which fails with an upcoming commit. --- lib/cretonne/src/regalloc/coloring.rs | 29 ++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 2aad6093a3..a7a0d23883 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -711,7 +711,9 @@ impl<'a> Context<'a> { if let Affinity::Reg(rci) = lv.affinity { let rc2 = self.reginfo.rc(rci); let reg2 = self.divert.reg(lv.value, &self.cur.func.locations); - if rc.contains(reg2) && self.solver.can_add_var(lv.value, rc2, reg2) { + if rc.contains(reg2) && self.solver.can_add_var(lv.value, rc2, reg2) && + !self.is_live_on_outgoing_edge(lv.value) + { // The new variable gets to roam the whole top-level register class because // it is not actually constrained by the instruction. We just want it out // of the way. @@ -724,6 +726,31 @@ impl<'a> Context<'a> { false } + /// Determine if `value` is live on a CFG edge from the current instruction. + /// + /// This means that the current instruction is a branch and `value` is live in to one of the + /// branch destinations. Branch arguments and EBB parameters are not considered live on the + /// edge. + fn is_live_on_outgoing_edge(&self, value: Value) -> bool { + use ir::instructions::BranchInfo::*; + + let inst = self.cur.current_inst().expect("Not on an instruction"); + match self.cur.func.dfg[inst].analyze_branch(&self.cur.func.dfg.value_lists) { + NotABranch => false, + SingleDest(ebb, _) => { + let lr = &self.liveness[value]; + lr.is_livein(ebb, &self.cur.func.layout) + } + Table(jt) => { + let lr = &self.liveness[value]; + !lr.is_local() && + self.cur.func.jump_tables[jt].entries().any(|(_, ebb)| { + lr.is_livein(ebb, &self.cur.func.layout) + }) + } + } + } + /// Emit `regmove` instructions as needed to move the live registers into place before the /// instruction. Also update `self.divert` accordingly. /// From ac8c8a676a799d7efdc24316a724398f0ac0f88b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 6 Oct 2017 15:03:50 -0700 Subject: [PATCH 1228/3084] Constrain solver variables as little as possible. When solver variables represent operands on the current instruction, they need to be constrained as required by the instructions, but variables that are simply moved out of the way should only be constrained to their top-level register class. The live range affinity is just a hint, not a requirement. --- lib/cretonne/src/isa/registers.rs | 5 +++++ lib/cretonne/src/regalloc/coloring.rs | 26 +++++++++++--------------- 2 files changed, 16 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index b8e7fe1148..db68399cb4 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -296,6 +296,11 @@ impl RegInfo { pub fn rc(&self, idx: RegClassIndex) -> RegClass { self.classes[idx.index()] } + + /// Get the top-level register class containing the `idx` class. + pub fn toprc(&self, idx: RegClassIndex) -> RegClass { + self.classes[self.rc(idx).toprc as usize] + } } /// Temporary object that holds enough information to print a register unit. diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index a7a0d23883..d5d7e804f7 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -572,10 +572,10 @@ impl<'a> Context<'a> { fn divert_fixed_input_conflicts(&mut self, live: &[LiveValue]) { for lv in live { if let Affinity::Reg(rci) = lv.affinity { - let rc = self.reginfo.rc(rci); + let toprc = self.reginfo.toprc(rci); let reg = self.divert.reg(lv.value, &self.cur.func.locations); - if self.solver.is_fixed_input_conflict(rc, reg) { - self.solver.add_var(lv.value, rc, reg); + if self.solver.is_fixed_input_conflict(toprc, reg) { + self.solver.add_var(lv.value, toprc, reg); } } } @@ -633,15 +633,12 @@ impl<'a> Context<'a> { // The fixed output conflicts with some of the live-through registers. for lv in throughs { if let Affinity::Reg(rci) = lv.affinity { - let rc2 = self.reginfo.rc(rci); + let toprc2 = self.reginfo.toprc(rci); let reg2 = self.divert.reg(lv.value, &self.cur.func.locations); - if regs_overlap(rc, reg, rc2, reg2) { + if regs_overlap(rc, reg, toprc2, reg2) { // This live-through value is interfering with the fixed output assignment. // Convert it to a solver variable. - // TODO: Use a looser constraint than the affinity hint. Any allocatable - // register in the top-level register class would be OK. Maybe `add_var` - // should take both a preferred class and a required constraint class. - self.solver.add_var(lv.value, rc2, reg2); + self.solver.add_var(lv.value, toprc2, reg2); } } } @@ -709,15 +706,14 @@ impl<'a> Context<'a> { for lv in throughs { if let Affinity::Reg(rci) = lv.affinity { - let rc2 = self.reginfo.rc(rci); + // The new variable gets to roam the whole top-level register class because it is + // not actually constrained by the instruction. We just want it out of the way. + let toprc2 = self.reginfo.rc(rci); let reg2 = self.divert.reg(lv.value, &self.cur.func.locations); - if rc.contains(reg2) && self.solver.can_add_var(lv.value, rc2, reg2) && + if rc.contains(reg2) && self.solver.can_add_var(lv.value, toprc2, reg2) && !self.is_live_on_outgoing_edge(lv.value) { - // The new variable gets to roam the whole top-level register class because - // it is not actually constrained by the instruction. We just want it out - // of the way. - self.solver.add_var(lv.value, rc2.toprc(), reg2); + self.solver.add_var(lv.value, toprc2, reg2); return true; } } From ecd537ecd68af43a55220d080ba16ccfe5fcd57e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 6 Oct 2017 14:13:15 -0700 Subject: [PATCH 1229/3084] Avoid widening TailRecipe register constraints automatically. Most recipes with an ABCD constraint can handle the full GPR register class when a REX prefix is applied, but not all. The "icscc" macro recipe always generates a setCC instruction with no REX prefix, so it can only write the ABCD registers, even in its REX form. Don't automatically rewrite ABCD constraints to GPR constraints when applying a REX prefix to a tail recipe. Instead, allow individual ABCD recipes to specify a "when_prefixed" alternative recipe to use. This also eliminates the spurious Rex*abcd recipe names which didn't have an ABCD constraint. Also allow recipes to specify that a REX prefix is required by setting the prefix_required flag. This is used by recipes like t8jccb which explicitly accesses an 8-bit register with a GPR constraint which is only valid with a prefix. --- lib/cretonne/meta/isa/intel/recipes.py | 91 ++++++++++++++++++-------- 1 file changed, 65 insertions(+), 26 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 9c4713500a..cd59cc1bbb 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -104,19 +104,10 @@ NOREX_MAP = { FPR: FPR8 } -# Register class mapping for REX instructions. The ABCD constraint no longer -# applies. -REX_MAP = { - ABCD: GPR - } - -def map_regs( - regs, # type: Sequence[OperandConstraint] - mapping # type: Dict[RegClass, RegClass] -): - # type: (...) -> Sequence[OperandConstraint] - return tuple(mapping.get(rc, rc) if isinstance(rc, RegClass) else rc +def map_regs_norex(regs): + # type: (Sequence[OperandConstraint]) -> Sequence[OperandConstraint] + return tuple(NOREX_MAP.get(rc, rc) if isinstance(rc, RegClass) else rc for rc in regs) @@ -134,6 +125,14 @@ class TailRecipe: The arguments are the same as for an `EncRecipe`, except for `size` which does not include the size of the opcode. + The `when_prefixed` parameter specifies a recipe that should be substituted + for this one when a REX (or VEX) prefix is present. This is relevant for + recipes that can only access the ABCD registers without a REX prefix, but + are able to access all registers with a prefix. + + The `requires_prefix` parameter indicates that the recipe can't be used + without a REX prefix. + The `emit` parameter contains Rust code to actually emit an encoding, like `EncRecipe` does it. Additionally, the text `PUT_OP` is substituted with the proper `put_*` function from the `intel/binemit.rs` module. @@ -141,15 +140,17 @@ class TailRecipe: def __init__( self, - name, # type: str - format, # type: InstructionFormat - size, # type: int - ins, # type: ConstraintSeq - outs, # type: ConstraintSeq - branch_range=None, # type: int - instp=None, # type: PredNode - isap=None, # type: PredNode - emit=None # type: str + name, # type: str + format, # type: InstructionFormat + size, # type: int + ins, # type: ConstraintSeq + outs, # type: ConstraintSeq + branch_range=None, # type: int + instp=None, # type: PredNode + isap=None, # type: PredNode + when_prefixed=None, # type: TailRecipe + requires_prefix=False, # type: bool + emit=None # type: str ): # type: (...) -> None self.name = name @@ -160,6 +161,8 @@ class TailRecipe: self.branch_range = branch_range self.instp = instp self.isap = isap + self.when_prefixed = when_prefixed + self.requires_prefix = requires_prefix self.emit = emit # Cached recipes, keyed by name prefix. @@ -171,6 +174,7 @@ class TailRecipe: Create an encoding recipe and encoding bits for the opcode bytes in `ops`. """ + assert not self.requires_prefix, "Tail recipe requires REX prefix." rrr = kwargs.get('rrr', 0) w = kwargs.get('w', 0) name, bits = decode_ops(ops, rrr, w) @@ -193,8 +197,8 @@ class TailRecipe: isap=self.isap, emit=replace_put_op(self.emit, name)) - recipe.ins = map_regs(recipe.ins, NOREX_MAP) - recipe.outs = map_regs(recipe.outs, NOREX_MAP) + recipe.ins = map_regs_norex(recipe.ins) + recipe.outs = map_regs_norex(recipe.outs) self.recipes[name] = recipe return (self.recipes[name], bits) @@ -208,6 +212,10 @@ class TailRecipe: not. For instructions that don't require a REX prefix, two encodings should be added: One with REX and one without. """ + # Use the prefixed alternative recipe when applicable. + if self.when_prefixed: + return self.when_prefixed.rex(*ops, **kwargs) + rrr = kwargs.get('rrr', 0) w = kwargs.get('w', 0) name, bits = decode_ops(ops, rrr, w) @@ -230,8 +238,6 @@ class TailRecipe: instp=self.instp, isap=self.isap, emit=replace_put_op(self.emit, name)) - recipe.ins = map_regs(recipe.ins, REX_MAP) - recipe.outs = map_regs(recipe.outs, REX_MAP) self.recipes[name] = recipe return (self.recipes[name], bits) @@ -314,6 +320,7 @@ urm = TailRecipe( # XX /r. Same as urm, but input limited to ABCD. urm_abcd = TailRecipe( 'urm_abcd', Unary, size=1, ins=ABCD, outs=GPR, + when_prefixed=urm, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_rr(in_reg0, out_reg0, sink); @@ -478,10 +485,11 @@ st = TailRecipe( ''') # XX /r register-indirect store with no offset. -# Only ABCD allowed for stored value. This is for byte stores. +# Only ABCD allowed for stored value. This is for byte stores with no REX. st_abcd = TailRecipe( 'st_abcd', Store, size=1, ins=(ABCD, GPR), outs=(), instp=IsEqual(Store.offset, 0), + when_prefixed=st, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_rm(in_reg1, in_reg0, sink); @@ -509,6 +517,7 @@ stDisp8 = TailRecipe( stDisp8_abcd = TailRecipe( 'stDisp8_abcd', Store, size=2, ins=(ABCD, GPR), outs=(), instp=IsSignedInt(Store.offset, 8), + when_prefixed=stDisp8, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp8(in_reg1, in_reg0, sink); @@ -536,6 +545,7 @@ stDisp32 = TailRecipe( ''') stDisp32_abcd = TailRecipe( 'stDisp32_abcd', Store, size=5, ins=(ABCD, GPR), outs=(), + when_prefixed=stDisp32, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp32(in_reg1, in_reg0, sink); @@ -786,9 +796,22 @@ tjccd = TailRecipe( # # Same as tjccb, but only looks at the low 8 bits of the register, for b1 # types. +t8jccb = TailRecipe( + 't8jccb', Branch, size=1 + 2, ins=GPR, outs=(), + branch_range=8, + requires_prefix=True, + emit=''' + // test8 r, r. + PUT_OP((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink); + modrm_rr(in_reg0, in_reg0, sink); + // Jcc instruction. + sink.put1(bits as u8); + disp1(destination, func, sink); + ''') t8jccb_abcd = TailRecipe( 't8jccb_abcd', Branch, size=1 + 2, ins=ABCD, outs=(), branch_range=8, + when_prefixed=t8jccb, emit=''' // test8 r, r. PUT_OP((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink); @@ -798,9 +821,23 @@ t8jccb_abcd = TailRecipe( disp1(destination, func, sink); ''') +t8jccd = TailRecipe( + 't8jccd', Branch, size=1 + 6, ins=GPR, outs=(), + branch_range=32, + requires_prefix=True, + emit=''' + // test8 r, r. + PUT_OP((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink); + modrm_rr(in_reg0, in_reg0, sink); + // Jcc instruction. + sink.put1(0x0f); + sink.put1(bits as u8); + disp4(destination, func, sink); + ''') t8jccd_abcd = TailRecipe( 't8jccd_abcd', Branch, size=1 + 6, ins=ABCD, outs=(), branch_range=32, + when_prefixed=t8jccd, emit=''' // test8 r, r. PUT_OP((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink); @@ -827,6 +864,7 @@ t8jccd_abcd = TailRecipe( # # This bandaid macro doesn't support a REX prefix for the final `setCC` # instruction, so it is limited to the `ABCD` register class for booleans. +# The omission of a `when_prefixed` alternative is deliberate here. icscc = TailRecipe( 'icscc', IntCompare, size=1 + 3, ins=(GPR, GPR), outs=ABCD, emit=''' @@ -866,6 +904,7 @@ icscc = TailRecipe( # EQ 100 000 # # Not all floating point condition codes are supported. +# The omission of a `when_prefixed` alternative is deliberate here. fcscc = TailRecipe( 'fcscc', FloatCompare, size=1 + 3, ins=(FPR, FPR), outs=ABCD, instp=Or(*(IsEqual(FloatCompare.cond, cc) From 4a2bf6d9a6c30910bbd75e3bedd2c3fd2819f32e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 6 Oct 2017 19:17:37 -0700 Subject: [PATCH 1230/3084] Use a more compact display of AllocatableSet. Since only Intel uses named registers, we can use a one-char shorthand for the registers. --- lib/cretonne/src/regalloc/allocatable_set.rs | 42 +++++++++++++++++--- lib/cretonne/src/regalloc/coloring.rs | 2 +- lib/cretonne/src/regalloc/solver.rs | 3 ++ 3 files changed, 41 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index c6f8c86a07..355443affc 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -6,6 +6,7 @@ //! share a register unit can't be in use at the same time. use isa::registers::{RegInfo, RegUnit, RegUnitMask, RegClass}; +use std::char; use std::fmt; use std::iter::ExactSizeIterator; use std::mem::size_of_val; @@ -51,14 +52,26 @@ impl AllocatableSet { /// It is an error to take a register that doesn't have all of its register units available. pub fn take(&mut self, rc: RegClass, reg: RegUnit) { let (idx, bits) = bitmask(rc, reg); - debug_assert_eq!(self.avail[idx] & bits, bits, "Not available"); + debug_assert!( + (self.avail[idx] & bits) == bits, + "{}:{} not available in {}", + rc, + rc.info.display_regunit(reg), + self.display(rc.info) + ); self.avail[idx] &= !bits; } /// Make `reg` available for allocation again. pub fn free(&mut self, rc: RegClass, reg: RegUnit) { let (idx, bits) = bitmask(rc, reg); - debug_assert_eq!(self.avail[idx] & bits, 0, "Not allocated"); + debug_assert!( + (self.avail[idx] & bits) == 0, + "{}:{} not allocated in {}", + rc, + rc.info.display_regunit(reg), + self.display(rc.info) + ); self.avail[idx] |= bits; } @@ -165,9 +178,28 @@ impl<'a> fmt::Display for DisplayAllocatableSet<'a> { .expect("No register banks"); for rc in ®info.classes[0..toprcs] { if rc.width == 1 { - write!(f, " {}:", rc)?; - for u in self.0.iter(rc) { - write!(f, " {}", reginfo.display_regunit(u))?; + let bank = ®info.banks[rc.bank as usize]; + write!(f, " {}: ", rc)?; + for offset in 0..bank.units { + let reg = bank.first_unit + offset; + if !rc.contains(reg) { + continue; + } + if !self.0.is_avail(rc, reg) { + write!(f, "-")?; + continue; + } + // Display individual registers as either the second letter of their + // name or the last digit of their number. + // This works for Intel (rax, rbx, ...) and for numbered regs. + write!( + f, + "{}", + bank.names + .get(offset as usize) + .and_then(|name| name.chars().skip(1).next()) + .unwrap_or(char::from_digit((offset % 10) as u32, 10).unwrap()) + )?; } } } diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index d5d7e804f7..8c4de9163f 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -287,7 +287,7 @@ impl<'a> Context<'a> { regs: &mut AllocatableSet, ) { dbg!( - "Coloring {}\n {}", + "Coloring {}\n from {}", self.cur.display_inst(inst), regs.display(&self.reginfo) ); diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index ae5f23afb4..f21993a4c7 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -962,7 +962,10 @@ impl Solver { impl fmt::Display for Solver { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + let reginfo = self.vars.first().map(|v| v.constraint.info); writeln!(f, "Solver {{ inputs_done: {},", self.inputs_done)?; + writeln!(f, " in: {}", self.regs_in.display(reginfo))?; + writeln!(f, " out: {}", self.regs_out.display(reginfo))?; writeln!( f, " assignments: {}", From 893a6716c6b1197a1a52c2d3e121d83e939490ff Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 6 Oct 2017 18:51:36 -0700 Subject: [PATCH 1231/3084] Enforce all instruction constraints in iterate_solution(). During iterate_solution(), live-through values may be converted to solver variables so they can be moved out of the way in order to satisfy all constraints. Make sure that the instruction's operand constraints are also considered for these new variables. Add a program_complete_input_constraints() which turns all the instruction's input operands into variables with the proper constraints. That makes it safe for try_add_var() to re-add these values as variables with looser generic constraints. The solver's add_var() function is split into three functions: add_var for use before inputs_done(), and add_killed_var/add_through_var for use after. --- lib/cretonne/src/regalloc/coloring.rs | 49 +++++++++++-- lib/cretonne/src/regalloc/solver.rs | 99 +++++++++++++++++++-------- 2 files changed, 116 insertions(+), 32 deletions(-) diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 8c4de9163f..cdea494438 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -374,7 +374,7 @@ impl<'a> Context<'a> { // Finally, we've fully programmed the constraint solver. // We expect a quick solution in most cases. let mut output_regs = self.solver.quick_solve().unwrap_or_else(|rc| { - dbg!("quick_solve needs more registers in {}", rc); + dbg!("quick_solve needs more {} regs for {}", rc, self.solver); self.iterate_solution(throughs) }); @@ -454,6 +454,44 @@ impl<'a> Context<'a> { } } + /// Program the complete set of input constraints into the solver. + /// + /// The `program_input_constraints()` function above will not tell the solver about any values + /// that are already assigned to appropriate registers. This is normally fine, but if we want + /// to add additional variables to help the solver, we need to make sure that they are + /// constrained properly. + /// + /// This function completes the work of `program_input_constraints()` by calling `add_var` for + /// all values used by the instruction. + fn program_complete_input_constraints(&mut self) { + let inst = self.cur.current_inst().expect("Not on an instruction"); + let constraints = self.encinfo + .operand_constraints(self.cur.func.encodings[inst]) + .expect("Current instruction not encoded") + .ins; + + for (op, &value) in constraints.iter().zip(self.cur.func.dfg.inst_args(inst)) { + match op.kind { + ConstraintKind::Reg | + ConstraintKind::Tied(_) => { + let cur_reg = self.divert.reg(value, &self.cur.func.locations); + // This is the opposite condition of `program_input_constraints()`. + if op.regclass.contains(cur_reg) { + // This code runs after calling `solver.inputs_done()` so we must identify + // the new variable as killed or live-through. + let layout = &self.cur.func.layout; + if self.liveness[value].killed_at(inst, layout.pp_ebb(inst), layout) { + self.solver.add_killed_var(value, op.regclass, cur_reg); + } else { + self.solver.add_through_var(value, op.regclass, cur_reg); + } + } + } + _ => {} + } + } + } + /// Prepare for a branch to `dest`. /// /// 1. Any values that are live-in to `dest` must be un-diverted so they live in their globally @@ -638,7 +676,7 @@ impl<'a> Context<'a> { if regs_overlap(rc, reg, toprc2, reg2) { // This live-through value is interfering with the fixed output assignment. // Convert it to a solver variable. - self.solver.add_var(lv.value, toprc2, reg2); + self.solver.add_through_var(lv.value, toprc2, reg2); } } } @@ -684,8 +722,11 @@ impl<'a> Context<'a> { /// We may need to move more registers around before a solution is possible. Use an iterative /// algorithm that adds one more variable until a solution can be found. fn iterate_solution(&mut self, throughs: &[LiveValue]) -> AllocatableSet { + // Make sure `try_add_var()` below doesn't create a variable with too loose constraints. + self.program_complete_input_constraints(); + loop { - dbg!("real_solve for {} variables", self.solver.vars().len()); + dbg!("real_solve for {}", self.solver); let rc = match self.solver.real_solve() { Ok(regs) => return regs, Err(rc) => rc, @@ -713,7 +754,7 @@ impl<'a> Context<'a> { if rc.contains(reg2) && self.solver.can_add_var(lv.value, toprc2, reg2) && !self.is_live_on_outgoing_edge(lv.value) { - self.solver.add_var(lv.value, toprc2, reg2); + self.solver.add_through_var(lv.value, toprc2, reg2); return true; } } diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index f21993a4c7..2a2b527561 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -148,13 +148,13 @@ pub struct Variable { } impl Variable { - fn new_live(value: Value, constraint: RegClass, from: RegUnit) -> Variable { + fn new_live(value: Value, constraint: RegClass, from: RegUnit, is_output: bool) -> Variable { Variable { value, constraint, from: Some(from), is_input: true, - is_output: true, + is_output, is_global: false, domain: 0, solution: !0, @@ -430,7 +430,7 @@ impl fmt::Debug for Move { /// calling `add_kill()`. /// 6. Program the output side constraints: Call `add_fixed_output()` for all fixed register /// constraints and `add_def()` for free defines. Resolve fixed output conflicts by calling -/// `add_var()`. +/// `add_through_var()`. /// pub struct Solver { /// Register reassignments that are required or decided as part of a full solution. @@ -565,28 +565,74 @@ impl Solver { /// /// It is assumed initially that the value is also live on the output side of the instruction. /// This can be changed by calling to `add_kill()`. + /// + /// This function can only be used before calling `inputs_done()`. Afterwards, more input-side + /// variables can be added by calling `add_killed_var()` and `add_through_var()` pub fn add_var(&mut self, value: Value, constraint: RegClass, from: RegUnit) { + dbg!( + "add_var({}:{}, from={})", + value, + constraint, + constraint.info.display_regunit(from) + ); + debug_assert!(!self.inputs_done); + self.add_live_var(value, constraint, from, true); + } + + /// Add an extra input-side variable representing a value that is killed by the current + /// instruction. + /// + /// This function should be called after `inputs_done()` only. Use `add_var()` before. + pub fn add_killed_var(&mut self, value: Value, constraint: RegClass, from: RegUnit) { + dbg!( + "add_killed_var({}:{}, from={})", + value, + constraint, + constraint.info.display_regunit(from) + ); + debug_assert!(self.inputs_done); + self.add_live_var(value, constraint, from, false); + } + + /// Add an extra input-side variable representing a value that is live through the current + /// instruction. + /// + /// This function should be called after `inputs_done()` only. Use `add_var()` before. + pub fn add_through_var(&mut self, value: Value, constraint: RegClass, from: RegUnit) { + dbg!( + "add_through_var({}:{}, from={})", + value, + constraint, + constraint.info.display_regunit(from) + ); + debug_assert!(self.inputs_done); + self.add_live_var(value, constraint, from, true); + } + + /// Shared code for `add_var`, `add_killed_var`, and `add_through_var`. + /// + /// Add a variable that is live before the instruction, and possibly live through. Merge + /// constraints if the value has already been added as a variable or fixed assignment. + fn add_live_var( + &mut self, + value: Value, + constraint: RegClass, + from: RegUnit, + live_through: bool, + ) { // Check for existing entries for this value. if self.regs_in.is_avail(constraint, from) { - dbg!( - "add_var({}:{}, from={}) for existing entry", - value, - constraint, - constraint.info.display_regunit(from), - ); - // There could be an existing variable entry. if let Some(v) = self.vars.iter_mut().find(|v| v.value == value) { - dbg!("-> combining constraint with {}", v); - // We have an existing variable entry for `value`. Combine the constraints. if let Some(rc) = v.constraint.intersect(constraint) { + dbg!("-> combining constraint with {} yields {}", v, rc); v.constraint = rc; return; } else { // The spiller should have made sure the same value is not used with disjoint // constraints. - panic!("Incompatible constraints: {} + {}", constraint, *v) + panic!("Incompatible constraints: {} + {}", constraint, v) } } @@ -605,17 +651,11 @@ impl Solver { panic!("Wrong from register for {}", value); } - let new_var = Variable::new_live(value, constraint, from); - dbg!( - "add_var({}:{}, from={}) new entry: {}", - value, - constraint, - constraint.info.display_regunit(from), - new_var - ); + let new_var = Variable::new_live(value, constraint, from, live_through); + dbg!("-> new var: {}", new_var); self.regs_in.free(constraint, from); - if self.inputs_done { + if self.inputs_done && live_through { self.regs_out.free(constraint, from); } self.vars.push(new_var); @@ -817,12 +857,15 @@ impl Solver { // Collect moves from the chosen solution for all non-define variables. for v in &self.vars { if let Some(from) = v.from { - self.moves.push(Move::Reg { - value: v.value, - from, - to: v.solution, - rc: v.constraint, - }); + // Omit variable solutions that don't require the value to be moved. + if from != v.solution { + self.moves.push(Move::Reg { + value: v.value, + from, + to: v.solution, + rc: v.constraint, + }); + } } } From 6aeeaebbd3d42314129b72b4a708a39c273ab124 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 9 Oct 2017 14:59:08 -0700 Subject: [PATCH 1232/3084] Disallow branching to the entry block. Functions that would otherwise start with a loop should start with a separate ebb which just branches to the header of the loop. --- cranelift/filetests/licm/basic.cton | 43 +++---- cranelift/filetests/licm/complex.cton | 126 +++++++++++---------- cranelift/filetests/licm/nested_loops.cton | 65 ++++++----- lib/cretonne/src/dominator_tree.rs | 4 + lib/cretonne/src/verifier/mod.rs | 11 +- 5 files changed, 133 insertions(+), 116 deletions(-) diff --git a/cranelift/filetests/licm/basic.cton b/cranelift/filetests/licm/basic.cton index 37dda60d2a..84d9242169 100644 --- a/cranelift/filetests/licm/basic.cton +++ b/cranelift/filetests/licm/basic.cton @@ -2,30 +2,33 @@ test licm function %simple_loop(i32) -> i32 { -ebb1(v0: i32): - v1 = iconst.i32 1 - v2 = iconst.i32 2 - v3 = iadd v1, v2 - brz v0, ebb2(v0) - v4 = isub v0, v1 - jump ebb1(v4) +ebb0(v0: i32): + jump ebb1(v0) -ebb2(v5: i32): - return v5 +ebb1(v1: i32): + v2 = iconst.i32 1 + v3 = iconst.i32 2 + v4 = iadd v2, v3 + brz v1, ebb2(v1) + v5 = isub v1, v2 + jump ebb1(v5) + +ebb2(v6: i32): + return v6 } ; sameln: function %simple_loop -; nextln: ebb2(v6: i32): -; nextln: v1 = iconst.i32 1 -; nextln: v2 = iconst.i32 2 -; nextln: v3 = iadd v1, v2 -; nextln: jump ebb0(v6) -; nextln: ; nextln: ebb0(v0: i32): -; nextln: brz v0, ebb1(v0) -; nextln: v4 = isub v0, v1 -; nextln: jump ebb0(v4) +; nextln: v2 = iconst.i32 1 +; nextln: v3 = iconst.i32 2 +; nextln: v4 = iadd v2, v3 +; nextln: jump ebb1(v0) ; nextln: -; nextln: ebb1(v5: i32): -; nextln: return v5 +; nextln: ebb1(v1: i32): +; nextln: brz v1, ebb2(v1) +; nextln: v5 = isub v1, v2 +; nextln: jump ebb1(v5) +; nextln: +; nextln: ebb2(v6: i32): +; nextln: return v6 ; nextln: } diff --git a/cranelift/filetests/licm/complex.cton b/cranelift/filetests/licm/complex.cton index 07efb9ff5f..e9012b1a4f 100644 --- a/cranelift/filetests/licm/complex.cton +++ b/cranelift/filetests/licm/complex.cton @@ -1,81 +1,83 @@ test licm -function %complex(i32) -> i32 { - +function %complex(i32) -> i32 native { ebb0(v0: i32): - v1 = iconst.i32 1 - v19 = iconst.i32 4 - v2 = iadd v1, v0 - brz v0, ebb1(v1) - jump ebb3(v2) + jump ebb1(v0) -ebb1(v3: i32): - v4 = iconst.i32 2 - v5 = iadd v3, v2 - v6 = iadd v4, v0 - jump ebb2(v6) +ebb1(v1: i32): + v2 = iconst.i32 1 + v3 = iconst.i32 4 + v4 = iadd v2, v1 + brz v1, ebb2(v2) + jump ebb4(v4) -ebb2(v7: i32): - v8 = iadd v7, v3 - v9 = iadd v0, v2 - brz v0, ebb1(v7) - jump ebb5(v8) +ebb2(v5: i32): + v6 = iconst.i32 2 + v7 = iadd v5, v4 + v8 = iadd v6, v1 + jump ebb3(v8) -ebb3(v10: i32): - v11 = iconst.i32 3 - v12 = iadd v10, v11 - v13 = iadd v2, v11 - jump ebb4(v11) +ebb3(v9: i32): + v10 = iadd v9, v5 + v11 = iadd.i32 v1, v4 + brz.i32 v1, ebb2(v9) + jump ebb6(v10) -ebb4(v14: i32): - v15 = iadd v12, v2 - brz v0, ebb3(v14) - jump ebb5(v14) +ebb4(v12: i32): + v13 = iconst.i32 3 + v14 = iadd v12, v13 + v15 = iadd.i32 v4, v13 + jump ebb5(v13) ebb5(v16: i32): - v17 = iadd v16, v1 - v18 = iadd v1, v19 - brz v0, ebb0(v18) - return v17 + v17 = iadd.i32 v14, v4 + brz.i32 v1, ebb4(v16) + jump ebb6(v16) + +ebb6(v18: i32): + v19 = iadd v18, v2 + v20 = iadd.i32 v2, v3 + brz.i32 v1, ebb1(v20) + return v19 } ; sameln: function %complex -; nextln: ebb6(v20: i32): -; nextln: v1 = iconst.i32 1 -; nextln: v2 = iconst.i32 4 -; nextln: v5 = iconst.i32 2 -; nextln: v12 = iconst.i32 3 -; nextln: v19 = iadd v1, v2 -; nextln: jump ebb0(v20) -; nextln: ; nextln: ebb0(v0: i32): -; nextln: v3 = iadd.i32 v1, v0 -; nextln: v7 = iadd.i32 v5, v0 -; nextln: v10 = iadd v0, v3 -; nextln: brz v0, ebb1(v1) -; nextln: v14 = iadd v3, v12 -; nextln: jump ebb3(v3) +; nextln: v2 = iconst.i32 1 +; nextln: v3 = iconst.i32 4 +; nextln: v6 = iconst.i32 2 +; nextln: v13 = iconst.i32 3 +; nextln: v20 = iadd v2, v3 +; nextln: jump ebb1(v0) ; nextln: -; nextln: ebb1(v4: i32): -; nextln: v6 = iadd v4, v3 -; nextln: jump ebb2(v7) +; nextln: ebb1(v1: i32): +; nextln: v4 = iadd.i32 v2, v1 +; nextln: v8 = iadd.i32 v6, v1 +; nextln: v11 = iadd v1, v4 +; nextln: brz v1, ebb2(v2) +; nextln: v15 = iadd v4, v13 +; nextln: jump ebb4(v4) ; nextln: -; nextln: ebb2(v8: i32): -; nextln: v9 = iadd v8, v4 -; nextln: brz.i32 v0, ebb1(v8) -; nextln: jump ebb5(v9) +; nextln: ebb2(v5: i32): +; nextln: v7 = iadd v5, v4 +; nextln: jump ebb3(v8) ; nextln: -; nextln: ebb3(v11: i32): -; nextln: v13 = iadd v11, v12 -; nextln: jump ebb4(v12) +; nextln: ebb3(v9: i32): +; nextln: v10 = iadd v9, v5 +; nextln: brz.i32 v1, ebb2(v9) +; nextln: jump ebb6(v10) ; nextln: -; nextln: ebb4(v15: i32): -; nextln: v16 = iadd.i32 v13, v3 -; nextln: brz.i32 v0, ebb3(v15) -; nextln: jump ebb5(v15) +; nextln: ebb4(v12: i32): +; nextln: v14 = iadd v12, v13 +; nextln: jump ebb5(v13) ; nextln: -; nextln: ebb5(v17: i32): -; nextln: v18 = iadd v17, v1 -; nextln: brz.i32 v0, ebb0(v19) -; nextln: return v18 +; nextln: ebb5(v16: i32): +; nextln: v17 = iadd.i32 v14, v4 +; nextln: brz.i32 v1, ebb4(v16) +; nextln: jump ebb6(v16) +; nextln: +; nextln: ebb6(v18: i32): +; nextln: v19 = iadd v18, v2 +; nextln: brz.i32 v1, ebb1(v20) +; nextln: return v19 ; nextln: } diff --git a/cranelift/filetests/licm/nested_loops.cton b/cranelift/filetests/licm/nested_loops.cton index a32dd4b498..d839bbce77 100644 --- a/cranelift/filetests/licm/nested_loops.cton +++ b/cranelift/filetests/licm/nested_loops.cton @@ -3,50 +3,53 @@ test licm function %nested_loops(i32) -> i32 { ebb0(v0: i32): - v1 = iconst.i32 1 - v2 = iconst.i32 2 - v3 = iadd v1, v2 - v4 = isub v0, v1 - jump ebb1(v4,v4) + jump ebb1(v0) -ebb1(v10: i32,v11: i32): - brz v11, ebb2(v10) +ebb1(v1: i32): + v2 = iconst.i32 1 + v3 = iconst.i32 2 + v4 = iadd v2, v3 + v5 = isub v1, v2 + jump ebb2(v5, v5) + +ebb2(v10: i32, v11: i32): + brz v11, ebb3(v10) v12 = iconst.i32 1 - v15 = iadd v12, v4 + v15 = iadd v12, v5 v13 = isub v11, v12 - jump ebb1(v10,v13) + jump ebb2(v10,v13) -ebb2(v20: i32): - brz v20, ebb3(v20) - jump ebb0(v20) +ebb3(v20: i32): + brz v20, ebb4(v20) + jump ebb1(v20) -ebb3(v30: i32): +ebb4(v30: i32): return v30 } ; sameln:function %nested_loops(i32) -> i32 { -; nextln: ebb4(v12: i32): -; nextln: v1 = iconst.i32 1 -; nextln: v2 = iconst.i32 2 -; nextln: v3 = iadd v1, v2 -; nextln: v7 = iconst.i32 1 -; nextln: jump ebb0(v12) -; nextln: ; nextln: ebb0(v0: i32): -; nextln: v4 = isub v0, v1 -; nextln: v8 = iadd.i32 v7, v4 -; nextln: jump ebb1(v4, v4) +; nextln: v2 = iconst.i32 1 +; nextln: v3 = iconst.i32 2 +; nextln: v4 = iadd v2, v3 +; nextln: v8 = iconst.i32 1 +; nextln: jump ebb1(v0) ; nextln: -; nextln: ebb1(v5: i32, v6: i32): -; nextln: brz v6, ebb2(v5) -; nextln: v9 = isub v6, v7 -; nextln: jump ebb1(v5, v9) +; nextln: ebb1(v1: i32): +; nextln: v5 = isub v1, v2 +; nextln: v9 = iadd.i32 v8, v5 +; nextln: jump ebb2(v5, v5) ; nextln: -; nextln: ebb2(v10: i32): -; nextln: brz v10, ebb3(v10) -; nextln: jump ebb0(v10) +; nextln: ebb2(v6: i32, v7: i32): +; nextln: brz v7, ebb3(v6) +; nextln: v10 = isub v7, v8 +; nextln: jump ebb2(v6, v10) ; nextln: ; nextln: ebb3(v11: i32): -; nextln: return v11 +; nextln: brz v11, ebb4(v11) +; nextln: jump ebb1(v11) +; nextln: +; nextln: ebb4(v12: i32): +; nextln: return v12 ; nextln: } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 7454cc0849..ba57069ab6 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -597,11 +597,15 @@ mod test { #[test] fn renumbering() { let mut func = Function::new(); + let entry = func.dfg.make_ebb(); let ebb0 = func.dfg.make_ebb(); let ebb100 = func.dfg.make_ebb(); let mut cur = FuncCursor::new(&mut func); + cur.insert_ebb(entry); + cur.ins().jump(ebb0, &[]); + cur.insert_ebb(ebb0); let cond = cur.ins().iconst(I32, 0); let inst2 = cur.ins().brz(cond, ebb0, &[]); diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 196054bec3..58da610829 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -14,6 +14,7 @@ //! - The instruction format must match the opcode. //! - All result values must be created for multi-valued instructions. //! - All referenced entities must exist. (Values, EBBs, stack slots, ...) +//! - Instructions must not reference (eg. branch to) the entry block. //! //! SSA form //! @@ -352,10 +353,14 @@ impl<'a> Verifier<'a> { fn verify_ebb(&self, inst: Inst, e: Ebb) -> Result { if !self.func.dfg.ebb_is_valid(e) || !self.func.layout.is_ebb_inserted(e) { - err!(inst, "invalid ebb reference {}", e) - } else { - Ok(()) + return err!(inst, "invalid ebb reference {}", e); } + if let Some(entry_block) = self.func.layout.entry_block() { + if e == entry_block { + return err!(inst, "invalid reference to entry ebb {}", e); + } + } + Ok(()) } fn verify_sig_ref(&self, inst: Inst, s: SigRef) -> Result { From 90ed698e83be186ea9c534553dc07ff116c9f600 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 9 Oct 2017 15:24:15 -0700 Subject: [PATCH 1233/3084] Add an unreachable code elimination pass. The register allocator doesn't even try to compile unreachable EBBs, so any values defined in such blocks won't be assigned registers. Since the dominator tree already has determined which EBBs are reachable, we should just eliminate any unreachable blocks instead o trying to do something with the dead code. Not that this is not a "dead code elimination" pass which would also remove individual instructions whose results are not used. --- .../filetests/regalloc/unreachable_code.cton | 45 +++++++++++++++++++ lib/cretonne/src/context.rs | 11 +++++ lib/cretonne/src/ir/layout.rs | 26 +++++++++++ lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/unreachable_code.rs | 43 ++++++++++++++++++ 5 files changed, 126 insertions(+) create mode 100644 cranelift/filetests/regalloc/unreachable_code.cton create mode 100644 lib/cretonne/src/unreachable_code.rs diff --git a/cranelift/filetests/regalloc/unreachable_code.cton b/cranelift/filetests/regalloc/unreachable_code.cton new file mode 100644 index 0000000000..efb1271105 --- /dev/null +++ b/cranelift/filetests/regalloc/unreachable_code.cton @@ -0,0 +1,45 @@ +test compile +set is_64bit +isa intel haswell + +; This function contains unreachable blocks which trip up the register +; allocator if they don't get cleared out. +function %unreachable_blocks(i64 vmctx) -> i32 spiderwasm { +ebb0(v0: i64): + v1 = iconst.i32 0 + v2 = iconst.i32 0 + jump ebb2 + +ebb2: + jump ebb4 + +ebb4: + jump ebb2 + +; Everything below this point is unreachable. + +ebb3(v3: i32): + v5 = iadd.i32 v2, v3 + jump ebb6 + +ebb6: + jump ebb6 + +ebb7(v6: i32): + v7 = iadd.i32 v5, v6 + jump ebb8 + +ebb8: + jump ebb10 + +ebb10: + jump ebb8 + +ebb9(v8: i32): + v10 = iadd.i32 v7, v8 + jump ebb1(v10) + +ebb1(v11: i32): + return v11 +} + diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 4b62721fff..6c04a7f213 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -19,6 +19,7 @@ use legalize_function; use regalloc; use result::{CtonError, CtonResult}; use settings::{FlagsOrIsa, OptLevel}; +use unreachable_code::eliminate_unreachable_code; use verifier; use simple_gvn::do_simple_gvn; use licm::do_licm; @@ -75,6 +76,7 @@ impl Context { self.simple_gvn(isa)?; } self.compute_domtree(); + self.eliminate_unreachable_code(isa)?; self.regalloc(isa)?; self.prologue_epilogue(isa)?; self.relax_branches(isa) @@ -159,6 +161,15 @@ impl Context { self.verify_if(fisa) } + /// Perform unreachable code elimination. + pub fn eliminate_unreachable_code<'a, FOI>(&mut self, fisa: FOI) -> CtonResult + where + FOI: Into>, + { + eliminate_unreachable_code(&mut self.func, &mut self.cfg, &self.domtree); + self.verify_if(fisa) + } + /// Run the register allocator. pub fn regalloc(&mut self, isa: &TargetIsa) -> CtonResult { self.regalloc.run( diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 2b8b5b6099..1664a3ba4d 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -360,6 +360,32 @@ impl Layout { self.assign_ebb_seq(ebb); } + /// Remove `ebb` from the layout. + pub fn remove_ebb(&mut self, ebb: Ebb) { + assert!(self.is_ebb_inserted(ebb), "EBB not in the layout"); + assert!(self.first_inst(ebb).is_none(), "EBB must be empty."); + + // Clear the `ebb` node and extract links. + let prev; + let next; + { + let n = &mut self.ebbs[ebb]; + prev = n.prev; + next = n.next; + n.prev = None.into(); + n.next = None.into(); + } + // Fix up links to `ebb`. + match prev.expand() { + None => self.first_ebb = next.expand(), + Some(p) => self.ebbs[p].next = next, + } + match next.expand() { + None => self.last_ebb = prev.expand(), + Some(n) => self.ebbs[n].prev = prev, + } + } + /// Return an iterator over all EBBs in layout order. pub fn ebbs<'f>(&'f self) -> Ebbs<'f> { Ebbs { diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 8fb1c393d9..6fbc651218 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -42,4 +42,5 @@ mod scoped_hash_map; mod simple_gvn; mod stack_layout; mod topo_order; +mod unreachable_code; mod write; diff --git a/lib/cretonne/src/unreachable_code.rs b/lib/cretonne/src/unreachable_code.rs new file mode 100644 index 0000000000..f5d49c77f9 --- /dev/null +++ b/lib/cretonne/src/unreachable_code.rs @@ -0,0 +1,43 @@ +//! Unreachable code elimination. + +use cursor::{Cursor, FuncCursor}; +use dominator_tree::DominatorTree; +use flowgraph::ControlFlowGraph; +use ir; + +/// Eliminate unreachable code. +/// +/// This pass deletes whole EBBs that can't be reached from the entry block. It does not delete +/// individual instructions whose results are unused. +/// +/// The reachability analysis is performed by the dominator tree analysis. +pub fn eliminate_unreachable_code( + func: &mut ir::Function, + cfg: &mut ControlFlowGraph, + domtree: &DominatorTree, +) { + let mut pos = FuncCursor::new(func); + while let Some(ebb) = pos.next_ebb() { + if domtree.is_reachable(ebb) { + continue; + } + + dbg!("Eliminating unreachable {}", ebb); + // Move the cursor out of the way and make sure the next lop iteration goes to the right + // EBB. + pos.prev_ebb(); + + // Remove all instructions from `ebb`. + while let Some(inst) = pos.func.layout.first_inst(ebb) { + dbg!(" - {}", pos.func.dfg.display_inst(inst, None)); + pos.func.layout.remove_inst(inst); + } + + // Once the EBB is completely empty, we can update the CFG which removes it from any + // predecessor lists. + cfg.recompute_ebb(pos.func, ebb); + + // Finally, remove the EBB from the layout. + pos.func.layout.remove_ebb(ebb); + } +} From 3841552b7c3da60ffc4992912d301b465bc3f105 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 6 Oct 2017 16:16:39 -0700 Subject: [PATCH 1234/3084] Use the WasmRuntime's type list rather than keeping a separate list. --- lib/wasm/src/module_translator.rs | 12 +++++------- lib/wasm/src/runtime/dummy.rs | 4 ++++ lib/wasm/src/runtime/spec.rs | 3 +++ lib/wasm/src/sections_translator.rs | 6 ++---- 4 files changed, 14 insertions(+), 11 deletions(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 001a85bd69..2715aeb394 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -5,7 +5,7 @@ use sections_translator::{SectionParsingError, parse_function_signatures, parse_ parse_function_section, parse_export_section, parse_memory_section, parse_global_section, parse_table_section, parse_elements_section, parse_data_section}; -use translation_utils::{Import, SignatureIndex, FunctionIndex}; +use translation_utils::{Import, FunctionIndex}; use cretonne::ir::{Function, FunctionName}; use func_translator::FuncTranslator; use std::collections::HashMap; @@ -39,7 +39,6 @@ pub fn translate_module( } ref s => panic!("modules should begin properly: {:?}", s), } - let mut functions: Vec = Vec::new(); let mut globals = Vec::new(); let mut exports: HashMap = HashMap::new(); let mut next_input = ParserInput::Default; @@ -62,7 +61,6 @@ pub fn translate_module( for import in imps { match import { Import::Function { sig_index } => { - functions.push(sig_index as SignatureIndex); function_index += 1; } Import::Memory(mem) => { @@ -86,9 +84,7 @@ pub fn translate_module( } ParserState::BeginSection { code: SectionCode::Function, .. } => { match parse_function_section(&mut parser, runtime) { - Ok(funcs) => { - functions.extend(funcs); - } + Ok(()) => {} Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the function section: {}", s)) } @@ -193,7 +189,9 @@ pub fn translate_module( runtime.next_function(); // First we build the Function object with its name and signature let mut func = Function::new(); - func.signature = runtime.get_signature(functions[function_index]).clone(); + func.signature = runtime + .get_signature(runtime.get_func_type(function_index)) + .clone(); if let Some(name) = exports.get(&function_index) { func.name = FunctionName::new(name.clone()); } diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index a2f58fb709..79921b7101 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -146,6 +146,10 @@ impl WasmRuntime for DummyRuntime { self.func_types.push(sig_index); } + fn get_func_type(&self, func_index: FunctionIndex) -> SignatureIndex { + self.func_types[func_index] + } + fn declare_global(&mut self, global: Global) { self.globals.push(global); } diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 74bc27770a..5a67980d1b 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -162,6 +162,9 @@ pub trait WasmRuntime: FuncEnvironment { /// Declares the type (signature) of a local function in the module. fn declare_func_type(&mut self, sig_index: SignatureIndex); + /// Return the signature index for the given function index. + fn get_func_type(&self, func_index: FunctionIndex) -> SignatureIndex; + /// Declares a global to the runtime. fn declare_global(&mut self, global: Global); /// Declares a table to the runtime. diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 8329220a21..ea3c0e97d7 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -113,19 +113,17 @@ pub fn parse_import_section( pub fn parse_function_section( parser: &mut Parser, runtime: &mut WasmRuntime, -) -> Result, SectionParsingError> { - let mut funcs = Vec::new(); +) -> Result<(), SectionParsingError> { loop { match *parser.read() { ParserState::FunctionSectionEntry(sigindex) => { runtime.declare_func_type(sigindex as SignatureIndex); - funcs.push(sigindex as SignatureIndex); } ParserState::EndSection => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; } - Ok(funcs) + Ok(()) } /// Retrieves the names of the functions from the export section From ef5ad630c873aa69211ed29b482247ca305ac3f2 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sat, 7 Oct 2017 08:12:07 -0700 Subject: [PATCH 1235/3084] Use the WasmRuntime's global list rather than keeping a separate list. --- lib/wasm/src/module_translator.rs | 41 ++++++------------------ lib/wasm/src/runtime/dummy.rs | 5 +++ lib/wasm/src/runtime/spec.rs | 4 +++ lib/wasm/src/sections_translator.rs | 49 ++++++++++++++--------------- lib/wasm/src/translation_utils.rs | 9 ------ 5 files changed, 41 insertions(+), 67 deletions(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 2715aeb394..e3395c0b9b 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -5,7 +5,7 @@ use sections_translator::{SectionParsingError, parse_function_signatures, parse_ parse_function_section, parse_export_section, parse_memory_section, parse_global_section, parse_table_section, parse_elements_section, parse_data_section}; -use translation_utils::{Import, FunctionIndex}; +use translation_utils::FunctionIndex; use cretonne::ir::{Function, FunctionName}; use func_translator::FuncTranslator; use std::collections::HashMap; @@ -39,7 +39,6 @@ pub fn translate_module( } ref s => panic!("modules should begin properly: {:?}", s), } - let mut globals = Vec::new(); let mut exports: HashMap = HashMap::new(); let mut next_input = ParserInput::Default; let mut function_index: FunctionIndex = 0; @@ -56,26 +55,8 @@ pub fn translate_module( next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Import, .. } => { - match parse_import_section(&mut parser, runtime) { - Ok(imps) => { - for import in imps { - match import { - Import::Function { sig_index } => { - function_index += 1; - } - Import::Memory(mem) => { - runtime.declare_memory(mem); - } - Import::Global(glob) => { - runtime.declare_global(glob); - globals.push(glob); - } - Import::Table(tab) => { - runtime.declare_table(tab); - } - } - } - } + match parse_import_section(&mut parser, runtime, &mut function_index) { + Ok(()) => {} Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the import section: {}", s)) } @@ -100,12 +81,8 @@ pub fn translate_module( } } ParserState::BeginSection { code: SectionCode::Memory, .. } => { - match parse_memory_section(&mut parser) { - Ok(mems) => { - for mem in mems { - runtime.declare_memory(mem); - } - } + match parse_memory_section(&mut parser, runtime) { + Ok(()) => {} Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the memory section: {}", s)) } @@ -114,7 +91,7 @@ pub fn translate_module( } ParserState::BeginSection { code: SectionCode::Global, .. } => { match parse_global_section(&mut parser, runtime) { - Ok(mut globs) => globals.append(&mut globs), + Ok(()) => {} Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the global section: {}", s)) } @@ -144,7 +121,7 @@ pub fn translate_module( next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Element, .. } => { - match parse_elements_section(&mut parser, runtime, &globals) { + match parse_elements_section(&mut parser, runtime) { Ok(()) => (), Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the element section: {}", s)) @@ -166,7 +143,7 @@ pub fn translate_module( }) } ParserState::BeginSection { code: SectionCode::Data, .. } => { - match parse_data_section(&mut parser, runtime, &globals) { + match parse_data_section(&mut parser, runtime) { Ok(()) => (), Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the data section: {}", s)) @@ -204,7 +181,7 @@ pub fn translate_module( loop { match *parser.read() { ParserState::BeginSection { code: SectionCode::Data, .. } => { - match parse_data_section(&mut parser, runtime, &globals) { + match parse_data_section(&mut parser, runtime) { Ok(()) => (), Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the data section: {}", s)) diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 79921b7101..6ecbce6cd7 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -153,6 +153,11 @@ impl WasmRuntime for DummyRuntime { fn declare_global(&mut self, global: Global) { self.globals.push(global); } + + fn get_global(&self, global_index: GlobalIndex) -> &Global { + &self.globals[global_index] + } + fn declare_table(&mut self, _: Table) { //We do nothing } diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 5a67980d1b..8d92dd9ed3 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -167,6 +167,10 @@ pub trait WasmRuntime: FuncEnvironment { /// Declares a global to the runtime. fn declare_global(&mut self, global: Global); + + /// Return the global for the given global index. + fn get_global(&self, global_index: GlobalIndex) -> &Global; + /// Declares a table to the runtime. fn declare_table(&mut self, table: Table); /// Fills a declared table with references to functions in the module. diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index ea3c0e97d7..b6feca54e4 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -7,9 +7,8 @@ //! The special case of the initialize expressions for table elements offsets or global variables //! is handled, according to the semantics of WebAssembly, to only specific expressions that are //! interpreted on the fly. -use translation_utils::{type_to_type, Import, TableIndex, FunctionIndex, GlobalIndex, - SignatureIndex, MemoryIndex, Global, GlobalInit, Table, TableElementType, - Memory}; +use translation_utils::{type_to_type, TableIndex, FunctionIndex, GlobalIndex, SignatureIndex, + MemoryIndex, Global, GlobalInit, Table, TableElementType, Memory}; use cretonne::ir::{Signature, ArgumentType, CallConv}; use cretonne; use wasmparser::{Parser, ParserState, FuncType, ImportSectionEntryType, ExternalKind, WasmDecoder, @@ -61,8 +60,8 @@ pub fn parse_function_signatures( pub fn parse_import_section( parser: &mut Parser, runtime: &mut WasmRuntime, -) -> Result, SectionParsingError> { - let mut imports = Vec::new(); + function_index: &mut FunctionIndex, +) -> Result<(), SectionParsingError> { loop { match *parser.read() { ParserState::ImportSectionEntry { @@ -71,42 +70,42 @@ pub fn parse_import_section( field, } => { runtime.declare_func_import(sig as SignatureIndex, module, field); - imports.push(Import::Function { sig_index: sig }); + *function_index += 1; } ParserState::ImportSectionEntry { ty: ImportSectionEntryType::Memory(MemoryType { limits: ref memlimits }), .. } => { - imports.push(Import::Memory(Memory { + runtime.declare_memory(Memory { pages_count: memlimits.initial as usize, maximum: memlimits.maximum.map(|x| x as usize), - })) + }); } ParserState::ImportSectionEntry { ty: ImportSectionEntryType::Global(ref ty), .. } => { - imports.push(Import::Global(Global { + runtime.declare_global(Global { ty: type_to_type(&ty.content_type).unwrap(), mutability: ty.mutable, initializer: GlobalInit::Import(), - })); + }); } ParserState::ImportSectionEntry { ty: ImportSectionEntryType::Table(ref tab), .. } => { - imports.push(Import::Table(Table { + runtime.declare_table(Table { ty: match type_to_type(&tab.element_type) { Ok(t) => TableElementType::Val(t), Err(()) => TableElementType::Func(), }, size: tab.limits.initial as usize, maximum: tab.limits.maximum.map(|x| x as usize), - })); + }); } ParserState::EndSection => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; } - Ok(imports) + Ok(()) } /// Retrieves the correspondances between functions and signatures from the function section @@ -156,29 +155,30 @@ pub fn parse_export_section( } /// Retrieves the size and maximum fields of memories from the memory section -pub fn parse_memory_section(parser: &mut Parser) -> Result, SectionParsingError> { - let mut memories: Vec = Vec::new(); +pub fn parse_memory_section( + parser: &mut Parser, + runtime: &mut WasmRuntime, +) -> Result<(), SectionParsingError> { loop { match *parser.read() { ParserState::MemorySectionEntry(ref ty) => { - memories.push(Memory { + runtime.declare_memory(Memory { pages_count: ty.limits.initial as usize, maximum: ty.limits.maximum.map(|x| x as usize), - }) + }); } ParserState::EndSection => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; } - Ok(memories) + Ok(()) } /// Retrieves the size and maximum fields of memories from the memory section pub fn parse_global_section( parser: &mut Parser, runtime: &mut WasmRuntime, -) -> Result, SectionParsingError> { - let mut globals = Vec::new(); +) -> Result<(), SectionParsingError> { loop { let (content_type, mutability) = match *parser.read() { ParserState::BeginGlobalSectionEntry(ref ty) => (ty.content_type, ty.mutable), @@ -218,19 +218,17 @@ pub fn parse_global_section( initializer: initializer, }; runtime.declare_global(global); - globals.push(global); match *parser.read() { ParserState::EndGlobalSectionEntry => (), ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), } } - Ok(globals) + Ok(()) } pub fn parse_data_section( parser: &mut Parser, runtime: &mut WasmRuntime, - globals: &[Global], ) -> Result<(), SectionParsingError> { loop { let memory_index = match *parser.read() { @@ -254,7 +252,7 @@ pub fn parse_data_section( } } ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => { - match globals[global_index as usize].initializer { + match runtime.get_global(global_index as GlobalIndex).initializer { GlobalInit::I32Const(value) => { if value < 0 { return Err(SectionParsingError::WrongSectionContent(String::from( @@ -332,7 +330,6 @@ pub fn parse_table_section( pub fn parse_elements_section( parser: &mut Parser, runtime: &mut WasmRuntime, - globals: &[Global], ) -> Result<(), SectionParsingError> { loop { let table_index = match *parser.read() { @@ -356,7 +353,7 @@ pub fn parse_elements_section( } } ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => { - match globals[global_index as usize].initializer { + match runtime.get_global(global_index as GlobalIndex).initializer { GlobalInit::I32Const(value) => { if value < 0 { return Err(SectionParsingError::WrongSectionContent(String::from( diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index c21b580b29..1f99287b32 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -14,15 +14,6 @@ pub type MemoryIndex = usize; /// Index of a signature (imported or defined) inside the WebAssembly module. pub type SignatureIndex = usize; -/// WebAssembly import. -#[derive(Debug, Clone, Copy)] -pub enum Import { - Function { sig_index: u32 }, - Memory(Memory), - Global(Global), - Table(Table), -} - /// WebAssembly global. #[derive(Debug, Clone, Copy)] pub struct Global { From e74bc0638022b8058c581025b4b5618475fcd537 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 9 Oct 2017 08:37:04 -0700 Subject: [PATCH 1236/3084] Move start_index out of TranslationResult and into the WasmRuntime. This makes it more consistent with how all the rest of the content of a wasm module is handled. And, now TranslationResult just has a Vec of translated functions, which will make it easier to refactor further. --- lib/wasm/src/module_translator.rs | 38 +++++++---------------------- lib/wasm/src/runtime/dummy.rs | 9 +++++++ lib/wasm/src/runtime/spec.rs | 4 +++ lib/wasm/src/sections_translator.rs | 17 +++++++++++++ 4 files changed, 39 insertions(+), 29 deletions(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index e3395c0b9b..baab08b63b 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -2,9 +2,9 @@ //! to deal with each part of it. use wasmparser::{ParserState, SectionCode, ParserInput, Parser, WasmDecoder, BinaryReaderError}; use sections_translator::{SectionParsingError, parse_function_signatures, parse_import_section, - parse_function_section, parse_export_section, parse_memory_section, - parse_global_section, parse_table_section, parse_elements_section, - parse_data_section}; + parse_function_section, parse_export_section, parse_start_section, + parse_memory_section, parse_global_section, parse_table_section, + parse_elements_section, parse_data_section}; use translation_utils::FunctionIndex; use cretonne::ir::{Function, FunctionName}; use func_translator::FuncTranslator; @@ -16,11 +16,6 @@ use runtime::WasmRuntime; pub struct TranslationResult { /// The translated functions. pub functions: Vec, - /// When present, the index of the function defined as `start` of the module. - /// - /// Note that this is a WebAssembly function index and not an index into the `functions` vector - /// above. The imported functions are numbered before the local functions. - pub start_index: Option, } /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IL @@ -42,7 +37,6 @@ pub fn translate_module( let mut exports: HashMap = HashMap::new(); let mut next_input = ParserInput::Default; let mut function_index: FunctionIndex = 0; - let mut start_index: Option = None; loop { match *parser.read_with_input(next_input) { ParserState::BeginSection { code: SectionCode::Type, .. } => { @@ -108,15 +102,11 @@ pub fn translate_module( next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Start, .. } => { - match *parser.read() { - ParserState::StartSectionEntry(index) => { - start_index = Some(index as FunctionIndex) + match parse_start_section(&mut parser, runtime) { + Ok(()) => (), + Err(SectionParsingError::WrongSectionContent(s)) => { + return Err(format!("wrong content in the start section: {}", s)) } - _ => return Err(String::from("wrong content in the start section")), - } - match *parser.read() { - ParserState::EndSection => {} - _ => return Err(String::from("wrong content in the start section")), } next_input = ParserInput::Default; } @@ -136,12 +126,7 @@ pub fn translate_module( ParserState::EndSection => { next_input = ParserInput::Default; } - ParserState::EndWasm => { - return Ok(TranslationResult { - functions: Vec::new(), - start_index: None, - }) - } + ParserState::EndWasm => return Ok(TranslationResult { functions: Vec::new() }), ParserState::BeginSection { code: SectionCode::Data, .. } => { match parse_data_section(&mut parser, runtime) { Ok(()) => (), @@ -188,12 +173,7 @@ pub fn translate_module( } } } - ParserState::EndWasm => { - return Ok(TranslationResult { - functions: il_functions, - start_index, - }) - } + ParserState::EndWasm => return Ok(TranslationResult { functions: il_functions }), _ => (), } } diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 6ecbce6cd7..810decce1e 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -22,6 +22,9 @@ pub struct DummyRuntime { // Compilation setting flags. flags: settings::Flags, + + // The start function. + start_func: Option, } impl DummyRuntime { @@ -38,6 +41,7 @@ impl DummyRuntime { func_types: Vec::new(), imported_funcs: Vec::new(), flags, + start_func: None, } } } @@ -177,6 +181,11 @@ impl WasmRuntime for DummyRuntime { Ok(()) } + fn declare_start_func(&mut self, func_index: FunctionIndex) { + debug_assert!(self.start_func.is_none()); + self.start_func = Some(func_index); + } + fn begin_translation(&mut self) { // We do nothing } diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 8d92dd9ed3..8c11901f19 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -189,6 +189,10 @@ pub trait WasmRuntime: FuncEnvironment { offset: usize, data: &[u8], ) -> Result<(), String>; + + /// Declares a start function. + fn declare_start_func(&mut self, index: FunctionIndex); + /// Call this function after having declared all the runtime elements but prior to the /// function body translation. fn begin_translation(&mut self); diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index b6feca54e4..04338af3e1 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -154,6 +154,23 @@ pub fn parse_export_section( Ok(exports) } +/// Retrieves the start function index from the start section +pub fn parse_start_section( + parser: &mut Parser, + runtime: &mut WasmRuntime, +) -> Result<(), SectionParsingError> { + loop { + match *parser.read() { + ParserState::StartSectionEntry(index) => { + runtime.declare_start_func(index as FunctionIndex); + } + ParserState::EndSection => break, + ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + }; + } + Ok(()) +} + /// Retrieves the size and maximum fields of memories from the memory section pub fn parse_memory_section( parser: &mut Parser, From 2c9d03f9bd1b7faaecb7431b139ba51130cdc64d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 9 Oct 2017 11:41:01 -0700 Subject: [PATCH 1237/3084] Let the runtime provide the number of imported functions. This obviates the need to keep a separate running total of the number of functions seen. --- lib/wasm/src/module_translator.rs | 6 +++--- lib/wasm/src/runtime/dummy.rs | 4 ++++ lib/wasm/src/runtime/spec.rs | 3 +++ lib/wasm/src/sections_translator.rs | 2 -- 4 files changed, 10 insertions(+), 5 deletions(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index baab08b63b..7aee73c974 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -36,7 +36,6 @@ pub fn translate_module( } let mut exports: HashMap = HashMap::new(); let mut next_input = ParserInput::Default; - let mut function_index: FunctionIndex = 0; loop { match *parser.read_with_input(next_input) { ParserState::BeginSection { code: SectionCode::Type, .. } => { @@ -49,7 +48,7 @@ pub fn translate_module( next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Import, .. } => { - match parse_import_section(&mut parser, runtime, &mut function_index) { + match parse_import_section(&mut parser, runtime) { Ok(()) => {} Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the import section: {}", s)) @@ -139,6 +138,7 @@ pub fn translate_module( }; } // At this point we've entered the code section + let num_func_imports = runtime.get_num_func_imports(); let mut il_functions: Vec = Vec::new(); let mut trans = FuncTranslator::new(); runtime.begin_translation(); @@ -151,6 +151,7 @@ pub fn translate_module( runtime.next_function(); // First we build the Function object with its name and signature let mut func = Function::new(); + let function_index = num_func_imports + il_functions.len(); func.signature = runtime .get_signature(runtime.get_func_type(function_index)) .clone(); @@ -161,7 +162,6 @@ pub fn translate_module( .translate_from_reader(parser.create_binary_reader(), &mut func, runtime) .map_err(|e| String::from(e.description()))?; il_functions.push(func); - function_index += 1; } loop { match *parser.read() { diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 810decce1e..6c7a12c88a 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -146,6 +146,10 @@ impl WasmRuntime for DummyRuntime { self.imported_funcs.push(ir::FunctionName::new(name)); } + fn get_num_func_imports(&self) -> usize { + self.imported_funcs.len() + } + fn declare_func_type(&mut self, sig_index: SignatureIndex) { self.func_types.push(sig_index); } diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 8c11901f19..f914f642d4 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -159,6 +159,9 @@ pub trait WasmRuntime: FuncEnvironment { /// Declares a function import to the runtime. fn declare_func_import(&mut self, sig_index: SignatureIndex, module: &[u8], field: &[u8]); + /// Return the number of imported funcs. + fn get_num_func_imports(&self) -> usize; + /// Declares the type (signature) of a local function in the module. fn declare_func_type(&mut self, sig_index: SignatureIndex); diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 04338af3e1..f0d5810253 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -60,7 +60,6 @@ pub fn parse_function_signatures( pub fn parse_import_section( parser: &mut Parser, runtime: &mut WasmRuntime, - function_index: &mut FunctionIndex, ) -> Result<(), SectionParsingError> { loop { match *parser.read() { @@ -70,7 +69,6 @@ pub fn parse_import_section( field, } => { runtime.declare_func_import(sig as SignatureIndex, module, field); - *function_index += 1; } ParserState::ImportSectionEntry { ty: ImportSectionEntryType::Memory(MemoryType { limits: ref memlimits }), .. From 653e8bb56300b12baf3a81d30a2b1e0c4cd93fa0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 9 Oct 2017 13:22:06 -0700 Subject: [PATCH 1238/3084] Declare exports via the WasmRuntime. Also, redo how functions are named in the DummyRuntime. Use the FunctionName field to just encode the wasm function index rather than trying to shoehorn a printable name into it. And to make up for that, teach the wasm printer to print export names as comments next to the function definitions. This also makes the fields of DummyRuntime public, in preparation for the DummyRuntime to have a more general-purpose debugging role, as well as possibly to allow it to serve as a base for other implementations. --- cranelift/src/wasm.rs | 14 ++- lib/wasm/src/module_translator.rs | 13 +-- lib/wasm/src/runtime/dummy.rs | 153 ++++++++++++++++++---------- lib/wasm/src/runtime/spec.rs | 14 ++- lib/wasm/src/sections_translator.rs | 30 +++--- 5 files changed, 146 insertions(+), 78 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 1c853efafa..f632a2162f 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -4,7 +4,7 @@ //! IL. Can also executes the `start` function of the module by laying out the memories, globals //! and tables, then emitting the translated code with hardcoded addresses to memory. -use cton_wasm::{translate_module, DummyRuntime}; +use cton_wasm::{translate_module, DummyRuntime, WasmRuntime}; use std::path::PathBuf; use cretonne::Context; use cretonne::settings::FlagsOrIsa; @@ -113,7 +113,9 @@ fn handle_module( vprint!(flag_verbose, "Compiling... "); } terminal.reset().unwrap(); - for func in &translation.functions { + let num_func_imports = dummy_runtime.get_num_func_imports(); + for (def_index, func) in translation.functions.iter().enumerate() { + let func_index = num_func_imports + def_index; let mut context = Context::new(); context.func = func.clone(); if flag_check_translation { @@ -131,6 +133,14 @@ fn handle_module( } if flag_print { vprintln!(flag_verbose, ""); + if let Some(start_func) = dummy_runtime.start_func { + if func_index == start_func { + println!("; Selected as wasm start function"); + } + } + for export_name in &dummy_runtime.functions[func_index].export_names { + println!("; Exported as \"{}\"", export_name); + } println!("{}", context.func.display(fisa.isa)); vprintln!(flag_verbose, ""); } diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 7aee73c974..434f2b0da3 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -5,10 +5,8 @@ use sections_translator::{SectionParsingError, parse_function_signatures, parse_ parse_function_section, parse_export_section, parse_start_section, parse_memory_section, parse_global_section, parse_table_section, parse_elements_section, parse_data_section}; -use translation_utils::FunctionIndex; -use cretonne::ir::{Function, FunctionName}; +use cretonne::ir::Function; use func_translator::FuncTranslator; -use std::collections::HashMap; use std::error::Error; use runtime::WasmRuntime; @@ -34,7 +32,6 @@ pub fn translate_module( } ref s => panic!("modules should begin properly: {:?}", s), } - let mut exports: HashMap = HashMap::new(); let mut next_input = ParserInput::Default; loop { match *parser.read_with_input(next_input) { @@ -92,8 +89,8 @@ pub fn translate_module( next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Export, .. } => { - match parse_export_section(&mut parser) { - Ok(exps) => exports = exps, + match parse_export_section(&mut parser, runtime) { + Ok(()) => {} Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the export section: {}", s)) } @@ -155,9 +152,7 @@ pub fn translate_module( func.signature = runtime .get_signature(runtime.get_func_type(function_index)) .clone(); - if let Some(name) = exports.get(&function_index) { - func.name = FunctionName::new(name.clone()); - } + func.name = runtime.get_name(function_index); trans .translate_from_reader(parser.create_binary_reader(), &mut func, runtime) .map_err(|e| String::from(e.description()))?; diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 6c7a12c88a..09a985a087 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -6,25 +6,51 @@ use cretonne::ir::types::*; use cretonne::cursor::FuncCursor; use cretonne::settings; +/// A collection of names under which a given entity is exported. +pub struct Exportable { + /// A wasm entity. + pub entity: T, + + /// Names under which the entity is exported. + pub export_names: Vec, +} + +impl Exportable { + pub fn new(entity: T) -> Self { + Self { + entity, + export_names: Vec::new(), + } + } +} + /// This runtime implementation is a "naïve" one, doing essentially nothing and emitting /// placeholders when forced to. Don't try to execute code translated with this runtime, it is /// essentially here for translation debug purposes. pub struct DummyRuntime { - // Unprocessed signatures exactly as provided by `declare_signature()`. - signatures: Vec, - globals: Vec, + /// Compilation setting flags. + pub flags: settings::Flags, - // Types of functions, imported and local. - func_types: Vec, + /// Signatures as provided by `declare_signature`. + pub signatures: Vec, - // Names of imported functions. - imported_funcs: Vec, + /// Module and field names of imported functions as provided by `declare_func_import`. + pub imported_funcs: Vec<(String, String)>, - // Compilation setting flags. - flags: settings::Flags, + /// Functions, imported and local. + pub functions: Vec>, - // The start function. - start_func: Option, + /// Tables as provided by `declare_table`. + pub tables: Vec>, + + /// Memories as provided by `declare_memory`. + pub memories: Vec>, + + /// Globals as provided by `declare_global`. + pub globals: Vec>, + + /// The start function. + pub start_func: Option, } impl DummyRuntime { @@ -36,11 +62,13 @@ impl DummyRuntime { /// Allocates the runtime data structures with the given flags. pub fn with_flags(flags: settings::Flags) -> Self { Self { - signatures: Vec::new(), - globals: Vec::new(), - func_types: Vec::new(), - imported_funcs: Vec::new(), flags, + signatures: Vec::new(), + imported_funcs: Vec::new(), + functions: Vec::new(), + tables: Vec::new(), + memories: Vec::new(), + globals: Vec::new(), start_func: None, } } @@ -57,7 +85,7 @@ impl FuncEnvironment for DummyRuntime { let gv = func.create_global_var(ir::GlobalVarData::VmCtx { offset }); GlobalValue::Memory { gv, - ty: self.globals[index].ty, + ty: self.globals[index].entity.ty, } } @@ -77,16 +105,11 @@ impl FuncEnvironment for DummyRuntime { } fn make_direct_func(&mut self, func: &mut ir::Function, index: FunctionIndex) -> ir::FuncRef { - let sigidx = self.func_types[index]; + let sigidx = self.functions[index].entity; // A real implementation would probably add a `vmctx` argument. // And maybe attempt some signature de-duplication. let signature = func.import_signature(self.signatures[sigidx].clone()); - - let name = match self.imported_funcs.get(index) { - Some(name) => name.clone(), - None => ir::FunctionName::new(format!("localfunc{}", index)), - }; - + let name = self.get_name(index); func.import_function(ir::ExtFuncData { name, signature }) } @@ -123,6 +146,10 @@ impl FuncEnvironment for DummyRuntime { } impl WasmRuntime for DummyRuntime { + fn get_name(&self, func_index: FunctionIndex) -> ir::FunctionName { + ir::FunctionName::new(format!("wasm_0x{:x}", func_index)) + } + fn declare_signature(&mut self, sig: &ir::Signature) { self.signatures.push(sig.clone()); } @@ -131,19 +158,17 @@ impl WasmRuntime for DummyRuntime { &self.signatures[sig_index] } - fn declare_func_import(&mut self, sig_index: SignatureIndex, module: &[u8], field: &[u8]) { + fn declare_func_import(&mut self, sig_index: SignatureIndex, module: &str, field: &str) { assert_eq!( - self.func_types.len(), + self.functions.len(), self.imported_funcs.len(), "Imported functions must be declared first" ); - self.func_types.push(sig_index); - - let mut name = Vec::new(); - name.extend(module.iter().cloned().map(name_fold)); - name.push(b'_'); - name.extend(field.iter().cloned().map(name_fold)); - self.imported_funcs.push(ir::FunctionName::new(name)); + self.functions.push(Exportable::new(sig_index)); + self.imported_funcs.push(( + String::from(module), + String::from(field), + )); } fn get_num_func_imports(&self) -> usize { @@ -151,40 +176,69 @@ impl WasmRuntime for DummyRuntime { } fn declare_func_type(&mut self, sig_index: SignatureIndex) { - self.func_types.push(sig_index); + self.functions.push(Exportable::new(sig_index)); } fn get_func_type(&self, func_index: FunctionIndex) -> SignatureIndex { - self.func_types[func_index] + self.functions[func_index].entity } fn declare_global(&mut self, global: Global) { - self.globals.push(global); + self.globals.push(Exportable::new(global)); } fn get_global(&self, global_index: GlobalIndex) -> &Global { - &self.globals[global_index] + &self.globals[global_index].entity } - fn declare_table(&mut self, _: Table) { - //We do nothing + fn declare_table(&mut self, table: Table) { + self.tables.push(Exportable::new(table)); } - fn declare_table_elements(&mut self, _: TableIndex, _: usize, _: &[FunctionIndex]) { - //We do nothing + fn declare_table_elements( + &mut self, + _table_index: TableIndex, + _offset: usize, + _elements: &[FunctionIndex], + ) { + // We do nothing } - fn declare_memory(&mut self, _: Memory) { - //We do nothing + fn declare_memory(&mut self, memory: Memory) { + self.memories.push(Exportable::new(memory)); } fn declare_data_initialization( &mut self, - _: MemoryIndex, - _: usize, - _: &[u8], + _memory_index: MemoryIndex, + _offset: usize, + _data: &[u8], ) -> Result<(), String> { // We do nothing Ok(()) } + fn declare_func_export(&mut self, func_index: FunctionIndex, name: &str) { + self.functions[func_index].export_names.push( + String::from(name), + ); + } + + fn declare_table_export(&mut self, table_index: TableIndex, name: &str) { + self.tables[table_index].export_names.push( + String::from(name), + ); + } + + fn declare_memory_export(&mut self, memory_index: MemoryIndex, name: &str) { + self.memories[memory_index].export_names.push( + String::from(name), + ); + } + + fn declare_global_export(&mut self, global_index: GlobalIndex, name: &str) { + self.globals[global_index].export_names.push( + String::from(name), + ); + } + fn declare_start_func(&mut self, func_index: FunctionIndex) { debug_assert!(self.start_func.is_none()); self.start_func = Some(func_index); @@ -197,12 +251,3 @@ impl WasmRuntime for DummyRuntime { // We do nothing } } - -// Generate characters suitable for printable `FuncName`s. -fn name_fold(c: u8) -> u8 { - if (c as char).is_alphanumeric() { - c - } else { - b'_' - } -} diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index f914f642d4..0eb74de809 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -150,6 +150,9 @@ pub trait FuncEnvironment { /// [`translate_module`](fn.translate_module.html) function. These methods should not be called /// by the user, they are only for `cretonne-wasm` internal use. pub trait WasmRuntime: FuncEnvironment { + /// Return the name for the given function index. + fn get_name(&self, func_index: FunctionIndex) -> ir::FunctionName; + /// Declares a function signature to the runtime. fn declare_signature(&mut self, sig: &ir::Signature); @@ -157,7 +160,7 @@ pub trait WasmRuntime: FuncEnvironment { fn get_signature(&self, sig_index: SignatureIndex) -> &ir::Signature; /// Declares a function import to the runtime. - fn declare_func_import(&mut self, sig_index: SignatureIndex, module: &[u8], field: &[u8]); + fn declare_func_import(&mut self, sig_index: SignatureIndex, module: &str, field: &str); /// Return the number of imported funcs. fn get_num_func_imports(&self) -> usize; @@ -193,6 +196,15 @@ pub trait WasmRuntime: FuncEnvironment { data: &[u8], ) -> Result<(), String>; + /// Declares a function export to the runtime. + fn declare_func_export(&mut self, func_index: FunctionIndex, name: &str); + /// Declares a table export to the runtime. + fn declare_table_export(&mut self, table_index: TableIndex, name: &str); + /// Declares a memory export to the runtime. + fn declare_memory_export(&mut self, memory_index: MemoryIndex, name: &str); + /// Declares a global export to the runtime. + fn declare_global_export(&mut self, global_index: GlobalIndex, name: &str); + /// Declares a start function. fn declare_start_func(&mut self, index: FunctionIndex); diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index f0d5810253..a63bad8e05 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -14,7 +14,6 @@ use cretonne; use wasmparser::{Parser, ParserState, FuncType, ImportSectionEntryType, ExternalKind, WasmDecoder, MemoryType, Operator}; use wasmparser; -use std::collections::HashMap; use std::str::from_utf8; use runtime::WasmRuntime; @@ -68,7 +67,12 @@ pub fn parse_import_section( module, field, } => { - runtime.declare_func_import(sig as SignatureIndex, module, field); + // The input has already been validated, so we should be able to + // assume valid UTF-8 and use `from_utf8_unchecked` if performance + // becomes a concern here. + let module_name = from_utf8(module).unwrap(); + let field_name = from_utf8(field).unwrap(); + runtime.declare_func_import(sig as SignatureIndex, module_name, field_name); } ParserState::ImportSectionEntry { ty: ImportSectionEntryType::Memory(MemoryType { limits: ref memlimits }), .. @@ -126,8 +130,8 @@ pub fn parse_function_section( /// Retrieves the names of the functions from the export section pub fn parse_export_section( parser: &mut Parser, -) -> Result, SectionParsingError> { - let mut exports: HashMap = HashMap::new(); + runtime: &mut WasmRuntime, +) -> Result<(), SectionParsingError> { loop { match *parser.read() { ParserState::ExportSectionEntry { @@ -135,21 +139,23 @@ pub fn parse_export_section( ref kind, index, } => { + // The input has already been validated, so we should be able to + // assume valid UTF-8 and use `from_utf8_unchecked` if performance + // becomes a concern here. + let name = from_utf8(field).unwrap(); + let func_index = index as FunctionIndex; match *kind { - ExternalKind::Function => { - exports.insert( - index as FunctionIndex, - String::from(from_utf8(field).unwrap()), - ); - } - _ => (),//TODO: deal with other kind of exports + ExternalKind::Function => runtime.declare_func_export(func_index, name), + ExternalKind::Table => runtime.declare_table_export(func_index, name), + ExternalKind::Memory => runtime.declare_memory_export(func_index, name), + ExternalKind::Global => runtime.declare_global_export(func_index, name), } } ParserState::EndSection => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; } - Ok(exports) + Ok(()) } /// Retrieves the start function index from the start section From 8e1ba080c0203b955c354e19eb2b35af83519633 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 10 Oct 2017 09:35:10 -0700 Subject: [PATCH 1239/3084] Complete support for global init expressions. --- lib/wasm/src/runtime/dummy.rs | 5 +- lib/wasm/src/runtime/spec.rs | 4 +- lib/wasm/src/sections_translator.rs | 73 ++++++++--------------------- 3 files changed, 25 insertions(+), 57 deletions(-) diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 09a985a087..4ebbdd39d3 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -197,6 +197,7 @@ impl WasmRuntime for DummyRuntime { fn declare_table_elements( &mut self, _table_index: TableIndex, + _base: Option, _offset: usize, _elements: &[FunctionIndex], ) { @@ -208,11 +209,11 @@ impl WasmRuntime for DummyRuntime { fn declare_data_initialization( &mut self, _memory_index: MemoryIndex, + _base: Option, _offset: usize, _data: &[u8], - ) -> Result<(), String> { + ) { // We do nothing - Ok(()) } fn declare_func_export(&mut self, func_index: FunctionIndex, name: &str) { diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 0eb74de809..dc51a20d8e 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -183,6 +183,7 @@ pub trait WasmRuntime: FuncEnvironment { fn declare_table_elements( &mut self, table_index: TableIndex, + base: Option, offset: usize, elements: &[FunctionIndex], ); @@ -192,9 +193,10 @@ pub trait WasmRuntime: FuncEnvironment { fn declare_data_initialization( &mut self, memory_index: MemoryIndex, + base: Option, offset: usize, data: &[u8], - ) -> Result<(), String>; + ); /// Declares a function export to the runtime. fn declare_func_export(&mut self, func_index: FunctionIndex, name: &str); diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index a63bad8e05..5641312440 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -101,7 +101,7 @@ pub fn parse_import_section( }, size: tab.limits.initial as usize, maximum: tab.limits.maximum.map(|x| x as usize), - }); + }) } ParserState::EndSection => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), @@ -227,7 +227,6 @@ pub fn parse_global_section( GlobalInit::GlobalRef(global_index as GlobalIndex) } ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), - }; match *parser.read() { ParserState::EndInitExpressionBody => (), @@ -261,35 +260,14 @@ pub fn parse_data_section( ParserState::BeginInitExpressionBody => (), ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; - let mut offset = match *parser.read() { + let (base, offset) = match *parser.read() { ParserState::InitExpressionOperator(Operator::I32Const { value }) => { - if value < 0 { - return Err(SectionParsingError::WrongSectionContent(String::from( - "negative \ - offset value", - ))); - } else { - value as usize - } + (None, value as u32 as usize) } ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => { match runtime.get_global(global_index as GlobalIndex).initializer { - GlobalInit::I32Const(value) => { - if value < 0 { - return Err(SectionParsingError::WrongSectionContent(String::from( - "\ - negative offset value", - ))); - } else { - value as usize - } - } - GlobalInit::Import() => { - return Err(SectionParsingError::WrongSectionContent(String::from( - "\ - imported globals not supported", - ))) - } // TODO: add runtime support + GlobalInit::I32Const(value) => (None, value as u32 as usize), + GlobalInit::Import() => (Some(global_index as GlobalIndex), 0), _ => panic!("should not happen"), } } @@ -303,17 +281,20 @@ pub fn parse_data_section( ParserState::BeginDataSectionEntryBody(_) => (), ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; + let mut running_offset = offset; loop { let data = match *parser.read() { ParserState::DataSectionEntryBodyChunk(data) => data, ParserState::EndDataSectionEntryBody => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; - match runtime.declare_data_initialization(memory_index as MemoryIndex, offset, data) { - Ok(()) => (), - Err(s) => return Err(SectionParsingError::WrongSectionContent(s)), - }; - offset += data.len(); + runtime.declare_data_initialization( + memory_index as MemoryIndex, + base, + running_offset, + data, + ); + running_offset += data.len(); } match *parser.read() { ParserState::EndDataSectionEntry => (), @@ -354,7 +335,7 @@ pub fn parse_elements_section( ) -> Result<(), SectionParsingError> { loop { let table_index = match *parser.read() { - ParserState::BeginElementSectionEntry(ref table_index) => *table_index as TableIndex, + ParserState::BeginElementSectionEntry(table_index) => table_index as TableIndex, ParserState::EndSection => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; @@ -362,30 +343,14 @@ pub fn parse_elements_section( ParserState::BeginInitExpressionBody => (), ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; - let offset = match *parser.read() { + let (base, offset) = match *parser.read() { ParserState::InitExpressionOperator(Operator::I32Const { value }) => { - if value < 0 { - return Err(SectionParsingError::WrongSectionContent(String::from( - "negative \ - offset value", - ))); - } else { - value as usize - } + (None, value as u32 as usize) } ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => { match runtime.get_global(global_index as GlobalIndex).initializer { - GlobalInit::I32Const(value) => { - if value < 0 { - return Err(SectionParsingError::WrongSectionContent(String::from( - "\ - negative offset value", - ))); - } else { - value as usize - } - } - GlobalInit::Import() => 0, // TODO: add runtime support + GlobalInit::I32Const(value) => (None, value as u32 as usize), + GlobalInit::Import() => (Some(global_index as GlobalIndex), 0), _ => panic!("should not happen"), } } @@ -399,7 +364,7 @@ pub fn parse_elements_section( ParserState::ElementSectionEntryBody(ref elements) => { let elems: Vec = elements.iter().map(|&x| x as FunctionIndex).collect(); - runtime.declare_table_elements(table_index, offset, &elems) + runtime.declare_table_elements(table_index, base, offset, &elems) } ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; From d4c0c5babc2a8a941bde84f839c7a4936f0f6df9 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 10 Oct 2017 10:21:06 -0700 Subject: [PATCH 1240/3084] Rename WasmRuntime's `get_name` to `get_func_name`. --- lib/wasm/src/module_translator.rs | 2 +- lib/wasm/src/runtime/dummy.rs | 4 ++-- lib/wasm/src/runtime/spec.rs | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 434f2b0da3..3cbb11de6d 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -152,7 +152,7 @@ pub fn translate_module( func.signature = runtime .get_signature(runtime.get_func_type(function_index)) .clone(); - func.name = runtime.get_name(function_index); + func.name = runtime.get_func_name(function_index); trans .translate_from_reader(parser.create_binary_reader(), &mut func, runtime) .map_err(|e| String::from(e.description()))?; diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 4ebbdd39d3..5dd39d1871 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -109,7 +109,7 @@ impl FuncEnvironment for DummyRuntime { // A real implementation would probably add a `vmctx` argument. // And maybe attempt some signature de-duplication. let signature = func.import_signature(self.signatures[sigidx].clone()); - let name = self.get_name(index); + let name = self.get_func_name(index); func.import_function(ir::ExtFuncData { name, signature }) } @@ -146,7 +146,7 @@ impl FuncEnvironment for DummyRuntime { } impl WasmRuntime for DummyRuntime { - fn get_name(&self, func_index: FunctionIndex) -> ir::FunctionName { + fn get_func_name(&self, func_index: FunctionIndex) -> ir::FunctionName { ir::FunctionName::new(format!("wasm_0x{:x}", func_index)) } diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index dc51a20d8e..35c25d3807 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -151,7 +151,7 @@ pub trait FuncEnvironment { /// by the user, they are only for `cretonne-wasm` internal use. pub trait WasmRuntime: FuncEnvironment { /// Return the name for the given function index. - fn get_name(&self, func_index: FunctionIndex) -> ir::FunctionName; + fn get_func_name(&self, func_index: FunctionIndex) -> ir::FunctionName; /// Declares a function signature to the runtime. fn declare_signature(&mut self, sig: &ir::Signature); From 733870aee4df84ec177d1124ed1c2bbd74d42ff1 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 10 Oct 2017 15:21:29 -0700 Subject: [PATCH 1241/3084] Make FuncEnvironment independent from ModuleEnvironment (formerly WasmRuntime). This renames WasmRuntime to ModuleEnvironment, and makes several changes to allow for more flexible compilation. ModuleEnvironment no longer derives from FuncEnvironment, and no longer has the `begin_translation` and `next_translation` functions, so that independent `FuncEnvironment` instances can operate within the same module. Also, this obviates the rest of TranslationResult, as it moves processing of function bodies into the environment. The DummyEnvironment implementation gives an example of decoding the function bodies as they are parsed, however other implementation strategies are now possible. --- cranelift/src/wasm.rs | 14 +-- lib/wasm/src/func_translator.rs | 26 ++-- lib/wasm/src/lib.rs | 14 +-- lib/wasm/src/module_translator.rs | 69 ++++------- lib/wasm/src/runtime/dummy.rs | 186 +++++++++++++++++++--------- lib/wasm/src/runtime/mod.rs | 4 +- lib/wasm/src/runtime/spec.rs | 50 ++++---- lib/wasm/src/sections_translator.rs | 58 ++++----- lib/wasm/tests/testsuite.rs | 11 +- 9 files changed, 245 insertions(+), 187 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index f632a2162f..8833dc81b9 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -4,7 +4,7 @@ //! IL. Can also executes the `start` function of the module by laying out the memories, globals //! and tables, then emitting the translated code with hardcoded addresses to memory. -use cton_wasm::{translate_module, DummyRuntime, WasmRuntime}; +use cton_wasm::{translate_module, DummyEnvironment, ModuleEnvironment}; use std::path::PathBuf; use cretonne::Context; use cretonne::settings::FlagsOrIsa; @@ -98,8 +98,8 @@ fn handle_module( |err| String::from(err.description()), )?; } - let mut dummy_runtime = DummyRuntime::with_flags(fisa.flags.clone()); - let translation = translate_module(&data, &mut dummy_runtime)?; + let mut dummy_environ = DummyEnvironment::with_flags(fisa.flags.clone()); + translate_module(&data, &mut dummy_environ)?; terminal.fg(term::color::GREEN).unwrap(); vprintln!(flag_verbose, "ok"); terminal.reset().unwrap(); @@ -113,8 +113,8 @@ fn handle_module( vprint!(flag_verbose, "Compiling... "); } terminal.reset().unwrap(); - let num_func_imports = dummy_runtime.get_num_func_imports(); - for (def_index, func) in translation.functions.iter().enumerate() { + let num_func_imports = dummy_environ.get_num_func_imports(); + for (def_index, func) in dummy_environ.info.function_bodies.iter().enumerate() { let func_index = num_func_imports + def_index; let mut context = Context::new(); context.func = func.clone(); @@ -133,12 +133,12 @@ fn handle_module( } if flag_print { vprintln!(flag_verbose, ""); - if let Some(start_func) = dummy_runtime.start_func { + if let Some(start_func) = dummy_environ.info.start_func { if func_index == start_func { println!("; Selected as wasm start function"); } } - for export_name in &dummy_runtime.functions[func_index].export_names { + for export_name in &dummy_environ.info.functions[func_index].export_names { println!("; Exported as \"{}\"", export_name); } println!("{}", context.func.display(fisa.isa)); diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index a5abdf382c..b93c3a368f 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -233,7 +233,7 @@ fn cur_srcloc(reader: &BinaryReader) -> ir::SourceLoc { mod tests { use cretonne::{ir, Context}; use cretonne::ir::types::I32; - use runtime::{DummyRuntime, FuncEnvironment}; + use runtime::{DummyEnvironment, FuncEnvironment}; use super::FuncTranslator; #[test] @@ -252,7 +252,7 @@ mod tests { ]; let mut trans = FuncTranslator::new(); - let mut runtime = DummyRuntime::default(); + let runtime = DummyEnvironment::default(); let mut ctx = Context::new(); ctx.func.name = ir::FunctionName::new("small1"); @@ -263,9 +263,11 @@ mod tests { ir::ArgumentType::new(I32), ); - trans.translate(&BODY, &mut ctx.func, &mut runtime).unwrap(); + trans + .translate(&BODY, &mut ctx.func, &mut runtime.func_env()) + .unwrap(); dbg!("{}", ctx.func.display(None)); - ctx.verify(runtime.flags()).unwrap(); + ctx.verify(runtime.func_env().flags()).unwrap(); } #[test] @@ -285,7 +287,7 @@ mod tests { ]; let mut trans = FuncTranslator::new(); - let mut runtime = DummyRuntime::default(); + let runtime = DummyEnvironment::default(); let mut ctx = Context::new(); ctx.func.name = ir::FunctionName::new("small2"); @@ -296,9 +298,11 @@ mod tests { ir::ArgumentType::new(I32), ); - trans.translate(&BODY, &mut ctx.func, &mut runtime).unwrap(); + trans + .translate(&BODY, &mut ctx.func, &mut runtime.func_env()) + .unwrap(); dbg!("{}", ctx.func.display(None)); - ctx.verify(runtime.flags()).unwrap(); + ctx.verify(runtime.func_env().flags()).unwrap(); } #[test] @@ -327,7 +331,7 @@ mod tests { ]; let mut trans = FuncTranslator::new(); - let mut runtime = DummyRuntime::default(); + let runtime = DummyEnvironment::default(); let mut ctx = Context::new(); ctx.func.name = ir::FunctionName::new("infloop"); @@ -335,8 +339,10 @@ mod tests { ir::ArgumentType::new(I32), ); - trans.translate(&BODY, &mut ctx.func, &mut runtime).unwrap(); + trans + .translate(&BODY, &mut ctx.func, &mut runtime.func_env()) + .unwrap(); dbg!("{}", ctx.func.display(None)); - ctx.verify(runtime.flags()).unwrap(); + ctx.verify(runtime.func_env().flags()).unwrap(); } } diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 9d1ad548bc..5aaee81d25 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -1,12 +1,10 @@ //! Performs the translation from a wasm module in binary format to the in-memory representation //! of the Cretonne IL. More particularly, it translates the code of all the functions bodies and -//! interacts with a runtime implementing the [`WasmRuntime`](trait.WasmRuntime.html) trait to -//! deal with tables, globals and linear memory. +//! interacts with a runtime implementing the [`ModuleEnvironment`](trait.ModuleEnvironment.html) +//! trait to deal with tables, globals and linear memory. //! -//! The crate provides a `DummyRuntime` trait that will allow to translate the code of the -//! functions but will fail at execution. You should use -//! [`wasmstandalone::StandaloneRuntime`](../wasmstandalone/struct.StandaloneRuntime.html) to be -//! able to execute the translated code. +//! The crate provides a `DummyEnvironment` struct that will allow to translate the code of the +//! functions but will fail at execution. //! //! The main function of this module is [`translate_module`](fn.translate_module.html). @@ -26,7 +24,7 @@ mod state; mod translation_utils; pub use func_translator::FuncTranslator; -pub use module_translator::{translate_module, TranslationResult}; -pub use runtime::{FuncEnvironment, WasmRuntime, DummyRuntime, GlobalValue}; +pub use module_translator::translate_module; +pub use runtime::{FuncEnvironment, ModuleEnvironment, DummyEnvironment, GlobalValue}; pub use translation_utils::{FunctionIndex, GlobalIndex, TableIndex, MemoryIndex, SignatureIndex, Global, GlobalInit, Table, Memory}; diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 3cbb11de6d..4aed1e9851 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -5,25 +5,16 @@ use sections_translator::{SectionParsingError, parse_function_signatures, parse_ parse_function_section, parse_export_section, parse_start_section, parse_memory_section, parse_global_section, parse_table_section, parse_elements_section, parse_data_section}; -use cretonne::ir::Function; -use func_translator::FuncTranslator; -use std::error::Error; -use runtime::WasmRuntime; - -/// Output of the [`translate_module`](fn.translate_module.html) function. -pub struct TranslationResult { - /// The translated functions. - pub functions: Vec, -} +use runtime::ModuleEnvironment; /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IL /// [`Function`](../cretonne/ir/function/struct.Function.html). /// Returns the functions and also the mappings for imported functions and signature between the /// indexes in the wasm module and the indexes inside each functions. -pub fn translate_module( - data: &[u8], - runtime: &mut WasmRuntime, -) -> Result { +pub fn translate_module<'data>( + data: &'data [u8], + environ: &mut ModuleEnvironment, +) -> Result<(), String> { let mut parser = Parser::new(data); match *parser.read() { ParserState::BeginWasm { .. } => {} @@ -36,7 +27,7 @@ pub fn translate_module( loop { match *parser.read_with_input(next_input) { ParserState::BeginSection { code: SectionCode::Type, .. } => { - match parse_function_signatures(&mut parser, runtime) { + match parse_function_signatures(&mut parser, environ) { Ok(()) => (), Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the type section: {}", s)) @@ -45,7 +36,7 @@ pub fn translate_module( next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Import, .. } => { - match parse_import_section(&mut parser, runtime) { + match parse_import_section(&mut parser, environ) { Ok(()) => {} Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the import section: {}", s)) @@ -54,7 +45,7 @@ pub fn translate_module( next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Function, .. } => { - match parse_function_section(&mut parser, runtime) { + match parse_function_section(&mut parser, environ) { Ok(()) => {} Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the function section: {}", s)) @@ -63,7 +54,7 @@ pub fn translate_module( next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Table, .. } => { - match parse_table_section(&mut parser, runtime) { + match parse_table_section(&mut parser, environ) { Ok(()) => (), Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the table section: {}", s)) @@ -71,7 +62,7 @@ pub fn translate_module( } } ParserState::BeginSection { code: SectionCode::Memory, .. } => { - match parse_memory_section(&mut parser, runtime) { + match parse_memory_section(&mut parser, environ) { Ok(()) => {} Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the memory section: {}", s)) @@ -80,7 +71,7 @@ pub fn translate_module( next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Global, .. } => { - match parse_global_section(&mut parser, runtime) { + match parse_global_section(&mut parser, environ) { Ok(()) => {} Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the global section: {}", s)) @@ -89,7 +80,7 @@ pub fn translate_module( next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Export, .. } => { - match parse_export_section(&mut parser, runtime) { + match parse_export_section(&mut parser, environ) { Ok(()) => {} Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the export section: {}", s)) @@ -98,7 +89,7 @@ pub fn translate_module( next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Start, .. } => { - match parse_start_section(&mut parser, runtime) { + match parse_start_section(&mut parser, environ) { Ok(()) => (), Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the start section: {}", s)) @@ -107,7 +98,7 @@ pub fn translate_module( next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Element, .. } => { - match parse_elements_section(&mut parser, runtime) { + match parse_elements_section(&mut parser, environ) { Ok(()) => (), Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the element section: {}", s)) @@ -122,9 +113,9 @@ pub fn translate_module( ParserState::EndSection => { next_input = ParserInput::Default; } - ParserState::EndWasm => return Ok(TranslationResult { functions: Vec::new() }), + ParserState::EndWasm => return Ok(()), ParserState::BeginSection { code: SectionCode::Data, .. } => { - match parse_data_section(&mut parser, runtime) { + match parse_data_section(&mut parser, environ) { Ok(()) => (), Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the data section: {}", s)) @@ -135,41 +126,33 @@ pub fn translate_module( }; } // At this point we've entered the code section - let num_func_imports = runtime.get_num_func_imports(); - let mut il_functions: Vec = Vec::new(); - let mut trans = FuncTranslator::new(); - runtime.begin_translation(); loop { match *parser.read() { ParserState::BeginFunctionBody { .. } => {} ParserState::EndSection => break, _ => return Err(String::from("wrong content in code section")), } - runtime.next_function(); - // First we build the Function object with its name and signature - let mut func = Function::new(); - let function_index = num_func_imports + il_functions.len(); - func.signature = runtime - .get_signature(runtime.get_func_type(function_index)) - .clone(); - func.name = runtime.get_func_name(function_index); - trans - .translate_from_reader(parser.create_binary_reader(), &mut func, runtime) - .map_err(|e| String::from(e.description()))?; - il_functions.push(func); + let mut reader = parser.create_binary_reader(); + let size = reader.bytes_remaining(); + environ.define_function_body( + reader.read_bytes(size).map_err(|e| { + format!("at offset {}: {}", e.offset, e.message) + })?, + )?; } loop { match *parser.read() { ParserState::BeginSection { code: SectionCode::Data, .. } => { - match parse_data_section(&mut parser, runtime) { + match parse_data_section(&mut parser, environ) { Ok(()) => (), Err(SectionParsingError::WrongSectionContent(s)) => { return Err(format!("wrong content in the data section: {}", s)) } } } - ParserState::EndWasm => return Ok(TranslationResult { functions: il_functions }), + ParserState::EndWasm => break, _ => (), } } + Ok(()) } diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 5dd39d1871..e3828d6e67 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -1,10 +1,18 @@ -use runtime::{FuncEnvironment, GlobalValue, WasmRuntime}; +use runtime::{FuncEnvironment, GlobalValue, ModuleEnvironment}; use translation_utils::{Global, Memory, Table, GlobalIndex, TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; +use func_translator::FuncTranslator; use cretonne::ir::{self, InstBuilder}; use cretonne::ir::types::*; use cretonne::cursor::FuncCursor; use cretonne::settings; +use wasmparser; +use std::error::Error; + +/// Compute a `ir::FunctionName` for a given wasm function index. +fn get_func_name(func_index: FunctionIndex) -> ir::FunctionName { + ir::FunctionName::new(format!("wasm_0x{:x}", func_index)) +} /// A collection of names under which a given entity is exported. pub struct Exportable { @@ -24,10 +32,10 @@ impl Exportable { } } -/// This runtime implementation is a "naïve" one, doing essentially nothing and emitting -/// placeholders when forced to. Don't try to execute code translated with this runtime, it is -/// essentially here for translation debug purposes. -pub struct DummyRuntime { +/// The main state belonging to a `DummyEnvironment`. This is split out from +/// `DummyEnvironment` to allow it to be borrowed separately from the +/// `FuncTranslator` field. +pub struct DummyModuleInfo { /// Compilation setting flags. pub flags: settings::Flags, @@ -40,6 +48,9 @@ pub struct DummyRuntime { /// Functions, imported and local. pub functions: Vec>, + /// Function bodies. + pub function_bodies: Vec, + /// Tables as provided by `declare_table`. pub tables: Vec>, @@ -53,12 +64,7 @@ pub struct DummyRuntime { pub start_func: Option, } -impl DummyRuntime { - /// Allocates the runtime data structures with default flags. - pub fn default() -> Self { - Self::with_flags(settings::Flags::new(&settings::builder())) - } - +impl DummyModuleInfo { /// Allocates the runtime data structures with the given flags. pub fn with_flags(flags: settings::Flags) -> Self { Self { @@ -66,6 +72,7 @@ impl DummyRuntime { signatures: Vec::new(), imported_funcs: Vec::new(), functions: Vec::new(), + function_bodies: Vec::new(), tables: Vec::new(), memories: Vec::new(), globals: Vec::new(), @@ -74,9 +81,52 @@ impl DummyRuntime { } } -impl FuncEnvironment for DummyRuntime { +/// This runtime implementation is a "naïve" one, doing essentially nothing and emitting +/// placeholders when forced to. Don't try to execute code translated with this runtime, it is +/// essentially here for translation debug purposes. +pub struct DummyEnvironment { + /// Module information. + pub info: DummyModuleInfo, + + /// Function translation. + trans: FuncTranslator, +} + +impl DummyEnvironment { + /// Allocates the runtime data structures with default flags. + pub fn default() -> Self { + Self::with_flags(settings::Flags::new(&settings::builder())) + } + + /// Allocates the runtime data structures with the given flags. + pub fn with_flags(flags: settings::Flags) -> Self { + Self { + info: DummyModuleInfo::with_flags(flags), + trans: FuncTranslator::new(), + } + } + + /// Return a `DummyFuncEnvironment` for translating functions within this + /// `DummyEnvironment`. + pub fn func_env(&self) -> DummyFuncEnvironment { + DummyFuncEnvironment::new(&self.info) + } +} + +/// The FuncEnvironment implementation for use by the DummyEnvironment. +pub struct DummyFuncEnvironment<'dummy_environment> { + pub mod_info: &'dummy_environment DummyModuleInfo, +} + +impl<'dummy_environment> DummyFuncEnvironment<'dummy_environment> { + pub fn new(mod_info: &'dummy_environment DummyModuleInfo) -> Self { + Self { mod_info } + } +} + +impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environment> { fn flags(&self) -> &settings::Flags { - &self.flags + &self.mod_info.flags } fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue { @@ -85,7 +135,7 @@ impl FuncEnvironment for DummyRuntime { let gv = func.create_global_var(ir::GlobalVarData::VmCtx { offset }); GlobalValue::Memory { gv, - ty: self.globals[index].entity.ty, + ty: self.mod_info.globals[index].entity.ty, } } @@ -101,15 +151,15 @@ impl FuncEnvironment for DummyRuntime { fn make_indirect_sig(&mut self, func: &mut ir::Function, index: SignatureIndex) -> ir::SigRef { // A real implementation would probably change the calling convention and add `vmctx` and // signature index arguments. - func.import_signature(self.signatures[index].clone()) + func.import_signature(self.mod_info.signatures[index].clone()) } fn make_direct_func(&mut self, func: &mut ir::Function, index: FunctionIndex) -> ir::FuncRef { - let sigidx = self.functions[index].entity; + let sigidx = self.mod_info.functions[index].entity; // A real implementation would probably add a `vmctx` argument. // And maybe attempt some signature de-duplication. - let signature = func.import_signature(self.signatures[sigidx].clone()); - let name = self.get_func_name(index); + let signature = func.import_signature(self.mod_info.signatures[sigidx].clone()); + let name = get_func_name(index); func.import_function(ir::ExtFuncData { name, signature }) } @@ -145,54 +195,59 @@ impl FuncEnvironment for DummyRuntime { } } -impl WasmRuntime for DummyRuntime { +impl ModuleEnvironment for DummyEnvironment { fn get_func_name(&self, func_index: FunctionIndex) -> ir::FunctionName { - ir::FunctionName::new(format!("wasm_0x{:x}", func_index)) + get_func_name(func_index) } fn declare_signature(&mut self, sig: &ir::Signature) { - self.signatures.push(sig.clone()); + self.info.signatures.push(sig.clone()); } fn get_signature(&self, sig_index: SignatureIndex) -> &ir::Signature { - &self.signatures[sig_index] + &self.info.signatures[sig_index] } - fn declare_func_import(&mut self, sig_index: SignatureIndex, module: &str, field: &str) { + fn declare_func_import<'data>( + &mut self, + sig_index: SignatureIndex, + module: &'data str, + field: &'data str, + ) { assert_eq!( - self.functions.len(), - self.imported_funcs.len(), + self.info.functions.len(), + self.info.imported_funcs.len(), "Imported functions must be declared first" ); - self.functions.push(Exportable::new(sig_index)); - self.imported_funcs.push(( + self.info.functions.push(Exportable::new(sig_index)); + self.info.imported_funcs.push(( String::from(module), String::from(field), )); } fn get_num_func_imports(&self) -> usize { - self.imported_funcs.len() + self.info.imported_funcs.len() } fn declare_func_type(&mut self, sig_index: SignatureIndex) { - self.functions.push(Exportable::new(sig_index)); + self.info.functions.push(Exportable::new(sig_index)); } fn get_func_type(&self, func_index: FunctionIndex) -> SignatureIndex { - self.functions[func_index].entity + self.info.functions[func_index].entity } fn declare_global(&mut self, global: Global) { - self.globals.push(Exportable::new(global)); + self.info.globals.push(Exportable::new(global)); } fn get_global(&self, global_index: GlobalIndex) -> &Global { - &self.globals[global_index].entity + &self.info.globals[global_index].entity } fn declare_table(&mut self, table: Table) { - self.tables.push(Exportable::new(table)); + self.info.tables.push(Exportable::new(table)); } fn declare_table_elements( &mut self, @@ -204,51 +259,68 @@ impl WasmRuntime for DummyRuntime { // We do nothing } fn declare_memory(&mut self, memory: Memory) { - self.memories.push(Exportable::new(memory)); + self.info.memories.push(Exportable::new(memory)); } - fn declare_data_initialization( + fn declare_data_initialization<'data>( &mut self, _memory_index: MemoryIndex, _base: Option, _offset: usize, - _data: &[u8], + _data: &'data [u8], ) { // We do nothing } - fn declare_func_export(&mut self, func_index: FunctionIndex, name: &str) { - self.functions[func_index].export_names.push( + fn declare_func_export<'data>(&mut self, func_index: FunctionIndex, name: &'data str) { + self.info.functions[func_index].export_names.push( + String::from( + name, + ), + ); + } + + fn declare_table_export<'data>(&mut self, table_index: TableIndex, name: &'data str) { + self.info.tables[table_index].export_names.push( String::from(name), ); } - fn declare_table_export(&mut self, table_index: TableIndex, name: &str) { - self.tables[table_index].export_names.push( - String::from(name), + fn declare_memory_export<'data>(&mut self, memory_index: MemoryIndex, name: &'data str) { + self.info.memories[memory_index].export_names.push( + String::from( + name, + ), ); } - fn declare_memory_export(&mut self, memory_index: MemoryIndex, name: &str) { - self.memories[memory_index].export_names.push( - String::from(name), - ); - } - - fn declare_global_export(&mut self, global_index: GlobalIndex, name: &str) { - self.globals[global_index].export_names.push( - String::from(name), + fn declare_global_export<'data>(&mut self, global_index: GlobalIndex, name: &'data str) { + self.info.globals[global_index].export_names.push( + String::from( + name, + ), ); } fn declare_start_func(&mut self, func_index: FunctionIndex) { - debug_assert!(self.start_func.is_none()); - self.start_func = Some(func_index); + debug_assert!(self.info.start_func.is_none()); + self.info.start_func = Some(func_index); } - fn begin_translation(&mut self) { - // We do nothing - } - fn next_function(&mut self) { - // We do nothing + /// Provides the contents of a function body. + fn define_function_body<'data>(&mut self, body_bytes: &'data [u8]) -> Result<(), String> { + let function_index = self.get_num_func_imports() + self.info.function_bodies.len(); + let name = get_func_name(function_index); + let sig = self.get_signature(self.get_func_type(function_index)) + .clone(); + let mut func = ir::Function::with_name_signature(name, sig); + { + let mut func_environ = DummyFuncEnvironment::new(&self.info); + let reader = wasmparser::BinaryReader::new(body_bytes); + self.trans + .translate_from_reader(reader, &mut func, &mut func_environ) + .map_err(|e| String::from(e.description()))?; + } + self.info.function_bodies.push(func); + Ok(()) } } diff --git a/lib/wasm/src/runtime/mod.rs b/lib/wasm/src/runtime/mod.rs index 548ad6bb6b..abee71b3f8 100644 --- a/lib/wasm/src/runtime/mod.rs +++ b/lib/wasm/src/runtime/mod.rs @@ -1,5 +1,5 @@ mod spec; mod dummy; -pub use runtime::spec::{WasmRuntime, FuncEnvironment, GlobalValue}; -pub use runtime::dummy::DummyRuntime; +pub use runtime::spec::{ModuleEnvironment, FuncEnvironment, GlobalValue}; +pub use runtime::dummy::DummyEnvironment; diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 35c25d3807..bd38c8c316 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -1,5 +1,5 @@ //! All the runtime support necessary for the wasm to cretonne translation is formalized by the -//! trait `WasmRuntime`. +//! traits `FunctionEnvironment` and `ModuleEnvironment`. use cretonne::ir::{self, InstBuilder}; use cretonne::cursor::FuncCursor; use cretonne::settings::Flags; @@ -146,21 +146,26 @@ pub trait FuncEnvironment { ) -> ir::Value; } -/// An object satisfyng the `WasmRuntime` trait can be passed as argument to the +/// An object satisfyng the `ModuleEnvironment` trait can be passed as argument to the /// [`translate_module`](fn.translate_module.html) function. These methods should not be called /// by the user, they are only for `cretonne-wasm` internal use. -pub trait WasmRuntime: FuncEnvironment { +pub trait ModuleEnvironment { /// Return the name for the given function index. fn get_func_name(&self, func_index: FunctionIndex) -> ir::FunctionName; - /// Declares a function signature to the runtime. + /// Declares a function signature to the environment. fn declare_signature(&mut self, sig: &ir::Signature); /// Return the signature with the given index. fn get_signature(&self, sig_index: SignatureIndex) -> &ir::Signature; - /// Declares a function import to the runtime. - fn declare_func_import(&mut self, sig_index: SignatureIndex, module: &str, field: &str); + /// Declares a function import to the environment. + fn declare_func_import<'data>( + &mut self, + sig_index: SignatureIndex, + module: &'data str, + field: &'data str, + ); /// Return the number of imported funcs. fn get_num_func_imports(&self) -> usize; @@ -171,13 +176,13 @@ pub trait WasmRuntime: FuncEnvironment { /// Return the signature index for the given function index. fn get_func_type(&self, func_index: FunctionIndex) -> SignatureIndex; - /// Declares a global to the runtime. + /// Declares a global to the environment. fn declare_global(&mut self, global: Global); /// Return the global for the given global index. fn get_global(&self, global_index: GlobalIndex) -> &Global; - /// Declares a table to the runtime. + /// Declares a table to the environment. fn declare_table(&mut self, table: Table); /// Fills a declared table with references to functions in the module. fn declare_table_elements( @@ -187,32 +192,29 @@ pub trait WasmRuntime: FuncEnvironment { offset: usize, elements: &[FunctionIndex], ); - /// Declares a memory to the runtime + /// Declares a memory to the environment fn declare_memory(&mut self, memory: Memory); /// Fills a declared memory with bytes at module instantiation. - fn declare_data_initialization( + fn declare_data_initialization<'data>( &mut self, memory_index: MemoryIndex, base: Option, offset: usize, - data: &[u8], + data: &'data [u8], ); - /// Declares a function export to the runtime. - fn declare_func_export(&mut self, func_index: FunctionIndex, name: &str); - /// Declares a table export to the runtime. - fn declare_table_export(&mut self, table_index: TableIndex, name: &str); - /// Declares a memory export to the runtime. - fn declare_memory_export(&mut self, memory_index: MemoryIndex, name: &str); - /// Declares a global export to the runtime. - fn declare_global_export(&mut self, global_index: GlobalIndex, name: &str); + /// Declares a function export to the environment. + fn declare_func_export<'data>(&mut self, func_index: FunctionIndex, name: &'data str); + /// Declares a table export to the environment. + fn declare_table_export<'data>(&mut self, table_index: TableIndex, name: &'data str); + /// Declares a memory export to the environment. + fn declare_memory_export<'data>(&mut self, memory_index: MemoryIndex, name: &'data str); + /// Declares a global export to the environment. + fn declare_global_export<'data>(&mut self, global_index: GlobalIndex, name: &'data str); /// Declares a start function. fn declare_start_func(&mut self, index: FunctionIndex); - /// Call this function after having declared all the runtime elements but prior to the - /// function body translation. - fn begin_translation(&mut self); - /// Call this function between each function body translation. - fn next_function(&mut self); + /// Provides the contents of a function body. + fn define_function_body<'data>(&mut self, body_bytes: &'data [u8]) -> Result<(), String>; } diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 5641312440..b76734101d 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -15,7 +15,7 @@ use wasmparser::{Parser, ParserState, FuncType, ImportSectionEntryType, External MemoryType, Operator}; use wasmparser; use std::str::from_utf8; -use runtime::WasmRuntime; +use runtime::ModuleEnvironment; pub enum SectionParsingError { WrongSectionContent(String), @@ -24,7 +24,7 @@ pub enum SectionParsingError { /// Reads the Type Section of the wasm module and returns the corresponding function signatures. pub fn parse_function_signatures( parser: &mut Parser, - runtime: &mut WasmRuntime, + environ: &mut ModuleEnvironment, ) -> Result<(), SectionParsingError> { loop { match *parser.read() { @@ -47,7 +47,7 @@ pub fn parse_function_signatures( ); ArgumentType::new(cret_arg) })); - runtime.declare_signature(&sig); + environ.declare_signature(&sig); } ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), } @@ -58,7 +58,7 @@ pub fn parse_function_signatures( /// Retrieves the imports from the imports section of the binary. pub fn parse_import_section( parser: &mut Parser, - runtime: &mut WasmRuntime, + environ: &mut ModuleEnvironment, ) -> Result<(), SectionParsingError> { loop { match *parser.read() { @@ -72,12 +72,12 @@ pub fn parse_import_section( // becomes a concern here. let module_name = from_utf8(module).unwrap(); let field_name = from_utf8(field).unwrap(); - runtime.declare_func_import(sig as SignatureIndex, module_name, field_name); + environ.declare_func_import(sig as SignatureIndex, module_name, field_name); } ParserState::ImportSectionEntry { ty: ImportSectionEntryType::Memory(MemoryType { limits: ref memlimits }), .. } => { - runtime.declare_memory(Memory { + environ.declare_memory(Memory { pages_count: memlimits.initial as usize, maximum: memlimits.maximum.map(|x| x as usize), }); @@ -85,7 +85,7 @@ pub fn parse_import_section( ParserState::ImportSectionEntry { ty: ImportSectionEntryType::Global(ref ty), .. } => { - runtime.declare_global(Global { + environ.declare_global(Global { ty: type_to_type(&ty.content_type).unwrap(), mutability: ty.mutable, initializer: GlobalInit::Import(), @@ -94,7 +94,7 @@ pub fn parse_import_section( ParserState::ImportSectionEntry { ty: ImportSectionEntryType::Table(ref tab), .. } => { - runtime.declare_table(Table { + environ.declare_table(Table { ty: match type_to_type(&tab.element_type) { Ok(t) => TableElementType::Val(t), Err(()) => TableElementType::Func(), @@ -113,12 +113,12 @@ pub fn parse_import_section( /// Retrieves the correspondances between functions and signatures from the function section pub fn parse_function_section( parser: &mut Parser, - runtime: &mut WasmRuntime, + environ: &mut ModuleEnvironment, ) -> Result<(), SectionParsingError> { loop { match *parser.read() { ParserState::FunctionSectionEntry(sigindex) => { - runtime.declare_func_type(sigindex as SignatureIndex); + environ.declare_func_type(sigindex as SignatureIndex); } ParserState::EndSection => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), @@ -130,7 +130,7 @@ pub fn parse_function_section( /// Retrieves the names of the functions from the export section pub fn parse_export_section( parser: &mut Parser, - runtime: &mut WasmRuntime, + environ: &mut ModuleEnvironment, ) -> Result<(), SectionParsingError> { loop { match *parser.read() { @@ -145,10 +145,10 @@ pub fn parse_export_section( let name = from_utf8(field).unwrap(); let func_index = index as FunctionIndex; match *kind { - ExternalKind::Function => runtime.declare_func_export(func_index, name), - ExternalKind::Table => runtime.declare_table_export(func_index, name), - ExternalKind::Memory => runtime.declare_memory_export(func_index, name), - ExternalKind::Global => runtime.declare_global_export(func_index, name), + ExternalKind::Function => environ.declare_func_export(func_index, name), + ExternalKind::Table => environ.declare_table_export(func_index, name), + ExternalKind::Memory => environ.declare_memory_export(func_index, name), + ExternalKind::Global => environ.declare_global_export(func_index, name), } } ParserState::EndSection => break, @@ -161,12 +161,12 @@ pub fn parse_export_section( /// Retrieves the start function index from the start section pub fn parse_start_section( parser: &mut Parser, - runtime: &mut WasmRuntime, + environ: &mut ModuleEnvironment, ) -> Result<(), SectionParsingError> { loop { match *parser.read() { ParserState::StartSectionEntry(index) => { - runtime.declare_start_func(index as FunctionIndex); + environ.declare_start_func(index as FunctionIndex); } ParserState::EndSection => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), @@ -178,12 +178,12 @@ pub fn parse_start_section( /// Retrieves the size and maximum fields of memories from the memory section pub fn parse_memory_section( parser: &mut Parser, - runtime: &mut WasmRuntime, + environ: &mut ModuleEnvironment, ) -> Result<(), SectionParsingError> { loop { match *parser.read() { ParserState::MemorySectionEntry(ref ty) => { - runtime.declare_memory(Memory { + environ.declare_memory(Memory { pages_count: ty.limits.initial as usize, maximum: ty.limits.maximum.map(|x| x as usize), }); @@ -198,7 +198,7 @@ pub fn parse_memory_section( /// Retrieves the size and maximum fields of memories from the memory section pub fn parse_global_section( parser: &mut Parser, - runtime: &mut WasmRuntime, + environ: &mut ModuleEnvironment, ) -> Result<(), SectionParsingError> { loop { let (content_type, mutability) = match *parser.read() { @@ -237,7 +237,7 @@ pub fn parse_global_section( mutability: mutability, initializer: initializer, }; - runtime.declare_global(global); + environ.declare_global(global); match *parser.read() { ParserState::EndGlobalSectionEntry => (), ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), @@ -248,7 +248,7 @@ pub fn parse_global_section( pub fn parse_data_section( parser: &mut Parser, - runtime: &mut WasmRuntime, + environ: &mut ModuleEnvironment, ) -> Result<(), SectionParsingError> { loop { let memory_index = match *parser.read() { @@ -265,7 +265,7 @@ pub fn parse_data_section( (None, value as u32 as usize) } ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => { - match runtime.get_global(global_index as GlobalIndex).initializer { + match environ.get_global(global_index as GlobalIndex).initializer { GlobalInit::I32Const(value) => (None, value as u32 as usize), GlobalInit::Import() => (Some(global_index as GlobalIndex), 0), _ => panic!("should not happen"), @@ -288,7 +288,7 @@ pub fn parse_data_section( ParserState::EndDataSectionEntryBody => break, ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; - runtime.declare_data_initialization( + environ.declare_data_initialization( memory_index as MemoryIndex, base, running_offset, @@ -307,12 +307,12 @@ pub fn parse_data_section( /// Retrieves the tables from the table section pub fn parse_table_section( parser: &mut Parser, - runtime: &mut WasmRuntime, + environ: &mut ModuleEnvironment, ) -> Result<(), SectionParsingError> { loop { match *parser.read() { ParserState::TableSectionEntry(ref table) => { - runtime.declare_table(Table { + environ.declare_table(Table { ty: match type_to_type(&table.element_type) { Ok(t) => TableElementType::Val(t), Err(()) => TableElementType::Func(), @@ -331,7 +331,7 @@ pub fn parse_table_section( /// Retrieves the tables from the table section pub fn parse_elements_section( parser: &mut Parser, - runtime: &mut WasmRuntime, + environ: &mut ModuleEnvironment, ) -> Result<(), SectionParsingError> { loop { let table_index = match *parser.read() { @@ -348,7 +348,7 @@ pub fn parse_elements_section( (None, value as u32 as usize) } ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => { - match runtime.get_global(global_index as GlobalIndex).initializer { + match environ.get_global(global_index as GlobalIndex).initializer { GlobalInit::I32Const(value) => (None, value as u32 as usize), GlobalInit::Import() => (Some(global_index as GlobalIndex), 0), _ => panic!("should not happen"), @@ -364,7 +364,7 @@ pub fn parse_elements_section( ParserState::ElementSectionEntryBody(ref elements) => { let elems: Vec = elements.iter().map(|&x| x as FunctionIndex).collect(); - runtime.declare_table_elements(table_index, base, offset, &elems) + environ.declare_table_elements(table_index, base, offset, &elems) } ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/testsuite.rs index d992de753e..687aa7c938 100644 --- a/lib/wasm/tests/testsuite.rs +++ b/lib/wasm/tests/testsuite.rs @@ -2,7 +2,7 @@ extern crate cton_wasm; extern crate cretonne; extern crate tempdir; -use cton_wasm::{translate_module, DummyRuntime, WasmRuntime}; +use cton_wasm::{translate_module, DummyEnvironment}; use std::path::PathBuf; use std::fs::File; use std::error::Error; @@ -92,12 +92,9 @@ fn handle_module(path: PathBuf, flags: &Flags) { } } }; - let mut dummy_runtime = DummyRuntime::with_flags(flags.clone()); - let translation = { - let runtime: &mut WasmRuntime = &mut dummy_runtime; - translate_module(&data, runtime).unwrap() - }; - for func in &translation.functions { + let mut dummy_environ = DummyEnvironment::with_flags(flags.clone()); + translate_module(&data, &mut dummy_environ).unwrap(); + for func in &dummy_environ.info.function_bodies { verifier::verify_function(func, flags) .map_err(|err| panic!(pretty_verifier_error(func, None, err))) .unwrap(); From 3f30171b79361bdc85f6a878122bd80a282d3d94 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 10 Oct 2017 16:28:29 -0700 Subject: [PATCH 1242/3084] Actually disable simple_gvn and licm by default. See https://github.com/stoklund/cretonne/pull/164#discussion_r142449999 for details. --- lib/cretonne/src/context.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 6c04a7f213..e69d5e9aa9 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -69,12 +69,14 @@ impl Context { self.compute_cfg(); self.legalize(isa)?; + /* TODO: Enable additional optimization passes. if isa.flags().opt_level() == OptLevel::Best { self.compute_domtree(); self.compute_loop_analysis(); self.licm(isa)?; self.simple_gvn(isa)?; } + */ self.compute_domtree(); self.eliminate_unreachable_code(isa)?; self.regalloc(isa)?; From ac85ba0d68c6cfae5926a75ecf87ba93eea1df9a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 11 Oct 2017 11:33:02 -0700 Subject: [PATCH 1243/3084] Update to wasmparser 0.12.1. --- lib/wasm/Cargo.toml | 2 +- lib/wasm/src/code_translator.rs | 56 +++++++++++++++++++-------------- 2 files changed, 34 insertions(+), 24 deletions(-) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 97c63ff367..0aacd4b6a9 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -11,7 +11,7 @@ license = "Apache-2.0" name = "cton_wasm" [dependencies] -wasmparser = "0.11.2" +wasmparser = "0.12.1" cretonne = { path = "../cretonne" } cretonne-frontend = { path = "../frontend" } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 24eae598a2..dab7d3a08d 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -410,46 +410,46 @@ pub fn translate_operator( * The memory base address is provided by the runtime. * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not ************************************************************************************/ - Operator::I32Load8U { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I32Load8U { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Uload8, I32, builder, state, environ); } - Operator::I32Load16U { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I32Load16U { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Uload16, I32, builder, state, environ); } - Operator::I32Load8S { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I32Load8S { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Sload8, I32, builder, state, environ); } - Operator::I32Load16S { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I32Load16S { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Sload16, I32, builder, state, environ); } - Operator::I64Load8U { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load8U { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Uload8, I64, builder, state, environ); } - Operator::I64Load16U { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load16U { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Uload16, I64, builder, state, environ); } - Operator::I64Load8S { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load8S { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Sload8, I64, builder, state, environ); } - Operator::I64Load16S { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load16S { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Sload16, I64, builder, state, environ); } - Operator::I64Load32S { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load32S { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Sload32, I64, builder, state, environ); } - Operator::I64Load32U { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load32U { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Uload32, I64, builder, state, environ); } - Operator::I32Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I32Load { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Load, I32, builder, state, environ); } - Operator::F32Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::F32Load { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Load, F32, builder, state, environ); } - Operator::I64Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Load, I64, builder, state, environ); } - Operator::F64Load { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::F64Load { memarg: MemoryImmediate { flags: _, offset } } => { translate_load(offset, ir::Opcode::Load, F64, builder, state, environ); } /****************************** Store instructions *********************************** @@ -457,21 +457,21 @@ pub fn translate_operator( * The memory base address is provided by the runtime. * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not ************************************************************************************/ - Operator::I32Store { memory_immediate: MemoryImmediate { flags: _, offset } } | - Operator::I64Store { memory_immediate: MemoryImmediate { flags: _, offset } } | - Operator::F32Store { memory_immediate: MemoryImmediate { flags: _, offset } } | - Operator::F64Store { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I32Store { memarg: MemoryImmediate { flags: _, offset } } | + Operator::I64Store { memarg: MemoryImmediate { flags: _, offset } } | + Operator::F32Store { memarg: MemoryImmediate { flags: _, offset } } | + Operator::F64Store { memarg: MemoryImmediate { flags: _, offset } } => { translate_store(offset, ir::Opcode::Store, builder, state, environ); } - Operator::I32Store8 { memory_immediate: MemoryImmediate { flags: _, offset } } | - Operator::I64Store8 { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I32Store8 { memarg: MemoryImmediate { flags: _, offset } } | + Operator::I64Store8 { memarg: MemoryImmediate { flags: _, offset } } => { translate_store(offset, ir::Opcode::Istore8, builder, state, environ); } - Operator::I32Store16 { memory_immediate: MemoryImmediate { flags: _, offset } } | - Operator::I64Store16 { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I32Store16 { memarg: MemoryImmediate { flags: _, offset } } | + Operator::I64Store16 { memarg: MemoryImmediate { flags: _, offset } } => { translate_store(offset, ir::Opcode::Istore16, builder, state, environ); } - Operator::I64Store32 { memory_immediate: MemoryImmediate { flags: _, offset } } => { + Operator::I64Store32 { memarg: MemoryImmediate { flags: _, offset } } => { translate_store(offset, ir::Opcode::Istore32, builder, state, environ); } /****************************** Nullary Operators ************************************/ @@ -601,6 +601,16 @@ pub fn translate_operator( let val = state.pop1(); state.push1(builder.ins().fcvt_to_uint(I32, val)); } + Operator::I64TruncSSatF64 | + Operator::I64TruncSSatF32 | + Operator::I32TruncSSatF64 | + Operator::I32TruncSSatF32 | + Operator::I64TruncUSatF64 | + Operator::I64TruncUSatF32 | + Operator::I32TruncUSatF64 | + Operator::I32TruncUSatF32 => { + panic!("proposed saturating conversion operators not yet supported"); + } Operator::F32ReinterpretI32 => { let val = state.pop1(); state.push1(builder.ins().bitcast(F32, val)); From 1a04c4260f8c388f72116b6834a4f8a7b4671f27 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 11 Oct 2017 10:04:23 -0700 Subject: [PATCH 1244/3084] Remove an unused import to silence a compiler warning. --- lib/cretonne/src/context.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index e69d5e9aa9..ca2cd43854 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -18,7 +18,7 @@ use isa::TargetIsa; use legalize_function; use regalloc; use result::{CtonError, CtonResult}; -use settings::{FlagsOrIsa, OptLevel}; +use settings::FlagsOrIsa; use unreachable_code::eliminate_unreachable_code; use verifier; use simple_gvn::do_simple_gvn; From 699cb9895e7486ede23a065c4765357def34ab73 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 11 Oct 2017 09:32:52 -0700 Subject: [PATCH 1245/3084] Enforce a 4-byte minimum spill slot size. This is primarily for the benefit of 32-bit x86 code which can't spill 1-byte types from arbitrary registers. This makes it possible to use 32-bit writes to spill types like b1 and i8. These small types are expected to be very rare since WebAssembly doesn't have then, and we tend to push integer arithmetic to at least i32. The effect of frame sizes should be minimal. --- lib/cretonne/src/ir/stackslot.rs | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index c1ad330e22..f46f91dd74 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -6,6 +6,7 @@ use entity::{PrimaryMap, Keys}; use ir::{Type, StackSlot}; use packed_option::PackedOption; +use std::cmp; use std::fmt; use std::ops::Index; use std::str::FromStr; @@ -22,6 +23,17 @@ pub type StackSize = u32; /// The location of a stack offset relative to a stack pointer or frame pointer. pub type StackOffset = i32; +/// The minimum size of a spill slot in bytes. +/// +/// ISA implementations are allowed to assume that small types like `b1` and `i8` get a full 4-byte +/// spill slot. +const MIN_SPILL_SLOT_SIZE: StackSize = 4; + +/// Get the spill slot size to use for `ty`. +fn spill_size(ty: Type) -> StackSize { + cmp::max(MIN_SPILL_SLOT_SIZE, ty.bytes()) +} + /// The kind of a stack slot. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum StackSlotKind { @@ -220,7 +232,7 @@ impl Index for StackSlots { impl StackSlots { /// Create a new spill slot for spilling values of type `ty`. pub fn make_spill_slot(&mut self, ty: Type) -> StackSlot { - self.push(StackSlotData::new(StackSlotKind::SpillSlot, ty.bytes())) + self.push(StackSlotData::new(StackSlotKind::SpillSlot, spill_size(ty))) } /// Create a stack slot representing an incoming function argument. @@ -267,7 +279,7 @@ impl StackSlots { ty: Type, in_use: &[PackedOption], ) -> StackSlot { - let size = ty.bytes(); + let size = spill_size(ty); // Find the smallest existing slot that can fit the type. if let Some(&ss) = self.emergency From ece09f2df2b9e9fba55f5e155d76c45d9d468609 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 11 Oct 2017 10:02:43 -0700 Subject: [PATCH 1246/3084] Add encodings for spill.b1, fill.b1 etc. These spills and fills use 32-bit writes, knowing that the spill slot is minimum 4 bytes which makes it safe. Also simplify the definition of load/store encodings a bit by introducing loops. --- lib/cretonne/meta/isa/intel/encodings.py | 79 +++++++++--------------- 1 file changed, 28 insertions(+), 51 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 6d79ca726e..10d89a33b4 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -120,6 +120,7 @@ enc_i32_i64(x86.sdivmodx, r.div, 0xf7, rrr=7) enc_i32_i64(x86.udivmodx, r.div, 0xf7, rrr=6) enc_i32_i64(base.copy, r.umr, 0x89) +enc_both(base.copy.b1, r.umr, 0x89) enc_i32_i64(base.regmove, r.rmov, 0x89) enc_both(base.regmove.b1, r.rmov, 0x89) @@ -188,66 +189,42 @@ I64.enc(base.ctz.i32, *r.urm(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) # # Loads and stores. # -enc_i32_i64_ld_st(base.store, True, r.st, 0x89) -enc_i32_i64_ld_st(base.store, True, r.stDisp8, 0x89) -enc_i32_i64_ld_st(base.store, True, r.stDisp32, 0x89) +for recipe in [r.st, r.stDisp8, r.stDisp32]: + enc_i32_i64_ld_st(base.store, True, recipe, 0x89) + enc_i64(base.istore32.i64.any, recipe, 0x89) + enc_i32_i64_ld_st(base.istore16, False, recipe, 0x66, 0x89) + +# Byte stores are more complicated because the registers they can address +# depends of the presence of a REX prefix. The st*_abcd recipes fall back to +# the corresponding st* recipes when a REX prefix is applied. +for recipe in [r.st_abcd, r.stDisp8_abcd, r.stDisp32_abcd]: + enc_both(base.istore8.i32.any, recipe, 0x88) + enc_i64(base.istore8.i64.any, recipe, 0x88) enc_i32_i64(base.spill, r.spSib32, 0x89) enc_i32_i64(base.regspill, r.rsp32, 0x89) -enc_i64(base.istore32.i64.any, r.st, 0x89) -enc_i64(base.istore32.i64.any, r.stDisp8, 0x89) -enc_i64(base.istore32.i64.any, r.stDisp32, 0x89) +# Use a 32-bit write for spilling `b1` to avoid constraining the permitted +# registers. +# See MIN_SPILL_SLOT_SIZE which makes this safe. +enc_both(base.spill.b1, r.spSib32, 0x89) +enc_both(base.regspill.b1, r.rsp32, 0x89) -enc_i32_i64_ld_st(base.istore16, False, r.st, 0x66, 0x89) -enc_i32_i64_ld_st(base.istore16, False, r.stDisp8, 0x66, 0x89) -enc_i32_i64_ld_st(base.istore16, False, r.stDisp32, 0x66, 0x89) - -# Byte stores are more complicated because the registers they can address -# depends of the presence of a REX prefix -I32.enc(base.istore8.i32.any, *r.st_abcd(0x88)) -I64.enc(base.istore8.i32.any, *r.st_abcd(0x88)) -I64.enc(base.istore8.i64.any, *r.st_abcd(0x88)) -I64.enc(base.istore8.i64.any, *r.st.rex(0x88)) -I32.enc(base.istore8.i32.any, *r.stDisp8_abcd(0x88)) -I64.enc(base.istore8.i32.any, *r.stDisp8_abcd(0x88)) -I64.enc(base.istore8.i64.any, *r.stDisp8_abcd(0x88)) -I64.enc(base.istore8.i64.any, *r.stDisp8.rex(0x88)) -I32.enc(base.istore8.i32.any, *r.stDisp32_abcd(0x88)) -I64.enc(base.istore8.i32.any, *r.stDisp32_abcd(0x88)) -I64.enc(base.istore8.i64.any, *r.stDisp32_abcd(0x88)) -I64.enc(base.istore8.i64.any, *r.stDisp32.rex(0x88)) - -enc_i32_i64_ld_st(base.load, True, r.ld, 0x8b) -enc_i32_i64_ld_st(base.load, True, r.ldDisp8, 0x8b) -enc_i32_i64_ld_st(base.load, True, r.ldDisp32, 0x8b) +for recipe in [r.ld, r.ldDisp8, r.ldDisp32]: + enc_i32_i64_ld_st(base.load, True, recipe, 0x8b) + enc_i64(base.uload32.i64, recipe, 0x8b) + I64.enc(base.sload32.i64, *recipe.rex(0x63, w=1)) + enc_i32_i64_ld_st(base.uload16, True, recipe, 0x0f, 0xb7) + enc_i32_i64_ld_st(base.sload16, True, recipe, 0x0f, 0xbf) + enc_i32_i64_ld_st(base.uload8, True, recipe, 0x0f, 0xb6) + enc_i32_i64_ld_st(base.sload8, True, recipe, 0x0f, 0xbe) enc_i32_i64(base.fill, r.fiSib32, 0x8b) enc_i32_i64(base.regfill, r.rfi32, 0x8b) -enc_i64(base.uload32.i64, r.ld, 0x8b) -enc_i64(base.uload32.i64, r.ldDisp8, 0x8b) -enc_i64(base.uload32.i64, r.ldDisp32, 0x8b) - -I64.enc(base.sload32.i64, *r.ld.rex(0x63, w=1)) -I64.enc(base.sload32.i64, *r.ldDisp8.rex(0x63, w=1)) -I64.enc(base.sload32.i64, *r.ldDisp32.rex(0x63, w=1)) - -enc_i32_i64_ld_st(base.uload16, True, r.ld, 0x0f, 0xb7) -enc_i32_i64_ld_st(base.uload16, True, r.ldDisp8, 0x0f, 0xb7) -enc_i32_i64_ld_st(base.uload16, True, r.ldDisp32, 0x0f, 0xb7) - -enc_i32_i64_ld_st(base.sload16, True, r.ld, 0x0f, 0xbf) -enc_i32_i64_ld_st(base.sload16, True, r.ldDisp8, 0x0f, 0xbf) -enc_i32_i64_ld_st(base.sload16, True, r.ldDisp32, 0x0f, 0xbf) - -enc_i32_i64_ld_st(base.uload8, True, r.ld, 0x0f, 0xb6) -enc_i32_i64_ld_st(base.uload8, True, r.ldDisp8, 0x0f, 0xb6) -enc_i32_i64_ld_st(base.uload8, True, r.ldDisp32, 0x0f, 0xb6) - -enc_i32_i64_ld_st(base.sload8, True, r.ld, 0x0f, 0xbe) -enc_i32_i64_ld_st(base.sload8, True, r.ldDisp8, 0x0f, 0xbe) -enc_i32_i64_ld_st(base.sload8, True, r.ldDisp32, 0x0f, 0xbe) +# Load 32 bits from `b1` spill slots. See `spill.b1` above. +enc_both(base.fill.b1, r.fiSib32, 0x8b) +enc_both(base.regfill.b1, r.rfi32, 0x8b) # # Float loads and stores. From ba52a38597d724e081c67ca83acccb951e84f11e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 11 Oct 2017 14:18:21 -0700 Subject: [PATCH 1247/3084] Add a t8jccd_long encoding recipe for brz.b1 and brnz.b1 in 32-bit mode. The register allocator can't handle branches with constrained register operands, and the brz.b1/brnz.b1 instructions only have the t8jccd_abcd in 32-bit mode where no REX prefixes are possible. This adds a worst case encoding for those cases where a b1 value lives in a non-ABCD register. --- cranelift/filetests/isa/intel/binary32.cton | 28 +++++++++++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 6 +++++ lib/cretonne/meta/isa/intel/recipes.py | 19 ++++++++++++++ 3 files changed, 53 insertions(+) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index e777cfb359..c1a7c7c935 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -1,5 +1,6 @@ ; binary emission of 32-bit code. test binemit +set is_compressed isa intel haswell ; The binary encodings can be verified with the command: @@ -393,3 +394,30 @@ ebb1: ebb2: trap user0 ; bin: 0f 0b } + +; Special branch encodings only for I32 mode. +function %special_branches() { +ebb0: + [-,%rcx] v1 = iconst.i32 1 + [-,%rsi] v2 = iconst.i32 2 + [-,%rdi] v3 = icmp eq v1, v2 + [-,%rbx] v4 = icmp ugt v1, v2 + + ; asm: testl $0xff, %edi + ; asm: je ebb1 + brz v3, ebb1 ; bin: f7 c7 000000ff 0f 84 00000015 + ; asm: testb %bl, %bl + ; asm: je ebb1 + brz v4, ebb1 ; bin: 84 db 74 11 + ; asm: testl $0xff, %edi + ; asm: jne ebb1 + brnz v3, ebb1 ; bin: f7 c7 000000ff 0f 85 00000005 + ; asm: testb %bl, %bl + ; asm: jne ebb1 + brnz v4, ebb1 ; bin: 84 db 75 01 + + return + +ebb1: + return +} diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 10d89a33b4..6c1de71d87 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -292,6 +292,12 @@ enc_i32_i64(base.brnz, r.tjccd, 0x85) # Branch on a b1 value in a register only looks at the low 8 bits. See also # bint encodings below. +# +# Start with the worst-case encoding for I32 only. The register allocator can't +# handle a branch with an ABCD-constrained operand. +I32.enc(base.brz.b1, *r.t8jccd_long(0x84)) +I32.enc(base.brnz.b1, *r.t8jccd_long(0x85)) + enc_both(base.brz.b1, r.t8jccb_abcd, 0x74) enc_both(base.brz.b1, r.t8jccd_abcd, 0x84) enc_both(base.brnz.b1, r.t8jccb_abcd, 0x75) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index cd59cc1bbb..e65f0a8217 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -848,6 +848,25 @@ t8jccd_abcd = TailRecipe( disp4(destination, func, sink); ''') +# Worst case test-and-branch recipe for brz.b1 and brnz.b1 in 32-bit mode. +# The register allocator can't handle a branch instruction with constrained +# operands like the t8jccd_abcd above. This variant can accept the b1 opernd in +# any register, but is is larger because it uses a 32-bit test instruction with +# a 0xff immediate. +t8jccd_long = TailRecipe( + 't8jccd_long', Branch, size=5 + 6, ins=GPR, outs=(), + branch_range=32, + emit=''' + // test32 r, 0xff. + PUT_OP((bits & 0xff00) | 0xf7, rex1(in_reg0), sink); + modrm_r_bits(in_reg0, bits, sink); + sink.put4(0xff); + // Jcc instruction. + sink.put1(0x0f); + sink.put1(bits as u8); + disp4(destination, func, sink); + ''') + # Comparison that produces a `b1` result in a GPR. # # This is a macro of a `cmp` instruction followed by a `setCC` instruction. From 994af598f5595b4dd0ef0b03ec69b49fad560db0 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 10 Oct 2017 09:45:06 -0700 Subject: [PATCH 1248/3084] Avoid interference on CFG edges. Track allocatable registers both locally and globally: Add a second AllocatableSet which tracks registers allocated to global values without accounting for register diversions. Since diversions are only local to an EBB, global values must be assigned un-diverted locations that don't interfere. Handle the third "global" interference domain in the constraint solver in addition to the existing "input" and "output" domains. Extend the solver error code to indicate when a global define just can't be allocated because there are not enough available global registers. Resolve this problem by replacing the instruction's global defines with local defines that are copied into their global destinations afterwards. --- .../regalloc/global-constraints.cton | 27 ++ lib/cretonne/src/regalloc/coloring.rs | 287 ++++++++++++++---- .../src/regalloc/live_value_tracker.rs | 7 + lib/cretonne/src/regalloc/solver.rs | 147 +++++++-- 4 files changed, 378 insertions(+), 90 deletions(-) create mode 100644 cranelift/filetests/regalloc/global-constraints.cton diff --git a/cranelift/filetests/regalloc/global-constraints.cton b/cranelift/filetests/regalloc/global-constraints.cton new file mode 100644 index 0000000000..5fc63d0c36 --- /dev/null +++ b/cranelift/filetests/regalloc/global-constraints.cton @@ -0,0 +1,27 @@ +test compile +isa intel + +; This test covers the troubles when values with global live ranges are defined +; by instructions with constrained register classes. +; +; The icmp_imm instrutions write their b1 result to the ABCD register class on +; 32-bit Intel. So if we define 5 live values, they can't all fit. +function %global_constraints(i32) { +ebb0(v0: i32): + v1 = icmp_imm eq v0, 1 + v2 = icmp_imm ugt v0, 2 + v3 = icmp_imm sle v0, 3 + v4 = icmp_imm ne v0, 4 + v5 = icmp_imm sge v0, 5 + brnz v5, ebb1 + return + +ebb1: + ; Make sure v1-v5 are live in. + v10 = band v1, v2 + v11 = bor v3, v4 + v12 = bor v10, v11 + v13 = bor v12, v5 + trapnz v13, user0 + return +} diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index cdea494438..a0375bb6c4 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -45,7 +45,7 @@ use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; use ir::{Ebb, Inst, Value, Function, ValueLoc, SigRef}; -use ir::{InstBuilder, ArgumentType, ArgumentLoc}; +use ir::{InstBuilder, ArgumentType, ArgumentLoc, ValueDef}; use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; use isa::{TargetIsa, EncInfo, RecipeConstraints, OperandConstraint, ConstraintKind}; use packed_option::PackedOption; @@ -55,7 +55,8 @@ use regalloc::allocatable_set::AllocatableSet; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; use regalloc::liverange::LiveRange; -use regalloc::solver::Solver; +use regalloc::solver::{Solver, SolverError}; +use std::mem; /// Data structures for the coloring pass. @@ -156,13 +157,16 @@ impl<'a> Context<'a> { self.cur.goto_top(ebb); while let Some(inst) = self.cur.next_inst() { self.cur.use_srcloc(inst); - if let Some(constraints) = - self.encinfo.operand_constraints( - self.cur.func.encodings[inst], - ) - { - self.visit_inst(inst, constraints, tracker, &mut regs); + let enc = self.cur.func.encodings[inst]; + if let Some(constraints) = self.encinfo.operand_constraints(enc) { + if self.visit_inst(inst, constraints, tracker, &mut regs) { + self.replace_global_defines(inst, tracker); + // Restore cursor location after `replace_global_defines` moves it. + // We want to revisit the copy instructions it inserted. + self.cur.goto_inst(inst); + } } else { + // This is a ghost instruction with no encoding. let (_throughs, kills) = tracker.process_ghost(inst); self.process_ghost_kills(kills, &mut regs); } @@ -173,7 +177,7 @@ impl<'a> Context<'a> { /// Visit the `ebb` header. /// /// Initialize the set of live registers and color the arguments to `ebb`. - fn visit_ebb_header(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) -> AllocatableSet { + fn visit_ebb_header(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) -> AvailableRegs { // Reposition the live value tracker and deal with the EBB arguments. tracker.ebb_top( ebb, @@ -198,31 +202,26 @@ impl<'a> Context<'a> { /// /// Also process the EBB arguments which were colored when the first predecessor branch was /// encountered. - fn livein_regs(&self, live: &[LiveValue]) -> AllocatableSet { + fn livein_regs(&self, live: &[LiveValue]) -> AvailableRegs { // Start from the registers that are actually usable. We don't want to include any reserved // registers in the set. - let mut regs = self.usable_regs.clone(); + let mut regs = AvailableRegs::new(&self.usable_regs); for lv in live.iter().filter(|lv| !lv.is_dead) { - let value = lv.value; - let affinity = self.liveness - .get(value) - .expect("No live range for live-in") - .affinity; dbg!( "Live-in: {}:{} in {}", - value, - affinity.display(&self.reginfo), - self.cur.func.locations[value].display(&self.reginfo) + lv.value, + lv.affinity.display(&self.reginfo), + self.cur.func.locations[lv.value].display(&self.reginfo) ); - if let Affinity::Reg(rci) = affinity { + if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); - let loc = self.cur.func.locations[value]; + let loc = self.cur.func.locations[lv.value]; match loc { - ValueLoc::Reg(reg) => regs.take(rc, reg), - ValueLoc::Unassigned => panic!("Live-in {} wasn't assigned", value), + ValueLoc::Reg(reg) => regs.take(rc, reg, lv.is_local), + ValueLoc::Unassigned => panic!("Live-in {} wasn't assigned", lv.value), ValueLoc::Stack(ss) => { - panic!("Live-in {} is in {}, should be register", value, ss) + panic!("Live-in {} is in {}, should be register", lv.value, ss) } } } @@ -237,11 +236,11 @@ impl<'a> Context<'a> { /// function signature. /// /// Return the set of remaining allocatable registers after filtering out the dead arguments. - fn color_entry_args(&mut self, args: &[LiveValue]) -> AllocatableSet { + fn color_entry_args(&mut self, args: &[LiveValue]) -> AvailableRegs { let sig = &self.cur.func.signature; assert_eq!(sig.argument_types.len(), args.len()); - let mut regs = self.usable_regs.clone(); + let mut regs = AvailableRegs::new(&self.usable_regs); for (lv, abi) in args.iter().zip(&sig.argument_types) { match lv.affinity { @@ -249,7 +248,7 @@ impl<'a> Context<'a> { let rc = self.reginfo.rc(rci); if let ArgumentLoc::Reg(reg) = abi.location { if !lv.is_dead { - regs.take(rc, reg); + regs.take(rc, reg, lv.is_local); } self.cur.func.locations[lv.value] = ValueLoc::Reg(reg); } else { @@ -279,17 +278,19 @@ impl<'a> Context<'a> { /// /// Update `regs` to reflect the allocated registers after `inst`, including removing any dead /// or killed values from the set. + /// + /// Returns true when the global values defined by `inst` must be replaced by local values. fn visit_inst( &mut self, inst: Inst, constraints: &RecipeConstraints, tracker: &mut LiveValueTracker, - regs: &mut AllocatableSet, - ) { + regs: &mut AvailableRegs, + ) -> bool { dbg!( "Coloring {}\n from {}", self.cur.display_inst(inst), - regs.display(&self.reginfo) + regs.input.display(&self.reginfo), ); // EBB whose arguments should be colored to match the current branch instruction's @@ -297,7 +298,7 @@ impl<'a> Context<'a> { let mut color_dest_args = None; // Program the solver with register constraints for the input side. - self.solver.reset(regs); + self.solver.reset(®s.input); self.program_input_constraints(inst, constraints.ins); let call_sig = self.cur.func.dfg.call_signature(inst); if let Some(sig) = call_sig { @@ -352,14 +353,31 @@ impl<'a> Context<'a> { // Get rid of the killed values. for lv in kills { if let Affinity::Reg(rci) = lv.affinity { - self.solver.add_kill( + let rc = self.reginfo.rc(rci); + let reg = self.divert.reg(lv.value, &self.cur.func.locations); + dbg!( + " kill {} in {} ({} {})", lv.value, - self.reginfo.rc(rci), - self.divert.reg(lv.value, &self.cur.func.locations), + self.reginfo.display_regunit(reg), + if lv.is_local { "local" } else { "global" }, + rc ); + self.solver.add_kill(lv.value, rc, reg); + + // Update the global register set which has no diversions. + if !lv.is_local { + regs.global.free( + rc, + self.cur.func.locations[lv.value].unwrap_reg(), + ); + } } } + // This aligns with the " from" line at the top of the function. + dbg!(" glob {}", regs.global.display(&self.reginfo)); + + // Program the fixed output constraints before the general defines. This allows us to // detect conflicts between fixed outputs and tied operands where the input value hasn't // been converted to a solver variable. @@ -371,17 +389,22 @@ impl<'a> Context<'a> { } self.program_output_constraints(inst, constraints.outs, defs); + // This flag is set when the solver failed to find a solution for the global defines that + // doesn't interfere with `regs.global`. We need to rewrite all of `inst`s global defines + // as local defines followed by copies. + let mut replace_global_defines = false; + // Finally, we've fully programmed the constraint solver. // We expect a quick solution in most cases. - let mut output_regs = self.solver.quick_solve().unwrap_or_else(|rc| { - dbg!("quick_solve needs more {} regs for {}", rc, self.solver); - self.iterate_solution(throughs) + let output_regs = self.solver.quick_solve(®s.global).unwrap_or_else(|_| { + dbg!("quick_solve failed for {}", self.solver); + self.iterate_solution(throughs, ®s.global, &mut replace_global_defines) }); // The solution and/or fixed input constraints may require us to shuffle the set of live // registers around. - self.shuffle_inputs(regs); + self.shuffle_inputs(&mut regs.input); // If this is the first time we branch to `dest`, color its arguments to match the current // register state. @@ -406,20 +429,42 @@ impl<'a> Context<'a> { } } - // Update `regs` for the next instruction, remove the dead defs. + // Update `regs` for the next instruction. + regs.input = output_regs; for lv in defs { - if lv.endpoint == inst { - if let Affinity::Reg(rci) = lv.affinity { - let rc = self.reginfo.rc(rci); - let reg = self.divert.reg(lv.value, &self.cur.func.locations); - output_regs.free(rc, reg); + let loc = self.cur.func.locations[lv.value]; + dbg!( + " color {} -> {}{}", + lv.value, + loc.display(&self.reginfo), + if lv.is_local { + "" + } else if replace_global_defines { + " (global to be replaced)" + } else { + " (global)" + } + ); + + if let Affinity::Reg(rci) = lv.affinity { + let rc = self.reginfo.rc(rci); + + // Remove the dead defs. + if lv.endpoint == inst { + regs.input.free(rc, loc.unwrap_reg()); + debug_assert!(lv.is_local); + } + + // Track globals in their undiverted locations. + if !lv.is_local && !replace_global_defines { + regs.global.take(rc, loc.unwrap_reg()); } } } self.forget_diverted(kills); - *regs = output_regs; + replace_global_defines } /// Program the input-side constraints for `inst` into the constraint solver. @@ -701,7 +746,7 @@ impl<'a> Context<'a> { ConstraintKind::FixedReg(_) | ConstraintKind::Stack => continue, ConstraintKind::Reg => { - self.solver.add_def(lv.value, op.regclass); + self.solver.add_def(lv.value, op.regclass, !lv.is_local); } ConstraintKind::Tied(num) => { // Find the input operand we're tied to. @@ -711,6 +756,7 @@ impl<'a> Context<'a> { arg, op.regclass, self.divert.reg(arg, &self.cur.func.locations), + !lv.is_local, ); } } @@ -721,23 +767,35 @@ impl<'a> Context<'a> { /// /// We may need to move more registers around before a solution is possible. Use an iterative /// algorithm that adds one more variable until a solution can be found. - fn iterate_solution(&mut self, throughs: &[LiveValue]) -> AllocatableSet { + fn iterate_solution( + &mut self, + throughs: &[LiveValue], + global_regs: &AllocatableSet, + replace_global_defines: &mut bool, + ) -> AllocatableSet { // Make sure `try_add_var()` below doesn't create a variable with too loose constraints. self.program_complete_input_constraints(); loop { - dbg!("real_solve for {}", self.solver); - let rc = match self.solver.real_solve() { + match self.solver.real_solve(global_regs) { Ok(regs) => return regs, - Err(rc) => rc, + Err(SolverError::Divert(rc)) => { + // Do we have any live-through `rc` registers that are not already variables? + assert!( + self.try_add_var(rc, throughs), + "Ran out of registers in {}", + rc + ); + } + Err(SolverError::Global(value)) => { + dbg!("Not enough global registers for {}, trying as local", value); + // We'll clear the `is_global` flag on all solver variables and instead make a + // note to replace all global defines with local defines followed by a copy. + *replace_global_defines = true; + self.solver.clear_all_global_flags(); + } }; - // Do we have any live-through `rc` registers that are not already variables? - assert!( - self.try_add_var(rc, throughs), - "Ran out of registers in {}", - rc - ); } } @@ -749,7 +807,7 @@ impl<'a> Context<'a> { if let Affinity::Reg(rci) = lv.affinity { // The new variable gets to roam the whole top-level register class because it is // not actually constrained by the instruction. We just want it out of the way. - let toprc2 = self.reginfo.rc(rci); + let toprc2 = self.reginfo.toprc(rci); let reg2 = self.divert.reg(lv.value, &self.cur.func.locations); if rc.contains(reg2) && self.solver.can_add_var(lv.value, toprc2, reg2) && !self.is_live_on_outgoing_edge(lv.value) @@ -856,10 +914,82 @@ impl<'a> Context<'a> { } } + /// Replace all global values define by `inst` with local values that are then copied into the + /// global value: + /// + /// v1 = foo + /// + /// becomes: + /// + /// v20 = foo + /// v1 = copy v20 + /// + /// This is sometimes necessary when there are no global registers available that can satisfy + /// the constraints on the instruction operands. + /// + fn replace_global_defines(&mut self, inst: Inst, tracker: &mut LiveValueTracker) { + dbg!("Replacing global defs on {}", self.cur.display_inst(inst)); + + // We'll insert copies *after `inst`. Our caller will move the cursor back. + self.cur.next_inst(); + + // The tracker keeps the defs from `inst` at the end. Any dead defs have already been + // removed, so it's not obvious how many defs to process + for lv in tracker.live_mut().iter_mut().rev() { + // Keep going until we reach a value that is not defined by `inst`. + if match self.cur.func.dfg.value_def(lv.value) { + ValueDef::Res(i, _) => i != inst, + _ => true, + } + { + break; + } + if lv.is_local || !lv.affinity.is_reg() { + continue; + } + + // Now `lv.value` is globally live and defined by `inst`. Replace it with a local live + // range that is copied after `inst`. + let ty = self.cur.func.dfg.value_type(lv.value); + let local = self.cur.func.dfg.replace_result(lv.value, ty); + self.cur.ins().with_result(lv.value).copy(local); + let copy = self.cur.built_inst(); + + // Create a live range for `local: inst -> copy`. + self.liveness.create_dead(local, inst, lv.affinity); + self.liveness.extend_locally( + local, + self.cur.func.layout.pp_ebb(inst), + copy, + &self.cur.func.layout, + ); + + // Move the definition of the global `lv.value`. + self.liveness.move_def_locally(lv.value, copy); + + // Transfer the register coloring to `local`. + let loc = mem::replace(&mut self.cur.func.locations[lv.value], ValueLoc::default()); + self.cur.func.locations[local] = loc; + + // Update `lv` to reflect the new `local` live range. + lv.value = local; + lv.endpoint = copy; + lv.is_local = true; + + dbg!( + " + {} with {} in {}", + self.cur.display_inst(copy), + local, + loc.display(&self.reginfo) + ); + } + dbg!("Done: {}", self.cur.display_inst(inst)); + } + /// Process kills on a ghost instruction. /// - Forget diversions. /// - Free killed registers. - fn process_ghost_kills(&mut self, kills: &[LiveValue], regs: &mut AllocatableSet) { + fn process_ghost_kills(&mut self, kills: &[LiveValue], regs: &mut AvailableRegs) { for lv in kills { if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); @@ -867,7 +997,13 @@ impl<'a> Context<'a> { Some(loc) => loc, None => self.cur.func.locations[lv.value], }; - regs.free(rc, loc.unwrap_reg()); + regs.input.free(rc, loc.unwrap_reg()); + if !lv.is_local { + regs.global.free( + rc, + self.cur.func.locations[lv.value].unwrap_reg(), + ); + } } } } @@ -902,3 +1038,36 @@ fn program_input_abi( } } } + +/// Keep track of the set of available registers in two interference domains: all registers +/// considering diversions and global registers not considering diversions. +struct AvailableRegs { + /// The exact set of registers available on the input side of the current instruction. This + /// takes into account register diversions, and it includes both local and global live ranges. + input: AllocatableSet, + + /// Registers available for allocating globally live values. This set ignores any local values, + /// and it does not account for register diversions. + /// + /// Global values must be allocated out of this set because conflicts with other global values + /// can't be resolved with local diversions. + global: AllocatableSet, +} + +impl AvailableRegs { + /// Initialize both the input and global sets from `regs`. + pub fn new(regs: &AllocatableSet) -> AvailableRegs { + AvailableRegs { + input: regs.clone(), + global: regs.clone(), + } + } + + /// Take an un-diverted register from one or both sets. + pub fn take(&mut self, rc: RegClass, reg: RegUnit, is_local: bool) { + self.input.take(rc, reg); + if !is_local { + self.global.take(rc, reg); + } + } +} diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 26d47d4b90..3007bcec7b 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -147,6 +147,13 @@ impl LiveValueTracker { &self.live.values } + /// Get a mutable set of currently live values. + /// + /// Use with care and don't move entries around. + pub fn live_mut(&mut self) -> &mut [LiveValue] { + &mut self.live.values + } + /// Move the current position to the top of `ebb`. /// /// This depends on the stored live value set at `ebb`'s immediate dominator, so that must have diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 2a2b527561..39f6e3a83c 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -103,6 +103,7 @@ use entity::{SparseMap, SparseMapValue}; use ir::Value; use isa::{RegClass, RegUnit}; use regalloc::allocatable_set::RegSetIter; +use std::cmp; use std::fmt; use std::mem; use super::AllocatableSet; @@ -161,14 +162,14 @@ impl Variable { } } - fn new_def(value: Value, constraint: RegClass) -> Variable { + fn new_def(value: Value, constraint: RegClass, is_global: bool) -> Variable { Variable { value, constraint, from: None, is_input: false, is_output: true, - is_global: false, + is_global, domain: 0, solution: !0, } @@ -180,17 +181,27 @@ impl Variable { } /// Get an iterator over possible register choices, given the available registers on the input - /// and output sides respectively. - fn iter(&self, iregs: &AllocatableSet, oregs: &AllocatableSet) -> RegSetIter { - if self.is_input && self.is_output { - let mut r = iregs.clone(); - r.intersect(oregs); - r.iter(self.constraint) - } else if self.is_input { - iregs.iter(self.constraint) - } else { - oregs.iter(self.constraint) + /// and output sides as well as the available global register set. + fn iter( + &self, + iregs: &AllocatableSet, + oregs: &AllocatableSet, + gregs: &AllocatableSet, + ) -> RegSetIter { + if !self.is_output { + debug_assert!(!self.is_global, "Global implies output"); + debug_assert!(self.is_input, "Missing interference set"); + return iregs.iter(self.constraint); } + + let mut r = oregs.clone(); + if self.is_input { + r.intersect(iregs); + } + if self.is_global { + r.intersect(gregs); + } + r.iter(self.constraint) } } @@ -736,7 +747,7 @@ impl Solver { /// /// The output value that must have the same register as the input value is not recorded in the /// solver. - pub fn add_tied_input(&mut self, value: Value, rc: RegClass, reg: RegUnit) { + pub fn add_tied_input(&mut self, value: Value, rc: RegClass, reg: RegUnit, is_global: bool) { debug_assert!(self.inputs_done); // If a fixed assignment is tied, the `to` register is not available on the output side. @@ -750,10 +761,22 @@ impl Solver { if let Some(v) = self.vars.iter_mut().find(|v| v.value == value) { assert!(v.is_input); v.is_output = true; + v.is_global = is_global; return; } - self.regs_out.take(rc, reg); + // No variable exists for `value` because its constraints are already satisfied. + // However, if the tied output value has a global live range, we must create a variable to + // avoid global interference too. + if is_global { + let mut new_var = Variable::new_live(value, rc, reg, true); + new_var.is_global = true; + dbg!("add_tied_input: new tied-global var: {}", new_var); + self.vars.push(new_var); + self.regs_in.free(rc, reg); + } else { + self.regs_out.take(rc, reg); + } } /// Add a fixed output assignment. @@ -778,10 +801,39 @@ impl Solver { /// Add a defined output value. /// /// This is similar to `add_var`, except the value doesn't have a prior register assignment. - pub fn add_def(&mut self, value: Value, constraint: RegClass) { + pub fn add_def(&mut self, value: Value, constraint: RegClass, is_global: bool) { debug_assert!(self.inputs_done); - self.vars.push(Variable::new_def(value, constraint)); + self.vars.push( + Variable::new_def(value, constraint, is_global), + ); } + + /// Clear the `is_global` flag on all solver variables. + /// + /// This is used when there are not enough global registers available, and global defines have + /// to be replaced with local defines followed by a copy. + pub fn clear_all_global_flags(&mut self) { + for v in &mut self.vars { + v.is_global = false; + } + } +} + +/// Error reported when the solver fails to find a solution with the current constraints. +/// +/// When no solution can be found, the error indicates how constraints could be loosened to help. +pub enum SolverError { + /// There are not available registers in the given register class. + /// + /// This should be resolved by turning live-through values into variables so they can be moved + /// out of the way. + Divert(RegClass), + + /// There are insufficient available registers in the global set to assign an `is_global` + /// variable with the given value. + /// + /// This should be resolved by converting the variable to a local one. + Global(Value), } /// Interface for searching for a solution. @@ -792,8 +844,11 @@ impl Solver { /// always trivial. /// /// Returns `Ok(regs)` if a solution was found. - pub fn quick_solve(&mut self) -> Result { - self.find_solution() + pub fn quick_solve( + &mut self, + global_regs: &AllocatableSet, + ) -> Result { + self.find_solution(global_regs) } /// Try harder to find a solution. @@ -803,9 +858,21 @@ impl Solver { /// This may return an error with a register class that has run out of registers. If registers /// can be freed up in the starving class, this method can be called again after adding /// variables for the freed registers. - pub fn real_solve(&mut self) -> Result { - // TODO: Sort variables to assign smallest register classes first. - self.find_solution() + pub fn real_solve( + &mut self, + global_regs: &AllocatableSet, + ) -> Result { + // Compute domain sizes for all the variables given the current register sets. + for v in &mut self.vars { + let d = v.iter(&self.regs_in, &self.regs_out, global_regs).len(); + v.domain = cmp::min(d, u16::max_value() as usize) as u16; + } + // Solve for vars with small domains first to increase the chance of finding a solution. + // Use the value number as a tie breaker to get a stable sort. + self.vars.sort_unstable_by_key(|v| (v.domain, v.value)); + + dbg!("real_solve for {}", self); + self.find_solution(global_regs) } /// Search for a solution with the current list of variables. @@ -813,16 +880,28 @@ impl Solver { /// If a solution was found, returns `Ok(regs)` with the set of available registers on the /// output side after the solution. If no solution could be found, returns `Err(rc)` with the /// constraint register class that needs more available registers. - fn find_solution(&mut self) -> Result { + fn find_solution( + &mut self, + global_regs: &AllocatableSet, + ) -> Result { // Available registers on the input and output sides respectively. let mut iregs = self.regs_in.clone(); let mut oregs = self.regs_out.clone(); + let mut gregs = global_regs.clone(); for v in &mut self.vars { let rc = v.constraint; - let reg = match v.iter(&iregs, &oregs).next() { - None => return Err(rc), + let reg = match v.iter(&iregs, &oregs, &gregs).next() { Some(reg) => reg, + None => { + // If `v` must avoid global interference, there is not point in requesting + // live registers be diverted. We need to make it a non-global variable. + if v.is_global && gregs.iter(rc).next().is_none() { + return Err(SolverError::Global(v.value)); + } else { + return Err(SolverError::Divert(rc)); + } + } }; v.solution = reg; @@ -832,6 +911,9 @@ impl Solver { if v.is_output { oregs.take(rc, reg); } + if v.is_global { + gregs.take(rc, reg); + } } Ok(oregs) @@ -1083,6 +1165,7 @@ mod tests { let r0 = gpr.unit(0); let r1 = gpr.unit(1); let r2 = gpr.unit(2); + let gregs = AllocatableSet::new(); let mut regs = AllocatableSet::new(); let mut solver = Solver::new(); let v10 = Value::new(10); @@ -1093,7 +1176,7 @@ mod tests { solver.reset(®s); solver.reassign_in(v10, gpr, r1, r0); solver.inputs_done(); - assert!(solver.quick_solve().is_ok()); + assert!(solver.quick_solve(&gregs).is_ok()); assert_eq!(solver.schedule_moves(®s), 0); assert_eq!(solver.moves(), &[mov(v10, gpr, r1, r0)]); @@ -1103,7 +1186,7 @@ mod tests { solver.reassign_in(v10, gpr, r0, r1); solver.reassign_in(v11, gpr, r1, r2); solver.inputs_done(); - assert!(solver.quick_solve().is_ok()); + assert!(solver.quick_solve(&gregs).is_ok()); assert_eq!(solver.schedule_moves(®s), 0); assert_eq!( solver.moves(), @@ -1115,7 +1198,7 @@ mod tests { solver.reassign_in(v10, gpr, r0, r1); solver.reassign_in(v11, gpr, r1, r0); solver.inputs_done(); - assert!(solver.quick_solve().is_ok()); + assert!(solver.quick_solve(&gregs).is_ok()); assert_eq!(solver.schedule_moves(®s), 0); assert_eq!( solver.moves(), @@ -1140,6 +1223,7 @@ mod tests { let s1 = s.unit(1); let s2 = s.unit(2); let s3 = s.unit(3); + let gregs = AllocatableSet::new(); let mut regs = AllocatableSet::new(); let mut solver = Solver::new(); let v10 = Value::new(10); @@ -1154,7 +1238,7 @@ mod tests { solver.reassign_in(v11, s, s2, s0); solver.reassign_in(v12, s, s3, s1); solver.inputs_done(); - assert!(solver.quick_solve().is_ok()); + assert!(solver.quick_solve(&gregs).is_ok()); assert_eq!(solver.schedule_moves(®s), 0); assert_eq!( solver.moves(), @@ -1175,7 +1259,7 @@ mod tests { solver.reassign_in(v12, s, s1, s3); solver.reassign_in(v10, d, d1, d0); solver.inputs_done(); - assert!(solver.quick_solve().is_ok()); + assert!(solver.quick_solve(&gregs).is_ok()); assert_eq!(solver.schedule_moves(®s), 0); assert_eq!( solver.moves(), @@ -1199,6 +1283,7 @@ mod tests { let r3 = gpr.unit(3); let r4 = gpr.unit(4); let r5 = gpr.unit(5); + let gregs = AllocatableSet::new(); let mut regs = AllocatableSet::new(); let mut solver = Solver::new(); let v10 = Value::new(10); @@ -1219,7 +1304,7 @@ mod tests { solver.reassign_in(v11, gpr, r1, r2); solver.reassign_in(v12, gpr, r2, r0); solver.inputs_done(); - assert!(solver.quick_solve().is_ok()); + assert!(solver.quick_solve(&gregs).is_ok()); assert_eq!(solver.schedule_moves(®s), 1); assert_eq!( solver.moves(), @@ -1243,7 +1328,7 @@ mod tests { solver.reassign_in(v15, gpr, r5, r3); solver.inputs_done(); - assert!(solver.quick_solve().is_ok()); + assert!(solver.quick_solve(&gregs).is_ok()); // We resolve two cycles with one spill. assert_eq!(solver.schedule_moves(®s), 1); assert_eq!( From 89a24b2f13eebabf179cb80e2112b1578d3b2afd Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 12 Oct 2017 10:39:12 -0700 Subject: [PATCH 1249/3084] Rename ScalarType to LaneType. The word "scalar" is a bit vague and tends to mean "non-vector". Since we are about to add new CPU flag value types that can't appear as vector lanes, make the distinction clear: LaneType represents value types that can appear as a vector lane. Also replace the Type::is_scalar() method with an is_vector() method. --- cranelift/docs/metaref.rst | 6 +++--- lib/cretonne/meta/cdsl/types.py | 31 +++++++++++++++---------------- lib/cretonne/meta/cdsl/typevar.py | 4 ++-- lib/cretonne/meta/gen_types.py | 4 ++-- lib/cretonne/src/abi.rs | 18 ++++++++---------- lib/cretonne/src/ir/types.rs | 24 ++++++++++++------------ lib/cretonne/src/isa/intel/abi.rs | 2 +- lib/cretonne/src/isa/riscv/abi.rs | 2 +- lib/cretonne/src/legalizer/mod.rs | 2 +- 9 files changed, 45 insertions(+), 48 deletions(-) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 05050f8f40..5548311cac 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -166,9 +166,9 @@ Concrete value types are represented as instances of :class:`ValueType`. There are subclasses to represent scalar and vector types. .. autoclass:: ValueType -.. inheritance-diagram:: ValueType ScalarType VectorType IntType FloatType BoolType +.. inheritance-diagram:: ValueType LaneType VectorType IntType FloatType BoolType :parts: 1 -.. autoclass:: ScalarType +.. autoclass:: LaneType :members: .. autoclass:: VectorType :members: @@ -183,7 +183,7 @@ are subclasses to represent scalar and vector types. :members: There are no predefined vector types, but they can be created as needed with -the :func:`ScalarType.by` function. +the :func:`LaneType.by` function. .. module:: cdsl.operands diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py index 8ae66ee359..85bf164c62 100644 --- a/lib/cretonne/meta/cdsl/types.py +++ b/lib/cretonne/meta/cdsl/types.py @@ -9,7 +9,7 @@ except ImportError: pass -# ValueType instances (i8, i32, ...) are provided in the cretonne.types module. +# ValueType instances (i8, i32, ...) are provided in the `base.types` module. class ValueType(object): """ A concrete SSA value type. @@ -21,8 +21,8 @@ class ValueType(object): # Map name -> ValueType. _registry = dict() # type: Dict[str, ValueType] - # List of all the scalar types. - all_scalars = list() # type: List[ScalarType] + # List of all the lane types. + all_lane_types = list() # type: List[LaneType] def __init__(self, name, membytes, doc): # type: (str, int, str) -> None @@ -75,9 +75,9 @@ class ValueType(object): self.lane_bits() >= other.lane_bits()) -class ScalarType(ValueType): +class LaneType(ValueType): """ - A concrete scalar (not vector) type. + A concrete scalar type that can appear as a vector lane too. Also tracks a unique set of :py:class:`VectorType` instances with this type as the lane type. @@ -85,16 +85,16 @@ class ScalarType(ValueType): def __init__(self, name, membytes, doc): # type: (str, int, str) -> None - super(ScalarType, self).__init__(name, membytes, doc) + super(LaneType, self).__init__(name, membytes, doc) self._vectors = dict() # type: Dict[int, VectorType] # Assign numbers starting from 1. (0 is VOID). - ValueType.all_scalars.append(self) - self.number = len(ValueType.all_scalars) - assert self.number < 16, 'Too many scalar types' + ValueType.all_lane_types.append(self) + self.number = len(ValueType.all_lane_types) + assert self.number < 16, 'Too many lane types' def __repr__(self): # type: () -> str - return 'ScalarType({})'.format(self.name) + return 'LaneType({})'.format(self.name) def by(self, lanes): # type: (int) -> VectorType @@ -120,13 +120,12 @@ class VectorType(ValueType): """ A concrete SIMD vector type. - A vector type has a lane type which is an instance of :class:`ScalarType`, + A vector type has a lane type which is an instance of :class:`LaneType`, and a positive number of lanes. """ def __init__(self, base, lanes): - # type: (ScalarType, int) -> None - assert isinstance(base, ScalarType), 'SIMD lanes must be scalar types' + # type: (LaneType, int) -> None super(VectorType, self).__init__( name='{}x{}'.format(base.name, lanes), membytes=lanes*base.membytes, @@ -153,7 +152,7 @@ class VectorType(ValueType): return self.base.lane_bits() -class IntType(ScalarType): +class IntType(LaneType): """A concrete scalar integer type.""" def __init__(self, bits): @@ -184,7 +183,7 @@ class IntType(ScalarType): return self.bits -class FloatType(ScalarType): +class FloatType(LaneType): """A concrete scalar floating point type.""" def __init__(self, bits, doc): @@ -215,7 +214,7 @@ class FloatType(ScalarType): return self.bits -class BoolType(ScalarType): +class BoolType(LaneType): """A concrete scalar boolean type.""" def __init__(self, bits): diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 988d34aaf8..0186ede5da 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -487,7 +487,7 @@ class TypeSet(object): def concrete_types(self): # type: () -> Iterable[types.ValueType] def by(scalar, lanes): - # type: (types.ScalarType, int) -> types.ValueType + # type: (types.LaneType, int) -> types.ValueType if (lanes == 1): return scalar else: @@ -577,7 +577,7 @@ class TypeVar(object): if isinstance(typ, types.VectorType): scalar = typ.base lanes = (typ.lanes, typ.lanes) - elif isinstance(typ, types.ScalarType): + elif isinstance(typ, types.LaneType): scalar = typ lanes = (1, 1) else: diff --git a/lib/cretonne/meta/gen_types.py b/lib/cretonne/meta/gen_types.py index fa033c5bdb..42ccc1bfe7 100644 --- a/lib/cretonne/meta/gen_types.py +++ b/lib/cretonne/meta/gen_types.py @@ -36,7 +36,7 @@ def emit_vectors(bits, fmt): Emit definition for all vector types with `bits` total size. """ size = bits // 8 - for ty in ValueType.all_scalars: + for ty in ValueType.all_lane_types: mb = ty.membytes if mb == 0 or mb >= size: continue @@ -45,7 +45,7 @@ def emit_vectors(bits, fmt): def emit_types(fmt): # type: (srcgen.Formatter) -> None - for ty in ValueType.all_scalars: + for ty in ValueType.all_lane_types: emit_type(ty, fmt) # Emit vector definitions for common SIMD sizes. emit_vectors(64, fmt) diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index 38bc6781d1..6728f57691 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -164,21 +164,19 @@ pub fn legalize_abi_value(have: Type, arg: &ArgumentType) -> ValueConversion { Ordering::Equal => { // This must be an integer vector that is split and then extended. assert!(arg.value_type.is_int()); - assert!(!have.is_scalar()); + assert!(have.is_vector()); ValueConversion::VectorSplit } // We have more bits than the argument. Ordering::Greater => { - if have.is_scalar() { - if have.is_float() { - // Convert a float to int so it can be split the next time. - // ARM would do this to pass an `f64` in two registers. - ValueConversion::IntBits - } else { - ValueConversion::IntSplit - } - } else { + if have.is_vector() { ValueConversion::VectorSplit + } else if have.is_float() { + // Convert a float to int so it can be split the next time. + // ARM would do this to pass an `f64` in two registers. + ValueConversion::IntBits + } else { + ValueConversion::IntSplit } } } diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 5b6f0fe7fb..3c0221d4cd 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -38,7 +38,7 @@ include!(concat!(env!("OUT_DIR"), "/types.rs")); impl Type { /// Get the lane type of this SIMD vector type. /// - /// A scalar type is the same as a SIMD vector type with one lane, so it returns itself. + /// A lane type is the same as a SIMD vector type with one lane, so it returns itself. pub fn lane_type(self) -> Type { Type(self.0 & 0x0f) } @@ -100,7 +100,7 @@ impl Type { /// /// Scalar types are all converted to `b1` which is usually what you want. pub fn as_bool(self) -> Type { - if self.is_scalar() { + if !self.is_vector() { B1 } else { self.as_bool_pedantic() @@ -173,16 +173,16 @@ impl Type { /// All SIMD types have a lane count that is a power of two and no larger than 256, so this /// will be a number in the range 0-8. /// - /// A scalar type is the same as a SIMD vector type with one lane, so it return 0. + /// A scalar type is the same as a SIMD vector type with one lane, so it returns 0. pub fn log2_lane_count(self) -> u8 { self.0 >> 4 } - /// Is this a scalar type? (That is, not a SIMD vector type). + /// Is this a SIMD vector type? /// - /// A scalar type is the same as a SIMD vector type with one lane. - pub fn is_scalar(self) -> bool { - self.log2_lane_count() == 0 + /// A vector type has 2 or more lanes. + pub fn is_vector(self) -> bool { + self.log2_lane_count() > 0 } /// Get the number of lanes in this SIMD vector type. @@ -225,10 +225,10 @@ impl Type { /// /// There is no `double_vector()` method. Use `t.by(2)` instead. pub fn half_vector(self) -> Option { - if self.is_scalar() { - None - } else { + if self.is_vector() { Some(Type(self.0 - 0x10)) + } else { + None } } @@ -255,7 +255,7 @@ impl Display for Type { write!(f, "i{}", self.lane_bits()) } else if self.is_float() { write!(f, "f{}", self.lane_bits()) - } else if !self.is_scalar() { + } else if self.is_vector() { write!(f, "{}x{}", self.lane_type(), self.lane_count()) } else { panic!("Invalid Type(0x{:x})", self.0) @@ -273,7 +273,7 @@ impl Debug for Type { write!(f, "types::I{}", self.lane_bits()) } else if self.is_float() { write!(f, "types::F{}", self.lane_bits()) - } else if !self.is_scalar() { + } else if self.is_vector() { write!(f, "{:?}X{}", self.lane_type(), self.lane_count()) } else { write!(f, "Type(0x{:x})", self.0) diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index c6bbe7fb2a..b45d439db3 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -46,7 +46,7 @@ impl ArgAssigner for Args { // Check for a legal type. // We don't support SIMD yet, so break all vectors down. - if !ty.is_scalar() { + if ty.is_vector() { return ValueConversion::VectorSplit.into(); } diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index 0859a53f31..f44ff705ec 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -45,7 +45,7 @@ impl ArgAssigner for Args { // Check for a legal type. // RISC-V doesn't have SIMD at all, so break all vectors down. - if !ty.is_scalar() { + if ty.is_vector() { return ValueConversion::VectorSplit.into(); } diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index a56de36a3c..75217eb239 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -217,7 +217,7 @@ fn expand_select(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowG /// Expand illegal `f32const` and `f64const` instructions. fn expand_fconst(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlowGraph) { let ty = func.dfg.value_type(func.dfg.first_result(inst)); - assert!(ty.is_scalar(), "Only scalar fconst supported: {}", ty); + assert!(!ty.is_vector(), "Only scalar fconst supported: {}", ty); // In the future, we may want to generate constant pool entries for these constants, but for // now use an `iconst` and a bit cast. From dbaa919ca945aec378787c6fd386fdee211bbda6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 12 Oct 2017 12:10:27 -0700 Subject: [PATCH 1250/3084] Make room for SpecialType in the value type numbering. The value types are now classified into three groups: 1. Lane types are scalar types that can also be used to form vectors. 2. Vector types 2-256 copies of a lane type. 3. Special types. This is where the CPU flag types will go. The special types can't be used to form vectors. Change the numbering scheme for value types to make room for the special types and add `is_lane()` and `is_special()` classification methods. The VOID type still has number 0, but it can no longer appear as a vector lane. It classifies as special now. --- cranelift/docs/metaref.rst | 7 ++-- lib/cretonne/meta/cdsl/types.py | 48 +++++++++++++++++++++++-- lib/cretonne/src/ir/types.rs | 63 +++++++++++++++++++++++---------- 3 files changed, 91 insertions(+), 27 deletions(-) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 5548311cac..209fa230d6 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -166,18 +166,15 @@ Concrete value types are represented as instances of :class:`ValueType`. There are subclasses to represent scalar and vector types. .. autoclass:: ValueType -.. inheritance-diagram:: ValueType LaneType VectorType IntType FloatType BoolType +.. inheritance-diagram:: ValueType LaneType VectorType SpecialType IntType FloatType BoolType :parts: 1 .. autoclass:: LaneType :members: .. autoclass:: VectorType - :members: +.. autoclass:: SpecialType .. autoclass:: IntType - :members: .. autoclass:: FloatType - :members: .. autoclass:: BoolType - :members: .. automodule:: base.types :members: diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py index 85bf164c62..505c0a1b2b 100644 --- a/lib/cretonne/meta/cdsl/types.py +++ b/lib/cretonne/meta/cdsl/types.py @@ -9,6 +9,18 @@ except ImportError: pass +# Numbering scheme for value types: +# +# 0: Void +# 0x01-0x6f: Special types +# 0x70-0x7f: Lane types +# 0x80-0xff: Vector types +# +# Vector types are encoded with the lane type in the low 4 bits and log2(lanes) +# in the high 4 bits, giving a range of 2-256 lanes. +LANE_BASE = 0x70 + + # ValueType instances (i8, i32, ...) are provided in the `base.types` module. class ValueType(object): """ @@ -24,6 +36,9 @@ class ValueType(object): # List of all the lane types. all_lane_types = list() # type: List[LaneType] + # List of all the special types (neither lanes nor vectors). + all_special_types = list() # type: List[SpecialType] + def __init__(self, name, membytes, doc): # type: (str, int, str) -> None self.name = name @@ -87,10 +102,11 @@ class LaneType(ValueType): # type: (str, int, str) -> None super(LaneType, self).__init__(name, membytes, doc) self._vectors = dict() # type: Dict[int, VectorType] - # Assign numbers starting from 1. (0 is VOID). + # Assign numbers starting from LANE_BASE. + n = len(ValueType.all_lane_types) ValueType.all_lane_types.append(self) - self.number = len(ValueType.all_lane_types) - assert self.number < 16, 'Too many lane types' + assert n < 16, 'Too many lane types' + self.number = LANE_BASE + n def __repr__(self): # type: () -> str @@ -132,6 +148,7 @@ class VectorType(ValueType): doc=""" A SIMD vector with {} lanes containing a `{}` each. """.format(lanes, base.name)) + assert lanes <= 256, "Too many lanes" self.base = base self.lanes = lanes self.number = 16*int(math.log(lanes, 2)) + base.number @@ -152,6 +169,31 @@ class VectorType(ValueType): return self.base.lane_bits() +class SpecialType(ValueType): + """ + A concrete scalar type that is neither a vector nor a lane type. + + Special types cannot be used to form vectors. + """ + + def __init__(self, name, membytes, doc): + # type: (str, int, str) -> None + super(SpecialType, self).__init__(name, membytes, doc) + # Assign numbers starting from 1. (0 is VOID) + ValueType.all_special_types.append(self) + self.number = len(ValueType.all_special_types) + assert self.number < LANE_BASE, 'Too many special types' + + def __repr__(self): + # type: () -> str + return 'SpecialType({})'.format(self.name) + + def lane_count(self): + # type: () -> int + """Return the number of lanes.""" + return 1 + + class IntType(LaneType): """A concrete scalar integer type.""" diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 3c0221d4cd..7efeb55558 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -30,6 +30,12 @@ pub struct Type(u8); /// a SIMD vector. pub const VOID: Type = Type(0); +// Start of the lane types. See also `meta/cdsl.types.py`. +const LANE_BASE: u8 = 0x70; + +// Start of the 2-lane vector types. +const VECTOR_BASE: u8 = LANE_BASE + 16; + // Include code generated by `lib/cretonne/meta/gen_types.py`. This file contains constant // definitions for all the scalar types as well as common vector types for 64, 128, 256, and // 512-bit SIMD vectors. @@ -40,7 +46,11 @@ impl Type { /// /// A lane type is the same as a SIMD vector type with one lane, so it returns itself. pub fn lane_type(self) -> Type { - Type(self.0 & 0x0f) + if self.0 < VECTOR_BASE { + self + } else { + Type(LANE_BASE | (self.0 & 0x0f)) + } } /// Get log_2 of the number of bits in a lane. @@ -78,6 +88,12 @@ impl Type { } } + /// Get a type with the same number of lanes as `self`, but using `lane` as the lane type. + fn replace_lanes(self, lane: Type) -> Type { + debug_assert!(lane.is_lane() && !self.is_special()); + Type((lane.0 & 0x0f) | (self.0 & 0xf0)) + } + /// Get a type with the same number of lanes as this type, but with the lanes replaced by /// booleans of the same size. /// @@ -85,14 +101,13 @@ impl Type { /// boolean types. pub fn as_bool_pedantic(self) -> Type { // Replace the low 4 bits with the boolean version, preserve the high 4 bits. - let lane = match self.lane_type() { + self.replace_lanes(match self.lane_type() { B8 | I8 => B8, B16 | I16 => B16, B32 | I32 | F32 => B32, B64 | I64 | F64 => B64, _ => B1, - }; - Type(lane.0 | (self.0 & 0xf0)) + }) } /// Get a type with the same number of lanes as this type, but with the lanes replaced by @@ -110,7 +125,7 @@ impl Type { /// Get a type with the same number of lanes as this type, but with lanes that are half the /// number of bits. pub fn half_width(self) -> Option { - let lane = match self.lane_type() { + Some(self.replace_lanes(match self.lane_type() { I16 => I8, I32 => I16, I64 => I32, @@ -119,14 +134,13 @@ impl Type { B32 => B16, B64 => B32, _ => return None, - }; - Some(Type(lane.0 | (self.0 & 0xf0))) + })) } /// Get a type with the same number of lanes as this type, but with lanes that are twice the /// number of bits. pub fn double_width(self) -> Option { - let lane = match self.lane_type() { + Some(self.replace_lanes(match self.lane_type() { I8 => I16, I16 => I32, I32 => I64, @@ -135,8 +149,7 @@ impl Type { B16 => B32, B32 => B64, _ => return None, - }; - Some(Type(lane.0 | (self.0 & 0xf0))) + })) } /// Is this the VOID type? @@ -144,6 +157,25 @@ impl Type { self == VOID } + /// Is this a special type? + pub fn is_special(self) -> bool { + self.0 < LANE_BASE + } + + /// Is this a lane type? + /// + /// This is a scalar type that can also appear as the lane type of a SIMD vector. + pub fn is_lane(self) -> bool { + LANE_BASE <= self.0 && self.0 < VECTOR_BASE + } + + /// Is this a SIMD vector type? + /// + /// A vector type has 2 or more lanes. + pub fn is_vector(self) -> bool { + self.0 >= VECTOR_BASE + } + /// Is this a scalar boolean type? pub fn is_bool(self) -> bool { match self { @@ -175,14 +207,7 @@ impl Type { /// /// A scalar type is the same as a SIMD vector type with one lane, so it returns 0. pub fn log2_lane_count(self) -> u8 { - self.0 >> 4 - } - - /// Is this a SIMD vector type? - /// - /// A vector type has 2 or more lanes. - pub fn is_vector(self) -> bool { - self.log2_lane_count() > 0 + self.0.saturating_sub(LANE_BASE) >> 4 } /// Get the number of lanes in this SIMD vector type. @@ -214,7 +239,7 @@ impl Type { } let log2_lanes: u32 = n.trailing_zeros(); let new_type = u32::from(self.0) + (log2_lanes << 4); - if new_type < 0x90 { + if new_type < 0x100 { Some(Type(new_type as u8)) } else { None From 15461c1e4b49d28633e4771d59bbfb05fa9b5ed3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 12 Oct 2017 12:49:10 -0700 Subject: [PATCH 1251/3084] Add two new value types: iflags and fflags. These two value types represent the state of CPU flags after an integer comparison and a floating point comparison respectively. Instructions using these types TBD. --- cranelift/docs/metaref.rst | 3 +- lib/cretonne/meta/base/types.py | 15 +++++++- lib/cretonne/meta/cdsl/types.py | 16 +++++++++ lib/cretonne/meta/cdsl/typevar.py | 60 ++++++++++++++++++++++++------- lib/cretonne/meta/gen_types.py | 2 ++ lib/cretonne/src/ir/types.rs | 34 +++++++++++++----- lib/reader/src/lexer.rs | 14 ++++++-- misc/vim/syntax/cton.vim | 2 +- 8 files changed, 119 insertions(+), 27 deletions(-) diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 209fa230d6..6f23647b0a 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -166,7 +166,7 @@ Concrete value types are represented as instances of :class:`ValueType`. There are subclasses to represent scalar and vector types. .. autoclass:: ValueType -.. inheritance-diagram:: ValueType LaneType VectorType SpecialType IntType FloatType BoolType +.. inheritance-diagram:: ValueType LaneType VectorType IntType FloatType BoolType SpecialType FlagsType :parts: 1 .. autoclass:: LaneType :members: @@ -175,6 +175,7 @@ are subclasses to represent scalar and vector types. .. autoclass:: IntType .. autoclass:: FloatType .. autoclass:: BoolType +.. autoclass:: FlagsType .. automodule:: base.types :members: diff --git a/lib/cretonne/meta/base/types.py b/lib/cretonne/meta/base/types.py index 7111626009..b60ca3719f 100644 --- a/lib/cretonne/meta/base/types.py +++ b/lib/cretonne/meta/base/types.py @@ -2,7 +2,7 @@ The base.types module predefines all the Cretonne scalar types. """ from __future__ import absolute_import -from cdsl.types import IntType, FloatType, BoolType +from cdsl.types import IntType, FloatType, BoolType, FlagsType #: Boolean. b1 = BoolType(1) #: 1-bit bool. Type is abstract (can't be stored in mem) @@ -31,3 +31,16 @@ f64 = FloatType( *binary64* interchange format. This corresponds to the :c:type:`double` type in most C implementations. """) +#: CPU flags from an integer comparison. +iflags = FlagsType( + 'iflags', """ + CPU flags representing the result of an integer comparison. These flags + can be tested with an `intcc` condition code. + """) + +#: CPU flags from a floating point comparison. +fflags = FlagsType( + 'fflags', """ + CPU flags representing the result of a floating point comparison. These + flags can be tested with a `floatcc` condition code. + """) diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/cretonne/meta/cdsl/types.py index 505c0a1b2b..3f2593fb78 100644 --- a/lib/cretonne/meta/cdsl/types.py +++ b/lib/cretonne/meta/cdsl/types.py @@ -287,6 +287,22 @@ class BoolType(LaneType): return self.bits +class FlagsType(SpecialType): + """ + A type representing CPU flags. + + Flags can't be stored in memory. + """ + + def __init__(self, name, doc): + # type: (str, str) -> None + super(FlagsType, self).__init__(name, 0, doc) + + def __repr__(self): + # type: () -> str + return 'FlagsType({})'.format(self.name) + + class BVType(ValueType): """A flat bitvector type. Used for semantics description only.""" diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index 0186ede5da..f818719a4d 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -163,6 +163,9 @@ class TypeSet(object): >>> TypeSet(lanes=True, ints=True) TypeSet(lanes={1, 2, 4, 8, 16, 32, 64, 128, 256}, ints={8, 16, 32, 64}) + Finally, a type set can contain special types (derived from `SpecialType`) + which can't appear as lane types. + :param lanes: `(min, max)` inclusive range of permitted vector lane counts. :param ints: `(min, max)` inclusive range of permitted scalar integer widths. @@ -172,11 +175,19 @@ class TypeSet(object): widths. :param bitvecs : `(min, max)` inclusive range of permitted bitvector widths. + :param specials: Sequence of speical types to appear in the set. """ - def __init__(self, lanes=None, ints=None, floats=None, bools=None, - bitvecs=None): - # type: (BoolInterval, BoolInterval, BoolInterval, BoolInterval, BoolInterval) -> None # noqa + def __init__( + self, + lanes=None, # type: BoolInterval + ints=None, # type: BoolInterval + floats=None, # type: BoolInterval + bools=None, # type: BoolInterval + bitvecs=None, # type: BoolInterval + specials=None # type: Iterable[types.SpecialType] + ): + # type: (...) -> None self.lanes = interval_to_set(decode_interval(lanes, (1, MAX_LANES), 1)) self.ints = interval_to_set(decode_interval(ints, (8, MAX_BITS))) self.floats = interval_to_set(decode_interval(floats, (32, 64))) @@ -184,6 +195,7 @@ class TypeSet(object): self.bools = set(filter(legal_bool, self.bools)) self.bitvecs = interval_to_set(decode_interval(bitvecs, (1, MAX_BITVEC))) + self.specials = set(specials) if specials else set() def copy(self): # type: (TypeSet) -> TypeSet @@ -194,13 +206,14 @@ class TypeSet(object): return deepcopy(self) def typeset_key(self): - # type: () -> Tuple[Tuple, Tuple, Tuple, Tuple, Tuple] + # type: () -> Tuple[Tuple, Tuple, Tuple, Tuple, Tuple, Tuple] """Key tuple used for hashing and equality.""" return (tuple(sorted(list(self.lanes))), tuple(sorted(list(self.ints))), tuple(sorted(list(self.floats))), tuple(sorted(list(self.bools))), - tuple(sorted(list(self.bitvecs)))) + tuple(sorted(list(self.bitvecs))), + tuple(sorted(list(self.specials)))) def __hash__(self): # type: () -> int @@ -231,6 +244,8 @@ class TypeSet(object): s += ', bools={}'.format(pp_set(self.bools)) if len(self.bitvecs) > 0: s += ', bitvecs={}'.format(pp_set(self.bitvecs)) + if len(self.specials) > 0: + s += ', specials=[{}]'.format(pp_set(self.specials)) return s + ')' def emit_fields(self, fmt): @@ -273,6 +288,7 @@ class TypeSet(object): self.floats.intersection_update(other.floats) self.bools.intersection_update(other.bools) self.bitvecs.intersection_update(other.bitvecs) + self.specials.intersection_update(other.specials) return self @@ -481,8 +497,9 @@ class TypeSet(object): """ Return the number of concrete types represented by this typeset """ - return len(self.lanes) * (len(self.ints) + len(self.floats) + - len(self.bools) + len(self.bitvecs)) + return (len(self.lanes) * (len(self.ints) + len(self.floats) + + len(self.bools) + len(self.bitvecs)) + + len(self.specials)) def concrete_types(self): # type: () -> Iterable[types.ValueType] @@ -504,6 +521,9 @@ class TypeSet(object): assert nlanes == 1 yield types.BVType.with_bits(bits) + for spec in self.specials: + yield spec + def get_singleton(self): # type: () -> types.ValueType """ @@ -545,11 +565,20 @@ class TypeVar(object): """ def __init__( - self, name, doc, - ints=False, floats=False, bools=False, - scalars=True, simd=False, bitvecs=False, - base=None, derived_func=None): - # type: (str, str, BoolInterval, BoolInterval, BoolInterval, bool, BoolInterval, BoolInterval, TypeVar, str) -> None # noqa + self, + name, # type: str + doc, # type: str + ints=False, # type: BoolInterval + floats=False, # type: BoolInterval + bools=False, # type: BoolInterval + scalars=True, # type: bool + simd=False, # type: BoolInterval + bitvecs=False, # type: BoolInterval + base=None, # type: TypeVar + derived_func=None, # type: str + specials=None # type: Iterable[types.SpecialType] + ): + # type: (...) -> None self.name = name self.__doc__ = doc self.is_derived = isinstance(base, TypeVar) @@ -567,7 +596,8 @@ class TypeVar(object): ints=ints, floats=floats, bools=bools, - bitvecs=bitvecs) + bitvecs=bitvecs, + specials=specials) @staticmethod def singleton(typ): @@ -580,6 +610,8 @@ class TypeVar(object): elif isinstance(typ, types.LaneType): scalar = typ lanes = (1, 1) + elif isinstance(typ, types.SpecialType): + return TypeVar(typ.name, typ.__doc__, specials=[typ]) else: assert isinstance(typ, types.BVType) scalar = typ @@ -681,6 +713,8 @@ class TypeVar(object): # Safety checks to avoid over/underflows. ts = base.get_typeset() + assert len(ts.specials) == 0, "Can't derive from special types" + if derived_func == TypeVar.HALFWIDTH: if len(ts.ints) > 0: assert min(ts.ints) > 8, "Can't halve all integer types" diff --git a/lib/cretonne/meta/gen_types.py b/lib/cretonne/meta/gen_types.py index 42ccc1bfe7..c5e36758ba 100644 --- a/lib/cretonne/meta/gen_types.py +++ b/lib/cretonne/meta/gen_types.py @@ -45,6 +45,8 @@ def emit_vectors(bits, fmt): def emit_types(fmt): # type: (srcgen.Formatter) -> None + for spec in ValueType.all_special_types: + emit_type(spec, fmt) for ty in ValueType.all_lane_types: emit_type(ty, fmt) # Emit vector definitions for common SIMD sizes. diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 7efeb55558..1a20f9e511 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -272,9 +272,7 @@ impl Type { impl Display for Type { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - if self.is_void() { - write!(f, "void") - } else if self.is_bool() { + if self.is_bool() { write!(f, "b{}", self.lane_bits()) } else if self.is_int() { write!(f, "i{}", self.lane_bits()) @@ -283,16 +281,19 @@ impl Display for Type { } else if self.is_vector() { write!(f, "{}x{}", self.lane_type(), self.lane_count()) } else { - panic!("Invalid Type(0x{:x})", self.0) + f.write_str(match *self { + VOID => "void", + IFLAGS => "iflags", + FFLAGS => "fflags", + _ => panic!("Invalid Type(0x{:x})", self.0), + }) } } } impl Debug for Type { fn fmt(&self, f: &mut Formatter) -> fmt::Result { - if self.is_void() { - write!(f, "types::VOID") - } else if self.is_bool() { + if self.is_bool() { write!(f, "types::B{}", self.lane_bits()) } else if self.is_int() { write!(f, "types::I{}", self.lane_bits()) @@ -301,7 +302,12 @@ impl Debug for Type { } else if self.is_vector() { write!(f, "{:?}X{}", self.lane_type(), self.lane_count()) } else { - write!(f, "Type(0x{:x})", self.0) + match *self { + VOID => write!(f, "types::VOID"), + IFLAGS => write!(f, "types::IFLAGS"), + FFLAGS => write!(f, "types::FFLAGS"), + _ => write!(f, "Type(0x{:x})", self.0), + } } } } @@ -320,6 +326,10 @@ mod tests { fn basic_scalars() { assert_eq!(VOID, VOID.lane_type()); assert_eq!(0, VOID.bits()); + assert_eq!(IFLAGS, IFLAGS.lane_type()); + assert_eq!(0, IFLAGS.bits()); + assert_eq!(FFLAGS, FFLAGS.lane_type()); + assert_eq!(0, FFLAGS.bits()); assert_eq!(B1, B1.lane_type()); assert_eq!(B8, B8.lane_type()); assert_eq!(B16, B16.lane_type()); @@ -333,6 +343,8 @@ mod tests { assert_eq!(F64, F64.lane_type()); assert_eq!(VOID.lane_bits(), 0); + assert_eq!(IFLAGS.lane_bits(), 0); + assert_eq!(FFLAGS.lane_bits(), 0); assert_eq!(B1.lane_bits(), 1); assert_eq!(B8.lane_bits(), 8); assert_eq!(B16.lane_bits(), 16); @@ -349,6 +361,8 @@ mod tests { #[test] fn typevar_functions() { assert_eq!(VOID.half_width(), None); + assert_eq!(IFLAGS.half_width(), None); + assert_eq!(FFLAGS.half_width(), None); assert_eq!(B1.half_width(), None); assert_eq!(B8.half_width(), None); assert_eq!(B16.half_width(), Some(B8)); @@ -363,6 +377,8 @@ mod tests { assert_eq!(F64.half_width(), Some(F32)); assert_eq!(VOID.double_width(), None); + assert_eq!(IFLAGS.double_width(), None); + assert_eq!(FFLAGS.double_width(), None); assert_eq!(B1.double_width(), None); assert_eq!(B8.double_width(), Some(B16)); assert_eq!(B16.double_width(), Some(B32)); @@ -397,6 +413,8 @@ mod tests { #[test] fn format_scalars() { assert_eq!(VOID.to_string(), "void"); + assert_eq!(IFLAGS.to_string(), "iflags"); + assert_eq!(FFLAGS.to_string(), "fflags"); assert_eq!(B1.to_string(), "b1"); assert_eq!(B8.to_string(), "b8"); assert_eq!(B16.to_string(), "b16"); diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 357880ef4a..10cd9737af 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -303,7 +303,11 @@ impl<'a> Lexer<'a> { Self::value_type(text, prefix, number) }) }) - .unwrap_or(Token::Identifier(text)), + .unwrap_or_else(|| match text { + "iflags" => Token::Type(types::IFLAGS), + "fflags" => Token::Type(types::FFLAGS), + _ => Token::Identifier(text), + }), loc, ) } @@ -554,7 +558,8 @@ mod tests { fn lex_identifiers() { let mut lex = Lexer::new( "v0 v00 vx01 ebb1234567890 ebb5234567890 v1x vx1 vxvx4 \ - function0 function b1 i32x4 f32x5", + function0 function b1 i32x4 f32x5 \ + iflags fflags iflagss", ); assert_eq!( lex.next(), @@ -573,8 +578,11 @@ mod tests { assert_eq!(lex.next(), token(Token::Identifier("function0"), 1)); assert_eq!(lex.next(), token(Token::Identifier("function"), 1)); assert_eq!(lex.next(), token(Token::Type(types::B1), 1)); - assert_eq!(lex.next(), token(Token::Type(types::I32.by(4).unwrap()), 1)); + assert_eq!(lex.next(), token(Token::Type(types::I32X4), 1)); assert_eq!(lex.next(), token(Token::Identifier("f32x5"), 1)); + assert_eq!(lex.next(), token(Token::Type(types::IFLAGS), 1)); + assert_eq!(lex.next(), token(Token::Type(types::FFLAGS), 1)); + assert_eq!(lex.next(), token(Token::Identifier("iflagss"), 1)); assert_eq!(lex.next(), None); } diff --git a/misc/vim/syntax/cton.vim b/misc/vim/syntax/cton.vim index 56602e8ded..6ccacae7fc 100644 --- a/misc/vim/syntax/cton.vim +++ b/misc/vim/syntax/cton.vim @@ -17,7 +17,7 @@ syn keyword ctonHeader test isa set syn keyword ctonDecl function jump_table incoming_arg outgoing_arg spill_slot local emergency_slot syn keyword ctonFilecheck check sameln nextln unordered not regex contained -syn match ctonType /\<[bif]\d\+\(x\d\+\)\?\>/ +syn match ctonType /\<\([bif]\d\+\(x\d\+\)\?\)\|[if]flags\>/ syn match ctonEntity /\<\(v\|ss\|jt\|fn\|sig\)\d\+\>/ syn match ctonLabel /\/ syn match ctonName /%\w\+\>/ From 1f98fc491c9cf6bb8ee093e21ad4097c6ab5116c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 12 Oct 2017 15:21:40 -0700 Subject: [PATCH 1252/3084] Add instructions using CPU flags. Add integer and floating comparison instructions that return CPU flags: ifcmp, ifcmp_imm, and ffcmp. Add conditional branch instructions that check CPU flags: brif, brff Add instructions that check a condition in the CPU flags and return a b1: trueif, trueff. --- cranelift/docs/langref.rst | 11 ++++ cranelift/filetests/parser/flags.cton | 46 +++++++++++++++ lib/cretonne/meta/base/formats.py | 4 ++ lib/cretonne/meta/base/instructions.py | 82 +++++++++++++++++++++++++- lib/cretonne/src/ir/instructions.rs | 22 +++++++ lib/cretonne/src/verifier/mod.rs | 12 ++++ lib/cretonne/src/write.rs | 53 +++++++++++------ lib/reader/src/parser.rs | 60 ++++++++++++++++--- 8 files changed, 262 insertions(+), 28 deletions(-) create mode 100644 cranelift/filetests/parser/flags.cton diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 5d2b46cc73..ea060551be 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -329,6 +329,8 @@ instruction in the EBB. .. autoinst:: brz .. autoinst:: brnz .. autoinst:: br_icmp +.. autoinst:: brif +.. autoinst:: brff .. autoinst:: br_table .. inst:: JT = jump_table EBB0, EBB1, ..., EBBn @@ -722,6 +724,8 @@ Integer operations .. autoinst:: icmp .. autoinst:: icmp_imm +.. autoinst:: ifcmp +.. autoinst:: ifcmp_imm .. autoinst:: iadd .. autoinst:: iadd_imm .. autoinst:: iadd_cin @@ -814,6 +818,7 @@ Floating point operations These operations generally follow IEEE 754-2008 semantics. .. autoinst:: fcmp +.. autoinst:: ffcmp .. autoinst:: fadd .. autoinst:: fsub .. autoinst:: fmul @@ -857,6 +862,12 @@ represented as a floating point number. .. autoinst:: trunc .. autoinst:: nearest +CPU flag operations +------------------- + +.. autoinst:: trueif +.. autoinst:: trueff + Conversion operations --------------------- diff --git a/cranelift/filetests/parser/flags.cton b/cranelift/filetests/parser/flags.cton new file mode 100644 index 0000000000..a315fbba56 --- /dev/null +++ b/cranelift/filetests/parser/flags.cton @@ -0,0 +1,46 @@ +test cat +test verifier + +function %iflags(i32) { +ebb0(v0: i32): + v1 = ifcmp_imm v0, 17 + brif eq v1, ebb1 + brif ugt v1, ebb2 + v2 = iconst.i32 34 + v3 = ifcmp v0, v2 + v4 = trueif eq v3 + brnz v4, ebb2 + return + +ebb1: + return + +ebb2: + trap oob +} +; check: $v1 = ifcmp_imm $v0, 17 +; check: brif eq $v1, $ebb1 +; check: brif ugt $v1, $ebb2 +; check: $v3 = ifcmp $v0, $v2 +; check: $v4 = trueif eq $v3 + +function %fflags(f32) { +ebb0(v0: f32): + v1 = f32const 0x34.0p0 + v2 = ffcmp v0, v1 + brff eq v2, ebb1 + brff ord v2, ebb2 + v3 = trueff gt v2 + brnz v3, ebb2 + return + +ebb1: + return + +ebb2: + trap oob +} +; check: $v2 = ffcmp $v0, $v1 +; check: brff eq $v2, $ebb1 +; check: brff ord $v2, $ebb2 +; check: $v3 = trueff gt $v2 diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 0f31ff2c5f..4e1ff3500a 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -37,10 +37,14 @@ ExtractLane = InstructionFormat(VALUE, ('lane', uimm8)) IntCompare = InstructionFormat(intcc, VALUE, VALUE) IntCompareImm = InstructionFormat(intcc, VALUE, imm64) +IntCond = InstructionFormat(intcc, VALUE) FloatCompare = InstructionFormat(floatcc, VALUE, VALUE) +FloatCond = InstructionFormat(floatcc, VALUE) Jump = InstructionFormat(ebb, VARIABLE_ARGS) Branch = InstructionFormat(VALUE, ebb, VARIABLE_ARGS) +BranchInt = InstructionFormat(intcc, VALUE, ebb, VARIABLE_ARGS) +BranchFloat = InstructionFormat(floatcc, VALUE, ebb, VARIABLE_ARGS) BranchIcmp = InstructionFormat(intcc, VALUE, VALUE, ebb, VARIABLE_ARGS) BranchTable = InstructionFormat(VALUE, entities.jump_table) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 6e94b4c553..48c37c280c 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -8,7 +8,7 @@ from __future__ import absolute_import from cdsl.operands import Operand, VARIABLE_ARGS from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup -from base.types import f32, f64, b1 +from base.types import f32, f64, b1, iflags, fflags from base.immediates import imm64, uimm8, uimm32, ieee32, ieee64, offset32 from base.immediates import boolean, intcc, floatcc, memflags, regunit from base.immediates import trapcode @@ -112,6 +112,23 @@ br_icmp = Instruction( """, ins=(Cond, x, y, EBB, args), is_branch=True) +f = Operand('f', iflags) + +brif = Instruction( + 'brif', r""" + Branch when condition is true in integer CPU flags. + """, + ins=(Cond, f, EBB, args), is_branch=True) + +Cond = Operand('Cond', floatcc) +f = Operand('f', fflags) + +brff = Instruction( + 'brff', r""" + Branch when condition is true in floating point CPU flags. + """, + ins=(Cond, f, EBB, args), is_branch=True) + x = Operand('x', iB, doc='index into jump table') JT = Operand('JT', entities.jump_table) br_table = Instruction( @@ -677,6 +694,28 @@ icmp_imm = Instruction( """, ins=(Cond, x, Y), outs=a) +f = Operand('f', iflags) +x = Operand('x', iB) +y = Operand('y', iB) + +ifcmp = Instruction( + 'ifcmp', r""" + Compare scalar integers and return flags. + + Compare two scalar integer values and return integer CPU flags + representing the result. + """, + ins=(x, y), outs=f) + +ifcmp_imm = Instruction( + 'ifcmp_imm', r""" + Compare scalar integer to a constant and return flags. + + Like :inst:`icmp_imm`, but returns integer CPU flags instead of testing + a specific condition code. + """, + ins=(x, Y), outs=f) + a = Operand('a', Int) x = Operand('x', Int) y = Operand('y', Int) @@ -1176,6 +1215,7 @@ popcnt = Instruction( Float = TypeVar( 'Float', 'A scalar or vector floating point number', floats=True, simd=True) +fB = TypeVar('fB', 'A scalar floating point number', floats=True) Cond = Operand('Cond', floatcc) x = Operand('x', Float) @@ -1246,6 +1286,17 @@ fcmp = Instruction( """, ins=(Cond, x, y), outs=a) +f = Operand('f', fflags) + +ffcmp = Instruction( + 'ffcmp', r""" + Floating point comparison returning flags. + + Compares two numbers like :inst:`fcmp`, but returns floating point CPU + flags instead of testing a specific condition. + """, + ins=(x, y), outs=f) + x = Operand('x', Float) y = Operand('y', Float) z = Operand('z', Float) @@ -1387,6 +1438,35 @@ nearest = Instruction( """, ins=x, outs=a) +# +# CPU flag operations +# + + +Cond = Operand('Cond', intcc) +f = Operand('f', iflags) +a = Operand('a', b1) + +trueif = Instruction( + 'trueif', r""" + Test integer CPU flags for a specific condition. + + Check the CPU flags in ``f`` against the ``Cond`` condition code and + return true when the condition code is satisfied. + """, + ins=(Cond, f), outs=a) + +Cond = Operand('Cond', floatcc) +f = Operand('f', fflags) + +trueff = Instruction( + 'trueff', r""" + Test floating point CPU flags for a specific condition. + + Check the CPU flags in ``f`` against the ``Cond`` condition code and + return true when the condition code is satisfied. + """, + ins=(Cond, f), outs=a) # # Conversions diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 2174bb4c99..0cb9234ed6 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -141,11 +141,21 @@ pub enum InstructionData { arg: Value, imm: Imm64, }, + IntCond { + opcode: Opcode, + cond: IntCC, + arg: Value, + }, FloatCompare { opcode: Opcode, cond: FloatCC, args: [Value; 2], }, + FloatCond { + opcode: Opcode, + cond: FloatCC, + arg: Value, + }, Jump { opcode: Opcode, destination: Ebb, @@ -162,6 +172,18 @@ pub enum InstructionData { destination: Ebb, args: ValueList, }, + BranchInt { + opcode: Opcode, + cond: IntCC, + destination: Ebb, + args: ValueList, + }, + BranchFloat { + opcode: Opcode, + cond: FloatCC, + destination: Ebb, + args: ValueList, + }, BranchTable { opcode: Opcode, arg: Value, diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 58da610829..f143531148 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -288,6 +288,16 @@ impl<'a> Verifier<'a> { ref args, .. } | + BranchInt { + destination, + ref args, + .. + } | + BranchFloat { + destination, + ref args, + .. + } | BranchIcmp { destination, ref args, @@ -340,7 +350,9 @@ impl<'a> Verifier<'a> { ExtractLane { .. } | IntCompare { .. } | IntCompareImm { .. } | + IntCond { .. } | FloatCompare { .. } | + FloatCond { .. } | Load { .. } | Store { .. } | RegMove { .. } | diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index a1e195cd17..ebcd703aaa 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -299,22 +299,16 @@ pub fn write_operands( ExtractLane { lane, arg, .. } => write!(w, " {}, {}", arg, lane), IntCompare { cond, args, .. } => write!(w, " {} {}, {}", cond, args[0], args[1]), IntCompareImm { cond, arg, imm, .. } => write!(w, " {} {}, {}", cond, arg, imm), + IntCond { cond, arg, .. } => write!(w, " {} {}", cond, arg), FloatCompare { cond, args, .. } => write!(w, " {} {}, {}", cond, args[0], args[1]), + FloatCond { cond, arg, .. } => write!(w, " {} {}", cond, arg), Jump { destination, ref args, .. } => { - if args.is_empty() { - write!(w, " {}", destination) - } else { - write!( - w, - " {}({})", - destination, - DisplayValues(args.as_slice(pool)) - ) - } + write!(w, " {}", destination)?; + write_ebb_args(w, args.as_slice(pool)) } Branch { destination, @@ -323,10 +317,27 @@ pub fn write_operands( } => { let args = args.as_slice(pool); write!(w, " {}, {}", args[0], destination)?; - if args.len() > 1 { - write!(w, "({})", DisplayValues(&args[1..]))?; - } - Ok(()) + write_ebb_args(w, &args[1..]) + } + BranchInt { + cond, + destination, + ref args, + .. + } => { + let args = args.as_slice(pool); + write!(w, " {} {}, {}", cond, args[0], destination)?; + write_ebb_args(w, &args[1..]) + } + BranchFloat { + cond, + destination, + ref args, + .. + } => { + let args = args.as_slice(pool); + write!(w, " {} {}, {}", cond, args[0], destination)?; + write_ebb_args(w, &args[1..]) } BranchIcmp { cond, @@ -336,10 +347,7 @@ pub fn write_operands( } => { let args = args.as_slice(pool); write!(w, " {} {}, {}, {}", cond, args[0], args[1], destination)?; - if args.len() > 2 { - write!(w, "({})", DisplayValues(&args[2..]))?; - } - Ok(()) + write_ebb_args(w, &args[2..]) } BranchTable { arg, table, .. } => write!(w, " {}, {}", arg, table), Call { func_ref, ref args, .. } => { @@ -406,6 +414,15 @@ pub fn write_operands( } } +/// Write EBB args using optional parantheses. +fn write_ebb_args(w: &mut Write, args: &[Value]) -> Result { + if args.is_empty() { + Ok(()) + } else { + write!(w, "({})", DisplayValues(args)) + } +} + /// Displayable slice of values. struct DisplayValues<'a>(&'a [Value]); diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 354cb02a22..86642277c9 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1975,6 +1975,38 @@ impl<'a> Parser<'a> { args: args.into_value_list(&[ctrl_arg], &mut ctx.function.dfg.value_lists), } } + InstructionFormat::BranchInt => { + let cond = self.match_enum("expected intcc condition code")?; + let arg = self.match_value("expected SSA value first operand")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; + let ebb_num = self.match_ebb("expected branch destination EBB")?; + let args = self.parse_opt_value_list()?; + InstructionData::BranchInt { + opcode, + cond, + destination: ebb_num, + args: args.into_value_list(&[arg], &mut ctx.function.dfg.value_lists), + } + } + InstructionFormat::BranchFloat => { + let cond = self.match_enum("expected floatcc condition code")?; + let arg = self.match_value("expected SSA value first operand")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; + let ebb_num = self.match_ebb("expected branch destination EBB")?; + let args = self.parse_opt_value_list()?; + InstructionData::BranchFloat { + opcode, + cond, + destination: ebb_num, + args: args.into_value_list(&[arg], &mut ctx.function.dfg.value_lists), + } + } InstructionFormat::BranchIcmp => { let cond = self.match_enum("expected intcc condition code")?; let lhs = self.match_value("expected SSA value first operand")?; @@ -1996,6 +2028,15 @@ impl<'a> Parser<'a> { args: args.into_value_list(&[lhs, rhs], &mut ctx.function.dfg.value_lists), } } + InstructionFormat::BranchTable => { + let arg = self.match_value("expected SSA value operand")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; + let table = self.match_jt().and_then(|num| ctx.get_jt(num, &self.loc))?; + InstructionData::BranchTable { opcode, arg, table } + } InstructionFormat::InsertLane => { let lhs = self.match_value("expected SSA value first operand")?; self.match_token( @@ -2052,6 +2093,11 @@ impl<'a> Parser<'a> { imm: rhs, } } + InstructionFormat::IntCond => { + let cond = self.match_enum("expected intcc condition code")?; + let arg = self.match_value("expected SSA value")?; + InstructionData::IntCond { opcode, cond, arg } + } InstructionFormat::FloatCompare => { let cond = self.match_enum("expected floatcc condition code")?; let lhs = self.match_value("expected SSA value first operand")?; @@ -2066,6 +2112,11 @@ impl<'a> Parser<'a> { args: [lhs, rhs], } } + InstructionFormat::FloatCond => { + let cond = self.match_enum("expected floatcc condition code")?; + let arg = self.match_value("expected SSA value")?; + InstructionData::FloatCond { opcode, cond, arg } + } InstructionFormat::Call => { let func_ref = self.match_fn("expected function reference").and_then( |num| { @@ -2121,15 +2172,6 @@ impl<'a> Parser<'a> { )?; InstructionData::FuncAddr { opcode, func_ref } } - InstructionFormat::BranchTable => { - let arg = self.match_value("expected SSA value operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; - let table = self.match_jt().and_then(|num| ctx.get_jt(num, &self.loc))?; - InstructionData::BranchTable { opcode, arg, table } - } InstructionFormat::StackLoad => { let ss = self.match_ss("expected stack slot number: ss«n»") .and_then(|num| ctx.get_ss(num, &self.loc))?; From ad2ffcd7fcad1a22242cede3b8dd975b5550bfda Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 11 Oct 2017 11:11:33 -0700 Subject: [PATCH 1253/3084] Update to wasmparser 0.13.0. --- lib/wasm/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 0aacd4b6a9..2aa984d720 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -11,7 +11,7 @@ license = "Apache-2.0" name = "cton_wasm" [dependencies] -wasmparser = "0.12.1" +wasmparser = "0.13.0" cretonne = { path = "../cretonne" } cretonne-frontend = { path = "../frontend" } From d6ab7e3abfd3420498a4f08ebeed0f3e95ae70b3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 11 Oct 2017 15:00:34 -0700 Subject: [PATCH 1254/3084] Minor comment cleanup. --- lib/wasm/src/runtime/dummy.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index e3828d6e67..65040a51e4 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -113,7 +113,7 @@ impl DummyEnvironment { } } -/// The FuncEnvironment implementation for use by the DummyEnvironment. +/// The FuncEnvironment implementation for use by the `DummyEnvironment`. pub struct DummyFuncEnvironment<'dummy_environment> { pub mod_info: &'dummy_environment DummyModuleInfo, } From bd94a3b20276ec484c2acb6cbb77b66a07db1737 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 11 Oct 2017 21:28:36 -0700 Subject: [PATCH 1255/3084] Move the 'data lifetime parameter to the ModuleEnvironment trait. This is needed to allow implementations to have 'data-lifetime references if they choose to. The DummyEnvironment is an example of an implementation that doesn't choose to. --- lib/wasm/src/module_translator.rs | 2 +- lib/wasm/src/runtime/dummy.rs | 16 ++++++++-------- lib/wasm/src/runtime/spec.rs | 16 ++++++++-------- lib/wasm/src/sections_translator.rs | 18 +++++++++--------- 4 files changed, 26 insertions(+), 26 deletions(-) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 4aed1e9851..ef68350e80 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -13,7 +13,7 @@ use runtime::ModuleEnvironment; /// indexes in the wasm module and the indexes inside each functions. pub fn translate_module<'data>( data: &'data [u8], - environ: &mut ModuleEnvironment, + environ: &mut ModuleEnvironment<'data>, ) -> Result<(), String> { let mut parser = Parser::new(data); match *parser.read() { diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 65040a51e4..085fb9b27e 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -195,7 +195,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ } } -impl ModuleEnvironment for DummyEnvironment { +impl<'data> ModuleEnvironment<'data> for DummyEnvironment { fn get_func_name(&self, func_index: FunctionIndex) -> ir::FunctionName { get_func_name(func_index) } @@ -208,7 +208,7 @@ impl ModuleEnvironment for DummyEnvironment { &self.info.signatures[sig_index] } - fn declare_func_import<'data>( + fn declare_func_import( &mut self, sig_index: SignatureIndex, module: &'data str, @@ -261,7 +261,7 @@ impl ModuleEnvironment for DummyEnvironment { fn declare_memory(&mut self, memory: Memory) { self.info.memories.push(Exportable::new(memory)); } - fn declare_data_initialization<'data>( + fn declare_data_initialization( &mut self, _memory_index: MemoryIndex, _base: Option, @@ -271,7 +271,7 @@ impl ModuleEnvironment for DummyEnvironment { // We do nothing } - fn declare_func_export<'data>(&mut self, func_index: FunctionIndex, name: &'data str) { + fn declare_func_export(&mut self, func_index: FunctionIndex, name: &'data str) { self.info.functions[func_index].export_names.push( String::from( name, @@ -279,13 +279,13 @@ impl ModuleEnvironment for DummyEnvironment { ); } - fn declare_table_export<'data>(&mut self, table_index: TableIndex, name: &'data str) { + fn declare_table_export(&mut self, table_index: TableIndex, name: &'data str) { self.info.tables[table_index].export_names.push( String::from(name), ); } - fn declare_memory_export<'data>(&mut self, memory_index: MemoryIndex, name: &'data str) { + fn declare_memory_export(&mut self, memory_index: MemoryIndex, name: &'data str) { self.info.memories[memory_index].export_names.push( String::from( name, @@ -293,7 +293,7 @@ impl ModuleEnvironment for DummyEnvironment { ); } - fn declare_global_export<'data>(&mut self, global_index: GlobalIndex, name: &'data str) { + fn declare_global_export(&mut self, global_index: GlobalIndex, name: &'data str) { self.info.globals[global_index].export_names.push( String::from( name, @@ -307,7 +307,7 @@ impl ModuleEnvironment for DummyEnvironment { } /// Provides the contents of a function body. - fn define_function_body<'data>(&mut self, body_bytes: &'data [u8]) -> Result<(), String> { + fn define_function_body(&mut self, body_bytes: &'data [u8]) -> Result<(), String> { let function_index = self.get_num_func_imports() + self.info.function_bodies.len(); let name = get_func_name(function_index); let sig = self.get_signature(self.get_func_type(function_index)) diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index bd38c8c316..7a235d524f 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -149,7 +149,7 @@ pub trait FuncEnvironment { /// An object satisfyng the `ModuleEnvironment` trait can be passed as argument to the /// [`translate_module`](fn.translate_module.html) function. These methods should not be called /// by the user, they are only for `cretonne-wasm` internal use. -pub trait ModuleEnvironment { +pub trait ModuleEnvironment<'data> { /// Return the name for the given function index. fn get_func_name(&self, func_index: FunctionIndex) -> ir::FunctionName; @@ -160,7 +160,7 @@ pub trait ModuleEnvironment { fn get_signature(&self, sig_index: SignatureIndex) -> &ir::Signature; /// Declares a function import to the environment. - fn declare_func_import<'data>( + fn declare_func_import( &mut self, sig_index: SignatureIndex, module: &'data str, @@ -195,7 +195,7 @@ pub trait ModuleEnvironment { /// Declares a memory to the environment fn declare_memory(&mut self, memory: Memory); /// Fills a declared memory with bytes at module instantiation. - fn declare_data_initialization<'data>( + fn declare_data_initialization( &mut self, memory_index: MemoryIndex, base: Option, @@ -204,17 +204,17 @@ pub trait ModuleEnvironment { ); /// Declares a function export to the environment. - fn declare_func_export<'data>(&mut self, func_index: FunctionIndex, name: &'data str); + fn declare_func_export(&mut self, func_index: FunctionIndex, name: &'data str); /// Declares a table export to the environment. - fn declare_table_export<'data>(&mut self, table_index: TableIndex, name: &'data str); + fn declare_table_export(&mut self, table_index: TableIndex, name: &'data str); /// Declares a memory export to the environment. - fn declare_memory_export<'data>(&mut self, memory_index: MemoryIndex, name: &'data str); + fn declare_memory_export(&mut self, memory_index: MemoryIndex, name: &'data str); /// Declares a global export to the environment. - fn declare_global_export<'data>(&mut self, global_index: GlobalIndex, name: &'data str); + fn declare_global_export(&mut self, global_index: GlobalIndex, name: &'data str); /// Declares a start function. fn declare_start_func(&mut self, index: FunctionIndex); /// Provides the contents of a function body. - fn define_function_body<'data>(&mut self, body_bytes: &'data [u8]) -> Result<(), String>; + fn define_function_body(&mut self, body_bytes: &'data [u8]) -> Result<(), String>; } diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index b76734101d..e0503c0065 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -56,9 +56,9 @@ pub fn parse_function_signatures( } /// Retrieves the imports from the imports section of the binary. -pub fn parse_import_section( - parser: &mut Parser, - environ: &mut ModuleEnvironment, +pub fn parse_import_section<'data>( + parser: &mut Parser<'data>, + environ: &mut ModuleEnvironment<'data>, ) -> Result<(), SectionParsingError> { loop { match *parser.read() { @@ -128,9 +128,9 @@ pub fn parse_function_section( } /// Retrieves the names of the functions from the export section -pub fn parse_export_section( - parser: &mut Parser, - environ: &mut ModuleEnvironment, +pub fn parse_export_section<'data>( + parser: &mut Parser<'data>, + environ: &mut ModuleEnvironment<'data>, ) -> Result<(), SectionParsingError> { loop { match *parser.read() { @@ -246,9 +246,9 @@ pub fn parse_global_section( Ok(()) } -pub fn parse_data_section( - parser: &mut Parser, - environ: &mut ModuleEnvironment, +pub fn parse_data_section<'data>( + parser: &mut Parser<'data>, + environ: &mut ModuleEnvironment<'data>, ) -> Result<(), SectionParsingError> { loop { let memory_index = match *parser.read() { From c8084474680aa8b2bfb268b4e2a0236c489334a6 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 13 Oct 2017 12:40:15 -0700 Subject: [PATCH 1256/3084] Have declare_table_elements consume its elements argument. The producer is already allocating a new Vec, so no need to require the consumer to allocate one too. --- lib/wasm/src/runtime/dummy.rs | 2 +- lib/wasm/src/runtime/spec.rs | 2 +- lib/wasm/src/sections_translator.rs | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/runtime/dummy.rs index 085fb9b27e..aab8ed6a4f 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/runtime/dummy.rs @@ -254,7 +254,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment { _table_index: TableIndex, _base: Option, _offset: usize, - _elements: &[FunctionIndex], + _elements: Vec, ) { // We do nothing } diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/runtime/spec.rs index 7a235d524f..c4de95a4f9 100644 --- a/lib/wasm/src/runtime/spec.rs +++ b/lib/wasm/src/runtime/spec.rs @@ -190,7 +190,7 @@ pub trait ModuleEnvironment<'data> { table_index: TableIndex, base: Option, offset: usize, - elements: &[FunctionIndex], + elements: Vec, ); /// Declares a memory to the environment fn declare_memory(&mut self, memory: Memory); diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index e0503c0065..ecaf3578ba 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -364,7 +364,7 @@ pub fn parse_elements_section( ParserState::ElementSectionEntryBody(ref elements) => { let elems: Vec = elements.iter().map(|&x| x as FunctionIndex).collect(); - environ.declare_table_elements(table_index, base, offset, &elems) + environ.declare_table_elements(table_index, base, offset, elems) } ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), }; From 1dbc55dadf062e6721b47e2c30dc277b877b6007 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 13 Oct 2017 10:54:44 -0700 Subject: [PATCH 1257/3084] Add a pressure_tracking flag to register banks. This makes it possible to define register banks that opt out of register pressure tracking. This will be used to define banks for special-purpose registers like the CPU flags. The pressure tracker does not need to use resources for a top-level register class in a non-tracked bank. The constant MAX_TOPRCS is renamed to MAX_TRACKED_TOPRCS to indicate that there may be top-level register classes with higher numbers, but they won't require pressure tracking. We won't be tracking register pressure for CPU flags since only one value is allowed to be live at a time. --- lib/cretonne/meta/cdsl/isa.py | 12 +++++-- lib/cretonne/meta/cdsl/registers.py | 15 ++++++-- lib/cretonne/meta/gen_registers.py | 3 ++ lib/cretonne/src/isa/registers.rs | 7 ++-- lib/cretonne/src/regalloc/pressure.rs | 50 +++++++++++++++++++-------- 5 files changed, 66 insertions(+), 21 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index ae517203e1..2fd8e4c7b5 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -136,13 +136,21 @@ class TargetISA(object): # Collect the top-level classes so they get numbered consecutively. for bank in self.regbanks: bank.finish_regclasses() - self.regclasses.extend(bank.toprcs) + # Always get the pressure tracking classes in first. + if bank.pressure_tracking: + self.regclasses.extend(bank.toprcs) # The limit on the number of top-level register classes can be raised. - # This should be coordinated with the `MAX_TOPRCS` constant in + # This should be coordinated with the `MAX_TRACKED_TOPRCS` constant in # `isa/registers.rs`. assert len(self.regclasses) <= 4, "Too many top-level register classes" + # Get the remaining top-level register classes which may exceed + # `MAX_TRACKED_TOPRCS`. + for bank in self.regbanks: + if not bank.pressure_tracking: + self.regclasses.extend(bank.toprcs) + # Collect all of the non-top-level register classes. # They are numbered strictly after the top-level classes. for bank in self.regbanks: diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index 89c6db0a28..37d75d3d58 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -61,17 +61,28 @@ class RegBank(object): :param name: Name of this register bank. :param doc: Documentation string. :param units: Number of register units. + :param pressure_tracking: Enable tracking of register pressure. :param prefix: Prefix for generated unit names. :param names: Special names for the first units. May be shorter than `units`, the remaining units are named using `prefix`. """ - def __init__(self, name, isa, doc, units, prefix='r', names=()): - # type: (str, TargetISA, str, int, str, Sequence[str]) -> None + def __init__( + self, + name, # type: str + isa, # type: TargetISA + doc, # type: str + units, # type: int + pressure_tracking=True, # type: bool + prefix='r', # type: str + names=() # type: Sequence[str] + ): + # type: (...) -> None self.name = name self.isa = isa self.first_unit = 0 self.units = units + self.pressure_tracking = pressure_tracking self.prefix = prefix self.names = names self.classes = list() # type: List[RegClass] diff --git a/lib/cretonne/meta/gen_registers.py b/lib/cretonne/meta/gen_registers.py index 9056441cb1..5917312d96 100644 --- a/lib/cretonne/meta/gen_registers.py +++ b/lib/cretonne/meta/gen_registers.py @@ -28,6 +28,9 @@ def gen_regbank(regbank, fmt): fmt.format('prefix: "{}",', regbank.prefix) fmt.format('first_toprc: {},', regbank.toprcs[0].index) fmt.format('num_toprcs: {},', len(regbank.toprcs)) + fmt.format( + 'pressure_tracking: {},', + 'true' if regbank.pressure_tracking else 'false') def gen_regbank_units(regbank, fmt): diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index db68399cb4..f21c946b99 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -27,10 +27,10 @@ pub type RegUnitMask = [u32; 3]; /// This type should be coordinated with meta/cdsl/isa.py. pub type RegClassMask = u32; -/// Guaranteed maximum number of top-level register classes in any ISA. +/// Guaranteed maximum number of top-level register classes with pressure tracking in any ISA. /// /// This can be increased, but should be coordinated with meta/cdsl/isa.py. -pub const MAX_TOPRCS: usize = 4; +pub const MAX_TRACKED_TOPRCS: usize = 4; /// The register units in a target ISA are divided into disjoint register banks. Each bank covers a /// contiguous range of register units. @@ -63,6 +63,9 @@ pub struct RegBank { /// The top-level register classes in a bank are guaranteed to be numbered sequentially from /// `first_toprc`, and all top-level register classes across banks come before any sub-classes. pub num_toprcs: usize, + + /// Is register pressure tracking enabled for this bank? + pub pressure_tracking: bool, } impl RegBank { diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index 28cf15f48f..7a3a3a00f6 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -36,7 +36,7 @@ // Remove once we're using the pressure tracker. #![allow(dead_code)] -use isa::registers::{RegInfo, MAX_TOPRCS, RegClass, RegClassMask}; +use isa::registers::{RegInfo, MAX_TRACKED_TOPRCS, RegClass, RegClassMask}; use regalloc::AllocatableSet; use std::cmp::min; use std::fmt; @@ -76,7 +76,7 @@ pub struct Pressure { aliased: RegClassMask, // Current register counts per top-level register class. - toprc: [TopRC; MAX_TOPRCS], + toprc: [TopRC; MAX_TRACKED_TOPRCS], } impl Pressure { @@ -88,17 +88,28 @@ impl Pressure { }; // Get the layout of aliasing top-level register classes from the register banks. - for bank in reginfo.banks { + for bank in reginfo.banks.iter() { let first = bank.first_toprc; let num = bank.num_toprcs; - for rc in &mut p.toprc[first..first + num] { - rc.first_toprc = first as u8; - rc.num_toprcs = num as u8; - } - // Flag the top-level register classes with aliases. - if num > 1 { - p.aliased |= ((1 << num) - 1) << first; + if bank.pressure_tracking { + for rc in &mut p.toprc[first..first + num] { + rc.first_toprc = first as u8; + rc.num_toprcs = num as u8; + } + + // Flag the top-level register classes with aliases. + if num > 1 { + p.aliased |= ((1 << num) - 1) << first; + } + } else { + // This bank has no pressure tracking, so its top-level register classes may exceed + // `MAX_TRACKED_TOPRCS`. Fill in dummy entries. + for rc in &mut p.toprc[first..min(first + num, MAX_TRACKED_TOPRCS)] { + // These aren't used if we don't set the `aliased` bit. + rc.first_toprc = !0; + rc.limit = !0; + } } } @@ -123,9 +134,12 @@ impl Pressure { /// pressure should be eased in one of the returned top-level register classes before calling /// `can_take()` to check again. fn check_avail(&self, rc: RegClass) -> RegClassMask { - let entry = &self.toprc[rc.toprc as usize]; + let entry = match self.toprc.get(rc.toprc as usize) { + None => return 0, // Not a pressure tracked bank. + Some(e) => e, + }; let mask = 1 << rc.toprc; - if self.aliased & mask == 0 { + if (self.aliased & mask) == 0 { // This is a simple unaliased top-level register class. if entry.total_count() < entry.limit { 0 @@ -189,12 +203,16 @@ impl Pressure { /// /// This does not check if there are enough registers available. pub fn take(&mut self, rc: RegClass) { - self.toprc[rc.toprc as usize].base_count += 1 + self.toprc.get_mut(rc.toprc as usize).map( + |t| t.base_count += 1, + ); } /// Free a register in `rc`. pub fn free(&mut self, rc: RegClass) { - self.toprc[rc.toprc as usize].base_count -= 1 + self.toprc.get_mut(rc.toprc as usize).map( + |t| t.base_count -= 1, + ); } /// Reset all counts to 0, both base and transient. @@ -211,7 +229,9 @@ impl Pressure { pub fn take_transient(&mut self, rc: RegClass) -> Result<(), RegClassMask> { let mask = self.check_avail(rc); if mask == 0 { - self.toprc[rc.toprc as usize].transient_count += 1; + self.toprc.get_mut(rc.toprc as usize).map(|t| { + t.transient_count += 1 + }); Ok(()) } else { Err(mask) From 0f4f663584f81509f080b70b1fc10276e862da79 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 13 Oct 2017 13:45:49 -0700 Subject: [PATCH 1258/3084] Add register banks for CPU flags to Intel and ARM ISAs. The arm32 ISA technically has separate floating point and integer flags, but the only useful thing you can do with the floating point flags is to copy them ti the integer flags, so there is not need to model them. The arm64 ISA fixes this and the fcmp instruction writes the integer nzcv flags directly. RISC-V does not have CPU flags. --- lib/cretonne/meta/isa/arm32/registers.py | 8 ++++++++ lib/cretonne/meta/isa/arm64/registers.py | 8 ++++++++ lib/cretonne/meta/isa/intel/registers.py | 8 ++++++++ lib/cretonne/src/isa/arm64/registers.rs | 3 ++- 4 files changed, 26 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/isa/arm32/registers.py b/lib/cretonne/meta/isa/arm32/registers.py index 9522057e34..054e95fa0e 100644 --- a/lib/cretonne/meta/isa/arm32/registers.py +++ b/lib/cretonne/meta/isa/arm32/registers.py @@ -29,9 +29,17 @@ IntRegs = RegBank( 'General purpose registers', units=16, prefix='r') +FlagRegs = RegBank( + 'FlagRegs', ISA, + 'Flag registers', + units=1, + pressure_tracking=False, + names=['nzcv']) + GPR = RegClass(IntRegs) S = RegClass(FloatRegs, count=32) D = RegClass(FloatRegs, width=2) Q = RegClass(FloatRegs, width=4) +FLAG = RegClass(FlagRegs) RegClass.extract_names(globals()) diff --git a/lib/cretonne/meta/isa/arm64/registers.py b/lib/cretonne/meta/isa/arm64/registers.py index b39bc917a1..df680b1a14 100644 --- a/lib/cretonne/meta/isa/arm64/registers.py +++ b/lib/cretonne/meta/isa/arm64/registers.py @@ -18,7 +18,15 @@ FloatRegs = RegBank( 'Floating point registers', units=32, prefix='v') +FlagRegs = RegBank( + 'FlagRegs', ISA, + 'Flag registers', + units=1, + pressure_tracking=False, + names=['nzcv']) + GPR = RegClass(IntRegs) FPR = RegClass(FloatRegs) +FLAG = RegClass(FlagRegs) RegClass.extract_names(globals()) diff --git a/lib/cretonne/meta/isa/intel/registers.py b/lib/cretonne/meta/isa/intel/registers.py index 886812d6ce..cbb6701660 100644 --- a/lib/cretonne/meta/isa/intel/registers.py +++ b/lib/cretonne/meta/isa/intel/registers.py @@ -38,11 +38,19 @@ FloatRegs = RegBank( 'SSE floating point registers', units=16, prefix='xmm') +FlagRegs = RegBank( + 'FlagRegs', ISA, + 'Flag registers', + units=1, + pressure_tracking=False, + names=['eflags']) + GPR = RegClass(IntRegs) GPR8 = GPR[0:8] ABCD = GPR[0:4] FPR = RegClass(FloatRegs) FPR8 = FPR[0:8] +FLAG = RegClass(FlagRegs) # Constraints for stack operands. diff --git a/lib/cretonne/src/isa/arm64/registers.rs b/lib/cretonne/src/isa/arm64/registers.rs index 0a0e043a20..d4cee48891 100644 --- a/lib/cretonne/src/isa/arm64/registers.rs +++ b/lib/cretonne/src/isa/arm64/registers.rs @@ -32,6 +32,7 @@ mod tests { assert_eq!(uname(32), "%v0"); assert_eq!(uname(33), "%v1"); assert_eq!(uname(63), "%v31"); - assert_eq!(uname(64), "%INVALID64"); + assert_eq!(uname(64), "%nzcv"); + assert_eq!(uname(65), "%INVALID65"); } } From 5d065c4d8f61a19feccfd72a3cffbf5e5edb1c04 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 13 Oct 2017 16:44:34 -0700 Subject: [PATCH 1259/3084] Add encodings for CPU flags instructions. Branch on flags: brif, brff, Compare integers to flags: ifcmp Compare floats to flags: ffcmp Convert flags to b1: trueif, trueff --- .../filetests/isa/intel/binary32-float.cton | 59 +++++++++ cranelift/filetests/isa/intel/binary32.cton | 58 +++++++++ .../filetests/isa/intel/binary64-float.cton | 59 +++++++++ cranelift/filetests/isa/intel/binary64.cton | 63 ++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 24 +++- lib/cretonne/meta/isa/intel/recipes.py | 113 +++++++++++++++++- lib/cretonne/src/isa/intel/binemit.rs | 63 ++++++++++ 7 files changed, 430 insertions(+), 9 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 6ac14039de..3116207a24 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -220,6 +220,13 @@ ebb0: ; asm: setbe %dl [-,%rdx] v307 = fcmp ule v11, v10 ; bin: 0f 2e d5 0f 96 c2 + ; asm: ucomiss %xmm2, %xmm5 + [-,%eflags] v310 = ffcmp v10, v11 ; bin: 0f 2e ea + ; asm: ucomiss %xmm2, %xmm5 + [-,%eflags] v311 = ffcmp v11, v10 ; bin: 0f 2e d5 + ; asm: ucomiss %xmm5, %xmm5 + [-,%eflags] v312 = ffcmp v10, v10 ; bin: 0f 2e ed + return } @@ -428,5 +435,57 @@ ebb0: ; asm: setbe %dl [-,%rdx] v307 = fcmp ule v11, v10 ; bin: 66 0f 2e d5 0f 96 c2 + ; asm: ucomisd %xmm2, %xmm5 + [-,%eflags] v310 = ffcmp v10, v11 ; bin: 66 0f 2e ea + ; asm: ucomisd %xmm2, %xmm5 + [-,%eflags] v311 = ffcmp v11, v10 ; bin: 66 0f 2e d5 + ; asm: ucomisd %xmm5, %xmm5 + [-,%eflags] v312 = ffcmp v10, v10 ; bin: 66 0f 2e ed + + return +} + +function %cpuflags_float(f32 [%xmm0]) { +ebb0(v0: f32 [%xmm0]): + ; asm: ucomiss %xmm0, %xmm0 + [-,%eflags] v1 = ffcmp v0, v0 ; bin: 0f 2e c0 + + jump ebb1 + +ebb1: + ; asm: jnp ebb1 + brff ord v1, ebb1 ; bin: 7b fe + ; asm: jp ebb1 + brff uno v1, ebb1 ; bin: 7a fc + ; asm: jne ebb1 + brff one v1, ebb1 ; bin: 75 fa + ; asm: je ebb1 + brff ueq v1, ebb1 ; bin: 74 f8 + ; asm: ja ebb1 + brff gt v1, ebb1 ; bin: 77 f6 + ; asm: jae ebb1 + brff ge v1, ebb1 ; bin: 73 f4 + ; asm: jb ebb1 + brff ult v1, ebb1 ; bin: 72 f2 + ; asm: jbe ebb1 + brff ule v1, ebb1 ; bin: 76 f0 + + ; asm: setnp %bl + [-,%rbx] v10 = trueff ord v1 ; bin: 0f 9b c3 + ; asm: setp %bl + [-,%rbx] v11 = trueff uno v1 ; bin: 0f 9a c3 + ; asm: setne %dl + [-,%rdx] v12 = trueff one v1 ; bin: 0f 95 c2 + ; asm: sete %dl + [-,%rdx] v13 = trueff ueq v1 ; bin: 0f 94 c2 + ; asm: seta %al + [-,%rax] v14 = trueff gt v1 ; bin: 0f 97 c0 + ; asm: setae %al + [-,%rax] v15 = trueff ge v1 ; bin: 0f 93 c0 + ; asm: setb %cl + [-,%rcx] v16 = trueff ult v1 ; bin: 0f 92 c1 + ; asm: setbe %cl + [-,%rcx] v17 = trueff ule v1 ; bin: 0f 96 c1 + return } diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index c1a7c7c935..51e739483f 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -421,3 +421,61 @@ ebb0: ebb1: return } + +; CPU flag instructions. +function %cpu_flags() { +ebb0: + [-,%rcx] v1 = iconst.i32 1 + [-,%rsi] v2 = iconst.i32 2 + jump ebb1 + +ebb1: + ; asm: cmpl %esi, %ecx + [-,%eflags] v10 = ifcmp v1, v2 ; bin: 39 f1 + ; asm: cmpl %ecx, %esi + [-,%eflags] v11 = ifcmp v2, v1 ; bin: 39 ce + + ; asm: je ebb1 + brif eq v11, ebb1 ; bin: 74 fa + ; asm: jne ebb1 + brif ne v11, ebb1 ; bin: 75 f8 + ; asm: jl ebb1 + brif slt v11, ebb1 ; bin: 7c f6 + ; asm: jge ebb1 + brif sge v11, ebb1 ; bin: 7d f4 + ; asm: jg ebb1 + brif sgt v11, ebb1 ; bin: 7f f2 + ; asm: jle ebb1 + brif sle v11, ebb1 ; bin: 7e f0 + ; asm: jb ebb1 + brif ult v11, ebb1 ; bin: 72 ee + ; asm: jae ebb1 + brif uge v11, ebb1 ; bin: 73 ec + ; asm: ja ebb1 + brif ugt v11, ebb1 ; bin: 77 ea + ; asm: jbe ebb1 + brif ule v11, ebb1 ; bin: 76 e8 + + ; asm: sete %bl + [-,%rbx] v20 = trueif eq v11 ; bin: 0f 94 c3 + ; asm: setne %bl + [-,%rbx] v21 = trueif ne v11 ; bin: 0f 95 c3 + ; asm: setl %dl + [-,%rdx] v22 = trueif slt v11 ; bin: 0f 9c c2 + ; asm: setge %dl + [-,%rdx] v23 = trueif sge v11 ; bin: 0f 9d c2 + ; asm: setg %bl + [-,%rbx] v24 = trueif sgt v11 ; bin: 0f 9f c3 + ; asm: setle %bl + [-,%rbx] v25 = trueif sle v11 ; bin: 0f 9e c3 + ; asm: setb %dl + [-,%rdx] v26 = trueif ult v11 ; bin: 0f 92 c2 + ; asm: setae %dl + [-,%rdx] v27 = trueif uge v11 ; bin: 0f 93 c2 + ; asm: seta %bl + [-,%rbx] v28 = trueif ugt v11 ; bin: 0f 97 c3 + ; asm: setbe %bl + [-,%rbx] v29 = trueif ule v11 ; bin: 0f 96 c3 + + return +} diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index 4ef385bbb2..86a6247b2e 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -229,6 +229,13 @@ ebb0: ; asm: setbe %dl [-,%rdx] v307 = fcmp ule v11, v10 ; bin: 44 0f 2e d5 0f 96 c2 + ; asm: ucomiss %xmm10, %xmm5 + [-,%eflags] v310 = ffcmp v10, v11 ; bin: 41 0f 2e ea + ; asm: ucomiss %xmm10, %xmm5 + [-,%eflags] v311 = ffcmp v11, v10 ; bin: 44 0f 2e d5 + ; asm: ucomiss %xmm5, %xmm5 + [-,%eflags] v312 = ffcmp v10, v10 ; bin: 0f 2e ed + return } @@ -452,5 +459,57 @@ ebb0: ; asm: setbe %dl [-,%rdx] v307 = fcmp ule v11, v10 ; bin: 66 44 0f 2e d5 0f 96 c2 + ; asm: ucomisd %xmm10, %xmm5 + [-,%eflags] v310 = ffcmp v10, v11 ; bin: 66 41 0f 2e ea + ; asm: ucomisd %xmm10, %xmm5 + [-,%eflags] v311 = ffcmp v11, v10 ; bin: 66 44 0f 2e d5 + ; asm: ucomisd %xmm5, %xmm5 + [-,%eflags] v312 = ffcmp v10, v10 ; bin: 66 0f 2e ed + + return +} + +function %cpuflags_float(f32 [%xmm0]) { +ebb0(v0: f32 [%xmm0]): + ; asm: ucomiss %xmm0, %xmm0 + [-,%eflags] v1 = ffcmp v0, v0 ; bin: 0f 2e c0 + + jump ebb1 + +ebb1: + ; asm: jnp ebb1 + brff ord v1, ebb1 ; bin: 7b fe + ; asm: jp ebb1 + brff uno v1, ebb1 ; bin: 7a fc + ; asm: jne ebb1 + brff one v1, ebb1 ; bin: 75 fa + ; asm: je ebb1 + brff ueq v1, ebb1 ; bin: 74 f8 + ; asm: ja ebb1 + brff gt v1, ebb1 ; bin: 77 f6 + ; asm: jae ebb1 + brff ge v1, ebb1 ; bin: 73 f4 + ; asm: jb ebb1 + brff ult v1, ebb1 ; bin: 72 f2 + ; asm: jbe ebb1 + brff ule v1, ebb1 ; bin: 76 f0 + + ; asm: setnp %bl + [-,%rbx] v10 = trueff ord v1 ; bin: 0f 9b c3 + ; asm: setp %bl + [-,%rbx] v11 = trueff uno v1 ; bin: 0f 9a c3 + ; asm: setne %dl + [-,%rdx] v12 = trueff one v1 ; bin: 0f 95 c2 + ; asm: sete %dl + [-,%rdx] v13 = trueff ueq v1 ; bin: 0f 94 c2 + ; asm: seta %r10b + [-,%r10] v14 = trueff gt v1 ; bin: 41 0f 97 c2 + ; asm: setae %r10b + [-,%r10] v15 = trueff ge v1 ; bin: 41 0f 93 c2 + ; asm: setb %r14b + [-,%r14] v16 = trueff ult v1 ; bin: 41 0f 92 c6 + ; asm: setbe %r14b + [-,%r14] v17 = trueff ule v1 ; bin: 41 0f 96 c6 + return } diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 03cb54e467..6d062c29b6 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -495,6 +495,64 @@ ebb2: jump ebb1 ; bin: eb fd } +; CPU flag instructions. +function %cpu_flags_I64() { +ebb0: + [-,%rcx] v1 = iconst.i64 1 + [-,%r10] v2 = iconst.i64 2 + jump ebb1 + +ebb1: + ; asm: cmpq %r10, %rcx + [-,%eflags] v10 = ifcmp v1, v2 ; bin: 4c 39 d1 + ; asm: cmpq %rcx, %r10 + [-,%eflags] v11 = ifcmp v2, v1 ; bin: 49 39 ca + + ; asm: je ebb1 + brif eq v11, ebb1 ; bin: 74 f8 + ; asm: jne ebb1 + brif ne v11, ebb1 ; bin: 75 f6 + ; asm: jl ebb1 + brif slt v11, ebb1 ; bin: 7c f4 + ; asm: jge ebb1 + brif sge v11, ebb1 ; bin: 7d f2 + ; asm: jg ebb1 + brif sgt v11, ebb1 ; bin: 7f f0 + ; asm: jle ebb1 + brif sle v11, ebb1 ; bin: 7e ee + ; asm: jb ebb1 + brif ult v11, ebb1 ; bin: 72 ec + ; asm: jae ebb1 + brif uge v11, ebb1 ; bin: 73 ea + ; asm: ja ebb1 + brif ugt v11, ebb1 ; bin: 77 e8 + ; asm: jbe ebb1 + brif ule v11, ebb1 ; bin: 76 e6 + + ; asm: sete %bl + [-,%rbx] v20 = trueif eq v11 ; bin: 0f 94 c3 + ; asm: setne %bl + [-,%rbx] v21 = trueif ne v11 ; bin: 0f 95 c3 + ; asm: setl %dl + [-,%rdx] v22 = trueif slt v11 ; bin: 0f 9c c2 + ; asm: setge %dl + [-,%rdx] v23 = trueif sge v11 ; bin: 0f 9d c2 + ; asm: setg %r10b + [-,%r10] v24 = trueif sgt v11 ; bin: 41 0f 9f c2 + ; asm: setle %r10b + [-,%r10] v25 = trueif sle v11 ; bin: 41 0f 9e c2 + ; asm: setb %r14b + [-,%r14] v26 = trueif ult v11 ; bin: 41 0f 92 c6 + ; asm: setae %r14b + [-,%r14] v27 = trueif uge v11 ; bin: 41 0f 93 c6 + ; asm: seta %r11b + [-,%r11] v28 = trueif ugt v11 ; bin: 41 0f 97 c3 + ; asm: setbe %r11b + [-,%r11] v29 = trueif ule v11 ; bin: 41 0f 96 c3 + + return +} + ; Tests for i32 instructions in 64-bit mode. ; ; Note that many i32 instructions can be encoded both with and without a REX @@ -860,6 +918,11 @@ ebb0: ; asm: movl 1032(%rsp), %ecx regfill v1, ss1 -> %rcx ; bin: 8b 8c 24 00000408 + ; asm: cmpl %esi, %ecx + [-,%eflags] v520 = ifcmp v1, v2 ; bin: 39 f1 + ; asm: cmpl %r10d, %esi + [-,%eflags] v521 = ifcmp v2, v3 ; bin: 44 39 d6 + ; asm: testl %ecx, %ecx ; asm: je ebb1x brz v1, ebb1 ; bin: 85 c9 74 18 diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 6c1de71d87..bdbb61fc05 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -279,10 +279,15 @@ I64.enc(base.x_return, *r.ret(0xc3)) # # Branches # -I32.enc(base.jump, *r.jmpb(0xeb)) -I32.enc(base.jump, *r.jmpd(0xe9)) -I64.enc(base.jump, *r.jmpb(0xeb)) -I64.enc(base.jump, *r.jmpd(0xe9)) +enc_both(base.jump, r.jmpb, 0xeb) +enc_both(base.jump, r.jmpd, 0xe9) + +enc_both(base.brif, r.brib, 0x70) +enc_both(base.brif, r.brid, 0x0f, 0x80) + +# Not all float condition codes are legal, see `supported_floatccs`. +enc_both(base.brff, r.brfb, 0x70) +enc_both(base.brff, r.brfd, 0x0f, 0x80) # Note that the tjccd opcode will be prefixed with 0x0f. enc_i32_i64(base.brz, r.tjccb, 0x74) @@ -313,6 +318,14 @@ I64.enc(base.trap, *r.trap(0x0f, 0x0b)) # Comparisons # enc_i32_i64(base.icmp, r.icscc, 0x39) +enc_i32_i64(base.ifcmp, r.rcmp, 0x39) + +# +# Convert flags to bool. +# +# This encodes `b1` as an 8-bit low register with the value 0 or 1. +enc_both(base.trueif, r.seti_abcd, 0x0f, 0x90) +enc_both(base.trueff, r.setf_abcd, 0x0f, 0x90) # # Convert bool to int. @@ -416,3 +429,6 @@ enc_both(base.band_not.f64, r.fax, 0x0f, 0x55) # handled by legalization patterns. enc_both(base.fcmp.f32, r.fcscc, 0x0f, 0x2e) enc_both(base.fcmp.f64, r.fcscc, 0x66, 0x0f, 0x2e) + +enc_both(base.ffcmp.f32, r.fcmp, 0x0f, 0x2e) +enc_both(base.ffcmp.f64, r.fcmp, 0x66, 0x0f, 0x2e) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index e65f0a8217..57506f7833 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -7,10 +7,11 @@ from cdsl.predicates import IsSignedInt, IsEqual, Or from cdsl.registers import RegClass from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry from base.formats import Trap, Call, IndirectCall, Store, Load -from base.formats import IntCompare, FloatCompare -from base.formats import Ternary, Jump, Branch, FuncAddr +from base.formats import IntCompare, FloatCompare, IntCond, FloatCond +from base.formats import Jump, Branch, BranchInt, BranchFloat +from base.formats import Ternary, FuncAddr from base.formats import RegMove, RegSpill, RegFill -from .registers import GPR, ABCD, FPR, GPR8, FPR8, StackGPR32, StackFPR32 +from .registers import GPR, ABCD, FPR, GPR8, FPR8, FLAG, StackGPR32, StackFPR32 from .defs import supported_floatccs try: @@ -250,6 +251,15 @@ class TailRecipe: assert name == obj.name, "Mismatched TailRecipe name: " + name +def floatccs(iform): + # type: (InstructionFormat) -> PredNode + """ + Return an instruction predicate that checks in `iform.cond` is one of the + directly supported floating point condition codes. + """ + return Or(*(IsEqual(iform.cond, cc) for cc in supported_floatccs)) + + # A null unary instruction that takes a GPR register. Can be used for identity # copies and no-op conversions. null = EncRecipe('null', Unary, size=0, ins=GPR, outs=0, emit='') @@ -754,6 +764,100 @@ jmpd = TailRecipe( disp4(destination, func, sink); ''') +brib = TailRecipe( + 'brib', BranchInt, size=1, ins=FLAG.eflags, outs=(), + branch_range=8, + emit=''' + PUT_OP(bits | icc2opc(cond), BASE_REX, sink); + disp1(destination, func, sink); + ''') + +brid = TailRecipe( + 'brid', BranchInt, size=4, ins=FLAG.eflags, outs=(), + branch_range=32, + emit=''' + PUT_OP(bits | icc2opc(cond), BASE_REX, sink); + disp4(destination, func, sink); + ''') + +brfb = TailRecipe( + 'brfb', BranchFloat, size=1, ins=FLAG.eflags, outs=(), + branch_range=8, + instp=floatccs(BranchFloat), + emit=''' + PUT_OP(bits | fcc2opc(cond), BASE_REX, sink); + disp1(destination, func, sink); + ''') + +brfd = TailRecipe( + 'brfd', BranchFloat, size=4, ins=FLAG.eflags, outs=(), + branch_range=32, + instp=floatccs(BranchFloat), + emit=''' + PUT_OP(bits | fcc2opc(cond), BASE_REX, sink); + disp4(destination, func, sink); + ''') + +# +# Test flags and set a register. +# +# These setCC instructions only set the low 8 bits, and they can only write +# ABCD registers without a REX prefix. +# +# Other instruction encodings accepting `b1` inputs have the same constraints +# and only look at the low 8 bits of the input register. +# + +seti = TailRecipe( + 'seti', IntCond, size=1, ins=FLAG.eflags, outs=GPR, + requires_prefix=True, + emit=''' + PUT_OP(bits | icc2opc(cond), rex1(out_reg0), sink); + modrm_r_bits(out_reg0, bits, sink); + ''') +seti_abcd = TailRecipe( + 'seti_abcd', IntCond, size=1, ins=FLAG.eflags, outs=ABCD, + when_prefixed=seti, + emit=''' + PUT_OP(bits | icc2opc(cond), rex1(out_reg0), sink); + modrm_r_bits(out_reg0, bits, sink); + ''') + +setf = TailRecipe( + 'setf', FloatCond, size=1, ins=FLAG.eflags, outs=GPR, + requires_prefix=True, + emit=''' + PUT_OP(bits | fcc2opc(cond), rex1(out_reg0), sink); + modrm_r_bits(out_reg0, bits, sink); + ''') +setf_abcd = TailRecipe( + 'setf_abcd', FloatCond, size=1, ins=FLAG.eflags, outs=ABCD, + when_prefixed=setf, + emit=''' + PUT_OP(bits | fcc2opc(cond), rex1(out_reg0), sink); + modrm_r_bits(out_reg0, bits, sink); + ''') + +# +# Compare and set flags. +# + +# XX /r, MR form. Compare two GPR registers and set flags. +rcmp = TailRecipe( + 'rcmp', Binary, size=1, ins=(GPR, GPR), outs=FLAG.eflags, + emit=''' + PUT_OP(bits, rex2(in_reg0, in_reg1), sink); + modrm_rr(in_reg0, in_reg1, sink); + ''') + +# XX /r, RM form. Compare two FPR registers and set flags. +fcmp = TailRecipe( + 'fcmp', Binary, size=1, ins=(FPR, FPR), outs=FLAG.eflags, + emit=''' + PUT_OP(bits, rex2(in_reg1, in_reg0), sink); + modrm_rr(in_reg1, in_reg0, sink); + ''') + # Test-and-branch. # # This recipe represents the macro fusion of a test and a conditional branch. @@ -926,8 +1030,7 @@ icscc = TailRecipe( # The omission of a `when_prefixed` alternative is deliberate here. fcscc = TailRecipe( 'fcscc', FloatCompare, size=1 + 3, ins=(FPR, FPR), outs=ABCD, - instp=Or(*(IsEqual(FloatCompare.cond, cc) - for cc in supported_floatccs)), + instp=floatccs(FloatCompare), emit=''' // Comparison instruction. PUT_OP(bits, rex2(in_reg1, in_reg0), sink); diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 4358c9da15..f1dcdce095 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -2,6 +2,7 @@ use binemit::{CodeSink, Reloc, bad_encoding}; use ir::{Function, Inst, Ebb, InstructionData, Opcode}; +use ir::condcodes::{IntCC, FloatCC}; use isa::{RegUnit, StackRef, StackBase, StackBaseMask}; use regalloc::RegDiversions; use super::registers::RU; @@ -237,6 +238,68 @@ fn sib_noindex(base: RegUnit, sink: &mut CS) { sink.put1(b); } +/// Get the low 4 bits of an opcode for an integer condition code. +/// +/// Add this offset to a base opcode for: +/// +/// ---- 0x70: Short conditional branch. +/// 0x0f 0x80: Long conditional branch. +/// 0x0f 0x90: SetCC. +/// +fn icc2opc(cond: IntCC) -> u16 { + use ir::condcodes::IntCC::*; + match cond { + // 0x0 = Overflow. + // 0x1 = !Overflow. + UnsignedLessThan => 0x2, + UnsignedGreaterThanOrEqual => 0x3, + Equal => 0x4, + NotEqual => 0x5, + UnsignedLessThanOrEqual => 0x6, + UnsignedGreaterThan => 0x7, + // 0x8 = Sign. + // 0x9 = !Sign. + // 0xa = Parity even. + // 0xb = Parity odd. + SignedLessThan => 0xc, + SignedGreaterThanOrEqual => 0xd, + SignedLessThanOrEqual => 0xe, + SignedGreaterThan => 0xf, + } +} + +/// Get the low 4 bits of an opcode for a floating point condition code. +/// +/// The ucomiss/ucomisd instructions set the EFLAGS bits CF/PF/CF like this: +/// +/// ZPC OSA +/// UN 111 000 +/// GT 000 000 +/// LT 001 000 +/// EQ 100 000 +/// +/// Not all floating point condition codes are supported. +fn fcc2opc(cond: FloatCC) -> u16 { + use ir::condcodes::FloatCC::*; + match cond { + Ordered => 0xb, // EQ|LT|GT => *np (P=0) + Unordered => 0xa, // UN => *p (P=1) + OrderedNotEqual => 0x5, // LT|GT => *ne (Z=0), + UnorderedOrEqual => 0x4, // UN|EQ => *e (Z=1) + GreaterThan => 0x7, // GT => *a (C=0&Z=0) + GreaterThanOrEqual => 0x3, // GT|EQ => *ae (C=0) + UnorderedOrLessThan => 0x2, // UN|LT => *b (C=1) + UnorderedOrLessThanOrEqual => 0x6, // UN|LT|EQ => *be (Z=1|C=1) + Equal | // EQ + NotEqual | // UN|LT|GT + LessThan | // LT + LessThanOrEqual | // LT|EQ + UnorderedOrGreaterThan | // UN|GT + UnorderedOrGreaterThanOrEqual // UN|GT|EQ + => panic!("{} not supported", cond), + } +} + /// Emit a single-byte branch displacement to `destination`. fn disp1(destination: Ebb, func: &Function, sink: &mut CS) { let delta = func.offsets[destination].wrapping_sub(sink.offset() + 1); From 620eb7effef027b643ce9a05e9cd8c765b50dd16 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 16 Oct 2017 14:06:04 -0700 Subject: [PATCH 1260/3084] Add a "clobbers_flags" flag to encoding recipes. On some ISAs like Intel's, all arithmetic instructions set all or some of the CPU flags, so flag values can't be live across these instructions. On ISAs like ARM's Aarch32, flags are clobbered by compact 16-bit encodings but not necessarily by 32-bit encodings of the same instruction. The "clobbers_flags" bit on the encoding recipe is used to indicate if CPU flag values can be live across an instruction, or conversely whether the encoding can be used where flag values are live. --- lib/cretonne/meta/cdsl/isa.py | 37 ++++++++++++--------- lib/cretonne/meta/gen_encoding.py | 3 ++ lib/cretonne/meta/isa/intel/recipes.py | 46 +++++++++++++++++++++++++- lib/cretonne/src/isa/constraints.rs | 5 +++ 4 files changed, 75 insertions(+), 16 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 2fd8e4c7b5..6d06f8ef0b 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -310,29 +310,35 @@ class EncRecipe(object): branch instructions. It is an `(origin, bits)` tuple describing the exact range that can be encoded in a branch instruction. + For ISAs that use CPU flags in `iflags` and `fflags` value types, the + `clobbers_flags` is used to indicate instruction encodings that clobbers + the CPU flags, so they can't be used where a flag value is live. + :param name: Short mnemonic name for this recipe. :param format: All encoded instructions must have this :py:class:`InstructionFormat`. :param size: Number of bytes in the binary encoded instruction. - :param: ins Tuple of register constraints for value operands. - :param: outs Tuple of register constraints for results. - :param: branch_range `(origin, bits)` range for branches. - :param: instp Instruction predicate. - :param: isap ISA predicate. - :param: emit Rust code for binary emission. + :param ins: Tuple of register constraints for value operands. + :param outs: Tuple of register constraints for results. + :param branch_range: `(origin, bits)` range for branches. + :param clobbers_flags: This instruction clobbers `iflags` and `fflags`. + :param instp: Instruction predicate. + :param isap: ISA predicate. + :param emit: Rust code for binary emission. """ def __init__( self, - name, # type: str - format, # type: InstructionFormat - size, # type: int - ins, # type: ConstraintSeq - outs, # type: ConstraintSeq - branch_range=None, # type: BranchRange - instp=None, # type: PredNode - isap=None, # type: PredNode - emit=None # type: str + name, # type: str + format, # type: InstructionFormat + size, # type: int + ins, # type: ConstraintSeq + outs, # type: ConstraintSeq + branch_range=None, # type: BranchRange + clobbers_flags=True, # type: bool + instp=None, # type: PredNode + isap=None, # type: PredNode + emit=None # type: str ): # type: (...) -> None self.name = name @@ -340,6 +346,7 @@ class EncRecipe(object): assert size >= 0 self.size = size self.branch_range = branch_range + self.clobbers_flags = clobbers_flags self.instp = instp self.isap = isap self.emit = emit diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 0228c33839..43b00d37f6 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -766,6 +766,9 @@ def emit_recipe_constraints(isa, fmt): str(any(isinstance(c, Register) for c in r.outs)).lower()) fmt.format('tied_ops: {},', str(bool(tied_i2o)).lower()) + fmt.format( + 'clobbers_flags: {},', + str(bool(r.clobbers_flags)).lower()) def emit_operand_constraints( diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 57506f7833..0aa3f0acd9 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -147,6 +147,7 @@ class TailRecipe: ins, # type: ConstraintSeq outs, # type: ConstraintSeq branch_range=None, # type: int + clobbers_flags=True, # type: bool instp=None, # type: PredNode isap=None, # type: PredNode when_prefixed=None, # type: TailRecipe @@ -160,6 +161,7 @@ class TailRecipe: self.ins = ins self.outs = outs self.branch_range = branch_range + self.clobbers_flags = clobbers_flags self.instp = instp self.isap = isap self.when_prefixed = when_prefixed @@ -194,6 +196,7 @@ class TailRecipe: ins=self.ins, outs=self.outs, branch_range=branch_range, + clobbers_flags=self.clobbers_flags, instp=self.instp, isap=self.isap, emit=replace_put_op(self.emit, name)) @@ -236,6 +239,7 @@ class TailRecipe: ins=self.ins, outs=self.outs, branch_range=branch_range, + clobbers_flags=self.clobbers_flags, instp=self.instp, isap=self.isap, emit=replace_put_op(self.emit, name)) @@ -302,9 +306,10 @@ fax = TailRecipe( ''') # XX /r, but for a unary operator with separate input/output register, like -# copies. MR form. +# copies. MR form, preserving flags. umr = TailRecipe( 'umr', Unary, size=1, ins=GPR, outs=GPR, + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(out_reg0, in_reg0), sink); modrm_rr(out_reg0, in_reg0, sink); @@ -313,6 +318,7 @@ umr = TailRecipe( # Same as umr, but with FPR -> GPR registers. rfumr = TailRecipe( 'rfumr', Unary, size=1, ins=FPR, outs=GPR, + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(out_reg0, in_reg0), sink); modrm_rr(out_reg0, in_reg0, sink); @@ -339,6 +345,7 @@ urm_abcd = TailRecipe( # XX /r, RM form, FPR -> FPR. furm = TailRecipe( 'furm', Unary, size=1, ins=FPR, outs=FPR, + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_rr(in_reg0, out_reg0, sink); @@ -347,6 +354,7 @@ furm = TailRecipe( # XX /r, RM form, GPR -> FPR. frurm = TailRecipe( 'frurm', Unary, size=1, ins=GPR, outs=FPR, + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_rr(in_reg0, out_reg0, sink); @@ -355,6 +363,7 @@ frurm = TailRecipe( # XX /r, RM form, FPR -> GPR. rfurm = TailRecipe( 'rfurm', Unary, size=1, ins=FPR, outs=GPR, + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_rr(in_reg0, out_reg0, sink); @@ -378,6 +387,7 @@ furmi_rnd = TailRecipe( # XX /r, for regmove instructions. rmov = TailRecipe( 'rmov', RegMove, size=1, ins=GPR, outs=(), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(dst, src), sink); modrm_rr(dst, src, sink); @@ -386,6 +396,7 @@ rmov = TailRecipe( # XX /r, for regmove instructions (FPR version, RM encoded). frmov = TailRecipe( 'frmov', RegMove, size=1, ins=FPR, outs=(), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(src, dst), sink); modrm_rr(src, dst, sink); @@ -489,6 +500,7 @@ fnaddr8 = TailRecipe( st = TailRecipe( 'st', Store, size=1, ins=(GPR, GPR), outs=(), instp=IsEqual(Store.offset, 0), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_rm(in_reg1, in_reg0, sink); @@ -500,6 +512,7 @@ st_abcd = TailRecipe( 'st_abcd', Store, size=1, ins=(ABCD, GPR), outs=(), instp=IsEqual(Store.offset, 0), when_prefixed=st, + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_rm(in_reg1, in_reg0, sink); @@ -509,6 +522,7 @@ st_abcd = TailRecipe( fst = TailRecipe( 'fst', Store, size=1, ins=(FPR, GPR), outs=(), instp=IsEqual(Store.offset, 0), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_rm(in_reg1, in_reg0, sink); @@ -518,6 +532,7 @@ fst = TailRecipe( stDisp8 = TailRecipe( 'stDisp8', Store, size=2, ins=(GPR, GPR), outs=(), instp=IsSignedInt(Store.offset, 8), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp8(in_reg1, in_reg0, sink); @@ -528,6 +543,7 @@ stDisp8_abcd = TailRecipe( 'stDisp8_abcd', Store, size=2, ins=(ABCD, GPR), outs=(), instp=IsSignedInt(Store.offset, 8), when_prefixed=stDisp8, + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp8(in_reg1, in_reg0, sink); @@ -537,6 +553,7 @@ stDisp8_abcd = TailRecipe( fstDisp8 = TailRecipe( 'fstDisp8', Store, size=2, ins=(FPR, GPR), outs=(), instp=IsSignedInt(Store.offset, 8), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp8(in_reg1, in_reg0, sink); @@ -547,6 +564,7 @@ fstDisp8 = TailRecipe( # XX /r register-indirect store with 32-bit offset. stDisp32 = TailRecipe( 'stDisp32', Store, size=5, ins=(GPR, GPR), outs=(), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp32(in_reg1, in_reg0, sink); @@ -556,6 +574,7 @@ stDisp32 = TailRecipe( stDisp32_abcd = TailRecipe( 'stDisp32_abcd', Store, size=5, ins=(ABCD, GPR), outs=(), when_prefixed=stDisp32, + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp32(in_reg1, in_reg0, sink); @@ -564,6 +583,7 @@ stDisp32_abcd = TailRecipe( ''') fstDisp32 = TailRecipe( 'fstDisp32', Store, size=5, ins=(FPR, GPR), outs=(), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp32(in_reg1, in_reg0, sink); @@ -574,6 +594,7 @@ fstDisp32 = TailRecipe( # Unary spill with SIB and 32-bit displacement. spSib32 = TailRecipe( 'spSib32', Unary, size=6, ins=GPR, outs=StackGPR32, + clobbers_flags=False, emit=''' let base = stk_base(out_stk0.base); PUT_OP(bits, rex2(base, in_reg0), sink); @@ -583,6 +604,7 @@ spSib32 = TailRecipe( ''') fspSib32 = TailRecipe( 'fspSib32', Unary, size=6, ins=FPR, outs=StackFPR32, + clobbers_flags=False, emit=''' let base = stk_base(out_stk0.base); PUT_OP(bits, rex2(base, in_reg0), sink); @@ -594,6 +616,7 @@ fspSib32 = TailRecipe( # Regspill using RSP-relative addressing. rsp32 = TailRecipe( 'rsp32', RegSpill, size=6, ins=GPR, outs=(), + clobbers_flags=False, emit=''' let dst = StackRef::sp(dst, &func.stack_slots); let base = stk_base(dst.base); @@ -604,6 +627,7 @@ rsp32 = TailRecipe( ''') frsp32 = TailRecipe( 'frsp32', RegSpill, size=6, ins=FPR, outs=(), + clobbers_flags=False, emit=''' let dst = StackRef::sp(dst, &func.stack_slots); let base = stk_base(dst.base); @@ -621,6 +645,7 @@ frsp32 = TailRecipe( ld = TailRecipe( 'ld', Load, size=1, ins=(GPR), outs=(GPR), instp=IsEqual(Load.offset, 0), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_rm(in_reg0, out_reg0, sink); @@ -630,6 +655,7 @@ ld = TailRecipe( fld = TailRecipe( 'fld', Load, size=1, ins=(GPR), outs=(FPR), instp=IsEqual(Load.offset, 0), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_rm(in_reg0, out_reg0, sink); @@ -639,6 +665,7 @@ fld = TailRecipe( ldDisp8 = TailRecipe( 'ldDisp8', Load, size=2, ins=(GPR), outs=(GPR), instp=IsSignedInt(Load.offset, 8), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_disp8(in_reg0, out_reg0, sink); @@ -650,6 +677,7 @@ ldDisp8 = TailRecipe( fldDisp8 = TailRecipe( 'fldDisp8', Load, size=2, ins=(GPR), outs=(FPR), instp=IsSignedInt(Load.offset, 8), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_disp8(in_reg0, out_reg0, sink); @@ -661,6 +689,7 @@ fldDisp8 = TailRecipe( ldDisp32 = TailRecipe( 'ldDisp32', Load, size=5, ins=(GPR), outs=(GPR), instp=IsSignedInt(Load.offset, 32), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_disp32(in_reg0, out_reg0, sink); @@ -672,6 +701,7 @@ ldDisp32 = TailRecipe( fldDisp32 = TailRecipe( 'fldDisp32', Load, size=5, ins=(GPR), outs=(FPR), instp=IsSignedInt(Load.offset, 32), + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_disp32(in_reg0, out_reg0, sink); @@ -682,6 +712,7 @@ fldDisp32 = TailRecipe( # Unary fill with SIB and 32-bit displacement. fiSib32 = TailRecipe( 'fiSib32', Unary, size=6, ins=StackGPR32, outs=GPR, + clobbers_flags=False, emit=''' let base = stk_base(in_stk0.base); PUT_OP(bits, rex2(base, out_reg0), sink); @@ -691,6 +722,7 @@ fiSib32 = TailRecipe( ''') ffiSib32 = TailRecipe( 'ffiSib32', Unary, size=6, ins=StackFPR32, outs=FPR, + clobbers_flags=False, emit=''' let base = stk_base(in_stk0.base); PUT_OP(bits, rex2(base, out_reg0), sink); @@ -702,6 +734,7 @@ ffiSib32 = TailRecipe( # Regfill with RSP-relative 32-bit displacement. rfi32 = TailRecipe( 'rfi32', RegFill, size=6, ins=StackGPR32, outs=(), + clobbers_flags=False, emit=''' let src = StackRef::sp(src, &func.stack_slots); let base = stk_base(src.base); @@ -712,6 +745,7 @@ rfi32 = TailRecipe( ''') frfi32 = TailRecipe( 'frfi32', RegFill, size=6, ins=StackFPR32, outs=(), + clobbers_flags=False, emit=''' let src = StackRef::sp(src, &func.stack_slots); let base = stk_base(src.base); @@ -751,6 +785,7 @@ ret = TailRecipe( jmpb = TailRecipe( 'jmpb', Jump, size=1, ins=(), outs=(), branch_range=8, + clobbers_flags=False, emit=''' PUT_OP(bits, BASE_REX, sink); disp1(destination, func, sink); @@ -759,6 +794,7 @@ jmpb = TailRecipe( jmpd = TailRecipe( 'jmpd', Jump, size=4, ins=(), outs=(), branch_range=32, + clobbers_flags=False, emit=''' PUT_OP(bits, BASE_REX, sink); disp4(destination, func, sink); @@ -767,6 +803,7 @@ jmpd = TailRecipe( brib = TailRecipe( 'brib', BranchInt, size=1, ins=FLAG.eflags, outs=(), branch_range=8, + clobbers_flags=False, emit=''' PUT_OP(bits | icc2opc(cond), BASE_REX, sink); disp1(destination, func, sink); @@ -775,6 +812,7 @@ brib = TailRecipe( brid = TailRecipe( 'brid', BranchInt, size=4, ins=FLAG.eflags, outs=(), branch_range=32, + clobbers_flags=False, emit=''' PUT_OP(bits | icc2opc(cond), BASE_REX, sink); disp4(destination, func, sink); @@ -783,6 +821,7 @@ brid = TailRecipe( brfb = TailRecipe( 'brfb', BranchFloat, size=1, ins=FLAG.eflags, outs=(), branch_range=8, + clobbers_flags=False, instp=floatccs(BranchFloat), emit=''' PUT_OP(bits | fcc2opc(cond), BASE_REX, sink); @@ -792,6 +831,7 @@ brfb = TailRecipe( brfd = TailRecipe( 'brfd', BranchFloat, size=4, ins=FLAG.eflags, outs=(), branch_range=32, + clobbers_flags=False, instp=floatccs(BranchFloat), emit=''' PUT_OP(bits | fcc2opc(cond), BASE_REX, sink); @@ -811,6 +851,7 @@ brfd = TailRecipe( seti = TailRecipe( 'seti', IntCond, size=1, ins=FLAG.eflags, outs=GPR, requires_prefix=True, + clobbers_flags=False, emit=''' PUT_OP(bits | icc2opc(cond), rex1(out_reg0), sink); modrm_r_bits(out_reg0, bits, sink); @@ -818,6 +859,7 @@ seti = TailRecipe( seti_abcd = TailRecipe( 'seti_abcd', IntCond, size=1, ins=FLAG.eflags, outs=ABCD, when_prefixed=seti, + clobbers_flags=False, emit=''' PUT_OP(bits | icc2opc(cond), rex1(out_reg0), sink); modrm_r_bits(out_reg0, bits, sink); @@ -826,6 +868,7 @@ seti_abcd = TailRecipe( setf = TailRecipe( 'setf', FloatCond, size=1, ins=FLAG.eflags, outs=GPR, requires_prefix=True, + clobbers_flags=False, emit=''' PUT_OP(bits | fcc2opc(cond), rex1(out_reg0), sink); modrm_r_bits(out_reg0, bits, sink); @@ -833,6 +876,7 @@ setf = TailRecipe( setf_abcd = TailRecipe( 'setf_abcd', FloatCond, size=1, ins=FLAG.eflags, outs=ABCD, when_prefixed=setf, + clobbers_flags=False, emit=''' PUT_OP(bits | fcc2opc(cond), rex1(out_reg0), sink); modrm_r_bits(out_reg0, bits, sink); diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index 9efcf7bb4f..aa3268187f 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -106,6 +106,11 @@ pub struct RecipeConstraints { /// Are there any tied operands? pub tied_ops: bool, + + /// Does this instruction clobber the CPU flags? + /// + /// When true, SSA values of type `iflags` or `fflags` can not be live across the instruction. + pub clobbers_flags: bool, } impl RecipeConstraints { From e6c6f09e417a3394443922bfdf1669212757fa62 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 16 Oct 2017 09:46:00 -0700 Subject: [PATCH 1261/3084] Tidy some formatting in the generated legalizer.rs. --- lib/cretonne/meta/gen_legalizer.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index c27df484cf..6d792a13d3 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -356,12 +356,12 @@ def gen_xform_group(xgrp, fmt, type_sets): # type: (XFormGroup, Formatter, UniqueTable) -> None fmt.doc_comment("Legalize the instruction pointed to by `pos`.") fmt.line('#[allow(unused_variables,unused_assignments)]') - with fmt.indented( - 'pub fn {}(inst: ir::Inst, ' - 'func: &mut ir::Function, ' - 'cfg: &mut ::flowgraph::ControlFlowGraph) -> ' - 'bool {{'.format(xgrp.name), '}'): - fmt.line('use ir::{InstBuilder};') + with fmt.indented('pub fn {}('.format(xgrp.name)): + fmt.line('inst: ir::Inst,') + fmt.line('func: &mut ir::Function,') + fmt.line('cfg: &mut ::flowgraph::ControlFlowGraph,') + with fmt.indented(') -> bool {', '}'): + fmt.line('use ir::InstBuilder;') fmt.line('use cursor::{Cursor, FuncCursor};') fmt.line('let pos = &mut FuncCursor::new(func).at_inst(inst);') fmt.line('pos.use_srcloc(inst);') From 35989f406914e7271de2367d74d8298739e3f622 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Oct 2017 11:48:29 -0700 Subject: [PATCH 1262/3084] Tidy up unneeded references. --- lib/cretonne/meta/gen_legalizer.py | 2 +- lib/cretonne/src/flowgraph.rs | 2 +- lib/wasm/src/func_translator.rs | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 6d792a13d3..dea0a53950 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -363,7 +363,7 @@ def gen_xform_group(xgrp, fmt, type_sets): with fmt.indented(') -> bool {', '}'): fmt.line('use ir::InstBuilder;') fmt.line('use cursor::{Cursor, FuncCursor};') - fmt.line('let pos = &mut FuncCursor::new(func).at_inst(inst);') + fmt.line('let mut pos = FuncCursor::new(func).at_inst(inst);') fmt.line('pos.use_srcloc(inst);') # Group the xforms by opcode so we can generate a big switch. diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index d4ec4ca717..4dfc72fe32 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -195,7 +195,7 @@ mod tests { let jmp_ebb1_ebb2; { - let cur = &mut FuncCursor::new(&mut func); + let mut cur = FuncCursor::new(&mut func); cur.insert_ebb(ebb0); br_ebb0_ebb2 = cur.ins().brnz(cond, ebb2, &[]); diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index b93c3a368f..9a4395c151 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -80,7 +80,7 @@ impl FuncTranslator { assert_eq!(func.dfg.num_insts(), 0, "Function must be empty"); // This clears the `ILBuilder`. - let builder = &mut FunctionBuilder::new(func, &mut self.il_builder); + let mut builder = FunctionBuilder::new(func, &mut self.il_builder); let entry_block = builder.create_ebb(); builder.switch_to_block(entry_block, &[]); // This also creates values for the arguments. builder.seal_block(entry_block); @@ -88,15 +88,15 @@ impl FuncTranslator { // `environ`. The callback functions may need to insert things in the entry block. builder.ensure_inserted_ebb(); - let num_args = declare_wasm_arguments(builder); + let num_args = declare_wasm_arguments(&mut builder); // Set up the translation state with a single pushed control block representing the whole // function and its return values. let exit_block = builder.create_ebb(); self.state.initialize(&builder.func.signature, exit_block); - parse_local_decls(&mut reader, builder, num_args)?; - parse_function_body(reader, builder, &mut self.state, environ) + parse_local_decls(&mut reader, &mut builder, num_args)?; + parse_function_body(reader, &mut builder, &mut self.state, environ) } } From 6f33066daa2606527b263e95e7ddb6ed5e4c33a1 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 18 Oct 2017 12:30:21 -0700 Subject: [PATCH 1263/3084] Add another link to the documentation. --- README.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.rst b/README.rst index 6f343fb258..cdce22ff92 100644 --- a/README.rst +++ b/README.rst @@ -38,6 +38,9 @@ Predictable performance the quirks of the target architecture. There are no advanced optimizations that sometimes work, sometimes fail. +For more information, see +`the documenation `_. + Building Cretonne ----------------- From c3446ee47227d00a781a32735f3f73ee417fb90a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 17 Oct 2017 12:08:58 -0700 Subject: [PATCH 1264/3084] Add CPU flags value types to the language reference manual. Clean up a few other things in the value types section too. --- cranelift/docs/langref.rst | 37 ++++++++++++++------------------- lib/cretonne/meta/base/types.py | 4 ++-- 2 files changed, 18 insertions(+), 23 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index ea060551be..44066fcabd 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -99,22 +99,7 @@ to represent, more bits are often used when holding a boolean value in a register or in memory. The :type:`b1` type represents an abstract boolean value. It can only exist as an SSA value, it can't be stored in memory or converted to another type. The larger boolean types can be stored in memory. - -.. todo:: Clarify the representation of larger boolean types. - - The multi-bit boolean types can be interpreted in different ways. We could - declare that zero means false and non-zero means true. This may require - unwanted normalization code in some places. - - We could specify a fixed encoding like all ones for true. This would then - lead to undefined behavior if untrusted code uses the multibit booleans - incorrectly. - - Something like this: - - - External code is not allowed to load/store multi-bit booleans or - otherwise expose the representation. - - Each target specifies the exact representation of a multi-bit boolean. +They are represented as either all zero bits or all one bits. .. autoctontype:: b1 .. autoctontype:: b8 @@ -144,6 +129,20 @@ double-double formats. .. autoctontype:: f32 .. autoctontype:: f64 +CPU flags types +--------------- + +Some target ISAs use CPU flags to represent the result of a comparison. These +CPU flags are represented as two value types depending on the type of values +compared. + +Since some ISAs don't have CPU flags, these value types should not be used +until the legalization phase of compilation where the code is adapted to fit +the target ISA. Use instructions like :inst:`icmp` instead. + +.. autoctontype:: iflags +.. autoctontype:: fflags + SIMD vector types ----------------- @@ -194,7 +193,7 @@ Pseudo-types and type classes These are not concrete types, but convenient names uses to refer to real types in this reference. -.. type:: iPtr +.. type:: iAddr A Pointer-sized integer. @@ -225,10 +224,6 @@ in this reference. Any type that can be stored in memory: :type:`Int` or :type:`Float`. -.. type:: Logic - - Either :type:`b1` or :type:`b1xN`. - .. type:: Testable Either :type:`b1` or :type:`iN`. diff --git a/lib/cretonne/meta/base/types.py b/lib/cretonne/meta/base/types.py index b60ca3719f..648220a41b 100644 --- a/lib/cretonne/meta/base/types.py +++ b/lib/cretonne/meta/base/types.py @@ -35,12 +35,12 @@ f64 = FloatType( iflags = FlagsType( 'iflags', """ CPU flags representing the result of an integer comparison. These flags - can be tested with an `intcc` condition code. + can be tested with an :type:`intcc` condition code. """) #: CPU flags from a floating point comparison. fflags = FlagsType( 'fflags', """ CPU flags representing the result of a floating point comparison. These - flags can be tested with a `floatcc` condition code. + flags can be tested with a :type:`floatcc` condition code. """) From b948de16932512e556b48e472e274d43cca31896 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 18 Oct 2017 14:03:46 -0700 Subject: [PATCH 1265/3084] Add a verifier pass for CPU flags. Only one CPU flags value can be live at a time, and some instructions clobber the flags. --- .../filetests/isa/riscv/verify-encoding.cton | 2 +- cranelift/filetests/verifier/flags.cton | 76 +++++++++ lib/cretonne/src/entity/sparse.rs | 5 + lib/cretonne/src/ir/dfg.rs | 7 +- lib/cretonne/src/ir/types.rs | 8 + lib/cretonne/src/verifier/flags.rs | 151 ++++++++++++++++++ lib/cretonne/src/verifier/mod.rs | 17 +- 7 files changed, 259 insertions(+), 7 deletions(-) create mode 100644 cranelift/filetests/verifier/flags.cton create mode 100644 lib/cretonne/src/verifier/flags.rs diff --git a/cranelift/filetests/isa/riscv/verify-encoding.cton b/cranelift/filetests/isa/riscv/verify-encoding.cton index 942179ba23..725b9e5744 100644 --- a/cranelift/filetests/isa/riscv/verify-encoding.cton +++ b/cranelift/filetests/isa/riscv/verify-encoding.cton @@ -16,6 +16,6 @@ function %RV32I(i32 link [%x1]) -> i32 link [%x1] { ebb0(v9999: i32): v1 = iconst.i32 1 v2 = iconst.i32 2 - [R#0,-] v3 = iadd v1, v2 ; error: Instruction encoding R#00 doesn't match any possibilities + [R#0,-] v3 = iadd v1, v2 ; error: encoding R#00 should be R#0c return v9999 } diff --git a/cranelift/filetests/verifier/flags.cton b/cranelift/filetests/verifier/flags.cton new file mode 100644 index 0000000000..dff01244e2 --- /dev/null +++ b/cranelift/filetests/verifier/flags.cton @@ -0,0 +1,76 @@ +test verifier +isa intel + +; Simple, correct use of CPU flags. +function %simple(i32) -> i32 { + ebb0(v0: i32): + [Op1rcmp#39] v1 = ifcmp v0, v0 + [Op2seti_abcd#490] v2 = trueif ugt v1 + [Op2urm_abcd#4b6] v3 = bint.i32 v2 + [Op1ret#c3] return v3 +} + +; Overlapping flag values of different types. +function %overlap(i32, f32) -> i32 { + ebb0(v0: i32, v1: f32): + [Op1rcmp#39] v2 = ifcmp v0, v0 + [Op2fcmp#42e] v3 = ffcmp v1, v1 + [Op2setf_abcd#490] v4 = trueff gt v3 ; error: conflicting live CPU flags: v2 and v3 + [Op2seti_abcd#490] v5 = trueif ugt v2 + [Op1rr#21] v6 = band v4, v5 + [Op2urm_abcd#4b6] v7 = bint.i32 v6 + [Op1ret#c3] return v7 +} + +; CPU flags clobbered by arithmetic. +function %clobbered(i32) -> i32 { + ebb0(v0: i32): + [Op1rcmp#39] v1 = ifcmp v0, v0 + [Op1rr#01] v2 = iadd v0, v0 ; error: encoding clobbers live CPU flags in v1 + [Op2seti_abcd#490] v3 = trueif ugt v1 + [Op2urm_abcd#4b6] v4 = bint.i32 v3 + [Op1ret#c3] return v4 +} + +; CPU flags not clobbered by load. +function %live_across_load(i32) -> i32 { + ebb0(v0: i32): + [Op1rcmp#39] v1 = ifcmp v0, v0 + [Op1ld#8b] v2 = load.i32 v0 + [Op2seti_abcd#490] v3 = trueif ugt v1 + [Op2urm_abcd#4b6] v4 = bint.i32 v3 + [Op1ret#c3] return v4 +} + +; Correct use of CPU flags across EBB. +function %live_across_ebb(i32) -> i32 { + ebb0(v0: i32): + [Op1rcmp#39] v1 = ifcmp v0, v0 + [Op1jmpb#eb] jump ebb1 + ebb1: + [Op2seti_abcd#490] v2 = trueif ugt v1 + [Op2urm_abcd#4b6] v3 = bint.i32 v2 + [Op1ret#c3] return v3 +} + +function %live_across_ebb_backwards(i32) -> i32 { + ebb0(v0: i32): + [Op1jmpb#eb] jump ebb2 + ebb1: + [Op2seti_abcd#490] v2 = trueif ugt v1 + [Op2urm_abcd#4b6] v3 = bint.i32 v2 + [Op1ret#c3] return v3 + ebb2: + [Op1rcmp#39] v1 = ifcmp v0, v0 + [Op1jmpb#eb] jump ebb1 +} + +; Flags live into loop. +function %live_into_loop(i32) -> i32 { + ebb0(v0: i32): + [Op1rcmp#39] v1 = ifcmp v0, v0 + [Op1jmpb#eb] jump ebb1 + ebb1: + [Op2seti_abcd#490] v2 = trueif ugt v1 + [Op1jmpb#eb] jump ebb1 +} \ No newline at end of file diff --git a/lib/cretonne/src/entity/sparse.rs b/lib/cretonne/src/entity/sparse.rs index 1476092121..26be799afd 100644 --- a/lib/cretonne/src/entity/sparse.rs +++ b/lib/cretonne/src/entity/sparse.rs @@ -176,6 +176,11 @@ where None } + /// Remove the last value from the map. + pub fn pop(&mut self) -> Option { + self.dense.pop() + } + /// Get an iterator over the values in the map. /// /// The iteration order is entirely determined by the preceding sequence of `insert` and diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index dbb643e7f3..b3d13c261d 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -4,7 +4,7 @@ use entity::{PrimaryMap, EntityMap}; use isa::TargetIsa; use ir::builder::{InsertBuilder, ReplaceBuilder}; use ir::extfunc::ExtFuncData; -use ir::instructions::{InstructionData, CallInfo}; +use ir::instructions::{InstructionData, CallInfo, BranchInfo}; use ir::layout::{Cursor, LayoutCursorInserter}; use ir::types; use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueList, ValueListPool}; @@ -613,6 +613,11 @@ impl DataFlowGraph { } } + /// Check if `inst` is a branch. + pub fn analyze_branch(&self, inst: Inst) -> BranchInfo { + self.insts[inst].analyze_branch(&self.value_lists) + } + /// Compute the type of an instruction result from opcode constraints and call signatures. /// /// This computes the same sequence of result types that `make_inst_results()` above would diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 1a20f9e511..e0cdcd02ce 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -200,6 +200,14 @@ impl Type { } } + /// Is this a CPU flags type? + pub fn is_flags(self) -> bool { + match self { + IFLAGS | FFLAGS => true, + _ => false, + } + } + /// Get log_2 of the number of lanes in this SIMD vector type. /// /// All SIMD types have a lane count that is a power of two and no larger than 256, so this diff --git a/lib/cretonne/src/verifier/flags.rs b/lib/cretonne/src/verifier/flags.rs new file mode 100644 index 0000000000..c2f3b5340a --- /dev/null +++ b/lib/cretonne/src/verifier/flags.rs @@ -0,0 +1,151 @@ +//! Verify CPU flags values. + +use entity::{EntityMap, SparseSet}; +use flowgraph::ControlFlowGraph; +use ir::instructions::BranchInfo; +use ir; +use isa; +use packed_option::PackedOption; +use std::result; +use verifier::{Result, Error}; + +/// Verify that CPU flags are used correctly. +/// +/// The value types `iflags` and `fflags` represent CPU flags which usually live in a +/// special-purpose register, so they can't be used as freely as other value types that can live in +/// any register. +/// +/// We verify the following conditions: +/// +/// - At most one flags value can be live at a time. +/// - A flags value can not be live across an instruction that clobbers the flags. +/// +/// +pub fn verify_flags( + func: &ir::Function, + cfg: &ControlFlowGraph, + isa: Option<&isa::TargetIsa>, +) -> Result { + let mut verifier = FlagsVerifier { + func, + cfg, + encinfo: isa.map(|isa| isa.encoding_info()), + livein: EntityMap::new(), + }; + verifier.check() +} + +struct FlagsVerifier<'a> { + func: &'a ir::Function, + cfg: &'a ControlFlowGraph, + encinfo: Option, + + /// The single live-in flags value (if any) for each EBB. + livein: EntityMap>, +} + +impl<'a> FlagsVerifier<'a> { + fn check(&mut self) -> Result { + // List of EBBs that need to be processed. EBBs may be re-added to this list when we detect + // that one of their successor blocks needs a live-in flags value. + let mut worklist = SparseSet::new(); + for ebb in self.func.layout.ebbs() { + worklist.insert(ebb); + } + + while let Some(ebb) = worklist.pop() { + if let Some(value) = self.visit_ebb(ebb)? { + // The EBB has live-in flags. Check if the value changed. + match self.livein[ebb].expand() { + // Revisit any predecessor blocks the first time we see a live-in for `ebb`. + None => { + self.livein[ebb] = value.into(); + for &(pred, _) in self.cfg.get_predecessors(ebb) { + worklist.insert(pred); + } + } + Some(old) if old != value => { + return err!(ebb, "conflicting live-in CPU flags: {} and {}", old, value); + } + x => assert_eq!(x, Some(value)), + } + } else { + // Existing live-in flags should never be able to disappear. + assert_eq!(self.livein[ebb].expand(), None); + } + } + + Ok(()) + } + + /// Check flags usage in `ebb` and return the live-in flags value, if any. + fn visit_ebb(&self, ebb: ir::Ebb) -> result::Result, Error> { + // The single currently live flags value. + let mut live_val = None; + + // Visit instructions backwards so we can track liveness accurately. + for inst in self.func.layout.ebb_insts(ebb).rev() { + // Check if `inst` interferes with existing live flags. + if let Some(live) = live_val { + for &res in self.func.dfg.inst_results(inst) { + if res == live { + // We've reached the def of `live_flags`, so it is no longer live above. + live_val = None; + } else if self.func.dfg.value_type(res).is_flags() { + return err!(inst, "{} clobbers live CPU flags in {}", res, live); + } + } + + // Does the instruction have an encoding that clobbers the CPU flags? + if self.encinfo + .as_ref() + .and_then(|ei| ei.operand_constraints(self.func.encodings[inst])) + .map(|c| c.clobbers_flags) + .unwrap_or(false) && live_val.is_some() + { + return err!(inst, "encoding clobbers live CPU flags in {}", live); + } + } + + // Now look for live ranges of CPU flags that end here. + for &arg in self.func.dfg.inst_args(inst) { + if self.func.dfg.value_type(arg).is_flags() { + merge(&mut live_val, arg, inst)?; + } + } + + // Include live-in flags to successor EBBs. + match self.func.dfg.analyze_branch(inst) { + BranchInfo::NotABranch => {} + BranchInfo::SingleDest(dest, _) => { + if let Some(val) = self.livein[dest].expand() { + merge(&mut live_val, val, inst)?; + } + } + BranchInfo::Table(jt) => { + for (_, dest) in self.func.jump_tables[jt].entries() { + if let Some(val) = self.livein[dest].expand() { + merge(&mut live_val, val, inst)?; + } + } + } + } + } + + // Return the required live-in flags value. + Ok(live_val) + } +} + +// Merge live flags values, or return an error on conflicting values. +fn merge(a: &mut Option, b: ir::Value, inst: ir::Inst) -> Result { + if let Some(va) = *a { + if b != va { + return err!(inst, "conflicting live CPU flags: {} and {}", va, b); + } + } else { + *a = Some(b); + } + + Ok(()) +} diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index f143531148..0b9b2db142 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -59,19 +59,20 @@ use dbg::DisplayList; use dominator_tree::DominatorTree; use entity::SparseSet; use flowgraph::ControlFlowGraph; -use ir; use ir::entities::AnyEntity; use ir::instructions::{InstructionFormat, BranchInfo, ResolvedConstraint, CallInfo}; use ir::{types, Function, ValueDef, Ebb, Inst, SigRef, FuncRef, ValueList, JumpTable, StackSlot, StackSlotKind, GlobalVar, Value, Type, Opcode, ValueLoc, ArgumentLoc}; +use ir; use isa::TargetIsa; +use iterators::IteratorExtras; +use self::flags::verify_flags; use settings::{Flags, FlagsOrIsa}; +use std::cmp::Ordering; +use std::collections::BTreeSet; use std::error as std_error; use std::fmt::{self, Display, Formatter, Write}; use std::result; -use std::collections::BTreeSet; -use std::cmp::Ordering; -use iterators::IteratorExtras; pub use self::cssa::verify_cssa; pub use self::liveness::verify_liveness; @@ -95,6 +96,7 @@ macro_rules! err { } mod cssa; +mod flags; mod liveness; mod locations; @@ -980,6 +982,7 @@ impl<'a> Verifier<'a> { if !has_valid_encoding { let mut possible_encodings = String::new(); + let mut multiple_encodings = false; for enc in isa.legal_encodings( &self.func.dfg, @@ -989,6 +992,7 @@ impl<'a> Verifier<'a> { { if possible_encodings.len() != 0 { possible_encodings.push_str(", "); + multiple_encodings = true; } possible_encodings .write_fmt(format_args!("{}", isa.encoding_info().display(enc))) @@ -997,8 +1001,9 @@ impl<'a> Verifier<'a> { return err!( inst, - "Instruction encoding {} doesn't match any possibilities: [{}]", + "encoding {} should be {}{}", isa.encoding_info().display(encoding), + if multiple_encodings { "one of: " } else { "" }, possible_encodings ); } @@ -1085,6 +1090,8 @@ impl<'a> Verifier<'a> { self.verify_return_at_end()?; } + verify_flags(self.func, &self.cfg, self.isa)?; + Ok(()) } } From a57a05cb92d845a303d99214f74c61b0cddf6cf1 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 18 Oct 2017 14:36:23 -0700 Subject: [PATCH 1266/3084] Expand on the floating-point section and provide the NaN rules. --- cranelift/docs/langref.rst | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 44066fcabd..01669f49d8 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -122,9 +122,27 @@ number, others don't care. Floating point types -------------------- -The floating point types have the IEEE semantics that are supported by most -hardware. There is no support for higher-precision types like quads or -double-double formats. +The floating point types have the IEEE 754 semantics that are supported by most +hardware, except that non-default rounding modes, unmasked exceptions, and +exception flags are not currently supported. + +There is currently no support for higher-precision types like quad-precision, +double-double, or extended-precision, nor for narrower-precision types like +half-precision. + +NaNs are encoded following the IEEE 754-2008 recommendation, with quiet NaN +being encoded with the MSB of the trailing significand set to 1, and signaling +NaNs being indicated by the MSB of the trailing significand set to 0. + +Except for bitwise and memory instructions, NaNs returned from arithmetic +instructions are encoded as follows: +- If all NaN inputs to an instruction are quiet NaNs with all bits of the + trailing significand other than the MSB set to 0, the result is a quiet + NaN with a nondeterministic sign bit and all bits of the trailing + significand other than the MSB set to 0. +- Otherwise the result is a quiet NaN with a nondeterministic sign bit + and all bits of the trailing significand other than the MSB set to + nondeterministic values. .. autoctontype:: f32 .. autoctontype:: f64 From 55bc368bf8c02efbc9a48c1ec4d8a232fb266831 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 18 Oct 2017 14:39:21 -0700 Subject: [PATCH 1267/3084] Remove minnum/maxnum. --- cranelift/docs/langref.rst | 8 +++----- lib/cretonne/meta/base/instructions.py | 18 ------------------ 2 files changed, 3 insertions(+), 23 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 01669f49d8..b4fc358283 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -853,16 +853,14 @@ significand bits are always preserved. Minimum and maximum ~~~~~~~~~~~~~~~~~~~ -These instructions return the larger or smaller of their operands. They differ -in their handling of quiet NaN inputs. Note that signaling NaN operands always -cause a NaN result. +These instructions return the larger or smaller of their operands. Note that +unlike the IEEE 754-2008 `minNum` and `maxNum` operations, these instructions +return NaN when either input is NaN. When comparing zeroes, these instructions behave as if :math:`-0.0 < 0.0`. .. autoinst:: fmin -.. autoinst:: fminnum .. autoinst:: fmax -.. autoinst:: fmaxnum Rounding ~~~~~~~~ diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 48c37c280c..3c473d9e22 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -1383,15 +1383,6 @@ fmin = Instruction( """, ins=(x, y), outs=a) -fminnum = Instruction( - 'fminnum', r""" - Floating point minimum, suppressing quiet NaNs. - - If either operand is a quiet NaN, the other operand is returned. If - either operand is a signaling NaN, NaN is returned. - """, - ins=(x, y), outs=a) - a = Operand('a', Float, 'The larger of ``x`` and ``y``') fmax = Instruction( @@ -1402,15 +1393,6 @@ fmax = Instruction( """, ins=(x, y), outs=a) -fmaxnum = Instruction( - 'fmaxnum', r""" - Floating point maximum, suppressing quiet NaNs. - - If either operand is a quiet NaN, the other operand is returned. If - either operand is a signaling NaN, NaN is returned. - """, - ins=(x, y), outs=a) - a = Operand('a', Float, '``x`` rounded to integral value') ceil = Instruction( From ee0f061ee8604ce6ba152a0c424a6dd3c78da171 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sat, 14 Oct 2017 09:46:13 -0700 Subject: [PATCH 1268/3084] Rename "runtime" to "environment". --- lib/wasm/src/code_translator.rs | 15 ++++++++------- lib/wasm/src/{runtime => environ}/dummy.rs | 14 +++++++------- lib/wasm/src/environ/mod.rs | 5 +++++ lib/wasm/src/{runtime => environ}/spec.rs | 0 lib/wasm/src/func_translator.rs | 4 ++-- lib/wasm/src/lib.rs | 7 ++++--- lib/wasm/src/module_translator.rs | 2 +- lib/wasm/src/runtime/mod.rs | 5 ----- lib/wasm/src/sections_translator.rs | 2 +- lib/wasm/src/state.rs | 2 +- 10 files changed, 29 insertions(+), 27 deletions(-) rename lib/wasm/src/{runtime => environ}/dummy.rs (95%) create mode 100644 lib/wasm/src/environ/mod.rs rename lib/wasm/src/{runtime => environ}/spec.rs (100%) delete mode 100644 lib/wasm/src/runtime/mod.rs diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index dab7d3a08d..a4a22435ae 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -11,7 +11,8 @@ //! Another data structure, the translation state, records information concerning unreachable code //! status and about if inserting a return at the end of the function is necessary. //! -//! Some of the WebAssembly instructions need information about the runtime to be translated: +//! Some of the WebAssembly instructions need information about the environment for which they +//! are being translated: //! //! - the loads and stores need the memory base address; //! - the `get_global` et `set_global` instructions depends on how the globals are implemented; @@ -30,7 +31,7 @@ use translation_utils::{f32_translation, f64_translation, type_to_type, num_retu use translation_utils::{TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; use state::{TranslationState, ControlStackFrame}; use std::collections::HashMap; -use runtime::{FuncEnvironment, GlobalValue}; +use environ::{FuncEnvironment, GlobalValue}; use std::u32; /// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted @@ -61,7 +62,7 @@ pub fn translate_operator( builder.def_var(Local(local_index), val); } /********************************** Globals **************************************** - * `get_global` and `set_global` are handled by the runtime. + * `get_global` and `set_global` are handled by the environment. ***********************************************************************************/ Operator::GetGlobal { global_index } => { let val = match state.get_global(builder.func, global_index, environ) { @@ -349,7 +350,7 @@ pub fn translate_operator( } /************************************ Calls **************************************** * The call instructions pop off their arguments from the stack and append their - * return values to it. `call_indirect` needs runtime support because there is an + * return values to it. `call_indirect` needs environment support because there is an * argument referring to an index in the external functions table of the module. ************************************************************************************/ Operator::Call { function_index } => { @@ -380,7 +381,7 @@ pub fn translate_operator( state.pushn(builder.func.dfg.inst_results(call)); } /******************************* Memory management *********************************** - * Memory management is handled by runtime. It is usually translated into calls to + * Memory management is handled by environment. It is usually translated into calls to * special functions. ************************************************************************************/ Operator::GrowMemory { reserved } => { @@ -407,7 +408,7 @@ pub fn translate_operator( } /******************************* Load instructions *********************************** * Wasm specifies an integer alignment flag but we drop it in Cretonne. - * The memory base address is provided by the runtime. + * The memory base address is provided by the environment. * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not ************************************************************************************/ Operator::I32Load8U { memarg: MemoryImmediate { flags: _, offset } } => { @@ -454,7 +455,7 @@ pub fn translate_operator( } /****************************** Store instructions *********************************** * Wasm specifies an integer alignment flag but we drop it in Cretonne. - * The memory base address is provided by the runtime. + * The memory base address is provided by the environment. * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not ************************************************************************************/ Operator::I32Store { memarg: MemoryImmediate { flags: _, offset } } | diff --git a/lib/wasm/src/runtime/dummy.rs b/lib/wasm/src/environ/dummy.rs similarity index 95% rename from lib/wasm/src/runtime/dummy.rs rename to lib/wasm/src/environ/dummy.rs index aab8ed6a4f..b2d4e66151 100644 --- a/lib/wasm/src/runtime/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -1,4 +1,4 @@ -use runtime::{FuncEnvironment, GlobalValue, ModuleEnvironment}; +use environ::{FuncEnvironment, GlobalValue, ModuleEnvironment}; use translation_utils::{Global, Memory, Table, GlobalIndex, TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; use func_translator::FuncTranslator; @@ -65,7 +65,7 @@ pub struct DummyModuleInfo { } impl DummyModuleInfo { - /// Allocates the runtime data structures with the given flags. + /// Allocates the data structures with the given flags. pub fn with_flags(flags: settings::Flags) -> Self { Self { flags, @@ -81,9 +81,9 @@ impl DummyModuleInfo { } } -/// This runtime implementation is a "naïve" one, doing essentially nothing and emitting -/// placeholders when forced to. Don't try to execute code translated with this runtime, it is -/// essentially here for translation debug purposes. +/// This `ModuleEnvironment` implementation is a "naïve" one, doing essentially nothing and +/// emitting placeholders when forced to. Don't try to execute code translated for this +/// environment, essentially here for translation debug purposes. pub struct DummyEnvironment { /// Module information. pub info: DummyModuleInfo, @@ -93,12 +93,12 @@ pub struct DummyEnvironment { } impl DummyEnvironment { - /// Allocates the runtime data structures with default flags. + /// Allocates the data structures with default flags. pub fn default() -> Self { Self::with_flags(settings::Flags::new(&settings::builder())) } - /// Allocates the runtime data structures with the given flags. + /// Allocates the data structures with the given flags. pub fn with_flags(flags: settings::Flags) -> Self { Self { info: DummyModuleInfo::with_flags(flags), diff --git a/lib/wasm/src/environ/mod.rs b/lib/wasm/src/environ/mod.rs new file mode 100644 index 0000000000..a31308b135 --- /dev/null +++ b/lib/wasm/src/environ/mod.rs @@ -0,0 +1,5 @@ +mod spec; +mod dummy; + +pub use environ::spec::{ModuleEnvironment, FuncEnvironment, GlobalValue}; +pub use environ::dummy::DummyEnvironment; diff --git a/lib/wasm/src/runtime/spec.rs b/lib/wasm/src/environ/spec.rs similarity index 100% rename from lib/wasm/src/runtime/spec.rs rename to lib/wasm/src/environ/spec.rs diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 9a4395c151..394da6ab36 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -9,7 +9,7 @@ use cretonne::entity::EntityRef; use cretonne::ir::{self, InstBuilder}; use cretonne::result::{CtonResult, CtonError}; use cton_frontend::{ILBuilder, FunctionBuilder}; -use runtime::FuncEnvironment; +use environ::FuncEnvironment; use state::TranslationState; use translation_utils::Local; use wasmparser::{self, BinaryReader}; @@ -233,7 +233,7 @@ fn cur_srcloc(reader: &BinaryReader) -> ir::SourceLoc { mod tests { use cretonne::{ir, Context}; use cretonne::ir::types::I32; - use runtime::{DummyEnvironment, FuncEnvironment}; + use environ::{DummyEnvironment, FuncEnvironment}; use super::FuncTranslator; #[test] diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 5aaee81d25..b5b79049fe 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -1,6 +1,7 @@ //! Performs the translation from a wasm module in binary format to the in-memory representation //! of the Cretonne IL. More particularly, it translates the code of all the functions bodies and -//! interacts with a runtime implementing the [`ModuleEnvironment`](trait.ModuleEnvironment.html) +//! interacts with an environment implementing the +//! [`ModuleEnvironment`](trait.ModuleEnvironment.html) //! trait to deal with tables, globals and linear memory. //! //! The crate provides a `DummyEnvironment` struct that will allow to translate the code of the @@ -18,13 +19,13 @@ extern crate cretonne; mod code_translator; mod func_translator; mod module_translator; -mod runtime; +mod environ; mod sections_translator; mod state; mod translation_utils; pub use func_translator::FuncTranslator; pub use module_translator::translate_module; -pub use runtime::{FuncEnvironment, ModuleEnvironment, DummyEnvironment, GlobalValue}; +pub use environ::{FuncEnvironment, ModuleEnvironment, DummyEnvironment, GlobalValue}; pub use translation_utils::{FunctionIndex, GlobalIndex, TableIndex, MemoryIndex, SignatureIndex, Global, GlobalInit, Table, Memory}; diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index ef68350e80..87e38a119c 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -5,7 +5,7 @@ use sections_translator::{SectionParsingError, parse_function_signatures, parse_ parse_function_section, parse_export_section, parse_start_section, parse_memory_section, parse_global_section, parse_table_section, parse_elements_section, parse_data_section}; -use runtime::ModuleEnvironment; +use environ::ModuleEnvironment; /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IL /// [`Function`](../cretonne/ir/function/struct.Function.html). diff --git a/lib/wasm/src/runtime/mod.rs b/lib/wasm/src/runtime/mod.rs deleted file mode 100644 index abee71b3f8..0000000000 --- a/lib/wasm/src/runtime/mod.rs +++ /dev/null @@ -1,5 +0,0 @@ -mod spec; -mod dummy; - -pub use runtime::spec::{ModuleEnvironment, FuncEnvironment, GlobalValue}; -pub use runtime::dummy::DummyEnvironment; diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index ecaf3578ba..21ade154ec 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -15,7 +15,7 @@ use wasmparser::{Parser, ParserState, FuncType, ImportSectionEntryType, External MemoryType, Operator}; use wasmparser; use std::str::from_utf8; -use runtime::ModuleEnvironment; +use environ::ModuleEnvironment; pub enum SectionParsingError { WrongSectionContent(String), diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index d862e8d9a0..1c9ce6d8cd 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -4,7 +4,7 @@ //! value and control stacks during the translation of a single function. use cretonne::ir::{self, Ebb, Inst, Value}; -use runtime::{FuncEnvironment, GlobalValue}; +use environ::{FuncEnvironment, GlobalValue}; use std::collections::HashMap; use translation_utils::{GlobalIndex, MemoryIndex, SignatureIndex, FunctionIndex}; From 6d4450805527a4ec377b9f649bc9656558833034 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 18 Oct 2017 14:46:33 -0700 Subject: [PATCH 1269/3084] Sort the glossary alphabetically. --- cranelift/docs/langref.rst | 80 +++++++++++++++++++------------------- 1 file changed, 40 insertions(+), 40 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index b4fc358283..98257a141d 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -1005,19 +1005,32 @@ Glossary .. glossary:: - intermediate language - IL - The language used to describe functions to Cretonne. This reference - describes the syntax and semantics of the Cretonne IL. The IL has two - forms: Textual and an in-memory intermediate representation - (:term:`IR`). + basic block + A maximal sequence of instructions that can only be entered from the + top, and that contains no branch or terminator instructions except for + the last instruction. - intermediate representation - IR - The in-memory representation of :term:`IL`. The data structures - Cretonne uses to represent a program internally are called the - intermediate representation. Cretonne's IR can be converted to text - losslessly. + entry block + The :term:`EBB` that is executed first in a function. Currently, a + Cretonne function must have exactly one entry block which must be the + first block in the function. The types of the entry block arguments must + match the types of arguments in the function signature. + + extended basic block + EBB + A maximal sequence of instructions that can only be entered from the + top, and that contains no :term:`terminator instruction`\s except for + the last one. An EBB can contain conditional branches that can fall + through to the following instructions in the block, but only the first + instruction in the EBB can be a branch target. + + The last instruction in an EBB must be a :term:`terminator instruction`, + so execution cannot flow through to the next EBB in the function. (But + there may be a branch to the next EBB.) + + Note that some textbooks define an EBB as a maximal *subtree* in the + control flow graph where only the root can be a join node. This + definition is not equivalent to Cretonne EBBs. function signature A function signature describes how to call a function. It consists of: @@ -1046,26 +1059,23 @@ Glossary The extended basic blocks which contain all the executable code in a function. The function body follows the function preamble. - basic block - A maximal sequence of instructions that can only be entered from the - top, and that contains no branch or terminator instructions except for - the last instruction. + intermediate language + IL + The language used to describe functions to Cretonne. This reference + describes the syntax and semantics of the Cretonne IL. The IL has two + forms: Textual and an in-memory intermediate representation + (:term:`IR`). - extended basic block - EBB - A maximal sequence of instructions that can only be entered from the - top, and that contains no :term:`terminator instruction`\s except for - the last one. An EBB can contain conditional branches that can fall - through to the following instructions in the block, but only the first - instruction in the EBB can be a branch target. + intermediate representation + IR + The in-memory representation of :term:`IL`. The data structures + Cretonne uses to represent a program internally are called the + intermediate representation. Cretonne's IR can be converted to text + losslessly. - The last instruction in an EBB must be a :term:`terminator instruction`, - so execution cannot flow through to the next EBB in the function. (But - there may be a branch to the next EBB.) - - Note that some textbooks define an EBB as a maximal *subtree* in the - control flow graph where only the root can be a join node. This - definition is not equivalent to Cretonne EBBs. + stack slot + A fixed size memory allocation in the current function's activation + frame. Also called a local variable. terminator instruction A control flow instruction that unconditionally directs the flow of @@ -1075,13 +1085,3 @@ Glossary The basic terminator instructions are :inst:`br`, :inst:`return`, and :inst:`trap`. Conditional branches and instructions that trap conditionally are not terminator instructions. - - entry block - The :term:`EBB` that is executed first in a function. Currently, a - Cretonne function must have exactly one entry block which must be the - first block in the function. The types of the entry block arguments must - match the types of arguments in the function signature. - - stack slot - A fixed size memory allocation in the current function's activation - frame. Also called a local variable. From 57b81a179efdffeca724549a4c117584d7a1f51b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Oct 2017 11:33:53 -0700 Subject: [PATCH 1270/3084] Move the CursorBase trait into the cursor module. Also move the CursorPosition type into the cursor module. Move layout::cursor into the tests module as LayoutCursor and remove its ability to insert instructions via the dfg.ins() method. This cursor type is only used in the layout unit tests now. The FuncCursor and EncCursor types are the commonly used cursors now. --- lib/cretonne/src/cursor.rs | 555 ++++++++++++++++++++++++- lib/cretonne/src/ir/dfg.rs | 12 +- lib/cretonne/src/ir/layout.rs | 734 +++------------------------------- lib/cretonne/src/ir/mod.rs | 2 +- 4 files changed, 614 insertions(+), 689 deletions(-) diff --git a/lib/cretonne/src/cursor.rs b/lib/cretonne/src/cursor.rs index 5e3ea866f5..56ad657043 100644 --- a/lib/cretonne/src/cursor.rs +++ b/lib/cretonne/src/cursor.rs @@ -5,10 +5,557 @@ use ir; use isa::TargetIsa; -// Re-export these types, anticipating their being moved here. -pub use ir::layout::CursorBase as Cursor; -pub use ir::layout::CursorPosition; -pub use ir::layout::Cursor as LayoutCursor; +/// The possible positions of a cursor. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum CursorPosition { + /// Cursor is not pointing anywhere. No instructions can be inserted. + Nowhere, + /// Cursor is pointing at an existing instruction. + /// New instructions will be inserted *before* the current instruction. + At(ir::Inst), + /// Cursor is before the beginning of an EBB. No instructions can be inserted. Calling + /// `next_inst()` will move to the first instruction in the EBB. + Before(ir::Ebb), + /// Cursor is pointing after the end of an EBB. + /// New instructions will be appended to the EBB. + After(ir::Ebb), +} + +/// All cursor types implement the `Cursor` which provides common navigation operations. +pub trait Cursor { + /// Get the current cursor position. + fn position(&self) -> CursorPosition; + + /// Set the current position. + fn set_position(&mut self, pos: CursorPosition); + + /// Get the source location that should be assigned to new instructions. + fn srcloc(&self) -> ir::SourceLoc; + + /// Set the source location that should be assigned to new instructions. + fn set_srcloc(&mut self, srcloc: ir::SourceLoc); + + /// Borrow a reference to the function layout that this cursor is navigating. + fn layout(&self) -> &ir::Layout; + + /// Borrow a mutable reference to the function layout that this cursor is navigating. + fn layout_mut(&mut self) -> &mut ir::Layout; + + /// Exchange this cursor for one with a set source location. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, SourceLoc}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_func(func: &mut Function, srcloc: SourceLoc) { + /// let mut pos = FuncCursor::new(func).with_srcloc(srcloc); + /// + /// // Use `pos`... + /// } + /// ``` + fn with_srcloc(mut self, srcloc: ir::SourceLoc) -> Self + where + Self: Sized, + { + self.set_srcloc(srcloc); + self + } + + /// Rebuild this cursor positioned at `pos`. + fn at_position(mut self, pos: CursorPosition) -> Self + where + Self: Sized, + { + self.set_position(pos); + self + } + + /// Rebuild this cursor positioned at `inst`. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_func(func: &mut Function, inst: Inst) { + /// let mut pos = FuncCursor::new(func).at_inst(inst); + /// + /// // Use `pos`... + /// } + /// ``` + fn at_inst(mut self, inst: ir::Inst) -> Self + where + Self: Sized, + { + self.goto_inst(inst); + self + } + + /// Rebuild this cursor positioned at the first insertion point for `ebb`. + /// This differs from `at_first_inst` in that it doesn't assume that any + /// instructions have been inserted into `ebb` yet. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_func(func: &mut Function, ebb: Ebb) { + /// let mut pos = FuncCursor::new(func).at_first_insertion_point(ebb); + /// + /// // Use `pos`... + /// } + /// ``` + fn at_first_insertion_point(mut self, ebb: ir::Ebb) -> Self + where + Self: Sized, + { + self.goto_first_insertion_point(ebb); + self + } + + /// Rebuild this cursor positioned at the first instruction in `ebb`. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_func(func: &mut Function, ebb: Ebb) { + /// let mut pos = FuncCursor::new(func).at_first_inst(ebb); + /// + /// // Use `pos`... + /// } + /// ``` + fn at_first_inst(mut self, ebb: ir::Ebb) -> Self + where + Self: Sized, + { + self.goto_first_inst(ebb); + self + } + + /// Rebuild this cursor positioned at the last instruction in `ebb`. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_func(func: &mut Function, ebb: Ebb) { + /// let mut pos = FuncCursor::new(func).at_last_inst(ebb); + /// + /// // Use `pos`... + /// } + /// ``` + fn at_last_inst(mut self, ebb: ir::Ebb) -> Self + where + Self: Sized, + { + self.goto_last_inst(ebb); + self + } + + /// Rebuild this cursor positioned after `inst`. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_func(func: &mut Function, inst: Inst) { + /// let mut pos = FuncCursor::new(func).after_inst(inst); + /// + /// // Use `pos`... + /// } + /// ``` + fn after_inst(mut self, inst: ir::Inst) -> Self + where + Self: Sized, + { + self.goto_after_inst(inst); + self + } + + /// Rebuild this cursor positioned at the top of `ebb`. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_func(func: &mut Function, ebb: Ebb) { + /// let mut pos = FuncCursor::new(func).at_top(ebb); + /// + /// // Use `pos`... + /// } + /// ``` + fn at_top(mut self, ebb: ir::Ebb) -> Self + where + Self: Sized, + { + self.goto_top(ebb); + self + } + + /// Rebuild this cursor positioned at the bottom of `ebb`. + /// + /// This is intended to be used as a builder method: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb, Inst}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_func(func: &mut Function, ebb: Ebb) { + /// let mut pos = FuncCursor::new(func).at_bottom(ebb); + /// + /// // Use `pos`... + /// } + /// ``` + fn at_bottom(mut self, ebb: ir::Ebb) -> Self + where + Self: Sized, + { + self.goto_bottom(ebb); + self + } + + /// Get the EBB corresponding to the current position. + fn current_ebb(&self) -> Option { + use self::CursorPosition::*; + match self.position() { + Nowhere => None, + At(inst) => self.layout().inst_ebb(inst), + Before(ebb) | After(ebb) => Some(ebb), + } + } + + /// Get the instruction corresponding to the current position, if any. + fn current_inst(&self) -> Option { + use self::CursorPosition::*; + match self.position() { + At(inst) => Some(inst), + _ => None, + } + } + + /// Go to the position after a specific instruction, which must be inserted + /// in the layout. New instructions will be inserted after `inst`. + fn goto_after_inst(&mut self, inst: ir::Inst) { + debug_assert!(self.layout().inst_ebb(inst).is_some()); + let new_pos = if let Some(next) = self.layout().next_inst(inst) { + CursorPosition::At(next) + } else { + CursorPosition::After(self.layout().inst_ebb(inst).expect( + "current instruction removed?", + )) + }; + self.set_position(new_pos); + } + + /// Go to a specific instruction which must be inserted in the layout. + /// New instructions will be inserted before `inst`. + fn goto_inst(&mut self, inst: ir::Inst) { + assert!(self.layout().inst_ebb(inst).is_some()); + self.set_position(CursorPosition::At(inst)); + } + + /// Go to the position for inserting instructions at the beginning of `ebb`, + /// which unlike `goto_first_inst` doesn't assume that any instructions have + /// been inserted into `ebb` yet. + fn goto_first_insertion_point(&mut self, ebb: ir::Ebb) { + if let Some(inst) = self.layout().first_inst(ebb) { + self.goto_inst(inst); + } else { + self.goto_bottom(ebb); + } + } + + /// Go to the first instruction in `ebb`. + fn goto_first_inst(&mut self, ebb: ir::Ebb) { + let inst = self.layout().first_inst(ebb).expect("Empty EBB"); + self.goto_inst(inst); + } + + /// Go to the last instruction in `ebb`. + fn goto_last_inst(&mut self, ebb: ir::Ebb) { + let inst = self.layout().last_inst(ebb).expect("Empty EBB"); + self.goto_inst(inst); + } + + /// Go to the top of `ebb` which must be inserted into the layout. + /// At this position, instructions cannot be inserted, but `next_inst()` will move to the first + /// instruction in `ebb`. + fn goto_top(&mut self, ebb: ir::Ebb) { + assert!(self.layout().is_ebb_inserted(ebb)); + self.set_position(CursorPosition::Before(ebb)); + } + + /// Go to the bottom of `ebb` which must be inserted into the layout. + /// At this position, inserted instructions will be appended to `ebb`. + fn goto_bottom(&mut self, ebb: ir::Ebb) { + assert!(self.layout().is_ebb_inserted(ebb)); + self.set_position(CursorPosition::After(ebb)); + } + + /// Go to the top of the next EBB in layout order and return it. + /// + /// - If the cursor wasn't pointing at anything, go to the top of the first EBB in the + /// function. + /// - If there are no more EBBs, leave the cursor pointing at nothing and return `None`. + /// + /// # Examples + /// + /// The `next_ebb()` method is intended for iterating over the EBBs in layout order: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_func(func: &mut Function) { + /// let mut cursor = FuncCursor::new(func); + /// while let Some(ebb) = cursor.next_ebb() { + /// // Edit ebb. + /// } + /// } + /// ``` + fn next_ebb(&mut self) -> Option { + let next = if let Some(ebb) = self.current_ebb() { + self.layout().next_ebb(ebb) + } else { + self.layout().entry_block() + }; + self.set_position(match next { + Some(ebb) => CursorPosition::Before(ebb), + None => CursorPosition::Nowhere, + }); + next + } + + /// Go to the bottom of the previous EBB in layout order and return it. + /// + /// - If the cursor wasn't pointing at anything, go to the bottom of the last EBB in the + /// function. + /// - If there are no more EBBs, leave the cursor pointing at nothing and return `None`. + /// + /// # Examples + /// + /// The `prev_ebb()` method is intended for iterating over the EBBs in backwards layout order: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_func(func: &mut Function) { + /// let mut cursor = FuncCursor::new(func); + /// while let Some(ebb) = cursor.prev_ebb() { + /// // Edit ebb. + /// } + /// } + /// ``` + fn prev_ebb(&mut self) -> Option { + let prev = if let Some(ebb) = self.current_ebb() { + self.layout().prev_ebb(ebb) + } else { + self.layout().last_ebb() + }; + self.set_position(match prev { + Some(ebb) => CursorPosition::After(ebb), + None => CursorPosition::Nowhere, + }); + prev + } + + /// Move to the next instruction in the same EBB and return it. + /// + /// - If the cursor was positioned before an EBB, go to the first instruction in that EBB. + /// - If there are no more instructions in the EBB, go to the `After(ebb)` position and return + /// `None`. + /// - If the cursor wasn't pointing anywhere, keep doing that. + /// + /// This method will never move the cursor to a different EBB. + /// + /// # Examples + /// + /// The `next_inst()` method is intended for iterating over the instructions in an EBB like + /// this: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_ebb(func: &mut Function, ebb: Ebb) { + /// let mut cursor = FuncCursor::new(func).at_top(ebb); + /// while let Some(inst) = cursor.next_inst() { + /// // Edit instructions... + /// } + /// } + /// ``` + /// The loop body can insert and remove instructions via the cursor. + /// + /// Iterating over all the instructions in a function looks like this: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_func(func: &mut Function) { + /// let mut cursor = FuncCursor::new(func); + /// while let Some(ebb) = cursor.next_ebb() { + /// while let Some(inst) = cursor.next_inst() { + /// // Edit instructions... + /// } + /// } + /// } + /// ``` + fn next_inst(&mut self) -> Option { + use self::CursorPosition::*; + match self.position() { + Nowhere | After(..) => None, + At(inst) => { + if let Some(next) = self.layout().next_inst(inst) { + self.set_position(At(next)); + Some(next) + } else { + let pos = After(self.layout().inst_ebb(inst).expect( + "current instruction removed?", + )); + self.set_position(pos); + None + } + } + Before(ebb) => { + if let Some(next) = self.layout().first_inst(ebb) { + self.set_position(At(next)); + Some(next) + } else { + self.set_position(After(ebb)); + None + } + } + } + } + + /// Move to the previous instruction in the same EBB and return it. + /// + /// - If the cursor was positioned after an EBB, go to the last instruction in that EBB. + /// - If there are no more instructions in the EBB, go to the `Before(ebb)` position and return + /// `None`. + /// - If the cursor wasn't pointing anywhere, keep doing that. + /// + /// This method will never move the cursor to a different EBB. + /// + /// # Examples + /// + /// The `prev_inst()` method is intended for iterating backwards over the instructions in an + /// EBB like this: + /// + /// ``` + /// # use cretonne::ir::{Function, Ebb}; + /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// fn edit_ebb(func: &mut Function, ebb: Ebb) { + /// let mut cursor = FuncCursor::new(func).at_bottom(ebb); + /// while let Some(inst) = cursor.prev_inst() { + /// // Edit instructions... + /// } + /// } + /// ``` + fn prev_inst(&mut self) -> Option { + use self::CursorPosition::*; + match self.position() { + Nowhere | Before(..) => None, + At(inst) => { + if let Some(prev) = self.layout().prev_inst(inst) { + self.set_position(At(prev)); + Some(prev) + } else { + let pos = Before(self.layout().inst_ebb(inst).expect( + "current instruction removed?", + )); + self.set_position(pos); + None + } + } + After(ebb) => { + if let Some(prev) = self.layout().last_inst(ebb) { + self.set_position(At(prev)); + Some(prev) + } else { + self.set_position(Before(ebb)); + None + } + } + } + } + + /// Insert an instruction at the current position. + /// + /// - If pointing at an instruction, the new instruction is inserted before the current + /// instruction. + /// - If pointing at the bottom of an EBB, the new instruction is appended to the EBB. + /// - Otherwise panic. + /// + /// In either case, the cursor is not moved, such that repeated calls to `insert_inst()` causes + /// instructions to appear in insertion order in the EBB. + fn insert_inst(&mut self, inst: ir::Inst) { + use self::CursorPosition::*; + match self.position() { + Nowhere | Before(..) => panic!("Invalid insert_inst position"), + At(cur) => self.layout_mut().insert_inst(inst, cur), + After(ebb) => self.layout_mut().append_inst(inst, ebb), + } + } + + /// Remove the instruction under the cursor. + /// + /// The cursor is left pointing at the position following the current instruction. + /// + /// Return the instruction that was removed. + fn remove_inst(&mut self) -> ir::Inst { + let inst = self.current_inst().expect("No instruction to remove"); + self.next_inst(); + self.layout_mut().remove_inst(inst); + inst + } + + /// Remove the instruction under the cursor. + /// + /// The cursor is left pointing at the position preceding the current instruction. + /// + /// Return the instruction that was removed. + fn remove_inst_and_step_back(&mut self) -> ir::Inst { + let inst = self.current_inst().expect("No instruction to remove"); + self.prev_inst(); + self.layout_mut().remove_inst(inst); + inst + } + + /// Insert an EBB at the current position and switch to it. + /// + /// As far as possible, this method behaves as if the EBB header were an instruction inserted + /// at the current position. + /// + /// - If the cursor is pointing at an existing instruction, *the current EBB is split in two* + /// and the current instruction becomes the first instruction in the inserted EBB. + /// - If the cursor points at the bottom of an EBB, the new EBB is inserted after the current + /// one, and moved to the bottom of the new EBB where instructions can be appended. + /// - If the cursor points to the top of an EBB, the new EBB is inserted above the current one. + /// - If the cursor is not pointing at anything, the new EBB is placed last in the layout. + /// + /// This means that it is always valid to call this method, and it always leaves the cursor in + /// a state that will insert instructions into the new EBB. + fn insert_ebb(&mut self, new_ebb: ir::Ebb) { + use self::CursorPosition::*; + match self.position() { + At(inst) => { + self.layout_mut().split_ebb(new_ebb, inst); + // All other cases move to `After(ebb)`, but in this case we'll stay `At(inst)`. + return; + } + Nowhere => self.layout_mut().append_ebb(new_ebb), + Before(ebb) => self.layout_mut().insert_ebb(new_ebb, ebb), + After(ebb) => self.layout_mut().insert_ebb_after(new_ebb, ebb), + } + // For everything but `At(inst)` we end up appending to the new EBB. + self.set_position(After(new_ebb)); + } +} /// Function cursor. /// diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index b3d13c261d..d663130280 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -2,10 +2,9 @@ use entity::{PrimaryMap, EntityMap}; use isa::TargetIsa; -use ir::builder::{InsertBuilder, ReplaceBuilder}; +use ir::builder::ReplaceBuilder; use ir::extfunc::ExtFuncData; use ir::instructions::{InstructionData, CallInfo, BranchInfo}; -use ir::layout::{Cursor, LayoutCursorInserter}; use ir::types; use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueList, ValueListPool}; use write::write_operands; @@ -479,14 +478,6 @@ impl DataFlowGraph { total_results } - /// Create an `InsertBuilder` that will insert an instruction at the cursor's current position. - pub fn ins<'c, 'fc: 'c, 'fd>( - &'fd mut self, - at: &'c mut Cursor<'fc>, - ) -> InsertBuilder<'fd, LayoutCursorInserter<'c, 'fc, 'fd>> { - InsertBuilder::new(LayoutCursorInserter::new(at, self)) - } - /// Create a `ReplaceBuilder` that will replace `inst` with a new instruction in place. pub fn replace(&mut self, inst: Inst) -> ReplaceBuilder { ReplaceBuilder::new(self, inst) @@ -508,7 +499,6 @@ impl DataFlowGraph { self.results[inst].clear(&mut self.value_lists) } - /// Attach an existing value to the result value list for `inst`. /// /// The `res` value is appended to the end of the result list. diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 1664a3ba4d..8bb61cb11f 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -3,13 +3,12 @@ //! The order of extended basic blocks in a function and the order of instructions in an EBB is //! determined by the `Layout` data structure defined in this module. +use entity::EntityMap; +use ir::{Ebb, Inst}; +use ir::progpoint::{ProgramOrder, ExpandedProgramPoint}; +use packed_option::PackedOption; use std::cmp; use std::iter::{Iterator, IntoIterator}; -use entity::EntityMap; -use packed_option::PackedOption; -use ir::{Ebb, Inst, Type, DataFlowGraph, SourceLoc, SourceLocs}; -use ir::builder::InstInserterBase; -use ir::progpoint::{ProgramOrder, ExpandedProgramPoint}; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not /// contain definitions of instructions or EBBs, but depends on `Inst` and `Ebb` entity references @@ -405,6 +404,11 @@ impl Layout { self.last_ebb } + /// Get the block preceding `ebb` in the layout order. + pub fn prev_ebb(&self, ebb: Ebb) -> Option { + self.ebbs[ebb].prev.expand() + } + /// Get the block following `ebb` in the layout order. pub fn next_ebb(&self, ebb: Ebb) -> Option { self.ebbs[ebb].next.expand() @@ -432,7 +436,7 @@ impl<'f> Iterator for Ebbs<'f> { fn next(&mut self) -> Option { match self.next { Some(ebb) => { - self.next = self.layout.ebbs[ebb].next.expand(); + self.next = self.layout.next_ebb(ebb); Some(ebb) } None => None, @@ -690,673 +694,57 @@ impl<'f> DoubleEndedIterator for Insts<'f> { } -/// Layout Cursor. -/// -/// A `Cursor` represents a position in a function layout where instructions can be inserted and -/// removed. It can be used to iterate through the instructions of a function while editing them at -/// the same time. A normal instruction iterator can't do this since it holds an immutable -/// reference to the Layout. -/// -/// When new instructions are added, the cursor can either append them to an EBB or insert them -/// before the current instruction. -pub struct Cursor<'f> { - /// Borrowed function layout. Public so it can be re-borrowed from this cursor. - pub layout: &'f mut Layout, - /// Borrowed source locations. - pub srclocs: Option<&'f mut SourceLocs>, - pos: CursorPosition, - srcloc: SourceLoc, -} - -/// The possible positions of a cursor. -#[derive(Clone, Copy, PartialEq, Eq, Debug)] -pub enum CursorPosition { - /// Cursor is not pointing anywhere. No instructions can be inserted. - Nowhere, - /// Cursor is pointing at an existing instruction. - /// New instructions will be inserted *before* the current instruction. - At(Inst), - /// Cursor is before the beginning of an EBB. No instructions can be inserted. Calling - /// `next_inst()` will move to the first instruction in the EBB. - Before(Ebb), - /// Cursor is pointing after the end of an EBB. - /// New instructions will be appended to the EBB. - After(Ebb), -} - -/// All cursor types implement the `CursorBase` which provides common navigation operations. -pub trait CursorBase { - /// Get the current cursor position. - fn position(&self) -> CursorPosition; - - /// Set the current position. - fn set_position(&mut self, pos: CursorPosition); - - /// Get the source location that should be assigned to new instructions. - fn srcloc(&self) -> SourceLoc; - - /// Set the source location that should be assigned to new instructions. - fn set_srcloc(&mut self, srcloc: SourceLoc); - - /// Borrow a reference to the function layout that this cursor is navigating. - fn layout(&self) -> &Layout; - - /// Borrow a mutable reference to the function layout that this cursor is navigating. - fn layout_mut(&mut self) -> &mut Layout; - - /// Exchange this cursor for one with a set source location. - /// - /// This is intended to be used as a builder method: - /// - /// ``` - /// # use cretonne::ir::{Function, Ebb, SourceLoc}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; - /// fn edit_func(func: &mut Function, srcloc: SourceLoc) { - /// let mut pos = FuncCursor::new(func).with_srcloc(srcloc); - /// - /// // Use `pos`... - /// } - /// ``` - fn with_srcloc(mut self, srcloc: SourceLoc) -> Self - where - Self: Sized, - { - self.set_srcloc(srcloc); - self - } - - /// Rebuild this cursor positioned at `pos`. - fn at_position(mut self, pos: CursorPosition) -> Self - where - Self: Sized, - { - self.set_position(pos); - self - } - - /// Rebuild this cursor positioned at `inst`. - /// - /// This is intended to be used as a builder method: - /// - /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; - /// fn edit_func(func: &mut Function, inst: Inst) { - /// let mut pos = FuncCursor::new(func).at_inst(inst); - /// - /// // Use `pos`... - /// } - /// ``` - fn at_inst(mut self, inst: Inst) -> Self - where - Self: Sized, - { - self.goto_inst(inst); - self - } - - /// Rebuild this cursor positioned at the first insertion point for `ebb`. - /// This differs from `at_first_inst` in that it doesn't assume that any - /// instructions have been inserted into `ebb` yet. - /// - /// This is intended to be used as a builder method: - /// - /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; - /// fn edit_func(func: &mut Function, ebb: Ebb) { - /// let mut pos = FuncCursor::new(func).at_first_insertion_point(ebb); - /// - /// // Use `pos`... - /// } - /// ``` - fn at_first_insertion_point(mut self, ebb: Ebb) -> Self - where - Self: Sized, - { - self.goto_first_insertion_point(ebb); - self - } - - /// Rebuild this cursor positioned at the first instruction in `ebb`. - /// - /// This is intended to be used as a builder method: - /// - /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; - /// fn edit_func(func: &mut Function, ebb: Ebb) { - /// let mut pos = FuncCursor::new(func).at_first_inst(ebb); - /// - /// // Use `pos`... - /// } - /// ``` - fn at_first_inst(mut self, ebb: Ebb) -> Self - where - Self: Sized, - { - self.goto_first_inst(ebb); - self - } - - /// Rebuild this cursor positioned at the last instruction in `ebb`. - /// - /// This is intended to be used as a builder method: - /// - /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; - /// fn edit_func(func: &mut Function, ebb: Ebb) { - /// let mut pos = FuncCursor::new(func).at_last_inst(ebb); - /// - /// // Use `pos`... - /// } - /// ``` - fn at_last_inst(mut self, ebb: Ebb) -> Self - where - Self: Sized, - { - self.goto_last_inst(ebb); - self - } - - /// Rebuild this cursor positioned after `inst`. - /// - /// This is intended to be used as a builder method: - /// - /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; - /// fn edit_func(func: &mut Function, inst: Inst) { - /// let mut pos = FuncCursor::new(func).after_inst(inst); - /// - /// // Use `pos`... - /// } - /// ``` - fn after_inst(mut self, inst: Inst) -> Self - where - Self: Sized, - { - self.goto_after_inst(inst); - self - } - - /// Rebuild this cursor positioned at the top of `ebb`. - /// - /// This is intended to be used as a builder method: - /// - /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; - /// fn edit_func(func: &mut Function, ebb: Ebb) { - /// let mut pos = FuncCursor::new(func).at_top(ebb); - /// - /// // Use `pos`... - /// } - /// ``` - fn at_top(mut self, ebb: Ebb) -> Self - where - Self: Sized, - { - self.goto_top(ebb); - self - } - - /// Rebuild this cursor positioned at the bottom of `ebb`. - /// - /// This is intended to be used as a builder method: - /// - /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; - /// fn edit_func(func: &mut Function, ebb: Ebb) { - /// let mut pos = FuncCursor::new(func).at_bottom(ebb); - /// - /// // Use `pos`... - /// } - /// ``` - fn at_bottom(mut self, ebb: Ebb) -> Self - where - Self: Sized, - { - self.goto_bottom(ebb); - self - } - - /// Get the EBB corresponding to the current position. - fn current_ebb(&self) -> Option { - use self::CursorPosition::*; - match self.position() { - Nowhere => None, - At(inst) => self.layout().inst_ebb(inst), - Before(ebb) | After(ebb) => Some(ebb), - } - } - - /// Get the instruction corresponding to the current position, if any. - fn current_inst(&self) -> Option { - use self::CursorPosition::*; - match self.position() { - At(inst) => Some(inst), - _ => None, - } - } - - /// Go to the position after a specific instruction, which must be inserted - /// in the layout. New instructions will be inserted after `inst`. - fn goto_after_inst(&mut self, inst: Inst) { - debug_assert!(self.layout().inst_ebb(inst).is_some()); - let new_pos = if let Some(next) = self.layout().insts[inst].next.expand() { - CursorPosition::At(next) - } else { - CursorPosition::After(self.layout().inst_ebb(inst).expect( - "current instruction removed?", - )) - }; - self.set_position(new_pos); - } - - /// Go to a specific instruction which must be inserted in the layout. - /// New instructions will be inserted before `inst`. - fn goto_inst(&mut self, inst: Inst) { - assert!(self.layout().inst_ebb(inst).is_some()); - self.set_position(CursorPosition::At(inst)); - } - - /// Go to the position for inserting instructions at the beginning of `ebb`, - /// which unlike `goto_first_inst` doesn't assume that any instructions have - /// been inserted into `ebb` yet. - fn goto_first_insertion_point(&mut self, ebb: Ebb) { - if let Some(inst) = self.layout().ebbs[ebb].first_inst.expand() { - self.goto_inst(inst); - } else { - self.goto_bottom(ebb); - } - } - - /// Go to the first instruction in `ebb`. - fn goto_first_inst(&mut self, ebb: Ebb) { - let inst = self.layout().ebbs[ebb].first_inst.expect("Empty EBB"); - self.goto_inst(inst); - } - - /// Go to the last instruction in `ebb`. - fn goto_last_inst(&mut self, ebb: Ebb) { - let inst = self.layout().ebbs[ebb].last_inst.expect("Empty EBB"); - self.goto_inst(inst); - } - - /// Go to the top of `ebb` which must be inserted into the layout. - /// At this position, instructions cannot be inserted, but `next_inst()` will move to the first - /// instruction in `ebb`. - fn goto_top(&mut self, ebb: Ebb) { - assert!(self.layout().is_ebb_inserted(ebb)); - self.set_position(CursorPosition::Before(ebb)); - } - - /// Go to the bottom of `ebb` which must be inserted into the layout. - /// At this position, inserted instructions will be appended to `ebb`. - fn goto_bottom(&mut self, ebb: Ebb) { - assert!(self.layout().is_ebb_inserted(ebb)); - self.set_position(CursorPosition::After(ebb)); - } - - /// Go to the top of the next EBB in layout order and return it. - /// - /// - If the cursor wasn't pointing at anything, go to the top of the first EBB in the - /// function. - /// - If there are no more EBBs, leave the cursor pointing at nothing and return `None`. - /// - /// # Examples - /// - /// The `next_ebb()` method is intended for iterating over the EBBs in layout order: - /// - /// ``` - /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; - /// fn edit_func(func: &mut Function) { - /// let mut cursor = FuncCursor::new(func); - /// while let Some(ebb) = cursor.next_ebb() { - /// // Edit ebb. - /// } - /// } - /// ``` - fn next_ebb(&mut self) -> Option { - let next = if let Some(ebb) = self.current_ebb() { - self.layout().ebbs[ebb].next.expand() - } else { - self.layout().first_ebb - }; - self.set_position(match next { - Some(ebb) => CursorPosition::Before(ebb), - None => CursorPosition::Nowhere, - }); - next - } - - /// Go to the bottom of the previous EBB in layout order and return it. - /// - /// - If the cursor wasn't pointing at anything, go to the bottom of the last EBB in the - /// function. - /// - If there are no more EBBs, leave the cursor pointing at nothing and return `None`. - /// - /// # Examples - /// - /// The `prev_ebb()` method is intended for iterating over the EBBs in backwards layout order: - /// - /// ``` - /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; - /// fn edit_func(func: &mut Function) { - /// let mut cursor = FuncCursor::new(func); - /// while let Some(ebb) = cursor.prev_ebb() { - /// // Edit ebb. - /// } - /// } - /// ``` - fn prev_ebb(&mut self) -> Option { - let prev = if let Some(ebb) = self.current_ebb() { - self.layout().ebbs[ebb].prev.expand() - } else { - self.layout().last_ebb - }; - self.set_position(match prev { - Some(ebb) => CursorPosition::After(ebb), - None => CursorPosition::Nowhere, - }); - prev - } - - /// Move to the next instruction in the same EBB and return it. - /// - /// - If the cursor was positioned before an EBB, go to the first instruction in that EBB. - /// - If there are no more instructions in the EBB, go to the `After(ebb)` position and return - /// `None`. - /// - If the cursor wasn't pointing anywhere, keep doing that. - /// - /// This method will never move the cursor to a different EBB. - /// - /// # Examples - /// - /// The `next_inst()` method is intended for iterating over the instructions in an EBB like - /// this: - /// - /// ``` - /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; - /// fn edit_ebb(func: &mut Function, ebb: Ebb) { - /// let mut cursor = FuncCursor::new(func).at_top(ebb); - /// while let Some(inst) = cursor.next_inst() { - /// // Edit instructions... - /// } - /// } - /// ``` - /// The loop body can insert and remove instructions via the cursor. - /// - /// Iterating over all the instructions in a function looks like this: - /// - /// ``` - /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; - /// fn edit_func(func: &mut Function) { - /// let mut cursor = FuncCursor::new(func); - /// while let Some(ebb) = cursor.next_ebb() { - /// while let Some(inst) = cursor.next_inst() { - /// // Edit instructions... - /// } - /// } - /// } - /// ``` - fn next_inst(&mut self) -> Option { - use self::CursorPosition::*; - match self.position() { - Nowhere | After(..) => None, - At(inst) => { - if let Some(next) = self.layout().insts[inst].next.expand() { - self.set_position(At(next)); - Some(next) - } else { - let pos = After(self.layout().inst_ebb(inst).expect( - "current instruction removed?", - )); - self.set_position(pos); - None - } - } - Before(ebb) => { - if let Some(next) = self.layout().ebbs[ebb].first_inst.expand() { - self.set_position(At(next)); - Some(next) - } else { - self.set_position(After(ebb)); - None - } - } - } - } - - /// Move to the previous instruction in the same EBB and return it. - /// - /// - If the cursor was positioned after an EBB, go to the last instruction in that EBB. - /// - If there are no more instructions in the EBB, go to the `Before(ebb)` position and return - /// `None`. - /// - If the cursor wasn't pointing anywhere, keep doing that. - /// - /// This method will never move the cursor to a different EBB. - /// - /// # Examples - /// - /// The `prev_inst()` method is intended for iterating backwards over the instructions in an - /// EBB like this: - /// - /// ``` - /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; - /// fn edit_ebb(func: &mut Function, ebb: Ebb) { - /// let mut cursor = FuncCursor::new(func).at_bottom(ebb); - /// while let Some(inst) = cursor.prev_inst() { - /// // Edit instructions... - /// } - /// } - /// ``` - fn prev_inst(&mut self) -> Option { - use self::CursorPosition::*; - match self.position() { - Nowhere | Before(..) => None, - At(inst) => { - if let Some(prev) = self.layout().insts[inst].prev.expand() { - self.set_position(At(prev)); - Some(prev) - } else { - let pos = Before(self.layout().inst_ebb(inst).expect( - "current instruction removed?", - )); - self.set_position(pos); - None - } - } - After(ebb) => { - if let Some(prev) = self.layout().ebbs[ebb].last_inst.expand() { - self.set_position(At(prev)); - Some(prev) - } else { - self.set_position(Before(ebb)); - None - } - } - } - } - - /// Insert an instruction at the current position. - /// - /// - If pointing at an instruction, the new instruction is inserted before the current - /// instruction. - /// - If pointing at the bottom of an EBB, the new instruction is appended to the EBB. - /// - Otherwise panic. - /// - /// In either case, the cursor is not moved, such that repeated calls to `insert_inst()` causes - /// instructions to appear in insertion order in the EBB. - fn insert_inst(&mut self, inst: Inst) { - use self::CursorPosition::*; - match self.position() { - Nowhere | Before(..) => panic!("Invalid insert_inst position"), - At(cur) => self.layout_mut().insert_inst(inst, cur), - After(ebb) => self.layout_mut().append_inst(inst, ebb), - } - } - - /// Remove the instruction under the cursor. - /// - /// The cursor is left pointing at the position following the current instruction. - /// - /// Return the instruction that was removed. - fn remove_inst(&mut self) -> Inst { - let inst = self.current_inst().expect("No instruction to remove"); - self.next_inst(); - self.layout_mut().remove_inst(inst); - inst - } - - /// Remove the instruction under the cursor. - /// - /// The cursor is left pointing at the position preceding the current instruction. - /// - /// Return the instruction that was removed. - fn remove_inst_and_step_back(&mut self) -> Inst { - let inst = self.current_inst().expect("No instruction to remove"); - self.prev_inst(); - self.layout_mut().remove_inst(inst); - inst - } - - /// Insert an EBB at the current position and switch to it. - /// - /// As far as possible, this method behaves as if the EBB header were an instruction inserted - /// at the current position. - /// - /// - If the cursor is pointing at an existing instruction, *the current EBB is split in two* - /// and the current instruction becomes the first instruction in the inserted EBB. - /// - If the cursor points at the bottom of an EBB, the new EBB is inserted after the current - /// one, and moved to the bottom of the new EBB where instructions can be appended. - /// - If the cursor points to the top of an EBB, the new EBB is inserted above the current one. - /// - If the cursor is not pointing at anything, the new EBB is placed last in the layout. - /// - /// This means that it is always valid to call this method, and it always leaves the cursor in - /// a state that will insert instructions into the new EBB. - fn insert_ebb(&mut self, new_ebb: Ebb) { - use self::CursorPosition::*; - match self.position() { - At(inst) => { - self.layout_mut().split_ebb(new_ebb, inst); - // All other cases move to `After(ebb)`, but in this case we'll stay `At(inst)`. - return; - } - Nowhere => self.layout_mut().append_ebb(new_ebb), - Before(ebb) => self.layout_mut().insert_ebb(new_ebb, ebb), - After(ebb) => self.layout_mut().insert_ebb_after(new_ebb, ebb), - } - // For everything but `At(inst)` we end up appending to the new EBB. - self.set_position(After(new_ebb)); - } -} - -impl<'f> CursorBase for Cursor<'f> { - fn position(&self) -> CursorPosition { - self.pos - } - - fn set_position(&mut self, pos: CursorPosition) { - self.pos = pos; - } - - fn srcloc(&self) -> SourceLoc { - self.srcloc - } - - fn set_srcloc(&mut self, srcloc: SourceLoc) { - self.srcloc = srcloc - } - - fn layout(&self) -> &Layout { - self.layout - } - - fn layout_mut(&mut self) -> &mut Layout { - self.layout - } -} - -impl<'f> Cursor<'f> { - /// Create a new `Cursor` for `layout`. - /// The cursor holds a mutable reference to `layout` for its entire lifetime. - pub fn new>>( - layout: &'f mut Layout, - srclocs: SL, - ) -> Cursor<'f> { - Cursor { - layout, - srclocs: srclocs.into(), - pos: CursorPosition::Nowhere, - srcloc: Default::default(), - } - } - - /// Use the source location of `inst` for future instructions. - pub fn use_srcloc(&mut self, inst: Inst) { - if let Some(ref mut ss) = self.srclocs { - self.srcloc = ss[inst]; - } - } -} - -/// An instruction inserter which can be used to build and insert instructions at a cursor -/// position. -/// -/// This is used by `dfg.ins()`. -pub struct LayoutCursorInserter<'c, 'fc: 'c, 'fd> { - pos: &'c mut Cursor<'fc>, - dfg: &'fd mut DataFlowGraph, -} - -impl<'c, 'fc: 'c, 'fd> LayoutCursorInserter<'c, 'fc, 'fd> { - /// Create a new inserter. Don't use this, use `dfg.ins(pos)`. - pub fn new( - pos: &'c mut Cursor<'fc>, - dfg: &'fd mut DataFlowGraph, - ) -> LayoutCursorInserter<'c, 'fc, 'fd> { - LayoutCursorInserter { pos, dfg } - } -} - -impl<'c, 'fc: 'c, 'fd> InstInserterBase<'fd> for LayoutCursorInserter<'c, 'fc, 'fd> { - fn data_flow_graph(&self) -> &DataFlowGraph { - self.dfg - } - - fn data_flow_graph_mut(&mut self) -> &mut DataFlowGraph { - self.dfg - } - - fn insert_built_inst(self, inst: Inst, _ctrl_typevar: Type) -> &'fd mut DataFlowGraph { - self.pos.insert_inst(inst); - if !self.pos.srcloc.is_default() { - if let Some(ref mut ss) = self.pos.srclocs { - ss[inst] = self.pos.srcloc; - } else { - panic!("layout::Cursor missing a SourceLocs reference"); - } - } - self.dfg - } -} - #[cfg(test)] mod tests { - use super::{Layout, Cursor, CursorBase, CursorPosition}; + use cursor::{Cursor, CursorPosition}; + use super::Layout; use entity::EntityRef; - use ir::{Ebb, Inst, ProgramOrder}; + use ir::{Ebb, Inst, ProgramOrder, SourceLoc}; use std::cmp::Ordering; + struct LayoutCursor<'f> { + /// Borrowed function layout. Public so it can be re-borrowed from this cursor. + pub layout: &'f mut Layout, + pos: CursorPosition, + } + + impl<'f> Cursor for LayoutCursor<'f> { + fn position(&self) -> CursorPosition { + self.pos + } + + fn set_position(&mut self, pos: CursorPosition) { + self.pos = pos; + } + + fn srcloc(&self) -> SourceLoc { + unimplemented!() + } + + fn set_srcloc(&mut self, _srcloc: SourceLoc) { + unimplemented!() + } + + fn layout(&self) -> &Layout { + self.layout + } + + fn layout_mut(&mut self) -> &mut Layout { + self.layout + } + } + + impl<'f> LayoutCursor<'f> { + /// Create a new `LayoutCursor` for `layout`. + /// The cursor holds a mutable reference to `layout` for its entire lifetime. + pub fn new(layout: &'f mut Layout) -> LayoutCursor<'f> { + LayoutCursor { + layout, + pos: CursorPosition::Nowhere, + } + } + } + fn verify(layout: &mut Layout, ebbs: &[(Ebb, &[Inst])]) { // Check that EBBs are inserted and instructions belong the right places. // Check forward linkage with iterators. @@ -1383,7 +771,7 @@ mod tests { } // Check backwards linkage with a cursor. - let mut cur = Cursor::new(layout, None); + let mut cur = LayoutCursor::new(layout); for &(ebb, insts) in ebbs.into_iter().rev() { assert_eq!(cur.prev_ebb(), Some(ebb)); for &inst in insts.into_iter().rev() { @@ -1439,7 +827,7 @@ mod tests { } // Test cursor positioning. - let mut cur = Cursor::new(&mut layout, None); + let mut cur = LayoutCursor::new(&mut layout); assert_eq!(cur.position(), CursorPosition::Nowhere); assert_eq!(cur.next_inst(), None); assert_eq!(cur.position(), CursorPosition::Nowhere); @@ -1557,7 +945,7 @@ mod tests { verify(&mut layout, &[(e1, &[i1, i2, i0])]); // Test cursor positioning. - let mut cur = Cursor::new(&mut layout, None).at_top(e1); + let mut cur = LayoutCursor::new(&mut layout).at_top(e1); assert_eq!(cur.position(), CursorPosition::Before(e1)); assert_eq!(cur.prev_inst(), None); assert_eq!(cur.position(), CursorPosition::Before(e1)); @@ -1677,7 +1065,7 @@ mod tests { assert_eq!(layout.inst_ebb(i0), Some(e1)); { - let mut cur = Cursor::new(&mut layout, None); + let mut cur = LayoutCursor::new(&mut layout); assert_eq!(cur.next_ebb(), Some(e0)); assert_eq!(cur.next_inst(), None); assert_eq!(cur.next_ebb(), Some(e1)); @@ -1705,7 +1093,7 @@ mod tests { assert_eq!(layout.inst_ebb(i3), Some(e2)); { - let mut cur = Cursor::new(&mut layout, None); + let mut cur = LayoutCursor::new(&mut layout); assert_eq!(cur.next_ebb(), Some(e0)); assert_eq!(cur.next_inst(), Some(i1)); assert_eq!(cur.next_inst(), None); diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 9fafd99b9a..2165937cee 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -32,7 +32,7 @@ pub use ir::globalvar::GlobalVarData; pub use ir::heap::{HeapData, HeapStyle, HeapBase}; pub use ir::instructions::{Opcode, InstructionData, VariableArgs, ValueList, ValueListPool}; pub use ir::jumptable::JumpTableData; -pub use ir::layout::{Layout, CursorBase, Cursor}; +pub use ir::layout::Layout; pub use ir::memflags::MemFlags; pub use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; pub use ir::sourceloc::SourceLoc; From 1a1774af15e0db949a803b5a81f476752b7e2b2f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Oct 2017 08:39:35 -0700 Subject: [PATCH 1271/3084] Fix a typo in a comment. --- lib/frontend/src/ssa.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 90090d6d0e..a359ab4b33 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -612,7 +612,7 @@ where } /// The main algorithm is naturally recursive: when there's a `use_var` in a - /// block with no correspondin local defs, it recurses and performs a + /// block with no corresponding local defs, it recurses and performs a /// `use_var` in each predecessor. To avoid risking running out of callstack /// space, we keep an explicit stack and use a small state machine rather /// than literal recursion. From 79a81d98c1ecf8b105a055b2d7930654500d2029 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Oct 2017 10:58:54 -0700 Subject: [PATCH 1272/3084] Mention `cton_frontend` when discussing SSA construction. --- cranelift/docs/langref.rst | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 98257a141d..0542a9445f 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -74,13 +74,15 @@ SSA values: In the entry block, ``v4`` is the initial value. In the loop block variable during each iteration. Finally, ``v12`` is computed as the induction variable value for the next iteration. -It can be difficult to generate correct SSA form if the program being converted -into Cretonne :term:`IL` contains multiple assignments to the same variables. -Such variables can be presented to Cretonne as :term:`stack slot`\s instead. +The `cton_frontend` crate contains utilities for translating from programs +containing multiple assignments to the same variables into SSA form for +Cretonne :term:`IL`. + +Such variables can also be presented to Cretonne as :term:`stack slot`\s. Stack slots are accessed with the :inst:`stack_store` and :inst:`stack_load` -instructions which behave more like variable accesses in a typical programming -language. Cretonne can perform the necessary data-flow analysis to convert stack -slots to SSA form. +instructions, and can have their address taken with :inst:`stack_addr`, which +supports C-like programming languages where local variables can have their +address taken. .. _value-types: From 3ccee371a7fbd4541ff5ebb04137337c88a5171a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Oct 2017 11:01:57 -0700 Subject: [PATCH 1273/3084] Remove the todo for smod. It's not present in either WebAssembly or Rust, for example. We can still add smod in the future if future use cases need it. --- lib/cretonne/meta/base/instructions.py | 5 ----- 1 file changed, 5 deletions(-) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 3c473d9e22..b27a3bf674 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -782,11 +782,6 @@ srem = Instruction( Signed integer remainder. The result has the sign of the dividend. This operation traps if the divisor is zero. - - .. todo:: Integer remainder vs modulus. - - Should we add a ``smod`` instruction for the case where - the result has the same sign as the divisor? """, ins=(x, y), outs=a, can_trap=True) From bce3c3804283ab0402552495710196f4c83de82a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Oct 2017 11:05:40 -0700 Subject: [PATCH 1274/3084] Clarify the todo about minimum / maximum. --- cranelift/docs/langref.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 0542a9445f..eea3e41c02 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -769,7 +769,7 @@ Integer operations .. autoinst:: srem .. autoinst:: srem_imm -.. todo:: Minimum / maximum. +.. todo:: Integer minimum / maximum. NEON has ``smin``, ``smax``, ``umin``, and ``umax`` instructions. We should replicate those for both scalar and vector integer types. Even if the From cc0bb70c5de82e811d6b5741af7a935904d0996a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Oct 2017 12:57:55 -0700 Subject: [PATCH 1275/3084] Make GVN aware of instructions that write to CPU flags. --- cranelift/filetests/simple_gvn/reject.cton | 16 ++++++++++++++++ lib/cretonne/meta/cdsl/instructions.py | 5 +++++ lib/cretonne/meta/cdsl/operands.py | 7 +++++++ lib/cretonne/src/simple_gvn.rs | 2 +- 4 files changed, 29 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/simple_gvn/reject.cton b/cranelift/filetests/simple_gvn/reject.cton index 1bb6ad16fa..5544ddbd75 100644 --- a/cranelift/filetests/simple_gvn/reject.cton +++ b/cranelift/filetests/simple_gvn/reject.cton @@ -23,3 +23,19 @@ ebb0: v5 = iadd v4, v3 return v5 } + +function %cpu_flags() -> b1 { +ebb0: + v0 = iconst.i32 7 + v1 = iconst.i32 8 + v2 = ifcmp v0, v1 + v3 = trueif eq v2 + v4 = ifcmp v0, v1 + v5 = trueif eq v4 + v6 = bor v3, v5 +; check: v2 = ifcmp v0, v1 +; check: v3 = trueif eq v2 +; check: v4 = ifcmp v0, v1 +; check: v5 = trueif eq v4 + return v6 +} diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index d295f59b41..8c66b400fb 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -110,6 +110,7 @@ class Instruction(object): 'can_trap': 'Can this instruction cause a trap?', 'other_side_effects': 'Does this instruction have other side effects besides can_*', + 'writes_cpu_flags': 'Does this instruction write to CPU flags?', } def __init__(self, name, doc, ins=(), outs=(), constraints=(), **kwargs): @@ -144,6 +145,10 @@ class Instruction(object): "unknown instruction attribute '" + attr + "'") for attr in Instruction.ATTRIBS: setattr(self, attr, not not kwargs.get(attr, False)) + + # Infer the 'writes_cpu_flags' field value. + setattr(self, 'writes_cpu_flags', any(out.is_cpu_flags() for out in self.outs)) + InstructionGroup.append(self) def __str__(self): diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/cretonne/meta/cdsl/operands.py index 85d4adef99..b6f81e4c26 100644 --- a/lib/cretonne/meta/cdsl/operands.py +++ b/lib/cretonne/meta/cdsl/operands.py @@ -242,3 +242,10 @@ class Operand(object): dependency. """ return self.kind is not VALUE and self.kind is not VARIABLE_ARGS + + def is_cpu_flags(self): + # type: () -> bool + """ + Is this a CPU flags operand? + """ + return self.kind is VALUE and self.typevar.name in ['iflags', 'fflags'] diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index b59f6bafd8..708a287904 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -10,7 +10,7 @@ use scoped_hash_map::ScopedHashMap; fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || opcode.is_return() || opcode.can_trap() || opcode.other_side_effects() || - opcode.can_store() || opcode.can_load() + opcode.can_store() || opcode.can_load() || opcode.writes_cpu_flags() } /// Perform simple GVN on `func`. From 7c9b9e3d2758eb4954b17219cf15791a6d357166 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Oct 2017 13:11:33 -0700 Subject: [PATCH 1276/3084] Mark spill and fill as can_store and can_load. This allows GVN to avoid hoisting them. These will be to coarse for things that want more precise dependence information, however we can work that out when we build such things. --- cranelift/filetests/simple_gvn/reject.cton | 15 +++++++++++++++ lib/cretonne/meta/base/instructions.py | 4 ++-- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/simple_gvn/reject.cton b/cranelift/filetests/simple_gvn/reject.cton index 5544ddbd75..00d88e5e66 100644 --- a/cranelift/filetests/simple_gvn/reject.cton +++ b/cranelift/filetests/simple_gvn/reject.cton @@ -39,3 +39,18 @@ ebb0: ; check: v5 = trueif eq v4 return v6 } + +function %spill() -> i32 { +ebb0: + v0 = iconst.i32 7 + v1 = spill v0 + v2 = fill v1 + v3 = spill v0 + v4 = fill v1 + v5 = bor v2, v4 +; check: v1 = spill v0 +; check: v2 = fill v1 +; check: v3 = spill v0 +; check: v4 = fill v1 + return v5 +} diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index b27a3bf674..f8b6ea989e 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -501,7 +501,7 @@ spill = Instruction( This instruction behaves exactly like :inst:`copy`, but the result value is assigned to a spill slot. """, - ins=x, outs=a) + ins=x, outs=a, can_store=True) fill = Instruction( 'fill', r""" @@ -510,7 +510,7 @@ fill = Instruction( This instruction behaves exactly like :inst:`copy`, but creates a new SSA value for the spilled input value. """, - ins=x, outs=a) + ins=x, outs=a, can_load=True) src = Operand('src', regunit) dst = Operand('dst', regunit) From ea68a69f8ba338a82c491a7883491742e549d62f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Oct 2017 16:16:27 -0700 Subject: [PATCH 1277/3084] Fix a flake8 lint. Also don't infer writes_cpu_flags if it is specified explicitly. --- lib/cretonne/meta/cdsl/instructions.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 8c66b400fb..2ca4951c58 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -147,7 +147,9 @@ class Instruction(object): setattr(self, attr, not not kwargs.get(attr, False)) # Infer the 'writes_cpu_flags' field value. - setattr(self, 'writes_cpu_flags', any(out.is_cpu_flags() for out in self.outs)) + if 'writes_cpu_flags' not in kwargs: + self.writes_cpu_flags = any( + out.is_cpu_flags() for out in self.outs) InstructionGroup.append(self) From 921bcc6c25aec6d72b93a78bd21318885626fea8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Oct 2017 14:15:23 -0700 Subject: [PATCH 1278/3084] Use the term "EBB parameter" everywhere. Add EBB parameter and EBB argument to the langref glossary to clarify the distinction between formal EBB parameter values and arguments passed to branches. - Replace "ebb_arg" with "ebb_param" in function names that deal with EBB parameters. - Rename the ValueDef variants to Result and Param. - A bunch of other small langref fixes. No functional changes intended. --- cranelift/docs/callex.cton | 13 + cranelift/docs/langref.rst | 88 +++--- cranelift/filetests/verifier/type_check.cton | 4 +- lib/cretonne/src/dominator_tree.rs | 4 +- lib/cretonne/src/flowgraph.rs | 2 +- lib/cretonne/src/ir/builder.rs | 8 +- lib/cretonne/src/ir/dfg.rs | 260 +++++++++--------- lib/cretonne/src/ir/function.rs | 2 +- lib/cretonne/src/ir/progpoint.rs | 8 +- lib/cretonne/src/isa/intel/enc_tables.rs | 8 +- lib/cretonne/src/isa/riscv/mod.rs | 10 +- lib/cretonne/src/legalizer/boundary.rs | 20 +- lib/cretonne/src/legalizer/globalvar.rs | 15 +- lib/cretonne/src/legalizer/mod.rs | 2 +- lib/cretonne/src/legalizer/split.rs | 26 +- lib/cretonne/src/licm.rs | 6 +- lib/cretonne/src/loop_analysis.rs | 4 +- lib/cretonne/src/regalloc/coalescing.rs | 18 +- lib/cretonne/src/regalloc/coloring.rs | 16 +- .../src/regalloc/live_value_tracker.rs | 12 +- lib/cretonne/src/regalloc/liveness.rs | 40 +-- lib/cretonne/src/regalloc/reload.rs | 9 +- lib/cretonne/src/verifier/cssa.rs | 10 +- lib/cretonne/src/verifier/liveness.rs | 4 +- lib/cretonne/src/verifier/mod.rs | 31 ++- lib/cretonne/src/write.rs | 10 +- lib/frontend/src/frontend.rs | 40 +-- lib/frontend/src/ssa.rs | 32 +-- lib/reader/src/parser.rs | 46 ++-- lib/wasm/src/code_translator.rs | 10 +- 30 files changed, 392 insertions(+), 366 deletions(-) create mode 100644 cranelift/docs/callex.cton diff --git a/cranelift/docs/callex.cton b/cranelift/docs/callex.cton new file mode 100644 index 0000000000..6690e78f90 --- /dev/null +++ b/cranelift/docs/callex.cton @@ -0,0 +1,13 @@ +test verifier + +function %gcd(i32 uext, i32 uext) -> i32 uext native { + fn1 = function %divmod(i32 uext, i32 uext) -> i32 uext, i32 uext + +ebb1(v1: i32, v2: i32): + brz v2, ebb2 + v3, v4 = call fn1(v1, v2) + return v3 + +ebb2: + return v1 +} diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index eea3e41c02..aa0913e745 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -34,7 +34,7 @@ Here is the same function compiled into Cretonne IL: :lines: 2- The first line of a function definition provides the function *name* and -the :term:`function signature` which declares the argument and return types. +the :term:`function signature` which declares the parameter and return types. Then follows the :term:`function preamble` which declares a number of entities that can be referenced inside the function. In the example above, the preamble declares a single local variable, ``ss1``. @@ -60,17 +60,18 @@ The instructions in the function body use and produce *values* in SSA form. This means that every value is defined exactly once, and every use of a value must be dominated by the definition. -Cretonne does not have phi instructions but uses *EBB arguments* instead. An EBB -can be defined with a list of typed arguments. Whenever control is transferred -to the EBB, values for the arguments must be provided. When entering a function, -the incoming function arguments are passed as arguments to the entry EBB. +Cretonne does not have phi instructions but uses :term:`EBB parameter`\s +instead. An EBB can be defined with a list of typed parameters. Whenever control +is transferred to the EBB, argument values for the parameters must be provided. +When entering a function, the incoming function parameters are passed as +arguments to the entry EBB's parameters. Instructions define zero, one, or more result values. All SSA values are either -EBB arguments or instruction results. +EBB parameters or instruction results. In the example above, the loop induction variable ``i`` is represented as three SSA values: In the entry block, ``v4`` is the initial value. In the loop block -``ebb2``, the EBB argument ``v5`` represents the value of the induction +``ebb2``, the EBB parameter ``v5`` represents the value of the induction variable during each iteration. Finally, ``v12`` is computed as the induction variable value for the next iteration. @@ -138,6 +139,7 @@ NaNs being indicated by the MSB of the trailing significand set to 0. Except for bitwise and memory instructions, NaNs returned from arithmetic instructions are encoded as follows: + - If all NaN inputs to an instruction are quiet NaNs with all bits of the trailing significand other than the MSB set to 0, the result is a quiet NaN with a nondeterministic sign bit and all bits of the trailing @@ -160,6 +162,11 @@ Since some ISAs don't have CPU flags, these value types should not be used until the legalization phase of compilation where the code is adapted to fit the target ISA. Use instructions like :inst:`icmp` instead. +The CPU flags types are also restricted such that two flags values can not be +live at the same time. After legalization, some instruction encodings will +clobber the flags, and flags values are not allowed to be live across such +instructions either. The verifier enforces these rules. + .. autoctontype:: iflags .. autoctontype:: fflags @@ -210,12 +217,12 @@ called a *lane*. The number of lanes must be a power of two in the range 2-256. Pseudo-types and type classes ----------------------------- -These are not concrete types, but convenient names uses to refer to real types +These are not concrete types, but convenient names used to refer to real types in this reference. .. type:: iAddr - A Pointer-sized integer. + A Pointer-sized integer representing an address. This is either :type:`i32`, or :type:`i64`, depending on whether the target platform has 32-bit or 64-bit pointers. @@ -379,21 +386,21 @@ Function calls ============== A function call needs a target function and a :term:`function signature`. The -target function may be determined dynamically at runtime, but the signature -must be known when the function call is compiled. The function signature -describes how to call the function, including arguments, return values, and the -calling convention: +target function may be determined dynamically at runtime, but the signature must +be known when the function call is compiled. The function signature describes +how to call the function, including parameters, return values, and the calling +convention: .. productionlist:: - signature : "(" [arglist] ")" ["->" retlist] [call_conv] - arglist : arg { "," arg } - retlist : arglist - arg : type [argext] [argspecial] - argext : "uext" | "sext" - argspecial: "sret" | "link" | "fp" | "csr" | "vmctx" - callconv : `string` + signature : "(" [paramlist] ")" ["->" retlist] [call_conv] + paramlist : param { "," param } + retlist : paramlist + param : type [paramext] [paramspecial] + paramext : "uext" | "sext" + paramspecial : "sret" | "link" | "fp" | "csr" | "vmctx" + callconv : "native" | "spiderwasm" -Arguments and return values have flags whose meaning is mostly target +Parameters and return values have flags whose meaning is mostly target dependent. They make it possible to call native functions on the target platform. When calling other Cretonne functions, the flags are not necessary. @@ -411,19 +418,11 @@ preamble`: .. autoinst:: call .. autoinst:: x_return -This simple example illustrates direct function calls and signatures:: +This simple example illustrates direct function calls and signatures: - function %gcd(i32 uext, i32 uext) -> i32 uext "C" { - fn1 = function %divmod(i32 uext, i32 uext) -> i32 uext, i32 uext - - ebb1(v1: i32, v2: i32): - brz v2, ebb2 - v3, v4 = call fn1(v1, v2) - br ebb1(v2, v4) - - ebb2: - return v1 - } +.. literalinclude:: callex.cton + :language: cton + :lines: 3- Indirect function calls use a signature declared in the preamble. @@ -682,15 +681,15 @@ bounds checking is required for each access: Operations ========== -A few instructions have variants that take immediate operands (e.g., -:inst:`band` / :inst:`band_imm`), but in general an instruction is required to -load a constant into an SSA value. - .. autoinst:: select Constant materialization ------------------------ +A few instructions have variants that take immediate operands (e.g., +:inst:`band` / :inst:`band_imm`), but in general an instruction is required to +load a constant into an SSA value. + .. autoinst:: iconst .. autoinst:: f32const .. autoinst:: f64const @@ -944,6 +943,9 @@ Instructions that can only be used by the Intel target ISA. .. autoinst:: isa.intel.instructions.sdivmodx .. autoinst:: isa.intel.instructions.udivmodx +.. autoinst:: isa.intel.instructions.cvtt2si +.. autoinst:: isa.intel.instructions.fmin +.. autoinst:: isa.intel.instructions.fmax Instruction groups ================== @@ -1034,6 +1036,18 @@ Glossary control flow graph where only the root can be a join node. This definition is not equivalent to Cretonne EBBs. + EBB parameter + A formal parameter for an EBB is an SSA value that dominates everything + in the EBB. For each parameter declared by an EBB, a corresponding + argument value must be passed when branching to the EBB. The function's + entry EBB has parameters that correspond to the function's parameters. + + EBB argument + Similar to function arguments, EBB arguments must be provided when + branching to an EBB that declares formal parameters. When execution + begins at the top of an EBB, the formal parameters have the values of + the arguments passed in the branch. + function signature A function signature describes how to call a function. It consists of: diff --git a/cranelift/filetests/verifier/type_check.cton b/cranelift/filetests/verifier/type_check.cton index d8250389c4..4bd386efbf 100644 --- a/cranelift/filetests/verifier/type_check.cton +++ b/cranelift/filetests/verifier/type_check.cton @@ -1,12 +1,12 @@ test verifier function %entry_block_signature_mismatch(i32) { - ebb0: ; error: entry block arguments must match function signature + ebb0: ; error: entry block parameters (0) must match function signature (1) return } function %entry_block_arg_type(i32) { - ebb0(v0: f32): ; error: entry block argument 0 expected to have type i32, got f32 + ebb0(v0: f32): ; error: entry block parameter 0 expected to have type i32, got f32 return } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index ba57069ab6..300c9453df 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -450,7 +450,7 @@ mod test { fn unreachable_node() { let mut func = Function::new(); let ebb0 = func.dfg.make_ebb(); - let v0 = func.dfg.append_ebb_arg(ebb0, types::I32); + let v0 = func.dfg.append_ebb_param(ebb0, types::I32); let ebb1 = func.dfg.make_ebb(); let ebb2 = func.dfg.make_ebb(); @@ -479,7 +479,7 @@ mod test { fn non_zero_entry_block() { let mut func = Function::new(); let ebb3 = func.dfg.make_ebb(); - let cond = func.dfg.append_ebb_arg(ebb3, types::I32); + let cond = func.dfg.append_ebb_param(ebb3, types::I32); let ebb1 = func.dfg.make_ebb(); let ebb2 = func.dfg.make_ebb(); let ebb0 = func.dfg.make_ebb(); diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index 4dfc72fe32..ce9744ac38 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -185,7 +185,7 @@ mod tests { fn branches_and_jumps() { let mut func = Function::new(); let ebb0 = func.dfg.make_ebb(); - let cond = func.dfg.append_ebb_arg(ebb0, types::I32); + let cond = func.dfg.append_ebb_param(ebb0, types::I32); let ebb1 = func.dfg.make_ebb(); let ebb2 = func.dfg.make_ebb(); diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 4a662d9d14..a69fd4e181 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -223,7 +223,7 @@ mod tests { fn types() { let mut func = Function::new(); let ebb0 = func.dfg.make_ebb(); - let arg0 = func.dfg.append_ebb_arg(ebb0, I32); + let arg0 = func.dfg.append_ebb_param(ebb0, I32); let mut pos = FuncCursor::new(&mut func); pos.insert_ebb(ebb0); @@ -244,14 +244,14 @@ mod tests { fn reuse_results() { let mut func = Function::new(); let ebb0 = func.dfg.make_ebb(); - let arg0 = func.dfg.append_ebb_arg(ebb0, I32); + let arg0 = func.dfg.append_ebb_param(ebb0, I32); let mut pos = FuncCursor::new(&mut func); pos.insert_ebb(ebb0); let v0 = pos.ins().iadd_imm(arg0, 17); assert_eq!(pos.func.dfg.value_type(v0), I32); let iadd = pos.prev_inst().unwrap(); - assert_eq!(pos.func.dfg.value_def(v0), ValueDef::Res(iadd, 0)); + assert_eq!(pos.func.dfg.value_def(v0), ValueDef::Result(iadd, 0)); // Detach v0 and reuse it for a different instruction. pos.func.dfg.clear_results(iadd); @@ -260,6 +260,6 @@ mod tests { assert_eq!(pos.current_inst(), Some(iadd)); let iconst = pos.prev_inst().unwrap(); assert!(iadd != iconst); - assert_eq!(pos.func.dfg.value_def(v0), ValueDef::Res(iconst, 0)); + assert_eq!(pos.func.dfg.value_def(v0), ValueDef::Result(iconst, 0)); } } diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index d663130280..6593b12f92 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -16,7 +16,7 @@ use std::u16; /// A data flow graph defines all instructions and extended basic blocks in a function as well as /// the data flow dependencies between them. The DFG also tracks values which can be either -/// instruction results or EBB arguments. +/// instruction results or EBB parameters. /// /// The layout of EBBs in the function and of instructions in each EBB is recorded by the /// `FunctionLayout` data structure which form the other half of the function representation. @@ -34,7 +34,8 @@ pub struct DataFlowGraph { /// primary `insts` map. results: EntityMap, - /// Extended basic blocks in the function and their arguments. + /// Extended basic blocks in the function and their parameters. + /// /// This map is not in program order. That is handled by `Layout`, and so is the sequence of /// instructions contained in each EBB. ebbs: PrimaryMap, @@ -45,7 +46,7 @@ pub struct DataFlowGraph { /// /// - Instructions in `insts` that don't have room for their entire argument list inline. /// - Instruction result values in `results`. - /// - EBB arguments in `ebbs`. + /// - EBB parameters in `ebbs`. pub value_lists: ValueListPool, /// Primary value table with entries for all values. @@ -135,7 +136,7 @@ fn resolve_aliases(values: &PrimaryMap, value: Value) -> Value /// Handling values. /// -/// Values are either EBB arguments or instruction results. +/// Values are either EBB parameters or instruction results. impl DataFlowGraph { /// Allocate an extended value entry. fn make_value(&mut self, data: ValueData) -> Value { @@ -151,7 +152,7 @@ impl DataFlowGraph { pub fn value_type(&self, v: Value) -> Type { match self.values[v] { ValueData::Inst { ty, .. } | - ValueData::Arg { ty, .. } | + ValueData::Param { ty, .. } | ValueData::Alias { ty, .. } => ty, } } @@ -159,7 +160,7 @@ impl DataFlowGraph { /// Get the definition of a value. /// /// This is either the instruction that defined it or the Ebb that has the value as an - /// argument. + /// parameter. pub fn value_def(&self, v: Value) -> ValueDef { match self.values[v] { ValueData::Inst { inst, num, .. } => { @@ -170,15 +171,15 @@ impl DataFlowGraph { v, self.display_inst(inst, None) ); - ValueDef::Res(inst, num as usize) + ValueDef::Result(inst, num as usize) } - ValueData::Arg { ebb, num, .. } => { + ValueData::Param { ebb, num, .. } => { assert_eq!( Some(v), - self.ebbs[ebb].args.get(num as usize, &self.value_lists), - "Dangling EBB argument value" + self.ebbs[ebb].params.get(num as usize, &self.value_lists), + "Dangling EBB parameter value" ); - ValueDef::Arg(ebb, num as usize) + ValueDef::Param(ebb, num as usize) } ValueData::Alias { original, .. } => { // Make sure we only recurse one level. `resolve_aliases` has safeguards to @@ -188,7 +189,7 @@ impl DataFlowGraph { } } - /// Determine if `v` is an attached instruction result / EBB argument. + /// Determine if `v` is an attached instruction result / EBB parameter. /// /// An attached value can't be attached to something else without first being detached. /// @@ -198,7 +199,7 @@ impl DataFlowGraph { use self::ValueData::*; match self.values[v] { Inst { inst, num, .. } => Some(&v) == self.inst_results(inst).get(num as usize), - Arg { ebb, num, .. } => Some(&v) == self.ebb_args(ebb).get(num as usize), + Param { ebb, num, .. } => Some(&v) == self.ebb_params(ebb).get(num as usize), Alias { .. } => false, } } @@ -317,16 +318,16 @@ impl DataFlowGraph { #[derive(Debug, PartialEq, Eq)] pub enum ValueDef { /// Value is the n'th result of an instruction. - Res(Inst, usize), - /// Value is the n'th argument to an EBB. - Arg(Ebb, usize), + Result(Inst, usize), + /// Value is the n'th parameter to an EBB. + Param(Ebb, usize), } impl ValueDef { /// Unwrap the instruction where the value was defined, or panic. pub fn unwrap_inst(&self) -> Inst { match *self { - ValueDef::Res(inst, _) => inst, + ValueDef::Result(inst, _) => inst, _ => panic!("Value is not an instruction result"), } } @@ -338,11 +339,11 @@ enum ValueData { // Value is defined by an instruction. Inst { ty: Type, num: u16, inst: Inst }, - // Value is an EBB argument. - Arg { ty: Type, num: u16, ebb: Ebb }, + // Value is an EBB parameter. + Param { ty: Type, num: u16, ebb: Ebb }, // Value is an alias of another value. - // An alias value can't be linked as an instruction result or EBB argument. It is used as a + // An alias value can't be linked as an instruction result or EBB parameter. It is used as a // placeholder when the original instruction or EBB has been rewritten or modified. Alias { ty: Type, original: Value }, } @@ -678,83 +679,82 @@ impl DataFlowGraph { self.ebbs.push(EbbData::new()) } - /// Get the number of arguments on `ebb`. - pub fn num_ebb_args(&self, ebb: Ebb) -> usize { - self.ebbs[ebb].args.len(&self.value_lists) + /// Get the number of parameters on `ebb`. + pub fn num_ebb_params(&self, ebb: Ebb) -> usize { + self.ebbs[ebb].params.len(&self.value_lists) } - /// Get the arguments to an EBB. - pub fn ebb_args(&self, ebb: Ebb) -> &[Value] { - self.ebbs[ebb].args.as_slice(&self.value_lists) + /// Get the parameters on `ebb`. + pub fn ebb_params(&self, ebb: Ebb) -> &[Value] { + self.ebbs[ebb].params.as_slice(&self.value_lists) } - /// Append an argument with type `ty` to `ebb`. - pub fn append_ebb_arg(&mut self, ebb: Ebb, ty: Type) -> Value { - let arg = self.values.next_key(); - let num = self.ebbs[ebb].args.push(arg, &mut self.value_lists); - assert!(num <= u16::MAX as usize, "Too many arguments to EBB"); - self.make_value(ValueData::Arg { + /// Append a parameter with type `ty` to `ebb`. + pub fn append_ebb_param(&mut self, ebb: Ebb, ty: Type) -> Value { + let param = self.values.next_key(); + let num = self.ebbs[ebb].params.push(param, &mut self.value_lists); + assert!(num <= u16::MAX as usize, "Too many parameters on EBB"); + self.make_value(ValueData::Param { ty, num: num as u16, ebb, }) } - /// Removes `val` from `ebb`'s argument by swapping it with the last argument of `ebb`. + /// Removes `val` from `ebb`'s parameters by swapping it with the last parameter on `ebb`. /// Returns the position of `val` before removal. /// - /// *Important*: to ensure O(1) deletion, this method swaps the removed argument with the - /// last `Ebb` argument. This can disrupt all the branch instructions jumping to this - /// `Ebb` for which you have to change the jump argument order if necessary. + /// *Important*: to ensure O(1) deletion, this method swaps the removed parameter with the + /// last `ebb`` parameter. This can disrupt all the branch instructions jumping to this + /// `ebb` for which you have to change the branch argument order if necessary. /// - /// Panics if `val` is not an `Ebb` argument. Returns `true` if `Ebb` arguments have been - /// swapped. - pub fn swap_remove_ebb_arg(&mut self, val: Value) -> usize { - let (ebb, num) = if let ValueData::Arg { num, ebb, .. } = self.values[val] { + /// Panics if `val` is not an EBB parameter. + pub fn swap_remove_ebb_param(&mut self, val: Value) -> usize { + let (ebb, num) = if let ValueData::Param { num, ebb, .. } = self.values[val] { (ebb, num) } else { - panic!("{} must be an EBB argument", val); + panic!("{} must be an EBB parameter", val); }; - self.ebbs[ebb].args.swap_remove( + self.ebbs[ebb].params.swap_remove( num as usize, &mut self.value_lists, ); - if let Some(last_arg_val) = self.ebbs[ebb].args.get(num as usize, &self.value_lists) { + if let Some(last_arg_val) = self.ebbs[ebb].params.get(num as usize, &self.value_lists) { // We update the position of the old last arg. - if let ValueData::Arg { num: ref mut old_num, .. } = self.values[last_arg_val] { + if let ValueData::Param { num: ref mut old_num, .. } = self.values[last_arg_val] { *old_num = num; } else { - panic!("{} should be an Ebb argument but is not", last_arg_val); + panic!("{} should be an Ebb parameter", last_arg_val); } } num as usize } - /// Removes `val` from `ebb`'s arguments by a standard linear time list removal which preserves - /// ordering. Also updates the values' data. - pub fn remove_ebb_arg(&mut self, val: Value) { - let (ebb, num) = if let ValueData::Arg { num, ebb, .. } = self.values[val] { + /// Removes `val` from `ebb`'s parameters by a standard linear time list removal which + /// preserves ordering. Also updates the values' data. + pub fn remove_ebb_param(&mut self, val: Value) { + let (ebb, num) = if let ValueData::Param { num, ebb, .. } = self.values[val] { (ebb, num) } else { - panic!("{} must be an EBB argument", val); + panic!("{} must be an EBB parameter", val); }; - self.ebbs[ebb].args.remove( + self.ebbs[ebb].params.remove( num as usize, &mut self.value_lists, ); - for index in num..(self.ebb_args(ebb).len() as u16) { + for index in num..(self.num_ebb_params(ebb) as u16) { match self.values[self.ebbs[ebb] - .args + .params .get(index as usize, &self.value_lists) .unwrap()] { - ValueData::Arg { ref mut num, .. } => { + ValueData::Param { ref mut num, .. } => { *num -= 1; } _ => { panic!( - "{} must be an EBB argument", + "{} must be an EBB parameter", self.ebbs[ebb] - .args + .params .get(index as usize, &self.value_lists) .unwrap() ) @@ -764,73 +764,73 @@ impl DataFlowGraph { } - /// Append an existing argument value to `ebb`. + /// Append an existing value to `ebb`'s parameters. /// /// The appended value can't already be attached to something else. /// - /// In almost all cases, you should be using `append_ebb_arg()` instead of this method. - pub fn attach_ebb_arg(&mut self, ebb: Ebb, arg: Value) { - assert!(!self.value_is_attached(arg)); - let num = self.ebbs[ebb].args.push(arg, &mut self.value_lists); - assert!(num <= u16::MAX as usize, "Too many arguments to EBB"); - let ty = self.value_type(arg); - self.values[arg] = ValueData::Arg { + /// In almost all cases, you should be using `append_ebb_param()` instead of this method. + pub fn attach_ebb_param(&mut self, ebb: Ebb, param: Value) { + assert!(!self.value_is_attached(param)); + let num = self.ebbs[ebb].params.push(param, &mut self.value_lists); + assert!(num <= u16::MAX as usize, "Too many parameters on EBB"); + let ty = self.value_type(param); + self.values[param] = ValueData::Param { ty, num: num as u16, ebb, }; } - /// Replace an EBB argument with a new value of type `ty`. + /// Replace an EBB parameter with a new value of type `ty`. /// - /// The `old_value` must be an attached EBB argument. It is removed from its place in the list - /// of arguments and replaced by a new value of type `new_type`. The new value gets the same - /// position in the list, and other arguments are not disturbed. + /// The `old_value` must be an attached EBB parameter. It is removed from its place in the list + /// of parameters and replaced by a new value of type `new_type`. The new value gets the same + /// position in the list, and other parameters are not disturbed. /// /// The old value is left detached, so it should probably be changed into something else. /// /// Returns the new value. - pub fn replace_ebb_arg(&mut self, old_arg: Value, new_type: Type) -> Value { + pub fn replace_ebb_param(&mut self, old_value: Value, new_type: Type) -> Value { // Create new value identical to the old one except for the type. - let (ebb, num) = if let ValueData::Arg { num, ebb, .. } = self.values[old_arg] { + let (ebb, num) = if let ValueData::Param { num, ebb, .. } = self.values[old_value] { (ebb, num) } else { - panic!("{} must be an EBB argument", old_arg); + panic!("{} must be an EBB parameter", old_value); }; - let new_arg = self.make_value(ValueData::Arg { + let new_arg = self.make_value(ValueData::Param { ty: new_type, num, ebb, }); - self.ebbs[ebb].args.as_mut_slice(&mut self.value_lists)[num as usize] = new_arg; + self.ebbs[ebb].params.as_mut_slice(&mut self.value_lists)[num as usize] = new_arg; new_arg } - /// Detach all the arguments from `ebb` and return them as a `ValueList`. + /// Detach all the parameters from `ebb` and return them as a `ValueList`. /// - /// This is a quite low-level operation. Sensible things to do with the detached EBB arguments - /// is to put them back on the same EBB with `attach_ebb_arg()` or change them into aliases + /// This is a quite low-level operation. Sensible things to do with the detached EBB parameters + /// is to put them back on the same EBB with `attach_ebb_param()` or change them into aliases /// with `change_to_alias()`. - pub fn detach_ebb_args(&mut self, ebb: Ebb) -> ValueList { - self.ebbs[ebb].args.take() + pub fn detach_ebb_params(&mut self, ebb: Ebb) -> ValueList { + self.ebbs[ebb].params.take() } } // Contents of an extended basic block. // -// Arguments for an extended basic block are values that dominate everything in the EBB. All +// Parameters on an extended basic block are values that dominate everything in the EBB. All // branches to this EBB must provide matching arguments, and the arguments to the entry EBB must // match the function arguments. #[derive(Clone)] struct EbbData { - // List of arguments to this EBB. - args: ValueList, + // List of parameters to this EBB. + params: ValueList, } impl EbbData { fn new() -> EbbData { - EbbData { args: ValueList::new() } + EbbData { params: ValueList::new() } } } @@ -899,7 +899,7 @@ mod tests { let val = dfg.first_result(inst); assert_eq!(dfg.inst_results(inst), &[val]); - assert_eq!(dfg.value_def(val), ValueDef::Res(inst, 0)); + assert_eq!(dfg.value_def(val), ValueDef::Result(inst, 0)); assert_eq!(dfg.value_type(val), types::I32); // Replacing results. @@ -908,7 +908,7 @@ mod tests { assert!(!dfg.value_is_attached(val)); assert!(dfg.value_is_attached(v2)); assert_eq!(dfg.inst_results(inst), &[v2]); - assert_eq!(dfg.value_def(v2), ValueDef::Res(inst, 0)); + assert_eq!(dfg.value_def(v2), ValueDef::Result(inst, 0)); assert_eq!(dfg.value_type(v2), types::F64); } @@ -933,90 +933,90 @@ mod tests { let ebb = dfg.make_ebb(); assert_eq!(ebb.to_string(), "ebb0"); - assert_eq!(dfg.num_ebb_args(ebb), 0); - assert_eq!(dfg.ebb_args(ebb), &[]); - assert!(dfg.detach_ebb_args(ebb).is_empty()); - assert_eq!(dfg.num_ebb_args(ebb), 0); - assert_eq!(dfg.ebb_args(ebb), &[]); + assert_eq!(dfg.num_ebb_params(ebb), 0); + assert_eq!(dfg.ebb_params(ebb), &[]); + assert!(dfg.detach_ebb_params(ebb).is_empty()); + assert_eq!(dfg.num_ebb_params(ebb), 0); + assert_eq!(dfg.ebb_params(ebb), &[]); - let arg1 = dfg.append_ebb_arg(ebb, types::F32); + let arg1 = dfg.append_ebb_param(ebb, types::F32); assert_eq!(arg1.to_string(), "v0"); - assert_eq!(dfg.num_ebb_args(ebb), 1); - assert_eq!(dfg.ebb_args(ebb), &[arg1]); + assert_eq!(dfg.num_ebb_params(ebb), 1); + assert_eq!(dfg.ebb_params(ebb), &[arg1]); - let arg2 = dfg.append_ebb_arg(ebb, types::I16); + let arg2 = dfg.append_ebb_param(ebb, types::I16); assert_eq!(arg2.to_string(), "v1"); - assert_eq!(dfg.num_ebb_args(ebb), 2); - assert_eq!(dfg.ebb_args(ebb), &[arg1, arg2]); + assert_eq!(dfg.num_ebb_params(ebb), 2); + assert_eq!(dfg.ebb_params(ebb), &[arg1, arg2]); - assert_eq!(dfg.value_def(arg1), ValueDef::Arg(ebb, 0)); - assert_eq!(dfg.value_def(arg2), ValueDef::Arg(ebb, 1)); + assert_eq!(dfg.value_def(arg1), ValueDef::Param(ebb, 0)); + assert_eq!(dfg.value_def(arg2), ValueDef::Param(ebb, 1)); assert_eq!(dfg.value_type(arg1), types::F32); assert_eq!(dfg.value_type(arg2), types::I16); - // Swap the two EBB arguments. - let vlist = dfg.detach_ebb_args(ebb); - assert_eq!(dfg.num_ebb_args(ebb), 0); - assert_eq!(dfg.ebb_args(ebb), &[]); + // Swap the two EBB parameters. + let vlist = dfg.detach_ebb_params(ebb); + assert_eq!(dfg.num_ebb_params(ebb), 0); + assert_eq!(dfg.ebb_params(ebb), &[]); assert_eq!(vlist.as_slice(&dfg.value_lists), &[arg1, arg2]); - dfg.attach_ebb_arg(ebb, arg2); - let arg3 = dfg.append_ebb_arg(ebb, types::I32); - dfg.attach_ebb_arg(ebb, arg1); - assert_eq!(dfg.ebb_args(ebb), &[arg2, arg3, arg1]); + dfg.attach_ebb_param(ebb, arg2); + let arg3 = dfg.append_ebb_param(ebb, types::I32); + dfg.attach_ebb_param(ebb, arg1); + assert_eq!(dfg.ebb_params(ebb), &[arg2, arg3, arg1]); } #[test] - fn replace_ebb_arguments() { + fn replace_ebb_params() { let mut dfg = DataFlowGraph::new(); let ebb = dfg.make_ebb(); - let arg1 = dfg.append_ebb_arg(ebb, types::F32); + let arg1 = dfg.append_ebb_param(ebb, types::F32); - let new1 = dfg.replace_ebb_arg(arg1, types::I64); + let new1 = dfg.replace_ebb_param(arg1, types::I64); assert_eq!(dfg.value_type(arg1), types::F32); assert_eq!(dfg.value_type(new1), types::I64); - assert_eq!(dfg.ebb_args(ebb), &[new1]); + assert_eq!(dfg.ebb_params(ebb), &[new1]); - dfg.attach_ebb_arg(ebb, arg1); - assert_eq!(dfg.ebb_args(ebb), &[new1, arg1]); + dfg.attach_ebb_param(ebb, arg1); + assert_eq!(dfg.ebb_params(ebb), &[new1, arg1]); - let new2 = dfg.replace_ebb_arg(arg1, types::I8); + let new2 = dfg.replace_ebb_param(arg1, types::I8); assert_eq!(dfg.value_type(arg1), types::F32); assert_eq!(dfg.value_type(new2), types::I8); - assert_eq!(dfg.ebb_args(ebb), &[new1, new2]); + assert_eq!(dfg.ebb_params(ebb), &[new1, new2]); - dfg.attach_ebb_arg(ebb, arg1); - assert_eq!(dfg.ebb_args(ebb), &[new1, new2, arg1]); + dfg.attach_ebb_param(ebb, arg1); + assert_eq!(dfg.ebb_params(ebb), &[new1, new2, arg1]); - let new3 = dfg.replace_ebb_arg(new2, types::I16); + let new3 = dfg.replace_ebb_param(new2, types::I16); assert_eq!(dfg.value_type(new1), types::I64); assert_eq!(dfg.value_type(new2), types::I8); assert_eq!(dfg.value_type(new3), types::I16); - assert_eq!(dfg.ebb_args(ebb), &[new1, new3, arg1]); + assert_eq!(dfg.ebb_params(ebb), &[new1, new3, arg1]); } #[test] - fn swap_remove_ebb_arguments() { + fn swap_remove_ebb_params() { let mut dfg = DataFlowGraph::new(); let ebb = dfg.make_ebb(); - let arg1 = dfg.append_ebb_arg(ebb, types::F32); - let arg2 = dfg.append_ebb_arg(ebb, types::F32); - let arg3 = dfg.append_ebb_arg(ebb, types::F32); - assert_eq!(dfg.ebb_args(ebb), &[arg1, arg2, arg3]); + let arg1 = dfg.append_ebb_param(ebb, types::F32); + let arg2 = dfg.append_ebb_param(ebb, types::F32); + let arg3 = dfg.append_ebb_param(ebb, types::F32); + assert_eq!(dfg.ebb_params(ebb), &[arg1, arg2, arg3]); - dfg.swap_remove_ebb_arg(arg1); + dfg.swap_remove_ebb_param(arg1); assert_eq!(dfg.value_is_attached(arg1), false); assert_eq!(dfg.value_is_attached(arg2), true); assert_eq!(dfg.value_is_attached(arg3), true); - assert_eq!(dfg.ebb_args(ebb), &[arg3, arg2]); - dfg.swap_remove_ebb_arg(arg2); + assert_eq!(dfg.ebb_params(ebb), &[arg3, arg2]); + dfg.swap_remove_ebb_param(arg2); assert_eq!(dfg.value_is_attached(arg2), false); assert_eq!(dfg.value_is_attached(arg3), true); - assert_eq!(dfg.ebb_args(ebb), &[arg3]); - dfg.swap_remove_ebb_arg(arg3); + assert_eq!(dfg.ebb_params(ebb), &[arg3]); + dfg.swap_remove_ebb_param(arg3); assert_eq!(dfg.value_is_attached(arg3), false); - assert_eq!(dfg.ebb_args(ebb), &[]); + assert_eq!(dfg.ebb_params(ebb), &[]); } #[test] @@ -1035,10 +1035,10 @@ mod tests { // Make sure we can resolve value aliases even when values is empty. assert_eq!(pos.func.dfg.resolve_aliases(v1), v1); - let arg0 = pos.func.dfg.append_ebb_arg(ebb0, types::I32); + let arg0 = pos.func.dfg.append_ebb_param(ebb0, types::I32); let (s, c) = pos.ins().iadd_cout(v1, arg0); let iadd = match pos.func.dfg.value_def(s) { - ValueDef::Res(i, 0) => i, + ValueDef::Result(i, 0) => i, _ => panic!(), }; diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index f8134b34d9..bc1cc980bc 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -150,7 +150,7 @@ impl Function { pub fn special_arg(&self, purpose: ir::ArgumentPurpose) -> Option { let entry = self.layout.entry_block().expect("Function is empty"); self.signature.special_arg_index(purpose).map(|i| { - self.dfg.ebb_args(entry)[i] + self.dfg.ebb_params(entry)[i] }) } } diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index d61d6755f6..ef3735c180 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -35,8 +35,8 @@ impl From for ProgramPoint { impl From for ProgramPoint { fn from(def: ValueDef) -> ProgramPoint { match def { - ValueDef::Res(inst, _) => inst.into(), - ValueDef::Arg(ebb, _) => ebb.into(), + ValueDef::Result(inst, _) => inst.into(), + ValueDef::Param(ebb, _) => ebb.into(), } } } @@ -66,8 +66,8 @@ impl From for ExpandedProgramPoint { impl From for ExpandedProgramPoint { fn from(def: ValueDef) -> ExpandedProgramPoint { match def { - ValueDef::Res(inst, _) => inst.into(), - ValueDef::Arg(ebb, _) => ebb.into(), + ValueDef::Result(inst, _) => inst.into(), + ValueDef::Param(ebb, _) => ebb.into(), } } } diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 23c73014d5..c58a362bb0 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -37,7 +37,7 @@ fn expand_srem(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGra let result = func.dfg.first_result(inst); let ty = func.dfg.value_type(result); func.dfg.clear_results(inst); - func.dfg.attach_ebb_arg(done, result); + func.dfg.attach_ebb_param(done, result); let mut pos = FuncCursor::new(func).at_inst(inst); pos.use_srcloc(inst); @@ -112,7 +112,7 @@ fn expand_minmax(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowG let result = func.dfg.first_result(inst); let ty = func.dfg.value_type(result); func.dfg.clear_results(inst); - func.dfg.attach_ebb_arg(done, result); + func.dfg.attach_ebb_param(done, result); // Test for case 1) ordered and not equal. let mut pos = FuncCursor::new(func).at_inst(inst); @@ -194,7 +194,7 @@ fn expand_fcvt_from_uint(inst: ir::Inst, func: &mut ir::Function, cfg: &mut Cont // Move the `inst` result value onto the `done` EBB. pos.func.dfg.clear_results(inst); - pos.func.dfg.attach_ebb_arg(done, result); + pos.func.dfg.attach_ebb_param(done, result); // If x as a signed int is not negative, we can use the existing `fcvt_from_sint` instruction. let is_neg = pos.ins().icmp_imm(IntCC::SignedLessThan, x, 0); @@ -328,7 +328,7 @@ fn expand_fcvt_to_uint(inst: ir::Inst, func: &mut ir::Function, cfg: &mut Contro // Move the `inst` result value onto the `done` EBB. func.dfg.clear_results(inst); - func.dfg.attach_ebb_arg(done, result); + func.dfg.attach_ebb_param(done, result); let mut pos = FuncCursor::new(func).at_inst(inst); pos.use_srcloc(inst); diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index e6841b7299..ab9eadc68b 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -136,8 +136,8 @@ mod tests { let mut dfg = DataFlowGraph::new(); let ebb = dfg.make_ebb(); - let arg64 = dfg.append_ebb_arg(ebb, types::I64); - let arg32 = dfg.append_ebb_arg(ebb, types::I32); + let arg64 = dfg.append_ebb_param(ebb, types::I64); + let arg32 = dfg.append_ebb_param(ebb, types::I32); // Try to encode iadd_imm.i64 v1, -10. let inst64 = InstructionData::BinaryImm { @@ -180,8 +180,8 @@ mod tests { let mut dfg = DataFlowGraph::new(); let ebb = dfg.make_ebb(); - let arg64 = dfg.append_ebb_arg(ebb, types::I64); - let arg32 = dfg.append_ebb_arg(ebb, types::I32); + let arg64 = dfg.append_ebb_param(ebb, types::I64); + let arg32 = dfg.append_ebb_param(ebb, types::I32); // Try to encode iadd_imm.i64 v1, -10. let inst64 = InstructionData::BinaryImm { @@ -237,7 +237,7 @@ mod tests { let mut dfg = DataFlowGraph::new(); let ebb = dfg.make_ebb(); - let arg32 = dfg.append_ebb_arg(ebb, types::I32); + let arg32 = dfg.append_ebb_param(ebb, types::I32); // Create an imul.i32 which is encodable in RV32M. let mul32 = InstructionData::Binary { diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index d1d48b053b..6276e0fb04 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -67,11 +67,11 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { // Keep track of the argument types in the ABI-legalized signature. let mut abi_arg = 0; - // Process the EBB arguments one at a time, possibly replacing one argument with multiple new - // ones. We do this by detaching the entry EBB arguments first. - let ebb_args = pos.func.dfg.detach_ebb_args(entry); + // Process the EBB parameters one at a time, possibly replacing one argument with multiple new + // ones. We do this by detaching the entry EBB parameters first. + let ebb_params = pos.func.dfg.detach_ebb_params(entry); let mut old_arg = 0; - while let Some(arg) = ebb_args.get(old_arg, &pos.func.dfg.value_lists) { + while let Some(arg) = ebb_params.get(old_arg, &pos.func.dfg.value_lists) { old_arg += 1; let abi_type = pos.func.signature.argument_types[abi_arg]; @@ -79,7 +79,7 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { if arg_type == abi_type.value_type { // No value translation is necessary, this argument matches the ABI type. // Just use the original EBB argument value. This is the most common case. - pos.func.dfg.attach_ebb_arg(entry, arg); + pos.func.dfg.attach_ebb_param(entry, arg); match abi_type.purpose { ArgumentPurpose::Normal => {} ArgumentPurpose::StructReturn => { @@ -108,7 +108,7 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { ); if ty == abi_type.value_type { abi_arg += 1; - Ok(func.dfg.append_ebb_arg(entry, ty)) + Ok(func.dfg.append_ebb_param(entry, ty)) } else { Err(abi_type) } @@ -155,7 +155,7 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { // Just create entry block values to match here. We will use them in `handle_return_abi()` // below. - pos.func.dfg.append_ebb_arg(entry, arg.value_type); + pos.func.dfg.append_ebb_param(entry, arg.value_type); } } @@ -584,7 +584,7 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph .expect("No matching special purpose argument."); // Get the corresponding entry block value and add it to the return instruction's // arguments. - let val = pos.func.dfg.ebb_args( + let val = pos.func.dfg.ebb_params( pos.func.layout.entry_block().unwrap(), ) [idx]; @@ -611,7 +611,9 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph /// stack slot already during legalization. fn spill_entry_arguments(func: &mut Function, entry: Ebb) { for (abi, &arg) in func.signature.argument_types.iter().zip( - func.dfg.ebb_args(entry), + func.dfg.ebb_params( + entry, + ), ) { if let ArgumentLoc::Stack(offset) = abi.location { diff --git a/lib/cretonne/src/legalizer/globalvar.rs b/lib/cretonne/src/legalizer/globalvar.rs index b4b28d3032..7c459fb8c4 100644 --- a/lib/cretonne/src/legalizer/globalvar.rs +++ b/lib/cretonne/src/legalizer/globalvar.rs @@ -26,19 +26,10 @@ pub fn expand_global_addr(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut Co /// Expand a `global_addr` instruction for a vmctx global. fn vmctx_addr(inst: ir::Inst, func: &mut ir::Function, offset: i64) { - // Find the incoming `vmctx` function argument. Start searching from the back since the special - // arguments are appended by signature legalization. - // - // This argument must exist; `vmctx` global variables can not be used in functions with calling - // conventions that don't add a `vmctx` argument. - let argidx = func.signature - .argument_types - .iter() - .rposition(|abi| abi.purpose == ir::ArgumentPurpose::VMContext) - .expect("Need vmctx argument for vmctx global"); - // Get the value representing the `vmctx` argument. - let vmctx = func.dfg.ebb_args(func.layout.entry_block().unwrap())[argidx]; + let vmctx = func.special_arg(ir::ArgumentPurpose::VMContext).expect( + "Missing vmctx parameter", + ); // Simply replace the `global_addr` instruction with an `iadd_imm`, reusing the result value. func.dfg.replace(inst).iadd_imm(vmctx, offset); diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 75217eb239..cf8cc5bc99 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -201,7 +201,7 @@ fn expand_select(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowG let result = func.dfg.first_result(inst); func.dfg.clear_results(inst); let new_ebb = func.dfg.make_ebb(); - func.dfg.attach_ebb_arg(new_ebb, result); + func.dfg.attach_ebb_param(new_ebb, result); func.dfg.replace(inst).brnz(ctrl, new_ebb, &[tval]); let mut pos = FuncCursor::new(func).after_inst(inst); diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index 96e1b03145..eff3b9f542 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -194,7 +194,7 @@ fn split_value( let mut reuse = None; match pos.func.dfg.value_def(value) { - ValueDef::Res(inst, num) => { + ValueDef::Result(inst, num) => { // This is an instruction result. See if the value was created by a `concat` // instruction. if let InstructionData::Binary { opcode, args, .. } = pos.func.dfg[inst] { @@ -204,11 +204,11 @@ fn split_value( } } } - ValueDef::Arg(ebb, num) => { - // This is an EBB argument. We can split the argument value unless this is the entry + ValueDef::Param(ebb, num) => { + // This is an EBB parameter. We can split the parameter value unless this is the entry // block. if pos.func.layout.entry_block() != Some(ebb) { - // We are going to replace the argument at `num` with two new arguments. + // We are going to replace the parameter at `num` with two new arguments. // Determine the new value types. let ty = pos.func.dfg.value_type(value); let split_type = match concat { @@ -217,20 +217,20 @@ fn split_value( _ => panic!("Unhandled concat opcode: {}", concat), }; - // Since the `repairs` stack potentially contains other argument numbers for `ebb`, - // avoid shifting and renumbering EBB arguments. It could invalidate other + // Since the `repairs` stack potentially contains other parameter numbers for + // `ebb`, avoid shifting and renumbering EBB parameters. It could invalidate other // `repairs` entries. // // Replace the original `value` with the low part, and append the high part at the // end of the argument list. - let lo = pos.func.dfg.replace_ebb_arg(value, split_type); - let hi_num = pos.func.dfg.num_ebb_args(ebb); - let hi = pos.func.dfg.append_ebb_arg(ebb, split_type); + let lo = pos.func.dfg.replace_ebb_param(value, split_type); + let hi_num = pos.func.dfg.num_ebb_params(ebb); + let hi = pos.func.dfg.append_ebb_param(ebb, split_type); reuse = Some((lo, hi)); // Now the original value is dangling. Insert a concatenation instruction that can - // compute it from the two new arguments. This also serves as a record of what we + // compute it from the two new parameters. This also serves as a record of what we // did so a future call to this function doesn't have to redo the work. // // Note that it is safe to move `pos` here since `reuse` was set above, so we don't @@ -243,7 +243,7 @@ fn split_value( hi, ); - // Finally, splitting the EBB argument is not enough. We also have to repair all + // Finally, splitting the EBB parameter is not enough. We also have to repair all // of the predecessor instructions that branch here. add_repair(concat, split_type, ebb, num, hi_num, repairs); } @@ -299,7 +299,7 @@ fn resolve_splits(dfg: &ir::DataFlowGraph, value: Value) -> Value { let split_res; let concat_opc; let split_arg; - if let ValueDef::Res(inst, num) = dfg.value_def(value) { + if let ValueDef::Result(inst, num) = dfg.value_def(value) { split_res = num; concat_opc = match dfg[inst].opcode() { Opcode::Isplit => Opcode::Iconcat, @@ -312,7 +312,7 @@ fn resolve_splits(dfg: &ir::DataFlowGraph, value: Value) -> Value { } // See if split_arg is defined by a concatenation instruction. - if let ValueDef::Res(inst, _) = dfg.value_def(split_arg) { + if let ValueDef::Result(inst, _) = dfg.value_def(split_arg) { if dfg[inst].opcode() == concat_opc { return dfg.inst_args(inst)[split_res]; } diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 490e0d8335..2fcd8241c2 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -63,7 +63,7 @@ fn create_pre_header( domtree: &DominatorTree, ) -> Ebb { let pool = &mut ListPool::::new(); - let header_args_values: Vec = func.dfg.ebb_args(header).into_iter().cloned().collect(); + let header_args_values: Vec = func.dfg.ebb_params(header).into_iter().cloned().collect(); let header_args_types: Vec = header_args_values .clone() .into_iter() @@ -72,7 +72,7 @@ fn create_pre_header( let pre_header = func.dfg.make_ebb(); let mut pre_header_args_value: EntityList = EntityList::new(); for typ in header_args_types { - pre_header_args_value.push(func.dfg.append_ebb_arg(pre_header, typ), pool); + pre_header_args_value.push(func.dfg.append_ebb_param(pre_header, typ), pool); } for &(_, last_inst) in cfg.get_predecessors(header) { // We only follow normal edges (not the back edges) @@ -143,7 +143,7 @@ fn remove_loop_invariant_instructions( // We traverse the loop EBB in reverse post-order. for ebb in postorder_ebbs_loop(loop_analysis, cfg, lp).iter().rev() { // Arguments of the EBB are loop values - for val in pos.func.dfg.ebb_args(*ebb) { + for val in pos.func.dfg.ebb_params(*ebb) { loop_values.insert(*val); } pos.goto_top(*ebb); diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index 21c612356e..95d3bace4f 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -236,7 +236,7 @@ mod test { let ebb1 = func.dfg.make_ebb(); let ebb2 = func.dfg.make_ebb(); let ebb3 = func.dfg.make_ebb(); - let cond = func.dfg.append_ebb_arg(ebb0, types::I32); + let cond = func.dfg.append_ebb_param(ebb0, types::I32); { let mut cur = FuncCursor::new(&mut func); @@ -288,7 +288,7 @@ mod test { let ebb3 = func.dfg.make_ebb(); let ebb4 = func.dfg.make_ebb(); let ebb5 = func.dfg.make_ebb(); - let cond = func.dfg.append_ebb_arg(ebb0, types::I32); + let cond = func.dfg.append_ebb_param(ebb0, types::I32); { let mut cur = FuncCursor::new(&mut func); diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 34a9b8bde0..b2897fe331 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -2,8 +2,8 @@ //! //! Conventional SSA form is a subset of SSA form where any (transitively) phi-related values do //! not interfere. We construct CSSA by building virtual registers that are as large as possible -//! and inserting copies where necessary such that all values passed to an EBB argument will belong -//! to the same virtual register as the EBB argument value itself. +//! and inserting copies where necessary such that all argument values passed to an EBB parameter +//! will belong to the same virtual register as the EBB parameter value itself. use cursor::{Cursor, EncCursor}; use dbg::DisplayList; @@ -289,8 +289,8 @@ impl Coalescing { for &ebb in domtree.cfg_postorder() { let preds = cfg.get_predecessors(ebb); if !preds.is_empty() { - for argnum in 0..context.func.dfg.num_ebb_args(ebb) { - context.coalesce_ebb_arg(ebb, argnum, preds) + for argnum in 0..context.func.dfg.num_ebb_params(ebb) { + context.coalesce_ebb_param(ebb, argnum, preds) } } } @@ -298,10 +298,10 @@ impl Coalescing { } impl<'a> Context<'a> { - /// Coalesce the `argnum`'th argument to `ebb`. - fn coalesce_ebb_arg(&mut self, ebb: Ebb, argnum: usize, preds: &[BasicBlock]) { + /// Coalesce the `argnum`'th parameter on `ebb`. + fn coalesce_ebb_param(&mut self, ebb: Ebb, argnum: usize, preds: &[BasicBlock]) { self.split_values.clear(); - let mut succ_val = self.func.dfg.ebb_args(ebb)[argnum]; + let mut succ_val = self.func.dfg.ebb_params(ebb)[argnum]; dbg!("Processing {}/{}: {}", ebb, argnum, succ_val); // We want to merge the virtual register for `succ_val` with the virtual registers for @@ -421,7 +421,7 @@ impl<'a> Context<'a> { // Never coalesce incoming function arguments on the stack. These arguments are // pre-spilled, and the rest of the virtual register would be forced to spill to the // `incoming_arg` stack slot too. - if let ValueDef::Arg(def_ebb, def_num) = self.func.dfg.value_def(pred_val) { + if let ValueDef::Param(def_ebb, def_num) = self.func.dfg.value_def(pred_val) { if Some(def_ebb) == self.func.layout.entry_block() && self.func.signature.argument_types[def_num] .location @@ -530,7 +530,7 @@ impl<'a> Context<'a> { /// Split the congruence class for the successor EBB value itself. fn split_succ(&mut self, ebb: Ebb, succ_val: Value) -> Value { let ty = self.func.dfg.value_type(succ_val); - let new_val = self.func.dfg.replace_ebb_arg(succ_val, ty); + let new_val = self.func.dfg.replace_ebb_param(succ_val, ty); // Insert a copy instruction at the top of ebb. let mut pos = EncCursor::new(self.func, self.isa).at_first_inst(ebb); diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index a0375bb6c4..89704b0865 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -409,7 +409,7 @@ impl<'a> Context<'a> { // If this is the first time we branch to `dest`, color its arguments to match the current // register state. if let Some(dest) = color_dest_args { - self.color_ebb_arguments(inst, dest); + self.color_ebb_params(inst, dest); } // Apply the solution to the defs. @@ -556,7 +556,7 @@ impl<'a> Context<'a> { // Now handle the EBB arguments. let br_args = self.cur.func.dfg.inst_variable_args(inst); - let dest_args = self.cur.func.dfg.ebb_args(dest); + let dest_args = self.cur.func.dfg.ebb_params(dest); assert_eq!(br_args.len(), dest_args.len()); for (&dest_arg, &br_arg) in dest_args.iter().zip(br_args) { // The first time we encounter a branch to `dest`, we get to pick the location. The @@ -565,7 +565,7 @@ impl<'a> Context<'a> { ValueLoc::Unassigned => { // This is the first branch to `dest`, so we should color `dest_arg` instead of // `br_arg`. However, we don't know where `br_arg` will end up until - // after `shuffle_inputs`. See `color_ebb_arguments` below. + // after `shuffle_inputs`. See `color_ebb_params` below. // // It is possible for `dest_arg` to have no affinity, and then it should simply // be ignored. @@ -595,13 +595,13 @@ impl<'a> Context<'a> { false } - /// Knowing that we've never seen a branch to `dest` before, color its arguments to match our + /// Knowing that we've never seen a branch to `dest` before, color its parameters to match our /// register state. /// /// This function is only called when `program_ebb_arguments()` returned `true`. - fn color_ebb_arguments(&mut self, inst: Inst, dest: Ebb) { + fn color_ebb_params(&mut self, inst: Inst, dest: Ebb) { let br_args = self.cur.func.dfg.inst_variable_args(inst); - let dest_args = self.cur.func.dfg.ebb_args(dest); + let dest_args = self.cur.func.dfg.ebb_params(dest); assert_eq!(br_args.len(), dest_args.len()); for (&dest_arg, &br_arg) in dest_args.iter().zip(br_args) { match self.cur.func.locations[dest_arg] { @@ -914,7 +914,7 @@ impl<'a> Context<'a> { } } - /// Replace all global values define by `inst` with local values that are then copied into the + /// Replace all global values defined by `inst` with local values that are then copied into the /// global value: /// /// v1 = foo @@ -938,7 +938,7 @@ impl<'a> Context<'a> { for lv in tracker.live_mut().iter_mut().rev() { // Keep going until we reach a value that is not defined by `inst`. if match self.cur.func.dfg.value_def(lv.value) { - ValueDef::Res(i, _) => i != inst, + ValueDef::Result(i, _) => i != inst, _ => true, } { diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 3007bcec7b..10fe9ff595 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -202,24 +202,22 @@ impl LiveValueTracker { } } - // Now add all the live arguments to `ebb`. + // Now add all the live parameters to `ebb`. let first_arg = self.live.values.len(); - for &value in dfg.ebb_args(ebb) { - let lr = liveness.get(value).expect( - "EBB argument value has no live range", - ); + for &value in dfg.ebb_params(ebb) { + let lr = &liveness[value]; assert_eq!(lr.def(), ebb.into()); match lr.def_local_end().into() { ExpandedProgramPoint::Inst(endpoint) => { self.live.push(value, endpoint, lr); } ExpandedProgramPoint::Ebb(local_ebb) => { - // This is a dead EBB argument which is not even live into the first + // This is a dead EBB parameter which is not even live into the first // instruction in the EBB. assert_eq!( local_ebb, ebb, - "EBB argument live range ends at wrong EBB header" + "EBB parameter live range ends at wrong EBB header" ); // Give this value a fake endpoint that is the first instruction in the EBB. // We expect it to be removed by calling `drop_dead_args()`. diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 5dd6ce71bb..a64aa9e017 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -205,7 +205,7 @@ fn get_or_create<'a>( let def; let affinity; match func.dfg.value_def(value) { - ValueDef::Res(inst, rnum) => { + ValueDef::Result(inst, rnum) => { def = inst.into(); // Initialize the affinity from the defining instruction's result constraints. // Don't do this for call return values which are always tied to a single register. @@ -221,14 +221,14 @@ fn get_or_create<'a>( }) .unwrap_or_default(); } - ValueDef::Arg(ebb, num) => { + ValueDef::Param(ebb, num) => { def = ebb.into(); if func.layout.entry_block() == Some(ebb) { - // The affinity for entry block arguments can be inferred from the function + // The affinity for entry block parameters can be inferred from the function // signature. affinity = Affinity::abi(&func.signature.argument_types[num], isa); } else { - // Don't apply any affinity to normal EBB arguments. + // Don't apply any affinity to normal EBB parameters. // They could be in a register or on the stack. affinity = Default::default(); } @@ -290,8 +290,8 @@ pub struct Liveness { /// It lives here to avoid repeated allocation of scratch memory. worklist: Vec, - /// Working space for the `propagate_ebb_arguments` algorithm. - ebb_args: Vec, + /// Working space for the `propagate_ebb_params` algorithm. + ebb_params: Vec, } impl Liveness { @@ -303,7 +303,7 @@ impl Liveness { Liveness { ranges: LiveRangeSet::new(), worklist: Vec::new(), - ebb_args: Vec::new(), + ebb_params: Vec::new(), } } @@ -378,10 +378,10 @@ impl Liveness { // elimination pass if we visit a post-order of the dominator tree? // TODO: Resolve value aliases while we're visiting instructions? for ebb in func.layout.ebbs() { - // Make sure we have created live ranges for dead EBB arguments. - // TODO: If these arguments are really dead, we could remove them, except for the entry - // block which must match the function signature. - for &arg in func.dfg.ebb_args(ebb) { + // Make sure we have created live ranges for dead EBB parameters. + // TODO: If these parameters are really dead, we could remove them, except for the + // entry block which must match the function signature. + for &arg in func.dfg.ebb_params(ebb) { get_or_create(&mut self.ranges, arg, isa, func, &enc_info); } @@ -431,28 +431,28 @@ impl Liveness { } } - self.propagate_ebb_arguments(func, cfg); + self.propagate_ebb_params(func, cfg); } - /// Propagate affinities for EBB arguments. + /// Propagate affinities for EBB parameters. /// /// If an EBB argument value has an affinity, all predecessors must pass a value with an /// affinity. - pub fn propagate_ebb_arguments(&mut self, func: &Function, cfg: &ControlFlowGraph) { - assert!(self.ebb_args.is_empty()); + pub fn propagate_ebb_params(&mut self, func: &Function, cfg: &ControlFlowGraph) { + assert!(self.ebb_params.is_empty()); for ebb in func.layout.ebbs() { - for &arg in func.dfg.ebb_args(ebb) { + for &arg in func.dfg.ebb_params(ebb) { let affinity = self.ranges.get(arg).unwrap().affinity; if affinity.is_none() { continue; } - self.ebb_args.push(arg); + self.ebb_params.push(arg); // Now apply the affinity to all predecessors recursively. - while let Some(succ_arg) = self.ebb_args.pop() { + while let Some(succ_arg) = self.ebb_params.pop() { let (succ_ebb, num) = match func.dfg.value_def(succ_arg) { - ValueDef::Arg(e, n) => (e, n), + ValueDef::Param(e, n) => (e, n), _ => continue, }; @@ -461,7 +461,7 @@ impl Liveness { let pred_affinity = &mut self.ranges.get_mut(pred_arg).unwrap().affinity; if pred_affinity.is_none() { *pred_affinity = affinity; - self.ebb_args.push(pred_arg); + self.ebb_params.push(pred_arg); } } } diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index aed4f040ba..de5683a444 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -139,7 +139,7 @@ impl<'a> Context<'a> { assert_eq!(liveins.len(), 0); self.visit_entry_args(ebb, args); } else { - self.visit_ebb_args(ebb, args); + self.visit_ebb_params(ebb, args); } } @@ -156,7 +156,10 @@ impl<'a> Context<'a> { if arg.affinity.is_stack() { // An incoming register parameter was spilled. Replace the parameter value // with a temporary register value that is immediately spilled. - let reg = self.cur.func.dfg.replace_ebb_arg(arg.value, abi.value_type); + let reg = self.cur.func.dfg.replace_ebb_param( + arg.value, + abi.value_type, + ); let affinity = Affinity::abi(&abi, self.cur.isa); self.liveness.create_dead(reg, ebb, affinity); self.insert_spill(ebb, arg.value, reg); @@ -170,7 +173,7 @@ impl<'a> Context<'a> { } } - fn visit_ebb_args(&mut self, ebb: Ebb, _args: &[LiveValue]) { + fn visit_ebb_params(&mut self, ebb: Ebb, _args: &[LiveValue]) { self.cur.goto_first_inst(ebb); } diff --git a/lib/cretonne/src/verifier/cssa.rs b/lib/cretonne/src/verifier/cssa.rs index e87c3bb39e..a311ba4876 100644 --- a/lib/cretonne/src/verifier/cssa.rs +++ b/lib/cretonne/src/verifier/cssa.rs @@ -101,22 +101,22 @@ impl<'a> CssaVerifier<'a> { fn check_cssa(&self) -> Result { for ebb in self.func.layout.ebbs() { - let ebb_args = self.func.dfg.ebb_args(ebb); + let ebb_params = self.func.dfg.ebb_params(ebb); for &(_, pred) in self.cfg.get_predecessors(ebb) { let pred_args = self.func.dfg.inst_variable_args(pred); // This should have been caught by an earlier verifier pass. assert_eq!( - ebb_args.len(), + ebb_params.len(), pred_args.len(), "Wrong arguments on branch." ); - for (&ebb_arg, &pred_arg) in ebb_args.iter().zip(pred_args) { - if !self.virtregs.same_class(ebb_arg, pred_arg) { + for (&ebb_param, &pred_arg) in ebb_params.iter().zip(pred_args) { + if !self.virtregs.same_class(ebb_param, pred_arg) { return err!( pred, "{} and {} must be in the same virtual register", - ebb_arg, + ebb_param, pred_arg ); } diff --git a/lib/cretonne/src/verifier/liveness.rs b/lib/cretonne/src/verifier/liveness.rs index fa7b23ec13..118347e2fb 100644 --- a/lib/cretonne/src/verifier/liveness.rs +++ b/lib/cretonne/src/verifier/liveness.rs @@ -49,7 +49,7 @@ impl<'a> LivenessVerifier<'a> { /// Check all EBB arguments. fn check_ebbs(&self) -> Result { for ebb in self.func.layout.ebbs() { - for &val in self.func.dfg.ebb_args(ebb) { + for &val in self.func.dfg.ebb_params(ebb) { let lr = match self.liveness.get(val) { Some(lr) => lr, None => return err!(ebb, "EBB arg {} has no live range", val), @@ -164,7 +164,7 @@ impl<'a> LivenessVerifier<'a> { // branch argument. self.func .dfg - .ebb_args(dest) + .ebb_params(dest) .get(argnum - fixed_args) .and_then(|&v| self.liveness.get(v)) .map(|lr| lr.affinity.is_none()) diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 0b9b2db142..9bfa531a9e 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -7,7 +7,7 @@ //! the EBB as reported by `inst_ebb()`. //! - Every EBB must end in a terminator instruction, and no other instruction //! can be a terminator. -//! - Every value in the `ebb_args` iterator belongs to the EBB as reported by `value_ebb`. +//! - Every value in the `ebb_params` iterator belongs to the EBB as reported by `value_ebb`. //! //! Instruction integrity //! @@ -212,10 +212,10 @@ impl<'a> Verifier<'a> { return err!(inst, "should belong to {} not {:?}", ebb, inst_ebb); } - // Arguments belong to the correct ebb. - for &arg in self.func.dfg.ebb_args(ebb) { + // Parameters belong to the correct ebb. + for &arg in self.func.dfg.ebb_params(ebb) { match self.func.dfg.value_def(arg) { - ValueDef::Arg(arg_ebb, _) => { + ValueDef::Param(arg_ebb, _) => { if ebb != arg_ebb { return err!(arg, "does not belong to {}", ebb); } @@ -441,7 +441,7 @@ impl<'a> Verifier<'a> { // SSA form match dfg.value_def(v) { - ValueDef::Res(def_inst, _) => { + ValueDef::Result(def_inst, _) => { // Value is defined by an instruction that exists. if !dfg.inst_is_valid(def_inst) { return err!( @@ -471,7 +471,7 @@ impl<'a> Verifier<'a> { return err!(loc_inst, "uses value from non-dominating {}", def_inst); } } - ValueDef::Arg(ebb, _) => { + ValueDef::Param(ebb, _) => { // Value is defined by an existing EBB. if !dfg.ebb_is_valid(ebb) { return err!(loc_inst, "{} is defined by invalid EBB {}", v, ebb); @@ -554,18 +554,23 @@ impl<'a> Verifier<'a> { fn typecheck_entry_block_arguments(&self) -> Result { if let Some(ebb) = self.func.layout.entry_block() { let expected_types = &self.func.signature.argument_types; - let ebb_arg_count = self.func.dfg.num_ebb_args(ebb); + let ebb_param_count = self.func.dfg.num_ebb_params(ebb); - if ebb_arg_count != expected_types.len() { - return err!(ebb, "entry block arguments must match function signature"); + if ebb_param_count != expected_types.len() { + return err!( + ebb, + "entry block parameters ({}) must match function signature ({})", + ebb_param_count, + expected_types.len() + ); } - for (i, &arg) in self.func.dfg.ebb_args(ebb).iter().enumerate() { + for (i, &arg) in self.func.dfg.ebb_params(ebb).iter().enumerate() { let arg_type = self.func.dfg.value_type(arg); if arg_type != expected_types[i].value_type { return err!( ebb, - "entry block argument {} expected to have type {}, got {}", + "entry block parameter {} expected to have type {}, got {}", i, expected_types[i], arg_type @@ -671,14 +676,14 @@ impl<'a> Verifier<'a> { fn typecheck_variable_args(&self, inst: Inst) -> Result { match self.func.dfg[inst].analyze_branch(&self.func.dfg.value_lists) { BranchInfo::SingleDest(ebb, _) => { - let iter = self.func.dfg.ebb_args(ebb).iter().map(|&v| { + let iter = self.func.dfg.ebb_params(ebb).iter().map(|&v| { self.func.dfg.value_type(v) }); self.typecheck_variable_args_iterator(inst, iter)?; } BranchInfo::Table(table) => { for (_, ebb) in self.func.jump_tables[table].entries() { - let arg_count = self.func.dfg.num_ebb_args(ebb); + let arg_count = self.func.dfg.num_ebb_params(ebb); if arg_count != 0 { return err!( inst, diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index ebcd703aaa..7068569f27 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -121,7 +121,7 @@ pub fn write_ebb_header( let regs = isa.map(TargetIsa::register_info); let regs = regs.as_ref(); - let mut args = func.dfg.ebb_args(ebb).iter().cloned(); + let mut args = func.dfg.ebb_params(ebb).iter().cloned(); match args.next() { None => return writeln!(w, ":"), Some(arg) => { @@ -177,8 +177,8 @@ fn type_suffix(func: &Function, inst: Inst) -> Option { if constraints.use_typevar_operand() { let ctrl_var = inst_data.typevar_operand(&func.dfg.value_lists).unwrap(); let def_ebb = match func.dfg.value_def(ctrl_var) { - ValueDef::Res(instr, _) => func.layout.inst_ebb(instr), - ValueDef::Arg(ebb, _) => Some(ebb), + ValueDef::Result(instr, _) => func.layout.inst_ebb(instr), + ValueDef::Param(ebb, _) => Some(ebb), }; if def_ebb.is_some() && def_ebb == func.layout.inst_ebb(inst) { return None; @@ -465,13 +465,13 @@ mod tests { "function %foo() native {\n ss0 = local 4\n\nebb0:\n}\n" ); - f.dfg.append_ebb_arg(ebb, types::I8); + f.dfg.append_ebb_param(ebb, types::I8); assert_eq!( f.to_string(), "function %foo() native {\n ss0 = local 4\n\nebb0(v0: i8):\n}\n" ); - f.dfg.append_ebb_arg(ebb, types::F32.by(4).unwrap()); + f.dfg.append_ebb_param(ebb, types::F32.by(4).unwrap()); assert_eq!( f.to_string(), "function %foo() native {\n ss0 = local 4\n\nebb0(v0: i8, v1: f32x4):\n}\n" diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 53b33334ea..54f9d3620e 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -136,7 +136,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short } _ => panic!("should not happen"), }; - self.builder.ebb_args_adjustment(dest_ebb, &args_types); + self.builder.ebb_params_adjustment(dest_ebb, &args_types); self.builder.declare_successor(dest_ebb, inst); } None => { @@ -273,8 +273,8 @@ where let basic_block = self.builder.ssa.header_block(ebb); // Then we change the cursor position. self.position = Position { ebb, basic_block }; - self.ebb_args_adjustment(ebb, jump_args); - self.func.dfg.ebb_args(ebb) + self.ebb_params_adjustment(ebb, jump_args); + self.func.dfg.ebb_params(ebb) } /// Declares that all the predecessors of this block are known. @@ -411,10 +411,10 @@ impl<'a, Variable> FunctionBuilder<'a, Variable> where Variable: EntityRef + Default, { - /// Retrieves all the arguments for an `Ebb` currently inferred from the jump instructions + /// Retrieves all the parameters for an `Ebb` currently inferred from the jump instructions /// inserted that target it and the SSA construction. - pub fn ebb_args(&self, ebb: Ebb) -> &[Value] { - self.func.dfg.ebb_args(ebb) + pub fn ebb_params(&self, ebb: Ebb) -> &[Value] { + self.func.dfg.ebb_params(ebb) } /// Retrieves the signature with reference `sigref` previously added with `import_signature`. @@ -422,14 +422,14 @@ where self.func.dfg.signatures.get(sigref) } - /// Creates an argument for a specific `Ebb` by appending it to the list of already existing - /// arguments. + /// Creates a parameter for a specific `Ebb` by appending it to the list of already existing + /// parameters. /// /// **Note:** this function has to be called at the creation of the `Ebb` before adding /// instructions to it, otherwise this could interfere with SSA construction. - pub fn append_ebb_arg(&mut self, ebb: Ebb, ty: Type) -> Value { + pub fn append_ebb_param(&mut self, ebb: Ebb, ty: Type) -> Value { debug_assert!(self.builder.ebbs[ebb].pristine); - self.func.dfg.append_ebb_arg(ebb, ty) + self.func.dfg.append_ebb_param(ebb, ty) } /// Returns the result values of an instruction. @@ -547,14 +547,14 @@ where debug_assert!(self.pristine); for argtyp in &self.func.signature.argument_types { self.builder.function_args_values.push( - self.func.dfg.append_ebb_arg(ebb, argtyp.value_type), + self.func.dfg.append_ebb_param(ebb, argtyp.value_type), ); } self.pristine = false; } - fn ebb_args_adjustment(&mut self, dest_ebb: Ebb, jump_args: &[Value]) { + fn ebb_params_adjustment(&mut self, dest_ebb: Ebb, jump_args: &[Value]) { if self.builder.ssa.predecessors(dest_ebb).is_empty() || self.builder.ebbs[dest_ebb].pristine { @@ -562,12 +562,12 @@ where // so the jump arguments supplied here are this Ebb' arguments // However some of the arguments might already be there // in the Ebb so we have to check they're consistent - let dest_ebb_args_len = { - let dest_ebb_args = self.func.dfg.ebb_args(dest_ebb); + let dest_ebb_params_len = { + let dest_ebb_params = self.func.dfg.ebb_params(dest_ebb); debug_assert!( - dest_ebb_args + dest_ebb_params .iter() - .zip(jump_args.iter().take(dest_ebb_args.len())) + .zip(jump_args.iter().take(dest_ebb_params.len())) .all(|(dest_arg, jump_arg)| { self.func.dfg.value_type(*jump_arg) == self.func.dfg.value_type(*dest_arg) @@ -575,12 +575,12 @@ where "the jump argument supplied has not the \ same type as the corresponding dest ebb argument" ); - dest_ebb_args.len() + dest_ebb_params.len() }; self.builder.ebbs[dest_ebb].user_arg_count = jump_args.len(); - for val in jump_args.iter().skip(dest_ebb_args_len) { + for val in jump_args.iter().skip(dest_ebb_params_len) { let ty = self.func.dfg.value_type(*val); - self.func.dfg.append_ebb_arg(dest_ebb, ty); + self.func.dfg.append_ebb_param(dest_ebb, ty); } } else { // The Ebb already has predecessors @@ -595,7 +595,7 @@ where debug_assert!( jump_args .iter() - .zip(self.func.dfg.ebb_args(dest_ebb).iter().take( + .zip(self.func.dfg.ebb_params(dest_ebb).iter().take( self.builder.ebbs[dest_ebb].user_arg_count, )) .all(|(jump_arg, dest_arg)| { diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index a359ab4b33..600143fecc 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -210,7 +210,7 @@ enum Call { /// call `seal_ebb_header_block` on it with the `Function` that you are building. /// /// This API will give you the correct SSA values to use as arguments of your instructions, -/// as well as modify the jump instruction and `Ebb` headers arguments to account for the SSA +/// as well as modify the jump instruction and `Ebb` headers parameters to account for the SSA /// Phi functions. /// impl SSABuilder @@ -261,18 +261,18 @@ where fn use_var_nonlocal(&mut self, func: &mut Function, var: Variable, ty: Type, block: Block) { let case = match self.blocks[block] { BlockData::EbbHeader(ref mut data) => { - // The block has multiple predecessors so we append an Ebb argument that + // The block has multiple predecessors so we append an Ebb parameter that // will serve as a value. if data.sealed { if data.predecessors.len() == 1 { // Only one predecessor, straightforward case UseVarCases::SealedOnePredecessor(data.predecessors[0].0) } else { - let val = func.dfg.append_ebb_arg(data.ebb, ty); + let val = func.dfg.append_ebb_param(data.ebb, ty); UseVarCases::SealedMultiplePredecessors(val, data.ebb) } } else { - let val = func.dfg.append_ebb_arg(data.ebb, ty); + let val = func.dfg.append_ebb_param(data.ebb, ty); data.undef_variables.push((var, val)); UseVarCases::Unsealed(val) } @@ -285,7 +285,7 @@ where self.calls.push(Call::FinishSealedOnePredecessor(block)); self.calls.push(Call::UseVar(pred)); } - // The block has multiple predecessors, we register the ebb argument as the current + // The block has multiple predecessors, we register the EBB parameter as the current // definition for the variable. UseVarCases::Unsealed(val) => { self.def_var(var, val, block); @@ -293,7 +293,7 @@ where } UseVarCases::SealedMultiplePredecessors(val, ebb) => { // If multiple predecessor we look up a use_var in each of them: - // if they all yield the same value no need for an Ebb argument + // if they all yield the same value no need for an EBB parameter self.def_var(var, val, block); self.begin_predecessors_lookup(val, ebb); } @@ -385,7 +385,7 @@ where } }; - // For each undef var we look up values in the predecessors and create an Ebb argument + // For each undef var we look up values in the predecessors and create an EBB parameter // only if necessary. for (var, val) in undef_vars { let ty = func.dfg.value_type(val); @@ -443,7 +443,7 @@ where } /// Examine the values from the predecessors and compute a result value, creating - /// block arguments as needed. + /// block parameters as needed. fn finish_predecessors_lookup( &mut self, func: &mut Function, @@ -499,7 +499,7 @@ where // so we don't need to have it as an ebb argument. // We need to replace all the occurences of val with pred_val but since // we can't afford a re-writing pass right now we just declare an alias. - func.dfg.remove_ebb_arg(temp_arg_val); + func.dfg.remove_ebb_param(temp_arg_val); func.dfg.change_to_alias(temp_arg_val, pred_val); pred_val } @@ -908,8 +908,8 @@ mod tests { ssa.declare_ebb_predecessor(ebb1, block3, jump_ebb2_ebb1); ssa.seal_ebb_header_block(ebb1, &mut func); - assert_eq!(func.dfg.ebb_args(ebb1)[0], z2); - assert_eq!(func.dfg.ebb_args(ebb1)[1], y3); + assert_eq!(func.dfg.ebb_params(ebb1)[0], z2); + assert_eq!(func.dfg.ebb_params(ebb1)[1], y3); assert_eq!(func.dfg.resolve_aliases(x3), x1); } @@ -1027,9 +1027,9 @@ mod tests { let block1 = ssa.declare_ebb_header_block(ebb1); ssa.declare_ebb_predecessor(ebb1, block0, jump_inst); let z2 = ssa.use_var(&mut func, z_var, I32, block1).0; - assert_eq!(func.dfg.ebb_args(ebb1)[0], z2); + assert_eq!(func.dfg.ebb_params(ebb1)[0], z2); let x2 = ssa.use_var(&mut func, x_var, I32, block1).0; - assert_eq!(func.dfg.ebb_args(ebb1)[1], x2); + assert_eq!(func.dfg.ebb_params(ebb1)[1], x2); let x3 = { let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1); cur.ins().iadd(x2, z2) @@ -1037,7 +1037,7 @@ mod tests { ssa.def_var(x_var, x3, block1); let x4 = ssa.use_var(&mut func, x_var, I32, block1).0; let y3 = ssa.use_var(&mut func, y_var, I32, block1).0; - assert_eq!(func.dfg.ebb_args(ebb1)[2], y3); + assert_eq!(func.dfg.ebb_params(ebb1)[2], y3); let y4 = { let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1); cur.ins().isub(y3, x4) @@ -1051,7 +1051,7 @@ mod tests { ssa.seal_ebb_header_block(ebb1, &mut func); // At sealing the "z" argument disappear but the remaining "x" and "y" args have to be // in the right order. - assert_eq!(func.dfg.ebb_args(ebb1)[1], y3); - assert_eq!(func.dfg.ebb_args(ebb1)[0], x2); + assert_eq!(func.dfg.ebb_params(ebb1)[1], y3); + assert_eq!(func.dfg.ebb_params(ebb1)[0], x2); } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 86642277c9..f4e3507754 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1362,7 +1362,7 @@ impl<'a> Parser<'a> { // Parse an extended basic block, add contents to `ctx`. // // extended-basic-block ::= * ebb-header { instruction } - // ebb-header ::= Ebb(ebb) [ebb-args] ":" + // ebb-header ::= Ebb(ebb) [ebb-params] ":" // fn parse_extended_basic_block(&mut self, ctx: &mut Context) -> Result<()> { let ebb_num = self.match_ebb("expected EBB header")?; @@ -1370,8 +1370,8 @@ impl<'a> Parser<'a> { self.gather_comments(ebb); if !self.optional(Token::Colon) { - // ebb-header ::= Ebb(ebb) [ * ebb-args ] ":" - self.parse_ebb_args(ctx, ebb)?; + // ebb-header ::= Ebb(ebb) [ * ebb-params ] ":" + self.parse_ebb_params(ctx, ebb)?; self.match_token( Token::Colon, "expected ':' after EBB arguments", @@ -1429,27 +1429,27 @@ impl<'a> Parser<'a> { Ok(()) } - // Parse parenthesized list of EBB arguments. Returns a vector of (u32, Type) pairs with the + // Parse parenthesized list of EBB parameters. Returns a vector of (u32, Type) pairs with the // source value numbers of the defined values and the defined types. // - // ebb-args ::= * "(" ebb-arg { "," ebb-arg } ")" - fn parse_ebb_args(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { - // ebb-args ::= * "(" ebb-arg { "," ebb-arg } ")" + // ebb-params ::= * "(" ebb-param { "," ebb-param } ")" + fn parse_ebb_params(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { + // ebb-params ::= * "(" ebb-param { "," ebb-param } ")" self.match_token( Token::LPar, "expected '(' before EBB arguments", )?; - // ebb-args ::= "(" * ebb-arg { "," ebb-arg } ")" - self.parse_ebb_arg(ctx, ebb)?; + // ebb-params ::= "(" * ebb-param { "," ebb-param } ")" + self.parse_ebb_param(ctx, ebb)?; - // ebb-args ::= "(" ebb-arg * { "," ebb-arg } ")" + // ebb-params ::= "(" ebb-param * { "," ebb-param } ")" while self.optional(Token::Comma) { - // ebb-args ::= "(" ebb-arg { "," * ebb-arg } ")" - self.parse_ebb_arg(ctx, ebb)?; + // ebb-params ::= "(" ebb-param { "," * ebb-param } ")" + self.parse_ebb_param(ctx, ebb)?; } - // ebb-args ::= "(" ebb-arg { "," ebb-arg } * ")" + // ebb-params ::= "(" ebb-param { "," ebb-param } * ")" self.match_token( Token::RPar, "expected ')' after EBB arguments", @@ -1458,27 +1458,27 @@ impl<'a> Parser<'a> { Ok(()) } - // Parse a single EBB argument declaration, and append it to `ebb`. + // Parse a single EBB parameter declaration, and append it to `ebb`. // - // ebb-arg ::= * Value(v) ":" Type(t) arg-loc? + // ebb-param ::= * Value(v) ":" Type(t) arg-loc? // arg-loc ::= "[" value-location "]" // - fn parse_ebb_arg(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { - // ebb-arg ::= * Value(v) ":" Type(t) arg-loc? + fn parse_ebb_param(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { + // ebb-param ::= * Value(v) ":" Type(t) arg-loc? let v = self.match_value("EBB argument must be a value")?; let v_location = self.loc; - // ebb-arg ::= Value(v) * ":" Type(t) arg-loc? + // ebb-param ::= Value(v) * ":" Type(t) arg-loc? self.match_token( Token::Colon, "expected ':' after EBB argument", )?; - // ebb-arg ::= Value(v) ":" * Type(t) arg-loc? + // ebb-param ::= Value(v) ":" * Type(t) arg-loc? let t = self.match_type("expected EBB argument type")?; // Allocate the EBB argument and add the mapping. - let value = ctx.function.dfg.append_ebb_arg(ebb, t); + let value = ctx.function.dfg.append_ebb_param(ebb, t); ctx.map.def_value(v, value, &v_location)?; - // ebb-arg ::= Value(v) ":" Type(t) * arg-loc? + // ebb-param ::= Value(v) ":" Type(t) * arg-loc? if self.optional(Token::LBracket) { let loc = self.parse_value_location(ctx)?; ctx.function.locations[value] = loc; @@ -2473,10 +2473,10 @@ mod tests { let mut ebbs = func.layout.ebbs(); let ebb0 = ebbs.next().unwrap(); - assert_eq!(func.dfg.ebb_args(ebb0), &[]); + assert_eq!(func.dfg.ebb_params(ebb0), &[]); let ebb4 = ebbs.next().unwrap(); - let ebb4_args = func.dfg.ebb_args(ebb4); + let ebb4_args = func.dfg.ebb_params(ebb4); assert_eq!(ebb4_args.len(), 1); assert_eq!(func.dfg.value_type(ebb4_args[0]), types::I32); } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index a4a22435ae..7c2d8ae15e 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -121,7 +121,7 @@ pub fn translate_operator( Operator::Block { ty } => { let next = builder.create_ebb(); if let Ok(ty_cre) = type_to_type(&ty) { - builder.append_ebb_arg(next, ty_cre); + builder.append_ebb_param(next, ty_cre); } state.push_block(next, num_return_values(ty)); } @@ -129,7 +129,7 @@ pub fn translate_operator( let loop_body = builder.create_ebb(); let next = builder.create_ebb(); if let Ok(ty_cre) = type_to_type(&ty) { - builder.append_ebb_arg(next, ty_cre); + builder.append_ebb_param(next, ty_cre); } builder.ins().jump(loop_body, &[]); state.push_loop(loop_body, next, num_return_values(ty)); @@ -146,7 +146,7 @@ pub fn translate_operator( // - either the If have an Else clause, in that case the destination of this jump // instruction will be changed later when we translate the Else operator. if let Ok(ty_cre) = type_to_type(&ty) { - builder.append_ebb_arg(if_not, ty_cre); + builder.append_ebb_param(if_not, ty_cre); } state.push_if(jump_inst, if_not, num_return_values(ty)); } @@ -190,7 +190,7 @@ pub fn translate_operator( } state.stack.truncate(frame.original_stack_size()); state.stack.extend_from_slice( - builder.ebb_args(frame.following_code()), + builder.ebb_params(frame.following_code()), ); } /**************************** Branch instructions ********************************* @@ -876,7 +876,7 @@ fn translate_unreachable_operator( // And add the return values of the block but only if the next block is reachble // (which corresponds to testing if the stack depth is 1) if state.real_unreachable_stack_depth == 1 { - stack.extend_from_slice(builder.ebb_args(frame.following_code())); + stack.extend_from_slice(builder.ebb_params(frame.following_code())); } state.real_unreachable_stack_depth -= 1; } From b3fb41087ed3200144a16846f759b03dd8dd8150 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 19 Oct 2017 17:39:23 -0700 Subject: [PATCH 1279/3084] Use the term "Function parameter" instead of "argument". Rename the ArgumentType type to AbiParam since it describes the ABI characteristics of a parameter or return value, not just the value type. In Signature, rename members argument_types and return_types to "params" and "returns". Again, they are not just types. Fix a couple lingering references to "EBB arguments". --- lib/cretonne/src/abi.rs | 14 +-- lib/cretonne/src/ir/dfg.rs | 6 +- lib/cretonne/src/ir/extfunc.rs | 110 +++++++++++------------- lib/cretonne/src/ir/function.rs | 8 +- lib/cretonne/src/ir/mod.rs | 2 +- lib/cretonne/src/isa/intel/abi.rs | 8 +- lib/cretonne/src/isa/riscv/abi.rs | 14 +-- lib/cretonne/src/legalizer/boundary.rs | 93 ++++++++++---------- lib/cretonne/src/legalizer/globalvar.rs | 2 +- lib/cretonne/src/regalloc/affinity.rs | 4 +- lib/cretonne/src/regalloc/coalescing.rs | 6 +- lib/cretonne/src/regalloc/coloring.rs | 31 +++---- lib/cretonne/src/regalloc/liveness.rs | 4 +- lib/cretonne/src/regalloc/reload.rs | 20 ++--- lib/cretonne/src/regalloc/spilling.rs | 2 +- lib/cretonne/src/verifier/locations.rs | 8 +- lib/cretonne/src/verifier/mod.rs | 26 +++--- lib/frontend/src/frontend.rs | 12 +-- lib/frontend/src/lib.rs | 6 +- lib/reader/src/parser.rs | 60 ++++++------- lib/wasm/src/func_translator.rs | 26 ++---- lib/wasm/src/sections_translator.rs | 10 +-- lib/wasm/src/state.rs | 8 +- 23 files changed, 225 insertions(+), 255 deletions(-) diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index 6728f57691..7eb459fe6b 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -3,7 +3,7 @@ //! This module provides functions and data structures that are useful for implementing the //! `TargetIsa::legalize_signature()` method. -use ir::{ArgumentLoc, ArgumentType, ArgumentExtension, Type}; +use ir::{ArgumentLoc, AbiParam, ArgumentExtension, Type}; use std::cmp::Ordering; /// Legalization action to perform on a single argument or return value when converting a @@ -81,13 +81,13 @@ impl ValueConversion { /// This will be implemented by individual ISAs. pub trait ArgAssigner { /// Pick an assignment action for function argument (or return value) `arg`. - fn assign(&mut self, arg: &ArgumentType) -> ArgAction; + fn assign(&mut self, arg: &AbiParam) -> ArgAction; } /// Legalize the arguments in `args` using the given argument assigner. /// /// This function can be used for both arguments and return values. -pub fn legalize_args(args: &mut Vec, aa: &mut AA) { +pub fn legalize_args(args: &mut Vec, aa: &mut AA) { // Iterate over the arguments. // We may need to mutate the vector in place, so don't use a normal iterator, and clone the // argument to avoid holding a reference. @@ -108,7 +108,7 @@ pub fn legalize_args(args: &mut Vec, aa: &mut AA) } // Split this argument into two smaller ones. Then revisit both. ArgAction::Convert(conv) => { - let new_arg = ArgumentType { + let new_arg = AbiParam { value_type: conv.apply(arg.value_type), ..arg }; @@ -143,7 +143,7 @@ pub fn legalize_args(args: &mut Vec, aa: &mut AA) /// It may be necessary to call `legalize_abi_value` more than once for a given argument before the /// desired argument type appears. This will happen when a vector or integer type needs to be split /// more than once, for example. -pub fn legalize_abi_value(have: Type, arg: &ArgumentType) -> ValueConversion { +pub fn legalize_abi_value(have: Type, arg: &AbiParam) -> ValueConversion { let have_bits = have.bits(); let arg_bits = arg.value_type.bits(); @@ -186,11 +186,11 @@ pub fn legalize_abi_value(have: Type, arg: &ArgumentType) -> ValueConversion { mod tests { use super::*; use ir::types; - use ir::ArgumentType; + use ir::AbiParam; #[test] fn legalize() { - let mut arg = ArgumentType::new(types::I32); + let mut arg = AbiParam::new(types::I32); assert_eq!( legalize_abi_value(types::I64X2, &arg), diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 6593b12f92..4a98fc6902 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -463,10 +463,10 @@ impl DataFlowGraph { // Get the call signature if this is a function call. if let Some(sig) = self.call_signature(inst) { // Create result values corresponding to the call return types. - let var_results = self.signatures[sig].return_types.len(); + let var_results = self.signatures[sig].returns.len(); total_results += var_results; for res_idx in 0..var_results { - let ty = self.signatures[sig].return_types[res_idx].value_type; + let ty = self.signatures[sig].returns[res_idx].value_type; if let Some(Some(v)) = reuse.next() { debug_assert_eq!(self.value_type(v), ty, "Reused {} is wrong type", ty); self.attach_result(inst, v); @@ -632,7 +632,7 @@ impl DataFlowGraph { // Not a fixed result, try to extract a return type from the call signature. self.call_signature(inst).and_then(|sigref| { self.signatures[sigref] - .return_types + .returns .get(result_idx - fixed_results) .map(|&arg| arg.value_type) }) diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 8b8cc9dca0..c82ff53bec 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -13,17 +13,17 @@ use std::str::FromStr; /// Function signature. /// -/// The function signature describes the types of arguments and return values along with other -/// details that are needed to call a function correctly. +/// The function signature describes the types of formal parameters and return values along with +/// other details that are needed to call a function correctly. /// /// A signature can optionally include ISA-specific ABI information which specifies exactly how /// arguments and return values are passed. #[derive(Clone, Debug)] pub struct Signature { - /// Types of the arguments passed to the function. - pub argument_types: Vec, - /// Types returned from the function. - pub return_types: Vec, + /// The arguments passed to the function. + pub params: Vec, + /// Values returned from the function. + pub returns: Vec, /// Calling convention. pub call_conv: CallConv, @@ -31,7 +31,7 @@ pub struct Signature { /// When the signature has been legalized to a specific ISA, this holds the size of the /// argument array on the stack. Before legalization, this is `None`. /// - /// This can be computed from the legalized `argument_types` array as the maximum (offset plus + /// This can be computed from the legalized `params` array as the maximum (offset plus /// byte size) of the `ArgumentLoc::Stack(offset)` argument. pub argument_bytes: Option, } @@ -40,8 +40,8 @@ impl Signature { /// Create a new blank signature. pub fn new(call_conv: CallConv) -> Signature { Signature { - argument_types: Vec::new(), - return_types: Vec::new(), + params: Vec::new(), + returns: Vec::new(), call_conv, argument_bytes: None, } @@ -49,18 +49,18 @@ impl Signature { /// Clear the signature so it is identical to a fresh one returned by `new()`. pub fn clear(&mut self, call_conv: CallConv) { - self.argument_types.clear(); - self.return_types.clear(); + self.params.clear(); + self.returns.clear(); self.call_conv = call_conv; self.argument_bytes = None; } /// Compute the size of the stack arguments and mark signature as legalized. /// - /// Even if there are no stack arguments, this will set `argument_types` to `Some(0)` instead + /// Even if there are no stack arguments, this will set `params` to `Some(0)` instead /// of `None`. This indicates that the signature has been legalized. pub fn compute_argument_bytes(&mut self) { - let bytes = self.argument_types + let bytes = self.params .iter() .filter_map(|arg| match arg.location { ArgumentLoc::Stack(offset) if offset >= 0 => { @@ -77,22 +77,16 @@ impl Signature { DisplaySignature(self, regs.into()) } - /// Find the index of a presumed unique special-purpose argument. - pub fn special_arg_index(&self, purpose: ArgumentPurpose) -> Option { - self.argument_types.iter().rposition( - |arg| arg.purpose == purpose, - ) + /// Find the index of a presumed unique special-purpose parameter. + pub fn special_param_index(&self, purpose: ArgumentPurpose) -> Option { + self.params.iter().rposition(|arg| arg.purpose == purpose) } } /// Wrapper type capable of displaying a `Signature` with correct register names. pub struct DisplaySignature<'a>(&'a Signature, Option<&'a RegInfo>); -fn write_list( - f: &mut fmt::Formatter, - args: &[ArgumentType], - regs: Option<&RegInfo>, -) -> fmt::Result { +fn write_list(f: &mut fmt::Formatter, args: &[AbiParam], regs: Option<&RegInfo>) -> fmt::Result { match args.split_first() { None => {} Some((first, rest)) => { @@ -108,11 +102,11 @@ fn write_list( impl<'a> fmt::Display for DisplaySignature<'a> { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "(")?; - write_list(f, &self.0.argument_types, self.1)?; + write_list(f, &self.0.params, self.1)?; write!(f, ")")?; - if !self.0.return_types.is_empty() { + if !self.0.returns.is_empty() { write!(f, " -> ")?; - write_list(f, &self.0.return_types, self.1)?; + write_list(f, &self.0.returns, self.1)?; } write!(f, " {}", self.0.call_conv) } @@ -124,12 +118,12 @@ impl fmt::Display for Signature { } } -/// Function argument or return value type. +/// Function parameter or return value descriptor. /// /// This describes the value type being passed to or from a function along with flags that affect /// how the argument is passed. #[derive(Copy, Clone, Debug)] -pub struct ArgumentType { +pub struct AbiParam { /// Type of the argument value. pub value_type: Type, /// Special purpose of argument, or `Normal`. @@ -142,10 +136,10 @@ pub struct ArgumentType { pub location: ArgumentLoc, } -impl ArgumentType { - /// Create an argument type with default flags. - pub fn new(vt: Type) -> ArgumentType { - ArgumentType { +impl AbiParam { + /// Create a parameter with default flags. + pub fn new(vt: Type) -> AbiParam { + AbiParam { value_type: vt, extension: ArgumentExtension::None, purpose: ArgumentPurpose::Normal, @@ -153,9 +147,9 @@ impl ArgumentType { } } - /// Create a special-purpose argument type that is not (yet) bound to a specific register. - pub fn special(vt: Type, purpose: ArgumentPurpose) -> ArgumentType { - ArgumentType { + /// Create a special-purpose parameter that is not (yet) bound to a specific register. + pub fn special(vt: Type, purpose: ArgumentPurpose) -> AbiParam { + AbiParam { value_type: vt, extension: ArgumentExtension::None, purpose, @@ -163,9 +157,9 @@ impl ArgumentType { } } - /// Create an argument type for a special-purpose register. - pub fn special_reg(vt: Type, purpose: ArgumentPurpose, regunit: RegUnit) -> ArgumentType { - ArgumentType { + /// Create a parameter for a special-purpose register. + pub fn special_reg(vt: Type, purpose: ArgumentPurpose, regunit: RegUnit) -> AbiParam { + AbiParam { value_type: vt, extension: ArgumentExtension::None, purpose, @@ -173,34 +167,34 @@ impl ArgumentType { } } - /// Convert `self` to an argument type with the `uext` flag set. - pub fn uext(self) -> ArgumentType { + /// Convert `self` to a parameter with the `uext` flag set. + pub fn uext(self) -> AbiParam { debug_assert!(self.value_type.is_int(), "uext on {} arg", self.value_type); - ArgumentType { + AbiParam { extension: ArgumentExtension::Uext, ..self } } - /// Convert `self` to an argument type with the `sext` flag set. - pub fn sext(self) -> ArgumentType { + /// Convert `self` to a parameter type with the `sext` flag set. + pub fn sext(self) -> AbiParam { debug_assert!(self.value_type.is_int(), "sext on {} arg", self.value_type); - ArgumentType { + AbiParam { extension: ArgumentExtension::Sext, ..self } } /// Return an object that can display `self` with correct register names. - pub fn display<'a, R: Into>>(&'a self, regs: R) -> DisplayArgumentType<'a> { - DisplayArgumentType(self, regs.into()) + pub fn display<'a, R: Into>>(&'a self, regs: R) -> DisplayAbiParam<'a> { + DisplayAbiParam(self, regs.into()) } } -/// Wrapper type capable of displaying an `ArgumentType` with correct register names. -pub struct DisplayArgumentType<'a>(&'a ArgumentType, Option<&'a RegInfo>); +/// Wrapper type capable of displaying a `AbiParam` with correct register names. +pub struct DisplayAbiParam<'a>(&'a AbiParam, Option<&'a RegInfo>); -impl<'a> fmt::Display for DisplayArgumentType<'a> { +impl<'a> fmt::Display for DisplayAbiParam<'a> { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "{}", self.0.value_type)?; match self.0.extension { @@ -220,7 +214,7 @@ impl<'a> fmt::Display for DisplayArgumentType<'a> { } } -impl fmt::Display for ArgumentType { +impl fmt::Display for AbiParam { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { self.display(None).fmt(f) } @@ -387,7 +381,7 @@ mod tests { #[test] fn argument_type() { - let t = ArgumentType::new(I32); + let t = AbiParam::new(I32); assert_eq!(t.to_string(), "i32"); let mut t = t.uext(); assert_eq!(t.to_string(), "i32 uext"); @@ -423,25 +417,23 @@ mod tests { fn signatures() { let mut sig = Signature::new(CallConv::SpiderWASM); assert_eq!(sig.to_string(), "() spiderwasm"); - sig.argument_types.push(ArgumentType::new(I32)); + sig.params.push(AbiParam::new(I32)); assert_eq!(sig.to_string(), "(i32) spiderwasm"); - sig.return_types.push(ArgumentType::new(F32)); + sig.returns.push(AbiParam::new(F32)); assert_eq!(sig.to_string(), "(i32) -> f32 spiderwasm"); - sig.argument_types.push( - ArgumentType::new(I32.by(4).unwrap()), - ); + sig.params.push(AbiParam::new(I32.by(4).unwrap())); assert_eq!(sig.to_string(), "(i32, i32x4) -> f32 spiderwasm"); - sig.return_types.push(ArgumentType::new(B8)); + sig.returns.push(AbiParam::new(B8)); assert_eq!(sig.to_string(), "(i32, i32x4) -> f32, b8 spiderwasm"); // Test the offset computation algorithm. assert_eq!(sig.argument_bytes, None); - sig.argument_types[1].location = ArgumentLoc::Stack(8); + sig.params[1].location = ArgumentLoc::Stack(8); sig.compute_argument_bytes(); // An `i32x4` at offset 8 requires a 24-byte argument array. assert_eq!(sig.argument_bytes, Some(24)); // Order does not matter. - sig.argument_types[0].location = ArgumentLoc::Stack(24); + sig.params[0].location = ArgumentLoc::Stack(24); sig.compute_argument_bytes(); assert_eq!(sig.argument_bytes, Some(28)); diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index bc1cc980bc..dac951ec99 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -144,12 +144,12 @@ impl Function { DisplayFunction(self, isa.into()) } - /// Find a presumed unique special-purpose function argument value. + /// Find a presumed unique special-purpose function parameter value. /// - /// Returns the value of the last `purpose` argument, or `None` if no such argument exists. - pub fn special_arg(&self, purpose: ir::ArgumentPurpose) -> Option { + /// Returns the value of the last `purpose` parameter, or `None` if no such parameter exists. + pub fn special_param(&self, purpose: ir::ArgumentPurpose) -> Option { let entry = self.layout.entry_block().expect("Function is empty"); - self.signature.special_arg_index(purpose).map(|i| { + self.signature.special_param_index(purpose).map(|i| { self.dfg.ebb_params(entry)[i] }) } diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 2165937cee..bb9730595f 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -24,7 +24,7 @@ mod valueloc; pub use ir::builder::{InstBuilder, InstBuilderBase, InstInserterBase, InsertBuilder}; pub use ir::dfg::{DataFlowGraph, ValueDef}; pub use ir::entities::{Ebb, Inst, Value, StackSlot, GlobalVar, JumpTable, FuncRef, SigRef, Heap}; -pub use ir::extfunc::{Signature, CallConv, ArgumentType, ArgumentExtension, ArgumentPurpose, +pub use ir::extfunc::{Signature, CallConv, AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData}; pub use ir::funcname::FunctionName; pub use ir::function::Function; diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index b45d439db3..ec623feed8 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -6,7 +6,7 @@ use regalloc::AllocatableSet; use settings as shared_settings; use super::registers::{GPR, FPR, RU}; use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; -use ir::{ArgumentType, ArgumentPurpose, ArgumentLoc, ArgumentExtension}; +use ir::{AbiParam, ArgumentPurpose, ArgumentLoc, ArgumentExtension}; /// Argument registers for x86-64 static ARG_GPRS: [RU; 6] = [RU::rdi, RU::rsi, RU::rdx, RU::rcx, RU::r8, RU::r9]; @@ -41,7 +41,7 @@ impl Args { } impl ArgAssigner for Args { - fn assign(&mut self, arg: &ArgumentType) -> ArgAction { + fn assign(&mut self, arg: &AbiParam) -> ArgAction { let ty = arg.value_type; // Check for a legal type. @@ -117,10 +117,10 @@ pub fn legalize_signature(sig: &mut ir::Signature, flags: &shared_settings::Flag args = Args::new(bits, &[], 0); } - legalize_args(&mut sig.argument_types, &mut args); + legalize_args(&mut sig.params, &mut args); let mut rets = Args::new(bits, &RET_GPRS, 2); - legalize_args(&mut sig.return_types, &mut rets); + legalize_args(&mut sig.returns, &mut rets); } /// Get register class for a type appearing in a legalized signature. diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index f44ff705ec..00ac99e305 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -6,7 +6,7 @@ //! This doesn't support the soft-float ABI at the moment. use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; -use ir::{self, Type, ArgumentType, ArgumentLoc, ArgumentExtension, ArgumentPurpose}; +use ir::{self, Type, AbiParam, ArgumentLoc, ArgumentExtension, ArgumentPurpose}; use isa::RegClass; use regalloc::AllocatableSet; use settings as shared_settings; @@ -36,7 +36,7 @@ impl Args { } impl ArgAssigner for Args { - fn assign(&mut self, arg: &ArgumentType) -> ArgAction { + fn assign(&mut self, arg: &AbiParam) -> ArgAction { fn align(value: u32, to: u32) -> u32 { (value + to - 1) & !(to - 1) } @@ -95,10 +95,10 @@ pub fn legalize_signature( let bits = if flags.is_64bit() { 64 } else { 32 }; let mut args = Args::new(bits, isa_flags.enable_e()); - legalize_args(&mut sig.argument_types, &mut args); + legalize_args(&mut sig.params, &mut args); let mut rets = Args::new(bits, isa_flags.enable_e()); - legalize_args(&mut sig.return_types, &mut rets); + legalize_args(&mut sig.returns, &mut rets); if current { let ptr = Type::int(bits).unwrap(); @@ -108,9 +108,9 @@ pub fn legalize_signature( // The `jalr` instruction implementing a return can technically accept the return address // in any register, but a micro-architecture with a return address predictor will only // recognize it as a return if the address is in `x1`. - let link = ArgumentType::special_reg(ptr, ArgumentPurpose::Link, GPR.unit(1)); - sig.argument_types.push(link); - sig.return_types.push(link); + let link = AbiParam::special_reg(ptr, ArgumentPurpose::Link, GPR.unit(1)); + sig.params.push(link); + sig.returns.push(link); } } diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 6276e0fb04..e9f7f4c564 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -21,7 +21,7 @@ use abi::{legalize_abi_value, ValueConversion}; use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::{Function, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, Signature, SigRef, - ArgumentType, ArgumentPurpose, ArgumentLoc, ValueLoc}; + AbiParam, ArgumentPurpose, ArgumentLoc, ValueLoc}; use ir::instructions::CallInfo; use isa::TargetIsa; use legalizer::split::{isplit, vsplit}; @@ -40,20 +40,20 @@ pub fn legalize_signatures(func: &mut Function, isa: &TargetIsa) { } if let Some(entry) = func.layout.entry_block() { - legalize_entry_arguments(func, entry); - spill_entry_arguments(func, entry); + legalize_entry_params(func, entry); + spill_entry_params(func, entry); } } -/// Legalize the entry block arguments after `func`'s signature has been legalized. +/// Legalize the entry block parameters after `func`'s signature has been legalized. /// -/// The legalized signature may contain more arguments than the original signature, and the -/// argument types have been changed. This function goes through the arguments to the entry EBB and -/// replaces them with arguments of the right type for the ABI. +/// The legalized signature may contain more parameters than the original signature, and the +/// parameter types have been changed. This function goes through the parameters of the entry EBB +/// and replaces them with parameters of the right type for the ABI. /// -/// The original entry EBB arguments are computed from the new ABI arguments by code inserted at +/// The original entry EBB parameters are computed from the new ABI parameters by code inserted at /// the top of the entry block. -fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { +fn legalize_entry_params(func: &mut Function, entry: Ebb) { let mut has_sret = false; let mut has_link = false; let mut has_vmctx = false; @@ -74,7 +74,7 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { while let Some(arg) = ebb_params.get(old_arg, &pos.func.dfg.value_lists) { old_arg += 1; - let abi_type = pos.func.signature.argument_types[abi_arg]; + let abi_type = pos.func.signature.params[abi_arg]; let arg_type = pos.func.dfg.value_type(arg); if arg_type == abi_type.value_type { // No value translation is necessary, this argument matches the ABI type. @@ -98,9 +98,9 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { } abi_arg += 1; } else { - // Compute the value we want for `arg` from the legalized ABI arguments. + // Compute the value we want for `arg` from the legalized ABI parameters. let mut get_arg = |func: &mut Function, ty| { - let abi_type = func.signature.argument_types[abi_arg]; + let abi_type = func.signature.params[abi_arg]; assert_eq!( abi_type.purpose, ArgumentPurpose::Normal, @@ -120,15 +120,15 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { } } - // The legalized signature may contain additional arguments representing special-purpose + // The legalized signature may contain additional parameters representing special-purpose // registers. - for &arg in &pos.func.signature.argument_types[abi_arg..] { + for &arg in &pos.func.signature.params[abi_arg..] { match arg.purpose { - // Any normal arguments should have been processed above. + // Any normal parameters should have been processed above. ArgumentPurpose::Normal => { panic!("Leftover arg: {}", arg); } - // The callee-save arguments should not appear until after register allocation is + // The callee-save parameters should not appear until after register allocation is // done. ArgumentPurpose::FramePointer | ArgumentPurpose::CalleeSaved => { @@ -136,19 +136,19 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { } // These can be meaningfully added by `legalize_signature()`. ArgumentPurpose::Link => { - assert!(!has_link, "Multiple link arguments found"); + assert!(!has_link, "Multiple link parameters found"); has_link = true; } ArgumentPurpose::StructReturn => { - assert!(!has_sret, "Multiple sret arguments found"); + assert!(!has_sret, "Multiple sret parameters found"); has_sret = true; } ArgumentPurpose::VMContext => { - assert!(!has_vmctx, "Multiple vmctx arguments found"); + assert!(!has_vmctx, "Multiple vmctx parameters found"); has_vmctx = true; } ArgumentPurpose::SignatureId => { - assert!(!has_sigid, "Multiple sigid arguments found"); + assert!(!has_sigid, "Multiple sigid parameters found"); has_sigid = true; } } @@ -164,12 +164,12 @@ fn legalize_entry_arguments(func: &mut Function, entry: Ebb) { /// The cursor `pos` points to a call instruction with at least one return value. The cursor will /// be left pointing after the instructions inserted to convert the return values. /// -/// This function is very similar to the `legalize_entry_arguments` function above. +/// This function is very similar to the `legalize_entry_params` function above. /// /// Returns the possibly new instruction representing the call. fn legalize_inst_results(pos: &mut FuncCursor, mut get_abi_type: ResType) -> Inst where - ResType: FnMut(&Function, usize) -> ArgumentType, + ResType: FnMut(&Function, usize) -> AbiParam, { let call = pos.current_inst().expect( "Cursor must point to a call instruction", @@ -230,7 +230,7 @@ fn convert_from_abi( get_arg: &mut GetArg, ) -> Value where - GetArg: FnMut(&mut Function, Type) -> Result, + GetArg: FnMut(&mut Function, Type) -> Result, { // Terminate the recursion when we get the desired type. let arg_type = match get_arg(pos.func, ty) { @@ -304,7 +304,7 @@ where /// 1. If the suggested argument has an acceptable value type, consume it by adding it to the list /// of arguments and return `Ok(())`. /// 2. If the suggested argument doesn't have the right value type, don't change anything, but -/// return the `Err(ArgumentType)` that is needed. +/// return the `Err(AbiParam)` that is needed. /// fn convert_to_abi( pos: &mut FuncCursor, @@ -312,7 +312,7 @@ fn convert_to_abi( value: Value, put_arg: &mut PutArg, ) where - PutArg: FnMut(&mut Function, Value) -> Result<(), ArgumentType>, + PutArg: FnMut(&mut Function, Value) -> Result<(), AbiParam>, { // Start by invoking the closure to either terminate the recursion or get the argument type // we're trying to match. @@ -355,7 +355,7 @@ fn convert_to_abi( } /// Check if a sequence of arguments match a desired sequence of argument types. -fn check_arg_types(dfg: &DataFlowGraph, args: &[Value], types: &[ArgumentType]) -> bool { +fn check_arg_types(dfg: &DataFlowGraph, args: &[Value], types: &[AbiParam]) -> bool { let arg_types = args.iter().map(|&v| dfg.value_type(v)); let sig_types = types.iter().map(|&at| at.value_type); arg_types.eq(sig_types) @@ -374,8 +374,8 @@ fn check_call_signature(dfg: &DataFlowGraph, inst: Inst) -> Result<(), SigRef> { }; let sig = &dfg.signatures[sig_ref]; - if check_arg_types(dfg, args, &sig.argument_types[..]) && - check_arg_types(dfg, dfg.inst_results(inst), &sig.return_types[..]) + if check_arg_types(dfg, args, &sig.params[..]) && + check_arg_types(dfg, dfg.inst_results(inst), &sig.returns[..]) { // All types check out. Ok(()) @@ -387,13 +387,13 @@ fn check_call_signature(dfg: &DataFlowGraph, inst: Inst) -> Result<(), SigRef> { /// Check if the arguments of the return `inst` match the signature. fn check_return_signature(dfg: &DataFlowGraph, inst: Inst, sig: &Signature) -> bool { - check_arg_types(dfg, dfg.inst_variable_args(inst), &sig.return_types) + check_arg_types(dfg, dfg.inst_variable_args(inst), &sig.returns) } /// Insert ABI conversion code for the arguments to the call or return instruction at `pos`. /// /// - `abi_args` is the number of arguments that the ABI signature requires. -/// - `get_abi_type` is a closure that can provide the desired `ArgumentType` for a given ABI +/// - `get_abi_type` is a closure that can provide the desired `AbiParam` for a given ABI /// argument number in `0..abi_args`. /// fn legalize_inst_arguments( @@ -402,7 +402,7 @@ fn legalize_inst_arguments( abi_args: usize, mut get_abi_type: ArgType, ) where - ArgType: FnMut(&Function, usize) -> ArgumentType, + ArgType: FnMut(&Function, usize) -> AbiParam, { let inst = pos.current_inst().expect( "Cursor must point to a call instruction", @@ -498,14 +498,14 @@ pub fn handle_call_abi(mut inst: Inst, func: &mut Function, cfg: &ControlFlowGra }; // OK, we need to fix the call arguments to match the ABI signature. - let abi_args = pos.func.dfg.signatures[sig_ref].argument_types.len(); + let abi_args = pos.func.dfg.signatures[sig_ref].params.len(); legalize_inst_arguments(pos, cfg, abi_args, |func, abi_arg| { - func.dfg.signatures[sig_ref].argument_types[abi_arg] + func.dfg.signatures[sig_ref].params[abi_arg] }); - if !pos.func.dfg.signatures[sig_ref].return_types.is_empty() { + if !pos.func.dfg.signatures[sig_ref].returns.is_empty() { inst = legalize_inst_results(pos, |func, abi_res| { - func.dfg.signatures[sig_ref].return_types[abi_res] + func.dfg.signatures[sig_ref].returns[abi_res] }); } @@ -537,7 +537,7 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph // Count the special-purpose return values (`link`, `sret`, and `vmctx`) that were appended to // the legalized signature. let special_args = func.signature - .return_types + .returns .iter() .rev() .take_while(|&rt| { @@ -545,13 +545,13 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph rt.purpose == ArgumentPurpose::VMContext }) .count(); - let abi_args = func.signature.return_types.len() - special_args; + let abi_args = func.signature.returns.len() - special_args; let pos = &mut FuncCursor::new(func).at_inst(inst); pos.use_srcloc(inst); legalize_inst_arguments(pos, cfg, abi_args, |func, abi_arg| { - func.signature.return_types[abi_arg] + func.signature.returns[abi_arg] }); assert_eq!(pos.func.dfg.inst_variable_args(inst).len(), abi_args); @@ -565,7 +565,7 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph pos.func.dfg.display_inst(inst, None) ); let mut vlist = pos.func.dfg[inst].take_value_list().unwrap(); - for arg in &pos.func.signature.return_types[abi_args..] { + for arg in &pos.func.signature.returns[abi_args..] { match arg.purpose { ArgumentPurpose::Link | ArgumentPurpose::StructReturn | @@ -578,7 +578,7 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph // the end. let idx = pos.func .signature - .argument_types + .params .iter() .rposition(|t| t.purpose == arg.purpose) .expect("No matching special purpose argument."); @@ -605,17 +605,12 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph true } -/// Assign stack slots to incoming function arguments on the stack. +/// Assign stack slots to incoming function parameters on the stack. /// /// Values that are passed into the function on the stack must be assigned to an `IncomingArg` /// stack slot already during legalization. -fn spill_entry_arguments(func: &mut Function, entry: Ebb) { - for (abi, &arg) in func.signature.argument_types.iter().zip( - func.dfg.ebb_params( - entry, - ), - ) - { +fn spill_entry_params(func: &mut Function, entry: Ebb) { + for (abi, &arg) in func.signature.params.iter().zip(func.dfg.ebb_params(entry)) { if let ArgumentLoc::Stack(offset) = abi.location { let ss = func.stack_slots.make_incoming_arg(abi.value_type, offset); func.locations[arg] = ValueLoc::Stack(ss); @@ -648,7 +643,7 @@ fn spill_call_arguments(pos: &mut FuncCursor) -> bool { .dfg .inst_variable_args(inst) .iter() - .zip(&pos.func.dfg.signatures[sig_ref].argument_types) + .zip(&pos.func.dfg.signatures[sig_ref].params) .enumerate() .filter_map(|(idx, (&arg, abi))| { match abi.location { diff --git a/lib/cretonne/src/legalizer/globalvar.rs b/lib/cretonne/src/legalizer/globalvar.rs index 7c459fb8c4..c51a8b8cfc 100644 --- a/lib/cretonne/src/legalizer/globalvar.rs +++ b/lib/cretonne/src/legalizer/globalvar.rs @@ -27,7 +27,7 @@ pub fn expand_global_addr(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut Co /// Expand a `global_addr` instruction for a vmctx global. fn vmctx_addr(inst: ir::Inst, func: &mut ir::Function, offset: i64) { // Get the value representing the `vmctx` argument. - let vmctx = func.special_arg(ir::ArgumentPurpose::VMContext).expect( + let vmctx = func.special_param(ir::ArgumentPurpose::VMContext).expect( "Missing vmctx parameter", ); diff --git a/lib/cretonne/src/regalloc/affinity.rs b/lib/cretonne/src/regalloc/affinity.rs index 5cfd3ada97..1c50cbbbab 100644 --- a/lib/cretonne/src/regalloc/affinity.rs +++ b/lib/cretonne/src/regalloc/affinity.rs @@ -9,7 +9,7 @@ //! larger register class instead. use std::fmt; -use ir::{ArgumentType, ArgumentLoc}; +use ir::{AbiParam, ArgumentLoc}; use isa::{TargetIsa, RegInfo, RegClassIndex, OperandConstraint, ConstraintKind}; /// Preferred register allocation for an SSA value. @@ -48,7 +48,7 @@ impl Affinity { } /// Create an affinity that matches an ABI argument for `isa`. - pub fn abi(arg: &ArgumentType, isa: &TargetIsa) -> Affinity { + pub fn abi(arg: &AbiParam, isa: &TargetIsa) -> Affinity { match arg.location { ArgumentLoc::Unassigned => Affinity::None, ArgumentLoc::Reg(_) => Affinity::Reg(isa.regclass_for_abi_type(arg.value_type).into()), diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index b2897fe331..28cde8c10c 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -418,14 +418,12 @@ impl<'a> Context<'a> { self.func.dfg.display_inst(pred_inst, self.isa) ); - // Never coalesce incoming function arguments on the stack. These arguments are + // Never coalesce incoming function parameters on the stack. These parameters are // pre-spilled, and the rest of the virtual register would be forced to spill to the // `incoming_arg` stack slot too. if let ValueDef::Param(def_ebb, def_num) = self.func.dfg.value_def(pred_val) { if Some(def_ebb) == self.func.layout.entry_block() && - self.func.signature.argument_types[def_num] - .location - .is_stack() + self.func.signature.params[def_num].location.is_stack() { dbg!("Isolating incoming stack parameter {}", pred_val); let new_val = self.split_pred(pred_inst, pred_ebb, argnum, pred_val); diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 89704b0865..b6b403fb30 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -45,7 +45,7 @@ use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; use ir::{Ebb, Inst, Value, Function, ValueLoc, SigRef}; -use ir::{InstBuilder, ArgumentType, ArgumentLoc, ValueDef}; +use ir::{InstBuilder, AbiParam, ArgumentLoc, ValueDef}; use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; use isa::{TargetIsa, EncInfo, RecipeConstraints, OperandConstraint, ConstraintKind}; use packed_option::PackedOption; @@ -188,10 +188,10 @@ impl<'a> Context<'a> { ); if self.cur.func.layout.entry_block() == Some(ebb) { - // Arguments to the entry block have ABI constraints. - self.color_entry_args(tracker.live()) + // Parameters on the entry block have ABI constraints. + self.color_entry_params(tracker.live()) } else { - // The live-ins and arguments to a non-entry EBB have already been assigned a register. + // The live-ins and parameters of a non-entry EBB have already been assigned a register. // Reconstruct the allocatable set. self.livein_regs(tracker.live()) } @@ -230,19 +230,19 @@ impl<'a> Context<'a> { regs } - /// Color the arguments to the entry block. + /// Color the parameters on the entry block. /// - /// These are function arguments that should already have assigned register units in the + /// These are function parameters that should already have assigned register units in the /// function signature. /// /// Return the set of remaining allocatable registers after filtering out the dead arguments. - fn color_entry_args(&mut self, args: &[LiveValue]) -> AvailableRegs { + fn color_entry_params(&mut self, args: &[LiveValue]) -> AvailableRegs { let sig = &self.cur.func.signature; - assert_eq!(sig.argument_types.len(), args.len()); + assert_eq!(sig.params.len(), args.len()); let mut regs = AvailableRegs::new(&self.usable_regs); - for (lv, abi) in args.iter().zip(&sig.argument_types) { + for (lv, abi) in args.iter().zip(&sig.params) { match lv.affinity { Affinity::Reg(rci) => { let rc = self.reginfo.rc(rci); @@ -305,7 +305,7 @@ impl<'a> Context<'a> { program_input_abi( &mut self.solver, inst, - &self.cur.func.dfg.signatures[sig].argument_types, + &self.cur.func.dfg.signatures[sig].params, &self.cur.func, &self.liveness, &self.reginfo, @@ -315,7 +315,7 @@ impl<'a> Context<'a> { program_input_abi( &mut self.solver, inst, - &self.cur.func.signature.return_types, + &self.cur.func.signature.returns, &self.cur.func, &self.liveness, &self.reginfo, @@ -687,12 +687,9 @@ impl<'a> Context<'a> { // It's technically possible for a call instruction to have fixed results before the // variable list of results, but we have no known instances of that. // Just assume all results are variable return values. - assert_eq!( - defs.len(), - self.cur.func.dfg.signatures[sig].return_types.len() - ); + assert_eq!(defs.len(), self.cur.func.dfg.signatures[sig].returns.len()); for (i, lv) in defs.iter().enumerate() { - let abi = self.cur.func.dfg.signatures[sig].return_types[i]; + let abi = self.cur.func.dfg.signatures[sig].returns[i]; if let ArgumentLoc::Reg(reg) = abi.location { if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); @@ -1015,7 +1012,7 @@ impl<'a> Context<'a> { fn program_input_abi( solver: &mut Solver, inst: Inst, - abi_types: &[ArgumentType], + abi_types: &[AbiParam], func: &Function, liveness: &Liveness, reginfo: &RegInfo, diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index a64aa9e017..0a449bd85c 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -216,7 +216,7 @@ fn get_or_create<'a>( .or_else(|| { // If this is a call, get the return value affinity. func.dfg.call_signature(inst).map(|sig| { - Affinity::abi(&func.dfg.signatures[sig].return_types[rnum], isa) + Affinity::abi(&func.dfg.signatures[sig].returns[rnum], isa) }) }) .unwrap_or_default(); @@ -226,7 +226,7 @@ fn get_or_create<'a>( if func.layout.entry_block() == Some(ebb) { // The affinity for entry block parameters can be inferred from the function // signature. - affinity = Affinity::abi(&func.signature.argument_types[num], isa); + affinity = Affinity::abi(&func.signature.params[num], isa); } else { // Don't apply any affinity to normal EBB parameters. // They could be in a register or on the stack. diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index de5683a444..97b849f3a9 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -13,7 +13,7 @@ use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; use entity::{SparseMap, SparseMapValue}; use ir::{Ebb, Inst, Value, Function}; -use ir::{InstBuilder, ArgumentType, ArgumentLoc}; +use ir::{InstBuilder, AbiParam, ArgumentLoc}; use isa::RegClass; use isa::{TargetIsa, Encoding, EncInfo, RecipeConstraints, ConstraintKind}; use regalloc::affinity::Affinity; @@ -137,20 +137,20 @@ impl<'a> Context<'a> { if self.cur.func.layout.entry_block() == Some(ebb) { assert_eq!(liveins.len(), 0); - self.visit_entry_args(ebb, args); + self.visit_entry_params(ebb, args); } else { self.visit_ebb_params(ebb, args); } } - /// Visit the arguments to the entry block. + /// Visit the parameters on the entry block. /// These values have ABI constraints from the function signature. - fn visit_entry_args(&mut self, ebb: Ebb, args: &[LiveValue]) { - assert_eq!(self.cur.func.signature.argument_types.len(), args.len()); + fn visit_entry_params(&mut self, ebb: Ebb, args: &[LiveValue]) { + assert_eq!(self.cur.func.signature.params.len(), args.len()); self.cur.goto_first_inst(ebb); for (arg_idx, arg) in args.iter().enumerate() { - let abi = self.cur.func.signature.argument_types[arg_idx]; + let abi = self.cur.func.signature.params[arg_idx]; match abi.location { ArgumentLoc::Reg(_) => { if arg.affinity.is_stack() { @@ -266,7 +266,7 @@ impl<'a> Context<'a> { "Extra results on non-call instruction", ); for (i, lv) in retvals.iter().enumerate() { - let abi = self.cur.func.dfg.signatures[sig].return_types[i]; + let abi = self.cur.func.dfg.signatures[sig].returns[i]; debug_assert!(abi.location.is_reg()); if lv.affinity.is_stack() { let reg = self.cur.func.dfg.replace_result(lv.value, abi.value_type); @@ -308,7 +308,7 @@ impl<'a> Context<'a> { if let Some(sig) = self.cur.func.dfg.call_signature(inst) { handle_abi_args( self.candidates, - &self.cur.func.dfg.signatures[sig].argument_types, + &self.cur.func.dfg.signatures[sig].params, var_args, self.cur.isa, self.liveness, @@ -316,7 +316,7 @@ impl<'a> Context<'a> { } else if self.cur.func.dfg[inst].opcode().is_return() { handle_abi_args( self.candidates, - &self.cur.func.signature.return_types, + &self.cur.func.signature.returns, var_args, self.cur.isa, self.liveness, @@ -348,7 +348,7 @@ impl<'a> Context<'a> { /// return values and call arguments. fn handle_abi_args( candidates: &mut Vec, - abi_types: &[ArgumentType], + abi_types: &[AbiParam], var_args: &[Value], isa: &TargetIsa, liveness: &Liveness, diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index a4fb4ededd..edda6f6f59 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -324,7 +324,7 @@ impl<'a> Context<'a> { let args = self.cur.func.dfg.inst_variable_args(inst); for (idx, (abi, &arg)) in self.cur.func.dfg.signatures[sig] - .argument_types + .params .iter() .zip(args) .enumerate() diff --git a/lib/cretonne/src/verifier/locations.rs b/lib/cretonne/src/verifier/locations.rs index 5f84f0403d..bb80507baa 100644 --- a/lib/cretonne/src/verifier/locations.rs +++ b/lib/cretonne/src/verifier/locations.rs @@ -129,7 +129,7 @@ impl<'a> LocationVerifier<'a> { let varargs = self.func.dfg.inst_variable_args(inst); let results = self.func.dfg.inst_results(inst); - for (abi, &value) in sig.argument_types.iter().zip(varargs) { + for (abi, &value) in sig.params.iter().zip(varargs) { self.check_abi_location( inst, value, @@ -139,7 +139,7 @@ impl<'a> LocationVerifier<'a> { )?; } - for (abi, &value) in sig.return_types.iter().zip(results) { + for (abi, &value) in sig.returns.iter().zip(results) { self.check_abi_location( inst, value, @@ -157,7 +157,7 @@ impl<'a> LocationVerifier<'a> { let sig = &self.func.signature; let varargs = self.func.dfg.inst_variable_args(inst); - for (abi, &value) in sig.return_types.iter().zip(varargs) { + for (abi, &value) in sig.returns.iter().zip(varargs) { self.check_abi_location( inst, value, @@ -175,7 +175,7 @@ impl<'a> LocationVerifier<'a> { &self, inst: ir::Inst, value: ir::Value, - abi: &ir::ArgumentType, + abi: &ir::AbiParam, loc: ir::ValueLoc, want_kind: ir::StackSlotKind, ) -> Result { diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 9bfa531a9e..342e501b0a 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -241,7 +241,7 @@ impl<'a> Verifier<'a> { let fixed_results = inst_data.opcode().constraints().fixed_results(); // var_results is 0 if we aren't a call instruction let var_results = dfg.call_signature(inst) - .map(|sig| dfg.signatures[sig].return_types.len()) + .map(|sig| dfg.signatures[sig].returns.len()) .unwrap_or(0); let total_results = fixed_results + var_results; @@ -551,9 +551,9 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck_entry_block_arguments(&self) -> Result { + fn typecheck_entry_block_params(&self) -> Result { if let Some(ebb) = self.func.layout.entry_block() { - let expected_types = &self.func.signature.argument_types; + let expected_types = &self.func.signature.params; let ebb_param_count = self.func.dfg.num_ebb_params(ebb); if ebb_param_count != expected_types.len() { @@ -700,18 +700,16 @@ impl<'a> Verifier<'a> { match self.func.dfg[inst].analyze_call(&self.func.dfg.value_lists) { CallInfo::Direct(func_ref, _) => { let sig_ref = self.func.dfg.ext_funcs[func_ref].signature; - let arg_types = self.func.dfg.signatures[sig_ref] - .argument_types - .iter() - .map(|a| a.value_type); + let arg_types = self.func.dfg.signatures[sig_ref].params.iter().map(|a| { + a.value_type + }); self.typecheck_variable_args_iterator(inst, arg_types)?; self.check_outgoing_args(inst, sig_ref)?; } CallInfo::Indirect(sig_ref, _) => { - let arg_types = self.func.dfg.signatures[sig_ref] - .argument_types - .iter() - .map(|a| a.value_type); + let arg_types = self.func.dfg.signatures[sig_ref].params.iter().map(|a| { + a.value_type + }); self.typecheck_variable_args_iterator(inst, arg_types)?; self.check_outgoing_args(inst, sig_ref)?; } @@ -772,7 +770,7 @@ impl<'a> Verifier<'a> { } let args = self.func.dfg.inst_variable_args(inst); - let expected_args = &sig.argument_types[..]; + let expected_args = &sig.params[..]; for (&arg, &abi) in args.iter().zip(expected_args) { // Value types have already been checked by `typecheck_variable_args_iterator()`. @@ -828,7 +826,7 @@ impl<'a> Verifier<'a> { fn typecheck_return(&self, inst: Inst) -> Result { if self.func.dfg[inst].opcode().is_return() { let args = self.func.dfg.inst_variable_args(inst); - let expected_types = &self.func.signature.return_types; + let expected_types = &self.func.signature.returns; if args.len() != expected_types.len() { return err!(inst, "arguments of return must match function signature"); } @@ -1081,7 +1079,7 @@ impl<'a> Verifier<'a> { pub fn run(&self) -> Result { self.verify_global_vars()?; - self.typecheck_entry_block_arguments()?; + self.typecheck_entry_block_params()?; for ebb in self.func.layout.ebbs() { for inst in self.func.layout.ebb_insts(ebb) { self.ebb_integrity(ebb, inst)?; diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 54f9d3620e..78f3725bce 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -529,14 +529,14 @@ where fn check_return_args(&self, args: &[Value]) { debug_assert_eq!( args.len(), - self.func.signature.return_types.len(), + self.func.signature.returns.len(), "the number of returned values doesn't match the function signature " ); for (i, arg) in args.iter().enumerate() { let valty = self.func.dfg.value_type(*arg); debug_assert_eq!( valty, - self.func.signature.return_types[i].value_type, + self.func.signature.returns[i].value_type, "the types of the values returned don't match the \ function signature" ); @@ -545,7 +545,7 @@ where fn fill_function_args_values(&mut self, ebb: Ebb) { debug_assert!(self.pristine); - for argtyp in &self.func.signature.argument_types { + for argtyp in &self.func.signature.params { self.builder.function_args_values.push( self.func.dfg.append_ebb_param(ebb, argtyp.value_type), ); @@ -621,7 +621,7 @@ where mod tests { use cretonne::entity::EntityRef; - use cretonne::ir::{FunctionName, Function, CallConv, Signature, ArgumentType, InstBuilder}; + use cretonne::ir::{FunctionName, Function, CallConv, Signature, AbiParam, InstBuilder}; use cretonne::ir::types::*; use frontend::{ILBuilder, FunctionBuilder}; use cretonne::verifier::verify_function; @@ -651,8 +651,8 @@ mod tests { #[test] fn sample_function() { let mut sig = Signature::new(CallConv::Native); - sig.return_types.push(ArgumentType::new(I32)); - sig.argument_types.push(ArgumentType::new(I32)); + sig.returns.push(AbiParam::new(I32)); + sig.params.push(AbiParam::new(I32)); let mut il_builder = ILBuilder::::new(); let mut func = Function::with_name_signature(FunctionName::new("sample_function"), sig); diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index cdf2361489..cf0044d1c0 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -36,7 +36,7 @@ //! extern crate cton_frontend; //! //! use cretonne::entity::EntityRef; -//! use cretonne::ir::{FunctionName, CallConv, Function, Signature, ArgumentType, InstBuilder}; +//! use cretonne::ir::{FunctionName, CallConv, Function, Signature, AbiParam, InstBuilder}; //! use cretonne::ir::types::*; //! use cretonne::settings; //! use cton_frontend::{ILBuilder, FunctionBuilder}; @@ -64,8 +64,8 @@ //! //! fn main() { //! let mut sig = Signature::new(CallConv::Native); -//! sig.return_types.push(ArgumentType::new(I32)); -//! sig.argument_types.push(ArgumentType::new(I32)); +//! sig.returns.push(AbiParam::new(I32)); +//! sig.params.push(AbiParam::new(I32)); //! let mut il_builder = ILBuilder::::new(); //! let mut func = Function::with_name_signature(FunctionName::new("sample_function"), sig); //! { diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index f4e3507754..cc2d65996b 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -10,9 +10,9 @@ use std::str::FromStr; use std::{u16, u32}; use std::mem; use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, CallConv, StackSlotData, - JumpTable, JumpTableData, Signature, ArgumentType, ArgumentExtension, - ExtFuncData, SigRef, FuncRef, StackSlot, ValueLoc, ArgumentLoc, MemFlags, - GlobalVar, GlobalVarData, Heap, HeapData, HeapStyle, HeapBase}; + JumpTable, JumpTableData, Signature, AbiParam, ArgumentExtension, ExtFuncData, + SigRef, FuncRef, StackSlot, ValueLoc, ArgumentLoc, MemFlags, GlobalVar, + GlobalVarData, Heap, HeapData, HeapStyle, HeapBase}; use cretonne::ir; use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Uimm32, Offset32, Ieee32, Ieee64}; @@ -918,7 +918,7 @@ impl<'a> Parser<'a> { // Parse a function signature. // - // signature ::= * "(" [arglist] ")" ["->" retlist] [callconv] + // signature ::= * "(" [paramlist] ")" ["->" retlist] [callconv] // fn parse_signature(&mut self, unique_isa: Option<&TargetIsa>) -> Result { // Calling convention defaults to `native`, but can be changed. @@ -928,16 +928,16 @@ impl<'a> Parser<'a> { Token::LPar, "expected function signature: ( args... )", )?; - // signature ::= "(" * [arglist] ")" ["->" retlist] [callconv] + // signature ::= "(" * [abi-param-list] ")" ["->" retlist] [callconv] if self.token() != Some(Token::RPar) { - sig.argument_types = self.parse_argument_list(unique_isa)?; + sig.params = self.parse_abi_param_list(unique_isa)?; } self.match_token( Token::RPar, "expected ')' after function arguments", )?; if self.optional(Token::Arrow) { - sig.return_types = self.parse_argument_list(unique_isa)?; + sig.returns = self.parse_abi_param_list(unique_isa)?; } // The calling convention is optional. @@ -951,38 +951,38 @@ impl<'a> Parser<'a> { } } - if sig.argument_types.iter().all(|a| a.location.is_assigned()) { + if sig.params.iter().all(|a| a.location.is_assigned()) { sig.compute_argument_bytes(); } Ok(sig) } - // Parse list of function argument / return value types. + // Parse list of function parameter / return value types. // - // arglist ::= * arg { "," arg } + // paramlist ::= * param { "," param } // - fn parse_argument_list(&mut self, unique_isa: Option<&TargetIsa>) -> Result> { + fn parse_abi_param_list(&mut self, unique_isa: Option<&TargetIsa>) -> Result> { let mut list = Vec::new(); - // arglist ::= * arg { "," arg } - list.push(self.parse_argument_type(unique_isa)?); + // abi-param-list ::= * abi-param { "," abi-param } + list.push(self.parse_abi_param(unique_isa)?); - // arglist ::= arg * { "," arg } + // abi-param-list ::= abi-param * { "," abi-param } while self.optional(Token::Comma) { - // arglist ::= arg { "," * arg } - list.push(self.parse_argument_type(unique_isa)?); + // abi-param-list ::= abi-param { "," * abi-param } + list.push(self.parse_abi_param(unique_isa)?); } Ok(list) } // Parse a single argument type with flags. - fn parse_argument_type(&mut self, unique_isa: Option<&TargetIsa>) -> Result { - // arg ::= * type { flag } [ argumentloc ] - let mut arg = ArgumentType::new(self.match_type("expected argument type")?); + fn parse_abi_param(&mut self, unique_isa: Option<&TargetIsa>) -> Result { + // abi-param ::= * type { flag } [ argumentloc ] + let mut arg = AbiParam::new(self.match_type("expected parameter type")?); - // arg ::= type * { flag } [ argumentloc ] + // abi-param ::= type * { flag } [ argumentloc ] while let Some(Token::Identifier(s)) = self.token() { match s { "uext" => arg.extension = ArgumentExtension::Uext, @@ -998,7 +998,7 @@ impl<'a> Parser<'a> { self.consume(); } - // arg ::= type { flag } * [ argumentloc ] + // abi-param ::= type { flag } * [ argumentloc ] arg.location = self.parse_argument_location(unique_isa)?; Ok(arg) @@ -1374,7 +1374,7 @@ impl<'a> Parser<'a> { self.parse_ebb_params(ctx, ebb)?; self.match_token( Token::Colon, - "expected ':' after EBB arguments", + "expected ':' after EBB parameters", )?; } @@ -1437,7 +1437,7 @@ impl<'a> Parser<'a> { // ebb-params ::= * "(" ebb-param { "," ebb-param } ")" self.match_token( Token::LPar, - "expected '(' before EBB arguments", + "expected '(' before EBB parameters", )?; // ebb-params ::= "(" * ebb-param { "," ebb-param } ")" @@ -1452,7 +1452,7 @@ impl<'a> Parser<'a> { // ebb-params ::= "(" ebb-param { "," ebb-param } * ")" self.match_token( Token::RPar, - "expected ')' after EBB arguments", + "expected ')' after EBB parameters", )?; Ok(()) @@ -2337,13 +2337,13 @@ mod tests { #[test] fn argument_type() { let mut p = Parser::new("i32 sext"); - let arg = p.parse_argument_type(None).unwrap(); + let arg = p.parse_abi_param(None).unwrap(); assert_eq!(arg.value_type, types::I32); assert_eq!(arg.extension, ArgumentExtension::Sext); assert_eq!(arg.purpose, ArgumentPurpose::Normal); - let Error { location, message } = p.parse_argument_type(None).unwrap_err(); + let Error { location, message } = p.parse_abi_param(None).unwrap_err(); assert_eq!(location.line_number, 1); - assert_eq!(message, "expected argument type"); + assert_eq!(message, "expected parameter type"); } #[test] @@ -2374,8 +2374,8 @@ mod tests { #[test] fn signature() { let sig = Parser::new("()native").parse_signature(None).unwrap(); - assert_eq!(sig.argument_types.len(), 0); - assert_eq!(sig.return_types.len(), 0); + assert_eq!(sig.params.len(), 0); + assert_eq!(sig.returns.len(), 0); assert_eq!(sig.call_conv, CallConv::Native); let sig2 = Parser::new("(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 spiderwasm") @@ -2406,7 +2406,7 @@ mod tests { .parse_signature(None) .unwrap_err() .to_string(), - "1: expected argument type" + "1: expected parameter type" ); assert_eq!( Parser::new("i8 -> i8") diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 394da6ab36..ebac333531 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -100,14 +100,14 @@ impl FuncTranslator { } } -/// Declare local variables for the signature arguments that correspond to WebAssembly locals. +/// Declare local variables for the signature parameters that correspond to WebAssembly locals. /// /// Return the number of local variables declared. fn declare_wasm_arguments(builder: &mut FunctionBuilder) -> usize { - let sig_len = builder.func.signature.argument_types.len(); + let sig_len = builder.func.signature.params.len(); let mut next_local = 0; for i in 0..sig_len { - let arg_type = builder.func.signature.argument_types[i]; + let arg_type = builder.func.signature.params[i]; // There may be additional special-purpose arguments following the normal WebAssembly // signature arguments. For example, a `vmctx` pointer. if arg_type.purpose == ir::ArgumentPurpose::Normal { @@ -256,12 +256,8 @@ mod tests { let mut ctx = Context::new(); ctx.func.name = ir::FunctionName::new("small1"); - ctx.func.signature.argument_types.push( - ir::ArgumentType::new(I32), - ); - ctx.func.signature.return_types.push( - ir::ArgumentType::new(I32), - ); + ctx.func.signature.params.push(ir::AbiParam::new(I32)); + ctx.func.signature.returns.push(ir::AbiParam::new(I32)); trans .translate(&BODY, &mut ctx.func, &mut runtime.func_env()) @@ -291,12 +287,8 @@ mod tests { let mut ctx = Context::new(); ctx.func.name = ir::FunctionName::new("small2"); - ctx.func.signature.argument_types.push( - ir::ArgumentType::new(I32), - ); - ctx.func.signature.return_types.push( - ir::ArgumentType::new(I32), - ); + ctx.func.signature.params.push(ir::AbiParam::new(I32)); + ctx.func.signature.returns.push(ir::AbiParam::new(I32)); trans .translate(&BODY, &mut ctx.func, &mut runtime.func_env()) @@ -335,9 +327,7 @@ mod tests { let mut ctx = Context::new(); ctx.func.name = ir::FunctionName::new("infloop"); - ctx.func.signature.return_types.push( - ir::ArgumentType::new(I32), - ); + ctx.func.signature.returns.push(ir::AbiParam::new(I32)); trans .translate(&BODY, &mut ctx.func, &mut runtime.func_env()) diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 21ade154ec..21f5abbdb5 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -9,7 +9,7 @@ //! interpreted on the fly. use translation_utils::{type_to_type, TableIndex, FunctionIndex, GlobalIndex, SignatureIndex, MemoryIndex, Global, GlobalInit, Table, TableElementType, Memory}; -use cretonne::ir::{Signature, ArgumentType, CallConv}; +use cretonne::ir::{Signature, AbiParam, CallConv}; use cretonne; use wasmparser::{Parser, ParserState, FuncType, ImportSectionEntryType, ExternalKind, WasmDecoder, MemoryType, Operator}; @@ -35,17 +35,17 @@ pub fn parse_function_signatures( ref returns, }) => { let mut sig = Signature::new(CallConv::Native); - sig.argument_types.extend(params.iter().map(|ty| { + sig.params.extend(params.iter().map(|ty| { let cret_arg: cretonne::ir::Type = type_to_type(ty).expect( "only numeric types are supported in function signatures", ); - ArgumentType::new(cret_arg) + AbiParam::new(cret_arg) })); - sig.return_types.extend(returns.iter().map(|ty| { + sig.returns.extend(returns.iter().map(|ty| { let cret_arg: cretonne::ir::Type = type_to_type(ty).expect( "only numeric types are supported in function signatures", ); - ArgumentType::new(cret_arg) + AbiParam::new(cret_arg) })); environ.declare_signature(&sig); } diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index 1c9ce6d8cd..4117d3509c 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -158,7 +158,7 @@ impl TranslationState { self.clear(); self.push_block( exit_block, - sig.return_types + sig.returns .iter() .filter(|arg| arg.purpose == ir::ArgumentPurpose::Normal) .count(), @@ -323,10 +323,10 @@ impl TranslationState { } } -/// Count the number of normal arguments in a signature. -/// Exclude special-purpose arguments that represent runtime stuff and not WebAssembly arguments. +/// Count the number of normal parameters in a signature. +/// Exclude special-purpose parameters that represent runtime stuff and not WebAssembly arguments. fn normal_args(sig: &ir::Signature) -> usize { - sig.argument_types + sig.params .iter() .filter(|arg| arg.purpose == ir::ArgumentPurpose::Normal) .count() From e5c0e06fa84370263bafd5f881a8111dc71fcda3 Mon Sep 17 00:00:00 2001 From: Julien Wajsberg Date: Fri, 20 Oct 2017 10:31:05 +0200 Subject: [PATCH 1280/3084] Fix typo in README.rst --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index cdce22ff92..edd726a033 100644 --- a/README.rst +++ b/README.rst @@ -39,7 +39,7 @@ Predictable performance that sometimes work, sometimes fail. For more information, see -`the documenation `_. +`the documentation `_. Building Cretonne ----------------- From 2569ef4c42c9bcc60d36b4bc7812a7944c616190 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Oct 2017 09:38:52 -0700 Subject: [PATCH 1281/3084] Clarify undefined behavior and notrap. (#170) * Clarify undefined behavior and notrap. Remove the "No undefined behavior" paragraph from the README. The other paragraphs, specifically "Portable semantics" and "Fast sandbox verification", describe Cretonne's goals in this area. Define *addressable* and *accessible* memory, so that trapping remains a fully defined part of the semantics, and we have a clear boundary around undefined behavior, and use these terms to describe related constructs. --- README.rst | 7 +-- cranelift/docs/langref.rst | 101 ++++++++++++++++++++++++++----------- 2 files changed, 73 insertions(+), 35 deletions(-) diff --git a/README.rst b/README.rst index edd726a033..1afcd5319d 100644 --- a/README.rst +++ b/README.rst @@ -18,9 +18,6 @@ target-independent intermediate language into executable machine code. Cretonne is designed to be a code generator for WebAssembly with these design goals: -No undefined behavior - Cretonne does not have a `nasal demons clause `_, and it won't generate code - with unexpected behavior if invariants are broken. Portable semantics As far as possible, Cretonne's input language has well-defined semantics that are the same on all target architectures. The semantics are usually @@ -28,8 +25,8 @@ Portable semantics Fast sandbox verification Cretonne's input language has a safe subset for sandboxed code. No advanced analysis is required to verify memory safety as long as only the safe - instructions are used. The safe instruction set is expressive enough to - implement WebAssembly. + subset is used. The safe subset is expressive enough to implement + WebAssembly. Scalable performance Cretonne can be configured to generate code as quickly as possible, or it can generate very good code at the cost of slower compile times. diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index aa0913e745..667bfcd3b3 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -429,41 +429,45 @@ Indirect function calls use a signature declared in the preamble. .. autoinst:: call_indirect .. autoinst:: func_addr +.. _memory: Memory ====== Cretonne provides fully general :inst:`load` and :inst:`store` instructions for accessing memory, as well as :ref:`extending loads and truncating stores -`. There are also more restricted operations for accessing -specific types of memory objects. +`. + +If the memory at the given addresss is not :term:`addressable`, the behavior of +these instructions is undefined. If it is addressable but not +:term:`accessible`, they :term:`trap`. .. autoinst:: load .. autoinst:: store +There are also more restricted operations for accessing specific types of memory +objects. + Memory operation flags ---------------------- Loads and stores can have flags that loosen their semantics in order to enable optimizations. -======= ========================================= +======= =========================================== Flag Description -======= ========================================= -notrap Trapping is not required. +======= =========================================== +notrap Memory is assumed to be :term:`accessible`. aligned Trapping allowed for misaligned accesses. -======= ========================================= +======= =========================================== -Trapping is part of the semantics of memory accesses. The operating system may -have configured parts of the address space to cause a trap when read and/or -written, and Cretonne's memory instructions respect that. When the ``notrap`` -flat is set, the trapping behavior is optional. This allows the optimizer to -delete loads whose results are not used. +When the ``accessible`` flag is set, the behavior is undefined if the memory +is not :term:`accessible`. Loads and stores are *misaligned* if the resultant address is not a multiple of the expected alignment. By default, misaligned loads and stores are allowed, but when the ``aligned`` flag is set, a misaligned memory access is allowed to -trap. +:term:`trap`. Local variables --------------- @@ -471,7 +475,8 @@ Local variables One set of restricted memory operations access the current function's stack frame. The stack frame is divided into fixed-size stack slots that are allocated in the :term:`function preamble`. Stack slots are not typed, they -simply represent a contiguous sequence of bytes in the stack frame. +simply represent a contiguous sequence of :term:`accessible` bytes in the stack +frame. .. inst:: SS = local Bytes, Flags... @@ -492,8 +497,8 @@ about because stack slots and offsets are fixed at compile time. For example, the alignment of these stack memory accesses can be inferred from the offsets and stack slot alignments. -It can be necessary to escape from the safety of the restricted instructions by -taking the address of a stack slot. +It can be necessary to escape from the :term:`safety` of the restricted +instructions by taking the address of a stack slot. .. autoinst:: stack_addr @@ -508,12 +513,12 @@ instructions before instruction selection:: Global variables ---------------- -A *global variable* is an object in memory whose address is not known at -compile time. The address is computed at runtime by :inst:`global_addr`, -possibly using information provided by the linker via relocations. There are -multiple kinds of global variables using different methods for determining -their address. Cretonne does not track the type or even the size of global -variables, they are just pointers to non-stack memory. +A *global variable* is an :term:`accessible` object in memory whose address is +not known at compile time. The address is computed at runtime by +:inst:`global_addr`, possibly using information provided by the linker via +relocations. There are multiple kinds of global variables using different +methods for determining their address. Cretonne does not track the type or even +the size of global variables, they are just pointers to non-stack memory. When Cretonne is generating code for a virtual machine environment, globals can be used to access data structures in the VM's runtime. This requires functions @@ -570,9 +575,9 @@ in, and all accesses are bounds checked. Cretonne models this through the concept of *heaps*. A heap is declared in the function preamble and can be accessed with the -:inst:`heap_addr` instruction that traps on out-of-bounds accesses or returns a -pointer that is guaranteed to trap. Heap addresses can be smaller than the -native pointer size, for example unsigned :type:`i32` offsets on a 64-bit +:inst:`heap_addr` instruction that :term:`traps` on out-of-bounds accesses or +returns a pointer that is guaranteed to trap. Heap addresses can be smaller than +the native pointer size, for example unsigned :type:`i32` offsets on a 64-bit architecture. .. digraph:: static @@ -588,18 +593,21 @@ architecture. A heap appears as three consecutive ranges of address space: -1. The *mapped pages* are the usable memory range in the heap. Loads and stores - to this range won't trap. A heap may have a minimum guaranteed size which - means that some mapped pages are always present. +1. The *mapped pages* are the :term:`accessible` memory range in the heap. A + heap may have a minimum guaranteed size which means that some mapped pages + are always present. 2. The *unmapped pages* is a possibly empty range of address space that may be - mapped in the future when the heap is grown. + mapped in the future when the heap is grown. They are :term:`addressable` but + not :term:`accessible`. 3. The *guard pages* is a range of address space that is guaranteed to cause a trap when accessed. It is used to optimize bounds checking for heap accesses - with a shared base pointer. + with a shared base pointer. They are :term:`addressable` but + not :term:`accessible`. The *heap bound* is the total size of the mapped and unmapped pages. This is the bound that :inst:`heap_addr` checks against. Memory accesses inside the -heap bounds can trap if they hit an unmapped page. +heap bounds can trap if they hit an unmapped page (which is not +:term:`accessible`). .. autoinst:: heap_addr @@ -920,6 +928,9 @@ In addition to the normal :inst:`load` and :inst:`store` instructions, Cretonne provides extending loads and truncation stores for 8, 16, and 32-bit memory accesses. +These instructions succeed, trap, or have undefined behavior, under the same +conditions as :ref:`normal loads and stores `. + .. autoinst:: uload8 .. autoinst:: sload8 .. autoinst:: istore8 @@ -1009,6 +1020,19 @@ Glossary .. glossary:: + addressable + Memory in which loads and stores have defined behavior. They either + succeed or :term:`trap`, depending on whether the memory is + :term:`accessible`. + + accessible + :term:`Addressable` memory in which loads and stores always succeed + without :term:`trapping`, except where specified otherwise (eg. with the + `aligned` flag). Heaps, globals, and the stack may contain accessible, + merely addressable, and outright unaddressable regions. There may also + be additional regions of addressable and/or accessible memory not + explicitly declared. + basic block A maximal sequence of instructions that can only be entered from the top, and that contains no branch or terminator instructions except for @@ -1089,6 +1113,15 @@ Glossary intermediate representation. Cretonne's IR can be converted to text losslessly. + safe + safety + Execution of exclusively defined behavior. Safe programs cannot + read, write, or execute memory outside of heaps, globals, stack + regions, and functions that have been explicitly provided to them. In + some instances, defined behavior can be nondeterministic, where the + specific behavior may vary among a bounded set of possibilities. + Execution of undefined behavior is unsafe. + stack slot A fixed size memory allocation in the current function's activation frame. Also called a local variable. @@ -1101,3 +1134,11 @@ Glossary The basic terminator instructions are :inst:`br`, :inst:`return`, and :inst:`trap`. Conditional branches and instructions that trap conditionally are not terminator instructions. + + trap + traps + trapping + Terminates execution of the current thread. The specific behavior after + a trap depends on the underlying OS. For example, a common behavior is + delivery of a signal, with the specific signal depending on the event + that triggered it. From 8e9ac361bd234856d75438afb817abe9d4d06a4e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 20 Oct 2017 09:19:28 -0700 Subject: [PATCH 1282/3084] Remove the CFG entry_block member. It was not used. --- lib/cretonne/src/flowgraph.rs | 3 --- 1 file changed, 3 deletions(-) diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index ce9744ac38..67ff150196 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -45,7 +45,6 @@ pub struct CFGNode { /// extended basic blocks. #[derive(Debug)] pub struct ControlFlowGraph { - entry_block: Option, data: EntityMap, valid: bool, } @@ -54,7 +53,6 @@ impl ControlFlowGraph { /// Allocate a new blank control flow graph. pub fn new() -> ControlFlowGraph { ControlFlowGraph { - entry_block: None, data: EntityMap::new(), valid: false, } @@ -71,7 +69,6 @@ impl ControlFlowGraph { /// /// This will clear and overwrite any information already stored in this data structure. pub fn compute(&mut self, func: &Function) { - self.entry_block = func.layout.entry_block(); self.data.clear(); self.data.resize(func.dfg.num_ebbs()); From d3077acf1c03dfc38279655278edf32ac7107315 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Oct 2017 13:36:59 -0700 Subject: [PATCH 1283/3084] TranslationState can assert that its stacks are empty between functions. --- lib/wasm/src/state.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index 4117d3509c..91737a370c 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -140,10 +140,10 @@ impl TranslationState { } fn clear(&mut self) { - self.stack.clear(); - self.control_stack.clear(); - self.phantom_unreachable_stack_depth = 0; - self.real_unreachable_stack_depth = 0; + debug_assert!(self.stack.is_empty()); + debug_assert!(self.control_stack.is_empty()); + debug_assert_eq!(self.phantom_unreachable_stack_depth, 0); + debug_assert_eq!(self.real_unreachable_stack_depth, 0); self.globals.clear(); self.heaps.clear(); self.signatures.clear(); From f5e4058eb8c61a3e8288d519f891bf555c9789bc Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Oct 2017 13:49:36 -0700 Subject: [PATCH 1284/3084] Update a stale comment. --- cranelift/src/wasm.rs | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 8833dc81b9..679fc79115 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -1,8 +1,6 @@ //! CLI tool to use the functions provided by the [cretonne-wasm](../cton_wasm/index.html) crate. //! -//! Reads Wasm binary files (one Wasm module per file), translates the functions' code to Cretonne -//! IL. Can also executes the `start` function of the module by laying out the memories, globals -//! and tables, then emitting the translated code with hardcoded addresses to memory. +//! Reads Wasm binary files, translates the functions' code to Cretonne IL. use cton_wasm::{translate_module, DummyEnvironment, ModuleEnvironment}; use std::path::PathBuf; From 9fdf44afd4043dc443809e4db111b4a199a857d0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Oct 2017 15:03:43 -0700 Subject: [PATCH 1285/3084] Clean up a stale comment. --- lib/wasm/src/environ/dummy.rs | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index b2d4e66151..5900b32ea2 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -306,7 +306,6 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment { self.info.start_func = Some(func_index); } - /// Provides the contents of a function body. fn define_function_body(&mut self, body_bytes: &'data [u8]) -> Result<(), String> { let function_index = self.get_num_func_imports() + self.info.function_bodies.len(); let name = get_func_name(function_index); From f4b25ff058f71eff2e5b15a4741dc98137811f01 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Oct 2017 15:13:59 -0700 Subject: [PATCH 1286/3084] Have FunctionBuilder clear the ILBuilder's state in its drop(). --- lib/frontend/src/frontend.rs | 23 ++++++++++++++++++----- lib/frontend/src/ssa.rs | 7 +++++++ 2 files changed, 25 insertions(+), 5 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 78f3725bce..708de69881 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -11,6 +11,10 @@ use ssa::{SSABuilder, SideEffects, Block}; use cretonne::entity::{EntityRef, EntityMap, EntitySet}; /// Permanent structure used for translating into Cretonne IL. +/// +/// In order to reduce memory reallocations whem compiling multiple functions, +/// `ILBuilder` holds various data structures which are cleared between +/// functions, rather than dropped, preserving the underlying allocations. pub struct ILBuilder where Variable: EntityRef + Default, @@ -55,8 +59,8 @@ impl ILBuilder where Variable: EntityRef + Default, { - /// Creates a ILBuilder structure. The structure is automatically cleared each time it is - /// passed to a [`FunctionBuilder`](struct.FunctionBuilder.html) for creation. + /// Creates a ILBuilder structure. The structure is automatically cleared after each + /// [`FunctionBuilder`](struct.FunctionBuilder.html) completes translating a function. pub fn new() -> Self { Self { ssa: SSABuilder::new(), @@ -72,6 +76,11 @@ where self.types.clear(); self.function_args_values.clear(); } + + fn is_empty(&self) -> bool { + self.ssa.is_empty() && self.ebbs.is_empty() && self.types.is_empty() && + self.function_args_values.is_empty() + } } /// Implementation of the [`InstBuilder`](../cretonne/ir/builder/trait.InstBuilder.html) that has @@ -216,7 +225,7 @@ where func: &'a mut Function, builder: &'a mut ILBuilder, ) -> FunctionBuilder<'a, Variable> { - builder.clear(); + debug_assert!(builder.is_empty()); FunctionBuilder { func: func, srcloc: Default::default(), @@ -491,15 +500,19 @@ where Variable: EntityRef + Default, { /// When a `FunctionBuilder` goes out of scope, it means that the function is fully built. - /// We then proceed to check if all the `Ebb`s are filled and sealed fn drop(&mut self) { + // Check that all the `Ebb`s are filled and sealed. debug_assert!( self.builder.ebbs.keys().all(|ebb| { self.builder.ebbs[ebb].pristine || (self.builder.ssa.is_sealed(ebb) && self.builder.ebbs[ebb].filled) }), "all blocks should be filled and sealed before dropping a FunctionBuilder" - ) + ); + + // Clear the state (but preserve the allocated buffers) in preparation + // for translation another function. + self.builder.clear(); } } diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 600143fecc..d735caabb7 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -168,6 +168,13 @@ where debug_assert!(self.results.is_empty()); debug_assert!(self.side_effects.is_empty()); } + + /// Tests whether an `SSABuilder` is in a cleared state. + pub fn is_empty(&self) -> bool { + self.variables.is_empty() && self.blocks.is_empty() && self.ebb_headers.is_empty() && + self.calls.is_empty() && + self.results.is_empty() && self.side_effects.is_empty() + } } // Small enum used for clarity in some functions. From 29a6ee77f3db3492c83a1da3262b7ddd7914cd8e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Oct 2017 11:30:22 -0700 Subject: [PATCH 1287/3084] Remove the definition of "safe". `stack_addr` and unrestricted loads and stores *can* be used with entirely defined behavior. The sense in which they're not "safe" is only that it's possible to misuse them. This subtlety wasn't captured in the definition of "safe" here, so for now, just remove the definition so that it doesn't cause confusion. --- cranelift/docs/langref.rst | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 667bfcd3b3..fb9f35fd94 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -497,8 +497,8 @@ about because stack slots and offsets are fixed at compile time. For example, the alignment of these stack memory accesses can be inferred from the offsets and stack slot alignments. -It can be necessary to escape from the :term:`safety` of the restricted -instructions by taking the address of a stack slot. +It's also possible to obtain the address of a stack slot, which can be used +in :ref:`unrestricted loads and stores `. .. autoinst:: stack_addr @@ -1113,15 +1113,6 @@ Glossary intermediate representation. Cretonne's IR can be converted to text losslessly. - safe - safety - Execution of exclusively defined behavior. Safe programs cannot - read, write, or execute memory outside of heaps, globals, stack - regions, and functions that have been explicitly provided to them. In - some instances, defined behavior can be nondeterministic, where the - specific behavior may vary among a bounded set of possibilities. - Execution of undefined behavior is unsafe. - stack slot A fixed size memory allocation in the current function's activation frame. Also called a local variable. From 250c515ba9a2d0e76fa4127b2c7dd70f0cb0ddf5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 20 Oct 2017 15:14:07 -0700 Subject: [PATCH 1288/3084] Update the register allocator document. - We have a coalescing pass which puts the code in CSSA form. - We do not have an EBB argument fixup pass. This isn't needed with CSSA. --- cranelift/docs/regalloc.rst | 129 +++++++++++++++++++++++++++++++----- 1 file changed, 111 insertions(+), 18 deletions(-) diff --git a/cranelift/docs/regalloc.rst b/cranelift/docs/regalloc.rst index 8c73df41ec..8f8781a520 100644 --- a/cranelift/docs/regalloc.rst +++ b/cranelift/docs/regalloc.rst @@ -3,7 +3,7 @@ Register Allocation in Cretonne ******************************* .. default-domain:: cton -.. highlight:: rust +.. highlight:: cton Cretonne uses a *decoupled, SSA-based* register allocator. Decoupled means that register allocation is split into two primary phases: *spilling* and @@ -29,6 +29,11 @@ The phases of the SSA-based register allocator are: Liveness analysis For each SSA value, determine exactly where it is live. +Coalescing + Form *virtual registers* which are sets of SSA values that should be + assigned to the same location. Split live ranges such that values that + belong to the same virtual register don't have interfering live ranges. + Spilling The process of deciding which SSA values go in a stack slot and which values go in a register. The spilling phase can also split live ranges by @@ -38,20 +43,20 @@ Spilling After spilling, the number of live register values never exceeds the number of available registers. +Reload + Insert :inst:`spill` and :inst:`fill` instructions as necessary such that + instructions that expect their operands in registers won't see values that + live on the stack and vice versa. + + Reuse registers containing values loaded from the stack as much as possible + without exceeding the maximum allowed register pressure. + Coloring The process of assigning specific registers to the live values. It's a property of SSA form that this can be done in a linear scan of the dominator tree without causing any additional spills. -EBB argument fixup - The coloring phase does not guarantee that EBB arguments are placed in the - correct registers and/or stack slots before jumping to the EBB. It will - try its best, but not making this guarantee is essential to the speed of - the coloring phase. (EBB arguments correspond to PHI nodes in traditional - SSA form). - - The argument fixup phase inserts 'shuffle code' before jumps and branches - to place the argument values in their expected locations. + Make sure that specific register operand constraints are satisfied. The contract between the spilling and coloring phases is that the number of values in registers never exceeds the number of available registers. This @@ -119,8 +124,8 @@ Early clobbers Liveness Analysis ================= -Both spilling and coloring need to know exactly where SSA values are live. The -liveness analysis computes this information. +All the register allocator passes need to know exactly where SSA values are +live. The liveness analysis computes this information. The data structure representing the live range of a value uses the linear layout of the function. All instructions and EBB headers are assigned a @@ -128,7 +133,7 @@ layout of the function. All instructions and EBB headers are assigned a following: - The instruction where the value is defined. -- The EBB header where the value is an EBB argument. +- The EBB header where the value is an EBB parameter. - An EBB header where the value is live-in because it was defined in a dominating block. @@ -185,14 +190,102 @@ with a few important differences: A consequence of Cretonne's more compact representation is that two program points can't be compared without the context of a function layout. +Coalescing algorithm +==================== + +Unconstrained SSA form is not well suited to register allocation because of the problems +that can arise around EBB parameters and arguments. Consider this simple example:: + + function %interference(i32, i32) -> i32 { + ebb0(v0: i32, v1: i32): + brz v0, ebb1(v1) + jump ebb1(v0) + + ebb1(v2: i32): + v3 = iadd v1, v2 + return v3 + } + +Here, the value ``v1`` is both passed as an argument to ``ebb1`` *and* it is +live in to the EBB because it is used by the :inst:`iadd` instruction. Since +EBB arguments on the :inst:`brz` instruction need to be in the same register as +the corresponding EBB parameter ``v2``, there is going to be interference +between ``v1`` and ``v2`` in the ``ebb1`` block. + +The interference can be resolved by isolating the SSA values passed as EBB arguments:: + + function %coalesced(i32, i32) -> i32 { + ebb0(v0: i32, v1: i32): + v5 = copy v1 + brz v0, ebb1(v5) + v6 = copy v0 + jump ebb1(v6) + + ebb1(v2: i32): + v3 = iadd.i32 v1, v2 + return v3 + } + +Now the EBB argument is ``v5`` which is *not* isself live into ``ebb1``, +resolving the interference. + +The coalescing pass groups the SSA values into sets called *virtual registers* +and inserts copies such that: + +1. Whenever a value is passed as an EBB argument, the corresponding EBB + parameter value belongs to the same virtual register as the passed argument + value. +2. The live ranges of values belonging to the same virtual register do not + interfere, i.e. they don't overlap anywhere. + +Most virtual registers contains only a single isolated SSA value because most +SSA values are never passed as EBB arguments. The ``VirtRegs`` data structure +doesn't store any information about these singleton virtual registers, it only +tracks larger virtual registers and assumes that any value it doesn't know about +is its own singleton virtual register + +Once the values have been partitioned into interference-free virtual registers, +the code is said to be in `conventional SSA form (CSSA) +`_. A program +in CSSA form can be register allocated correctly by assigning all the values in +a virtual register to the same stack or register location. + +Conventional SSA form and the virtual registers are maintained through all the +register allocator passes. + Spilling algorithm ================== -There is no one way of implementing spilling, and different tradeoffs between -compilation time and code quality are possible. Any spilling algorithm will -need a way of tracking the register pressure so the colorability condition can -be satisfied. +The spilling pass is responsible for lowering the register pressure enough that +the coloring pass is guaranteed to be able to find a coloring solution. It does +this by assigning whole virtual registers to stack slots. + +Besides just counting registers, the spiller also has to look at the +instruction's operand constraints because sometimes the constraints can require +extra registers to solve, raising the register pressure: + +- If a single value is used more than once by an instruction, and the operands + have conflicting constraints, two registers must be used. The most common case is + when a single value is passed as two separate arguments to a function call. +- If an instruction has a *tied operand constraint* where one of the input operands + must use the same register as the output operand, the spiller makes sure that + the tied input value doesn't interfere with the output value by inserting a copy + if needed. + +The spilling heuristic used by Cretonne is very simple. Whenever the spiller +determines that the register pressure is too high at some instruction, it picks +the live SSA value whose definition is farthest away as the spill candidate. +Then it spills all values in the corresponding virtual register to the same +spill slot. It is important that all values in a virtual register get the same +spill slot, otherwise we could need memory-to-memory copies when passing spilled +arguments to a spilled EBB parameter. + +This simple heuristic tends to spill values with long live ranges, and it +depends on the reload pass to do a good job of reusing registers reloaded from +spill slots if the spilled value gets used a lot. The idea is to minimize stack +*write* traffic with the spilling heuristic and to minimize stack *read* traffic +with the reload pass. Coloring algorithm ================== @@ -223,7 +316,7 @@ instruction. At the top of an EBB, this set can be computed as the union of: instruction. The topological iteration order guarantees that this set is available. Values whose live range indicate that they are not live-in to the current EBB should be filtered out. -- The set of arguments to the EBB. These values should all be live-in, although +- The set of parameters the EBB. These values should all be live-in, although it is possible that some are dead and never used anywhere. For each live value, we also track its kill point in the current EBB. This is From c93959de7e5fa3a413bb699b217f285f8389cf52 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Oct 2017 15:41:09 -0700 Subject: [PATCH 1289/3084] Fix a bug handling a use_var of a value not defined in the entry block. --- lib/frontend/src/ssa.rs | 43 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index d735caabb7..68e234018f 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -484,6 +484,7 @@ where // The variable is used but never defined before. This is an irregularity in the // code, but rather than throwing an error we silently initialize the variable to // 0. This will have no effect since this situation happens in unreachable code. + func.dfg.remove_ebb_param(temp_arg_val); if !func.layout.is_ebb_inserted(dest_ebb) { func.layout.append_ebb(dest_ebb) }; @@ -654,7 +655,7 @@ where mod tests { use cretonne::cursor::{Cursor, FuncCursor}; use cretonne::entity::EntityRef; - use cretonne::ir::{Function, InstBuilder, Inst, JumpTableData}; + use cretonne::ir::{Function, InstBuilder, Inst, JumpTableData, Opcode}; use cretonne::ir::types::*; use cretonne::verify_function; use cretonne::ir::instructions::BranchInfo; @@ -1061,4 +1062,44 @@ mod tests { assert_eq!(func.dfg.ebb_params(ebb1)[1], y3); assert_eq!(func.dfg.ebb_params(ebb1)[0], x2); } + + #[test] + fn undef_in_entry() { + // Use a var which has not been defined. The search should hit the + // top of the entry block, and then fall back to inserting an iconst. + let mut func = Function::new(); + let mut ssa: SSABuilder = SSABuilder::new(); + let ebb0 = func.dfg.make_ebb(); + let block = ssa.declare_ebb_header_block(ebb0); + ssa.seal_ebb_header_block(ebb0, &mut func); + let x_var = Variable(0); + assert_eq!(func.dfg.num_ebb_params(ebb0), 0); + ssa.use_var(&mut func, x_var, I32, block); + assert_eq!(func.dfg.num_ebb_params(ebb0), 0); + assert_eq!( + func.dfg[func.layout.first_inst(ebb0).unwrap()].opcode(), + Opcode::Iconst + ); + } + + #[test] + fn undef_in_entry_sealed_after() { + // Use a var which has not been defined, but the block is not sealed + // until afterward. Before sealing, the SSA builder should insert an + // ebb param; after sealing, it should be removed. + let mut func = Function::new(); + let mut ssa: SSABuilder = SSABuilder::new(); + let ebb0 = func.dfg.make_ebb(); + let block = ssa.declare_ebb_header_block(ebb0); + let x_var = Variable(0); + assert_eq!(func.dfg.num_ebb_params(ebb0), 0); + ssa.use_var(&mut func, x_var, I32, block); + assert_eq!(func.dfg.num_ebb_params(ebb0), 1); + ssa.seal_ebb_header_block(ebb0, &mut func); + assert_eq!(func.dfg.num_ebb_params(ebb0), 0); + assert_eq!( + func.dfg[func.layout.first_inst(ebb0).unwrap()].opcode(), + Opcode::Iconst + ); + } } From 8387c53c3fab3e8b0a87997ea2bae0883344614c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Oct 2017 16:01:39 -0700 Subject: [PATCH 1290/3084] Fix a typo. --- cranelift/docs/regalloc.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/docs/regalloc.rst b/cranelift/docs/regalloc.rst index 8f8781a520..74bd6e8575 100644 --- a/cranelift/docs/regalloc.rst +++ b/cranelift/docs/regalloc.rst @@ -226,7 +226,7 @@ The interference can be resolved by isolating the SSA values passed as EBB argum return v3 } -Now the EBB argument is ``v5`` which is *not* isself live into ``ebb1``, +Now the EBB argument is ``v5`` which is *not* itself live into ``ebb1``, resolving the interference. The coalescing pass groups the SSA values into sets called *virtual registers* From bdf726d101fcd89ddb6d5191ae83f517376d2b86 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Sat, 21 Oct 2017 12:30:46 -0700 Subject: [PATCH 1291/3084] Go back to tracking the latest mypy release. Our bug (https://github.com/python/mypy/issues/4069) was fixed in the mypy-0.540 release. --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 1309bfce63..03dd6f76e6 100644 --- a/.travis.yml +++ b/.travis.yml @@ -9,7 +9,7 @@ addons: packages: - python3-pip install: - - pip3 install --user --upgrade mypy==0.521 flake8 + - pip3 install --user --upgrade mypy flake8 - travis_wait ./check-rustfmt.sh --install script: ./test-all.sh cache: From fdf2486c0a38b0cb2409d44c381cdbdc3a5e9667 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 23 Oct 2017 14:57:20 -0700 Subject: [PATCH 1292/3084] Handle uninitialized values of type b1 and vector. --- lib/frontend/src/ssa.rs | 52 +++++++++++++++++++++++++++++++---------- 1 file changed, 40 insertions(+), 12 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 68e234018f..aed83e4a12 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -488,19 +488,26 @@ where if !func.layout.is_ebb_inserted(dest_ebb) { func.layout.append_ebb(dest_ebb) }; - let ty = func.dfg.value_type(temp_arg_val); - let mut cur = FuncCursor::new(func).at_first_insertion_point(dest_ebb); - let val = if ty.is_int() { - cur.ins().iconst(ty, 0) - } else if ty == F32 { - cur.ins().f32const(Ieee32::with_bits(0)) - } else if ty == F64 { - cur.ins().f64const(Ieee64::with_bits(0)) - } else { - panic!("value used but never declared and initialization not supported") - }; self.side_effects.instructions_added_to_ebbs.push(dest_ebb); - val + fn emit_zero(ty: Type, mut cur: FuncCursor) -> Value { + if ty.is_int() { + cur.ins().iconst(ty, 0) + } else if ty.is_bool() { + cur.ins().bconst(ty, false) + } else if ty == F32 { + cur.ins().f32const(Ieee32::with_bits(0)) + } else if ty == F64 { + cur.ins().f64const(Ieee64::with_bits(0)) + } else if ty.is_vector() { + emit_zero(ty.lane_type(), cur) + } else { + panic!("use of undefined value unsupported for type {}", ty) + } + } + emit_zero( + func.dfg.value_type(temp_arg_val), + FuncCursor::new(func).at_first_insertion_point(dest_ebb), + ) } ZeroOneOrMore::One(pred_val) => { // Here all the predecessors use a single value to represent our variable @@ -1063,6 +1070,27 @@ mod tests { assert_eq!(func.dfg.ebb_params(ebb1)[0], x2); } + #[test] + fn undef() { + // Use vars of varous types which have not been defined. + let mut func = Function::new(); + let mut ssa: SSABuilder = SSABuilder::new(); + let ebb0 = func.dfg.make_ebb(); + let block = ssa.declare_ebb_header_block(ebb0); + ssa.seal_ebb_header_block(ebb0, &mut func); + let i32_var = Variable(0); + let f32_var = Variable(1); + let f64_var = Variable(2); + let b1_var = Variable(3); + let f32x4_var = Variable(4); + ssa.use_var(&mut func, i32_var, I32, block); + ssa.use_var(&mut func, f32_var, F32, block); + ssa.use_var(&mut func, f64_var, F64, block); + ssa.use_var(&mut func, b1_var, B1, block); + ssa.use_var(&mut func, f32x4_var, F32X4, block); + assert_eq!(func.dfg.num_ebb_params(ebb0), 0); + } + #[test] fn undef_in_entry() { // Use a var which has not been defined. The search should hit the From bbdce7e3e077245bdfb904204d63e8eec756255a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 24 Oct 2017 10:39:01 -0700 Subject: [PATCH 1293/3084] Update the comment for 'notrap()'. --- lib/cretonne/src/ir/memflags.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/ir/memflags.rs b/lib/cretonne/src/ir/memflags.rs index 8d0694e1a0..fd836d2752 100644 --- a/lib/cretonne/src/ir/memflags.rs +++ b/lib/cretonne/src/ir/memflags.rs @@ -54,8 +54,9 @@ impl MemFlags { /// would cause a trap when accessing the effective address, the Cretonne memory operation is /// also required to trap. /// - /// The `notrap` flag gives a Cretonne operation permission to not trap. This makes it possible - /// to delete an unused load or a dead store instruction. + /// The `notrap` flag tells Cretonne that the memory is *accessible*, which means that + /// accesses will not trap. This makes it possible to delete an unused load or a dead store + /// instruction. pub fn notrap(self) -> bool { self.read(FlagBit::Notrap) } From 2d5faa01af36ab27bb15dedc25ae20c81990bfcf Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 24 Oct 2017 15:28:28 -0700 Subject: [PATCH 1294/3084] Simplify code using the FunctionBuilder helper functions. --- lib/wasm/src/code_translator.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 7c2d8ae15e..d74bbbc688 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -362,7 +362,7 @@ pub fn translate_operator( state.peekn(num_args), ); state.popn(num_args); - state.pushn(builder.func.dfg.inst_results(call)); + state.pushn(builder.inst_results(call)); } Operator::CallIndirect { index, table_index } => { // `index` is the index of the function's signature and `table_index` is the index of @@ -378,7 +378,7 @@ pub fn translate_operator( state.peekn(num_args), ); state.popn(num_args); - state.pushn(builder.func.dfg.inst_results(call)); + state.pushn(builder.inst_results(call)); } /******************************* Memory management *********************************** * Memory management is handled by environment. It is usually translated into calls to From fc0671a0cfcd5c0ea785669556574cda3e08e22a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 24 Oct 2017 22:26:03 -0700 Subject: [PATCH 1295/3084] Avoid dangling references to block params when sealing an unreachable block. --- lib/cretonne/meta/base/instructions.py | 10 ++- lib/frontend/src/ssa.rs | 85 ++++++++++++++++++++------ 2 files changed, 71 insertions(+), 24 deletions(-) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index f8b6ea989e..76682d2dff 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -429,24 +429,22 @@ iconst = Instruction( ins=N, outs=a) N = Operand('N', ieee32) -a = Operand('a', f32, doc='A constant integer scalar or vector value') +a = Operand('a', f32, doc='A constant f32 scalar value') f32const = Instruction( 'f32const', r""" Floating point constant. - Create a :type:`f32` SSA value with an immediate constant value, or a - floating point vector where all the lanes have the same value. + Create a :type:`f32` SSA value with an immediate constant value. """, ins=N, outs=a) N = Operand('N', ieee64) -a = Operand('a', f64, doc='A constant integer scalar or vector value') +a = Operand('a', f64, doc='A constant f64 scalar value') f64const = Instruction( 'f64const', r""" Floating point constant. - Create a :type:`f64` SSA value with an immediate constant value, or a - floating point vector where all the lanes have the same value. + Create a :type:`f64` SSA value with an immediate constant value. """, ins=N, outs=a) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index aed83e4a12..68f999c889 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -199,6 +199,35 @@ enum Call { FinishPredecessorsLookup(Value, Ebb), } +/// Emit instructions to produce a zero value in the given type. +fn emit_zero(ty: Type, mut cur: FuncCursor) -> Value { + if ty.is_int() { + cur.ins().iconst(ty, 0) + } else if ty.is_bool() { + cur.ins().bconst(ty, false) + } else if ty == F32 { + cur.ins().f32const(Ieee32::with_bits(0)) + } else if ty == F64 { + cur.ins().f64const(Ieee64::with_bits(0)) + } else if ty.is_vector() { + let scalar_ty = ty.lane_type(); + if scalar_ty.is_int() { + cur.ins().iconst(ty, 0) + } else if scalar_ty.is_bool() { + cur.ins().bconst(ty, false) + } else if scalar_ty == F32 { + let scalar = cur.ins().f32const(Ieee32::with_bits(0)); + cur.ins().splat(ty, scalar) + } else if scalar_ty == F64 { + let scalar = cur.ins().f64const(Ieee64::with_bits(0)); + cur.ins().splat(ty, scalar) + } else { + panic!("unimplemented scalar type: {:?}", ty) + } + } else { + panic!("unimplemented type: {:?}", ty) + } +} /// The following methods are the API of the SSA builder. Here is how it should be used when /// translating to Cretonne IL: /// @@ -484,30 +513,17 @@ where // The variable is used but never defined before. This is an irregularity in the // code, but rather than throwing an error we silently initialize the variable to // 0. This will have no effect since this situation happens in unreachable code. - func.dfg.remove_ebb_param(temp_arg_val); if !func.layout.is_ebb_inserted(dest_ebb) { func.layout.append_ebb(dest_ebb) }; self.side_effects.instructions_added_to_ebbs.push(dest_ebb); - fn emit_zero(ty: Type, mut cur: FuncCursor) -> Value { - if ty.is_int() { - cur.ins().iconst(ty, 0) - } else if ty.is_bool() { - cur.ins().bconst(ty, false) - } else if ty == F32 { - cur.ins().f32const(Ieee32::with_bits(0)) - } else if ty == F64 { - cur.ins().f64const(Ieee64::with_bits(0)) - } else if ty.is_vector() { - emit_zero(ty.lane_type(), cur) - } else { - panic!("use of undefined value unsupported for type {}", ty) - } - } - emit_zero( + let zero = emit_zero( func.dfg.value_type(temp_arg_val), FuncCursor::new(func).at_first_insertion_point(dest_ebb), - ) + ); + func.dfg.remove_ebb_param(temp_arg_val); + func.dfg.change_to_alias(temp_arg_val, zero); + zero } ZeroOneOrMore::One(pred_val) => { // Here all the predecessors use a single value to represent our variable @@ -1130,4 +1146,37 @@ mod tests { Opcode::Iconst ); } + + #[test] + fn unreachable_use() { + let mut func = Function::new(); + let mut ssa: SSABuilder = SSABuilder::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + // Here is the pseudo-program we want to translate: + // ebb0: + // return + // ebb1: + // brz v1, ebb1 + // jump ebb1 + let _block0 = ssa.declare_ebb_header_block(ebb0); + ssa.seal_ebb_header_block(ebb0, &mut func); + let block1 = ssa.declare_ebb_header_block(ebb1); + let block2 = ssa.declare_ebb_body_block(block1); + { + let mut cur = FuncCursor::new(&mut func); + cur.insert_ebb(ebb0); + cur.insert_ebb(ebb1); + cur.goto_bottom(ebb0); + cur.ins().return_(&[]); + let x_var = Variable(0); + cur.goto_bottom(ebb1); + let val = ssa.use_var(&mut cur.func, x_var, I32, block1).0; + let brz = cur.ins().brz(val, ebb1, &[]); + ssa.declare_ebb_predecessor(ebb1, block1, brz); + let j = cur.ins().jump(ebb1, &[]); + ssa.declare_ebb_predecessor(ebb1, block2, j); + } + ssa.seal_ebb_header_block(ebb1, &mut func); + } } From b6eae2cfb669e7b6a1bcb107ff25d00c95adfb32 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 25 Oct 2017 10:11:36 -0700 Subject: [PATCH 1296/3084] Put `FlagsOrIsa` in code-comments. --- cranelift/src/utils.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index b243b8953b..25c6dbf067 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -73,7 +73,7 @@ pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CtonError } } -/// Like FlagsOrIsa, but holds ownership. +/// Like `FlagsOrIsa`, but holds ownership. pub enum OwnedFlagsOrIsa { Flags(settings::Flags), Isa(Box), From dbd75483e8b1f8aecca26312d04fce46b419fae3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 25 Oct 2017 10:14:10 -0700 Subject: [PATCH 1297/3084] Rename `arg_value` to `param_value` for consistency. Also rename FunctionBuilder's `pristine` to `params_values_initialized` to avoid confusion with the unrelated `is_pristine` accessor function. --- lib/frontend/src/frontend.rs | 49 ++++++++++++++++++++++----------- lib/frontend/src/lib.rs | 2 +- lib/wasm/src/func_translator.rs | 28 +++++++++---------- 3 files changed, 48 insertions(+), 31 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 708de69881..0534fa487d 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -22,7 +22,7 @@ where ssa: SSABuilder, ebbs: EntityMap, types: EntityMap, - function_args_values: Vec, + function_params_values: Vec, } @@ -40,7 +40,9 @@ where builder: &'a mut ILBuilder, position: Position, - pristine: bool, + + /// Has builder.function_params_values been populated yet? + params_values_initialized: bool, } #[derive(Clone, Default)] @@ -66,7 +68,7 @@ where ssa: SSABuilder::new(), ebbs: EntityMap::new(), types: EntityMap::new(), - function_args_values: Vec::new(), + function_params_values: Vec::new(), } } @@ -74,12 +76,12 @@ where self.ssa.clear(); self.ebbs.clear(); self.types.clear(); - self.function_args_values.clear(); + self.function_params_values.clear(); } fn is_empty(&self) -> bool { self.ssa.is_empty() && self.ebbs.is_empty() && self.types.is_empty() && - self.function_args_values.is_empty() + self.function_params_values.is_empty() } } @@ -234,7 +236,7 @@ where ebb: Ebb::new(0), basic_block: Block::new(0), }, - pristine: true, + params_values_initialized: false, } } @@ -263,8 +265,8 @@ where /// successor), the block will be declared filled and it will not be possible to append /// instructions to it. pub fn switch_to_block(&mut self, ebb: Ebb, jump_args: &[Value]) -> &[Value] { - if self.pristine { - self.fill_function_args_values(ebb); + if !self.params_values_initialized { + self.fill_function_params_values(ebb); } if !self.builder.ebbs[self.position.ebb].pristine { // First we check that the previous block has been filled. @@ -332,12 +334,21 @@ where ); } - /// Returns the value corresponding to the `i`-th argument of the function as defined by + /// Returns the value corresponding to the `i`-th parameter of the function as defined by /// the function signature. Panics if `i` is out of bounds or if called before the first call /// to `switch_to_block`. + pub fn param_value(&self, i: usize) -> Value { + debug_assert!( + self.params_values_initialized, + "you have to call switch_to_block first." + ); + self.builder.function_params_values[i] + } + + /// Use param_value instead. + #[deprecated(since = "forever", note = "arg_value is renamed to param_value")] pub fn arg_value(&self, i: usize) -> Value { - debug_assert!(!self.pristine, "you have to call switch_to_block first."); - self.builder.function_args_values[i] + self.param_value(i) } /// Creates a jump table in the function, to be used by `br_table` instructions. @@ -475,6 +486,12 @@ where self.builder.ssa.predecessors(self.position.ebb).is_empty()) } + /// Returns `true` if and only if the entry block has been started and `param_value` + /// may be called. + pub fn entry_block_started(&self) -> bool { + self.params_values_initialized + } + /// Returns `true` if and only if no instructions have been added since the last call to /// `switch_to_block`. pub fn is_pristine(&self) -> bool { @@ -556,14 +573,14 @@ where } } - fn fill_function_args_values(&mut self, ebb: Ebb) { - debug_assert!(self.pristine); + fn fill_function_params_values(&mut self, ebb: Ebb) { + debug_assert!(!self.params_values_initialized); for argtyp in &self.func.signature.params { - self.builder.function_args_values.push( + self.builder.function_params_values.push( self.func.dfg.append_ebb_param(ebb, argtyp.value_type), ); } - self.pristine = false; + self.params_values_initialized = true; } @@ -685,7 +702,7 @@ mod tests { builder.switch_to_block(block0, &[]); builder.seal_block(block0); { - let tmp = builder.arg_value(0); + let tmp = builder.param_value(0); builder.def_var(x, tmp); } { diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index cf0044d1c0..bca94f842c 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -84,7 +84,7 @@ //! builder.switch_to_block(block0, &[]); //! builder.seal_block(block0); //! { -//! let tmp = builder.arg_value(0); +//! let tmp = builder.param_value(0); //! builder.def_var(x, tmp); //! } //! { diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index ebac333531..2de7d38191 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -88,14 +88,14 @@ impl FuncTranslator { // `environ`. The callback functions may need to insert things in the entry block. builder.ensure_inserted_ebb(); - let num_args = declare_wasm_arguments(&mut builder); + let num_params = declare_wasm_parameters(&mut builder); // Set up the translation state with a single pushed control block representing the whole // function and its return values. let exit_block = builder.create_ebb(); self.state.initialize(&builder.func.signature, exit_block); - parse_local_decls(&mut reader, &mut builder, num_args)?; + parse_local_decls(&mut reader, &mut builder, num_params)?; parse_function_body(reader, &mut builder, &mut self.state, environ) } } @@ -103,21 +103,21 @@ impl FuncTranslator { /// Declare local variables for the signature parameters that correspond to WebAssembly locals. /// /// Return the number of local variables declared. -fn declare_wasm_arguments(builder: &mut FunctionBuilder) -> usize { +fn declare_wasm_parameters(builder: &mut FunctionBuilder) -> usize { let sig_len = builder.func.signature.params.len(); let mut next_local = 0; for i in 0..sig_len { - let arg_type = builder.func.signature.params[i]; - // There may be additional special-purpose arguments following the normal WebAssembly - // signature arguments. For example, a `vmctx` pointer. - if arg_type.purpose == ir::ArgumentPurpose::Normal { - // This is a normal WebAssembly signature argument, so create a local for it. + let param_type = builder.func.signature.params[i]; + // There may be additional special-purpose parameters following the normal WebAssembly + // signature parameters. For example, a `vmctx` pointer. + if param_type.purpose == ir::ArgumentPurpose::Normal { + // This is a normal WebAssembly signature parameter, so create a local for it. let local = Local::new(next_local); - builder.declare_var(local, arg_type.value_type); + builder.declare_var(local, param_type.value_type); next_local += 1; - let arg_value = builder.arg_value(i); - builder.def_var(local, arg_value); + let param_value = builder.param_value(i); + builder.def_var(local, param_value); } } @@ -126,13 +126,13 @@ fn declare_wasm_arguments(builder: &mut FunctionBuilder) -> usize { /// Parse the local variable declarations that precede the function body. /// -/// Declare local variables, starting from `num_args`. +/// Declare local variables, starting from `num_params`. fn parse_local_decls( reader: &mut BinaryReader, builder: &mut FunctionBuilder, - num_args: usize, + num_params: usize, ) -> CtonResult { - let mut next_local = num_args; + let mut next_local = num_params; let local_count = reader.read_local_count().map_err( |_| CtonError::InvalidInput, )?; From 2932a9314dca1a6ccb3de70bf505c84419a25aa9 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 25 Oct 2017 11:17:29 -0700 Subject: [PATCH 1298/3084] Don't allocate srclocs if we only have default SourceLoc values. This avoids needless extra indentation in `write_function` in the case where no SourceLoc information is available. --- lib/frontend/src/frontend.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 0534fa487d..92eca41dcc 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -132,7 +132,9 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short let inst = self.builder.func.dfg.make_inst(data.clone()); self.builder.func.dfg.make_inst_results(inst, ctrl_typevar); self.builder.func.layout.append_inst(inst, self.ebb); - self.builder.func.srclocs[inst] = self.builder.srcloc; + if !self.builder.srcloc.is_default() { + self.builder.func.srclocs[inst] = self.builder.srcloc; + } if data.opcode().is_branch() { match data.branch_destination() { From e8ecf1f809e46eef18e3eed2e018aabb9c4072b7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Oct 2017 11:14:11 -0700 Subject: [PATCH 1299/3084] Add a FixedTied constraint kind for operand constraints. Fixes #175. The Intel division instructions have fixed input operands that are clobbered by fixed output operands, so the value passed as an input will be clobbered just like a tied operand. The FixedTied operand constraint is used to indicate a fixed input operand that has a corresponding output operand with the same fixed register. Teach the spiller to teach a FixedTied operand the same as a Tied operand constraint and make sure that the input value is killed by the instruction. --- .../regalloc/output-interference.cton | 15 +++++++++++ lib/cretonne/meta/cdsl/isa.py | 10 +++++++ lib/cretonne/meta/cdsl/registers.py | 5 +++- lib/cretonne/meta/gen_encoding.py | 27 ++++++++++--------- lib/cretonne/src/isa/constraints.rs | 10 ++++++- lib/cretonne/src/regalloc/coloring.rs | 19 ++++++++++--- lib/cretonne/src/regalloc/spilling.rs | 4 +++ 7 files changed, 72 insertions(+), 18 deletions(-) create mode 100644 cranelift/filetests/regalloc/output-interference.cton diff --git a/cranelift/filetests/regalloc/output-interference.cton b/cranelift/filetests/regalloc/output-interference.cton new file mode 100644 index 0000000000..11079a383f --- /dev/null +++ b/cranelift/filetests/regalloc/output-interference.cton @@ -0,0 +1,15 @@ +test compile +set is_64bit=1 +isa intel haswell + +function %test(i64) -> i64 native { +ebb0(v0: i64): + v2 = iconst.i64 12 + ; This division clobbers two of its fixed input registers on Intel. + ; These are FixedTied constraints that the spiller needs to resolve. + v5 = udiv v0, v2 + v6 = iconst.i64 13 + v9 = udiv v0, v6 + v10 = iadd v5, v9 + return v10 +} diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 6d06f8ef0b..9fcfa440a3 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -394,6 +394,16 @@ class EncRecipe(object): o2i[o] = i return (i2o, o2i) + def fixed_ops(self): + # type: () -> Tuple[Set[Register], Set[Register]] + """ + Return two sets of registers representing the fixed input and output + operands. + """ + i = set(r for r in self.ins if isinstance(r, Register)) + o = set(r for r in self.outs if isinstance(r, Register)) + return (i, o) + def recipe_pred(self): # type: () -> RecipePred """ diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index 37d75d3d58..22e954ee8a 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -286,7 +286,10 @@ class RegClass(object): For example: `GPR.r5`. """ - return Register(self, self.bank.unit_by_name(attr)) + reg = Register(self, self.bank.unit_by_name(attr)) + # Save this register so we won't have to create it again. + setattr(self, attr, reg) + return reg def mask(self): # type: () -> List[int] diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 43b00d37f6..283b9d1e0d 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -754,17 +754,14 @@ def emit_recipe_constraints(isa, fmt): for r in isa.all_recipes: fmt.comment(r.name) tied_i2o, tied_o2i = r.ties() + fixed_ins, fixed_outs = r.fixed_ops() with fmt.indented('RecipeConstraints {', '},'): - emit_operand_constraints(r, r.ins, 'ins', tied_i2o, fmt) - emit_operand_constraints(r, r.outs, 'outs', tied_o2i, fmt) - fmt.format( - 'fixed_ins: {},', - str(any(isinstance(c, Register) - for c in r.ins)).lower()) - fmt.format( - 'fixed_outs: {},', - str(any(isinstance(c, Register) - for c in r.outs)).lower()) + emit_operand_constraints( + r, r.ins, 'ins', tied_i2o, fixed_outs, fmt) + emit_operand_constraints( + r, r.outs, 'outs', tied_o2i, fixed_ins, fmt) + fmt.format('fixed_ins: {},', str(bool(fixed_ins)).lower()) + fmt.format('fixed_outs: {},', str(bool(fixed_outs)).lower()) fmt.format('tied_ops: {},', str(bool(tied_i2o)).lower()) fmt.format( 'clobbers_flags: {},', @@ -776,11 +773,16 @@ def emit_operand_constraints( seq, # type: Sequence[OperandConstraint] field, # type: str tied, # type: Dict[int, int] + fixops, # type: Set[Register] fmt # type: srcgen.Formatter ): # type: (...) -> None """ Emit a struct field initializer for an array of operand constraints. + + :param field: The name of the struct field to emit. + :param tied: Map of tied opnums to counterparts. + :param fix_ops: Set of fixed operands on the other side of the inst. """ if len(seq) == 0: fmt.line('{}: &[],'.format(field)) @@ -796,8 +798,9 @@ def emit_operand_constraints( fmt.format('regclass: &{}_DATA,', cons) elif isinstance(cons, Register): assert n not in tied, "Can't tie fixed register operand" - fmt.format( - 'kind: ConstraintKind::FixedReg({}),', cons.unit) + # See if this fixed register is also on the other side. + t = 'FixedTied' if cons in fixops else 'FixedReg' + fmt.format('kind: ConstraintKind::{}({}),', t, cons.unit) fmt.format('regclass: &{}_DATA,', cons.regclass) elif isinstance(cons, int): # This is a tied output constraint. It should never happen diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index aa3268187f..eb6b33c3cb 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -37,7 +37,8 @@ impl OperandConstraint { false } } - ConstraintKind::FixedReg(reg) => { + ConstraintKind::FixedReg(reg) | + ConstraintKind::FixedTied(reg) => { loc == ValueLoc::Reg(reg) && self.regclass.contains(reg) } ConstraintKind::Stack => { @@ -73,6 +74,13 @@ pub enum ConstraintKind { /// the out operand is `Tied(in)`. Tied(u8), + /// This operand must be a fixed register, and it has a tied counterpart. + /// + /// This works just like `FixedReg`, but additionally indicates that there are identical + /// input/output operands for this fixed register. For an input operand, this means that the + /// value will be clobbered by the instruction + FixedTied(RegUnit), + /// This operand must be a value in a stack slot. /// /// The constraint's `regclass` field is the register class that would normally be used to load diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index b6b403fb30..b44860c958 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -478,7 +478,8 @@ impl<'a> Context<'a> { // already in a register. let cur_reg = self.divert.reg(value, &self.cur.func.locations); match op.kind { - ConstraintKind::FixedReg(regunit) => { + ConstraintKind::FixedReg(regunit) | + ConstraintKind::FixedTied(regunit) => { if regunit != cur_reg { self.solver.reassign_in( value, @@ -532,7 +533,9 @@ impl<'a> Context<'a> { } } } - _ => {} + ConstraintKind::FixedReg(_) | + ConstraintKind::FixedTied(_) | + ConstraintKind::Stack => {} } } } @@ -674,8 +677,15 @@ impl<'a> Context<'a> { throughs: &[LiveValue], ) { for (op, lv) in constraints.iter().zip(defs) { - if let ConstraintKind::FixedReg(reg) = op.kind { - self.add_fixed_output(lv.value, op.regclass, reg, throughs); + match op.kind { + ConstraintKind::FixedReg(reg) | + ConstraintKind::FixedTied(reg) => { + self.add_fixed_output(lv.value, op.regclass, reg, throughs); + } + ConstraintKind::Reg | + ConstraintKind::Tied(_) | + ConstraintKind::Stack => {} + } } } @@ -741,6 +751,7 @@ impl<'a> Context<'a> { for (op, lv) in constraints.iter().zip(defs) { match op.kind { ConstraintKind::FixedReg(_) | + ConstraintKind::FixedTied(_) | ConstraintKind::Stack => continue, ConstraintKind::Reg => { self.solver.add_def(lv.value, op.regclass, !lv.is_local); diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index edda6f6f59..85d7b20b4a 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -301,6 +301,10 @@ impl<'a> Context<'a> { // A tied operand must kill the used value. reguse.tied = !lr.killed_at(inst, ebb, &self.cur.func.layout); } + ConstraintKind::FixedTied(_) => { + reguse.fixed = true; + reguse.tied = !lr.killed_at(inst, ebb, &self.cur.func.layout); + } ConstraintKind::Reg => {} } if lr.affinity.is_stack() { From 02e81dd1d7adb9294e2122c997f6f8c4c5c43852 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Oct 2017 11:40:37 -0700 Subject: [PATCH 1300/3084] Fix build after flake8 update. There's a new version of flake8 out which doesn't like variables names i, l, I. No functional change intended. --- lib/cretonne/meta/cdsl/settings.py | 8 ++++---- lib/cretonne/meta/cdsl/test_ti.py | 12 ++++++------ lib/cretonne/meta/isa/riscv/encodings.py | 8 ++++---- lib/cretonne/meta/isa/riscv/recipes.py | 4 ++-- lib/cretonne/src/isa/riscv/mod.rs | 15 ++++++++++++--- 5 files changed, 28 insertions(+), 19 deletions(-) diff --git a/lib/cretonne/meta/cdsl/settings.py b/lib/cretonne/meta/cdsl/settings.py index cea8ec7eee..3a9f803c05 100644 --- a/lib/cretonne/meta/cdsl/settings.py +++ b/lib/cretonne/meta/cdsl/settings.py @@ -389,7 +389,7 @@ class Preset(object): The list will have an entry for each setting byte in the settings group. """ - l = [(0, 0)] * self.group.settings_size + lst = [(0, 0)] * self.group.settings_size # Apply setting values in order. for s, v in self.values: @@ -397,11 +397,11 @@ class Preset(object): s_mask = s.byte_mask() s_val = s.byte_for_value(v) assert (s_val & ~s_mask) == 0 - l_mask, l_val = l[ofs] + l_mask, l_val = lst[ofs] # Accumulated mask of modified bits. l_mask |= s_mask # Overwrite the relevant bits with the new value. l_val = (l_val & ~s_mask) | s_val - l[ofs] = (l_mask, l_val) + lst[ofs] = (l_mask, l_val) - return l + return lst diff --git a/lib/cretonne/meta/cdsl/test_ti.py b/lib/cretonne/meta/cdsl/test_ti.py index bffbbd527d..b9b3e0b646 100644 --- a/lib/cretonne/meta/cdsl/test_ti.py +++ b/lib/cretonne/meta/cdsl/test_ti.py @@ -356,9 +356,9 @@ class TestRTL(TypeCheckingBaseTest): typing = ti_rtl(r, ti).extract() # The number of possible typings is 9 * (3+ 2*2 + 3) = 90 - l = [(t[self.v0], t[self.v1]) for t in typing.concrete_typings()] - assert (len(l) == len(set(l)) and len(l) == 90) - for (tv0, tv1) in l: + lst = [(t[self.v0], t[self.v1]) for t in typing.concrete_typings()] + assert (len(lst) == len(set(lst)) and len(lst) == 90) + for (tv0, tv1) in lst: typ0, typ1 = (tv0.singleton_type(), tv1.singleton_type()) if (op == ireduce): assert typ0.wider_or_equal(typ1) @@ -396,9 +396,9 @@ class TestRTL(TypeCheckingBaseTest): typing = ti_rtl(r, ti).extract() # The number of possible typings is 9*(2 + 1) = 27 - l = [(t[self.v0], t[self.v1]) for t in typing.concrete_typings()] - assert (len(l) == len(set(l)) and len(l) == 27) - for (tv0, tv1) in l: + lst = [(t[self.v0], t[self.v1]) for t in typing.concrete_typings()] + assert (len(lst) == len(set(lst)) and len(lst) == 27) + for (tv0, tv1) in lst: (typ0, typ1) = (tv0.singleton_type(), tv1.singleton_type()) if (op == fdemote): assert typ0.wider_or_equal(typ1) diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index 990b5c4112..ac719460a1 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -7,7 +7,7 @@ from base.immediates import intcc from .defs import RV32, RV64 from .recipes import OPIMM, OPIMM32, OP, OP32, LUI, BRANCH, JALR, JAL from .recipes import LOAD, STORE -from .recipes import R, Rshamt, Ricmp, I, Iz, Iicmp, Iret, Icall, Icopy +from .recipes import R, Rshamt, Ricmp, Ii, Iz, Iicmp, Iret, Icall, Icopy from .recipes import U, UJ, UJcall, SB, SBzero, GPsp, GPfi, Irmov from .settings import use_m from cdsl.ast import Var @@ -47,14 +47,14 @@ for inst, inst_imm, f3, f7 in [ # Immediate versions for add/xor/or/and. if inst_imm: - RV32.enc(inst_imm.i32, I, OPIMM(f3)) - RV64.enc(inst_imm.i64, I, OPIMM(f3)) + RV32.enc(inst_imm.i32, Ii, OPIMM(f3)) + RV64.enc(inst_imm.i64, Ii, OPIMM(f3)) # 32-bit ops in RV64. RV64.enc(base.iadd.i32, R, OP32(0b000, 0b0000000)) RV64.enc(base.isub.i32, R, OP32(0b000, 0b0100000)) # There are no andiw/oriw/xoriw variations. -RV64.enc(base.iadd_imm.i32, I, OPIMM32(0b000)) +RV64.enc(base.iadd_imm.i32, Ii, OPIMM32(0b000)) # Use iadd_imm with %x0 to materialize constants. RV32.enc(base.iconst.i32, Iz, OPIMM(0b000)) diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 87dce643eb..6ce25af6fc 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -106,8 +106,8 @@ Ricmp = EncRecipe( 'Ricmp', IntCompare, size=4, ins=(GPR, GPR), outs=GPR, emit='put_r(bits, in_reg0, in_reg1, out_reg0, sink);') -I = EncRecipe( - 'I', BinaryImm, size=4, ins=GPR, outs=GPR, +Ii = EncRecipe( + 'Ii', BinaryImm, size=4, ins=GPR, outs=GPR, instp=IsSignedInt(BinaryImm.imm, 12), emit='put_i(bits, in_reg0, imm.into(), out_reg0, sink);') diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index ab9eadc68b..1cb76aef41 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -147,7 +147,10 @@ mod tests { }; // ADDI is I/0b00100 - assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst64, types::I64)), "I#04"); + assert_eq!( + encstr(&*isa, isa.encode(&dfg, &inst64, types::I64)), + "Ii#04" + ); // Try to encode iadd_imm.i64 v1, -10000. let inst64_large = InstructionData::BinaryImm { @@ -167,7 +170,10 @@ mod tests { }; // ADDIW is I/0b00110 - assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32, types::I32)), "I#06"); + assert_eq!( + encstr(&*isa, isa.encode(&dfg, &inst32, types::I32)), + "Ii#06" + ); } // Same as above, but for RV32. @@ -211,7 +217,10 @@ mod tests { }; // ADDI is I/0b00100 - assert_eq!(encstr(&*isa, isa.encode(&dfg, &inst32, types::I32)), "I#04"); + assert_eq!( + encstr(&*isa, isa.encode(&dfg, &inst32, types::I32)), + "Ii#04" + ); // Create an imul.i32 which is encodable in RV32, but only when use_m is true. let mul32 = InstructionData::Binary { From 1b71285b34d6bbbbc718523a795e525f971bd6a6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Oct 2017 13:34:55 -0700 Subject: [PATCH 1301/3084] Return bools in GPR registers. Boolean types are returned in %rax, so regclass_for_abi_type() should return GPR. Fixes #179. --- cranelift/filetests/isa/intel/abi-bool.cton | 20 ++++++++++++++++++++ lib/cretonne/src/isa/intel/abi.rs | 6 +++++- 2 files changed, 25 insertions(+), 1 deletion(-) create mode 100644 cranelift/filetests/isa/intel/abi-bool.cton diff --git a/cranelift/filetests/isa/intel/abi-bool.cton b/cranelift/filetests/isa/intel/abi-bool.cton new file mode 100644 index 0000000000..bc1a1ba1d5 --- /dev/null +++ b/cranelift/filetests/isa/intel/abi-bool.cton @@ -0,0 +1,20 @@ +test compile +set is_64bit=1 +isa intel haswell + +function %foo(i64, i64, i64, i32) -> b1 native { +ebb3(v0: i64, v1: i64, v2: i64, v3: i32): + v5 = icmp ne v2, v2 + v8 = iconst.i64 0 + jump ebb2(v8, v3, v5) + +ebb2(v10: i64, v30: i32, v37: b1): + v18 = load.i32 notrap aligned v2 + v27 = iadd.i64 v10, v10 + v31 = icmp eq v30, v30 + brz v31, ebb2(v27, v30, v37) + jump ebb0(v37) + +ebb0(v35: b1): + return v35 +} diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index ec623feed8..4b247b5a13 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -125,7 +125,11 @@ pub fn legalize_signature(sig: &mut ir::Signature, flags: &shared_settings::Flag /// Get register class for a type appearing in a legalized signature. pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { - if ty.is_int() { GPR } else { FPR } + if ty.is_int() || ty.is_bool() { + GPR + } else { + FPR + } } /// Get the set of allocatable registers for `func`. From d37126565eaf2b45c2fbd815059bfe1cb03f665f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Oct 2017 14:28:30 -0700 Subject: [PATCH 1302/3084] Also consider fixed outputs for replace_global_defines. Fixes #178. When an instruction with a fixed output operand defines a globally live SSA value, we need to check if the fixed register is available in the `regs.global` set of registers that can be used across EBB boundaries. If the fixed output register is not available in regs.global, set the replace_global_defines flag so the output operands are rewritten as local values. --- .../filetests/regalloc/global-fixed.cton | 17 ++++++ lib/cretonne/src/regalloc/coloring.rs | 54 ++++++++++++++++--- 2 files changed, 63 insertions(+), 8 deletions(-) create mode 100644 cranelift/filetests/regalloc/global-fixed.cton diff --git a/cranelift/filetests/regalloc/global-fixed.cton b/cranelift/filetests/regalloc/global-fixed.cton new file mode 100644 index 0000000000..9bdcde3be0 --- /dev/null +++ b/cranelift/filetests/regalloc/global-fixed.cton @@ -0,0 +1,17 @@ +test compile +set is_64bit=1 +isa intel haswell + +function %foo() native { +ebb4: + v3 = iconst.i32 0 + jump ebb3 + +ebb3: + v9 = udiv v3, v3 + jump ebb1 + +ebb1: + v19 = iadd.i32 v9, v9 + jump ebb3 +} diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index b44860c958..d8607e4bd1 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -377,23 +377,34 @@ impl<'a> Context<'a> { // This aligns with the " from" line at the top of the function. dbg!(" glob {}", regs.global.display(&self.reginfo)); + // This flag is set when the solver failed to find a solution for the global defines that + // doesn't interfere with `regs.global`. We need to rewrite all of `inst`s global defines + // as local defines followed by copies. + let mut replace_global_defines = false; // Program the fixed output constraints before the general defines. This allows us to // detect conflicts between fixed outputs and tied operands where the input value hasn't // been converted to a solver variable. if constraints.fixed_outs { - self.program_fixed_outputs(constraints.outs, defs, throughs); + self.program_fixed_outputs( + constraints.outs, + defs, + throughs, + &mut replace_global_defines, + ®s.global, + ); } if let Some(sig) = call_sig { - self.program_output_abi(sig, defs, throughs); + self.program_output_abi( + sig, + defs, + throughs, + &mut replace_global_defines, + ®s.global, + ); } self.program_output_constraints(inst, constraints.outs, defs); - // This flag is set when the solver failed to find a solution for the global defines that - // doesn't interfere with `regs.global`. We need to rewrite all of `inst`s global defines - // as local defines followed by copies. - let mut replace_global_defines = false; - // Finally, we've fully programmed the constraint solver. // We expect a quick solution in most cases. let output_regs = self.solver.quick_solve(®s.global).unwrap_or_else(|_| { @@ -675,12 +686,23 @@ impl<'a> Context<'a> { constraints: &[OperandConstraint], defs: &[LiveValue], throughs: &[LiveValue], + replace_global_defines: &mut bool, + global_regs: &AllocatableSet, ) { for (op, lv) in constraints.iter().zip(defs) { match op.kind { ConstraintKind::FixedReg(reg) | ConstraintKind::FixedTied(reg) => { self.add_fixed_output(lv.value, op.regclass, reg, throughs); + if !lv.is_local && !global_regs.is_avail(op.regclass, reg) { + dbg!( + "Fixed output {} in {}:{} is not available in global regs", + lv.value, + op.regclass, + self.reginfo.display_regunit(reg) + ); + *replace_global_defines = true; + } } ConstraintKind::Reg | ConstraintKind::Tied(_) | @@ -693,7 +715,14 @@ impl<'a> Context<'a> { /// Program the output-side ABI constraints for `inst` into the constraint solver. /// /// That means return values for a call instruction. - fn program_output_abi(&mut self, sig: SigRef, defs: &[LiveValue], throughs: &[LiveValue]) { + fn program_output_abi( + &mut self, + sig: SigRef, + defs: &[LiveValue], + throughs: &[LiveValue], + replace_global_defines: &mut bool, + global_regs: &AllocatableSet, + ) { // It's technically possible for a call instruction to have fixed results before the // variable list of results, but we have no known instances of that. // Just assume all results are variable return values. @@ -704,6 +733,15 @@ impl<'a> Context<'a> { if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); self.add_fixed_output(lv.value, rc, reg, throughs); + if !lv.is_local && !global_regs.is_avail(rc, reg) { + dbg!( + "ABI output {} in {}:{} is not available in global regs", + lv.value, + rc, + self.reginfo.display_regunit(reg) + ); + *replace_global_defines = true; + } } else { panic!("ABI argument {} should be in a register", lv.value); } From 91b1566acaadcbdc8e503ed6f485b2c65fa31655 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 25 Oct 2017 18:31:14 -0700 Subject: [PATCH 1303/3084] Use "test regalloc" for the register allocator tests. These tests were only using "test compile" because it doesn't require any filecheck directives to be present, so just stop requiring filecheck directives for "test regalloc" and other filecheck-based test drivers. --- cranelift/filetests/regalloc/aliases.cton | 2 +- cranelift/filetests/regalloc/global-constraints.cton | 2 +- cranelift/filetests/regalloc/global-fixed.cton | 2 +- cranelift/filetests/regalloc/iterate.cton | 2 +- cranelift/filetests/regalloc/output-interference.cton | 2 +- cranelift/filetests/regalloc/schedule-moves.cton | 2 +- cranelift/filetests/regalloc/unreachable_code.cton | 2 ++ cranelift/src/filetest/subtest.rs | 7 +------ 8 files changed, 9 insertions(+), 12 deletions(-) diff --git a/cranelift/filetests/regalloc/aliases.cton b/cranelift/filetests/regalloc/aliases.cton index 575fedd923..a16fe7231b 100644 --- a/cranelift/filetests/regalloc/aliases.cton +++ b/cranelift/filetests/regalloc/aliases.cton @@ -1,4 +1,4 @@ -test compile +test regalloc set is_64bit isa intel haswell diff --git a/cranelift/filetests/regalloc/global-constraints.cton b/cranelift/filetests/regalloc/global-constraints.cton index 5fc63d0c36..b2a3fc0a7e 100644 --- a/cranelift/filetests/regalloc/global-constraints.cton +++ b/cranelift/filetests/regalloc/global-constraints.cton @@ -1,4 +1,4 @@ -test compile +test regalloc isa intel ; This test covers the troubles when values with global live ranges are defined diff --git a/cranelift/filetests/regalloc/global-fixed.cton b/cranelift/filetests/regalloc/global-fixed.cton index 9bdcde3be0..a332311b6e 100644 --- a/cranelift/filetests/regalloc/global-fixed.cton +++ b/cranelift/filetests/regalloc/global-fixed.cton @@ -1,4 +1,4 @@ -test compile +test regalloc set is_64bit=1 isa intel haswell diff --git a/cranelift/filetests/regalloc/iterate.cton b/cranelift/filetests/regalloc/iterate.cton index 15e0145407..6f7e6547fd 100644 --- a/cranelift/filetests/regalloc/iterate.cton +++ b/cranelift/filetests/regalloc/iterate.cton @@ -1,4 +1,4 @@ -test compile +test regalloc set is_64bit isa intel haswell diff --git a/cranelift/filetests/regalloc/output-interference.cton b/cranelift/filetests/regalloc/output-interference.cton index 11079a383f..666c13c0d7 100644 --- a/cranelift/filetests/regalloc/output-interference.cton +++ b/cranelift/filetests/regalloc/output-interference.cton @@ -1,4 +1,4 @@ -test compile +test regalloc set is_64bit=1 isa intel haswell diff --git a/cranelift/filetests/regalloc/schedule-moves.cton b/cranelift/filetests/regalloc/schedule-moves.cton index e0d9cf4e6e..e3bae45be3 100644 --- a/cranelift/filetests/regalloc/schedule-moves.cton +++ b/cranelift/filetests/regalloc/schedule-moves.cton @@ -1,4 +1,4 @@ -test compile +test regalloc isa intel haswell function %pr165() native { diff --git a/cranelift/filetests/regalloc/unreachable_code.cton b/cranelift/filetests/regalloc/unreachable_code.cton index efb1271105..ac667e7721 100644 --- a/cranelift/filetests/regalloc/unreachable_code.cton +++ b/cranelift/filetests/regalloc/unreachable_code.cton @@ -1,4 +1,6 @@ +; Use "test compile" here otherwise the dead blocks won't be eliminated. test compile + set is_64bit isa intel haswell diff --git a/cranelift/src/filetest/subtest.rs b/cranelift/src/filetest/subtest.rs index bbb8b666ca..6c49b6e5a9 100644 --- a/cranelift/src/filetest/subtest.rs +++ b/cranelift/src/filetest/subtest.rs @@ -113,10 +113,5 @@ pub fn build_filechecker(context: &Context) -> Result { format!("filecheck: {}", e) })?; } - let checker = builder.finish(); - if checker.is_empty() { - Err("no filecheck directives in function".to_string()) - } else { - Ok(checker) - } + Ok(builder.finish()) } From 53e93d94a85f8245d4b36e027492ad2c19465d32 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 26 Oct 2017 11:37:43 -0700 Subject: [PATCH 1304/3084] Avoid creating cyclic aliases in unreachable code. --- lib/frontend/src/ssa.rs | 50 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 2 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 68f999c889..3c7e112b56 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -531,8 +531,13 @@ where // We need to replace all the occurences of val with pred_val but since // we can't afford a re-writing pass right now we just declare an alias. func.dfg.remove_ebb_param(temp_arg_val); - func.dfg.change_to_alias(temp_arg_val, pred_val); - pred_val + // Resolve aliases eagerly so that we can check for cyclic aliasing, + // which can occur in unreachable code. + let resolved = func.dfg.resolve_aliases(pred_val); + if temp_arg_val != resolved { + func.dfg.change_to_alias(temp_arg_val, resolved); + } + resolved } ZeroOneOrMore::More() => { // There is disagreement in the predecessors on which value to use so we have @@ -1179,4 +1184,45 @@ mod tests { } ssa.seal_ebb_header_block(ebb1, &mut func); } + + #[test] + fn unreachable_use_with_multiple_preds() { + let mut func = Function::new(); + let mut ssa: SSABuilder = SSABuilder::new(); + let ebb0 = func.dfg.make_ebb(); + let ebb1 = func.dfg.make_ebb(); + let ebb2 = func.dfg.make_ebb(); + // Here is the pseudo-program we want to translate: + // ebb0: + // return + // ebb1: + // brz v1, ebb2 + // jump ebb1 + // ebb2: + // jump ebb1 + let _block0 = ssa.declare_ebb_header_block(ebb0); + ssa.seal_ebb_header_block(ebb0, &mut func); + let block1 = ssa.declare_ebb_header_block(ebb1); + let block2 = ssa.declare_ebb_header_block(ebb2); + { + let mut cur = FuncCursor::new(&mut func); + let x_var = Variable(0); + cur.insert_ebb(ebb0); + cur.insert_ebb(ebb1); + cur.insert_ebb(ebb2); + cur.goto_bottom(ebb0); + cur.ins().return_(&[]); + cur.goto_bottom(ebb1); + let v = ssa.use_var(&mut cur.func, x_var, I32, block1).0; + let brz = cur.ins().brz(v, ebb2, &[]); + let j0 = cur.ins().jump(ebb1, &[]); + cur.goto_bottom(ebb2); + let j1 = cur.ins().jump(ebb1, &[]); + ssa.declare_ebb_predecessor(ebb1, block2, brz); + ssa.declare_ebb_predecessor(ebb1, block1, j0); + ssa.declare_ebb_predecessor(ebb2, block1, j1); + } + ssa.seal_ebb_header_block(ebb1, &mut func); + ssa.seal_ebb_header_block(ebb2, &mut func); + } } From 683408f30b37a84e84ece98ec55703d83604c51d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 26 Oct 2017 12:22:16 -0700 Subject: [PATCH 1305/3084] Break alias cycles by inserting zero values. --- lib/frontend/src/ssa.rs | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 3c7e112b56..65d1873fca 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -530,13 +530,18 @@ where // so we don't need to have it as an ebb argument. // We need to replace all the occurences of val with pred_val but since // we can't afford a re-writing pass right now we just declare an alias. - func.dfg.remove_ebb_param(temp_arg_val); // Resolve aliases eagerly so that we can check for cyclic aliasing, // which can occur in unreachable code. - let resolved = func.dfg.resolve_aliases(pred_val); - if temp_arg_val != resolved { - func.dfg.change_to_alias(temp_arg_val, resolved); + let mut resolved = func.dfg.resolve_aliases(pred_val); + if temp_arg_val == resolved { + // Cycle detected. Break it by creating a zero value. + resolved = emit_zero( + func.dfg.value_type(temp_arg_val), + FuncCursor::new(func).at_first_insertion_point(dest_ebb), + ); } + func.dfg.remove_ebb_param(temp_arg_val); + func.dfg.change_to_alias(temp_arg_val, resolved); resolved } ZeroOneOrMore::More() => { @@ -1183,6 +1188,11 @@ mod tests { ssa.declare_ebb_predecessor(ebb1, block2, j); } ssa.seal_ebb_header_block(ebb1, &mut func); + let flags = settings::Flags::new(&settings::builder()); + match verify_function(&func, &flags) { + Ok(()) => {} + Err(err) => panic!(err.message), + } } #[test] @@ -1224,5 +1234,10 @@ mod tests { } ssa.seal_ebb_header_block(ebb1, &mut func); ssa.seal_ebb_header_block(ebb2, &mut func); + let flags = settings::Flags::new(&settings::builder()); + match verify_function(&func, &flags) { + Ok(()) => {} + Err(err) => panic!(err.message), + } } } From 1b61d5cd1eaad11a932dcc48ad3a452e7dace29e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 26 Oct 2017 20:55:45 -0700 Subject: [PATCH 1306/3084] Enable the verifier by default in "cton-util compile". --- cranelift/src/compile.rs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index f9f718685d..6b0bda21bd 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -12,9 +12,13 @@ use utils::{pretty_error, read_to_string, parse_sets_and_isa}; pub fn run( files: Vec, flag_print: bool, - flag_set: Vec, + mut flag_set: Vec, flag_isa: String, ) -> Result<(), String> { + // Enable the verifier by default, since we're reading IL in from a + // text file. + flag_set.insert(0, "enable_verifier=1".to_string()); + let parsed = parse_sets_and_isa(flag_set, flag_isa)?; for filename in files { From fae5ffb556069319c84c5fc5eaab45bd4cd0d56c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 30 Oct 2017 10:06:23 -0700 Subject: [PATCH 1307/3084] Make generated code more consistent with current rustfmt. --- lib/cretonne/meta/gen_binemit.py | 22 +++++++++++++--------- lib/cretonne/meta/gen_instr.py | 10 +++++----- lib/cretonne/meta/gen_settings.py | 4 ++-- lib/cretonne/meta/isa/riscv/recipes.py | 24 ++++++++++++++---------- 4 files changed, 34 insertions(+), 26 deletions(-) diff --git a/lib/cretonne/meta/gen_binemit.py b/lib/cretonne/meta/gen_binemit.py index 8c591f0513..c3d1b18e86 100644 --- a/lib/cretonne/meta/gen_binemit.py +++ b/lib/cretonne/meta/gen_binemit.py @@ -133,17 +133,21 @@ def gen_isa(isa, fmt): '''.format(isa.name)) if len(isa.all_recipes) == 0: # No encoding recipes: Emit a stub. - with fmt.indented( - 'pub fn emit_inst' - '(func: &Function, inst: Inst, ' - '_divert: &mut RegDiversions, _sink: &mut CS) {', '}'): + with fmt.indented('pub fn emit_inst('): + fmt.line('func: &Function,') + fmt.line('inst: Inst,') + fmt.line('_divert: &mut RegDiversions,') + fmt.line('_sink: &mut CS,') + with fmt.indented(') {', '}'): fmt.line('bad_encoding(func, inst)') else: fmt.line('#[allow(unused_variables, unreachable_code)]') - with fmt.indented( - 'pub fn emit_inst' - '(func: &Function, inst: Inst, ' - 'divert: &mut RegDiversions, sink: &mut CS) {', '}'): + with fmt.indented('pub fn emit_inst('): + fmt.line('func: &Function,') + fmt.line('inst: Inst,') + fmt.line('divert: &mut RegDiversions,') + fmt.line('sink: &mut CS,') + with fmt.indented(') {', '}'): fmt.line('let encoding = func.encodings[inst];') fmt.line('let bits = encoding.bits();') with fmt.indented('match func.encodings[inst].recipe() {', '}'): @@ -151,7 +155,7 @@ def gen_isa(isa, fmt): fmt.comment(recipe.name) with fmt.indented('{} => {{'.format(i), '}'): gen_recipe(recipe, fmt) - fmt.line('_ => {}') + fmt.line('_ => {},') # Allow for un-encoded ghost instructions. # Verifier checks the details. with fmt.indented('if encoding.is_legal() {', '}'): diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index b004aaadf1..4c537acf12 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -97,7 +97,7 @@ def gen_arguments_method(fmt, is_mut): capture = 'ref {}args, '.format(mut) arg = 'args' fmt.line( - '{} {{ {} .. }} => {},' + '{} {{ {}.. }} => {},' .format(n, capture, arg)) @@ -271,7 +271,7 @@ def gen_opcodes(groups, fmt): 'Opcode::{} => true,', i.camel_name, i.name) - fmt.line('_ => false') + fmt.line('_ => false,') # Generate a private opcode_format table. with fmt.indented( @@ -352,7 +352,7 @@ def gen_typesets_table(fmt, type_sets): fmt.comment('Table of value type sets.') assert len(type_sets.table) <= typeset_limit, "Too many type sets" with fmt.indented( - 'const TYPE_SETS : [ir::instructions::ValueTypeSet; {}] = [' + 'const TYPE_SETS: [ir::instructions::ValueTypeSet; {}] = [' .format(len(type_sets.table)), '];'): for ts in type_sets.table: with fmt.indented('ir::instructions::ValueTypeSet {', '},'): @@ -385,7 +385,7 @@ def gen_type_constraints(fmt, instrs): fmt.comment('Table of opcode constraints.') with fmt.indented( - 'const OPCODE_CONSTRAINTS : [OpcodeConstraints; {}] = [' + 'const OPCODE_CONSTRAINTS: [OpcodeConstraints; {}] = [' .format(len(instrs)), '];'): for i in instrs: # Collect constraints for the value results, not including @@ -442,7 +442,7 @@ def gen_type_constraints(fmt, instrs): fmt.comment('Table of operand constraint sequences.') with fmt.indented( - 'const OPERAND_CONSTRAINTS : [OperandConstraint; {}] = [' + 'const OPERAND_CONSTRAINTS: [OperandConstraint; {}] = [' .format(len(operand_seqs.table)), '];'): for c in operand_seqs.table: fmt.line('OperandConstraint::{},'.format(c)) diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index 223e33f478..9f80a391ff 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -62,7 +62,7 @@ def gen_getter(setting, sgrp, fmt): .format(setting.byte_offset), '}'): for i, v in enumerate(setting.values): fmt.line('{} => {}::{},'.format(i, ty, camel_case(v))) - fmt.line('_ => panic!("Invalid enum value")') + fmt.line('_ => panic!("Invalid enum value"),') else: raise AssertionError("Unknown setting kind") @@ -197,7 +197,7 @@ def gen_template(sgrp, fmt): fmt.line('enumerators: &ENUMERATORS,') fmt.line('hash_table: &HASH_TABLE,') vs = ', '.join('{:#04x}'.format(x) for x in v) - fmt.line('defaults: &[ {} ],'.format(vs)) + fmt.line('defaults: &[{}],'.format(vs)) fmt.line('presets: &PRESETS,') fmt.doc_comment( diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 6ce25af6fc..8cbbd40a80 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -131,11 +131,13 @@ Iret = EncRecipe( emit=''' // Return instructions are always a jalr to %x1. // The return address is provided as a special-purpose link argument. - put_i(bits, - 1, // rs1 = %x1 - 0, // no offset. - 0, // rd = %x0: no address written. - sink); + put_i( + bits, + 1, // rs1 = %x1 + 0, // no offset. + 0, // rd = %x0: no address written. + sink, + ); ''') # I-type encoding for `jalr` as an indirect call. @@ -143,11 +145,13 @@ Icall = EncRecipe( 'Icall', IndirectCall, size=4, ins=GPR, outs=(), emit=''' // Indirect instructions are jalr with rd=%x1. - put_i(bits, - in_reg0, - 0, // no offset. - 1, // rd = %x1: link register. - sink); + put_i( + bits, + in_reg0, + 0, // no offset. + 1, // rd = %x1: link register. + sink, + ); ''') From c2665385b1932cab59c9defe4268c2e84e277ce1 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 27 Oct 2017 13:05:49 -0700 Subject: [PATCH 1308/3084] Rename FunctionName to ExternalName. --- lib/cretonne/src/ir/extfunc.rs | 4 +- .../src/ir/{funcname.rs => extname.rs} | 57 ++++++++++--------- lib/cretonne/src/ir/function.rs | 8 +-- lib/cretonne/src/ir/mod.rs | 4 +- lib/cretonne/src/write.rs | 4 +- lib/frontend/src/frontend.rs | 4 +- lib/frontend/src/lib.rs | 4 +- lib/reader/src/parser.rs | 10 ++-- lib/wasm/src/environ/dummy.rs | 8 +-- lib/wasm/src/environ/spec.rs | 2 +- lib/wasm/src/func_translator.rs | 6 +- 11 files changed, 56 insertions(+), 55 deletions(-) rename lib/cretonne/src/ir/{funcname.rs => extname.rs} (62%) diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index c82ff53bec..4403b6ebef 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -5,7 +5,7 @@ //! //! This module declares the data types used to represent external functions and call signatures. -use ir::{Type, FunctionName, SigRef, ArgumentLoc}; +use ir::{Type, ExternalName, SigRef, ArgumentLoc}; use isa::{RegInfo, RegUnit}; use std::cmp; use std::fmt; @@ -323,7 +323,7 @@ impl FromStr for ArgumentPurpose { #[derive(Clone, Debug)] pub struct ExtFuncData { /// Name of the external function. - pub name: FunctionName, + pub name: ExternalName, /// Call signature of function. pub signature: SigRef, } diff --git a/lib/cretonne/src/ir/funcname.rs b/lib/cretonne/src/ir/extname.rs similarity index 62% rename from lib/cretonne/src/ir/funcname.rs rename to lib/cretonne/src/ir/extname.rs index 0cffae6d7c..bd8c4e631e 100644 --- a/lib/cretonne/src/ir/funcname.rs +++ b/lib/cretonne/src/ir/extname.rs @@ -1,39 +1,40 @@ -//! Function names. +//! External names. //! -//! The name of a function doesn't have any meaning to Cretonne which compiles functions -//! independently. +//! These are identifiers for declaring entities defined outside the current +//! function. The name of an external declaration doesn't have any meaning to +//! Cretonne, which compiles functions independently. use std::fmt::{self, Write}; use std::ascii::AsciiExt; -/// The name of a function can be any sequence of bytes. +/// The name of an external can be any sequence of bytes. /// -/// Function names are primarily used as keys by code using Cretonne to map -/// from a cretonne::ir::Function to additional associated data. +/// External names are primarily used as keys by code using Cretonne to map +/// from a cretonne::ir::FuncRef or similar to additional associated data. /// -/// Function names can also serve as a primitive testing and debugging tool. +/// External names can also serve as a primitive testing and debugging tool. /// In particular, many `.cton` test files use function names to identify /// functions. #[derive(Debug, Clone, PartialEq, Eq, Default)] -pub struct FunctionName(NameRepr); +pub struct ExternalName(NameRepr); -impl FunctionName { - /// Creates a new function name from a sequence of bytes. +impl ExternalName { + /// Creates a new external name from a sequence of bytes. /// /// # Examples /// /// ```rust - /// # use cretonne::ir::FunctionName; - /// // Create `FunctionName` from a string. - /// let name = FunctionName::new("hello"); + /// # use cretonne::ir::ExternalName; + /// // Create `ExternalName` from a string. + /// let name = ExternalName::new("hello"); /// assert_eq!(name.to_string(), "%hello"); /// - /// // Create `FunctionName` from a sequence of bytes. + /// // Create `ExternalName` from a sequence of bytes. /// let bytes: &[u8] = &[10, 9, 8]; - /// let name = FunctionName::new(bytes); + /// let name = ExternalName::new(bytes); /// assert_eq!(name.to_string(), "#0a0908"); /// ``` - pub fn new(v: T) -> FunctionName + pub fn new(v: T) -> ExternalName where T: Into>, { @@ -43,12 +44,12 @@ impl FunctionName { for (i, &byte) in vec.iter().enumerate() { bytes[i] = byte; } - FunctionName(NameRepr::Short { + ExternalName(NameRepr::Short { length: vec.len() as u8, bytes: bytes, }) } else { - FunctionName(NameRepr::Long(vec)) + ExternalName(NameRepr::Long(vec)) } } } @@ -86,7 +87,7 @@ impl AsRef<[u8]> for NameRepr { } } -impl AsRef<[u8]> for FunctionName { +impl AsRef<[u8]> for ExternalName { fn as_ref(&self) -> &[u8] { self.0.as_ref() } @@ -101,7 +102,7 @@ impl Default for NameRepr { } } -impl fmt::Display for FunctionName { +impl fmt::Display for ExternalName { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { if let Some(name) = try_as_name(self.0.as_ref()) { write!(f, "%{}", name) @@ -117,24 +118,24 @@ impl fmt::Display for FunctionName { #[cfg(test)] mod tests { - use super::FunctionName; + use super::ExternalName; #[test] fn displaying() { - assert_eq!(FunctionName::new("").to_string(), "%"); - assert_eq!(FunctionName::new("x").to_string(), "%x"); - assert_eq!(FunctionName::new("x_1").to_string(), "%x_1"); - assert_eq!(FunctionName::new(" ").to_string(), "#20"); + assert_eq!(ExternalName::new("").to_string(), "%"); + assert_eq!(ExternalName::new("x").to_string(), "%x"); + assert_eq!(ExternalName::new("x_1").to_string(), "%x_1"); + assert_eq!(ExternalName::new(" ").to_string(), "#20"); assert_eq!( - FunctionName::new("кретон").to_string(), + ExternalName::new("кретон").to_string(), "#d0bad180d0b5d182d0bed0bd" ); assert_eq!( - FunctionName::new("印花棉布").to_string(), + ExternalName::new("印花棉布").to_string(), "#e58db0e88ab1e6a389e5b883" ); assert_eq!( - FunctionName::new(vec![0, 1, 2, 3, 4, 5]).to_string(), + ExternalName::new(vec![0, 1, 2, 3, 4, 5]).to_string(), "#000102030405" ); } diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index dac951ec99..4c348b6ce9 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -5,7 +5,7 @@ use entity::{PrimaryMap, EntityMap}; use ir; -use ir::{FunctionName, CallConv, Signature, DataFlowGraph, Layout}; +use ir::{ExternalName, CallConv, Signature, DataFlowGraph, Layout}; use ir::{InstEncodings, ValueLocations, JumpTables, StackSlots, EbbOffsets, SourceLocs}; use ir::{Ebb, JumpTableData, JumpTable, StackSlotData, StackSlot, SigRef, ExtFuncData, FuncRef, GlobalVarData, GlobalVar, HeapData, Heap}; @@ -20,7 +20,7 @@ use write::write_function; #[derive(Clone)] pub struct Function { /// Name of this function. Mostly used by `.cton` files. - pub name: FunctionName, + pub name: ExternalName, /// Signature of this function. pub signature: Signature, @@ -66,7 +66,7 @@ pub struct Function { impl Function { /// Create a function with the given name and signature. - pub fn with_name_signature(name: FunctionName, sig: Signature) -> Function { + pub fn with_name_signature(name: ExternalName, sig: Signature) -> Function { Function { name, signature: sig, @@ -100,7 +100,7 @@ impl Function { /// Create a new empty, anonymous function with a native calling convention. pub fn new() -> Function { - Self::with_name_signature(FunctionName::default(), Signature::new(CallConv::Native)) + Self::with_name_signature(ExternalName::default(), Signature::new(CallConv::Native)) } /// Creates a jump table in the function, to be used by `br_table` instructions. diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index bb9730595f..3444a52dc6 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -12,7 +12,7 @@ pub mod layout; pub mod function; mod builder; mod extfunc; -mod funcname; +mod extname; mod globalvar; mod heap; mod memflags; @@ -26,7 +26,7 @@ pub use ir::dfg::{DataFlowGraph, ValueDef}; pub use ir::entities::{Ebb, Inst, Value, StackSlot, GlobalVar, JumpTable, FuncRef, SigRef, Heap}; pub use ir::extfunc::{Signature, CallConv, AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData}; -pub use ir::funcname::FunctionName; +pub use ir::extname::ExternalName; pub use ir::function::Function; pub use ir::globalvar::GlobalVarData; pub use ir::heap::{HeapData, HeapStyle, HeapBase}; diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 7068569f27..b4276349de 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -441,7 +441,7 @@ impl<'a> fmt::Display for DisplayValues<'a> { #[cfg(test)] mod tests { - use ir::{Function, FunctionName, StackSlotData, StackSlotKind}; + use ir::{Function, ExternalName, StackSlotData, StackSlotKind}; use ir::types; #[test] @@ -449,7 +449,7 @@ mod tests { let mut f = Function::new(); assert_eq!(f.to_string(), "function %() native {\n}\n"); - f.name = FunctionName::new("foo"); + f.name = ExternalName::new("foo"); assert_eq!(f.to_string(), "function %foo() native {\n}\n"); f.create_stack_slot(StackSlotData::new(StackSlotKind::Local, 4)); diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 92eca41dcc..80d9768b67 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -653,7 +653,7 @@ where mod tests { use cretonne::entity::EntityRef; - use cretonne::ir::{FunctionName, Function, CallConv, Signature, AbiParam, InstBuilder}; + use cretonne::ir::{ExternalName, Function, CallConv, Signature, AbiParam, InstBuilder}; use cretonne::ir::types::*; use frontend::{ILBuilder, FunctionBuilder}; use cretonne::verifier::verify_function; @@ -687,7 +687,7 @@ mod tests { sig.params.push(AbiParam::new(I32)); let mut il_builder = ILBuilder::::new(); - let mut func = Function::with_name_signature(FunctionName::new("sample_function"), sig); + let mut func = Function::with_name_signature(ExternalName::new("sample_function"), sig); { let mut builder = FunctionBuilder::::new(&mut func, &mut il_builder); diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index bca94f842c..bcab9c2d61 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -36,7 +36,7 @@ //! extern crate cton_frontend; //! //! use cretonne::entity::EntityRef; -//! use cretonne::ir::{FunctionName, CallConv, Function, Signature, AbiParam, InstBuilder}; +//! use cretonne::ir::{ExternalName, CallConv, Function, Signature, AbiParam, InstBuilder}; //! use cretonne::ir::types::*; //! use cretonne::settings; //! use cton_frontend::{ILBuilder, FunctionBuilder}; @@ -67,7 +67,7 @@ //! sig.returns.push(AbiParam::new(I32)); //! sig.params.push(AbiParam::new(I32)); //! let mut il_builder = ILBuilder::::new(); -//! let mut func = Function::with_name_signature(FunctionName::new("sample_function"), sig); +//! let mut func = Function::with_name_signature(ExternalName::new("sample_function"), sig); //! { //! let mut builder = FunctionBuilder::::new(&mut func, &mut il_builder); //! diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index cc2d65996b..2162b167c5 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -9,7 +9,7 @@ use std::collections::HashMap; use std::str::FromStr; use std::{u16, u32}; use std::mem; -use cretonne::ir::{Function, Ebb, Opcode, Value, Type, FunctionName, CallConv, StackSlotData, +use cretonne::ir::{Function, Ebb, Opcode, Value, Type, ExternalName, CallConv, StackSlotData, JumpTable, JumpTableData, Signature, AbiParam, ArgumentExtension, ExtFuncData, SigRef, FuncRef, StackSlot, ValueLoc, ArgumentLoc, MemFlags, GlobalVar, GlobalVarData, Heap, HeapData, HeapStyle, HeapBase}; @@ -872,7 +872,7 @@ impl<'a> Parser<'a> { fn parse_function_spec( &mut self, unique_isa: Option<&TargetIsa>, - ) -> Result<(Location, FunctionName, Signature)> { + ) -> Result<(Location, ExternalName, Signature)> { self.match_identifier("function", "expected 'function'")?; let location = self.loc; @@ -889,11 +889,11 @@ impl<'a> Parser<'a> { // // function ::= "function" * name signature { ... } // - fn parse_function_name(&mut self) -> Result { + fn parse_function_name(&mut self) -> Result { match self.token() { Some(Token::Name(s)) => { self.consume(); - Ok(FunctionName::new(s)) + Ok(ExternalName::new(s)) } Some(Token::HexSequence(s)) => { if s.len() % 2 != 0 { @@ -910,7 +910,7 @@ impl<'a> Parser<'a> { i += 2; } self.consume(); - Ok(FunctionName::new(bin_name)) + Ok(ExternalName::new(bin_name)) } _ => err!(self.loc, "expected function name"), } diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 5900b32ea2..64fb19e262 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -9,9 +9,9 @@ use cretonne::settings; use wasmparser; use std::error::Error; -/// Compute a `ir::FunctionName` for a given wasm function index. -fn get_func_name(func_index: FunctionIndex) -> ir::FunctionName { - ir::FunctionName::new(format!("wasm_0x{:x}", func_index)) +/// Compute a `ir::ExternalName` for a given wasm function index. +fn get_func_name(func_index: FunctionIndex) -> ir::ExternalName { + ir::ExternalName::new(format!("wasm_0x{:x}", func_index)) } /// A collection of names under which a given entity is exported. @@ -196,7 +196,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ } impl<'data> ModuleEnvironment<'data> for DummyEnvironment { - fn get_func_name(&self, func_index: FunctionIndex) -> ir::FunctionName { + fn get_func_name(&self, func_index: FunctionIndex) -> ir::ExternalName { get_func_name(func_index) } diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index c4de95a4f9..31e4782918 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -151,7 +151,7 @@ pub trait FuncEnvironment { /// by the user, they are only for `cretonne-wasm` internal use. pub trait ModuleEnvironment<'data> { /// Return the name for the given function index. - fn get_func_name(&self, func_index: FunctionIndex) -> ir::FunctionName; + fn get_func_name(&self, func_index: FunctionIndex) -> ir::ExternalName; /// Declares a function signature to the environment. fn declare_signature(&mut self, sig: &ir::Signature); diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 2de7d38191..9723f22ca5 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -255,7 +255,7 @@ mod tests { let runtime = DummyEnvironment::default(); let mut ctx = Context::new(); - ctx.func.name = ir::FunctionName::new("small1"); + ctx.func.name = ir::ExternalName::new("small1"); ctx.func.signature.params.push(ir::AbiParam::new(I32)); ctx.func.signature.returns.push(ir::AbiParam::new(I32)); @@ -286,7 +286,7 @@ mod tests { let runtime = DummyEnvironment::default(); let mut ctx = Context::new(); - ctx.func.name = ir::FunctionName::new("small2"); + ctx.func.name = ir::ExternalName::new("small2"); ctx.func.signature.params.push(ir::AbiParam::new(I32)); ctx.func.signature.returns.push(ir::AbiParam::new(I32)); @@ -326,7 +326,7 @@ mod tests { let runtime = DummyEnvironment::default(); let mut ctx = Context::new(); - ctx.func.name = ir::FunctionName::new("infloop"); + ctx.func.name = ir::ExternalName::new("infloop"); ctx.func.signature.returns.push(ir::AbiParam::new(I32)); trans From 6fc45b070a4fb39713ece40203c9cfe6d73909c7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 27 Oct 2017 13:27:10 -0700 Subject: [PATCH 1309/3084] Add a new kind of GlobalVar for symbolic addresses. These addresses will allow referencing C/C++/Rust-style global variables by name directly. --- cranelift/docs/langref.rst | 10 ++++++++++ cranelift/filetests/parser/memory.cton | 14 ++++++++++++++ lib/cretonne/src/ir/globalvar.rs | 11 ++++++++++- lib/cretonne/src/legalizer/globalvar.rs | 1 + lib/reader/src/parser.rs | 19 +++++++++++++------ 5 files changed, 48 insertions(+), 7 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index fb9f35fd94..b5471d2a0c 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -563,6 +563,16 @@ runtime data structures. variable. :result GV: Global variable. +.. inst:: GV = globalsym name + + Declare a global variable at a symbolic address. + + The address of GV is symbolic and will be assigned a relocation, so that + it can be resolved by a later linking phase. + + :arg name: External name. + :result GV: Global variable. + .. autoinst:: global_addr diff --git a/cranelift/filetests/parser/memory.cton b/cranelift/filetests/parser/memory.cton index fc26169961..6fa594c4c1 100644 --- a/cranelift/filetests/parser/memory.cton +++ b/cranelift/filetests/parser/memory.cton @@ -36,6 +36,20 @@ ebb0: return v1 } +function %sym() -> i32 { + gv0 = globalsym %something + ; check: $gv0 = globalsym %something + gv1 = globalsym #d0bad180d0b5d182d0bed0bd + ; check: $gv1 = globalsym #d0bad180d0b5d182d0bed0bd +ebb0: + v0 = global_addr.i32 gv0 + ; check: $v0 = global_addr.i32 $gv0 + v1 = global_addr.i32 gv1 + ; check: $v1 = global_addr.i32 $gv1 + v2 = bxor v0, v1 + return v2 +} + ; Declare static heaps. function %sheap(i32) -> i64 { heap1 = static reserved_reg, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000 diff --git a/lib/cretonne/src/ir/globalvar.rs b/lib/cretonne/src/ir/globalvar.rs index 110cf687ce..1e7fa2efe0 100644 --- a/lib/cretonne/src/ir/globalvar.rs +++ b/lib/cretonne/src/ir/globalvar.rs @@ -1,6 +1,6 @@ //! Global variables. -use ir::GlobalVar; +use ir::{ExternalName, GlobalVar}; use ir::immediates::Offset32; use std::fmt; @@ -25,6 +25,14 @@ pub enum GlobalVarData { /// Byte offset to be added to the pointer loaded from `base`. offset: Offset32, }, + + /// Variable is at an address identified by a symbolic name. Cretonne itself + /// does not interpret this name; it's used by embedders to link with other + /// data structures. + Sym { + /// The symbolic name. + name: ExternalName, + }, } impl fmt::Display for GlobalVarData { @@ -32,6 +40,7 @@ impl fmt::Display for GlobalVarData { match *self { GlobalVarData::VmCtx { offset } => write!(f, "vmctx{}", offset), GlobalVarData::Deref { base, offset } => write!(f, "deref({}){}", base, offset), + GlobalVarData::Sym { ref name } => write!(f, "globalsym {}", name), } } } diff --git a/lib/cretonne/src/legalizer/globalvar.rs b/lib/cretonne/src/legalizer/globalvar.rs index c51a8b8cfc..73855f4410 100644 --- a/lib/cretonne/src/legalizer/globalvar.rs +++ b/lib/cretonne/src/legalizer/globalvar.rs @@ -21,6 +21,7 @@ pub fn expand_global_addr(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut Co match func.global_vars[gv] { ir::GlobalVarData::VmCtx { offset } => vmctx_addr(inst, func, offset.into()), ir::GlobalVarData::Deref { base, offset } => deref_addr(inst, func, base, offset.into()), + ir::GlobalVarData::Sym { .. } => (), } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 2162b167c5..dd08b0ddcb 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -877,7 +877,7 @@ impl<'a> Parser<'a> { let location = self.loc; // function-spec ::= "function" * name signature - let name = self.parse_function_name()?; + let name = self.parse_external_name()?; // function-spec ::= "function" name * signature let sig = self.parse_signature(unique_isa)?; @@ -885,11 +885,13 @@ impl<'a> Parser<'a> { Ok((location, name, sig)) } - // Parse a function name. + // Parse an external name. + // + // For example, in a function spec, the parser would be in this state: // // function ::= "function" * name signature { ... } // - fn parse_function_name(&mut self) -> Result { + fn parse_external_name(&mut self) -> Result { match self.token() { Some(Token::Name(s)) => { self.consume(); @@ -899,7 +901,7 @@ impl<'a> Parser<'a> { if s.len() % 2 != 0 { return err!( self.loc, - "expected binary function name to have length multiple of two" + "expected binary external name to have length multiple of two" ); } let mut bin_name = Vec::with_capacity(s.len() / 2); @@ -912,7 +914,7 @@ impl<'a> Parser<'a> { self.consume(); Ok(ExternalName::new(bin_name)) } - _ => err!(self.loc, "expected function name"), + _ => err!(self.loc, "expected external name"), } } @@ -1142,6 +1144,7 @@ impl<'a> Parser<'a> { // global-var-decl ::= * GlobalVar(gv) "=" global-var-desc // global-var-desc ::= "vmctx" offset32 // | "deref" "(" GlobalVar(base) ")" offset32 + // | "globalsym" name // fn parse_global_var_decl(&mut self) -> Result<(u32, GlobalVarData)> { let number = self.match_gv("expected global variable number: gv«n»")?; @@ -1168,6 +1171,10 @@ impl<'a> Parser<'a> { let offset = self.optional_offset32()?; GlobalVarData::Deref { base, offset } } + "globalsym" => { + let name = self.parse_external_name()?; + GlobalVarData::Sym { name } + } other => return err!(self.loc, "Unknown global variable kind '{}'", other), }; @@ -1292,7 +1299,7 @@ impl<'a> Parser<'a> { Some(Token::SigRef(sig_src)) => { let sig = ctx.get_sig(sig_src, &self.loc)?; self.consume(); - let name = self.parse_function_name()?; + let name = self.parse_external_name()?; ExtFuncData { name, signature: sig, From cb805f704dc8b2d10bb173beb43623f7600bab40 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sat, 28 Oct 2017 07:12:46 -0700 Subject: [PATCH 1310/3084] Put BaldrMonkey-specific behavior under a setting. BaldrMonkey will need to enable allones_funcaddrs. --- .../isa/intel/allones_funcaddrs32.cton | 25 +++++++++++++++++ .../isa/intel/allones_funcaddrs64.cton | 28 +++++++++++++++++++ cranelift/filetests/isa/intel/binary32.cton | 4 +-- cranelift/filetests/isa/intel/binary64.cton | 6 ++-- lib/cretonne/meta/base/settings.py | 8 ++++++ lib/cretonne/meta/isa/intel/encodings.py | 14 ++++++++-- lib/cretonne/meta/isa/intel/recipes.py | 22 +++++++++++++-- lib/cretonne/src/settings.rs | 3 +- 8 files changed, 99 insertions(+), 11 deletions(-) create mode 100644 cranelift/filetests/isa/intel/allones_funcaddrs32.cton create mode 100644 cranelift/filetests/isa/intel/allones_funcaddrs64.cton diff --git a/cranelift/filetests/isa/intel/allones_funcaddrs32.cton b/cranelift/filetests/isa/intel/allones_funcaddrs32.cton new file mode 100644 index 0000000000..0c769d45ec --- /dev/null +++ b/cranelift/filetests/isa/intel/allones_funcaddrs32.cton @@ -0,0 +1,25 @@ +; binary emission of 32-bit code. +test binemit +set is_compressed +set allones_funcaddrs +isa intel haswell + +; The binary encodings can be verified with the command: +; +; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/allones_funcaddrs32.cton | llvm-mc -show-encoding -triple=i386 +; + +; Tests from binary32.cton affected by allones_funcaddrs. +function %I32() { + fn0 = function %foo() + sig0 = () + +ebb0: + + ; asm: movl $-1, %ecx + [-,%rcx] v400 = func_addr.i32 fn0 ; bin: b9 Abs4(fn0) ffffffff + ; asm: movl $-1, %esi + [-,%rsi] v401 = func_addr.i32 fn0 ; bin: be Abs4(fn0) ffffffff + + return ; bin: c3 +} diff --git a/cranelift/filetests/isa/intel/allones_funcaddrs64.cton b/cranelift/filetests/isa/intel/allones_funcaddrs64.cton new file mode 100644 index 0000000000..5670c6ee2e --- /dev/null +++ b/cranelift/filetests/isa/intel/allones_funcaddrs64.cton @@ -0,0 +1,28 @@ +; binary emission of 64-bit code. +test binemit +set is_64bit +set is_compressed +set allones_funcaddrs +isa intel haswell + +; The binary encodings can be verified with the command: +; +; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/allones_funcaddrs64.cton | llvm-mc -show-encoding -triple=x86_64 +; + +; Tests from binary64.cton affected by allones_funcaddrs. +function %I64() { + fn0 = function %foo() + sig0 = () + +ebb0: + + ; asm: movabsq $-1, %rcx + [-,%rcx] v400 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(fn0) ffffffffffffffff + ; asm: movabsq $-1, %rsi + [-,%rsi] v401 = func_addr.i64 fn0 ; bin: 48 be Abs8(fn0) ffffffffffffffff + ; asm: movabsq $-1, %r10 + [-,%r10] v402 = func_addr.i64 fn0 ; bin: 49 ba Abs8(fn0) ffffffffffffffff + + return ; bin: c3 +} diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 51e739483f..ab0f060488 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -343,9 +343,9 @@ ebb0: call fn0() ; bin: e8 PCRel4(fn0) 00000000 ; asm: movl $-1, %ecx - [-,%rcx] v400 = func_addr.i32 fn0 ; bin: b9 Abs4(fn0) ffffffff + [-,%rcx] v400 = func_addr.i32 fn0 ; bin: b9 Abs4(fn0) 00000000 ; asm: movl $-1, %esi - [-,%rsi] v401 = func_addr.i32 fn0 ; bin: be Abs4(fn0) ffffffff + [-,%rsi] v401 = func_addr.i32 fn0 ; bin: be Abs4(fn0) 00000000 ; asm: call *%ecx call_indirect sig0, v400() ; bin: ff d1 diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 6d062c29b6..a770416e30 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -430,11 +430,11 @@ ebb0: call fn0() ; bin: e8 PCRel4(fn0) 00000000 ; asm: movabsq $-1, %rcx - [-,%rcx] v400 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(fn0) ffffffffffffffff + [-,%rcx] v400 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(fn0) 0000000000000000 ; asm: movabsq $-1, %rsi - [-,%rsi] v401 = func_addr.i64 fn0 ; bin: 48 be Abs8(fn0) ffffffffffffffff + [-,%rsi] v401 = func_addr.i64 fn0 ; bin: 48 be Abs8(fn0) 0000000000000000 ; asm: movabsq $-1, %r10 - [-,%r10] v402 = func_addr.i64 fn0 ; bin: 49 ba Abs8(fn0) ffffffffffffffff + [-,%r10] v402 = func_addr.i64 fn0 ; bin: 49 ba Abs8(fn0) 0000000000000000 ; asm: call *%rcx call_indirect sig0, v400() ; bin: ff d1 diff --git a/lib/cretonne/meta/base/settings.py b/lib/cretonne/meta/base/settings.py index 434bd28a52..11f0728e11 100644 --- a/lib/cretonne/meta/base/settings.py +++ b/lib/cretonne/meta/base/settings.py @@ -69,5 +69,13 @@ spiderwasm_prologue_words = NumSetting( pushed return address on Intel ISAs. """) +# +# BaldrMonkey requires that not-yet-relocated function addresses be encoded +# as all-ones bitpatterns. +# +allones_funcaddrs = BoolSetting( + """ + Emit not-yet-relocated function addresses as all-ones bit patterns. + """) group.close(globals()) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index bdbb61fc05..9ae4601a57 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -2,7 +2,7 @@ Intel Encodings. """ from __future__ import absolute_import -from cdsl.predicates import IsUnsignedInt +from cdsl.predicates import IsUnsignedInt, Not from base import instructions as base from base.formats import UnaryImm from .defs import I32, I64 @@ -11,6 +11,7 @@ from . import settings as cfg from . import instructions as x86 from .legalize import intel_expand from base.legalize import narrow, expand +from base.settings import allones_funcaddrs from .settings import use_sse41 try: @@ -260,8 +261,15 @@ enc_both(base.regspill.f64, r.frsp32, 0x66, 0x0f, 0xd6) # Function addresses. # -I32.enc(base.func_addr.i32, *r.fnaddr4(0xb8)) -I64.enc(base.func_addr.i64, *r.fnaddr8.rex(0xb8, w=1)) +I32.enc(base.func_addr.i32, *r.fnaddr4(0xb8), + isap=Not(allones_funcaddrs)) +I64.enc(base.func_addr.i64, *r.fnaddr8.rex(0xb8, w=1), + isap=Not(allones_funcaddrs)) + +I32.enc(base.func_addr.i32, *r.allones_fnaddr4(0xb8), + isap=allones_funcaddrs) +I64.enc(base.func_addr.i64, *r.allones_fnaddr8.rex(0xb8, w=1), + isap=allones_funcaddrs) # # Call/return diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 0aa3f0acd9..89f0e00ac0 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -478,8 +478,7 @@ fnaddr4 = TailRecipe( emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); sink.reloc_func(RelocKind::Abs4.into(), func_ref); - // Write the immediate as `!0` for the benefit of BaldrMonkey. - sink.put4(!0); + sink.put4(0); ''') # XX+rd iq with Abs8 function relocation. @@ -488,6 +487,25 @@ fnaddr8 = TailRecipe( emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); sink.reloc_func(RelocKind::Abs8.into(), func_ref); + sink.put8(0); + ''') + +# Similar to fnaddr4, but writes !0 (this is used by BaldrMonkey). +allones_fnaddr4 = TailRecipe( + 'allones_fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, + emit=''' + PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); + sink.reloc_func(RelocKind::Abs4.into(), func_ref); + // Write the immediate as `!0` for the benefit of BaldrMonkey. + sink.put4(!0); + ''') + +# Similar to fnaddr8, but writes !0 (this is used by BaldrMonkey). +allones_fnaddr8 = TailRecipe( + 'allones_fnaddr8', FuncAddr, size=8, ins=(), outs=GPR, + emit=''' + PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); + sink.reloc_func(RelocKind::Abs8.into(), func_ref); // Write the immediate as `!0` for the benefit of BaldrMonkey. sink.put8(!0); ''') diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 67ee68a214..d04dd4991f 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -352,7 +352,8 @@ mod tests { enable_float = true\n\ enable_simd = true\n\ enable_atomics = true\n\ - spiderwasm_prologue_words = 0\n" + spiderwasm_prologue_words = 0\n\ + allones_funcaddrs = false\n" ); assert_eq!(f.opt_level(), super::OptLevel::Default); assert_eq!(f.enable_simd(), true); From 9c54c3fff066c173a296636a30698afa63351fbb Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 30 Oct 2017 10:02:29 -0700 Subject: [PATCH 1311/3084] Introduce globalsym_addr. This is an instruction used in legalization of GlobalVarData::Sym global variables. --- cranelift/filetests/isa/intel/binary32.cton | 11 ++++++++-- cranelift/filetests/isa/intel/binary64.cton | 15 +++++++++++--- .../filetests/isa/intel/legalize-memory.cton | 13 ++++++++++++ cranelift/src/filetest/binemit.rs | 4 ++++ cranelift/src/filetest/compile.rs | 1 + lib/cretonne/meta/base/instructions.py | 8 ++++++++ lib/cretonne/meta/isa/intel/encodings.py | 7 +++++++ lib/cretonne/meta/isa/intel/recipes.py | 20 ++++++++++++++++++- lib/cretonne/src/binemit/memorysink.rs | 12 ++++++++++- lib/cretonne/src/binemit/mod.rs | 6 +++++- lib/cretonne/src/legalizer/globalvar.rs | 8 +++++++- 11 files changed, 96 insertions(+), 9 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index ab0f060488..c37079b063 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -12,6 +12,8 @@ function %I32() { fn0 = function %foo() sig0 = () + gv0 = globalsym %some_gv + ss0 = incoming_arg 8, offset 0 ss1 = incoming_arg 1024, offset -1024 ss2 = incoming_arg 1024, offset -2048 @@ -342,9 +344,9 @@ ebb0: ; asm: call foo call fn0() ; bin: e8 PCRel4(fn0) 00000000 - ; asm: movl $-1, %ecx + ; asm: movl $0, %ecx [-,%rcx] v400 = func_addr.i32 fn0 ; bin: b9 Abs4(fn0) 00000000 - ; asm: movl $-1, %esi + ; asm: movl $0, %esi [-,%rsi] v401 = func_addr.i32 fn0 ; bin: be Abs4(fn0) 00000000 ; asm: call *%ecx @@ -352,6 +354,11 @@ ebb0: ; asm: call *%esi call_indirect sig0, v401() ; bin: ff d6 + ; asm: movl $0, %ecx + [-,%rcx] v450 = globalsym_addr.i32 gv0 ; bin: b9 Abs4(gv0) 00000000 + ; asm: movl $0, %esi + [-,%rsi] v451 = globalsym_addr.i32 gv0 ; bin: be Abs4(gv0) 00000000 + ; Spill / Fill. ; asm: movl %ecx, 1032(%esp) diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index a770416e30..7ad9d47c4d 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -14,6 +14,8 @@ function %I64() { fn0 = function %foo() sig0 = () + gv0 = globalsym %some_gv + ; Use incoming_arg stack slots because they won't be relocated by the frame ; layout. ss0 = incoming_arg 8, offset 0 @@ -429,11 +431,11 @@ ebb0: ; asm: call foo call fn0() ; bin: e8 PCRel4(fn0) 00000000 - ; asm: movabsq $-1, %rcx + ; asm: movabsq $0, %rcx [-,%rcx] v400 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(fn0) 0000000000000000 - ; asm: movabsq $-1, %rsi + ; asm: movabsq $0, %rsi [-,%rsi] v401 = func_addr.i64 fn0 ; bin: 48 be Abs8(fn0) 0000000000000000 - ; asm: movabsq $-1, %r10 + ; asm: movabsq $0, %r10 [-,%r10] v402 = func_addr.i64 fn0 ; bin: 49 ba Abs8(fn0) 0000000000000000 ; asm: call *%rcx @@ -443,6 +445,13 @@ ebb0: ; asm: call *%r10 call_indirect sig0, v402() ; bin: 41 ff d2 + ; asm: movabsq $-1, %rcx + [-,%rcx] v450 = globalsym_addr.i64 gv0 ; bin: 48 b9 Abs8(gv0) 0000000000000000 + ; asm: movabsq $-1, %rsi + [-,%rsi] v451 = globalsym_addr.i64 gv0 ; bin: 48 be Abs8(gv0) 0000000000000000 + ; asm: movabsq $-1, %r10 + [-,%r10] v452 = globalsym_addr.i64 gv0 ; bin: 49 ba Abs8(gv0) 0000000000000000 + ; Spill / Fill. ; asm: movq %rcx, 1032(%rsp) diff --git a/cranelift/filetests/isa/intel/legalize-memory.cton b/cranelift/filetests/isa/intel/legalize-memory.cton index c7d5ed3617..b9e8e99696 100644 --- a/cranelift/filetests/isa/intel/legalize-memory.cton +++ b/cranelift/filetests/isa/intel/legalize-memory.cton @@ -29,6 +29,19 @@ ebb1(v1: i64): ; check: return $v2 } +function %sym() -> i64 { + gv0 = globalsym %something + gv1 = globalsym #d0bad180d0b5d182d0bed0bd + +ebb1: + v0 = global_addr.i64 gv0 + ; check: $v0 = globalsym_addr.i64 gv0 + v1 = global_addr.i64 gv1 + ; check: $v1 = globalsym_addr.i64 gv1 + v2 = bxor v0, v1 + return v2 +} + ; SpiderMonkey VM-style static 4+2 GB heap. ; This eliminates bounds checks completely for offsets < 2GB. function %staticheap_sm64(i32, i64 vmctx) -> f32 spiderwasm { diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index ae1c741d13..67bdac475c 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -79,6 +79,10 @@ impl binemit::CodeSink for TextSink { write!(self.text, "{}({}) ", self.rnames[reloc.0 as usize], fref).unwrap(); } + fn reloc_globalsym(&mut self, reloc: binemit::Reloc, global: ir::GlobalVar) { + write!(self.text, "{}({}) ", self.rnames[reloc.0 as usize], global).unwrap(); + } + fn reloc_jt(&mut self, reloc: binemit::Reloc, jt: ir::JumpTable) { write!(self.text, "{}({}) ", self.rnames[reloc.0 as usize], jt).unwrap(); } diff --git a/cranelift/src/filetest/compile.rs b/cranelift/src/filetest/compile.rs index 799bcc0a0a..8d6c8e726c 100644 --- a/cranelift/src/filetest/compile.rs +++ b/cranelift/src/filetest/compile.rs @@ -99,5 +99,6 @@ impl binemit::CodeSink for SizeSink { fn reloc_ebb(&mut self, _reloc: binemit::Reloc, _ebb: ir::Ebb) {} fn reloc_func(&mut self, _reloc: binemit::Reloc, _fref: ir::FuncRef) {} + fn reloc_globalsym(&mut self, _reloc: binemit::Reloc, _global: ir::GlobalVar) {} fn reloc_jt(&mut self, _reloc: binemit::Reloc, _jt: ir::JumpTable) {} } diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 76682d2dff..26a6f66077 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -389,6 +389,14 @@ global_addr = Instruction( """, ins=GV, outs=addr) +# A specialized form of global_addr instructions that only handles +# symbolic names. +globalsym_addr = Instruction( + 'globalsym_addr', r""" + Compute the address of global variable GV, which is a symbolic name. + """, + ins=GV, outs=addr) + # # WebAssembly bounds-checked heap accesses. # diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 9ae4601a57..f9c7f041c8 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -271,6 +271,13 @@ I32.enc(base.func_addr.i32, *r.allones_fnaddr4(0xb8), I64.enc(base.func_addr.i64, *r.allones_fnaddr8.rex(0xb8, w=1), isap=allones_funcaddrs) +# +# Global addresses. +# + +I32.enc(base.globalsym_addr.i32, *r.gvaddr4(0xb8)) +I64.enc(base.globalsym_addr.i64, *r.gvaddr8.rex(0xb8, w=1)) + # # Call/return # diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 89f0e00ac0..0a3d05d096 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -9,7 +9,7 @@ from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare, FloatCompare, IntCond, FloatCond from base.formats import Jump, Branch, BranchInt, BranchFloat -from base.formats import Ternary, FuncAddr +from base.formats import Ternary, FuncAddr, UnaryGlobalVar from base.formats import RegMove, RegSpill, RegFill from .registers import GPR, ABCD, FPR, GPR8, FPR8, FLAG, StackGPR32, StackFPR32 from .defs import supported_floatccs @@ -510,6 +510,24 @@ allones_fnaddr8 = TailRecipe( sink.put8(!0); ''') +# XX+rd id with Abs4 globalsym relocation. +gvaddr4 = TailRecipe( + 'gvaddr4', UnaryGlobalVar, size=4, ins=(), outs=GPR, + emit=''' + PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); + sink.reloc_globalsym(RelocKind::Abs4.into(), global_var); + sink.put4(0); + ''') + +# XX+rd iq with Abs8 globalsym relocation. +gvaddr8 = TailRecipe( + 'gvaddr8', UnaryGlobalVar, size=8, ins=(), outs=GPR, + emit=''' + PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); + sink.reloc_globalsym(RelocKind::Abs8.into(), global_var); + sink.put8(0); + ''') + # # Store recipes. # diff --git a/lib/cretonne/src/binemit/memorysink.rs b/lib/cretonne/src/binemit/memorysink.rs index f3fd5cebba..610fd9666d 100644 --- a/lib/cretonne/src/binemit/memorysink.rs +++ b/lib/cretonne/src/binemit/memorysink.rs @@ -14,7 +14,7 @@ //! relocations to a `RelocSink` trait object. Relocations are less frequent than the //! `CodeSink::put*` methods, so the performance impact of the virtual callbacks is less severe. -use ir::{Ebb, FuncRef, JumpTable}; +use ir::{Ebb, FuncRef, GlobalVar, JumpTable}; use super::{CodeSink, CodeOffset, Reloc}; use std::ptr::write_unaligned; @@ -54,6 +54,11 @@ pub trait RelocSink { /// Add a relocation referencing an external function at the current offset. fn reloc_func(&mut self, CodeOffset, Reloc, FuncRef); + /// Add a relocation referencing an external global variable symbol at the + /// current offset. + fn reloc_globalsym(&mut self, CodeOffset, Reloc, GlobalVar); + + /// Add a relocation referencing a jump table. /// Add a relocation referencing a jump table. fn reloc_jt(&mut self, CodeOffset, Reloc, JumpTable); } @@ -101,6 +106,11 @@ impl<'a> CodeSink for MemoryCodeSink<'a> { self.relocs.reloc_func(ofs, rel, func); } + fn reloc_globalsym(&mut self, rel: Reloc, global: GlobalVar) { + let ofs = self.offset(); + self.relocs.reloc_globalsym(ofs, rel, global); + } + fn reloc_jt(&mut self, rel: Reloc, jt: JumpTable) { let ofs = self.offset(); self.relocs.reloc_jt(ofs, rel, jt); diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 5715b1a86d..0ba7ea5668 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -9,7 +9,7 @@ mod memorysink; pub use self::relaxation::relax_branches; pub use self::memorysink::{MemoryCodeSink, RelocSink}; -use ir::{Ebb, FuncRef, JumpTable, Function, Inst}; +use ir::{Ebb, FuncRef, GlobalVar, JumpTable, Function, Inst}; use regalloc::RegDiversions; /// Offset in bytes from the beginning of the function. @@ -47,6 +47,10 @@ pub trait CodeSink { /// Add a relocation referencing an external function at the current offset. fn reloc_func(&mut self, Reloc, FuncRef); + /// Add a relocation referencing an external global variable symbol at the + /// current offset. This is only used for `GlobalVarData::Sym` globals. + fn reloc_globalsym(&mut self, Reloc, GlobalVar); + /// Add a relocation referencing a jump table. fn reloc_jt(&mut self, Reloc, JumpTable); } diff --git a/lib/cretonne/src/legalizer/globalvar.rs b/lib/cretonne/src/legalizer/globalvar.rs index 73855f4410..717be0d4f0 100644 --- a/lib/cretonne/src/legalizer/globalvar.rs +++ b/lib/cretonne/src/legalizer/globalvar.rs @@ -21,7 +21,7 @@ pub fn expand_global_addr(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut Co match func.global_vars[gv] { ir::GlobalVarData::VmCtx { offset } => vmctx_addr(inst, func, offset.into()), ir::GlobalVarData::Deref { base, offset } => deref_addr(inst, func, base, offset.into()), - ir::GlobalVarData::Sym { .. } => (), + ir::GlobalVarData::Sym { .. } => globalsym(inst, func, gv), } } @@ -50,3 +50,9 @@ fn deref_addr(inst: ir::Inst, func: &mut ir::Function, base: ir::GlobalVar, offs let base_ptr = pos.ins().load(ptr_ty, ir::MemFlags::new(), base_addr, 0); pos.func.dfg.replace(inst).iadd_imm(base_ptr, offset); } + +/// Expand a `global_addr` instruction for a symbolic name global. +fn globalsym(inst: ir::Inst, func: &mut ir::Function, gv: ir::GlobalVar) { + let ptr_ty = func.dfg.value_type(func.dfg.first_result(inst)); + func.dfg.replace(inst).globalsym_addr(ptr_ty, gv); +} From 149a41a684d39b95714271485fac4cd87d1615fe Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 31 Oct 2017 12:19:58 -0700 Subject: [PATCH 1312/3084] Clean up copypasta. --- lib/cretonne/src/binemit/memorysink.rs | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/cretonne/src/binemit/memorysink.rs b/lib/cretonne/src/binemit/memorysink.rs index 610fd9666d..00ac5fe349 100644 --- a/lib/cretonne/src/binemit/memorysink.rs +++ b/lib/cretonne/src/binemit/memorysink.rs @@ -58,7 +58,6 @@ pub trait RelocSink { /// current offset. fn reloc_globalsym(&mut self, CodeOffset, Reloc, GlobalVar); - /// Add a relocation referencing a jump table. /// Add a relocation referencing a jump table. fn reloc_jt(&mut self, CodeOffset, Reloc, JumpTable); } From b60b2ce135d59f2952ea3fc329bf74ba9c493cfe Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 31 Oct 2017 10:51:49 -0700 Subject: [PATCH 1313/3084] Change parse_multiline to follow PEP 257. The main change is that it avoids creating blank lines when processing docstrings. This also adds blank lines in various places to make the generated code prettier. --- lib/cretonne/meta/gen_instr.py | 31 +++++++++++++++++++++++-------- lib/cretonne/meta/gen_types.py | 1 + lib/cretonne/meta/srcgen.py | 32 +++++++++++++++++++++++++------- 3 files changed, 49 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 4c537acf12..2bd34a926c 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -28,11 +28,13 @@ def gen_formats(fmt): # type: (srcgen.Formatter) -> None """Generate an instruction format enumeration""" - fmt.doc_comment('An instruction format') - fmt.doc_comment('') - fmt.doc_comment('Every opcode has a corresponding instruction format') - fmt.doc_comment('which is represented by both the `InstructionFormat`') - fmt.doc_comment('and the `InstructionData` enums.') + fmt.doc_comment(''' + An instruction format + + Every opcode has a corresponding instruction format + which is represented by both the `InstructionFormat` + and the `InstructionData` enums. + ''') fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug)]') with fmt.indented('pub enum InstructionFormat {', '}'): for f in InstructionFormat.all_formats: @@ -126,6 +128,7 @@ def gen_instruction_data_impl(fmt): fmt.line( 'InstructionData::{} {{ opcode, .. }} => opcode,' .format(f.name)) + fmt.line() fmt.doc_comment('Get the controlling type variable operand.') with fmt.indented( @@ -153,18 +156,22 @@ def gen_instruction_data_impl(fmt): n + ' {{ ref args, .. }} => Some(args[{}]),' .format(i)) + fmt.line() fmt.doc_comment( """ Get the value arguments to this instruction. """) gen_arguments_method(fmt, False) + fmt.line() + fmt.doc_comment( """ Get mutable references to the value arguments to this instruction. """) gen_arguments_method(fmt, True) + fmt.line() fmt.doc_comment( """ @@ -184,6 +191,7 @@ def gen_instruction_data_impl(fmt): fmt.line( n + ' { ref mut args, .. } => Some(args.take()),') fmt.line('_ => None,') + fmt.line() fmt.doc_comment( """ @@ -227,9 +235,11 @@ def gen_opcodes(groups, fmt): Return a list of all instructions. """ - fmt.doc_comment('An instruction opcode.') - fmt.doc_comment('') - fmt.doc_comment('All instructions from all supported ISAs are present.') + fmt.doc_comment(''' + An instruction opcode. + + All instructions from all supported ISAs are present. + ''') fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)]') instrs = [] @@ -272,6 +282,8 @@ def gen_opcodes(groups, fmt): i.camel_name, i.name) fmt.line('_ => false,') + fmt.line() + fmt.line() # Generate a private opcode_format table. with fmt.indented( @@ -437,8 +449,10 @@ def gen_type_constraints(fmt, instrs): fmt.line('flags: {:#04x},'.format(flags)) fmt.line('typeset_offset: {},'.format(ctrl_typeset)) fmt.line('constraint_offset: {},'.format(offset)) + fmt.line() gen_typesets_table(fmt, type_sets) + fmt.line() fmt.comment('Table of operand constraint sequences.') with fmt.indented( @@ -668,6 +682,7 @@ def generate(isas, out_dir): fmt = srcgen.Formatter() gen_formats(fmt) gen_instruction_data_impl(fmt) + fmt.line() instrs = gen_opcodes(groups, fmt) gen_type_constraints(fmt, instrs) fmt.update_file('opcodes.rs', out_dir) diff --git a/lib/cretonne/meta/gen_types.py b/lib/cretonne/meta/gen_types.py index c5e36758ba..bc912e78f7 100644 --- a/lib/cretonne/meta/gen_types.py +++ b/lib/cretonne/meta/gen_types.py @@ -28,6 +28,7 @@ def emit_type(ty, fmt): fmt.line( 'pub const {}: Type = Type({:#x});' .format(name, ty.number)) + fmt.line() def emit_vectors(bits, fmt): diff --git a/lib/cretonne/meta/srcgen.py b/lib/cretonne/meta/srcgen.py index f0f7931ea1..9761160487 100644 --- a/lib/cretonne/meta/srcgen.py +++ b/lib/cretonne/meta/srcgen.py @@ -168,12 +168,30 @@ def parse_multiline(s): # type: (str) -> List[str] """ Given a multi-line string, split it into a sequence of lines after - stripping a common indentation. This is useful for strings defined with doc - strings: + stripping a common indentation, as described in the "trim" function + from PEP 257. This is useful for strings defined with doc strings: >>> parse_multiline('\\n hello\\n world\\n') - [None, 'hello', 'world'] + ['hello', 'world'] """ - lines = s.splitlines() - indents = list(i for i in (_indent(l) for l in lines) if i) - indent = min(indents) if indents else 0 - return list(l[indent:] if len(l) > indent else None for l in lines) + if not s: + return [] + # Convert tabs to spaces (following the normal Python rules) + # and split into a list of lines: + lines = s.expandtabs().splitlines() + # Determine minimum indentation (first line doesn't count): + indent = sys.maxsize + for line in lines[1:]: + stripped = line.lstrip() + if stripped: + indent = min(indent, len(line) - len(stripped)) + # Remove indentation (first line is special): + trimmed = [lines[0].strip()] + if indent < sys.maxsize: + for line in lines[1:]: + trimmed.append(line[indent:].rstrip()) + # Strip off trailing and leading blank lines: + while trimmed and not trimmed[-1]: + trimmed.pop() + while trimmed and not trimmed[0]: + trimmed.pop(0) + return trimmed From 5d063eb8bc05e735fd04abfdce9d3ddf6e797f13 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 31 Oct 2017 12:06:13 -0700 Subject: [PATCH 1314/3084] Merge reloc_func and reloc_globalsym into reloc_external. --- .../isa/intel/allones_funcaddrs32.cton | 4 ++-- .../isa/intel/allones_funcaddrs64.cton | 6 +++--- cranelift/filetests/isa/intel/binary32.cton | 10 +++++----- cranelift/filetests/isa/intel/binary64.cton | 14 +++++++------- cranelift/filetests/isa/riscv/binary32.cton | 2 +- cranelift/src/filetest/binemit.rs | 13 +++++++------ cranelift/src/filetest/compile.rs | 3 +-- lib/cretonne/meta/isa/intel/recipes.py | 14 +++++++------- lib/cretonne/meta/isa/riscv/recipes.py | 2 +- lib/cretonne/src/binemit/memorysink.rs | 19 +++++-------------- lib/cretonne/src/binemit/mod.rs | 10 +++------- lib/cretonne/src/ir/globalvar.rs | 10 ++++++++++ 12 files changed, 52 insertions(+), 55 deletions(-) diff --git a/cranelift/filetests/isa/intel/allones_funcaddrs32.cton b/cranelift/filetests/isa/intel/allones_funcaddrs32.cton index 0c769d45ec..3e08d876d1 100644 --- a/cranelift/filetests/isa/intel/allones_funcaddrs32.cton +++ b/cranelift/filetests/isa/intel/allones_funcaddrs32.cton @@ -17,9 +17,9 @@ function %I32() { ebb0: ; asm: movl $-1, %ecx - [-,%rcx] v400 = func_addr.i32 fn0 ; bin: b9 Abs4(fn0) ffffffff + [-,%rcx] v400 = func_addr.i32 fn0 ; bin: b9 Abs4(%foo) ffffffff ; asm: movl $-1, %esi - [-,%rsi] v401 = func_addr.i32 fn0 ; bin: be Abs4(fn0) ffffffff + [-,%rsi] v401 = func_addr.i32 fn0 ; bin: be Abs4(%foo) ffffffff return ; bin: c3 } diff --git a/cranelift/filetests/isa/intel/allones_funcaddrs64.cton b/cranelift/filetests/isa/intel/allones_funcaddrs64.cton index 5670c6ee2e..af1988f65c 100644 --- a/cranelift/filetests/isa/intel/allones_funcaddrs64.cton +++ b/cranelift/filetests/isa/intel/allones_funcaddrs64.cton @@ -18,11 +18,11 @@ function %I64() { ebb0: ; asm: movabsq $-1, %rcx - [-,%rcx] v400 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(fn0) ffffffffffffffff + [-,%rcx] v400 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(%foo) ffffffffffffffff ; asm: movabsq $-1, %rsi - [-,%rsi] v401 = func_addr.i64 fn0 ; bin: 48 be Abs8(fn0) ffffffffffffffff + [-,%rsi] v401 = func_addr.i64 fn0 ; bin: 48 be Abs8(%foo) ffffffffffffffff ; asm: movabsq $-1, %r10 - [-,%r10] v402 = func_addr.i64 fn0 ; bin: 49 ba Abs8(fn0) ffffffffffffffff + [-,%r10] v402 = func_addr.i64 fn0 ; bin: 49 ba Abs8(%foo) ffffffffffffffff return ; bin: c3 } diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index c37079b063..81274ab91f 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -342,12 +342,12 @@ ebb0: [-,%rsi] v351 = bint.i32 v301 ; bin: 0f b6 f2 ; asm: call foo - call fn0() ; bin: e8 PCRel4(fn0) 00000000 + call fn0() ; bin: e8 PCRel4(%foo) 00000000 ; asm: movl $0, %ecx - [-,%rcx] v400 = func_addr.i32 fn0 ; bin: b9 Abs4(fn0) 00000000 + [-,%rcx] v400 = func_addr.i32 fn0 ; bin: b9 Abs4(%foo) 00000000 ; asm: movl $0, %esi - [-,%rsi] v401 = func_addr.i32 fn0 ; bin: be Abs4(fn0) 00000000 + [-,%rsi] v401 = func_addr.i32 fn0 ; bin: be Abs4(%foo) 00000000 ; asm: call *%ecx call_indirect sig0, v400() ; bin: ff d1 @@ -355,9 +355,9 @@ ebb0: call_indirect sig0, v401() ; bin: ff d6 ; asm: movl $0, %ecx - [-,%rcx] v450 = globalsym_addr.i32 gv0 ; bin: b9 Abs4(gv0) 00000000 + [-,%rcx] v450 = globalsym_addr.i32 gv0 ; bin: b9 Abs4(%some_gv) 00000000 ; asm: movl $0, %esi - [-,%rsi] v451 = globalsym_addr.i32 gv0 ; bin: be Abs4(gv0) 00000000 + [-,%rsi] v451 = globalsym_addr.i32 gv0 ; bin: be Abs4(%some_gv) 00000000 ; Spill / Fill. diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 7ad9d47c4d..f695d1b227 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -429,14 +429,14 @@ ebb0: [-,%rsi] v351 = bint.i64 v301 ; bin: 0f b6 f2 ; asm: call foo - call fn0() ; bin: e8 PCRel4(fn0) 00000000 + call fn0() ; bin: e8 PCRel4(%foo) 00000000 ; asm: movabsq $0, %rcx - [-,%rcx] v400 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(fn0) 0000000000000000 + [-,%rcx] v400 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(%foo) 0000000000000000 ; asm: movabsq $0, %rsi - [-,%rsi] v401 = func_addr.i64 fn0 ; bin: 48 be Abs8(fn0) 0000000000000000 + [-,%rsi] v401 = func_addr.i64 fn0 ; bin: 48 be Abs8(%foo) 0000000000000000 ; asm: movabsq $0, %r10 - [-,%r10] v402 = func_addr.i64 fn0 ; bin: 49 ba Abs8(fn0) 0000000000000000 + [-,%r10] v402 = func_addr.i64 fn0 ; bin: 49 ba Abs8(%foo) 0000000000000000 ; asm: call *%rcx call_indirect sig0, v400() ; bin: ff d1 @@ -446,11 +446,11 @@ ebb0: call_indirect sig0, v402() ; bin: 41 ff d2 ; asm: movabsq $-1, %rcx - [-,%rcx] v450 = globalsym_addr.i64 gv0 ; bin: 48 b9 Abs8(gv0) 0000000000000000 + [-,%rcx] v450 = globalsym_addr.i64 gv0 ; bin: 48 b9 Abs8(%some_gv) 0000000000000000 ; asm: movabsq $-1, %rsi - [-,%rsi] v451 = globalsym_addr.i64 gv0 ; bin: 48 be Abs8(gv0) 0000000000000000 + [-,%rsi] v451 = globalsym_addr.i64 gv0 ; bin: 48 be Abs8(%some_gv) 0000000000000000 ; asm: movabsq $-1, %r10 - [-,%r10] v452 = globalsym_addr.i64 gv0 ; bin: 49 ba Abs8(gv0) 0000000000000000 + [-,%r10] v452 = globalsym_addr.i64 gv0 ; bin: 49 ba Abs8(%some_gv) 0000000000000000 ; Spill / Fill. diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 1ed2fcabea..5f101cafb8 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -88,7 +88,7 @@ ebb0(v9999: i32): ; Control Transfer Instructions ; jal %x1, fn0 - call fn0() ; bin: Call(fn0) 000000ef + call fn0() ; bin: Call(%foo) 000000ef ; jalr %x1, %x10 call_indirect sig0, v1() ; bin: 000500e7 diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 67bdac475c..7ffef262ac 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -75,12 +75,13 @@ impl binemit::CodeSink for TextSink { write!(self.text, "{}({}) ", self.rnames[reloc.0 as usize], ebb).unwrap(); } - fn reloc_func(&mut self, reloc: binemit::Reloc, fref: ir::FuncRef) { - write!(self.text, "{}({}) ", self.rnames[reloc.0 as usize], fref).unwrap(); - } - - fn reloc_globalsym(&mut self, reloc: binemit::Reloc, global: ir::GlobalVar) { - write!(self.text, "{}({}) ", self.rnames[reloc.0 as usize], global).unwrap(); + fn reloc_external(&mut self, reloc: binemit::Reloc, name: &ir::ExternalName) { + write!( + self.text, + "{}({}) ", + self.rnames[reloc.0 as usize], + name, + ).unwrap(); } fn reloc_jt(&mut self, reloc: binemit::Reloc, jt: ir::JumpTable) { diff --git a/cranelift/src/filetest/compile.rs b/cranelift/src/filetest/compile.rs index 8d6c8e726c..e76bf56743 100644 --- a/cranelift/src/filetest/compile.rs +++ b/cranelift/src/filetest/compile.rs @@ -98,7 +98,6 @@ impl binemit::CodeSink for SizeSink { } fn reloc_ebb(&mut self, _reloc: binemit::Reloc, _ebb: ir::Ebb) {} - fn reloc_func(&mut self, _reloc: binemit::Reloc, _fref: ir::FuncRef) {} - fn reloc_globalsym(&mut self, _reloc: binemit::Reloc, _global: ir::GlobalVar) {} + fn reloc_external(&mut self, _reloc: binemit::Reloc, _name: &ir::ExternalName) {} fn reloc_jt(&mut self, _reloc: binemit::Reloc, _jt: ir::JumpTable) {} } diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 0a3d05d096..a32d3ec8df 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -477,7 +477,7 @@ fnaddr4 = TailRecipe( 'fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_func(RelocKind::Abs4.into(), func_ref); + sink.reloc_external(RelocKind::Abs4.into(), &func.dfg.ext_funcs[func_ref].name); sink.put4(0); ''') @@ -486,7 +486,7 @@ fnaddr8 = TailRecipe( 'fnaddr8', FuncAddr, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_func(RelocKind::Abs8.into(), func_ref); + sink.reloc_external(RelocKind::Abs8.into(), &func.dfg.ext_funcs[func_ref].name); sink.put8(0); ''') @@ -495,7 +495,7 @@ allones_fnaddr4 = TailRecipe( 'allones_fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_func(RelocKind::Abs4.into(), func_ref); + sink.reloc_external(RelocKind::Abs4.into(), &func.dfg.ext_funcs[func_ref].name); // Write the immediate as `!0` for the benefit of BaldrMonkey. sink.put4(!0); ''') @@ -505,7 +505,7 @@ allones_fnaddr8 = TailRecipe( 'allones_fnaddr8', FuncAddr, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_func(RelocKind::Abs8.into(), func_ref); + sink.reloc_external(RelocKind::Abs8.into(), &func.dfg.ext_funcs[func_ref].name); // Write the immediate as `!0` for the benefit of BaldrMonkey. sink.put8(!0); ''') @@ -515,7 +515,7 @@ gvaddr4 = TailRecipe( 'gvaddr4', UnaryGlobalVar, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_globalsym(RelocKind::Abs4.into(), global_var); + sink.reloc_external(RelocKind::Abs4.into(), &func.global_vars[global_var].symbol_name()); sink.put4(0); ''') @@ -524,7 +524,7 @@ gvaddr8 = TailRecipe( 'gvaddr8', UnaryGlobalVar, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_globalsym(RelocKind::Abs8.into(), global_var); + sink.reloc_external(RelocKind::Abs8.into(), &func.global_vars[global_var].symbol_name()); sink.put8(0); ''') @@ -798,7 +798,7 @@ call_id = TailRecipe( 'call_id', Call, size=4, ins=(), outs=(), emit=''' PUT_OP(bits, BASE_REX, sink); - sink.reloc_func(RelocKind::PCRel4.into(), func_ref); + sink.reloc_external(RelocKind::PCRel4.into(), &func.dfg.ext_funcs[func_ref].name); sink.put4(0); ''') diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 8cbbd40a80..bfe30a0025 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -183,7 +183,7 @@ UJ = EncRecipe( UJcall = EncRecipe( 'UJcall', Call, size=4, ins=(), outs=(), emit=''' - sink.reloc_func(RelocKind::Call.into(), func_ref); + sink.reloc_external(RelocKind::Call.into(), &func.dfg.ext_funcs[func_ref].name); // rd=%x1 is the standard link register. put_uj(bits, 0, 1, sink); ''') diff --git a/lib/cretonne/src/binemit/memorysink.rs b/lib/cretonne/src/binemit/memorysink.rs index 00ac5fe349..7bb6de4c31 100644 --- a/lib/cretonne/src/binemit/memorysink.rs +++ b/lib/cretonne/src/binemit/memorysink.rs @@ -14,7 +14,7 @@ //! relocations to a `RelocSink` trait object. Relocations are less frequent than the //! `CodeSink::put*` methods, so the performance impact of the virtual callbacks is less severe. -use ir::{Ebb, FuncRef, GlobalVar, JumpTable}; +use ir::{ExternalName, Ebb, JumpTable}; use super::{CodeSink, CodeOffset, Reloc}; use std::ptr::write_unaligned; @@ -51,12 +51,8 @@ pub trait RelocSink { /// Add a relocation referencing an EBB at the current offset. fn reloc_ebb(&mut self, CodeOffset, Reloc, Ebb); - /// Add a relocation referencing an external function at the current offset. - fn reloc_func(&mut self, CodeOffset, Reloc, FuncRef); - - /// Add a relocation referencing an external global variable symbol at the - /// current offset. - fn reloc_globalsym(&mut self, CodeOffset, Reloc, GlobalVar); + /// Add a relocation referencing an external symbol at the current offset. + fn reloc_external(&mut self, CodeOffset, Reloc, &ExternalName); /// Add a relocation referencing a jump table. fn reloc_jt(&mut self, CodeOffset, Reloc, JumpTable); @@ -100,14 +96,9 @@ impl<'a> CodeSink for MemoryCodeSink<'a> { self.relocs.reloc_ebb(ofs, rel, ebb); } - fn reloc_func(&mut self, rel: Reloc, func: FuncRef) { + fn reloc_external(&mut self, rel: Reloc, name: &ExternalName) { let ofs = self.offset(); - self.relocs.reloc_func(ofs, rel, func); - } - - fn reloc_globalsym(&mut self, rel: Reloc, global: GlobalVar) { - let ofs = self.offset(); - self.relocs.reloc_globalsym(ofs, rel, global); + self.relocs.reloc_external(ofs, rel, name); } fn reloc_jt(&mut self, rel: Reloc, jt: JumpTable) { diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 0ba7ea5668..e9424f829b 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -9,7 +9,7 @@ mod memorysink; pub use self::relaxation::relax_branches; pub use self::memorysink::{MemoryCodeSink, RelocSink}; -use ir::{Ebb, FuncRef, GlobalVar, JumpTable, Function, Inst}; +use ir::{ExternalName, Ebb, JumpTable, Function, Inst}; use regalloc::RegDiversions; /// Offset in bytes from the beginning of the function. @@ -44,12 +44,8 @@ pub trait CodeSink { /// Add a relocation referencing an EBB at the current offset. fn reloc_ebb(&mut self, Reloc, Ebb); - /// Add a relocation referencing an external function at the current offset. - fn reloc_func(&mut self, Reloc, FuncRef); - - /// Add a relocation referencing an external global variable symbol at the - /// current offset. This is only used for `GlobalVarData::Sym` globals. - fn reloc_globalsym(&mut self, Reloc, GlobalVar); + /// Add a relocation referencing an external symbol at the current offset. + fn reloc_external(&mut self, Reloc, &ExternalName); /// Add a relocation referencing a jump table. fn reloc_jt(&mut self, Reloc, JumpTable); diff --git a/lib/cretonne/src/ir/globalvar.rs b/lib/cretonne/src/ir/globalvar.rs index 1e7fa2efe0..78e1c6cb98 100644 --- a/lib/cretonne/src/ir/globalvar.rs +++ b/lib/cretonne/src/ir/globalvar.rs @@ -35,6 +35,16 @@ pub enum GlobalVarData { }, } +impl GlobalVarData { + /// Assume that `self` is an `GlobalVarData::Sym` and return its name. + pub fn symbol_name(&self) -> &ExternalName { + match *self { + GlobalVarData::Sym { ref name } => name, + _ => panic!("only symbols have names"), + } + } +} + impl fmt::Display for GlobalVarData { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match *self { From 871bf95acfa81210fa703458997c70031900dd1a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 31 Oct 2017 12:18:17 -0700 Subject: [PATCH 1315/3084] Change reloc_ebb to pass a CodeOffset rather than an Ebb index. --- cranelift/src/filetest/binemit.rs | 9 +++++++-- cranelift/src/filetest/compile.rs | 2 +- lib/cretonne/src/binemit/memorysink.rs | 8 ++++---- lib/cretonne/src/binemit/mod.rs | 4 ++-- 4 files changed, 14 insertions(+), 9 deletions(-) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 7ffef262ac..cc8dcad0b7 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -71,8 +71,13 @@ impl binemit::CodeSink for TextSink { self.offset += 8; } - fn reloc_ebb(&mut self, reloc: binemit::Reloc, ebb: ir::Ebb) { - write!(self.text, "{}({}) ", self.rnames[reloc.0 as usize], ebb).unwrap(); + fn reloc_ebb(&mut self, reloc: binemit::Reloc, ebb_offset: binemit::CodeOffset) { + write!( + self.text, + "{}({}) ", + self.rnames[reloc.0 as usize], + ebb_offset + ).unwrap(); } fn reloc_external(&mut self, reloc: binemit::Reloc, name: &ir::ExternalName) { diff --git a/cranelift/src/filetest/compile.rs b/cranelift/src/filetest/compile.rs index e76bf56743..3aaa0a6b8a 100644 --- a/cranelift/src/filetest/compile.rs +++ b/cranelift/src/filetest/compile.rs @@ -97,7 +97,7 @@ impl binemit::CodeSink for SizeSink { self.offset += 8; } - fn reloc_ebb(&mut self, _reloc: binemit::Reloc, _ebb: ir::Ebb) {} + fn reloc_ebb(&mut self, _reloc: binemit::Reloc, _ebb_offset: binemit::CodeOffset) {} fn reloc_external(&mut self, _reloc: binemit::Reloc, _name: &ir::ExternalName) {} fn reloc_jt(&mut self, _reloc: binemit::Reloc, _jt: ir::JumpTable) {} } diff --git a/lib/cretonne/src/binemit/memorysink.rs b/lib/cretonne/src/binemit/memorysink.rs index 7bb6de4c31..92766b2d56 100644 --- a/lib/cretonne/src/binemit/memorysink.rs +++ b/lib/cretonne/src/binemit/memorysink.rs @@ -14,7 +14,7 @@ //! relocations to a `RelocSink` trait object. Relocations are less frequent than the //! `CodeSink::put*` methods, so the performance impact of the virtual callbacks is less severe. -use ir::{ExternalName, Ebb, JumpTable}; +use ir::{ExternalName, JumpTable}; use super::{CodeSink, CodeOffset, Reloc}; use std::ptr::write_unaligned; @@ -49,7 +49,7 @@ impl<'a> MemoryCodeSink<'a> { /// A trait for receiving relocations for code that is emitted directly into memory. pub trait RelocSink { /// Add a relocation referencing an EBB at the current offset. - fn reloc_ebb(&mut self, CodeOffset, Reloc, Ebb); + fn reloc_ebb(&mut self, CodeOffset, Reloc, CodeOffset); /// Add a relocation referencing an external symbol at the current offset. fn reloc_external(&mut self, CodeOffset, Reloc, &ExternalName); @@ -91,9 +91,9 @@ impl<'a> CodeSink for MemoryCodeSink<'a> { self.offset += 8; } - fn reloc_ebb(&mut self, rel: Reloc, ebb: Ebb) { + fn reloc_ebb(&mut self, rel: Reloc, ebb_offset: CodeOffset) { let ofs = self.offset(); - self.relocs.reloc_ebb(ofs, rel, ebb); + self.relocs.reloc_ebb(ofs, rel, ebb_offset); } fn reloc_external(&mut self, rel: Reloc, name: &ExternalName) { diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index e9424f829b..f216fe6514 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -9,7 +9,7 @@ mod memorysink; pub use self::relaxation::relax_branches; pub use self::memorysink::{MemoryCodeSink, RelocSink}; -use ir::{ExternalName, Ebb, JumpTable, Function, Inst}; +use ir::{ExternalName, JumpTable, Function, Inst}; use regalloc::RegDiversions; /// Offset in bytes from the beginning of the function. @@ -42,7 +42,7 @@ pub trait CodeSink { fn put8(&mut self, u64); /// Add a relocation referencing an EBB at the current offset. - fn reloc_ebb(&mut self, Reloc, Ebb); + fn reloc_ebb(&mut self, Reloc, CodeOffset); /// Add a relocation referencing an external symbol at the current offset. fn reloc_external(&mut self, Reloc, &ExternalName); From 3d83d0e4f0abea56e80c461f541d23d7952f0f51 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 31 Oct 2017 12:45:22 -0700 Subject: [PATCH 1316/3084] Clean up redundant blank lines. --- lib/frontend/src/frontend.rs | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 80d9768b67..e2d090f014 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -585,7 +585,6 @@ where self.params_values_initialized = true; } - fn ebb_params_adjustment(&mut self, dest_ebb: Ebb, jump_args: &[Value]) { if self.builder.ssa.predecessors(dest_ebb).is_empty() || self.builder.ebbs[dest_ebb].pristine From 5f8b1b9f048f3b33b463febb02e61d67b5d9db7e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 31 Oct 2017 13:05:26 -0700 Subject: [PATCH 1317/3084] Fix a flake8 lint. --- lib/cretonne/meta/isa/intel/recipes.py | 21 ++++++++++++++------- lib/cretonne/meta/isa/riscv/recipes.py | 3 ++- 2 files changed, 16 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index a32d3ec8df..531e291630 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -477,7 +477,8 @@ fnaddr4 = TailRecipe( 'fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(RelocKind::Abs4.into(), &func.dfg.ext_funcs[func_ref].name); + sink.reloc_external(RelocKind::Abs4.into(), + &func.dfg.ext_funcs[func_ref].name); sink.put4(0); ''') @@ -486,7 +487,8 @@ fnaddr8 = TailRecipe( 'fnaddr8', FuncAddr, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(RelocKind::Abs8.into(), &func.dfg.ext_funcs[func_ref].name); + sink.reloc_external(RelocKind::Abs8.into(), + &func.dfg.ext_funcs[func_ref].name); sink.put8(0); ''') @@ -495,7 +497,8 @@ allones_fnaddr4 = TailRecipe( 'allones_fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(RelocKind::Abs4.into(), &func.dfg.ext_funcs[func_ref].name); + sink.reloc_external(RelocKind::Abs4.into(), + &func.dfg.ext_funcs[func_ref].name); // Write the immediate as `!0` for the benefit of BaldrMonkey. sink.put4(!0); ''') @@ -505,7 +508,8 @@ allones_fnaddr8 = TailRecipe( 'allones_fnaddr8', FuncAddr, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(RelocKind::Abs8.into(), &func.dfg.ext_funcs[func_ref].name); + sink.reloc_external(RelocKind::Abs8.into(), + &func.dfg.ext_funcs[func_ref].name); // Write the immediate as `!0` for the benefit of BaldrMonkey. sink.put8(!0); ''') @@ -515,7 +519,8 @@ gvaddr4 = TailRecipe( 'gvaddr4', UnaryGlobalVar, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(RelocKind::Abs4.into(), &func.global_vars[global_var].symbol_name()); + sink.reloc_external(RelocKind::Abs4.into(), + &func.global_vars[global_var].symbol_name()); sink.put4(0); ''') @@ -524,7 +529,8 @@ gvaddr8 = TailRecipe( 'gvaddr8', UnaryGlobalVar, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(RelocKind::Abs8.into(), &func.global_vars[global_var].symbol_name()); + sink.reloc_external(RelocKind::Abs8.into(), + &func.global_vars[global_var].symbol_name()); sink.put8(0); ''') @@ -798,7 +804,8 @@ call_id = TailRecipe( 'call_id', Call, size=4, ins=(), outs=(), emit=''' PUT_OP(bits, BASE_REX, sink); - sink.reloc_external(RelocKind::PCRel4.into(), &func.dfg.ext_funcs[func_ref].name); + sink.reloc_external(RelocKind::PCRel4.into(), + &func.dfg.ext_funcs[func_ref].name); sink.put4(0); ''') diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index bfe30a0025..6874f95c36 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -183,7 +183,8 @@ UJ = EncRecipe( UJcall = EncRecipe( 'UJcall', Call, size=4, ins=(), outs=(), emit=''' - sink.reloc_external(RelocKind::Call.into(), &func.dfg.ext_funcs[func_ref].name); + sink.reloc_external(RelocKind::Call.into(), + &func.dfg.ext_funcs[func_ref].name); // rd=%x1 is the standard link register. put_uj(bits, 0, 1, sink); ''') From 09b1bdc1084a1509c94225a29a6e14c611da63fc Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 31 Oct 2017 14:12:29 -0700 Subject: [PATCH 1318/3084] Make "cton-util compile" default to the isa specified in the test file. --- cranelift/src/compile.rs | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 6b0bda21bd..828625120a 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -2,7 +2,7 @@ //! //! Reads IR files into Cretonne IL and compiles it. -use cton_reader::parse_functions; +use cton_reader::parse_test; use std::path::PathBuf; use cretonne::Context; use cretonne::settings::FlagsOrIsa; @@ -38,22 +38,28 @@ fn handle_module( let buffer = read_to_string(&path).map_err( |e| format!("{}: {}", name, e), )?; - let items = parse_functions(&buffer).map_err( - |e| format!("{}: {}", name, e), - )?; - for func in items.into_iter() { + let test_file = parse_test(&buffer).map_err(|e| format!("{}: {}", name, e))?; + + // If we have an isa from the command-line, use that. Otherwise if the + // file contins a unique isa, use that. + let isa = if let Some(isa) = fisa.isa { + isa + } else if let Some(isa) = test_file.isa_spec.unique_isa() { + isa + } else { + return Err(String::from("compilation requires a target isa")); + }; + + for (func, _) in test_file.functions.into_iter() { let mut context = Context::new(); context.func = func; - if let Some(isa) = fisa.isa { - context.compile(isa).map_err(|err| { - pretty_error(&context.func, fisa.isa, err) - })?; - } else { - return Err(String::from("compilation requires a target isa")); - } + context.compile(isa).map_err(|err| { + pretty_error(&context.func, Some(isa), err) + })?; if flag_print { - println!("{}", context.func.display(fisa.isa)); + println!("{}", context.func.display(isa)); } } + Ok(()) } From aa724846f6a5947864ed49f00219f260def1f757 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 1 Nov 2017 11:25:22 -0700 Subject: [PATCH 1319/3084] Clear the wasm operand stack at the end of a function. This preserves an invariant that the stack is empty between function body translations. --- lib/wasm/src/func_translator.rs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 9723f22ca5..aad791c9ac 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -215,6 +215,10 @@ fn parse_function_body( builder.ins().return_(&state.stack); } + // Discard any remaining values on the stack. Either we just returned them, + // or the end of the function is unreachable. + state.stack.clear(); + debug_assert!(reader.eof()); Ok(()) From 1663f141c89453f60f03e02aeb7d4e4ee908ad06 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 1 Nov 2017 11:26:55 -0700 Subject: [PATCH 1320/3084] Tidy up unneeded parentheses. --- lib/frontend/src/frontend.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index e2d090f014..1c41e18994 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -484,8 +484,8 @@ where None => false, Some(entry) => self.position.ebb == entry, }; - (!is_entry && self.builder.ssa.is_sealed(self.position.ebb) && - self.builder.ssa.predecessors(self.position.ebb).is_empty()) + !is_entry && self.builder.ssa.is_sealed(self.position.ebb) && + self.builder.ssa.predecessors(self.position.ebb).is_empty() } /// Returns `true` if and only if the entry block has been started and `param_value` From 39c0fbb6c70138852a5eac9499590b31460977ca Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 1 Nov 2017 11:27:54 -0700 Subject: [PATCH 1321/3084] Assert that sealed blocks cannot accept new predecessors. --- lib/frontend/src/ssa.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 65d1873fca..c8f80205e9 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -90,6 +90,7 @@ impl BlockData { match *self { BlockData::EbbBody { .. } => panic!("you can't add a predecessor to a body block"), BlockData::EbbHeader(ref mut data) => { + debug_assert!(!data.sealed, "sealed blocks cannot accept new predecessors"); data.predecessors.push((pred, inst)); } } From 57041ea9ebe47e37a703218bdba2c4e2bd51cc4b Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 1 Nov 2017 11:30:37 -0700 Subject: [PATCH 1322/3084] Tidy up semicolons. --- lib/frontend/src/ssa.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index c8f80205e9..35d3486da9 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -442,7 +442,7 @@ where debug_assert!(data.undef_variables.is_empty()); data.sealed = true; } - }; + } } /// Look up in the predecessors of an Ebb the def for a value an decides wether or not @@ -507,7 +507,7 @@ where } } ZeroOneOrMore::More() => {} - }; + } } let result_val = match pred_values { ZeroOneOrMore::Zero() => { @@ -515,8 +515,8 @@ where // code, but rather than throwing an error we silently initialize the variable to // 0. This will have no effect since this situation happens in unreachable code. if !func.layout.is_ebb_inserted(dest_ebb) { - func.layout.append_ebb(dest_ebb) - }; + func.layout.append_ebb(dest_ebb); + } self.side_effects.instructions_added_to_ebbs.push(dest_ebb); let zero = emit_zero( func.dfg.value_type(temp_arg_val), From 61a51f5275d06b8fc79187307932e49de0e672ab Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 1 Nov 2017 11:34:16 -0700 Subject: [PATCH 1323/3084] Rename a variable to clarify its purpose. --- lib/frontend/src/ssa.rs | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 35d3486da9..8ba9bfdb9a 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -611,9 +611,9 @@ where // We have to split the critical edge let middle_ebb = func.dfg.make_ebb(); func.layout.append_ebb(middle_ebb); - let block = self.declare_ebb_header_block(middle_ebb); - self.blocks[block].add_predecessor(jump_inst_block, jump_inst); - self.mark_ebb_header_block_sealed(block); + let middle_block = self.declare_ebb_header_block(middle_ebb); + self.blocks[middle_block].add_predecessor(jump_inst_block, jump_inst); + self.mark_ebb_header_block_sealed(middle_block); for old_dest in func.jump_tables[jt].as_mut_slice() { if old_dest.unwrap() == dest_ebb { *old_dest = PackedOption::from(middle_ebb); @@ -621,8 +621,8 @@ where } let mut cur = FuncCursor::new(func).at_bottom(middle_ebb); let middle_jump_inst = cur.ins().jump(dest_ebb, &[val]); - self.def_var(var, val, block); - Some((middle_ebb, block, middle_jump_inst)) + self.def_var(var, val, middle_block); + Some((middle_ebb, middle_block, middle_jump_inst)) } } } From c6d39dea5d76d3d09e89813126cb11039a8c5a3e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 1 Nov 2017 11:35:15 -0700 Subject: [PATCH 1324/3084] Add a comment about shrink_to_fit(). --- lib/frontend/src/ssa.rs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 8ba9bfdb9a..12e77170d0 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -441,6 +441,9 @@ where debug_assert!(!data.sealed); debug_assert!(data.undef_variables.is_empty()); data.sealed = true; + // We could call data.predecessors.shrink_to_fit() here, if + // important, because no further predecessors will be added + // to this block. } } } From 1cbf921d9d55011b8c6f8f59c49860a5158f7dad Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 1 Nov 2017 13:09:20 -0700 Subject: [PATCH 1325/3084] Add a function to seal all the blocks at once. --- lib/frontend/src/frontend.rs | 40 +++++++++++++++++++++++++++++++----- lib/frontend/src/ssa.rs | 24 +++++++++++++++++++++- 2 files changed, 58 insertions(+), 6 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 1c41e18994..fc7e8a90b2 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -300,6 +300,17 @@ where self.handle_ssa_side_effects(side_effects); } + /// Effectively calls seal_block on all blocks in the function. + /// + /// It's more efficient to seal `Ebb`s as soon as possible, during + /// translation, but for frontends where this is impractical to do, this + /// function can be used at the end of translating all blocks to ensure + /// that everything is sealed. + pub fn seal_all_blocks(&mut self) { + let side_effects = self.builder.ssa.seal_all_ebb_header_blocks(self.func); + self.handle_ssa_side_effects(side_effects); + } + /// In order to use a variable in a `use_var`, you need to declare its type with this method. pub fn declare_var(&mut self, var: Variable, ty: Type) { self.builder.types[var] = ty; @@ -679,8 +690,7 @@ mod tests { } } - #[test] - fn sample_function() { + fn sample_function(lazy_seal: bool) { let mut sig = Signature::new(CallConv::Native); sig.returns.push(AbiParam::new(I32)); sig.params.push(AbiParam::new(I32)); @@ -701,7 +711,9 @@ mod tests { builder.declare_var(z, I32); builder.switch_to_block(block0, &[]); - builder.seal_block(block0); + if !lazy_seal { + builder.seal_block(block0); + } { let tmp = builder.param_value(0); builder.def_var(x, tmp); @@ -741,7 +753,9 @@ mod tests { } builder.switch_to_block(block2, &[]); - builder.seal_block(block2); + if !lazy_seal { + builder.seal_block(block2); + } { let arg1 = builder.use_var(y); @@ -750,7 +764,13 @@ mod tests { builder.def_var(y, tmp); } builder.ins().jump(block1, &[]); - builder.seal_block(block1); + if !lazy_seal { + builder.seal_block(block1); + } + + if lazy_seal { + builder.seal_all_blocks(); + } } let flags = settings::Flags::new(&settings::builder()); @@ -761,4 +781,14 @@ mod tests { Err(err) => panic!("{}{}", func.display(None), err), } } + + #[test] + fn sample() { + sample_function(false) + } + + #[test] + fn sample_with_lazy_seal() { + sample_function(true) + } } diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 12e77170d0..f8018ad415 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -409,6 +409,29 @@ where /// /// Returns the list of newly created ebbs for critical edge splitting. pub fn seal_ebb_header_block(&mut self, ebb: Ebb, func: &mut Function) -> SideEffects { + self.seal_one_ebb_header_block(ebb, func); + mem::replace(&mut self.side_effects, SideEffects::new()) + } + + /// Completes the global value numbering for all `Ebb`s in `func`. + /// + /// It's more efficient to seal `Ebb`s as soon as possible, during + /// translation, but for frontends where this is impractical to do, this + /// function can be used at the end of translating all blocks to ensure + /// that everything is sealed. + pub fn seal_all_ebb_header_blocks(&mut self, func: &mut Function) -> SideEffects { + // Seal all `Ebb`s currently in the function. This can entail splitting + // and creation of new blocks, however such new blocks are sealed on + // the fly, so we don't need to accout for them here. + for ebb in self.ebb_headers.keys() { + self.seal_one_ebb_header_block(ebb, func); + } + mem::replace(&mut self.side_effects, SideEffects::new()) + } + + /// Helper function for `seal_ebb_header_block` and + /// `seal_all_ebb_header_blocks`. + fn seal_one_ebb_header_block(&mut self, ebb: Ebb, func: &mut Function) { let block = self.header_block(ebb); let (undef_vars, ebb): (Vec<(Variable, Value)>, Ebb) = match self.blocks[block] { @@ -429,7 +452,6 @@ where self.predecessors_lookup(func, val, var, ty, ebb); } self.mark_ebb_header_block_sealed(block); - mem::replace(&mut self.side_effects, SideEffects::new()) } /// Set the `sealed` flag for `block`. From e21479a84340b1d2ab4e78eb26f5cd269b638be4 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 2 Nov 2017 12:28:01 -0700 Subject: [PATCH 1326/3084] Mark check_return_args as #[cfg(debug_assertions)]. --- lib/frontend/src/frontend.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index fc7e8a90b2..b382951f7c 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -122,7 +122,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short // instruction being inserted to add related info to the DFG and the SSA building system, // and perform debug sanity checks. fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'short mut DataFlowGraph) { - if data.opcode().is_return() { + if cfg!(debug_assertions) && data.opcode().is_return() { self.builder .check_return_args(data.arguments(&self.builder.func.dfg.value_lists)); } @@ -569,6 +569,7 @@ where ); } + #[cfg(debug_assertions)] fn check_return_args(&self, args: &[Value]) { debug_assert_eq!( args.len(), From c7f01f88b2ae7b8364ed1478f12ba60b5f2691ec Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 3 Nov 2017 11:44:58 -0700 Subject: [PATCH 1327/3084] Clarify some comments. --- lib/frontend/src/frontend.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index b382951f7c..4cbdcd527f 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -10,7 +10,7 @@ use cretonne::isa::TargetIsa; use ssa::{SSABuilder, SideEffects, Block}; use cretonne::entity::{EntityRef, EntityMap, EntitySet}; -/// Permanent structure used for translating into Cretonne IL. +/// Structure used for translating a series of functions into Cretonne IL. /// /// In order to reduce memory reallocations whem compiling multiple functions, /// `ILBuilder` holds various data structures which are cleared between @@ -26,7 +26,7 @@ where } -/// Temporary object used to build a Cretonne IL `Function`. +/// Temporary object used to build a single Cretonne IL `Function`. pub struct FunctionBuilder<'a, Variable: 'a> where Variable: EntityRef + Default, From acc6d941a365318cc3ba93cd9c141b7df1e52d9c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 3 Nov 2017 16:40:51 -0700 Subject: [PATCH 1328/3084] Combine redundant match arm bodies. --- lib/cretonne/src/abi.rs | 4 ++-- lib/cretonne/src/ir/instructions.rs | 8 ++++---- lib/cretonne/src/verifier/locations.rs | 2 +- lib/filecheck/src/error.rs | 6 +++--- lib/filecheck/src/pattern.rs | 2 +- lib/reader/src/parser.rs | 6 +++--- 6 files changed, 14 insertions(+), 14 deletions(-) diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index 7eb459fe6b..5c121824aa 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -61,7 +61,7 @@ impl ValueConversion { ValueConversion::IntSplit => ty.half_width().expect("Integer type too small to split"), ValueConversion::VectorSplit => ty.half_vector().expect("Not a vector"), ValueConversion::IntBits => Type::int(ty.bits()).expect("Bad integer size"), - ValueConversion::Sext(nty) => nty, + ValueConversion::Sext(nty) | ValueConversion::Uext(nty) => nty, } } @@ -69,7 +69,7 @@ impl ValueConversion { /// Is this a split conversion that results in two arguments? pub fn is_split(self) -> bool { match self { - ValueConversion::IntSplit => true, + ValueConversion::IntSplit | ValueConversion::VectorSplit => true, _ => false, } diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 0cb9234ed6..2a8d5cdf2d 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -356,8 +356,8 @@ impl InstructionData { /// Multi-destination branches like `br_table` return `None`. pub fn branch_destination(&self) -> Option { match *self { - InstructionData::Jump { destination, .. } => Some(destination), - InstructionData::Branch { destination, .. } => Some(destination), + InstructionData::Jump { destination, .. } | + InstructionData::Branch { destination, .. } | InstructionData::BranchIcmp { destination, .. } => Some(destination), _ => None, } @@ -369,8 +369,8 @@ impl InstructionData { /// Multi-destination branches like `br_table` return `None`. pub fn branch_destination_mut(&mut self) -> Option<&mut Ebb> { match *self { - InstructionData::Jump { ref mut destination, .. } => Some(destination), - InstructionData::Branch { ref mut destination, .. } => Some(destination), + InstructionData::Jump { ref mut destination, .. } | + InstructionData::Branch { ref mut destination, .. } | InstructionData::BranchIcmp { ref mut destination, .. } => Some(destination), _ => None, } diff --git a/lib/cretonne/src/verifier/locations.rs b/lib/cretonne/src/verifier/locations.rs index bb80507baa..94958c2f89 100644 --- a/lib/cretonne/src/verifier/locations.rs +++ b/lib/cretonne/src/verifier/locations.rs @@ -233,7 +233,7 @@ impl<'a> LocationVerifier<'a> { /// Update diversions to reflect the current instruction and check their consistency. fn update_diversions(&self, inst: ir::Inst, divert: &mut RegDiversions) -> Result { let (arg, src) = match self.func.dfg[inst] { - ir::InstructionData::RegMove { arg, src, .. } => (arg, ir::ValueLoc::Reg(src)), + ir::InstructionData::RegMove { arg, src, .. } | ir::InstructionData::RegSpill { arg, src, .. } => (arg, ir::ValueLoc::Reg(src)), ir::InstructionData::RegFill { arg, src, .. } => (arg, ir::ValueLoc::Stack(src)), _ => return Ok(()), diff --git a/lib/filecheck/src/error.rs b/lib/filecheck/src/error.rs index 1cc17001fc..5580cc4c3c 100644 --- a/lib/filecheck/src/error.rs +++ b/lib/filecheck/src/error.rs @@ -39,9 +39,9 @@ impl StdError for Error { fn description(&self) -> &str { use Error::*; match *self { - Syntax(ref s) => s, - UndefVariable(ref s) => s, - Backref(ref s) => s, + Syntax(ref s) | + UndefVariable(ref s) | + Backref(ref s) | DuplicateDef(ref s) => s, Regex(ref err) => err.description(), } diff --git a/lib/filecheck/src/pattern.rs b/lib/filecheck/src/pattern.rs index 0d1da57f12..7969bc00ec 100644 --- a/lib/filecheck/src/pattern.rs +++ b/lib/filecheck/src/pattern.rs @@ -45,7 +45,7 @@ impl Part { /// Get the variabled referenced by this part, if any. pub fn ref_var(&self) -> Option<&str> { match *self { - Part::Var(ref var) => Some(var), + Part::Var(ref var) | Part::DefVar { ref var, .. } => Some(var), _ => None, } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index dd08b0ddcb..0a63f19029 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1387,9 +1387,9 @@ impl<'a> Parser<'a> { // extended-basic-block ::= ebb-header * { instruction } while match self.token() { - Some(Token::Value(_)) => true, - Some(Token::Identifier(_)) => true, - Some(Token::LBracket) => true, + Some(Token::Value(_)) | + Some(Token::Identifier(_)) | + Some(Token::LBracket) | Some(Token::SourceLoc(_)) => true, _ => false, } From 8501cb798e9f9262307f4c26c089d4aae3760864 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 3 Nov 2017 16:43:29 -0700 Subject: [PATCH 1329/3084] Minor comment cleanups. --- lib/cretonne/src/bitset.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/bitset.rs b/lib/cretonne/src/bitset.rs index a9c41ce78a..2ed4efe15b 100644 --- a/lib/cretonne/src/bitset.rs +++ b/lib/cretonne/src/bitset.rs @@ -1,10 +1,10 @@ //! Small Bitset //! -//! This module defines a struct BitSet encapsulating a bitset built over the type T. +//! This module defines a struct `BitSet` encapsulating a bitset built over the type T. //! T is intended to be a primitive unsigned type. Currently it can be any type between u8 and u32 //! //! If you would like to add support for larger bitsets in the future, you need to change the trait -//! bound Into and the u32 in the implementation of max_bits() +//! bound Into and the u32 in the implementation of `max_bits()`. use std::mem::size_of; use std::ops::{Shl, BitOr, Sub, Add}; use std::convert::{Into, From}; From d9743290ea9863b3c63af039194bce286b47951a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 6 Nov 2017 11:09:56 -0800 Subject: [PATCH 1330/3084] Elide elidable lifetime parameters. --- lib/cretonne/src/ir/dfg.rs | 4 ++-- lib/cretonne/src/ir/instructions.rs | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 4a98fc6902..83a32f7ff8 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -660,14 +660,14 @@ impl DataFlowGraph { impl Index for DataFlowGraph { type Output = InstructionData; - fn index<'a>(&'a self, inst: Inst) -> &'a InstructionData { + fn index(&self, inst: Inst) -> &InstructionData { &self.insts[inst] } } /// Allow mutable access to instructions via indexing. impl IndexMut for DataFlowGraph { - fn index_mut<'a>(&'a mut self, inst: Inst) -> &'a mut InstructionData { + fn index_mut(&mut self, inst: Inst) -> &mut InstructionData { &mut self.insts[inst] } } diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 2a8d5cdf2d..6c101d751e 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -289,13 +289,13 @@ impl VariableArgs { impl Deref for VariableArgs { type Target = [Value]; - fn deref<'a>(&'a self) -> &'a [Value] { + fn deref(&self) -> &[Value] { &self.0 } } impl DerefMut for VariableArgs { - fn deref_mut<'a>(&'a mut self) -> &'a mut [Value] { + fn deref_mut(&mut self) -> &mut [Value] { &mut self.0 } } From 4091688cc0abd30cb7734923c624273495e2a6f0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 6 Nov 2017 11:10:03 -0800 Subject: [PATCH 1331/3084] Avoid unnecessary name qualification. --- lib/cretonne/src/dominator_tree.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 300c9453df..043ebad4eb 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -431,7 +431,7 @@ mod test { use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::types::*; - use ir::{Function, InstBuilder, types, TrapCode}; + use ir::{Function, InstBuilder, TrapCode}; use settings; use super::*; use verifier::verify_context; @@ -450,7 +450,7 @@ mod test { fn unreachable_node() { let mut func = Function::new(); let ebb0 = func.dfg.make_ebb(); - let v0 = func.dfg.append_ebb_param(ebb0, types::I32); + let v0 = func.dfg.append_ebb_param(ebb0, I32); let ebb1 = func.dfg.make_ebb(); let ebb2 = func.dfg.make_ebb(); @@ -479,7 +479,7 @@ mod test { fn non_zero_entry_block() { let mut func = Function::new(); let ebb3 = func.dfg.make_ebb(); - let cond = func.dfg.append_ebb_param(ebb3, types::I32); + let cond = func.dfg.append_ebb_param(ebb3, I32); let ebb1 = func.dfg.make_ebb(); let ebb2 = func.dfg.make_ebb(); let ebb0 = func.dfg.make_ebb(); From 1d8df2ed1ee25bb7ce46f4d337ec0d9931c37548 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 6 Nov 2017 11:10:14 -0800 Subject: [PATCH 1332/3084] Remove the requirement that `Variable` implement `Default`. --- lib/frontend/src/frontend.rs | 25 ++++++++++--------------- lib/frontend/src/lib.rs | 5 ----- lib/frontend/src/ssa.rs | 11 +++-------- 3 files changed, 13 insertions(+), 28 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 4cbdcd527f..9ae7cfe0fb 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -17,7 +17,7 @@ use cretonne::entity::{EntityRef, EntityMap, EntitySet}; /// functions, rather than dropped, preserving the underlying allocations. pub struct ILBuilder where - Variable: EntityRef + Default, + Variable: EntityRef, { ssa: SSABuilder, ebbs: EntityMap, @@ -29,7 +29,7 @@ where /// Temporary object used to build a single Cretonne IL `Function`. pub struct FunctionBuilder<'a, Variable: 'a> where - Variable: EntityRef + Default, + Variable: EntityRef, { /// The function currently being built. /// This field is public so the function can be re-borrowed. @@ -59,7 +59,7 @@ struct Position { impl ILBuilder where - Variable: EntityRef + Default, + Variable: EntityRef, { /// Creates a ILBuilder structure. The structure is automatically cleared after each /// [`FunctionBuilder`](struct.FunctionBuilder.html) completes translating a function. @@ -89,7 +89,7 @@ where /// one convenience method per Cretonne IL instruction. pub struct FuncInstBuilder<'short, 'long: 'short, Variable: 'long> where - Variable: EntityRef + Default, + Variable: EntityRef, { builder: &'short mut FunctionBuilder<'long, Variable>, ebb: Ebb, @@ -97,7 +97,7 @@ where impl<'short, 'long, Variable> FuncInstBuilder<'short, 'long, Variable> where - Variable: EntityRef + Default, + Variable: EntityRef, { fn new<'s, 'l>( builder: &'s mut FunctionBuilder<'l, Variable>, @@ -108,7 +108,7 @@ where } impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long, Variable> - where Variable: EntityRef + Default + where Variable: EntityRef { fn data_flow_graph(&self) -> &DataFlowGraph { &self.builder.func.dfg @@ -221,7 +221,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short /// return instruction with arguments that don't match the function's signature. impl<'a, Variable> FunctionBuilder<'a, Variable> where - Variable: EntityRef + Default, + Variable: EntityRef, { /// Creates a new FunctionBuilder structure that will operate on a `Function` using a /// `IlBuilder`. @@ -442,7 +442,7 @@ where /// in ways that can be unsafe if used incorrectly. impl<'a, Variable> FunctionBuilder<'a, Variable> where - Variable: EntityRef + Default, + Variable: EntityRef, { /// Retrieves all the parameters for an `Ebb` currently inferred from the jump instructions /// inserted that target it and the SSA construction. @@ -527,7 +527,7 @@ where impl<'a, Variable> Drop for FunctionBuilder<'a, Variable> where - Variable: EntityRef + Default, + Variable: EntityRef, { /// When a `FunctionBuilder` goes out of scope, it means that the function is fully built. fn drop(&mut self) { @@ -549,7 +549,7 @@ where // Helper functions impl<'a, Variable> FunctionBuilder<'a, Variable> where - Variable: EntityRef + Default, + Variable: EntityRef, { fn move_to_next_basic_block(&mut self) { self.position.basic_block = self.builder.ssa.declare_ebb_body_block( @@ -685,11 +685,6 @@ mod tests { self.0 as usize } } - impl Default for Variable { - fn default() -> Variable { - Variable(u32::MAX) - } - } fn sample_function(lazy_seal: bool) { let mut sig = Signature::new(CallConv::Native); diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index bcab9c2d61..6e10d6481a 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -56,11 +56,6 @@ //! self.0 as usize //! } //! } -//! impl Default for Variable { -//! fn default() -> Variable { -//! Variable(u32::MAX) -//! } -//! } //! //! fn main() { //! let mut sig = Signature::new(CallConv::Native); diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index f8018ad415..c7c4d08029 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -33,7 +33,7 @@ use std::mem; /// can be declared. pub struct SSABuilder where - Variable: EntityRef + Default, + Variable: EntityRef, { // Records for every variable and for every revelant block, the last definition of // the variable in the block. @@ -145,7 +145,7 @@ impl ReservedValue for Block { impl SSABuilder where - Variable: EntityRef + Default, + Variable: EntityRef, { /// Allocate a new blank SSA builder struct. Use the API function to interact with the struct. pub fn new() -> Self { @@ -252,7 +252,7 @@ fn emit_zero(ty: Type, mut cur: FuncCursor) -> Value { /// impl SSABuilder where - Variable: EntityRef + Default, + Variable: EntityRef, { /// Declares a new definition of a variable in a given basic block. /// The SSA value is passed as an argument because it should be created with @@ -735,11 +735,6 @@ mod tests { self.0 as usize } } - impl Default for Variable { - fn default() -> Variable { - Variable(u32::MAX) - } - } #[test] fn simple_block() { From 5dda19035d8d037e326adb697d3c613ca6a4a620 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 6 Nov 2017 11:34:09 -0800 Subject: [PATCH 1333/3084] Implement Hash for ExternalName. This allows users to have HashMaps etc. with ExternalNames as keys. --- lib/cretonne/src/ir/extname.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/ir/extname.rs b/lib/cretonne/src/ir/extname.rs index bd8c4e631e..273006b570 100644 --- a/lib/cretonne/src/ir/extname.rs +++ b/lib/cretonne/src/ir/extname.rs @@ -15,7 +15,7 @@ use std::ascii::AsciiExt; /// External names can also serve as a primitive testing and debugging tool. /// In particular, many `.cton` test files use function names to identify /// functions. -#[derive(Debug, Clone, PartialEq, Eq, Default)] +#[derive(Debug, Clone, PartialEq, Eq, Default, Hash)] pub struct ExternalName(NameRepr); impl ExternalName { @@ -69,7 +69,7 @@ fn try_as_name(bytes: &[u8]) -> Option { const NAME_LENGTH_THRESHOLD: usize = 22; -#[derive(Debug, Clone, PartialEq, Eq)] +#[derive(Debug, Clone, PartialEq, Eq, Hash)] enum NameRepr { Short { length: u8, From f76640778c2959971916ad23f297ef9a55a927d0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 6 Nov 2017 16:28:15 -0800 Subject: [PATCH 1334/3084] Fix printing with -p when -c is also given. --- cranelift/src/wasm.rs | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 679fc79115..04810e2983 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -120,14 +120,14 @@ fn handle_module( context.verify(fisa).map_err(|err| { pretty_verifier_error(&context.func, fisa.isa, err) })?; - continue; - } - if let Some(isa) = fisa.isa { - context.compile(isa).map_err(|err| { - pretty_error(&context.func, fisa.isa, err) - })?; } else { - return Err(String::from("compilation requires a target isa")); + if let Some(isa) = fisa.isa { + context.compile(isa).map_err(|err| { + pretty_error(&context.func, fisa.isa, err) + })?; + } else { + return Err(String::from("compilation requires a target isa")); + } } if flag_print { vprintln!(flag_verbose, ""); From cce2384edef10eb17a35f9e4c366645e9ea5a6af Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 6 Nov 2017 16:38:13 -0800 Subject: [PATCH 1335/3084] Remove switch_to_block's jump_args argument. switch_to_block doesn't need its jump_args argument, since jump arguments are handled by the `jump` instruction and others, rather than on the switch to a new ebb itself. --- lib/frontend/src/frontend.rs | 10 ++++------ lib/frontend/src/lib.rs | 6 +++--- lib/wasm/src/code_translator.rs | 12 ++++++------ lib/wasm/src/func_translator.rs | 2 +- 4 files changed, 14 insertions(+), 16 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 9ae7cfe0fb..6dab86b245 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -266,7 +266,7 @@ where /// When inserting the terminator instruction (which doesn't have a falltrough to its immediate /// successor), the block will be declared filled and it will not be possible to append /// instructions to it. - pub fn switch_to_block(&mut self, ebb: Ebb, jump_args: &[Value]) -> &[Value] { + pub fn switch_to_block(&mut self, ebb: Ebb) { if !self.params_values_initialized { self.fill_function_params_values(ebb); } @@ -286,8 +286,6 @@ where let basic_block = self.builder.ssa.header_block(ebb); // Then we change the cursor position. self.position = Position { ebb, basic_block }; - self.ebb_params_adjustment(ebb, jump_args); - self.func.dfg.ebb_params(ebb) } /// Declares that all the predecessors of this block are known. @@ -706,7 +704,7 @@ mod tests { builder.declare_var(y, I32); builder.declare_var(z, I32); - builder.switch_to_block(block0, &[]); + builder.switch_to_block(block0); if !lazy_seal { builder.seal_block(block0); } @@ -726,7 +724,7 @@ mod tests { } builder.ins().jump(block1, &[]); - builder.switch_to_block(block1, &[]); + builder.switch_to_block(block1); { let arg1 = builder.use_var(y); let arg2 = builder.use_var(z); @@ -748,7 +746,7 @@ mod tests { builder.ins().return_(&[arg]); } - builder.switch_to_block(block2, &[]); + builder.switch_to_block(block2); if !lazy_seal { builder.seal_block(block2); } diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 6e10d6481a..9b23e80a66 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -76,7 +76,7 @@ //! builder.declare_var(y, I32); //! builder.declare_var(z, I32); //! -//! builder.switch_to_block(block0, &[]); +//! builder.switch_to_block(block0); //! builder.seal_block(block0); //! { //! let tmp = builder.param_value(0); @@ -94,7 +94,7 @@ //! } //! builder.ins().jump(block1, &[]); //! -//! builder.switch_to_block(block1, &[]); +//! builder.switch_to_block(block1); //! { //! let arg1 = builder.use_var(y); //! let arg2 = builder.use_var(z); @@ -116,7 +116,7 @@ //! builder.ins().return_(&[arg]); //! } //! -//! builder.switch_to_block(block2, &[]); +//! builder.switch_to_block(block2); //! builder.seal_block(block2); //! //! { diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index d74bbbc688..040a53fe5d 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -133,7 +133,7 @@ pub fn translate_operator( } builder.ins().jump(loop_body, &[]); state.push_loop(loop_body, next, num_return_values(ty)); - builder.switch_to_block(loop_body, &[]); + builder.switch_to_block(loop_body); } Operator::If { ty } => { let val = state.pop1(); @@ -170,7 +170,7 @@ pub fn translate_operator( let else_ebb = builder.create_ebb(); builder.change_jump_destination(branch_inst, else_ebb); builder.seal_block(else_ebb); - builder.switch_to_block(else_ebb, &[]); + builder.switch_to_block(else_ebb); } Operator::End => { let frame = state.control_stack.pop().unwrap(); @@ -181,7 +181,7 @@ pub fn translate_operator( state.peekn(return_count), ); } - builder.switch_to_block(frame.following_code(), state.peekn(return_count)); + builder.switch_to_block(frame.following_code()); builder.seal_block(frame.following_code()); // If it is a loop we also have to seal the body loop block match frame { @@ -316,7 +316,7 @@ pub fn translate_operator( .br_destination(); builder.ins().jump(default_ebb, state.peekn(return_count)); for (depth, dest_ebb) in dest_ebbs { - builder.switch_to_block(dest_ebb, &[]); + builder.switch_to_block(dest_ebb); builder.seal_block(dest_ebb); let i = state.control_stack.len() - 1 - depth; let real_dest_ebb = { @@ -856,7 +856,7 @@ fn translate_unreachable_operator( // a jump instruction since the code is still unreachable let frame = control_stack.pop().unwrap(); - builder.switch_to_block(frame.following_code(), &[]); + builder.switch_to_block(frame.following_code()); builder.seal_block(frame.following_code()); match frame { // If it is a loop we also have to seal the body loop block @@ -900,7 +900,7 @@ fn translate_unreachable_operator( let else_ebb = builder.create_ebb(); builder.change_jump_destination(branch_inst, else_ebb); builder.seal_block(else_ebb); - builder.switch_to_block(else_ebb, &[]); + builder.switch_to_block(else_ebb); // Now we have to split off the stack the values not used // by unreachable code that hasn't been translated stack.truncate(original_stack_size); diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index aad791c9ac..acdd400a8f 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -82,7 +82,7 @@ impl FuncTranslator { // This clears the `ILBuilder`. let mut builder = FunctionBuilder::new(func, &mut self.il_builder); let entry_block = builder.create_ebb(); - builder.switch_to_block(entry_block, &[]); // This also creates values for the arguments. + builder.switch_to_block(entry_block); // This also creates values for the arguments. builder.seal_block(entry_block); // Make sure the entry block is inserted in the layout before we make any callbacks to // `environ`. The callback functions may need to insert things in the entry block. From 9b15fe7dfd61f0347f1e4411f7d59c6456e1a1e7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 7 Nov 2017 14:55:08 -0800 Subject: [PATCH 1336/3084] Simplify Ebb parameter creation. Ebb parameters are appended explicitly by whoever calls create_block, so they don't need to also be inferred from branches. This makes the frontend code more flexible for producers that want to create things in a different order, and it eliminates more temporary allocations. FunctionBuilder no longer maintains its own list of the function parameter values; these can be obtained from the ebb parameter list on the entry block. --- lib/frontend/src/frontend.rs | 179 +++++++------------------------- lib/frontend/src/lib.rs | 3 +- lib/wasm/src/func_translator.rs | 10 +- 3 files changed, 47 insertions(+), 145 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 6dab86b245..95900b8166 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -4,7 +4,6 @@ use cretonne::ir; use cretonne::ir::{Ebb, Type, Value, Function, Inst, JumpTable, StackSlot, JumpTableData, StackSlotData, DataFlowGraph, InstructionData, ExtFuncData, FuncRef, SigRef, Signature, InstBuilderBase, GlobalVarData, GlobalVar, HeapData, Heap}; -use cretonne::ir::instructions::BranchInfo; use cretonne::ir::function::DisplayFunction; use cretonne::isa::TargetIsa; use ssa::{SSABuilder, SideEffects, Block}; @@ -22,7 +21,6 @@ where ssa: SSABuilder, ebbs: EntityMap, types: EntityMap, - function_params_values: Vec, } @@ -40,9 +38,6 @@ where builder: &'a mut ILBuilder, position: Position, - - /// Has builder.function_params_values been populated yet? - params_values_initialized: bool, } #[derive(Clone, Default)] @@ -68,7 +63,6 @@ where ssa: SSABuilder::new(), ebbs: EntityMap::new(), types: EntityMap::new(), - function_params_values: Vec::new(), } } @@ -76,12 +70,10 @@ where self.ssa.clear(); self.ebbs.clear(); self.types.clear(); - self.function_params_values.clear(); } fn is_empty(&self) -> bool { - self.ssa.is_empty() && self.ebbs.is_empty() && self.types.is_empty() && - self.function_params_values.is_empty() + self.ssa.is_empty() && self.ebbs.is_empty() && self.types.is_empty() } } @@ -122,10 +114,6 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short // instruction being inserted to add related info to the DFG and the SSA building system, // and perform debug sanity checks. fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'short mut DataFlowGraph) { - if cfg!(debug_assertions) && data.opcode().is_return() { - self.builder - .check_return_args(data.arguments(&self.builder.func.dfg.value_lists)); - } // We only insert the Ebb in the layout when an instruction is added to it self.builder.ensure_inserted_ebb(); @@ -141,15 +129,6 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short Some(dest_ebb) => { // If the user has supplied jump arguments we must adapt the arguments of // the destination ebb -// TODO: find a way not to allocate a vector - let args_types: Vec = - match data.analyze_branch(&self.builder.func.dfg.value_lists) { - BranchInfo::SingleDest(_, args) => { - args.to_vec() - } - _ => panic!("should not happen"), - }; - self.builder.ebb_params_adjustment(dest_ebb, &args_types); self.builder.declare_successor(dest_ebb, inst); } None => { @@ -238,7 +217,6 @@ where ebb: Ebb::new(0), basic_block: Block::new(0), }, - params_values_initialized: false, } } @@ -247,7 +225,7 @@ where self.srcloc = srcloc; } - /// Creates a new `Ebb` for the function and returns its reference. + /// Creates a new `Ebb` and returns its reference. pub fn create_ebb(&mut self) -> Ebb { let ebb = self.func.dfg.make_ebb(); self.builder.ssa.declare_ebb_header_block(ebb); @@ -267,9 +245,6 @@ where /// successor), the block will be declared filled and it will not be possible to append /// instructions to it. pub fn switch_to_block(&mut self, ebb: Ebb) { - if !self.params_values_initialized { - self.fill_function_params_values(ebb); - } if !self.builder.ebbs[self.position.ebb].pristine { // First we check that the previous block has been filled. debug_assert!( @@ -333,11 +308,6 @@ where /// Register a new definition of a user variable. Panics if the type of the value is not the /// same as the type registered for the variable. pub fn def_var(&mut self, var: Variable, val: Value) { - debug_assert_eq!( - self.func.dfg.value_type(val), - self.builder.types[var], - "the type of the value is not the type registered for the variable" - ); self.builder.ssa.def_var( var, val, @@ -345,23 +315,6 @@ where ); } - /// Returns the value corresponding to the `i`-th parameter of the function as defined by - /// the function signature. Panics if `i` is out of bounds or if called before the first call - /// to `switch_to_block`. - pub fn param_value(&self, i: usize) -> Value { - debug_assert!( - self.params_values_initialized, - "you have to call switch_to_block first." - ); - self.builder.function_params_values[i] - } - - /// Use param_value instead. - #[deprecated(since = "forever", note = "arg_value is renamed to param_value")] - pub fn arg_value(&self, i: usize) -> Value { - self.param_value(i) - } - /// Creates a jump table in the function, to be used by `br_table` instructions. pub fn create_jump_table(&mut self, data: JumpTableData) -> JumpTable { self.func.create_jump_table(data) @@ -431,6 +384,28 @@ where .with_srcloc(self.srcloc) .at_bottom(self.position.ebb) } + + /// Append parameters to the given `Ebb` corresponding to the function + /// parameters. This can be used to set up the ebb parameters for the + /// entry block. + pub fn append_ebb_params_for_function_params(&mut self, ebb: Ebb) { + let user_arg_count = &mut self.builder.ebbs[ebb].user_arg_count; + for argtyp in &self.func.signature.params { + *user_arg_count += 1; + self.func.dfg.append_ebb_param(ebb, argtyp.value_type); + } + } + + /// Append parameters to the given `Ebb` corresponding to the function + /// return values. This can be used to set up the ebb parameters for a + /// function exit block. + pub fn append_ebb_params_for_function_returns(&mut self, ebb: Ebb) { + let user_arg_count = &mut self.builder.ebbs[ebb].user_arg_count; + for argtyp in &self.func.signature.returns { + *user_arg_count += 1; + self.func.dfg.append_ebb_param(ebb, argtyp.value_type); + } + } } /// All the functions documented in the previous block are write-only and help you build a valid @@ -460,6 +435,11 @@ where /// instructions to it, otherwise this could interfere with SSA construction. pub fn append_ebb_param(&mut self, ebb: Ebb, ty: Type) -> Value { debug_assert!(self.builder.ebbs[ebb].pristine); + debug_assert_eq!( + self.builder.ebbs[ebb].user_arg_count, + self.func.dfg.num_ebb_params(ebb) + ); + self.builder.ebbs[ebb].user_arg_count += 1; self.func.dfg.append_ebb_param(ebb, ty) } @@ -497,12 +477,6 @@ where self.builder.ssa.predecessors(self.position.ebb).is_empty() } - /// Returns `true` if and only if the entry block has been started and `param_value` - /// may be called. - pub fn entry_block_started(&self) -> bool { - self.params_values_initialized - } - /// Returns `true` if and only if no instructions have been added since the last call to /// `switch_to_block`. pub fn is_pristine(&self) -> bool { @@ -532,10 +506,15 @@ where // Check that all the `Ebb`s are filled and sealed. debug_assert!( self.builder.ebbs.keys().all(|ebb| { - self.builder.ebbs[ebb].pristine || - (self.builder.ssa.is_sealed(ebb) && self.builder.ebbs[ebb].filled) + self.builder.ebbs[ebb].pristine || self.builder.ssa.is_sealed(ebb) }), - "all blocks should be filled and sealed before dropping a FunctionBuilder" + "all blocks should be sealed before dropping a FunctionBuilder" + ); + debug_assert!( + self.builder.ebbs.keys().all(|ebb| { + self.builder.ebbs[ebb].pristine || self.builder.ebbs[ebb].filled + }), + "all blocks should be filled before dropping a FunctionBuilder" ); // Clear the state (but preserve the allocated buffers) in preparation @@ -567,87 +546,6 @@ where ); } - #[cfg(debug_assertions)] - fn check_return_args(&self, args: &[Value]) { - debug_assert_eq!( - args.len(), - self.func.signature.returns.len(), - "the number of returned values doesn't match the function signature " - ); - for (i, arg) in args.iter().enumerate() { - let valty = self.func.dfg.value_type(*arg); - debug_assert_eq!( - valty, - self.func.signature.returns[i].value_type, - "the types of the values returned don't match the \ - function signature" - ); - } - } - - fn fill_function_params_values(&mut self, ebb: Ebb) { - debug_assert!(!self.params_values_initialized); - for argtyp in &self.func.signature.params { - self.builder.function_params_values.push( - self.func.dfg.append_ebb_param(ebb, argtyp.value_type), - ); - } - self.params_values_initialized = true; - } - - fn ebb_params_adjustment(&mut self, dest_ebb: Ebb, jump_args: &[Value]) { - if self.builder.ssa.predecessors(dest_ebb).is_empty() || - self.builder.ebbs[dest_ebb].pristine - { - // This is the first jump instruction targeting this Ebb - // so the jump arguments supplied here are this Ebb' arguments - // However some of the arguments might already be there - // in the Ebb so we have to check they're consistent - let dest_ebb_params_len = { - let dest_ebb_params = self.func.dfg.ebb_params(dest_ebb); - debug_assert!( - dest_ebb_params - .iter() - .zip(jump_args.iter().take(dest_ebb_params.len())) - .all(|(dest_arg, jump_arg)| { - self.func.dfg.value_type(*jump_arg) == - self.func.dfg.value_type(*dest_arg) - }), - "the jump argument supplied has not the \ - same type as the corresponding dest ebb argument" - ); - dest_ebb_params.len() - }; - self.builder.ebbs[dest_ebb].user_arg_count = jump_args.len(); - for val in jump_args.iter().skip(dest_ebb_params_len) { - let ty = self.func.dfg.value_type(*val); - self.func.dfg.append_ebb_param(dest_ebb, ty); - } - } else { - // The Ebb already has predecessors - // We check that the arguments supplied match those supplied - // previously. - debug_assert_eq!( - jump_args.len(), - self.builder.ebbs[dest_ebb].user_arg_count, - "the jump instruction doesn't have the same \ - number of arguments as its destination Ebb.", - ); - debug_assert!( - jump_args - .iter() - .zip(self.func.dfg.ebb_params(dest_ebb).iter().take( - self.builder.ebbs[dest_ebb].user_arg_count, - )) - .all(|(jump_arg, dest_arg)| { - self.func.dfg.value_type(*jump_arg) == self.func.dfg.value_type(*dest_arg) - }), - "the jump argument supplied has not the \ - same type as the corresponding dest ebb argument" - ); - } - } - fn handle_ssa_side_effects(&mut self, side_effects: SideEffects) { for split_ebb in side_effects.split_ebbs_created { self.builder.ebbs[split_ebb].filled = true @@ -703,13 +601,14 @@ mod tests { builder.declare_var(x, I32); builder.declare_var(y, I32); builder.declare_var(z, I32); + builder.append_ebb_params_for_function_params(block0); builder.switch_to_block(block0); if !lazy_seal { builder.seal_block(block0); } { - let tmp = builder.param_value(0); + let tmp = builder.ebb_params(block0)[0]; // the first function parameter builder.def_var(x, tmp); } { diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 9b23e80a66..ad76797129 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -75,11 +75,12 @@ //! builder.declare_var(x, I32); //! builder.declare_var(y, I32); //! builder.declare_var(z, I32); +//! builder.append_ebb_params_for_function_params(block0); //! //! builder.switch_to_block(block0); //! builder.seal_block(block0); //! { -//! let tmp = builder.param_value(0); +//! let tmp = builder.ebb_params(block0)[0]; // the first function parameter //! builder.def_var(x, tmp); //! } //! { diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index acdd400a8f..fd784a4491 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -6,7 +6,7 @@ use code_translator::translate_operator; use cretonne::entity::EntityRef; -use cretonne::ir::{self, InstBuilder}; +use cretonne::ir::{self, InstBuilder, Ebb}; use cretonne::result::{CtonResult, CtonError}; use cton_frontend::{ILBuilder, FunctionBuilder}; use environ::FuncEnvironment; @@ -82,17 +82,19 @@ impl FuncTranslator { // This clears the `ILBuilder`. let mut builder = FunctionBuilder::new(func, &mut self.il_builder); let entry_block = builder.create_ebb(); + builder.append_ebb_params_for_function_params(entry_block); builder.switch_to_block(entry_block); // This also creates values for the arguments. builder.seal_block(entry_block); // Make sure the entry block is inserted in the layout before we make any callbacks to // `environ`. The callback functions may need to insert things in the entry block. builder.ensure_inserted_ebb(); - let num_params = declare_wasm_parameters(&mut builder); + let num_params = declare_wasm_parameters(&mut builder, entry_block); // Set up the translation state with a single pushed control block representing the whole // function and its return values. let exit_block = builder.create_ebb(); + builder.append_ebb_params_for_function_returns(exit_block); self.state.initialize(&builder.func.signature, exit_block); parse_local_decls(&mut reader, &mut builder, num_params)?; @@ -103,7 +105,7 @@ impl FuncTranslator { /// Declare local variables for the signature parameters that correspond to WebAssembly locals. /// /// Return the number of local variables declared. -fn declare_wasm_parameters(builder: &mut FunctionBuilder) -> usize { +fn declare_wasm_parameters(builder: &mut FunctionBuilder, entry_block: Ebb) -> usize { let sig_len = builder.func.signature.params.len(); let mut next_local = 0; for i in 0..sig_len { @@ -116,7 +118,7 @@ fn declare_wasm_parameters(builder: &mut FunctionBuilder) -> usize { builder.declare_var(local, param_type.value_type); next_local += 1; - let param_value = builder.param_value(i); + let param_value = builder.ebb_params(entry_block)[i]; builder.def_var(local, param_value); } } From cbb99bf233f4bec60d924fb0040e2def856e9f4f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 7 Nov 2017 14:56:32 -0800 Subject: [PATCH 1337/3084] Rename "user_arg_count" to "user_param_count" for consistency. --- lib/frontend/src/frontend.rs | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 95900b8166..69d9473a58 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -44,7 +44,7 @@ where struct EbbData { filled: bool, pristine: bool, - user_arg_count: usize, + user_param_count: usize, } struct Position { @@ -232,7 +232,7 @@ where self.builder.ebbs[ebb] = EbbData { filled: false, pristine: true, - user_arg_count: 0, + user_param_count: 0, }; ebb } @@ -389,9 +389,9 @@ where /// parameters. This can be used to set up the ebb parameters for the /// entry block. pub fn append_ebb_params_for_function_params(&mut self, ebb: Ebb) { - let user_arg_count = &mut self.builder.ebbs[ebb].user_arg_count; + let user_param_count = &mut self.builder.ebbs[ebb].user_param_count; for argtyp in &self.func.signature.params { - *user_arg_count += 1; + *user_param_count += 1; self.func.dfg.append_ebb_param(ebb, argtyp.value_type); } } @@ -400,9 +400,9 @@ where /// return values. This can be used to set up the ebb parameters for a /// function exit block. pub fn append_ebb_params_for_function_returns(&mut self, ebb: Ebb) { - let user_arg_count = &mut self.builder.ebbs[ebb].user_arg_count; + let user_param_count = &mut self.builder.ebbs[ebb].user_param_count; for argtyp in &self.func.signature.returns { - *user_arg_count += 1; + *user_param_count += 1; self.func.dfg.append_ebb_param(ebb, argtyp.value_type); } } @@ -436,10 +436,10 @@ where pub fn append_ebb_param(&mut self, ebb: Ebb, ty: Type) -> Value { debug_assert!(self.builder.ebbs[ebb].pristine); debug_assert_eq!( - self.builder.ebbs[ebb].user_arg_count, + self.builder.ebbs[ebb].user_param_count, self.func.dfg.num_ebb_params(ebb) ); - self.builder.ebbs[ebb].user_arg_count += 1; + self.builder.ebbs[ebb].user_param_count += 1; self.func.dfg.append_ebb_param(ebb, ty) } From 6d91a32d396aef926ae82ebc252c79b5023395e0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 7 Nov 2017 14:57:35 -0800 Subject: [PATCH 1338/3084] Add comments about "user" parameters. --- lib/frontend/src/frontend.rs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 69d9473a58..ac86e3d64f 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -389,6 +389,8 @@ where /// parameters. This can be used to set up the ebb parameters for the /// entry block. pub fn append_ebb_params_for_function_params(&mut self, ebb: Ebb) { + // These parameters count as "user" parameters here because they aren't + // inserted by the SSABuilder. let user_param_count = &mut self.builder.ebbs[ebb].user_param_count; for argtyp in &self.func.signature.params { *user_param_count += 1; @@ -400,6 +402,8 @@ where /// return values. This can be used to set up the ebb parameters for a /// function exit block. pub fn append_ebb_params_for_function_returns(&mut self, ebb: Ebb) { + // These parameters count as "user" parameters here because they aren't + // inserted by the SSABuilder. let user_param_count = &mut self.builder.ebbs[ebb].user_param_count; for argtyp in &self.func.signature.returns { *user_param_count += 1; From 15aff899ee5f30be81a0d7a915ed796044c1fb20 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 7 Nov 2017 15:19:55 -0800 Subject: [PATCH 1339/3084] Block no longer needs to implement Hash. --- lib/frontend/src/ssa.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index c7c4d08029..2b758ef18b 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -124,7 +124,7 @@ struct EbbHeaderBlockData { } /// A opaque reference to a basic block. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Debug)] pub struct Block(u32); impl EntityRef for Block { fn new(index: usize) -> Self { From 849f090562eac5574f8412091c5b4173bc609970 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 7 Nov 2017 15:33:35 -0800 Subject: [PATCH 1340/3084] Fix nondeterminism in br_table translation. --- lib/wasm/src/code_translator.rs | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 040a53fe5d..6528416b28 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -22,7 +22,7 @@ //! //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as //! argument. -use cretonne::ir::{self, InstBuilder, Ebb, MemFlags, JumpTableData}; +use cretonne::ir::{self, InstBuilder, MemFlags, JumpTableData}; use cretonne::ir::types::*; use cretonne::ir::condcodes::{IntCC, FloatCC}; use cton_frontend::FunctionBuilder; @@ -30,7 +30,7 @@ use wasmparser::{Operator, MemoryImmediate}; use translation_utils::{f32_translation, f64_translation, type_to_type, num_return_values, Local}; use translation_utils::{TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; use state::{TranslationState, ControlStackFrame}; -use std::collections::HashMap; +use std::collections::{HashMap, hash_map}; use environ::{FuncEnvironment, GlobalValue}; use std::u32; @@ -297,25 +297,26 @@ pub fn translate_operator( let val = state.pop1(); let return_count = jump_args_count; let mut data = JumpTableData::with_capacity(depths.len()); - let dest_ebbs: HashMap = depths.iter().fold(HashMap::new(), |mut acc, - &depth| { - if acc.get(&(depth as usize)).is_none() { - let branch_ebb = builder.create_ebb(); - data.push_entry(branch_ebb); - acc.insert(depth as usize, branch_ebb); - return acc; + let mut dest_ebb_sequence = Vec::new(); + let mut dest_ebb_map = HashMap::new(); + for depth in depths { + let branch_ebb = match dest_ebb_map.entry(depth as usize) { + hash_map::Entry::Occupied(entry) => *entry.get(), + hash_map::Entry::Vacant(entry) => { + let ebb = builder.create_ebb(); + dest_ebb_sequence.push((depth as usize, ebb)); + *entry.insert(ebb) + } }; - let branch_ebb = acc[&(depth as usize)]; data.push_entry(branch_ebb); - acc - }); + } let jt = builder.create_jump_table(data); builder.ins().br_table(val, jt); let default_ebb = state.control_stack[state.control_stack.len() - 1 - (default as usize)] .br_destination(); builder.ins().jump(default_ebb, state.peekn(return_count)); - for (depth, dest_ebb) in dest_ebbs { + for (depth, dest_ebb) in dest_ebb_sequence { builder.switch_to_block(dest_ebb); builder.seal_block(dest_ebb); let i = state.control_stack.len() - 1 - depth; From 2b6502ac6e634bc1a2a68b47849dffd9302e38a2 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 7 Nov 2017 16:57:22 -0800 Subject: [PATCH 1341/3084] Fix handling of CFG triangles in compute_postorder. For example, in `loops_one`, ebb3 is the bottom of a triangle, so postorder should order it after the rest of the triangle. --- cranelift/tests/cfg_traversal.rs | 8 +++--- lib/cretonne/src/dominator_tree.rs | 40 +++++++++++------------------- lib/cretonne/src/loop_analysis.rs | 12 ++++----- 3 files changed, 24 insertions(+), 36 deletions(-) diff --git a/cranelift/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs index 90fb900dfe..73b4b9fad4 100644 --- a/cranelift/tests/cfg_traversal.rs +++ b/cranelift/tests/cfg_traversal.rs @@ -51,7 +51,7 @@ fn simple_traversal() { trap user0 } ", - vec![0, 1, 3, 2, 4, 5], + vec![0, 2, 5, 4, 1, 3], ); } @@ -71,7 +71,7 @@ fn loops_one() { return } ", - vec![0, 1, 3, 2], + vec![0, 1, 2, 3], ); } @@ -98,7 +98,7 @@ fn loops_two() { return } ", - vec![0, 1, 2, 4, 3, 5], + vec![0, 2, 1, 3, 4, 5], ); } @@ -130,7 +130,7 @@ fn loops_three() { return } ", - vec![0, 1, 2, 4, 3, 6, 7, 5], + vec![0, 2, 1, 3, 4, 6, 7, 5], ); } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 043ebad4eb..4a3965540e 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -36,7 +36,7 @@ pub struct DominatorTree { postorder: Vec, // Scratch memory used by `compute_postorder()`. - stack: Vec, + stack: Vec<(Ebb, usize)>, valid: bool, } @@ -255,39 +255,27 @@ impl DominatorTree { // During this algorithm only, use `rpo_number` to hold the following state: // - // 0: EBB never reached. - // 2: EBB has been pushed once, so it shouldn't be pushed again. - // 1: EBB has already been popped once, and should be added to the post-order next time. - const SEEN: u32 = 2; - const DONE: u32 = 1; + // 0: EBB is not yet on the stack. + // 1: EBB is on the stack or in postorder. + const SEEN: u32 = 1; match func.layout.entry_block() { Some(ebb) => { + self.stack.push((ebb, 0)); self.nodes[ebb].rpo_number = SEEN; - self.stack.push(ebb) } None => return, } - while let Some(ebb) = self.stack.pop() { - match self.nodes[ebb].rpo_number { - // This is the first time we visit `ebb`, forming a pre-order. - SEEN => { - // Mark it as done and re-queue it to be visited after its children. - self.nodes[ebb].rpo_number = DONE; - self.stack.push(ebb); - for &succ in cfg.get_successors(ebb) { - // Only push children that haven't been seen before. - if self.nodes[succ].rpo_number == 0 { - self.nodes[succ].rpo_number = SEEN; - self.stack.push(succ); - } - } + while let Some((ebb, succ_index)) = self.stack.pop() { + if let Some(&succ) = cfg.get_successors(ebb).get(succ_index) { + self.stack.push((ebb, succ_index + 1)); + if self.nodes[succ].rpo_number == 0 { + self.stack.push((succ, 0)); + self.nodes[succ].rpo_number = SEEN; } - // This is the second time we popped `ebb`, so all its children have been visited. - // This is the post-order. - DONE => self.postorder.push(ebb), - _ => panic!("Inconsistent stack rpo_number"), + } else { + self.postorder.push(ebb); } } } @@ -522,7 +510,7 @@ mod test { Ordering::Less ); - assert_eq!(dt.cfg_postorder(), &[ebb2, ebb0, ebb1, ebb3]); + assert_eq!(dt.cfg_postorder(), &[ebb0, ebb2, ebb1, ebb3]); } #[test] diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index 95d3bace4f..c994ff3027 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -326,16 +326,16 @@ mod test { let loops = loop_analysis.loops().collect::>(); assert_eq!(loops.len(), 3); assert_eq!(loop_analysis.loop_header(loops[0]), ebb0); - assert_eq!(loop_analysis.loop_header(loops[1]), ebb1); - assert_eq!(loop_analysis.loop_header(loops[2]), ebb3); + assert_eq!(loop_analysis.loop_header(loops[1]), ebb3); + assert_eq!(loop_analysis.loop_header(loops[2]), ebb1); assert_eq!(loop_analysis.loop_parent(loops[1]), Some(loops[0])); assert_eq!(loop_analysis.loop_parent(loops[2]), Some(loops[0])); assert_eq!(loop_analysis.loop_parent(loops[0]), None); assert_eq!(loop_analysis.is_in_loop(ebb0, loops[0]), true); - assert_eq!(loop_analysis.is_in_loop(ebb1, loops[1]), true); - assert_eq!(loop_analysis.is_in_loop(ebb2, loops[1]), true); - assert_eq!(loop_analysis.is_in_loop(ebb3, loops[2]), true); - assert_eq!(loop_analysis.is_in_loop(ebb4, loops[2]), true); + assert_eq!(loop_analysis.is_in_loop(ebb3, loops[1]), true); + assert_eq!(loop_analysis.is_in_loop(ebb4, loops[1]), true); + assert_eq!(loop_analysis.is_in_loop(ebb1, loops[2]), true); + assert_eq!(loop_analysis.is_in_loop(ebb2, loops[2]), true); assert_eq!(loop_analysis.is_in_loop(ebb5, loops[0]), true); } } From d7796cbf25b6916d41e229895abb342ce9915a4a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 10:40:29 -0800 Subject: [PATCH 1342/3084] Suppress an unused-import warning on AsciiExt. See https://users.rust-lang.org/t/psa-dealing-with-warning-unused-import-std-ascii-asciiext-in-today-s-nightly/13726 for details. --- lib/cretonne/src/ir/extname.rs | 1 + lib/reader/src/lexer.rs | 1 + 2 files changed, 2 insertions(+) diff --git a/lib/cretonne/src/ir/extname.rs b/lib/cretonne/src/ir/extname.rs index 273006b570..ba5ddba40b 100644 --- a/lib/cretonne/src/ir/extname.rs +++ b/lib/cretonne/src/ir/extname.rs @@ -5,6 +5,7 @@ //! Cretonne, which compiles functions independently. use std::fmt::{self, Write}; +#[allow(unused_imports)] use std::ascii::AsciiExt; /// The name of an external can be any sequence of bytes. diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 10cd9737af..c8b0ca96ad 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -7,6 +7,7 @@ use std::str::CharIndices; use std::u16; +#[allow(unused_imports)] use std::ascii::AsciiExt; use cretonne::ir::types; use cretonne::ir::{Value, Ebb}; From 24e185c8c5451d030aef9be604adcc8a6e1eaf92 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 10:40:37 -0800 Subject: [PATCH 1343/3084] Remove unnecessary parens. --- lib/native/src/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index 49cb173383..6cdb39a15e 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -38,7 +38,7 @@ pub fn builders() -> Result<(settings::Builder, isa::Builder), ()> { let mut isa_builder = isa::lookup(name).map_err(|err| match err { isa::LookupError::Unknown => panic!(), - isa::LookupError::Unsupported => (()), + isa::LookupError::Unsupported => (), })?; if cfg!(any(target_arch = "x86", target_arch = "x86_64")) { From 8b053aa923ef28559c3816ab71bae9f4ca9fbc40 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 10:40:39 -0800 Subject: [PATCH 1344/3084] Remove a trivial cast. --- lib/cretonne/src/regalloc/solver.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 39f6e3a83c..69b7bd1dd3 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -421,7 +421,8 @@ impl fmt::Display for Move { impl fmt::Debug for Move { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - (self as &fmt::Display).fmt(f) + let as_display: &fmt::Display = self; + as_display.fmt(f) } } From 388ebde41de8f49b504fee38cb6c07befc3ae2a4 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 10:40:42 -0800 Subject: [PATCH 1345/3084] Use .any(...) rather than .position(...).is_some(). --- lib/cretonne/src/verifier/mod.rs | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 342e501b0a..c2753d1335 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -979,9 +979,7 @@ impl<'a> Verifier<'a> { ); } - let has_valid_encoding = encodings - .position(|possible_enc| encoding == possible_enc) - .is_some(); + let has_valid_encoding = encodings.any(|possible_enc| encoding == possible_enc); if !has_valid_encoding { let mut possible_encodings = String::new(); From b7f979a8be432b0fa20d2802ec418074ba7e5281 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 10:40:43 -0800 Subject: [PATCH 1346/3084] Combine identical match arms. --- lib/wasm/src/code_translator.rs | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 6528416b28..133004aa13 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -486,26 +486,15 @@ pub fn translate_operator( state.push1(builder.ins().f64const(f64_translation(value))); } /******************************* Unary Operators *************************************/ - Operator::I32Clz => { + Operator::I32Clz | Operator::I64Clz => { let arg = state.pop1(); state.push1(builder.ins().clz(arg)); } - Operator::I64Clz => { - let arg = state.pop1(); - state.push1(builder.ins().clz(arg)); - } - Operator::I32Ctz => { + Operator::I32Ctz | Operator::I64Ctz => { let arg = state.pop1(); state.push1(builder.ins().ctz(arg)); } - Operator::I64Ctz => { - let arg = state.pop1(); - state.push1(builder.ins().ctz(arg)); - } - Operator::I32Popcnt => { - let arg = state.pop1(); - state.push1(builder.ins().popcnt(arg)); - } + Operator::I32Popcnt | Operator::I64Popcnt => { let arg = state.pop1(); state.push1(builder.ins().popcnt(arg)); From 3ab4349c1bdc2a13767371f771d33d7d722b0a81 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 10:40:47 -0800 Subject: [PATCH 1347/3084] Use `Self` instead of repeating the type name. --- cranelift/src/filetest/binemit.rs | 4 ++-- cranelift/src/filetest/concurrent.rs | 4 ++-- cranelift/src/filetest/runner.rs | 4 ++-- lib/cretonne/src/bitset.rs | 2 +- lib/cretonne/src/context.rs | 4 ++-- lib/cretonne/src/dominator_tree.rs | 8 +++---- lib/cretonne/src/entity/keys.rs | 4 ++-- lib/cretonne/src/entity/list.rs | 10 ++++---- lib/cretonne/src/entity/map.rs | 4 ++-- lib/cretonne/src/entity/primary.rs | 2 +- lib/cretonne/src/entity/set.rs | 2 +- lib/cretonne/src/entity/sparse.rs | 4 ++-- lib/cretonne/src/flowgraph.rs | 8 +++---- lib/cretonne/src/ir/dfg.rs | 8 +++---- lib/cretonne/src/ir/extfunc.rs | 24 +++++++++---------- lib/cretonne/src/ir/function.rs | 6 ++--- lib/cretonne/src/ir/instructions.rs | 6 ++--- lib/cretonne/src/ir/jumptable.rs | 8 +++---- lib/cretonne/src/ir/layout.rs | 4 ++-- lib/cretonne/src/ir/memflags.rs | 4 ++-- lib/cretonne/src/ir/sourceloc.rs | 2 +- lib/cretonne/src/ir/stackslot.rs | 4 ++-- lib/cretonne/src/ir/types.rs | 2 +- lib/cretonne/src/isa/registers.rs | 2 +- lib/cretonne/src/loop_analysis.rs | 4 ++-- lib/cretonne/src/regalloc/allocatable_set.rs | 4 ++-- lib/cretonne/src/regalloc/coalescing.rs | 8 +++---- lib/cretonne/src/regalloc/coloring.rs | 4 ++-- lib/cretonne/src/regalloc/context.rs | 4 ++-- lib/cretonne/src/regalloc/diversion.rs | 4 ++-- .../src/regalloc/live_value_tracker.rs | 8 +++---- lib/cretonne/src/regalloc/liveness.rs | 4 ++-- lib/cretonne/src/regalloc/reload.rs | 4 ++-- lib/cretonne/src/regalloc/solver.rs | 4 ++-- lib/cretonne/src/regalloc/spilling.rs | 4 ++-- lib/cretonne/src/regalloc/virtregs.rs | 4 ++-- lib/cretonne/src/topo_order.rs | 4 ++-- lib/filecheck/src/checker.rs | 8 +++---- lib/filecheck/src/pattern.rs | 4 ++-- lib/wasm/src/func_translator.rs | 4 ++-- lib/wasm/src/state.rs | 4 ++-- 41 files changed, 105 insertions(+), 105 deletions(-) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index cc8dcad0b7..9c19533c64 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -35,8 +35,8 @@ struct TextSink { impl TextSink { /// Create a new empty TextSink. - pub fn new(isa: &TargetIsa) -> TextSink { - TextSink { + pub fn new(isa: &TargetIsa) -> Self { + Self { rnames: isa.reloc_names(), offset: 0, text: String::new(), diff --git a/cranelift/src/filetest/concurrent.rs b/cranelift/src/filetest/concurrent.rs index 430fb54142..4f7cca6935 100644 --- a/cranelift/src/filetest/concurrent.rs +++ b/cranelift/src/filetest/concurrent.rs @@ -38,7 +38,7 @@ pub struct ConcurrentRunner { impl ConcurrentRunner { /// Create a new `ConcurrentRunner` with threads spun up. - pub fn new() -> ConcurrentRunner { + pub fn new() -> Self { let (request_tx, request_rx) = channel(); let request_mutex = Arc::new(Mutex::new(request_rx)); let (reply_tx, reply_rx) = channel(); @@ -51,7 +51,7 @@ impl ConcurrentRunner { }) .collect(); - ConcurrentRunner { + Self { request_tx: Some(request_tx), reply_rx, handles, diff --git a/cranelift/src/filetest/runner.rs b/cranelift/src/filetest/runner.rs index 6425482084..8e0f7721a7 100644 --- a/cranelift/src/filetest/runner.rs +++ b/cranelift/src/filetest/runner.rs @@ -81,8 +81,8 @@ pub struct TestRunner { impl TestRunner { /// Create a new blank TrstRunner. - pub fn new(verbose: bool) -> TestRunner { - TestRunner { + pub fn new(verbose: bool) -> Self { + Self { verbose, dir_stack: Vec::new(), tests: Vec::new(), diff --git a/lib/cretonne/src/bitset.rs b/lib/cretonne/src/bitset.rs index 2ed4efe15b..62e0286bb3 100644 --- a/lib/cretonne/src/bitset.rs +++ b/lib/cretonne/src/bitset.rs @@ -61,7 +61,7 @@ where } /// Construct a BitSet with the half-open range [lo,hi) filled in - pub fn from_range(lo: u8, hi: u8) -> BitSet { + pub fn from_range(lo: u8, hi: u8) -> Self { assert!(lo <= hi); assert!((hi as usize) <= Self::bits()); let one: T = T::from(1); diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index ca2cd43854..6f7f7a65c1 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -47,8 +47,8 @@ impl Context { /// /// The returned instance should be reused for compiling multiple functions in order to avoid /// needless allocator thrashing. - pub fn new() -> Context { - Context { + pub fn new() -> Self { + Self { func: Function::new(), cfg: ControlFlowGraph::new(), domtree: DominatorTree::new(), diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 4a3965540e..2c2fd1b48c 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -204,8 +204,8 @@ impl DominatorTree { impl DominatorTree { /// Allocate a new blank dominator tree. Use `compute` to compute the dominator tree for a /// function. - pub fn new() -> DominatorTree { - DominatorTree { + pub fn new() -> Self { + Self { nodes: EntityMap::new(), postorder: Vec::new(), stack: Vec::new(), @@ -214,8 +214,8 @@ impl DominatorTree { } /// Allocate and compute a dominator tree. - pub fn with_function(func: &Function, cfg: &ControlFlowGraph) -> DominatorTree { - let mut domtree = DominatorTree::new(); + pub fn with_function(func: &Function, cfg: &ControlFlowGraph) -> Self { + let mut domtree = Self::new(); domtree.compute(func, cfg); domtree } diff --git a/lib/cretonne/src/entity/keys.rs b/lib/cretonne/src/entity/keys.rs index 795e896a00..3ad5b660b6 100644 --- a/lib/cretonne/src/entity/keys.rs +++ b/lib/cretonne/src/entity/keys.rs @@ -12,8 +12,8 @@ pub struct Keys { impl Keys { /// Create a `Keys` iterator that visits `count` entities starting from 0. - pub fn new(count: usize) -> Keys { - Keys { + pub fn new(count: usize) -> Self { + Self { pos: 0, rev_pos: count, unused: PhantomData, diff --git a/lib/cretonne/src/entity/list.rs b/lib/cretonne/src/entity/list.rs index 03c66a8ec0..c9bb5b1a69 100644 --- a/lib/cretonne/src/entity/list.rs +++ b/lib/cretonne/src/entity/list.rs @@ -68,7 +68,7 @@ pub struct EntityList { /// Create an empty list. impl Default for EntityList { fn default() -> Self { - EntityList { + Self { index: 0, unused: PhantomData, } @@ -82,7 +82,7 @@ impl Hash for EntityList { } impl PartialEq for EntityList { - fn eq(&self, _: &EntityList) -> bool { + fn eq(&self, _: &Self) -> bool { panic!("eq called on EntityList"); } } @@ -121,8 +121,8 @@ fn is_sclass_min_length(len: usize) -> bool { impl ListPool { /// Create a new list pool. - pub fn new() -> ListPool { - ListPool { + pub fn new() -> Self { + Self { data: Vec::new(), free: Vec::new(), } @@ -311,7 +311,7 @@ impl EntityList { /// Take all elements from this list and return them as a new list. Leave this list empty. /// /// This is the equivalent of `Option::take()`. - pub fn take(&mut self) -> EntityList { + pub fn take(&mut self) -> Self { mem::replace(self, Default::default()) } diff --git a/lib/cretonne/src/entity/map.rs b/lib/cretonne/src/entity/map.rs index 200aa464c3..9622f93485 100644 --- a/lib/cretonne/src/entity/map.rs +++ b/lib/cretonne/src/entity/map.rs @@ -34,7 +34,7 @@ where where V: Default, { - EntityMap { + Self { elems: Vec::new(), default: Default::default(), unused: PhantomData, @@ -45,7 +45,7 @@ where /// /// This constructor does not require V to implement Default. pub fn with_default(default: V) -> Self { - EntityMap { + Self { elems: Vec::new(), default: default, unused: PhantomData, diff --git a/lib/cretonne/src/entity/primary.rs b/lib/cretonne/src/entity/primary.rs index c0347304aa..137320f3e5 100644 --- a/lib/cretonne/src/entity/primary.rs +++ b/lib/cretonne/src/entity/primary.rs @@ -27,7 +27,7 @@ where { /// Create a new empty map. pub fn new() -> Self { - PrimaryMap { + Self { elems: Vec::new(), unused: PhantomData, } diff --git a/lib/cretonne/src/entity/set.rs b/lib/cretonne/src/entity/set.rs index b1c351b650..8b5d32e790 100644 --- a/lib/cretonne/src/entity/set.rs +++ b/lib/cretonne/src/entity/set.rs @@ -24,7 +24,7 @@ where { /// Create a new empty set. pub fn new() -> Self { - EntitySet { + Self { elems: Vec::new(), len: 0, unused: PhantomData, diff --git a/lib/cretonne/src/entity/sparse.rs b/lib/cretonne/src/entity/sparse.rs index 26be799afd..dacbf66031 100644 --- a/lib/cretonne/src/entity/sparse.rs +++ b/lib/cretonne/src/entity/sparse.rs @@ -66,7 +66,7 @@ where { /// Create a new empty mapping. pub fn new() -> Self { - SparseMap { + Self { sparse: EntityMap::new(), dense: Vec::new(), } @@ -215,7 +215,7 @@ impl SparseMapValue for T where T: EntityRef, { - fn key(&self) -> T { + fn key(&self) -> Self { *self } } diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index 67ff150196..bc6096ad9a 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -51,16 +51,16 @@ pub struct ControlFlowGraph { impl ControlFlowGraph { /// Allocate a new blank control flow graph. - pub fn new() -> ControlFlowGraph { - ControlFlowGraph { + pub fn new() -> Self { + Self { data: EntityMap::new(), valid: false, } } /// Allocate and compute the control flow graph for `func`. - pub fn with_function(func: &Function) -> ControlFlowGraph { - let mut cfg = ControlFlowGraph::new(); + pub fn with_function(func: &Function) -> Self { + let mut cfg = Self::new(); cfg.compute(func); cfg } diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 83a32f7ff8..e2328e46dc 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -62,8 +62,8 @@ pub struct DataFlowGraph { impl DataFlowGraph { /// Create a new empty `DataFlowGraph`. - pub fn new() -> DataFlowGraph { - DataFlowGraph { + pub fn new() -> Self { + Self { insts: PrimaryMap::new(), results: EntityMap::new(), ebbs: PrimaryMap::new(), @@ -829,8 +829,8 @@ struct EbbData { } impl EbbData { - fn new() -> EbbData { - EbbData { params: ValueList::new() } + fn new() -> Self { + Self { params: ValueList::new() } } } diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 4403b6ebef..b3c3a274e7 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -38,8 +38,8 @@ pub struct Signature { impl Signature { /// Create a new blank signature. - pub fn new(call_conv: CallConv) -> Signature { - Signature { + pub fn new(call_conv: CallConv) -> Self { + Self { params: Vec::new(), returns: Vec::new(), call_conv, @@ -138,8 +138,8 @@ pub struct AbiParam { impl AbiParam { /// Create a parameter with default flags. - pub fn new(vt: Type) -> AbiParam { - AbiParam { + pub fn new(vt: Type) -> Self { + Self { value_type: vt, extension: ArgumentExtension::None, purpose: ArgumentPurpose::Normal, @@ -148,8 +148,8 @@ impl AbiParam { } /// Create a special-purpose parameter that is not (yet) bound to a specific register. - pub fn special(vt: Type, purpose: ArgumentPurpose) -> AbiParam { - AbiParam { + pub fn special(vt: Type, purpose: ArgumentPurpose) -> Self { + Self { value_type: vt, extension: ArgumentExtension::None, purpose, @@ -158,8 +158,8 @@ impl AbiParam { } /// Create a parameter for a special-purpose register. - pub fn special_reg(vt: Type, purpose: ArgumentPurpose, regunit: RegUnit) -> AbiParam { - AbiParam { + pub fn special_reg(vt: Type, purpose: ArgumentPurpose, regunit: RegUnit) -> Self { + Self { value_type: vt, extension: ArgumentExtension::None, purpose, @@ -168,18 +168,18 @@ impl AbiParam { } /// Convert `self` to a parameter with the `uext` flag set. - pub fn uext(self) -> AbiParam { + pub fn uext(self) -> Self { debug_assert!(self.value_type.is_int(), "uext on {} arg", self.value_type); - AbiParam { + Self { extension: ArgumentExtension::Uext, ..self } } /// Convert `self` to a parameter type with the `sext` flag set. - pub fn sext(self) -> AbiParam { + pub fn sext(self) -> Self { debug_assert!(self.value_type.is_int(), "sext on {} arg", self.value_type); - AbiParam { + Self { extension: ArgumentExtension::Sext, ..self } diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 4c348b6ce9..457b47a5e0 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -66,8 +66,8 @@ pub struct Function { impl Function { /// Create a function with the given name and signature. - pub fn with_name_signature(name: ExternalName, sig: Signature) -> Function { - Function { + pub fn with_name_signature(name: ExternalName, sig: Signature) -> Self { + Self { name, signature: sig, stack_slots: StackSlots::new(), @@ -99,7 +99,7 @@ impl Function { } /// Create a new empty, anonymous function with a native calling convention. - pub fn new() -> Function { + pub fn new() -> Self { Self::with_name_signature(ExternalName::default(), Signature::new(CallConv::Native)) } diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 6c101d751e..0038fa42c4 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -262,7 +262,7 @@ pub struct VariableArgs(Vec); impl VariableArgs { /// Create an empty argument list. - pub fn new() -> VariableArgs { + pub fn new() -> Self { VariableArgs(Vec::new()) } @@ -314,8 +314,8 @@ impl Display for VariableArgs { } impl Default for VariableArgs { - fn default() -> VariableArgs { - VariableArgs::new() + fn default() -> Self { + Self::new() } } diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index a6cfad3346..2d09aab4da 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -24,16 +24,16 @@ pub struct JumpTableData { impl JumpTableData { /// Create a new empty jump table. - pub fn new() -> JumpTableData { - JumpTableData { + pub fn new() -> Self { + Self { table: Vec::new(), holes: 0, } } /// Create a new empty jump table with the specified capacity. - pub fn with_capacity(capacity: usize) -> JumpTableData { - JumpTableData { + pub fn with_capacity(capacity: usize) -> Self { + Self { table: Vec::with_capacity(capacity), holes: 0, } diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 8bb61cb11f..551055394a 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -42,8 +42,8 @@ pub struct Layout { impl Layout { /// Create a new empty `Layout`. - pub fn new() -> Layout { - Layout { + pub fn new() -> Self { + Self { ebbs: EntityMap::new(), insts: EntityMap::new(), first_ebb: None, diff --git a/lib/cretonne/src/ir/memflags.rs b/lib/cretonne/src/ir/memflags.rs index fd836d2752..e87fb3c4ec 100644 --- a/lib/cretonne/src/ir/memflags.rs +++ b/lib/cretonne/src/ir/memflags.rs @@ -21,8 +21,8 @@ pub struct MemFlags { impl MemFlags { /// Create a new empty set of flags. - pub fn new() -> MemFlags { - MemFlags { bits: 0 } + pub fn new() -> Self { + Self { bits: 0 } } /// Read a flag bit. diff --git a/lib/cretonne/src/ir/sourceloc.rs b/lib/cretonne/src/ir/sourceloc.rs index 104bc541b6..ffcf0db943 100644 --- a/lib/cretonne/src/ir/sourceloc.rs +++ b/lib/cretonne/src/ir/sourceloc.rs @@ -33,7 +33,7 @@ impl SourceLoc { } impl Default for SourceLoc { - fn default() -> SourceLoc { + fn default() -> Self { SourceLoc(!0) } } diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index f46f91dd74..dc02f0df6b 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -172,8 +172,8 @@ pub struct StackSlots { /// Stack slot manager functions that behave mostly like an entity map. impl StackSlots { /// Create an empty stack slot manager. - pub fn new() -> StackSlots { - StackSlots { + pub fn new() -> Self { + Self { slots: PrimaryMap::new(), outgoing: Vec::new(), emergency: Vec::new(), diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index e0cdcd02ce..da502f9bf7 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -321,7 +321,7 @@ impl Debug for Type { } impl Default for Type { - fn default() -> Type { + fn default() -> Self { VOID } } diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index f21c946b99..55ec2bb8a8 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -214,7 +214,7 @@ impl fmt::Debug for RegClassData { /// Within an ISA, register classes are uniquely identified by their index. impl PartialEq for RegClassData { - fn eq(&self, other: &RegClassData) -> bool { + fn eq(&self, other: &Self) -> bool { self.index == other.index } } diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index c994ff3027..e6390aa100 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -42,8 +42,8 @@ impl LoopData { impl LoopAnalysis { /// Allocate a new blank loop analysis struct. Use `compute` to compute the loop analysis for /// a function. - pub fn new() -> LoopAnalysis { - LoopAnalysis { + pub fn new() -> Self { + Self { valid: false, loops: PrimaryMap::new(), ebb_loop_map: EntityMap::new(), diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index 355443affc..0e8e559cca 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -37,8 +37,8 @@ impl AllocatableSet { /// /// Note that this includes *all* registers. Query the `TargetIsa` object to get a set of /// allocatable registers where reserved registers have been filtered out. - pub fn new() -> AllocatableSet { - AllocatableSet { avail: [!0; 3] } + pub fn new() -> Self { + Self { avail: [!0; 3] } } /// Returns `true` if the specified register is available. diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 28cde8c10c..cd54f2ece2 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -74,8 +74,8 @@ impl Node { impl DomForest { /// Create a new empty dominator forest. - pub fn new() -> DomForest { - DomForest { + pub fn new() -> Self { + Self { values: Vec::new(), stack: Vec::new(), } @@ -252,8 +252,8 @@ struct Context<'a> { impl Coalescing { /// Create a new coalescing pass. - pub fn new() -> Coalescing { - Coalescing { + pub fn new() -> Self { + Self { forest: DomForest::new(), values: Vec::new(), split_values: Vec::new(), diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index d8607e4bd1..ff5bc0ef69 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -101,8 +101,8 @@ struct Context<'a> { impl Coloring { /// Allocate scratch space data structures for the coloring pass. - pub fn new() -> Coloring { - Coloring { + pub fn new() -> Self { + Self { divert: RegDiversions::new(), solver: Solver::new(), } diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 06d27ff325..a35ddb3484 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -36,8 +36,8 @@ impl Context { /// /// This context should be reused for multiple functions in order to avoid repeated memory /// allocations. - pub fn new() -> Context { - Context { + pub fn new() -> Self { + Self { liveness: Liveness::new(), virtregs: VirtRegs::new(), coalescing: Coalescing::new(), diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index c6d25f543e..588ad040a5 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -44,8 +44,8 @@ pub struct RegDiversions { impl RegDiversions { /// Create a new empty diversion tracker. - pub fn new() -> RegDiversions { - RegDiversions { current: Vec::new() } + pub fn new() -> Self { + Self { current: Vec::new() } } /// Clear the tracker, preparing for a new EBB. diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 10fe9ff595..cb57683661 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -65,8 +65,8 @@ struct LiveValueVec { } impl LiveValueVec { - fn new() -> LiveValueVec { - LiveValueVec { + fn new() -> Self { + Self { values: Vec::new(), live_prefix: None, } @@ -124,8 +124,8 @@ impl LiveValueVec { impl LiveValueTracker { /// Create a new blank tracker. - pub fn new() -> LiveValueTracker { - LiveValueTracker { + pub fn new() -> Self { + Self { live: LiveValueVec::new(), idom_sets: HashMap::new(), idom_pool: ListPool::new(), diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 0a449bd85c..e5face5ef4 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -299,8 +299,8 @@ impl Liveness { /// /// The memory allocated for this analysis can be reused for multiple functions. Use the /// `compute` method to actually runs the analysis for a function. - pub fn new() -> Liveness { - Liveness { + pub fn new() -> Self { + Self { ranges: LiveRangeSet::new(), worklist: Vec::new(), ebb_params: Vec::new(), diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 97b849f3a9..9b0006b990 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -46,8 +46,8 @@ struct Context<'a> { impl Reload { /// Create a new blank reload pass. - pub fn new() -> Reload { - Reload { + pub fn new() -> Self { + Self { candidates: Vec::new(), reloads: SparseMap::new(), } diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 69b7bd1dd3..a18599a127 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -502,8 +502,8 @@ pub struct Solver { /// Interface for programming the constraints into the solver. impl Solver { /// Create a new empty solver. - pub fn new() -> Solver { - Solver { + pub fn new() -> Self { + Self { assignments: SparseMap::new(), vars: Vec::new(), inputs_done: false, diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 85d7b20b4a..65d2f2d076 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -63,8 +63,8 @@ struct Context<'a> { impl Spilling { /// Create a new spilling data structure. - pub fn new() -> Spilling { - Spilling { + pub fn new() -> Self { + Self { spills: Vec::new(), reg_uses: Vec::new(), } diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index 21a2a917d5..c2cb6c8ffe 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -45,8 +45,8 @@ pub struct VirtRegs { #[allow(dead_code)] impl VirtRegs { /// Create a new virtual register collection. - pub fn new() -> VirtRegs { - VirtRegs { + pub fn new() -> Self { + Self { pool: ListPool::new(), vregs: PrimaryMap::new(), value_vregs: EntityMap::new(), diff --git a/lib/cretonne/src/topo_order.rs b/lib/cretonne/src/topo_order.rs index db566ae7c8..d71dc63ccf 100644 --- a/lib/cretonne/src/topo_order.rs +++ b/lib/cretonne/src/topo_order.rs @@ -26,8 +26,8 @@ pub struct TopoOrder { impl TopoOrder { /// Create a new empty topological order. - pub fn new() -> TopoOrder { - TopoOrder { + pub fn new() -> Self { + Self { preferred: Vec::new(), next: 0, visited: SparseSet::new(), diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs index bad8e4774e..b3a68fba4c 100644 --- a/lib/filecheck/src/checker.rs +++ b/lib/filecheck/src/checker.rs @@ -95,8 +95,8 @@ pub struct CheckerBuilder { impl CheckerBuilder { /// Create a new, blank `CheckerBuilder`. - pub fn new() -> CheckerBuilder { - CheckerBuilder { + pub fn new() -> Self { + Self { directives: Vec::new(), linerx: Regex::new(DIRECTIVE_RX).unwrap(), } @@ -146,8 +146,8 @@ pub struct Checker { } impl Checker { - fn new(directives: Vec) -> Checker { - Checker { directives: directives } + fn new(directives: Vec) -> Self { + Self { directives: directives } } /// An empty checker contains no directives, and will match any input string. diff --git a/lib/filecheck/src/pattern.rs b/lib/filecheck/src/pattern.rs index 7969bc00ec..934150dca7 100644 --- a/lib/filecheck/src/pattern.rs +++ b/lib/filecheck/src/pattern.rs @@ -54,8 +54,8 @@ impl Part { impl Pattern { /// Create a new blank pattern. Use the `FromStr` trait to generate Patterns with content. - fn new() -> Pattern { - Pattern { + fn new() -> Self { + Self { parts: Vec::new(), defs: Vec::new(), } diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index fd784a4491..57821c49c5 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -30,8 +30,8 @@ pub struct FuncTranslator { impl FuncTranslator { /// Create a new translator. - pub fn new() -> FuncTranslator { - FuncTranslator { + pub fn new() -> Self { + Self { il_builder: ILBuilder::new(), state: TranslationState::new(), } diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index 91737a370c..f9f76d2934 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -126,8 +126,8 @@ pub struct TranslationState { } impl TranslationState { - pub fn new() -> TranslationState { - TranslationState { + pub fn new() -> Self { + Self { stack: Vec::new(), control_stack: Vec::new(), phantom_unreachable_stack_depth: 0, From 0d825b4643acabd670f3d11c1c6137d945076c50 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 10:40:52 -0800 Subject: [PATCH 1348/3084] Remove a redundant `.into()`. --- lib/cretonne/src/isa/mod.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index f6c95edec8..78a7e79176 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -177,7 +177,7 @@ pub trait TargetIsa { ctrl_typevar: ir::Type, ) -> Result { let mut iter = self.legal_encodings(dfg, inst, ctrl_typevar); - iter.next().ok_or_else(|| iter.legalize().into()) + iter.next().ok_or_else(|| iter.legalize()) } /// Get a data structure describing the instruction encodings in this ISA. From 809e2f0c910c7426b064f4fb262bfb61e4dfbad3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 10:40:54 -0800 Subject: [PATCH 1349/3084] Avoid unneeded return keywords. --- lib/cretonne/src/isa/stack.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/isa/stack.rs b/lib/cretonne/src/isa/stack.rs index ea7ee37248..907afe0208 100644 --- a/lib/cretonne/src/isa/stack.rs +++ b/lib/cretonne/src/isa/stack.rs @@ -41,10 +41,10 @@ impl StackRef { // Offset where SP is pointing. (All ISAs have stacks growing downwards.) let sp_offset = -(size as StackOffset); - return StackRef { + StackRef { base: StackBase::SP, offset: frame[ss].offset - sp_offset, - }; + } } } From 5d3ae0596c5265616157659935b40c78b8ff337b Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 10:40:56 -0800 Subject: [PATCH 1350/3084] Avoid matching with reference patterns. --- lib/cretonne/src/regalloc/solver.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index a18599a127..c13ec65292 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -317,10 +317,10 @@ impl Move { /// Replace the "to" register with `new` and return the old value. fn replace_to_reg(&mut self, new: RegUnit) -> RegUnit { mem::replace( - match self { - &mut Move::Reg { ref mut to, .. } | - &mut Move::Fill { ref mut to, .. } => to, - &mut Move::Spill { .. } => panic!("No to register in a spill {}", self), + match *self { + Move::Reg { ref mut to, .. } | + Move::Fill { ref mut to, .. } => to, + Move::Spill { .. } => panic!("No to register in a spill {}", self), }, new, ) From d939bc5137266288eea7f47d4c5be47182817014 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 10:40:58 -0800 Subject: [PATCH 1351/3084] Use is_empty() instead of comparing len() with 0. --- lib/cretonne/src/verifier/mod.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index c2753d1335..7d2338c821 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -991,7 +991,7 @@ impl<'a> Verifier<'a> { self.func.dfg.ctrl_typevar(inst), ) { - if possible_encodings.len() != 0 { + if !possible_encodings.is_empty() { possible_encodings.push_str(", "); multiple_encodings = true; } From 7c579a80c28b0e86a4f03c33aa7adbbcd1590581 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 10:41:00 -0800 Subject: [PATCH 1352/3084] Avoid an unneeded .into_iter(). --- cranelift/src/compile.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 828625120a..9e828cfed2 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -50,7 +50,7 @@ fn handle_module( return Err(String::from("compilation requires a target isa")); }; - for (func, _) in test_file.functions.into_iter() { + for (func, _) in test_file.functions { let mut context = Context::new(); context.func = func; context.compile(isa).map_err(|err| { From 889b06fd16678f931bae2960417f7797d422699e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 10:41:09 -0800 Subject: [PATCH 1353/3084] Replace `as` casts with type-conversion functions. --- lib/cretonne/src/entity/set.rs | 3 +- lib/cretonne/src/ir/immediates.rs | 18 +++++------ lib/cretonne/src/isa/intel/abi.rs | 3 +- lib/cretonne/src/isa/intel/enc_tables.rs | 2 +- lib/cretonne/src/isa/mod.rs | 2 +- lib/cretonne/src/isa/registers.rs | 4 +-- lib/cretonne/src/isa/riscv/abi.rs | 3 +- lib/cretonne/src/isa/riscv/binemit.rs | 33 ++++++++++---------- lib/cretonne/src/legalizer/heap.rs | 4 +-- lib/cretonne/src/legalizer/mod.rs | 2 +- lib/cretonne/src/regalloc/allocatable_set.rs | 4 ++- lib/cretonne/src/regalloc/pressure.rs | 4 +-- lib/cretonne/src/regalloc/solver.rs | 3 +- lib/reader/src/lexer.rs | 2 +- lib/wasm/src/code_translator.rs | 14 ++++----- 15 files changed, 54 insertions(+), 47 deletions(-) diff --git a/lib/cretonne/src/entity/set.rs b/lib/cretonne/src/entity/set.rs index 8b5d32e790..82dc1384ee 100644 --- a/lib/cretonne/src/entity/set.rs +++ b/lib/cretonne/src/entity/set.rs @@ -78,6 +78,7 @@ where #[cfg(test)] mod tests { use super::*; + use std::u32; // `EntityRef` impl for testing. #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -133,7 +134,7 @@ mod tests { assert!(!m.contains(E(16))); assert!(!m.contains(E(19))); assert!(!m.contains(E(20))); - assert!(!m.contains(E(u32::max_value()))); + assert!(!m.contains(E(u32::MAX))); m.clear(); assert!(m.is_empty()); diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index ecaa6d5e4d..4af9452bcf 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -163,7 +163,7 @@ impl Into for Uimm32 { impl Into for Uimm32 { fn into(self) -> i64 { - self.0 as i64 + i64::from(self.0) } } @@ -178,7 +178,7 @@ impl Display for Uimm32 { if self.0 < 10_000 { write!(f, "{}", self.0) } else { - write_hex(self.0 as i64, f) + write_hex(i64::from(self.0), f) } } @@ -189,7 +189,7 @@ impl FromStr for Uimm32 { // Parse a decimal or hexadecimal `Uimm32`, formatted as above. fn from_str(s: &str) -> Result { - parse_i64(s).and_then(|x| if 0 <= x && x <= u32::MAX as i64 { + parse_i64(s).and_then(|x| if 0 <= x && x <= i64::from(u32::MAX) { Ok(Uimm32(x as u32)) } else { Err("Uimm32 out of range") @@ -439,7 +439,7 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { let exp_str = &s3[1 + idx..]; match exp_str.parse::() { Ok(e) => { - exponent = e as i32; + exponent = i32::from(e); break; } Err(_) => return Err("Bad exponent"), @@ -473,7 +473,7 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { // Number of bits appearing after the radix point. match digits_before_period { None => {} // No radix point present. - Some(d) => exponent -= 4 * (digits - d) as i32, + Some(d) => exponent -= 4 * i32::from(digits - d), }; // Normalize the significand and exponent. @@ -485,11 +485,11 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { } // Adjust significand down. significand >>= adjust; - exponent += adjust as i32; + exponent += i32::from(adjust); } else { let adjust = t + 1 - significant_bits; significand <<= adjust; - exponent -= adjust as i32; + exponent -= i32::from(adjust); } assert_eq!(significand >> t, 1); @@ -498,7 +498,7 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { let max_exp = (1i32 << w) - 2; let bias: i32 = (1 << (w - 1)) - 1; - exponent += bias + t as i32; + exponent += bias + i32::from(t); if exponent > max_exp { Err("Magnitude too large") @@ -506,7 +506,7 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { // This is a normal number. let e_bits = (exponent as u64) << t; Ok(sign_bit | e_bits | t_bits) - } else if 1 - exponent <= t as i32 { + } else if 1 - exponent <= i32::from(t) { // This is a subnormal number: e = 0, t = significand bits. // Renormalize significand for exponent = 1. let adjust = 1 - exponent; diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 4b247b5a13..3cd3dce931 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -7,6 +7,7 @@ use settings as shared_settings; use super::registers::{GPR, FPR, RU}; use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; use ir::{AbiParam, ArgumentPurpose, ArgumentLoc, ArgumentExtension}; +use std::i32; /// Argument registers for x86-64 static ARG_GPRS: [RU; 6] = [RU::rdi, RU::rsi, RU::rdx, RU::rcx, RU::r8, RU::r9]; @@ -99,7 +100,7 @@ impl ArgAssigner for Args { // Assign a stack location. let loc = ArgumentLoc::Stack(self.offset as i32); self.offset += self.pointer_bytes; - assert!(self.offset <= i32::max_value() as u32); + assert!(self.offset <= i32::MAX as u32); loc.into() } } diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index c58a362bb0..8265b731e7 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -48,7 +48,7 @@ fn expand_srem(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGra // Now it is safe to execute the `x86_sdivmodx` instruction which will still trap on division // by zero. - let xhi = pos.ins().sshr_imm(x, ty.lane_bits() as i64 - 1); + let xhi = pos.ins().sshr_imm(x, i64::from(ty.lane_bits()) - 1); let (_qout, rem) = pos.ins().x86_sdivmodx(x, xhi, y); pos.ins().jump(done, &[rem]); diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 78a7e79176..e2f28ed119 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -244,7 +244,7 @@ pub trait TargetIsa { // Account for the SpiderMonkey standard prologue pushes. if func.signature.call_conv == ir::CallConv::SpiderWASM { - let bytes = self.flags().spiderwasm_prologue_words() as StackSize * word_size; + let bytes = StackSize::from(self.flags().spiderwasm_prologue_words()) * word_size; let mut ss = ir::StackSlotData::new(ir::StackSlotKind::IncomingArg, bytes); ss.offset = -(bytes as StackOffset); func.stack_slots.push(ss); diff --git a/lib/cretonne/src/isa/registers.rs b/lib/cretonne/src/isa/registers.rs index 55ec2bb8a8..7defead608 100644 --- a/lib/cretonne/src/isa/registers.rs +++ b/lib/cretonne/src/isa/registers.rs @@ -253,8 +253,8 @@ impl fmt::Display for RegClassIndex { /// A register is identified as a `(RegClass, RegUnit)` pair. The register class is needed to /// determine the width (in regunits) of the register. pub fn regs_overlap(rc1: RegClass, reg1: RegUnit, rc2: RegClass, reg2: RegUnit) -> bool { - let end1 = reg1 + rc1.width as RegUnit; - let end2 = reg2 + rc2.width as RegUnit; + let end1 = reg1 + RegUnit::from(rc1.width); + let end2 = reg2 + RegUnit::from(rc2.width); !(end1 <= reg2 || end2 <= reg1) } diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index 00ac99e305..52e715be43 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -12,6 +12,7 @@ use regalloc::AllocatableSet; use settings as shared_settings; use super::registers::{GPR, FPR}; use super::settings; +use std::i32; struct Args { pointer_bits: u16, @@ -79,7 +80,7 @@ impl ArgAssigner for Args { // Assign a stack location. let loc = ArgumentLoc::Stack(self.offset as i32); self.offset += self.pointer_bytes; - assert!(self.offset <= i32::max_value() as u32); + assert!(self.offset <= i32::MAX as u32); loc.into() } } diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 61a497000f..a6a633a524 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -5,6 +5,7 @@ use ir::{Function, Inst, InstructionData}; use isa::{RegUnit, StackRef, StackBaseMask}; use predicates::is_signed_int; use regalloc::RegDiversions; +use std::u32; include!(concat!(env!("OUT_DIR"), "/binemit-riscv.rs")); @@ -30,13 +31,13 @@ impl Into for RelocKind { /// /// Encoding bits: `opcode[6:2] | (funct3 << 5) | (funct7 << 8)`. fn put_r(bits: u16, rs1: RegUnit, rs2: RegUnit, rd: RegUnit, sink: &mut CS) { - let bits = bits as u32; + let bits = u32::from(bits); let opcode5 = bits & 0x1f; let funct3 = (bits >> 5) & 0x7; let funct7 = (bits >> 8) & 0x7f; - let rs1 = rs1 as u32 & 0x1f; - let rs2 = rs2 as u32 & 0x1f; - let rd = rd as u32 & 0x1f; + let rs1 = u32::from(rs1) & 0x1f; + let rs2 = u32::from(rs2) & 0x1f; + let rd = u32::from(rd) & 0x1f; // 0-6: opcode let mut i = 0x3; @@ -66,13 +67,13 @@ fn put_rshamt( rd: RegUnit, sink: &mut CS, ) { - let bits = bits as u32; + let bits = u32::from(bits); let opcode5 = bits & 0x1f; let funct3 = (bits >> 5) & 0x7; let funct7 = (bits >> 8) & 0x7f; - let rs1 = rs1 as u32 & 0x1f; + let rs1 = u32::from(rs1) & 0x1f; let shamt = shamt as u32 & 0x3f; - let rd = rd as u32 & 0x1f; + let rd = u32::from(rd) & 0x1f; // 0-6: opcode let mut i = 0x3; @@ -94,11 +95,11 @@ fn put_rshamt( /// /// Encoding bits: `opcode[6:2] | (funct3 << 5)` fn put_i(bits: u16, rs1: RegUnit, imm: i64, rd: RegUnit, sink: &mut CS) { - let bits = bits as u32; + let bits = u32::from(bits); let opcode5 = bits & 0x1f; let funct3 = (bits >> 5) & 0x7; - let rs1 = rs1 as u32 & 0x1f; - let rd = rd as u32 & 0x1f; + let rs1 = u32::from(rs1) & 0x1f; + let rd = u32::from(rd) & 0x1f; // 0-6: opcode let mut i = 0x3; @@ -121,7 +122,7 @@ fn put_i(bits: u16, rs1: RegUnit, imm: i64, rd: RegUnit, fn put_u(bits: u16, imm: i64, rd: RegUnit, sink: &mut CS) { let bits = bits as u32; let opcode5 = bits & 0x1f; - let rd = rd as u32 & 0x1f; + let rd = u32::from(rd) & 0x1f; // 0-6: opcode let mut i = 0x3; @@ -140,11 +141,11 @@ fn put_u(bits: u16, imm: i64, rd: RegUnit, sink: &mut CS) /// /// Encoding bits: `opcode[6:2] | (funct3 << 5)` fn put_sb(bits: u16, imm: i64, rs1: RegUnit, rs2: RegUnit, sink: &mut CS) { - let bits = bits as u32; + let bits = u32::from(bits); let opcode5 = bits & 0x1f; let funct3 = (bits >> 5) & 0x7; - let rs1 = rs1 as u32 & 0x1f; - let rs2 = rs2 as u32 & 0x1f; + let rs1 = u32::from(rs1) & 0x1f; + let rs2 = u32::from(rs2) & 0x1f; assert!(is_signed_int(imm, 13, 1), "SB out of range {:#x}", imm); let imm = imm as u32; @@ -173,9 +174,9 @@ fn put_sb(bits: u16, imm: i64, rs1: RegUnit, rs2: RegUnit /// /// Encoding bits: `opcode[6:2]` fn put_uj(bits: u16, imm: i64, rd: RegUnit, sink: &mut CS) { - let bits = bits as u32; + let bits = u32::from(bits); let opcode5 = bits & 0x1f; - let rd = rd as u32 & 0x1f; + let rd = u32::from(rd) & 0x1f; assert!(is_signed_int(imm, 21, 1), "UJ out of range {:#x}", imm); let imm = imm as u32; diff --git a/lib/cretonne/src/legalizer/heap.rs b/lib/cretonne/src/legalizer/heap.rs index 775b745725..09736925b4 100644 --- a/lib/cretonne/src/legalizer/heap.rs +++ b/lib/cretonne/src/legalizer/heap.rs @@ -43,7 +43,7 @@ fn dynamic_addr( bound_gv: ir::GlobalVar, func: &mut ir::Function, ) { - let size = size as i64; + let size = i64::from(size); let offset_ty = func.dfg.value_type(offset); let addr_ty = func.dfg.value_type(func.dfg.first_result(inst)); let min_size = func.heaps[heap].min_size.into(); @@ -96,7 +96,7 @@ fn static_addr( bound: i64, func: &mut ir::Function, ) { - let size = size as i64; + let size = i64::from(size); let offset_ty = func.dfg.value_type(offset); let addr_ty = func.dfg.value_type(func.dfg.first_result(inst)); let mut pos = FuncCursor::new(func).at_inst(inst); diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index cf8cc5bc99..4a6ff6c337 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -227,7 +227,7 @@ fn expand_fconst(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlow ir::InstructionData::UnaryIeee32 { opcode: ir::Opcode::F32const, imm, - } => pos.ins().iconst(ir::types::I32, imm.bits() as i64), + } => pos.ins().iconst(ir::types::I32, i64::from(imm.bits())), ir::InstructionData::UnaryIeee64 { opcode: ir::Opcode::F64const, imm, diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index 0e8e559cca..cc72066b5d 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -198,7 +198,9 @@ impl<'a> fmt::Display for DisplayAllocatableSet<'a> { bank.names .get(offset as usize) .and_then(|name| name.chars().skip(1).next()) - .unwrap_or(char::from_digit((offset % 10) as u32, 10).unwrap()) + .unwrap_or( + char::from_digit(u32::from(offset % 10), 10).unwrap(), + ) )?; } } diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index 7a3a3a00f6..0ee39b07d5 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -158,13 +158,13 @@ impl Pressure { fn check_avail_aliased(&self, entry: &TopRC) -> RegClassMask { let first = usize::from(entry.first_toprc); let num = usize::from(entry.num_toprcs); - let width = entry.width as u32; + let width = u32::from(entry.width); let ulimit = entry.limit * width; // Count up the number of available register units. let mut units = 0; for (rc, rci) in self.toprc[first..first + num].iter().zip(first..) { - let rcw = rc.width as u32; + let rcw = u32::from(rc.width); // If `rc.width` is smaller than `width`, each register in `rc` could potentially block // one of ours. This is assuming that none of the smaller registers are straddling the // bigger ones. diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index c13ec65292..760f16aa79 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -107,6 +107,7 @@ use std::cmp; use std::fmt; use std::mem; use super::AllocatableSet; +use std::u16; /// A variable in the constraint problem. /// @@ -866,7 +867,7 @@ impl Solver { // Compute domain sizes for all the variables given the current register sets. for v in &mut self.vars { let d = v.iter(&self.regs_in, &self.regs_out, global_regs).len(); - v.domain = cmp::min(d, u16::max_value() as usize) as u16; + v.domain = cmp::min(d, u16::MAX as usize) as u16; } // Solve for vars with small domains first to increase the chance of finding a solution. // Use the value number as a tie breaker to get a stable sort. diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index c8b0ca96ad..090b85baad 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -352,7 +352,7 @@ impl<'a> Lexer<'a> { _ => return None, }; if is_vector { - if number <= u16::MAX as u32 { + if number <= u32::from(u16::MAX) { base_type.by(number as u16).map(Token::Type) } else { None diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 133004aa13..1e65f2e607 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -32,7 +32,7 @@ use translation_utils::{TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; use state::{TranslationState, ControlStackFrame}; use std::collections::{HashMap, hash_map}; use environ::{FuncEnvironment, GlobalValue}; -use std::u32; +use std::{i32, u32}; /// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted /// a return. @@ -477,7 +477,7 @@ pub fn translate_operator( translate_store(offset, ir::Opcode::Istore32, builder, state, environ); } /****************************** Nullary Operators ************************************/ - Operator::I32Const { value } => state.push1(builder.ins().iconst(I32, value as i64)), + Operator::I32Const { value } => state.push1(builder.ins().iconst(I32, i64::from(value))), Operator::I64Const { value } => state.push1(builder.ins().iconst(I64, value)), Operator::F32Const { value } => { state.push1(builder.ins().f32const(f32_translation(value))); @@ -924,17 +924,17 @@ fn get_heap_addr( // even if the access goes beyond the guard pages. This is because the first byte pointed to is // inside the guard pages. let check_size = min( - u32::max_value() as i64, - 1 + (offset as i64 / guard_size) * guard_size, + i64::from(u32::MAX), + 1 + (i64::from(offset) / guard_size) * guard_size, ) as u32; let base = builder.ins().heap_addr(addr_ty, heap, addr32, check_size); // Native load/store instructions take a signed `Offset32` immediate, so adjust the base // pointer if necessary. - if offset > i32::max_value() as u32 { + if offset > i32::MAX as u32 { // Offset doesn't fit in the load/store instruction. - let adj = builder.ins().iadd_imm(base, i32::max_value() as i64 + 1); - (adj, (offset - (i32::max_value() as u32 + 1)) as i32) + let adj = builder.ins().iadd_imm(base, i64::from(i32::MAX) + 1); + (adj, (offset - (i32::MAX as u32 + 1)) as i32) } else { (base, offset as i32) } From e213c2654f1fa5755642c8f959e9ee3be0c86dcf Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 10:27:55 -0800 Subject: [PATCH 1354/3084] Fix branch_destination/analyze_branch for BranchInt/BranchFloat. --- cranelift/filetests/parser/flags.cton | 32 ++++++++++++------------ lib/cretonne/src/ir/instructions.rs | 36 ++++++++++++++++++++++++--- lib/cretonne/src/verifier/mod.rs | 11 +++++++- 3 files changed, 58 insertions(+), 21 deletions(-) diff --git a/cranelift/filetests/parser/flags.cton b/cranelift/filetests/parser/flags.cton index a315fbba56..c7b42f3ac2 100644 --- a/cranelift/filetests/parser/flags.cton +++ b/cranelift/filetests/parser/flags.cton @@ -2,45 +2,45 @@ test cat test verifier function %iflags(i32) { -ebb0(v0: i32): +ebb200(v0: i32): v1 = ifcmp_imm v0, 17 - brif eq v1, ebb1 - brif ugt v1, ebb2 + brif eq v1, ebb201 + brif ugt v1, ebb202 v2 = iconst.i32 34 v3 = ifcmp v0, v2 v4 = trueif eq v3 - brnz v4, ebb2 + brnz v4, ebb202 return -ebb1: +ebb201: return -ebb2: +ebb202: trap oob } ; check: $v1 = ifcmp_imm $v0, 17 -; check: brif eq $v1, $ebb1 -; check: brif ugt $v1, $ebb2 +; check: brif eq $v1, $ebb201 +; check: brif ugt $v1, $ebb202 ; check: $v3 = ifcmp $v0, $v2 ; check: $v4 = trueif eq $v3 function %fflags(f32) { -ebb0(v0: f32): +ebb200(v0: f32): v1 = f32const 0x34.0p0 v2 = ffcmp v0, v1 - brff eq v2, ebb1 - brff ord v2, ebb2 + brff eq v2, ebb201 + brff ord v2, ebb202 v3 = trueff gt v2 - brnz v3, ebb2 + brnz v3, ebb202 return -ebb1: +ebb201: return -ebb2: +ebb202: trap oob } ; check: $v2 = ffcmp $v0, $v1 -; check: brff eq $v2, $ebb1 -; check: brff ord $v2, $ebb2 +; check: brff eq $v2, $ebb201 +; check: brff ord $v2, $ebb202 ; check: $v3 = trueff gt $v2 diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 0038fa42c4..f110cd14db 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -335,6 +335,16 @@ impl InstructionData { ref args, .. } => BranchInfo::SingleDest(destination, args.as_slice(pool)), + InstructionData::BranchInt { + destination, + ref args, + .. + } | + InstructionData::BranchFloat { + destination, + ref args, + .. + } | InstructionData::Branch { destination, ref args, @@ -346,7 +356,10 @@ impl InstructionData { .. } => BranchInfo::SingleDest(destination, &args.as_slice(pool)[2..]), InstructionData::BranchTable { table, .. } => BranchInfo::Table(table), - _ => BranchInfo::NotABranch, + _ => { + debug_assert!(!self.opcode().is_branch()); + BranchInfo::NotABranch + } } } @@ -358,8 +371,14 @@ impl InstructionData { match *self { InstructionData::Jump { destination, .. } | InstructionData::Branch { destination, .. } | + InstructionData::BranchInt { destination, .. } | + InstructionData::BranchFloat { destination, .. } | InstructionData::BranchIcmp { destination, .. } => Some(destination), - _ => None, + InstructionData::BranchTable { .. } => None, + _ => { + debug_assert!(!self.opcode().is_branch()); + None + } } } @@ -371,8 +390,14 @@ impl InstructionData { match *self { InstructionData::Jump { ref mut destination, .. } | InstructionData::Branch { ref mut destination, .. } | + InstructionData::BranchInt { ref mut destination, .. } | + InstructionData::BranchFloat { ref mut destination, .. } | InstructionData::BranchIcmp { ref mut destination, .. } => Some(destination), - _ => None, + InstructionData::BranchTable { .. } => None, + _ => { + debug_assert!(!self.opcode().is_branch()); + None + } } } @@ -387,7 +412,10 @@ impl InstructionData { InstructionData::IndirectCall { sig_ref, ref args, .. } => { CallInfo::Indirect(sig_ref, &args.as_slice(pool)[1..]) } - _ => CallInfo::NotACall, + _ => { + debug_assert!(!self.opcode().is_call()); + CallInfo::NotACall + } } } } diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 7d2338c821..7332f2469a 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -1102,6 +1102,7 @@ mod tests { use super::{Verifier, Error}; use ir::Function; use ir::instructions::{InstructionData, Opcode}; + use entity::EntityList; use settings; macro_rules! assert_err_with_msg { @@ -1131,10 +1132,18 @@ mod tests { let ebb0 = func.dfg.make_ebb(); func.layout.append_ebb(ebb0); let nullary_with_bad_opcode = func.dfg.make_inst(InstructionData::UnaryImm { - opcode: Opcode::Jump, + opcode: Opcode::F32const, imm: 0.into(), }); func.layout.append_inst(nullary_with_bad_opcode, ebb0); + func.layout.append_inst( + func.dfg.make_inst(InstructionData::Jump { + opcode: Opcode::Jump, + destination: ebb0, + args: EntityList::default(), + }), + ebb0, + ); let flags = &settings::Flags::new(&settings::builder()); let verifier = Verifier::new(&func, flags.into()); assert_err_with_msg!(verifier.run(), "instruction format"); From ced7a88ecca40c2b9abc02b336ad4e146bb6ea0c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 8 Nov 2017 14:49:27 -0800 Subject: [PATCH 1355/3084] Use consistent formatting for module-level comments. --- cranelift/src/filetest/domtree.rs | 1 - lib/cretonne/src/ir/immediates.rs | 1 - lib/cretonne/src/ir/types.rs | 6 ------ lib/reader/src/lexer.rs | 7 +------ lib/reader/src/parser.rs | 7 +------ lib/wasm/src/environ/dummy.rs | 2 ++ lib/wasm/src/environ/mod.rs | 2 ++ lib/wasm/src/translation_utils.rs | 2 +- 8 files changed, 7 insertions(+), 21 deletions(-) diff --git a/cranelift/src/filetest/domtree.rs b/cranelift/src/filetest/domtree.rs index 622b01a891..1063429b17 100644 --- a/cranelift/src/filetest/domtree.rs +++ b/cranelift/src/filetest/domtree.rs @@ -1,4 +1,3 @@ - //! Test command for verifying dominator trees. //! //! The `test domtree` test command looks for annotations on instructions like this: diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 4af9452bcf..fb55c97006 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -1,4 +1,3 @@ - //! Immediate operands for Cretonne instructions //! //! This module defines the types of immediate operands that can appear on Cretonne instructions. diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index da502f9bf7..8b93e82267 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -3,12 +3,6 @@ use std::default::Default; use std::fmt::{self, Display, Debug, Formatter}; -// ====--------------------------------------------------------------------------------------====// -// -// Value types -// -// ====--------------------------------------------------------------------------------------====// - /// The type of an SSA value. /// /// The `VOID` type is only used for instructions that produce no value. It can't be part of a SIMD diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 090b85baad..1c422a8027 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -1,9 +1,4 @@ - -// ====--------------------------------------------------------------------------------------====// -// -// Lexical analysis for .cton files. -// -// ====--------------------------------------------------------------------------------------====// +//! Lexical analysis for .cton files. use std::str::CharIndices; use std::u16; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 0a63f19029..55cf342ad3 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1,9 +1,4 @@ - -// ====--------------------------------------------------------------------------------------====// -// -// Parser for .cton files. -// -// ====--------------------------------------------------------------------------------------====// +//! Parser for .cton files. use std::collections::HashMap; use std::str::FromStr; diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 64fb19e262..64733684be 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -1,3 +1,5 @@ +//! "Dummy" environment for testing wasm translation. + use environ::{FuncEnvironment, GlobalValue, ModuleEnvironment}; use translation_utils::{Global, Memory, Table, GlobalIndex, TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; diff --git a/lib/wasm/src/environ/mod.rs b/lib/wasm/src/environ/mod.rs index a31308b135..f45dbd3161 100644 --- a/lib/wasm/src/environ/mod.rs +++ b/lib/wasm/src/environ/mod.rs @@ -1,3 +1,5 @@ +//! Support for configurable wasm translation. + mod spec; mod dummy; diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 1f99287b32..8b1bc5fac2 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -1,4 +1,4 @@ -///! Helper functions and structures for the translation. +//! Helper functions and structures for the translation. use wasmparser; use cretonne; use std::u32; From 8dd46054a5afccde105014d3b57cdc5cbc022067 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 9 Nov 2017 22:02:07 -0800 Subject: [PATCH 1356/3084] In the verifier, clarify expected versus observed values. --- lib/cretonne/src/verifier/mod.rs | 56 +++++++++++++++++++------------- 1 file changed, 34 insertions(+), 22 deletions(-) diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 7332f2469a..f82043fce3 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -149,20 +149,20 @@ pub fn verify_context<'a, FOI: Into>>( struct Verifier<'a> { func: &'a Function, - cfg: ControlFlowGraph, - domtree: DominatorTree, + expected_cfg: ControlFlowGraph, + expected_domtree: DominatorTree, flags: &'a Flags, isa: Option<&'a TargetIsa>, } impl<'a> Verifier<'a> { pub fn new(func: &'a Function, fisa: FlagsOrIsa<'a>) -> Verifier<'a> { - let cfg = ControlFlowGraph::with_function(func); - let domtree = DominatorTree::with_function(func, &cfg); + let expected_cfg = ControlFlowGraph::with_function(func); + let expected_domtree = DominatorTree::with_function(func, &expected_cfg); Verifier { func, - cfg, - domtree, + expected_cfg, + expected_domtree, flags: fisa.flags, isa: fisa.isa, } @@ -461,8 +461,10 @@ impl<'a> Verifier<'a> { ); } // Defining instruction dominates the instruction that uses the value. - if self.domtree.is_reachable(self.func.layout.pp_ebb(loc_inst)) && - !self.domtree.dominates( + if self.expected_domtree.is_reachable( + self.func.layout.pp_ebb(loc_inst), + ) && + !self.expected_domtree.dominates( def_inst, loc_inst, &self.func.layout, @@ -486,8 +488,12 @@ impl<'a> Verifier<'a> { ); } // The defining EBB dominates the instruction using this value. - if self.domtree.is_reachable(ebb) && - !self.domtree.dominates(ebb, loc_inst, &self.func.layout) + if self.expected_domtree.is_reachable(ebb) && + !self.expected_domtree.dominates( + ebb, + loc_inst, + &self.func.layout, + ) { return err!(loc_inst, "uses value arg from non-dominating {}", ebb); } @@ -501,8 +507,8 @@ impl<'a> Verifier<'a> { // dominator for each EBB. Therefore the current domtree is valid if it matches the freshly // computed one. for ebb in self.func.layout.ebbs() { - let expected = domtree.idom(ebb); - let got = self.domtree.idom(ebb); + let expected = self.expected_domtree.idom(ebb); + let got = domtree.idom(ebb); if got != expected { return err!( ebb, @@ -514,20 +520,20 @@ impl<'a> Verifier<'a> { } } // We also verify if the postorder defined by `DominatorTree` is sane - if self.domtree.cfg_postorder().len() != domtree.cfg_postorder().len() { + if domtree.cfg_postorder().len() != self.expected_domtree.cfg_postorder().len() { return err!( AnyEntity::Function, "incorrect number of Ebbs in postorder traversal" ); } - for (index, (&true_ebb, &test_ebb)) in - self.domtree + for (index, (&test_ebb, &true_ebb)) in + domtree .cfg_postorder() .iter() - .zip(domtree.cfg_postorder().iter()) + .zip(self.expected_domtree.cfg_postorder().iter()) .enumerate() { - if true_ebb != test_ebb { + if test_ebb != true_ebb { return err!( test_ebb, "invalid domtree, postorder ebb number {} should be {}, got {}", @@ -538,8 +544,13 @@ impl<'a> Verifier<'a> { } } // We verify rpo_cmp on pairs of adjacent ebbs in the postorder - for (&prev_ebb, &next_ebb) in self.domtree.cfg_postorder().iter().adjacent_pairs() { - if domtree.rpo_cmp(prev_ebb, next_ebb, &self.func.layout) != Ordering::Greater { + for (&prev_ebb, &next_ebb) in domtree.cfg_postorder().iter().adjacent_pairs() { + if self.expected_domtree.rpo_cmp( + prev_ebb, + next_ebb, + &self.func.layout, + ) != Ordering::Greater + { return err!( next_ebb, "invalid domtree, rpo_cmp does not says {} is greater than {}", @@ -905,7 +916,7 @@ impl<'a> Verifier<'a> { let mut got_preds = BTreeSet::::new(); for ebb in self.func.layout.ebbs() { - expected_succs.extend(self.cfg.get_successors(ebb)); + expected_succs.extend(self.expected_cfg.get_successors(ebb)); got_succs.extend(cfg.get_successors(ebb)); let missing_succs: Vec = expected_succs.difference(&got_succs).cloned().collect(); @@ -922,7 +933,8 @@ impl<'a> Verifier<'a> { return err!(ebb, "cfg had unexpected successor(s) {:?}", excess_succs); } - expected_preds.extend(self.cfg.get_predecessors(ebb).iter().map(|&(_, inst)| inst)); + expected_preds.extend(self.expected_cfg.get_predecessors(ebb).iter().map(|&(_, + inst)| inst)); got_preds.extend(cfg.get_predecessors(ebb).iter().map(|&(_, inst)| inst)); let missing_preds: Vec = expected_preds.difference(&got_preds).cloned().collect(); @@ -1091,7 +1103,7 @@ impl<'a> Verifier<'a> { self.verify_return_at_end()?; } - verify_flags(self.func, &self.cfg, self.isa)?; + verify_flags(self.func, &self.expected_cfg, self.isa)?; Ok(()) } From 54e4ab71d918c778fa380a25250942c77a793be9 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sat, 2 Sep 2017 08:06:55 -0700 Subject: [PATCH 1357/3084] Enable pager in cton-util. --- cranelift/Cargo.toml | 3 +++ cranelift/src/cton-util.rs | 18 ++++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 45b39ef0a3..b328febdd0 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -26,6 +26,9 @@ num_cpus = "1.5.1" tempdir="0.3.5" term = "0.4.6" +[target.'cfg(unix)'.dependencies] +pager = "0.13.0" + [workspace] # Enable debug assertions and parallel compilation when building cretonne-tools diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index b08fa8cde3..15eb250f5e 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -10,11 +10,17 @@ extern crate num_cpus; extern crate tempdir; extern crate term; +#[cfg(unix)] +extern crate pager; + use cretonne::VERSION; use docopt::Docopt; use std::io::{self, Write}; use std::process; +#[cfg(unix)] +use pager::Pager; + mod utils; mod filetest; mod cat; @@ -107,7 +113,19 @@ fn cton_util() -> CommandResult { } } +#[cfg(unix)] +fn enable_pager() { + Pager::new().skip_on_notty().setup(); +} + +#[cfg(not(unix))] +fn enable_pager() { + // For now, pager only supports unix-type platforms. +} + fn main() { + enable_pager(); + if let Err(mut msg) = cton_util() { if !msg.ends_with('\n') { msg.push('\n'); From 78f2edefc2112061f728e94f5ed8991ee8b26cfc Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 10 Nov 2017 17:45:09 -0800 Subject: [PATCH 1358/3084] Add todos for add/sub with signed overflow, saturating fcvt_to_[su]int. --- cranelift/docs/langref.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index b5471d2a0c..d422163885 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -769,6 +769,12 @@ Integer operations .. autoinst:: isub_bout .. autoinst:: isub_borrow +.. todo:: Add and subtract with signed overflow. + + For example, see + `llvm.sadd.with.overflow.*` and `llvm.ssub.with.overflow.*` in + LLVM `_. + .. autoinst:: imul .. autoinst:: imul_imm @@ -916,6 +922,12 @@ Conversion operations .. autoinst:: fcvt_from_uint .. autoinst:: fcvt_from_sint +.. todo:: Saturating fcvt_to_sint and fcvt_to_uint. + + For example, these appear in + `Rust `_ and + `WebAssembly https://github.com/WebAssembly/nontrapping-float-to-int-conversions>`_. + Legalization operations ----------------------- From 648c1b33ba18bd8f88c2eee5b5cd04d4330d0d6f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 13 Nov 2017 14:05:47 -0800 Subject: [PATCH 1359/3084] Fix sphinx hyperlink syntax. --- cranelift/docs/langref.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index d422163885..b5abe0d956 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -926,7 +926,7 @@ Conversion operations For example, these appear in `Rust `_ and - `WebAssembly https://github.com/WebAssembly/nontrapping-float-to-int-conversions>`_. + `WebAssembly `_. Legalization operations ----------------------- From 355e3e3c15abca3fae611dfa91d7de0ec2a273bb Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 13 Nov 2017 14:34:35 -0800 Subject: [PATCH 1360/3084] Remove redundant validation for function locals. Remove the MAX_LOCALS constraint, and the local type validation. These are checked by whatever is performing proper validation. --- lib/wasm/src/func_translator.rs | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-) diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 57821c49c5..73aa7b433a 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -14,10 +14,6 @@ use state::TranslationState; use translation_utils::Local; use wasmparser::{self, BinaryReader}; -/// Maximum number of local variables permitted in a function. The translation fails with a -/// `CtonError::ImplLimitExceeded` error if the limit is exceeded. -const MAX_LOCALS: usize = 50_000; - /// WebAssembly to Cretonne IL function translator. /// /// A `FuncTranslator` is used to translate a binary WebAssembly function into Cretonne IL guided @@ -145,7 +141,7 @@ fn parse_local_decls( let (count, ty) = reader.read_local_decl(&mut locals_total).map_err(|_| { CtonError::InvalidInput })?; - declare_locals(builder, count, ty, &mut next_local)?; + declare_locals(builder, count, ty, &mut next_local); } Ok(()) @@ -159,7 +155,7 @@ fn declare_locals( count: u32, wasm_type: wasmparser::Type, next_local: &mut usize, -) -> CtonResult { +) { // All locals are initialized to 0. use wasmparser::Type::*; let zeroval = match wasm_type { @@ -167,24 +163,16 @@ fn declare_locals( I64 => builder.ins().iconst(ir::types::I64, 0), F32 => builder.ins().f32const(ir::immediates::Ieee32::with_bits(0)), F64 => builder.ins().f64const(ir::immediates::Ieee64::with_bits(0)), - _ => return Err(CtonError::InvalidInput), + _ => panic!("invalid local type"), }; let ty = builder.func.dfg.value_type(zeroval); for _ in 0..count { - // This implementation limit is arbitrary, but it ensures that a small function can't blow - // up the compiler by declaring millions of locals. - if *next_local >= MAX_LOCALS { - return Err(CtonError::ImplLimitExceeded); - } - let local = Local::new(*next_local); builder.declare_var(local, ty); builder.def_var(local, zeroval); *next_local += 1; } - - Ok(()) } /// Parse the function body in `reader`. From 4c829f7c7f5d03d40612178e19f3da86dc44de81 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 14 Nov 2017 14:09:35 -0800 Subject: [PATCH 1361/3084] Fix sphinx hyperlink syntax. --- cranelift/docs/langref.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index b5abe0d956..a6fe32ff85 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -773,7 +773,7 @@ Integer operations For example, see `llvm.sadd.with.overflow.*` and `llvm.ssub.with.overflow.*` in - LLVM `_. + `LLVM `_. .. autoinst:: imul .. autoinst:: imul_imm From 4d9aedbacad450f1e21eebb2209a927ad48f5cb6 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 15 Nov 2017 10:49:32 -0800 Subject: [PATCH 1362/3084] Add a 'clear()' function to Context. This includes adding `clear()` functions to its (transitive) members. --- lib/cretonne/src/context.rs | 9 +++++++++ lib/cretonne/src/flowgraph.rs | 6 ++++++ lib/cretonne/src/regalloc/coalescing.rs | 16 ++++++++++++++-- lib/cretonne/src/regalloc/coloring.rs | 6 ++++++ lib/cretonne/src/regalloc/context.rs | 12 ++++++++++++ lib/cretonne/src/regalloc/liveness.rs | 7 +++++++ lib/cretonne/src/regalloc/reload.rs | 6 ++++++ lib/cretonne/src/regalloc/solver.rs | 11 +++++++++++ lib/cretonne/src/regalloc/spilling.rs | 6 ++++++ lib/cretonne/src/topo_order.rs | 8 ++++++++ 10 files changed, 85 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 6f7f7a65c1..aa900ff9e1 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -57,6 +57,15 @@ impl Context { } } + /// Clear all data structures in this context. + pub fn clear(&mut self) { + self.func.clear(); + self.cfg.clear(); + self.domtree.clear(); + self.regalloc.clear(); + self.loop_analysis.clear(); + } + /// Compile the function. /// /// Run the function through all the passes necessary to generate code for the target ISA diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index bc6096ad9a..1f51ef0cf5 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -58,6 +58,12 @@ impl ControlFlowGraph { } } + /// Clear all data structures in this control flow graph. + pub fn clear(&mut self) { + self.data.clear(); + self.valid = false; + } + /// Allocate and compute the control flow graph for `func`. pub fn with_function(func: &Function) -> Self { let mut cfg = Self::new(); diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index cd54f2ece2..445a127ace 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -81,6 +81,12 @@ impl DomForest { } } + /// Clear all data structures in this dominator forest. + pub fn clear(&mut self) { + self.values.clear(); + self.stack.clear(); + } + /// Swap the merged list with `buffer`, leaving the dominator forest empty. /// /// This is typically called after a successful merge to extract the merged value list. @@ -141,8 +147,7 @@ impl DomForest { domtree: &DominatorTree, liveness: &Liveness, ) -> Result<(), (Value, Value)> { - self.stack.clear(); - self.values.clear(); + self.clear(); self.values.reserve(va.len() + vb.len()); // Convert the two value lists into a merged sequence of nodes. @@ -261,6 +266,13 @@ impl Coalescing { } + /// Clear all data structures in this coalescing pass. + pub fn clear(&mut self) { + self.forest.clear(); + self.values.clear(); + self.split_values.clear(); + } + /// Convert `func` to conventional SSA form and build virtual registers in the process. pub fn conventional_ssa( &mut self, diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index ff5bc0ef69..23cc3e27c2 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -108,6 +108,12 @@ impl Coloring { } } + /// Clear all data structures in this coloring pass. + pub fn clear(&mut self) { + self.divert.clear(); + self.solver.clear(); + } + /// Run the coloring algorithm over `func`. pub fn run( &mut self, diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index a35ddb3484..d101cce30b 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -49,6 +49,18 @@ impl Context { } } + /// Clear all data structures in this context. + pub fn clear(&mut self) { + self.liveness.clear(); + self.virtregs.clear(); + self.coalescing.clear(); + self.topo.clear(); + self.tracker.clear(); + self.spilling.clear(); + self.reload.clear(); + self.coloring.clear(); + } + /// Allocate registers in `func`. /// /// After register allocation, all values in `func` have been assigned to a register or stack diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index e5face5ef4..3180b57d4c 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -307,6 +307,13 @@ impl Liveness { } } + /// Clear all data structures in this liveness analysis. + pub fn clear(&mut self) { + self.ranges.clear(); + self.worklist.clear(); + self.ebb_params.clear(); + } + /// Get the live range for `value`, if it exists. pub fn get(&self, value: Value) -> Option<&LiveRange> { self.ranges.get(value) diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 9b0006b990..aa43d6771a 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -53,6 +53,12 @@ impl Reload { } } + /// Clear all data structures in this reload pass. + pub fn clear(&mut self) { + self.candidates.clear(); + self.reloads.clear(); + } + /// Run the reload algorithm over `func`. pub fn run( &mut self, diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 760f16aa79..c7b884ce43 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -515,6 +515,17 @@ impl Solver { } } + /// Clear all data structures in this coloring pass. + pub fn clear(&mut self) { + self.assignments.clear(); + self.vars.clear(); + self.inputs_done = false; + self.regs_in = AllocatableSet::new(); + self.regs_out = AllocatableSet::new(); + self.moves.clear(); + self.fills.clear(); + } + /// Reset the solver state and prepare solving for a new instruction with an initial set of /// allocatable registers. /// diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 65d2f2d076..253b96c25a 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -70,6 +70,12 @@ impl Spilling { } } + /// Clear all data structures in this spilling pass. + pub fn clear(&mut self) { + self.spills.clear(); + self.reg_uses.clear(); + } + /// Run the spilling algorithm over `func`. pub fn run( &mut self, diff --git a/lib/cretonne/src/topo_order.rs b/lib/cretonne/src/topo_order.rs index d71dc63ccf..cd783c928a 100644 --- a/lib/cretonne/src/topo_order.rs +++ b/lib/cretonne/src/topo_order.rs @@ -35,6 +35,14 @@ impl TopoOrder { } } + /// Clear all data structures in this topological order. + pub fn clear(&mut self) { + self.preferred.clear(); + self.next = 0; + self.visited.clear(); + self.stack.clear(); + } + /// Reset and initialize with a preferred sequence of EBBs. The resulting topological order is /// guaranteed to contain all of the EBBs in `preferred` as well as any dominators. pub fn reset(&mut self, preferred: Ebbs) From bfa5a72b77c71e00e346607bd411001d172d5d98 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 16 Nov 2017 15:16:23 -0800 Subject: [PATCH 1363/3084] Use reserve_exact instead of reserve when the max size is known. --- lib/cretonne/src/regalloc/coalescing.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 445a127ace..5f41ddc72a 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -148,7 +148,7 @@ impl DomForest { liveness: &Liveness, ) -> Result<(), (Value, Value)> { self.clear(); - self.values.reserve(va.len() + vb.len()); + self.values.reserve_exact(va.len() + vb.len()); // Convert the two value lists into a merged sequence of nodes. let merged = MergedNodes { From 94921c0b7448c982448cfc8308f9a12732c3c8af Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 24 Oct 2017 09:09:02 -0700 Subject: [PATCH 1364/3084] Remove the dead B-Tree interface. We're starting over from scratch. --- lib/cretonne/src/btree.rs | 112 -------------------------------------- 1 file changed, 112 deletions(-) delete mode 100644 lib/cretonne/src/btree.rs diff --git a/lib/cretonne/src/btree.rs b/lib/cretonne/src/btree.rs deleted file mode 100644 index e625b6108e..0000000000 --- a/lib/cretonne/src/btree.rs +++ /dev/null @@ -1,112 +0,0 @@ -//! Generic B-Tree implementation. -//! -//! This module defines a `Btree` type which provides similar functionality to -//! `BtreeMap`, but with some important differences in the implementation: -//! -//! 1. Memory is allocated from a `NodePool` instead of the global heap. -//! 2. The footprint of a BTree is only 4 bytes. -//! 3. A BTree doesn't implement `Drop`, leaving it to the pool to manage memory. -//! -//! The node pool is intended to be used as a LIFO allocator. After building up a larger data -//! structure with many list references, the whole thing can be discarded quickly by clearing the -//! pool. - -use std::marker::PhantomData; - -// A Node reference is a direct index to an element of the pool. -type NodeRef = u32; - -/// A B-tree data structure which nodes are allocated from a pool. -pub struct BTree { - index: NodeRef, - unused1: PhantomData, - unused2: PhantomData, -} - -/// An enum representing a B-tree node. -/// Keys and values are required to implement Default. -enum Node { - Inner { - size: u8, - keys: [K; 7], - nodes: [NodeRef; 8], - }, - Leaf { - size: u8, - keys: [K; 7], - values: [V; 7], - }, -} - -/// Memory pool for nodes. -struct NodePool { - // The array containing the nodes. - data: Vec>, - - // A free list - freelist: Vec, -} - -impl NodePool { - /// Create a new NodePool. - pub fn new() -> NodePool { - NodePool { - data: Vec::new(), - freelist: Vec::new(), - } - } - - /// Get a B-tree node. - pub fn get(&self, index: u32) -> Option<&Node> { - unimplemented!() - } -} - -impl BTree { - /// Search for `key` and return a `Cursor` that either points at `key` or the position - /// where it would be inserted. - pub fn search(&mut self, key: K) -> Cursor { - unimplemented!() - } -} - -pub struct Cursor<'a, K: 'a, V: 'a> { - pool: &'a mut NodePool, - height: usize, - path: [(NodeRef, u8); 16], -} - -impl<'a, K: Default, V: Default> Cursor<'a, K, V> { - /// The key at the cursor position. Returns `None` when the cursor points off the end. - pub fn key(&self) -> Option { - unimplemented!() - } - - /// The value at the cursor position. Returns `None` when the cursor points off the end. - pub fn value(&self) -> Option<&V> { - unimplemented!() - } - - /// Move to the next element. - /// Returns `false` if that moves the cursor off the end. - pub fn next(&mut self) -> bool { - unimplemented!() - } - - /// Move to the previous element. - /// Returns `false` if this moves the cursor before the beginning. - pub fn prev(&mut self) -> bool { - unimplemented!() - } - - /// Insert a `(key, value)` pair at the cursor position. - /// It is an error to insert a key that would be out of order at this position. - pub fn insert(&mut self, key: K, value: V) { - unimplemented!() - } - - /// Remove the current element. - pub fn remove(&mut self) { - unimplemented!() - } -} From aa6f5c0db3b5667114781a646b17f14b76d7f715 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 24 Oct 2017 09:44:23 -0700 Subject: [PATCH 1365/3084] Forests of B+ trees. Add new ordered set and map data structures based on B+-trees. These are not general-purpose data structures like the BTreeSet and BTreeMap types in the standard library. They are specialized for: - Keys and values are small `Copy` types, optimized for 32-bit entities. - Each set or map has a very small footprint, using only 32 bits of memory when empty. - Keys are compared using a borrowed comparator object which can provide context for comparing tiny types that don't contain enough information to implement `Ord`. - A whole forest of B-trees can be cleared in constant time without having to traverse the whole data structure. --- lib/cretonne/src/bforest/map.rs | 753 ++++++++++++++++++++++++++++ lib/cretonne/src/bforest/mod.rs | 172 +++++++ lib/cretonne/src/bforest/node.rs | 813 ++++++++++++++++++++++++++++++ lib/cretonne/src/bforest/path.rs | 817 +++++++++++++++++++++++++++++++ lib/cretonne/src/bforest/pool.rs | 197 ++++++++ lib/cretonne/src/bforest/set.rs | 486 ++++++++++++++++++ lib/cretonne/src/lib.rs | 1 + 7 files changed, 3239 insertions(+) create mode 100644 lib/cretonne/src/bforest/map.rs create mode 100644 lib/cretonne/src/bforest/mod.rs create mode 100644 lib/cretonne/src/bforest/node.rs create mode 100644 lib/cretonne/src/bforest/path.rs create mode 100644 lib/cretonne/src/bforest/pool.rs create mode 100644 lib/cretonne/src/bforest/set.rs diff --git a/lib/cretonne/src/bforest/map.rs b/lib/cretonne/src/bforest/map.rs new file mode 100644 index 0000000000..43855c4760 --- /dev/null +++ b/lib/cretonne/src/bforest/map.rs @@ -0,0 +1,753 @@ +//! Forest of maps. + +use packed_option::PackedOption; +use std::marker::PhantomData; +use super::{INNER_SIZE, BPlusComparator, Forest, NodePool, Node, NodeData, Path}; + +/// Tag type defining forest types for a map. +struct MapTypes(PhantomData<(K, V, C)>); + +impl Forest for MapTypes +where + K: Copy, + V: Copy, + C: BPlusComparator, +{ + type Key = K; + type Value = V; + type LeafKeys = [K; INNER_SIZE - 1]; + type LeafValues = [V; INNER_SIZE - 1]; + type Comparator = C; + + fn splat_key(key: Self::Key) -> Self::LeafKeys { + [key; INNER_SIZE - 1] + } + + fn splat_value(value: Self::Value) -> Self::LeafValues { + [value; INNER_SIZE - 1] + } +} + +/// Memory pool for a forest of `BPlusMap` instances. +pub struct MapForest +where + K: Copy, + V: Copy, + C: BPlusComparator, +{ + nodes: NodePool>, +} + +impl MapForest +where + K: Copy, + V: Copy, + C: BPlusComparator, +{ + /// Create a new empty forest. + pub fn new() -> MapForest { + MapForest { nodes: NodePool::new() } + } + + /// Clear all maps in the forest. + /// + /// All `BPlusMap` instances belong to this forest are invalidated and should no longer be used. + pub fn clear(&mut self) { + self.nodes.clear(); + } +} + +/// B-tree mapping from `K` to `V` using `C` for comparing keys. +/// +/// This is not a general-purpose replacement for `BTreeMap`. See the [module +/// documentation](index.html) for more information about design tradeoffs. +pub struct BPlusMap +where + K: Copy, + V: Copy, + C: BPlusComparator, +{ + root: PackedOption, + unused: PhantomData<(K, V, C)>, +} + +impl BPlusMap +where + K: Copy, + V: Copy, + C: BPlusComparator, +{ + /// Make an empty map. + pub fn new() -> BPlusMap { + BPlusMap { + root: None.into(), + unused: PhantomData, + } + } + + /// Is this an empty map? + pub fn is_empty(&self) -> bool { + self.root.is_none() + } + + /// Get the value stored for `key`. + pub fn get(&self, key: K, forest: &MapForest, comp: &C) -> Option { + self.root.expand().and_then(|root| { + Path::default().find(key, root, &forest.nodes, comp) + }) + } + + /// Insert `key, value` into the map and return the old value stored for `key`, if any. + pub fn insert( + &mut self, + key: K, + value: V, + forest: &mut MapForest, + comp: &C, + ) -> Option { + self.cursor(forest, comp).insert(key, value) + } + + /// Remove `key` from the map and return the removed value for `key`, if any. + pub fn remove(&mut self, key: K, forest: &mut MapForest, comp: &C) -> Option { + let mut c = self.cursor(forest, comp); + if c.goto(key).is_some() { + c.remove() + } else { + None + } + } + + /// Create a cursor for navigating this map. The cursor is initially positioned off the end of + /// the map. + pub fn cursor<'a>( + &'a mut self, + forest: &'a mut MapForest, + comp: &'a C, + ) -> MapCursor<'a, K, V, C> { + MapCursor::new(self, forest, comp) + } +} + +#[cfg(test)] +impl BPlusMap +where + K: Copy + ::std::fmt::Display, + V: Copy, + C: BPlusComparator, +{ + /// Verify consistency. + fn verify(&self, forest: &MapForest, comp: &C) + where + NodeData>: ::std::fmt::Display, + { + if let Some(root) = self.root.expand() { + forest.nodes.verify_tree(root, comp); + } + } + + /// Get a text version of the path to `key`. + fn tpath(&self, key: K, forest: &MapForest, comp: &C) -> String { + match self.root.expand() { + None => "map(empty)".to_string(), + Some(root) => { + let mut path = Path::default(); + path.find(key, root, &forest.nodes, comp); + path.to_string() + } + } + } +} + +/// A position in a `BPlusMap` used to navigate and modify the ordered map. +/// +/// A cursor always points at a key-value pair in the map, or "off the end" which is a position +/// after the last entry in the map. +pub struct MapCursor<'a, K, V, C> +where + K: 'a + Copy, + V: 'a + Copy, + C: 'a + BPlusComparator, +{ + root: &'a mut PackedOption, + pool: &'a mut NodePool>, + comp: &'a C, + path: Path>, +} + +impl<'a, K, V, C> MapCursor<'a, K, V, C> +where + K: Copy, + V: Copy, + C: BPlusComparator, +{ + /// Create a cursor with a default (off-the-end) location. + fn new( + container: &'a mut BPlusMap, + forest: &'a mut MapForest, + comp: &'a C, + ) -> MapCursor<'a, K, V, C> { + MapCursor { + root: &mut container.root, + pool: &mut forest.nodes, + comp, + path: Path::default(), + } + } + + /// Is this cursor pointing to an empty map? + pub fn is_empty(&self) -> bool { + self.root.is_none() + } + + /// Move cursor to the next key-value pair and return it. + /// + /// If the cursor reaches the end, return `None` and leave the cursor at the off-the-end + /// position. + pub fn next(&mut self) -> Option<(K, V)> { + self.path.next(self.pool) + } + + /// Move cursor to the previous key-value pair and return it. + /// + /// If the cursor is already pointing at the first entry, leave it there and return `None`. + pub fn prev(&mut self) -> Option<(K, V)> { + self.root.expand().and_then( + |root| self.path.prev(root, self.pool), + ) + } + + /// Get the current key, or `None` if the cursor is at the end. + pub fn key(&self) -> Option { + self.path.leaf_pos().and_then(|(node, entry)| { + self.pool[node].unwrap_leaf().0.get(entry).cloned() + }) + } + + /// Get the current value, or `None` if the cursor is at the end. + pub fn value(&self) -> Option { + self.path.leaf_pos().and_then(|(node, entry)| { + self.pool[node].unwrap_leaf().1.get(entry).cloned() + }) + } + + /// Move this cursor to `key`. + /// + /// If `key` is in the map, place the cursor at `key` and return the corresponding value. + /// + /// If `key` is not in the set, place the cursor at the next larger element (or the end) and + /// return `None`. + pub fn goto(&mut self, elem: K) -> Option { + self.root.expand().and_then(|root| { + let v = self.path.find(elem, root, self.pool, self.comp); + if v.is_none() { + self.path.normalize(self.pool); + } + v + }) + } + + /// Insert `(key, value))` into the map and leave the cursor at the inserted pair. + /// + /// If the map did not contain `key`, return `None`. + /// + /// If `key` is already present, replace the existing with `value` and return the old value. + pub fn insert(&mut self, key: K, value: V) -> Option { + match self.root.expand() { + None => { + let root = self.pool.alloc_node(NodeData::leaf(key, value)); + *self.root = root.into(); + self.path.set_root_node(root); + None + } + Some(root) => { + // TODO: Optimize the case where `self.path` is already at the correct insert pos. + let old = self.path.find(key, root, self.pool, self.comp); + if old.is_some() { + *self.path.value_mut(self.pool) = value; + } else { + *self.root = self.path.insert(key, value, self.pool).into(); + } + old + } + } + } + + /// Remove the current entry (if any) and return the mapped value. + /// This advances the cursor to the next entry after the removed one. + pub fn remove(&mut self) -> Option { + let value = self.value(); + if value.is_some() { + *self.root = self.path.remove(self.pool).into(); + } + value + } +} + +#[cfg(test)] +impl<'a, K, V, C> MapCursor<'a, K, V, C> +where + K: Copy + ::std::fmt::Display, + V: Copy + ::std::fmt::Display, + C: BPlusComparator, +{ + fn verify(&self) { + self.path.verify(self.pool); + self.root.map(|root| self.pool.verify_tree(root, self.comp)); + } + + /// Get a text version of the path to the current position. + fn tpath(&self) -> String { + self.path.to_string() + } +} + +#[cfg(test)] +mod test { + use std::mem; + use super::*; + use super::super::NodeData; + + #[test] + fn node_size() { + // check that nodes are cache line sized when keys and values are 32 bits. + type F = MapTypes; + assert_eq!(mem::size_of::>(), 64); + } + + #[test] + fn empty() { + let mut f = MapForest::::new(); + f.clear(); + + let mut m = BPlusMap::::new(); + assert!(m.is_empty()); + + assert_eq!(m.get(7, &f, &()), None); + + let mut c = m.cursor(&mut f, &()); + assert!(c.is_empty()); + assert_eq!(c.key(), None); + assert_eq!(c.value(), None); + assert_eq!(c.next(), None); + assert_eq!(c.prev(), None); + c.verify(); + assert_eq!(c.tpath(), ""); + } + + #[test] + fn inserting() { + let f = &mut MapForest::::new(); + let mut m = BPlusMap::::new(); + + // The first seven values stay in a single leaf node. + assert_eq!(m.insert(50, 5.0, f, &()), None); + assert_eq!(m.insert(50, 5.5, f, &()), Some(5.0)); + assert_eq!(m.insert(20, 2.0, f, &()), None); + assert_eq!(m.insert(80, 8.0, f, &()), None); + assert_eq!(m.insert(40, 4.0, f, &()), None); + assert_eq!(m.insert(60, 6.0, f, &()), None); + assert_eq!(m.insert(90, 9.0, f, &()), None); + assert_eq!(m.insert(200, 20.0, f, &()), None); + + m.verify(f, &()); + + // [ 20 40 50 60 80 90 200 ] + + assert_eq!(m.get(0, f, &()), None); + assert_eq!(m.get(20, f, &()), Some(2.0)); + assert_eq!(m.get(30, f, &()), None); + assert_eq!(m.get(40, f, &()), Some(4.0)); + assert_eq!(m.get(50, f, &()), Some(5.5)); + assert_eq!(m.get(60, f, &()), Some(6.0)); + assert_eq!(m.get(70, f, &()), None); + assert_eq!(m.get(80, f, &()), Some(8.0)); + assert_eq!(m.get(100, f, &()), None); + + { + let mut c = m.cursor(f, &()); + assert_eq!(c.prev(), Some((200, 20.0))); + assert_eq!(c.prev(), Some((90, 9.0))); + assert_eq!(c.prev(), Some((80, 8.0))); + assert_eq!(c.prev(), Some((60, 6.0))); + assert_eq!(c.prev(), Some((50, 5.5))); + assert_eq!(c.prev(), Some((40, 4.0))); + assert_eq!(c.prev(), Some((20, 2.0))); + assert_eq!(c.prev(), None); + } + + // Test some removals where the node stays healthy. + assert_eq!(m.tpath(50, f, &()), "node0[2]"); + assert_eq!(m.tpath(80, f, &()), "node0[4]"); + assert_eq!(m.tpath(200, f, &()), "node0[6]"); + + assert_eq!(m.remove(80, f, &()), Some(8.0)); + assert_eq!(m.tpath(50, f, &()), "node0[2]"); + assert_eq!(m.tpath(80, f, &()), "node0[4]"); + assert_eq!(m.tpath(200, f, &()), "node0[5]"); + assert_eq!(m.remove(80, f, &()), None); + m.verify(f, &()); + + assert_eq!(m.remove(20, f, &()), Some(2.0)); + assert_eq!(m.tpath(50, f, &()), "node0[1]"); + assert_eq!(m.tpath(80, f, &()), "node0[3]"); + assert_eq!(m.tpath(200, f, &()), "node0[4]"); + assert_eq!(m.remove(20, f, &()), None); + m.verify(f, &()); + + // [ 40 50 60 90 200 ] + + { + let mut c = m.cursor(f, &()); + assert_eq!(c.goto(0), None); + assert_eq!(c.key(), Some(40)); + assert_eq!(c.value(), Some(4.0)); + assert_eq!(c.next(), Some((50, 5.5))); + assert_eq!(c.next(), Some((60, 6.0))); + assert_eq!(c.next(), Some((90, 9.0))); + assert_eq!(c.next(), Some((200, 20.0))); + c.verify(); + assert_eq!(c.next(), None); + c.verify(); + } + + // Removals from the root leaf node beyond underflow. + assert_eq!(m.remove(200, f, &()), Some(20.0)); + assert_eq!(m.remove(40, f, &()), Some(4.0)); + assert_eq!(m.remove(60, f, &()), Some(6.0)); + m.verify(f, &()); + assert_eq!(m.remove(50, f, &()), Some(5.5)); + m.verify(f, &()); + assert_eq!(m.remove(90, f, &()), Some(9.0)); + m.verify(f, &()); + assert!(m.is_empty()); + } + + #[test] + fn split_level0_leaf() { + // Various ways of splitting a full leaf node at level 0. + let f = &mut MapForest::::new(); + + fn full_leaf(f: &mut MapForest) -> BPlusMap { + let mut m = BPlusMap::new(); + for n in 1..8 { + m.insert(n * 10, n as f32 * 1.1, f, &()); + } + m + } + + // Insert at front of leaf. + let mut m = full_leaf(f); + m.insert(5, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(5, f, &()), Some(4.2)); + + // Insert at back of leaf. + let mut m = full_leaf(f); + m.insert(80, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(80, f, &()), Some(4.2)); + + // Insert before middle (40). + let mut m = full_leaf(f); + m.insert(35, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(35, f, &()), Some(4.2)); + + // Insert after middle (40). + let mut m = full_leaf(f); + m.insert(45, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(45, f, &()), Some(4.2)); + } + + #[test] + fn split_level1_leaf() { + // Various ways of splitting a full leaf node at level 1. + let f = &mut MapForest::::new(); + + // Return a map whose root node is a full inner node, and the leaf nodes are all full + // containing: + // + // 110, 120, ..., 170 + // 210, 220, ..., 270 + // ... + // 810, 820, ..., 870 + fn full(f: &mut MapForest) -> BPlusMap { + let mut m = BPlusMap::new(); + + // Start by inserting elements in order. + // This should leave 8 leaf nodes with 4 elements in each. + for row in 1..9 { + for col in 1..5 { + m.insert(row * 100 + col * 10, row as f32 + col as f32 * 0.1, f, &()); + } + } + + // Then top up the leaf nodes without splitting them. + for row in 1..9 { + for col in 5..8 { + m.insert(row * 100 + col * 10, row as f32 + col as f32 * 0.1, f, &()); + } + } + + m + } + + let mut m = full(f); + // Verify geometry. Get get node2 as the root and leaves node0, 1, 3, ... + m.verify(f, &()); + assert_eq!(m.tpath(110, f, &()), "node2[0]--node0[0]"); + assert_eq!(m.tpath(140, f, &()), "node2[0]--node0[3]"); + assert_eq!(m.tpath(210, f, &()), "node2[1]--node1[0]"); + assert_eq!(m.tpath(270, f, &()), "node2[1]--node1[6]"); + assert_eq!(m.tpath(310, f, &()), "node2[2]--node3[0]"); + assert_eq!(m.tpath(810, f, &()), "node2[7]--node8[0]"); + assert_eq!(m.tpath(870, f, &()), "node2[7]--node8[6]"); + + // Front of first leaf. + m.insert(0, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(0, f, &()), Some(4.2)); + + // First leaf split 4-4 after appending to LHS. + f.clear(); + m = full(f); + m.insert(135, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(135, f, &()), Some(4.2)); + + // First leaf split 4-4 after prepending to RHS. + f.clear(); + m = full(f); + m.insert(145, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(145, f, &()), Some(4.2)); + + // First leaf split 4-4 after appending to RHS. + f.clear(); + m = full(f); + m.insert(175, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(175, f, &()), Some(4.2)); + + // Left-middle leaf split, ins LHS. + f.clear(); + m = full(f); + m.insert(435, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(435, f, &()), Some(4.2)); + + // Left-middle leaf split, ins RHS. + f.clear(); + m = full(f); + m.insert(445, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(445, f, &()), Some(4.2)); + + // Right-middle leaf split, ins LHS. + f.clear(); + m = full(f); + m.insert(535, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(535, f, &()), Some(4.2)); + + // Right-middle leaf split, ins RHS. + f.clear(); + m = full(f); + m.insert(545, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(545, f, &()), Some(4.2)); + + // Last leaf split, ins LHS. + f.clear(); + m = full(f); + m.insert(835, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(835, f, &()), Some(4.2)); + + // Last leaf split, ins RHS. + f.clear(); + m = full(f); + m.insert(845, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(845, f, &()), Some(4.2)); + + // Front of last leaf. + f.clear(); + m = full(f); + m.insert(805, 4.2, f, &()); + m.verify(f, &()); + assert_eq!(m.get(805, f, &()), Some(4.2)); + } + + // Make a tree with two barely healthy leaf nodes: + // [ 10 20 30 40 ] [ 50 60 70 80 ] + fn two_leaf(f: &mut MapForest) -> BPlusMap { + f.clear(); + let mut m = BPlusMap::new(); + for n in 1..9 { + m.insert(n * 10, n as f32, f, &()); + } + m + } + + #[test] + fn remove_level1() { + let f = &mut MapForest::::new(); + let mut m = two_leaf(f); + + // Verify geometry. + m.verify(f, &()); + assert_eq!(m.tpath(10, f, &()), "node2[0]--node0[0]"); + assert_eq!(m.tpath(40, f, &()), "node2[0]--node0[3]"); + assert_eq!(m.tpath(49, f, &()), "node2[0]--node0[4]"); + assert_eq!(m.tpath(50, f, &()), "node2[1]--node1[0]"); + assert_eq!(m.tpath(80, f, &()), "node2[1]--node1[3]"); + + // Remove the front entry from a node that stays healthy. + assert_eq!(m.insert(55, 5.5, f, &()), None); + assert_eq!(m.remove(50, f, &()), Some(5.0)); + m.verify(f, &()); + assert_eq!(m.tpath(49, f, &()), "node2[0]--node0[4]"); + assert_eq!(m.tpath(50, f, &()), "node2[0]--node0[4]"); + assert_eq!(m.tpath(55, f, &()), "node2[1]--node1[0]"); + + // Remove the front entry from the first leaf node: No critical key to update. + assert_eq!(m.insert(15, 1.5, f, &()), None); + assert_eq!(m.remove(10, f, &()), Some(1.0)); + m.verify(f, &()); + + // [ 15 20 30 40 ] [ 55 60 70 80 ] + + // Remove the front entry from a right-most node that underflows. + // No rebalancing for the right-most node. Still need critical key update. + assert_eq!(m.remove(55, f, &()), Some(5.5)); + m.verify(f, &()); + assert_eq!(m.tpath(55, f, &()), "node2[0]--node0[4]"); + assert_eq!(m.tpath(60, f, &()), "node2[1]--node1[0]"); + + // [ 15 20 30 40 ] [ 60 70 80 ] + + // Replenish the right leaf. + assert_eq!(m.insert(90, 9.0, f, &()), None); + assert_eq!(m.insert(100, 10.0, f, &()), None); + m.verify(f, &()); + assert_eq!(m.tpath(55, f, &()), "node2[0]--node0[4]"); + assert_eq!(m.tpath(60, f, &()), "node2[1]--node1[0]"); + + // [ 15 20 30 40 ] [ 60 70 80 90 100 ] + + // Removing one entry from the left leaf should trigger a rebalancing from the right + // sibling. + assert_eq!(m.remove(20, f, &()), Some(2.0)); + m.verify(f, &()); + + // [ 15 30 40 60 ] [ 70 80 90 100 ] + // Check that the critical key was updated correctly. + assert_eq!(m.tpath(50, f, &()), "node2[0]--node0[3]"); + assert_eq!(m.tpath(60, f, &()), "node2[0]--node0[3]"); + assert_eq!(m.tpath(70, f, &()), "node2[1]--node1[0]"); + + // Remove front entry from the left-most leaf node, underflowing. + // This should cause two leaf nodes to be merged and the root node to go away. + assert_eq!(m.remove(15, f, &()), Some(1.5)); + m.verify(f, &()); + } + + #[test] + fn remove_level1_rightmost() { + let f = &mut MapForest::::new(); + let mut m = two_leaf(f); + + // [ 10 20 30 40 ] [ 50 60 70 80 ] + + // Remove entries from the right leaf. This doesn't trigger a rebalancing. + assert_eq!(m.remove(60, f, &()), Some(6.0)); + assert_eq!(m.remove(80, f, &()), Some(8.0)); + assert_eq!(m.remove(50, f, &()), Some(5.0)); + m.verify(f, &()); + + // [ 10 20 30 40 ] [ 70 ] + assert_eq!(m.tpath(50, f, &()), "node2[0]--node0[4]"); + assert_eq!(m.tpath(70, f, &()), "node2[1]--node1[0]"); + + // Removing the last entry from the right leaf should cause a collapse. + assert_eq!(m.remove(70, f, &()), Some(7.0)); + m.verify(f, &()); + } + + // Make a 3-level tree with barely healthy nodes. + // 1 root, 8 inner nodes, 7*4+5=33 leaf nodes, 4 entries each. + fn level3_sparse(f: &mut MapForest) -> BPlusMap { + f.clear(); + let mut m = BPlusMap::new(); + for n in 1..133 { + m.insert(n * 10, n as f32, f, &()); + } + m + } + + #[test] + fn level3_removes() { + let f = &mut MapForest::::new(); + let mut m = level3_sparse(f); + m.verify(f, &()); + + // Check geometry. + // Root: node11 + // [ node2 170 node10 330 node16 490 node21 650 node26 810 node31 970 node36 1130 node41 ] + // L1: node11 + assert_eq!(m.tpath(0, f, &()), "node11[0]--node2[0]--node0[0]"); + assert_eq!(m.tpath(10000, f, &()), "node11[7]--node41[4]--node40[4]"); + + // 650 is a critical key in the middle of the root. + assert_eq!(m.tpath(640, f, &()), "node11[3]--node21[3]--node19[3]"); + assert_eq!(m.tpath(650, f, &()), "node11[4]--node26[0]--node20[0]"); + + // Deleting 640 triggers a rebalance from node19 to node 20, cascading to n21 -> n26. + assert_eq!(m.remove(640, f, &()), Some(64.0)); + m.verify(f, &()); + assert_eq!(m.tpath(650, f, &()), "node11[3]--node26[3]--node20[3]"); + + // 1130 is in the first leaf of the last L1 node. Deleting it triggers a rebalance node35 + // -> node37, but no rebalance above where there is no right sibling. + assert_eq!(m.tpath(1130, f, &()), "node11[6]--node41[0]--node35[0]"); + assert_eq!(m.tpath(1140, f, &()), "node11[6]--node41[0]--node35[1]"); + assert_eq!(m.remove(1130, f, &()), Some(113.0)); + m.verify(f, &()); + assert_eq!(m.tpath(1140, f, &()), "node11[6]--node41[0]--node37[0]"); + } + + #[test] + fn insert_many() { + let f = &mut MapForest::::new(); + let mut m = BPlusMap::::new(); + + let mm = 4096; + let mut x = 0; + + for n in 0..mm { + assert_eq!(m.insert(x, n as f32, f, &()), None); + m.verify(f, &()); + + x = (x + n + 1) % mm; + } + + x = 0; + for n in 0..mm { + assert_eq!(m.get(x, f, &()), Some(n as f32)); + x = (x + n + 1) % mm; + } + + x = 0; + for n in 0..mm { + assert_eq!(m.remove(x, f, &()), Some(n as f32)); + m.verify(f, &()); + + x = (x + n + 1) % mm; + } + + assert!(m.is_empty()); + } +} diff --git a/lib/cretonne/src/bforest/mod.rs b/lib/cretonne/src/bforest/mod.rs new file mode 100644 index 0000000000..de8ee1092d --- /dev/null +++ b/lib/cretonne/src/bforest/mod.rs @@ -0,0 +1,172 @@ +//! A forest of B+-trees. +//! +//! This module provides a data structures representing a set of small ordered sets or maps. +//! It is implemented as a forest of B+-trees all allocating nodes out of the same pool. +//! +//! **These are not general purpose data structures that are somehow magically faster that the +//! standard library's `BTreeSet` and `BTreeMap` types.** +//! +//! The tradeoffs are different: +//! +//! - Keys and values are expected to be small and copyable. We optimize for 32-bit types. +//! - A comparator object is used to compare keys, allowing smaller "context free" keys. +//! - Empty trees have a very small 32-bit footprint. +//! - All the trees in a forest can be cleared in constant time. + +use std::borrow::BorrowMut; +use std::cmp::Ordering; + +mod map; +mod node; +mod path; +mod pool; +mod set; + +pub use self::map::{MapForest, BPlusMap, MapCursor}; +pub use self::set::{SetForest, BPlusSet, SetCursor}; + +use self::node::NodeData; +use self::path::Path; +use self::pool::NodePool; + +/// The maximum branching factor of an inner node in a B+-tree. +/// The minimum number of outgoing edges is `INNER_SIZE/2`. +const INNER_SIZE: usize = 8; + +/// Given the worst case branching factor of `INNER_SIZE/2` = 4, this is the +/// worst case path length from the root node to a leaf node in a tree with 2^32 +/// entries. We would run out of node references before we hit `MAX_PATH`. +const MAX_PATH: usize = 16; + +/// Key comparator. +/// +/// Keys don't need to implement `Ord`. They are compared using a comparator object which +/// provides a context for comparison. +pub trait BPlusComparator +where + K: Copy, +{ + /// Compare keys `a` and `b`. + /// + /// This relation must provide a total ordering or the key space. + fn cmp(&self, a: K, b: K) -> Ordering; + + /// Binary search for `k` in an ordered slice. + /// + /// Assume that `s` is already sorted according to this ordering, search for the key `k`. + /// + /// Returns `Ok(idx)` if `k` was found in the slice or `Err(idx)` with the position where it + /// should be inserted to preserve the ordering. + fn search(&self, k: K, s: &[K]) -> Result { + s.binary_search_by(|x| self.cmp(*x, k)) + } +} + +/// Trivial comparator that doesn't actually provide any context. +impl BPlusComparator for () +where + K: Copy + Ord, +{ + fn cmp(&self, a: K, b: K) -> Ordering { + a.cmp(&b) + } +} + +/// Family of types shared by the map and set forest implementations. +trait Forest { + /// The key type is present for both sets and maps. + type Key: Copy; + + /// The value type is `()` for sets. + type Value: Copy; + + /// An array of keys for the leaf nodes. + type LeafKeys: Copy + BorrowMut<[Self::Key]>; + + /// An array of values for the leaf nodes. + type LeafValues: Copy + BorrowMut<[Self::Value]>; + + /// Type used for key comparisons. + type Comparator: BPlusComparator; + + /// Splat a single key into a whole array. + fn splat_key(key: Self::Key) -> Self::LeafKeys; + + /// Splat a single value inst a whole array + fn splat_value(value: Self::Value) -> Self::LeafValues; +} + +/// A reference to a B+-tree node. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +struct Node(u32); +entity_impl!(Node, "node"); + +/// Empty type to be used as the "value" in B-trees representing sets. +#[derive(Clone, Copy)] +struct SetValue(); + +/// Insert `x` into `s` at position `i`, pushing out the last element. +fn slice_insert(s: &mut [T], i: usize, x: T) { + for j in (i + 1..s.len()).rev() { + s[j] = s[j - 1]; + } + s[i] = x; +} + +/// Shift elements in `s` to the left by `n` positions. +fn slice_shift(s: &mut [T], n: usize) { + for j in 0..s.len() - n { + s[j] = s[j + n]; + } +} + +#[cfg(test)] +mod test { + use super::*; + use entity::EntityRef; + use ir::Ebb; + + #[test] + fn comparator() { + let ebb1 = Ebb::new(1); + let ebb2 = Ebb::new(2); + let ebb3 = Ebb::new(3); + let ebb4 = Ebb::new(4); + let vals = [ebb1, ebb2, ebb4]; + let comp = (); + assert_eq!(comp.search(ebb1, &vals), Ok(0)); + assert_eq!(comp.search(ebb3, &vals), Err(2)); + assert_eq!(comp.search(ebb4, &vals), Ok(2)); + } + + #[test] + fn slice_insertion() { + let mut a = ['a', 'b', 'c', 'd']; + + slice_insert(&mut a[0..1], 0, 'e'); + assert_eq!(a, ['e', 'b', 'c', 'd']); + + slice_insert(&mut a, 0, 'a'); + assert_eq!(a, ['a', 'e', 'b', 'c']); + + slice_insert(&mut a, 3, 'g'); + assert_eq!(a, ['a', 'e', 'b', 'g']); + + slice_insert(&mut a, 1, 'h'); + assert_eq!(a, ['a', 'h', 'e', 'b']); + } + + #[test] + fn slice_shifting() { + let mut a = ['a', 'b', 'c', 'd']; + + slice_shift(&mut a[0..1], 1); + assert_eq!(a, ['a', 'b', 'c', 'd']); + + slice_shift(&mut a[1..], 1); + assert_eq!(a, ['a', 'c', 'd', 'd']); + + slice_shift(&mut a, 2); + assert_eq!(a, ['d', 'd', 'd', 'd']); + } +} diff --git a/lib/cretonne/src/bforest/node.rs b/lib/cretonne/src/bforest/node.rs new file mode 100644 index 0000000000..53a96f9b14 --- /dev/null +++ b/lib/cretonne/src/bforest/node.rs @@ -0,0 +1,813 @@ +//! B+-tree nodes. + +use std::borrow::{Borrow, BorrowMut}; +use std::fmt; +use super::{Forest, Node, INNER_SIZE, SetValue, slice_insert, slice_shift}; + +/// B+-tree node. +/// +/// A B+-tree has different node types for inner nodes and leaf nodes. Inner nodes contain M node +/// references and M-1 keys while leaf nodes contain N keys and values. Values for M and N are +/// chosen such that a node is exactly 64 bytes (a cache line) when keys and values are 32 bits +/// each. +/// +/// An inner node contains at least M/2 node references unless it is the right-most node at its +/// level. A leaf node contains at least N/2 keys unless it is the right-most leaf. +pub(super) enum NodeData { + Inner { + /// The number of keys in this node. + /// The number of node references is always one more. + size: u8, + + /// Keys discriminating sub-trees. + /// + /// The key in `keys[i]` is greater than all keys in `tree[i]` and less than or equal to + /// all keys in `tree[i+1]`. + keys: [F::Key; INNER_SIZE - 1], + + /// Sub-trees. + tree: [Node; INNER_SIZE], + }, + Leaf { + /// Number of key-value pairs in this node. + size: u8, + + // Key array. + keys: F::LeafKeys, + + // Value array. + vals: F::LeafValues, + }, + /// An unused node on the free list. + Free { next: Option }, +} + +// Implement `Clone` and `Copy` manually, because deriving them would also require `Forest` to +// implement `Clone`. +impl Copy for NodeData {} +impl Clone for NodeData { + fn clone(&self) -> Self { + *self + } +} + +impl NodeData { + /// Is this a free/unused node? + pub fn is_free(&self) -> bool { + match self { + &NodeData::Free { .. } => true, + _ => false, + } + } + + /// Get the number of entries in this node. + /// + /// This is the number of outgoing edges in an inner node, or the number of key-value pairs in + /// a leaf node. + pub fn entries(&self) -> usize { + match self { + &NodeData::Inner { size, .. } => usize::from(size) + 1, + &NodeData::Leaf { size, .. } => usize::from(size), + &NodeData::Free { .. } => panic!("freed node"), + } + } + + /// Create an inner node with a single key and two sub-trees. + pub fn inner(left: Node, key: F::Key, right: Node) -> NodeData { + // Splat the key and right node to the whole array. + // Saves us from inventing a default/reserved value. + let mut tree = [right; INNER_SIZE]; + tree[0] = left; + NodeData::Inner { + size: 1, + keys: [key; INNER_SIZE - 1], + tree, + } + } + + /// Create a leaf node with a single key-value pair. + pub fn leaf(key: F::Key, value: F::Value) -> NodeData { + NodeData::Leaf { + size: 1, + keys: F::splat_key(key), + vals: F::splat_value(value), + } + } + + /// Unwrap an inner node into two slices (keys, trees). + pub fn unwrap_inner(&self) -> (&[F::Key], &[Node]) { + match self { + &NodeData::Inner { + size, + ref keys, + ref tree, + } => { + let size = usize::from(size); + // TODO: We could probably use `get_unchecked()` here since `size` is always in + // range. + (&keys[0..size], &tree[0..size + 1]) + } + _ => panic!("Expected inner node"), + } + } + + /// Unwrap a leaf node into two slices (keys, values) of the same length. + pub fn unwrap_leaf(&self) -> (&[F::Key], &[F::Value]) { + match self { + &NodeData::Leaf { + size, + ref keys, + ref vals, + } => { + let size = usize::from(size); + let keys = keys.borrow(); + let vals = vals.borrow(); + // TODO: We could probably use `get_unchecked()` here since `size` is always in + // range. + (&keys[0..size], &vals[0..size]) + } + _ => panic!("Expected leaf node"), + } + } + + /// Unwrap a mutable leaf node into two slices (keys, values) of the same length. + pub fn unwrap_leaf_mut(&mut self) -> (&mut [F::Key], &mut [F::Value]) { + match self { + &mut NodeData::Leaf { + size, + ref mut keys, + ref mut vals, + } => { + let size = usize::from(size); + let keys = keys.borrow_mut(); + let vals = vals.borrow_mut(); + // TODO: We could probably use `get_unchecked_mut()` here since `size` is always in + // range. + (&mut keys[0..size], &mut vals[0..size]) + } + _ => panic!("Expected leaf node"), + } + } + + /// Get the critical key for a leaf node. + /// This is simply the first key. + pub fn leaf_crit_key(&self) -> F::Key { + match self { + &NodeData::Leaf { size, ref keys, .. } => { + debug_assert!(size > 0, "Empty leaf node"); + keys.borrow()[0] + } + _ => panic!("Expected leaf node"), + } + } + + /// Try to insert `(key, node)` at key-position `index` in an inner node. + /// This means that `key` is inserted at `keys[i]` and `node` is inserted at `tree[i + 1]`. + /// If the node is full, this leaves the node unchanged and returns false. + pub fn try_inner_insert(&mut self, index: usize, key: F::Key, node: Node) -> bool { + match self { + &mut NodeData::Inner { + ref mut size, + ref mut keys, + ref mut tree, + } => { + let sz = usize::from(*size); + debug_assert!(sz <= keys.len()); + debug_assert!(index <= sz, "Can't insert at {} with {} keys", index, sz); + + if let Some(ks) = keys.get_mut(0..sz + 1) { + *size = (sz + 1) as u8; + slice_insert(ks, index, key); + slice_insert(&mut tree[1..sz + 2], index, node); + true + } else { + false + } + } + _ => panic!("Expected inner node"), + } + } + + /// Try to insert `key, value` at `index` in a leaf node, but fail and return false if the node + /// is full. + pub fn try_leaf_insert(&mut self, index: usize, key: F::Key, value: F::Value) -> bool { + match self { + &mut NodeData::Leaf { + ref mut size, + ref mut keys, + ref mut vals, + } => { + let sz = usize::from(*size); + let keys = keys.borrow_mut(); + let vals = vals.borrow_mut(); + debug_assert!(sz <= keys.len()); + debug_assert!(index <= sz); + + if let Some(ks) = keys.get_mut(0..sz + 1) { + *size = (sz + 1) as u8; + slice_insert(ks, index, key); + slice_insert(&mut vals[0..sz + 1], index, value); + true + } else { + false + } + } + _ => panic!("Expected leaf node"), + } + } + + /// Split off the second half of this node. + /// It is assumed that this a completely full inner or leaf node. + /// + /// The `insert_index` parameter is the position where an insertion was tried and failed. The + /// node will be split in half with a bias towards an even split after the insertion is retried. + pub fn split(&mut self, insert_index: usize) -> SplitOff { + match self { + &mut NodeData::Inner { + ref mut size, + ref keys, + ref tree, + } => { + debug_assert_eq!(usize::from(*size), keys.len(), "Node not full"); + + // Number of tree entries in the lhs node. + let l_ents = split_pos(tree.len(), insert_index + 1); + let r_ents = tree.len() - l_ents; + + // With INNER_SIZE=8, we get l_ents=4 and: + // + // self: [ n0 k0 n1 k1 n2 k2 n3 k3 n4 k4 n5 k5 n6 k6 n7 ] + // lhs: [ n0 k0 n1 k1 n2 k2 n3 ] + // crit_key = k3 (not present in either node) + // rhs: [ n4 k4 n5 k5 n6 k6 n7 ] + + // 1. Truncate the LHS. + *size = (l_ents - 1) as u8; + + // 2. Copy second half to `rhs_data`. + let mut r_keys = *keys; + r_keys[0..r_ents - 1].copy_from_slice(&keys[l_ents..]); + + let mut r_tree = *tree; + r_tree[0..r_ents].copy_from_slice(&tree[l_ents..]); + + SplitOff { + lhs_entries: l_ents, + rhs_entries: r_ents, + crit_key: keys[l_ents - 1], + rhs_data: NodeData::Inner { + size: (r_ents - 1) as u8, + keys: r_keys, + tree: r_tree, + }, + } + } + &mut NodeData::Leaf { + ref mut size, + ref keys, + ref vals, + } => { + let o_keys = keys.borrow(); + let o_vals = vals.borrow(); + debug_assert_eq!(usize::from(*size), o_keys.len(), "Node not full"); + + let l_size = split_pos(o_keys.len(), insert_index); + let r_size = o_keys.len() - l_size; + + // 1. Truncate the LHS node at `l_size`. + *size = l_size as u8; + + // 2. Copy second half to `rhs_data`. + let mut r_keys = *keys; + r_keys.borrow_mut()[0..r_size].copy_from_slice(&o_keys[l_size..]); + + let mut r_vals = *vals; + r_vals.borrow_mut()[0..r_size].copy_from_slice(&o_vals[l_size..]); + + SplitOff { + lhs_entries: l_size, + rhs_entries: r_size, + crit_key: o_keys[l_size], + rhs_data: NodeData::Leaf { + size: r_size as u8, + keys: r_keys, + vals: r_vals, + }, + } + } + _ => panic!("Expected leaf node"), + } + } + + /// Remove the sub-tree at `index` from this inner node. + /// + /// Note that `index` refers to a sub-tree entry and not a key entry as it does for + /// `try_inner_insert()`. It is possible to remove the first sub-tree (which can't be inserted + /// by `try_inner_insert()`). + /// + /// Return an indication of the node's health (i.e. below half capacity). + pub fn inner_remove(&mut self, index: usize) -> Removed { + match self { + &mut NodeData::Inner { + ref mut size, + ref mut keys, + ref mut tree, + } => { + let ents = usize::from(*size) + 1; + debug_assert!(ents <= tree.len()); + debug_assert!(index < ents); + // Leave an invalid 0xff size when node becomes empty. + *size = ents.wrapping_sub(2) as u8; + if ents > 1 { + slice_shift(&mut keys[index.saturating_sub(1)..ents - 1], 1); + } + slice_shift(&mut tree[index..ents], 1); + Removed::new(index, ents - 1, tree.len()) + } + _ => panic!("Expected inner node"), + } + } + + /// Remove the key-value pair at `index` from this leaf node. + /// + /// Return an indication of the node's health (i.e. below half capacity). + pub fn leaf_remove(&mut self, index: usize) -> Removed { + match self { + &mut NodeData::Leaf { + ref mut size, + ref mut keys, + ref mut vals, + } => { + let sz = usize::from(*size); + let keys = keys.borrow_mut(); + let vals = vals.borrow_mut(); + *size -= 1; + slice_shift(&mut keys[index..sz], 1); + slice_shift(&mut vals[index..sz], 1); + Removed::new(index, sz - 1, keys.len()) + } + _ => panic!("Expected leaf node"), + } + } + + /// Balance this node with its right sibling. + /// + /// It is assumed that the current node has underflowed. Look at the right sibling node and do + /// one of two things: + /// + /// 1. Move all entries to the right node, leaving this node empty, or + /// 2. Distribute entries evenly between the two nodes. + /// + /// In the first case, `None` is returned. In the second case, the new critical key for the + /// right sibling node is returned. + pub fn balance(&mut self, crit_key: F::Key, rhs: &mut NodeData) -> Option { + match (self, rhs) { + (&mut NodeData::Inner { + size: ref mut l_size, + keys: ref mut l_keys, + tree: ref mut l_tree, + }, + &mut NodeData::Inner { + size: ref mut r_size, + keys: ref mut r_keys, + tree: ref mut r_tree, + }) => { + let l_ents = usize::from(*l_size) + 1; + let r_ents = usize::from(*r_size) + 1; + let ents = l_ents + r_ents; + + if ents <= r_tree.len() { + // All entries will fit in the RHS node. + // We'll leave the LHS node empty, but first use it as a scratch space. + *l_size = 0; + // Insert `crit_key` between the two nodes. + l_keys[l_ents - 1] = crit_key; + l_keys[l_ents..ents - 1].copy_from_slice(&r_keys[0..r_ents - 1]); + r_keys[0..ents - 1].copy_from_slice(&l_keys[0..ents - 1]); + l_tree[l_ents..ents].copy_from_slice(&r_tree[0..r_ents]); + r_tree[0..ents].copy_from_slice(&l_tree[0..ents]); + *r_size = (ents - 1) as u8; + None + } else { + // The entries don't all fit in one node. Distribute some from RHS -> LHS. + // Split evenly with a bias to putting one entry in LHS. + let r_goal = ents / 2; + let l_goal = ents - r_goal; + debug_assert!(l_goal > l_ents, "Node must be underflowed"); + + l_keys[l_ents - 1] = crit_key; + l_keys[l_ents..l_goal - 1].copy_from_slice(&r_keys[0..l_goal - 1 - l_ents]); + l_tree[l_ents..l_goal].copy_from_slice(&r_tree[0..l_goal - l_ents]); + *l_size = (l_goal - 1) as u8; + + let new_crit = r_keys[r_ents - r_goal - 1]; + slice_shift(&mut r_keys[0..r_ents - 1], r_ents - r_goal); + slice_shift(&mut r_tree[0..r_ents], r_ents - r_goal); + *r_size = (r_goal - 1) as u8; + + Some(new_crit) + } + } + (&mut NodeData::Leaf { + size: ref mut l_size, + keys: ref mut l_keys, + vals: ref mut l_vals, + }, + &mut NodeData::Leaf { + size: ref mut r_size, + keys: ref mut r_keys, + vals: ref mut r_vals, + }) => { + let l_ents = usize::from(*l_size); + let l_keys = l_keys.borrow_mut(); + let l_vals = l_vals.borrow_mut(); + let r_ents = usize::from(*r_size); + let r_keys = r_keys.borrow_mut(); + let r_vals = r_vals.borrow_mut(); + let ents = l_ents + r_ents; + + if ents <= r_vals.len() { + // We can fit all entries in the RHS node. + // We'll leave the LHS node empty, but first use it as a scratch space. + *l_size = 0; + l_keys[l_ents..ents].copy_from_slice(&r_keys[0..r_ents]); + r_keys[0..ents].copy_from_slice(&l_keys[0..ents]); + l_vals[l_ents..ents].copy_from_slice(&r_vals[0..r_ents]); + r_vals[0..ents].copy_from_slice(&l_vals[0..ents]); + *r_size = ents as u8; + None + } else { + // The entries don't all fit in one node. Distribute some from RHS -> LHS. + // Split evenly with a bias to putting one entry in LHS. + let r_goal = ents / 2; + let l_goal = ents - r_goal; + debug_assert!(l_goal > l_ents, "Node must be underflowed"); + + l_keys[l_ents..l_goal].copy_from_slice(&r_keys[0..l_goal - l_ents]); + l_vals[l_ents..l_goal].copy_from_slice(&r_vals[0..l_goal - l_ents]); + *l_size = l_goal as u8; + + slice_shift(&mut r_keys[0..r_ents], r_ents - r_goal); + slice_shift(&mut r_vals[0..r_ents], r_ents - r_goal); + *r_size = r_goal as u8; + + Some(r_keys[0]) + } + } + _ => panic!("Mismatched nodes"), + } + } +} + +/// Find the right split position for halving a full node with `len` entries to recover from a +/// failed insertion at `ins`. +/// +/// If `len` is even, we should split straight down the middle regardless of `len`. +/// +/// If `len` is odd, we should split the node such that the two halves are the same size after the +/// insertion is retried. +fn split_pos(len: usize, ins: usize) -> usize { + // Anticipate `len` being a compile time constant, so this all folds away when `len` is even. + if ins <= len / 2 { + len / 2 + } else { + (len + 1) / 2 + } +} + +/// The result of splitting off the second half of a node. +pub(super) struct SplitOff { + /// The number of entries left in the original node which becomes the left-hand-side of the + /// pair. This is the number of outgoing node edges for an inner node, and the number of + /// key-value pairs for a leaf node. + pub lhs_entries: usize, + + /// The number of entries in the new RHS node. + pub rhs_entries: usize, + + /// The critical key separating the LHS and RHS nodes. All keys in the LHS sub-tree are less + /// than the critical key, and all entries in the RHS sub-tree are greater or equal to the + /// critical key. + pub crit_key: F::Key, + + /// The RHS node data containing the elements that were removed from the original node (now the + /// LHS). + pub rhs_data: NodeData, +} + +/// The result of removing an entry from a node. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) enum Removed { + /// An entry was removed, and the node is still in good shape. + Healthy, + + /// The node is in good shape after removing the rightmost element. + Rightmost, + + /// The node has too few entries now, and it should be balanced with a sibling node. + Underflow, + + /// The last entry was removed. For an inner node, this means that the `keys` array is empty + /// and there is just a single sub-tree left. + Empty, +} + +impl Removed { + /// Create a `Removed` status from a size and capacity. + fn new(removed: usize, new_size: usize, capacity: usize) -> Removed { + if 2 * new_size >= capacity { + if removed == new_size { + Removed::Rightmost + } else { + Removed::Healthy + } + } else if new_size > 0 { + Removed::Underflow + } else { + Removed::Empty + } + } +} + +// Display ": value" or nothing at all for `()`. +pub(super) trait ValDisp { + fn valfmt(&self, f: &mut fmt::Formatter) -> fmt::Result; +} + +impl ValDisp for SetValue { + fn valfmt(&self, _: &mut fmt::Formatter) -> fmt::Result { + Ok(()) + } +} + +impl ValDisp for T { + fn valfmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, ":{}", self) + } +} + +impl fmt::Display for NodeData +where + F: Forest, + F::Key: fmt::Display, + F::Value: ValDisp, +{ + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self { + &NodeData::Inner { size, keys, tree } => { + write!(f, "[ {}", tree[0])?; + for i in 0..usize::from(size) { + write!(f, " {} {}", keys[i], tree[i + 1])?; + } + write!(f, " ]") + } + &NodeData::Leaf { size, keys, vals } => { + let keys = keys.borrow(); + let vals = vals.borrow(); + write!(f, "[")?; + for i in 0..usize::from(size) { + write!(f, " {}", keys[i])?; + vals[i].valfmt(f)?; + } + write!(f, " ]") + } + &NodeData::Free { next: Some(n) } => write!(f, "[ free -> {} ]", n), + &NodeData::Free { next: None } => write!(f, "[ free ]"), + } + } +} + +#[cfg(test)] +mod test { + use std::mem; + use super::*; + + // Forest impl for a set implementation. + struct TF(); + + impl Forest for TF { + type Key = char; + type Value = SetValue; + type LeafKeys = [char; 15]; + type LeafValues = [SetValue; 15]; + type Comparator = (); + + fn splat_key(key: Self::Key) -> Self::LeafKeys { + [key; 15] + } + + fn splat_value(value: Self::Value) -> Self::LeafValues { + [value; 15] + } + } + + #[test] + fn inner() { + let n1 = Node(1); + let n2 = Node(2); + let n3 = Node(3); + let n4 = Node(4); + let mut inner = NodeData::::inner(n1, 'c', n4); + assert_eq!(mem::size_of_val(&inner), 64); + assert_eq!(inner.to_string(), "[ node1 c node4 ]"); + + assert!(inner.try_inner_insert(0, 'a', n2)); + assert_eq!(inner.to_string(), "[ node1 a node2 c node4 ]"); + + assert!(inner.try_inner_insert(1, 'b', n3)); + assert_eq!(inner.to_string(), "[ node1 a node2 b node3 c node4 ]"); + + for i in 3..7 { + assert!(inner.try_inner_insert( + usize::from(i), + ('a' as u8 + i) as char, + Node(i as u32 + 2), + )); + } + assert_eq!( + inner.to_string(), + "[ node1 a node2 b node3 c node4 d node5 e node6 f node7 g node8 ]" + ); + + // Now the node is full and insertion should fail anywhere. + assert!(!inner.try_inner_insert(0, 'x', n3)); + assert!(!inner.try_inner_insert(4, 'x', n3)); + assert!(!inner.try_inner_insert(7, 'x', n3)); + + // Splitting should be independent of the hint because we have an even number of node + // references. + let saved = inner.clone(); + let sp = inner.split(1); + assert_eq!(sp.lhs_entries, 4); + assert_eq!(sp.rhs_entries, 4); + assert_eq!(sp.crit_key, 'd'); + // The critical key is not present in either of the resulting nodes. + assert_eq!(inner.to_string(), "[ node1 a node2 b node3 c node4 ]"); + assert_eq!(sp.rhs_data.to_string(), "[ node5 e node6 f node7 g node8 ]"); + + assert_eq!(inner.inner_remove(0), Removed::Underflow); + assert_eq!(inner.to_string(), "[ node2 b node3 c node4 ]"); + + assert_eq!(inner.inner_remove(1), Removed::Underflow); + assert_eq!(inner.to_string(), "[ node2 c node4 ]"); + + assert_eq!(inner.inner_remove(1), Removed::Underflow); + assert_eq!(inner.to_string(), "[ node2 ]"); + + assert_eq!(inner.inner_remove(0), Removed::Empty); + + inner = saved; + let sp = inner.split(6); + assert_eq!(sp.lhs_entries, 4); + assert_eq!(sp.rhs_entries, 4); + assert_eq!(sp.crit_key, 'd'); + assert_eq!(inner.to_string(), "[ node1 a node2 b node3 c node4 ]"); + assert_eq!(sp.rhs_data.to_string(), "[ node5 e node6 f node7 g node8 ]"); + } + + #[test] + fn leaf() { + let mut leaf = NodeData::::leaf('d', SetValue()); + assert_eq!(leaf.to_string(), "[ d ]"); + + assert!(leaf.try_leaf_insert(0, 'a', SetValue())); + assert_eq!(leaf.to_string(), "[ a d ]"); + assert!(leaf.try_leaf_insert(1, 'b', SetValue())); + assert!(leaf.try_leaf_insert(2, 'c', SetValue())); + assert_eq!(leaf.to_string(), "[ a b c d ]"); + for i in 4..15 { + assert!(leaf.try_leaf_insert( + usize::from(i), + ('a' as u8 + i) as char, + SetValue(), + )); + } + assert_eq!(leaf.to_string(), "[ a b c d e f g h i j k l m n o ]"); + + // Now the node is full and insertion should fail anywhere. + assert!(!leaf.try_leaf_insert(0, 'x', SetValue())); + assert!(!leaf.try_leaf_insert(8, 'x', SetValue())); + assert!(!leaf.try_leaf_insert(15, 'x', SetValue())); + + // The index given to `split` is not the split position, it's a hint for balancing the node. + let saved = leaf.clone(); + let sp = leaf.split(12); + assert_eq!(sp.lhs_entries, 8); + assert_eq!(sp.rhs_entries, 7); + assert_eq!(sp.crit_key, 'i'); + assert_eq!(leaf.to_string(), "[ a b c d e f g h ]"); + assert_eq!(sp.rhs_data.to_string(), "[ i j k l m n o ]"); + + assert!(leaf.try_leaf_insert(8, 'i', SetValue())); + assert_eq!(leaf.leaf_remove(2), Removed::Healthy); + assert_eq!(leaf.to_string(), "[ a b d e f g h i ]"); + assert_eq!(leaf.leaf_remove(7), Removed::Underflow); + assert_eq!(leaf.to_string(), "[ a b d e f g h ]"); + + leaf = saved; + let sp = leaf.split(7); + assert_eq!(sp.lhs_entries, 7); + assert_eq!(sp.rhs_entries, 8); + assert_eq!(sp.crit_key, 'h'); + assert_eq!(leaf.to_string(), "[ a b c d e f g ]"); + assert_eq!(sp.rhs_data.to_string(), "[ h i j k l m n o ]"); + } + + #[test] + fn optimal_split_pos() { + // An even split is easy. + assert_eq!(split_pos(8, 0), 4); + assert_eq!(split_pos(8, 8), 4); + + // Easy cases for odd splits. + assert_eq!(split_pos(7, 0), 3); + assert_eq!(split_pos(7, 7), 4); + + // If the insertion point is the same as the split position, we + // will append to the lhs node. + assert_eq!(split_pos(7, 3), 3); + assert_eq!(split_pos(7, 4), 4); + } + + #[test] + fn inner_balance() { + let n1 = Node(1); + let n2 = Node(2); + let n3 = Node(3); + let mut lhs = NodeData::::inner(n1, 'a', n2); + assert!(lhs.try_inner_insert(1, 'b', n3)); + assert_eq!(lhs.to_string(), "[ node1 a node2 b node3 ]"); + + let n11 = Node(11); + let n12 = Node(12); + let mut rhs = NodeData::::inner(n11, 'p', n12); + + for i in 1..4 { + assert!(rhs.try_inner_insert( + usize::from(i), + ('p' as u8 + i) as char, + Node(i as u32 + 12), + )); + } + assert_eq!( + rhs.to_string(), + "[ node11 p node12 q node13 r node14 s node15 ]" + ); + + // 3+5 elements fit in RHS. + assert_eq!(lhs.balance('o', &mut rhs), None); + assert_eq!( + rhs.to_string(), + "[ node1 a node2 b node3 o node11 p node12 q node13 r node14 s node15 ]" + ); + + // 2+8 elements are redistributed. + lhs = NodeData::::inner(Node(20), 'x', Node(21)); + assert_eq!(lhs.balance('y', &mut rhs), Some('o')); + assert_eq!( + lhs.to_string(), + "[ node20 x node21 y node1 a node2 b node3 ]" + ); + assert_eq!( + rhs.to_string(), + "[ node11 p node12 q node13 r node14 s node15 ]" + ); + } + + #[test] + fn leaf_balance() { + let mut lhs = NodeData::::leaf('a', SetValue()); + for i in 1..6 { + assert!(lhs.try_leaf_insert( + usize::from(i), + ('a' as u8 + i) as char, + SetValue(), + )); + } + assert_eq!(lhs.to_string(), "[ a b c d e f ]"); + + let mut rhs = NodeData::::leaf('0', SetValue()); + for i in 1..8 { + assert!(rhs.try_leaf_insert( + usize::from(i), + ('0' as u8 + i) as char, + SetValue(), + )); + } + assert_eq!(rhs.to_string(), "[ 0 1 2 3 4 5 6 7 ]"); + + // 6+8 elements all fits in rhs. + assert_eq!(lhs.balance('0', &mut rhs), None); + assert_eq!(rhs.to_string(), "[ a b c d e f 0 1 2 3 4 5 6 7 ]"); + + assert!(lhs.try_leaf_insert(0, 'x', SetValue())); + assert!(lhs.try_leaf_insert(1, 'y', SetValue())); + assert!(lhs.try_leaf_insert(2, 'z', SetValue())); + assert_eq!(lhs.to_string(), "[ x y z ]"); + + // 3+14 elements need redistribution. + assert_eq!(lhs.balance('a', &mut rhs), Some('0')); + assert_eq!(lhs.to_string(), "[ x y z a b c d e f ]"); + assert_eq!(rhs.to_string(), "[ 0 1 2 3 4 5 6 7 ]"); + } +} diff --git a/lib/cretonne/src/bforest/path.rs b/lib/cretonne/src/bforest/path.rs new file mode 100644 index 0000000000..2dceb805f4 --- /dev/null +++ b/lib/cretonne/src/bforest/path.rs @@ -0,0 +1,817 @@ +//! A path from the root of a B+-tree to a leaf node. + +use std::borrow::Borrow; +use std::marker::PhantomData; +use super::{Forest, Node, NodeData, NodePool, MAX_PATH, BPlusComparator, slice_insert, slice_shift}; +use super::node::Removed; + +#[cfg(test)] +use std::fmt; + +pub(super) struct Path { + /// Number of path entries including the root and leaf nodes. + size: usize, + + /// Path of node references from the root to a leaf node. + node: [Node; MAX_PATH], + + /// Entry number in each node. + entry: [u8; MAX_PATH], + + unused: PhantomData, +} + +impl Default for Path { + fn default() -> Path { + Path { + size: 0, + node: [Node(0); MAX_PATH], + entry: [0; MAX_PATH], + unused: PhantomData, + } + } +} + +impl Path { + /// Reset path by searching for `key` starting from `root`. + /// + /// If `key` is in the tree, returns the corresponding value and leaved the path pointing at + /// the entry. Otherwise returns `None` and: + /// + /// - A key smaller than all stored keys returns a path to the first entry of the first leaf. + /// - A key larger than all stored keys returns a path to one beyond the last element of the + /// last leaf. + /// - A key between the stored keys of adjacent leaf nodes returns a path to one beyond the + /// last entry of the first of the leaf nodes. + /// + pub fn find( + &mut self, + key: F::Key, + root: Node, + pool: &NodePool, + comp: &F::Comparator, + ) -> Option { + let mut node = root; + for level in 0.. { + self.size = level + 1; + self.node[level] = node; + match &pool[node] { + &NodeData::Inner { size, keys, tree } => { + // Invariant: `tree[i]` contains keys smaller than + // `keys[i]`, greater or equal to `keys[i-1]`. + let i = match comp.search(key, &keys[0..size.into()]) { + // We hit an existing key, so follow the >= branch. + Ok(i) => i + 1, + // Key is less than `keys[i]`, so follow the < branch. + Err(i) => i, + }; + self.entry[level] = i as u8; + node = tree[i]; + } + &NodeData::Leaf { size, keys, vals } => { + // For a leaf we want either the found key or an insert position. + return match comp.search(key, &keys.borrow()[0..size.into()]) { + Ok(i) => { + self.entry[level] = i as u8; + Some(vals.borrow()[i]) + } + Err(i) => { + self.entry[level] = i as u8; + None + } + }; + } + &NodeData::Free { .. } => panic!("Free {} reached from {}", node, root), + } + } + unreachable!(); + } + + /// Move this path to the next key-value pair and return it. + pub fn next(&mut self, pool: &NodePool) -> Option<(F::Key, F::Value)> { + match self.leaf_pos() { + None => return None, + Some((node, entry)) => { + let (keys, vals) = pool[node].unwrap_leaf(); + if entry + 1 < keys.len() { + self.entry[self.size - 1] += 1; + return Some((keys[entry + 1], vals[entry + 1])); + } + } + } + + // The current leaf node is exhausted. Move to the next one. + let leaf_level = self.size - 1; + self.next_node(leaf_level, pool).map(|node| { + let (keys, vals) = pool[node].unwrap_leaf(); + (keys[0], vals[0]) + }) + } + + /// Move this path to the previous key-value pair and return it. + /// + /// If the path is at the off-the-end position, go to the last key-value pair. + /// + /// If the path is already at the first key-value pair, leave it there and return `None`. + pub fn prev(&mut self, root: Node, pool: &NodePool) -> Option<(F::Key, F::Value)> { + // We use `size == 0` as a generic off-the-end position. + if self.size == 0 { + self.goto_subtree_last(0, root, pool); + let (node, entry) = self.leaf_pos().unwrap(); + let (keys, vals) = pool[node].unwrap_leaf(); + return Some((keys[entry], vals[entry])); + } + + match self.leaf_pos() { + None => return None, + Some((node, entry)) => { + if entry > 0 { + self.entry[self.size - 1] -= 1; + let (keys, vals) = pool[node].unwrap_leaf(); + return Some((keys[entry - 1], vals[entry - 1])); + } + } + } + + // The current leaf node is exhausted. Move to the previous one. + self.prev_leaf(pool).map(|node| { + let (keys, vals) = pool[node].unwrap_leaf(); + let e = self.leaf_entry(); + (keys[e], vals[e]) + }) + } + + /// Move path to the first entry of the next node at level, if one exists. + /// + /// Returns the new node if it exists. + /// + /// Reset the path to `size = 0` and return `None` if there is no next node. + fn next_node(&mut self, level: usize, pool: &NodePool) -> Option { + match self.right_sibling_branch_level(level, pool) { + None => { + self.size = 0; + None + } + Some(bl) => { + let (_, bnodes) = pool[self.node[bl]].unwrap_inner(); + self.entry[bl] += 1; + let mut node = bnodes[usize::from(self.entry[bl])]; + + for l in bl + 1..level { + self.node[l] = node; + self.entry[l] = 0; + node = pool[node].unwrap_inner().1[0]; + } + + self.node[level] = node; + self.entry[level] = 0; + Some(node) + } + } + } + + /// Move the path to the last entry of the previous leaf node, if one exists. + /// + /// Returns the new leaf node if it exists. + /// + /// Leave the path unchanged and returns `None` if we are already at the first leaf node. + fn prev_leaf(&mut self, pool: &NodePool) -> Option { + self.left_sibling_branch_level(self.size - 1).map(|bl| { + let entry = self.entry[bl] - 1; + self.entry[bl] = entry; + let (_, bnodes) = pool[self.node[bl]].unwrap_inner(); + self.goto_subtree_last(bl + 1, bnodes[usize::from(entry)], pool) + }) + } + + /// Move this path to the last position for the sub-tree at `level, root`. + fn goto_subtree_last(&mut self, level: usize, root: Node, pool: &NodePool) -> Node { + let mut node = root; + for l in level.. { + self.node[l] = node; + match &pool[node] { + &NodeData::Inner { size, ref tree, .. } => { + self.entry[l] = size; + node = tree[usize::from(size)]; + } + &NodeData::Leaf { size, .. } => { + self.entry[l] = size - 1; + self.size = l + 1; + break; + } + &NodeData::Free { .. } => panic!("Free {} reached from {}", node, root), + } + } + node + } + + /// Set the root node and point the path at the first entry of the node. + pub fn set_root_node(&mut self, root: Node) { + self.size = 1; + self.node[0] = root; + self.entry[0] = 0; + } + + /// Get the current leaf node and entry, if any. + pub fn leaf_pos(&self) -> Option<(Node, usize)> { + let i = self.size.wrapping_sub(1); + self.node.get(i).map(|&n| (n, self.entry[i].into())) + } + + /// Get the current leaf node. + fn leaf_node(&self) -> Node { + self.node[self.size - 1] + } + + /// Get the current entry in the leaf node. + fn leaf_entry(&self) -> usize { + self.entry[self.size - 1].into() + } + + /// Is this path pointing to the first entry in the tree? + /// This corresponds to the smallest key. + fn at_first_entry(&self) -> bool { + self.entry[0..self.size].iter().all(|&i| i == 0) + } + + /// Get a mutable reference to the current value. + /// This assumes that there is a current value. + pub fn value_mut<'a>(&self, pool: &'a mut NodePool) -> &'a mut F::Value { + &mut pool[self.leaf_node()].unwrap_leaf_mut().1[self.leaf_entry()] + } + + /// Insert the key-value pair at the current position. + /// The current position must be the correct insertion location for the key. + /// This function does not check for duplicate keys. Use `find` or similar for that. + /// Returns the new root node. + pub fn insert(&mut self, key: F::Key, value: F::Value, pool: &mut NodePool) -> Node { + if !self.try_leaf_insert(key, value, pool) { + self.split_and_insert(key, value, pool); + } + self.node[0] + } + + /// Try to insert `key, value` at the current position, but fail and return false if the leaf + /// node is full. + fn try_leaf_insert(&self, key: F::Key, value: F::Value, pool: &mut NodePool) -> bool { + let index = self.leaf_entry(); + + // The case `index == 0` should only ever happen when there are no earlier leaf nodes, + // otherwise we should have appended to the previous leaf node instead. This invariant + // means that we don't need to update keys stored in inner nodes here. + debug_assert!(index > 0 || self.at_first_entry()); + + pool[self.leaf_node()].try_leaf_insert(index, key, value) + } + + /// Split the current leaf node and then insert `key, value`. + /// This should only be used if `try_leaf_insert()` fails. + fn split_and_insert(&mut self, mut key: F::Key, value: F::Value, pool: &mut NodePool) { + let orig_root = self.node[0]; + + // Loop invariant: We need to split the node at `level` and then retry a failed insertion. + // The items to insert are either `(key, ins_node)` or `(key, value)`. + let mut ins_node = None; + let mut split; + for level in (0..self.size).rev() { + // Split the current node. + let mut node = self.node[level]; + let mut entry = self.entry[level].into(); + split = pool[node].split(entry); + let rhs_node = pool.alloc_node(split.rhs_data); + + // Should the path be moved to the new RHS node? + // Prefer the smaller node if we're right in the middle. + // Prefer to append to LHS all other things being equal. + // + // When inserting into an inner node (`ins_node.is_some()`), we must point to a valid + // entry in the current node since the new entry is inserted *after* the insert + // location. + if entry > split.lhs_entries || + (entry == split.lhs_entries && + (split.lhs_entries > split.rhs_entries || ins_node.is_some())) + { + node = rhs_node; + entry -= split.lhs_entries; + self.node[level] = node; + self.entry[level] = entry as u8; + } + + // Now that we have a not-full node, it must be possible to insert. + match ins_node { + None => { + assert!(pool[node].try_leaf_insert(entry, key, value)); + // If we inserted at the front of the new rhs_node leaf, we need to propagate + // the inserted key as the critical key instead of the previous front key. + if entry == 0 && node == rhs_node { + split.crit_key = key; + } + } + Some(n) => { + assert!(pool[node].try_inner_insert(entry, key, n)); + // The lower level was moved to the new RHS node, so make sure that is + // reflected here. + if n == self.node[level + 1] { + self.entry[level] += 1; + } + } + } + + // We are now done with the current level, but `rhs_node` must be inserted in the inner + // node above us. If we're already at level 0, the root node needs to be split. + key = split.crit_key; + ins_node = Some(rhs_node); + if level > 0 { + let pnode = &mut pool[self.node[level - 1]]; + let pentry = self.entry[level - 1].into(); + if pnode.try_inner_insert(pentry, key, rhs_node) { + // If this level level was moved to the new RHS node, update parent entry. + if node == rhs_node { + self.entry[level - 1] += 1; + } + return; + } + } + } + + // If we get here we have split the original root node and need to add an extra level. + let rhs_node = ins_node.expect("empty path"); + let root = pool.alloc_node(NodeData::inner(orig_root, key, rhs_node)); + let entry = if self.node[0] == rhs_node { 1 } else { 0 }; + self.size += 1; + slice_insert(&mut self.node[0..self.size], 0, root); + slice_insert(&mut self.entry[0..self.size], 0, entry); + } + + /// Remove the key-value pair at the current position and advance the path to the next + /// key-value pair, leaving the path in a normalized state. + /// + /// Return the new root node. + pub fn remove(&mut self, pool: &mut NodePool) -> Option { + let e = self.leaf_entry(); + match pool[self.leaf_node()].leaf_remove(e) { + Removed::Healthy => { + if e == 0 { + self.update_crit_key(pool) + } + Some(self.node[0]) + } + status => self.balance_nodes(status, pool), + } + } + + /// Get the critical key for the current node at `level`. + /// + /// The critical key is less than or equal to all keys in the sub-tree at `level` and greater + /// than all keys to the left of the current node at `level`. + /// + /// The left-most node at any level does not have a critical key. + fn current_crit_key(&self, level: usize, pool: &NodePool) -> Option { + // Find the level containing the critical key for the current node. + self.left_sibling_branch_level(level).map(|bl| { + let (keys, _) = pool[self.node[bl]].unwrap_inner(); + keys[usize::from(self.entry[bl]) - 1] + }) + } + + /// Update the critical key after removing the front entry of the leaf node. + fn update_crit_key(&mut self, pool: &mut NodePool) { + // Find the inner level containing the critical key for the current leaf node. + let crit_level = match self.left_sibling_branch_level(self.size - 1) { + None => return, + Some(l) => l, + }; + let crit_kidx = self.entry[crit_level] - 1; + + // Extract the new critical key from the leaf node. + let crit_key = pool[self.leaf_node()].leaf_crit_key(); + let crit_node = self.node[crit_level]; + + match &mut pool[crit_node] { + &mut NodeData::Inner { size, ref mut keys, .. } => { + debug_assert!(crit_kidx < size); + keys[usize::from(crit_kidx)] = crit_key; + } + _ => panic!("Expected inner node"), + } + } + + + /// Given that the current leaf node is in an unhealthy (underflowed or even empty) status, + /// balance it with sibling nodes. + /// + /// Return the new root node. + fn balance_nodes(&mut self, status: Removed, pool: &mut NodePool) -> Option { + // The current leaf node is not in a healthy state, and its critical key may have changed + // too. + // + // Start by dealing with a changed critical key for the leaf level. + if status != Removed::Empty && self.leaf_entry() == 0 { + self.update_crit_key(pool); + } + + let leaf_level = self.size - 1; + if self.heal_level(status, leaf_level, pool) { + // Tree has become empty. + self.size = 0; + return None; + } + + // Discard the root node if it has shrunk to a single sub-tree. + let mut ns = 0; + while let &NodeData::Inner { size: 0, ref tree, .. } = &pool[self.node[ns]] { + ns += 1; + self.node[ns] = tree[0]; + } + + if ns > 0 { + for l in 0..ns { + pool.free_node(self.node[l]); + } + + // Shift the whole array instead of just 0..size because `self.size` may be cleared + // here if the path is pointing off-the-end. + slice_shift(&mut self.node, ns); + slice_shift(&mut self.entry, ns); + + if self.size > 0 { + self.size -= ns; + } + } + + // Return the root node, even when `size=0` indicating that we're at the off-the-end + // position. + Some(self.node[0]) + } + + /// After removing an entry from the node at `level`, check its health and rebalance as needed. + /// + /// Leave the path up to and including `level` in a normalized state where all entries are in + /// bounds. + /// + /// Returns true if the tree becomes empty. + fn heal_level(&mut self, status: Removed, level: usize, pool: &mut NodePool) -> bool { + match status { + Removed::Healthy => {} + Removed::Rightmost => { + // The rightmost entry was removed from the curent node, so move the path so it + // points at the first entry of the next node at this level. + debug_assert_eq!( + usize::from(self.entry[level]), + pool[self.node[level]].entries() + ); + self.next_node(level, pool); + } + Removed::Underflow => self.underflowed_node(level, pool), + Removed::Empty => return self.empty_node(level, pool), + } + false + } + + /// The current node at `level` has underflowed, meaning that it is below half capacity but + /// not completely empty. + /// + /// Handle this by balancing entries with the right sibling node. + /// + /// Leave the path up to and including `level` in a valid state that points to the same entry. + fn underflowed_node(&mut self, level: usize, pool: &mut NodePool) { + // Look for a right sibling node at this level. If none exists, we allow the underflowed + // node to persist as the right-most node at its level. + if let Some((crit_key, rhs_node)) = self.right_sibling(level, pool) { + // New critical key for the updated right sibling node. + let new_ck: Option; + let empty; + // Make a COPY of the sibling node to avoid fighting the borrow checker. + let mut rhs = pool[rhs_node]; + match pool[self.node[level]].balance(crit_key, &mut rhs) { + None => { + // Everything got moved to the RHS node. + new_ck = self.current_crit_key(level, pool); + empty = true; + } + Some(key) => { + // Entries moved from RHS node. + new_ck = Some(key); + empty = false; + } + } + // Put back the updated RHS node data. + pool[rhs_node] = rhs; + // Update the critical key for the RHS node unless it has become a left-most + // node. + if let Some(ck) = new_ck { + self.update_right_crit_key(level, ck, pool); + } + if empty { + let empty_tree = self.empty_node(level, pool); + debug_assert!(!empty_tree); + } + + // Any Removed::Rightmost state must have been cleared above by merging nodes. If the + // current entry[level] was one off the end of the node, it will now point at a proper + // entry. + debug_assert!(usize::from(self.entry[level]) < pool[self.node[level]].entries()); + } else { + // There's no right sibling at this level, so the node can't be rebalanced. + // Check if we are in an off-the-end position. + if usize::from(self.entry[level]) >= pool[self.node[level]].entries() { + self.size = 0; + } + } + } + + /// The current node at `level` has become empty. + /// + /// Remove the node from its parent node and leave the path in a normalized state. This means + /// that the path at this level will go through the right sibling of this node. + /// + /// If the current node has no right sibling, set `self.size = 0`. + /// + /// Returns true if the tree becomes empty. + fn empty_node(&mut self, level: usize, pool: &mut NodePool) -> bool { + pool.free_node(self.node[level]); + if level == 0 { + // We just deleted the root node, so the tree is now empty. + return true; + } + + // Get the right sibling node before recursively removing nodes. + let rhs_node = self.right_sibling(level, pool).map(|(_, n)| n); + + // Remove the current sub-tree from the parent node. + let pl = level - 1; + let pe = self.entry[pl].into(); + let status = pool[self.node[pl]].inner_remove(pe); + self.heal_level(status, pl, pool); + + // Finally update the path at this level. + match rhs_node { + // We'll leave `self.entry[level]` unchanged. It can be non-zero after moving node + // entries to the right sibling node. + Some(rhs) => self.node[level] = rhs, + // We have no right sibling, so we must have deleted the right-most + // entry. The path should be moved to the "off-the-end" position. + None => self.size = 0, + } + false + } + + /// Find the level where the right sibling to the current node at `level` branches off. + /// + /// This will be an inner node with two adjacent sub-trees: In one the current node at level is + /// a right-most node, in the other, the right sibling is a left-most node. + /// + /// Returns `None` if the current node is a right-most node so no right sibling exists. + fn right_sibling_branch_level(&self, level: usize, pool: &NodePool) -> Option { + (0..level).rposition(|l| match &pool[self.node[l]] { + &NodeData::Inner { size, .. } => self.entry[l] < size, + _ => panic!("Expected inner node"), + }) + } + + /// Find the level where the left sibling to the current node at `level` branches off. + fn left_sibling_branch_level(&self, level: usize) -> Option { + self.entry[0..level].iter().rposition(|&e| e != 0) + } + + /// Get the right sibling node to the current node at `level`. + /// Also return the critical key between the current node and the right sibling. + fn right_sibling(&self, level: usize, pool: &NodePool) -> Option<(F::Key, Node)> { + // Find the critical level: The deepest level where two sibling subtrees contain the + // current node and its right sibling. + self.right_sibling_branch_level(level, pool).map(|bl| { + // Extract the critical key and the `bl+1` node. + let be = usize::from(self.entry[bl]); + let crit_key; + let mut node; + { + let (keys, tree) = pool[self.node[bl]].unwrap_inner(); + crit_key = keys[be]; + node = tree[be + 1]; + } + + // Follow left-most links back down to `level`. + for _ in bl + 1..level { + node = pool[node].unwrap_inner().1[0]; + } + + (crit_key, node) + }) + } + + /// Update the critical key for the right sibling node at `level`. + fn update_right_crit_key(&self, level: usize, crit_key: F::Key, pool: &mut NodePool) { + let bl = self.right_sibling_branch_level(level, pool).expect( + "No right sibling exists", + ); + match &mut pool[self.node[bl]] { + &mut NodeData::Inner { ref mut keys, .. } => { + keys[usize::from(self.entry[bl])] = crit_key; + } + _ => panic!("Expected inner node"), + } + } + + /// Normalize the path position such that it is either pointing at a real entry or `size=0` + /// indicating "off-the-end". + pub fn normalize(&mut self, pool: &mut NodePool) { + if let Some((leaf, entry)) = self.leaf_pos() { + if entry >= pool[leaf].entries() { + let leaf_level = self.size - 1; + self.next_node(leaf_level, pool); + } + } + } +} + +#[cfg(test)] +impl Path { + /// Check the internal consistency of this path. + pub fn verify(&self, pool: &NodePool) { + for level in 0..self.size { + match &pool[self.node[level]] { + &NodeData::Inner { size, tree, .. } => { + assert!( + level < self.size - 1, + "Expected leaf node at level {}", + level + ); + assert!( + self.entry[level] <= size, + "OOB inner entry {}/{} at level {}", + self.entry[level], + size, + level + ); + assert_eq!( + self.node[level + 1], + tree[usize::from(self.entry[level])], + "Node mismatch at level {}", + level + ); + } + &NodeData::Leaf { size, .. } => { + assert_eq!(level, self.size - 1, "Expected inner node"); + assert!( + self.entry[level] <= size, + "OOB leaf entry {}/{}", + self.entry[level], + size, + ); + } + &NodeData::Free { .. } => { + panic!("Free {} in path", self.node[level]); + } + } + } + } +} + +#[cfg(test)] +impl fmt::Display for Path { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + if self.size == 0 { + write!(f, "") + } else { + write!(f, "{}[{}]", self.node[0], self.entry[0])?; + for i in 1..self.size { + write!(f, "--{}[{}]", self.node[i], self.entry[i])?; + } + Ok(()) + } + } +} + +#[cfg(test)] +mod test { + use std::cmp::Ordering; + use super::*; + use super::super::{Forest, NodePool, NodeData}; + + struct TC(); + + impl BPlusComparator for TC { + fn cmp(&self, a: i32, b: i32) -> Ordering { + a.cmp(&b) + } + } + + struct TF(); + + impl Forest for TF { + type Key = i32; + type Value = char; + type LeafKeys = [i32; 7]; + type LeafValues = [char; 7]; + type Comparator = TC; + + fn splat_key(key: Self::Key) -> Self::LeafKeys { + [key; 7] + } + + fn splat_value(value: Self::Value) -> Self::LeafValues { + [value; 7] + } + } + + #[test] + fn search_single_leaf() { + // Testing Path::new() for trees with a single leaf node. + let mut pool = NodePool::::new(); + let root = pool.alloc_node(NodeData::leaf(10, 'a')); + let mut p = Path::default(); + let comp = TC(); + + // Search for key less than stored key. + assert_eq!(p.find(5, root, &pool, &comp), None); + assert_eq!(p.size, 1); + assert_eq!(p.node[0], root); + assert_eq!(p.entry[0], 0); + + // Search for stored key. + assert_eq!(p.find(10, root, &pool, &comp), Some('a')); + assert_eq!(p.size, 1); + assert_eq!(p.node[0], root); + assert_eq!(p.entry[0], 0); + + // Search for key greater than stored key. + assert_eq!(p.find(15, root, &pool, &comp), None); + assert_eq!(p.size, 1); + assert_eq!(p.node[0], root); + assert_eq!(p.entry[0], 1); + + // Modify leaf node to contain two values. + match pool[root] { + NodeData::Leaf { + ref mut size, + ref mut keys, + ref mut vals, + } => { + *size = 2; + keys[1] = 20; + vals[1] = 'b'; + } + _ => unreachable!(), + } + + // Search for key between stored keys. + assert_eq!(p.find(15, root, &pool, &comp), None); + assert_eq!(p.size, 1); + assert_eq!(p.node[0], root); + assert_eq!(p.entry[0], 1); + + // Search for key greater than stored keys. + assert_eq!(p.find(25, root, &pool, &comp), None); + assert_eq!(p.size, 1); + assert_eq!(p.node[0], root); + assert_eq!(p.entry[0], 2); + } + + #[test] + fn search_single_inner() { + // Testing Path::new() for trees with a single inner node and two leaves. + let mut pool = NodePool::::new(); + let leaf1 = pool.alloc_node(NodeData::leaf(10, 'a')); + let leaf2 = pool.alloc_node(NodeData::leaf(20, 'b')); + let root = pool.alloc_node(NodeData::inner(leaf1, 20, leaf2)); + let mut p = Path::default(); + let comp = TC(); + + // Search for key less than stored keys. + assert_eq!(p.find(5, root, &pool, &comp), None); + assert_eq!(p.size, 2); + assert_eq!(p.node[0], root); + assert_eq!(p.entry[0], 0); + assert_eq!(p.node[1], leaf1); + assert_eq!(p.entry[1], 0); + + assert_eq!(p.find(10, root, &pool, &comp), Some('a')); + assert_eq!(p.size, 2); + assert_eq!(p.node[0], root); + assert_eq!(p.entry[0], 0); + assert_eq!(p.node[1], leaf1); + assert_eq!(p.entry[1], 0); + + // Midway between the two leaf nodes. + assert_eq!(p.find(15, root, &pool, &comp), None); + assert_eq!(p.size, 2); + assert_eq!(p.node[0], root); + assert_eq!(p.entry[0], 0); + assert_eq!(p.node[1], leaf1); + assert_eq!(p.entry[1], 1); + + assert_eq!(p.find(20, root, &pool, &comp), Some('b')); + assert_eq!(p.size, 2); + assert_eq!(p.node[0], root); + assert_eq!(p.entry[0], 1); + assert_eq!(p.node[1], leaf2); + assert_eq!(p.entry[1], 0); + + assert_eq!(p.find(25, root, &pool, &comp), None); + assert_eq!(p.size, 2); + assert_eq!(p.node[0], root); + assert_eq!(p.entry[0], 1); + assert_eq!(p.node[1], leaf2); + assert_eq!(p.entry[1], 1); + } +} diff --git a/lib/cretonne/src/bforest/pool.rs b/lib/cretonne/src/bforest/pool.rs new file mode 100644 index 0000000000..2dfcb35074 --- /dev/null +++ b/lib/cretonne/src/bforest/pool.rs @@ -0,0 +1,197 @@ +//! B+-tree node pool. + +use entity::PrimaryMap; +use std::ops::{Index, IndexMut}; +use super::{Forest, Node, NodeData}; + +/// A pool of nodes, including a free list. +pub(super) struct NodePool { + nodes: PrimaryMap>, + freelist: Option, +} + +impl NodePool { + /// Allocate a new empty pool of nodes. + pub fn new() -> NodePool { + NodePool { + nodes: PrimaryMap::new(), + freelist: None, + } + } + + /// Free all nodes. + pub fn clear(&mut self) { + self.nodes.clear(); + self.freelist = None; + } + + /// Allocate a new node containing `data`. + pub fn alloc_node(&mut self, data: NodeData) -> Node { + debug_assert!(!data.is_free(), "can't allocate free node"); + match self.freelist { + Some(node) => { + // Remove this node from the free list. + match self.nodes[node] { + NodeData::Free { next } => self.freelist = next, + _ => panic!("Invalid {} on free list", node), + } + self.nodes[node] = data; + node + } + None => { + // The free list is empty. Allocate a new node. + self.nodes.push(data) + } + } + } + + /// Free a node. + pub fn free_node(&mut self, node: Node) { + // Quick check for a double free. + debug_assert!(!self.nodes[node].is_free(), "{} is already free", node); + self.nodes[node] = NodeData::Free { next: self.freelist }; + self.freelist = Some(node); + } +} + +#[cfg(test)] +impl NodePool { + /// Verify the consistency of the tree rooted at `node`. + pub fn verify_tree(&self, node: Node, comp: &F::Comparator) + where + NodeData: ::std::fmt::Display, + F::Key: ::std::fmt::Display, + { + use std::borrow::Borrow; + use std::cmp::Ordering; + use super::BPlusComparator; + use entity::SparseSet; + + // The root node can't be an inner node with just a single sub-tree. It should have been + // pruned. + if let &NodeData::Inner { size, .. } = &self[node] { + assert!(size > 0, "Root must have more than one sub-tree"); + } + + let mut done = SparseSet::new(); + let mut todo = Vec::new(); + + // Todo-list entries are: + // 1. Optional LHS key which must be <= all node entries. + // 2. The node reference. + // 3. Optional RHS key which must be > all node entries. + todo.push((None, node, None)); + + while let Some((lkey, node, rkey)) = todo.pop() { + assert_eq!( + done.insert(node), + None, + "Node appears more than once in tree" + ); + let mut lower = lkey; + + match &self[node] { + &NodeData::Inner { size, keys, tree } => { + let size = size as usize; + let capacity = tree.len(); + let keys = &keys[0..size]; + + // Verify occupancy. + // Right-most nodes can be small, but others must be at least half full. + assert!( + rkey.is_none() || (size + 1) * 2 >= capacity, + "Only {}/{} entries in {}:{}, upper={}", + size + 1, + capacity, + node, + self[node], + rkey.unwrap() + ); + + // Queue up the sub-trees, checking for duplicates. + for i in 0..size + 1 { + // Get an upper bound for node[i]. + let upper = keys.get(i).cloned().or(rkey); + + // Check that keys are strictly monotonic. + if let (Some(a), Some(b)) = (lower, upper) { + assert_eq!( + comp.cmp(a, b), + Ordering::Less, + "Key order {} < {} failed in {}: {}", + a, + b, + node, + self[node] + ); + } + + // Queue up the sub-tree. + todo.push((lower, tree[i], upper)); + + // Set a lower bound for the next tree. + lower = upper; + } + } + &NodeData::Leaf { size, keys, .. } => { + let size = size as usize; + let capacity = keys.borrow().len(); + let keys = &keys.borrow()[0..size]; + + // Verify occupancy. + // Right-most nodes can be small, but others must be at least half full. + assert!(size > 0, "Leaf {} is empty", node); + assert!( + rkey.is_none() || size * 2 >= capacity, + "Only {}/{} entries in {}:{}, upper={}", + size, + capacity, + node, + self[node], + rkey.unwrap() + ); + + for i in 0..size + 1 { + let upper = keys.get(i).cloned().or(rkey); + + // Check that keys are strictly monotonic. + if let (Some(a), Some(b)) = (lower, upper) { + let wanted = if i == 0 { + Ordering::Equal + } else { + Ordering::Less + }; + assert_eq!( + comp.cmp(a, b), + wanted, + "Key order for {} - {} failed in {}: {}", + a, + b, + node, + self[node] + ); + } + + // Set a lower bound for the next key. + lower = upper; + } + } + &NodeData::Free { .. } => panic!("Free {} reached", node), + } + } + } +} + +impl Index for NodePool { + type Output = NodeData; + + fn index(&self, index: Node) -> &Self::Output { + self.nodes.index(index) + } +} + +impl IndexMut for NodePool { + fn index_mut(&mut self, index: Node) -> &mut Self::Output { + self.nodes.index_mut(index) + } +} diff --git a/lib/cretonne/src/bforest/set.rs b/lib/cretonne/src/bforest/set.rs new file mode 100644 index 0000000000..e768ab64b9 --- /dev/null +++ b/lib/cretonne/src/bforest/set.rs @@ -0,0 +1,486 @@ +//! Forest of sets. + +use packed_option::PackedOption; +use std::marker::PhantomData; +use super::{INNER_SIZE, BPlusComparator, Forest, NodePool, Node, NodeData, Path, SetValue}; + +/// Tag type defining forest types for a set. +struct SetTypes(PhantomData<(K, C)>); + +impl Forest for SetTypes +where + K: Copy, + C: BPlusComparator, +{ + type Key = K; + type Value = SetValue; + type LeafKeys = [K; 2 * INNER_SIZE - 1]; + type LeafValues = [SetValue; 2 * INNER_SIZE - 1]; + type Comparator = C; + + fn splat_key(key: Self::Key) -> Self::LeafKeys { + [key; 2 * INNER_SIZE - 1] + } + + fn splat_value(value: Self::Value) -> Self::LeafValues { + [value; 2 * INNER_SIZE - 1] + } +} + +/// Memory pool for a forest of `BPlusSet` instances. +pub struct SetForest +where + K: Copy, + C: BPlusComparator, +{ + nodes: NodePool>, +} + +impl SetForest +where + K: Copy, + C: BPlusComparator, +{ + /// Create a new empty forest. + pub fn new() -> SetForest { + SetForest { nodes: NodePool::new() } + } + + /// Clear all sets in the forest. + /// + /// All `BPlusSet` instances belong to this forest are invalidated and should no longer be used. + pub fn clear(&mut self) { + self.nodes.clear(); + } +} + +/// B-tree representing an ordered set of `K`s using `C` for comparing elements. +/// +/// This is not a general-purpose replacement for `BTreeSet`. See the [module +/// documentation](index.html) for more information about design tradeoffs. +pub struct BPlusSet +where + K: Copy, + C: BPlusComparator, +{ + root: PackedOption, + unused: PhantomData<(K, C)>, +} + +impl BPlusSet +where + K: Copy, + C: BPlusComparator, +{ + /// Make an empty set. + pub fn new() -> BPlusSet { + BPlusSet { + root: None.into(), + unused: PhantomData, + } + } + + /// Is this an empty set? + pub fn is_empty(&self) -> bool { + self.root.is_none() + } + + /// Does the set contain `key`?. + pub fn contains(&self, key: K, forest: &SetForest, comp: &C) -> bool { + self.root + .expand() + .and_then(|root| Path::default().find(key, root, &forest.nodes, comp)) + .is_some() + } + + /// Try to insert `key` into the set. + /// + /// If the set did not contain `key`, insert it and return true. + /// + /// If `key` is already present, don't change the set and return false. + pub fn insert(&mut self, key: K, forest: &mut SetForest, comp: &C) -> bool { + self.cursor(forest, comp).insert(key) + } + + /// Remove `key` from the set and return true. + /// + /// If `key` was not present in the set, return false. + pub fn remove(&mut self, key: K, forest: &mut SetForest, comp: &C) -> bool { + let mut c = self.cursor(forest, comp); + if c.goto(key) { + c.remove(); + true + } else { + false + } + } + + /// Create a cursor for navigating this set. The cursor is initially positioned off the end of + /// the set. + pub fn cursor<'a>( + &'a mut self, + forest: &'a mut SetForest, + comp: &'a C, + ) -> SetCursor<'a, K, C> { + SetCursor::new(self, forest, comp) + } +} + +/// A position in a `BPlusSet` used to navigate and modify the ordered set. +/// +/// A cursor always points at an element in the set, or "off the end" which is a position after the +/// last element in the set. +pub struct SetCursor<'a, K, C> +where + K: 'a + Copy, + C: 'a + BPlusComparator, +{ + root: &'a mut PackedOption, + pool: &'a mut NodePool>, + comp: &'a C, + path: Path>, +} + +impl<'a, K, C> SetCursor<'a, K, C> +where + K: Copy, + C: BPlusComparator, +{ + /// Create a cursor with a default (invalid) location. + fn new( + container: &'a mut BPlusSet, + forest: &'a mut SetForest, + comp: &'a C, + ) -> SetCursor<'a, K, C> { + SetCursor { + root: &mut container.root, + pool: &mut forest.nodes, + comp, + path: Path::default(), + } + } + + /// Is this cursor pointing to an empty set? + pub fn is_empty(&self) -> bool { + self.root.is_none() + } + + /// Move cursor to the next element and return it. + /// + /// If the cursor reaches the end, return `None` and leave the cursor at the off-the-end + /// position. + pub fn next(&mut self) -> Option { + self.path.next(self.pool).map(|(k, _)| k) + } + + /// Move cursor to the previous element and return it. + /// + /// If the cursor is already pointing at the first element, leave it there and return `None`. + pub fn prev(&mut self) -> Option { + self.root.expand().and_then(|root| { + self.path.prev(root, self.pool).map(|(k, _)| k) + }) + } + + /// Get the current element, or `None` if the cursor is at the end. + pub fn elem(&self) -> Option { + self.path.leaf_pos().and_then(|(node, entry)| { + self.pool[node].unwrap_leaf().0.get(entry).cloned() + }) + } + + /// Move this cursor to `elem`. + /// + /// If `elem` is in the set, place the cursor at `elem` and return true. + /// + /// If `elem` is not in the set, place the cursor at the next larger element (or the end) and + /// return false. + pub fn goto(&mut self, elem: K) -> bool { + match self.root.expand() { + None => false, + Some(root) => { + if self.path.find(elem, root, self.pool, self.comp).is_some() { + true + } else { + self.path.normalize(self.pool); + false + } + } + } + } + + /// Try to insert `elem` into the set and leave the cursor at the inserted element. + /// + /// If the set did not contain `elem`, insert it and return true. + /// + /// If `elem` is already present, don't change the set, place the cursor at `goto(elem)`, and + /// return false. + pub fn insert(&mut self, elem: K) -> bool { + match self.root.expand() { + None => { + let root = self.pool.alloc_node(NodeData::leaf(elem, SetValue())); + *self.root = root.into(); + self.path.set_root_node(root); + true + } + Some(root) => { + // TODO: Optimize the case where `self.path` is already at the correct insert pos. + if self.path.find(elem, root, self.pool, self.comp).is_none() { + *self.root = self.path.insert(elem, SetValue(), self.pool).into(); + true + } else { + false + } + } + } + } + + /// Remove the current element (if any) and return it. + /// This advances the cursor to the next element after the removed one. + pub fn remove(&mut self) -> Option { + let elem = self.elem(); + if elem.is_some() { + *self.root = self.path.remove(self.pool).into(); + } + elem + } +} + +#[cfg(test)] +impl<'a, K, C> SetCursor<'a, K, C> +where + K: Copy + ::std::fmt::Display, + C: BPlusComparator, +{ + fn verify(&self) { + self.path.verify(self.pool); + self.root.map(|root| self.pool.verify_tree(root, self.comp)); + } + + /// Get a text version of the path to the current position. + fn tpath(&self) -> String { + self.path.to_string() + } +} + +#[cfg(test)] +mod test { + use std::mem; + use super::*; + use super::super::NodeData; + + #[test] + fn node_size() { + // check that nodes are cache line sized when keys are 32 bits. + type F = SetTypes; + assert_eq!(mem::size_of::>(), 64); + } + + #[test] + fn empty() { + let mut f = SetForest::::new(); + f.clear(); + + let mut s = BPlusSet::::new(); + assert!(s.is_empty()); + assert!(!s.contains(7, &f, &())); + + let c = SetCursor::new(&mut s, &mut f, &()); + c.verify(); + assert_eq!(c.elem(), None); + } + + #[test] + fn simple_cursor() { + let mut f = SetForest::::new(); + let mut s = BPlusSet::::new(); + let mut c = SetCursor::new(&mut s, &mut f, &()); + + assert!(c.insert(50)); + c.verify(); + assert_eq!(c.elem(), Some(50)); + + assert!(c.insert(100)); + c.verify(); + assert_eq!(c.elem(), Some(100)); + + assert!(c.insert(10)); + c.verify(); + assert_eq!(c.elem(), Some(10)); + + // Basic movement. + assert_eq!(c.next(), Some(50)); + assert_eq!(c.next(), Some(100)); + assert_eq!(c.next(), None); + assert_eq!(c.next(), None); + assert_eq!(c.prev(), Some(100)); + assert_eq!(c.prev(), Some(50)); + assert_eq!(c.prev(), Some(10)); + assert_eq!(c.prev(), None); + assert_eq!(c.prev(), None); + + assert!(c.goto(50)); + assert_eq!(c.elem(), Some(50)); + assert_eq!(c.remove(), Some(50)); + c.verify(); + + assert_eq!(c.elem(), Some(100)); + assert_eq!(c.remove(), Some(100)); + c.verify(); + assert_eq!(c.elem(), None); + assert_eq!(c.remove(), None); + c.verify(); + } + + #[test] + fn two_level_sparse_tree() { + let mut f = SetForest::::new(); + let mut s = BPlusSet::::new(); + let mut c = SetCursor::new(&mut s, &mut f, &()); + + // Insert enough elements that we get a two-level tree. + // Each leaf node holds 8 elements + assert!(c.is_empty()); + for i in 0..50 { + assert!(c.insert(i)); + assert_eq!(c.elem(), Some(i)); + } + assert!(!c.is_empty()); + + assert!(c.goto(0)); + assert_eq!(c.tpath(), "node2[0]--node0[0]"); + + assert_eq!(c.prev(), None); + for i in 1..50 { + assert_eq!(c.next(), Some(i)); + } + assert_eq!(c.next(), None); + for i in (0..50).rev() { + assert_eq!(c.prev(), Some(i)); + } + assert_eq!(c.prev(), None); + + assert!(c.goto(25)); + for i in 25..50 { + assert_eq!(c.remove(), Some(i)); + assert!(!c.is_empty()); + c.verify(); + } + + for i in (0..25).rev() { + assert!(!c.is_empty()); + assert_eq!(c.elem(), None); + assert_eq!(c.prev(), Some(i)); + assert_eq!(c.remove(), Some(i)); + c.verify(); + } + assert_eq!(c.elem(), None); + assert!(c.is_empty()); + } + + #[test] + fn three_level_sparse_tree() { + let mut f = SetForest::::new(); + let mut s = BPlusSet::::new(); + let mut c = SetCursor::new(&mut s, &mut f, &()); + + // Insert enough elements that we get a 3-level tree. + // Each leaf node holds 8 elements when filled up sequentially. + // Inner nodes hold 8 node pointers. + assert!(c.is_empty()); + for i in 0..150 { + assert!(c.insert(i)); + assert_eq!(c.elem(), Some(i)); + } + assert!(!c.is_empty()); + + assert!(c.goto(0)); + assert_eq!(c.tpath(), "node11[0]--node2[0]--node0[0]"); + + assert_eq!(c.prev(), None); + for i in 1..150 { + assert_eq!(c.next(), Some(i)); + } + assert_eq!(c.next(), None); + for i in (0..150).rev() { + assert_eq!(c.prev(), Some(i)); + } + assert_eq!(c.prev(), None); + + assert!(c.goto(125)); + for i in 125..150 { + assert_eq!(c.remove(), Some(i)); + assert!(!c.is_empty()); + c.verify(); + } + + for i in (0..125).rev() { + assert!(!c.is_empty()); + assert_eq!(c.elem(), None); + assert_eq!(c.prev(), Some(i)); + assert_eq!(c.remove(), Some(i)); + c.verify(); + } + assert_eq!(c.elem(), None); + assert!(c.is_empty()); + } + + // Generate a densely populated 4-level tree. + // + // Level 1: 1 root + // Level 2: 8 inner + // Level 3: 64 inner + // Level 4: 512 leafs, up to 7680 elements + // + // A 3-level tree can hold at most 960 elements. + fn dense4l(f: &mut SetForest) -> BPlusSet { + f.clear(); + let mut s = BPlusSet::new(); + + // Insert 400 elements in 7 passes over the range to avoid the half-full leaf node pattern + // that comes from sequential insertion. This will generate a normal leaf layer. + for n in 0..4000 { + assert!(s.insert((n * 7) % 4000, f, &())); + } + s + } + + #[test] + fn four_level() { + let mut f = SetForest::::new(); + let mut s = dense4l(&mut f); + let mut c = s.cursor(&mut f, &()); + + c.verify(); + + // Peel off a whole sub-tree of the root by deleting from the front. + // The 900 element is near the front of the second sub-tree. + assert!(c.goto(900)); + assert_eq!(c.tpath(), "node48[1]--node47[0]--node26[0]--node20[4]"); + assert!(c.goto(0)); + for i in 0..900 { + assert!(!c.is_empty()); + assert_eq!(c.remove(), Some(i)); + } + c.verify(); + assert_eq!(c.elem(), Some(900)); + + // Delete backwards from somewhere in the middle. + assert!(c.goto(3000)); + for i in (2000..3000).rev() { + assert_eq!(c.prev(), Some(i)); + assert_eq!(c.remove(), Some(i)); + assert_eq!(c.elem(), Some(3000)); + } + c.verify(); + + // Remove everything in a scattered manner, triggering many collapsing patterns. + for i in 0..4000 { + if c.goto((i * 7) % 4000) { + c.remove(); + } + } + assert!(c.is_empty()); + } + +} diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 6fbc651218..5de3c1dba3 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -15,6 +15,7 @@ pub mod dbg; #[macro_use] pub mod entity; +pub mod bforest; pub mod binemit; pub mod bitset; pub mod cursor; From d51a4c1065f2e274763a11d3214de716b1ec6aea Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 17 Nov 2017 16:03:40 -0800 Subject: [PATCH 1366/3084] Replace FunctionBuilder's Drop impl with a finalize function. (#193) * Replace FunctionBuilder's Drop impl with a finalize function. This has the advantage of not triggering assertion failures in the event of abandoning a partially-built function. It has the disadvantage of requiring users to call finalize() explicitly. --- lib/frontend/src/frontend.rs | 135 +++++++++++++++++++------------- lib/frontend/src/lib.rs | 2 + lib/wasm/src/func_translator.rs | 5 +- 3 files changed, 85 insertions(+), 57 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index ac86e3d64f..5127270cbd 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -8,6 +8,7 @@ use cretonne::ir::function::DisplayFunction; use cretonne::isa::TargetIsa; use ssa::{SSABuilder, SideEffects, Block}; use cretonne::entity::{EntityRef, EntityMap, EntitySet}; +use cretonne::packed_option::PackedOption; /// Structure used for translating a series of functions into Cretonne IL. /// @@ -48,8 +49,28 @@ struct EbbData { } struct Position { - ebb: Ebb, - basic_block: Block, + ebb: PackedOption, + basic_block: PackedOption, +} + +impl Position { + fn at(ebb: Ebb, basic_block: Block) -> Self { + Self { + ebb: PackedOption::from(ebb), + basic_block: PackedOption::from(basic_block), + } + } + + fn default() -> Self { + Self { + ebb: PackedOption::default(), + basic_block: PackedOption::default(), + } + } + + fn is_default(&self) -> bool { + self.ebb.is_none() && self.basic_block.is_none() + } } impl ILBuilder @@ -149,7 +170,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short .filter(|dest_ebb| unique.insert(*dest_ebb)) { self.builder.builder.ssa.declare_ebb_predecessor( dest_ebb, - self.builder.position.basic_block, + self.builder.position.basic_block.unwrap(), inst, ) } @@ -213,10 +234,7 @@ where func: func, srcloc: Default::default(), builder: builder, - position: Position { - ebb: Ebb::new(0), - basic_block: Block::new(0), - }, + position: Position::default(), } } @@ -245,13 +263,12 @@ where /// successor), the block will be declared filled and it will not be possible to append /// instructions to it. pub fn switch_to_block(&mut self, ebb: Ebb) { - if !self.builder.ebbs[self.position.ebb].pristine { - // First we check that the previous block has been filled. - debug_assert!( - self.is_unreachable() || self.builder.ebbs[self.position.ebb].filled, - "you have to fill your block before switching" - ); - } + // First we check that the previous block has been filled. + debug_assert!( + self.position.is_default() || self.is_unreachable() || self.is_pristine() || + self.is_filled(), + "you have to fill your block before switching" + ); // We cannot switch to a filled block debug_assert!( !self.builder.ebbs[ebb].filled, @@ -260,7 +277,7 @@ where let basic_block = self.builder.ssa.header_block(ebb); // Then we change the cursor position. - self.position = Position { ebb, basic_block }; + self.position = Position::at(ebb, basic_block); } /// Declares that all the predecessors of this block are known. @@ -299,7 +316,7 @@ where self.func, var, ty, - self.position.basic_block, + self.position.basic_block.unwrap(), ); self.handle_ssa_side_effects(side_effects); val @@ -311,7 +328,7 @@ where self.builder.ssa.def_var( var, val, - self.position.basic_block, + self.position.basic_block.unwrap(), ); } @@ -354,13 +371,13 @@ where /// Returns an object with the [`InstBuilder`](../cretonne/ir/builder/trait.InstBuilder.html) /// trait that allows to conveniently append an instruction to the current `Ebb` being built. pub fn ins<'short>(&'short mut self) -> FuncInstBuilder<'short, 'a, Variable> { - let ebb = self.position.ebb; + let ebb = self.position.ebb.unwrap(); FuncInstBuilder::new(self, ebb) } /// Make sure that the current EBB is inserted in the layout. pub fn ensure_inserted_ebb(&mut self) { - let ebb = self.position.ebb; + let ebb = self.position.ebb.unwrap(); if self.builder.ebbs[ebb].pristine { if !self.func.layout.is_ebb_inserted(ebb) { self.func.layout.append_ebb(ebb); @@ -382,7 +399,7 @@ where self.ensure_inserted_ebb(); FuncCursor::new(self.func) .with_srcloc(self.srcloc) - .at_bottom(self.position.ebb) + .at_bottom(self.position.ebb.unwrap()) } /// Append parameters to the given `Ebb` corresponding to the function @@ -410,6 +427,33 @@ where self.func.dfg.append_ebb_param(ebb, argtyp.value_type); } } + + /// Declare that translation of the current function is complete. This + /// resets the state of the `FunctionBuilder` in preparation to be used + /// for another function. + pub fn finalize(&mut self) { + // Check that all the `Ebb`s are filled and sealed. + debug_assert!( + self.builder.ebbs.keys().all(|ebb| { + self.builder.ebbs[ebb].pristine || self.builder.ssa.is_sealed(ebb) + }), + "all blocks should be sealed before dropping a FunctionBuilder" + ); + debug_assert!( + self.builder.ebbs.keys().all(|ebb| { + self.builder.ebbs[ebb].pristine || self.builder.ebbs[ebb].filled + }), + "all blocks should be filled before dropping a FunctionBuilder" + ); + + // Clear the state (but preserve the allocated buffers) in preparation + // for translation another function. + self.builder.clear(); + + // Reset srcloc and position to initial states. + self.srcloc = Default::default(); + self.position = Position::default(); + } } /// All the functions documented in the previous block are write-only and help you build a valid @@ -475,22 +519,25 @@ where pub fn is_unreachable(&self) -> bool { let is_entry = match self.func.layout.entry_block() { None => false, - Some(entry) => self.position.ebb == entry, + Some(entry) => self.position.ebb.unwrap() == entry, }; - !is_entry && self.builder.ssa.is_sealed(self.position.ebb) && - self.builder.ssa.predecessors(self.position.ebb).is_empty() + !is_entry && self.builder.ssa.is_sealed(self.position.ebb.unwrap()) && + self.builder + .ssa + .predecessors(self.position.ebb.unwrap()) + .is_empty() } /// Returns `true` if and only if no instructions have been added since the last call to /// `switch_to_block`. pub fn is_pristine(&self) -> bool { - self.builder.ebbs[self.position.ebb].pristine + self.builder.ebbs[self.position.ebb.unwrap()].pristine } /// Returns `true` if and only if a terminator instruction has been inserted since the /// last call to `switch_to_block`. pub fn is_filled(&self) -> bool { - self.builder.ebbs[self.position.ebb].filled + self.builder.ebbs[self.position.ebb.unwrap()].filled } /// Returns a displayable object for the function as it is. @@ -501,51 +548,25 @@ where } } -impl<'a, Variable> Drop for FunctionBuilder<'a, Variable> -where - Variable: EntityRef, -{ - /// When a `FunctionBuilder` goes out of scope, it means that the function is fully built. - fn drop(&mut self) { - // Check that all the `Ebb`s are filled and sealed. - debug_assert!( - self.builder.ebbs.keys().all(|ebb| { - self.builder.ebbs[ebb].pristine || self.builder.ssa.is_sealed(ebb) - }), - "all blocks should be sealed before dropping a FunctionBuilder" - ); - debug_assert!( - self.builder.ebbs.keys().all(|ebb| { - self.builder.ebbs[ebb].pristine || self.builder.ebbs[ebb].filled - }), - "all blocks should be filled before dropping a FunctionBuilder" - ); - - // Clear the state (but preserve the allocated buffers) in preparation - // for translation another function. - self.builder.clear(); - } -} - // Helper functions impl<'a, Variable> FunctionBuilder<'a, Variable> where Variable: EntityRef, { fn move_to_next_basic_block(&mut self) { - self.position.basic_block = self.builder.ssa.declare_ebb_body_block( - self.position.basic_block, - ); + self.position.basic_block = PackedOption::from(self.builder.ssa.declare_ebb_body_block( + self.position.basic_block.unwrap(), + )); } fn fill_current_block(&mut self) { - self.builder.ebbs[self.position.ebb].filled = true; + self.builder.ebbs[self.position.ebb.unwrap()].filled = true; } fn declare_successor(&mut self, dest_ebb: Ebb, jump_inst: Inst) { self.builder.ssa.declare_ebb_predecessor( dest_ebb, - self.position.basic_block, + self.position.basic_block.unwrap(), jump_inst, ); } @@ -668,6 +689,8 @@ mod tests { if lazy_seal { builder.seal_all_blocks(); } + + builder.finalize(); } let flags = settings::Flags::new(&settings::builder()); diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index ad76797129..12b8c8db73 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -128,6 +128,8 @@ //! } //! builder.ins().jump(block1, &[]); //! builder.seal_block(block1); +//! +//! builder.finalize(); //! } //! //! let flags = settings::Flags::new(&settings::builder()); diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 73aa7b433a..b5407decd9 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -94,7 +94,10 @@ impl FuncTranslator { self.state.initialize(&builder.func.signature, exit_block); parse_local_decls(&mut reader, &mut builder, num_params)?; - parse_function_body(reader, &mut builder, &mut self.state, environ) + parse_function_body(reader, &mut builder, &mut self.state, environ)?; + + builder.finalize(); + Ok(()) } } From d3778e56bbfb4f7dc6de4c9b4cf70708e2c7a6f1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 20 Nov 2017 14:09:25 -0800 Subject: [PATCH 1367/3084] Remove the "BPlus" prefix from bforest::* types. We'll just use the bforest:: namespace for these types, avoiding the confusing mix of prefixed and non-prefixed names. No functional change intended. --- lib/cretonne/src/bforest/map.rs | 60 ++++++++++++++++---------------- lib/cretonne/src/bforest/mod.rs | 10 +++--- lib/cretonne/src/bforest/path.rs | 4 +-- lib/cretonne/src/bforest/pool.rs | 2 +- lib/cretonne/src/bforest/set.rs | 46 ++++++++++++------------ 5 files changed, 61 insertions(+), 61 deletions(-) diff --git a/lib/cretonne/src/bforest/map.rs b/lib/cretonne/src/bforest/map.rs index 43855c4760..e9709c0b17 100644 --- a/lib/cretonne/src/bforest/map.rs +++ b/lib/cretonne/src/bforest/map.rs @@ -2,7 +2,7 @@ use packed_option::PackedOption; use std::marker::PhantomData; -use super::{INNER_SIZE, BPlusComparator, Forest, NodePool, Node, NodeData, Path}; +use super::{INNER_SIZE, Comparator, Forest, NodePool, Node, NodeData, Path}; /// Tag type defining forest types for a map. struct MapTypes(PhantomData<(K, V, C)>); @@ -11,7 +11,7 @@ impl Forest for MapTypes where K: Copy, V: Copy, - C: BPlusComparator, + C: Comparator, { type Key = K; type Value = V; @@ -28,12 +28,12 @@ where } } -/// Memory pool for a forest of `BPlusMap` instances. +/// Memory pool for a forest of `Map` instances. pub struct MapForest where K: Copy, V: Copy, - C: BPlusComparator, + C: Comparator, { nodes: NodePool>, } @@ -42,7 +42,7 @@ impl MapForest where K: Copy, V: Copy, - C: BPlusComparator, + C: Comparator, { /// Create a new empty forest. pub fn new() -> MapForest { @@ -51,7 +51,7 @@ where /// Clear all maps in the forest. /// - /// All `BPlusMap` instances belong to this forest are invalidated and should no longer be used. + /// All `Map` instances belong to this forest are invalidated and should no longer be used. pub fn clear(&mut self) { self.nodes.clear(); } @@ -61,25 +61,25 @@ where /// /// This is not a general-purpose replacement for `BTreeMap`. See the [module /// documentation](index.html) for more information about design tradeoffs. -pub struct BPlusMap +pub struct Map where K: Copy, V: Copy, - C: BPlusComparator, + C: Comparator, { root: PackedOption, unused: PhantomData<(K, V, C)>, } -impl BPlusMap +impl Map where K: Copy, V: Copy, - C: BPlusComparator, + C: Comparator, { /// Make an empty map. - pub fn new() -> BPlusMap { - BPlusMap { + pub fn new() -> Map { + Map { root: None.into(), unused: PhantomData, } @@ -130,11 +130,11 @@ where } #[cfg(test)] -impl BPlusMap +impl Map where K: Copy + ::std::fmt::Display, V: Copy, - C: BPlusComparator, + C: Comparator, { /// Verify consistency. fn verify(&self, forest: &MapForest, comp: &C) @@ -159,7 +159,7 @@ where } } -/// A position in a `BPlusMap` used to navigate and modify the ordered map. +/// A position in a `Map` used to navigate and modify the ordered map. /// /// A cursor always points at a key-value pair in the map, or "off the end" which is a position /// after the last entry in the map. @@ -167,7 +167,7 @@ pub struct MapCursor<'a, K, V, C> where K: 'a + Copy, V: 'a + Copy, - C: 'a + BPlusComparator, + C: 'a + Comparator, { root: &'a mut PackedOption, pool: &'a mut NodePool>, @@ -179,11 +179,11 @@ impl<'a, K, V, C> MapCursor<'a, K, V, C> where K: Copy, V: Copy, - C: BPlusComparator, + C: Comparator, { /// Create a cursor with a default (off-the-end) location. fn new( - container: &'a mut BPlusMap, + container: &'a mut Map, forest: &'a mut MapForest, comp: &'a C, ) -> MapCursor<'a, K, V, C> { @@ -289,7 +289,7 @@ impl<'a, K, V, C> MapCursor<'a, K, V, C> where K: Copy + ::std::fmt::Display, V: Copy + ::std::fmt::Display, - C: BPlusComparator, + C: Comparator, { fn verify(&self) { self.path.verify(self.pool); @@ -320,7 +320,7 @@ mod test { let mut f = MapForest::::new(); f.clear(); - let mut m = BPlusMap::::new(); + let mut m = Map::::new(); assert!(m.is_empty()); assert_eq!(m.get(7, &f, &()), None); @@ -338,7 +338,7 @@ mod test { #[test] fn inserting() { let f = &mut MapForest::::new(); - let mut m = BPlusMap::::new(); + let mut m = Map::::new(); // The first seven values stay in a single leaf node. assert_eq!(m.insert(50, 5.0, f, &()), None); @@ -428,8 +428,8 @@ mod test { // Various ways of splitting a full leaf node at level 0. let f = &mut MapForest::::new(); - fn full_leaf(f: &mut MapForest) -> BPlusMap { - let mut m = BPlusMap::new(); + fn full_leaf(f: &mut MapForest) -> Map { + let mut m = Map::new(); for n in 1..8 { m.insert(n * 10, n as f32 * 1.1, f, &()); } @@ -473,8 +473,8 @@ mod test { // 210, 220, ..., 270 // ... // 810, 820, ..., 870 - fn full(f: &mut MapForest) -> BPlusMap { - let mut m = BPlusMap::new(); + fn full(f: &mut MapForest) -> Map { + let mut m = Map::new(); // Start by inserting elements in order. // This should leave 8 leaf nodes with 4 elements in each. @@ -583,9 +583,9 @@ mod test { // Make a tree with two barely healthy leaf nodes: // [ 10 20 30 40 ] [ 50 60 70 80 ] - fn two_leaf(f: &mut MapForest) -> BPlusMap { + fn two_leaf(f: &mut MapForest) -> Map { f.clear(); - let mut m = BPlusMap::new(); + let mut m = Map::new(); for n in 1..9 { m.insert(n * 10, n as f32, f, &()); } @@ -679,9 +679,9 @@ mod test { // Make a 3-level tree with barely healthy nodes. // 1 root, 8 inner nodes, 7*4+5=33 leaf nodes, 4 entries each. - fn level3_sparse(f: &mut MapForest) -> BPlusMap { + fn level3_sparse(f: &mut MapForest) -> Map { f.clear(); - let mut m = BPlusMap::new(); + let mut m = Map::new(); for n in 1..133 { m.insert(n * 10, n as f32, f, &()); } @@ -722,7 +722,7 @@ mod test { #[test] fn insert_many() { let f = &mut MapForest::::new(); - let mut m = BPlusMap::::new(); + let mut m = Map::::new(); let mm = 4096; let mut x = 0; diff --git a/lib/cretonne/src/bforest/mod.rs b/lib/cretonne/src/bforest/mod.rs index de8ee1092d..8234551b23 100644 --- a/lib/cretonne/src/bforest/mod.rs +++ b/lib/cretonne/src/bforest/mod.rs @@ -22,8 +22,8 @@ mod path; mod pool; mod set; -pub use self::map::{MapForest, BPlusMap, MapCursor}; -pub use self::set::{SetForest, BPlusSet, SetCursor}; +pub use self::map::{MapForest, Map, MapCursor}; +pub use self::set::{SetForest, Set, SetCursor}; use self::node::NodeData; use self::path::Path; @@ -42,7 +42,7 @@ const MAX_PATH: usize = 16; /// /// Keys don't need to implement `Ord`. They are compared using a comparator object which /// provides a context for comparison. -pub trait BPlusComparator +pub trait Comparator where K: Copy, { @@ -63,7 +63,7 @@ where } /// Trivial comparator that doesn't actually provide any context. -impl BPlusComparator for () +impl Comparator for () where K: Copy + Ord, { @@ -87,7 +87,7 @@ trait Forest { type LeafValues: Copy + BorrowMut<[Self::Value]>; /// Type used for key comparisons. - type Comparator: BPlusComparator; + type Comparator: Comparator; /// Splat a single key into a whole array. fn splat_key(key: Self::Key) -> Self::LeafKeys; diff --git a/lib/cretonne/src/bforest/path.rs b/lib/cretonne/src/bforest/path.rs index 2dceb805f4..aa42e2ecc8 100644 --- a/lib/cretonne/src/bforest/path.rs +++ b/lib/cretonne/src/bforest/path.rs @@ -2,7 +2,7 @@ use std::borrow::Borrow; use std::marker::PhantomData; -use super::{Forest, Node, NodeData, NodePool, MAX_PATH, BPlusComparator, slice_insert, slice_shift}; +use super::{Forest, Node, NodeData, NodePool, MAX_PATH, Comparator, slice_insert, slice_shift}; use super::node::Removed; #[cfg(test)] @@ -690,7 +690,7 @@ mod test { struct TC(); - impl BPlusComparator for TC { + impl Comparator for TC { fn cmp(&self, a: i32, b: i32) -> Ordering { a.cmp(&b) } diff --git a/lib/cretonne/src/bforest/pool.rs b/lib/cretonne/src/bforest/pool.rs index 2dfcb35074..778fa082f4 100644 --- a/lib/cretonne/src/bforest/pool.rs +++ b/lib/cretonne/src/bforest/pool.rs @@ -64,7 +64,7 @@ impl NodePool { { use std::borrow::Borrow; use std::cmp::Ordering; - use super::BPlusComparator; + use super::Comparator; use entity::SparseSet; // The root node can't be an inner node with just a single sub-tree. It should have been diff --git a/lib/cretonne/src/bforest/set.rs b/lib/cretonne/src/bforest/set.rs index e768ab64b9..efdf39b4c7 100644 --- a/lib/cretonne/src/bforest/set.rs +++ b/lib/cretonne/src/bforest/set.rs @@ -2,7 +2,7 @@ use packed_option::PackedOption; use std::marker::PhantomData; -use super::{INNER_SIZE, BPlusComparator, Forest, NodePool, Node, NodeData, Path, SetValue}; +use super::{INNER_SIZE, Comparator, Forest, NodePool, Node, NodeData, Path, SetValue}; /// Tag type defining forest types for a set. struct SetTypes(PhantomData<(K, C)>); @@ -10,7 +10,7 @@ struct SetTypes(PhantomData<(K, C)>); impl Forest for SetTypes where K: Copy, - C: BPlusComparator, + C: Comparator, { type Key = K; type Value = SetValue; @@ -27,11 +27,11 @@ where } } -/// Memory pool for a forest of `BPlusSet` instances. +/// Memory pool for a forest of `Set` instances. pub struct SetForest where K: Copy, - C: BPlusComparator, + C: Comparator, { nodes: NodePool>, } @@ -39,7 +39,7 @@ where impl SetForest where K: Copy, - C: BPlusComparator, + C: Comparator, { /// Create a new empty forest. pub fn new() -> SetForest { @@ -48,7 +48,7 @@ where /// Clear all sets in the forest. /// - /// All `BPlusSet` instances belong to this forest are invalidated and should no longer be used. + /// All `Set` instances belong to this forest are invalidated and should no longer be used. pub fn clear(&mut self) { self.nodes.clear(); } @@ -58,23 +58,23 @@ where /// /// This is not a general-purpose replacement for `BTreeSet`. See the [module /// documentation](index.html) for more information about design tradeoffs. -pub struct BPlusSet +pub struct Set where K: Copy, - C: BPlusComparator, + C: Comparator, { root: PackedOption, unused: PhantomData<(K, C)>, } -impl BPlusSet +impl Set where K: Copy, - C: BPlusComparator, + C: Comparator, { /// Make an empty set. - pub fn new() -> BPlusSet { - BPlusSet { + pub fn new() -> Set { + Set { root: None.into(), unused: PhantomData, } @@ -126,14 +126,14 @@ where } } -/// A position in a `BPlusSet` used to navigate and modify the ordered set. +/// A position in a `Set` used to navigate and modify the ordered set. /// /// A cursor always points at an element in the set, or "off the end" which is a position after the /// last element in the set. pub struct SetCursor<'a, K, C> where K: 'a + Copy, - C: 'a + BPlusComparator, + C: 'a + Comparator, { root: &'a mut PackedOption, pool: &'a mut NodePool>, @@ -144,11 +144,11 @@ where impl<'a, K, C> SetCursor<'a, K, C> where K: Copy, - C: BPlusComparator, + C: Comparator, { /// Create a cursor with a default (invalid) location. fn new( - container: &'a mut BPlusSet, + container: &'a mut Set, forest: &'a mut SetForest, comp: &'a C, ) -> SetCursor<'a, K, C> { @@ -250,7 +250,7 @@ where impl<'a, K, C> SetCursor<'a, K, C> where K: Copy + ::std::fmt::Display, - C: BPlusComparator, + C: Comparator, { fn verify(&self) { self.path.verify(self.pool); @@ -281,7 +281,7 @@ mod test { let mut f = SetForest::::new(); f.clear(); - let mut s = BPlusSet::::new(); + let mut s = Set::::new(); assert!(s.is_empty()); assert!(!s.contains(7, &f, &())); @@ -293,7 +293,7 @@ mod test { #[test] fn simple_cursor() { let mut f = SetForest::::new(); - let mut s = BPlusSet::::new(); + let mut s = Set::::new(); let mut c = SetCursor::new(&mut s, &mut f, &()); assert!(c.insert(50)); @@ -335,7 +335,7 @@ mod test { #[test] fn two_level_sparse_tree() { let mut f = SetForest::::new(); - let mut s = BPlusSet::::new(); + let mut s = Set::::new(); let mut c = SetCursor::new(&mut s, &mut f, &()); // Insert enough elements that we get a two-level tree. @@ -381,7 +381,7 @@ mod test { #[test] fn three_level_sparse_tree() { let mut f = SetForest::::new(); - let mut s = BPlusSet::::new(); + let mut s = Set::::new(); let mut c = SetCursor::new(&mut s, &mut f, &()); // Insert enough elements that we get a 3-level tree. @@ -433,9 +433,9 @@ mod test { // Level 4: 512 leafs, up to 7680 elements // // A 3-level tree can hold at most 960 elements. - fn dense4l(f: &mut SetForest) -> BPlusSet { + fn dense4l(f: &mut SetForest) -> Set { f.clear(); - let mut s = BPlusSet::new(); + let mut s = Set::new(); // Insert 400 elements in 7 passes over the range to avoid the half-full leaf node pattern // that comes from sequential insertion. This will generate a normal leaf layer. From 3389eaef296cb04068cb32b1142fb5ef2b984802 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 20 Nov 2017 11:08:25 -0800 Subject: [PATCH 1368/3084] Add a clear() method to bforest::{Set,Map}. This is a lot more efficient that removing entries one by one. --- lib/cretonne/src/bforest/map.rs | 14 ++++++++++++++ lib/cretonne/src/bforest/pool.rs | 14 ++++++++++++++ lib/cretonne/src/bforest/set.rs | 14 ++++++++++++++ 3 files changed, 42 insertions(+) diff --git a/lib/cretonne/src/bforest/map.rs b/lib/cretonne/src/bforest/map.rs index e9709c0b17..45ff290ccd 100644 --- a/lib/cretonne/src/bforest/map.rs +++ b/lib/cretonne/src/bforest/map.rs @@ -118,6 +118,13 @@ where } } + /// Remove all entries. + pub fn clear(&mut self, forest: &mut MapForest) { + if let Some(root) = self.root.take() { + forest.nodes.free_tree(root); + } + } + /// Create a cursor for navigating this map. The cursor is initially positioned off the end of /// the map. pub fn cursor<'a>( @@ -322,6 +329,7 @@ mod test { let mut m = Map::::new(); assert!(m.is_empty()); + m.clear(&mut f); assert_eq!(m.get(7, &f, &()), None); @@ -459,6 +467,9 @@ mod test { m.insert(45, 4.2, f, &()); m.verify(f, &()); assert_eq!(m.get(45, f, &()), Some(4.2)); + + m.clear(f); + assert!(m.is_empty()); } #[test] @@ -579,6 +590,9 @@ mod test { m.insert(805, 4.2, f, &()); m.verify(f, &()); assert_eq!(m.get(805, f, &()), Some(4.2)); + + m.clear(f); + m.verify(f, &()); } // Make a tree with two barely healthy leaf nodes: diff --git a/lib/cretonne/src/bforest/pool.rs b/lib/cretonne/src/bforest/pool.rs index 778fa082f4..37801ee4ab 100644 --- a/lib/cretonne/src/bforest/pool.rs +++ b/lib/cretonne/src/bforest/pool.rs @@ -52,6 +52,20 @@ impl NodePool { self.nodes[node] = NodeData::Free { next: self.freelist }; self.freelist = Some(node); } + + /// Free the entire tree rooted at `node`. + pub fn free_tree(&mut self, node: Node) { + if let NodeData::Inner { size, tree, .. } = self[node] { + // Note that we have to capture `tree` by value to avoid borrow checker trouble. + for i in 0..usize::from(size + 1) { + // Recursively free sub-trees. This recursion can never be deeper than `MAX_PATH`, + // and since most trees have less than a handful of nodes, it is worthwhile to + // avoid the heap allocation for an iterative tree traversal. + self.free_tree(tree[i]); + } + } + self.free_node(node); + } } #[cfg(test)] diff --git a/lib/cretonne/src/bforest/set.rs b/lib/cretonne/src/bforest/set.rs index efdf39b4c7..b716cb2e33 100644 --- a/lib/cretonne/src/bforest/set.rs +++ b/lib/cretonne/src/bforest/set.rs @@ -115,6 +115,13 @@ where } } + /// Remove all entries. + pub fn clear(&mut self, forest: &mut SetForest) { + if let Some(root) = self.root.take() { + forest.nodes.free_tree(root); + } + } + /// Create a cursor for navigating this set. The cursor is initially positioned off the end of /// the set. pub fn cursor<'a>( @@ -283,6 +290,7 @@ mod test { let mut s = Set::::new(); assert!(s.is_empty()); + s.clear(&mut f); assert!(!s.contains(7, &f, &())); let c = SetCursor::new(&mut s, &mut f, &()); @@ -483,4 +491,10 @@ mod test { assert!(c.is_empty()); } + #[test] + fn four_level_clear() { + let mut f = SetForest::::new(); + let mut s = dense4l(&mut f); + s.clear(&mut f); + } } From a2ff2a6836119167217f65ee220bdf45d0e3960b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 20 Nov 2017 12:31:51 -0800 Subject: [PATCH 1369/3084] Add goto_first() methods to SetCursor and MapCursor. --- lib/cretonne/src/bforest/map.rs | 15 ++++++++++++++- lib/cretonne/src/bforest/path.rs | 16 ++++++++++++++++ lib/cretonne/src/bforest/set.rs | 12 ++++++++++-- 3 files changed, 40 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/bforest/map.rs b/lib/cretonne/src/bforest/map.rs index 45ff290ccd..0cc1364c47 100644 --- a/lib/cretonne/src/bforest/map.rs +++ b/lib/cretonne/src/bforest/map.rs @@ -254,6 +254,11 @@ where }) } + /// Move this cursor to the first element. + pub fn goto_first(&mut self) -> Option { + self.root.map(|root| self.path.first(root, self.pool).1) + } + /// Insert `(key, value))` into the map and leave the cursor at the inserted pair. /// /// If the map did not contain `key`, return `None`. @@ -341,6 +346,8 @@ mod test { assert_eq!(c.prev(), None); c.verify(); assert_eq!(c.tpath(), ""); + assert_eq!(c.goto_first(), None); + assert_eq!(c.tpath(), ""); } #[test] @@ -407,7 +414,7 @@ mod test { { let mut c = m.cursor(f, &()); - assert_eq!(c.goto(0), None); + assert_eq!(c.goto_first(), Some(4.0)); assert_eq!(c.key(), Some(40)); assert_eq!(c.value(), Some(4.0)); assert_eq!(c.next(), Some((50, 5.5))); @@ -516,6 +523,12 @@ mod test { assert_eq!(m.tpath(810, f, &()), "node2[7]--node8[0]"); assert_eq!(m.tpath(870, f, &()), "node2[7]--node8[6]"); + { + let mut c = m.cursor(f, &()); + assert_eq!(c.goto_first(), Some(1.1)); + assert_eq!(c.key(), Some(110)); + } + // Front of first leaf. m.insert(0, 4.2, f, &()); m.verify(f, &()); diff --git a/lib/cretonne/src/bforest/path.rs b/lib/cretonne/src/bforest/path.rs index aa42e2ecc8..3393014fee 100644 --- a/lib/cretonne/src/bforest/path.rs +++ b/lib/cretonne/src/bforest/path.rs @@ -87,6 +87,22 @@ impl Path { unreachable!(); } + /// Move path to the first entry of the tree starting at `root` and return it. + pub fn first(&mut self, root: Node, pool: &NodePool) -> (F::Key, F::Value) { + let mut node = root; + for level in 0.. { + self.size = level + 1; + self.node[level] = node; + self.entry[level] = 0; + match &pool[node] { + &NodeData::Inner { tree, .. } => node = tree[0], + &NodeData::Leaf { keys, vals, .. } => return (keys.borrow()[0], vals.borrow()[0]), + &NodeData::Free { .. } => panic!("Free {} reached from {}", node, root), + } + } + unreachable!(); + } + /// Move this path to the next key-value pair and return it. pub fn next(&mut self, pool: &NodePool) -> Option<(F::Key, F::Value)> { match self.leaf_pos() { diff --git a/lib/cretonne/src/bforest/set.rs b/lib/cretonne/src/bforest/set.rs index b716cb2e33..ce809ac648 100644 --- a/lib/cretonne/src/bforest/set.rs +++ b/lib/cretonne/src/bforest/set.rs @@ -216,6 +216,11 @@ where } } + /// Move this cursor to the first element. + pub fn goto_first(&mut self) -> Option { + self.root.map(|root| self.path.first(root, self.pool).0) + } + /// Try to insert `elem` into the set and leave the cursor at the inserted element. /// /// If the set did not contain `elem`, insert it and return true. @@ -293,9 +298,12 @@ mod test { s.clear(&mut f); assert!(!s.contains(7, &f, &())); - let c = SetCursor::new(&mut s, &mut f, &()); + let mut c = SetCursor::new(&mut s, &mut f, &()); c.verify(); assert_eq!(c.elem(), None); + + assert_eq!(c.goto_first(), None); + assert_eq!(c.tpath(), ""); } #[test] @@ -355,7 +363,7 @@ mod test { } assert!(!c.is_empty()); - assert!(c.goto(0)); + assert_eq!(c.goto_first(), Some(0)); assert_eq!(c.tpath(), "node2[0]--node0[0]"); assert_eq!(c.prev(), None); From 8eaf7d3904dff4545ea84e0f9011d9914e9b1a30 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 20 Nov 2017 13:24:19 -0800 Subject: [PATCH 1370/3084] Add iterators for bforest::{Set,Map}. The iter() methods return an iterator that traverses all set elements / map key-value pairs. The iterator doesn't require a mutable container and forest reference, unlike the cursor types. --- lib/cretonne/src/bforest/map.rs | 54 ++++++++++++++++++++++++++++++++- lib/cretonne/src/bforest/mod.rs | 4 +-- lib/cretonne/src/bforest/set.rs | 47 ++++++++++++++++++++++++++++ 3 files changed, 102 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/bforest/map.rs b/lib/cretonne/src/bforest/map.rs index 0cc1364c47..23cac5600d 100644 --- a/lib/cretonne/src/bforest/map.rs +++ b/lib/cretonne/src/bforest/map.rs @@ -134,6 +134,15 @@ where ) -> MapCursor<'a, K, V, C> { MapCursor::new(self, forest, comp) } + + /// Create an iterator traversing this map. The iterator type is `(K, V)`. + pub fn iter<'a>(&'a self, forest: &'a MapForest) -> MapIter<'a, K, V, C> { + MapIter { + root: self.root, + pool: &forest.nodes, + path: Path::default(), + } + } } #[cfg(test)] @@ -296,6 +305,37 @@ where } } +/// An iterator visiting the key-value pairs of a `Map`. +pub struct MapIter<'a, K, V, C> +where + K: 'a + Copy, + V: 'a + Copy, + C: 'a + Comparator, +{ + root: PackedOption, + pool: &'a NodePool>, + path: Path>, +} + +impl<'a, K, V, C> Iterator for MapIter<'a, K, V, C> +where + K: 'a + Copy, + V: 'a + Copy, + C: 'a + Comparator, +{ + type Item = (K, V); + + fn next(&mut self) -> Option { + // We use `self.root` to indicate if we need to go to the first element. Reset to `None` + // once we've returned the first element. This also works for an empty tree since the + // `path.next()` call returns `None` when the path is empty. This also fuses the iterator. + match self.root.take() { + Some(root) => Some(self.path.first(root, self.pool)), + None => self.path.next(self.pool), + } + } +} + #[cfg(test)] impl<'a, K, V, C> MapCursor<'a, K, V, C> where @@ -337,6 +377,7 @@ mod test { m.clear(&mut f); assert_eq!(m.get(7, &f, &()), None); + assert_eq!(m.iter(&f).next(), None); let mut c = m.cursor(&mut f, &()); assert!(c.is_empty()); @@ -367,7 +408,18 @@ mod test { m.verify(f, &()); - // [ 20 40 50 60 80 90 200 ] + assert_eq!( + m.iter(f).collect::>(), + [ + (20, 2.0), + (40, 4.0), + (50, 5.5), + (60, 6.0), + (80, 8.0), + (90, 9.0), + (200, 20.0), + ] + ); assert_eq!(m.get(0, f, &()), None); assert_eq!(m.get(20, f, &()), Some(2.0)); diff --git a/lib/cretonne/src/bforest/mod.rs b/lib/cretonne/src/bforest/mod.rs index 8234551b23..b1d4aa3c28 100644 --- a/lib/cretonne/src/bforest/mod.rs +++ b/lib/cretonne/src/bforest/mod.rs @@ -22,8 +22,8 @@ mod path; mod pool; mod set; -pub use self::map::{MapForest, Map, MapCursor}; -pub use self::set::{SetForest, Set, SetCursor}; +pub use self::map::{MapForest, Map, MapCursor, MapIter}; +pub use self::set::{SetForest, Set, SetCursor, SetIter}; use self::node::NodeData; use self::path::Path; diff --git a/lib/cretonne/src/bforest/set.rs b/lib/cretonne/src/bforest/set.rs index ce809ac648..4956a26147 100644 --- a/lib/cretonne/src/bforest/set.rs +++ b/lib/cretonne/src/bforest/set.rs @@ -131,6 +131,15 @@ where ) -> SetCursor<'a, K, C> { SetCursor::new(self, forest, comp) } + + /// Create an iterator traversing this set. The iterator type is `K`. + pub fn iter<'a>(&'a self, forest: &'a SetForest) -> SetIter<'a, K, C> { + SetIter { + root: self.root, + pool: &forest.nodes, + path: Path::default(), + } + } } /// A position in a `Set` used to navigate and modify the ordered set. @@ -275,6 +284,35 @@ where } } +/// An iterator visiting the elements of a `Set`. +pub struct SetIter<'a, K, C> +where + K: 'a + Copy, + C: 'a + Comparator, +{ + root: PackedOption, + pool: &'a NodePool>, + path: Path>, +} + +impl<'a, K, C> Iterator for SetIter<'a, K, C> +where + K: 'a + Copy, + C: 'a + Comparator, +{ + type Item = K; + + fn next(&mut self) -> Option { + // We use `self.root` to indicate if we need to go to the first element. Reset to `None` + // once we've returned the first element. This also works for an empty tree since the + // `path.next()` call returns `None` when the path is empty. This also fuses the iterator. + match self.root.take() { + Some(root) => Some(self.path.first(root, self.pool).0), + None => self.path.next(self.pool).map(|(k, _)| k), + } + } +} + #[cfg(test)] mod test { use std::mem; @@ -298,6 +336,9 @@ mod test { s.clear(&mut f); assert!(!s.contains(7, &f, &())); + // Iterator for an empty set. + assert_eq!(s.iter(&f).next(), None); + let mut c = SetCursor::new(&mut s, &mut f, &()); c.verify(); assert_eq!(c.elem(), None); @@ -465,6 +506,12 @@ mod test { fn four_level() { let mut f = SetForest::::new(); let mut s = dense4l(&mut f); + + assert_eq!( + s.iter(&f).collect::>()[0..10], + [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] + ); + let mut c = s.cursor(&mut f, &()); c.verify(); From 2d7b54373f0581a894a7c7c6eed641c44d678594 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 20 Nov 2017 14:42:47 -0800 Subject: [PATCH 1371/3084] Implement Clone and Default for bforest::{Set,Map}. The default container is empty. We need a manual implementation of Default because deriving it seems to imply that K and V generic parameter types must also implement Default. Cloning can be used to clone an empty container or for cloning the whole forest. We can derive this trait because we already require Copy for K and V. --- lib/cretonne/src/bforest/map.rs | 16 ++++++++++++++++ lib/cretonne/src/bforest/set.rs | 15 +++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/lib/cretonne/src/bforest/map.rs b/lib/cretonne/src/bforest/map.rs index 23cac5600d..601e9febe6 100644 --- a/lib/cretonne/src/bforest/map.rs +++ b/lib/cretonne/src/bforest/map.rs @@ -61,6 +61,11 @@ where /// /// This is not a general-purpose replacement for `BTreeMap`. See the [module /// documentation](index.html) for more information about design tradeoffs. +/// +/// Maps can be cloned, but that operation should only be used as part of cloning the whole forest +/// they belong to. *Cloning a map does not allocate new memory for the clone*. It creates an alias +/// of the same memory. +#[derive(Clone)] pub struct Map where K: Copy, @@ -145,6 +150,17 @@ where } } +impl Default for Map +where + K: Copy, + V: Copy, + C: Comparator, +{ + fn default() -> Self { + Self::new() + } +} + #[cfg(test)] impl Map where diff --git a/lib/cretonne/src/bforest/set.rs b/lib/cretonne/src/bforest/set.rs index 4956a26147..025efc0225 100644 --- a/lib/cretonne/src/bforest/set.rs +++ b/lib/cretonne/src/bforest/set.rs @@ -58,6 +58,11 @@ where /// /// This is not a general-purpose replacement for `BTreeSet`. See the [module /// documentation](index.html) for more information about design tradeoffs. +/// +/// Sets can be cloned, but that operation should only be used as part of cloning the whole forest +/// they belong to. *Cloning a set does not allocate new memory for the clone*. It creates an alias +/// of the same memory. +#[derive(Clone)] pub struct Set where K: Copy, @@ -142,6 +147,16 @@ where } } +impl Default for Set +where + K: Copy, + C: Comparator, +{ + fn default() -> Self { + Self::new() + } +} + /// A position in a `Set` used to navigate and modify the ordered set. /// /// A cursor always points at an element in the set, or "off the end" which is a position after the From 2e0b931590197e46edf55cc88fcb27f5504000ea Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 21 Nov 2017 10:00:30 -0800 Subject: [PATCH 1372/3084] Provide an fmt::Debug impl for entity references. Instead of deriving a Debug impl: Ebb(45), use the Display version for Debug too: ebb45. This is more readable, and no information is lost. --- lib/cretonne/src/bforest/mod.rs | 2 +- lib/cretonne/src/entity/mod.rs | 6 ++++++ lib/cretonne/src/ir/entities.rs | 26 ++++++++++++++++---------- lib/cretonne/src/loop_analysis.rs | 2 +- lib/cretonne/src/regalloc/virtregs.rs | 2 +- 5 files changed, 25 insertions(+), 13 deletions(-) diff --git a/lib/cretonne/src/bforest/mod.rs b/lib/cretonne/src/bforest/mod.rs index b1d4aa3c28..ff1b94cc0f 100644 --- a/lib/cretonne/src/bforest/mod.rs +++ b/lib/cretonne/src/bforest/mod.rs @@ -97,7 +97,7 @@ trait Forest { } /// A reference to a B+-tree node. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[derive(Clone, Copy, PartialEq, Eq)] struct Node(u32); entity_impl!(Node, "node"); diff --git a/lib/cretonne/src/entity/mod.rs b/lib/cretonne/src/entity/mod.rs index 4fc40ea56e..837bb5f107 100644 --- a/lib/cretonne/src/entity/mod.rs +++ b/lib/cretonne/src/entity/mod.rs @@ -87,5 +87,11 @@ macro_rules! entity_impl { write!(f, "{}{}", $display_prefix, self.0) } } + + impl ::std::fmt::Debug for $entity { + fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result { + (self as &::std::fmt::Display).fmt(f) + } + } } } diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 4060e6ff37..4a4a40df42 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -23,7 +23,7 @@ use std::fmt; use std::u32; /// An opaque reference to an extended basic block in a function. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] +#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] pub struct Ebb(u32); entity_impl!(Ebb, "ebb"); @@ -37,7 +37,7 @@ impl Ebb { } /// An opaque reference to an SSA value. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] +#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] pub struct Value(u32); entity_impl!(Value, "v"); @@ -56,17 +56,17 @@ impl Value { } /// An opaque reference to an instruction in a function. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] +#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] pub struct Inst(u32); entity_impl!(Inst, "inst"); /// An opaque reference to a stack slot. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash)] pub struct StackSlot(u32); entity_impl!(StackSlot, "ss"); /// An opaque reference to a global variable. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash)] pub struct GlobalVar(u32); entity_impl!(GlobalVar, "gv"); @@ -84,27 +84,27 @@ impl GlobalVar { } /// An opaque reference to a jump table. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash)] pub struct JumpTable(u32); entity_impl!(JumpTable, "jt"); /// A reference to an external function. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash)] pub struct FuncRef(u32); entity_impl!(FuncRef, "fn"); /// A reference to a function signature. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash)] pub struct SigRef(u32); entity_impl!(SigRef, "sig"); /// A reference to a heap. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash)] pub struct Heap(u32); entity_impl!(Heap, "heap"); /// A reference to any of the entities defined in this module. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash)] pub enum AnyEntity { /// The whole function. Function, @@ -145,6 +145,12 @@ impl fmt::Display for AnyEntity { } } +impl fmt::Debug for AnyEntity { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + (self as &fmt::Display).fmt(f) + } +} + impl From for AnyEntity { fn from(r: Ebb) -> AnyEntity { AnyEntity::Ebb(r) diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index e6390aa100..ad3239f5e5 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -9,7 +9,7 @@ use ir::{Function, Ebb, Layout}; use packed_option::PackedOption; /// A opaque reference to a code loop. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Hash)] pub struct Loop(u32); entity_impl!(Loop, "loop"); diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index c2cb6c8ffe..852f1e0af0 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -18,7 +18,7 @@ use packed_option::PackedOption; use ref_slice::ref_slice; /// A virtual register reference. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)] +#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] pub struct VirtReg(u32); entity_impl!(VirtReg, "vreg"); From cf45afa1e77eb0e192391d01c72278edc33b0d1f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 21 Nov 2017 09:45:52 -0800 Subject: [PATCH 1373/3084] Avoid the CFG get_successors() when computing a post-order. The control flow graph does not guarantee any particular ordering for its successor lists, and the post-order we are computing for building the dominator tree needs to be "split-invariant". See #146 for details. - Discover EBB successors directly from the EBB instruction sequence to guarantee that the post-order we compute is canonical/split-invariant. - Use an alternative graph DFS algorithm which doesn't require indexing into a slice of successors. This changes cfg_postorder in some cases because the edge pruning when converting the (DAG) CFG to a tree for the DFT is different. --- cranelift/tests/cfg_traversal.rs | 76 ++++++++++++++- lib/cretonne/src/dominator_tree.rs | 149 +++++++++++++++++++++++++---- lib/cretonne/src/loop_analysis.rs | 12 +-- 3 files changed, 206 insertions(+), 31 deletions(-) diff --git a/cranelift/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs index 73b4b9fad4..6709da2e9a 100644 --- a/cranelift/tests/cfg_traversal.rs +++ b/cranelift/tests/cfg_traversal.rs @@ -26,6 +26,30 @@ fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) #[test] fn simple_traversal() { + // Fall-through-first, prune-at-source DFT: + // + // ebb0 { + // ebb0:brz v0, ebb1 { + // ebb0:jump ebb2 { + // ebb2 { + // ebb2:brz v2, ebb2 - + // ebb2:brz v3, ebb1 - + // ebb2:brz v4, ebb4 { + // ebb2: jump ebb5 { + // ebb5 {} + // } + // ebb4 {} + // } + // } ebb2 + // } + // ebb1 { + // ebb1:jump ebb3 { + // ebb3 {} + // } + // } ebb1 + // } + // } ebb0 + test_reverse_postorder_traversal( " function %test(i32) native { @@ -51,12 +75,28 @@ fn simple_traversal() { trap user0 } ", - vec![0, 2, 5, 4, 1, 3], + vec![0, 1, 3, 2, 4, 5], ); } #[test] fn loops_one() { + // Fall-through-first, prune-at-source DFT: + // ebb0 { + // ebb0:jump ebb1 { + // ebb1 { + // ebb1:brnz v0, ebb3 { + // ebb1:jump ebb2 { + // ebb2 { + // ebb2:jump ebb3 - + // } ebb2 + // } + // ebb3 {} + // } + // } ebb1 + // } + // } ebb0 + test_reverse_postorder_traversal( " function %test(i32) native { @@ -71,12 +111,40 @@ fn loops_one() { return } ", - vec![0, 1, 2, 3], + vec![0, 1, 3, 2], ); } #[test] fn loops_two() { + // Fall-through-first, prune-at-source DFT: + // ebb0 { + // ebb0:brz v0, ebb1 { + // ebb0:jump ebb2 { + // ebb2 { + // ebb2:brz v0, ebb4 { + // ebb2:jump ebb5 { + // ebb5 { + // brz v0, ebb4 - + // } ebb5 + // } + // ebb4 { + // ebb4:brz v0, ebb3 { + // ebb4:jump ebb5 - + // ebb3 { + // ebb3:jump ebb4 - + // } ebb3 + // } + // } ebb4 + // } + // } ebb2 + // } + // ebb1 { + // ebb1:jump ebb3 - + // } ebb1 + // } + // } ebb0 + test_reverse_postorder_traversal( " function %test(i32) native { @@ -98,7 +166,7 @@ fn loops_two() { return } ", - vec![0, 2, 1, 3, 4, 5], + vec![0, 1, 2, 4, 3, 5], ); } @@ -130,7 +198,7 @@ fn loops_three() { return } ", - vec![0, 2, 1, 3, 4, 6, 7, 5], + vec![0, 1, 2, 4, 3, 6, 7, 5], ); } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 2c2fd1b48c..2507fd25e4 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -3,6 +3,7 @@ use entity::EntityMap; use flowgraph::{ControlFlowGraph, BasicBlock}; use ir::{Ebb, Inst, Function, Layout, ProgramOrder, ExpandedProgramPoint}; +use ir::instructions::BranchInfo; use packed_option::PackedOption; use std::cmp::Ordering; @@ -11,6 +12,10 @@ use std::cmp::Ordering; // room for modifications of the dominator tree. const STRIDE: u32 = 4; +// Special RPO numbers used during `compute_postorder`. +const DONE: u32 = 1; +const SEEN: u32 = 2; + // Dominator tree node. We keep one of these per EBB. #[derive(Clone, Default)] struct DomNode { @@ -36,7 +41,7 @@ pub struct DominatorTree { postorder: Vec, // Scratch memory used by `compute_postorder()`. - stack: Vec<(Ebb, usize)>, + stack: Vec, valid: bool, } @@ -223,7 +228,7 @@ impl DominatorTree { /// Reset and compute a CFG post-order and dominator tree. pub fn compute(&mut self, func: &Function, cfg: &ControlFlowGraph) { debug_assert!(cfg.is_valid()); - self.compute_postorder(func, cfg); + self.compute_postorder(func); self.compute_domtree(func, cfg); self.valid = true; } @@ -246,36 +251,108 @@ impl DominatorTree { self.valid } - /// Reset all internal data structures and compute a post-order for `cfg`. + /// Reset all internal data structures and compute a post-order of the control flow graph. /// /// This leaves `rpo_number == 1` for all reachable EBBs, 0 for unreachable ones. - fn compute_postorder(&mut self, func: &Function, cfg: &ControlFlowGraph) { + fn compute_postorder(&mut self, func: &Function) { self.clear(); self.nodes.resize(func.dfg.num_ebbs()); + // This algorithm is a depth first traversal (DFT) of the control flow graph, computing a + // post-order of the EBBs that are reachable form the entry block. A DFT post-order is not + // unique. The specific order we get is controlled by two factors: + // + // 1. The order each node's children are visited, and + // 2. The method used for pruning graph edges to get a tree. + // + // There are two ways of viewing the CFG as a graph: + // + // 1. Each EBB is a node, with outgoing edges for all the branches in the EBB> + // 2. Each basic block is a node, with outgoing edges for the single branch at the end of + // the BB. (An EBB is a linear sequence of basic blocks). + // + // The first graph is a contraction of the second one. We want to compute an EBB post-order + // that is compatible both graph interpretations. That is, if you compute a BB post-order + // and then remove those BBs that do not correspond to EBB headers, you get a post-order of + // the EBB graph. + // + // Node child order: + // + // In the BB graph, we always go down the fall-through path first and follow the branch + // destination second. + // + // In the EBB graph, this is equivalent to visiting EBB successors in a bottom-up + // order, starting from the destination of the EBB's terminating jump, ending at the + // destination of the first branch in the EBB. + // + // Edge pruning: + // + // In the BB graph, we keep an edge to an EBB the first time we visit the *source* side + // of the edge. Any subsequent edges to the same EBB are pruned. + // + // The equivalent tree is reached in the EBB graph by keeping the first edge to an EBB + // in a top-down traversal of the successors. (And then visiting edges in a bottom-up + // order). + // + // This pruning method makes it possible to compute the DFT without storing lots of + // information about the progress through an EBB. + // During this algorithm only, use `rpo_number` to hold the following state: // - // 0: EBB is not yet on the stack. - // 1: EBB is on the stack or in postorder. - const SEEN: u32 = 1; + // 0: EBB has not yet been reached in the pre-order. + // SEEN: EBB has been pushed on the stack but successors not yet pushed. + // DONE: Successors pushed. match func.layout.entry_block() { Some(ebb) => { - self.stack.push((ebb, 0)); + self.stack.push(ebb); self.nodes[ebb].rpo_number = SEEN; } None => return, } - while let Some((ebb, succ_index)) = self.stack.pop() { - if let Some(&succ) = cfg.get_successors(ebb).get(succ_index) { - self.stack.push((ebb, succ_index + 1)); - if self.nodes[succ].rpo_number == 0 { - self.stack.push((succ, 0)); - self.nodes[succ].rpo_number = SEEN; + while let Some(ebb) = self.stack.pop() { + match self.nodes[ebb].rpo_number { + SEEN => { + // This is the first time we pop the EBB, so we need to scan its successors and + // then revisit it. + self.nodes[ebb].rpo_number = DONE; + self.stack.push(ebb); + self.push_successors(func, ebb); } - } else { - self.postorder.push(ebb); + DONE => { + // This is the second time we pop the EBB, so all successors have been + // processed. + self.postorder.push(ebb); + } + _ => unreachable!(), + } + } + } + + /// Push `ebb` successors onto `self.stack`, filtering out those that have already been seen. + /// + /// The successors are pushed in program order which is important to get a split-invariant + /// post-order. Split-invariant means that if an EBB is split in two, we get the same + /// post-order except for the insertion of the new EBB header at the split point. + fn push_successors(&mut self, func: &Function, ebb: Ebb) { + for inst in func.layout.ebb_insts(ebb) { + match func.dfg[inst].analyze_branch(&func.dfg.value_lists) { + BranchInfo::SingleDest(succ, _) => { + if self.nodes[succ].rpo_number == 0 { + self.nodes[succ].rpo_number = SEEN; + self.stack.push(succ); + } + } + BranchInfo::Table(jt) => { + for (_, succ) in func.jump_tables[jt].entries() { + if self.nodes[succ].rpo_number == 0 { + self.nodes[succ].rpo_number = SEEN; + self.stack.push(succ); + } + } + } + BranchInfo::NotABranch => {} } } } @@ -458,6 +535,18 @@ mod test { let cfg = ControlFlowGraph::with_function(cur.func); let dt = DominatorTree::with_function(cur.func, &cfg); + + // Fall-through-first, prune-at-source DFT: + // + // ebb0 { + // brnz ebb2 { + // trap + // ebb2 { + // return + // } ebb2 + // } ebb0 + assert_eq!(dt.cfg_postorder(), &[ebb2, ebb0]); + let v2_def = cur.func.dfg.value_def(v2).unwrap_inst(); assert!(!dt.dominates(v2_def, ebb0, &cur.func.layout)); assert!(!dt.dominates(ebb0, v2_def, &cur.func.layout)); @@ -466,11 +555,11 @@ mod test { #[test] fn non_zero_entry_block() { let mut func = Function::new(); - let ebb3 = func.dfg.make_ebb(); - let cond = func.dfg.append_ebb_param(ebb3, I32); + let ebb0 = func.dfg.make_ebb(); let ebb1 = func.dfg.make_ebb(); let ebb2 = func.dfg.make_ebb(); - let ebb0 = func.dfg.make_ebb(); + let ebb3 = func.dfg.make_ebb(); + let cond = func.dfg.append_ebb_param(ebb3, I32); let mut cur = FuncCursor::new(&mut func); @@ -489,6 +578,26 @@ mod test { let cfg = ControlFlowGraph::with_function(cur.func); let dt = DominatorTree::with_function(cur.func, &cfg); + // Fall-through-first, prune-at-source DFT: + // + // ebb3 { + // ebb3:jump ebb1 { + // ebb1 { + // ebb1:brnz ebb0 { + // ebb1:jump ebb2 { + // ebb2 { + // ebb2:jump ebb0 (seen) + // } ebb2 + // } ebb1:jump ebb2 + // ebb0 { + // } ebb0 + // } ebb1:brnz ebb0 + // } ebb1 + // } ebb3:jump ebb1 + // } ebb3 + + assert_eq!(dt.cfg_postorder(), &[ebb2, ebb0, ebb1, ebb3]); + assert_eq!(cur.func.layout.entry_block().unwrap(), ebb3); assert_eq!(dt.idom(ebb3), None); assert_eq!(dt.idom(ebb1).unwrap(), jmp_ebb3_ebb1); @@ -509,8 +618,6 @@ mod test { dt.rpo_cmp(jmp_ebb3_ebb1, jmp_ebb1_ebb2, &cur.func.layout), Ordering::Less ); - - assert_eq!(dt.cfg_postorder(), &[ebb0, ebb2, ebb1, ebb3]); } #[test] diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index ad3239f5e5..b44bd77427 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -326,16 +326,16 @@ mod test { let loops = loop_analysis.loops().collect::>(); assert_eq!(loops.len(), 3); assert_eq!(loop_analysis.loop_header(loops[0]), ebb0); - assert_eq!(loop_analysis.loop_header(loops[1]), ebb3); - assert_eq!(loop_analysis.loop_header(loops[2]), ebb1); + assert_eq!(loop_analysis.loop_header(loops[1]), ebb1); + assert_eq!(loop_analysis.loop_header(loops[2]), ebb3); assert_eq!(loop_analysis.loop_parent(loops[1]), Some(loops[0])); assert_eq!(loop_analysis.loop_parent(loops[2]), Some(loops[0])); assert_eq!(loop_analysis.loop_parent(loops[0]), None); assert_eq!(loop_analysis.is_in_loop(ebb0, loops[0]), true); - assert_eq!(loop_analysis.is_in_loop(ebb3, loops[1]), true); - assert_eq!(loop_analysis.is_in_loop(ebb4, loops[1]), true); - assert_eq!(loop_analysis.is_in_loop(ebb1, loops[2]), true); - assert_eq!(loop_analysis.is_in_loop(ebb2, loops[2]), true); + assert_eq!(loop_analysis.is_in_loop(ebb1, loops[1]), true); + assert_eq!(loop_analysis.is_in_loop(ebb2, loops[1]), true); + assert_eq!(loop_analysis.is_in_loop(ebb3, loops[2]), true); + assert_eq!(loop_analysis.is_in_loop(ebb4, loops[2]), true); assert_eq!(loop_analysis.is_in_loop(ebb5, loops[0]), true); } } From b5d4fb66d906d67155f7f55538a130baa2d2b3cb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 20 Nov 2017 14:02:16 -0800 Subject: [PATCH 1374/3084] Use a bforest::Set to represent CFG successor sets. This has two advantages over the previous Vec: - Duplicates are removed. - Clearing the control flow graph is constant time. The set of EBB successors is simply ordered by EBB number. --- lib/cretonne/src/flowgraph.rs | 72 ++++++++++++++++---------------- lib/cretonne/src/licm.rs | 6 +-- lib/cretonne/src/verifier/mod.rs | 4 +- 3 files changed, 40 insertions(+), 42 deletions(-) diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index 1f51ef0cf5..4b411ff2bd 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -23,6 +23,7 @@ //! Here `Ebb1` and `Ebb2` would each have a single predecessor denoted as `(Ebb0, brz)` //! and `(Ebb0, jmp Ebb2)` respectively. +use bforest; use ir::{Function, Inst, Ebb}; use ir::instructions::BranchInfo; use entity::EntityMap; @@ -32,10 +33,12 @@ use std::mem; pub type BasicBlock = (Ebb, Inst); /// A container for the successors and predecessors of some Ebb. -#[derive(Debug, Clone, Default)] +#[derive(Clone, Default)] pub struct CFGNode { - /// EBBs that are the targets of branches and jumps in this EBB. - pub successors: Vec, + /// Set of EBBs that are the targets of branches and jumps in this EBB. + /// The set is ordered by EBB number, indicated by the `()` comparator type. + pub successors: bforest::Set, + /// Basic blocks that can branch or jump to this EBB. pub predecessors: Vec, } @@ -43,9 +46,9 @@ pub struct CFGNode { /// The Control Flow Graph maintains a mapping of ebbs to their predecessors /// and successors where predecessors are basic blocks and successors are /// extended basic blocks. -#[derive(Debug)] pub struct ControlFlowGraph { data: EntityMap, + succ_forest: bforest::SetForest, valid: bool, } @@ -55,12 +58,14 @@ impl ControlFlowGraph { Self { data: EntityMap::new(), valid: false, + succ_forest: bforest::SetForest::new(), } } /// Clear all data structures in this control flow graph. pub fn clear(&mut self) { self.data.clear(); + self.succ_forest.clear(); self.valid = false; } @@ -75,7 +80,7 @@ impl ControlFlowGraph { /// /// This will clear and overwrite any information already stored in this data structure. pub fn compute(&mut self, func: &Function) { - self.data.clear(); + self.clear(); self.data.resize(func.dfg.num_ebbs()); for ebb in &func.layout { @@ -105,12 +110,11 @@ impl ControlFlowGraph { // Temporarily take ownership because we need mutable access to self.data inside the loop. // Unfortunately borrowck cannot see that our mut accesses to predecessors don't alias // our iteration over successors. - let mut successors = mem::replace(&mut self.data[ebb].successors, Vec::new()); - for suc in successors.iter().cloned() { - self.data[suc].predecessors.retain(|&(e, _)| e != ebb); + let mut successors = mem::replace(&mut self.data[ebb].successors, Default::default()); + for succ in successors.iter(&self.succ_forest) { + self.data[succ].predecessors.retain(|&(e, _)| e != ebb); } - successors.clear(); - self.data[ebb].successors = successors; + successors.clear(&mut self.succ_forest); } /// Recompute the control flow graph of `ebb`. @@ -126,7 +130,11 @@ impl ControlFlowGraph { } fn add_edge(&mut self, from: BasicBlock, to: Ebb) { - self.data[from.0].successors.push(to); + self.data[from.0].successors.insert( + to, + &mut self.succ_forest, + &(), + ); self.data[to].predecessors.push(from); } @@ -136,10 +144,10 @@ impl ControlFlowGraph { &self.data[ebb].predecessors } - /// Get the CFG successors to `ebb`. - pub fn get_successors(&self, ebb: Ebb) -> &[Ebb] { + /// Get an iterator over the CFG successors to `ebb`. + pub fn succ_iter(&self, ebb: Ebb) -> bforest::SetIter { debug_assert!(self.is_valid()); - &self.data[ebb].successors + self.data[ebb].successors.iter(&self.succ_forest) } /// Check if the CFG is in a valid state. @@ -180,7 +188,7 @@ mod tests { for ebb in func.layout.ebbs() { assert_eq!(ebb, fun_ebbs.next().unwrap()); assert_eq!(cfg.get_predecessors(ebb).len(), 0); - assert_eq!(cfg.get_successors(ebb).len(), 0); + assert_eq!(cfg.succ_iter(ebb).count(), 0); } } @@ -218,9 +226,9 @@ mod tests { let ebb1_predecessors = cfg.get_predecessors(ebb1); let ebb2_predecessors = cfg.get_predecessors(ebb2); - let ebb0_successors = cfg.get_successors(ebb0); - let ebb1_successors = cfg.get_successors(ebb1); - let ebb2_successors = cfg.get_successors(ebb2); + let ebb0_successors = cfg.succ_iter(ebb0); + let ebb1_successors = cfg.succ_iter(ebb1); + let ebb2_successors = cfg.succ_iter(ebb2); assert_eq!(ebb0_predecessors.len(), 0); assert_eq!(ebb1_predecessors.len(), 2); @@ -231,14 +239,9 @@ mod tests { assert_eq!(ebb2_predecessors.contains(&(ebb0, br_ebb0_ebb2)), true); assert_eq!(ebb2_predecessors.contains(&(ebb1, jmp_ebb1_ebb2)), true); - assert_eq!(ebb0_successors.len(), 2); - assert_eq!(ebb1_successors.len(), 2); - assert_eq!(ebb2_successors.len(), 0); - - assert_eq!(ebb0_successors.contains(&ebb1), true); - assert_eq!(ebb0_successors.contains(&ebb2), true); - assert_eq!(ebb1_successors.contains(&ebb1), true); - assert_eq!(ebb1_successors.contains(&ebb2), true); + assert_eq!(ebb0_successors.collect::>(), [ebb1, ebb2]); + assert_eq!(ebb1_successors.collect::>(), [ebb1, ebb2]); + assert_eq!(ebb2_successors.collect::>(), []); } // Change some instructions and recompute ebb0 @@ -252,9 +255,9 @@ mod tests { let ebb1_predecessors = cfg.get_predecessors(ebb1); let ebb2_predecessors = cfg.get_predecessors(ebb2); - let ebb0_successors = cfg.get_successors(ebb0); - let ebb1_successors = cfg.get_successors(ebb1); - let ebb2_successors = cfg.get_successors(ebb2); + let ebb0_successors = cfg.succ_iter(ebb0); + let ebb1_successors = cfg.succ_iter(ebb1); + let ebb2_successors = cfg.succ_iter(ebb2); assert_eq!(ebb0_predecessors.len(), 0); assert_eq!(ebb1_predecessors.len(), 2); @@ -265,14 +268,9 @@ mod tests { assert_eq!(ebb2_predecessors.contains(&(ebb0, br_ebb0_ebb2)), false); assert_eq!(ebb2_predecessors.contains(&(ebb1, jmp_ebb1_ebb2)), true); - assert_eq!(ebb0_successors.len(), 1); - assert_eq!(ebb1_successors.len(), 2); - assert_eq!(ebb2_successors.len(), 0); - - assert_eq!(ebb0_successors.contains(&ebb1), true); - assert_eq!(ebb0_successors.contains(&ebb2), false); - assert_eq!(ebb1_successors.contains(&ebb1), true); - assert_eq!(ebb1_successors.contains(&ebb2), true); + assert_eq!(ebb0_successors.collect::>(), [ebb1]); + assert_eq!(ebb1_successors.collect::>(), [ebb1, ebb2]); + assert_eq!(ebb2_successors.collect::>(), []); } } } diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 2fcd8241c2..7ca08620b6 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -184,9 +184,9 @@ fn postorder_ebbs_loop(loop_analysis: &LoopAnalysis, cfg: &ControlFlowGraph, lp: grey.insert(node); stack.push(node); // Get any children we've never seen before. - for child in cfg.get_successors(node) { - if loop_analysis.is_in_loop(*child, lp) && !grey.contains(child) { - stack.push(*child); + for child in cfg.succ_iter(node) { + if loop_analysis.is_in_loop(child, lp) && !grey.contains(&child) { + stack.push(child); } } } else if !black.contains(&node) { diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index f82043fce3..3df4975026 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -916,8 +916,8 @@ impl<'a> Verifier<'a> { let mut got_preds = BTreeSet::::new(); for ebb in self.func.layout.ebbs() { - expected_succs.extend(self.expected_cfg.get_successors(ebb)); - got_succs.extend(cfg.get_successors(ebb)); + expected_succs.extend(self.expected_cfg.succ_iter(ebb)); + got_succs.extend(cfg.succ_iter(ebb)); let missing_succs: Vec = expected_succs.difference(&got_succs).cloned().collect(); if !missing_succs.is_empty() { From 7956084121622e44c64262bb8c8ececfbd7848b9 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Tue, 21 Nov 2017 17:08:31 -0800 Subject: [PATCH 1375/3084] Treat VmContext as positional when using Native CallConv (#195) * Treat VMContext as standard positional argument when using Native CallConv. This requires threading the CallConv through legalize_args and into ArgAssigner. * Stash CallConv in the intel-specific Args struct, for use ArgAssigner. --- lib/cretonne/src/isa/intel/abi.rs | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 3cd3dce931..e8be53e848 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -6,7 +6,7 @@ use regalloc::AllocatableSet; use settings as shared_settings; use super::registers::{GPR, FPR, RU}; use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; -use ir::{AbiParam, ArgumentPurpose, ArgumentLoc, ArgumentExtension}; +use ir::{AbiParam, ArgumentPurpose, ArgumentLoc, ArgumentExtension, CallConv}; use std::i32; /// Argument registers for x86-64 @@ -24,10 +24,11 @@ struct Args { fpr_limit: usize, fpr_used: usize, offset: u32, + call_conv: CallConv, } impl Args { - fn new(bits: u16, gpr: &'static [RU], fpr_limit: usize) -> Args { + fn new(bits: u16, gpr: &'static [RU], fpr_limit: usize, call_conv: CallConv) -> Args { Args { pointer_bytes: u32::from(bits) / 8, pointer_bits: bits, @@ -37,6 +38,7 @@ impl Args { fpr_limit, fpr_used: 0, offset: 0, + call_conv: call_conv, } } } @@ -66,8 +68,7 @@ impl ArgAssigner for Args { } // Handle special-purpose arguments. - // TODO: The registers below are for `spiderwasm`. Should we check the calling convention? - if ty.is_int() { + if ty.is_int() && self.call_conv == CallConv::SpiderWASM { match arg.purpose { // This is SpiderMonkey's `WasmTlsReg`. ArgumentPurpose::VMContext => { @@ -112,15 +113,15 @@ pub fn legalize_signature(sig: &mut ir::Signature, flags: &shared_settings::Flag if flags.is_64bit() { bits = 64; - args = Args::new(bits, &ARG_GPRS, 8); + args = Args::new(bits, &ARG_GPRS, 8, sig.call_conv); } else { bits = 32; - args = Args::new(bits, &[], 0); + args = Args::new(bits, &[], 0, sig.call_conv); } legalize_args(&mut sig.params, &mut args); - let mut rets = Args::new(bits, &RET_GPRS, 2); + let mut rets = Args::new(bits, &RET_GPRS, 2, sig.call_conv); legalize_args(&mut sig.returns, &mut rets); } From 92f378de76a3d00de703547ce515bf3e5d2a1e38 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 22 Nov 2017 09:13:04 -0800 Subject: [PATCH 1376/3084] Expose CFG predecessors only as an iterator. Define two public iterator types in the flowgraph module, PredIter and SuccIter, which are by-value iterators over an EBB's predecessors and successors respectively. Provide matching pred_iter() and succ_iter() methods for inspecting the CFG. Remove the get_predecessors() method which returned a slice. Update the uses of get_predecessors(), none of which depended on it being a slice. This abstraction makes it possible to change the internal representation of the CFG. --- cranelift/src/print_cfg.rs | 2 +- lib/cretonne/src/dominator_tree.rs | 8 ++-- lib/cretonne/src/flowgraph.rs | 52 ++++++++++++++++--------- lib/cretonne/src/legalizer/split.rs | 2 +- lib/cretonne/src/licm.rs | 4 +- lib/cretonne/src/loop_analysis.rs | 6 +-- lib/cretonne/src/regalloc/coalescing.rs | 26 +++++-------- lib/cretonne/src/regalloc/liveness.rs | 4 +- lib/cretonne/src/verifier/cssa.rs | 2 +- lib/cretonne/src/verifier/flags.rs | 2 +- lib/cretonne/src/verifier/liveness.rs | 2 +- lib/cretonne/src/verifier/mod.rs | 5 +-- 12 files changed, 61 insertions(+), 54 deletions(-) diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index bc3fc3dd9b..8d2b06c8f1 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -76,7 +76,7 @@ impl<'a> CFGPrinter<'a> { fn cfg_connections(&self, w: &mut Write) -> Result { for ebb in &self.func.layout { - for &(parent, inst) in self.cfg.get_predecessors(ebb) { + for (parent, inst) in self.cfg.pred_iter(ebb) { writeln!(w, " {}:{} -> {}", parent, inst, ebb)?; } } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 2507fd25e4..d9f2362e3b 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -415,11 +415,9 @@ impl DominatorTree { // Get an iterator with just the reachable, already visited predecessors to `ebb`. // Note that during the first pass, `rpo_number` is 1 for reachable blocks that haven't // been visited yet, 0 for unreachable blocks. - let mut reachable_preds = cfg.get_predecessors(ebb).iter().cloned().filter( - |&(pred, _)| { - self.nodes[pred].rpo_number > 1 - }, - ); + let mut reachable_preds = cfg.pred_iter(ebb).filter(|&(pred, _)| { + self.nodes[pred].rpo_number > 1 + }); // The RPO must visit at least one predecessor before this node. let mut idom = reachable_preds.next().expect( diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index 4b411ff2bd..a2c27828fb 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -28,6 +28,7 @@ use ir::{Function, Inst, Ebb}; use ir::instructions::BranchInfo; use entity::EntityMap; use std::mem; +use std::slice; /// A basic block denoted by its enclosing Ebb and last instruction. pub type BasicBlock = (Ebb, Inst); @@ -138,14 +139,13 @@ impl ControlFlowGraph { self.data[to].predecessors.push(from); } - /// Get the CFG predecessor basic blocks to `ebb`. - pub fn get_predecessors(&self, ebb: Ebb) -> &[BasicBlock] { - debug_assert!(self.is_valid()); - &self.data[ebb].predecessors + /// Get an iterator over the CFG predecessors to `ebb`. + pub fn pred_iter(&self, ebb: Ebb) -> PredIter { + PredIter(self.data[ebb].predecessors.iter()) } /// Get an iterator over the CFG successors to `ebb`. - pub fn succ_iter(&self, ebb: Ebb) -> bforest::SetIter { + pub fn succ_iter(&self, ebb: Ebb) -> SuccIter { debug_assert!(self.is_valid()); self.data[ebb].successors.iter(&self.succ_forest) } @@ -160,6 +160,22 @@ impl ControlFlowGraph { } } +/// An iterator over EBB predecessors. The iterator type is `BasicBlock`. +/// +/// Each predecessor is an instruction that branches to the EBB. +pub struct PredIter<'a>(slice::Iter<'a, BasicBlock>); + +impl<'a> Iterator for PredIter<'a> { + type Item = BasicBlock; + + fn next(&mut self) -> Option { + self.0.next().cloned() + } +} + +/// An iterator over EBB successors. The iterator type is `Ebb`. +pub type SuccIter<'a> = bforest::SetIter<'a, Ebb, ()>; + #[cfg(test)] mod tests { use super::*; @@ -187,7 +203,7 @@ mod tests { let mut fun_ebbs = func.layout.ebbs(); for ebb in func.layout.ebbs() { assert_eq!(ebb, fun_ebbs.next().unwrap()); - assert_eq!(cfg.get_predecessors(ebb).len(), 0); + assert_eq!(cfg.pred_iter(ebb).count(), 0); assert_eq!(cfg.succ_iter(ebb).count(), 0); } } @@ -222,13 +238,13 @@ mod tests { let mut cfg = ControlFlowGraph::with_function(&func); { - let ebb0_predecessors = cfg.get_predecessors(ebb0); - let ebb1_predecessors = cfg.get_predecessors(ebb1); - let ebb2_predecessors = cfg.get_predecessors(ebb2); + let ebb0_predecessors = cfg.pred_iter(ebb0).collect::>(); + let ebb1_predecessors = cfg.pred_iter(ebb1).collect::>(); + let ebb2_predecessors = cfg.pred_iter(ebb2).collect::>(); - let ebb0_successors = cfg.succ_iter(ebb0); - let ebb1_successors = cfg.succ_iter(ebb1); - let ebb2_successors = cfg.succ_iter(ebb2); + let ebb0_successors = cfg.succ_iter(ebb0).collect::>(); + let ebb1_successors = cfg.succ_iter(ebb1).collect::>(); + let ebb2_successors = cfg.succ_iter(ebb2).collect::>(); assert_eq!(ebb0_predecessors.len(), 0); assert_eq!(ebb1_predecessors.len(), 2); @@ -239,9 +255,9 @@ mod tests { assert_eq!(ebb2_predecessors.contains(&(ebb0, br_ebb0_ebb2)), true); assert_eq!(ebb2_predecessors.contains(&(ebb1, jmp_ebb1_ebb2)), true); - assert_eq!(ebb0_successors.collect::>(), [ebb1, ebb2]); - assert_eq!(ebb1_successors.collect::>(), [ebb1, ebb2]); - assert_eq!(ebb2_successors.collect::>(), []); + assert_eq!(ebb0_successors, [ebb1, ebb2]); + assert_eq!(ebb1_successors, [ebb1, ebb2]); + assert_eq!(ebb2_successors, []); } // Change some instructions and recompute ebb0 @@ -251,9 +267,9 @@ mod tests { let br_ebb0_ebb1 = br_ebb0_ebb2; { - let ebb0_predecessors = cfg.get_predecessors(ebb0); - let ebb1_predecessors = cfg.get_predecessors(ebb1); - let ebb2_predecessors = cfg.get_predecessors(ebb2); + let ebb0_predecessors = cfg.pred_iter(ebb0).collect::>(); + let ebb1_predecessors = cfg.pred_iter(ebb1).collect::>(); + let ebb2_predecessors = cfg.pred_iter(ebb2).collect::>(); let ebb0_successors = cfg.succ_iter(ebb0); let ebb1_successors = cfg.succ_iter(ebb1); diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index eff3b9f542..0c73e7c2d4 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -125,7 +125,7 @@ fn split_any( // We have split the value requested, and now we may need to fix some EBB predecessors. while let Some(repair) = repairs.pop() { - for &(_, inst) in cfg.get_predecessors(repair.ebb) { + for (_, inst) in cfg.pred_iter(repair.ebb) { let branch_opc = pos.func.dfg[inst].opcode(); assert!( branch_opc.is_branch(), diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 7ca08620b6..628f8e0aaa 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -74,7 +74,7 @@ fn create_pre_header( for typ in header_args_types { pre_header_args_value.push(func.dfg.append_ebb_param(pre_header, typ), pool); } - for &(_, last_inst) in cfg.get_predecessors(header) { + for (_, last_inst) in cfg.pred_iter(header) { // We only follow normal edges (not the back edges) if !domtree.dominates(header, last_inst, &func.layout) { change_branch_jump_destination(last_inst, pre_header, func); @@ -103,7 +103,7 @@ fn has_pre_header( ) -> Option<(Ebb, Inst)> { let mut result = None; let mut found = false; - for &(pred_ebb, last_inst) in cfg.get_predecessors(header) { + for (pred_ebb, last_inst) in cfg.pred_iter(header) { // We only count normal edges (not the back edges) if !domtree.dominates(header, last_inst, layout) { if found { diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index b44bd77427..173ceb46ee 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -134,7 +134,7 @@ impl LoopAnalysis { ) { // We traverse the CFG in reverse postorder for &ebb in domtree.cfg_postorder().iter().rev() { - for &(_, pred_inst) in cfg.get_predecessors(ebb) { + for (_, pred_inst) in cfg.pred_iter(ebb) { // If the ebb dominates one of its predecessors it is a back edge if domtree.dominates(ebb, pred_inst, layout) { // This ebb is a loop header, so we create its associated loop @@ -160,7 +160,7 @@ impl LoopAnalysis { // We handle each loop header in reverse order, corresponding to a pseudo postorder // traversal of the graph. for lp in self.loops().rev() { - for &(pred, pred_inst) in cfg.get_predecessors(self.loops[lp].header) { + for (pred, pred_inst) in cfg.pred_iter(self.loops[lp].header) { // We follow the back edges if domtree.dominates(self.loops[lp].header, pred_inst, layout) { stack.push(pred); @@ -210,7 +210,7 @@ impl LoopAnalysis { // Now we have handled the popped node and need to continue the DFS by adding the // predecessors of that node if let Some(continue_dfs) = continue_dfs { - for &(pred, _) in cfg.get_predecessors(continue_dfs) { + for (pred, _) in cfg.pred_iter(continue_dfs) { stack.push(pred) } } diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 5f41ddc72a..224e97840a 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -8,7 +8,7 @@ use cursor::{Cursor, EncCursor}; use dbg::DisplayList; use dominator_tree::DominatorTree; -use flowgraph::{ControlFlowGraph, BasicBlock}; +use flowgraph::ControlFlowGraph; use ir::{DataFlowGraph, Layout, InstBuilder, ValueDef}; use ir::{Function, Ebb, Inst, Value, ExpandedProgramPoint}; use regalloc::affinity::Affinity; @@ -246,6 +246,7 @@ struct Context<'a> { encinfo: EncInfo, func: &'a mut Function, + cfg: &'a ControlFlowGraph, domtree: &'a DominatorTree, liveness: &'a mut Liveness, virtregs: &'a mut VirtRegs, @@ -288,6 +289,7 @@ impl Coalescing { isa, encinfo: isa.encoding_info(), func, + cfg, domtree, liveness, virtregs, @@ -299,11 +301,8 @@ impl Coalescing { // TODO: The iteration order matters here. We should coalesce in the most important blocks // first, so they get first pick at forming virtual registers. for &ebb in domtree.cfg_postorder() { - let preds = cfg.get_predecessors(ebb); - if !preds.is_empty() { - for argnum in 0..context.func.dfg.num_ebb_params(ebb) { - context.coalesce_ebb_param(ebb, argnum, preds) - } + for argnum in 0..context.func.dfg.num_ebb_params(ebb) { + context.coalesce_ebb_param(ebb, argnum) } } } @@ -311,7 +310,7 @@ impl Coalescing { impl<'a> Context<'a> { /// Coalesce the `argnum`'th parameter on `ebb`. - fn coalesce_ebb_param(&mut self, ebb: Ebb, argnum: usize, preds: &[BasicBlock]) { + fn coalesce_ebb_param(&mut self, ebb: Ebb, argnum: usize) { self.split_values.clear(); let mut succ_val = self.func.dfg.ebb_params(ebb)[argnum]; dbg!("Processing {}/{}: {}", ebb, argnum, succ_val); @@ -342,7 +341,7 @@ impl<'a> Context<'a> { // A successor copy is always required if the `succ_val` virtual register is live at // any predecessor branch. - while let Some(bad_value) = self.try_coalesce(argnum, succ_val, preds) { + while let Some(bad_value) = self.try_coalesce(argnum, succ_val, ebb) { dbg!("Isolating interfering value {}", bad_value); // The bad value has some conflict that can only be reconciled by excluding its // congruence class from the new virtual register. @@ -363,7 +362,7 @@ impl<'a> Context<'a> { } // Check the predecessors. - for &(pred_ebb, pred_inst) in preds { + for (pred_ebb, pred_inst) in self.cfg.pred_iter(ebb) { let pred_val = self.func.dfg.inst_variable_args(pred_inst)[argnum]; if self.virtregs.same_class(bad_value, pred_val) { self.split_pred(pred_inst, pred_ebb, argnum, pred_val); @@ -404,12 +403,7 @@ impl<'a> Context<'a> { /// /// Returns a value from a congruence class that needs to be split before starting over, or /// `None` if everything was successfully coalesced into `self.values`. - fn try_coalesce( - &mut self, - argnum: usize, - succ_val: Value, - preds: &[BasicBlock], - ) -> Option { + fn try_coalesce(&mut self, argnum: usize, succ_val: Value, succ_ebb: Ebb) -> Option { // Initialize the value list with the split values. These are guaranteed to be // interference free, and anything that interferes with them must be split away. self.reset_values(); @@ -421,7 +415,7 @@ impl<'a> Context<'a> { return Some(succ_val); } - for &(pred_ebb, pred_inst) in preds { + for (pred_ebb, pred_inst) in self.cfg.pred_iter(succ_ebb) { let pred_val = self.func.dfg.inst_variable_args(pred_inst)[argnum]; dbg!( "Checking {}: {}: {}", diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 3180b57d4c..e152cfaa32 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -269,7 +269,7 @@ fn extend_to_use( while let Some(livein) = worklist.pop() { // We've learned that the value needs to be live-in to the `livein` EBB. // Make sure it is also live at all predecessor branches to `livein`. - for &(pred, branch) in cfg.get_predecessors(livein) { + for (pred, branch) in cfg.pred_iter(livein) { if lr.extend_in_ebb(pred, branch, &func.layout) { // This predecessor EBB also became live-in. We need to process it later. worklist.push(pred); @@ -463,7 +463,7 @@ impl Liveness { _ => continue, }; - for &(_, pred_branch) in cfg.get_predecessors(succ_ebb) { + for (_, pred_branch) in cfg.pred_iter(succ_ebb) { let pred_arg = func.dfg.inst_variable_args(pred_branch)[num]; let pred_affinity = &mut self.ranges.get_mut(pred_arg).unwrap().affinity; if pred_affinity.is_none() { diff --git a/lib/cretonne/src/verifier/cssa.rs b/lib/cretonne/src/verifier/cssa.rs index a311ba4876..1010b45d0f 100644 --- a/lib/cretonne/src/verifier/cssa.rs +++ b/lib/cretonne/src/verifier/cssa.rs @@ -102,7 +102,7 @@ impl<'a> CssaVerifier<'a> { fn check_cssa(&self) -> Result { for ebb in self.func.layout.ebbs() { let ebb_params = self.func.dfg.ebb_params(ebb); - for &(_, pred) in self.cfg.get_predecessors(ebb) { + for (_, pred) in self.cfg.pred_iter(ebb) { let pred_args = self.func.dfg.inst_variable_args(pred); // This should have been caught by an earlier verifier pass. assert_eq!( diff --git a/lib/cretonne/src/verifier/flags.rs b/lib/cretonne/src/verifier/flags.rs index c2f3b5340a..f007844b51 100644 --- a/lib/cretonne/src/verifier/flags.rs +++ b/lib/cretonne/src/verifier/flags.rs @@ -60,7 +60,7 @@ impl<'a> FlagsVerifier<'a> { // Revisit any predecessor blocks the first time we see a live-in for `ebb`. None => { self.livein[ebb] = value.into(); - for &(pred, _) in self.cfg.get_predecessors(ebb) { + for (pred, _) in self.cfg.pred_iter(ebb) { worklist.insert(pred); } } diff --git a/lib/cretonne/src/verifier/liveness.rs b/lib/cretonne/src/verifier/liveness.rs index 118347e2fb..4e72d59224 100644 --- a/lib/cretonne/src/verifier/liveness.rs +++ b/lib/cretonne/src/verifier/liveness.rs @@ -226,7 +226,7 @@ impl<'a> LivenessVerifier<'a> { // Check all the EBBs in the interval independently. loop { // If `val` is live-in at `ebb`, it must be live at all the predecessors. - for &(_, pred) in self.cfg.get_predecessors(ebb) { + for (_, pred) in self.cfg.pred_iter(ebb) { if !self.live_at_use(lr, pred) { return err!( pred, diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 3df4975026..2144264f40 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -933,9 +933,8 @@ impl<'a> Verifier<'a> { return err!(ebb, "cfg had unexpected successor(s) {:?}", excess_succs); } - expected_preds.extend(self.expected_cfg.get_predecessors(ebb).iter().map(|&(_, - inst)| inst)); - got_preds.extend(cfg.get_predecessors(ebb).iter().map(|&(_, inst)| inst)); + expected_preds.extend(self.expected_cfg.pred_iter(ebb).map(|(_, inst)| inst)); + got_preds.extend(cfg.pred_iter(ebb).map(|(_, inst)| inst)); let missing_preds: Vec = expected_preds.difference(&got_preds).cloned().collect(); if !missing_preds.is_empty() { From 8e2ce6ded284018b7d45ba97102f26d20cfbaee7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 22 Nov 2017 11:35:03 -0800 Subject: [PATCH 1377/3084] Revert "Enable pager in cton-util." This reverts commit 0538615ccc0b600d4f534dae2ee966d5ed0df9b7. Fixes #196. The pager functionality wasn't working as intended since long error messages appear on stdout which isn't captured by the pager. --- cranelift/Cargo.toml | 3 --- cranelift/src/cton-util.rs | 18 ------------------ 2 files changed, 21 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index b328febdd0..45b39ef0a3 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -26,9 +26,6 @@ num_cpus = "1.5.1" tempdir="0.3.5" term = "0.4.6" -[target.'cfg(unix)'.dependencies] -pager = "0.13.0" - [workspace] # Enable debug assertions and parallel compilation when building cretonne-tools diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 15eb250f5e..b08fa8cde3 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -10,17 +10,11 @@ extern crate num_cpus; extern crate tempdir; extern crate term; -#[cfg(unix)] -extern crate pager; - use cretonne::VERSION; use docopt::Docopt; use std::io::{self, Write}; use std::process; -#[cfg(unix)] -use pager::Pager; - mod utils; mod filetest; mod cat; @@ -113,19 +107,7 @@ fn cton_util() -> CommandResult { } } -#[cfg(unix)] -fn enable_pager() { - Pager::new().skip_on_notty().setup(); -} - -#[cfg(not(unix))] -fn enable_pager() { - // For now, pager only supports unix-type platforms. -} - fn main() { - enable_pager(); - if let Err(mut msg) = cton_util() { if !msg.ends_with('\n') { msg.push('\n'); From c810b21488f721e7c97a34c28df278dec30fa032 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 22 Nov 2017 11:48:47 -0800 Subject: [PATCH 1378/3084] Be more forgiving about what's a "slow" test. This was supposed to be Q3 + 1.5 IQR, but a braino meant we actually used Q3 + 2/3 IQR. Since the distribution of test case times is far from gaussian, bump the "slow" limit up even further to Q3 + 3 IQR. --- cranelift/src/filetest/runner.rs | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/cranelift/src/filetest/runner.rs b/cranelift/src/filetest/runner.rs index 8e0f7721a7..320db9d942 100644 --- a/cranelift/src/filetest/runner.rs +++ b/cranelift/src/filetest/runner.rs @@ -302,9 +302,12 @@ impl TestRunner { // Inter-quartile range. let iqr = q3 - q1; - // Cut-off for what we consider a 'slow' test: 1.5 IQR from the 75% quartile. - // These are the data points that would be plotted as outliers outside a box plot. - let cut = q3 + iqr * 2 / 3; + + // Cut-off for what we consider a 'slow' test: 3 IQR from the 75% quartile. + // + // Q3 + 1.5 IQR are the data points that would be plotted as outliers outside a box plot, + // but we have a wider distribution of test times, so double it to 3 IQR. + let cut = q3 + iqr * 3; if cut > *times.last().unwrap() { return; } From a960730add4ad4d3d5248187da6a14a78b621097 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 22 Nov 2017 13:43:41 -0800 Subject: [PATCH 1379/3084] Add retain() methods to bforest::{Set,Map}. These work just like their counterparts in HashMap and HashSet. --- lib/cretonne/src/bforest/map.rs | 37 +++++++++++++++++++++++++++++++++ lib/cretonne/src/bforest/set.rs | 22 ++++++++++++++++++++ 2 files changed, 59 insertions(+) diff --git a/lib/cretonne/src/bforest/map.rs b/lib/cretonne/src/bforest/map.rs index 601e9febe6..bf288582eb 100644 --- a/lib/cretonne/src/bforest/map.rs +++ b/lib/cretonne/src/bforest/map.rs @@ -130,6 +130,32 @@ where } } + /// Retains only the elements specified by the predicate. + /// + /// Remove all key-value pairs where the predicate returns false. + /// + /// The predicate is allowed to update the values stored in the map. + pub fn retain(&mut self, forest: &mut MapForest, mut predicate: F) + where + F: FnMut(K, &mut V) -> bool, + { + let mut path = Path::default(); + if let Some(root) = self.root.expand() { + path.first(root, &forest.nodes); + } + while let Some((node, entry)) = path.leaf_pos() { + let keep = { + let (ks, vs) = forest.nodes[node].unwrap_leaf_mut(); + predicate(ks[entry], &mut vs[entry]) + }; + if keep { + path.next(&forest.nodes); + } else { + self.root = path.remove(&mut forest.nodes).into(); + } + } + } + /// Create a cursor for navigating this map. The cursor is initially positioned off the end of /// the map. pub fn cursor<'a>( @@ -394,6 +420,7 @@ mod test { assert_eq!(m.get(7, &f, &()), None); assert_eq!(m.iter(&f).next(), None); + m.retain(&mut f, |_, _| unreachable!()); let mut c = m.cursor(&mut f, &()); assert!(c.is_empty()); @@ -525,6 +552,16 @@ mod test { m.verify(f, &()); assert_eq!(m.get(5, f, &()), Some(4.2)); + // Retain even entries, with altered values. + m.retain(f, |k, v| { + *v = (k / 10) as f32; + (k % 20) == 0 + }); + assert_eq!( + m.iter(f).collect::>(), + [(20, 2.0), (40, 4.0), (60, 6.0)] + ); + // Insert at back of leaf. let mut m = full_leaf(f); m.insert(80, 4.2, f, &()); diff --git a/lib/cretonne/src/bforest/set.rs b/lib/cretonne/src/bforest/set.rs index 025efc0225..6992265773 100644 --- a/lib/cretonne/src/bforest/set.rs +++ b/lib/cretonne/src/bforest/set.rs @@ -127,6 +127,26 @@ where } } + /// Retains only the elements specified by the predicate. + /// + /// Remove all elements where the predicate returns false. + pub fn retain(&mut self, forest: &mut SetForest, mut predicate: F) + where + F: FnMut(K) -> bool, + { + let mut path = Path::default(); + if let Some(root) = self.root.expand() { + path.first(root, &forest.nodes); + } + while let Some((node, entry)) = path.leaf_pos() { + if predicate(forest.nodes[node].unwrap_leaf().0[entry]) { + path.next(&forest.nodes); + } else { + self.root = path.remove(&mut forest.nodes).into(); + } + } + } + /// Create a cursor for navigating this set. The cursor is initially positioned off the end of /// the set. pub fn cursor<'a>( @@ -354,6 +374,8 @@ mod test { // Iterator for an empty set. assert_eq!(s.iter(&f).next(), None); + s.retain(&mut f, |_| unreachable!()); + let mut c = SetCursor::new(&mut s, &mut f, &()); c.verify(); assert_eq!(c.elem(), None); From 95bdf91e3efd45ca1ca12e14893d173b3427596a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 22 Nov 2017 13:52:42 -0800 Subject: [PATCH 1380/3084] Use a bforest::Map for the predecessors in the CFG. Like before, we store a redundant EBB with each predecessor instruction which allows invalidate_ebb_successors() to clean up the successor's predecessor lists even after instructions have been moved around in the layout. --- lib/cretonne/src/flowgraph.rs | 39 +++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 9 deletions(-) diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index a2c27828fb..10eff77857 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -28,7 +28,6 @@ use ir::{Function, Inst, Ebb}; use ir::instructions::BranchInfo; use entity::EntityMap; use std::mem; -use std::slice; /// A basic block denoted by its enclosing Ebb and last instruction. pub type BasicBlock = (Ebb, Inst); @@ -36,12 +35,23 @@ pub type BasicBlock = (Ebb, Inst); /// A container for the successors and predecessors of some Ebb. #[derive(Clone, Default)] pub struct CFGNode { + /// Instructions that can branch or jump to this EBB. + /// + /// This maps branch instruction -> predecessor EBB which is redundant since the EBB containing + /// the branch instruction is available from the `layout.inst_ebb()` method. We store the + /// redundant information because: + /// + /// 1. Many `pred_iter()` consumers want the EBB anyway, so it is handily available. + /// 2. The `invalidate_ebb_successors()` may be called *after* branches have been removed from + /// their EBB, but we still need to remove them form the old EBB predecessor map. + /// + /// The redundant EBB stored here is always consistent with the CFG successor lists, even after + /// the IR has been edited. + pub predecessors: bforest::Map, + /// Set of EBBs that are the targets of branches and jumps in this EBB. /// The set is ordered by EBB number, indicated by the `()` comparator type. pub successors: bforest::Set, - - /// Basic blocks that can branch or jump to this EBB. - pub predecessors: Vec, } /// The Control Flow Graph maintains a mapping of ebbs to their predecessors @@ -49,6 +59,7 @@ pub struct CFGNode { /// extended basic blocks. pub struct ControlFlowGraph { data: EntityMap, + pred_forest: bforest::MapForest, succ_forest: bforest::SetForest, valid: bool, } @@ -59,6 +70,7 @@ impl ControlFlowGraph { Self { data: EntityMap::new(), valid: false, + pred_forest: bforest::MapForest::new(), succ_forest: bforest::SetForest::new(), } } @@ -66,6 +78,7 @@ impl ControlFlowGraph { /// Clear all data structures in this control flow graph. pub fn clear(&mut self) { self.data.clear(); + self.pred_forest.clear(); self.succ_forest.clear(); self.valid = false; } @@ -113,7 +126,10 @@ impl ControlFlowGraph { // our iteration over successors. let mut successors = mem::replace(&mut self.data[ebb].successors, Default::default()); for succ in successors.iter(&self.succ_forest) { - self.data[succ].predecessors.retain(|&(e, _)| e != ebb); + self.data[succ].predecessors.retain( + &mut self.pred_forest, + |_, &mut e| e != ebb, + ); } successors.clear(&mut self.succ_forest); } @@ -136,12 +152,17 @@ impl ControlFlowGraph { &mut self.succ_forest, &(), ); - self.data[to].predecessors.push(from); + self.data[to].predecessors.insert( + from.1, + from.0, + &mut self.pred_forest, + &(), + ); } /// Get an iterator over the CFG predecessors to `ebb`. pub fn pred_iter(&self, ebb: Ebb) -> PredIter { - PredIter(self.data[ebb].predecessors.iter()) + PredIter(self.data[ebb].predecessors.iter(&self.pred_forest)) } /// Get an iterator over the CFG successors to `ebb`. @@ -163,13 +184,13 @@ impl ControlFlowGraph { /// An iterator over EBB predecessors. The iterator type is `BasicBlock`. /// /// Each predecessor is an instruction that branches to the EBB. -pub struct PredIter<'a>(slice::Iter<'a, BasicBlock>); +pub struct PredIter<'a>(bforest::MapIter<'a, Inst, Ebb, ()>); impl<'a> Iterator for PredIter<'a> { type Item = BasicBlock; fn next(&mut self) -> Option { - self.0.next().cloned() + self.0.next().map(|(i, e)| (e, i)) } } From 894268233edf7bba32714f298dbcffae90d0feb7 Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Wed, 22 Nov 2017 14:40:54 -0800 Subject: [PATCH 1381/3084] ir::ExternalName representation and constructors rewritten --- lib/cretonne/src/ir/extname.rs | 169 ++++++++++++++++----------------- 1 file changed, 83 insertions(+), 86 deletions(-) diff --git a/lib/cretonne/src/ir/extname.rs b/lib/cretonne/src/ir/extname.rs index ba5ddba40b..290a40d9f4 100644 --- a/lib/cretonne/src/ir/extname.rs +++ b/lib/cretonne/src/ir/extname.rs @@ -5,10 +5,12 @@ //! Cretonne, which compiles functions independently. use std::fmt::{self, Write}; -#[allow(unused_imports)] -use std::ascii::AsciiExt; -/// The name of an external can be any sequence of bytes. +const TESTCASE_NAME_LENGTH: usize = 10; + +/// The name of an external is either a reference to a user-defined symbol +/// table, or a short sequence of ascii bytes so that test cases do not have +/// to keep track of a sy mbol table. /// /// External names are primarily used as keys by code using Cretonne to map /// from a cretonne::ir::FuncRef or similar to additional associated data. @@ -16,103 +18,92 @@ use std::ascii::AsciiExt; /// External names can also serve as a primitive testing and debugging tool. /// In particular, many `.cton` test files use function names to identify /// functions. -#[derive(Debug, Clone, PartialEq, Eq, Default, Hash)] -pub struct ExternalName(NameRepr); +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum ExternalName { + /// A name in a user-defined symbol table. Cretonne does not interpret + /// these numbers in any way. + User { + /// Arbitrary + namespace: u32, + /// Arbitrary + index: u32, + }, + /// A test case function name of up to 10 ascii characters. This is + /// not intended to be used outside test cases. + TestCase { + /// How many of the bytes in `ascii` are valid + length: u8, + /// Ascii bytes of the name + ascii: [u8; TESTCASE_NAME_LENGTH], + }, +} impl ExternalName { - /// Creates a new external name from a sequence of bytes. + /// Creates a new external name from a sequence of bytes. Caller is expected + /// to guarantee bytes are only ascii alphanumeric or `_`. /// /// # Examples /// /// ```rust /// # use cretonne::ir::ExternalName; /// // Create `ExternalName` from a string. - /// let name = ExternalName::new("hello"); + /// let name = ExternalName::testcase("hello"); /// assert_eq!(name.to_string(), "%hello"); - /// - /// // Create `ExternalName` from a sequence of bytes. - /// let bytes: &[u8] = &[10, 9, 8]; - /// let name = ExternalName::new(bytes); - /// assert_eq!(name.to_string(), "#0a0908"); /// ``` - pub fn new(v: T) -> ExternalName + pub fn testcase(v: T) -> ExternalName where T: Into>, { let vec = v.into(); - if vec.len() <= NAME_LENGTH_THRESHOLD { - let mut bytes = [0u8; NAME_LENGTH_THRESHOLD]; - for (i, &byte) in vec.iter().enumerate() { - bytes[i] = byte; - } - ExternalName(NameRepr::Short { - length: vec.len() as u8, - bytes: bytes, - }) + let len = if vec.len() > TESTCASE_NAME_LENGTH { + TESTCASE_NAME_LENGTH } else { - ExternalName(NameRepr::Long(vec)) + vec.len() + }; + let mut bytes = [0u8; TESTCASE_NAME_LENGTH]; + for (i, &byte) in vec.iter().take(len).enumerate() { + bytes[i] = byte; + } + ExternalName::TestCase { + length: len as u8, + ascii: bytes, + } + } + + /// Create a new external name from user-provided integer indicies. + /// + /// # Examples + /// ```rust + /// # use cretonne::ir::ExternalName; + /// // Create `ExternalName` from integer indicies + /// let name = ExternalName::user(123, 456); + /// assert_eq!(name.to_string(), "u123:456"); + /// ``` + pub fn user(namespace: u32, index: u32) -> ExternalName { + ExternalName::User { + namespace: namespace, + index: index, } } } -/// Tries to interpret bytes as ASCII alphanumerical characters and `_`. -fn try_as_name(bytes: &[u8]) -> Option { - let mut name = String::with_capacity(bytes.len()); - for c in bytes.iter().map(|&b| b as char) { - if c.is_ascii() && c.is_alphanumeric() || c == '_' { - name.push(c); - } else { - return None; - } - } - Some(name) -} - -const NAME_LENGTH_THRESHOLD: usize = 22; - -#[derive(Debug, Clone, PartialEq, Eq, Hash)] -enum NameRepr { - Short { - length: u8, - bytes: [u8; NAME_LENGTH_THRESHOLD], - }, - Long(Vec), -} - -impl AsRef<[u8]> for NameRepr { - fn as_ref(&self) -> &[u8] { - match *self { - NameRepr::Short { length, ref bytes } => &bytes[0..length as usize], - NameRepr::Long(ref vec) => vec.as_ref(), - } - } -} - -impl AsRef<[u8]> for ExternalName { - fn as_ref(&self) -> &[u8] { - self.0.as_ref() - } -} - -impl Default for NameRepr { - fn default() -> Self { - NameRepr::Short { - length: 0, - bytes: [0; NAME_LENGTH_THRESHOLD], - } +impl Default for ExternalName { + fn default() -> ExternalName { + ExternalName::user(0, 0) } } impl fmt::Display for ExternalName { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - if let Some(name) = try_as_name(self.0.as_ref()) { - write!(f, "%{}", name) - } else { - f.write_char('#')?; - for byte in self.0.as_ref() { - write!(f, "{:02x}", byte)?; + match *self { + ExternalName::User { namespace, index } => write!(f, "u{}:{}", namespace, index), + ExternalName::TestCase { length, ascii } => { + f.write_char('%')?; + for byte in ascii.iter().take(length as usize) { + f.write_char(*byte as char)?; + } + Ok(()) } - Ok(()) } } } @@ -122,22 +113,28 @@ mod tests { use super::ExternalName; #[test] - fn displaying() { - assert_eq!(ExternalName::new("").to_string(), "%"); - assert_eq!(ExternalName::new("x").to_string(), "%x"); - assert_eq!(ExternalName::new("x_1").to_string(), "%x_1"); - assert_eq!(ExternalName::new(" ").to_string(), "#20"); + fn display_testcase() { + assert_eq!(ExternalName::testcase("").to_string(), "%"); + assert_eq!(ExternalName::testcase("x").to_string(), "%x"); + assert_eq!(ExternalName::testcase("x_1").to_string(), "%x_1"); assert_eq!( - ExternalName::new("кретон").to_string(), - "#d0bad180d0b5d182d0bed0bd" + ExternalName::testcase("long123456").to_string(), + "%long123456" ); + // Constructor will silently drop bytes beyond the 10th assert_eq!( - ExternalName::new("印花棉布").to_string(), - "#e58db0e88ab1e6a389e5b883" + ExternalName::testcase("long1234567").to_string(), + "%long123456" ); + } + + #[test] + fn display_user() { + assert_eq!(ExternalName::user(0, 0).to_string(), "u0:0"); + assert_eq!(ExternalName::user(1, 1).to_string(), "u1:1"); assert_eq!( - ExternalName::new(vec![0, 1, 2, 3, 4, 5]).to_string(), - "#000102030405" + ExternalName::user(::std::u32::MAX, ::std::u32::MAX).to_string(), + "u4294967295:4294967295" ); } } From 2dfc78dbcd7a501456020ce0db5834fc96dd58bc Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Wed, 22 Nov 2017 16:15:31 -0800 Subject: [PATCH 1382/3084] reader: lex UserRefs and parse ExternalName::Users --- lib/reader/src/lexer.rs | 15 +++++++++++++++ lib/reader/src/parser.rs | 34 ++++++++++++++++++---------------- 2 files changed, 33 insertions(+), 16 deletions(-) diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 1c422a8027..ca315d3b58 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -38,6 +38,7 @@ pub enum Token<'a> { JumpTable(u32), // jt2 FuncRef(u32), // fn2 SigRef(u32), // sig2 + UserRef(u32), // u345 Name(&'a str), // %9arbitrary_alphanum, %x3, %0, %function ... HexSequence(&'a str), // #89AF Identifier(&'a str), // Unrecognized identifier (opcode, enumerator, ...) @@ -320,6 +321,7 @@ impl<'a> Lexer<'a> { "jt" => Some(Token::JumpTable(number)), "fn" => Some(Token::FuncRef(number)), "sig" => Some(Token::SigRef(number)), + "u" => Some(Token::UserRef(number)), _ => None, } } @@ -604,4 +606,17 @@ mod tests { assert_eq!(lex.next(), token(Token::Name("ebb11"), 1)); assert_eq!(lex.next(), token(Token::Name("_"), 1)); } + + #[test] + fn lex_userrefs() { + let mut lex = Lexer::new("u0 u1 u234567890 u9:8765"); + + assert_eq!(lex.next(), token(Token::UserRef(0), 1)); + assert_eq!(lex.next(), token(Token::UserRef(1), 1)); + assert_eq!(lex.next(), token(Token::UserRef(234567890), 1)); + assert_eq!(lex.next(), token(Token::UserRef(9), 1)); + assert_eq!(lex.next(), token(Token::Colon, 1)); + assert_eq!(lex.next(), token(Token::Integer("8765"), 1)); + assert_eq!(lex.next(), None); + } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 55cf342ad3..03516fa91d 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -890,24 +890,26 @@ impl<'a> Parser<'a> { match self.token() { Some(Token::Name(s)) => { self.consume(); - Ok(ExternalName::new(s)) + Ok(ExternalName::testcase(s)) } - Some(Token::HexSequence(s)) => { - if s.len() % 2 != 0 { - return err!( - self.loc, - "expected binary external name to have length multiple of two" - ); - } - let mut bin_name = Vec::with_capacity(s.len() / 2); - let mut i = 0; - while i + 2 <= s.len() { - let byte = u8::from_str_radix(&s[i..i + 2], 16).unwrap(); - bin_name.push(byte); - i += 2; - } + Some(Token::UserRef(namespace)) => { self.consume(); - Ok(ExternalName::new(bin_name)) + match self.token() { + Some(Token::Colon) => { + self.consume(); + match self.token() { + Some(Token::Integer(index_str)) => { + let index: u32 = u32::from_str_radix(index_str, 10).map_err(|_| { + self.error("the integer given overflows the u32 type") + })?; + self.consume(); + Ok(ExternalName::user(namespace, index)) + } + _ => err!(self.loc, "expected integer"), + } + } + _ => err!(self.loc, "expected colon"), + } } _ => err!(self.loc, "expected external name"), } From 1216f81b90793f57f51d691a399b66c53c977e40 Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Wed, 22 Nov 2017 16:52:08 -0800 Subject: [PATCH 1383/3084] parser: replace binary function name test case with user names --- lib/reader/src/parser.rs | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 03516fa91d..d809cd192c 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -2584,45 +2584,50 @@ mod tests { } #[test] - fn binary_function_name() { - // Valid characters in the name. + fn user_function_name() { + // Valid characters in the name: let func = Parser::new( - "function #1234567890AbCdEf() native { + "function u1:2() native { ebb0: trap int_divz }", ).parse_function(None) .unwrap() .0; - assert_eq!(func.name.to_string(), "#1234567890abcdef"); + assert_eq!(func.name.to_string(), "u1:2"); - // Invalid characters in the name. + // Invalid characters in the name: let mut parser = Parser::new( - "function #12ww() native { + "function u123:abc() native { ebb0: trap stk_ovf }", ); assert!(parser.parse_function(None).is_err()); - // The length of binary function name should be multiple of two. + // Incomplete function names should not be valid: let mut parser = Parser::new( - "function #1() native { + "function u() native { ebb0: - trap user0 + trap int_ovf }", ); assert!(parser.parse_function(None).is_err()); - // Empty binary function name should be valid. - let func = Parser::new( - "function #() native { + let mut parser = Parser::new( + "function u0() native { ebb0: trap int_ovf }", - ).parse_function(None) - .unwrap() - .0; - assert_eq!(func.name.to_string(), "%"); + ); + assert!(parser.parse_function(None).is_err()); + + let mut parser = Parser::new( + "function u0:() native { + ebb0: + trap int_ovf + }", + ); + assert!(parser.parse_function(None).is_err()); } } From 803b83c87a260e7f5e6d23dc6ccd81c1150b12b7 Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Wed, 22 Nov 2017 17:18:54 -0800 Subject: [PATCH 1384/3084] wasm: change to new externalname api --- lib/wasm/src/environ/dummy.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 64733684be..8c9c7c3d8d 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -13,7 +13,7 @@ use std::error::Error; /// Compute a `ir::ExternalName` for a given wasm function index. fn get_func_name(func_index: FunctionIndex) -> ir::ExternalName { - ir::ExternalName::new(format!("wasm_0x{:x}", func_index)) + ir::ExternalName::user(0, func_index as u32) } /// A collection of names under which a given entity is exported. From eb49d9f8bffe3849b545fd203cc0b25369e9381e Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Wed, 22 Nov 2017 17:37:08 -0800 Subject: [PATCH 1385/3084] tests: update ExternalName api --- lib/cretonne/src/write.rs | 4 ++-- lib/frontend/src/frontend.rs | 2 +- lib/frontend/src/lib.rs | 2 +- lib/wasm/src/func_translator.rs | 6 +++--- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index b4276349de..9de3e93d58 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -447,9 +447,9 @@ mod tests { #[test] fn basic() { let mut f = Function::new(); - assert_eq!(f.to_string(), "function %() native {\n}\n"); + assert_eq!(f.to_string(), "function u0:0() native {\n}\n"); - f.name = ExternalName::new("foo"); + f.name = ExternalName::testcase("foo"); assert_eq!(f.to_string(), "function %foo() native {\n}\n"); f.create_stack_slot(StackSlotData::new(StackSlotKind::Local, 4)); diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 5127270cbd..432288e4eb 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -613,7 +613,7 @@ mod tests { sig.params.push(AbiParam::new(I32)); let mut il_builder = ILBuilder::::new(); - let mut func = Function::with_name_signature(ExternalName::new("sample_function"), sig); + let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig); { let mut builder = FunctionBuilder::::new(&mut func, &mut il_builder); diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 12b8c8db73..4307023ab9 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -62,7 +62,7 @@ //! sig.returns.push(AbiParam::new(I32)); //! sig.params.push(AbiParam::new(I32)); //! let mut il_builder = ILBuilder::::new(); -//! let mut func = Function::with_name_signature(ExternalName::new("sample_function"), sig); +//! let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig); //! { //! let mut builder = FunctionBuilder::::new(&mut func, &mut il_builder); //! diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index b5407decd9..f68996d9b6 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -252,7 +252,7 @@ mod tests { let runtime = DummyEnvironment::default(); let mut ctx = Context::new(); - ctx.func.name = ir::ExternalName::new("small1"); + ctx.func.name = ir::ExternalName::testcase("small1"); ctx.func.signature.params.push(ir::AbiParam::new(I32)); ctx.func.signature.returns.push(ir::AbiParam::new(I32)); @@ -283,7 +283,7 @@ mod tests { let runtime = DummyEnvironment::default(); let mut ctx = Context::new(); - ctx.func.name = ir::ExternalName::new("small2"); + ctx.func.name = ir::ExternalName::testcase("small2"); ctx.func.signature.params.push(ir::AbiParam::new(I32)); ctx.func.signature.returns.push(ir::AbiParam::new(I32)); @@ -323,7 +323,7 @@ mod tests { let runtime = DummyEnvironment::default(); let mut ctx = Context::new(); - ctx.func.name = ir::ExternalName::new("infloop"); + ctx.func.name = ir::ExternalName::testcase("infloop"); ctx.func.signature.returns.push(ir::AbiParam::new(I32)); trans From 9108725b74c83d8438c57e7ca78d796c35ed2ffa Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Wed, 22 Nov 2017 17:51:42 -0800 Subject: [PATCH 1386/3084] extname: extend testcases to 16 chars this is sufficient for all filetests to pass without truncating --- lib/cretonne/src/ir/extname.rs | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/src/ir/extname.rs b/lib/cretonne/src/ir/extname.rs index 290a40d9f4..36cc18d5c3 100644 --- a/lib/cretonne/src/ir/extname.rs +++ b/lib/cretonne/src/ir/extname.rs @@ -6,7 +6,7 @@ use std::fmt::{self, Write}; -const TESTCASE_NAME_LENGTH: usize = 10; +const TESTCASE_NAME_LENGTH: usize = 16; /// The name of an external is either a reference to a user-defined symbol /// table, or a short sequence of ascii bytes so that test cases do not have @@ -118,13 +118,13 @@ mod tests { assert_eq!(ExternalName::testcase("x").to_string(), "%x"); assert_eq!(ExternalName::testcase("x_1").to_string(), "%x_1"); assert_eq!( - ExternalName::testcase("long123456").to_string(), - "%long123456" + ExternalName::testcase("longname12345678").to_string(), + "%longname12345678" ); - // Constructor will silently drop bytes beyond the 10th + // Constructor will silently drop bytes beyond the 16th assert_eq!( - ExternalName::testcase("long1234567").to_string(), - "%long123456" + ExternalName::testcase("longname123456789").to_string(), + "%longname12345678" ); } From b5601d57c8930c880bb26a8eb6a6498932192897 Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Wed, 22 Nov 2017 17:52:05 -0800 Subject: [PATCH 1387/3084] filetests: change hex function names to user function numbers --- cranelift/filetests/isa/intel/abi64.cton | 2 +- cranelift/filetests/isa/intel/legalize-memory.cton | 2 +- cranelift/filetests/parser/memory.cton | 4 ++-- cranelift/filetests/regalloc/iterate.cton | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/cranelift/filetests/isa/intel/abi64.cton b/cranelift/filetests/isa/intel/abi64.cton index ec943203ee..d56a31d479 100644 --- a/cranelift/filetests/isa/intel/abi64.cton +++ b/cranelift/filetests/isa/intel/abi64.cton @@ -21,7 +21,7 @@ ebb0: function %pass_stack_int64(i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) spiderwasm { sig0 = (i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) spiderwasm - fn0 = sig0 #00000000 + fn0 = sig0 u0:0 ebb0(v0: i64, v1: i64, v2: i64, v3: i64, v4: i64, v5: i64, v6: i64, v7: i64, v8: i64, v9: i64, v10: i64, v11: i64, v12: i64, v13: i64, v14: i64, v15: i64, v16: i64, v17: i64, v18: i64, v19: i64, v20: i64): call fn0(v0, v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16, v17, v18, v19, v20) diff --git a/cranelift/filetests/isa/intel/legalize-memory.cton b/cranelift/filetests/isa/intel/legalize-memory.cton index b9e8e99696..0257c7001c 100644 --- a/cranelift/filetests/isa/intel/legalize-memory.cton +++ b/cranelift/filetests/isa/intel/legalize-memory.cton @@ -31,7 +31,7 @@ ebb1(v1: i64): function %sym() -> i64 { gv0 = globalsym %something - gv1 = globalsym #d0bad180d0b5d182d0bed0bd + gv1 = globalsym u123:456 ebb1: v0 = global_addr.i64 gv0 diff --git a/cranelift/filetests/parser/memory.cton b/cranelift/filetests/parser/memory.cton index 6fa594c4c1..0b137335c9 100644 --- a/cranelift/filetests/parser/memory.cton +++ b/cranelift/filetests/parser/memory.cton @@ -39,8 +39,8 @@ ebb0: function %sym() -> i32 { gv0 = globalsym %something ; check: $gv0 = globalsym %something - gv1 = globalsym #d0bad180d0b5d182d0bed0bd - ; check: $gv1 = globalsym #d0bad180d0b5d182d0bed0bd + gv1 = globalsym u8:9 + ; check: $gv1 = globalsym u8:9 ebb0: v0 = global_addr.i32 gv0 ; check: $v0 = global_addr.i32 $gv0 diff --git a/cranelift/filetests/regalloc/iterate.cton b/cranelift/filetests/regalloc/iterate.cton index 6f7e6547fd..2c585e941d 100644 --- a/cranelift/filetests/regalloc/iterate.cton +++ b/cranelift/filetests/regalloc/iterate.cton @@ -2,7 +2,7 @@ test regalloc set is_64bit isa intel haswell -function #00000009(i64 [%rdi], f32 [%xmm0], f64 [%xmm1], i32 [%rsi], i32 [%rdx], i64 vmctx [%r14]) -> i64 [%rax] spiderwasm { +function u0:9(i64 [%rdi], f32 [%xmm0], f64 [%xmm1], i32 [%rsi], i32 [%rdx], i64 vmctx [%r14]) -> i64 [%rax] spiderwasm { ebb0(v0: i64, v1: f32, v2: f64, v3: i32, v4: i32, v5: i64): v32 = iconst.i32 0 v6 = bitcast.f32 v32 @@ -104,7 +104,7 @@ ebb1(v31: i64): return v31 } -function #0000001a(i64 vmctx [%r14]) -> i64 [%rax] spiderwasm { +function u0:26(i64 vmctx [%r14]) -> i64 [%rax] spiderwasm { gv0 = vmctx+48 sig0 = (i32 [%rdi], i64 [%rsi], i64 vmctx [%r14], i64 sigid [%rbx]) -> i64 [%rax] spiderwasm From cced2c8b0c101246f108c8f302c78b81e738f5e5 Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Mon, 27 Nov 2017 12:46:53 -0800 Subject: [PATCH 1388/3084] Fix wat syntax so wasm tests pass (#199) * wasm testsuite: ignore hidden files in test dir and report a rejected file. it was picking up vim .swp files * wasmtests: correct wat syntax in icall.wat --- cranelift/wasmtests/icall.wat | 2 +- lib/wasm/tests/testsuite.rs | 11 ++++++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/cranelift/wasmtests/icall.wat b/cranelift/wasmtests/icall.wat index 21894c55ca..76f28f47a9 100644 --- a/cranelift/wasmtests/icall.wat +++ b/cranelift/wasmtests/icall.wat @@ -1,7 +1,7 @@ (module (type $ft (func (param f32) (result i32))) (func $foo (export "foo") (param i32 f32) (result i32) - (call_indirect $ft (get_local 1) (get_local 0)) + (call_indirect (type $ft) (get_local 1) (get_local 0)) ) (table (;0;) 23 23 anyfunc) ) diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/testsuite.rs index 687aa7c938..c551dbf783 100644 --- a/lib/wasm/tests/testsuite.rs +++ b/lib/wasm/tests/testsuite.rs @@ -23,6 +23,15 @@ fn testsuite() { let mut paths: Vec<_> = fs::read_dir("../../wasmtests") .unwrap() .map(|r| r.unwrap()) + .filter(|p| { + // Ignore files starting with `.`, which could be editor temporary files + if let Some(stem) = p.path().file_stem() { + if let Some(stemstr) = stem.to_str() { + return !stemstr.starts_with("."); + } + } + false + }) .collect(); paths.sort_by_key(|dir| dir.path()); let flags = Flags::new(&settings::builder()); @@ -88,7 +97,7 @@ fn handle_module(path: PathBuf, flags: &Flags) { } read_wasm_file(file_path).expect("error reading converted wasm file") } - None | Some(&_) => panic!("the file extension is not wasm or wat"), + None | Some(&_) => panic!("the file extension for {:?} is not wasm or wat", path), } } }; From 27f0b2918832c82d191e06e524df144755a3418c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 1 Dec 2017 06:32:38 -0800 Subject: [PATCH 1389/3084] Replace `hex` with `"0x%x" % ...` to fix a trailing 'L' on 32-bit python2 This test was failing because the hash computes values greater than 0x7fffffff, which 32-bit python2 promotes to long, which `hex` formats with a trailing 'L'. The code that uses it appears to be ok with a long value, so it's just the test that needs to handle it. Fix as suggested here: https://stackoverflow.com/questions/5917203/python-trailing-l-problem --- lib/cretonne/meta/constant_hash.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/meta/constant_hash.py b/lib/cretonne/meta/constant_hash.py index 47f5eebe33..9ef19ff098 100644 --- a/lib/cretonne/meta/constant_hash.py +++ b/lib/cretonne/meta/constant_hash.py @@ -20,9 +20,9 @@ def simple_hash(s): Compute a primitive hash of a string. Example: - >>> hex(simple_hash("Hello")) + >>> "0x%x" % simple_hash("Hello") '0x2fa70c01' - >>> hex(simple_hash("world")) + >>> "0x%x" % simple_hash("world") '0x5b0c31d5' """ h = 5381 From 0676e8cbd732becc2be95e1bddbc987e5978086a Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Mon, 27 Nov 2017 16:05:28 -0800 Subject: [PATCH 1390/3084] install wasm-toolchain build 26619 from deb --- .travis.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.travis.yml b/.travis.yml index 03dd6f76e6..ce1a269955 100644 --- a/.travis.yml +++ b/.travis.yml @@ -8,6 +8,9 @@ addons: apt: packages: - python3-pip +before_install: + - wget https://storage.googleapis.com/wasm-llvm/builds/linux/26619/wasm-toolchain_0.1.26619_amd64.deb + - sudo dpkg -i wasm-toolchain_0.1.26619_amd64.deb install: - pip3 install --user --upgrade mypy flake8 - travis_wait ./check-rustfmt.sh --install From 8e0110b0760f0f80f3143c5f7746bb603f18c882 Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Mon, 27 Nov 2017 16:08:13 -0800 Subject: [PATCH 1391/3084] add wat2wasm note to readme --- README.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.rst b/README.rst index 1afcd5319d..d62e5b2777 100644 --- a/README.rst +++ b/README.rst @@ -59,6 +59,11 @@ You can then run tests with:: ./test-all.sh +You may need to install the *wat2wasm* tool from the `wabt +`_ project in order to run all of the +WebAssembly tests. Tests requiring wat2wasm are ignored if the tool is not +installed. + Building the documentation -------------------------- From 919de82e9c1354f448573a657701638e391d4ac8 Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Mon, 27 Nov 2017 16:36:40 -0800 Subject: [PATCH 1392/3084] rename wasm testsuite to wasm_testsuite this makes it a lot more obvious in the travis build output --- lib/wasm/tests/{testsuite.rs => wasm_testsuite.rs} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename lib/wasm/tests/{testsuite.rs => wasm_testsuite.rs} (100%) diff --git a/lib/wasm/tests/testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs similarity index 100% rename from lib/wasm/tests/testsuite.rs rename to lib/wasm/tests/wasm_testsuite.rs From 04f6ccabe51d879b89622f3ad7b193cecb3a2f15 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 4 Dec 2017 09:18:30 -0800 Subject: [PATCH 1393/3084] Allow filecheck directives with "test compile". Things like inserted prologues and epilogues in #201 can be tested this way. --- cranelift/docs/testing.rst | 11 +++++++++++ cranelift/src/filetest/compile.rs | 11 ++++++++--- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index edc8ee6570..6df29d6fd1 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -352,3 +352,14 @@ Test the simple GVN pass. The simple GVN pass is run on each function, and then results are run through filecheck. + +`test compile` +-------------- + +Test the whole code generation pipeline. + +Each function is passed through the full ``Context::compile()`` function +which is normally used to compile code. This type of test often depends +on assertions or verifier errors, but it is also possible to use +filecheck directives which will be matched against the final form of the +Cretonne IL right before binary machine code emission. diff --git a/cranelift/src/filetest/compile.rs b/cranelift/src/filetest/compile.rs index 3aaa0a6b8a..0e271b7db4 100644 --- a/cranelift/src/filetest/compile.rs +++ b/cranelift/src/filetest/compile.rs @@ -6,8 +6,9 @@ use cretonne::binemit; use cretonne::ir; use cretonne; use cton_reader::TestCommand; -use filetest::subtest::{SubTest, Context, Result}; +use filetest::subtest::{SubTest, Context, Result, run_filecheck}; use std::borrow::Cow; +use std::fmt::Write; use utils::pretty_error; struct TestCompile; @@ -51,7 +52,7 @@ impl SubTest for TestCompile { comp_ctx.func.display(isa) ); - // Finally verify that the returned code size matches the emitted bytes. + // Verify that the returned code size matches the emitted bytes. let mut sink = SizeSink { offset: 0 }; binemit::emit_function( &comp_ctx.func, @@ -67,7 +68,11 @@ impl SubTest for TestCompile { )); } - Ok(()) + // Run final code through filecheck. + let mut text = String::new(); + write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))) + .map_err(|e| e.to_string())?; + run_filecheck(&text, context) } } From 8ed37e352e064a5f8211a18f222873dbeb15f9e4 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Wed, 22 Nov 2017 12:25:01 -0800 Subject: [PATCH 1394/3084] Add x86_push and x86_pop instructions. --- lib/cretonne/meta/base/formats.py | 2 ++ lib/cretonne/meta/isa/intel/instructions.py | 13 +++++++++++++ lib/cretonne/src/ir/instructions.rs | 1 + lib/cretonne/src/verifier/mod.rs | 3 ++- lib/cretonne/src/write.rs | 1 + lib/reader/src/parser.rs | 1 + 6 files changed, 20 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 4e1ff3500a..1961d9d4a9 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -32,6 +32,8 @@ Ternary = InstructionFormat(VALUE, VALUE, VALUE, typevar_operand=1) # operands. MultiAry = InstructionFormat(VARIABLE_ARGS) +NullAry = InstructionFormat() + InsertLane = InstructionFormat(VALUE, ('lane', uimm8), VALUE) ExtractLane = InstructionFormat(VALUE, ('lane', uimm8)) diff --git a/lib/cretonne/meta/isa/intel/instructions.py b/lib/cretonne/meta/isa/intel/instructions.py index 27a0063557..3389f77708 100644 --- a/lib/cretonne/meta/isa/intel/instructions.py +++ b/lib/cretonne/meta/isa/intel/instructions.py @@ -99,4 +99,17 @@ fmax = Instruction( """, ins=(x, y), outs=a) +WideInt = TypeVar( + 'WideInt', 'An integer type with 16 to 64 bits', + ints=(16, 64)) +x = Operand('x', WideInt) + +push = Instruction( + 'x86_push', "Pushes onto the stack.", + ins=x, can_store=True, other_side_effects=True) + +pop = Instruction( + 'x86_pop', "Pops from the stack.", + outs=x, can_load=True, other_side_effects=True) + GROUP.close() diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index f110cd14db..c474258f37 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -120,6 +120,7 @@ pub enum InstructionData { }, Ternary { opcode: Opcode, args: [Value; 3] }, MultiAry { opcode: Opcode, args: ValueList }, + NullAry { opcode: Opcode }, InsertLane { opcode: Opcode, lane: Uimm8, diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 2144264f40..6b67ed1bed 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -359,7 +359,8 @@ impl<'a> Verifier<'a> { Store { .. } | RegMove { .. } | Trap { .. } | - CondTrap { .. } => {} + CondTrap { .. } | + NullAry { .. } => {} } Ok(()) diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 9de3e93d58..1604ca559a 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -295,6 +295,7 @@ pub fn write_operands( write!(w, " {}", DisplayValues(args.as_slice(pool))) } } + NullAry { .. } => write!(w, " "), InsertLane { lane, args, .. } => write!(w, " {}, {}, {}", args[0], lane, args[1]), ExtractLane { lane, arg, .. } => write!(w, " {}, {}", arg, lane), IntCompare { cond, args, .. } => write!(w, " {} {}, {}", cond, args[0], args[1]), diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index d809cd192c..b96cba20b2 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1955,6 +1955,7 @@ impl<'a> Parser<'a> { args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } } + InstructionFormat::NullAry => InstructionData::NullAry { opcode }, InstructionFormat::Jump => { // Parse the destination EBB number. Don't translate source to local numbers yet. let ebb_num = self.match_ebb("expected jump destination EBB")?; From cdf70ccb7733b80579f06d8dcdcb3a38d3a27ccc Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Wed, 22 Nov 2017 19:18:11 -0800 Subject: [PATCH 1395/3084] Add copy_special instruction. --- lib/cretonne/meta/base/formats.py | 1 + lib/cretonne/meta/base/instructions.py | 7 +++++++ lib/cretonne/src/ir/instructions.rs | 5 +++++ lib/cretonne/src/verifier/mod.rs | 1 + lib/cretonne/src/write.rs | 13 +++++++++++++ lib/reader/src/parser.rs | 9 +++++++++ 6 files changed, 36 insertions(+) diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 1961d9d4a9..539d87561b 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -64,6 +64,7 @@ StackStore = InstructionFormat(VALUE, stack_slot, offset32) HeapAddr = InstructionFormat(heap, VALUE, uimm32) RegMove = InstructionFormat(VALUE, ('src', regunit), ('dst', regunit)) +CopySpecial = InstructionFormat(('src', regunit), ('dst', regunit)) RegSpill = InstructionFormat( VALUE, ('src', regunit), ('dst', entities.stack_slot)) RegFill = InstructionFormat( diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 26a6f66077..195f255046 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -537,6 +537,13 @@ regmove = Instruction( ins=(x, src, dst), other_side_effects=True) +copy_special = Instruction( + 'copy_special', r""" + Copies a value from one special register to another. e.g. rbp -> rsp. + """, + ins=(src, dst), + other_side_effects=True) + regspill = Instruction( 'regspill', r""" Temporarily divert ``x`` from ``src`` to ``SS``. diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index c474258f37..37ca23150c 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -236,6 +236,11 @@ pub enum InstructionData { src: RegUnit, dst: RegUnit, }, + CopySpecial { + opcode: Opcode, + src: RegUnit, + dst: RegUnit, + }, RegSpill { opcode: Opcode, arg: Value, diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 6b67ed1bed..259cf48a5e 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -358,6 +358,7 @@ impl<'a> Verifier<'a> { Load { .. } | Store { .. } | RegMove { .. } | + CopySpecial { .. } | Trap { .. } | CondTrap { .. } | NullAry { .. } => {} diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 1604ca559a..8c8c280ca9 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -394,6 +394,19 @@ pub fn write_operands( write!(w, " {}, %{} -> %{}", arg, src, dst) } } + CopySpecial { src, dst, .. } => { + if let Some(isa) = isa { + let regs = isa.register_info(); + write!( + w, + " {} -> {}", + regs.display_regunit(src), + regs.display_regunit(dst) + ) + } else { + write!(w, " %{} -> %{}", src, dst) + } + } RegSpill { arg, src, dst, .. } => { if let Some(isa) = isa { let regs = isa.register_info(); diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index b96cba20b2..b3827de62c 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -2270,6 +2270,15 @@ impl<'a> Parser<'a> { dst, } } + InstructionFormat::CopySpecial => { + let src = self.match_regunit(ctx.unique_isa)?; + self.match_token( + Token::Arrow, + "expected '->' between register units", + )?; + let dst = self.match_regunit(ctx.unique_isa)?; + InstructionData::CopySpecial { opcode, src, dst } + } InstructionFormat::RegSpill => { let arg = self.match_value("expected SSA value operand")?; self.match_token( From b8275f57130ad456208e7938379bea7fe595b67c Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Wed, 22 Nov 2017 19:18:51 -0800 Subject: [PATCH 1396/3084] Add (some) encodings for x86_push/pop instructions. Simple uses actually pass the legalizer now. --- lib/cretonne/meta/isa/intel/encodings.py | 4 ++++ lib/cretonne/meta/isa/intel/instructions.py | 7 +++---- lib/cretonne/meta/isa/intel/recipes.py | 14 +++++++++++++- lib/cretonne/src/legalizer/boundary.rs | 2 ++ 4 files changed, 22 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index f9c7f041c8..75378db853 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -227,6 +227,10 @@ enc_i32_i64(base.regfill, r.rfi32, 0x8b) enc_both(base.fill.b1, r.fiSib32, 0x8b) enc_both(base.regfill.b1, r.rfi32, 0x8b) +# Push and Pop +I64.enc(x86.push.i64, *r.pushq(0x50)) +I64.enc(x86.pop.i64, *r.popq(0x58)) + # # Float loads and stores. # diff --git a/lib/cretonne/meta/isa/intel/instructions.py b/lib/cretonne/meta/isa/intel/instructions.py index 3389f77708..82ff8ec90d 100644 --- a/lib/cretonne/meta/isa/intel/instructions.py +++ b/lib/cretonne/meta/isa/intel/instructions.py @@ -99,10 +99,9 @@ fmax = Instruction( """, ins=(x, y), outs=a) -WideInt = TypeVar( - 'WideInt', 'An integer type with 16 to 64 bits', - ints=(16, 64)) -x = Operand('x', WideInt) + +Int = TypeVar('Int', 'A scalar integer type', ints=True) +x = Operand('x', Int) push = Instruction( 'x86_push', "Pushes onto the stack.", diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 531e291630..a746ac7a38 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -5,7 +5,7 @@ from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt, IsEqual, Or from cdsl.registers import RegClass -from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry +from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry, NullAry from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare, FloatCompare, IntCond, FloatCond from base.formats import Jump, Branch, BranchInt, BranchFloat @@ -472,6 +472,18 @@ puiq = TailRecipe( sink.put8(imm as u64); ''') +pushq = TailRecipe( + 'pushq', Unary, size=0, ins=GPR, outs=(), + emit=''' + PUT_OP(bits | (in_reg0 & 7), rex1(in_reg0), sink); + ''') + +popq = TailRecipe( + 'popq', NullAry, size=0, ins=(), outs=GPR, + emit=''' + PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); + ''') + # XX+rd id with Abs4 function relocation. fnaddr4 = TailRecipe( 'fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index e9f7f4c564..ea81c1ff95 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -82,6 +82,8 @@ fn legalize_entry_params(func: &mut Function, entry: Ebb) { pos.func.dfg.attach_ebb_param(entry, arg); match abi_type.purpose { ArgumentPurpose::Normal => {} + ArgumentPurpose::FramePointer => {} + ArgumentPurpose::CalleeSaved => {} ArgumentPurpose::StructReturn => { assert!(!has_sret, "Multiple sret arguments found"); has_sret = true; From 32509ebacd4da5a480e60740d04c4b24b4284f2d Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Mon, 27 Nov 2017 13:30:11 -0800 Subject: [PATCH 1397/3084] Fix push/pop encoding for extended registers. Add copy_special encoding. --- lib/cretonne/meta/isa/intel/encodings.py | 5 +++++ lib/cretonne/meta/isa/intel/recipes.py | 12 +++++++++++- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 75378db853..0b1f33524f 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -228,9 +228,14 @@ enc_both(base.fill.b1, r.fiSib32, 0x8b) enc_both(base.regfill.b1, r.rfi32, 0x8b) # Push and Pop +I64.enc(x86.push.i64, *r.pushq.rex(0x50)) I64.enc(x86.push.i64, *r.pushq(0x50)) +I64.enc(x86.pop.i64, *r.popq.rex(0x58)) I64.enc(x86.pop.i64, *r.popq(0x58)) +# Copy Special +I64.enc(base.copy_special, *r.copysp.rex(0x89)) + # # Float loads and stores. # diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index a746ac7a38..71c280ff2a 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -10,7 +10,7 @@ from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare, FloatCompare, IntCond, FloatCond from base.formats import Jump, Branch, BranchInt, BranchFloat from base.formats import Ternary, FuncAddr, UnaryGlobalVar -from base.formats import RegMove, RegSpill, RegFill +from base.formats import RegMove, RegSpill, RegFill, CopySpecial from .registers import GPR, ABCD, FPR, GPR8, FPR8, FLAG, StackGPR32, StackFPR32 from .defs import supported_floatccs @@ -484,6 +484,16 @@ popq = TailRecipe( PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); ''') +# XX /r, for regmove instructions. +copysp = TailRecipe( + 'copysp', CopySpecial, size=1, ins=(), outs=(), + clobbers_flags=False, + emit=''' + PUT_OP(bits, rex2(dst, src), sink); + modrm_rr(dst, src, sink); + ''') + + # XX+rd id with Abs4 function relocation. fnaddr4 = TailRecipe( 'fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, From ffab87318ecbad9085d7c7184be9b5038b15e62b Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Mon, 27 Nov 2017 15:32:28 -0800 Subject: [PATCH 1398/3084] Add adjust_sp_imm instruction. Note: This enables using rsp and rbp as normal registers. Which is... wrong. --- lib/cretonne/meta/base/formats.py | 1 + lib/cretonne/meta/base/instructions.py | 8 ++++++++ lib/cretonne/meta/isa/intel/encodings.py | 5 ++++- lib/cretonne/meta/isa/intel/recipes.py | 11 ++++++++++- lib/cretonne/src/ir/instructions.rs | 1 + lib/cretonne/src/isa/intel/abi.rs | 4 ++-- lib/cretonne/src/verifier/mod.rs | 1 + lib/cretonne/src/write.rs | 1 + lib/reader/src/parser.rs | 4 ++++ 9 files changed, 32 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 539d87561b..df8539656b 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -69,6 +69,7 @@ RegSpill = InstructionFormat( VALUE, ('src', regunit), ('dst', entities.stack_slot)) RegFill = InstructionFormat( VALUE, ('src', entities.stack_slot), ('dst', regunit)) +AdjustSpImm = InstructionFormat(offset32) Trap = InstructionFormat(trapcode) CondTrap = InstructionFormat(VALUE, trapcode) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 195f255046..4eef992e99 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -544,6 +544,14 @@ copy_special = Instruction( ins=(src, dst), other_side_effects=True) +Offset = Operand('Offset', offset32, 'Offset from current stack pointer') +adjust_sp_imm = Instruction( + 'adjust_sp_imm', r""" + Adds an immediate offset value to the stack pointer register. + """, + ins=(Offset,), + other_side_effects=True) + regspill = Instruction( 'regspill', r""" Temporarily divert ``x`` from ``src`` to ``SS``. diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 0b1f33524f..c4b4a730e6 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -234,7 +234,10 @@ I64.enc(x86.pop.i64, *r.popq.rex(0x58)) I64.enc(x86.pop.i64, *r.popq(0x58)) # Copy Special -I64.enc(base.copy_special, *r.copysp.rex(0x89)) +I64.enc(base.copy_special, *r.copysp.rex(0x89, w=1)) + +# Adjust SP Imm +I64.enc(base.adjust_sp_imm, *r.adjustsp.rex(0x81, w=1)) # # Float loads and stores. diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 71c280ff2a..a478b1b54e 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -10,7 +10,7 @@ from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare, FloatCompare, IntCond, FloatCond from base.formats import Jump, Branch, BranchInt, BranchFloat from base.formats import Ternary, FuncAddr, UnaryGlobalVar -from base.formats import RegMove, RegSpill, RegFill, CopySpecial +from base.formats import RegMove, RegSpill, RegFill, CopySpecial, AdjustSpImm from .registers import GPR, ABCD, FPR, GPR8, FPR8, FLAG, StackGPR32, StackFPR32 from .defs import supported_floatccs @@ -493,6 +493,15 @@ copysp = TailRecipe( modrm_rr(dst, src, sink); ''') +adjustsp = TailRecipe( + 'adjustsp', AdjustSpImm, size=5, ins=(), outs=(), + emit=''' + PUT_OP(bits, rex1(4), sink); + modrm_r_bits(4, bits, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + ''') + # XX+rd id with Abs4 function relocation. fnaddr4 = TailRecipe( diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 37ca23150c..040d6cb98e 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -241,6 +241,7 @@ pub enum InstructionData { src: RegUnit, dst: RegUnit, }, + AdjustSpImm { opcode: Opcode, offset: Offset32 }, RegSpill { opcode: Opcode, arg: Value, diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index e8be53e848..0c45a0e223 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -140,8 +140,8 @@ pub fn allocatable_registers( flags: &shared_settings::Flags, ) -> AllocatableSet { let mut regs = AllocatableSet::new(); - regs.take(GPR, RU::rsp as RegUnit); - regs.take(GPR, RU::rbp as RegUnit); + //regs.take(GPR, RU::rsp as RegUnit); + //regs.take(GPR, RU::rbp as RegUnit); // 32-bit arch only has 8 registers. if !flags.is_64bit() { diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 259cf48a5e..57a94b6d0d 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -359,6 +359,7 @@ impl<'a> Verifier<'a> { Store { .. } | RegMove { .. } | CopySpecial { .. } | + AdjustSpImm { .. } | Trap { .. } | CondTrap { .. } | NullAry { .. } => {} diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 8c8c280ca9..ba951c4c84 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -407,6 +407,7 @@ pub fn write_operands( write!(w, " %{} -> %{}", src, dst) } } + AdjustSpImm { offset, .. } => write!(w, " {}", offset), RegSpill { arg, src, dst, .. } => { if let Some(isa) = isa { let regs = isa.register_info(); diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index b3827de62c..d567da7741 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -2279,6 +2279,10 @@ impl<'a> Parser<'a> { let dst = self.match_regunit(ctx.unique_isa)?; InstructionData::CopySpecial { opcode, src, dst } } + InstructionFormat::AdjustSpImm => { + let offset = self.optional_offset32()?; + InstructionData::AdjustSpImm { opcode, offset } + } InstructionFormat::RegSpill => { let arg = self.match_value("expected SSA value operand")?; self.match_token( From d4311d2b1d4b123312bd27eb7584acb83ba023f7 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Mon, 27 Nov 2017 15:38:38 -0800 Subject: [PATCH 1399/3084] Add prologue-epilogue test that exercises new instructions and binary emission. --- .../isa/intel/prologue-epilogue.cton | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 cranelift/filetests/isa/intel/prologue-epilogue.cton diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/intel/prologue-epilogue.cton new file mode 100644 index 0000000000..384451533a --- /dev/null +++ b/cranelift/filetests/isa/intel/prologue-epilogue.cton @@ -0,0 +1,25 @@ +test legalizer +test binemit +set is_64bit=1 +isa intel haswell + +function %foo(f64 [%xmm0], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12]) -> i64 csr [%r12] { + ss0 = local 168, offset 0 + ss1 = incoming_arg 32, offset -32 + +ebb0(v0: f64 [%xmm0], v1: i64 [%rbp], v2: i64 [%rbx], v3: i64 [%r12]): + x86_push v1 ; bin: 40 55 + copy_special %rsp -> %rbp ; bin: 48 89 e5 + x86_push v2 ; bin: 40 53 + x86_push v3 ; bin: 41 54 + adjust_sp_imm -168 ; bin: 48 81 c4 ffffff58 + + ; ... function body ... + + + adjust_sp_imm +168 ; bin: 48 81 c4 000000a8 + [-,%r12] v100 = x86_pop.i64 ; bin: 41 5c + [-,%rbx] v101 = x86_pop.i64 ; bin: 40 5b + [-,%rbp] v102 = x86_pop.i64 ; bin: 40 5d + return v102 +} \ No newline at end of file From fdfe24760a16ef5a6cb7248d4e39c5d08798d929 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Mon, 27 Nov 2017 23:35:55 -0800 Subject: [PATCH 1400/3084] Add missing newline to prologue epilogue test --- cranelift/filetests/isa/intel/prologue-epilogue.cton | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/intel/prologue-epilogue.cton index 384451533a..e08fec1823 100644 --- a/cranelift/filetests/isa/intel/prologue-epilogue.cton +++ b/cranelift/filetests/isa/intel/prologue-epilogue.cton @@ -22,4 +22,4 @@ ebb0(v0: f64 [%xmm0], v1: i64 [%rbp], v2: i64 [%rbx], v3: i64 [%r12]): [-,%rbx] v101 = x86_pop.i64 ; bin: 40 5b [-,%rbp] v102 = x86_pop.i64 ; bin: 40 5d return v102 -} \ No newline at end of file +} From 60b6bc3ab726672bf5ddf0cdadb5a3cf8b790d87 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Tue, 28 Nov 2017 17:25:39 -0800 Subject: [PATCH 1401/3084] Sketch of prologue generation --- lib/cretonne/src/isa/intel/mod.rs | 47 ++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index a8b6314521..48106cce20 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -10,9 +10,14 @@ use binemit::{CodeSink, MemoryCodeSink, emit_function}; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo, RegUnit}; +use self::registers::RU; use ir; use regalloc; +use result; +use ir::InstBuilder; +use legalizer; + #[allow(dead_code)] struct Isa { @@ -111,4 +116,44 @@ impl TargetIsa for Isa { fn reloc_names(&self) -> &'static [&'static str] { &binemit::RELOC_NAMES } + + fn prologue_epilogue(&self, func: &mut ir::Function) -> result::CtonResult { + use stack_layout::layout_stack; + use cursor::{Cursor, EncCursor}; + + let word_size = if self.flags().is_64bit() { 8 } else { 4 }; + let stack_size = layout_stack(&mut func.stack_slots, word_size)?; + + // Append frame pointer to function signature + let rbp_arg = ir::AbiParam::special_reg( + ir::types::I64, + ir::ArgumentPurpose::FramePointer, + RU::rbp as RegUnit, + ); + func.signature.params.push(rbp_arg); + + // Append param to entry EBB + let entry_ebb = func.layout.entry_block().expect("missing entry block"); + func.dfg.append_ebb_param(entry_ebb, ir::types::I64); + + // Find our frame pointer parameter Value + let fp = func.special_param(ir::ArgumentPurpose::FramePointer) + .expect("missing frame pointer"); + + // Assign it a location + func.locations[fp] = ir::ValueLoc::Reg(RU::rbp as RegUnit); + + // Insert prologue + let mut pos = EncCursor::new(func, self).at_first_insertion_point(entry_ebb); + pos.ins().x86_push(fp); + pos.ins().copy_special( + RU::rbp as RegUnit, + RU::rsp as RegUnit, + ); + pos.ins().adjust_sp_imm(-(stack_size as i32)); + + //legalizer::legalize_function(func, &mut func.cfg, self); + + Ok(()) + } } From c1a64a5dc7affbf557f04c18d2e87dcdfc3c46ba Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Tue, 28 Nov 2017 19:51:19 -0800 Subject: [PATCH 1402/3084] Insert a basic epilogue. --- lib/cretonne/src/isa/intel/mod.rs | 59 ++++++++++++++++++++++++------- 1 file changed, 46 insertions(+), 13 deletions(-) diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 48106cce20..60982b438f 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -15,8 +15,9 @@ use self::registers::RU; use ir; use regalloc; use result; -use ir::InstBuilder; -use legalizer; +use ir::{InstBuilder, InstructionData, Opcode}; +use stack_layout::layout_stack; +use cursor::{Cursor, EncCursor}; #[allow(dead_code)] @@ -118,10 +119,10 @@ impl TargetIsa for Isa { } fn prologue_epilogue(&self, func: &mut ir::Function) -> result::CtonResult { - use stack_layout::layout_stack; - use cursor::{Cursor, EncCursor}; - let word_size = if self.flags().is_64bit() { 8 } else { 4 }; + + // TODO: Insert stack slot for FP and CSRs we will push + let stack_size = layout_stack(&mut func.stack_slots, word_size)?; // Append frame pointer to function signature @@ -131,6 +132,7 @@ impl TargetIsa for Isa { RU::rbp as RegUnit, ); func.signature.params.push(rbp_arg); + func.signature.returns.push(rbp_arg); // Append param to entry EBB let entry_ebb = func.layout.entry_block().expect("missing entry block"); @@ -144,16 +146,47 @@ impl TargetIsa for Isa { func.locations[fp] = ir::ValueLoc::Reg(RU::rbp as RegUnit); // Insert prologue - let mut pos = EncCursor::new(func, self).at_first_insertion_point(entry_ebb); - pos.ins().x86_push(fp); - pos.ins().copy_special( - RU::rbp as RegUnit, - RU::rsp as RegUnit, - ); - pos.ins().adjust_sp_imm(-(stack_size as i32)); + { + let mut pos = EncCursor::new(func, self).at_first_insertion_point(entry_ebb); + pos.ins().x86_push(fp); + pos.ins().copy_special( + RU::rbp as RegUnit, + RU::rsp as RegUnit, + ); + pos.ins().adjust_sp_imm(-(stack_size as i32)); + } - //legalizer::legalize_function(func, &mut func.cfg, self); + let mut return_insts = Vec::new(); + + for ebb in func.layout.ebbs() { + for inst in func.layout.ebb_insts(ebb) { + if let InstructionData::MultiAry { opcode, .. } = func.dfg[inst] { + if opcode == Opcode::Return { + return_insts.push(inst); + } + } + + } + } + + for inst in return_insts { + let fp_ret = self.insert_epilogue(inst, stack_size as i32, func); + func.dfg.append_inst_arg(inst, fp_ret); + } Ok(()) } } + +impl Isa { + fn insert_epilogue( + &self, + inst: ir::Inst, + stack_size: i32, + func: &mut ir::Function, + ) -> ir::Value { + let mut pos = EncCursor::new(func, self).at_inst(inst); + pos.ins().adjust_sp_imm(stack_size); + pos.ins().x86_pop(ir::types::I64) + } +} From 544c148b25926af010518b706af4f23e721e98e6 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Tue, 28 Nov 2017 20:09:27 -0800 Subject: [PATCH 1403/3084] Fix the stack slots, so the locations of local var slots are correct. --- lib/cretonne/src/isa/intel/mod.rs | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 60982b438f..d96fbf1cde 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -122,8 +122,17 @@ impl TargetIsa for Isa { let word_size = if self.flags().is_64bit() { 8 } else { 4 }; // TODO: Insert stack slot for FP and CSRs we will push + let meta_stack_size = word_size; // TODO: this needs to change depending on CSRs + let stack_offset = -(meta_stack_size as i32); + let slot = ir::StackSlotData { + kind: ir::StackSlotKind::IncomingArg, + size: meta_stack_size, + offset: stack_offset, + }; + func.create_stack_slot(slot); - let stack_size = layout_stack(&mut func.stack_slots, word_size)?; + let total_stack_size = layout_stack(&mut func.stack_slots, word_size)?; + let local_stack_size = total_stack_size - meta_stack_size; // Append frame pointer to function signature let rbp_arg = ir::AbiParam::special_reg( @@ -153,7 +162,7 @@ impl TargetIsa for Isa { RU::rbp as RegUnit, RU::rsp as RegUnit, ); - pos.ins().adjust_sp_imm(-(stack_size as i32)); + pos.ins().adjust_sp_imm(-(local_stack_size as i32)); } let mut return_insts = Vec::new(); @@ -170,7 +179,7 @@ impl TargetIsa for Isa { } for inst in return_insts { - let fp_ret = self.insert_epilogue(inst, stack_size as i32, func); + let fp_ret = self.insert_epilogue(inst, local_stack_size as i32, func); func.dfg.append_inst_arg(inst, fp_ret); } From d12587f2189ebefedc10d29557ed24d90d8fb01b Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Wed, 29 Nov 2017 02:28:18 -0800 Subject: [PATCH 1404/3084] Assign explicit register location to result of frame-pointer pop. --- lib/cretonne/src/isa/intel/mod.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index d96fbf1cde..c6dfeb937a 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -180,6 +180,7 @@ impl TargetIsa for Isa { for inst in return_insts { let fp_ret = self.insert_epilogue(inst, local_stack_size as i32, func); + func.locations[fp_ret] = ir::ValueLoc::Reg(RU::rbp as RegUnit); func.dfg.append_inst_arg(inst, fp_ret); } From e0c8ab49e15215a8c917845f6a1b1648e97c4fc9 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Wed, 29 Nov 2017 02:32:20 -0800 Subject: [PATCH 1405/3084] Only emit an adjust_sp_imm if the stack_size is non-zero. --- lib/cretonne/src/isa/intel/mod.rs | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index c6dfeb937a..cfca41083a 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -162,11 +162,13 @@ impl TargetIsa for Isa { RU::rbp as RegUnit, RU::rsp as RegUnit, ); - pos.ins().adjust_sp_imm(-(local_stack_size as i32)); + if local_stack_size > 0 { + pos.ins().adjust_sp_imm(-(local_stack_size as i32)); + } } + // Find all 'return' instructions let mut return_insts = Vec::new(); - for ebb in func.layout.ebbs() { for inst in func.layout.ebb_insts(ebb) { if let InstructionData::MultiAry { opcode, .. } = func.dfg[inst] { @@ -178,6 +180,7 @@ impl TargetIsa for Isa { } } + // Insert an epilogue directly before every 'return' for inst in return_insts { let fp_ret = self.insert_epilogue(inst, local_stack_size as i32, func); func.locations[fp_ret] = ir::ValueLoc::Reg(RU::rbp as RegUnit); @@ -196,7 +199,9 @@ impl Isa { func: &mut ir::Function, ) -> ir::Value { let mut pos = EncCursor::new(func, self).at_inst(inst); - pos.ins().adjust_sp_imm(stack_size); + if stack_size > 0 { + pos.ins().adjust_sp_imm(stack_size); + } pos.ins().x86_pop(ir::types::I64) } } From b049916d353980a1fea5779ffc9a5826a9c4fe09 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Wed, 29 Nov 2017 03:16:37 -0800 Subject: [PATCH 1406/3084] Move %rsp to %rbp, not the reverse. This also takes away %rsp and %rbp from regalloc again. This may cause tests to fail temporarily. --- lib/cretonne/src/isa/intel/abi.rs | 4 ++-- lib/cretonne/src/isa/intel/mod.rs | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 0c45a0e223..e8be53e848 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -140,8 +140,8 @@ pub fn allocatable_registers( flags: &shared_settings::Flags, ) -> AllocatableSet { let mut regs = AllocatableSet::new(); - //regs.take(GPR, RU::rsp as RegUnit); - //regs.take(GPR, RU::rbp as RegUnit); + regs.take(GPR, RU::rsp as RegUnit); + regs.take(GPR, RU::rbp as RegUnit); // 32-bit arch only has 8 registers. if !flags.is_64bit() { diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index cfca41083a..662faddba1 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -159,8 +159,8 @@ impl TargetIsa for Isa { let mut pos = EncCursor::new(func, self).at_first_insertion_point(entry_ebb); pos.ins().x86_push(fp); pos.ins().copy_special( - RU::rbp as RegUnit, RU::rsp as RegUnit, + RU::rbp as RegUnit, ); if local_stack_size > 0 { pos.ins().adjust_sp_imm(-(local_stack_size as i32)); From f31a764fc8d1ac386fab16b5436ec71b6f09cb1a Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Wed, 29 Nov 2017 16:34:21 -0800 Subject: [PATCH 1407/3084] Save all callee-saved registers in prologue. --- lib/cretonne/src/isa/intel/abi.rs | 3 +++ lib/cretonne/src/isa/intel/mod.rs | 45 +++++++++++++++++++++++++++---- 2 files changed, 43 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index e8be53e848..2aa83703cd 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -15,6 +15,9 @@ static ARG_GPRS: [RU; 6] = [RU::rdi, RU::rsi, RU::rdx, RU::rcx, RU::r8, RU::r9]; /// Return value registers. static RET_GPRS: [RU; 3] = [RU::rax, RU::rdx, RU::rcx]; +/// Callee-saved registers +pub static CSR_GPRS: [RU; 5] = [RU::rbx, RU::r12, RU::r13, RU::r14, RU::r15]; + struct Args { pointer_bytes: u32, pointer_bits: u16, diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 662faddba1..50575d28ad 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -121,18 +121,21 @@ impl TargetIsa for Isa { fn prologue_epilogue(&self, func: &mut ir::Function) -> result::CtonResult { let word_size = if self.flags().is_64bit() { 8 } else { 4 }; - // TODO: Insert stack slot for FP and CSRs we will push - let meta_stack_size = word_size; // TODO: this needs to change depending on CSRs - let stack_offset = -(meta_stack_size as i32); + let mut csr_stack_size = word_size; // Size of RBP to start with + for _reg in abi::CSR_GPRS.iter() { + csr_stack_size += word_size; + } + + let stack_offset = -(csr_stack_size as i32); let slot = ir::StackSlotData { kind: ir::StackSlotKind::IncomingArg, - size: meta_stack_size, + size: csr_stack_size, offset: stack_offset, }; func.create_stack_slot(slot); let total_stack_size = layout_stack(&mut func.stack_slots, word_size)?; - let local_stack_size = total_stack_size - meta_stack_size; + let local_stack_size = total_stack_size - csr_stack_size; // Append frame pointer to function signature let rbp_arg = ir::AbiParam::special_reg( @@ -143,6 +146,16 @@ impl TargetIsa for Isa { func.signature.params.push(rbp_arg); func.signature.returns.push(rbp_arg); + for reg in abi::CSR_GPRS.iter() { + let csr_arg = ir::AbiParam::special_reg( + ir::types::I64, + ir::ArgumentPurpose::CalleeSaved, + *reg as RegUnit, + ); + func.signature.params.push(csr_arg); + func.signature.returns.push(csr_arg); + } + // Append param to entry EBB let entry_ebb = func.layout.entry_block().expect("missing entry block"); func.dfg.append_ebb_param(entry_ebb, ir::types::I64); @@ -154,6 +167,23 @@ impl TargetIsa for Isa { // Assign it a location func.locations[fp] = ir::ValueLoc::Reg(RU::rbp as RegUnit); + let mut csr_vals = Vec::new(); + for reg in abi::CSR_GPRS.iter() { + // Append param to entry EBB + func.dfg.append_ebb_param(entry_ebb, ir::types::I64); + + let csr_arg = func.dfg.ebb_params(entry_ebb).last().expect( + "no last argument", + ); + + // Assign it a location + func.locations[*csr_arg] = ir::ValueLoc::Reg(*reg as RegUnit); + + // Remember it so we can push it momentarily + csr_vals.push(*csr_arg); + } + + // Insert prologue { let mut pos = EncCursor::new(func, self).at_first_insertion_point(entry_ebb); @@ -165,6 +195,10 @@ impl TargetIsa for Isa { if local_stack_size > 0 { pos.ins().adjust_sp_imm(-(local_stack_size as i32)); } + + for csr_arg in csr_vals { + pos.ins().x86_push(csr_arg); + } } // Find all 'return' instructions @@ -187,6 +221,7 @@ impl TargetIsa for Isa { func.dfg.append_inst_arg(inst, fp_ret); } + Ok(()) } } From cf9d287beaeea82d577dce8c48eff70a1513c2ce Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Wed, 29 Nov 2017 17:43:06 -0800 Subject: [PATCH 1408/3084] Pop the callee-saved registers in the function epilogue(s). --- lib/cretonne/src/isa/intel/mod.rs | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 50575d28ad..0d63a15220 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -216,9 +216,7 @@ impl TargetIsa for Isa { // Insert an epilogue directly before every 'return' for inst in return_insts { - let fp_ret = self.insert_epilogue(inst, local_stack_size as i32, func); - func.locations[fp_ret] = ir::ValueLoc::Reg(RU::rbp as RegUnit); - func.dfg.append_inst_arg(inst, fp_ret); + self.insert_epilogue(inst, local_stack_size as i32, func); } @@ -227,16 +225,24 @@ impl TargetIsa for Isa { } impl Isa { - fn insert_epilogue( - &self, - inst: ir::Inst, - stack_size: i32, - func: &mut ir::Function, - ) -> ir::Value { + fn insert_epilogue(&self, inst: ir::Inst, stack_size: i32, func: &mut ir::Function) { + let mut return_values = Vec::new(); + let mut pos = EncCursor::new(func, self).at_inst(inst); if stack_size > 0 { pos.ins().adjust_sp_imm(stack_size); } - pos.ins().x86_pop(ir::types::I64) + for reg in abi::CSR_GPRS.iter().rev() { + let csr_ret = pos.ins().x86_pop(ir::types::I64); + return_values.push((csr_ret, *reg)); + } + let fp_ret = pos.ins().x86_pop(ir::types::I64); + return_values.push((fp_ret, RU::rbp)); + + let func = pos.func; + for (val, reg) in return_values { + func.locations[val] = ir::ValueLoc::Reg(reg as RegUnit); + func.dfg.append_inst_arg(inst, val); + } } } From c92d49963a69a56981f63767dbc8dfd0d2cdf405 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Wed, 29 Nov 2017 21:06:55 -0800 Subject: [PATCH 1409/3084] Simplify x86_(push|pop) encodings. --- lib/cretonne/meta/isa/intel/encodings.py | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index c4b4a730e6..a806267e7b 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -228,10 +228,8 @@ enc_both(base.fill.b1, r.fiSib32, 0x8b) enc_both(base.regfill.b1, r.rfi32, 0x8b) # Push and Pop -I64.enc(x86.push.i64, *r.pushq.rex(0x50)) -I64.enc(x86.push.i64, *r.pushq(0x50)) -I64.enc(x86.pop.i64, *r.popq.rex(0x58)) -I64.enc(x86.pop.i64, *r.popq(0x58)) +enc_i64(x86.push.i64, r.pushq, 0x50) +enc_i64(x86.pop.i64, r.popq, 0x58) # Copy Special I64.enc(base.copy_special, *r.copysp.rex(0x89, w=1)) From e6481bb4ebdbd415beef326f05b258e6802dba70 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Thu, 30 Nov 2017 18:44:00 -0800 Subject: [PATCH 1410/3084] Add 32-bit encodings for x86_push, x86_pop, copy_special, and adjust_sp_imm. --- lib/cretonne/meta/isa/intel/encodings.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index a806267e7b..0e92bf4597 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -228,13 +228,15 @@ enc_both(base.fill.b1, r.fiSib32, 0x8b) enc_both(base.regfill.b1, r.rfi32, 0x8b) # Push and Pop -enc_i64(x86.push.i64, r.pushq, 0x50) -enc_i64(x86.pop.i64, r.popq, 0x58) +enc_i32_i64(x86.push, r.pushq, 0x50) +enc_i32_i64(x86.pop, r.popq, 0x58) # Copy Special -I64.enc(base.copy_special, *r.copysp.rex(0x89, w=1)) +enc_i64(base.copy_special, r.copysp, 0x89) +I32.enc(base.copy_special, *r.copysp(0x89)) # Adjust SP Imm +I32.enc(base.adjust_sp_imm, *r.adjustsp(0x81)) I64.enc(base.adjust_sp_imm, *r.adjustsp.rex(0x81, w=1)) # From daa7a21d056c4a29c1d8292d71b053176e252476 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Thu, 30 Nov 2017 18:50:56 -0800 Subject: [PATCH 1411/3084] Adjust prologue/epilogue generation to work with 32-bit Intel arch. --- lib/cretonne/src/isa/intel/abi.rs | 11 ++++++--- lib/cretonne/src/isa/intel/mod.rs | 37 +++++++++++++++++++++---------- 2 files changed, 33 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 2aa83703cd..0257c66586 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -15,9 +15,6 @@ static ARG_GPRS: [RU; 6] = [RU::rdi, RU::rsi, RU::rdx, RU::rcx, RU::r8, RU::r9]; /// Return value registers. static RET_GPRS: [RU; 3] = [RU::rax, RU::rdx, RU::rcx]; -/// Callee-saved registers -pub static CSR_GPRS: [RU; 5] = [RU::rbx, RU::r12, RU::r13, RU::r14, RU::r15]; - struct Args { pointer_bytes: u32, pointer_bits: u16, @@ -156,3 +153,11 @@ pub fn allocatable_registers( regs } + +pub fn callee_saved_registers(flags: &shared_settings::Flags) -> Vec { + if flags.is_64bit() { + return vec![RU::rbx, RU::r12, RU::r13, RU::r14, RU::r15]; + } else { + return vec![RU::rbx, RU::rsi, RU::rdi]; + } +} diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 0d63a15220..3e90acc933 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -120,9 +120,15 @@ impl TargetIsa for Isa { fn prologue_epilogue(&self, func: &mut ir::Function) -> result::CtonResult { let word_size = if self.flags().is_64bit() { 8 } else { 4 }; + let csr_type = if self.flags().is_64bit() { + ir::types::I64 + } else { + ir::types::I32 + }; + let csrs = abi::callee_saved_registers(&self.shared_flags); let mut csr_stack_size = word_size; // Size of RBP to start with - for _reg in abi::CSR_GPRS.iter() { + for _reg in &csrs { csr_stack_size += word_size; } @@ -139,16 +145,16 @@ impl TargetIsa for Isa { // Append frame pointer to function signature let rbp_arg = ir::AbiParam::special_reg( - ir::types::I64, + csr_type, ir::ArgumentPurpose::FramePointer, RU::rbp as RegUnit, ); func.signature.params.push(rbp_arg); func.signature.returns.push(rbp_arg); - for reg in abi::CSR_GPRS.iter() { + for reg in &csrs { let csr_arg = ir::AbiParam::special_reg( - ir::types::I64, + csr_type, ir::ArgumentPurpose::CalleeSaved, *reg as RegUnit, ); @@ -158,7 +164,7 @@ impl TargetIsa for Isa { // Append param to entry EBB let entry_ebb = func.layout.entry_block().expect("missing entry block"); - func.dfg.append_ebb_param(entry_ebb, ir::types::I64); + func.dfg.append_ebb_param(entry_ebb, csr_type); // Find our frame pointer parameter Value let fp = func.special_param(ir::ArgumentPurpose::FramePointer) @@ -168,9 +174,9 @@ impl TargetIsa for Isa { func.locations[fp] = ir::ValueLoc::Reg(RU::rbp as RegUnit); let mut csr_vals = Vec::new(); - for reg in abi::CSR_GPRS.iter() { + for reg in &csrs { // Append param to entry EBB - func.dfg.append_ebb_param(entry_ebb, ir::types::I64); + func.dfg.append_ebb_param(entry_ebb, csr_type); let csr_arg = func.dfg.ebb_params(entry_ebb).last().expect( "no last argument", @@ -216,7 +222,7 @@ impl TargetIsa for Isa { // Insert an epilogue directly before every 'return' for inst in return_insts { - self.insert_epilogue(inst, local_stack_size as i32, func); + self.insert_epilogue(inst, local_stack_size as i32, func, &csrs, csr_type); } @@ -225,18 +231,25 @@ impl TargetIsa for Isa { } impl Isa { - fn insert_epilogue(&self, inst: ir::Inst, stack_size: i32, func: &mut ir::Function) { + fn insert_epilogue( + &self, + inst: ir::Inst, + stack_size: i32, + func: &mut ir::Function, + csrs: &Vec, + csr_type: ir::types::Type, + ) { let mut return_values = Vec::new(); let mut pos = EncCursor::new(func, self).at_inst(inst); if stack_size > 0 { pos.ins().adjust_sp_imm(stack_size); } - for reg in abi::CSR_GPRS.iter().rev() { - let csr_ret = pos.ins().x86_pop(ir::types::I64); + for reg in csrs.iter().rev() { + let csr_ret = pos.ins().x86_pop(csr_type); return_values.push((csr_ret, *reg)); } - let fp_ret = pos.ins().x86_pop(ir::types::I64); + let fp_ret = pos.ins().x86_pop(csr_type); return_values.push((fp_ret, RU::rbp)); let func = pos.func; From 6ec4bfc4ca3288759af85bc60eea52f8e87445fd Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Thu, 30 Nov 2017 22:34:13 -0800 Subject: [PATCH 1412/3084] Fix up the encodings for new instructions, both expected and actual. Make the test more accurate. --- .../filetests/isa/intel/prologue-epilogue.cton | 16 ++++++++-------- lib/cretonne/meta/isa/intel/encodings.py | 2 +- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/intel/prologue-epilogue.cton index e08fec1823..773a47da0c 100644 --- a/cranelift/filetests/isa/intel/prologue-epilogue.cton +++ b/cranelift/filetests/isa/intel/prologue-epilogue.cton @@ -3,23 +3,23 @@ test binemit set is_64bit=1 isa intel haswell -function %foo(f64 [%xmm0], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12]) -> i64 csr [%r12] { +function %foo(f64 [%xmm0], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12]) -> i64 csr [%r12], i64 csr [%rbx], i64 fp [%rbp] { ss0 = local 168, offset 0 ss1 = incoming_arg 32, offset -32 ebb0(v0: f64 [%xmm0], v1: i64 [%rbp], v2: i64 [%rbx], v3: i64 [%r12]): - x86_push v1 ; bin: 40 55 + x86_push v1 ; bin: 48 55 copy_special %rsp -> %rbp ; bin: 48 89 e5 - x86_push v2 ; bin: 40 53 - x86_push v3 ; bin: 41 54 + x86_push v2 ; bin: 48 53 + x86_push v3 ; bin: 49 54 adjust_sp_imm -168 ; bin: 48 81 c4 ffffff58 ; ... function body ... adjust_sp_imm +168 ; bin: 48 81 c4 000000a8 - [-,%r12] v100 = x86_pop.i64 ; bin: 41 5c - [-,%rbx] v101 = x86_pop.i64 ; bin: 40 5b - [-,%rbp] v102 = x86_pop.i64 ; bin: 40 5d - return v102 + [-,%r12] v100 = x86_pop.i64 ; bin: 49 5c + [-,%rbx] v101 = x86_pop.i64 ; bin: 48 5b + [-,%rbp] v102 = x86_pop.i64 ; bin: 48 5d + return v100, v101, v102 } diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 0e92bf4597..fd75fef2a2 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -232,7 +232,7 @@ enc_i32_i64(x86.push, r.pushq, 0x50) enc_i32_i64(x86.pop, r.popq, 0x58) # Copy Special -enc_i64(base.copy_special, r.copysp, 0x89) +enc_i64(base.copy_special, r.copysp, 0x89, w=1) I32.enc(base.copy_special, *r.copysp(0x89)) # Adjust SP Imm From 2f3edc1bc601fd7a1996febee9a096a21dbb2eb5 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Thu, 30 Nov 2017 22:44:38 -0800 Subject: [PATCH 1413/3084] Fix issue in which CSR returns were incorrectly ordered. --- lib/cretonne/src/isa/intel/mod.rs | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 3e90acc933..a5a3954478 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -143,23 +143,27 @@ impl TargetIsa for Isa { let total_stack_size = layout_stack(&mut func.stack_slots, word_size)?; let local_stack_size = total_stack_size - csr_stack_size; - // Append frame pointer to function signature - let rbp_arg = ir::AbiParam::special_reg( + // Build up list of args, which we'll append forwards to the params and + // backwards to the returns. + let mut csr_args = Vec::new(); + csr_args.push(ir::AbiParam::special_reg( csr_type, ir::ArgumentPurpose::FramePointer, RU::rbp as RegUnit, - ); - func.signature.params.push(rbp_arg); - func.signature.returns.push(rbp_arg); - + )); for reg in &csrs { - let csr_arg = ir::AbiParam::special_reg( + csr_args.push(ir::AbiParam::special_reg( csr_type, ir::ArgumentPurpose::CalleeSaved, *reg as RegUnit, - ); - func.signature.params.push(csr_arg); - func.signature.returns.push(csr_arg); + )); + } + + for csr_arg in &csr_args { + func.signature.params.push(*csr_arg); + } + for csr_arg in csr_args.iter().rev() { + func.signature.returns.push(*csr_arg); } // Append param to entry EBB From 4eb9a54096ae4f4423307731d1caa19e68aa37a5 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Fri, 1 Dec 2017 16:55:26 -0800 Subject: [PATCH 1414/3084] Convert x86_(push|pop) operations to be explicitly limited to 32-bit and 64-bit values. --- lib/cretonne/meta/isa/intel/instructions.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/instructions.py b/lib/cretonne/meta/isa/intel/instructions.py index 82ff8ec90d..42dd8074bd 100644 --- a/lib/cretonne/meta/isa/intel/instructions.py +++ b/lib/cretonne/meta/isa/intel/instructions.py @@ -100,8 +100,7 @@ fmax = Instruction( ins=(x, y), outs=a) -Int = TypeVar('Int', 'A scalar integer type', ints=True) -x = Operand('x', Int) +x = Operand('x', iWord) push = Instruction( 'x86_push', "Pushes onto the stack.", From 3b1b33e0ac376499f85031dc535aec1d0aa4d10c Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Fri, 1 Dec 2017 18:01:00 -0800 Subject: [PATCH 1415/3084] Add docs and tests for copy_special instruction. Fixes encoding issue that tests revealed. --- cranelift/filetests/isa/intel/binary32.cton | 7 +++++++ cranelift/filetests/isa/intel/binary64.cton | 11 +++++++++++ .../filetests/isa/intel/prologue-epilogue.cton | 4 ++-- cranelift/filetests/parser/tiny.cton | 14 ++++++++++++++ lib/cretonne/meta/base/instructions.py | 7 ++++++- lib/cretonne/meta/isa/intel/encodings.py | 2 +- 6 files changed, 41 insertions(+), 4 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 81274ab91f..a4834baaab 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -140,6 +140,13 @@ ebb0: ; asm: movl %ecx, %esi [-,%rsi] v81 = copy v1 ; bin: 89 ce + ; Copy Special + ; asm: movl %esp, %ebp + copy_special %rsp -> %rbp ; bin: 89 e5 + ; asm: movl %ebp, %esp + copy_special %rbp -> %rsp ; bin: 89 ec + + ; Load/Store instructions. ; Register indirect addressing with no displacement. diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index f695d1b227..8d57b231e6 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -155,6 +155,17 @@ ebb0: ; asm: movq %rcx, %r10 [-,%r10] v112 = copy v1 ; bin: 49 89 ca + ; Copy Special + ; asm: movq %rsp, %rbp + copy_special %rsp -> %rbp ; bin: 48 89 e5 + ; asm: movq %r10, %r11 + copy_special %r10 -> %r11 ; bin: 4d 89 d3 + ; asm: movq %rsp, %r11 + copy_special %rsp -> %r11 ; bin: 49 89 e3 + ; asm: movq %r10, %rsp + copy_special %r10 -> %rsp ; bin: 4c 89 d4 + + ; Load/Store instructions. ; Register indirect addressing with no displacement. diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/intel/prologue-epilogue.cton index 773a47da0c..10b6e9cf93 100644 --- a/cranelift/filetests/isa/intel/prologue-epilogue.cton +++ b/cranelift/filetests/isa/intel/prologue-epilogue.cton @@ -1,6 +1,6 @@ -test legalizer test binemit -set is_64bit=1 +set is_64bit +set is_compressed isa intel haswell function %foo(f64 [%xmm0], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12]) -> i64 csr [%r12], i64 csr [%rbx], i64 fp [%rbp] { diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 13513a2b6c..20c4bbe2b3 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -186,3 +186,17 @@ ebb0(v1: i32): ; nextln: regfill $v1, $ss0 -> %10 ; nextln: return ; nextln: } + +; Register copies. +function %copy_special() { +ebb0: + copy_special %10 -> %20 + copy_special %20 -> %10 + return +} +; sameln: function %copy_special() native { +; nextln: ebb0: +; nextln: copy_special %10 -> %20 +; nextln: copy_special %20 -> %10 +; nextln: return +; nextln: } diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 4eef992e99..bbfb9ddbe8 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -539,7 +539,12 @@ regmove = Instruction( copy_special = Instruction( 'copy_special', r""" - Copies a value from one special register to another. e.g. rbp -> rsp. + Copies the contents of ''src'' register to ''dst'' register. + + This instructions copies the contents of one register to another + register without involving any SSA values. This is used for copying + special registers, e.g. copying the stack register to the frame + register in a function prologue. """, ins=(src, dst), other_side_effects=True) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index fd75fef2a2..42f125c93f 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -232,7 +232,7 @@ enc_i32_i64(x86.push, r.pushq, 0x50) enc_i32_i64(x86.pop, r.popq, 0x58) # Copy Special -enc_i64(base.copy_special, r.copysp, 0x89, w=1) +I64.enc(base.copy_special, *r.copysp.rex(0x89, w=1)) I32.enc(base.copy_special, *r.copysp(0x89)) # Adjust SP Imm From 1a11c351b5cf9ed21bb637a4f11e37b747babba8 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Fri, 1 Dec 2017 19:15:31 -0800 Subject: [PATCH 1416/3084] Add tests and documentation for x86_(push|pop). Fix up encoding issues revealed by tests. --- cranelift/filetests/isa/intel/binary32.cton | 6 ++++++ cranelift/filetests/isa/intel/binary64.cton | 11 ++++++++++- lib/cretonne/meta/isa/intel/encodings.py | 7 +++++-- lib/cretonne/meta/isa/intel/instructions.py | 19 +++++++++++++++++-- 4 files changed, 38 insertions(+), 5 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index a4834baaab..a2f287bd4c 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -383,6 +383,12 @@ ebb0: ; asm: movl 1032(%esp), %ecx regfill v1, ss1 -> %rcx ; bin: 8b 8c 24 00000408 + ; Push and Pop + ; asm: pushl %ecx + x86_push v1 ; bin: 51 + ; asm: popl %ecx + [-,%rcx] v512 = x86_pop.i32 ; bin: 59 + ; asm: testl %ecx, %ecx ; asm: je ebb1 brz v1, ebb1 ; bin: 85 c9 74 0e diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 8d57b231e6..26e2de02dc 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -165,7 +165,6 @@ ebb0: ; asm: movq %r10, %rsp copy_special %r10 -> %rsp ; bin: 4c 89 d4 - ; Load/Store instructions. ; Register indirect addressing with no displacement. @@ -484,6 +483,16 @@ ebb0: ; asm: movq 1032(%rsp), %rcx regfill v1, ss1 -> %rcx ; bin: 48 8b 8c 24 00000408 + ; Push and Pop + ; asm: pushq %rcx + x86_push v1 ; bin: 51 + ; asm: pushq %r10 + x86_push v3 ; bin: 41 52 + ; asm: popq %rcx + [-,%rcx] v513 = x86_pop.i64 ; bin: 59 + ; asm: popq %r10 + [-,%r10] v514 = x86_pop.i64 ; bin: 41 5a + ; asm: testq %rcx, %rcx ; asm: je ebb1 brz v1, ebb1 ; bin: 48 85 c9 74 1b diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 42f125c93f..fed3d10c17 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -228,8 +228,11 @@ enc_both(base.fill.b1, r.fiSib32, 0x8b) enc_both(base.regfill.b1, r.rfi32, 0x8b) # Push and Pop -enc_i32_i64(x86.push, r.pushq, 0x50) -enc_i32_i64(x86.pop, r.popq, 0x58) +I32.enc(x86.push.i32, *r.pushq(0x50)) +enc_i64(x86.push.i64, r.pushq, 0x50) + +I32.enc(x86.pop.i32, *r.popq(0x58)) +enc_i64(x86.pop.i64, r.popq, 0x58) # Copy Special I64.enc(base.copy_special, *r.copysp.rex(0x89, w=1)) diff --git a/lib/cretonne/meta/isa/intel/instructions.py b/lib/cretonne/meta/isa/intel/instructions.py index 42dd8074bd..0852d49c73 100644 --- a/lib/cretonne/meta/isa/intel/instructions.py +++ b/lib/cretonne/meta/isa/intel/instructions.py @@ -103,11 +103,26 @@ fmax = Instruction( x = Operand('x', iWord) push = Instruction( - 'x86_push', "Pushes onto the stack.", + 'x86_push', r""" + Pushes a value onto the stack. + + Decrements the stack pointer and stores the specified value on to the top. + + This is polymorphic in i32 and i64. However, it is only implemented for i64 + in 64-bit mode, and only for i32 in 32-bit mode. + """, ins=x, can_store=True, other_side_effects=True) pop = Instruction( - 'x86_pop', "Pops from the stack.", + 'x86_pop', r""" + Pops a value from the stack. + + Loads a value from the top of the stack and then increments the stack + pointer. + + This is polymorphic in i32 and i64. However, it is only implemented for i64 + in 64-bit mode, and only for i32 in 32-bit mode. + """, outs=x, can_load=True, other_side_effects=True) GROUP.close() From ced39f5186e5d2b3c5867c26b95537cf99b758a3 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Sat, 2 Dec 2017 15:37:04 -0800 Subject: [PATCH 1417/3084] Fix up adjust_sp_imm instruction. * Use imm64 rather than offset32 * Add predicate to enforce signed 32-bit limit to imm * Remove AdjustSpImm format * Add encoding tests for adjust_sp_imm * Adjust use of adjust_sp_imm in Intel prologue_epilogue to match --- cranelift/filetests/isa/intel/binary32.cton | 11 +++++++++++ cranelift/filetests/isa/intel/binary64.cton | 10 ++++++++++ .../filetests/isa/intel/prologue-epilogue.cton | 14 +++++++------- lib/cretonne/meta/base/formats.py | 1 - lib/cretonne/meta/base/instructions.py | 10 +++++++--- lib/cretonne/meta/isa/intel/recipes.py | 9 +++++---- lib/cretonne/src/ir/instructions.rs | 1 - lib/cretonne/src/isa/intel/mod.rs | 11 ++++++----- lib/cretonne/src/verifier/mod.rs | 1 - lib/cretonne/src/write.rs | 1 - lib/reader/src/parser.rs | 4 ---- 11 files changed, 46 insertions(+), 27 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index a2f287bd4c..41d364c63d 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -389,6 +389,17 @@ ebb0: ; asm: popl %ecx [-,%rcx] v512 = x86_pop.i32 ; bin: 59 + ; Adjust Stack Pointer + ; asm: addq $1024, %rsp + adjust_sp_imm 1024 ; bin: 81 c4 00000400 + ; asm: addq $-1024, %rsp + adjust_sp_imm -1024 ; bin: 81 c4 fffffc00 + ; asm: addq $2147483647, %rsp + adjust_sp_imm 2147483647 ; bin: 81 c4 7fffffff + ; asm: addq $-2147483648, %rsp + adjust_sp_imm -2147483648 ; bin: 81 c4 80000000 + + ; asm: testl %ecx, %ecx ; asm: je ebb1 brz v1, ebb1 ; bin: 85 c9 74 0e diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 26e2de02dc..6af8f91a89 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -493,6 +493,16 @@ ebb0: ; asm: popq %r10 [-,%r10] v514 = x86_pop.i64 ; bin: 41 5a + ; Adjust Stack Pointer + ; asm: addq $1024, %rsp + adjust_sp_imm 1024 ; bin: 48 81 c4 00000400 + ; asm: addq $-1024, %rsp + adjust_sp_imm -1024 ; bin: 48 81 c4 fffffc00 + ; asm: addq $2147483647, %rsp + adjust_sp_imm 2147483647 ; bin: 48 81 c4 7fffffff + ; asm: addq $-2147483648, %rsp + adjust_sp_imm -2147483648 ; bin: 48 81 c4 80000000 + ; asm: testq %rcx, %rcx ; asm: je ebb1 brz v1, ebb1 ; bin: 48 85 c9 74 1b diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/intel/prologue-epilogue.cton index 10b6e9cf93..75eb62f1d4 100644 --- a/cranelift/filetests/isa/intel/prologue-epilogue.cton +++ b/cranelift/filetests/isa/intel/prologue-epilogue.cton @@ -8,18 +8,18 @@ function %foo(f64 [%xmm0], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12]) -> i64 ss1 = incoming_arg 32, offset -32 ebb0(v0: f64 [%xmm0], v1: i64 [%rbp], v2: i64 [%rbx], v3: i64 [%r12]): - x86_push v1 ; bin: 48 55 + x86_push v1 ; bin: 55 copy_special %rsp -> %rbp ; bin: 48 89 e5 - x86_push v2 ; bin: 48 53 - x86_push v3 ; bin: 49 54 + x86_push v2 ; bin: 53 + x86_push v3 ; bin: 41 54 adjust_sp_imm -168 ; bin: 48 81 c4 ffffff58 ; ... function body ... - adjust_sp_imm +168 ; bin: 48 81 c4 000000a8 - [-,%r12] v100 = x86_pop.i64 ; bin: 49 5c - [-,%rbx] v101 = x86_pop.i64 ; bin: 48 5b - [-,%rbp] v102 = x86_pop.i64 ; bin: 48 5d + adjust_sp_imm 168 ; bin: 48 81 c4 000000a8 + [-,%r12] v100 = x86_pop.i64 ; bin: 41 5c + [-,%rbx] v101 = x86_pop.i64 ; bin: 5b + [-,%rbp] v102 = x86_pop.i64 ; bin: 5d return v100, v101, v102 } diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index df8539656b..539d87561b 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -69,7 +69,6 @@ RegSpill = InstructionFormat( VALUE, ('src', regunit), ('dst', entities.stack_slot)) RegFill = InstructionFormat( VALUE, ('src', entities.stack_slot), ('dst', regunit)) -AdjustSpImm = InstructionFormat(offset32) Trap = InstructionFormat(trapcode) CondTrap = InstructionFormat(VALUE, trapcode) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index bbfb9ddbe8..6a41725ff9 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -549,12 +549,16 @@ copy_special = Instruction( ins=(src, dst), other_side_effects=True) -Offset = Operand('Offset', offset32, 'Offset from current stack pointer') +StackOffset = Operand('Offset', imm64, 'Offset from current stack pointer') adjust_sp_imm = Instruction( 'adjust_sp_imm', r""" - Adds an immediate offset value to the stack pointer register. + Adds ``Offset`` immediate offset value to the stack pointer register. + + This instruction is used to adjust the stack pointer, primarily in function + prologues and epilogues. ``Offset`` is constrained to the size of a signed + 32-bit integer. """, - ins=(Offset,), + ins=(StackOffset,), other_side_effects=True) regspill = Instruction( diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index a478b1b54e..be15c3053d 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -10,7 +10,7 @@ from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare, FloatCompare, IntCond, FloatCond from base.formats import Jump, Branch, BranchInt, BranchFloat from base.formats import Ternary, FuncAddr, UnaryGlobalVar -from base.formats import RegMove, RegSpill, RegFill, CopySpecial, AdjustSpImm +from base.formats import RegMove, RegSpill, RegFill, CopySpecial from .registers import GPR, ABCD, FPR, GPR8, FPR8, FLAG, StackGPR32, StackFPR32 from .defs import supported_floatccs @@ -494,12 +494,13 @@ copysp = TailRecipe( ''') adjustsp = TailRecipe( - 'adjustsp', AdjustSpImm, size=5, ins=(), outs=(), + 'adjustsp', UnaryImm, size=5, ins=(), outs=(), + instp=IsSignedInt(UnaryImm.imm, 32), emit=''' PUT_OP(bits, rex1(4), sink); modrm_r_bits(4, bits, sink); - let offset: i32 = offset.into(); - sink.put4(offset as u32); + let imm: i64 = imm.into(); + sink.put4(imm as u32); ''') diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 040d6cb98e..37ca23150c 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -241,7 +241,6 @@ pub enum InstructionData { src: RegUnit, dst: RegUnit, }, - AdjustSpImm { opcode: Opcode, offset: Offset32 }, RegSpill { opcode: Opcode, arg: Value, diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index a5a3954478..78bec9dbfb 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -16,6 +16,7 @@ use ir; use regalloc; use result; use ir::{InstBuilder, InstructionData, Opcode}; +use ir::immediates::Imm64; use stack_layout::layout_stack; use cursor::{Cursor, EncCursor}; @@ -141,7 +142,7 @@ impl TargetIsa for Isa { func.create_stack_slot(slot); let total_stack_size = layout_stack(&mut func.stack_slots, word_size)?; - let local_stack_size = total_stack_size - csr_stack_size; + let local_stack_size = (total_stack_size - csr_stack_size) as i64; // Build up list of args, which we'll append forwards to the params and // backwards to the returns. @@ -203,7 +204,7 @@ impl TargetIsa for Isa { RU::rbp as RegUnit, ); if local_stack_size > 0 { - pos.ins().adjust_sp_imm(-(local_stack_size as i32)); + pos.ins().adjust_sp_imm(Imm64::new(-local_stack_size)); } for csr_arg in csr_vals { @@ -226,7 +227,7 @@ impl TargetIsa for Isa { // Insert an epilogue directly before every 'return' for inst in return_insts { - self.insert_epilogue(inst, local_stack_size as i32, func, &csrs, csr_type); + self.insert_epilogue(inst, local_stack_size, func, &csrs, csr_type); } @@ -238,7 +239,7 @@ impl Isa { fn insert_epilogue( &self, inst: ir::Inst, - stack_size: i32, + stack_size: i64, func: &mut ir::Function, csrs: &Vec, csr_type: ir::types::Type, @@ -247,7 +248,7 @@ impl Isa { let mut pos = EncCursor::new(func, self).at_inst(inst); if stack_size > 0 { - pos.ins().adjust_sp_imm(stack_size); + pos.ins().adjust_sp_imm(Imm64::new(stack_size)); } for reg in csrs.iter().rev() { let csr_ret = pos.ins().x86_pop(csr_type); diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 57a94b6d0d..259cf48a5e 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -359,7 +359,6 @@ impl<'a> Verifier<'a> { Store { .. } | RegMove { .. } | CopySpecial { .. } | - AdjustSpImm { .. } | Trap { .. } | CondTrap { .. } | NullAry { .. } => {} diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index ba951c4c84..8c8c280ca9 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -407,7 +407,6 @@ pub fn write_operands( write!(w, " %{} -> %{}", src, dst) } } - AdjustSpImm { offset, .. } => write!(w, " {}", offset), RegSpill { arg, src, dst, .. } => { if let Some(isa) = isa { let regs = isa.register_info(); diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index d567da7741..b3827de62c 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -2279,10 +2279,6 @@ impl<'a> Parser<'a> { let dst = self.match_regunit(ctx.unique_isa)?; InstructionData::CopySpecial { opcode, src, dst } } - InstructionFormat::AdjustSpImm => { - let offset = self.optional_offset32()?; - InstructionData::AdjustSpImm { opcode, offset } - } InstructionFormat::RegSpill => { let arg = self.match_value("expected SSA value operand")?; self.match_token( From a26d438b3045e9e67199f228f485d47f1a3109cd Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Sat, 2 Dec 2017 15:43:41 -0800 Subject: [PATCH 1418/3084] Use returned Value from append_ebb_param in prologue_epilogue. --- lib/cretonne/src/isa/intel/mod.rs | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 78bec9dbfb..d39905c9bb 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -169,11 +169,7 @@ impl TargetIsa for Isa { // Append param to entry EBB let entry_ebb = func.layout.entry_block().expect("missing entry block"); - func.dfg.append_ebb_param(entry_ebb, csr_type); - - // Find our frame pointer parameter Value - let fp = func.special_param(ir::ArgumentPurpose::FramePointer) - .expect("missing frame pointer"); + let fp = func.dfg.append_ebb_param(entry_ebb, csr_type); // Assign it a location func.locations[fp] = ir::ValueLoc::Reg(RU::rbp as RegUnit); @@ -181,17 +177,13 @@ impl TargetIsa for Isa { let mut csr_vals = Vec::new(); for reg in &csrs { // Append param to entry EBB - func.dfg.append_ebb_param(entry_ebb, csr_type); - - let csr_arg = func.dfg.ebb_params(entry_ebb).last().expect( - "no last argument", - ); + let csr_arg = func.dfg.append_ebb_param(entry_ebb, csr_type); // Assign it a location - func.locations[*csr_arg] = ir::ValueLoc::Reg(*reg as RegUnit); + func.locations[csr_arg] = ir::ValueLoc::Reg(*reg as RegUnit); // Remember it so we can push it momentarily - csr_vals.push(*csr_arg); + csr_vals.push(csr_arg); } From 66eccb785995bdc2f2f4f1297a9e288b0e76fb44 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Sat, 2 Dec 2017 15:45:47 -0800 Subject: [PATCH 1419/3084] Use opcode's is_return() rather than pattern-matching. --- lib/cretonne/src/isa/intel/mod.rs | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index d39905c9bb..100621bacd 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -208,12 +208,9 @@ impl TargetIsa for Isa { let mut return_insts = Vec::new(); for ebb in func.layout.ebbs() { for inst in func.layout.ebb_insts(ebb) { - if let InstructionData::MultiAry { opcode, .. } = func.dfg[inst] { - if opcode == Opcode::Return { - return_insts.push(inst); - } + if func.dfg[inst].opcode().is_return() { + return_insts.push(inst); } - } } From c78a191294518c9c2e3a9152f7933efad9d309e7 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Sat, 2 Dec 2017 17:20:42 -0800 Subject: [PATCH 1420/3084] Use layout.last_inst to find 'return' opcodes, rather than iterating. --- lib/cretonne/src/isa/intel/mod.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 100621bacd..6446c755db 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -15,7 +15,7 @@ use self::registers::RU; use ir; use regalloc; use result; -use ir::{InstBuilder, InstructionData, Opcode}; +use ir::InstBuilder; use ir::immediates::Imm64; use stack_layout::layout_stack; use cursor::{Cursor, EncCursor}; @@ -207,7 +207,7 @@ impl TargetIsa for Isa { // Find all 'return' instructions let mut return_insts = Vec::new(); for ebb in func.layout.ebbs() { - for inst in func.layout.ebb_insts(ebb) { + if let Some(inst) = func.layout.last_inst(ebb) { if func.dfg[inst].opcode().is_return() { return_insts.push(inst); } From c156eb9ff72e571bf167f11baba3fca918a0a03d Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Sun, 3 Dec 2017 23:25:32 -0800 Subject: [PATCH 1421/3084] Refactor prologue_epilogue. Break out into functions. Remove Vecs. --- lib/cretonne/src/isa/intel/abi.rs | 6 +- lib/cretonne/src/isa/intel/mod.rs | 161 +++++++++++++----------------- 2 files changed, 74 insertions(+), 93 deletions(-) diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 0257c66586..fdb0d0fe17 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -154,10 +154,10 @@ pub fn allocatable_registers( regs } -pub fn callee_saved_registers(flags: &shared_settings::Flags) -> Vec { +pub fn callee_saved_registers(flags: &shared_settings::Flags) -> &'static [RU] { if flags.is_64bit() { - return vec![RU::rbx, RU::r12, RU::r13, RU::r14, RU::r15]; + &[RU::rbx, RU::r12, RU::r13, RU::r14, RU::r15] } else { - return vec![RU::rbx, RU::rsi, RU::rdi]; + &[RU::rbx, RU::rsi, RU::rdi] } } diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 6446c755db..3abad93aa3 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -127,129 +127,110 @@ impl TargetIsa for Isa { ir::types::I32 }; let csrs = abi::callee_saved_registers(&self.shared_flags); + let csr_stack_size = ((csrs.len() + 1) * word_size as usize) as i32; - let mut csr_stack_size = word_size; // Size of RBP to start with - for _reg in &csrs { - csr_stack_size += word_size; - } - - let stack_offset = -(csr_stack_size as i32); - let slot = ir::StackSlotData { + func.create_stack_slot(ir::StackSlotData { kind: ir::StackSlotKind::IncomingArg, - size: csr_stack_size, - offset: stack_offset, - }; - func.create_stack_slot(slot); + size: csr_stack_size as u32, + offset: -csr_stack_size, + }); - let total_stack_size = layout_stack(&mut func.stack_slots, word_size)?; + let total_stack_size = layout_stack(&mut func.stack_slots, word_size)? as i32; let local_stack_size = (total_stack_size - csr_stack_size) as i64; - // Build up list of args, which we'll append forwards to the params and - // backwards to the returns. - let mut csr_args = Vec::new(); - csr_args.push(ir::AbiParam::special_reg( + // Add CSRs to function signature + let fp_arg = ir::AbiParam::special_reg( csr_type, ir::ArgumentPurpose::FramePointer, RU::rbp as RegUnit, - )); - for reg in &csrs { - csr_args.push(ir::AbiParam::special_reg( + ); + func.signature.params.push(fp_arg); + func.signature.returns.push(fp_arg); + + for csr in csrs.iter() { + let csr_arg = ir::AbiParam::special_reg( csr_type, ir::ArgumentPurpose::CalleeSaved, - *reg as RegUnit, - )); - } - - for csr_arg in &csr_args { - func.signature.params.push(*csr_arg); - } - for csr_arg in csr_args.iter().rev() { - func.signature.returns.push(*csr_arg); - } - - // Append param to entry EBB - let entry_ebb = func.layout.entry_block().expect("missing entry block"); - let fp = func.dfg.append_ebb_param(entry_ebb, csr_type); - - // Assign it a location - func.locations[fp] = ir::ValueLoc::Reg(RU::rbp as RegUnit); - - let mut csr_vals = Vec::new(); - for reg in &csrs { - // Append param to entry EBB - let csr_arg = func.dfg.append_ebb_param(entry_ebb, csr_type); - - // Assign it a location - func.locations[csr_arg] = ir::ValueLoc::Reg(*reg as RegUnit); - - // Remember it so we can push it momentarily - csr_vals.push(csr_arg); - } - - - // Insert prologue - { - let mut pos = EncCursor::new(func, self).at_first_insertion_point(entry_ebb); - pos.ins().x86_push(fp); - pos.ins().copy_special( - RU::rsp as RegUnit, - RU::rbp as RegUnit, + *csr as RegUnit, ); - if local_stack_size > 0 { - pos.ins().adjust_sp_imm(Imm64::new(-local_stack_size)); - } - - for csr_arg in csr_vals { - pos.ins().x86_push(csr_arg); - } + func.signature.params.push(csr_arg); + func.signature.returns.push(csr_arg); } - // Find all 'return' instructions - let mut return_insts = Vec::new(); - for ebb in func.layout.ebbs() { - if let Some(inst) = func.layout.last_inst(ebb) { - if func.dfg[inst].opcode().is_return() { - return_insts.push(inst); - } - } - } - // Insert an epilogue directly before every 'return' - for inst in return_insts { - self.insert_epilogue(inst, local_stack_size, func, &csrs, csr_type); - } + let entry_ebb = func.layout.entry_block().expect("missing entry block"); + let mut pos = EncCursor::new(func, self).at_first_insertion_point(entry_ebb); + self.insert_prologue(&mut pos, local_stack_size, csr_type); + self.insert_epilogues(&mut pos, local_stack_size, csr_type); Ok(()) } } impl Isa { + fn insert_prologue(&self, pos: &mut EncCursor, stack_size: i64, csr_type: ir::types::Type) { + // Append param to entry EBB + let ebb = pos.current_ebb().expect("missing ebb under cursor"); + let fp = pos.func.dfg.append_ebb_param(ebb, csr_type); + pos.func.locations[fp] = ir::ValueLoc::Reg(RU::rbp as RegUnit); + + pos.ins().x86_push(fp); + pos.ins().copy_special( + RU::rsp as RegUnit, + RU::rbp as RegUnit, + ); + + if stack_size > 0 { + pos.ins().adjust_sp_imm(Imm64::new(-stack_size)); + } + + let csrs = abi::callee_saved_registers(&self.shared_flags); + for reg in csrs.iter() { + // Append param to entry EBB + let csr_arg = pos.func.dfg.append_ebb_param(ebb, csr_type); + + // Assign it a location + pos.func.locations[csr_arg] = ir::ValueLoc::Reg(*reg as RegUnit); + + // Remember it so we can push it momentarily + pos.ins().x86_push(csr_arg); + } + } + + fn insert_epilogues(&self, pos: &mut EncCursor, stack_size: i64, csr_type: ir::types::Type) { + while let Some(ebb) = pos.next_ebb() { + pos.goto_last_inst(ebb); + if let Some(inst) = pos.current_inst() { + if pos.func.dfg[inst].opcode().is_return() { + self.insert_epilogue(inst, stack_size, pos, csr_type); + } + } + } + + } + fn insert_epilogue( &self, inst: ir::Inst, stack_size: i64, - func: &mut ir::Function, - csrs: &Vec, + pos: &mut EncCursor, csr_type: ir::types::Type, ) { - let mut return_values = Vec::new(); - - let mut pos = EncCursor::new(func, self).at_inst(inst); if stack_size > 0 { pos.ins().adjust_sp_imm(Imm64::new(stack_size)); } + + let csrs = abi::callee_saved_registers(&self.shared_flags); for reg in csrs.iter().rev() { let csr_ret = pos.ins().x86_pop(csr_type); - return_values.push((csr_ret, *reg)); + pos.func.locations[csr_ret] = ir::ValueLoc::Reg(*reg as RegUnit); + pos.func.dfg.append_inst_arg(inst, csr_ret); } - let fp_ret = pos.ins().x86_pop(csr_type); - return_values.push((fp_ret, RU::rbp)); - let func = pos.func; - for (val, reg) in return_values { - func.locations[val] = ir::ValueLoc::Reg(reg as RegUnit); - func.dfg.append_inst_arg(inst, val); - } + let fp_ret = pos.ins().x86_pop(csr_type); + pos.func.locations[fp_ret] = ir::ValueLoc::Reg(RU::rbp as RegUnit); + pos.func.dfg.append_inst_arg(inst, fp_ret); + } } From 0fb59dc5895edb86d9129d6e7a501d49e4e1d235 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Sun, 3 Dec 2017 23:40:26 -0800 Subject: [PATCH 1422/3084] Fix the ordering of return values. --- lib/cretonne/src/isa/intel/mod.rs | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 3abad93aa3..541f329726 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -221,16 +221,19 @@ impl Isa { pos.ins().adjust_sp_imm(Imm64::new(stack_size)); } - let csrs = abi::callee_saved_registers(&self.shared_flags); - for reg in csrs.iter().rev() { - let csr_ret = pos.ins().x86_pop(csr_type); - pos.func.locations[csr_ret] = ir::ValueLoc::Reg(*reg as RegUnit); - pos.func.dfg.append_inst_arg(inst, csr_ret); - } - let fp_ret = pos.ins().x86_pop(csr_type); + pos.prev_inst(); + pos.func.locations[fp_ret] = ir::ValueLoc::Reg(RU::rbp as RegUnit); pos.func.dfg.append_inst_arg(inst, fp_ret); + let csrs = abi::callee_saved_registers(&self.shared_flags); + for reg in csrs.iter() { + let csr_ret = pos.ins().x86_pop(csr_type); + pos.prev_inst(); + + pos.func.locations[csr_ret] = ir::ValueLoc::Reg(*reg as RegUnit); + pos.func.dfg.append_inst_arg(inst, csr_ret); + } } } From 694658b9494f8e6e3d764aa58e2441ba8b81fb73 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Mon, 4 Dec 2017 00:01:53 -0800 Subject: [PATCH 1423/3084] Move entirety of prologue_epilogue logic to abi module. --- lib/cretonne/src/isa/intel/abi.rs | 137 +++++++++++++++++++++++++++++- lib/cretonne/src/isa/intel/mod.rs | 123 +-------------------------- 2 files changed, 137 insertions(+), 123 deletions(-) diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index fdb0d0fe17..1a6aa0aeff 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -1,13 +1,18 @@ //! Intel ABI implementation. use ir; -use isa::{RegClass, RegUnit}; +use isa::{RegClass, RegUnit, TargetIsa}; use regalloc::AllocatableSet; use settings as shared_settings; use super::registers::{GPR, FPR, RU}; use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; -use ir::{AbiParam, ArgumentPurpose, ArgumentLoc, ArgumentExtension, CallConv}; +use ir::{AbiParam, ArgumentPurpose, ArgumentLoc, ArgumentExtension, CallConv, InstBuilder}; +use ir::immediates::Imm64; +use stack_layout::layout_stack; use std::i32; +use cursor::{Cursor, EncCursor}; +use result; + /// Argument registers for x86-64 static ARG_GPRS: [RU; 6] = [RU::rdi, RU::rsi, RU::rdx, RU::rcx, RU::r8, RU::r9]; @@ -154,6 +159,7 @@ pub fn allocatable_registers( regs } +/// Get the set of callee-saved registers. pub fn callee_saved_registers(flags: &shared_settings::Flags) -> &'static [RU] { if flags.is_64bit() { &[RU::rbx, RU::r12, RU::r13, RU::r14, RU::r15] @@ -161,3 +167,130 @@ pub fn callee_saved_registers(flags: &shared_settings::Flags) -> &'static [RU] { &[RU::rbx, RU::rsi, RU::rdi] } } + +/// Insert a System V-compatible prologue and epilogue. +pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { + let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; + let csr_type = if isa.flags().is_64bit() { + ir::types::I64 + } else { + ir::types::I32 + }; + let csrs = callee_saved_registers(isa.flags()); + let csr_stack_size = ((csrs.len() + 1) * word_size as usize) as i32; + + func.create_stack_slot(ir::StackSlotData { + kind: ir::StackSlotKind::IncomingArg, + size: csr_stack_size as u32, + offset: -csr_stack_size, + }); + + let total_stack_size = layout_stack(&mut func.stack_slots, word_size)? as i32; + let local_stack_size = (total_stack_size - csr_stack_size) as i64; + + // Add CSRs to function signature + let fp_arg = ir::AbiParam::special_reg( + csr_type, + ir::ArgumentPurpose::FramePointer, + RU::rbp as RegUnit, + ); + func.signature.params.push(fp_arg); + func.signature.returns.push(fp_arg); + + for csr in csrs.iter() { + let csr_arg = + ir::AbiParam::special_reg(csr_type, ir::ArgumentPurpose::CalleeSaved, *csr as RegUnit); + func.signature.params.push(csr_arg); + func.signature.returns.push(csr_arg); + } + + // Finally, insert the prologue and epilogues + let entry_ebb = func.layout.entry_block().expect("missing entry block"); + let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_ebb); + + insert_prologue(&mut pos, local_stack_size, csr_type, csrs); + insert_epilogues(&mut pos, local_stack_size, csr_type, csrs); + + Ok(()) +} + +/// Insert the prologue for a given function. +fn insert_prologue( + pos: &mut EncCursor, + stack_size: i64, + csr_type: ir::types::Type, + csrs: &'static [RU], +) { + // Append param to entry EBB + let ebb = pos.current_ebb().expect("missing ebb under cursor"); + let fp = pos.func.dfg.append_ebb_param(ebb, csr_type); + pos.func.locations[fp] = ir::ValueLoc::Reg(RU::rbp as RegUnit); + + pos.ins().x86_push(fp); + pos.ins().copy_special( + RU::rsp as RegUnit, + RU::rbp as RegUnit, + ); + + if stack_size > 0 { + pos.ins().adjust_sp_imm(Imm64::new(-stack_size)); + } + + for reg in csrs.iter() { + // Append param to entry EBB + let csr_arg = pos.func.dfg.append_ebb_param(ebb, csr_type); + + // Assign it a location + pos.func.locations[csr_arg] = ir::ValueLoc::Reg(*reg as RegUnit); + + // Remember it so we can push it momentarily + pos.ins().x86_push(csr_arg); + } +} + +/// Find all `return` instructions and insert epilogues before them. +fn insert_epilogues( + pos: &mut EncCursor, + stack_size: i64, + csr_type: ir::types::Type, + csrs: &'static [RU], +) { + while let Some(ebb) = pos.next_ebb() { + pos.goto_last_inst(ebb); + if let Some(inst) = pos.current_inst() { + if pos.func.dfg[inst].opcode().is_return() { + insert_epilogue(inst, stack_size, pos, csr_type, csrs); + } + } + } + +} + +/// Insert an epilogue given a specific `return` instruction. +fn insert_epilogue( + inst: ir::Inst, + stack_size: i64, + pos: &mut EncCursor, + csr_type: ir::types::Type, + csrs: &'static [RU], +) { + if stack_size > 0 { + pos.ins().adjust_sp_imm(Imm64::new(stack_size)); + } + + // Pop all the callee-saved registers, stepping backward each time to + // preserve the correct order. + let fp_ret = pos.ins().x86_pop(csr_type); + pos.prev_inst(); + + pos.func.locations[fp_ret] = ir::ValueLoc::Reg(RU::rbp as RegUnit); + pos.func.dfg.append_inst_arg(inst, fp_ret); + + for reg in csrs.iter() { + let csr_ret = pos.ins().x86_pop(csr_type); + pos.prev_inst(); + + pos.func.locations[csr_ret] = ir::ValueLoc::Reg(*reg as RegUnit); + pos.func.dfg.append_inst_arg(inst, csr_ret); + } +} diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 541f329726..96048c5fbd 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -10,15 +10,10 @@ use binemit::{CodeSink, MemoryCodeSink, emit_function}; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo, RegUnit}; -use self::registers::RU; +use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; use result; -use ir::InstBuilder; -use ir::immediates::Imm64; -use stack_layout::layout_stack; -use cursor::{Cursor, EncCursor}; #[allow(dead_code)] @@ -120,120 +115,6 @@ impl TargetIsa for Isa { } fn prologue_epilogue(&self, func: &mut ir::Function) -> result::CtonResult { - let word_size = if self.flags().is_64bit() { 8 } else { 4 }; - let csr_type = if self.flags().is_64bit() { - ir::types::I64 - } else { - ir::types::I32 - }; - let csrs = abi::callee_saved_registers(&self.shared_flags); - let csr_stack_size = ((csrs.len() + 1) * word_size as usize) as i32; - - func.create_stack_slot(ir::StackSlotData { - kind: ir::StackSlotKind::IncomingArg, - size: csr_stack_size as u32, - offset: -csr_stack_size, - }); - - let total_stack_size = layout_stack(&mut func.stack_slots, word_size)? as i32; - let local_stack_size = (total_stack_size - csr_stack_size) as i64; - - // Add CSRs to function signature - let fp_arg = ir::AbiParam::special_reg( - csr_type, - ir::ArgumentPurpose::FramePointer, - RU::rbp as RegUnit, - ); - func.signature.params.push(fp_arg); - func.signature.returns.push(fp_arg); - - for csr in csrs.iter() { - let csr_arg = ir::AbiParam::special_reg( - csr_type, - ir::ArgumentPurpose::CalleeSaved, - *csr as RegUnit, - ); - func.signature.params.push(csr_arg); - func.signature.returns.push(csr_arg); - } - - - let entry_ebb = func.layout.entry_block().expect("missing entry block"); - let mut pos = EncCursor::new(func, self).at_first_insertion_point(entry_ebb); - - self.insert_prologue(&mut pos, local_stack_size, csr_type); - self.insert_epilogues(&mut pos, local_stack_size, csr_type); - - Ok(()) - } -} - -impl Isa { - fn insert_prologue(&self, pos: &mut EncCursor, stack_size: i64, csr_type: ir::types::Type) { - // Append param to entry EBB - let ebb = pos.current_ebb().expect("missing ebb under cursor"); - let fp = pos.func.dfg.append_ebb_param(ebb, csr_type); - pos.func.locations[fp] = ir::ValueLoc::Reg(RU::rbp as RegUnit); - - pos.ins().x86_push(fp); - pos.ins().copy_special( - RU::rsp as RegUnit, - RU::rbp as RegUnit, - ); - - if stack_size > 0 { - pos.ins().adjust_sp_imm(Imm64::new(-stack_size)); - } - - let csrs = abi::callee_saved_registers(&self.shared_flags); - for reg in csrs.iter() { - // Append param to entry EBB - let csr_arg = pos.func.dfg.append_ebb_param(ebb, csr_type); - - // Assign it a location - pos.func.locations[csr_arg] = ir::ValueLoc::Reg(*reg as RegUnit); - - // Remember it so we can push it momentarily - pos.ins().x86_push(csr_arg); - } - } - - fn insert_epilogues(&self, pos: &mut EncCursor, stack_size: i64, csr_type: ir::types::Type) { - while let Some(ebb) = pos.next_ebb() { - pos.goto_last_inst(ebb); - if let Some(inst) = pos.current_inst() { - if pos.func.dfg[inst].opcode().is_return() { - self.insert_epilogue(inst, stack_size, pos, csr_type); - } - } - } - - } - - fn insert_epilogue( - &self, - inst: ir::Inst, - stack_size: i64, - pos: &mut EncCursor, - csr_type: ir::types::Type, - ) { - if stack_size > 0 { - pos.ins().adjust_sp_imm(Imm64::new(stack_size)); - } - - let fp_ret = pos.ins().x86_pop(csr_type); - pos.prev_inst(); - - pos.func.locations[fp_ret] = ir::ValueLoc::Reg(RU::rbp as RegUnit); - pos.func.dfg.append_inst_arg(inst, fp_ret); - - let csrs = abi::callee_saved_registers(&self.shared_flags); - for reg in csrs.iter() { - let csr_ret = pos.ins().x86_pop(csr_type); - pos.prev_inst(); - - pos.func.locations[csr_ret] = ir::ValueLoc::Reg(*reg as RegUnit); - pos.func.dfg.append_inst_arg(inst, csr_ret); - } + abi::prologue_epilogue(func, self) } } From ebcbd54f61c1e66c6dbfd083461ac25bd462811f Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Mon, 4 Dec 2017 11:15:26 -0800 Subject: [PATCH 1424/3084] Add 'compile' test and confirm the pro/epilogue is added. Fix regression this revealed. --- .../isa/intel/prologue-epilogue.cton | 43 ++++++++++--------- lib/cretonne/src/isa/intel/abi.rs | 9 ++-- 2 files changed, 28 insertions(+), 24 deletions(-) diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/intel/prologue-epilogue.cton index 75eb62f1d4..edbe90cbc5 100644 --- a/cranelift/filetests/isa/intel/prologue-epilogue.cton +++ b/cranelift/filetests/isa/intel/prologue-epilogue.cton @@ -1,25 +1,28 @@ -test binemit +test compile set is_64bit set is_compressed isa intel haswell -function %foo(f64 [%xmm0], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12]) -> i64 csr [%r12], i64 csr [%rbx], i64 fp [%rbp] { - ss0 = local 168, offset 0 - ss1 = incoming_arg 32, offset -32 - -ebb0(v0: f64 [%xmm0], v1: i64 [%rbp], v2: i64 [%rbx], v3: i64 [%r12]): - x86_push v1 ; bin: 55 - copy_special %rsp -> %rbp ; bin: 48 89 e5 - x86_push v2 ; bin: 53 - x86_push v3 ; bin: 41 54 - adjust_sp_imm -168 ; bin: 48 81 c4 ffffff58 - - ; ... function body ... - - - adjust_sp_imm 168 ; bin: 48 81 c4 000000a8 - [-,%r12] v100 = x86_pop.i64 ; bin: 41 5c - [-,%rbx] v101 = x86_pop.i64 ; bin: 5b - [-,%rbp] v102 = x86_pop.i64 ; bin: 5d - return v100, v101, v102 +function %foo() { +ebb0: + return } + +; check: function %foo(i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] native { +; nextln: ss0 = incoming_arg 48, offset -48 +; check: ebb0(v0: i64 [%rbp], v1: i64 [%rbx], v2: i64 [%r12], v3: i64 [%r13], v4: i64 [%r14], v5: i64 [%r15]): +; nextln: x86_push v0 +; nextln: copy_special %rsp -> %rbp +; nextln: x86_push v1 +; nextln: x86_push v2 +; nextln: x86_push v3 +; nextln: x86_push v4 +; nextln: x86_push v5 +; nextln: v11 = x86_pop.i64 +; nextln: v10 = x86_pop.i64 +; nextln: v9 = x86_pop.i64 +; nextln: v8 = x86_pop.i64 +; nextln: v7 = x86_pop.i64 +; nextln: v6 = x86_pop.i64 +; nextln: return v6, v7, v8, v9, v10, v11 +; nextln: } \ No newline at end of file diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 1a6aa0aeff..89c24787f6 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -10,7 +10,7 @@ use ir::{AbiParam, ArgumentPurpose, ArgumentLoc, ArgumentExtension, CallConv, In use ir::immediates::Imm64; use stack_layout::layout_stack; use std::i32; -use cursor::{Cursor, EncCursor}; +use cursor::{Cursor, EncCursor, CursorPosition}; use result; @@ -204,11 +204,13 @@ pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::Ct func.signature.returns.push(csr_arg); } - // Finally, insert the prologue and epilogues + // Set up the cursor and insert the prologue let entry_ebb = func.layout.entry_block().expect("missing entry block"); let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_ebb); - insert_prologue(&mut pos, local_stack_size, csr_type, csrs); + + // Reset the cursor and insert the epilogue + let mut pos = pos.at_position(CursorPosition::Nowhere); insert_epilogues(&mut pos, local_stack_size, csr_type, csrs); Ok(()) @@ -263,7 +265,6 @@ fn insert_epilogues( } } } - } /// Insert an epilogue given a specific `return` instruction. From a75248d2cfe77f64b72e893e2d9a242acbaaf75b Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Mon, 4 Dec 2017 16:03:04 -0800 Subject: [PATCH 1425/3084] Move the initial stack pointer adjustment to after the CSR pushes. --- cranelift/filetests/isa/intel/prologue-epilogue.cton | 6 +++++- lib/cretonne/src/isa/intel/abi.rs | 8 ++++---- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/intel/prologue-epilogue.cton index edbe90cbc5..df0bf28ff0 100644 --- a/cranelift/filetests/isa/intel/prologue-epilogue.cton +++ b/cranelift/filetests/isa/intel/prologue-epilogue.cton @@ -4,12 +4,14 @@ set is_compressed isa intel haswell function %foo() { + ss0 = local 168 ebb0: return } ; check: function %foo(i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] native { -; nextln: ss0 = incoming_arg 48, offset -48 +; nextln: ss0 = local 168, offset -216 +; nextln: ss1 = incoming_arg 48, offset -48 ; check: ebb0(v0: i64 [%rbp], v1: i64 [%rbx], v2: i64 [%r12], v3: i64 [%r13], v4: i64 [%r14], v5: i64 [%r15]): ; nextln: x86_push v0 ; nextln: copy_special %rsp -> %rbp @@ -18,6 +20,8 @@ ebb0: ; nextln: x86_push v3 ; nextln: x86_push v4 ; nextln: x86_push v5 +; nextln: adjust_sp_imm -168 +; nextln: adjust_sp_imm 168 ; nextln: v11 = x86_pop.i64 ; nextln: v10 = x86_pop.i64 ; nextln: v9 = x86_pop.i64 diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 89c24787f6..ad85a8e0af 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -234,10 +234,6 @@ fn insert_prologue( RU::rbp as RegUnit, ); - if stack_size > 0 { - pos.ins().adjust_sp_imm(Imm64::new(-stack_size)); - } - for reg in csrs.iter() { // Append param to entry EBB let csr_arg = pos.func.dfg.append_ebb_param(ebb, csr_type); @@ -248,6 +244,10 @@ fn insert_prologue( // Remember it so we can push it momentarily pos.ins().x86_push(csr_arg); } + + if stack_size > 0 { + pos.ins().adjust_sp_imm(Imm64::new(-stack_size)); + } } /// Find all `return` instructions and insert epilogues before them. From 5783ea2c9a95116a5c73e917ebe321b5eea513ab Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Mon, 4 Dec 2017 17:19:27 -0800 Subject: [PATCH 1426/3084] Account for return address when reserving stack space for CSRs. --- cranelift/filetests/isa/intel/prologue-epilogue.cton | 4 ++-- lib/cretonne/src/isa/intel/abi.rs | 9 ++++++++- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/intel/prologue-epilogue.cton index df0bf28ff0..425b790fbf 100644 --- a/cranelift/filetests/isa/intel/prologue-epilogue.cton +++ b/cranelift/filetests/isa/intel/prologue-epilogue.cton @@ -10,8 +10,8 @@ ebb0: } ; check: function %foo(i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] native { -; nextln: ss0 = local 168, offset -216 -; nextln: ss1 = incoming_arg 48, offset -48 +; nextln: ss0 = local 168, offset -224 +; nextln: ss1 = incoming_arg 56, offset -56 ; check: ebb0(v0: i64 [%rbp], v1: i64 [%rbx], v2: i64 [%r12], v3: i64 [%r13], v4: i64 [%r14], v5: i64 [%r15]): ; nextln: x86_push v0 ; nextln: copy_special %rsp -> %rbp diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index ad85a8e0af..733c2a3eb5 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -177,8 +177,15 @@ pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::Ct ir::types::I32 }; let csrs = callee_saved_registers(isa.flags()); - let csr_stack_size = ((csrs.len() + 1) * word_size as usize) as i32; + // The reserved stack area is composed of: + // return address + frame pointer + all callee-saved registers + // + // Pushing the return address is an implicit function of the `call` + // instruction. Each of the others we will then push explicitly. Then we + // will adjust the stack pointer to make room for the rest of the required + // space for this frame. + let csr_stack_size = ((csrs.len() + 2) * word_size as usize) as i32; func.create_stack_slot(ir::StackSlotData { kind: ir::StackSlotKind::IncomingArg, size: csr_stack_size as u32, From 3b937f59170b5b479b8616fe847aedf88f658869 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Tue, 5 Dec 2017 10:55:24 -0800 Subject: [PATCH 1427/3084] Add separate spiderwasm prologue/epilogue to intel's abi.rs --- lib/cretonne/src/isa/intel/abi.rs | 37 +++++++++++++++++++++++++------ 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 733c2a3eb5..965283d82f 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -7,6 +7,7 @@ use settings as shared_settings; use super::registers::{GPR, FPR, RU}; use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; use ir::{AbiParam, ArgumentPurpose, ArgumentLoc, ArgumentExtension, CallConv, InstBuilder}; +use ir::stackslot::{StackSize, StackOffset}; use ir::immediates::Imm64; use stack_layout::layout_stack; use std::i32; @@ -168,8 +169,30 @@ pub fn callee_saved_registers(flags: &shared_settings::Flags) -> &'static [RU] { } } -/// Insert a System V-compatible prologue and epilogue. pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { + match func.signature.call_conv { + ir::CallConv::Native => native_prologue_epilogue(func, isa), + ir::CallConv::SpiderWASM => spiderwasm_prologue_epilogue(func, isa), + } +} + +pub fn spiderwasm_prologue_epilogue( + func: &mut ir::Function, + isa: &TargetIsa, +) -> result::CtonResult { + let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; + let bytes = StackSize::from(isa.flags().spiderwasm_prologue_words()) * word_size; + + let mut ss = ir::StackSlotData::new(ir::StackSlotKind::IncomingArg, bytes); + ss.offset = -(bytes as StackOffset); + func.stack_slots.push(ss); + + layout_stack(&mut func.stack_slots, word_size)?; + Ok(()) +} + +/// Insert a System V-compatible prologue and epilogue. +pub fn native_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; let csr_type = if isa.flags().is_64bit() { ir::types::I64 @@ -214,17 +237,17 @@ pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::Ct // Set up the cursor and insert the prologue let entry_ebb = func.layout.entry_block().expect("missing entry block"); let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_ebb); - insert_prologue(&mut pos, local_stack_size, csr_type, csrs); + insert_native_prologue(&mut pos, local_stack_size, csr_type, csrs); // Reset the cursor and insert the epilogue let mut pos = pos.at_position(CursorPosition::Nowhere); - insert_epilogues(&mut pos, local_stack_size, csr_type, csrs); + insert_native_epilogues(&mut pos, local_stack_size, csr_type, csrs); Ok(()) } /// Insert the prologue for a given function. -fn insert_prologue( +fn insert_native_prologue( pos: &mut EncCursor, stack_size: i64, csr_type: ir::types::Type, @@ -258,7 +281,7 @@ fn insert_prologue( } /// Find all `return` instructions and insert epilogues before them. -fn insert_epilogues( +fn insert_native_epilogues( pos: &mut EncCursor, stack_size: i64, csr_type: ir::types::Type, @@ -268,14 +291,14 @@ fn insert_epilogues( pos.goto_last_inst(ebb); if let Some(inst) = pos.current_inst() { if pos.func.dfg[inst].opcode().is_return() { - insert_epilogue(inst, stack_size, pos, csr_type, csrs); + insert_native_epilogue(inst, stack_size, pos, csr_type, csrs); } } } } /// Insert an epilogue given a specific `return` instruction. -fn insert_epilogue( +fn insert_native_epilogue( inst: ir::Inst, stack_size: i64, pos: &mut EncCursor, From 7988d0c54cbad0ec0545706cbd781dee425a1eb3 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Tue, 5 Dec 2017 10:56:22 -0800 Subject: [PATCH 1428/3084] Add 8-bit variation of adjust_sp_imm for 32-bit and 64-bit Intel. --- cranelift/filetests/isa/intel/binary32.cton | 12 ++++++++---- cranelift/filetests/isa/intel/binary64.cton | 4 ++++ lib/cretonne/meta/isa/intel/encodings.py | 6 ++++-- lib/cretonne/meta/isa/intel/recipes.py | 14 ++++++++++++-- 4 files changed, 28 insertions(+), 8 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 41d364c63d..02e7fc0364 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -390,13 +390,17 @@ ebb0: [-,%rcx] v512 = x86_pop.i32 ; bin: 59 ; Adjust Stack Pointer - ; asm: addq $1024, %rsp + ; asm: addl $64, %esp + adjust_sp_imm 64 ; bin: 83 c4 40 + ; asm: addl $-64, %esp + adjust_sp_imm -64 ; bin: 83 c4 c0 + ; asm: addl $1024, %esp adjust_sp_imm 1024 ; bin: 81 c4 00000400 - ; asm: addq $-1024, %rsp + ; asm: addl $-1024, %esp adjust_sp_imm -1024 ; bin: 81 c4 fffffc00 - ; asm: addq $2147483647, %rsp + ; asm: addl $2147483647, %esp adjust_sp_imm 2147483647 ; bin: 81 c4 7fffffff - ; asm: addq $-2147483648, %rsp + ; asm: addl $-2147483648, %esp adjust_sp_imm -2147483648 ; bin: 81 c4 80000000 diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 6af8f91a89..719e6b1de5 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -494,6 +494,10 @@ ebb0: [-,%r10] v514 = x86_pop.i64 ; bin: 41 5a ; Adjust Stack Pointer + ; asm: addq $64, %rsp + adjust_sp_imm 64 ; bin: 48 83 c4 40 + ; asm: addq $-64, %rsp + adjust_sp_imm -64 ; bin: 48 83 c4 c0 ; asm: addq $1024, %rsp adjust_sp_imm 1024 ; bin: 48 81 c4 00000400 ; asm: addq $-1024, %rsp diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index fed3d10c17..52a56a514d 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -239,8 +239,10 @@ I64.enc(base.copy_special, *r.copysp.rex(0x89, w=1)) I32.enc(base.copy_special, *r.copysp(0x89)) # Adjust SP Imm -I32.enc(base.adjust_sp_imm, *r.adjustsp(0x81)) -I64.enc(base.adjust_sp_imm, *r.adjustsp.rex(0x81, w=1)) +I32.enc(base.adjust_sp_imm, *r.adjustsp8(0x83)) +I32.enc(base.adjust_sp_imm, *r.adjustsp32(0x81)) +I64.enc(base.adjust_sp_imm, *r.adjustsp8.rex(0x83, w=1)) +I64.enc(base.adjust_sp_imm, *r.adjustsp32.rex(0x81, w=1)) # # Float loads and stores. diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index be15c3053d..e4ae7c74c4 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -493,8 +493,18 @@ copysp = TailRecipe( modrm_rr(dst, src, sink); ''') -adjustsp = TailRecipe( - 'adjustsp', UnaryImm, size=5, ins=(), outs=(), +adjustsp8 = TailRecipe( + 'adjustsp8', UnaryImm, size=2, ins=(), outs=(), + instp=IsSignedInt(UnaryImm.imm, 8), + emit=''' + PUT_OP(bits, rex1(4), sink); + modrm_r_bits(4, bits, sink); + let imm: i64 = imm.into(); + sink.put1(imm as u8); + ''') + +adjustsp32 = TailRecipe( + 'adjustsp32', UnaryImm, size=5, ins=(), outs=(), instp=IsSignedInt(UnaryImm.imm, 32), emit=''' PUT_OP(bits, rex1(4), sink); From c64428b6984b3a3314ddd5ffe985ae4c14f4b894 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 4 Dec 2017 14:03:54 -0800 Subject: [PATCH 1429/3084] Add a Map::get_or_less() method. Find the largest (k,v) pair with k <= key. --- lib/cretonne/src/bforest/map.rs | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/lib/cretonne/src/bforest/map.rs b/lib/cretonne/src/bforest/map.rs index bf288582eb..3c5fe7fb9e 100644 --- a/lib/cretonne/src/bforest/map.rs +++ b/lib/cretonne/src/bforest/map.rs @@ -102,6 +102,23 @@ where }) } + /// Look up the value stored for `key`. + /// + /// If it exists, return the stored key-value pair. + /// + /// Otherwise, return the last key-value pair with a key that is less than or equal to `key`. + /// + /// If no stored keys are less than or equal to `key`, return `None`. + pub fn get_or_less(&self, key: K, forest: &MapForest, comp: &C) -> Option<(K, V)> { + self.root.expand().and_then(|root| { + let mut path = Path::default(); + match path.find(key, root, &forest.nodes, comp) { + Some(v) => Some((key, v)), + None => path.prev(root, &forest.nodes), + } + }) + } + /// Insert `key, value` into the map and return the old value stored for `key`, if any. pub fn insert( &mut self, @@ -420,6 +437,7 @@ mod test { assert_eq!(m.get(7, &f, &()), None); assert_eq!(m.iter(&f).next(), None); + assert_eq!(m.get_or_less(7, &f, &()), None); m.retain(&mut f, |_, _| unreachable!()); let mut c = m.cursor(&mut f, &()); @@ -474,6 +492,13 @@ mod test { assert_eq!(m.get(80, f, &()), Some(8.0)); assert_eq!(m.get(100, f, &()), None); + assert_eq!(m.get_or_less(0, f, &()), None); + assert_eq!(m.get_or_less(20, f, &()), Some((20, 2.0))); + assert_eq!(m.get_or_less(30, f, &()), Some((20, 2.0))); + assert_eq!(m.get_or_less(40, f, &()), Some((40, 4.0))); + assert_eq!(m.get_or_less(200, f, &()), Some((200, 20.0))); + assert_eq!(m.get_or_less(201, f, &()), Some((200, 20.0))); + { let mut c = m.cursor(f, &()); assert_eq!(c.prev(), Some((200, 20.0))); From b8fe6bf0f5d8099c93f9484d60def1e508b18ce3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Dec 2017 10:01:35 -0800 Subject: [PATCH 1430/3084] Add a MapCursor::value_mut() method. It's ok to alter a value stored in a map, but not the keys. --- lib/cretonne/src/bforest/map.rs | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/cretonne/src/bforest/map.rs b/lib/cretonne/src/bforest/map.rs index 3c5fe7fb9e..5aa19bceda 100644 --- a/lib/cretonne/src/bforest/map.rs +++ b/lib/cretonne/src/bforest/map.rs @@ -306,6 +306,13 @@ where }) } + /// Get a mutable reference to the current value, or `None` if the cursor is at the end. + pub fn value_mut(&mut self) -> Option<&mut V> { + self.path.leaf_pos().and_then(move |(node, entry)| { + self.pool[node].unwrap_leaf_mut().1.get_mut(entry) + }) + } + /// Move this cursor to `key`. /// /// If `key` is in the map, place the cursor at `key` and return the corresponding value. From c09ad06f96f055b1207032dee3c6779677922b84 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 5 Dec 2017 16:31:10 -0800 Subject: [PATCH 1431/3084] Stop generating reserved_reg heaps in DummyEnvironment. The reserved register heaps are not implemented in the Cretonne legalizer, so IR generated by the dummy environment would trip assertions when compiled. Use a heap with a vmctx base address instead, and also demonstrate how vmctx arguments are added to all signatures to achieve this. --- lib/wasm/src/environ/dummy.rs | 85 ++++++++++++++++++++++++++++++----- 1 file changed, 74 insertions(+), 11 deletions(-) diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 8c9c7c3d8d..90ed3ffb1e 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -124,6 +124,17 @@ impl<'dummy_environment> DummyFuncEnvironment<'dummy_environment> { pub fn new(mod_info: &'dummy_environment DummyModuleInfo) -> Self { Self { mod_info } } + + // Create a signature for `sigidx` amended with a `vmctx` argument after the standard wasm + // arguments. + fn vmctx_sig(&self, sigidx: SignatureIndex) -> ir::Signature { + let mut sig = self.mod_info.signatures[sigidx].clone(); + sig.params.push(ir::AbiParam::special( + self.native_pointer(), + ir::ArgumentPurpose::VMContext, + )); + sig + } } impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environment> { @@ -142,8 +153,11 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ } fn make_heap(&mut self, func: &mut ir::Function, _index: MemoryIndex) -> ir::Heap { + // Create a static heap whose base address is stored at `vmctx+0`. + let gv = func.create_global_var(ir::GlobalVarData::VmCtx { offset: 0.into() }); + func.create_heap(ir::HeapData { - base: ir::HeapBase::ReservedReg, + base: ir::HeapBase::GlobalVar(gv), min_size: 0.into(), guard_size: 0x8000_0000.into(), style: ir::HeapStyle::Static { bound: 0x1_0000_0000.into() }, @@ -153,14 +167,14 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ fn make_indirect_sig(&mut self, func: &mut ir::Function, index: SignatureIndex) -> ir::SigRef { // A real implementation would probably change the calling convention and add `vmctx` and // signature index arguments. - func.import_signature(self.mod_info.signatures[index].clone()) + func.import_signature(self.vmctx_sig(index)) } fn make_direct_func(&mut self, func: &mut ir::Function, index: FunctionIndex) -> ir::FuncRef { let sigidx = self.mod_info.functions[index].entity; // A real implementation would probably add a `vmctx` argument. // And maybe attempt some signature de-duplication. - let signature = func.import_signature(self.mod_info.signatures[sigidx].clone()); + let signature = func.import_signature(self.vmctx_sig(sigidx)); let name = get_func_name(index); func.import_function(ir::ExtFuncData { name, signature }) } @@ -174,7 +188,56 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ callee: ir::Value, call_args: &[ir::Value], ) -> ir::Inst { - pos.ins().call_indirect(sig_ref, callee, call_args) + // Pass the current function's vmctx parameter on to the callee. + let vmctx = pos.func + .special_param(ir::ArgumentPurpose::VMContext) + .expect("Missing vmctx parameter"); + + // The `callee` value is an index into a table of function pointers. + // Apparently, that table is stored at absolute address 0 in this dummy environment. + // TODO: Generate bounds checking code. + let ptr = self.native_pointer(); + let callee_offset = if ptr == I32 { + pos.ins().imul_imm(callee, 4) + } else { + let ext = pos.ins().uextend(I64, callee); + pos.ins().imul_imm(ext, 4) + }; + let func_ptr = pos.ins().load(ptr, ir::MemFlags::new(), callee_offset, 0); + + // Build a value list for the indirect call instruction containing the callee, call_args, + // and the vmctx parameter. + let mut args = ir::ValueList::default(); + args.push(func_ptr, &mut pos.func.dfg.value_lists); + args.extend(call_args.iter().cloned(), &mut pos.func.dfg.value_lists); + args.push(vmctx, &mut pos.func.dfg.value_lists); + + pos.ins() + .IndirectCall(ir::Opcode::CallIndirect, ir::types::VOID, sig_ref, args) + .0 + } + + fn translate_call( + &mut self, + mut pos: FuncCursor, + _callee_index: FunctionIndex, + callee: ir::FuncRef, + call_args: &[ir::Value], + ) -> ir::Inst { + // Pass the current function's vmctx parameter on to the callee. + let vmctx = pos.func + .special_param(ir::ArgumentPurpose::VMContext) + .expect("Missing vmctx parameter"); + + // Build a value list for the call instruction containing the call_args and the vmctx + // parameter. + let mut args = ir::ValueList::default(); + args.extend(call_args.iter().cloned(), &mut pos.func.dfg.value_lists); + args.push(vmctx, &mut pos.func.dfg.value_lists); + + pos.ins() + .Call(ir::Opcode::Call, ir::types::VOID, callee, args) + .0 } fn translate_grow_memory( @@ -309,18 +372,18 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment { } fn define_function_body(&mut self, body_bytes: &'data [u8]) -> Result<(), String> { - let function_index = self.get_num_func_imports() + self.info.function_bodies.len(); - let name = get_func_name(function_index); - let sig = self.get_signature(self.get_func_type(function_index)) - .clone(); - let mut func = ir::Function::with_name_signature(name, sig); - { + let func = { let mut func_environ = DummyFuncEnvironment::new(&self.info); + let function_index = self.get_num_func_imports() + self.info.function_bodies.len(); + let name = get_func_name(function_index); + let sig = func_environ.vmctx_sig(self.get_func_type(function_index)); + let mut func = ir::Function::with_name_signature(name, sig); let reader = wasmparser::BinaryReader::new(body_bytes); self.trans .translate_from_reader(reader, &mut func, &mut func_environ) .map_err(|e| String::from(e.description()))?; - } + func + }; self.info.function_bodies.push(func); Ok(()) } From f106e4266ad91115df8abd1279adc48c6187c11c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 6 Dec 2017 08:30:48 -0800 Subject: [PATCH 1432/3084] Enable the IL verifier by default. Change the default value for the "enable_verifier" setting so the verifier runs unless it is explicitly disabled. Most projects using Cretonne are best off running the verifier always until they start caring about compile time performance. Then they can easily disable the verifier. --- cranelift/src/compile.rs | 6 +----- lib/cretonne/meta/base/settings.py | 3 ++- lib/cretonne/src/settings.rs | 2 +- lib/reader/src/parser.rs | 9 +-------- 4 files changed, 5 insertions(+), 15 deletions(-) diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 9e828cfed2..ae0c87ad36 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -12,13 +12,9 @@ use utils::{pretty_error, read_to_string, parse_sets_and_isa}; pub fn run( files: Vec, flag_print: bool, - mut flag_set: Vec, + flag_set: Vec, flag_isa: String, ) -> Result<(), String> { - // Enable the verifier by default, since we're reading IL in from a - // text file. - flag_set.insert(0, "enable_verifier=1".to_string()); - let parsed = parse_sets_and_isa(flag_set, flag_isa)?; for filename in files { diff --git a/lib/cretonne/meta/base/settings.py b/lib/cretonne/meta/base/settings.py index 11f0728e11..a08d5b4c3f 100644 --- a/lib/cretonne/meta/base/settings.py +++ b/lib/cretonne/meta/base/settings.py @@ -24,7 +24,8 @@ enable_verifier = BoolSetting( This makes compilation slower but catches many bugs. The verifier is disabled by default, except when reading Cretonne IL from a text file. - """) + """, + default=True) is_64bit = BoolSetting("Enable 64-bit code generation") diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index d04dd4991f..4893ecf35a 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -345,7 +345,7 @@ mod tests { f.to_string(), "[shared]\n\ opt_level = \"default\"\n\ - enable_verifier = false\n\ + enable_verifier = true\n\ is_64bit = false\n\ return_at_end = false\n\ is_compressed = false\n\ diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index b3827de62c..ac25f1e6e6 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -14,7 +14,7 @@ use cretonne::ir::immediates::{Imm64, Uimm32, Offset32, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs}; use cretonne::isa::{self, TargetIsa, Encoding, RegUnit}; -use cretonne::settings::{self, Configurable}; +use cretonne::settings; use testfile::{TestFile, Details, Comment}; use error::{Location, Error, Result}; use lexer::{self, Lexer, Token}; @@ -731,13 +731,6 @@ impl<'a> Parser<'a> { let mut isas = Vec::new(); let mut flag_builder = settings::builder(); - // Change the default for `enable_verifier` to `true`. It defaults to `false` because it - // would slow down normal compilation, but when we're reading IL from a text file we're - // either testing or debugging Cretonne, and verification makes sense. - flag_builder.enable("enable_verifier").expect( - "Missing enable_verifier setting", - ); - while let Some(Token::Identifier(command)) = self.token() { match command { "set" => { From 27d5543adcf863f62d7462dfa83259975401ccda Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 4 Dec 2017 12:57:04 -0800 Subject: [PATCH 1433/3084] Make LiveRange a type alias for GenLiveRange. This makes the whole LiveRange generic over the program order instead of having a number of methods that are individually program order-generic. This makes is possible to have data members that depend on the program order, as we will shortly. This also gives us stronger type checking on the public LiveRange methods which now require a Layout argument, not just any program order. --- lib/cretonne/src/regalloc/liverange.rs | 63 +++++++++++++------------- 1 file changed, 32 insertions(+), 31 deletions(-) diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 7b94fce280..840c882223 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -108,9 +108,10 @@ //! use entity::SparseMapValue; -use ir::{Inst, Ebb, Value, ProgramPoint, ExpandedProgramPoint, ProgramOrder}; +use ir::{Inst, Ebb, Value, Layout, ProgramPoint, ExpandedProgramPoint, ProgramOrder}; use regalloc::affinity::Affinity; use std::cmp::Ordering; +use std::marker::PhantomData; /// Global live range of a single SSA value. /// @@ -139,7 +140,13 @@ use std::cmp::Ordering; /// Inserting new instructions in the layout is safe, but removing instructions is not. Besides the /// instructions using or defining their value, `LiveRange` structs can contain references to /// branch and jump instructions. -pub struct LiveRange { +pub type LiveRange = GenLiveRange; + +/// Generic live range implementation. +/// +/// The intended generic parameter is `PO=Layout`, but tests are simpler with a mock order. +/// Use `LiveRange` instead of using this generic directly. +pub struct GenLiveRange { /// The value described by this live range. /// This member can't be modified in case the live range is stored in a `SparseMap`. value: Value, @@ -166,6 +173,8 @@ pub struct LiveRange { /// - Not overlapping defining EBB: For all `i`: /// `liveins[i].end < def_begin` or `liveins[i].begin > def_end`. liveins: Vec, + + unused: PhantomData, } /// An additional contiguous interval of a global live range. @@ -202,17 +211,18 @@ impl Interval { } } -impl LiveRange { +impl GenLiveRange { /// Create a new live range for `value` defined at `def`. /// /// The live range will be created as dead, but it can be extended with `extend_in_ebb()`. - pub fn new(value: Value, def: ProgramPoint, affinity: Affinity) -> LiveRange { - LiveRange { + pub fn new(value: Value, def: ProgramPoint, affinity: Affinity) -> GenLiveRange { + GenLiveRange { value, affinity, def_begin: def, def_end: def, liveins: Vec::new(), + unused: PhantomData, } } @@ -220,7 +230,7 @@ impl LiveRange { /// /// Return `Ok(n)` if `liveins[n]` already contains `ebb`. /// Otherwise, return `Err(n)` with the index where such an interval should be inserted. - fn find_ebb_interval(&self, ebb: Ebb, order: &PO) -> Result { + fn find_ebb_interval(&self, ebb: Ebb, order: &PO) -> Result { self.liveins .binary_search_by(|intv| order.cmp(intv.begin, ebb)) .or_else(|n| { @@ -244,7 +254,7 @@ impl LiveRange { /// /// The return value can be used to detect if we just learned that the value is live-in to /// `ebb`. This can trigger recursive extensions in `ebb`'s CFG predecessor blocks. - pub fn extend_in_ebb(&mut self, ebb: Ebb, to: Inst, order: &PO) -> bool { + pub fn extend_in_ebb(&mut self, ebb: Ebb, to: Inst, order: &PO) -> bool { // First check if we're extending the def interval. // // We're assuming here that `to` never precedes `def_begin` in the same EBB, but we can't @@ -377,7 +387,7 @@ impl LiveRange { /// If the live range is live through all of `ebb`, the terminator of `ebb` is a correct /// answer, but it is also possible that an even later program point is returned. So don't /// depend on the returned `Inst` to belong to `ebb`. - pub fn livein_local_end(&self, ebb: Ebb, order: &PO) -> Option { + pub fn livein_local_end(&self, ebb: Ebb, order: &PO) -> Option { self.find_ebb_interval(ebb, order).ok().map(|n| { self.liveins[n].end }) @@ -386,7 +396,7 @@ impl LiveRange { /// Is this value live-in to `ebb`? /// /// An EBB argument is not considered to be live in. - pub fn is_livein(&self, ebb: Ebb, order: &PO) -> bool { + pub fn is_livein(&self, ebb: Ebb, order: &PO) -> bool { self.livein_local_end(ebb, order).is_some() } @@ -396,10 +406,7 @@ impl LiveRange { } /// Check if this live range overlaps a definition in `ebb`. - pub fn overlaps_def(&self, def: ExpandedProgramPoint, ebb: Ebb, order: &PO) -> bool - where - PO: ProgramOrder, - { + pub fn overlaps_def(&self, def: ExpandedProgramPoint, ebb: Ebb, order: &PO) -> bool { // Check for an overlap with the local range. if order.cmp(def, self.def_begin) != Ordering::Less && order.cmp(def, self.def_end) == Ordering::Less @@ -415,10 +422,7 @@ impl LiveRange { } /// Check if this live range reaches a use at `user` in `ebb`. - pub fn reaches_use(&self, user: Inst, ebb: Ebb, order: &PO) -> bool - where - PO: ProgramOrder, - { + pub fn reaches_use(&self, user: Inst, ebb: Ebb, order: &PO) -> bool { // Check for an overlap with the local range. if order.cmp(user, self.def_begin) == Ordering::Greater && order.cmp(user, self.def_end) != Ordering::Greater @@ -434,16 +438,13 @@ impl LiveRange { } /// Check if this live range is killed at `user` in `ebb`. - pub fn killed_at(&self, user: Inst, ebb: Ebb, order: &PO) -> bool - where - PO: ProgramOrder, - { + pub fn killed_at(&self, user: Inst, ebb: Ebb, order: &PO) -> bool { self.def_local_end() == user.into() || self.livein_local_end(ebb, order) == Some(user) } } /// Allow a `LiveRange` to be stored in a `SparseMap` indexed by values. -impl SparseMapValue for LiveRange { +impl SparseMapValue for GenLiveRange { fn key(&self) -> Value { self.value } @@ -451,7 +452,7 @@ impl SparseMapValue for LiveRange { #[cfg(test)] mod tests { - use super::LiveRange; + use super::GenLiveRange; use ir::{Inst, Ebb, Value}; use entity::EntityRef; use ir::{ProgramOrder, ExpandedProgramPoint}; @@ -502,7 +503,7 @@ mod tests { } // Validate the live range invariants. - fn validate(&self, lr: &LiveRange) { + fn validate(&self, lr: &GenLiveRange) { // The def interval must cover a single EBB. let def_ebb = self.pp_ebb(lr.def_begin); assert_eq!(def_ebb, self.pp_ebb(lr.def_end)); @@ -545,7 +546,7 @@ mod tests { let v0 = Value::new(0); let i1 = Inst::new(1); let e2 = Ebb::new(2); - let lr = LiveRange::new(v0, i1.into(), Default::default()); + let lr = GenLiveRange::new(v0, i1.into(), Default::default()); assert!(lr.is_dead()); assert!(lr.is_local()); assert_eq!(lr.def(), i1.into()); @@ -558,7 +559,7 @@ mod tests { fn dead_arg_range() { let v0 = Value::new(0); let e2 = Ebb::new(2); - let lr = LiveRange::new(v0, e2.into(), Default::default()); + let lr = GenLiveRange::new(v0, e2.into(), Default::default()); assert!(lr.is_dead()); assert!(lr.is_local()); assert_eq!(lr.def(), e2.into()); @@ -575,7 +576,7 @@ mod tests { let i11 = Inst::new(11); let i12 = Inst::new(12); let i13 = Inst::new(13); - let mut lr = LiveRange::new(v0, i11.into(), Default::default()); + let mut lr = GenLiveRange::new(v0, i11.into(), Default::default()); assert_eq!(lr.extend_in_ebb(e10, i13, PO), false); PO.validate(&lr); @@ -598,7 +599,7 @@ mod tests { let i11 = Inst::new(11); let i12 = Inst::new(12); let i13 = Inst::new(13); - let mut lr = LiveRange::new(v0, e10.into(), Default::default()); + let mut lr = GenLiveRange::new(v0, e10.into(), Default::default()); // Extending a dead EBB argument in its own block should not indicate that a live-in // interval was created. @@ -632,7 +633,7 @@ mod tests { let i21 = Inst::new(21); let i22 = Inst::new(22); let i23 = Inst::new(23); - let mut lr = LiveRange::new(v0, i11.into(), Default::default()); + let mut lr = GenLiveRange::new(v0, i11.into(), Default::default()); assert_eq!(lr.extend_in_ebb(e10, i12, PO), false); @@ -661,7 +662,7 @@ mod tests { let i31 = Inst::new(31); let e40 = Ebb::new(40); let i41 = Inst::new(41); - let mut lr = LiveRange::new(v0, i11.into(), Default::default()); + let mut lr = GenLiveRange::new(v0, i11.into(), Default::default()); assert_eq!(lr.extend_in_ebb(e30, i31, PO), true); assert_eq!(lr.liveins.len(), 1); @@ -678,7 +679,7 @@ mod tests { assert_eq!(lr.liveins[0].begin, e20); assert_eq!(lr.liveins[0].end, i41); - let mut lr = LiveRange::new(v0, i11.into(), Default::default()); + let mut lr = GenLiveRange::new(v0, i11.into(), Default::default()); assert_eq!(lr.extend_in_ebb(e40, i41, PO), true); assert_eq!(lr.liveins.len(), 1); From feaea238bccb03dd6c99469832c6af89991ee94d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 4 Dec 2017 13:43:10 -0800 Subject: [PATCH 1434/3084] Use bforest::Map for representing live ranges. Get rid of the per-value Vec in the LiveRange data type and use a bforest::Map instead to represent the live-in intervals for non-local live ranges. This has some advantages: - The memory footprint of a local live range is reduced from 40 to 20 bytes, and - Clearing the Liveness data structure is now a constant time operation which doesn't call free(). - The potentially quadratic behavior when computing large live ranges is controlled by the logarithmic B-tree operations. --- lib/cretonne/src/regalloc/coalescing.rs | 12 +- lib/cretonne/src/regalloc/coloring.rs | 19 +- .../src/regalloc/live_value_tracker.rs | 3 +- lib/cretonne/src/regalloc/liveness.rs | 29 +- lib/cretonne/src/regalloc/liverange.rs | 402 ++++++++++-------- lib/cretonne/src/regalloc/spilling.rs | 5 +- lib/cretonne/src/verifier/cssa.rs | 3 +- lib/cretonne/src/verifier/liveness.rs | 17 +- lib/cretonne/src/verifier/locations.rs | 4 +- 9 files changed, 279 insertions(+), 215 deletions(-) diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 224e97840a..31228f680a 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -157,13 +157,11 @@ impl DomForest { layout, domtree, }; + let ctx = liveness.context(layout); for node in merged { if let Some(parent) = self.push_node(node, layout, domtree) { // Check if `parent` live range contains `node.def`. - let lr = liveness.get(parent).expect( - "No live range for parent value", - ); - if lr.overlaps_def(node.def, layout.pp_ebb(node.def), layout) { + if liveness[parent].overlaps_def(node.def, layout.pp_ebb(node.def), ctx) { // Interference detected. Get the `(a, b)` order right in the error. return Err(if node.set == 0 { (node.value, parent) @@ -447,7 +445,11 @@ impl<'a> Context<'a> { if self.liveness .get(a) .expect("No live range for interfering value") - .reaches_use(pred_inst, pred_ebb, &self.func.layout) + .reaches_use( + pred_inst, + pred_ebb, + self.liveness.context(&self.func.layout), + ) { // Splitting at `pred_inst` wouldn't resolve the interference, so we need to // start over. diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 23cc3e27c2..25b271d17e 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -44,7 +44,7 @@ use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; -use ir::{Ebb, Inst, Value, Function, ValueLoc, SigRef}; +use ir::{Ebb, Inst, Value, Function, Layout, ValueLoc, SigRef}; use ir::{InstBuilder, AbiParam, ArgumentLoc, ValueDef}; use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; use isa::{TargetIsa, EncInfo, RecipeConstraints, OperandConstraint, ConstraintKind}; @@ -54,7 +54,7 @@ use regalloc::affinity::Affinity; use regalloc::allocatable_set::AllocatableSet; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; -use regalloc::liverange::LiveRange; +use regalloc::liverange::{LiveRange, LiveRangeContext}; use regalloc::solver::{Solver, SolverError}; use std::mem; @@ -542,8 +542,8 @@ impl<'a> Context<'a> { if op.regclass.contains(cur_reg) { // This code runs after calling `solver.inputs_done()` so we must identify // the new variable as killed or live-through. - let layout = &self.cur.func.layout; - if self.liveness[value].killed_at(inst, layout.pp_ebb(inst), layout) { + let ctx = self.liveness.context(&self.cur.func.layout); + if self.liveness[value].killed_at(inst, ctx.order.pp_ebb(inst), ctx) { self.solver.add_killed_var(value, op.regclass, cur_reg); } else { self.solver.add_through_var(value, op.regclass, cur_reg); @@ -572,7 +572,7 @@ impl<'a> Context<'a> { // // Values with a global live range that are not live in to `dest` could appear as branch // arguments, so they can't always be un-diverted. - self.undivert_regs(|lr, func| lr.is_livein(dest, &func.layout)); + self.undivert_regs(|lr, ctx| lr.is_livein(dest, ctx)); // Now handle the EBB arguments. let br_args = self.cur.func.dfg.inst_variable_args(inst); @@ -642,13 +642,13 @@ impl<'a> Context<'a> { /// are reallocated to their global register assignments. fn undivert_regs(&mut self, mut pred: Pred) where - Pred: FnMut(&LiveRange, &Function) -> bool, + Pred: FnMut(&LiveRange, LiveRangeContext) -> bool, { for rdiv in self.divert.all() { let lr = self.liveness.get(rdiv.value).expect( "Missing live range for diverted register", ); - if pred(lr, &self.cur.func) { + if pred(lr, self.liveness.context(&self.cur.func.layout)) { if let Affinity::Reg(rci) = lr.affinity { let rc = self.reginfo.rc(rci); // Stack diversions should not be possible here. The only live transiently @@ -882,17 +882,18 @@ impl<'a> Context<'a> { use ir::instructions::BranchInfo::*; let inst = self.cur.current_inst().expect("Not on an instruction"); + let ctx = self.liveness.context(&self.cur.func.layout); match self.cur.func.dfg[inst].analyze_branch(&self.cur.func.dfg.value_lists) { NotABranch => false, SingleDest(ebb, _) => { let lr = &self.liveness[value]; - lr.is_livein(ebb, &self.cur.func.layout) + lr.is_livein(ebb, ctx) } Table(jt) => { let lr = &self.liveness[value]; !lr.is_local() && self.cur.func.jump_tables[jt].entries().any(|(_, ebb)| { - lr.is_livein(ebb, &self.cur.func.layout) + lr.is_livein(ebb, ctx) }) } } diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index cb57683661..e6f53151e2 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -189,6 +189,7 @@ impl LiveValueTracker { let idom_live_list = self.idom_sets.get(&idom).expect( "No stored live set for dominator", ); + let ctx = liveness.context(layout); // Get just the values that are live-in to `ebb`. for &value in idom_live_list.as_slice(&self.idom_pool) { let lr = liveness.get(value).expect( @@ -196,7 +197,7 @@ impl LiveValueTracker { ); // Check if this value is live-in here. - if let Some(endpoint) = lr.livein_local_end(ebb, layout) { + if let Some(endpoint) = lr.livein_local_end(ebb, ctx) { self.live.push(value, endpoint, lr); } } diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index e152cfaa32..20208e36a1 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -181,7 +181,7 @@ use ir::dfg::ValueDef; use ir::{Function, Value, Inst, Ebb, Layout, ProgramPoint}; use isa::{TargetIsa, EncInfo}; use regalloc::affinity::Affinity; -use regalloc::liverange::LiveRange; +use regalloc::liverange::{LiveRange, LiveRangeForest, LiveRangeContext}; use std::mem; use std::ops::Index; @@ -247,13 +247,14 @@ fn extend_to_use( worklist: &mut Vec, func: &Function, cfg: &ControlFlowGraph, + forest: &mut LiveRangeForest, ) { // This is our scratch working space, and we'll leave it empty when we return. assert!(worklist.is_empty()); // Extend the range locally in `ebb`. // If there already was a live interval in that block, we're done. - if lr.extend_in_ebb(ebb, to, &func.layout) { + if lr.extend_in_ebb(ebb, to, &func.layout, forest) { worklist.push(ebb); } @@ -270,7 +271,7 @@ fn extend_to_use( // We've learned that the value needs to be live-in to the `livein` EBB. // Make sure it is also live at all predecessor branches to `livein`. for (pred, branch) in cfg.pred_iter(livein) { - if lr.extend_in_ebb(pred, branch, &func.layout) { + if lr.extend_in_ebb(pred, branch, &func.layout, forest) { // This predecessor EBB also became live-in. We need to process it later. worklist.push(pred); } @@ -285,6 +286,9 @@ pub struct Liveness { /// The live ranges that have been computed so far. ranges: LiveRangeSet, + /// Memory pool for the live ranges. + forest: LiveRangeForest, + /// Working space for the `extend_to_use` algorithm. /// This vector is always empty, except for inside that function. /// It lives here to avoid repeated allocation of scratch memory. @@ -302,14 +306,21 @@ impl Liveness { pub fn new() -> Self { Self { ranges: LiveRangeSet::new(), + forest: LiveRangeForest::new(), worklist: Vec::new(), ebb_params: Vec::new(), } } + /// Get a context needed for working with a `LiveRange`. + pub fn context<'a>(&'a self, layout: &'a Layout) -> LiveRangeContext<'a, Layout> { + LiveRangeContext::new(layout, &self.forest) + } + /// Clear all data structures in this liveness analysis. pub fn clear(&mut self) { self.ranges.clear(); + self.forest.clear(); self.worklist.clear(); self.ebb_params.clear(); } @@ -359,7 +370,7 @@ impl Liveness { ) -> &mut Affinity { debug_assert_eq!(Some(ebb), layout.inst_ebb(user)); let lr = self.ranges.get_mut(value).expect("Value has no live range"); - let livein = lr.extend_in_ebb(ebb, user, layout); + let livein = lr.extend_in_ebb(ebb, user, layout, &mut self.forest); assert!(!livein, "{} should already be live in {}", value, ebb); &mut lr.affinity } @@ -416,7 +427,15 @@ impl Liveness { let lr = get_or_create(&mut self.ranges, arg, isa, func, &enc_info); // Extend the live range to reach this use. - extend_to_use(lr, ebb, inst, &mut self.worklist, func, cfg); + extend_to_use( + lr, + ebb, + inst, + &mut self.worklist, + func, + cfg, + &mut self.forest, + ); // Apply operand constraint, ignoring any variable arguments after the fixed // operands described by `operand_constraints`. Variable arguments are either diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 840c882223..60079b3705 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -107,11 +107,11 @@ //! of coalescing, so we would need to roll our own. //! +use bforest; use entity::SparseMapValue; use ir::{Inst, Ebb, Value, Layout, ProgramPoint, ExpandedProgramPoint, ProgramOrder}; use regalloc::affinity::Affinity; use std::cmp::Ordering; -use std::marker::PhantomData; /// Global live range of a single SSA value. /// @@ -165,52 +165,54 @@ pub struct GenLiveRange { /// Additional live-in intervals sorted in program order. /// - /// This vector is empty for most values which are only used in one EBB. + /// This map is empty for most values which are only used in one EBB. /// - /// Invariants: + /// A map entry `ebb -> inst` means that the live range is live-in to `ebb`, continuing up to + /// `inst` which may belong to a later EBB in the program order. /// - /// - Sorted, disjoint: For all `i < j`: `liveins[i].end < liveins[j].begin`. - /// - Not overlapping defining EBB: For all `i`: - /// `liveins[i].end < def_begin` or `liveins[i].begin > def_end`. - liveins: Vec, - - unused: PhantomData, + /// The entries are non-overlapping, and none of them overlap the EBB where the value is + /// defined. + liveins: bforest::Map, } -/// An additional contiguous interval of a global live range. -/// -/// This represents a live-in interval for a single EBB, or a coalesced set of live-in intervals -/// for contiguous EBBs where all but the last live-in interval covers the whole EBB. -/// -#[derive(Copy, Clone)] -pub struct Interval { - /// Interval starting point. - /// - /// Since this interval does not represent the def of the value, it must begin at an EBB header - /// where the value is live-in. - pub begin: Ebb, - - /// Interval end point. - /// - /// The only interval end point that can be an EBB header is `def_end` above in a dead def - /// live range for an unused EBB argument. All other intervals must end at an instruction -- - /// either the last use in the EBB or the last branch/jump that can reach a use. - /// - /// When this represents multiple contiguous live-in intervals, this is the end point of the - /// last interval. The other intervals end at the terminator instructions of their respective - /// EBB. - pub end: Inst, +/// Context information needed to query a `LiveRange`. +pub struct LiveRangeContext<'a, PO: 'a + ProgramOrder> { + /// Ordering of EBBs. + pub order: &'a PO, + /// Memory pool. + pub forest: &'a bforest::MapForest, } -impl Interval { - /// Extend the interval end point to reach `to`, but only if it would make the interval longer. - fn extend_to(&mut self, to: Inst, order: &PO) { - if order.cmp(to, self.end) == Ordering::Greater { - self.end = to; +impl<'a, PO: ProgramOrder> LiveRangeContext<'a, PO> { + /// Make a new context. + pub fn new( + order: &'a PO, + forest: &'a bforest::MapForest, + ) -> LiveRangeContext<'a, PO> { + LiveRangeContext { order, forest } + } +} + +impl<'a, PO: ProgramOrder> Clone for LiveRangeContext<'a, PO> { + fn clone(&self) -> Self { + LiveRangeContext { + order: self.order, + forest: self.forest, } } } +impl<'a, PO: ProgramOrder> Copy for LiveRangeContext<'a, PO> {} + +/// Forest of B-trees used for storing live ranges. +pub type LiveRangeForest = bforest::MapForest; + +impl bforest::Comparator for PO { + fn cmp(&self, a: Ebb, b: Ebb) -> Ordering { + self.cmp(a, b) + } +} + impl GenLiveRange { /// Create a new live range for `value` defined at `def`. /// @@ -221,28 +223,10 @@ impl GenLiveRange { affinity, def_begin: def, def_end: def, - liveins: Vec::new(), - unused: PhantomData, + liveins: bforest::Map::new(), } } - /// Find the live-in interval containing `ebb`, if any. - /// - /// Return `Ok(n)` if `liveins[n]` already contains `ebb`. - /// Otherwise, return `Err(n)` with the index where such an interval should be inserted. - fn find_ebb_interval(&self, ebb: Ebb, order: &PO) -> Result { - self.liveins - .binary_search_by(|intv| order.cmp(intv.begin, ebb)) - .or_else(|n| { - // The interval at `n-1` may cover `ebb`. - if n > 0 && order.cmp(self.liveins[n - 1].end, ebb) == Ordering::Greater { - Ok(n - 1) - } else { - Err(n) - } - }) - } - /// Extend the local interval for `ebb` so it reaches `to` which must belong to `ebb`. /// Create a live-in interval if necessary. /// @@ -254,7 +238,13 @@ impl GenLiveRange { /// /// The return value can be used to detect if we just learned that the value is live-in to /// `ebb`. This can trigger recursive extensions in `ebb`'s CFG predecessor blocks. - pub fn extend_in_ebb(&mut self, ebb: Ebb, to: Inst, order: &PO) -> bool { + pub fn extend_in_ebb( + &mut self, + ebb: Ebb, + to: Inst, + order: &PO, + forest: &mut bforest::MapForest, + ) -> bool { // First check if we're extending the def interval. // // We're assuming here that `to` never precedes `def_begin` in the same EBB, but we can't @@ -275,68 +265,60 @@ impl GenLiveRange { } // Now check if we're extending any of the existing live-in intervals. - match self.find_ebb_interval(ebb, order) { - Ok(n) => { - // We have an interval that contains `ebb`, so we can simply extend it. - self.liveins[n].extend_to(to, order); + let mut c = self.liveins.cursor(forest, order); + let first_time_livein; - // If `to` is the terminator and the value lives in the successor EBB, - // coalesce the two intervals. - if let Some(next) = self.liveins.get(n + 1).cloned() { - if order.is_ebb_gap(to, next.begin) { - self.liveins[n].extend_to(next.end, order); - self.liveins.remove(n + 1); - } - } - - false + if let Some(end) = c.goto(ebb) { + // There's an interval beginning at `ebb`. See if it extends. + first_time_livein = false; + if order.cmp(end, to) == Ordering::Less { + *c.value_mut().unwrap() = to; + } else { + return first_time_livein; } - Err(n) => { - // Insert a new live-in interval at `n`, or coalesce to predecessor or successor - // if possible. - - // Determine if the new live-in range touches the predecessor or successor range - // and can therefore be coalesced to them. - let (coalesce_prev, coalesce_next) = { - let prev = n.checked_sub(1).and_then(|i| self.liveins.get(i)); - let next = self.liveins.get(n); - - ( - prev.map_or(false, |prev| order.is_ebb_gap(prev.end, ebb)), - next.map_or(false, |next| order.is_ebb_gap(to, next.begin)), - ) - }; - - match (coalesce_prev, coalesce_next) { - // Extend predecessor interval to cover new and successor intervals - (true, true) => { - let end = self.liveins[n].end; - self.liveins[n - 1].extend_to(end, order); - self.liveins.remove(n); + } else { + // There's no interval beginning at `ebb`, but we could still be live-in at `ebb` with + // a coalesced interval that begins before and ends after. + if let Some((_, end)) = c.prev() { + if order.cmp(end, ebb) == Ordering::Greater { + // Yep, the previous interval overlaps `ebb`. + first_time_livein = false; + if order.cmp(end, to) == Ordering::Less { + *c.value_mut().unwrap() = to; + } else { + return first_time_livein; } - // Extend predecessor interval to cover new interval - (true, false) => { - self.liveins[n - 1].extend_to(to, order); - } - // Extend successor interval to cover new interval - (false, true) => { - self.liveins[n].begin = ebb; - } - // Cannot coalesce; insert new interval - (false, false) => { - self.liveins.insert( - n, - Interval { - begin: ebb, - end: to, - }, - ); + } else { + first_time_livein = true; + // The current interval does not overlap `ebb`, but it may still be possible to + // coalesce with it. + if order.is_ebb_gap(end, ebb) { + *c.value_mut().unwrap() = to; + } else { + c.insert(ebb, to); } } - - true + } else { + // There is no existing interval before `ebb`. + first_time_livein = true; + c.insert(ebb, to); } } + + // Now `c` to left pointing at an interval that ends in `to`. + debug_assert_eq!(c.value(), Some(to)); + + // See if it can be coalesced with the following interval. + if let Some((next_ebb, next_end)) = c.next() { + if order.is_ebb_gap(to, next_ebb) { + // Remove this interval and extend the previous end point to `next_end`. + c.remove(); + c.prev(); + *c.value_mut().unwrap() = next_end; + } + } + + first_time_livein } /// Is this the live range of a dead value? @@ -387,59 +369,77 @@ impl GenLiveRange { /// If the live range is live through all of `ebb`, the terminator of `ebb` is a correct /// answer, but it is also possible that an even later program point is returned. So don't /// depend on the returned `Inst` to belong to `ebb`. - pub fn livein_local_end(&self, ebb: Ebb, order: &PO) -> Option { - self.find_ebb_interval(ebb, order).ok().map(|n| { - self.liveins[n].end - }) + pub fn livein_local_end(&self, ebb: Ebb, ctx: LiveRangeContext) -> Option { + self.liveins + .get_or_less(ebb, ctx.forest, ctx.order) + .and_then(|(_, inst)| { + // We have an entry that ends at `inst`. + if ctx.order.cmp(inst, ebb) == Ordering::Greater { + Some(inst) + } else { + None + } + }) } /// Is this value live-in to `ebb`? /// /// An EBB argument is not considered to be live in. - pub fn is_livein(&self, ebb: Ebb, order: &PO) -> bool { - self.livein_local_end(ebb, order).is_some() + pub fn is_livein(&self, ebb: Ebb, ctx: LiveRangeContext) -> bool { + self.livein_local_end(ebb, ctx).is_some() } /// Get all the live-in intervals. - pub fn liveins(&self) -> &[Interval] { - &self.liveins + /// + /// Note that the intervals are stored in a compressed form so each entry may span multiple + /// EBBs where the value is live in. + pub fn liveins<'a>( + &'a self, + ctx: LiveRangeContext<'a, PO>, + ) -> bforest::MapIter<'a, Ebb, Inst, PO> { + self.liveins.iter(ctx.forest) } /// Check if this live range overlaps a definition in `ebb`. - pub fn overlaps_def(&self, def: ExpandedProgramPoint, ebb: Ebb, order: &PO) -> bool { + pub fn overlaps_def( + &self, + def: ExpandedProgramPoint, + ebb: Ebb, + ctx: LiveRangeContext, + ) -> bool { // Check for an overlap with the local range. - if order.cmp(def, self.def_begin) != Ordering::Less && - order.cmp(def, self.def_end) == Ordering::Less + if ctx.order.cmp(def, self.def_begin) != Ordering::Less && + ctx.order.cmp(def, self.def_end) == Ordering::Less { return true; } // Check for an overlap with a live-in range. - match self.livein_local_end(ebb, order) { - Some(inst) => order.cmp(def, inst) == Ordering::Less, + match self.livein_local_end(ebb, ctx) { + Some(inst) => ctx.order.cmp(def, inst) == Ordering::Less, None => false, } } /// Check if this live range reaches a use at `user` in `ebb`. - pub fn reaches_use(&self, user: Inst, ebb: Ebb, order: &PO) -> bool { + pub fn reaches_use(&self, user: Inst, ebb: Ebb, ctx: LiveRangeContext) -> bool { // Check for an overlap with the local range. - if order.cmp(user, self.def_begin) == Ordering::Greater && - order.cmp(user, self.def_end) != Ordering::Greater + if ctx.order.cmp(user, self.def_begin) == Ordering::Greater && + ctx.order.cmp(user, self.def_end) != Ordering::Greater { return true; } // Check for an overlap with a live-in range. - match self.livein_local_end(ebb, order) { - Some(inst) => order.cmp(user, inst) != Ordering::Greater, + match self.livein_local_end(ebb, ctx) { + Some(inst) => ctx.order.cmp(user, inst) != Ordering::Greater, None => false, } } /// Check if this live range is killed at `user` in `ebb`. - pub fn killed_at(&self, user: Inst, ebb: Ebb, order: &PO) -> bool { - self.def_local_end() == user.into() || self.livein_local_end(ebb, order) == Some(user) + pub fn killed_at(&self, user: Inst, ebb: Ebb, ctx: LiveRangeContext) -> bool { + self.def_local_end() == user.into() || self.livein_local_end(ebb, ctx) == Some(user) } } @@ -452,7 +452,8 @@ impl SparseMapValue for GenLiveRange { #[cfg(test)] mod tests { - use super::GenLiveRange; + use super::{GenLiveRange, LiveRangeContext}; + use bforest; use ir::{Inst, Ebb, Value}; use entity::EntityRef; use ir::{ProgramOrder, ExpandedProgramPoint}; @@ -503,7 +504,11 @@ mod tests { } // Validate the live range invariants. - fn validate(&self, lr: &GenLiveRange) { + fn validate( + &self, + lr: &GenLiveRange, + forest: &bforest::MapForest, + ) { // The def interval must cover a single EBB. let def_ebb = self.pp_ebb(lr.def_begin); assert_eq!(def_ebb, self.pp_ebb(lr.def_end)); @@ -519,20 +524,20 @@ mod tests { // Check the live-in intervals. let mut prev_end = None; - for li in &lr.liveins { - assert_eq!(self.cmp(li.begin, li.end), Ordering::Less); + for (begin, end) in lr.liveins.iter(forest) { + assert_eq!(self.cmp(begin, end), Ordering::Less); if let Some(e) = prev_end { - assert_eq!(self.cmp(e, li.begin), Ordering::Less); + assert_eq!(self.cmp(e, begin), Ordering::Less); } assert!( - self.cmp(lr.def_end, li.begin) == Ordering::Less || - self.cmp(lr.def_begin, li.end) == Ordering::Greater, + self.cmp(lr.def_end, begin) == Ordering::Less || + self.cmp(lr.def_begin, end) == Ordering::Greater, "Interval can't overlap the def EBB" ); // Save for next round. - prev_end = Some(li.end); + prev_end = Some(end); } } @@ -547,12 +552,14 @@ mod tests { let i1 = Inst::new(1); let e2 = Ebb::new(2); let lr = GenLiveRange::new(v0, i1.into(), Default::default()); + let forest = &bforest::MapForest::new(); + let ctx = LiveRangeContext::new(PO, forest); assert!(lr.is_dead()); assert!(lr.is_local()); assert_eq!(lr.def(), i1.into()); assert_eq!(lr.def_local_end(), i1.into()); - assert_eq!(lr.livein_local_end(e2, PO), None); - PO.validate(&lr); + assert_eq!(lr.livein_local_end(e2, ctx), None); + PO.validate(&lr, ctx.forest); } #[test] @@ -560,13 +567,15 @@ mod tests { let v0 = Value::new(0); let e2 = Ebb::new(2); let lr = GenLiveRange::new(v0, e2.into(), Default::default()); + let forest = &bforest::MapForest::new(); + let ctx = LiveRangeContext::new(PO, forest); assert!(lr.is_dead()); assert!(lr.is_local()); assert_eq!(lr.def(), e2.into()); assert_eq!(lr.def_local_end(), e2.into()); // The def interval of an EBB argument does not count as live-in. - assert_eq!(lr.livein_local_end(e2, PO), None); - PO.validate(&lr); + assert_eq!(lr.livein_local_end(e2, ctx), None); + PO.validate(&lr, ctx.forest); } #[test] @@ -577,17 +586,18 @@ mod tests { let i12 = Inst::new(12); let i13 = Inst::new(13); let mut lr = GenLiveRange::new(v0, i11.into(), Default::default()); + let forest = &mut bforest::MapForest::new(); - assert_eq!(lr.extend_in_ebb(e10, i13, PO), false); - PO.validate(&lr); + assert_eq!(lr.extend_in_ebb(e10, i13, PO, forest), false); + PO.validate(&lr, forest); assert!(!lr.is_dead()); assert!(lr.is_local()); assert_eq!(lr.def(), i11.into()); assert_eq!(lr.def_local_end(), i13.into()); // Extending to an already covered inst should not change anything. - assert_eq!(lr.extend_in_ebb(e10, i12, PO), false); - PO.validate(&lr); + assert_eq!(lr.extend_in_ebb(e10, i12, PO, forest), false); + PO.validate(&lr, forest); assert_eq!(lr.def(), i11.into()); assert_eq!(lr.def_local_end(), i13.into()); } @@ -600,25 +610,26 @@ mod tests { let i12 = Inst::new(12); let i13 = Inst::new(13); let mut lr = GenLiveRange::new(v0, e10.into(), Default::default()); + let forest = &mut bforest::MapForest::new(); // Extending a dead EBB argument in its own block should not indicate that a live-in // interval was created. - assert_eq!(lr.extend_in_ebb(e10, i12, PO), false); - PO.validate(&lr); + assert_eq!(lr.extend_in_ebb(e10, i12, PO, forest), false); + PO.validate(&lr, forest); assert!(!lr.is_dead()); assert!(lr.is_local()); assert_eq!(lr.def(), e10.into()); assert_eq!(lr.def_local_end(), i12.into()); // Extending to an already covered inst should not change anything. - assert_eq!(lr.extend_in_ebb(e10, i11, PO), false); - PO.validate(&lr); + assert_eq!(lr.extend_in_ebb(e10, i11, PO, forest), false); + PO.validate(&lr, forest); assert_eq!(lr.def(), e10.into()); assert_eq!(lr.def_local_end(), i12.into()); // Extending further. - assert_eq!(lr.extend_in_ebb(e10, i13, PO), false); - PO.validate(&lr); + assert_eq!(lr.extend_in_ebb(e10, i13, PO, forest), false); + PO.validate(&lr, forest); assert_eq!(lr.def(), e10.into()); assert_eq!(lr.def_local_end(), i13.into()); } @@ -634,22 +645,32 @@ mod tests { let i22 = Inst::new(22); let i23 = Inst::new(23); let mut lr = GenLiveRange::new(v0, i11.into(), Default::default()); + let forest = &mut bforest::MapForest::new(); - assert_eq!(lr.extend_in_ebb(e10, i12, PO), false); + assert_eq!(lr.extend_in_ebb(e10, i12, PO, forest), false); // Adding a live-in interval. - assert_eq!(lr.extend_in_ebb(e20, i22, PO), true); - PO.validate(&lr); - assert_eq!(lr.livein_local_end(e20, PO), Some(i22)); + assert_eq!(lr.extend_in_ebb(e20, i22, PO, forest), true); + PO.validate(&lr, forest); + assert_eq!( + lr.livein_local_end(e20, LiveRangeContext::new(PO, forest)), + Some(i22) + ); // Non-extending the live-in. - assert_eq!(lr.extend_in_ebb(e20, i21, PO), false); - assert_eq!(lr.livein_local_end(e20, PO), Some(i22)); + assert_eq!(lr.extend_in_ebb(e20, i21, PO, forest), false); + assert_eq!( + lr.livein_local_end(e20, LiveRangeContext::new(PO, forest)), + Some(i22) + ); // Extending the existing live-in. - assert_eq!(lr.extend_in_ebb(e20, i23, PO), false); - PO.validate(&lr); - assert_eq!(lr.livein_local_end(e20, PO), Some(i23)); + assert_eq!(lr.extend_in_ebb(e20, i23, PO, forest), false); + PO.validate(&lr, forest); + assert_eq!( + lr.livein_local_end(e20, LiveRangeContext::new(PO, forest)), + Some(i23) + ); } #[test] @@ -663,35 +684,54 @@ mod tests { let e40 = Ebb::new(40); let i41 = Inst::new(41); let mut lr = GenLiveRange::new(v0, i11.into(), Default::default()); + let forest = &mut bforest::MapForest::new(); - assert_eq!(lr.extend_in_ebb(e30, i31, PO), true); - assert_eq!(lr.liveins.len(), 1); + assert_eq!(lr.extend_in_ebb(e30, i31, PO, forest), true); + assert_eq!( + lr.liveins(LiveRangeContext::new(PO, forest)) + .collect::>(), + [(e30, i31)] + ); // Coalesce to previous - assert_eq!(lr.extend_in_ebb(e40, i41, PO), true); - assert_eq!(lr.liveins.len(), 1); - assert_eq!(lr.liveins[0].begin, e30); - assert_eq!(lr.liveins[0].end, i41); + assert_eq!(lr.extend_in_ebb(e40, i41, PO, forest), true); + assert_eq!( + lr.liveins(LiveRangeContext::new(PO, forest)) + .collect::>(), + [(e30, i41)] + ); // Coalesce to next - assert_eq!(lr.extend_in_ebb(e20, i21, PO), true); - assert_eq!(lr.liveins.len(), 1); - assert_eq!(lr.liveins[0].begin, e20); - assert_eq!(lr.liveins[0].end, i41); + assert_eq!(lr.extend_in_ebb(e20, i21, PO, forest), true); + assert_eq!( + lr.liveins(LiveRangeContext::new(PO, forest)) + .collect::>(), + [(e20, i41)] + ); let mut lr = GenLiveRange::new(v0, i11.into(), Default::default()); - assert_eq!(lr.extend_in_ebb(e40, i41, PO), true); - assert_eq!(lr.liveins.len(), 1); + assert_eq!(lr.extend_in_ebb(e40, i41, PO, forest), true); + assert_eq!( + lr.liveins(LiveRangeContext::new(PO, forest)) + .collect::>(), + [(e40, i41)] + ); - assert_eq!(lr.extend_in_ebb(e20, i21, PO), true); - assert_eq!(lr.liveins.len(), 2); + assert_eq!(lr.extend_in_ebb(e20, i21, PO, forest), true); + assert_eq!( + lr.liveins(LiveRangeContext::new(PO, forest)) + .collect::>(), + [(e20, i21), (e40, i41)] + ); // Coalesce to previous and next - assert_eq!(lr.extend_in_ebb(e30, i31, PO), true); - assert_eq!(lr.liveins.len(), 1); - assert_eq!(lr.liveins[0].begin, e20); - assert_eq!(lr.liveins[0].end, i41); + assert_eq!(lr.extend_in_ebb(e30, i31, PO, forest), true); + assert_eq!( + lr.liveins(LiveRangeContext::new(PO, forest)) + .collect::>(), + [(e20, i41)] + ); } // TODO: Add more tests that exercise the binary search algorithm. diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 253b96c25a..0bbb4c4225 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -300,16 +300,17 @@ impl<'a> Context<'a> { for (idx, (op, &arg)) in constraints.ins.iter().zip(args).enumerate() { let mut reguse = RegUse::new(arg, idx, op.regclass.into()); let lr = &self.liveness[arg]; + let ctx = self.liveness.context(&self.cur.func.layout); match op.kind { ConstraintKind::Stack => continue, ConstraintKind::FixedReg(_) => reguse.fixed = true, ConstraintKind::Tied(_) => { // A tied operand must kill the used value. - reguse.tied = !lr.killed_at(inst, ebb, &self.cur.func.layout); + reguse.tied = !lr.killed_at(inst, ebb, ctx); } ConstraintKind::FixedTied(_) => { reguse.fixed = true; - reguse.tied = !lr.killed_at(inst, ebb, &self.cur.func.layout); + reguse.tied = !lr.killed_at(inst, ebb, ctx); } ConstraintKind::Reg => {} } diff --git a/lib/cretonne/src/verifier/cssa.rs b/lib/cretonne/src/verifier/cssa.rs index 1010b45d0f..853f27251c 100644 --- a/lib/cretonne/src/verifier/cssa.rs +++ b/lib/cretonne/src/verifier/cssa.rs @@ -89,7 +89,8 @@ impl<'a> CssaVerifier<'a> { // Knowing that values are in RPO order, we can check for interference this // way. - if self.liveness[prev_val].overlaps_def(def, def_ebb, &self.func.layout) { + let ctx = self.liveness.context(&self.func.layout); + if self.liveness[prev_val].overlaps_def(def, def_ebb, ctx) { return err!(val, "Value def in {} interferes with {}", vreg, prev_val); } } diff --git a/lib/cretonne/src/verifier/liveness.rs b/lib/cretonne/src/verifier/liveness.rs index 4e72d59224..78eb3e17ca 100644 --- a/lib/cretonne/src/verifier/liveness.rs +++ b/lib/cretonne/src/verifier/liveness.rs @@ -129,18 +129,18 @@ impl<'a> LivenessVerifier<'a> { /// Is `lr` live at the use `inst`? fn live_at_use(&self, lr: &LiveRange, inst: Inst) -> bool { - let l = &self.func.layout; + let ctx = self.liveness.context(&self.func.layout); // Check if `inst` is in the def range, not including the def itself. - if l.cmp(lr.def(), inst) == Ordering::Less && - l.cmp(inst, lr.def_local_end()) != Ordering::Greater + if ctx.order.cmp(lr.def(), inst) == Ordering::Less && + ctx.order.cmp(inst, lr.def_local_end()) != Ordering::Greater { return true; } // Otherwise see if `inst` is in one of the live-in ranges. - match lr.livein_local_end(l.inst_ebb(inst).unwrap(), l) { - Some(end) => l.cmp(inst, end) != Ordering::Greater, + match lr.livein_local_end(ctx.order.inst_ebb(inst).unwrap(), ctx) { + Some(end) => ctx.order.cmp(inst, end) != Ordering::Greater, None => false, } } @@ -205,12 +205,11 @@ impl<'a> LivenessVerifier<'a> { } // Now check the live-in intervals against the CFG. - for &livein in lr.liveins() { - let mut ebb = livein.begin; + for (mut ebb, end) in lr.liveins(self.liveness.context(l)) { if !l.is_ebb_inserted(ebb) { return err!(loc, "{} livein at {} which is not in the layout", val, ebb); } - let end_ebb = match l.inst_ebb(livein.end) { + let end_ebb = match l.inst_ebb(end) { Some(e) => e, None => { return err!( @@ -218,7 +217,7 @@ impl<'a> LivenessVerifier<'a> { "{} livein for {} ends at {} which is not in the layout", val, ebb, - livein.end + end ) } }; diff --git a/lib/cretonne/src/verifier/locations.rs b/lib/cretonne/src/verifier/locations.rs index 94958c2f89..8df40ba924 100644 --- a/lib/cretonne/src/verifier/locations.rs +++ b/lib/cretonne/src/verifier/locations.rs @@ -282,7 +282,7 @@ impl<'a> LocationVerifier<'a> { SingleDest(ebb, _) => { for d in divert.all() { let lr = &liveness[d.value]; - if lr.is_livein(ebb, &self.func.layout) { + if lr.is_livein(ebb, liveness.context(&self.func.layout)) { return err!( inst, "{} is diverted to {} and live in to {}", @@ -297,7 +297,7 @@ impl<'a> LocationVerifier<'a> { for d in divert.all() { let lr = &liveness[d.value]; for (_, ebb) in self.func.jump_tables[jt].entries() { - if lr.is_livein(ebb, &self.func.layout) { + if lr.is_livein(ebb, liveness.context(&self.func.layout)) { return err!( inst, "{} is diverted to {} and live in to {}", From 60c456c1ec0a2af69a9cfb62ace2b9ec186ea105 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 6 Dec 2017 10:56:17 -0800 Subject: [PATCH 1435/3084] Add a compilation pass timing facility. Individual compilation passes call the corresponding timing::*() function and hold on to their timing token while they run. This causes nested per-pass timing information to be recorded in thread-local storage. The --time-passes command line option prints a pass timing report to stdout. --- cranelift/src/cton-util.rs | 19 +- cranelift/src/filetest/concurrent.rs | 15 +- cranelift/src/filetest/runone.rs | 3 +- lib/cretonne/src/context.rs | 3 + lib/cretonne/src/dominator_tree.rs | 2 + lib/cretonne/src/flowgraph.rs | 4 +- lib/cretonne/src/isa/intel/mod.rs | 2 + lib/cretonne/src/isa/mod.rs | 2 + lib/cretonne/src/legalizer/mod.rs | 2 + lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/licm.rs | 2 + lib/cretonne/src/loop_analysis.rs | 2 + lib/cretonne/src/regalloc/coalescing.rs | 2 + lib/cretonne/src/regalloc/coloring.rs | 2 + lib/cretonne/src/regalloc/context.rs | 2 + lib/cretonne/src/regalloc/liveness.rs | 2 + lib/cretonne/src/regalloc/reload.rs | 2 + lib/cretonne/src/regalloc/spilling.rs | 2 + lib/cretonne/src/simple_gvn.rs | 2 + lib/cretonne/src/timing.rs | 225 ++++++++++++++++++++++++ lib/cretonne/src/unreachable_code.rs | 2 + lib/cretonne/src/verifier/cssa.rs | 2 + lib/cretonne/src/verifier/flags.rs | 2 + lib/cretonne/src/verifier/liveness.rs | 2 + lib/cretonne/src/verifier/locations.rs | 2 + lib/cretonne/src/verifier/mod.rs | 3 + lib/reader/src/parser.rs | 4 +- lib/wasm/src/func_translator.rs | 2 + lib/wasm/src/module_translator.rs | 2 + 29 files changed, 305 insertions(+), 12 deletions(-) create mode 100644 lib/cretonne/src/timing.rs diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index b08fa8cde3..fe121d0055 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -10,7 +10,7 @@ extern crate num_cpus; extern crate tempdir; extern crate term; -use cretonne::VERSION; +use cretonne::{VERSION, timing}; use docopt::Docopt; use std::io::{self, Write}; use std::process; @@ -27,16 +27,18 @@ const USAGE: &str = " Cretonne code generator utility Usage: - cton-util test [-v] ... + cton-util test [-vT] ... cton-util cat ... cton-util filecheck [-v] cton-util print-cfg ... - cton-util compile [-vp] [--set ]... [--isa ] ... - cton-util wasm [-ctvp] [--set ]... [--isa ] ... + cton-util compile [-vpT] [--set ]... [--isa ] ... + cton-util wasm [-ctvpT] [--set ]... [--isa ] ... cton-util --help | --version Options: -v, --verbose be more verbose + -T, --time-passes + print pass timing report -t, --just-decode just decode WebAssembly to Cretonne IL -c, --check-translation @@ -64,6 +66,7 @@ struct Args { flag_verbose: bool, flag_set: Vec, flag_isa: String, + flag_time_passes: bool, } /// A command either succeeds or fails with an error message. @@ -81,7 +84,7 @@ fn cton_util() -> CommandResult { .unwrap_or_else(|e| e.exit()); // Find the sub-command to execute. - if args.cmd_test { + let result = if args.cmd_test { filetest::run(args.flag_verbose, args.arg_file) } else if args.cmd_cat { cat::run(args.arg_file) @@ -104,7 +107,13 @@ fn cton_util() -> CommandResult { } else { // Debugging / shouldn't happen with proper command line handling above. Err(format!("Unhandled args: {:?}", args)) + }; + + if args.flag_time_passes { + print!("{}", timing::take_current()); } + + result } fn main() { diff --git a/cranelift/src/filetest/concurrent.rs b/cranelift/src/filetest/concurrent.rs index 4f7cca6935..cb188577e0 100644 --- a/cranelift/src/filetest/concurrent.rs +++ b/cranelift/src/filetest/concurrent.rs @@ -3,6 +3,7 @@ //! This module provides the `ConcurrentRunner` struct which uses a pool of threads to run tests //! concurrently. +use cretonne::timing; use std::panic::catch_unwind; use std::path::{Path, PathBuf}; use std::sync::mpsc::{channel, Sender, Receiver}; @@ -33,7 +34,7 @@ pub struct ConcurrentRunner { // Workers have their own `Sender`. reply_rx: Receiver, - handles: Vec>, + handles: Vec>, } impl ConcurrentRunner { @@ -64,11 +65,13 @@ impl ConcurrentRunner { } /// Join all the worker threads. + /// Transfer pass timings from the worker threads to the current thread. pub fn join(&mut self) { assert!(self.request_tx.is_none(), "must shutdown before join"); for h in self.handles.drain(..) { - if let Err(e) = h.join() { - println!("worker panicked: {:?}", e); + match h.join() { + Ok(t) => timing::add_to_current(t), + Err(e) => println!("worker panicked: {:?}", e), } } } @@ -109,7 +112,7 @@ fn worker_thread( thread_num: usize, requests: Arc>>, replies: Sender, -) -> thread::JoinHandle<()> { +) -> thread::JoinHandle { thread::Builder::new() .name(format!("worker #{}", thread_num)) .spawn(move || { @@ -142,6 +145,10 @@ fn worker_thread( replies.send(Reply::Done { jobid, result }).unwrap(); } + + // Timing is accumulated independently per thread. + // Timings from this worker thread will be aggregated by `ConcurrentRunner::join()`. + timing::take_current() }) .unwrap() } diff --git a/cranelift/src/filetest/runone.rs b/cranelift/src/filetest/runone.rs index 67ea25b08d..1dc8eb9849 100644 --- a/cranelift/src/filetest/runone.rs +++ b/cranelift/src/filetest/runone.rs @@ -6,6 +6,7 @@ use std::time; use cretonne::ir::Function; use cretonne::isa::TargetIsa; use cretonne::settings::Flags; +use cretonne::timing; use cretonne::verify_function; use cton_reader::parse_test; use cton_reader::IsaSpec; @@ -17,6 +18,7 @@ use filetest::subtest::{SubTest, Context, Result}; /// /// If running this test causes a panic, it will propagate as normal. pub fn run(path: &Path) -> TestResult { + let _tt = timing::process_file(); dbg!("---\nFile: {}", path.to_string_lossy()); let started = time::Instant::now(); let buffer = read_to_string(path).map_err(|e| e.to_string())?; @@ -71,7 +73,6 @@ pub fn run(path: &Path) -> TestResult { } - // TODO: Actually run the tests. Ok(started.elapsed()) } diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index aa900ff9e1..221481a039 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -23,6 +23,7 @@ use unreachable_code::eliminate_unreachable_code; use verifier; use simple_gvn::do_simple_gvn; use licm::do_licm; +use timing; /// Persistent data structures and compilation pipeline. pub struct Context { @@ -74,6 +75,7 @@ impl Context { /// /// Returns the size of the function's code. pub fn compile(&mut self, isa: &TargetIsa) -> Result { + let _tt = timing::compile(); self.verify_if(isa)?; self.compute_cfg(); @@ -100,6 +102,7 @@ impl Context { /// /// The machine code is not relocated. Instead, any relocations are emitted into `relocs`. pub fn emit_to_memory(&self, mem: *mut u8, relocs: &mut RelocSink, isa: &TargetIsa) { + let _tt = timing::binemit(); isa.emit_function(&self.func, &mut MemoryCodeSink::new(mem, relocs)); } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index d9f2362e3b..0da024f36d 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -5,6 +5,7 @@ use flowgraph::{ControlFlowGraph, BasicBlock}; use ir::{Ebb, Inst, Function, Layout, ProgramOrder, ExpandedProgramPoint}; use ir::instructions::BranchInfo; use packed_option::PackedOption; +use timing; use std::cmp::Ordering; @@ -227,6 +228,7 @@ impl DominatorTree { /// Reset and compute a CFG post-order and dominator tree. pub fn compute(&mut self, func: &Function, cfg: &ControlFlowGraph) { + let _tt = timing::domtree(); debug_assert!(cfg.is_valid()); self.compute_postorder(func); self.compute_domtree(func, cfg); diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index 10eff77857..d1e45312ac 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -28,13 +28,14 @@ use ir::{Function, Inst, Ebb}; use ir::instructions::BranchInfo; use entity::EntityMap; use std::mem; +use timing; /// A basic block denoted by its enclosing Ebb and last instruction. pub type BasicBlock = (Ebb, Inst); /// A container for the successors and predecessors of some Ebb. #[derive(Clone, Default)] -pub struct CFGNode { +struct CFGNode { /// Instructions that can branch or jump to this EBB. /// /// This maps branch instruction -> predecessor EBB which is redundant since the EBB containing @@ -94,6 +95,7 @@ impl ControlFlowGraph { /// /// This will clear and overwrite any information already stored in this data structure. pub fn compute(&mut self, func: &Function) { + let _tt = timing::flowgraph(); self.clear(); self.data.resize(func.dfg.num_ebbs()); diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 96048c5fbd..b4ca044c2f 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -14,6 +14,7 @@ use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; use result; +use timing; #[allow(dead_code)] @@ -115,6 +116,7 @@ impl TargetIsa for Isa { } fn prologue_epilogue(&self, func: &mut ir::Function) -> result::CtonResult { + let _tt = timing::prologue_epilogue(); abi::prologue_epilogue(func, self) } } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index e2f28ed119..ccfced715e 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -51,6 +51,7 @@ use settings; use ir; use regalloc; use result; +use timing; use isa::enc_tables::Encodings; #[cfg(build_riscv)] @@ -236,6 +237,7 @@ pub trait TargetIsa { /// /// Return an error if the stack frame is too large. fn prologue_epilogue(&self, func: &mut ir::Function) -> result::CtonResult { + let _tt = timing::prologue_epilogue(); // This default implementation is unlikely to be good enough. use stack_layout::layout_stack; use ir::stackslot::{StackSize, StackOffset}; diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 4a6ff6c337..2781f5fe84 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -18,6 +18,7 @@ use flowgraph::ControlFlowGraph; use ir::{self, InstBuilder}; use isa::TargetIsa; use bitset::BitSet; +use timing; mod boundary; mod globalvar; @@ -33,6 +34,7 @@ use self::heap::expand_heap_addr; /// - Fill out `func.encodings`. /// pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, isa: &TargetIsa) { + let _tt = timing::legalize(); debug_assert!(cfg.is_valid()); boundary::legalize_signatures(func, isa); diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 5de3c1dba3..9abffe0abc 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -28,6 +28,7 @@ pub mod packed_option; pub mod regalloc; pub mod result; pub mod settings; +pub mod timing; pub mod verifier; mod abi; diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 628f8e0aaa..1eb035d9bf 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -7,6 +7,7 @@ use std::collections::HashSet; use dominator_tree::DominatorTree; use entity::{EntityList, ListPool}; use loop_analysis::{Loop, LoopAnalysis}; +use timing; /// Performs the LICM pass by detecting loops within the CFG and moving /// loop-invariant instructions out of them. @@ -17,6 +18,7 @@ pub fn do_licm( domtree: &mut DominatorTree, loop_analysis: &mut LoopAnalysis, ) { + let _tt = timing::licm(); debug_assert!(cfg.is_valid()); debug_assert!(domtree.is_valid()); debug_assert!(loop_analysis.is_valid()); diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index 173ceb46ee..f6dee54d71 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -7,6 +7,7 @@ use entity::EntityMap; use flowgraph::ControlFlowGraph; use ir::{Function, Ebb, Layout}; use packed_option::PackedOption; +use timing; /// A opaque reference to a code loop. #[derive(Copy, Clone, PartialEq, Eq, Hash)] @@ -98,6 +99,7 @@ impl LoopAnalysis { impl LoopAnalysis { /// Detects the loops in a function. Needs the control flow graph and the dominator tree. pub fn compute(&mut self, func: &Function, cfg: &ControlFlowGraph, domtree: &DominatorTree) { + let _tt = timing::loop_analysis(); self.loops.clear(); self.ebb_loop_map.clear(); self.ebb_loop_map.resize(func.dfg.num_ebbs()); diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 31228f680a..3ce58821bc 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -18,6 +18,7 @@ use std::cmp::Ordering; use std::iter::Peekable; use std::mem; use isa::{TargetIsa, EncInfo}; +use timing; /// Dominator forest. /// @@ -282,6 +283,7 @@ impl Coalescing { liveness: &mut Liveness, virtregs: &mut VirtRegs, ) { + let _tt = timing::ra_cssa(); dbg!("Coalescing for:\n{}", func.display(isa)); let mut context = Context { isa, diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 25b271d17e..2f52dfaabc 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -57,6 +57,7 @@ use regalloc::liveness::Liveness; use regalloc::liverange::{LiveRange, LiveRangeContext}; use regalloc::solver::{Solver, SolverError}; use std::mem; +use timing; /// Data structures for the coloring pass. @@ -123,6 +124,7 @@ impl Coloring { liveness: &mut Liveness, tracker: &mut LiveValueTracker, ) { + let _tt = timing::ra_coloring(); dbg!("Coloring for:\n{}", func.display(isa)); let mut ctx = Context { usable_regs: isa.allocatable_registers(func), diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index d101cce30b..765ecf155c 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -16,6 +16,7 @@ use regalloc::reload::Reload; use regalloc::spilling::Spilling; use regalloc::virtregs::VirtRegs; use result::CtonResult; +use timing; use topo_order::TopoOrder; use verifier::{verify_context, verify_liveness, verify_cssa, verify_locations}; @@ -72,6 +73,7 @@ impl Context { cfg: &ControlFlowGraph, domtree: &mut DominatorTree, ) -> CtonResult { + let _tt = timing::regalloc(); debug_assert!(domtree.is_valid()); // `Liveness` and `Coloring` are self-clearing. diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 20208e36a1..6eee4f0c8c 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -184,6 +184,7 @@ use regalloc::affinity::Affinity; use regalloc::liverange::{LiveRange, LiveRangeForest, LiveRangeContext}; use std::mem; use std::ops::Index; +use timing; /// A set of live ranges, indexed by value number. type LiveRangeSet = SparseMap; @@ -385,6 +386,7 @@ impl Liveness { /// Compute the live ranges of all SSA values used in `func`. /// This clears out any existing analysis stored in this data structure. pub fn compute(&mut self, isa: &TargetIsa, func: &mut Function, cfg: &ControlFlowGraph) { + let _tt = timing::ra_liveness(); self.ranges.clear(); // Get ISA data structures used for computing live range affinities. diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index aa43d6771a..ab9798d4c9 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -19,6 +19,7 @@ use isa::{TargetIsa, Encoding, EncInfo, RecipeConstraints, ConstraintKind}; use regalloc::affinity::Affinity; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; +use timing; use topo_order::TopoOrder; /// Reusable data structures for the reload pass. @@ -69,6 +70,7 @@ impl Reload { topo: &mut TopoOrder, tracker: &mut LiveValueTracker, ) { + let _tt = timing::ra_reload(); dbg!("Reload for:\n{}", func.display(isa)); let mut ctx = Context { cur: EncCursor::new(func, isa), diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 0bbb4c4225..6eef88141d 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -26,6 +26,7 @@ use regalloc::liveness::Liveness; use regalloc::pressure::Pressure; use regalloc::virtregs::VirtRegs; use std::fmt; +use timing; use topo_order::TopoOrder; /// Persistent data structures for the spilling pass. @@ -87,6 +88,7 @@ impl Spilling { topo: &mut TopoOrder, tracker: &mut LiveValueTracker, ) { + let _tt = timing::ra_spilling(); dbg!("Spilling for:\n{}", func.display(isa)); let reginfo = isa.register_info(); let usable_regs = isa.allocatable_registers(func); diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index 708a287904..8642122593 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -5,6 +5,7 @@ use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; use ir::{InstructionData, Function, Inst, Opcode, Type}; use scoped_hash_map::ScopedHashMap; +use timing; /// Test whether the given opcode is unsafe to even consider for GVN. fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { @@ -16,6 +17,7 @@ fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { /// Perform simple GVN on `func`. /// pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph, domtree: &mut DominatorTree) { + let _tt = timing::gvn(); debug_assert!(cfg.is_valid()); debug_assert!(domtree.is_valid()); diff --git a/lib/cretonne/src/timing.rs b/lib/cretonne/src/timing.rs new file mode 100644 index 0000000000..e1855cf65a --- /dev/null +++ b/lib/cretonne/src/timing.rs @@ -0,0 +1,225 @@ +//! Pass timing. +//! +//! This modules provides facilities for timing the execution of individual compilation passes. + +use std::fmt; + +pub use self::details::{TimingToken, PassTimes, take_current, add_to_current}; + +// Each pass that can be timed is predefined with the `define_passes!` macro. Each pass has a +// snake_case name and a plain text description used when printing out the timing report. +// +// This macro defines: +// +// - A C-style enum containing all the pass names and a `NoPass` variant. +// - A usize constant with the number of defined passes. +// - A const array of pass descriptions. +// - A public function per pass used to start the timing of that pass. +macro_rules! define_passes { + { $enum:ident, $num_passes:ident, $descriptions:ident; + $($pass:ident: $desc:expr,)+ + } => { + #[allow(non_camel_case_types)] + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + enum $enum { $($pass,)+ NoPass } + + const $num_passes: usize = $enum::NoPass as usize; + + const $descriptions: [&str; $num_passes] = [ $($desc),+ ]; + + $( + #[doc=$desc] + pub fn $pass() -> TimingToken { + details::start_pass($enum::$pass) + } + )+ + } +} + +// Pass definitions. +define_passes!{ + Pass, NUM_PASSES, DESCRIPTIONS; + + process_file: "Processing test file", + parse_text: "Parsing textual Cretonne IL", + wasm_translate_module: "Translate WASM module", + wasm_translate_function: "Translate WASM function", + + verifier: "Verify Cretonne IL", + verify_cssa: "Verify CSSA", + verify_liveness: "Verify live ranges", + verify_locations: "Verify value locations", + verify_flags: "Verify CPU flags", + + compile: "Compilation passes", + flowgraph: "Control flow graph", + domtree: "Dominator tree", + loop_analysis: "Loop analysis", + legalize: "Legalization", + gvn: "Global value numbering", + licm: "Loop invariant code motion", + unreachable_code: "Remove unreachable blocks", + + regalloc: "Register allocation", + ra_liveness: "RA liveness analysis", + ra_cssa: "RA coalescing CSSA", + ra_spilling: "RA spilling", + ra_reload: "RA reloading", + ra_coloring: "RA coloring", + + prologue_epilogue: "Prologue/epilogue insertion", + binemit: "Binary machine code emission", +} + +impl Pass { + pub fn idx(self) -> usize { + self as usize + } +} + +impl fmt::Display for Pass { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match DESCRIPTIONS.get(self.idx()) { + Some(s) => f.write_str(s), + None => f.write_str(""), + } + } +} + + +/// Implementation details. +/// +/// This whole module can be gated on a `cfg` feature to provide a dummy implementation for +/// performance-sensitive builds or restricted environments. The dummy implementation must provide +/// `TimingToken` and `PassTimings` types and a `take_current` function. +mod details { + use super::{Pass, NUM_PASSES, DESCRIPTIONS}; + use std::cell::{Cell, RefCell}; + use std::fmt; + use std::mem; + use std::time::{Instant, Duration}; + + /// A timing token is responsible for timing the currently running pass. Timing starts when it + /// is created and ends when it is dropped. + /// + /// Multiple passes can be active at the same time, but they must be started and stopped in a + /// LIFO fashion. + pub struct TimingToken { + /// Start time for this pass. + start: Instant, + + // Pass being timed by this token. + pass: Pass, + + // The previously active pass which will be restored when this token is dropped. + prev: Pass, + } + + /// Accumulated timing information for a single pass. + #[derive(Default)] + struct PassTime { + /// Total time spent running this pas including children. + total: Duration, + + /// Time spent running in child passes. + child: Duration, + } + + /// Accumulated timing for all passes. + #[derive(Default)] + pub struct PassTimes { + pass: [PassTime; NUM_PASSES], + } + + impl fmt::Display for PassTimes { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + writeln!(f, "======= ======= ==================================")?; + writeln!(f, " Total Self Pass")?; + writeln!(f, "------- ------- ----------------------------------")?; + for (time, desc) in self.pass.iter().zip(&DESCRIPTIONS) { + // Omit passes that haven't run. + if time.total == Duration::default() { + continue; + } + + // Write a duration as secs.milis, trailing space. + fn fmtdur(mut dur: Duration, f: &mut fmt::Formatter) -> fmt::Result { + // Round to nearest ms by adding 500us. + dur += Duration::new(0, 500_000); + let ms = dur.subsec_nanos() / 1_000_000; + write!(f, "{:3}.{:03} ", dur.as_secs(), ms) + } + + fmtdur(time.total, f)?; + if let Some(s) = time.total.checked_sub(time.child) { + fmtdur(s, f)?; + } + writeln!(f, " {}", desc)?; + } + writeln!(f, "======= ======= ==================================") + } + } + + /// Information about passes in a single thread. + thread_local!{ + static CURRENT_PASS: Cell = Cell::new(Pass::NoPass); + static PASS_TIME: RefCell = RefCell::new(Default::default()); + } + + /// Start timing `pass` as a child of the currently running pass, if any. + /// + /// This function is called by the publicly exposed pass functions. + pub(super) fn start_pass(pass: Pass) -> TimingToken { + let prev = CURRENT_PASS.with(|p| p.replace(pass)); + dbg!("timing: Starting {}, (during {})", pass, prev); + TimingToken { + start: Instant::now(), + pass, + prev, + } + } + + /// Dropping a timing token indicated the end of the pass. + impl Drop for TimingToken { + fn drop(&mut self) { + let duration = self.start.elapsed(); + dbg!("timing: Ending {}", self.pass); + let old_cur = CURRENT_PASS.with(|p| p.replace(self.prev)); + assert_eq!(self.pass, old_cur, "Timing tokens dropped out of order"); + PASS_TIME.with(|rc| { + let mut table = rc.borrow_mut(); + table.pass[self.pass.idx()].total += duration; + if let Some(parent) = table.pass.get_mut(self.prev.idx()) { + parent.child += duration; + } + }) + } + } + + /// Take the current accumulated pass timings and reset the timings for the current thread. + pub fn take_current() -> PassTimes { + PASS_TIME.with(|rc| mem::replace(&mut *rc.borrow_mut(), Default::default())) + } + + /// Add `timings` to the accumulated timings for the current thread. + pub fn add_to_current(times: PassTimes) { + PASS_TIME.with(|rc| for (a, b) in rc.borrow_mut().pass.iter_mut().zip( + ×.pass, + ) + { + a.total += b.total; + a.child += b.child; + }) + } +} + +#[cfg(test)] +mod test { + use super::*; + + #[test] + fn display() { + assert_eq!(Pass::NoPass.to_string(), ""); + assert_eq!(Pass::regalloc.to_string(), "Register allocation"); + } +} diff --git a/lib/cretonne/src/unreachable_code.rs b/lib/cretonne/src/unreachable_code.rs index f5d49c77f9..36ec7e673c 100644 --- a/lib/cretonne/src/unreachable_code.rs +++ b/lib/cretonne/src/unreachable_code.rs @@ -4,6 +4,7 @@ use cursor::{Cursor, FuncCursor}; use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir; +use timing; /// Eliminate unreachable code. /// @@ -16,6 +17,7 @@ pub fn eliminate_unreachable_code( cfg: &mut ControlFlowGraph, domtree: &DominatorTree, ) { + let _tt = timing::unreachable_code(); let mut pos = FuncCursor::new(func); while let Some(ebb) = pos.next_ebb() { if domtree.is_reachable(ebb) { diff --git a/lib/cretonne/src/verifier/cssa.rs b/lib/cretonne/src/verifier/cssa.rs index 853f27251c..935279c7f1 100644 --- a/lib/cretonne/src/verifier/cssa.rs +++ b/lib/cretonne/src/verifier/cssa.rs @@ -6,6 +6,7 @@ use ir::Function; use regalloc::liveness::Liveness; use regalloc::virtregs::VirtRegs; use std::cmp::Ordering; +use timing; use verifier::Result; /// Verify conventional SSA form for `func`. @@ -29,6 +30,7 @@ pub fn verify_cssa( liveness: &Liveness, virtregs: &VirtRegs, ) -> Result { + let _tt = timing::verify_cssa(); let verifier = CssaVerifier { func, cfg, diff --git a/lib/cretonne/src/verifier/flags.rs b/lib/cretonne/src/verifier/flags.rs index f007844b51..da47fdfcf8 100644 --- a/lib/cretonne/src/verifier/flags.rs +++ b/lib/cretonne/src/verifier/flags.rs @@ -8,6 +8,7 @@ use isa; use packed_option::PackedOption; use std::result; use verifier::{Result, Error}; +use timing; /// Verify that CPU flags are used correctly. /// @@ -26,6 +27,7 @@ pub fn verify_flags( cfg: &ControlFlowGraph, isa: Option<&isa::TargetIsa>, ) -> Result { + let _tt = timing::verify_flags(); let mut verifier = FlagsVerifier { func, cfg, diff --git a/lib/cretonne/src/verifier/liveness.rs b/lib/cretonne/src/verifier/liveness.rs index 78eb3e17ca..a58d04caf7 100644 --- a/lib/cretonne/src/verifier/liveness.rs +++ b/lib/cretonne/src/verifier/liveness.rs @@ -8,6 +8,7 @@ use regalloc::liveness::Liveness; use regalloc::liverange::LiveRange; use std::cmp::Ordering; use verifier::Result; +use timing; /// Verify liveness information for `func`. /// @@ -27,6 +28,7 @@ pub fn verify_liveness( cfg: &ControlFlowGraph, liveness: &Liveness, ) -> Result { + let _tt = timing::verify_liveness(); let verifier = LivenessVerifier { isa, func, diff --git a/lib/cretonne/src/verifier/locations.rs b/lib/cretonne/src/verifier/locations.rs index 8df40ba924..2c93d0de3e 100644 --- a/lib/cretonne/src/verifier/locations.rs +++ b/lib/cretonne/src/verifier/locations.rs @@ -5,6 +5,7 @@ use isa; use regalloc::RegDiversions; use regalloc::liveness::Liveness; use verifier::Result; +use timing; /// Verify value locations for `func`. /// @@ -22,6 +23,7 @@ pub fn verify_locations( func: &ir::Function, liveness: Option<&Liveness>, ) -> Result { + let _tt = timing::verify_locations(); let verifier = LocationVerifier { isa, func, diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 259cf48a5e..ef534bcd87 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -73,6 +73,7 @@ use std::collections::BTreeSet; use std::error as std_error; use std::fmt::{self, Display, Formatter, Write}; use std::result; +use timing; pub use self::cssa::verify_cssa; pub use self::liveness::verify_liveness; @@ -126,6 +127,7 @@ pub type Result = result::Result<(), Error>; /// Verify `func`. pub fn verify_function<'a, FOI: Into>>(func: &Function, fisa: FOI) -> Result { + let _tt = timing::verifier(); Verifier::new(func, fisa.into()).run() } @@ -137,6 +139,7 @@ pub fn verify_context<'a, FOI: Into>>( domtree: &DominatorTree, fisa: FOI, ) -> Result { + let _tt = timing::verifier(); let verifier = Verifier::new(func, fisa.into()); if cfg.is_valid() { verifier.cfg_integrity(cfg)?; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index ac25f1e6e6..c897d68a3f 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -14,7 +14,7 @@ use cretonne::ir::immediates::{Imm64, Uimm32, Offset32, Ieee32, Ieee64}; use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs}; use cretonne::isa::{self, TargetIsa, Encoding, RegUnit}; -use cretonne::settings; +use cretonne::{settings, timing}; use testfile::{TestFile, Details, Comment}; use error::{Location, Error, Result}; use lexer::{self, Lexer, Token}; @@ -26,6 +26,7 @@ use sourcemap::{SourceMap, MutableSourceMap}; /// /// Any test commands or ISA declarations are ignored. pub fn parse_functions(text: &str) -> Result> { + let _tt = timing::parse_text(); parse_test(text).map(|file| { file.functions.into_iter().map(|(func, _)| func).collect() }) @@ -35,6 +36,7 @@ pub fn parse_functions(text: &str) -> Result> { /// /// The returned `TestFile` contains direct references to substrings of `text`. pub fn parse_test<'a>(text: &'a str) -> Result> { + let _tt = timing::parse_text(); let mut parser = Parser::new(text); // Gather the preamble comments as 'Function'. parser.gather_comments(AnyEntity::Function); diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index f68996d9b6..36c1b58a17 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -8,6 +8,7 @@ use code_translator::translate_operator; use cretonne::entity::EntityRef; use cretonne::ir::{self, InstBuilder, Ebb}; use cretonne::result::{CtonResult, CtonError}; +use cretonne::timing; use cton_frontend::{ILBuilder, FunctionBuilder}; use environ::FuncEnvironment; use state::TranslationState; @@ -66,6 +67,7 @@ impl FuncTranslator { func: &mut ir::Function, environ: &mut FE, ) -> CtonResult { + let _tt = timing::wasm_translate_function(); dbg!( "translate({} bytes, {}{})", reader.bytes_remaining(), diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 87e38a119c..d75f60a35e 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -1,5 +1,6 @@ //! Translation skeletton that traverses the whole WebAssembly module and call helper functions //! to deal with each part of it. +use cretonne::timing; use wasmparser::{ParserState, SectionCode, ParserInput, Parser, WasmDecoder, BinaryReaderError}; use sections_translator::{SectionParsingError, parse_function_signatures, parse_import_section, parse_function_section, parse_export_section, parse_start_section, @@ -15,6 +16,7 @@ pub fn translate_module<'data>( data: &'data [u8], environ: &mut ModuleEnvironment<'data>, ) -> Result<(), String> { + let _tt = timing::wasm_translate_module(); let mut parser = Parser::new(data); match *parser.read() { ParserState::BeginWasm { .. } => {} From 362a4bdc4c0f01ee8747940544c21f3cc582005c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 7 Dec 2017 17:50:22 -0800 Subject: [PATCH 1436/3084] Add well-known names for runtime library functions. Add a LibCall type which represents runtime library functions that many be synthesized by Cretonne from pure instructions. Add a LibCall variant to ExternalName to represent one of these runtime functions. --- lib/cretonne/src/ir/extname.rs | 57 ++++++++++++----- lib/cretonne/src/ir/libcall.rs | 114 +++++++++++++++++++++++++++++++++ lib/cretonne/src/ir/mod.rs | 2 + lib/reader/src/parser.rs | 4 +- 4 files changed, 159 insertions(+), 18 deletions(-) create mode 100644 lib/cretonne/src/ir/libcall.rs diff --git a/lib/cretonne/src/ir/extname.rs b/lib/cretonne/src/ir/extname.rs index 36cc18d5c3..f58f8d24ec 100644 --- a/lib/cretonne/src/ir/extname.rs +++ b/lib/cretonne/src/ir/extname.rs @@ -4,7 +4,10 @@ //! function. The name of an external declaration doesn't have any meaning to //! Cretonne, which compiles functions independently. +use ir::LibCall; +use std::cmp; use std::fmt::{self, Write}; +use std::str::FromStr; const TESTCASE_NAME_LENGTH: usize = 16; @@ -23,19 +26,21 @@ pub enum ExternalName { /// A name in a user-defined symbol table. Cretonne does not interpret /// these numbers in any way. User { - /// Arbitrary + /// Arbitrary. namespace: u32, - /// Arbitrary + /// Arbitrary. index: u32, }, /// A test case function name of up to 10 ascii characters. This is /// not intended to be used outside test cases. TestCase { - /// How many of the bytes in `ascii` are valid + /// How many of the bytes in `ascii` are valid? length: u8, - /// Ascii bytes of the name + /// Ascii bytes of the name. ascii: [u8; TESTCASE_NAME_LENGTH], }, + /// A well-known runtime library function. + LibCall(LibCall), } impl ExternalName { @@ -50,20 +55,12 @@ impl ExternalName { /// let name = ExternalName::testcase("hello"); /// assert_eq!(name.to_string(), "%hello"); /// ``` - pub fn testcase(v: T) -> ExternalName - where - T: Into>, - { - let vec = v.into(); - let len = if vec.len() > TESTCASE_NAME_LENGTH { - TESTCASE_NAME_LENGTH - } else { - vec.len() - }; + pub fn testcase>(v: T) -> ExternalName { + let vec = v.as_ref(); + let len = cmp::min(vec.len(), TESTCASE_NAME_LENGTH); let mut bytes = [0u8; TESTCASE_NAME_LENGTH]; - for (i, &byte) in vec.iter().take(len).enumerate() { - bytes[i] = byte; - } + bytes[0..len].copy_from_slice(&vec[0..len]); + ExternalName::TestCase { length: len as u8, ascii: bytes, @@ -104,6 +101,19 @@ impl fmt::Display for ExternalName { } Ok(()) } + ExternalName::LibCall(lc) => write!(f, "%{}", lc), + } + } +} + +impl FromStr for ExternalName { + type Err = (); + + fn from_str(s: &str) -> Result { + // Try to parse as a libcall name, otherwise it's a test case. + match s.parse() { + Ok(lc) => Ok(ExternalName::LibCall(lc)), + Err(_) => Ok(ExternalName::testcase(s.as_bytes())), } } } @@ -111,6 +121,7 @@ impl fmt::Display for ExternalName { #[cfg(test)] mod tests { use super::ExternalName; + use ir::LibCall; #[test] fn display_testcase() { @@ -137,4 +148,16 @@ mod tests { "u4294967295:4294967295" ); } + + #[test] + fn parsing() { + assert_eq!( + "FloorF32".parse(), + Ok(ExternalName::LibCall(LibCall::FloorF32)) + ); + assert_eq!( + ExternalName::LibCall(LibCall::FloorF32).to_string(), + "%FloorF32" + ); + } } diff --git a/lib/cretonne/src/ir/libcall.rs b/lib/cretonne/src/ir/libcall.rs new file mode 100644 index 0000000000..e8c848f71d --- /dev/null +++ b/lib/cretonne/src/ir/libcall.rs @@ -0,0 +1,114 @@ +//! Naming well-known routines in the runtime library. + +use ir::{types, Opcode, Type}; +use std::fmt; +use std::str::FromStr; + +/// The name of a runtime library routine. +/// +/// Runtime library calls are generated for Cretonne IL instructions that don't have an equivalent +/// ISA instruction or an easy macro expansion. A `LibCall` is used as a well-known name to refer to +/// the runtime library routine. This way, Cretonne doesn't have to know about the naming +/// convention in the embedding VM's runtime library. +/// +/// This list is likely to grow over time. +#[derive(Copy, Clone, Debug, PartialEq, Eq)] +pub enum LibCall { + /// ceil.f32 + CeilF32, + /// ceil.f64 + CeilF64, + /// floor.f32 + FloorF32, + /// floor.f64 + FloorF64, + /// trunc.f32 + TruncF32, + /// frunc.f64 + TruncF64, + /// nearest.f32 + NearestF32, + /// nearest.f64 + NearestF64, +} + +const NAME: [&str; 8] = [ + "CeilF32", + "CeilF64", + "FloorF32", + "FloorF64", + "TruncF32", + "TruncF64", + "NearestF32", + "NearestF64", +]; + +impl fmt::Display for LibCall { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + f.write_str(NAME[*self as usize]) + } +} + +impl FromStr for LibCall { + type Err = (); + + fn from_str(s: &str) -> Result { + match s { + "CeilF32" => Ok(LibCall::CeilF32), + "CeilF64" => Ok(LibCall::CeilF64), + "FloorF32" => Ok(LibCall::FloorF32), + "FloorF64" => Ok(LibCall::FloorF64), + "TruncF32" => Ok(LibCall::TruncF32), + "TruncF64" => Ok(LibCall::TruncF64), + "NearestF32" => Ok(LibCall::NearestF32), + "NearestF64" => Ok(LibCall::NearestF64), + _ => Err(()), + } + } +} + +impl LibCall { + /// Get the well-known library call name to use as a replacement for an instruction with the + /// given opcode and controlling type variable. + /// + /// Returns `None` if no well-known library routine name exists for that instruction. + pub fn for_inst(opcode: Opcode, ctrl_type: Type) -> Option { + Some(match ctrl_type { + types::F32 => { + match opcode { + Opcode::Ceil => LibCall::CeilF32, + Opcode::Floor => LibCall::FloorF32, + Opcode::Trunc => LibCall::TruncF32, + Opcode::Nearest => LibCall::NearestF32, + _ => return None, + } + } + types::F64 => { + match opcode { + Opcode::Ceil => LibCall::CeilF64, + Opcode::Floor => LibCall::FloorF64, + Opcode::Trunc => LibCall::TruncF64, + Opcode::Nearest => LibCall::NearestF64, + _ => return None, + } + } + _ => return None, + }) + } +} + +#[cfg(test)] +mod test { + use super::*; + + #[test] + fn display() { + assert_eq!(LibCall::CeilF32.to_string(), "CeilF32"); + assert_eq!(LibCall::NearestF64.to_string(), "NearestF64"); + } + + #[test] + fn parsing() { + assert_eq!("FloorF32".parse(), Ok(LibCall::FloorF32)); + } +} diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 3444a52dc6..2144ad6811 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -15,6 +15,7 @@ mod extfunc; mod extname; mod globalvar; mod heap; +mod libcall; mod memflags; mod progpoint; mod sourceloc; @@ -33,6 +34,7 @@ pub use ir::heap::{HeapData, HeapStyle, HeapBase}; pub use ir::instructions::{Opcode, InstructionData, VariableArgs, ValueList, ValueListPool}; pub use ir::jumptable::JumpTableData; pub use ir::layout::Layout; +pub use ir::libcall::LibCall; pub use ir::memflags::MemFlags; pub use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; pub use ir::sourceloc::SourceLoc; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index c897d68a3f..17f14af9b1 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -885,7 +885,9 @@ impl<'a> Parser<'a> { match self.token() { Some(Token::Name(s)) => { self.consume(); - Ok(ExternalName::testcase(s)) + s.parse().map_err( + |_| self.error("invalid test case or libcall name"), + ) } Some(Token::UserRef(namespace)) => { self.consume(); From f03729d742891b04efa924610b59b2b2e40cd77a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Dec 2017 10:22:01 -0800 Subject: [PATCH 1437/3084] Fix generated code for ISA predicates on encoding recipes. The generated code had syntax errors and inverted logic. Add an SSE 4.1 requirement to the floating point rounding instructions. --- cranelift/filetests/isa/intel/binary32-float.cton | 6 +++--- cranelift/filetests/isa/intel/binary64-float.cton | 2 +- lib/cretonne/meta/gen_encoding.py | 12 ++++++++---- lib/cretonne/meta/isa/intel/recipes.py | 2 ++ 4 files changed, 14 insertions(+), 8 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 3116207a24..9b12506b80 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -1,6 +1,6 @@ ; Binary emission of 32-bit floating point code. test binemit -isa intel has_sse2 +isa intel haswell ; The binary encodings can be verified with the command: ; @@ -52,7 +52,7 @@ ebb0: ; asm: addss %xmm2, %xmm5 [-,%xmm5] v20 = fadd v10, v11 ; bin: f3 0f 58 ea ; asm: addss %xmm5, %xmm2 - [-,%xmm2] v21 = fadd v11, v10 ; bin: f3 0f 58 d5 + [-,%xmm2] v21 = fadd v11, v10 ; bin: f3 0f 58 d5 ; asm: subss %xmm2, %xmm5 [-,%xmm5] v22 = fsub v10, v11 ; bin: f3 0f 5c ea @@ -267,7 +267,7 @@ ebb0: ; asm: addsd %xmm2, %xmm5 [-,%xmm5] v20 = fadd v10, v11 ; bin: f2 0f 58 ea ; asm: addsd %xmm5, %xmm2 - [-,%xmm2] v21 = fadd v11, v10 ; bin: f2 0f 58 d5 + [-,%xmm2] v21 = fadd v11, v10 ; bin: f2 0f 58 d5 ; asm: subsd %xmm2, %xmm5 [-,%xmm5] v22 = fsub v10, v11 ; bin: f2 0f 5c ea diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index 86a6247b2e..8ca4efd8d8 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -2,7 +2,7 @@ test binemit set is_64bit set is_compressed -isa intel has_sse2 +isa intel haswell ; The binary encodings can be verified with the command: ; diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 283b9d1e0d..46fe22a913 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -169,15 +169,19 @@ def emit_recipe_predicates(isa, fmt): # Generate the predicate function. with fmt.indented( 'fn {}({}: ::settings::PredicateView, ' - 'inst: &ir::InstructionData) -> bool {{' + '{}: &ir::InstructionData) -> bool {{' .format( name, - 'isap' if isap else '_'), '}'): + 'isap' if isap else '_', + 'inst' if instp else '_'), '}'): if isap: n = isa.settings.predicate_number[isap] - with fmt.indented('if isap.test({})'.format(n), '}'): + with fmt.indented('if !isap.test({}) {{'.format(n), '}'): fmt.line('return false;') - emit_instp(instp, fmt) + if instp: + emit_instp(instp, fmt) + else: + fmt.line('true') # Generate the static table. with fmt.indented( diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index e4ae7c74c4..a970b342aa 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -13,6 +13,7 @@ from base.formats import Ternary, FuncAddr, UnaryGlobalVar from base.formats import RegMove, RegSpill, RegFill, CopySpecial from .registers import GPR, ABCD, FPR, GPR8, FPR8, FLAG, StackGPR32, StackFPR32 from .defs import supported_floatccs +from .settings import use_sse41 try: from typing import Tuple, Dict, Sequence, Any # noqa @@ -372,6 +373,7 @@ rfurm = TailRecipe( # XX /r, RMI form for one of the roundXX SSE 4.1 instructions. furmi_rnd = TailRecipe( 'furmi_rnd', Unary, size=2, ins=FPR, outs=FPR, + isap=use_sse41, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_rr(in_reg0, out_reg0, sink); From a7eb13a1518d94c7ddbbfef4f19a7952bac21731 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Dec 2017 10:10:26 -0800 Subject: [PATCH 1438/3084] Expand unknown instructions to runtime library calls. --- .../filetests/isa/intel/legalize-libcall.cton | 15 +++++ lib/cretonne/src/legalizer/libcall.rs | 61 +++++++++++++++++++ lib/cretonne/src/legalizer/mod.rs | 9 +++ 3 files changed, 85 insertions(+) create mode 100644 cranelift/filetests/isa/intel/legalize-libcall.cton create mode 100644 lib/cretonne/src/legalizer/libcall.rs diff --git a/cranelift/filetests/isa/intel/legalize-libcall.cton b/cranelift/filetests/isa/intel/legalize-libcall.cton new file mode 100644 index 0000000000..2d249fd8df --- /dev/null +++ b/cranelift/filetests/isa/intel/legalize-libcall.cton @@ -0,0 +1,15 @@ +test legalizer + +; Pre-SSE 4.1, we need to use runtime library calls for floating point rounding operations. +set is_64bit +isa intel + +function %floor(f32) -> f32 { +ebb0(v0: f32): + v1 = floor v0 + return v1 +} +; check: function %floor(f32 [%xmm0]) -> f32 [%xmm0] native { +; check: sig0 = (f32) -> f32 native +; check: fn0 = sig0 %FloorF32 +; check: $v1 = call fn0($v0) diff --git a/lib/cretonne/src/legalizer/libcall.rs b/lib/cretonne/src/legalizer/libcall.rs new file mode 100644 index 0000000000..156a05b813 --- /dev/null +++ b/lib/cretonne/src/legalizer/libcall.rs @@ -0,0 +1,61 @@ +//! Expanding instructions as runtime library calls. + +use ir; +use ir::InstBuilder; + +/// Try to expand `inst` as a library call, returning true is successful. +pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function) -> bool { + // Does the opcode/ctrl_type combo even have a well-known runtime library name. + let libcall = + match ir::LibCall::for_inst(func.dfg[inst].opcode(), func.dfg.ctrl_typevar(inst)) { + Some(lc) => lc, + None => return false, + }; + + let funcref = find_funcref(libcall, func).unwrap_or_else(|| make_funcref(libcall, inst, func)); + + // Now we convert `inst` to a call. First save the arguments. + let mut args = Vec::new(); + args.extend_from_slice(func.dfg.inst_args(inst)); + // The replace builder will preserve the instruction result values. + func.dfg.replace(inst).call(funcref, &args); + + // TODO: ask the ISA to legalize the signature. + + true +} + +/// Get the existing function reference for `libcall` in `func` if it exists. +fn find_funcref(libcall: ir::LibCall, func: &ir::Function) -> Option { + // We're assuming that all libcall function decls are at the end. + // If we get this wrong, worst case we'll have duplicate libcall decls which is harmless. + for fref in func.dfg.ext_funcs.keys().rev() { + match func.dfg.ext_funcs[fref].name { + ir::ExternalName::LibCall(lc) => { + if lc == libcall { + return Some(fref); + } + } + _ => break, + } + } + None +} + +/// Create a funcref for `libcall` with a signature matching `inst`. +fn make_funcref(libcall: ir::LibCall, inst: ir::Inst, func: &mut ir::Function) -> ir::FuncRef { + // Start with a native calling convention. We'll give the ISA a chance to change it. + let mut sig = ir::Signature::new(ir::CallConv::Native); + for &v in func.dfg.inst_args(inst) { + sig.params.push(ir::AbiParam::new(func.dfg.value_type(v))); + } + for &v in func.dfg.inst_results(inst) { + sig.returns.push(ir::AbiParam::new(func.dfg.value_type(v))); + } + let sigref = func.import_signature(sig); + + func.import_function(ir::ExtFuncData { + name: ir::ExternalName::LibCall(libcall), + signature: sigref, + }) +} diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 2781f5fe84..8849e215e8 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -23,10 +23,12 @@ use timing; mod boundary; mod globalvar; mod heap; +mod libcall; mod split; use self::globalvar::expand_global_addr; use self::heap::expand_heap_addr; +use self::libcall::expand_as_libcall; /// Legalize `func` for `isa`. /// @@ -88,6 +90,13 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is pos.set_position(prev_pos); continue; } + + // We don't have any pattern expansion for this instruction either. + // Try converting it to a library call as a last resort. + if expand_as_libcall(inst, pos.func) { + pos.set_position(prev_pos); + continue; + } } } From 7d5f2f0404f66f814fad2e236d184cca3acd3f35 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Dec 2017 13:54:19 -0800 Subject: [PATCH 1439/3084] Convert the CFG traversal tests to file tests. Add a "cfg_postorder:" printout to the "test domtree" file tests and use that to check the computed CFG post-order instead of doing it manually with Rust code. --- cranelift/docs/testing.rst | 14 +- cranelift/filetests/cfg/loop.cton | 2 +- cranelift/filetests/cfg/traps_early.cton | 2 +- cranelift/filetests/cfg/unused_node.cton | 2 +- cranelift/filetests/domtree/basic.cton | 5 + cranelift/filetests/domtree/loops.cton | 64 +++++++ cranelift/filetests/domtree/loops2.cton | 26 +++ cranelift/src/filetest/domtree.rs | 20 +- cranelift/src/print_cfg.rs | 2 +- cranelift/tests/cfg_traversal.rs | 228 ----------------------- 10 files changed, 125 insertions(+), 240 deletions(-) delete mode 100644 cranelift/tests/cfg_traversal.rs diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index 6df29d6fd1..c7654713ea 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -225,7 +225,7 @@ instruction that produces the verifier error*. Both the error message and reported location of the error is verified:: test verifier - + function %test(i32) { ebb0(v0: i32): jump ebb1 ; error: terminator @@ -249,20 +249,20 @@ command:: ; For testing cfg generation. This code is nonsense. test print-cfg test verifier - + function %nonsense(i32, i32) -> f32 { ; check: digraph %nonsense { ; regex: I=\binst\d+\b ; check: label="{ebb0 | <$(BRZ=$I)>brz ebb2 | <$(JUMP=$I)>jump ebb1}"] - + ebb0(v1: i32, v2: i32): brz v2, ebb2 ; unordered: ebb0:$BRZ -> ebb2 v4 = iconst.i32 0 jump ebb1(v4) ; unordered: ebb0:$JUMP -> ebb1 - + ebb1(v5: i32): return v1 - + ebb2: v100 = f32const 0.0 return v100 @@ -275,7 +275,7 @@ Compute the dominator tree of each function and validate it against the ``dominates:`` annotations:: test domtree - + function %test(i32) { ebb0(v0: i32): jump ebb1 ; dominates: ebb1 @@ -293,6 +293,8 @@ Every reachable extended basic block except for the entry block has an if the ``dominates:`` annotations on the immediate dominator instructions are both correct and complete. +This test also sends the computed CFG post-order through filecheck. + `test legalizer` ---------------- diff --git a/cranelift/filetests/cfg/loop.cton b/cranelift/filetests/cfg/loop.cton index 06aa848c7f..728240940f 100644 --- a/cranelift/filetests/cfg/loop.cton +++ b/cranelift/filetests/cfg/loop.cton @@ -3,7 +3,7 @@ test print-cfg test verifier function %nonsense(i32, i32) -> f32 { -; check: digraph %nonsense { +; check: digraph "%nonsense" { ; regex: I=\binst\d+\b ; check: label="{ebb0 | <$(BRZ=$I)>brz ebb2 | <$(JUMP=$I)>jump ebb1}"] diff --git a/cranelift/filetests/cfg/traps_early.cton b/cranelift/filetests/cfg/traps_early.cton index e12c3c65ee..36f3016d5c 100644 --- a/cranelift/filetests/cfg/traps_early.cton +++ b/cranelift/filetests/cfg/traps_early.cton @@ -4,7 +4,7 @@ test print-cfg test verifier function %nonsense(i32) { -; check: digraph %nonsense { +; check: digraph "%nonsense" { ebb0(v1: i32): trap user0 ; error: terminator instruction was encountered before the end diff --git a/cranelift/filetests/cfg/unused_node.cton b/cranelift/filetests/cfg/unused_node.cton index d339e5ea83..80f2402c07 100644 --- a/cranelift/filetests/cfg/unused_node.cton +++ b/cranelift/filetests/cfg/unused_node.cton @@ -2,7 +2,7 @@ test print-cfg function %not_reached(i32) -> i32 { -; check: digraph %not_reached { +; check: digraph "%not_reached" { ; check: ebb0 [shape=record, label="{ebb0 | brnz ebb2}"] ; check: ebb1 [shape=record, label="{ebb1 | jump ebb0}"] ; check: ebb2 [shape=record, label="{ebb2}"] diff --git a/cranelift/filetests/domtree/basic.cton b/cranelift/filetests/domtree/basic.cton index e46c73e67e..94f6d88cdb 100644 --- a/cranelift/filetests/domtree/basic.cton +++ b/cranelift/filetests/domtree/basic.cton @@ -11,3 +11,8 @@ function %test(i32) { ebb3: return } +; check: cfg_postorder: +; sameln: ebb2 +; sameln: ebb3 +; sameln: ebb1 +; sameln: ebb0 diff --git a/cranelift/filetests/domtree/loops.cton b/cranelift/filetests/domtree/loops.cton index 43ad1e1c08..fd659ed421 100644 --- a/cranelift/filetests/domtree/loops.cton +++ b/cranelift/filetests/domtree/loops.cton @@ -18,3 +18,67 @@ function %test(i32) { brz v0, ebb4 return } +; Fall-through-first, prune-at-source DFT: +; +; ebb0 { +; ebb0:brz v0, ebb1 { +; ebb0:jump ebb2 { +; ebb2 { +; ebb2:brz v2, ebb2 - +; ebb2:brz v3, ebb1 - +; ebb2:brz v4, ebb4 { +; ebb2: jump ebb5 { +; ebb5 {} +; } +; ebb4 {} +; } +; } ebb2 +; } +; ebb1 { +; ebb1:jump ebb3 { +; ebb3 {} +; } +; } ebb1 +; } +; } ebb0 +; +; check: cfg_postorder: +; sameln: ebb5 +; sameln: ebb3 +; sameln: ebb4 +; sameln: ebb2 +; sameln: ebb1 +; sameln: ebb0 + +function %loop2(i32) native { + ebb0(v0: i32): + brz v0, ebb1 ; dominates: ebb1 ebb3 ebb4 ebb5 + jump ebb2 ; dominates: ebb2 + ebb1: + jump ebb3 + ebb2: + brz v0, ebb4 + jump ebb5 + ebb3: + jump ebb4 + ebb4: + brz v0, ebb3 + brnz v0, ebb5 + jump ebb6 ; dominates: ebb6 + ebb5: + brz v0, ebb4 + trap user0 + ebb6: + jump ebb7 ; dominates: ebb7 + ebb7: + return +} +; check: cfg_postorder: +; sameln: ebb5 +; sameln: ebb7 +; sameln: ebb6 +; sameln: ebb3 +; sameln: ebb4 +; sameln: ebb2 +; sameln: ebb1 +; sameln: ebb0 diff --git a/cranelift/filetests/domtree/loops2.cton b/cranelift/filetests/domtree/loops2.cton index eeac8343bd..59d98119bd 100644 --- a/cranelift/filetests/domtree/loops2.cton +++ b/cranelift/filetests/domtree/loops2.cton @@ -29,3 +29,29 @@ function %test(i32) { ebb9: return } + +function %test(i32) native { + ebb0(v0: i32): + brz v0, ebb1 ; dominates: ebb1 ebb3 ebb4 ebb5 + jump ebb2 ; dominates: ebb2 + ebb1: + jump ebb3 + ebb2: + brz v0, ebb4 + jump ebb5 + ebb3: + jump ebb4 + ebb4: + brz v0, ebb3 + jump ebb5 + ebb5: + brz v0, ebb4 + return +} +; check: cfg_postorder: +; sameln: ebb5 +; sameln: ebb3 +; sameln: ebb4 +; sameln: ebb2 +; sameln: ebb1 +; sameln: ebb0 diff --git a/cranelift/src/filetest/domtree.rs b/cranelift/src/filetest/domtree.rs index 1063429b17..97fd97bc9a 100644 --- a/cranelift/src/filetest/domtree.rs +++ b/cranelift/src/filetest/domtree.rs @@ -15,9 +15,11 @@ use cretonne::flowgraph::ControlFlowGraph; use cretonne::ir::Function; use cretonne::ir::entities::AnyEntity; use cton_reader::TestCommand; -use filetest::subtest::{SubTest, Context, Result}; +use filetest::subtest::{SubTest, Context, Result, run_filecheck}; use std::borrow::{Borrow, Cow}; use std::collections::HashMap; +use std::fmt::{self, Write}; +use std::result; use utils::match_directive; struct TestDomtree; @@ -108,6 +110,20 @@ impl SubTest for TestDomtree { } } - Ok(()) + let text = filecheck_text(&domtree).expect("formatting error"); + run_filecheck(&text, context) } } + +// Generate some output for filecheck testing +fn filecheck_text(domtree: &DominatorTree) -> result::Result { + let mut s = String::new(); + + write!(s, "cfg_postorder:")?; + for &ebb in domtree.cfg_postorder() { + write!(s, " {}", ebb)?; + } + writeln!(s, "")?; + + Ok(s) +} diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index 8d2b06c8f1..f3b958be77 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -46,7 +46,7 @@ impl<'a> CFGPrinter<'a> { } fn header(&self, w: &mut Write) -> Result { - writeln!(w, "digraph {} {{", self.func.name)?; + writeln!(w, "digraph \"{}\" {{", self.func.name)?; if let Some(entry) = self.func.layout.entry_block() { writeln!(w, " {{rank=min; {}}}", entry)?; } diff --git a/cranelift/tests/cfg_traversal.rs b/cranelift/tests/cfg_traversal.rs deleted file mode 100644 index 6709da2e9a..0000000000 --- a/cranelift/tests/cfg_traversal.rs +++ /dev/null @@ -1,228 +0,0 @@ -extern crate cretonne; -extern crate cton_reader; - -use self::cretonne::flowgraph::ControlFlowGraph; -use self::cretonne::dominator_tree::DominatorTree; -use self::cretonne::ir::Ebb; -use self::cton_reader::parse_functions; - -fn test_reverse_postorder_traversal(function_source: &str, ebb_order: Vec) { - let func = &parse_functions(function_source).unwrap()[0]; - let cfg = ControlFlowGraph::with_function(&func); - let domtree = DominatorTree::with_function(&func, &cfg); - - let got = domtree - .cfg_postorder() - .iter() - .rev() - .cloned() - .collect::>(); - let want = ebb_order - .iter() - .map(|&n| Ebb::with_number(n).unwrap()) - .collect::>(); - assert_eq!(got, want); -} - -#[test] -fn simple_traversal() { - // Fall-through-first, prune-at-source DFT: - // - // ebb0 { - // ebb0:brz v0, ebb1 { - // ebb0:jump ebb2 { - // ebb2 { - // ebb2:brz v2, ebb2 - - // ebb2:brz v3, ebb1 - - // ebb2:brz v4, ebb4 { - // ebb2: jump ebb5 { - // ebb5 {} - // } - // ebb4 {} - // } - // } ebb2 - // } - // ebb1 { - // ebb1:jump ebb3 { - // ebb3 {} - // } - // } ebb1 - // } - // } ebb0 - - test_reverse_postorder_traversal( - " - function %test(i32) native { - ebb0(v0: i32): - brz v0, ebb1 - jump ebb2 - ebb1: - jump ebb3 - ebb2: - v1 = iconst.i32 1 - v2 = iadd v1, v0 - brz v2, ebb2 - v3 = iadd v1, v2 - brz v3, ebb1 - v4 = iadd v1, v3 - brz v4, ebb4 - jump ebb5 - ebb3: - trap user0 - ebb4: - trap user0 - ebb5: - trap user0 - } - ", - vec![0, 1, 3, 2, 4, 5], - ); -} - -#[test] -fn loops_one() { - // Fall-through-first, prune-at-source DFT: - // ebb0 { - // ebb0:jump ebb1 { - // ebb1 { - // ebb1:brnz v0, ebb3 { - // ebb1:jump ebb2 { - // ebb2 { - // ebb2:jump ebb3 - - // } ebb2 - // } - // ebb3 {} - // } - // } ebb1 - // } - // } ebb0 - - test_reverse_postorder_traversal( - " - function %test(i32) native { - ebb0(v0: i32): - jump ebb1 - ebb1: - brnz v0, ebb3 - jump ebb2 - ebb2: - jump ebb3 - ebb3: - return - } - ", - vec![0, 1, 3, 2], - ); -} - -#[test] -fn loops_two() { - // Fall-through-first, prune-at-source DFT: - // ebb0 { - // ebb0:brz v0, ebb1 { - // ebb0:jump ebb2 { - // ebb2 { - // ebb2:brz v0, ebb4 { - // ebb2:jump ebb5 { - // ebb5 { - // brz v0, ebb4 - - // } ebb5 - // } - // ebb4 { - // ebb4:brz v0, ebb3 { - // ebb4:jump ebb5 - - // ebb3 { - // ebb3:jump ebb4 - - // } ebb3 - // } - // } ebb4 - // } - // } ebb2 - // } - // ebb1 { - // ebb1:jump ebb3 - - // } ebb1 - // } - // } ebb0 - - test_reverse_postorder_traversal( - " - function %test(i32) native { - ebb0(v0: i32): - brz v0, ebb1 - jump ebb2 - ebb1: - jump ebb3 - ebb2: - brz v0, ebb4 - jump ebb5 - ebb3: - jump ebb4 - ebb4: - brz v0, ebb3 - jump ebb5 - ebb5: - brz v0, ebb4 - return - } - ", - vec![0, 1, 2, 4, 3, 5], - ); -} - -#[test] -fn loops_three() { - test_reverse_postorder_traversal( - " - function %test(i32) native { - ebb0(v0: i32): - brz v0, ebb1 - jump ebb2 - ebb1: - jump ebb3 - ebb2: - brz v0, ebb4 - jump ebb5 - ebb3: - jump ebb4 - ebb4: - brz v0, ebb3 - brnz v0, ebb5 - jump ebb6 - ebb5: - brz v0, ebb4 - trap user0 - ebb6: - jump ebb7 - ebb7: - return - } - ", - vec![0, 1, 2, 4, 3, 6, 7, 5], - ); -} - -#[test] -fn back_edge_one() { - test_reverse_postorder_traversal( - " - function %test(i32) native { - ebb0(v0: i32): - brz v0, ebb1 - jump ebb2 - ebb1: - jump ebb3 - ebb2: - brz v0, ebb0 - jump ebb4 - ebb3: - brz v0, ebb2 - brnz v0, ebb0 - return - ebb4: - trap user0 - } - ", - vec![0, 1, 3, 2, 4], - ); -} From a888b2a6f1914ff40acaa055e6a44b4395b80f2a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 8 Dec 2017 14:56:16 -0800 Subject: [PATCH 1440/3084] Dominator tree pre-order. Add a DominatorTreePreorder data structure which can be initialized for a DominatorTree and used for queries involving a pre-order of the dominator tree. Print out the pre-order and send it through filecheck in "test domtree" file tests. --- cranelift/filetests/domtree/basic.cton | 7 + cranelift/filetests/domtree/loops.cton | 20 +++ cranelift/filetests/domtree/loops2.cton | 26 ++- cranelift/filetests/domtree/tall-tree.cton | 15 ++ cranelift/filetests/domtree/wide-tree.cton | 17 ++ cranelift/src/filetest/domtree.rs | 25 ++- lib/cretonne/src/dominator_tree.rs | 177 ++++++++++++++++++++- 7 files changed, 281 insertions(+), 6 deletions(-) diff --git a/cranelift/filetests/domtree/basic.cton b/cranelift/filetests/domtree/basic.cton index 94f6d88cdb..37cb20d41d 100644 --- a/cranelift/filetests/domtree/basic.cton +++ b/cranelift/filetests/domtree/basic.cton @@ -16,3 +16,10 @@ function %test(i32) { ; sameln: ebb3 ; sameln: ebb1 ; sameln: ebb0 + +; check: domtree_preorder { +; nextln: ebb0: ebb1 +; nextln: ebb1: ebb3 ebb2 +; nextln: ebb3: +; nextln: ebb2: +; nextln: } diff --git a/cranelift/filetests/domtree/loops.cton b/cranelift/filetests/domtree/loops.cton index fd659ed421..70c62d4130 100644 --- a/cranelift/filetests/domtree/loops.cton +++ b/cranelift/filetests/domtree/loops.cton @@ -50,6 +50,15 @@ function %test(i32) { ; sameln: ebb1 ; sameln: ebb0 +; check: domtree_preorder { +; nextln: ebb0: ebb1 ebb2 ebb4 ebb3 ebb5 +; nextln: ebb1: +; nextln: ebb2: +; nextln: ebb4: +; nextln: ebb3: +; nextln: ebb5: +; nextln: } + function %loop2(i32) native { ebb0(v0: i32): brz v0, ebb1 ; dominates: ebb1 ebb3 ebb4 ebb5 @@ -82,3 +91,14 @@ function %loop2(i32) native { ; sameln: ebb2 ; sameln: ebb1 ; sameln: ebb0 + +; check: domtree_preorder { +; nextln: ebb0: ebb1 ebb2 ebb4 ebb3 ebb5 +; nextln: ebb1: +; nextln: ebb2: +; nextln: ebb4: ebb6 +; nextln: ebb6: ebb7 +; nextln: ebb7: +; nextln: ebb3: +; nextln: ebb5: +; nextln: } diff --git a/cranelift/filetests/domtree/loops2.cton b/cranelift/filetests/domtree/loops2.cton index 59d98119bd..029cea922a 100644 --- a/cranelift/filetests/domtree/loops2.cton +++ b/cranelift/filetests/domtree/loops2.cton @@ -1,6 +1,6 @@ test domtree -function %test(i32) { +function %loop1(i32) { ebb0(v0: i32): brz v0, ebb1 ; dominates: ebb1 ebb6 brnz v0, ebb2 ; dominates: ebb2 ebb9 @@ -30,7 +30,20 @@ function %test(i32) { return } -function %test(i32) native { +; check: domtree_preorder { +; nextln: ebb0: ebb1 ebb2 ebb6 ebb3 ebb9 +; nextln: ebb1: +; nextln: ebb2: ebb4 ebb5 ebb7 ebb8 +; nextln: ebb4: +; nextln: ebb5: +; nextln: ebb7: +; nextln: ebb8: +; nextln: ebb6: +; nextln: ebb3: +; nextln: ebb9: +; nextln: } + +function %loop2(i32) native { ebb0(v0: i32): brz v0, ebb1 ; dominates: ebb1 ebb3 ebb4 ebb5 jump ebb2 ; dominates: ebb2 @@ -55,3 +68,12 @@ function %test(i32) native { ; sameln: ebb2 ; sameln: ebb1 ; sameln: ebb0 + +; check: domtree_preorder { +; nextln: ebb0: ebb1 ebb2 ebb4 ebb3 ebb5 +; nextln: ebb1: +; nextln: ebb2: +; nextln: ebb4: +; nextln: ebb3: +; nextln: ebb5: +; nextln: } diff --git a/cranelift/filetests/domtree/tall-tree.cton b/cranelift/filetests/domtree/tall-tree.cton index 89821d0744..cad763fc36 100644 --- a/cranelift/filetests/domtree/tall-tree.cton +++ b/cranelift/filetests/domtree/tall-tree.cton @@ -31,3 +31,18 @@ function %test(i32) { ebb11: return } + +; check: domtree_preorder { +; nextln: ebb0: ebb1 ebb2 ebb3 ebb5 +; nextln: ebb1: ebb4 +; nextln: ebb4: ebb6 ebb7 ebb10 +; nextln: ebb6: ebb8 ebb9 ebb11 +; nextln: ebb8: +; nextln: ebb9: +; nextln: ebb11: +; nextln: ebb7: +; nextln: ebb10: +; nextln: ebb2: +; nextln: ebb3: +; nextln: ebb5: +; nextln: } diff --git a/cranelift/filetests/domtree/wide-tree.cton b/cranelift/filetests/domtree/wide-tree.cton index 3a9b4fff37..ae943dd7c2 100644 --- a/cranelift/filetests/domtree/wide-tree.cton +++ b/cranelift/filetests/domtree/wide-tree.cton @@ -39,3 +39,20 @@ function %test(i32) { ebb13: return } + +; check: domtree_preorder { +; nextln: ebb0: ebb13 ebb1 +; nextln: ebb13: +; nextln: ebb1: ebb2 ebb3 ebb4 ebb5 ebb6 ebb7 +; nextln: ebb2: +; nextln: ebb3: +; nextln: ebb4: +; nextln: ebb5: +; nextln: ebb6: +; nextln: ebb7: ebb8 ebb9 ebb10 ebb12 ebb11 +; nextln: ebb8: +; nextln: ebb9: +; nextln: ebb10: +; nextln: ebb12: +; nextln: ebb11: +; nextln: } diff --git a/cranelift/src/filetest/domtree.rs b/cranelift/src/filetest/domtree.rs index 97fd97bc9a..b118dbd314 100644 --- a/cranelift/src/filetest/domtree.rs +++ b/cranelift/src/filetest/domtree.rs @@ -10,7 +10,7 @@ //! We verify that the dominator tree annotations are complete and correct. //! -use cretonne::dominator_tree::DominatorTree; +use cretonne::dominator_tree::{DominatorTree, DominatorTreePreorder}; use cretonne::flowgraph::ControlFlowGraph; use cretonne::ir::Function; use cretonne::ir::entities::AnyEntity; @@ -110,13 +110,13 @@ impl SubTest for TestDomtree { } } - let text = filecheck_text(&domtree).expect("formatting error"); + let text = filecheck_text(func, &domtree).expect("formatting error"); run_filecheck(&text, context) } } // Generate some output for filecheck testing -fn filecheck_text(domtree: &DominatorTree) -> result::Result { +fn filecheck_text(func: &Function, domtree: &DominatorTree) -> result::Result { let mut s = String::new(); write!(s, "cfg_postorder:")?; @@ -125,5 +125,24 @@ fn filecheck_text(domtree: &DominatorTree) -> result::Result } writeln!(s, "")?; + // Compute and print out a pre-order of the dominator tree. + writeln!(s, "domtree_preorder {{")?; + let mut dtpo = DominatorTreePreorder::new(); + dtpo.compute(domtree, &func.layout); + let mut stack = Vec::new(); + stack.extend(func.layout.entry_block()); + while let Some(ebb) = stack.pop() { + write!(s, " {}:", ebb)?; + let i = stack.len(); + for ch in dtpo.children(ebb) { + write!(s, " {}", ch)?; + stack.push(ch); + } + writeln!(s, "")?; + // Reverse the children we just pushed so we'll pop them in order. + stack[i..].reverse(); + } + writeln!(s, "}}")?; + Ok(s) } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 0da024f36d..96fffc5b09 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -5,6 +5,8 @@ use flowgraph::{ControlFlowGraph, BasicBlock}; use ir::{Ebb, Inst, Function, Layout, ProgramOrder, ExpandedProgramPoint}; use ir::instructions::BranchInfo; use packed_option::PackedOption; +use std::cmp; +use std::mem; use timing; use std::cmp::Ordering; @@ -83,7 +85,6 @@ impl DominatorTree { /// Compare two EBBs relative to the reverse post-order. fn rpo_cmp_ebb(&self, a: Ebb, b: Ebb) -> Ordering { - self.nodes[a].rpo_number.cmp(&self.nodes[b].rpo_number) } @@ -491,6 +492,165 @@ impl DominatorTree { } } +/// Optional pre-order information that can be computed for a dominator tree. +/// +/// This data structure is computed from a `DominatorTree` and provides: +/// +/// - A forward traversable dominator tree through the `children()` iterator. +/// - An ordering of EBBs according to a dominator tree pre-order. +/// - Constant time dominance checks at the EBB granularity. +/// +/// The information in this auxillary data structure is not easy to update when the control flow +/// graph changes, which is why it is kept separate. +pub struct DominatorTreePreorder { + nodes: EntityMap, + + // Scratch memory used by `compute_postorder()`. + stack: Vec, +} + +#[derive(Default, Clone)] +struct ExtraNode { + /// First child node in the domtree. + child: PackedOption, + + /// Next sibling node in the domtree. This linked list is ordered according to the CFG RPO. + sibling: PackedOption, + + /// Sequence number for this node in a pre-order traversal of the dominator tree. + /// Unreachable blocks have number 0, the entry block is 1. + pre_number: u32, + + /// Maximum `pre_number` for the sub-tree of the dominator tree that is rooted at this node. + /// This is always >= `pre_number`. + pre_max: u32, +} + +/// Creating and computing the dominator tree pre-order. +impl DominatorTreePreorder { + /// Create a new blank `DominatorTreePreorder`. + pub fn new() -> DominatorTreePreorder { + DominatorTreePreorder { + nodes: EntityMap::new(), + stack: Vec::new(), + } + } + + /// Recompute this data structure to match `domtree`. + pub fn compute(&mut self, domtree: &DominatorTree, layout: &Layout) { + self.nodes.clear(); + assert_eq!(self.stack.len(), 0); + + // Step 1: Populate the child and sibling links. + // + // By following the CFG post-order and pushing to the front of the lists, we make sure that + // sibling lists are ordered according to the CFG reverse post-order. + for &ebb in domtree.cfg_postorder() { + if let Some(idom_inst) = domtree.idom(ebb) { + let idom = layout.pp_ebb(idom_inst); + let sib = mem::replace(&mut self.nodes[idom].child, ebb.into()); + self.nodes[ebb].sibling = sib; + } else { + // The only EBB without an immediate dominator is the entry. + self.stack.push(ebb); + } + } + + // Step 2. Assign pre-order numbers from a DFS of the dominator tree. + assert!(self.stack.len() <= 1); + let mut n = 0; + while let Some(ebb) = self.stack.pop() { + n += 1; + let node = &mut self.nodes[ebb]; + node.pre_number = n; + node.pre_max = n; + if let Some(n) = node.sibling.expand() { + self.stack.push(n); + } + if let Some(n) = node.child.expand() { + self.stack.push(n); + } + } + + // Step 3. Propagate the `pre_max` numbers up the tree. + // The CFG post-order is topologically ordered w.r.t. dominance so a node comes after all + // its dominator tree children. + for &ebb in domtree.cfg_postorder() { + if let Some(idom_inst) = domtree.idom(ebb) { + let idom = layout.pp_ebb(idom_inst); + let pre_max = cmp::max(self.nodes[ebb].pre_max, self.nodes[idom].pre_max); + self.nodes[idom].pre_max = pre_max; + } + } + } +} + +/// An iterator that enumerates the direct children of an EBB in the dominator tree. +pub struct ChildIter<'a> { + dtpo: &'a DominatorTreePreorder, + next: PackedOption, +} + +impl<'a> Iterator for ChildIter<'a> { + type Item = Ebb; + + fn next(&mut self) -> Option { + let n = self.next.expand(); + if let Some(ebb) = n { + self.next = self.dtpo.nodes[ebb].sibling; + } + n + } +} + +/// Query interface for the dominator tree pre-order. +impl DominatorTreePreorder { + /// Get an iterator over the direct children of `ebb` in the dominator tree. + /// + /// These are the EBB's whose immediate dominator is an instruction in `ebb`, ordered according + /// to the CFG reverse post-order. + pub fn children(&self, ebb: Ebb) -> ChildIter { + ChildIter { + dtpo: self, + next: self.nodes[ebb].child, + } + } + + /// Fast, constant time dominance check with EBB granularity. + /// + /// This computes the same result as `domtree.dominates(a, b)`, but in guaranteed fast constant + /// time. This is less general than the `DominatorTree` method because it only works with EBB + /// program points. + /// + /// An EBB is considered to dominate itself. + pub fn dominates(&self, a: Ebb, b: Ebb) -> bool { + let na = &self.nodes[a]; + let nb = &self.nodes[b]; + na.pre_number <= nb.pre_number && na.pre_max >= nb.pre_max + } + + /// Compare two EBBs according to the dominator pre-order. + pub fn pre_cmp_ebb(&self, a: Ebb, b: Ebb) -> Ordering { + self.nodes[a].pre_number.cmp(&self.nodes[b].pre_number) + } + + /// Compare two program points according to the dominator tree pre-order. + /// + /// This ordering of program points have the property that given a program point, pp, all the + /// program points dominated by pp follow immediately and contiguously after pp in the order. + pub fn pre_cmp(&self, a: A, b: B, layout: &Layout) -> Ordering + where + A: Into, + B: Into, + { + let a = a.into(); + let b = b.into(); + self.pre_cmp_ebb(layout.pp_ebb(a), layout.pp_ebb(b)).then( + layout.cmp(a, b), + ) + } +} + #[cfg(test)] mod test { use cursor::{Cursor, FuncCursor}; @@ -509,6 +669,9 @@ mod test { let dtree = DominatorTree::with_function(&func, &cfg); assert_eq!(0, dtree.nodes.keys().count()); assert_eq!(dtree.cfg_postorder(), &[]); + + let mut dtpo = DominatorTreePreorder::new(); + dtpo.compute(&dtree, &func.layout); } #[test] @@ -550,6 +713,18 @@ mod test { let v2_def = cur.func.dfg.value_def(v2).unwrap_inst(); assert!(!dt.dominates(v2_def, ebb0, &cur.func.layout)); assert!(!dt.dominates(ebb0, v2_def, &cur.func.layout)); + + let mut dtpo = DominatorTreePreorder::new(); + dtpo.compute(&dt, &cur.func.layout); + assert!(dtpo.dominates(ebb0, ebb0)); + assert!(!dtpo.dominates(ebb0, ebb1)); + assert!(dtpo.dominates(ebb0, ebb2)); + assert!(!dtpo.dominates(ebb1, ebb0)); + assert!(dtpo.dominates(ebb1, ebb1)); + assert!(!dtpo.dominates(ebb1, ebb2)); + assert!(!dtpo.dominates(ebb2, ebb0)); + assert!(!dtpo.dominates(ebb2, ebb1)); + assert!(dtpo.dominates(ebb2, ebb2)); } #[test] From 88b30ff386aa04a680912ad0f1ad17bbd417957e Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Tue, 12 Dec 2017 11:57:25 -0800 Subject: [PATCH 1441/3084] refactor Reloc to an enum of every architecture's reloc types https://github.com/stoklund/cretonne/pull/206#issuecomment-350905016 --- cranelift/src/filetest/binemit.rs | 18 ++++----------- lib/cretonne/meta/isa/intel/recipes.py | 14 +++++------ lib/cretonne/meta/isa/riscv/recipes.py | 2 +- lib/cretonne/src/binemit/mod.rs | 32 ++++++++++++++++++++++++-- lib/cretonne/src/isa/arm32/binemit.rs | 2 -- lib/cretonne/src/isa/arm32/mod.rs | 4 ---- lib/cretonne/src/isa/arm64/binemit.rs | 2 -- lib/cretonne/src/isa/arm64/mod.rs | 4 ---- lib/cretonne/src/isa/intel/binemit.rs | 20 ---------------- lib/cretonne/src/isa/intel/mod.rs | 4 ---- lib/cretonne/src/isa/mod.rs | 6 ----- lib/cretonne/src/isa/riscv/binemit.rs | 14 ----------- lib/cretonne/src/isa/riscv/mod.rs | 4 ---- 13 files changed, 43 insertions(+), 83 deletions(-) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 9c19533c64..24de3d7e69 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -9,7 +9,6 @@ use std::fmt::Write; use cretonne::binemit; use cretonne::ir; use cretonne::ir::entities::AnyEntity; -use cretonne::isa::TargetIsa; use cretonne::regalloc::RegDiversions; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result}; @@ -28,16 +27,14 @@ pub fn subtest(parsed: &TestCommand) -> Result> { // Code sink that generates text. struct TextSink { - rnames: &'static [&'static str], offset: binemit::CodeOffset, text: String, } impl TextSink { /// Create a new empty TextSink. - pub fn new(isa: &TargetIsa) -> Self { + pub fn new() -> Self { Self { - rnames: isa.reloc_names(), offset: 0, text: String::new(), } @@ -72,25 +69,20 @@ impl binemit::CodeSink for TextSink { } fn reloc_ebb(&mut self, reloc: binemit::Reloc, ebb_offset: binemit::CodeOffset) { - write!( - self.text, - "{}({}) ", - self.rnames[reloc.0 as usize], - ebb_offset - ).unwrap(); + write!(self.text, "{}({}) ", reloc, ebb_offset).unwrap(); } fn reloc_external(&mut self, reloc: binemit::Reloc, name: &ir::ExternalName) { write!( self.text, "{}({}) ", - self.rnames[reloc.0 as usize], + reloc, name, ).unwrap(); } fn reloc_jt(&mut self, reloc: binemit::Reloc, jt: ir::JumpTable) { - write!(self.text, "{}({}) ", self.rnames[reloc.0 as usize], jt).unwrap(); + write!(self.text, "{}({}) ", reloc, jt).unwrap(); } } @@ -191,7 +183,7 @@ impl SubTest for TestBinEmit { } // Now emit all instructions. - let mut sink = TextSink::new(isa); + let mut sink = TextSink::new(); for ebb in func.layout.ebbs() { divert.clear(); // Correct header offsets should have been computed by `relax_branches()`. diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index a970b342aa..2afd7be588 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -521,7 +521,7 @@ fnaddr4 = TailRecipe( 'fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(RelocKind::Abs4.into(), + sink.reloc_external(Reloc::IntelAbs4, &func.dfg.ext_funcs[func_ref].name); sink.put4(0); ''') @@ -531,7 +531,7 @@ fnaddr8 = TailRecipe( 'fnaddr8', FuncAddr, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(RelocKind::Abs8.into(), + sink.reloc_external(Reloc::IntelAbs8, &func.dfg.ext_funcs[func_ref].name); sink.put8(0); ''') @@ -541,7 +541,7 @@ allones_fnaddr4 = TailRecipe( 'allones_fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(RelocKind::Abs4.into(), + sink.reloc_external(Reloc::IntelAbs4, &func.dfg.ext_funcs[func_ref].name); // Write the immediate as `!0` for the benefit of BaldrMonkey. sink.put4(!0); @@ -552,7 +552,7 @@ allones_fnaddr8 = TailRecipe( 'allones_fnaddr8', FuncAddr, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(RelocKind::Abs8.into(), + sink.reloc_external(Reloc::IntelAbs8, &func.dfg.ext_funcs[func_ref].name); // Write the immediate as `!0` for the benefit of BaldrMonkey. sink.put8(!0); @@ -563,7 +563,7 @@ gvaddr4 = TailRecipe( 'gvaddr4', UnaryGlobalVar, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(RelocKind::Abs4.into(), + sink.reloc_external(Reloc::IntelAbs4, &func.global_vars[global_var].symbol_name()); sink.put4(0); ''') @@ -573,7 +573,7 @@ gvaddr8 = TailRecipe( 'gvaddr8', UnaryGlobalVar, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(RelocKind::Abs8.into(), + sink.reloc_external(Reloc::IntelAbs8, &func.global_vars[global_var].symbol_name()); sink.put8(0); ''') @@ -848,7 +848,7 @@ call_id = TailRecipe( 'call_id', Call, size=4, ins=(), outs=(), emit=''' PUT_OP(bits, BASE_REX, sink); - sink.reloc_external(RelocKind::PCRel4.into(), + sink.reloc_external(Reloc::IntelPCRel4, &func.dfg.ext_funcs[func_ref].name); sink.put4(0); ''') diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 6874f95c36..1fbba83036 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -183,7 +183,7 @@ UJ = EncRecipe( UJcall = EncRecipe( 'UJcall', Call, size=4, ins=(), outs=(), emit=''' - sink.reloc_external(RelocKind::Call.into(), + sink.reloc_external(Reloc::RiscvCall, &func.dfg.ext_funcs[func_ref].name); // rd=%x1 is the standard link register. put_uj(bits, 0, 1, sink); diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index f216fe6514..7851e491a5 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -11,6 +11,7 @@ pub use self::memorysink::{MemoryCodeSink, RelocSink}; use ir::{ExternalName, JumpTable, Function, Inst}; use regalloc::RegDiversions; +use std::fmt; /// Offset in bytes from the beginning of the function. /// @@ -18,8 +19,35 @@ use regalloc::RegDiversions; /// depends on the *host* platform, not the *target* platform. pub type CodeOffset = u32; -/// Relocation kinds depend on the current ISA. -pub struct Reloc(pub u16); +/// Relocation kinds for every ISA +#[derive(Debug)] +pub enum Reloc { + /// Intel PC-relative 4-byte + IntelPCRel4, + /// Intel absolute 4-byte + IntelAbs4, + /// Intel absolute 8-byte + IntelAbs8, + /// Arm32 call target + Arm32Call, + /// Arm64 call target + Arm64Call, + /// RISC-V call target + RiscvCall, +} + +impl fmt::Display for Reloc { + /// Display trait implementation drops the arch, since its used in contexts where the arch is + /// already unambigious, e.g. cton syntax with isa specified. In other contexts, use Debug. + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match *self { + Reloc::IntelPCRel4 => write!(f, "{}", "PCRel4"), + Reloc::IntelAbs4 => write!(f, "{}", "Abs4"), + Reloc::IntelAbs8 => write!(f, "{}", "Abs8"), + Reloc::Arm32Call | Reloc::Arm64Call | Reloc::RiscvCall => write!(f, "{}", "Call"), + } + } +} /// Abstract interface for adding bytes to the code segment. /// diff --git a/lib/cretonne/src/isa/arm32/binemit.rs b/lib/cretonne/src/isa/arm32/binemit.rs index bbae03432c..78fa2fd657 100644 --- a/lib/cretonne/src/isa/arm32/binemit.rs +++ b/lib/cretonne/src/isa/arm32/binemit.rs @@ -5,5 +5,3 @@ use ir::{Function, Inst}; use regalloc::RegDiversions; include!(concat!(env!("OUT_DIR"), "/binemit-arm32.rs")); - -pub static RELOC_NAMES: [&'static str; 1] = ["Call"]; diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 552e159ca1..49987f1e18 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -107,8 +107,4 @@ impl TargetIsa for Isa { fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { emit_function(func, binemit::emit_inst, sink) } - - fn reloc_names(&self) -> &'static [&'static str] { - &binemit::RELOC_NAMES - } } diff --git a/lib/cretonne/src/isa/arm64/binemit.rs b/lib/cretonne/src/isa/arm64/binemit.rs index ecff1662bc..a4b79cda55 100644 --- a/lib/cretonne/src/isa/arm64/binemit.rs +++ b/lib/cretonne/src/isa/arm64/binemit.rs @@ -5,5 +5,3 @@ use ir::{Function, Inst}; use regalloc::RegDiversions; include!(concat!(env!("OUT_DIR"), "/binemit-arm64.rs")); - -pub static RELOC_NAMES: [&'static str; 1] = ["Call"]; diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 88086b233e..f8e7737c31 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -100,8 +100,4 @@ impl TargetIsa for Isa { fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { emit_function(func, binemit::emit_inst, sink) } - - fn reloc_names(&self) -> &'static [&'static str] { - &binemit::RELOC_NAMES - } } diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index f1dcdce095..ba93ce296e 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -9,26 +9,6 @@ use super::registers::RU; include!(concat!(env!("OUT_DIR"), "/binemit-intel.rs")); -/// Intel relocations. -pub enum RelocKind { - /// A 4-byte relative function reference. Based from relocation + 4 bytes. - PCRel4, - - /// A 4-byte absolute function reference. - Abs4, - - /// An 8-byte absolute function reference. - Abs8, -} - -pub static RELOC_NAMES: [&'static str; 3] = ["PCRel4", "Abs4", "Abs8"]; - -impl Into for RelocKind { - fn into(self) -> Reloc { - Reloc(self as u16) - } -} - // Convert a stack base to the corresponding register. fn stk_base(base: StackBase) -> RegUnit { let ru = match base { diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index b4ca044c2f..3123e7f0cb 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -111,10 +111,6 @@ impl TargetIsa for Isa { emit_function(func, binemit::emit_inst, sink) } - fn reloc_names(&self) -> &'static [&'static str] { - &binemit::RELOC_NAMES - } - fn prologue_epilogue(&self, func: &mut ir::Function) -> result::CtonResult { let _tt = timing::prologue_epilogue(); abi::prologue_epilogue(func, self) diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index ccfced715e..1fdaa79f09 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -272,10 +272,4 @@ pub trait TargetIsa { /// /// This is more performant than calling `emit_inst` for each instruction. fn emit_function(&self, func: &ir::Function, sink: &mut binemit::MemoryCodeSink); - - /// Get a static array of names associated with relocations in this ISA. - /// - /// This array can be indexed by the contents of `binemit::Reloc` objects passed to a - /// `CodeSink`. - fn reloc_names(&self) -> &'static [&'static str]; } diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index a6a633a524..c20b1ca004 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -9,20 +9,6 @@ use std::u32; include!(concat!(env!("OUT_DIR"), "/binemit-riscv.rs")); -/// RISC-V relocation kinds. -pub enum RelocKind { - /// A jal call to a function. - Call, -} - -pub static RELOC_NAMES: [&'static str; 1] = ["Call"]; - -impl Into for RelocKind { - fn into(self) -> Reloc { - Reloc(self as u16) - } -} - /// R-type instructions. /// /// 31 24 19 14 11 6 diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 1cb76aef41..790a788a24 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -107,10 +107,6 @@ impl TargetIsa for Isa { fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { emit_function(func, binemit::emit_inst, sink) } - - fn reloc_names(&self) -> &'static [&'static str] { - &binemit::RELOC_NAMES - } } #[cfg(test)] From 90bc798e4f9646e88c171500ede581204e5051d4 Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Tue, 12 Dec 2017 13:52:01 -0800 Subject: [PATCH 1442/3084] settings: add "is_pic" boolean setting to base --- lib/cretonne/meta/base/settings.py | 2 ++ lib/cretonne/src/settings.rs | 1 + 2 files changed, 3 insertions(+) diff --git a/lib/cretonne/meta/base/settings.py b/lib/cretonne/meta/base/settings.py index a08d5b4c3f..68df47d4eb 100644 --- a/lib/cretonne/meta/base/settings.py +++ b/lib/cretonne/meta/base/settings.py @@ -29,6 +29,8 @@ enable_verifier = BoolSetting( is_64bit = BoolSetting("Enable 64-bit code generation") +is_pic = BoolSetting("Enable Position-Independent Code generation") + return_at_end = BoolSetting( """ Generate functions with at most a single return instruction at the diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 4893ecf35a..f08bea95c2 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -347,6 +347,7 @@ mod tests { opt_level = \"default\"\n\ enable_verifier = true\n\ is_64bit = false\n\ + is_pic = false\n\ return_at_end = false\n\ is_compressed = false\n\ enable_float = true\n\ From 5834520bfebff4ff9884487e72e48225b7b5cc13 Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Tue, 12 Dec 2017 13:52:46 -0800 Subject: [PATCH 1443/3084] binemit: add PIC relocation types for Intel --- lib/cretonne/src/binemit/mod.rs | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 7851e491a5..43665d945b 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -28,6 +28,10 @@ pub enum Reloc { IntelAbs4, /// Intel absolute 8-byte IntelAbs8, + /// Intel GOT PC-relative 4-byte + IntelGotPCRel4, + /// Intel PLT-relative 4-byte + IntelPLTRel4, /// Arm32 call target Arm32Call, /// Arm64 call target @@ -44,6 +48,8 @@ impl fmt::Display for Reloc { Reloc::IntelPCRel4 => write!(f, "{}", "PCRel4"), Reloc::IntelAbs4 => write!(f, "{}", "Abs4"), Reloc::IntelAbs8 => write!(f, "{}", "Abs8"), + Reloc::IntelGotPCRel4 => write!(f, "{}", "GotPCRel4"), + Reloc::IntelPLTRel4 => write!(f, "{}", "PLTRel4"), Reloc::Arm32Call | Reloc::Arm64Call | Reloc::RiscvCall => write!(f, "{}", "Call"), } } From 6d44debc18a94754f5f850ae560c9659e14eb1aa Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Tue, 12 Dec 2017 13:54:11 -0800 Subject: [PATCH 1444/3084] intel: add PIC variants to recipes and encodings --- lib/cretonne/meta/isa/intel/encodings.py | 20 +++++++++----- lib/cretonne/meta/isa/intel/recipes.py | 34 ++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 52a56a514d..4e4ebc229c 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -2,7 +2,7 @@ Intel Encodings. """ from __future__ import absolute_import -from cdsl.predicates import IsUnsignedInt, Not +from cdsl.predicates import IsUnsignedInt, Not, And from base import instructions as base from base.formats import UnaryImm from .defs import I32, I64 @@ -11,7 +11,7 @@ from . import settings as cfg from . import instructions as x86 from .legalize import intel_expand from base.legalize import narrow, expand -from base.settings import allones_funcaddrs +from base.settings import allones_funcaddrs, is_pic from .settings import use_sse41 try: @@ -281,25 +281,33 @@ enc_both(base.regspill.f64, r.frsp32, 0x66, 0x0f, 0xd6) I32.enc(base.func_addr.i32, *r.fnaddr4(0xb8), isap=Not(allones_funcaddrs)) I64.enc(base.func_addr.i64, *r.fnaddr8.rex(0xb8, w=1), - isap=Not(allones_funcaddrs)) + isap=And(Not(allones_funcaddrs), Not(is_pic))) I32.enc(base.func_addr.i32, *r.allones_fnaddr4(0xb8), isap=allones_funcaddrs) I64.enc(base.func_addr.i64, *r.allones_fnaddr8.rex(0xb8, w=1), - isap=allones_funcaddrs) + isap=And(allones_funcaddrs, Not(is_pic))) + +I64.enc(base.func_addr.i64, *r.got_fnaddr8.rex(0x8b, w=1), + isap=is_pic) # # Global addresses. # I32.enc(base.globalsym_addr.i32, *r.gvaddr4(0xb8)) -I64.enc(base.globalsym_addr.i64, *r.gvaddr8.rex(0xb8, w=1)) +I64.enc(base.globalsym_addr.i64, *r.gvaddr8.rex(0xb8, w=1), + isap=Not(is_pic)) + +I64.enc(base.globalsym_addr.i64, *r.got_gvaddr8.rex(0x8b, w=1), + isap=is_pic) # # Call/return # I32.enc(base.call, *r.call_id(0xe8)) -I64.enc(base.call, *r.call_id(0xe8)) +I64.enc(base.call, *r.call_id(0xe8), isap=Not(is_pic)) +I64.enc(base.call, *r.call_plt_id(0xe8), isap=is_pic) I32.enc(base.call_indirect.i32, *r.call_r(0xff, rrr=2)) I64.enc(base.call_indirect.i64, *r.call_r.rex(0xff, rrr=2)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 2afd7be588..b9b7a4ec59 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -558,6 +558,19 @@ allones_fnaddr8 = TailRecipe( sink.put8(!0); ''') +got_fnaddr8 = TailRecipe( + 'got_fnaddr8', FuncAddr, size=5, ins=(), outs=GPR, + # rex2 is a hack at the moment to get one that sets the `r` bit in rex + # rm for modrm_rm is 5 to get RIP-relative addressing + emit=''' + PUT_OP(bits, rex2(0, out_reg0), sink); + modrm_rm(5, out_reg0, sink); + sink.reloc_external(Reloc::IntelGotPCRel4, + &func.dfg.ext_funcs[func_ref].name); + sink.put4(0); + ''') + + # XX+rd id with Abs4 globalsym relocation. gvaddr4 = TailRecipe( 'gvaddr4', UnaryGlobalVar, size=4, ins=(), outs=GPR, @@ -578,6 +591,18 @@ gvaddr8 = TailRecipe( sink.put8(0); ''') +# XX+rd iq with Abs8 globalsym relocation. +got_gvaddr8 = TailRecipe( + 'got_gvaddr8', UnaryGlobalVar, size=5, ins=(), outs=GPR, + emit=''' + PUT_OP(bits, rex2(0, out_reg0), sink); + modrm_rm(5, out_reg0, sink); + sink.reloc_external(Reloc::IntelGotPCRel4, + &func.global_vars[global_var].symbol_name()); + sink.put4(0); + ''') + + # # Store recipes. # @@ -853,6 +878,15 @@ call_id = TailRecipe( sink.put4(0); ''') +call_plt_id = TailRecipe( + 'call_plt_id', Call, size=4, ins=(), outs=(), + emit=''' + PUT_OP(bits, BASE_REX, sink); + sink.reloc_external(Reloc::IntelPLTRel4, + &func.dfg.ext_funcs[func_ref].name); + sink.put4(0); + ''') + call_r = TailRecipe( 'call_r', IndirectCall, size=1, ins=GPR, outs=(), emit=''' From ed81bc21be0b7e117d59905ae904c28c79720e6a Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Tue, 12 Dec 2017 13:54:59 -0800 Subject: [PATCH 1445/3084] filetests: add filetests for intel PIC encodings --- .../filetests/isa/intel/binary64-pic.cton | 54 +++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 cranelift/filetests/isa/intel/binary64-pic.cton diff --git a/cranelift/filetests/isa/intel/binary64-pic.cton b/cranelift/filetests/isa/intel/binary64-pic.cton new file mode 100644 index 0000000000..fa422741e4 --- /dev/null +++ b/cranelift/filetests/isa/intel/binary64-pic.cton @@ -0,0 +1,54 @@ +; binary emission of 64-bit code. +test binemit +set is_64bit +set is_compressed +set is_pic +isa intel haswell + +; The binary encodings can be verified with the command: +; +; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/binary64-pic.cton | llvm-mc -show-encoding -triple=x86_64 +; + +; Tests for i64 instructions. +function %I64() { + fn0 = function %foo() + sig0 = () + + gv0 = globalsym %some_gv + + ; Use incoming_arg stack slots because they won't be relocated by the frame + ; layout. + ss0 = incoming_arg 8, offset 0 + ss1 = incoming_arg 1024, offset -1024 + ss2 = incoming_arg 1024, offset -2048 + ss3 = incoming_arg 8, offset -2056 + +ebb0: + + ; asm: call foo@PLT + call fn0() ; bin: e8 PLTRel4(%foo) 00000000 + + ; asm: mov 0x0(%rip), %rax + [-,%rax] v0 = func_addr.i64 fn0 ; bin: 48 8b 05 GotPCRel4(%foo) 00000000 + ; asm: mov 0x0(%rip), %rsi + [-,%rsi] v1 = func_addr.i64 fn0 ; bin: 48 8b 35 GotPCRel4(%foo) 00000000 + ; asm: mov 0x0(%rip), %r10 + [-,%r10] v2 = func_addr.i64 fn0 ; bin: 4c 8b 15 GotPCRel4(%foo) 00000000 + + ; asm: call *%rax + call_indirect sig0, v0() ; bin: ff d0 + ; asm: call *%rsi + call_indirect sig0, v1() ; bin: ff d6 + ; asm: call *%r10 + call_indirect sig0, v2() ; bin: 41 ff d2 + + ; asm: mov 0x0(%rip), %rcx + [-,%rcx] v3 = globalsym_addr.i64 gv0 ; bin: 48 8b 0d GotPCRel4(%some_gv) 00000000 + ; asm: mov 0x0(%rip), %rsi + [-,%rsi] v4 = globalsym_addr.i64 gv0 ; bin: 48 8b 35 GotPCRel4(%some_gv) 00000000 + ; asm: mov 0x0(%rip), %r10 + [-,%r10] v5 = globalsym_addr.i64 gv0 ; bin: 4c 8b 15 GotPCRel4(%some_gv) 00000000 + + return +} From d444044e9e194960af5b81b33e1054661502b981 Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Tue, 12 Dec 2017 15:13:11 -0800 Subject: [PATCH 1446/3084] intel isa: comments to explain rip-relative addressing encoding --- lib/cretonne/meta/isa/intel/recipes.py | 6 +++--- lib/cretonne/src/isa/intel/binemit.rs | 7 +++++++ 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index b9b7a4ec59..947d5839e7 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -560,11 +560,11 @@ allones_fnaddr8 = TailRecipe( got_fnaddr8 = TailRecipe( 'got_fnaddr8', FuncAddr, size=5, ins=(), outs=GPR, - # rex2 is a hack at the moment to get one that sets the `r` bit in rex - # rm for modrm_rm is 5 to get RIP-relative addressing + # rex2 gets passed 0 for r/m register because the upper bit of + # r/m doesnt get decoded when in rip-relative addressing mode. emit=''' PUT_OP(bits, rex2(0, out_reg0), sink); - modrm_rm(5, out_reg0, sink); + modrm_riprel(out_reg0, sink); sink.reloc_external(Reloc::IntelGotPCRel4, &func.dfg.ext_funcs[func_ref].name); sink.put4(0); diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index ba93ce296e..0f07f049f3 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -180,6 +180,13 @@ fn modrm_rm(rm: RegUnit, reg: RegUnit, sink: &mut CS) { sink.put1(b); } +/// Emit a mode 00 Mod/RM byte, with a rip-relative displacement in 64-bit mode. Effective address +/// is calculated by adding displacement to 64-bit rip of next instruction. See intel Sw dev manual +/// section 2.2.1.6. +fn modrm_riprel(reg: RegUnit, sink: &mut CS) { + modrm_rm(0b101, reg, sink) +} + /// Emit a mode 01 ModR/M byte. This is a register-indirect addressing mode with 8-bit /// displacement. /// Register %rsp is invalid for `rm`. It indicates the presence of a SIB byte. From a825427786f6f508f79f292706d0d95e6d7435f4 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 13 Dec 2017 15:13:37 -0600 Subject: [PATCH 1447/3084] Avoid reloading spilled EBB arguments. The coalescer makes sure that matching EBB arguments and parameters are always in the same virtual registers, and therefore also in the same stack slot if they are spilled. This means that the reload pass should never rewrite an EBB argument if the argument value is spilled. This comes up in cases where the branch instruction needs the same value in a register: brnz v9, ebb3(v9) If the virtual register containing v9 is spilled, the branch instruction must be reloaded like: v52 = fill v9 brnz v52, ebb3(v9) The branch register argument must be rewritten, and the EBB argument must be referring to the original stack value. Fixes #208. --- cranelift/filetests/regalloc/reload-208.cton | 94 ++++++++++++++++++++ lib/cretonne/src/regalloc/reload.rs | 37 +++++--- 2 files changed, 120 insertions(+), 11 deletions(-) create mode 100644 cranelift/filetests/regalloc/reload-208.cton diff --git a/cranelift/filetests/regalloc/reload-208.cton b/cranelift/filetests/regalloc/reload-208.cton new file mode 100644 index 0000000000..29c6e09840 --- /dev/null +++ b/cranelift/filetests/regalloc/reload-208.cton @@ -0,0 +1,94 @@ +test regalloc +set is_64bit +isa intel haswell + +; regex: V=v\d+ + +; Filed as https://github.com/stoklund/cretonne/issues/208 +; +; The verifier complains about a branch argument that is not in the same virtual register as the +; corresponding EBB argument. +; +; The problem was the reload pass rewriting EBB arguments on "brnz v9, ebb3(v9)" + +function %pr208(i64 vmctx [%rdi]) native { + gv0 = vmctx-8 + heap0 = static gv0, min 0, bound 0x5000, guard 0x0040_0000 + sig0 = (i64 vmctx [%rdi]) -> i32 [%rax] native + sig1 = (i64 vmctx [%rdi], i32 [%rsi]) native + fn0 = sig0 u0:1 + fn1 = sig1 u0:3 + +ebb0(v0: i64): + v1 = iconst.i32 0 + v2 = call fn0(v0) + v20 = iconst.i32 0x4ffe + v16 = icmp uge v2, v20 + brz v16, ebb5 + trap heap_oob + +ebb5: + v17 = uextend.i64 v2 + v18 = iadd_imm.i64 v0, -8 + v19 = load.i64 v18 + v3 = iadd v19, v17 + v4 = load.i32 v3 + v21 = iconst.i32 0 + v5 = icmp eq v4, v21 + v6 = bint.i32 v5 + brnz v6, ebb2 + jump ebb3(v4) + +ebb3(v7: i32): + call fn1(v0, v7) + v26 = iconst.i32 0x4ffe + v22 = icmp uge v7, v26 + brz v22, ebb6 + trap heap_oob + +ebb6: + v23 = uextend.i64 v7 + v24 = iadd_imm.i64 v0, -8 + v25 = load.i64 v24 + v8 = iadd v25, v23 + v9 = load.i32 v8+56 + ; check: $v9 = spill + ; check: brnz $V, $ebb3($v9) + brnz v9, ebb3(v9) + jump ebb4 + +ebb4: + jump ebb2 + +ebb2: + v10 = iconst.i32 0 + v31 = iconst.i32 0x4ffe + v27 = icmp uge v10, v31 + brz v27, ebb7 + trap heap_oob + +ebb7: + v28 = uextend.i64 v10 + v29 = iadd_imm.i64 v0, -8 + v30 = load.i64 v29 + v11 = iadd v30, v28 + v12 = load.i32 v11+12 + call fn1(v0, v12) + v13 = iconst.i32 0 + v36 = iconst.i32 0x4ffe + v32 = icmp uge v13, v36 + brz v32, ebb8 + trap heap_oob + +ebb8: + v33 = uextend.i64 v13 + v34 = iadd_imm.i64 v0, -8 + v35 = load.i64 v34 + v14 = iadd v35, v33 + v15 = load.i32 v14+12 + call fn1(v0, v15) + jump ebb1 + +ebb1: + return +} diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index ab9798d4c9..8ed95628fd 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -90,6 +90,7 @@ impl Reload { /// This represents a stack value that is used by the current instruction where a register is /// needed. struct ReloadCandidate { + argidx: usize, value: Value, regclass: RegClass, } @@ -205,9 +206,10 @@ impl<'a> Context<'a> { assert!(self.candidates.is_empty()); self.find_candidates(inst, constraints); - // Insert fill instructions before `inst`. - while let Some(cand) = self.candidates.pop() { - if let Some(_reload) = self.reloads.get_mut(cand.value) { + // Insert fill instructions before `inst` and replace `cand.value` with the filled value. + for cand in self.candidates.iter_mut() { + if let Some(reload) = self.reloads.get(cand.value) { + cand.value = reload.reg; continue; } @@ -218,6 +220,7 @@ impl<'a> Context<'a> { stack: cand.value, reg: reg, }); + cand.value = reg; // Create a live range for the new reload. let affinity = Affinity::Reg(cand.regclass.into()); @@ -230,10 +233,16 @@ impl<'a> Context<'a> { ); } - // Rewrite arguments. - for arg in self.cur.func.dfg.inst_args_mut(inst) { - if let Some(reload) = self.reloads.get(*arg) { - *arg = reload.reg; + // Rewrite instruction arguments. + // + // Only rewrite those arguments that were identified as candidates. This leaves EBB + // arguments on branches as-is without rewriting them. A spilled EBB argument needs to stay + // spilled because the matching EBB parameter is going to be in the same virtual register + // and therefore the same stack slot as the EBB argument value. + if !self.candidates.is_empty() { + let args = self.cur.func.dfg.inst_args_mut(inst); + while let Some(cand) = self.candidates.pop() { + args[cand.argidx] = cand.value; } } @@ -295,10 +304,11 @@ impl<'a> Context<'a> { fn find_candidates(&mut self, inst: Inst, constraints: &RecipeConstraints) { let args = self.cur.func.dfg.inst_args(inst); - for (op, &arg) in constraints.ins.iter().zip(args) { + for (argidx, (op, &arg)) in constraints.ins.iter().zip(args).enumerate() { if op.kind != ConstraintKind::Stack { if self.liveness[arg].affinity.is_stack() { self.candidates.push(ReloadCandidate { + argidx, value: arg, regclass: op.regclass, }) @@ -307,10 +317,11 @@ impl<'a> Context<'a> { } // If we only have the fixed arguments, we're done now. - if args.len() == constraints.ins.len() { + let offset = constraints.ins.len(); + if args.len() == offset { return; } - let var_args = &args[constraints.ins.len()..]; + let var_args = &args[offset..]; // Handle ABI arguments. if let Some(sig) = self.cur.func.dfg.call_signature(inst) { @@ -318,6 +329,7 @@ impl<'a> Context<'a> { self.candidates, &self.cur.func.dfg.signatures[sig].params, var_args, + offset, self.cur.isa, self.liveness, ); @@ -326,6 +338,7 @@ impl<'a> Context<'a> { self.candidates, &self.cur.func.signature.returns, var_args, + offset, self.cur.isa, self.liveness, ); @@ -358,15 +371,17 @@ fn handle_abi_args( candidates: &mut Vec, abi_types: &[AbiParam], var_args: &[Value], + offset: usize, isa: &TargetIsa, liveness: &Liveness, ) { assert_eq!(abi_types.len(), var_args.len()); - for (abi, &arg) in abi_types.iter().zip(var_args) { + for ((abi, &arg), argidx) in abi_types.iter().zip(var_args).zip(offset..) { if abi.location.is_reg() { let lv = liveness.get(arg).expect("Missing live range for ABI arg"); if lv.affinity.is_stack() { candidates.push(ReloadCandidate { + argidx, value: arg, regclass: isa.regclass_for_abi_type(abi.value_type), }); From 2473661d49b9bc0996b8ef3254066237b98ef276 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 12 Dec 2017 16:16:01 -0600 Subject: [PATCH 1448/3084] Loosen the required order of values in a virtual register. Instead of requiring the values in a virtual register to be sorted according to the domtree.rpo_cmp() order, just require any topological ordering w.r.t. dominance. The coalescer with stop using the RPO shortly. --- lib/cretonne/src/regalloc/virtregs.rs | 9 +++---- lib/cretonne/src/verifier/cssa.rs | 39 ++++++++++++++------------- 2 files changed, 23 insertions(+), 25 deletions(-) diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index 852f1e0af0..b9388277e0 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -33,9 +33,6 @@ pub struct VirtRegs { pool: ListPool, /// The primary table of virtual registers. - /// - /// The list of values ion a virtual register is kept sorted according to the dominator tree's - /// RPO of the value defs. vregs: PrimaryMap, /// Each value belongs to at most one virtual register. @@ -65,8 +62,8 @@ impl VirtRegs { self.value_vregs[value].into() } - /// Get the list of values in `vreg`. The values are ordered according to `DomTree::rpo_cmp` of - /// their definition points. + /// Get the list of values in `vreg`. + /// The values are topologically ordered according dominance of their definition points. pub fn values(&self, vreg: VirtReg) -> &[Value] { self.vregs[vreg].as_slice(&self.pool) } @@ -103,7 +100,7 @@ impl VirtRegs { /// If a value belongs to a virtual register, all of the values in that register must be /// present. /// - /// The values are assumed to already be in RPO order. + /// The values are assumed to already be in topological order. pub fn unify(&mut self, values: &[Value]) -> VirtReg { // Start by clearing all virtual registers involved. // Pick a virtual register to reuse (the smallest number) or allocate a new one. diff --git a/lib/cretonne/src/verifier/cssa.rs b/lib/cretonne/src/verifier/cssa.rs index 935279c7f1..5c35f8811e 100644 --- a/lib/cretonne/src/verifier/cssa.rs +++ b/lib/cretonne/src/verifier/cssa.rs @@ -1,11 +1,11 @@ //! Verify conventional SSA form. +use dbg::DisplayList; use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir::Function; use regalloc::liveness::Liveness; use regalloc::virtregs::VirtRegs; -use std::cmp::Ordering; use timing; use verifier::Result; @@ -20,7 +20,7 @@ use verifier::Result; /// /// Additionally, we verify this property of virtual registers: /// -/// - The values in a virtual register are ordered according to the dominator tree's `rpo_cmp()`. +/// - The values in a virtual register are topologically ordered w.r.t. dominance. /// /// We don't verify that virtual registers are minimal. Minimal CSSA is not required. pub fn verify_cssa( @@ -67,33 +67,34 @@ impl<'a> CssaVerifier<'a> { return err!(val, "Value in {} has no live range", vreg); }; - // Check RPO ordering with the previous values in the virtual register. + // Check topological ordering with the previous values in the virtual register. let def = self.func.dfg.value_def(val).into(); let def_ebb = self.func.layout.pp_ebb(def); for &prev_val in &values[0..idx] { let prev_def = self.func.dfg.value_def(prev_val); - // Enforce RPO of defs in the virtual register. - match self.domtree.rpo_cmp(prev_def, def, &self.func.layout) { - Ordering::Less => {} - Ordering::Equal => { - return err!(val, "Value in {} has same def as {}", vreg, prev_val); - } - Ordering::Greater => { - return err!( - val, - "Value in {} in wrong order relative to {}", - vreg, - prev_val - ); - } + // Enforce topological ordering of defs in the virtual register. + if self.domtree.dominates(def, prev_def, &self.func.layout) { + return err!( + val, + "Value in {} = {} def dominates previous {}", + vreg, + DisplayList(values), + prev_val + ); } - // Knowing that values are in RPO order, we can check for interference this + // Knowing that values are in topo order, we can check for interference this // way. let ctx = self.liveness.context(&self.func.layout); if self.liveness[prev_val].overlaps_def(def, def_ebb, ctx) { - return err!(val, "Value def in {} interferes with {}", vreg, prev_val); + return err!( + val, + "Value def in {} = {} interferes with {}", + vreg, + DisplayList(values), + prev_val + ); } } } From d617d5e0f3664b6129ed0aba96d18e411f931678 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 13 Dec 2017 09:22:44 -0600 Subject: [PATCH 1449/3084] Use a domtree pre-order instead of a CFG RPO for coalescing. The stack implementation if the Budimlic dominator forest doesn't work correctly with a CFG RPO. It needs the domtree pre-order. Also handle EBB pre-order vs inst-level preorder. Manage the stack according to EBB dominance. Look for a dominating value by searching the stack. This is different from the Budimlic algorithm because we're computing the dominator tree pre-order with EBB granularity only. Fixes #207. --- .../filetests/regalloc/coalescing-207.cton | 1263 +++++++++++++++++ lib/cretonne/src/regalloc/coalescing.rs | 113 +- 2 files changed, 1343 insertions(+), 33 deletions(-) create mode 100644 cranelift/filetests/regalloc/coalescing-207.cton diff --git a/cranelift/filetests/regalloc/coalescing-207.cton b/cranelift/filetests/regalloc/coalescing-207.cton new file mode 100644 index 0000000000..9485c89f36 --- /dev/null +++ b/cranelift/filetests/regalloc/coalescing-207.cton @@ -0,0 +1,1263 @@ +test regalloc +set is_64bit +isa intel haswell + +; Reported as https://github.com/stoklund/cretonne/issues/207 +; +; The coalescer creates a virtual register with two interfering values. +function %pr207(i64 vmctx, i32, i32) -> i32 native { + gv0 = vmctx-8 + heap0 = static gv0, min 0, bound 0x5000, guard 0x0040_0000 + sig0 = (i64 vmctx, i32, i32) -> i32 native + sig1 = (i64 vmctx, i32, i32, i32) -> i32 native + sig2 = (i64 vmctx, i32, i32, i32) -> i32 native + fn0 = sig0 u0:2 + fn1 = sig1 u0:0 + fn2 = sig2 u0:1 + +ebb0(v0: i64, v1: i32, v2: i32): + v3 = iconst.i32 0 + v4 = iconst.i32 0 + v5 = iconst.i32 0 + v6 = iconst.i32 0x4ffe + v7 = icmp uge v5, v6 + brz v7, ebb1 + trap heap_oob + +ebb1: + v8 = uextend.i64 v5 + v9 = iadd_imm.i64 v0, -8 + v10 = load.i64 v9 + v11 = iadd v10, v8 + v12 = load.i32 v11+4 + v13 = iconst.i32 1056 + v14 = isub v12, v13 + v15 = iconst.i32 0x4ffe + v16 = icmp.i32 uge v4, v15 + brz v16, ebb2 + trap heap_oob + +ebb2: + v17 = uextend.i64 v4 + v18 = iadd_imm.i64 v0, -8 + v19 = load.i64 v18 + v20 = iadd v19, v17 + store.i32 v14, v20+4 + v21 = iconst.i32 0x4ffe + v22 = icmp.i32 uge v2, v21 + brz v22, ebb3 + trap heap_oob + +ebb3: + v23 = uextend.i64 v2 + v24 = iadd_imm.i64 v0, -8 + v25 = load.i64 v24 + v26 = iadd v25, v23 + v27 = sload8.i32 v26 + v28 = iconst.i32 255 + v29 = band v27, v28 + v30 = iconst.i32 0 + v31 = icmp eq v29, v30 + v32 = bint.i32 v31 + brnz v32, ebb90(v14, v1) + v33 = call fn0(v0, v1, v27) + v34 = iconst.i32 0 + v35 = iconst.i32 0 + v36 = icmp eq v33, v35 + v37 = bint.i32 v36 + brnz v37, ebb90(v14, v34) + v38 = iconst.i32 0x4ffe + v39 = icmp.i32 uge v2, v38 + brz v39, ebb4 + trap heap_oob + +ebb4: + v40 = uextend.i64 v2 + v41 = iadd_imm.i64 v0, -8 + v42 = load.i64 v41 + v43 = iadd v42, v40 + v44 = uload8.i32 v43+1 + v45 = iconst.i32 0 + v46 = icmp eq v44, v45 + v47 = bint.i32 v46 + brnz v47, ebb56(v33, v14) + v48 = iconst.i32 0x4ffe + v49 = icmp.i32 uge v33, v48 + brz v49, ebb5 + trap heap_oob + +ebb5: + v50 = uextend.i64 v33 + v51 = iadd_imm.i64 v0, -8 + v52 = load.i64 v51 + v53 = iadd v52, v50 + v54 = uload8.i32 v53+1 + v55 = iconst.i32 0 + v56 = icmp eq v54, v55 + v57 = bint.i32 v56 + brnz v57, ebb90(v14, v34) + v58 = iconst.i32 0x4ffe + v59 = icmp.i32 uge v2, v58 + brz v59, ebb6 + trap heap_oob + +ebb6: + v60 = uextend.i64 v2 + v61 = iadd_imm.i64 v0, -8 + v62 = load.i64 v61 + v63 = iadd v62, v60 + v64 = uload8.i32 v63+2 + v65 = iconst.i32 0 + v66 = icmp eq v64, v65 + v67 = bint.i32 v66 + brnz v67, ebb42 + v68 = iconst.i32 0x4ffe + v69 = icmp.i32 uge v33, v68 + brz v69, ebb7 + trap heap_oob + +ebb7: + v70 = uextend.i64 v33 + v71 = iadd_imm.i64 v0, -8 + v72 = load.i64 v71 + v73 = iadd v72, v70 + v74 = uload8.i32 v73+2 + v75 = iconst.i32 0 + v76 = icmp eq v74, v75 + v77 = bint.i32 v76 + brnz v77, ebb90(v14, v34) + v78 = iconst.i32 0x4ffe + v79 = icmp.i32 uge v2, v78 + brz v79, ebb8 + trap heap_oob + +ebb8: + v80 = uextend.i64 v2 + v81 = iadd_imm.i64 v0, -8 + v82 = load.i64 v81 + v83 = iadd v82, v80 + v84 = uload8.i32 v83+3 + v85 = iconst.i32 0 + v86 = icmp eq v84, v85 + v87 = bint.i32 v86 + brnz v87, ebb46 + v88 = iconst.i32 0x4ffe + v89 = icmp.i32 uge v33, v88 + brz v89, ebb9 + trap heap_oob + +ebb9: + v90 = uextend.i64 v33 + v91 = iadd_imm.i64 v0, -8 + v92 = load.i64 v91 + v93 = iadd v92, v90 + v94 = uload8.i32 v93+3 + v95 = iconst.i32 0 + v96 = icmp eq v94, v95 + v97 = bint.i32 v96 + brnz v97, ebb90(v14, v34) + v98 = iconst.i32 0x4ffe + v99 = icmp.i32 uge v2, v98 + brz v99, ebb10 + trap heap_oob + +ebb10: + v100 = uextend.i64 v2 + v101 = iadd_imm.i64 v0, -8 + v102 = load.i64 v101 + v103 = iadd v102, v100 + v104 = uload8.i32 v103+4 + v105 = iconst.i32 0 + v106 = icmp eq v104, v105 + v107 = bint.i32 v106 + brnz v107, ebb54 + v108 = iconst.i32 1 + v109 = iadd.i32 v2, v108 + v110 = iconst.i32 1048 + v111 = iadd.i32 v14, v110 + v112 = iconst.i64 0 + v113 = iconst.i32 0x4ffe + v114 = icmp uge v111, v113 + brz v114, ebb11 + trap heap_oob + +ebb11: + v115 = uextend.i64 v111 + v116 = iadd_imm.i64 v0, -8 + v117 = load.i64 v116 + v118 = iadd v117, v115 + store.i64 v112, v118 + v119 = iconst.i32 1040 + v120 = iadd.i32 v14, v119 + v121 = iconst.i64 0 + v122 = iconst.i32 0x4ffe + v123 = icmp uge v120, v122 + brz v123, ebb12 + trap heap_oob + +ebb12: + v124 = uextend.i64 v120 + v125 = iadd_imm.i64 v0, -8 + v126 = load.i64 v125 + v127 = iadd v126, v124 + store.i64 v121, v127 + v128 = iconst.i64 0 + v129 = iconst.i32 0x4ffe + v130 = icmp.i32 uge v14, v129 + brz v130, ebb13 + trap heap_oob + +ebb13: + v131 = uextend.i64 v14 + v132 = iadd_imm.i64 v0, -8 + v133 = load.i64 v132 + v134 = iadd v133, v131 + store.i64 v128, v134+1032 + v135 = iconst.i64 0 + v136 = iconst.i32 0x4ffe + v137 = icmp.i32 uge v14, v136 + brz v137, ebb14 + trap heap_oob + +ebb14: + v138 = uextend.i64 v14 + v139 = iadd_imm.i64 v0, -8 + v140 = load.i64 v139 + v141 = iadd v140, v138 + store.i64 v135, v141+1024 + v142 = iconst.i32 -1 + jump ebb15(v142, v27) + +ebb15(v143: i32, v144: i32): + v145 = iadd.i32 v33, v143 + v146 = iconst.i32 1 + v147 = iadd v145, v146 + v148 = iconst.i32 0x4ffe + v149 = icmp uge v147, v148 + brz v149, ebb16 + trap heap_oob + +ebb16: + v150 = uextend.i64 v147 + v151 = iadd_imm.i64 v0, -8 + v152 = load.i64 v151 + v153 = iadd v152, v150 + v154 = uload8.i32 v153 + v155 = iconst.i32 0 + v156 = icmp eq v154, v155 + v157 = bint.i32 v156 + brnz v157, ebb89(v14) + v158 = iconst.i32 255 + v159 = band.i32 v144, v158 + v160 = iconst.i32 2 + v161 = ishl v159, v160 + v162 = iadd.i32 v14, v161 + v163 = iconst.i32 2 + v164 = iadd.i32 v143, v163 + v165 = iconst.i32 0x4ffe + v166 = icmp uge v162, v165 + brz v166, ebb17 + trap heap_oob + +ebb17: + v167 = uextend.i64 v162 + v168 = iadd_imm.i64 v0, -8 + v169 = load.i64 v168 + v170 = iadd v169, v167 + store.i32 v164, v170 + v171 = iconst.i32 1024 + v172 = iadd.i32 v14, v171 + v173 = iconst.i32 3 + v174 = ushr.i32 v159, v173 + v175 = iconst.i32 28 + v176 = band v174, v175 + v177 = iadd v172, v176 + v178 = iconst.i32 0x4ffe + v179 = icmp uge v177, v178 + brz v179, ebb18 + trap heap_oob + +ebb18: + v180 = uextend.i64 v177 + v181 = iadd_imm.i64 v0, -8 + v182 = load.i64 v181 + v183 = iadd v182, v180 + v184 = load.i32 v183 + v185 = iconst.i32 1 + v186 = iconst.i32 31 + v187 = band.i32 v144, v186 + v188 = ishl v185, v187 + v189 = bor v184, v188 + v190 = iconst.i32 0x4ffe + v191 = icmp.i32 uge v177, v190 + brz v191, ebb19 + trap heap_oob + +ebb19: + v192 = uextend.i64 v177 + v193 = iadd_imm.i64 v0, -8 + v194 = load.i64 v193 + v195 = iadd v194, v192 + store.i32 v189, v195 + v196 = iadd.i32 v109, v143 + v197 = iconst.i32 1 + v198 = iadd.i32 v143, v197 + v199 = iconst.i32 1 + v200 = iadd v196, v199 + v201 = iconst.i32 0x4ffe + v202 = icmp uge v200, v201 + brz v202, ebb20 + trap heap_oob + +ebb20: + v203 = uextend.i64 v200 + v204 = iadd_imm.i64 v0, -8 + v205 = load.i64 v204 + v206 = iadd v205, v203 + v207 = uload8.i32 v206 + brnz v207, ebb15(v198, v207) + jump ebb21 + +ebb21: + v208 = iconst.i32 -1 + v209 = iconst.i32 1 + v210 = iconst.i32 -1 + v211 = iconst.i32 1 + v212 = iconst.i32 1 + v213 = iadd.i32 v198, v212 + v214 = iconst.i32 2 + v215 = icmp ult v213, v214 + v216 = bint.i32 v215 + brnz v216, ebb38(v2, v211, v209, v210, v208, v198, v213, v33, v14) + v217 = iconst.i32 -1 + v218 = iconst.i32 0 + v219 = iconst.i32 1 + v220 = iconst.i32 1 + v221 = iconst.i32 1 + v222 = copy.i32 v44 + jump ebb22(v217, v221, v44, v220, v218, v219, v213, v222, v198, v33, v14) + +ebb22(v223: i32, v224: i32, v225: i32, v226: i32, v227: i32, v228: i32, v229: i32, v230: i32, v231: i32, v232: i32, v233: i32): + v234 = copy v228 + v235 = iadd v223, v224 + v236 = iadd.i32 v2, v235 + v237 = iconst.i32 0x4ffe + v238 = icmp uge v236, v237 + brz v238, ebb23 + trap heap_oob + +ebb23: + v239 = uextend.i64 v236 + v240 = iadd_imm.i64 v0, -8 + v241 = load.i64 v240 + v242 = iadd v241, v239 + v243 = uload8.i32 v242 + v244 = iconst.i32 255 + v245 = band.i32 v225, v244 + v246 = icmp ne v243, v245 + v247 = bint.i32 v246 + brnz v247, ebb24 + v248 = icmp.i32 ne v224, v226 + v249 = bint.i32 v248 + brnz v249, ebb25 + v250 = iadd.i32 v227, v226 + v251 = iconst.i32 1 + jump ebb27(v251, v250, v223, v226) + +ebb24: + v252 = icmp.i32 ule v243, v245 + v253 = bint.i32 v252 + brnz v253, ebb26 + v254 = isub.i32 v234, v223 + v255 = iconst.i32 1 + jump ebb27(v255, v234, v223, v254) + +ebb25: + v256 = iconst.i32 1 + v257 = iadd.i32 v224, v256 + v258 = copy.i32 v227 + jump ebb27(v257, v258, v223, v226) + +ebb26: + v259 = iconst.i32 1 + v260 = iconst.i32 1 + v261 = iadd.i32 v227, v260 + v262 = iconst.i32 1 + v263 = copy.i32 v227 + jump ebb27(v259, v261, v263, v262) + +ebb27(v264: i32, v265: i32, v266: i32, v267: i32): + v268 = iadd v264, v265 + v269 = icmp uge v268, v229 + v270 = bint.i32 v269 + brnz v270, ebb29 + v271 = iadd.i32 v2, v268 + v272 = iconst.i32 0x4ffe + v273 = icmp uge v271, v272 + brz v273, ebb28 + trap heap_oob + +ebb28: + v274 = uextend.i64 v271 + v275 = iadd_imm.i64 v0, -8 + v276 = load.i64 v275 + v277 = iadd v276, v274 + v278 = uload8.i32 v277 + v279 = copy.i32 v265 + jump ebb22(v266, v264, v278, v267, v279, v268, v229, v230, v231, v232, v233) + +ebb29: + jump ebb30 + +ebb30: + v280 = iconst.i32 -1 + v281 = iconst.i32 0 + v282 = iconst.i32 1 + v283 = iconst.i32 1 + v284 = iconst.i32 1 + jump ebb31(v280, v284, v230, v283, v281, v282, v229, v267, v266, v231, v232, v233) + +ebb31(v285: i32, v286: i32, v287: i32, v288: i32, v289: i32, v290: i32, v291: i32, v292: i32, v293: i32, v294: i32, v295: i32, v296: i32): + v297 = copy v290 + v298 = iadd v285, v286 + v299 = iadd.i32 v2, v298 + v300 = iconst.i32 0x4ffe + v301 = icmp uge v299, v300 + brz v301, ebb32 + trap heap_oob + +ebb32: + v302 = uextend.i64 v299 + v303 = iadd_imm.i64 v0, -8 + v304 = load.i64 v303 + v305 = iadd v304, v302 + v306 = uload8.i32 v305 + v307 = iconst.i32 255 + v308 = band.i32 v287, v307 + v309 = icmp ne v306, v308 + v310 = bint.i32 v309 + brnz v310, ebb33 + v311 = icmp.i32 ne v286, v288 + v312 = bint.i32 v311 + brnz v312, ebb34 + v313 = iadd.i32 v289, v288 + v314 = iconst.i32 1 + jump ebb36(v314, v313, v285, v288) + +ebb33: + v315 = icmp.i32 uge v306, v308 + v316 = bint.i32 v315 + brnz v316, ebb35 + v317 = isub.i32 v297, v285 + v318 = iconst.i32 1 + jump ebb36(v318, v297, v285, v317) + +ebb34: + v319 = iconst.i32 1 + v320 = iadd.i32 v286, v319 + v321 = copy.i32 v289 + jump ebb36(v320, v321, v285, v288) + +ebb35: + v322 = iconst.i32 1 + v323 = iconst.i32 1 + v324 = iadd.i32 v289, v323 + v325 = iconst.i32 1 + v326 = copy.i32 v289 + jump ebb36(v322, v324, v326, v325) + +ebb36(v327: i32, v328: i32, v329: i32, v330: i32): + v331 = iadd v327, v328 + v332 = icmp uge v331, v291 + v333 = bint.i32 v332 + brnz v333, ebb38(v2, v330, v292, v329, v293, v294, v291, v295, v296) + v334 = iadd.i32 v2, v331 + v335 = iconst.i32 0x4ffe + v336 = icmp uge v334, v335 + brz v336, ebb37 + trap heap_oob + +ebb37: + v337 = uextend.i64 v334 + v338 = iadd_imm.i64 v0, -8 + v339 = load.i64 v338 + v340 = iadd v339, v337 + v341 = uload8.i32 v340 + v342 = copy.i32 v328 + jump ebb31(v329, v327, v341, v330, v342, v331, v291, v292, v293, v294, v295, v296) + +ebb38(v343: i32, v344: i32, v345: i32, v346: i32, v347: i32, v348: i32, v349: i32, v350: i32, v351: i32): + v352 = iconst.i32 1 + v353 = iadd v346, v352 + v354 = iconst.i32 1 + v355 = iadd v347, v354 + v356 = icmp ugt v353, v355 + v357 = bint.i32 v356 + brnz v357, ebb39(v344) + v358 = copy v345 + jump ebb39(v358) + +ebb39(v359: i32): + v360 = iadd.i32 v343, v359 + brnz.i32 v357, ebb40(v346) + v361 = copy.i32 v347 + jump ebb40(v361) + +ebb40(v362: i32): + v363 = iconst.i32 1 + v364 = iadd v362, v363 + v365 = call fn1(v0, v343, v360, v364) + v366 = iconst.i32 0 + v367 = icmp eq v365, v366 + v368 = bint.i32 v367 + brnz v368, ebb63 + v369 = iconst.i32 1 + v370 = iadd v362, v369 + v371 = isub.i32 v348, v370 + v372 = iconst.i32 1 + v373 = iadd v371, v372 + v374 = icmp ugt v362, v373 + v375 = bint.i32 v374 + v376 = copy v362 + brnz v375, ebb41(v376) + v377 = copy v373 + jump ebb41(v377) + +ebb41(v378: i32): + v379 = iconst.i32 1 + v380 = iadd v378, v379 + v381 = iconst.i32 0 + jump ebb64(v380, v381) + +ebb42: + v382 = iconst.i32 8 + v383 = ishl.i32 v29, v382 + v384 = bor v383, v44 + v385 = iconst.i32 0x4ffe + v386 = icmp.i32 uge v33, v385 + brz v386, ebb43 + trap heap_oob + +ebb43: + v387 = uextend.i64 v33 + v388 = iadd_imm.i64 v0, -8 + v389 = load.i64 v388 + v390 = iadd v389, v387 + v391 = uload8.i32 v390 + jump ebb44(v391, v54, v33) + +ebb44(v392: i32, v393: i32, v394: i32): + v395 = iconst.i32 8 + v396 = ishl v392, v395 + v397 = iconst.i32 0xff00 + v398 = band v396, v397 + v399 = iconst.i32 255 + v400 = band v393, v399 + v401 = bor v398, v400 + v402 = icmp eq v401, v384 + v403 = bint.i32 v402 + brnz v403, ebb56(v394, v14) + v404 = iconst.i32 2 + v405 = iadd v394, v404 + v406 = iconst.i32 1 + v407 = iadd v394, v406 + v408 = iconst.i32 0x4ffe + v409 = icmp uge v405, v408 + brz v409, ebb45 + trap heap_oob + +ebb45: + v410 = uextend.i64 v405 + v411 = iadd_imm.i64 v0, -8 + v412 = load.i64 v411 + v413 = iadd v412, v410 + v414 = uload8.i32 v413 + brnz v414, ebb44(v401, v414, v407) + jump ebb90(v14, v34) + +ebb46: + v415 = iconst.i32 8 + v416 = ishl.i32 v74, v415 + v417 = iconst.i32 16 + v418 = ishl.i32 v54, v417 + v419 = bor v416, v418 + v420 = iconst.i32 0x4ffe + v421 = icmp.i32 uge v33, v420 + brz v421, ebb47 + trap heap_oob + +ebb47: + v422 = uextend.i64 v33 + v423 = iadd_imm.i64 v0, -8 + v424 = load.i64 v423 + v425 = iadd v424, v422 + v426 = uload8.i32 v425 + v427 = iconst.i32 24 + v428 = ishl v426, v427 + v429 = bor.i32 v419, v428 + v430 = iconst.i32 16 + v431 = ishl.i32 v44, v430 + v432 = iconst.i32 24 + v433 = ishl.i32 v29, v432 + v434 = bor v431, v433 + v435 = iconst.i32 8 + v436 = ishl.i32 v64, v435 + v437 = bor v434, v436 + v438 = icmp eq v429, v437 + v439 = bint.i32 v438 + brnz v439, ebb56(v33, v14) + jump ebb48(v33, v429) + +ebb48(v440: i32, v441: i32): + v442 = iconst.i32 1 + v443 = iadd v440, v442 + v444 = iconst.i32 3 + v445 = iadd v440, v444 + v446 = iconst.i32 0x4ffe + v447 = icmp uge v445, v446 + brz v447, ebb49 + trap heap_oob + +ebb49: + v448 = uextend.i64 v445 + v449 = iadd_imm.i64 v0, -8 + v450 = load.i64 v449 + v451 = iadd v450, v448 + v452 = uload8.i32 v451 + v453 = iconst.i32 0 + v454 = icmp eq v452, v453 + v455 = bint.i32 v454 + brnz v455, ebb51(v14) + v456 = bor.i32 v441, v452 + v457 = iconst.i32 8 + v458 = ishl v456, v457 + v459 = icmp ne v458, v437 + v460 = bint.i32 v459 + v461 = copy.i32 v443 + brnz v460, ebb48(v461, v458) + jump ebb50 + +ebb50: + jump ebb51(v14) + +ebb51(v462: i32): + v463 = iconst.i32 0 + v464 = iconst.i32 1056 + v465 = iadd v462, v464 + v466 = iconst.i32 0x4ffe + v467 = icmp uge v463, v466 + brz v467, ebb52 + trap heap_oob + +ebb52: + v468 = uextend.i64 v463 + v469 = iadd_imm.i64 v0, -8 + v470 = load.i64 v469 + v471 = iadd v470, v468 + store.i32 v465, v471+4 + v472 = iconst.i32 0 + brnz.i32 v452, ebb53(v443) + v473 = copy v472 + jump ebb53(v473) + +ebb53(v474: i32): + return v474 + +ebb54: + v475 = iconst.i32 8 + v476 = ishl.i32 v74, v475 + v477 = iconst.i32 16 + v478 = ishl.i32 v54, v477 + v479 = bor v476, v478 + v480 = bor v479, v94 + v481 = iconst.i32 0x4ffe + v482 = icmp.i32 uge v33, v481 + brz v482, ebb55 + trap heap_oob + +ebb55: + v483 = uextend.i64 v33 + v484 = iadd_imm.i64 v0, -8 + v485 = load.i64 v484 + v486 = iadd v485, v483 + v487 = uload8.i32 v486 + v488 = iconst.i32 24 + v489 = ishl v487, v488 + v490 = bor.i32 v480, v489 + v491 = iconst.i32 16 + v492 = ishl.i32 v44, v491 + v493 = iconst.i32 24 + v494 = ishl.i32 v29, v493 + v495 = bor v492, v494 + v496 = iconst.i32 8 + v497 = ishl.i32 v64, v496 + v498 = bor v495, v497 + v499 = bor v498, v84 + v500 = icmp ne v490, v499 + v501 = bint.i32 v500 + brnz v501, ebb57 + jump ebb56(v33, v14) + +ebb56(v502: i32, v503: i32): + v504 = copy v502 + jump ebb90(v503, v504) + +ebb57: + jump ebb58(v33, v490) + +ebb58(v505: i32, v506: i32): + v507 = iconst.i32 4 + v508 = iadd v505, v507 + v509 = iconst.i32 1 + v510 = iadd v505, v509 + v511 = iconst.i32 0x4ffe + v512 = icmp uge v508, v511 + brz v512, ebb59 + trap heap_oob + +ebb59: + v513 = uextend.i64 v508 + v514 = iadd_imm.i64 v0, -8 + v515 = load.i64 v514 + v516 = iadd v515, v513 + v517 = uload8.i32 v516 + v518 = iconst.i32 0 + v519 = icmp eq v517, v518 + v520 = bint.i32 v519 + brnz v520, ebb61(v14) + v521 = iconst.i32 8 + v522 = ishl.i32 v506, v521 + v523 = bor v522, v517 + v524 = icmp ne v523, v499 + v525 = bint.i32 v524 + brnz v525, ebb58(v510, v523) + jump ebb60 + +ebb60: + jump ebb61(v14) + +ebb61(v526: i32): + v527 = iconst.i32 0 + brnz.i32 v517, ebb62(v510) + v528 = copy v527 + jump ebb62(v528) + +ebb62(v529: i32): + v530 = copy v529 + jump ebb90(v526, v530) + +ebb63: + v531 = isub.i32 v348, v359 + v532 = iconst.i32 1 + v533 = iadd v531, v532 + jump ebb64(v359, v533) + +ebb64(v534: i32, v535: i32): + v536 = iconst.i32 1 + v537 = iadd.i32 v343, v536 + v538 = iconst.i32 0 + v539 = isub v538, v362 + v540 = iconst.i32 63 + v541 = bor.i32 v349, v540 + v542 = isub.i32 v348, v534 + v543 = iconst.i32 1 + v544 = iadd v542, v543 + v545 = iconst.i32 0 + v546 = copy.i32 v350 + jump ebb65(v350, v546, v349, v541, v348, v351, v544, v534, v545, v535, v343, v364, v537, v539, v362) + +ebb65(v547: i32, v548: i32, v549: i32, v550: i32, v551: i32, v552: i32, v553: i32, v554: i32, v555: i32, v556: i32, v557: i32, v558: i32, v559: i32, v560: i32, v561: i32): + v562 = copy v556 + v563 = isub v547, v548 + v564 = icmp uge v563, v549 + v565 = bint.i32 v564 + brnz v565, ebb67(v547) + v566 = iconst.i32 0 + v567 = call fn2(v0, v547, v566, v550) + brnz v567, ebb66 + v568 = iadd v547, v550 + jump ebb67(v568) + +ebb66: + v569 = isub.i32 v567, v548 + v570 = icmp ult v569, v549 + v571 = bint.i32 v570 + brnz v571, ebb89(v552) + v572 = copy.i32 v567 + jump ebb67(v572) + +ebb67(v573: i32): + v574 = iconst.i32 1 + v575 = iadd.i32 v548, v551 + v576 = iconst.i32 0x4ffe + v577 = icmp uge v575, v576 + brz v577, ebb68 + trap heap_oob + +ebb68: + v578 = uextend.i64 v575 + v579 = iadd_imm.i64 v0, -8 + v580 = load.i64 v579 + v581 = iadd v580, v578 + v582 = uload8.i32 v581 + v583 = iconst.i32 31 + v584 = band v582, v583 + v585 = ishl.i32 v574, v584 + v586 = iconst.i32 1024 + v587 = iadd.i32 v552, v586 + v588 = iconst.i32 3 + v589 = ushr v582, v588 + v590 = iconst.i32 28 + v591 = band v589, v590 + v592 = iadd v587, v591 + v593 = iconst.i32 0x4ffe + v594 = icmp uge v592, v593 + brz v594, ebb69 + trap heap_oob + +ebb69: + v595 = uextend.i64 v592 + v596 = iadd_imm.i64 v0, -8 + v597 = load.i64 v596 + v598 = iadd v597, v595 + v599 = load.i32 v598 + v600 = band.i32 v585, v599 + v601 = iconst.i32 0 + v602 = icmp eq v600, v601 + v603 = bint.i32 v602 + brnz v603, ebb74 + v604 = iconst.i32 2 + v605 = ishl.i32 v582, v604 + v606 = iadd.i32 v552, v605 + v607 = iconst.i32 0x4ffe + v608 = icmp uge v606, v607 + brz v608, ebb70 + trap heap_oob + +ebb70: + v609 = uextend.i64 v606 + v610 = iadd_imm.i64 v0, -8 + v611 = load.i64 v610 + v612 = iadd v611, v609 + v613 = load.i32 v612 + v614 = isub.i32 v551, v613 + v615 = iconst.i32 -1 + v616 = icmp eq v614, v615 + v617 = bint.i32 v616 + brnz v617, ebb75 + v618 = iconst.i32 1 + v619 = iadd v614, v618 + v620 = icmp ult v619, v554 + v621 = bint.i32 v620 + v622 = copy.i32 v553 + brnz v621, ebb71(v622) + v623 = copy v619 + jump ebb71(v623) + +ebb71(v624: i32): + v625 = copy v624 + brnz.i32 v555, ebb72(v625) + jump ebb72(v619) + +ebb72(v626: i32): + brnz.i32 v562, ebb73(v626) + jump ebb73(v619) + +ebb73(v627: i32): + v628 = copy.i32 v554 + v629 = copy.i32 v562 + jump ebb87(v548, v627, v573, v549, v550, v551, v552, v553, v628, v629, v557, v558, v559, v560, v561) + +ebb74: + v630 = copy.i32 v549 + v631 = copy.i32 v554 + v632 = copy.i32 v562 + jump ebb87(v548, v630, v573, v549, v550, v551, v552, v553, v631, v632, v557, v558, v559, v560, v561) + +ebb75: + v633 = icmp.i32 ugt v558, v555 + v634 = bint.i32 v633 + v635 = copy.i32 v558 + brnz v634, ebb76(v635) + v636 = copy.i32 v555 + jump ebb76(v636) + +ebb76(v637: i32): + v638 = iadd.i32 v557, v637 + v639 = iconst.i32 0x4ffe + v640 = icmp uge v638, v639 + brz v640, ebb77 + trap heap_oob + +ebb77: + v641 = uextend.i64 v638 + v642 = iadd_imm.i64 v0, -8 + v643 = load.i64 v642 + v644 = iadd v643, v641 + v645 = uload8.i32 v644 + v646 = iconst.i32 0 + v647 = icmp eq v645, v646 + v648 = bint.i32 v647 + brnz v648, ebb82(v548, v549, v551, v552) + v649 = iadd.i32 v548, v637 + v650 = iadd.i32 v559, v637 + v651 = iadd.i32 v560, v637 + jump ebb78(v645, v649, v651, v650) + +ebb78(v652: i32, v653: i32, v654: i32, v655: i32): + v656 = iconst.i32 255 + v657 = band v652, v656 + v658 = iconst.i32 0x4ffe + v659 = icmp uge v653, v658 + brz v659, ebb79 + trap heap_oob + +ebb79: + v660 = uextend.i64 v653 + v661 = iadd_imm.i64 v0, -8 + v662 = load.i64 v661 + v663 = iadd v662, v660 + v664 = uload8.i32 v663 + v665 = icmp.i32 ne v657, v664 + v666 = bint.i32 v665 + v667 = copy.i32 v554 + v668 = copy.i32 v562 + brnz v666, ebb87(v548, v654, v573, v549, v550, v551, v552, v553, v667, v668, v557, v558, v559, v560, v561) + v669 = iconst.i32 1 + v670 = iadd.i32 v653, v669 + v671 = iconst.i32 1 + v672 = iadd.i32 v654, v671 + v673 = iconst.i32 0x4ffe + v674 = icmp.i32 uge v655, v673 + brz v674, ebb80 + trap heap_oob + +ebb80: + v675 = uextend.i64 v655 + v676 = iadd_imm.i64 v0, -8 + v677 = load.i64 v676 + v678 = iadd v677, v675 + v679 = uload8.i32 v678 + v680 = iconst.i32 1 + v681 = iadd.i32 v655, v680 + brnz v679, ebb78(v679, v670, v672, v681) + jump ebb81 + +ebb81: + jump ebb82(v548, v549, v551, v552) + +ebb82(v682: i32, v683: i32, v684: i32, v685: i32): + v686 = icmp.i32 ule v558, v555 + v687 = bint.i32 v686 + brnz v687, ebb90(v685, v682) + v688 = copy.i32 v561 + jump ebb83(v688) + +ebb83(v689: i32): + v690 = iadd.i32 v557, v689 + v691 = iconst.i32 0x4ffe + v692 = icmp uge v690, v691 + brz v692, ebb84 + trap heap_oob + +ebb84: + v693 = uextend.i64 v690 + v694 = iadd_imm.i64 v0, -8 + v695 = load.i64 v694 + v696 = iadd v695, v693 + v697 = uload8.i32 v696 + v698 = iadd.i32 v682, v689 + v699 = iconst.i32 0x4ffe + v700 = icmp uge v698, v699 + brz v700, ebb85 + trap heap_oob + +ebb85: + v701 = uextend.i64 v698 + v702 = iadd_imm.i64 v0, -8 + v703 = load.i64 v702 + v704 = iadd v703, v701 + v705 = uload8.i32 v704 + v706 = icmp.i32 ne v697, v705 + v707 = bint.i32 v706 + brnz v707, ebb86 + v708 = icmp.i32 ule v689, v555 + v709 = bint.i32 v708 + v710 = iconst.i32 -1 + v711 = iadd.i32 v689, v710 + v712 = iconst.i32 0 + v713 = icmp eq v709, v712 + v714 = bint.i32 v713 + brnz v714, ebb83(v711) + jump ebb90(v685, v682) + +ebb86: + v715 = copy.i32 v554 + v716 = copy.i32 v562 + jump ebb88(v682, v554, v573, v683, v550, v684, v685, v553, v715, v562, v716, v557, v558, v559, v560, v561) + +ebb87(v717: i32, v718: i32, v719: i32, v720: i32, v721: i32, v722: i32, v723: i32, v724: i32, v725: i32, v726: i32, v727: i32, v728: i32, v729: i32, v730: i32, v731: i32): + v732 = copy v718 + v733 = iconst.i32 0 + jump ebb88(v717, v732, v719, v720, v721, v722, v723, v724, v725, v733, v726, v727, v728, v729, v730, v731) + +ebb88(v734: i32, v735: i32, v736: i32, v737: i32, v738: i32, v739: i32, v740: i32, v741: i32, v742: i32, v743: i32, v744: i32, v745: i32, v746: i32, v747: i32, v748: i32, v749: i32): + v750 = iadd v734, v735 + v751 = copy v742 + v752 = copy v743 + v753 = copy v744 + jump ebb65(v736, v750, v737, v738, v739, v740, v741, v751, v752, v753, v745, v746, v747, v748, v749) + +ebb89(v754: i32): + v755 = iconst.i32 0 + jump ebb90(v754, v755) + +ebb90(v756: i32, v757: i32): + v758 = iconst.i32 0 + v759 = iconst.i32 1056 + v760 = iadd v756, v759 + v761 = iconst.i32 0x4ffe + v762 = icmp uge v758, v761 + brz v762, ebb91 + trap heap_oob + +ebb91: + v763 = uextend.i64 v758 + v764 = iadd_imm.i64 v0, -8 + v765 = load.i64 v764 + v766 = iadd v765, v763 + store.i32 v760, v766+4 + jump ebb92(v757) + +ebb92(v767: i32): + return v767 +} + +; Same problem from musl.wasm. +function %musl(f64 [%xmm0], i64 vmctx [%rdi]) -> f64 [%xmm0] native { + gv0 = vmctx + heap0 = static gv0, min 0, bound 0x0001_0000_0000, guard 0x8000_0000 + sig0 = (f64 [%xmm0], i32 [%rdi], i64 vmctx [%rsi]) -> f64 [%xmm0] native + fn0 = sig0 u0:517 + +ebb0(v0: f64, v1: i64): + v3 = iconst.i64 0 + v4 = iconst.i32 0 + v131 = iconst.i64 0 + v5 = bitcast.f64 v131 + v6 = iconst.i32 0 + v7 = iconst.i32 0 + v8 = iconst.i32 0 + v132 = uextend.i64 v8 + v133 = iadd_imm v1, 0 + v134 = load.i64 v133 + v9 = iadd v134, v132 + v10 = load.i32 v9+4 + v11 = iconst.i32 16 + v12 = isub v10, v11 + v135 = uextend.i64 v7 + v136 = iadd_imm v1, 0 + v137 = load.i64 v136 + v13 = iadd v137, v135 + store v12, v13+4 + v14 = bitcast.i64 v0 + v15 = iconst.i64 63 + v16 = ushr v14, v15 + v17 = ireduce.i32 v16 + v18 = iconst.i64 32 + v19 = ushr v14, v18 + v20 = ireduce.i32 v19 + v21 = iconst.i32 0x7fff_ffff + v22 = band v20, v21 + v23 = iconst.i32 0x4086_232b + v24 = icmp ult v22, v23 + v25 = bint.i32 v24 + brnz v25, ebb10 + v26 = iconst.i64 0x7fff_ffff_ffff_ffff + v27 = band v14, v26 + v28 = iconst.i64 0x7ff0_0000_0000_0000 + v29 = icmp ule v27, v28 + v30 = bint.i32 v29 + brnz v30, ebb9 + jump ebb2(v12, v0) + +ebb10: + v31 = iconst.i32 0x3fd6_2e43 + v32 = icmp.i32 ult v22, v31 + v33 = bint.i32 v32 + brnz v33, ebb8 + v34 = iconst.i32 0x3ff0_a2b2 + v35 = icmp.i32 uge v22, v34 + v36 = bint.i32 v35 + brnz v36, ebb6 + v37 = iconst.i32 1 + v38 = bxor.i32 v17, v37 + v39 = isub v38, v17 + jump ebb5(v0, v39) + +ebb9: + v138 = iconst.i64 0x4086_2e42_fefa_39ef + v40 = bitcast.f64 v138 + v41 = fcmp ge v40, v0 + v42 = bint.i32 v41 + v139 = fcmp.f64 uno v0, v0 + v140 = fcmp.f64 one v0, v0 + v43 = bor v139, v140 + v44 = bint.i32 v43 + v45 = bor v42, v44 + brnz v45, ebb7 + v141 = iconst.i64 0x7fe0_0000_0000_0000 + v46 = bitcast.f64 v141 + v47 = fmul.f64 v0, v46 + jump ebb2(v12, v47) + +ebb8: + v48 = iconst.i32 0x3e30_0000 + v49 = icmp.i32 ule v22, v48 + v50 = bint.i32 v49 + brnz v50, ebb3 + v51 = iconst.i32 0 + v142 = iconst.i64 0 + v52 = bitcast.f64 v142 + v178 = copy.f64 v0 + jump ebb4(v0, v178, v52, v51) + +ebb7: + v143 = iconst.i64 0xc086_232b_dd7a_bcd2 + v53 = bitcast.f64 v143 + v54 = fcmp.f64 ge v0, v53 + v55 = bint.i32 v54 + v56 = bor v55, v44 + brnz v56, ebb6 + v144 = iconst.i64 0xb6a0_0000_0000_0000 + v57 = bitcast.f64 v144 + v58 = fdiv v57, v0 + v59 = fdemote.f32 v58 + v145 = uextend.i64 v12 + v146 = iadd_imm.i64 v1, 0 + v147 = load.i64 v146 + v60 = iadd v147, v145 + store v59, v60+12 + v148 = iconst.i64 0 + v61 = bitcast.f64 v148 + v149 = iconst.i64 0xc087_4910_d52d_3051 + v62 = bitcast.f64 v149 + v63 = fcmp gt v62, v0 + v64 = bint.i32 v63 + brnz v64, ebb2(v12, v61) + jump ebb6 + +ebb6: + v150 = iconst.i64 0x3ff7_1547_652b_82fe + v66 = bitcast.f64 v150 + v67 = fmul.f64 v0, v66 + v69 = iconst.i32 3 + v70 = ishl.i32 v17, v69 + v71 = iconst.i32 5040 + v72 = iadd v70, v71 + v151 = uextend.i64 v72 + v152 = iadd_imm.i64 v1, 0 + v153 = load.i64 v152 + v73 = iadd v153, v151 + v74 = load.f64 v73 + v75 = fadd v67, v74 + v76 = x86_cvtt2si.i32 v75 + v158 = iconst.i32 0x8000_0000 + v154 = icmp ne v76, v158 + brnz v154, ebb11 + v155 = fcmp uno v75, v75 + brz v155, ebb12 + trap bad_toint + +ebb12: + v159 = iconst.i64 0xc1e0_0000_0020_0000 + v156 = bitcast.f64 v159 + v157 = fcmp ge v156, v75 + brz v157, ebb13 + trap int_ovf + +ebb13: + jump ebb11 + +ebb11: + jump ebb5(v0, v76) + +ebb5(v77: f64, v78: i32): + v79 = fcvt_from_sint.f64 v78 + v160 = iconst.i64 0xbfe6_2e42_fee0_0000 + v80 = bitcast.f64 v160 + v81 = fmul v79, v80 + v82 = fadd v77, v81 + v161 = iconst.i64 0x3dea_39ef_3579_3c76 + v83 = bitcast.f64 v161 + v84 = fmul v79, v83 + v85 = fsub v82, v84 + jump ebb4(v82, v85, v84, v78) + +ebb4(v86: f64, v87: f64, v108: f64, v113: i32): + v88 = fmul v87, v87 + v162 = iconst.i64 0x3e66_3769_72be_a4d0 + v89 = bitcast.f64 v162 + v90 = fmul v88, v89 + v163 = iconst.i64 0xbebb_bd41_c5d2_6bf1 + v91 = bitcast.f64 v163 + v92 = fadd v90, v91 + v93 = fmul v88, v92 + v164 = iconst.i64 0x3f11_566a_af25_de2c + v94 = bitcast.f64 v164 + v95 = fadd v93, v94 + v96 = fmul v88, v95 + v165 = iconst.i64 0xbf66_c16c_16be_bd93 + v97 = bitcast.f64 v165 + v98 = fadd v96, v97 + v99 = fmul v88, v98 + v166 = iconst.i64 0x3fc5_5555_5555_553e + v100 = bitcast.f64 v166 + v101 = fadd v99, v100 + v102 = fmul v88, v101 + v103 = fsub v87, v102 + v104 = fmul v87, v103 + v167 = iconst.i64 0x4000_0000_0000_0000 + v105 = bitcast.f64 v167 + v106 = fsub v105, v103 + v107 = fdiv v104, v106 + v109 = fsub v107, v108 + v110 = fadd v86, v109 + v168 = iconst.i64 0x3ff0_0000_0000_0000 + v111 = bitcast.f64 v168 + v112 = fadd v110, v111 + v169 = iconst.i32 0 + v114 = icmp eq v113, v169 + v115 = bint.i32 v114 + brnz v115, ebb2(v12, v112) + v116 = call fn0(v112, v113, v1) + jump ebb2(v12, v116) + +ebb3: + v170 = iconst.i64 0x7fe0_0000_0000_0000 + v117 = bitcast.f64 v170 + v118 = fadd.f64 v0, v117 + v171 = uextend.i64 v12 + v172 = iadd_imm.i64 v1, 0 + v173 = load.i64 v172 + v119 = iadd v173, v171 + store v118, v119 + v174 = iconst.i64 0x3ff0_0000_0000_0000 + v120 = bitcast.f64 v174 + v121 = fadd.f64 v0, v120 + jump ebb2(v12, v121) + +ebb2(v123: i32, v130: f64): + v122 = iconst.i32 0 + v127 = iconst.i32 16 + v128 = iadd v123, v127 + v175 = uextend.i64 v122 + v176 = iadd_imm.i64 v1, 0 + v177 = load.i64 v176 + v129 = iadd v177, v175 + store v128, v129+4 + jump ebb1(v130) + +ebb1(v2: f64): + return v2 +} diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 3ce58821bc..4819969605 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -7,14 +7,15 @@ use cursor::{Cursor, EncCursor}; use dbg::DisplayList; -use dominator_tree::DominatorTree; +use dominator_tree::{DominatorTree, DominatorTreePreorder}; use flowgraph::ControlFlowGraph; -use ir::{DataFlowGraph, Layout, InstBuilder, ValueDef}; +use ir::{Layout, InstBuilder, ValueDef}; use ir::{Function, Ebb, Inst, Value, ExpandedProgramPoint}; use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; use regalloc::virtregs::VirtRegs; use std::cmp::Ordering; +use std::fmt; use std::iter::Peekable; use std::mem; use isa::{TargetIsa, EncInfo}; @@ -23,10 +24,10 @@ use timing; /// Dominator forest. /// /// This is a utility type used for merging virtual registers, where each virtual register is a -/// list of values ordered according to `DomTree::rpo_cmp`. +/// list of values ordered according to the dominator tree pre-order. /// /// A `DomForest` object is used as a buffer for building virtual registers. It lets you merge two -/// sorted lists of values while checking for interference only whee necessary. +/// sorted lists of values while checking for interference only where necessary. /// /// The idea of a dominator forest was introduced here: /// @@ -37,14 +38,20 @@ use timing; /// The linear stack representation here: /// /// Boissinot, B., Darte, A., & Rastello, F. (2009). Revisiting out-of-SSA translation for -/// correctness, code quality and efficiency. Presented at the Proceedings of the 7th …. +/// correctness, code quality and efficiency. +/// +/// Our version of the linear stack is slightly modified because we have a pre-order of the +/// dominator tree at the EBB granularity, not basic block granularity. struct DomForest { - // The sequence of values that have been merged so far. In RPO order of their defs. + // The sequence of values that have been merged so far. + // In domtree pre-order order of their definitions. values: Vec, // Stack representing the rightmost edge of the dominator forest so far, ending in the last - // element of `values`. At all times, each element in the stack dominates the next one, and all - // elements dominating the end of `values` are on the stack. + // element of `values`. + // + // At all times, the EBB of each element in the stack dominates the EBB of the next one, and + // all elements dominating the end of `values` are on the stack. stack: Vec, } @@ -56,22 +63,29 @@ struct Node { set: u8, /// The program point where `value` is defined. def: ExpandedProgramPoint, + /// EBB containing `def`. + ebb: Ebb, } impl Node { /// Create a node for `value`. - pub fn new(value: Value, set: u8, dfg: &DataFlowGraph) -> Node { + pub fn new(value: Value, set: u8, func: &Function) -> Node { + let def = func.dfg.value_def(value).into(); + let ebb = func.layout.pp_ebb(def); Node { value, set, - def: dfg.value_def(value).into(), + def, + ebb, } } } -/// Push a node to `stack` and update `stack` so it contains all dominator forest ancestors of -/// the pushed value. -/// +impl fmt::Display for Node { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{}@{}:{}", self.value, self.ebb, self.set) + } +} impl DomForest { /// Create a new empty dominator forest. @@ -103,23 +117,51 @@ impl DomForest { /// /// If the pushed node's parent in the dominator forest belongs to a different set, returns /// `Some(parent)`. - fn push_node(&mut self, node: Node, layout: &Layout, domtree: &DominatorTree) -> Option { + fn push_node( + &mut self, + node: Node, + layout: &Layout, + domtree: &DominatorTree, + preorder: &DominatorTreePreorder, + ) -> Option { self.values.push(node.value); // The stack contains the current sequence of dominating defs. Pop elements until we - // find one that dominates `node`. + // find one whose EBB dominates `node.ebb`. while let Some(top) = self.stack.pop() { - if domtree.dominates(top.def, node.def, layout) { + if preorder.dominates(top.ebb, node.ebb) { // This is the right insertion spot for `node`. self.stack.push(top); self.stack.push(node); + + // We know here that `top.ebb` dominates `node.ebb`, and thus `node.def`. This does + // not necessarily mean that `top.def` dominates `node.def`, though. The `top.def` + // program point may be below the last branch in `top.ebb` that dominates + // `node.def`. + debug_assert!(domtree.dominates(top.ebb, node.def, layout)); + + // We do know, though, that if there is a nearest value dominating `node.def`, it + // will be on the stack. We just need to find the last stack entry that actually + // dominates. + // + // TODO: This search could be more efficient if we had access to + // `domtree.last_dominator()`. Each call to `dominates()` here ends up walking up + // the dominator tree starting from `node.ebb`. + let dom = self.stack[0..self.stack.len() - 1].iter().rposition(|n| { + domtree.dominates(n.def, node.def, layout) + }); + // If the parent value comes from a different set, return it for interference // checking. If the sets are equal, assume that interference is already handled. - if top.set != node.set { - return Some(top.value); - } else { - return None; + if let Some(pos) = dom { + let parent = &self.stack[pos]; + if parent.set != node.set { + return Some(parent.value); + } } + + // There was no opposite-set value dominating `node.def`. + return None; } } @@ -143,9 +185,9 @@ impl DomForest { &mut self, va: &[Value], vb: &[Value], - dfg: &DataFlowGraph, - layout: &Layout, + func: &Function, domtree: &DominatorTree, + preorder: &DominatorTreePreorder, liveness: &Liveness, ) -> Result<(), (Value, Value)> { self.clear(); @@ -153,16 +195,16 @@ impl DomForest { // Convert the two value lists into a merged sequence of nodes. let merged = MergedNodes { - a: va.iter().map(|&value| Node::new(value, 0, dfg)).peekable(), - b: vb.iter().map(|&value| Node::new(value, 1, dfg)).peekable(), - layout, - domtree, + a: va.iter().map(|&value| Node::new(value, 0, func)).peekable(), + b: vb.iter().map(|&value| Node::new(value, 1, func)).peekable(), + layout: &func.layout, + preorder, }; - let ctx = liveness.context(layout); + let ctx = liveness.context(&func.layout); for node in merged { - if let Some(parent) = self.push_node(node, layout, domtree) { + if let Some(parent) = self.push_node(node, &func.layout, domtree, preorder) { // Check if `parent` live range contains `node.def`. - if liveness[parent].overlaps_def(node.def, layout.pp_ebb(node.def), ctx) { + if liveness[parent].overlaps_def(node.def, func.layout.pp_ebb(node.def), ctx) { // Interference detected. Get the `(a, b)` order right in the error. return Err(if node.set == 0 { (node.value, parent) @@ -189,7 +231,7 @@ where a: Peekable, b: Peekable, layout: &'a Layout, - domtree: &'a DominatorTree, + preorder: &'a DominatorTreePreorder, } impl<'a, IA, IB> Iterator for MergedNodes<'a, IA, IB> @@ -205,7 +247,7 @@ where // If the two values are defined at the same point, compare value numbers instead // this is going to cause an interference conflict unless its actually the same // value appearing in both streams. - self.domtree.rpo_cmp(a.def, b.def, self.layout).then( + self.preorder.pre_cmp(a.def, b.def, self.layout).then( Ord::cmp( &a.value, &b.value, @@ -231,6 +273,7 @@ where /// Data structures to be used by the coalescing pass. pub struct Coalescing { forest: DomForest, + preorder: DominatorTreePreorder, // Current set of coalesced values. Kept sorted and interference free. values: Vec, @@ -247,6 +290,7 @@ struct Context<'a> { func: &'a mut Function, cfg: &'a ControlFlowGraph, domtree: &'a DominatorTree, + preorder: &'a DominatorTreePreorder, liveness: &'a mut Liveness, virtregs: &'a mut VirtRegs, @@ -260,6 +304,7 @@ impl Coalescing { pub fn new() -> Self { Self { forest: DomForest::new(), + preorder: DominatorTreePreorder::new(), values: Vec::new(), split_values: Vec::new(), } @@ -285,12 +330,14 @@ impl Coalescing { ) { let _tt = timing::ra_cssa(); dbg!("Coalescing for:\n{}", func.display(isa)); + self.preorder.compute(domtree, &func.layout); let mut context = Context { isa, encinfo: isa.encoding_info(), func, cfg, domtree, + preorder: &self.preorder, liveness, virtregs, forest: &mut self.forest, @@ -486,9 +533,9 @@ impl<'a> Context<'a> { self.forest.try_merge( self.values, self.virtregs.congruence_class(&value), - &self.func.dfg, - &self.func.layout, + self.func, self.domtree, + self.preorder, self.liveness, )?; self.forest.swap(&mut self.values); From febe8e0e517f91ff831c01bc77f25d3f5fe49b9c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 14 Dec 2017 10:57:10 -0600 Subject: [PATCH 1450/3084] Allow spilling of EBB arguments. When the spiller needs to make a register available for a conditional branch instruction, it can be necessary to spill some of the EBB arguments on the branch instruction. This is ok because EBB argument values belong to the same virtual register as the corresponding EBB parameter and we spill the whole virtreg to the same slot. Also make sure free_regs() can handle values that are killed by the current instruction *and* spilled. --- cranelift/filetests/regalloc/spill.cton | 24 ++++++++++++++++++++++++ lib/cretonne/src/regalloc/spilling.rs | 17 ++++++++++++++--- 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index 8f00c93184..adafc89b43 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -145,6 +145,30 @@ ebb1(v10: i32, v11: i32, v12: i32, v13: i32, v14: i32, v15: i32, v16: i32, v17: return v33 } +; Spilling an EBB argument to make room for a branch operand. +function %brargs(i32) -> i32 { +ebb0(v1: i32): + ; check: $v1 = spill + v2 = iconst.i32 1 + brnz v1, ebb1(v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2) + return v1 + +ebb1(v10: i32, v11: i32, v12: i32, v13: i32, v14: i32, v15: i32, v16: i32, v17: i32, v18: i32, v19: i32, v20: i32, v21: i32): + v22 = iadd v10, v11 + v23 = iadd v22, v12 + v24 = iadd v23, v13 + v25 = iadd v24, v14 + v26 = iadd v25, v15 + v27 = iadd v26, v16 + v28 = iadd v27, v17 + v29 = iadd v28, v18 + v30 = iadd v29, v19 + v31 = iadd v30, v20 + v32 = iadd v31, v21 + v33 = iadd v32, v1 + return v33 +} + ; In straight-line code, the first value defined is spilled. ; That is in order: ; 1. The argument v1. diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 6eef88141d..4c5b8b9ec7 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -155,8 +155,10 @@ impl<'a> Context<'a> { fn free_regs(&mut self, kills: &[LiveValue]) { for lv in kills { if let Affinity::Reg(rci) = lv.affinity { - let rc = self.reginfo.rc(rci); - self.pressure.free(rc); + if !self.spills.contains(&lv.value) { + let rc = self.reginfo.rc(rci); + self.pressure.free(rc); + } } } } @@ -402,8 +404,17 @@ impl<'a> Context<'a> { dbg!("Copy of {} reg causes spill", rc); // Spill a live register that is *not* used by the current instruction. // Spilling a use wouldn't help. + // + // Do allow spilling of EBB arguments on branches. This is safe since we spill + // the whole virtual register which includes the matching EBB parameter value + // at the branch destination. It is also necessary since there can be + // arbitrarily many EBB arguments. match { - let args = self.cur.func.dfg.inst_args(inst); + let args = if self.cur.func.dfg[inst].opcode().is_branch() { + self.cur.func.dfg.inst_fixed_args(inst) + } else { + self.cur.func.dfg.inst_args(inst) + }; self.spill_candidate( mask, tracker.live().iter().filter(|lv| !args.contains(&lv.value)), From 66073eb26cb4db538e84805c55c0ee97c4326a02 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 14 Dec 2017 17:04:16 -0600 Subject: [PATCH 1451/3084] Better verifier error for coinciding defs. If a virtual register contains values that a defined at the same program point, say so. Don't cryptically claim that one dominates the other. --- lib/cretonne/src/verifier/cssa.rs | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/verifier/cssa.rs b/lib/cretonne/src/verifier/cssa.rs index 5c35f8811e..d29b0d72b5 100644 --- a/lib/cretonne/src/verifier/cssa.rs +++ b/lib/cretonne/src/verifier/cssa.rs @@ -3,7 +3,7 @@ use dbg::DisplayList; use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; -use ir::Function; +use ir::{Function, ExpandedProgramPoint}; use regalloc::liveness::Liveness; use regalloc::virtregs::VirtRegs; use timing; @@ -68,10 +68,21 @@ impl<'a> CssaVerifier<'a> { }; // Check topological ordering with the previous values in the virtual register. - let def = self.func.dfg.value_def(val).into(); + let def: ExpandedProgramPoint = self.func.dfg.value_def(val).into(); let def_ebb = self.func.layout.pp_ebb(def); for &prev_val in &values[0..idx] { - let prev_def = self.func.dfg.value_def(prev_val); + let prev_def: ExpandedProgramPoint = self.func.dfg.value_def(prev_val).into(); + + if prev_def == def { + return err!( + val, + "Values {} and {} in {} = {} defined at the same program point", + prev_val, + val, + vreg, + DisplayList(values) + ); + } // Enforce topological ordering of defs in the virtual register. if self.domtree.dominates(def, prev_def, &self.func.layout) { From fc857247e467ec8a66cdac7ef87a1688393588fb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 14 Dec 2017 17:16:19 -0600 Subject: [PATCH 1452/3084] Fix overlaps_def for dead live ranges. A dead live range ends at the same point it is defined, but it is still considered to overlap a def at the same program point. --- lib/cretonne/src/regalloc/liverange.rs | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 60079b3705..7d7d6a8775 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -407,6 +407,11 @@ impl GenLiveRange { ebb: Ebb, ctx: LiveRangeContext, ) -> bool { + // Two defs at the same program point always overlap, even if one is dead. + if def == self.def_begin.into() { + return true; + } + // Check for an overlap with the local range. if ctx.order.cmp(def, self.def_begin) != Ordering::Less && ctx.order.cmp(def, self.def_end) == Ordering::Less @@ -549,7 +554,9 @@ mod tests { #[test] fn dead_def_range() { let v0 = Value::new(0); + let e0 = Ebb::new(0); let i1 = Inst::new(1); + let i2 = Inst::new(2); let e2 = Ebb::new(2); let lr = GenLiveRange::new(v0, i1.into(), Default::default()); let forest = &bforest::MapForest::new(); @@ -560,6 +567,11 @@ mod tests { assert_eq!(lr.def_local_end(), i1.into()); assert_eq!(lr.livein_local_end(e2, ctx), None); PO.validate(&lr, ctx.forest); + + // A dead live range overlaps its own def program point. + assert!(lr.overlaps_def(i1.into(), e0, ctx)); + assert!(!lr.overlaps_def(i2.into(), e0, ctx)); + assert!(!lr.overlaps_def(e0.into(), e0, ctx)); } #[test] From 76e31cc1ad94a406f946c682fb442bb982fd3fd5 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 14 Dec 2017 06:43:55 -0600 Subject: [PATCH 1453/3084] Rename GotPCRel4 to GOTPCRel4. This emphasizes that GOT is being used as an abbreviation rather than the word "got". --- cranelift/filetests/isa/intel/binary64-pic.cton | 12 ++++++------ lib/cretonne/meta/isa/intel/recipes.py | 4 ++-- lib/cretonne/src/binemit/mod.rs | 4 ++-- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary64-pic.cton b/cranelift/filetests/isa/intel/binary64-pic.cton index fa422741e4..50a0361ebf 100644 --- a/cranelift/filetests/isa/intel/binary64-pic.cton +++ b/cranelift/filetests/isa/intel/binary64-pic.cton @@ -30,11 +30,11 @@ ebb0: call fn0() ; bin: e8 PLTRel4(%foo) 00000000 ; asm: mov 0x0(%rip), %rax - [-,%rax] v0 = func_addr.i64 fn0 ; bin: 48 8b 05 GotPCRel4(%foo) 00000000 + [-,%rax] v0 = func_addr.i64 fn0 ; bin: 48 8b 05 GOTPCRel4(%foo) 00000000 ; asm: mov 0x0(%rip), %rsi - [-,%rsi] v1 = func_addr.i64 fn0 ; bin: 48 8b 35 GotPCRel4(%foo) 00000000 + [-,%rsi] v1 = func_addr.i64 fn0 ; bin: 48 8b 35 GOTPCRel4(%foo) 00000000 ; asm: mov 0x0(%rip), %r10 - [-,%r10] v2 = func_addr.i64 fn0 ; bin: 4c 8b 15 GotPCRel4(%foo) 00000000 + [-,%r10] v2 = func_addr.i64 fn0 ; bin: 4c 8b 15 GOTPCRel4(%foo) 00000000 ; asm: call *%rax call_indirect sig0, v0() ; bin: ff d0 @@ -44,11 +44,11 @@ ebb0: call_indirect sig0, v2() ; bin: 41 ff d2 ; asm: mov 0x0(%rip), %rcx - [-,%rcx] v3 = globalsym_addr.i64 gv0 ; bin: 48 8b 0d GotPCRel4(%some_gv) 00000000 + [-,%rcx] v3 = globalsym_addr.i64 gv0 ; bin: 48 8b 0d GOTPCRel4(%some_gv) 00000000 ; asm: mov 0x0(%rip), %rsi - [-,%rsi] v4 = globalsym_addr.i64 gv0 ; bin: 48 8b 35 GotPCRel4(%some_gv) 00000000 + [-,%rsi] v4 = globalsym_addr.i64 gv0 ; bin: 48 8b 35 GOTPCRel4(%some_gv) 00000000 ; asm: mov 0x0(%rip), %r10 - [-,%r10] v5 = globalsym_addr.i64 gv0 ; bin: 4c 8b 15 GotPCRel4(%some_gv) 00000000 + [-,%r10] v5 = globalsym_addr.i64 gv0 ; bin: 4c 8b 15 GOTPCRel4(%some_gv) 00000000 return } diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 947d5839e7..0ed5431e8d 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -565,7 +565,7 @@ got_fnaddr8 = TailRecipe( emit=''' PUT_OP(bits, rex2(0, out_reg0), sink); modrm_riprel(out_reg0, sink); - sink.reloc_external(Reloc::IntelGotPCRel4, + sink.reloc_external(Reloc::IntelGOTPCRel4, &func.dfg.ext_funcs[func_ref].name); sink.put4(0); ''') @@ -597,7 +597,7 @@ got_gvaddr8 = TailRecipe( emit=''' PUT_OP(bits, rex2(0, out_reg0), sink); modrm_rm(5, out_reg0, sink); - sink.reloc_external(Reloc::IntelGotPCRel4, + sink.reloc_external(Reloc::IntelGOTPCRel4, &func.global_vars[global_var].symbol_name()); sink.put4(0); ''') diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 43665d945b..9814b2727f 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -29,7 +29,7 @@ pub enum Reloc { /// Intel absolute 8-byte IntelAbs8, /// Intel GOT PC-relative 4-byte - IntelGotPCRel4, + IntelGOTPCRel4, /// Intel PLT-relative 4-byte IntelPLTRel4, /// Arm32 call target @@ -48,7 +48,7 @@ impl fmt::Display for Reloc { Reloc::IntelPCRel4 => write!(f, "{}", "PCRel4"), Reloc::IntelAbs4 => write!(f, "{}", "Abs4"), Reloc::IntelAbs8 => write!(f, "{}", "Abs8"), - Reloc::IntelGotPCRel4 => write!(f, "{}", "GotPCRel4"), + Reloc::IntelGOTPCRel4 => write!(f, "{}", "GOTPCRel4"), Reloc::IntelPLTRel4 => write!(f, "{}", "PLTRel4"), Reloc::Arm32Call | Reloc::Arm64Call | Reloc::RiscvCall => write!(f, "{}", "Call"), } From 4f53cc1dad32e672d8a2fabf312b5c5c2bca2244 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 14 Dec 2017 07:13:17 -0600 Subject: [PATCH 1454/3084] Align IntelGOTPCRel4 with R_X86_64_GOTPCREL. Add an addend field to reloc_external, and use it to move the responsibility for accounting for the difference between the end of an instruction (where the PC is considered to be in PC-relative on intel) and the beginning of the immediate field into the encoding code. Specifically, this makes IntelGOTPCRel4 directly correspond to R_X86_64_GOTPCREL, instead of also carrying an implicit `- 4`. --- .../filetests/isa/intel/binary64-pic.cton | 12 +++---- cranelift/src/filetest/binemit.rs | 20 +++++++++-- cranelift/src/filetest/compile.rs | 8 ++++- lib/cretonne/meta/isa/intel/recipes.py | 34 +++++++++++++------ lib/cretonne/meta/isa/riscv/recipes.py | 3 +- lib/cretonne/src/binemit/memorysink.rs | 8 ++--- lib/cretonne/src/binemit/mod.rs | 7 ++-- 7 files changed, 66 insertions(+), 26 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary64-pic.cton b/cranelift/filetests/isa/intel/binary64-pic.cton index 50a0361ebf..70dc7bed42 100644 --- a/cranelift/filetests/isa/intel/binary64-pic.cton +++ b/cranelift/filetests/isa/intel/binary64-pic.cton @@ -30,11 +30,11 @@ ebb0: call fn0() ; bin: e8 PLTRel4(%foo) 00000000 ; asm: mov 0x0(%rip), %rax - [-,%rax] v0 = func_addr.i64 fn0 ; bin: 48 8b 05 GOTPCRel4(%foo) 00000000 + [-,%rax] v0 = func_addr.i64 fn0 ; bin: 48 8b 05 GOTPCRel4(%foo-4) 00000000 ; asm: mov 0x0(%rip), %rsi - [-,%rsi] v1 = func_addr.i64 fn0 ; bin: 48 8b 35 GOTPCRel4(%foo) 00000000 + [-,%rsi] v1 = func_addr.i64 fn0 ; bin: 48 8b 35 GOTPCRel4(%foo-4) 00000000 ; asm: mov 0x0(%rip), %r10 - [-,%r10] v2 = func_addr.i64 fn0 ; bin: 4c 8b 15 GOTPCRel4(%foo) 00000000 + [-,%r10] v2 = func_addr.i64 fn0 ; bin: 4c 8b 15 GOTPCRel4(%foo-4) 00000000 ; asm: call *%rax call_indirect sig0, v0() ; bin: ff d0 @@ -44,11 +44,11 @@ ebb0: call_indirect sig0, v2() ; bin: 41 ff d2 ; asm: mov 0x0(%rip), %rcx - [-,%rcx] v3 = globalsym_addr.i64 gv0 ; bin: 48 8b 0d GOTPCRel4(%some_gv) 00000000 + [-,%rcx] v3 = globalsym_addr.i64 gv0 ; bin: 48 8b 0d GOTPCRel4(%some_gv-4) 00000000 ; asm: mov 0x0(%rip), %rsi - [-,%rsi] v4 = globalsym_addr.i64 gv0 ; bin: 48 8b 35 GOTPCRel4(%some_gv) 00000000 + [-,%rsi] v4 = globalsym_addr.i64 gv0 ; bin: 48 8b 35 GOTPCRel4(%some_gv-4) 00000000 ; asm: mov 0x0(%rip), %r10 - [-,%r10] v5 = globalsym_addr.i64 gv0 ; bin: 4c 8b 15 GOTPCRel4(%some_gv) 00000000 + [-,%r10] v5 = globalsym_addr.i64 gv0 ; bin: 4c 8b 15 GOTPCRel4(%some_gv-4) 00000000 return } diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 24de3d7e69..c51f4b613d 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -72,13 +72,29 @@ impl binemit::CodeSink for TextSink { write!(self.text, "{}({}) ", reloc, ebb_offset).unwrap(); } - fn reloc_external(&mut self, reloc: binemit::Reloc, name: &ir::ExternalName) { + fn reloc_external( + &mut self, + reloc: binemit::Reloc, + name: &ir::ExternalName, + addend: binemit::Addend, + ) { write!( self.text, - "{}({}) ", + "{}({}", reloc, name, ).unwrap(); + if addend != 0 { + write!( + self.text, + "{:+}", + addend, + ).unwrap(); + } + write!( + self.text, + ") ", + ).unwrap(); } fn reloc_jt(&mut self, reloc: binemit::Reloc, jt: ir::JumpTable) { diff --git a/cranelift/src/filetest/compile.rs b/cranelift/src/filetest/compile.rs index 0e271b7db4..8c8db46591 100644 --- a/cranelift/src/filetest/compile.rs +++ b/cranelift/src/filetest/compile.rs @@ -103,6 +103,12 @@ impl binemit::CodeSink for SizeSink { } fn reloc_ebb(&mut self, _reloc: binemit::Reloc, _ebb_offset: binemit::CodeOffset) {} - fn reloc_external(&mut self, _reloc: binemit::Reloc, _name: &ir::ExternalName) {} + fn reloc_external( + &mut self, + _reloc: binemit::Reloc, + _name: &ir::ExternalName, + _addend: binemit::Addend, + ) { + } fn reloc_jt(&mut self, _reloc: binemit::Reloc, _jt: ir::JumpTable) {} } diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 0ed5431e8d..fb7b993c80 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -522,7 +522,8 @@ fnaddr4 = TailRecipe( emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); sink.reloc_external(Reloc::IntelAbs4, - &func.dfg.ext_funcs[func_ref].name); + &func.dfg.ext_funcs[func_ref].name, + 0); sink.put4(0); ''') @@ -532,7 +533,8 @@ fnaddr8 = TailRecipe( emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); sink.reloc_external(Reloc::IntelAbs8, - &func.dfg.ext_funcs[func_ref].name); + &func.dfg.ext_funcs[func_ref].name, + 0); sink.put8(0); ''') @@ -542,7 +544,8 @@ allones_fnaddr4 = TailRecipe( emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); sink.reloc_external(Reloc::IntelAbs4, - &func.dfg.ext_funcs[func_ref].name); + &func.dfg.ext_funcs[func_ref].name, + 0); // Write the immediate as `!0` for the benefit of BaldrMonkey. sink.put4(!0); ''') @@ -553,7 +556,8 @@ allones_fnaddr8 = TailRecipe( emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); sink.reloc_external(Reloc::IntelAbs8, - &func.dfg.ext_funcs[func_ref].name); + &func.dfg.ext_funcs[func_ref].name, + 0); // Write the immediate as `!0` for the benefit of BaldrMonkey. sink.put8(!0); ''') @@ -565,8 +569,11 @@ got_fnaddr8 = TailRecipe( emit=''' PUT_OP(bits, rex2(0, out_reg0), sink); modrm_riprel(out_reg0, sink); + // The addend adjusts for the difference between the end of the + // instruction and the beginning of the immediate field. sink.reloc_external(Reloc::IntelGOTPCRel4, - &func.dfg.ext_funcs[func_ref].name); + &func.dfg.ext_funcs[func_ref].name, + -4); sink.put4(0); ''') @@ -577,7 +584,8 @@ gvaddr4 = TailRecipe( emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); sink.reloc_external(Reloc::IntelAbs4, - &func.global_vars[global_var].symbol_name()); + &func.global_vars[global_var].symbol_name(), + 0); sink.put4(0); ''') @@ -587,7 +595,8 @@ gvaddr8 = TailRecipe( emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); sink.reloc_external(Reloc::IntelAbs8, - &func.global_vars[global_var].symbol_name()); + &func.global_vars[global_var].symbol_name(), + 0); sink.put8(0); ''') @@ -597,8 +606,11 @@ got_gvaddr8 = TailRecipe( emit=''' PUT_OP(bits, rex2(0, out_reg0), sink); modrm_rm(5, out_reg0, sink); + // The addend adjusts for the difference between the end of the + // instruction and the beginning of the immediate field. sink.reloc_external(Reloc::IntelGOTPCRel4, - &func.global_vars[global_var].symbol_name()); + &func.global_vars[global_var].symbol_name(), + -4); sink.put4(0); ''') @@ -874,7 +886,8 @@ call_id = TailRecipe( emit=''' PUT_OP(bits, BASE_REX, sink); sink.reloc_external(Reloc::IntelPCRel4, - &func.dfg.ext_funcs[func_ref].name); + &func.dfg.ext_funcs[func_ref].name, + 0); sink.put4(0); ''') @@ -883,7 +896,8 @@ call_plt_id = TailRecipe( emit=''' PUT_OP(bits, BASE_REX, sink); sink.reloc_external(Reloc::IntelPLTRel4, - &func.dfg.ext_funcs[func_ref].name); + &func.dfg.ext_funcs[func_ref].name, + 0); sink.put4(0); ''') diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/cretonne/meta/isa/riscv/recipes.py index 1fbba83036..c657c7de47 100644 --- a/lib/cretonne/meta/isa/riscv/recipes.py +++ b/lib/cretonne/meta/isa/riscv/recipes.py @@ -184,7 +184,8 @@ UJcall = EncRecipe( 'UJcall', Call, size=4, ins=(), outs=(), emit=''' sink.reloc_external(Reloc::RiscvCall, - &func.dfg.ext_funcs[func_ref].name); + &func.dfg.ext_funcs[func_ref].name, + 0); // rd=%x1 is the standard link register. put_uj(bits, 0, 1, sink); ''') diff --git a/lib/cretonne/src/binemit/memorysink.rs b/lib/cretonne/src/binemit/memorysink.rs index 92766b2d56..1d6758afe6 100644 --- a/lib/cretonne/src/binemit/memorysink.rs +++ b/lib/cretonne/src/binemit/memorysink.rs @@ -15,7 +15,7 @@ //! `CodeSink::put*` methods, so the performance impact of the virtual callbacks is less severe. use ir::{ExternalName, JumpTable}; -use super::{CodeSink, CodeOffset, Reloc}; +use super::{CodeSink, CodeOffset, Reloc, Addend}; use std::ptr::write_unaligned; /// A `CodeSink` that writes binary machine code directly into memory. @@ -52,7 +52,7 @@ pub trait RelocSink { fn reloc_ebb(&mut self, CodeOffset, Reloc, CodeOffset); /// Add a relocation referencing an external symbol at the current offset. - fn reloc_external(&mut self, CodeOffset, Reloc, &ExternalName); + fn reloc_external(&mut self, CodeOffset, Reloc, &ExternalName, Addend); /// Add a relocation referencing a jump table. fn reloc_jt(&mut self, CodeOffset, Reloc, JumpTable); @@ -96,9 +96,9 @@ impl<'a> CodeSink for MemoryCodeSink<'a> { self.relocs.reloc_ebb(ofs, rel, ebb_offset); } - fn reloc_external(&mut self, rel: Reloc, name: &ExternalName) { + fn reloc_external(&mut self, rel: Reloc, name: &ExternalName, addend: Addend) { let ofs = self.offset(); - self.relocs.reloc_external(ofs, rel, name); + self.relocs.reloc_external(ofs, rel, name, addend); } fn reloc_jt(&mut self, rel: Reloc, jt: JumpTable) { diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 9814b2727f..fe6262cb70 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -19,6 +19,9 @@ use std::fmt; /// depends on the *host* platform, not the *target* platform. pub type CodeOffset = u32; +/// Addend to add to the symbol value. +pub type Addend = i64; + /// Relocation kinds for every ISA #[derive(Debug)] pub enum Reloc { @@ -78,8 +81,8 @@ pub trait CodeSink { /// Add a relocation referencing an EBB at the current offset. fn reloc_ebb(&mut self, Reloc, CodeOffset); - /// Add a relocation referencing an external symbol at the current offset. - fn reloc_external(&mut self, Reloc, &ExternalName); + /// Add a relocation referencing an external symbol plus the addend at the current offset. + fn reloc_external(&mut self, Reloc, &ExternalName, Addend); /// Add a relocation referencing a jump table. fn reloc_jt(&mut self, Reloc, JumpTable); From 4afa19ddff99da15b973d9025d9686fdff4f5b42 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 3 Jan 2018 12:13:13 -0800 Subject: [PATCH 1455/3084] Fix some mypy errors. It looks like mypy 0.560 doesn't like when a local variable changes its type inside a function. Fixes introduce a new variable instead of reusing an existing one. --- lib/cretonne/meta/cdsl/instructions.py | 16 ++++++++-------- lib/cretonne/meta/unique_table.py | 17 ++++++++++------- 2 files changed, 18 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/cretonne/meta/cdsl/instructions.py index 2ca4951c58..3ef95c9cf9 100644 --- a/lib/cretonne/meta/cdsl/instructions.py +++ b/lib/cretonne/meta/cdsl/instructions.py @@ -290,12 +290,12 @@ class Instruction(object): # Allow a single Operand instance instead of the awkward singleton # tuple syntax. if isinstance(x, Operand): - x = (x,) + y = (x,) # type: Tuple[Operand, ...] else: - x = tuple(x) - for op in x: + y = tuple(x) + for op in y: assert isinstance(op, Operand) - return x + return y @staticmethod def _to_constraint_tuple(x): @@ -307,12 +307,12 @@ class Instruction(object): # import placed here to avoid circular dependency from .ti import TypeConstraint # noqa if isinstance(x, TypeConstraint): - x = (x,) + y = (x,) # type: Tuple[TypeConstraint, ...] else: - x = tuple(x) - for op in x: + y = tuple(x) + for op in y: assert isinstance(op, TypeConstraint) - return x + return y def bind(self, *args): # type: (*ValueType) -> BoundInstruction diff --git a/lib/cretonne/meta/unique_table.py b/lib/cretonne/meta/unique_table.py index 0e18ae3afd..d8482b3c7d 100644 --- a/lib/cretonne/meta/unique_table.py +++ b/lib/cretonne/meta/unique_table.py @@ -62,16 +62,19 @@ class UniqueSeqTable: """ if len(seq) == 0: return 0 - seq = tuple(seq) - if seq in self.index: - return self.index[seq] + tseq = tuple(seq) + if tseq in self.index: + return self.index[tseq] idx = len(self.table) - self.table.extend(seq) + self.table.extend(tseq) # Add seq and all sub-sequences to `index`. - for length in range(1, len(seq) + 1): - for offset in range(len(seq) - length + 1): - self.index[seq[offset:offset+length]] = idx + offset + index = self.index # type: Dict[Tuple[Any, ...], int] + assert index is not None + for length in range(1, len(tseq) + 1): + for offset in range(len(tseq) - length + 1): + key = tseq[offset:offset+length] + index[key] = idx + offset return idx From af89006b09e91517e98947827adf37e7dba430eb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 8 Jan 2018 16:18:10 -0800 Subject: [PATCH 1456/3084] Fix some markdown issues. Work around some cases where the old markdown parser differs from the new Pulldown parser for the documentation. --- lib/cretonne/src/ir/dfg.rs | 2 +- lib/cretonne/src/ir/immediates.rs | 4 +- lib/cretonne/src/ir/types.rs | 5 +- lib/cretonne/src/result.rs | 6 +-- lib/cretonne/src/verifier/mod.rs | 78 +++++++++++++++---------------- lib/filecheck/src/lib.rs | 2 + lib/wasm/src/func_translator.rs | 5 +- 7 files changed, 53 insertions(+), 49 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index e2328e46dc..50842b3fa5 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -705,7 +705,7 @@ impl DataFlowGraph { /// Returns the position of `val` before removal. /// /// *Important*: to ensure O(1) deletion, this method swaps the removed parameter with the - /// last `ebb`` parameter. This can disrupt all the branch instructions jumping to this + /// last `ebb` parameter. This can disrupt all the branch instructions jumping to this /// `ebb` for which you have to change the branch argument order if necessary. /// /// Panics if `val` is not an EBB parameter. diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index fb55c97006..69607754a2 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -526,7 +526,7 @@ impl Ieee32 { Ieee32(x) } - /// Create an `Ieee32` number representing 2.0^n. + /// Create an `Ieee32` number representing `2.0^n`. pub fn pow2>(n: I) -> Ieee32 { let n = n.into(); let w = 8; @@ -578,7 +578,7 @@ impl Ieee64 { Ieee64(x) } - /// Create an `Ieee64` number representing 2.0^n. + /// Create an `Ieee64` number representing `2.0^n`. pub fn pow2>(n: I) -> Ieee64 { let n = n.into(); let w = 11; diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 8b93e82267..f8774250ec 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -265,8 +265,9 @@ impl Type { } /// True iff: - /// 1) self.lane_count() == other.lane_count() and - /// 2) self.lane_bits() >= other.lane_bits() + /// + /// 1. `self.lane_count() == other.lane_count()` and + /// 2. `self.lane_bits() >= other.lane_bits()` pub fn wider_or_equal(self, other: Type) -> bool { self.lane_count() == other.lane_count() && self.lane_bits() >= other.lane_bits() } diff --git a/lib/cretonne/src/result.rs b/lib/cretonne/src/result.rs index b5e9861f95..387eaa9a2a 100644 --- a/lib/cretonne/src/result.rs +++ b/lib/cretonne/src/result.rs @@ -23,10 +23,10 @@ pub enum CtonError { /// An implementation limit was exceeded. /// - /// Cretonne can compile very large and complicated functions, but the implementation has - /// limits that cause compilation to fail when they are exceeded. + /// Cretonne can compile very large and complicated functions, but the [implementation has + /// limits][limits] that cause compilation to fail when they are exceeded. /// - /// See http://cretonne.readthedocs.io/en/latest/langref.html#implementation-limits + /// [limits]: http://cretonne.readthedocs.io/en/latest/langref.html#implementation-limits ImplLimitExceeded, /// The code size for the function is too large. diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index ef534bcd87..affd8a3ec6 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -1,59 +1,59 @@ //! A verifier for ensuring that functions are well formed. //! It verifies: //! -//! EBB integrity +//! EBB integrity //! -//! - All instructions reached from the `ebb_insts` iterator must belong to -//! the EBB as reported by `inst_ebb()`. -//! - Every EBB must end in a terminator instruction, and no other instruction -//! can be a terminator. -//! - Every value in the `ebb_params` iterator belongs to the EBB as reported by `value_ebb`. +//! - All instructions reached from the `ebb_insts` iterator must belong to +//! the EBB as reported by `inst_ebb()`. +//! - Every EBB must end in a terminator instruction, and no other instruction +//! can be a terminator. +//! - Every value in the `ebb_params` iterator belongs to the EBB as reported by `value_ebb`. //! -//! Instruction integrity +//! Instruction integrity //! -//! - The instruction format must match the opcode. -//! - All result values must be created for multi-valued instructions. -//! - All referenced entities must exist. (Values, EBBs, stack slots, ...) -//! - Instructions must not reference (eg. branch to) the entry block. +//! - The instruction format must match the opcode. +//! - All result values must be created for multi-valued instructions. +//! - All referenced entities must exist. (Values, EBBs, stack slots, ...) +//! - Instructions must not reference (eg. branch to) the entry block. //! -//! SSA form +//! SSA form //! -//! - Values must be defined by an instruction that exists and that is inserted in -//! an EBB, or be an argument of an existing EBB. -//! - Values used by an instruction must dominate the instruction. +//! - Values must be defined by an instruction that exists and that is inserted in +//! an EBB, or be an argument of an existing EBB. +//! - Values used by an instruction must dominate the instruction. //! -//! Control flow graph and dominator tree integrity: +//! Control flow graph and dominator tree integrity: //! -//! - All predecessors in the CFG must be branches to the EBB. -//! - All branches to an EBB must be present in the CFG. -//! - A recomputed dominator tree is identical to the existing one. +//! - All predecessors in the CFG must be branches to the EBB. +//! - All branches to an EBB must be present in the CFG. +//! - A recomputed dominator tree is identical to the existing one. //! -//! Type checking +//! Type checking //! -//! - Compare input and output values against the opcode's type constraints. -//! For polymorphic opcodes, determine the controlling type variable first. -//! - Branches and jumps must pass arguments to destination EBBs that match the -//! expected types exactly. The number of arguments must match. -//! - All EBBs in a jump_table must take no arguments. -//! - Function calls are type checked against their signature. -//! - The entry block must take arguments that match the signature of the current -//! function. -//! - All return instructions must have return value operands matching the current -//! function signature. +//! - Compare input and output values against the opcode's type constraints. +//! For polymorphic opcodes, determine the controlling type variable first. +//! - Branches and jumps must pass arguments to destination EBBs that match the +//! expected types exactly. The number of arguments must match. +//! - All EBBs in a jump_table must take no arguments. +//! - Function calls are type checked against their signature. +//! - The entry block must take arguments that match the signature of the current +//! function. +//! - All return instructions must have return value operands matching the current +//! function signature. //! -//! Global variables +//! Global variables //! -//! - Detect cycles in deref(base) declarations. +//! - Detect cycles in deref(base) declarations. //! //! TODO: -//! Ad hoc checking +//! Ad hoc checking //! -//! - Stack slot loads and stores must be in-bounds. -//! - Immediate constraints for certain opcodes, like `udiv_imm v3, 0`. -//! - `Insertlane` and `extractlane` instructions have immediate lane numbers that must be in -//! range for their polymorphic type. -//! - Swizzle and shuffle instructions take a variable number of lane arguments. The number -//! of arguments must match the destination type, and the lane indexes must be in range. +//! - Stack slot loads and stores must be in-bounds. +//! - Immediate constraints for certain opcodes, like `udiv_imm v3, 0`. +//! - `Insertlane` and `extractlane` instructions have immediate lane numbers that must be in +//! range for their polymorphic type. +//! - Swizzle and shuffle instructions take a variable number of lane arguments. The number +//! of arguments must match the destination type, and the lane indexes must be in range. use dbg::DisplayList; use dominator_tree::DominatorTree; diff --git a/lib/filecheck/src/lib.rs b/lib/filecheck/src/lib.rs index 8cca73736b..a189acebdb 100644 --- a/lib/filecheck/src/lib.rs +++ b/lib/filecheck/src/lib.rs @@ -10,6 +10,7 @@ //! # Directives //! //! These are the directives recognized by *filecheck*: +//! //! 
 //! check: <pattern>
 //! sameln: <pattern>
@@ -18,6 +19,7 @@
 //! not: <pattern>
 //! regex: <variable>=<regex>
 //!
+//! //! Each directive is described in more detail below. //! //! ## Example diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 36c1b58a17..5e8555c194 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -43,8 +43,9 @@ impl FuncTranslator { /// - The declaration of *locals*, and /// - The function *body* as an expression. /// - /// See [the WebAssembly specification] - /// (http://webassembly.github.io/spec/binary/modules.html#code-section). + /// See [the WebAssembly specification][wasm]. + /// + /// [wasm]: http://webassembly.github.io/spec/binary/modules.html#code-section /// /// The Cretonne IR function `func` should be completely empty except for the `func.signature` /// and `func.name` fields. The signature may contain special-purpose arguments which are not From 5e094034d45b341bc426f97f4fd321134085cc00 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 9 Jan 2018 10:45:47 -0800 Subject: [PATCH 1457/3084] Fix verifier bug in unreachable code. We want to disable dominance checks in unreachable code. The is_reachable() check for EBB parameter values was checking if the defining EBB was reachable, not the EBB using the value. This bug showed up in fuzzing and in #213. --- .../filetests/verifier/unreachable_code.cton | 22 +++++++++++++++++++ lib/cretonne/src/verifier/mod.rs | 8 +++---- 2 files changed, 26 insertions(+), 4 deletions(-) diff --git a/cranelift/filetests/verifier/unreachable_code.cton b/cranelift/filetests/verifier/unreachable_code.cton index 474eaec6b4..7ea7dd49b0 100644 --- a/cranelift/filetests/verifier/unreachable_code.cton +++ b/cranelift/filetests/verifier/unreachable_code.cton @@ -21,3 +21,25 @@ ebb9(v7: i32): return v9 } + +; Using a function argument in an unreachable block is ok. +function %arg(i32) -> i32 { +ebb0(v0: i32): + v1 = iadd_imm v0, 1 + return v1 + +ebb1: + v10 = iadd_imm v0, 10 + return v10 +} + +; Using an EBB argument from an unreachable block is not ok. +function %arg2(i32) -> i32 { +ebb0(v0: i32): + v1 = iadd v0, v10 ; error: uses value arg from non-dominating + return v1 + +ebb1(v10: i32): + v11 = iadd v0, v10 + return v11 +} diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index affd8a3ec6..1860fecbbf 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -443,6 +443,8 @@ impl<'a> Verifier<'a> { if !dfg.value_is_valid(v) { return err!(loc_inst, "invalid value reference {}", v); } + let loc_ebb = self.func.layout.pp_ebb(loc_inst); + let is_reachable = self.expected_domtree.is_reachable(loc_ebb); // SSA form match dfg.value_def(v) { @@ -466,9 +468,7 @@ impl<'a> Verifier<'a> { ); } // Defining instruction dominates the instruction that uses the value. - if self.expected_domtree.is_reachable( - self.func.layout.pp_ebb(loc_inst), - ) && + if is_reachable && !self.expected_domtree.dominates( def_inst, loc_inst, @@ -493,7 +493,7 @@ impl<'a> Verifier<'a> { ); } // The defining EBB dominates the instruction using this value. - if self.expected_domtree.is_reachable(ebb) && + if is_reachable && !self.expected_domtree.dominates( ebb, loc_inst, From cacba1a58f223111aa3eaf9130c24315cd3ccb81 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 11 Jan 2018 16:35:19 -0800 Subject: [PATCH 1458/3084] Don't allow EBB parameters to be ghost values. Ghost instructions and values are supposed to be stored as metadata alongside the compiled program such that the ghost values can be computed from the real register/stack values when the program is stopped for debugging or de-optimization. If we allow an EBB parameter to be a ghost value, we have no way of computing its real value using ghost instructions. We would need to know a complete execution trace of the stopped program to figure out which values were passed to the ghost parameter. Instead we require EBB parameters to be real values materialized in registers or on the stack. We use the regclass_for_abi_type() TargetIsa callback to determine the initial register class for these parameters. They can then be spilled later if needed. Fixes #215. --- cranelift/filetests/regalloc/ghost-param.cton | 43 ++++++++++++++ lib/cretonne/src/regalloc/liveness.rs | 58 +------------------ lib/cretonne/src/verifier/liveness.rs | 33 +---------- 3 files changed, 48 insertions(+), 86 deletions(-) create mode 100644 cranelift/filetests/regalloc/ghost-param.cton diff --git a/cranelift/filetests/regalloc/ghost-param.cton b/cranelift/filetests/regalloc/ghost-param.cton new file mode 100644 index 0000000000..8b0ba6c2ab --- /dev/null +++ b/cranelift/filetests/regalloc/ghost-param.cton @@ -0,0 +1,43 @@ +test regalloc +set is_64bit +isa intel haswell + +; This test case would create an EBB parameter that was a ghost value. +; The coalescer would insert a copy of the ghost value, leading to verifier errors. +; +; We don't allow EBB parameters to be ghost values any longer. +; +; Test case by binaryen fuzzer! + +function %pr215(i64 vmctx [%rdi]) native { +ebb0(v0: i64): + v10 = iconst.i64 0 + v1 = bitcast.f64 v10 + jump ebb5(v1) + +ebb5(v9: f64): + v11 = iconst.i64 0xffff_ffff_ff9a_421a + v4 = bitcast.f64 v11 + v6 = iconst.i32 0 + v7 = iconst.i32 1 + brnz v7, ebb4(v6) + v8 = iconst.i32 0 + jump ebb7(v8) + +ebb7(v5: i32): + brnz v5, ebb3(v4) + jump ebb5(v4) + +ebb4(v3: i32): + brnz v3, ebb2 + jump ebb3(v9) + +ebb3(v2: f64): + jump ebb2 + +ebb2: + jump ebb1 + +ebb1: + return +} diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 6eee4f0c8c..2778f47f63 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -229,9 +229,9 @@ fn get_or_create<'a>( // signature. affinity = Affinity::abi(&func.signature.params[num], isa); } else { - // Don't apply any affinity to normal EBB parameters. - // They could be in a register or on the stack. - affinity = Default::default(); + // Give normal EBB parameters a register affinity matching their type. + let rc = isa.regclass_for_abi_type(func.dfg.value_type(value)); + affinity = Affinity::Reg(rc.into()); } } }; @@ -294,9 +294,6 @@ pub struct Liveness { /// This vector is always empty, except for inside that function. /// It lives here to avoid repeated allocation of scratch memory. worklist: Vec, - - /// Working space for the `propagate_ebb_params` algorithm. - ebb_params: Vec, } impl Liveness { @@ -309,7 +306,6 @@ impl Liveness { ranges: LiveRangeSet::new(), forest: LiveRangeForest::new(), worklist: Vec::new(), - ebb_params: Vec::new(), } } @@ -323,7 +319,6 @@ impl Liveness { self.ranges.clear(); self.forest.clear(); self.worklist.clear(); - self.ebb_params.clear(); } /// Get the live range for `value`, if it exists. @@ -444,53 +439,6 @@ impl Liveness { // EBB arguments or call/return ABI arguments. if let Some(constraint) = operand_constraints.next() { lr.affinity.merge(constraint, ®_info); - } else if lr.affinity.is_none() && encoding.is_legal() && - !func.dfg[inst].opcode().is_branch() - { - // This is a real encoded instruction using a value that doesn't yet have a - // concrete affinity. Most likely a call argument or a return value. Give - // the value a register affinity matching the ABI type. - // - // EBB arguments on a branch are not required to have an affinity. - let rc = isa.regclass_for_abi_type(func.dfg.value_type(arg)); - lr.affinity = Affinity::Reg(rc.into()); - } - } - } - } - - self.propagate_ebb_params(func, cfg); - } - - /// Propagate affinities for EBB parameters. - /// - /// If an EBB argument value has an affinity, all predecessors must pass a value with an - /// affinity. - pub fn propagate_ebb_params(&mut self, func: &Function, cfg: &ControlFlowGraph) { - assert!(self.ebb_params.is_empty()); - - for ebb in func.layout.ebbs() { - for &arg in func.dfg.ebb_params(ebb) { - let affinity = self.ranges.get(arg).unwrap().affinity; - if affinity.is_none() { - continue; - } - self.ebb_params.push(arg); - - // Now apply the affinity to all predecessors recursively. - while let Some(succ_arg) = self.ebb_params.pop() { - let (succ_ebb, num) = match func.dfg.value_def(succ_arg) { - ValueDef::Param(e, n) => (e, n), - _ => continue, - }; - - for (_, pred_branch) in cfg.pred_iter(succ_ebb) { - let pred_arg = func.dfg.inst_variable_args(pred_branch)[num]; - let pred_affinity = &mut self.ranges.get_mut(pred_arg).unwrap().affinity; - if pred_affinity.is_none() { - *pred_affinity = affinity; - self.ebb_params.push(pred_arg); - } } } } diff --git a/lib/cretonne/src/verifier/liveness.rs b/lib/cretonne/src/verifier/liveness.rs index a58d04caf7..95691625c8 100644 --- a/lib/cretonne/src/verifier/liveness.rs +++ b/lib/cretonne/src/verifier/liveness.rs @@ -100,7 +100,7 @@ impl<'a> LivenessVerifier<'a> { } // Check the uses. - for (idx, &val) in self.func.dfg.inst_args(inst).iter().enumerate() { + for &val in self.func.dfg.inst_args(inst) { let lr = match self.liveness.get(val) { Some(lr) => lr, None => return err!(inst, "{} has no live range", val), @@ -111,10 +111,7 @@ impl<'a> LivenessVerifier<'a> { if encoding.is_legal() { // A legal instruction is not allowed to depend on ghost values. - // - // A branch argument can be a ghost value if the corresponding destination - // EBB argument is a ghost value. - if lr.affinity.is_none() && !self.is_ghost_branch_argument(inst, idx) { + if lr.affinity.is_none() { return err!( inst, "{} is a ghost value used by a real [{}] instruction", @@ -147,32 +144,6 @@ impl<'a> LivenessVerifier<'a> { } } - /// Is argument `argnum` on `inst` a branch argument that leads to a ghost EBB argument? - fn is_ghost_branch_argument(&self, inst: Inst, argnum: usize) -> bool { - let dest = match self.func.dfg[inst].branch_destination() { - Some(d) => d, - None => return false, - }; - - let fixed_args = self.func.dfg[inst] - .opcode() - .constraints() - .fixed_value_arguments(); - if argnum < fixed_args { - return false; - } - - // If the EBB argument value in the destination is a ghost value, we'll allow a ghost - // branch argument. - self.func - .dfg - .ebb_params(dest) - .get(argnum - fixed_args) - .and_then(|&v| self.liveness.get(v)) - .map(|lr| lr.affinity.is_none()) - .unwrap_or(false) - } - /// Check the integrity of the live range `lr`. fn check_lr(&self, def: ProgramPoint, val: Value, lr: &LiveRange) -> Result { let l = &self.func.layout; From 567e570c02e98fe83abcfa8fd5d074677eb06958 Mon Sep 17 00:00:00 2001 From: Yury Delendik Date: Fri, 12 Jan 2018 14:56:24 -0600 Subject: [PATCH 1459/3084] Allow to print translated wasm file. --- cranelift/src/wasm.rs | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 04810e2983..800bf7d856 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -102,6 +102,26 @@ fn handle_module( vprintln!(flag_verbose, "ok"); terminal.reset().unwrap(); if flag_just_decode { + if flag_print { + let num_func_imports = dummy_environ.get_num_func_imports(); + for (def_index, func) in dummy_environ.info.function_bodies.iter().enumerate() { + let func_index = num_func_imports + def_index; + let mut context = Context::new(); + context.func = func.clone(); + if let Some(start_func) = dummy_environ.info.start_func { + if func_index == start_func { + println!("; Selected as wasm start function"); + } + } + vprintln!(flag_verbose, ""); + for export_name in &dummy_environ.info.functions[func_index].export_names { + println!("; Exported as \"{}\"", export_name); + } + println!("{}", context.func.display(None)); + vprintln!(flag_verbose, ""); + } + terminal.reset().unwrap(); + } return Ok(()); } terminal.fg(term::color::MAGENTA).unwrap(); From 85aab278ddb451752d0b07b5fbef2d2931befdd8 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 12 Jan 2018 11:11:49 -0800 Subject: [PATCH 1460/3084] Add RISC-V encodings for b1 copy/spill/fill. We allow b1 values in general purpose registers, so we need to be able to move them around. --- lib/cretonne/meta/isa/riscv/encodings.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/cretonne/meta/isa/riscv/encodings.py index ac719460a1..374aef17d1 100644 --- a/lib/cretonne/meta/isa/riscv/encodings.py +++ b/lib/cretonne/meta/isa/riscv/encodings.py @@ -155,3 +155,8 @@ RV64.enc(base.copy.i32, Icopy, OPIMM32(0b000)) RV32.enc(base.regmove.i32, Irmov, OPIMM(0b000)) RV64.enc(base.regmove.i64, Irmov, OPIMM(0b000)) RV64.enc(base.regmove.i32, Irmov, OPIMM32(0b000)) + +RV32.enc(base.copy.b1, Icopy, OPIMM(0b000)) +RV64.enc(base.copy.b1, Icopy, OPIMM(0b000)) +RV32.enc(base.regmove.b1, Irmov, OPIMM(0b000)) +RV64.enc(base.regmove.b1, Irmov, OPIMM(0b000)) From ce4cc8ce12f28f6bb1cd756494c16a5e89880cf7 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 16 Jan 2018 10:26:13 -0800 Subject: [PATCH 1461/3084] Fix the handling of special types in type variables. - Allow the syntax "specials=True" to indicate that a type variable can assume all special types. Use this for the unconstrained type variable created in ast.py. - Fix TypeSet.copy() to avoid deepcopy() which doesn't do the right thing for the self.specials set. - Fix TypeSet.typeset_key() to just use the name of special types instead of the full SpecialType objects. --- lib/cretonne/meta/cdsl/ast.py | 3 +- lib/cretonne/meta/cdsl/typevar.py | 47 ++++++++++++++++++++++--------- 2 files changed, 35 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index d6c0f42952..38d633e17d 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -247,7 +247,8 @@ class Var(Atom): 'typeof_{}'.format(self), 'Type of the pattern variable `{}`'.format(self), ints=True, floats=True, bools=True, - scalars=True, simd=True, bitvecs=True) + scalars=True, simd=True, bitvecs=True, + specials=True) self.original_typevar = tv self.typevar = tv return self.typevar diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/cretonne/meta/cdsl/typevar.py index f818719a4d..8cda58785e 100644 --- a/lib/cretonne/meta/cdsl/typevar.py +++ b/lib/cretonne/meta/cdsl/typevar.py @@ -7,16 +7,17 @@ polymorphic by using type variables. from __future__ import absolute_import import math from . import types, is_power_of_two -from copy import deepcopy +from copy import copy try: from typing import Tuple, Union, Iterable, Any, Set, TYPE_CHECKING # noqa if TYPE_CHECKING: from srcgen import Formatter # noqa - from .types import ValueType # noqa Interval = Tuple[int, int] # An Interval where `True` means 'everything' BoolInterval = Union[bool, Interval] + # Set of special types: None, False, True, or iterable. + SpecialSpec = Union[bool, Iterable[types.SpecialType]] except ImportError: pass @@ -88,7 +89,7 @@ def decode_interval(intv, full_range, default=None): An explicit interval """ if isinstance(intv, tuple): - # mypy buig here: 'builtins.None' object is not iterable + # mypy bug here: 'builtins.None' object is not iterable lo, hi = intv assert is_power_of_two(lo) assert is_power_of_two(hi) @@ -175,7 +176,7 @@ class TypeSet(object): widths. :param bitvecs : `(min, max)` inclusive range of permitted bitvector widths. - :param specials: Sequence of speical types to appear in the set. + :param specials: Sequence of special types to appear in the set. """ def __init__( @@ -185,7 +186,7 @@ class TypeSet(object): floats=None, # type: BoolInterval bools=None, # type: BoolInterval bitvecs=None, # type: BoolInterval - specials=None # type: Iterable[types.SpecialType] + specials=None # type: SpecialSpec ): # type: (...) -> None self.lanes = interval_to_set(decode_interval(lanes, (1, MAX_LANES), 1)) @@ -195,15 +196,27 @@ class TypeSet(object): self.bools = set(filter(legal_bool, self.bools)) self.bitvecs = interval_to_set(decode_interval(bitvecs, (1, MAX_BITVEC))) - self.specials = set(specials) if specials else set() + # Allow specials=None, specials=True, specials=(...) + self.specials = set() # type: Set[types.SpecialType] + if isinstance(specials, bool): + if specials: + self.specials = set(types.ValueType.all_special_types) + elif specials: + self.specials = set(specials) def copy(self): # type: (TypeSet) -> TypeSet """ - Return a copy of our self. deepcopy is sufficient and safe here, since - TypeSet contains only sets of numbers. + Return a copy of our self. """ - return deepcopy(self) + n = TypeSet() + n.lanes = copy(self.lanes) + n.ints = copy(self.ints) + n.floats = copy(self.floats) + n.bools = copy(self.bools) + n.bitvecs = copy(self.bitvecs) + n.specials = copy(self.specials) + return n def typeset_key(self): # type: () -> Tuple[Tuple, Tuple, Tuple, Tuple, Tuple, Tuple] @@ -213,7 +226,7 @@ class TypeSet(object): tuple(sorted(list(self.floats))), tuple(sorted(list(self.bools))), tuple(sorted(list(self.bitvecs))), - tuple(sorted(list(self.specials)))) + tuple(sorted(s.name for s in self.specials))) def __hash__(self): # type: () -> int @@ -301,7 +314,8 @@ class TypeSet(object): self.ints.issubset(other.ints) and \ self.floats.issubset(other.floats) and \ self.bools.issubset(other.bools) and \ - self.bitvecs.issubset(other.bitvecs) + self.bitvecs.issubset(other.bitvecs) and \ + self.specials.issubset(other.specials) def lane_of(self): # type: () -> TypeSet @@ -340,6 +354,7 @@ class TypeSet(object): new.floats = set([x//2 for x in self.floats if x > 32]) new.bools = set([x//2 for x in self.bools if x > 8]) new.bitvecs = set([x//2 for x in self.bitvecs if x > 1]) + new.specials = set() return new @@ -354,6 +369,7 @@ class TypeSet(object): new.bools = set(filter(legal_bool, set([x*2 for x in self.bools if x < MAX_BITS]))) new.bitvecs = set([x*2 for x in self.bitvecs if x < MAX_BITVEC]) + new.specials = set() return new @@ -365,6 +381,7 @@ class TypeSet(object): new = self.copy() new.bitvecs = set() new.lanes = set([x//2 for x in self.lanes if x > 1]) + new.specials = set() return new @@ -376,6 +393,7 @@ class TypeSet(object): new = self.copy() new.bitvecs = set() new.lanes = set([x*2 for x in self.lanes if x < MAX_LANES]) + new.specials = set() return new @@ -394,6 +412,7 @@ class TypeSet(object): new.floats = set() new.bitvecs = set([lane_w * nlanes for lane_w in all_scalars for nlanes in self.lanes]) + new.specials = set() return new @@ -576,7 +595,7 @@ class TypeVar(object): bitvecs=False, # type: BoolInterval base=None, # type: TypeVar derived_func=None, # type: str - specials=None # type: Iterable[types.SpecialType] + specials=None # type: SpecialSpec ): # type: (...) -> None self.name = name @@ -603,7 +622,7 @@ class TypeVar(object): def singleton(typ): # type: (types.ValueType) -> TypeVar """Create a type variable that can only assume a single type.""" - scalar = None # type: ValueType + scalar = None # type: types.ValueType if isinstance(typ, types.VectorType): scalar = typ.base lanes = (typ.lanes, typ.lanes) @@ -806,7 +825,7 @@ class TypeVar(object): return TypeVar.derived(self, self.TOBITVEC) def singleton_type(self): - # type: () -> ValueType + # type: () -> types.ValueType """ If the associated typeset has a single type return it. Otherwise return None From 16ac4f65b3dc3437eb1caa5b8751ec939409312d Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 8 Jan 2018 13:29:53 -0800 Subject: [PATCH 1462/3084] Add support for textbook union-find to VirtRegs. The initial phase of computing virtual registers can now be implemented with a textbook union-find algorithm using a disjoint set forest complete with rank and path compression optimizations. The disjoint set forest is converted to virtual register value lists in a single linear scan implemented in finish_union_find(). This union-find algorithm will soon be used by the coalescer. --- lib/cretonne/src/regalloc/virtregs.rs | 335 +++++++++++++++++++++++++- 1 file changed, 333 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index b9388277e0..ce38dc3687 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -11,11 +11,15 @@ //! If any values in a virtual register are spilled, they will use the same stack slot. This avoids //! memory-to-memory copies when a spilled value is passed as an EBB argument. +use dbg::DisplayList; use entity::{EntityList, ListPool}; use entity::{PrimaryMap, EntityMap, Keys}; +use entity::EntityRef; use ir::Value; use packed_option::PackedOption; use ref_slice::ref_slice; +use std::cmp::Ordering; +use std::fmt; /// A virtual register reference. #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] @@ -35,26 +39,41 @@ pub struct VirtRegs { /// The primary table of virtual registers. vregs: PrimaryMap, + /// Allocated virtual register numbers that are no longer in use. + unused_vregs: Vec, + /// Each value belongs to at most one virtual register. value_vregs: EntityMap>, + + /// Table used during the union-find phase while `vregs` is empty. + union_find: EntityMap, + + /// Values that have been activated in the `union_find` table, but not yet added to any virtual + /// registers by the `finish_union_find()` function. + pending_values: Vec, } -#[allow(dead_code)] impl VirtRegs { /// Create a new virtual register collection. pub fn new() -> Self { Self { pool: ListPool::new(), vregs: PrimaryMap::new(), + unused_vregs: Vec::new(), value_vregs: EntityMap::new(), + union_find: EntityMap::new(), + pending_values: Vec::new(), } } /// Clear all virtual registers. pub fn clear(&mut self) { self.vregs.clear(); + self.unused_vregs.clear(); self.value_vregs.clear(); self.pool.clear(); + self.union_find.clear(); + self.pending_values.clear(); } /// Get the virtual register containing `value`, if any. @@ -63,7 +82,6 @@ impl VirtRegs { } /// Get the list of values in `vreg`. - /// The values are topologically ordered according dominance of their definition points. pub fn values(&self, vreg: VirtReg) -> &[Value] { self.vregs[vreg].as_slice(&self.pool) } @@ -94,6 +112,44 @@ impl VirtRegs { } } + /// Sort the values in `vreg` according to `compare`. + /// + /// If the ordering defined by `compare` is not total, value numbers are used as a last resort + /// tie-breaker. This makes it possible to use an unstable sorting algorithm which can be + /// faster because it doesn't allocate memory. + /// + /// Returns the slice of sorted values which `values(vreg)` will also return from now on. + pub fn sort_values(&mut self, vreg: VirtReg, mut compare: F) -> &[Value] + where + F: FnMut(Value, Value) -> Ordering, + { + let s = self.vregs[vreg].as_mut_slice(&mut self.pool); + s.sort_unstable_by(|&a, &b| compare(a, b).then(a.cmp(&b))); + s + } + + /// Remove a virtual register. + /// + /// The values in `vreg` become singletons, and the virtual register number may be reused in + /// the future. + pub fn remove(&mut self, vreg: VirtReg) { + // Start by reassigning all the values. + for &v in self.vregs[vreg].as_slice(&self.pool) { + let old = self.value_vregs[v].take(); + debug_assert_eq!(old, Some(vreg)); + } + + self.vregs[vreg].clear(&mut self.pool); + self.unused_vregs.push(vreg); + } + + /// Allocate a new empty virtual register. + fn alloc(&mut self) -> VirtReg { + self.unused_vregs.pop().unwrap_or_else(|| { + self.vregs.push(Default::default()) + }) + } + /// Unify `values` into a single virtual register. /// /// The values in the slice can be singletons or they can belong to a virtual register already. @@ -138,3 +194,278 @@ impl VirtRegs { vreg } } + +impl fmt::Display for VirtRegs { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + for vreg in self.all_virtregs() { + write!(f, "\n{} = {}", vreg, DisplayList(self.values(vreg)))?; + } + Ok(()) + } +} + +/// Expanded version of a union-find table entry. +enum UFEntry { + /// This value is a a set leader. The embedded number is the set's rank. + Rank(u32), + + /// This value belongs to the same set as the linked value. + Link(Value), +} + +/// The `union_find` table contains `i32` entries that are interpreted as follows: +/// +/// x = 0: The value belongs to its own singleton set. +/// x > 0: The value is the leader of a set with rank x. +/// x < 0: The value belongs to the same set as the value numbered !x. +/// +/// The rank of a set is an upper bound on the number of links that must be followed from a member +/// of the set to the set leader. +/// +/// A singleton set is the same as a set with rank 0. It contains only the leader value. +impl UFEntry { + /// Decode a table entry. + fn decode(x: i32) -> UFEntry { + if x < 0 { + UFEntry::Link(Value::new((!x) as usize)) + } else { + UFEntry::Rank(x as u32) + } + } + + /// Encode a link entry. + fn encode_link(v: Value) -> i32 { + !(v.index() as i32) + } +} + +/// Union-find algorithm for building virtual registers. +/// +/// Before values are added to virtual registers, it is possible to use a union-find algorithm to +/// construct virtual registers efficiently. This support implemented here is used as follows: +/// +/// 1. Repeatedly call the `union(a, b)` method to request that `a` and `b` are placed in the same +/// virtual register. +/// 2. When done, call `finish_union_find()` to construct the virtual register sets based on the +/// `union()` calls. +/// +/// The values that were passed to `union(a, b)` mist not belong to any existing virtual registers +/// by the time `finish_union_find()` is called. +/// +/// For more information on the algorithm implemented here, see Chapter 21 "Data Structures for +/// Disjoint Sets" of Cormen, Leiserson, Rivest, Stein, "Introduction to algorithms", 3rd Ed. +/// +/// The [Wikipedia entry on disjoint-set data +/// structures](https://en.wikipedia.org/wiki/Disjoint-set_data_structure) is also good. +impl VirtRegs { + /// Find the leader value and rank of the set containing `v`. + /// Compress the path if needed. + fn find(&mut self, val: Value) -> (Value, u32) { + match UFEntry::decode(self.union_find[val]) { + UFEntry::Rank(rank) => (val, rank), + UFEntry::Link(parent) => { + // TODO: This recursion would be more efficient as an iteration that pushes + // elements onto a SmallVector. + let found = self.find(parent); + // Compress the path if needed. + if found.0 != parent { + self.union_find[val] = UFEntry::encode_link(found.0); + } + found + } + } + } + + /// Union the two sets containing `a` and `b`. + /// + /// This ensures that `a` and `b` will belong to the same virtual register after calling + /// `finish_union_find()`. + pub fn union(&mut self, a: Value, b: Value) { + let (leader_a, rank_a) = self.find(a); + let (leader_b, rank_b) = self.find(b); + + if leader_a == leader_b { + return; + } + + // The first time we see a value, its rank will be 0. Add it to the list of pending values. + if rank_a == 0 { + debug_assert_eq!(a, leader_a); + self.pending_values.push(a); + } + if rank_b == 0 { + debug_assert_eq!(b, leader_b); + self.pending_values.push(b); + } + + // Merge into the set with the greater rank. This preserves the invariant that the rank is + // an upper bound on the number of links to the leader. + match rank_a.cmp(&rank_b) { + Ordering::Less => { + self.union_find[leader_a] = UFEntry::encode_link(leader_b); + } + Ordering::Greater => { + self.union_find[leader_b] = UFEntry::encode_link(leader_a); + } + Ordering::Equal => { + // When the two sets have the same rank, we arbitrarily pick the a-set to preserve. + // We need to increase the rank by one since the elements in the b-set are now one + // link further away from the leader. + self.union_find[leader_a] += 1; + self.union_find[leader_b] = UFEntry::encode_link(leader_a); + } + } + } + + /// Compute virtual registers based on previous calls to `union(a, b)`. + /// + /// This terminates the union-find algorithm, so the next time `union()` is called, it is for a + /// new independent batch of values. + /// + /// The values in each virtual register will be ordered according to when they were first + /// passed to `union()`, but backwards. It is expected that `sort_values()` will be used to + /// create a more sensible value order. + /// + /// The new virtual registers will be appended to `new_vregs`, if present. + pub fn finish_union_find(&mut self, mut new_vregs: Option<&mut Vec>) { + debug_assert_eq!( + self.pending_values.iter().find(|&&v| self.get(v).is_some()), + None, + "Values participating in union-find must not belong to existing virtual registers" + ); + + while let Some(val) = self.pending_values.pop() { + let (leader, _) = self.find(val); + + // Get the vreg for `leader`, or create it. + let vreg = self.get(leader).unwrap_or_else(|| { + // Allocate a vreg for `leader`, but leave it empty. + let vr = self.alloc(); + if let &mut Some(ref mut vec) = &mut new_vregs { + vec.push(vr); + } + self.value_vregs[leader] = vr.into(); + vr + }); + + // Push values in `pending_values` order, including when `v == leader`. + self.vregs[vreg].push(val, &mut self.pool); + self.value_vregs[val] = vreg.into(); + + // Clear the entry in the union-find table. The `find(val)` call may still look at this + // entry in a future iteration, but that it ok. It will return a rank 0 leader that has + // already been assigned to the correct virtual register. + self.union_find[val] = 0; + } + + // We do *not* call `union_find.clear()` table here because re-initializing the table for + // sparse use takes time linear in the number of values in the function. Instead we reset + // the entries that are known to be non-zero in the loop above. + } +} + +#[cfg(test)] +mod test { + use super::*; + use entity::EntityRef; + use ir::Value; + + #[test] + fn empty_union_find() { + let mut vregs = VirtRegs::new(); + vregs.finish_union_find(None); + assert_eq!(vregs.all_virtregs().count(), 0); + } + + #[test] + fn union_self() { + let mut vregs = VirtRegs::new(); + let v1 = Value::new(1); + vregs.union(v1, v1); + vregs.finish_union_find(None); + assert_eq!(vregs.get(v1), None); + assert_eq!(vregs.all_virtregs().count(), 0); + } + + #[test] + fn union_pair() { + let mut vregs = VirtRegs::new(); + let v1 = Value::new(1); + let v2 = Value::new(2); + vregs.union(v1, v2); + vregs.finish_union_find(None); + assert_eq!(vregs.congruence_class(&v1), &[v2, v1]); + assert_eq!(vregs.congruence_class(&v2), &[v2, v1]); + assert_eq!(vregs.all_virtregs().count(), 1); + } + + #[test] + fn union_pair_backwards() { + let mut vregs = VirtRegs::new(); + let v1 = Value::new(1); + let v2 = Value::new(2); + vregs.union(v2, v1); + vregs.finish_union_find(None); + assert_eq!(vregs.congruence_class(&v1), &[v1, v2]); + assert_eq!(vregs.congruence_class(&v2), &[v1, v2]); + assert_eq!(vregs.all_virtregs().count(), 1); + } + + #[test] + fn union_tree() { + let mut vregs = VirtRegs::new(); + let v1 = Value::new(1); + let v2 = Value::new(2); + let v3 = Value::new(3); + let v4 = Value::new(4); + + vregs.union(v2, v4); + vregs.union(v3, v1); + // Leaders: v2, v3 + vregs.union(v4, v1); + vregs.finish_union_find(None); + assert_eq!(vregs.congruence_class(&v1), &[v1, v3, v4, v2]); + assert_eq!(vregs.congruence_class(&v2), &[v1, v3, v4, v2]); + assert_eq!(vregs.congruence_class(&v3), &[v1, v3, v4, v2]); + assert_eq!(vregs.congruence_class(&v4), &[v1, v3, v4, v2]); + assert_eq!(vregs.all_virtregs().count(), 1); + } + + #[test] + fn union_two() { + let mut vregs = VirtRegs::new(); + let v1 = Value::new(1); + let v2 = Value::new(2); + let v3 = Value::new(3); + let v4 = Value::new(4); + + vregs.union(v2, v4); + vregs.union(v3, v1); + // Leaders: v2, v3 + vregs.finish_union_find(None); + assert_eq!(vregs.congruence_class(&v1), &[v1, v3]); + assert_eq!(vregs.congruence_class(&v2), &[v4, v2]); + assert_eq!(vregs.congruence_class(&v3), &[v1, v3]); + assert_eq!(vregs.congruence_class(&v4), &[v4, v2]); + assert_eq!(vregs.all_virtregs().count(), 2); + } + + #[test] + fn union_uneven() { + let mut vregs = VirtRegs::new(); + let v1 = Value::new(1); + let v2 = Value::new(2); + let v3 = Value::new(3); + let v4 = Value::new(4); + + vregs.union(v2, v4); // Rank 0-0 + vregs.union(v3, v2); // Rank 0-1 + vregs.union(v2, v1); // Rank 1-0 + vregs.finish_union_find(None); + assert_eq!(vregs.congruence_class(&v1), &[v1, v3, v4, v2]); + assert_eq!(vregs.congruence_class(&v2), &[v1, v3, v4, v2]); + assert_eq!(vregs.congruence_class(&v3), &[v1, v3, v4, v2]); + assert_eq!(vregs.congruence_class(&v4), &[v1, v3, v4, v2]); + assert_eq!(vregs.all_virtregs().count(), 1); + } +} From d1f236b00aa1ec3f06c11862f3466302fd2377ac Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 9 Jan 2018 10:33:02 -0800 Subject: [PATCH 1463/3084] Reimplement coalescer following the Budimlic paper. The old coalescing algorithm had some algorithmic complexity issues when dealing with large virtual registers. Reimplement to use a proper union-find algorithm so we only need one pass through the dominator forests for virtual registers that are interference free. Virtual registers that do have interference are split and new registers built. This pass is about twice as fast as the old one when dealing with complex virtual registers. --- cranelift/filetests/regalloc/coalesce.cton | 12 +- .../regalloc/infinite-interference.cton | 37 + lib/cretonne/src/ir/dfg.rs | 19 +- lib/cretonne/src/regalloc/coalescing.rs | 1021 +++++++++-------- 4 files changed, 604 insertions(+), 485 deletions(-) create mode 100644 cranelift/filetests/regalloc/infinite-interference.cton diff --git a/cranelift/filetests/regalloc/coalesce.cton b/cranelift/filetests/regalloc/coalesce.cton index bdddf26ef4..0deac6fc85 100644 --- a/cranelift/filetests/regalloc/coalesce.cton +++ b/cranelift/filetests/regalloc/coalesce.cton @@ -40,9 +40,8 @@ ebb1(v10: i32): function %dualuse(i32) -> i32 { ebb0(v0: i32): ; check: $(cp1=$V) = copy $v0 - ; nextln: brnz $v0, $ebb1($v0, $cp1) + ; nextln: brnz $v0, $ebb1($cp1, $v0) brnz v0, ebb1(v0, v0) - ; not: copy v1 = iadd_imm v0, 7 v2 = iadd_imm v1, 56 jump ebb1(v1, v2) @@ -56,14 +55,15 @@ ebb1(v10: i32, v11: i32): ; The interference can be broken with a copy at either branch. function %interference(i32) -> i32 { ebb0(v0: i32): + ; check: $(cp0=$V) = copy $v0 ; not: copy + ; check: brnz $v0, ebb1($cp0) brnz v0, ebb1(v0) v1 = iadd_imm v0, 7 ; v1 and v0 interfere here: v2 = iadd_imm v0, 8 - ; check: $(cp1=$V) = copy $v1 ; not: copy - ; check: jump $ebb1($cp1) + ; check: jump $ebb1($v1) jump ebb1(v1) ebb1(v10: i32): @@ -75,7 +75,6 @@ ebb1(v10: i32): ; A loop where one induction variable is used as a backedge argument. function %fibonacci(i32) -> i32 { ebb0(v0: i32): - ; not: copy v1 = iconst.i32 1 v2 = iconst.i32 2 jump ebb1(v1, v2) @@ -103,8 +102,7 @@ function %stackarg(i32, i32, i32, i32, i32, i32, i32, i32, i32) -> i32 { ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32, v7: i32, v8: i32): ; check: fill v8 ; not: v8 - brnz v0, ebb1(v8) - jump ebb1(v7) + jump ebb1(v8) ebb1(v10: i32): v11 = iadd_imm v10, 1 diff --git a/cranelift/filetests/regalloc/infinite-interference.cton b/cranelift/filetests/regalloc/infinite-interference.cton new file mode 100644 index 0000000000..8f07c4cf77 --- /dev/null +++ b/cranelift/filetests/regalloc/infinite-interference.cton @@ -0,0 +1,37 @@ +test regalloc +isa riscv + +; Here, the coalescer initially builds vreg0 = [v1, v2, v3] +; +; There's interference between v1 and v2 at the brz instruction. Isolating v2 is not going to +; resolve that conflict since v1 will just interfere with the inserted copy too. + +;function %c1(i32) -> i32 { +;ebb0(v0: i32): +; v1 = iadd_imm v0, 1 +; v2 = iconst.i32 1 +; brz v1, ebb1(v2) +; jump ebb2 +; +;ebb1(v3: i32): +; return v3 +; +;ebb2: +; jump ebb1(v1) +;} + +; Same thing with v1 and v2 swapped to reverse the order of definitions. + +function %c2(i32) -> i32 { +ebb0(v0: i32): + v1 = iadd_imm v0, 1 + v2 = iconst.i32 1 + brz v2, ebb1(v1) + jump ebb2 + +ebb1(v3: i32): + return v3 + +ebb2: + jump ebb1(v2) +} diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 50842b3fa5..25457000d7 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -2,6 +2,7 @@ use entity::{PrimaryMap, EntityMap}; use isa::TargetIsa; +use ir; use ir::builder::ReplaceBuilder; use ir::extfunc::ExtFuncData; use ir::instructions::{InstructionData, CallInfo, BranchInfo}; @@ -315,7 +316,7 @@ impl DataFlowGraph { } /// Where did a value come from? -#[derive(Debug, PartialEq, Eq)] +#[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum ValueDef { /// Value is the n'th result of an instruction. Result(Inst, usize), @@ -331,6 +332,22 @@ impl ValueDef { _ => panic!("Value is not an instruction result"), } } + + /// Get the program point where the value was defined. + pub fn pp(self) -> ir::ExpandedProgramPoint { + self.into() + } + + /// Get the number component of this definition. + /// + /// When multiple values are defined at the same program point, this indicates the index of + /// this value. + pub fn num(self) -> usize { + match self { + ValueDef::Result(_, n) | + ValueDef::Param(_, n) => n, + } + } } // Internal table storage for extended values. diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 4819969605..ff01542bd8 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -9,277 +9,55 @@ use cursor::{Cursor, EncCursor}; use dbg::DisplayList; use dominator_tree::{DominatorTree, DominatorTreePreorder}; use flowgraph::ControlFlowGraph; -use ir::{Layout, InstBuilder, ValueDef}; +use ir::{self, InstBuilder}; use ir::{Function, Ebb, Inst, Value, ExpandedProgramPoint}; use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; -use regalloc::virtregs::VirtRegs; -use std::cmp::Ordering; +use regalloc::virtregs::{VirtReg, VirtRegs}; use std::fmt; -use std::iter::Peekable; -use std::mem; use isa::{TargetIsa, EncInfo}; use timing; -/// Dominator forest. -/// -/// This is a utility type used for merging virtual registers, where each virtual register is a -/// list of values ordered according to the dominator tree pre-order. -/// -/// A `DomForest` object is used as a buffer for building virtual registers. It lets you merge two -/// sorted lists of values while checking for interference only where necessary. -/// -/// The idea of a dominator forest was introduced here: -/// -/// Budimlic, Z., Budimlic, Z., Cooper, K. D., Cooper, K. D., Harvey, T. J., Harvey, T. J., et al. -/// (2002). Fast copy coalescing and live-range identification (Vol. 37, pp. 25–32). ACM. -/// http://doi.org/10.1145/543552.512534 -/// -/// The linear stack representation here: -/// -/// Boissinot, B., Darte, A., & Rastello, F. (2009). Revisiting out-of-SSA translation for -/// correctness, code quality and efficiency. -/// -/// Our version of the linear stack is slightly modified because we have a pre-order of the -/// dominator tree at the EBB granularity, not basic block granularity. -struct DomForest { - // The sequence of values that have been merged so far. - // In domtree pre-order order of their definitions. - values: Vec, - - // Stack representing the rightmost edge of the dominator forest so far, ending in the last - // element of `values`. - // - // At all times, the EBB of each element in the stack dominates the EBB of the next one, and - // all elements dominating the end of `values` are on the stack. - stack: Vec, -} - -/// A node in the dominator forest. -#[derive(Clone, Copy, Debug)] -struct Node { - value: Value, - /// Set identifier. Values in the same set are assumed to be non-interfering. - set: u8, - /// The program point where `value` is defined. - def: ExpandedProgramPoint, - /// EBB containing `def`. - ebb: Ebb, -} - -impl Node { - /// Create a node for `value`. - pub fn new(value: Value, set: u8, func: &Function) -> Node { - let def = func.dfg.value_def(value).into(); - let ebb = func.layout.pp_ebb(def); - Node { - value, - set, - def, - ebb, - } - } -} - -impl fmt::Display for Node { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - write!(f, "{}@{}:{}", self.value, self.ebb, self.set) - } -} - -impl DomForest { - /// Create a new empty dominator forest. - pub fn new() -> Self { - Self { - values: Vec::new(), - stack: Vec::new(), - } - } - - /// Clear all data structures in this dominator forest. - pub fn clear(&mut self) { - self.values.clear(); - self.stack.clear(); - } - - /// Swap the merged list with `buffer`, leaving the dominator forest empty. - /// - /// This is typically called after a successful merge to extract the merged value list. - pub fn swap(&mut self, buffer: &mut Vec) { - buffer.clear(); - mem::swap(&mut self.values, buffer); - } - - /// Add a single node to the forest. - /// - /// Update the stack so its dominance invariants are preserved. Detect a parent node on the - /// stack which is the closest one dominating the new node. - /// - /// If the pushed node's parent in the dominator forest belongs to a different set, returns - /// `Some(parent)`. - fn push_node( - &mut self, - node: Node, - layout: &Layout, - domtree: &DominatorTree, - preorder: &DominatorTreePreorder, - ) -> Option { - self.values.push(node.value); - - // The stack contains the current sequence of dominating defs. Pop elements until we - // find one whose EBB dominates `node.ebb`. - while let Some(top) = self.stack.pop() { - if preorder.dominates(top.ebb, node.ebb) { - // This is the right insertion spot for `node`. - self.stack.push(top); - self.stack.push(node); - - // We know here that `top.ebb` dominates `node.ebb`, and thus `node.def`. This does - // not necessarily mean that `top.def` dominates `node.def`, though. The `top.def` - // program point may be below the last branch in `top.ebb` that dominates - // `node.def`. - debug_assert!(domtree.dominates(top.ebb, node.def, layout)); - - // We do know, though, that if there is a nearest value dominating `node.def`, it - // will be on the stack. We just need to find the last stack entry that actually - // dominates. - // - // TODO: This search could be more efficient if we had access to - // `domtree.last_dominator()`. Each call to `dominates()` here ends up walking up - // the dominator tree starting from `node.ebb`. - let dom = self.stack[0..self.stack.len() - 1].iter().rposition(|n| { - domtree.dominates(n.def, node.def, layout) - }); - - // If the parent value comes from a different set, return it for interference - // checking. If the sets are equal, assume that interference is already handled. - if let Some(pos) = dom { - let parent = &self.stack[pos]; - if parent.set != node.set { - return Some(parent.value); - } - } - - // There was no opposite-set value dominating `node.def`. - return None; - } - } - - // No dominators, start a new tree in the forest. - self.stack.push(node); - None - } - - /// Try to merge two sorted sets of values. Each slice must already be sorted and free of any - /// interference. - /// - /// It is permitted for a value to appear in both lists. The merged sequence will only have one - /// copy of the value. - /// - /// If an interference is detected, returns `Err((a, b))` with the two conflicting values form - /// `va` and `vb` respectively. - /// - /// If the merge succeeds, returns `Ok(())`. The merged sequence can be extracted with - /// `swap()`. - pub fn try_merge( - &mut self, - va: &[Value], - vb: &[Value], - func: &Function, - domtree: &DominatorTree, - preorder: &DominatorTreePreorder, - liveness: &Liveness, - ) -> Result<(), (Value, Value)> { - self.clear(); - self.values.reserve_exact(va.len() + vb.len()); - - // Convert the two value lists into a merged sequence of nodes. - let merged = MergedNodes { - a: va.iter().map(|&value| Node::new(value, 0, func)).peekable(), - b: vb.iter().map(|&value| Node::new(value, 1, func)).peekable(), - layout: &func.layout, - preorder, - }; - let ctx = liveness.context(&func.layout); - for node in merged { - if let Some(parent) = self.push_node(node, &func.layout, domtree, preorder) { - // Check if `parent` live range contains `node.def`. - if liveness[parent].overlaps_def(node.def, func.layout.pp_ebb(node.def), ctx) { - // Interference detected. Get the `(a, b)` order right in the error. - return Err(if node.set == 0 { - (node.value, parent) - } else { - (parent, node.value) - }); - } - } - } - - Ok(()) - } -} - -/// Node-merging iterator. -/// -/// Given two ordered sequences of nodes, yield an ordered sequence containing all of them. -/// Duplicates are removed. -struct MergedNodes<'a, IA, IB> -where - IA: Iterator, - IB: Iterator, -{ - a: Peekable, - b: Peekable, - layout: &'a Layout, - preorder: &'a DominatorTreePreorder, -} - -impl<'a, IA, IB> Iterator for MergedNodes<'a, IA, IB> -where - IA: Iterator, - IB: Iterator, -{ - type Item = Node; - - fn next(&mut self) -> Option { - let ord = match (self.a.peek(), self.b.peek()) { - (Some(a), Some(b)) => { - // If the two values are defined at the same point, compare value numbers instead - // this is going to cause an interference conflict unless its actually the same - // value appearing in both streams. - self.preorder.pre_cmp(a.def, b.def, self.layout).then( - Ord::cmp( - &a.value, - &b.value, - ), - ) - } - (Some(_), None) => Ordering::Less, - (None, Some(_)) => Ordering::Greater, - (None, None) => return None, - }; - match ord { - Ordering::Equal => { - // The two iterators produced the same value. Just return the first one. - self.b.next(); - self.a.next() - } - Ordering::Less => self.a.next(), - Ordering::Greater => self.b.next(), - } - } -} +// # Implementation +// +// The coalescing algorithm implemented follows this paper fairly closely: +// +// Budimlic, Z., Cooper, K. D., Harvey, T. J., et al. (2002). Fast copy coalescing and +// live-range identification (Vol. 37, pp. 25–32). ACM. http://doi.org/10.1145/543552.512534 +// +// We use a more efficient dominator forest representation (a linear stack) described here: +// +// Boissinot, B., Darte, A., & Rastello, F. (2009). Revisiting out-of-SSA translation for +// correctness, code quality and efficiency. +// +// The algorithm has two main phases: +// +// Phase 1: Union-find. +// +// We use the union-find support in `VirtRegs` to build virtual registers such that EBB parameter +// values always belong to the same virtual register as their corresponding EBB arguments at the +// predecessor branches. Trivial interferences between parameter and argument value live ranges are +// detected and resolved before unioning congruence classes, but non-trivial interferences between +// values that end up in the same congruence class are possible. +// +// Phase 2: Dominator forests. +// +// The virtual registers formed in phase 1 can contain interferences that we need to detect and +// eliminate. By ordering the values in a virtual register according to a dominator tree pre-order, +// we can identify all interferences in the virtual register in linear time. +// +// Interfering values are isolated and virtual registers rebuilt. /// Data structures to be used by the coalescing pass. pub struct Coalescing { forest: DomForest, preorder: DominatorTreePreorder, - // Current set of coalesced values. Kept sorted and interference free. - values: Vec, + /// EBB parameter values present in the current virtual register. + params: Vec, - // New values that were created when splitting interferences. - split_values: Vec, + /// Worklist of virtual registers that need to be processed. + worklist: Vec, } /// One-shot context created once per invocation. @@ -295,8 +73,8 @@ struct Context<'a> { virtregs: &'a mut VirtRegs, forest: &'a mut DomForest, - values: &'a mut Vec, - split_values: &'a mut Vec, + params: &'a mut Vec, + worklist: &'a mut Vec, } impl Coalescing { @@ -305,8 +83,8 @@ impl Coalescing { Self { forest: DomForest::new(), preorder: DominatorTreePreorder::new(), - values: Vec::new(), - split_values: Vec::new(), + params: Vec::new(), + worklist: Vec::new(), } } @@ -314,8 +92,8 @@ impl Coalescing { /// Clear all data structures in this coalescing pass. pub fn clear(&mut self) { self.forest.clear(); - self.values.clear(); - self.split_values.clear(); + self.params.clear(); + self.worklist.clear(); } /// Convert `func` to conventional SSA form and build virtual registers in the process. @@ -341,212 +119,216 @@ impl Coalescing { liveness, virtregs, forest: &mut self.forest, - values: &mut self.values, - split_values: &mut self.split_values, + params: &mut self.params, + worklist: &mut self.worklist, }; - // TODO: The iteration order matters here. We should coalesce in the most important blocks - // first, so they get first pick at forming virtual registers. + // Run phase 1 (union-find) of the coalescing algorithm on the current function. for &ebb in domtree.cfg_postorder() { - for argnum in 0..context.func.dfg.num_ebb_params(ebb) { - context.coalesce_ebb_param(ebb, argnum) - } + context.union_find_ebb(ebb); } + context.finish_union_find(); + + // Run phase 2 (dominator forests) on the current function. + context.process_vregs(); } } +/// Phase 1: Union-find. +/// +/// The two entry points for phase 1 are `union_find_ebb()` and `finish_union_find`. impl<'a> Context<'a> { - /// Coalesce the `argnum`'th parameter on `ebb`. - fn coalesce_ebb_param(&mut self, ebb: Ebb, argnum: usize) { - self.split_values.clear(); - let mut succ_val = self.func.dfg.ebb_params(ebb)[argnum]; - dbg!("Processing {}/{}: {}", ebb, argnum, succ_val); - - // We want to merge the virtual register for `succ_val` with the virtual registers for - // the branch arguments in the predecessors. This may not be possible if any live - // ranges interfere, so we can insert copies to break interferences: - // - // pred: - // jump ebb1(v1) - // - // ebb1(v10: i32): - // ... - // - // In the predecessor: - // - // v2 = copy v1 - // jump ebb(v2) - // - // A predecessor copy is always required if the branch argument virtual register is - // live into the successor. - // - // In the successor: - // - // ebb1(v11: i32): - // v10 = copy v11 - // - // A successor copy is always required if the `succ_val` virtual register is live at - // any predecessor branch. - - while let Some(bad_value) = self.try_coalesce(argnum, succ_val, ebb) { - dbg!("Isolating interfering value {}", bad_value); - // The bad value has some conflict that can only be reconciled by excluding its - // congruence class from the new virtual register. - // - // Try to catch infinite splitting loops. The values created by splitting should never - // have irreconcilable interferences. - assert!( - !self.split_values.contains(&bad_value), - "{} was already isolated", - bad_value - ); - let split_len = self.split_values.len(); - - // The bad value can be both the successor value and a predecessor value at the same - // time. - if self.virtregs.same_class(bad_value, succ_val) { - succ_val = self.split_succ(ebb, succ_val); - } - - // Check the predecessors. - for (pred_ebb, pred_inst) in self.cfg.pred_iter(ebb) { - let pred_val = self.func.dfg.inst_variable_args(pred_inst)[argnum]; - if self.virtregs.same_class(bad_value, pred_val) { - self.split_pred(pred_inst, pred_ebb, argnum, pred_val); - } - } - - // Second loop check. - assert_ne!( - split_len, - self.split_values.len(), - "Couldn't isolate {}", - bad_value - ); - } - - let vreg = self.virtregs.unify(self.values); - dbg!( - "Coalesced {} arg {} into {} = {}", - ebb, - argnum, - vreg, - DisplayList(self.virtregs.values(vreg)) - ); - } - - /// Reset `self.values` to just the set of split values. - fn reset_values(&mut self) { - self.values.clear(); - self.values.extend_from_slice(self.split_values); - let domtree = &self.domtree; - let func = &self.func; - self.values.sort_by(|&a, &b| { - domtree.rpo_cmp(func.dfg.value_def(a), func.dfg.value_def(b), &func.layout) - }); - } - - /// Try coalescing predecessors with `succ_val`. + /// Run the union-find algorithm on the parameter values on `ebb`. /// - /// Returns a value from a congruence class that needs to be split before starting over, or - /// `None` if everything was successfully coalesced into `self.values`. - fn try_coalesce(&mut self, argnum: usize, succ_val: Value, succ_ebb: Ebb) -> Option { - // Initialize the value list with the split values. These are guaranteed to be - // interference free, and anything that interferes with them must be split away. - self.reset_values(); - dbg!("Trying {} with split values: {:?}", succ_val, self.values); - - // Start by adding `succ_val` so we can determine if it interferes with any of the new - // split values. If it does, we must split it. - if self.add_class(succ_val).is_err() { - return Some(succ_val); + /// This ensure that all EBB parameters will belong to the same virtual register as their + /// corresponding arguments at all predecessor branches. + pub fn union_find_ebb(&mut self, ebb: Ebb) { + let num_params = self.func.dfg.num_ebb_params(ebb); + if num_params == 0 { + return; } - for (pred_ebb, pred_inst) in self.cfg.pred_iter(succ_ebb) { - let pred_val = self.func.dfg.inst_variable_args(pred_inst)[argnum]; + self.isolate_conflicting_params(ebb, num_params); + + for i in 0..num_params { + self.union_pred_args(ebb, i); + } + } + + // Identify EBB parameter values that are live at one of the predecessor branches. + // + // Such a parameter value will conflict with any argument value at the predecessor branch, so + // it must be isolated by inserting a copy. + fn isolate_conflicting_params(&mut self, ebb: Ebb, num_params: usize) { + debug_assert_eq!(num_params, self.func.dfg.num_ebb_params(ebb)); + // The only way a parameter value can interfere with a predecessor branch is if the EBB is + // dominating the predecessor branch. That is, we are looking for loop back-edges. + for (pred_ebb, pred_inst) in self.cfg.pred_iter(ebb) { + // The quick pre-order dominance check is accurate because the EBB parameter is defined + // at the top of the EBB before any branches. + if !self.preorder.dominates(ebb, pred_ebb) { + continue; + } + dbg!( - "Checking {}: {}: {}", - pred_val, + " - checking {} params at back-edge {}: {}", + num_params, pred_ebb, self.func.dfg.display_inst(pred_inst, self.isa) ); + // Now `pred_inst` is known to be a back-edge, so it is possible for parameter values + // to be live at the use. + for i in 0..num_params { + let param = self.func.dfg.ebb_params(ebb)[i]; + if self.liveness[param].reaches_use( + pred_inst, + pred_ebb, + self.liveness.context(&self.func.layout), + ) + { + self.isolate_param(ebb, param); + } + } + } + } + + // Union EBB parameter value `num` with the corresponding EBB arguments on the predecessor + // branches. + // + // Detect cases where the argument value is live-in to `ebb` so it conflicts with any EBB + // parameter. Isolate the argument in those cases before unioning it with the parameter value. + fn union_pred_args(&mut self, ebb: Ebb, argnum: usize) { + let param = self.func.dfg.ebb_params(ebb)[argnum]; + + for (pred_ebb, pred_inst) in self.cfg.pred_iter(ebb) { + let arg = self.func.dfg.inst_variable_args(pred_inst)[argnum]; + // Never coalesce incoming function parameters on the stack. These parameters are // pre-spilled, and the rest of the virtual register would be forced to spill to the // `incoming_arg` stack slot too. - if let ValueDef::Param(def_ebb, def_num) = self.func.dfg.value_def(pred_val) { + if let ir::ValueDef::Param(def_ebb, def_num) = self.func.dfg.value_def(arg) { if Some(def_ebb) == self.func.layout.entry_block() && self.func.signature.params[def_num].location.is_stack() { - dbg!("Isolating incoming stack parameter {}", pred_val); - let new_val = self.split_pred(pred_inst, pred_ebb, argnum, pred_val); - assert!(self.add_class(new_val).is_ok()); + dbg!("-> isolating function stack parameter {}", arg); + let new_arg = self.isolate_arg(pred_ebb, pred_inst, argnum, arg); + self.virtregs.union(param, new_arg); continue; } } - if let Err((a, b)) = self.add_class(pred_val) { - dbg!("Found conflict between {} and {}", a, b); - // We have a conflict between the already merged value `a` and one of the new - // values `b`. + // Check for basic interference: If `arg` overlaps a value defined at the entry to + // `ebb`, it can never be used as an EBB argument. + let interference = { + let lr = &self.liveness[arg]; + let ctx = self.liveness.context(&self.func.layout); + + // There are two ways the argument value can interfere with `ebb`: // - // Check if the `a` live range is fundamentally incompatible with `pred_inst`. - if self.liveness - .get(a) - .expect("No live range for interfering value") - .reaches_use( - pred_inst, - pred_ebb, - self.liveness.context(&self.func.layout), - ) - { - // Splitting at `pred_inst` wouldn't resolve the interference, so we need to - // start over. - return Some(a); - } + // 1. It is defined in a dominating EBB and live-in to `ebb`. + // 2. If is itself a parameter value for `ebb`. This case should already have been + // eliminated by `isolate_conflicting_params()`. + assert!( + lr.def() != ebb.into(), + "{} parameter {} was missed by isolate_conflicting_params()", + ebb, + arg + ); - // The local conflict could likely be avoided by splitting at this predecessor, so - // try that. This split is not necessarily required, but it allows us to make - // progress. - let new_val = self.split_pred(pred_inst, pred_ebb, argnum, pred_val); + // The only other possibility is that `arg` is live-in to `ebb`. + lr.is_livein(ebb, ctx) + }; - // If this tiny new live range can't be merged, there is something in the already - // merged values that is fundamentally incompatible with `pred_inst`, and we need - // to start over after removing that value. - // TODO: It is unfortunate that we discover this *after* splitting. It would have - // been better if we could detect and isolate `merged` before splitting. - if let Err((merged, _)) = self.add_class(new_val) { - dbg!("Splitting didn't help: {} interferes", merged); - // We need to start over, isolating the bad value. - return Some(merged); - } + if interference { + let new_arg = self.isolate_arg(pred_ebb, pred_inst, argnum, arg); + self.virtregs.union(param, new_arg); + } else { + self.virtregs.union(param, arg); } } - - None } - /// Try merging the congruence class for `value` into `self.values`. - /// - /// Leave `self.values` unchanged on failure. - fn add_class(&mut self, value: Value) -> Result<(), (Value, Value)> { - self.forest.try_merge( - self.values, - self.virtregs.congruence_class(&value), - self.func, - self.domtree, - self.preorder, - self.liveness, - )?; - self.forest.swap(&mut self.values); - Ok(()) + // Isolate EBB parameter value `param` on `ebb`. + // + // When `param=v10`: + // + // ebb1(v10: i32): + // foo + // + // becomes: + // + // ebb1(v11: i32): + // v10 = copy v11 + // foo + // + // This function inserts the copy and updates the live ranges of the old and new parameter + // values. Returns the new parameter value. + fn isolate_param(&mut self, ebb: Ebb, param: Value) -> Value { + debug_assert_eq!( + self.func.dfg.value_def(param).pp(), + ExpandedProgramPoint::Ebb(ebb) + ); + let ty = self.func.dfg.value_type(param); + let new_val = self.func.dfg.replace_ebb_param(param, ty); + + // Insert a copy instruction at the top of `ebb`. + let mut pos = EncCursor::new(self.func, self.isa).at_first_inst(ebb); + pos.ins().with_result(param).copy(new_val); + let inst = pos.built_inst(); + self.liveness.move_def_locally(param, inst); + + dbg!( + "-> inserted {}, following {}({}: {})", + pos.display_inst(inst), + ebb, + new_val, + ty + ); + + // Create a live range for the new value. + // TODO: Should we handle ghost values? + let affinity = Affinity::new( + &self.encinfo + .operand_constraints(pos.func.encodings[inst]) + .expect("Bad copy encoding") + .outs + [0], + ); + self.liveness.create_dead(new_val, ebb, affinity); + self.liveness.extend_locally( + new_val, + ebb, + inst, + &pos.func.layout, + ); + + new_val } - /// Split the congruence class for the `argnum` argument to `pred_inst` by inserting a copy. - fn split_pred( + // Isolate the EBB argument `pred_val` from the predecessor `(pred_ebb, pred_inst)`. + // + // It is assumed that `pred_inst` is a branch instruction in `pred_ebb` whose `argnum`'th EBB + // argument is `pred_val`. Since the argument value interferes with the corresponding EBB + // parameter at the destination, a copy is used instead: + // + // brnz v1, ebb2(v10) + // + // Becomes: + // + // v11 = copy v10 + // brnz v1, ebb2(v11) + // + // This way the interference with the EBB parameter is avoided. + // + // A live range for the new value is created while the live range for `pred_val` is left + // unaltered. + // + // The new argument value is returned. + fn isolate_arg( &mut self, - pred_inst: Inst, pred_ebb: Ebb, + pred_inst: Inst, argnum: usize, pred_val: Value, ) -> Value { @@ -554,14 +336,8 @@ impl<'a> Context<'a> { let copy = pos.ins().copy(pred_val); let inst = pos.built_inst(); - dbg!( - "Inserted {}, before {}: {}", - pos.display_inst(inst), - pred_ebb, - pos.display_inst(pred_inst) - ); - // Create a live range for the new value. + // TODO: Handle affinity for ghost values. let affinity = Affinity::new( &self.encinfo .operand_constraints(pos.func.encodings[inst]) @@ -578,46 +354,337 @@ impl<'a> Context<'a> { ); pos.func.dfg.inst_variable_args_mut(pred_inst)[argnum] = copy; - self.split_values.push(copy); + + dbg!( + "-> inserted {}, before {}: {}", + pos.display_inst(inst), + pred_ebb, + pos.display_inst(pred_inst) + ); + copy } - /// Split the congruence class for the successor EBB value itself. - fn split_succ(&mut self, ebb: Ebb, succ_val: Value) -> Value { - let ty = self.func.dfg.value_type(succ_val); - let new_val = self.func.dfg.replace_ebb_param(succ_val, ty); - - // Insert a copy instruction at the top of ebb. - let mut pos = EncCursor::new(self.func, self.isa).at_first_inst(ebb); - pos.ins().with_result(succ_val).copy(new_val); - let inst = pos.built_inst(); - self.liveness.move_def_locally(succ_val, inst); - - dbg!( - "Inserted {}, following {}({}: {})", - pos.display_inst(inst), - ebb, - new_val, - ty - ); - - // Create a live range for the new value. - let affinity = Affinity::new( - &self.encinfo - .operand_constraints(pos.func.encodings[inst]) - .expect("Bad copy encoding") - .outs - [0], - ); - self.liveness.create_dead(new_val, ebb, affinity); - self.liveness.extend_locally( - new_val, - ebb, - inst, - &pos.func.layout, - ); - - self.split_values.push(new_val); - new_val + /// Finish the union-find part of the coalescing algorithm. + /// /// + /// This builds the initial set of virtual registers as the transitive/reflexive/symmetric + /// closure of the relation formed by EBB parameter-argument pairs found by `union_find_ebb()`. + fn finish_union_find(&mut self) { + self.virtregs.finish_union_find(None); + dbg!("After union-find phase:{}", self.virtregs); + } +} + +/// Phase 2: Dominator forests. +/// +/// The main entry point is `process_vregs()`. +impl<'a> Context<'a> { + /// Check al virtual registers for interference and fix conflicts. + pub fn process_vregs(&mut self) { + for vreg in self.virtregs.all_virtregs() { + self.process_vreg(vreg); + while let Some(vr) = self.worklist.pop() { + self.process_vreg(vr); + } + } + } + + // Check `vreg` for interferences and fix conflicts. + fn process_vreg(&mut self, vreg: VirtReg) { + if self.analyze_vreg(vreg) { + self.synthesize_vreg(vreg); + } + } + + // Check `vreg` for interferences and choose values to isolate. + // + // We use a Budimlic dominator forest to check for interferences between the values in `vreg` + // and identify values that should be isolated. + // + // Returns true if `vreg` has conflicts that need to be fixed. Additionally leaves state in + // member variables: + // + // - `self.params` contains all the EBB parameter values that were present in the virtual + // register. + // - `self.forest` contains the set of values that should be isolated from the virtual register. + fn analyze_vreg(&mut self, vreg: VirtReg) -> bool { + // Order the values according to the dominator pre-order of their definition. + let dfg = &self.func.dfg; + let layout = &self.func.layout; + let preorder = self.preorder; + let values = self.virtregs.sort_values(vreg, |a, b| { + let da = dfg.value_def(a); + let db = dfg.value_def(b); + preorder.pre_cmp(da, db, layout).then( + da.num().cmp(&db.num()), + ) + }); + dbg!("Analyzing {} = {}", vreg, DisplayList(values)); + + // Now push the values in order to the dominator forest. This gives us the closest + // dominating value def for each of the values. + self.params.clear(); + self.forest.clear(); + for &value in values { + let node = Node::new(value, self.func); + + // Remember the parameter values in case we need to re-synthesize virtual registers. + if let ExpandedProgramPoint::Ebb(_) = node.def { + self.params.push(value); + } + + // Push this value and get the nearest dominating def back. + let parent = match self.forest.push_value( + node, + self.func, + self.domtree, + self.preorder, + ) { + None => continue, + Some(p) => p, + }; + + // Check for interference between `parent` and `value`. Since `parent` dominates + // `value`, we only have to check if it overlaps the definition. + let ctx = self.liveness.context(&self.func.layout); + if !self.liveness[parent].overlaps_def(node.def, node.ebb, ctx) { + // No interference, both values can stay in the virtual register. + continue; + } + + // The two values are interfering, so they can't both be in the same virtual register. + // We need to pick one to isolate. It's hard to pick a heuristic that only looks at two + // values since an optimal solution is a global problem involving all the values in the + // virtual register. + // + // We choose to always isolate the dominating parent value for two reasons: + // + // 1. We avoid the case of a parent value with a very long live range pushing many + // following values out of the virtual register. + // + // 2. In the case of a value that is live across a branch to the definition of a + // parameter in the virtual register, our splitting method in `synthesize_vreg` + // doesn't actually resolve the interference unless we're trying to isolate the + // first value. This heuristic will at least pick the first value on a second + // attempt. This is actually a correctness issue - we could loop infinitely + // otherwise. See the `infinite-interference.cton` test case. + dbg!("-> isolating {} which overlaps def of {}", parent, value); + self.forest.drop_value(parent); + } + + let dropped = self.forest.prepare_dropped(); + assert!(dropped < values.len()); + dropped != 0 + } + + /// Destroy and rebuild `vreg`. + /// + /// Use `self.params` to rebuild the virtual register, but this time making sure that dropped + /// values in `self.forest` are isolated from non-dropped values. This may cause multiple new + /// virtual registers to be formed. + /// + /// All new virtual registers are appended to `self.worklist`. + fn synthesize_vreg(&mut self, vreg: VirtReg) { + dbg!("Synthesizing {} from {}", vreg, DisplayList(self.params)); + self.virtregs.remove(vreg); + + while let Some(param) = self.params.pop() { + let param_dropped = self.forest.is_dropped(param); + let (ebb, argnum) = match self.func.dfg.value_def(param) { + ir::ValueDef::Param(e, n) => (e, n), + ir::ValueDef::Result(_, _) => panic!("{} expected to be EBB parameter"), + }; + + // Union the EBB parameter with corresponding arguments on the predecessor branches, + // but make sure to isolate dropped values. + // + // Compare `union_pred_args()` which runs during phase 1. We don't need to check for + // special cases here since they have already been eliminated during phase 1. We + // already know that: + // + // 1. `arg` is not live-in to `ebb`. + // 2. `arg` is not a function argument on the stack. + for (pred_ebb, pred_inst) in self.cfg.pred_iter(ebb) { + let arg = self.func.dfg.inst_variable_args(pred_inst)[argnum]; + let arg_dropped = self.forest.is_dropped(arg); + + // We don't want to union dropped values with each other because we can't ensure + // that we are actually making progress -- the new virtual register of dropped + // values may have its own interferences and so on. + // + // TODO: Maintain a secondary dominator forest to keep track of dropped values that + // would be allowed to be unioned together. + if param_dropped || arg_dropped { + dbg!(" - {}#{}: {} isolated from {}", ebb, argnum, param, arg); + let new_arg = self.isolate_arg(pred_ebb, pred_inst, argnum, arg); + self.virtregs.union(param, new_arg); + } else { + self.virtregs.union(param, arg); + } + } + } + + // TODO: Get back the new vregs so they can be re-checked. + let old_len = self.worklist.len(); + self.virtregs.finish_union_find(Some(self.worklist)); + dbg!("-> new vregs {}", DisplayList(&self.worklist[old_len..])); + } +} + +/// Dominator forest. +/// +/// This is a utility type used for detecting interference in virtual registers, where each virtual +/// register is a list of values ordered according to the dominator tree pre-order. +/// +/// The idea of a dominator forest was introduced on the Budimlic paper and the linear stack +/// representation in the Boissinot paper. Our version of the linear stack is slightly modified +/// because we have a pre-order of the dominator tree at the EBB granularity, not basic block +/// granularity. +/// +/// Values are pushed in dominator tree pre-order of their definitions, and for each value pushed, +/// `push_value` will return the nearest previously pushed value that dominates the definition. +#[allow(dead_code)] +struct DomForest { + // Stack representing the rightmost edge of the dominator forest so far, ending in the last + // element of `values`. + // + // At all times, the EBB of each element in the stack dominates the EBB of the next one, and + // all elements dominating the end of `values` are on the stack. + stack: Vec, + + // The index into `stack` of the last dominating node returned by `push_value`. + last_dom: Option, + + // List of values that have been dropped from the forest because they were interfering with + // another member. + // + // This list is initially just appended to, then it sorted for quick member checks with + // `is_dropped()`. + dropped: Vec, +} + +/// A node in the dominator forest. +#[derive(Clone, Copy, Debug)] +#[allow(dead_code)] +struct Node { + value: Value, + /// The program point where `value` is defined. + def: ExpandedProgramPoint, + /// EBB containing `def`. + ebb: Ebb, +} + +impl Node { + /// Create a node for `value`. + pub fn new(value: Value, func: &Function) -> Node { + let def = func.dfg.value_def(value).pp(); + let ebb = func.layout.pp_ebb(def); + Node { value, def, ebb } + } +} + +impl fmt::Display for Node { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{}@{}", self.value, self.ebb) + } +} + +impl DomForest { + /// Create a new empty dominator forest. + pub fn new() -> Self { + Self { + stack: Vec::new(), + last_dom: None, + dropped: Vec::new(), + } + } + + /// Clear all data structures in this dominator forest. + pub fn clear(&mut self) { + self.stack.clear(); + self.last_dom = None; + self.dropped.clear(); + } + + /// Add a single value to the forest. + /// + /// Update the stack so its dominance invariants are preserved. Detect a parent node on the + /// stack which is the closest one dominating the new node and return it. + fn push_value( + &mut self, + node: Node, + func: &Function, + domtree: &DominatorTree, + preorder: &DominatorTreePreorder, + ) -> Option { + // The stack contains the current sequence of dominating defs. Pop elements until we + // find one whose EBB dominates `node.ebb`. + while let Some(top) = self.stack.pop() { + if preorder.dominates(top.ebb, node.ebb) { + // This is the right insertion spot for `node`. + self.stack.push(top); + self.stack.push(node); + + // We know here that `top.ebb` dominates `node.ebb`, and thus `node.def`. This does + // not necessarily mean that `top.def` dominates `node.def`, though. The `top.def` + // program point may be below the last branch in `top.ebb` that dominates + // `node.def`. + debug_assert!(domtree.dominates(top.ebb, node.def, &func.layout)); + + // We do know, though, that if there is a nearest value dominating `node.def`, it + // will be on the stack. We just need to find the last stack entry that actually + // dominates. + // + // TODO: This search could be more efficient if we had access to + // `domtree.last_dominator()`. Each call to `dominates()` here ends up walking up + // the dominator tree starting from `node.ebb`. + self.last_dom = self.stack[0..self.stack.len() - 1].iter().rposition(|n| { + domtree.dominates(n.def, node.def, &func.layout) + }); + + // If there is a dominating parent value, return it for interference checking. + return self.last_dom.map(|pos| self.stack[pos].value); + } + } + + // No dominators, start a new tree in the forest. + self.stack.push(node); + None + } + + /// Drop `value` from the forest and add it to the `dropped` list. + /// + /// The value must be either the last value passed to `push_value` or the dominating value + /// returned from the call. + pub fn drop_value(&mut self, value: Value) { + self.dropped.push(value); + + // Are they dropping the last value pushed? + if self.stack.last().expect("Nothing pushed").value == value { + self.stack.pop(); + } else { + // Otherwise, they must be dropping the last dominator. + let pos = self.last_dom.take().expect("No last dominator"); + let node = self.stack.remove(pos); + assert_eq!(node.value, value, "Inconsistent value to drop_value"); + } + } + + /// Prepare the set of dropped values to be queried with `is_dropped()`. + /// + /// Returns the number of dropped values. + pub fn prepare_dropped(&mut self) -> usize { + self.stack.clear(); + if !self.dropped.is_empty() { + self.dropped.sort_unstable(); + dbg!("-> dropped {}", DisplayList(&self.dropped)); + } + self.dropped.len() + } + + /// Check if `value` was dropped. + pub fn is_dropped(&self, value: Value) -> bool { + debug_assert!(self.stack.is_empty(), "Call prepare_dropped first"); + self.dropped.binary_search(&value).is_ok() } } From 1e2b7de141221a702e14f08808b5eec5c297adb6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 16 Jan 2018 12:34:32 -0800 Subject: [PATCH 1464/3084] Remove dead code. --- lib/cretonne/src/regalloc/virtregs.rs | 44 --------------------------- 1 file changed, 44 deletions(-) diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index ce38dc3687..4fe724cf74 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -149,50 +149,6 @@ impl VirtRegs { self.vregs.push(Default::default()) }) } - - /// Unify `values` into a single virtual register. - /// - /// The values in the slice can be singletons or they can belong to a virtual register already. - /// If a value belongs to a virtual register, all of the values in that register must be - /// present. - /// - /// The values are assumed to already be in topological order. - pub fn unify(&mut self, values: &[Value]) -> VirtReg { - // Start by clearing all virtual registers involved. - // Pick a virtual register to reuse (the smallest number) or allocate a new one. - let mut singletons = 0; - let mut cleared = 0; - let vreg = values - .iter() - .filter_map(|&v| { - let vr = self.get(v); - match vr { - None => singletons += 1, - Some(vr) => { - if !self.vregs[vr].is_empty() { - cleared += self.vregs[vr].len(&self.pool); - self.vregs[vr].clear(&mut self.pool); - } - } - } - vr - }) - .min() - .unwrap_or_else(|| self.vregs.push(Default::default())); - - assert_eq!( - values.len(), - singletons + cleared, - "Can't unify partial virtual registers" - ); - - self.vregs[vreg].extend(values.iter().cloned(), &mut self.pool); - for &v in values { - self.value_vregs[v] = vreg.into(); - } - - vreg - } } impl fmt::Display for VirtRegs { From eb85aa833c19aeafa749fd72d724ad2f2289b955 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Tue, 16 Jan 2018 21:05:53 -0700 Subject: [PATCH 1465/3084] Illegalize rbp/r13 for zero-offset loads on Intel x64 (#225) * Switch RegClass to a bitmap implementation. * Add special RegClass to remove r13 from 'ld' recipe. * Use MASK_LEN constant instead of magic number. * Enforce that RegClass slicing is only valid on contiguous classes. * Use Optional[int] for RegClass optional bitmask parameter. * Add comment explaining use of Intel ISA's GPR_NORIP register class. --- lib/cretonne/meta/cdsl/registers.py | 102 +++++++++++++++-------- lib/cretonne/meta/gen_registers.py | 2 +- lib/cretonne/meta/isa/intel/recipes.py | 6 +- lib/cretonne/meta/isa/intel/registers.py | 3 + 4 files changed, 77 insertions(+), 36 deletions(-) diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/cretonne/meta/cdsl/registers.py index 22e954ee8a..6f57e380d9 100644 --- a/lib/cretonne/meta/cdsl/registers.py +++ b/lib/cretonne/meta/cdsl/registers.py @@ -26,12 +26,12 @@ from . import is_power_of_two, next_power_of_two try: - from typing import Sequence, Tuple, List, Dict, Any, TYPE_CHECKING # noqa + from typing import Sequence, Tuple, List, Dict, Any, Optional, TYPE_CHECKING # noqa if TYPE_CHECKING: from .isa import TargetISA # noqa # A tuple uniquely identifying a register class inside a register bank. - # (count, width, start) - RCTup = Tuple[int, int, int] + # (width, bitmask) + RCTup = Tuple[int, int] except ImportError: pass @@ -195,24 +195,29 @@ class RegClass(object): :param start: The first unit to allocate, relative to `bank.first.unit`. """ - def __init__(self, bank, count=None, width=1, start=0): - # type: (RegBank, int, int, int) -> None + def __init__(self, bank, count=0, width=1, start=0, bitmask=None): + # type: (RegBank, int, int, int, Optional[int]) -> None self.name = None # type: str self.index = None # type: int self.bank = bank - self.start = start self.width = width + self.bitmask = 0 # This is computed later in `finish_regclasses()`. self.subclasses = list() # type: List[RegClass] self.toprc = None # type: RegClass assert width > 0 - assert start >= 0 and start < bank.units - if count is None: - count = bank.units // width - self.count = count + if bitmask: + self.bitmask = bitmask + else: + assert start >= 0 and start < bank.units + if count == 0: + count = bank.units // width + for a in range(count): + u = start + a * self.width + self.bitmask |= 1 << u bank.classes.append(self) @@ -238,7 +243,7 @@ class RegClass(object): The tuple can be used as a dictionary key to ensure that there are no duplicate register classes. """ - return (self.count, self.width, self.start) + return (self.width, self.bitmask) def intersect(self, other): # type: (RegClass) -> RCTup @@ -249,17 +254,11 @@ class RegClass(object): """ if self.width != other.width: return None - s_end = self.start + self.count * self.width - o_end = other.start + other.count * other.width - if self.start >= o_end or other.start >= s_end: + intersection = self.bitmask & other.bitmask + if intersection == 0: return None - # We have an overlap. - start = max(self.start, other.start) - end = min(s_end, o_end) - count = (end - start) // self.width - assert count > 0 - return (count, self.width, start) + return (self.width, intersection) def __getitem__(self, sliced): # type: (slice) -> RegClass @@ -271,14 +270,57 @@ class RegClass(object): assert isinstance(sliced, slice), "RegClass slicing can't be 1 reg" # We could add strided sub-classes if needed. assert sliced.step is None, 'Subclass striding not supported' + # Can't slice a non-contiguous class + assert self.is_contiguous(), 'Cannot slice non-contiguous RegClass' w = self.width - s = self.start + sliced.start * w + s = self.start() + sliced.start * w c = sliced.stop - sliced.start assert c > 1, "Can't have single-register classes" return RegClass(self.bank, count=c, width=w, start=s) + def without(self, *registers): + # type: (*Register) -> RegClass + """ + Create a sub-class of a register class excluding a specific set of + registers. + + For example: GPR.without(GPR.r9) + """ + bm = self.bitmask + w = self.width + fmask = (1 << self.width) - 1 + for reg in registers: + bm &= ~(fmask << (reg.unit * w)) + + return RegClass(self.bank, bitmask=bm) + + def is_contiguous(self): + # type: () -> bool + """ + Returns boolean indicating whether a register class is a contiguous set + of register units. + """ + x = self.bitmask | (self.bitmask-1) + return self.bitmask != 0 and ((x+1) & x) == 0 + + def start(self): + # type: () -> int + """ + Returns the first valid register unit in this class. + """ + start = 0 + bm = self.bitmask + fmask = (1 << self.width) - 1 + while True: + if bm & fmask > 0: + break + start += 1 + bm >>= self.width + + return start + def __getattr__(self, attr): # type: (str) -> Register """ @@ -299,19 +341,13 @@ class RegClass(object): Return as a list of 32-bit integers. """ - mask = [0] * MASK_LEN + out_mask = [] + mask32 = (1 << 32) - 1 + bitmask = self.bitmask << self.bank.first_unit + for i in range(MASK_LEN): + out_mask.append((bitmask >> (i * 32)) & mask32) - start = self.bank.first_unit + self.start - for a in range(self.count): - u = start + a * self.width - b = u % 32 - # We need fancier masking code if a register can straddle mask - # words. This will only happen with widths that are not powers of - # two. - assert b + self.width <= 32, 'Register straddles words' - mask[u // 32] |= 1 << b - - return mask + return out_mask def subclass_mask(self): # type: () -> int diff --git a/lib/cretonne/meta/gen_registers.py b/lib/cretonne/meta/gen_registers.py index 5917312d96..8e12fbf34f 100644 --- a/lib/cretonne/meta/gen_registers.py +++ b/lib/cretonne/meta/gen_registers.py @@ -59,7 +59,7 @@ def gen_regclass(rc, fmt): fmt.format('width: {},', rc.width) fmt.format('bank: {},', rc.bank.index) fmt.format('toprc: {},', rc.toprc.index) - fmt.format('first: {},', rc.bank.first_unit + rc.start) + fmt.format('first: {},', rc.bank.first_unit + rc.start()) fmt.format('subclasses: 0x{:x},', rc.subclass_mask()) mask = ', '.join('0x{:08x}'.format(x) for x in rc.mask()) fmt.format('mask: [{}],', mask) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index fb7b993c80..e9c896123b 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -11,7 +11,8 @@ from base.formats import IntCompare, FloatCompare, IntCond, FloatCond from base.formats import Jump, Branch, BranchInt, BranchFloat from base.formats import Ternary, FuncAddr, UnaryGlobalVar from base.formats import RegMove, RegSpill, RegFill, CopySpecial -from .registers import GPR, ABCD, FPR, GPR8, FPR8, FLAG, StackGPR32, StackFPR32 +from .registers import GPR, ABCD, FPR, GPR_NORIP, GPR8, FPR8, GPR8_NORIP +from .registers import FLAG, StackGPR32, StackFPR32 from .defs import supported_floatccs from .settings import use_sse41 @@ -103,6 +104,7 @@ def replace_put_op(emit, prefix): # Register class mapping for no-REX instructions. NOREX_MAP = { GPR: GPR8, + GPR_NORIP: GPR8_NORIP, FPR: FPR8 } @@ -766,7 +768,7 @@ frsp32 = TailRecipe( # XX /r load with no offset. ld = TailRecipe( - 'ld', Load, size=1, ins=(GPR), outs=(GPR), + 'ld', Load, size=1, ins=(GPR_NORIP), outs=(GPR), instp=IsEqual(Load.offset, 0), clobbers_flags=False, emit=''' diff --git a/lib/cretonne/meta/isa/intel/registers.py b/lib/cretonne/meta/isa/intel/registers.py index cbb6701660..9411b23245 100644 --- a/lib/cretonne/meta/isa/intel/registers.py +++ b/lib/cretonne/meta/isa/intel/registers.py @@ -47,6 +47,9 @@ FlagRegs = RegBank( GPR = RegClass(IntRegs) GPR8 = GPR[0:8] +# In certain instructions, RBP and R13 are interpreted as RIP-relative. +GPR_NORIP = GPR.without(GPR.rbp, GPR.r13) +GPR8_NORIP = GPR8.without(GPR.rbp) ABCD = GPR[0:4] FPR = RegClass(FloatRegs) FPR8 = FPR[0:8] From 457e1619745fe142763f25d159262a71c04bc373 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 17 Jan 2018 11:24:45 -0800 Subject: [PATCH 1466/3084] Add comments explaining why the rustfmt version is still at 0.9.0. --- check-rustfmt.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/check-rustfmt.sh b/check-rustfmt.sh index 0439f35ff7..483a45396a 100755 --- a/check-rustfmt.sh +++ b/check-rustfmt.sh @@ -14,7 +14,11 @@ # # With the --install option, also tries to install the right version. -# This version should always be bumped to the newest version available. +# This version should always be bumped to the newest version available that +# works with stable Rust. +# ... but not 0.10.0, since it's the same as 0.9.0 except for a deprecation +# error (and it requires --force to disable the error and enable normal +# operation, however that doesn't appear to be possible through "cargo fmt"). VERS="0.9.0" if cargo install --list | grep -q "^rustfmt v$VERS"; then From 5463fde203e68ee33aa7509a541c59be897b45ed Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 17 Jan 2018 09:52:31 -0800 Subject: [PATCH 1467/3084] Don't print unused pressure classes. --- lib/cretonne/src/regalloc/pressure.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index 0ee39b07d5..c71037efdd 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -258,7 +258,7 @@ impl fmt::Display for Pressure { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "Pressure[")?; for rc in &self.toprc { - if rc.limit > 0 { + if rc.limit > 0 && rc.limit < !0 { write!(f, " {}+{}/{}", rc.base_count, rc.transient_count, rc.limit)?; } } From 13af22b46bf6e9c912a53182ef3ae6f0b4abdf3a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 17 Jan 2018 12:39:10 -0800 Subject: [PATCH 1468/3084] Track register pressure for dead EBB parameters. The spiller wasn't tracking register pressure correctly for dead EBB parameters in visit_ebb_header(). Make sure we free any dead EBB parameters. Fixes #223 --- .../filetests/regalloc/spill-noregs.cton | 176 ++++++++++++++++++ lib/cretonne/src/regalloc/coloring.rs | 2 +- .../src/regalloc/live_value_tracker.rs | 8 +- lib/cretonne/src/regalloc/reload.rs | 2 +- lib/cretonne/src/regalloc/spilling.rs | 33 ++-- 5 files changed, 204 insertions(+), 17 deletions(-) create mode 100644 cranelift/filetests/regalloc/spill-noregs.cton diff --git a/cranelift/filetests/regalloc/spill-noregs.cton b/cranelift/filetests/regalloc/spill-noregs.cton new file mode 100644 index 0000000000..de4fa618bf --- /dev/null +++ b/cranelift/filetests/regalloc/spill-noregs.cton @@ -0,0 +1,176 @@ +test regalloc +set is_64bit +isa intel + +; Test case found by the Binaryen fuzzer. +; +; The spiller panics with a +; 'Ran out of GPR registers when inserting copy before v68 = icmp.i32 eq v66, v67', +; lib/cretonne/src/regalloc/spilling.rs:425:28 message. +; +; The process_reg_uses() function is trying to insert a copy before the icmp instruction in ebb4 +; and runs out of registers to spill. Note that ebb7 has a lot of dead parameter values. +; +; The spiller was not releasing register pressure for dead EBB parameters. + +function %pr223(i32 [%rdi], i64 vmctx [%rsi]) -> i64 [%rax] native { +ebb0(v0: i32, v1: i64): + v2 = iconst.i32 0 + v3 = iconst.i64 0 + v4 = iconst.i32 0xffff_ffff_bb3f_4a2c + brz v4, ebb5 + jump ebb1 + +ebb1: + v5 = iconst.i32 0 + v6 = copy.i64 v3 + v7 = copy.i64 v3 + v8 = copy.i64 v3 + v9 = copy.i64 v3 + v10 = copy.i64 v3 + v11 = copy.i64 v3 + v12 = copy.i64 v3 + v13 = copy.i64 v3 + v14 = copy.i64 v3 + v15 = copy.i64 v3 + v16 = copy.i64 v3 + brnz v5, ebb4(v2, v3, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16) + jump ebb2 + +ebb2: + v17 = iconst.i32 0 + v18 = copy.i64 v3 + v19 = copy.i64 v3 + v20 = copy.i64 v3 + v21 = copy.i64 v3 + v22 = copy.i64 v3 + v23 = copy.i64 v3 + v24 = copy.i64 v3 + v25 = copy.i64 v3 + v26 = copy.i64 v3 + v27 = copy.i64 v3 + v28 = copy.i64 v3 + brnz v17, ebb4(v2, v3, v18, v19, v20, v21, v22, v23, v24, v25, v26, v27, v28) + jump ebb3 + +ebb3: + jump ebb1 + +ebb4(v29: i32, v30: i64, v31: i64, v32: i64, v33: i64, v34: i64, v35: i64, v36: i64, v37: i64, v38: i64, v39: i64, v40: i64, v41: i64): + jump ebb7(v29, v30, v31, v32, v33, v34, v35, v36, v37, v38, v39, v40, v41) + +ebb5: + jump ebb6 + +ebb6: + v42 = copy.i64 v3 + v43 = copy.i64 v3 + v44 = copy.i64 v3 + v45 = copy.i64 v3 + v46 = copy.i64 v3 + v47 = copy.i64 v3 + v48 = copy.i64 v3 + v49 = copy.i64 v3 + v50 = copy.i64 v3 + v51 = copy.i64 v3 + v52 = copy.i64 v3 + jump ebb7(v2, v3, v42, v43, v44, v45, v46, v47, v48, v49, v50, v51, v52) + +ebb7(v53: i32, v54: i64, v55: i64, v56: i64, v57: i64, v58: i64, v59: i64, v60: i64, v61: i64, v62: i64, v63: i64, v64: i64, v65: i64): + v66 = iconst.i32 0 + v67 = iconst.i32 0 + v68 = icmp eq v66, v67 + v69 = bint.i32 v68 + jump ebb8 + +ebb8: + jump ebb9 + +ebb9: + v70 = iconst.i32 0xffff_ffff_ffff_912f + brz v70, ebb10 + jump ebb35 + +ebb10: + v71 = iconst.i32 0 + brz v71, ebb11 + jump ebb27 + +ebb11: + jump ebb12 + +ebb12: + jump ebb13 + +ebb13: + jump ebb14 + +ebb14: + jump ebb15 + +ebb15: + jump ebb16 + +ebb16: + jump ebb17 + +ebb17: + jump ebb18 + +ebb18: + jump ebb19 + +ebb19: + jump ebb20 + +ebb20: + jump ebb21 + +ebb21: + jump ebb22 + +ebb22: + jump ebb23 + +ebb23: + jump ebb24 + +ebb24: + jump ebb25 + +ebb25: + jump ebb26 + +ebb26: + jump ebb27 + +ebb27: + jump ebb28 + +ebb28: + jump ebb29 + +ebb29: + jump ebb30 + +ebb30: + jump ebb31 + +ebb31: + jump ebb32 + +ebb32: + jump ebb33 + +ebb33: + jump ebb34 + +ebb34: + jump ebb35 + +ebb35: + jump ebb36 + +ebb36: + trap user0 +} diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 2f52dfaabc..9d82c0b21a 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -158,7 +158,7 @@ impl<'a> Context<'a> { fn visit_ebb(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) { dbg!("Coloring {}:", ebb); let mut regs = self.visit_ebb_header(ebb, tracker); - tracker.drop_dead_args(); + tracker.drop_dead_params(); self.divert.clear(); // Now go through the instructions in `ebb` and color the values they define. diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index e6f53151e2..d0810a5cf7 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -160,9 +160,9 @@ impl LiveValueTracker { /// been visited first. /// /// Returns `(liveins, args)` as a pair of slices. The first slice is the set of live-in values - /// from the immediate dominator. The second slice is the set of `ebb` arguments that are live. + /// from the immediate dominator. The second slice is the set of `ebb` parameters. /// - /// Dead arguments with no uses are included in `args`. Call `drop_dead_args()` to remove them. + /// Dead parameters with no uses are included in `args`. Call `drop_dead_args()` to remove them. pub fn ebb_top( &mut self, ebb: Ebb, @@ -314,8 +314,8 @@ impl LiveValueTracker { /// Drop any values that are marked as `is_dead`. /// - /// Use this after calling `ebb_top` to clean out dead EBB arguments. - pub fn drop_dead_args(&mut self) { + /// Use this after calling `ebb_top` to clean out dead EBB parameters. + pub fn drop_dead_params(&mut self) { self.live.remove_dead_values(); } diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 8ed95628fd..2cb37d7d6a 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -120,7 +120,7 @@ impl<'a> Context<'a> { fn visit_ebb(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) { dbg!("Reloading {}:", ebb); self.visit_ebb_header(ebb, tracker); - tracker.drop_dead_args(); + tracker.drop_dead_params(); // visit_ebb_header() places us at the first interesting instruction in the EBB. while let Some(inst) = self.cur.current_inst() { diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 4c5b8b9ec7..84ef76fdff 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -120,7 +120,8 @@ impl<'a> Context<'a> { dbg!("Spilling {}:", ebb); self.cur.goto_top(ebb); self.visit_ebb_header(ebb, tracker); - tracker.drop_dead_args(); + tracker.drop_dead_params(); + self.process_spills(tracker); while let Some(inst) = self.cur.next_inst() { if let Some(constraints) = @@ -163,8 +164,22 @@ impl<'a> Context<'a> { } } + // Free all dead registers in `regs` from the pressure set. + fn free_dead_regs(&mut self, regs: &[LiveValue]) { + for lv in regs { + if lv.is_dead { + if let Affinity::Reg(rci) = lv.affinity { + if !self.spills.contains(&lv.value) { + let rc = self.reginfo.rc(rci); + self.pressure.free(rc); + } + } + } + } + } + fn visit_ebb_header(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) { - let (liveins, args) = tracker.ebb_top( + let (liveins, params) = tracker.ebb_top( ebb, &self.cur.func.dfg, self.liveness, @@ -177,22 +192,17 @@ impl<'a> Context<'a> { self.pressure.reset(); self.take_live_regs(liveins); - // An EBB can have an arbitrary (up to 2^16...) number of EBB arguments, so they are not + // An EBB can have an arbitrary (up to 2^16...) number of parameters, so they are not // guaranteed to fit in registers. - for lv in args { + for lv in params { if let Affinity::Reg(rci) = lv.affinity { let rc = self.reginfo.rc(rci); 'try_take: while let Err(mask) = self.pressure.take_transient(rc) { - dbg!( - "Need {} reg for EBB argument {} from {} live-ins", - rc, - lv.value, - liveins.len() - ); + dbg!("Need {} reg for EBB param {}", rc, lv.value); match self.spill_candidate(mask, liveins) { Some(cand) => { dbg!( - "Spilling live-in {} to make room for {} EBB argument {}", + "Spilling live-in {} to make room for {} EBB param {}", cand, rc, lv.value @@ -217,6 +227,7 @@ impl<'a> Context<'a> { // The transient pressure counts for the EBB arguments are accurate. Just preserve them. self.pressure.preserve_transient(); + self.free_dead_regs(params); } fn visit_inst( From 0a6500c99a7a14223ac18a288ce93885dee6addb Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 17 Jan 2018 14:58:05 -0800 Subject: [PATCH 1469/3084] Avoid making solver variables for fixed input constraints. When the coloring pass sees an instruction with a fixed input register constraint that is already satisfied, make sure to tell the solver about it anyway. There are situations where the solver wants to convert a value to a solver variable, and we can't allow that if the same value is also used for a fixed register operand. Fixes #221. --- .../filetests/regalloc/multi-constraints.cton | 29 +++++++++++++++++++ lib/cretonne/src/regalloc/coloring.rs | 17 ++++++----- 2 files changed, 38 insertions(+), 8 deletions(-) create mode 100644 cranelift/filetests/regalloc/multi-constraints.cton diff --git a/cranelift/filetests/regalloc/multi-constraints.cton b/cranelift/filetests/regalloc/multi-constraints.cton new file mode 100644 index 0000000000..80dc8f0658 --- /dev/null +++ b/cranelift/filetests/regalloc/multi-constraints.cton @@ -0,0 +1,29 @@ +test regalloc +set is_64bit +isa intel haswell + +; Test combinations of constraints. +; +; The Intel ushr instruction requires its second operand to be passed in %rcx and its output is +; tied to the first input operand. +; +; If we pass the same value to both operands, both constraints must be satisfied. + +; Found by the Binaryen fuzzer in PR221. +; +; Conditions triggering the problem: +; +; - The same value used for a tied operand and a fixed operand. +; - The common value is already in %rcx. +; - The tied output value is live outside the EBB. +; +; Under these conditions, Solver::add_tied_input() would create a variable for the tied input +; without considering the fixed constraint. +function %pr221(i64 [%rdi], i64 [%rsi], i64 [%rdx], i64 [%rcx]) -> i64 [%rax] { +ebb0(v0: i64, v1: i64, v2: i64, v3: i64): + v4 = ushr v3, v3 + jump ebb1 + +ebb1: + return v4 +} diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 9d82c0b21a..18d141eb75 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -499,14 +499,15 @@ impl<'a> Context<'a> { match op.kind { ConstraintKind::FixedReg(regunit) | ConstraintKind::FixedTied(regunit) => { - if regunit != cur_reg { - self.solver.reassign_in( - value, - op.regclass, - cur_reg, - regunit, - ); - } + // Add the fixed constraint even if `cur_reg == regunit`. + // It is possible that we will want to convert the value to a variable later, + // and this identity assignment prevents that from happening. + self.solver.reassign_in( + value, + op.regclass, + cur_reg, + regunit, + ); } ConstraintKind::Reg | ConstraintKind::Tied(_) => { From dcad3fa33952c54b05f84b95767277c07dae14d1 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 17 Jan 2018 15:17:34 -0800 Subject: [PATCH 1470/3084] Fix coloring bug with combined constraints and global values. The Intel instruction "v1 = ushr v2, v2" will implicitly fix the output register for v2 to %rcx because the output is tied to the first input operand and the second input operand is fixed to %rcx. Make sure we handle this transitive constraint when checking for interference with the globally live registers. Fixes #218 --- .../filetests/regalloc/multi-constraints.cton | 23 +++++++++++++++ lib/cretonne/src/regalloc/coloring.rs | 28 +++++++++++++++++-- lib/cretonne/src/regalloc/solver.rs | 16 +++++++++-- 3 files changed, 61 insertions(+), 6 deletions(-) diff --git a/cranelift/filetests/regalloc/multi-constraints.cton b/cranelift/filetests/regalloc/multi-constraints.cton index 80dc8f0658..ecb3aa8bad 100644 --- a/cranelift/filetests/regalloc/multi-constraints.cton +++ b/cranelift/filetests/regalloc/multi-constraints.cton @@ -27,3 +27,26 @@ ebb0(v0: i64, v1: i64, v2: i64, v3: i64): ebb1: return v4 } + +; Found by the Binaryen fuzzer in PR218. +; +; This is a similar situation involving combined constraints on the ushr instruction: +; +; - The %rcx register is already in use by a globally live value. +; - The ushr x, x result is also a globally live value. +; +; Since the ushr x, x result is forced to be placed in %rcx, we must set the replace_global_defines +; flag so it can be reassigned to a different global register. +function %pr218(i64 [%rdi], i64 [%rsi], i64 [%rdx], i64 [%rcx]) -> i64 [%rax] { +ebb0(v0: i64, v1: i64, v2: i64, v3: i64): + ; check: regmove $v3, %rcx -> + v4 = ushr v0, v0 + ; check: $v4 = copy + jump ebb1 + +ebb1: + ; v3 is globally live in %rcx. + ; v4 is also globally live. Needs to be assigned something else for the trip across the CFG edge. + v5 = iadd v3, v4 + return v5 +} diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 18d141eb75..74d80e1832 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -411,7 +411,13 @@ impl<'a> Context<'a> { ®s.global, ); } - self.program_output_constraints(inst, constraints.outs, defs); + self.program_output_constraints( + inst, + constraints.outs, + defs, + &mut replace_global_defines, + ®s.global, + ); // Finally, we've fully programmed the constraint solver. // We expect a quick solution in most cases. @@ -794,6 +800,8 @@ impl<'a> Context<'a> { inst: Inst, constraints: &[OperandConstraint], defs: &[LiveValue], + replace_global_defines: &mut bool, + global_regs: &AllocatableSet, ) { for (op, lv) in constraints.iter().zip(defs) { match op.kind { @@ -807,12 +815,26 @@ impl<'a> Context<'a> { // Find the input operand we're tied to. // The solver doesn't care about the output value. let arg = self.cur.func.dfg.inst_args(inst)[num as usize]; - self.solver.add_tied_input( + if let Some(reg) = self.solver.add_tied_input( arg, op.regclass, self.divert.reg(arg, &self.cur.func.locations), !lv.is_local, - ); + ) + { + // The value we're tied to has been assigned to a fixed register. + // We need to make sure that fixed output register is compatible with the + // global register set. + if !lv.is_local && !global_regs.is_avail(op.regclass, reg) { + dbg!( + "Tied output {} in {}:{} is not available in global regs", + lv.value, + op.regclass, + self.reginfo.display_regunit(reg) + ); + *replace_global_defines = true; + } + } } } } diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index c7b884ce43..b5d5aa661b 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -760,14 +760,22 @@ impl Solver { /// /// The output value that must have the same register as the input value is not recorded in the /// solver. - pub fn add_tied_input(&mut self, value: Value, rc: RegClass, reg: RegUnit, is_global: bool) { + /// + /// If the value has already been assigned to a fixed register, return that. + pub fn add_tied_input( + &mut self, + value: Value, + rc: RegClass, + reg: RegUnit, + is_global: bool, + ) -> Option { debug_assert!(self.inputs_done); // If a fixed assignment is tied, the `to` register is not available on the output side. if let Some(a) = self.assignments.get(value) { debug_assert_eq!(a.from, reg); self.regs_out.take(a.rc, a.to); - return; + return Some(a.to); } // Check if a variable was created. @@ -775,7 +783,7 @@ impl Solver { assert!(v.is_input); v.is_output = true; v.is_global = is_global; - return; + return None; } // No variable exists for `value` because its constraints are already satisfied. @@ -790,6 +798,8 @@ impl Solver { } else { self.regs_out.take(rc, reg); } + + None } /// Add a fixed output assignment. From 1e49431804d26418490e4350300c892aee1605e5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 17 Jan 2018 16:18:07 -0800 Subject: [PATCH 1471/3084] Add test case from #216. The error exposed by this test case no longer happens after the coalescer was rewritten to to follow the Budimlic paper. It's still a good coalescer test. Fixes #216 by including the test case. --- .../filetests/regalloc/coalescing-216.cton | 73 +++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 cranelift/filetests/regalloc/coalescing-216.cton diff --git a/cranelift/filetests/regalloc/coalescing-216.cton b/cranelift/filetests/regalloc/coalescing-216.cton new file mode 100644 index 0000000000..83dda0e56a --- /dev/null +++ b/cranelift/filetests/regalloc/coalescing-216.cton @@ -0,0 +1,73 @@ +test regalloc +set is_64bit +isa intel haswell + +; Reported as https://github.com/stoklund/cretonne/issues/216 from the Binaryen fuzzer. +; +; The (old) coalescer creates a virtual register with two identical values. +function %pr216(i32 [%rdi], i64 vmctx [%rsi]) -> i64 [%rax] native { +ebb0(v0: i32, v1: i64): + v3 = iconst.i64 0 + v5 = iconst.i32 0 + brz v5, ebb3(v3) + jump ebb4(v3, v3) + +ebb4(v11: i64, v29: i64): + v6 = iconst.i32 0 + brz v6, ebb14 + v9 = iconst.i32 -17 + v12 = iconst.i32 0xffff_ffff_ffff_8000 + jump ebb9(v12) + +ebb9(v10: i32): + brnz v10, ebb8(v9, v11, v11) + brz.i32 v9, ebb13 + v13 = iconst.i32 0 + brnz v13, ebb6(v11, v11) + v14 = iconst.i32 0 + brz v14, ebb12 + jump ebb11 + +ebb12: + jump ebb4(v11, v11) + +ebb11: + jump ebb10(v11) + +ebb13: + v15 = iconst.i64 1 + jump ebb10(v15) + +ebb10(v21: i64): + v16 = iconst.i32 0 + brnz v16, ebb6(v21, v11) + v17 = iconst.i32 0xffff_ffff_ffff_9f35 + jump ebb8(v17, v21, v11) + +ebb8(v8: i32, v23: i64, v28: i64): + jump ebb7(v8, v23, v28) + +ebb14: + v18 = iconst.i32 0 + jump ebb7(v18, v11, v29) + +ebb7(v7: i32, v22: i64, v27: i64): + jump ebb6(v22, v27) + +ebb6(v20: i64, v25: i64): + v19 = iconst.i32 0xffc7 + brnz v19, ebb4(v20, v25) + jump ebb5 + +ebb5: + jump ebb3(v25) + +ebb3(v24: i64): + jump ebb2(v24) + +ebb2(v4: i64): + jump ebb1(v4) + +ebb1(v2: i64): + return v2 +} From 850896f05e4ef04230b513440b05b9e3f29569d8 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Thu, 18 Jan 2018 13:36:15 -0800 Subject: [PATCH 1472/3084] The addend for a PLTRel4 reloc should be -4. --- lib/cretonne/meta/isa/intel/recipes.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index e9c896123b..e613e9aeeb 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -899,7 +899,7 @@ call_plt_id = TailRecipe( PUT_OP(bits, BASE_REX, sink); sink.reloc_external(Reloc::IntelPLTRel4, &func.dfg.ext_funcs[func_ref].name, - 0); + -4); sink.put4(0); ''') From df210bfdea6bcf997678e75af46d073b5deb59f6 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Thu, 18 Jan 2018 13:40:08 -0800 Subject: [PATCH 1473/3084] Fix the Intel x64 PIC 'call' test, adding correct addend. --- cranelift/filetests/isa/intel/binary64-pic.cton | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/filetests/isa/intel/binary64-pic.cton b/cranelift/filetests/isa/intel/binary64-pic.cton index 70dc7bed42..fd6a5dad03 100644 --- a/cranelift/filetests/isa/intel/binary64-pic.cton +++ b/cranelift/filetests/isa/intel/binary64-pic.cton @@ -27,7 +27,7 @@ function %I64() { ebb0: ; asm: call foo@PLT - call fn0() ; bin: e8 PLTRel4(%foo) 00000000 + call fn0() ; bin: e8 PLTRel4(%foo-4) 00000000 ; asm: mov 0x0(%rip), %rax [-,%rax] v0 = func_addr.i64 fn0 ; bin: 48 8b 05 GOTPCRel4(%foo-4) 00000000 From 14e39db42814445580ff90473cbec642d9067ad8 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Thu, 18 Jan 2018 15:32:58 -0800 Subject: [PATCH 1474/3084] Add filetest for statically out-of-bound heap addresses. --- .../filetests/isa/intel/legalize-memory.cton | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/cranelift/filetests/isa/intel/legalize-memory.cton b/cranelift/filetests/isa/intel/legalize-memory.cton index 0257c7001c..fc17cfd2dc 100644 --- a/cranelift/filetests/isa/intel/legalize-memory.cton +++ b/cranelift/filetests/isa/intel/legalize-memory.cton @@ -65,6 +65,27 @@ ebb0(v0: i32, v999: i64): return v4 } +function %staticheap_static_oob_sm64(i32, i64 vmctx) -> f32 spiderwasm { + gv0 = vmctx+64 + heap0 = static gv0, min 0x1000, bound 0x1000_0000, guard 0x8000_0000 + +ebb0(v0: i32, v999: i64): + ; Everything after the obviously OOB access should be eliminated, leaving + ; the `trap heap_oob` instruction as the terminator of the Ebb and moving + ; the remainder of the instructions into an inaccessible Ebb. + ; check: $ebb0( + ; nextln: trap heap_oob + ; check: ebb1: + ; nextln: v2 = iconst.i64 0 + ; nextln: v3 = load.f32 v2+16 + ; nextln: return v3 + ; nextln: } + v1 = heap_addr.i64 heap0, v0, 0x1000_0001 + v2 = load.f32 v1+16 + return v2 +} + + ; SpiderMonkey VM-style static 4+2 GB heap. ; Offsets >= 2 GB do require a boundscheck. function %staticheap_sm64(i32, i64 vmctx) -> f32 spiderwasm { From 7826fce44f5e17004f2ce8f6b6ba0a185fbf16d9 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Thu, 18 Jan 2018 15:33:28 -0800 Subject: [PATCH 1475/3084] On finding an static OOB heap addr, split the Ebb and recompute the CFG. --- lib/cretonne/src/legalizer/heap.rs | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/legalizer/heap.rs b/lib/cretonne/src/legalizer/heap.rs index 09736925b4..6dd5c11095 100644 --- a/lib/cretonne/src/legalizer/heap.rs +++ b/lib/cretonne/src/legalizer/heap.rs @@ -9,7 +9,7 @@ use ir::{self, InstBuilder, MemFlags}; use ir::condcodes::IntCC; /// Expand a `heap_addr` instruction according to the definition of the heap. -pub fn expand_heap_addr(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlowGraph) { +pub fn expand_heap_addr(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { // Unpack the instruction. let (heap, offset, size) = match func.dfg[inst] { ir::InstructionData::HeapAddr { @@ -29,7 +29,7 @@ pub fn expand_heap_addr(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut Cont dynamic_addr(inst, heap, offset, size, bound_gv, func) } ir::HeapStyle::Static { bound } => { - static_addr(inst, heap, offset, size, bound.into(), func) + static_addr(inst, heap, offset, size, bound.into(), func, cfg) } } } @@ -95,6 +95,7 @@ fn static_addr( size: u32, bound: i64, func: &mut ir::Function, + cfg: &mut ControlFlowGraph, ) { let size = i64::from(size); let offset_ty = func.dfg.value_type(offset); @@ -107,6 +108,13 @@ fn static_addr( // This will simply always trap since `offset >= 0`. pos.ins().trap(ir::TrapCode::HeapOutOfBounds); pos.func.dfg.replace(inst).iconst(addr_ty, 0); + + // Split Ebb, as the trap is a terminator instruction. + let curr_ebb = pos.current_ebb().expect("Cursor is not in an ebb"); + let new_ebb = pos.func.dfg.make_ebb(); + pos.insert_ebb(new_ebb); + cfg.recompute_ebb(pos.func, curr_ebb); + cfg.recompute_ebb(pos.func, new_ebb); return; } From 1bbc529ef980c722aaaa65393c19245b7c7aa729 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 19 Jan 2018 13:31:26 -0800 Subject: [PATCH 1476/3084] Improve the variable ordering used by the coloring constraint solver. The fuzzer bugs #219 and #227 are both cases where the register allocator coloring pass "runs out of registers". What's really happening is that the constraint solver failed to find a solution, even when one existed. Suppose we have three solver variables: v0(GPR, out, global) v1(GPR, in) v2(GPR, in, out) And suppose registers %r0 and %r1 are available on both input and output sides of the instruction, but only %r1 is available for global outputs. A valid solution would be: v0 -> %r1 v1 -> %r1 v2 -> %r0 However, the solver would pick registers for the three values in numerical order because v1 and v2 have the same domain size (=2). This would assign v1 -> %r0 and then fail to find a free register for v2. Fix this by prioritizing in+out variables over single-sided variables even when their domains are equal. This means the v2 gets assigned a register before v1, and it gets a chance to pick a register that is still available on both in and out sides. Also try to avoid depending on value numbers in the solver. These bugs were hard to reproduce because a test case invariably would have different value numbers, causing the solver to order its variables differently and succeed. Throw in the previous solution and original register assignments as tie breakers which are stable and not dependent on value numbers. This is still not a substitute for a proper solver search algorithm that we will probably have to write eventually. Fixes #219 Fixes #227 --- .../filetests/regalloc/coloring-227.cton | 101 ++++++++++++++++++ lib/cretonne/src/regalloc/solver.rs | 33 +++++- 2 files changed, 132 insertions(+), 2 deletions(-) create mode 100644 cranelift/filetests/regalloc/coloring-227.cton diff --git a/cranelift/filetests/regalloc/coloring-227.cton b/cranelift/filetests/regalloc/coloring-227.cton new file mode 100644 index 0000000000..07681a7509 --- /dev/null +++ b/cranelift/filetests/regalloc/coloring-227.cton @@ -0,0 +1,101 @@ +test regalloc +set is_64bit +isa intel haswell + +function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) native { + gv0 = vmctx + heap0 = static gv0, min 0, bound 0x0001_0000_0000, guard 0x8000_0000 + + ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i64): +@0001 [RexOp1puid#b8] v5 = iconst.i32 0 +@0003 [RexOp1puid#b8] v6 = iconst.i32 0 +@0005 [RexOp1tjccb#74] brz v6, ebb10 +@0007 [RexOp1jmpb#eb] jump ebb3(v5, v5, v5, v5, v5, v5, v0, v1, v2, v3) + + ebb3(v15: i32, v17: i32, v25: i32, v31: i32, v40: i32, v47: i32, v54: i32, v61: i32, v68: i32, v75: i32): +@000b [RexOp1jmpb#eb] jump ebb6 + + ebb6: +@000d [RexOp1puid#b8] v8 = iconst.i32 0 +@000f [RexOp1tjccb#75] brnz v8, ebb5 +@0011 [RexOp1puid#b8] v9 = iconst.i32 0 +@0015 [RexOp1puid#b8] v11 = iconst.i32 0 +@0017 [RexOp1icscc#39] v12 = icmp.i32 eq v15, v11 +@0017 [RexOp2urm#4b6] v13 = bint.i32 v12 +@001a [RexOp1rr#21] v14 = band v9, v13 +@001b [RexOp1tjccb#75] brnz v14, ebb6 +@001d [RexOp1jmpb#eb] jump ebb7 + + ebb7: +@0020 [RexOp1tjccb#74] brz.i32 v17, ebb8 +@0022 [RexOp1puid#b8] v18 = iconst.i32 0 +@0024 [RexOp1tjccb#74] brz v18, ebb9 +@0028 [RexOp1puid#b8] v21 = iconst.i32 0 +@002a [RexOp1umr#89] v79 = uextend.i64 v5 +@002a [RexOp1rib#8083] v80 = iadd_imm.i64 v4, 0 +@002a [RexOp1ld#808b] v81 = load.i64 v80 +@002a [RexOp1rr#8001] v22 = iadd v81, v79 +@002a [RexMp1st#189] istore16 v21, v22 +@002d [RexOp1jmpb#eb] jump ebb9 + + ebb9: +@002e [RexOp1jmpb#eb] jump ebb8 + + ebb8: +@0033 [RexOp1puid#b8] v27 = iconst.i32 3 +@0035 [RexOp1puid#b8] v28 = iconst.i32 4 +@003b [RexOp1rr#09] v35 = bor.i32 v31, v13 +@003c [RexOp1tjccb#75] brnz v35, ebb15(v27) +@003c [RexOp1jmpb#eb] jump ebb15(v28) + + ebb15(v36: i32): +@003f [RexOp1jmpb#eb] jump ebb3(v25, v36, v25, v31, v40, v47, v54, v61, v68, v75) + + ebb5: +@0042 [RexOp1jmpb#eb] jump ebb4 + + ebb4: +@0045 [RexOp1jmpb#eb] jump ebb2(v40, v47, v54, v61, v68, v75) + + ebb10: +@0046 [RexOp1puid#b8] v43 = iconst.i32 0 +@0048 [RexOp1jmpb#eb] jump ebb2(v43, v5, v0, v1, v2, v3) + + ebb2(v7: i32, v45: i32, v52: i32, v59: i32, v66: i32, v73: i32): +@004c [RexOp1puid#b8] v44 = iconst.i32 0 +@004e [RexOp1tjccb#74] brz v44, ebb12 +@0052 [RexOp1puid#b8] v50 = iconst.i32 11 +@0054 [RexOp1tjccb#74] brz v50, ebb14 +@0058 [RexOp1umr#89] v82 = uextend.i64 v52 +@0058 [RexOp1rib#8083] v83 = iadd_imm.i64 v4, 0 +@0058 [RexOp1ld#808b] v84 = load.i64 v83 +@0058 [RexOp1rr#8001] v57 = iadd v84, v82 +@0058 [RexOp1ld#8b] v58 = load.i32 v57 +@005d [RexOp1umr#89] v85 = uextend.i64 v58 +@005d [RexOp1rib#8083] v86 = iadd_imm.i64 v4, 0 +@005d [RexOp1ld#808b] v87 = load.i64 v86 +@005d [RexOp1rr#8001] v64 = iadd v87, v85 +@005d [RexOp1st#88] istore8 v59, v64 +@0060 [RexOp1puid#b8] v65 = iconst.i32 0 +@0062 [RexOp1jmpb#eb] jump ebb13(v65) + + ebb14: +@0065 [RexOp1jmpb#eb] jump ebb13(v66) + + ebb13(v51: i32): +@0066 [RexOp1umr#89] v88 = uextend.i64 v45 +@0066 [RexOp1rib#8083] v89 = iadd_imm.i64 v4, 0 +@0066 [RexOp1ld#808b] v90 = load.i64 v89 +@0066 [RexOp1rr#8001] v71 = iadd v90, v88 +@0066 [RexOp1st#89] store v51, v71 +@0069 [RexOp1jmpb#eb] jump ebb12 + + ebb12: +@006a [RexOp1jmpb#eb] jump ebb11 + + ebb11: +@006e [RexOp1jmpb#eb] jump ebb1 + + ebb1: +@006e [Op1ret#c3] return +} diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index b5d5aa661b..a3b732799c 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -537,6 +537,8 @@ impl Solver { self.regs_in = regs.clone(); // Used for tracking fixed input assignments while `!inputs_done`: self.regs_out = AllocatableSet::new(); + self.moves.clear(); + self.fills.clear(); } /// Add a fixed input reassignment of `value`. @@ -891,8 +893,35 @@ impl Solver { v.domain = cmp::min(d, u16::MAX as usize) as u16; } // Solve for vars with small domains first to increase the chance of finding a solution. - // Use the value number as a tie breaker to get a stable sort. - self.vars.sort_unstable_by_key(|v| (v.domain, v.value)); + // + // Also consider this case: + // + // v0: out, global + // v1: in + // v2: in+out + // + // If only %r0 and %r1 are available, the global constraint may cause us to assign: + // + // v0 -> %r1 + // v1 -> %r0 + // v2 -> ! + // + // Usually in+out variables will have a smaller domain, but in the above case the domain + // size is the same, so we also prioritize in+out variables. + // + // Include the reversed previous solution for this variable partly as a stable tie breaker, + // partly to shake things up on a second attempt. + // + // Use the `from` register and value number as a tie breaker to get a stable sort. + self.vars.sort_unstable_by_key(|v| { + ( + v.domain, + !(v.is_input && v.is_output), + !v.solution, + v.from.unwrap_or(0), + v.value, + ) + }); dbg!("real_solve for {}", self); self.find_solution(global_regs) From 10845995a8ef7e77cc05fec1675fa2e918525608 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 22 Jan 2018 07:51:25 -0800 Subject: [PATCH 1477/3084] Make room for 4-digit pass timings. (sigh) --- lib/cretonne/src/timing.rs | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/src/timing.rs b/lib/cretonne/src/timing.rs index e1855cf65a..42bbea884b 100644 --- a/lib/cretonne/src/timing.rs +++ b/lib/cretonne/src/timing.rs @@ -133,9 +133,9 @@ mod details { impl fmt::Display for PassTimes { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - writeln!(f, "======= ======= ==================================")?; - writeln!(f, " Total Self Pass")?; - writeln!(f, "------- ------- ----------------------------------")?; + writeln!(f, "======== ======== ==================================")?; + writeln!(f, " Total Self Pass")?; + writeln!(f, "-------- -------- ----------------------------------")?; for (time, desc) in self.pass.iter().zip(&DESCRIPTIONS) { // Omit passes that haven't run. if time.total == Duration::default() { @@ -147,7 +147,7 @@ mod details { // Round to nearest ms by adding 500us. dur += Duration::new(0, 500_000); let ms = dur.subsec_nanos() / 1_000_000; - write!(f, "{:3}.{:03} ", dur.as_secs(), ms) + write!(f, "{:4}.{:03} ", dur.as_secs(), ms) } fmtdur(time.total, f)?; @@ -156,7 +156,7 @@ mod details { } writeln!(f, " {}", desc)?; } - writeln!(f, "======= ======= ==================================") + writeln!(f, "======== ======== ==================================") } } From 91343f208d2c6c8c164b23828b3c6d461999aa70 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 22 Jan 2018 09:21:42 -0800 Subject: [PATCH 1478/3084] Fix quadratic behavior in sequence numbering. The ir::layout module is assigning sequence numbers to all EBBs and instructions so relative positions can be computed in constant time. This works a lot like BASIC line numbers where we initially use numbers 10, 20, 30, ... so we can insert new instructions in the middle of the sequence without renumbering everything. In some cases where the coalescer is misbehaving and inserting a lot of copy instructions, we end up having to renumber a larger and larger number of instructions to make space in the sequence. This causes the following reload pass to be very slow, spending most of its time renumbering instructions. Fix this by putting an upper limit on the number of instructions we're willing to renumber locally. When the limit is reached, switch to a full function renumbering with the major stride of 10. This gives us new elasticity in the sequence numbers. - Time to compile the Python interpreter in #229 drops from 4826 s -> 15.8 s. - The godot benchmark in #226 drops from 1257 s -> 75 s. - The AngryBots1 benchmark does not have the coalescer misbehavior. Its compilation time changes 22.9 s -> 23.1 s. It's worth noting that the sequence numbering is still technically quadratic with this fix. The system is not designed to handle a large number of instructions inserted in a single location. It expects a more even distribution of new instructions. We still need to fix the coalescer. It should not insert so many copies in degenerate cases. --- lib/cretonne/src/ir/layout.rs | 58 ++++++++++++++++++++++++++++++----- lib/cretonne/src/timing.rs | 1 + 2 files changed, 51 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 551055394a..e18ffd7a33 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -9,6 +9,7 @@ use ir::progpoint::{ProgramOrder, ExpandedProgramPoint}; use packed_option::PackedOption; use std::cmp; use std::iter::{Iterator, IntoIterator}; +use timing; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not /// contain definitions of instructions or EBBs, but depends on `Inst` and `Ebb` entity references @@ -80,6 +81,10 @@ const MAJOR_STRIDE: SequenceNumber = 10; // Secondary stride used when renumbering locally. const MINOR_STRIDE: SequenceNumber = 2; +// Limit on the sequence number range we'll renumber locally. If this limit is exceeded, we'll +// switch to a full function renumbering. +const LOCAL_LIMIT: SequenceNumber = 100 * MINOR_STRIDE; + // Compute the midpoint between `a` and `b`. // Return `None` if the midpoint would be equal to either. fn midpoint(a: SequenceNumber, b: SequenceNumber) -> Option { @@ -167,7 +172,7 @@ impl Layout { self.ebbs[ebb].seq = seq; } else { // No available integers between `prev_seq` and `next_seq`. We have to renumber. - self.renumber_from_ebb(ebb, prev_seq + MINOR_STRIDE); + self.renumber_from_ebb(ebb, prev_seq + MINOR_STRIDE, prev_seq + LOCAL_LIMIT); } } @@ -200,7 +205,7 @@ impl Layout { self.insts[inst].seq = seq; } else { // No available integers between `prev_seq` and `next_seq`. We have to renumber. - self.renumber_from_inst(inst, prev_seq + MINOR_STRIDE); + self.renumber_from_inst(inst, prev_seq + MINOR_STRIDE, prev_seq + LOCAL_LIMIT); } } @@ -209,7 +214,14 @@ impl Layout { /// /// Return `None` if renumbering has caught up and the sequence is monotonic again. Otherwise /// return the last used sequence number. - fn renumber_insts(&mut self, inst: Inst, seq: SequenceNumber) -> Option { + /// + /// If sequence numbers exceed `limit`, switch to a full function renumbering and return `None`. + fn renumber_insts( + &mut self, + inst: Inst, + seq: SequenceNumber, + limit: SequenceNumber, + ) -> Option { let mut inst = inst; let mut seq = seq; @@ -227,13 +239,20 @@ impl Layout { return None; } + if seq > limit { + // We're pushing too many instructions in front of us. + // Switch to a full function renumbering to make some space. + self.full_renumber(); + return None; + } + seq += MINOR_STRIDE; } } /// Renumber starting from `ebb` to `seq` and continuing until the sequence numbers are /// monotonic again. - fn renumber_from_ebb(&mut self, ebb: Ebb, first_seq: SequenceNumber) { + fn renumber_from_ebb(&mut self, ebb: Ebb, first_seq: SequenceNumber, limit: SequenceNumber) { let mut ebb = ebb; let mut seq = first_seq; @@ -242,7 +261,7 @@ impl Layout { // Renumber instructions in `ebb`. Stop when the numbers catch up. if let Some(inst) = self.ebbs[ebb].first_inst.expand() { - seq = match self.renumber_insts(inst, seq + MINOR_STRIDE) { + seq = match self.renumber_insts(inst, seq + MINOR_STRIDE, limit) { Some(s) => s, None => return, } @@ -265,14 +284,37 @@ impl Layout { /// Renumber starting from `inst` to `seq` and continuing until the sequence numbers are /// monotonic again. - fn renumber_from_inst(&mut self, inst: Inst, first_seq: SequenceNumber) { - if let Some(seq) = self.renumber_insts(inst, first_seq) { + fn renumber_from_inst(&mut self, inst: Inst, first_seq: SequenceNumber, limit: SequenceNumber) { + if let Some(seq) = self.renumber_insts(inst, first_seq, limit) { // Renumbering spills over into next EBB. if let Some(next_ebb) = self.ebbs[self.inst_ebb(inst).unwrap()].next.expand() { - self.renumber_from_ebb(next_ebb, seq + MINOR_STRIDE); + self.renumber_from_ebb(next_ebb, seq + MINOR_STRIDE, limit); } } } + + /// Renumber all EBBs and instructions in the layout. + /// + /// This doesn't affect the position of anything, but it gives more room in the internal + /// sequence numbers for inserting instructions later. + fn full_renumber(&mut self) { + let _tt = timing::layout_renumber(); + let mut seq = 0; + let mut next_ebb = self.first_ebb; + while let Some(ebb) = next_ebb { + self.ebbs[ebb].seq = seq; + seq += MAJOR_STRIDE; + next_ebb = self.ebbs[ebb].next.expand(); + + let mut next_inst = self.ebbs[ebb].first_inst.expand(); + while let Some(inst) = next_inst { + self.insts[inst].seq = seq; + seq += MAJOR_STRIDE; + next_inst = self.insts[inst].next.expand(); + } + } + dbg!("Renumbered {} program points", seq / MAJOR_STRIDE); + } } /// Methods for laying out EBBs. diff --git a/lib/cretonne/src/timing.rs b/lib/cretonne/src/timing.rs index 42bbea884b..d120d43c32 100644 --- a/lib/cretonne/src/timing.rs +++ b/lib/cretonne/src/timing.rs @@ -69,6 +69,7 @@ define_passes!{ prologue_epilogue: "Prologue/epilogue insertion", binemit: "Binary machine code emission", + layout_renumber: "Layout full renumbering", } impl Pass { From c6bb7afa07eb0a87b3c9df9486cec8b68696dc23 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 22 Jan 2018 10:48:14 -0800 Subject: [PATCH 1479/3084] Dampen quadratic behavior in check_cssa(). Use a dominator tree pre-order to speed up the dominance checks and only check for interference with the nearest dominating predecessor in a virtual register. This makes the CSSA verification about 2x as fast. --- lib/cretonne/src/verifier/cssa.rs | 47 ++++++++++++++++++++++--------- 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/lib/cretonne/src/verifier/cssa.rs b/lib/cretonne/src/verifier/cssa.rs index d29b0d72b5..c508ed8d49 100644 --- a/lib/cretonne/src/verifier/cssa.rs +++ b/lib/cretonne/src/verifier/cssa.rs @@ -1,7 +1,7 @@ //! Verify conventional SSA form. use dbg::DisplayList; -use dominator_tree::DominatorTree; +use dominator_tree::{DominatorTree, DominatorTreePreorder}; use flowgraph::ControlFlowGraph; use ir::{Function, ExpandedProgramPoint}; use regalloc::liveness::Liveness; @@ -31,12 +31,17 @@ pub fn verify_cssa( virtregs: &VirtRegs, ) -> Result { let _tt = timing::verify_cssa(); + + let mut preorder = DominatorTreePreorder::new(); + preorder.compute(domtree, &func.layout); + let verifier = CssaVerifier { func, cfg, domtree, virtregs, liveness, + preorder, }; verifier.check_virtregs()?; verifier.check_cssa()?; @@ -49,6 +54,7 @@ struct CssaVerifier<'a> { domtree: &'a DominatorTree, virtregs: &'a VirtRegs, liveness: &'a Liveness, + preorder: DominatorTreePreorder, } impl<'a> CssaVerifier<'a> { @@ -72,6 +78,7 @@ impl<'a> CssaVerifier<'a> { let def_ebb = self.func.layout.pp_ebb(def); for &prev_val in &values[0..idx] { let prev_def: ExpandedProgramPoint = self.func.dfg.value_def(prev_val).into(); + let prev_ebb = self.func.layout.pp_ebb(prev_def); if prev_def == def { return err!( @@ -85,7 +92,9 @@ impl<'a> CssaVerifier<'a> { } // Enforce topological ordering of defs in the virtual register. - if self.domtree.dominates(def, prev_def, &self.func.layout) { + if self.preorder.dominates(def_ebb, prev_ebb) && + self.domtree.dominates(def, prev_def, &self.func.layout) + { return err!( val, "Value in {} = {} def dominates previous {}", @@ -94,18 +103,30 @@ impl<'a> CssaVerifier<'a> { prev_val ); } + } - // Knowing that values are in topo order, we can check for interference this - // way. - let ctx = self.liveness.context(&self.func.layout); - if self.liveness[prev_val].overlaps_def(def, def_ebb, ctx) { - return err!( - val, - "Value def in {} = {} interferes with {}", - vreg, - DisplayList(values), - prev_val - ); + // Knowing that values are in topo order, we can check for interference this + // way. + // We only have to check against the nearest dominating value. + for &prev_val in values[0..idx].iter().rev() { + let prev_def: ExpandedProgramPoint = self.func.dfg.value_def(prev_val).into(); + let prev_ebb = self.func.layout.pp_ebb(prev_def); + + if self.preorder.dominates(prev_ebb, def_ebb) && + self.domtree.dominates(prev_def, def, &self.func.layout) + { + let ctx = self.liveness.context(&self.func.layout); + if self.liveness[prev_val].overlaps_def(def, def_ebb, ctx) { + return err!( + val, + "Value def in {} = {} interferes with {}", + vreg, + DisplayList(values), + prev_val + ); + } else { + break; + } } } } From 085e228358ab472193d0faffb87e486f34383e13 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 22 Jan 2018 11:31:21 -0800 Subject: [PATCH 1480/3084] Remove assertions from the hot value_def() function. The errors caught by these assertions are also detected by the IL verifier. Speed up compilation by 13%. --- lib/cretonne/src/ir/dfg.rs | 20 ++------------------ 1 file changed, 2 insertions(+), 18 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 25457000d7..5374138e51 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -164,24 +164,8 @@ impl DataFlowGraph { /// parameter. pub fn value_def(&self, v: Value) -> ValueDef { match self.values[v] { - ValueData::Inst { inst, num, .. } => { - assert_eq!( - Some(v), - self.results[inst].get(num as usize, &self.value_lists), - "Dangling result value {}: {}", - v, - self.display_inst(inst, None) - ); - ValueDef::Result(inst, num as usize) - } - ValueData::Param { ebb, num, .. } => { - assert_eq!( - Some(v), - self.ebbs[ebb].params.get(num as usize, &self.value_lists), - "Dangling EBB parameter value" - ); - ValueDef::Param(ebb, num as usize) - } + ValueData::Inst { inst, num, .. } => ValueDef::Result(inst, num as usize), + ValueData::Param { ebb, num, .. } => ValueDef::Param(ebb, num as usize), ValueData::Alias { original, .. } => { // Make sure we only recurse one level. `resolve_aliases` has safeguards to // detect alias loops without overrunning the stack. From d2e786a78a1927fedf6d021d72b6b218992e0f20 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 22 Jan 2018 12:40:51 -0800 Subject: [PATCH 1481/3084] Resurrect the VirtRegs::unify() function. The coalescer needs to be able to merge virtual registers with this function. Updated to recycle all virtual registers. --- lib/cretonne/src/regalloc/virtregs.rs | 39 +++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index 4fe724cf74..9c3c6616cf 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -149,6 +149,45 @@ impl VirtRegs { self.vregs.push(Default::default()) }) } + + /// Unify `values` into a single virtual register. + /// + /// The values in the slice can be singletons or they can belong to a virtual register already. + /// If a value belongs to a virtual register, all of the values in that register must be + /// present. + /// + /// The values are assumed to already be in topological order. + pub fn unify(&mut self, values: &[Value]) -> VirtReg { + // Start by clearing all virtual registers involved. + let mut singletons = 0; + let mut cleared = 0; + for &val in values { + match self.get(val) { + None => singletons += 1, + Some(vreg) => { + if !self.vregs[vreg].is_empty() { + cleared += self.vregs[vreg].len(&self.pool); + self.vregs[vreg].clear(&mut self.pool); + self.unused_vregs.push(vreg); + } + } + } + } + + assert_eq!( + values.len(), + singletons + cleared, + "Can't unify partial virtual registers" + ); + + let vreg = self.alloc(); + self.vregs[vreg].extend(values.iter().cloned(), &mut self.pool); + for &v in values { + self.value_vregs[v] = vreg.into(); + } + + vreg + } } impl fmt::Display for VirtRegs { From b124eaf77d016d471eeb7479f2b4ae826e885e48 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 23 Jan 2018 14:53:59 -0800 Subject: [PATCH 1482/3084] Add a preorder pre_cmp_def() function. This provides a total ordering of values according to when their definition appears in the dominator tree pre-order. --- lib/cretonne/src/dominator_tree.rs | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 96fffc5b09..4d4e73d406 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -2,7 +2,7 @@ use entity::EntityMap; use flowgraph::{ControlFlowGraph, BasicBlock}; -use ir::{Ebb, Inst, Function, Layout, ProgramOrder, ExpandedProgramPoint}; +use ir::{Ebb, Inst, Value, Function, Layout, ProgramOrder, ExpandedProgramPoint}; use ir::instructions::BranchInfo; use packed_option::PackedOption; use std::cmp; @@ -649,6 +649,20 @@ impl DominatorTreePreorder { layout.cmp(a, b), ) } + + /// Compare two value defs according to the dominator tree pre-order. + /// + /// Two values defined at the same program point are compared according to their parameter or + /// result order. + /// + /// This is a total ordering of the values in the function. + pub fn pre_cmp_def(&self, a: Value, b: Value, func: &Function) -> Ordering { + let da = func.dfg.value_def(a); + let db = func.dfg.value_def(b); + self.pre_cmp(da, db, &func.layout).then_with( + || da.num().cmp(&db.num()), + ) + } } #[cfg(test)] From 416b21c18dac3300c5eb95b46c29597da7b32824 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 22 Jan 2018 12:27:04 -0800 Subject: [PATCH 1483/3084] Pairwise virtual register coalescing. Use a better algorithm for resolving interferences in virtual registers. This improves code quality by generating much fewer copies on some complicated functions. After the initial union-find phase, the check_vreg() function uses a Budimlic forest to check for interference between the values in the virtual registers, as before. All the interference-free vregs are done. Others are passed to synthesize_vreg() which dissolves the vreg and then attempts to rebuild one or more vregs from the contained values. The pairwise interference checks use *virtual copies* to make sure that any future conflicts can be resolved by inserting a copy instruction. This technique was not present in the old coalescer which caused some correctness issues. This coalescing algorithm makes much better code, and it is generally a bit slower than before. Some of the slowdown is made up by the following passes being faster because they have to process less code. Example 1, the Python interpreter which contains a very large function with a lot of variables. Before: 15.664 0.011 Register allocation 1.535 1.535 RA liveness analysis 2.872 1.911 RA coalescing CSSA 4.436 4.436 RA spilling 2.610 2.598 RA reloading 4.200 4.199 RA coloring After: 9.795 0.013 Register allocation 1.372 1.372 RA liveness analysis 6.231 6.227 RA coalescing CSSA 0.712 0.712 RA spilling 0.598 0.598 RA reloading 0.869 0.869 RA coloring Coalescing is more than twice as slow, but because of the vastly better code quality, overall register allocation time is improved by 37%. Example 2, the clang compiler. Before: 57.148 0.035 Register allocation 9.630 9.630 RA liveness analysis 7.210 7.169 RA coalescing CSSA 9.972 9.972 RA spilling 11.602 11.572 RA reloading 18.698 18.672 RA coloring After: 64.792 0.042 Register allocation 8.630 8.630 RA liveness analysis 22.937 22.928 RA coalescing CSSA 8.684 8.684 RA spilling 9.559 9.551 RA reloading 14.939 14.936 RA coloring Here coalescing is 3x slower, but overall regalloc time only regresses by 13%. Most examples are less extreme than these two. They just get better code at about the same compile time. --- lib/cretonne/src/ir/dfg.rs | 8 + lib/cretonne/src/ir/progpoint.rs | 10 + lib/cretonne/src/regalloc/coalescing.rs | 782 ++++++++++++++++++------ lib/cretonne/src/regalloc/virtregs.rs | 56 +- 4 files changed, 664 insertions(+), 192 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 5374138e51..923579d45d 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -317,6 +317,14 @@ impl ValueDef { } } + /// Unwrap the EBB there the parameter is defined, or panic. + pub fn unwrap_ebb(&self) -> Ebb { + match *self { + ValueDef::Param(ebb, _) => ebb, + _ => panic!("Value is not an EBB parameter"), + } + } + /// Get the program point where the value was defined. pub fn pp(self) -> ir::ExpandedProgramPoint { self.into() diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index ef3735c180..480160ea96 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -51,6 +51,16 @@ pub enum ExpandedProgramPoint { Ebb(Ebb), } +impl ExpandedProgramPoint { + /// Get the instruction we know is inside. + pub fn unwrap_inst(self) -> Inst { + match self { + ExpandedProgramPoint::Inst(x) => x, + ExpandedProgramPoint::Ebb(x) => panic!("expected inst: {}", x), + } + } +} + impl From for ExpandedProgramPoint { fn from(inst: Inst) -> ExpandedProgramPoint { ExpandedProgramPoint::Inst(inst) diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index ff01542bd8..2471c74284 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -9,12 +9,15 @@ use cursor::{Cursor, EncCursor}; use dbg::DisplayList; use dominator_tree::{DominatorTree, DominatorTreePreorder}; use flowgraph::ControlFlowGraph; -use ir::{self, InstBuilder}; +use ir::{self, InstBuilder, ProgramOrder}; use ir::{Function, Ebb, Inst, Value, ExpandedProgramPoint}; use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; use regalloc::virtregs::{VirtReg, VirtRegs}; +use std::cmp; +use std::iter; use std::fmt; +use std::slice; use isa::{TargetIsa, EncInfo}; use timing; @@ -50,14 +53,12 @@ use timing; /// Data structures to be used by the coalescing pass. pub struct Coalescing { - forest: DomForest, preorder: DominatorTreePreorder, - - /// EBB parameter values present in the current virtual register. - params: Vec, - - /// Worklist of virtual registers that need to be processed. - worklist: Vec, + forest: DomForest, + vcopies: VirtualCopies, + values: Vec, + predecessors: Vec, + backedges: Vec, } /// One-shot context created once per invocation. @@ -73,8 +74,10 @@ struct Context<'a> { virtregs: &'a mut VirtRegs, forest: &'a mut DomForest, - params: &'a mut Vec, - worklist: &'a mut Vec, + vcopies: &'a mut VirtualCopies, + values: &'a mut Vec, + predecessors: &'a mut Vec, + backedges: &'a mut Vec, } impl Coalescing { @@ -83,8 +86,10 @@ impl Coalescing { Self { forest: DomForest::new(), preorder: DominatorTreePreorder::new(), - params: Vec::new(), - worklist: Vec::new(), + vcopies: VirtualCopies::new(), + values: Vec::new(), + predecessors: Vec::new(), + backedges: Vec::new(), } } @@ -92,8 +97,10 @@ impl Coalescing { /// Clear all data structures in this coalescing pass. pub fn clear(&mut self) { self.forest.clear(); - self.params.clear(); - self.worklist.clear(); + self.vcopies.clear(); + self.values.clear(); + self.predecessors.clear(); + self.backedges.clear(); } /// Convert `func` to conventional SSA form and build virtual registers in the process. @@ -119,8 +126,10 @@ impl Coalescing { liveness, virtregs, forest: &mut self.forest, - params: &mut self.params, - worklist: &mut self.worklist, + vcopies: &mut self.vcopies, + values: &mut self.values, + predecessors: &mut self.predecessors, + backedges: &mut self.backedges, }; // Run phase 1 (union-find) of the coalescing algorithm on the current function. @@ -383,151 +392,275 @@ impl<'a> Context<'a> { pub fn process_vregs(&mut self) { for vreg in self.virtregs.all_virtregs() { self.process_vreg(vreg); - while let Some(vr) = self.worklist.pop() { - self.process_vreg(vr); - } } } // Check `vreg` for interferences and fix conflicts. fn process_vreg(&mut self, vreg: VirtReg) { - if self.analyze_vreg(vreg) { + if !self.check_vreg(vreg) { self.synthesize_vreg(vreg); } } - // Check `vreg` for interferences and choose values to isolate. + // Check `vreg` for interferences. // // We use a Budimlic dominator forest to check for interferences between the values in `vreg` // and identify values that should be isolated. // - // Returns true if `vreg` has conflicts that need to be fixed. Additionally leaves state in - // member variables: - // - // - `self.params` contains all the EBB parameter values that were present in the virtual - // register. - // - `self.forest` contains the set of values that should be isolated from the virtual register. - fn analyze_vreg(&mut self, vreg: VirtReg) -> bool { + // Returns true if `vreg` is free of interference. + fn check_vreg(&mut self, vreg: VirtReg) -> bool { // Order the values according to the dominator pre-order of their definition. - let dfg = &self.func.dfg; - let layout = &self.func.layout; - let preorder = self.preorder; - let values = self.virtregs.sort_values(vreg, |a, b| { - let da = dfg.value_def(a); - let db = dfg.value_def(b); - preorder.pre_cmp(da, db, layout).then( - da.num().cmp(&db.num()), - ) - }); - dbg!("Analyzing {} = {}", vreg, DisplayList(values)); + let values = self.virtregs.sort_values(vreg, self.func, self.preorder); + dbg!("Checking {} = {}", vreg, DisplayList(values)); - // Now push the values in order to the dominator forest. This gives us the closest - // dominating value def for each of the values. - self.params.clear(); + // Now push the values in order to the dominator forest. + // This gives us the closest dominating value def for each of the values. self.forest.clear(); for &value in values { - let node = Node::new(value, self.func); - - // Remember the parameter values in case we need to re-synthesize virtual registers. - if let ExpandedProgramPoint::Ebb(_) = node.def { - self.params.push(value); - } + let node = Node::value(value, 0, self.func); // Push this value and get the nearest dominating def back. - let parent = match self.forest.push_value( + let parent = match self.forest.push_node( node, self.func, self.domtree, self.preorder, ) { None => continue, - Some(p) => p, + Some(n) => n, }; // Check for interference between `parent` and `value`. Since `parent` dominates // `value`, we only have to check if it overlaps the definition. let ctx = self.liveness.context(&self.func.layout); - if !self.liveness[parent].overlaps_def(node.def, node.ebb, ctx) { - // No interference, both values can stay in the virtual register. + if self.liveness[parent.value].overlaps_def(node.def, node.ebb, ctx) { + // The two values are interfering, so they can't be in the same virtual register. + dbg!("-> interference: {} overlaps def of {}", parent, value); + return false; + } + } + + // No interference found. + true + } + + /// Destroy and rebuild `vreg` by iterative coalescing. + /// + /// When detecting that a virtual register formed in phase 1 contains interference, we have to + /// start over in a more careful way. We'll split the vreg into individual values and then + /// reassemble virtual registers using an iterative algorithm of pairwise merging. + /// + /// It is possible to recover multiple large virtual registers this way while still avoiding + /// a lot of copies. + fn synthesize_vreg(&mut self, vreg: VirtReg) { + self.vcopies.initialize( + self.virtregs.values(vreg), + self.func, + self.cfg, + self.preorder, + ); + dbg!( + "Synthesizing {} from {} branches and params {}", + vreg, + self.vcopies.branches.len(), + DisplayList(&self.vcopies.params) + ); + self.virtregs.remove(vreg); + + while let Some(param) = self.vcopies.next_param() { + self.merge_param(param); + self.vcopies.merged_param(param, self.func); + } + } + + /// Merge EBB parameter value `param` with virtual registers at its predecessors. + fn merge_param(&mut self, param: Value) { + let (ebb, argnum) = match self.func.dfg.value_def(param) { + ir::ValueDef::Param(e, n) => (e, n), + ir::ValueDef::Result(_, _) => panic!("Expected parameter"), + }; + + // Collect all the predecessors and rearrange them. + // + // The order we process the predecessors matters because once one predecessor's virtual + // register is merged, it can cause interference with following merges. This means that the + // first predecessors processed are more likely to be copy-free. We want an ordering that + // is a) good for performance and b) as stable as possible. The pred_iter() iterator uses + // instruction numbers which is not great for reproducible test cases. + // + // First merge loop back-edges in layout order, on the theory that shorter back-edges are + // more sensitive to inserted copies. + // + // Second everything else in reverse layout order. Again, short forward branches get merged + // first. There can also be backwards branches mixed in here, though, as long as they are + // not loop backedges. + assert!(self.predecessors.is_empty()); + assert!(self.backedges.is_empty()); + for (pred_ebb, pred_inst) in self.cfg.pred_iter(ebb) { + if self.preorder.dominates(ebb, pred_ebb) { + self.backedges.push(pred_inst); + } else { + self.predecessors.push(pred_inst); + } + } + // Order instructions in reverse order so we can pop them off the back. + { + let l = &self.func.layout; + self.backedges.sort_unstable_by(|&a, &b| l.cmp(b, a)); + self.predecessors.sort_unstable_by(|&a, &b| l.cmp(a, b)); + self.predecessors.extend_from_slice(&self.backedges); + self.backedges.clear(); + } + + while let Some(pred_inst) = self.predecessors.pop() { + let arg = self.func.dfg.inst_variable_args(pred_inst)[argnum]; + + // We want to merge the vreg containing `param` with the vreg containing `arg`. + if self.try_merge_vregs(param, arg) { continue; } - // The two values are interfering, so they can't both be in the same virtual register. - // We need to pick one to isolate. It's hard to pick a heuristic that only looks at two - // values since an optimal solution is a global problem involving all the values in the - // virtual register. - // - // We choose to always isolate the dominating parent value for two reasons: - // - // 1. We avoid the case of a parent value with a very long live range pushing many - // following values out of the virtual register. - // - // 2. In the case of a value that is live across a branch to the definition of a - // parameter in the virtual register, our splitting method in `synthesize_vreg` - // doesn't actually resolve the interference unless we're trying to isolate the - // first value. This heuristic will at least pick the first value on a second - // attempt. This is actually a correctness issue - we could loop infinitely - // otherwise. See the `infinite-interference.cton` test case. - dbg!("-> isolating {} which overlaps def of {}", parent, value); - self.forest.drop_value(parent); + // Can't merge because of interference. Insert a copy instead. + let pred_ebb = self.func.layout.pp_ebb(pred_inst); + let new_arg = self.isolate_arg(pred_ebb, pred_inst, argnum, arg); + self.virtregs.insert_single( + param, + new_arg, + self.func, + self.preorder, + ); } - - let dropped = self.forest.prepare_dropped(); - assert!(dropped < values.len()); - dropped != 0 } - /// Destroy and rebuild `vreg`. + /// Merge the virtual registers containing `param` and `arg` if possible. /// - /// Use `self.params` to rebuild the virtual register, but this time making sure that dropped - /// values in `self.forest` are isolated from non-dropped values. This may cause multiple new - /// virtual registers to be formed. + /// Use self.vcopies to check for virtual copy interference too. /// - /// All new virtual registers are appended to `self.worklist`. - fn synthesize_vreg(&mut self, vreg: VirtReg) { - dbg!("Synthesizing {} from {}", vreg, DisplayList(self.params)); - self.virtregs.remove(vreg); + /// Returns true if the virtual registers are successfully merged. + fn try_merge_vregs(&mut self, param: Value, arg: Value) -> bool { + if self.virtregs.same_class(param, arg) { + return true; + } - while let Some(param) = self.params.pop() { - let param_dropped = self.forest.is_dropped(param); - let (ebb, argnum) = match self.func.dfg.value_def(param) { - ir::ValueDef::Param(e, n) => (e, n), - ir::ValueDef::Result(_, _) => panic!("{} expected to be EBB parameter"), + if !self.can_merge_vregs(param, arg) { + return false; + } + + let vreg = self.virtregs.unify(self.values); + dbg!("-> merged into {} = {}", vreg, DisplayList(self.values)); + true + } + + /// Check if it is possible to merge two virtual registers. + /// + /// Also leave `self.values` with the ordered list of values in the merged vreg. + fn can_merge_vregs(&mut self, param: Value, arg: Value) -> bool { + // We only need an immutable function reference. + let func = &*self.func; + let domtree = self.domtree; + let preorder = self.preorder; + + // Restrict the virtual copy nodes we look at and key the `set_id` and `value` properties + // of the nodes. Set_id 0 will be `param` and set_id 1 will be `arg`. + self.vcopies.set_filter( + [param, arg], + func, + self.virtregs, + preorder, + ); + + // Now create an ordered sequence of dom-forest nodes from three sources: The two virtual + // registers and the filtered virtual copies. + let v0 = self.virtregs.congruence_class(¶m); + let v1 = self.virtregs.congruence_class(&arg); + dbg!( + " - set 0: {}\n - set 1: {}", + DisplayList(v0), + DisplayList(v1) + ); + let nodes = MergeNodes::new( + func, + preorder, + MergeNodes::new( + func, + preorder, + v0.iter().map(|&value| Node::value(value, 0, func)), + v1.iter().map(|&value| Node::value(value, 1, func)), + ), + self.vcopies.iter(func), + ); + + // Now push the values in order to the dominator forest. + // This gives us the closest dominating value def for each of the values. + self.forest.clear(); + self.values.clear(); + let ctx = self.liveness.context(&self.func.layout); + for node in nodes { + // Accumulate ordered values for the new vreg. + if node.is_value() { + self.values.push(node.value); + } + + // Push this value and get the nearest dominating def back. + let parent = match self.forest.push_node(node, func, domtree, preorder) { + None => { + if node.is_vcopy { + self.forest.pop_last(); + } + continue; + } + Some(n) => n, }; - // Union the EBB parameter with corresponding arguments on the predecessor branches, - // but make sure to isolate dropped values. - // - // Compare `union_pred_args()` which runs during phase 1. We don't need to check for - // special cases here since they have already been eliminated during phase 1. We - // already know that: - // - // 1. `arg` is not live-in to `ebb`. - // 2. `arg` is not a function argument on the stack. - for (pred_ebb, pred_inst) in self.cfg.pred_iter(ebb) { - let arg = self.func.dfg.inst_variable_args(pred_inst)[argnum]; - let arg_dropped = self.forest.is_dropped(arg); - - // We don't want to union dropped values with each other because we can't ensure - // that we are actually making progress -- the new virtual register of dropped - // values may have its own interferences and so on. - // - // TODO: Maintain a secondary dominator forest to keep track of dropped values that - // would be allowed to be unioned together. - if param_dropped || arg_dropped { - dbg!(" - {}#{}: {} isolated from {}", ebb, argnum, param, arg); - let new_arg = self.isolate_arg(pred_ebb, pred_inst, argnum, arg); - self.virtregs.union(param, new_arg); - } else { - self.virtregs.union(param, arg); + if node.is_vcopy { + // Vcopy nodes don't represent interference if they are copies of the parent value. + // In that case, the node must be removed because the parent value can still be + // live belong the vcopy. + if parent.is_vcopy || node.value == parent.value { + self.forest.pop_last(); + continue; } + + // Check if the parent value interferes with the virtual copy. + let inst = node.def.unwrap_inst(); + if node.set_id != parent.set_id && + self.liveness[parent.value].reaches_use(inst, node.ebb, ctx) + { + dbg!( + " - interference: {} overlaps vcopy at {}:{}", + parent, + node.ebb, + self.func.dfg.display_inst(inst, self.isa) + ); + return false; + } + + // Keep this vcopy on the stack. It will save us a few interference checks. + continue; + } + + // Parent vcopies never represent any interference. We only keep them on the stack to + // avoid an interference check against a value higher up. + if parent.is_vcopy { + continue; + } + + // Both node and parent are values, so check for interference. + debug_assert!(node.is_value() && parent.is_value()); + if node.set_id != parent.set_id && + self.liveness[parent.value].overlaps_def(node.def, node.ebb, ctx) + { + // The two values are interfering. + dbg!(" - interference: {} overlaps def of {}", parent, node.value); + return false; } } - // TODO: Get back the new vregs so they can be re-checked. - let old_len = self.worklist.len(); - self.virtregs.finish_union_find(Some(self.worklist)); - dbg!("-> new vregs {}", DisplayList(&self.worklist[old_len..])); + // The values vector should receive all values. + debug_assert_eq!(v0.len() + v1.len(), self.values.len()); + + // No interference found. + true } } @@ -542,81 +675,98 @@ impl<'a> Context<'a> { /// granularity. /// /// Values are pushed in dominator tree pre-order of their definitions, and for each value pushed, -/// `push_value` will return the nearest previously pushed value that dominates the definition. +/// `push_node` will return the nearest previously pushed value that dominates the definition. #[allow(dead_code)] struct DomForest { // Stack representing the rightmost edge of the dominator forest so far, ending in the last // element of `values`. // - // At all times, the EBB of each element in the stack dominates the EBB of the next one, and - // all elements dominating the end of `values` are on the stack. + // At all times, the EBB of each element in the stack dominates the EBB of the next one. stack: Vec, - - // The index into `stack` of the last dominating node returned by `push_value`. - last_dom: Option, - - // List of values that have been dropped from the forest because they were interfering with - // another member. - // - // This list is initially just appended to, then it sorted for quick member checks with - // `is_dropped()`. - dropped: Vec, } /// A node in the dominator forest. #[derive(Clone, Copy, Debug)] #[allow(dead_code)] struct Node { - value: Value, - /// The program point where `value` is defined. + /// The program point where the live range is defined. def: ExpandedProgramPoint, /// EBB containing `def`. ebb: Ebb, + /// Is this a virtual copy or a value? + is_vcopy: bool, + /// Set identifier. + set_id: u8, + /// For a value node: The value defined at `def`. + /// For a vcopy node: The relevant branch argument at `def`. + value: Value, } impl Node { - /// Create a node for `value`. - pub fn new(value: Value, func: &Function) -> Node { + /// Create a node representing `value`. + pub fn value(value: Value, set_id: u8, func: &Function) -> Node { let def = func.dfg.value_def(value).pp(); let ebb = func.layout.pp_ebb(def); - Node { value, def, ebb } + Node { + def, + ebb, + is_vcopy: false, + set_id, + value, + } + } + + /// Create a node representing a virtual copy. + pub fn vcopy(branch: Inst, value: Value, set_id: u8, func: &Function) -> Node { + let def = branch.into(); + let ebb = func.layout.pp_ebb(def); + Node { + def, + ebb, + is_vcopy: true, + set_id, + value, + } + } + + /// IF this a value node? + pub fn is_value(&self) -> bool { + !self.is_vcopy } } impl fmt::Display for Node { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - write!(f, "{}@{}", self.value, self.ebb) + if self.is_vcopy { + write!(f, "{}:vcopy({})@{}", self.set_id, self.value, self.ebb) + } else { + write!(f, "{}:{}@{}", self.set_id, self.value, self.ebb) + } } } impl DomForest { /// Create a new empty dominator forest. pub fn new() -> Self { - Self { - stack: Vec::new(), - last_dom: None, - dropped: Vec::new(), - } + Self { stack: Vec::new() } } /// Clear all data structures in this dominator forest. pub fn clear(&mut self) { self.stack.clear(); - self.last_dom = None; - self.dropped.clear(); } - /// Add a single value to the forest. + /// Add a single node to the forest. /// /// Update the stack so its dominance invariants are preserved. Detect a parent node on the /// stack which is the closest one dominating the new node and return it. - fn push_value( + fn push_node( &mut self, node: Node, func: &Function, domtree: &DominatorTree, preorder: &DominatorTreePreorder, - ) -> Option { + ) -> Option { // The stack contains the current sequence of dominating defs. Pop elements until we // find one whose EBB dominates `node.ebb`. while let Some(top) = self.stack.pop() { @@ -638,12 +788,12 @@ impl DomForest { // TODO: This search could be more efficient if we had access to // `domtree.last_dominator()`. Each call to `dominates()` here ends up walking up // the dominator tree starting from `node.ebb`. - self.last_dom = self.stack[0..self.stack.len() - 1].iter().rposition(|n| { + let last_dom = self.stack[0..self.stack.len() - 1].iter().rposition(|n| { domtree.dominates(n.def, node.def, &func.layout) }); // If there is a dominating parent value, return it for interference checking. - return self.last_dom.map(|pos| self.stack[pos].value); + return last_dom.map(|pos| self.stack[pos]); } } @@ -652,39 +802,309 @@ impl DomForest { None } - /// Drop `value` from the forest and add it to the `dropped` list. - /// - /// The value must be either the last value passed to `push_value` or the dominating value - /// returned from the call. - pub fn drop_value(&mut self, value: Value) { - self.dropped.push(value); - - // Are they dropping the last value pushed? - if self.stack.last().expect("Nothing pushed").value == value { - self.stack.pop(); - } else { - // Otherwise, they must be dropping the last dominator. - let pos = self.last_dom.take().expect("No last dominator"); - let node = self.stack.remove(pos); - assert_eq!(node.value, value, "Inconsistent value to drop_value"); - } - } - - /// Prepare the set of dropped values to be queried with `is_dropped()`. - /// - /// Returns the number of dropped values. - pub fn prepare_dropped(&mut self) -> usize { - self.stack.clear(); - if !self.dropped.is_empty() { - self.dropped.sort_unstable(); - dbg!("-> dropped {}", DisplayList(&self.dropped)); - } - self.dropped.len() - } - - /// Check if `value` was dropped. - pub fn is_dropped(&self, value: Value) -> bool { - debug_assert!(self.stack.is_empty(), "Call prepare_dropped first"); - self.dropped.binary_search(&value).is_ok() + pub fn pop_last(&mut self) { + self.stack.pop().expect("Stack is empty"); + } +} + +/// Virtual copies. +/// +/// When building a full virtual register at once, like phase 1 does with union-find, it is good +/// enough to check for interference between the values in the full virtual register like +/// `check_vreg()` does. However, in phase 2 we are doing pairwise merges of partial virtual +/// registers that don't represent the full transitive closure of the EBB argument-parameter +/// relation. This means that just checking for interference between values is inadequate. +/// +/// Example: +/// +/// v1 = iconst.i32 1 +/// brnz v10, ebb1(v1) +/// v2 = iconst.i32 2 +/// brnz v11, ebb1(v2) +/// return v1 +/// +/// ebb1(v3: i32): +/// v4 = iadd v3, v1 +/// +/// With just value interference checking, we could build the virtual register [v3, v1] since those +/// two values don't interfere. We can't merge v2 into this virtual register because v1 and v2 +/// interfere. However, we can't resolve that interference either by inserting a copy: +/// +/// v1 = iconst.i32 1 +/// brnz v10, ebb1(v1) +/// v2 = iconst.i32 2 +/// v20 = copy v2 <-- new value +/// brnz v11, ebb1(v20) +/// return v1 +/// +/// ebb1(v3: i32): +/// v4 = iadd v3, v1 +/// +/// The new value v20 still interferes with v1 because v1 is live across the "brnz v11" branch. We +/// shouldn't have placed v1 and v3 in the same virtual register to begin with. +/// +/// LLVM detects this form of interference by inserting copies in the predecessors of all phi +/// instructions, then attempting to delete the copies. This is quite expensive because it involves +/// creating a large number of copies and value. +/// +/// We'll detect this form of interference with *virtual copies*: Each EBB parameter value that +/// hasn't yet been fully merged with its EBB argument values is given a set of virtual copies at +/// the predecessors. Any candidate value to be merged is checked for interference against both the +/// virtual register and the virtual copies. +/// +/// In the general case, we're checking if two virtual registers can be merged, and both can +/// contain incomplete EBB parameter values with associated virtual copies. +/// +/// The `VirtualCopies` struct represents a set of incomplete parameters and their associated +/// virtual copies. Given two virtual registers, it can produce an ordered sequence of nodes +/// representing the virtual copies in both vregs. +struct VirtualCopies { + // Incomplete EBB parameters. These don't need to belong to the same virtual register. + params: Vec, + + // Set of `(branch, destination)` pairs. These are all the predecessor branches for the EBBs + // whose parameters can be found in `params`. + // + // Ordered by dominator tree pre-order of the branch instructions. + branches: Vec<(Inst, Ebb)>, + + // Filter for the currently active node iterator. + // + // An (ebb, set_id, num) entry means that branches to `ebb` are active in `set_id` with branch + // argument number `num`. + // + // This is ordered by EBB number for fast binary search. + filter: Vec<(Ebb, u8, usize)>, +} + +impl VirtualCopies { + /// Create an empty VirtualCopies struct. + pub fn new() -> VirtualCopies { + VirtualCopies { + params: Vec::new(), + branches: Vec::new(), + filter: Vec::new(), + } + } + + /// Clear all state. + pub fn clear(&mut self) { + self.params.clear(); + self.branches.clear(); + self.filter.clear(); + } + + /// Initialise virtual copies from the (interfering) values in a union-find virtual register + /// that is going to be broken up and reassembled iteratively. + /// + /// The values are assumed to be in domtree pre-order. + /// + /// This will extract the EBB parameter values and associate virtual copies all of them. + pub fn initialize( + &mut self, + values: &[Value], + func: &Function, + cfg: &ControlFlowGraph, + preorder: &DominatorTreePreorder, + ) { + self.clear(); + + let mut last_ebb = None; + for &val in values { + if let ir::ValueDef::Param(ebb, _) = func.dfg.value_def(val) { + self.params.push(val); + + // We may have multiple parameters from the same EBB, but we only need to collect + // predecessors once. Also verify the ordering of values. + if let Some(last) = last_ebb { + match preorder.pre_cmp_ebb(last, ebb) { + cmp::Ordering::Less => {} + cmp::Ordering::Equal => continue, + cmp::Ordering::Greater => panic!("values in wrong order"), + } + } + + // This EBB hasn't been seen before. + for (_, pred_inst) in cfg.pred_iter(ebb) { + self.branches.push((pred_inst, ebb)); + } + last_ebb = Some(ebb); + } + } + + // Reorder the predecessor branches as required by the dominator forest. + self.branches.sort_unstable_by(|&(a, _), &(b, _)| { + preorder.pre_cmp(a, b, &func.layout) + }); + } + + /// Get the next unmerged parameter value. + pub fn next_param(&self) -> Option { + self.params.last().cloned() + } + + /// Indicate that `param` is now fully merged. + pub fn merged_param(&mut self, param: Value, func: &Function) { + assert_eq!(self.params.pop(), Some(param)); + + // The domtree pre-order in `self.params` guarantees that all parameters defined at the + // same EBB will be adjacent. This means we can see when all parameters at an EBB have been + // merged. + // + // We don't care about the last parameter - when that is merged we are done. + let last = match self.params.last() { + None => return, + Some(x) => *x, + }; + let ebb = func.dfg.value_def(param).unwrap_ebb(); + if func.dfg.value_def(last).unwrap_ebb() == ebb { + // We're not done with `ebb` parameters yet. + return; + } + + // Alright, we know there are no remaining `ebb` parameters in `self.params`. This means we + // can get rid of the `ebb` predecessors in `self.branches`. We don't have to, the + // `VCopyIter` will just skip them, but this reduces its workload. + self.branches.retain(|&(_, dest)| dest != ebb); + } + + /// Set a filter for the virtual copy nodes we're generating. + /// + /// Only generate nodes for parameter values that are in the same congruence class as `reprs`. + /// Assign a set_id to each node corresponding to the index into `reprs` of the parameter's + /// congruence class. + pub fn set_filter( + &mut self, + reprs: [Value; 2], + func: &Function, + virtregs: &VirtRegs, + preorder: &DominatorTreePreorder, + ) { + self.filter.clear(); + + // Parameters in `self.params` are ordered according to the domtree per-order, and they are + // removed from the back once they are fully merged. This means we can stop looking for + // parameters once we're beyond the last one. + let last_param = *self.params.last().expect("No more parameters"); + let limit = func.dfg.value_def(last_param).unwrap_ebb(); + + for (set_id, repr) in reprs.iter().enumerate() { + let set_id = set_id as u8; + for &value in virtregs.congruence_class(repr) { + if let ir::ValueDef::Param(ebb, num) = func.dfg.value_def(value) { + if preorder.pre_cmp_ebb(ebb, limit) == cmp::Ordering::Greater { + // Stop once we're outside the bounds of `self.params`. + break; + } + self.filter.push((ebb, set_id, num)); + } + } + } + // We'll be using `binary_search_by` with the numerical EBB ordering. + self.filter.sort_unstable(); + } + + /// Look up the set_id and argument number for `ebb` in the current filter. + /// + /// Returns `None` if none of the currently active parameters are defined at `ebb`. Otherwise + /// returns `(set_id, argnum)` for an active parameter defined at `ebb`. + fn lookup(&self, ebb: Ebb) -> Option<(u8, usize)> { + self.filter + .binary_search_by(|&(e, _, _)| e.cmp(&ebb)) + .ok() + .map(|i| { + let t = self.filter[i]; + (t.1, t.2) + }) + } + + /// Get an iterator of dom-forest nodes corresponding to the current filter. + pub fn iter<'a>(&'a self, func: &'a Function) -> VCopyIter { + VCopyIter { + func, + vcopies: self, + branches: self.branches.iter(), + } + } +} + +/// Virtual copy iterator. +/// +/// This iterator produces dom-forest nodes corresponding to the current filter in the virtual +/// copies container. +struct VCopyIter<'a> { + func: &'a Function, + vcopies: &'a VirtualCopies, + branches: slice::Iter<'a, (Inst, Ebb)>, +} + +impl<'a> Iterator for VCopyIter<'a> { + type Item = Node; + + fn next(&mut self) -> Option { + while let Some(&(branch, dest)) = self.branches.next() { + if let Some((set_id, argnum)) = self.vcopies.lookup(dest) { + let arg = self.func.dfg.inst_variable_args(branch)[argnum]; + return Some(Node::vcopy(branch, arg, set_id, self.func)); + } + } + None + } +} + +/// Node-merging iterator. +/// +/// Given two ordered sequences of nodes, yield an ordered sequence containing all of them. +struct MergeNodes<'a, IA, IB> +where + IA: Iterator, + IB: Iterator, +{ + a: iter::Peekable, + b: iter::Peekable, + layout: &'a ir::Layout, + preorder: &'a DominatorTreePreorder, +} + +impl<'a, IA, IB> MergeNodes<'a, IA, IB> +where + IA: Iterator, + IB: Iterator, +{ + pub fn new(func: &'a Function, preorder: &'a DominatorTreePreorder, a: IA, b: IB) -> Self { + MergeNodes { + a: a.peekable(), + b: b.peekable(), + layout: &func.layout, + preorder, + } + } +} + +impl<'a, IA, IB> Iterator for MergeNodes<'a, IA, IB> +where + IA: Iterator, + IB: Iterator, +{ + type Item = Node; + + fn next(&mut self) -> Option { + let ord = match (self.a.peek(), self.b.peek()) { + (Some(a), Some(b)) => { + let layout = self.layout; + self.preorder.pre_cmp_ebb(a.ebb, b.ebb).then_with(|| { + layout.cmp(a.def, b.def) + }) + } + (Some(_), None) => cmp::Ordering::Less, + (None, Some(_)) => cmp::Ordering::Greater, + (None, None) => return None, + }; + // When the nodes compare equal, prefer the `a` side. + if ord != cmp::Ordering::Greater { + self.a.next() + } else { + self.b.next() + } } } diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index 9c3c6616cf..5116e8f02a 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -12,10 +12,11 @@ //! memory-to-memory copies when a spilled value is passed as an EBB argument. use dbg::DisplayList; +use dominator_tree::DominatorTreePreorder; use entity::{EntityList, ListPool}; use entity::{PrimaryMap, EntityMap, Keys}; use entity::EntityRef; -use ir::Value; +use ir::{Value, Function}; use packed_option::PackedOption; use ref_slice::ref_slice; use std::cmp::Ordering; @@ -112,22 +113,55 @@ impl VirtRegs { } } - /// Sort the values in `vreg` according to `compare`. - /// - /// If the ordering defined by `compare` is not total, value numbers are used as a last resort - /// tie-breaker. This makes it possible to use an unstable sorting algorithm which can be - /// faster because it doesn't allocate memory. + /// Sort the values in `vreg` according to the dominator tree pre-order. /// /// Returns the slice of sorted values which `values(vreg)` will also return from now on. - pub fn sort_values(&mut self, vreg: VirtReg, mut compare: F) -> &[Value] - where - F: FnMut(Value, Value) -> Ordering, - { + pub fn sort_values( + &mut self, + vreg: VirtReg, + func: &Function, + preorder: &DominatorTreePreorder, + ) -> &[Value] { let s = self.vregs[vreg].as_mut_slice(&mut self.pool); - s.sort_unstable_by(|&a, &b| compare(a, b).then(a.cmp(&b))); + s.sort_unstable_by(|&a, &b| preorder.pre_cmp_def(a, b, func)); s } + /// Insert a single value into a sorted virtual register. + /// + /// It is assumed that the virtual register containing `big` is already sorted by + /// `sort_values()`, and that `single` does not already belong to a virtual register. + /// + /// If `big` is not part of a virtual register, one will be created. + pub fn insert_single( + &mut self, + big: Value, + single: Value, + func: &Function, + preorder: &DominatorTreePreorder, + ) -> VirtReg { + assert_eq!(self.get(single), None, "Expected singleton {}", single); + + // Make sure `big` has a vreg. + let vreg = self.get(big).unwrap_or_else(|| { + let vr = self.alloc(); + self.vregs[vr].push(big, &mut self.pool); + self.value_vregs[big] = vr.into(); + vr + }); + + // Determine the insertion position for `single`. + let index = match self.values(vreg).binary_search_by( + |&v| preorder.pre_cmp_def(v, single, func), + ) { + Ok(_) => panic!("{} already in {}", single, vreg), + Err(i) => i, + }; + self.vregs[vreg].insert(index, single, &mut self.pool); + self.value_vregs[single] = vreg.into(); + vreg + } + /// Remove a virtual register. /// /// The values in `vreg` become singletons, and the virtual register number may be reused in From d56ce9e8bfa0eaf8e3af5c13d92d7b4ea0dcde5c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 24 Jan 2018 13:44:51 -0800 Subject: [PATCH 1484/3084] Optimize DomForest::push_node(). The dominator tree pre-order is defined at the EBB granularity, but we are looking for dominating nodes at the instruction level. This means that we sometimes need to look higher up the DomForest stack for a dominating node, using DominatorTree::dominates() instead of DominatorTreePreorder::dominates(). Each dominance check involves the domtree.last_dominator() function scanning up the dominator tree, starting from the new node that was pushed. We can eliminate this duplicate work by exposing the last_dominator() function to push_node(). As we are searching through nodes on the stack, maintain a last_dom program point representing the previous return value from last_dominator(). This way, we're only scanning the dominator tree once. --- lib/cretonne/src/dominator_tree.rs | 2 +- lib/cretonne/src/regalloc/coalescing.rs | 34 +++++++++++++++++-------- 2 files changed, 24 insertions(+), 12 deletions(-) diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 4d4e73d406..f7b7273681 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -138,7 +138,7 @@ impl DominatorTree { /// Find the last instruction in `a` that dominates `b`. /// If no instructions in `a` dominate `b`, return `None`. - fn last_dominator(&self, a: Ebb, b: B, layout: &Layout) -> Option + pub fn last_dominator(&self, a: Ebb, b: B, layout: &Layout) -> Option where B: Into, { diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 2471c74284..416783b6c0 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -779,21 +779,33 @@ impl DomForest { // not necessarily mean that `top.def` dominates `node.def`, though. The `top.def` // program point may be below the last branch in `top.ebb` that dominates // `node.def`. - debug_assert!(domtree.dominates(top.ebb, node.def, &func.layout)); - + // // We do know, though, that if there is a nearest value dominating `node.def`, it // will be on the stack. We just need to find the last stack entry that actually // dominates. - // - // TODO: This search could be more efficient if we had access to - // `domtree.last_dominator()`. Each call to `dominates()` here ends up walking up - // the dominator tree starting from `node.ebb`. - let last_dom = self.stack[0..self.stack.len() - 1].iter().rposition(|n| { - domtree.dominates(n.def, node.def, &func.layout) - }); + let mut last_dom = node.def; + for &n in self.stack.iter().rev().skip(1) { + // If the node is defined at the EBB header, it does in fact dominate + // everything else pushed on the stack. + let def_inst = match n.def { + ExpandedProgramPoint::Ebb(_) => return Some(n), + ExpandedProgramPoint::Inst(i) => i, + }; - // If there is a dominating parent value, return it for interference checking. - return last_dom.map(|pos| self.stack[pos]); + // We need to find the last program point in `n.ebb` to dominate `node.def`. + last_dom = match domtree.last_dominator(n.ebb, last_dom, &func.layout) { + None => n.ebb.into(), + Some(inst) => { + if func.layout.cmp(def_inst, inst) != cmp::Ordering::Greater { + return Some(n); + } + inst.into() + } + }; + } + + // No real dominator found on the stack. + return None; } } From ef2640d8a49bcabc1e29191ddae73dd3ad6a76e3 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 25 Jan 2018 14:07:50 -0800 Subject: [PATCH 1485/3084] Add information about SpiderMonkey and rustc plans. --- README.rst | 37 ++++++++++--------------- cranelift/media/spidermonkey1.png | Bin 0 -> 127604 bytes cranelift/media/spidermonkey2.png | Bin 0 -> 123766 bytes rustc.rst | 16 +++++++++++ spidermonkey.rst | 44 ++++++++++++++++++++++++++++++ 5 files changed, 75 insertions(+), 22 deletions(-) create mode 100644 cranelift/media/spidermonkey1.png create mode 100644 cranelift/media/spidermonkey2.png create mode 100644 rustc.rst create mode 100644 spidermonkey.rst diff --git a/README.rst b/README.rst index d62e5b2777..31e674350b 100644 --- a/README.rst +++ b/README.rst @@ -2,8 +2,9 @@ Cretonne Code Generator ======================= -Cretonne is a low-level retargetable code generator. It translates a -target-independent intermediate language into executable machine code. +Cretonne is a low-level retargetable code generator. It translates a `target-independent +intermediate language `_ into executable +machine code. *This is a work in progress that is not yet functional.* @@ -15,28 +16,20 @@ target-independent intermediate language into executable machine code. :target: https://travis-ci.org/stoklund/cretonne :alt: Build Status -Cretonne is designed to be a code generator for WebAssembly with these design -goals: +For more information, see `the documentation +`_. -Portable semantics - As far as possible, Cretonne's input language has well-defined semantics - that are the same on all target architectures. The semantics are usually - the same as WebAssembly's. -Fast sandbox verification - Cretonne's input language has a safe subset for sandboxed code. No advanced - analysis is required to verify memory safety as long as only the safe - subset is used. The safe subset is expressive enough to implement - WebAssembly. -Scalable performance - Cretonne can be configured to generate code as quickly as possible, or it - can generate very good code at the cost of slower compile times. -Predictable performance - When optimizing, Cretonne focuses on adapting the target-independent IL to - the quirks of the target architecture. There are no advanced optimizations - that sometimes work, sometimes fail. +Planned uses +------------ -For more information, see -`the documentation `_. +Cretonne is designed to be a code generator for WebAssembly, but it is general enough to be useful +elsewhere too. The initial planned uses that affected its design are: + +1. `WebAssembly compiler for the SpiderMonkey engine in Firefox + `_. +2. `Backend for the IonMonkey JavaScript JIT compiler in Firefox + `_. +3. `Debug build backend for the Rust compiler `_. Building Cretonne ----------------- diff --git a/cranelift/media/spidermonkey1.png b/cranelift/media/spidermonkey1.png new file mode 100644 index 0000000000000000000000000000000000000000..0304a0a4f3422f8df6b69ecab7a673e21aa5b827 GIT binary patch literal 127604 zcmc$`cRbbo8$XUzW<&@X5wf#6W+B<5BJ(7h4%vI%h3sTy%M79HeY))&duE616~~_6 z*Ez@i`E-9Czu))YZ;yvZ&hdV~-mllSujh4L?_doz1!4jk0xT>nV#P;tT3A?D{IIZa z+VQS{Z=ekP>EJ(XM=ga1SVi5pmca-7SC4cZv9KsvP(Rp;T6cbf%L%QXJac-cssb^w zgYg)f+PyI2af7`AS7TvGxIw_LFf%7(MmN|?TSthSBoq1y2>2cKH7^q*`VuD_Nv3D2 z8jP}b4rYu(Jc2y;n4}0885t!UOwA!$at|+V2cIOFES;QQL3nvxU0r!x1$gWnEO`0E z#Kd^-@$>TYbAv0m9o=o6jNQ0x9hotU{IiannWKq=)hj0}J6lH7y2dZ;oSh_@m{1S; z&p*sQovh6N`y^Y(i(`QU@}mC2%g1w%_djccTP0B6LbS{r?Or;gmVat%i|haV?tib>u(Wfs1GILq zGEua3GIIbAySV$G4gUXMaj}*JFN$1$kPpKf^tXUHQUnsb|6#NgL12H;9V{#vEJeBd zPu#HAQt&69${)5VXp+S;3dyS8SNz7!a)Y>uTR~Zd%aNO9N@|^bEJWlCZbhi+5G(Sw; z_g+juCMVcmyNZn`gM~xthlR_C_5b=;8E)L+PxCp#^|@@j<*E$Vy8{2SE8^SlY#!xF zR%-05q#my=i{Uc%p5fjkUp*$GCSOHF_oFs1m?YB{8M-c|LA(vWKJHFkqZjDrE6eMB zX?eX#^v_d%{*4!Rc{`b;2g&bReu6kg*6+xPw7b&O*RV;s9}!{W?L5H=3m#SXc?WO) z@S!V+6Ga|KluTPh=(;JEmcGc=wh!;|=Q!@V$k+ebA*r8CJ)Xo2VPb=x?e^J#5oEpK z-j4OThCQ0L8gW=AFLdQh(RB8wmrLhvYta_L%JW|Jq+XoI=L>$>buv+2QUPqE3+ituO~XA@G7UD}@DG|cXx;Jmx@idN`Py;Hb)9tLHK z#+-~56>CehNN!$+trGLRz}@A6w)GMOk8KjH!d^D$S{Gs=$!_ClZ(aZ0RJ|C^ryH_G z_y3V4|I`Q1TS6#oM@!5u>sGz`m?Z9MYeesJVC=d-=~_FFh>vgj+F^0#4| z#J+m#=iaCVL*=+_tr(F+wxvDsf8{w{3rFk7qVvN{j(XZgQSAoAb;Qs$wszMcZD!Tz ztR7E~#13<9k**52?BCGx%ZWG5{J}ZK-e)+-g)YZvHg%VHs84m0w6Bl0neu7C<9dUi zX7ej5UBal-UWvpgU-<2BAAZ~{iXlKVb6qh#yj9*Jy2bo`1!SINdt0BEx&B7=OjA%T zJ~?X@pGj+#&3$IRvaPShFJrZ09L|g zG|g;p0qtM04{oFMoG}#0BPDe|`a?P!ndOZA*WaQq`=33seD!T5T6!1}V06^xyiqow zc8I^{f0h0wl1pDd_ZXwpaQ`btSU81`o^Q>zY`h^?Yqukb?{APkEK_%LZwaOTt5f~EktUgk zv;2p4T=YJ>SOGs@&JJR@Td0yw+uEXXE8GY>OEF(jn>8o+Pkvd9t=r{CKT&^89S` zyk>UfIGN>k)iW|p9kvYJ0R3uV&7nKHti0j@B1E@DdsP$o zTf%ShF{!Wui}xQ%@~>dJ=eVFEVml#P`TXy{swN>^OQ%M09(@Fi`j<@`)H#-W3zTPP zn9ydZj0U^4S=*DT>?(%S(K46XT0&~f$U8##bv}IWh~e$4up4pjqk4C_vgw+5)oL+( z3Rbh66Cn8KNl7hra6&1$9ga8QaL=FqS7&)9Hps92elh-c!dBafD7>YwARJc;^FRvl zzzPooK}z%jG%D%Xb1z!BjY?;eW6i?Y^2T0l5b)y9^(^&ePQofD1GjSdbAJ8xUmIOr5qjZ!u;7LH1nb<*0&e6GmWY@dn)u zKIdQE6~tEtWj z_V6Z~IEH#ij@&Ku#`O|``YB7((_myol~p6x{_BeN_^*-`+DzKdx02F37*^-^jiq46 zImJwTXK$Y1p*My(i4%=Ki*j6*tzE8MMFg8F9+VsbLyK@^^-B|kMErg?!2wS|1a~?>F_PnESA6Z2V7#K)2%6ST6h*2 zF}E~?E&3P66*T~RJib5L=YW>bPtyK0;1IMQn@nqGXBG2j+1w7FJX;#INL!aWnIQ>M zeDlZl8GE;Ivwqkd&#e~K@~o|zOKm5|*S?n9S;4w;AHCX=1fWpLzRL1-429IvL+roq zDV?MwS`M$Q86D~=-tqN|zgoA0eD7+~8fO1E*E-1&bLtCq@nH4;YSLC?HINgK0$pJi zchjwd*biAxYcFL2^Zplxxhe9%{*|V()%&Bk=U;UFt^kj9?W-<|_0DHV&Z5CEQHp6& z)4_Vy#7{~*yADTG4RefFy9qh3CvtmY&p zyCWeFXKZqeCNJGVooq*NnGk(4r&yT;d(Bn`0jvOIYe+$%GofhGheY1Fo(Q9EVjHQs zFoB8V>GQLD!T%Ctk6Zr0EKL_yPI+u;ztY2Y3Xiv_YxdX`yM1S8<=quyGz~2+z?0Tb zb}Yp)G)$5Sw%@Xbj=*d6(pVFRAhik&g(p$PO)uMYjm6i?qH7mZJq?PD%Evx@zT@>@ ztAmYKChZUGTTP1V)M!_#r1)Cr4F)lOmESW^&-=v@>7)T@+I6EmrC87VR5bl%M(|^9 zQnelT!zkVr;%Ws{yp|j7FssUI8KnwYAlV8C!zWA>wP$KgBgMmP_9FOFm0@_Os1T9t zwDi$vt%)O`uYmcH9L5or0`uuIjLZ_jNS_c6ORM~-VdoVSpQo`;ZLms|UYy%B3_s9q z3LZjH=Wzlu?kpT6?{bTGS_`XguaI6QRNtKA76&6}j+kVC*9yV&4&jA&Ay7C$4 za4#UkYhCxuk<+v{{r*Ou0)w+nQJ(U@qGK7xJ727aI7HljKNs;mJGpllTEM)h$K+8S z^$oMM8Vle;o_d@Q#^rG1^0&6kdFjIU$kHxnIr?>^UVai9%PNdZNAF^OagA^yfKAhnk-q*eoEv5uZEQ(|OEj?u&*=2YG#xtHg${b5ZW85Lzw zhl((A#@&cuzB5I9gqS*`H24*oRsZBJGBp0~$c5^UvO$6IYY1*2s00|^FS(%>KYzQ- zh+h23ebnNA&$IJBEyl@rxvcRLZ){vg??YC^Eitk=bqbQ`&ws2d8SN+a;r+%f(ScH= zBCt->7uKl{SSRtLt$s0#b$Tz;T6I@kMYX_>gAZ%eRIqEB^W`}RPe~u*|6#p>s8Hi7 zEq7-ZncYy}$4E~)3_pfuU&+vB{Mx`Pj9SRx2;sX)YLG5S^Jp*RScZ_SecoZ?J6oDlq}0); zf%A3l%87p1m-~&W(|;A_D56dTiAPg>xEgnf8Y6l?iP0yask?ZXq`yqZ&AXmqYy6z` zZz>Ysi*``^eNGqjt^e{X=1M?3N@XNtzJ*Y5PW1SmA>}rMNEkLAM$)G|V(!0*!&|@e zUVJgxabnVSdY#_=S5)21Vyupl;*;TGD7@~r0R-ecE+E!FJ>Q(=F5vG1u(rRn$ga(kCZBzo&t4Iw@C5D}AB zB2>1?V(YPAJH5U)W>YLja~(Y66qsfpXIZ0GT-@=)t6FvkrmFqUhxJUl<84$P%j+zg zAgg&gQtNu#`a9U83<=v8r)4cvpotq0tp@p?;ykPyEXcIWk{&CCN-?L8hASsBM9F6`NrdM*SPqACpAvQpY`33!Ab^W){nz+6M2x>^(nGYvdTY$zKstN0zx1_baD;JdxtPvesqm2ZDL`+}DM|XMh_^XS@Dr;Nw8Q2O0ShMuQ}y zF^Vp*+`DQ{?sk4W4Sh0?I%Gzf1HSL^_;g5^bj@NQt?gY`o0Z)%Vap&w5OVdp<7&0a zjGA-`laIe7lKGITj^M{?5qb8v5-&$H{~X$g$n2B3t$l4vj)A)=R+U>9mJd6iV1dks z;T?9z6$^=@P2U;=2q%4x*B6Vb7ienf;1Re+ZZns2i#r}vpPwDCKD#7;q0$#B0hrB= z+fO9p3wBw+!-41(uo>ps?-b~H@^tF_bo#s+^@m{3E7xv)ICm3>dIU1mB_|y7w7hon z=+gyWx@ZdP*sMh+3bfTuRC_+(5qnrD^MXaoGceU<%*yufzb>nJQcIs5%&9?%zdy@z zv^RCCTFEb%NcG&%9Waq}yBK0m!*IY7{Y^#ohA1ZBr9S^ZI|z2w2q z*TEe7)4sm0qh|5d;UZI_QMOS)sN2@RG+lG27I*n9j_eee%*#z(DeL)EZH9=eF7}$| z-00veF3YL_3Htd--4=qT0R;AXr2lnMWfK0IKKr#B^YKugGRNcfO5Qj48253C|FUA# z+!i#PhmVPP*v36^JjTzup$NeG~jO>TVrn;Q3*X z?}83tfR)vi{NE7Z>cDNy6$zq7gUed_-zd!X@Ht3P=~sFlu6B_@f==>6SJL7_Pg)uk zchr>AfNIL#eC#8>f?o(Kn8_ZHt2<-GNc&;Ef*Ri->*$7I{7L=kBkjtwqtzfq`5BKr z7A^@}1TcK@i)0IVtP-z&%H(qYO7+_Q-Uzd&3m~M%X0L+`ZXc}{yAo3IbRzpl)LvgD zanXmB$BvsP!6pEHw&bz9606ym^u0&w_~ont6L$*KbG@99nvz@Zmu<~5tBB3;=abaV zw8Ksd10AC00e?NlIa*OV^*GsTn&jr*V z%UKy-R`@k2tJz6RBlzhP5hudSj;KAHR)l9UHg_WbX+eJoB3Jdpy^o)`eAU#<&ps2v z!Hz;Y5~RF6wD$~{kI1z5QZ~JEKS`dkD1FFY`5o!AlPU4oJ~kQHP>R@@o@*Va2W`^b zhg7%aVI_-ZXT$n-DM=FU4&9vDEC_lr2h)tt)B@(ekgK}qYZlT7_a^<>qb$W}vUzuA z<8InS0tdi5)*}68&Y=K_j^e!pWNY@s-&Yxuk*#fVhQm~cQKTXdwTWy;nrFC zh!JO~!#6+d;O(=$O8ESyrnmIL-y{xGj4@TVzOLK#U)fzZ>-Sk#-Weg+?R>=n_zLel zKH6E1K3>s*C+dHFAwN6i@$iM=g1Ya~gi{m_MG)W)hz(@tx?DK2ukkS}V76aIe8@y6O*RLla;s z0$7KR!43?E?Fw3}A;dCYx81(8#Agk&$V25TLKKZ`dNH?zVcBn zlsd*2x8KHP!oDqJy!v>fcG2{n^_$S+-8ShnzS6{p+g+by2uVMY`ybeqsu^>tDKqY6omq%3IF5dowVq%4K&*XFU4})C2)p{TyAjSik9QXKlnHY{I1@ z8Gg%d*1q_0tD}9>{d5|ft#by3c*BQ#)T36W#q2g%92VC*^ps*^H&&Zb(Z~{we{QM= z2VCfb2z@qXR^e2cP+Wz-vSYlVOTzKVqI8Hfpq z!9BqeUgJw;DSB&lAG-`4yM*i=nr0&x~CcRD33P@qWg(goSNmb=bOr6F#j_U22+$+CcT1D z+itvOI7z>Z^r?3AD_b|HN=dfufSl(7d$nS>=Qc3V>d4o6pYgTK5u%qSV&f}?%ox{b zg&&~~R{cOO8&}WJv!ZrFP*=Q#GgO4t&yiTh%U#~z#NhGac+>Y>-!B&?Q}^|DF3YIT z`Cjw+ozut?ZfOFM&)1TD&kp6?Q9)nZDF58lx0ouZ{$Hc<88fO~OQA!#+=C~njlnL# z{uQm_Tg~K-uG8L?X0W%Lv3(PJ<95&|=UNuiW0Hi0>Fy;`eRU+*r;{>prYc zany5W9ggeqf~iy01TpxC=JLYtP(i)yHGND!sbhv|=#S^C^+ z>C_&fI~3F*W#ne@so#<|^q^-#&RNmhI!x zHL}8OhL5a8LHdzw{6!I2>v!ydEK!-kTAvjuCGkx)&l%mcTimK!`WS`f&yV|3g2-#s ztX~`Jd&J`b%2!{pEt`fYo?~$!UsZtuO5Oc*y-V+m-d#N(+@67+%u<=zAMw5j=}tbJ z+HZ*WNHfY0;@y3kc1|WF1XyvtE!g)^^=!aE;D~IPQ;o0o9?x8RE}+c`A_hc`{B`9B z<8@Dl;q2catC~qBlGW{IXh8aF?!MRtpaR&WnC%lKZh;#s$og}0&dqs8lZ=OcH+Udyk#UrAg;se9hsbAeU|q0iKFsuCO2JuVDA)Q-sE%!}5w`MQFruaAb0C%K)Q;;&-C+5I6`fL_}&)rxdf6%^HU9LBQ- z@7$Y(J9LcDS3QvektxhZNYW9`&jmj*>?zmm=A7s*zNI|5(6h4hFh7HPKh=9r zcM(om!G%U2fJKXAL0L5YhCsFPk%L$vV=c`eX@4&^fQW>(zr4O{qS}$%Dw}BxauBC) zYmdrr9+w_Qx=(=0V4TM~h*IN@k4E4dCv9wbBlXzr#PkxP_Uk!Gwo1)O6w5pfT`LT` zs2pgC2gho>&B)#Wa)RI~r?B{Gve?|6VCr|PcZB9dkj zR3^7@R}`@>X-HaW0X(+X3}=fXz`he|8OUmO zauUQj9|`rNELqN$)22-tihp3@^)rOC=#>vAxz0)*9pW5V{aOSsd?MvlBYwB9{M?DK z$HqI^YxX|XHPM>6T8ou1<$HbJ{CH;dckvNJQ%7gte%<>%2_o-cH*GnB(&-n4SyAk( zQ`Y^SyDRf2@o<)KY3RzFxrgxgqStvTqOkJ}46S(gO@ADs0?_=W6L)!Q=u&=PX5(=U%?3|{AxvZ+mFHVJ z<9LjuSNkrVxTG7Gk;YwbwY#kM)+V(u^0EmPZ9oZ;%+b&L43vR9u+IKn^RW5++}hTp zRQ}XZzH2zUcix>*KbEvf=R*0Hr=-6vTB~@ymz4v=LB~ScgQ_u^JPK}z%rj`DWlL~1 zP<0a9wI5$8Rd-LTxq_YQ*Y@=T@APx^e{Oi@y?yE~cSynJQ_JrP?iGhCG$8eJ-{|r; zq2hm^%c0ZbQqYJn0n{1-@VN57Sk+Ts{$01(gX6&e=E=%6$WnUq?Zl%8IuG--)E*|d z4Qb|`7*;P-mN?1=66ICBMFC5O?(?w$CqAJxi#_=cAiKA}9e$AwrVrr>%G&i|cYhqn zUx_#JV1FDRf((SHW^?*pC8qfWP@bLu`Ru?`T{-#XvoLswlwhz9R=|;&=0{?crE3ku zYuDt_q@>0<`liLeXuU}4xRrlPIg(fm#65OrCfK#MXJ^NonOH>CsC_#V zwAJIVKLH<0GeI|}s!r2G2)}{LsKJJF({Ba${R^NxN2I^6B8Lrn*DI%uw(7O=%MA4_ zJQcUfLW29&pq>sqADqrwLS`of3Xi+2G~#Eb#;3FXA_dQF&}O zo~6-E8b1`&d7?(L734Dl_4Uq}22OR>FYBI_nv+sDX+dvVt?DAam0pQz7zNsT2D|Y=i03&s1c3^3F%gx#GU<4o3G+2hz@dHEf03{fZS{ zmo(_egDs^zHGYmfJ&a#-+vXy$_|No+ZV(>VcpU?C@ZM|VQbm1z^rxPk)Oua zE+pDeo*Erok}TkO-FXq>X|rjBPQBmbYew~VR7^M^PB`f5pACehd(?nRwe}N(vElNq zj?@r>?F5d)A36Oc`QJzAyb7!s+)=fMG6DZGl1nCYVHG~1Ju~b=@bKc6R_3jc%EWT z`^xnu-(F`3F&#nYgE>+kU{al z$`&NI?y|j94Zj1L&AfMS)apUwn&)FeY^*#5k}C!avWvQe$GQ6I++v^1G6Con1%Swe zl6|V2Vyd^9Z4N#}5;XB3mNEq{v*=}W)2T8%w{+GRzCSW^n%$`X@~eCvf~D9>-dVhQ z;dgr$$Hn!JxasmUcy=?p5m2A!1 zps~vGrauLzb`>zTV$}3Im%C6c9q($&2p$y}cP^jM4$;}oe|)7|ws7;J-uIoDYa<*>32>iWxWJ?&qm{b9`0zb*suADkzPmSud$r zE^w{4^Ip}rIjT`2f8F~P=Wf=0On;1Cf=uRa910IREWI+HE9RZd4_tno<>k_Alf#H2 zR9Xi2-bAHAF8bzUphGRyy>KExsUm_?=V58FNRnOYv4S^B)@nliq^=Ea(9F0OZxN)s zDMWo^9@aUO<3qZ4IB;yKHyXfsa#@y(oU1v$}-cSELne$%g zwHYCM9~KWPZ~Bz@4V21P07Q^`QeitbRhYb`(0VgP+;KLTP1|q;B=X;z3C~b{8%ee` zd7o}k%Qdb`WF0)s@DnWMl@BkveII()(7uJj{mRx%^ca(R!J7wcMjr)lD4i3lzxgsZ8vmEvpNKxUu1SuPhw|7Q>v_M)ScmSIIgM46 z>8jtMk1pMI*76WIbRiQt!a|b9qEKAHhIxdDq`ic?XG0b$*HJuHWx0|b=Aa0&at{J*@8G=G z__CUMEJcaB7EyCOGHt`H9b+NDBms}C2F$gQ?~ z4(%nPs)?D)ITF+*cb{ZbLOzjW`YlK;&2YN<^~3xt+(A5Iedg8pJ(_K?2PaIYSI-EU zjkJ>A)`1^ARc`EI#F{3$cL(K|mT>%kU}eP$yDapeY7AwLZ>v#3F+4o4rS#Xsdp6CZVe>RPciU0h7RPJ5Ap%t61f# zp5p8vAA#fTd0yYqWe^wa*DXrgg4HPtN6!K9v>^r1PqqzTV28=KfXqRz$8RY@Wk7c( zThvrhlwP=%y2O@T2I8h4ndsXK%W~Z^XNt_#(B#*_)*O2^RC3VXbt3H3+w;W0!aUY=s!kXDsV#g$J)0dv$TkUg%{#XhddvQslu7s_QjmUecENss#tcExhoON1R~~bULsGIRI{n za3hxzY9w9q0pUr_Idv}%9wWsCqOF=f-Ea-IiG6uK+}ZH5y2XlP*Rw+??3$QSz+C@Ncs zk^mOy{W1xbm}h!X-$JQ>L+(n2wmm<`*(&rrKVPiI0(h-pPZv;JkxRyOf61iB;t4*s z%gdRr_uO}3IO+|W0NDR#1SkbH7wWO^pb+r>w@(zX8iux*L{PTn`rGh(kjev4OL0K~ z@HiUPNMEkfZo#u}SEZi!)$y8W)t!z5Z@xNP?d$fqAY$H{c~12p3r{xv5ceLIP@0&~ zgY}Rp1!(a5Dz_3B& z`7Vl2EB+dBk1^~!B@1wt!EK=XO}dN&vK>m?Spea9tS?HRyZdlDt$x$iht1Ef_pj?E zG89t+`nLq;Rwbi?{pc-LR`pvrh3X(xT9szj!}GUgd*|Y#`0{4Ewdu)F$ac$k2)iou z)frey^UF_EJoBK+K!$gsxNf_{VU)jk6GTFX^&tMYy&YdMs!27&*|>!gn4l0BB0}N#>%)@~Bt`745TYqCfrQ+$ZU(XG1tS zrmmCt%)rw;6GHqkeztP4~U$->~+(T^mX#VBGG8Qzw_oe3aVoTiQO^OJAKji&Rz9r98D$=I&~jevvs;jH%w2q_mzODElB1IF zLh&+Gzv}7Si@As}0|2x>`iMs-Q@5OxL^+#{)3t$6WJ~$A!*s`~)S|})Rg0{hkJE4b z_1+kE%(}4&uz7>e%|3gjY0eN{b{AC1s*TRofJwLDIRFI9*s5dYndAXR7S7(Yb+@u zDuxMCy643Xe??CUkH4HN(^BA>%HIG5Xos0Y$(d6tV{S7aKra2NGSK++;F*6W=MTVL zOBd*le4%!lWx4iJ8I8OJyW;&J$3APybxp2RGFVf_X@FT+-%Q7*$4G5*s^&p?RL8)` zBfaJ5c{LERj8xi9H=CXMv@>o4z`t)Rz!iI-DfzGz`aht^=WXxsR0AQ#ZC;?x?DP;Fl|TGjn$93h+) z==*$_K=&iBlk}&OkYn{l?AGz_{GOu^3I*SHnbtvk%=5e#QNHQD=@}sEFM{_iei|3$ z{JL(ZbN>=bQ+5RX3=T&>iuHh>d(!dKbXU-ZOUFA*Rj8|LO$y0iJ^)&EVrx>y)yWV(=aZrZmxPNd-Gg1o17=FQ&eyA$fz4M!8d? zz+1DwK^{wO&ITYQr@NpM1DCB;tkg@E8FR~>?($fDK7G(CqSYo9UVSDTGZ7@U^Kk+d z=qLMD$07Bbau#-ZQ~3d^-EdCIB4FqCcX2XLzD-|-ym4ITm-C;@_QldC;sV7ykL(Kf zHXHhVgc9w(N7!G-ac0z5pMFS}pM{4vD94zZIks6eO`MsteRB0vwUqbKjc^ills+viVo>RAxsv8(3Pex|+Ki|q3!ac)m{mmjAc zJ&}iRYN?%e)O42cVa1-5|76021j?M8SD8Q_DvX85lCsdz?lS91W9UUp4%%i5iWHl~&PNCJZ zA3-NRED2!g4Q7VJM+@)WIA6|%i?~Fys&*_!_IxOjDEw*TDS6)V)CW=Hz~w~m50#HO zujkr7E@)V4pwo<1=x1qglMR!)Y(16Dl2 z*D|`bs>2S9`W#ur+!S%eEOM+i$Gv4QDWraJ zwF15S9p8#Dqm@;&iZB8GD@!UVN=9#THIW6(+q_d|q#jEjgSM|+riY&nGL+)I zOA=DQe7?tQi_>`f`1>mTI-T9ajKL(DQU>q+Mk zX8@4|AoUaGy#o{?=_gq(>Al?ok-qwwtDMEc0dLgy@xf}?M^x&>3qdNh#%eo2(jF01 zh{190+86bJo`pq}Jw?@Idn7?&O)W@^1VpAVP%c&rvQ;uJp3jcymh7GG1>k%N)ILU9 z#1j0;(Z~6T>b&mp)@>~RVYYPm`N4_s&dRW_ZOT+5$kHK>y~Xq{=b(a`vJw-KA-H_? z=&gLlR}BW0gQrW99x8*%Eg-L;^4WE2&FE(1n*XUEa&1H}3d`gLrxeB5U=c2e8$wCu zRBO-TdN;J$-tqGk{f5!6%?t4sUoGhm(s7UCef_TS@(9JH-qVJdN?k?JX!3ddl`Ks1 zpC3eGwg&<+qOYkE?(89_zGlG|f@Ndb1kY7EW@)GG!#$H?0%*A?epAj0C zvb+OY%eUYS4X&PfzytV5=d2mk-7wOtc&U4n<

Y`mD;?{Z{wvt!u<5f|ZR1=~B} zv+V40Z-wQ5kYpUt*8IGY{_*|mhg^nySoJrx-}k5g<~K$^>;gEIVPrzDBwu+QH2!XDKd<(6J_PqHx2z)F_e@MLlN2%H1i6wgys)p-Jqn@UIGJ{Grwv6 zP3wPm21CNWp)x~MZ_}U=3XHJ@V}5G&51DYs5-^RF-QHR27wlC+b+Y1ns$|c{jN|Yi z-e>JJr`n;EeeQXO(${DbO!|$YSb8q`yA3Wnh|AIt-}PBiKfmh-vsCaU(6wTz>4oP+ zgbjv>++&TJ~R`Y?^=vGisv=2xZs~o$<+6v3Hb%eK$S6fV&02aYU(S}=fsH~O&BoE z0lL`l#a1z8Y6x-qJ-PqPuQ50TmB%X4+($Pg04YO|Ju?XxGPuhp;`CR*hqx z4Q$#T0_YmU@@1oDwFY3xM04u8TBb`kaI#zLwMHP>=R21$)F%ZJcWfFI3=^2)PMG=p zR7xq{zCA0R#HRe+A;ey*Bq1K|$sUu*8yiSOJsbW1HBB*J=}b_Rz#nNuCvzJ!aGatJ zMw*ame3Ea_O&J~dreUF*NES=Nsu69(>J)Uj#S3O(^1@L$Z70y5jp5Tzh1u4j!}Il1 z@g`3;_0(JcI}dT)|3=E8RGPfjD|UB+k$)@!BtWLat?k1Pc9M7>42(5|W)l>g#xSv- zwL2O*^Ejpoa{qF3z_Y}|j-%Y6L*x`dEH)gf4=V0m{g;^U7~nu`R3MJO&KBn0|0P?@ zlaA&*s#yV?MV5hU5x5DuNK9%9&guj~@>Js#lx$ zI9KlI)FPOtISm+9T$ zsfwu!utCGYfJp0VjAqFuDlhkgl+X#OOu9K*%6LlPAGld}IP-Ip!Bj(s?%$Nc)BU{L z_z=^(n*i*dFRI_u3!Ozlq5qg*EF3V#h?!+N)#r72o*RJF?Jc;Zg^QW+Kn{{S$`$nGyY4z+QIDhzvp@- z6a1V^mfNFhTiU4H_$Kv?vnl?Z3L-@f?rbwIUqPR7FaR}V;~BcfBdinVU7)|b(D+J< z8-bM-YP?YOyX+u9JM?6<)S?!22}}Q{5{y2RKRuYJ{p$3q{dBAiwi|h<#3{P7fbp>DzPE)D5(L*x+t=HRcQZeiZ(a*?}x0kI? z$FSmuKVTugk7{tnyECV5iPisRFid*JU53DJX5Q84$m4r0C{iwj%biEGmJMc9D96j#?*MvBS-sjL3rC0!duV%x4)JkcmAn+cFxo zjMB)aH&-3lL+8kiI(tnP&+is}A(H-x{m4-T`IYekD_gsHWCC&9gG0%FcDU%XT; z;Z&5F!JDYMIzU9pSjo|6y6~AYZP(v*b54rzpM6Tpb=O^LY54Jeo9A@&h8H11hkkxY z@^X-oq0Gg2!k8a$T(lB*5;5jqm09Bj@Xsw$HwL{lm^@W(3DbrjN-0-xT3`_ne*JQ& zZNIi!({H=IJ^p0{Oi6yDb7;S93p}^+zoR_h^_Oe<-LsD$s?}^c@tU7R@$$_@9>(#c z=owGGHd?=p?)8yLuLq{4Vg#8{jxjaQaLig;lDyUvFC}`*&phiCHff-_6-c38<#2@& z@%cfguK)JjmAgfX`PpgnO_gr!RUk!}^#1*t!IVYAA9DbVk@Hu(onBm>wW_ojKEW+@ z1~*bszZNc$BOB!5i-vrrgJNYfam_1%DCFPYGzYq38=knukLx5&S2>25``p40s zJW&}R_WIms*tC}R*~DtCb4I*QT^6-x3?*csa$b2P?rvQK)s8g@Hm1Z?s0wydo<>rL zj)VOd$UZ$c7>u(z!s~SjxHsL6Rbn%v@H$Z;hdO{PWk9YZnUFBJ>EmUNq6wPc`Nco) z@FE*S1o=>t-AUkOFdtO@_1*!Pf()e4Y5c}qq|@`|#{S@J24#zD@*(xJIY0G9J2Ss* z$QSRCV#0*pF`&Hq$A`TTjPmmM6SRSG*$4WeU35)lKZMC!1;n$9@j2})a2~l2G{(iv zBea||zer5y?p>Rw82v6 z$)z^U5~v>k93T=m|EWpOzU8n0zwSAg>jVR1o);s&mh8Y=aPNv*5~00?lYl=@MCi;! zAJKgZYVSr{9cwIt&F}o%rrG25iJ#Ag5YKsE*ZrCVMRJfe8ZK_Y&wVOEj~D8F4AVUS z)?PDwE~C01bfEuYW39BurQeF#wII^B3!!YJVjmeUlheHF@u^+8IKuElpb_1`ZOqFu zCQ-qK{?=pgz8VydZ~8;RLWP53?2r!+XX^SISv;ZP*pH&slHB45$9I3|Fz0$k#aejs zUKX+FzU2y_YE%`n2i34_+a<&~no{&i{J#_e3knAKg-L>*i&R<>+vFtFi#Sr?$YAN; zC5_6TJ_jjz-He;H?WiN6jPVHQ`lv&TkD0i(q(;#QdDjwC8!E%{zY-QY~qp?6Y@UQLS$l(=pHk!e)6Ij#=~1u}0 z_7ijiJ2QYn2;8eKXWL;0lBIep`&Yp@5l@)q={`$X%<@OdVSG zi;Y2m!OLOJeSi@sV;pB1D%jK~cVp4q!3lLMVLL%M*L79;0(;?6K`6rM4*#8m&+|=W zE!;2d5zNqi*uwl(ql#UvUx8p!kX7Zw&(;XK)!}EEV>y%*3ZSadzY5M(B8bj_+|-l*FMI3V%>@J*1xf)|DBXAS%@r+4mHFQzW>zJKfAJ-$6 zaj_E>EB~IW`bhTDy8v0~23J|=5>?$=abxMk|GrMAdaB;j!i3MT4B4v3=bBo7w?<08X#RiLI=OPp30eyU1bPH@k&k8UgEy{d=TlvvdDt|B@zdRvE8;#b-u<7C1RBc6T9?#T=7~q`<5ZZ<}1lKDUw2`uufX ztvcC&i5t7UBAM!I|A(-%j;gZT-ZrVEAdMg(-KnTZiIg-bAt?ePN_THix23JDOmd)M?hodS-NyFxh8^QU=5N3JM;6>jHRI50LXH4mYR$R+&*}Tu z!u^l`^wzkw9c0y}sLy>=SQi{}`>?=Z*`F4hof<|*p--^&WE4+qH`|uKco2eFye#H3 z3iI%$xZx_XpiB|A71*gW5(6(Iw=mqH;nDa`@HcttKfdLkcwqpmjnM8pxy6BFa7;{f z?E4lx!CjL<@S&8>UE9Pryw|ne!#j$1-ShM9d>TmaQ@(iZkTn^?htv6uFIotlp5grz zkAY1zL!(OUX6o;q^zWO-@ALTIueRQ&Xvo8oZe6Y;XXG=UmkYczA$N9}UZU`$D=+)| zfsD}98q!E>BQdl<09Xf)N1!ST%9@>tpCrq zt;)^bF&OXkuAXhgtr4Cx@s4S&;XHgHh)>Ju6V2=va6)!1R493lg)7I66FzkPOjQ0L zFZ&AJP^O=cQfM@)jOV8!zcp?5@BO8RwmC5#HJR}W8ShR&_MN;8+{?=Wir|+C@Rd7x zoE>t>f~tCft)+jj#y?|lJ4XE5BS+f5@mO!7etj-;jay&r8;Y~=G()WZOm{1N;&<}S z12=5DFoXP8xgw1EDHP8nIbiRDW5!|bDCFZ5jf0&%$<|7rt&oxPRKxCoWaYR1#m})1 zWomsbhQfD!Rvf1~0@k+2-q@u_3_Biuc6xSySMlp8J12bP^-M!h=keEa_vX8YyNW#i z&h7RLGGX>tsja;xOcOmWOL2L(jZ^0T>+!uy8IocQL7vnaI5(k+=Py;@O3-mIC!IKF zs*k7=N@ebXj_Nn(J=Xd;7^xtu4=$=78K z0mXo)v1eRA>~L3|#{(d<9=^L$0jfEx7{cx~;8XH{T^733xc^?=y>S>&VIc6zu5-iB zZ6V%8j#BM1kM6Qb7P(^l;wrzHXKH4oSYt&#|LfWZ!@-WFS~K$Rjly(%d@i$OgqdFA zR3U++`)@fF8Mk{f^Nyr<5~%Np@P$5f?rzvEY6=nD?OrDKF?(*5n8M;CmHp>-b9ct6V5 z8hTl!1bl+pSjEe+TJoCZ83h>gCW8!AI(j-WG;2X*1RlPRu<`G{Yy;!H+vFMB+F)2= z_(jj}b;s)%TpXNyez1^G+d{;42@Y)YVm$SUEHl11c5oa(Boz8Df1D*lOUj>k~_aLfHv<1zb`N9utNZ|fb&-?u>(D>NZ+2D(iB4z zNKKKJv`JavFn>{_Y9s%lzGa?FkU?oZM&LY?nxnC!*tKhGSAajQq!UvLQ>$XvWS!5z z(BkCKh^N)hJx6+mJrcOOT&|w=hy?B<}M5Tq%i`>P%U!Ur}^>b7yMQ+^)mD< z@bH*+{cDGj*5khvJI=gN(TP?h|6ZWKr~DbDBg7*ER3;kJ%0GLIx9jWU=ZzUZkj=W6 zdEfqCxMy%UvZ-_02D{6~h`#A}VA6;2Ntju%@VWq)nGC~ z1HX7)bH6XQ4Ne;Nw?%Q+?)w-vul?Y<0F%R*Dyb^xr5E(+Ba-%JBOh&C#QR_wIsjP^+8YP z4@1-3yfdVW%!}V(AYzXX;+eSGTAtLUw*`KEZC-fuk!4@K zXdd4wy=yp?oI$Q z(YYdhdtR%G$Ij!_Wijxr7=2MN;;{>4D!&hGZ!!jz48DS9S1CJo#{7F7cj+#l=T%67 z_44QG%mx~d^}982Z2X@C zbr%h)|E4{ky*dygPJSv^yTFu?lWqAq-n82WzO+Z39r7!7;l@9=8g(>1YGR!do#Jd^ zZ_`vxDk+;3D?;9j9%n7#ZcXh^%hSdC#r!!5>Bs4VjwSrj#DM!idzdmLhMB>%11xS- z955Wozi;H1S{4B6EoU$NA*~s?e}U}~QF)t3()_R@9i~2dq|bezX5D3j%li8tiU$)f zH^h$4i=?_RVQ~iL72MRsU<~?1lWB~baUzF1z(oVHW2wdILTrV44rW0+A8UIl^>N@& zh4A)3Ms~IBS+OdgXA|r-~-Q4Q|A!6t?lj~`TG1Y%9 zU_GosUgmM!4aXJw4Fw{tk0F({s?{tq@)wr=Bb?NyhFW-}%JM~TcB{G{f7=Mcjedc< zHkqhXv;G3}Xmjd;)j1{K3!nd#u72!yUYpv;H zA!9D?vcbRp2h>O9&E6tOMYYlFSmh9XD0F0tVK~J)O`IBiF6!^n4`~6x*?Mb^EkbpL z8Z0JSO0~5C{77e7L{!Y%ODv+lSkBG4|F|Ii7IA+Oje_$1J)QRkz&L5x{P{HCum#>4 ztby7-_L{ES>W)p!cL7+~?tje6=j)l)3WF4xe0?oad(mOV1n-`t8uSb}d&a@xCFfzd z-w`xjzJ~H^js_j(xk^W$+E5rNXBS0zd`)(KbKjH>9{+C;DSmXF0-!2B3tDK8qF7)a z(eNM-U+@&$PyMRg+MaW(;cX=){Vn9#<9YKNy?Aww-P#(q0wY*yd!s5g5;iaS;Y$2% zngZ1fUE$miOU3LV6yuKklb0x!O-H}1*Hq!27n9&-GUFYXW=+axwP}=I^W-xdVM#2-Fq6Udy!9(& zq(cRLYVaihiQ*)rw9Ily?gNn&Vtu(0b0Hk(0&xTn-;tsFpLyurz}UA*`StDn2XE7> zJagw7K(h}K6YZkb{6`wPj=@erym8a|*xScQ-J#L%H=;qt>33P*CX*g^==*X zjFWZE>YNg7l9yp?|7e{>Ae9z)n5Su!Tyu5PoN)gd$WJ9D^=kBrof+MoL7-L-%{<_) zhJoPlzY$l^XM-xCuU$H=$5&o^a{OO}IHfJuJ2a~z`8DXJP7S*tpDq5GF!L9|Frw{1 z#wrEfGi=+|`@rchG92gsvFz8bYn8l?rV@6}hq$(*?b*FU_;#FU zX4>^NTpvFm$b4dBuLi$|p7|l+WvIuuK{=zA#M{UP!zsYeZ9OJNyY-(Y0`S`9k`brn zdX2bI5q?_!_3!iM8dC%x^XeC?@v7(YYbrPFanZg=iNo@v@0~T4MvLv1oSF*o*j1Bs z#657zqvAIXQ2sxTIqjIbYL~sq-sDG(SH@!sK|I3#C!Sk?#=gbrE0{-FOpiMPbQ+VZq;54i|?DX~P?i;V#oanHmW#jJI(i~aSYV0EZheN-F%+TefA z*pJLAuW58vwdI|HTK*i`ujwNAH&>><*Id0egFm7oR0hUu>S2-^dIX}b+ZVQujou4D zJw`6S(>(v|oy$edrJOVhSCI_l!w)OI(DQY=NZv!zd2@^Z& z%m^P@*FBzH1Y^{3t>`%&%uuEUkSihprm~W6`0t$w?r^s+D}x|87ZjDSjJH$Y^?vvJ zQm_w@cSiGJwQJ#SefP!Zo{0+?v@~6{0iwEqm-cq@?W)H^IVeG)4<~#=ho;;92MKq% zw1==?(NKLskFw&oo78~W7XLOEtSZtODGWY%Uh@bH1xE@(KcI^)jF9<6~tDxZmn5C4zLMWL~8WExQ(q9o6 z`mQTXPCxE1l>;*~P~h4aTpD{gzE?+PW*WdfLBZW@{ky66GBhjKM30}BwcMENjcA+x z_#PC!jRyKwKt^E5#6`Ao9RcjzYY1?>jSP+eH}ix=ai6qH-^1k3=7o{Z97h` zgY#?me-hpWf^Vh$8z`n8fg>DVKLd_lYw*}^t0hVb!r+i2o)QAJ`VeT5*i4d6h3EGO z`EU7fnX9vhU2l6e`zWay%b3z@-~SEwwF%n$99|)WAiM%s8nD5tpgQ6}H)Wv}Mm%wB z`h(C{X+1yNKTMU?9&nYt>Pw0A<MFbU(Z%=3 zKFUk7(lZP-eaD^mM(A@#j2Dj1j#W&EIsxa)ethU}E={{w zRNRzt#oZ;sH^LrTO1TdK7LLXmOylPVtC^{wc&K0d98^P8fF(I@CIyuOCw%lR7P_9# zbyCcg*#72wjYdX4m1Y;1{UyW_4AHvM+xsT3BHe~mUDdX#FefG0_%~fvi12c^4JES& zWeRv}42rEzHkYz*W@YDP>9|10Ur9p8=?zmKjm zNpXf=CU$%*Wb=E`KSTq-rPzVPmvbZjh5{qnZ*tEy@c-h&HU8nljqvjvH+tI39Ycn6 zA!mUo2!(A?nK5Ky)EZ_HIP>1obWD)oi{C$+Hcp&lg?1_=MJF3@yjrIqXzrq*Fes0M z9H@HS{%Anusk}b>#d=CbjS%CLJRPW8$|p60%~E(tNL-$F#k`leT~O>cC<2u$XuN)Q z1Xk(6c~BA%ExQg@hqA5pmQYvBlPc_%Dibf(O*aAMh)eg0_z3VfD?|8?(G`7N)9c5; zg|-38x)R6OE|l8yQ#{6yTS!FD1*r72<^hr#JrWdYY>FNhgfGIvr%!%3G3QSes(}Wes zc3%bvrk(Rpb%5BK)YXa7><6ICaIb~)4NQ|IOGByZj3wS$Lfuc~LLZ_%S%OmBz`RbC zOp#d@#ZOVPEKrfc?*sJ(+A=NQ`@_&TRQJAk6t~ zo$1K(9G<-dU<;Piejo*T*sh74Z@)D!8PeDoJpX8xeHehKC-9twG|P+)j7Lzaa52=^ z@GoayuR3@>Jti&qSqVcOpc`tjs+elM_I1uxdO!|pkhpv&4s6E3lD+*#@?Y$(>f( z#I|e;N`S?LgDNuMv>GSdh5$rpUK*&~pYQTO2E~5@o55g|Q<)msVg0buNSeErx+z|H zIkRzq8!_;)@fWCKLIDL1YtJBe)iMp*&e`m}X_4F%4G@uVdduO78{Rry_#Fj-$n_E_ zr!C~VznoGmM_db-HsmH>Ktfyf2DV5{9bj9i4!V3H_W9!#1=AKv?}g}NEsTz?0(@g9 zurQeTl>r2wt%i)8D*UOVu7+l5Om>>$B;YfXU~OBLdx6b9{!ZLB@#EczF511%oLC?L z<5Cbe@SAd+us9*TvbWUZM5K5{aLRt*oRWDw?0JvKkv&vR)p01H6K7T;v1~7`RGp^o zEW(u|cmWvOO1iFVIy0-o2CxfIUGvS{9>}qz#^upp`t!nlz-bIsg^OBskAwLp3PE*#z#h=qIPe(fVvh z{qj5jEFSZ#s2cEiq@E!GN#Wi>v+ks_{FWsQh$I}ZC7d0jxNAX48uw;uDSluX1t&YJ zoJxf&B6(ozeJO?+hJ3Db706moL15NP)zYIM`4LJMU#78v8^`d;>uV!>TMW6^PYqJJ zZkD)i0StBbk(9%#lN^#3bgHoe^3&Pkg4vPgr}QOWKvB(r<6msn0)VO|lt%%K%?_mE ztF;8z%9kREuBCB>zat}24E8>;j-f#;y=vVri+OinbLN&Ia`-&I*Bo~ARG3?C?+++1 zEt$!EZ+YC>fBs_<_~6$77K85SVhc7C`FfWs4VejNp8)r5bluFOPV2Od^{>Q_X(O+h z3&rf`f@5Xc(jtK8LRM$BIj_G$vSvS9dJTChQt$8pwMPD>Rf{DI?>49zCGOQ>uW=Zl zDt4%Tm1`l!GcF)&rFw9v0vp8K3Bra%XmR!5ygJ$!NM(pu7u3;P1C^}(fXi~SH+JyP zP@&s;%DGW z@ppxpR|`Dbj+VF=&rj|-%4Z=zB%U+wA@VtK&xZ8AEk6AmtMUErwz`hOE%>oERW}^m z*>OimxBdK6gimX-hQEN7DYdwKl5x;J9{DTAXeG3<0t#vFTj{cA@o@G~@!BCczsBGF zr9Khuj*ve0@WJn!=?4d7>>dxZ+~FgSUB9i#f>xT(do5l00YMGZ5ea7yYH81}pv$#- zj?<}wE#oKjp6A)Cnz^u~w@NeewzM4H`TiohPn%Y?mHAZELP|`AgibA z&{CPTA7h)>XJj^+{(9fEM8wn6({05)MoD<5sma+Bu-c;6UiUeyY|YKggKG9ZcN(O( z7X4JBs&Iy>N&=xvT5#YCkSbJA8DwhJwR-Bi`cj37MqO-SIP{7 zm9V_^K#k`_Q~bDQ$B+*Ebuq#qi$^y*RQ&wG(t;~_j#Go{-5jSTSIr!!7FVYnr#9D^ zkT)WL-sUy@<`&ngAlgfm{){-lW^g*)Z{kt0Kq{33f9Y~N#_KyMQ!wO(i9u5fGL_SF zP*uj{hWEMVY9R7WCFq31r!Gc6WpvOU^PatPdoZ(BE0qUSSV`&(CV&a*;l4XG68sa? z;O1qD+{!M_7jt7>f>U&wzZu8SApR8Y{*&RGgYT6j=LW}1Djoy6%BUb*jxAByuV%E4 zKw@wvex+7;9^x1)p!1R ztic>Gqfx@n#~w>r+yhv|PS58fSQlA+r!YP~wMscU#Fssr8eAm-l$9yID|7_OQhMjG zB&pAji*7tdSuAIYb7klr8B{2cHeC{q4!@LKG!XvVdzzR_>{5^ z4C_VIaj=0XZ<{z+UTL3=%32C||Je$V;1CH$Ft_ehY8-S%I z)fALTQDF2Ug}DW;_aT#yuVwT_D0BU7`9Yso3L!zvE*pV3uW~5lS>L|x=e&JgDuTXd z^5`U2xHEpt&ar26YN~4S$X(PfBR=lFz>M9Ig3FJt<9f`4o;U#+=(Xwsd`S{p3zlQQ zWa=m%%^(p+7qhZ(M|&wvZUE*MCZc4aB?ktb&&LK$j#q3J4UnfX)E@hNDd_9vk2~0D zN{L+nuN&Q@CAnr4t&h|^dwGjEd?{pOZyu+V)CCxj(EF4@jj9(sF9C$D-%|{}q?-%G zu+4x-P#gqH)_v**aVgAsg3HL2($CUod9(^+7X0I*^xYdXhPWF>;Ua?XxhD{I1*c-3Y1^Z+^syI) zY*Hk)WG>2TQnX)b42RchmYk_85uJC_u<5BNwtHxV?-C>|ESJl@57mIHl)$IYTT|}Sdj=Wi?+jrHraw*Jf0#kOB{u0^#MKp+ z5q?sZm-!&p2fNhQZlNKxu!$7rM)-AP&zz`zuYNa>B|(;rb~CvnVjQ{zkQpqkewEH; z43724`f z8rZoFhhtEv@b-i4M6;ro-9!^1lOWm)Ji4tRJqLT%^Afk|U@In%tFAlvz+H{K0|m@R z5c4d&k-f6;K)2#_6{b}ZJVf4labEj+5GUzHX_{wT;fh$Q5@N3rhm2V4`z1gk zq~bVVeVSpkQP!%@ieC^Xp|QfY6n;sfF%mwco8X$|{g%zs=v4t68}zm_WWr|sU!w#} ztwp6`pQ}xM8Ew5VpIOO2_Kbyh=@Ww+YdrQ17JM}dVr z6+a`R&R`2vGnvgzPElPzd00FrG{1GdG&(`ds~Kf*6|@wYZNhMmDr}<7Z&pWnhR81{ zbJ42Ev7}f&Hf?<_&v%ZLc|2-`j%@WMgrS5Bfxk1?^$D09n10mMBj0yoTE-tmDV^sE zZ*wjLeHW(?vVZleUsp@Qd&%m|&ouTir8{tOrt#J}TBkq9=6IeAqjerR>5+o}P#pW% z!!rTw=%06C*&G%0q~XUK$4B*&ij_#6_26)Ul5_iQqg__HpcD6$jyh{FqaBR1@No8` zk7=ser|U2;alW&8PlG#hs872}2YH?dUYKt0CDMxiqV6pB8hw@}~HD6>>jAOfmftjmn^8JJ<^p0l@;iDL*W1_Ifv1)HK{AHx_ksDz#1; zewf%itK0+>?!i?J?Gi2gI}WRZA#RcrA`RwvjOV1hDjl=g)e1z8Dy$lM$MXcwbGfzV zqLkWw+|N;UN^$7oS3W467?~U#J(8gW9T?QA;$Qe*2=*&}%`lFT@)Gls8RM@<=5XoROn$mhVN>I%p&h7*ct}b^i2sFsu=@Lmf={24jz*2bwb!c> zAM>8fee9WBArWtO6^-lovBoHl7A;dZ{qUuxKE8a0^uZVs#rMTw^F{6JShUPhi*E+Z zXNlnhuq&V*v;v1tx0N#UBa8X|ngI5XM@>PL^8N(M`GwPrd1#qh7qlEPVhQ3W1?bIg zTQ#Pt^cxWI{+U6x)8PF`$0>6I)M8NmrMuh>$0LfMOj3I8Owp?6?-{{Y+ewD4)aDH9AHcMww~$)zvDdcjk~R4 zEBScThq2$3$XxZ;^rNGku=rP88hSZhDr&kQfzQ@KQC6*LiC3HNfXckjXZR)5$6-4e zFOWm)+w8}+aM-z#Tz@_)=M^y9hVS7NmXRL825pn^+UT1F*>o+xk z+*Rv=;{j`dUk2;A+Gfg!xck48(N7;Xn=7j8SrJ5#JsP1eR$Qg4wtPkt{|O3d5tr1(1;4k` zBtoG7tfxm4C3)+oXrsW7d+6=`9iY(&k?cEye%-)q=!MzZ@hhA~WlJG@mMb^Gg+#^m zR&JGEN%OmnU7rbcl=_YE&xA`rzq2h0uqQ8qsfQq7#!@b~IM5Kt_|tDFYFH+$h_h6@ z>^l09XcQ6=5+XWOB34{9*V=&P-Gi_7bDt`L*d^&d%=fxZS#nzI!F>ph8--oD<>@G{ zW~{9yi^sYIsgqYFdMdI&L|sh(f$T^{dtX1jj#q155}<_gaeZ=fWk!iRC?+PIf(sc5 zNm2j+K3zfrCZiA~4VL^;eT}XQfPw2;Wl~)oTdNb=*A_*0pnA}!PEb&U;iPHrcD2tg zYceUwdVx?!Pl}~d>a#*D?=6cju7~Svz?UaZrxZUk4uWcG*;F~4m0naG)e$}G?Oe+z zkzKN7Z$E%&Ky)R1;oesD9$KIM89^V{eKf;u5)c!~?beALaMJaWI(z3UBtT*?p$TZ! znUbB1X6S0ESr>jbD{@1#a89RdiMZ@MLWBAPkJsH1N1XsIF-doKQu96}(FVCMoCw$;yq#f(o{Y+^ybnHTDKq5QIGy^@i+R z_chftzceXjXJbp2Dd4hRhniJ1B}r*o`1~qz-g&VTOa=IA5jDuTG zw+Gc>LR1!Pbl)(~Yd>X_cNoGf=jedAsyO(N;58{8MWbXHjOe0)!@6B_8H3x;`3Io% z_X#|W)EqBue)eGn^Ft4i9pq@)2s9mrBs40LbSqsYc)cyrM8SP0(?z~zSfrC7F`JZ# zn5(jqIf=92O@3=QZKBHI3V8%}TB=*O!`jFfs6PP9JNp^uK7jxXWVT^Sb~=$u)kuU) ztt7!RzHypCn;hsGif$MutRh>HqC>+RWti7r*$WQ`(71p+6}#YDuKMbAw5@jF78;lr zfFo=wJmGOT#`0dbXM|{qoj+p8Bu0;H+jLi)4eC(PxA&D_D*A;;VhE-o2DGy2ysL>> zUm{a$Pv;GLgQhhq5sysDjs8oDNSGoBuUL^_x z?IwW+S9x|2sNZFRdiM~gTK=dUWLp0b+uLlsZv{$un|^XJr|y_clWOYTs=RpA@(_15 zYr-14w##m9W4iNZ)3alsNp?|own(CAsKwQzuzbJ+-8`Q}RqM9m#F$H#waA_%Is@ou zQG3cO8&HhON$at%s*Ns%SNpT6gDhv?TdhDjQPi7XV#fwtaoa%9K!xQb;#{gy`ANL} z7@RtBcbkVBz0i?%THWb7B20Jl5iyo56@)0HO%~w$)&6VS2`3Dm40#;=3_N^D6Qy2YHM4Sb$~O&44`9*emC% zH11CwkLUa|SYg&M5u_mdGdX9E{qJ($kcntMs1-!{_l=9}y8TyVy)baZ={9lb5MH<_ ziO;}$dGHGgb7DaxrjFTMC)VIoEjT+CtQFNOdZ|d~4CHw;O(mrE=!H`Yb z#~Lg_Ue`!HT@5BL;=<3q+hDsgBE4ubr-EXL`T#QV60>GQ} zgq~0awTafg_LTrFv@HqiPN*U2U;M=P+x|{&M`XtPwL%4Kf6L%C7v)0OW zP1L6#iW{3B*Lv?w%FR34fcxP{YtEbJNYHAUy6x9$_V!=7740f37X!uis`U1 zNzkCBl&*9>8gA#@PNPY%D}7rZ#$kx=dE{DAk!)OtYy_zuVkD0SuptBF)i6nOgkGy4 zVICS5oJAx=BcG>|V{~`{!oa;_Oq_sM1mRZ&T6}ng(6!*TN$Z6Zh{57#NK5omb@F(f zTElLl*K@NVVH7rYs%S%1f?zO*V%2C(8H=yGG%Qpr@566D_OMvF&e{m~uG1N=IR`~4K6QgGCS%l%6X8c}pG8J4gp7C3uF?XHVouqs$%Vxv z>z$s}P*(K)=EUM4p(?gGx`yjA_jSP$tg3M30}vumu%mdr&Ch7l`pJ1->S(XGI)PmL zRZGfJ{CIff8DMMoxOA!HAe58T60YL3zrfKTJfW4GYfJ@1le-#xtJyGh_6wv|1Vtb; z5gqa3HahA2{>}5?qrVc>X&BzaZ6Y-%CoxSCTV1EP%QF(%2;aFAn{96M-Dv*0f9SkC z%jmr-be#%^<=$HbwwR{Fu}#ljhi_C`iyfU4nWOtJ9_AgR$ALHIL=E*Sx3u2&otUaG zq_xuM!IfAV7=}!(1hC^qaY3Hi;};zx?ugmeqpSR^0^#4=0(IQuvlnn`t#c&3`AwI5 zlfCVV*4KLSPv>F8Iu#JpPhP*ps#|lCzAGAe21nuo_2NxMd80<#b9oYTfhU?fngS>A zt1|{8XC<%t=p!(vhbn@QFsKZHxo9`;;jQ@K;|*CY8bl3YhDs01Is%UY984hAvp^f$ z7Yphm50~$>n!y?S*uwRscHWQ*$}DlmO#uWA5l8b>Z8ZGiGt5zi-v&}CMx;;kqj!pb^5yB!R0%(9N@jnAh@n5;nvLjBu5GL+l``i@Lv5cjL^4wy{ICShS% zQyhcD)b;WKs2@yFjQI0_))^u;Mmc^g13N>VI|e7{nB73W;Q?14j<&-WSBA)N|6u`) zUiqPp0^w+EL8>%Rari~J$>0b^w(jn|W1bvS`NO?=(`L7MDZ!;x+M`>`> zs};vy!l~sB9zU80)x5T;3y*?)TS>0bStt@r7o;qs)9C)=(ri83iz(SCH^e;MB`WZI z`7!)?9`!8;Axl!-xX7JG_v(iIMZ{Wn8w>TyNlX% zb~{7+OdKsuicSfcs^zg$ZXEPyNP&g?L4jN2Wm~xUg<>d~j_r$yX8p1Ulxf9P-_*bu zF>xsCDO9;gH1=NWLuI;ADk%oC`$TEw7l#v=awCn)Ao^?&(-$Z@X0*LzvT8eA=rk8`NoiajE#ES}PR zz@uTco6OD0&uaPNbEt`&4K7hQ6D7~)M!Zf0_oOt|UjC8{&%95KIJr8# z=u;N^_a65^vpJ=0UJY(dQwLWT%eY<2o-O#odM{`<%xpQw{*G|AwY!uW95BW@c@j;u zKuVP>Jr?hu`GTJ9GZ~(BE_pBpXZJ3)mOP%9TrX(2XBF+nUCksH=E;t(Xl3%GOo;)I|E@QAX~|+y|FNch$wtM8rOe z80#A`4UFR;YJ)K!g#t`}_|ym_mFH~wA8S!QZ=vlqj((QFV6^=y6l`x~1n~V_jr0*o z75Cvk0K`kxbL1XSR!wt4zYjw9NFp7$VW)`G;>|0;o|7SQtpQ%{1j5^O;$0w1!n8J| z?|fC_F#woVG2;DT+SBX<7eQ}(XjqL5DZGD&OFyw7P23~uMHojz61cSESHY4Sd#$X+TpL>7-pjsHMO+Qi z!0i)869(pMdQhyC!EM56tZn|p6J!R`_q(n0(#5V7*bX`&JGKVMtB_XIuYNB^AW>#U zCUUPyqe;;bmKslLUXvtU0GUk1il0_oeprS&I2FVZY;n%0%j>5GF)ph8t_1ggMl^Mx z1p91+T8jq4FCfOh3D9d^44Z&Pxz}M-GCn$^IJ9gKs(tL10Zzt#pW$B^F=Ud?(kO9*HyFtw+V_M!4K0zI!k0~EO^a62bZ}PE=|#eIXqQO-mu+P(@}?8=1$mm z^_ZQ&ffd)Gnm9rLtQAlz8gf=N$HwygetC4mybP@igQJ9B++cA>`wBwg#Zz`5G_oJG z7wR*rFrq1b79`Dax531D3!0}@K4h~rmpKHn5@v%94NeqU?ejt#@{7pn=z9ji45B}- zp8y~<)jhm{Zk8}A^gsw4OGv8oq+$YYm_Of&%{aQs_7X${BU0pgB&=7MVabx%4~kd_ z`5N9uKaHJqOat|7yF=eShw$YpWx}W_M-Oq|ZGsXSHW!gE=muH0YO!l^sDbC$Wo@bj z`Ft|^8pY4k?A~+lXX!YLw!c);h4(x>fJ3DaSO?lcgAwYs8$BCQPn58Iq+HIwqwMI| zmA!pb3Ef@TU2?YVISWElVPs>V^(4jFIVZI(-eMAmssqpd7P_2FHC$o43Jy}BU7!1E zcRM&thO*S#osV;&=9rI9yLUh~`A0l9mBHcLXVmqYqqUK2dX?b%X1_Rl=g&*}R)pf^ z#F%4&ebJq+dlGb(YvsYw%Tj+o?`X=&fbI-*+W;cS&=U}=&G~&O%ZEu@@2+GW5AEy$}zc;f#5U2A5bqcS+{p30&E%`p~zV<~VXT*%I)$jJ{QZVNGKm4$vlO;N(A0oaOKB#;FQMLy%Ww(jb;Nna5+{*^&O5alRZl| zLje2Bl}$sWjH~q14X`?HeriE;$XCxcox$jTmzt+g#L8Bh=lE@iiyF-gg6&2*pJ4S4 zk$aydSyqY|<>c%j(s_Zr8C|wI zrtSKM`~VBFwSZd8qmp_q~O4 zsPi3sW!7_o^vLx>RkJxtBs%kS%$O1aH{p6!=3|8z@(QCrzC0*%oGiD&mck=(M|>?E z(kolb(yMZ`7_WSE-T$$hypiCQm#2%c4bae-XvM)nf{TPphStMgZW64CDh*9^-{}l) zbS+rj$6Pcl2X9-nkZq*#hba}6WyDaRb@ul+YwVf5mF3=|cIRSiBgrMuvV7z4rn+g* zc42At@ai^eM&7GnlSv_|3K)=OP4sSyP_Zq->UPBd>bUR+kIu)+X2V>|H?eA0Se15^ z2KL+|O~iHe@HMitO^CX3;`5V_-$gQffsRhplJEJ_WA>!#T+{6;^KsAVAHv&mhoaxH zIrtY6n}iZoFEvgJsKn2~G|FfmGy6*DY0_hMr0LLK*^E;HiKL|uoVRa6`}<*?hkv6Y z8?`scG?7&C(ENJ|p`S02%b@y%7oX3>p6v9=GPg?u2>?ORu^!GMCxz-$XqaOf(j9i1 znOA$Jd9sO2MMPCz&qq@JEA!xenR&nh5zElV&Gtb?m1qp!xJ;g}@!i1m{P{hV#+c6( zdIu&e7AT|CnkcoHh`M*2qR|vhog~D_$~)36&a$Z!u1q9Sst)#zHZUhre>Y{LW&b!@cRsqX$V$BD7pOYA zn1uIyMII@5JsOM1%Y!;;Jyl1WGc@47K|BiX14y(TyMtqN=OY>in|JlV!XE#m!2V55 z8}gyZ1I{&JJHx=hcll0!eD%m{)Av_AHC0BN-%B7wtzoP>Ig_=a*i@{gfQcWjU4F5) z{pdZZDL*<6B7dLkN?MJjOeE1&cpO+ow+U$u4zhPRf9(~x^@FQJraRcd{8hVebYXXL zwh%H}V%K_V2p59PPqUg|%NMad8HEKR&*xdaw)teA8Zam!xv4M-J9Kjp;bI8^1E#id zWUny^HAlyij0WdxkVTFw%0w$lK`W{!ZP7~$M zz?n5}`TF#S5Jfa~8I+0274SM}B*-y|M|g1oO{W7tC;lD)rWa-rXnZ#wpKcIeea4^6v9|s!st|4Yjlr4r zi#juh_TiR8XTzX2D75^nPVbq=SrvL~%Z0ASP$*!DLg zG8)X!UqbwXPUcjPwcN7w3;4knDyB$64`?`n9xuS=r4j)S$q-UgAVqrz(apkB*;CY* zw6P7mXN!886aO5TXIag!pU+m&%QVN-Xy7um}fXXTzS>R3zM zd-9qg&?K-jfkowGvEdgUn?KK@02GesEd)$!isIuDTJ zZE$P3sp=bS95e$;U<%%GyX`N(Z&TK!YzGFMM|)w!nH;K;S3k0;_5j0iRK5c0focpO z(aYOciT3R|NFKp{JZ*|v6HF%Ua#aN3A44Jx;qpR_ z_NJTyp3%H=&tQ9ITOj>6gAf+xnPfs&5PT8iU|zT?B(3+o)02X~##PK(Y70mR{1A&Z z^gRK?%--s!WMwag6jn}n<+7$@$`Xt*ipcdaa+#g2k5nme>8q(n0F5qBCD)S%Kb;wn;kxj_9{gnYJcpZrz=?99~bg0<+$taDFqA&GV6~WzcdKL;b95**cNl z&1k!X_O~(l3(5X8a;!ibR^fki0!bq}D#MO?T5m;2Y?Z|>imQ*N-BJu>aF~;l+Wd&{ zdN4lRaQ&z_ytqpYoma>wtm`|lP*(_w-w)a#H$Av<(3dM~+!S#AZF_+(Cftlwc87Wg z9z1g&;q+dNBn3bbUMmGHXYP{hok&J|&w&lIs9cxm^NXVFJkc!v#-NOWuu=td`Cxl; zzm-iInGt{D6J0ZUku9+gbmiuJXN+?})h7;5-aXuAZZ+39BD<5%$A@<}tr+W=_)xXd ztDpUj{0B`mRRWPPv}F%2?2ji*P##X+1z>7ix@#*)y;aycIAqQ92|8M*j7QL2@V)V$XH> ziOo~}xfs}>e+hn6SkZ2M&?6$#$UGA;xD*y%qOBuih6f1DkeMj#xiU@t5#kyct0^d_ z4K+{NI_ql`?q2?~m%nloy)|fXe|28ZzE28OOytjJMp&U4E{p;8-%QL2(89nl1b{Kn zMq=LDFnE5gyjZTYAN-jyy$U&IukGDE;FB!>kKg&pA{66Ap=;9&>7#hEc9K9^kdb|-A; z$KSI_F(1vZ$4~j4o4VW#h4lGg!*{l`Byx_@ya0topl>z9pWjoQ8-q8j*?E{e%*KQM zx2A^x%Rdp8?$FMcqF8+Y`DbFmD(-hFNtLRI&W;BwaDpjngF>cx{7Z<2CtL?1Grd@cA5zi5n4 zXAqy5`t}=?v}x?)9v@=L1B7glftPs?iY?M?)>E%LP-7jW5xsHf-aY)2rc1$6nR^o3 zS^?R~@qGVG;c75SOMz+wpCt*U-{tbO#27{Pgz6Ni`4$Jpo*gW`3s$F|_6M7r0At=9 z)t|Nk`RJ~gKH6i-2ae|bidX+xEdN?HAk!k`ZND&TKO$G7&q@2|{~OEGeVr<*n6ck$ z9Gmvr0h1ggOZR##rV2B@3Z!5#sY_vvg4XTsfT+s^pwHYdNS!(UtYm03|M_W4s9PT^ zN&{(TONq15_eQ!gI8(p(8L2W|iel5D)#w%O*SPm45TQDC(Z65;2XAuh+YzCdy?O3v zD;g*#pY(e3d5gq`@{tv>(&024t}B1$>z{A9oV!RyLVnLLS(v_$N;e|iC|%MW(p}PZ&wzgS zyZ`-Nvs|MKX3l%g+5PNiyTtDQm@MqRm9VMG&)iij4^#_G7+*3Iq2$vTHc43!*&!Q- z-$lbM?6+QuWo}8<|GW5qo+8TzqFH7s_5Nk8y-=oS>=)bj3F41#b`s`~_tq4jV|l#? zQFF7oX@eA0LVWNZfTHKZov{m_gfo(mqG77;BtbP$4B`0aw*BALKiB~UEsU1J4|z&Q zotl`FXh01}_T4;amNEpc0A6z}TkO{tXlE0vv*Be)LgENu$99i5o?2Uf#l7^8FGgi2 z;kW<7Z}z|1;GYAAwd10al-FhXYt;@lBt?c*v|IdRO)BwG&6e3mg86409h$Ou3Y5Uk zNyow>A#E4Cr`Bh4)rV6ro{}F%g)t~)w1FZ~m(0J!^v^DX-v~fx$t5lq@ZH^bV{Vy< z!P{DaMJaHj7#Jg9;t6jnbu92bCSa74-<>pxueXR2CmliT=OjhJ)#IZJ3iET5sZ%)O ztHgERnD^~}Jp?rfai)IFTD#0zN+z;21#!j8=Me-S>>}t_ZVoC72Z9TB>J`VxdhgY# z=VxzQba2!`c;KkTK3-5b$5JF2_+62p$!5qWF@n}aPPXf+)jy~F=Tlt%e9gsL>*$@A z98C=&MgRA}ao`8u8eFN9u#cI5?cB`hhn)Ke5+#-TDBgUI^me4UF==*nG3lsWS(Wx# zIq9JaH=5Z*S$=S%@3c54Zk0sDN`*9rQFl0IdCv+oJNqycLKa&P>)s z#?Q$Cy~X&A)GW#asd1NHcdCpJG3RmR5eb9Ede9>xeaxkP&X)ci74~ddT0q;IEOZN< z3gL%5Ek_9JuQ;-@iw5m87t%U(-1|(bd0`k}(2NZB6#zo~F&zKAfeGU^91kj8&U&v; z42^bB%(QTSmqKl=c;b7VGYx~m?+ZRhW^kppOTkFetuj|VWdNicq^VUVlYn-^=$*!@YLRD+RWY;k0gLxq+p4tCvq252s#{ z=oYpusM@X=ro8$UR07(WWsm-#WB#oqfP+AW`2I1+y@NCH{9-CWIZEvNC*K|t?>ZK zyg)4?63pCd2W@FAfPhFL;3tSSZk)D%{q0ZXy-U}UMjZOu{>>?e^^e#)FS#dTx?|5d z4d&jysgV3XQb>ObVmAJL{oAqWGJ7T&&qy$P0ydB4n{YkbYuyqFUj!kUs5SmsS~Hd# zLu$nPo#nxEZMo;5phOF1awX8I%+GF!;!> z8q68bd|B0TT)zMV0owqm>qiufmEd2#@n6y0dxG$i-(WImH(H*$?iD?#QQ$8gL-&0%A;@;;c74G5!0lhtJDN z$lsw280K1ej&={dfBrQl{>yE0KE;+0GCJ)lN2&jQ5Ki~K_c4n$HQPE;c)4dV26bi7 zl{QA9M3yekJ6jrv_w{dE8oV6+*Lqg*-;UA#dV?}D`V?m?5NB8?KD^C}ONK)HAR4E) zo$$8}y~-2H1VR|hK-7Bl@QsUAIlk4F6X}sY_t!ycq@-QI=-2@|?IZO58jOE51$?&) zpzL+u`MAVS-t5x1q!Xu%s}ETw%-Yk{PqGJ_@U&J`icn_!K_8I%kWYx0I&EncR!;HH zeH!xNr2;uAL7xhA%A=?Emdaut#fA{IqH{0jG!m%+ENR?I;*4GE-FEiBru^@5z-JO=`dBMbx&r|MtPr*8!h$8_fB!VXr+CcsL-kn7n%o^#C>dl1w-yG>57(6YXW1Ng0^O|>caR&MzSlu;TFtMI#>HOu& zoj+d1+Yr^khN!^i%%T%8k?I#KYE#i z`irHa0lwOnU$oW!lFrv47QeXBg&@&7{}iBC|0uxJY1D-@xza=2P14C>o==#56XQ?6 za=(3!3@te7c=cvEvm6~YwK+cI_^ytrS-%G^VHtW`T_kCNS>dx=Mv5@d3!?ycK|l~h zg(H9M4F8990i8fk3hGYR=zsJw()`BYX3Hr?WBIz-X$&w<6@ovJ8i>I>nQi?%3j@us z{4GfQe`NX@+Cs5hDNq0SlTqt4(+-N!$UVwg(OgR>#KmVa(VQ$kTKB4Qp(R)xmg~Qp z%6j0LFuglobWT^#yAik#`IAcV!=fpyVy9Og5@Am7Jtq38&J)ag)CQ#G-huXg zR(mi*xqUS=g%$YF@|8WH!{+}geRz8YyW629iSX-EP~N`9>9RGMCGD9;w57cNmQ*eK zO%O9!peF_w4emyf6yAHAsbbca!pqTN;50Cj|JlJgRDv`)aW_+|&u2b<-#!`>6@+*3 zr+~|6w10Y1gR;+P2dNZs{NfPePIh*7(ECn||8FQ3C3FFrXAkspw_&i=z(L1RX!x{-ISqUvw$<_J=9VQLv_)DI zks@_gX6#b>_%uAzPOVk&)jl(r#EPeAtw8s=iLb)Qd<05^QdaNgdpJypIhXbh;HhAwAK>ERs5ONc~>5};fl?NS}_nesH=L>WH2rZNxRYj|kj`O~nWDixy z=7c!8u0-GA5hycm4(TZz8^(1OY$=!!_z~T*w_4zuSg>Tv-97j>W%O^I_^JJ!f?(gexc>RaggGlqG~R;B*b=F&K$Acv;7~WGa*RRDr`k3MuU4 zK*F(9+no z4%E8RWrZ{OBKDqR$ZG~8Nrc`$!k*mfhdJLd{bjYkU(t!_pf(57N>9cFR;Jzdov@_$ zl4bsI^54Rje@qUH8%I4aDzRor_PJXSX#(2;p(tv3lu$uGUB@P4FtftUFf&{!E=s2s zpi>T#u_3AL{PKp-NuwFE;eO%Z1z(}HT9Ev8kUTp;vw`7{%ka;q|N9N>2hL~~x5j)d z;hIjWab3KtxrpK|q%gt2wJMG)bWuyLItc1q*&2$q5;AbE7w48RdBORncA+giMl?j7 zjDhHo>q%37w*z=3X^PIwP=Ov52fz5D^*ajj+!nN8_-Hg>RgeRj;V5=c4~$aHe?cP$ z{&0%irB<|p+Q!Zjy>$Qh5TLa`!qc2+K>w4+aT>kL(+p%W7(n;zWy=Ql@64R?(2e)3 zv+wlGLPp)q47W!SSTC~~3cy1V1CSbOU|I_uU<#LtwVtYyS|84j2-wC>mjtdlr4Kq) z*jCsS372u(_aJS^_*nU%$;I!RMH zFiBH#{X_Sc3<(3X@lVg1tOu#B2C4{=Sgw_pk3D6P>-?RuBIAg>mtXny3QrxAn3L+C zM0dYrky$Q?4mHV(lp|v&kY!qASC!R7)L};Tp)fDFT3&Duth!n~UcHTny8iSXzi7dH@X__!6ZWF+4AD=+W<36m_ zz33hp5=SQx0U4VLa>u?$C15HXDxK+rXS)zILu&7!BGB#M4&Zn%T>(!79y;hEO8&XN zQsYs_o$Q3()e|PP9kW^cHEHR+{L;_Wts*$Of}@rEL31E>3>XZ*bMX=DP9Z$g2qVOU zRUiMSj{C$e|KR*lcS%0(dbMz=b8&6-vKaSMD;BjZB>&!m$7L~FZZ>q^$K9 zEs>%{a)^fSQfNt?%i=o+OP@th9W!2O&YY??>84}DZ!J5tNML0ykU~BdXgl$#1&mtT znE0ndsSyap$UipN9%CGhOmn{qO-x;K=gx1kE(f5hHnYqm+RPNss^0+Y$c)~6PzLfF3(4Y%#}-6CM?{`=yPAtl{3uIGX!ri? z962>~vyy8S_6w(>BXzacoq0P;qnOX0DCjvcQ%g41JVA6r=)<#tM9s;JEtI>WNsl0& z^*_%GFv~4u^==7vRlK9n;`LJ3;sq$5#^?QN1<;MS7bjMex}M`Iuv8rEsXf0<(wAs{ zK@<1Pw^V_*O+dnDo2msSS=w(Dw@Nn$NXo-wSY#hiOl$*?4Hm%7k_+Zsv8+XJuOQ`( zpP5=fEtI=HNUO!z9$oZHa8*6)_Aa_z{9*?VMF&n`@^&aY><`xVtsKJUFYte?fKyzy z{KcKkUn=HzvWv_5Hb##+s6I{5Qfa7hKN#{W3r=#jU_nV7dwhC(ogS0)#dGREerp)a z9r8`n+-Ue%Ve#NhBvN51@13+dg~5-k684&7XexrhKqU(S>`2g4pq?6MSfP}Cgoihh z*S0xUP&VxiyV7y&W~;+~k&xcwuFL+``bGH$uVix6$TR%zv~&O0E75G0#HKb~0l!eT zY_b}oNvwvThM-^+lUqydLfTIbZ*%8>2_ULaWP} zb*Me)2pM{4%Pp!j@Insm{`G!`PJxCL$lU*y5m158N4x9O2+*+2B@Po37AW0V!6`Da z6Zc-6wmBCY6sp>{xz>JJMif-n`jg?H4kLnW*Vl>b)U6Aj{1#?Dm%=TVD-9l|jklUB zWeRSPc#-rqiSa^b>PXr|EVoJRdKAmxq{Z9xDN@8*Kvd!QzOmQp{ERek z6_v3(7aJG58Wbt>--aW{B6WG53cK{=J(D&r%Wj3 z`b&)r_>bL$jSApGzwIr}V=P2f&%_+$^7=1owiWA-UJ$EZ zL0N~X+*=u1Sk~%Vchz#av|sksBA1_ZTa_KK%moT2?auox`k3g=*(x!HA|w|wkrP({ z8?3z&XpLXifVQaMz)g(fB_{{l`F3gJb*8xr4vI1nlq7um@9%*0dmHG6U_CzN7f&83 zk{JXs1g5OEAl~5z-zjBG+5s#zd>r622UP&13rf=}+_J!pix{4I%`3ZK&9^MEiKu$b zh$FeyhblJpB?r|XnPN;i7eBe+&E@Y>m~tD%vR|ON_@aXHFzF_D0VGh!hb_7w#>6K1 z{?}GEv}%p}E${S>^wiC88fqFU<)5*CXx+t*RE7*c{Z8mm%`;iYK@(l$Fog+N`2DMnZhT^7DPu* z1KgIAuz;(g@{RrOq$%CM>4sEOD6KH(OP%9p03Bj5I*#cr*8jQIOtvUK%PDu@LAPi# zKWFT}X^1X<$aehKq}kTP;OH z){kcHTmAkeI`rG+fz$T1_C3%yHU0)?r zt*9eK88-D3_xy){uG*hD|{sKINPXIb^`Hxm|o=-x#i zi|D9ve74*B_=b)Yz=huqnBKm#B;bYmcuA%Tq3NzC00^QKKz_}s&3ch~@1yq3y9koK zUFH=*<=UOO|H@0Zcjf*a#KQ)kTSfq_)t@dy#2foJ0iC4=0Tff=fN<{xoUbir9usqC zLgIUShMYK=+2*fHNz0Pvg8YRqn#eNin|5)3-Q)uW-650pg|rmek2DnSb40Ww#h+5J zFVbaZrCG)$vTz5jlZ_0SX7g?HH{ zJDZE1X(e}=PFGjitRQL@yR6Rdj%A##rgsM_L%c=#Z7$g-PDATLCePXTC%8uf5}pe~ zcj{-~b=OwvIjZsbxC=XWqLh&&o;iFabr~}eJ9!f&3jEkY+tR#yXv)|=-z-LG!Bjqa zpUXUgf@kwyQ4l`s2Ly*$jcQ_BjUOS4y#Vjpe1x*{H`x-6F#hB72LVP|G(YEEMs6PZ z^0Ge5L54gz@I(=GKF+uuDK;}zF%Qk&+0gz|_y4#UFb-m#uboEwMW&Z=^Lp#{8qF%+cWq=B)vrqqVd3R`~J~g zUzImUm5xQ5697h3x!3A1c2fg~wc|BNr|LBa_*VJ zR{hp$M*d=6MC;_1{ht(YIIVXS4Es{|`KYt4DgHnRp-hzyvlg|F#X-kn&rnM4nahXz z@ucBpjQLp|u5#dUe+d+2?kZ@^Q07HEyJANtHc`xZ4->gpB?$8;doy^?yof^CJ~SiA zd0Yzc*mQDv5up1+8KV$-paSr;0BAn3CREEZi3NdDQzYzC5V{?@YaZ?$AlgOA-Z8*__tQMQBL$3IYXc==SkGlmS5APt%dH6O?b5d!6YYs~7bWiIZB!HOjO@5L@?lgrus zWLOcJHnl2syOzh*Vxv*tclG2kyA8J)f5^6Du;7!AD0 z3jKX=_~-u?=R161xtj^n1F9j5Z+N4t7QgZQyvwBjYhql&5xfmB_3ch6o+d=+-n3 zf5{-W>i`gVW;pQRBpx*pjN{-m?jkPlP@fi6@7CvD$x8ittN2K<<7^&OO=CdY!A7V0 zxy+v=j*;vvJlEGs;We3$IR4|2Id~>ZoKO+&@h7FhN`usAL$!gMlys97naF>1g=DHb znvnC5a-6VdkS3x_-dnp*q8wW->D9l{&SisNB`I~S4_#oiDwyygu44pf%ek;Hi!=j} z#PO9N4~of%nHnPEZFUFT^RIq!FU?-rhmBV`S)71Az66;|*~j2)mY_Qmt%41>HDuO~ z=UYHFJ~f_u%sYk|6vGdPQsok=)rsE~Wq8cs^yD6wE{&6dyQ=9bff+X+b-nxoXkT)g_WpNpqp__L zP4R!dHMbs5?`?7`MPp0RTywyf|IQ8)(1oe29cl*%ZIet_;t@b z?YzsEegVic!k>h&N2kLR4NQnNw{}=}pxfv*TsxOOQm$U_$@#;#v5_9%NKY0s@a%uK zxP}*N3oX>>^cUeS%L`fmzA33jav+vkn9c|l^ZZK?_4s1-vqx6AO0W6&;_i3(_>n;Wvlb3yxPiy6{02_(}5D zDg+cx$RTiL!yjIzy-})ZvFh4@CGRM3Q*W0{F?3=MBA5*bt`eDO9=FpV-SV7TQ44U zsp|uc2a-)8khYvno*KzenktqJ{c`K61%O}XzNx=U*5L10+d{YB7v}QI%k5nh!xId< ztth`A$>aU|nerbeETfK#l?IuTMQrL%^mKpeE&3q;@Sh(kK}}71>;``1Q5V-q2L&B^ z5{-PrOGI%b#UN4kJE1_Cl%RbF$SG-x8IO|S6_5gnerQR|tI10PfAA=2gl%XInb4yK zDeF;?-e@@r#PGZE0)5_5dpqM=CDWPW-wIiTXDpC0`AvS=eTuoUPQEu^jJ6baUP(8y zEsR*){rrC^GRePv+698C92zT3r=hUaGjkF6`RIHVMLGplIVH5 zD#Z(zO885$9V#|n6X|SsG>BF#{iLbt2HvL3;1hO;1@_pcsU|`YTP593{V#l>EeS@B z6gDw>y7l;A7*z%pt8{ZHH^%iknFn8ut=8#xq_`aTBQRKB=uPq!ccE$6{#@L>so zWxXF^)fEQyJS9^-EJ>Y?e}DlnBRrT}pmQ&7X;5HJaT!OK^QQp~eP{(F>-#;_V7O|V zf&#$!`>=^ws=Knf2fZMPWNmlzt>`|E+iwj%@YB+W-1?Ip0_<*IEY6(omUz&pB{|*H z1u1_SPmb@YKpo@cQ7xGFt0qGDU>DAg6aH@H*J~ zf+KnKfmp=-m9#tZpCS`@63LN6rx~|u=>h@JPj|DKh>-i6_m&bSPw0ZTdlWR_$df-o zDk>a7GdxOFS=qf+peLwN*B#Ft0VqaVZvcFh{s_~!uhfqFYg`>jz<31;*=tZ9Ts1sI zEcGzG!t#oA%XMp{er#2Y;o&yA893wZZV z_;0yN#BX5~2MljCm~HaSCSAuUaj1T^Os9vN2@E8Ke}Q!pc&HY)1RZUjeRMKP|C`bS z;PON4eR&=fBc2lPgun3(HO?ED08-dYMf^-%9jd8&6RnfMWD4A%0>5#z`V+u=sXXx+ zy0IaYK5S@y-+U%EC&OR;y$fpqnB^7g(xesH{dlZ~V6?*em?8bKHtcY9HnD543x-qk`fLp1FJ7_Wpa^cq+$7c!@Z$RXzGtiJ)I_E z2#5QsoSAr^<*FhUx&>Y95lK8$H?>mhDFN3ctiNlgg|A&MxMj%C+v)FSLE$@&FSjKW zxlLQyPYu64CtB)^@Y_QZX6+ahF!bHVa1Ps2{vJ=ZMbNdldEX1nXQ`z8S)5~uf<29wd;MCapz6`mH9P0CGY z%X_a`CK{Y_c{+aEQ3TSiPiBT}ls>KQvq0@yJn3(l!OxKfc_Gs(WHc zvUAhiusZ35!*tZFID2zyc#TQx@S(L7adSA&NH)YkEGz!9R zoHytpo{agr9Y8@_v_NP1JjNYvzZ&S^$Lr0{i}IHXAmMy(;OAcvi;|?w!@PY0slvfqU_wz4y=M#zQwy^B@Ny(0{| z_7bGr_0PmLCP;W-BP~C*6Nu}VFk@^X4{$5G?S zU*&eA;^4uC0KWR}ZdFa*CylS}Z@S#}_XT1yl{sTY7adL}#%$VtjWNljOEN!Q#GMsV zJ`q;(Lc&?b>D(qM`3?k)P@EC~Gt1xU!9Sn%e$^lM;#vT1#DFm#L3PWKq_0fgDN*=R zDSK+8H)NQhGY_BFY+4fe!wR0@RCI`@VfVX`lIOiKFN7>qq^m;K%i2Dx$wqNqmDHE6 z^*vk^xFwnT0Xlh-S#**uVBEWa1R7=*CQa16CpcuTAIQfM3J4hT^qdAXiNQh$|NBD9 ze8@cYY|Q#7kLGsXFYlK}X%!SDl$%jKU8_{AY$Hm^k6E@(q2YRoI&3f$e!w}GvTt~d zz2_kRoYlx6{v8btH@4AST}hh$qaytqZB8AA;bU3m-B}xxAk!Wj@JI6h=&$QdSPQz2 zg}79rV`1&ZAOo`0J+5aREfOr1$=FS3ix0ymMZX6iq}uE(`cmw~14JfV9=+6xF!b`w#p#$m=`G zzM^NhR?nh7_*8Z6sMbMQjd!{{Pw=yyem@WnbPS{I{&l;Z%TQ~fbC&a04q|_urlRXN z)$*XTS1s?TVgkx@A&^V*$n8Q1)JRQ&(Ys^oV{rp`@+eu#iD_EW7dOxWAgBL&DL?cX zXZhBDRMg`m(DNEin%v8p3x}qE-sWmpVZY$uD`NyMZ&~An+0o#Bn5st=YsqB|4B|ws zNL&9wU^u+}&jq6Fg?{Qm&~J`6>FfINpWE8ubDiw|`FnBi(jujOnIiw=P$y4Y#z8{5 zYc;7^?PI0Yx)JAogBh&;PablrLmIV~wU%~~bopf#jF_a>WUs}FIc++YOIDH=eZLwH z7fv&?jC22VwKjX!!NQK9zWDCbLp0>60s*s7nUi{l7?N`C+ z>-ek8a`PFO>C1h07lzzKi&9IzR^K*d8fft(r$3UOq74-Tp@Vk}=v-p4$9*LT)S6IP z?Mm(2Kh8fvGCm2g0SFYfeU+7KDq!QKXA34Toh?kTbH}viCAGTt7EN+m0=$q1#P3 zTPHt%-}1Tu*B|g=H7uodu{N;88r=hpvyo3@y8u4Q8@eV0{GX>J znc4;|YFfP9r)#U2>5|O#E7l*%>0ipH6AiCc@K7-xp3_Ps3?AvYSe6STdfMIoY6fIM z?_;0qhRc$DZF~1j(JiSA6fKNM)4-b{{%h54sg`7m&=4~8^uqK zr=rrLT5>+9-YwGBTw8?eAN-M878@Zx-UI!BFqM@22(NR~66RzQ&zYhHMIm8_f!imR zTSR&>`=n*Ax#1?6!s9fmj2cT8F*cE|5ixPRpF>`=CYA}26b2QXukt&d`B5A&sGV#y z51RB9$gJh0+UHYr0#_xB3-Rj56C>>&`ncFK}p1gO}AU9q_;t9@uTPXDoK>m#kJl+KH4 z{WmpNrzSondch*+g^H4?Feo2|3S|*eTQqtq1ZETsSDIl=K9+>A%IRy$wSQ|$@Lu|T zgT-f2Sq+~7CtXec)r{INW}UOP#c9cB14gh3zg0KW%jA`elpOcb8oS_!=6EH{ z3}G@`_Y#8)eXVA-koq=aM58{lZpJ9S*GQzFn7Qnpdfrp}RQdEd#?!bfk7h@WZ-xnb zSVog)2~*PfS0OcT}r*4U8W~p7vrA6iuW$?`>)x%0Ih=Qe|hh zdfr`#O*zDd+NgikpcSsMijtkx_si5+Z?Qj3I?JHgE6NoEGaL3{vz~sD`Bryo8_AO z6!NIUwabVTo%dRuSu)p(^1$xDgU#;BdY-VPe#)g0XL;+sOrM}Sn5Xa_$hdRBP zdiTrj@a7DJ=-mo%KpyKKs7`7YoVdS%ef(zLE}-qS*t;&RFEKh|Lri=Y*ILUq54@U4 z7Q`zF+`>on6Kf^2@LQ!g4)@(Rmz(=<0^$EX4%j2qnv36`V0B4nWgmNLC!kH7uE^v+ z`BMQ8Y%D05N8UX{E_cA_%p5wDGupP5m6Hmr>CPP?#vmgoq z%t1#ZoG&!XPVRRL>1vr5lcC1IVy)T5owyS*MUE&|&%tlVISYb2UH3SDzrWY7*!%Vq zd>sv{kUGd)-J83<_7TC&>;N_8nyKkudrj$cO6jd~xARUO+T`mUUT5^<`B$%3-$e(6 zyrFPmizT(O;7NQiNF6-H4xDM|J7r5GoJfOBm+`U}W*2L6dyQ8j7?JGLsE3iP|i!yj`{IGPU zL==Ya<^O#x%FTXuc^>t!GEr_IPX*s>z-(fSFSW#RdQIJS+{Nl1Q?i8lIA?{=j||L& z&F4QiJOiE3Cov`(B!cQYGL9MrB|p>ZJ>JP8mlh;95vl#1$!~vEiEnsNa*(dA9$oMp zZL}t8Eb2wCeQr!4;nF@sIbGWxNG&K6_~>E_RGGVLJpDRH_@6#2Jz%U|qp(~avainI zA4*$v<}Nw^{H?uoUA1K*KHqGew*8?UIi|F(^7wX9&)&EP1#R%zbgqoy{AqU@Q=IXu ziM)fm)BJG-B6nHFN7~RHp!{GIv$D$1yV8%yQ~J7ap|W?mp3-hYflTNwGkqm?uTnoJ z(e>!Ec+Zd;FYAF1-YfjUV&tflMD)}Sua>;~KrxfxzWt1#KmXzhcBI6g810|K1s#yK zs-aBz2k(;ta-ba&FLJ~hqrPzv!a|%ditQtS!$EV`*^EJE*)tnDR^j}$=4#x;cXLBa zL12XL=L@@n^e6z6ug zQNKrFf0jY2?Qdl&pGh6nC`n%@fTD&Ax6f^8M`3GHnplnOe;zV*SaL&^N~4NEzTi^X zDcuGuHw&phQl{Txj;%Ck8AiNtA`D{!=6B;Y{#G0ta(TeIhvZYU!HV<1nODXS3P>q0 zu&_B{>BRzXuY-SS5ThU4T;C%a@zis-R2pc6#GgRtf49Y+vhRzAqSRNT2+@~B^>=ck zp}k&aBk2<7-;Ia|L{U)Rg<@9WwGjd;FHf41iv-OUttm`LwmRV_j#5%}?mx?{5R_Os zfzJf;|HqilTboKKJ4}=Obg%K)JY3Ceps0hu>c__r?tBS$VQ|9 zASIH%Kr43tX9a)Wgqc7t(J2kDPMWwiW{kGJ7(HKK4H`HW(BjL?$?3;34wMm)y!#$QhGf8lta%*f@q;Uk22V#rDbRxfHU9Nc9z04d zD^tD0tWF8I&corAeK1Z&mEy;&@Qu^{4wmZFn$e#{%SY~hwi;Sew_n}Mg7qaE#ChYJ z>L=)_&f4)Z%XL=aPBS&ds!KI{*#P?YLd0V`*zi^GA$Px(qLI~=aa67ut}=qvVE)eE zlI!)C0}{Pa1E&0v;)gejQGvlx#P)zX3)x%%sxxi7g8Dkf9E3TH;)Lyr|SSL zz6)Tmc#PA!uL-QwYC;v>l?q58A_0B&V^Cl%M8onbpgviDdBv!!XAfv%Wv(tSx^LRo z7(r5WaSHicL?wPX!`TdQzsb*&l7A^|JVk7H7dpmfBY1|qdw6QhYc^T>Oga1=`nK0A z6CkffCGuR!SK_Mf&Fw1M6apBy@nuu-{UgCT&2Nf!s3 z{`@s5GY0(j0Dl#fu{+?bP+2Dmo_qL=-VpJ<^+xHGpP!$0MycsA0H}do;ZMZg= zjkXU;jkE+0VqSVy!u!BMw{S$MS?A&}?h&mA!9+G^z89b}k_O4rx_7Tr3fM_H$V+}N z-b$BWSK+R@7++e{y8Li=27idMNR0Gznzw};4j)G zKfKx;BxAH%V?{wQD>gHi6fwfo{V@k&oJx~tL^ZD|w?CO;{V@cI8iu5ZILCPtpGL!M zyS>V?NmBmC2B~?d052yOaGQh*KH=%fqJsL5X^$gq{`I`Ccuc-Ju8UDc^tU^2A=0k` z%n30&8LxHMG`0tnDu7E_-Z9el>E5tQYucNcQwHXmqKYv#olm5s1w_*+jJn77S z7SQ6TwkPu>#Y>I&SVgEzmSn3&v%N_8UB1?*T($X=b+z!}Q@Q=YZK)Bj#hQske!&l$;G8;^QqrtaT?E{=a z41hk~&{XGk9vOU!pfo65K$ZYST!ChW>~%WwJ`{_;Jm%qH59IY?;dZCegUn9fauMvj z9gTY92y2pZ<>YFsXvz;vAqhN@7gsh!yjhHs7g)hMa|2Jyy)62QILp17oAb>`CLvgQ zoubnj2yaq=o(1GJAkEH5A;<);6y!jki;gi2g_mgkc^nvoAAhvQ^{e~E`pXyvQOx9_ zK2UV51Wy2V8ssw>LrAR>x*Ac^Z3xqUVovJWm!IF6=Gr|f8nracnrA_w6sLMo7mVE(5r%pd8Ziv%pPn$C(Ub!=>;p~a_NiPEkdZHE$4ATB)0|X&}U9?;_MMOAk1zh;(v#n|c0p|^3R9*!zlEiSf!3WKu z35~8zj(3V04x@uNi+^eZoDK`Wg4v=0CpkTdsx^Gr3tFTGEqo%%O%E(g1!r_wQ&8qJ zE#_ieg2@*~PDj%1uh?&v$14=$A6n42>eN>M7U8R5v71-@L^e^gS!3cBC%U)~J9yz@ z$*Ap8XelANzL|i$cDr|>^$v9zq2)xSYX>)S7%Um}xWiw_sMC@Pbt83-H`@oVm4JWg z@S+W(H#KK#7+Xc5GT(Z1A?eE=J25Df8vu^8D5RlxXv`*omu{q4Bcs13O)JKT^a&dW zd@n|C>Dk?!gk&lX)Vl6u304NC{X_@>tycnI;z&)Og+{Tq09#i~GnSw%me3ANqu$el za}5qK?Tls#PEzC43*!4AdU=e;fEIk+zjyFd7xT1Pcf1!;&KKx@F)`=FC{9_3I4sGoj<6l4^@(2YV)KJBG7cT&OrD;ANC~BX8jSz4 z2ZL8pnpWDAV58gmKueYY%=~gPgHkjzqF4BF2@$qTUJLyg;WreAFK$vTgj2!d+rq3;meHI$}}$fujEylorHW4arwv&SuG zEDKp0kt<;cbLfHPZQ8rsUry;yXl;v)U&+*Q$wka4z23AiE7-?_7m%6DFD2!Q?>NpT z1=mi$Ico!*|23!t>l7I^Sw*FP_0l6Kt!!-A^8~=T%tym)-2i7mB0%Z`J$Km%?=Zr4 zB`I(DU|R4*dkYB9?k3i*JjrN_;6#h2HC%^!iMDFk9F( zc{tr<*r_Fzx`776~PjbDoh&6-ogkA7*u|#!Ekf z?ggYKcZIsOx#jhkbfeB_G+NJ^ip#C3YK*cl_&7(-Ag|B`f7DjcJ+lbf*P`s^TiOAE zIbKZ>UY<#BatWNeSd}@1m_v+|yNPPy_a68mq#l|k#%5Z%vJeF#mKW5$qE|Jtahwv- ztlEG)XUip>6}`wYz0sU$a2yWMbGE3K{l@{{?UC~ zFtwn)aygkP_We=0me{zab*01;K=jy#yu1Ln8@YW0j1P~ngn#$-eK{!z z^unjVqvZn#`)3sBH79p0Cq<7Nv7wsbalf}`W)jUdG0Aw^0B$pWwgwDiiL3wsWi3ui zFsd>F&f(lBn)C}^RVQ?b8i4bRz*V8HTRWVHNx|z>=>%xY@DqS>twRLirA9Cjo4iiz z)gWLGjryGcryM1u$CFFV^6kgv1cP$KHB#WyCOlX^j2+iEd3XXl)0v%_Y~F}SX7qz! z=9zsLM=O*yttSg3eGk>o9Z}mg$}r@Q7HfC_YHwES4X$>pECN3Dt9OIb5KaT9Q8cgM z^}!}I-Qe%(=9i*w%ZM)}qAo82AA<2rueViJr|p-B4BloFU67uX^;J-MK>EWG6ww8?z*xrI_y z_r*o98Mz}-aS>?OGVVqpz&s_ZnJiDxD&RPC`r`8w%)19#U_9J0p44BM0!clw+(NJ-*k+kdVlziql_To%h`I9`$1*$ z0TicblW#W1<}VX)3Z`f8b#`~4;KTb0(}`{6b3hk(|2GRYO&+aC|4;7T>jj1NE!&?V zV8~al_VRPBb$a5H!ci%@q=l|z>y521W3z zM@CvhxN}Qi4rhy~IEhz2XQUt*T4}%>CgB>x;`Gib~ORs6HD9fp~`L zTC{&6ECL+cv8!Q%$PT*)J*Eq2vyBI}`uCi(*SzF8LL0ZFDDR=Zf7d`o#qk6h-FP47 z4aPd$y^+Q51c)`0ft_so=OoO4j2e*>7>==8xAMgvi?hSc;YxNzzJ`2NU}LP-{i<0K zG3Fydze7hfl1xZpop0y*tYMQMn*HyUX9`C2X{iYI(sqT-$2oud+)HkXahYDd0gJMhj?0Z-C!bMvgfH8V9R)~p)&2ZE$j*y z;w)Y3@Zi0WA0N!6H?i6_g6Rr^jjpnDVW8PWeOTyplD?wxjXmx%!rV6UB3}af(Mj1bPurb&Wl3miHJ6ySC;78f?sIFoNy4SI7($fBAtB+Rpw#PgN$k;qM{ZVWTd2UYa%IlEMwu}ASV}>u+iIDS)xoW`*xb}=xF0ktRI-xi1Y|ZilEw77%Jpd0uXO3uMz!v@; z#{8nRZJ1^@`KD7S0)ukzgws|5*czRD;qAPLxpM`)~n=H zvUO6tJh}hAE$2R7!!3`-`>FSNuoQ(~M%7cKVg&==6&>|Ga3U-9ue{5+4Jlc}iV&e80*HwF$m4A zNX!f^Fb>4HAD?Z@=@qg7F5+h}i*_(>Q>ytBx+)!f3TAqk*}g%ilr<-u>+>3R^@1>Sr=zIUYf0IspPyddS51*nd7V0{P;K0E#ADQ9>CbA zuM(~GYR(@L%=wZPg+nUQV8OAn++kTBP*PpaS0NkI^7ydG97#+Z|93+n*4}#6|+H+=hJpEN=@r}xKN?c zkuP6WoYr-*h8lvHSc{)3*Lqiql8}<6EZ*dru*%qpR|n=D~wzd%4Vk zC2qWR_SVPlfW->JM4BncX(Ie(Bv0c3IQP0q_xRqzJfmdm#sk7r5)+xPz>MDuMlVM% zzFVunE)Gkrqg&8aFntO0mZIdPd&$ExZDGtI|0j5~lS4SWcPRRpC-)UQIwGV0ZByL0 zo>?vQJ1%$Y5LYi*+?mIe^l?TrqDhWvBvzRF=uR`f{pVHY58ttr- z_c7}1h+?uDllT9S-$&!a!NJBmG|yYRoaV za7*!px>|3yu>Nw5TWGS%p9)M+II>T;ljQ+)^(_@NP1n$S!eVO}?{{2wPnMA=(Y5U;5g@t(nqaTYT}MtaNThC!63yl2xm$2=M?m=#ua&}Cz0WFslDsjcQdm4 zE;M6dB?*2H!al2>`=DQ84RrkfhYfoR#Ukg^)hKC*8U{YR6{3;X5hpQVSy|kocuV07*%rA*9R=VyFO7wojZR3eaSkLs`Z9yW#+0()J1G}DY7Lb zc;*x0)&bRh+ibPiV)K!*D^=$gQWE-G=d!{xD1vC!5Z1%t>BWVW6FlW$lV0v|y`!-iddhQ0uZD#L+Z9(G>(xiw{r zYfZE~CVkky^ToY#6+>YgsWy`B<#gv-cIu$Z?RPv%rih09x|7sJ>a{q6^QSEkqX#$C z0N&7F9~*Yh=Y21F?z2Y?qunlAX2w z+BW zjUoE0INyD@#Lccf^Cj}aP zQL0@uL*n~Ymo{6M(amu(tHl%zU%jBk#bJP`qty203`RQQIymDBJmST8vKPjDr>Iuq z!ykyjp-~0I-?|1lo#T1u76DfNamb!8?*3$=?%MzG^_5Xou2HwJ1%ZuHf`}q1(j`ha z(%l^@A>A!4qM*_n>6Q-ZE>Suq1qG1}(%p5R7sT&;cZ_?_@SGod)V<&5c~{Ie=Uj@Q z6NJ_R0_BBsfkBqS-8|9}I>;2%*`{xIWdePKub0Ce=B19SsSn&zDy&JX{2bwg!XJUb zPQP5uSL*WjKy^R9D(m7>vGhW;J(?u zSn=Ac7~t)`?MHA?WxceNm0SLA4Pm=n92>O?g%_TTDYvkWL5(-~QU1;ZCfGAal#qTy z3WtpUR(5a}Da)5>ou0A^5z45CTtA#5oq(LCW##ELn`v-i`T9r!bU~>2HpjaceA`Uz>(g4t==DZiPz~51<NEW{wz-RH7&gm4*eAAG8e_gy|9nj|)rvAG{issZ{}~5?9iDF& z3SD2H>nuk6;|c3#4b%EDYGLe@v0AePcIX{%B?1lDx?i$>NlD2v39U-;AHU3SdL=uk z`aIlQBUd{3&0qO-xlNO_Goia%PLZ9-LQa-AYp0JUMvp+L8Nb6>tx#{A!}T$kT+C$( zm#Skv@bi8`Y0?RnwDx2?v>fS1NVK@0)iJaxT|K=iT5j}t)6O)CzL&$V`@z>6h$h9k z&aC=RDw|vT-d)Bm+!tZT>PAz;tCJqe{Y8s+@UiV<^dE2I)9))k81=Fk+Xy7RZHVeo zt3hXMW(5ex6zRcDDRkB$Y)tVI!Jx(tr|LNe_9)(C-J_-TELa5_iwxXtNNnH~h8p|| zUD2wF0>i^Zu@l_wjyl=g%x@;YM}wiu)oZtm7$5v-0t><=D@u$e8^s;&KC&Ds>%qOL z*%kBI#C70{+S}AhfB=m1I)ed}LH&0PS|wQi(s159mQt~Jh;V9^+r~`T#{f=X?_6HK zFj);YGoWG7!8nPpO=yyZO$nAV;z*zG#RC2K(^!<UcMSX?ulx4QELt%A)&cPfCt9bMcb$oc+;k7tDSJ){Uc;RXFLA2s z_dve)&GI(n$);(LO;TN5J)<_q9W4)5*b&Y1@BpY@L`p1Gkjd1^uIBc&oYJS+$30xd z2wVff<8?8xYbdn9Z$h9%-G~YwYdZ2z$6QNRW4R2DF-Dda*Tla#>s2Y@&tOnu=qffK#X4~<{KRpX7umQ6xYjYp0Iwx|zn>M*( zF<=F;)XCwFz;*UeumnjO%9Yh~w-xUihD#NtgPv!`>B?R&-`r16qx&prlg!DSTQ zpYROi9D_s}1r_WSr*^ph0(=-d32+2h@VQoTD#ObT6s;*gnwn>pW6hF2$&<)%aLLl^ z(%dHqM=@}yPisM~-%rPIMAptD7Vi~Z)GYV#O3{jwk))j*^PvI0N$SmNheoVJ=B@Y? zrpHC3r@9enc0v+?+rn zE8|qx9r!Ew^5B$(}vF&s!YMJ$5W zv0}Kry*&(Aq4ptyS|b$&>46JT(hIsKP)MJ-51vwqy}I^^(5FUWXK@(n7kNpDG>@-0 z02^769(1@^7SZ?XT$6Zex@rvF?qVXYy}{r%u&TP`7#;oAIsJwcdqn>;_R?2}f)mS# z{ImC5Sd6+y7JTfiY*5m$hyUMWQb9B7Nnev?b97^EJ#60Y;Dz3eMtV$_O8V2B6qfHw zGYL?O?Hy3f4^^D62~o4|Cs6fx(_yefa93K z%DY7k!Vz9>F4IZws@Mwb$trY41tbok9>A2=AFlnVtsKf#?~2?dezVQHTDPT;qn!TA zTE^#iBPhY;yE#<)f8RWzncltL4bq|0c!O7M!6MlFLdVV8K@UC16U~R{XqqZ68(3i6R$~BI&03{y zV(@~XNtX3xpIY&ceR@{bm;#@fZj(z!Uh7TNWy7&H(h{jmAr+A|uS0ksY({4wy@-MF zp2?L()!>+>+ruYic&~h0AU1&#Kbk~fzcrn+*dK&?`aAH@23#b$$NjW(%^H79=p1+~M3hZuxeZgk3 zpEl?sOz3fhlr2n$E`mM0Bh-Pju+j#z1)*dAiDHKdhM@2Q6voqfp+0)xPfG~ z1Cq%E&Qvbv_Xysr40`6$d{3Aq0&E`8Woo`L&n+MKP&(x;30Tkb(0uT*olk<<7+8(n zS+dQ3#Q8gvfAh7mscD?Hf8%w!$546(wy~c&?Sy_2;EhOEPI=cBc!SN6C-0x+k^X80 z=6Xz;u0}5#X-_I&AA?k^yyXQB3DpBU%U0U8bcR*2ZHMaGiCR z45~RzP5JTqS?e5zpOUGg7TPm)^a-&1CXZu1awQwkuDh(6hd}ZQJi$KtKsh$Gs*ZY; zsJfB9C+al}r_jZaFno((@(UNi5fU3_+z)1Hm&gYx4DX@ep1S<8(4(-F-VegjT4dyC zlY;5uWUweP%nper+HuSr1}mjz>Rv|@e)P9cVeT6s40W+QhbB49k#9cSL!oI0X%1V! zl@1QmewE<4T5T}jVkAs=D<#9vWB@o4)lbGZ^J0v!DBUX1uN~oR<8@xt3JwE!>?mSQ zuI}`3X{76gnKx*xm}y?AHqYdneUyEAx7fc1%r@(KSJ~$SxubCHCa$_#`=O1ra^5-u?)ouJ!R_)*c%8IdBI9-LTdkw9ioi-S9Ha-&xCVfeY=W)I6OlNC%{=^b zu~Nk&_NtFXdC-?=nWw=K%=3uv^Z9-uB7EC0f*q2>_0vtv73mS^4=h5bDL?CbpX_Mz zAfr?#v19M3qUkYBlvR-Q&AJT?s|IFui>q5lQN}b5N)K{uXhA+QP@n1dhsMtMjGctT zxNup^tBI$hR_&`RLlYBuL;6O_Yoo8q9|JV5F61$!J;w5Lupryd?K&Nf*_*wZlh0LO z8VhPHJPDX#cEhq^pNH#dW@k+kUp&u-lbndN+H&Axg-O zlFR$pExD7tvVkf(C}tNz4dv-yega*og|^~>%Bh~`O8Mq@%OdoI`%KRRszS2jU0v-`TmmY1S{W;PQ#u_Dqk$908w;68s&&^mysCRhhgz`DlcV^PcISG zGX;mi0&6hp54G=S4IwNx<`?>=o=I3r$zQhkr(?MbOD7%`6v6IslqyJnWjpiY(I4=t}871lf$qM2q5P zBA;soVfcRhwK>~*0A(Ps8vwyY(nFiHXcmjx(3kSWAO5X7$6BvgQn2ASS98qpg}fi6 z6i%?(qge4tqXVX()qXPE=ICyNm9QY8K#d-wU-q%GQks^;)ejT%xi4&ed6^*lQ(nI} zxgl0UKT@ta(_x9C*w7^S^P|-Tib-+8^qc5!fj=0mmQ2B}O4|sD9wu zHjyPyzree`5F0H24&;pDP{N$nOEfZgRa%)xX52v_o1+98NZIjkm9~b- ztDlU1_?K!Mv7^Pgsxu0Qb=wNbhwZ)W&l@>{5CbCScsHj-Y z9~zsPb%E0`K0Q7_+-U*cYBGRDGdmlpc(bbWv%6N{r$5b8bf-t!6;`pN+;&a8T-jhm z30ul;*(RnpAfHbCycy9q9BAUZW42pg~2{Dk2tn2JH4 zzu==1qld{irz0!`lg+Xp;)Ge4QqdU-=pPDO~2;jb}#-jrDDx&bfR^nrD3s?lh|#Y%MppDoerL&;;~y z87&X72+2{8gxD?QU@Jp4~|u?;|Kign%w1?U2xEo zrwNl$dFtCF@u+Q0e~#xnO{g{qAYTByBD-hf&NBRgND+{Pg0pHn%Aw45`J?p;>dB^T zBdOT;x761gdG~VH!{MCTpk#DkDj`8UQu0n3mF3pezyJ54rO=@j_cbkI9O3Me6E_P6 zp1_X~jy~T!<5f2TJjR-~qk}2Xsb3jqJS(&lD<2DQy9Yk@#tmQO-AOA85mNUkH0rPj zqa~!nUwX{NNl<*p`a^8|fwuRoIfUH{C+vXW+Qz(^FJD!Vdb?V41gP^1ii(NlChrMc zeH8xnNA4n(!6m(h2~DGClJeV6Eqc)V*)8FYGRU7}kf9Gj8`+ka{|3ZwxhkOl`R3>fosMrz_nnKklzHqLb zuvCzCB5YbdV`qrIb21;#`dFrts#5Qe(#cmDT8O2?MM949lK%AWOw!Ps9V%*GCjobls62QoGdewJGk@uHMe?5K38i1A;%<#t ziAje+M^CfMZ%w5qx$tx~Uq+RB*M&Td)jxR^rvsX^OEp!CUfu5M;xtr;SFZU4V5qhk z9=Wl2YB1xNy>}A`Xx&Py<|F|9gbse!+07reIJ*hL1agVOUQt@r6B2>^XvfMF(`}hJ zO1CI8a*IXKLzl<=ree=O6{cK@h5C`mP+Mr*_L<}r)(?-sHJEwjyLG0?FH2)%WBrDu z6}gt38W-kG|IyI@$@|`UC;q(O<9b4Q@qKHrm&^_i8N4&$W_e+Se+^#2nha4C!RZFg zUjjrUIVsUxRf{LGZQ-0mf~I<-!Rey$kFm;|rw}l!Kli43s6N`sV0@cW z{>xwJbVCf~rV96@0!CGlUTRCp4b=IG0`=x7sb%Fkx4>XNE*>Y4ytgniPL-w6@h~A) zMKCmOqx$vi+4-B`$4xyghMHj*_RHi0{h{{VzIiu;(Y}Sl+^y}>I`6`^@&vem`hG3^ zu9`5}WwiIF^o`ML%s~lxpTj_^hU)%tP=e8Gn<)4$$Dexn`2~+EsuUqJCG9U%9=NUL zAK_*>{j%uGkvnQBe*>j&WRpjDR^{- zlGt?i?Qd}+*k^V-i=GUPSlL0r3gepOdSeZ%M*%zu;_^`L|7}kA7C) zTd2!TD~9hXVip$9Eo8MsGbH5rv#2UxcA!-R*;x)ng80Q|C)q8Y3~x125n}BDf0hh$ z2u??>1H{N+-uI}c+tP8$P*m6;4w^UwLZ)pzJ7=|0`0Z0oKSfnDEKu^Hc}M*05c$EN z%3Gt2sBghdf-89ha?u z`{SvIgzpo51rfXj3cJOr#**NJf**j;B95x2$Sm8)_POi9@$-Jxw(TvxLIv^A)912= ziW<%!@duL+?aUY^qd?@Z_i!C8!l@YCOoxl+_t0q}S&hueqOfxYA-a#q{?) z1Yz!Nn>6!Eg`@&eX8L=!LR8~RxE+N%7&M+MBI6}a$+L4tOB;;YDB}I`3{1)*jBeXR zNZ$91gS@z!jy~9XT@M@^AJ-MY_wxD;(HrmRc*q1h4exW8NWU((|j1$A#vw>0sj1tIT7!o$dUi$|sbm?q?l(_=L$S;=02< zVxfU!_-=(g1AltU3!;|#${ZFk;(E$B=I|ij>-ZvPzC(t@Ph(`&p9H|kAOMcG7Zb`+ z`_#hdVM4r27qx`cLzp0;a2p}U$zb;j>5m)x9tLe3p3X#MYQL!T$5-Hiis^;w8QyI& zDzzr)ss3&62)3YvK*5eBEK+8^VW7=r8v|yE-&8QsDcD7#g)&<>iMCHOi#`rw*gG12 zV{Nkwgzi}X+3;HRb}jSrn|-Py2OoWrU*EEn9T$`v-#F#yD`rn`NQRg|M;YdZmO^pp z{LLe_!X{Hrr@kBfxhZTC(Vu4skJ5fF^=D)skse}7Z-0HxIT(NM|9cxWnX!{&%o!<8 zO0H=#&vDeA8ex;0UsppFdf2%XJXiiN&7mR$&qE$KbIeyLOaV9Lzf*;%kJS>9P3xH? zcLtW2WWalINxJLh7=5>~*-swG8S3-lhUHAzE}YWzty9_YpH(p(nb&rc)=N|rG`Wu!ZpL@k zD+HD^Oa~J0vY*W4RYeIM>fbE9uHFPShzU8wb$!#@0)z z38dseQBO1HHJ^J?hD~DaE2EfWQ=Dvezc4oyEd4w@!N}m|^-Hfn`qlQWp~tsff5-H! zZpU7jSSe}2@=RC5%x&|dRv2-soc)TDg0fLY?ZEMD%d+ZG9v*5k?ECESF*_|q6D$!2{}3cP83yPsD#2gSQ&;%3o)i{rfsDM zY((q+J@+%aGTTYG9wKnOmSj{)Ca4<=cyC->2WaKnfqNUWvX8TUyHcZLAYiPd4vm~{ zG7H9}+e2*NA+%+IXZ*_BHM3~!MzZ>^n`TFwncqzQka!s@1=&O-JPskJCI5Kw@LD7( zScMFK?puH47xao)=388Ruw(M*y8wgHw;|8ehF7*!5{fpBS9j*5u_>+)rh^ag3%vVE z`&uQm7uj%(7B2(V?vxlHo|U(r1Sh6=B2&7mp1G$_D=mQ>H0At#`I)WIaE!^L*OGJ@ zx_rL>;`~a?umvZ=$wQ&Rdu(_ngML=z9ct20+mf{305|ZQLGRJ&T>p8JAt7P9PF?(u zy%ydya^(nFvG%o#xQn=UCu2|i`RndbY9->%l;Yr@+&v0C@=-U1?~|OzcTp(58_B&^ z@IG4Mf`Coz@{yq-wH%eqU?ONh0tP=!YC=|oyndgawRM6CfxvaJk&_bQJ!JH>18xX4 z1}@pQaUw9TaR945JqO>00j36~3hGPptA}ltj_=Gji5}cLp9O)bU<7nRMM=F74|C{(E;2&Z4LTjXplrzu)uSqo$^2@+!IE z3cCdF;D#?Y#18<>zakxvPd6uLG5{ENkTAd$BuR=@EHovQUokL$4HS`stwcCyfyKe% z!r)p*g-W`(yZS$~$_VQJ^vN+x@j^rSy(-OWuqSEFbpG453|?Q380dWJc3VA=rL&*C zHKl`YJNHAj?&mF-OjMJtE>~yf@2TjWqm-pj9WSC63t;dvSavbF^cXMajWM>%uz|OS zHrJQT8Y+00{SY8a4PZu8WyneljA`(Q(bgld(+pjjzw~euS#>7ozI!m+$;GRBa7E=J zIQtRM`i9~cvp(^0km8F@N`dWi)K{hk|6Id!y`isZxWSV57F@ljUAd2p7o)|pkL}aj z&u0vvEu!?rB`D^466%2IJ^F<8bJDAN`8CJG+>&4xH58&mO3%N^o)Q+%Ws3kdsmfS{ zVta2@Y5dS=e1p7Sd+#f1Yz#1r=}iuqN76uG=7YRx{o9w{KP>=dmu6XUEQ=Jjm-$Ss zso~=ux7xrF+04g3yobSiS~(1LS^n0F`~?bP2dA6ODvYK6<4OOp6MNWLe6i${A_vUf z2g~MT6T#wGQ8Tx~q%SK%3mC58eu}Y)`$^b#HxDbTBADXSdhRZV1IbM%0KTb$aPDqK zQ3JjPBXGw51lR|ni$kfFn0H8_rYOzN*Bk4jq06tsDENGQO*Kj$)q`~ z!I=w=h4+;7(7Iy`t`AYL#$AanN@5{CYuxo`&HMM@R?vzKpG>u!BEz`UZsYpQH3Tt8 zi8(b?9E<$W%#XNnHpG}rW2pBcaXQc9vWjnop)PBW)fNZwpm~z+@@mySUHf*F66~=n z(JDsZ^zW6fqP06f#>fIqmetZrWXC45%i3KI>PLhL@%nKO`*j*~{O8#MGNM$VaKOJs z3%4Q2Ifd}m!|xk z3qB12hNmrLuj;$;mhH!uisu(j6xTu|%KOtmJo_3_evz)8eh}VU{ZVfk>ga#2altb| z+ZH<=wlKKcUAuq- z{0jAJJ$mk%p#?IyySs0K$)om&7~lgCy#!1aX~6Eu-+E&wp$riF3_!O{by>+tPE2G5 zyFT&gvhg&E4488|HLaoy+8T#s>1!vYVOlRgnJZo)g zPZQu13dW12zS~_zw;9gDZG6zo&o;t3MZQ0!n z6TBz27E*&VD1uESERz0I{5%Z#Y0;vHzf%R9Vph91V}I7V7EMp40PW?t%He)~HasQ2 z8seOSwGUB^U{5r{zunMF-v!JL-A@nZO~#6!APB_f`HH}<(hr%_w*UqD+UFk7fX*9tj1L-_{>zfSDzOv+-MozUPm@u0%g9FyBD^F+|9mm(`~1^20IszAJW#O#UvpqOWBS|p^kV7FzgS=^ z6ROq_cP3zU^fHZZw*+EmiUeGB0Q&?m)J{_Z0G+X;%Kmo-u!KXvko)%ahP5uXCQrSg zkrYtxwHIwl?iD|4!>I=P2qsHvtvWQu-)*5W;I4G6|N0DAy8Tm-*bmH1Zqg}z1k0J0 z@kB-g)I&-9e`W9)HhR}*_5-Pn=P`qi68Ro33rgxbTs-ZKI^CA(j}Npc@`{FzL}ruy zS`QdvI7@F`L^(iRxaWIv?I=!+MSY=yVOk(s<9IsVWL6(crCucsNLm;hdw@#n0s$6g zy3quZX%FSt(f90>f+ywJ++NOVdx@Z=UYP`X9&(w|r+L^4^&UGVdZXxkn0i&$$@sN^ChE3Y zq}vmsp{LsP-!Uhx6tZ<9uPyXf7RG+Lto2#}GH*FHMe0x_^V^R&m>jKjO{JOII(Ilq zVTJ$%tR| zJYcC<7#0!*WNYO9f^sBmM!wLV%W1M}X7gYYuWy&s5A-2OpoEA9vgWv9Bv2YflY8$s zBeHIAH_ie2Ol5E3fV&byJ(#ME#PS!f`jvM922B2zui*al6_2q}cg$oUO&lgZ1ZL}ucI+7FA@oM}E9IkydO_NiRlk9{D%B+M zrG1JUA23EX0i*^*^dMlPDj+%9*wfgxonS94g3;_ zMT4Hb3iBqN34#AhYj;uUo*`HbS=_?U0L{Xa-tKhm>{QX8=ge%p#N>g`>5Dt$2{c!b z{;FWK1&y#Ode{6^G9ir|;5{go8+spz0~$Y*qK+g`?a@ZkDvn6gYZfMeE!NWCD+1d> zU?OMSkr=4(P{zWl7M{&*?#yb8mPSMGJlcDwH=LrkhqH%7#8C z<(c`8zh5fMyoO>3_2a8%!gg&w`cKf%EPpM^kqhCJSLR zm5EikA)Dx$^@xOTALf5)v+ZAa?g&rfH<1O^+niQ3{&dYoU#z!#9sFAl^{yN?#q0r5 zU#M(jxz>fZqcBWaGzFxx5p90%&^G3wBEu<%B@MttTAI9_VE9aDtts(4FuSG!m%~tr zF^Mr)QER570cF{JP^F5x-@q} z8J!jOZ+>uUaX|y^SW9Dm7fiM6 zaF(80&5%Sa$B%zoW;`m?Wyi6#YxH?DO2-LV{%tX~^>89&%N+ANi?^3Nlw**Ag9Q5U zmIKF#Q&B|OK+>i6&qh%h`s)JmMuY;?6c7VJz+*cM&^0j1|3JDb>&hQsQOD3<>{kentA+t zTWQa~8>v5Q;bFXcO!+c}bml(%f4HH4zZT{CL=l*EUc>iSIZa6>Jidgx{Xop%FpP-& zw03Zl4zQKSpQ(C$DEB#BpyfH){)PmkJ$$W}{YW5~J7V0`<@7pqYDCr)d+x1*K0f(R z23Q0l?y@(C)}bIeoY&M5kdr&V%{s$aK6G7GvjXA&HjgYajVDCh=) z)9|()X|Np>x57=XPgl9D=qQekkB=u+k-U-Am^-``$_y*0GkpDJ1jemb>T7J+*j7^PMnE+}RXq&>zoXW!i3+FK;-`o%8?5rc z5=ved{R_PLJ-?6QYRe3gCx!N#||pMVjN0rnigy5fQIf%ka`g_)$%E1o_whnjbe751`b zCMH<1v07=7$RXi{bpby9Ei>phc;MML3~bs!U~=P zZ#Y!uWQY&>p1cj?K6=TMD}sXbBz~d{g!JkR(;VSQ8CEjwz?L`8=|RE(wN5j8kbw2K z5+me5UG@ayK*zh8iquCzdWF#+fHBC$?BFqq=hP%~g)2)|9`cXgmB0B!(JAhJ}TS&yE86%iqhpZj6QlJq(tp zGR>G{a4_$C-6N&R#_Dp{t`JQP8%q;|?Hk!spH#LLBRDP(jSau4VXv!wxUhaFn>-F# zX-V6C`AfH`eCyXb3>&!2u5sCYhqbH0%)j6gityT0ls*)%xb}BiDn*fmH3>WdK#^sx zOGU_gYW%&-9C!@ybp>Ot{(#Z@L_|c))@}7hf)#K3OCx#9ei&q7VDGC8U=dzPazv9& z2hwxB!unkuhzG^LGo%`k92zRLJhPO4$Fra%bK0~h0}WnpCtFtn=c3)!JVsL)jL2~5 zf`1Xhle~ajDE#O^^#Js9s)NHEhB4oF62|2&n-jeoO+SMK`tcIL%n6-~*v?x&Y0QJ2 z*ZA^Q&P1T}`tyLl)%qqIkn?R+k3f6afj*vDVb82a1TJb|;%=2io?dmy zi{p(pXpzgGTtr|f?JsKo3Q$fHYo$*wCqooMHjD)3S=kEPDI0CBm9bBsOH3au(a`=L zZ(?EADZ}hQCe}NP}fjzA~48(nT8-Jjg=3hcbklT!qD@9pNRt;42PS`f;-&agxIyu_I z%|7nv`iwy$PMmOk2=ZnD+s%=npeoJ}xUY~v2e!TLF1`TuXazr4=2r-~aQ?6#f}*{_ zN~wdXmaEC4^RdPKvH@wutXlEuVF2WqPj{O9EBm|%aN4qwmm$%s>4gw;Z)%W#Qn4V zHlbPY>7zk5n$T%d!5op(u@h7ZgzuI<27$q8S+k^M%P4miQ>{=EJ0&8$FZ?1T$^a(( zDfBZncS!%5;b&k~E&E*`N=WI$lEwnw`_fa=?=Re&@swc@W&l`;-@oz$7Y}RmM7xot z@zN|7_+kxEEcosS9xZd2_ow*LLcO!m9eq@Uh*%KLu=JA0@)$dNo4kxn#5V=L9t|_F z#5E!rm+Mn=7bu6A_8dG4fi*%DKqIbBiCi1Hmv^g58tcIqKt#YQQDKSv8{y&5V|0E{ z3TI%uw1>xf++>2ApN+6(@R{E1xmM_ujpFd7MegE&-R{}_>8FHx%R0meptTk8YYi-e zHwJ;5DOp@%)<;Id=VH~g2w2HC=ro<0NQ6?Qke$)>y)3x~^)&?{f?kJXgT*Mfg}m#S zS(RoW279l};8wnR9s&R()Jmlv7-{o9)SB4y{3qnju%E)%YNy4ef6xcn*Wg z`D?e>noREolmIxj%OpnsW1f5A!omVVy%)&%1l+N2GOf3sh#ToU0$P3-IQkRy(oSgV z0KUf7kbb>KIVh54ft}FpVrGV8fC6&H*eXFyZrg|pK;IFWcu^`kNgj*zOQh6V_~;S9 zfky1>Tassab6>kPC?#$f6Qy&p^nTQ@VG>+S3lrYQTK?gor6E`8GpSVMC7v^KkK;Jt zi*{%&Cu!7hX^C~7#S06f94)?MmOI#<$)^b=@?aWM-oqaCSVkuzDkqG=q+~EdFY{rJ zT2_9P4E#6&uk%*EEbJ6IR+7vdl+Cuc=fhQD*oG`P*4BV5)q{XC4cIS_N$Gp{iG(4Y zJCuhj5J1KE3Qlnc|2#)fe8LhwC;=44=2&h!I{-LSm%@LvTsk9*1w)4Apxl0B z5#>CG6UEtprXj{P^`i9c9j@Ly83ThPO#=sq5@gy6@YUFgx=!-oob(O4`JLHn@#3$b zXP?p-s&+Kfz5NDgmd=B~8``AxsFTy9HH4@Y*pwL&9{#JtRl@u9z^r;En6Z_=F&l`4 zHIhng_%k{f$X1^&7d5Y7{~0qR(!{%t=2FN*H-&3x8s9m!`dkD(?A>e6rxTke2Fufh zx!i=pf%S@|LI$+)#B0um!jukOrkOIoZPYWgLoEikw=Dwy_{mFAJcQXCC3lVXr8;+B zn#JR#1JS$iHyCCo8R&#|y6&o_y#y+fFzcUn<2UWduF!a6ttYju)gwF8B87pJg$Dd% zw2o2p=_)d`{gpk$uqGOtVC1L|-8IiGzv~2 za-n3X0tVNKM6Q}8!L_=#9kFE;kklUCKOQQ4K${6`N)=dipN?mKuzGs1+1xk}&rhMd zr<8y;x!YJa>u)wYLmKaOnXAk2!7HdbOj-`qAgSBzL_(&MIs~qRY*XVP-pfu@SX{O6+jSEHDLEGQBZbK9y`T{!~1y_JpRuUFfHvaHd- z=t0@e8<+6m(r8gkz2}PI=8h+i6@wgkLf1Ko+?m zZufKsm0Ipz%8H@N5&-FR-k`ZQ2qynQLnIuB_E&3!C4!n~?RR z@BtslL>yf=f`+es7qNE)8M*^33ZJGIL-V@Jej$p{q42I>IwXTY@a=LiQDcsyP8iMsA}$Z++#AIgP6P`D=u6}E^q zHuthANFh%0VKT&75x9Req^+zQv$AL5x-8Z3wssA-`f!CMLOFK6#0~Z7C2|fjD4`H) z)~2K`%0k}gwQQ8le=Uw11xlbexU>2FRrb%i2(kd62_tq$#ms+dM03UUIiy_!?B*jY zoJ@krCJ~bKJ|d6Va7km(`8{~{^;+_Z5D?y-CMVil{OmS2jpts|HdGBv_?fOehKpPkcH_BYK!@GMN78nBnlu z2Rr)qn}SLj@dZyWFZFuwnmt@EwE5AGE4cg=zrVhVjO8%v1UHxx+Adc0k`Z834=sk; z;Yt?G^?%&w6X%sVd5*TCzjA}6gF%yapRYdv761oe*$@*pwbq%iy+*Bzb(g$8)ICMW zvV?KQzFbw48Fl`HNg6$ZNHewYOP2*JbPROiV$vQfu0m{jC+ z^xv8RhW8v`CloIklyaL6F)42~YYUC)yCxF(=NOLhl8}%Lf}u4dVMXX;b4iUi4q4R- z7p=2A1$}i?!jX~ec-8Zw<)3y^W?fH^_iefc>s2bP#TCh@|H3Edof$r*5reza$_kbbHV2#`(m!%cTLr}kW(hui`gMOcUQn|JtgJ( zW5XUDKG)Tu>XRM!m1fK)~+6wQwn4H#wq;b+~-F{y2TNGlUUO68pMkDTcCrtI6&C@r~P=@^U(abPpU4qy!S9% zky5Jz{PdC0_hZ#%WlwOTGoH+Ms#HAHkrKvP<<0djb>~-?lE@rXVURv6B;qOT964#w z2AI+hbrrT4Dm$vAYg7czNn3#WwNa0vT-n3n1?;u?1S(_M5h#DofcWALdZv@ra|s#DRohTs$l3wF)u#DA!o_YV^=*t|f- zL$h>@1n~GzUa^@0N%CmaD_5)3H#!+R_3C+5Nq)`VUiEx`@XFD9?(lQZ!16y;!}2|*M-?Tl=H)W`v_2}eI{En^wRW2H zHeJHpi#i2UpR!)iy5|!pp(Hx4NvqVVBT8Q{P!VJ+rN7n0#%| z+6_o_)0VE1s_d z&p2$R1Q`AM?%pTPP8~EcT7|vc$HE&ez2*0+uIhwnlPw$&A<}uw{;wCTNzi$u>xio_ zoLP2K;6@X{z4Ql@4?qxD3AvaT_Mwq!^AfmWto9V05)4%Fv0FlU9%}(JL3x=J1)h&nVSwFSz;Jk}*T& zN6BEmXL*XqyIR`@(smYZvyUC->iN4w7Krc%L@Fjn!6zm5&1Bl!D>plOI`+0h_WP(c zy2KUBHfGsX++aa_bM3&S!TN5~53*Ih%V93`{!tua-OGSUG5X|c6fhJw{OL~(SW!p> zcgX>WW10tJZL*wh5Zr}@0jwHYx1rxwS zOVjAsPIfzghkUghn8`B_kDAP|78227tHXJrGjm7FJ+Pp7wDYr0K)BMpPJ&*DN8hKP zGYQE;NvRIoqr_eR6HEY6Kc?yJ-iVv_^pK6d2+OG<2EBgosp(Hv1cwSYckNNM zNsZCYz=;^zfi4i4dCXq@>um~{B{g-I=p~+joKaDib|1`r&&DPjz+>)uiFKncN~qT! z7&8iru%WL`PcnIQ^P1`<|Gt8LJzv=_Nr|nKycE~Yfx)NSp*7VwfF_I{6jdcrdZtN@ zZrP5^(uJoZsa-M_-l+~lTgc10<^0U;F`G7@&3Rf{V;4?v%Ca2z+y$T-Gz43Z>fGEr zQminUC+&=^PR_4HqRV8MeQb{|a@|!q{jd_3WFTK-IMv!6k?`5QL_`}y87_&Up@u!A zEkj+DNMmVgHo@b(@JO$T`E ztr0>EhYR1!^O`d6>V=uRgCKD3r8qgU!0$6^G=KZL-bq1v#yl~Tq>rT$$zKEn!oq}) z2|k=5?{{$MywYdriZjF}#M~0pXU)?8CS}A-_|~bu&QC30 zpVMe~WPT*V+_n&W#D&P(B^Vm)M8XFsd4f0%8qOaZ(W+B!|L>=1Debt^hGIg))2PIKRR^Q%zm?QXWRric#|O(uf9R7wMXB*quo8gS%BA>Nt04w+KT6Y^O*(aEx$A&AFJz6r zJ+-Bgr$+WNt4a4sMQ;v&v+gMwy<3*)MZ++Ha~dG9tz47vqgvcXA*rY^mYfPTVQL%X zjoBY}%(|NOe;qKB&Jb!P{!aldl@f-wtk@B_d~YO90(R4Byd=> zRyV5N0uH8QoBqCj$gKl58BbhUJ z9_u1Yh`wb^MOp{M-Sk!@69{9JR2%zb(U>1st?E+viE+s#pO+%}=U$NBiAGGyp)t*W zPxv)Kzsxl~flmLIYABgm6OE!Ff2`TAPL#GCb=BmkW&xKaH}gYRXQQM5-QRXFcW{eV%N{ zy@q~tY3m6+f!6{s$=!(prw4iC$x|UEX=NqNiA$ zU(FBrJ6XS|Lue$L4onan*6~_S&qb|NWfC4clOVIpWz(EvTK)j zzM90i6Lk%>O4QcThFSaYL}LPN*NVk%!Hm)9+~@yq_NG{b_T&0jw#sk#ZTa*H>`f9R zsIQ9wtdMWNQ=$j-1h?tKTXYD>IKlV3Qph#Wfd4Gb#VlgVWpRY)$?RRir?-)=L{0M(VFWt*q9ND*fae5d3{UqOHj{0(V`KKG*taw3Ut?YZgZ3*pi zz!4!a#^Rz&eBw$P;TZ+izBbU1{%2jA|9zl@=+AC3WT&T>u5ns;y5t7-0Z{4=0~dV} zV7+lqDT4wqLqq^!({!n_ia-e1DM{&v0!L0TGBC4*qS*IAarukqp?pv(@ZaQWb)!9B zLnq?fvJesRi*ou|ac293|3lYVKt(BqS^Z|%Z z9EXPd>gT?)M}Ez1eKF2AM-!GH$ehveq0f_fE)*(SYVvH8yKh4MP^>LB4#KZXef1v@ zmigyPegV&IkU4B;bt^d;cv36&Btq&t|j`t%1&8>Ld)~7p$Wfz=6 zF>ZALZ1l#yKCk9K zi2aGE&C$(hvx|h_=m%$Utff8X$=xc(f_;b!s6@GGw7VC0Qcsno@Z-~;iG+ykO~70t zUDPuu(ieHC^x2;QuI}|l=83s^7vTO_Jpqm~s2KQh;D2MgDOgpnYFG2}?Z$x(Ndlt# z@N?Y{C9%`(R({W2^Tt-tfAIj=cbMj*wvWy>KY!{uCx0)-Hs^raHhckm^|=4|>dg31 zsy>(kOybs4ob#1Jja^Ws|14ICw%SW$t99LVSx>g-NmjK+1UdCIJknP^TOCaI}lpb@xs`Mq-yP8K%&e;7vN$33>#vn8yaKPC{Q;rY1X52ChYI&2&XiYE%8qVPn*HZc zrjChncu8C?p%Hfd5A&{uf9}uwOvI{*yHM-8#)=@WWG^A0L}An^d?0zc0#0;F@}S}Y zdKpBWKuaf zIpD$U17eB}c0K}2ee~>ktJ4Y~ce`<7>lrdvzW@iZF1db)oys`TzZZcPLIjqgpL$BY zf$o|6eBJUp?m+sidwGI4*|K8fyCc5!ctc-lm1HicIW?OVX4&26z%l(vzy{P9wyU;& zeF<;5IXvJ3|1oiEFu$Pd^G!vz+`K(ovw@jQgRC2m14pl` zlS-77G~x38fY;#CW-OOQ-Gvn0FJ5(O#mIj6D)rm*XRTphAHIJ!8P?MSlc;S%#86w&8kJ5oTTjnq7dv4jH(NeW*E5ZL-# zHp33Lxj*lN9xe{hWSE-mpX^)ix~8e@1uP|YL2j}egvjQg%olh|=;_4pTU%hWR z@IqcK?47OZrbrd(!Kr*qy~yXsA|P3;3Uht06LO^_l0!rBaBC7sWj|09SBQB`W3ShT z9Ucq(tj0c@HWr*(gxKj<$>Kkui*NOge3*9!BPZEuCqea=0hLQ0Gw4l?4)6KZxJI1J z9|G7cp+pGXGkiQ%a_ous;rYeH;MF-+ijnB%UT=LrJT;Hs_{D37d%+nT3(1((t6>4C zV&WklHTAEvQRIj|c#vRbBJDnFrY3e^n^i~Jn5rJSGv8Bx+xgF=1%?BFPNM*7Ms_%L z8~Lq}QwUF}LuPYJ}ly3VNEP1A5O{wzpPV&d$`KL2LON({tC`W~{yjaU%n_RkWj{6e_a*`; z-VHn1u%p~=#y@*wCp8%8J4%%Rbl0tzr_OE8{cW7oLm4|L`E7S`Iq_N$=4;!#=Jtr2 zPwxO!BT2VxH5qivuHwr!9^`f3G1Qcpcd?Ln*SB1s2>dpe0>ADE(9M88e$|v3CFOx& zFdC*he%DLnF)#;fVrmHxkyIH%$jNn39)DDPxgJ3M`cE7r|0bZZ-u`=?0gIR@{*{M% zG*go=#Hm1p`0_GMiNIx+Zn{Tn>E)@j^T|Yunqx3K1U7*-2bt^O3n2cBQb9Z2@Xfc} zPy_}gTz}q2GTV?ic952Tsi^?j4zL8$>HPu*goG$i(8UV_ZA~&(&3S)5gRmB@WXWVp zV4*6ENjva2r#dJ0Dl;xMMcm}qx#$1m*DoiN%Qvtfn4826$cS+EeLxeUbs)<4GPyi^ zQSMIbphh}Q=UIo|h}PZZrb!dB|`PS?`2b3C$BEv5J}P#!Sn zW-?*0SQ2XUEA*E%wY2bD9N$uZ{Cz0aJz;!&=EP@OvuN}Dm(r+uqo)RQR1=z5M;*(m zQ~GBvx&gS`Q;I)g5N0mq2zM-RThBk=(NQYk+I!(DEHTib4IM&5c%Nk}jZ6BmUjH*v z<>XLwbaVpw23SZmsBB=^7Dzv;tmch_em@_2WUS<#J1i7i|D4uJ8Kh8E?5fH@S_6uS z3A)NuUl7pI-c`7LHP5DGj6v5J_`oPL4vDjH#%JBlG@a%@KN;NV{%tZ0#v5@&)9Oks z+C0tGX;WowE#~gvq7m_M%Pm_NBye4{y zPymyKnF=|=`K6nz=O6dbod@88J!+Z6=!m=Cp_T1hKy*LJfu5g?Slp{MFCEi3yj9y0 zv!iJ%j!|%4Gd{(3(ENMBcg;Qn$gZ=(>)w zrVb(LO*I7iWdxeyUmvfFLNR3MF^7I|6#1C)@JIkx(!n>FF+wbEoPWOD)yZSaQMB(j z#{xWr1x*x+fsg*H^JfAE1_O&zUXR4=uU0}W6hWl+wUjx}XsKz&0N1AyYBm>W>mYg` ztaab2yAS=d9!Hg+@qPprHC=TQ;htylHW`t>&cAlh$Bobb*kNLW``Z@_PW~l|RmUK8 zl!s?foZE?fFnu2HRR8z}7E?5#u-|T`-JL3<#b=XTjYT>PGFm(dFK?)sH@m?d-P+D| z$sUeg_8C+sNI~_z7sbG)4ZX)006F%K_bb|m9Jf3Or(giRGKmN&tR>%7sa{_e$&9i? zFwv)AkmAI6dxtnKy*9iE}v#j)W?Cs?B#j}3KGjAS9xTuiNNK;COHO!Dbe!v$9sG#}Dl2Jmt7($8 zucfvJWhqx(R3}t&Sz;ySqW2&?-0`c4!#aPbfEM9c>1vcQwR5*kJZ*j)=WtaMFXfx2 z^!d)rnO>~jxzicuy||P7IQL1{lLJhb?Xsi6DpNAgtf%BF+U2S47K^{TI|2!3k>1e@ z#ek06>SUyb!n?{GzB<-#)fD*o`7=Rjstj~J{p*{eBZnJMwDKlU2J#g}s)c*AZ}*zqdGIc4l*6*fB&3!?c|NQX3lQ)Pf<`z*Ry(EhYGk(wAcyNv)bhTl zULcEC(g+j6qc{MFi=Zde7CFyrtju8PDV+md$EPC)XSEnYKv}^0)=aT=D zVr;D}eslq0eJDz*ellr`XkLH7C{2x>kc16r0f5%Y1t>JZ+iUx6S)qqa?;Dqw9%sLR zTs)3H&)L77mB6L$5RESRVCU5%C>5mKO@b?b@Ax>L71hqw+quIOOK8|GA4554e4NmBWv*8446!7z&p*|AK&)4UYF$}+8pO1N8+ z@a{GyLxfn$h#AxF7mVXFbjY^`PdkYNw6d31;%WjAq7Zbj5zG&my81ylZ*|@iANh)U zllGY*b!zKe`26k{3NGRwahwB@q3{ zKggq_TD53`EdpXLw8goopLf-VYRc)*UpwB`a?0ngtZhK~&) zMqUA=?QjadmNo6M&Z5Xcm#mM-vYkXfEE4^8vOXZA&ofL&@$yKde@avH*6+RJDbLza zt9u(V#YS^6A{Ca%SrGP@e&7wN^0bK;_4R6MAGdzpq3CEMk8j}!+QAJxq9vVwk57~h z7|KmrJ@n4&%zkvvnzYs<^0UaKg~M7W7!?Ix(BA2q?oFHS{-;jc+BMmhUsE$=nVI(1 ztJQDn>SMFAj6bc+qKRXFH-ddUmgskH(|$_2C{J^U=YU)!DV|;6c5&j-WMx-BleOmL zURZpT4U2-pko=OuO@%2M)h$_k+Rjm>pZ8U{la{V0i}owOI+=295bf>liO~VDlPcGI z^Rx5%k)y0MF$Q?-tB3K#S(zK?6`8HUuIOjM990kY^Rv`;h({zI-AQrfPyo{$;{5q3 zv4=$#jU*5JRjLGS#J3XZ$53=F-CrYdyo-cf)Yjhy(tJ=b{F6tjhYr@?78RI$V-r-@fFS1_z)~DVj zl^G5z?k}zu^S8i&S-~rJlLBEF4fD&+XY&v__zxEDj4vm9PrAIFvqgHopEV=x-(A{F z*y%S|R9adno$BoGKUv!ZCs@o75QWcuF*N}5Nt?)_@w6q;-8zfM(Ur}7`UZ0651fGNg#INjiNQcS-({?XZtDiKBcYit20P|5R z!E{tLOe2BYi@L1}BiO1nlJW!HfFluPZWiy;{%GOgXRW8XuJ_UNNzSiMJnM`l&qkgl z{p!{ouNc{bmv}3-nefq?Qn!wM?7Tx((Q)r|85vGWT!NKJdx!o8{(AXPsD8Sb+iHU7 zKs5QIADge*Nc>BGqIad)VY7|8utgPJ%q#X6J_`$9H(#H?zO$@gG$>t$M&?HY&j@OLZ#BVPYL`EeEB8G zpz{Kur}r;)%ezse2_M%^zq2t=`nzcH-Vqn^t16f%e{2~{y4Y6wfiWEU7b4V38B94BD;2XCM|*o zp96i02AP%O@H|Zik9hx_Y&Qj6v@8KVI`juKZ|B%M*+fBA?;mZiaP=zGujRuhr_?ky zUO(ok&WXFJ-e$num|0K~{$LZuqt7MxyVP6t0*RY!K zWi5i(88&Q&OZSS4oukz`H74ZgG)|DkKj1$8SU2@Cc9`pNe}A6Nv7Nr9Y4Kap7X&f4 zY_I?oLG^_DyZb|&>N*?O>V-Ovkp|gqpTJ7|a2liBtRG60zx23*wX*FzS9A|d%WpL9 z=!N(`-T0Zosh=%8l{sOkk{A9%TQOmMg0+#;Y_<|cF8u3x%e0ENwEcFU``X95t=e>) zng>G5!KElzX!OQPRIaxc?(z}J@L}2pH)HCmSm&wfxox>gP_Lu>zIjde-%F#XSy&k- zeQi7(d_jB0B^_!RbtE=8z((`q9nS66&4S_LPgY;0L1~h4W3n<4c=yPsOHbP_9d87n zU*>_Z9?{D@N&pOmryHC?ZtL0zlj4#!nqrQ9ts1RdztSw3x-WyY;!kqsZ^{}3d96QI zRjBKbI1{HfWBl67 zn>;s(K?cW?fOZm2b&t(*m2bh^xPEag4}I zMf#(eff`70e2<)D#^L=A$E-0Q+^}laVQS|>s3!b(m^qtF{Dpuuh@zX1A%K3^J|n7_ z3{tPy{Hn4+zEQ&~ZO~2LRg_Ge+anqSJkVkw4 z`F&x_#T7S91~1%uG@-OOt~WA?WJArY?&=;?D7PD_9;Xa?#~T(7TY}wQY}wwZC=n%i zR{{QNxL5;n)5h~!e14H%g_j`_yZWa9Ql7Ww!Gc$eVmU$--feSgarVy0C%PsRMP_C0 zd#cdvf$2xG7AJHVaPt?mVW)8$FjLP}61VpCcgEXeSfzm23_~a|^0-P=v z3Pv15D|h^>L>J4H@=v$V_iem_>%N{pU@u6J#bJgh-I(V+Zwe4RFOub&S8Z}D6qN^| zEQ(CjbuJ7ct8uZTy)155P4&6yeWu+JE}!Iq9*u82anY6MI$`KXJL|W|N8%%Era0VR zQ`-qC!Gw8G7YE|VhpR&vvtBX7;=E~{J=m)!zpG@Y*JX+&8NR*dy=N^_5V7aI+EMcB ze%Lp}x@%*IyT=M`QvO)i9i#;k%~5-3`{qbS$3#Nt9kdHfE{P0uziO) zOD67_T9`=E?b$aB;i9^(H1QT#i|&LJfi5?O)1v|&85ckC&O1G{H*4Qaj{W=zfuD{ga|Wj*FKSDM5Bi3k91}J3s5D0_h3tOhG#Yt2a8JK{%7Lmt2LJX`yF4F7>&|0r z0^S4SfNeu;4O7TJyvks7=5#ddlzwY7T5zh-NBAV!=S4S7A1WMHt)=~nf< zjWd$4ChyL;#rS*$r@MBgv^6dTIs>Xl=9&q;GYP9*ub4|umQ8fZ2dXPiWFEB`>2AK5 zAb;3=szgwjlzSsERju+kF{u56v=g6f+B0Q7*>p~#e6#kwGgk& z>G83ayHS<;Xhzp$nCKqZ8WhPaRlXW>&CB;XV6WG-_VG7UXOsIAJ^n5i{Z9M|vjuqS zlnWdSktwMhR}8+EH|hG?LH!iTDa7miW1cRoAzgCVabQOUYzeQVbl@yzJvp9VM9rg?{i3^O>jtO<2~R@^#(kfPg%;9f#ER$%9l3mI zwy@jHosCwNPcf>9$+9>@wj1)%)!h1?mEqT_YteJWy~lBLjvtp+)Uzf3t)(GX1liWX zC<&Ut9w{DetX>S$~WyJy0 zZ>(RBeli_K($)-dN~SbLZ#uGC4S$bsYTYPR~Ed9huirwzP5l;v_h@^Rf(qyUL+mfpgJ z;3%mg)_^Tu-HILA5HBlP-5aj^_wFXnEin$TZ{+bdgsx4s>A{UP-!7Mrbrygq!YH^d}$v0SMS z4c-ue*OH53Dj+vG{__-CC4y}47PhSF!Pyxv&);@xI_ zWi=L@a;1dfOi=IZxtYJv!ZgFKyLinet2`)`v(TFuKYRJ5GdQc%Siw}GZf2(K<3K)6 zFgQ4;tC-S&H8Yw1edGEXRsjCb(j7ds2lPQg9KJ@m0|RP2_4{hJ{H8oc?^myKcK;fy zAD9E^MqRh2zYij;S0!uo>O4wowNB;i0|(jSBh+mAH(aK0HOQ#^k-%Z&OQxv?yJ1cV zb-(a4a-T-i^!=wpB6{I$%PyyP?2O=X`H-BA-cWvcMcns%E%)dmS4c$J(z>-_ZNbox z&0g~vk$WBY#`}zo;4Ijk!Iy7B4^G~N_If^=BWi-F|E@N-;UFub(0lP4cSMI|&75cL z+h*aVeiU#T{3mh$XPp3#g8l^>??sCk?jBz+ z40tRE_0IFA4LOS%^reg(;XZ6){Q@F((ITd;{v_5FHl=0rwz)6y3d|NW&ty;BKnitm zJUYiP?^7I?e%se?RHKeG!6L~y@Li;qxzYP2!_y0|ZCMm3zbWP>VI%~+IykYNCg|8> zMJ)P&6|NC0B}URz-1=UZu(P<+Pc;epYo{*ugi4@1nB7Xb;;?&K?o=)`j211Z7ZkvD=`! z+vSFG43m_m>0~yk8_e&`FaXlUT_7seYrHM6_lGeZ!a``XctM_2i?%$?KF^ZX zdbj8ola&X~loGcgSQKd33(7G|x&~$o$q~O&Bn+?HDgW#yxHS(lIu?a682sy#!p!^e zfM`mjS#N7Q;yT<}DqCc@*@cs^Eb5zVCsv|-o9>RWb^H$&MC!8nEjsr=A|y>;XG2=H zzBmrKYs$M@bvattZ8s=fUT5y6=O=4UcD48KnINhpAfH1LFH+3u0=A~${;*r1o@}Zh z<-amYf*iCRO`G?r)N}d4Ybu@CFu>1A8owft{%toy!b&znd6Nw~d_3PN@TjhA#ha#D z_MfOc8o;>B0vnJE#lPy&a#_f-piaEuf2a>it?OL`2?3*hlGzbtl_5A{8ir534N4Y? z5@#lJs^grvQ4baix&wVfrK4FyE4$q}lWA{P8VxhYebl5m7_{4bQEc$NJ=%5oknhO& zTv68WecP+BTx(tq&5|gw@ehyQvtntu6>Zr|H!rlCi-K`jM#C-&>aw8$0q14*N+`4b zrxj`;OUj@{3n9?~V=x{Qzm@e*y!)?FSjrl3M78+zO=IIzmTm ztTD7%biuk#R42o)Kip^;Pa?Bm-=0gvf!b7*xNM|wPaQGm!Fn9Nj6_W36XhND^C|@b z_QhJCbme+E0r{Kr17>|R#X@KgQbc8O-6)jmjys(kEAdWxRMiX3t*gt^E0Hl3idk|k zmK2*SFk)Ea`x9%aZo8^O6ThuBK5|*`WKrpS%C-7!ADi9i(;KayLA|itQnn^8A>?Dc#cB{pPdzEc69J^sO^|0$XxR7J@|H$gwqE&sLQ z+*pK%yIAd-#t)0EVvbu&P>zQT$o44YHwjX}$s0n`c%~hZB4(|1hb4Pq=zjUaUVAyb zXAcqwhGy)ZLPAVfySyB-1=_gVVHBvXEBk4}V!5YE+n-rJNUi@+|Ed1{=OnUN;e@hx zRSfNO6@nZV)tR^x+=!flzHV-w>$LFU%U(sDdiTWB!%kCA?|$_Pqc1wj=_p5gu-SK; z32OK2W?nPr?&h(`>%!Vra6F$47AHg(JR8SW<7I!yldtxuhm<|G{za|6l5I*%F*sHqzbfT-w z0@f?wE61|PnPO)S3O@nOIS?pp_*`rot!8V<-l7sVk8v%AGIx#p=hKyxSmklLnN^33 zNXrQKjb&#O4@xIz<(2I|*|MWqzgiiOr71C&_whyA1y#Gr>^$t=?y#2yinS4xhzfgk z-75N%1m)8_n_KWnGMtT5-_D30$&cZmhlaL%^(mprCp69Z%N6;>v6igocB3-pUAyQB zfm&y~OYB=8e9BA@lq|eNZQ};`xa!}bD@O4XnkAooK%T} ze#@;+Cl$ZBb}<+8V5XVh^-T2p2Z^W#x`z}##s#{dcmO+m{K@z(?9kcl`wPRw1h!qL z)$>KHH+5ZhOdvx0eCal$O2~^Rm6qctztVu6d&?sP??#8XzaZg@|DO6$Fic}*`rb9( zMuE)Cu*Q7Bhqj0QQ8 zzaNq`%J1$K09El_ywId%io33P=R238|K=lsZec7B;;{rb6_BB@3_3hV3x@gqS z!7Sh~GvZ{klnd{An=fvEC+`fpnj`M;;~~B!bt2Z56x1{VUQh7I- zv^3z{E~-aigCN#`e&r&c2!j51|HVE16kr1}1Y zr1?m~>$1lwlhgYdY6Zp&BFfwvT3H_K76JK%xYl@?h`p|+Cl+4X(CA5QLtDGN;4it> zV5$ur8d&Bdqt$xr_@MMImEc-y%Q(rw#z55^3`oDs_M3N>kf<5s>7tNJB zMPmQI#xOCPUrsAoiSj^~)W`K7y7|kGA`U`Y^nP# zs4~i%1C#Urqn!?8^yX|SJJ_&(kt`2oEB|uuq5azf_xEo6h(KEMcXU_qkiLAJ!0PWB zu=-1{qbBa^LqiB~fEEYVm)PX|zfT1-8fnr04Tjp^4>OekZys~7KtOtbpynjaW^)J2cpq-k)VJaD+j*uUQ_%o@&8_}AaXA%AKgEH)Laq~^`xzhIl z-!zd8vCD=AAdPtQsOPAlagYDcjg_OXBx&-AIzfW}s_ojld}yJvU`Vyk8_bDBOVKi^ z0F1MkTYH`U|D+C(W097ES@Et0JH0pJ){0Y)kxPFkD7y0gju$|9^8-#0^DetOb`58iCZfEZZ4b_}?Q2+)MDsDrfGdN?!gk zG2)wqk!Avcye1{4UwdU}%t#A{c9R2`HlTg$nodO?xA3xJ5l z-T@~c#iF`h`2qeX;Ch)^p$PH`2Az+wkv7%=!n9=_}RD{BoE0n&ooUBGAfgG4_tM=;-A=vF;w#s9lT{Ktz~eK03GXMbV>FP?U$J_PD)cVVG7tvMR)<}4#|Qn zkv}f=T=VhQ7(%x`+2a<0+ctTV3h<&#|>}HTGQ9m1y>;V{^6??dxMxHtmmk9{DGKj`}{=^RT9$N()=*1ea+o2(|; zj?*M;N}nYB57#$l8jT#z!GWp4RdOepL%+&l?Cil1eeISqS1Q(5eY(lNWAJhXJ7gmq zw;S~L=lZAG0i|X`LPdy+vt%P_cL8*CqBDezz_98C)3br}wc^HU-SfT;WKx?n;2Rc) zKpP`m7GTRHIW)MO9zWpd{Rq16u{6Y_O!_!B>dxO?3v)C2_Tz zH89icz3&V~{+mb7!C#rV>!_N+EcBB0h&D27cofiAT4PpmR-7RCqhN(I;GV zHzutG1OqpmmOyMfK%-K(!lsC-^x@)gePAih!3q$pMgf%8QT-c)?FxTjJ&-OAg1a7t zceixYD2+G#1|Ukt16`ReKDF@sa$ji*T1N;%6%4(5>kR^+6>&juI%DYEcPk-!yYb?z z7=cSfVG2rYvib#tjPkLpkJe3ncpOqaZqL$sKLn@y{mt)s=Rb@6MR1c1UC-c9R0#rJ zE3a-KjiqDP`&{r-@@&j3^1cssaYOyCsjGk4Uec`VG@`lr@6HEabV;D-qwt(u+0B7} zggW0tx<=mx{9WdRQ7N1il~H8V(}KK@Dkq=S_T^DNle9upehw1(T>=AQB*TZ#kE`&L zok5dW0r?_tSrEteZ7aat`H30WbssllL5L)eKOu z64<(A2*ei-(5uI34RO+Ze>np!1yH>m*?UD3}YJ2V}~xR>5cO@4Xtjl zT#4nsQ}|rX3HocLXnprp-9OI=hO6=ZI0XnXzqKlylBw$n!S8@WL~WOR>j{d&oG14~ zjCs=VN878R@e9NCEf@2JmD3!;sd^=+6k;iL6z_R(MZs!}W%VW8_2M{rE(O6 zYab!3qFKfg)L5RleG#s6D`9@W^748Pk%aSM8ZrTwYn6F-jMO(8i^J2_F4DZ&@#o?# zAX7+YLy}LM0BhajAT@icO_Sdv;_J`o$r|MSsNhPh^=G#rqy!rn<9vta-`&IUp41IX zW5utuqR0DFwBmPWID=GM^;hw{FPemrzjkihEcLhIs8U22AM&Cr41+-gn;=JnAv_RZ z6Gj*>-T{^R$ONpdAd+RIq)~|;deDqGxB+)2H}!l#ItV_?;A5 z5DvN$wO}r^(^`~n(sc$jeS!L>_6}g}?o=bxvRp+m9!_#=`6tZ+5ih^!uM)6Chxn2sh-w=U}$NW{oho0OS zPC0MJ#lNQbhPRoY;|=78iGItV;dz9`XM$U)j}Ndrgt+d1%+fA3O{k;{Cq75>=0w;x zm>36y+z_)i#^hreIh)r$q&A@x8rsWtSs5I;_&m97lX_!Q@|-{sA?y#RA6+VQg7>5< zgu_??7eLm4zu_ILu*cIRp+%GRZ*~7?KAyF1H(JY+@JxxY}ha`2|?Kmcnjl-K$MLbqEluI!14AtKS((N_rF}zt$92HjHu33C*drD9 zruoOx>Uq$fBBmQ2*MB7hx(rdax6?0Ibi_|7X(9)2rl2xh>I8oQabkcidLwVP&JgT$QF-obBYmCEmjSic$%FjGaCH7)b%SKO1c`oj-_>dI&t*%6V;vSj z3FlGEy#EOjLT^DuH>+f(efHL47{!Y(gmA9trrOdUd98G;e6x2=LLyIHfw z)9lBuJNvZ)gv8d|os5g`0 zg>!co45u{@*2sR02Z)D$nias`5?#+A%T0!zkn{^M46d17+9Ui8k)U0ON+6}s=YLLp zW+=S;#z_L}uBgp0M+%~6H7l$G<Vev8QlYrB+@oDwxc{c zwQhwOJh*q}x)6xru~+vMcnNTZpA)f-4O--ckS}j$_0W1Q5RnG6Y`3-e2~kpNj~sqd zOp_3w#FY|pEOr>#j%p${(M9-PoJGE-7U9p?d9RqnkAlZsy6F#&oe`~Ok{LkEC2Lp` zLtCz~I&!;K=%A5uBQFbuq|#&*EI3s~nHI zx?*Oi|6Kj%I*5(R{c0(P;`u%11MKXtU{gpdKLXWycnc2XQ=J|H&hLqKq>jzUlyGJi zNq%6czG<-{E%J!M3UDD;b81XlugOkuIs#~$hsDW)C%5nNuk{`41-4dw{6S?h&Krio z;2?*k-ky!yQ2_cmMRqgzkW$u8NLB@!9hrKE=8+(&cdlefdQ3%Xk!KrasB`%nAqPE+Pda zE9rVz)c^rxiBltwRQ_?=9bWaMX(jDzJP9%153rZQ5*NUUE*7{h2rHcdySTvmrRKsWOTZD7>Jv^SGOrbChO{6o zkMjHB>lR^fR+ealpJPvrlD-C+hnX~$G^}%N-n9SBZ_v9TG6XcgmG76ST?c$DW69)K zJT+YtdZ^jZ#N!Ov$OKw(FX4V{h~W;u9?mbV-@KLwWTxX~z*!ZB==wLCO*|fFIXT!m zSm8+amGhwrc|&AH$5M`SaMY~87-vz}E8#*7vq&PwK~>?}ikCP*Sneg1BBdu9KNR$s zkeczB$FsFh2L6#9x;5M2=oSa^5FWS6_e{Svl5c@>CXS#k6q)M{BCn=R)(DS2A2{0h zqJczEB-i#73A>%7XU4Jr=NZ<|q}i?0r2mdTMG&w#$C9-$(B<1 zVAp;u4KEC8lG8Q0ni_?JLR|J|hl$XM%AX(uht$`=fLYKP%gHS2N5=?J9^ zxhg!VuXSziX%BXOw13x8%`)>X?v3?E*l8ldwTp7&4lD|8+1FGo{>MXUXsN8=tuOed+Re5z=_>eI1=@VFw|vf-;+;W2EpGq$ zzfNqS;$UR9r}9Pq&BQxIU>Z|aA&^MFzW$WpT!d+XOWLe|zTRPSuY^IgE_4cJu)v)(vR=a)XFL%2^j2HSXHj9dz%J9M})&TtrDz~7pLXMBHWn2a&@X4(#bmMxY9?!N+C+~5DgNv1vsQY zn29TmFopvUv*V5uv+uj7#=`vQhDhhXg!$Lt>QlacW7*7VFXNGKW^cwfu5~y)2Q=|d zq#Z4ASRyQr{jvFvwwDQayKFr_fz3m!WON8|U@g`Ym8?CMm3($I+IY@_Sn)(f2eRrD zjBwDU-OiQw&V!_rUH_)S5HINMi9>w~p>^G_5KYRL&kEEVtHiw&Bk2z6sbE1@EP!46 z$|3+jWy57pY*6KXl(_cEP-|O%FN4$tSK;+R6Hbsix z{8>A8Ns%j&tq{0l;4$+GYK<%XqV^f{`|DiVvsNP@z4%_iEeQ{?Te13zy>v-WM79E} zTa9q$c6OQ{vE75>TcSlqbYmOHXatD>=C%4GO2(tL>b2>1KsP>!g2=k7uXCu&6*ZoF zQ3^dv7CT!B*y8Lng(aoH{JtSq)fSJ#)uyP#5EF)pCEMxdQtTt)eAAkkED4zC1f}EUbQ*H z!O_r}x*R0OMy4M|^kR~La2Ju`1ze$$3aw&8hBQ7gcSElg7Fpl*Z1eN9wwjAm(CrYf zWw;_7LMiY>4&*(VAlfJ}q!y2Co!Ro0Uz^w?vSJC!Zw1*4hFX?3!TfW32TVYFKvK#s zyY=<9-K2h^;AGLr^{Yy`L@Updh>o<|q6CEz8mS0ywv{It#eUw$fvH zSgcRiPpWbz_kAaz7M^uBjOY(X!$lSP?h+f#q^Y-#P8#-}wk}-}!7)z?=ZM&|1vuo4 zNJztB%g&EU&}@XHOd+J$JT3IBHAaWlMvQHXtggcN^b2=`E!#L_o;F(9m`LS)vrxrxMDmsE0x%D0% ze8r?^FiW8?b4Iw1ofZ(dn|D>xmRWKM2DanH9q-(@2} zZ2PN4J+&Kd?ZT%cF%VXFq9fVlYIv-Vzbw;Eyt-akv+;rjndpC#)L1p6fwEEHZLHaw z7SK1ru`qDPJ)j0Kw#}m~F1n2QY#YhxfxI#tOfP(0^9b+Kt-T$ueRAQrgRU=6Oat={ zn9w4c6RUXPJu`5p&*^%i!D-&&$ch$X5n+1yH^Ro_zW$Vm&Im}#&5*U zR4u@cHt5dtJ-pk1c}mc3yZLE;tQMx-pnPy<<7kKAv&}zF2|jW;4J?O?t_phI~ zauTn-?@OLw725jNN>#buLeDUI@uu@#Jb4cQdTXaiL^!x%b%sekTYC6OUHCwnb~UKp zxjAehgAU99Q|puewySVeuq4HHlC#u2i%w8;ZiLTcdX!qKG0dZ-vKnNIuQ zyvWRb(MI5@51S2Ul6!F-PBY=cfk!v2zokU?$@&CoD=2SU%rm54^77o6k#Dr1!fCAC z=vBOB2lVZp&F6`b$KQ3hZ+yps)>ig3Di6Q+vaSElR54GICnN^%)t71^H|gCt4z%O-+x0JKJ;_&%0UB^{K* z>XW8{9HG7pzIzy7R;cI;ZhB=<^8Zli4IVHKg5=i*=gE0e%RppKHzl0U>}xA9CSBgJ z+&SM_0b(ex50(`g_9=|zRV1(xX^LEpWWkdsNN>~}54qKjI%+V1NFOT@6k`SV%_|AN z{gLq*5P&b@A@GtM1>-8B?nAs}0D2RtZ*B`{mQe#Sma5-EO;nw{uBUS0Me-vxpu~ak z*6(prZ^Br+3aVFc-pQ}EaYzQU{8lUnZ!>T)%nMz)BqdC-HTlUW0VE~OCW60FQ8ZdC z?W|Ax@>nlGO`M+;I%-fPWaQPQqG&U#B#|XpUY}+Kti%d& zj&>jOcF&_%Qo6UP7%$$CY*H_BKDTvueWN(Ak&KjhIdaDApi!`KCx~w4LwG`s4o~go z+j;e%H441@3GzzA!|WCyI74NB!DsY_ZsW5Vl?S6zuU{f5w|E}&NM|A~LF%?2R}rmq zddp}J9#tDJT3A$|)oIx2wLVBZ%qd5t|CDATBQ&fe+CqzPXT z`T1Ivp_^!`JjTft9J2q)4EKVLhjD||$ai-ZU#&~|N-N+<$8@w_auUSV_1lhpdPYE9 zY!Zk=fQ(=pNZ}Iii?GJ|6GSZrrS{Z*Pu-~*KENQZ##Zj7{8KG-8vVjdaYeY=Vvb;NuG`Py%#iLhz2fQs^+cQYm8(`7ulAb)5 zaYGhAIay!*cuQbA%M=k&u87w)#$hl7`fc6PxtYYBorr8`@meN%E_nngcp2k z&NYJ!OFKxqw*g`2Mzgx}PWpc8A=_RD<0qm4x>EmX&DTgy6x%U|sge*l-3Q_^P$ra* zWQ=lcnx{(gq=R`ozBZOZtGu3?YXlnrIs?#RtE`sPM~5!gBF=nWB~fEcz}E-;*ql!C zhYyn?)a?&Hi|3x^_@=$6247V8aEH7j3&K`r{P0O6`@|w=YFEehvBr zupFp~{oYAYTy5PP{K4tT>k6OwARgQV7Ooer`ZjSj6}J885TJSGKTtyG^+5HHs{QjH z8`lX6Jtps$=@b9%EZn$E5XMVmEOlQ|fRs6;8G(Y^4O)CA1(x(Nl?IgFBvwXh^41#R zzf86tTVX?JjQLc0{DQv?^cRz6<{BZRDYLlWr>IBAp=9l8_oM9rQ~k#~AG_CJZTyk7V}0pbKw(n!Z_v3ic-F)vi_k9h$qW9=9YLDg^zG6nJv#q9 zv4QcyU;1%s6v5+Sirw7CNIr4?44Ss~0K{AaH%NF5Kprxh*2b=RW?M)7L~t>KrLFC~ zSJD6Pg?WVd57gW+V>$u#n3wX``3ZD1Ikk3<=stvx!MT0Oce?R7^yucDqnjlI!0(Ka z|53`*@DA+$1K>p_xy1e+2>j2(!-dJ+J1BL~qmjbblXQOn|LgddHP?v@e+!;y&OUrD z{qva;2QcdxOf0|o{SSnS3|`o1t=r!80q#nylwX}P-w#w6 z)30B)x#=7$;Z^?6qy8~$1H_>a%&m6H(Z|a(^06g?@0rA_s~`L>A$klbGd|~tP6Q|} zQ^7MvNgeZLwJds*30n0~HeY@G^(Tl&A=5Z<8^CMS-g zRvsx8vWHG+{YuL>oCjMru1@sYGL5%5KTAdLDk*aX?U#F?*9~O&oteUawB;2YBleVi z#Q>UT{$Q~F*x=#Nu}d_MBt?7=muiRte;qHB@P$8Ou;Pwcyz6=Xg4jz)x!b&uMOkBp zg93^b@8y*buH1gHsR4MNnusCnG8}b_j7?4fLXLqj@HNY31eBC{bkHUXI=vP{R-mr4vg3k>3b09%8~?`eUIkk= z#ur(ulU@Gh<7a=6)9`neE|D&*QM-BilJQ2;&0w~lb)rhloSbc*LLc2tI_@H>-bg<^ zK8g8E@?8EO&G=Ic?rP#Cj&b3Xjqr_;_I=mp+nuyZ%)fduu3QPo)A)g7)X=)Am~EN) z%~7+@k^fE|Yw9_FXQL

^CKiXERX>+1c3}pju+-$`2fPI0zR~CY^YCQDfcG;*8^u z35(!rACUjAa%gZv-TMJ(^hmjc(DFVy-_f-jPrVxbCN~=O^I2e(01BC=T}boFZ z@5tICmT;;vAJSCVF>^?FsEfryYoeR(tf;HKA3hqy7>?&4c!c>Q`0u}X?=S<9kBY$S zS{C_BFVJ(v2Gb)lBvU)>y0q!JKpPL!{wx0+$Q5v_exPbePE#oRCKobnJWb_Ow8IZc zO+D-Oee_y)^0nqhZoP7wSdjO-#g3MLuI#_}Op6{P`V- zDSeAbbh{QnjDW!n44))}K3Zm~$t#A>hXZNX6cAgd8z9^7UG@O;xcD3VB{HYRHve)-8K?KJ5gyR_oOx5!xMbXC&eH+QYyzMxZmQ8W?z z@qAtOhMlUF{PR<+4*}3${lf#9A1FZPFq*+ir_55DA$2OI7NPs)4H2UQ%Azbh*$*ue zLG1pT9tiV*vS|4~_{Beo8sHAj61y%BUIK}Pcc2U>R)2g!W7L-kBJ8eisESU% znnIl3BRd0hNU<`=u{Ac?q)(HV1%Ej{uD)wo^|`cEz+-F3!sRWv6`+il{gHKTJ4A|I z`wC^V!wm1+%g-{1yQ_kUVTmEn*46*7+%c~oGfA4~J2GM1`JjHd@s;5y`!J4HKA9^+BVxqcf>u1(?!WS*;JmmLU_-w2gPXj#-8=_i)}t zxR9exmzJ$MgSrWIP|#%wa=D#&L|FI)F@8w(?vAOqpBs5zKc;hL;u4%mm3;m9xBg{L zWbX%Ye_%(kN5kxJgNk&&8JGXL$!5gB0_Q>35Xmuxc&Dp-7BaO{W5Xh`)ocf!cAs?1 zDsY2#o{9)=bxqKoWAx{o0u1F$8V<(OaW&^xF-OK`TpBMvsCexz(0F7Z!YrA)(fV4E zC4r3s;VFjH9|M5DsaRl4L3O>VNYyB>_boo$HTc04?l|3Y^Cf|DZAl zXmmfCzBH6rkQRMILjRhZUUS(DzREhj_Ss{_J4h8LLdawCEkKC)zPQV?BduVT9+rTo z8A~uUi!)+9a9a<)rS9N<(i>D8QTBVlfy%jZoxrFP| zi}&6#+TRWMatl8>5F~^+;(uQRe_rQG_z1ZPz6YmmC&*)*QepiRB~QT(sXNa+QahzD z?vhKXy`VFRng||B{7m+xyfo;?PX3GzQV{;2SiX`vH$6d{i;%P({5GAYJk0=5Z06G* zE5L%PA`<9#WLOINC>@arIG)*{@Z07Mh_5LX6_NqF7hH=IC80lQRwu=nj`esmAL><-z=Et zr6(uv&a*TXz973+W`;>6>c%2?W*T@8RO5cXJ@|*FXIrknx_^CF0nXdMxSc+CE(=3r z9n_B9 zA1NR?vh%8%80jysu>Puxn&YsBJVKTpfiVT;kKJX~I+9Xn&iz4I|1lCtf2iz`-2}xt z4a@fZ8$7+TOf?I)zbE$dG^o<;e^u7DYI^|?TL=}He3jGr$JGhH8NAQv2+!r-zymjr zwKVN2fb~17gqSD!B67wH85A$e3Y@_r|6N3?DXXI$X!r8~&eCkwRo zjVxK=QRz7_HRQcfW+Rj_mkfnT+fUa7NO2P#AZa?&-&FM-DQ@(6ps3jrt$?kQBo9lL zXDPsRFx_4Fpb?T2SFsS#TfJ#h!1(C?%r}bZH~oYhRLxg&r*GcV_8^K7(y~$vB0Vt! zJmUz3$mR9n>v+N>z&$r4N$GBNe8yO$Rt1hJOs;00kR)GM>fPo&3Xd{ohyUr^81{>IBrwVAzJ_uX3v-e_(;#_3_M>y(-@GV=ykeX0}DKRGc8yv zi`GLXQZm1^T;1S;cGeZC@hyvN_4g6~0AK+nW)MTxR_Eb@wLJA2OBR99h+$d2raK3 zmuIFsiNDvrgOQKLzK5RMdWG8O!#E}0_X->>d=w(Q^kj)=@1~|+d5U+AI*>|Gh06y9 z)LD*J35G(2IDP}a{={M?zMs}DG6}%P-2q$dJNtAJ=Tc>=5OJ*29SNx>FSc{RoBM1G za^s5NuGBwutQC;s%s>NQ))n0_qO3tf_zbdV&?_z;TrK{?^^fie^o?zJCKhOmhpEpRVtAnICC=L=$W{P9` zZMgmMoS$7HXs5+zjuyvitGSGii@gdXj^}(}mo*KDy-|Iju@=x*6+Q$8r(41?tbUT3nZLX~s15nU9Hw@^nhCc`(a_+hXhzD~85@{f=DTNgF?H-KrcGrvab zs^4I8lE3OCQ`6p!x!q37E8J^YB6gT`P4(GcZse+}KFsnKJnAA++!8#v$~;(;Q%~P2 zIzQ!KG|UP+_;>y#`tvn5btcP>$NK$)CM}B9ZMz)l{Yoe0=%3ImRV{Q~6fPrZ^)AfG z*?Fmi5gG={dF^i(GGJjVt(8Q z&*AeSF_fReiJHSgQe_NNJLM8@D}0+1hi`M_7xc>gAmF(*3qE$LeJQH5nFT*6w9=QbeR-QWav(UGM?rdR6n+LR2UbkVl#l3a>gW<6jz#P8y5253c~|C8 zF`blo^I5s4YJaNG#)4{nLm!V_hib*GL`DTBwWUe|RUjKPAWKwM3Q8(+F5u0ov&5jb zffGNVTR>2-Z~Rh`=nz7|Pu=Q*OX(W~!+WZY6nBwJy|87i-c#Z#Yu*h`1gXB-6T2zX z=I1srAXt2%yTn4J63pF92SpS)V*h2kgaeo@Pz`xz`ypWX1HG`_=72zV<@meLlpW}xm1>pq8P%>g5Ih0R5h*fJ z4ZYLVsd>nEHmJlm0EKSmUW&;XiPG6s;b@*%)>+FvqGGt&s->kKPj+c;2ob6F-Gl%5h6 zGUpG9Wx20Um2Z#~!o0a!iHO(Whu~`~2g0B=F`MQx`6>AE^0_1)0ut(|YYO%1aqrJq zY5>aZ)LuZkbctaFKa={N-uq;NIm{(1PJV}8sE^krFka!5a&~IRF3rA8jg1A91HR3+!Zez z=yB4smaxtTx!E^JbY2+^6hc5uvj8Y9_RV0DOk{BqgxBxr;6akGy|HS8#qQ(7$;W*& zxcnV!gWNP+{yCiRd)FMCNcG1?K||g zXrfQ&6Jd3a+$<4~1l^biOe0b2-I0-Z??y6Ly~)u0owC&NFVg`G)SEMr`iOyn@~;;- z?T!1Ya@mz?%7u5!Ij~`w(MMsW(MQy-2h{e41D?Kebs8AOWb)7rL*l)+qZYE{zL)2;Wz0k zJqgOK+Hp;x`(>fSb@?hℑFvQf;Q(s3;@s8yaYeC}EZV!r5laD3@=ZZN+;BQ}4TZ z9vFLeAr!+Dyha8vwt`8I6E>^0x2*kP)KjQC7k8fTSNb-+?LGPLRsC9NN-SaAi1%>u zoEq)HH>U3Wi&@C$o)ZY7n|*}Py7mFB#p$fxLCMUx&fMYmU*%8dFvxp|X(Ur+slTA^ zMa8vi8Nti@o)iqn(ex`4+n0L3n+-D<8{KH7#+Y!*lQVbm1~ixNE>mNIwCf%W`NjJ> zsx7+IFEr7mD!ABAqxNGQEno8AGUDzLid3)3AK)~SI=J~BdzYNqA42%we?ZfSy%{x} zVi%d3yw(tk)NAhN#|dV97!RH!Hg-A&X>Zc2H5Pq>}zYl013P}&6 zwwi+OlCa6?3^*awK!5D}hIv$oem~`r6NRfX498kQWm>_(%)a;Gs&5TZ%hN@R=YB7jGOEJufK}4PB${4H5O5O7jqSoUxvGAHPWNNOwesY!)N}s)BG3KT@8sll0vnr2zL$i=g$oTL%a(sV^uK+<8 zxpRBkg?cGHAn9Yii;qMoL)dw=B4V~)4X2cAgTj6;WEmML$C>K|Fomk|Vpm3lmqEql z{pyu+N&rW@I+rRP@)X1;BOWJ9`f%7G3I8g|Gz)u#oZ5NhAluJ53+haQ0pg1e14{Q8 zL7!tl5vWwrXV2M1l5xSTcP-l-ClEp6j$MfT?g;Vp^X; zyAjhCC@O~fXI`eI{M$E518pEa!yG5-WF8~w;~67ls~-c#Z*fP9*#_f2n4_0=52eT@tf=Sb?gy=D`$@ zG46b0VN@wPU}!_lowx~-$qwzjh$q0O5jxH!DQ-`@_xCq|1XK}&>OdMWwf79@&%1c` zx}mORys&K}JgTt#;I$bwarWj!7@?FB<q|Ji-2Xf_7`^h+RX_i_!mR07xPtn zhR|tKcZspEfoXLYmEou0%Q(4aqUc>WMi*_Y(;(|&R_H~;&TUQNJH_jLFH*tSw~GS@ z!zJdtD#2|XUPyf$Guy_6Hm~GbsX`S@zT)pB2klgpwVswv;8OkTSKW0k07M6afwleg z*lD&WA&j0`-6>ACjTKxxrOS=UB9MsAvKCBs*_^s<>(Uo?gWSTtEx4qecmT>Jxm7DR z+9z-z+OoQI)f5hYt?Bb~LF4*uZ51hG0sXXDF1%~Aj@h1ZJriM32mS2 zKJAT-AnjQkc0A*&$i52{&WLq=GaC@aZwYD^uNga2i_V1KlREMD&j6Zoo)DC$I<1F& zn{SVOCwxHPpUr-5^pWU^qb5xLVvW_9uI&=$dB2ct@6$-14efY$qjwdKqwFqy^H>G5 zn#6(HSKFyoZ=1_T$<~r)+rcgztg&c-xO9?v!s%6^U=yDFH zM{NH~%3UU6Pk;aGXOn6eqkVawD^e3S!^r|ljrF(d+>aMpkLS2uS==s6FD{(@5=C8- zAn(~8f4v}xjJk8P=dX$fsP>y0CZII0SxS9idh&6%QomcmYy~&xPTH;pPyX=au4bs3 zf@64k@6eMMXgM>(?TOdhUe(%*sQJj7-s5umI`Kd_Pdy{LVXULajc8=b*U4Tn-FaN; znLMCXF5FyiFR`?u-{yq~0_9lj34TYrlMx!q&#SXp{(3b-2y?Y2Ko7)mP(AAj`U`{) z9(GIgU$EbGX78xY!vEm$CcshDRtD5E7p%UkgQ}wLT)GZ)27>5qQfhVBVv1Rms=wwJh z6itkTm+LtxUvGDi_2`-Q7`#bURMS+9thP#g-I;IoN`2#bzq%Z|rgt(B?y>~ZfQ|CL zRK%e#$Jo5B0&?+2qD~PoD|FI!wK+sNJWVz_`r%^Y~xp(pddMKx;LkZ|&jkWn~xsdm# zviuRj$39X|Ki)%G&0`lurj9-)_V=+}>N-tw@IKXP+tWC^?&J7~C2UM&33c~{AL7Uh zmwh6}qcSW|s$$I3dSRr(UT&+o{O`AvKQxm9bd`!X-b z^nTKZqoFQ9)OE!028sDSIJ5NkU)*Iz?9AQ;0uCFDWNs0OO=sF9+2cFg%25}6rpk=y zg)P%|VrqoZ$atkhxig#z3_+8xLNb3|SkiPjoXF^FEQI5*Fo z?^8e9%|9lDmqgImz0sLm{0@3(Vlg#Xwv0Wf2A4NS^yL6kLJ@Y44M#-hozO~)2o zn>w%w5v_wnDY8zvbeol)Tv#QHBouP2AE=FxhkK^gZtlOr#PuiL_n;nE19ruW#$+J& zo^-5#|7FZwW+Z^OYjm6tH27SW3?a+oZQ0n55|pC1ur8NaWJMYA?#?(<&{XtPb@H#I zh041fQsr5F-X8U%=aBdUAG$_$;C^^#WHEZ0K_x+i_ac~D>`3SGzi#k0isR?`(sy!R z0ynOCz9?JJxGWEp@?X%dFGf;K@f}&@zAJGP9K-9f^@0w1yc^8wZ&| zlC?+5?1Kx!c7`#0GGTRB{@2GdsyOkC9mu;%)g-G;IW*F=INpEpC)eF`a zO^48g8)5;a%A;9a?Yd%F zr-{js3|TTO*2l6Dq%28t$v^^Hj_t^BaH~`pMHI*YIEA&yOt|!g3LROzi}^QmN?#X> zNsXfmeLJTpXdl-J_Y~7Fdt4^4sP91=g|}l)(5}6VD3C(wUzgZZ@&fs5e$DW|JqP?O zfRQDfpC`vdqa)dhv2R{;WF|Ef@8Up9NDc_$>U3O4Sn(c@&pfCWrv!DlnhSl3FM;p$ zS2YDi(tv2K)}U2x{-;mDl+&k=as+)%MSbYP$A~mBM$q=+mI$C#QicdEnHMP6slr6( z&IL;GlK+p-VBsX;UYdJH7@IsM^Tl)%Cg%%BWc=_CArsJGt`k{&Fzca_%Tr`nMb(ik zC3T#sz%3p)p?}@8fF#x;lg7Xs51O9sq`O3ivE73eeV?oL|44K-BORPs$CD%*c)4B;>`F?92Yp&m_(PIT72T+o1b? z@S{A#-_MR_!I4lDlfHl9He6I2t8zCh&&-9u$kf?!qqw3kvZCnV+3bTQpf~Wk3EV(j%B}mLRAjvp{V>#mtMK3k$_MHx6&|h4tr8c*ioZoVXjHr8^uC~s z?QplY$#A*z`lk+_%VhnJjZ!(&C2qOF$jtjlvTt6yR@2W=N9-Lr#7yK7Sn6rwgUd7U z0x<)`{*dd4?KIo}5CUilf?}pU+Dfu7xpW4T$C1lM2-Wh?eFh<9IJ5Rr)mu&7$6b8& z*-4hE|KBGFCYnZuA?xS#PlAbdsi%R7cXfh}<;1j0BqMx`gHdWe@GMWp|M-5%H_rtm z?5cNUwqSB4=Q-YeOya!^iI_|Y03X}gXFQU|I+eaP$-qx-`G1@nZEGw~pSP8GJqNwL zj`L-R=9{M^APfNw*m;nSQpUM210T2P|M%bLAe^Qu`*UQiwT-^s#@}&=gua))a_%>O zo-^#4YW`)mEqovZqucM`oyDPDCeMY``!Bi6ucGV~5un!j6X;7;hzH}`ITO4$zQA2| zat&RF%qH=y5QZ%RvWqn3pw~e?o`Nuu+X0w(A;M$sX{95ll|?QA8;1(+%EIZ5qt70~ zv{bE>j|3KH`fLrcFN~l6?M3BD{@km%juKDUCG?|XklF065O883j)8awdouM_%-=BN z)Q|5YfP>I)w+;3Qw2AtGrsfJ*GqnSwDm^Bs**m&4U&II^T_V$q~v~s#j?s z9G0rLeq0PDSn?TSe}Uu+$U@IiYSoou=J0X88O|Ty23TKni-!I-r3Mr1mRYi7ta;6| zo4?F~8FG9#_Q9NR{TID%Y_-7p4#>n^>bS^Pi6_! zFC7Lu8!!??zvz;M1elZ4Y+63$F#U0lZQcz4M=s_x252tsJL_S735#jqYyA0df9j{M ze3GR&dsawqemzmbMZ#R8hBTxvv*pPCwO`@J6y*TG273xRb9hR&o*X;<>79yX>|Y*3 z++EpMYUElhPFV2j2^BHTqqqLo@8azX{A(?wG=Nm+OSgjqRQ-{)<(1m~IY3}NUNP!w zkX^{n0p?U32Za>mlxMlfDe10r%?bAFvH(o(xI3M1*4Z6`g zm+eMf22KFND{^yI5Ku7=9R`ftf`+?bdLcFefLLp=L27{u4-me8wQ9 zm^R%%DG3c-u>r3+3GM_+BGThmM>57<%laUfhQ}7qLfn6mlmwsv1jtBJxfvc@1 zZQK=HNytkf7f*p=I3uKk)_@0au08`Kz-J8=30i>Os+@AutViR+eE5ZLOGwIf5(ZqqmVyiH^*d>;ZO+hk$W>_V;nPO1H`ZI6 zB&YXvi;4ZWM6{mVIYj%x_O`omE2UqsflA$H0lGk=+y;OeMAK&64mUHUI&AD4!w+bk zJfpev)J1Ry7)qfxG3P&r@m}Xv)xi7KlOPeu&FI$YK;repsKarJ7m&2HgkCAB(%Goj zMHGV1wELz-f}`xvPW|DEy-c=_|1tNfhEv}c!Mo@xnJS0YVP6vitM5Tbdtw!S8rtF6~QGDJxp)4ucL4oZvKG`J1VTkv@nHnA${Jv%v7qzgLo=K8%o42%%Uy%$Ci68z(W z^d$5kAS3aQST3xh1ZW#q^KRvLo^h{o*~^3>o2@Kiy~oW5>$hX4H@1NGil!&Fx&yhI zI9PYE(WZF3c*AtgIpBbJkfF+*aw=@rrMfNM0UU|Wh2qbL z+3yIUu5`EW0}^Mf6J}aok)(LJGiA#0QJ^O%?<%ks3&mus_~fKK-||%Fud|5U%Eb)c z4`Eo6q!#{A{hq?UZr2=szm0957E1T$k_?5~k5&R_7u`7lH-dQw0n z0BSQ>VrwvM6{F)n1t?0dLh|+Oo5us0oJ%DAm~i%f`v&&`z%6fNKx!Z4{%8y8_NwzX>ARp6$Gv=xR!%_5Hsxv&>X@83Sf6u*-CJ)5sn4 zGNOi+IS+S{epIdlw>q$57_TjGpFaVVuAWJdm1vVFunN1703u*J!vl3u)kemdor?XQ zh7W4L5SmXuIju}CBo9Ng+3KqxN`W0u$E2i-e(Kp9-8mAd=VIIawLaL~@5uAwd&UY^ zQ1&^mKQ!vttnJoa0)g-p$ltl8DQ7+>p{_c>e%Q|iO{%sX&PGQTFq|b;pwqHJc@?75 zeUY|G`3IHdSVLW=pBB{hE8#dM{Y#ol=-ZhsslXA@Ik_h0zo}~JM{MX#*2SJDh0(F( zX0(}K`>C*WZ3+8UA>FqY?Op(Ssw4)TLjt1!^C(q1Ozlt%(5fC*v|SiWn0T?nxX(82 zvJV^FXSkY;Lg<4o-l)*fwXenUAu@4H6JZRP6V5tMI<eI@V=Fa+J;>XWy>g;(Q zk0D8Y;pbt0dd&=l`sR_XQI)(^s*(B87EOF@u*r9Q-Q06?5EzI$UDKFN@`_FGewg3k zz2rK2&nKgR+^KqS^u-!1xFau(*V%a8ja4GV+!xvHv#6%$;ihNsf4qir_7aoX}_f=-TPR1R~Q$7afatH-d90R)A$uzyWqE`>@)&etQ z=ev))MOESEKFm_W3#MZU+fyE?4tX(OAVnmGdq5} zM9R;I46~Xy3}8_dQn+2@%kgV|T<;khEzqcIl6xIvUo`jb&Ez3v#zU>-r_*s~AB})m zc%}oOgwb99I1xDIqVGnxCiHim^5>fPewrUuanL>7vHX0Gz8W!q z0qQoEiQ@qvIMG3ubp|pFrt?vrAxJ6MDm;PsVc-=F6I+q^JDX5!JqqkP3gtFavf6KY z01vXahTOds=gyRaThgh|5%A8&HIgxLc@@4GQ&GFNSj)5MDca|E zqWpZ*S*#V3(Ccc&M`=r)%AXn5V&UR zhMBSR1aaLn8-b^WwZpFVup9^sB}Q*_D&7iK$S$d~7_J{voS8?@0_>-;s!58vev!$o zp=zz=f{O2Q1lq;@Xv}Igx}fys)E%D-BBLfjCpa!xr#Qo~r2Ypr*qyu8psJtu+PN$@ zo83JQY+brl*rql1=LyU|;OWDjyF+p&7Y=Hc^Ot$!0iV^*Zej%SZ9Z{T5e><4UFj7| z?nd?^c7Tb$C>ZkDXRkod%y`f%8?kuh%Ps_0Gth1I}6^88@pY)S}O|s?TXj&cF||DDM-& z5)Un%9~qD*A{h=LJomUTs_tNxCz>I?ysuh1hjnzQ2$*eV;4dj0@Xoy?%Toq!EX~P% zpu1ULuyqRI-J>LIR^D^2Dwk8qqkl`}Q;r*S_uL|b#W0}e@q0V!#jfUZ6EKs+Cjq~( zn74<@NCWI#Zf#*sq#tF+(Sb`x*QoG^0TA)cspIF^I;zL**SD8nn6E!V)lX?DU_n0T zES|2f=a-r2%CRsiUG|pHYn|7#Rv(fDtSbHaw#UE4)*m-=G18U_vQEX^!}78^4~_H(}RxD9n>u?W$Jl#zbqtd-cZoy=)BT`1{re zO)CRQmN^*ovtek-Vlhz#TP;8^E4Wv!F1=qQ_KAc0I(8&&@b;c@jCn2DUG#|_3q8)E zCch!E!p+z5HN8te?*C|GPs<{)FMI-0`$Nb;l0<%=$O~uls6Y*@gZX+eo$)!Y!FP)q zp95tYSMRlLqQ;49!}7In0_Xeon0tuyM2k==iL?YBajGPly zF6I#^L0FI{wQ@Izvd>s#TyMNYpvXMTj4~ORnLW~lgGZTudztOUV(1$Fo=)r5$p#|o zb_30R3%A(_AD!IF$P=m-h>Uo<3v~cyJ5W|0*dA|R=V5MC-NV|mC7X*o8+W)112xjh zYFxC$6=LmtK4lkR2UB$Xxiuhh-nh!vzL|X=o$+E52{xPMXT6Fs8 zh|ptcuNj@4Pjq#vH9emZwE!y-REtP|z%1>zo}|2rQhrt=;Ip@RZyTiaZ;#jc@PqX& z!VXcLX#k@f{F?ZC&Pop>t(I%1{Mep{8>q>8v7yb4Dskg6+goP18teLf;T1Yo;sWO& z3vz1osX)p*y_g;mU`pKfJ}O&H>84pUk{11JUbS2+b*auEN4RmqfxZ0>z zF(4&$8!wOQ!#xcxe<~c3r8#6WlM}JmJ&Oz@73l$|L)WR4VhPe4c-If!F)L-!nV@bE z2`syjSl9fggvo&JXBq?M66b44bNUqsQYX>=a&1%bqP0MVwEPdzk_rH1u&em!%ze-m z0e@HGCVGa6RTD|kfmzeg+vfhC=nck!rh706x!u2=p}g##N*P&W=W|gOq24w%=T=g#uPqr(5hM zM`-f)qteaJB!!WT!>wg(Y3}ICmN(K_M=Neqx1#*Y=t;O( z{bN;iNLzXQeQ%DfF}3=k5paO4%ww9~a*K>XvtK7ydSDsv8K2I($CSL=tDzEkq6DA@ zqOpBTUG|mQkgPuZ)z$jz>hjEi5TqpSeQ_Ei7mzavr4GRsi+bJH{1pvorQK;_mpr*F zvx3j)UgNHaP7Fc&ZdXs@O(Cpc&U?9cqfST(yd}vwMw=o=Brc5kV8$SNgm<+WTnAI# z+a+HIitkzP*F;DjDTPV2DSAq^jo8;G>TQD0=FB)j#A06B8`8U=j#U$(y<#A9x_w1y zug*o*YIFNuz!!ls;h4zR8u_(As&xzu%WXmh<8Xf2nAK$tvGA<4q~9Ch>`kHqs5Td` z^~1DMc_e^#`68{|f4j8bERLYJQfpX6rV`ATDU? z#1iQgTH-hytRUYZ>LSt44~zS%2-dmk#CEM@2lTRp9>eaJSS5QWR7Mebq`^ z-y+c^PSCj;VY{_`FWD_&GmE>HDQD=~k&=)X4EuWg)$d~yfM_npsi7^-6h=#)y#ai# z?SdLl!W5toT9kiBe(w`%&BH2*vO;|5MT~xlB}nwF@}$stqGTqd_P&KFuFO8Ofg`7{uaJtVwbcMs0SSTSTCU(|fo;EZA|PBzS-=jAf(-EaHP#rrVluE8qK z!^b|PIaPwlufk^(uY)P#HFufTH%G)G@O_@e$9j&9-T*e1Ej}Y>Ub`hAH1zP@mQ6ov z#o{Zp-i6J(BSxnRf~l(d{3sXa0d7uV_d}Qu7k?8t`)JDw_{n<8dn2}$vr(}f8mOH^ z`&e3Q0b>BQxM2O{dv(mmmcECkBtKtCyBd6keMcu9Apa5a6EL3e8s9hG8OI%tx9_w3 z2oJyvVaRFUDVAjZh-&IJPn3wAoKP^?Y&tJKz#E!K9{pe~8ze$6Cck03+ z=zn&U%T63Sbm~=;(>g&jqxAZ^Zc&N=W8^SlB36Q4(%XHgF?}3eOuLX}@N1Evr~vh< zZ~SPhqALj@vUD;Yv;cR$iX0<|AoT}K(8M(m2zXeYo%wAHLNoBu(^>Cl`Dj2NClvAs zyI11&N3MWB&z9oWztEW|G793Bx5C7%72|++Y>ow8=X6Z$(TjQA?!ZCdQ|u9Y&POSh zor(}(@R)fMTibjcaAeT@Z8;K0N}Njt9HXpf*xBSvLx-`za-AiCB>Ck|IgyVst*k+AW2Xx=Ip%P3Fu%bNmQlNym|j9AiVLd=!XItSYAIz zu304`5nU!I0z1oY=u#6Vx%bHO&6v{%ATN=_9+Aeip1~?UOB%5jp8(A$)Rqo= z*?k2K;)*Zy(3%Qp!Yn~q9iuT|7=hR(>F6c8Ogzogu0PM(XH;;~0~pIWl}ajlsXrwM zD_Od^@3K1^U9mfj8%6s@56QZ5EqT(g8*z5KCVkF`f9*AXaX($UcO z(D04$OVXb}0NmQPe>U1}Zu{Qpx#YuT3x4yx*GooF?3H|B+u_po9B@r^lzMrl{q!RQ zkAU7!+@<)Db^`RX_DwD8h3yo7=Y-9jPJyD{3j6UCF}~?^xBA5oPS^}4+ztX;Z~^c5 z?qAV8`7`Xn$OgQx5TA*J>wkUhDLBR<|C;4k&(D` zuofb9Jko}rQ{S{RL3FaB{?ImK|0V3;LB^;Ned3`+JcxYgTIZK3B*ZxYIweMN#VYch z^37|Hd{pS|YaH=Gi&}Y0!u%tHM_coLtCjQStL@(yqUj7BZjFwI;;n-pmJ}AiQ<;>- zR!Y^y3KBq=+p%P)?vjz)k~{#zoeYYacIy=;s9O; zexjM0;o`ehl57^00>$ZpuwS&67r4EpJN9Ox_~wmH=El~Y;L=!yJ}roZ&rKKgQ$((;ONQh?Sz%W2fs@e z!dXn*-akC*=&er%(?h&FmLy)h-NwRC z&3I?4qTZ-T4jEal2XNYoE-Og3HcI_M_2TGIi~a%KEuGb10NbcD2^-_UJ0xcD!=~MQ zBMUJoC82=0p?)RJ;fH>;pU(^ckfcoG7@elj7?s$v>PO&9QEo{lcf0Ektp0zEoq0Ue zYx~E?7AZ!jq$69qeJNYE8Evve_Kany=7gwJgHhJ%)Tz)z2#1t?m%Xu9D3c`_WG!oU zV;fs}t}k^?ox}6{^;ceAGrr$@x$bLuU!OZpv>Cs^(<3<6G^#6tXjaD(w;O9lq(Sk8 z{CjReASB$*xPwC0P)p<9;sdSF>JiryORm+N}Wq7CNH~T|A|6Vt&yA=8rcnXy#HNQ z2@5~|0LIfd9jiTGL~{m>a(4EOV^Gm%p%!5zx!Ka_3drdt>#|W$Pkq5xz9qqdpQUX? zw_3+Dr!=xAnTj#x-?akIdXEZyT=LT(gfn4P7;B~g(5aJU_;Qz{?g!A-&vFeNbXG9zvlJ~} zH1C`@iB~hw_D{+xiKMrOXcc+xOE1YjbnEVk?5%?7*=uvfIMX~8Q3xrT7v~@ulIb|9G@KS84ZU1K=@(KIiCiV9P!+J*9z|wl{BGOE>L0c9SAmAP&|2B3X5hMzeFbz;CHcHco+( zR1H#u@DbCFPu$_!i8Jx6%8Wc=rZZN&a3Mn@$~AC>x_}j9Q7XW? z6=5w-V?RSh@Q8EM3}-|J+DzmufjCwIls1&#guh z1b9A0Sv#8-_WP)wmTcY?=rdXjT$`z;k~2#-h~dYL_XO&#YiEd2c9(Cz`o7E3SZUvh z*_LaHVf zr%PI4rY#$%4HxPn{y>7T3>PQjS)NcwI5(>K#J^$l9HX+gD_%l%t15}fG2e5;i;_WeiO z;!<#?c;%tB)m0Hm|JR2CWK_U z8ybkTXYa=D(WT)D|1Ux?l+++9M3~HE_DZsS1{e^pC>P$#sMaDs?nHQFU4+cG)#p{H zj>>$wPgBl6tR}UQh=X}LeR>ZJcD$FQ&OrH_*3b=2DP~zOK=`KR`F8o0nDn3T0UU0w zSD(f*1+K2+mc@ka zevFwuXe`xDotVE%S&u5$L7>uij2<-o0g=&yYNe0`AJAU@9WIdh+ur@S0+81QxD>!= zjxa%hXivy_^47-BcNMDPO+?qDf`k)b-379A-3w^y%y!qkM_0Caar1vn*9-uiPs0?b zi;1(V|%DOxzoKBdQlZfwK$}0BEw9o<9 zY~y5K^#*9ZnOS>_wz47262b%^ZYS+vyRENh-ePet{j!#-(wi&+?)YK#9myUK^O?S{ zNbgz39-f$Nl}_i9w`c^T`UGj{sl971rc`FxrBG&1SxgcMsLr^_czdz(E2)B0Mzi=Y z#%hQGw9WSFWQcM4q8pWee~A4^@)y5e7YWan%Q6F;Q5^EkLcE|Fvjlr zBXB!6BS@q4<(}hXPgl^DRd#7gDaic1{QNHZC5YN~U2$#Tt;Cs}>*(%mi05Oy)unvP zL0_K6mY>2Z0q>LIC5hGYFn=22^sptmQIyE=eo{i)-{<|&OW?hILasQxu-3^`M`V$$ zO`NL%(>8a~b4*VK&s!Tk-}y#Hdq%f@e}RSIEB&?j;Zu5< zDH;MIdTu|<3)j7cEN^#+D*1PeQ=}K7%P(7MB2V^N5 zx(>fDzYIK}4agDwk4Yjh%X%h~-v0T~zfW0iEOAZ`jL4PPr1;6$fj0ZLTQM#{;tfg!1AMTa*@Z#9 zHo^Do5RVe1u0Q=pLts4+PI}NJQX;kHMpCjwn*Ba$~iqfgzaHx|_FNmMMZIT3+A{muAdo zKmz|31bP)bYcl^QHY53mm!MofcL_2-%+-c_HT~Tdq$nj>DMEUD&6+pVirchPlMm`k z8Pgr8(A;XBoL+7KZxPkaXeX_aHy|v=H9|U(ES5Y%yttqwn$sQ^>}gH9Mj_Y%cglgF ziI5W5^@8{Or9?5Nr*P!2HUKRW8K8nTZcIOE#g$-yI$kD($H%s4ocemVw~Yq{ zH%y81U3n|>2`)h80IkzJD$>;Qk0WvUN9@AC`;s3(Uvj#*-m{b_KUPnpq>(ay^sMXS zkm9x*7*W;n=*%=&dXjrh4xzl)t)G?CAn>6kqI9o^`1>eV32eMex!TI%P-G}B=ZBrY~#2*v3ejE9i&%r=?RM`Y0aO0pZbNk=_g&(+)dZkZ7J=NMsKHl6f3 zhZK4XK`%x6M`4YL-W*-ivM^=wrIv}!!Q3%_Fe!Ry=vlbc?UPVF+0%MHMbSK~IA*=! zvcdbnNS<+>LHUK)J=n*wuAJf?p5KrkZmw1_0j(xyLJ`BU+L*}VKJsrLccuhNV9p=p zSajvu$GMmgxz4?-x7asig#Bx?+TG0=qDXcRdyZZBcyx?G*>+g4%GbKE{;b{&JWVi6 zt+R@vK?`ObO_>55dEo>^sN&y8DdM!uunP-K@=)9wS|gC1ug=TjsKOQlYrsjRwA=f! zB3ln7`$D0wf3(3H3K~aTlnQ)86nX~MS@vk@4Oyqqbyr1Myt}?C+QyJ8Cie?mJ3Q36 ztTs_$60p{v#4$>>vS(r5qH%LB6!zDz4sw2dRmO)Ne?{P9n~3rcQPWj%4!?nMc36}v z)P`hpd*R2nA8!I-VRIVrlU zcaV9Hij7A`w6yAbyd1o@A}LXO%F5vb(|!{PDz%?sdyo~~+OG;{);c>+b1lXMjcA3j zEqg4#HNp_^{Uv;~4+46hi z)Pr1~HyQU4(DSY+VNOJC#NDIGUy2T|%WF=lcRyvi5U^JtnbXq2t#wDM10^rBwJS;_t2=Oz|Mp%c;=$CD8E5Oj6g1J-pH*zDK_`J%EIbf6aHAl z?S@bi-oj0lv(eR(U^V0AA2OiC`@ZsIdzP_wWt1DV_(XBip2@%lIfdm>Vlg+W(BwIc zelc4vEy>;XPRVV`oNtk4?a%62Xyh4|nw*X944I!WNbM!HBJDz}8H8j#p@=Iw{;xtR z2`a1uM08sMbX1$N&%YX)KjoCztzTdR2lPwfUy*ou_XDe*h8b&L=O=F0ZHz2=^gtDW zAu50LOb?5Y$&;hfn{(WdNY!R;xu`T&^@-Hc-ii#2aTFc7!XjYue_t}8=o5xVpo*^U}>fO z=%0qi`@#;cT6H&7N%yZZl5$H4&d{Zv`<~b(Qz&_a8XnN4Sra$xr4ETsZ%+1Eq87nI z@5aH7G~;y*Z6r83%Bt&gV2@!|MD*$3CsQetD3^kR*X*4 z6+^K8tjyh~nXh*_?&)`=hKD_Ba}brfu?xa+=EWB7kfeCC0=vnGFP&Tr@md>lbGbp9R2TJ}_FI)Owea`!l1T%6aP*^=q1I=IHDg)Mj6lYj2E zrv^WIYvDLGybU}-AujxTq*vZ%xmSM2w_f>G2g{b90o`Dv{M0p~kW4ogmCKb%&V8(F zR5)w#0>2gNlJEaiDf@d;uey6{F@)!L4p|MRh*8at-F#CW zJSVKk1v&-MJa^69GuK=*vp*=xOJJfCqaz?7U`k0oQ$|3z?Tdf_`H6ZPd}2Vs zkq-Wc=%6g|6rrexbQS!9_DWLA0RaJ@2KFDKl=8zJ@HmF0s-~l+oGibwtu>3GiLH?- zi>vi3@H7H~pesN4(c09}kkZxqrHuo>s}R-A6a3(3*vG6?lsAt!S_x5U$|+Kc+S;2^ zK4#%!VWSd8r=+A5v^O#1SAHh`=XLO#5S4|a<12nvRu>l+78gzyTYGa>c0N8nRyGb+ z4i0AU1ha#ijiaF}vyB5ae2~A!d1mThY;XC>(bCq25;m@(k*$-X5ET_{qW}GYuhY@e z?C+Uu9R6$zY>*Z9FRbh=Y^?tq8@wtA`;=eV)WP@7{f{$SGxv;TSipP&8rc|{9bM_X{#_Ljy{HjbwD zV6s1N|FyvX>nHw za#F}W3vbB5ht2c5t2mQ!v^7t_9`bnnVz&vB8Zy`BeW)vs2Yf0H_K)7_*P;vz*RlU| zhnidWnJ^ddwo{i)w_cpaRV{n02Iqv?a?P*xoGe*#B2o5fq4m+4HJu7y-S;x$rbvaI zkcAk8d6Jbo3L@z={iCz3VWUpF@__sHKaXxx>=$2JfB7)S@Fc6dWrdnyaQ>xC%gymZ zrNC-w?%zked7l5iU(N;ri5_CsS>d*6bX;cr#(Z}F`bPeHzrW>Rfe2B)m|8Whk3-Yf zKOQLPpllt!S~R>7Q~C=qoLVOJgBz)pQ~Kqs5|Ah*7t+05?RSI0T&?8Lo&1Ur>K zlI6nk!{q>5q95oUhS9>-`k=2=p8~UE!rc>cZ_C73h{r|9CEzgv$*v z%I^$q*`EwVHhdY;i+0Ls;jlGnjSgao)w8?t{iK2+MW?&)vL2dR%CxC_~w7> zPx1e{x7ay7kS()c`urP&$7O3uGSzhsjmNn4PSwltQe|HCGgO}g`=Yo24G9}Av3v7t z!>qS4ZWcTMjz7miiEcKrbbJR1w!nOBi$(ALM*(D~TLpJ1_F}5PcCHx35OJ8iClU6n zo^+Y@n%=70-csDb&1~WPTciE#(JcBzyv`hXtS7j5-1bb4zuv-%mUe#=wqwnDtwmOn zi3iR>rhz?G^yVD2kXCTCWOYYa(Ia5z?1&BNo?@MNSVFBI6EodH;GGdi;l3K79wF>? zR^6W|QI{FY-*LE}8}*y*uXBb;JCE7VZGWj|cfPHT#HRXNcbbSu{ui37Z(90AJEcZW zS?-Ll2{npuhaumbcEtl|nkGrn0y_NcrP4o~iJ3>WxJOoUm>%!$>9C5{Lic8V(UPYb~x{1qqxKC@z?kcYE28UFd#`3BhoqYjF)!MQ4$U^^$Bl?ePBSx(6M z3YT=Fw!>9X`#tp5VG#Rge(yqZaU9jCiu&_heU`n_(75{5{>*^7U#~HyVl?zu&yKg# zu-Xv+VUhgADZg}cPgc)|iOg7+_ls{eo^LtWHeEVvGKW};tS{cuzglXE7Rf2tePYec zofF$#UE9t3XV%X&;E0*|25#;Hlo8v)*P9*B%0pgU1!Kog{oPBsK%e((CQ6>4o}Zj0+uLW)Fzt9RmOuq5AqBgk09;K|Ln`c> z6B3DuwX>I)N%`*;3H(xYtKK|5E~&vHDV*N6zKp@x_{Df| ze*RNQs++~;;Q_H~N6;AD{AM8zaoQpXqI$9ZjEm8sbM(>Y#4P#4QkD30m+cvw=C^1z zJHNgevf2D=pLl)=oaVJknc_9(X7AWDH5ROR{yvM=$L36+1c87}? zOegL^J%n;2Y5qtk=6OqLqA5;CDODU%`(8A2e~E2Wg9Wn5%pFqRKDh*r7VFy8oKrT^ zf8G`gbqEFliz5o@D_@U;eu*f#>QR(C$3|`66873QOg#5Fi@34JZeHk91=lla%}GfM zzFT%b;)5X5QP=KL<5&WAKB?jR4m|E3=zZQ{k}Pf2ER#ozEV9$!|E;!AoO^xEk3#Go zkH`@sTgVoalrTpse^n$N*CY=&YduCfNs`7g>Z1L@oYH* z_M(C$GS5Huk$+#a-nC@Xp;u@y^o_t&l~V=**|k~5(xlz1;C!)LaG%aB3N~S?PlI{m zQ~aDM``&q%d>*iVt)(_csg!pT;bFyk?9pz!zdBu4Rkz)E&U?h{`wr(ul<-wKp{*n@ ze-sUUwB;k0$Y>lCwH|BI8)!Cg^+awrmZiT{#lICHZ>; z2oPok=q(R2A3?Xhg)fCkZoQ29W=MX)`L_sp;iDa*_R&_(2xN=h=$E39ANpC#8pbaD z(2=4=`5+J=2q6-R6l+E!n?XP z6P09Vz`fr3_8O)aej`-Sv@z}Ns1Z06m_&_UA#l>IUJ7(JFBmr^cp;r!ii31NX5<-; zT6=|lD42{~$i14+VX0cbfy)ODApR&=x&yXmxkUsC+jC!?(r4~z^{0g{WXoc2TuFXT zo+R2ik>lKj-+dYY5DE{tEivFiih+LcR_dToLGa&iEe$F>Lgh`0Ph?)~4b^AClxc)n z-8PY{T{hHzXc+<$Kbha*sKIPu)r@<4=CATwE9$JQZc?H7tY{C|i|$(fy3u++vqxZe zUAAGhaLx9>p&MYGnd>}L(CHj}R;FxhX)@VEF<#r{9Bo<&{7DT8g=e4Bs&ZjyyM6P& z?#(`qW{Hviq-mVq)W&$(_g2}R)XeV)+l@0A^J_@B^>7ow4@`j0vfcnTe47G(RF`uW zX?2^B(iA8|pL{r-Ry8iqFrm&z3ulqE_V;X=CB|(T7ifO)#lkJ@$`DdHo3!!5+kV2@ zCITHz8Xhpg!#@*f!zNhw8X$&GKmsPnIAFpqpa-o*$pe26TUbU0rc*b!29paBFDjrS{Y$kV5N*0_c86xq zk#ici@74Ta5EQtg(gACN?^VJD2ZEoe!uwgJuc?IO^Wx|Qd}tWj-X`R;MRnJEq{ev) zJ1qA&U7oGlQ2pnK#GlX9yA}CdU;U=mscEU^&w33v3fif_Gj}!_5u@Jd0p7izD+JlE z+`h@Q)cG<@BSxztZO&tKD<}utA$q-+`#Mfa^=7j-2->el3W|}#<$O0dq%0G-dqko? z5*>g^^z3ke_vYEWaHx0_osg_roo|N{!ng@n_l!Wn_0I3_qOAY&vrx)6(2^2|5_6*Q z`*+}4X`u~iIcqTLO|1N;3gR2stcq`;(z8`nVW~N3C11Lxz=&E#owAh!Qb*+hrN`4Q zvwcDTGI#fWQYyaO(B04ycSgFojR;7IvCtCl;<2(EYc{+9xDmm95I%4$g=Ng2ps1ko zFitHDwN@i42* zvW-Tg`-j>6bMpugG>E0~O8J*(9B_+-;1^3BK60JyU!nWm@*aE=N`Lr>j$Aou>|P;%a z9C0|5CC;H&yti@(N%;)AYMd3P^_ymB7`H_UNRcReeBGZfSC!5)7%2Bj7E&y$!It-Kkiida z5sN-bo|j`^Gzn8D?zzr~@J!Lf>CTyT#s5zA+-sNI-5=4hZ9!iLky#e%+GJI+z}ZUX z^fwfWjS)@J>m!{uU9ZDod8V$7s#$NFy3I;XR&JOG?^1_zzmvyU|AMZ}23CC!DbhIM z&0YSV)(wXvm`Xehh<-;@Z~U_{x)(KFxmUUF z&S@TiXsOTZ6S%If(QtssrgDAa@pi>?Z~@b!a5PIzKCF+InFlvp=B6yM)UITJLpY*I zPZ6GP3v)3~wtJ^j{mM`*L5AJ_Y&Ey8*XL^AX0zgOyYaO9QPRsfd}i%v+n~9` zwyNtXVAj?zlq%gHd)s67+^B`KDD?MNfE!&Q?Im5^JhK~jg$@+Qpy}B#fHbuJ@~m}b zg(fBSej+TewXPh0bcbQp`*2ue8b##fNtOL#ho)}#u+POjMO$;nZZPrV4!LArc6_xU zo2I>(;vdaB;C^*0P-YwGZs}xuxX$cm=s`cOqy^(bcYU~pp&cn)I1KM1ZBA*}t+6`N zxPsr_@HrxW5)@uTn;_8JiZ>4ZqzP2PLKJ$G z%Zw65C@@9*IPi;ZZuC?Kj`AF^Z?uk|mP}jE7D91-(uNUu0q~B}J#~g(wm#=NetQy@ zHcgjD%2Fz6xx-Irb)Umf8}*GXFs@C=>S;xI`&&sf7DAWsl|$Y2$s>FsxZUNC{W_>1 zvdMKKV}n+uU05-qU8ImEY_rn$5kx*#MG4cGOqyT5ZZ=(CZ2{G+SQy)s{8cfk@1U)Q z-$5NFI1ow`jt9Td@bDOB{(v);Lp(Tcx^9>&D8&^jGwZfzZ8%UCJH0%gy>`An?e#IT zbU)rAW1a~{o&);@fMeCXV_@9G=TxT6BtiSZY6+_Ujcuxh0V**^TuJ+TpEyN(`IbqK zH_CeXQ%ovGz*g0;(~N=Ki~5(%|9ffWRDNE^ z(^ctOmUXu1R5XyS>iN)TKN*VU-~M!uP5g?I6O)#c*CDOI3sf8%iD9I-Kux@CkV6bn z^)_z?%*5u=`Y{&Q4+tHN8JAFD@%}<^nRXhj3E`RH#kWay8qsRQB;{p`jG&gN#bB-P zO97wAF9#=T)jEC>I2=-$mgAW=c_L?S;C;FnvDeAaIB8Rtfb+KplL1CI9*CQ zb>I5({jM9&A2jR(NwTg?fT89jPIGzT{ThemS`B@-7VGNY?@qrMh5B5s8z3K}`tMc< zSRAbM8!9UCpr>cJx}y5^Jf$4!%v!ZZ=fzb)(7J6%g@J*Ao(2#2{O(e{>K7wuk)H9~ z@16#`Sm7Rf{zlU^R-)}&iuc*7JmY(?QLJ22gzOngpPvfd zI6K(sLR*lkr|OJM9>R{i&kT}C5`a#*IT3-q6fSy~o^g9&*Oubq7s|BH_4%ylMAOw_ zvAkPRiMU&u)?jG5F^VEN?Ojwye2CT=5k@b~6IFX8UAOCyNSa?a4&N^kr!QS~88Kk# zH3mzbv>~VK&U2T!547*JY5;9Edb!Bz^TGf3jSq(dkw0=TXYk`%l%8NoE;ypagU+6` z^5_D$zu)ha)(hbI{*}?)O+oY+rNI2@bc(aSfIymgBCz-gq>h2n=u8BT&4}O9XUwF3 z?#X*#1qBrq_r4f@M{n-H?EA`PY1WgRaOY3JeS>!XcuNZ@NG+`ft?AnwUF1FhGNy!% zr*IxQp^!N$I?jm`#;aoc_dY!q0T;odN|7)&Cmk*h?MSFHPt*YcSMAvSSI7}wz8pVp zG>*{^k5T>Etz4Ml0$5fegLxpyruNfdO+7YdSBKVVhH=-7`+A|~rISdHww2n^wxZL( zi&tDI_2MRPLe*b3y_IhhXxB@@b7H#9yfM`t$6K3@Z>RQqMbZ>vnW(QRzkft5 zW~oT#Fb&lC^8CpUnf#Qqqs{fo8F#Cy&C1DIm`iY49iX8OIDQRN-m!*f6Z}m_6IRRA zOuE&5aw+`7-HDGG`J+x{49+60f*?$3c7dY#MMFjAscx-c{|-f9%KP~Wo^v=#3GHZ4 z<6B5<>wm{GX<;KbfMI@T{w$nF7{cdlch=^G`7|KNbaJE@k8RM_i}1PuQZy)!(yB{H z14@|}Hfxa?lH5vtzFCzvIPJ9$?=64R3aUV znCqv1&;tt3oo_4E2geQDUSdHpd$bov>DvSpr#~fvP0$h@~Y>S$G z58rZhOhA(Woy&~d^5R86_-p?0a}Ez+TG~S>M1+@7vc>kvIZPy(idP47C3X*sfD=wz zuAgH*@|>!+E4iqTWLw6QbE^RoGw$u4M;Sil4J-!iuD z_d60MDZk68w2OzF*70FAmpw(V1z{FhJIwd8J89aTIZLctJbh#yf7f-to5#VpEwuX- znTzIJyor)(ZF|;jIVFcexf?Kpk$Gon??tm23n@iUTDyR6<1wmJuhT94!q zhrUawxAN;^BP+JbJRpSJdk;+6_v_yGQhey?669_VmtBH>4AAVi?@>5(u_s7VHhE-H z_5!Bn2czCQ1X?F+394?FP1-hjJ<)rr4m|%F z_c5Sg%}3bf5Er&-I>?;I3_qFD;6dBJ%>&33r5$^F&+zg>KASEz+Cz#)g^+bs_@GZT z)9+R3JzS_1)d_$Z_a(2e{LqmA2P)Lce07qXj7!`;d#QW{=rjDwF&dk8VFUDZKp`(i z8h)|iqq5#5HnP>O{ z&b#B2sZZ3#v39ly0nZ&Ih6mvn*YlSHosuHdLLmo_!5*`@o@(iQC~fvixZxzMslmlz zeJ1h7ZBglkIRBcQgz+k2HJ1a-PZxV1=39p8USC~egdX?ZH|4*ZrJI>q2ZPd@j>V4B zg$m4**y_->Pc$JKgjX*xn!&e-uYQ+f@J+kWcLS7)VBoh zS*==F=5CsE_qXA(iw}*d1AIT)_7JqE6aGfd43_9Zv86F{a}%yY35SSe7(1Y@Rr{l$SQ?sesyh| zGs5Z@69EXz6k>S|R+wcfW54cw0uagwA_0~mOOMNvk-1{a*Y8T3=4o%!8cc*=@u-p_cnPR)!0SDfJA~nbHY6elp`mVr_)f!OZJn#t(Iy6a z`39E^72fS?b?uu6v0*pS9pI{UjG4ZP;pZfz=Rx8W~IGYmurKU zucKB@#zCaB>=>LFiXu-Eb>cx`wD`jFGxfs7Tw=2G9V@{v$1@&nb9pwzSa0Wty8_iu zC|?G2NKGQDVm0EaC#~Gei}yYBH`7Q)M59!^1@YLd(Ac|;54jOa&pB| zr&)MGH8i%rv$CuCV>q=u>fu)Xe$VT?!<9}o3!$?llO}?*q#6-cS4$*lIuhbK)-4_P zgmNi%#VbVyB+<4j)O>J;7N01pt4+3h>`0{R> z6c376n&VuM9ZJR()oL|pOrEswWHw%PZ1tId@_#}YaBEXfZHnepKEhLpefvf%n^6PN z#Pp}|@3%P0>t>sE*H`Blq0B=+pgy-OA8p!-HnoA-YUv3Fzs{5?@a}Z?ddw5?{CPe!x5Wq~O)zb=qkV>Ywd)LB`Ru?f5|`QP|79 z%a*(;V~`9WkEM{8WUEUqjz5|Q1ohj0UG!v(QTrqqciE=8m$!68GRsSl+C3x0b@0dz z+pm+KS{Au>TaJ0sGiy5^%D+apyp0tY%Cj37r+Hy7bfxR%vYH*&n&FWheEyx{>eq!t zl&0&-Zhmq`>3243YOJb`fZnH7WQ92{TQ2X!CT9=ztxPX`Cr?w1Keb(GCFo13y;_** zWNq5Gcwrdy`cJCQLKAXGShezVv~T5vxTy>T)vwM4PNv4q)5tJJcew$-8RCz(zq0bH zZmYHvMj=RrED`Kif5#FLJRB!7PyXF_xoIoVbn@`9MgyS+u(hqJ082hzDvry|JQ>ON ziiTx^(lb%B$f{zU?j+BWsqA!9x-BHlx~&~eIrM5e1d`o~V`ALYMU zb6{_UrVF8^BfL;ZTmVc>d4g|GC1!}o8S??Y({}B8QOWXEd{Ml}0jbwP|3{wMTaqpf z(HuCm@{Zof7Q%_b(k8M17}p0oWCsA30|H<$Wu$7+Eb# zMzn1}yX-Ui1Tpw9iD3S4vV~f-hL9}lyRT;dFMpB#8+FKdD)Qp_P-Xx=b1XM;?gqHr zCGWjy?Ya!QsF9UFl@17RNtNtRZ+lm1j`VwpeKxRQAd>K3omktVYqT!pS%tYAE`CHW ztG+1_t}EWYz9bhJwD~Yd_xa=A5u<9J0mNaM`N~H;hQRIH;s3!u(Q}nsXkqRXMPA^Tz@CR;M^dmQG3uTbenq8f=iPVCy zH<4%f=(zMp^I^#>1ras|r$WPo(PI12@qIA?`3pPKn~d>Qu;+4^JX)qntpTuX7AQd? zlfoSm9SJaABO%|EITO?!BypBr>OtiL!|l7DppfC#JyT(H%#H@kYqZoBKLdUfbMJ+c zbCE%lx2d_^gk^3Gh=}MZW+{~BFZ9%G0I${)@cT~`$VCjZ;f{U|7UlV8AjJJiDnm2> z&N$KDi_%aZm1Oq{nu4$zryrAI1bPcSLdUwfF;c{DrODkE?vrI%_j6dc04s7e zSI#KCS-lV`ql{8GF+Rs?8M;XAFHf@sD2|}e%rdTQ^%aIx_2)7C9Ibm+`6-X9dpb0n z1|>de7~3KBboh7Lc8Vbp-={l0oORRkfxGxkei zwflRk!yQj+qIZ2M~YHAc4(XDOHL>F=TpET5C_ zHe+854WqehCI?~*ULQVrT9?-YeA6WVDVN6m6pOx`vVb6>isRBwmY|*% ztVtWot)^>lhb8Mx($7CES_++&6CPGI=Pp)U9!=&Vk3M`^+;H@nZWf^GvR}u|1}j=h z20Z|UJi=AO>N;X|jt7Qv0q2ReJ?sYd&g3uRe*lJ$`i-K#zgOUavtpI1B8 zY$OhNQa*{;hPlX)-$O;q+CGREcrw!^6MPqCk&4(UdYrOj)6cr<4B9dDJ?>{-Xb^bnZVnDuD577RAPOj zjYkz@#)akkbO;qE>->yPk4Mfa@Oy2va(}p^cf?C4@|X zySA9^Wofi82|x-gjmjG1wJRWJ0i`R&57#4%v8C* zuXDPEFmv_?B6?r(U_wLv+)O!*0?mlyb=QN1DYolAfh>etfYtV&Xor6`9_wC_>|aQ6 zu?(aON;9i+F(~5T&uChey>y%Kcz6e5iXAR2(RLM2MkAwwki8hdK^KB<2o$j!mC2xS zDujfmM`|}6j|*5a8@Y}F7Rt?c?3w4bs?K|KVaOQx&jPW7a`AW;8GP4<@$+?PDuqi| zZUrp>>+29EbfoF{a%vl}SYuz++(L;sTfKf`7`#szq|mpqTwJrs)=nxgKAWhU0_5kV zT4B+^4@#~L1oXcW&Ek&~*X=rw8;=`Sik^&HiU+CWD5Q~paAsQ{u=d*Ehec%b9-~_3 zDdi>!hBt9RnKD)fAAgZ5`JuPIZR4R0#UhMXUZI0%ljaRRZa&wK67Y?r4KxA86V_kO z{bVhH?>C3x3 zBm`?PRY!mnN~P=1$8z@Yeiu6V^``&xs)vS<}=9i z-uv=waXy!CZ8$D}!~KXSr9?!*qv`DvkhWC1lmpRsuqPV0xBM80g+GQ`-hQMjAXRBc z(mRz);%4SoIQl@FD%q_sVYgDR<%Prx_^V6-58$9_a#IA_&t1!(=0^8Bzbvdp3kmg; zA*T0z>M&*3=0BtA(lD{NmS5aD(+whmNbNLfNCou@sFGrlv|mnfX>E>NUZlM&`6;*Z z$vONKgsDHY^Mm;Rly>?7^(eLrV((+O1yCe-iZsKlX`DsYjuk5{sTPjAjx^ zFmE`wk5TZ6^29iK(C3QOAp|MfeIXwL{7~)vcuw=sLAR=p{h#HW4zguw2u{~{*~#+p zx5|~Sl>3rjHKW<|3LQIE;?=6w#0ZYEzxvIPlw{jDez_Od>~x z6amp#@)oIVs7j>UyvyCj3w+pH79x-}k8+Yj;Lf_u8E8Tp)!qh@2u%SutH&o1Cxhq+ z9NqH%`MSY1u@{aT;1DsE4qI@cWaDx9dj~wH8t-pGFeQA+pqR(fdH0aCgM33bB@!kpdpgTpx8LzWPgqv1*t96uQ{?-rqyD$%Zb+j;xq@ zu=ttvQto8rm1z1rz{_hVE>KL$z8$229<$LG(e|>55IQRY1^dyCzK^0RVGlYPcxnV_ zYNF^ErvZvyzbmluWo&+juo}jImY6S+UaNyGm$pADIkU{(sCU~pG8}e)+7iDv22!2I zTG{lzser%LOjaf@fq8;`DrB8ZLVCRG!(tybRkG{^5g%`+?xMOa;{EwQY(5vZpL?hd zgQd%8k=}H!@F6{qX$LNh`)`eWk=q9d8a#~eDny7U>a9UH6?V@kKfUd5Xt8ZP&8b^X z^)T|4p1wAYaLV1K72ZYL7q5(#YM-C2oU{>$OFj?yKwQD9LAGk!%;Qf2Ii&T4S!jWRFXWDK|BDQacgOxo=8l*w8&6^+d5%&A1om<-sa==oQ?DO zSA?qbt-2(hL7@HR6@r#UfU7f6Y`KD^Cum~$+7WYEKFZES?E`8uC!-YCK)_5kvJF!` zS__j(Ayu#@L)Yhga<;Z5h@fpCt07YpNf4e+@Vom^G0ZNgRUOd4?VTEpdoiUt1u;JW z=W=Yk>qKFy)9PK)rTz8REWYapE>M_lfevXWgYvs?5;UEFtXb{j=O8wAZl536)Jja5 zVvf#1kY-4YJRJfgb;1FU)E+PeCrFA2kFNwW-rWP3%JMkwA>J*m7}+9irTT>uBlJP^ z9xzvv{ZhBOj&+Pnoc+q1bCAMPWzb&Q2edsSf9EJw9*g$3D8Z{uKmbbjMqyEiw$6@31JyJBu@gcxGIU z#{q~N9S6iOL4IkpnWaFsQ3Tu$fSd3h+R)0phEd`B^O?(qsu96s$Man6Lj)hsLk+K%W; zS{)%{HaHuIQ)Wf$k&Skx6l+zIAj88L?l<|pYs-($N<>>~y~UM_u4P+FQ7hyL>(bwe%8vl7-bv{IgYx5J%$O9uznDdOq_xL3{hIc` z{CAE*gFXOck$N0wF9<+I3A<>qe#6vKlC3_2mZopWwNVjYto`(j+{0OAnPJw{&7bkrmyO~QLgk*KC1@us_kAe8VXY4J8*J2ZfdIF zk{{S(Z<%~g>(spIC^Ms{5#nY#p9!D>$+Q>bs0e!%D zhh=S~#CGngB+RRe2BpN{YA-HjOGY{oR^x>u&-(zsK#y%X*+HI(X|)DKLFmq!A>=~* zwQL+jRhgH$sqUWv%R3pUE*hsq8-@)MfVqQpQk=?pG}rj#J{P;?c{eG0So%jA#?gA! z4^%(o(nigoelA1A%y72l(nJz8(sjIU$sQfrk4ZC9%^-XYm zH=b)=*4kQNNoPwv1M|DTeS*9NqgkV`au9?N?^(D)=zNK<@RXjQa@|Hu87$J{_}H(t zL7cUMS`15-G3(WGP2>Q=lAsO*9KDyoG-75-tcHP_2TjctJ^+?Pi-Ed~CP%YO_-AHT zJk9iQlNY&kgk|aYH-{pm$kmPkuBhI*N@ep()X_;0IqMHwqF~mkWZI6MAL9u1DMLE6 z2~r<=as0vA%3r)r$uTbScAh)yM0W;fV2etIv8H3=muU7{-BUmkeHj!$*C3%6DHI=& zPh)Zz(K4?AflM3u`mrc_1PlULX6Fabe5x$7av9(OZeQ~PL>mOm)c7?$Jsi==#3(*2`qPBqr;U?0 zTD=Iez3-X5`-;e?FLTckqS|^scWZZLPA{B=I|DGa#J(~=f#tpPrklA?&I(&sa3w4Q&OZ7*|v~V|G}EhxM$N zMs5F{zGGkM(ZssdOZ|&BHjtC+n?nlaTZ+G!^*lK?D4JwMlg8c)^c&)+w|}sVhg9Qt z3uWm76k_a%FjHeE*3-YKbhdn3Lm(y6jRzI?Nn!teA# z%WVW$64Mvcv(b~=G~b5GL=<8}C?FM-&#!eLI_^ZP=2^KBjpFVktF~Ef5yBGgDJSUO z6yaP_$bJ065cG8I`X_sDp;7l;0djM=x*I=)_~BYaiEC;bsr^(fnMP;BC2%3@3w?~i z$rOb`oCf)c9?d69k^O|;XeOQe`k;I&S$Hepv~x3zj-Jtr@Kfdlhr?1=@@%(i-#e2e z>C?k+5`vd3abL8=rmyH4lV?LzmBz17YmyCyb+~31e%0!uQ_`1TcuDj5Jk%GnK5Hgu zR2k!3EJ}HLU2RG%E}9q!^IOCksuRWUe5k#1?R#bu;@T^T zgb9b05ml`k6MnfN_&HPEyK3kJq;@9ujzm;W&phaTQFDqJ#O2klre9R#%hC6$SyNuW z${{-2x1T|?qwG_Mw0rlT58^@xW=$A;0hpJ+9s+jhioUzqoM%L_FQrp!QA4$q_f9Qdh(3LLN76 ziodX)f$yrjV=J!`#ys|5@lW@9A8{^gq-#5MR9><$x|}Y?36OovHJ-m<$SNHSCFL6s z0y!~Q)k^0SwrzrS{cfub$QmnMERd<-=iYxwR=ljCD<&}#WAyPH^@1g1<+yZcyLh4! z#BkygJEwR(A$_WAu*5$CBqjdF-OD_79#m?KA&n>lbuu*oVfC)|G%S?-NI9v%<05NB zmW1V#DHU-bmTjfh0Cx717EeG1L;VIX>-Eu8)ngCWI@O5Il1lZ_A}AHHV1cjJq~FW@ z@Vn*FI+d085(+k_72G1s=sCR2eFxhNVb`7y)fKby92DX&z>4xN`mMHrHxAy4?U(TK z>WZ}&??9OTq|y_x!}Y@3<=Me+J@k)l6h4B=bStI0^d5k@_%mqzpA+HEV~YwDzT*8pjRB3O1LMWlnNbdj9< zrDhVY#e-HD^y=S!zty$N7?I&{;ndd0J^2#n!`4NFeRc$h4+jt?gJ%(O>qeJB!;u}V zsAd`v>l>*aTYR}PpFsBP7Yym<%J4jXUc8hUy(QE)+c#C?kTPqBJ55p!+>1byz~|2w z1y=w%L+5e&W_|XqxLb!qu4l9x-W?KGt@J!;@))}W)fA2lD#p0}mY(TiOxTD1+HAjF zKDw+4D^cJPfU3Of;X$&X+O#k6ezxrVf>zH!VRgufdY(!dcAka-i|_$Y_nFSq29gx0 zbW`;d$ic?B)Yp^zMBmn56|2O33TnW0aG_x*y28)-8ZP!aZN6hV9gzNMQRquX0mcZQf6?oQj2L?Q zz%sjO)m+FU3%-+dT^>c!6%d zL%P?W3ZeUwtGA5L;_c}x#1PAQnM}UDM*c|fugY>t#n;dYLVo6gXWX#NoOb-zKs?5b zbCU0mY`%QE4SQJoHB|dL4u1qYB>VoKNC6wvV}6Yn&Uv&>wHXz#(4&)u+pnd+s^Q+2DTY-ungv!u*u7@LC4LDq?Bi&P_eI zFH{La@AX`kLWRq2=T+fXy--e4Ct-oqP}vVFLzQ{>9gsZbyne8}cy|&H zi0Y>tY_2X&OGbdVrf>wo46IxW%6RXls8Rc4VL801lu|Je6a^Em89x=0>n{MU3jPWH z$hEYou2i1y{;rXO#m00`;2WVLA&2=m@#h`U^joLPX$scEc5P(KBl;ejL*__NVU#MY zlDT_IeC_UyVYx>a2b*HBEjmfz6ZNlBE-Cb|lv8fd(b?7#y=X)WZ+W{3zoF)|=_O^Y zeNv#fUkVBI4n9X}1XrvMv|}U^kN_M1pXMZ3oj-+&tU;8vJw38w{7(*^bL8t+AH_5) zUnV38d)2y?z+&@1kv+iNR7hOTBZT~xgc`ITeg#06Y5KBd2u7QHe0TtaJ_eCutDb#3 zjZI<2gnhWgvZ+)j39K#8xQXHg%FQr`G~olR(pc19!NIHMUHW*5yPQ@Fc|WX7ve4dK zuL@-{J)bDIgb_DfH$}qWo%Bzrpu~k7RzM`DX3hYF#Knpjl2UeOeht=1neK3}FJT+h z4P7U1ov%(*SS?#{6>bcH?f?PMA-a2`L_ox_NmWAW717m7Z%HD*CHhIm=(@MizNq?| z_m3{VqxsTGY*npDT$!PI2i#kCNK0!^OX_2<@0Kug9r+c;9RO3t87)F1b^oIpD(?Dm zppaF}(IC9o(HAO7nY|rF{^(N@T9ip`ne!=~ZpVWtmYKror)KM|Qtf8?S6eh`Vd@Nr z>!ZCW@P=X-Uj#-+@Z*B)FLeinI-#eIKToxDLO`Ykp8|ap27MVLK5gaqZXWEQ1@iLl zT280_HKWD6?e5`EQ09~z0*cSQ`WTQE7UW7ul(qXvq>VAK#yP{EYXKKi4A#IRmB{5K$01+-J(u#8lP`y) znUSFx<64?}TUY_C$9B=a-Nf+Dzb09!M{*pEZ|y?c=h=>1z?@$bL?^vjL|+E1W#gb_ zfayk`@&ZB2hQ%4dV0%1SYy>{O5bN??N*Sg}SwiOkEL}4!fG)Qm$tPRr|ECjzlFf49 zVJ^GF^O?OZ5A~p_2t!@#hhJ{AG+hleFjuZ6W$WEcsEGMjBhFK=VAB+Qv)(;tK%sh? zY@>2MK3*60v?sLO7%TbPkahohPy5$*0l)?GB@x=8c`ARs5;9EXGL!YATjcwn=2#17 z2>m6gmr+@h>KeRzQ4L3Ol_1+C;;O}w{iB{bUp6kZL$Q^EyJ{TYI|i$dMl>Sfu~jTA zoa+kuXU{3D`kR>AssME2ee_vxVcFnv!S%Ouf};OXf&Vu8fu5h}nHER3k456yTGchC z1o2GB;ifrAQdOp;6TK&{6M@m7**n;dQN7^jx!wP+F6=6C;`EJaf(ERb&hxZGBV(%? zyjY&#{CzsOB0z)gfu6qz0x8a${7AYUs>wmnK$YaSkwZ(CwJGD2>`G`eNWQugip!g& zE;S8EM+kvX7Y3;Up8Tsnqt6n};$;E5eY5WNW#xI>OaH}x>H!r@4;IA!m}(0G2oY%Y zVzzXTH;Es9@*CMK@X&k{$ayl12uWNH_Vn zRys;Uh;eA6MQVRKBokmxSbw#tY(9{+`@fwOIQxNO$==8upZ;W}M?=E}=?$V#u9%LjLM2qRsAH%YWLt%SjPK+#Vmv(fJJZIMg%k7%^6IxsINU z-#%H2E#wW%=lW+a9v338`Hx(sRPW!|pf&My$tF6lKxW(_Pem#(*#T>Q;xe!3mnzbT zxMoH+Wre@83YXE7))uP_`@@9K@&p#bMW;ahcJuRpnPm$i2=V%huKW_)W{a1C-9hv| zCNi?8#+Ck7vdmQjY(+=+;50^`F1VdZ>t0Gaa4XI1hkZGxYEZl6n`00mK%bMuu7h`X zdaq5E9ywf0{gD}y=y*H&&tj?1{ss%vE^h5fAZ0P?wHK9ULaUgH`o}fixH$C(>BG9P zH6*dQxLqqWjPdfR?u`-hlSMpp`TM9K$;DvF)ST6g^VKgX zjC*fOfx10}4-Hq4#CYExJHX3<=0@_pZvOF39sPl5A#{w8kSbHF0_F%0GS=XHAtYFt zGr$|%Kp?H&B13oK0+C4v-J=pYmiNPq;JW^j=qhFCtXlvBF0xKmK_1$3@cSap#G-agu! zM3|j2B&~^wwW|p5y-6(iwZd9{$F>6|;9scVg%&-|D!q-pru}k@@Ewg4=SPt;C2cv) z&=N9eVA4|M7bcy=&jR%uE2cQC5n7;;=^XHKo1k-k6;!eEV6^@z0D$ZN;5|Yfn`8ly zXqg7#bz;Fj6UB}it4zZfg0NhPF*z}!=&eL!J&-5yaR>3(;aPR#j z;Mu1@6|imCzc26?B+xsx5Gs6T;R0wyfe=faC7bQ**f^g0v9Cg(kV>D_rv}_|3(1dq zbjMgXCF>2vjcoX%fk_FN%YAjDycTSb%|6x%Qsv5S64QWvaR&lxFB)`8IezOotq$wn zs7eOD?x1o(0oV2Dnev&)LqZNULYn@pR#zT0^Nw08u{Dbsjgh0f{u5 zqm6M()2kkotDl|rtiPWNk^TssO|#j@>?Vfof^8&D*t$u{u++IyS47 z<7OltZ~uuN{O;1enzXK}>Gj%j*qX2^b|S8O6ddY)-WWIQE>$#yjwkHA9frAd*mZj! zozwW%`GV`jMR0xyF5HIb5FO1F4d5yq&CeO`~H+N`C!I0aQtM1`_IdVpJ z?r|=iNymrE4-*O&r?F4Ft-XV&bym6e_o!JX-RRk1UD+_qK`jyLXjPzOsfJNX{TH^q z!ToqRnl<*>NnDnZBp`Q?^jK!}EGx0iYb8Nnf5+&J6G?q%>_FSYI~uwqKRwLLzkVJg zOtPz>gM;LBQP2(NjOvUIZ?C#dqnPPkGF0CmEmA6z_@bO4=OV&7YLJUYF%}{jr5?np za}}ZAI7f2`N&olv#fAI6EW%&Vl~-bk&JTueWyk`n%Y7I~GTv zLaImcv&}ClSp{nBWsDs|lD_XcsJJ3Z>tF;3kBm`LK zaqnJUhC%XNRF!E_o?AvY(f691ZnEyXFA}y2#1XRqVUMe_^cH7$;m6QLAWmR*OrfCD zZZEt3(Qd{U*+KBdM7bZizL5wR z!uhNnt;&~L^pA)=XBQXUSry#-zY36sPeO&ttM}2*w^>m;C{0vt+O95xdm@(%0DAblo-?xa@O|9cy!^3 zB8PVqb^jmA-a4wvt?U1WO@m0cbeDukNNvI(l#-AJ36<_<1B%ihf`k%M(kUGUS|N`R0lc`uiw|}d*^@oS7gDcr*0KHG^t*v zdM5af=mKPL?gXLd~$sWrwrKdN`ch-J+Qw5epHms)lSbr0_! z^j~%nImVf11m*n7w%2Iy(1{)=bZ9h!5h-EAOLbK!$KNb>M?h<#t~6Vv@zj||GCMurp4#f=fMDfFCdw9HI5&}L!Do$ta! zO)B95#laSc&~1S&bO8(Y)d$5|BJEUYzWH;&^sBG@kjnJbsUs^m4t@E zcVl!@qak<}Kk>|RM@o8kL3#A?8~ql6%xd$m!R=YNaWU0M#v$hvuDE{(&|Ijed=mp5l+L)Q!ZD@=|gIvnDRq7^NIQ( z$@yas8H zwS@$ zZ!8nH`5fQQ=xt#;{>)U7W<2WnAvm#G}DCxwOlx*KubY&R`fqpHb}d3hxnhjnw1uLK*tqYfvZ8) zy@1Dwef^5sHtwQWT6CX02iGUI78R(MtN!#Yfgh%Z;7_n?4U-Q2jA-(e+Mvf8M~u{V zp!w0NHvE9{zu^Rx3|~~nRiet4^r)D&5RfK8Cx@)xtTm}CXUTDnIq;b z8tI#Ig81Ge)paX=femC#uOlH)D0WKG*;tN(O=5fZvLo(CwbjoU6f-ir&pi9w7Q_9! z%)ai0Z5ePt8aZ7`H_bkmOMIVcnY@yGgEo5HI|Y`sEpjv2(3$b8@7ew}(hL;OosBLg zM)ytr;jjK#(AaC&{^D0z~xFKfC@^*%){P zNT+%m2;d%P`Ltl&TFaznjq3Yk=RSPxWGj!Jj zHJ4k332QTZ9tXx0gxF;K%O4GtWtSr7&ChrVyb4=*d#6gROsyFv4HNWyh50|$N5oCm z-N3`6617kL1e%8@orfQ{{xe*@ejP#YAF@xF76kR%gAeH+FYoygh<1loV2@|$UBBi# zSIVCjT#q7M%Fm)s(!@G$iDdZyIXcw-Q1O%C@EUHWbaGyws4TuBa3QWZpSR!Qj1z*oiyX!f!D{$wLp+OHQVYX^3k%d!kGUb z=Z(;lSvg^)8DTP6=-Yeo{;f}5O;j8W*_WDJ9r>XzR!5RbN znTmC>mRgb^_U7io3yrr42{U_~*XtH5uzb^oMuh82b>HoWze$$OJxHIXk)sNy+E~qe zjO6q++t^MCt@FOytBZc#`uIEU{)aA?jXp8PF8r!2TiXFr6Jq{~(7eKJhvL+s;bBudjNBA1 zq-XLnt^DQhmmWG|E+saqlfPPGR5;Y(-V>p&h{f=V@3R2Nwt3k_()@RB;q{|@V@lN= z@Y=4noN%!3I1@z7vx3M`!Bey`R?kVlAhG_J(hRr?)0_VMApVn7d>_--m0|1Vgkv-B z5X|n!@oq4l-31;N;{~XsfLmfgG5^$MS8=&Fsg`!SzqQ`ui6(dMl$_fiyHobEdiZyM zf`0q*CAZDc-&2!F&jUJ+?u=+grpEv&fhOzOyyS(unz1@ghPxFy(~^oEpPT2}Z&txx zhj8>ifte7VKi9eZMd~!qnv;%Rk{P1f#vgRf>a}A#uBW|IUubIrRp90P zUTn3kVVK`+1b(NEP1T&N*z(o9;1UF(M0kw!Yb=6_f6;`lpU42a_p{g?bqKFfg9ML? z2u_I8BLr8~e)k+!`hPlVgRJ5oM>Uls61YS&Qgth0Sk$*G~Es@-?pQ&jo)lF|M4zn%@61PJfjyAo(Y; z^Uoni7I}9St(Y*PGUDRi z)h|}I0Te_FK&1@+s|<7f($GFMUhI5tJCvbdg5gX$)o%!2Xp;(Bt?qM?UYsTXULZHu zpDPQb)L75ItDcb@NAeECpZtjG^Px$>nW%26t-N_Ds!ha^S*{?(w`3<;IKt&;&2L@7 z=~uYNE777mp8k*k$jxqz=EWWUZ`WVPbuRmFu(>2RpRQ20L*bmt>TOM_O(&4my*2vZ zbT-EjyYFBn?_*RmAG0!%`m&!!^ovH=)U}MkwWmKCTfb>Fc04GsdpOb)lVm}W_fP}AHA(Nt8mmoB zq5Zc)HZKw!dfB{@ijxH&FYu@7m#qEth@u z9Ngm|u1;9+#Qy7DvVllA`N!T!ooF#OmA6+JQV`LOJrreplE z2ArODAt}(;6^@8lS8r8>8*fGG!w%VM=2kfPv~_GQfa{*@xz1U+oEQ@D?{@>FMCtwk zs*4pj|F%MmXZGLwSrZghg4xfHBTh%Y2*B8W99WTd!md+>^?L4vMkl`lUL=obBi+* zBz6-B#9vq7i>+Ck#Yi&}{iE!G+}t0l=t7Z?r_e|>6U1!O8^zZDND;OkGo-8h6fyRZ z11r=M{?Z$A3w88#CwuuW&qGu**pDvheP5F;%o~)1p8q1nAx%Oekh^#b^X79>%duXU zjIP^Sh7fBs_WQNPip6&AHXPv@ciEbV?C)Yu-53Wn=yCdr2@~7#ti3Dyu_mBn`(L** z3&>GY9^+Ysw5uvy8#q5G4R*#M>>Tr!FG`v$}dl&e+8k4`@KT1NGRpaLMe59 z1M6J=I6cbL`Q&VcQ0jm4$sR#o9*mb4zs;A3y{4lAZ2_}?3|wW_uhp)FtM774NATH* z9~3m$yW)2<^kA~9MnF0f3E?K()0oH5V{$!fqW^!(P>ftxg32$X(Z7PB``A_A)}P*P z6C(6i=bMIS4{si|x;g`b`QiMvo%Cm8it1THl6`iNR_7~XMbGA2?4A|+=&{1o~aENp5P!hAguuapQo_| z!3_2tv~Hro3CNCw^OWbl*I(xw+x70}xk%nqf+xMabQpdtQjNry7!{mX(Yj3u?AY-= zl)M-+-Pu99{XmrTGR^{dzzkqlM>!t&bP{7*KrE9ppewLDq{J6~cjFDo6MLj1h}155 z{k2k;Y}U5i>b^112k9l7i#Hh}{H*d%G~y=AIQxeLr?RXH#G@eZ^9B(*vK;bWIi8UZ!ezKHZ?bg*nlMvGtf<(8ON%h#&Q76-zMHA- z9u%}%B;t(%d646G){69PWyNFaXj*h;j4VhTDD*j@JHTUKWA2gK>gev2a%#}a@OKo*QhuWSBJw-Iz9#F5wyN41RV{7 zYWQ;RZi41#J*U3SZ4)5hvjt*nyLf_26LHOgXOOp$QEhkt`U7gI%F5jy|${&Gf%clR`4e*s*nVUp@yUINq z4U6Vh&|46Vt6{Q+x0FWs;pu%m#)^P&uw0{}p=j5>oj7Rpm{#JeC(X;Oa6z^cMYy5^ z%%Mf0Qrzvcl^TLaZfJc;;-goWE7ER2df*06xaH})`2HbMU)RFPoAnLb8-Ns@fDEPA zI>z^FhR$I1CQ^Vp4XDZlC!lwveZCPWaJS1+uB;Ee$^oISBl&O4H3dkRbhLcpXi<_l z(B>&HS40>Z-rkecGmXwM>=9tcKCdW^z-G%n> z(7z|5{^vV9pL0GLq_%pBSPNR1O+2~MIQYmKb@X_wr5f>7z{p1(^*eaky;bfOXdY8Z z2exU=I#Dd&|5K6t84PtVsR{XM8-{F&)j0c*|1skAFl!ztNTmQyO^rhk!5D>$!PM<|dSX(?6ESqAWyK z5L-7M-Xk(@#TV(d%ogYQCReAM{sC_r}b72J64d5AkN+=07B( z_$&drEoD+9Do!lK?C(el&FcDg%z8y0@h9Eav)CY!fyuYqRdbNaW=xtlGUnA- z1azBEB=)}R=+pM>#x-8DlE3|WK>VY`56NhHj#7^I0`xzW@Jn2K_eftPwH})v{(>)h zWFDz^MVnj%djKQd0ROF((TdO4OvdCLnlz_QQRe zu*d=r|DW|Jw0wf?6|$#N)UNJncpNp!~HH>j)eM5SKWkTo&H@HCVar7W^bze zEfJDs&*FySfUd7mS8-fm_VHJw^z!kNvMqSXaO@Z6V@QPG7Js}ntl`58!Q!>7PY>MU zS49KBmDA(1Do4-t42{-2YZ~{OWTn(K=@PH90Ynw-%2>J%q`K$$JU`s zN%`-z`hyj|Wq}z6gG7$^el;tX-oWFt;M1$9-3tWzG{}swTFj`SR{_I1Xlpn6S`n{MG29|P zF$0`gyj;Q=+RS%Fjqn(st7-vX!r%tRY&ZE;bhc zS_09T^08rxzy2M-&WT^yth${fxy?9$4*0;0;jb!(B45(ICWe1*T$NiUb^YOLPrQSQ zEkvrbj+pWo=Ju)&SdGM0oSZ)Mxt#IZq3x@-w%rA7#t#mJ0<}$r+J`Rtd%?-rxOs1t z;RJal1mu=l-G!@p#a1R5)_YdlPw}!bf*-vRZLg^#m>NxSm>ZWCPa`H;T|0{V-P1X!;PthL&%_YO*!7|HhE^ zQy%P3YGQ8>T;qKY|hjI}-8t%!6YqkiszjfSwp{zqqdVxZCfubToq-=xSNH zM3K0I0D&p#;?hocS6Ir?oLl3Cfs7P@i}Zvf=iS%KFGeB=FkJR(1vVWtkE>UbY4%7p zy2MsKYq&}wCyF3JP(20AJ8M|etgo^A@dss2yhn+c(mzl zo>e(4?4O-QPKy{>L(caJvcD_D+?SA>lW?Y9~}Yn-oBrk?)j3AxY>nQoFm-# zmQwU9#3H+Nmoojbxou-D_uUjlwBha}+y=$6t_}*Ly!7!C01bgx-!j}&C4AVZ9h(TQ zr*a1RdPKSKGlQW`b)I@hQMYc+uZwk#IjMF0(nU?ZXSGp|h6&m)HPTMMonBybr!{6G z(ON6g%A_{$CKkzNtAVIoY9%(ogxtKko=NiLHT@;S3fc37CHJ;x+W-pNi@XQA4*(@L zqK&tbPj%k2KBbgeg$0d z$8>j{1CoU@G}J4E@ncy^;&LfxyAdY`_zF)Rr|M$-0O(uweL3rWSj<@@aIlnFql_}? zE*i6hz(tdNO3N7@%dV%G%hRS&Fku?oFdxEg>Z0azyLMC1OJ7@$J{sU`E&|TqYd&P3Rr1tIn%uUmzS}Cbi3nR_Zf zvpJ~6bP}0%{$Q>#8<1aYaAY~H^)K3(SJ zc9JZ4;U$$)tp9BGQ{zu%|WHHC{YFVX>i3)J`z%>+QmTj2gK#B1BDsLiTX?V6$*Cc zy?Dgx3_rc91~)Z+qI|I{1OEZ5d!x}vUs|fJ8Z=AY!cBN0 zA~khJ3RQ)#jw?`^Q{J&qo*~%m9&F@tLr8z1l%rN`LZoO&F9)=eLgU4=AdeWoKeheg zSiKW)$5i(Td0rTLb`meMzQtn*i>fDbc8#(PepYw-kKPhml8;Tp);56aMYiDY_@aTR zbOPsx$dlf!M_3xtPkG)<;5&z9d1Uoa+j%MA5`;6o)L%&{fLbW!+~~V2rrHu98^qmm z>|{(s+sszT<)W(~@c;~PH1qZ^^<$7hGV8q69g@hEu&skM+zK)e_}L2Dpo-}4H-3#@ zk^uui^Du8X%w~Ci=>j}eg$)jh<>HnLAQnl@ZQ9WRg_GT?P)t@fkMEaSXRPsh~II! z#6Av)UpS8CF~tHd* zoW`~wM!&u{2}@toyqj<@)Jk{9i^<55v= zCSvCxH3gj*+LYqm>3T~UlH1Kis>m>jn|eEusJ#BidCf%YTdt1@$>@(#F>y$1AlP{? z-rk@hc0ZXjcoR#<*Klfj1t#8n%nITQb2crZGkWW?JtfB62SAny5gYda)T}+M$`d<> zhV`o+nX>=k7`Zb0>pMeF({5LBR!ICfh=Q0y?pTMu=~W!z6H!Jh;>p&w~!HV-SIaKRw# z{<;#Eu~XK1Q#@n8jZ08cDbp~PBSQ!6|p=O*pQEi@D7x!Cv1z{627 zdU#86pmQLNqY;6@-mTk9QRN=to@HyG=EjdhVvV=b9!O&z%mqJYcvQ49o)H}vumpIp zF=|p%a31w_^VdYDpZ#tVbA4*3lj*hxj>Cg37K(O_Q=s`UdH6jQG(g$YKu`S!0+i*>>3xpQd zL?-hbVjzd!sX3mceX6(?#rA|dqA5(u{5jiCHWz#=cARy*rF)GmiryuktEs~kf414) zcpxP*O3;RlX@QJ6LlRFbFU-o29baJ=^bX&Nj^a=I*+7GBm#8$l-lI1*nJWaw0w~|k zw*Vv}2j|qfn;6zau8C@P*4jR^Y_e2ZW0Ja%Li?JxZ5NU9xW$*|RQmS2Q#ALVoVgHe z%U{YVQj#v)`0z;m+x81}U-jqhmi#TLG`%rekn>4Ckhc=!O28&XBwD?@=YX#v^cuC| z^&Jc)9n_wyi>BIs*zI!{+x7}?-*Grd2v}aut}7PihdjJ=y}U3^V;e>A9=CNQ==$C} z74V38%ys5n8teZX7=SL8xyH!oXFcvea|JHw8C1I(s%EwO05Q@`Z#^z23V!GGY=D5? zF@Ekj@k!;I1AoV(KptA_!?GAQdqdResZ1pSpm~na)oim3S#&*;^9OF4isw4dxgSdvw3zFNs9WEjhN)uLY68{zoQj`v}S50~~nKB^mQ6 z4F7|vj;Wn<<&$_)50Lw7EgitV7%B24dlc$j%9)GYI&mCs`>C2RN&%*o1YO%eXPP#s z?&1NsN2bD~9iL}@k&d4Q8MXD29bc*D#dmsOtd2U0=K9$pw|LP96daaFzxvm`PYm%F zG}7BQVSci#7xTVGca>3}u7PX$!_~mUHR;zWh)aD8!or0cPkTs&&yUW~R~Sv(!ZKUm zvFkjuBi2NHngfrF?V(pmy&UtO2N|eRj%Qokg;h)jMtN=(PApqB;|-7(Z6Rbwt1II! zWlrH@QvsT}FwJ0iqaI&PD7!I2F5)ppR7lSY0Vnly$ha`{a;lCQrlW$E6*AnRxv8$bZ~ z2(I}&<~*Y5F%~N`=7otmhtg%_MIMTij|sJ_3Zh3z62Zlz>r)M**YRI+47dRO$T+-8 z6;R1Ct|+XA`8F+wLSyq77`-H3UwHH|ZN*~+VPF+gRQhEPm~f(?OgN`IrSe*WY_a=f z*isj=SAmDpeyPzu4vj_B*XJzc{K6cvlYM2Gagvf!gd#iJ<@1(3$(cBMDF#UF??qwK zXH@@Zlls6W&75g9>*-1!aHJl}>bljw3dNdS!l}$d37Di8VeCgz6fNG57<65qvuE7h zco;;xh?Soz7*Vp>vS-yF$9>*irr~ZhF^NQ>kV=vgGoTV@<3c!sB%A0>DwuNxX zm$J2eIjWh~nXB=|q2GPyaj5?h%7638=!l#@|H0GT+h8tY;uEBA-TeD7*NnvAi%H!6 z#J$ug;3>)5zsTNx$&17}-9IVn|8$cG47{WX`f7<9{NycGi=&4fHOlLlr;$pvh5sJA>Kk*t3dKDcMVw63CTd z@NZ{VeS|>EEO0iGp#P#+k!fPi>4JDRu#GeIK)RUZ(8np5s~Hk^B=Ol#I`^ykRf_cz zvimGMKwxqSdJ4cotTpRWgjw`=F)q&}DD!BYf5;#a;aF+wxi*7N6lf;n60w^`Z6%15 zaJ*Ip21fLT!}YCTzv`K<&6UbGWB)~hJ zBNJLUf!$|0n09}jo86hp=Y>AHR_<+I)WaUJK{E0HnPs4PVF%(>`EKZ!^c1wS!8?~h z>3X7_r+W(;EU7%{*vt|0UT|CZ)~I!lvu&@)-KebktY{4428=9Oh%t zP<(ijxbdqM3FgbwS;L9?(++kg=pLECyNOn;e5Rtj}yH_AHv0T;3MXWWNW(;|N z#>O`{u;Q=o4yIaT|dEm8R|8aC#*;TvX51${kwWey;@ZxjJ!nll4T2K-U@nL?seR+RJmDOze?T|KwzPKxY-t?-g>D9+#Q%>?26v(xt5I2LvMFc zyO}!YkEgI8uGLWF3KyT2Mt}+DDLX2cU+zJ5Z234{MyAQpeivd5kSl7Fa8S{^&!HYzjCoSp&%tiyjXQ_7 zgt@n&Lvt`NVz@!+z@`5ERD5~UXi8xni0leQ_`6If6wfZ~V_$uQ#mq*w-K9~~mOSXH znp0HMqfqBDjBUeRCyu5q1*N`a1%Qp zfaB7^0wcX(Fc%T6iRX3d*-vv3AM*4mvB;Tfrd=6ol1-?0Ef40>OTmCta}Jn2OcEii zR~ok4zG~E#gGfPrgW5(6h__{-eec#3Rg9D3ma>{F-TY@3iGyk&LDF-JaDrkB7S$1S z{Tn*l}43hHW74AmDTm)vz+bPl`=3CvPMg6EjTy zqJpENR(#@i4vd)YsN6eRqcHL&!cLa(aO?-+PG>}iky<5~=)97f(HP`{rgV0(3xjzI z2dvT^ZErEvehG3xXisZen|#m$)sV2ymG=*0mC3g!V!frdeb(5(y9sViFy-E#A=s9f7L}YmaB5q|mW|33Q)&%{;BZ#Kzs*G>a6q6N$S6nuK=b&n9L(H01GI*jZq97aj<_CA~Hl(gOACRK?%dKDyFo*_@{ z69+B>JT>nOr)-lY3JMWTSpcnG6dd&YVxdJJyBT#b-jE5fQI^lmz(-%ij|G-~N9UGI z0~rcU3U0lpG~9y4Ef|=3?k4+m+!DJ>GeEn6Hm(maQ;s12W~RIi15iYN_uW%VZgY7K z8EA#gxnh*dJv{H6cPskY#>ESYV}7&shBmewRJ({`Ff8jQA~cphWE`*u+2jPwMoaMD z$c)o8Ij*oZYdG(aelt}j<#2*=k6FGo=bz=SFg=&4-tTec1^t;0eASD@ z&0M*l(N#}XtG47^+hGl1iP4IuhGX9^%9Ya|x}p;AfT6k|8Tz2s>35lSxYpB|c;s1Q zxqDU%jOSM^YU5y-)Di*Pn!|Q4k}w>wM+?Yd{t6%sC1KL?Cd@mP)S>aXz}OIUkgEt8nq4OI;ZgxWQ27yV|62nA9OCnRD}RHUZ?ej|?$E>aRZ38o$=H-yb57FFGk=SCbz zBS=(-YDPmG-W5bOf8-dm$oev0WU&t;XlVR$Vg-`!40v^Up=1bzE4WXUESK)QmaU@P zn|xm^ap}eQq⁢?276EGE>AtXa@nFE7u@Jq+*ZVZ1?!b2fP|N^CmP^$Y3;S8f4kk zx!a~2VjHhM!CY1EIiNqxJeMX4QGd@t>4Y436Cfo3x3ka6>p~`K*7a6sKAj_AkBoxh zn)>I6_E2FQmUzELNKlO*y7Z>C=7QY7gSW^4^*#0+|8A1&1Sg>zs2y>(DKYSWU$6@% zC1$04Q&e4CkXUr-AFzBltP%$X%fy&_30R}8aQV@Bt}<4GGHx52RY`~7x>bPY?bBDh zalmy272im6fOA+&gEGEtb)n%pjULcOF{!<6l={XA!zj*t@?G zKy)5^sQua4i{RF)Pmev`@=+LI{P-SBqS0;65yL3zO%?J$`?833+N!OpH5scy>N`1g z=11h4ds))xRT8a)x;Wb*y}_|AnVylico3{>+m3VrfSIN6=eUXlnUwVC95X%YZ_1e= z$nKY)5m;bbl+1%8hv|Gzf5)^yP({m-5F~}!o7@33r8Se5&kErxHBI5Ge#H2CX-Eeo zM_nHzg&>)c;_Xs^N25W{w|oz)RNz9}VHF`V&%6kT#Ommme5lnJ`k9>_t5Q>t+Ybm{ z-)HpKW%lz)_33hb!Nz6+DpgldXAV&ytw3aG-CUfM>YNtlic+8>h!NY?($djfcp`g| z2itAaZSYCq)}(-aLrxmIL;Auam)?`=%+w`@8>#26=M=&8Za5wI(P+r)dr41P6M*k> zz+QcI$3Hg{uGYP_!yZwD*V@2eUI!rDMHuj+rV}t;sl11?@pa|8d?=NrHwD*-womxC+2Urq=Vc+Z`T4=hhb4cr^*oma*_3h4F&GENCc? zD<&=Pb0dgGu;(L`kqjSi43R_3uEYV1@XB|T0m2Ja@b<_j$7Obcn0Z8K7c;ib1`o}+)dZYy^b3A<#D z5}GCUk}rH}aLN01_pVM3TbV&7<@zE$ab$5Zj%_}KQ@)A%HPszBN@-M`A{9kQ5Wt6~ zr}DhF0HtVuAhc7=S#nbix}qIxQY@@UH*VtYh>^k@6WDe>1{7>Iz$XliVteXsZ3ZueE>nW;(^> zKi3jrCPFQz!TqUjthG}5$DEkGOSpLdO@jt+k9G1wL2OjtDcK&i9JtDtC9a2;pb5>) zER=xdYFeyVvGXdsTjgi`Ri5Y}!iVng0~CUziW<}OPWMG@HVRh^6h zeO>*cc!Xh`6beeA1XEvs6rkcJ<-6IvwVBhG!I~gNxnt4$$f(7ztMx6u(nXmVxbeN! zmU=)!!%{liSXvAHG~|}i_#5RCjCYOGqqFei%7E>6r#e)|?esrLW1a;wKi}02LMmk9 zGL_;mqu)rTL*H;6DJM%jyF?s-K~ju%Hr|>2pcwFB;v8RZyPA84U*~XPxlckCeX$md zvB5HZk|>ljH)80HC96;hiZod_`dJUO2V*Wy{O=52&d%CYPLpTrkrKe|4E*;gjs#>R zukMV%0TOg#>CrLhOyU9cWzU%^;9hy)wf+^0lrSTc?k8$oVujeRwYn;VHsMP* zR&nU@=JTmHpX>sqO{QI>4XmWVO)djOm2YE8KEEt^dr(( zwPC9|;RCQcSQS?-sTW8I|GDRFtcIs@-Xz{+Re)q1UpZ)VN$|j6%}=JV5x0*%eko5? z!1Cr&Pub5=%f3Wb;6uHWxY#W7&)=H@qtL3LI>M0%G6r9(Z7uspZlH#2sYJ&qL7`;5 zu$pq;wT(<$UP>D8-7Pt>RmF;$xQi3iOy(IQw$o|kD3s+sD=J@~qHtmm2BSMM1r>Nr4SFGpZw$1*Qm(Q#)tg z7i8ma9xB*JYyc$%RVN&bpT#*1Jwh6C>oczRH*Z|>*6cC<&`^;i^So%^HquN((Ca#2 zMni1~cgdiN9iQ<&xTjm{h^mPY480U01Yx$-ugfIery~ml|t`% z*{!Yne|~utT6159f&NOJZ%#WiB!wlfG1^5fP^G?50y~E61`>Z7C z<&6gKh1EIBjz!Y#B4twDk3MW5(1O9(7-@?PF>tG+oh3dbyXjwrRE!yPTU}T?hvxqf z0tZchj_w$iVJSZ%4d$7CdC8-SrFZ0c{Fp&fbiG?BG}>h2Rh;K^3uDwW*uu^r?GO|L zAEGLP^{mM|54x!lMm0X#l2u7g{bEPC44s{LXKP}D@Au_0fb}afj3r<5ZfZ9Fq8Bx? z;i2{Zfcx!IW}IPSL!+dXWK5M|ETh;?-elXzhSa)$xu>sYza{dz;mSf=lJNo4=!}-~ zv2HU(j7)C=#jdo~y`=K4AQ<56j^vf{S|x(GwZxDNLA7QteJO4J0c72*&5cFDjtk<9 zUG#67{Su`6_No{D5M;DdX}=Ze0z>&Kq4cWqbU;N~qKEu0ZQKs+iVcD45%IYm+HGT#J@m&Ev`+~vM+wB$t!>;nQZM-3a%TOksK+pw`GyG|HR3?s8G)Og)`@d+UMc z6u}-Yk--!C&em>JJ#BaS{@fv#!4b-%8g=gS&6-%P^@LUUDwKSGvjv#+>+~RI zi($gVm4c~Ht`|QUzC2+omeIkGTK<$#>? z>uG>b>Y4KPhVd1u-WObkpBUXZ{4IzNk6q8@>A_ z4Ja^2HFpiyJ*xLYT`thvV>GnO?+e-{l~%(54W3kANRozg0oJrHy%eK?+h3&mv8K(W z-qtE!4U^-^-jm@2?;&%jz5%{gQJ%SaZ~o`vFQ>1gnd6XmE9$qR;s@i-WSWvQU8?g5 z#!NDeodu(@uc=(#m6DkEvAGYttG7%QZAIkUN=84y4Xi1Y$Bj{{l)o{BtVl%mC-j4X8#m4zaZF6F(Snvlp5NHJzws4$b6PTh@lFM zJx7^za!@D)+q_k0fbvt)R(cIEAh9)w$a4+q2TgFxTc2<=-P`x{v9l^Hmjke+MMd*fBD-1dB zt5w{q`CZjm8`4_;Rp79-m!N@?(DK8dO?!$#C>kg!r|c&;9+g*!tf^MyqPMg*=nJtG zBf*E5^AsC(PnYL0nb}jwwo<8RPE>parc$Qa|0O@ ziu>JX7fn}RtVBw9OZp*l)E<{x_7ZC+3p~{19{EVFo>hz>Mx5PO zL;3wY%GxKKXe!sUM~}-OMc<~wH@9Vs@(Nhm44itl3imgs!r&|1)q}97KFLpPlLjW z`V1=qB}C@+?B3{!M*qc}5|akqC9twL!}Gk5beqL)jo016irmsiy8=z!UqL zdTxBA*s#TWBO{rWq^7h#`TplWlbY4hp5!Tg5w7UK(~w6Zg##^OLyUNo@(;(6xHA)PTT-#wloTBjJmd^fybKMgCEIt*$}gxFkdxN zXy+kY%1z5@JTB6jd%y|LMP8yR0b&#p|J2h&`K z!*r%8wJ80gPVD0$*iP+JkSC1abzDbFt9NGNp6hFo7bv3_OFt)Vmc|W4W|>Nfe-zCv z0T4IQ^!md?rG4N};7NKO5+>;=)1u_Q#ynBa))ODyZ>O;f;?id^LgJf4X`RPw-FBY( z)^tBYW;qhwVxR`WO!ta=gYJQ}Xjri~YPeb)7~mRWJIe0a5U~m$I17n)Fu=lf1S8VA zfo$|XR#w`iqWSkOvQ#~p;N(db&)XkX<1DMwvTpL$1JSC+o@Q{|9oy#yKZ6|-fPgkz zkcSUXlPSp7F77;~_oUJ4h)U2b2>4zkK=e6gA(kBQI>v?nw)jq68FIFsBbe+PYD*vQ zNiZj(lfSAcr55P;U>`ZJm&!y95@L>Xe11Q7b+jsA7ADn<_-1Ruk^A`rm!|Tvkd$4Fi_QSfEV4CH82Ak|r9=ius7Z6O&92u=F! z^Vp=-GkVU^hjC|Ky3&4 z`0s1CWni*_YI1$k>PKG)hLS$IB8tJ*huaZ_DAAzrJo5_7)Xo9Zi)hmJol(_lh}Y((>Zx9pkvGdB!IF=3AN-o z625p(fv>}xZCuO9(@ck34dyl|16@1sU=cG0iK?oZ9L`(I00n%>o~${?Gn{Tv*GF)Y z9-lTokd!)GBzXzoGV8SA=c`&Qf}V#jniq9S0%nCsGu$IiXWMpeA!#+GB323 zR)IkbH5tS{wMolb&%u0c%t6|luYH1fw^%^nh6t-6y6ew*Ww#I^x#`!m(n)ursf@U( zm9YdL&oV&;iCDQ`6&z9IABLCc`<#B!TC<`*SHOS*!f8VmgBTE>JpnVelgm?XUT0NM zi8)wNy;)NAt}sg%DMkEV(;@$wgB$f20bZ!oUb%_y-x6i$?>4 z=Jw^}T~VCXd|!ypv_^iuGfAONNz2g?km+43(ZE0N6E(g0l-xo3$P2WMU|t^ndE29) zoH%d7En8a@Jye$OU;p2`WZ-u#9HkHr!7-No7BqD~vJs%d@I*J)@8<|Wo}!G5p;@~8 z3GyB>Y@-)np;fTnX%_Yz6O5Mq>(?zY5ekm8nwTurNFT5wD{{JQO>kIUvhSmdhZw4L zbZU&5SkRK#hC- z#6CvL53heALkswDIG*f8W794BI>nx$MaH=zlr#tjK{1P=Z4l%|4jvMgT^MKk*AgRF zG`PaL4~gyV4}y|B|YwSNCYgv2k4l)Fi-wxbK@E0!>ER$11-(02Ej`N#HneI zTRIz;B(p{>}(h*^H$SVh_) zHq)D9-;-!6_$T&Qo5-MhKEh<&&wwjgX}nXF92NK4E8%b74x3P?-t$`+jmT|t2!*gG zsM)Y1qpdO4x9_iwTohE{uSN9Ku(-}Z-+kbBg;;l=W85+_zFBqeRkOM+3lxJ8LB=vm zZa-dLgAJQp5^-8q9#snot5?7FsL7?6+_Bve2;r8~w# zLPEMjq`SLOR6>xFE@>FLQ;<%PZYf0?r0YKe_`LDG|61yr#ge)2bIv|HuYGOXJ-Kht z*sx1EPr-xeFN_xzJ=%Gtb-53|zYrVPmpjGrctWfP`DYw8n{)Ul?`>!ms@C=zK?G+q zq@y7FY@j!3K_@}jwFD$UMzGW3FTd>a>R{e40_o;%(AUE}cJCzs!80+iQePD&!eAUw z_R!Y<*tHdmq6=NOOzch$>39?SCSH0Ar^i;UWl>aM-=copS*_*nR1ge!ECBu9!!gE* zgBHKtB3zhR8yAU)#t~y^b_cI&b(Fqe@v;kX_if|3J4XWOGP{U!IzZl<9x?y#w+&7!aK(ip-5J3pgfF5M+f{#RBoeSNRP;7vFqX3Or zQZGf`M}6}X_U|_sIB`5agTJ8cg_}&iTT^6qG|$sj-=Z=kz*2Fi&Zt_>(&EzBE5PXB zTSPE4f8h8bBqI%7{)b{xdv!1GB>K%uF94_*vLK|k;_(23DT`MmvWTnEim!Lk$Zr)T zMFiw$V>B5Tu4LD^lo@ic$=+DL`qy)8k#;{~d-)2A*NpAAnBw@vx({_7I{ z)gLr?&XDT1;%azny$%b-^?Cuc^z(FS^{m!z%j=mp+sx8U+QA$=9z3&b|tXSnLAPtAYcDy**&b3S#TOMn5dw!!E& z153}^2cES?wsb(dUZYcUt+D>&Nzu?pZH3>sq}!vRMk{bjb!^jhq%uJQF{UPBi?ob&hF&ZEv7K3fw)iApWFGU<8!ee_Cx#u!alhE0K>fZqp%@l$WUC z18p$L`eb2mdcVbaU9;>n{~9wb=wizS%BipJ`W62}3MhG@>^iMg!}!6*n=cF&w5kN! z9Fe|BUTXTV=MeRYCp~)_@xLa~AV28flg%dW{yv3B^Oz|v!ZGio?#BZ+-Nw5LNb^Bs z-uA_M^fBn>i3V)%wGSQs*Vd&IH1QKHD9a_ixbWbSKW=I|3y%BGGeSq4QF~Cej`#WK z0Xb@b;=FXzEClZA5!CfD${)stinsCeqNwHc*B$E92NHUmmJ{67pgpQr`rj1TKMN#F zL~6?+DRi@oIp(NnRSjmhp=M#dvyk~>;NjH*@h%s*{SrR9?9sC9KV1|sAYs9CN(56v zqTPQHsG2Y7V#9Q6R4;{w^ZWDZLwetkrI$9f`-og=T%^O=*b6WLh8Ez}ODV5w;Qw)x z2N)2Ht~B6o#VmEl)E;qj!jrx8NNl;@=8r_pI00*Kgb@ z>cvX`4JY@K^1!e`mG?-Nc`EfLO8Drj3dZ<*N@wM;>Rz3K*e`45u`f~OW7xD`fz(~u z|J(!l7m!h1?|Lo!UY>K2!eO6IaumzvZn|>zzOzTYOg!$^!4VTneg<0U53dd|g$!{^ z3rA||KBKLt7VL0LVsmBS>D-2B28TK>zUvl^q{duCGG6==k0J6bx^WcqJLta}$aO+}p zNH|BTp^%YYoqKKELomCRAuo> zNQ?oWx@lNvMP*43ln?AflluY)i*PI(%qBjJTb%f|>`%Lp>J@hUQL*{;iUI^&JD^Uu zSc{8*y{{qdczOC+jP*B)+e3@nrv z-@ba}{K&rk6K(~2Joy`$92|%X96r}Ve5GwlD0sp?ON^-{;MZPD)54XpC6y*y7zb9c<>XB166YRb?3E zv-m?SR?!K8o|QMfd-r>m{d-4J97v`j4+#(%>UmA2==?hfUwkd;&^CSy7nG``krQ79 zh~+F>n+xiubklt;FI&NzkH?<3w;IJJTT_AyD~dn4+kyM}Lm- zN{RxD?*un)!!%~$vt-2LoImce6yHoOAmzJ28+NUsqW8>@Lcfw$~LInc@6-A^U zt7^}~K2cZ&#LmuBv$j`Z`d&HW@^_&-W$!-Kmh<+NTLOjRWs7aaHT=RgngBYGZ2KCM zGDaM`3Umyu1>Q9ZtS31w!irj8W|MYkeilsfh09lq;eTjM?oUX4d`&J~PmcU;8QZep zFWYYk6QfN!PCAy_h8SqQhFxvgTQOgB%QQx}(<7}>3gCe0ZwEFJ95ed>k1ym*?#T`71jPm_YJ@Oq7fq;#W-c>L%exD=vVnX-w$(Z2s%?DEexfFS0OV4)|>a+dvG zYt^j~+U8vB29p$MMqogMGlfnM?s&@R9VpcPnoD`MPRs%9xfoEaf97GK`@-KcQy zrK`p>@?(E1RKdg7UZq?7L-#5!5T}v%#}Kh0 z$=&Q4UB^qG4g@-@i^pnHTkL$zTCS2J)_J6vnHkz@rA8sa6k<96%b`CGI@b}9(@N?8 z=~X<{0~5vUK-XUv0P(PqJaa2~bQY6+6i*gEfLY3sx+LeI{)y3(TS{Ylp+=w$75jPv z(+L^$<4r6x%3m8Pu6+L`SN#x`q#(!H-j1YjgYiYx{`8;18*@XMJ_ zMGV#b;A$Jy1pSYnZr&ACLJWq_DRe%W@v=DnzmDifSv^pJ&)KGGdq?uXNPW*AA!TxT zUp@`BC6BjdRbY0&sT-2pJ{Z{cRvgNiPDBj-o2pr&me1tHUniU3<~lzuJnB~ESk}UU zoDJJ+|ljjoVA))4c~ zC~u{V)0h25;jf==8>k0sTPAdKz?)O9zK$Em7qjf032uR9S;ei2@^B+Q9Hg0Y63D1Y z8if!>|BC|(kUzno*Xv4nl=sd7&S4ljDX>o@x?CArIv(w0-@D7g6S&CK1G=85K-Xw? z@ErjKK+j0$=;&CTwtH33{P#->Bf?j!bt=MaD{+DSsQ7FmxrV^XS|uGN(^AP1fTl ziQ^`%v3fsnzcnx|u;i|-de5$GKv1P%?UkhHvpam|-Y#d{jBzUu2|~Pz*HVE4A0OG~ z5mLP|KKLnwR@7i*h|0B`qa?~8NB@1@3arT~^aA^-4JUufz-8t?kU z011>mwj5M<%hZeXe0!2agbWdmp8R1xs6gp{W9Qf_i>n>#d0{}PY%~dt8RsoFtau$L zV`u8LvJDSW1PgLXHJFOvmJ79g7=2|O!sz7ZdAD(TtCTF$%Cj;LePwcFA*v?D4;B76s0@H94fwY7#f-sr8F+N6y@0NkQN-xonWI7O|u1C+2BBl{%-*Mk8}C& zAI$uamY?gs;L_>+Zl;p^@Kauy?_{JvktgBPBY$BTt;*Nm;P)V_b!in_r++~B`1lqK zMz{|tijn1YsM;V`gv4UghP`X$5&O5K&VRcEpq`Bi2`c-j$QQ?G_CUp8H=|+x#*gYq zu?Ef#Wn%TDh5~hk+)qfiyx`nQ6m!c@or&-mT2b1izYx{&r!@KArQ0X?!eig^W~V($ z=QnBkG9?cp=l?fx_-_jLpLG#DM#|%Au<7A+1YT4UnZ@dUC>ev#lz`G`$dT%ye*88O z482lW%qE_oW2%T1m(RM2aagWN6$!vpLB7+^Odb<8YbC{>pJI>W zyagqm8@U*@R@h(xxnqCQ`hn-kzDz6_Nh?w7v_6;#hqK)U>B?*#2>ck9RR$-Ti zM_`tsK(jJ>+IdzR4d)I62ukE)0da-Y$^QEHH=DQ_5(rnq7mDBE*s%cy+Zck>_zT=5 zZ}H+3Z4Yzmq^k135w`trJ5SAS$B*(v6`(nVu}xM)^;VcNYgd?3uh&+5tl{m=FpM{1 ztW^(6kXaSKRd%&~xxP)uj^wY_Z|io&Js8PaBcdpbmYY3!6kS)TAespNCRA%RXa%L} z$>zK%e%Xh)^->-E6lO2-rSnMKaTZ*Rn*`{q&on-n*IJ$K&jtEU^66tzDjtvVeCDWJ zGH`9qqzUwCMAc&4L+(tCoQoTqE4rmzR;L&L1@zsF8KofkkzZKY#INLT1PyyA%Bc7@ z8w1ka%jRd0A*4%4>3#qHwglw^ z#pw3kn70ktqUVw36}MR1ADC$E(@3Jo5pXf6e#FY)Q9)i&{b$0FYlR~_UH%K<`**&V zUlajNq0~5RJq+2k0fuqOB+#}ngHC(FR-|CI5 z#>Q_(@>eZ0*=toYx;#Hx(D5g48>~GTm{a1nklG_`@*rK_dCuF zfiWtLI~h(XMSA!Q`Q<(sdVW^!j6FmBEJr|FrnYF(5}*<6nlb1*-q9y90JjAzsQC)X zh)jS^%MPeONx@W%DUi+K1RW>?{A6#Pw&{3qv%k!dDXf6)dHSdPgNl7z+<;TDL8S`k zuepL(&v?{U9$!ZYz6{VJPLno5iY{`qXx6TppeJ|Q-oMCg;a{RfLd9D?dY*z*(;{5P zg?@M@zFW&X1-B*Q%j~eVRNeNs*{!whSzlhA;Yv@rnXLgKA=j&W=_er4lm5aizz3UH zT3x%Fu)^Gf7LU%#J+Jy{uRISar^}WeVC}G91QpJROquWm#ExJfQcYoq>IuAhnJiY+ z)2%|YA@z2Vne~r0^r&A9RhuyzZnMloHz!;!d|nUxAwVK`z;JR8Z)nHBKceVu-puT{_gPxYA6Np%sSsVw``ePfJ{+_L zSj@fQsARPDn7k!wbTT?E%py02SK7FU*y{4%40$RV}Z1%9=qj4S<%{* z)3=F-r7iQf370DZqeabY$I7jR#y86v28|0@S=jy>2L=q^N?k3M{+ZKgk5gje^H*;t z=_a0<6ZTfvOmyByLLJDFjY>gkZl5h@J38N7uQ6{~Cugw7Gm^SH8d3z9&-fCP=})f~ zuYjJAPgLEK)06o7-jy{bhehYbb<9R@G`%rRU3NAbpoBsL3lYFCL^Fm&snGx-A^@6) zU3%rft?L8;LUQ%u`+t}V-OCcZ<^4m^z6xR+c@t?&>AK5}pDbp@g}A4@Vj2y-E7S#i zesh6^V6MY+hEOiWt3tsO8XqT+%(9M@W~+V@&8kTp{o4xrNv#Qr__xB}^u=pk zgoWW{7L!{0XI>PfZm)oY+7qY1hxRvk9k79rkBoLRkG8M*FP{& zTOd44B}SfXianLQyRolGtYhv3Z%A>+w%U!Kv{=<_#&u>}XvgSA>2GTOkqJCdTMgE_ zAer;b(BN%+xm}Wa zp!qKc7q>`6>#cB(zIn^ym(SdCx`fv43v)AEoPoET~p_^LhY6E-Jp zkubRBd&;kS)5Oa^QN+Hk6?&Y;KY_HHWSl1}7P}9h-M#dSBF$q%u;qK-?A*|-Ka=kD zW%~|NdL4!_+R?{$jeuovgBFqNekqRB_<})G(0PM}kY4sy!n?T+r{6Qmy$fb4dBw7U zE?WTqqx4nUU3M+158s@^SO6S)lFN zEWAD{fLmRByCp)LIWB*Mg3d#ERvrp(HV!@3el3qpnZ3?5p&F3!SNkCv#`qf91cqGEf%!<`fLmzx!H;+{8vq6-GQNvdnf{~w;Ot!J zv%mOZVBFoQx1jU<*RX{RK`8uQ_hX<>#R&?(W>>NC;wBmKwK5xtwb_6xD(89wm}LeL_@(H2=p ziWdhzu5x(~4Rdl$kG}AJmzrJ}T3elYHP_^82Wske-E@e|k-Q5 z&<7qx$qa6%Bb`M!Z^pgI4$qv?XDG(997x#WZr&4CyXSpsyE0Q>i~8+v=}Vj*RPZS} zr~>cY2k`NO?wyZlhhVw42IebuCU}3~EAqoT$A7CM@oEvZgM*|v)ntqJZ zC)oJa{UKPcfcT*;otW7WFbI+u4Uta#*qWspOWRT5G07e*0Y$qmz?#AWM#)PBe!EJy z!yvX)MGh+pm-#+Gtm73vz~&`~U9rYcCi8^+o!1_#8DxEaM9A@@^_i8i>2SkqowV!F z7^GJ6b&6%rUB7U!)E&)9d*DEoRbXH!!Y=$#R4?R^V;pU z^jME-ky}Yw;*(?ypM6j?Oi*@9S;*r00cHQ9fw@g@^`pI7Ol{sZ%^iioyJ#oyr(>1U zFFN1fv;(=yc|+AOK+lracz0@FgZa)H&%=;)F>_u)>g9?DDm@jUU4=Y26(6OsBE=Bn zbjqwhOj}=PqiFENASSmzi0qFIsk;IJNz#Os1(`Il&taOQu=Y$ynB(d{_R66fV0(15 zn`r(Hv0`gVt$_#GbAE-0rBIVDGXZf@r+jv)t%HXLvh3SnGT-)X1taEy=-Xm*| z7t?Zi@9bmA^DIVYOHW#Gop~oiZby$aa|ititVB;A;`OyGJ2tojCHXwHav4@CPH~sT zb+q%ypWjJE%kv)evz)!>x`+=yl_X5{hkksF#O#Zp>I<;E;aoDZ^MYsOT`|r($46@- ziw%#9%(i4>a4J0G`EBp3o>$5Ly6&rFMb3wkalCvo=44}XyZA=s?7f=xv`M-&-7`lX zUnl&@hP&P^R$52tdViC2bUsnAgHszR#4ixqhY&4jb40U|`%ZQ23Li|pxyTi>8F5-q$?lDNDL;<4eIsX!PmxPSM&syv+EXk9gH-mZ zDY|JUtuGVh0Epd&tnW6uK1=DSwoIo;bQ?`Em_Odesa6L6Ur|++9_U%IC;(Br(hTz2 zP|#hFWY#heE#SwCV|jN%xDxx%>-ep%d0KOv7HO zckO9L)$rHn+~RT#=Y*FdBlWVH6-PdH8;X0JHlQTgmBb6heeZelQ@Wx zPJBv_r5o|Jrz!syn^Tz3%O>kG$}C2;W?WYkE9*w+Y7Cw}_t32U6-u#!6PmTqg_4Jj zV|R~J<=(0L0#}H6EP1-tzW2SQnWlFx@I$WyvT2$*f~L724)jR7_Iv_ff`qb7i7u>V zxf14Rn3SKa%ybkZ&ZY3*Cj;D^qCY~i%b$Rrbh&86I9lblopbFW76t!MOhF+$cU;3d zu~6Y130FXg+3>!#@vooT3PmKHU4gcrCQH%2lZLi_|0~+|zCO`JtLpIQli@RQ3#w<@ z#a#3MxHe`O!ZzQyW4UaLHNBkkP4eyyu*KOf4BqvI7@Ol4I=w%7=9s(+!en|q&o4!J z)$rG?#v~Y3P$&6*a@nT%-}e?TmB3B*AbiEs!+mS+(m#~tAjN>_B_WOwLjduV1dvbn zw{dKsn`q7ydIx8|&M)T9z7-ohLzGq%7aaf|9HEC{djzofHx<43E(80q&FvFUgZy~8 z5>pH(m+c0sQP#=+dgUOfGU*?H3(S8YN|a{IV)QeyR(RhU``UBOv>)Fyfj_3QxXY+Z zw$whVMQxDI_a(XwR@KXFS_T;oOv<_{nK|)$?2m^|Kg5-3A(|`z6;0-u19<)r&gENf zEvoHper}r~-xnp#*Y`bWM|jdn8t zP@Ii@__uf%#0BFv^$2C{{*je9=pji@0Fi(d9>VwL5l7lYs>2fgQVZF{sq?nMA?-mm z8!Q2!qi#~;gz)|Ee@&nyrB4O6WdgkkN`c4|kca0SDBPH+OuQ*lk`)uIb6x(*)`6Ie z4m7yFGT5c=LAP*bi^#^FrT*2&o%em5@WE@iPPU%Ot}@HS+irYa z5PxuW&P0T;+Bf^Xgyw{n@XpHmUw?qJJcgO_i8!kDzHc0&8AwKWc`b@X3HgjE_}J(hPS z1fSVe1Zi5$>Kw0?za~tSz1{-}G~_aClb?5rdGUFJdqo6Z{K|(QR*s7M?rA)z%A8r_Ezd`g2W|)4@9N)T zD${%pAK(u*|01~$Vm1rEh)z8RWG)DI`q+m5ZLerj9u9cVWkj)+!`}` zRQX{MPGbHa&?_v)sqe{f5Wwh=YtGg;8oe!woP$jQ#@)D>OMKR6jJfH#yKa`75lRWb z?|KXXcrl@J{~UEJ(u8oCXDD={6_K?B8Ov3z%zq>h?8Ds&k!J&R8-;BXZS0fpWR3rQ ztt0NxI9D$>LRrf9kPf^mY_`EKP?d{dGM=vaOVpb0O-S_&K7v(K+~aQ<2lf$T(d-0d z^jtsY_0{~9EFYLeZQv1Iyu01cbjZrDceDWD^nS@eKh-S3rxE*3KXPuW_ghe}fUVG` z+U6KQYSUe$-*T$=x1>h-4LY!&iv1_URa+5}wxoD2c3sA$iAI_bDf151LmVmZ4MX0v zDyu`r?fVFflG{BNYz!aJc09aw@i4`)_sbVD)zWaQ6moAcEm}cU^(Wf5%U3{#xICNF z{(;p8n>%UAdUW;GutAKJ_1!*XDP`BZyMj`s#4l=BB(uIAU+wV_1i8%J7An||6?PG|P53`Ti;VflW2LU$AS2B%Kl&tavALq-6bCDWvAC90+Ym@eGWFylJSKzZ9YY#;Y(Q6)a{1iC;Y+?q9R`8dpmtc(1W%g5}OH&qOz z!&GbD#2<-RuV7rZj(PyiwW&$0WN8rF3Bc7S4@joXaFeL8x@ZT#`7l`|0(h-{wZ$R*oS*1CxLv#)ZF7sJobl_M_K%MSb@a|v#NgCQb(Nx2GN?90venU<5 zN}U#7?wJW@{D%_uv>JT*`J&0dD&ncv{0$BiE<|9V-pZo1wl z7SX?+iv*`{5pNMFVo1U{S5=H4-NXeSt0RbSY?Sn}C1#lGdf5>A5X`G{yI0p;91n&u z*Mw|~YD#1kXawpobKC5_cjR2y#(f;&xH|i=VkSrN7Y~$p*TYz3FWlpNB`)7}xptrq zo>R6l2FT8GBP!yDJB>6e!|7zESNuE#wgfSrPy-ma{!C|LAZMi^dSjtQk$D;U2e1a7 zd+sx_&=Le*kvx_Ry+0Gl&fUIYl5DE!V{(z?G}&A`6FAgc2ob~QJyO=I2>vU@%923D zZXQ8um{!{!@!|N&&1f^q=?Dqg?4tCO0z>X2QDrms`jA?X&#fgPEDIQZ5CPdRe( zuv}O_U8*_g<#9Aa-u?P(hXM%D%~9gZJ4T<%G2B!Mg1&sNH}ah#D&K*No`p$WCnWsa zZRyd@Kid>tXBP)?r|ZuYUv>ILg}k^q!{rOCideh}C+efIDJ6)cl4lg)=lM-R#SA!k zv=A-)WaUz?L)i+7%iH*Vdm=i_|4-d7_hVBDu4>FHqF(Tnu2WUfJSjB&*It{${fqmX z4KzW^EV#8j>;w#`$bFBa{9YRF0?F>$S)P`l#01J<=)TnbHc2!vGOww+9LwAy2gwfO zO0``}@)guK%mkuV{oPZ(9c|dQ@9T#QuX-`Qi4+zr*{L)~E>htd_eBTt5zDX#(FGAp zZVf|e7`&GPGl&Nk>Ux5Nw*cKC#!may(!(-UAeH1DH^hj3g01 zwno1FlzyN|lhaiTebH@fq^QQQoV^m+sqlY-e%?|0Sk8%`XD4%;DxBK5EC&o)(yFq| zjoj7fc<9Ea#IbWkZ@$>MYSf*d<+rC+>=IIU`r(LkCon^RKIc;jhs;ZiSMh-*TW!PH zbj0rdOv_>s^SW`^OlX7v>c)?_@zo9xSx&(SSV|b{++nk9GokU_|GW1)1SXZP(xw?* z1cVF%Gv1y#u2>bSUKJ`c?v9dz+NCv_gRZN*rrupYl@aFRg37tLN@nMty~_UEU043Bq&NmUOy ztAZO$!W=e8m2?UQE$2aHGru|G$GVx1kJyc+6LX}8!^+=O4Hse&WdpJp#Jy&_5U+Pn zt(R#iBj2;#Uj7f!xD=2mO8LfA9ke?GtTuV${jhfn{g!BHul4}10rpVc>REp7tFIxJ zTNN`LyTYGAWwlSkTZhsYjUfA@?lTng*x<#`Tp{JuM?ddHJoGK6^mRdK5S{+?w2aP* zSqk!z_o8Ss!T&A#^AGh}j@{Jx4RY$vn(24On;*Pd`dGfR6t_26qnKDWo0jp1tS0u) zOnp$GjEPjA!{b9-lqDVb+uq_ZSt0@-$Z~l*ao11Y7`ODv~6zs$M!?$VNX53Uw2jSB89SYU#f zyxZDXXQLAF#hV9wpKmFl>~As^3U7HG-&5;Ep;e>eJc<2)qB!KKe55#9qpl}i^=K#P z@f7g+jc=t_Cz(H?^5w)Ua%Cv(w!tz|e}o7#9)0S050!cX%)S(~gsil4RH%ET$_T4a z9F2wR?*jg=WQZF%-==!?Pt4NxY;R9a0Efh0Zkg+P_-niF?f&k3w1l8ii*HQTt7S1u zE95_y>o@0}MPq8c`JZNfIB9zSLdD%5j^wA8$E#|zsxB(8L3AIeSIj3A>~`Ievw)%S ztvqTHEmlJZgJB^2Tcj^32GBO+;Ff!L5hsbz>*U-*zCi6XR%YC(!^L<-KgaB&@#;et zvm|wx9CBpYaf#^v6>1#cB{!??Y2)Y7xU*Ut0akV|m7mqJjlnGyKUI$u%uwQLAJ^15 z^b2FXwQDP;{>)H{xHyPodcEEMiJ*(s~LnL1hE z7zsX0moCdF=INy5yRLtG&A5vz?d7VybSTx%BSQVMQM?4x#MSpzjHtN7VuP!Xxz_V1vS3C!5 zfqwRQx^K`kFw{5BdHuVNlkjJEv+|pfN_+7Ru$@pQijl%e%3h5C7HP~yg}NKY_ha?Q z`Bjm*e~y1xnoGuc&e+Bx%R4!H-aD<(SO2Wyk1f`-Sc>zdo9e|b5j!}BQ+t{s3o1&{ zpx~Hdi_UMos)XEikY<%|ybY(34?(E*Z-VB-YKqYU{z$sbq!J`ow)Wrt?WFfnzVje7 z@piUJKULzQkm_b!>M6z(1-VU;nvT?q%Ey#0>$g`8yFBAnI-zVv+hu06TC#Pf2RD~K zkb9nZ)^xqSSYJi!oKRjh^o8T(M#8p-izhR8EmaB+6_#brxi{BiyvsC*`_!UCRYx;p zEh>#;${OSDsJZUdBsDsmFh1z4eQ;xS#<;N8FssE)NzqYF1!%4v{ISm>7r*a;`!71u z6AMiv82i)Or^@K3d-SCD^>*$@Bnl^GQC{M>|5n*#;&bKCh`Zd3&3P7`n#RvEWc#z< z0gF|U@khn5t@dL2^UY7k{Rgi$IVLewu%3qNw_7tOx|em5J8u2j*jdw^!ThS3YJ6K+ z0qqyB&BfW|=^s{O*+U7Cka;>mvHWST=b7)^`oZ|4IP0l%R-076qzBP(nSz=3Zru+= zHJRmK6#&OkRr_eS=A#Dwey=W0=xI;-Ow2a8J-N}q%qg#@8+;*r?t_^E6`S%*O zHDj%pdwh{Jss7s_BK~e-ONHZnjpa?FkKy^Bkx`yGVlntE%W5PD#%H@~?}zZ%Szf3N zHWP)7$C&eW0Jr6R4&-B2I`0eG*%|tS=pXcZ(V1@Iw=Pj_yaZc3t=}5ZDMi6?*b!_A z$$$VQq0Atf&)t;6ostFTzjbe9Jd%ndi~82nIo{ek#g0uBt>0Bu%eXD?B}{6vvrddy z@IT9bikznp9Db}@IfIy#4DEu}^`_;6aZ$QmLKkN=@^qsJ?R>P=yR?!Ly&jiU(IcV^D;^VQK4`+8*^4&r1W>Zu5h$y6v9CmS**K%`iVo6;> zO8!mA0a;H|GaqLZcWZ*o_grnPm7=B0Qd^s!az6`ubdf^7RXC`7Jv?biQu?!LMA z=L*liIO$mOnX4<^ov>1&m%a^`%F?c%nl+>WXgQPdhN9{~Iu4R39O0DsI-8%hD{ggr zr!UnjjXZxzWya3%nxlw2NoScqe-_{8Xg9R<_&!l;LxawU$9YD+O6lP`5kehy&~Ks< zc?8X#Yaz;q0!Psz)JvTbGy^aQhYKNh5n88Q9}%q6yDeXT4}VL@_=O{6SXD8a-yaqk z*xSZ~LXPr(jDQ=)T>d@PiB(b|8D$Km;4urch$bFH z5xknt-P_s6N=C0JwKrS=uGF`DnaRd-piU~LJ-PU#(yMKazJZb_)BL$+r&pHcg9Y!C zH_s_9^eN8*xnjCcRih;eQ@67XQf&Uvgu`qfH`Qm13ySWC(C z(TWw&4_}q3HWs-%MFyYs(#}u^1T0<)u_3^ry?KY|m5-ExBi`i|i3AsVdlukco>7i% z6Ea$mI-UM*As~XK9J{BWDn-8lmi50j%wcGjDf{#IR>s8_CkH8J;v46Zc{{BOG4M7JHux7p-JR2ZSMT)Kb$8ueN@a4xJvsg{}Xf&QS{Pgw%MdYEHUv;@uKa3{J zqJXi^2s&Ph!`>OOr^q^!CeDxiu$(V(qUmgws+sO6dlE6%L~M7C2{)urK#mMB?iLJ# zVIVS~7?hk?1PmO23|Z=CSzJjvBs(sNxcmQHJX*?uPDLco?Qsndh=>`-T!@o5Oi?n& z95VZSEI@)p`JVII$XQP8^v<3aG+JGZa3fDPR$pMNJ$$nprlo%J)P^Y1Jp^f=mQppzhX@nKZ@CncmPf2Q5*O!B`j6A9s#DpG28a{a~AFSJQ~hVO>l-O^=6 z%;b>&^hqao0Da@BCDi=_4LQ4zg^9%OIboT_ZDFzFZqVG``sYQl8jZXuL?@3j!9D*E z;oO-7Wdt~9L(ZI7yoLbtr2+M>6Y-5@74iPM|rZU; zSkV9dho#`CWIt8s#Cb21>L^_{l17&1kINPl7-&MnyTF#9^nwCDl%PYRa9XC@zIq=D zw4C)D{=V1iripU%rObFMOs#2v&v^x~ir4|)6x|@qs_;mFMp2=YOJYGFX2JL*O_f5N zBxFQ|Yh}8&^3TfmXR?ATG)HZG4OIN60+M#C1(ld;pFo;d%4tmh##Mu|?-PQi+Pa@% zvYggCbcPqJAG|0}yb3=Y5}u4B>9w{e81p51`nx=C99E30w1{SvOlx)~8I~dL)kuPW z@4=wxEW!0(|K)S$cv5N9X~tL3>}cuNAIt0$eAAL3+ZEEl%YRt@uy=t9iNNF*BE>g^ z!QEwGk`TY6&xhkY6Loo~^r(FSPPlX2IQj0}c0}@N0A>nO_@GE6KXFv~z6(qp zAWagOzJ-Ym^-~;2jVr(Puu2&Z>gikLX4vjVcW?LfS_5AvzT(U*HP3thP_|G@lEBfj z>0Gl*(K{{z{=GIrEK2UD^>GT)ZM5LE=!PWjXPFQae+xC;WVP3-gGm&tk0FLMK$Z;y zy7MJdBp~OQl)s_xq3R)>*{=)Rvf4PW8{uH#&`h~xHqf9E0k~BK3U}z&h{2$f%wSS3 zy7K!bhza#CNK_yY`N8^&^AqG!tHg_MDi~H@S%Athi-T33-+eXkMH7|edEJtw>-oMj zL|^~j*I2tvcVFqpAEq=W9CEP&Xve>UYW|>sO}hgTpNIMbX)(2^{mRur1-FeM#&x;4 zbhzX4)VH|)aTVHifl^T#c_<>A0wIeCgT-PTVmoj)EVFQxeSRf}DzA`4v--EN=ZX%7 zxe58L_v!#F=fJ1lc)l30zbc@~Fll1H!1VjtG&F$-aOU6W1mnL5;%=jmiz6mM50)DD z`?#L%jgm(*Dk63PhVFYNyUyHOU+P0QeOw{zc63vHYr@!Zfr0PNGP&ly*CM~MPC|K-sHZiO79sTXDp!qiGg|)}xU#Y7p@LH9TY+L7 zv@u%uL0eck17P*Tyu|u~sZlb4%JB;Tka>%t^!fgjy`243(I<|wXtJ_0+=w-F^+M}QXs7KcPOx0&-LhUC-afhAwhl%~0`>YLo396V0b<=T8Zyu0R( zv)OMH453>v#(#ov40@YkWmQzODg{lzKb|Za zD03;I#oo0>OH)&oP= z2Xhp+d=`^W7wPAjS1Y*R@)<1K`JHdR`HU_^}M(9y;2c(1(t)JuUq-htMLNLC&VJO znT{41V{mexYjqfVnA+kd-Cdr~#|#;C?9aJ8iicvCJ2t87-<)M^dmzsA2Q($}O!_dR zBtUjh>schNv5|%oB7NlawGR0*RB5^}-V|>7%uvDWq1RaiMeY<}%6@{cgtrAr^9wNl z>N5)c)N1qeN&S7Bv-)fXIzMz0*vB+KbP0l6_+d1Z;2sb=9j%&Yr`BZAulM!z^?3bs z#nH~m%bU?DSP}J<4Tmig9D)|VIy4(FcFQdaRdWmAVcbZ407~`=ihL2$cqkNtCn4Y^ zz5~Kj_Z(SGVJGkNnpv2kTa3wd>NH#GI?ZCEVhA+#8 z^U+PW%@!{df8VSo$-xMDuhhQAw*xgndM|9C<9lfnDVw1Rii(C{6nBYHFWGpNZF*u#Djm;bPDd+6d0J-ukc7+r0ZG}{bAAE4fC{ab!nIC6TFS8$iu4{b$~RqD zm)i6$G%NRKBp7rii#4`>9cLT!c#TN}rCO}8n*J@&>WVG*`U9ZKreAEQ@Q!3u%P*{WJLiD?H7#k#?$UuRbyUGK~u9UF#28T;uj9DZgL3BOQ}gtv5u z!GsC7$opz%Ty6yq$X^^y|+0_wX8F~jcOsIL2K)p!nrH{8}U zQ+q=;{VlHeB*f1jdI4I5=;BruT-xdMXSfd~I9X$K$fj=iGBPr9+RVX%cEP~yKPyQK za$;wTlB#(sGQeya1~rPcbcXQFFl5Gj2W=*w!SD;et&c!aM}v9i4@2*@ZZ6ml7U@5Z zM@DbUPZV}u=<<#_-!MzRu*Xc zvjcRaD=#PU4O&H|8y4S?TNNqm_kF^m-A1qSKu&atSPZ@nT$&y>8J^W{q4a%=9?vEu z$$^i_nSP$7W>X&ZL7=iAgfR3f6A~XU`B4Ep7k&%;JgA5F8?rp=<*Ms1fZ?kFItHbD zwHI#tYor5*D0+Zc4}lVsTpcOM0A>dMU}bv0$d`j&=?JiCT4d=ZAgE9ao*Uoiz@4rX zjb>3-pciQe_&2w~70IS-2(Jbh|8=*B+~s&EPqm-3ql(l!;Vy!?2b0+u;JDs@f4gL{ z+S3yM8$cnWo>6!X8J*fzJiccsvsd{bb{xmF1GIFp`VRuvclFrWkVoO7$siXD9JlLc ztwa}2VW61{Y666Up321u*TEY>v`_c3GrK-Lxi{fs7yr#Pb_=(QC&-)+!r7r)PmQ>5 zW~LLqQ6o+SGN=Bh&Hj@90rLo*vKzZcs3HbVM3(DsJo}W=imhia1^?4@yPTCm$phYk(fS80Ojh1~G zK^}py%A?Y8RFetVMUC>6R?`WsqPy&K2quphGn}t(R_Al~-4bMjXf>$d$s#H9#vxkbEr8%Z`R5|(CA=z3I zlpqAD&gVWO`Xq-&vhd3GcL z+QD7+!oV}DB8rcUBnDZsi7bylk#_Z35W>0P!yrO2m%CHiAID?f1<)7yB0fb21do2O zYE{W2Xr^3umwDn54QNZiTP}_|>H>bUIPf?(ltb_=0NEq{TKg@W@nVC(rQY;|kVhoK zB)5xk@y@N}Kh)>gTqL5gMS26`<_diTA>0mVQqvH0_$9A-mlK^jYt{Cvmt^Eunj-Jr z;;#uL##eM@Fo+j=RlfJ?8Mm?WAB;KX%4z(%wGX+QShlg$E>uo{y88*zj6nF!2OMn8 zXY>OFZ>MtxkZ#o;yrI4V1lUj&x$T*$*9>he1>asVXb&x~bH=>QwkE**EcA|F%nl0L z%`nsNnUZ#PC0RHaYb*1SlUB7dV33I+GY;*|J32k_ej)};%DJzI)mJ3?Tk0tH`@tyc z59+Tn@QF9i4rVL&!x>fK3Bp@VC_7+K6S{CjGb5tG0PwF&);uBMUW@m)@nG-K3uf=( zHW_Hb#1C}W0zB}O0>9dIU3jNm22vghGJ;-VqBZ6+>sit6fijovm$qa_!Xb~JH+A@n zgiwAE-g(RO><_kFiLmPq`lHP+hx*?fIfx^0-_lAESkwN@P&d*cW`vg#b zU^$9}-qv;qFU8o!c%(NgPUb_fsA5NUB}-HYF&xSH?&*4d$b|djy#?8Q4p%nBQAp*-@#c{l-gC#nDK}pH~A=eosFA-4$0CHo(vz4#tKBmX1XPJ|n7= z1(T>{yD1f`?F`K?5fiK71R|Btfw0Oz`W-^Y!2 zva+*hR`$%wiWIV7@7H}__jR4ud7c+&cO31-AagVr(aV8Ny9R0V_9@|QDe<_qtSq)|nEk!j zzm>Vu#HjpY#JjNT>UmEd7DJXhC^al=?H-Fy_`aRz519@{XD}8S`{WcW3*7{-&x_*I zyn7PW+<&G1+?dpgi3Clc-yl1l#SyIa4{Ul$CQpp>WXr^<{@xCzKlRsT|@kvI$$31`llc7lWaN_jV~~bKgNoTg>BQ(t10*8IGN@P z*VBx$z5{C}2tB#@XYlpvd5&$69Dhz8YA?MR$ps5SNv&}ExM6D?c<>K)R_B<;IeRly zi4iKYC=6aMF_M`}Lw}}NiVVJ00Cx&osqK3|Ytr@KJDn}66lK^~yKPsV zG*F5gwqAj|@rGZ8nTPalFcnEtJ$sw6)>!uqa*HRaIKN)2ycx+Hkk6_)g{`|%GrI1C z5;kThoeO>w%Sm9=w}WpGqGhcy8C(Z^A!tF1{dXjjwo$j~{tH+bICyu?TueGa41&N5sS|y_tD-9}AFCw?FI{ zUA(+!$CnkVT&Br9wT^)&D5MW&cTS{;+QvwO5brjLo!9yg|C}aY__(*KVDf{5Yk&Xl zH@)RK65K_~`*9~PHsYt?+~Br>P+qF{MLu=T*eaj1+sx$ zD;btB5vJoUbpw|<$TN;=ag1UZ!Bk{Gb(tqo5sWf+UggYjeJ$a1j`yoKltC_hBm^ZQ zO%*0RaI{rDb$k?=&-KE}uLIJfdMdX|lsvkf$?0jVn<78p*5cJ3UUif04-7pU4F&qi zn4FrwnkP|PsL~NhlrgDk>{;p~YNt}>T`I!n=SZnRtatMvzqqj4&r&+FwgnUk&eu)B zGh5Q|>lA8c$PNgHQA7s?i5nJ@bd2}w|ulMdWo)b<@z}T_OX>fQwZb+3T zlH)K~{#rP0PT126Wsh;cMSA06mXe7> z)BVSwf?mPjJ{q@ThW>Y&JCTeA_nIt?M^d_1#Di}VSnM`j@LrPX z1yjzbnOl(14jQd6NqcGmqWAUVQNj~5Ba*mgm{ViV3Pe`S*AipZN`^eot1BmMLh4Jv7 z5@R|0(X`L6ZW7eoO!Zkq{U~63T>TX0qI_~E!Rg`>#7F{wNl#=kIigQ`b9wZ}D}kL& z7Ez}b6yq7l5H}VqU^7&f(@QQzv~<-JJJVaGhNPP0mO6eCGw1~Ze_Mie2wV^A=N$wq z)D9zo0;!Y!?1~bpnZ%?L2Zb5q+l{KLdnx)TlNyx{^lQ^?x|G0mFK)c30P#T0%NNfz zW@Z(ve-V(VDQSq2U^VdezJ8W5uJ7 z+-1(WGHS|hXNA#wu@regu4Mrt-lvoJ*(+dWk%*Kc9_(D#vl8W>CzMHo!sRq2zfC0i7b^=N`o}Cl+Uxqq)N$#2!27x3W=S7Tvfq zOW5!uREGE2YOB`=_2;CxULwVdUSS{$$y4$S_Rm^(8O>W?V!`eiB-Mm;bA@CeUwf4M z+rD;V?=X*$db}VikSbw|AHt9Cs9eOa@H3HyexJF8?&b$ijY_K2?uN{xY3Fj#Xd&b8&6D!-oQ=@(L+6g>Pt9*0u=7yj1IarxjMdKfTuK$lj;6g5UA%sW2ql zE#&=h8_?^!y$>fv#o~GNG*Wk_eD}6zS+3!K85hd@NMk3M)pg?CRwxrmj;!Z4xLi(+ zo!Jp7|0IK7cfL|h3u}vwcAQFDcsADE#m(3$#JyWZ?J(*R!VM|5V=l0q{>CoPpX*}_ z)B0WgcXQbIWt;=8oX?oHW@iFih08TnmZHaiCGLScwx6!pE0n6TKrhhYEA?ZXO9)AH zsb8_s|Dqo>vtquGQA=&XG+u{fBuy_~Hd8ARm&Y5lNd_(NZL%tlSemEdu5Z8BOSh$s)j2Dp{IrW@fl7?VHtf zp)@vJ=J_Ta@JPzzoS(1#WnGH~$={KTjT*Danya;}`{nsk(swDU>Y2oS3XjCY<*bH! zG0BruO|D2i-P4!q8b^T526D82;+DV7NwWtEB<=Sc=sqE+-&PsMdNfHwu+Og@R2pixBFFvg+C)8`J;V%QWsDTgP_n03 z`^4wwBvi4;GB8hK-r&xKc}?}ad{4HFm;XJR z?nx+o6Jppe423+ictck8W1(<*Hr>B6ey4+ZiSD5>g4+0ezi!YA@)+e(;~1TSOZg{| z^0}DWNU)@KGr71k2|6-2Yov>+-R4gSw!BW3{`Ns<4a)c(hCV<$nrT!^oyHq)`%H2O z=1K0KzEb6qZJ?9D&VgAocUwct?$xTuhym;AeW}!k70|x_#XBHN#^@6$GAGi1x@OG% z(LP-^xrNzr2^KKo`op&tSWOBq_wJGmm2<%!`hn+5)|ii*II&MKLLz0}J8%NRi^FH) zh!7;a0P+m_^`&nvb(-n2J=N~^JYoG%AWi|;B7F0!A#@8e4E=w8Lkws?`L4RuLn2ct zEaD1-6jJmheYZ-{D_ENJ!TrKVeAYqi>k2be5^-zXnY*;yVblh>D=`xY187aFX~nNE zamhMQ1apR;6&xSkzRMbb@zbn8rnqr2*5A<9;XDm}_Fw zeAiSAb^tlrsSPQ1juUB(WO||SIgoTOZDaIAJB-P)EZga?-jAl-EJfPq*qY+BQp|n0Nl1{T8K=lD^Yv!Nd zjFjZhGWC@o5DT^k<3y`u7`EXU$@aq4=?i4`*T$5lI81OSV^cD&BZJKCyK`Y&+Zz~B zJtO%ItheI9EvPe=I1Vi>;?! zQwpsZlSxk&%%FzI4E3a>>7%-q1>{gCQJ@j^JsS_6sm9dUJOr2UxrQ!ZCn}lrZ}^`Yn$HZ zU%`G}kj37nV-|}pWe%C%fAES~%7h#elPFxm%f;9~{bP;1SXjdATSPHlkIGEmMbbBB zE4-=p_zNUy3n@41qG4>Jbe{8x&f`W;np{u%Y@$2JqWS>UWrUX1>Fo?b~B(l{FN+i}b{ta8a~N&urcHBj3+ zc)ehbGlsrL(AP{@2BcD%AFisHwwzjSejreH1?I9@Jiqx>V0yX6roSQM6j<$cP*dJj z0(J$TBHJEE(6&K6pd3XczU-=TcQ>DT)^>7^`C^9bep0ngQvCE|_RRJ|z8uO}_n-aL zhih`HQR>dOb{3J#d;4+K#W%OgF{eFE^B1H03S#1@*)f(G{l{}k?~ON8Gn@`p!n&tw zQky)mff1XDc@iVTv*g?ryxa`J`!Af{u_@AEX*3{Sl@?(bifGF)M@AF z9RS`I5Y%+<3e*g=5;#&@n~D^QE2k)NSrE4vkVj$eI3B$&;Z3)>cMaB_ zyuo5kBfKlr3T+aph>1dEblaD!(Yzf+URl}MZ@^KLH?vuwe)j~Vu!&G*8+_-2v``xW z;Eq>E&(A&YH%hranH0;U&^oboaQ)X-xmiXgfEeB6YrR2w`Lyo&)izJ{R2$EUO>|7eAqmm0`z-h25S~^g+^Z3lQeOJo163|!^3z;Vl7j|5UAYf;JF3CU5 z3#G_i=|Bqid)2C6PrmDY~b+X4y(eB2k z9nJm(nPl~Hoc?Rk!$*k2=wm37yAoQ;C^=;v-&%bql6vFnjNzo4QU9b3ZQ!K@GEP*; zt1#Cr?)4u{PPVPkdkt(O*=o-Iq+j5WazXL|Sr~v4cpi$942O&y1n7UsnPKdgB+%3; zxyo z0LwmT57%sT2-a)}8%HJCE~Ts_^vuRL^0cw$Mm#F3Yuwu~Cb0PI+fU_^R{2li;|4sX-FwpGPv^yfWV4~@ z))q@r{xN&jZVN2(wrI5LBNb2!l{|nVav)#_8

7s@iFTfEgcau}RX?44j%PIV^GV%M zNJ4d1#-+M~hnMpFqvP@~E62DjJaxqlkK^(b zIe9o+<-R4_wGl}nOfN)-=3Q<@$R;yMJ+yVlXV<_b)SE2I+%QQEOKf5bERr81KlSES zd?WVV7$w|y7;#G1MRR}8p7}H zpML154odWT=hf7IA;NGl`$P7fESo#N3D#Qg*S6WVC!}L|jEUYn5_lukBcFBhm!S&t z8C*0v)@3&=RVvMnA#zEUYe$TU%Jaw4CpSZnB55{j?b6|`F<_pTcLS)oe2v1PM++9jjRWik#C5h$oS6x_`GD|=P9M`x-Gz8F5} ziXYlPmJ|LnjJmyw?}>NAYbHzzVz?&+Z_iw+R?0uWxzymw@oQURk^f3x-w#$e|J=X$ z$?zfcgp?hyy7rBo1>&3QM&Bnt>KMJq?(XPI`o;Im%@-e$%fnt~nlTVM&D0_E?knz^1X-nrym-1BkvcuPgNs`|YLRLPzv-R`e9Xv=9$ zC}T&(QX)MX7IHZG78cS6Fm@o`mLSHw44H{KKKI>2F94LV(!Or%4Oi-$oudm@Nn$v* ztRxWwRC!g}!6AjY6qbj1{{b%riC$=wSRsc6FUf|Hdf;*fv+(ZB(g zfY75qF#=h`aeeqLh1)jhr1__ET!)dLWeV&uI(el^Nwjk|vFM{Lhd z9OWha(*&}m52_-U#@-)Vh5kmJAmB4#4PC*WJl69g@BH0ReCsONqvt)b&Nww!9giEy zeHO1B5}mL|s7{ZennkSb1K}C;FZfE+=6p<<(R7!mqYGQlvv${VXv(2=f%_G-r?+Ar)B(7eINcM-EtyVR=*G%x%W_@ zq$`O#RP&MB;-1Xqts-sdSz7_-TWAH5dYIY%0 z8iiD#kan+GyyPco#Cp68Jp8EOYYlL5)d!sVV!V8m>94$ zWs0zGZwx|`N8g=tSK^?G032{z|={l^9i-^ct2q-leT=frmfloObo%-$2E$x-cb9gFo(+ zigVw;A}JfmBx2eqMMQ}A3j_{ z5IuTmW$x|m+(-h@onFpwxNi?gb%<1?9fT*@4*DS%${S4KL^4z2KEz@E+0eIWXCj^) z)YR1U;)M*e*T*#Le_eUVhZ$hklF&3--2O6C64F*vOxGDWQY5^8d1&32rLiF?=CPBL zC`QuBgOcm67Z%`qiNcocD4DLfI0`T5|7YD8t*dm^NR#G5xKn3>a0c6x?(qjU1RW*# zdWUj}=`7rvD=+u#Cp8UTPqD-jG#>T+ye&)XuXggV!T8TRkS8QZxB7TSyz<*PT5i^@ z;hOdNvQ=6S-Yh|r73v9QTXW~{N=Lqtv@M<|u?Efr0&BJbn$Eeh_g>hdjl>J))v`%S z>Qk*!TVgRtx!qkm?j?y}dPCT_I2miIN0T%8jY~^?T^H+7#!jLRxv*0^d$F6!j+I0@U?uD!zb5m~%j4U!UI!sJu$j#XTFoi|P%PbO(Ae1}{WnUlzUvyYI)#Ro>sbPlD~;Gh3aKi{|Ofzl3(p)R?>HT4T|*s*4ZJ z7e^;J^7UeT5|h*(9#qMP=?Cmv)__?l=~2n7x^~uG@wB7QNlz+GAtQEU_Ana8Sjb_Fnb;?o-A5OA91D=a%W0BES(0@2CBNZGU@TG?Qcdz3*VO<8 zPCi-Gc7jI7iUY{L>|;-}E#Ab$$n4x%zS7gvGX-dV!Vv<>HsCBzuZ%Y#RH8+{XLcV| z?fRbTxq`8BJ^FC7k@pO0nQhCQzW~A&IKK7i$Y(nI5-RtJ&`i*1( z?(0g~X~){}XS!IIpDjErC7)^j>yqXnL%WW2r}VuU*WvuOK~T_4r^<8*-#k)m%S4m; zk?xzZsm9!jhUeBizxEvDt_D~udCscv3Lhyw-G;s#(%Z_00jQYaa&u##UlYDQEMfb( z`njdNq>tDNZTKqa#MHgNEldLe?dVJEZbmQQdGg(0^{hehfKzJX!pFi6pm0z2Zw2iE zxr!5MaiIfX6mgh|0IAz)&uZZI1ft&`Wn;F9 z)Ajw{9Bd?k2tyD8s-sBtcnDpV$r;}(+v91gnX3#w83-A-`b4RmMoWYd}enfZ5?$XIz zr6N9w=Qi&f@vl)WDUmdSr|NY8O$#zKR48=Ms3(M*g=3h>`b@1oxbUXw8?DdUVNw3i z7KXt*M3~ac<5k@(xOfp%XJ~}6-eXKWhO`ms@)3>eZD*w6Ce$?QJgE4$BK-j zyHRuTT__jO)f2YYX9*gEOqWv;2FWJ*)iSd?$o;OJrA%5;g&0EvRpvTRFU^v%E5tN? zpZ+;v>UaeBwbx|y{Eq%K4V3miKY1YERauhsI64QDBBr%g(R$v{+g}4ae%zSrxzblP*f7PlDBmud> zT#ti&g}cqAR`F_J>}-P2>r^g6A0WSGdWEE{g7jRmR_?H9e(RQS!ieA#CxtA~`6k_6FZ2vt3OODqw%CCr#^Y=q; zZbRqS$J4+ge(rr{Pxb{Wd%tQT!?FBW9zFhRQ<9C!cLcBJ%xVH{tGEh{F?FG|^&Ln@ ztqPwSR+R@3gxOs0$-Bt94Sg`_%Rr`mD!n;io>r5DmX)PWp`>;$osH5-{o*D=XgTV; z9v+_lw29Q6;k3syuk#I=`1HdPM(hdA0~OZP*7M@>siVanW78r5&--l z%k3Bo{AsVY%uutt4|<&2!h0C_VIsg4J{h%o^b6b*0y5qv;dRUel)NvagtP@IlX}W6 zFVwd%J>%2L2$;?`PYW%zDby z5u6vq6yTw5SDQ>)cLBUUc7JzPuF{NpdZz1$^pYdu!D@(KF_q|8}HwceE>E8Z^n z(pO2uy5gYAebvcNum{?tRbQ(Y9n_%#Zrfpf4u{e3a=|4$&v{@vcia1O^q#cwh{`MK zloV!=L$y@r+{8U{6q|%%lim@d=lta<0~?IB zw;`u~e+S;VF^Fy;J>#_PD@x)!n^X_~)z>*B>^2&^*1LDr-gKJ;F2;P8`!0OKsev-{ zN?~J~XQz&tL9E)eL)iKD{wJTA(u&2sC>?r&Sw_J6;JtVoQs{T2E-lIAb{ElkqP z3PHkq8B9z68MyM2B|yeFCZ_c3HYK$;5u>z?i1)){WCVA=p=-Da6SHUIR_yi9)!T)e z2I4T?rT$xWIdGzdJ(jLKiib0x46IU%iROC~!y+BMbG}%n?=to*TEb{5?Sj=hQmB}s zQ38%b1LCaGw|+dcJlk>i4DVK<&x>GA-IJ~ut>JXy-crCsN#}IrAd8Ndl*^fXfPVD6 z)^Dq|A0=`1?^qOtJ0@AWFVOqeH91es7W&0z%S_Y(8+&qss58D_kDSGRCn>-C+b)<{ z1DDy5wnpwYspos3vTGv|_>P~*_trqn$t3HvXvi}RS`tit5ab+_zR>sd8V@9zu{TG& zI3MaD^gyF8p!TK8zmy*{3~ZvFx?4US_FVpZyGCEIj-#n(hJD&-#Q3$C@&pWDmcC}# zA+#L?v6sfVpI4sSeT;evGY8Jg z|C|=6o*_ZTUX|Y3AsX=E4UEoB>Jf-*#K^krgSjobmsh!gfra{Q03mNi7SMp&S&{eI zT>4cTBpP|Y5z`MAGXN%3u$keRxr!>++4lR#d3a06JQHlxHLpB_1{I?4$_9t@%Cd%l zZy@S5NQQDb-?+Zc68H8^66uGC7Q=oy;f&`?$Rj=#Kt@UZZvBl%<96p@#pWo5L8Yf( z*vnIqHZ+9VkCiJOeq>UQTC;QWjEY#P^ODw4*5@R3|RsRrZh1v4zuAj zEmVp*+#gNHOmOjSP*09VJwZwsxM-Apu_RrlN(p%d3pN(}yG2Z?ufrs(IZNs_QHgYA z6_scxe)Q911T)YEY^iHN(ZUa-kf_F!P8fTGeTSt`J`f8Davi7VB2Ise0`bPnM;trf zm%nqTgNK2<2zSF*ow)alFCQ_fZi&BG|J+U%AOFcvpK=G4d>Qv`2j8Z3GwI)N1Eq-3 z9dv8ku6KSWWEIv_~zFXz2uzDLK z<-Tym=q!kbh;4S?q{+M*cI#E|0j-?>XYY?ccSiRUAhcbBBTwR#&gx04o-C^JYp|i# z&t5-GPdVP8V$Q@Ak=9NhRi@+K!rLhtc}vzEuNYb*(@q1N*1m(N+YpWjoc z!TLM!c9NpknbQc4H89E_j{~HQ7*MXv$bzQ!xDcp~`K!!vFZBvkG^sDm`GikP*$C`~`k{ekY z-Uc&*U+m{FOdo>n7|4JOM$}<%menM6dZ?&uIhrIw?G{3@*5$W($1qdDj*Uo_1Sih>&c)MCV*$gUJTq!xq7FtTBEam;dbVoZ zb9#_1C~KPjBuT`&9dwEoG{M7-CAB{{KLKiV9LcC<))nP6&fwMHWf!=5})5PS3&q1$jQlv7cckwVQv7}FcWa_!jI^q=iB zBpYIm#d|4DrF%*7whSj5D+$2tzN}u*vPRZtj_k@yr^mbom7N~%q1XN)p`XV371Sw< z1sM@Mx<$AH*I{gqcA1${q%72YnBDJ>jnRC_)nt0~JOVVBlPnS_ zQzTf{(2=8_QncdYT@=rKIR^4{oY6D%&)5njUaII@+_j7)u#Qf0N&frdzm76YBY2SK zk+XV~WmL_5v89~4O1CuX074=QN3Tw{PJ>y;w`5(Q+|{&?DS_7mqk&`?V?#UOGnSnX zg)mbf)Lt_&e1BiYPa*n=(gW`=*DgIdi)TOu7k=gRIHo9&T@3UyI^m8C*6XBJp~k_IFNn}dVVN2&_iTl1~kH6Yg1)4E}i$~_IG?RE2Ga-4$Yh%byaHX{PI8nlU1TOyV8=@Q_^X}>J} zoH+*4-Y65r9zOl@v)}~pxY!$>RD34Ys*PLYhT-jFujH?THf|}wMe5&cO+tp5INRu3 zfStQ~)XNdICcHOx+u0W5Dv+)>7Cv_5Jg(twYCiBdxTlWMLKW;oC+quC`sYS3A{REW z5tZ2yANAECQGKg!LN7cGw2kM4DppA%$4VX}0z;rJ&lbf1&R0^7FsDfzWxO{?e3wX+ z1zSOQ+|Wn3?xqToMCrY)$;ye>52!B8c0HPg3IYgYR$qs!5E<=2m=cH#VQf>iL<+~* zv?8(zv6i1~hIJeFcO;$va^^w^u1;S3qA~$1Cpv6*oo`hghhD&#kWxK+UCv0|tyk)0 zOxWzscq?g~Uu&m3&0>+`!v?u3=5x5m5?Z?hh$H1=mu{Z0pn|E!)v=GRFdv`; z03Uu3uT&4X2D8}01jruX&?HH^=HyE2JvCf|!0x?fg~c-bw0eQNq75HAw4BLr>kJ@A z)A>!+cYa11`K~G-mY9p-zwvxl0f$j!A^Dh0 zXA8^2@81g4D(ygO<8?DIz!D2{glm+_2s)>=K$_4M?(+kITOGVxV>gMA(++ zD?nd}db(2B!er$KOxc=#wOm)RIo`l{?mAq0g(hd$H~~DV*rakHAP-&l_&RL+w)O2+ zkWBOlNX6y{D{|{Eu&FWVvv(Fxo&7t!C^KKgl?ay`!d7Ogol(J#q9UU_NE)y(sd?e- z^MD398m_H3paZe+sspG&JV9K&)HPU-u*RZPKy!0xFHzld9%-{!bv`up`Z7ifFi--A zdNQiU5QuHX>m=uh#usnLBdrdExFvXLd#IyrsqpMQGMF1Fy*;|W$EkFmLa5NNcOWwl zio`VtVXEK1e!ljJ&5lK%ec{n`&3}7temwNMNh{QvB#n4Kvm>9vdEldw3KQM;bglfH zUpDXl48`)1#IH4O@8?cO{oXf$ijz7$158f<1FWZXo5lF#3dMk)U;)b~WI>c@a>}{gw_C$ zl!aMC<`siPFLe#T)1P0H9Kq6PiWFHs z70f#hUjE;&t6T`0?Uk(Bnt0&2BauP0!?FuWr0y1-!_qY^ZnICgjulO7FY)r+LYn#BIJL2;L~_5$-|fFYyBy3oAu!_2RJh9xa7Q*r@?Em7l1P9p?!yCZoEK|kcf%Sy(rqB+^ zVivAB*G+-Y>cz*OTa%I6FRouRNJ?rLAx1ifz7WxG|FrZ)u;Sv{C?vsslPHj0VpZv~ z^avFclK>_rzoeEvr6$;`2ylZn8A2JRLl;nr=?SBU-Qvf7oD2%!+B9t(9=3pacYTax za0biWz77u--_z4S*Rt?2i5r)*-)Yn}Ogwc`WR~fAgw^O@ZKXeoS7cv=2NXV@-Thd* zQ0{;$;h76QW#^2%rjG-o_f`kU3)qoc%rfe|WwaE8FnwqMupvZa<4NMD_0E&$U}8E< zKODLQI=a}eEDaXSE$B#g?ELQZqVEfuD3X0mRzP>d6ZgQtRmZAWXcmu?1K28mj zl7e^aab?N1);N0WKLUH+Ga7qK;v^qk{CD>G=c|GavI1Ikg}M0S?CbI@OkC40IMK`( z5_nhfTJ!^{X1Zl}iU6W)(LY|fn|CpD3byA9h1xqk^Kb`bL4E}I$AFeb#Zcq=Y<_N9 z3=qM^rd*vHp_pJ>aT2Z#!E#5HWWpJKi|gWJr_G%Uy?1aQ&BVbK-yqy9(E*UL0QD%yZLfdXhb&avzT^qO z*#>FiRy*K`W2hQ;($ou;G`-h@PUSpvO6mh5>Y<5mgpO5ocf;HfRA?K#vrj?z$Z!hsxa)vH{b=y3NF=FqB zIm2IN?4P?!eu>nng3y+LtR@Jw0*xl9xA|@lv=LoTYixuYmJ#Pbe>26BT#%sEb?ri3 zzJ9?Qkjqhd8Y7-6Uql0Q+o8c^e@D9a*xGFO4KGZ%881)`MhsNsB5_|h>^qFNEsam2 zq1Gm8aopegnKl6)HTath)i9z;+On$|l8>qVz2$v}*T=apkx=Wo%rx_@QBuct$^^v3 z#GDOvc?A_4fT$9b;0>~9vGm=wS6$pIJ`*F=guWx;?H5hesYtoKDF40n^vY5X!AivC zLr_orP1}|C#?M~n%%+`p^`)7f^SB{=HDrHNEl`i_kk<%$^4aBFa)Dj(!Y5-je{bk6 z^0rU=lAI4X3dUY)FO^_*$Lpx-#6jdAs5r9VHJHqff zAkOu$;=1D~Doad*_}~SNze!vYn3o`D5p?C<26hog#>a-Gy*tIMuG0CFcoU7B6g*o+ z#M#H?oV^m!CO)(k;*}rp-&rWo+To(bm?s0G7B_ zbhq0{^S2I9WjUSePKdNp;kU}KoBXE9{P7_FW5MJd$l79XG9U+6L4*KxtV$0kpFan+ zL;zoUG+bJ)4!NwCuAVe+dc$W>`RtqXOef7JS-DEp^EnV1wO@HSp0{1m4-O+xa;j~- zL-0vMzuvDFx0*AE3w%Ik&Oyycqb8E>m^wb|D<*A>xlA5>Tf`Ee zRjWR0se!TZ~DbtaNOQm_~QOA7Z#lr~F|j~HMyvN(4%NabV{^F5=_c;IbipwC}*?;D1^EW2$ntlm=a7PPJGZ&xV|hp zh5FP0oG1r%L?P8-ElR#_1;Kw;(L!d*WH3vw_C41Kk;E zngEmCZ73=nCW+&atPC>FPstyP4f+$hZ8-X>T7GN^x2Mnk-=V`o7I=2TtBe-iY1bg3 zv7l>!i$h_v0K9S@ARG)6<-oo|Jxw^omW$%rN#JM{`r+Hw>LfzCsB{H}#{d6y_9X^w z@O3&&=>V|3b>Jb0Lzp=hfFjF!RaDRtW2*^}B5CY6!@wezd zx4UehFY$ZLs3LsOvj172KTh30zX|$*WAYbGmLZLXzd;9|lGldrHHhK(1QSrKa0kY;v}o-XXv@^1^G-VfuC;8R3_)U=J4jM}~0;|n)bP0!;`i{Mh-?3VdH zaK9stixo+79WA4~YBqW<2!nF7K9NSo``O0a`=&mekP?W9gYAv+_c;2smL+GX->%+y zz_s&V3iLnY9ut-*#+_}WhAmLHav78$`Ed>SbO^gITxv*@JW9;??4ul@P}=o=w+seh zk}@ZZw~tQu`=J6>VPEo3L$?dPI0Kh!B5-UBSVr9}q0eg37R6%3s8lVh-3pWzTxl;b zk>i5We$7(=NznkGu#T78o{9x$5t>gVV8fhrH{#S)!(P{J%oA59E6)5ETIioE$$WvN zEt=hmW?DrI>YJ7bhQ+FdKY!nljy0ihAz$)epGmnnd~TwqW8oRc1Z|r#60e zW&JOZ^Uq3H9;@sOb4rJx7S3xrZ=0CMxHL!K=5Htz3CAOrTR#$^x5wEDI^N<^$JPjj zip^ID@$yo^C6zH5Xw@^yaD$ma9Q0q`08&(RD`0Fs=;s~PM1H=x*#v{@(IBAvvB**6 z4jhU?r0bL>TlW1e!hPKaQCtVy)?6Kl0*Mf!@n#Ic=*zwc8R%t#Ir1dsKV&R)=!2KTb{CZPePPf0yJU z%*R}vNYDt=9gHr*Y^iG&OvM8sK~#e<0y8#;3f=;S_wn_HA?3m_}>jA5-wB9 zf^P?atNH6(kVRnhqk~^%r;_}8OThh38(=o0>%`0!<&L*9t=CukJyjCu+;_I$r_IBJiqZ=ngO-`4c4Qq0m!Ne@jt7l8E>uvE`L(!n#D^rb;*~ zfKeSukEDaz0X)8{fnm?oC;$7=|5z7zjI+1pBb43RWyxdZGtY%v0Kj)oQnHJ?7Y}+U zieOcfJU=;xad0CsUqzK}{ugSgGP5>n4wHhb9B=4GrNdzJ2h<~u+;}{c-`04j+DmSM zLxd{ne1N_2OxbMwzw;Sp9z-$8Gi=!vjhYBRLov?*Ew+0>(Jprn)-B2z>&p87`oqD% zf4=NWp&iS2T?dt3g`*A;Q1=gmi}^yfmLm$3&bPm!2N&t zZjN^^Zj45CkK-LY7l|h7W#8-G&l6YvS|mSYH`afTVC4)vdm<`_vjQ9xk>0O+!uf+( zuY3~g%4BID&4fO7xGr6A;^3M-ElN$q22hsRUO8zT_N*UzsyAlr*Ra83tz6LNI22iX3) z6185k%F-?h{M`cxl;Ch$=%g9+TDrTWU2QObPb4>6In7sfUA5}A($vImAFBp^ca-8k z>WsU3um-OeA5FegVJ^cRo9>s*R*aL)&lQ8*i$>HY3SqBR-#vUQcTZzo=_t*SM2%|_ zpPXKHK1%&-Q2^OqlJLG#XYfiJlop0ojz*kZew%{D+F1`pxeP9T&AY6ZtXD)y+&fAA z&l-{iV=zgOPn`pkOOjcX3C3+RreIxFfy;I(-Dxt}V9tu9L)V&F4|Z_#u|T*+@%n^A zi*&b{;)LldunB*?Q<6m#QMwe@tyzuO@BZnW`PRL+X}?xu`y7#S8C-Tzy4SOIwSq3| z@-*$4M;!-bF=Piskz)DIb>hmTAxDiL%hTm{+QlFDdstQZAXxosM?kXWP8xZ*m8ejq z9;U~du3EKSjQ?6fVV`ESA7*#3FU{i7!SxAmESY}lja-x91ZA-+|Am{5;5!haT9P#S zCh6HFYj?w0_*^^xiTZROG~I?L!2{g&f`jYu$tOu)p{Ip>FLQDdl$T-Oz>p&SZ< zmLCg}H-?_63_Y`Dh15F%YVSp}f4I+9c%U_SclA|ih~yMs{ODtf`lhn4^QW}f4_cgM zswM+5#@VfL&a$D2r@2Do1TLLbozYC7R~fYIZNLWNJf`K8^Zw}WzTYEuf1Z*I0jXiW z!3y@47gZ|}1!7yr=rX;&%kY0wO?VzJbU1gGjYI~vi&EsN60nUKO9}AfZqG>`vmEGF zZ9Q3gIiDRe`B+GX-he9>`;C#B>WBYq97!xrjnvXU-lA(n4Jl2MsIAe}-6J zVGB5gZ-2axH#EbfRJ*n-$v!QU>mXE=DVS{LxIb-4zwjVFiso8%$DBcRoBBsPFB*Tr z|BtS(4ybb3-UdVj0Ywoh1*D}Lr9lPh-ZV&eNF%KxQqtYsAl;#aun~~hbVxTy!=}F3 zp!avK=YIcjj_|%S@64=O>silwmWT5}GyU>0HKU<&6@G@;JM;Q1PJ>RA*y^#hcgV$# z|0wnH-b366{DaOwmPx1@qxo7#?g(*eRe#%J+bXI0E>GoW{IMNE1g#Q zAO(6Zg2IC!Ze#~uil%^zr$Y)wHT*MI&3pG2qJsp50@JaR8|H~J`q;l6xmGf7fwHWQ z#b-AxDC*W{6?Qf(Z@*0CTfvk(8`|HDb5X6~<#RdQQ5noq5R4{GtP9~QunMGa@zU4bKYz5^%f78K-q8jj-TG!GReW=Jvq9iKT zJV9peh3`p}&#Y@FIVa)^Qqy{x z*O7=n9}1U7B@rTwj5E7P$%iV1Tp#Chsa3BvuY17b@)cHQvbC)r7WPJ$p=}p0Btcv} z$*zj@DJ47i8RN44g`d9dG@c5SbAirv)t#y*BNLj?y|!@pP{VMzgkq=$e@QmV_By6y zLzp2|435*cLH5y9MVbQ12l9`06W*5SyQwGW4=?oSo2kcrpRSb8DmalGA{{Xf+G>p* zP++k3c$}#DGEO;%d{`xqX_@)n7rwr#sIco{g$;4`0vi@_->1s8JtiB)f5tk@4?cbz zwKc){x$unRT)*L?Bpb;GzMz;tE)UC{m(tP|0({G$1J45<`iOZ^OFH}9h<`62(`VYH8iK5zuD~!%5o61qTEqtEc z=luPj?On<`B%BC%r3g(couS5)yE&S-2kkm(zeHxRQpW znYlHhqQ%^i#E0hFWznVg%IkC+v?gOxJ=SpKY((4H$}g@8ga{B5;{|=dF5j;^?~Be* z@_rj7zdY$=c!AL-Tn6)@}V5YvQR`$2)pG8!nZo6(h zvAz3}62*g`MOg-3-lN2Kba`wcRqeqMi2?+rpbYS5k~5)?_S(*5V| zut06RICouvIO?_+aR%MHVp zFK3l^I3XmwumEif`mUFYMPvKdhEUb4040wOWWzi}4 z&|v5I5)mW_-)-(Q8Vd zLK%{!U8*bnW$Z0yyYKc5^SeXT1@dniY&E*R@5>asKKk^r`^s@47k_m=6r??=qw=V< z{dO@X{IS=PY!zwatQba}<=Rkko!R*Y1pk?E#X2=GJsRZ*3~tV>jJa*g%e9kEQenvk%RXfG+?dP^ zKq0xBFfwwt$uifhr`Ki3O?3$5`-1pGJW+F z&?i-Yv;dANpQ?};N4nJ+`51E8LYmDNKjcNfoV;s@dUC-4(Pn9~nr5)8Du`B#9~zAl zJUt^Yd@mlgh?nmIwJvbXji6T=fs(46AoHhTDQ4C8?_G@Y;ycr4pI#JOWh*=uc}$gR z>&KMw8lOTp@coxomL^uTcQpi_S*-0Lg!)DV*!o5j=HIK8Hd!Nh#d9;{#TCRQB0v$r z#lFkl4qVqnM? zux7krqpBIxklAyp|L|aS>0oVSbfZGccX@)M(5+wMAtCSPo<^HHVFQzgvV_LWM(sY= zX0?^;2iW{xG6%e9xq!6GzT9}*Qpe-lkcLYX@;RY|d8!BYHg*30)usANdq>cT?IN#gPeO3c{`4KZ=hSCp2s4ynU|3>x=IbJ#JQ z$&irIDbX$JHY)uHXu1N=ZjNE`58-zzlioWJ8xoAoulk5tPqM(7k6=UtAg%+*KR54h zO;1NgWNhgtR-oKPJU}Q#2A6gpR|TMnfjyDL+*S!d8(P8T@Z^0j%hxsAVxx6dZVr@O z65?SF$I(Mm?C1R$+U%u^{elLkmjeL3{V)6wj^eNM6dkLlC^#L=vz7)fk8;~Ncxv^=R7Yy)Bc4m8bC~Fe7=k{Z#ieb6(g{gNm z7s=!ORqXMBo=%@migb=~lk^AkGIHY=l9jpcYl)t6?FS}rRGO@iJlXy}$R1;VCN`b4 z(l{5SM0~bNA0BnCzNKUDLvN~dPz8fIJgms9O)8$`W&%rqjIKoLz1Y-5=~8kA1cRjW z-klD)z=XXN&dB;aB937A*Gc6aBX&a35zA#p{VLqhc0Wix+~96a9Bnx5loDF19LKg} z=CEtMNAH@hc45My2UT1LPzGCwECK}rGH9tll)oBy5n^MY_+uDI_66#ZvDsGRcp;&# zq9GGHgG_y7AE;A8FjK{N0HRO)r$8T8)eBPLuRRkfx|>yc_c* zd@C240k$5WosMKCC#KO7=Ve@N46=UZ^*d}Xb~4xM99J&UTgGuZ^(J_DSY#@cu9wtp z&J_L1k1cYRX*xTSfSi~&)Z2JW8B^e`@o*j=6VYp;BgQ=9hHC36IpVf43kBmpBY%Z! zV1#sYOS_$s>&`D&WBh9#D?ZWfjhvQ zi__~Hmz05afQ`s-pr9>d@S$j0iBx}cexy7J=sN_yuo;eb@UR<+c~DQXX@5uUFV%h{ z%#N3~Eurta8X6(K6TEin4AU>pZqW@N>0kG7X}+^Ldbn7NfB$<~&N4$v9vjl+OqH!h zm9QY>T)l|*wAwaEFv`^?IB7F}3TH1u2iw4QHzIn%uyK-2dx1i5%hN<=E*SRuO|QEi z8d)3jjxqWIeOtARv}faS=_At1vEK_8@<`X+E|Tp=jPH3f^_TAzo$NmQTq&P5cQtBj zbe|qCxi++~RdLlQ)_0-C`+8livoI=$qX`zqw}@kbf)Y`ExhLl%F$PvotbFF8p*d4> z?l|6;1|?49)js~%iv{yhwgsugV#WOF;m^X?4znp56Ni1a-1Ngvz9yIGr?J2FaCmi| zudlRu!w)79eo82;ex`tDdq})p76^xL%)Y7EQk8>EkES~@T+n!2A1=S`U|@0F|D|}H zr)kT3!Fca!`Avoz>=X#g0!3Tx%nA->XiE2}#ch`EFF1jYGW(%O#CK09)xZ#6~``JgyVKM|%Cdmw-Ra{2N?yn4Rn zc9!-YBu;3buP3`b&vz0TH9qMna>0GLmeBNIw8E8P7al}BIp2n<{~8a2;l!=N^QjTKS#DghYA;} zV4b77+lGIg@aESY<&9~}ct;oInyKQ`*uC?&s#8+beNY+82{+yT;#E3qm&_sQD+5fuPa3f z-~D!<%dgrk^(w#hCK;y~y|$S@a*k>GX_l>k4C`CIv&{!Ig#~U=PiS(<_X~d+@4xR) zg+&#~KXYBcRD>Q{uVD+Nx+R?5c&N)YT>n~ke5UOtJ`QcWgBg665$U$t)gJhWx^prl z*NP^`i8zYVs1_npMTDO7aWN%gy1VpSa>Zj&H2c|lTDA7CK~bUuD4nbY9U|g;KV0r& zB-DP44k3G>v?&IUpFVBxj%EKA^x$bf;6_8clLZ9}Lp+>qVul4>8V1ZwPD7{LS%Xwk zw`m^iy4^*&{N$W|%}g4Widj#0RooP)i`ecEpFZU=7IZ8=(R+uD*MmJ#<4ZhNW^I@k z6xn^~7N?rZcWsT!q{Mt-KKwnhi7%v(UPV`~)-b8=DMwMtm&esg+EaR(*;#_q_01s| zm)64{v8a7_?7-{DZUajc{vKCvrZu*o<8_vflH|PO0om;N?eZHlntXTPiYCXE_W%ax^q#yGO9SI(vO{nte_mT2;Ft|BA7u zj9X%~xL|x}y!O5ek(e!PYAP8+No_Pu+-<)be~K0&uU5ahw(F<#O~$fZ{P|>5+_~Et zm;2@|#O1&7$O}2=fRA4NrgyDOGj+3t!gtL&>jvx&bjm{wZEK)DNTUZd`TePvzcRYJ z30&QYWOBMF@JC-l9z7#>2j!yaqy$p&T#RbT9wEOXhA|ri<~bjH`%YRbfZ0weEQBy` zgB&QpyR&DMjH`uZuCYm<^nM?njJ!drOx2F744>!a5oKZm>0wE}w6N6j!u28^`w-ru z@ubLt8Z-Z`QU|w8*xSftrRv$Xd`LTVbI@o_wVDYHY{kNW?=>`t(^nDdMXh)T@e`+y z7j-)&m+*Z!UUuQM$I?DiG3%Tx*mjekc4QfF86NQVepz~UX>`G<6%`U>`JtN5DyAPdb2$~RH3TfK47?l3sFxOJ9}=m=CtOuj2KZuQ@) zv2?A{ntc$j`r#&OrB&sNSOsfO%?kQhsAh`S$HehgY#~WOosFSFW^kMA6LVKnB^UZ@ z)d&!Hfe8fh1N({CDW6BtUm=ZIEZ4B0l7;wqNk^%b`bfI9(iShNzpWWtv;?KTn{P3h zY{20tHM#PFyoBV@O$z#A(o4kj<@`+w`eQOny4qR0qOdghuKdzLG8CH4N~hSNiC?RB zR^9kI5q``Zv*%S=bx%o|1MqkYx{69{lu>!AtmY~$#uy;5alf_pyZKK)+v!~oVY-mH zUO5)e;#oAl9AuViUwA{K8Me1XEO7huYQ)Eg5xcF|uz?ZF2sOn-#dQ?xC<)FgYcv@? z_jOpq9|39*uR$ZH-qyKG*8(`oH;{w)iQaMCHjjC2r$ZKXlRK`%XN%^I{Y_^*jA{>cUgOriO!9uHUg0QB+$Kw1y%sdQ!*J$Md2-IZ zzvgvldeX?`XN=z^Cx*g#c06?F`ve-ojH%0lGhgWWegb&c_mJap!p%LT6Zlv5n}5lG zeph0LVBy7@ns3c*gZmxvA@btAAlFQ#`bQ#gg0y6vVXD*^-?)jF8YO0!ujWLH5=6{p z8@=i29LuG3d>U>Ay@A=)x`}H>$!A(+?b3R$9XREHbE0L~<9>O?Sfa8Z!76D)T_4FgVT?SFk^a;P^SFNK zI>}qVk;LyRifah~A?)-&bK@dL4iaZWF%6XH&=xo)EhZZH$|^Hzu^vu()F1nJVi^O_*)@HidVR=W?EwoP^!DkEf#u( zZ#$54sdq znxuuw@ifB%<>Ugbxmyh<)J}cj`jcYGp7$jsG3)pHd^bPJd)f_4yJbo3Fus$qXM|c& zs8j2_pip<2>pebc)y_=_wj#Tp)VdwU} zgxtztZ){p7$ie*eF~H6A#vG&=N_5RRQx5?Q<+@rgeB0D*;vZd;yl)8gNs?@2ZGKof$E2$y ze>w);Tya`zg)s7SEJ<@OZ5nPm`97Qj;Q>K~2L6R!3mUiUv4Yd|MOKYq1H3?xd3h~T z1RNN9il#HI;aWBLi7EwPwZo$NxL!L_M1S(+U<-2?f1@#+Ax}|d?JHxIZ=|7WJAb|y zl0FeLYTbIYnvmfnsJ7WNQJ1_1E7EU&SqUx1dDCtEmpu-I02*w?s5n5&Ay_o+7kY-!nz@ z?vLAe97Yd;PPi2iAs{)TNZI0d3)YBI-KMzK ziHW)NBF=i8_-Gng)2X!zTJXKI$}RQSz?snDRL-fj>3s2i!6~JpX1?90ri%mGiJ}*h zO}1H?@nvqYCwg-qq>_H@jz2@IIpf(R5RKd>9m&nFGhSSaJqjN7*z`F5t{?kr*Z6dV z*n4*Ap{a5o%$HphDXE4HZ;XVyDaCFKXDKUm7F5RgCRqe!_`C2RW#=+w$fe5cy$R) zqOxT>sd;`N16$+XzD7}Ir*CrML3<=sZhLZ85K$fRC%&mOj$jxkzVffrhn9-mdOtBd zUNP`;Kz~x`!{Qs=P1}~WM$fVaHG7Un%JoVR-LWJH?xWXbAh4?kBIqnT^1v5ep}fJ;<| zsX-W}+d>4#adl>wQj}u;%nj${o)D2HDNOczUen!p&FN8a0xx)%(QKoIgRa>_`a=K9^+hw zbUo8vTqw|SGQ0xV=E*;UwrtelPCIwTjDKPktN8kHPdTU8{d|l0mG*Ah{y9jgrd8X+&uRUM_^?OADoN1XxYIaav#dQx>xT9G z>aq&vMx!*Z6Tt0w2^;9hGl_@lYDhij7v1W=7cwyh#jH14@(x!>YYaCNM?u561vJ-5 z;!gw*p=rmP&!#IE%mthn<07gzP9Y8t)hiFss!`dMV9U3xWf&yr?6`XhlC`SVk7Tja$F`6HVc zdj=KO%lwLSX(8Zq{tU`mq?C1%;J@8Mn9R@jxH2~S-=I_Bp~_Q-4~vss+?jCK_zAp3 zi2j(=D+B_V4OtXrU8Jzk8;G0ea}7qvFTU^Q?MDNk<%bifSG3-AeC>{~*x*P-;{1ro zDq+MqGdLD0Ubw*Gp?!aK9sPdqJBPWGYcLKfPi;svPGyzeq<^L8_ zQb<3pz4HBog_cNOX+HT!!dLGAV?e+Vh}U_6Q(!=8^9aeiTm0w5uUX*7+k$o}+5|pV zi%llChGAkcKoA|WMg0}^_eJq%R^SQBB8ZUoRmNR#qQX$oJ}mfx&;`(s?}^U%UcF}U z4GPgXM?2O0&*w2@|M%kGV@Qzz9%A0zKqcTwD}@PVF|UG59WQ)2onGx>+n;+W+^hXb zkb}2Ddo$UsD~IWF?0+87kLo;Az#&41aJ0gjK~I;wQl3>{j}Vg_e1a0_z_}UvgY2)V zJA?rMiRV}q+?0bdD2VVAbbu~O49bRG&CZ(-!=zv!#`m9Zx^m|H#le(WR}{Fxx5x1Y z9z*R1?3JjSZ$*tRUy6P&L01JlnC%I?!>}`5HU)CeE)NlkC5A& zL(u-77*37^_#Am*)DFPM#UuX#Du8cBrsMqQU9YeTHe^E%e*5dchmMm|gYbnXjGuLp z$yuYb94lLsaB|^RzcEpIecLr2op&^KJR>Bm8Gm2z68as9xj>iUOZt-$%$Y{`3t{Z=3{m zcrkEipjY!`7>GX(mWxn;Mr$*hpnun(9`N)XZalCNG9wBFrfHIr#q-?l5H#=Mw6g=u zT?atfRioC7OhV{c0lfuE#f3+{4`vTQk6I}LS=gN%OZ``{$)}u%u;LQx_&sf*5{y?! zbX>Ztz(hc)3bgXl2ERJ)uus8=h1mZHS* zTqLFEuH3~=w&2-gJ5Q}TN5(khm&36j*R-e(?lTIs6s(}U{lQAs&t_XmV}Dli&+AiS zP)AUTV(fE?0hs253dC;C>FSn%K1&sJHI4kQ)9{}`r+6Tog{CBKCro7Yd$G-+zlka_sq){(R@(tNMSF2G*azYjw$PsR*1!^1C_yk-VU6(*YrorbnMk z^xq%n|Nc6s3$gnavb!3<2aNaes`NeqfhE1eMji{favt)%-=^c=7RrnhY>f)|cAqn_ z%G5Z^BPHqPpdnNiqE5^fs4yx}YvvdO<(W;O8`6CbB!gndfvyz0t_z*{D}QCt{~P#U zkIu0}45&s7iVX$?lTHCdQ7ceJP|^i4y51YZsz31oo0x4Mks$;Eh~&8K*ZzB(|KqtJ zwL;(prFhN*cD1)Hy2V6wpZgN;wBG^|D3wrjdqHW+r!d)B3p&Tih2}l}|2}6B7kJ~H z?zZuWmGT$;3AVBgsLsd%S|EuhK(#D=4bh}$^LQyacowtf7oeo4*+Ps07%Oc+kEDV+ zaA7j!y;8~d8;kHVK8ggj|2gA zBI!gP7^6<(#M#dqf;KlE+RIV%cT?O)6mz()mwcN*C}p;4SVAUeZANr`Ybb*G*luP= zGCOKI@Y8SK^#v8jqC9{S_i=-3++xlVpbSo}cf_iJPEaqWm1apXMqANzQVU?~PulhJ zFKGjL$Yr(FTKEGXw=lW^#5Juh&JL>zTh57DP4bfpPv3`*ACq$7xu7ojzNYmH%zJak_tJH8me1r7IO6|pY{ZXp0~kscM%h`@ zcpL8!{~3g8fMhoEIhA+26?amadorS?6tZLTwE{=H=mF1SjDFh2bQrzYPOcrO?%5pD zuy*h~ThpOPmn~vrD=@hW_Z2)%Jz3;#TDhsEOZluD6uWF8KvaW5d{(p z#gnGliT7WZ9cARZCIgE>{RXJTh02TIE+GoS>p;QIB%n92BlP*KA=g5VCOu9zHo=(- z#=0IWg3${W`aU6l56@Zwtu>+?;i*4vL6I?)est0RXzQ~k0BUdwxOV|gP0jXJ3rIk2 zCfN0Kj^ik;!wbK#Ss=6kHi5o}El?2pyv$aF75pYj@N7jkF);V|WwCaH-US7GnjgfF z6whXoFHVh}TClmF5I*XE{-I3pbmn%E)7Dh1fjieAWFPxYk7<9siycsiTdJ8p+jL^3 zFjk+kc&(S(`#vzvbmyDkd7ov&{!rh=KG^y?Giw>J#6bGR-CU^0qfx7`e=h}+cde)w zyropdysc+-dR;U|Nc4PaEa2hBWyBDF75g18;JTg#oRb>=xm8R*Qwj2 z6DoKZqDwk<@asJ4KJ|30f;_=Vm(F?Ez)||C`oey`&KY!neRM!NhUH5h?FcN7jL=S5 zg=;RDF0#Y~q*69P6lPISQlZ{(WVIo@9r$p&CHe zn@R))PYdUso&dTIWnK5WcFX2XV_Qh~wi{hP^S!yLT-^V$p%FK~sqFg$U^Oc~c@cVX zjh4e_& zxT9V%gYQp&HD8+qQjUscz8aNO>{798ZHYh@rgOPKmoU?y)#a!6jnYMNYLk3MOUy$c znL{0(_8FpWLSJ9bxMGkHRw7Q%<4*S?x?2QzCVm|?GF%~Z%MnX-$ zGdP|N85REMcb{kbsyg)p2C-C6WCvw)EIiCjkPQ=!W*biRqPOsgl?zd+^2r&ntagUM znJc@@6pD>$jbg^{`S*T2JHGF?W7@^HuX0Raftl-U$CvEChv_qc>+PM|28+xxVQ6$Hy+ zSqjJUP6g1VT6TL#XazJ`!e;*qwlcejTth>}<8mr;jal zvm>9Rempz`SBvdMaSPjyd0jcegYh3y6Jk?ey>JA?ZTa}}G`eF?ewQ7C>|XCNgVMt( z*E)o<82?TO#bz76PGDuR>C)@Vx17Ji>m{W*QN zK}ZKRkA^7_a%P=-3#l@t3JN4ky;tXZI)PArW*d4wQrUfpi zgI@9mdkwgIv#m0zTT&t4fJVNyu0!YraHt&EMzSnTM}Xu=2W`6D$34M>h;lAyN~Y!A z@H9LT_1DqVY&+*0)}D6b7G=`vU{{7AGjLF^X=VTyq!q%XJ~O{0D>{J5RNWk=Z{;A6 zY*2S|178ojSlHX(qH3{!wvewtxJp{_Z^xGG#wEWfwzer)wM=~YjZ@-qzjWTly9{FP zT&G61U(ZR-FTFXT1q>5g1>>!lxCfM4+)>0R_dY(@@ECN z1W$hzpb-{Y^!caD-4Zx@bqFx#MUHdY3+|?4X(u2o^WjChGF#_KkYSEcN^MePplj+! z4W_x^9kUbR!ZeH*^qFD#xe?myaEBn!evrZFv@hd=y_ZkXs4a2l1386JKN2Mcq!K2j zJNko}A~TTBw4HQeK!U`wEsEOggxO;|KnXmntD|;!wBwv3L7MTwz>mE+xtvMTC%(I=50 zmjrA#OAL&eqDSWfKAa#*jTeP1rV;0pR#;IEry?Z;Eqn>DO?CiMOK6)3&pj+d9_M>1C|;?NUXNtl|{% zl`zzW3iYK=n)ZD!Vl6uDk?@m`7@m3+EE#AG-xGo0lkT947hfiB1%Ks5Mv^)8_|OisL6B`h6<>GJe@2 z9Qg44Z?D(zq%@}O07P|O8Qfo)RzSbGYv_2b)G>!k z(nQ|6m>@hf?B?CVul+9EsoOKai7RHiCg;FMKlvFVX5z{c4%&&6!7tu`;HHc7fkg!j zA`>U+uXO=R2=t$=Tzm^R)qUbp zq|l#pe^DBa$$8%#?}y{K<D9W+9*Kvs!QCb0z3 zyBUM`e1?UskL<`V8(VE4_vqK?L@$?C??$;|dFcXpQ!oTIvNOW?IqTZ+F;iz*PS|+=!hjT^F^}UqK^Gh_mD|6L zq5xGs{mKkgx3L4?&Pe_el&2^&jv-G=U*rb>iKP&F$yg4;G6dl$pg0FPhng?h?ZjsU z8~3kJ1+65Pn_bFwhxzK-g2ecvG%j}T#yN#kLM3(c9F&7M1s9v z3b$?wsd3@^cO>xUIRH2L-;7Mq0WZxR)t+<558btY3pj_<8blng)yMfN+2;R7tsmkkatx5@F~XR_14agOhR* z@?^&PuRPz23rj^Fe?ZQO>4lD+0}q~!?Oum8uBBgABv(I>!sp^4UINj3a3v$6SGN|> zG4A&_ob^}sB-)}$cIy|Xr)Tq8pa8<_P#O)hB>(g2Oi0w-TxCtX&O z7)EO?0wT%!o**PiR|KsKp&8*0MDCr`4Q;zQhpT}akCEFQ|1naCT_o2g!Ia;GaxAcb53H%a!znDqZJNbZlCZ>GR^GHPT2)-#4-LR^iw zVq(|E)aYmC&23Mr=0}BNzq-7!C#!?tP*v5H&W?qdfxZtxJ)WWYyGrHzDgYTu#8Tp( zv=2AzuB}koDF`=sqWHyV$D*X|ehIy~gXD_^AOsqW2bA~C88l;s9Qg7LPWB2AxLmpI zvy0%dIJu1<$0DN!z_|4tyapwFc0dBuz0yM7O?f!kD5pP4XU3S;_?<1KRnytQQw4J| zh5nIP>wHjL@Ssb*FY-BGGJ)v7!YL$gA~CQPgIp;&&;e9Ok6P=ncE7MSmdjGzUXrJ4 ze#V#7aZEunt;|dtbV%pkYa3EPqD^oceh;3JC&!ux_JzT6;vpq2jP zP580llbbK{kxCP-?=9vtd6v5KWM`x)FaXoapKozkScnF&AEMkqAB3`<>td|&1OVwD z629-R6srUMLK1r3z(MqyP&g*#rVJ~76Sf25fS!jSM(kXE15lE;jHEcq`<~sqoxWTy ze_2i7d&LI}wH=%SM)Cq+Bm>il=RNPAWYZ=K)(BVT$roiK&LHY`d`9(qbGzZmr>*>8 z>m$&$1tL!_ixC|9ct)}oh(g$~be4*-GXP%wv}EX2cP2OK8FC7DqZrigiq_<*BP(h3 z(hVIkcq6DEfUHrA$z0FECI`*=1#qcaa37UQ`2p3G1}ROKJCV79-OVfr1yv?B>;CW$ z32xSWYDH-m!@vysh2Q?yihOnf0qz*zL=D0;$f8qX3jqh*4#wkLM&JPfw%?Z2SAm!9Z5rFIkwc!!bW_TnD zX0T_8d_mgd#CA8rTu6l>>;$~_?pA&d^2a6ImY4s48-M>OH-`w(UkATO1NfzbpI6V& z2wb=0o-uufyV+H`G zY7ED~r}Oja!lbKs4*$XB z#WqEV>CP0MUkA-k&&moW6>Ln&S}CgnyivSY%K3^n`VU6!@@z6rPAyBdSy-!qGsI7|aL2~v7(*2+B zWi720_ZVlw)Y&{@%#;-$^@9R3wnGN}O!1PnVF!e$cXWVep)(=KsWKGRHy3;C(`PH2 zwS#Qhts-#ME#T$jtpJmx7#nF;Lj)ql3;w8rvJ|?Ow_s((|HsOvBUV;E_+2|VXbz#x z;xvi#1`#ZMv3yRG4$;>s5P`@iA582HmcF*)J@f^@^Fx29cBU@9{f_js+0;D8%@Fjs zk)gjKWiWUInmw6(NkE{+q~oGeXg+|sxx}DoGP?!3lo0Gj(yP=qoh~rDsMw7fM%N%j zEq1CbM*Gj!jRm{j)6eDh7{h?T{xHzlma;QM1zL7`)S5|40FeylrzgBw{uP=rvcJZI zaQ@5(vVyeOY%jqmjeG5E@MaJ-P9Shf)jb4Ii8;LJ1u>We%Fiy7Z?Sr34&Jfxta&!D zecjE);)hmJJkol&_D;tO3_kz92A=sp)LPTK{#YS0IZzGM+z#>^)j3 zOJUl%fIIvNjMyLIjc6jKtCXkXVpTo!L3PY!DRHTC)F@t6(+~v&FC1TzCq$&to>(@i zBz2HE1L&*0)MuQKE)`6jXza}v@P;F}DHOA19Mn1Stqh~YvG{*umunpj3io^c@D%qkdyS7b8fIgu-1HfyZ(mf9H5F-sn4_al#CV+b`klYpYad-+)~De0g^~ zb>6cT0Gkh(rRjZKwFClr1*?cFcoWDv+p}T4r%wVT^zrmucPK9OzJ|pNQ{&MrOJ4u^ zsv87c!-?iVlQOGY9hS}>ZGS&7FjO4TyTcX8{!aqUp$QQDhKmEW?AI#VE{4<=NqWP@ zYaPzRO`e{02J>dbPufg*zuXo)WeLx+^S5vBM|j#!%c;Ji*KY1Tft!D&K*3D~5-V$j zsq4{4Y_4}7U%HF{1hTwFGbb`OvtJYV?0C@9r5tYnI0>Y@I%4$vGW`2Whb^S%KUV7R zOCK?-er`6EESD?4xdPCUAS_YPj3PL!@pMOW7!dePx{6zF@qV8K?frD`Q~`cUEpXXe zRst!ch2jGN(OaBK!?Ge0AtK7~c#%$e10-Hd85;9Y)Lk-_#qqW5<}3ig_qiCWt7tp7jckM-3)rb178w( zmP?4x>y_h0RCY=D2cq!I(MW&5BD`CegWem9v1`bFZi4x=@Mp+&YD`*Ti@kuv*iCvu zHv^#`YT^JZnG6pUj5A>VptMw03slY(st5=D5mL6bARETOL$g$~loU{yRdN#bMM5;g zLIJ=qTJ8{g4|!W+9`gZ2^6X-(Tv?jOX$7c8=j}vlI&O3?K!1@_m^FEvK{+19Z;R!B zI0x!yVxlu_K6H1!?B=5K`TyHY?a+jj!zf!y;`*_KAk7o1$XBk094m?`y?Z%Z4qb0k zXsWm6VjUcm2Pfj-Jva*t~TGM>UOItvrr8+5-HJ z6`(H3dk`k&J?nY8l+bY$xwXH?w^v{U)E*`kOU6{ej?kD{d1^l11a7)TcS(RFCcA>; zofxQGb1oa8)x=qixcMF5vb{37uI7?<@h$xm53~WknlH2T1oq<24NDk}zOijr%lz3U z$XnbANIu`o?#!UW*mcW8rHK589s2zvl!9zL$h4;(yEFsJlEKkE-^r{p-WJQX?UDFO zdzSNVB3|k)G7cg{sX<>JJ_ap}0uB3^VJm`qFY_o_z+N{orzL!kxc**fB(A8y)g9QK zfBhU(zE=Aga$-XD5rVbik{j6buG6^UQxAy?IQeOeFnhERNDL5xW$IfcFJ$SD6&!_N zDtAgpD|P}5e-ikBcM0#5>&=72wLca)o7Bq1&PR$*A1m*Lzs};xO1~{V)jYii=rh zaMIi@-u0hA_9eI@qzYi98lgN}tGn47_!Zc|IPRwPCRaKQ8q4PI9{{J$Ouu=zle7Xp zIuD7V_k0~hZyM1n(|xStoA7sRD(8wGVWqmIu5;M+`HRj`^CNA+!`k-z%C)|!EuNoh zrV0AM*=a|4l=v32AZm#X3{UXqY)4qWn9SyAFFhM?hc92lS_!8prJ!7EfFMH@#I zv@RGoinr7|1RMi;O+?Yw>%Sonp_Fs*dyDKppTp{qAzNtpHLzY&mVM6* z3HpF!KHrVI0CI!nvR(>G5i+E*jK%tg%t?xo5#((b#hf2ZTW$Pzj6L!kJjzD?#ZT~{ zo5ifmSQ*l*A670oaUX@$bwat0uQOi#jgUK82p3wO@zxH5`f*0TK5X3LuTn z2@j^c_x18a$t_`+Xbuv@M0Aoe*aMy-O*+Qkm!%LtS<0bs13tCEdDiE3cBAwXhR)Az zbrxtT1=rcA5=X)B=Icfa+L1N^}Yj=qm44)P*F@xWJPiv!gi2*eGPQ~wyr zW@UM7K3wcn+^CaE>O_@Kj`d#vEtpnCpx4xmeQkn|m42$ToaN&k)3dOZ@3WI@W|nO z{pdb2@*!yH`HvLIV}pys&&0`ceRmj6MSLrAvSnx44Xdv1eh}9}Nre z9vwTwZ~oB&IEwRve`q}xEcs*xI#F8nB#iXw>EO3tk)CjJ^ieq$P1i9gxtPqFmZ}7` z?^FH@(gyWQLR;Q`yPkVRInQ7liG7->EKL(iorXxIUv-58*i~G68;FdmfwpD~*(HPQ zj(0?(Zk%k05klgCQ1ExYOh}75gl(~V=aBe>s+HfmQ9JsMjrBjz5&}RB0sD;PNsF=C zCd`6OoLl}GUrZO0!PAzT1G6NhFKgoF&vqp7gb_>a_Tm+Yq~)hWex~wM=%0vwBI5Kv zr$F{2yW9wa{3t~DoQqV?HRxrr<~C!GdFJ%cQGXk-Kar3(kJPKVXyU}FYq&cFx2^Yx z#!SB#7cAb^Y7+|!Mg|E1XLv_&D2;p8begsq|p=JQU>dUx4BQtNfsF9}g zqy)~H6DY|kA-k#$0=w`o0k0z>Ru#bkiBq7J;IMH683ypt&nQCrY53+I zyH$!W^mYI#T6T{=JE-<0nPI^Ilr$&+E+2^{OjvUex&5Cd03R4}NWi;u>VT2BFgQJ) z$R~9F&gm-M&v=a{9{J^U>9BKy%$pYj`Aj68frRW6in9EM!k{cFg#=elCq>{9&OYA7 zj+wjPY3#ZF>R|3Bxaw|~{@F5v4^fEbIEJpLvHlV+w#|g=DjR7goX_YTw(?pZ)Bd)C zzefO8dzB8k;c(@QIb^;rt$`g{0ZMG|+BMz0?gJ@CIt#STBvIgBLOU)MXSc+MJa;Yzbz6I&X18SR;EXX z$G_ebb1^~P8NTnf_urnF*72kY{Q4KE5!m7PUQ>sAQ_&$M0v20-T+wtYbQ(c5sgvzu z-!g^y2rwT@(@}()QfyhTD_ez2pZ_>}7tA|kP6_IJZ4phC5hSh`GvEr`f8PwmPM6__ zTU(IgAy!?&(8niGDca4NuQ!5Ab3l@;4j9iGgtqOyKf8A5u4<`)-`Z3|-O1j7Y>O+w z)-@hJ_ERgZ*%o-C`={ZvdeKtD0V6B;Xp_x`MrA4O?{c|+Y^w#-r~>VU1Nb-62~f2( zlUP50&2zhUD8C0u9XDq-q6SdMZ}=Rfi?E{|=M+Exkhnw1VBk?>qA2gqK~eNfY#h9G|P$ovFRQg+?K z{ch8?=a98wuC|=;(pwTmaGWwD($9}&%TaOw*%QPfY=T~ulz$2f3m>t7HVRH>`|8WM z;Z4UOiTmj#KRjX#r}OvCi&~KV;KJ*?U*Fw81;0!te+AMZ{{Toyq!(29FSzjyP4picm%5AK0h9Quz z*K_SPpq;|_)jT9#fix+iZhA9-d63l5AnP!3LHS|Cb2PFw&#&Ul`&3TOlP%_yfM&c2 zQc3L}Erh`Ife}f8+=&4VjK@L9uVtFsSZqA+IptKO666gRj#AQ3X%y?49?Wml=sJrW zX40lG?f)F=dJwp)m3+Xf`e(!a!v?aU6f>*MBMQL|+eAlg*mjBz&)UsxySUG6rtVL7 zRFGnsp+N9r9=+U(VJ*=iYA07jp(DsUd!u93UyLy&S#Jd9`1}{>3c6B$_!O|>2enY( zfoo++Eay4Q)r0vwl>7Lr6yLGW{1r(Sy7x%qScusSK5EoEmjsLc-WnCYUhPalx)^i8 z?bSLiFnmXM@nKb375$X9&SK{p&%-?ClhEabpM$JUb)b+#{eR4X9OX^&fUDNmqx3_8 zjeELV?~F!DoJ29rZ^1aqM>Foy_97YJ<$ONwt_Yyi%4FVy6BiRAWZ!r}+9$DRAC&WY z*NO=xTe`RJtI2jm3KR$kyZbmOtyYUvI4{U+NB%#)zB;bT?ECtHAP6cLzwK)PF`ySr6j6ai`J?(PN!5$WdANO$+;#(N%P#@}~l-hccs3~UAnIeYQ{~GWxlLX({AZ@ej9VUVHHaW1uK_P=jZ(j;p?AcdgoBQ_$fs+i`901L z1_|CC;3t0pc+-}|!0p|2JtI%?+YSA+vH_XY> z3wd0YS~u?y7nK&|8-SCZ7tqzI9hTaAieo4?^Q6mSbI$T`tkp~u6sUOtTz>>kdd{t) zXf6;Zz&GfUIK9d?G}_w}5oqeTC8Z8@=T~}_czI*g&d1LANaU3Rj^V;DQX3+`-h2h> zgN#UiNRWsRWouyT51z*r*b^IXNY6tR+sNt6THI4i-5?<4!K~DlmFb7^928{FR%{~` z3TvIazR-+UIFTfxyb2!X4oCC-h*JIzY-KqKmpf@KP9sZpjS8%04zwLG!BZ6JF)~`U z;tv}seM=}`xQCRaoEvukbROOwyI5JprE;aA$8~agjYpdv;tfxL+hAv~K85V07?DBc;#0gvD-$CT+x}bdfdgVJ|kVnn6&1 z9iz7Iz5kof-K_0zFXF@E%{vC|S9z_!!a>92IDO90xsYVX zYm=%%>1D35Ex*Of{W7>I(sD2NP&nJ^iCgggR7q$d4l|nxN6JYJORlK!f`e7zxLe>n zgv3G|ut{JRzJfg5FzW+11pETiHb7+Bm`F zd0P3(c%g@|Ft0$Gtuxv;poHLl(mXC&eFN9DB>@iunXA_$x{04myETT z-2`Ib|P09qsj!VK^`;; zFf9~IOB9^c^M=}<27~q?-!42r5p1Rg@IYwpE$0Npd*)(fm{+5f z4hldv6}+W7^RO}Sbz8CQ^icCo@7w!1dNGI~?Qxet{kbTe8f@?sM5HS%7wPrqh+s&KM~s2t}>Ee&eLN2Rl1U6Y%)pbg6kE=+)bm2Um;jO!LCLQ z1Z3T`-&h!&WxrnIT1~xG9)S>w=$i-JaYuTwAbSn-V&6638fZ2myHyDkL9$l}jpf$> zQ{K)f!NpKNPs(J#%YDMCKBu&QNi(PW3m1P|y7O9EIO3B)UeWX&uVIWHWO4w}zd?gL z>(lj8610Ecr*F(9pA^#GDreB_sQSi3zbcafeg9kZ3m|wVu&4_+(@Ux5Im~{(x7Jad zJ*z0~@4Kp*bX%MAuG`67r`CP(g2!Q(5t-8`gnSmx6RThot9QEY zF(PV)DTPbah3I-i&lpvJGUT7_pV1c|q-<|>p%!EMTA2(Os~>o2qgz2SJY@(DkpH5!iRw~Fy`e<#{U##KlE($h zzDjT*TwwxSQZrnKH*Y3_!)?Oi{9sjdK~!(qHYR4%Ht{`SsTv)Prxki79CVf739 z%j29{ne5|LS`|~nq2EKJU#H>CxqCa^8k_t6m<7@9_v`z@k)#_qOwB!bg@(Q2z{zG? z@9d50NEI@_tx({!wWQ<`SrQtj${w3){`B~=Y2$-%R=rS@IVsQT4BeQ8>5rK@8%hMD zks(l(8p2UG0buBG;dtRWrSKzok}6(V{kUGR5V?g3r=FiOLq_{k+EdoTfVvo6G3pRX z?|ELOI26xE2Hx-@4!Oi+OcU}Vv4*BLS`0MdnFNgSiE~VGaO|bf&)>5W{42dxe4tG{ zFT||@{UXg5h$$(!FUIV&HWZDkX4%O2UC;JpX0V8~X4iI+sLSbe)b80H8#SXGt<@qp z!Zz4c2{xQ2ybUl|CdK*t$p02RTH{`8oLHub5gO1sZFlBHH@A$ksD8b&c~~=;yC~LG zME^2)Wej9D7cShNxt=@4lny#VPOZz!oq1=fsOBcejd*)+aSX3OYQnD|LN4J-J?4wq zDh?cwTWp!)?d8)zJFkFmW;iW$cbqmgoj_Sitj2n#$vmoeM!(Y8~nESQ4C8X{nJZ`jK4vpSz-+e*3VsV*)@Vn*g||(J0ytGaoHqT{>6D;aVPX z&m`CL0sT8lZ+(E~&yTHx}If6Cu|Wol5lvqRxJ zL~N$qSBMPenuNb3}M99LYYZ?cwF)wpm}j+2&F@IZs0F z*Jgr2c+59ztGEuKvP`OSTF;Bvc5Um~ME10gUeB!;^RvrsaCYnqFnrv?Xt6MMGp~B4 zBNrvEC;Q&gpmSq!FLCAIPT`vl79d=*^FZ#~sFUgEl)i)rN^*?OrL*+OG^D+ch9Rj7 z5^yyH^LPL7fRun^<*PcjZv-NNA>fKbkc#ia3l~IoHT?B0-&-y;Nw*^}ubpYaBpai} z_7=P1&p!bDi>)VXoq`o}FKCpTM#v#auw37s(BV(-&n-}4TrpP*G)|*l0d(28^TKAs z%VO&MEuULz-3!`-0%n}E0!-P;Y-GF3QepfXxn)eT26p< z=wXD&->4{v)uFuUU@(vo65#Vi;nAohT{i)buYZ zdDvU_eKLF8(%wEqoL+r6Yg$nM$&Vv7S?7eUP&3y^Y)fPk_?8`*i2)cWOB=W3e0DR{ z%7IYfsy>$-1Mj+hQ#V_el}neB%j;_7gk-3P&jGbbt1-*Qk81Ms*yg)^Ll&RvMR--A&mQmy-Ts82jnM zoRX+6t65l;t%kEu=Net%*yjf}(A9b!cNJdt$R~VMc07=93g*=+>DYG|T)Z1J27&|3Gnmj_zuh_ae`d%h0`?w<5?J^<6$=aaoj} zok?K;)ijAzDHx(xaTt#?wbdSCGN6#kQWovMox#V=9Q*15cO#?SLR4-^Os|sal_mkz zBe}w0m&xHZVeM1P{==!q;QDL7q&gcQGny7wU)!1M!IN(nV&=JK-z3c(t7E{{I7C?^ z%x^fT6w|IBR+M*AdO$lI%uM^^zJo&2zk#@KUnb=cj@-*wyc;I+C+}Qvwc*!4cr|hK zZ46Q8)=ExPV+q)u6H0TG2Sy5yx`saGqGmYuHYiTXndZVyA`HRxr0hQwYi|HsU3CyE zjIVdx&3R~5NvPoYYvo&YsU5t-OQI7 zWJI@z4eH06mbOHT*e~$P0b%$-gg2#H(`jW@?*+7O=4D5&Mc!2D{@yEo5s2OF&7$sY zb<-#h(G5=a%3e$}GGx+Yl-4z3M$AD!0=UD}g2Frkl;tf8+>SOw99IY1Kto`bRDhwV zb%g!TwErC}40_^%25FW60;ALdcrzreNrN(Mph`qc0K?$+TDdN-hQ)*Xg?n{=ezcss zuS97BB&ocb_?2T-4u&3!F>7FT{SqA;COGP8w*Kj9tNXR>%^SS`DOdd51Mm-5-uV~<&7ft|t8<{-VPgpj z5eqgh!$!ycD%9mS&IIQp&jmi%9XGw9adM@*)Ne1lty)osOQcvFfwGNC>{u;5r)g)g z%d-*_!nDG;cPp=^v)r{ysbfCs{i#xx`N*uz4Ov>kn}P(y{spLHKfi07^REghT|zD2 zx1x0GXSfGO#*Gldh{cQp0>k4z@?ayE`cG?-CJ~M+{Vf-$1qS0ikt+qDXQheSpZ{cT z|4hx_QTRY|SI`}%1}J!|f$Y_YwRDVWV)a0{(5yb@#uH%Br+<741V;$1&#Ea^?Qa+@ z7GvW^pIZgGfnhZqn~%&ag7C+WXF@A(Ha0Bw3_``zFSxR+m0a3 zqQZqTDM&Zn>`&qZs6$=KE9Vop?SKtt%nlP5jA2Q=`DAc+QQjk>x`k20Zi(gI(CP&y z>n^J+)wTnJjLz&*3!$F5tSdR`Z^EG2UZ39W?}(&3i_0AacZv5&2Qw}%MdcMvL*124m&Qx6szh(3hSms1u%)=^Ko_k77`FX|;D#e|Y z0*ltY*lXGkJzw{ih!>QZ8~o`HmtuI=QFS>(jVa~yPGhwpN^az>!03d2LYY95*%zIx z5u|J?kQPNsJsad}QMtic0lW8L7{au<}J2`&e zmF$#4rar=e`U}7P91Hxai^Y`rimqGpgWlq=!hW)upr5qi1bN+W-aN?&>Lxl@rCGkdWRd zp|9Cqv=-3w?#J4+&a!JJn4Ju0a!(h8>;sp?B;iYh7w#-Z#Y#Q|0wDup4eH3-1>{Da z)npOdd-?fOH2bcys%qlmuTKH@Z~1oB*B5uF{-?tBOVsbvCGpK9XEuPtF+E*<`$`yv zhw+~X8PGVOa91V!m0u`JKA)t4W4-lr-Y%=Nbm_jBvZC3W%E2e8U!-;xTYcdk^rVhd zU^x34ag=ubO>JBqS88T+tqpFTL`Uuz)d4M8I-o(NLOhH$Ai?@A@PC{fe%T@LK)P=T zkNy`@APwAN80VZ6yFTOKs}k#^l+fOtpA+vw<=<8G!enGtpHLfhgSBUS$ZPic0O!|KQQ;cYGm)x`&AcxKd%t8+T7EC?sl zbY{x<%IV*dG>bPj5#p`VLE4xVM9pII6a-tsym`+V^zWM>Ho zq3wafrm(Y|8LFZFeeM1FWs>?isAuwI^*2)UYg=Z&-n7WG(%L}Tsq6&aKiN5(=I z7q1yf6{E;6PHLLy3ZqD~0`WRk!-fy3A(gbjz$C*o$et&PS7$IeW9xMn#tt_C zE7fR0h7kWhm#3^5p)o@*puERbP)|6aq~7kB+Lg4qj1oYAaC-ZMpu z{sxgXcwS2VX60$uax=Cu$7N-Y(o~Pub84-@F&Q+p{&GA}X{8g;^iOfx&&ifGI|t;^ z40mxOA|w5dwqr=HuRGMMboKbHzVr>as8$&GJZwN5rpvd2@L-%HY)&h{keIW5#ZC%e zvz7XkK607Lv(qb~arvm37<4H;*aRK!Nm94}m)G(4hRZ6)1G|8f+5w_FY|{uJKiC%D z#Z!81++WTkS#Y={k=u3jbo_?wCEV)s0@eL2w)yVuBTEXF4-0L_-Qs7;%Mnk`lF+Pw ze%m)3^&N3ybVL=VG0F)Z>bUC(YPB zpK{f|AvC|3Dt$u=vDBw{mTrYwTjqdD95OToYY=!sl@Q&CvfL^Utkb!&z ztHDv*m&W=nw!*uZ_c%}X>hXI{hKJd|qUwJK;4C}CGe2581JvmFTHp*?6Hu!+flTNB zKV6)7AnjZMLYA4!8XryYdz-aI^40@Qaml|CrnK<#dP2-tz2<{<1wK*r<*IP-0;Mw| z>k^Nt-i!a-T!0dBjmI2<|AGPhtL|)<>hUZ~MFU@-)hC>3D*_(l3~~zKzr%#1aiV9* z>$USy_Y81@2-;!7d-2~3Hw<>c8jJ^bSs!zo|ACX;+l#6c>M#=?tBjqm^Mlw;Q?qx8dEw zSxXzlny+w|9>f;hH6OXd|NmJKt`F%9v=#R7nnpgSdt98Fjd(gKCD6 zGnUk7m1>v7RFN&VFYTRs>jV}I-;-(tB zv&4@X^7m|Ee8+_sPo-@Q%J>177mJ`YUt!FbP@${` z(DV2_zf35|c4&^skL~997l*8$lR3x%p;9&gTp+Ph&vob0MK&N_ zsNfXiGCz6^ULZ50Puh*;J6^?acQS!h_HSNq#1m`&={WHHulY~`d*vX+Jrg=;>P&r= zadcrsPiDb^k`w1GkY(;_>|72h8otxX{+gn{cRaY>R;hi$UHTsiy)w7Ud(*@6n|Ddd z4ZvE>dVs4-knSnwYORU|Z0)A+oS~l&@z>;hyCV*Q;O+Oa7v(IZx#Nu3qokzQWUJHL zpcsda7w6!Q)woWJd31k#UBe7_D3o`Ru_{-EhF<>vo_t$`;gOLwJQibB+-AeCjvAs! z>ed{Z8HZfp0<>Q=x7b-@-hbN&S6_O+6_$T}(lv?BT`t6!>PWs7xp4{g?*j$<+~a1F zOaCW+cvI4eVyp(Y%1*M*%=>4sH#5-eT=nsGc(9l=J<3og9ZVzi|9|L6c)NS{qJl*x z18654t8fRl@aZAsMJkRvd=8{%sL3YRS|GnX`_oMJU#35+5>I|nk~=XbPLGT>)=SF3 z7l)aQ7_y^cIHziqX4=Vt$Qu;&KghH<*1h|UwDs@Lb>R1XEX^lMv36u(y|j9-aG3qR zdC#=qHFdY5Y%4Do4glrxPoKko8H+c(p5fhHot%?FYfmL}HWYTemcm&uQxH2R`RLD2 zLJZ0MRjdwDZ^SRDb3{qsdY+?jbuWf8UpC)Vkb3N*g*rAeBZea1<~2_?73b{jM^`Sh zgb-b-e4_c_MV1EPgWp{+Yqcjr!=;C%N1olnZi%Hm{D}vyLcsA8GcIv8l^Q*R8#}a1m|Qr zaLdsWfY}2gcon$a(vIu^pZPLOf}7di@57eoQEjJ_Eg;vMsl)!<@-lL7np}$GH0-pd z(00~!eJCDE3jO5w$?WCF<;S5r$n-E{E{{Evlctk|= z`%yV(hLFm0!YYEMBig3)Fnz9hLb_gK9Jj9{FU4CcD?}3m=5Bm@y}fgIz=MeG2*?^7 zYEOs5)m Q4^V_#_kWtN*hK25&q==lUWw&%uUnblxVUla2Bg3SYmWgt$L)?%P!2 z;0a%K22(QuIOZOFKiMjljg6yM(2__?>& zc)R1BXJ141Qd^Ek!^78V=0hbXKSpgD~OarhUoNyYjljtjpn4f>&Clo$X7=zeS z|NID+_4d_SJW^wCZz1odhiV$fNkj;RpL57VwCuq@ef*hsB97nA6(x88I3rm+tHu1d zlVaXmUPmWv)mNYe@og|~;OQ78GnrYBXx!W6$w4sOE>gPXf%8g1uc0aFrN;-%r^ZeX z5lBuL)rd042qgNy8wOrU&OA6R0n5_xBBeIXk-Iw*5Sefpm3XU)qDal=qGxUA5IFNuItrCEFdpJ zyMHr8jK(H_g2OQ;!5X4v4J2a91c73F0{lOf3Yk7XQIDQCaaJe03+aie;N?t$qHfl{ zixp6mNuSlQs&`oNY&dS=sNqIvmVTMZvE2K7&8ljxuyINVl(8Uv9*2k6!CpOZ@TqAg z5vj8;$RWtmGeDiJx!1dTpq!F(RSF#eW27>c-_CFaIPeA|T?TsgKx(uuycgXwXfwQi zJAb$IU40ITSr-O%pw+O|a4I3fikvuL>*en{268zz{@kglw8Z-!L7?Qp>JbSzdu$$+ zKyfq66nHouO`N;N%O|wXM0`BRWYP)+~4a! zdXA28#3Bud8;ynA?RT(zH@@#@Z+OrL@Q7iUhO+7S20Njh__8)gLR$f$jQnD4R>*@w zulUhvf`_}johHDl)Wr59KE=%j`akT>*M7cpH5@I%c=%ai`;|`g0ok!o`pF}MG->{*owg2;~4j6p~CjzaUqQc;1hFG*!vPqiFNgmCwUXG)#oKz8lC9~@cq1u!ngyrLmbQ}0{rHO zS8ty_Gq>HWUSbrk(%d;+c)8mKL_%R~8Wnl=6If&+w%r_(e3J|7uzD5vTD`dNebXDu z;hcIE(Wx}%AXSEXaxlsVwK-TNbkS-s`k2Gb-4{A_|!9 zjt9`GU@YWQ=lv2FE`cem2?H(-f&-wnjgZ4~t95WY!iHdK`kfswD97R2O`j zi~@!rQYhy9rltNP?L@D7oFuSqh<%DLeLsWQZ$HE8kj}5FC}DGuYD3fNoUliazOaUX zOZFCSgx~MVrdI8bt!2L-_BFX5Ac+y`OJwJUiEAb;>MKEXg{ZQ&^9+9GN+IgY+lgk= z99}7{G?TBp{j`{E0;e&P%!PbY)7ZS(l@6|!s0=~Y7%sA5pj?g|oggSG;fPZ0~j4tZ~ za!-{dG3vpI2Kq)LVsbWX;4xMRnL64Q=8h=>K2?w7G-{F8?x@3T(4$R#fPq{}_}QYg z@QMDC;7Y(|f5wKr6eLuXRJcMuDW~AE(%^9?xA%5Fs#ln8|dn4ToKf((W+#yleN(xuZ^UQe1 zc9izP$EbM#=H$C{Sy1Doh1_;Y6$;{Ce6J6zAzynXin~f<3toS6Vq=^Jbgr;Ko-t-n zbH>7A9XlO+vj``1$)~ebE6~7Gec)j1bE8PBqtu?W{=<8?vw$6aOK=m5>cOU+B3O>U zDj9n5k3V_utP{BZC)P{MVLCb%1^gDC?yDeU&UO+cOL)0#34ZOUKn%JRE4U}Pt`;k( z9&O=?ySI9pxN=GfInD+0GR6qYC{(o^m4c|yXl#G7_o^dM_}Z}!U-&kFJ(((vdM) z<;6;8>~n^c3R8IS<}gq*VYHGw5Nby8^8kpGgdB1E?bK6=5AIR_@%+)^^&NgQDI7$5 zq8T^S@Q?R6gg1+2+IKj}T?~ZDi?AttaKXC_4lA7`d6@APFH@l;LYPDOG>;|+v<2ar z=LU&jD@|9|^P?T`p zl60q`PF2ns!YN1!!M#3Hj3$g%_m@J@0zbEC53-3bKqI{`KU!uL!;`OEv;vGNZ5EF~ zCpQ9>HM`DQKU0h1FqbTavEaqF$pdh(N$kG3Jf@-$ZbBsD;JB1tcPcNNEWety72REU zba@o5;W-~7#z=HtQ3A+RINU1V&>lnzl09QEL%UA_=e-DNxz()TGnujGCE}0usF{UT z`e5%SD>H)kQx#dX!(RBfwnkbMYHms9@TBYHV%fM(_Yiis_vO_F=vb0e*#UW^+B^ob z4i(|uwrK5JqE3|Dg5?~p4+%CRSjh*$y=) zg3%kA^Jm3^FX&4PU)+k2FU5rmOn(WX=R_nYeTMGb75RQgkw6WTc?oB1#FS4;2-S?E ztIiuy=d0q7guqaTI`hf9{4d28v3#;gcFf*4j!ai{Fbg{E1yZN0lQ)J!MS09iyLh5K z4sK^jWT!S5t$rFdPTMXW0~D*Avuf)xpgKF;=_%kR>dadIIU_s2jSklVn?vX5!R zo;G$i2y_Dhf0%VxS)J)njy{mv_Lw)W71T1i9?9mXeb9CeR4yQgnJ)B{52H(HzQ<1m zjL#ZlU)FWu&`sgYA3XUbqqR5i0d7arz+RZyj`*8#>J-yJ);u=!o*lEufkuXQ{%q#` z_PhjMEA5Oaxrjnw5F4$qLVXSRU$vJlC_ydKP=rzTN>~h9YYNSwR`N*;SU9gZvK(iJ z$I)tqndB;-^aXp)qrRkcuI|jW8m)F^9n%GkKRbiQ(Oua0<#10nPbXRDqn%(iBe}M3 z8Fq|WP`u?Zc+@E~9(nTALC916_R<;0h}-NIh}vM(lc2JKCgA6A_dtRyzB!mMw@kp| zNTIm;xc*L;E6Ka5YZ*Pj;g%X}uvOPDXt^u@eL*&&c01x}#UqnXx~x3s+S%g)=Fw36SdNM70n3wlo0wtea&kEKJ#A>2h`dS9{_qj}9 zgTu0zypR)uebUSdI5swGWQc($XTH%Tgw2eebos#M5YC3U)QnP-EzrxLO57c&0qAP;fLZ68Br#g%NCa#xm|xoST0Jb-kcyL;jv+U@g&F1?cSl$iSa;SSH^9{>2lh8_!sVp*VP)eOt#~7+M>aGQb znmO84b>+{k=tQcHlUqGa12ud}VQSF|{6J%&<%}&Cm0cm-$x4oEHjF{Jd1>;g=g1c^ zlqy_frD_|F+{zM-VAHI+BNX6OsjPmH_{U}AQv=YLyYtqbG2Kt=pFHldL$ANOZy<6Z z383k!HMOXv&gigVrQ<1pBxWOWn76Az5M2eiWejH&fRG}FqUY@Q{kSiSg9{hhf4Nlg zBQpl`S(Eqy*paSp)~wwIb|@wLi>TV`5(?+YR+6NhOyT$AhoHa+a=fTrP6?p0{`C3! zgMkkwRJSEdYUY9jB|dVR5!(fl5{a={fIh^K@c}XKtq63ji^B|ew<3FcTgW-CEr3u ziBw#!yB{JAr8R~V)@T7`69Ug95$3IO@5H<;5~F5UrnxrMwEIuhrT{gkGY9f%Nq2%7 zP!TK6!rFl84AHvMG~?>C0^XI=;Y-bBb1XScRp_ksyE8v~U=$;T|@|VC9zr z*I8=cB_)61E=NL#IQICMDB&U~Sjw_#D<=JMJwLTDo+c)j((B-ge1N2VV%q#7-jakeZ-mzzQt!&SdOI2?_1d$snzTJ9C;V)wV0O_kMwKTwRra!v5~`c|~?)*X`^_Ilc!(Oal!^TX;l zG>w!$*X$fjHr+_HuZMZ!b^`gGi?AU^A9~7f z#nlNsQixHqFUmWRt#MPpz$pTDw)La$NXM^{R)CY6u!^p?t)9^p7R9 ziTe}>F)5Y*?DoKq-R$mKjubZI0(Xu?>Z_(lOU7ApwG(pF zrFF0mkO%N7E+da|8Vp8PB>w2W;!E~@Ey{b&)-}xVC&<4h`#z=67FSA9{t7Tc3J7?) z1X(OE-f=nF1yy~cid*upry0U4Wpcd=To6yM{WzQX#9)6~;rtaK!Eh&4zXE$Iwrk0$ zeS#3cI#Ua!9e2lRoWJ@+yXLopkXhL*Oj@lFoR0Q@HXS9P=};dGT!LyOwyH-!X4vnl zoeXSIHX*#la0s}w%_$dZ$D8CT_1Vj<&Qk-kUz0w;h*>#Uc~GAc!fEix96+h|R5vIx zfU&ng-Y}b)TVPC)0k|vkt7T+Hf2^c6596JqHGmA6@3mr=M6mp&!J=%dE~Jb?tg}uz zTAn8hGmP9Fzyr_LZmB!JS_?cSx0UyWAL}d`7th;unV`qw?=sQ;c^@gB(KPe2pefcJ zDElOZPlZ3QVbU)@!w$q`S^xtc;#nzkpcj4195_)-+&d^fkSXJ|vu_22385D9+#>Vu z9^PH!#)Sxx(G}_ucHR7~CR}fD@2m5GlpAQyKB(6^oE%V}2r2lV0K|RaZFEDbD}`)G z+6L64cyu4~YaDcQO^^5%$*mnjzTS(h)-7ZzRX~|8IG~)Ll1PA`)||E?IQKyw1W@s! z`hMbDo&#lej|adj?nsHPfl#DryldYBxZY+wLfAnON!+i;oC|I%wNsoRFmUJO#ef@7 ze>Eb*K*4{GRD_D*>0^w#=D68@M<6(KB!Rlmrf8TUzU#eJb z`LH_cf};N^K85DH;aoD&K&pkZ`t&ZI@y-q4p`*-jgqx3nARa96r0#j!+4pBh!0no> zZKpY?n}tS&xmt|}NLD6+7C3sixe0jWkq|+2WI7dI|+08)Itz{~Lt- za+sV$0g;`P>z0(zt<8xBh!6-&cyeeYQcd=?!$C4uU|xG%Pd;;Yb$K1s+ehYCT{0gp z!jxESV8#%^_2Uct6&j*AldIApX}4lP9ByU6;{hNLi7l#}0|XWdTLgg>J?F>Y>-84F zHtV@wD5xT^>z0vS3P~CD`G!oqK#9;`Z1#=7&fKNv$hTLY-t_o+^neq&rzBA;aPt&k z(#SvLeM2jt#3xM64N|Q_rG~M*T@vgz$tJ7L$r~lf>V`}Rsa|vm9sQtU*d4dBpHNkS zH==>I3)~73_+=9law)^HyGqa0>Ulg+{Rj>;I)(RltJ$b7)UJ5r&6!y&OXSZ=89%P) zO^HRP3{S7y+I4(J9nYmaa}rDDMz{1u`qO9cqfS}d5BIkEo}dI-E>F9>AML?DcG<1p z^5ED}=t(~{2mJy})Ry+*7hYH7hNC(@bB?d6VhSh1p!8u!!HS`<5#u^q;2BfaDO|Id z(&m(8c*uQap`o;KSyy5_U$yx6oh@P(!KrS+t+)vwaRrE)jO#s%H8BMSOdSMc=scj~xiW1tThnp*@(In>fABjvA zEH7Cx-CIt2dH3v4LH73>3_$jKuM<<)=hC+jubC3wA0NGB5N0;dYXdh<1Vk={qH!K)9D)D*|$s*CeC{1s0$R6z$Onq?u{st-i`w z&AGL7P^g38Clzu+tjD&T8aiiHM@b|XsWG&bgHbtX&Jj^Ym!J$n#prH$!|lNKc37BZ z(fpRs!aL4!3Bby17uxkf2H9i3e}ZLT9?He{#+#eA39U^O=p8!utv+zgyy*TYkLUi= z*=gUfW0U#z;ghKJD1mW3XNM2t4hsuM(_UV6l9!##RaRjf<9)m+;1ky&G!5!}ip6#! zTYSSALDO}oBOw&+Tf1%tsygu*0og}x=ArsEZZ_N=Hz7Un5gX2^tIfh)ge~7XU^c1mr|@ayrV)_ zwcN7|fff^bBRf~irtJr$Ix6HoOa>2jmZ=c0uQY(bVTe2Q$9aAc_t5R$m+|Yc=R|yk zVjwupIJZoD`R0$S*!U8TR6Ynyqks}VBfpJ6F^*&;d?VvXu`@wvd`0WBYtQ5N7x})z zGPAn?wZFE1-*84C0&SepS*vJ{c4|DGycjPpsW5cXm{2*8Sv}PlhDflKP5vsVE;+#J z0gUInCYyqBSzJyU(>OE=oo2qde1mwGMFVk#3haHx9>6V+%&-Ryu5@l&p$gc>pD6q( z701_S0i4Q*A0^QL2t2qM(lVy^aT4xQOVvYuVf^(9>{e7f9!!M|NYkSQ4zFtS>AJHb zrmaw1I$LVjm-4tf_-6Zh&I&td(;<5FTE_kN0qxv=0D9!F83}#g{*zF&Yj%S5wJ)t* zas=Vj0Dg|}D*^Os73}-$xW0#a1Wo{n*wkX}VJf9Wd_<~fVEabrd6FvNMOXz)S(`ww zChiS7ECdGu;uwvka&aA)r)tQX(4r#Rkk{CAm!5wDco1tlri4igWas4@zbrt5x0k!; z6|rTHQWw9^FVgaUFEuu9>ZngaLCmwKb3`U=0VvR4_t(al;)|gPiFh5?NBR$mtV2zS z%f2UI08UB2dXod`(RZ45HGJ4@7{b}dK71sHtmP*m`!>+^Y$b`iG3!)$TfAr7RUX{( z2NPFLf23OY_@;-ocWE^#P>R+1cbx2ioAsmnbXI13X$LXtD&wk zD>I0F&jx(RDYgfEs&6S|gyTMd80#jFLNLE;rr!AG(Ez+{7p8o?RNg>jyKGLyRGqS5 z?FZT>SA8qgfP28+R-QG+gR)!Zk0n&NxO6JVp!k?lXn__^J~$zUARA+8dT*|NBhYq> zIWVVRD7zWa=%1Q9A@8qg1VZPgSP)*SNYMXII*{ZZVKarL|;QM45WIk1r>40kGetxpbMK4WQUpgAtiwTLoK-pl9?0*x-ud0X$_&& zLeN95e@|iBKuE;lV&KzzfDs($MWNy*9Wq)8GNjIypNU=o;qvr@K6AVn;b+$x^Y%Lk zK9;mcshmEp#(lt&nZ1U}y;ZMI1V{w4xQ}t+c4(nAFQQzpS8*RjIUwVz4O z{X0xi@y!PKoe*3re}ax*=sJI+RzJTI5(R7@u7O1Lt1n=(a<>4|IucO<~T#Gq>@)o`yJb#Uj6#k()4%7eC z%T$gY7ydYXxAe}2`6hW_0Z7}2Rhv*&OQBc6B8Zqa3k zB0(x#Gyyo~eQ5*Ms=s6uzy5BL**Uv;c8gDQL;nt`e|`HfTp@WlI!$g{ z44t(qqf~-X>JTx{-v;GjvS-Gkz##>nNXQ-4XY|VGy?}$MYWmx6_uxPO+2`|#^Xz{r zVoLU|#8V9yu2iP#3&XuYMo0^1F{04~=(g!{8{cog#y-Ccqjw0FwasgCl=YuIt& zzVAl}aX{1h-j}yL`P78=z5+E?B&7R|&(!7p&uJ&l@=`>n-Iv$NLErEwZy0_S6&!%) zEnJZVVyLbwMM*Nweo-C^m?Ef!qi)0x*x?_$qE`=$MQq3`$8(Tx;h^&KI3F(D;3g=f zz|Y{#8kb!(v53sw5Jk841AjWr(5tSFqqhMC>gUS-KVQM4C{*iR%N4N3;d$6Wq#`>B zZ-*IcJ`?QfT|mI{5hWgJ7OPG#vM90Av}6T5nq`3g8!JJat^d35e}D&Kgqnc+EsajS zFW;DrlxfB&YQ!T9*ZSjNvJ&?p0uLLhaCjc7LS!jzyDn8M_+svVu5L??cLN+bo(czv z>N=+HDR${#ihsTi1S{tvj?laC^^f`n9rg-dD}YwHL_UZED=R? z%6)mSY@}2^3HQ3$km0bU!GZzcVNssD0FESS9Jk+A?%0gdJk`bIO-nzd9I!teHL3;j zTZoJIAJgY_RaPVm`7kvNBYzOkW+yP_cTg$K=%MA#vng)rzTz?veYqeh70`nsqp)2SaEp>`W|4#7cV zg4fF!YYufiwKDtt$A$XO52*VvQmi)#=wq{|vdvE$jb8=h3fmM^@ZFh@-3=4a5(jSz z&WYDV7yP1&hb%6`tCS((Y(cON?-WPk4mixfAqKkFc~}A>_VAG%`K)nw5gFV@pZsy}ujI7(1%!m%vVXFFc&cXxP z>gV9P3h^TYE0Ye#jZTe5xNNp$H2`gE>AVTG)u0Ahh$ZM0q04FTUu(XDFw08;9k;sm zHWc}Ah$9Chr!kmF6_p=6WZNjd02qTM7tFv*_Q0Ka_@zB^{3UBQP#7toc9*vIp?)mi z7-qW*9nFUA|2}$ecn5KojsiM%bqRdJ{UO>MPq{VPz*8xU_=7?gl@g?X`Enzk_wS@R z;32G`?e*ahE}w8jT6)4-zWJbWVk3$R>h!h=3I)11Wmz786XM?v7IiH+z1Lu2Hl~Yt za${|kpuC~K$ZK#c+#j9Ope~`?m|JdOTb~vW<)K`X@Sfi^u3A?ZNa<{DP;?-WSp_^A zWd$I9KI};F6d$-`$* zwk1Xq6+t^>p>_RwJ2BCTmaz0&7;@wVLe2I-O!lE30&UlxP;iC zKHdPkqvrWZ#{9=Ks*A$WQLC+;=P+4~3h0DYtYV|dh)Ry_gbuiD{2UrxJo|Mt zBCvJh=?l#*B5WVmFr6CkHlE*=yj2gLWfL%nY_C~XC65E7F+ch{zxN*>MY?S716s+| z0O)PF9)n!lDZe*=O?t26Zlh|E7*3)@0KPvFz6Z*t{}GHiJqg?x)O519z9?i*-WI=~ zB#15v)bTq`4~GIv06fhD-qjjp5d_GOR=NKf8C;}*ECf%Hh{jDobeS9UqD(hw@Ql=% z+6lmS&%yU$J5IfK9;P!j8rOKg5Tej=FTTcan-a5uf?S*5Z^tV0(>fwLh%jo4xhk%sMGIMJ_|0_5b8eguASxpVg6o2{zjC%E5NlS zQ++IJRv3Amx1T{s?oRyB&r!Rs<^gkSIMyF<-R^k}?)JgPj)(ja%7r@efd-18(_ z$8lFu4PqKh00a09$WOiUGu6G-c5mbVSj9E?05IJs0Kf^a>%Q?(??cyo8B5oR%7nQn z3RDW6^!4qN50#@S4!WZi<=!Fqo@__4Dl+?Tl#g`T!E`9ooyS#!F>??q-h!n)dj}XF z2dQY=H#)XDFfO2ybOJ3sYoUfK7NExH$@yPLoNVF;(7kCAFcc?2I0AM3Wu>0CXq=UA ze7Ya-LVz44S5SlR#P(IsB68f77{groA$a!zxRh@&@_=mo?70z6F_+(9B}vkJ?P4FP z*~!ln?Kw|Np&T<$7i1e1p8{|-N)WWQngBSWOC_j=Re{b5iW8CG-2B&o=ffK^R;Us> zIhd^=S(x;m+f11~d>nmN54{_s(*z4@=V@o^p8F6y+mclVH#hnce~FjuT-r!fNV3y0Ub2Tdl#}C4XG0Xz_L<>LUj+ zcxO_-eJH*pq-%LUk*^XSvQen#8&F6txq62I;wn+ICm98^lYc0_cX9XA$0Y0ejVp@+ zo0%xvI-Q2EQ4XjF-?)_jvRJ6tw-ZFZUTu83-C^hX`iuzqr*mgSSX%EVfqVj2Zmg*U-_x8{Fvyl2jk5LD;o_c1@RYmu zUl~u93LGbz0G*OOwRzYf=vhn`_=1NVPhPh51t0m>;tw*v;gvXy@l3P^rE0eAl9=IR zB#n%5Zt~m*;0&SaJI2mAZ|QV8*%6wvfr`UBY$U-#A1X=qI(ma6yWFM!>j_QI)vfcs zNB8Y|_vjo1;NG3NeAcV!SrEQ+)4lU}YQ3$tg3`|GJOo~-@*TwX0|+C_JogGG=8pB1 zvkpzqWYSA4tXhj;?-m?sCbJ?4kh{vBC*eA0PYom+^R}WKUK+6LFn(u?{qL%SACal9 zSZz!`bLohpI9YIMytfo(gf98SLH=&1r7Vw!c!asK1m7+kp04?`*kf~iD?QMSQ6S;`?*h8iCE9#7lLsp*gfv{9+f=wevH|*&kYIYp>t3 z;(XxiDbWV%I?=N~ZCI0=v+?iY;vZL+KGs@qEy%EHcAu=E!-7VpsvVj;7TuX7*HFng z{ezpNHT1M|xeaPJ= zi$Xc}>NRrt>A+RvZ~14(HJ>U~XL2aLyiplgoJKhCTe8MXY4_xJT>3#VBbBu@gs*t& zW7p%Mvx~HUCq->9nz1^^^EZxhj0L|rj{Kdl3$(aHyZxY)w)XjGV3}FDr?)Tjc)(4u zc%Su-Ps>!99Ns=kP6t*a8ye(FxVCBVPnK&FYuVVII755K;#`xj(Gq_)CiJt`>x5VD z`LXPaOqt%+r%R(=fBpvAA%rajS@|oxEACl;4ff1b?mHU&r&on@{c5>=YvZAq z>x-)n3o$Ne3oKFq+Bw1D&xGK+O{P|@rye#3rrO%*a)_`s9NHwStk3bJ{QR85uTFgi zsgmTtnX3QMc9THUOJkm*z&FPz z%QI{L8=krPe+4M=u0PH_lpGP+p=GYxI(Lq*B_lg%a{rcQ!gfGV`l zTKWaT+aeh}auY4ruQOA0l=KekYc>fB&pRlS&HXwN)G#`? zblWoZ*MD+;by|RCT`w>&R;RV@WX;!lS}35AA0HcaBX4?EZkRguo>b#IRROl-H8Rh% zo?pEhrUE=?P1|zy>N{6lo~H#@T3P_ptLK4vv5D)7_PqE~7#*?wYvs;cQ?}UR^89=U znVyYRaqG72an-7PTY3CNfTNX}^_1C@fdy$Z>l;b2XN8ep_n5_c-H^NXc+)FfnRp(P z*qV!XBQ5pAqi^JW-J$#5BJ1^ql)3X|tAhO6K+XGz1M^ZVD=(VuR=IueRbpIj><(ER z-Jc9!M(%4*VqTf6-Br1c4t8?Azz^m zW}AnZtJZRF&t7RXcaHAY!U98Js#(NjqaFRF`Qaw4d2&VY0&lj`S9`+CAS?Z_c9yW0 l99YdmE3}E{Hkxt#sZWW@Qr&Q0=m!H3c)I$ztaD0e0su%iQ9J+u literal 0 HcmV?d00001 diff --git a/rustc.rst b/rustc.rst new file mode 100644 index 0000000000..90ab7df8b4 --- /dev/null +++ b/rustc.rst @@ -0,0 +1,16 @@ +================= +Cretonne in Rustc +================= + +The Rust compiler currently uses LLVM as its optimizer and code generator for both debug and +release builds. The Cretonne project does not intend to compete with LLVM when it comes to +optimizing release builds, but for debug builds where compilation speed is paramount, it makes +sense to use Cretonne instead of LLVM. + +- Cretonne is designed to take advantage of multi-core CPUs, making parallel code generation quite + easy. This is harder with LLVM which was designed before multi-core CPUs where mainstream. +- Cretonne is designed with compilation speed in mind. It makes engineering tradeoffs that favor + compilation speed over advanced optimizations. + +See `the discussion on the Rust internals forum +`_. diff --git a/spidermonkey.rst b/spidermonkey.rst new file mode 100644 index 0000000000..57cbccc03e --- /dev/null +++ b/spidermonkey.rst @@ -0,0 +1,44 @@ +======================== +Cretonne in SpiderMonkey +======================== + +`SpiderMonkey `_ is the +JavaScript and WebAssembly engine in Firefox. Cretonne is designed to be used in SpiderMonkey with +the goal of enabling better code generation for ARM's 32-bit and 64-bit architectures, and building +a framework for improved low-level code optimizations in the future. + +Phase 1: WebAssembly +-------------------- + +SpiderMonkey currently has two WebAssembly compilers: The tier 1 baseline compiler (not shown +below) and the tier 2 compiler using the IonMonkey JavaScript compiler's optimizations and register +allocation. + +.. image:: media/spidermonkey1.png + :align: center + :width: 80% + :alt: Cretonne in SpiderMonkey phase 1 + +In phase 1, Cretonne aims to replace the IonMonkey-based tier 2 compiler for WebAssembly only. It +will still be orchestrated by the BaldrMonkey engine and compile WebAssembly modules on multiple +threads. Cretonne translates binary wasm functions directly into its own intermediate +representation, and it generates binary machine code without depending on SpiderMonkey's macro +assembler. + +Phase 2: IonMonkey +------------------ + +The IonMonkey JIT compiler is designed to compile JavaScript code. It uses two separate +intermediate representations to do that: + +- MIR is used for optimizations that are specific to JavaScript JIT compilation. It has good + support for JS types and the special tricks needed to make JS fast. +- LIR is used for register allocation. + +.. image:: media/spidermonkey2.png + :align: center + :width: 80% + :alt: Cretonne in SpiderMonkey phase 2 + +Cretonne has its own register allocator, so the LIR representation can be skipped when using +Cretonne as a backend for IonMonkey. From 3f69581d03ca4974ae6c6d3a7f1a58a1f4050aba Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Thu, 25 Jan 2018 17:11:13 -0800 Subject: [PATCH 1486/3084] cretonne::Context: add for_function constructor --- lib/cretonne/src/context.rs | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 221481a039..8aff5aa706 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -49,8 +49,16 @@ impl Context { /// The returned instance should be reused for compiling multiple functions in order to avoid /// needless allocator thrashing. pub fn new() -> Self { + Context::for_function(Function::new()) + } + + /// Allocate a new compilation context with an existing Function. + /// + /// The returned instance should be reused for compiling multiple functions in order to avoid + /// needless allocator thrashing. + pub fn for_function(func: Function) -> Self { Self { - func: Function::new(), + func: func, cfg: ControlFlowGraph::new(), domtree: DominatorTree::new(), regalloc: regalloc::Context::new(), From 584a33bca77a05cad59b99e408ee8f9bd6790776 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 29 Jan 2018 09:16:33 -0800 Subject: [PATCH 1487/3084] Give better error messages in "test binemit". When an instruction can't be encoded, provide a bit more help: - Detect missing register assignments for input and output operands. - List encodings that where considered and rejected. --- cranelift/src/filetest/binemit.rs | 85 +++++++++++++++++++++++-------- 1 file changed, 63 insertions(+), 22 deletions(-) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index c51f4b613d..aca308021b 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -7,6 +7,7 @@ use std::borrow::Cow; use std::collections::HashMap; use std::fmt::Write; use cretonne::binemit; +use cretonne::dbg::DisplayList; use cretonne::ir; use cretonne::ir::entities::AnyEntity; use cretonne::regalloc::RegDiversions; @@ -137,26 +138,27 @@ impl SubTest for TestBinEmit { divert.clear(); for inst in func.layout.ebb_insts(ebb) { if !func.encodings[inst].is_legal() { - let mut legal_encodings = isa.legal_encodings( - &func.dfg, - &func.dfg[inst], - func.dfg.ctrl_typevar(inst), - ); - - let enc = if is_compressed { - // Get the smallest legal encoding - legal_encodings - .filter(|e| { + // Find an encoding that satisfies both immediate field and register + // constraints. + if let Some(enc) = { + let mut legal_encodings = isa.legal_encodings( + &func.dfg, + &func.dfg[inst], + func.dfg.ctrl_typevar(inst), + ).filter(|e| { let recipe_constraints = &encinfo.constraints[e.recipe()]; recipe_constraints.satisfied(inst, &divert, &func) - }) - .min_by_key(|&e| encinfo.bytes(e)) - } else { - // If not using compressed, just use the first encoding. - legal_encodings.next() - }; + }); - if let Some(enc) = enc { + if is_compressed { + // Get the smallest legal encoding + legal_encodings.min_by_key(|&e| encinfo.bytes(e)) + } else { + // If not using compressed, just use the first encoding. + legal_encodings.next() + } + } + { func.encodings[inst] = enc; } } @@ -215,6 +217,17 @@ impl SubTest for TestBinEmit { // Send legal encodings into the emitter. if enc.is_legal() { + // Generate a better error message if output locations are not specified. + if let Some(&v) = func.dfg.inst_results(inst).iter().find(|&&v| { + !func.locations[v].is_assigned() + }) + { + return Err(format!( + "Missing register/stack slot for {} in {}", + v, + func.dfg.display_inst(inst, isa) + )); + } let before = sink.offset; isa.emit_inst(&func, inst, &mut divert, &mut sink); let emitted = sink.offset - before; @@ -231,11 +244,39 @@ impl SubTest for TestBinEmit { // Check against bin: directives. if let Some(want) = bins.remove(&inst) { if !enc.is_legal() { - return Err(format!( - "{} can't be encoded: {}", - inst, - func.dfg.display_inst(inst, isa) - )); + // A possible cause of an unencoded instruction is a missing location for + // one of the input operands. + if let Some(&v) = func.dfg.inst_args(inst).iter().find(|&&v| { + !func.locations[v].is_assigned() + }) + { + return Err(format!( + "Missing register/stack slot for {} in {}", + v, + func.dfg.display_inst(inst, isa) + )); + } + + // Do any encodings exist? + let encodings = isa.legal_encodings( + &func.dfg, + &func.dfg[inst], + func.dfg.ctrl_typevar(inst), + ).map(|e| encinfo.display(e)) + .collect::>(); + + if encodings.is_empty() { + return Err(format!( + "No encodings found for: {}", + func.dfg.display_inst(inst, isa) + )); + } else { + return Err(format!( + "No matching encodings for {} in {}", + func.dfg.display_inst(inst, isa), + DisplayList(&encodings), + )); + } } let have = sink.text.trim(); if have != want { From 21f0fc39ad65d6333469321a11345b8e2aee04fb Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Mon, 29 Jan 2018 13:15:39 -0800 Subject: [PATCH 1488/3084] Further restrict Intel register classes to prevent incorrect encoding of R12 derefs. --- lib/cretonne/meta/isa/intel/recipes.py | 30 +++++++++++++----------- lib/cretonne/meta/isa/intel/registers.py | 9 ++++--- 2 files changed, 22 insertions(+), 17 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index e613e9aeeb..3b9b508ce4 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -11,8 +11,9 @@ from base.formats import IntCompare, FloatCompare, IntCond, FloatCond from base.formats import Jump, Branch, BranchInt, BranchFloat from base.formats import Ternary, FuncAddr, UnaryGlobalVar from base.formats import RegMove, RegSpill, RegFill, CopySpecial -from .registers import GPR, ABCD, FPR, GPR_NORIP, GPR8, FPR8, GPR8_NORIP -from .registers import FLAG, StackGPR32, StackFPR32 +from .registers import GPR, ABCD, FPR, GPR_DEREF_SAFE, GPR_ZERO_DEREF_SAFE +from .registers import GPR8, FPR8, GPR8_ZERO_DEREF_SAFE, FLAG, StackGPR32 +from .registers import StackFPR32 from .defs import supported_floatccs from .settings import use_sse41 @@ -104,7 +105,8 @@ def replace_put_op(emit, prefix): # Register class mapping for no-REX instructions. NOREX_MAP = { GPR: GPR8, - GPR_NORIP: GPR8_NORIP, + GPR_DEREF_SAFE: GPR8, + GPR_ZERO_DEREF_SAFE: GPR8_ZERO_DEREF_SAFE, FPR: FPR8 } @@ -623,7 +625,7 @@ got_gvaddr8 = TailRecipe( # XX /r register-indirect store with no offset. st = TailRecipe( - 'st', Store, size=1, ins=(GPR, GPR), outs=(), + 'st', Store, size=1, ins=(GPR, GPR_ZERO_DEREF_SAFE), outs=(), instp=IsEqual(Store.offset, 0), clobbers_flags=False, emit=''' @@ -655,7 +657,7 @@ fst = TailRecipe( # XX /r register-indirect store with 8-bit offset. stDisp8 = TailRecipe( - 'stDisp8', Store, size=2, ins=(GPR, GPR), outs=(), + 'stDisp8', Store, size=2, ins=(GPR, GPR_DEREF_SAFE), outs=(), instp=IsSignedInt(Store.offset, 8), clobbers_flags=False, emit=''' @@ -676,7 +678,7 @@ stDisp8_abcd = TailRecipe( sink.put1(offset as u8); ''') fstDisp8 = TailRecipe( - 'fstDisp8', Store, size=2, ins=(FPR, GPR), outs=(), + 'fstDisp8', Store, size=2, ins=(FPR, GPR_DEREF_SAFE), outs=(), instp=IsSignedInt(Store.offset, 8), clobbers_flags=False, emit=''' @@ -688,7 +690,7 @@ fstDisp8 = TailRecipe( # XX /r register-indirect store with 32-bit offset. stDisp32 = TailRecipe( - 'stDisp32', Store, size=5, ins=(GPR, GPR), outs=(), + 'stDisp32', Store, size=5, ins=(GPR, GPR_DEREF_SAFE), outs=(), clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); @@ -707,7 +709,7 @@ stDisp32_abcd = TailRecipe( sink.put4(offset as u32); ''') fstDisp32 = TailRecipe( - 'fstDisp32', Store, size=5, ins=(FPR, GPR), outs=(), + 'fstDisp32', Store, size=5, ins=(FPR, GPR_DEREF_SAFE), outs=(), clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); @@ -768,7 +770,7 @@ frsp32 = TailRecipe( # XX /r load with no offset. ld = TailRecipe( - 'ld', Load, size=1, ins=(GPR_NORIP), outs=(GPR), + 'ld', Load, size=1, ins=(GPR_ZERO_DEREF_SAFE), outs=(GPR), instp=IsEqual(Load.offset, 0), clobbers_flags=False, emit=''' @@ -778,7 +780,7 @@ ld = TailRecipe( # XX /r float load with no offset. fld = TailRecipe( - 'fld', Load, size=1, ins=(GPR), outs=(FPR), + 'fld', Load, size=1, ins=(GPR_ZERO_DEREF_SAFE), outs=(FPR), instp=IsEqual(Load.offset, 0), clobbers_flags=False, emit=''' @@ -788,7 +790,7 @@ fld = TailRecipe( # XX /r load with 8-bit offset. ldDisp8 = TailRecipe( - 'ldDisp8', Load, size=2, ins=(GPR), outs=(GPR), + 'ldDisp8', Load, size=2, ins=(GPR_DEREF_SAFE), outs=(GPR), instp=IsSignedInt(Load.offset, 8), clobbers_flags=False, emit=''' @@ -800,7 +802,7 @@ ldDisp8 = TailRecipe( # XX /r float load with 8-bit offset. fldDisp8 = TailRecipe( - 'fldDisp8', Load, size=2, ins=(GPR), outs=(FPR), + 'fldDisp8', Load, size=2, ins=(GPR_DEREF_SAFE), outs=(FPR), instp=IsSignedInt(Load.offset, 8), clobbers_flags=False, emit=''' @@ -812,7 +814,7 @@ fldDisp8 = TailRecipe( # XX /r load with 32-bit offset. ldDisp32 = TailRecipe( - 'ldDisp32', Load, size=5, ins=(GPR), outs=(GPR), + 'ldDisp32', Load, size=5, ins=(GPR_DEREF_SAFE), outs=(GPR), instp=IsSignedInt(Load.offset, 32), clobbers_flags=False, emit=''' @@ -824,7 +826,7 @@ ldDisp32 = TailRecipe( # XX /r float load with 32-bit offset. fldDisp32 = TailRecipe( - 'fldDisp32', Load, size=5, ins=(GPR), outs=(FPR), + 'fldDisp32', Load, size=5, ins=(GPR_DEREF_SAFE), outs=(FPR), instp=IsSignedInt(Load.offset, 32), clobbers_flags=False, emit=''' diff --git a/lib/cretonne/meta/isa/intel/registers.py b/lib/cretonne/meta/isa/intel/registers.py index 9411b23245..ef7b5c6d97 100644 --- a/lib/cretonne/meta/isa/intel/registers.py +++ b/lib/cretonne/meta/isa/intel/registers.py @@ -46,10 +46,13 @@ FlagRegs = RegBank( names=['eflags']) GPR = RegClass(IntRegs) +# Certain types of deref encodings cannot be used with all registers. +# R13/RBP cannot be used with zero-offset load or store instructions. +# R12 cannot be used with a non-SIB-byte encoding of all derefs. +GPR_DEREF_SAFE = GPR.without(GPR.r12) +GPR_ZERO_DEREF_SAFE = GPR_DEREF_SAFE.without(GPR.rbp, GPR.r13) GPR8 = GPR[0:8] -# In certain instructions, RBP and R13 are interpreted as RIP-relative. -GPR_NORIP = GPR.without(GPR.rbp, GPR.r13) -GPR8_NORIP = GPR8.without(GPR.rbp) +GPR8_ZERO_DEREF_SAFE = GPR8.without(GPR.rbp) ABCD = GPR[0:4] FPR = RegClass(FloatRegs) FPR8 = FPR[0:8] From 099b959d57f5b55d623bc489745b9e64370b6354 Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Mon, 29 Jan 2018 13:38:26 -0800 Subject: [PATCH 1489/3084] TargetIsa implies a Display of shared and isa-specific flags --- lib/cretonne/src/isa/arm32/mod.rs | 7 +++++++ lib/cretonne/src/isa/arm64/mod.rs | 7 +++++++ lib/cretonne/src/isa/intel/mod.rs | 7 +++++++ lib/cretonne/src/isa/mod.rs | 6 ++++-- lib/cretonne/src/isa/riscv/mod.rs | 7 +++++++ 5 files changed, 32 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 49987f1e18..c3f295d4fb 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -13,6 +13,7 @@ use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; +use std::fmt; #[allow(dead_code)] struct Isa { @@ -108,3 +109,9 @@ impl TargetIsa for Isa { emit_function(func, binemit::emit_inst, sink) } } + +impl fmt::Display for Isa { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{}\n{}", self.shared_flags, self.isa_flags) + } +} diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index f8e7737c31..d0644ae2f6 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -13,6 +13,7 @@ use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; +use std::fmt; #[allow(dead_code)] struct Isa { @@ -101,3 +102,9 @@ impl TargetIsa for Isa { emit_function(func, binemit::emit_inst, sink) } } + +impl fmt::Display for Isa { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{}\n{}", self.shared_flags, self.isa_flags) + } +} diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 3123e7f0cb..3f02890a82 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -15,6 +15,7 @@ use ir; use regalloc; use result; use timing; +use std::fmt; #[allow(dead_code)] @@ -116,3 +117,9 @@ impl TargetIsa for Isa { abi::prologue_epilogue(func, self) } } + +impl fmt::Display for Isa { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{}\n{}", self.shared_flags, self.isa_flags) + } +} diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 1fdaa79f09..b4227d2550 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -53,6 +53,7 @@ use regalloc; use result; use timing; use isa::enc_tables::Encodings; +use std::fmt; #[cfg(build_riscv)] pub mod riscv; @@ -146,8 +147,9 @@ pub type Legalize = fn(ir::Inst, &mut flowgraph::ControlFlowGraph) -> bool; -/// Methods that are specialized to a target ISA. -pub trait TargetIsa { +/// Methods that are specialized to a target ISA. Implies a Display trait that shows the +/// shared flags, as well as any isa-specific flags. +pub trait TargetIsa: fmt::Display { /// Get the name of this ISA. fn name(&self) -> &'static str; diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 790a788a24..2fd0d5cdb9 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -13,6 +13,7 @@ use isa::Builder as IsaBuilder; use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; +use std::fmt; #[allow(dead_code)] struct Isa { @@ -252,3 +253,9 @@ mod tests { assert_eq!(encstr(&*isa, isa.encode(&dfg, &mul32, types::I32)), "R#10c"); } } + +impl fmt::Display for Isa { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{}\n{}", self.shared_flags, self.isa_flags) + } +} From ff16583c595423ae51567e11aecc22b56a1a4de8 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Mon, 29 Jan 2018 14:12:01 -0800 Subject: [PATCH 1490/3084] Remove RSP from deref safe register class as well. --- lib/cretonne/meta/isa/intel/recipes.py | 6 +++--- lib/cretonne/meta/isa/intel/registers.py | 5 +++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 3b9b508ce4..313ca7abd6 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -12,8 +12,8 @@ from base.formats import Jump, Branch, BranchInt, BranchFloat from base.formats import Ternary, FuncAddr, UnaryGlobalVar from base.formats import RegMove, RegSpill, RegFill, CopySpecial from .registers import GPR, ABCD, FPR, GPR_DEREF_SAFE, GPR_ZERO_DEREF_SAFE -from .registers import GPR8, FPR8, GPR8_ZERO_DEREF_SAFE, FLAG, StackGPR32 -from .registers import StackFPR32 +from .registers import GPR8, FPR8, GPR8_DEREF_SAFE, GPR8_ZERO_DEREF_SAFE, FLAG +from .registers import StackGPR32, StackFPR32 from .defs import supported_floatccs from .settings import use_sse41 @@ -105,7 +105,7 @@ def replace_put_op(emit, prefix): # Register class mapping for no-REX instructions. NOREX_MAP = { GPR: GPR8, - GPR_DEREF_SAFE: GPR8, + GPR_DEREF_SAFE: GPR8_DEREF_SAFE, GPR_ZERO_DEREF_SAFE: GPR8_ZERO_DEREF_SAFE, FPR: FPR8 } diff --git a/lib/cretonne/meta/isa/intel/registers.py b/lib/cretonne/meta/isa/intel/registers.py index ef7b5c6d97..354001a102 100644 --- a/lib/cretonne/meta/isa/intel/registers.py +++ b/lib/cretonne/meta/isa/intel/registers.py @@ -49,10 +49,11 @@ GPR = RegClass(IntRegs) # Certain types of deref encodings cannot be used with all registers. # R13/RBP cannot be used with zero-offset load or store instructions. # R12 cannot be used with a non-SIB-byte encoding of all derefs. -GPR_DEREF_SAFE = GPR.without(GPR.r12) +GPR_DEREF_SAFE = GPR.without(GPR.rsp, GPR.r12) GPR_ZERO_DEREF_SAFE = GPR_DEREF_SAFE.without(GPR.rbp, GPR.r13) GPR8 = GPR[0:8] -GPR8_ZERO_DEREF_SAFE = GPR8.without(GPR.rbp) +GPR8_DEREF_SAFE = GPR8.without(GPR.rsp) +GPR8_ZERO_DEREF_SAFE = GPR8_DEREF_SAFE.without(GPR.rbp) ABCD = GPR[0:4] FPR = RegClass(FloatRegs) FPR8 = FPR[0:8] From 429027e2f2b69bd74395960bb64964edfc3fbb25 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 31 Jan 2018 16:17:53 -0800 Subject: [PATCH 1491/3084] Stack alignment is 16 bytes on x86_64. Make sure that the stack frame size is always a multiple of 16 as specified by the x86_64 ABI. Heads up @pchickey --- lib/cretonne/src/isa/intel/abi.rs | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 965283d82f..21442250c5 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -180,20 +180,28 @@ pub fn spiderwasm_prologue_epilogue( func: &mut ir::Function, isa: &TargetIsa, ) -> result::CtonResult { - let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; + let (word_size, stack_align) = if isa.flags().is_64bit() { + (8, 16) + } else { + (4, 4) + }; let bytes = StackSize::from(isa.flags().spiderwasm_prologue_words()) * word_size; let mut ss = ir::StackSlotData::new(ir::StackSlotKind::IncomingArg, bytes); ss.offset = -(bytes as StackOffset); func.stack_slots.push(ss); - layout_stack(&mut func.stack_slots, word_size)?; + layout_stack(&mut func.stack_slots, stack_align)?; Ok(()) } /// Insert a System V-compatible prologue and epilogue. pub fn native_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { - let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; + let (word_size, stack_align) = if isa.flags().is_64bit() { + (8, 16) + } else { + (4, 4) + }; let csr_type = if isa.flags().is_64bit() { ir::types::I64 } else { @@ -215,7 +223,7 @@ pub fn native_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> res offset: -csr_stack_size, }); - let total_stack_size = layout_stack(&mut func.stack_slots, word_size)? as i32; + let total_stack_size = layout_stack(&mut func.stack_slots, stack_align)? as i32; let local_stack_size = (total_stack_size - csr_stack_size) as i64; // Add CSRs to function signature From e3714ddd10a8864a7557baff858f38b2f3042474 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 1 Feb 2018 17:10:21 -0800 Subject: [PATCH 1492/3084] Add a func.inst_offsets() iterator. This Function method can be used after the final code layout has been computed. It returns all the instructions in an EBB along with their encoded size and offset from the beginning of the function. This is useful for extracting additional metadata about trapping instructions and other things that may be needed by a VM. --- cranelift/src/filetest/binemit.rs | 5 ++-- lib/cretonne/src/ir/function.rs | 46 ++++++++++++++++++++++++++++++- 2 files changed, 48 insertions(+), 3 deletions(-) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index aca308021b..9b18a439a1 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -211,7 +211,8 @@ impl SubTest for TestBinEmit { "Inconsistent {} header offset", ebb ); - for inst in func.layout.ebb_insts(ebb) { + for (offset, inst, enc_bytes) in func.inst_offsets(ebb, &encinfo) { + assert_eq!(sink.offset, offset); sink.text.clear(); let enc = func.encodings[inst]; @@ -234,7 +235,7 @@ impl SubTest for TestBinEmit { // Verify the encoding recipe sizes against the ISAs emit_inst implementation. assert_eq!( emitted, - encinfo.bytes(enc), + enc_bytes, "Inconsistent size for [{}] {}", encinfo.display(enc), func.dfg.display_inst(inst, isa) diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 457b47a5e0..91ee7eca75 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -3,13 +3,14 @@ //! The `Function` struct defined in this module owns all of its extended basic blocks and //! instructions. +use binemit::CodeOffset; use entity::{PrimaryMap, EntityMap}; use ir; use ir::{ExternalName, CallConv, Signature, DataFlowGraph, Layout}; use ir::{InstEncodings, ValueLocations, JumpTables, StackSlots, EbbOffsets, SourceLocs}; use ir::{Ebb, JumpTableData, JumpTable, StackSlotData, StackSlot, SigRef, ExtFuncData, FuncRef, GlobalVarData, GlobalVar, HeapData, Heap}; -use isa::TargetIsa; +use isa::{TargetIsa, EncInfo}; use std::fmt; use write::write_function; @@ -153,6 +154,28 @@ impl Function { self.dfg.ebb_params(entry)[i] }) } + + /// Get an iterator over the instructions in `ebb`, including offsets and encoded instruction + /// sizes. + /// + /// The iterator returns `(offset, inst, size)` tuples, where `offset` if the offset in bytes + /// from the beginning of the function to the instruction, and `size` is the size of the + /// instruction in bytes, or 0 for unencoded instructions. + /// + /// This function can only be used after the code layout has been computed by the + /// `binemit::relax_branches()` function. + pub fn inst_offsets<'a>(&'a self, ebb: Ebb, encinfo: &EncInfo) -> InstOffsetIter<'a> { + assert!( + !self.offsets.is_empty(), + "Code layout must be computed first" + ); + InstOffsetIter { + encinfo: encinfo.clone(), + encodings: &self.encodings, + offset: self.offsets[ebb], + iter: self.layout.ebb_insts(ebb), + } + } } /// Wrapper type capable of displaying a `Function` with correct ISA annotations. @@ -175,3 +198,24 @@ impl fmt::Debug for Function { write_function(fmt, self, None) } } + +/// Iterator returning instruction offsets and sizes: `(offset, inst, size)`. +pub struct InstOffsetIter<'a> { + encinfo: EncInfo, + encodings: &'a InstEncodings, + offset: CodeOffset, + iter: ir::layout::Insts<'a>, +} + +impl<'a> Iterator for InstOffsetIter<'a> { + type Item = (CodeOffset, ir::Inst, CodeOffset); + + fn next(&mut self) -> Option { + self.iter.next().map(|inst| { + let size = self.encinfo.bytes(self.encodings[inst]); + let offset = self.offset; + self.offset += size; + (offset, inst, size) + }) + } +} From 6f8a54b6a5d4c5f5cca8bad36fc19047cc106bb7 Mon Sep 17 00:00:00 2001 From: Julian Seward Date: Wed, 17 Jan 2018 06:23:30 +0100 Subject: [PATCH 1493/3084] Adds support for legalizing CLZ, CTZ and POPCOUNT on baseline x86_64 targets. Changes: * Adds a new generic instruction, SELECTIF, that does value selection (a la conditional move) similarly to existing SELECT, except that it is controlled by condition code input and flags-register inputs. * Adds a new Intel x86_64 variant, 'baseline', that supports SSE2 and nothing else. * Adds new Intel x86_64 instructions BSR and BSF. * Implements generic CLZ, CTZ and POPCOUNT on x86_64 'baseline' targets using the new BSR, BSF and SELECTIF instructions. * Implements SELECTIF on x86_64 targets using conditional-moves. * new test filetests/isa/intel/baseline_clz_ctz_popcount.cton (for legalization) * new test filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton (for encoding) * Allow lib/cretonne/meta/gen_legalizer.py to generate non-snake-caseified Rust without rustc complaining. Fixes #238. --- cranelift/docs/langref.rst | 5 + .../isa/intel/baseline_clz_ctz_popcount.cton | 104 ++++++++++++++ .../baseline_clz_ctz_popcount_encoding.cton | 89 ++++++++++++ cranelift/filetests/parser/tiny.cton | 12 +- lib/cretonne/meta/base/formats.py | 2 + lib/cretonne/meta/base/instructions.py | 9 ++ lib/cretonne/meta/gen_legalizer.py | 2 +- lib/cretonne/meta/isa/intel/encodings.py | 11 ++ lib/cretonne/meta/isa/intel/instructions.py | 23 ++++ lib/cretonne/meta/isa/intel/legalize.py | 130 +++++++++++++++++- lib/cretonne/meta/isa/intel/recipes.py | 27 ++++ lib/cretonne/meta/isa/intel/settings.py | 1 + lib/cretonne/src/ir/instructions.rs | 5 + lib/cretonne/src/verifier/mod.rs | 1 + lib/cretonne/src/write.rs | 3 + lib/reader/src/parser.rs | 19 +++ 16 files changed, 440 insertions(+), 3 deletions(-) create mode 100644 cranelift/filetests/isa/intel/baseline_clz_ctz_popcount.cton create mode 100644 cranelift/filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index a6fe32ff85..44204ad53f 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -700,6 +700,7 @@ Operations ========== .. autoinst:: select +.. autoinst:: selectif Constant materialization ------------------------ @@ -979,6 +980,10 @@ Instructions that can only be used by the Intel target ISA. .. autoinst:: isa.intel.instructions.cvtt2si .. autoinst:: isa.intel.instructions.fmin .. autoinst:: isa.intel.instructions.fmax +.. autoinst:: isa.intel.instructions.bsf +.. autoinst:: isa.intel.instructions.bsr +.. autoinst:: isa.intel.instructions.push +.. autoinst:: isa.intel.instructions.pop Instruction groups ================== diff --git a/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount.cton b/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount.cton new file mode 100644 index 0000000000..62a793de60 --- /dev/null +++ b/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount.cton @@ -0,0 +1,104 @@ + +test compile +set is_64bit +isa intel baseline + + +; clz/ctz on 64 bit operands + +function %i64_clz(i64) -> i64 { +ebb0(v10: i64): + v11 = clz v10 + ; check: x86_bsr + ; check: selectif.i64 + return v11 +} + +function %i64_ctz(i64) -> i64 { +ebb1(v20: i64): + v21 = ctz v20 + ; check: x86_bsf + ; check: selectif.i64 + return v21 +} + + +; clz/ctz on 32 bit operands + +function %i32_clz(i32) -> i32 { +ebb0(v10: i32): + v11 = clz v10 + ; check: x86_bsr + ; check: selectif.i32 + return v11 +} + +function %i32_ctz(i32) -> i32 { +ebb1(v20: i32): + v21 = ctz v20 + ; check: x86_bsf + ; check: selectif.i32 + return v21 +} + + +; popcount on 64 bit operands + +function %i64_popcount(i64) -> i64 { +ebb0(v30: i64): + v31 = popcnt v30; + ; check: iconst.i32 + ; check: ushr + ; check: iconst.i64 + ; check: band + ; check: isub + ; check: iconst.i32 + ; check: ushr + ; check: band + ; check: isub + ; check: iconst.i32 + ; check: ushr + ; check: band + ; check: isub + ; check: iconst.i32 + ; check: ushr + ; check: iadd + ; check: iconst.i64 + ; check: band + ; check: iconst.i64 + ; check: imul + ; check: iconst.i32 + ; check: ushr + return v31; +} + + +; popcount on 32 bit operands + +function %i32_popcount(i32) -> i32 { +ebb0(v40: i32): + v41 = popcnt v40; + ; check: iconst.i32 + ; check: ushr + ; check: iconst.i32 + ; check: band + ; check: isub + ; check: iconst.i32 + ; check: ushr + ; check: band + ; check: isub + ; check: iconst.i32 + ; check: ushr + ; check: band + ; check: isub + ; check: iconst.i32 + ; check: ushr + ; check: iadd + ; check: iconst.i32 + ; check: band + ; check: iconst.i32 + ; check: imul + ; check: iconst.i32 + ; check: ushr + return v41; +} diff --git a/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton b/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton new file mode 100644 index 0000000000..0b7003449d --- /dev/null +++ b/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton @@ -0,0 +1,89 @@ + +test binemit +set is_64bit +set is_compressed +isa intel baseline + +; The binary encodings can be verified with the command: +; +; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton | llvm-mc -show-encoding -triple=x86_64 +; + +function %Foo() { +ebb0: + ; 64-bit wide bsf + + [-,%r11] v10 = iconst.i64 0x1234 + ; asm: bsfq %r11, %rcx + [-,%rcx,%eflags] v11, v12 = x86_bsf v10 ; bin: 49 0f bc cb + + [-,%rdx] v14 = iconst.i64 0x5678 + ; asm: bsfq %rdx, %r12 + [-,%r12,%eflags] v15, v16 = x86_bsf v14 ; bin: 4c 0f bc e2 + + ; asm: bsfq %rdx, %rdi + [-,%rdi,%eflags] v17, v18 = x86_bsf v14 ; bin: 48 0f bc fa + + + ; 32-bit wide bsf + + [-,%r11] v20 = iconst.i32 0x1234 + ; asm: bsfl %r11d, %ecx + [-,%rcx,%eflags] v21, v22 = x86_bsf v20 ; bin: 41 0f bc cb + + [-,%rdx] v24 = iconst.i32 0x5678 + ; asm: bsfl %edx, %r12d + [-,%r12,%eflags] v25, v26 = x86_bsf v24 ; bin: 44 0f bc e2 + + ; asm: bsfl %edx, %esi + [-,%rsi,%eflags] v27, v28 = x86_bsf v24 ; bin: 0f bc f2 + + + ; 64-bit wide bsr + + [-,%r11] v30 = iconst.i64 0x1234 + ; asm: bsrq %r11, %rcx + [-,%rcx,%eflags] v31, v32 = x86_bsr v30 ; bin: 49 0f bd cb + + [-,%rdx] v34 = iconst.i64 0x5678 + ; asm: bsrq %rdx, %r12 + [-,%r12,%eflags] v35, v36 = x86_bsr v34 ; bin: 4c 0f bd e2 + + ; asm: bsrq %rdx, %rdi + [-,%rdi,%eflags] v37, v38 = x86_bsr v34 ; bin: 48 0f bd fa + + + ; 32-bit wide bsr + + [-,%r11] v40 = iconst.i32 0x1234 + ; asm: bsrl %r11d, %ecx + [-,%rcx,%eflags] v41, v42 = x86_bsr v40 ; bin: 41 0f bd cb + + [-,%rdx] v44 = iconst.i32 0x5678 + ; asm: bsrl %edx, %r12d + [-,%r12,%eflags] v45, v46 = x86_bsr v44 ; bin: 44 0f bd e2 + + ; asm: bsrl %edx, %esi + [-,%rsi,%eflags] v47, v48 = x86_bsr v44 ; bin: 0f bd f2 + + + ; 64-bit wide cmov + + ; asm: cmoveq %r11, %rdx + [-,%rdx] v51 = selectif.i64 eq v48, v30, v34 ; bin: 49 0f 44 d3 + + ; asm: cmoveq %rdi, %rdx + [-,%rdx] v52 = selectif.i64 eq v48, v37, v34 ; bin: 48 0f 44 d7 + + + ; 32-bit wide cmov + + ; asm: cmovnel %r11d, %edx + [-,%rdx] v60 = selectif.i32 ne v48, v40, v44 ; bin: 41 0f 45 d3 + + ; asm: cmovlel %esi, %edx + [-,%rdx] v61 = selectif.i32 sle v48, v27, v44 ; bin: 0f 4e d6 + + + trap user0 +} diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 20c4bbe2b3..5e8c8e85c3 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -42,7 +42,7 @@ ebb0: ; nextln: $v3 = bxor v0, v2 ; nextln: } -; Polymorphic istruction controlled by second operand. +; Polymorphic instruction controlled by second operand. function %select() { ebb0(v90: i32, v91: i32, v92: b1): v0 = select v92, v90, v91 @@ -52,6 +52,16 @@ ebb0(v90: i32, v91: i32, v92: b1): ; nextln: $v0 = select $v92, $v90, $v91 ; nextln: } +; Polymorphic instruction controlled by third operand. +function %selectif() native { +ebb0(v95: i32, v96: i32, v97: b1): + v98 = selectif.i32 eq v97, v95, v96 +} +; sameln: function %selectif() native { +; nextln: ebb0(v0: i32, v1: i32, v2: b1): +; nextln: v3 = selectif.i32 eq v2, v0, v1 +; nextln: } + ; Lane indexes. function %lanes() { ebb0: diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 539d87561b..394e1f4dc4 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -43,6 +43,8 @@ IntCond = InstructionFormat(intcc, VALUE) FloatCompare = InstructionFormat(floatcc, VALUE, VALUE) FloatCond = InstructionFormat(floatcc, VALUE) +IntSelect = InstructionFormat(intcc, VALUE, VALUE, VALUE) + Jump = InstructionFormat(ebb, VARIABLE_ARGS) Branch = InstructionFormat(VALUE, ebb, VARIABLE_ARGS) BranchInt = InstructionFormat(intcc, VALUE, ebb, VARIABLE_ARGS) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 6a41725ff9..fad863d976 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -485,6 +485,15 @@ select = Instruction( """, ins=(c, x, y), outs=a) +cc = Operand('cc', intcc, doc='Controlling condition code') +flags = Operand('flags', iflags, doc='The machine\'s flag register') + +selectif = Instruction( + 'selectif', r""" + Conditional select, dependent on integer condition codes. + """, + ins=(cc, flags, x, y), outs=a) + x = Operand('x', Any) copy = Instruction( diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index dea0a53950..54678b4284 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -355,7 +355,7 @@ def gen_xform(xform, fmt, type_sets): def gen_xform_group(xgrp, fmt, type_sets): # type: (XFormGroup, Formatter, UniqueTable) -> None fmt.doc_comment("Legalize the instruction pointed to by `pos`.") - fmt.line('#[allow(unused_variables,unused_assignments)]') + fmt.line('#[allow(unused_variables,unused_assignments,non_snake_case)]') with fmt.indented('pub fn {}('.format(xgrp.name)): fmt.line('inst: ir::Inst,') fmt.line('func: &mut ir::Function,') diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 4e4ebc229c..0e962a66f1 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -367,6 +367,17 @@ enc_i32_i64(base.ifcmp, r.rcmp, 0x39) enc_both(base.trueif, r.seti_abcd, 0x0f, 0x90) enc_both(base.trueff, r.setf_abcd, 0x0f, 0x90) +# +# Conditional move (a.k.a integer select) +# +enc_i32_i64(base.selectif, r.cmov, 0x0F, 0x40) + +# +# Bit scan forwards and reverse +# +enc_i32_i64(x86.bsf, r.bsf_and_bsr, 0x0F, 0xBC) +enc_i32_i64(x86.bsr, r.bsf_and_bsr, 0x0F, 0xBD) + # # Convert bool to int. # diff --git a/lib/cretonne/meta/isa/intel/instructions.py b/lib/cretonne/meta/isa/intel/instructions.py index 0852d49c73..277cf62b4a 100644 --- a/lib/cretonne/meta/isa/intel/instructions.py +++ b/lib/cretonne/meta/isa/intel/instructions.py @@ -5,6 +5,7 @@ This module defines additional instructions that are useful only to the Intel target ISA. """ +from base.types import iflags from cdsl.operands import Operand from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup @@ -125,4 +126,26 @@ pop = Instruction( """, outs=x, can_load=True, other_side_effects=True) +y = Operand('y', iWord) +rflags = Operand('rflags', iflags) + +bsr = Instruction( + 'x86_bsr', r""" + Bit Scan Reverse -- returns the bit-index of the most significant 1 + in the word. Result is undefined if the argument is zero. However, it + sets the Z flag depending on the argument, so it is at least easy to + detect and handle that case. + + This is polymorphic in i32 and i64. It is implemented for both i64 and + i32 in 64-bit mode, and only for i32 in 32-bit mode. + """, + ins=x, outs=(y, rflags)) + +bsf = Instruction( + 'x86_bsf', r""" + Bit Scan Forwards -- returns the bit-index of the least significant 1 + in the word. Is otherwise identical to 'bsr', just above. + """, + ins=x, outs=(y, rflags)) + GROUP.close() diff --git a/lib/cretonne/meta/isa/intel/legalize.py b/lib/cretonne/meta/isa/intel/legalize.py index 5c883baf09..70a8e9166b 100644 --- a/lib/cretonne/meta/isa/intel/legalize.py +++ b/lib/cretonne/meta/isa/intel/legalize.py @@ -4,7 +4,7 @@ Custom legalization patterns for Intel. from __future__ import absolute_import from cdsl.ast import Var from cdsl.xform import Rtl, XFormGroup -from base.immediates import imm64, floatcc +from base.immediates import imm64, intcc, floatcc from base.types import i32, i64 from base import legalize as shared from base import instructions as insts @@ -100,3 +100,131 @@ intel_expand.custom_legalize(insts.fcvt_from_uint, 'expand_fcvt_from_uint') # Conversions from float to int can trap. intel_expand.custom_legalize(insts.fcvt_to_sint, 'expand_fcvt_to_sint') intel_expand.custom_legalize(insts.fcvt_to_uint, 'expand_fcvt_to_uint') + +# Count leading and trailing zeroes, for baseline x86_64 +c_minus_one = Var('c_minus_one') +c_thirty_one = Var('c_thirty_one') +c_thirty_two = Var('c_thirty_two') +c_sixty_three = Var('c_sixty_three') +c_sixty_four = Var('c_sixty_four') +index1 = Var('index1') +r2flags = Var('r2flags') +index2 = Var('index2') + +intel_expand.legalize( + a << insts.clz.i64(x), + Rtl( + c_minus_one << insts.iconst(imm64(-1)), + c_sixty_three << insts.iconst(imm64(63)), + (index1, r2flags) << x86.bsr(x), + index2 << insts.selectif(intcc.eq, r2flags, c_minus_one, index1), + a << insts.isub(c_sixty_three, index2), + )) + +intel_expand.legalize( + a << insts.clz.i32(x), + Rtl( + c_minus_one << insts.iconst(imm64(-1)), + c_thirty_one << insts.iconst(imm64(31)), + (index1, r2flags) << x86.bsr(x), + index2 << insts.selectif(intcc.eq, r2flags, c_minus_one, index1), + a << insts.isub(c_thirty_one, index2), + )) + +intel_expand.legalize( + a << insts.ctz.i64(x), + Rtl( + c_sixty_four << insts.iconst(imm64(64)), + (index1, r2flags) << x86.bsf(x), + a << insts.selectif(intcc.eq, r2flags, c_sixty_four, index1), + )) + +intel_expand.legalize( + a << insts.ctz.i32(x), + Rtl( + c_thirty_two << insts.iconst(imm64(32)), + (index1, r2flags) << x86.bsf(x), + a << insts.selectif(intcc.eq, r2flags, c_thirty_two, index1), + )) + + +# Population count for baseline x86_64 +qv1 = Var('qv1') +qv3 = Var('qv3') +qv4 = Var('qv4') +qv5 = Var('qv5') +qv6 = Var('qv6') +qv7 = Var('qv7') +qv8 = Var('qv8') +qv9 = Var('qv9') +qv10 = Var('qv10') +qv11 = Var('qv11') +qv12 = Var('qv12') +qv13 = Var('qv13') +qv14 = Var('qv14') +qv15 = Var('qv15') +qv16 = Var('qv16') +qc77 = Var('qc77') +qc0F = Var('qc0F') +qc01 = Var('qc01') +intel_expand.legalize( + qv16 << insts.popcnt.i64(qv1), + Rtl( + qv3 << insts.ushr_imm(qv1, imm64(1)), + qc77 << insts.iconst(imm64(0x7777777777777777)), + qv4 << insts.band(qv3, qc77), + qv5 << insts.isub(qv1, qv4), + qv6 << insts.ushr_imm(qv4, imm64(1)), + qv7 << insts.band(qv6, qc77), + qv8 << insts.isub(qv5, qv7), + qv9 << insts.ushr_imm(qv7, imm64(1)), + qv10 << insts.band(qv9, qc77), + qv11 << insts.isub(qv8, qv10), + qv12 << insts.ushr_imm(qv11, imm64(4)), + qv13 << insts.iadd(qv11, qv12), + qc0F << insts.iconst(imm64(0x0F0F0F0F0F0F0F0F)), + qv14 << insts.band(qv13, qc0F), + qc01 << insts.iconst(imm64(0x0101010101010101)), + qv15 << insts.imul(qv14, qc01), + qv16 << insts.ushr_imm(qv15, imm64(56)) + )) + +lv1 = Var('lv1') +lv3 = Var('lv3') +lv4 = Var('lv4') +lv5 = Var('lv5') +lv6 = Var('lv6') +lv7 = Var('lv7') +lv8 = Var('lv8') +lv9 = Var('lv9') +lv10 = Var('lv10') +lv11 = Var('lv11') +lv12 = Var('lv12') +lv13 = Var('lv13') +lv14 = Var('lv14') +lv15 = Var('lv15') +lv16 = Var('lv16') +lc77 = Var('lc77') +lc0F = Var('lc0F') +lc01 = Var('lc01') +intel_expand.legalize( + lv16 << insts.popcnt.i32(lv1), + Rtl( + lv3 << insts.ushr_imm(lv1, imm64(1)), + lc77 << insts.iconst(imm64(0x77777777)), + lv4 << insts.band(lv3, lc77), + lv5 << insts.isub(lv1, lv4), + lv6 << insts.ushr_imm(lv4, imm64(1)), + lv7 << insts.band(lv6, lc77), + lv8 << insts.isub(lv5, lv7), + lv9 << insts.ushr_imm(lv7, imm64(1)), + lv10 << insts.band(lv9, lc77), + lv11 << insts.isub(lv8, lv10), + lv12 << insts.ushr_imm(lv11, imm64(4)), + lv13 << insts.iadd(lv11, lv12), + lc0F << insts.iconst(imm64(0x0F0F0F0F)), + lv14 << insts.band(lv13, lc0F), + lc01 << insts.iconst(imm64(0x01010101)), + lv15 << insts.imul(lv14, lc01), + lv16 << insts.ushr_imm(lv15, imm64(24)) + )) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 313ca7abd6..622430b9b2 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -8,6 +8,7 @@ from cdsl.registers import RegClass from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry, NullAry from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare, FloatCompare, IntCond, FloatCond +from base.formats import IntSelect from base.formats import Jump, Branch, BranchInt, BranchFloat from base.formats import Ternary, FuncAddr, UnaryGlobalVar from base.formats import RegMove, RegSpill, RegFill, CopySpecial @@ -1021,6 +1022,32 @@ setf_abcd = TailRecipe( modrm_r_bits(out_reg0, bits, sink); ''') +# +# Conditional move (a.k.a integer select) +# (maybe-REX.W) 0F 4x modrm(r,r) +# 1 byte, modrm(r,r), is after the opcode +# +cmov = TailRecipe( + 'cmov', IntSelect, size=1, ins=(FLAG.eflags, GPR, GPR), outs=2, + requires_prefix=False, + clobbers_flags=False, + emit=''' + PUT_OP(bits | icc2opc(cond), rex2(in_reg1, in_reg2), sink); + modrm_rr(in_reg1, in_reg2, sink); + ''') + +# +# Bit scan forwards and reverse +# +bsf_and_bsr = TailRecipe( + 'bsf_and_bsr', Unary, size=1, ins=GPR, outs=(GPR, FLAG.eflags), + requires_prefix=False, + clobbers_flags=True, + emit=''' + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); + modrm_rr(in_reg0, out_reg0, sink); + ''') + # # Compare and set flags. # diff --git a/lib/cretonne/meta/isa/intel/settings.py b/lib/cretonne/meta/isa/intel/settings.py index a16303a3b5..5817c48c0e 100644 --- a/lib/cretonne/meta/isa/intel/settings.py +++ b/lib/cretonne/meta/isa/intel/settings.py @@ -40,6 +40,7 @@ use_lzcnt = And(has_lzcnt) # Presets corresponding to Intel CPUs. +baseline = Preset(has_sse2) nehalem = Preset( has_sse2, has_sse3, has_ssse3, has_sse41, has_sse42, has_popcnt) haswell = Preset(nehalem, has_bmi1, has_lzcnt) diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 37ca23150c..9e28616377 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -157,6 +157,11 @@ pub enum InstructionData { cond: FloatCC, arg: Value, }, + IntSelect { + opcode: Opcode, + cond: IntCC, + args: [Value; 3], + }, Jump { opcode: Opcode, destination: Ebb, diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 1860fecbbf..1b27c41766 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -358,6 +358,7 @@ impl<'a> Verifier<'a> { IntCond { .. } | FloatCompare { .. } | FloatCond { .. } | + IntSelect { .. } | Load { .. } | Store { .. } | RegMove { .. } | diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 8c8c280ca9..2312e5644f 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -303,6 +303,9 @@ pub fn write_operands( IntCond { cond, arg, .. } => write!(w, " {} {}", cond, arg), FloatCompare { cond, args, .. } => write!(w, " {} {}, {}", cond, args[0], args[1]), FloatCond { cond, arg, .. } => write!(w, " {} {}", cond, arg), + IntSelect { cond, args, .. } => { + write!(w, " {} {}, {}, {}", cond, args[0], args[1], args[2]) + } Jump { destination, ref args, diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 17f14af9b1..95923d013d 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -2119,6 +2119,25 @@ impl<'a> Parser<'a> { let arg = self.match_value("expected SSA value")?; InstructionData::FloatCond { opcode, cond, arg } } + InstructionFormat::IntSelect => { + let cond = self.match_enum("expected intcc condition code")?; + let guard = self.match_value("expected SSA value first operand")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; + let v_true = self.match_value("expected SSA value second operand")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; + let v_false = self.match_value("expected SSA value third operand")?; + InstructionData::IntSelect { + opcode, + cond, + args: [guard, v_true, v_false], + } + } InstructionFormat::Call => { let func_ref = self.match_fn("expected function reference").and_then( |num| { From 3eeef1c75245ce62fea8730562ae6e36fd220db2 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 6 Feb 2018 09:55:53 -0800 Subject: [PATCH 1494/3084] Add some missing instructions to the language reference. --- cranelift/docs/langref.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 44204ad53f..b4a9f1dcb2 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -574,6 +574,7 @@ runtime data structures. :result GV: Global variable. .. autoinst:: global_addr +.. autoinst:: globalsym_addr Heaps @@ -938,6 +939,15 @@ the target ISA. .. autoinst:: isplit .. autoinst:: iconcat +Special register operations +--------------------------- + +The prologue and epilogue of a function needs to manipulate special registers like the stack +pointer and the frame pointer. These instructions should not be used in regular code. + +.. autoinst:: adjust_sp_imm +.. autoinst:: copy_special + .. _extload-truncstore: Extending loads and truncating stores From 11c721934c095a709f79350192ae6941581560c6 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 8 Feb 2018 14:39:06 -0800 Subject: [PATCH 1495/3084] Add a trapif instruction. This is a conditional trap controlled by integer CPU flags. Compare to brif. --- cranelift/docs/langref.rst | 1 + cranelift/filetests/parser/tiny.cton | 15 +++++++++++++++ lib/cretonne/meta/base/formats.py | 1 + lib/cretonne/meta/base/instructions.py | 9 +++++++++ lib/cretonne/src/ir/instructions.rs | 6 ++++++ lib/cretonne/src/verifier/mod.rs | 1 + lib/cretonne/src/write.rs | 1 + lib/reader/src/parser.rs | 15 +++++++++++++++ 8 files changed, 49 insertions(+) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index b4a9f1dcb2..17a146ba07 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -380,6 +380,7 @@ is zero. .. autoinst:: trap .. autoinst:: trapz .. autoinst:: trapnz +.. autoinst:: trapif Function calls diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 5e8c8e85c3..748c8f9a13 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -210,3 +210,18 @@ ebb0: ; nextln: copy_special %20 -> %10 ; nextln: return ; nextln: } + +function %cond_traps(i32) { +ebb0(v0: i32): + trapz v0, stk_ovf + v1 = ifcmp_imm v0, 5 + trapif ugt v1, oob + return +} +; sameln: function %cond_traps(i32) +; nextln: ebb0($v0: i32): +; nextln: trapz $v0, stk_ovf +; nextln: $v1 = ifcmp_imm v0, 5 +; nextln: trapif ugt $v1, oob +; nextln: return +; nextln: } diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 394e1f4dc4..1be94a7b09 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -74,6 +74,7 @@ RegFill = InstructionFormat( Trap = InstructionFormat(trapcode) CondTrap = InstructionFormat(VALUE, trapcode) +IntCondTrap = InstructionFormat(intcc, VALUE, trapcode) # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index fad863d976..875a0e7537 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -167,6 +167,15 @@ trapnz = Instruction( """, ins=(c, code), can_trap=True) +Cond = Operand('Cond', intcc) +f = Operand('f', iflags) + +trapif = Instruction( + 'trapif', r""" + Trap when condition is true in integer CPU flags. + """, + ins=(Cond, f, code), can_trap=True) + rvals = Operand('rvals', VARIABLE_ARGS, doc='return values') x_return = Instruction( diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 9e28616377..449da29b93 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -264,6 +264,12 @@ pub enum InstructionData { arg: Value, code: ir::TrapCode, }, + IntCondTrap { + opcode: Opcode, + cond: IntCC, + arg: Value, + code: ir::TrapCode, + }, } /// A variable list of `Value` operands used for function call arguments and passing arguments to diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 1b27c41766..fa5c751430 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -365,6 +365,7 @@ impl<'a> Verifier<'a> { CopySpecial { .. } | Trap { .. } | CondTrap { .. } | + IntCondTrap { .. } | NullAry { .. } => {} } diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 2312e5644f..ea9e3a3e4a 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -428,6 +428,7 @@ pub fn write_operands( } Trap { code, .. } => write!(w, " {}", code), CondTrap { arg, code, .. } => write!(w, " {}, {}", arg, code), + IntCondTrap { cond, arg, code, .. } => write!(w, " {} {}, {}", cond, arg, code), } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 95923d013d..c4c7b6e852 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -2348,6 +2348,21 @@ impl<'a> Parser<'a> { let code = self.match_enum("expected trap code")?; InstructionData::CondTrap { opcode, arg, code } } + InstructionFormat::IntCondTrap => { + let cond = self.match_enum("expected intcc condition code")?; + let arg = self.match_value("expected SSA value operand")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; + let code = self.match_enum("expected trap code")?; + InstructionData::IntCondTrap { + opcode, + cond, + arg, + code, + } + } }; Ok(idata) } From 69f70fc61dad5ba93f4809f11515401ecddbf924 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Thu, 8 Feb 2018 15:15:15 -0800 Subject: [PATCH 1496/3084] Add Intel encodings for trapif. This is implemented as a macro with a conditional jump over a ud2. This way, we don't have to split up EBBs at every conditional trap. --- cranelift/filetests/isa/intel/binary32.cton | 22 +++++++++++++++++++++ cranelift/filetests/isa/intel/binary64.cton | 22 +++++++++++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 4 ++++ lib/cretonne/meta/isa/intel/recipes.py | 15 +++++++++++++- lib/cretonne/src/isa/intel/binemit.rs | 2 +- 5 files changed, 63 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 02e7fc0364..5da0c709a6 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -512,5 +512,27 @@ ebb1: ; asm: setbe %bl [-,%rbx] v29 = trueif ule v11 ; bin: 0f 96 c3 + ; The trapif instructions are encoded as macros: a conditional jump over a ud2. + ; asm: jne .+4; ud2 + trapif eq v11, user0 ; bin: 75 02 0f 0b + ; asm: je .+4; ud2 + trapif ne v11, user0 ; bin: 74 02 0f 0b + ; asm: jnl .+4; ud2 + trapif slt v11, user0 ; bin: 7d 02 0f 0b + ; asm: jnge .+4; ud2 + trapif sge v11, user0 ; bin: 7c 02 0f 0b + ; asm: jng .+4; ud2 + trapif sgt v11, user0 ; bin: 7e 02 0f 0b + ; asm: jnle .+4; ud2 + trapif sle v11, user0 ; bin: 7f 02 0f 0b + ; asm: jnb .+4; ud2 + trapif ult v11, user0 ; bin: 73 02 0f 0b + ; asm: jnae .+4; ud2 + trapif uge v11, user0 ; bin: 72 02 0f 0b + ; asm: jna .+4; ud2 + trapif ugt v11, user0 ; bin: 76 02 0f 0b + ; asm: jnbe .+4; ud2 + trapif ule v11, user0 ; bin: 77 02 0f 0b + return } diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 719e6b1de5..29f75085d9 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -593,6 +593,28 @@ ebb1: ; asm: setbe %r11b [-,%r11] v29 = trueif ule v11 ; bin: 41 0f 96 c3 + ; The trapif instructions are encoded as macros: a conditional jump over a ud2. + ; asm: jne .+4; ud2 + trapif eq v11, user0 ; bin: 75 02 0f 0b + ; asm: je .+4; ud2 + trapif ne v11, user0 ; bin: 74 02 0f 0b + ; asm: jnl .+4; ud2 + trapif slt v11, user0 ; bin: 7d 02 0f 0b + ; asm: jnge .+4; ud2 + trapif sge v11, user0 ; bin: 7c 02 0f 0b + ; asm: jng .+4; ud2 + trapif sgt v11, user0 ; bin: 7e 02 0f 0b + ; asm: jnle .+4; ud2 + trapif sle v11, user0 ; bin: 7f 02 0f 0b + ; asm: jnb .+4; ud2 + trapif ult v11, user0 ; bin: 73 02 0f 0b + ; asm: jnae .+4; ud2 + trapif uge v11, user0 ; bin: 72 02 0f 0b + ; asm: jna .+4; ud2 + trapif ugt v11, user0 ; bin: 76 02 0f 0b + ; asm: jnbe .+4; ud2 + trapif ule v11, user0 ; bin: 77 02 0f 0b + return } diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 0e962a66f1..5b9f8ae422 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -354,6 +354,10 @@ enc_both(base.brnz.b1, r.t8jccd_abcd, 0x85) I32.enc(base.trap, *r.trap(0x0f, 0x0b)) I64.enc(base.trap, *r.trap(0x0f, 0x0b)) +# Using a standard EncRecipe, not the TailRecipe. +I32.enc(base.trapif, r.trapif, 0) +I64.enc(base.trapif, r.trapif, 0) + # # Comparisons # diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 622430b9b2..7e74abae47 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -8,7 +8,7 @@ from cdsl.registers import RegClass from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry, NullAry from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare, FloatCompare, IntCond, FloatCond -from base.formats import IntSelect +from base.formats import IntSelect, IntCondTrap from base.formats import Jump, Branch, BranchInt, BranchFloat from base.formats import Ternary, FuncAddr, UnaryGlobalVar from base.formats import RegMove, RegSpill, RegFill, CopySpecial @@ -279,6 +279,19 @@ trap = TailRecipe( 'trap', Trap, size=0, ins=(), outs=(), emit='PUT_OP(bits, BASE_REX, sink);') +# Macro: conditional jump over a ud2. +trapif = EncRecipe( + 'trapif', IntCondTrap, size=4, ins=FLAG.eflags, outs=(), + clobbers_flags=False, + emit=''' + // Jump over a 2-byte ud2. + sink.put1(0x70 | (icc2opc(cond.inverse()) as u8)); + sink.put1(2); + // ud2. + sink.put1(0x0f); + sink.put1(0x0b); + ''') + # XX /r rr = TailRecipe( 'rr', Binary, size=1, ins=(GPR, GPR), outs=0, diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 0f07f049f3..2d045a0882 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -2,7 +2,7 @@ use binemit::{CodeSink, Reloc, bad_encoding}; use ir::{Function, Inst, Ebb, InstructionData, Opcode}; -use ir::condcodes::{IntCC, FloatCC}; +use ir::condcodes::{CondCode, IntCC, FloatCC}; use isa::{RegUnit, StackRef, StackBase, StackBaseMask}; use regalloc::RegDiversions; use super::registers::RU; From 73c4c356c9b40934e7044031eb3bcb993b731911 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 9 Feb 2018 13:59:49 -0800 Subject: [PATCH 1497/3084] Add an ifcmp_sp instruction. This will be used to implement the stack_check macro. --- cranelift/docs/langref.rst | 1 + lib/cretonne/meta/base/instructions.py | 11 +++++++++++ 2 files changed, 12 insertions(+) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 17a146ba07..b7fe64b924 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -947,6 +947,7 @@ The prologue and epilogue of a function needs to manipulate special registers li pointer and the frame pointer. These instructions should not be used in regular code. .. autoinst:: adjust_sp_imm +.. autoinst:: ifcmp_sp .. autoinst:: copy_special .. _extload-truncstore: diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 875a0e7537..b6ab96a16e 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -579,6 +579,17 @@ adjust_sp_imm = Instruction( ins=(StackOffset,), other_side_effects=True) +f = Operand('f', iflags) + +ifcmp_sp = Instruction( + 'ifcmp_sp', r""" + Compare ``addr`` with the stack pointer and set the CPU flags. + + This is like :inst:`ifcmp` where ``addr`` is the LHS operand and the stack + pointer is the RHS. + """, + ins=addr, outs=f) + regspill = Instruction( 'regspill', r""" Temporarily divert ``x`` from ``src`` to ``SS``. From 788a78caf4e1631e7ff394d64ce33f057b349ace Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 9 Feb 2018 14:32:29 -0800 Subject: [PATCH 1498/3084] Add Intel encodings for ifcmp_sp. Also generate an Into implementation for the RU enums. --- cranelift/filetests/isa/intel/binary32.cton | 6 ++++++ cranelift/filetests/isa/intel/binary64.cton | 6 ++++++ lib/cretonne/meta/gen_registers.py | 3 +++ lib/cretonne/meta/isa/intel/encodings.py | 3 +++ lib/cretonne/meta/isa/intel/recipes.py | 16 ++++++++++++---- lib/cretonne/src/isa/arm32/registers.rs | 2 +- lib/cretonne/src/isa/arm64/registers.rs | 2 +- lib/cretonne/src/isa/intel/registers.rs | 2 +- lib/cretonne/src/isa/riscv/registers.rs | 2 +- 9 files changed, 34 insertions(+), 8 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 5da0c709a6..e507e5fbcd 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -534,5 +534,11 @@ ebb1: ; asm: jnbe .+4; ud2 trapif ule v11, user0 ; bin: 77 02 0f 0b + ; Stack check. + ; asm: cmpl %esp, %ecx + [-,%eflags] v40 = ifcmp_sp v1 ; bin: 39 e1 + ; asm: cmpl %esp, %esi + [-,%eflags] v41 = ifcmp_sp v2 ; bin: 39 e6 + return } diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 29f75085d9..486223ec6f 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -615,6 +615,12 @@ ebb1: ; asm: jnbe .+4; ud2 trapif ule v11, user0 ; bin: 77 02 0f 0b + ; Stack check. + ; asm: cmpq %rsp, %rcx + [-,%eflags] v40 = ifcmp_sp v1 ; bin: 48 39 e1 + ; asm: cmpq %rsp, %r10 + [-,%eflags] v41 = ifcmp_sp v2 ; bin: 49 39 e2 + return } diff --git a/lib/cretonne/meta/gen_registers.py b/lib/cretonne/meta/gen_registers.py index 8e12fbf34f..166bc611da 100644 --- a/lib/cretonne/meta/gen_registers.py +++ b/lib/cretonne/meta/gen_registers.py @@ -96,6 +96,9 @@ def gen_isa(isa, fmt): with fmt.indented('pub enum RU {', '}'): for regbank in isa.regbanks: gen_regbank_units(regbank, fmt) + with fmt.indented('impl Into for RU {', '}'): + with fmt.indented('fn into(self) -> RegUnit {', '}'): + fmt.line('self as RegUnit') def generate(isas, out_dir): diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 5b9f8ae422..449e159f71 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -364,6 +364,9 @@ I64.enc(base.trapif, r.trapif, 0) enc_i32_i64(base.icmp, r.icscc, 0x39) enc_i32_i64(base.ifcmp, r.rcmp, 0x39) +I32.enc(base.ifcmp_sp.i32, *r.rcmp_sp(0x39)) +I64.enc(base.ifcmp_sp.i64, *r.rcmp_sp.rex(0x39, w=1)) + # # Convert flags to bool. # diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 7e74abae47..56f67306cb 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -517,8 +517,8 @@ adjustsp8 = TailRecipe( 'adjustsp8', UnaryImm, size=2, ins=(), outs=(), instp=IsSignedInt(UnaryImm.imm, 8), emit=''' - PUT_OP(bits, rex1(4), sink); - modrm_r_bits(4, bits, sink); + PUT_OP(bits, rex1(RU::rsp.into()), sink); + modrm_r_bits(RU::rsp.into(), bits, sink); let imm: i64 = imm.into(); sink.put1(imm as u8); ''') @@ -527,8 +527,8 @@ adjustsp32 = TailRecipe( 'adjustsp32', UnaryImm, size=5, ins=(), outs=(), instp=IsSignedInt(UnaryImm.imm, 32), emit=''' - PUT_OP(bits, rex1(4), sink); - modrm_r_bits(4, bits, sink); + PUT_OP(bits, rex1(RU::rsp.into()), sink); + modrm_r_bits(RU::rsp.into(), bits, sink); let imm: i64 = imm.into(); sink.put4(imm as u32); ''') @@ -1081,6 +1081,14 @@ fcmp = TailRecipe( modrm_rr(in_reg1, in_reg0, sink); ''') +# Same as rcmp, but second operand is the stack pointer. +rcmp_sp = TailRecipe( + 'rcmp_sp', Unary, size=1, ins=GPR, outs=FLAG.eflags, + emit=''' + PUT_OP(bits, rex2(in_reg0, RU::rsp.into()), sink); + modrm_rr(in_reg0, RU::rsp.into(), sink); + ''') + # Test-and-branch. # # This recipe represents the macro fusion of a test and a conditional branch. diff --git a/lib/cretonne/src/isa/arm32/registers.rs b/lib/cretonne/src/isa/arm32/registers.rs index 283f113d1e..7c6ac406e1 100644 --- a/lib/cretonne/src/isa/arm32/registers.rs +++ b/lib/cretonne/src/isa/arm32/registers.rs @@ -1,6 +1,6 @@ //! ARM32 register descriptions. -use isa::registers::{RegBank, RegClass, RegClassData, RegInfo}; +use isa::registers::{RegBank, RegClass, RegClassData, RegInfo, RegUnit}; include!(concat!(env!("OUT_DIR"), "/registers-arm32.rs")); diff --git a/lib/cretonne/src/isa/arm64/registers.rs b/lib/cretonne/src/isa/arm64/registers.rs index d4cee48891..62311aaebe 100644 --- a/lib/cretonne/src/isa/arm64/registers.rs +++ b/lib/cretonne/src/isa/arm64/registers.rs @@ -1,6 +1,6 @@ //! ARM64 register descriptions. -use isa::registers::{RegBank, RegClass, RegClassData, RegInfo}; +use isa::registers::{RegBank, RegClass, RegClassData, RegInfo, RegUnit}; include!(concat!(env!("OUT_DIR"), "/registers-arm64.rs")); diff --git a/lib/cretonne/src/isa/intel/registers.rs b/lib/cretonne/src/isa/intel/registers.rs index fcb1cf4445..c2b96f77d3 100644 --- a/lib/cretonne/src/isa/intel/registers.rs +++ b/lib/cretonne/src/isa/intel/registers.rs @@ -1,6 +1,6 @@ //! Intel register descriptions. -use isa::registers::{RegBank, RegClass, RegClassData, RegInfo}; +use isa::registers::{RegBank, RegClass, RegClassData, RegInfo, RegUnit}; include!(concat!(env!("OUT_DIR"), "/registers-intel.rs")); diff --git a/lib/cretonne/src/isa/riscv/registers.rs b/lib/cretonne/src/isa/riscv/registers.rs index 9447e5ea29..3cce8c4988 100644 --- a/lib/cretonne/src/isa/riscv/registers.rs +++ b/lib/cretonne/src/isa/riscv/registers.rs @@ -1,6 +1,6 @@ //! RISC-V register descriptions. -use isa::registers::{RegBank, RegClass, RegClassData, RegInfo}; +use isa::registers::{RegBank, RegClass, RegClassData, RegInfo, RegUnit}; include!(concat!(env!("OUT_DIR"), "/registers-riscv.rs")); From 60e70da0e69060b6063cd6cd4ae9f245030e362a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 13 Feb 2018 10:37:26 -0800 Subject: [PATCH 1499/3084] Add Intel encodings for ifcmp_imm. The instruction set has variants with 8-bit and 32-bit signed immediate operands. Add a TODO to use a TEST instruction for the special case ifcmp_imm x, 0. --- cranelift/filetests/isa/intel/binary32.cton | 10 ++++++++++ cranelift/filetests/isa/intel/binary64.cton | 20 +++++++++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 3 +++ lib/cretonne/meta/isa/intel/recipes.py | 22 +++++++++++++++++++++ 4 files changed, 55 insertions(+) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index e507e5fbcd..ea9613f9fa 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -540,5 +540,15 @@ ebb1: ; asm: cmpl %esp, %esi [-,%eflags] v41 = ifcmp_sp v2 ; bin: 39 e6 + ; asm: cmpl $-100, %ecx + [-,%eflags] v42 = ifcmp_imm v1, -100 ; bin: 83 f9 9c + ; asm: cmpl $100, %esi + [-,%eflags] v43 = ifcmp_imm v2, 100 ; bin: 83 fe 64 + + ; asm: cmpl $-10000, %ecx + [-,%eflags] v44 = ifcmp_imm v1, -10000 ; bin: 81 f9 ffffd8f0 + ; asm: cmpl $10000, %esi + [-,%eflags] v45 = ifcmp_imm v2, 10000 ; bin: 81 fe 00002710 + return } diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 486223ec6f..59c5f0b2f0 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -621,6 +621,16 @@ ebb1: ; asm: cmpq %rsp, %r10 [-,%eflags] v41 = ifcmp_sp v2 ; bin: 49 39 e2 + ; asm: cmpq $-100, %rcx + [-,%eflags] v522 = ifcmp_imm v1, -100 ; bin: 48 83 f9 9c + ; asm: cmpq $100, %r10 + [-,%eflags] v523 = ifcmp_imm v2, 100 ; bin: 49 83 fa 64 + + ; asm: cmpq $-10000, %rcx + [-,%eflags] v524 = ifcmp_imm v1, -10000 ; bin: 48 81 f9 ffffd8f0 + ; asm: cmpq $10000, %r10 + [-,%eflags] v525 = ifcmp_imm v2, 10000 ; bin: 49 81 fa 00002710 + return } @@ -994,6 +1004,16 @@ ebb0: ; asm: cmpl %r10d, %esi [-,%eflags] v521 = ifcmp v2, v3 ; bin: 44 39 d6 + ; asm: cmpl $-100, %ecx + [-,%eflags] v522 = ifcmp_imm v1, -100 ; bin: 83 f9 9c + ; asm: cmpl $100, %r10d + [-,%eflags] v523 = ifcmp_imm v3, 100 ; bin: 41 83 fa 64 + + ; asm: cmpl $-10000, %ecx + [-,%eflags] v524 = ifcmp_imm v1, -10000 ; bin: 81 f9 ffffd8f0 + ; asm: cmpl $10000, %r10d + [-,%eflags] v525 = ifcmp_imm v3, 10000 ; bin: 41 81 fa 00002710 + ; asm: testl %ecx, %ecx ; asm: je ebb1x brz v1, ebb1 ; bin: 85 c9 74 18 diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 449e159f71..1e1eff820c 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -363,6 +363,9 @@ I64.enc(base.trapif, r.trapif, 0) # enc_i32_i64(base.icmp, r.icscc, 0x39) enc_i32_i64(base.ifcmp, r.rcmp, 0x39) +enc_i32_i64(base.ifcmp_imm, r.rcmpib, 0x83, rrr=7) +enc_i32_i64(base.ifcmp_imm, r.rcmpid, 0x81, rrr=7) +# TODO: We could special-case ifcmp_imm(x, 0) to TEST(x, x). I32.enc(base.ifcmp_sp.i32, *r.rcmp_sp(0x39)) I64.enc(base.ifcmp_sp.i64, *r.rcmp_sp.rex(0x39, w=1)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 56f67306cb..9696a7ed43 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -1081,6 +1081,28 @@ fcmp = TailRecipe( modrm_rr(in_reg1, in_reg0, sink); ''') +# XX /n, MI form with imm8. +rcmpib = TailRecipe( + 'rcmpib', BinaryImm, size=2, ins=GPR, outs=FLAG.eflags, + instp=IsSignedInt(BinaryImm.imm, 8), + emit=''' + PUT_OP(bits, rex1(in_reg0), sink); + modrm_r_bits(in_reg0, bits, sink); + let imm: i64 = imm.into(); + sink.put1(imm as u8); + ''') + +# XX /n, MI form with imm32. +rcmpid = TailRecipe( + 'rcmpid', BinaryImm, size=5, ins=GPR, outs=FLAG.eflags, + instp=IsSignedInt(BinaryImm.imm, 32), + emit=''' + PUT_OP(bits, rex1(in_reg0), sink); + modrm_r_bits(in_reg0, bits, sink); + let imm: i64 = imm.into(); + sink.put4(imm as u32); + ''') + # Same as rcmp, but second operand is the stack pointer. rcmp_sp = TailRecipe( 'rcmp_sp', Unary, size=1, ins=GPR, outs=FLAG.eflags, From a73fcb26914a97e877f1f060fe4d2ffb20158a0f Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 12 Feb 2018 14:21:41 -0800 Subject: [PATCH 1500/3084] Pass an ISA argument to legalization functions. This lets them look at the ISA flags. --- lib/cretonne/meta/gen_legalizer.py | 5 ++-- lib/cretonne/src/isa/intel/enc_tables.rs | 35 ++++++++++++++++++++---- lib/cretonne/src/isa/mod.rs | 3 +- lib/cretonne/src/legalizer/globalvar.rs | 8 +++++- lib/cretonne/src/legalizer/heap.rs | 8 +++++- lib/cretonne/src/legalizer/mod.rs | 30 ++++++++++++++++---- 6 files changed, 74 insertions(+), 15 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 54678b4284..51368fea44 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -360,6 +360,7 @@ def gen_xform_group(xgrp, fmt, type_sets): fmt.line('inst: ir::Inst,') fmt.line('func: &mut ir::Function,') fmt.line('cfg: &mut ::flowgraph::ControlFlowGraph,') + fmt.line('isa: &::isa::TargetIsa,') with fmt.indented(') -> bool {', '}'): fmt.line('use ir::InstBuilder;') fmt.line('use cursor::{Cursor, FuncCursor};') @@ -387,7 +388,7 @@ def gen_xform_group(xgrp, fmt, type_sets): with fmt.indented( 'ir::Opcode::{} => {{' .format(inst.camel_name), '}'): - fmt.format('{}(inst, pos.func, cfg);', funcname) + fmt.format('{}(inst, pos.func, cfg, isa);', funcname) fmt.line('return true;') # We'll assume there are uncovered opcodes. @@ -395,7 +396,7 @@ def gen_xform_group(xgrp, fmt, type_sets): # If we fall through, nothing was expanded. Call the chain if any. if xgrp.chain: - fmt.format('{}(inst, pos.func, cfg)', xgrp.chain.rust_name()) + fmt.format('{}(inst, pos.func, cfg, isa)', xgrp.chain.rust_name()) else: fmt.line('false') diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 8265b731e7..a97607c4c7 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -15,7 +15,12 @@ include!(concat!(env!("OUT_DIR"), "/encoding-intel.rs")); include!(concat!(env!("OUT_DIR"), "/legalize-intel.rs")); /// Expand the `srem` instruction using `x86_sdivmodx`. -fn expand_srem(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { +fn expand_srem( + inst: ir::Inst, + func: &mut ir::Function, + cfg: &mut ControlFlowGraph, + _isa: &isa::TargetIsa, +) { use ir::condcodes::IntCC; let (x, y) = match func.dfg[inst] { @@ -70,7 +75,12 @@ fn expand_srem(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGra /// Expand the `fmin` and `fmax` instructions using the Intel `x86_fmin` and `x86_fmax` /// instructions. -fn expand_minmax(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { +fn expand_minmax( + inst: ir::Inst, + func: &mut ir::Function, + cfg: &mut ControlFlowGraph, + _isa: &isa::TargetIsa, +) { use ir::condcodes::FloatCC; let (x, y, x86_opc, bitwise_opc) = match func.dfg[inst] { @@ -159,7 +169,12 @@ fn expand_minmax(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowG /// Intel has no unsigned-to-float conversions. We handle the easy case of zero-extending i32 to /// i64 with a pattern, the rest needs more code. -fn expand_fcvt_from_uint(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { +fn expand_fcvt_from_uint( + inst: ir::Inst, + func: &mut ir::Function, + cfg: &mut ControlFlowGraph, + _isa: &isa::TargetIsa, +) { use ir::condcodes::IntCC; let x; @@ -227,7 +242,12 @@ fn expand_fcvt_from_uint(inst: ir::Inst, func: &mut ir::Function, cfg: &mut Cont cfg.recompute_ebb(pos.func, done); } -fn expand_fcvt_to_sint(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { +fn expand_fcvt_to_sint( + inst: ir::Inst, + func: &mut ir::Function, + cfg: &mut ControlFlowGraph, + _isa: &isa::TargetIsa, +) { use ir::condcodes::{IntCC, FloatCC}; use ir::immediates::{Ieee32, Ieee64}; @@ -303,7 +323,12 @@ fn expand_fcvt_to_sint(inst: ir::Inst, func: &mut ir::Function, cfg: &mut Contro cfg.recompute_ebb(pos.func, done); } -fn expand_fcvt_to_uint(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { +fn expand_fcvt_to_uint( + inst: ir::Inst, + func: &mut ir::Function, + cfg: &mut ControlFlowGraph, + _isa: &isa::TargetIsa, +) { use ir::condcodes::{IntCC, FloatCC}; use ir::immediates::{Ieee32, Ieee64}; diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index b4227d2550..f1ebf39a77 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -144,7 +144,8 @@ impl settings::Configurable for Builder { /// The `Encodings` iterator returns a legalization function to call. pub type Legalize = fn(ir::Inst, &mut ir::Function, - &mut flowgraph::ControlFlowGraph) + &mut flowgraph::ControlFlowGraph, + &TargetIsa) -> bool; /// Methods that are specialized to a target ISA. Implies a Display trait that shows the diff --git a/lib/cretonne/src/legalizer/globalvar.rs b/lib/cretonne/src/legalizer/globalvar.rs index 717be0d4f0..b945923f2f 100644 --- a/lib/cretonne/src/legalizer/globalvar.rs +++ b/lib/cretonne/src/legalizer/globalvar.rs @@ -6,9 +6,15 @@ use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::{self, InstBuilder}; +use isa::TargetIsa; /// Expand a `global_addr` instruction according to the definition of the global variable. -pub fn expand_global_addr(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlowGraph) { +pub fn expand_global_addr( + inst: ir::Inst, + func: &mut ir::Function, + _cfg: &mut ControlFlowGraph, + _isa: &TargetIsa, +) { // Unpack the instruction. let gv = match func.dfg[inst] { ir::InstructionData::UnaryGlobalVar { opcode, global_var } => { diff --git a/lib/cretonne/src/legalizer/heap.rs b/lib/cretonne/src/legalizer/heap.rs index 6dd5c11095..af40181d1a 100644 --- a/lib/cretonne/src/legalizer/heap.rs +++ b/lib/cretonne/src/legalizer/heap.rs @@ -7,9 +7,15 @@ use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::{self, InstBuilder, MemFlags}; use ir::condcodes::IntCC; +use isa::TargetIsa; /// Expand a `heap_addr` instruction according to the definition of the heap. -pub fn expand_heap_addr(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { +pub fn expand_heap_addr( + inst: ir::Inst, + func: &mut ir::Function, + cfg: &mut ControlFlowGraph, + _isa: &TargetIsa, +) { // Unpack the instruction. let (heap, offset, size) = match func.dfg[inst] { ir::InstructionData::HeapAddr { diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 8849e215e8..565f3e0971 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -80,7 +80,7 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is Ok(encoding) => pos.func.encodings[inst] = encoding, Err(action) => { // We should transform the instruction into legal equivalents. - let changed = action(inst, pos.func, cfg); + let changed = action(inst, pos.func, cfg, isa); // If the current instruction was replaced, we need to double back and revisit // the expanded sequence. This is both to assign encodings and possible to // expand further. @@ -114,7 +114,12 @@ include!(concat!(env!("OUT_DIR"), "/legalizer.rs")); /// Custom expansion for conditional trap instructions. /// TODO: Add CFG support to the Python patterns so we won't have to do this. -fn expand_cond_trap(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { +fn expand_cond_trap( + inst: ir::Inst, + func: &mut ir::Function, + cfg: &mut ControlFlowGraph, + _isa: &TargetIsa, +) { // Parse the instruction. let trapz; let (arg, code) = match func.dfg[inst] { @@ -159,7 +164,12 @@ fn expand_cond_trap(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFl } /// Jump tables. -fn expand_br_table(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { +fn expand_br_table( + inst: ir::Inst, + func: &mut ir::Function, + cfg: &mut ControlFlowGraph, + _isa: &TargetIsa, +) { use ir::condcodes::IntCC; let (arg, table) = match func.dfg[inst] { @@ -194,7 +204,12 @@ fn expand_br_table(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlo /// /// Conditional moves are available in some ISAs for some register classes. The remaining selects /// are handled by a branch. -fn expand_select(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph) { +fn expand_select( + inst: ir::Inst, + func: &mut ir::Function, + cfg: &mut ControlFlowGraph, + _isa: &TargetIsa, +) { let (ctrl, tval, fval) = match func.dfg[inst] { ir::InstructionData::Ternary { opcode: ir::Opcode::Select, @@ -226,7 +241,12 @@ fn expand_select(inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowG /// Expand illegal `f32const` and `f64const` instructions. -fn expand_fconst(inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlowGraph) { +fn expand_fconst( + inst: ir::Inst, + func: &mut ir::Function, + _cfg: &mut ControlFlowGraph, + _isa: &TargetIsa, +) { let ty = func.dfg.value_type(func.dfg.first_result(inst)); assert!(!ty.is_vector(), "Only scalar fconst supported: {}", ty); From 3ccc3f4f9b83ad2dc401420e1e8db13898cbd58b Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Mon, 12 Feb 2018 13:50:22 -0800 Subject: [PATCH 1501/3084] Add a stack_check instruction. This instruction loads a stack limit from a global variable and compares it to the stack pointer, trapping if the stack has grown beyond the limit. Also add a expand_flags transform group containing legalization patterns for ISAs with CPU flags. Fixes #234. --- cranelift/docs/langref.rst | 5 +++ .../filetests/isa/intel/legalize-custom.cton | 10 +++--- .../filetests/isa/intel/legalize-memory.cton | 14 ++++++++ lib/cretonne/meta/base/instructions.py | 15 ++++++++ lib/cretonne/meta/base/legalize.py | 28 ++++++++++++++- lib/cretonne/meta/isa/intel/encodings.py | 10 +++--- lib/cretonne/meta/isa/intel/legalize.py | 2 +- lib/cretonne/src/legalizer/mod.rs | 36 +++++++++++++++++++ 8 files changed, 107 insertions(+), 13 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index b7fe64b924..4001cf3688 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -511,6 +511,11 @@ instructions before instruction selection:: v9 = stack_addr ss3, 16 v1 = load.f64 v9 +When Cretonne code is running in a sandbox, it can also be necessary to include +stack overflow checks in the prologue. + +.. autoinst:: stack_check + Global variables ---------------- diff --git a/cranelift/filetests/isa/intel/legalize-custom.cton b/cranelift/filetests/isa/intel/legalize-custom.cton index 0327e0a317..41411ec21a 100644 --- a/cranelift/filetests/isa/intel/legalize-custom.cton +++ b/cranelift/filetests/isa/intel/legalize-custom.cton @@ -12,9 +12,8 @@ ebb0(v1: i32): trapz v1, user67 return ; check: $ebb0($v1: i32 - ; nextln: brnz $v1, $(new=$EBB) - ; nextln: trap user67 - ; check: $new: + ; nextln: $(f=$V) = ifcmp_imm $v1, 0 + ; nextln: trapif eq $f, user67 ; nextln: return } @@ -23,9 +22,8 @@ ebb0(v1: i32): trapnz v1, int_ovf return ; check: $ebb0($v1: i32 - ; nextln: brz $v1, $(new=$EBB) - ; nextln: trap int_ovf - ; check: $new: + ; nextln: $(f=$V) = ifcmp_imm $v1, 0 + ; nextln: trapif ne $f, int_ovf ; nextln: return } diff --git a/cranelift/filetests/isa/intel/legalize-memory.cton b/cranelift/filetests/isa/intel/legalize-memory.cton index fc17cfd2dc..1cdcf680ba 100644 --- a/cranelift/filetests/isa/intel/legalize-memory.cton +++ b/cranelift/filetests/isa/intel/legalize-memory.cton @@ -109,3 +109,17 @@ ebb0(v0: i32, v999: i64): ; nextln: $v2 = load.f32 $v1+0x7fff_ffff return v2 } + +; Stack overflow check. +; The stack limit is stored in a pointer-sized global variable. +function %stkchk(i64 vmctx) spiderwasm { + gv0 = vmctx+64 + +ebb0(v0: i64): + ; check: $ebb0( + stack_check gv0 + ; check: $(limit=$V) = load.i64 notrap aligned + ; check: $(flags=$V) = ifcmp_sp $limit + ; check: trapif uge $flags, stk_ovf + return +} diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index b6ab96a16e..89fd36812d 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -567,6 +567,21 @@ copy_special = Instruction( ins=(src, dst), other_side_effects=True) +GV = Operand( + 'GV', entities.global_var, doc=r""" + Global variable containing the stack limit. + """) + +stack_check = Instruction( + 'stack_check', r""" + Check for stack overflow. + + Read the stack limit from ``GV`` and compare it to the stack pointer. If + the stack pointer has reached or exceeded the limit, generate a trap with a + ``stk_ovf`` code. + """, + ins=GV, can_trap=True) + StackOffset = Operand('Offset', imm64, 'Offset from current stack pointer') adjust_sp_imm = Instruction( 'adjust_sp_imm', r""" diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index ebcda3fee1..b809708072 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -7,7 +7,7 @@ patterns that describe how base instructions can be transformed to other base instructions that are legal. """ from __future__ import absolute_import -from .immediates import intcc, ieee32, ieee64 +from .immediates import intcc, imm64, ieee32, ieee64 from . import instructions as insts from . import types from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm @@ -47,6 +47,15 @@ expand = XFormGroup('expand', """ operating on the same types as the original instructions. """) +expand_flags = XFormGroup('expand_flags', """ + Instruction expansions for architectures with flags. + + Expand some instructions using CPU flags, then fall back to the normal + expansions. Not all architectures support CPU flags, so these patterns + are kept separate. + """, chain=expand) + + # Custom expansions for memory objects. expand.custom_legalize(insts.global_addr, 'expand_global_addr') expand.custom_legalize(insts.heap_addr, 'expand_heap_addr') @@ -245,3 +254,20 @@ for ty, minus_zero in [ a2 << band(y, b), a << bor(a1, a2) )) + + +# Expansions using CPU flags. +expand_flags.custom_legalize(insts.stack_check, 'expand_stack_check') + +expand_flags.legalize( + insts.trapnz(x, c), + Rtl( + a << insts.ifcmp_imm(x, imm64(0)), + insts.trapif(intcc.ne, a, c) + )) +expand_flags.legalize( + insts.trapz(x, c), + Rtl( + a << insts.ifcmp_imm(x, imm64(0)), + insts.trapif(intcc.eq, a, c) + )) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 1e1eff820c..80614cb8b4 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -10,7 +10,7 @@ from . import recipes as r from . import settings as cfg from . import instructions as x86 from .legalize import intel_expand -from base.legalize import narrow, expand +from base.legalize import narrow, expand_flags from base.settings import allones_funcaddrs, is_pic from .settings import use_sse41 @@ -22,18 +22,18 @@ except ImportError: pass -I32.legalize_monomorphic(expand) +I32.legalize_monomorphic(expand_flags) I32.legalize_type( default=narrow, - b1=expand, + b1=expand_flags, i32=intel_expand, f32=intel_expand, f64=intel_expand) -I64.legalize_monomorphic(expand) +I64.legalize_monomorphic(expand_flags) I64.legalize_type( default=narrow, - b1=expand, + b1=expand_flags, i32=intel_expand, i64=intel_expand, f32=intel_expand, diff --git a/lib/cretonne/meta/isa/intel/legalize.py b/lib/cretonne/meta/isa/intel/legalize.py index 70a8e9166b..f6e5b1dc9d 100644 --- a/lib/cretonne/meta/isa/intel/legalize.py +++ b/lib/cretonne/meta/isa/intel/legalize.py @@ -18,7 +18,7 @@ intel_expand = XFormGroup( Use Intel-specific instructions if needed. """, - isa=ISA, chain=shared.expand) + isa=ISA, chain=shared.expand_flags) a = Var('a') dead = Var('dead') diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 565f3e0971..81dde2f457 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -267,3 +267,39 @@ fn expand_fconst( }; pos.func.dfg.replace(inst).bitcast(ty, ival); } + +/// Expand the stack check instruction. +pub fn expand_stack_check( + inst: ir::Inst, + func: &mut ir::Function, + _cfg: &mut ControlFlowGraph, + isa: &TargetIsa, +) { + use ir::condcodes::IntCC; + + let gv = match func.dfg[inst] { + ir::InstructionData::UnaryGlobalVar { global_var, .. } => global_var, + _ => panic!("Want stack_check: {}", func.dfg.display_inst(inst, isa)), + }; + let ptr_ty = if isa.flags().is_64bit() { + ir::types::I64 + } else { + ir::types::I32 + }; + + let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); + + let limit_addr = pos.ins().global_addr(ptr_ty, gv); + + let mut mflags = ir::MemFlags::new(); + mflags.set_aligned(); + mflags.set_notrap(); + let limit = pos.ins().load(ptr_ty, mflags, limit_addr, 0); + let cflags = pos.ins().ifcmp_sp(limit); + pos.func.dfg.replace(inst).trapif( + IntCC::UnsignedGreaterThanOrEqual, + cflags, + ir::TrapCode::StackOverflow, + ); +} From 42e1616b82f9e7f39aa6d08ad845c8985559a24a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 13 Feb 2018 20:14:01 -0800 Subject: [PATCH 1502/3084] Update to wasmparser 0.14.1. --- lib/wasm/Cargo.toml | 2 +- lib/wasm/src/code_translator.rs | 75 +++++++++++++++++++++++++++++ lib/wasm/src/sections_translator.rs | 8 ++- lib/wasm/src/translation_utils.rs | 2 + 4 files changed, 85 insertions(+), 2 deletions(-) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 2aa984d720..56d49d7cf6 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -11,7 +11,7 @@ license = "Apache-2.0" name = "cton_wasm" [dependencies] -wasmparser = "0.13.0" +wasmparser = "0.14.1" cretonne = { path = "../cretonne" } cretonne-frontend = { path = "../frontend" } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 1e65f2e607..903b9cd9e1 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -618,6 +618,13 @@ pub fn translate_operator( let val = state.pop1(); state.push1(builder.ins().bitcast(I64, val)); } + Operator::I32Extend8S | + Operator::I32Extend16S | + Operator::I64Extend8S | + Operator::I64Extend16S | + Operator::I64Extend32S => { + panic!("proposed sign-extend operators not yet supported"); + } /****************************** Binary Operators ************************************/ Operator::I32Add | Operator::I64Add => { let (arg1, arg2) = state.pop2(); @@ -814,6 +821,74 @@ pub fn translate_operator( let val = builder.ins().fcmp(FloatCC::LessThanOrEqual, arg1, arg2); state.push1(builder.ins().bint(I32, val)); } + Operator::Wake { .. } | + Operator::I32Wait { .. } | + Operator::I64Wait { .. } | + Operator::I32AtomicLoad { .. } | + Operator::I64AtomicLoad { .. } | + Operator::I32AtomicLoad8U { .. } | + Operator::I32AtomicLoad16U { .. } | + Operator::I64AtomicLoad8U { .. } | + Operator::I64AtomicLoad16U { .. } | + Operator::I64AtomicLoad32U { .. } | + Operator::I32AtomicStore { .. } | + Operator::I64AtomicStore { .. } | + Operator::I32AtomicStore8 { .. } | + Operator::I32AtomicStore16 { .. } | + Operator::I64AtomicStore8 { .. } | + Operator::I64AtomicStore16 { .. } | + Operator::I64AtomicStore32 { .. } | + Operator::I32AtomicRmwAdd { .. } | + Operator::I64AtomicRmwAdd { .. } | + Operator::I32AtomicRmw8UAdd { .. } | + Operator::I32AtomicRmw16UAdd { .. } | + Operator::I64AtomicRmw8UAdd { .. } | + Operator::I64AtomicRmw16UAdd { .. } | + Operator::I64AtomicRmw32UAdd { .. } | + Operator::I32AtomicRmwSub { .. } | + Operator::I64AtomicRmwSub { .. } | + Operator::I32AtomicRmw8USub { .. } | + Operator::I32AtomicRmw16USub { .. } | + Operator::I64AtomicRmw8USub { .. } | + Operator::I64AtomicRmw16USub { .. } | + Operator::I64AtomicRmw32USub { .. } | + Operator::I32AtomicRmwAnd { .. } | + Operator::I64AtomicRmwAnd { .. } | + Operator::I32AtomicRmw8UAnd { .. } | + Operator::I32AtomicRmw16UAnd { .. } | + Operator::I64AtomicRmw8UAnd { .. } | + Operator::I64AtomicRmw16UAnd { .. } | + Operator::I64AtomicRmw32UAnd { .. } | + Operator::I32AtomicRmwOr { .. } | + Operator::I64AtomicRmwOr { .. } | + Operator::I32AtomicRmw8UOr { .. } | + Operator::I32AtomicRmw16UOr { .. } | + Operator::I64AtomicRmw8UOr { .. } | + Operator::I64AtomicRmw16UOr { .. } | + Operator::I64AtomicRmw32UOr { .. } | + Operator::I32AtomicRmwXor { .. } | + Operator::I64AtomicRmwXor { .. } | + Operator::I32AtomicRmw8UXor { .. } | + Operator::I32AtomicRmw16UXor { .. } | + Operator::I64AtomicRmw8UXor { .. } | + Operator::I64AtomicRmw16UXor { .. } | + Operator::I64AtomicRmw32UXor { .. } | + Operator::I32AtomicRmwXchg { .. } | + Operator::I64AtomicRmwXchg { .. } | + Operator::I32AtomicRmw8UXchg { .. } | + Operator::I32AtomicRmw16UXchg { .. } | + Operator::I64AtomicRmw8UXchg { .. } | + Operator::I64AtomicRmw16UXchg { .. } | + Operator::I64AtomicRmw32UXchg { .. } | + Operator::I32AtomicRmwCmpxchg { .. } | + Operator::I64AtomicRmwCmpxchg { .. } | + Operator::I32AtomicRmw8UCmpxchg { .. } | + Operator::I32AtomicRmw16UCmpxchg { .. } | + Operator::I64AtomicRmw8UCmpxchg { .. } | + Operator::I64AtomicRmw16UCmpxchg { .. } | + Operator::I64AtomicRmw32UCmpxchg { .. } => { + panic!("proposed thread operators not yet supported"); + } } } diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 21f5abbdb5..d02a4a7c2a 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -75,11 +75,16 @@ pub fn parse_import_section<'data>( environ.declare_func_import(sig as SignatureIndex, module_name, field_name); } ParserState::ImportSectionEntry { - ty: ImportSectionEntryType::Memory(MemoryType { limits: ref memlimits }), .. + ty: ImportSectionEntryType::Memory(MemoryType { + limits: ref memlimits, + shared, + }), + .. } => { environ.declare_memory(Memory { pages_count: memlimits.initial as usize, maximum: memlimits.maximum.map(|x| x as usize), + shared, }); } ParserState::ImportSectionEntry { @@ -186,6 +191,7 @@ pub fn parse_memory_section( environ.declare_memory(Memory { pages_count: ty.limits.initial as usize, maximum: ty.limits.maximum.map(|x| x as usize), + shared: ty.shared, }); } ParserState::EndSection => break, diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 8b1bc5fac2..4c70d708ed 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -67,6 +67,8 @@ pub struct Memory { pub pages_count: usize, /// The maximum number of pages in the memory. pub maximum: Option, + /// Whether the memory may be shared between multiple threads. + pub shared: bool, } /// Wrapper to a `get_local` and `set_local` index. They are WebAssembly's non-SSA variables. From ed24320eda181720f48b619c7053b304a84d810d Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Wed, 14 Feb 2018 11:51:40 -0800 Subject: [PATCH 1503/3084] gen_settings: dont try to display a Preset descriptor in Flags (#241) * gen_settings: dont try to display a Preset descriptor in Flags Trying to display a preset doesnt make sense, and before this commit it does not display anything meaningful - the printout just says e.g. "haswell =\n". The offset byte a preset descriptor isnt a valid offset into the flag bytes, it is actually an offset into the PRESETS table. It will cause a panic when the offset is out of bounds for the flag bytes, which happens in the intel isa as of this commit. * intel settings: test that display impl doesnt panic --- lib/cretonne/meta/gen_settings.py | 11 ++++++----- lib/cretonne/src/isa/intel/settings.rs | 19 +++++++++++++++++++ lib/cretonne/src/settings.rs | 11 +++++++++++ 3 files changed, 36 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index 9f80a391ff..693b7baa76 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -218,11 +218,12 @@ def gen_display(sgrp, fmt): '}'): fmt.line('writeln!(f, "[{}]")?;'.format(sgrp.name)) with fmt.indented('for d in &DESCRIPTORS {', '}'): - fmt.line('write!(f, "{} = ", d.name)?;') - fmt.line( - 'TEMPLATE.format_toml_value(d.detail,' + - 'self.bytes[d.offset as usize], f)?;') - fmt.line('writeln!(f, "")?;') + with fmt.indented('if !d.detail.is_preset() {', '}'): + fmt.line('write!(f, "{} = ", d.name)?;') + fmt.line( + 'TEMPLATE.format_toml_value(d.detail,' + + 'self.bytes[d.offset as usize], f)?;') + fmt.line('writeln!(f, "")?;') fmt.line('Ok(())') diff --git a/lib/cretonne/src/isa/intel/settings.rs b/lib/cretonne/src/isa/intel/settings.rs index 7ca4886588..f933e91b87 100644 --- a/lib/cretonne/src/isa/intel/settings.rs +++ b/lib/cretonne/src/isa/intel/settings.rs @@ -30,4 +30,23 @@ mod tests { assert_eq!(f2.has_sse41(), true); assert_eq!(f2.has_bmi1(), true); } + #[test] + fn display_presets() { + // Spot check that the flags Display impl does not cause a panic + let shared = settings::Flags::new(&settings::builder()); + + let b1 = builder(); + let f1 = Flags::new(&shared, &b1); + let _ = format!("{}", f1); + + let mut b2 = builder(); + b2.enable("nehalem").unwrap(); + let f2 = Flags::new(&shared, &b1); + let _ = format!("{}", f2); + + let mut b3 = builder(); + b3.enable("haswell").unwrap(); + let f3 = Flags::new(&shared, &b1); + let _ = format!("{}", f3); + } } diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index f08bea95c2..cc9f91f452 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -297,6 +297,17 @@ pub mod detail { /// The `Descriptor::offset` field refers to the `PRESETS` table. Preset, } + + impl Detail { + /// Check if a detail is a Detail::Preset. Useful because the Descriptor + /// offset field has a different meaning when the detail is a preset. + pub fn is_preset(&self) -> bool { + match self { + &Detail::Preset => true, + _ => false, + } + } + } } // Include code generated by `meta/gen_settings.py`. This file contains a public `Flags` struct From a9e799debb07f33a434fd16407cd833c70627888 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Feb 2018 11:23:37 -0800 Subject: [PATCH 1504/3084] Add an avoid_div_traps setting. This enables code generation that never causes a SIGFPE signal to be raised from a division instruction. Instead, division and remainder calculations are protected by explicit traps. --- .../isa/intel/legalize-div-traps.cton | 71 ++++++++++ .../filetests/isa/intel/legalize-div.cton | 58 ++++++++ lib/cretonne/meta/base/settings.py | 12 ++ lib/cretonne/meta/isa/intel/legalize.py | 30 +---- lib/cretonne/src/isa/intel/enc_tables.rs | 125 ++++++++++++++---- lib/cretonne/src/settings.rs | 1 + 6 files changed, 247 insertions(+), 50 deletions(-) create mode 100644 cranelift/filetests/isa/intel/legalize-div-traps.cton create mode 100644 cranelift/filetests/isa/intel/legalize-div.cton diff --git a/cranelift/filetests/isa/intel/legalize-div-traps.cton b/cranelift/filetests/isa/intel/legalize-div-traps.cton new file mode 100644 index 0000000000..0da41bc5ad --- /dev/null +++ b/cranelift/filetests/isa/intel/legalize-div-traps.cton @@ -0,0 +1,71 @@ +; Test the division legalizations. +test legalizer +set is_64bit +; See also legalize-div.cton. +set avoid_div_traps=1 +isa intel + +; regex: V=v\d+ +; regex: EBB=ebb\d+ + +function %udiv(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + ; check: $ebb0( + v2 = udiv v0, v1 + ; nextln: $(fz=$V) = ifcmp_imm $v1, 0 + ; nextln: trapif eq $fz, int_divz + ; nextln: $(hi=$V) = iconst.i64 0 + ; nextln: $(d=$V), $(r=$V) = x86_udivmodx $v0, $hi, $v1 + return v2 + ; nextln: return $d +} + +function %urem(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + ; check: $ebb0( + v2 = urem v0, v1 + ; nextln: $(fz=$V) = ifcmp_imm $v1, 0 + ; nextln: trapif eq $fz, int_divz + ; nextln: $(hi=$V) = iconst.i64 0 + ; nextln: $(d=$V), $(r=$V) = x86_udivmodx $v0, $hi, $v1 + return v2 + ; nextln: return $r +} + +function %sdiv(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + ; check: $ebb0( + v2 = sdiv v0, v1 + ; nextln: $(fm1=$V) = ifcmp_imm $v1, -1 + ; nextln: brif eq $fm1, $(m1=$EBB) + ; nextln: $(fz=$V) = ifcmp_imm $v1, 0 + ; nextln: trapif eq $fz, int_divz + ; check: $(hi=$V) = sshr + ; nextln: $(q=$V), $(r=$V) = x86_sdivmodx $v0, $hi, $v1 + ; nextln: jump $(done=$EBB)($q) + ; check: $m1: + ; nextln: $(fm=$V) = ifcmp_imm.i64 $v0, 0x8000_0000_0000_0000 + ; nextln: trapif eq $fm, int_ovf + ; check: $done($v2: i64): + return v2 + ; nextln: return $v2 +} + +; The srem expansion needs to special-case x % -1 since x86_sdivmodx traps on INT_MIN/-1. +; TODO: Add more explicit pattern matching once we've cleaned up the ifcmp+brif pattern. +function %srem(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + ; check: $ebb0( + v2 = srem v0, v1 + ; nextln: $(fm1=$V) = ifcmp_imm $v1, -1 + ; nextln: brif eq $fm1, $(m1=$EBB) + ; check: $(hi=$V) = sshr + ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx $v0, $hi, $v1 + ; nextln: jump $(done=$EBB)($r) + ; check: $m1: + ; nextln: $(zero=$V) = iconst.i64 0 + ; nextln: jump $(done=$EBB)($zero) + ; check: $done($v2: i64): + return v2 + ; nextln: return $v2 +} diff --git a/cranelift/filetests/isa/intel/legalize-div.cton b/cranelift/filetests/isa/intel/legalize-div.cton new file mode 100644 index 0000000000..8e34c01b86 --- /dev/null +++ b/cranelift/filetests/isa/intel/legalize-div.cton @@ -0,0 +1,58 @@ +; Test the division legalizations. +test legalizer +set is_64bit +; See also legalize-div-traps.cton. +set avoid_div_traps=0 +isa intel + +; regex: V=v\d+ +; regex: EBB=ebb\d+ + +function %udiv(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + ; check: $ebb0( + v2 = udiv v0, v1 + ; nextln: $(hi=$V) = iconst.i64 0 + ; nextln: $(d=$V), $(r=$V) = x86_udivmodx $v0, $hi, $v1 + return v2 + ; nextln: return $d +} + +function %urem(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + ; check: $ebb0( + v2 = urem v0, v1 + ; nextln: $(hi=$V) = iconst.i64 0 + ; nextln: $(d=$V), $(r=$V) = x86_udivmodx $v0, $hi, $v1 + return v2 + ; nextln: return $r +} + +function %sdiv(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + ; check: $ebb0( + v2 = sdiv v0, v1 + ; check: $(hi=$V) = sshr + ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx $v0, $hi, $v1 + return v2 + ; nextln: return $d +} + +; The srem expansion needs to special-case x % -1 since x86_sdivmodx traps on INT_MIN/-1. +; TODO: Add more explicit pattern matching once we've cleaned up the ifcmp+brif pattern. +function %srem(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): + ; check: $ebb0( + v2 = srem v0, v1 + ; nextln: $(fm1=$V) = ifcmp_imm $v1, -1 + ; nextln: brif eq $fm1, $(m1=$EBB) + ; check: $(hi=$V) = sshr + ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx $v0, $hi, $v1 + ; nextln: jump $(done=$EBB)($r) + ; check: $m1: + ; nextln: $(zero=$V) = iconst.i64 0 + ; nextln: jump $(done=$EBB)($zero) + ; check: $done($v2: i64): + return v2 + ; nextln: return $v2 +} diff --git a/lib/cretonne/meta/base/settings.py b/lib/cretonne/meta/base/settings.py index 68df47d4eb..8d0522af79 100644 --- a/lib/cretonne/meta/base/settings.py +++ b/lib/cretonne/meta/base/settings.py @@ -41,6 +41,18 @@ return_at_end = BoolSetting( instruction at the end. """) +avoid_div_traps = BoolSetting( + """ + Generate explicit checks around native division instructions to avoid + their trapping. + + This is primarily used by SpiderMonkey which doesn't install a signal + handler for SIGFPE, but expects a SIGILL trap for division by zero. + + On ISAs like ARM where the native division instructions don't trap, + this setting has no effect - explicit checks are always inserted. + """) + is_compressed = BoolSetting("Enable compressed instructions") enable_float = BoolSetting( diff --git a/lib/cretonne/meta/isa/intel/legalize.py b/lib/cretonne/meta/isa/intel/legalize.py index f6e5b1dc9d..32f0a98153 100644 --- a/lib/cretonne/meta/isa/intel/legalize.py +++ b/lib/cretonne/meta/isa/intel/legalize.py @@ -5,7 +5,6 @@ from __future__ import absolute_import from cdsl.ast import Var from cdsl.xform import Rtl, XFormGroup from base.immediates import imm64, intcc, floatcc -from base.types import i32, i64 from base import legalize as shared from base import instructions as insts from . import instructions as x86 @@ -31,31 +30,12 @@ a2 = Var('a2') # # Division and remainder. # -intel_expand.legalize( - a << insts.udiv(x, y), - Rtl( - xhi << insts.iconst(imm64(0)), - (a, dead) << x86.udivmodx(x, xhi, y) - )) - -intel_expand.legalize( - a << insts.urem(x, y), - Rtl( - xhi << insts.iconst(imm64(0)), - (dead, a) << x86.udivmodx(x, xhi, y) - )) - -for ty in [i32, i64]: - intel_expand.legalize( - a << insts.sdiv.bind(ty)(x, y), - Rtl( - xhi << insts.sshr_imm(x, imm64(ty.lane_bits() - 1)), - (a, dead) << x86.sdivmodx(x, xhi, y) - )) - # The srem expansion requires custom code because srem INT_MIN, -1 is not -# allowed to trap. -intel_expand.custom_legalize(insts.srem, 'expand_srem') +# allowed to trap. The other ops need to check avoid_div_traps. +intel_expand.custom_legalize(insts.sdiv, 'expand_sdivrem') +intel_expand.custom_legalize(insts.srem, 'expand_sdivrem') +intel_expand.custom_legalize(insts.udiv, 'expand_udivrem') +intel_expand.custom_legalize(insts.urem, 'expand_udivrem') # Floating point condition codes. # diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index a97607c4c7..98de07db48 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -1,9 +1,9 @@ //! Encoding tables for Intel ISAs. -use bitset::BitSet; use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::{self, InstBuilder}; +use ir::condcodes::IntCC; use isa::constraints::*; use isa::enc_tables::*; use isa::encoding::RecipeSizing; @@ -14,55 +14,87 @@ use super::registers::*; include!(concat!(env!("OUT_DIR"), "/encoding-intel.rs")); include!(concat!(env!("OUT_DIR"), "/legalize-intel.rs")); -/// Expand the `srem` instruction using `x86_sdivmodx`. -fn expand_srem( +/// Expand the `sdiv` and `srem` instructions using `x86_sdivmodx`. +fn expand_sdivrem( inst: ir::Inst, func: &mut ir::Function, cfg: &mut ControlFlowGraph, - _isa: &isa::TargetIsa, + isa: &isa::TargetIsa, ) { - use ir::condcodes::IntCC; - let (x, y) = match func.dfg[inst] { + let (x, y, is_srem) = match func.dfg[inst] { + ir::InstructionData::Binary { + opcode: ir::Opcode::Sdiv, + args, + } => (args[0], args[1], false), ir::InstructionData::Binary { opcode: ir::Opcode::Srem, args, - } => (args[0], args[1]), - _ => panic!("Need srem: {}", func.dfg.display_inst(inst, None)), + } => (args[0], args[1], true), + _ => panic!("Need sdiv/srem: {}", func.dfg.display_inst(inst, None)), }; + let avoid_div_traps = isa.flags().avoid_div_traps(); let old_ebb = func.layout.pp_ebb(inst); - - // EBB handling the -1 divisor case. - let minus_one = func.dfg.make_ebb(); - - // Final EBB with one argument representing the final result value. - let done = func.dfg.make_ebb(); - - // Move the `inst` result value onto the `done` EBB. let result = func.dfg.first_result(inst); let ty = func.dfg.value_type(result); - func.dfg.clear_results(inst); - func.dfg.attach_ebb_param(done, result); let mut pos = FuncCursor::new(func).at_inst(inst); pos.use_srcloc(inst); + pos.func.dfg.clear_results(inst); + + // If we can tolerate native division traps, sdiv doesn't need branching. + if !avoid_div_traps && !is_srem { + let xhi = pos.ins().sshr_imm(x, i64::from(ty.lane_bits()) - 1); + pos.ins().with_result(result).x86_sdivmodx(x, xhi, y); + pos.remove_inst(); + return; + } + + // EBB handling the -1 divisor case. + let minus_one = pos.func.dfg.make_ebb(); + + // Final EBB with one argument representing the final result value. + let done = pos.func.dfg.make_ebb(); + + // Move the `inst` result value onto the `done` EBB. + pos.func.dfg.attach_ebb_param(done, result); // Start by checking for a -1 divisor which needs to be handled specially. - let is_m1 = pos.ins().icmp_imm(IntCC::Equal, y, -1); - pos.ins().brnz(is_m1, minus_one, &[]); + let is_m1 = pos.ins().ifcmp_imm(y, -1); + pos.ins().brif(IntCC::Equal, is_m1, minus_one, &[]); + + // Put in an explicit division-by-zero trap if the environment requires it. + if avoid_div_traps { + pos.ins().trapz(y, ir::TrapCode::IntegerDivisionByZero); + } // Now it is safe to execute the `x86_sdivmodx` instruction which will still trap on division // by zero. let xhi = pos.ins().sshr_imm(x, i64::from(ty.lane_bits()) - 1); - let (_qout, rem) = pos.ins().x86_sdivmodx(x, xhi, y); - pos.ins().jump(done, &[rem]); + let (quot, rem) = pos.ins().x86_sdivmodx(x, xhi, y); + let divres = if is_srem { rem } else { quot }; + pos.ins().jump(done, &[divres]); - // Now deal with the -1 divisor which always yields a 0 remainder. + // Now deal with the -1 divisor case. pos.insert_ebb(minus_one); - let zero = pos.ins().iconst(ty, 0); + let m1_result = if is_srem { + // x % -1 = 0. + pos.ins().iconst(ty, 0) + } else { + // Explicitly check for overflow: Trap when x == INT_MIN. + debug_assert!(avoid_div_traps, "Native trapping divide handled above"); + let f = pos.ins().ifcmp_imm(x, -1 << (ty.lane_bits() - 1)); + pos.ins().trapif( + IntCC::Equal, + f, + ir::TrapCode::IntegerOverflow, + ); + // x / -1 = -x. + pos.ins().irsub_imm(x, 0) + }; // Recycle the original instruction as a jump. - pos.func.dfg.replace(inst).jump(done, &[zero]); + pos.func.dfg.replace(inst).jump(done, &[m1_result]); // Finally insert a label for the completion. pos.next_inst(); @@ -73,6 +105,49 @@ fn expand_srem( cfg.recompute_ebb(pos.func, done); } +/// Expand the `udiv` and `urem` instructions using `x86_udivmodx`. +fn expand_udivrem( + inst: ir::Inst, + func: &mut ir::Function, + _cfg: &mut ControlFlowGraph, + isa: &isa::TargetIsa, +) { + + let (x, y, is_urem) = match func.dfg[inst] { + ir::InstructionData::Binary { + opcode: ir::Opcode::Udiv, + args, + } => (args[0], args[1], false), + ir::InstructionData::Binary { + opcode: ir::Opcode::Urem, + args, + } => (args[0], args[1], true), + _ => panic!("Need udiv/urem: {}", func.dfg.display_inst(inst, None)), + }; + let avoid_div_traps = isa.flags().avoid_div_traps(); + let result = func.dfg.first_result(inst); + let ty = func.dfg.value_type(result); + + let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); + pos.func.dfg.clear_results(inst); + + // Put in an explicit division-by-zero trap if the environment requires it. + if avoid_div_traps { + pos.ins().trapz(y, ir::TrapCode::IntegerDivisionByZero); + } + + // Now it is safe to execute the `x86_udivmodx` instruction. + let xhi = pos.ins().iconst(ty, 0); + let reuse = if is_urem { + [None, Some(result)] + } else { + [Some(result), None] + }; + pos.ins().with_results(reuse).x86_udivmodx(x, xhi, y); + pos.remove_inst(); +} + /// Expand the `fmin` and `fmax` instructions using the Intel `x86_fmin` and `x86_fmax` /// instructions. fn expand_minmax( diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index cc9f91f452..36a48e04b4 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -360,6 +360,7 @@ mod tests { is_64bit = false\n\ is_pic = false\n\ return_at_end = false\n\ + avoid_div_traps = false\n\ is_compressed = false\n\ enable_float = true\n\ enable_simd = true\n\ From a6ab90f20540287276758a5632dc0ce11d77f7ad Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Fri, 16 Feb 2018 15:50:36 -0800 Subject: [PATCH 1505/3084] Legalize irsub_imm. --- lib/cretonne/meta/base/legalize.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index b809708072..511bee9a07 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -11,7 +11,7 @@ from .immediates import intcc, imm64, ieee32, ieee64 from . import instructions as insts from . import types from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm -from .instructions import isub, isub_bin, isub_bout, isub_borrow +from .instructions import isub, isub_bin, isub_bout, isub_borrow, irsub_imm from .instructions import imul, imul_imm from .instructions import band, bor, bxor, isplit, iconcat from .instructions import bnot, band_not, bor_not, bxor_not @@ -196,6 +196,12 @@ for inst_imm, inst in [ a1 << iconst(y), a << inst(x, a1) )) +expand.legalize( + a << irsub_imm(y, x), + Rtl( + a1 << iconst(x), + a << isub(a1, y) + )) # Rotates and shifts. for inst_imm, inst in [ From ad896d97906ede1b52c8a97a1feb00520c7ee74c Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Feb 2018 10:47:46 -0800 Subject: [PATCH 1506/3084] Add more legalization patterns for *_imm instructions. When the imediate value is out of range for the legal encodings, convert these instructions to an iconst followed by their register counterparts. --- .../filetests/isa/intel/legalize-div-traps.cton | 3 ++- lib/cretonne/meta/base/legalize.py | 15 +++++++++++++-- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/cranelift/filetests/isa/intel/legalize-div-traps.cton b/cranelift/filetests/isa/intel/legalize-div-traps.cton index 0da41bc5ad..2077ab9bf8 100644 --- a/cranelift/filetests/isa/intel/legalize-div-traps.cton +++ b/cranelift/filetests/isa/intel/legalize-div-traps.cton @@ -44,7 +44,8 @@ ebb0(v0: i64, v1: i64): ; nextln: $(q=$V), $(r=$V) = x86_sdivmodx $v0, $hi, $v1 ; nextln: jump $(done=$EBB)($q) ; check: $m1: - ; nextln: $(fm=$V) = ifcmp_imm.i64 $v0, 0x8000_0000_0000_0000 + ; nextln: $(imin=$V) = iconst.i64 0x8000_0000_0000_0000 + ; nextln: $(fm=$V) = ifcmp.i64 $v0, $imin ; nextln: trapif eq $fm, int_ovf ; check: $done($v2: i64): return v2 diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index 511bee9a07..ecc8d5a436 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -13,9 +13,12 @@ from . import types from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm from .instructions import isub, isub_bin, isub_bout, isub_borrow, irsub_imm from .instructions import imul, imul_imm +from .instructions import sdiv, sdiv_imm, udiv, udiv_imm +from .instructions import srem, srem_imm, urem, urem_imm from .instructions import band, bor, bxor, isplit, iconcat from .instructions import bnot, band_not, bor_not, bxor_not -from .instructions import icmp, icmp_imm +from .instructions import band_imm, bor_imm, bxor_imm +from .instructions import icmp, icmp_imm, ifcmp, ifcmp_imm from .instructions import iconst, bint, select from .instructions import ishl, ishl_imm, sshr, sshr_imm, ushr, ushr_imm from .instructions import rotl, rotl_imm, rotr, rotr_imm @@ -189,7 +192,15 @@ expand.legalize( # Expansions for immediate operands that are out of range. for inst_imm, inst in [ (iadd_imm, iadd), - (imul_imm, imul)]: + (imul_imm, imul), + (sdiv_imm, sdiv), + (udiv_imm, udiv), + (srem_imm, srem), + (urem_imm, urem), + (band_imm, band), + (bor_imm, bor), + (bxor_imm, bor), + (ifcmp_imm, ifcmp)]: expand.legalize( a << inst_imm(x, y), Rtl( From 21215529fe4f9221d152e265f4410de93727f980 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 20 Feb 2018 12:15:23 -0800 Subject: [PATCH 1507/3084] Simplify the code in make_inst_results_reusing; NFC. --- lib/cretonne/src/ir/dfg.rs | 37 ++++++++++++++++++------------------- 1 file changed, 18 insertions(+), 19 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 923579d45d..1c3a0fce3c 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -452,29 +452,15 @@ impl DataFlowGraph { I: Iterator>, { let mut reuse = reuse.fuse(); - let constraints = self.insts[inst].opcode().constraints(); - let fixed_results = constraints.fixed_results(); - let mut total_results = fixed_results; self.results[inst].clear(&mut self.value_lists); - // The fixed results will appear at the front of the list. - for res_idx in 0..fixed_results { - let ty = constraints.result_type(res_idx, ctrl_typevar); - if let Some(Some(v)) = reuse.next() { - debug_assert_eq!(self.value_type(v), ty, "Reused {} is wrong type", ty); - self.attach_result(inst, v); - } else { - self.append_result(inst, ty); - } - } - // Get the call signature if this is a function call. if let Some(sig) = self.call_signature(inst) { // Create result values corresponding to the call return types. - let var_results = self.signatures[sig].returns.len(); - total_results += var_results; - for res_idx in 0..var_results { + debug_assert_eq!(self.insts[inst].opcode().constraints().fixed_results(), 0); + let num_results = self.signatures[sig].returns.len(); + for res_idx in 0..num_results { let ty = self.signatures[sig].returns[res_idx].value_type; if let Some(Some(v)) = reuse.next() { debug_assert_eq!(self.value_type(v), ty, "Reused {} is wrong type", ty); @@ -483,9 +469,22 @@ impl DataFlowGraph { self.append_result(inst, ty); } } + num_results + } else { + // Create result values corresponding to the opcode's constraints. + let constraints = self.insts[inst].opcode().constraints(); + let num_results = constraints.fixed_results(); + for res_idx in 0..num_results { + let ty = constraints.result_type(res_idx, ctrl_typevar); + if let Some(Some(v)) = reuse.next() { + debug_assert_eq!(self.value_type(v), ty, "Reused {} is wrong type", ty); + self.attach_result(inst, v); + } else { + self.append_result(inst, ty); + } + } + num_results } - - total_results } /// Create a `ReplaceBuilder` that will replace `inst` with a new instruction in place. From 1e56d4446560fef29ecb97a2f0a3348e05cbb63e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Feb 2018 13:16:36 -0800 Subject: [PATCH 1508/3084] Remember the positive numbers in expand_fcvt_to_sint. We can get an INT_MIN result when converting a positive number that is too large to fit in an integer too. --- lib/cretonne/src/isa/intel/enc_tables.rs | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 98de07db48..238088698e 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -391,6 +391,15 @@ fn expand_fcvt_to_sint( let overflow = pos.ins().fcmp(overflow_cc, x, flimit); pos.ins().trapnz(overflow, ir::TrapCode::IntegerOverflow); + // Finally, we could have a positive value that is too large. + let fzero = match xty { + ir::types::F32 => pos.ins().f32const(Ieee32::with_float(0.0)), + ir::types::F64 => pos.ins().f64const(Ieee64::with_float(0.0)), + _ => panic!("Can't convert {}", xty), + }; + let overflow = pos.ins().fcmp(FloatCC::GreaterThanOrEqual, x, fzero); + pos.ins().trapnz(overflow, ir::TrapCode::IntegerOverflow); + pos.ins().jump(done, &[]); pos.insert_ebb(done); From b9b1d0fcd5024c49505b7c4984340bb1478eb75e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Feb 2018 14:35:41 -0800 Subject: [PATCH 1509/3084] Add a trapff instruction. This is the floating point equivalent of trapif: Trap when a given condition is in the floating-point flags. Define Intel encodings comparable to the trapif encodings. --- cranelift/docs/langref.rst | 1 + .../filetests/isa/intel/binary32-float.cton | 17 +++++++++++++++++ .../filetests/isa/intel/binary64-float.cton | 17 +++++++++++++++++ cranelift/filetests/parser/tiny.cton | 6 ++++++ lib/cretonne/meta/base/formats.py | 1 + lib/cretonne/meta/base/instructions.py | 9 +++++++++ lib/cretonne/meta/isa/intel/encodings.py | 2 ++ lib/cretonne/meta/isa/intel/recipes.py | 16 +++++++++++++++- lib/cretonne/src/ir/instructions.rs | 6 ++++++ lib/cretonne/src/verifier/mod.rs | 1 + lib/cretonne/src/write.rs | 1 + lib/reader/src/parser.rs | 15 +++++++++++++++ 12 files changed, 91 insertions(+), 1 deletion(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 4001cf3688..d80554dc67 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -381,6 +381,7 @@ is zero. .. autoinst:: trapz .. autoinst:: trapnz .. autoinst:: trapif +.. autoinst:: trapff Function calls diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 9b12506b80..8f60cf20be 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -470,6 +470,23 @@ ebb1: ; asm: jbe ebb1 brff ule v1, ebb1 ; bin: 76 f0 + ; asm: jp .+4; ud2 + trapff ord v1, user0 ; bin: 7a 02 0f 0b + ; asm: jnp .+4; ud2 + trapff uno v1, user0 ; bin: 7b 02 0f 0b + ; asm: je .+4; ud2 + trapff one v1, user0 ; bin: 74 02 0f 0b + ; asm: jne .+4; ud2 + trapff ueq v1, user0 ; bin: 75 02 0f 0b + ; asm: jna .+4; ud2 + trapff gt v1, user0 ; bin: 76 02 0f 0b + ; asm: jnae .+4; ud2 + trapff ge v1, user0 ; bin: 72 02 0f 0b + ; asm: jnb .+4; ud2 + trapff ult v1, user0 ; bin: 73 02 0f 0b + ; asm: jnbe .+4; ud2 + trapff ule v1, user0 ; bin: 77 02 0f 0b + ; asm: setnp %bl [-,%rbx] v10 = trueff ord v1 ; bin: 0f 9b c3 ; asm: setp %bl diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index 8ca4efd8d8..a6f47e53db 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -494,6 +494,23 @@ ebb1: ; asm: jbe ebb1 brff ule v1, ebb1 ; bin: 76 f0 + ; asm: jp .+4; ud2 + trapff ord v1, user0 ; bin: 7a 02 0f 0b + ; asm: jnp .+4; ud2 + trapff uno v1, user0 ; bin: 7b 02 0f 0b + ; asm: je .+4; ud2 + trapff one v1, user0 ; bin: 74 02 0f 0b + ; asm: jne .+4; ud2 + trapff ueq v1, user0 ; bin: 75 02 0f 0b + ; asm: jna .+4; ud2 + trapff gt v1, user0 ; bin: 76 02 0f 0b + ; asm: jnae .+4; ud2 + trapff ge v1, user0 ; bin: 72 02 0f 0b + ; asm: jnb .+4; ud2 + trapff ult v1, user0 ; bin: 73 02 0f 0b + ; asm: jnbe .+4; ud2 + trapff ule v1, user0 ; bin: 77 02 0f 0b + ; asm: setnp %bl [-,%rbx] v10 = trueff ord v1 ; bin: 0f 9b c3 ; asm: setp %bl diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 748c8f9a13..b16dd058ab 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -216,6 +216,9 @@ ebb0(v0: i32): trapz v0, stk_ovf v1 = ifcmp_imm v0, 5 trapif ugt v1, oob + v2 = bitcast.f32 v1 + v3 = ffcmp v2, v2 + trapff uno v3, int_ovf return } ; sameln: function %cond_traps(i32) @@ -223,5 +226,8 @@ ebb0(v0: i32): ; nextln: trapz $v0, stk_ovf ; nextln: $v1 = ifcmp_imm v0, 5 ; nextln: trapif ugt $v1, oob +; nextln: $v2 = bitcast.f32 $v1 +; nextln: $v3 = ffcmp $v2, $v2 +; nextln: trapff uno $v3, int_ovf ; nextln: return ; nextln: } diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index 1be94a7b09..b50824581b 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -75,6 +75,7 @@ RegFill = InstructionFormat( Trap = InstructionFormat(trapcode) CondTrap = InstructionFormat(VALUE, trapcode) IntCondTrap = InstructionFormat(intcc, VALUE, trapcode) +FloatCondTrap = InstructionFormat(floatcc, VALUE, trapcode) # Finally extract the names of global variables in this module. InstructionFormat.extract_names(globals()) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 89fd36812d..7200e45e7d 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -176,6 +176,15 @@ trapif = Instruction( """, ins=(Cond, f, code), can_trap=True) +Cond = Operand('Cond', floatcc) +f = Operand('f', fflags) + +trapff = Instruction( + 'trapff', r""" + Trap when condition is true in floating point CPU flags. + """, + ins=(Cond, f, code), can_trap=True) + rvals = Operand('rvals', VARIABLE_ARGS, doc='return values') x_return = Instruction( diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 80614cb8b4..1f47a5da06 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -357,6 +357,8 @@ I64.enc(base.trap, *r.trap(0x0f, 0x0b)) # Using a standard EncRecipe, not the TailRecipe. I32.enc(base.trapif, r.trapif, 0) I64.enc(base.trapif, r.trapif, 0) +I32.enc(base.trapff, r.trapff, 0) +I64.enc(base.trapff, r.trapff, 0) # # Comparisons diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 9696a7ed43..965bc4331d 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -8,7 +8,7 @@ from cdsl.registers import RegClass from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry, NullAry from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare, FloatCompare, IntCond, FloatCond -from base.formats import IntSelect, IntCondTrap +from base.formats import IntSelect, IntCondTrap, FloatCondTrap from base.formats import Jump, Branch, BranchInt, BranchFloat from base.formats import Ternary, FuncAddr, UnaryGlobalVar from base.formats import RegMove, RegSpill, RegFill, CopySpecial @@ -292,6 +292,20 @@ trapif = EncRecipe( sink.put1(0x0b); ''') +trapff = EncRecipe( + 'trapff', FloatCondTrap, size=4, ins=FLAG.eflags, outs=(), + clobbers_flags=False, + instp=floatccs(FloatCondTrap), + emit=''' + // Jump over a 2-byte ud2. + sink.put1(0x70 | (fcc2opc(cond.inverse()) as u8)); + sink.put1(2); + // ud2. + sink.put1(0x0f); + sink.put1(0x0b); + ''') + + # XX /r rr = TailRecipe( 'rr', Binary, size=1, ins=(GPR, GPR), outs=0, diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 449da29b93..af570f7ad5 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -270,6 +270,12 @@ pub enum InstructionData { arg: Value, code: ir::TrapCode, }, + FloatCondTrap { + opcode: Opcode, + cond: FloatCC, + arg: Value, + code: ir::TrapCode, + }, } /// A variable list of `Value` operands used for function call arguments and passing arguments to diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index fa5c751430..a346573e67 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -366,6 +366,7 @@ impl<'a> Verifier<'a> { Trap { .. } | CondTrap { .. } | IntCondTrap { .. } | + FloatCondTrap { .. } | NullAry { .. } => {} } diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index ea9e3a3e4a..860550f666 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -429,6 +429,7 @@ pub fn write_operands( Trap { code, .. } => write!(w, " {}", code), CondTrap { arg, code, .. } => write!(w, " {}, {}", arg, code), IntCondTrap { cond, arg, code, .. } => write!(w, " {} {}, {}", cond, arg, code), + FloatCondTrap { cond, arg, code, .. } => write!(w, " {} {}, {}", cond, arg, code), } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index c4c7b6e852..6be56523df 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -2363,6 +2363,21 @@ impl<'a> Parser<'a> { code, } } + InstructionFormat::FloatCondTrap => { + let cond = self.match_enum("expected floatcc condition code")?; + let arg = self.match_value("expected SSA value operand")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; + let code = self.match_enum("expected trap code")?; + InstructionData::FloatCondTrap { + opcode, + cond, + arg, + code, + } + } }; Ok(idata) } From c846ec1626120a73ba33fc85203649c58b4ca7ea Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Tue, 20 Feb 2018 14:44:57 -0800 Subject: [PATCH 1510/3084] Catch NaN explicitly in expand_fcvt_to_uint(). When the input is a NaN, we need to generate a different trap code, so use the new trapff instruction to generate such a trap after the first floating point comparison. --- lib/cretonne/src/isa/intel/enc_tables.rs | 38 +++++++++++++++++++----- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 238088698e..1a602cf8a1 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -449,21 +449,43 @@ fn expand_fcvt_to_uint( ir::types::F64 => pos.ins().f64const(Ieee64::pow2(ty.lane_bits() - 1)), _ => panic!("Can't convert {}", xty), }; - let is_large = pos.ins().fcmp(FloatCC::GreaterThanOrEqual, x, pow2nm1); - pos.ins().brnz(is_large, large, &[]); + let is_large = pos.ins().ffcmp(x, pow2nm1); + pos.ins().brff( + FloatCC::GreaterThanOrEqual, + is_large, + large, + &[], + ); - // Now we know that x < 2^(N-1) or x is NaN. + // We need to generate a specific trap code when `x` is NaN, so reuse the flags from the + // previous comparison. + pos.ins().trapff( + FloatCC::Unordered, + is_large, + ir::TrapCode::BadConversionToInteger, + ); + + // Now we know that x < 2^(N-1) and not NaN. let sres = pos.ins().x86_cvtt2si(ty, x); - let is_neg = pos.ins().icmp_imm(IntCC::SignedLessThan, sres, 0); - pos.ins().brz(is_neg, done, &[sres]); - pos.ins().trap(ir::TrapCode::BadConversionToInteger); + let is_neg = pos.ins().ifcmp_imm(sres, 0); + pos.ins().brif( + IntCC::SignedGreaterThanOrEqual, + is_neg, + done, + &[sres], + ); + pos.ins().trap(ir::TrapCode::IntegerOverflow); // Handle the case where x >= 2^(N-1) and not NaN. pos.insert_ebb(large); let adjx = pos.ins().fsub(x, pow2nm1); let lres = pos.ins().x86_cvtt2si(ty, adjx); - let is_neg = pos.ins().icmp_imm(IntCC::SignedLessThan, lres, 0); - pos.ins().trapnz(is_neg, ir::TrapCode::IntegerOverflow); + let is_neg = pos.ins().ifcmp_imm(lres, 0); + pos.ins().trapif( + IntCC::SignedLessThan, + is_neg, + ir::TrapCode::IntegerOverflow, + ); let lfinal = pos.ins().iadd_imm(lres, 1 << (ty.lane_bits() - 1)); // Recycle the original instruction as a jump. From a5b00b173e010b5e48f8519edd36df43ddb92184 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 15 Feb 2018 21:13:25 -0800 Subject: [PATCH 1511/3084] Don't renumber entities in the parser. This makes it easier to debug testcases: - the entity numbers in a .cton file match the entity numbers used within Cretonne. - serializing and deserializing doesn't cause indices to change. One disadvantage is that if a .cton file uses sparse entity numbers, deserializing to the in-memory form doesn't compact it. However, the text format is not intended to be performance-critical, so this isn't expected to be a big burden. --- .../isa/intel/allones_funcaddrs32.cton | 2 +- .../isa/intel/allones_funcaddrs64.cton | 2 +- cranelift/filetests/isa/intel/binary32.cton | 2 +- .../filetests/isa/intel/binary64-pic.cton | 2 +- cranelift/filetests/isa/intel/binary64.cton | 4 +- .../filetests/isa/intel/legalize-memory.cton | 6 +- cranelift/filetests/isa/riscv/binary32.cton | 2 +- .../filetests/isa/riscv/parse-encoding.cton | 2 +- cranelift/filetests/licm/multiple-blocks.cton | 28 +- cranelift/filetests/licm/nested_loops.cton | 22 +- cranelift/filetests/parser/branch.cton | 15 +- cranelift/filetests/parser/call.cton | 16 +- cranelift/filetests/parser/rewrite.cton | 4 +- cranelift/filetests/parser/tiny.cton | 14 +- cranelift/filetests/verifier/memory.cton | 2 +- cranelift/src/filetest/domtree.rs | 2 +- lib/cretonne/src/ir/dfg.rs | 106 ++- lib/cretonne/src/ir/entities.rs | 53 ++ lib/cretonne/src/ir/stackslot.rs | 8 +- lib/reader/src/parser.rs | 728 +++++++++--------- lib/reader/src/sourcemap.rs | 304 +++----- lib/reader/src/testfile.rs | 3 +- 22 files changed, 677 insertions(+), 650 deletions(-) diff --git a/cranelift/filetests/isa/intel/allones_funcaddrs32.cton b/cranelift/filetests/isa/intel/allones_funcaddrs32.cton index 3e08d876d1..90af70fe5b 100644 --- a/cranelift/filetests/isa/intel/allones_funcaddrs32.cton +++ b/cranelift/filetests/isa/intel/allones_funcaddrs32.cton @@ -11,8 +11,8 @@ isa intel haswell ; Tests from binary32.cton affected by allones_funcaddrs. function %I32() { - fn0 = function %foo() sig0 = () + fn0 = function %foo() ebb0: diff --git a/cranelift/filetests/isa/intel/allones_funcaddrs64.cton b/cranelift/filetests/isa/intel/allones_funcaddrs64.cton index af1988f65c..7be06c3963 100644 --- a/cranelift/filetests/isa/intel/allones_funcaddrs64.cton +++ b/cranelift/filetests/isa/intel/allones_funcaddrs64.cton @@ -12,8 +12,8 @@ isa intel haswell ; Tests from binary64.cton affected by allones_funcaddrs. function %I64() { - fn0 = function %foo() sig0 = () + fn0 = function %foo() ebb0: diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index ea9613f9fa..7d3e3a683b 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -9,8 +9,8 @@ isa intel haswell ; function %I32() { - fn0 = function %foo() sig0 = () + fn0 = function %foo() gv0 = globalsym %some_gv diff --git a/cranelift/filetests/isa/intel/binary64-pic.cton b/cranelift/filetests/isa/intel/binary64-pic.cton index fd6a5dad03..3129bd9563 100644 --- a/cranelift/filetests/isa/intel/binary64-pic.cton +++ b/cranelift/filetests/isa/intel/binary64-pic.cton @@ -12,8 +12,8 @@ isa intel haswell ; Tests for i64 instructions. function %I64() { - fn0 = function %foo() sig0 = () + fn0 = function %foo() gv0 = globalsym %some_gv diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 59c5f0b2f0..8ac8deb58e 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -11,8 +11,8 @@ isa intel haswell ; Tests for i64 instructions. function %I64() { - fn0 = function %foo() sig0 = () + fn0 = function %foo() gv0 = globalsym %some_gv @@ -641,8 +641,8 @@ ebb1: ; encodings which are chosen by default. Switching to non-REX encodings should ; be done by an instruction shrinking pass. function %I32() { - fn0 = function %foo() sig0 = () + fn0 = function %foo() ss0 = incoming_arg 8, offset 0 ss1 = incoming_arg 1024, offset -1024 diff --git a/cranelift/filetests/isa/intel/legalize-memory.cton b/cranelift/filetests/isa/intel/legalize-memory.cton index 1cdcf680ba..28a0fd6a15 100644 --- a/cranelift/filetests/isa/intel/legalize-memory.cton +++ b/cranelift/filetests/isa/intel/legalize-memory.cton @@ -76,9 +76,9 @@ ebb0(v0: i32, v999: i64): ; check: $ebb0( ; nextln: trap heap_oob ; check: ebb1: - ; nextln: v2 = iconst.i64 0 - ; nextln: v3 = load.f32 v2+16 - ; nextln: return v3 + ; nextln: v1 = iconst.i64 0 + ; nextln: v2 = load.f32 v1+16 + ; nextln: return v2 ; nextln: } v1 = heap_addr.i64 heap0, v0, 0x1000_0001 v2 = load.f32 v1+16 diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 5f101cafb8..459d9d7419 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -3,8 +3,8 @@ test binemit isa riscv function %RV32I(i32 link [%x1]) -> i32 link [%x1] { - fn0 = function %foo() sig0 = () + fn0 = function %foo() ebb0(v9999: i32): [-,%x10] v1 = iconst.i32 1 diff --git a/cranelift/filetests/isa/riscv/parse-encoding.cton b/cranelift/filetests/isa/riscv/parse-encoding.cton index 3fdb8f62d6..e955ad296a 100644 --- a/cranelift/filetests/isa/riscv/parse-encoding.cton +++ b/cranelift/filetests/isa/riscv/parse-encoding.cton @@ -27,7 +27,7 @@ function %parse_encoding(i32 [%x5]) -> i32 [%x10] { ; check: sig5 = () -> f32 [0] native ; function + signature - fn15 = function %bar(i32 [%x10]) -> b1 [%x10] native + fn0 = function %bar(i32 [%x10]) -> b1 [%x10] native ; check: sig6 = (i32 [%x10]) -> b1 [%x10] native ; nextln: fn0 = sig6 %bar diff --git a/cranelift/filetests/licm/multiple-blocks.cton b/cranelift/filetests/licm/multiple-blocks.cton index 4738081d9a..d47f8658d2 100644 --- a/cranelift/filetests/licm/multiple-blocks.cton +++ b/cranelift/filetests/licm/multiple-blocks.cton @@ -25,22 +25,22 @@ ebb3(v30: i32): } ; sameln:function %multiple_blocks(i32) -> i32 { ; nextln: ebb0(v0: i32): -; nextln: v2 = iconst.i32 1 -; nextln: v3 = iconst.i32 2 -; nextln: v4 = iadd v2, v3 -; nextln: v9 = iadd v2, v4 +; nextln: v11 = iconst.i32 1 +; nextln: v12 = iconst.i32 2 +; nextln: v13 = iadd v11, v12 +; nextln: v31 = iadd v11, v13 ; nextln: jump ebb1(v0) ; nextln: -; nextln: ebb1(v1: i32): -; nextln: brz v1, ebb2(v1) -; nextln: v5 = isub v1, v2 -; nextln: brz v5, ebb3(v5) -; nextln: v6 = isub v1, v2 -; nextln: jump ebb1(v6) +; nextln: ebb1(v10: i32): +; nextln: brz v10, ebb2(v10) +; nextln: v15 = isub v10, v11 +; nextln: brz v15, ebb3(v15) +; nextln: v14 = isub v10, v11 +; nextln: jump ebb1(v14) ; nextln: -; nextln: ebb2(v7: i32): -; nextln: return v7 +; nextln: ebb2(v20: i32): +; nextln: return v20 ; nextln: -; nextln: ebb3(v8: i32): -; nextln: jump ebb1(v8) +; nextln: ebb3(v30: i32): +; nextln: jump ebb1(v30) ; nextln: } diff --git a/cranelift/filetests/licm/nested_loops.cton b/cranelift/filetests/licm/nested_loops.cton index d839bbce77..9fb3c08806 100644 --- a/cranelift/filetests/licm/nested_loops.cton +++ b/cranelift/filetests/licm/nested_loops.cton @@ -33,23 +33,23 @@ ebb4(v30: i32): ; nextln: v2 = iconst.i32 1 ; nextln: v3 = iconst.i32 2 ; nextln: v4 = iadd v2, v3 -; nextln: v8 = iconst.i32 1 +; nextln: v12 = iconst.i32 1 ; nextln: jump ebb1(v0) ; nextln: ; nextln: ebb1(v1: i32): ; nextln: v5 = isub v1, v2 -; nextln: v9 = iadd.i32 v8, v5 +; nextln: v15 = iadd.i32 v12, v5 ; nextln: jump ebb2(v5, v5) ; nextln: -; nextln: ebb2(v6: i32, v7: i32): -; nextln: brz v7, ebb3(v6) -; nextln: v10 = isub v7, v8 -; nextln: jump ebb2(v6, v10) +; nextln: ebb2(v10: i32, v11: i32): +; nextln: brz v11, ebb3(v10) +; nextln: v13 = isub v11, v12 +; nextln: jump ebb2(v10, v13) ; nextln: -; nextln: ebb3(v11: i32): -; nextln: brz v11, ebb4(v11) -; nextln: jump ebb1(v11) +; nextln: ebb3(v20: i32): +; nextln: brz v20, ebb4(v20) +; nextln: jump ebb1(v20) ; nextln: -; nextln: ebb4(v12: i32): -; nextln: return v12 +; nextln: ebb4(v30: i32): +; nextln: return v30 ; nextln: } diff --git a/cranelift/filetests/parser/branch.cton b/cranelift/filetests/parser/branch.cton index 7b3ca787b6..7c653ed047 100644 --- a/cranelift/filetests/parser/branch.cton +++ b/cranelift/filetests/parser/branch.cton @@ -95,19 +95,18 @@ ebb40: trap user4 } ; sameln: function %jumptable(i32) native { -; nextln: jt0 = jump_table 0 -; nextln: jt1 = jump_table 0, 0, ebb0, ebb3, ebb1, ebb2 -; nextln: -; nextln: ebb0($v3: i32): -; nextln: br_table $v3, jt1 +; check: jt2 = jump_table 0, 0, ebb10, ebb40, ebb20, ebb30 +; check: jt200 = jump_table 0 +; check: ebb10($v3: i32): +; nextln: br_table $v3, jt2 ; nextln: trap user1 ; nextln: -; nextln: ebb1: +; nextln: ebb20: ; nextln: trap user2 ; nextln: -; nextln: ebb2: +; nextln: ebb30: ; nextln: trap user3 ; nextln: -; nextln: ebb3: +; nextln: ebb40: ; nextln: trap user4 ; nextln: } diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton index f4ed00c222..eab0799458 100644 --- a/cranelift/filetests/parser/call.cton +++ b/cranelift/filetests/parser/call.cton @@ -6,7 +6,7 @@ ebb1: return } ; sameln: function %mini() native { -; nextln: ebb0: +; nextln: ebb1: ; nextln: return ; nextln: } @@ -17,7 +17,7 @@ ebb1: return v1, v2 } ; sameln: function %r1() -> i32, f32 spiderwasm { -; nextln: ebb0: +; nextln: ebb1: ; nextln: $v1 = iconst.i32 3 ; nextln: $v2 = f32const 0.0 ; nextln: return $v1, $v2 @@ -30,12 +30,12 @@ function %signatures() { fn8 = function %bar(i32) -> b1 } ; sameln: function %signatures() native { -; nextln: $sig10 = () native -; nextln: $sig11 = (i32, f64) -> i32, b1 spiderwasm -; nextln: sig2 = (i32) -> b1 native -; nextln: $fn5 = $sig11 %foo -; nextln: $fn8 = sig2 %bar -; nextln: } +; check: $sig10 = () native +; check: $sig11 = (i32, f64) -> i32, b1 spiderwasm +; check: sig12 = (i32) -> b1 native +; check: $fn5 = $sig11 %foo +; check: $fn8 = sig12 %bar +; check: } function %direct() { fn0 = function %none() diff --git a/cranelift/filetests/parser/rewrite.cton b/cranelift/filetests/parser/rewrite.cton index 1601002b27..33d23baef4 100644 --- a/cranelift/filetests/parser/rewrite.cton +++ b/cranelift/filetests/parser/rewrite.cton @@ -30,8 +30,8 @@ ebb100(v20: i32): jump ebb100(v1000) } ; sameln: function %use_value() native { -; nextln: ebb0($v20: i32): +; nextln: ebb100($v20: i32): ; nextln: $v1000 = iadd_imm $v20, 5 ; nextln: $v200 = iadd $v20, $v1000 -; nextln: jump ebb0($v1000) +; nextln: jump ebb100($v1000) ; nextln: } diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index b16dd058ab..1355d0a21d 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -58,8 +58,8 @@ ebb0(v95: i32, v96: i32, v97: b1): v98 = selectif.i32 eq v97, v95, v96 } ; sameln: function %selectif() native { -; nextln: ebb0(v0: i32, v1: i32, v2: b1): -; nextln: v3 = selectif.i32 eq v2, v0, v1 +; nextln: ebb0(v95: i32, v96: i32, v97: b1): +; nextln: v98 = selectif.i32 eq v97, v95, v96 ; nextln: } ; Lane indexes. @@ -136,11 +136,11 @@ ebb0: stack_store v2, ss2 } ; sameln: function %stack() native { -; nextln: $ss10 = spill_slot 8 -; nextln: $ss2 = local 4 -; nextln: $ss3 = incoming_arg 4, offset 8 -; nextln: $ss4 = outgoing_arg 4 -; nextln: $ss5 = emergency_slot 4 +; check: $ss2 = local 4 +; check: $ss3 = incoming_arg 4, offset 8 +; check: $ss4 = outgoing_arg 4 +; check: $ss5 = emergency_slot 4 +; check: $ss10 = spill_slot 8 ; check: ebb0: ; nextln: $v1 = stack_load.i32 $ss10 diff --git a/cranelift/filetests/verifier/memory.cton b/cranelift/filetests/verifier/memory.cton index 46bf18d636..4818d4c686 100644 --- a/cranelift/filetests/verifier/memory.cton +++ b/cranelift/filetests/verifier/memory.cton @@ -1,7 +1,7 @@ test verifier function %deref_cycle() { - gv1 = deref(gv2)-32 ; error: deref cycle: [gv0, gv1] + gv1 = deref(gv2)-32 ; error: deref cycle: [gv1, gv2] gv2 = deref(gv1) ebb1: diff --git a/cranelift/src/filetest/domtree.rs b/cranelift/src/filetest/domtree.rs index b118dbd314..9c0fe5a841 100644 --- a/cranelift/src/filetest/domtree.rs +++ b/cranelift/src/filetest/domtree.rs @@ -61,7 +61,7 @@ impl SubTest for TestDomtree { for src_ebb in tail.split_whitespace() { let ebb = match context.details.map.lookup_str(src_ebb) { Some(AnyEntity::Ebb(ebb)) => ebb, - _ => return Err(format!("expected EBB: {}", src_ebb)), + _ => return Err(format!("expected defined EBB, got {}", src_ebb)), }; // Annotations say that `inst` is the idom of `ebb`. diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 1c3a0fce3c..023d8751b6 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -8,6 +8,7 @@ use ir::extfunc::ExtFuncData; use ir::instructions::{InstructionData, CallInfo, BranchInfo}; use ir::types; use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueList, ValueListPool}; +use packed_option::ReservedValue; use write::write_operands; use std::fmt; use std::iter; @@ -287,16 +288,6 @@ impl DataFlowGraph { self.clear_results(dest_inst); } - - /// Create a new value alias. - /// - /// Note that this function should only be called by the parser. - pub fn make_value_alias(&mut self, src: Value) -> Value { - let ty = self.value_type(src); - - let data = ValueData::Alias { ty, original: src }; - self.make_value(data) - } } /// Where did a value come from? @@ -370,14 +361,6 @@ impl DataFlowGraph { self.insts.push(data) } - /// Get the instruction reference that will be assigned to the next instruction created by - /// `make_inst`. - /// - /// This is only really useful to the parser. - pub fn next_inst(&self) -> Inst { - self.insts.next_key() - } - /// Returns an object that displays `inst`. pub fn display_inst<'a, I: Into>>( &'a self, @@ -870,6 +853,91 @@ impl<'a> fmt::Display for DisplayInst<'a> { } } +/// Parser routines. These routines should not be used outside the parser. +impl DataFlowGraph { + /// Set the type of a value. This is only for use in the parser, which needs + /// to create invalid values for index padding which may be reassigned later. + #[cold] + fn set_value_type_for_parser(&mut self, v: Value, t: Type) { + debug_assert!( + self.value_type(v) == types::VOID, + "this function is only for assigning types to previously invalid values" + ); + match self.values[v] { + ValueData::Inst { ref mut ty, .. } | + ValueData::Param { ref mut ty, .. } | + ValueData::Alias { ref mut ty, .. } => *ty = t, + } + } + + /// Create result values for `inst`, reusing the provided detached values. + /// This is similar to `make_inst_results_reusing` except it's only for use + /// in the parser, which needs to reuse previously invalid values. + #[cold] + pub fn make_inst_results_for_parser( + &mut self, + inst: Inst, + ctrl_typevar: Type, + reuse: &[Value], + ) -> usize { + // Get the call signature if this is a function call. + if let Some(sig) = self.call_signature(inst) { + debug_assert_eq!(self.insts[inst].opcode().constraints().fixed_results(), 0); + for res_idx in 0..self.signatures[sig].returns.len() { + let ty = self.signatures[sig].returns[res_idx].value_type; + if let Some(v) = reuse.get(res_idx) { + self.set_value_type_for_parser(*v, ty); + } + } + } else { + let constraints = self.insts[inst].opcode().constraints(); + for res_idx in 0..constraints.fixed_results() { + let ty = constraints.result_type(res_idx, ctrl_typevar); + if let Some(v) = reuse.get(res_idx) { + self.set_value_type_for_parser(*v, ty); + } + } + } + + self.make_inst_results_reusing(inst, ctrl_typevar, reuse.iter().map(|x| Some(*x))) + } + + /// Similar to `append_ebb_param`, append a parameter with type `ty` to + /// `ebb`, but using value `val`. This is only for use by the parser to + /// create parameters with specific values. + #[cold] + pub fn append_ebb_param_for_parser(&mut self, ebb: Ebb, ty: Type, val: Value) { + let num = self.ebbs[ebb].params.push(val, &mut self.value_lists); + assert!(num <= u16::MAX as usize, "Too many parameters on EBB"); + self.values[val] = ValueData::Param { + ty, + num: num as u16, + ebb, + }; + } + + /// Create a new value alias. This is only for use by the parser to create + /// aliases with specific values. + #[cold] + pub fn make_value_alias_for_parser(&mut self, src: Value, dest: Value) { + let ty = self.value_type(src); + + let data = ValueData::Alias { ty, original: src }; + self.values[dest] = data; + } + + /// Create an invalid value, to pad the index space. This is only for use by + /// the parser to pad out the value index space. + #[cold] + pub fn make_invalid_value_for_parser(&mut self) { + let data = ValueData::Alias { + ty: types::VOID, + original: Value::reserved_value(), + }; + self.make_value(data); + } +} + #[cfg(test)] mod tests { use super::*; @@ -885,9 +953,7 @@ mod tests { opcode: Opcode::Iconst, imm: 0.into(), }; - let next = dfg.next_inst(); let inst = dfg.make_inst(idata); - assert_eq!(next, inst); dfg.make_inst_results(inst, types::I32); assert_eq!(inst.to_string(), "inst0"); diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 4a4a40df42..a9e6705d98 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -65,6 +65,19 @@ entity_impl!(Inst, "inst"); pub struct StackSlot(u32); entity_impl!(StackSlot, "ss"); +impl StackSlot { + /// Create a new stack slot reference from its number. + /// + /// This method is for use by the parser. + pub fn with_number(n: u32) -> Option { + if n < u32::MAX { + Some(StackSlot(n)) + } else { + None + } + } +} + /// An opaque reference to a global variable. #[derive(Copy, Clone, PartialEq, Eq, Hash)] pub struct GlobalVar(u32); @@ -88,21 +101,61 @@ impl GlobalVar { pub struct JumpTable(u32); entity_impl!(JumpTable, "jt"); +impl JumpTable { + /// Create a new jump table reference from its number. + /// + /// This method is for use by the parser. + pub fn with_number(n: u32) -> Option { + if n < u32::MAX { + Some(JumpTable(n)) + } else { + None + } + } +} + /// A reference to an external function. #[derive(Copy, Clone, PartialEq, Eq, Hash)] pub struct FuncRef(u32); entity_impl!(FuncRef, "fn"); +impl FuncRef { + /// Create a new external function reference from its number. + /// + /// This method is for use by the parser. + pub fn with_number(n: u32) -> Option { + if n < u32::MAX { Some(FuncRef(n)) } else { None } + } +} + /// A reference to a function signature. #[derive(Copy, Clone, PartialEq, Eq, Hash)] pub struct SigRef(u32); entity_impl!(SigRef, "sig"); +impl SigRef { + /// Create a new function signature reference from its number. + /// + /// This method is for use by the parser. + pub fn with_number(n: u32) -> Option { + if n < u32::MAX { Some(SigRef(n)) } else { None } + } +} + /// A reference to a heap. #[derive(Copy, Clone, PartialEq, Eq, Hash)] pub struct Heap(u32); entity_impl!(Heap, "heap"); +impl Heap { + /// Create a new heap reference from its number. + /// + /// This method is for use by the parser. + pub fn with_number(n: u32) -> Option { + if n < u32::MAX { Some(Heap(n)) } else { None } + } +} + /// A reference to any of the entities defined in this module. #[derive(Copy, Clone, PartialEq, Eq, Hash)] pub enum AnyEntity { diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index dc02f0df6b..4a0804787b 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -8,7 +8,7 @@ use ir::{Type, StackSlot}; use packed_option::PackedOption; use std::cmp; use std::fmt; -use std::ops::Index; +use std::ops::{Index, IndexMut}; use std::str::FromStr; /// The size of an object on the stack, or the size of a stack frame. @@ -228,6 +228,12 @@ impl Index for StackSlots { } } +impl IndexMut for StackSlots { + fn index_mut(&mut self, ss: StackSlot) -> &mut StackSlotData { + &mut self.slots[ss] + } +} + /// Higher-level stack frame manipulation functions. impl StackSlots { /// Create a new spill slot for spilling values of type `ty`. diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 6be56523df..a0a17faa16 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1,13 +1,13 @@ //! Parser for .cton files. -use std::collections::HashMap; use std::str::FromStr; use std::{u16, u32}; use std::mem; use cretonne::ir::{Function, Ebb, Opcode, Value, Type, ExternalName, CallConv, StackSlotData, - JumpTable, JumpTableData, Signature, AbiParam, ArgumentExtension, ExtFuncData, - SigRef, FuncRef, StackSlot, ValueLoc, ArgumentLoc, MemFlags, GlobalVar, - GlobalVarData, Heap, HeapData, HeapStyle, HeapBase}; + StackSlotKind, JumpTable, JumpTableData, Signature, AbiParam, + ArgumentExtension, ExtFuncData, SigRef, FuncRef, StackSlot, ValueLoc, + ArgumentLoc, MemFlags, GlobalVar, GlobalVarData, Heap, HeapData, HeapStyle, + HeapBase}; use cretonne::ir; use cretonne::ir::types::VOID; use cretonne::ir::immediates::{Imm64, Uimm32, Offset32, Ieee32, Ieee64}; @@ -15,12 +15,14 @@ use cretonne::ir::entities::AnyEntity; use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs}; use cretonne::isa::{self, TargetIsa, Encoding, RegUnit}; use cretonne::{settings, timing}; +use cretonne::entity::EntityRef; +use cretonne::packed_option::ReservedValue; use testfile::{TestFile, Details, Comment}; use error::{Location, Error, Result}; use lexer::{self, Lexer, Token}; use testcommand::TestCommand; use isaspec; -use sourcemap::{SourceMap, MutableSourceMap}; +use sourcemap::SourceMap; /// Parse the entire `text` into a list of functions. /// @@ -38,11 +40,15 @@ pub fn parse_functions(text: &str) -> Result> { pub fn parse_test<'a>(text: &'a str) -> Result> { let _tt = timing::parse_text(); let mut parser = Parser::new(text); - // Gather the preamble comments as 'Function'. - parser.gather_comments(AnyEntity::Function); + // Gather the preamble comments. + parser.start_gathering_comments(); let commands = parser.parse_test_commands(); let isa_spec = parser.parse_isa_specs()?; + + parser.token(); + parser.claim_gathered_comments(AnyEntity::Function); + let preamble_comments = parser.take_comments(); let functions = parser.parse_function_list(isa_spec.unique_isa())?; @@ -65,9 +71,11 @@ pub struct Parser<'a> { // Location of lookahead. loc: Location, - // The currently active entity that should be associated with collected comments, or `None` if - // comments are ignored. - comment_entity: Option, + // Are we gathering any comments that we encounter? + gathering_comments: bool, + + // The gathered comments; claim them with `claim_gathered_comments`. + gathered_comments: Vec<&'a str>, // Comments collected so far. comments: Vec>, @@ -80,8 +88,6 @@ pub struct Parser<'a> { struct Context<'a> { function: Function, map: SourceMap, - // Store aliases until the values can be reliably looked up. - aliases: HashMap, // Reference to the unique_isa for things like parsing ISA-specific instruction encoding // information. This is only `Some` if exactly one set of `isa` directives were found in the @@ -95,7 +101,6 @@ impl<'a> Context<'a> { Context { function: f, map: SourceMap::new(), - aliases: HashMap::new(), unique_isa, } } @@ -115,200 +120,134 @@ impl<'a> Context<'a> { } // Allocate a new stack slot and add a mapping number -> StackSlot. - fn add_ss(&mut self, number: u32, data: StackSlotData, loc: &Location) -> Result<()> { - self.map.def_ss( - number, - self.function.create_stack_slot(data), - loc, - ) + fn add_ss(&mut self, ss: StackSlot, data: StackSlotData, loc: &Location) -> Result<()> { + while self.function.stack_slots.next_key().index() <= ss.index() { + self.function.create_stack_slot( + StackSlotData::new(StackSlotKind::SpillSlot, 0), + ); + } + self.function.stack_slots[ss] = data; + self.map.def_ss(ss, loc) } // Resolve a reference to a stack slot. - fn get_ss(&self, number: u32, loc: &Location) -> Result { - match self.map.get_ss(number) { - Some(sig) => Ok(sig), - None => err!(loc, "undefined stack slot ss{}", number), + fn check_ss(&self, ss: StackSlot, loc: &Location) -> Result<()> { + if !self.map.contains_ss(ss) { + err!(loc, "undefined stack slot {}", ss) + } else { + Ok(()) } } // Allocate a global variable slot and add a mapping number -> GlobalVar. - fn add_gv(&mut self, number: u32, data: GlobalVarData, loc: &Location) -> Result<()> { - self.map.def_gv( - number, - self.function.create_global_var(data), - loc, - ) + fn add_gv(&mut self, gv: GlobalVar, data: GlobalVarData, loc: &Location) -> Result<()> { + while self.function.global_vars.next_key().index() <= gv.index() { + self.function.create_global_var(GlobalVarData::Sym { + name: ExternalName::testcase(""), + }); + } + self.function.global_vars[gv] = data; + self.map.def_gv(gv, loc) } // Resolve a reference to a global variable. - fn get_gv(&self, number: u32, loc: &Location) -> Result { - match self.map.get_gv(number) { - Some(gv) => Ok(gv), - None => err!(loc, "undefined global variable gv{}", number), + fn check_gv(&self, gv: GlobalVar, loc: &Location) -> Result<()> { + if !self.map.contains_gv(gv) { + err!(loc, "undefined global variable {}", gv) + } else { + Ok(()) } } // Allocate a heap slot and add a mapping number -> Heap. - fn add_heap(&mut self, number: u32, data: HeapData, loc: &Location) -> Result<()> { - self.map.def_heap( - number, - self.function.create_heap(data), - loc, - ) + fn add_heap(&mut self, heap: Heap, data: HeapData, loc: &Location) -> Result<()> { + while self.function.heaps.next_key().index() <= heap.index() { + self.function.create_heap(HeapData { + base: HeapBase::ReservedReg, + min_size: Imm64::new(0), + guard_size: Imm64::new(0), + style: HeapStyle::Static { bound: Imm64::new(0) }, + }); + } + self.function.heaps[heap] = data; + self.map.def_heap(heap, loc) } // Resolve a reference to a heap. - fn get_heap(&self, number: u32, loc: &Location) -> Result { - match self.map.get_heap(number) { - Some(heap) => Ok(heap), - None => err!(loc, "undefined heap heap{}", number), + fn check_heap(&self, heap: Heap, loc: &Location) -> Result<()> { + if !self.map.contains_heap(heap) { + err!(loc, "undefined heap {}", heap) + } else { + Ok(()) } } // Allocate a new signature and add a mapping number -> SigRef. - fn add_sig(&mut self, number: u32, data: Signature, loc: &Location) -> Result<()> { - self.map.def_sig( - number, - self.function.import_signature(data), - loc, - ) + fn add_sig(&mut self, sig: SigRef, data: Signature, loc: &Location) -> Result<()> { + while self.function.dfg.signatures.next_key().index() <= sig.index() { + self.function.import_signature( + Signature::new(CallConv::Native), + ); + } + self.function.dfg.signatures[sig] = data; + self.map.def_sig(sig, loc) } // Resolve a reference to a signature. - fn get_sig(&self, number: u32, loc: &Location) -> Result { - match self.map.get_sig(number) { - Some(sig) => Ok(sig), - None => err!(loc, "undefined signature sig{}", number), + fn check_sig(&self, sig: SigRef, loc: &Location) -> Result<()> { + if !self.map.contains_sig(sig) { + err!(loc, "undefined signature {}", sig) + } else { + Ok(()) } } // Allocate a new external function and add a mapping number -> FuncRef. - fn add_fn(&mut self, number: u32, data: ExtFuncData, loc: &Location) -> Result<()> { - self.map.def_fn( - number, - self.function.import_function(data), - loc, - ) + fn add_fn(&mut self, fn_: FuncRef, data: ExtFuncData, loc: &Location) -> Result<()> { + while self.function.dfg.ext_funcs.next_key().index() <= fn_.index() { + self.function.import_function(ExtFuncData { + name: ExternalName::testcase(""), + signature: SigRef::reserved_value(), + }); + } + self.function.dfg.ext_funcs[fn_] = data; + self.map.def_fn(fn_, loc) } // Resolve a reference to a function. - fn get_fn(&self, number: u32, loc: &Location) -> Result { - match self.map.get_fn(number) { - Some(fnref) => Ok(fnref), - None => err!(loc, "undefined function fn{}", number), + fn check_fn(&self, fn_: FuncRef, loc: &Location) -> Result<()> { + if !self.map.contains_fn(fn_) { + err!(loc, "undefined function {}", fn_) + } else { + Ok(()) } } // Allocate a new jump table and add a mapping number -> JumpTable. - fn add_jt(&mut self, number: u32, data: JumpTableData, loc: &Location) -> Result<()> { - self.map.def_jt( - number, - self.function.create_jump_table(data), - loc, - ) + fn add_jt(&mut self, jt: JumpTable, data: JumpTableData, loc: &Location) -> Result<()> { + while self.function.jump_tables.next_key().index() <= jt.index() { + self.function.create_jump_table(JumpTableData::new()); + } + self.function.jump_tables[jt] = data; + self.map.def_jt(jt, loc) } // Resolve a reference to a jump table. - fn get_jt(&self, number: u32, loc: &Location) -> Result { - match self.map.get_jt(number) { - Some(jt) => Ok(jt), - None => err!(loc, "undefined jump table jt{}", number), + fn check_jt(&self, jt: JumpTable, loc: &Location) -> Result<()> { + if !self.map.contains_jt(jt) { + err!(loc, "undefined jump table {}", jt) + } else { + Ok(()) } } // Allocate a new EBB and add a mapping src_ebb -> Ebb. - fn add_ebb(&mut self, src_ebb: Ebb, loc: &Location) -> Result { - let ebb = self.function.dfg.make_ebb(); + fn add_ebb(&mut self, ebb: Ebb, loc: &Location) -> Result { + while self.function.dfg.num_ebbs() <= ebb.index() { + self.function.dfg.make_ebb(); + } self.function.layout.append_ebb(ebb); - self.map.def_ebb(src_ebb, ebb, loc).and(Ok(ebb)) - } - - fn add_alias(&mut self, src: Value, dest: Value, loc: Location) -> Result<()> { - match self.aliases.insert(src, (dest, loc)) { - Some((v, _)) if v != dest => err!(loc, "duplicate alias: {} -> {}", src, dest), - _ => Ok(()), - } - } - - // The parser creates all instructions with Ebb and Value references using the source file - // numbering. These references need to be rewritten after parsing is complete since forward - // references are allowed. - fn rewrite_references(&mut self) -> Result<()> { - for (&source_from, &(source_to, source_loc)) in &self.aliases { - let ir_to = match self.map.get_value(source_to) { - Some(v) => v, - None => { - return err!( - source_loc, - "IR destination value alias not found for {}", - source_to - ); - } - }; - let dest_loc = self.map.location(AnyEntity::from(ir_to)).expect(&*format!( - "Error in looking up location of IR destination value alias \ - for {}", - ir_to - )); - let ir_from = self.function.dfg.make_value_alias(ir_to); - self.map.def_value(source_from, ir_from, &dest_loc)?; - } - - for ebb in self.function.layout.ebbs() { - for inst in self.function.layout.ebb_insts(ebb) { - let loc = inst.into(); - self.map.rewrite_values( - self.function.dfg.inst_args_mut(inst), - loc, - )?; - if let Some(dest) = self.function.dfg[inst].branch_destination_mut() { - self.map.rewrite_ebb(dest, loc)?; - } - } - } - - // Rewrite EBB references in jump tables. - for jt in self.function.jump_tables.keys() { - let loc = jt.into(); - for ebb_ref in self.function.jump_tables[jt].as_mut_slice() { - if let Some(mut ebb) = ebb_ref.expand() { - self.map.rewrite_ebb(&mut ebb, loc)?; - // Convert back to a packed option. - *ebb_ref = ebb.into(); - } - } - } - - // Rewrite base references in `deref` globals. Other `GlobalVar` references are already - // resolved. - for gv in self.function.global_vars.keys() { - let loc = gv.into(); - match self.function.global_vars[gv] { - GlobalVarData::Deref { ref mut base, .. } => { - self.map.rewrite_gv(base, loc)?; - } - _ => {} - } - } - - // Rewrite references to global variables in heaps. - for heap in self.function.heaps.keys() { - let loc = heap.into(); - match self.function.heaps[heap].base { - HeapBase::GlobalVar(ref mut base) => { - self.map.rewrite_gv(base, loc)?; - } - _ => {} - } - match self.function.heaps[heap].style { - HeapStyle::Dynamic { ref mut bound_gv } => { - self.map.rewrite_gv(bound_gv, loc)?; - } - _ => {} - } - } - - Ok(()) + self.map.def_ebb(ebb, loc).and(Ok(ebb)) } } @@ -320,7 +259,8 @@ impl<'a> Parser<'a> { lex_error: None, lookahead: None, loc: Location { line_number: 0 }, - comment_entity: None, + gathering_comments: false, + gathered_comments: Vec::new(), comments: Vec::new(), } } @@ -345,9 +285,8 @@ impl<'a> Parser<'a> { Some(Ok(lexer::LocatedToken { token, location })) => { match token { Token::Comment(text) => { - // Gather comments, associate them with `comment_entity`. - if let Some(entity) = self.comment_entity { - self.comments.push(Comment { entity, text }); + if self.gathering_comments { + self.gathered_comments.push(text); } } _ => self.lookahead = Some(token), @@ -365,13 +304,29 @@ impl<'a> Parser<'a> { self.lookahead } - // Begin gathering comments associated with `entity`. - fn gather_comments>(&mut self, entity: E) { - self.comment_entity = Some(entity.into()); + // Enable gathering of all comments encountered. + fn start_gathering_comments(&mut self) { + debug_assert!(!self.gathering_comments); + self.gathering_comments = true; + debug_assert!(self.gathered_comments.is_empty()); } - // Get the comments gathered so far, clearing out the internal list. + // Claim the comments gathered up to the current position for the + // given entity. + fn claim_gathered_comments>(&mut self, entity: E) { + debug_assert!(self.gathering_comments); + let entity = entity.into(); + self.comments.extend( + self.gathered_comments.drain(..).map(|text| { + Comment { entity, text } + }), + ); + self.gathering_comments = false; + } + + // Get the comments collected so far, clearing out the internal list. fn take_comments(&mut self) -> Vec> { + debug_assert!(!self.gathering_comments); mem::replace(&mut self.comments, Vec::new()) } @@ -415,74 +370,69 @@ impl<'a> Parser<'a> { } // Match and consume a stack slot reference. - fn match_ss(&mut self, err_msg: &str) -> Result { + fn match_ss(&mut self, err_msg: &str) -> Result { if let Some(Token::StackSlot(ss)) = self.token() { self.consume(); - Ok(ss) - } else { - err!(self.loc, err_msg) + if let Some(ss) = StackSlot::with_number(ss) { + return Ok(ss); + } } + err!(self.loc, err_msg) } // Match and consume a global variable reference. - fn match_gv(&mut self, err_msg: &str) -> Result { + fn match_gv(&mut self, err_msg: &str) -> Result { if let Some(Token::GlobalVar(gv)) = self.token() { self.consume(); - Ok(gv) - } else { - err!(self.loc, err_msg) - } - } - - // Match and consume a global variable reference in the preamble where it can't be rewritten. - // - // Any global variable references appearing in the preamble need to be rewritten after parsing - // the whole preamble. - fn match_gv_preamble(&mut self, err_msg: &str) -> Result { - match GlobalVar::with_number(self.match_gv(err_msg)?) { - Some(gv) => Ok(gv), - None => err!(self.loc, "Invalid global variable number"), + if let Some(gv) = GlobalVar::with_number(gv) { + return Ok(gv); + } } + err!(self.loc, err_msg) } // Match and consume a function reference. - fn match_fn(&mut self, err_msg: &str) -> Result { + fn match_fn(&mut self, err_msg: &str) -> Result { if let Some(Token::FuncRef(fnref)) = self.token() { self.consume(); - Ok(fnref) - } else { - err!(self.loc, err_msg) + if let Some(fnref) = FuncRef::with_number(fnref) { + return Ok(fnref); + } } + err!(self.loc, err_msg) } // Match and consume a signature reference. - fn match_sig(&mut self, err_msg: &str) -> Result { + fn match_sig(&mut self, err_msg: &str) -> Result { if let Some(Token::SigRef(sigref)) = self.token() { self.consume(); - Ok(sigref) - } else { - err!(self.loc, err_msg) + if let Some(sigref) = SigRef::with_number(sigref) { + return Ok(sigref); + } } + err!(self.loc, err_msg) } // Match and consume a heap reference. - fn match_heap(&mut self, err_msg: &str) -> Result { + fn match_heap(&mut self, err_msg: &str) -> Result { if let Some(Token::Heap(heap)) = self.token() { self.consume(); - Ok(heap) - } else { - err!(self.loc, err_msg) + if let Some(heap) = Heap::with_number(heap) { + return Ok(heap); + } } + err!(self.loc, err_msg) } // Match and consume a jump table reference. - fn match_jt(&mut self) -> Result { + fn match_jt(&mut self) -> Result { if let Some(Token::JumpTable(jt)) = self.token() { self.consume(); - Ok(jt) - } else { - err!(self.loc, "expected jump table number: jt«n»") + if let Some(jt) = JumpTable::with_number(jt) { + return Ok(jt); + } } + err!(self.loc, "expected jump table number: jt«n»") } // Match and consume an ebb reference. @@ -816,8 +766,8 @@ impl<'a> Parser<'a> { // Begin gathering comments. // Make sure we don't include any comments before the `function` keyword. self.token(); - self.comments.clear(); - self.gather_comments(AnyEntity::Function); + debug_assert!(self.comments.is_empty()); + self.start_gathering_comments(); let (location, name, sig) = self.parse_function_spec(unique_isa)?; let mut ctx = Context::new(Function::with_name_signature(name, sig), unique_isa); @@ -827,6 +777,10 @@ impl<'a> Parser<'a> { Token::LBrace, "expected '{' before function body", )?; + + self.token(); + self.claim_gathered_comments(AnyEntity::Function); + // function ::= function-spec "{" * preamble function-body "}" self.parse_preamble(&mut ctx)?; // function ::= function-spec "{" preamble * function-body "}" @@ -838,13 +792,9 @@ impl<'a> Parser<'a> { )?; // Collect any comments following the end of the function, then stop gathering comments. - self.gather_comments(AnyEntity::Function); + self.start_gathering_comments(); self.token(); - self.comment_entity = None; - - // Rewrite references to values and EBBs after parsing everything to allow forward - // references. - ctx.rewrite_references()?; + self.claim_gathered_comments(AnyEntity::Function); let details = Details { location, @@ -1051,42 +1001,42 @@ impl<'a> Parser<'a> { loop { match self.token() { Some(Token::StackSlot(..)) => { - self.gather_comments(ctx.function.stack_slots.next_key()); + self.start_gathering_comments(); let loc = self.loc; - self.parse_stack_slot_decl().and_then(|(num, dat)| { - ctx.add_ss(num, dat, &loc) + self.parse_stack_slot_decl().and_then(|(ss, dat)| { + ctx.add_ss(ss, dat, &loc) }) } Some(Token::GlobalVar(..)) => { - self.gather_comments(ctx.function.global_vars.next_key()); - self.parse_global_var_decl().and_then(|(num, dat)| { - ctx.add_gv(num, dat, &self.loc) + self.start_gathering_comments(); + self.parse_global_var_decl().and_then(|(gv, dat)| { + ctx.add_gv(gv, dat, &self.loc) }) } Some(Token::Heap(..)) => { - self.gather_comments(ctx.function.heaps.next_key()); - self.parse_heap_decl().and_then(|(num, dat)| { - ctx.add_heap(num, dat, &self.loc) + self.start_gathering_comments(); + self.parse_heap_decl().and_then(|(heap, dat)| { + ctx.add_heap(heap, dat, &self.loc) }) } Some(Token::SigRef(..)) => { - self.gather_comments(ctx.function.dfg.signatures.next_key()); + self.start_gathering_comments(); self.parse_signature_decl(ctx.unique_isa).and_then( - |(num, dat)| { - ctx.add_sig(num, dat, &self.loc) + |(sig, dat)| { + ctx.add_sig(sig, dat, &self.loc) }, ) } Some(Token::FuncRef(..)) => { - self.gather_comments(ctx.function.dfg.ext_funcs.next_key()); - self.parse_function_decl(ctx).and_then(|(num, dat)| { - ctx.add_fn(num, dat, &self.loc) + self.start_gathering_comments(); + self.parse_function_decl(ctx).and_then(|(fn_, dat)| { + ctx.add_fn(fn_, dat, &self.loc) }) } Some(Token::JumpTable(..)) => { - self.gather_comments(ctx.function.jump_tables.next_key()); - self.parse_jump_table_decl().and_then(|(num, dat)| { - ctx.add_jt(num, dat, &self.loc) + self.start_gathering_comments(); + self.parse_jump_table_decl().and_then(|(jt, dat)| { + ctx.add_jt(jt, dat, &self.loc) }) } // More to come.. @@ -1102,8 +1052,8 @@ impl<'a> Parser<'a> { // | "spill_slot" // | "incoming_arg" // | "outgoing_arg" - fn parse_stack_slot_decl(&mut self) -> Result<(u32, StackSlotData)> { - let number = self.match_ss("expected stack slot number: ss«n»")?; + fn parse_stack_slot_decl(&mut self) -> Result<(StackSlot, StackSlotData)> { + let ss = self.match_ss("expected stack slot number: ss«n»")?; self.match_token( Token::Equal, "expected '=' in stack slot declaration", @@ -1129,8 +1079,12 @@ impl<'a> Parser<'a> { } } + // Collect any trailing comments. + self.token(); + self.claim_gathered_comments(ss); + // TBD: stack-slot-decl ::= StackSlot(ss) "=" stack-slot-kind Bytes * {"," stack-slot-flag} - Ok((number, data)) + Ok((ss, data)) } // Parse a global variable decl. @@ -1140,8 +1094,9 @@ impl<'a> Parser<'a> { // | "deref" "(" GlobalVar(base) ")" offset32 // | "globalsym" name // - fn parse_global_var_decl(&mut self) -> Result<(u32, GlobalVarData)> { - let number = self.match_gv("expected global variable number: gv«n»")?; + fn parse_global_var_decl(&mut self) -> Result<(GlobalVar, GlobalVarData)> { + let gv = self.match_gv("expected global variable number: gv«n»")?; + self.match_token( Token::Equal, "expected '=' in global variable declaration", @@ -1157,7 +1112,7 @@ impl<'a> Parser<'a> { Token::LPar, "expected '(' in 'deref' global variable decl", )?; - let base = self.match_gv_preamble("expected global variable: gv«n»")?; + let base = self.match_gv("expected global variable: gv«n»")?; self.match_token( Token::RPar, "expected ')' in 'deref' global variable decl", @@ -1172,7 +1127,11 @@ impl<'a> Parser<'a> { other => return err!(self.loc, "Unknown global variable kind '{}'", other), }; - Ok((number, data)) + // Collect any trailing comments. + self.token(); + self.claim_gathered_comments(gv); + + Ok((gv, data)) } // Parse a heap decl. @@ -1186,8 +1145,8 @@ impl<'a> Parser<'a> { // | "max" Imm64(bytes) // | "guard" Imm64(bytes) // - fn parse_heap_decl(&mut self) -> Result<(u32, HeapData)> { - let number = self.match_heap("expected heap number: heap«n»")?; + fn parse_heap_decl(&mut self) -> Result<(Heap, HeapData)> { + let heap = self.match_heap("expected heap number: heap«n»")?; self.match_token( Token::Equal, "expected '=' in heap declaration", @@ -1227,9 +1186,7 @@ impl<'a> Parser<'a> { "bound" => { data.style = match style_name { "dynamic" => { - HeapStyle::Dynamic { - bound_gv: self.match_gv_preamble("expected gv bound")?, - } + HeapStyle::Dynamic { bound_gv: self.match_gv("expected gv bound")? } } "static" => { HeapStyle::Static { bound: self.match_imm64("expected integer bound")? } @@ -1244,21 +1201,33 @@ impl<'a> Parser<'a> { } } - Ok((number, data)) + // Collect any trailing comments. + self.token(); + self.claim_gathered_comments(heap); + + Ok((heap, data)) } // Parse a signature decl. // // signature-decl ::= SigRef(sigref) "=" signature // - fn parse_signature_decl(&mut self, unique_isa: Option<&TargetIsa>) -> Result<(u32, Signature)> { - let number = self.match_sig("expected signature number: sig«n»")?; + fn parse_signature_decl( + &mut self, + unique_isa: Option<&TargetIsa>, + ) -> Result<(SigRef, Signature)> { + let sig = self.match_sig("expected signature number: sig«n»")?; self.match_token( Token::Equal, "expected '=' in signature decl", )?; let data = self.parse_signature(unique_isa)?; - Ok((number, data)) + + // Collect any trailing comments. + self.token(); + self.claim_gathered_comments(sig); + + Ok((sig, data)) } // Parse a function decl. @@ -1271,8 +1240,8 @@ impl<'a> Parser<'a> { // The first variant allocates a new signature reference. The second references an existing // signature which must be declared first. // - fn parse_function_decl(&mut self, ctx: &mut Context) -> Result<(u32, ExtFuncData)> { - let number = self.match_fn("expected function number: fn«n»")?; + fn parse_function_decl(&mut self, ctx: &mut Context) -> Result<(FuncRef, ExtFuncData)> { + let fn_ = self.match_fn("expected function number: fn«n»")?; self.match_token( Token::Equal, "expected '=' in function decl", @@ -1291,7 +1260,13 @@ impl<'a> Parser<'a> { } } Some(Token::SigRef(sig_src)) => { - let sig = ctx.get_sig(sig_src, &self.loc)?; + let sig = match SigRef::with_number(sig_src) { + None => { + return err!(self.loc, "attempted to use invalid signature ss{}", sig_src) + } + Some(sig) => sig, + }; + ctx.check_sig(sig, &self.loc)?; self.consume(); let name = self.parse_external_name()?; ExtFuncData { @@ -1301,14 +1276,19 @@ impl<'a> Parser<'a> { } _ => return err!(self.loc, "expected 'function' or sig«n» in function decl"), }; - Ok((number, data)) + + // Collect any trailing comments. + self.token(); + self.claim_gathered_comments(fn_); + + Ok((fn_, data)) } // Parse a jump table decl. // // jump-table-decl ::= * JumpTable(jt) "=" "jump_table" jt-entry {"," jt-entry} - fn parse_jump_table_decl(&mut self) -> Result<(u32, JumpTableData)> { - let number = self.match_jt()?; + fn parse_jump_table_decl(&mut self) -> Result<(JumpTable, JumpTableData)> { + let jt = self.match_jt()?; self.match_token( Token::Equal, "expected '=' in jump_table decl", @@ -1323,7 +1303,11 @@ impl<'a> Parser<'a> { data.set_entry(idx, dest); } if !self.optional(Token::Comma) { - return Ok((number, data)); + // Collect any trailing comments. + self.token(); + self.claim_gathered_comments(jt); + + return Ok((jt, data)); } } @@ -1366,9 +1350,11 @@ impl<'a> Parser<'a> { // ebb-header ::= Ebb(ebb) [ebb-params] ":" // fn parse_extended_basic_block(&mut self, ctx: &mut Context) -> Result<()> { + // Collect comments for the next ebb. + self.start_gathering_comments(); + let ebb_num = self.match_ebb("expected EBB header")?; let ebb = ctx.add_ebb(ebb_num, &self.loc)?; - self.gather_comments(ebb); if !self.optional(Token::Colon) { // ebb-header ::= Ebb(ebb) [ * ebb-params ] ":" @@ -1379,6 +1365,10 @@ impl<'a> Parser<'a> { )?; } + // Collect any trailing comments. + self.token(); + self.claim_gathered_comments(ebb); + // extended-basic-block ::= ebb-header * { instruction } while match self.token() { Some(Token::Value(_)) | @@ -1397,6 +1387,12 @@ impl<'a> Parser<'a> { // inst-results ::= Value(v) { "," Value(v) } let results = self.parse_inst_results()?; + for result in &results { + while ctx.function.dfg.num_values() <= result.index() { + ctx.function.dfg.make_invalid_value_for_parser(); + } + } + match self.token() { Some(Token::Arrow) => { self.consume(); @@ -1474,15 +1470,20 @@ impl<'a> Parser<'a> { "expected ':' after EBB argument", )?; // ebb-param ::= Value(v) ":" * Type(t) arg-loc? + + while ctx.function.dfg.num_values() <= v.index() { + ctx.function.dfg.make_invalid_value_for_parser(); + } + let t = self.match_type("expected EBB argument type")?; // Allocate the EBB argument and add the mapping. - let value = ctx.function.dfg.append_ebb_param(ebb, t); - ctx.map.def_value(v, value, &v_location)?; + ctx.function.dfg.append_ebb_param_for_parser(ebb, t, v); + ctx.map.def_value(v, &v_location)?; // ebb-param ::= Value(v) ":" Type(t) * arg-loc? if self.optional(Token::LBracket) { let loc = self.parse_value_location(ctx)?; - ctx.function.locations[value] = loc; + ctx.function.locations[v] = loc; self.match_token( Token::RBracket, "expected ']' after value location", @@ -1496,15 +1497,18 @@ impl<'a> Parser<'a> { match self.token() { Some(Token::StackSlot(src_num)) => { self.consume(); - if let Some(ss) = ctx.map.get_ss(src_num) { - Ok(ValueLoc::Stack(ss)) - } else { - err!( - self.loc, - "attempted to use undefined stack slot ss{}", - src_num - ) - } + let ss = match StackSlot::with_number(src_num) { + None => { + return err!( + self.loc, + "attempted to use invalid stack slot ss{}", + src_num + ) + } + Some(ss) => ss, + }; + ctx.check_ss(ss, &self.loc)?; + Ok(ValueLoc::Stack(ss)) } Some(Token::Name(name)) => { self.consume(); @@ -1584,6 +1588,7 @@ impl<'a> Parser<'a> { // inst-results ::= * Value(v) { "," Value(v) } if let Some(Token::Value(v)) = self.token() { self.consume(); + results.push(v); // inst-results ::= Value(v) * { "," Value(v) } @@ -1605,7 +1610,13 @@ impl<'a> Parser<'a> { return err!(self.loc, "wrong number of aliases"); } let dest = self.match_value("expected value alias")?; - ctx.add_alias(results[0], dest, self.loc) + + ctx.function.dfg.make_value_alias_for_parser( + dest, + results[0], + ); + ctx.map.def_value(results[0], &self.loc)?; + Ok(()) } // Parse an instruction, append it to `ebb`. @@ -1621,8 +1632,8 @@ impl<'a> Parser<'a> { ctx: &mut Context, ebb: Ebb, ) -> Result<()> { - // Collect comments for the next instruction to be allocated. - self.gather_comments(ctx.function.dfg.next_inst()); + // Collect comments for the next instruction. + self.start_gathering_comments(); // instruction ::= [inst-results "="] * Opcode(opc) ["." Type] ... let opcode = if let Some(Token::Identifier(text)) = self.token() { @@ -1659,7 +1670,11 @@ impl<'a> Parser<'a> { &inst_data, )?; let inst = ctx.function.dfg.make_inst(inst_data); - let num_results = ctx.function.dfg.make_inst_results(inst, ctrl_typevar); + let num_results = ctx.function.dfg.make_inst_results_for_parser( + inst, + ctrl_typevar, + &results, + ); ctx.function.layout.append_inst(inst, ebb); ctx.map.def_entity(inst.into(), &opcode_loc).expect( "duplicate inst references created", @@ -1697,11 +1712,7 @@ impl<'a> Parser<'a> { // Now map the source result values to the just created instruction results. // Pass a reference to `ctx.values` instead of `ctx` itself since the `Values` iterator // holds a reference to `ctx.function`. - self.add_values( - &mut ctx.map, - results.into_iter(), - ctx.function.dfg.inst_results(inst).iter().cloned(), - )?; + self.add_values(&mut ctx.map, results.into_iter())?; if let Some(result_locations) = result_locations { for (&value, loc) in ctx.function.dfg.inst_results(inst).iter().zip( @@ -1712,6 +1723,10 @@ impl<'a> Parser<'a> { } } + // Collect any trailing comments. + self.token(); + self.claim_gathered_comments(inst); + Ok(()) } @@ -1732,54 +1747,35 @@ impl<'a> Parser<'a> { inst_data: &InstructionData, ) -> Result { let constraints = opcode.constraints(); - let ctrl_type = - match explicit_ctrl_type { - Some(t) => t, - None => { - if constraints.use_typevar_operand() { - // This is an opcode that supports type inference, AND there was no - // explicit type specified. Look up `ctrl_value` to see if it was defined - // already. - // TBD: If it is defined in another block, the type should have been - // specified explicitly. It is unfortunate that the correctness of IL - // depends on the layout of the blocks. - let ctrl_src_value = inst_data - .typevar_operand(&ctx.function.dfg.value_lists) - .expect("Constraints <-> Format inconsistency"); - ctx.function.dfg.value_type( - match ctx.map.get_value(ctrl_src_value) { - Some(v) => v, - None => { - if let Some(v) = ctx.aliases.get(&ctrl_src_value).and_then( - |&(aliased, _)| ctx.map.get_value(aliased), - ) - { - v - } else { - return err!( - self.loc, - "cannot determine type of operand {}", - ctrl_src_value - ); - } - } - }, - ) - } else if constraints.is_polymorphic() { - // This opcode does not support type inference, so the explicit type - // variable is required. - return err!( - self.loc, - "type variable required for polymorphic opcode, e.g. '{}.{}'", - opcode, - constraints.ctrl_typeset().unwrap().example() - ); - } else { - // This is a non-polymorphic opcode. No typevar needed. - VOID - } + let ctrl_type = match explicit_ctrl_type { + Some(t) => t, + None => { + if constraints.use_typevar_operand() { + // This is an opcode that supports type inference, AND there was no + // explicit type specified. Look up `ctrl_value` to see if it was defined + // already. + // TBD: If it is defined in another block, the type should have been + // specified explicitly. It is unfortunate that the correctness of IL + // depends on the layout of the blocks. + let ctrl_src_value = inst_data + .typevar_operand(&ctx.function.dfg.value_lists) + .expect("Constraints <-> Format inconsistency"); + ctx.function.dfg.value_type(ctrl_src_value) + } else if constraints.is_polymorphic() { + // This opcode does not support type inference, so the explicit type + // variable is required. + return err!( + self.loc, + "type variable required for polymorphic opcode, e.g. '{}.{}'", + opcode, + constraints.ctrl_typeset().unwrap().example() + ); + } else { + // This is a non-polymorphic opcode. No typevar needed. + VOID } - }; + } + }; // Verify that `ctrl_type` is valid for the controlling type variable. We don't want to // attempt deriving types from an incorrect basis. @@ -1805,13 +1801,12 @@ impl<'a> Parser<'a> { } // Add mappings for a list of source values to their corresponding new values. - fn add_values(&self, map: &mut SourceMap, results: S, new_results: V) -> Result<()> + fn add_values(&self, map: &mut SourceMap, new_results: V) -> Result<()> where - S: Iterator, V: Iterator, { - for (src, val) in results.zip(new_results) { - map.def_value(src, val, &self.loc)?; + for val in new_results { + map.def_value(val, &self.loc)?; } Ok(()) } @@ -1892,11 +1887,11 @@ impl<'a> Parser<'a> { } } InstructionFormat::UnaryGlobalVar => { + let gv = self.match_gv("expected global variable")?; + ctx.check_gv(gv, &self.loc)?; InstructionData::UnaryGlobalVar { opcode, - global_var: self.match_gv("expected global variable").and_then(|num| { - ctx.get_gv(num, &self.loc) - })?, + global_var: gv, } } InstructionFormat::Binary => { @@ -2036,7 +2031,8 @@ impl<'a> Parser<'a> { Token::Comma, "expected ',' between operands", )?; - let table = self.match_jt().and_then(|num| ctx.get_jt(num, &self.loc))?; + let table = self.match_jt()?; + ctx.check_jt(table, &self.loc)?; InstructionData::BranchTable { opcode, arg, table } } InstructionFormat::InsertLane => { @@ -2139,11 +2135,8 @@ impl<'a> Parser<'a> { } } InstructionFormat::Call => { - let func_ref = self.match_fn("expected function reference").and_then( - |num| { - ctx.get_fn(num, &self.loc) - }, - )?; + let func_ref = self.match_fn("expected function reference")?; + ctx.check_fn(func_ref, &self.loc)?; self.match_token( Token::LPar, "expected '(' before arguments", @@ -2160,11 +2153,8 @@ impl<'a> Parser<'a> { } } InstructionFormat::IndirectCall => { - let sig_ref = self.match_sig("expected signature reference").and_then( - |num| { - ctx.get_sig(num, &self.loc) - }, - )?; + let sig_ref = self.match_sig("expected signature reference")?; + ctx.check_sig(sig_ref, &self.loc)?; self.match_token( Token::Comma, "expected ',' between operands", @@ -2186,16 +2176,13 @@ impl<'a> Parser<'a> { } } InstructionFormat::FuncAddr => { - let func_ref = self.match_fn("expected function reference").and_then( - |num| { - ctx.get_fn(num, &self.loc) - }, - )?; + let func_ref = self.match_fn("expected function reference")?; + ctx.check_fn(func_ref, &self.loc)?; InstructionData::FuncAddr { opcode, func_ref } } InstructionFormat::StackLoad => { - let ss = self.match_ss("expected stack slot number: ss«n»") - .and_then(|num| ctx.get_ss(num, &self.loc))?; + let ss = self.match_ss("expected stack slot number: ss«n»")?; + ctx.check_ss(ss, &self.loc)?; let offset = self.optional_offset32()?; InstructionData::StackLoad { opcode, @@ -2209,8 +2196,8 @@ impl<'a> Parser<'a> { Token::Comma, "expected ',' between operands", )?; - let ss = self.match_ss("expected stack slot number: ss«n»") - .and_then(|num| ctx.get_ss(num, &self.loc))?; + let ss = self.match_ss("expected stack slot number: ss«n»")?; + ctx.check_ss(ss, &self.loc)?; let offset = self.optional_offset32()?; InstructionData::StackStore { opcode, @@ -2220,9 +2207,8 @@ impl<'a> Parser<'a> { } } InstructionFormat::HeapAddr => { - let heap = self.match_heap("expected heap identifier").and_then(|h| { - ctx.get_heap(h, &self.loc) - })?; + let heap = self.match_heap("expected heap identifier")?; + ctx.check_heap(heap, &self.loc)?; self.match_token( Token::Comma, "expected ',' between operands", @@ -2306,8 +2292,8 @@ impl<'a> Parser<'a> { Token::Arrow, "expected '->' before destination stack slot", )?; - let dst = self.match_ss("expected stack slot number: ss«n»") - .and_then(|num| ctx.get_ss(num, &self.loc))?; + let dst = self.match_ss("expected stack slot number: ss«n»")?; + ctx.check_ss(dst, &self.loc)?; InstructionData::RegSpill { opcode, arg, @@ -2321,8 +2307,8 @@ impl<'a> Parser<'a> { Token::Comma, "expected ',' between operands", )?; - let src = self.match_ss("expected stack slot number: ss«n»") - .and_then(|num| ctx.get_ss(num, &self.loc))?; + let src = self.match_ss("expected stack slot number: ss«n»")?; + ctx.check_ss(src, &self.loc)?; self.match_token( Token::Arrow, "expected '->' before destination register units", @@ -2419,13 +2405,13 @@ mod tests { .unwrap(); assert_eq!(func.name.to_string(), "%qux"); let v4 = details.map.lookup_str("v4").unwrap(); - assert_eq!(v4.to_string(), "v0"); + assert_eq!(v4.to_string(), "v4"); let v3 = details.map.lookup_str("v3").unwrap(); - assert_eq!(v3.to_string(), "v2"); + assert_eq!(v3.to_string(), "v3"); match v3 { AnyEntity::Value(v3) => { let aliased_to = func.dfg.resolve_aliases(v3); - assert_eq!(aliased_to.to_string(), "v0"); + assert_eq!(aliased_to.to_string(), "v4"); } _ => panic!("expected value: {}", v3), } @@ -2495,14 +2481,16 @@ mod tests { .unwrap(); assert_eq!(func.name.to_string(), "%foo"); let mut iter = func.stack_slots.keys(); - let ss0 = iter.next().unwrap(); - assert_eq!(ss0.to_string(), "ss0"); - assert_eq!(func.stack_slots[ss0].kind, StackSlotKind::IncomingArg); - assert_eq!(func.stack_slots[ss0].size, 13); + let _ss0 = iter.next().unwrap(); let ss1 = iter.next().unwrap(); assert_eq!(ss1.to_string(), "ss1"); assert_eq!(func.stack_slots[ss1].kind, StackSlotKind::SpillSlot); assert_eq!(func.stack_slots[ss1].size, 1); + let _ss2 = iter.next().unwrap(); + let ss3 = iter.next().unwrap(); + assert_eq!(ss3.to_string(), "ss3"); + assert_eq!(func.stack_slots[ss3].kind, StackSlotKind::IncomingArg); + assert_eq!(func.stack_slots[ss3].size, 13); assert_eq!(iter.next(), None); // Catch duplicate definitions. @@ -2515,7 +2503,7 @@ mod tests { ).parse_function(None) .unwrap_err() .to_string(), - "3: duplicate stack slot: ss1" + "3: duplicate entity: ss1" ); } @@ -2565,10 +2553,10 @@ mod tests { text: "; decl", } ); - assert_eq!(comments[1].entity.to_string(), "ss0"); - assert_eq!(comments[2].entity.to_string(), "ss0"); + assert_eq!(comments[1].entity.to_string(), "ss10"); + assert_eq!(comments[2].entity.to_string(), "ss10"); assert_eq!(comments[2].text, "; Still stackslot."); - assert_eq!(comments[3].entity.to_string(), "jt0"); + assert_eq!(comments[3].entity.to_string(), "jt10"); assert_eq!(comments[3].text, "; Jumptable"); assert_eq!(comments[4].entity.to_string(), "ebb0"); assert_eq!(comments[4].text, "; Basic block"); diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 05d7e3ca3e..b3a2f0462f 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -1,31 +1,20 @@ -//! Source map for translating source entity names to parsed entities. +//! Source map associating entities with their source locations. //! -//! When the parser reads in a source file, entities like instructions, EBBs, and values get new -//! entity numbers. The parser maintains a mapping from the entity names in the source to the final -//! entity references. +//! When the parser reads in a source file, it records the locations of the +//! definitions of entities like instructions, EBBs, and values. //! -//! The `SourceMap` struct defined in this module makes the same mapping available to parser -//! clients. +//! The `SourceMap` struct defined in this module makes this mapping available +//! to parser clients. -use cretonne::entity::EntityRef; use cretonne::ir::entities::AnyEntity; use cretonne::ir::{StackSlot, GlobalVar, Heap, JumpTable, Ebb, Value, SigRef, FuncRef}; use error::{Result, Location}; use lexer::split_entity_name; use std::collections::HashMap; -/// Mapping from source entity names to entity references that are valid in the parsed function. +/// Mapping from entity names to source locations. #[derive(Debug)] pub struct SourceMap { - values: HashMap, // vNN - ebbs: HashMap, // ebbNN - stack_slots: HashMap, // ssNN - global_vars: HashMap, // gvNN - heaps: HashMap, // heapNN - signatures: HashMap, // sigNN - functions: HashMap, // fnNN - jump_tables: HashMap, // jtNN - // Store locations for entities, including instructions. locations: HashMap, } @@ -33,43 +22,43 @@ pub struct SourceMap { /// Read-only interface which is exposed outside the parser crate. impl SourceMap { /// Look up a value entity by its source number. - pub fn get_value(&self, src: Value) -> Option { - self.values.get(&src).cloned() + pub fn contains_value(&self, v: Value) -> bool { + self.locations.contains_key(&v.into()) } /// Look up a EBB entity by its source number. - pub fn get_ebb(&self, src: Ebb) -> Option { - self.ebbs.get(&src).cloned() + pub fn contains_ebb(&self, ebb: Ebb) -> bool { + self.locations.contains_key(&ebb.into()) } /// Look up a stack slot entity by its source number. - pub fn get_ss(&self, src_num: u32) -> Option { - self.stack_slots.get(&src_num).cloned() + pub fn contains_ss(&self, ss: StackSlot) -> bool { + self.locations.contains_key(&ss.into()) } /// Look up a global variable entity by its source number. - pub fn get_gv(&self, src_num: u32) -> Option { - self.global_vars.get(&src_num).cloned() + pub fn contains_gv(&self, gv: GlobalVar) -> bool { + self.locations.contains_key(&gv.into()) } /// Look up a heap entity by its source number. - pub fn get_heap(&self, src_num: u32) -> Option { - self.heaps.get(&src_num).cloned() + pub fn contains_heap(&self, heap: Heap) -> bool { + self.locations.contains_key(&heap.into()) } /// Look up a signature entity by its source number. - pub fn get_sig(&self, src_num: u32) -> Option { - self.signatures.get(&src_num).cloned() + pub fn contains_sig(&self, sig: SigRef) -> bool { + self.locations.contains_key(&sig.into()) } /// Look up a function entity by its source number. - pub fn get_fn(&self, src_num: u32) -> Option { - self.functions.get(&src_num).cloned() + pub fn contains_fn(&self, fn_: FuncRef) -> bool { + self.locations.contains_key(&fn_.into()) } /// Look up a jump table entity by its source number. - pub fn get_jt(&self, src_num: u32) -> Option { - self.jump_tables.get(&src_num).cloned() + pub fn contains_jt(&self, jt: JumpTable) -> bool { + self.locations.contains_key(&jt.into()) } /// Look up an entity by source name. @@ -77,193 +66,120 @@ impl SourceMap { pub fn lookup_str(&self, name: &str) -> Option { split_entity_name(name).and_then(|(ent, num)| match ent { "v" => { - Value::with_number(num) - .and_then(|v| self.get_value(v)) - .map(AnyEntity::Value) + Value::with_number(num).and_then(|v| if !self.contains_value(v) { + None + } else { + Some(v.into()) + }) } "ebb" => { - Ebb::with_number(num).and_then(|e| self.get_ebb(e)).map( - AnyEntity::Ebb, - ) + Ebb::with_number(num).and_then(|ebb| if !self.contains_ebb(ebb) { + None + } else { + Some(ebb.into()) + }) + } + "ss" => { + StackSlot::with_number(num).and_then(|ss| if !self.contains_ss(ss) { + None + } else { + Some(ss.into()) + }) + } + "gv" => { + GlobalVar::with_number(num).and_then(|gv| if !self.contains_gv(gv) { + None + } else { + Some(gv.into()) + }) + } + "heap" => { + Heap::with_number(num).and_then(|heap| if !self.contains_heap(heap) { + None + } else { + Some(heap.into()) + }) + } + "sig" => { + SigRef::with_number(num).and_then(|sig| if !self.contains_sig(sig) { + None + } else { + Some(sig.into()) + }) + } + "fn" => { + FuncRef::with_number(num).and_then(|fn_| if !self.contains_fn(fn_) { + None + } else { + Some(fn_.into()) + }) + } + "jt" => { + JumpTable::with_number(num).and_then(|jt| if !self.contains_jt(jt) { + None + } else { + Some(jt.into()) + }) } - "ss" => self.get_ss(num).map(AnyEntity::StackSlot), - "gv" => self.get_gv(num).map(AnyEntity::GlobalVar), - "heap" => self.get_heap(num).map(AnyEntity::Heap), - "sig" => self.get_sig(num).map(AnyEntity::SigRef), - "fn" => self.get_fn(num).map(AnyEntity::FuncRef), - "jt" => self.get_jt(num).map(AnyEntity::JumpTable), _ => None, }) } /// Get the source location where an entity was defined. - /// This looks up entities in the parsed function, not the source entity numbers. pub fn location(&self, entity: AnyEntity) -> Option { self.locations.get(&entity).cloned() } - - /// Rewrite an Ebb reference. - pub fn rewrite_ebb(&self, ebb: &mut Ebb, loc: AnyEntity) -> Result<()> { - match self.get_ebb(*ebb) { - Some(new) => { - *ebb = new; - Ok(()) - } - None => { - err!( - self.location(loc).unwrap_or_default(), - "undefined reference: {}", - ebb - ) - } - } - } - - /// Rewrite a value reference. - pub fn rewrite_value(&self, val: &mut Value, loc: AnyEntity) -> Result<()> { - match self.get_value(*val) { - Some(new) => { - *val = new; - Ok(()) - } - None => { - err!( - self.location(loc).unwrap_or_default(), - "undefined reference: {}", - val - ) - } - } - } - - /// Rewrite a slice of value references. - pub fn rewrite_values(&self, vals: &mut [Value], loc: AnyEntity) -> Result<()> { - for val in vals { - self.rewrite_value(val, loc)?; - } - Ok(()) - } - - /// Rewrite a `GlobalVar` reference. - pub fn rewrite_gv(&self, gv: &mut GlobalVar, loc: AnyEntity) -> Result<()> { - match self.get_gv(gv.index() as u32) { - Some(new) => { - *gv = new; - Ok(()) - } - None => { - err!( - self.location(loc).unwrap_or_default(), - "undefined reference: {}", - gv - ) - } - } - } } - -/// Interface for mutating a source map. -/// -/// This interface is provided for the parser itself, it is not made available outside the crate. -pub trait MutableSourceMap { - fn new() -> Self; - - /// Define a value mapping from the source name `src` to the final `entity`. - fn def_value(&mut self, src: Value, entity: Value, loc: &Location) -> Result<()>; - fn def_ebb(&mut self, src: Ebb, entity: Ebb, loc: &Location) -> Result<()>; - fn def_ss(&mut self, src_num: u32, entity: StackSlot, loc: &Location) -> Result<()>; - fn def_gv(&mut self, src_num: u32, entity: GlobalVar, loc: &Location) -> Result<()>; - fn def_heap(&mut self, src_num: u32, entity: Heap, loc: &Location) -> Result<()>; - fn def_sig(&mut self, src_num: u32, entity: SigRef, loc: &Location) -> Result<()>; - fn def_fn(&mut self, src_num: u32, entity: FuncRef, loc: &Location) -> Result<()>; - fn def_jt(&mut self, src_num: u32, entity: JumpTable, loc: &Location) -> Result<()>; - - /// Define an entity without an associated source number. This can be used for instructions - /// whose numbers never appear in source, or implicitly defined signatures. - fn def_entity(&mut self, entity: AnyEntity, loc: &Location) -> Result<()>; -} - -impl MutableSourceMap for SourceMap { - fn new() -> Self { - Self { - values: HashMap::new(), - ebbs: HashMap::new(), - stack_slots: HashMap::new(), - global_vars: HashMap::new(), - heaps: HashMap::new(), - signatures: HashMap::new(), - functions: HashMap::new(), - jump_tables: HashMap::new(), - locations: HashMap::new(), - } +impl SourceMap { + /// Create a new empty `SourceMap`. + pub fn new() -> Self { + Self { locations: HashMap::new() } } - fn def_value(&mut self, src: Value, entity: Value, loc: &Location) -> Result<()> { - if self.values.insert(src, entity).is_some() { - err!(loc, "duplicate value: {}", src) - } else { - self.def_entity(entity.into(), loc) - } + /// Define the value `entity`. + pub fn def_value(&mut self, entity: Value, loc: &Location) -> Result<()> { + self.def_entity(entity.into(), loc) } - fn def_ebb(&mut self, src: Ebb, entity: Ebb, loc: &Location) -> Result<()> { - if self.ebbs.insert(src, entity).is_some() { - err!(loc, "duplicate EBB: {}", src) - } else { - self.def_entity(entity.into(), loc) - } + /// Define the ebb `entity`. + pub fn def_ebb(&mut self, entity: Ebb, loc: &Location) -> Result<()> { + self.def_entity(entity.into(), loc) } - fn def_ss(&mut self, src_num: u32, entity: StackSlot, loc: &Location) -> Result<()> { - if self.stack_slots.insert(src_num, entity).is_some() { - err!(loc, "duplicate stack slot: ss{}", src_num) - } else { - self.def_entity(entity.into(), loc) - } + /// Define the stack slot `entity`. + pub fn def_ss(&mut self, entity: StackSlot, loc: &Location) -> Result<()> { + self.def_entity(entity.into(), loc) } - fn def_gv(&mut self, src_num: u32, entity: GlobalVar, loc: &Location) -> Result<()> { - if self.global_vars.insert(src_num, entity).is_some() { - err!(loc, "duplicate global variable: gv{}", src_num) - } else { - self.def_entity(entity.into(), loc) - } + /// Define the global variable `entity`. + pub fn def_gv(&mut self, entity: GlobalVar, loc: &Location) -> Result<()> { + self.def_entity(entity.into(), loc) } - fn def_heap(&mut self, src_num: u32, entity: Heap, loc: &Location) -> Result<()> { - if self.heaps.insert(src_num, entity).is_some() { - err!(loc, "duplicate heap: heap{}", src_num) - } else { - self.def_entity(entity.into(), loc) - } + /// Define the heap `entity`. + pub fn def_heap(&mut self, entity: Heap, loc: &Location) -> Result<()> { + self.def_entity(entity.into(), loc) } - fn def_sig(&mut self, src_num: u32, entity: SigRef, loc: &Location) -> Result<()> { - if self.signatures.insert(src_num, entity).is_some() { - err!(loc, "duplicate signature: sig{}", src_num) - } else { - self.def_entity(entity.into(), loc) - } + /// Define the signature `entity`. + pub fn def_sig(&mut self, entity: SigRef, loc: &Location) -> Result<()> { + self.def_entity(entity.into(), loc) } - fn def_fn(&mut self, src_num: u32, entity: FuncRef, loc: &Location) -> Result<()> { - if self.functions.insert(src_num, entity).is_some() { - err!(loc, "duplicate function: fn{}", src_num) - } else { - self.def_entity(entity.into(), loc) - } + /// Define the external function `entity`. + pub fn def_fn(&mut self, entity: FuncRef, loc: &Location) -> Result<()> { + self.def_entity(entity.into(), loc) } - fn def_jt(&mut self, src_num: u32, entity: JumpTable, loc: &Location) -> Result<()> { - if self.jump_tables.insert(src_num, entity).is_some() { - err!(loc, "duplicate jump table: jt{}", src_num) - } else { - self.def_entity(entity.into(), loc) - } + /// Define the jump table `entity`. + pub fn def_jt(&mut self, entity: JumpTable, loc: &Location) -> Result<()> { + self.def_entity(entity.into(), loc) } - fn def_entity(&mut self, entity: AnyEntity, loc: &Location) -> Result<()> { + /// Define an entity. This can be used for instructions whose numbers never + /// appear in source, or implicitly defined signatures. + pub fn def_entity(&mut self, entity: AnyEntity, loc: &Location) -> Result<()> { if self.locations.insert(entity, *loc).is_some() { err!(loc, "duplicate entity: {}", entity) } else { @@ -290,11 +206,11 @@ mod tests { assert_eq!(map.lookup_str("v0"), None); assert_eq!(map.lookup_str("ss1"), None); - assert_eq!(map.lookup_str("ss10").unwrap().to_string(), "ss0"); - assert_eq!(map.lookup_str("jt10").unwrap().to_string(), "jt0"); + assert_eq!(map.lookup_str("ss10").unwrap().to_string(), "ss10"); + assert_eq!(map.lookup_str("jt10").unwrap().to_string(), "jt10"); assert_eq!(map.lookup_str("ebb0").unwrap().to_string(), "ebb0"); - assert_eq!(map.lookup_str("v4").unwrap().to_string(), "v0"); - assert_eq!(map.lookup_str("v7").unwrap().to_string(), "v1"); - assert_eq!(map.lookup_str("v10").unwrap().to_string(), "v2"); + assert_eq!(map.lookup_str("v4").unwrap().to_string(), "v4"); + assert_eq!(map.lookup_str("v7").unwrap().to_string(), "v7"); + assert_eq!(map.lookup_str("v10").unwrap().to_string(), "v10"); } } diff --git a/lib/reader/src/testfile.rs b/lib/reader/src/testfile.rs index fcb11590e8..20e03939a1 100644 --- a/lib/reader/src/testfile.rs +++ b/lib/reader/src/testfile.rs @@ -36,8 +36,7 @@ pub struct Details<'a> { pub location: Location, /// Annotation comments that appeared inside or after the function. pub comments: Vec>, - /// Mapping of source entity numbers to parsed entity numbers. - /// Source locations of parsed entities. + /// Mapping of entity numbers to source locations. pub map: SourceMap, } From 10dcfcacdb312262b2e037789cd98325d72803f7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 20 Feb 2018 14:31:01 -0800 Subject: [PATCH 1512/3084] Remove support for entity variables in filecheck. Now that the parser doesn't renumber indices, there's no need for entity variables like $v0. --- cranelift/docs/testing.rst | 37 +----- .../filetests/isa/intel/legalize-custom.cton | 38 +++--- .../isa/intel/legalize-div-traps.cton | 36 +++--- .../filetests/isa/intel/legalize-div.cton | 22 ++-- .../filetests/isa/intel/legalize-libcall.cton | 2 +- .../filetests/isa/intel/legalize-memory.cton | 40 +++--- cranelift/filetests/isa/riscv/encoding.cton | 6 +- cranelift/filetests/isa/riscv/expand-i32.cton | 10 +- .../filetests/isa/riscv/legalize-abi.cton | 56 ++++----- .../filetests/isa/riscv/legalize-i64.cton | 16 +-- cranelift/filetests/isa/riscv/split-args.cton | 24 ++-- cranelift/filetests/parser/branch.cton | 34 ++--- cranelift/filetests/parser/call.cton | 34 ++--- cranelift/filetests/parser/flags.cton | 18 +-- .../parser/instruction_encoding.cton | 12 +- cranelift/filetests/parser/memory.cton | 36 +++--- cranelift/filetests/parser/rewrite.cton | 24 ++-- cranelift/filetests/parser/ternary.cton | 12 +- cranelift/filetests/parser/tiny.cton | 116 +++++++++--------- cranelift/filetests/regalloc/basic.cton | 14 +-- cranelift/filetests/regalloc/coalesce.cton | 22 ++-- cranelift/filetests/regalloc/constraints.cton | 26 ++-- .../filetests/regalloc/intel-regres.cton | 6 +- .../filetests/regalloc/multi-constraints.cton | 4 +- cranelift/filetests/regalloc/reload-208.cton | 4 +- cranelift/filetests/regalloc/reload.cton | 8 +- cranelift/filetests/regalloc/spill.cton | 48 ++++---- cranelift/filetests/simple_gvn/basic.cton | 4 +- cranelift/src/filetest/subtest.rs | 29 ++--- 29 files changed, 343 insertions(+), 395 deletions(-) diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index c7654713ea..951149c044 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -160,20 +160,6 @@ directives in the test file. LLVM's :command:`FileCheck` command has a ``CHECK-LABEL:`` directive to help separate the output from different functions. Cretonne's tests don't need this. -Filecheck variables -~~~~~~~~~~~~~~~~~~~ - -Cretonne's IL parser causes entities like values and EBBs to be renumbered. It -maintains a source mapping to resolve references in the text, but when a -function is written out as text as part of a test, all of the entities have the -new numbers. This can complicate the filecheck directives since they need to -refer to the new entity numbers, not the ones in the adjacent source text. - -To help with this, the parser's source-to-entity mapping is made available as -predefined filecheck variables. A value by the source name ``v10`` can be -referenced as the filecheck variable ``$v10``. The variable expands to the -renumbered entity name. - `test cat` ---------- @@ -192,26 +178,9 @@ Example:: } ; sameln: function %r1() -> i32, f32 { ; nextln: ebb0: - ; nextln: v0 = iconst.i32 3 - ; nextln: v1 = f32const 0.0 - ; nextln: return v0, v1 - ; nextln: } - -Notice that the values ``v10`` and ``v20`` in the source were renumbered to -``v0`` and ``v1`` respectively during parsing. The equivalent test using -filecheck variables would be:: - - function %r1() -> i32, f32 { - ebb1: - v10 = iconst.i32 3 - v20 = f32const 0.0 - return v10, v20 - } - ; sameln: function %r1() -> i32, f32 { - ; nextln: ebb0: - ; nextln: $v10 = iconst.i32 3 - ; nextln: $v20 = f32const 0.0 - ; nextln: return $v10, $v20 + ; nextln: v10 = iconst.i32 3 + ; nextln: v20 = f32const 0.0 + ; nextln: return v10, v20 ; nextln: } `test verifier` diff --git a/cranelift/filetests/isa/intel/legalize-custom.cton b/cranelift/filetests/isa/intel/legalize-custom.cton index 41411ec21a..5cb5ac0462 100644 --- a/cranelift/filetests/isa/intel/legalize-custom.cton +++ b/cranelift/filetests/isa/intel/legalize-custom.cton @@ -11,8 +11,8 @@ function %cond_trap(i32) { ebb0(v1: i32): trapz v1, user67 return - ; check: $ebb0($v1: i32 - ; nextln: $(f=$V) = ifcmp_imm $v1, 0 + ; check: ebb0(v1: i32 + ; nextln: $(f=$V) = ifcmp_imm v1, 0 ; nextln: trapif eq $f, user67 ; nextln: return } @@ -21,8 +21,8 @@ function %cond_trap2(i32) { ebb0(v1: i32): trapnz v1, int_ovf return - ; check: $ebb0($v1: i32 - ; nextln: $(f=$V) = ifcmp_imm $v1, 0 + ; check: ebb0(v1: i32 + ; nextln: $(f=$V) = ifcmp_imm v1, 0 ; nextln: trapif ne $f, int_ovf ; nextln: return } @@ -32,8 +32,8 @@ ebb0(v1: i32): v2 = icmp_imm eq v1, 6 trapz v2, user7 return - ; check: $ebb0($v1: i32 - ; check: brnz $v2, $(new=$EBB) + ; check: ebb0(v1: i32 + ; check: brnz v2, $(new=$EBB) ; nextln: trap user7 ; check: $new: ; nextln: return @@ -44,8 +44,8 @@ ebb0(v1: i32): v2 = icmp_imm eq v1, 6 trapnz v2, user9 return - ; check: $ebb0($v1: i32 - ; check: brz $v2, $(new=$EBB) + ; check: ebb0(v1: i32 + ; check: brz v2, $(new=$EBB) ; nextln: trap user9 ; check: $new: ; nextln: return @@ -55,7 +55,7 @@ function %f32const() -> f32 { ebb0: v1 = f32const 0x1.0p1 ; check: $(tmp=$V) = iconst.i32 - ; check: $v1 = bitcast.f32 $tmp + ; check: v1 = bitcast.f32 $tmp return v1 } @@ -63,17 +63,17 @@ function %f64const() -> f64 { ebb0: v1 = f64const 0x1.0p1 ; check: $(tmp=$V) = iconst.i64 - ; check: $v1 = bitcast.f64 $tmp + ; check: v1 = bitcast.f64 $tmp return v1 } function %select_f64(f64, f64, i32) -> f64 { ebb0(v0: f64, v1: f64, v2: i32): v3 = select v2, v0, v1 - ; check: brnz v2, $(new=$EBB)($v0) - ; nextln: jump $new($v1) - ; check: $new($v3: f64): - ; nextln: return $v3 + ; check: brnz v2, $(new=$EBB)(v0) + ; nextln: jump $new(v1) + ; check: $new(v3: f64): + ; nextln: return v3 return v3 } @@ -81,17 +81,17 @@ function %f32_min(f32, f32) -> f32 { ebb0(v0: f32, v1: f32): v2 = fmin v0, v1 return v2 - ; check: $(vnat=$V) = x86_fmin $v0, $v1 + ; check: $(vnat=$V) = x86_fmin v0, v1 ; nextln: jump $(done=$EBB)($vnat) ; check: $(uno=$EBB): - ; nextln: $(vuno=$V) = fadd.f32 $v0, $v1 + ; nextln: $(vuno=$V) = fadd.f32 v0, v1 ; nextln: jump $(done=$EBB)($vuno) ; check: $(ueq=$EBB): - ; check: $(veq=$V) = bor.f32 $v0, $v1 + ; check: $(veq=$V) = bor.f32 v0, v1 ; nextln: jump $(done=$EBB)($veq) - ; check: $done($v2: f32): - ; nextln: return $v2 + ; check: $done(v2: f32): + ; nextln: return v2 } diff --git a/cranelift/filetests/isa/intel/legalize-div-traps.cton b/cranelift/filetests/isa/intel/legalize-div-traps.cton index 2077ab9bf8..fa070fba1a 100644 --- a/cranelift/filetests/isa/intel/legalize-div-traps.cton +++ b/cranelift/filetests/isa/intel/legalize-div-traps.cton @@ -10,63 +10,63 @@ isa intel function %udiv(i64, i64) -> i64 { ebb0(v0: i64, v1: i64): - ; check: $ebb0( + ; check: ebb0( v2 = udiv v0, v1 - ; nextln: $(fz=$V) = ifcmp_imm $v1, 0 + ; nextln: $(fz=$V) = ifcmp_imm v1, 0 ; nextln: trapif eq $fz, int_divz ; nextln: $(hi=$V) = iconst.i64 0 - ; nextln: $(d=$V), $(r=$V) = x86_udivmodx $v0, $hi, $v1 + ; nextln: $(d=$V), $(r=$V) = x86_udivmodx v0, $hi, v1 return v2 ; nextln: return $d } function %urem(i64, i64) -> i64 { ebb0(v0: i64, v1: i64): - ; check: $ebb0( + ; check: ebb0( v2 = urem v0, v1 - ; nextln: $(fz=$V) = ifcmp_imm $v1, 0 + ; nextln: $(fz=$V) = ifcmp_imm v1, 0 ; nextln: trapif eq $fz, int_divz ; nextln: $(hi=$V) = iconst.i64 0 - ; nextln: $(d=$V), $(r=$V) = x86_udivmodx $v0, $hi, $v1 + ; nextln: $(d=$V), $(r=$V) = x86_udivmodx v0, $hi, v1 return v2 ; nextln: return $r } function %sdiv(i64, i64) -> i64 { ebb0(v0: i64, v1: i64): - ; check: $ebb0( + ; check: ebb0( v2 = sdiv v0, v1 - ; nextln: $(fm1=$V) = ifcmp_imm $v1, -1 + ; nextln: $(fm1=$V) = ifcmp_imm v1, -1 ; nextln: brif eq $fm1, $(m1=$EBB) - ; nextln: $(fz=$V) = ifcmp_imm $v1, 0 + ; nextln: $(fz=$V) = ifcmp_imm v1, 0 ; nextln: trapif eq $fz, int_divz ; check: $(hi=$V) = sshr - ; nextln: $(q=$V), $(r=$V) = x86_sdivmodx $v0, $hi, $v1 + ; nextln: $(q=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1 ; nextln: jump $(done=$EBB)($q) ; check: $m1: ; nextln: $(imin=$V) = iconst.i64 0x8000_0000_0000_0000 - ; nextln: $(fm=$V) = ifcmp.i64 $v0, $imin + ; nextln: $(fm=$V) = ifcmp.i64 v0, $imin ; nextln: trapif eq $fm, int_ovf - ; check: $done($v2: i64): + ; check: $done(v2: i64): return v2 - ; nextln: return $v2 + ; nextln: return v2 } ; The srem expansion needs to special-case x % -1 since x86_sdivmodx traps on INT_MIN/-1. ; TODO: Add more explicit pattern matching once we've cleaned up the ifcmp+brif pattern. function %srem(i64, i64) -> i64 { ebb0(v0: i64, v1: i64): - ; check: $ebb0( + ; check: ebb0( v2 = srem v0, v1 - ; nextln: $(fm1=$V) = ifcmp_imm $v1, -1 + ; nextln: $(fm1=$V) = ifcmp_imm v1, -1 ; nextln: brif eq $fm1, $(m1=$EBB) ; check: $(hi=$V) = sshr - ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx $v0, $hi, $v1 + ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1 ; nextln: jump $(done=$EBB)($r) ; check: $m1: ; nextln: $(zero=$V) = iconst.i64 0 ; nextln: jump $(done=$EBB)($zero) - ; check: $done($v2: i64): + ; check: $done(v2: i64): return v2 - ; nextln: return $v2 + ; nextln: return v2 } diff --git a/cranelift/filetests/isa/intel/legalize-div.cton b/cranelift/filetests/isa/intel/legalize-div.cton index 8e34c01b86..ec3f8ec5d3 100644 --- a/cranelift/filetests/isa/intel/legalize-div.cton +++ b/cranelift/filetests/isa/intel/legalize-div.cton @@ -10,30 +10,30 @@ isa intel function %udiv(i64, i64) -> i64 { ebb0(v0: i64, v1: i64): - ; check: $ebb0( + ; check: ebb0( v2 = udiv v0, v1 ; nextln: $(hi=$V) = iconst.i64 0 - ; nextln: $(d=$V), $(r=$V) = x86_udivmodx $v0, $hi, $v1 + ; nextln: $(d=$V), $(r=$V) = x86_udivmodx v0, $hi, v1 return v2 ; nextln: return $d } function %urem(i64, i64) -> i64 { ebb0(v0: i64, v1: i64): - ; check: $ebb0( + ; check: ebb0( v2 = urem v0, v1 ; nextln: $(hi=$V) = iconst.i64 0 - ; nextln: $(d=$V), $(r=$V) = x86_udivmodx $v0, $hi, $v1 + ; nextln: $(d=$V), $(r=$V) = x86_udivmodx v0, $hi, v1 return v2 ; nextln: return $r } function %sdiv(i64, i64) -> i64 { ebb0(v0: i64, v1: i64): - ; check: $ebb0( + ; check: ebb0( v2 = sdiv v0, v1 ; check: $(hi=$V) = sshr - ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx $v0, $hi, $v1 + ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1 return v2 ; nextln: return $d } @@ -42,17 +42,17 @@ ebb0(v0: i64, v1: i64): ; TODO: Add more explicit pattern matching once we've cleaned up the ifcmp+brif pattern. function %srem(i64, i64) -> i64 { ebb0(v0: i64, v1: i64): - ; check: $ebb0( + ; check: ebb0( v2 = srem v0, v1 - ; nextln: $(fm1=$V) = ifcmp_imm $v1, -1 + ; nextln: $(fm1=$V) = ifcmp_imm v1, -1 ; nextln: brif eq $fm1, $(m1=$EBB) ; check: $(hi=$V) = sshr - ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx $v0, $hi, $v1 + ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1 ; nextln: jump $(done=$EBB)($r) ; check: $m1: ; nextln: $(zero=$V) = iconst.i64 0 ; nextln: jump $(done=$EBB)($zero) - ; check: $done($v2: i64): + ; check: $done(v2: i64): return v2 - ; nextln: return $v2 + ; nextln: return v2 } diff --git a/cranelift/filetests/isa/intel/legalize-libcall.cton b/cranelift/filetests/isa/intel/legalize-libcall.cton index 2d249fd8df..562efb6beb 100644 --- a/cranelift/filetests/isa/intel/legalize-libcall.cton +++ b/cranelift/filetests/isa/intel/legalize-libcall.cton @@ -12,4 +12,4 @@ ebb0(v0: f32): ; check: function %floor(f32 [%xmm0]) -> f32 [%xmm0] native { ; check: sig0 = (f32) -> f32 native ; check: fn0 = sig0 %FloorF32 -; check: $v1 = call fn0($v0) +; check: v1 = call fn0(v0) diff --git a/cranelift/filetests/isa/intel/legalize-memory.cton b/cranelift/filetests/isa/intel/legalize-memory.cton index 28a0fd6a15..5b7246a2e3 100644 --- a/cranelift/filetests/isa/intel/legalize-memory.cton +++ b/cranelift/filetests/isa/intel/legalize-memory.cton @@ -11,9 +11,9 @@ function %vmctx(i64 vmctx) -> i64 { ebb1(v1: i64): v2 = global_addr.i64 gv1 - ; check: $v2 = iadd_imm $v1, -16 + ; check: v2 = iadd_imm v1, -16 return v2 - ; check: return $v2 + ; check: return v2 } function %deref(i64 vmctx) -> i64 { @@ -22,11 +22,11 @@ function %deref(i64 vmctx) -> i64 { ebb1(v1: i64): v2 = global_addr.i64 gv2 - ; check: $(a1=$V) = iadd_imm $v1, -16 + ; check: $(a1=$V) = iadd_imm v1, -16 ; check: $(p1=$V) = load.i64 $a1 - ; check: $v2 = iadd_imm $p1, 32 + ; check: v2 = iadd_imm $p1, 32 return v2 - ; check: return $v2 + ; check: return v2 } function %sym() -> i64 { @@ -35,9 +35,9 @@ function %sym() -> i64 { ebb1: v0 = global_addr.i64 gv0 - ; check: $v0 = globalsym_addr.i64 gv0 + ; check: v0 = globalsym_addr.i64 gv0 v1 = global_addr.i64 gv1 - ; check: $v1 = globalsym_addr.i64 gv1 + ; check: v1 = globalsym_addr.i64 gv1 v2 = bxor v0, v1 return v2 } @@ -49,18 +49,18 @@ function %staticheap_sm64(i32, i64 vmctx) -> f32 spiderwasm { heap0 = static gv0, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000 ebb0(v0: i32, v999: i64): - ; check: $ebb0( + ; check: ebb0( v1 = heap_addr.i64 heap0, v0, 1 ; Boundscheck should be eliminated. ; Checks here are assuming that no pipehole opts fold the load offsets. - ; nextln: $(xoff=$V) = uextend.i64 $v0 - ; nextln: $(haddr=$V) = iadd_imm $v999, 64 + ; nextln: $(xoff=$V) = uextend.i64 v0 + ; nextln: $(haddr=$V) = iadd_imm v999, 64 ; nextln: $(hbase=$V) = load.i64 $haddr - ; nextln: $v1 = iadd $hbase, $xoff + ; nextln: v1 = iadd $hbase, $xoff v2 = load.f32 v1+16 - ; nextln: $v2 = load.f32 $v1+16 + ; nextln: v2 = load.f32 v1+16 v3 = load.f32 v1+20 - ; nextln: $v3 = load.f32 $v1+20 + ; nextln: v3 = load.f32 v1+20 v4 = fadd v2, v3 return v4 } @@ -73,7 +73,7 @@ ebb0(v0: i32, v999: i64): ; Everything after the obviously OOB access should be eliminated, leaving ; the `trap heap_oob` instruction as the terminator of the Ebb and moving ; the remainder of the instructions into an inaccessible Ebb. - ; check: $ebb0( + ; check: ebb0( ; nextln: trap heap_oob ; check: ebb1: ; nextln: v1 = iconst.i64 0 @@ -93,7 +93,7 @@ function %staticheap_sm64(i32, i64 vmctx) -> f32 spiderwasm { heap0 = static gv0, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000 ebb0(v0: i32, v999: i64): - ; check: $ebb0( + ; check: ebb0( v1 = heap_addr.i64 heap0, v0, 0x8000_0000 ; Boundscheck code ; check: $(oob=$V) = icmp @@ -101,12 +101,12 @@ ebb0(v0: i32, v999: i64): ; nextln: trap heap_oob ; check: $ok: ; Checks here are assuming that no pipehole opts fold the load offsets. - ; nextln: $(xoff=$V) = uextend.i64 $v0 - ; nextln: $(haddr=$V) = iadd_imm.i64 $v999, 64 + ; nextln: $(xoff=$V) = uextend.i64 v0 + ; nextln: $(haddr=$V) = iadd_imm.i64 v999, 64 ; nextln: $(hbase=$V) = load.i64 $haddr - ; nextln: $v1 = iadd $hbase, $xoff + ; nextln: v1 = iadd $hbase, $xoff v2 = load.f32 v1+0x7fff_ffff - ; nextln: $v2 = load.f32 $v1+0x7fff_ffff + ; nextln: v2 = load.f32 v1+0x7fff_ffff return v2 } @@ -116,7 +116,7 @@ function %stkchk(i64 vmctx) spiderwasm { gv0 = vmctx+64 ebb0(v0: i64): - ; check: $ebb0( + ; check: ebb0( stack_check gv0 ; check: $(limit=$V) = load.i64 notrap aligned ; check: $(flags=$V) = ifcmp_sp $limit diff --git a/cranelift/filetests/isa/riscv/encoding.cton b/cranelift/filetests/isa/riscv/encoding.cton index 71961ba353..9dc6f388be 100644 --- a/cranelift/filetests/isa/riscv/encoding.cton +++ b/cranelift/filetests/isa/riscv/encoding.cton @@ -5,15 +5,15 @@ function %int32(i32, i32) { ebb0(v1: i32, v2: i32): v10 = iadd v1, v2 ; check: [R#0c] - ; sameln: $v10 = iadd + ; sameln: v10 = iadd v11 = isub v1, v2 ; check: [R#200c] - ; sameln: $v11 = isub + ; sameln: v11 = isub v12 = imul v1, v2 ; check: [R#10c] - ; sameln: $v12 = imul + ; sameln: v12 = imul return ; check: [Iret#19] diff --git a/cranelift/filetests/isa/riscv/expand-i32.cton b/cranelift/filetests/isa/riscv/expand-i32.cton index 885d97d309..890352e72f 100644 --- a/cranelift/filetests/isa/riscv/expand-i32.cton +++ b/cranelift/filetests/isa/riscv/expand-i32.cton @@ -14,9 +14,9 @@ ebb0(v1: i32, v2: i32): v3, v4 = iadd_cout v1, v2 return v3, v4 } -; check: $v3 = iadd $v1, $v2 -; check: $v4 = icmp ult $v3, $v1 -; check: return $v3, $v4 +; check: v3 = iadd v1, v2 +; check: v4 = icmp ult v3, v1 +; check: return v3, v4 ; Expanding illegal immediate constants. ; Note that at some point we'll probably expand the iconst as well. @@ -26,8 +26,8 @@ ebb0(v0: i32): return v1 } ; check: $(cst=$V) = iconst.i32 0x3b9a_ca00 -; check: $v1 = iadd $v0, $cst -; check: return $v1 +; check: v1 = iadd v0, $cst +; check: return v1 function %bitclear(i32, i32) -> i32 { ebb0(v0: i32, v1: i32): diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index f80494cc1a..3234407052 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -8,10 +8,10 @@ isa riscv function %int_split_args(i64) -> i64 { ebb0(v0: i64): - ; check: $ebb0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32): - ; check: $v0 = iconcat $v0l, $v0h + ; check: ebb0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32): + ; check: v0 = iconcat $v0l, $v0h v1 = iadd_imm v0, 1 - ; check: $(v1l=$V), $(v1h=$V) = isplit $v1 + ; check: $(v1l=$V), $(v1h=$V) = isplit v1 ; check: return $v1l, $v1h, $link return v1 } @@ -22,10 +22,10 @@ function %split_call_arg(i32) { ebb0(v0: i32): v1 = uextend.i64 v0 call fn1(v1) - ; check: $(v1l=$V), $(v1h=$V) = isplit $v1 - ; check: call $fn1($v1l, $v1h) + ; check: $(v1l=$V), $(v1h=$V) = isplit v1 + ; check: call fn1($v1l, $v1h) call fn2(v0, v1) - ; check: call $fn2($v0, $V, $V) + ; check: call fn2(v0, $V, $V) return } @@ -33,11 +33,11 @@ function %split_ret_val() { fn1 = function %foo() -> i64 ebb0: v1 = call fn1() - ; check: $ebb0($(link=$V): i32): - ; nextln: $(v1l=$V), $(v1h=$V) = call $fn1() - ; check: $v1 = iconcat $v1l, $v1h + ; check: ebb0($(link=$V): i32): + ; nextln: $(v1l=$V), $(v1h=$V) = call fn1() + ; check: v1 = iconcat $v1l, $v1h jump ebb1(v1) - ; check: jump $ebb1($v1) + ; check: jump ebb1(v1) ebb1(v10: i64): jump ebb1(v10) @@ -48,11 +48,11 @@ function %split_ret_val2() { fn1 = function %foo() -> i32, i64 ebb0: v1, v2 = call fn1() - ; check: $ebb0($(link=$V): i32): - ; nextln: $v1, $(v2l=$V), $(v2h=$V) = call $fn1() - ; check: $v2 = iconcat $v2l, $v2h + ; check: ebb0($(link=$V): i32): + ; nextln: v1, $(v2l=$V), $(v2h=$V) = call fn1() + ; check: v2 = iconcat $v2l, $v2h jump ebb1(v1, v2) - ; check: jump $ebb1($v1, $v2) + ; check: jump ebb1(v1, v2) ebb1(v9: i32, v10: i64): jump ebb1(v9, v10) @@ -60,10 +60,10 @@ ebb1(v9: i32, v10: i64): function %int_ext(i8, i8 sext, i8 uext) -> i8 uext { ebb0(v1: i8, v2: i8, v3: i8): - ; check: $ebb0($v1: i8, $(v2x=$V): i32, $(v3x=$V): i32, $(link=$V): i32): - ; check: $v2 = ireduce.i8 $v2x - ; check: $v3 = ireduce.i8 $v3x - ; check: $(v1x=$V) = uextend.i32 $v1 + ; check: ebb0(v1: i8, $(v2x=$V): i32, $(v3x=$V): i32, $(link=$V): i32): + ; check: v2 = ireduce.i8 $v2x + ; check: v3 = ireduce.i8 $v3x + ; check: $(v1x=$V) = uextend.i32 v1 ; check: return $v1x, $link return v1 } @@ -73,11 +73,11 @@ function %ext_ret_val() { fn1 = function %foo() -> i8 sext ebb0: v1 = call fn1() - ; check: $ebb0($V: i32): - ; nextln: $(rv=$V) = call $fn1() - ; check: $v1 = ireduce.i8 $rv + ; check: ebb0($V: i32): + ; nextln: $(rv=$V) = call fn1() + ; check: v1 = ireduce.i8 $rv jump ebb1(v1) - ; check: jump $ebb1($v1) + ; check: jump ebb1(v1) ebb1(v10: i8): jump ebb1(v10) @@ -85,16 +85,16 @@ ebb1(v10: i8): function %vector_split_args(i64x4) -> i64x4 { ebb0(v0: i64x4): - ; check: $ebb0($(v0al=$V): i32, $(v0ah=$V): i32, $(v0bl=$V): i32, $(v0bh=$V): i32, $(v0cl=$V): i32, $(v0ch=$V): i32, $(v0dl=$V): i32, $(v0dh=$V): i32, $(link=$V): i32): + ; check: ebb0($(v0al=$V): i32, $(v0ah=$V): i32, $(v0bl=$V): i32, $(v0bh=$V): i32, $(v0cl=$V): i32, $(v0ch=$V): i32, $(v0dl=$V): i32, $(v0dh=$V): i32, $(link=$V): i32): ; check: $(v0a=$V) = iconcat $v0al, $v0ah ; check: $(v0b=$V) = iconcat $v0bl, $v0bh ; check: $(v0ab=$V) = vconcat $v0a, $v0b ; check: $(v0c=$V) = iconcat $v0cl, $v0ch ; check: $(v0d=$V) = iconcat $v0dl, $v0dh ; check: $(v0cd=$V) = vconcat $v0c, $v0d - ; check: $v0 = vconcat $v0ab, $v0cd + ; check: v0 = vconcat $v0ab, $v0cd v1 = bxor v0, v0 - ; check: $(v1ab=$V), $(v1cd=$V) = vsplit $v1 + ; check: $(v1ab=$V), $(v1cd=$V) = vsplit v1 ; check: $(v1a=$V), $(v1b=$V) = vsplit $v1ab ; check: $(v1al=$V), $(v1ah=$V) = isplit $v1a ; check: $(v1bl=$V), $(v1bh=$V) = isplit $v1b @@ -117,7 +117,7 @@ function %indirect_arg(i32, f32x2) { sig1 = (f32x2) native ebb0(v0: i32, v1: f32x2): call_indirect sig1, v0(v1) - ; check: call_indirect $sig1, $v0($V, $V) + ; check: call_indirect sig1, v0($V, $V) return } @@ -128,7 +128,7 @@ function %stack_args(i32) { ebb0(v0: i32): v1 = iconst.i64 1 call fn1(v1, v1, v1, v1, v0) - ; check: [GPsp#48,$ss0]$WS $(v0s=$V) = spill $v0 - ; check: call $fn1($(=.*), $v0s) + ; check: [GPsp#48,$ss0]$WS $(v0s=$V) = spill v0 + ; check: call fn1($(=.*), $v0s) return } diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton index e7a3441ee9..f06cf0803a 100644 --- a/cranelift/filetests/isa/riscv/legalize-i64.cton +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -9,12 +9,12 @@ ebb0(v1: i64, v2: i64): v3 = band v1, v2 return v3 } -; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): +; check: ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): ; check: [R#ec ; sameln: $(v3l=$V) = band $v1l, $v2l ; check: [R#ec ; sameln: $(v3h=$V) = band $v1h, $v2h -; check: $v3 = iconcat $v3l, $v3h +; check: v3 = iconcat $v3l, $v3h ; check: return $v3l, $v3h, $link function %bitwise_or(i64, i64) -> i64 { @@ -22,12 +22,12 @@ ebb0(v1: i64, v2: i64): v3 = bor v1, v2 return v3 } -; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): +; check: ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): ; check: [R#cc ; sameln: $(v3l=$V) = bor $v1l, $v2l ; check: [R#cc ; sameln: $(v3h=$V) = bor $v1h, $v2h -; check: $v3 = iconcat $v3l, $v3h +; check: v3 = iconcat $v3l, $v3h ; check: return $v3l, $v3h, $link function %bitwise_xor(i64, i64) -> i64 { @@ -35,12 +35,12 @@ ebb0(v1: i64, v2: i64): v3 = bxor v1, v2 return v3 } -; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): +; check: ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): ; check: [R#8c ; sameln: $(v3l=$V) = bxor $v1l, $v2l ; check: [R#8c ; sameln: $(v3h=$V) = bxor $v1h, $v2h -; check: $v3 = iconcat $v3l, $v3h +; check: v3 = iconcat $v3l, $v3h ; check: return $v3l, $v3h, $link function %arith_add(i64, i64) -> i64 { @@ -51,7 +51,7 @@ ebb0(v1: i64, v2: i64): v3 = iadd v1, v2 return v3 } -; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): +; check: ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): ; check: [R#0c ; sameln: $(v3l=$V) = iadd $v1l, $v2l ; check: $(c=$V) = icmp ult $v3l, $v1l @@ -60,5 +60,5 @@ ebb0(v1: i64, v2: i64): ; check: $(c_int=$V) = bint.i32 $c ; check: [R#0c ; sameln: $(v3h=$V) = iadd $v3h1, $c_int -; check: $v3 = iconcat $v3l, $v3h +; check: v3 = iconcat $v3l, $v3h ; check: return $v3l, $v3h, $link diff --git a/cranelift/filetests/isa/riscv/split-args.cton b/cranelift/filetests/isa/riscv/split-args.cton index be1370dc12..80de18fd7a 100644 --- a/cranelift/filetests/isa/riscv/split-args.cton +++ b/cranelift/filetests/isa/riscv/split-args.cton @@ -6,12 +6,12 @@ isa riscv function %simple(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): -; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): +; check: ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): jump ebb1(v1) - ; check: jump $ebb1($v1l, $v1h) + ; check: jump ebb1($v1l, $v1h) ebb1(v3: i64): -; check: $ebb1($(v3l=$V): i32, $(v3h=$V): i32): +; check: ebb1($(v3l=$V): i32, $(v3h=$V): i32): v4 = band v3, v2 ; check: $(v4l=$V) = band $v3l, $v2l ; check: $(v4h=$V) = band $v3h, $v2h @@ -21,17 +21,17 @@ ebb1(v3: i64): function %multi(i64) -> i64 { ebb1(v1: i64): -; check: $ebb1($(v1l=$V): i32, $(v1h=$V): i32, $(link=$V): i32): +; check: ebb1($(v1l=$V): i32, $(v1h=$V): i32, $(link=$V): i32): jump ebb2(v1, v1) - ; check: jump $ebb2($v1l, $v1l, $v1h, $v1h) + ; check: jump ebb2($v1l, $v1l, $v1h, $v1h) ebb2(v2: i64, v3: i64): -; check: $ebb2($(v2l=$V): i32, $(v3l=$V): i32, $(v2h=$V): i32, $(v3h=$V): i32): +; check: ebb2($(v2l=$V): i32, $(v3l=$V): i32, $(v2h=$V): i32, $(v3h=$V): i32): jump ebb3(v2) - ; check: jump $ebb3($v2l, $v2h) + ; check: jump ebb3($v2l, $v2h) ebb3(v4: i64): -; check: $ebb3($(v4l=$V): i32, $(v4h=$V): i32): +; check: ebb3($(v4l=$V): i32, $(v4h=$V): i32): v5 = band v4, v3 ; check: $(v5l=$V) = band $v4l, $v3l ; check: $(v5h=$V) = band $v4h, $v3h @@ -41,15 +41,15 @@ ebb3(v4: i64): function %loop(i64, i64) -> i64 { ebb0(v1: i64, v2: i64): -; check: $ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): +; check: ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32): jump ebb1(v1) - ; check: jump $ebb1($v1l, $v1h) + ; check: jump ebb1($v1l, $v1h) ebb1(v3: i64): -; check: $ebb1($(v3l=$V): i32, $(v3h=$V): i32): +; check: ebb1($(v3l=$V): i32, $(v3h=$V): i32): v4 = band v3, v2 ; check: $(v4l=$V) = band $v3l, $v2l ; check: $(v4h=$V) = band $v3h, $v2h jump ebb1(v4) - ; check: jump $ebb1($v4l, $v4h) + ; check: jump ebb1($v4l, $v4h) } diff --git a/cranelift/filetests/parser/branch.cton b/cranelift/filetests/parser/branch.cton index 7c653ed047..868b92d024 100644 --- a/cranelift/filetests/parser/branch.cton +++ b/cranelift/filetests/parser/branch.cton @@ -26,11 +26,11 @@ ebb1(v91: i32): jump ebb0(v91) } ; sameln: function %onearg(i32) native { -; nextln: ebb0($v90: i32): -; nextln: jump ebb1($v90) +; nextln: ebb0(v90: i32): +; nextln: jump ebb1(v90) ; nextln: -; nextln: ebb1($v91: i32): -; nextln: jump ebb0($v91) +; nextln: ebb1(v91: i32): +; nextln: jump ebb0(v91) ; nextln: } ; Jumps with 2 args. @@ -42,11 +42,11 @@ ebb1(v92: i32, v93: f32): jump ebb0(v92, v93) } ; sameln: function %twoargs(i32, f32) native { -; nextln: ebb0($v90: i32, $v91: f32): -; nextln: jump ebb1($v90, $v91) +; nextln: ebb0(v90: i32, v91: f32): +; nextln: jump ebb1(v90, v91) ; nextln: -; nextln: ebb1($v92: i32, $v93: f32): -; nextln: jump ebb0($v92, $v93) +; nextln: ebb1(v92: i32, v93: f32): +; nextln: jump ebb0(v92, v93) ; nextln: } ; Branches with no arguments. The '()' empty argument list is optional. @@ -58,11 +58,11 @@ ebb1: brnz v90, ebb1() } ; sameln: function %minimal(i32) native { -; nextln: ebb0($v90: i32): -; nextln: brz $v90, ebb1 +; nextln: ebb0(v90: i32): +; nextln: brz v90, ebb1 ; nextln: ; nextln: ebb1: -; nextln: brnz.i32 $v90, ebb1 +; nextln: brnz.i32 v90, ebb1 ; nextln: } function %twoargs(i32, f32) { @@ -73,11 +73,11 @@ ebb1(v92: i32, v93: f32): brnz v90, ebb0(v92, v93) } ; sameln: function %twoargs(i32, f32) native { -; nextln: ebb0($v90: i32, $v91: f32): -; nextln: brz $v90, ebb1($v90, $v91) +; nextln: ebb0(v90: i32, v91: f32): +; nextln: brz v90, ebb1(v90, v91) ; nextln: -; nextln: ebb1($v92: i32, $v93: f32): -; nextln: brnz.i32 $v90, ebb0($v92, $v93) +; nextln: ebb1(v92: i32, v93: f32): +; nextln: brnz.i32 v90, ebb0(v92, v93) ; nextln: } function %jumptable(i32) { @@ -97,8 +97,8 @@ ebb40: ; sameln: function %jumptable(i32) native { ; check: jt2 = jump_table 0, 0, ebb10, ebb40, ebb20, ebb30 ; check: jt200 = jump_table 0 -; check: ebb10($v3: i32): -; nextln: br_table $v3, jt2 +; check: ebb10(v3: i32): +; nextln: br_table v3, jt2 ; nextln: trap user1 ; nextln: ; nextln: ebb20: diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton index eab0799458..b6972817ca 100644 --- a/cranelift/filetests/parser/call.cton +++ b/cranelift/filetests/parser/call.cton @@ -18,9 +18,9 @@ ebb1: } ; sameln: function %r1() -> i32, f32 spiderwasm { ; nextln: ebb1: -; nextln: $v1 = iconst.i32 3 -; nextln: $v2 = f32const 0.0 -; nextln: return $v1, $v2 +; nextln: v1 = iconst.i32 3 +; nextln: v2 = f32const 0.0 +; nextln: return v1, v2 ; nextln: } function %signatures() { @@ -30,11 +30,11 @@ function %signatures() { fn8 = function %bar(i32) -> b1 } ; sameln: function %signatures() native { -; check: $sig10 = () native -; check: $sig11 = (i32, f64) -> i32, b1 spiderwasm +; check: sig10 = () native +; check: sig11 = (i32, f64) -> i32, b1 spiderwasm ; check: sig12 = (i32) -> b1 native -; check: $fn5 = $sig11 %foo -; check: $fn8 = sig12 %bar +; check: fn5 = sig11 %foo +; check: fn8 = sig12 %bar ; check: } function %direct() { @@ -48,9 +48,9 @@ ebb0: v2, v3 = call fn2() return } -; check: call $fn0() -; check: $v1 = call $fn1() -; check: $v2, $v3 = call $fn2() +; check: call fn0() +; check: v1 = call fn1() +; check: v2, v3 = call fn2() ; check: return function %indirect(i64) { @@ -64,9 +64,9 @@ ebb0(v0: i64): v3, v4 = call_indirect sig2, v1() return } -; check: $v1 = call_indirect $sig1, $v0() -; check: call_indirect $sig0, $v1($v0) -; check: $v3, $v4 = call_indirect $sig2, $v1() +; check: v1 = call_indirect sig1, v0() +; check: call_indirect sig0, v1(v0) +; check: v3, v4 = call_indirect sig2, v1() ; check: return function %long_call() { @@ -78,8 +78,8 @@ ebb0: call_indirect sig0, v0() return } -; check: $v0 = func_addr.i32 $fn0 -; check: call_indirect $sig0, $v0() +; check: v0 = func_addr.i32 fn0 +; check: call_indirect sig0, v0() ; check: return ; Special purpose function arguments @@ -88,6 +88,6 @@ ebb0(v1: i32, v2: i32, v3: i32, v4: i32): return v4, v2, v3, v1 } ; check: function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret native { -; check: ebb0($v1: i32, $v2: i32, $v3: i32, $v4: i32): -; check: return $v4, $v2, $v3, $v1 +; check: ebb0(v1: i32, v2: i32, v3: i32, v4: i32): +; check: return v4, v2, v3, v1 ; check: } diff --git a/cranelift/filetests/parser/flags.cton b/cranelift/filetests/parser/flags.cton index c7b42f3ac2..d3c096f1c5 100644 --- a/cranelift/filetests/parser/flags.cton +++ b/cranelift/filetests/parser/flags.cton @@ -18,11 +18,11 @@ ebb201: ebb202: trap oob } -; check: $v1 = ifcmp_imm $v0, 17 -; check: brif eq $v1, $ebb201 -; check: brif ugt $v1, $ebb202 -; check: $v3 = ifcmp $v0, $v2 -; check: $v4 = trueif eq $v3 +; check: v1 = ifcmp_imm v0, 17 +; check: brif eq v1, ebb201 +; check: brif ugt v1, ebb202 +; check: v3 = ifcmp v0, v2 +; check: v4 = trueif eq v3 function %fflags(f32) { ebb200(v0: f32): @@ -40,7 +40,7 @@ ebb201: ebb202: trap oob } -; check: $v2 = ffcmp $v0, $v1 -; check: brff eq $v2, $ebb201 -; check: brff ord $v2, $ebb202 -; check: $v3 = trueff gt $v2 +; check: v2 = ffcmp v0, v1 +; check: brff eq v2, ebb201 +; check: brff ord v2, ebb202 +; check: v3 = trueff gt v2 diff --git a/cranelift/filetests/parser/instruction_encoding.cton b/cranelift/filetests/parser/instruction_encoding.cton index 6bc7dcd6ce..a16c24a364 100644 --- a/cranelift/filetests/parser/instruction_encoding.cton +++ b/cranelift/filetests/parser/instruction_encoding.cton @@ -14,11 +14,11 @@ ebb1(v0: i32 [%x8], v1: i32): @a5 [Iret#5] return v0, v8 } ; sameln: function %foo(i32, i32) native { -; nextln: $ebb1($v0: i32 [%x8], $v1: i32): -; nextln: [-,-]$WS $v2 = iadd $v0, $v1 +; nextln: ebb1(v0: i32 [%x8], v1: i32): +; nextln: [-,-]$WS v2 = iadd v0, v1 ; nextln: [-]$WS trap heap_oob -; nextln: [R#1234,%x5,%x11]$WS $v6, $v7 = iadd_cout $v2, $v0 -; nextln: [Rshamt#beef,%x25]$WS $v8 = ishl_imm $v6, 2 -; nextln: @0055 [-,-]$WS $v9 = iadd $v8, $v7 -; nextln: @00a5 [Iret#05]$WS return $v0, $v8 +; nextln: [R#1234,%x5,%x11]$WS v6, v7 = iadd_cout v2, v0 +; nextln: [Rshamt#beef,%x25]$WS v8 = ishl_imm v6, 2 +; nextln: @0055 [-,-]$WS v9 = iadd v8, v7 +; nextln: @00a5 [Iret#05]$WS return v0, v8 ; nextln: } diff --git a/cranelift/filetests/parser/memory.cton b/cranelift/filetests/parser/memory.cton index 0b137335c9..1b36071243 100644 --- a/cranelift/filetests/parser/memory.cton +++ b/cranelift/filetests/parser/memory.cton @@ -3,34 +3,34 @@ test verifier function %vmglobal() -> i32 { gv3 = vmctx+16 - ; check: $gv3 = vmctx+16 + ; check: gv3 = vmctx+16 gv4 = vmctx+0 - ; check: $gv4 = vmctx + ; check: gv4 = vmctx ; not: +0 gv5 = vmctx -256 - ; check: $gv5 = vmctx-256 + ; check: gv5 = vmctx-256 ebb0: v1 = global_addr.i32 gv3 - ; check: $v1 = global_addr.i32 $gv3 + ; check: v1 = global_addr.i32 gv3 return v1 } function %deref() -> i32 { gv3 = vmctx+16 gv4 = deref(gv3)-32 - ; check: $gv4 = deref($gv3)-32 + ; check: gv4 = deref(gv3)-32 ebb0: v1 = global_addr.i32 gv4 - ; check: $v1 = global_addr.i32 $gv4 + ; check: v1 = global_addr.i32 gv4 return v1 } ; Refer to a global variable before it's been declared. function %backref() -> i32 { gv1 = deref(gv2)-32 - ; check: $gv1 = deref($gv2)-32 + ; check: gv1 = deref(gv2)-32 gv2 = vmctx+16 - ; check: $gv2 = vmctx+16 + ; check: gv2 = vmctx+16 ebb0: v1 = global_addr.i32 gv1 return v1 @@ -38,14 +38,14 @@ ebb0: function %sym() -> i32 { gv0 = globalsym %something - ; check: $gv0 = globalsym %something + ; check: gv0 = globalsym %something gv1 = globalsym u8:9 - ; check: $gv1 = globalsym u8:9 + ; check: gv1 = globalsym u8:9 ebb0: v0 = global_addr.i32 gv0 - ; check: $v0 = global_addr.i32 $gv0 + ; check: v0 = global_addr.i32 gv0 v1 = global_addr.i32 gv1 - ; check: $v1 = global_addr.i32 $gv1 + ; check: v1 = global_addr.i32 gv1 v2 = bxor v0, v1 return v2 } @@ -56,11 +56,11 @@ function %sheap(i32) -> i64 { heap2 = static gv5, guard 0x1000, bound 0x1_0000 gv5 = vmctx+64 - ; check: $heap1 = static reserved_reg, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 - ; check: $heap2 = static $gv5, min 0, bound 0x0001_0000, guard 4096 + ; check: heap1 = static reserved_reg, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 + ; check: heap2 = static gv5, min 0, bound 0x0001_0000, guard 4096 ebb0(v1: i32): v2 = heap_addr.i64 heap1, v1, 0 - ; check: $v2 = heap_addr.i64 $heap1, $v1, 0 + ; check: v2 = heap_addr.i64 heap1, v1, 0 return v2 } @@ -71,10 +71,10 @@ function %dheap(i32) -> i64 { gv5 = vmctx+64 gv6 = vmctx+72 - ; check: $heap1 = dynamic reserved_reg, min 0x0001_0000, bound $gv6, guard 0x8000_0000 - ; check: $heap2 = dynamic $gv5, min 0, bound $gv6, guard 4096 + ; check: heap1 = dynamic reserved_reg, min 0x0001_0000, bound gv6, guard 0x8000_0000 + ; check: heap2 = dynamic gv5, min 0, bound gv6, guard 4096 ebb0(v1: i32): v2 = heap_addr.i64 heap2, v1, 0 - ; check: $v2 = heap_addr.i64 $heap2, $v1, 0 + ; check: v2 = heap_addr.i64 heap2, v1, 0 return v2 } diff --git a/cranelift/filetests/parser/rewrite.cton b/cranelift/filetests/parser/rewrite.cton index 33d23baef4..a62157082a 100644 --- a/cranelift/filetests/parser/rewrite.cton +++ b/cranelift/filetests/parser/rewrite.cton @@ -1,14 +1,8 @@ -; The .cton parser can't preserve the actual entity numbers in the input file -; since entities are numbered as they are created. For entities declared in the -; preamble, this is no problem, but for EBB and value references, mapping -; source numbers to real numbers can be a problem. -; ; It is possible to refer to instructions and EBBs that have not yet been -; defined in the lexical order, so the parser needs to rewrite these references -; after the fact. +; defined in the lexical order. test cat -; Check that defining numbers are rewritten. +; Defining numbers. function %defs() { ebb100(v20: i32): v1000 = iconst.i32x8 5 @@ -16,9 +10,9 @@ ebb100(v20: i32): trap user4 } ; sameln: function %defs() native { -; nextln: $ebb100($v20: i32): -; nextln: $v1000 = iconst.i32x8 5 -; nextln: $v9200 = f64const 0x1.0000000000000p2 +; nextln: ebb100(v20: i32): +; nextln: v1000 = iconst.i32x8 5 +; nextln: v9200 = f64const 0x1.0000000000000p2 ; nextln: trap user4 ; nextln: } @@ -30,8 +24,8 @@ ebb100(v20: i32): jump ebb100(v1000) } ; sameln: function %use_value() native { -; nextln: ebb100($v20: i32): -; nextln: $v1000 = iadd_imm $v20, 5 -; nextln: $v200 = iadd $v20, $v1000 -; nextln: jump ebb100($v1000) +; nextln: ebb100(v20: i32): +; nextln: v1000 = iadd_imm v20, 5 +; nextln: v200 = iadd v20, v1000 +; nextln: jump ebb100(v1000) ; nextln: } diff --git a/cranelift/filetests/parser/ternary.cton b/cranelift/filetests/parser/ternary.cton index 3f320a61f0..75f09bb223 100644 --- a/cranelift/filetests/parser/ternary.cton +++ b/cranelift/filetests/parser/ternary.cton @@ -4,21 +4,21 @@ test verifier function %add_i96(i32, i32, i32, i32, i32, i32) -> i32, i32, i32 { ebb1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32): v10, v11 = iadd_cout v1, v4 - ;check: $v10, $v11 = iadd_cout $v1, $v4 + ;check: v10, v11 = iadd_cout v1, v4 v20, v21 = iadd_carry v2, v5, v11 - ; check: $v20, $v21 = iadd_carry $v2, $v5, $v11 + ; check: v20, v21 = iadd_carry v2, v5, v11 v30 = iadd_cin v3, v6, v21 - ; check: $v30 = iadd_cin $v3, $v6, $v21 + ; check: v30 = iadd_cin v3, v6, v21 return v10, v20, v30 } function %sub_i96(i32, i32, i32, i32, i32, i32) -> i32, i32, i32 { ebb1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32): v10, v11 = isub_bout v1, v4 - ;check: $v10, $v11 = isub_bout $v1, $v4 + ;check: v10, v11 = isub_bout v1, v4 v20, v21 = isub_borrow v2, v5, v11 - ; check: $v20, $v21 = isub_borrow $v2, $v5, $v11 + ; check: v20, v21 = isub_borrow v2, v5, v11 v30 = isub_bin v3, v6, v21 - ; check: $v30 = isub_bin $v3, $v6, $v21 + ; check: v30 = isub_bin v3, v6, v21 return v10, v20, v30 } diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 1355d0a21d..a5a6489808 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -20,9 +20,9 @@ ebb0: } ; sameln: function %ivalues() native { ; nextln: ebb0: -; nextln: $v0 = iconst.i32 2 -; nextln: $v1 = iconst.i8 6 -; nextln: $v2 = ishl $v0, $v1 +; nextln: v0 = iconst.i32 2 +; nextln: v1 = iconst.i8 6 +; nextln: v2 = ishl v0, v1 ; nextln: } ; Create and use values. @@ -36,10 +36,10 @@ ebb0: } ; sameln: function %bvalues() native { ; nextln: ebb0: -; nextln: $v0 = bconst.b32 true -; nextln: $v1 = bconst.b8 false -; nextln: $v2 = bextend.b32 v1 -; nextln: $v3 = bxor v0, v2 +; nextln: v0 = bconst.b32 true +; nextln: v1 = bconst.b8 false +; nextln: v2 = bextend.b32 v1 +; nextln: v3 = bxor v0, v2 ; nextln: } ; Polymorphic instruction controlled by second operand. @@ -48,8 +48,8 @@ ebb0(v90: i32, v91: i32, v92: b1): v0 = select v92, v90, v91 } ; sameln: function %select() native { -; nextln: ebb0($v90: i32, $v91: i32, $v92: b1): -; nextln: $v0 = select $v92, $v90, $v91 +; nextln: ebb0(v90: i32, v91: i32, v92: b1): +; nextln: v0 = select v92, v90, v91 ; nextln: } ; Polymorphic instruction controlled by third operand. @@ -71,9 +71,9 @@ ebb0: } ; sameln: function %lanes() native { ; nextln: ebb0: -; nextln: $v0 = iconst.i32x4 2 -; nextln: $v1 = extractlane $v0, 3 -; nextln: $v2 = insertlane $v0, 1, $v1 +; nextln: v0 = iconst.i32x4 2 +; nextln: v1 = extractlane v0, 3 +; nextln: v2 = insertlane v0, 1, v1 ; nextln: } ; Integer condition codes. @@ -86,12 +86,12 @@ ebb0(v90: i32, v91: i32): br_icmp eq v90, v91, ebb0(v91, v90) } ; sameln: function %icmp(i32, i32) native { -; nextln: ebb0($v90: i32, $v91: i32): -; nextln: $v0 = icmp eq $v90, $v91 -; nextln: $v1 = icmp ult $v90, $v91 -; nextln: $v2 = icmp_imm sge $v90, -12 -; nextln: $v3 = irsub_imm $v91, 45 -; nextln: br_icmp eq $v90, $v91, ebb0($v91, $v90) +; nextln: ebb0(v90: i32, v91: i32): +; nextln: v0 = icmp eq v90, v91 +; nextln: v1 = icmp ult v90, v91 +; nextln: v2 = icmp_imm sge v90, -12 +; nextln: v3 = irsub_imm v91, 45 +; nextln: br_icmp eq v90, v91, ebb0(v91, v90) ; nextln: } ; Floating condition codes. @@ -102,10 +102,10 @@ ebb0(v90: f32, v91: f32): v2 = fcmp lt v90, v91 } ; sameln: function %fcmp(f32, f32) native { -; nextln: ebb0($v90: f32, $v91: f32): -; nextln: $v0 = fcmp eq $v90, $v91 -; nextln: $v1 = fcmp uno $v90, $v91 -; nextln: $v2 = fcmp lt $v90, $v91 +; nextln: ebb0(v90: f32, v91: f32): +; nextln: v0 = fcmp eq v90, v91 +; nextln: v1 = fcmp uno v90, v91 +; nextln: v2 = fcmp lt v90, v91 ; nextln: } ; The bitcast instruction has two type variables: The controlling type variable @@ -116,9 +116,9 @@ ebb0(v90: i32, v91: f32): v1 = bitcast.i32 v91 } ; sameln: function %bitcast(i32, f32) native { -; nextln: ebb0($v90: i32, $v91: f32): -; nextln: $v0 = bitcast.i8x4 $v90 -; nextln: $v1 = bitcast.i32 $v91 +; nextln: ebb0(v90: i32, v91: f32): +; nextln: v0 = bitcast.i8x4 v90 +; nextln: v1 = bitcast.i32 v91 ; nextln: } ; Stack slot references @@ -136,17 +136,17 @@ ebb0: stack_store v2, ss2 } ; sameln: function %stack() native { -; check: $ss2 = local 4 -; check: $ss3 = incoming_arg 4, offset 8 -; check: $ss4 = outgoing_arg 4 -; check: $ss5 = emergency_slot 4 -; check: $ss10 = spill_slot 8 +; check: ss2 = local 4 +; check: ss3 = incoming_arg 4, offset 8 +; check: ss4 = outgoing_arg 4 +; check: ss5 = emergency_slot 4 +; check: ss10 = spill_slot 8 ; check: ebb0: -; nextln: $v1 = stack_load.i32 $ss10 -; nextln: $v2 = stack_load.i32 $ss10+4 -; nextln: stack_store $v1, $ss10+2 -; nextln: stack_store $v2, $ss2 +; nextln: v1 = stack_load.i32 ss10 +; nextln: v2 = stack_load.i32 ss10+4 +; nextln: stack_store v1, ss10+2 +; nextln: stack_store v2, ss2 ; Memory access instructions. function %memory(i32) { @@ -163,17 +163,17 @@ ebb0(v1: i32): store notrap aligned v3, v1-12 } ; sameln: function %memory(i32) native { -; nextln: ebb0($v1: i32): -; nextln: $v2 = load.i64 $v1 -; nextln: $v3 = load.i64 aligned $v1 -; nextln: $v4 = load.i64 notrap $v1 -; nextln: $v5 = load.i64 notrap aligned $v1 -; nextln: $v6 = load.i64 notrap aligned $v1 -; nextln: $v7 = load.i64 $v1-12 -; nextln: $v8 = load.i64 notrap $v1+0x0001_0000 -; nextln: store $v2, $v1 -; nextln: store aligned $v3, $v1+12 -; nextln: store notrap aligned $v3, $v1-12 +; nextln: ebb0(v1: i32): +; nextln: v2 = load.i64 v1 +; nextln: v3 = load.i64 aligned v1 +; nextln: v4 = load.i64 notrap v1 +; nextln: v5 = load.i64 notrap aligned v1 +; nextln: v6 = load.i64 notrap aligned v1 +; nextln: v7 = load.i64 v1-12 +; nextln: v8 = load.i64 notrap v1+0x0001_0000 +; nextln: store v2, v1 +; nextln: store aligned v3, v1+12 +; nextln: store notrap aligned v3, v1-12 ; Register diversions. ; This test file has no ISA, so we can unly use register unit numbers. @@ -188,12 +188,12 @@ ebb0(v1: i32): return } ; sameln: function %diversion(i32) native { -; nextln: $ss0 = spill_slot 4 -; check: ebb0($v1: i32): -; nextln: regmove $v1, %10 -> %20 -; nextln: regmove $v1, %20 -> %10 -; nextln: regspill $v1, %10 -> $ss0 -; nextln: regfill $v1, $ss0 -> %10 +; nextln: ss0 = spill_slot 4 +; check: ebb0(v1: i32): +; nextln: regmove v1, %10 -> %20 +; nextln: regmove v1, %20 -> %10 +; nextln: regspill v1, %10 -> ss0 +; nextln: regfill v1, ss0 -> %10 ; nextln: return ; nextln: } @@ -222,12 +222,12 @@ ebb0(v0: i32): return } ; sameln: function %cond_traps(i32) -; nextln: ebb0($v0: i32): -; nextln: trapz $v0, stk_ovf -; nextln: $v1 = ifcmp_imm v0, 5 -; nextln: trapif ugt $v1, oob -; nextln: $v2 = bitcast.f32 $v1 -; nextln: $v3 = ffcmp $v2, $v2 -; nextln: trapff uno $v3, int_ovf +; nextln: ebb0(v0: i32): +; nextln: trapz v0, stk_ovf +; nextln: v1 = ifcmp_imm v0, 5 +; nextln: trapif ugt v1, oob +; nextln: v2 = bitcast.f32 v1 +; nextln: v3 = ffcmp v2, v2 +; nextln: trapff uno v3, int_ovf ; nextln: return ; nextln: } diff --git a/cranelift/filetests/regalloc/basic.cton b/cranelift/filetests/regalloc/basic.cton index 89e160d75a..b9399f464d 100644 --- a/cranelift/filetests/regalloc/basic.cton +++ b/cranelift/filetests/regalloc/basic.cton @@ -17,7 +17,7 @@ ebb0(v1: i32, v2: i32): function %dead_arg(i32, i32) -> i32{ ebb0(v1: i32, v2: i32): ; not: regmove -; check: return $v1 +; check: return v1 return v1 } @@ -25,8 +25,8 @@ ebb0(v1: i32, v2: i32): function %move1(i32, i32) -> i32 { ebb0(v1: i32, v2: i32): ; not: regmove -; check: regmove $v2, %x11 -> %x10 -; nextln: return $v2 +; check: regmove v2, %x11 -> %x10 +; nextln: return v2 return v2 } @@ -34,10 +34,10 @@ ebb0(v1: i32, v2: i32): function %swap(i32, i32) -> i32, i32 { ebb0(v1: i32, v2: i32): ; not: regmove -; check: regmove $v2, %x11 -> $(tmp=$RX) -; nextln: regmove $v1, %x10 -> %x11 -; nextln: regmove $v2, $tmp -> %x10 -; nextln: return $v2, $v1 +; check: regmove v2, %x11 -> $(tmp=$RX) +; nextln: regmove v1, %x10 -> %x11 +; nextln: regmove v2, $tmp -> %x10 +; nextln: return v2, v1 return v2, v1 } diff --git a/cranelift/filetests/regalloc/coalesce.cton b/cranelift/filetests/regalloc/coalesce.cton index 0deac6fc85..b2700b01dd 100644 --- a/cranelift/filetests/regalloc/coalesce.cton +++ b/cranelift/filetests/regalloc/coalesce.cton @@ -23,8 +23,8 @@ ebb1(v10: i32): function %trivial(i32) -> i32 { ebb0(v0: i32): - ; check: $(cp1=$V) = copy $v0 - ; nextln: brnz $v0, $ebb1($cp1) + ; check: $(cp1=$V) = copy v0 + ; nextln: brnz v0, ebb1($cp1) brnz v0, ebb1(v0) ; not: copy v1 = iadd_imm v0, 7 @@ -39,8 +39,8 @@ ebb1(v10: i32): ; A value is used as an SSA argument twice in the same branch. function %dualuse(i32) -> i32 { ebb0(v0: i32): - ; check: $(cp1=$V) = copy $v0 - ; nextln: brnz $v0, $ebb1($cp1, $v0) + ; check: $(cp1=$V) = copy v0 + ; nextln: brnz v0, ebb1($cp1, v0) brnz v0, ebb1(v0, v0) v1 = iadd_imm v0, 7 v2 = iadd_imm v1, 56 @@ -55,15 +55,15 @@ ebb1(v10: i32, v11: i32): ; The interference can be broken with a copy at either branch. function %interference(i32) -> i32 { ebb0(v0: i32): - ; check: $(cp0=$V) = copy $v0 + ; check: $(cp0=$V) = copy v0 ; not: copy - ; check: brnz $v0, ebb1($cp0) + ; check: brnz v0, ebb1($cp0) brnz v0, ebb1(v0) v1 = iadd_imm v0, 7 ; v1 and v0 interfere here: v2 = iadd_imm v0, 8 ; not: copy - ; check: jump $ebb1($v1) + ; check: jump ebb1(v1) jump ebb1(v1) ebb1(v10: i32): @@ -81,13 +81,13 @@ ebb0(v0: i32): ebb1(v10: i32, v11: i32): ; v11 needs to be isolated because it interferes with v10. - ; check: $ebb1($v10: i32 [$LOC], $(nv11a=$V): i32 [$LOC]) - ; check: $v11 = copy $nv11a + ; check: ebb1(v10: i32 [$LOC], $(nv11a=$V): i32 [$LOC]) + ; check: v11 = copy $nv11a v12 = iadd v10, v11 v13 = icmp ult v12, v0 - ; check: $(nv11b=$V) = copy $v11 + ; check: $(nv11b=$V) = copy v11 ; not: copy - ; check: brnz $v13, $ebb1($nv11b, $v12) + ; check: brnz v13, ebb1($nv11b, v12) brnz v13, ebb1(v11, v12) return v12 } diff --git a/cranelift/filetests/regalloc/constraints.cton b/cranelift/filetests/regalloc/constraints.cton index 720a886263..bda37df149 100644 --- a/cranelift/filetests/regalloc/constraints.cton +++ b/cranelift/filetests/regalloc/constraints.cton @@ -20,10 +20,10 @@ function %tied_alive() -> i32 { ebb0: v0 = iconst.i32 12 v1 = iconst.i32 13 - ; check: $(v0c=$V) = copy $v0 - ; check: $v2 = isub $v0c, $v1 + ; check: $(v0c=$V) = copy v0 + ; check: v2 = isub $v0c, v1 v2 = isub v0, v1 - ; check: $v3 = iadd $v2, $v0 + ; check: v3 = iadd v2, v0 v3 = iadd v2, v0 return v3 } @@ -32,11 +32,11 @@ ebb0: function %fixed_op() -> i32 { ebb0: ; check: ,%rax] - ; sameln: $v0 = iconst.i32 12 + ; sameln: v0 = iconst.i32 12 v0 = iconst.i32 12 v1 = iconst.i32 13 ; The dynamic shift amount must be in %rcx - ; check: regmove $v0, %rax -> %rcx + ; check: regmove v0, %rax -> %rcx v2 = ishl v1, v0 return v2 } @@ -45,14 +45,14 @@ ebb0: function %fixed_op_twice() -> i32 { ebb0: ; check: ,%rax] - ; sameln: $v0 = iconst.i32 12 + ; sameln: v0 = iconst.i32 12 v0 = iconst.i32 12 v1 = iconst.i32 13 ; The dynamic shift amount must be in %rcx - ; check: regmove $v0, %rax -> %rcx + ; check: regmove v0, %rax -> %rcx v2 = ishl v1, v0 - ; check: regmove $v0, %rcx -> $REG - ; check: regmove $v2, $REG -> %rcx + ; check: regmove v0, %rcx -> $REG + ; check: regmove v2, $REG -> %rcx v3 = ishl v0, v2 return v3 @@ -62,12 +62,12 @@ ebb0: function %fixed_op_twice() -> i32 { ebb0: ; check: ,%rax] - ; sameln: $v0 = iconst.i32 12 + ; sameln: v0 = iconst.i32 12 v0 = iconst.i32 12 v1 = iconst.i32 13 ; The dynamic shift amount must be in %rcx - ; check: regmove $v0, %rax -> %rcx - ; check: $v2 = ishl $v1, $v0 + ; check: regmove v0, %rax -> %rcx + ; check: v2 = ishl v1, v0 v2 = ishl v1, v0 ; Now v0 is globally allocated to %rax, but diverted to %rcx. @@ -77,6 +77,6 @@ ebb0: ; check: ,%rcx] ; sameln: isub ; Move it into place for the return value. - ; check: regmove $v3, %rcx -> %rax + ; check: regmove v3, %rcx -> %rax return v3 } diff --git a/cranelift/filetests/regalloc/intel-regres.cton b/cranelift/filetests/regalloc/intel-regres.cton index 0de1b620c5..50384d6349 100644 --- a/cranelift/filetests/regalloc/intel-regres.cton +++ b/cranelift/filetests/regalloc/intel-regres.cton @@ -19,12 +19,12 @@ ebb0(v0: i32): jump ebb2(v3, v2, v0) ebb2(v4: i32, v5: i32, v7: i32): - ; check: $ebb2 + ; check: ebb2 v6 = iadd v4, v5 v8 = iconst.i32 -1 ; v7 is killed here and v9 gets the same register. v9 = iadd v7, v8 - ; check: $v9 = iadd $v7, $v8 + ; check: v9 = iadd v7, v8 ; Here v9 the brnz control appears to interfere with v9 the EBB argument, ; so divert_fixed_input_conflicts() calls add_var(v9), which is ok. The ; add_var sanity checks got confused when no fixed assignment could be @@ -32,7 +32,7 @@ ebb2(v4: i32, v5: i32, v7: i32): ; ; We should be able to handle this situation without making copies of v9. brnz v9, ebb2(v5, v6, v9) - ; check: brnz $v9, $ebb2($V, $V, $v9) + ; check: brnz v9, ebb2($V, $V, v9) jump ebb3 ebb3: diff --git a/cranelift/filetests/regalloc/multi-constraints.cton b/cranelift/filetests/regalloc/multi-constraints.cton index ecb3aa8bad..afbbf6fc2b 100644 --- a/cranelift/filetests/regalloc/multi-constraints.cton +++ b/cranelift/filetests/regalloc/multi-constraints.cton @@ -39,9 +39,9 @@ ebb1: ; flag so it can be reassigned to a different global register. function %pr218(i64 [%rdi], i64 [%rsi], i64 [%rdx], i64 [%rcx]) -> i64 [%rax] { ebb0(v0: i64, v1: i64, v2: i64, v3: i64): - ; check: regmove $v3, %rcx -> + ; check: regmove v3, %rcx -> v4 = ushr v0, v0 - ; check: $v4 = copy + ; check: v4 = copy jump ebb1 ebb1: diff --git a/cranelift/filetests/regalloc/reload-208.cton b/cranelift/filetests/regalloc/reload-208.cton index 29c6e09840..1bf8120d57 100644 --- a/cranelift/filetests/regalloc/reload-208.cton +++ b/cranelift/filetests/regalloc/reload-208.cton @@ -52,8 +52,8 @@ ebb6: v25 = load.i64 v24 v8 = iadd v25, v23 v9 = load.i32 v8+56 - ; check: $v9 = spill - ; check: brnz $V, $ebb3($v9) + ; check: v9 = spill + ; check: brnz $V, ebb3(v9) brnz v9, ebb3(v9) jump ebb4 diff --git a/cranelift/filetests/regalloc/reload.cton b/cranelift/filetests/regalloc/reload.cton index 3db769eea8..68a6e8a149 100644 --- a/cranelift/filetests/regalloc/reload.cton +++ b/cranelift/filetests/regalloc/reload.cton @@ -9,11 +9,11 @@ function %spill_return() -> i32 { ebb0: v0 = call fn0() - ; check: $(reg=$V) = call $fn0 - ; check: $v0 = spill $reg + ; check: $(reg=$V) = call fn0 + ; check: v0 = spill $reg v2 = call fn0() - ; check: $v2 = call $fn0 + ; check: v2 = call fn0 return v0 - ; check: $(reload=$V) = fill $v0 + ; check: $(reload=$V) = fill v0 ; check: return $reload } diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index adafc89b43..8d18baa525 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -25,13 +25,13 @@ function %pyramid(i32) -> i32 { ; check: ss2 = spill_slot 4 ; not: spill_slot ebb0(v1: i32): -; check: $ebb0($(rv1=$V): i32 [%x10], $(rlink=$V): i32 [%x1]) - ; check: ,ss0]$WS $v1 = spill $rv1 +; check: ebb0($(rv1=$V): i32 [%x10], $(rlink=$V): i32 [%x1]) + ; check: ,ss0]$WS v1 = spill $rv1 ; nextln: ,ss1]$WS $(link=$V) = spill $rlink ; not: spill v2 = iadd_imm v1, 12 ; check: $(r1v2=$V) = iadd_imm - ; nextln: ,ss2]$WS $v2 = spill $r1v2 + ; nextln: ,ss2]$WS v2 = spill $r1v2 ; not: spill v3 = iadd_imm v2, 12 v4 = iadd_imm v3, 12 @@ -46,7 +46,7 @@ ebb0(v1: i32): v13 = iadd_imm v12, 12 v14 = iadd_imm v13, 12 v33 = iadd v13, v14 - ; check: iadd $v13 + ; check: iadd v13 v32 = iadd v33, v12 v31 = iadd v32, v11 v30 = iadd v31, v10 @@ -58,26 +58,26 @@ ebb0(v1: i32): v24 = iadd v25, v4 v23 = iadd v24, v3 v22 = iadd v23, v2 - ; check: $(r2v2=$V) = fill $v2 - ; check: $v22 = iadd $v23, $r2v2 + ; check: $(r2v2=$V) = fill v2 + ; check: v22 = iadd v23, $r2v2 v21 = iadd v22, v1 - ; check: $(r2v1=$V) = fill $v1 - ; check: $v21 = iadd $v22, $r2v1 + ; check: $(r2v1=$V) = fill v1 + ; check: v21 = iadd v22, $r2v1 ; check: $(rlink2=$V) = fill $link return v21 - ; check: return $v21, $rlink2 + ; check: return v21, $rlink2 } ; All values live across a call must be spilled function %across_call(i32) { fn0 = function %foo(i32) ebb0(v1: i32): - ; check: $v1 = spill + ; check: v1 = spill call fn0(v1) - ; check: call $fn0 + ; check: call fn0 call fn0(v1) - ; check: fill $v1 - ; check: call $fn0 + ; check: fill v1 + ; check: call fn0 return } @@ -85,9 +85,9 @@ ebb0(v1: i32): function %doubleuse(i32) { fn0 = function %xx(i32, i32) ebb0(v0: i32): - ; check: $(c=$V) = copy $v0 + ; check: $(c=$V) = copy v0 call fn0(v0, v0) - ; check: call $fn0($v0, $c) + ; check: call fn0(v0, $c) return } @@ -104,9 +104,9 @@ ebb0(v0: i32): function %doubleuse_icall2(i32) { sig0 = (i32, i32) native ebb0(v0: i32): - ; check: $(c=$V) = copy $v0 + ; check: $(c=$V) = copy v0 call_indirect sig0, v0(v0, v0) - ; check: call_indirect $sig0, $v0($v0, $c) + ; check: call_indirect sig0, v0(v0, $c) return } @@ -116,8 +116,8 @@ function %stackargs(i32, i32, i32, i32, i32, i32, i32, i32) -> i32 { ; check: ss1 = incoming_arg 4, offset 4 ; not: incoming_arg ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32, v7: i32): - ; unordered: fill $v6 - ; unordered: fill $v7 + ; unordered: fill v6 + ; unordered: fill v7 v10 = iadd v6, v7 return v10 } @@ -125,7 +125,7 @@ ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32, v7: i32): ; More EBB arguments than registers. function %ebbargs(i32) -> i32 { ebb0(v1: i32): - ; check: $v1 = spill + ; check: v1 = spill v2 = iconst.i32 1 jump ebb1(v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2) @@ -148,7 +148,7 @@ ebb1(v10: i32, v11: i32, v12: i32, v13: i32, v14: i32, v15: i32, v16: i32, v17: ; Spilling an EBB argument to make room for a branch operand. function %brargs(i32) -> i32 { ebb0(v1: i32): - ; check: $v1 = spill + ; check: v1 = spill v2 = iconst.i32 1 brnz v1, ebb1(v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2) return v1 @@ -179,13 +179,13 @@ function %use_spilled_value(i32) -> i32 { ; check: ss1 = spill_slot 4 ; check: ss2 = spill_slot 4 ebb0(v1: i32): -; check: $ebb0($(rv1=$V): i32 [%x10], $(rlink=$V): i32 [%x1]) - ; check: ,ss0]$WS $v1 = spill $rv1 +; check: ebb0($(rv1=$V): i32 [%x10], $(rlink=$V): i32 [%x1]) + ; check: ,ss0]$WS v1 = spill $rv1 ; nextln: ,ss1]$WS $(link=$V) = spill $rlink ; not: spill v2 = iadd_imm v1, 12 ; check: $(r1v2=$V) = iadd_imm - ; nextln: ,ss2]$WS $v2 = spill $r1v2 + ; nextln: ,ss2]$WS v2 = spill $r1v2 v3 = iadd_imm v2, 12 v4 = iadd_imm v3, 12 v5 = iadd_imm v4, 12 diff --git a/cranelift/filetests/simple_gvn/basic.cton b/cranelift/filetests/simple_gvn/basic.cton index 6ff45d1aef..df8fd495c9 100644 --- a/cranelift/filetests/simple_gvn/basic.cton +++ b/cranelift/filetests/simple_gvn/basic.cton @@ -5,7 +5,7 @@ ebb0(v0: i32, v1: i32): v2 = iadd v0, v1 v3 = iadd v0, v1 v4 = imul v2, v3 -; check: v4 = imul $v2, $v2 +; check: v4 = imul v2, v2 return v4 } @@ -16,7 +16,7 @@ ebb0(v0: i32, v1: i32): v4 = imul v2, v3 v5 = imul v2, v2 v6 = iadd v4, v5 -; check: v6 = iadd $v4, $v4 +; check: v6 = iadd v4, v4 return v6 } diff --git a/cranelift/src/filetest/subtest.rs b/cranelift/src/filetest/subtest.rs index 6c49b6e5a9..a9cf657d8e 100644 --- a/cranelift/src/filetest/subtest.rs +++ b/cranelift/src/filetest/subtest.rs @@ -6,7 +6,7 @@ use cretonne::ir::Function; use cretonne::isa::TargetIsa; use cretonne::settings::{Flags, FlagsOrIsa}; use cton_reader::{Details, Comment}; -use filecheck::{self, CheckerBuilder, Checker, Value as FCValue}; +use filecheck::{CheckerBuilder, Checker, NO_VARIABLES}; pub type Result = result::Result; @@ -67,34 +67,19 @@ pub trait SubTest { fn run(&self, func: Cow, context: &Context) -> Result<()>; } -/// Make the parser's source map available as filecheck variables. -/// -/// This means that the filecheck directives can refer to entities like `jump $ebb3`, where `$ebb3` -/// will expand to the EBB number that was assigned to `ebb3` in the input source. -/// -/// The expanded entity names are wrapped in word boundary regex guards so that 'inst1' doesn't -/// match 'inst10'. -impl<'a> filecheck::VariableMap for Context<'a> { - fn lookup(&self, varname: &str) -> Option { - self.details.map.lookup_str(varname).map(|e| { - FCValue::Regex(format!(r"\b{}\b", e).into()) - }) - } -} - /// Run filecheck on `text`, using directives extracted from `context`. pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { let checker = build_filechecker(context)?; - if checker.check(text, context).map_err( - |e| format!("filecheck: {}", e), - )? + if checker.check(text, NO_VARIABLES).map_err(|e| { + format!("filecheck: {}", e) + })? { Ok(()) } else { // Filecheck mismatch. Emit an explanation as output. - let (_, explain) = checker.explain(text, context).map_err( - |e| format!("explain: {}", e), - )?; + let (_, explain) = checker.explain(text, NO_VARIABLES).map_err(|e| { + format!("explain: {}", e) + })?; Err(format!("filecheck failed:\n{}{}", checker, explain)) } } From 69cc6c8af375ff8465b3038afeea682d315df6f5 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 20 Feb 2018 14:44:05 -0800 Subject: [PATCH 1513/3084] Clean up more comments referring to "source numbers". --- lib/reader/src/parser.rs | 2 -- lib/reader/src/sourcemap.rs | 16 ++++++++-------- 2 files changed, 8 insertions(+), 10 deletions(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index a0a17faa16..43c149956a 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1735,8 +1735,6 @@ impl<'a> Parser<'a> { // The controlling type variable can be specified explicitly as 'splat.i32x4 v5', or it can be // inferred from `inst_data.typevar_operand` for some opcodes. // - // The value operands in `inst_data` are expected to use source numbering. - // // Returns the controlling typevar for a polymorphic opcode, or `VOID` for a non-polymorphic // opcode. fn infer_typevar( diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index b3a2f0462f..704c097855 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -21,42 +21,42 @@ pub struct SourceMap { /// Read-only interface which is exposed outside the parser crate. impl SourceMap { - /// Look up a value entity by its source number. + /// Look up a value entity. pub fn contains_value(&self, v: Value) -> bool { self.locations.contains_key(&v.into()) } - /// Look up a EBB entity by its source number. + /// Look up a EBB entity. pub fn contains_ebb(&self, ebb: Ebb) -> bool { self.locations.contains_key(&ebb.into()) } - /// Look up a stack slot entity by its source number. + /// Look up a stack slot entity. pub fn contains_ss(&self, ss: StackSlot) -> bool { self.locations.contains_key(&ss.into()) } - /// Look up a global variable entity by its source number. + /// Look up a global variable entity. pub fn contains_gv(&self, gv: GlobalVar) -> bool { self.locations.contains_key(&gv.into()) } - /// Look up a heap entity by its source number. + /// Look up a heap entity. pub fn contains_heap(&self, heap: Heap) -> bool { self.locations.contains_key(&heap.into()) } - /// Look up a signature entity by its source number. + /// Look up a signature entity. pub fn contains_sig(&self, sig: SigRef) -> bool { self.locations.contains_key(&sig.into()) } - /// Look up a function entity by its source number. + /// Look up a function entity. pub fn contains_fn(&self, fn_: FuncRef) -> bool { self.locations.contains_key(&fn_.into()) } - /// Look up a jump table entity by its source number. + /// Look up a jump table entity. pub fn contains_jt(&self, jt: JumpTable) -> bool { self.locations.contains_key(&jt.into()) } From 8d388b2218bdeb78f429705993b1201d60989294 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 21 Feb 2018 10:34:41 -0800 Subject: [PATCH 1514/3084] Fix stack pointer offsets for outgoing arguments. StackSlotKind::OutgoingArg stack slots have an offset that is relative to our own stack pointer, while all other stack slot kinds have offsets that are relative to the caller's stack pointer. Make sure we generate the right sp-relative offsets for outgoing arguments too. --- cranelift/filetests/isa/intel/binary64.cton | 17 +++++++++++++++++ lib/cretonne/src/isa/stack.rs | 17 ++++++++++++----- 2 files changed, 29 insertions(+), 5 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 8ac8deb58e..0b546961d8 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -634,6 +634,23 @@ ebb1: return } +; Test for the encoding of outgoing_arg stack slots. +function %outargs() { + ss0 = incoming_arg 16, offset -16 + ss1 = outgoing_arg 8, offset 8 + ss2 = outgoing_arg 8, offset 0 + +ebb0: + [-,%rcx] v1 = iconst.i64 1 + + ; asm: movq %rcx, 8(%rsp) + [-,ss1] v10 = spill v1 ; bin: 48 89 8c 24 00000008 + ; asm: movq %rcx, (%rsp) + [-,ss2] v11 = spill v1 ; bin: 48 89 8c 24 00000000 + + return +} + ; Tests for i32 instructions in 64-bit mode. ; ; Note that many i32 instructions can be encoded both with and without a REX diff --git a/lib/cretonne/src/isa/stack.rs b/lib/cretonne/src/isa/stack.rs index 907afe0208..730db28e18 100644 --- a/lib/cretonne/src/isa/stack.rs +++ b/lib/cretonne/src/isa/stack.rs @@ -4,7 +4,7 @@ //! defined in this module expresses the low-level details of accessing a stack slot from an //! encoded instruction. -use ir::stackslot::{StackSlots, StackOffset}; +use ir::stackslot::{StackSlots, StackOffset, StackSlotKind}; use ir::StackSlot; /// A method for referencing a stack slot in the current stack frame. @@ -38,12 +38,19 @@ impl StackRef { let size = frame.frame_size.expect( "Stack layout must be computed before referencing stack slots", ); - - // Offset where SP is pointing. (All ISAs have stacks growing downwards.) - let sp_offset = -(size as StackOffset); + let slot = &frame[ss]; + let offset = if slot.kind == StackSlotKind::OutgoingArg { + // Outgoing argument slots have offsets relative to our stack pointer. + slot.offset + } else { + // All other slots have offsets relative to our caller's stack frame. + // Offset where SP is pointing. (All ISAs have stacks growing downwards.) + let sp_offset = -(size as StackOffset); + slot.offset - sp_offset + }; StackRef { base: StackBase::SP, - offset: frame[ss].offset - sp_offset, + offset, } } } From 2f58c371bc0d9beb53b81c0ba097dd62c5ef5771 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 21 Feb 2018 12:06:58 -0800 Subject: [PATCH 1515/3084] Make specific ISA sub-modules private. We don't want ISA-specific details exposed in the public Cretonne APIs. --- lib/cretonne/meta/gen_settings.py | 1 + lib/cretonne/src/isa/mod.rs | 8 ++++---- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index 693b7baa76..0cd4e0e860 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -86,6 +86,7 @@ def gen_getters(sgrp, fmt): Emit getter functions for all the settings in fmt. """ fmt.doc_comment("User-defined settings.") + fmt.line('#[allow(dead_code)]') with fmt.indented('impl Flags {', '}'): fmt.doc_comment('Get a view of the boolean predicates.') with fmt.indented( diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index f1ebf39a77..38164e5167 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -56,16 +56,16 @@ use isa::enc_tables::Encodings; use std::fmt; #[cfg(build_riscv)] -pub mod riscv; +mod riscv; #[cfg(build_intel)] -pub mod intel; +mod intel; #[cfg(build_arm32)] -pub mod arm32; +mod arm32; #[cfg(build_arm64)] -pub mod arm64; +mod arm64; pub mod registers; mod encoding; From eae8261b076c76df007e7b6ac8448ea4c4e2f1a5 Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 21 Feb 2018 12:11:50 -0800 Subject: [PATCH 1516/3084] Make the regalloc module private. Cretonne clients don't need to know how the register allocator works. Export the RegDiversions type from the binemit module instead. It is used by the "test binemit" driver. --- cranelift/src/filetest/binemit.rs | 2 +- lib/cretonne/src/binemit/mod.rs | 2 +- lib/cretonne/src/lib.rs | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 9b18a439a1..550703f0bf 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -10,7 +10,7 @@ use cretonne::binemit; use cretonne::dbg::DisplayList; use cretonne::ir; use cretonne::ir::entities::AnyEntity; -use cretonne::regalloc::RegDiversions; +use cretonne::binemit::RegDiversions; use cton_reader::TestCommand; use filetest::subtest::{SubTest, Context, Result}; use utils::{match_directive, pretty_error}; diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index fe6262cb70..713d25bed5 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -6,11 +6,11 @@ mod relaxation; mod memorysink; +pub use regalloc::RegDiversions; pub use self::relaxation::relax_branches; pub use self::memorysink::{MemoryCodeSink, RelocSink}; use ir::{ExternalName, JumpTable, Function, Inst}; -use regalloc::RegDiversions; use std::fmt; /// Offset in bytes from the beginning of the function. diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 9abffe0abc..f947f68621 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -25,7 +25,6 @@ pub mod ir; pub mod isa; pub mod loop_analysis; pub mod packed_option; -pub mod regalloc; pub mod result; pub mod settings; pub mod timing; @@ -40,6 +39,7 @@ mod licm; mod partition_slice; mod predicates; mod ref_slice; +mod regalloc; mod scoped_hash_map; mod simple_gvn; mod stack_layout; From 043d80f7e1855f3694f8f4bff172373e087df30e Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 21 Feb 2018 12:14:50 -0800 Subject: [PATCH 1517/3084] Make the bitset module private. This is just an implementation detail. --- lib/cretonne/src/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index f947f68621..b4c386477c 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -17,7 +17,6 @@ pub mod entity; pub mod bforest; pub mod binemit; -pub mod bitset; pub mod cursor; pub mod dominator_tree; pub mod flowgraph; @@ -31,6 +30,7 @@ pub mod timing; pub mod verifier; mod abi; +mod bitset; mod constant_hash; mod context; mod iterators; From 20500913adf7b80e3ea6df33b4506815c5ab86a4 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 21 Feb 2018 07:25:43 -0800 Subject: [PATCH 1518/3084] Change tabs to spaces, for consistency with rustfmt. This code is in a macro, which is presumably why rustfmt itself doesn't reformat it. --- lib/cretonne/src/entity/mod.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/src/entity/mod.rs b/lib/cretonne/src/entity/mod.rs index 837bb5f107..3cc8ec6f6d 100644 --- a/lib/cretonne/src/entity/mod.rs +++ b/lib/cretonne/src/entity/mod.rs @@ -90,8 +90,8 @@ macro_rules! entity_impl { impl ::std::fmt::Debug for $entity { fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result { - (self as &::std::fmt::Display).fmt(f) - } - } + (self as &::std::fmt::Display).fmt(f) + } + } } } From a55a582d4e432fd8cce0c5a0f257590bb27401ea Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 21 Feb 2018 13:10:58 -0800 Subject: [PATCH 1519/3084] Suppress printing of placeholder external functions. With the change to the parser to preserve indices, it now inserts placeholders to pad out index spaces as needed. Placeholder functions use reserved signature indices, so skip them when writing them out, to avoid writing them out as "sig4294967295". --- cranelift/filetests/parser/call.cton | 1 + lib/cretonne/src/write.rs | 8 ++++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton index b6972817ca..77543800d0 100644 --- a/cranelift/filetests/parser/call.cton +++ b/cranelift/filetests/parser/call.cton @@ -33,6 +33,7 @@ function %signatures() { ; check: sig10 = () native ; check: sig11 = (i32, f64) -> i32, b1 spiderwasm ; check: sig12 = (i32) -> b1 native +; not: fn0 ; check: fn5 = sig11 %foo ; check: fn8 = sig12 %bar ; check: } diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 860550f666..d9853d5768 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -4,10 +4,11 @@ //! equivalent textual representation. This textual representation can be read back by the //! `cretonne-reader` crate. -use ir::{Function, DataFlowGraph, Ebb, Inst, Value, ValueDef, Type}; +use ir::{Function, DataFlowGraph, Ebb, Inst, Value, ValueDef, Type, SigRef}; use isa::{TargetIsa, RegInfo}; use std::fmt::{self, Result, Error, Write}; use std::result; +use packed_option::ReservedValue; /// Write `func` to `w` as equivalent text. /// Use `isa` to emit ISA-dependent annotations. @@ -74,7 +75,10 @@ fn write_preamble( for fnref in func.dfg.ext_funcs.keys() { any = true; - writeln!(w, " {} = {}", fnref, func.dfg.ext_funcs[fnref])?; + let ext_func = &func.dfg.ext_funcs[fnref]; + if ext_func.signature != SigRef::reserved_value() { + writeln!(w, " {} = {}", fnref, ext_func)?; + } } for jt in func.jump_tables.keys() { From c3c31e5cdc694125a845b73e870b44243c47fefb Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 21 Feb 2018 15:14:03 -0800 Subject: [PATCH 1520/3084] Add an empty .rustfmt.toml. The existence of this file signals to some editors that this repository wishes to be formatted with rustfmt. --- cranelift/.rustfmt.toml | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 cranelift/.rustfmt.toml diff --git a/cranelift/.rustfmt.toml b/cranelift/.rustfmt.toml new file mode 100644 index 0000000000..e69de29bb2 From 8e6e976e6851aba55b61c95ef49f8c9171747349 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 21 Feb 2018 19:04:35 -0800 Subject: [PATCH 1521/3084] Add .swx to .gitignore to hide vim files from `cargo watch`. --- cranelift/.gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/cranelift/.gitignore b/cranelift/.gitignore index 14318728b6..a481ff48e8 100644 --- a/cranelift/.gitignore +++ b/cranelift/.gitignore @@ -2,6 +2,7 @@ *.bk *.swp *.swo +*.swx tags target Cargo.lock From c7655c4928324994973d780c8065ecb861f5ed83 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 21 Feb 2018 19:52:54 -0800 Subject: [PATCH 1522/3084] Add minimal README.md files to published crates. This will put descriptions on the packages' crates.io pages. --- lib/cretonne/Cargo.toml | 1 + lib/cretonne/README.md | 2 ++ lib/frontend/Cargo.toml | 1 + lib/frontend/README.md | 5 +++++ lib/frontend/src/lib.rs | 2 +- lib/native/Cargo.toml | 3 ++- lib/native/README.md | 3 +++ lib/reader/Cargo.toml | 1 + lib/reader/README.md | 3 +++ lib/wasm/Cargo.toml | 1 + lib/wasm/README.md | 3 +++ 11 files changed, 23 insertions(+), 2 deletions(-) create mode 100644 lib/cretonne/README.md create mode 100644 lib/frontend/README.md create mode 100644 lib/native/README.md create mode 100644 lib/reader/README.md create mode 100644 lib/wasm/README.md diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 60e8eb2be9..e85779913c 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -7,6 +7,7 @@ license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/stoklund/cretonne" publish = false +readme = "README.md" build = "build.rs" [lib] diff --git a/lib/cretonne/README.md b/lib/cretonne/README.md new file mode 100644 index 0000000000..73b2806f5d --- /dev/null +++ b/lib/cretonne/README.md @@ -0,0 +1,2 @@ +This crate contains the core Cretonne code generator. It translates code from an +intermediate language into executable machine code. diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 7ff44a1f6f..704517c176 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -7,6 +7,7 @@ license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/stoklund/cretonne" publish = false +readme = "README.md" [lib] name = "cton_frontend" diff --git a/lib/frontend/README.md b/lib/frontend/README.md new file mode 100644 index 0000000000..53248b68ab --- /dev/null +++ b/lib/frontend/README.md @@ -0,0 +1,5 @@ +This crate provides a straightforward way to create a +[Cretonne](https://crates.io/crates/cretonne) IL function and fill it with +instructions translated from another language. It contains an SSA construction +module that provides convenient methods for translating non-SSA variables into +SSA Cretonne IL values via `use_var` and `def_var` calls. diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 4307023ab9..9db5a4a34b 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -1,7 +1,7 @@ //! Cretonne IL builder library. //! //! Provides a straightforward way to create a Cretonne IL function and fill it with instructions -//! translated from another language. Contains a SSA construction module that lets you translate +//! translated from another language. Contains an SSA construction module that lets you translate //! your non-SSA variables into SSA Cretonne IL values via `use_var` and `def_var` calls. //! //! To get started, create an [`IlBuilder`](struct.ILBuilder.html) and pass it as an argument diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 0f66bc6037..c7389127c6 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -3,9 +3,10 @@ name = "cretonne-native" version = "0.0.0" authors = ["The Cretonne Project Developers"] publish = false -description = "Support for targetting the host with Cretonne" +description = "Support for targeting the host with Cretonne" repository = "https://github.com/stoklund/cretonne" license = "Apache-2.0" +readme = "README.md" [lib] name = "cton_native" diff --git a/lib/native/README.md b/lib/native/README.md new file mode 100644 index 0000000000..2e91c82ab1 --- /dev/null +++ b/lib/native/README.md @@ -0,0 +1,3 @@ +This crate performs autodetection of the host architecture, which can be used to +configure [Cretonne](https://crates.io/crates/cretonne) to generate code +specialized for the machine it's running on. diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 030d20ff9f..4ea993ab0c 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -7,6 +7,7 @@ license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/stoklund/cretonne" publish = false +readme = "README.md" [lib] name = "cton_reader" diff --git a/lib/reader/README.md b/lib/reader/README.md new file mode 100644 index 0000000000..8430d11919 --- /dev/null +++ b/lib/reader/README.md @@ -0,0 +1,3 @@ +This crate library supports reading .cton files. This functionality is needed +for testing [Cretonne](https://crates.io/crates/cretonne), but is not essential +for a JIT compiler. diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 56d49d7cf6..7947348d98 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -6,6 +6,7 @@ publish = false description = "Translator from WebAssembly to Cretonne IL" repository = "https://github.com/stoklund/cretonne" license = "Apache-2.0" +readme = "README.md" [lib] name = "cton_wasm" diff --git a/lib/wasm/README.md b/lib/wasm/README.md new file mode 100644 index 0000000000..f12f00d46a --- /dev/null +++ b/lib/wasm/README.md @@ -0,0 +1,3 @@ +This crate performs the translation from a wasm module in binary format to the +in-memory representation of the [Cretonne](https://crates.io/crates/cretonne) +IL. From f02c8fd1ff352364f33b9045aba116dbf5bf3e6e Mon Sep 17 00:00:00 2001 From: Benjamin Bouvier Date: Fri, 23 Feb 2018 17:21:44 +0100 Subject: [PATCH 1523/3084] Fixes #244: Prints the generated code size and wasm bytecode size in wasm command; --- cranelift/src/cton-util.rs | 6 +++++- cranelift/src/wasm.rs | 39 ++++++++++++++++++++++++++++++++++- lib/wasm/src/environ/dummy.rs | 5 +++++ 3 files changed, 48 insertions(+), 2 deletions(-) diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index fe121d0055..9f6891489e 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -32,7 +32,7 @@ Usage: cton-util filecheck [-v] cton-util print-cfg ... cton-util compile [-vpT] [--set ]... [--isa ] ... - cton-util wasm [-ctvpT] [--set ]... [--isa ] ... + cton-util wasm [-ctvpTs] [--set ]... [--isa ] ... cton-util --help | --version Options: @@ -41,6 +41,8 @@ Options: print pass timing report -t, --just-decode just decode WebAssembly to Cretonne IL + -s, --print-size + prints generated code size -c, --check-translation just checks the correctness of Cretonne IL translated from WebAssembly -p, --print print the resulting Cretonne IL @@ -67,6 +69,7 @@ struct Args { flag_set: Vec, flag_isa: String, flag_time_passes: bool, + flag_print_size: bool, } /// A command either succeeds or fails with an error message. @@ -103,6 +106,7 @@ fn cton_util() -> CommandResult { args.flag_print, args.flag_set, args.flag_isa, + args.flag_print_size, ) } else { // Debugging / shouldn't happen with proper command line handling above. diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 800bf7d856..ab6a245d3a 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -39,6 +39,7 @@ pub fn run( flag_print: bool, flag_set: Vec, flag_isa: String, + flag_print_size: bool, ) -> Result<(), String> { let parsed = parse_sets_and_isa(flag_set, flag_isa)?; @@ -50,6 +51,7 @@ pub fn run( flag_just_decode, flag_check_translation, flag_print, + flag_print_size, path.to_path_buf(), name, parsed.as_fisa(), @@ -63,6 +65,7 @@ fn handle_module( flag_just_decode: bool, flag_check_translation: bool, flag_print: bool, + flag_print_size: bool, path: PathBuf, name: String, fisa: FlagsOrIsa, @@ -75,6 +78,7 @@ fn handle_module( terminal.fg(term::color::MAGENTA).unwrap(); vprint!(flag_verbose, "Translating... "); terminal.reset().unwrap(); + let mut data = read_to_end(path.clone()).map_err(|err| { String::from(err.description()) })?; @@ -96,11 +100,14 @@ fn handle_module( |err| String::from(err.description()), )?; } + let mut dummy_environ = DummyEnvironment::with_flags(fisa.flags.clone()); translate_module(&data, &mut dummy_environ)?; + terminal.fg(term::color::GREEN).unwrap(); vprintln!(flag_verbose, "ok"); terminal.reset().unwrap(); + if flag_just_decode { if flag_print { let num_func_imports = dummy_environ.get_num_func_imports(); @@ -124,6 +131,7 @@ fn handle_module( } return Ok(()); } + terminal.fg(term::color::MAGENTA).unwrap(); if flag_check_translation { vprint!(flag_verbose, "Checking... "); @@ -131,7 +139,13 @@ fn handle_module( vprint!(flag_verbose, "Compiling... "); } terminal.reset().unwrap(); + + if flag_print_size { + vprintln!(flag_verbose, ""); + } + let num_func_imports = dummy_environ.get_num_func_imports(); + let mut total_module_code_size = 0; for (def_index, func) in dummy_environ.info.function_bodies.iter().enumerate() { let func_index = num_func_imports + def_index; let mut context = Context::new(); @@ -142,9 +156,22 @@ fn handle_module( })?; } else { if let Some(isa) = fisa.isa { - context.compile(isa).map_err(|err| { + let compiled_size = context.compile(isa).map_err(|err| { pretty_error(&context.func, fisa.isa, err) })?; + if flag_print_size { + println!( + "Function #{} code size: {} bytes", + func_index, + compiled_size + ); + total_module_code_size += compiled_size; + println!( + "Function #{} bytecode size: {} bytes", + func_index, + dummy_environ.func_bytecode_sizes[func_index] + ); + } } else { return Err(String::from("compilation requires a target isa")); } @@ -163,6 +190,16 @@ fn handle_module( vprintln!(flag_verbose, ""); } } + + if !flag_check_translation && flag_print_size { + println!("Total module code size: {} bytes", total_module_code_size); + let total_bytecode_size = dummy_environ.func_bytecode_sizes.iter().fold( + 0, + |sum, x| sum + x, + ); + println!("Total module bytecode size: {} bytes", total_bytecode_size); + } + terminal.fg(term::color::GREEN).unwrap(); vprintln!(flag_verbose, "ok"); terminal.reset().unwrap(); diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 90ed3ffb1e..5557d842f6 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -92,6 +92,9 @@ pub struct DummyEnvironment { /// Function translation. trans: FuncTranslator, + + /// Vector of wasm bytecode size for each function. + pub func_bytecode_sizes: Vec, } impl DummyEnvironment { @@ -105,6 +108,7 @@ impl DummyEnvironment { Self { info: DummyModuleInfo::with_flags(flags), trans: FuncTranslator::new(), + func_bytecode_sizes: Vec::new(), } } @@ -384,6 +388,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment { .map_err(|e| String::from(e.description()))?; func }; + self.func_bytecode_sizes.push(body_bytes.len()); self.info.function_bodies.push(func); Ok(()) } From ec746c3359656d051a0105f5b968900b46af1d0a Mon Sep 17 00:00:00 2001 From: Jakob Stoklund Olesen Date: Wed, 21 Feb 2018 15:22:59 -0800 Subject: [PATCH 1524/3084] Add a publish-all.sh script. Set identical version numbers on all cretonne-* crates and print "cargo publish" commands. Update all crates to version 0.1.0. --- cranelift/Cargo.toml | 12 ++++++------ cranelift/publish-all.sh | 34 ++++++++++++++++++++++++++++++++++ lib/cretonne/Cargo.toml | 3 +-- lib/filecheck/Cargo.toml | 4 ++-- lib/frontend/Cargo.toml | 5 ++--- lib/native/Cargo.toml | 5 ++--- lib/reader/Cargo.toml | 5 ++--- lib/wasm/Cargo.toml | 7 +++---- 8 files changed, 52 insertions(+), 23 deletions(-) create mode 100755 cranelift/publish-all.sh diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 45b39ef0a3..fa275ca153 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.0.0" +version = "0.1.0" description = "Binaries for testing the Cretonne library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,11 +13,11 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne = { path = "lib/cretonne" } -cretonne-reader = { path = "lib/reader" } -cretonne-frontend = { path = "lib/frontend" } -cretonne-wasm = { path = "lib/wasm" } -cretonne-native = { path = "lib/native" } +cretonne = { path = "lib/cretonne", version = "0.1.0" } +cretonne-reader = { path = "lib/reader", version = "0.1.0" } +cretonne-frontend = { path = "lib/frontend", version = "0.1.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.1.0" } +cretonne-native = { path = "lib/native", version = "0.1.0" } filecheck = { path = "lib/filecheck" } docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh new file mode 100755 index 0000000000..144aa519f8 --- /dev/null +++ b/cranelift/publish-all.sh @@ -0,0 +1,34 @@ +#!/bin/bash +set -e +cd $(dirname "$0") +topdir=$(pwd) + +# All the cretonne-* crates have the same version number +# The filecheck crate version is managed independently. +version="0.1.0" + +# Update all of the Cargo.toml files. +# +# The main Cargo.toml in the top-level directory is the cretonne-tools crate which we don't publish. +echo "Updating crate versions to $version" +for crate in . lib/*; do + if [ "$crate" = "lib/filecheck" ]; then + continue + fi + # Update the version number of this crate to $version. + sed -i "" -e "s/^version = .*/version = \"$version\"/" $crate/Cargo.toml + # Update the required version number of any cretonne* dependencies. + sed -i "" -e "/^cretonne/s/version = \"[^\"]*\"/version = \"$version\"/" $crate/Cargo.toml +done + +# Update our local Cargo.lock (not checked in). +cargo update +./test-all.sh + +# Commands needed to publish. +# +# Note that libraries need to be published in topological order. + +for crate in filecheck cretonne frontend native reader wasm; do + echo cargo publish --manifest-path lib/$crate/Cargo.toml +done diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index e85779913c..042d372efc 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -1,12 +1,11 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.0.0" +version = "0.1.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/stoklund/cretonne" -publish = false readme = "README.md" build = "build.rs" diff --git a/lib/filecheck/Cargo.toml b/lib/filecheck/Cargo.toml index 41effb059d..c539213d29 100644 --- a/lib/filecheck/Cargo.toml +++ b/lib/filecheck/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "filecheck" -version = "0.0.1" +version = "0.1.0" description = "Library for matching test outputs against filecheck directives" license = "Apache-2.0" repository = "https://github.com/stoklund/cretonne" @@ -11,4 +11,4 @@ documentation = "https://docs.rs/filecheck" name = "filecheck" [dependencies] -regex = "0.2.2" +regex = "0.2.6" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 704517c176..6f8e46b67c 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,16 +1,15 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.0.0" +version = "0.1.0" description = "Cretonne IL builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/stoklund/cretonne" -publish = false readme = "README.md" [lib] name = "cton_frontend" [dependencies] -cretonne = { path = "../cretonne" } +cretonne = { path = "../cretonne", version = "0.1.0" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index c7389127c6..32230306b1 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,8 +1,7 @@ [package] name = "cretonne-native" -version = "0.0.0" +version = "0.1.0" authors = ["The Cretonne Project Developers"] -publish = false description = "Support for targeting the host with Cretonne" repository = "https://github.com/stoklund/cretonne" license = "Apache-2.0" @@ -12,7 +11,7 @@ readme = "README.md" name = "cton_native" [dependencies] -cretonne = { path = "../cretonne" } +cretonne = { path = "../cretonne", version = "0.1.0" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 4ea993ab0c..47b2eb98b7 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,16 +1,15 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.0.0" +version = "0.1.0" description = "Cretonne textual IL reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/stoklund/cretonne" -publish = false readme = "README.md" [lib] name = "cton_reader" [dependencies] -cretonne = { path = "../cretonne" } +cretonne = { path = "../cretonne", version = "0.1.0" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 7947348d98..fd1fe693ae 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,8 +1,7 @@ [package] name = "cretonne-wasm" -version = "0.0.0" +version = "0.1.0" authors = ["The Cretonne Project Developers"] -publish = false description = "Translator from WebAssembly to Cretonne IL" repository = "https://github.com/stoklund/cretonne" license = "Apache-2.0" @@ -13,8 +12,8 @@ name = "cton_wasm" [dependencies] wasmparser = "0.14.1" -cretonne = { path = "../cretonne" } -cretonne-frontend = { path = "../frontend" } +cretonne = { path = "../cretonne", version = "0.1.0" } +cretonne-frontend = { path = "../frontend", version = "0.1.0" } [dev-dependencies] tempdir = "0.3.5" From 2a26b70854a4be14c4fb5071567d67379304ec3a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 23 Feb 2018 16:16:44 -0800 Subject: [PATCH 1525/3084] Update URLs. --- README.rst | 4 ++-- cranelift/Cargo.toml | 2 +- cranelift/filetests/regalloc/coalescing-207.cton | 2 +- cranelift/filetests/regalloc/coalescing-216.cton | 2 +- cranelift/filetests/regalloc/reload-208.cton | 2 +- lib/cretonne/Cargo.toml | 2 +- lib/filecheck/Cargo.toml | 2 +- lib/frontend/Cargo.toml | 2 +- lib/native/Cargo.toml | 2 +- lib/reader/Cargo.toml | 2 +- lib/wasm/Cargo.toml | 2 +- 11 files changed, 12 insertions(+), 12 deletions(-) diff --git a/README.rst b/README.rst index 31e674350b..05ddeb6b70 100644 --- a/README.rst +++ b/README.rst @@ -12,8 +12,8 @@ machine code. :target: https://cretonne.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status -.. image:: https://travis-ci.org/stoklund/cretonne.svg?branch=master - :target: https://travis-ci.org/stoklund/cretonne +.. image:: https://travis-ci.org/Cretonne/cretonne.svg?branch=master + :target: https://travis-ci.org/Cretonne/cretonne :alt: Build Status For more information, see `the documentation diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index fa275ca153..91fe8ed6d9 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -5,7 +5,7 @@ version = "0.1.0" description = "Binaries for testing the Cretonne library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/stoklund/cretonne" +repository = "https://github.com/Cretonne/cretonne" publish = false [[bin]] diff --git a/cranelift/filetests/regalloc/coalescing-207.cton b/cranelift/filetests/regalloc/coalescing-207.cton index 9485c89f36..8641c87354 100644 --- a/cranelift/filetests/regalloc/coalescing-207.cton +++ b/cranelift/filetests/regalloc/coalescing-207.cton @@ -2,7 +2,7 @@ test regalloc set is_64bit isa intel haswell -; Reported as https://github.com/stoklund/cretonne/issues/207 +; Reported as https://github.com/Cretonne/cretonne/issues/207 ; ; The coalescer creates a virtual register with two interfering values. function %pr207(i64 vmctx, i32, i32) -> i32 native { diff --git a/cranelift/filetests/regalloc/coalescing-216.cton b/cranelift/filetests/regalloc/coalescing-216.cton index 83dda0e56a..de44d53d5b 100644 --- a/cranelift/filetests/regalloc/coalescing-216.cton +++ b/cranelift/filetests/regalloc/coalescing-216.cton @@ -2,7 +2,7 @@ test regalloc set is_64bit isa intel haswell -; Reported as https://github.com/stoklund/cretonne/issues/216 from the Binaryen fuzzer. +; Reported as https://github.com/Cretonne/cretonne/issues/216 from the Binaryen fuzzer. ; ; The (old) coalescer creates a virtual register with two identical values. function %pr216(i32 [%rdi], i64 vmctx [%rsi]) -> i64 [%rax] native { diff --git a/cranelift/filetests/regalloc/reload-208.cton b/cranelift/filetests/regalloc/reload-208.cton index 1bf8120d57..1897a03c5d 100644 --- a/cranelift/filetests/regalloc/reload-208.cton +++ b/cranelift/filetests/regalloc/reload-208.cton @@ -4,7 +4,7 @@ isa intel haswell ; regex: V=v\d+ -; Filed as https://github.com/stoklund/cretonne/issues/208 +; Filed as https://github.com/Cretonne/cretonne/issues/208 ; ; The verifier complains about a branch argument that is not in the same virtual register as the ; corresponding EBB argument. diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 042d372efc..88ae712c07 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -5,7 +5,7 @@ version = "0.1.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/stoklund/cretonne" +repository = "https://github.com/Cretonne/cretonne" readme = "README.md" build = "build.rs" diff --git a/lib/filecheck/Cargo.toml b/lib/filecheck/Cargo.toml index c539213d29..aec22b0f51 100644 --- a/lib/filecheck/Cargo.toml +++ b/lib/filecheck/Cargo.toml @@ -4,7 +4,7 @@ name = "filecheck" version = "0.1.0" description = "Library for matching test outputs against filecheck directives" license = "Apache-2.0" -repository = "https://github.com/stoklund/cretonne" +repository = "https://github.com/Cretonne/cretonne" documentation = "https://docs.rs/filecheck" [lib] diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 6f8e46b67c..eea524ef1c 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -5,7 +5,7 @@ version = "0.1.0" description = "Cretonne IL builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/stoklund/cretonne" +repository = "https://github.com/Cretonne/cretonne" readme = "README.md" [lib] diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 32230306b1..13556b0eb0 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -3,7 +3,7 @@ name = "cretonne-native" version = "0.1.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" -repository = "https://github.com/stoklund/cretonne" +repository = "https://github.com/Cretonne/cretonne" license = "Apache-2.0" readme = "README.md" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 47b2eb98b7..636f6b40e0 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -5,7 +5,7 @@ version = "0.1.0" description = "Cretonne textual IL reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/stoklund/cretonne" +repository = "https://github.com/Cretonne/cretonne" readme = "README.md" [lib] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index fd1fe693ae..2bad5f8d2a 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -3,7 +3,7 @@ name = "cretonne-wasm" version = "0.1.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IL" -repository = "https://github.com/stoklund/cretonne" +repository = "https://github.com/Cretonne/cretonne" license = "Apache-2.0" readme = "README.md" From 7375088c3ec10c9f12298a011d2eec69aed7a633 Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Mon, 15 Jan 2018 14:05:38 -0500 Subject: [PATCH 1526/3084] Most of the way to no_std support --- lib/cretonne/Cargo.toml | 8 +++++ lib/cretonne/src/abi.rs | 1 + lib/cretonne/src/dbg.rs | 18 ++++++++++ lib/cretonne/src/dominator_tree.rs | 1 + lib/cretonne/src/entity/list.rs | 1 + lib/cretonne/src/entity/map.rs | 1 + lib/cretonne/src/entity/primary.rs | 1 + lib/cretonne/src/entity/set.rs | 1 + lib/cretonne/src/entity/sparse.rs | 1 + lib/cretonne/src/ir/extfunc.rs | 1 + lib/cretonne/src/ir/instructions.rs | 1 + lib/cretonne/src/ir/jumptable.rs | 1 + lib/cretonne/src/ir/stackslot.rs | 1 + lib/cretonne/src/isa/arm32/mod.rs | 2 ++ lib/cretonne/src/isa/arm64/mod.rs | 2 ++ lib/cretonne/src/isa/intel/mod.rs | 1 + lib/cretonne/src/isa/mod.rs | 2 ++ lib/cretonne/src/isa/riscv/mod.rs | 2 ++ lib/cretonne/src/legalizer/boundary.rs | 1 + lib/cretonne/src/legalizer/libcall.rs | 2 +- lib/cretonne/src/legalizer/split.rs | 1 + lib/cretonne/src/lib.rs | 34 ++++++++++++++++++- lib/cretonne/src/licm.rs | 1 + lib/cretonne/src/loop_analysis.rs | 1 + lib/cretonne/src/regalloc/coalescing.rs | 3 ++ lib/cretonne/src/regalloc/diversion.rs | 1 + .../src/regalloc/live_value_tracker.rs | 1 + lib/cretonne/src/regalloc/liveness.rs | 1 + lib/cretonne/src/regalloc/reload.rs | 1 + lib/cretonne/src/regalloc/solver.rs | 1 + lib/cretonne/src/regalloc/spilling.rs | 1 + lib/cretonne/src/settings.rs | 1 + lib/cretonne/src/simple_gvn.rs | 3 ++ lib/cretonne/src/timing.rs | 25 ++++++++++++-- lib/cretonne/src/topo_order.rs | 3 ++ lib/cretonne/src/verifier/mod.rs | 2 ++ lib/cretonne/src/write.rs | 1 + lib/frontend/Cargo.toml | 3 ++ lib/frontend/src/lib.rs | 12 +++++++ lib/frontend/src/ssa.rs | 1 + lib/native/Cargo.toml | 3 ++ lib/native/src/lib.rs | 2 +- 42 files changed, 146 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 88ae712c07..91dc6ccdac 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -17,3 +17,11 @@ name = "cretonne" # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be # accomodated in `tests`. +[dependencies.hashmap_core] +version = "0.1.1" +optional = true + +[features] +# Currently, the only feature is the `no_std` feature. +# Enabling this disables use of `stdlib`. +no_std = ["hashmap_core"] \ No newline at end of file diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index 5c121824aa..45840d63db 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -5,6 +5,7 @@ use ir::{ArgumentLoc, AbiParam, ArgumentExtension, Type}; use std::cmp::Ordering; +use std::vec::Vec; /// Legalization action to perform on a single argument or return value when converting a /// signature. diff --git a/lib/cretonne/src/dbg.rs b/lib/cretonne/src/dbg.rs index 5acadd2098..cb8dd07118 100644 --- a/lib/cretonne/src/dbg.rs +++ b/lib/cretonne/src/dbg.rs @@ -10,12 +10,17 @@ /// thread doing the logging. use std::cell::RefCell; +#[cfg(not(feature = "no_std"))] use std::env; +#[cfg(not(feature = "no_std"))] use std::ffi::OsStr; use std::fmt; +#[cfg(not(feature = "no_std"))] use std::fs::File; +#[cfg(not(feature = "no_std"))] use std::io::{self, Write}; use std::sync::atomic; +#[cfg(not(feature = "no_std"))] use std::thread; static STATE: atomic::AtomicIsize = atomic::ATOMIC_ISIZE_INIT; @@ -26,6 +31,7 @@ static STATE: atomic::AtomicIsize = atomic::ATOMIC_ISIZE_INIT; /// other than `0`. /// /// This inline function turns into a constant `false` when debug assertions are disabled. +#[cfg(not(feature = "no_std"))] #[inline] pub fn enabled() -> bool { if cfg!(debug_assertions) { @@ -38,7 +44,15 @@ pub fn enabled() -> bool { } } +/// Does nothing +#[cfg(feature = "no_std")] +#[inline] +pub fn enabled() -> bool { + false +} + /// Initialize `STATE` from the environment variable. +#[cfg(not(feature = "no_std"))] fn initialize() -> bool { let enable = match env::var_os("CRETONNE_DBG") { Some(s) => s != OsStr::new("0"), @@ -54,6 +68,7 @@ fn initialize() -> bool { enable } +#[cfg(not(feature = "no_std"))] thread_local! { static WRITER : RefCell> = RefCell::new(open_file()); } @@ -61,6 +76,7 @@ thread_local! { /// Write a line with the given format arguments. /// /// This is for use by the `dbg!` macro. +#[cfg(not(feature = "no_std"))] pub fn writeln_with_format_args(args: fmt::Arguments) -> io::Result<()> { WRITER.with(|rc| { let mut w = rc.borrow_mut(); @@ -70,6 +86,7 @@ pub fn writeln_with_format_args(args: fmt::Arguments) -> io::Result<()> { } /// Open the tracing file for the current thread. +#[cfg(not(feature = "no_std"))] fn open_file() -> io::BufWriter { let curthread = thread::current(); let tmpstr; @@ -97,6 +114,7 @@ macro_rules! dbg { if $crate::dbg::enabled() { // Drop the error result so we don't get compiler errors for ignoring it. // What are you going to do, log the error? + #[cfg(not(feature = "no_std"))] $crate::dbg::writeln_with_format_args(format_args!($($arg)+)).ok(); } } diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index f7b7273681..a7625f840e 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -10,6 +10,7 @@ use std::mem; use timing; use std::cmp::Ordering; +use std::vec::Vec; // RPO numbers are not first assigned in a contiguous way but as multiples of STRIDE, to leave // room for modifications of the dominator tree. diff --git a/lib/cretonne/src/entity/list.rs b/lib/cretonne/src/entity/list.rs index c9bb5b1a69..58c91d9287 100644 --- a/lib/cretonne/src/entity/list.rs +++ b/lib/cretonne/src/entity/list.rs @@ -3,6 +3,7 @@ use entity::EntityRef; use std::hash::{Hash, Hasher}; use std::marker::PhantomData; use std::mem; +use std::vec::Vec; /// A small list of entity references allocated from a pool. /// diff --git a/lib/cretonne/src/entity/map.rs b/lib/cretonne/src/entity/map.rs index 9622f93485..f251c2f831 100644 --- a/lib/cretonne/src/entity/map.rs +++ b/lib/cretonne/src/entity/map.rs @@ -3,6 +3,7 @@ use entity::{EntityRef, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; +use std::vec::Vec; /// A mapping `K -> V` for densely indexed entity references. /// diff --git a/lib/cretonne/src/entity/primary.rs b/lib/cretonne/src/entity/primary.rs index 137320f3e5..c06b818355 100644 --- a/lib/cretonne/src/entity/primary.rs +++ b/lib/cretonne/src/entity/primary.rs @@ -2,6 +2,7 @@ use entity::{EntityRef, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; +use std::vec::Vec; /// A primary mapping `K -> V` allocating dense entity references. /// diff --git a/lib/cretonne/src/entity/set.rs b/lib/cretonne/src/entity/set.rs index 82dc1384ee..e4acf47723 100644 --- a/lib/cretonne/src/entity/set.rs +++ b/lib/cretonne/src/entity/set.rs @@ -2,6 +2,7 @@ use entity::{EntityRef, Keys}; use std::marker::PhantomData; +use std::vec::Vec; /// A set of `K` for densely indexed entity references. /// diff --git a/lib/cretonne/src/entity/sparse.rs b/lib/cretonne/src/entity/sparse.rs index dacbf66031..a4836cb9d5 100644 --- a/lib/cretonne/src/entity/sparse.rs +++ b/lib/cretonne/src/entity/sparse.rs @@ -11,6 +11,7 @@ use entity::{EntityRef, EntityMap}; use std::mem; use std::slice; use std::u32; +use std::vec::Vec; /// Trait for extracting keys from values stored in a `SparseMap`. /// diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index b3c3a274e7..0aa87fa60e 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -10,6 +10,7 @@ use isa::{RegInfo, RegUnit}; use std::cmp; use std::fmt; use std::str::FromStr; +use std::vec::Vec; /// Function signature. /// diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index af570f7ad5..1340c14055 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -9,6 +9,7 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::ops::{Deref, DerefMut}; +use std::vec::Vec; use ir; use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef, StackSlot, MemFlags}; diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index 2d09aab4da..4f6e117b64 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -8,6 +8,7 @@ use ir::entities::Ebb; use std::iter; use std::slice; use std::fmt::{self, Display, Formatter}; +use std::vec::Vec; /// Contents of a jump table. /// diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index 4a0804787b..744bd06ac3 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -10,6 +10,7 @@ use std::cmp; use std::fmt; use std::ops::{Index, IndexMut}; use std::str::FromStr; +use std::vec::Vec; /// The size of an object on the stack, or the size of a stack frame. /// diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index c3f295d4fb..50ef3ea7db 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -15,6 +15,8 @@ use ir; use regalloc; use std::fmt; +use std::boxed::Box; + #[allow(dead_code)] struct Isa { shared_flags: shared_settings::Flags, diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index d0644ae2f6..b69e044bf4 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -15,6 +15,8 @@ use ir; use regalloc; use std::fmt; +use std::boxed::Box; + #[allow(dead_code)] struct Isa { shared_flags: shared_settings::Flags, diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 3f02890a82..e4841558e4 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -17,6 +17,7 @@ use result; use timing; use std::fmt; +use std::boxed::Box; #[allow(dead_code)] struct Isa { diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 38164e5167..cedfdaa301 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -55,6 +55,8 @@ use timing; use isa::enc_tables::Encodings; use std::fmt; +use std::boxed::Box; + #[cfg(build_riscv)] mod riscv; diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 2fd0d5cdb9..1993ecb9d6 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -15,6 +15,8 @@ use ir; use regalloc; use std::fmt; +use std::boxed::Box; + #[allow(dead_code)] struct Isa { shared_flags: shared_settings::Flags, diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index ea81c1ff95..5d9dd71dc0 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -25,6 +25,7 @@ use ir::{Function, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, Signature use ir::instructions::CallInfo; use isa::TargetIsa; use legalizer::split::{isplit, vsplit}; +use std::vec::Vec; /// Legalize all the function signatures in `func`. /// diff --git a/lib/cretonne/src/legalizer/libcall.rs b/lib/cretonne/src/legalizer/libcall.rs index 156a05b813..cdf6a763a4 100644 --- a/lib/cretonne/src/legalizer/libcall.rs +++ b/lib/cretonne/src/legalizer/libcall.rs @@ -15,7 +15,7 @@ pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function) -> bool { let funcref = find_funcref(libcall, func).unwrap_or_else(|| make_funcref(libcall, inst, func)); // Now we convert `inst` to a call. First save the arguments. - let mut args = Vec::new(); + let mut args = vec![]; args.extend_from_slice(func.dfg.inst_args(inst)); // The replace builder will preserve the instruction result values. func.dfg.replace(inst).call(funcref, &args); diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index 0c73e7c2d4..76ad05a2bc 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -68,6 +68,7 @@ use cursor::{Cursor, CursorPosition, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::{self, Ebb, Inst, Value, Type, Opcode, ValueDef, InstructionData, InstBuilder}; use std::iter; +use std::vec::Vec; /// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values /// if possible. diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index b4c386477c..2bed9df4fb 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -1,7 +1,17 @@ //! Cretonne code generation library. - +#![cfg_attr(feature = "no_std", no_std)] #![deny(missing_docs)] +// Turns on alloc feature if no_std +#![cfg_attr(feature = "no_std", feature(alloc))] + +// Include the `hashmap_core` crate if no_std +#[cfg(feature = "no_std")] +extern crate hashmap_core; +#[cfg(feature = "no_std")] +#[macro_use] +extern crate alloc; + pub use context::Context; pub use legalizer::legalize_function; pub use verifier::verify_function; @@ -46,3 +56,25 @@ mod stack_layout; mod topo_order; mod unreachable_code; mod write; + +/// This replaces `std` in builds with no_std. +#[cfg(feature = "no_std")] +mod std { + pub use core::*; + #[macro_use] + pub use alloc::{boxed, vec, string}; + pub mod prelude { + pub use core::prelude as v1; + } + pub mod collections { + pub use hashmap_core::{HashMap, HashSet}; + pub use hashmap_core::map as hash_map; + pub use alloc::BTreeSet; + } + pub mod error { + pub trait Error { + fn description(&self) -> &str; + fn cause(&self) -> Option<&Error> { None } + } + } +} \ No newline at end of file diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 1eb035d9bf..e767fc838b 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -8,6 +8,7 @@ use dominator_tree::DominatorTree; use entity::{EntityList, ListPool}; use loop_analysis::{Loop, LoopAnalysis}; use timing; +use std::vec::Vec; /// Performs the LICM pass by detecting loops within the CFG and moving /// loop-invariant instructions out of them. diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index f6dee54d71..dfd5cf135a 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -8,6 +8,7 @@ use flowgraph::ControlFlowGraph; use ir::{Function, Ebb, Layout}; use packed_option::PackedOption; use timing; +use std::vec::Vec; /// A opaque reference to a code loop. #[derive(Copy, Clone, PartialEq, Eq, Hash)] diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 416783b6c0..225aa8e9d8 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -18,6 +18,9 @@ use std::cmp; use std::iter; use std::fmt; use std::slice; +use std::iter::Peekable; +use std::mem; +use std::vec::Vec; use isa::{TargetIsa, EncInfo}; use timing; diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index 588ad040a5..16eb0e9b50 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -11,6 +11,7 @@ use ir::{Value, ValueLoc, ValueLocations, StackSlot}; use ir::{InstructionData, Opcode}; use isa::{RegUnit, RegInfo}; use std::fmt; +use std::vec::Vec; /// A diversion of a value from its original location to a new register or stack location. /// diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index d0810a5cf7..de300ced0e 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -13,6 +13,7 @@ use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; use regalloc::liverange::LiveRange; use std::collections::HashMap; +use std::vec::Vec; type ValueList = EntityList; diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 2778f47f63..83b63366a5 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -184,6 +184,7 @@ use regalloc::affinity::Affinity; use regalloc::liverange::{LiveRange, LiveRangeForest, LiveRangeContext}; use std::mem; use std::ops::Index; +use std::vec::Vec; use timing; /// A set of live ranges, indexed by value number. diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 2cb37d7d6a..10b803c831 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -21,6 +21,7 @@ use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; use timing; use topo_order::TopoOrder; +use std::vec::Vec; /// Reusable data structures for the reload pass. pub struct Reload { diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index a3b732799c..c2f8017c3a 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -108,6 +108,7 @@ use std::fmt; use std::mem; use super::AllocatableSet; use std::u16; +use std::vec::Vec; /// A variable in the constraint problem. /// diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 84ef76fdff..5f6677d650 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -26,6 +26,7 @@ use regalloc::liveness::Liveness; use regalloc::pressure::Pressure; use regalloc::virtregs::VirtRegs; use std::fmt; +use std::vec::Vec; use timing; use topo_order::TopoOrder; diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 36a48e04b4..a7da8d6f17 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -24,6 +24,7 @@ use constant_hash::{probe, simple_hash}; use isa::TargetIsa; use std::fmt; use std::result; +use std::vec::Vec; /// A string-based configurator for settings groups. /// diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index 8642122593..e0350ad687 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -7,6 +7,9 @@ use ir::{InstructionData, Function, Inst, Opcode, Type}; use scoped_hash_map::ScopedHashMap; use timing; +#[cfg(feature = "no_std")] +use alloc::Vec; + /// Test whether the given opcode is unsafe to even consider for GVN. fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || diff --git a/lib/cretonne/src/timing.rs b/lib/cretonne/src/timing.rs index d120d43c32..90d15353e1 100644 --- a/lib/cretonne/src/timing.rs +++ b/lib/cretonne/src/timing.rs @@ -87,12 +87,12 @@ impl fmt::Display for Pass { } } - /// Implementation details. /// /// This whole module can be gated on a `cfg` feature to provide a dummy implementation for /// performance-sensitive builds or restricted environments. The dummy implementation must provide -/// `TimingToken` and `PassTimings` types and a `take_current` function. +/// `TimingToken` and `PassTimes` types and `take_current`, `add_to_current`, and `start_pass` functions. +#[cfg(not(feature = "no_std"))] mod details { use super::{Pass, NUM_PASSES, DESCRIPTIONS}; use std::cell::{Cell, RefCell}; @@ -214,6 +214,27 @@ mod details { } } +/// Dummy `debug` implementation +#[cfg(feature = "no_std")] +mod details { + use super::Pass; + /// Dummy `TimingToken` + pub struct TimingToken; + /// Dummy `PassTimes` + pub struct PassTimes; + /// Returns dummy `PassTimes` + pub fn take_current() -> PassTimes { + PassTimes + } + /// does nothing + pub fn add_to_current(_times: PassTimes) { } + + /// does nothing + pub(super) fn start_pass(_pass: Pass) -> TimingToken { + TimingToken + } +} + #[cfg(test)] mod test { use super::*; diff --git a/lib/cretonne/src/topo_order.rs b/lib/cretonne/src/topo_order.rs index cd783c928a..660179f1f9 100644 --- a/lib/cretonne/src/topo_order.rs +++ b/lib/cretonne/src/topo_order.rs @@ -4,6 +4,9 @@ use entity::SparseSet; use dominator_tree::DominatorTree; use ir::{Ebb, Layout}; +#[cfg(feature = "no_std")] +use alloc::Vec; + /// Present EBBs in a topological order such that all dominating EBBs are guaranteed to be visited /// before the current EBB. /// diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index a346573e67..c3bebc5ae6 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -73,6 +73,8 @@ use std::collections::BTreeSet; use std::error as std_error; use std::fmt::{self, Display, Formatter, Write}; use std::result; +use std::vec::Vec; +use std::string::String; use timing; pub use self::cssa::verify_cssa; diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index d9853d5768..4ca8ddaadb 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -9,6 +9,7 @@ use isa::{TargetIsa, RegInfo}; use std::fmt::{self, Result, Error, Write}; use std::result; use packed_option::ReservedValue; +use std::string::String; /// Write `func` to `w` as equivalent text. /// Use `isa` to emit ISA-dependent annotations. diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index eea524ef1c..32c65e4ded 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -13,3 +13,6 @@ name = "cton_frontend" [dependencies] cretonne = { path = "../cretonne", version = "0.1.0" } + +[features] +no_std = ["cretonne/no_std"] diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 9db5a4a34b..37a3504bf5 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -142,11 +142,23 @@ //! } //! ``` +#![cfg_attr(feature = "no_std", no_std)] #![deny(missing_docs)] +#![cfg_attr(feature = "no_std", feature(alloc))] + extern crate cretonne; +#[cfg(feature = "no_std")] +extern crate alloc; + pub use frontend::{ILBuilder, FunctionBuilder}; mod frontend; mod ssa; + +#[cfg(feature = "no_std")] +mod std { + pub use alloc::vec; + pub use core::*; +} \ No newline at end of file diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 2b758ef18b..f99ff5e179 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -15,6 +15,7 @@ use std::u32; use cretonne::ir::types::{F32, F64}; use cretonne::ir::immediates::{Ieee32, Ieee64}; use std::mem; +use std::vec::Vec; /// Structure containing the data relevant the construction of SSA for a given function. /// diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 13556b0eb0..57a2d0bc3c 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -15,3 +15,6 @@ cretonne = { path = "../cretonne", version = "0.1.0" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" + +[features] +no_std = ["cretonne/no_std"] diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index 6cdb39a15e..284366a65c 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -1,6 +1,6 @@ //! Performs autodetection of the host for the purposes of running //! Cretonne to generate code to run on the same machine. - +#![cfg_attr(feature = "no_std", no_std)] #![deny(missing_docs)] extern crate cretonne; From 299e8a9737259e6fd8052f3e520533f8fc3cd7f9 Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Mon, 15 Jan 2018 16:21:26 -0500 Subject: [PATCH 1527/3084] lib/cretonne works with no_std --- lib/cretonne/Cargo.toml | 5 ++++- lib/cretonne/src/dbg.rs | 1 + lib/cretonne/src/lib.rs | 7 +++---- 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 91dc6ccdac..0cf3fffd9b 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -20,8 +20,11 @@ name = "cretonne" [dependencies.hashmap_core] version = "0.1.1" optional = true +[dependencies.error_core] +version = "0.1.0" +optional = true [features] # Currently, the only feature is the `no_std` feature. # Enabling this disables use of `stdlib`. -no_std = ["hashmap_core"] \ No newline at end of file +no_std = ["hashmap_core", "error_core"] \ No newline at end of file diff --git a/lib/cretonne/src/dbg.rs b/lib/cretonne/src/dbg.rs index cb8dd07118..7e5cd2777c 100644 --- a/lib/cretonne/src/dbg.rs +++ b/lib/cretonne/src/dbg.rs @@ -9,6 +9,7 @@ /// The output will appear in files named `cretonne.dbg.*`, where the suffix is named after the /// thread doing the logging. +#[cfg(not(feature = "no_std"))] use std::cell::RefCell; #[cfg(not(feature = "no_std"))] use std::env; diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 2bed9df4fb..c448cc1d0f 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -9,6 +9,8 @@ #[cfg(feature = "no_std")] extern crate hashmap_core; #[cfg(feature = "no_std")] +extern crate error_core; +#[cfg(feature = "no_std")] #[macro_use] extern crate alloc; @@ -72,9 +74,6 @@ mod std { pub use alloc::BTreeSet; } pub mod error { - pub trait Error { - fn description(&self) -> &str; - fn cause(&self) -> Option<&Error> { None } - } + pub use error_core::Error; } } \ No newline at end of file From 66a150e67acded290bd7259ca7cc5113e428f5c0 Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Mon, 15 Jan 2018 16:21:42 -0500 Subject: [PATCH 1528/3084] lib/wasm works with no_std --- lib/wasm/Cargo.toml | 12 ++++++++++++ lib/wasm/src/code_translator.rs | 3 +++ lib/wasm/src/environ/dummy.rs | 4 ++++ lib/wasm/src/environ/spec.rs | 4 ++++ lib/wasm/src/lib.rs | 25 +++++++++++++++++++++++++ lib/wasm/src/module_translator.rs | 3 +++ lib/wasm/src/sections_translator.rs | 4 ++++ lib/wasm/src/state.rs | 3 +++ 8 files changed, 58 insertions(+) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 2bad5f8d2a..448166d568 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -15,5 +15,17 @@ wasmparser = "0.14.1" cretonne = { path = "../cretonne", version = "0.1.0" } cretonne-frontend = { path = "../frontend", version = "0.1.0" } +[dependencies.hashmap_core] +version = "0.1.1" +optional = true +[dependencies.error_core] +version = "0.1.0" +optional = true + [dev-dependencies] tempdir = "0.3.5" + +[features] +# Currently, the only feature is the `no_std` feature. +# Enabling this disables use of `stdlib`. +no_std = ["hashmap_core", "error_core", "cretonne/no_std", "cretonne-frontend/no_std"] \ No newline at end of file diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 903b9cd9e1..ea33f311a9 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -34,6 +34,9 @@ use std::collections::{HashMap, hash_map}; use environ::{FuncEnvironment, GlobalValue}; use std::{i32, u32}; +// this is for no_std builds, but has no affect on regular builds +use std::vec::Vec; + /// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted /// a return. pub fn translate_operator( diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 5557d842f6..9238cbf665 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -11,6 +11,10 @@ use cretonne::settings; use wasmparser; use std::error::Error; +// this is for no_std builds, but has no affect on regular builds +use std::vec::Vec; +use std::string::String; + /// Compute a `ir::ExternalName` for a given wasm function index. fn get_func_name(func_index: FunctionIndex) -> ir::ExternalName { ir::ExternalName::user(0, func_index as u32) diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 31e4782918..c8a8f54111 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -6,6 +6,10 @@ use cretonne::settings::Flags; use translation_utils::{SignatureIndex, FunctionIndex, TableIndex, GlobalIndex, MemoryIndex, Global, Table, Memory}; +// this is for no_std builds, but has no affect on regular builds +use std::vec::Vec; +use std::string::String; + /// The value of a WebAssembly global variable. #[derive(Clone, Copy)] pub enum GlobalValue { diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index b5b79049fe..dde500af97 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -9,8 +9,20 @@ //! //! The main function of this module is [`translate_module`](fn.translate_module.html). +#![cfg_attr(feature = "no_std", no_std)] #![deny(missing_docs)] +#![cfg_attr(feature = "no_std", feature(alloc))] + +#[cfg(feature = "no_std")] +#[macro_use] +extern crate alloc; + +#[cfg(feature = "no_std")] +extern crate hashmap_core; +#[cfg(feature = "no_std")] +extern crate error_core; + extern crate wasmparser; extern crate cton_frontend; #[macro_use(dbg)] @@ -29,3 +41,16 @@ pub use module_translator::translate_module; pub use environ::{FuncEnvironment, ModuleEnvironment, DummyEnvironment, GlobalValue}; pub use translation_utils::{FunctionIndex, GlobalIndex, TableIndex, MemoryIndex, SignatureIndex, Global, GlobalInit, Table, Memory}; + +#[cfg(feature = "no_std")] +mod std { + pub use alloc::vec; + pub use alloc::string; + pub use core::{u32, i32, str, cmp}; + pub mod collections { + pub use hashmap_core::{HashMap, map as hash_map}; + } + pub mod error { + pub use error_core::Error; + } +} \ No newline at end of file diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index d75f60a35e..2786f9094e 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -8,6 +8,9 @@ use sections_translator::{SectionParsingError, parse_function_signatures, parse_ parse_elements_section, parse_data_section}; use environ::ModuleEnvironment; +// this is for no_std builds, but has no affect on regular builds +use std::string::String; + /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IL /// [`Function`](../cretonne/ir/function/struct.Function.html). /// Returns the functions and also the mappings for imported functions and signature between the diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index d02a4a7c2a..ff641a6889 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -17,6 +17,10 @@ use wasmparser; use std::str::from_utf8; use environ::ModuleEnvironment; +// this is for no_std builds, but has no affect on regular builds +use std::vec::Vec; +use std::string::String; + pub enum SectionParsingError { WrongSectionContent(String), } diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index f9f76d2934..625e39b4e9 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -8,6 +8,9 @@ use environ::{FuncEnvironment, GlobalValue}; use std::collections::HashMap; use translation_utils::{GlobalIndex, MemoryIndex, SignatureIndex, FunctionIndex}; +// this is for no_std builds, but has no affect on regular builds +use std::vec::Vec; + /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following /// fields: /// From 5c85c1ba4a1890fa0ec05b3355c85ffab8435fc9 Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Mon, 15 Jan 2018 17:02:28 -0500 Subject: [PATCH 1529/3084] Fixed formatting issues --- lib/cretonne/src/timing.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/timing.rs b/lib/cretonne/src/timing.rs index 90d15353e1..c05ee93004 100644 --- a/lib/cretonne/src/timing.rs +++ b/lib/cretonne/src/timing.rs @@ -91,7 +91,7 @@ impl fmt::Display for Pass { /// /// This whole module can be gated on a `cfg` feature to provide a dummy implementation for /// performance-sensitive builds or restricted environments. The dummy implementation must provide -/// `TimingToken` and `PassTimes` types and `take_current`, `add_to_current`, and `start_pass` functions. +/// `TimingToken` and `PassTimes` types and `take_current`, `add_to_current`, and `start_pass` funcs #[cfg(not(feature = "no_std"))] mod details { use super::{Pass, NUM_PASSES, DESCRIPTIONS}; @@ -227,7 +227,7 @@ mod details { PassTimes } /// does nothing - pub fn add_to_current(_times: PassTimes) { } + pub fn add_to_current(_times: PassTimes) {} /// does nothing pub(super) fn start_pass(_pass: Pass) -> TimingToken { From 2462a065add5e516794cd225e1bbd45e69d1c1e5 Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Mon, 15 Jan 2018 17:14:48 -0500 Subject: [PATCH 1530/3084] Fixed formatting issues --- lib/cretonne/src/dbg.rs | 1 - lib/cretonne/src/lib.rs | 2 +- lib/cretonne/src/simple_gvn.rs | 3 +-- lib/cretonne/src/topo_order.rs | 3 +-- lib/frontend/src/lib.rs | 2 +- lib/wasm/src/code_translator.rs | 1 - lib/wasm/src/environ/dummy.rs | 1 - lib/wasm/src/environ/spec.rs | 1 - lib/wasm/src/lib.rs | 2 +- lib/wasm/src/module_translator.rs | 1 - lib/wasm/src/sections_translator.rs | 1 - lib/wasm/src/state.rs | 1 - 12 files changed, 5 insertions(+), 14 deletions(-) diff --git a/lib/cretonne/src/dbg.rs b/lib/cretonne/src/dbg.rs index 7e5cd2777c..ace22b6c47 100644 --- a/lib/cretonne/src/dbg.rs +++ b/lib/cretonne/src/dbg.rs @@ -8,7 +8,6 @@ /// /// The output will appear in files named `cretonne.dbg.*`, where the suffix is named after the /// thread doing the logging. - #[cfg(not(feature = "no_std"))] use std::cell::RefCell; #[cfg(not(feature = "no_std"))] diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index c448cc1d0f..22914cc17b 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -76,4 +76,4 @@ mod std { pub mod error { pub use error_core::Error; } -} \ No newline at end of file +} diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index e0350ad687..8f09398c82 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -7,8 +7,7 @@ use ir::{InstructionData, Function, Inst, Opcode, Type}; use scoped_hash_map::ScopedHashMap; use timing; -#[cfg(feature = "no_std")] -use alloc::Vec; +use std::vec::Vec; /// Test whether the given opcode is unsafe to even consider for GVN. fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { diff --git a/lib/cretonne/src/topo_order.rs b/lib/cretonne/src/topo_order.rs index 660179f1f9..f2c57f8166 100644 --- a/lib/cretonne/src/topo_order.rs +++ b/lib/cretonne/src/topo_order.rs @@ -4,8 +4,7 @@ use entity::SparseSet; use dominator_tree::DominatorTree; use ir::{Ebb, Layout}; -#[cfg(feature = "no_std")] -use alloc::Vec; +use std::vec::Vec; /// Present EBBs in a topological order such that all dominating EBBs are guaranteed to be visited /// before the current EBB. diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 37a3504bf5..84af6f976b 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -161,4 +161,4 @@ mod ssa; mod std { pub use alloc::vec; pub use core::*; -} \ No newline at end of file +} diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index ea33f311a9..4a66bce463 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -34,7 +34,6 @@ use std::collections::{HashMap, hash_map}; use environ::{FuncEnvironment, GlobalValue}; use std::{i32, u32}; -// this is for no_std builds, but has no affect on regular builds use std::vec::Vec; /// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 9238cbf665..d0dba2c817 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -11,7 +11,6 @@ use cretonne::settings; use wasmparser; use std::error::Error; -// this is for no_std builds, but has no affect on regular builds use std::vec::Vec; use std::string::String; diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index c8a8f54111..dfc2685f33 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -6,7 +6,6 @@ use cretonne::settings::Flags; use translation_utils::{SignatureIndex, FunctionIndex, TableIndex, GlobalIndex, MemoryIndex, Global, Table, Memory}; -// this is for no_std builds, but has no affect on regular builds use std::vec::Vec; use std::string::String; diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index dde500af97..9005e6cd42 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -53,4 +53,4 @@ mod std { pub mod error { pub use error_core::Error; } -} \ No newline at end of file +} diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 2786f9094e..43a47b68af 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -8,7 +8,6 @@ use sections_translator::{SectionParsingError, parse_function_signatures, parse_ parse_elements_section, parse_data_section}; use environ::ModuleEnvironment; -// this is for no_std builds, but has no affect on regular builds use std::string::String; /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IL diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index ff641a6889..a1e172aa7c 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -17,7 +17,6 @@ use wasmparser; use std::str::from_utf8; use environ::ModuleEnvironment; -// this is for no_std builds, but has no affect on regular builds use std::vec::Vec; use std::string::String; diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index 625e39b4e9..cb8aebb653 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -8,7 +8,6 @@ use environ::{FuncEnvironment, GlobalValue}; use std::collections::HashMap; use translation_utils::{GlobalIndex, MemoryIndex, SignatureIndex, FunctionIndex}; -// this is for no_std builds, but has no affect on regular builds use std::vec::Vec; /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following From 6bd6a80b785acda493087f8323b29c194e714d46 Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Mon, 15 Jan 2018 18:32:27 -0500 Subject: [PATCH 1531/3084] Updated top-level README.rst --- README.rst | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/README.rst b/README.rst index 05ddeb6b70..4672c092d4 100644 --- a/README.rst +++ b/README.rst @@ -57,6 +57,37 @@ You may need to install the *wat2wasm* tool from the `wabt WebAssembly tests. Tests requiring wat2wasm are ignored if the tool is not installed. +Building with `no_std` +---------------------- + +To build cretonne without libstd, enable the `no_std` feature on `lib/cretonne`, +`lib/frontend`, `lib/native`, and `lib/wasm`. + +For example, to build `cretonne`: + + cd lib/cretonne + cargo build --features no_std + +Or, when using `cretonne` as a dependency (in Cargo.toml): + + [dependency.cretonne] + path = "..." + features = ["no_std"] + +Currently, tests don't test the `no_std` feature. + +It's important to note that cretonne still needs liballoc to compile. +Thus, whatever environment is used must implement an allocator. + +Also, to allow the use of HashMaps in `no_std` mode, an external crate +called `hashmap_core` is pulled in (only in `no_std` builds). This +is mostly the same as `std::collections::HashMap`, except that it doesn't +have DOS protection. Just something to think about. + +Lastly, to support `std::error`, which isn't is `std` or `alloc` for +an inexplicable reason, the `error_core` crate is also used in `no_std` builds. +You might need it, as well, when interfacing with `CtonError`. + Building the documentation -------------------------- From 5590abcfd95671baa9109662fbae5606826de47d Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Mon, 15 Jan 2018 18:48:32 -0500 Subject: [PATCH 1532/3084] Expanded on `no_std` in README.rst --- README.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 4672c092d4..71d4b3902e 100644 --- a/README.rst +++ b/README.rst @@ -74,7 +74,11 @@ Or, when using `cretonne` as a dependency (in Cargo.toml): path = "..." features = ["no_std"] -Currently, tests don't test the `no_std` feature. +Currently, tests don't test the `no_std` feature: + +1. `cargo test --features no_std` won't compile. + +1. `./test_all.sh` doesn't test the `no_std` feature. It's important to note that cretonne still needs liballoc to compile. Thus, whatever environment is used must implement an allocator. From ddfa88c8ba909cf78d007c2d92b05ad59605a2e2 Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Mon, 15 Jan 2018 19:00:40 -0500 Subject: [PATCH 1533/3084] Removed extraneous newlines --- lib/cretonne/src/dominator_tree.rs | 1 - lib/cretonne/src/isa/arm32/mod.rs | 1 - lib/cretonne/src/isa/arm64/mod.rs | 1 - lib/cretonne/src/isa/intel/mod.rs | 1 - lib/cretonne/src/isa/mod.rs | 1 - lib/cretonne/src/isa/riscv/mod.rs | 1 - lib/cretonne/src/simple_gvn.rs | 1 - lib/cretonne/src/topo_order.rs | 1 - lib/wasm/src/code_translator.rs | 1 - lib/wasm/src/environ/dummy.rs | 1 - lib/wasm/src/environ/spec.rs | 1 - lib/wasm/src/sections_translator.rs | 1 - lib/wasm/src/state.rs | 1 - 13 files changed, 13 deletions(-) diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index a7625f840e..6617085834 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -8,7 +8,6 @@ use packed_option::PackedOption; use std::cmp; use std::mem; use timing; - use std::cmp::Ordering; use std::vec::Vec; diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 50ef3ea7db..02e0d5f1af 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -14,7 +14,6 @@ use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; use std::fmt; - use std::boxed::Box; #[allow(dead_code)] diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index b69e044bf4..8239cb01e9 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -14,7 +14,6 @@ use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; use std::fmt; - use std::boxed::Box; #[allow(dead_code)] diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index e4841558e4..578dac2d16 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -16,7 +16,6 @@ use regalloc; use result; use timing; use std::fmt; - use std::boxed::Box; #[allow(dead_code)] diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index cedfdaa301..7e10f12620 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -54,7 +54,6 @@ use result; use timing; use isa::enc_tables::Encodings; use std::fmt; - use std::boxed::Box; #[cfg(build_riscv)] diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 1993ecb9d6..8efae74af7 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -14,7 +14,6 @@ use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; use std::fmt; - use std::boxed::Box; #[allow(dead_code)] diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index 8f09398c82..dd0fb3b1aa 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -6,7 +6,6 @@ use dominator_tree::DominatorTree; use ir::{InstructionData, Function, Inst, Opcode, Type}; use scoped_hash_map::ScopedHashMap; use timing; - use std::vec::Vec; /// Test whether the given opcode is unsafe to even consider for GVN. diff --git a/lib/cretonne/src/topo_order.rs b/lib/cretonne/src/topo_order.rs index f2c57f8166..373ab69e0d 100644 --- a/lib/cretonne/src/topo_order.rs +++ b/lib/cretonne/src/topo_order.rs @@ -3,7 +3,6 @@ use entity::SparseSet; use dominator_tree::DominatorTree; use ir::{Ebb, Layout}; - use std::vec::Vec; /// Present EBBs in a topological order such that all dominating EBBs are guaranteed to be visited diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 4a66bce463..c62e1fe76a 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -33,7 +33,6 @@ use state::{TranslationState, ControlStackFrame}; use std::collections::{HashMap, hash_map}; use environ::{FuncEnvironment, GlobalValue}; use std::{i32, u32}; - use std::vec::Vec; /// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index d0dba2c817..de941f3b9a 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -10,7 +10,6 @@ use cretonne::cursor::FuncCursor; use cretonne::settings; use wasmparser; use std::error::Error; - use std::vec::Vec; use std::string::String; diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index dfc2685f33..7872bdb0ea 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -5,7 +5,6 @@ use cretonne::cursor::FuncCursor; use cretonne::settings::Flags; use translation_utils::{SignatureIndex, FunctionIndex, TableIndex, GlobalIndex, MemoryIndex, Global, Table, Memory}; - use std::vec::Vec; use std::string::String; diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index a1e172aa7c..a12a92e100 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -16,7 +16,6 @@ use wasmparser::{Parser, ParserState, FuncType, ImportSectionEntryType, External use wasmparser; use std::str::from_utf8; use environ::ModuleEnvironment; - use std::vec::Vec; use std::string::String; diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index cb8aebb653..40de47d7c6 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -7,7 +7,6 @@ use cretonne::ir::{self, Ebb, Inst, Value}; use environ::{FuncEnvironment, GlobalValue}; use std::collections::HashMap; use translation_utils::{GlobalIndex, MemoryIndex, SignatureIndex, FunctionIndex}; - use std::vec::Vec; /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following From 4cdbf2f56e134a52cd6be3fceff8d283547cbcdb Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Mon, 15 Jan 2018 19:11:52 -0500 Subject: [PATCH 1534/3084] Removed unused prelude --- lib/cretonne/src/lib.rs | 3 --- 1 file changed, 3 deletions(-) diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 22914cc17b..8302095998 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -65,9 +65,6 @@ mod std { pub use core::*; #[macro_use] pub use alloc::{boxed, vec, string}; - pub mod prelude { - pub use core::prelude as v1; - } pub mod collections { pub use hashmap_core::{HashMap, HashSet}; pub use hashmap_core::map as hash_map; From 48229f02e3b9712f10f128bfc548dbf22b6e90a7 Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Mon, 15 Jan 2018 19:12:06 -0500 Subject: [PATCH 1535/3084] Fixed typo in README.rst --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 71d4b3902e..8c685ac1ae 100644 --- a/README.rst +++ b/README.rst @@ -78,7 +78,7 @@ Currently, tests don't test the `no_std` feature: 1. `cargo test --features no_std` won't compile. -1. `./test_all.sh` doesn't test the `no_std` feature. +1. `./test-all.sh` doesn't test the `no_std` feature. It's important to note that cretonne still needs liballoc to compile. Thus, whatever environment is used must implement an allocator. From d8c8e4af3d4c5d74c5d7dd463a1d678f046c78d0 Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Mon, 15 Jan 2018 20:50:19 -0500 Subject: [PATCH 1536/3084] 2 is after 1 --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 8c685ac1ae..e9c9b568d9 100644 --- a/README.rst +++ b/README.rst @@ -78,7 +78,7 @@ Currently, tests don't test the `no_std` feature: 1. `cargo test --features no_std` won't compile. -1. `./test-all.sh` doesn't test the `no_std` feature. +2. `./test-all.sh` doesn't test the `no_std` feature. It's important to note that cretonne still needs liballoc to compile. Thus, whatever environment is used must implement an allocator. From 023f8d798013db09c7df804f21ca0b644b4a5481 Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Thu, 18 Jan 2018 21:42:58 -0500 Subject: [PATCH 1537/3084] Fixed missing vec --- lib/cretonne/src/regalloc/virtregs.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index 5116e8f02a..df2dbca78e 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -21,6 +21,7 @@ use packed_option::PackedOption; use ref_slice::ref_slice; use std::cmp::Ordering; use std::fmt; +use std::vec::Vec; /// A virtual register reference. #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] From 61db54c44705488188340e97a0d3f8e785d6e767 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 18 Jan 2018 22:05:51 -0800 Subject: [PATCH 1538/3084] Add support for running tests in `no_std` mode. --- README.rst | 10 ++++--- cranelift/test-no_std.sh | 28 ++++++++++++++++++++ lib/cretonne/src/bforest/map.rs | 7 +++-- lib/cretonne/src/bforest/node.rs | 1 + lib/cretonne/src/bforest/pool.rs | 1 + lib/cretonne/src/bforest/set.rs | 4 ++- lib/cretonne/src/dbg.rs | 2 ++ lib/cretonne/src/flowgraph.rs | 1 + lib/cretonne/src/ir/condcodes.rs | 1 + lib/cretonne/src/ir/dfg.rs | 1 + lib/cretonne/src/ir/entities.rs | 1 + lib/cretonne/src/ir/extfunc.rs | 1 + lib/cretonne/src/ir/extname.rs | 1 + lib/cretonne/src/ir/immediates.rs | 1 + lib/cretonne/src/ir/instructions.rs | 1 + lib/cretonne/src/ir/jumptable.rs | 2 ++ lib/cretonne/src/ir/layout.rs | 1 + lib/cretonne/src/ir/libcall.rs | 1 + lib/cretonne/src/ir/progpoint.rs | 1 + lib/cretonne/src/ir/sourceloc.rs | 1 + lib/cretonne/src/ir/stackslot.rs | 1 + lib/cretonne/src/ir/trapcode.rs | 1 + lib/cretonne/src/ir/types.rs | 1 + lib/cretonne/src/isa/arm32/registers.rs | 1 + lib/cretonne/src/isa/arm64/registers.rs | 1 + lib/cretonne/src/isa/intel/registers.rs | 1 + lib/cretonne/src/isa/riscv/mod.rs | 1 + lib/cretonne/src/isa/riscv/registers.rs | 1 + lib/cretonne/src/isa/riscv/settings.rs | 1 + lib/cretonne/src/iterators.rs | 2 ++ lib/cretonne/src/lib.rs | 1 - lib/cretonne/src/loop_analysis.rs | 1 + lib/cretonne/src/partition_slice.rs | 1 + lib/cretonne/src/regalloc/allocatable_set.rs | 1 + lib/cretonne/src/regalloc/coalescing.rs | 2 -- lib/cretonne/src/regalloc/liverange.rs | 1 + lib/cretonne/src/regalloc/pressure.rs | 1 + lib/cretonne/src/regalloc/solver.rs | 1 + lib/cretonne/src/settings.rs | 1 + lib/cretonne/src/timing.rs | 1 + lib/cretonne/src/verifier/mod.rs | 3 +++ lib/cretonne/src/write.rs | 1 + lib/frontend/src/ssa.rs | 21 ++++++++++++--- 43 files changed, 101 insertions(+), 13 deletions(-) create mode 100755 cranelift/test-no_std.sh diff --git a/README.rst b/README.rst index e9c9b568d9..690472578b 100644 --- a/README.rst +++ b/README.rst @@ -74,11 +74,13 @@ Or, when using `cretonne` as a dependency (in Cargo.toml): path = "..." features = ["no_std"] -Currently, tests don't test the `no_std` feature: +`no_std` is currently "best effort". We won't try to break it, and we'll +accept patches fixing problems, however we don't expect all developers to +build and test with `no_std` when submitting patches. Accordingly, the +`./test-all.sh` script does not test `no_std`. -1. `cargo test --features no_std` won't compile. - -2. `./test-all.sh` doesn't test the `no_std` feature. +There is a separate `./test-no_std.sh` script that tests the `no_std` +feature in packages which support it. It's important to note that cretonne still needs liballoc to compile. Thus, whatever environment is used must implement an allocator. diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh new file mode 100755 index 0000000000..c9d3fe38f7 --- /dev/null +++ b/cranelift/test-no_std.sh @@ -0,0 +1,28 @@ +#!/bin/bash + +# This is the test script for testing the no_std configuration of +# packages which support it. + +# Exit immediately on errors. +set -e + +# Repository top-level directory. +cd $(dirname "$0") +topdir=$(pwd) + +function banner() { + echo "====== $@ ======" +} + +# Test those packages which have no_std support. +LIBS="cretonne frontend wasm native" +cd "$topdir" +for LIB in $LIBS +do + banner "Rust unit tests in $LIB" + cd "lib/$LIB" + cargo test --features no_std + cd "$topdir" +done + +banner "OK" diff --git a/lib/cretonne/src/bforest/map.rs b/lib/cretonne/src/bforest/map.rs index 5aa19bceda..073f27529b 100644 --- a/lib/cretonne/src/bforest/map.rs +++ b/lib/cretonne/src/bforest/map.rs @@ -222,7 +222,8 @@ where } /// Get a text version of the path to `key`. - fn tpath(&self, key: K, forest: &MapForest, comp: &C) -> String { + fn tpath(&self, key: K, forest: &MapForest, comp: &C) -> ::std::string::String { + use std::string::ToString; match self.root.expand() { None => "map(empty)".to_string(), Some(root) => { @@ -415,7 +416,8 @@ where } /// Get a text version of the path to the current position. - fn tpath(&self) -> String { + fn tpath(&self) -> ::std::string::String { + use std::string::ToString; self.path.to_string() } } @@ -423,6 +425,7 @@ where #[cfg(test)] mod test { use std::mem; + use std::vec::Vec; use super::*; use super::super::NodeData; diff --git a/lib/cretonne/src/bforest/node.rs b/lib/cretonne/src/bforest/node.rs index 53a96f9b14..779554f0c8 100644 --- a/lib/cretonne/src/bforest/node.rs +++ b/lib/cretonne/src/bforest/node.rs @@ -580,6 +580,7 @@ where #[cfg(test)] mod test { use std::mem; + use std::string::ToString; use super::*; // Forest impl for a set implementation. diff --git a/lib/cretonne/src/bforest/pool.rs b/lib/cretonne/src/bforest/pool.rs index 37801ee4ab..18c455fdd1 100644 --- a/lib/cretonne/src/bforest/pool.rs +++ b/lib/cretonne/src/bforest/pool.rs @@ -78,6 +78,7 @@ impl NodePool { { use std::borrow::Borrow; use std::cmp::Ordering; + use std::vec::Vec; use super::Comparator; use entity::SparseSet; diff --git a/lib/cretonne/src/bforest/set.rs b/lib/cretonne/src/bforest/set.rs index 6992265773..22f5122c7c 100644 --- a/lib/cretonne/src/bforest/set.rs +++ b/lib/cretonne/src/bforest/set.rs @@ -314,7 +314,8 @@ where } /// Get a text version of the path to the current position. - fn tpath(&self) -> String { + fn tpath(&self) -> ::std::string::String { + use std::string::ToString; self.path.to_string() } } @@ -351,6 +352,7 @@ where #[cfg(test)] mod test { use std::mem; + use std::vec::Vec; use super::*; use super::super::NodeData; diff --git a/lib/cretonne/src/dbg.rs b/lib/cretonne/src/dbg.rs index ace22b6c47..8f949f00e1 100644 --- a/lib/cretonne/src/dbg.rs +++ b/lib/cretonne/src/dbg.rs @@ -19,10 +19,12 @@ use std::fmt; use std::fs::File; #[cfg(not(feature = "no_std"))] use std::io::{self, Write}; +#[cfg(not(feature = "no_std"))] use std::sync::atomic; #[cfg(not(feature = "no_std"))] use std::thread; +#[cfg(not(feature = "no_std"))] static STATE: atomic::AtomicIsize = atomic::ATOMIC_ISIZE_INIT; /// Is debug tracing enabled? diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index d1e45312ac..9c0fa09f26 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -204,6 +204,7 @@ mod tests { use super::*; use cursor::{Cursor, FuncCursor}; use ir::{Function, InstBuilder, types}; + use std::vec::Vec; #[test] fn empty() { diff --git a/lib/cretonne/src/ir/condcodes.rs b/lib/cretonne/src/ir/condcodes.rs index 4bbf82cb6f..11c438b10a 100644 --- a/lib/cretonne/src/ir/condcodes.rs +++ b/lib/cretonne/src/ir/condcodes.rs @@ -266,6 +266,7 @@ impl FromStr for FloatCC { #[cfg(test)] mod tests { use super::*; + use std::string::ToString; static INT_ALL: [IntCC; 10] = [ IntCC::Equal, diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 023d8751b6..2f1cc8763b 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -944,6 +944,7 @@ mod tests { use cursor::{Cursor, FuncCursor}; use ir::types; use ir::{Function, Opcode, InstructionData, TrapCode}; + use std::string::ToString; #[test] fn make_inst() { diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index a9e6705d98..1a4e772be4 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -262,6 +262,7 @@ impl From for AnyEntity { mod tests { use super::*; use std::u32; + use std::string::ToString; #[test] fn value_with_number() { diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 0aa87fa60e..f8fe767970 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -379,6 +379,7 @@ impl FromStr for CallConv { mod tests { use super::*; use ir::types::{I32, F32, B8}; + use std::string::ToString; #[test] fn argument_type() { diff --git a/lib/cretonne/src/ir/extname.rs b/lib/cretonne/src/ir/extname.rs index f58f8d24ec..b7bcc7d94f 100644 --- a/lib/cretonne/src/ir/extname.rs +++ b/lib/cretonne/src/ir/extname.rs @@ -122,6 +122,7 @@ impl FromStr for ExternalName { mod tests { use super::ExternalName; use ir::LibCall; + use std::string::ToString; #[test] fn display_testcase() { diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 69607754a2..0dca3144cb 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -630,6 +630,7 @@ mod tests { use std::{f32, f64}; use std::str::FromStr; use std::fmt::Display; + use std::string::ToString; #[test] fn format_imm64() { diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 1340c14055..22800ac634 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -724,6 +724,7 @@ pub enum ResolvedConstraint { #[cfg(test)] mod tests { use super::*; + use std::string::ToString; #[test] fn opcodes() { diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index 4f6e117b64..1c7bf79157 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -142,6 +142,8 @@ mod tests { use super::JumpTableData; use ir::Ebb; use entity::EntityRef; + use std::vec::Vec; + use std::string::ToString; #[test] fn empty() { diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index e18ffd7a33..1fc1c048fb 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -743,6 +743,7 @@ mod tests { use entity::EntityRef; use ir::{Ebb, Inst, ProgramOrder, SourceLoc}; use std::cmp::Ordering; + use std::vec::Vec; struct LayoutCursor<'f> { /// Borrowed function layout. Public so it can be re-borrowed from this cursor. diff --git a/lib/cretonne/src/ir/libcall.rs b/lib/cretonne/src/ir/libcall.rs index e8c848f71d..85133611be 100644 --- a/lib/cretonne/src/ir/libcall.rs +++ b/lib/cretonne/src/ir/libcall.rs @@ -100,6 +100,7 @@ impl LibCall { #[cfg(test)] mod test { use super::*; + use std::string::ToString; #[test] fn display() { diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index 480160ea96..72a00602ee 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -148,6 +148,7 @@ mod tests { use super::*; use entity::EntityRef; use ir::{Inst, Ebb}; + use std::string::ToString; #[test] fn convert() { diff --git a/lib/cretonne/src/ir/sourceloc.rs b/lib/cretonne/src/ir/sourceloc.rs index ffcf0db943..36e5247488 100644 --- a/lib/cretonne/src/ir/sourceloc.rs +++ b/lib/cretonne/src/ir/sourceloc.rs @@ -51,6 +51,7 @@ impl fmt::Display for SourceLoc { #[cfg(test)] mod tests { use ir::SourceLoc; + use std::string::ToString; #[test] fn display() { diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index 744bd06ac3..7baadde636 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -320,6 +320,7 @@ mod tests { use ir::Function; use ir::types; use super::*; + use std::string::ToString; #[test] fn stack_slot() { diff --git a/lib/cretonne/src/ir/trapcode.rs b/lib/cretonne/src/ir/trapcode.rs index 2374c206f3..1487a63aad 100644 --- a/lib/cretonne/src/ir/trapcode.rs +++ b/lib/cretonne/src/ir/trapcode.rs @@ -83,6 +83,7 @@ impl FromStr for TrapCode { #[cfg(test)] mod tests { use super::*; + use std::string::ToString; // Everything but user-defined codes. const CODES: [TrapCode; 8] = [ diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index f8774250ec..07cf97aefa 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -324,6 +324,7 @@ impl Default for Type { #[cfg(test)] mod tests { use super::*; + use std::string::ToString; #[test] fn basic_scalars() { diff --git a/lib/cretonne/src/isa/arm32/registers.rs b/lib/cretonne/src/isa/arm32/registers.rs index 7c6ac406e1..e2c4813bdd 100644 --- a/lib/cretonne/src/isa/arm32/registers.rs +++ b/lib/cretonne/src/isa/arm32/registers.rs @@ -8,6 +8,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-arm32.rs")); mod tests { use super::{INFO, GPR, S, D}; use isa::RegUnit; + use std::string::{String, ToString}; #[test] fn unit_encodings() { diff --git a/lib/cretonne/src/isa/arm64/registers.rs b/lib/cretonne/src/isa/arm64/registers.rs index 62311aaebe..2c1d85e091 100644 --- a/lib/cretonne/src/isa/arm64/registers.rs +++ b/lib/cretonne/src/isa/arm64/registers.rs @@ -8,6 +8,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-arm64.rs")); mod tests { use super::INFO; use isa::RegUnit; + use std::string::{String, ToString}; #[test] fn unit_encodings() { diff --git a/lib/cretonne/src/isa/intel/registers.rs b/lib/cretonne/src/isa/intel/registers.rs index c2b96f77d3..c972c10a13 100644 --- a/lib/cretonne/src/isa/intel/registers.rs +++ b/lib/cretonne/src/isa/intel/registers.rs @@ -8,6 +8,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-intel.rs")); mod tests { use super::*; use isa::RegUnit; + use std::string::{String, ToString}; #[test] fn unit_encodings() { diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 8efae74af7..6fc976686b 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -117,6 +117,7 @@ mod tests { use isa; use ir::{DataFlowGraph, InstructionData, Opcode}; use ir::{types, immediates}; + use std::string::{String, ToString}; fn encstr(isa: &isa::TargetIsa, enc: Result) -> String { match enc { diff --git a/lib/cretonne/src/isa/riscv/registers.rs b/lib/cretonne/src/isa/riscv/registers.rs index 3cce8c4988..e2073899b6 100644 --- a/lib/cretonne/src/isa/riscv/registers.rs +++ b/lib/cretonne/src/isa/riscv/registers.rs @@ -8,6 +8,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-riscv.rs")); mod tests { use super::{INFO, GPR, FPR}; use isa::RegUnit; + use std::string::{String, ToString}; #[test] fn unit_encodings() { diff --git a/lib/cretonne/src/isa/riscv/settings.rs b/lib/cretonne/src/isa/riscv/settings.rs index 8cb376e5da..7b609acf20 100644 --- a/lib/cretonne/src/isa/riscv/settings.rs +++ b/lib/cretonne/src/isa/riscv/settings.rs @@ -12,6 +12,7 @@ include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs")); mod tests { use super::{builder, Flags}; use settings::{self, Configurable}; + use std::string::ToString; #[test] fn display_default() { diff --git a/lib/cretonne/src/iterators.rs b/lib/cretonne/src/iterators.rs index 08866717e8..0524343028 100644 --- a/lib/cretonne/src/iterators.rs +++ b/lib/cretonne/src/iterators.rs @@ -48,6 +48,8 @@ where #[cfg(test)] mod tests { + use std::vec::Vec; + #[test] fn adjpairs() { use super::IteratorExtras; diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 8302095998..95868a6933 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -63,7 +63,6 @@ mod write; #[cfg(feature = "no_std")] mod std { pub use core::*; - #[macro_use] pub use alloc::{boxed, vec, string}; pub mod collections { pub use hashmap_core::{HashMap, HashSet}; diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index dfd5cf135a..431496b6e4 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -231,6 +231,7 @@ mod test { use loop_analysis::{Loop, LoopAnalysis}; use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; + use std::vec::Vec; #[test] fn nested_loops_detection() { diff --git a/lib/cretonne/src/partition_slice.rs b/lib/cretonne/src/partition_slice.rs index 6f106e5bfc..7a94a9fe0d 100644 --- a/lib/cretonne/src/partition_slice.rs +++ b/lib/cretonne/src/partition_slice.rs @@ -33,6 +33,7 @@ where #[cfg(test)] mod tests { use super::partition_slice; + use std::vec::Vec; fn check(x: &[u32], want: &[u32]) { assert_eq!(x.len(), want.len()); diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index cc72066b5d..e0abf76d37 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -221,6 +221,7 @@ impl fmt::Display for AllocatableSet { mod tests { use super::*; use isa::registers::{RegClass, RegClassData}; + use std::vec::Vec; // Register classes for testing. const GPR: RegClass = &RegClassData { diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 225aa8e9d8..5933170e33 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -18,8 +18,6 @@ use std::cmp; use std::iter; use std::fmt; use std::slice; -use std::iter::Peekable; -use std::mem; use std::vec::Vec; use isa::{TargetIsa, EncInfo}; use timing; diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 7d7d6a8775..93d5b0c244 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -463,6 +463,7 @@ mod tests { use entity::EntityRef; use ir::{ProgramOrder, ExpandedProgramPoint}; use std::cmp::Ordering; + use std::vec::Vec; // Dummy program order which simply compares indexes. // It is assumed that EBBs have indexes that are multiples of 10, and instructions have indexes diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index c71037efdd..67e0e99b70 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -273,6 +273,7 @@ mod tests { use regalloc::AllocatableSet; use std::borrow::Borrow; use super::Pressure; + use std::boxed::Box; // Make an arm32 `TargetIsa`, if possible. fn arm32() -> Option> { diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index c2f8017c3a..db5efcab06 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -1163,6 +1163,7 @@ mod tests { use isa::{TargetIsa, RegClass, RegUnit, RegInfo}; use regalloc::AllocatableSet; use super::{Solver, Move}; + use std::boxed::Box; // Make an arm32 `TargetIsa`, if possible. fn arm32() -> Option> { diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index a7da8d6f17..b6ef9eae30 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -348,6 +348,7 @@ mod tests { use super::{builder, Flags}; use super::Error::*; use super::Configurable; + use std::string::ToString; #[test] fn display_default() { diff --git a/lib/cretonne/src/timing.rs b/lib/cretonne/src/timing.rs index c05ee93004..15496ed9ce 100644 --- a/lib/cretonne/src/timing.rs +++ b/lib/cretonne/src/timing.rs @@ -238,6 +238,7 @@ mod details { #[cfg(test)] mod test { use super::*; + use std::string::ToString; #[test] fn display() { diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index c3bebc5ae6..117de2f38b 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -1132,7 +1132,10 @@ mod tests { Ok(_) => { panic!("Expected an error!") }, Err(Error { message, .. } ) => { if !message.contains($msg) { + #[cfg(not(feature = "no_std"))] panic!(format!("'{}' did not contain the substring '{}'", message, $msg)); + #[cfg(feature = "no_std")] + panic!("error message did not contain the expected substring"); } } } diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 4ca8ddaadb..ff60033af2 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -467,6 +467,7 @@ impl<'a> fmt::Display for DisplayValues<'a> { mod tests { use ir::{Function, ExternalName, StackSlotData, StackSlotKind}; use ir::types; + use std::string::ToString; #[test] fn basic() { diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index f99ff5e179..fc2ebcca08 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -1039,7 +1039,12 @@ mod tests { let flags = settings::Flags::new(&settings::builder()); match verify_function(&func, &flags) { Ok(()) => {} - Err(err) => panic!(err.message), + Err(err) => { + #[cfg(not(feature = "no_std"))] + panic!(err.message); + #[cfg(feature = "no_std")] + panic!("function failed to verify"); + } } } @@ -1213,7 +1218,12 @@ mod tests { let flags = settings::Flags::new(&settings::builder()); match verify_function(&func, &flags) { Ok(()) => {} - Err(err) => panic!(err.message), + Err(err) => { + #[cfg(not(feature = "no_std"))] + panic!(err.message); + #[cfg(feature = "no_std")] + panic!("function failed to verify"); + } } } @@ -1259,7 +1269,12 @@ mod tests { let flags = settings::Flags::new(&settings::builder()); match verify_function(&func, &flags) { Ok(()) => {} - Err(err) => panic!(err.message), + Err(err) => { + #[cfg(not(feature = "no_std"))] + panic!(err.message); + #[cfg(feature = "no_std")] + panic!("function failed to verify"); + } } } } From e37f45667fc37c0813def2359acd2a54cc6744f6 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 13 Feb 2018 17:48:04 -0800 Subject: [PATCH 1539/3084] Add an explicit `std` feature so that features are purely additive. --- README.rst | 10 ++++++---- cranelift/test-no_std.sh | 1 + lib/cretonne/Cargo.toml | 11 ++++++++--- lib/cretonne/src/lib.rs | 8 ++++---- lib/frontend/Cargo.toml | 4 +++- lib/frontend/src/lib.rs | 8 ++++---- lib/native/Cargo.toml | 4 +++- lib/native/src/lib.rs | 3 ++- lib/wasm/Cargo.toml | 11 ++++++----- lib/wasm/src/lib.rs | 8 ++++---- 10 files changed, 41 insertions(+), 27 deletions(-) diff --git a/README.rst b/README.rst index 690472578b..caa24342d9 100644 --- a/README.rst +++ b/README.rst @@ -60,18 +60,20 @@ installed. Building with `no_std` ---------------------- -To build cretonne without libstd, enable the `no_std` feature on `lib/cretonne`, -`lib/frontend`, `lib/native`, and `lib/wasm`. +To build cretonne without libstd, disable the `std` feature on `lib/cretonne`, +`lib/frontend`, `lib/native`, and `lib/wasm`, which is otherwise enabled by +default, and enable the `no_std` feature. For example, to build `cretonne`: cd lib/cretonne - cargo build --features no_std + cargo build --no-default-features --features no_std Or, when using `cretonne` as a dependency (in Cargo.toml): [dependency.cretonne] - path = "..." + ... + default-features = false features = ["no_std"] `no_std` is currently "best effort". We won't try to break it, and we'll diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh index c9d3fe38f7..d3edbb921f 100755 --- a/cranelift/test-no_std.sh +++ b/cranelift/test-no_std.sh @@ -21,6 +21,7 @@ for LIB in $LIBS do banner "Rust unit tests in $LIB" cd "lib/$LIB" + cargo test --no-default-features --features no_std cargo test --features no_std cd "$topdir" done diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 0cf3fffd9b..c1c666fa77 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -17,14 +17,19 @@ name = "cretonne" # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be # accomodated in `tests`. + [dependencies.hashmap_core] version = "0.1.1" optional = true + [dependencies.error_core] version = "0.1.0" optional = true [features] -# Currently, the only feature is the `no_std` feature. -# Enabling this disables use of `stdlib`. -no_std = ["hashmap_core", "error_core"] \ No newline at end of file +# The "std" feature enables use of libstd. The "no_std" feature enables use +# of some minimal std-like replacement libraries. At least one of these two +# features to be enabled. +default = ["std"] +std = [] +no_std = ["hashmap_core", "error_core"] diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 95868a6933..0952969bb1 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -1,16 +1,16 @@ //! Cretonne code generation library. -#![cfg_attr(feature = "no_std", no_std)] #![deny(missing_docs)] // Turns on alloc feature if no_std -#![cfg_attr(feature = "no_std", feature(alloc))] +#![cfg_attr(not(feature = "std"), no_std)] +#![cfg_attr(not(feature = "std"), feature(alloc))] // Include the `hashmap_core` crate if no_std #[cfg(feature = "no_std")] extern crate hashmap_core; #[cfg(feature = "no_std")] extern crate error_core; -#[cfg(feature = "no_std")] +#[cfg(not(feature = "std"))] #[macro_use] extern crate alloc; @@ -60,7 +60,7 @@ mod unreachable_code; mod write; /// This replaces `std` in builds with no_std. -#[cfg(feature = "no_std")] +#[cfg(not(feature = "std"))] mod std { pub use core::*; pub use alloc::{boxed, vec, string}; diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 32c65e4ded..bccc84f6bc 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -12,7 +12,9 @@ readme = "README.md" name = "cton_frontend" [dependencies] -cretonne = { path = "../cretonne", version = "0.1.0" } +cretonne = { path = "../cretonne", version = "0.1.0", default-features = false } [features] +default = ["std"] +std = ["cretonne/std"] no_std = ["cretonne/no_std"] diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 84af6f976b..52765b5a20 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -142,14 +142,14 @@ //! } //! ``` -#![cfg_attr(feature = "no_std", no_std)] #![deny(missing_docs)] -#![cfg_attr(feature = "no_std", feature(alloc))] +#![cfg_attr(not(feature = "std"), no_std)] +#![cfg_attr(not(feature = "std"), feature(alloc))] extern crate cretonne; -#[cfg(feature = "no_std")] +#[cfg(not(feature = "std"))] extern crate alloc; pub use frontend::{ILBuilder, FunctionBuilder}; @@ -157,7 +157,7 @@ pub use frontend::{ILBuilder, FunctionBuilder}; mod frontend; mod ssa; -#[cfg(feature = "no_std")] +#[cfg(not(feature = "std"))] mod std { pub use alloc::vec; pub use core::*; diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 57a2d0bc3c..32417e8718 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -11,10 +11,12 @@ readme = "README.md" name = "cton_native" [dependencies] -cretonne = { path = "../cretonne", version = "0.1.0" } +cretonne = { path = "../cretonne", version = "0.1.0", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" [features] +default = ["std"] +std = ["cretonne/std"] no_std = ["cretonne/no_std"] diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index 284366a65c..514e6df314 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -1,8 +1,9 @@ //! Performs autodetection of the host for the purposes of running //! Cretonne to generate code to run on the same machine. -#![cfg_attr(feature = "no_std", no_std)] #![deny(missing_docs)] +#![cfg_attr(not(feature = "std"), no_std)] + extern crate cretonne; #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 448166d568..3bd2b36aa8 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -12,12 +12,13 @@ name = "cton_wasm" [dependencies] wasmparser = "0.14.1" -cretonne = { path = "../cretonne", version = "0.1.0" } -cretonne-frontend = { path = "../frontend", version = "0.1.0" } +cretonne = { path = "../cretonne", version = "0.1.0", default_features = false } +cretonne-frontend = { path = "../frontend", version = "0.1.0", default_features = false } [dependencies.hashmap_core] version = "0.1.1" optional = true + [dependencies.error_core] version = "0.1.0" optional = true @@ -26,6 +27,6 @@ optional = true tempdir = "0.3.5" [features] -# Currently, the only feature is the `no_std` feature. -# Enabling this disables use of `stdlib`. -no_std = ["hashmap_core", "error_core", "cretonne/no_std", "cretonne-frontend/no_std"] \ No newline at end of file +default = ["std"] +std = ["cretonne/std", "cretonne-frontend/std"] +no_std = ["hashmap_core", "error_core", "cretonne/no_std", "cretonne-frontend/no_std"] diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 9005e6cd42..76e69887d9 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -9,12 +9,12 @@ //! //! The main function of this module is [`translate_module`](fn.translate_module.html). -#![cfg_attr(feature = "no_std", no_std)] #![deny(missing_docs)] -#![cfg_attr(feature = "no_std", feature(alloc))] +#![cfg_attr(not(feature = "std"), no_std)] +#![cfg_attr(not(feature = "std"), feature(alloc))] -#[cfg(feature = "no_std")] +#[cfg(not(feature = "std"))] #[macro_use] extern crate alloc; @@ -42,7 +42,7 @@ pub use environ::{FuncEnvironment, ModuleEnvironment, DummyEnvironment, GlobalVa pub use translation_utils::{FunctionIndex, GlobalIndex, TableIndex, MemoryIndex, SignatureIndex, Global, GlobalInit, Table, Memory}; -#[cfg(feature = "no_std")] +#[cfg(not(feature = "std"))] mod std { pub use alloc::vec; pub use alloc::string; From 6c9cf2bacfafd9b90e828616e8cc9f8c343fcc50 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 15 Feb 2018 23:14:33 -0800 Subject: [PATCH 1540/3084] Switch from error_core to failure. --- README.rst | 4 --- lib/cretonne/Cargo.toml | 8 +++--- lib/cretonne/src/lib.rs | 8 +++--- lib/cretonne/src/result.rs | 43 +++++++------------------------- lib/cretonne/src/verifier/mod.rs | 9 +------ lib/filecheck/Cargo.toml | 2 ++ lib/filecheck/src/error.rs | 41 ++++++++---------------------- lib/filecheck/src/lib.rs | 3 +++ lib/wasm/Cargo.toml | 6 +---- lib/wasm/src/environ/dummy.rs | 3 +-- lib/wasm/src/lib.rs | 5 ---- 11 files changed, 33 insertions(+), 99 deletions(-) diff --git a/README.rst b/README.rst index caa24342d9..621217f888 100644 --- a/README.rst +++ b/README.rst @@ -92,10 +92,6 @@ called `hashmap_core` is pulled in (only in `no_std` builds). This is mostly the same as `std::collections::HashMap`, except that it doesn't have DOS protection. Just something to think about. -Lastly, to support `std::error`, which isn't is `std` or `alloc` for -an inexplicable reason, the `error_core` crate is also used in `no_std` builds. -You might need it, as well, when interfacing with `CtonError`. - Building the documentation -------------------------- diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index c1c666fa77..09d3a7dd7b 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -17,19 +17,17 @@ name = "cretonne" # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be # accomodated in `tests`. +failure = { version = "0.1.1", default-features = false, features = ["derive"] } +failure_derive = { version = "0.1.1", default-features = false } [dependencies.hashmap_core] version = "0.1.1" optional = true -[dependencies.error_core] -version = "0.1.0" -optional = true - [features] # The "std" feature enables use of libstd. The "no_std" feature enables use # of some minimal std-like replacement libraries. At least one of these two # features to be enabled. default = ["std"] std = [] -no_std = ["hashmap_core", "error_core"] +no_std = ["hashmap_core"] diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 0952969bb1..ddb391852d 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -8,11 +8,12 @@ // Include the `hashmap_core` crate if no_std #[cfg(feature = "no_std")] extern crate hashmap_core; -#[cfg(feature = "no_std")] -extern crate error_core; #[cfg(not(feature = "std"))] #[macro_use] extern crate alloc; +extern crate failure; +#[macro_use] +extern crate failure_derive; pub use context::Context; pub use legalizer::legalize_function; @@ -69,7 +70,4 @@ mod std { pub use hashmap_core::map as hash_map; pub use alloc::BTreeSet; } - pub mod error { - pub use error_core::Error; - } } diff --git a/lib/cretonne/src/result.rs b/lib/cretonne/src/result.rs index 387eaa9a2a..2b89c774d3 100644 --- a/lib/cretonne/src/result.rs +++ b/lib/cretonne/src/result.rs @@ -1,25 +1,28 @@ //! Result and error types representing the outcome of compiling a function. use verifier; -use std::error::Error as StdError; -use std::fmt; /// A compilation error. /// /// When Cretonne fails to compile a function, it will return one of these error codes. -#[derive(Debug, PartialEq, Eq)] +#[derive(Fail, Debug, PartialEq, Eq)] pub enum CtonError { /// The input is invalid. /// /// This error code is used by a WebAssembly translator when it encounters invalid WebAssembly /// code. This should never happen for validated WebAssembly code. + #[fail(display = "Invalid input code")] InvalidInput, /// An IL verifier error. /// /// This always represents a bug, either in the code that generated IL for Cretonne, or a bug /// in Cretonne itself. - Verifier(verifier::Error), + #[fail(display = "Verifier error: {}", _0)] + Verifier( + #[cause] + verifier::Error + ), /// An implementation limit was exceeded. /// @@ -27,48 +30,20 @@ pub enum CtonError { /// limits][limits] that cause compilation to fail when they are exceeded. /// /// [limits]: http://cretonne.readthedocs.io/en/latest/langref.html#implementation-limits + #[fail(display = "Implementation limit exceeded")] ImplLimitExceeded, /// The code size for the function is too large. /// /// Different target ISAs may impose a limit on the size of a compiled function. If that limit /// is exceeded, compilation fails. + #[fail(display = "Code for function is too large")] CodeTooLarge, } /// A Cretonne compilation result. pub type CtonResult = Result<(), CtonError>; -impl fmt::Display for CtonError { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match *self { - CtonError::Verifier(ref e) => write!(f, "Verifier error: {}", e), - CtonError::InvalidInput | - CtonError::ImplLimitExceeded | - CtonError::CodeTooLarge => f.write_str(self.description()), - } - } -} - -impl StdError for CtonError { - fn description(&self) -> &str { - match *self { - CtonError::InvalidInput => "Invalid input code", - CtonError::Verifier(ref e) => &e.message, - CtonError::ImplLimitExceeded => "Implementation limit exceeded", - CtonError::CodeTooLarge => "Code for function is too large", - } - } - fn cause(&self) -> Option<&StdError> { - match *self { - CtonError::Verifier(ref e) => Some(e), - CtonError::InvalidInput | - CtonError::ImplLimitExceeded | - CtonError::CodeTooLarge => None, - } - } -} - impl From for CtonError { fn from(e: verifier::Error) -> CtonError { CtonError::Verifier(e) diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 117de2f38b..821bc3d623 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -70,7 +70,6 @@ use self::flags::verify_flags; use settings::{Flags, FlagsOrIsa}; use std::cmp::Ordering; use std::collections::BTreeSet; -use std::error as std_error; use std::fmt::{self, Display, Formatter, Write}; use std::result; use std::vec::Vec; @@ -104,7 +103,7 @@ mod liveness; mod locations; /// A verifier error. -#[derive(Debug, PartialEq, Eq)] +#[derive(Fail, Debug, PartialEq, Eq)] pub struct Error { /// The entity causing the verifier error. pub location: AnyEntity, @@ -118,12 +117,6 @@ impl Display for Error { } } -impl std_error::Error for Error { - fn description(&self) -> &str { - &self.message - } -} - /// Verifier result. pub type Result = result::Result<(), Error>; diff --git a/lib/filecheck/Cargo.toml b/lib/filecheck/Cargo.toml index aec22b0f51..b29eaa5c53 100644 --- a/lib/filecheck/Cargo.toml +++ b/lib/filecheck/Cargo.toml @@ -12,3 +12,5 @@ name = "filecheck" [dependencies] regex = "0.2.6" +failure = "0.1.1" +failure_derive = "0.1.1" diff --git a/lib/filecheck/src/error.rs b/lib/filecheck/src/error.rs index 5580cc4c3c..af9657cdcc 100644 --- a/lib/filecheck/src/error.rs +++ b/lib/filecheck/src/error.rs @@ -1,21 +1,21 @@ use std::result; use std::convert::From; -use std::error::Error as StdError; -use std::fmt; use regex; /// A result from the filecheck library. pub type Result = result::Result; /// A filecheck error. -#[derive(Debug)] +#[derive(Fail, Debug)] pub enum Error { /// A syntax error in a check line. + #[fail(display = "{}", _0)] Syntax(String), /// A check refers to an undefined variable. /// /// The pattern contains `$foo` where the `foo` variable has not yet been defined. /// Use `$$` to match a literal dollar sign. + #[fail(display = "{}", _0)] UndefVariable(String), /// A pattern contains a back-reference to a variable that was defined in the same pattern. /// @@ -26,40 +26,19 @@ pub enum Error { /// check: Hello $(world=[^ ]*) /// sameln: $world /// ``` + #[fail(display = "{}", _0)] Backref(String), /// A pattern contains multiple definitions of the same variable. + #[fail(display = "{}", _0)] DuplicateDef(String), /// An error in a regular expression. /// /// Use `cause()` to get the underlying `Regex` library error. - Regex(regex::Error), -} - -impl StdError for Error { - fn description(&self) -> &str { - use Error::*; - match *self { - Syntax(ref s) | - UndefVariable(ref s) | - Backref(ref s) | - DuplicateDef(ref s) => s, - Regex(ref err) => err.description(), - } - } - - fn cause(&self) -> Option<&StdError> { - use Error::*; - match *self { - Regex(ref err) => Some(err), - _ => None, - } - } -} - -impl fmt::Display for Error { - fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result { - write!(fmt, "{}", self.description()) - } + #[fail(display = "{}", _0)] + Regex( + #[cause] + regex::Error + ), } impl From for Error { diff --git a/lib/filecheck/src/lib.rs b/lib/filecheck/src/lib.rs index a189acebdb..b29377f9ac 100644 --- a/lib/filecheck/src/lib.rs +++ b/lib/filecheck/src/lib.rs @@ -243,6 +243,9 @@ pub use variable::{VariableMap, Value, NO_VARIABLES}; pub use checker::{Checker, CheckerBuilder}; extern crate regex; +extern crate failure; +#[macro_use] +extern crate failure_derive; mod error; mod variable; diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 3bd2b36aa8..7954ebc209 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -19,14 +19,10 @@ cretonne-frontend = { path = "../frontend", version = "0.1.0", default_features version = "0.1.1" optional = true -[dependencies.error_core] -version = "0.1.0" -optional = true - [dev-dependencies] tempdir = "0.3.5" [features] default = ["std"] std = ["cretonne/std", "cretonne-frontend/std"] -no_std = ["hashmap_core", "error_core", "cretonne/no_std", "cretonne-frontend/no_std"] +no_std = ["hashmap_core", "cretonne/no_std", "cretonne-frontend/no_std"] diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index de941f3b9a..102e83ef79 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -9,7 +9,6 @@ use cretonne::ir::types::*; use cretonne::cursor::FuncCursor; use cretonne::settings; use wasmparser; -use std::error::Error; use std::vec::Vec; use std::string::String; @@ -387,7 +386,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment { let reader = wasmparser::BinaryReader::new(body_bytes); self.trans .translate_from_reader(reader, &mut func, &mut func_environ) - .map_err(|e| String::from(e.description()))?; + .map_err(|e| format!("{}", e))?; func }; self.func_bytecode_sizes.push(body_bytes.len()); diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 76e69887d9..ae5c99e05e 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -20,8 +20,6 @@ extern crate alloc; #[cfg(feature = "no_std")] extern crate hashmap_core; -#[cfg(feature = "no_std")] -extern crate error_core; extern crate wasmparser; extern crate cton_frontend; @@ -50,7 +48,4 @@ mod std { pub mod collections { pub use hashmap_core::{HashMap, map as hash_map}; } - pub mod error { - pub use error_core::Error; - } } From 81c126619b3386f1ba3364e1ede499300516e34f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 26 Feb 2018 15:24:25 -0800 Subject: [PATCH 1541/3084] Ignore unknown custom wasm sections. --- lib/wasm/src/module_translator.rs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index d75f60a35e..d50031fb04 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -124,6 +124,10 @@ pub fn translate_module<'data>( } } } + ParserState::BeginSection { code: SectionCode::Custom { .. }, .. } => { + // Ignore unknown custom sections. + next_input = ParserInput::SkipSection; + } _ => return Err(String::from("wrong content in the preamble")), }; } From 0e22c740858ae18f4a213ba8306108fdd00264b0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 26 Feb 2018 14:12:13 -0800 Subject: [PATCH 1542/3084] Track wasm reachability explicitly. Maintain an explicit "reachable" flag when decoding wasm. Push placeholder frames on the control-flow stack instead of just maintaining a count of the stack depth in unreachable code, so that we can whether If blocks have Elses, and whether block exits are branched to, in all contexts. Fixes #217. --- cranelift/wasmtests/unreachable_code.wat | 77 +++++++++ lib/wasm/src/code_translator.rs | 206 ++++++++++++----------- lib/wasm/src/func_translator.rs | 8 +- lib/wasm/src/state.rs | 49 +++--- 4 files changed, 211 insertions(+), 129 deletions(-) create mode 100644 cranelift/wasmtests/unreachable_code.wat diff --git a/cranelift/wasmtests/unreachable_code.wat b/cranelift/wasmtests/unreachable_code.wat new file mode 100644 index 0000000000..38c1a315ce --- /dev/null +++ b/cranelift/wasmtests/unreachable_code.wat @@ -0,0 +1,77 @@ +(module + (type (;0;) (func (param i32 i64 f64) (result f64))) + (type (;1;) (func)) + (type (;2;) (func (result f32))) + (type (;3;) (func (result f64))) + (type (;4;) (func (param f64 f64) (result f64))) + (type (;5;) (func (result i32))) + (func (result i32) + block (result i32) + unreachable + end + block + end + i32.clz + ) + (func (result i32) + loop (result i32) + unreachable + end + block + end + i32.clz + ) + (func (;0;) (type 5) (result i32) + nop + block (result i32) ;; label = @1 + block ;; label = @2 + block ;; label = @3 + nop + block ;; label = @4 + i32.const 1 + if ;; label = @5 + nop + block ;; label = @6 + nop + nop + loop (result i32) ;; label = @7 + nop + block (result i32) ;; label = @8 + nop + nop + block (result i32) ;; label = @9 + nop + unreachable + end + end + end + block (result i32) ;; label = @7 + block ;; label = @8 + nop + end + i32.const 0 + end + br_if 5 (;@1;) + drop + end + else + nop + end + nop + end + end + end + unreachable + end) + (func + block (result i32) + block (result i32) + i32.const 1 + br 1 + end + end + drop + ) + (table (;0;) 16 anyfunc) + (elem (i32.const 0)) +) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 903b9cd9e1..abaaaef3bf 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -25,6 +25,7 @@ use cretonne::ir::{self, InstBuilder, MemFlags, JumpTableData}; use cretonne::ir::types::*; use cretonne::ir::condcodes::{IntCC, FloatCC}; +use cretonne::packed_option::ReservedValue; use cton_frontend::FunctionBuilder; use wasmparser::{Operator, MemoryImmediate}; use translation_utils::{f32_translation, f64_translation, type_to_type, num_return_values, Local}; @@ -42,7 +43,7 @@ pub fn translate_operator( state: &mut TranslationState, environ: &mut FE, ) { - if state.in_unreachable_code() { + if !state.reachable { return translate_unreachable_operator(op, builder, state); } @@ -105,7 +106,7 @@ pub fn translate_operator( // We use `trap user0` to indicate a user-generated trap. // We could make the trap code configurable if need be. builder.ins().trap(ir::TrapCode::User(0)); - state.real_unreachable_stack_depth = 1; + state.reachable = false; } /***************************** Control flow blocks ********************************** * When starting a control flow block, we create a new `Ebb` that will hold the code @@ -155,15 +156,24 @@ pub fn translate_operator( // and push a new control frame with a new ebb for the code after the if/then/else // At the end of the then clause we jump to the destination let i = state.control_stack.len() - 1; - let (destination, return_count, branch_inst) = match state.control_stack[i] { - ControlStackFrame::If { - destination, - num_return_values, - branch_inst, - .. - } => (destination, num_return_values, branch_inst), - _ => panic!("should not happen"), - }; + let (destination, return_count, branch_inst, ref mut reachable_from_top) = + match state.control_stack[i] { + ControlStackFrame::If { + destination, + num_return_values, + branch_inst, + reachable_from_top, + .. + } => ( + destination, + num_return_values, + branch_inst, + reachable_from_top, + ), + _ => panic!("should not happen"), + }; + // The if has an else, so there's no branch to the end from the top. + *reachable_from_top = false; builder.ins().jump(destination, state.peekn(return_count)); state.popn(return_count); // We change the target of the branch instruction @@ -219,7 +229,7 @@ pub fn translate_operator( let (return_count, br_destination) = { let frame = &mut state.control_stack[i]; // We signal that all the code that follows until the next End is unreachable - frame.set_reachable(); + frame.set_branched_to_exit(); let return_count = if frame.is_loop() { 0 } else { @@ -232,7 +242,7 @@ pub fn translate_operator( state.peekn(return_count), ); state.popn(return_count); - state.real_unreachable_stack_depth = 1 + relative_depth as usize; + state.reachable = false; } Operator::BrIf { relative_depth } => { let val = state.pop1(); @@ -241,7 +251,7 @@ pub fn translate_operator( let frame = &mut state.control_stack[i]; // The values returned by the branch are still available for the reachable // code that comes after it - frame.set_reachable(); + frame.set_branched_to_exit(); let return_count = if frame.is_loop() { 0 } else { @@ -277,20 +287,23 @@ pub fn translate_operator( let val = state.pop1(); let mut data = JumpTableData::with_capacity(depths.len()); for depth in depths { - let i = state.control_stack.len() - 1 - (depth as usize); - let frame = &mut state.control_stack[i]; - let ebb = frame.br_destination(); + let ebb = { + let i = state.control_stack.len() - 1 - (depth as usize); + let frame = &mut state.control_stack[i]; + frame.set_branched_to_exit(); + frame.br_destination() + }; data.push_entry(ebb); - frame.set_reachable(); } let jt = builder.create_jump_table(data); builder.ins().br_table(val, jt); - let i = state.control_stack.len() - 1 - (default as usize); - let frame = &mut state.control_stack[i]; - let ebb = frame.br_destination(); + let ebb = { + let i = state.control_stack.len() - 1 - (default as usize); + let frame = &mut state.control_stack[i]; + frame.set_branched_to_exit(); + frame.br_destination() + }; builder.ins().jump(ebb, &[]); - state.real_unreachable_stack_depth = 1 + min_depth as usize; - frame.set_reachable(); } else { // Here we have jump arguments, but Cretonne's br_table doesn't support them // We then proceed to split the edges going out of the br_table @@ -312,29 +325,32 @@ pub fn translate_operator( } let jt = builder.create_jump_table(data); builder.ins().br_table(val, jt); - let default_ebb = state.control_stack[state.control_stack.len() - 1 - - (default as usize)] - .br_destination(); + let default_ebb = { + let i = state.control_stack.len() - 1 - (default as usize); + let frame = &mut state.control_stack[i]; + frame.set_branched_to_exit(); + frame.br_destination() + }; builder.ins().jump(default_ebb, state.peekn(return_count)); for (depth, dest_ebb) in dest_ebb_sequence { builder.switch_to_block(dest_ebb); builder.seal_block(dest_ebb); - let i = state.control_stack.len() - 1 - depth; let real_dest_ebb = { + let i = state.control_stack.len() - 1 - depth; let frame = &mut state.control_stack[i]; - frame.set_reachable(); + frame.set_branched_to_exit(); frame.br_destination() }; builder.ins().jump(real_dest_ebb, state.peekn(return_count)); } state.popn(return_count); - state.real_unreachable_stack_depth = 1 + min_depth as usize; } + state.reachable = false; } Operator::Return => { let (return_count, br_destination) = { let frame = &mut state.control_stack[0]; - frame.set_reachable(); + frame.set_branched_to_exit(); let return_count = frame.num_return_values(); (return_count, frame.br_destination()) }; @@ -347,7 +363,7 @@ pub fn translate_operator( } } state.popn(return_count); - state.real_unreachable_stack_depth = 1; + state.reachable = false; } /************************************ Calls **************************************** * The call instructions pop off their arguments from the stack and append their @@ -900,76 +916,74 @@ fn translate_unreachable_operator( builder: &mut FunctionBuilder, state: &mut TranslationState, ) { - let stack = &mut state.stack; - let control_stack = &mut state.control_stack; - - // We don't translate because the code is unreachable - // Nevertheless we have to record a phantom stack for this code - // to know when the unreachable code ends match *op { - Operator::If { ty: _ } | + Operator::If { ty: _ } => { + // Push a placeholder control stack entry. The if isn't reachable, + // so we don't have any branches anywhere. + state.push_if(ir::Inst::reserved_value(), ir::Ebb::reserved_value(), 0); + } Operator::Loop { ty: _ } | Operator::Block { ty: _ } => { - state.phantom_unreachable_stack_depth += 1; - } - Operator::End => { - if state.phantom_unreachable_stack_depth > 0 { - state.phantom_unreachable_stack_depth -= 1; - } else { - // This End corresponds to a real control stack frame - // We switch to the destination block but we don't insert - // a jump instruction since the code is still unreachable - let frame = control_stack.pop().unwrap(); - - builder.switch_to_block(frame.following_code()); - builder.seal_block(frame.following_code()); - match frame { - // If it is a loop we also have to seal the body loop block - ControlStackFrame::Loop { header, .. } => builder.seal_block(header), - // If it is an if then the code after is reachable again - ControlStackFrame::If { .. } => { - state.real_unreachable_stack_depth = 1; - } - _ => {} - } - if frame.is_reachable() { - state.real_unreachable_stack_depth = 1; - } - // Now we have to split off the stack the values not used - // by unreachable code that hasn't been translated - stack.truncate(frame.original_stack_size()); - // And add the return values of the block but only if the next block is reachble - // (which corresponds to testing if the stack depth is 1) - if state.real_unreachable_stack_depth == 1 { - stack.extend_from_slice(builder.ebb_params(frame.following_code())); - } - state.real_unreachable_stack_depth -= 1; - } + state.push_block(ir::Ebb::reserved_value(), 0); } Operator::Else => { - if state.phantom_unreachable_stack_depth > 0 { - // This is part of a phantom if-then-else, we do nothing - } else { - // Encountering an real else means that the code in the else - // clause is reachable again - let (branch_inst, original_stack_size) = match control_stack[control_stack.len() - - 1] { - ControlStackFrame::If { - branch_inst, - original_stack_size, - .. - } => (branch_inst, original_stack_size), - _ => panic!("should not happen"), - }; - // We change the target of the branch instruction - let else_ebb = builder.create_ebb(); - builder.change_jump_destination(branch_inst, else_ebb); - builder.seal_block(else_ebb); - builder.switch_to_block(else_ebb); - // Now we have to split off the stack the values not used - // by unreachable code that hasn't been translated - stack.truncate(original_stack_size); - state.real_unreachable_stack_depth = 0; + let i = state.control_stack.len() - 1; + match state.control_stack[i] { + ControlStackFrame::If { + branch_inst, + ref mut reachable_from_top, + .. + } => { + if *reachable_from_top { + // We have a branch from the top of the if to the else. + state.reachable = true; + // And because there's an else, there can no longer be a + // branch from the top directly to the end. + *reachable_from_top = false; + + // We change the target of the branch instruction + let else_ebb = builder.create_ebb(); + builder.change_jump_destination(branch_inst, else_ebb); + builder.seal_block(else_ebb); + builder.switch_to_block(else_ebb); + } + } + _ => {} + } + } + Operator::End => { + let stack = &mut state.stack; + let control_stack = &mut state.control_stack; + let frame = control_stack.pop().unwrap(); + + // Now we have to split off the stack the values not used + // by unreachable code that hasn't been translated + stack.truncate(frame.original_stack_size()); + + let reachable_anyway = match frame { + // If it is a loop we also have to seal the body loop block + ControlStackFrame::Loop { header, .. } => { + builder.seal_block(header); + // And loops can't have branches to the end. + false + } + ControlStackFrame::If { reachable_from_top, .. } => { + // A reachable if without an else has a branch from the top + // directly to the bottom. + reachable_from_top + } + // All other control constructs are already handled. + _ => false, + }; + + if frame.exit_is_branched_to() || reachable_anyway { + builder.switch_to_block(frame.following_code()); + builder.seal_block(frame.following_code()); + + // And add the return values of the block but only if the next block is reachble + // (which corresponds to testing if the stack depth is 1) + stack.extend_from_slice(builder.ebb_params(frame.following_code())); + state.reachable = true; } } _ => { diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 5e8555c194..a9389342e9 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -206,9 +206,11 @@ fn parse_function_body( // // If the exit block is unreachable, it may not have the correct arguments, so we would // generate a return instruction that doesn't match the signature. - debug_assert!(builder.is_pristine()); - if !builder.is_unreachable() { - builder.ins().return_(&state.stack); + if state.reachable { + debug_assert!(builder.is_pristine()); + if !builder.is_unreachable() { + builder.ins().return_(&state.stack); + } } // Discard any remaining values on the stack. Either we just returned them, diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index f9f76d2934..1943f31326 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -25,20 +25,20 @@ pub enum ControlStackFrame { branch_inst: Inst, num_return_values: usize, original_stack_size: usize, - reachable: bool, + exit_is_branched_to: bool, + reachable_from_top: bool, }, Block { destination: Ebb, num_return_values: usize, original_stack_size: usize, - reachable: bool, + exit_is_branched_to: bool, }, Loop { destination: Ebb, header: Ebb, num_return_values: usize, original_stack_size: usize, - reachable: bool, }, } @@ -80,19 +80,21 @@ impl ControlStackFrame { } } - pub fn is_reachable(&self) -> bool { + pub fn exit_is_branched_to(&self) -> bool { match *self { - ControlStackFrame::If { reachable, .. } | - ControlStackFrame::Block { reachable, .. } | - ControlStackFrame::Loop { reachable, .. } => reachable, + ControlStackFrame::If { exit_is_branched_to, .. } | + ControlStackFrame::Block { exit_is_branched_to, .. } => exit_is_branched_to, + ControlStackFrame::Loop { .. } => false, } } - pub fn set_reachable(&mut self) { + pub fn set_branched_to_exit(&mut self) { match *self { - ControlStackFrame::If { ref mut reachable, .. } | - ControlStackFrame::Block { ref mut reachable, .. } | - ControlStackFrame::Loop { ref mut reachable, .. } => *reachable = true, + ControlStackFrame::If { ref mut exit_is_branched_to, .. } | + ControlStackFrame::Block { ref mut exit_is_branched_to, .. } => { + *exit_is_branched_to = true + } + ControlStackFrame::Loop { .. } => {} } } } @@ -105,8 +107,7 @@ impl ControlStackFrame { pub struct TranslationState { pub stack: Vec, pub control_stack: Vec, - pub phantom_unreachable_stack_depth: usize, - pub real_unreachable_stack_depth: usize, + pub reachable: bool, // Map of global variables that have already been created by `FuncEnvironment::make_global`. globals: HashMap, @@ -130,8 +131,7 @@ impl TranslationState { Self { stack: Vec::new(), control_stack: Vec::new(), - phantom_unreachable_stack_depth: 0, - real_unreachable_stack_depth: 0, + reachable: true, globals: HashMap::new(), heaps: HashMap::new(), signatures: HashMap::new(), @@ -142,8 +142,7 @@ impl TranslationState { fn clear(&mut self) { debug_assert!(self.stack.is_empty()); debug_assert!(self.control_stack.is_empty()); - debug_assert_eq!(self.phantom_unreachable_stack_depth, 0); - debug_assert_eq!(self.real_unreachable_stack_depth, 0); + self.reachable = true; self.globals.clear(); self.heaps.clear(); self.signatures.clear(); @@ -219,7 +218,7 @@ impl TranslationState { destination: following_code, original_stack_size: self.stack.len(), num_return_values: num_result_types, - reachable: false, + exit_is_branched_to: false, }); } @@ -230,7 +229,6 @@ impl TranslationState { destination: following_code, original_stack_size: self.stack.len(), num_return_values: num_result_types, - reachable: false, }); } @@ -241,19 +239,10 @@ impl TranslationState { destination: following_code, original_stack_size: self.stack.len(), num_return_values: num_result_types, - reachable: false, + exit_is_branched_to: false, + reachable_from_top: self.reachable, }); } - - /// Test if the translation state is currently in unreachable code. - pub fn in_unreachable_code(&self) -> bool { - if self.real_unreachable_stack_depth > 0 { - true - } else { - debug_assert_eq!(self.phantom_unreachable_stack_depth, 0, "in reachable code"); - false - } - } } /// Methods for handling entity references. From 234d097f656b468492ddc73573e9a18d8ae1839f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 21 Feb 2018 12:43:36 -0800 Subject: [PATCH 1543/3084] Change `println!("")` to `println!()`. https://rust-lang-nursery.github.io/rust-clippy/v0.0.186/index.html#print_with_newline --- cranelift/src/cat.rs | 4 ++-- cranelift/src/print_cfg.rs | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs index e8519b10d9..36ae818e2c 100644 --- a/cranelift/src/cat.rs +++ b/cranelift/src/cat.rs @@ -13,7 +13,7 @@ use filetest::subtest::{self, SubTest, Context, Result as STResult}; pub fn run(files: Vec) -> CommandResult { for (i, f) in files.into_iter().enumerate() { if i != 0 { - println!(""); + println!(); } cat_one(f)? } @@ -30,7 +30,7 @@ fn cat_one(filename: String) -> CommandResult { for (idx, func) in items.into_iter().enumerate() { if idx != 0 { - println!(""); + println!(); } print!("{}", func); } diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index f3b958be77..df624993ad 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -17,7 +17,7 @@ use utils::read_to_string; pub fn run(files: Vec) -> CommandResult { for (i, f) in files.into_iter().enumerate() { if i != 0 { - println!(""); + println!(); } print_cfg(f)? } @@ -100,7 +100,7 @@ fn print_cfg(filename: String) -> CommandResult { for (idx, func) in items.into_iter().enumerate() { if idx != 0 { - println!(""); + println!(); } print!("{}", CFGPrinter::new(&func)); } From e943d932b90f4d3467a4f14fc064ff2324bbce45 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 21 Feb 2018 12:42:30 -0800 Subject: [PATCH 1544/3084] Change `match self` to `match *self` to avoid adding `&` to all patterns in a match. https://rust-lang-nursery.github.io/rust-clippy/v0.0.186/index.html#match_ref_pats --- lib/cretonne/src/bforest/node.rs | 60 ++++++++++++++++---------------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/lib/cretonne/src/bforest/node.rs b/lib/cretonne/src/bforest/node.rs index 53a96f9b14..3c6b8c8774 100644 --- a/lib/cretonne/src/bforest/node.rs +++ b/lib/cretonne/src/bforest/node.rs @@ -54,8 +54,8 @@ impl Clone for NodeData { impl NodeData { /// Is this a free/unused node? pub fn is_free(&self) -> bool { - match self { - &NodeData::Free { .. } => true, + match *self { + NodeData::Free { .. } => true, _ => false, } } @@ -65,10 +65,10 @@ impl NodeData { /// This is the number of outgoing edges in an inner node, or the number of key-value pairs in /// a leaf node. pub fn entries(&self) -> usize { - match self { - &NodeData::Inner { size, .. } => usize::from(size) + 1, - &NodeData::Leaf { size, .. } => usize::from(size), - &NodeData::Free { .. } => panic!("freed node"), + match *self { + NodeData::Inner { size, .. } => usize::from(size) + 1, + NodeData::Leaf { size, .. } => usize::from(size), + NodeData::Free { .. } => panic!("freed node"), } } @@ -96,8 +96,8 @@ impl NodeData { /// Unwrap an inner node into two slices (keys, trees). pub fn unwrap_inner(&self) -> (&[F::Key], &[Node]) { - match self { - &NodeData::Inner { + match *self { + NodeData::Inner { size, ref keys, ref tree, @@ -113,8 +113,8 @@ impl NodeData { /// Unwrap a leaf node into two slices (keys, values) of the same length. pub fn unwrap_leaf(&self) -> (&[F::Key], &[F::Value]) { - match self { - &NodeData::Leaf { + match *self { + NodeData::Leaf { size, ref keys, ref vals, @@ -132,8 +132,8 @@ impl NodeData { /// Unwrap a mutable leaf node into two slices (keys, values) of the same length. pub fn unwrap_leaf_mut(&mut self) -> (&mut [F::Key], &mut [F::Value]) { - match self { - &mut NodeData::Leaf { + match *self { + NodeData::Leaf { size, ref mut keys, ref mut vals, @@ -152,8 +152,8 @@ impl NodeData { /// Get the critical key for a leaf node. /// This is simply the first key. pub fn leaf_crit_key(&self) -> F::Key { - match self { - &NodeData::Leaf { size, ref keys, .. } => { + match *self { + NodeData::Leaf { size, ref keys, .. } => { debug_assert!(size > 0, "Empty leaf node"); keys.borrow()[0] } @@ -165,8 +165,8 @@ impl NodeData { /// This means that `key` is inserted at `keys[i]` and `node` is inserted at `tree[i + 1]`. /// If the node is full, this leaves the node unchanged and returns false. pub fn try_inner_insert(&mut self, index: usize, key: F::Key, node: Node) -> bool { - match self { - &mut NodeData::Inner { + match *self { + NodeData::Inner { ref mut size, ref mut keys, ref mut tree, @@ -191,8 +191,8 @@ impl NodeData { /// Try to insert `key, value` at `index` in a leaf node, but fail and return false if the node /// is full. pub fn try_leaf_insert(&mut self, index: usize, key: F::Key, value: F::Value) -> bool { - match self { - &mut NodeData::Leaf { + match *self { + NodeData::Leaf { ref mut size, ref mut keys, ref mut vals, @@ -222,8 +222,8 @@ impl NodeData { /// The `insert_index` parameter is the position where an insertion was tried and failed. The /// node will be split in half with a bias towards an even split after the insertion is retried. pub fn split(&mut self, insert_index: usize) -> SplitOff { - match self { - &mut NodeData::Inner { + match *self { + NodeData::Inner { ref mut size, ref keys, ref tree, @@ -262,7 +262,7 @@ impl NodeData { }, } } - &mut NodeData::Leaf { + NodeData::Leaf { ref mut size, ref keys, ref vals, @@ -307,8 +307,8 @@ impl NodeData { /// /// Return an indication of the node's health (i.e. below half capacity). pub fn inner_remove(&mut self, index: usize) -> Removed { - match self { - &mut NodeData::Inner { + match *self { + NodeData::Inner { ref mut size, ref mut keys, ref mut tree, @@ -332,8 +332,8 @@ impl NodeData { /// /// Return an indication of the node's health (i.e. below half capacity). pub fn leaf_remove(&mut self, index: usize) -> Removed { - match self { - &mut NodeData::Leaf { + match *self { + NodeData::Leaf { ref mut size, ref mut keys, ref mut vals, @@ -553,15 +553,15 @@ where F::Value: ValDisp, { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match self { - &NodeData::Inner { size, keys, tree } => { + match *self { + NodeData::Inner { size, keys, tree } => { write!(f, "[ {}", tree[0])?; for i in 0..usize::from(size) { write!(f, " {} {}", keys[i], tree[i + 1])?; } write!(f, " ]") } - &NodeData::Leaf { size, keys, vals } => { + NodeData::Leaf { size, keys, vals } => { let keys = keys.borrow(); let vals = vals.borrow(); write!(f, "[")?; @@ -571,8 +571,8 @@ where } write!(f, " ]") } - &NodeData::Free { next: Some(n) } => write!(f, "[ free -> {} ]", n), - &NodeData::Free { next: None } => write!(f, "[ free ]"), + NodeData::Free { next: Some(n) } => write!(f, "[ free -> {} ]", n), + NodeData::Free { next: None } => write!(f, "[ free ]"), } } } From af154655d7e45c7756a06b5920f405f27f1b96ee Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 21 Feb 2018 12:35:37 -0800 Subject: [PATCH 1545/3084] Replace `as` casts with type-conversion functions. https://github.com/rust-lang-nursery/rust-clippy/wiki#cast_lossless --- lib/cretonne/src/isa/intel/abi.rs | 2 +- lib/cretonne/src/isa/riscv/binemit.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 21442250c5..cf30058364 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -224,7 +224,7 @@ pub fn native_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> res }); let total_stack_size = layout_stack(&mut func.stack_slots, stack_align)? as i32; - let local_stack_size = (total_stack_size - csr_stack_size) as i64; + let local_stack_size = i64::from(total_stack_size - csr_stack_size); // Add CSRs to function signature let fp_arg = ir::AbiParam::special_reg( diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index c20b1ca004..369a9bf09f 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -106,7 +106,7 @@ fn put_i(bits: u16, rs1: RegUnit, imm: i64, rd: RegUnit, /// /// Encoding bits: `opcode[6:2] | (funct3 << 5)` fn put_u(bits: u16, imm: i64, rd: RegUnit, sink: &mut CS) { - let bits = bits as u32; + let bits = u32::from(bits); let opcode5 = bits & 0x1f; let rd = u32::from(rd) & 0x1f; From 5ffdc5174238fab5ed3c596105ab9b4345342f72 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 27 Feb 2018 12:41:43 -0800 Subject: [PATCH 1546/3084] Fix formatting of no_std instructions in README.rst. --- README.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.rst b/README.rst index 621217f888..0186bc57f8 100644 --- a/README.rst +++ b/README.rst @@ -66,11 +66,15 @@ default, and enable the `no_std` feature. For example, to build `cretonne`: +.. code-block:: sh + cd lib/cretonne cargo build --no-default-features --features no_std Or, when using `cretonne` as a dependency (in Cargo.toml): +.. code-block:: + [dependency.cretonne] ... default-features = false From 6a962e8b2c6359bd8d9f6eb66565da2d39138731 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 27 Feb 2018 15:26:06 -0800 Subject: [PATCH 1547/3084] Add links to the docs.rs API documentation. --- cranelift/docs/index.rst | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst index 301346a9e3..c2e5b6d5ef 100644 --- a/cranelift/docs/index.rst +++ b/cranelift/docs/index.rst @@ -12,6 +12,28 @@ Contents: regalloc compare-llvm +Rust Crate Documentation +======================== + +`cretonne `_ + This is the core code generator crate. It takes Cretonne IR as input + and emits encoded machine instructions, along with symbolic relocations, + as output. + +`cretonne-wasm `_ + This crate translates WebAssembly code into Cretonne IR. + +`cretonne-frontend `_ + This crate provides utilities for translating code into Cretonne IR. + +`cretonne-native `_ + This crate performs auto-detection of the host, allowing Cretonne to + generate code optimized for the machine it's running on. + +`cretonne-reader `_ + This crate translates from Cretonne IR's text format into Cretonne IR + in in-memory data structures. + Indices and tables ================== From d394ae0902824bcfd27b83dd97d5550465c08ee0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 27 Feb 2018 15:32:21 -0800 Subject: [PATCH 1548/3084] Enable "set -euo pipefail" in all bash scripts. This enables "set -e", "set -u", and "set -o pipefail", which catch common errors. --- check-rustfmt.sh | 3 ++- cranelift/format-all.sh | 4 +--- cranelift/publish-all.sh | 2 +- cranelift/test-all.sh | 4 +--- lib/cretonne/meta/check.sh | 2 +- 5 files changed, 6 insertions(+), 9 deletions(-) diff --git a/check-rustfmt.sh b/check-rustfmt.sh index 483a45396a..1983493342 100755 --- a/check-rustfmt.sh +++ b/check-rustfmt.sh @@ -1,5 +1,6 @@ #!/bin/bash -# +set -euo pipefail + # Usage: check-rustfmt.sh [--install] # # Check that the desired version of rustfmt is installed. diff --git a/cranelift/format-all.sh b/cranelift/format-all.sh index 6ccac9c289..a99da7db09 100755 --- a/cranelift/format-all.sh +++ b/cranelift/format-all.sh @@ -1,10 +1,8 @@ #!/bin/bash +set -euo pipefail # Format all sources using rustfmt. -# Exit immediately on errors. -set -e - cd $(dirname "$0") # Make sure we can find rustfmt. diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 144aa519f8..1826287213 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -1,5 +1,5 @@ #!/bin/bash -set -e +set -euo pipefail cd $(dirname "$0") topdir=$(pwd) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index a5b2e92066..6a9291d82e 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -1,4 +1,5 @@ #!/bin/bash +set -euo pipefail # This is the top-level test script: # @@ -10,9 +11,6 @@ # # All tests run by this script should be passing at all times. -# Exit immediately on errors. -set -e - # Repository top-level directory. cd $(dirname "$0") topdir=$(pwd) diff --git a/lib/cretonne/meta/check.sh b/lib/cretonne/meta/check.sh index 655092e6cb..aa0e88ce1e 100755 --- a/lib/cretonne/meta/check.sh +++ b/lib/cretonne/meta/check.sh @@ -1,5 +1,5 @@ #!/bin/bash -set -e +set -euo pipefail cd $(dirname "$0") runif() { From 111c97df6dcdcd8e5ef52322ac43aa7c526a292c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 27 Feb 2018 15:43:13 -0800 Subject: [PATCH 1549/3084] Quote shell variables in more places. --- cranelift/publish-all.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 1826287213..b914a46394 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -1,7 +1,7 @@ #!/bin/bash set -euo pipefail cd $(dirname "$0") -topdir=$(pwd) +topdir="$(pwd)" # All the cretonne-* crates have the same version number # The filecheck crate version is managed independently. @@ -16,9 +16,9 @@ for crate in . lib/*; do continue fi # Update the version number of this crate to $version. - sed -i "" -e "s/^version = .*/version = \"$version\"/" $crate/Cargo.toml + sed -i "" -e "s/^version = .*/version = \"$version\"/" "$crate/Cargo.toml" # Update the required version number of any cretonne* dependencies. - sed -i "" -e "/^cretonne/s/version = \"[^\"]*\"/version = \"$version\"/" $crate/Cargo.toml + sed -i "" -e "/^cretonne/s/version = \"[^\"]*\"/version = \"$version\"/" "$crate/Cargo.toml" done # Update our local Cargo.lock (not checked in). @@ -30,5 +30,5 @@ cargo update # Note that libraries need to be published in topological order. for crate in filecheck cretonne frontend native reader wasm; do - echo cargo publish --manifest-path lib/$crate/Cargo.toml + echo cargo publish --manifest-path "lib/$crate/Cargo.toml" done From 42a0c2e44b6297bedfea568781409fdd19744773 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 27 Feb 2018 15:44:36 -0800 Subject: [PATCH 1550/3084] Enable backup files when setting version numbers with sed. --- cranelift/publish-all.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index b914a46394..91e6be826a 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -16,9 +16,9 @@ for crate in . lib/*; do continue fi # Update the version number of this crate to $version. - sed -i "" -e "s/^version = .*/version = \"$version\"/" "$crate/Cargo.toml" + sed -i.bk -e "s/^version = .*/version = \"$version\"/" "$crate/Cargo.toml" # Update the required version number of any cretonne* dependencies. - sed -i "" -e "/^cretonne/s/version = \"[^\"]*\"/version = \"$version\"/" "$crate/Cargo.toml" + sed -i.bk -e "/^cretonne/s/version = \"[^\"]*\"/version = \"$version\"/" "$crate/Cargo.toml" done # Update our local Cargo.lock (not checked in). From 2addcfd6429531d6f212ffe5287f61cfda5ff485 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 27 Feb 2018 15:54:29 -0800 Subject: [PATCH 1551/3084] Make publish-all.sh print the git commit line too, for convenience. --- cranelift/publish-all.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 91e6be826a..607a39d37e 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -29,6 +29,7 @@ cargo update # # Note that libraries need to be published in topological order. +echo git commit -a -m "Bump version to $version" for crate in filecheck cretonne frontend native reader wasm; do echo cargo publish --manifest-path "lib/$crate/Cargo.toml" done From dabfc55c6ecdad4380c07987a8db4f43a19a9b6c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 27 Feb 2018 15:58:25 -0800 Subject: [PATCH 1552/3084] Bump version to 0.2.0 --- cranelift/Cargo.toml | 12 ++++++------ cranelift/publish-all.sh | 4 ++-- lib/cretonne/Cargo.toml | 2 +- lib/frontend/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/wasm/Cargo.toml | 6 +++--- 7 files changed, 18 insertions(+), 18 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 91fe8ed6d9..20ec002a29 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.1.0" +version = "0.2.0" description = "Binaries for testing the Cretonne library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,11 +13,11 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne = { path = "lib/cretonne", version = "0.1.0" } -cretonne-reader = { path = "lib/reader", version = "0.1.0" } -cretonne-frontend = { path = "lib/frontend", version = "0.1.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.1.0" } -cretonne-native = { path = "lib/native", version = "0.1.0" } +cretonne = { path = "lib/cretonne", version = "0.2.0" } +cretonne-reader = { path = "lib/reader", version = "0.2.0" } +cretonne-frontend = { path = "lib/frontend", version = "0.2.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.2.0" } +cretonne-native = { path = "lib/native", version = "0.2.0" } filecheck = { path = "lib/filecheck" } docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 607a39d37e..5844621079 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -5,7 +5,7 @@ topdir="$(pwd)" # All the cretonne-* crates have the same version number # The filecheck crate version is managed independently. -version="0.1.0" +version="0.2.0" # Update all of the Cargo.toml files. # @@ -29,7 +29,7 @@ cargo update # # Note that libraries need to be published in topological order. -echo git commit -a -m "Bump version to $version" +echo git commit -a -m "\"Bump version to $version"\" for crate in filecheck cretonne frontend native reader wasm; do echo cargo publish --manifest-path "lib/$crate/Cargo.toml" done diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 88ae712c07..e9bed118cd 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.1.0" +version = "0.2.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index eea524ef1c..7ef8cee698 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.1.0" +version = "0.2.0" description = "Cretonne IL builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,4 +12,4 @@ readme = "README.md" name = "cton_frontend" [dependencies] -cretonne = { path = "../cretonne", version = "0.1.0" } +cretonne = { path = "../cretonne", version = "0.2.0" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 13556b0eb0..4401e41aab 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.1.0" +version = "0.2.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -11,7 +11,7 @@ readme = "README.md" name = "cton_native" [dependencies] -cretonne = { path = "../cretonne", version = "0.1.0" } +cretonne = { path = "../cretonne", version = "0.2.0" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 636f6b40e0..a732eda896 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.1.0" +version = "0.2.0" description = "Cretonne textual IL reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,4 +12,4 @@ readme = "README.md" name = "cton_reader" [dependencies] -cretonne = { path = "../cretonne", version = "0.1.0" } +cretonne = { path = "../cretonne", version = "0.2.0" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 2bad5f8d2a..efd4641a0e 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.1.0" +version = "0.2.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IL" repository = "https://github.com/Cretonne/cretonne" @@ -12,8 +12,8 @@ name = "cton_wasm" [dependencies] wasmparser = "0.14.1" -cretonne = { path = "../cretonne", version = "0.1.0" } -cretonne-frontend = { path = "../frontend", version = "0.1.0" } +cretonne = { path = "../cretonne", version = "0.2.0" } +cretonne-frontend = { path = "../frontend", version = "0.2.0" } [dev-dependencies] tempdir = "0.3.5" From 7dda61543f0769254f141daada24b69792bb1ba2 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 27 Feb 2018 17:01:02 -0800 Subject: [PATCH 1553/3084] Add more release instructions. --- cranelift/publish-all.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 5844621079..5cdc281387 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -30,6 +30,9 @@ cargo update # Note that libraries need to be published in topological order. echo git commit -a -m "\"Bump version to $version"\" +echo git push for crate in filecheck cretonne frontend native reader wasm; do echo cargo publish --manifest-path "lib/$crate/Cargo.toml" done +echo +echo Then, go to https://github.com/Cretonne/cretonne/releases/ and define a new release. From 60c6154b9438bcad93991c83ddc5a6e4ca92d2f5 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Feb 2018 03:38:48 -0800 Subject: [PATCH 1554/3084] Add some crate keywords. --- lib/cretonne/Cargo.toml | 1 + lib/wasm/Cargo.toml | 1 + 2 files changed, 2 insertions(+) diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index e9bed118cd..5dea4bb3f7 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -7,6 +7,7 @@ license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/Cretonne/cretonne" readme = "README.md" +keywords = [ "compile", "compiler", "jit" ] build = "build.rs" [lib] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index efd4641a0e..cd48f36959 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -6,6 +6,7 @@ description = "Translator from WebAssembly to Cretonne IL" repository = "https://github.com/Cretonne/cretonne" license = "Apache-2.0" readme = "README.md" +keywords = [ "webassembly", "wasm" ] [lib] name = "cton_wasm" From d71756f298d5140e86faa0385be4197a17159a88 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Feb 2018 03:43:55 -0800 Subject: [PATCH 1555/3084] Delete an extraneous file. --- lib/cretonne/src/isa/riscv/registers.py | 1 - 1 file changed, 1 deletion(-) delete mode 100644 lib/cretonne/src/isa/riscv/registers.py diff --git a/lib/cretonne/src/isa/riscv/registers.py b/lib/cretonne/src/isa/riscv/registers.py deleted file mode 100644 index 8b13789179..0000000000 --- a/lib/cretonne/src/isa/riscv/registers.py +++ /dev/null @@ -1 +0,0 @@ - From 9010e576dd047edff867f29c71303c9b4893ec2d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Feb 2018 10:10:42 -0800 Subject: [PATCH 1556/3084] Simply instruction result value handling in the parser. Also, move the handling earlier so that they're checked before the rest of the instruction is checked. --- lib/reader/src/parser.rs | 21 +++++---------------- 1 file changed, 5 insertions(+), 16 deletions(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 43c149956a..b4252d4cb9 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1632,6 +1632,11 @@ impl<'a> Parser<'a> { ctx: &mut Context, ebb: Ebb, ) -> Result<()> { + // Define the result values. + for val in &results { + ctx.map.def_value(*val, &self.loc)?; + } + // Collect comments for the next instruction. self.start_gathering_comments(); @@ -1709,11 +1714,6 @@ impl<'a> Parser<'a> { } } - // Now map the source result values to the just created instruction results. - // Pass a reference to `ctx.values` instead of `ctx` itself since the `Values` iterator - // holds a reference to `ctx.function`. - self.add_values(&mut ctx.map, results.into_iter())?; - if let Some(result_locations) = result_locations { for (&value, loc) in ctx.function.dfg.inst_results(inst).iter().zip( result_locations, @@ -1798,17 +1798,6 @@ impl<'a> Parser<'a> { Ok(ctrl_type) } - // Add mappings for a list of source values to their corresponding new values. - fn add_values(&self, map: &mut SourceMap, new_results: V) -> Result<()> - where - V: Iterator, - { - for val in new_results { - map.def_value(val, &self.loc)?; - } - Ok(()) - } - // Parse comma-separated value list into a VariableArgs struct. // // value_list ::= [ value { "," value } ] From ab9298eafa835b19388538d5aa9b1de87150cca2 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Feb 2018 10:12:40 -0800 Subject: [PATCH 1557/3084] Make the `fst` recipe use the deref-safe register class as well. --- cranelift/filetests/isa/intel/binary64-float.cton | 10 ++++++++++ lib/cretonne/meta/isa/intel/recipes.py | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index a6f47e53db..69bfeb6c36 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -20,6 +20,7 @@ ebb0: [-,%rsi] v1 = iconst.i32 2 [-,%rax] v2 = iconst.i64 11 [-,%r14] v3 = iconst.i64 12 + [-,%r13] v4 = iconst.i64 13 ; asm: cvtsi2ssl %r11d, %xmm5 [-,%xmm5] v10 = fcvt_from_sint.f32 v0 ; bin: f3 41 0f 2a eb @@ -173,6 +174,10 @@ ebb0: [-] store.f32 v100, v3 ; bin: 66 41 0f 7e 2e ; asm: movd %xmm10, (%rax) [-] store.f32 v101, v2 ; bin: 66 44 0f 7e 10 + ; asm: movd %xmm5, (%r13) + [-] store.f32 v100, v4 ; bin: 66 41 0f 7e 6d 00 + ; asm: movd %xmm10, (%r13) + [-] store.f32 v101, v4 ; bin: 66 45 0f 7e 55 00 ; asm: movd %xmm5, 50(%r14) [-] store.f32 v100, v3+50 ; bin: 66 41 0f 7e 6e 32 ; asm: movd %xmm10, -50(%rax) @@ -250,6 +255,7 @@ ebb0: [-,%rsi] v1 = iconst.i32 2 [-,%rax] v2 = iconst.i64 11 [-,%r14] v3 = iconst.i64 12 + [-,%r13] v4 = iconst.i64 13 ; asm: cvtsi2sdl %r11d, %xmm5 [-,%xmm5] v10 = fcvt_from_sint.f64 v0 ; bin: f2 41 0f 2a eb @@ -403,6 +409,10 @@ ebb0: [-] store.f64 v100, v3 ; bin: 66 41 0f d6 2e ; asm: movq %xmm10, (%rax) [-] store.f64 v101, v2 ; bin: 66 44 0f d6 10 + ; asm: movq %xmm5, (%r13) + [-] store.f64 v100, v4 ; bin: 66 41 0f d6 6d 00 + ; asm: movq %xmm10, (%r13) + [-] store.f64 v101, v4 ; bin: 66 45 0f d6 55 00 ; asm: movq %xmm5, 50(%r14) [-] store.f64 v100, v3+50 ; bin: 66 41 0f d6 6e 32 ; asm: movq %xmm10, -50(%rax) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 965bc4331d..133837fc78 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -675,7 +675,7 @@ st_abcd = TailRecipe( # XX /r register-indirect store of FPR with no offset. fst = TailRecipe( - 'fst', Store, size=1, ins=(FPR, GPR), outs=(), + 'fst', Store, size=1, ins=(FPR, GPR_ZERO_DEREF_SAFE), outs=(), instp=IsEqual(Store.offset, 0), clobbers_flags=False, emit=''' From 6fcbb20e10a7e489deda28191ca8fabcc19ea7ce Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Feb 2018 10:31:00 -0800 Subject: [PATCH 1558/3084] Bump version to 0.3.0 --- cranelift/Cargo.toml | 12 ++++++------ cranelift/publish-all.sh | 2 +- lib/cretonne/Cargo.toml | 2 +- lib/frontend/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/wasm/Cargo.toml | 6 +++--- 7 files changed, 17 insertions(+), 17 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 20ec002a29..d762d64d82 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.2.0" +version = "0.3.0" description = "Binaries for testing the Cretonne library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,11 +13,11 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne = { path = "lib/cretonne", version = "0.2.0" } -cretonne-reader = { path = "lib/reader", version = "0.2.0" } -cretonne-frontend = { path = "lib/frontend", version = "0.2.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.2.0" } -cretonne-native = { path = "lib/native", version = "0.2.0" } +cretonne = { path = "lib/cretonne", version = "0.3.0" } +cretonne-reader = { path = "lib/reader", version = "0.3.0" } +cretonne-frontend = { path = "lib/frontend", version = "0.3.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.3.0" } +cretonne-native = { path = "lib/native", version = "0.3.0" } filecheck = { path = "lib/filecheck" } docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 5cdc281387..e8b22b4f5d 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -5,7 +5,7 @@ topdir="$(pwd)" # All the cretonne-* crates have the same version number # The filecheck crate version is managed independently. -version="0.2.0" +version="0.3.0" # Update all of the Cargo.toml files. # diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 5dea4bb3f7..12ddbf2abe 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.2.0" +version = "0.3.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 7ef8cee698..67297562b8 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.2.0" +version = "0.3.0" description = "Cretonne IL builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,4 +12,4 @@ readme = "README.md" name = "cton_frontend" [dependencies] -cretonne = { path = "../cretonne", version = "0.2.0" } +cretonne = { path = "../cretonne", version = "0.3.0" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 4401e41aab..1fe35a7190 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.2.0" +version = "0.3.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -11,7 +11,7 @@ readme = "README.md" name = "cton_native" [dependencies] -cretonne = { path = "../cretonne", version = "0.2.0" } +cretonne = { path = "../cretonne", version = "0.3.0" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index a732eda896..14b721af35 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.2.0" +version = "0.3.0" description = "Cretonne textual IL reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,4 +12,4 @@ readme = "README.md" name = "cton_reader" [dependencies] -cretonne = { path = "../cretonne", version = "0.2.0" } +cretonne = { path = "../cretonne", version = "0.3.0" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index cd48f36959..45e153357c 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.2.0" +version = "0.3.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IL" repository = "https://github.com/Cretonne/cretonne" @@ -13,8 +13,8 @@ name = "cton_wasm" [dependencies] wasmparser = "0.14.1" -cretonne = { path = "../cretonne", version = "0.2.0" } -cretonne-frontend = { path = "../frontend", version = "0.2.0" } +cretonne = { path = "../cretonne", version = "0.3.0" } +cretonne-frontend = { path = "../frontend", version = "0.3.0" } [dev-dependencies] tempdir = "0.3.5" From e4b30d328467c3e30976a4a55710b79bd3229bb7 Mon Sep 17 00:00:00 2001 From: Nathan Froyd Date: Wed, 28 Feb 2018 14:33:28 -0500 Subject: [PATCH 1559/3084] fix typo in build.rs --- lib/cretonne/build.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/build.rs b/lib/cretonne/build.rs index 4d1b0edf72..d2ef3905b1 100644 --- a/lib/cretonne/build.rs +++ b/lib/cretonne/build.rs @@ -46,7 +46,7 @@ fn main() { let cur_dir = env::current_dir().expect("Can't access current working directory"); let crate_dir = cur_dir.as_path(); - // Make sure we rebuild is this build script changes. + // Make sure we rebuild if this build script changes. // I guess that won't happen if you have non-UTF8 bytes in your path names. // The `build.py` script prints out its own dependencies. println!( From 7054f25abbb009796e4009661f62fb33c130cdaa Mon Sep 17 00:00:00 2001 From: Julian Seward Date: Wed, 28 Feb 2018 12:28:55 +0100 Subject: [PATCH 1560/3084] Adds support to transform integer div and rem by constants into cheaper equivalents. Adds support for transforming integer division and remainder by constants into sequences that do not involve division instructions. * div/rem by constant powers of two are turned into right shifts, plus some fixups for the signed cases. * div/rem by constant non-powers of two are turned into double length multiplies by a magic constant, plus some fixups involving shifts, addition and subtraction, that depends on the constant, the word size and the signedness involved. * The following cases are transformed: div and rem, signed or unsigned, 32 or 64 bit. The only un-transformed cases are: unsigned div and rem by zero, signed div and rem by zero or -1. * This is all incorporated within a new transformation pass, "preopt", in lib/cretonne/src/preopt.rs. * In preopt.rs, fn do_preopt() is the main driver. It is designed to be extensible to transformations of other kinds of instructions. Currently it merely uses a helper to identify div/rem transformation candidates and another helper to perform the transformation. * In preopt.rs, fn get_div_info() pattern matches to find candidates, both cases where the second arg is an immediate, and cases where the second arg is an identifier bound to an immediate at its definition point. * In preopt.rs, fn do_divrem_transformation() does the heavy lifting of the transformation proper. It in turn uses magic{S,U}{32,64} to calculate the magic numbers required for the transformations. * There are many test cases for the transformation proper: filetests/preopt/div_by_const_non_power_of_2.cton filetests/preopt/div_by_const_power_of_2.cton filetests/preopt/rem_by_const_non_power_of_2.cton filetests/preopt/rem_by_const_power_of_2.cton filetests/preopt/div_by_const_indirect.cton preopt.rs also contains a set of tests for magic number generation. * The main (non-power-of-2) transformation requires instructions that return the high word of a double-length multiply. For this, instructions umulhi and smulhi have been added to the core instruction set. These will map directly to single instructions on most non-intel targets. * intel does not have an instruction exactly like that. For intel, instructions x86_umulx and x86_smulx have been added. These map to real instructions and return both result words. The intel legaliser will rewrite {s,u}mulhi into x86_{s,u}mulx uses that throw away the lower half word. Tests: filetests/isa/intel/legalize-mulhi.cton (new file) filetests/isa/intel/binary64.cton (added x86_{s,u}mulx encoding tests) --- cranelift/filetests/isa/intel/binary64.cton | 22 + .../filetests/isa/intel/legalize-mulhi.cton | 45 ++ .../preopt/div_by_const_indirect.cton | 60 ++ .../preopt/div_by_const_non_power_of_2.cton | 267 +++++++++ .../preopt/div_by_const_power_of_2.cton | 293 ++++++++++ .../preopt/rem_by_const_non_power_of_2.cton | 286 +++++++++ .../preopt/rem_by_const_power_of_2.cton | 292 ++++++++++ cranelift/src/filetest/mod.rs | 2 + cranelift/src/filetest/preopt.rs | 50 ++ lib/cretonne/meta/base/instructions.py | 20 + lib/cretonne/meta/isa/intel/encodings.py | 3 + lib/cretonne/meta/isa/intel/instructions.py | 22 + lib/cretonne/meta/isa/intel/legalize.py | 17 + lib/cretonne/meta/isa/intel/recipes.py | 9 + lib/cretonne/src/context.rs | 9 + lib/cretonne/src/divconst_magic_numbers.rs | 542 ++++++++++++++++++ lib/cretonne/src/isa/intel/enc_tables.rs | 1 + lib/cretonne/src/lib.rs | 2 + lib/cretonne/src/preopt.rs | 521 +++++++++++++++++ lib/cretonne/src/timing.rs | 1 + 20 files changed, 2464 insertions(+) create mode 100644 cranelift/filetests/isa/intel/legalize-mulhi.cton create mode 100644 cranelift/filetests/preopt/div_by_const_indirect.cton create mode 100644 cranelift/filetests/preopt/div_by_const_non_power_of_2.cton create mode 100644 cranelift/filetests/preopt/div_by_const_power_of_2.cton create mode 100644 cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton create mode 100644 cranelift/filetests/preopt/rem_by_const_power_of_2.cton create mode 100644 cranelift/src/filetest/preopt.rs create mode 100644 lib/cretonne/src/divconst_magic_numbers.rs create mode 100644 lib/cretonne/src/preopt.rs diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 0b546961d8..b579b4d58d 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -336,6 +336,28 @@ ebb0: ; asm: divq %r10 [-,%rax,%rdx] v202, v203 = x86_udivmodx v190, v191, v3 ; bin: 49 f7 f2 + ; double-length multiply instructions, 64 bit + [-,%rax] v1001 = iconst.i64 1 + [-,%r15] v1002 = iconst.i64 2 + ; asm: mulq %r15 + [-,%rax,%rdx] v1003, v1004 = x86_umulx v1001, v1002 ; bin: 49 f7 e7 + ; asm: imulq %r15 + [-,%rax,%rdx] v1005, v1006 = x86_smulx v1001, v1002 ; bin: 49 f7 ef + + ; double-length multiply instructions, 32 bit + [-,%rax] v1011 = iconst.i32 1 + [-,%r15] v1012 = iconst.i32 2 + [-,%rcx] v1017 = iconst.i32 3 + ; asm: mull %r15d + [-,%rax,%rdx] v1013, v1014 = x86_umulx v1011, v1012 ; bin: 41 f7 e7 + ; asm: imull %r15d + [-,%rax,%rdx] v1015, v1016 = x86_smulx v1011, v1012 ; bin: 41 f7 ef + + ; asm: mull %ecx + [-,%rax,%rdx] v1018, v1019 = x86_umulx v1011, v1017 ; bin: f7 e1 + ; asm: imull %ecx + [-,%rax,%rdx] v1020, v1021 = x86_smulx v1011, v1017 ; bin: f7 e9 + ; Bit-counting instructions. ; asm: popcntq %rsi, %rcx diff --git a/cranelift/filetests/isa/intel/legalize-mulhi.cton b/cranelift/filetests/isa/intel/legalize-mulhi.cton new file mode 100644 index 0000000000..673a19db3b --- /dev/null +++ b/cranelift/filetests/isa/intel/legalize-mulhi.cton @@ -0,0 +1,45 @@ + +test compile +set is_64bit +isa intel baseline + +; umulhi/smulhi on 64 bit operands + +function %i64_umulhi(i64, i64) -> i64 { +ebb0(v10: i64, v11: i64): + v12 = umulhi v10, v11 + ; check: %rdi -> %rax + ; check: x86_umulx + ; check: %rdx -> %rax + return v12 +} + +function %i64_smulhi(i64, i64) -> i64 { +ebb0(v20: i64, v21: i64): + v22 = smulhi v20, v21 + ; check: %rdi -> %rax + ; check: x86_smulx + ; check: %rdx -> %rax + return v22 +} + + +; umulhi/smulhi on 32 bit operands + +function %i32_umulhi(i32, i32) -> i32 { +ebb0(v30: i32, v31: i32): + v32 = umulhi v30, v31 + ; check: %rdi -> %rax + ; check: x86_umulx + ; check: %rdx -> %rax + return v32 +} + +function %i32_smulhi(i32, i32) -> i32 { +ebb0(v40: i32, v41: i32): + v42 = smulhi v40, v41 + ; check: %rdi -> %rax + ; check: x86_smulx + ; check: %rdx -> %rax + return v42 +} diff --git a/cranelift/filetests/preopt/div_by_const_indirect.cton b/cranelift/filetests/preopt/div_by_const_indirect.cton new file mode 100644 index 0000000000..ccc83cd49b --- /dev/null +++ b/cranelift/filetests/preopt/div_by_const_indirect.cton @@ -0,0 +1,60 @@ + +test preopt +isa intel baseline + +; Cases where the denominator is created by an iconst + +function %indir_udiv32(i32) -> i32 { +ebb0(v0: i32): + v1 = iconst.i32 7 + v2 = udiv v0, v1 + ; check: iconst.i32 7 + ; check: iconst.i32 0x2492_4925 + ; check: umulhi v0, v3 + ; check: isub v0, v4 + ; check: ushr_imm v5, 1 + ; check: iadd v6, v4 + ; check: ushr_imm v7, 2 + ; check: copy v8 + return v2 +} + +function %indir_sdiv32(i32) -> i32 { +ebb0(v0: i32): + v1 = iconst.i32 -17 + v2 = sdiv v0, v1 + ; check: iconst.i32 -17 + ; check: iconst.i32 0xffff_ffff_8787_8787 + ; check: smulhi v0, v3 + ; check: sshr_imm v4, 3 + ; check: ushr_imm v5, 31 + ; check: iadd v5, v6 + ; check: copy v7 + return v2 +} + +function %indir_udiv64(i64) -> i64 { +ebb0(v0: i64): + v1 = iconst.i64 1337 + v2 = udiv v0, v1 + ; check: iconst.i64 1337 + ; check: iconst.i64 0xc411_9d95_2866_a139 + ; check: umulhi v0, v3 + ; check: ushr_imm v4, 10 + ; check: copy v5 + return v2 +} + +function %indir_sdiv64(i64) -> i64 { +ebb0(v0: i64): + v1 = iconst.i64 -90210 + v2 = sdiv v0, v1 + ; check: iconst.i64 0xffff_ffff_fffe_9f9e + ; check: iconst.i64 0xd181_4ee8_939c_b8bb + ; check: smulhi v0, v3 + ; check: sshr_imm v4, 14 + ; check: ushr_imm v5, 63 + ; check: iadd v5, v6 + ; check: copy v7 + return v2 +} diff --git a/cranelift/filetests/preopt/div_by_const_non_power_of_2.cton b/cranelift/filetests/preopt/div_by_const_non_power_of_2.cton new file mode 100644 index 0000000000..18811fcd82 --- /dev/null +++ b/cranelift/filetests/preopt/div_by_const_non_power_of_2.cton @@ -0,0 +1,267 @@ + +test preopt +isa intel baseline + +; -------- U32 -------- + +; complex case (mul, sub, shift, add, shift) +function %t_udiv32_p7(i32) -> i32 { +ebb0(v0: i32): + v1 = udiv_imm v0, 7 + ; check: iconst.i32 0x2492_4925 + ; check: umulhi v0, v2 + ; check: isub v0, v3 + ; check: ushr_imm v4, 1 + ; check: iadd v5, v3 + ; check: ushr_imm v6, 2 + ; check: copy v7 + return v1 +} + +; simple case (mul, shift) +function %t_udiv32_p125(i32) -> i32 { +ebb0(v0: i32): + v1 = udiv_imm v0, 125 + ; check: iconst.i32 0x1062_4dd3 + ; check: umulhi v0, v2 + ; check: ushr_imm v3, 3 + ; check: copy v4 + return v1 +} + +; simple case w/ shift by zero (mul) +function %t_udiv32_p641(i32) -> i32 { +ebb0(v0: i32): + v1 = udiv_imm v0, 641 + ; check: iconst.i32 0x0066_3d81 + ; check: umulhi v0, v2 + ; check: copy v3 + return v1 +} + + +; -------- S32 -------- + +; simple case w/ shift by zero (mul, add-sign-bit) +function %t_sdiv32_n6(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, -6 + ; check: iconst.i32 0xffff_ffff_d555_5555 + ; check: smulhi v0, v2 + ; check: ushr_imm v3, 31 + ; check: iadd v3, v4 + ; check: copy v5 + return v1 +} + +; simple case (mul, shift, add-sign-bit) +function %t_sdiv32_n5(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, -5 + ; check: iconst.i32 0xffff_ffff_9999_9999 + ; check: smulhi v0, v2 + ; check: sshr_imm v3, 1 + ; check: ushr_imm v4, 31 + ; check: iadd v4, v5 + ; check: copy v6 + return v1 +} + +; case d < 0 && M > 0 (mul, sub, shift, add-sign-bit) +function %t_sdiv32_n3(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, -3 + ; check: iconst.i32 0x5555_5555 + ; check: smulhi v0, v2 + ; check: isub v3, v0 + ; check: sshr_imm v4, 1 + ; check: ushr_imm v5, 31 + ; check: iadd v5, v6 + ; check: copy v7 + return v1 +} + +; simple case w/ shift by zero (mul, add-sign-bit) +function %t_sdiv32_p6(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, 6 + ; check: iconst.i32 0x2aaa_aaab + ; check: smulhi v0, v2 + ; check: ushr_imm v3, 31 + ; check: iadd v3, v4 + ; check: copy v5 + return v1 +} + +; case d > 0 && M < 0 (mull, add, shift, add-sign-bit) +function %t_sdiv32_p7(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, 7 + ; check: iconst.i32 0xffff_ffff_9249_2493 + ; check: smulhi v0, v2 + ; check: iadd v3, v0 + ; check: sshr_imm v4, 2 + ; check: ushr_imm v5, 31 + ; check: iadd v5, v6 + ; check: copy v7 + return v1 +} + +; simple case (mul, shift, add-sign-bit) +function %t_sdiv32_p625(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, 625 + ; check: iconst.i32 0x68db_8bad + ; check: smulhi v0, v2 + ; check: sshr_imm v3, 8 + ; check: ushr_imm v4, 31 + ; check: iadd v4, v5 + ; check: copy v6 + return v1 +} + + +; -------- U64 -------- + +; complex case (mul, sub, shift, add, shift) +function %t_udiv64_p7(i64) -> i64 { +ebb0(v0: i64): + v1 = udiv_imm v0, 7 + ; check: iconst.i64 0x2492_4924_9249_2493 + ; check: umulhi v0, v2 + ; check: isub v0, v3 + ; check: ushr_imm v4, 1 + ; check: iadd v5, v3 + ; check: ushr_imm v6, 2 + ; check: copy v7 + return v1 +} + +; simple case (mul, shift) +function %t_udiv64_p9(i64) -> i64 { +ebb0(v0: i64): + v1 = udiv_imm v0, 9 + ; check: iconst.i64 0xe38e_38e3_8e38_e38f + ; check: umulhi v0, v2 + ; check: ushr_imm v3, 3 + ; check: copy v4 + return v1 +} + +; complex case (mul, sub, shift, add, shift) +function %t_udiv64_p125(i64) -> i64 { +ebb0(v0: i64): + v1 = udiv_imm v0, 125 + ; check: iconst.i64 0x0624_dd2f_1a9f_be77 + ; check: umulhi v0, v2 + ; check: isub v0, v3 + ; check: ushr_imm v4, 1 + ; check: iadd v5, v3 + ; check: ushr_imm v6, 6 + ; check: copy v7 + return v1 +} + +; simple case w/ shift by zero (mul) +function %t_udiv64_p274177(i64) -> i64 { +ebb0(v0: i64): + v1 = udiv_imm v0, 274177 + ; check: iconst.i64 0x3d30_f19c_d101 + ; check: umulhi v0, v2 + ; check: copy v3 + return v1 +} + + +; -------- S64 -------- + +; simple case (mul, shift, add-sign-bit) +function %t_sdiv64_n625(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, -625 + ; check: iconst.i64 0xcb92_3a29_c779_a6b5 + ; check: smulhi v0, v2 + ; check: sshr_imm v3, 7 + ; check: ushr_imm v4, 63 + ; check: iadd v4, v5 + ; check: copy v6 + return v1 +} + +; simple case w/ zero shift (mul, add-sign-bit) +function %t_sdiv64_n6(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, -6 + ; check: iconst.i64 0xd555_5555_5555_5555 + ; check: smulhi v0, v2 + ; check: ushr_imm v3, 63 + ; check: iadd v3, v4 + ; check: copy v5 + return v1 +} + +; simple case w/ zero shift (mul, add-sign-bit) +function %t_sdiv64_n5(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, -5 + ; check: iconst.i64 0x9999_9999_9999_9999 + ; check: smulhi v0, v2 + ; check: sshr_imm v3, 1 + ; check: ushr_imm v4, 63 + ; check: iadd v4, v5 + ; check: copy v6 + return v1 +} + +; case d < 0 && M > 0 (mul, sub, shift, add-sign-bit) +function %t_sdiv64_n3(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, -3 + ; check: iconst.i64 0x5555_5555_5555_5555 + ; check: smulhi v0, v2 + ; check: isub v3, v0 + ; check: sshr_imm v4, 1 + ; check: ushr_imm v5, 63 + ; check: iadd v5, v6 + ; check: copy v7 + return v1 +} + +; simple case w/ zero shift (mul, add-sign-bit) +function %t_sdiv64_p6(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, 6 + ; check: iconst.i64 0x2aaa_aaaa_aaaa_aaab + ; check: smulhi v0, v2 + ; check: ushr_imm v3, 63 + ; check: iadd v3, v4 + ; check: copy v5 + return v1 +} + +; case d > 0 && M < 0 (mul, add, shift, add-sign-bit) +function %t_sdiv64_p15(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, 15 + ; check: iconst.i64 0x8888_8888_8888_8889 + ; check: smulhi v0, v2 + ; check: iadd v3, v0 + ; check: sshr_imm v4, 3 + ; check: ushr_imm v5, 63 + ; check: iadd v5, v6 + ; check: copy v7 + return v1 +} + +; simple case (mul, shift, add-sign-bit) +function %t_sdiv64_p625(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, 625 + ; check: iconst.i64 0x346d_c5d6_3886_594b + ; check: smulhi v0, v2 + ; check: sshr_imm v3, 7 + ; check: ushr_imm v4, 63 + ; check: iadd v4, v5 + ; check: copy v6 + return v1 +} diff --git a/cranelift/filetests/preopt/div_by_const_power_of_2.cton b/cranelift/filetests/preopt/div_by_const_power_of_2.cton new file mode 100644 index 0000000000..dc51c5395d --- /dev/null +++ b/cranelift/filetests/preopt/div_by_const_power_of_2.cton @@ -0,0 +1,293 @@ + +test preopt +isa intel baseline + +; -------- U32 -------- + +; ignored +function %t_udiv32_p0(i32) -> i32 { +ebb0(v0: i32): + v1 = udiv_imm v0, 0 + ; check: udiv_imm v0, 0 + return v1 +} + +; converted to a copy +function %t_udiv32_p1(i32) -> i32 { +ebb0(v0: i32): + v1 = udiv_imm v0, 1 + ; check: copy v0 + return v1 +} + +; shift +function %t_udiv32_p2(i32) -> i32 { +ebb0(v0: i32): + v1 = udiv_imm v0, 2 + ; check: ushr_imm v0, 1 + return v1 +} + +; shift +function %t_udiv32_p2p31(i32) -> i32 { +ebb0(v0: i32): + v1 = udiv_imm v0, 0x8000_0000 + ; check: ushr_imm v0, 31 + return v1 +} + + +; -------- U64 -------- + +; ignored +function %t_udiv64_p0(i64) -> i64 { +ebb0(v0: i64): + v1 = udiv_imm v0, 0 + ; check: udiv_imm v0, 0 + return v1 +} + +; converted to a copy +function %t_udiv64_p1(i64) -> i64 { +ebb0(v0: i64): + v1 = udiv_imm v0, 1 + ; check: copy v0 + return v1 +} + +; shift +function %t_udiv64_p2(i64) -> i64 { +ebb0(v0: i64): + v1 = udiv_imm v0, 2 + ; check: ushr_imm v0, 1 + return v1 +} + +; shift +function %t_udiv64_p2p63(i64) -> i64 { +ebb0(v0: i64): + v1 = udiv_imm v0, 0x8000_0000_0000_0000 + ; check: ushr_imm v0, 63 + return v1 +} + + +; -------- S32 -------- + +; ignored +function %t_sdiv32_p0(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, 0 + ; check: sdiv_imm v0, 0 + return v1 +} + +; converted to a copy +function %t_sdiv32_p1(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, 1 + ; check: copy v0 + return v1 +} + +; ignored +function %t_sdiv32_n1(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, -1 + ; check: sdiv_imm v0, -1 + return v1 +} + +; shift +function %t_sdiv32_p2(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, 2 + ; check: ushr_imm v0, 31 + ; check: iadd v0, v2 + ; check: sshr_imm v3, 1 + ; check: copy v4 + return v1 +} + +; shift +function %t_sdiv32_n2(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, -2 + ; check: ushr_imm v0, 31 + ; check: iadd v0, v2 + ; check: sshr_imm v3, 1 + ; check: irsub_imm v4, 0 + return v1 +} + +; shift +function %t_sdiv32_p4(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, 4 + ; check: v2 = sshr_imm v0, 1 + ; check: ushr_imm v2, 30 + ; check: iadd v0, v3 + ; check: sshr_imm v4, 2 + ; check: copy v5 + + return v1 +} + +; shift +function %t_sdiv32_n4(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, -4 + ; check: sshr_imm v0, 1 + ; check: ushr_imm v2, 30 + ; check: iadd v0, v3 + ; check: sshr_imm v4, 2 + ; check: irsub_imm v5, 0 + return v1 +} + +; shift +function %t_sdiv32_p2p30(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, 0x4000_0000 + ; check: sshr_imm v0, 29 + ; check: ushr_imm v2, 2 + ; check: iadd v0, v3 + ; check: sshr_imm v4, 30 + ; check: copy v5 + return v1 +} + +; shift +function %t_sdiv32_n2p30(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, -0x4000_0000 + ; check: sshr_imm v0, 29 + ; check: ushr_imm v2, 2 + ; check: iadd v0, v3 + ; check: sshr_imm v4, 30 + ; check: irsub_imm v5, 0 + return v1 +} + +; there's no positive version of this, since -(-0x8000_0000) isn't +; representable. +function %t_sdiv32_n2p31(i32) -> i32 { +ebb0(v0: i32): + v1 = sdiv_imm v0, -0x8000_0000 + ; check: sshr_imm v0, 30 + ; check: ushr_imm v2, 1 + ; check: iadd v0, v3 + ; check: sshr_imm v4, 31 + ; check: irsub_imm v5, 0 + return v1 +} + + +; -------- S64 -------- + +; ignored +function %t_sdiv64_p0(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, 0 + ; check: sdiv_imm v0, 0 + return v1 +} + +; converted to a copy +function %t_sdiv64_p1(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, 1 + ; check: copy v0 + return v1 +} + +; ignored +function %t_sdiv64_n1(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, -1 + ; check: sdiv_imm v0, -1 + return v1 +} + +; shift +function %t_sdiv64_p2(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, 2 + ; check: ushr_imm v0, 63 + ; check: iadd v0, v2 + ; check: sshr_imm v3, 1 + ; check: copy v4 + return v1 +} + +; shift +function %t_sdiv64_n2(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, -2 + ; check: ushr_imm v0, 63 + ; check: iadd v0, v2 + ; check: sshr_imm v3, 1 + ; check: irsub_imm v4, 0 + return v1 +} + +; shift +function %t_sdiv64_p4(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, 4 + ; check: sshr_imm v0, 1 + ; check: ushr_imm v2, 62 + ; check: iadd v0, v3 + ; check: sshr_imm v4, 2 + ; check: copy v5 + return v1 +} + +; shift +function %t_sdiv64_n4(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, -4 + ; check: sshr_imm v0, 1 + ; check: ushr_imm v2, 62 + ; check: iadd v0, v3 + ; check: sshr_imm v4, 2 + ; check: irsub_imm v5, 0 + return v1 +} + +; shift +function %t_sdiv64_p2p62(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, 0x4000_0000_0000_0000 + ; check: sshr_imm v0, 61 + ; check: ushr_imm v2, 2 + ; check: iadd v0, v3 + ; check: sshr_imm v4, 62 + ; check: copy v5 + return v1 +} + +; shift +function %t_sdiv64_n2p62(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, -0x4000_0000_0000_0000 + ; check: sshr_imm v0, 61 + ; check: ushr_imm v2, 2 + ; check: iadd v0, v3 + ; check: sshr_imm v4, 62 + ; check: irsub_imm v5, 0 + return v1 +} + +; there's no positive version of this, since -(-0x8000_0000_0000_0000) isn't +; representable. +function %t_sdiv64_n2p63(i64) -> i64 { +ebb0(v0: i64): + v1 = sdiv_imm v0, -0x8000_0000_0000_0000 + ; check: sshr_imm v0, 62 + ; check: ushr_imm v2, 1 + ; check: iadd v0, v3 + ; check: sshr_imm v4, 63 + ; check: irsub_imm v5, 0 + return v1 +} diff --git a/cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton b/cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton new file mode 100644 index 0000000000..c142a16359 --- /dev/null +++ b/cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton @@ -0,0 +1,286 @@ + +test preopt +isa intel baseline + +; -------- U32 -------- + +; complex case (mul, sub, shift, add, shift) +function %t_urem32_p7(i32) -> i32 { +ebb0(v0: i32): + v1 = urem_imm v0, 7 + ; check: iconst.i32 0x2492_4925 + ; check: umulhi v0, v2 + ; check: isub v0, v3 + ; check: ushr_imm v4, 1 + ; check: iadd v5, v3 + ; check: ushr_imm v6, 2 + ; check: imul_imm v7, 7 + ; check: isub v0, v8 + return v1 +} + +; simple case (mul, shift) +function %t_urem32_p125(i32) -> i32 { +ebb0(v0: i32): + v1 = urem_imm v0, 125 + ; check: iconst.i32 0x1062_4dd3 + ; check: umulhi v0, v2 + ; check: ushr_imm v3, 3 + ; check: imul_imm v4, 125 + ; check: isub v0, v5 + return v1 +} + +; simple case w/ shift by zero (mul) +function %t_urem32_p641(i32) -> i32 { +ebb0(v0: i32): + v1 = urem_imm v0, 641 + ; check: iconst.i32 0x0066_3d81 + ; check: umulhi v0, v2 + ; check: imul_imm v3, 641 + ; check: isub v0, v4 + return v1 +} + + +; -------- S32 -------- + +; simple case w/ shift by zero (mul, add-sign-bit) +function %t_srem32_n6(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, -6 + ; check: iconst.i32 0xffff_ffff_d555_5555 + ; check: smulhi v0, v2 + ; check: ushr_imm v3, 31 + ; check: iadd v3, v4 + ; check: imul_imm v5, -6 + ; check: isub v0, v6 + return v1 +} + +; simple case (mul, shift, add-sign-bit) +function %t_srem32_n5(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, -5 + ; check: iconst.i32 0xffff_ffff_9999_9999 + ; check: smulhi v0, v2 + ; check: sshr_imm v3, 1 + ; check: ushr_imm v4, 31 + ; check: iadd v4, v5 + ; check: imul_imm v6, -5 + ; check: isub v0, v7 + return v1 +} + +; case d < 0 && M > 0 (mul, sub, shift, add-sign-bit) +function %t_srem32_n3(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, -3 + ; check: iconst.i32 0x5555_5555 + ; check: smulhi v0, v2 + ; check: isub v3, v0 + ; check: sshr_imm v4, 1 + ; check: ushr_imm v5, 31 + ; check: iadd v5, v6 + ; check: imul_imm v7, -3 + ; check: isub v0, v8 + return v1 +} + +; simple case w/ shift by zero (mul, add-sign-bit) +function %t_srem32_p6(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, 6 + ; check: iconst.i32 0x2aaa_aaab + ; check: smulhi v0, v2 + ; check: ushr_imm v3, 31 + ; check: iadd v3, v4 + ; check: imul_imm v5, 6 + ; check: isub v0, v6 + return v1 +} + +; case d > 0 && M < 0 (mull, add, shift, add-sign-bit) +function %t_srem32_p7(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, 7 + ; check: iconst.i32 0xffff_ffff_9249_2493 + ; check: smulhi v0, v2 + ; check: iadd v3, v0 + ; check: sshr_imm v4, 2 + ; check: ushr_imm v5, 31 + ; check: iadd v5, v6 + ; check: imul_imm v7, 7 + ; check: isub v0, v8 + return v1 +} + +; simple case (mul, shift, add-sign-bit) +function %t_srem32_p625(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, 625 + ; check: iconst.i32 0x68db_8bad + ; check: smulhi v0, v2 + ; check: sshr_imm v3, 8 + ; check: ushr_imm v4, 31 + ; check: iadd v4, v5 + ; check: imul_imm v6, 625 + ; check: isub v0, v7 + return v1 +} + + +; -------- U64 -------- + +; complex case (mul, sub, shift, add, shift) +function %t_urem64_p7(i64) -> i64 { +ebb0(v0: i64): + v1 = urem_imm v0, 7 + ; check: umulhi v0, v2 + ; check: isub v0, v3 + ; check: ushr_imm v4, 1 + ; check: iadd v5, v3 + ; check: ushr_imm v6, 2 + ; check: imul_imm v7, 7 + ; check: isub v0, v8 + return v1 +} + +; simple case (mul, shift) +function %t_urem64_p9(i64) -> i64 { +ebb0(v0: i64): + v1 = urem_imm v0, 9 + ; check: iconst.i64 0xe38e_38e3_8e38_e38f + ; check: umulhi v0, v2 + ; check: ushr_imm v3, 3 + ; check: imul_imm v4, 9 + ; check: isub v0, v5 + return v1 +} + +; complex case (mul, sub, shift, add, shift) +function %t_urem64_p125(i64) -> i64 { +ebb0(v0: i64): + v1 = urem_imm v0, 125 + ; check: iconst.i64 0x0624_dd2f_1a9f_be77 + ; check: umulhi v0, v2 + ; check: isub v0, v3 + ; check: ushr_imm v4, 1 + ; check: iadd v5, v3 + ; check: ushr_imm v6, 6 + ; check: imul_imm v7, 125 + ; check: isub v0, v8 + return v1 +} + +; simple case w/ shift by zero (mul) +function %t_urem64_p274177(i64) -> i64 { +ebb0(v0: i64): + v1 = urem_imm v0, 274177 + ; check: iconst.i64 0x3d30_f19c_d101 + ; check: umulhi v0, v2 + ; check: imul_imm v3, 0x0004_2f01 + ; check: isub v0, v4 + return v1 +} + + +; -------- S64 -------- + +; simple case (mul, shift, add-sign-bit) +function %t_srem64_n625(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, -625 + ; check: iconst.i64 0xcb92_3a29_c779_a6b5 + ; check: smulhi v0, v2 + ; check: sshr_imm v3, 7 + ; check: ushr_imm v4, 63 + ; check: iadd v4, v5 + ; check: imul_imm v6, -625 + ; check: isub v0, v7 + return v1 +} + +; simple case w/ zero shift (mul, add-sign-bit) +function %t_srem64_n6(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, -6 + ; check: iconst.i64 0xd555_5555_5555_5555 + ; check: smulhi v0, v2 + ; check: ushr_imm v3, 63 + ; check: iadd v3, v4 + ; check: imul_imm v5, -6 + ; check: isub v0, v6 + return v1 +} + +; simple case w/ zero shift (mul, add-sign-bit) +function %t_srem64_n5(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, -5 + ; check: iconst.i64 0x9999_9999_9999_9999 + ; check: smulhi v0, v2 + ; check: sshr_imm v3, 1 + ; check: ushr_imm v4, 63 + ; check: iadd v4, v5 + ; check: imul_imm v6, -5 + ; check: isub v0, v7 + return v1 +} + +; case d < 0 && M > 0 (mul, sub, shift, add-sign-bit) +function %t_srem64_n3(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, -3 + ; check: iconst.i64 0x5555_5555_5555_5555 + ; check: smulhi v0, v2 + ; check: isub v3, v0 + ; check: sshr_imm v4, 1 + ; check: ushr_imm v5, 63 + ; check: iadd v5, v6 + ; check: imul_imm v7, -3 + ; check: isub v0, v8 + return v1 +} + +; simple case w/ zero shift (mul, add-sign-bit) +function %t_srem64_p6(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, 6 + ; check: iconst.i64 0x2aaa_aaaa_aaaa_aaab + ; check: smulhi v0, v2 + ; check: ushr_imm v3, 63 + ; check: iadd v3, v4 + ; check: imul_imm v5, 6 + ; check: isub v0, v6 + return v1 +} + +; case d > 0 && M < 0 (mul, add, shift, add-sign-bit) +function %t_srem64_p15(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, 15 + ; check: iconst.i64 0x8888_8888_8888_8889 + ; check: smulhi v0, v2 + ; check: iadd v3, v0 + ; check: sshr_imm v4, 3 + ; check: ushr_imm v5, 63 + ; check: iadd v5, v6 + ; check: imul_imm v7, 15 + ; check: isub v0, v8 + return v1 +} + +; simple case (mul, shift, add-sign-bit) +function %t_srem64_p625(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, 625 + ; check: iconst.i64 0x346d_c5d6_3886_594b + ; check: smulhi v0, v2 + ; check: sshr_imm v3, 7 + ; check: ushr_imm v4, 63 + ; check: iadd v4, v5 + ; check: imul_imm v6, 625 + ; check: isub v0, v7 + return v1 +} diff --git a/cranelift/filetests/preopt/rem_by_const_power_of_2.cton b/cranelift/filetests/preopt/rem_by_const_power_of_2.cton new file mode 100644 index 0000000000..931623d2e7 --- /dev/null +++ b/cranelift/filetests/preopt/rem_by_const_power_of_2.cton @@ -0,0 +1,292 @@ + +test preopt +isa intel baseline + +; -------- U32 -------- + +; ignored +function %t_urem32_p0(i32) -> i32 { +ebb0(v0: i32): + v1 = urem_imm v0, 0 + ; check: urem_imm v0, 0 + return v1 +} + +; converted to constant zero +function %t_urem32_p1(i32) -> i32 { +ebb0(v0: i32): + v1 = urem_imm v0, 1 + ; check: iconst.i32 0 + return v1 +} + +; shift +function %t_urem32_p2(i32) -> i32 { +ebb0(v0: i32): + v1 = urem_imm v0, 2 + ; check: band_imm v0, 1 + return v1 +} + +; shift +function %t_urem32_p2p31(i32) -> i32 { +ebb0(v0: i32): + v1 = urem_imm v0, 0x8000_0000 + ; check: band_imm v0, 0x7fff_ffff + return v1 +} + + +; -------- U64 -------- + +; ignored +function %t_urem64_p0(i64) -> i64 { +ebb0(v0: i64): + v1 = urem_imm v0, 0 + ; check: urem_imm v0, 0 + return v1 +} + +; converted to constant zero +function %t_urem64_p1(i64) -> i64 { +ebb0(v0: i64): + v1 = urem_imm v0, 1 + ; check: iconst.i64 0 + return v1 +} + +; shift +function %t_urem64_p2(i64) -> i64 { +ebb0(v0: i64): + v1 = urem_imm v0, 2 + ; check: band_imm v0, 1 + return v1 +} + +; shift +function %t_urem64_p2p63(i64) -> i64 { +ebb0(v0: i64): + v1 = urem_imm v0, 0x8000_0000_0000_0000 + ; check: band_imm v0, 0x7fff_ffff_ffff_ffff + return v1 +} + + +; -------- S32 -------- + +; ignored +function %t_srem32_n1(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, -1 + ; check: srem_imm v0, -1 + return v1 +} + +; ignored +function %t_srem32_p0(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, 0 + ; check: srem_imm v0, 0 + return v1 +} + +; converted to constant zero +function %t_srem32_p1(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, 1 + ; check: iconst.i32 0 + return v1 +} + +; shift +function %t_srem32_p2(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, 2 + ; check: ushr_imm v0, 31 + ; check: iadd v0, v2 + ; check: band_imm v3, -2 + ; check: isub v0, v4 + return v1 +} + +; shift +function %t_srem32_n2(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, -2 + ; check: ushr_imm v0, 31 + ; check: iadd v0, v2 + ; check: band_imm v3, -2 + ; check: isub v0, v4 + return v1 +} + +; shift +function %t_srem32_p4(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, 4 + ; check: sshr_imm v0, 1 + ; check: ushr_imm v2, 30 + ; check: iadd v0, v3 + ; check: band_imm v4, -4 + ; check: isub v0, v5 + return v1 +} + +; shift +function %t_srem32_n4(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, -4 + ; check: sshr_imm v0, 1 + ; check: ushr_imm v2, 30 + ; check: iadd v0, v3 + ; check: band_imm v4, -4 + ; check: isub v0, v5 + return v1 +} + +; shift +function %t_srem32_p2p30(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, 0x4000_0000 + ; check: sshr_imm v0, 29 + ; check: ushr_imm v2, 2 + ; check: iadd v0, v3 + ; check: band_imm v4, 0xffff_ffff_c000_0000 + ; check: isub v0, v5 + return v1 +} + +; shift +function %t_srem32_n2p30(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, -0x4000_0000 + ; check: sshr_imm v0, 29 + ; check: ushr_imm v2, 2 + ; check: iadd v0, v3 + ; check: band_imm v4, 0xffff_ffff_c000_0000 + ; check: isub v0, v5 + return v1 +} + +; there's no positive version of this, since -(-0x8000_0000) isn't +; representable. +function %t_srem32_n2p31(i32) -> i32 { +ebb0(v0: i32): + v1 = srem_imm v0, -0x8000_0000 + ; check: sshr_imm v0, 30 + ; check: ushr_imm v2, 1 + ; check: iadd v0, v3 + ; check: band_imm v4, 0xffff_ffff_8000_0000 + ; check: isub v0, v5 + return v1 +} + + +; -------- S64 -------- + +; ignored +function %t_srem64_n1(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, -1 + ; check: srem_imm v0, -1 + return v1 +} + +; ignored +function %t_srem64_p0(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, 0 + ; check: srem_imm v0, 0 + return v1 +} + +; converted to constant zero +function %t_srem64_p1(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, 1 + ; check: iconst.i64 0 + return v1 +} + +; shift +function %t_srem64_p2(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, 2 + ; check: ushr_imm v0, 63 + ; check: iadd v0, v2 + ; check: band_imm v3, -2 + ; check: isub v0, v4 + return v1 +} + +; shift +function %t_srem64_n2(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, -2 + ; check: ushr_imm v0, 63 + ; check: iadd v0, v2 + ; check: band_imm v3, -2 + ; check: isub v0, v4 + return v1 +} + +; shift +function %t_srem64_p4(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, 4 + ; check: sshr_imm v0, 1 + ; check: ushr_imm v2, 62 + ; check: iadd v0, v3 + ; check: band_imm v4, -4 + ; check: isub v0, v5 + return v1 +} + +; shift +function %t_srem64_n4(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, -4 + ; check: sshr_imm v0, 1 + ; check: ushr_imm v2, 62 + ; check: iadd v0, v3 + ; check: band_imm v4, -4 + ; check: isub v0, v5 + return v1 +} + +; shift +function %t_srem64_p2p62(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, 0x4000_0000_0000_0000 + ; check: sshr_imm v0, 61 + ; check: ushr_imm v2, 2 + ; check: iadd v0, v3 + ; check: band_imm v4, 0xc000_0000_0000_0000 + ; check: isub v0, v5 + return v1 +} + +; shift +function %t_srem64_n2p62(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, -0x4000_0000_0000_0000 + ; check: sshr_imm v0, 61 + ; check: ushr_imm v2, 2 + ; check: iadd v0, v3 + ; check: band_imm v4, 0xc000_0000_0000_0000 + ; check: isub v0, v5 + return v1 +} + +; there's no positive version of this, since -(-0x8000_0000_0000_0000) isn't +; representable. +function %t_srem64_n2p63(i64) -> i64 { +ebb0(v0: i64): + v1 = srem_imm v0, -0x8000_0000_0000_0000 + ; check: sshr_imm v0, 62 + ; check: ushr_imm v2, 1 + ; check: iadd v0, v3 + ; check: band_imm v4, 0x8000_0000_0000_0000 + ; check: isub v0, v5 + return v1 +} diff --git a/cranelift/src/filetest/mod.rs b/cranelift/src/filetest/mod.rs index 2d2a9c6cac..5286307811 100644 --- a/cranelift/src/filetest/mod.rs +++ b/cranelift/src/filetest/mod.rs @@ -19,6 +19,7 @@ mod concurrent; mod domtree; mod legalizer; mod licm; +mod preopt; mod regalloc; mod runner; mod runone; @@ -64,6 +65,7 @@ fn new_subtest(parsed: &TestCommand) -> subtest::Result> { "domtree" => domtree::subtest(parsed), "legalizer" => legalizer::subtest(parsed), "licm" => licm::subtest(parsed), + "preopt" => preopt::subtest(parsed), "print-cfg" => print_cfg::subtest(parsed), "regalloc" => regalloc::subtest(parsed), "simple-gvn" => simple_gvn::subtest(parsed), diff --git a/cranelift/src/filetest/preopt.rs b/cranelift/src/filetest/preopt.rs new file mode 100644 index 0000000000..60d03f8207 --- /dev/null +++ b/cranelift/src/filetest/preopt.rs @@ -0,0 +1,50 @@ +//! Test command for testing the preopt pass. +//! +//! The resulting function is sent to `filecheck`. + +use cretonne::ir::Function; +use cretonne; +use cton_reader::TestCommand; +use filetest::subtest::{SubTest, Context, Result, run_filecheck}; +use std::borrow::Cow; +use std::fmt::Write; +use utils::pretty_error; + +struct TestPreopt; + +pub fn subtest(parsed: &TestCommand) -> Result> { + assert_eq!(parsed.command, "preopt"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestPreopt)) + } +} + +impl SubTest for TestPreopt { + fn name(&self) -> Cow { + Cow::from("preopt") + } + + fn is_mutating(&self) -> bool { + true + } + + fn run(&self, func: Cow, context: &Context) -> Result<()> { + // Create a compilation context, and drop in the function. + let mut comp_ctx = cretonne::Context::new(); + comp_ctx.func = func.into_owned(); + let isa = context.isa.expect("preopt needs an ISA"); + + comp_ctx.flowgraph(); + comp_ctx.preopt(isa).map_err(|e| { + pretty_error(&comp_ctx.func, context.isa, Into::into(e)) + })?; + + let mut text = String::new(); + write!(&mut text, "{}", &comp_ctx.func).map_err( + |e| e.to_string(), + )?; + run_filecheck(&text, context) + } +} diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 7200e45e7d..7db8bed202 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -833,6 +833,26 @@ imul = Instruction( """, ins=(x, y), outs=a) +umulhi = Instruction( + 'umulhi', r""" + Unsigned integer multiplication, producing the high half of a + double-length result. + + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(x, y), outs=a) + +smulhi = Instruction( + 'smulhi', """ + Signed integer multiplication, producing the high half of a + double-length result. + + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(x, y), outs=a) + udiv = Instruction( 'udiv', r""" Unsigned integer division: :math:`a := \lfloor {x \over y} \rfloor`. diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 1f47a5da06..f9b77a04bd 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -120,6 +120,9 @@ enc_i32_i64(base.imul, r.rrx, 0x0f, 0xaf) enc_i32_i64(x86.sdivmodx, r.div, 0xf7, rrr=7) enc_i32_i64(x86.udivmodx, r.div, 0xf7, rrr=6) +enc_i32_i64(x86.smulx, r.mulx, 0xf7, rrr=5) +enc_i32_i64(x86.umulx, r.mulx, 0xf7, rrr=4) + enc_i32_i64(base.copy, r.umr, 0x89) enc_both(base.copy.b1, r.umr, 0x89) enc_i32_i64(base.regmove, r.rmov, 0x89) diff --git a/lib/cretonne/meta/isa/intel/instructions.py b/lib/cretonne/meta/isa/intel/instructions.py index 277cf62b4a..e5265e2d15 100644 --- a/lib/cretonne/meta/isa/intel/instructions.py +++ b/lib/cretonne/meta/isa/intel/instructions.py @@ -47,6 +47,28 @@ sdivmodx = Instruction( """, ins=(nlo, nhi, d), outs=(q, r), can_trap=True) +argL = Operand('argL', iWord) +argR = Operand('argR', iWord) +resLo = Operand('resLo', iWord) +resHi = Operand('resHi', iWord) + +umulx = Instruction( + 'x86_umulx', r""" + Unsigned integer multiplication, producing a double-length result. + + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(argL, argR), outs=(resLo, resHi)) + +smulx = Instruction( + 'x86_smulx', r""" + Signed integer multiplication, producing a double-length result. + + Polymorphic over all scalar integer types, but does not support vector + types. + """, + ins=(argL, argR), outs=(resLo, resHi)) Float = TypeVar( 'Float', 'A scalar or vector floating point number', diff --git a/lib/cretonne/meta/isa/intel/legalize.py b/lib/cretonne/meta/isa/intel/legalize.py index 32f0a98153..5806bb9284 100644 --- a/lib/cretonne/meta/isa/intel/legalize.py +++ b/lib/cretonne/meta/isa/intel/legalize.py @@ -37,6 +37,23 @@ intel_expand.custom_legalize(insts.srem, 'expand_sdivrem') intel_expand.custom_legalize(insts.udiv, 'expand_udivrem') intel_expand.custom_legalize(insts.urem, 'expand_udivrem') +# +# Double length (widening) multiplication +# +resLo = Var('resLo') +resHi = Var('resHi') +intel_expand.legalize( + resHi << insts.umulhi(x, y), + Rtl( + (resLo, resHi) << x86.umulx(x, y) + )) + +intel_expand.legalize( + resHi << insts.smulhi(x, y), + Rtl( + (resLo, resHi) << x86.smulx(x, y) + )) + # Floating point condition codes. # # The 8 condition codes in `supported_floatccs` are directly supported by a diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 133837fc78..9d03d02053 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -453,6 +453,15 @@ div = TailRecipe( modrm_r_bits(in_reg2, bits, sink); ''') +# XX /n for {s,u}mulx: inputs in %rax, r. Outputs in %rdx(hi):%rax(lo) +mulx = TailRecipe( + 'mulx', Binary, size=1, + ins=(GPR.rax, GPR), outs=(GPR.rax, GPR.rdx), + emit=''' + PUT_OP(bits, rex1(in_reg1), sink); + modrm_r_bits(in_reg1, bits, sink); + ''') + # XX /n ib with 8-bit immediate sign-extended. rib = TailRecipe( 'rib', BinaryImm, size=2, ins=GPR, outs=0, diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 8aff5aa706..78b5d5bebf 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -23,6 +23,7 @@ use unreachable_code::eliminate_unreachable_code; use verifier; use simple_gvn::do_simple_gvn; use licm::do_licm; +use preopt::do_preopt; use timing; /// Persistent data structures and compilation pipeline. @@ -87,6 +88,7 @@ impl Context { self.verify_if(isa)?; self.compute_cfg(); + self.preopt(isa)?; self.legalize(isa)?; /* TODO: Enable additional optimization passes. if isa.flags().opt_level() == OptLevel::Best { @@ -131,6 +133,13 @@ impl Context { } } + /// Perform pre-legalization rewrites on the function. + pub fn preopt(&mut self, isa: &TargetIsa) -> CtonResult { + do_preopt(&mut self.func); + self.verify_if(isa)?; + Ok(()) + } + /// Run the legalizer for `isa` on the function. pub fn legalize(&mut self, isa: &TargetIsa) -> CtonResult { // Legalization invalidates the domtree and loop_analysis by mutating the CFG. diff --git a/lib/cretonne/src/divconst_magic_numbers.rs b/lib/cretonne/src/divconst_magic_numbers.rs new file mode 100644 index 0000000000..64416e48fe --- /dev/null +++ b/lib/cretonne/src/divconst_magic_numbers.rs @@ -0,0 +1,542 @@ +//! Compute "magic numbers" for division-by-constants transformations. + +#![allow(non_snake_case)] + +//---------------------------------------------------------------------- +// +// Math helpers for division by (non-power-of-2) constants. This is based +// on the presentation in "Hacker's Delight" by Henry Warren, 2003. There +// are four cases: {unsigned, signed} x {32 bit, 64 bit}. The word size +// makes little difference, but the signed-vs-unsigned aspect has a large +// effect. Therefore everything is presented in the order U32 U64 S32 S64 +// so as to emphasise the similarity of the U32 and U64 cases and the S32 +// and S64 cases. + +// Structures to hold the "magic numbers" computed. + +#[derive(PartialEq, Debug)] +pub struct MU32 { + pub mulBy: u32, + pub doAdd: bool, + pub shiftBy: i32, +} + +#[derive(PartialEq, Debug)] +pub struct MU64 { + pub mulBy: u64, + pub doAdd: bool, + pub shiftBy: i32, +} + +#[derive(PartialEq, Debug)] +pub struct MS32 { + pub mulBy: i32, + pub shiftBy: i32, +} + +#[derive(PartialEq, Debug)] +pub struct MS64 { + pub mulBy: i64, + pub shiftBy: i32, +} + +// The actual "magic number" generators follow. + +pub fn magicU32(d: u32) -> MU32 { + assert_ne!(d, 0); + assert_ne!(d, 1); // d==1 generates out of range shifts. + + let mut do_add: bool = false; + let mut p: i32 = 31; + let nc: u32 = 0xFFFFFFFFu32 - u32::wrapping_neg(d) % d; + let mut q1: u32 = 0x80000000u32 / nc; + let mut r1: u32 = 0x80000000u32 - q1 * nc; + let mut q2: u32 = 0x7FFFFFFFu32 / d; + let mut r2: u32 = 0x7FFFFFFFu32 - q2 * d; + loop { + p = p + 1; + if r1 >= nc - r1 { + q1 = u32::wrapping_add(u32::wrapping_mul(2, q1), 1); + r1 = u32::wrapping_sub(u32::wrapping_mul(2, r1), nc); + } else { + q1 = 2 * q1; + r1 = 2 * r1; + } + if r2 + 1 >= d - r2 { + if q2 >= 0x7FFFFFFFu32 { + do_add = true; + } + q2 = 2 * q2 + 1; + r2 = u32::wrapping_sub(u32::wrapping_add(u32::wrapping_mul(2, r2), 1), d); + } else { + if q2 >= 0x80000000u32 { + do_add = true; + } + q2 = u32::wrapping_mul(2, q2); + r2 = 2 * r2 + 1; + } + let delta: u32 = d - 1 - r2; + if !(p < 64 && (q1 < delta || (q1 == delta && r1 == 0))) { + break; + } + } + + MU32 { + mulBy: q2 + 1, + doAdd: do_add, + shiftBy: p - 32, + } +} + +pub fn magicU64(d: u64) -> MU64 { + assert_ne!(d, 0); + assert_ne!(d, 1); // d==1 generates out of range shifts. + + let mut do_add: bool = false; + let mut p: i32 = 63; + let nc: u64 = 0xFFFFFFFFFFFFFFFFu64 - u64::wrapping_neg(d) % d; + let mut q1: u64 = 0x8000000000000000u64 / nc; + let mut r1: u64 = 0x8000000000000000u64 - q1 * nc; + let mut q2: u64 = 0x7FFFFFFFFFFFFFFFu64 / d; + let mut r2: u64 = 0x7FFFFFFFFFFFFFFFu64 - q2 * d; + loop { + p = p + 1; + if r1 >= nc - r1 { + q1 = u64::wrapping_add(u64::wrapping_mul(2, q1), 1); + r1 = u64::wrapping_sub(u64::wrapping_mul(2, r1), nc); + } else { + q1 = 2 * q1; + r1 = 2 * r1; + } + if r2 + 1 >= d - r2 { + if q2 >= 0x7FFFFFFFFFFFFFFFu64 { + do_add = true; + } + q2 = 2 * q2 + 1; + r2 = u64::wrapping_sub(u64::wrapping_add(u64::wrapping_mul(2, r2), 1), d); + } else { + if q2 >= 0x8000000000000000u64 { + do_add = true; + } + q2 = u64::wrapping_mul(2, q2); + r2 = 2 * r2 + 1; + } + let delta: u64 = d - 1 - r2; + if !(p < 128 && (q1 < delta || (q1 == delta && r1 == 0))) { + break; + } + } + + MU64 { + mulBy: q2 + 1, + doAdd: do_add, + shiftBy: p - 64, + } +} + +pub fn magicS32(d: i32) -> MS32 { + assert_ne!(d, -1); + assert_ne!(d, 0); + assert_ne!(d, 1); + let two31: u32 = 0x80000000u32; + let mut p: i32 = 31; + let ad: u32 = i32::wrapping_abs(d) as u32; + let t: u32 = two31 + ((d as u32) >> 31); + let anc: u32 = u32::wrapping_sub(t - 1, t % ad); + let mut q1: u32 = two31 / anc; + let mut r1: u32 = two31 - q1 * anc; + let mut q2: u32 = two31 / ad; + let mut r2: u32 = two31 - q2 * ad; + loop { + p = p + 1; + q1 = 2 * q1; + r1 = 2 * r1; + if r1 >= anc { + q1 = q1 + 1; + r1 = r1 - anc; + } + q2 = 2 * q2; + r2 = 2 * r2; + if r2 >= ad { + q2 = q2 + 1; + r2 = r2 - ad; + } + let delta: u32 = ad - r2; + if !(q1 < delta || (q1 == delta && r1 == 0)) { + break; + } + } + + MS32 { + mulBy: (if d < 0 { + u32::wrapping_neg(q2 + 1) + } else { + q2 + 1 + }) as i32, + shiftBy: p - 32, + } +} + +pub fn magicS64(d: i64) -> MS64 { + assert_ne!(d, -1); + assert_ne!(d, 0); + assert_ne!(d, 1); + let two63: u64 = 0x8000000000000000u64; + let mut p: i32 = 63; + let ad: u64 = i64::wrapping_abs(d) as u64; + let t: u64 = two63 + ((d as u64) >> 63); + let anc: u64 = u64::wrapping_sub(t - 1, t % ad); + let mut q1: u64 = two63 / anc; + let mut r1: u64 = two63 - q1 * anc; + let mut q2: u64 = two63 / ad; + let mut r2: u64 = two63 - q2 * ad; + loop { + p = p + 1; + q1 = 2 * q1; + r1 = 2 * r1; + if r1 >= anc { + q1 = q1 + 1; + r1 = r1 - anc; + } + q2 = 2 * q2; + r2 = 2 * r2; + if r2 >= ad { + q2 = q2 + 1; + r2 = r2 - ad; + } + let delta: u64 = ad - r2; + if !(q1 < delta || (q1 == delta && r1 == 0)) { + break; + } + } + + MS64 { + mulBy: (if d < 0 { + u64::wrapping_neg(q2 + 1) + } else { + q2 + 1 + }) as i64, + shiftBy: p - 64, + } +} + +#[cfg(test)] +mod tests { + use super::{magicU32, magicU64, magicS32, magicS64}; + use super::{MU32, MU64, MS32, MS64}; + + fn mkMU32(mulBy: u32, doAdd: bool, shiftBy: i32) -> MU32 { + MU32 { + mulBy, + doAdd, + shiftBy, + } + } + + fn mkMU64(mulBy: u64, doAdd: bool, shiftBy: i32) -> MU64 { + MU64 { + mulBy, + doAdd, + shiftBy, + } + } + + fn mkMS32(mulBy: i32, shiftBy: i32) -> MS32 { + MS32 { mulBy, shiftBy } + } + + fn mkMS64(mulBy: i64, shiftBy: i32) -> MS64 { + MS64 { mulBy, shiftBy } + } + + #[test] + fn test_magicU32() { + assert_eq!(magicU32(2u32), mkMU32(0x80000000u32, false, 0)); + assert_eq!(magicU32(3u32), mkMU32(0xaaaaaaabu32, false, 1)); + assert_eq!(magicU32(4u32), mkMU32(0x40000000u32, false, 0)); + assert_eq!(magicU32(5u32), mkMU32(0xcccccccdu32, false, 2)); + assert_eq!(magicU32(6u32), mkMU32(0xaaaaaaabu32, false, 2)); + assert_eq!(magicU32(7u32), mkMU32(0x24924925u32, true, 3)); + assert_eq!(magicU32(9u32), mkMU32(0x38e38e39u32, false, 1)); + assert_eq!(magicU32(10u32), mkMU32(0xcccccccdu32, false, 3)); + assert_eq!(magicU32(11u32), mkMU32(0xba2e8ba3u32, false, 3)); + assert_eq!(magicU32(12u32), mkMU32(0xaaaaaaabu32, false, 3)); + assert_eq!(magicU32(25u32), mkMU32(0x51eb851fu32, false, 3)); + assert_eq!(magicU32(125u32), mkMU32(0x10624dd3u32, false, 3)); + assert_eq!(magicU32(625u32), mkMU32(0xd1b71759u32, false, 9)); + assert_eq!(magicU32(1337u32), mkMU32(0x88233b2bu32, true, 11)); + assert_eq!(magicU32(65535u32), mkMU32(0x80008001u32, false, 15)); + assert_eq!(magicU32(65536u32), mkMU32(0x00010000u32, false, 0)); + assert_eq!(magicU32(65537u32), mkMU32(0xffff0001u32, false, 16)); + assert_eq!(magicU32(31415927u32), mkMU32(0x445b4553u32, false, 23)); + assert_eq!(magicU32(0xdeadbeefu32), mkMU32(0x93275ab3u32, false, 31)); + assert_eq!(magicU32(0xfffffffdu32), mkMU32(0x40000001u32, false, 30)); + assert_eq!(magicU32(0xfffffffeu32), mkMU32(0x00000003u32, true, 32)); + assert_eq!(magicU32(0xffffffffu32), mkMU32(0x80000001u32, false, 31)); + } + #[test] + fn test_magicU64() { + assert_eq!(magicU64(2u64), mkMU64(0x8000000000000000u64, false, 0)); + assert_eq!(magicU64(3u64), mkMU64(0xaaaaaaaaaaaaaaabu64, false, 1)); + assert_eq!(magicU64(4u64), mkMU64(0x4000000000000000u64, false, 0)); + assert_eq!(magicU64(5u64), mkMU64(0xcccccccccccccccdu64, false, 2)); + assert_eq!(magicU64(6u64), mkMU64(0xaaaaaaaaaaaaaaabu64, false, 2)); + assert_eq!(magicU64(7u64), mkMU64(0x2492492492492493u64, true, 3)); + assert_eq!(magicU64(9u64), mkMU64(0xe38e38e38e38e38fu64, false, 3)); + assert_eq!(magicU64(10u64), mkMU64(0xcccccccccccccccdu64, false, 3)); + assert_eq!(magicU64(11u64), mkMU64(0x2e8ba2e8ba2e8ba3u64, false, 1)); + assert_eq!(magicU64(12u64), mkMU64(0xaaaaaaaaaaaaaaabu64, false, 3)); + assert_eq!(magicU64(25u64), mkMU64(0x47ae147ae147ae15u64, true, 5)); + assert_eq!(magicU64(125u64), mkMU64(0x0624dd2f1a9fbe77u64, true, 7)); + assert_eq!(magicU64(625u64), mkMU64(0x346dc5d63886594bu64, false, 7)); + assert_eq!(magicU64(1337u64), mkMU64(0xc4119d952866a139u64, false, 10)); + assert_eq!( + magicU64(31415927u64), + mkMU64(0x116d154b9c3d2f85u64, true, 25) + ); + assert_eq!( + magicU64(0x00000000deadbeefu64), + mkMU64(0x93275ab2dfc9094bu64, false, 31) + ); + assert_eq!( + magicU64(0x00000000fffffffdu64), + mkMU64(0x8000000180000005u64, false, 31) + ); + assert_eq!( + magicU64(0x00000000fffffffeu64), + mkMU64(0x0000000200000005u64, true, 32) + ); + assert_eq!( + magicU64(0x00000000ffffffffu64), + mkMU64(0x8000000080000001u64, false, 31) + ); + assert_eq!( + magicU64(0x0000000100000000u64), + mkMU64(0x0000000100000000u64, false, 0) + ); + assert_eq!( + magicU64(0x0000000100000001u64), + mkMU64(0xffffffff00000001u64, false, 32) + ); + assert_eq!( + magicU64(0x0ddc0ffeebadf00du64), + mkMU64(0x2788e9d394b77da1u64, true, 60) + ); + assert_eq!( + magicU64(0xfffffffffffffffdu64), + mkMU64(0x4000000000000001u64, false, 62) + ); + assert_eq!( + magicU64(0xfffffffffffffffeu64), + mkMU64(0x0000000000000003u64, true, 64) + ); + assert_eq!( + magicU64(0xffffffffffffffffu64), + mkMU64(0x8000000000000001u64, false, 63) + ); + } + #[test] + fn test_magicS32() { + assert_eq!(magicS32(-0x80000000i32), mkMS32(0x7fffffffu32 as i32, 30)); + assert_eq!(magicS32(-0x7FFFFFFFi32), mkMS32(0xbfffffffu32 as i32, 29)); + assert_eq!(magicS32(-0x7FFFFFFEi32), mkMS32(0x7ffffffdu32 as i32, 30)); + assert_eq!(magicS32(-31415927i32), mkMS32(0xbba4baadu32 as i32, 23)); + assert_eq!(magicS32(-1337i32), mkMS32(0x9df73135u32 as i32, 9)); + assert_eq!(magicS32(-256i32), mkMS32(0x7fffffffu32 as i32, 7)); + assert_eq!(magicS32(-5i32), mkMS32(0x99999999u32 as i32, 1)); + assert_eq!(magicS32(-3i32), mkMS32(0x55555555u32 as i32, 1)); + assert_eq!(magicS32(-2i32), mkMS32(0x7fffffffu32 as i32, 0)); + assert_eq!(magicS32(2i32), mkMS32(0x80000001u32 as i32, 0)); + assert_eq!(magicS32(3i32), mkMS32(0x55555556u32 as i32, 0)); + assert_eq!(magicS32(4i32), mkMS32(0x80000001u32 as i32, 1)); + assert_eq!(magicS32(5i32), mkMS32(0x66666667u32 as i32, 1)); + assert_eq!(magicS32(6i32), mkMS32(0x2aaaaaabu32 as i32, 0)); + assert_eq!(magicS32(7i32), mkMS32(0x92492493u32 as i32, 2)); + assert_eq!(magicS32(9i32), mkMS32(0x38e38e39u32 as i32, 1)); + assert_eq!(magicS32(10i32), mkMS32(0x66666667u32 as i32, 2)); + assert_eq!(magicS32(11i32), mkMS32(0x2e8ba2e9u32 as i32, 1)); + assert_eq!(magicS32(12i32), mkMS32(0x2aaaaaabu32 as i32, 1)); + assert_eq!(magicS32(25i32), mkMS32(0x51eb851fu32 as i32, 3)); + assert_eq!(magicS32(125i32), mkMS32(0x10624dd3u32 as i32, 3)); + assert_eq!(magicS32(625i32), mkMS32(0x68db8badu32 as i32, 8)); + assert_eq!(magicS32(1337i32), mkMS32(0x6208cecbu32 as i32, 9)); + assert_eq!(magicS32(31415927i32), mkMS32(0x445b4553u32 as i32, 23)); + assert_eq!(magicS32(0x7ffffffei32), mkMS32(0x80000003u32 as i32, 30)); + assert_eq!(magicS32(0x7fffffffi32), mkMS32(0x40000001u32 as i32, 29)); + } + #[test] + fn test_magicS64() { + assert_eq!( + magicS64(-0x8000000000000000i64), + mkMS64(0x7fffffffffffffffu64 as i64, 62) + ); + assert_eq!( + magicS64(-0x7FFFFFFFFFFFFFFFi64), + mkMS64(0xbfffffffffffffffu64 as i64, 61) + ); + assert_eq!( + magicS64(-0x7FFFFFFFFFFFFFFEi64), + mkMS64(0x7ffffffffffffffdu64 as i64, 62) + ); + assert_eq!( + magicS64(-0x0ddC0ffeeBadF00di64), + mkMS64(0x6c3b8b1635a4412fu64 as i64, 59) + ); + assert_eq!( + magicS64(-0x100000001i64), + mkMS64(0x800000007fffffffu64 as i64, 31) + ); + assert_eq!( + magicS64(-0x100000000i64), + mkMS64(0x7fffffffffffffffu64 as i64, 31) + ); + assert_eq!( + magicS64(-0xFFFFFFFFi64), + mkMS64(0x7fffffff7fffffffu64 as i64, 31) + ); + assert_eq!( + magicS64(-0xFFFFFFFEi64), + mkMS64(0x7ffffffefffffffdu64 as i64, 31) + ); + assert_eq!( + magicS64(-0xFFFFFFFDi64), + mkMS64(0x7ffffffe7ffffffbu64 as i64, 31) + ); + assert_eq!( + magicS64(-0xDeadBeefi64), + mkMS64(0x6cd8a54d2036f6b5u64 as i64, 31) + ); + assert_eq!( + magicS64(-31415927i64), + mkMS64(0x7749755a31e1683du64 as i64, 24) + ); + assert_eq!(magicS64(-1337i64), mkMS64(0x9df731356bccaf63u64 as i64, 9)); + assert_eq!(magicS64(-256i64), mkMS64(0x7fffffffffffffffu64 as i64, 7)); + assert_eq!(magicS64(-5i64), mkMS64(0x9999999999999999u64 as i64, 1)); + assert_eq!(magicS64(-3i64), mkMS64(0x5555555555555555u64 as i64, 1)); + assert_eq!(magicS64(-2i64), mkMS64(0x7fffffffffffffffu64 as i64, 0)); + assert_eq!(magicS64(2i64), mkMS64(0x8000000000000001u64 as i64, 0)); + assert_eq!(magicS64(3i64), mkMS64(0x5555555555555556u64 as i64, 0)); + assert_eq!(magicS64(4i64), mkMS64(0x8000000000000001u64 as i64, 1)); + assert_eq!(magicS64(5i64), mkMS64(0x6666666666666667u64 as i64, 1)); + assert_eq!(magicS64(6i64), mkMS64(0x2aaaaaaaaaaaaaabu64 as i64, 0)); + assert_eq!(magicS64(7i64), mkMS64(0x4924924924924925u64 as i64, 1)); + assert_eq!(magicS64(9i64), mkMS64(0x1c71c71c71c71c72u64 as i64, 0)); + assert_eq!(magicS64(10i64), mkMS64(0x6666666666666667u64 as i64, 2)); + assert_eq!(magicS64(11i64), mkMS64(0x2e8ba2e8ba2e8ba3u64 as i64, 1)); + assert_eq!(magicS64(12i64), mkMS64(0x2aaaaaaaaaaaaaabu64 as i64, 1)); + assert_eq!(magicS64(25i64), mkMS64(0xa3d70a3d70a3d70bu64 as i64, 4)); + assert_eq!(magicS64(125i64), mkMS64(0x20c49ba5e353f7cfu64 as i64, 4)); + assert_eq!(magicS64(625i64), mkMS64(0x346dc5d63886594bu64 as i64, 7)); + assert_eq!(magicS64(1337i64), mkMS64(0x6208ceca9433509du64 as i64, 9)); + assert_eq!( + magicS64(31415927i64), + mkMS64(0x88b68aa5ce1e97c3u64 as i64, 24) + ); + assert_eq!( + magicS64(0x00000000deadbeefi64), + mkMS64(0x93275ab2dfc9094bu64 as i64, 31) + ); + assert_eq!( + magicS64(0x00000000fffffffdi64), + mkMS64(0x8000000180000005u64 as i64, 31) + ); + assert_eq!( + magicS64(0x00000000fffffffei64), + mkMS64(0x8000000100000003u64 as i64, 31) + ); + assert_eq!( + magicS64(0x00000000ffffffffi64), + mkMS64(0x8000000080000001u64 as i64, 31) + ); + assert_eq!( + magicS64(0x0000000100000000i64), + mkMS64(0x8000000000000001u64 as i64, 31) + ); + assert_eq!( + magicS64(0x0000000100000001i64), + mkMS64(0x7fffffff80000001u64 as i64, 31) + ); + assert_eq!( + magicS64(0x0ddc0ffeebadf00di64), + mkMS64(0x93c474e9ca5bbed1u64 as i64, 59) + ); + assert_eq!( + magicS64(0x7ffffffffffffffdi64), + mkMS64(0x2000000000000001u64 as i64, 60) + ); + assert_eq!( + magicS64(0x7ffffffffffffffei64), + mkMS64(0x8000000000000003u64 as i64, 62) + ); + assert_eq!( + magicS64(0x7fffffffffffffffi64), + mkMS64(0x4000000000000001u64 as i64, 61) + ); + } + #[test] + fn test_magic_generators_dont_panic() { + // The point of this is to check that the magic number generators + // don't panic with integer wraparounds, especially at boundary + // cases for their arguments. The actual results are thrown away. + let mut total: u64 = 0; + println!("Testing UP magicU32"); + for x in 2..(200 * 1000u32) { + let m = magicU32(x); + total = total ^ (m.mulBy as u64); + total = total + (m.shiftBy as u64); + total = total - (if m.doAdd { 123 } else { 456 }); + } + println!("Testing DOWN magicU32"); + for x in 0..(200 * 1000u32) { + let m = magicU32(0xFFFF_FFFFu32 - x); + total = total ^ (m.mulBy as u64); + total = total + (m.shiftBy as u64); + total = total - (if m.doAdd { 123 } else { 456 }); + } + + println!("Testing UP magicU64"); + for x in 2..(200 * 1000u64) { + let m = magicU64(x); + total = total ^ (m.mulBy as u64); + total = total + (m.shiftBy as u64); + total = total - (if m.doAdd { 123 } else { 456 }); + } + println!("Testing DOWN magicU64"); + for x in 0..(200 * 1000u64) { + let m = magicU64(0xFFFF_FFFF_FFFF_FFFFu64 - x); + total = total ^ (m.mulBy as u64); + total = total + (m.shiftBy as u64); + total = total - (if m.doAdd { 123 } else { 456 }); + } + + println!("Testing UP magicS32"); + for x in 0..(200 * 1000i32) { + let m = magicS32(-0x8000_0000i32 + x); + total = total ^ (m.mulBy as u64); + total = total + (m.shiftBy as u64); + } + println!("Testing DOWN magicS32"); + for x in 0..(200 * 1000i32) { + let m = magicS32(0x7FFF_FFFFi32 - x); + total = total ^ (m.mulBy as u64); + total = total + (m.shiftBy as u64); + } + + println!("Testing UP magicS64"); + for x in 0..(200 * 1000i64) { + let m = magicS64(-0x8000_0000_0000_0000i64 + x); + total = total ^ (m.mulBy as u64); + total = total + (m.shiftBy as u64); + } + println!("Testing DOWN magicS64"); + for x in 0..(200 * 1000i64) { + let m = magicS64(0x7FFF_FFFF_FFFF_FFFFi64 - x); + total = total ^ (m.mulBy as u64); + total = total + (m.shiftBy as u64); + } + // Force `total` -- and hence, the entire computation -- to + // be used, so that rustc can't optimise it out. + assert_eq!(total, 7547519887532559585u64); + } +} diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 1a602cf8a1..00c5ab99cb 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -1,5 +1,6 @@ //! Encoding tables for Intel ISAs. +use bitset::BitSet; use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::{self, InstBuilder}; diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index b4c386477c..f34bed638b 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -33,11 +33,13 @@ mod abi; mod bitset; mod constant_hash; mod context; +mod divconst_magic_numbers; mod iterators; mod legalizer; mod licm; mod partition_slice; mod predicates; +mod preopt; mod ref_slice; mod regalloc; mod scoped_hash_map; diff --git a/lib/cretonne/src/preopt.rs b/lib/cretonne/src/preopt.rs new file mode 100644 index 0000000000..0e4582d7a7 --- /dev/null +++ b/lib/cretonne/src/preopt.rs @@ -0,0 +1,521 @@ +//! A pre-legalization rewriting pass. + +#![allow(non_snake_case)] + +use cursor::{Cursor, FuncCursor}; +use ir::dfg::ValueDef; +use ir::{Function, InstructionData, Value, DataFlowGraph, InstBuilder, Type}; +use ir::Inst; +use ir::types::{I32, I64}; +use ir::instructions::Opcode; +use divconst_magic_numbers::{MU32, MU64, MS32, MS64}; +use divconst_magic_numbers::{magicU32, magicU64, magicS32, magicS64}; +use timing; + + +//---------------------------------------------------------------------- +// +// Pattern-match helpers and transformation for div and rem by constants. + +// Simple math helpers + +// if `x` is a power of two, or the negation thereof, return the power along +// with a boolean that indicates whether `x` is negative. Else return None. +#[inline] +fn isPowerOf2_S32(x: i32) -> Option<(bool, u32)> { + // We have to special-case this because abs(x) isn't representable. + if x == -0x8000_0000 { + return Some((true, 31)); + } + let abs_x = i32::wrapping_abs(x) as u32; + if abs_x.is_power_of_two() { + return Some((x < 0, abs_x.trailing_zeros())); + } + None +} + +// Same comments as for isPowerOf2_S64 apply. +#[inline] +fn isPowerOf2_S64(x: i64) -> Option<(bool, u32)> { + // We have to special-case this because abs(x) isn't representable. + if x == -0x8000_0000_0000_0000 { + return Some((true, 63)); + } + let abs_x = i64::wrapping_abs(x) as u64; + if abs_x.is_power_of_two() { + return Some((x < 0, abs_x.trailing_zeros())); + } + None +} + +#[derive(Debug)] +enum DivRemByConstInfo { + DivU32(Value, u32), // In all cases, the arguments are: + DivU64(Value, u64), // left operand, right operand + DivS32(Value, i32), + DivS64(Value, i64), + RemU32(Value, u32), + RemU64(Value, u64), + RemS32(Value, i32), + RemS64(Value, i64), +} + +// Possibly create a DivRemByConstInfo from the given components, by +// figuring out which, if any, of the 8 cases apply, and also taking care to +// sanity-check the immediate. +fn package_up_divrem_info( + argL: Value, + argL_ty: Type, + argRs: i64, + isSigned: bool, + isRem: bool, +) -> Option { + let argRu: u64 = argRs as u64; + if !isSigned && argL_ty == I32 && argRu < 0x1_0000_0000 { + let con = if isRem { + DivRemByConstInfo::RemU32 + } else { + DivRemByConstInfo::DivU32 + }; + return Some(con(argL, argRu as u32)); + } + if !isSigned && argL_ty == I64 { + // unsigned 64, no range constraint + let con = if isRem { + DivRemByConstInfo::RemU64 + } else { + DivRemByConstInfo::DivU64 + }; + return Some(con(argL, argRu)); + } + if isSigned && argL_ty == I32 && (argRu <= 0x7fff_ffff || argRu >= 0xffff_ffff_8000_0000) { + let con = if isRem { + DivRemByConstInfo::RemS32 + } else { + DivRemByConstInfo::DivS32 + }; + return Some(con(argL, argRu as i32)); + } + if isSigned && argL_ty == I64 { + // signed 64, no range constraint + let con = if isRem { + DivRemByConstInfo::RemS64 + } else { + DivRemByConstInfo::DivS64 + }; + return Some(con(argL, argRu as i64)); + } + None +} + +// Examine `idata` to see if it is a div or rem by a constant, and if so +// return the operands, signedness, operation size and div-vs-rem-ness in a +// handy bundle. +fn get_div_info(inst: Inst, dfg: &DataFlowGraph) -> Option { + let idata: &InstructionData = &dfg[inst]; + + if let &InstructionData::BinaryImm { opcode, arg, imm } = idata { + let (isSigned, isRem) = match opcode { + Opcode::UdivImm => (false, false), + Opcode::UremImm => (false, true), + Opcode::SdivImm => (true, false), + Opcode::SremImm => (true, true), + _other => return None, + }; + // Pull the operation size (type) from the left arg + let argL_ty = dfg.value_type(arg); + return package_up_divrem_info(arg, argL_ty, imm.into(), isSigned, isRem); + } + + // TODO: should we actually bother to do this (that is, manually match + // the case that the second argument is an iconst)? Or should we assume + // that some previous constant propagation pass has pushed all such + // immediates to their use points, creating BinaryImm instructions + // instead? For now we take the conservative approach. + if let &InstructionData::Binary { opcode, args } = idata { + let (isSigned, isRem) = match opcode { + Opcode::Udiv => (false, false), + Opcode::Urem => (false, true), + Opcode::Sdiv => (true, false), + Opcode::Srem => (true, true), + _other => return None, + }; + let argR: Value = args[1]; + if let Some(simm64) = get_const(argR, dfg) { + let argL: Value = args[0]; + // Pull the operation size (type) from the left arg + let argL_ty = dfg.value_type(argL); + return package_up_divrem_info(argL, argL_ty, simm64, isSigned, isRem); + } + } + + None +} + +// Actually do the transformation given a bundle containing the relevant +// information. `divrem_info` describes a div or rem by a constant, that +// `pos` currently points at, and `inst` is the associated instruction. +// `inst` is replaced by a sequence of other operations that calculate the +// same result. Note that there are various `divrem_info` cases where we +// cannot do any transformation, in which case `inst` is left unchanged. +fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCursor, inst: Inst) { + let isRem = match *divrem_info { + DivRemByConstInfo::DivU32(_, _) | + DivRemByConstInfo::DivU64(_, _) | + DivRemByConstInfo::DivS32(_, _) | + DivRemByConstInfo::DivS64(_, _) => false, + DivRemByConstInfo::RemU32(_, _) | + DivRemByConstInfo::RemU64(_, _) | + DivRemByConstInfo::RemS32(_, _) | + DivRemByConstInfo::RemS64(_, _) => true, + }; + + match divrem_info { + + // -------------------- U32 -------------------- + + // U32 div, rem by zero: ignore + &DivRemByConstInfo::DivU32(_n1, 0) | + &DivRemByConstInfo::RemU32(_n1, 0) => {} + + // U32 div by 1: identity + // U32 rem by 1: zero + &DivRemByConstInfo::DivU32(n1, 1) | + &DivRemByConstInfo::RemU32(n1, 1) => { + if isRem { + pos.func.dfg.replace(inst).iconst(I32, 0); + } else { + pos.func.dfg.replace(inst).copy(n1); + } + } + + // U32 div, rem by a power-of-2 + &DivRemByConstInfo::DivU32(n1, d) | + &DivRemByConstInfo::RemU32(n1, d) if d.is_power_of_two() => { + assert!(d >= 2); + // compute k where d == 2^k + let k = d.trailing_zeros(); + assert!(k >= 1 && k <= 31); + if isRem { + let mask = (1u64 << k) - 1; + pos.func.dfg.replace(inst).band_imm(n1, mask as i64); + } else { + pos.func.dfg.replace(inst).ushr_imm(n1, k as i64); + } + } + + // U32 div, rem by non-power-of-2 + &DivRemByConstInfo::DivU32(n1, d) | + &DivRemByConstInfo::RemU32(n1, d) => { + assert!(d >= 3); + let MU32 { + mulBy, + doAdd, + shiftBy, + } = magicU32(d); + let qf; // final quotient + let q0 = pos.ins().iconst(I32, mulBy as i64); + let q1 = pos.ins().umulhi(n1, q0); + if doAdd { + assert!(shiftBy >= 1 && shiftBy <= 32); + let t1 = pos.ins().isub(n1, q1); + let t2 = pos.ins().ushr_imm(t1, 1); + let t3 = pos.ins().iadd(t2, q1); + // I never found any case where shiftBy == 1 here. + // So there's no attempt to fold out a zero shift. + debug_assert!(shiftBy != 1); + qf = pos.ins().ushr_imm(t3, (shiftBy - 1) as i64); + } else { + assert!(shiftBy >= 0 && shiftBy <= 31); + // Whereas there are known cases here for shiftBy == 0. + if shiftBy > 0 { + qf = pos.ins().ushr_imm(q1, shiftBy as i64); + } else { + qf = q1; + } + } + // Now qf holds the final quotient. If necessary calculate the + // remainder instead. + if isRem { + let tt = pos.ins().imul_imm(qf, d as i64); + pos.func.dfg.replace(inst).isub(n1, tt); + } else { + pos.func.dfg.replace(inst).copy(qf); + } + } + + // -------------------- U64 -------------------- + + // U64 div, rem by zero: ignore + &DivRemByConstInfo::DivU64(_n1, 0) | + &DivRemByConstInfo::RemU64(_n1, 0) => {} + + // U64 div by 1: identity + // U64 rem by 1: zero + &DivRemByConstInfo::DivU64(n1, 1) | + &DivRemByConstInfo::RemU64(n1, 1) => { + if isRem { + pos.func.dfg.replace(inst).iconst(I64, 0); + } else { + pos.func.dfg.replace(inst).copy(n1); + } + } + + // U64 div, rem by a power-of-2 + &DivRemByConstInfo::DivU64(n1, d) | + &DivRemByConstInfo::RemU64(n1, d) if d.is_power_of_two() => { + assert!(d >= 2); + // compute k where d == 2^k + let k = d.trailing_zeros(); + assert!(k >= 1 && k <= 63); + if isRem { + let mask = (1u64 << k) - 1; + pos.func.dfg.replace(inst).band_imm(n1, mask as i64); + } else { + pos.func.dfg.replace(inst).ushr_imm(n1, k as i64); + } + } + + // U64 div, rem by non-power-of-2 + &DivRemByConstInfo::DivU64(n1, d) | + &DivRemByConstInfo::RemU64(n1, d) => { + assert!(d >= 3); + let MU64 { + mulBy, + doAdd, + shiftBy, + } = magicU64(d); + let qf; // final quotient + let q0 = pos.ins().iconst(I64, mulBy as i64); + let q1 = pos.ins().umulhi(n1, q0); + if doAdd { + assert!(shiftBy >= 1 && shiftBy <= 64); + let t1 = pos.ins().isub(n1, q1); + let t2 = pos.ins().ushr_imm(t1, 1); + let t3 = pos.ins().iadd(t2, q1); + // I never found any case where shiftBy == 1 here. + // So there's no attempt to fold out a zero shift. + debug_assert!(shiftBy != 1); + qf = pos.ins().ushr_imm(t3, (shiftBy - 1) as i64); + } else { + assert!(shiftBy >= 0 && shiftBy <= 63); + // Whereas there are known cases here for shiftBy == 0. + if shiftBy > 0 { + qf = pos.ins().ushr_imm(q1, shiftBy as i64); + } else { + qf = q1; + } + } + // Now qf holds the final quotient. If necessary calculate the + // remainder instead. + if isRem { + let tt = pos.ins().imul_imm(qf, d as i64); + pos.func.dfg.replace(inst).isub(n1, tt); + } else { + pos.func.dfg.replace(inst).copy(qf); + } + } + + // -------------------- S32 -------------------- + + // S32 div, rem by zero or -1: ignore + &DivRemByConstInfo::DivS32(_n1, -1) | + &DivRemByConstInfo::RemS32(_n1, -1) | + &DivRemByConstInfo::DivS32(_n1, 0) | + &DivRemByConstInfo::RemS32(_n1, 0) => {} + + // S32 div by 1: identity + // S32 rem by 1: zero + &DivRemByConstInfo::DivS32(n1, 1) | + &DivRemByConstInfo::RemS32(n1, 1) => { + if isRem { + pos.func.dfg.replace(inst).iconst(I32, 0); + } else { + pos.func.dfg.replace(inst).copy(n1); + } + } + + &DivRemByConstInfo::DivS32(n1, d) | + &DivRemByConstInfo::RemS32(n1, d) => { + if let Some((isNeg, k)) = isPowerOf2_S32(d) { + // k can be 31 only in the case that d is -2^31. + assert!(k >= 1 && k <= 31); + let t1 = if k - 1 == 0 { + n1 + } else { + pos.ins().sshr_imm(n1, (k - 1) as i64) + }; + let t2 = pos.ins().ushr_imm(t1, (32 - k) as i64); + let t3 = pos.ins().iadd(n1, t2); + if isRem { + // S32 rem by a power-of-2 + let t4 = pos.ins().band_imm(t3, i32::wrapping_neg(1 << k) as i64); + // Curiously, we don't care here what the sign of d is. + pos.func.dfg.replace(inst).isub(n1, t4); + } else { + // S32 div by a power-of-2 + let t4 = pos.ins().sshr_imm(t3, k as i64); + if isNeg { + pos.func.dfg.replace(inst).irsub_imm(t4, 0); + } else { + pos.func.dfg.replace(inst).copy(t4); + } + } + } else { + // S32 div, rem by a non-power-of-2 + assert!(d < -2 || d > 2); + let MS32 { mulBy, shiftBy } = magicS32(d); + let q0 = pos.ins().iconst(I32, mulBy as i64); + let q1 = pos.ins().smulhi(n1, q0); + let q2 = if d > 0 && mulBy < 0 { + pos.ins().iadd(q1, n1) + } else if d < 0 && mulBy > 0 { + pos.ins().isub(q1, n1) + } else { + q1 + }; + assert!(shiftBy >= 0 && shiftBy <= 31); + let q3 = if shiftBy == 0 { + q2 + } else { + pos.ins().sshr_imm(q2, shiftBy as i64) + }; + let t1 = pos.ins().ushr_imm(q3, 31); + let qf = pos.ins().iadd(q3, t1); + // Now qf holds the final quotient. If necessary calculate + // the remainder instead. + if isRem { + let tt = pos.ins().imul_imm(qf, d as i64); + pos.func.dfg.replace(inst).isub(n1, tt); + } else { + pos.func.dfg.replace(inst).copy(qf); + } + } + } + + // -------------------- S64 -------------------- + + // S64 div, rem by zero or -1: ignore + &DivRemByConstInfo::DivS64(_n1, -1) | + &DivRemByConstInfo::RemS64(_n1, -1) | + &DivRemByConstInfo::DivS64(_n1, 0) | + &DivRemByConstInfo::RemS64(_n1, 0) => {} + + // S64 div by 1: identity + // S64 rem by 1: zero + &DivRemByConstInfo::DivS64(n1, 1) | + &DivRemByConstInfo::RemS64(n1, 1) => { + if isRem { + pos.func.dfg.replace(inst).iconst(I64, 0); + } else { + pos.func.dfg.replace(inst).copy(n1); + } + } + + &DivRemByConstInfo::DivS64(n1, d) | + &DivRemByConstInfo::RemS64(n1, d) => { + if let Some((isNeg, k)) = isPowerOf2_S64(d) { + // k can be 63 only in the case that d is -2^63. + assert!(k >= 1 && k <= 63); + let t1 = if k - 1 == 0 { + n1 + } else { + pos.ins().sshr_imm(n1, (k - 1) as i64) + }; + let t2 = pos.ins().ushr_imm(t1, (64 - k) as i64); + let t3 = pos.ins().iadd(n1, t2); + if isRem { + // S64 rem by a power-of-2 + let t4 = pos.ins().band_imm(t3, i64::wrapping_neg(1 << k)); + // Curiously, we don't care here what the sign of d is. + pos.func.dfg.replace(inst).isub(n1, t4); + } else { + // S64 div by a power-of-2 + let t4 = pos.ins().sshr_imm(t3, k as i64); + if isNeg { + pos.func.dfg.replace(inst).irsub_imm(t4, 0); + } else { + pos.func.dfg.replace(inst).copy(t4); + } + } + } else { + // S64 div, rem by a non-power-of-2 + assert!(d < -2 || d > 2); + let MS64 { mulBy, shiftBy } = magicS64(d); + let q0 = pos.ins().iconst(I64, mulBy); + let q1 = pos.ins().smulhi(n1, q0); + let q2 = if d > 0 && mulBy < 0 { + pos.ins().iadd(q1, n1) + } else if d < 0 && mulBy > 0 { + pos.ins().isub(q1, n1) + } else { + q1 + }; + assert!(shiftBy >= 0 && shiftBy <= 63); + let q3 = if shiftBy == 0 { + q2 + } else { + pos.ins().sshr_imm(q2, shiftBy as i64) + }; + let t1 = pos.ins().ushr_imm(q3, 63); + let qf = pos.ins().iadd(q3, t1); + // Now qf holds the final quotient. If necessary calculate + // the remainder instead. + if isRem { + let tt = pos.ins().imul_imm(qf, d); + pos.func.dfg.replace(inst).isub(n1, tt); + } else { + pos.func.dfg.replace(inst).copy(qf); + } + } + } + + } +} + + +//---------------------------------------------------------------------- +// +// General pattern-match helpers. + +// Find out if `value` actually resolves to a constant, and if so what its +// value is. +fn get_const(value: Value, dfg: &DataFlowGraph) -> Option { + match dfg.value_def(value) { + ValueDef::Result(definingInst, resultNo) => { + let definingIData: &InstructionData = &dfg[definingInst]; + if let &InstructionData::UnaryImm { opcode, imm } = definingIData { + if opcode == Opcode::Iconst && resultNo == 0 { + return Some(imm.into()); + } + } + None + } + ValueDef::Param(_definingEbb, _paramNo) => None, + } +} + + +//---------------------------------------------------------------------- +// +// The main pre-opt pass. + +pub fn do_preopt(func: &mut Function) { + let _tt = timing::preopt(); + let mut pos = FuncCursor::new(func); + while let Some(_ebb) = pos.next_ebb() { + + while let Some(inst) = pos.next_inst() { + + //-- BEGIN -- division by constants ---------------- + + let mb_dri = get_div_info(inst, &pos.func.dfg); + if let Some(divrem_info) = mb_dri { + do_divrem_transformation(&divrem_info, &mut pos, inst); + continue; + } + + //-- END -- division by constants ------------------ + } + } +} diff --git a/lib/cretonne/src/timing.rs b/lib/cretonne/src/timing.rs index d120d43c32..5419895fdc 100644 --- a/lib/cretonne/src/timing.rs +++ b/lib/cretonne/src/timing.rs @@ -55,6 +55,7 @@ define_passes!{ flowgraph: "Control flow graph", domtree: "Dominator tree", loop_analysis: "Loop analysis", + preopt: "Pre-legalization rewriting", legalize: "Legalization", gvn: "Global value numbering", licm: "Loop invariant code motion", From 227baaadb8652eda9fa09fbfa73ee7d3d7ec32bd Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Feb 2018 11:42:43 -0800 Subject: [PATCH 1561/3084] Enable the simple_gvn and licm passes at OptLevel::Best. --- lib/cretonne/src/context.rs | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 78b5d5bebf..5b8db2414c 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -18,7 +18,7 @@ use isa::TargetIsa; use legalize_function; use regalloc; use result::{CtonError, CtonResult}; -use settings::FlagsOrIsa; +use settings::{FlagsOrIsa, OptLevel}; use unreachable_code::eliminate_unreachable_code; use verifier; use simple_gvn::do_simple_gvn; @@ -90,14 +90,12 @@ impl Context { self.compute_cfg(); self.preopt(isa)?; self.legalize(isa)?; - /* TODO: Enable additional optimization passes. if isa.flags().opt_level() == OptLevel::Best { self.compute_domtree(); self.compute_loop_analysis(); self.licm(isa)?; self.simple_gvn(isa)?; } - */ self.compute_domtree(); self.eliminate_unreachable_code(isa)?; self.regalloc(isa)?; From 1cf9a8d669d453fd9e505db49c1cf7085b0d8f0a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Feb 2018 12:02:14 -0800 Subject: [PATCH 1562/3084] Implement the wasm sign-extension-ops proposal. https://github.com/WebAssembly/sign-extension-ops/ --- lib/wasm/src/code_translator.rs | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index abaaaef3bf..21df608f74 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -634,12 +634,35 @@ pub fn translate_operator( let val = state.pop1(); state.push1(builder.ins().bitcast(I64, val)); } - Operator::I32Extend8S | - Operator::I32Extend16S | - Operator::I64Extend8S | - Operator::I64Extend16S | + Operator::I32Extend8S => { + let val = state.pop1(); + state.push1(builder.ins().ireduce(I8, val)); + let val = state.pop1(); + state.push1(builder.ins().sextend(I32, val)); + } + Operator::I32Extend16S => { + let val = state.pop1(); + state.push1(builder.ins().ireduce(I16, val)); + let val = state.pop1(); + state.push1(builder.ins().sextend(I32, val)); + } + Operator::I64Extend8S => { + let val = state.pop1(); + state.push1(builder.ins().ireduce(I8, val)); + let val = state.pop1(); + state.push1(builder.ins().sextend(I64, val)); + } + Operator::I64Extend16S => { + let val = state.pop1(); + state.push1(builder.ins().ireduce(I8, val)); + let val = state.pop1(); + state.push1(builder.ins().sextend(I64, val)); + } Operator::I64Extend32S => { - panic!("proposed sign-extend operators not yet supported"); + let val = state.pop1(); + state.push1(builder.ins().ireduce(I32, val)); + let val = state.pop1(); + state.push1(builder.ins().sextend(I64, val)); } /****************************** Binary Operators ************************************/ Operator::I32Add | Operator::I64Add => { From b9f51d7850e969d714936bc0d4a12c15e29e83ae Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 23 Feb 2018 22:27:52 -0800 Subject: [PATCH 1563/3084] Enable more compliler lints. --- lib/cretonne/src/lib.rs | 4 +++- lib/filecheck/src/lib.rs | 4 +++- lib/frontend/src/lib.rs | 4 +++- lib/native/src/lib.rs | 4 +++- lib/reader/src/lib.rs | 4 +++- lib/wasm/src/lib.rs | 4 +++- 6 files changed, 18 insertions(+), 6 deletions(-) diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index f34bed638b..ab7bb51f03 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -1,6 +1,8 @@ //! Cretonne code generation library. -#![deny(missing_docs)] +#![deny(missing_docs, + trivial_numeric_casts, + unused_extern_crates)] pub use context::Context; pub use legalizer::legalize_function; diff --git a/lib/filecheck/src/lib.rs b/lib/filecheck/src/lib.rs index a189acebdb..4c8a242897 100644 --- a/lib/filecheck/src/lib.rs +++ b/lib/filecheck/src/lib.rs @@ -236,7 +236,9 @@ //! This will match `"one, two"` , but not `"one,two"`. Without the `$()`, trailing whitespace //! would be trimmed from the pattern. -#![deny(missing_docs)] +#![deny(missing_docs, + trivial_numeric_casts, + unused_extern_crates)] pub use error::{Error, Result}; pub use variable::{VariableMap, Value, NO_VARIABLES}; diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 9db5a4a34b..8d2de2237b 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -142,7 +142,9 @@ //! } //! ``` -#![deny(missing_docs)] +#![deny(missing_docs, + trivial_numeric_casts, + unused_extern_crates)] extern crate cretonne; diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index 6cdb39a15e..d7c6a8d0ab 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -1,7 +1,9 @@ //! Performs autodetection of the host for the purposes of running //! Cretonne to generate code to run on the same machine. -#![deny(missing_docs)] +#![deny(missing_docs, + trivial_numeric_casts, + unused_extern_crates)] extern crate cretonne; diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index 06a03dd6fe..85ea048950 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -3,7 +3,9 @@ //! The cton_reader library supports reading .cton files. This functionality is needed for testing //! Cretonne, but is not essential for a JIT compiler. -#![deny(missing_docs)] +#![deny(missing_docs, + trivial_numeric_casts, + unused_extern_crates)] extern crate cretonne; diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index b5b79049fe..6e9a6a365a 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -9,7 +9,9 @@ //! //! The main function of this module is [`translate_module`](fn.translate_module.html). -#![deny(missing_docs)] +#![deny(missing_docs, + trivial_numeric_casts, + unused_extern_crates)] extern crate wasmparser; extern crate cton_frontend; From c93f29ad1e2d4b2228d788506f6d1d00d7c0b322 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Feb 2018 13:30:37 -0800 Subject: [PATCH 1564/3084] Remove more obsolete comments about entity number remapping. --- lib/reader/src/parser.rs | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index b4252d4cb9..47f14129a9 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -82,9 +82,6 @@ pub struct Parser<'a> { } // Context for resolving references when parsing a single function. -// -// Many entities like values, stack slots, and function signatures are referenced in the `.cton` -// file by number. We need to map these numbers to real references. struct Context<'a> { function: Function, map: SourceMap, @@ -119,7 +116,7 @@ impl<'a> Context<'a> { } } - // Allocate a new stack slot and add a mapping number -> StackSlot. + // Allocate a new stack slot. fn add_ss(&mut self, ss: StackSlot, data: StackSlotData, loc: &Location) -> Result<()> { while self.function.stack_slots.next_key().index() <= ss.index() { self.function.create_stack_slot( @@ -139,7 +136,7 @@ impl<'a> Context<'a> { } } - // Allocate a global variable slot and add a mapping number -> GlobalVar. + // Allocate a global variable slot. fn add_gv(&mut self, gv: GlobalVar, data: GlobalVarData, loc: &Location) -> Result<()> { while self.function.global_vars.next_key().index() <= gv.index() { self.function.create_global_var(GlobalVarData::Sym { @@ -159,7 +156,7 @@ impl<'a> Context<'a> { } } - // Allocate a heap slot and add a mapping number -> Heap. + // Allocate a heap slot. fn add_heap(&mut self, heap: Heap, data: HeapData, loc: &Location) -> Result<()> { while self.function.heaps.next_key().index() <= heap.index() { self.function.create_heap(HeapData { @@ -182,7 +179,7 @@ impl<'a> Context<'a> { } } - // Allocate a new signature and add a mapping number -> SigRef. + // Allocate a new signature. fn add_sig(&mut self, sig: SigRef, data: Signature, loc: &Location) -> Result<()> { while self.function.dfg.signatures.next_key().index() <= sig.index() { self.function.import_signature( @@ -202,7 +199,7 @@ impl<'a> Context<'a> { } } - // Allocate a new external function and add a mapping number -> FuncRef. + // Allocate a new external function. fn add_fn(&mut self, fn_: FuncRef, data: ExtFuncData, loc: &Location) -> Result<()> { while self.function.dfg.ext_funcs.next_key().index() <= fn_.index() { self.function.import_function(ExtFuncData { @@ -223,7 +220,7 @@ impl<'a> Context<'a> { } } - // Allocate a new jump table and add a mapping number -> JumpTable. + // Allocate a new jump table. fn add_jt(&mut self, jt: JumpTable, data: JumpTableData, loc: &Location) -> Result<()> { while self.function.jump_tables.next_key().index() <= jt.index() { self.function.create_jump_table(JumpTableData::new()); @@ -241,7 +238,7 @@ impl<'a> Context<'a> { } } - // Allocate a new EBB and add a mapping src_ebb -> Ebb. + // Allocate a new EBB. fn add_ebb(&mut self, ebb: Ebb, loc: &Location) -> Result { while self.function.dfg.num_ebbs() <= ebb.index() { self.function.dfg.make_ebb(); @@ -446,7 +443,6 @@ impl<'a> Parser<'a> { } // Match and consume a value reference, direct or vtable. - // This does not convert from the source value numbering to our in-memory value numbering. fn match_value(&mut self, err_msg: &str) -> Result { if let Some(Token::Value(v)) = self.token() { self.consume(); @@ -1427,7 +1423,7 @@ impl<'a> Parser<'a> { } // Parse parenthesized list of EBB parameters. Returns a vector of (u32, Type) pairs with the - // source value numbers of the defined values and the defined types. + // value numbers of the defined values and the defined types. // // ebb-params ::= * "(" ebb-param { "," ebb-param } ")" fn parse_ebb_params(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { @@ -1476,7 +1472,7 @@ impl<'a> Parser<'a> { } let t = self.match_type("expected EBB argument type")?; - // Allocate the EBB argument and add the mapping. + // Allocate the EBB argument. ctx.function.dfg.append_ebb_param_for_parser(ebb, t, v); ctx.map.def_value(v, &v_location)?; @@ -1936,7 +1932,7 @@ impl<'a> Parser<'a> { } InstructionFormat::NullAry => InstructionData::NullAry { opcode }, InstructionFormat::Jump => { - // Parse the destination EBB number. Don't translate source to local numbers yet. + // Parse the destination EBB number. let ebb_num = self.match_ebb("expected jump destination EBB")?; let args = self.parse_opt_value_list()?; InstructionData::Jump { From 5dc449ec9e331edebeb4e7419d8c88a4a302c3a1 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Feb 2018 13:58:46 -0800 Subject: [PATCH 1565/3084] Rename "local variables" to "explicit stack slots". The term "local variables" predated the SSA builder in the front-end crate, which also provides a way to implement source-language local variables. The name "explicit stack slot" makes it clear what this construct is. --- cranelift/docs/example.cton | 2 +- cranelift/docs/langref.rst | 23 ++++++++++++------- .../isa/intel/prologue-epilogue.cton | 6 ++--- cranelift/filetests/parser/tiny.cton | 4 ++-- lib/cretonne/src/ir/stackslot.rs | 12 +++++----- lib/cretonne/src/stack_layout.rs | 17 +++++++------- lib/cretonne/src/write.rs | 10 ++++---- lib/reader/src/parser.rs | 2 +- 8 files changed, 42 insertions(+), 34 deletions(-) diff --git a/cranelift/docs/example.cton b/cranelift/docs/example.cton index 1a3117c521..a36fef3238 100644 --- a/cranelift/docs/example.cton +++ b/cranelift/docs/example.cton @@ -1,7 +1,7 @@ test verifier function %average(i32, i32) -> f32 native { - ss1 = local 8 ; Stack slot for ``sum``. + ss1 = explicit_slot 8 ; Stack slot for ``sum``. ebb1(v1: i32, v2: i32): v3 = f64const 0x0.0 diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index d80554dc67..476e0a262d 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -37,7 +37,7 @@ The first line of a function definition provides the function *name* and the :term:`function signature` which declares the parameter and return types. Then follows the :term:`function preamble` which declares a number of entities that can be referenced inside the function. In the example above, the preamble -declares a single local variable, ``ss1``. +declares a single explicit stack slot, ``ss1``. After the preamble follows the :term:`function body` which consists of :term:`extended basic block`\s (EBBs), the first of which is the @@ -471,8 +471,8 @@ the expected alignment. By default, misaligned loads and stores are allowed, but when the ``aligned`` flag is set, a misaligned memory access is allowed to :term:`trap`. -Local variables ---------------- +Explicit Stack Slots +-------------------- One set of restricted memory operations access the current function's stack frame. The stack frame is divided into fixed-size stack slots that are @@ -480,9 +480,9 @@ allocated in the :term:`function preamble`. Stack slots are not typed, they simply represent a contiguous sequence of :term:`accessible` bytes in the stack frame. -.. inst:: SS = local Bytes, Flags... +.. inst:: SS = explicit_slot Bytes, Flags... - Allocate a stack slot for a local variable in the preamble. + Allocate a stack slot in the preamble. If no alignment is specified, Cretonne will pick an appropriate alignment for the stack slot based on its size and access patterns. @@ -1135,7 +1135,7 @@ Glossary A list of declarations of entities that are used by the function body. Some of the entities that can be declared in the preamble are: - - Local variables. + - Stack slots. - Functions that are called directly. - Function signatures for indirect function calls. - Function flags and attributes that are not part of the signature. @@ -1158,9 +1158,16 @@ Glossary intermediate representation. Cretonne's IR can be converted to text losslessly. - stack slot + explicit stack slot A fixed size memory allocation in the current function's activation - frame. Also called a local variable. + frame. These differ from :term:`spill stack slots` in that they can + be created by frontends and they may have their addresses taken. + + spill stack slot + A fixed size memory allocation in the current function's activation + frame. These differ from :term:`explicit stack slots` in that they are + only created during register allocation, and they may not have their + address taken. terminator instruction A control flow instruction that unconditionally directs the flow of diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/intel/prologue-epilogue.cton index 425b790fbf..fc53f72008 100644 --- a/cranelift/filetests/isa/intel/prologue-epilogue.cton +++ b/cranelift/filetests/isa/intel/prologue-epilogue.cton @@ -4,13 +4,13 @@ set is_compressed isa intel haswell function %foo() { - ss0 = local 168 + ss0 = explicit_slot 168 ebb0: return } ; check: function %foo(i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] native { -; nextln: ss0 = local 168, offset -224 +; nextln: ss0 = explicit_slot 168, offset -224 ; nextln: ss1 = incoming_arg 56, offset -56 ; check: ebb0(v0: i64 [%rbp], v1: i64 [%rbx], v2: i64 [%r12], v3: i64 [%r13], v4: i64 [%r14], v5: i64 [%r15]): ; nextln: x86_push v0 @@ -29,4 +29,4 @@ ebb0: ; nextln: v7 = x86_pop.i64 ; nextln: v6 = x86_pop.i64 ; nextln: return v6, v7, v8, v9, v10, v11 -; nextln: } \ No newline at end of file +; nextln: } diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index a5a6489808..0c619565a0 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -124,7 +124,7 @@ ebb0(v90: i32, v91: f32): ; Stack slot references function %stack() { ss10 = spill_slot 8 - ss2 = local 4 + ss2 = explicit_slot 4 ss3 = incoming_arg 4, offset 8 ss4 = outgoing_arg 4 ss5 = emergency_slot 4 @@ -136,7 +136,7 @@ ebb0: stack_store v2, ss2 } ; sameln: function %stack() native { -; check: ss2 = local 4 +; check: ss2 = explicit_slot 4 ; check: ss3 = incoming_arg 4, offset 8 ; check: ss4 = outgoing_arg 4 ; check: ss5 = emergency_slot 4 diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index 4a0804787b..172a43e3af 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -40,9 +40,9 @@ pub enum StackSlotKind { /// A spill slot. This is a stack slot created by the register allocator. SpillSlot, - /// A local variable. This is a chunk of local stack memory for use by the `stack_load` and - /// `stack_store` instructions. - Local, + /// An explicit stack slot. This is a chunk of stack memory for use by the `stack_load` + /// and `stack_store` instructions. + ExplicitSlot, /// An incoming function argument. /// @@ -71,7 +71,7 @@ impl FromStr for StackSlotKind { fn from_str(s: &str) -> Result { use self::StackSlotKind::*; match s { - "local" => Ok(Local), + "explicit_slot" => Ok(ExplicitSlot), "spill_slot" => Ok(SpillSlot), "incoming_arg" => Ok(IncomingArg), "outgoing_arg" => Ok(OutgoingArg), @@ -85,7 +85,7 @@ impl fmt::Display for StackSlotKind { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { use self::StackSlotKind::*; f.write_str(match *self { - Local => "local", + ExplicitSlot => "explicit_slot", SpillSlot => "spill_slot", IncomingArg => "incoming_arg", OutgoingArg => "outgoing_arg", @@ -366,7 +366,7 @@ mod tests { assert_eq!(slot.alignment(8), 8); assert_eq!(slot.alignment(16), 8); - let slot2 = StackSlotData::new(StackSlotKind::Local, 24); + let slot2 = StackSlotData::new(StackSlotKind::ExplicitSlot, 24); assert_eq!(slot2.alignment(4), 4); assert_eq!(slot2.alignment(8), 8); diff --git a/lib/cretonne/src/stack_layout.rs b/lib/cretonne/src/stack_layout.rs index 796850e059..7d0550dbba 100644 --- a/lib/cretonne/src/stack_layout.rs +++ b/lib/cretonne/src/stack_layout.rs @@ -7,8 +7,8 @@ use std::cmp::{min, max}; /// Compute the stack frame layout. /// -/// Determine the total size of this stack frame and assign offsets to all `Spill` and `Local` -/// stack slots. +/// Determine the total size of this stack frame and assign offsets to all `Spill` and +/// `Explicit` stack slots. /// /// The total frame size will be a multiple of `alignment` which must be a power of two. /// @@ -25,7 +25,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result Result { - // Determine the smallest alignment of any local or spill slot. + // Determine the smallest alignment of any explicit or spill slot. min_align = slot.alignment(min_align); } } } - // Lay out spill slots and locals below the incoming arguments. + // Lay out spill slots and explicit slots below the incoming arguments. // The offset is negative, growing downwards. // Start with the smallest alignments for better packing. let mut offset = incoming_min; @@ -74,9 +74,10 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result { + StackSlotKind::SpillSlot | + StackSlotKind::ExplicitSlot => { if slot.alignment(alignment) != min_align { continue; } diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index d9853d5768..02f0fa3ab2 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -475,29 +475,29 @@ mod tests { f.name = ExternalName::testcase("foo"); assert_eq!(f.to_string(), "function %foo() native {\n}\n"); - f.create_stack_slot(StackSlotData::new(StackSlotKind::Local, 4)); + f.create_stack_slot(StackSlotData::new(StackSlotKind::ExplicitSlot, 4)); assert_eq!( f.to_string(), - "function %foo() native {\n ss0 = local 4\n}\n" + "function %foo() native {\n ss0 = explicit_slot 4\n}\n" ); let ebb = f.dfg.make_ebb(); f.layout.append_ebb(ebb); assert_eq!( f.to_string(), - "function %foo() native {\n ss0 = local 4\n\nebb0:\n}\n" + "function %foo() native {\n ss0 = explicit_slot 4\n\nebb0:\n}\n" ); f.dfg.append_ebb_param(ebb, types::I8); assert_eq!( f.to_string(), - "function %foo() native {\n ss0 = local 4\n\nebb0(v0: i8):\n}\n" + "function %foo() native {\n ss0 = explicit_slot 4\n\nebb0(v0: i8):\n}\n" ); f.dfg.append_ebb_param(ebb, types::F32.by(4).unwrap()); assert_eq!( f.to_string(), - "function %foo() native {\n ss0 = local 4\n\nebb0(v0: i8, v1: f32x4):\n}\n" + "function %foo() native {\n ss0 = explicit_slot 4\n\nebb0(v0: i8, v1: f32x4):\n}\n" ); } } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 47f14129a9..62a26128ca 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1044,7 +1044,7 @@ impl<'a> Parser<'a> { // Parse a stack slot decl. // // stack-slot-decl ::= * StackSlot(ss) "=" stack-slot-kind Bytes {"," stack-slot-flag} - // stack-slot-kind ::= "local" + // stack-slot-kind ::= "explicit_slot" // | "spill_slot" // | "incoming_arg" // | "outgoing_arg" From 162ca42b08c83403985389c1a267b6ef6a5df1b7 Mon Sep 17 00:00:00 2001 From: Bruce Mitchener Date: Mon, 5 Mar 2018 10:52:19 +0700 Subject: [PATCH 1566/3084] Update term dep to 0.5 --- cranelift/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index d762d64d82..37b4456e82 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -24,7 +24,7 @@ serde = "1.0.8" serde_derive = "1.0.8" num_cpus = "1.5.1" tempdir="0.3.5" -term = "0.4.6" +term = "0.5" [workspace] From 125270e2b0aed7efcd0fa41c23c6648d4bb13d6a Mon Sep 17 00:00:00 2001 From: Bruce Mitchener Date: Mon, 5 Mar 2018 10:48:48 +0700 Subject: [PATCH 1567/3084] Fix some typos. --- cranelift/docs/langref.rst | 4 ++-- cranelift/src/compile.rs | 2 +- cranelift/src/filetest/concurrent.rs | 2 +- cranelift/src/filetest/runner.rs | 2 +- lib/cretonne/src/regalloc/reload.rs | 2 +- lib/filecheck/src/error.rs | 2 +- lib/filecheck/src/pattern.rs | 6 +++--- lib/filecheck/src/variable.rs | 2 +- lib/filecheck/tests/basic.rs | 2 +- lib/frontend/src/frontend.rs | 6 +++--- lib/frontend/src/ssa.rs | 16 ++++++++-------- lib/reader/src/testcommand.rs | 2 +- lib/reader/src/testfile.rs | 2 +- lib/wasm/src/code_translator.rs | 4 ++-- lib/wasm/src/environ/spec.rs | 2 +- lib/wasm/src/module_translator.rs | 2 +- lib/wasm/src/sections_translator.rs | 2 +- 17 files changed, 30 insertions(+), 30 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 476e0a262d..b91f1b3f47 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -440,7 +440,7 @@ Cretonne provides fully general :inst:`load` and :inst:`store` instructions for accessing memory, as well as :ref:`extending loads and truncating stores `. -If the memory at the given addresss is not :term:`addressable`, the behavior of +If the memory at the given address is not :term:`addressable`, the behavior of these instructions is undefined. If it is addressable but not :term:`accessible`, they :term:`trap`. @@ -559,7 +559,7 @@ runtime data structures. The address of GV can be computed by first loading a pointer from BaseGV and adding Offset to it. - It is assumed the BaseGV resides in readable memory with the apropriate + It is assumed the BaseGV resides in readable memory with the appropriate alignment for storing a pointer. Chains of ``deref`` global variables are possible, but cycles are not diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index ae0c87ad36..fd2e5d5552 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -37,7 +37,7 @@ fn handle_module( let test_file = parse_test(&buffer).map_err(|e| format!("{}: {}", name, e))?; // If we have an isa from the command-line, use that. Otherwise if the - // file contins a unique isa, use that. + // file contains a unique isa, use that. let isa = if let Some(isa) = fisa.isa { isa } else if let Some(isa) = test_file.isa_spec.unique_isa() { diff --git a/cranelift/src/filetest/concurrent.rs b/cranelift/src/filetest/concurrent.rs index cb188577e0..a651c14071 100644 --- a/cranelift/src/filetest/concurrent.rs +++ b/cranelift/src/filetest/concurrent.rs @@ -119,7 +119,7 @@ fn worker_thread( loop { // Lock the mutex only long enough to extract a request. let Request(jobid, path) = match requests.lock().unwrap().recv() { - Err(..) => break, // TX end shuit down. exit thread. + Err(..) => break, // TX end shut down. exit thread. Ok(req) => req, }; diff --git a/cranelift/src/filetest/runner.rs b/cranelift/src/filetest/runner.rs index 320db9d942..09ec3a19a9 100644 --- a/cranelift/src/filetest/runner.rs +++ b/cranelift/src/filetest/runner.rs @@ -221,7 +221,7 @@ impl TestRunner { } self.tests[jobid].state = State::Done(result); - // Rports jobs in order. + // Reports jobs in order. while self.report_job() { self.reported_tests += 1; } diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 2cb37d7d6a..12fd8d310a 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -298,7 +298,7 @@ impl<'a> Context<'a> { } } - // Find reload candidates for `inst` and add them to `self.condidates`. + // Find reload candidates for `inst` and add them to `self.candidates`. // // These are uses of spilled values where the operand constraint requires a register. fn find_candidates(&mut self, inst: Inst, constraints: &RecipeConstraints) { diff --git a/lib/filecheck/src/error.rs b/lib/filecheck/src/error.rs index 5580cc4c3c..f15cb17124 100644 --- a/lib/filecheck/src/error.rs +++ b/lib/filecheck/src/error.rs @@ -19,7 +19,7 @@ pub enum Error { UndefVariable(String), /// A pattern contains a back-reference to a variable that was defined in the same pattern. /// - /// For example, `check: Hello $(world=.*) $world`. Backreferences are not support. Often the + /// For example, `check: Hello $(world=.*) $world`. Backreferences are not supported. Often the /// desired effect can be achieved with the `sameln` check: /// /// ```text diff --git a/lib/filecheck/src/pattern.rs b/lib/filecheck/src/pattern.rs index 934150dca7..97977b191c 100644 --- a/lib/filecheck/src/pattern.rs +++ b/lib/filecheck/src/pattern.rs @@ -42,7 +42,7 @@ pub enum Part { } impl Part { - /// Get the variabled referenced by this part, if any. + /// Get the variable referenced by this part, if any. pub fn ref_var(&self) -> Option<&str> { match *self { Part::Var(ref var) | @@ -217,10 +217,10 @@ impl Pattern { } /// Compute the length of a regular expression terminated by `)` or `}`. -/// Handle nested and escaped parentheses in the rx, but don't actualy parse it. +/// Handle nested and escaped parentheses in the rx, but don't actually parse it. /// Return the position of the terminating brace or the length of the string. fn regex_prefix(s: &str) -> usize { - // The prevous char was a backslash. + // The previous char was a backslash. let mut escape = false; // State around parsing charsets. enum State { diff --git a/lib/filecheck/src/variable.rs b/lib/filecheck/src/variable.rs index 1a43f1428a..5977f06354 100644 --- a/lib/filecheck/src/variable.rs +++ b/lib/filecheck/src/variable.rs @@ -3,7 +3,7 @@ use std::borrow::Cow; /// A variable name is one or more ASCII alphanumerical characters, including underscore. /// Note that numerical variable names like `$45` are allowed too. /// -/// Try to parse a variable name from the begining of `s`. +/// Try to parse a variable name from the beginning of `s`. /// Return the index of the character following the varname. /// This returns 0 if `s` doesn't have a prefix that is a variable name. pub fn varname_prefix(s: &str) -> usize { diff --git a/lib/filecheck/tests/basic.rs b/lib/filecheck/tests/basic.rs index debced2a1b..1ccc5991a5 100644 --- a/lib/filecheck/tests/basic.rs +++ b/lib/filecheck/tests/basic.rs @@ -153,7 +153,7 @@ fn nextln() { #[test] fn leading_nextln() { // A leading nextln directive should match from line 2. - // This is somewhat arbitrary, but consistent with a preceeding 'check: $()' directive. + // This is somewhat arbitrary, but consistent with a preceding 'check: $()' directive. let c = CheckerBuilder::new() .text( " diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 432288e4eb..fba2f89a01 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -12,7 +12,7 @@ use cretonne::packed_option::PackedOption; /// Structure used for translating a series of functions into Cretonne IL. /// -/// In order to reduce memory reallocations whem compiling multiple functions, +/// In order to reduce memory reallocations when compiling multiple functions, /// `ILBuilder` holds various data structures which are cleared between /// functions, rather than dropped, preserving the underlying allocations. pub struct ILBuilder @@ -259,7 +259,7 @@ where /// block, in the order they are declared. You must declare the types of the Ebb arguments /// you will use here. /// - /// When inserting the terminator instruction (which doesn't have a falltrough to its immediate + /// When inserting the terminator instruction (which doesn't have a fallthrough to its immediate /// successor), the block will be declared filled and it will not be possible to append /// instructions to it. pub fn switch_to_block(&mut self, ebb: Ebb) { @@ -283,7 +283,7 @@ where /// Declares that all the predecessors of this block are known. /// /// Function to call with `ebb` as soon as the last branch instruction to `ebb` has been - /// created. Forgetting to call this method on every block will cause inconsistences in the + /// created. Forgetting to call this method on every block will cause inconsistencies in the /// produced functions. pub fn seal_block(&mut self, ebb: Ebb) { let side_effects = self.builder.ssa.seal_ebb_header_block(ebb, self.func); diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 2b758ef18b..aa6aa81166 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -35,7 +35,7 @@ pub struct SSABuilder where Variable: EntityRef, { - // Records for every variable and for every revelant block, the last definition of + // Records for every variable and for every relevant block, the last definition of // the variable in the block. variables: EntityMap>>, // Records the position of the basic blocks and the list of values used but not defined in the @@ -81,7 +81,7 @@ enum BlockData { // A block at the top of an `Ebb`. EbbHeader(EbbHeaderBlockData), // A block inside an `Ebb` with an unique other block as its predecessor. - // The block is implicitely sealed at creation. + // The block is implicitly sealed at creation. EbbBody { predecessor: Block }, } @@ -394,7 +394,7 @@ where /// Remove a previously declared Ebb predecessor by giving a reference to the jump /// instruction. Returns the basic block containing the instruction. /// - /// Note: use only when you know what you are doing, this might break the SSA bbuilding problem + /// Note: use only when you know what you are doing, this might break the SSA building problem pub fn remove_ebb_predecessor(&mut self, ebb: Ebb, inst: Inst) -> Block { debug_assert!(!self.is_sealed(ebb)); let header_block = self.header_block(ebb); @@ -422,7 +422,7 @@ where pub fn seal_all_ebb_header_blocks(&mut self, func: &mut Function) -> SideEffects { // Seal all `Ebb`s currently in the function. This can entail splitting // and creation of new blocks, however such new blocks are sealed on - // the fly, so we don't need to accout for them here. + // the fly, so we don't need to account for them here. for ebb in self.ebb_headers.keys() { self.seal_one_ebb_header_block(ebb, func); } @@ -470,8 +470,8 @@ where } } - /// Look up in the predecessors of an Ebb the def for a value an decides wether or not - /// to keep the eeb arg, and act accordingly. Returns the chosen value and optionnaly a + /// Look up in the predecessors of an Ebb the def for a value an decides whether or not + /// to keep the eeb arg, and act accordingly. Returns the chosen value and optionally a /// list of Ebb that are the middle of newly created critical edges splits. fn predecessors_lookup( &mut self, @@ -554,7 +554,7 @@ where ZeroOneOrMore::One(pred_val) => { // Here all the predecessors use a single value to represent our variable // so we don't need to have it as an ebb argument. - // We need to replace all the occurences of val with pred_val but since + // We need to replace all the occurrences of val with pred_val but since // we can't afford a re-writing pass right now we just declare an alias. // Resolve aliases eagerly so that we can check for cyclic aliasing, // which can occur in unreachable code. @@ -1119,7 +1119,7 @@ mod tests { #[test] fn undef() { - // Use vars of varous types which have not been defined. + // Use vars of various types which have not been defined. let mut func = Function::new(); let mut ssa: SSABuilder = SSABuilder::new(); let ebb0 = func.dfg.make_ebb(); diff --git a/lib/reader/src/testcommand.rs b/lib/reader/src/testcommand.rs index ad856e9949..2b848a9348 100644 --- a/lib/reader/src/testcommand.rs +++ b/lib/reader/src/testcommand.rs @@ -9,7 +9,7 @@ //! //! The options are either a single identifier flag, or setting values like `identifier=value`. //! -//! The parser does not understand the test commands or which options are alid. It simply parses +//! The parser does not understand the test commands or which options are valid. It simply parses //! the general format into a `TestCommand` data structure. use std::fmt::{self, Display, Formatter}; diff --git a/lib/reader/src/testfile.rs b/lib/reader/src/testfile.rs index 20e03939a1..8ec5271a07 100644 --- a/lib/reader/src/testfile.rs +++ b/lib/reader/src/testfile.rs @@ -42,7 +42,7 @@ pub struct Details<'a> { /// A comment in a parsed function. /// -/// The comment belongs to the immediately preceeding entity, whether that is an EBB header, and +/// The comment belongs to the immediately preceding entity, whether that is an EBB header, and /// instruction, or one of the preamble declarations. /// /// Comments appearing inside the function but before the preamble, as well as comments appearing diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 21df608f74..086646fa29 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -51,7 +51,7 @@ pub fn translate_operator( match *op { /********************************** Locals **************************************** * `get_local` and `set_local` are treated as non-SSA variables and will completely - * diseappear in the Cretonne Code + * disappear in the Cretonne Code ***********************************************************************************/ Operator::GetLocal { local_index } => state.push1(builder.use_var(Local(local_index))), Operator::SetLocal { local_index } => { @@ -1003,7 +1003,7 @@ fn translate_unreachable_operator( builder.switch_to_block(frame.following_code()); builder.seal_block(frame.following_code()); - // And add the return values of the block but only if the next block is reachble + // And add the return values of the block but only if the next block is reachable // (which corresponds to testing if the stack depth is 1) stack.extend_from_slice(builder.ebb_params(frame.following_code())); state.reachable = true; diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 31e4782918..732438a573 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -146,7 +146,7 @@ pub trait FuncEnvironment { ) -> ir::Value; } -/// An object satisfyng the `ModuleEnvironment` trait can be passed as argument to the +/// An object satisfying the `ModuleEnvironment` trait can be passed as argument to the /// [`translate_module`](fn.translate_module.html) function. These methods should not be called /// by the user, they are only for `cretonne-wasm` internal use. pub trait ModuleEnvironment<'data> { diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index d50031fb04..df640cbb49 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -1,4 +1,4 @@ -//! Translation skeletton that traverses the whole WebAssembly module and call helper functions +//! Translation skeleton that traverses the whole WebAssembly module and call helper functions //! to deal with each part of it. use cretonne::timing; use wasmparser::{ParserState, SectionCode, ParserInput, Parser, WasmDecoder, BinaryReaderError}; diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index d02a4a7c2a..359510547a 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -115,7 +115,7 @@ pub fn parse_import_section<'data>( Ok(()) } -/// Retrieves the correspondances between functions and signatures from the function section +/// Retrieves the correspondences between functions and signatures from the function section pub fn parse_function_section( parser: &mut Parser, environ: &mut ModuleEnvironment, From 505fe9277af946cc683fcf856f3129148217f5e0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 1 Mar 2018 06:43:16 -0800 Subject: [PATCH 1568/3084] Tidy up calls to `analyze_branch`. --- lib/cretonne/src/dominator_tree.rs | 2 +- lib/cretonne/src/flowgraph.rs | 2 +- lib/cretonne/src/regalloc/coloring.rs | 2 +- lib/cretonne/src/regalloc/live_value_tracker.rs | 2 +- lib/cretonne/src/verifier/locations.rs | 2 +- lib/cretonne/src/verifier/mod.rs | 2 +- lib/frontend/src/ssa.rs | 4 ++-- 7 files changed, 8 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index f7b7273681..27caf338dc 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -340,7 +340,7 @@ impl DominatorTree { /// post-order except for the insertion of the new EBB header at the split point. fn push_successors(&mut self, func: &Function, ebb: Ebb) { for inst in func.layout.ebb_insts(ebb) { - match func.dfg[inst].analyze_branch(&func.dfg.value_lists) { + match func.dfg.analyze_branch(inst) { BranchInfo::SingleDest(succ, _) => { if self.nodes[succ].rpo_number == 0 { self.nodes[succ].rpo_number = SEEN; diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index d1e45312ac..fbc74b79d0 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -108,7 +108,7 @@ impl ControlFlowGraph { fn compute_ebb(&mut self, func: &Function, ebb: Ebb) { for inst in func.layout.ebb_insts(ebb) { - match func.dfg[inst].analyze_branch(&func.dfg.value_lists) { + match func.dfg.analyze_branch(inst) { BranchInfo::SingleDest(dest, _) => { self.add_edge((ebb, inst), dest); } diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 74d80e1832..fb60a979c8 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -908,7 +908,7 @@ impl<'a> Context<'a> { let inst = self.cur.current_inst().expect("Not on an instruction"); let ctx = self.liveness.context(&self.cur.func.layout); - match self.cur.func.dfg[inst].analyze_branch(&self.cur.func.dfg.value_lists) { + match self.cur.func.dfg.analyze_branch(inst) { NotABranch => false, SingleDest(ebb, _) => { let lr = &self.liveness[value]; diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index d0810a5cf7..3d7eeaa6e5 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -260,7 +260,7 @@ impl LiveValueTracker { ) -> (&[LiveValue], &[LiveValue], &[LiveValue]) { // Save a copy of the live values before any branches or jumps that could be somebody's // immediate dominator. - match dfg[inst].analyze_branch(&dfg.value_lists) { + match dfg.analyze_branch(inst) { BranchInfo::NotABranch => {} _ => self.save_idom_live_set(inst), } diff --git a/lib/cretonne/src/verifier/locations.rs b/lib/cretonne/src/verifier/locations.rs index 2c93d0de3e..535d04f0ef 100644 --- a/lib/cretonne/src/verifier/locations.rs +++ b/lib/cretonne/src/verifier/locations.rs @@ -274,7 +274,7 @@ impl<'a> LocationVerifier<'a> { }; let dfg = &self.func.dfg; - match dfg[inst].analyze_branch(&dfg.value_lists) { + match dfg.analyze_branch(inst) { NotABranch => { panic!( "No branch information for {}", diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index a346573e67..9bc4c0459d 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -693,7 +693,7 @@ impl<'a> Verifier<'a> { } fn typecheck_variable_args(&self, inst: Inst) -> Result { - match self.func.dfg[inst].analyze_branch(&self.func.dfg.value_lists) { + match self.func.dfg.analyze_branch(inst) { BranchInfo::SingleDest(ebb, _) => { let iter = self.func.dfg.ebb_params(ebb).iter().map(|&v| { self.func.dfg.value_type(v) diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index aa6aa81166..06dba9d4ed 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -620,7 +620,7 @@ where val: Value, var: Variable, ) -> Option<(Ebb, Block, Inst)> { - match func.dfg[jump_inst].analyze_branch(&func.dfg.value_lists) { + match func.dfg.analyze_branch(jump_inst) { BranchInfo::NotABranch => { panic!("you have declared a non-branch instruction as a predecessor to an ebb"); } @@ -855,7 +855,7 @@ mod tests { cur.ins().iadd(x_use3, y_use3) }; ssa.def_var(y_var, y2_ssa, block2); - match func.dfg[jump_inst].analyze_branch(&func.dfg.value_lists) { + match func.dfg.analyze_branch(jump_inst) { BranchInfo::SingleDest(dest, jump_args) => { assert_eq!(dest, ebb1); assert_eq!(jump_args.len(), 0); From 804b56d0f2b254ee5f0eb9d05b153bf961777551 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 2 Mar 2018 13:16:01 -0800 Subject: [PATCH 1569/3084] Document that "enable_float=false" isn't implemented yet. --- lib/cretonne/meta/base/settings.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/meta/base/settings.py b/lib/cretonne/meta/base/settings.py index 8d0522af79..e8785fbc08 100644 --- a/lib/cretonne/meta/base/settings.py +++ b/lib/cretonne/meta/base/settings.py @@ -56,7 +56,11 @@ avoid_div_traps = BoolSetting( is_compressed = BoolSetting("Enable compressed instructions") enable_float = BoolSetting( - """Enable the use of floating-point instructions""", + """ + Enable the use of floating-point instructions + + Disabling use of floating-point instructions is not yet implemented. + """, default=True) enable_simd = BoolSetting( From aac006ed4933c230a99736f4ebb8afe4929677ca Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 2 Mar 2018 13:20:11 -0800 Subject: [PATCH 1570/3084] Avoid trivial numeric casts. --- lib/cretonne/src/divconst_magic_numbers.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/divconst_magic_numbers.rs b/lib/cretonne/src/divconst_magic_numbers.rs index 64416e48fe..a34608359f 100644 --- a/lib/cretonne/src/divconst_magic_numbers.rs +++ b/lib/cretonne/src/divconst_magic_numbers.rs @@ -498,14 +498,14 @@ mod tests { println!("Testing UP magicU64"); for x in 2..(200 * 1000u64) { let m = magicU64(x); - total = total ^ (m.mulBy as u64); + total = total ^ m.mulBy; total = total + (m.shiftBy as u64); total = total - (if m.doAdd { 123 } else { 456 }); } println!("Testing DOWN magicU64"); for x in 0..(200 * 1000u64) { let m = magicU64(0xFFFF_FFFF_FFFF_FFFFu64 - x); - total = total ^ (m.mulBy as u64); + total = total ^ m.mulBy; total = total + (m.shiftBy as u64); total = total - (if m.doAdd { 123 } else { 456 }); } From a301280d9456ffc1f3170b5038dbc192ec290ff0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 2 Mar 2018 13:22:09 -0800 Subject: [PATCH 1571/3084] Change the stack alignment for 32-bit x86 to 16. Spiderwasm on 32-bit x86 always uses a 16-byte-aligned stack pointer. Change the setting for the "native" convention as well, for compatibility with Linux and Darwin ABIs, and so that if a platform has different ABI rules, the problem will be detected in code emitted by Cretonne, rather than somewhere else. --- lib/cretonne/src/isa/intel/abi.rs | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index cf30058364..c7dd1af93c 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -180,11 +180,9 @@ pub fn spiderwasm_prologue_epilogue( func: &mut ir::Function, isa: &TargetIsa, ) -> result::CtonResult { - let (word_size, stack_align) = if isa.flags().is_64bit() { - (8, 16) - } else { - (4, 4) - }; + // Spiderwasm on 32-bit x86 always aligns its stack pointer to 16 bytes. + let stack_align = 16; + let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; let bytes = StackSize::from(isa.flags().spiderwasm_prologue_words()) * word_size; let mut ss = ir::StackSlotData::new(ir::StackSlotKind::IncomingArg, bytes); @@ -197,11 +195,10 @@ pub fn spiderwasm_prologue_epilogue( /// Insert a System V-compatible prologue and epilogue. pub fn native_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { - let (word_size, stack_align) = if isa.flags().is_64bit() { - (8, 16) - } else { - (4, 4) - }; + // The original 32-bit x86 ELF ABI had a 4-byte aligned stack pointer, but + // newer versions use a 16-byte aligned stack pointer. + let stack_align = 16; + let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; let csr_type = if isa.flags().is_64bit() { ir::types::I64 } else { From 13b167770c112930604b6da18c4897f808d518bf Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sat, 3 Mar 2018 13:21:10 -0800 Subject: [PATCH 1572/3084] Include emergency stack slots when laying out the stack. Emergency stack slots are a new kind of stack slot added relatively recently. They need to be allocated a stack offset just like explicit and spill slots. Also, make StackSlotData's offset field an Option, to catch problems like this in the future. Previously the value 0 was used when offsets weren't assigned yet, however that made it non-obvious when the field meant "not assigned yet" and when it meant "assigned the value 0". --- cranelift/src/filetest/binemit.rs | 2 +- lib/cretonne/src/ir/stackslot.rs | 22 +++--- lib/cretonne/src/isa/intel/abi.rs | 4 +- lib/cretonne/src/isa/mod.rs | 2 +- lib/cretonne/src/isa/stack.rs | 4 +- lib/cretonne/src/stack_layout.rs | 99 ++++++++++++++++---------- lib/cretonne/src/verifier/locations.rs | 4 +- lib/cretonne/src/verifier/mod.rs | 2 +- lib/reader/src/parser.rs | 2 +- 9 files changed, 81 insertions(+), 60 deletions(-) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index 550703f0bf..d2d179fea8 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -126,7 +126,7 @@ impl SubTest for TestBinEmit { // Fix the stack frame layout so we can test spill/fill encodings. let min_offset = func.stack_slots .keys() - .map(|ss| func.stack_slots[ss].offset) + .map(|ss| func.stack_slots[ss].offset.unwrap()) .min(); func.stack_slots.frame_size = min_offset.map(|off| (-off) as u32); diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index 172a43e3af..d3578a7570 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -111,7 +111,7 @@ pub struct StackSlotData { /// /// For `OutgoingArg` stack slots, the offset is relative to the current function's stack /// pointer immediately before the call. - pub offset: StackOffset, + pub offset: Option, } impl StackSlotData { @@ -120,7 +120,7 @@ impl StackSlotData { StackSlotData { kind, size, - offset: 0, + offset: None, } } @@ -138,8 +138,8 @@ impl StackSlotData { impl fmt::Display for StackSlotData { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "{} {}", self.kind, self.size)?; - if self.offset != 0 { - write!(f, ", offset {}", self.offset)?; + if let Some(offset) = self.offset { + write!(f, ", offset {}", offset)?; } Ok(()) } @@ -204,7 +204,7 @@ impl StackSlots { /// Set the offset of a stack slot. pub fn set_offset(&mut self, ss: StackSlot, offset: StackOffset) { - self.slots[ss].offset = offset; + self.slots[ss].offset = Some(offset); } /// Get an iterator over all the stack slot keys. @@ -245,7 +245,7 @@ impl StackSlots { pub fn make_incoming_arg(&mut self, ty: Type, offset: StackOffset) -> StackSlot { let mut data = StackSlotData::new(StackSlotKind::IncomingArg, ty.bytes()); assert!(offset <= StackOffset::max_value() - data.size as StackOffset); - data.offset = offset; + data.offset = Some(offset); self.push(data) } @@ -261,7 +261,7 @@ impl StackSlots { // Look for an existing outgoing stack slot with the same offset and size. let inspos = match self.outgoing.binary_search_by_key(&(offset, size), |&ss| { - (self[ss].offset, self[ss].size) + (self[ss].offset.unwrap(), self[ss].size) }) { Ok(idx) => return self.outgoing[idx], Err(idx) => idx, @@ -270,7 +270,7 @@ impl StackSlots { // No existing slot found. Make one and insert it into `outgoing`. let mut data = StackSlotData::new(StackSlotKind::OutgoingArg, size); assert!(offset <= StackOffset::max_value() - size as StackOffset); - data.offset = offset; + data.offset = Some(offset); let ss = self.slots.push(data); self.outgoing.insert(inspos, ss); ss @@ -344,13 +344,13 @@ mod tests { let ss1 = sss.get_outgoing_arg(types::I32, 4); let ss2 = sss.get_outgoing_arg(types::I64, 8); - assert_eq!(sss[ss0].offset, 8); + assert_eq!(sss[ss0].offset, Some(8)); assert_eq!(sss[ss0].size, 4); - assert_eq!(sss[ss1].offset, 4); + assert_eq!(sss[ss1].offset, Some(4)); assert_eq!(sss[ss1].size, 4); - assert_eq!(sss[ss2].offset, 8); + assert_eq!(sss[ss2].offset, Some(8)); assert_eq!(sss[ss2].size, 8); assert_eq!(sss.get_outgoing_arg(types::I32, 8), ss0); diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index c7dd1af93c..2296a9292b 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -186,7 +186,7 @@ pub fn spiderwasm_prologue_epilogue( let bytes = StackSize::from(isa.flags().spiderwasm_prologue_words()) * word_size; let mut ss = ir::StackSlotData::new(ir::StackSlotKind::IncomingArg, bytes); - ss.offset = -(bytes as StackOffset); + ss.offset = Some(-(bytes as StackOffset)); func.stack_slots.push(ss); layout_stack(&mut func.stack_slots, stack_align)?; @@ -217,7 +217,7 @@ pub fn native_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> res func.create_stack_slot(ir::StackSlotData { kind: ir::StackSlotKind::IncomingArg, size: csr_stack_size as u32, - offset: -csr_stack_size, + offset: Some(-csr_stack_size), }); let total_stack_size = layout_stack(&mut func.stack_slots, stack_align)? as i32; diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 38164e5167..28b297860c 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -251,7 +251,7 @@ pub trait TargetIsa: fmt::Display { if func.signature.call_conv == ir::CallConv::SpiderWASM { let bytes = StackSize::from(self.flags().spiderwasm_prologue_words()) * word_size; let mut ss = ir::StackSlotData::new(ir::StackSlotKind::IncomingArg, bytes); - ss.offset = -(bytes as StackOffset); + ss.offset = Some(-(bytes as StackOffset)); func.stack_slots.push(ss); } diff --git a/lib/cretonne/src/isa/stack.rs b/lib/cretonne/src/isa/stack.rs index 730db28e18..e8a9b6981b 100644 --- a/lib/cretonne/src/isa/stack.rs +++ b/lib/cretonne/src/isa/stack.rs @@ -41,12 +41,12 @@ impl StackRef { let slot = &frame[ss]; let offset = if slot.kind == StackSlotKind::OutgoingArg { // Outgoing argument slots have offsets relative to our stack pointer. - slot.offset + slot.offset.unwrap() } else { // All other slots have offsets relative to our caller's stack frame. // Offset where SP is pointing. (All ISAs have stacks growing downwards.) let sp_offset = -(size as StackOffset); - slot.offset - sp_offset + slot.offset.unwrap() - sp_offset }; StackRef { base: StackBase::SP, diff --git a/lib/cretonne/src/stack_layout.rs b/lib/cretonne/src/stack_layout.rs index 7d0550dbba..16dcea8d5f 100644 --- a/lib/cretonne/src/stack_layout.rs +++ b/lib/cretonne/src/stack_layout.rs @@ -48,12 +48,13 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result { - incoming_min = min(incoming_min, slot.offset); + incoming_min = min(incoming_min, slot.offset.unwrap()); } StackSlotKind::OutgoingArg => { - let offset = slot.offset.checked_add(slot.size as StackOffset).ok_or( - CtonError::ImplLimitExceeded, - )?; + let offset = slot.offset + .unwrap() + .checked_add(slot.size as StackOffset) + .ok_or(CtonError::ImplLimitExceeded)?; outgoing_max = max(outgoing_max, offset); } StackSlotKind::SpillSlot | @@ -77,12 +78,14 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result { + StackSlotKind::ExplicitSlot | + StackSlotKind::EmergencySlot => { if slot.alignment(alignment) != min_align { continue; } } - _ => continue, + StackSlotKind::IncomingArg | + StackSlotKind::OutgoingArg => continue, } offset = offset.checked_sub(slot.size as StackOffset).ok_or( @@ -111,7 +114,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result LocationVerifier<'a> { slot.kind ); } - if slot.offset != offset { + if slot.offset.unwrap() != offset { return err!( inst, "ABI expects {} at stack offset {}, but {} is at {}", value, offset, ss, - slot.offset + slot.offset.unwrap() ); } } else { diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 9bc4c0459d..33fd3582cb 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -808,7 +808,7 @@ impl<'a> Verifier<'a> { slot ); } - if slot.offset != offset { + if slot.offset != Some(offset) { return err!( inst, "Outgoing stack argument {} should have offset {}: {} = {}", diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 62a26128ca..a9568942ed 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1070,7 +1070,7 @@ impl<'a> Parser<'a> { // Take additional options. while self.optional(Token::Comma) { match self.match_any_identifier("expected stack slot flags")? { - "offset" => data.offset = self.match_imm32("expected byte offset")?, + "offset" => data.offset = Some(self.match_imm32("expected byte offset")?), other => return err!(self.loc, "Unknown stack slot flag '{}'", other), } } From d2d02565fbf2d550a9cccf4fd1c8a844e6a400ed Mon Sep 17 00:00:00 2001 From: Bruce Mitchener Date: Mon, 5 Mar 2018 20:23:50 +0700 Subject: [PATCH 1573/3084] I64Extend16S should operate on 16 bits, not 8. This showed up in clippy warnings as the code for this case was the same as for `I64Extend8S`. --- lib/wasm/src/code_translator.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 086646fa29..765b1a4577 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -654,7 +654,7 @@ pub fn translate_operator( } Operator::I64Extend16S => { let val = state.pop1(); - state.push1(builder.ins().ireduce(I8, val)); + state.push1(builder.ins().ireduce(I16, val)); let val = state.pop1(); state.push1(builder.ins().sextend(I64, val)); } From 0ea0146e58a7b54793435e347f8a26de9f8b8b26 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 2 Mar 2018 22:53:07 -0800 Subject: [PATCH 1574/3084] Update to sphinx-autogen 1.5.6. --- cranelift/docs/Makefile | 192 ++----------------------------- cranelift/docs/conf.py | 63 +++++++--- cranelift/docs/make.bat | 247 ++-------------------------------------- 3 files changed, 70 insertions(+), 432 deletions(-) diff --git a/cranelift/docs/Makefile b/cranelift/docs/Makefile index 635aafde1c..335779f1f3 100644 --- a/cranelift/docs/Makefile +++ b/cranelift/docs/Makefile @@ -1,196 +1,24 @@ -# Makefile for Sphinx documentation +# Minimal makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build SPHINXABUILD = sphinx-autobuild -PAPER = +SPHINXPROJ = cretonne +SOURCEDIR = . BUILDDIR = _build -# User-friendly check for sphinx-build -ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) -$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) -endif - -# Internal variables. -PAPEROPT_a4 = -D latex_paper_size=a4 -PAPEROPT_letter = -D latex_paper_size=letter -ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . -# the i18n builder cannot share the environment and doctrees with the others -I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . - -.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext - +# Put it first so that "make" without argument is like "make help". help: - @echo "Please use \`make ' where is one of" - @echo " html to make standalone HTML files" - @echo " dirhtml to make HTML files named index.html in directories" - @echo " singlehtml to make a single large HTML file" - @echo " pickle to make pickle files" - @echo " json to make JSON files" - @echo " htmlhelp to make HTML files and a HTML help project" - @echo " qthelp to make HTML files and a qthelp project" - @echo " applehelp to make an Apple Help Book" - @echo " devhelp to make HTML files and a Devhelp project" - @echo " epub to make an epub" - @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" - @echo " latexpdf to make LaTeX files and run them through pdflatex" - @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" - @echo " text to make text files" - @echo " man to make manual pages" - @echo " texinfo to make Texinfo files" - @echo " info to make Texinfo files and run them through makeinfo" - @echo " gettext to make PO message catalogs" - @echo " changes to make an overview of all changed/added/deprecated items" - @echo " xml to make Docutils-native XML files" - @echo " pseudoxml to make pseudoxml-XML files for display purposes" - @echo " linkcheck to check all external links for integrity" - @echo " doctest to run all doctests embedded in the documentation (if enabled)" - @echo " coverage to run coverage check of the documentation (if enabled)" - -clean: - rm -rf $(BUILDDIR)/* - -html: - $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) autohtml: html $(SPHINXABUILD) -z ../lib/cretonne/meta --ignore '.*' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html -dirhtml: - $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." +.PHONY: help Makefile -singlehtml: - $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml - @echo - @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." - -pickle: - $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle - @echo - @echo "Build finished; now you can process the pickle files." - -json: - $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json - @echo - @echo "Build finished; now you can process the JSON files." - -htmlhelp: - $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp - @echo - @echo "Build finished; now you can run HTML Help Workshop with the" \ - ".hhp project file in $(BUILDDIR)/htmlhelp." - -qthelp: - $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp - @echo - @echo "Build finished; now you can run "qcollectiongenerator" with the" \ - ".qhcp project file in $(BUILDDIR)/qthelp, like this:" - @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/cretonne.qhcp" - @echo "To view the help file:" - @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/cretonne.qhc" - -applehelp: - $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp - @echo - @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." - @echo "N.B. You won't be able to view it unless you put it in" \ - "~/Library/Documentation/Help or install it in your application" \ - "bundle." - -devhelp: - $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp - @echo - @echo "Build finished." - @echo "To view the help file:" - @echo "# mkdir -p $$HOME/.local/share/devhelp/cretonne" - @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/cretonne" - @echo "# devhelp" - -epub: - $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub - @echo - @echo "Build finished. The epub file is in $(BUILDDIR)/epub." - -latex: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo - @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." - @echo "Run \`make' in that directory to run these through (pdf)latex" \ - "(use \`make latexpdf' here to do that automatically)." - -latexpdf: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through pdflatex..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -latexpdfja: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through platex and dvipdfmx..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -text: - $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text - @echo - @echo "Build finished. The text files are in $(BUILDDIR)/text." - -man: - $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man - @echo - @echo "Build finished. The manual pages are in $(BUILDDIR)/man." - -texinfo: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo - @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." - @echo "Run \`make' in that directory to run these through makeinfo" \ - "(use \`make info' here to do that automatically)." - -info: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo "Running Texinfo files through makeinfo..." - make -C $(BUILDDIR)/texinfo info - @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." - -gettext: - $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale - @echo - @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." - -changes: - $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes - @echo - @echo "The overview file is in $(BUILDDIR)/changes." - -linkcheck: - $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck - @echo - @echo "Link check complete; look for any errors in the above output " \ - "or in $(BUILDDIR)/linkcheck/output.txt." - -doctest: - $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest - @echo "Testing of doctests in the sources finished, look at the " \ - "results in $(BUILDDIR)/doctest/output.txt." - -coverage: - $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage - @echo "Testing of coverage in the sources finished, look at the " \ - "results in $(BUILDDIR)/coverage/python.txt." - -xml: - $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml - @echo - @echo "Build finished. The XML files are in $(BUILDDIR)/xml." - -pseudoxml: - $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml - @echo - @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py index 0603a27bfd..20cfd32f91 100644 --- a/cranelift/docs/conf.py +++ b/cranelift/docs/conf.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # # cretonne documentation build configuration file, created by -# sphinx-quickstart on Fri Jan 8 10:11:19 2016. +# sphinx-quickstart on Fri Mar 2 12:49:24 2018. # # This file is execfile()d with the current directory set to its # containing dir. @@ -12,14 +12,13 @@ # All configuration values have a default; values that are commented out # serve to show the default. -from __future__ import absolute_import -import sys -import os - - # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. + +from __future__ import absolute_import +import os +import sys sys.path.insert(0, os.path.abspath('.')) # Also add the meta directory to sys.path so autodoc can find the Cretonne meta @@ -28,6 +27,10 @@ sys.path.insert(0, os.path.abspath('../lib/cretonne/meta')) # -- General configuration ------------------------------------------------ +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. @@ -47,6 +50,7 @@ templates_path = ['_templates'] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: +# # source_suffix = ['.rst', '.md'] source_suffix = '.rst' @@ -55,7 +59,7 @@ master_doc = 'index' # General information about the project. project = u'cretonne' -copyright = u'2016, Cretonne Developers' +copyright = u'2018, Cretonne Developers' author = u'Cretonne Developers' # The version info for the project you're documenting, acts as replacement for @@ -76,7 +80,8 @@ language = None # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. -exclude_patterns = ['_build'] +# This patterns also effect to html_static_path and html_extra_path +exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' @@ -89,22 +94,54 @@ todo_include_todos = True # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. +# html_theme = 'sphinx_rtd_theme' +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +# +# html_static_path = ['_static'] + + +# -- Options for HTMLHelp output ------------------------------------------ + # Output file base name for HTML help builder. htmlhelp_basename = 'cretonnedoc' + # -- Options for LaTeX output --------------------------------------------- latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ - (master_doc, 'cretonne.tex', u'cretonne Documentation', - author, 'manual'), + (master_doc, 'cretonne.tex', u'cretonne Documentation', + author, 'manual'), ] @@ -124,9 +161,9 @@ man_pages = [ # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ - (master_doc, 'cretonne', u'cretonne Documentation', - author, 'cretonne', 'One line description of project.', - 'Miscellaneous'), + (master_doc, 'cretonne', u'cretonne Documentation', + author, 'cretonne', 'One line description of project.', + 'Miscellaneous'), ] diff --git a/cranelift/docs/make.bat b/cranelift/docs/make.bat index 3f6fe2e48d..9267ac038e 100644 --- a/cranelift/docs/make.bat +++ b/cranelift/docs/make.bat @@ -1,62 +1,19 @@ @ECHO OFF +pushd %~dp0 + REM Command file for Sphinx documentation if "%SPHINXBUILD%" == "" ( set SPHINXBUILD=sphinx-build ) +set SOURCEDIR=. set BUILDDIR=_build -set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . -set I18NSPHINXOPTS=%SPHINXOPTS% . -if NOT "%PAPER%" == "" ( - set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% - set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS% -) +set SPHINXPROJ=cretonne if "%1" == "" goto help -if "%1" == "help" ( - :help - echo.Please use `make ^` where ^ is one of - echo. html to make standalone HTML files - echo. dirhtml to make HTML files named index.html in directories - echo. singlehtml to make a single large HTML file - echo. pickle to make pickle files - echo. json to make JSON files - echo. htmlhelp to make HTML files and a HTML help project - echo. qthelp to make HTML files and a qthelp project - echo. devhelp to make HTML files and a Devhelp project - echo. epub to make an epub - echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter - echo. text to make text files - echo. man to make manual pages - echo. texinfo to make Texinfo files - echo. gettext to make PO message catalogs - echo. changes to make an overview over all changed/added/deprecated items - echo. xml to make Docutils-native XML files - echo. pseudoxml to make pseudoxml-XML files for display purposes - echo. linkcheck to check all external links for integrity - echo. doctest to run all doctests embedded in the documentation if enabled - echo. coverage to run coverage check of the documentation if enabled - goto end -) - -if "%1" == "clean" ( - for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i - del /q /s %BUILDDIR%\* - goto end -) - - -REM Check if sphinx-build is available and fallback to Python version if any -%SPHINXBUILD% 1>NUL 2>NUL -if errorlevel 9009 goto sphinx_python -goto sphinx_ok - -:sphinx_python - -set SPHINXBUILD=python -m sphinx.__init__ -%SPHINXBUILD% 2> nul +%SPHINXBUILD% >NUL 2>NUL if errorlevel 9009 ( echo. echo.The 'sphinx-build' command was not found. Make sure you have Sphinx @@ -69,195 +26,11 @@ if errorlevel 9009 ( exit /b 1 ) -:sphinx_ok +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% +goto end - -if "%1" == "html" ( - %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/html. - goto end -) - -if "%1" == "dirhtml" ( - %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. - goto end -) - -if "%1" == "singlehtml" ( - %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. - goto end -) - -if "%1" == "pickle" ( - %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can process the pickle files. - goto end -) - -if "%1" == "json" ( - %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can process the JSON files. - goto end -) - -if "%1" == "htmlhelp" ( - %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can run HTML Help Workshop with the ^ -.hhp project file in %BUILDDIR%/htmlhelp. - goto end -) - -if "%1" == "qthelp" ( - %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can run "qcollectiongenerator" with the ^ -.qhcp project file in %BUILDDIR%/qthelp, like this: - echo.^> qcollectiongenerator %BUILDDIR%\qthelp\cretonne.qhcp - echo.To view the help file: - echo.^> assistant -collectionFile %BUILDDIR%\qthelp\cretonne.ghc - goto end -) - -if "%1" == "devhelp" ( - %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. - goto end -) - -if "%1" == "epub" ( - %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The epub file is in %BUILDDIR%/epub. - goto end -) - -if "%1" == "latex" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "latexpdf" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - cd %BUILDDIR%/latex - make all-pdf - cd %~dp0 - echo. - echo.Build finished; the PDF files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "latexpdfja" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - cd %BUILDDIR%/latex - make all-pdf-ja - cd %~dp0 - echo. - echo.Build finished; the PDF files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "text" ( - %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The text files are in %BUILDDIR%/text. - goto end -) - -if "%1" == "man" ( - %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The manual pages are in %BUILDDIR%/man. - goto end -) - -if "%1" == "texinfo" ( - %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo. - goto end -) - -if "%1" == "gettext" ( - %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The message catalogs are in %BUILDDIR%/locale. - goto end -) - -if "%1" == "changes" ( - %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes - if errorlevel 1 exit /b 1 - echo. - echo.The overview file is in %BUILDDIR%/changes. - goto end -) - -if "%1" == "linkcheck" ( - %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck - if errorlevel 1 exit /b 1 - echo. - echo.Link check complete; look for any errors in the above output ^ -or in %BUILDDIR%/linkcheck/output.txt. - goto end -) - -if "%1" == "doctest" ( - %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest - if errorlevel 1 exit /b 1 - echo. - echo.Testing of doctests in the sources finished, look at the ^ -results in %BUILDDIR%/doctest/output.txt. - goto end -) - -if "%1" == "coverage" ( - %SPHINXBUILD% -b coverage %ALLSPHINXOPTS% %BUILDDIR%/coverage - if errorlevel 1 exit /b 1 - echo. - echo.Testing of coverage in the sources finished, look at the ^ -results in %BUILDDIR%/coverage/python.txt. - goto end -) - -if "%1" == "xml" ( - %SPHINXBUILD% -b xml %ALLSPHINXOPTS% %BUILDDIR%/xml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The XML files are in %BUILDDIR%/xml. - goto end -) - -if "%1" == "pseudoxml" ( - %SPHINXBUILD% -b pseudoxml %ALLSPHINXOPTS% %BUILDDIR%/pseudoxml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The pseudo-XML files are in %BUILDDIR%/pseudoxml. - goto end -) +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% :end +popd From c59e9180decea441e783979caaa4df84b35273f5 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sun, 4 Mar 2018 22:37:34 -0800 Subject: [PATCH 1575/3084] Tidy up whitespace. --- lib/cretonne/meta/base/instructions.py | 6 +++--- lib/cretonne/meta/cdsl/test_ti.py | 2 +- lib/cretonne/meta/cdsl/ti.py | 12 ++++++------ lib/cretonne/meta/cdsl/xform.py | 4 ++-- lib/cretonne/meta/gen_legalizer.py | 2 +- lib/cretonne/meta/isa/intel/instructions.py | 4 ++-- lib/cretonne/meta/semantics/__init__.py | 2 +- lib/cretonne/src/bforest/path.rs | 2 +- lib/cretonne/src/divconst_magic_numbers.rs | 10 +++++----- lib/cretonne/src/legalizer/boundary.rs | 2 +- lib/cretonne/src/preopt.rs | 18 +++++++++--------- lib/wasm/src/code_translator.rs | 2 +- 12 files changed, 33 insertions(+), 33 deletions(-) diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 7db8bed202..8ba1aa65fa 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -1699,7 +1699,7 @@ fpromote = Instruction( This is an exact operation. Cretonne currently only supports two floating point formats - - :type:`f32` and :type:`f64`. This may change in the future. + - :type:`f32` and :type:`f64`. This may change in the future. The result type must have the same number of vector lanes as the input, and the result lanes must not have fewer bits than the input lanes. If @@ -1715,10 +1715,10 @@ fdemote = Instruction( by rounding to nearest, ties to even. Cretonne currently only supports two floating point formats - - :type:`f32` and :type:`f64`. This may change in the future. + - :type:`f32` and :type:`f64`. This may change in the future. The result type must have the same number of vector lanes as the input, - and the result lanes must not have more bits than the input lanes. If + and the result lanes must not have more bits than the input lanes. If the input and output types are the same, this is a no-op. """, ins=x, outs=a, constraints=WiderOrEq(Float, FloatTo)) diff --git a/lib/cretonne/meta/cdsl/test_ti.py b/lib/cretonne/meta/cdsl/test_ti.py index b9b3e0b646..d3baa4d3c5 100644 --- a/lib/cretonne/meta/cdsl/test_ti.py +++ b/lib/cretonne/meta/cdsl/test_ti.py @@ -96,7 +96,7 @@ def check_concrete_typing_rtl(var_types, rtl): # type: (VarTyping, Rtl) -> None """ Check that a concrete type assignment var_types (Dict[Var, TypeVar]) is - valid for an Rtl rtl. Specifically check that: + valid for an Rtl rtl. Specifically check that: 1) For each Var v \in rtl, v is defined in var_types diff --git a/lib/cretonne/meta/cdsl/ti.py b/lib/cretonne/meta/cdsl/ti.py index b2673366be..bc2c16d5b1 100644 --- a/lib/cretonne/meta/cdsl/ti.py +++ b/lib/cretonne/meta/cdsl/ti.py @@ -322,7 +322,7 @@ class TypeEnv(object): # type: (TypeVar, TypeVar) -> None """ Record a that the free tv1 is part of the same equivalence class as - tv2. The canonical representative of the merged class is tv2's + tv2. The canonical representative of the merged class is tv2's cannonical representative. """ assert not tv1.is_derived @@ -364,9 +364,9 @@ class TypeEnv(object): # type: (TypeVar) -> int """ Get the rank of tv in the partial order. TVs directly associated with a - Var get their rank from the Var (see register()). Internally generated + Var get their rank from the Var (see register()). Internally generated non-derived TVs implicitly get the lowest rank (0). Derived variables - get their rank from their free typevar. Singletons have the highest + get their rank from their free typevar. Singletons have the highest rank. TVs associated with vars in a source pattern have a higher rank than TVs associted with temporary vars. """ @@ -381,7 +381,7 @@ class TypeEnv(object): def register(self, v): # type: (Var) -> None """ - Register a new Var v. This computes a rank for the associated TypeVar + Register a new Var v. This computes a rank for the associated TypeVar for v, which is used to impose a partial order on type variables. """ self.vars.add(v) @@ -848,7 +848,7 @@ def ti_def(definition, typ): def ti_rtl(rtl, typ): # type: (Rtl, TypeEnv) -> TypingOrError """ - Perform type inference on an Rtl in a starting type env typ. Return an + Perform type inference on an Rtl in a starting type env typ. Return an updated type environment or error. """ for (i, d) in enumerate(rtl.rtl): @@ -866,7 +866,7 @@ def ti_rtl(rtl, typ): def ti_xform(xform, typ): # type: (XForm, TypeEnv) -> TypingOrError """ - Perform type inference on an Rtl in a starting type env typ. Return an + Perform type inference on an Rtl in a starting type env typ. Return an updated type environment or error. """ typ_or_err = ti_rtl(xform.src, typ) diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/cretonne/meta/cdsl/xform.py index bad15e0245..607c1776a2 100644 --- a/lib/cretonne/meta/cdsl/xform.py +++ b/lib/cretonne/meta/cdsl/xform.py @@ -113,8 +113,8 @@ class Rtl(object): # type: (Rtl) -> None """ Given that there is only 1 possible concrete typing T for self, assign - a singleton TV with type t=T[v] for each Var v \in self. Its an error - to call this on an Rtl with more than 1 possible typing. This modifies + a singleton TV with type t=T[v] for each Var v \in self. Its an error + to call this on an Rtl with more than 1 possible typing. This modifies the Rtl in-place. """ from .ti import ti_rtl, TypeEnv diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 51368fea44..0f7b1f68e4 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -211,7 +211,7 @@ def unwrap_inst(iref, node, fmt): fmt.format('let typeof_{0} = pos.func.dfg.value_type({0});', v) # If the node has results, detach the values. - # Place the values in locals. + # Place the values in locals. replace_inst = False if len(node.defs) > 0: if node.defs == node.defs[0].dst_def.defs: diff --git a/lib/cretonne/meta/isa/intel/instructions.py b/lib/cretonne/meta/isa/intel/instructions.py index e5265e2d15..1fe75ff984 100644 --- a/lib/cretonne/meta/isa/intel/instructions.py +++ b/lib/cretonne/meta/isa/intel/instructions.py @@ -154,7 +154,7 @@ rflags = Operand('rflags', iflags) bsr = Instruction( 'x86_bsr', r""" Bit Scan Reverse -- returns the bit-index of the most significant 1 - in the word. Result is undefined if the argument is zero. However, it + in the word. Result is undefined if the argument is zero. However, it sets the Z flag depending on the argument, so it is at least easy to detect and handle that case. @@ -166,7 +166,7 @@ bsr = Instruction( bsf = Instruction( 'x86_bsf', r""" Bit Scan Forwards -- returns the bit-index of the least significant 1 - in the word. Is otherwise identical to 'bsr', just above. + in the word. Is otherwise identical to 'bsr', just above. """, ins=x, outs=(y, rflags)) diff --git a/lib/cretonne/meta/semantics/__init__.py b/lib/cretonne/meta/semantics/__init__.py index 94e32b652c..8a55b1d595 100644 --- a/lib/cretonne/meta/semantics/__init__.py +++ b/lib/cretonne/meta/semantics/__init__.py @@ -17,7 +17,7 @@ def verify_semantics(inst, src, xforms): # type: (Instruction, Rtl, InstructionSemantics) -> None """ Verify that the semantics transforms in xforms correctly describe the - instruction described by the src Rtl. This involves checking that: + instruction described by the src Rtl. This involves checking that: 0) src is a single instance of inst 1) For all x\in xforms x.src is a single instance of inst 2) For any concrete values V of Literals in inst: diff --git a/lib/cretonne/src/bforest/path.rs b/lib/cretonne/src/bforest/path.rs index 3393014fee..c0f9550645 100644 --- a/lib/cretonne/src/bforest/path.rs +++ b/lib/cretonne/src/bforest/path.rs @@ -285,7 +285,7 @@ impl Path { fn split_and_insert(&mut self, mut key: F::Key, value: F::Value, pool: &mut NodePool) { let orig_root = self.node[0]; - // Loop invariant: We need to split the node at `level` and then retry a failed insertion. + // Loop invariant: We need to split the node at `level` and then retry a failed insertion. // The items to insert are either `(key, ins_node)` or `(key, value)`. let mut ins_node = None; let mut split; diff --git a/lib/cretonne/src/divconst_magic_numbers.rs b/lib/cretonne/src/divconst_magic_numbers.rs index a34608359f..968ca67f5c 100644 --- a/lib/cretonne/src/divconst_magic_numbers.rs +++ b/lib/cretonne/src/divconst_magic_numbers.rs @@ -4,11 +4,11 @@ //---------------------------------------------------------------------- // -// Math helpers for division by (non-power-of-2) constants. This is based -// on the presentation in "Hacker's Delight" by Henry Warren, 2003. There -// are four cases: {unsigned, signed} x {32 bit, 64 bit}. The word size +// Math helpers for division by (non-power-of-2) constants. This is based +// on the presentation in "Hacker's Delight" by Henry Warren, 2003. There +// are four cases: {unsigned, signed} x {32 bit, 64 bit}. The word size // makes little difference, but the signed-vs-unsigned aspect has a large -// effect. Therefore everything is presented in the order U32 U64 S32 S64 +// effect. Therefore everything is presented in the order U32 U64 S32 S64 // so as to emphasise the similarity of the U32 and U64 cases and the S32 // and S64 cases. @@ -478,7 +478,7 @@ mod tests { fn test_magic_generators_dont_panic() { // The point of this is to check that the magic number generators // don't panic with integer wraparounds, especially at boundary - // cases for their arguments. The actual results are thrown away. + // cases for their arguments. The actual results are thrown away. let mut total: u64 = 0; println!("Testing UP magicU32"); for x in 2..(200 * 1000u32) { diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index ea81c1ff95..9ea50b202d 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -180,7 +180,7 @@ where // We theoretically allow for call instructions that return a number of fixed results before // the call return values. In practice, it doesn't happen. let fixed_results = pos.func.dfg[call].opcode().constraints().fixed_results(); - assert_eq!(fixed_results, 0, "Fixed results on calls not supported"); + assert_eq!(fixed_results, 0, "Fixed results on calls not supported"); let results = pos.func.dfg.detach_results(call); let mut next_res = 0; diff --git a/lib/cretonne/src/preopt.rs b/lib/cretonne/src/preopt.rs index 0e4582d7a7..1fdee0ad80 100644 --- a/lib/cretonne/src/preopt.rs +++ b/lib/cretonne/src/preopt.rs @@ -20,7 +20,7 @@ use timing; // Simple math helpers // if `x` is a power of two, or the negation thereof, return the power along -// with a boolean that indicates whether `x` is negative. Else return None. +// with a boolean that indicates whether `x` is negative. Else return None. #[inline] fn isPowerOf2_S32(x: i32) -> Option<(bool, u32)> { // We have to special-case this because abs(x) isn't representable. @@ -128,10 +128,10 @@ fn get_div_info(inst: Inst, dfg: &DataFlowGraph) -> Option { } // TODO: should we actually bother to do this (that is, manually match - // the case that the second argument is an iconst)? Or should we assume + // the case that the second argument is an iconst)? Or should we assume // that some previous constant propagation pass has pushed all such // immediates to their use points, creating BinaryImm instructions - // instead? For now we take the conservative approach. + // instead? For now we take the conservative approach. if let &InstructionData::Binary { opcode, args } = idata { let (isSigned, isRem) = match opcode { Opcode::Udiv => (false, false), @@ -153,10 +153,10 @@ fn get_div_info(inst: Inst, dfg: &DataFlowGraph) -> Option { } // Actually do the transformation given a bundle containing the relevant -// information. `divrem_info` describes a div or rem by a constant, that +// information. `divrem_info` describes a div or rem by a constant, that // `pos` currently points at, and `inst` is the associated instruction. // `inst` is replaced by a sequence of other operations that calculate the -// same result. Note that there are various `divrem_info` cases where we +// same result. Note that there are various `divrem_info` cases where we // cannot do any transformation, in which case `inst` is left unchanged. fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCursor, inst: Inst) { let isRem = match *divrem_info { @@ -234,7 +234,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso qf = q1; } } - // Now qf holds the final quotient. If necessary calculate the + // Now qf holds the final quotient. If necessary calculate the // remainder instead. if isRem { let tt = pos.ins().imul_imm(qf, d as i64); @@ -306,7 +306,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso qf = q1; } } - // Now qf holds the final quotient. If necessary calculate the + // Now qf holds the final quotient. If necessary calculate the // remainder instead. if isRem { let tt = pos.ins().imul_imm(qf, d as i64); @@ -382,7 +382,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso }; let t1 = pos.ins().ushr_imm(q3, 31); let qf = pos.ins().iadd(q3, t1); - // Now qf holds the final quotient. If necessary calculate + // Now qf holds the final quotient. If necessary calculate // the remainder instead. if isRem { let tt = pos.ins().imul_imm(qf, d as i64); @@ -459,7 +459,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso }; let t1 = pos.ins().ushr_imm(q3, 63); let qf = pos.ins().iadd(q3, t1); - // Now qf holds the final quotient. If necessary calculate + // Now qf holds the final quotient. If necessary calculate // the remainder instead. if isRem { let tt = pos.ins().imul_imm(qf, d); diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 765b1a4577..5d7e59f829 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -90,7 +90,7 @@ pub fn translate_operator( } } /********************************* Stack misc *************************************** - * `drop`, `nop`, `unreachable` and `select`. + * `drop`, `nop`, `unreachable` and `select`. ***********************************************************************************/ Operator::Drop => { state.pop1(); From be84db8e86d44c3d07f80febea32e319e0b3933c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sun, 4 Mar 2018 22:40:36 -0800 Subject: [PATCH 1576/3084] Fix glossary references. --- cranelift/docs/langref.rst | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index b91f1b3f47..3219f14b08 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -1158,14 +1158,19 @@ Glossary intermediate representation. Cretonne's IR can be converted to text losslessly. + stack slot + A fixed size memory allocation in the current function's activation + frame. These include :term:`explicit stack slot`\s and + :term:`spill stack slot`\s. + explicit stack slot A fixed size memory allocation in the current function's activation - frame. These differ from :term:`spill stack slots` in that they can + frame. These differ from :term:`spill stack slot`\s in that they can be created by frontends and they may have their addresses taken. spill stack slot A fixed size memory allocation in the current function's activation - frame. These differ from :term:`explicit stack slots` in that they are + frame. These differ from :term:`explicit stack slot`\s in that they are only created during register allocation, and they may not have their address taken. From 6e94e70f308e97c1a96eb07ca3d674426e928bc2 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 5 Mar 2018 05:37:19 -0800 Subject: [PATCH 1577/3084] Use an https URL rather than http. Found by sphinx's linkcheck. --- lib/cretonne/meta/isa/riscv/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/meta/isa/riscv/__init__.py b/lib/cretonne/meta/isa/riscv/__init__.py index cf61cbdcf5..f40086414d 100644 --- a/lib/cretonne/meta/isa/riscv/__init__.py +++ b/lib/cretonne/meta/isa/riscv/__init__.py @@ -2,7 +2,7 @@ RISC-V Target ------------- -`RISC-V `_ is an open instruction set architecture +`RISC-V `_ is an open instruction set architecture originally developed at UC Berkeley. It is a RISC-style ISA with either a 32-bit (RV32I) or 64-bit (RV32I) base instruction set and a number of optional extensions: From bf480c341b4624ebd0114a0b9fd732a4c409a9ad Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 5 Mar 2018 05:47:56 -0800 Subject: [PATCH 1578/3084] Use https rather than http for several URLs. --- README.rst | 4 ++-- cranelift/docs/compare-llvm.rst | 14 +++++++------- cranelift/docs/langref.rst | 2 +- cranelift/docs/make.bat | 2 +- lib/cretonne/src/regalloc/coalescing.rs | 2 +- lib/cretonne/src/result.rs | 2 +- lib/filecheck/src/lib.rs | 2 +- lib/wasm/src/func_translator.rs | 2 +- 8 files changed, 15 insertions(+), 15 deletions(-) diff --git a/README.rst b/README.rst index 05ddeb6b70..6e6ab59bb3 100644 --- a/README.rst +++ b/README.rst @@ -3,7 +3,7 @@ Cretonne Code Generator ======================= Cretonne is a low-level retargetable code generator. It translates a `target-independent -intermediate language `_ into executable +intermediate language `_ into executable machine code. *This is a work in progress that is not yet functional.* @@ -61,7 +61,7 @@ Building the documentation -------------------------- To build the Cretonne documentation, you need the `Sphinx documentation -generator `_:: +generator `_:: $ pip install sphinx sphinx-autobuild sphinx_rtd_theme $ cd cretonne/docs diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst index 03b82dc7f7..0180f6cc92 100644 --- a/cranelift/docs/compare-llvm.rst +++ b/cranelift/docs/compare-llvm.rst @@ -2,9 +2,9 @@ Cretonne compared to LLVM ************************* -`LLVM `_ is a collection of compiler components implemented as +`LLVM `_ is a collection of compiler components implemented as a set of C++ libraries. It can be used to build both JIT compilers and static -compilers like `Clang `_, and it is deservedly very +compilers like `Clang `_, and it is deservedly very popular. `Chris Lattner's chapter about LLVM `_ in the `Architecture of Open Source Applications `_ book gives an excellent @@ -40,7 +40,7 @@ Intermediate representations LLVM uses multiple intermediate representations as it translates a program to binary machine code: -`LLVM IR `_ +`LLVM IR `_ This is the primary intermediate language which has textual, binary, and in-memory representations. It serves two main purposes: @@ -49,7 +49,7 @@ binary machine code: - Intermediate representation for common mid-level optimizations. A large library of code analysis and transformation passes operate on LLVM IR. -`SelectionDAG `_ +`SelectionDAG `_ A graph-based representation of the code in a single basic block is used by the instruction selector. It has both ISA-agnostic and ISA-specific opcodes. These main passes are run on the SelectionDAG representation: @@ -65,7 +65,7 @@ binary machine code: The SelectionDAG representation automatically eliminates common subexpressions and dead code. -`MachineInstr `_ +`MachineInstr `_ A linear representation of ISA-specific instructions that initially is in SSA form, but it can also represent non-SSA form during and after register allocation. Many low-level optimizations run on MI code. The most important @@ -74,7 +74,7 @@ binary machine code: - Scheduling. - Register allocation. -`MC `_ +`MC `_ MC serves as the output abstraction layer and is the basis for LLVM's integrated assembler. It is used for: @@ -126,7 +126,7 @@ condition is false. The Cretonne representation is closer to how machine code works; LLVM's representation is more abstract. LLVM uses `phi instructions -`_ in its SSA +`_ in its SSA representation. Cretonne passes arguments to EBBs instead. The two representations are equivalent, but the EBB arguments are better suited to handle EBBs that may contain multiple branches to the same destination block diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 3219f14b08..3a855c32b4 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -782,7 +782,7 @@ Integer operations For example, see `llvm.sadd.with.overflow.*` and `llvm.ssub.with.overflow.*` in - `LLVM `_. + `LLVM `_. .. autoinst:: imul .. autoinst:: imul_imm diff --git a/cranelift/docs/make.bat b/cranelift/docs/make.bat index 9267ac038e..2958a2ba6e 100644 --- a/cranelift/docs/make.bat +++ b/cranelift/docs/make.bat @@ -22,7 +22,7 @@ if errorlevel 9009 ( echo.may add the Sphinx directory to PATH. echo. echo.If you don't have Sphinx installed, grab it from - echo.http://sphinx-doc.org/ + echo.https://sphinx-doc.org/ exit /b 1 ) diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 416783b6c0..1dc06c6c2b 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -26,7 +26,7 @@ use timing; // The coalescing algorithm implemented follows this paper fairly closely: // // Budimlic, Z., Cooper, K. D., Harvey, T. J., et al. (2002). Fast copy coalescing and -// live-range identification (Vol. 37, pp. 25–32). ACM. http://doi.org/10.1145/543552.512534 +// live-range identification (Vol. 37, pp. 25–32). ACM. https://doi.org/10.1145/543552.512534 // // We use a more efficient dominator forest representation (a linear stack) described here: // diff --git a/lib/cretonne/src/result.rs b/lib/cretonne/src/result.rs index 387eaa9a2a..aa209a0845 100644 --- a/lib/cretonne/src/result.rs +++ b/lib/cretonne/src/result.rs @@ -26,7 +26,7 @@ pub enum CtonError { /// Cretonne can compile very large and complicated functions, but the [implementation has /// limits][limits] that cause compilation to fail when they are exceeded. /// - /// [limits]: http://cretonne.readthedocs.io/en/latest/langref.html#implementation-limits + /// [limits]: https://cretonne.readthedocs.io/en/latest/langref.html#implementation-limits ImplLimitExceeded, /// The code size for the function is too large. diff --git a/lib/filecheck/src/lib.rs b/lib/filecheck/src/lib.rs index 4c8a242897..4dec7a9b65 100644 --- a/lib/filecheck/src/lib.rs +++ b/lib/filecheck/src/lib.rs @@ -1,5 +1,5 @@ //! This crate provides a text pattern matching library with functionality similar to the LLVM -//! project's [FileCheck command](http://llvm.org/docs/CommandGuide/FileCheck.html). +//! project's [FileCheck command](https://llvm.org/docs/CommandGuide/FileCheck.html). //! //! A list of directives is typically extracted from a file containing a test case. The test case //! is then run through the program under test, and its output matched against the directives. diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index a9389342e9..7e8955e816 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -45,7 +45,7 @@ impl FuncTranslator { /// /// See [the WebAssembly specification][wasm]. /// - /// [wasm]: http://webassembly.github.io/spec/binary/modules.html#code-section + /// [wasm]: https://webassembly.github.io/spec/binary/modules.html#code-section /// /// The Cretonne IR function `func` should be completely empty except for the `func.signature` /// and `func.name` fields. The signature may contain special-purpose arguments which are not From d119524c90b98d9e293d0f850a191273b1151ab0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 5 Mar 2018 07:11:23 -0800 Subject: [PATCH 1579/3084] Bump version to 0.3.1 --- cranelift/Cargo.toml | 12 ++++++------ cranelift/publish-all.sh | 2 +- lib/cretonne/Cargo.toml | 2 +- lib/frontend/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/wasm/Cargo.toml | 6 +++--- 7 files changed, 17 insertions(+), 17 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 37b4456e82..c948ea54ff 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.3.0" +version = "0.3.1" description = "Binaries for testing the Cretonne library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,11 +13,11 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne = { path = "lib/cretonne", version = "0.3.0" } -cretonne-reader = { path = "lib/reader", version = "0.3.0" } -cretonne-frontend = { path = "lib/frontend", version = "0.3.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.3.0" } -cretonne-native = { path = "lib/native", version = "0.3.0" } +cretonne = { path = "lib/cretonne", version = "0.3.1" } +cretonne-reader = { path = "lib/reader", version = "0.3.1" } +cretonne-frontend = { path = "lib/frontend", version = "0.3.1" } +cretonne-wasm = { path = "lib/wasm", version = "0.3.1" } +cretonne-native = { path = "lib/native", version = "0.3.1" } filecheck = { path = "lib/filecheck" } docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index e8b22b4f5d..746d85a1fc 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -5,7 +5,7 @@ topdir="$(pwd)" # All the cretonne-* crates have the same version number # The filecheck crate version is managed independently. -version="0.3.0" +version="0.3.1" # Update all of the Cargo.toml files. # diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 12ddbf2abe..d830e1ece5 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.3.0" +version = "0.3.1" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 67297562b8..ace336de71 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.3.0" +version = "0.3.1" description = "Cretonne IL builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,4 +12,4 @@ readme = "README.md" name = "cton_frontend" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.0" } +cretonne = { path = "../cretonne", version = "0.3.1" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 1fe35a7190..3158a88a4e 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.3.0" +version = "0.3.1" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -11,7 +11,7 @@ readme = "README.md" name = "cton_native" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.0" } +cretonne = { path = "../cretonne", version = "0.3.1" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 14b721af35..250f296066 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.3.0" +version = "0.3.1" description = "Cretonne textual IL reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,4 +12,4 @@ readme = "README.md" name = "cton_reader" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.0" } +cretonne = { path = "../cretonne", version = "0.3.1" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 45e153357c..7acb1660ce 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.3.0" +version = "0.3.1" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IL" repository = "https://github.com/Cretonne/cretonne" @@ -13,8 +13,8 @@ name = "cton_wasm" [dependencies] wasmparser = "0.14.1" -cretonne = { path = "../cretonne", version = "0.3.0" } -cretonne-frontend = { path = "../frontend", version = "0.3.0" } +cretonne = { path = "../cretonne", version = "0.3.1" } +cretonne-frontend = { path = "../frontend", version = "0.3.1" } [dev-dependencies] tempdir = "0.3.5" From 1a4723831b6f5bc8e8a5dee33b530caa9b34a7c3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 5 Mar 2018 14:02:45 -0800 Subject: [PATCH 1580/3084] Add an encoding step to "cton-util compile". --- cranelift/src/compile.rs | 58 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-) diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index fd2e5d5552..9a9644b1cf 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -6,9 +6,45 @@ use cton_reader::parse_test; use std::path::PathBuf; use cretonne::Context; use cretonne::settings::FlagsOrIsa; +use cretonne::{binemit, ir}; use std::path::Path; use utils::{pretty_error, read_to_string, parse_sets_and_isa}; +struct PrintRelocs { + flag_print: bool, +} + +impl binemit::RelocSink for PrintRelocs { + fn reloc_ebb( + &mut self, + where_: binemit::CodeOffset, + r: binemit::Reloc, + offset: binemit::CodeOffset, + ) { + if self.flag_print { + println!("reloc_ebb: {} {} at {}", r, offset, where_); + } + } + + fn reloc_external( + &mut self, + where_: binemit::CodeOffset, + r: binemit::Reloc, + name: &ir::ExternalName, + addend: binemit::Addend, + ) { + if self.flag_print { + println!("reloc_ebb: {} {} {} at {}", r, name, addend, where_); + } + } + + fn reloc_jt(&mut self, where_: binemit::CodeOffset, r: binemit::Reloc, jt: ir::JumpTable) { + if self.flag_print { + println!("reloc_ebb: {} {} at {}", r, jt, where_); + } + } +} + pub fn run( files: Vec, flag_print: bool, @@ -49,12 +85,32 @@ fn handle_module( for (func, _) in test_file.functions { let mut context = Context::new(); context.func = func; - context.compile(isa).map_err(|err| { + let size = context.compile(isa).map_err(|err| { pretty_error(&context.func, Some(isa), err) })?; if flag_print { println!("{}", context.func.display(isa)); } + + // Encode the result as machine code. + let mut mem = Vec::new(); + let mut relocs = PrintRelocs { flag_print }; + mem.resize(size as usize, 0); + context.emit_to_memory(mem.as_mut_ptr(), &mut relocs, &*isa); + + if flag_print { + print!(".byte "); + let mut first = true; + for byte in &mem { + if first { + first = false; + } else { + print!(", "); + } + print!("{}", byte); + } + println!(); + } } Ok(()) From ae0801e23b4a17be08fd24f0b41da836f43abf23 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 5 Mar 2018 14:05:13 -0800 Subject: [PATCH 1581/3084] Disable creation of .pyc files. Cretonne's python scripts aren't run very often, so there's little benefit in creating .pyc files. And the .pyc files cause trouble for some vendoring scripts. So disable them. --- cranelift/test-all.sh | 4 ++++ lib/cretonne/build.rs | 3 +++ 2 files changed, 7 insertions(+) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 6a9291d82e..2402eb7a39 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -11,6 +11,10 @@ set -euo pipefail # # All tests run by this script should be passing at all times. +# Disable generation of .pyc files because they cause trouble for vendoring +# scripts, and this is a build step that isn't run very often anyway. +export PYTHONDONTWRITEBYTECODE=1 + # Repository top-level directory. cd $(dirname "$0") topdir=$(pwd) diff --git a/lib/cretonne/build.rs b/lib/cretonne/build.rs index d2ef3905b1..69be14de94 100644 --- a/lib/cretonne/build.rs +++ b/lib/cretonne/build.rs @@ -59,8 +59,11 @@ fn main() { let build_script = meta_dir.join("build.py"); // Launch build script with Python. We'll just find python in the path. + // Use -B to disable .pyc files, because they cause trouble for vendoring + // scripts, and this is a build step that isn't run very often anyway. let status = process::Command::new("python") .current_dir(crate_dir) + .arg("-B") .arg(build_script) .arg("--out-dir") .arg(out_dir) From b1697dd1dc9e7479ee331603d76d04f4f93ea660 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 5 Mar 2018 14:34:35 -0800 Subject: [PATCH 1582/3084] Style: Don't use `else` after a `return`. --- cranelift/src/filetest/binemit.rs | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/binemit.rs index d2d179fea8..3c52a63b2b 100644 --- a/cranelift/src/filetest/binemit.rs +++ b/cranelift/src/filetest/binemit.rs @@ -271,13 +271,12 @@ impl SubTest for TestBinEmit { "No encodings found for: {}", func.dfg.display_inst(inst, isa) )); - } else { - return Err(format!( + } + return Err(format!( "No matching encodings for {} in {}", func.dfg.display_inst(inst, isa), DisplayList(&encodings), )); - } } let have = sink.text.trim(); if have != want { From 136d6f5c4b0ee9727d5db24c545547d6099134bd Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 5 Mar 2018 15:13:59 -0800 Subject: [PATCH 1583/3084] Implement ireduce, sextend, and uextend between i8/i16 and i32/i64. --- cranelift/filetests/isa/intel/binary32.cton | 32 ++++++ cranelift/filetests/isa/intel/binary64.cton | 112 ++++++++++++++++++++ lib/cretonne/meta/isa/intel/encodings.py | 48 ++++++++- 3 files changed, 191 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 7d3e3a683b..df98a86515 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -552,3 +552,35 @@ ebb1: return } + +; Tests for i32/i8 conversion instructions. +function %I32_I8() { +ebb0: + [-,%rcx] v1 = iconst.i32 1 + + [-,%rcx] v11 = ireduce.i8 v1 ; bin: + + ; asm: movsbl %cl, %esi + [-,%rsi] v20 = sextend.i32 v11 ; bin: 0f be f1 + + ; asm: movzbl %cl, %esi + [-,%rsi] v30 = uextend.i32 v11 ; bin: 0f b6 f1 + + trap user0 ; bin: 0f 0b +} + +; Tests for i32/i16 conversion instructions. +function %I32_I16() { +ebb0: + [-,%rcx] v1 = iconst.i32 1 + + [-,%rcx] v11 = ireduce.i16 v1 ; bin: + + ; asm: movswl %cx, %esi + [-,%rsi] v20 = sextend.i32 v11 ; bin: 0f bf f1 + + ; asm: movzwl %cx, %esi + [-,%rsi] v30 = uextend.i32 v11 ; bin: 0f b7 f1 + + trap user0 ; bin: 0f 0b +} diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index b579b4d58d..9eca950ee6 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -1084,6 +1084,118 @@ ebb2: jump ebb1 ; bin: eb fd } +; Tests for i32/i8 conversion instructions. +function %I32_I8() { +ebb0: + [-,%rcx] v1 = iconst.i32 1 + [-,%rsi] v2 = iconst.i32 2 + [-,%r10] v3 = iconst.i32 3 + + [-,%rcx] v11 = ireduce.i8 v1 ; bin: + [-,%rsi] v12 = ireduce.i8 v2 ; bin: + [-,%r10] v13 = ireduce.i8 v3 ; bin: + + ; asm: movsbl %cl, %esi + [-,%rsi] v20 = sextend.i32 v11 ; bin: 0f be f1 + ; asm: movsbl %sil, %r10d + [-,%r10] v21 = sextend.i32 v12 ; bin: 44 0f be d6 + ; asm: movsbl %r10b, %ecx + [-,%rcx] v22 = sextend.i32 v13 ; bin: 41 0f be ca + + ; asm: movzbl %cl, %esi + [-,%rsi] v30 = uextend.i32 v11 ; bin: 0f b6 f1 + ; asm: movzbl %sil, %r10d + [-,%r10] v31 = uextend.i32 v12 ; bin: 44 0f b6 d6 + ; asm: movzbl %r10b, %ecx + [-,%rcx] v32 = uextend.i32 v13 ; bin: 41 0f b6 ca + + trap user0 ; bin: 0f 0b +} + +; Tests for i32/i16 conversion instructions. +function %I32_I16() { +ebb0: + [-,%rcx] v1 = iconst.i32 1 + [-,%rsi] v2 = iconst.i32 2 + [-,%r10] v3 = iconst.i32 3 + + [-,%rcx] v11 = ireduce.i16 v1 ; bin: + [-,%rsi] v12 = ireduce.i16 v2 ; bin: + [-,%r10] v13 = ireduce.i16 v3 ; bin: + + ; asm: movswl %cx, %esi + [-,%rsi] v20 = sextend.i32 v11 ; bin: 0f bf f1 + ; asm: movswl %si, %r10d + [-,%r10] v21 = sextend.i32 v12 ; bin: 44 0f bf d6 + ; asm: movswl %r10w, %ecx + [-,%rcx] v22 = sextend.i32 v13 ; bin: 41 0f bf ca + + ; asm: movzwl %cx, %esi + [-,%rsi] v30 = uextend.i32 v11 ; bin: 0f b7 f1 + ; asm: movzwl %si, %r10d + [-,%r10] v31 = uextend.i32 v12 ; bin: 44 0f b7 d6 + ; asm: movzwl %r10w, %ecx + [-,%rcx] v32 = uextend.i32 v13 ; bin: 41 0f b7 ca + + trap user0 ; bin: 0f 0b +} + +; Tests for i64/i8 conversion instructions. +function %I64_I8() { +ebb0: + [-,%rcx] v1 = iconst.i64 1 + [-,%rsi] v2 = iconst.i64 2 + [-,%r10] v3 = iconst.i64 3 + + [-,%rcx] v11 = ireduce.i8 v1 ; bin: + [-,%rsi] v12 = ireduce.i8 v2 ; bin: + [-,%r10] v13 = ireduce.i8 v3 ; bin: + + ; asm: movsbq %cl, %rsi + [-,%rsi] v20 = sextend.i64 v11 ; bin: 48 0f be f1 + ; asm: movsbq %sil, %r10 + [-,%r10] v21 = sextend.i64 v12 ; bin: 4c 0f be d6 + ; asm: movsbq %r10b, %rcx + [-,%rcx] v22 = sextend.i64 v13 ; bin: 49 0f be ca + + ; asm: movzbl %cl, %esi + [-,%rsi] v30 = uextend.i64 v11 ; bin: 0f b6 f1 + ; asm: movzbl %sil, %r10d + [-,%r10] v31 = uextend.i64 v12 ; bin: 44 0f b6 d6 + ; asm: movzbl %r10b, %ecx + [-,%rcx] v32 = uextend.i64 v13 ; bin: 41 0f b6 ca + + trap user0 ; bin: 0f 0b +} + +; Tests for i64/i16 conversion instructions. +function %I64_I16() { +ebb0: + [-,%rcx] v1 = iconst.i64 1 + [-,%rsi] v2 = iconst.i64 2 + [-,%r10] v3 = iconst.i64 3 + + [-,%rcx] v11 = ireduce.i16 v1 ; bin: + [-,%rsi] v12 = ireduce.i16 v2 ; bin: + [-,%r10] v13 = ireduce.i16 v3 ; bin: + + ; asm: movswq %cx, %rsi + [-,%rsi] v20 = sextend.i64 v11 ; bin: 48 0f bf f1 + ; asm: movswq %si, %r10 + [-,%r10] v21 = sextend.i64 v12 ; bin: 4c 0f bf d6 + ; asm: movswq %r10w, %rcx + [-,%rcx] v22 = sextend.i64 v13 ; bin: 49 0f bf ca + + ; asm: movzwl %cx, %esi + [-,%rsi] v30 = uextend.i64 v11 ; bin: 0f b7 f1 + ; asm: movzwl %si, %r10d + [-,%r10] v31 = uextend.i64 v12 ; bin: 44 0f b7 d6 + ; asm: movzwl %r10w, %ecx + [-,%rcx] v32 = uextend.i64 v13 ; bin: 41 0f b7 ca + + trap user0 ; bin: 0f 0b +} + ; Tests for i64/i32 conversion instructions. function %I64_I32() { ebb0: diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index f9b77a04bd..162caccf65 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -406,9 +406,55 @@ I64.enc(base.bint.i32.b1, *r.urm_abcd(0x0f, 0xb6)) # Numerical conversions. -# Converting i64 to i32 is a no-op in 64-bit mode. +# Reducing an integer is a no-op. +I32.enc(base.ireduce.i8.i32, r.null, 0) +I32.enc(base.ireduce.i16.i32, r.null, 0) +I64.enc(base.ireduce.i8.i32, r.null, 0) +I64.enc(base.ireduce.i16.i32, r.null, 0) +I64.enc(base.ireduce.i8.i64, r.null, 0) +I64.enc(base.ireduce.i16.i64, r.null, 0) I64.enc(base.ireduce.i32.i64, r.null, 0) + +# TODO: Add encodings for cbw, cwde, cdqe, which are sign-extending +# instructions for %al/%ax/%eax to %ax/%eax/%rax. + +# movsbl +I32.enc(base.sextend.i32.i8, *r.urm(0x0f, 0xbe)) +I64.enc(base.sextend.i32.i8, *r.urm.rex(0x0f, 0xbe)) +I64.enc(base.sextend.i32.i8, *r.urm(0x0f, 0xbe)) + +# movswl +I32.enc(base.sextend.i32.i16, *r.urm(0x0f, 0xbf)) +I64.enc(base.sextend.i32.i16, *r.urm.rex(0x0f, 0xbf)) +I64.enc(base.sextend.i32.i16, *r.urm(0x0f, 0xbf)) + +# movsbq +I64.enc(base.sextend.i64.i8, *r.urm.rex(0x0f, 0xbe, w=1)) + +# movswq +I64.enc(base.sextend.i64.i16, *r.urm.rex(0x0f, 0xbf, w=1)) + +# movslq I64.enc(base.sextend.i64.i32, *r.urm.rex(0x63, w=1)) + +# movzbl +I32.enc(base.uextend.i32.i8, *r.urm(0x0f, 0xb6)) +I64.enc(base.uextend.i32.i8, *r.urm.rex(0x0f, 0xb6)) +I64.enc(base.uextend.i32.i8, *r.urm(0x0f, 0xb6)) + +# movzwl +I32.enc(base.uextend.i32.i16, *r.urm(0x0f, 0xb7)) +I64.enc(base.uextend.i32.i16, *r.urm.rex(0x0f, 0xb7)) +I64.enc(base.uextend.i32.i16, *r.urm(0x0f, 0xb7)) + +# movzbq, encoded as movzbl because it's equivalent and shorter +I64.enc(base.uextend.i64.i8, *r.urm.rex(0x0f, 0xb6)) +I64.enc(base.uextend.i64.i8, *r.urm(0x0f, 0xb6)) + +# movzwq, encoded as movzwl because it's equivalent and shorter +I64.enc(base.uextend.i64.i16, *r.urm.rex(0x0f, 0xb7)) +I64.enc(base.uextend.i64.i16, *r.urm(0x0f, 0xb7)) + # A 32-bit register copy clears the high 32 bits. I64.enc(base.uextend.i64.i32, *r.umr.rex(0x89)) I64.enc(base.uextend.i64.i32, *r.umr(0x89)) From ee0bce4106711b686be12bf813e428ac8b58e0d7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 5 Mar 2018 16:17:00 -0800 Subject: [PATCH 1584/3084] Bump version to 0.3.2 --- cranelift/Cargo.toml | 12 ++++++------ cranelift/publish-all.sh | 2 +- lib/cretonne/Cargo.toml | 2 +- lib/frontend/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/wasm/Cargo.toml | 6 +++--- 7 files changed, 17 insertions(+), 17 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index c948ea54ff..a1479c63fc 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.3.1" +version = "0.3.2" description = "Binaries for testing the Cretonne library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,11 +13,11 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne = { path = "lib/cretonne", version = "0.3.1" } -cretonne-reader = { path = "lib/reader", version = "0.3.1" } -cretonne-frontend = { path = "lib/frontend", version = "0.3.1" } -cretonne-wasm = { path = "lib/wasm", version = "0.3.1" } -cretonne-native = { path = "lib/native", version = "0.3.1" } +cretonne = { path = "lib/cretonne", version = "0.3.2" } +cretonne-reader = { path = "lib/reader", version = "0.3.2" } +cretonne-frontend = { path = "lib/frontend", version = "0.3.2" } +cretonne-wasm = { path = "lib/wasm", version = "0.3.2" } +cretonne-native = { path = "lib/native", version = "0.3.2" } filecheck = { path = "lib/filecheck" } docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 746d85a1fc..b380211212 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -5,7 +5,7 @@ topdir="$(pwd)" # All the cretonne-* crates have the same version number # The filecheck crate version is managed independently. -version="0.3.1" +version="0.3.2" # Update all of the Cargo.toml files. # diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index d830e1ece5..d6e91e695d 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.3.1" +version = "0.3.2" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index ace336de71..a09c692126 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.3.1" +version = "0.3.2" description = "Cretonne IL builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,4 +12,4 @@ readme = "README.md" name = "cton_frontend" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.1" } +cretonne = { path = "../cretonne", version = "0.3.2" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 3158a88a4e..5d2920a67a 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.3.1" +version = "0.3.2" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -11,7 +11,7 @@ readme = "README.md" name = "cton_native" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.1" } +cretonne = { path = "../cretonne", version = "0.3.2" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 250f296066..b0d32b2a61 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.3.1" +version = "0.3.2" description = "Cretonne textual IL reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,4 +12,4 @@ readme = "README.md" name = "cton_reader" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.1" } +cretonne = { path = "../cretonne", version = "0.3.2" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 7acb1660ce..086e8c9735 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.3.1" +version = "0.3.2" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IL" repository = "https://github.com/Cretonne/cretonne" @@ -13,8 +13,8 @@ name = "cton_wasm" [dependencies] wasmparser = "0.14.1" -cretonne = { path = "../cretonne", version = "0.3.1" } -cretonne-frontend = { path = "../frontend", version = "0.3.1" } +cretonne = { path = "../cretonne", version = "0.3.2" } +cretonne-frontend = { path = "../frontend", version = "0.3.2" } [dev-dependencies] tempdir = "0.3.5" From 6cf9bf36b8c66ad4ea82245d2f3aaffd25dc1c7e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 8 Mar 2018 02:26:15 -0800 Subject: [PATCH 1585/3084] Fix a typo in a comment. --- lib/cretonne/src/isa/enc_tables.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index 5154138c1b..7453b7c044 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -225,7 +225,7 @@ impl<'a> Encodings<'a> { self.legalize_actions[self.legalize as usize] } - /// Check if the `rpred` recipe predicate s satisfied. + /// Check if the `rpred` recipe predicate is satisfied. fn check_recipe(&self, rpred: RecipePredicate) -> bool { match rpred { Some(p) => p(self.isa_preds, self.inst), From 40ec50d0b6036f6c97804e8d99b994a5d43e1ca1 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 8 Mar 2018 02:26:26 -0800 Subject: [PATCH 1586/3084] Don't relax a branch to have different input constraints. When relaxing a branch, restrict the set of candidate encodings to those which have the same input constraints as the original encoding choice. This prevents situations where relaxation prefers a non-REX-prefixed encoding over a REX prefixed one because the end of the instruction can be one byte closer to the destination, in a situation where the encoding needs to be REX-prefixed because of one of the operand registers. This also makes the Context class perform encoding verification after relaxation, to catch similar problems in the future. Fixes #256. --- lib/cretonne/src/binemit/relaxation.rs | 24 +++++++++++++++++------- lib/cretonne/src/context.rs | 19 ++++++++++++++++++- lib/cretonne/src/isa/constraints.rs | 7 ++++--- 3 files changed, 39 insertions(+), 11 deletions(-) diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index 94835b736e..c6ea41f451 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -152,13 +152,23 @@ fn relax_branch( if let Some(enc) = isa.legal_encodings(dfg, &dfg[inst], ctrl_type).find( |&enc| { let range = encinfo.branch_range(enc).expect("Branch with no range"); - let in_range = range.contains(offset, dest_offset); - dbg!( - " trying [{}]: {}", - encinfo.display(enc), - if in_range { "OK" } else { "out of range" } - ); - in_range + if !range.contains(offset, dest_offset) { + dbg!(" trying [{}]: out of range", encinfo.display(enc)); + false + } else if encinfo.operand_constraints(enc) != + encinfo.operand_constraints(cur.func.encodings[inst]) + { + // Conservatively give up if the encoding has different constraints + // than the original, so that we don't risk picking a new encoding + // which the existing operands don't satisfy. We can't check for + // validity directly because we don't have a RegDiversions active so + // we don't know which registers are actually in use. + dbg!(" trying [{}]: constraints differ", encinfo.display(enc)); + false + } else { + dbg!(" trying [{}]: OK", encinfo.display(enc)); + true + } }, ) { diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 5b8db2414c..0eb7ff5c05 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -131,6 +131,20 @@ impl Context { } } + /// Run the locations verifier on the function. + pub fn verify_locations<'a>(&self, isa: &TargetIsa) -> verifier::Result { + verifier::verify_locations(isa, &self.func, None) + } + + /// Run the locations verifier only if the `enable_verifier` setting is true. + pub fn verify_locations_if<'a>(&self, isa: &TargetIsa) -> CtonResult { + if isa.flags().enable_verifier() { + self.verify_locations(isa).map_err(Into::into) + } else { + Ok(()) + } + } + /// Perform pre-legalization rewrites on the function. pub fn preopt(&mut self, isa: &TargetIsa) -> CtonResult { do_preopt(&mut self.func); @@ -212,13 +226,16 @@ impl Context { /// Insert prologue and epilogues after computing the stack frame layout. pub fn prologue_epilogue(&mut self, isa: &TargetIsa) -> CtonResult { isa.prologue_epilogue(&mut self.func)?; - self.verify_if(isa) + self.verify_if(isa)?; + self.verify_locations_if(isa)?; + Ok(()) } /// Run the branch relaxation pass and return the final code size. pub fn relax_branches(&mut self, isa: &TargetIsa) -> Result { let code_size = relax_branches(&mut self.func, isa)?; self.verify_if(isa)?; + self.verify_locations_if(isa)?; Ok(code_size) } diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index eb6b33c3cb..f831c0e187 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -13,6 +13,7 @@ use ir::{Function, ValueLoc, Inst}; use regalloc::RegDiversions; /// Register constraint for a single value operand or instruction result. +#[derive(PartialEq, Debug)] pub struct OperandConstraint { /// The kind of constraint. pub kind: ConstraintKind, @@ -53,7 +54,7 @@ impl OperandConstraint { } /// The different kinds of operand constraints. -#[derive(Clone, Copy, PartialEq, Eq)] +#[derive(Clone, Copy, PartialEq, Eq, Debug)] pub enum ConstraintKind { /// This operand or result must be a register from the given register class. Reg, @@ -89,7 +90,7 @@ pub enum ConstraintKind { } /// Value operand constraints for an encoding recipe. -#[derive(Clone)] +#[derive(PartialEq, Clone)] pub struct RecipeConstraints { /// Constraints for the instruction's fixed value operands. /// @@ -160,7 +161,7 @@ impl RecipeConstraints { /// - Intel uses the address of the instruction following the branch, `origin = 2` for a 2-byte /// branch instruction. /// - ARM's A32 encoding uses the address of the branch instruction + 8 bytes, `origin = 8`. -#[derive(Clone, Copy)] +#[derive(Clone, Copy, Debug)] pub struct BranchRange { /// Offset in bytes from the address of the branch instruction to the origin used for computing /// the branch displacement. This is the destination of a branch that encodes a 0 displacement. From b5f428b6f285d5482aa02db3bc14c4c07d9ba987 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 8 Mar 2018 02:50:09 -0800 Subject: [PATCH 1587/3084] Bump version to 0.3.3 --- cranelift/Cargo.toml | 12 ++++++------ cranelift/publish-all.sh | 2 +- lib/cretonne/Cargo.toml | 2 +- lib/frontend/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/wasm/Cargo.toml | 6 +++--- 7 files changed, 17 insertions(+), 17 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index a1479c63fc..cddfc4ff35 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.3.2" +version = "0.3.3" description = "Binaries for testing the Cretonne library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,11 +13,11 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne = { path = "lib/cretonne", version = "0.3.2" } -cretonne-reader = { path = "lib/reader", version = "0.3.2" } -cretonne-frontend = { path = "lib/frontend", version = "0.3.2" } -cretonne-wasm = { path = "lib/wasm", version = "0.3.2" } -cretonne-native = { path = "lib/native", version = "0.3.2" } +cretonne = { path = "lib/cretonne", version = "0.3.3" } +cretonne-reader = { path = "lib/reader", version = "0.3.3" } +cretonne-frontend = { path = "lib/frontend", version = "0.3.3" } +cretonne-wasm = { path = "lib/wasm", version = "0.3.3" } +cretonne-native = { path = "lib/native", version = "0.3.3" } filecheck = { path = "lib/filecheck" } docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index b380211212..ac0f3218cc 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -5,7 +5,7 @@ topdir="$(pwd)" # All the cretonne-* crates have the same version number # The filecheck crate version is managed independently. -version="0.3.2" +version="0.3.3" # Update all of the Cargo.toml files. # diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index d6e91e695d..bbf8901ae4 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.3.2" +version = "0.3.3" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index a09c692126..8a22f61efc 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.3.2" +version = "0.3.3" description = "Cretonne IL builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,4 +12,4 @@ readme = "README.md" name = "cton_frontend" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.2" } +cretonne = { path = "../cretonne", version = "0.3.3" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 5d2920a67a..c8ca693e71 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.3.2" +version = "0.3.3" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -11,7 +11,7 @@ readme = "README.md" name = "cton_native" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.2" } +cretonne = { path = "../cretonne", version = "0.3.3" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index b0d32b2a61..311ffe31cf 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.3.2" +version = "0.3.3" description = "Cretonne textual IL reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,4 +12,4 @@ readme = "README.md" name = "cton_reader" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.2" } +cretonne = { path = "../cretonne", version = "0.3.3" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 086e8c9735..1df8e0da15 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.3.2" +version = "0.3.3" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IL" repository = "https://github.com/Cretonne/cretonne" @@ -13,8 +13,8 @@ name = "cton_wasm" [dependencies] wasmparser = "0.14.1" -cretonne = { path = "../cretonne", version = "0.3.2" } -cretonne-frontend = { path = "../frontend", version = "0.3.2" } +cretonne = { path = "../cretonne", version = "0.3.3" } +cretonne-frontend = { path = "../frontend", version = "0.3.3" } [dev-dependencies] tempdir = "0.3.5" From b9d7a43439ba2fc63671106e25065d5c67109f40 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 8 Mar 2018 06:07:18 -0800 Subject: [PATCH 1588/3084] Elaborate on Cretonne's rustc backend goals. (#257) * Elaborate on Cretonne's rustc backend goals. * Remove these extra newlines. They don't show up in the rendered form anyway. * Fix typo. * The document is meant to speak with the voice of the project. --- rustc.rst | 68 +++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 58 insertions(+), 10 deletions(-) diff --git a/rustc.rst b/rustc.rst index 90ab7df8b4..20cebeb5d6 100644 --- a/rustc.rst +++ b/rustc.rst @@ -2,15 +2,63 @@ Cretonne in Rustc ================= -The Rust compiler currently uses LLVM as its optimizer and code generator for both debug and -release builds. The Cretonne project does not intend to compete with LLVM when it comes to -optimizing release builds, but for debug builds where compilation speed is paramount, it makes -sense to use Cretonne instead of LLVM. +One goal for Cretonne is to be usable as a backend suitable for compiling Rust +in debug mode. This mode doesn't require a lot of mid-level optimization, and it +does want very fast compile times, and this matches up fairly well with what we +expect Cretonne's initial strengths and weaknesses will be. Cretonne is being +designed to take aggressive advantage of multiple cores, and to be very efficient +with its use of memory. -- Cretonne is designed to take advantage of multi-core CPUs, making parallel code generation quite - easy. This is harder with LLVM which was designed before multi-core CPUs where mainstream. -- Cretonne is designed with compilation speed in mind. It makes engineering tradeoffs that favor - compilation speed over advanced optimizations. +Another goal is a "pretty good" backend. The idea here is to do the work to get +MIR-level inlining enabled, do some basic optimizations in Cretonne to capture the +low-hanging fruit, and then use that along with good low-level optimizations to +produce code which has a chance of being decently fast, with quite fast compile +times. It obviously wouldn't compete with LLVM-based release builds in terms of +optimization, but for some users, completely unoptimized code is too slow to test +with, so a "pretty good" mode might be good enough. -See `the discussion on the Rust internals forum -`_. +There's plenty of work to do to achieve these goals, and if achieve them, we'll have +enabled a Rust compiler written entirely in Rust, and enabled faster Rust compile +times for important use cases. + +With all that said, there is a potential goal beyond that, which is to build a +full optimizing release-capable backend. We can't predict how far Cretonne will go +yet, but we do have some crazy ideas about what such a thing might look like, +including: + +- Take advantage of Rust language properties in the optimizer. With LLVM, Rust is + able to use annotations to describe some of its aliasing guarantees, however the + annotations are awkward and limited. An optimizer that can represent the core + aliasing relationships that Rust provides directly has the potential to be very + powerful without the need for complex alias analysis logic. Unsafe blocks are an + interesting challenge, however in many simple cases, like Vec, it may be possible + to recover what the optimizer needs to know. + +- Design for superoptimization. Traditionally, compiler development teams have + spent many years of manual effort to identify patterns of code that can be + matched and replaced. Superoptimizers have been contributing some to this + effort, but in the future, we may be able to reverse roles. + Superoptimizers will do the bulk of the work, and humans will contribute + specialized optimizations that superoptimizers miss. This has the potential to + take a new optimizer from scratch to diminishing-returns territory with much + less manual effort. + +- Build an optimizer IR without the constraints of fast-debug-build compilation. + Cretonne's base IR is focused on Codegen, so a full-strength optimizer would either + use an IR layer on top of it (possibly using Cretonne's flexible EntityMap system), + or possibly an independent IR that could be translated to/from the base IR. Either + way, this overall architecture would keep the optimizer out of the way of the + non-optimizing build path, which keeps that path fast and simple, and gives the + optimizer more flexibility. If we then want to base the IR on a powerful data + structure like the Value State Dependence Graph (VSDG), we can do so with fewer + compromises. + +And, these ideas build on each other. For example, one of the challenges for +dependence-graph-oriented IRs like the VSDG is getting good enough memory dependence +information. But if we can get high-quality aliasing information directly from the +Rust front-end, we should be in great shape. As another example, it's often harder +for superoptimizers to reason about control flow than expression graphs. But, +graph-oriented IRs like the VSDG represent control flow as control dependencies. +It's difficult to say how powerful this combination will be until we try it, but +if nothing else, it should be very convenient to express pattern-matching over a +single graph that includes both data and control dependencies. From 8df9fe6c8787e84e15905f094a325259cd29c6fd Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 8 Mar 2018 12:12:10 -0800 Subject: [PATCH 1589/3084] Fix obsolete paths in comments. --- cranelift/docs/testing.rst | 2 +- lib/cretonne/src/isa/arm32/settings.rs | 2 +- lib/cretonne/src/isa/arm64/settings.rs | 2 +- lib/cretonne/src/isa/intel/settings.rs | 2 +- lib/cretonne/src/isa/riscv/settings.rs | 2 +- lib/cretonne/src/legalizer/mod.rs | 2 +- lib/cretonne/src/predicates.rs | 2 +- lib/cretonne/src/settings.rs | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index 951149c044..2b84b1c19b 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -119,7 +119,7 @@ All types of tests allow shared Cretonne settings to be modified: option : flag | setting "=" value The shared settings available for all target ISAs are defined in -:file:`lib/cretonne/meta/cretonne/settings.py`. +:file:`lib/cretonne/meta/base/settings.py`. The ``set`` lines apply settings cumulatively:: diff --git a/lib/cretonne/src/isa/arm32/settings.rs b/lib/cretonne/src/isa/arm32/settings.rs index e857716a64..b502deee40 100644 --- a/lib/cretonne/src/isa/arm32/settings.rs +++ b/lib/cretonne/src/isa/arm32/settings.rs @@ -5,5 +5,5 @@ use std::fmt; // Include code generated by `lib/cretonne/meta/gen_settings.py`. This file contains a public // `Flags` struct with an impl for all of the settings defined in -// `lib/cretonne/meta/cretonne/settings.py`. +// `lib/cretonne/meta/isa/arm32/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings-arm32.rs")); diff --git a/lib/cretonne/src/isa/arm64/settings.rs b/lib/cretonne/src/isa/arm64/settings.rs index 6427d7be99..b575168361 100644 --- a/lib/cretonne/src/isa/arm64/settings.rs +++ b/lib/cretonne/src/isa/arm64/settings.rs @@ -5,5 +5,5 @@ use std::fmt; // Include code generated by `lib/cretonne/meta/gen_settings.py`. This file contains a public // `Flags` struct with an impl for all of the settings defined in -// `lib/cretonne/meta/cretonne/settings.py`. +// `lib/cretonne/meta/isa/arm64/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings-arm64.rs")); diff --git a/lib/cretonne/src/isa/intel/settings.rs b/lib/cretonne/src/isa/intel/settings.rs index f933e91b87..147af4c2fa 100644 --- a/lib/cretonne/src/isa/intel/settings.rs +++ b/lib/cretonne/src/isa/intel/settings.rs @@ -5,7 +5,7 @@ use std::fmt; // Include code generated by `lib/cretonne/meta/gen_settings.py`. This file contains a public // `Flags` struct with an impl for all of the settings defined in -// `lib/cretonne/meta/cretonne/settings.py`. +// `lib/cretonne/meta/isa/intel/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings-intel.rs")); #[cfg(test)] diff --git a/lib/cretonne/src/isa/riscv/settings.rs b/lib/cretonne/src/isa/riscv/settings.rs index 8cb376e5da..2aa78fa853 100644 --- a/lib/cretonne/src/isa/riscv/settings.rs +++ b/lib/cretonne/src/isa/riscv/settings.rs @@ -5,7 +5,7 @@ use std::fmt; // Include code generated by `lib/cretonne/meta/gen_settings.py`. This file contains a public // `Flags` struct with an impl for all of the settings defined in -// `lib/cretonne/meta/cretonne/settings.py`. +// `lib/cretonne/meta/isa/riscv/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs")); #[cfg(test)] diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 81dde2f457..189a3e735d 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -107,7 +107,7 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is } // Include legalization patterns that were generated by `gen_legalizer.py` from the `XForms` in -// `meta/cretonne/legalize.py`. +// `lib/cretonne/meta/base/legalize.py`. // // Concretely, this defines private functions `narrow()`, and `expand()`. include!(concat!(env!("OUT_DIR"), "/legalizer.rs")); diff --git a/lib/cretonne/src/predicates.rs b/lib/cretonne/src/predicates.rs index 6ce3e0a799..63d2e79af3 100644 --- a/lib/cretonne/src/predicates.rs +++ b/lib/cretonne/src/predicates.rs @@ -1,7 +1,7 @@ //! Predicate functions for testing instruction fields. //! //! This module defines functions that are used by the instruction predicates defined by -//! `lib/cretonne/meta/cretonne/predicates.py` classes. +//! `lib/cretonne/meta/cdsl/predicates.py` classes. //! //! The predicates the operate on integer fields use `Into` as a shared trait bound. This //! bound is implemented by all the native integer types as well as `Imm64`. diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 36a48e04b4..a87ceda695 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -311,7 +311,7 @@ pub mod detail { } // Include code generated by `meta/gen_settings.py`. This file contains a public `Flags` struct -// with an impl for all of the settings defined in `meta/cretonne/settings.py`. +// with an impl for all of the settings defined in `lib/cretonne/meta/base/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings.rs")); /// Wrapper containing flags and optionally a `TargetIsa` trait object. From 55d0efcb1495029fe96a808052e2da6626761537 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 9 Mar 2018 14:30:26 -0800 Subject: [PATCH 1590/3084] Pass the wasmparser::Operator by value, simplifying the code. --- lib/wasm/src/code_translator.rs | 10 +++++----- lib/wasm/src/func_translator.rs | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 5d7e59f829..f08df8b0e1 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -38,7 +38,7 @@ use std::{i32, u32}; /// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted /// a return. pub fn translate_operator( - op: &Operator, + op: Operator, builder: &mut FunctionBuilder, state: &mut TranslationState, environ: &mut FE, @@ -48,7 +48,7 @@ pub fn translate_operator( } // This big match treats all Wasm code operators. - match *op { + match op { /********************************** Locals **************************************** * `get_local` and `set_local` are treated as non-SSA variables and will completely * disappear in the Cretonne Code @@ -265,7 +265,7 @@ pub fn translate_operator( state.peekn(return_count), ); } - Operator::BrTable { ref table } => { + Operator::BrTable { table } => { let (depths, default) = table.read_table(); let mut min_depth = default; for depth in &depths { @@ -935,11 +935,11 @@ pub fn translate_operator( /// are dropped but special ones like `End` or `Else` signal the potential end of the unreachable /// portion so the translation state muts be updated accordingly. fn translate_unreachable_operator( - op: &Operator, + op: Operator, builder: &mut FunctionBuilder, state: &mut TranslationState, ) { - match *op { + match op { Operator::If { ty: _ } => { // Push a placeholder control stack entry. The if isn't reachable, // so we don't have any branches anywhere. diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 7e8955e816..d5737bd446 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -198,7 +198,7 @@ fn parse_function_body( while !state.control_stack.is_empty() { builder.set_srcloc(cur_srcloc(&reader)); let op = reader.read_operator().map_err(|_| CtonError::InvalidInput)?; - translate_operator(&op, builder, state, environ); + translate_operator(op, builder, state, environ); } // The final `End` operator left us in the exit block where we need to manually add a return From 56c7d857276b2d4d47e6112f444133aa9760f793 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 9 Mar 2018 14:31:12 -0800 Subject: [PATCH 1591/3084] More minor code simplifications. --- lib/wasm/src/code_translator.rs | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index f08df8b0e1..100dfbab41 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -282,10 +282,10 @@ pub fn translate_operator( min_depth_frame.num_return_values() } }; + let val = state.pop1(); + let mut data = JumpTableData::with_capacity(depths.len()); if jump_args_count == 0 { // No jump arguments - let val = state.pop1(); - let mut data = JumpTableData::with_capacity(depths.len()); for depth in depths { let ebb = { let i = state.control_stack.len() - 1 - (depth as usize); @@ -307,9 +307,7 @@ pub fn translate_operator( } else { // Here we have jump arguments, but Cretonne's br_table doesn't support them // We then proceed to split the edges going out of the br_table - let val = state.pop1(); let return_count = jump_args_count; - let mut data = JumpTableData::with_capacity(depths.len()); let mut dest_ebb_sequence = Vec::new(); let mut dest_ebb_map = HashMap::new(); for depth in depths { From 1c72ccfe0ab8a6b83cc2a8946c68dad339436c61 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 9 Mar 2018 15:01:13 -0800 Subject: [PATCH 1592/3084] Define a `Variable` struct so that frontends don't have to. Frontends can still use their own types with `ILBuilder` and `FunctionBuilder`. This just provides a basic `Variable` struct for frontends that want it. --- lib/frontend/src/frontend.rs | 27 +++++----------- lib/frontend/src/lib.rs | 25 ++++----------- lib/frontend/src/ssa.rs | 60 ++++++++++++++---------------------- lib/frontend/src/variable.rs | 24 +++++++++++++++ 4 files changed, 61 insertions(+), 75 deletions(-) create mode 100644 lib/frontend/src/variable.rs diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index fba2f89a01..6802884b6a 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -15,6 +15,10 @@ use cretonne::packed_option::PackedOption; /// In order to reduce memory reallocations when compiling multiple functions, /// `ILBuilder` holds various data structures which are cleared between /// functions, rather than dropped, preserving the underlying allocations. +/// +/// The `Variable` parameter can be any index-like type that can be made to +/// implement `EntityRef`. For frontends that don't have an obvious type to +/// use here, `variable::Variable` can be used. pub struct ILBuilder where Variable: EntityRef, @@ -590,22 +594,7 @@ mod tests { use frontend::{ILBuilder, FunctionBuilder}; use cretonne::verifier::verify_function; use cretonne::settings; - - use std::u32; - - // An opaque reference to variable. - #[derive(Copy, Clone, PartialEq, Eq, Debug)] - pub struct Variable(u32); - impl EntityRef for Variable { - fn new(index: usize) -> Self { - assert!(index < (u32::MAX as usize)); - Variable(index as u32) - } - - fn index(self) -> usize { - self.0 as usize - } - } + use Variable; fn sample_function(lazy_seal: bool) { let mut sig = Signature::new(CallConv::Native); @@ -620,9 +609,9 @@ mod tests { let block0 = builder.create_ebb(); let block1 = builder.create_ebb(); let block2 = builder.create_ebb(); - let x = Variable(0); - let y = Variable(1); - let z = Variable(2); + let x = Variable::new(0); + let y = Variable::new(1); + let z = Variable::new(2); builder.declare_var(x, I32); builder.declare_var(y, I32); builder.declare_var(z, I32); diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 8d2de2237b..a95d662893 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -39,23 +39,8 @@ //! use cretonne::ir::{ExternalName, CallConv, Function, Signature, AbiParam, InstBuilder}; //! use cretonne::ir::types::*; //! use cretonne::settings; -//! use cton_frontend::{ILBuilder, FunctionBuilder}; +//! use cton_frontend::{ILBuilder, FunctionBuilder, Variable}; //! use cretonne::verifier::verify_function; -//! use std::u32; -//! -//! // An opaque reference to variable. -//! #[derive(Copy, Clone, PartialEq, Eq, Debug)] -//! pub struct Variable(u32); -//! impl EntityRef for Variable { -//! fn new(index: usize) -> Self { -//! assert!(index < (u32::MAX as usize)); -//! Variable(index as u32) -//! } -//! -//! fn index(self) -> usize { -//! self.0 as usize -//! } -//! } //! //! fn main() { //! let mut sig = Signature::new(CallConv::Native); @@ -69,9 +54,9 @@ //! let block0 = builder.create_ebb(); //! let block1 = builder.create_ebb(); //! let block2 = builder.create_ebb(); -//! let x = Variable(0); -//! let y = Variable(1); -//! let z = Variable(2); +//! let x = Variable::new(0); +//! let y = Variable::new(1); +//! let z = Variable::new(2); //! builder.declare_var(x, I32); //! builder.declare_var(y, I32); //! builder.declare_var(z, I32); @@ -149,6 +134,8 @@ extern crate cretonne; pub use frontend::{ILBuilder, FunctionBuilder}; +pub use variable::Variable; mod frontend; mod ssa; +mod variable; diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 06dba9d4ed..5b974799bf 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -720,21 +720,7 @@ mod tests { use cretonne::ir::instructions::BranchInfo; use cretonne::settings; use ssa::SSABuilder; - use std::u32; - - /// An opaque reference to variable. - #[derive(Copy, Clone, PartialEq, Eq, Debug)] - pub struct Variable(u32); - impl EntityRef for Variable { - fn new(index: usize) -> Self { - assert!(index < (u32::MAX as usize)); - Variable(index as u32) - } - - fn index(self) -> usize { - self.0 as usize - } - } + use Variable; #[test] fn simple_block() { @@ -748,14 +734,14 @@ mod tests { // z = x + z; let block = ssa.declare_ebb_header_block(ebb0); - let x_var = Variable(0); + let x_var = Variable::new(0); let x_ssa = { let mut cur = FuncCursor::new(&mut func); cur.insert_ebb(ebb0); cur.ins().iconst(I32, 1) }; ssa.def_var(x_var, x_ssa, block); - let y_var = Variable(1); + let y_var = Variable::new(1); let y_ssa = { let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); cur.ins().iconst(I32, 2) @@ -764,7 +750,7 @@ mod tests { assert_eq!(ssa.use_var(&mut func, x_var, I32, block).0, x_ssa); assert_eq!(ssa.use_var(&mut func, y_var, I32, block).0, y_ssa); - let z_var = Variable(2); + let z_var = Variable::new(2); let x_use1 = ssa.use_var(&mut func, x_var, I32, block).0; let y_use1 = ssa.use_var(&mut func, y_var, I32, block).0; let z1_ssa = { @@ -800,7 +786,7 @@ mod tests { // y = x + y; let block0 = ssa.declare_ebb_header_block(ebb0); - let x_var = Variable(0); + let x_var = Variable::new(0); let x_ssa = { let mut cur = FuncCursor::new(&mut func); cur.insert_ebb(ebb0); @@ -809,7 +795,7 @@ mod tests { cur.ins().iconst(I32, 1) }; ssa.def_var(x_var, x_ssa, block0); - let y_var = Variable(1); + let y_var = Variable::new(1); let y_ssa = { let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); cur.ins().iconst(I32, 2) @@ -817,7 +803,7 @@ mod tests { ssa.def_var(y_var, y_ssa, block0); assert_eq!(ssa.use_var(&mut func, x_var, I32, block0).0, x_ssa); assert_eq!(ssa.use_var(&mut func, y_var, I32, block0).0, y_ssa); - let z_var = Variable(2); + let z_var = Variable::new(2); let x_use1 = ssa.use_var(&mut func, x_var, I32, block0).0; let y_use1 = ssa.use_var(&mut func, y_var, I32, block0).0; let z1_ssa = { @@ -888,7 +874,7 @@ mod tests { let block0 = ssa.declare_ebb_header_block(ebb0); ssa.seal_ebb_header_block(ebb0, &mut func); - let x_var = Variable(0); + let x_var = Variable::new(0); let x1 = { let mut cur = FuncCursor::new(&mut func); cur.insert_ebb(ebb0); @@ -899,14 +885,14 @@ mod tests { }; ssa.def_var(x_var, x1, block0); assert_eq!(ssa.use_var(&mut func, x_var, I32, block0).0, x1); - let y_var = Variable(1); + let y_var = Variable::new(1); let y1 = { let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0); cur.ins().iconst(I32, 2) }; ssa.def_var(y_var, y1, block0); assert_eq!(ssa.use_var(&mut func, y_var, I32, block0).0, y1); - let z_var = Variable(2); + let z_var = Variable::new(2); let x2 = ssa.use_var(&mut func, x_var, I32, block0).0; assert_eq!(x2, x1); let y2 = ssa.use_var(&mut func, y_var, I32, block0).0; @@ -995,7 +981,7 @@ mod tests { // let block0 = ssa.declare_ebb_header_block(ebb0); ssa.seal_ebb_header_block(ebb0, &mut func); - let x_var = Variable(0); + let x_var = Variable::new(0); let x1 = { let mut cur = FuncCursor::new(&mut func); cur.insert_ebb(ebb0); @@ -1060,9 +1046,9 @@ mod tests { // jump ebb1 // let block0 = ssa.declare_ebb_header_block(ebb0); - let x_var = Variable(0); - let y_var = Variable(1); - let z_var = Variable(2); + let x_var = Variable::new(0); + let y_var = Variable::new(1); + let z_var = Variable::new(2); ssa.seal_ebb_header_block(ebb0, &mut func); let x1 = { let mut cur = FuncCursor::new(&mut func); @@ -1125,11 +1111,11 @@ mod tests { let ebb0 = func.dfg.make_ebb(); let block = ssa.declare_ebb_header_block(ebb0); ssa.seal_ebb_header_block(ebb0, &mut func); - let i32_var = Variable(0); - let f32_var = Variable(1); - let f64_var = Variable(2); - let b1_var = Variable(3); - let f32x4_var = Variable(4); + let i32_var = Variable::new(0); + let f32_var = Variable::new(1); + let f64_var = Variable::new(2); + let b1_var = Variable::new(3); + let f32x4_var = Variable::new(4); ssa.use_var(&mut func, i32_var, I32, block); ssa.use_var(&mut func, f32_var, F32, block); ssa.use_var(&mut func, f64_var, F64, block); @@ -1147,7 +1133,7 @@ mod tests { let ebb0 = func.dfg.make_ebb(); let block = ssa.declare_ebb_header_block(ebb0); ssa.seal_ebb_header_block(ebb0, &mut func); - let x_var = Variable(0); + let x_var = Variable::new(0); assert_eq!(func.dfg.num_ebb_params(ebb0), 0); ssa.use_var(&mut func, x_var, I32, block); assert_eq!(func.dfg.num_ebb_params(ebb0), 0); @@ -1166,7 +1152,7 @@ mod tests { let mut ssa: SSABuilder = SSABuilder::new(); let ebb0 = func.dfg.make_ebb(); let block = ssa.declare_ebb_header_block(ebb0); - let x_var = Variable(0); + let x_var = Variable::new(0); assert_eq!(func.dfg.num_ebb_params(ebb0), 0); ssa.use_var(&mut func, x_var, I32, block); assert_eq!(func.dfg.num_ebb_params(ebb0), 1); @@ -1200,7 +1186,7 @@ mod tests { cur.insert_ebb(ebb1); cur.goto_bottom(ebb0); cur.ins().return_(&[]); - let x_var = Variable(0); + let x_var = Variable::new(0); cur.goto_bottom(ebb1); let val = ssa.use_var(&mut cur.func, x_var, I32, block1).0; let brz = cur.ins().brz(val, ebb1, &[]); @@ -1237,7 +1223,7 @@ mod tests { let block2 = ssa.declare_ebb_header_block(ebb2); { let mut cur = FuncCursor::new(&mut func); - let x_var = Variable(0); + let x_var = Variable::new(0); cur.insert_ebb(ebb0); cur.insert_ebb(ebb1); cur.insert_ebb(ebb2); diff --git a/lib/frontend/src/variable.rs b/lib/frontend/src/variable.rs new file mode 100644 index 0000000000..6b322e184f --- /dev/null +++ b/lib/frontend/src/variable.rs @@ -0,0 +1,24 @@ +//! A basic `Variable` implementation. +//! +//! `ILBuilder`, `FunctionBuilder`, and related types have a `Variable` +//! type parameter, to allow frontends that identify variables with +//! their own index types to use them directly. Frontends which don't +//! can use the `Variable` defined here. + +use cretonne::entity::EntityRef; +use std::u32; + +///! An opaque reference to a variable. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct Variable(u32); + +impl EntityRef for Variable { + fn new(index: usize) -> Self { + assert!(index < (u32::MAX as usize)); + Variable(index as u32) + } + + fn index(self) -> usize { + self.0 as usize + } +} From e441337e4bb8cac1a8cee1ca469067d0de0a9526 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 9 Mar 2018 16:32:13 -0800 Subject: [PATCH 1593/3084] Bump version to 0.3.4 --- cranelift/Cargo.toml | 12 ++++++------ cranelift/publish-all.sh | 2 +- lib/cretonne/Cargo.toml | 2 +- lib/frontend/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/wasm/Cargo.toml | 6 +++--- 7 files changed, 17 insertions(+), 17 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index cddfc4ff35..297e8c31b3 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.3.3" +version = "0.3.4" description = "Binaries for testing the Cretonne library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,11 +13,11 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne = { path = "lib/cretonne", version = "0.3.3" } -cretonne-reader = { path = "lib/reader", version = "0.3.3" } -cretonne-frontend = { path = "lib/frontend", version = "0.3.3" } -cretonne-wasm = { path = "lib/wasm", version = "0.3.3" } -cretonne-native = { path = "lib/native", version = "0.3.3" } +cretonne = { path = "lib/cretonne", version = "0.3.4" } +cretonne-reader = { path = "lib/reader", version = "0.3.4" } +cretonne-frontend = { path = "lib/frontend", version = "0.3.4" } +cretonne-wasm = { path = "lib/wasm", version = "0.3.4" } +cretonne-native = { path = "lib/native", version = "0.3.4" } filecheck = { path = "lib/filecheck" } docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index ac0f3218cc..7d8b7f7f3c 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -5,7 +5,7 @@ topdir="$(pwd)" # All the cretonne-* crates have the same version number # The filecheck crate version is managed independently. -version="0.3.3" +version="0.3.4" # Update all of the Cargo.toml files. # diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index bbf8901ae4..597ff1efd6 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.3.3" +version = "0.3.4" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 8a22f61efc..0085036ed0 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.3.3" +version = "0.3.4" description = "Cretonne IL builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,4 +12,4 @@ readme = "README.md" name = "cton_frontend" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.3" } +cretonne = { path = "../cretonne", version = "0.3.4" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index c8ca693e71..10856d98a5 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.3.3" +version = "0.3.4" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -11,7 +11,7 @@ readme = "README.md" name = "cton_native" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.3" } +cretonne = { path = "../cretonne", version = "0.3.4" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 311ffe31cf..0d2061e361 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.3.3" +version = "0.3.4" description = "Cretonne textual IL reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,4 +12,4 @@ readme = "README.md" name = "cton_reader" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.3" } +cretonne = { path = "../cretonne", version = "0.3.4" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 1df8e0da15..8e61d440c0 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.3.3" +version = "0.3.4" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IL" repository = "https://github.com/Cretonne/cretonne" @@ -13,8 +13,8 @@ name = "cton_wasm" [dependencies] wasmparser = "0.14.1" -cretonne = { path = "../cretonne", version = "0.3.3" } -cretonne-frontend = { path = "../frontend", version = "0.3.3" } +cretonne = { path = "../cretonne", version = "0.3.4" } +cretonne-frontend = { path = "../frontend", version = "0.3.4" } [dev-dependencies] tempdir = "0.3.5" From b8a106adf010743b86cce8ef24fae03b7883de05 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 12 Mar 2018 10:28:19 -0700 Subject: [PATCH 1594/3084] Remove the "has_sse2" flag. Cretonne currently requires SSE2 support pervasively, so it's not meaningful to have a setting for it. --- lib/cretonne/meta/isa/intel/settings.py | 7 ++----- lib/native/src/lib.rs | 15 ++++++++------- 2 files changed, 10 insertions(+), 12 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/settings.py b/lib/cretonne/meta/isa/intel/settings.py index 5817c48c0e..c62012e0c1 100644 --- a/lib/cretonne/meta/isa/intel/settings.py +++ b/lib/cretonne/meta/isa/intel/settings.py @@ -11,9 +11,6 @@ ISA.settings = SettingGroup('intel', parent=shared.group) # The has_* settings here correspond to CPUID bits. -# CPUID.01H:EDX -has_sse2 = BoolSetting("SSE2: CPUID.01H:EDX.SSE2[bit 26]") - # CPUID.01H:ECX has_sse3 = BoolSetting("SSE3: CPUID.01H:ECX.SSE3[bit 0]") has_ssse3 = BoolSetting("SSSE3: CPUID.01H:ECX.SSSE3[bit 9]") @@ -40,9 +37,9 @@ use_lzcnt = And(has_lzcnt) # Presets corresponding to Intel CPUs. -baseline = Preset(has_sse2) +baseline = Preset() nehalem = Preset( - has_sse2, has_sse3, has_ssse3, has_sse41, has_sse42, has_popcnt) + has_sse3, has_ssse3, has_sse41, has_sse42, has_popcnt) haswell = Preset(nehalem, has_bmi1, has_lzcnt) ISA.settings.close(globals()) diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index d7c6a8d0ab..ea828ea252 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -19,7 +19,7 @@ use raw_cpuid::CpuId; /// Return `settings` and `isa` builders configured for the current host /// machine, or `Err(())` if the host machine is not supported /// in the current configuration. -pub fn builders() -> Result<(settings::Builder, isa::Builder), ()> { +pub fn builders() -> Result<(settings::Builder, isa::Builder), &'static str> { let mut flag_builder = settings::builder(); // TODO: Add RISC-V support once Rust supports it. @@ -35,28 +35,28 @@ pub fn builders() -> Result<(settings::Builder, isa::Builder), ()> { } else if cfg!(target_arch = "aarch64") { "arm64" } else { - return Err(()); + return Err("unrecognized architecture"); }; let mut isa_builder = isa::lookup(name).map_err(|err| match err { isa::LookupError::Unknown => panic!(), - isa::LookupError::Unsupported => (), + isa::LookupError::Unsupported => "unsupported architecture", })?; if cfg!(any(target_arch = "x86", target_arch = "x86_64")) { - parse_x86_cpuid(&mut isa_builder); + parse_x86_cpuid(&mut isa_builder)?; } Ok((flag_builder, isa_builder)) } #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] -fn parse_x86_cpuid(isa_builder: &mut isa::Builder) { +fn parse_x86_cpuid(isa_builder: &mut isa::Builder) -> Result<(), &'static str> { let cpuid = CpuId::new(); if let Some(info) = cpuid.get_feature_info() { - if info.has_sse2() { - isa_builder.enable("has_sse2").unwrap(); + if !info.has_sse2() { + return Err("x86 support requires SSE2"); } if info.has_sse3() { isa_builder.enable("has_sse3").unwrap(); @@ -87,4 +87,5 @@ fn parse_x86_cpuid(isa_builder: &mut isa::Builder) { isa_builder.enable("has_lzcnt").unwrap(); } } + Ok(()) } From ad363d7e6be281782311f30e88d5c591f1830672 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 12 Mar 2018 10:28:25 -0700 Subject: [PATCH 1595/3084] Replace cretonne-wasm's Local with cretonne-frontend's Variable. Previously, cretonne-wasm used its own Local struct for identifying local variables. However, now that cretonne-frontend provides a Variable struct, just use that instead. --- lib/frontend/src/variable.rs | 8 ++++++++ lib/wasm/src/code_translator.rs | 22 ++++++++++++---------- lib/wasm/src/func_translator.rs | 17 ++++++++--------- lib/wasm/src/translation_utils.rs | 19 ------------------- 4 files changed, 28 insertions(+), 38 deletions(-) diff --git a/lib/frontend/src/variable.rs b/lib/frontend/src/variable.rs index 6b322e184f..5b35a3426b 100644 --- a/lib/frontend/src/variable.rs +++ b/lib/frontend/src/variable.rs @@ -12,6 +12,14 @@ use std::u32; #[derive(Copy, Clone, PartialEq, Eq, Debug)] pub struct Variable(u32); +impl Variable { + /// Create a new Variable with the given index. + pub fn with_u32(index: u32) -> Self { + assert!(index < u32::MAX); + Variable(index) + } +} + impl EntityRef for Variable { fn new(index: usize) -> Self { assert!(index < (u32::MAX as usize)); diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 100dfbab41..60b432a582 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -26,9 +26,9 @@ use cretonne::ir::{self, InstBuilder, MemFlags, JumpTableData}; use cretonne::ir::types::*; use cretonne::ir::condcodes::{IntCC, FloatCC}; use cretonne::packed_option::ReservedValue; -use cton_frontend::FunctionBuilder; +use cton_frontend::{FunctionBuilder, Variable}; use wasmparser::{Operator, MemoryImmediate}; -use translation_utils::{f32_translation, f64_translation, type_to_type, num_return_values, Local}; +use translation_utils::{f32_translation, f64_translation, type_to_type, num_return_values}; use translation_utils::{TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; use state::{TranslationState, ControlStackFrame}; use std::collections::{HashMap, hash_map}; @@ -39,7 +39,7 @@ use std::{i32, u32}; /// a return. pub fn translate_operator( op: Operator, - builder: &mut FunctionBuilder, + builder: &mut FunctionBuilder, state: &mut TranslationState, environ: &mut FE, ) { @@ -53,14 +53,16 @@ pub fn translate_operator( * `get_local` and `set_local` are treated as non-SSA variables and will completely * disappear in the Cretonne Code ***********************************************************************************/ - Operator::GetLocal { local_index } => state.push1(builder.use_var(Local(local_index))), + Operator::GetLocal { local_index } => { + state.push1(builder.use_var(Variable::with_u32(local_index))) + } Operator::SetLocal { local_index } => { let val = state.pop1(); - builder.def_var(Local(local_index), val); + builder.def_var(Variable::with_u32(local_index), val); } Operator::TeeLocal { local_index } => { let val = state.peek1(); - builder.def_var(Local(local_index), val); + builder.def_var(Variable::with_u32(local_index), val); } /********************************** Globals **************************************** * `get_global` and `set_global` are handled by the environment. @@ -934,7 +936,7 @@ pub fn translate_operator( /// portion so the translation state muts be updated accordingly. fn translate_unreachable_operator( op: Operator, - builder: &mut FunctionBuilder, + builder: &mut FunctionBuilder, state: &mut TranslationState, ) { match op { @@ -1019,7 +1021,7 @@ fn get_heap_addr( addr32: ir::Value, offset: u32, addr_ty: ir::Type, - builder: &mut FunctionBuilder, + builder: &mut FunctionBuilder, ) -> (ir::Value, i32) { use std::cmp::min; @@ -1055,7 +1057,7 @@ fn translate_load( offset: u32, opcode: ir::Opcode, result_ty: ir::Type, - builder: &mut FunctionBuilder, + builder: &mut FunctionBuilder, state: &mut TranslationState, environ: &mut FE, ) { @@ -1078,7 +1080,7 @@ fn translate_load( fn translate_store( offset: u32, opcode: ir::Opcode, - builder: &mut FunctionBuilder, + builder: &mut FunctionBuilder, state: &mut TranslationState, environ: &mut FE, ) { diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index d5737bd446..0632630095 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -9,10 +9,9 @@ use cretonne::entity::EntityRef; use cretonne::ir::{self, InstBuilder, Ebb}; use cretonne::result::{CtonResult, CtonError}; use cretonne::timing; -use cton_frontend::{ILBuilder, FunctionBuilder}; +use cton_frontend::{ILBuilder, FunctionBuilder, Variable}; use environ::FuncEnvironment; use state::TranslationState; -use translation_utils::Local; use wasmparser::{self, BinaryReader}; /// WebAssembly to Cretonne IL function translator. @@ -21,7 +20,7 @@ use wasmparser::{self, BinaryReader}; /// by a `FuncEnvironment` object. A single translator instance can be reused to translate multiple /// functions which will reduce heap allocation traffic. pub struct FuncTranslator { - il_builder: ILBuilder, + il_builder: ILBuilder, state: TranslationState, } @@ -107,7 +106,7 @@ impl FuncTranslator { /// Declare local variables for the signature parameters that correspond to WebAssembly locals. /// /// Return the number of local variables declared. -fn declare_wasm_parameters(builder: &mut FunctionBuilder, entry_block: Ebb) -> usize { +fn declare_wasm_parameters(builder: &mut FunctionBuilder, entry_block: Ebb) -> usize { let sig_len = builder.func.signature.params.len(); let mut next_local = 0; for i in 0..sig_len { @@ -116,7 +115,7 @@ fn declare_wasm_parameters(builder: &mut FunctionBuilder, entry_block: Eb // signature parameters. For example, a `vmctx` pointer. if param_type.purpose == ir::ArgumentPurpose::Normal { // This is a normal WebAssembly signature parameter, so create a local for it. - let local = Local::new(next_local); + let local = Variable::new(next_local); builder.declare_var(local, param_type.value_type); next_local += 1; @@ -133,7 +132,7 @@ fn declare_wasm_parameters(builder: &mut FunctionBuilder, entry_block: Eb /// Declare local variables, starting from `num_params`. fn parse_local_decls( reader: &mut BinaryReader, - builder: &mut FunctionBuilder, + builder: &mut FunctionBuilder, num_params: usize, ) -> CtonResult { let mut next_local = num_params; @@ -157,7 +156,7 @@ fn parse_local_decls( /// /// Fail of too many locals are declared in the function, or if the type is not valid for a local. fn declare_locals( - builder: &mut FunctionBuilder, + builder: &mut FunctionBuilder, count: u32, wasm_type: wasmparser::Type, next_local: &mut usize, @@ -174,7 +173,7 @@ fn declare_locals( let ty = builder.func.dfg.value_type(zeroval); for _ in 0..count { - let local = Local::new(*next_local); + let local = Variable::new(*next_local); builder.declare_var(local, ty); builder.def_var(local, zeroval); *next_local += 1; @@ -187,7 +186,7 @@ fn declare_locals( /// arguments and locals are declared in the builder. fn parse_function_body( mut reader: BinaryReader, - builder: &mut FunctionBuilder, + builder: &mut FunctionBuilder, state: &mut TranslationState, environ: &mut FE, ) -> CtonResult { diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 4c70d708ed..44c03fb6dd 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -71,25 +71,6 @@ pub struct Memory { pub shared: bool, } -/// Wrapper to a `get_local` and `set_local` index. They are WebAssembly's non-SSA variables. -#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug)] -pub struct Local(pub u32); -impl cretonne::entity::EntityRef for Local { - fn new(index: usize) -> Self { - debug_assert!(index < (u32::MAX as usize)); - Local(index as u32) - } - - fn index(self) -> usize { - self.0 as usize - } -} -impl Default for Local { - fn default() -> Self { - Local(u32::MAX) - } -} - /// Helper function translating wasmparser types to Cretonne types when possible. pub fn type_to_type(ty: &wasmparser::Type) -> Result { match *ty { From e81a27fb5d57cc2782bc442e33e87a6ceb9160cd Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 12 Mar 2018 10:28:32 -0700 Subject: [PATCH 1596/3084] Implement Debug for `ArgAction` and `Affinity`. --- lib/cretonne/src/abi.rs | 2 +- lib/cretonne/src/regalloc/affinity.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index 5c121824aa..05c96c79d5 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -11,7 +11,7 @@ use std::cmp::Ordering; /// /// An argument may go through a sequence of legalization steps before it reaches the final /// `Assign` action. -#[derive(Clone, Copy)] +#[derive(Clone, Copy, Debug)] pub enum ArgAction { /// Assign the argument to the given location. Assign(ArgumentLoc), diff --git a/lib/cretonne/src/regalloc/affinity.rs b/lib/cretonne/src/regalloc/affinity.rs index 1c50cbbbab..7d85ae8721 100644 --- a/lib/cretonne/src/regalloc/affinity.rs +++ b/lib/cretonne/src/regalloc/affinity.rs @@ -13,7 +13,7 @@ use ir::{AbiParam, ArgumentLoc}; use isa::{TargetIsa, RegInfo, RegClassIndex, OperandConstraint, ConstraintKind}; /// Preferred register allocation for an SSA value. -#[derive(Clone, Copy)] +#[derive(Clone, Copy, Debug)] pub enum Affinity { /// No affinity. /// From 30f8daa9d646e51903d5e3440cfae7f3681a158e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 12 Mar 2018 10:28:35 -0700 Subject: [PATCH 1597/3084] Replace `assert!` with `debug_assert!` in production code paths. This allows the assertions to be disabled in release builds, so that the code is faster and smaller, at the expense of not performing the checks. Assertions can be re-enabled in release builds with the debug-assertions flag in Cargo.toml, as the top-level Cargo.toml file does. --- lib/cretonne/meta/gen_instr.py | 2 +- lib/cretonne/meta/gen_legalizer.py | 3 +- lib/cretonne/meta/gen_settings.py | 2 +- lib/cretonne/src/abi.rs | 6 ++-- lib/cretonne/src/bforest/path.rs | 6 ++-- lib/cretonne/src/binemit/mod.rs | 2 +- lib/cretonne/src/binemit/relaxation.rs | 4 +-- lib/cretonne/src/bitset.rs | 8 ++--- lib/cretonne/src/cursor.rs | 6 ++-- lib/cretonne/src/divconst_magic_numbers.rs | 20 ++++++------ lib/cretonne/src/dominator_tree.rs | 8 ++--- lib/cretonne/src/entity/list.rs | 16 +++++----- lib/cretonne/src/entity/mod.rs | 2 +- lib/cretonne/src/entity/sparse.rs | 2 +- lib/cretonne/src/ir/dfg.rs | 26 +++++++-------- lib/cretonne/src/ir/immediates.rs | 2 +- lib/cretonne/src/ir/instructions.rs | 4 +-- lib/cretonne/src/ir/layout.rs | 30 ++++++++--------- lib/cretonne/src/ir/progpoint.rs | 4 +-- lib/cretonne/src/ir/stackslot.rs | 4 +-- lib/cretonne/src/isa/intel/abi.rs | 2 +- lib/cretonne/src/isa/riscv/abi.rs | 2 +- lib/cretonne/src/isa/riscv/binemit.rs | 4 +-- lib/cretonne/src/legalizer/boundary.rs | 30 ++++++++--------- lib/cretonne/src/legalizer/globalvar.rs | 2 +- lib/cretonne/src/legalizer/heap.rs | 2 +- lib/cretonne/src/legalizer/mod.rs | 2 +- lib/cretonne/src/legalizer/split.rs | 4 +-- lib/cretonne/src/preopt.rs | 32 +++++++++---------- lib/cretonne/src/regalloc/coalescing.rs | 9 +++--- lib/cretonne/src/regalloc/coloring.rs | 23 ++++++------- .../src/regalloc/live_value_tracker.rs | 6 ++-- lib/cretonne/src/regalloc/liveness.rs | 6 ++-- lib/cretonne/src/regalloc/liverange.rs | 2 +- lib/cretonne/src/regalloc/reload.rs | 10 +++--- lib/cretonne/src/regalloc/solver.rs | 12 +++---- lib/cretonne/src/regalloc/spilling.rs | 2 +- lib/cretonne/src/regalloc/virtregs.rs | 4 +-- lib/cretonne/src/stack_layout.rs | 4 +-- lib/cretonne/src/timing.rs | 2 +- lib/frontend/src/variable.rs | 4 +-- lib/wasm/src/code_translator.rs | 2 +- lib/wasm/src/func_translator.rs | 6 ++-- 43 files changed, 165 insertions(+), 164 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 2bd34a926c..ac8e338eda 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -211,7 +211,7 @@ def gen_instruction_data_impl(fmt): if f.has_value_list: fmt.line(n + ' { ref mut args, .. } => args,') fmt.line('_ => panic!("No value list: {:?}", self),') - fmt.line('assert!(args.is_empty(), "Value list already in use");') + fmt.line('debug_assert!(args.is_empty(), "Value list already in use");') fmt.line('*args = vlist;') diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 0f7b1f68e4..04c48ade51 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -348,7 +348,8 @@ def gen_xform(xform, fmt, type_sets): # Delete the original instruction if we didn't have an opportunity to # replace it. if not replace_inst: - fmt.line('assert_eq!(pos.remove_inst(), inst);') + fmt.line('let removed = pos.remove_inst();') + fmt.line('debug_assert_eq!(removed, inst);') fmt.line('return true;') diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index 0cd4e0e860..dfc2c2b70a 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -245,7 +245,7 @@ def gen_constructor(sgrp, parent, fmt): 'pub fn new({}) -> Flags {{'.format(args), '}'): fmt.line('let bvec = builder.state_for("{}");'.format(sgrp.name)) fmt.line('let mut bytes = [0; {}];'.format(sgrp.byte_size())) - fmt.line('assert_eq!(bvec.len(), {});'.format(sgrp.settings_size)) + fmt.line('debug_assert_eq!(bvec.len(), {});'.format(sgrp.settings_size)) with fmt.indented( 'for (i, b) in bvec.iter().enumerate() {', '}'): fmt.line('bytes[i] = *b;') diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index 05c96c79d5..801d4bfe9e 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -150,7 +150,7 @@ pub fn legalize_abi_value(have: Type, arg: &AbiParam) -> ValueConversion { match have_bits.cmp(&arg_bits) { // We have fewer bits than the ABI argument. Ordering::Less => { - assert!( + debug_assert!( have.is_int() && arg.value_type.is_int(), "Can only extend integer values" ); @@ -163,8 +163,8 @@ pub fn legalize_abi_value(have: Type, arg: &AbiParam) -> ValueConversion { // We have the same number of bits as the argument. Ordering::Equal => { // This must be an integer vector that is split and then extended. - assert!(arg.value_type.is_int()); - assert!(have.is_vector()); + debug_assert!(arg.value_type.is_int()); + debug_assert!(have.is_vector()); ValueConversion::VectorSplit } // We have more bits than the argument. diff --git a/lib/cretonne/src/bforest/path.rs b/lib/cretonne/src/bforest/path.rs index c0f9550645..1add8f6e63 100644 --- a/lib/cretonne/src/bforest/path.rs +++ b/lib/cretonne/src/bforest/path.rs @@ -316,7 +316,8 @@ impl Path { // Now that we have a not-full node, it must be possible to insert. match ins_node { None => { - assert!(pool[node].try_leaf_insert(entry, key, value)); + let inserted = pool[node].try_leaf_insert(entry, key, value); + debug_assert!(inserted); // If we inserted at the front of the new rhs_node leaf, we need to propagate // the inserted key as the critical key instead of the previous front key. if entry == 0 && node == rhs_node { @@ -324,7 +325,8 @@ impl Path { } } Some(n) => { - assert!(pool[node].try_inner_insert(entry, key, n)); + let inserted = pool[node].try_inner_insert(entry, key, n); + debug_assert!(inserted); // The lower level was moved to the new RHS node, so make sure that is // reflected here. if n == self.node[level + 1] { diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 713d25bed5..bca1f6fa4d 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -110,7 +110,7 @@ where let mut divert = RegDiversions::new(); for ebb in func.layout.ebbs() { divert.clear(); - assert_eq!(func.offsets[ebb], sink.offset()); + debug_assert_eq!(func.offsets[ebb], sink.offset()); for inst in func.layout.ebb_insts(ebb) { emit_inst(func, inst, &mut divert, sink); } diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index c6ea41f451..b0bf6196c6 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -60,7 +60,7 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> Result { // Somebody used a fall-through instruction before the branch relaxation pass. // Make sure it is correct, i.e. the destination is the layout successor. - assert_eq!(destination, succ, "Illegal fall-through in {}", ebb) + debug_assert_eq!(destination, succ, "Illegal fall-through in {}", ebb) } Opcode::Jump => { // If this is a jump to the successor EBB, change it to a fall-through. diff --git a/lib/cretonne/src/bitset.rs b/lib/cretonne/src/bitset.rs index 62e0286bb3..f212790be0 100644 --- a/lib/cretonne/src/bitset.rs +++ b/lib/cretonne/src/bitset.rs @@ -36,8 +36,8 @@ where /// Check if this BitSet contains the number num pub fn contains(&self, num: u8) -> bool { - assert!((num as usize) < Self::bits()); - assert!((num as usize) < Self::max_bits()); + debug_assert!((num as usize) < Self::bits()); + debug_assert!((num as usize) < Self::max_bits()); self.0.into() & (1 << num) != 0 } @@ -62,8 +62,8 @@ where /// Construct a BitSet with the half-open range [lo,hi) filled in pub fn from_range(lo: u8, hi: u8) -> Self { - assert!(lo <= hi); - assert!((hi as usize) <= Self::bits()); + debug_assert!(lo <= hi); + debug_assert!((hi as usize) <= Self::bits()); let one: T = T::from(1); // I can't just do (one << hi) - one here as the shift may overflow let hi_rng = if hi >= 1 { diff --git a/lib/cretonne/src/cursor.rs b/lib/cretonne/src/cursor.rs index 56ad657043..ed2528d5be 100644 --- a/lib/cretonne/src/cursor.rs +++ b/lib/cretonne/src/cursor.rs @@ -256,7 +256,7 @@ pub trait Cursor { /// Go to a specific instruction which must be inserted in the layout. /// New instructions will be inserted before `inst`. fn goto_inst(&mut self, inst: ir::Inst) { - assert!(self.layout().inst_ebb(inst).is_some()); + debug_assert!(self.layout().inst_ebb(inst).is_some()); self.set_position(CursorPosition::At(inst)); } @@ -287,14 +287,14 @@ pub trait Cursor { /// At this position, instructions cannot be inserted, but `next_inst()` will move to the first /// instruction in `ebb`. fn goto_top(&mut self, ebb: ir::Ebb) { - assert!(self.layout().is_ebb_inserted(ebb)); + debug_assert!(self.layout().is_ebb_inserted(ebb)); self.set_position(CursorPosition::Before(ebb)); } /// Go to the bottom of `ebb` which must be inserted into the layout. /// At this position, inserted instructions will be appended to `ebb`. fn goto_bottom(&mut self, ebb: ir::Ebb) { - assert!(self.layout().is_ebb_inserted(ebb)); + debug_assert!(self.layout().is_ebb_inserted(ebb)); self.set_position(CursorPosition::After(ebb)); } diff --git a/lib/cretonne/src/divconst_magic_numbers.rs b/lib/cretonne/src/divconst_magic_numbers.rs index 968ca67f5c..b046e22411 100644 --- a/lib/cretonne/src/divconst_magic_numbers.rs +++ b/lib/cretonne/src/divconst_magic_numbers.rs @@ -43,8 +43,8 @@ pub struct MS64 { // The actual "magic number" generators follow. pub fn magicU32(d: u32) -> MU32 { - assert_ne!(d, 0); - assert_ne!(d, 1); // d==1 generates out of range shifts. + debug_assert_ne!(d, 0); + debug_assert_ne!(d, 1); // d==1 generates out of range shifts. let mut do_add: bool = false; let mut p: i32 = 31; @@ -89,8 +89,8 @@ pub fn magicU32(d: u32) -> MU32 { } pub fn magicU64(d: u64) -> MU64 { - assert_ne!(d, 0); - assert_ne!(d, 1); // d==1 generates out of range shifts. + debug_assert_ne!(d, 0); + debug_assert_ne!(d, 1); // d==1 generates out of range shifts. let mut do_add: bool = false; let mut p: i32 = 63; @@ -135,9 +135,9 @@ pub fn magicU64(d: u64) -> MU64 { } pub fn magicS32(d: i32) -> MS32 { - assert_ne!(d, -1); - assert_ne!(d, 0); - assert_ne!(d, 1); + debug_assert_ne!(d, -1); + debug_assert_ne!(d, 0); + debug_assert_ne!(d, 1); let two31: u32 = 0x80000000u32; let mut p: i32 = 31; let ad: u32 = i32::wrapping_abs(d) as u32; @@ -178,9 +178,9 @@ pub fn magicS32(d: i32) -> MS32 { } pub fn magicS64(d: i64) -> MS64 { - assert_ne!(d, -1); - assert_ne!(d, 0); - assert_ne!(d, 1); + debug_assert_ne!(d, -1); + debug_assert_ne!(d, 0); + debug_assert_ne!(d, 1); let two63: u64 = 0x8000000000000000u64; let mut p: i32 = 63; let ad: u64 = i64::wrapping_abs(d) as u64; diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 27caf338dc..e643a42592 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -197,7 +197,7 @@ impl DominatorTree { } } - assert_eq!(a.0, b.0, "Unreachable block passed to common_dominator?"); + debug_assert_eq!(a.0, b.0, "Unreachable block passed to common_dominator?"); // We're in the same EBB. The common dominator is the earlier instruction. if layout.cmp(a.1, b.1) == Ordering::Less { @@ -241,7 +241,7 @@ impl DominatorTree { pub fn clear(&mut self) { self.nodes.clear(); self.postorder.clear(); - assert!(self.stack.is_empty()); + debug_assert!(self.stack.is_empty()); self.valid = false; } @@ -539,7 +539,7 @@ impl DominatorTreePreorder { /// Recompute this data structure to match `domtree`. pub fn compute(&mut self, domtree: &DominatorTree, layout: &Layout) { self.nodes.clear(); - assert_eq!(self.stack.len(), 0); + debug_assert_eq!(self.stack.len(), 0); // Step 1: Populate the child and sibling links. // @@ -557,7 +557,7 @@ impl DominatorTreePreorder { } // Step 2. Assign pre-order numbers from a DFS of the dominator tree. - assert!(self.stack.len() <= 1); + debug_assert!(self.stack.len() <= 1); let mut n = 0; while let Some(ebb) = self.stack.pop() { n += 1; diff --git a/lib/cretonne/src/entity/list.rs b/lib/cretonne/src/entity/list.rs index c9bb5b1a69..6c963bc377 100644 --- a/lib/cretonne/src/entity/list.rs +++ b/lib/cretonne/src/entity/list.rs @@ -219,8 +219,8 @@ impl ListPool { to_sclass: SizeClass, elems_to_copy: usize, ) -> usize { - assert!(elems_to_copy <= sclass_size(from_sclass)); - assert!(elems_to_copy <= sclass_size(to_sclass)); + debug_assert!(elems_to_copy <= sclass_size(from_sclass)); + debug_assert!(elems_to_copy <= sclass_size(to_sclass)); let new_block = self.alloc(to_sclass); if elems_to_copy > 0 { @@ -301,7 +301,7 @@ impl EntityList { pub fn clear(&mut self, pool: &mut ListPool) { let idx = self.index as usize; match pool.len_of(self) { - None => assert_eq!(idx, 0, "Invalid pool"), + None => debug_assert_eq!(idx, 0, "Invalid pool"), Some(len) => pool.free(idx - 1, sclass_for_length(len)), } // Switch back to the empty list representation which has no storage. @@ -322,7 +322,7 @@ impl EntityList { match pool.len_of(self) { None => { // This is an empty list. Allocate a block and set length=1. - assert_eq!(idx, 0, "Invalid pool"); + debug_assert_eq!(idx, 0, "Invalid pool"); let block = pool.alloc(sclass_for_length(1)); pool.data[block] = T::new(1); pool.data[block + 1] = element; @@ -358,7 +358,7 @@ impl EntityList { match pool.len_of(self) { None => { // This is an empty list. Allocate a block. - assert_eq!(idx, 0, "Invalid pool"); + debug_assert_eq!(idx, 0, "Invalid pool"); if count == 0 { return &mut []; } @@ -409,7 +409,7 @@ impl EntityList { } tail[0] = element; } else { - assert_eq!(index, seq.len()); + debug_assert_eq!(index, seq.len()); } } @@ -419,7 +419,7 @@ impl EntityList { { let seq = self.as_mut_slice(pool); len = seq.len(); - assert!(index < len); + debug_assert!(index < len); // Copy elements down. for i in index..len - 1 { @@ -449,7 +449,7 @@ impl EntityList { /// the list. pub fn swap_remove(&mut self, index: usize, pool: &mut ListPool) { let len = self.len(pool); - assert!(index < len); + debug_assert!(index < len); if index == len - 1 { self.remove(index, pool); } else { diff --git a/lib/cretonne/src/entity/mod.rs b/lib/cretonne/src/entity/mod.rs index 3cc8ec6f6d..6457e47f56 100644 --- a/lib/cretonne/src/entity/mod.rs +++ b/lib/cretonne/src/entity/mod.rs @@ -61,7 +61,7 @@ macro_rules! entity_impl { ($entity:ident) => { impl $crate::entity::EntityRef for $entity { fn new(index: usize) -> Self { - assert!(index < (::std::u32::MAX as usize)); + debug_assert!(index < (::std::u32::MAX as usize)); $entity(index as u32) } diff --git a/lib/cretonne/src/entity/sparse.rs b/lib/cretonne/src/entity/sparse.rs index dacbf66031..82e1b49546 100644 --- a/lib/cretonne/src/entity/sparse.rs +++ b/lib/cretonne/src/entity/sparse.rs @@ -149,7 +149,7 @@ where // There was no previous entry for `key`. Add it to the end of `dense`. let idx = self.dense.len(); - assert!(idx <= u32::MAX as usize, "SparseMap overflow"); + debug_assert!(idx <= u32::MAX as usize, "SparseMap overflow"); self.dense.push(value); self.sparse[key] = idx as u32; None diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 023d8751b6..ccd08f1bb5 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -217,11 +217,11 @@ impl DataFlowGraph { /// /// The `dest` value can't be attached to an instruction or EBB. pub fn change_to_alias(&mut self, dest: Value, src: Value) { - assert!(!self.value_is_attached(dest)); + debug_assert!(!self.value_is_attached(dest)); // Try to create short alias chains by finding the original source value. // This also avoids the creation of loops. let original = self.resolve_aliases(src); - assert_ne!( + debug_assert_ne!( dest, original, "Aliasing {} to {} would create a loop", @@ -229,7 +229,7 @@ impl DataFlowGraph { src ); let ty = self.value_type(original); - assert_eq!( + debug_assert_eq!( self.value_type(dest), ty, "Aliasing {} to {} would change its type {} to {}", @@ -273,7 +273,7 @@ impl DataFlowGraph { { let original = src; let ty = self.value_type(original); - assert_eq!( + debug_assert_eq!( self.value_type(dest), ty, "Aliasing {} to {} would change its type {} to {}", @@ -498,9 +498,9 @@ impl DataFlowGraph { /// This is a very low-level operation. Usually, instruction results with the correct types are /// created automatically. The `res` value must not be attached to anything else. pub fn attach_result(&mut self, inst: Inst, res: Value) { - assert!(!self.value_is_attached(res)); + debug_assert!(!self.value_is_attached(res)); let num = self.results[inst].push(res, &mut self.value_lists); - assert!(num <= u16::MAX as usize, "Too many result values"); + debug_assert!(num <= u16::MAX as usize, "Too many result values"); let ty = self.value_type(res); self.values[res] = ValueData::Inst { ty, @@ -533,7 +533,7 @@ impl DataFlowGraph { .expect("Replacing detached result"), new_value, ); - assert_eq!( + debug_assert_eq!( attached, old_value, "{} wasn't detached from {}", @@ -547,7 +547,7 @@ impl DataFlowGraph { pub fn append_result(&mut self, inst: Inst, ty: Type) -> Value { let res = self.values.next_key(); let num = self.results[inst].push(res, &mut self.value_lists); - assert!(num <= u16::MAX as usize, "Too many result values"); + debug_assert!(num <= u16::MAX as usize, "Too many result values"); self.make_value(ValueData::Inst { ty, inst, @@ -684,7 +684,7 @@ impl DataFlowGraph { pub fn append_ebb_param(&mut self, ebb: Ebb, ty: Type) -> Value { let param = self.values.next_key(); let num = self.ebbs[ebb].params.push(param, &mut self.value_lists); - assert!(num <= u16::MAX as usize, "Too many parameters on EBB"); + debug_assert!(num <= u16::MAX as usize, "Too many parameters on EBB"); self.make_value(ValueData::Param { ty, num: num as u16, @@ -761,9 +761,9 @@ impl DataFlowGraph { /// /// In almost all cases, you should be using `append_ebb_param()` instead of this method. pub fn attach_ebb_param(&mut self, ebb: Ebb, param: Value) { - assert!(!self.value_is_attached(param)); + debug_assert!(!self.value_is_attached(param)); let num = self.ebbs[ebb].params.push(param, &mut self.value_lists); - assert!(num <= u16::MAX as usize, "Too many parameters on EBB"); + debug_assert!(num <= u16::MAX as usize, "Too many parameters on EBB"); let ty = self.value_type(param); self.values[param] = ValueData::Param { ty, @@ -859,7 +859,7 @@ impl DataFlowGraph { /// to create invalid values for index padding which may be reassigned later. #[cold] fn set_value_type_for_parser(&mut self, v: Value, t: Type) { - debug_assert!( + assert!( self.value_type(v) == types::VOID, "this function is only for assigning types to previously invalid values" ); @@ -882,7 +882,7 @@ impl DataFlowGraph { ) -> usize { // Get the call signature if this is a function call. if let Some(sig) = self.call_signature(inst) { - debug_assert_eq!(self.insts[inst].opcode().constraints().fixed_results(), 0); + assert_eq!(self.insts[inst].opcode().constraints().fixed_results(), 0); for res_idx in 0..self.signatures[sig].returns.len() { let ty = self.signatures[sig].returns[res_idx].value_type; if let Some(v) = reuse.get(res_idx) { diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 69607754a2..2d3b1236ef 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -490,7 +490,7 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { significand <<= adjust; exponent -= i32::from(adjust); } - assert_eq!(significand >> t, 1); + debug_assert_eq!(significand >> t, 1); // Trailing significand excludes the high bit. let t_bits = significand & ((1 << t) - 1); diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index af570f7ad5..2d462064d1 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -560,7 +560,7 @@ impl OpcodeConstraints { /// Get the value type of result number `n`, having resolved the controlling type variable to /// `ctrl_type`. pub fn result_type(self, n: usize, ctrl_type: Type) -> Type { - assert!(n < self.fixed_results(), "Invalid result index"); + debug_assert!(n < self.fixed_results(), "Invalid result index"); if let ResolvedConstraint::Bound(t) = OPERAND_CONSTRAINTS[self.constraint_offset() + n].resolve(ctrl_type) { @@ -576,7 +576,7 @@ impl OpcodeConstraints { /// Unlike results, it is possible for some input values to vary freely within a specific /// `ValueTypeSet`. This is represented with the `ArgumentConstraint::Free` variant. pub fn value_argument_constraint(self, n: usize, ctrl_type: Type) -> ResolvedConstraint { - assert!( + debug_assert!( n < self.fixed_value_arguments(), "Invalid value argument index" ); diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index e18ffd7a33..de117b3ed8 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -88,7 +88,7 @@ const LOCAL_LIMIT: SequenceNumber = 100 * MINOR_STRIDE; // Compute the midpoint between `a` and `b`. // Return `None` if the midpoint would be equal to either. fn midpoint(a: SequenceNumber, b: SequenceNumber) -> Option { - assert!(a < b); + debug_assert!(a < b); // Avoid integer overflow. let m = a + (b - a) / 2; if m > a { Some(m) } else { None } @@ -148,7 +148,7 @@ impl Layout { /// Assign a valid sequence number to `ebb` such that the numbers are still monotonic. This may /// require renumbering. fn assign_ebb_seq(&mut self, ebb: Ebb) { - assert!(self.is_ebb_inserted(ebb)); + debug_assert!(self.is_ebb_inserted(ebb)); // Get the sequence number immediately before `ebb`, or 0. let prev_seq = self.ebbs[ebb] @@ -334,13 +334,13 @@ impl Layout { /// Insert `ebb` as the last EBB in the layout. pub fn append_ebb(&mut self, ebb: Ebb) { - assert!( + debug_assert!( !self.is_ebb_inserted(ebb), "Cannot append EBB that is already in the layout" ); { let node = &mut self.ebbs[ebb]; - assert!(node.first_inst.is_none() && node.last_inst.is_none()); + debug_assert!(node.first_inst.is_none() && node.last_inst.is_none()); node.prev = self.last_ebb.into(); node.next = None.into(); } @@ -355,11 +355,11 @@ impl Layout { /// Insert `ebb` in the layout before the existing EBB `before`. pub fn insert_ebb(&mut self, ebb: Ebb, before: Ebb) { - assert!( + debug_assert!( !self.is_ebb_inserted(ebb), "Cannot insert EBB that is already in the layout" ); - assert!( + debug_assert!( self.is_ebb_inserted(before), "EBB Insertion point not in the layout" ); @@ -379,11 +379,11 @@ impl Layout { /// Insert `ebb` in the layout *after* the existing EBB `after`. pub fn insert_ebb_after(&mut self, ebb: Ebb, after: Ebb) { - assert!( + debug_assert!( !self.is_ebb_inserted(ebb), "Cannot insert EBB that is already in the layout" ); - assert!( + debug_assert!( self.is_ebb_inserted(after), "EBB Insertion point not in the layout" ); @@ -403,8 +403,8 @@ impl Layout { /// Remove `ebb` from the layout. pub fn remove_ebb(&mut self, ebb: Ebb) { - assert!(self.is_ebb_inserted(ebb), "EBB not in the layout"); - assert!(self.first_inst(ebb).is_none(), "EBB must be empty."); + debug_assert!(self.is_ebb_inserted(ebb), "EBB not in the layout"); + debug_assert!(self.first_inst(ebb).is_none(), "EBB must be empty."); // Clear the `ebb` node and extract links. let prev; @@ -521,8 +521,8 @@ impl Layout { /// Append `inst` to the end of `ebb`. pub fn append_inst(&mut self, inst: Inst, ebb: Ebb) { - assert_eq!(self.inst_ebb(inst), None); - assert!( + debug_assert_eq!(self.inst_ebb(inst), None); + debug_assert!( self.is_ebb_inserted(ebb), "Cannot append instructions to EBB not in layout" ); @@ -532,7 +532,7 @@ impl Layout { let inst_node = &mut self.insts[inst]; inst_node.ebb = ebb.into(); inst_node.prev = ebb_node.last_inst; - assert!(inst_node.next.is_none()); + debug_assert!(inst_node.next.is_none()); } if ebb_node.first_inst.is_none() { ebb_node.first_inst = inst.into(); @@ -566,7 +566,7 @@ impl Layout { /// Insert `inst` before the instruction `before` in the same EBB. pub fn insert_inst(&mut self, inst: Inst, before: Inst) { - assert_eq!(self.inst_ebb(inst), None); + debug_assert_eq!(self.inst_ebb(inst), None); let ebb = self.inst_ebb(before).expect( "Instruction before insertion point not in the layout", ); @@ -645,7 +645,7 @@ impl Layout { let old_ebb = self.inst_ebb(before).expect( "The `before` instruction must be in the layout", ); - assert!(!self.is_ebb_inserted(new_ebb)); + debug_assert!(!self.is_ebb_inserted(new_ebb)); // Insert new_ebb after old_ebb. let next_ebb = self.ebbs[old_ebb].next; diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index 480160ea96..b7b2994c3b 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -19,7 +19,7 @@ pub struct ProgramPoint(u32); impl From for ProgramPoint { fn from(inst: Inst) -> ProgramPoint { let idx = inst.index(); - assert!(idx < (u32::MAX / 2) as usize); + debug_assert!(idx < (u32::MAX / 2) as usize); ProgramPoint((idx * 2) as u32) } } @@ -27,7 +27,7 @@ impl From for ProgramPoint { impl From for ProgramPoint { fn from(ebb: Ebb) -> ProgramPoint { let idx = ebb.index(); - assert!(idx < (u32::MAX / 2) as usize); + debug_assert!(idx < (u32::MAX / 2) as usize); ProgramPoint((idx * 2 + 1) as u32) } } diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index d3578a7570..ebae8e2391 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -244,7 +244,7 @@ impl StackSlots { /// Create a stack slot representing an incoming function argument. pub fn make_incoming_arg(&mut self, ty: Type, offset: StackOffset) -> StackSlot { let mut data = StackSlotData::new(StackSlotKind::IncomingArg, ty.bytes()); - assert!(offset <= StackOffset::max_value() - data.size as StackOffset); + debug_assert!(offset <= StackOffset::max_value() - data.size as StackOffset); data.offset = Some(offset); self.push(data) } @@ -269,7 +269,7 @@ impl StackSlots { // No existing slot found. Make one and insert it into `outgoing`. let mut data = StackSlotData::new(StackSlotKind::OutgoingArg, size); - assert!(offset <= StackOffset::max_value() - size as StackOffset); + debug_assert!(offset <= StackOffset::max_value() - size as StackOffset); data.offset = Some(offset); let ss = self.slots.push(data); self.outgoing.insert(inspos, ss); diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 2296a9292b..0a14566b50 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -107,7 +107,7 @@ impl ArgAssigner for Args { // Assign a stack location. let loc = ArgumentLoc::Stack(self.offset as i32); self.offset += self.pointer_bytes; - assert!(self.offset <= i32::MAX as u32); + debug_assert!(self.offset <= i32::MAX as u32); loc.into() } } diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index 52e715be43..1db377b0ef 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -80,7 +80,7 @@ impl ArgAssigner for Args { // Assign a stack location. let loc = ArgumentLoc::Stack(self.offset as i32); self.offset += self.pointer_bytes; - assert!(self.offset <= i32::MAX as u32); + debug_assert!(self.offset <= i32::MAX as u32); loc.into() } } diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 369a9bf09f..8cc1ef17d1 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -133,7 +133,7 @@ fn put_sb(bits: u16, imm: i64, rs1: RegUnit, rs2: RegUnit let rs1 = u32::from(rs1) & 0x1f; let rs2 = u32::from(rs2) & 0x1f; - assert!(is_signed_int(imm, 13, 1), "SB out of range {:#x}", imm); + debug_assert!(is_signed_int(imm, 13, 1), "SB out of range {:#x}", imm); let imm = imm as u32; // 0-6: opcode @@ -164,7 +164,7 @@ fn put_uj(bits: u16, imm: i64, rd: RegUnit, sink: &mut CS let opcode5 = bits & 0x1f; let rd = u32::from(rd) & 0x1f; - assert!(is_signed_int(imm, 21, 1), "UJ out of range {:#x}", imm); + debug_assert!(is_signed_int(imm, 21, 1), "UJ out of range {:#x}", imm); let imm = imm as u32; // 0-6: opcode diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 9ea50b202d..29af50b3ad 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -85,15 +85,15 @@ fn legalize_entry_params(func: &mut Function, entry: Ebb) { ArgumentPurpose::FramePointer => {} ArgumentPurpose::CalleeSaved => {} ArgumentPurpose::StructReturn => { - assert!(!has_sret, "Multiple sret arguments found"); + debug_assert!(!has_sret, "Multiple sret arguments found"); has_sret = true; } ArgumentPurpose::VMContext => { - assert!(!has_vmctx, "Multiple vmctx arguments found"); + debug_assert!(!has_vmctx, "Multiple vmctx arguments found"); has_vmctx = true; } ArgumentPurpose::SignatureId => { - assert!(!has_sigid, "Multiple sigid arguments found"); + debug_assert!(!has_sigid, "Multiple sigid arguments found"); has_sigid = true; } _ => panic!("Unexpected special-purpose arg {}", abi_type), @@ -103,7 +103,7 @@ fn legalize_entry_params(func: &mut Function, entry: Ebb) { // Compute the value we want for `arg` from the legalized ABI parameters. let mut get_arg = |func: &mut Function, ty| { let abi_type = func.signature.params[abi_arg]; - assert_eq!( + debug_assert_eq!( abi_type.purpose, ArgumentPurpose::Normal, "Can't legalize special-purpose argument" @@ -118,7 +118,7 @@ fn legalize_entry_params(func: &mut Function, entry: Ebb) { let converted = convert_from_abi(&mut pos, arg_type, Some(arg), &mut get_arg); // The old `arg` is no longer an attached EBB argument, but there are probably still // uses of the value. - assert_eq!(pos.func.dfg.resolve_aliases(arg), converted); + debug_assert_eq!(pos.func.dfg.resolve_aliases(arg), converted); } } @@ -138,19 +138,19 @@ fn legalize_entry_params(func: &mut Function, entry: Ebb) { } // These can be meaningfully added by `legalize_signature()`. ArgumentPurpose::Link => { - assert!(!has_link, "Multiple link parameters found"); + debug_assert!(!has_link, "Multiple link parameters found"); has_link = true; } ArgumentPurpose::StructReturn => { - assert!(!has_sret, "Multiple sret parameters found"); + debug_assert!(!has_sret, "Multiple sret parameters found"); has_sret = true; } ArgumentPurpose::VMContext => { - assert!(!has_vmctx, "Multiple vmctx parameters found"); + debug_assert!(!has_vmctx, "Multiple vmctx parameters found"); has_vmctx = true; } ArgumentPurpose::SignatureId => { - assert!(!has_sigid, "Multiple sigid parameters found"); + debug_assert!(!has_sigid, "Multiple sigid parameters found"); has_sigid = true; } } @@ -180,7 +180,7 @@ where // We theoretically allow for call instructions that return a number of fixed results before // the call return values. In practice, it doesn't happen. let fixed_results = pos.func.dfg[call].opcode().constraints().fixed_results(); - assert_eq!(fixed_results, 0, "Fixed results on calls not supported"); + debug_assert_eq!(fixed_results, 0, "Fixed results on calls not supported"); let results = pos.func.dfg.detach_results(call); let mut next_res = 0; @@ -209,7 +209,7 @@ where } }; let v = convert_from_abi(pos, res_type, Some(res), &mut get_res); - assert_eq!(pos.func.dfg.resolve_aliases(res), v); + debug_assert_eq!(pos.func.dfg.resolve_aliases(res), v); } } @@ -238,7 +238,7 @@ where let arg_type = match get_arg(pos.func, ty) { Ok(v) => { debug_assert_eq!(pos.func.dfg.value_type(v), ty); - assert_eq!(into_result, None); + debug_assert_eq!(into_result, None); return v; } Err(t) => t, @@ -274,7 +274,7 @@ where } // Construct a `ty` by bit-casting from an integer type. ValueConversion::IntBits => { - assert!(!ty.is_int()); + debug_assert!(!ty.is_int()); let abi_ty = Type::int(ty.bits()).expect("Invalid type for conversion"); let arg = convert_from_abi(pos, abi_ty, None, get_arg); pos.ins().with_results([into_result]).bitcast(ty, arg) @@ -340,7 +340,7 @@ fn convert_to_abi( convert_to_abi(pos, cfg, hi, put_arg); } ValueConversion::IntBits => { - assert!(!ty.is_int()); + debug_assert!(!ty.is_int()); let abi_ty = Type::int(ty.bits()).expect("Invalid type for conversion"); let arg = pos.ins().bitcast(abi_ty, value); convert_to_abi(pos, cfg, arg, put_arg); @@ -555,7 +555,7 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph legalize_inst_arguments(pos, cfg, abi_args, |func, abi_arg| { func.signature.returns[abi_arg] }); - assert_eq!(pos.func.dfg.inst_variable_args(inst).len(), abi_args); + debug_assert_eq!(pos.func.dfg.inst_variable_args(inst).len(), abi_args); // Append special return arguments for any `sret`, `link`, and `vmctx` return values added to // the legalized signature. These values should simply be propagated from the entry block diff --git a/lib/cretonne/src/legalizer/globalvar.rs b/lib/cretonne/src/legalizer/globalvar.rs index b945923f2f..5f6427822b 100644 --- a/lib/cretonne/src/legalizer/globalvar.rs +++ b/lib/cretonne/src/legalizer/globalvar.rs @@ -18,7 +18,7 @@ pub fn expand_global_addr( // Unpack the instruction. let gv = match func.dfg[inst] { ir::InstructionData::UnaryGlobalVar { opcode, global_var } => { - assert_eq!(opcode, ir::Opcode::GlobalAddr); + debug_assert_eq!(opcode, ir::Opcode::GlobalAddr); global_var } _ => panic!("Wanted global_addr: {}", func.dfg.display_inst(inst, None)), diff --git a/lib/cretonne/src/legalizer/heap.rs b/lib/cretonne/src/legalizer/heap.rs index af40181d1a..37ce226351 100644 --- a/lib/cretonne/src/legalizer/heap.rs +++ b/lib/cretonne/src/legalizer/heap.rs @@ -24,7 +24,7 @@ pub fn expand_heap_addr( arg, imm, } => { - assert_eq!(opcode, ir::Opcode::HeapAddr); + debug_assert_eq!(opcode, ir::Opcode::HeapAddr); (heap, arg, imm.into()) } _ => panic!("Wanted heap_addr: {}", func.dfg.display_inst(inst, None)), diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 189a3e735d..51198d05a0 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -248,7 +248,7 @@ fn expand_fconst( _isa: &TargetIsa, ) { let ty = func.dfg.value_type(func.dfg.first_result(inst)); - assert!(!ty.is_vector(), "Only scalar fconst supported: {}", ty); + debug_assert!(!ty.is_vector(), "Only scalar fconst supported: {}", ty); // In the future, we may want to generate constant pool entries for these constants, but for // now use an `iconst` and a bit cast. diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index 0c73e7c2d4..aafc050c26 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -127,7 +127,7 @@ fn split_any( while let Some(repair) = repairs.pop() { for (_, inst) in cfg.pred_iter(repair.ebb) { let branch_opc = pos.func.dfg[inst].opcode(); - assert!( + debug_assert!( branch_opc.is_branch(), "Predecessor not a branch: {}", pos.func.dfg.display_inst(inst, None) @@ -198,7 +198,7 @@ fn split_value( // This is an instruction result. See if the value was created by a `concat` // instruction. if let InstructionData::Binary { opcode, args, .. } = pos.func.dfg[inst] { - assert_eq!(num, 0); + debug_assert_eq!(num, 0); if opcode == concat { reuse = Some((args[0], args[1])); } diff --git a/lib/cretonne/src/preopt.rs b/lib/cretonne/src/preopt.rs index 1fdee0ad80..8e4bef8da3 100644 --- a/lib/cretonne/src/preopt.rs +++ b/lib/cretonne/src/preopt.rs @@ -192,10 +192,10 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso // U32 div, rem by a power-of-2 &DivRemByConstInfo::DivU32(n1, d) | &DivRemByConstInfo::RemU32(n1, d) if d.is_power_of_two() => { - assert!(d >= 2); + debug_assert!(d >= 2); // compute k where d == 2^k let k = d.trailing_zeros(); - assert!(k >= 1 && k <= 31); + debug_assert!(k >= 1 && k <= 31); if isRem { let mask = (1u64 << k) - 1; pos.func.dfg.replace(inst).band_imm(n1, mask as i64); @@ -207,7 +207,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso // U32 div, rem by non-power-of-2 &DivRemByConstInfo::DivU32(n1, d) | &DivRemByConstInfo::RemU32(n1, d) => { - assert!(d >= 3); + debug_assert!(d >= 3); let MU32 { mulBy, doAdd, @@ -217,7 +217,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso let q0 = pos.ins().iconst(I32, mulBy as i64); let q1 = pos.ins().umulhi(n1, q0); if doAdd { - assert!(shiftBy >= 1 && shiftBy <= 32); + debug_assert!(shiftBy >= 1 && shiftBy <= 32); let t1 = pos.ins().isub(n1, q1); let t2 = pos.ins().ushr_imm(t1, 1); let t3 = pos.ins().iadd(t2, q1); @@ -226,7 +226,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso debug_assert!(shiftBy != 1); qf = pos.ins().ushr_imm(t3, (shiftBy - 1) as i64); } else { - assert!(shiftBy >= 0 && shiftBy <= 31); + debug_assert!(shiftBy >= 0 && shiftBy <= 31); // Whereas there are known cases here for shiftBy == 0. if shiftBy > 0 { qf = pos.ins().ushr_imm(q1, shiftBy as i64); @@ -264,10 +264,10 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso // U64 div, rem by a power-of-2 &DivRemByConstInfo::DivU64(n1, d) | &DivRemByConstInfo::RemU64(n1, d) if d.is_power_of_two() => { - assert!(d >= 2); + debug_assert!(d >= 2); // compute k where d == 2^k let k = d.trailing_zeros(); - assert!(k >= 1 && k <= 63); + debug_assert!(k >= 1 && k <= 63); if isRem { let mask = (1u64 << k) - 1; pos.func.dfg.replace(inst).band_imm(n1, mask as i64); @@ -279,7 +279,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso // U64 div, rem by non-power-of-2 &DivRemByConstInfo::DivU64(n1, d) | &DivRemByConstInfo::RemU64(n1, d) => { - assert!(d >= 3); + debug_assert!(d >= 3); let MU64 { mulBy, doAdd, @@ -289,7 +289,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso let q0 = pos.ins().iconst(I64, mulBy as i64); let q1 = pos.ins().umulhi(n1, q0); if doAdd { - assert!(shiftBy >= 1 && shiftBy <= 64); + debug_assert!(shiftBy >= 1 && shiftBy <= 64); let t1 = pos.ins().isub(n1, q1); let t2 = pos.ins().ushr_imm(t1, 1); let t3 = pos.ins().iadd(t2, q1); @@ -298,7 +298,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso debug_assert!(shiftBy != 1); qf = pos.ins().ushr_imm(t3, (shiftBy - 1) as i64); } else { - assert!(shiftBy >= 0 && shiftBy <= 63); + debug_assert!(shiftBy >= 0 && shiftBy <= 63); // Whereas there are known cases here for shiftBy == 0. if shiftBy > 0 { qf = pos.ins().ushr_imm(q1, shiftBy as i64); @@ -339,7 +339,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso &DivRemByConstInfo::RemS32(n1, d) => { if let Some((isNeg, k)) = isPowerOf2_S32(d) { // k can be 31 only in the case that d is -2^31. - assert!(k >= 1 && k <= 31); + debug_assert!(k >= 1 && k <= 31); let t1 = if k - 1 == 0 { n1 } else { @@ -363,7 +363,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } } else { // S32 div, rem by a non-power-of-2 - assert!(d < -2 || d > 2); + debug_assert!(d < -2 || d > 2); let MS32 { mulBy, shiftBy } = magicS32(d); let q0 = pos.ins().iconst(I32, mulBy as i64); let q1 = pos.ins().smulhi(n1, q0); @@ -374,7 +374,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } else { q1 }; - assert!(shiftBy >= 0 && shiftBy <= 31); + debug_assert!(shiftBy >= 0 && shiftBy <= 31); let q3 = if shiftBy == 0 { q2 } else { @@ -416,7 +416,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso &DivRemByConstInfo::RemS64(n1, d) => { if let Some((isNeg, k)) = isPowerOf2_S64(d) { // k can be 63 only in the case that d is -2^63. - assert!(k >= 1 && k <= 63); + debug_assert!(k >= 1 && k <= 63); let t1 = if k - 1 == 0 { n1 } else { @@ -440,7 +440,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } } else { // S64 div, rem by a non-power-of-2 - assert!(d < -2 || d > 2); + debug_assert!(d < -2 || d > 2); let MS64 { mulBy, shiftBy } = magicS64(d); let q0 = pos.ins().iconst(I64, mulBy); let q1 = pos.ins().smulhi(n1, q0); @@ -451,7 +451,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } else { q1 }; - assert!(shiftBy >= 0 && shiftBy <= 63); + debug_assert!(shiftBy >= 0 && shiftBy <= 63); let q3 = if shiftBy == 0 { q2 } else { diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 1dc06c6c2b..10d0df2e62 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -238,7 +238,7 @@ impl<'a> Context<'a> { // 1. It is defined in a dominating EBB and live-in to `ebb`. // 2. If is itself a parameter value for `ebb`. This case should already have been // eliminated by `isolate_conflicting_params()`. - assert!( + debug_assert!( lr.def() != ebb.into(), "{} parameter {} was missed by isolate_conflicting_params()", ebb, @@ -494,8 +494,8 @@ impl<'a> Context<'a> { // Second everything else in reverse layout order. Again, short forward branches get merged // first. There can also be backwards branches mixed in here, though, as long as they are // not loop backedges. - assert!(self.predecessors.is_empty()); - assert!(self.backedges.is_empty()); + debug_assert!(self.predecessors.is_empty()); + debug_assert!(self.backedges.is_empty()); for (pred_ebb, pred_inst) in self.cfg.pred_iter(ebb) { if self.preorder.dominates(ebb, pred_ebb) { self.backedges.push(pred_inst); @@ -957,7 +957,8 @@ impl VirtualCopies { /// Indicate that `param` is now fully merged. pub fn merged_param(&mut self, param: Value, func: &Function) { - assert_eq!(self.params.pop(), Some(param)); + let popped = self.params.pop(); + debug_assert_eq!(popped, Some(param)); // The domtree pre-order in `self.params` guarantees that all parameters defined at the // same EBB will be adjacent. This means we can see when all parameters at an EBB have been diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index fb60a979c8..2de41e67c6 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -246,7 +246,7 @@ impl<'a> Context<'a> { /// Return the set of remaining allocatable registers after filtering out the dead arguments. fn color_entry_params(&mut self, args: &[LiveValue]) -> AvailableRegs { let sig = &self.cur.func.signature; - assert_eq!(sig.params.len(), args.len()); + debug_assert_eq!(sig.params.len(), args.len()); let mut regs = AvailableRegs::new(&self.usable_regs); @@ -271,7 +271,7 @@ impl<'a> Context<'a> { } // The spiller will have assigned an incoming stack slot already. - Affinity::Stack => assert!(abi.location.is_stack()), + Affinity::Stack => debug_assert!(abi.location.is_stack()), // This is a ghost value, unused in the function. Don't assign it to a location // either. Affinity::None => {} @@ -340,7 +340,7 @@ impl<'a> Context<'a> { } else { // This is a multi-way branch like `br_table`. We only support arguments on // single-destination branches. - assert_eq!( + debug_assert_eq!( self.cur.func.dfg.inst_variable_args(inst).len(), 0, "Can't handle EBB arguments: {}", @@ -586,7 +586,7 @@ impl<'a> Context<'a> { // Now handle the EBB arguments. let br_args = self.cur.func.dfg.inst_variable_args(inst); let dest_args = self.cur.func.dfg.ebb_params(dest); - assert_eq!(br_args.len(), dest_args.len()); + debug_assert_eq!(br_args.len(), dest_args.len()); for (&dest_arg, &br_arg) in dest_args.iter().zip(br_args) { // The first time we encounter a branch to `dest`, we get to pick the location. The // following times we see a branch to `dest`, we must follow suit. @@ -631,7 +631,7 @@ impl<'a> Context<'a> { fn color_ebb_params(&mut self, inst: Inst, dest: Ebb) { let br_args = self.cur.func.dfg.inst_variable_args(inst); let dest_args = self.cur.func.dfg.ebb_params(dest); - assert_eq!(br_args.len(), dest_args.len()); + debug_assert_eq!(br_args.len(), dest_args.len()); for (&dest_arg, &br_arg) in dest_args.iter().zip(br_args) { match self.cur.func.locations[dest_arg] { ValueLoc::Unassigned => { @@ -741,7 +741,7 @@ impl<'a> Context<'a> { // It's technically possible for a call instruction to have fixed results before the // variable list of results, but we have no known instances of that. // Just assume all results are variable return values. - assert_eq!(defs.len(), self.cur.func.dfg.signatures[sig].returns.len()); + debug_assert_eq!(defs.len(), self.cur.func.dfg.signatures[sig].returns.len()); for (i, lv) in defs.iter().enumerate() { let abi = self.cur.func.dfg.signatures[sig].returns[i]; if let ArgumentLoc::Reg(reg) = abi.location { @@ -787,7 +787,7 @@ impl<'a> Context<'a> { } let ok = self.solver.add_fixed_output(rc, reg); - assert!(ok, "Couldn't clear fixed output interference for {}", value); + debug_assert!(ok, "Couldn't clear fixed output interference for {}", value); } self.cur.func.locations[value] = ValueLoc::Reg(reg); } @@ -858,11 +858,8 @@ impl<'a> Context<'a> { Ok(regs) => return regs, Err(SolverError::Divert(rc)) => { // Do we have any live-through `rc` registers that are not already variables? - assert!( - self.try_add_var(rc, throughs), - "Ran out of registers in {}", - rc - ); + let added = self.try_add_var(rc, throughs); + debug_assert!(added, "Ran out of registers in {}", rc); } Err(SolverError::Global(value)) => { dbg!("Not enough global registers for {}, trying as local", value); @@ -941,7 +938,7 @@ impl<'a> Context<'a> { // It is very unlikely (impossible?) that we would need more than one spill per top-level // register class, so avoid allocation by using a fixed array here. let mut slot = [PackedOption::default(); 8]; - assert!(spills <= slot.len(), "Too many spills ({})", spills); + debug_assert!(spills <= slot.len(), "Too many spills ({})", spills); for m in self.solver.moves() { match *m { diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 3d7eeaa6e5..f53b5ec8bd 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -207,7 +207,7 @@ impl LiveValueTracker { let first_arg = self.live.values.len(); for &value in dfg.ebb_params(ebb) { let lr = &liveness[value]; - assert_eq!(lr.def(), ebb.into()); + debug_assert_eq!(lr.def(), ebb.into()); match lr.def_local_end().into() { ExpandedProgramPoint::Inst(endpoint) => { self.live.push(value, endpoint, lr); @@ -215,7 +215,7 @@ impl LiveValueTracker { ExpandedProgramPoint::Ebb(local_ebb) => { // This is a dead EBB parameter which is not even live into the first // instruction in the EBB. - assert_eq!( + debug_assert_eq!( local_ebb, ebb, "EBB parameter live range ends at wrong EBB header" @@ -273,7 +273,7 @@ impl LiveValueTracker { let first_def = self.live.values.len(); for &value in dfg.inst_results(inst) { let lr = &liveness[value]; - assert_eq!(lr.def(), inst.into()); + debug_assert_eq!(lr.def(), inst.into()); match lr.def_local_end().into() { ExpandedProgramPoint::Inst(endpoint) => { self.live.push(value, endpoint, lr); diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 2778f47f63..11e7d8cc3d 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -251,7 +251,7 @@ fn extend_to_use( forest: &mut LiveRangeForest, ) { // This is our scratch working space, and we'll leave it empty when we return. - assert!(worklist.is_empty()); + debug_assert!(worklist.is_empty()); // Extend the range locally in `ebb`. // If there already was a live interval in that block, we're done. @@ -338,7 +338,7 @@ impl Liveness { let old = self.ranges.insert( LiveRange::new(value, def.into(), affinity), ); - assert!(old.is_none(), "{} already has a live range", value); + debug_assert!(old.is_none(), "{} already has a live range", value); } /// Move the definition of `value` to `def`. @@ -367,7 +367,7 @@ impl Liveness { debug_assert_eq!(Some(ebb), layout.inst_ebb(user)); let lr = self.ranges.get_mut(value).expect("Value has no live range"); let livein = lr.extend_in_ebb(ebb, user, layout, &mut self.forest); - assert!(!livein, "{} should already be live in {}", value, ebb); + debug_assert!(!livein, "{} should already be live in {}", value, ebb); &mut lr.affinity } diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 7d7d6a8775..9462d32ad0 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -253,7 +253,7 @@ impl GenLiveRange { order.cmp(to, self.def_begin) != Ordering::Less { let to_pp = to.into(); - assert_ne!( + debug_assert_ne!( to_pp, self.def_begin, "Can't use value in the defining instruction." diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 12fd8d310a..2967081807 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -145,7 +145,7 @@ impl<'a> Context<'a> { ); if self.cur.func.layout.entry_block() == Some(ebb) { - assert_eq!(liveins.len(), 0); + debug_assert_eq!(liveins.len(), 0); self.visit_entry_params(ebb, args); } else { self.visit_ebb_params(ebb, args); @@ -155,7 +155,7 @@ impl<'a> Context<'a> { /// Visit the parameters on the entry block. /// These values have ABI constraints from the function signature. fn visit_entry_params(&mut self, ebb: Ebb, args: &[LiveValue]) { - assert_eq!(self.cur.func.signature.params.len(), args.len()); + debug_assert_eq!(self.cur.func.signature.params.len(), args.len()); self.cur.goto_first_inst(ebb); for (arg_idx, arg) in args.iter().enumerate() { @@ -175,7 +175,7 @@ impl<'a> Context<'a> { } } ArgumentLoc::Stack(_) => { - assert!(arg.affinity.is_stack()); + debug_assert!(arg.affinity.is_stack()); } ArgumentLoc::Unassigned => panic!("Unexpected ABI location"), } @@ -203,7 +203,7 @@ impl<'a> Context<'a> { ); // Identify reload candidates. - assert!(self.candidates.is_empty()); + debug_assert!(self.candidates.is_empty()); self.find_candidates(inst, constraints); // Insert fill instructions before `inst` and replace `cand.value` with the filled value. @@ -375,7 +375,7 @@ fn handle_abi_args( isa: &TargetIsa, liveness: &Liveness, ) { - assert_eq!(abi_types.len(), var_args.len()); + debug_assert_eq!(abi_types.len(), var_args.len()); for ((abi, &arg), argidx) in abi_types.iter().zip(var_args).zip(offset..) { if abi.location.is_reg() { let lv = liveness.get(arg).expect("Missing live range for ABI arg"); diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index a3b732799c..0e68099a67 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -565,7 +565,7 @@ impl Solver { dbg!("-> converting variable {} to a fixed constraint", v); // The spiller is responsible for ensuring that all constraints on the uses of a // value are compatible. - assert!( + debug_assert!( v.constraint.contains(to), "Incompatible constraints for {}", value @@ -665,7 +665,7 @@ impl Solver { // No variable, then it must be a fixed reassignment. if let Some(a) = self.assignments.get(value) { dbg!("-> already fixed assignment {}", a); - assert!( + debug_assert!( constraint.contains(a.to), "Incompatible constraints for {}", value @@ -708,7 +708,7 @@ impl Solver { /// Call this method to indicate that there will be no more fixed input reassignments added /// and prepare for the output side constraints. pub fn inputs_done(&mut self) { - assert!(!self.has_fixed_input_conflicts()); + debug_assert!(!self.has_fixed_input_conflicts()); // At this point, `regs_out` contains the `to` side of the input reassignments, and the // `from` side has already been marked as available in `regs_in`. @@ -746,7 +746,7 @@ impl Solver { // interference constraints on the output side. // Variables representing tied operands will get their `is_output` flag set again later. if let Some(v) = self.vars.iter_mut().find(|v| v.value == value) { - assert!(v.is_input); + debug_assert!(v.is_input); v.is_output = false; return; } @@ -782,7 +782,7 @@ impl Solver { // Check if a variable was created. if let Some(v) = self.vars.iter_mut().find(|v| v.value == value) { - assert!(v.is_input); + debug_assert!(v.is_input); v.is_output = true; v.is_global = is_global; return None; @@ -1026,7 +1026,7 @@ impl Solver { /// Returns the number of spills that had to be emitted. pub fn schedule_moves(&mut self, regs: &AllocatableSet) -> usize { self.collect_moves(); - assert!(self.fills.is_empty()); + debug_assert!(self.fills.is_empty()); let mut num_spill_slots = 0; let mut avail = regs.clone(); diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 84ef76fdff..15739cd260 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -242,7 +242,7 @@ impl<'a> Context<'a> { debug_assert_eq!(self.cur.current_ebb(), Some(ebb)); // We may need to resolve register constraints if there are any noteworthy uses. - assert!(self.reg_uses.is_empty()); + debug_assert!(self.reg_uses.is_empty()); self.collect_reg_uses(inst, ebb, constraints); // Calls usually have fixed register uses. diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index 5116e8f02a..fca213a62b 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -140,7 +140,7 @@ impl VirtRegs { func: &Function, preorder: &DominatorTreePreorder, ) -> VirtReg { - assert_eq!(self.get(single), None, "Expected singleton {}", single); + debug_assert_eq!(self.get(single), None, "Expected singleton {}", single); // Make sure `big` has a vreg. let vreg = self.get(big).unwrap_or_else(|| { @@ -208,7 +208,7 @@ impl VirtRegs { } } - assert_eq!( + debug_assert_eq!( values.len(), singletons + cleared, "Can't unify partial virtual registers" diff --git a/lib/cretonne/src/stack_layout.rs b/lib/cretonne/src/stack_layout.rs index 16dcea8d5f..f1cf35f150 100644 --- a/lib/cretonne/src/stack_layout.rs +++ b/lib/cretonne/src/stack_layout.rs @@ -19,7 +19,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result Result Self { - assert!(index < u32::MAX); + debug_assert!(index < u32::MAX); Variable(index) } } impl EntityRef for Variable { fn new(index: usize) -> Self { - assert!(index < (u32::MAX as usize)); + debug_assert!(index < (u32::MAX as usize)); Variable(index as u32) } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 60b432a582..5535a1555e 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -1026,7 +1026,7 @@ fn get_heap_addr( use std::cmp::min; let guard_size: i64 = builder.func.heaps[heap].guard_size.into(); - assert!(guard_size > 0, "Heap guard pages currently required"); + debug_assert!(guard_size > 0, "Heap guard pages currently required"); // Generate `heap_addr` instructions that are friendly to CSE by checking offsets that are // multiples of the guard size. Add one to make sure that we check the pointer itself is in diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 0632630095..cef629c7dc 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -74,8 +74,8 @@ impl FuncTranslator { func.name, func.signature ); - assert_eq!(func.dfg.num_ebbs(), 0, "Function must be empty"); - assert_eq!(func.dfg.num_insts(), 0, "Function must be empty"); + debug_assert_eq!(func.dfg.num_ebbs(), 0, "Function must be empty"); + debug_assert_eq!(func.dfg.num_insts(), 0, "Function must be empty"); // This clears the `ILBuilder`. let mut builder = FunctionBuilder::new(func, &mut self.il_builder); @@ -191,7 +191,7 @@ fn parse_function_body( environ: &mut FE, ) -> CtonResult { // The control stack is initialized with a single block representing the whole function. - assert_eq!(state.control_stack.len(), 1, "State not initialized"); + debug_assert_eq!(state.control_stack.len(), 1, "State not initialized"); // Keep going until the final `End` operator which pops the outermost block. while !state.control_stack.is_empty() { From f04e02c0a15850e141347d5d4e36ebedaa901b30 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 12 Mar 2018 10:28:39 -0700 Subject: [PATCH 1598/3084] Clarify comments about Conventional SSA form. Captialize "Conventional" so that it's clear that "Conventional SSA" is a specific concept being referenced. --- lib/cretonne/src/regalloc/coalescing.rs | 12 ++++++------ lib/cretonne/src/regalloc/coloring.rs | 2 +- lib/cretonne/src/regalloc/context.rs | 2 +- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 10d0df2e62..3b9d6d03bb 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -1,9 +1,9 @@ -//! Constructing conventional SSA form. +//! Constructing Conventional SSA form. //! -//! Conventional SSA form is a subset of SSA form where any (transitively) phi-related values do -//! not interfere. We construct CSSA by building virtual registers that are as large as possible -//! and inserting copies where necessary such that all argument values passed to an EBB parameter -//! will belong to the same virtual register as the EBB parameter value itself. +//! Conventional SSA (CSSA) form is a subset of SSA form where any (transitively) phi-related +//! values do not interfere. We construct CSSA by building virtual registers that are as large as +//! possible and inserting copies where necessary such that all argument values passed to an EBB +//! parameter will belong to the same virtual register as the EBB parameter value itself. use cursor::{Cursor, EncCursor}; use dbg::DisplayList; @@ -103,7 +103,7 @@ impl Coalescing { self.backedges.clear(); } - /// Convert `func` to conventional SSA form and build virtual registers in the process. + /// Convert `func` to Conventional SSA form and build virtual registers in the process. pub fn conventional_ssa( &mut self, isa: &TargetIsa, diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index 2de41e67c6..d7ab5dd0f3 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -23,7 +23,7 @@ //! operands are allowed to read spilled values, but each such instance must be counted as using //! a register. //! -//! 5. The code must be in conventional SSA form. Among other things, this means that values passed +//! 5. The code must be in Conventional SSA form. Among other things, this means that values passed //! as arguments when branching to an EBB must belong to the same virtual register as the //! corresponding EBB argument value. //! diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index 765ecf155c..a8284a9579 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -90,7 +90,7 @@ impl Context { verify_liveness(isa, func, cfg, &self.liveness)?; } - // Pass: Coalesce and create conventional SSA form. + // Pass: Coalesce and create Conventional SSA form. self.coalescing.conventional_ssa( isa, func, From 11eddafef8cea4ced9cc36a47bb511119320f8ff Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 12 Mar 2018 10:28:42 -0700 Subject: [PATCH 1599/3084] Avoid using floating-point values in expand_fcvt_to_sint. Compute the bound values for expand_fcvt_to_sint using bitwise integer arithmetic rather than floating-point arithmetic, to avoid relying on host floating point arithmetic. --- lib/cretonne/src/ir/immediates.rs | 40 ++++++++++++++++++++++++ lib/cretonne/src/isa/intel/enc_tables.rs | 17 +++++++--- 2 files changed, 53 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 2d3b1236ef..69c9aab51a 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -538,6 +538,17 @@ impl Ieee32 { Ieee32(exponent << t) } + /// Create an `Ieee32` number representing the greatest negative value + /// not convertable from f32 to a signed integer with width n. + pub fn fcvt_to_sint_negative_overflow>(n: I) -> Ieee32 { + let n = n.into(); + debug_assert!(n < 32); + debug_assert!(23 + 1 - n < 32); + Self::with_bits( + (1u32 << (32 - 1)) | Self::pow2(n - 1).0 | (1u32 << (23 + 1 - n)), + ) + } + /// Return self negated. pub fn neg(self) -> Ieee32 { Ieee32(self.0 ^ (1 << 31)) @@ -590,6 +601,17 @@ impl Ieee64 { Ieee64(exponent << t) } + /// Create an `Ieee64` number representing the greatest negative value + /// not convertable from f64 to a signed integer with width n. + pub fn fcvt_to_sint_negative_overflow>(n: I) -> Ieee64 { + let n = n.into(); + debug_assert!(n < 64); + debug_assert!(52 + 1 - n < 64); + Self::with_bits( + (1u64 << (64 - 1)) | Self::pow2(n - 1).0 | (1u64 << (52 + 1 - n)), + ) + } + /// Return self negated. pub fn neg(self) -> Ieee64 { Ieee64(self.0 ^ (1 << 63)) @@ -857,6 +879,15 @@ mod tests { assert_eq!(Ieee32::pow2(1).neg().to_string(), "-0x1.000000p1"); } + #[test] + fn fcvt_to_sint_negative_overflow_ieee32() { + for n in &[8, 16] { + assert_eq!(-((1u32 << (n - 1)) as f32) - 1.0, unsafe { + mem::transmute(Ieee32::fcvt_to_sint_negative_overflow(*n)) + }); + } + } + #[test] fn format_ieee64() { assert_eq!(Ieee64::with_float(0.0).to_string(), "0.0"); @@ -985,4 +1016,13 @@ mod tests { assert_eq!(Ieee64::pow2(1).neg().to_string(), "-0x1.0000000000000p1"); } + + #[test] + fn fcvt_to_sint_negative_overflow_ieee64() { + for n in &[8, 16, 32] { + assert_eq!(-((1u64 << (n - 1)) as f64) - 1.0, unsafe { + mem::transmute(Ieee64::fcvt_to_sint_negative_overflow(*n)) + }); + } + } } diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index 00c5ab99cb..d3f62adf7c 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -376,13 +376,22 @@ fn expand_fcvt_to_sint( let mut overflow_cc = FloatCC::LessThan; let output_bits = ty.lane_bits(); let flimit = match xty { - ir::types::F32 => pos.ins().f32const(Ieee32::pow2(output_bits - 1).neg()), + // An f32 can represent `i16::min_value() - 1` exactly with precision to spare, so + // there are values less than -2^(N-1) that convert correctly to INT_MIN. + ir::types::F32 => { + pos.ins().f32const(if output_bits < 32 { + overflow_cc = FloatCC::LessThanOrEqual; + Ieee32::fcvt_to_sint_negative_overflow(output_bits) + } else { + Ieee32::pow2(output_bits - 1).neg() + }) + } ir::types::F64 => { // An f64 can represent `i32::min_value() - 1` exactly with precision to spare, so // there are values less than -2^(N-1) that convert correctly to INT_MIN. pos.ins().f64const(if output_bits < 64 { overflow_cc = FloatCC::LessThanOrEqual; - Ieee64::with_float(-((1u64 << (output_bits - 1)) as f64) - 1.0) + Ieee64::fcvt_to_sint_negative_overflow(output_bits) } else { Ieee64::pow2(output_bits - 1).neg() }) @@ -394,8 +403,8 @@ fn expand_fcvt_to_sint( // Finally, we could have a positive value that is too large. let fzero = match xty { - ir::types::F32 => pos.ins().f32const(Ieee32::with_float(0.0)), - ir::types::F64 => pos.ins().f64const(Ieee64::with_float(0.0)), + ir::types::F32 => pos.ins().f32const(Ieee32::with_bits(0)), + ir::types::F64 => pos.ins().f64const(Ieee64::with_bits(0)), _ => panic!("Can't convert {}", xty), }; let overflow = pos.ins().fcmp(FloatCC::GreaterThanOrEqual, x, fzero); From 2b9229d715443c1e8db67482612c68f15841c1a5 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 12 Mar 2018 13:02:55 -0700 Subject: [PATCH 1600/3084] Fix > 80-column lines for flake8. --- lib/cretonne/meta/gen_instr.py | 3 ++- lib/cretonne/meta/gen_settings.py | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index ac8e338eda..28db9e81d2 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -211,7 +211,8 @@ def gen_instruction_data_impl(fmt): if f.has_value_list: fmt.line(n + ' { ref mut args, .. } => args,') fmt.line('_ => panic!("No value list: {:?}", self),') - fmt.line('debug_assert!(args.is_empty(), "Value list already in use");') + fmt.line( + 'debug_assert!(args.is_empty(), "Value list already in use");') fmt.line('*args = vlist;') diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index dfc2c2b70a..1560157915 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -245,7 +245,8 @@ def gen_constructor(sgrp, parent, fmt): 'pub fn new({}) -> Flags {{'.format(args), '}'): fmt.line('let bvec = builder.state_for("{}");'.format(sgrp.name)) fmt.line('let mut bytes = [0; {}];'.format(sgrp.byte_size())) - fmt.line('debug_assert_eq!(bvec.len(), {});'.format(sgrp.settings_size)) + fmt.line( + 'debug_assert_eq!(bvec.len(), {});'.format(sgrp.settings_size)) with fmt.indented( 'for (i, b) in bvec.iter().enumerate() {', '}'): fmt.line('bytes[i] = *b;') From 1a671cf82f6144856b257a193ba6605f06872c15 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 12 Mar 2018 14:06:15 -0700 Subject: [PATCH 1601/3084] Remove uses of `println!` in tests. In this case, it's a little nicer to just use more assertions, which will print their line number indicating how far the test got, when they fail. And this allows the tests to be run in no_std configurations. --- lib/cretonne/src/divconst_magic_numbers.rs | 23 ++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/src/divconst_magic_numbers.rs b/lib/cretonne/src/divconst_magic_numbers.rs index b046e22411..f189e23d1a 100644 --- a/lib/cretonne/src/divconst_magic_numbers.rs +++ b/lib/cretonne/src/divconst_magic_numbers.rs @@ -480,56 +480,63 @@ mod tests { // don't panic with integer wraparounds, especially at boundary // cases for their arguments. The actual results are thrown away. let mut total: u64 = 0; - println!("Testing UP magicU32"); + // Testing UP magicU32 for x in 2..(200 * 1000u32) { let m = magicU32(x); total = total ^ (m.mulBy as u64); total = total + (m.shiftBy as u64); total = total - (if m.doAdd { 123 } else { 456 }); } - println!("Testing DOWN magicU32"); + assert_eq!(total, 1747815691); + // Testing DOWN magicU32 for x in 0..(200 * 1000u32) { let m = magicU32(0xFFFF_FFFFu32 - x); total = total ^ (m.mulBy as u64); total = total + (m.shiftBy as u64); total = total - (if m.doAdd { 123 } else { 456 }); } + assert_eq!(total, 2210292772); - println!("Testing UP magicU64"); + // Testing UP magicU64 for x in 2..(200 * 1000u64) { let m = magicU64(x); total = total ^ m.mulBy; total = total + (m.shiftBy as u64); total = total - (if m.doAdd { 123 } else { 456 }); } - println!("Testing DOWN magicU64"); + assert_eq!(total, 7430004084791260605); + // Testing DOWN magicU64 for x in 0..(200 * 1000u64) { let m = magicU64(0xFFFF_FFFF_FFFF_FFFFu64 - x); total = total ^ m.mulBy; total = total + (m.shiftBy as u64); total = total - (if m.doAdd { 123 } else { 456 }); } + assert_eq!(total, 7547519887519825919); - println!("Testing UP magicS32"); + // Testing UP magicS32 for x in 0..(200 * 1000i32) { let m = magicS32(-0x8000_0000i32 + x); total = total ^ (m.mulBy as u64); total = total + (m.shiftBy as u64); } - println!("Testing DOWN magicS32"); + assert_eq!(total, 10899224186731671235); + // Testing DOWN magicS32 for x in 0..(200 * 1000i32) { let m = magicS32(0x7FFF_FFFFi32 - x); total = total ^ (m.mulBy as u64); total = total + (m.shiftBy as u64); } + assert_eq!(total, 7547519887517897369); - println!("Testing UP magicS64"); + // Testing UP magicS64 for x in 0..(200 * 1000i64) { let m = magicS64(-0x8000_0000_0000_0000i64 + x); total = total ^ (m.mulBy as u64); total = total + (m.shiftBy as u64); } - println!("Testing DOWN magicS64"); + assert_eq!(total, 8029756891368555163); + // Testing DOWN magicS64 for x in 0..(200 * 1000i64) { let m = magicS64(0x7FFF_FFFF_FFFF_FFFFi64 - x); total = total ^ (m.mulBy as u64); From aaf0def241e40b44badedf97b113be49bb00df9e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 12 Mar 2018 14:09:34 -0700 Subject: [PATCH 1602/3084] Add lint overrides for unused `extern crate hashmap_core`. This allows these files to build with both the `no_std` and `std` features enabled at the same time. --- lib/cretonne/src/lib.rs | 1 + lib/wasm/src/lib.rs | 1 + 2 files changed, 2 insertions(+) diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 9656f6d1e1..29643d552f 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -9,6 +9,7 @@ #![cfg_attr(not(feature = "std"), feature(alloc))] // Include the `hashmap_core` crate if no_std +#[allow(unused_extern_crates)] #[cfg(feature = "no_std")] extern crate hashmap_core; #[cfg(not(feature = "std"))] diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 4dc802d50f..a6492b5d66 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -20,6 +20,7 @@ #[macro_use] extern crate alloc; +#[allow(unused_extern_crates)] #[cfg(feature = "no_std")] extern crate hashmap_core; From e776d987fdae7e225892ab72026b3d31f6d5a06e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 12 Mar 2018 16:07:13 -0700 Subject: [PATCH 1603/3084] Add Cargo.toml badges for published packages. This adds a basic travis badge, as well as a badge indicating "experimental" status, since the APIs are still evolving. --- lib/cretonne/Cargo.toml | 4 ++++ lib/frontend/Cargo.toml | 4 ++++ lib/native/Cargo.toml | 4 ++++ lib/reader/Cargo.toml | 4 ++++ lib/wasm/Cargo.toml | 4 ++++ 5 files changed, 20 insertions(+) diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 597ff1efd6..adcaa28f76 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -18,3 +18,7 @@ name = "cretonne" # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be # accomodated in `tests`. + +[badges] +maintenance = { status = "experimental" } +travis-ci = { repository = "Cretonne/cretonne" } diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 0085036ed0..368b6f058e 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -13,3 +13,7 @@ name = "cton_frontend" [dependencies] cretonne = { path = "../cretonne", version = "0.3.4" } + +[badges] +maintenance = { status = "experimental" } +travis-ci = { repository = "Cretonne/cretonne" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 10856d98a5..89676fc1b3 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -15,3 +15,7 @@ cretonne = { path = "../cretonne", version = "0.3.4" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" + +[badges] +maintenance = { status = "experimental" } +travis-ci = { repository = "Cretonne/cretonne" } diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 0d2061e361..bd26dc0f7c 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -13,3 +13,7 @@ name = "cton_reader" [dependencies] cretonne = { path = "../cretonne", version = "0.3.4" } + +[badges] +maintenance = { status = "experimental" } +travis-ci = { repository = "Cretonne/cretonne" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 8e61d440c0..bc50b4a303 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -18,3 +18,7 @@ cretonne-frontend = { path = "../frontend", version = "0.3.4" } [dev-dependencies] tempdir = "0.3.5" + +[badges] +maintenance = { status = "experimental" } +travis-ci = { repository = "Cretonne/cretonne" } From 44ca27beecc596a5eec2225ffa3cbcbd7e8d11c8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 12 Mar 2018 16:11:14 -0700 Subject: [PATCH 1604/3084] Update to wasmparser 0.15.1. --- lib/wasm/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index bc50b4a303..140ab69cfe 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -12,7 +12,7 @@ keywords = [ "webassembly", "wasm" ] name = "cton_wasm" [dependencies] -wasmparser = "0.14.1" +wasmparser = "0.15.1" cretonne = { path = "../cretonne", version = "0.3.4" } cretonne-frontend = { path = "../frontend", version = "0.3.4" } From 9128290fb4fe73104b9a48e3986b28e0d6f8f0cb Mon Sep 17 00:00:00 2001 From: Afnan Enayet Date: Wed, 14 Mar 2018 10:48:06 -0700 Subject: [PATCH 1605/3084] Rename `ILBuilder` to `FunctionBuilderContext` (#268) * Rename `ILBuilder` to `FunctionBuilderContext` and update corresponding code * Refactor usages of ILBuilder to become FunctionBuilderContext, update variable names to reflect this change * Reformat to ensure that lines stay under 100 char limit * Apply corrections from `rustfmt` to pass tests * Rename variables to be more consistent with refactor of ILBuilder --- lib/frontend/src/frontend.rs | 103 ++++++++++++++++---------------- lib/frontend/src/lib.rs | 14 ++--- lib/frontend/src/variable.rs | 2 +- lib/wasm/src/func_translator.rs | 10 ++-- 4 files changed, 65 insertions(+), 64 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 6802884b6a..de89698f15 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -13,13 +13,13 @@ use cretonne::packed_option::PackedOption; /// Structure used for translating a series of functions into Cretonne IL. /// /// In order to reduce memory reallocations when compiling multiple functions, -/// `ILBuilder` holds various data structures which are cleared between +/// `FunctionBuilderContext` holds various data structures which are cleared between /// functions, rather than dropped, preserving the underlying allocations. /// /// The `Variable` parameter can be any index-like type that can be made to /// implement `EntityRef`. For frontends that don't have an obvious type to /// use here, `variable::Variable` can be used. -pub struct ILBuilder +pub struct FunctionBuilderContext where Variable: EntityRef, { @@ -41,7 +41,7 @@ where /// Source location to assign to all new instructions. srcloc: ir::SourceLoc, - builder: &'a mut ILBuilder, + func_ctx: &'a mut FunctionBuilderContext, position: Position, } @@ -77,12 +77,12 @@ impl Position { } } -impl ILBuilder +impl FunctionBuilderContext where Variable: EntityRef, { - /// Creates a ILBuilder structure. The structure is automatically cleared after each - /// [`FunctionBuilder`](struct.FunctionBuilder.html) completes translating a function. + /// Creates a FunctionBuilderContext structure. The structure is automatically cleared after + /// each [`FunctionBuilder`](struct.FunctionBuilder.html) completes translating a function. pub fn new() -> Self { Self { ssa: SSABuilder::new(), @@ -172,7 +172,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short .entries() .map(|(_, ebb)| ebb) .filter(|dest_ebb| unique.insert(*dest_ebb)) { - self.builder.builder.ssa.declare_ebb_predecessor( + self.builder.func_ctx.ssa.declare_ebb_predecessor( dest_ebb, self.builder.position.basic_block.unwrap(), inst, @@ -213,9 +213,10 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short /// /// At creation, a `FunctionBuilder` instance borrows an already allocated `Function` which it /// modifies with the information stored in the mutable borrowed -/// [`ILBuilder`](struct.ILBuilder.html). The function passed in argument should be newly created -/// with [`Function::with_name_signature()`](../function/struct.Function.html), whereas the -/// `ILBuilder` can be kept as is between two function translations. +/// [`FunctionBuilderContext`](struct.FunctionBuilderContext.html). The function passed in +/// argument should be newly created with +/// [`Function::with_name_signature()`](../function/struct.Function.html), whereas the +/// `FunctionBuilderContext` can be kept as is between two function translations. /// /// # Errors /// @@ -228,16 +229,16 @@ where Variable: EntityRef, { /// Creates a new FunctionBuilder structure that will operate on a `Function` using a - /// `IlBuilder`. + /// `FunctionBuilderContext`. pub fn new( func: &'a mut Function, - builder: &'a mut ILBuilder, + func_ctx: &'a mut FunctionBuilderContext, ) -> FunctionBuilder<'a, Variable> { - debug_assert!(builder.is_empty()); + debug_assert!(func_ctx.is_empty()); FunctionBuilder { func: func, srcloc: Default::default(), - builder: builder, + func_ctx: func_ctx, position: Position::default(), } } @@ -250,8 +251,8 @@ where /// Creates a new `Ebb` and returns its reference. pub fn create_ebb(&mut self) -> Ebb { let ebb = self.func.dfg.make_ebb(); - self.builder.ssa.declare_ebb_header_block(ebb); - self.builder.ebbs[ebb] = EbbData { + self.func_ctx.ssa.declare_ebb_header_block(ebb); + self.func_ctx.ebbs[ebb] = EbbData { filled: false, pristine: true, user_param_count: 0, @@ -275,11 +276,11 @@ where ); // We cannot switch to a filled block debug_assert!( - !self.builder.ebbs[ebb].filled, + !self.func_ctx.ebbs[ebb].filled, "you cannot switch to a block which is already filled" ); - let basic_block = self.builder.ssa.header_block(ebb); + let basic_block = self.func_ctx.ssa.header_block(ebb); // Then we change the cursor position. self.position = Position::at(ebb, basic_block); } @@ -290,7 +291,7 @@ where /// created. Forgetting to call this method on every block will cause inconsistencies in the /// produced functions. pub fn seal_block(&mut self, ebb: Ebb) { - let side_effects = self.builder.ssa.seal_ebb_header_block(ebb, self.func); + let side_effects = self.func_ctx.ssa.seal_ebb_header_block(ebb, self.func); self.handle_ssa_side_effects(side_effects); } @@ -301,22 +302,22 @@ where /// function can be used at the end of translating all blocks to ensure /// that everything is sealed. pub fn seal_all_blocks(&mut self) { - let side_effects = self.builder.ssa.seal_all_ebb_header_blocks(self.func); + let side_effects = self.func_ctx.ssa.seal_all_ebb_header_blocks(self.func); self.handle_ssa_side_effects(side_effects); } /// In order to use a variable in a `use_var`, you need to declare its type with this method. pub fn declare_var(&mut self, var: Variable, ty: Type) { - self.builder.types[var] = ty; + self.func_ctx.types[var] = ty; } /// Returns the Cretonne IL value corresponding to the utilization at the current program /// position of a previously defined user variable. pub fn use_var(&mut self, var: Variable) -> Value { - let ty = *self.builder.types.get(var).expect( + let ty = *self.func_ctx.types.get(var).expect( "this variable is used but its type has not been declared", ); - let (val, side_effects) = self.builder.ssa.use_var( + let (val, side_effects) = self.func_ctx.ssa.use_var( self.func, var, ty, @@ -329,7 +330,7 @@ where /// Register a new definition of a user variable. Panics if the type of the value is not the /// same as the type registered for the variable. pub fn def_var(&mut self, var: Variable, val: Value) { - self.builder.ssa.def_var( + self.func_ctx.ssa.def_var( var, val, self.position.basic_block.unwrap(), @@ -382,14 +383,14 @@ where /// Make sure that the current EBB is inserted in the layout. pub fn ensure_inserted_ebb(&mut self) { let ebb = self.position.ebb.unwrap(); - if self.builder.ebbs[ebb].pristine { + if self.func_ctx.ebbs[ebb].pristine { if !self.func.layout.is_ebb_inserted(ebb) { self.func.layout.append_ebb(ebb); } - self.builder.ebbs[ebb].pristine = false; + self.func_ctx.ebbs[ebb].pristine = false; } else { debug_assert!( - !self.builder.ebbs[ebb].filled, + !self.func_ctx.ebbs[ebb].filled, "you cannot add an instruction to a block already filled" ); } @@ -412,7 +413,7 @@ where pub fn append_ebb_params_for_function_params(&mut self, ebb: Ebb) { // These parameters count as "user" parameters here because they aren't // inserted by the SSABuilder. - let user_param_count = &mut self.builder.ebbs[ebb].user_param_count; + let user_param_count = &mut self.func_ctx.ebbs[ebb].user_param_count; for argtyp in &self.func.signature.params { *user_param_count += 1; self.func.dfg.append_ebb_param(ebb, argtyp.value_type); @@ -425,7 +426,7 @@ where pub fn append_ebb_params_for_function_returns(&mut self, ebb: Ebb) { // These parameters count as "user" parameters here because they aren't // inserted by the SSABuilder. - let user_param_count = &mut self.builder.ebbs[ebb].user_param_count; + let user_param_count = &mut self.func_ctx.ebbs[ebb].user_param_count; for argtyp in &self.func.signature.returns { *user_param_count += 1; self.func.dfg.append_ebb_param(ebb, argtyp.value_type); @@ -438,21 +439,21 @@ where pub fn finalize(&mut self) { // Check that all the `Ebb`s are filled and sealed. debug_assert!( - self.builder.ebbs.keys().all(|ebb| { - self.builder.ebbs[ebb].pristine || self.builder.ssa.is_sealed(ebb) + self.func_ctx.ebbs.keys().all(|ebb| { + self.func_ctx.ebbs[ebb].pristine || self.func_ctx.ssa.is_sealed(ebb) }), "all blocks should be sealed before dropping a FunctionBuilder" ); debug_assert!( - self.builder.ebbs.keys().all(|ebb| { - self.builder.ebbs[ebb].pristine || self.builder.ebbs[ebb].filled + self.func_ctx.ebbs.keys().all(|ebb| { + self.func_ctx.ebbs[ebb].pristine || self.func_ctx.ebbs[ebb].filled }), "all blocks should be filled before dropping a FunctionBuilder" ); // Clear the state (but preserve the allocated buffers) in preparation // for translation another function. - self.builder.clear(); + self.func_ctx.clear(); // Reset srcloc and position to initial states. self.srcloc = Default::default(); @@ -486,12 +487,12 @@ where /// **Note:** this function has to be called at the creation of the `Ebb` before adding /// instructions to it, otherwise this could interfere with SSA construction. pub fn append_ebb_param(&mut self, ebb: Ebb, ty: Type) -> Value { - debug_assert!(self.builder.ebbs[ebb].pristine); + debug_assert!(self.func_ctx.ebbs[ebb].pristine); debug_assert_eq!( - self.builder.ebbs[ebb].user_param_count, + self.func_ctx.ebbs[ebb].user_param_count, self.func.dfg.num_ebb_params(ebb) ); - self.builder.ebbs[ebb].user_param_count += 1; + self.func_ctx.ebbs[ebb].user_param_count += 1; self.func.dfg.append_ebb_param(ebb, ty) } @@ -508,9 +509,9 @@ where let old_dest = self.func.dfg[inst].branch_destination_mut().expect( "you want to change the jump destination of a non-jump instruction", ); - let pred = self.builder.ssa.remove_ebb_predecessor(*old_dest, inst); + let pred = self.func_ctx.ssa.remove_ebb_predecessor(*old_dest, inst); *old_dest = new_dest; - self.builder.ssa.declare_ebb_predecessor( + self.func_ctx.ssa.declare_ebb_predecessor( new_dest, pred, inst, @@ -525,8 +526,8 @@ where None => false, Some(entry) => self.position.ebb.unwrap() == entry, }; - !is_entry && self.builder.ssa.is_sealed(self.position.ebb.unwrap()) && - self.builder + !is_entry && self.func_ctx.ssa.is_sealed(self.position.ebb.unwrap()) && + self.func_ctx .ssa .predecessors(self.position.ebb.unwrap()) .is_empty() @@ -535,13 +536,13 @@ where /// Returns `true` if and only if no instructions have been added since the last call to /// `switch_to_block`. pub fn is_pristine(&self) -> bool { - self.builder.ebbs[self.position.ebb.unwrap()].pristine + self.func_ctx.ebbs[self.position.ebb.unwrap()].pristine } /// Returns `true` if and only if a terminator instruction has been inserted since the /// last call to `switch_to_block`. pub fn is_filled(&self) -> bool { - self.builder.ebbs[self.position.ebb.unwrap()].filled + self.func_ctx.ebbs[self.position.ebb.unwrap()].filled } /// Returns a displayable object for the function as it is. @@ -558,17 +559,17 @@ where Variable: EntityRef, { fn move_to_next_basic_block(&mut self) { - self.position.basic_block = PackedOption::from(self.builder.ssa.declare_ebb_body_block( + self.position.basic_block = PackedOption::from(self.func_ctx.ssa.declare_ebb_body_block( self.position.basic_block.unwrap(), )); } fn fill_current_block(&mut self) { - self.builder.ebbs[self.position.ebb.unwrap()].filled = true; + self.func_ctx.ebbs[self.position.ebb.unwrap()].filled = true; } fn declare_successor(&mut self, dest_ebb: Ebb, jump_inst: Inst) { - self.builder.ssa.declare_ebb_predecessor( + self.func_ctx.ssa.declare_ebb_predecessor( dest_ebb, self.position.basic_block.unwrap(), jump_inst, @@ -577,10 +578,10 @@ where fn handle_ssa_side_effects(&mut self, side_effects: SideEffects) { for split_ebb in side_effects.split_ebbs_created { - self.builder.ebbs[split_ebb].filled = true + self.func_ctx.ebbs[split_ebb].filled = true } for modified_ebb in side_effects.instructions_added_to_ebbs { - self.builder.ebbs[modified_ebb].pristine = false + self.func_ctx.ebbs[modified_ebb].pristine = false } } } @@ -591,7 +592,7 @@ mod tests { use cretonne::entity::EntityRef; use cretonne::ir::{ExternalName, Function, CallConv, Signature, AbiParam, InstBuilder}; use cretonne::ir::types::*; - use frontend::{ILBuilder, FunctionBuilder}; + use frontend::{FunctionBuilderContext, FunctionBuilder}; use cretonne::verifier::verify_function; use cretonne::settings; use Variable; @@ -601,10 +602,10 @@ mod tests { sig.returns.push(AbiParam::new(I32)); sig.params.push(AbiParam::new(I32)); - let mut il_builder = ILBuilder::::new(); + let mut fn_ctx = FunctionBuilderContext::::new(); let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig); { - let mut builder = FunctionBuilder::::new(&mut func, &mut il_builder); + let mut builder = FunctionBuilder::::new(&mut func, &mut fn_ctx); let block0 = builder.create_ebb(); let block1 = builder.create_ebb(); diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index a95d662893..f5b4514e56 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -4,8 +4,8 @@ //! translated from another language. Contains an SSA construction module that lets you translate //! your non-SSA variables into SSA Cretonne IL values via `use_var` and `def_var` calls. //! -//! To get started, create an [`IlBuilder`](struct.ILBuilder.html) and pass it as an argument -//! to a [`FunctionBuilder`](struct.FunctionBuilder.html). +//! To get started, create an [`FunctionBuilderContext`](struct.FunctionBuilderContext.html) and +//! pass it as an argument to a [`FunctionBuilder`](struct.FunctionBuilder.html). //! //! # Example //! @@ -29,7 +29,7 @@ //! } //! ``` //! -//! Here is how you build the corresponding Cretonne IL function using `ILBuilder`: +//! Here is how you build the corresponding Cretonne IL function using `FunctionBuilderContext`: //! //! ```rust //! extern crate cretonne; @@ -39,17 +39,17 @@ //! use cretonne::ir::{ExternalName, CallConv, Function, Signature, AbiParam, InstBuilder}; //! use cretonne::ir::types::*; //! use cretonne::settings; -//! use cton_frontend::{ILBuilder, FunctionBuilder, Variable}; +//! use cton_frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; //! use cretonne::verifier::verify_function; //! //! fn main() { //! let mut sig = Signature::new(CallConv::Native); //! sig.returns.push(AbiParam::new(I32)); //! sig.params.push(AbiParam::new(I32)); -//! let mut il_builder = ILBuilder::::new(); +//! let mut fn_builder_ctx = FunctionBuilderContext::::new(); //! let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig); //! { -//! let mut builder = FunctionBuilder::::new(&mut func, &mut il_builder); +//! let mut builder = FunctionBuilder::::new(&mut func, &mut fn_builder_ctx); //! //! let block0 = builder.create_ebb(); //! let block1 = builder.create_ebb(); @@ -133,7 +133,7 @@ extern crate cretonne; -pub use frontend::{ILBuilder, FunctionBuilder}; +pub use frontend::{FunctionBuilderContext, FunctionBuilder}; pub use variable::Variable; mod frontend; diff --git a/lib/frontend/src/variable.rs b/lib/frontend/src/variable.rs index b69a63afeb..b4e3c75da2 100644 --- a/lib/frontend/src/variable.rs +++ b/lib/frontend/src/variable.rs @@ -1,6 +1,6 @@ //! A basic `Variable` implementation. //! -//! `ILBuilder`, `FunctionBuilder`, and related types have a `Variable` +//! `FunctionBuilderContext`, `FunctionBuilder`, and related types have a `Variable` //! type parameter, to allow frontends that identify variables with //! their own index types to use them directly. Frontends which don't //! can use the `Variable` defined here. diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index cef629c7dc..5ac69b07b8 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -9,7 +9,7 @@ use cretonne::entity::EntityRef; use cretonne::ir::{self, InstBuilder, Ebb}; use cretonne::result::{CtonResult, CtonError}; use cretonne::timing; -use cton_frontend::{ILBuilder, FunctionBuilder, Variable}; +use cton_frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; use environ::FuncEnvironment; use state::TranslationState; use wasmparser::{self, BinaryReader}; @@ -20,7 +20,7 @@ use wasmparser::{self, BinaryReader}; /// by a `FuncEnvironment` object. A single translator instance can be reused to translate multiple /// functions which will reduce heap allocation traffic. pub struct FuncTranslator { - il_builder: ILBuilder, + func_ctx: FunctionBuilderContext, state: TranslationState, } @@ -28,7 +28,7 @@ impl FuncTranslator { /// Create a new translator. pub fn new() -> Self { Self { - il_builder: ILBuilder::new(), + func_ctx: FunctionBuilderContext::new(), state: TranslationState::new(), } } @@ -77,8 +77,8 @@ impl FuncTranslator { debug_assert_eq!(func.dfg.num_ebbs(), 0, "Function must be empty"); debug_assert_eq!(func.dfg.num_insts(), 0, "Function must be empty"); - // This clears the `ILBuilder`. - let mut builder = FunctionBuilder::new(func, &mut self.il_builder); + // This clears the `FunctionBuilderContext`. + let mut builder = FunctionBuilder::new(func, &mut self.func_ctx); let entry_block = builder.create_ebb(); builder.append_ebb_params_for_function_params(entry_block); builder.switch_to_block(entry_block); // This also creates values for the arguments. From 3afe85ff17ca1dbd8e7bf53b83599a7a31ef1609 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 13 Mar 2018 10:06:45 -0700 Subject: [PATCH 1606/3084] Auto-generate `InstructionData`. The meta description has all the information to generate the `InstructionData` enum, so generate it rather than having a manually-maintained copy. --- lib/cretonne/meta/gen_instr.py | 34 +++++ lib/cretonne/src/ir/instructions.rs | 190 +--------------------------- 2 files changed, 36 insertions(+), 188 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 28db9e81d2..ab42b01850 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -103,6 +103,38 @@ def gen_arguments_method(fmt, is_mut): .format(n, capture, arg)) +def gen_instruction_data(fmt): + # type: (srcgen.Formatter) -> None + """ + Generate the InstructionData enum. + + Every variant must contain `opcode` and `ty` fields. An instruction that + doesn't produce a value should have its `ty` field set to `VOID`. The size + of `InstructionData` should be kept at 16 bytes on 64-bit architectures. If + more space is needed to represent an instruction, use a `Box` to + store the additional information out of line. + """ + + fmt.line('#[derive(Clone, Debug, Hash, PartialEq, Eq)]') + fmt.line('#[allow(missing_docs)]') + with fmt.indented('pub enum InstructionData {', '}'): + for f in InstructionFormat.all_formats: + with fmt.indented('{} {{'.format(f.name), '},'): + fmt.line('opcode: Opcode,') + if f.typevar_operand is None: + pass + elif f.has_value_list: + fmt.line('args: ValueList,') + elif f.num_value_operands == 1: + fmt.line('arg: Value,') + else: + fmt.line('args: [Value; {}],'.format(f.num_value_operands)) + for field in f.imm_fields: + fmt.line( + '{}: {},' + .format(field.member, field.kind.rust_type)) + + def gen_instruction_data_impl(fmt): # type: (srcgen.Formatter) -> None """ @@ -682,6 +714,8 @@ def generate(isas, out_dir): # opcodes.rs fmt = srcgen.Formatter() gen_formats(fmt) + gen_instruction_data(fmt) + fmt.line() gen_instruction_data_impl(fmt) fmt.line() instrs = gen_opcodes(groups, fmt) diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 2d462064d1..358c63d942 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -11,11 +11,9 @@ use std::str::FromStr; use std::ops::{Deref, DerefMut}; use ir; -use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef, StackSlot, MemFlags}; -use ir::immediates::{Imm64, Uimm8, Uimm32, Ieee32, Ieee64, Offset32}; -use ir::condcodes::*; +use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef}; use ir::types; -use isa::RegUnit; +use isa; use entity; use bitset::BitSet; @@ -94,190 +92,6 @@ impl FromStr for Opcode { } } -/// Contents on an instruction. -/// -/// Every variant must contain `opcode` and `ty` fields. An instruction that doesn't produce a -/// value should have its `ty` field set to `VOID`. The size of `InstructionData` should be kept at -/// 16 bytes on 64-bit architectures. If more space is needed to represent an instruction, use a -/// `Box` to store the additional information out of line. -#[derive(Clone, Debug, Hash, PartialEq, Eq)] -#[allow(missing_docs)] -pub enum InstructionData { - Unary { opcode: Opcode, arg: Value }, - UnaryImm { opcode: Opcode, imm: Imm64 }, - UnaryIeee32 { opcode: Opcode, imm: Ieee32 }, - UnaryIeee64 { opcode: Opcode, imm: Ieee64 }, - UnaryBool { opcode: Opcode, imm: bool }, - UnaryGlobalVar { - opcode: Opcode, - global_var: ir::GlobalVar, - }, - Binary { opcode: Opcode, args: [Value; 2] }, - BinaryImm { - opcode: Opcode, - arg: Value, - imm: Imm64, - }, - Ternary { opcode: Opcode, args: [Value; 3] }, - MultiAry { opcode: Opcode, args: ValueList }, - NullAry { opcode: Opcode }, - InsertLane { - opcode: Opcode, - lane: Uimm8, - args: [Value; 2], - }, - ExtractLane { - opcode: Opcode, - lane: Uimm8, - arg: Value, - }, - IntCompare { - opcode: Opcode, - cond: IntCC, - args: [Value; 2], - }, - IntCompareImm { - opcode: Opcode, - cond: IntCC, - arg: Value, - imm: Imm64, - }, - IntCond { - opcode: Opcode, - cond: IntCC, - arg: Value, - }, - FloatCompare { - opcode: Opcode, - cond: FloatCC, - args: [Value; 2], - }, - FloatCond { - opcode: Opcode, - cond: FloatCC, - arg: Value, - }, - IntSelect { - opcode: Opcode, - cond: IntCC, - args: [Value; 3], - }, - Jump { - opcode: Opcode, - destination: Ebb, - args: ValueList, - }, - Branch { - opcode: Opcode, - destination: Ebb, - args: ValueList, - }, - BranchIcmp { - opcode: Opcode, - cond: IntCC, - destination: Ebb, - args: ValueList, - }, - BranchInt { - opcode: Opcode, - cond: IntCC, - destination: Ebb, - args: ValueList, - }, - BranchFloat { - opcode: Opcode, - cond: FloatCC, - destination: Ebb, - args: ValueList, - }, - BranchTable { - opcode: Opcode, - arg: Value, - table: JumpTable, - }, - Call { - opcode: Opcode, - func_ref: FuncRef, - args: ValueList, - }, - IndirectCall { - opcode: Opcode, - sig_ref: SigRef, - args: ValueList, - }, - FuncAddr { opcode: Opcode, func_ref: FuncRef }, - StackLoad { - opcode: Opcode, - stack_slot: StackSlot, - offset: Offset32, - }, - StackStore { - opcode: Opcode, - arg: Value, - stack_slot: StackSlot, - offset: Offset32, - }, - HeapAddr { - opcode: Opcode, - heap: ir::Heap, - arg: Value, - imm: Uimm32, - }, - Load { - opcode: Opcode, - flags: MemFlags, - arg: Value, - offset: Offset32, - }, - Store { - opcode: Opcode, - flags: MemFlags, - args: [Value; 2], - offset: Offset32, - }, - RegMove { - opcode: Opcode, - arg: Value, - src: RegUnit, - dst: RegUnit, - }, - CopySpecial { - opcode: Opcode, - src: RegUnit, - dst: RegUnit, - }, - RegSpill { - opcode: Opcode, - arg: Value, - src: RegUnit, - dst: StackSlot, - }, - RegFill { - opcode: Opcode, - arg: Value, - src: StackSlot, - dst: RegUnit, - }, - Trap { opcode: Opcode, code: ir::TrapCode }, - CondTrap { - opcode: Opcode, - arg: Value, - code: ir::TrapCode, - }, - IntCondTrap { - opcode: Opcode, - cond: IntCC, - arg: Value, - code: ir::TrapCode, - }, - FloatCondTrap { - opcode: Opcode, - cond: FloatCC, - arg: Value, - code: ir::TrapCode, - }, -} - /// A variable list of `Value` operands used for function call arguments and passing arguments to /// basic blocks. #[derive(Clone, Debug)] From cc8d6400f49f541384f51f5a848a992c58041a59 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 13 Mar 2018 17:00:23 -0700 Subject: [PATCH 1607/3084] Rename builder.rs to inst_builder.rs. This reflects its purpose, to define the `InstBuilder` trait. --- lib/cretonne/meta/gen_instr.py | 6 +++--- lib/cretonne/src/ir/builder.rs | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index ab42b01850..38d120fa7e 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -687,7 +687,7 @@ def gen_builder(insts, fmt): fmt.doc_comment(""" Convenience methods for building instructions. - The `InstrBuilder` trait has one method per instruction opcode for + The `InstBuilder` trait has one method per instruction opcode for conveniently constructing the instruction with minimum arguments. Polymorphic instructions infer their result types from the input arguments when possible. In some cases, an explicit `ctrl_typevar` @@ -722,7 +722,7 @@ def generate(isas, out_dir): gen_type_constraints(fmt, instrs) fmt.update_file('opcodes.rs', out_dir) - # builder.rs + # inst_builder.rs fmt = srcgen.Formatter() gen_builder(instrs, fmt) - fmt.update_file('builder.rs', out_dir) + fmt.update_file('inst_builder.rs', out_dir) diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index a69fd4e181..7e1480186d 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -36,7 +36,7 @@ pub trait InstBuilderBase<'f>: Sized { // // This file defines the `InstBuilder` trait as an extension of `InstBuilderBase` with methods per // instruction format and per opcode. -include!(concat!(env!("OUT_DIR"), "/builder.rs")); +include!(concat!(env!("OUT_DIR"), "/inst_builder.rs")); /// Any type implementing `InstBuilderBase` gets all the `InstBuilder` methods for free. impl<'f, T: InstBuilderBase<'f>> InstBuilder<'f> for T {} From d9712f5d7dee23289ce944471dd167b888603c2c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 13 Mar 2018 20:56:53 -0700 Subject: [PATCH 1608/3084] Elaborate on some comments in generated source files. Recipe names are fairly obscure, so the more context we can give when using them the better. --- lib/cretonne/meta/gen_binemit.py | 2 +- lib/cretonne/meta/gen_encoding.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/meta/gen_binemit.py b/lib/cretonne/meta/gen_binemit.py index c3d1b18e86..5888d89e10 100644 --- a/lib/cretonne/meta/gen_binemit.py +++ b/lib/cretonne/meta/gen_binemit.py @@ -152,7 +152,7 @@ def gen_isa(isa, fmt): fmt.line('let bits = encoding.bits();') with fmt.indented('match func.encodings[inst].recipe() {', '}'): for i, recipe in enumerate(isa.all_recipes): - fmt.comment(recipe.name) + fmt.comment('Recipe {}'.format(recipe.name)) with fmt.indented('{} => {{'.format(i), '}'): gen_recipe(recipe, fmt) fmt.line('_ => {},') diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 46fe22a913..da382a1774 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -756,7 +756,7 @@ def emit_recipe_constraints(isa, fmt): 'static RECIPE_CONSTRAINTS: [RecipeConstraints; {}] = [' .format(len(isa.all_recipes)), '];'): for r in isa.all_recipes: - fmt.comment(r.name) + fmt.comment('Constraints for recipe {}:'.format(r.name)) tied_i2o, tied_o2i = r.ties() fixed_ins, fixed_outs = r.fixed_ops() with fmt.indented('RecipeConstraints {', '},'): @@ -830,7 +830,7 @@ def emit_recipe_sizing(isa, fmt): 'static RECIPE_SIZING: [RecipeSizing; {}] = [' .format(len(isa.all_recipes)), '];'): for r in isa.all_recipes: - fmt.comment(r.name) + fmt.comment('Code size information for recipe {}:'.format(r.name)) with fmt.indented('RecipeSizing {', '},'): fmt.format('bytes: {},', r.size) if r.branch_range: From 272d03d8fc0771bb7416d81e4dc1bb2695b0deb7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 14 Mar 2018 10:40:47 -0700 Subject: [PATCH 1609/3084] Add a utility for generating Rust 'match' expressions. This makes it a little simpler to generate 'match' statements, and it performs deduplication of identical arms. And it means I don't have to think about as many strings like '{} {{ {}.. }} => {}' when I'm trying to think about how instructions work :-). --- lib/cretonne/meta/gen_instr.py | 146 +++++++++++++++--------------- lib/cretonne/meta/gen_settings.py | 11 +-- lib/cretonne/meta/srcgen.py | 82 ++++++++++++++++- 3 files changed, 157 insertions(+), 82 deletions(-) diff --git a/lib/cretonne/meta/gen_instr.py b/lib/cretonne/meta/gen_instr.py index 38d120fa7e..4e0bc6cb78 100644 --- a/lib/cretonne/meta/gen_instr.py +++ b/lib/cretonne/meta/gen_instr.py @@ -49,11 +49,11 @@ def gen_formats(fmt): with fmt.indented( "fn from(inst: &'a InstructionData) -> InstructionFormat {", '}'): - with fmt.indented('match *inst {', '}'): - for f in InstructionFormat.all_formats: - fmt.line(('InstructionData::{} {{ .. }} => ' + - 'InstructionFormat::{},') - .format(f.name, f.name)) + m = srcgen.Match('*inst') + for f in InstructionFormat.all_formats: + m.arm('InstructionData::' + f.name, ['..'], + 'InstructionFormat::' + f.name) + fmt.match(m) fmt.line() @@ -74,33 +74,32 @@ def gen_arguments_method(fmt, is_mut): 'pool: &\'a {m}ir::ValueListPool) -> ' '&{m}[Value] {{' .format(f=method, m=mut), '}'): - with fmt.indented('match *self {', '}'): - for f in InstructionFormat.all_formats: - n = 'InstructionData::' + f.name + m = srcgen.Match('*self') + for f in InstructionFormat.all_formats: + n = 'InstructionData::' + f.name - # Formats with a value list put all of their arguments in the - # list. We don't split them up, just return it all as variable - # arguments. (I expect the distinction to go away). - if f.has_value_list: - arg = ''.format(mut) - fmt.line( - '{} {{ ref {}args, .. }} => args.{}(pool),' - .format(n, mut, as_slice)) - continue + # Formats with a value list put all of their arguments in the + # list. We don't split them up, just return it all as variable + # arguments. (I expect the distinction to go away). + if f.has_value_list: + m.arm(n, ['ref {}args'.format(mut), '..'], + 'args.{}(pool)'.format(as_slice)) + continue - # Fixed args. - if f.num_value_operands == 0: - arg = '&{}[]'.format(mut) - capture = '' - elif f.num_value_operands == 1: - capture = 'ref {}arg, '.format(mut) - arg = '{}(arg)'.format(rslice) - else: - capture = 'ref {}args, '.format(mut) - arg = 'args' - fmt.line( - '{} {{ {}.. }} => {},' - .format(n, capture, arg)) + # Fixed args. + fields = [] + if f.num_value_operands == 0: + arg = '&{}[]'.format(mut) + elif f.num_value_operands == 1: + fields.append('ref {}arg'.format(mut)) + arg = '{}(arg)'.format(rslice) + else: + args = 'args_arity{}'.format(f.num_value_operands) + fields.append('args: ref {}{}'.format(mut, args)) + arg = args + fields.append('..') + m.arm(n, fields, arg) + fmt.match(m) def gen_instruction_data(fmt): @@ -155,39 +154,37 @@ def gen_instruction_data_impl(fmt): with fmt.indented('impl InstructionData {', '}'): fmt.doc_comment('Get the opcode of this instruction.') with fmt.indented('pub fn opcode(&self) -> Opcode {', '}'): - with fmt.indented('match *self {', '}'): - for f in InstructionFormat.all_formats: - fmt.line( - 'InstructionData::{} {{ opcode, .. }} => opcode,' - .format(f.name)) + m = srcgen.Match('*self') + for f in InstructionFormat.all_formats: + m.arm('InstructionData::' + f.name, ['opcode', '..'], + 'opcode') + fmt.match(m) fmt.line() fmt.doc_comment('Get the controlling type variable operand.') with fmt.indented( 'pub fn typevar_operand(&self, pool: &ir::ValueListPool) -> ' 'Option {', '}'): - with fmt.indented('match *self {', '}'): - for f in InstructionFormat.all_formats: - n = 'InstructionData::' + f.name - if f.typevar_operand is None: - fmt.line(n + ' { .. } => None,') - elif f.has_value_list: - # We keep all arguments in a value list. - i = f.typevar_operand - fmt.line( - '{} {{ ref args, .. }} => ' - 'args.get({}, pool),'.format(n, i)) - elif f.num_value_operands == 1: - # We have a single value operand called 'arg'. - fmt.line(n + ' { arg, .. } => Some(arg),') - else: - # We have multiple value operands and an array `args`. - # Which `args` index to use? - i = f.typevar_operand - fmt.line( - n + - ' {{ ref args, .. }} => Some(args[{}]),' - .format(i)) + m = srcgen.Match('*self') + for f in InstructionFormat.all_formats: + n = 'InstructionData::' + f.name + if f.typevar_operand is None: + m.arm(n, ['..'], 'None') + elif f.has_value_list: + # We keep all arguments in a value list. + i = f.typevar_operand + m.arm(n, ['ref args', '..'], + 'args.get({}, pool)'.format(i)) + elif f.num_value_operands == 1: + # We have a single value operand called 'arg'. + m.arm(n, ['arg', '..'], 'Some(arg)') + else: + # We have multiple value operands and an array `args`. + # Which `args` index to use? + args = 'args_arity{}'.format(f.num_value_operands) + m.arm(n, ['args: ref {}'.format(args), '..'], + 'Some({}[{}])'.format(args, f.typevar_operand)) + fmt.match(m) fmt.line() fmt.doc_comment( @@ -216,13 +213,13 @@ def gen_instruction_data_impl(fmt): with fmt.indented( 'pub fn take_value_list(&mut self) -> Option {', '}'): - with fmt.indented('match *self {', '}'): - for f in InstructionFormat.all_formats: - n = 'InstructionData::' + f.name - if f.has_value_list: - fmt.line( - n + ' { ref mut args, .. } => Some(args.take()),') - fmt.line('_ => None,') + m = srcgen.Match('*self') + for f in InstructionFormat.all_formats: + n = 'InstructionData::' + f.name + if f.has_value_list: + m.arm(n, ['ref mut args', '..'], 'Some(args.take())') + m.arm('_', [], 'None') + fmt.match(m) fmt.line() fmt.doc_comment( @@ -307,14 +304,12 @@ def gen_opcodes(groups, fmt): fmt.doc_comment(Instruction.ATTRIBS[attr]) with fmt.indented('pub fn {}(self) -> bool {{' .format(attr), '}'): - with fmt.indented('match self {', '}'): - for i in instrs: - if getattr(i, attr): - fmt.format( - 'Opcode::{} => true,', - i.camel_name, i.name) - - fmt.line('_ => false,') + m = srcgen.Match('self') + for i in instrs: + if getattr(i, attr): + m.arm('Opcode::' + i.camel_name, [], 'true') + m.arm('_', [], 'false') + fmt.match(m) fmt.line() fmt.line() @@ -331,9 +326,10 @@ def gen_opcodes(groups, fmt): # Generate a private opcode_name function. with fmt.indented('fn opcode_name(opc: Opcode) -> &\'static str {', '}'): - with fmt.indented('match opc {', '}'): - for i in instrs: - fmt.format('Opcode::{} => "{}",', i.camel_name, i.name) + m = srcgen.Match('opc') + for i in instrs: + m.arm('Opcode::' + i.camel_name, [], '"{}"'.format(i.name)) + fmt.match(m) fmt.line() # Generate an opcode hash table for looking up opcodes by name. diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index 1560157915..b5f0de3488 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -57,12 +57,11 @@ def gen_getter(setting, sgrp, fmt): ty = camel_case(setting.name) proto = 'pub fn {}(&self) -> {}'.format(setting.name, ty) with fmt.indented(proto + ' {', '}'): - with fmt.indented( - 'match self.bytes[{}] {{' - .format(setting.byte_offset), '}'): - for i, v in enumerate(setting.values): - fmt.line('{} => {}::{},'.format(i, ty, camel_case(v))) - fmt.line('_ => panic!("Invalid enum value"),') + m = srcgen.Match('self.bytes[{}]'.format(setting.byte_offset)) + for i, v in enumerate(setting.values): + m.arm(str(i), [], '{}::{}'.format(ty, camel_case(v))) + m.arm('_', [], 'panic!("Invalid enum value")') + fmt.match(m) else: raise AssertionError("Unknown setting kind") diff --git a/lib/cretonne/meta/srcgen.py b/lib/cretonne/meta/srcgen.py index 9761160487..2201d997d5 100644 --- a/lib/cretonne/meta/srcgen.py +++ b/lib/cretonne/meta/srcgen.py @@ -8,9 +8,10 @@ source code. from __future__ import absolute_import import sys import os +import collections try: - from typing import Any, List # noqa + from typing import Any, List, Set, Tuple # noqa except ImportError: pass @@ -146,6 +147,52 @@ class Formatter(object): for l in parse_multiline(s): self.line('/// ' + l if l else '///') + def match(self, m): + # type: (Match) -> None + """ + Add a match expression. + + Example: + + >>> f = Formatter() + >>> m = Match('x') + >>> m.arm('Orange', ['a', 'b'], 'some body') + >>> m.arm('Yellow', ['a', 'b'], 'some body') + >>> m.arm('Green', ['a', 'b'], 'different body') + >>> m.arm('Blue', ['x', 'y'], 'some body') + >>> f.match(m) + >>> f.writelines() + match x { + Orange { a, b } | + Yellow { a, b } => { + some body + } + Green { a, b } => { + different body + } + Blue { x, y } => { + some body + } + } + + """ + with self.indented('match {} {{'.format(m.expr), '}'): + for (fields, body), names in m.arms.items(): + with self.indented('', '}'): + names_left = len(names) + for name in names.keys(): + fields_str = ', '.join(fields) + if len(fields) != 0: + fields_str = '{{ {} }} '.format(fields_str) + names_left -= 1 + if names_left > 0: + suffix = '|' + else: + suffix = '=> {' + self.outdented_line(name + ' ' + fields_str + suffix) + if names_left == 0: + self.multi_line(body) + def _indent(s): # type: (str) -> int @@ -195,3 +242,36 @@ def parse_multiline(s): while trimmed and not trimmed[0]: trimmed.pop(0) return trimmed + + +class Match(object): + """ + Match formatting class. + + Match objects collect all the information needed to emit a Rust `match` + expression, automatically deduplicating overlapping identical arms. + + Example: + + >>> m = Match('x') + >>> m.arm('Orange', ['a', 'b'], 'some body') + >>> m.arm('Yellow', ['a', 'b'], 'some body') + >>> m.arm('Green', ['a', 'b'], 'different body') + >>> m.arm('Blue', ['x', 'y'], 'some body') + >>> assert(len(m.arms) == 3) + + Note that this class is ignorant of Rust types, and considers two fields + with the same name to be equivalent. + """ + + def __init__(self, expr): + # type: (str) -> None + self.expr = expr + self.arms = collections.OrderedDict() # type: collections.OrderedDict[Tuple[Tuple[str, ...], str], collections.OrderedDict[str, None]] # noqa + + def arm(self, name, fields, body): + # type: (str, List[str], str) -> None + key = (tuple(fields), body) + if key not in self.arms: + self.arms[key] = collections.OrderedDict() + self.arms[key][name] = None From 48fc161b8bc437b68e95667a3fdf6e3e49f6bfac Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 14 Mar 2018 11:26:02 -0700 Subject: [PATCH 1610/3084] Add a badge for a Gitter chat room. I don't know how much interest there will be in Gitter, but if there is some, I'm interested in trying it out. --- README.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.rst b/README.rst index 6e6ab59bb3..7f77aa62f0 100644 --- a/README.rst +++ b/README.rst @@ -16,6 +16,10 @@ machine code. :target: https://travis-ci.org/Cretonne/cretonne :alt: Build Status +.. image:: https://badges.gitter.im/gitterHQ/gitter.png + :target: https://gitter.im/Cretonne/Lobby + :alt: Gitter chat + For more information, see `the documentation `_. From bb82bac3af562aa03a00add1fed190b7440fa08c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 14 Mar 2018 12:40:20 -0700 Subject: [PATCH 1611/3084] Tweak the Gitter URLs. --- README.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.rst b/README.rst index 7f77aa62f0..12b889f56d 100644 --- a/README.rst +++ b/README.rst @@ -16,8 +16,8 @@ machine code. :target: https://travis-ci.org/Cretonne/cretonne :alt: Build Status -.. image:: https://badges.gitter.im/gitterHQ/gitter.png - :target: https://gitter.im/Cretonne/Lobby +.. image:: https://badges.gitter.im/Cretonne/cretonne.png + :target: https://gitter.im/Cretonne/Lobby/~chat :alt: Gitter chat For more information, see `the documentation From c842b9aaa149fad6aa07e434b43e108a262efbc3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 15 Mar 2018 10:13:26 -0700 Subject: [PATCH 1612/3084] Code cleanup: import OrderedDict rather than collections.OrderedDict --- lib/cretonne/meta/srcgen.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/meta/srcgen.py b/lib/cretonne/meta/srcgen.py index 2201d997d5..df5794cedb 100644 --- a/lib/cretonne/meta/srcgen.py +++ b/lib/cretonne/meta/srcgen.py @@ -8,7 +8,7 @@ source code. from __future__ import absolute_import import sys import os -import collections +from collections import OrderedDict try: from typing import Any, List, Set, Tuple # noqa @@ -267,11 +267,11 @@ class Match(object): def __init__(self, expr): # type: (str) -> None self.expr = expr - self.arms = collections.OrderedDict() # type: collections.OrderedDict[Tuple[Tuple[str, ...], str], collections.OrderedDict[str, None]] # noqa + self.arms = OrderedDict() # type: OrderedDict[Tuple[Tuple[str, ...], str], OrderedDict[str, None]] # noqa def arm(self, name, fields, body): # type: (str, List[str], str) -> None key = (tuple(fields), body) if key not in self.arms: - self.arms[key] = collections.OrderedDict() + self.arms[key] = OrderedDict() self.arms[key][name] = None From b2acd457d5a99aba90f9e3dad04cb0e1313a9739 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 15 Mar 2018 10:18:51 -0700 Subject: [PATCH 1613/3084] Use OrderedDict rather than explicit sorting. This reduces churn in the generated files, making it easier to inspect changes. --- lib/cretonne/meta/cdsl/isa.py | 5 ++--- lib/cretonne/meta/gen_encoding.py | 4 ++-- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 9fcfa440a3..064a353117 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -172,8 +172,7 @@ class TargetISA(object): """ for cpumode in self.cpumodes: self.legalize_code(cpumode.default_legalize) - for x in sorted(cpumode.type_legalize.values(), - key=lambda x: x.name): + for x in cpumode.type_legalize.values(): self.legalize_code(x) def legalize_code(self, xgrp): @@ -232,7 +231,7 @@ class CPUMode(object): # Tables for configuring legalization actions when no valid encoding # exists for an instruction. self.default_legalize = None # type: XFormGroup - self.type_legalize = dict() # type: Dict[ValueType, XFormGroup] + self.type_legalize = OrderedDict() # type: OrderedDict[ValueType, XFormGroup] # noqa def __str__(self): # type: () -> str diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index da382a1774..61ceb8d498 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -600,8 +600,8 @@ def make_tables(cpumode): table[ty][inst].encodings.append(enc) # Ensure there are level 1 table entries for all types with a custom - # legalize action. Try to be stable relative to dict ordering. - for ty in sorted(cpumode.type_legalize.keys(), key=str): + # legalize action. + for ty in cpumode.type_legalize.keys(): table[ty] return table From 00af7a28f39bede5fdea183c118866e4d864fb88 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 Aug 2017 10:43:41 -0700 Subject: [PATCH 1614/3084] Run the filetests as part of "cargo test". Refactor the filetests harness so that it can be run as part of `cargo test`. And begin reorganizing the test harness code in preparation for moving it out of the src directory. - Test subcommand files are now named `test_*.rs`. - cton-util subcommand files now just export their `run` and nothing else. - src/filetest/mod.rs now also just exports `run` and nothing else. - Tests are now run in release mode (with debug assertions enabled). --- cranelift/Cargo.toml | 2 +- cranelift/docs/testing.rst | 16 +++ cranelift/src/cat.rs | 36 +------ cranelift/src/filetest/mod.rs | 50 ++++----- .../filetest/{binemit.rs => test_binemit.rs} | 0 cranelift/src/filetest/test_cat.rs | 37 +++++++ .../filetest/{compile.rs => test_compile.rs} | 0 .../filetest/{domtree.rs => test_domtree.rs} | 0 .../{legalizer.rs => test_legalizer.rs} | 0 .../src/filetest/{licm.rs => test_licm.rs} | 0 .../filetest/{preopt.rs => test_preopt.rs} | 0 cranelift/src/filetest/test_print_cfg.rs | 37 +++++++ .../{regalloc.rs => test_regalloc.rs} | 0 .../{simple_gvn.rs => test_simple_gvn.rs} | 0 .../{verifier.rs => test_verifier.rs} | 0 cranelift/src/print_cfg.rs | 102 +----------------- cranelift/test-all.sh | 26 +++-- cranelift/tests/cton-util-test.rs | 34 ++++++ lib/cretonne/src/cfg_printer.rs | 76 +++++++++++++ lib/cretonne/src/lib.rs | 1 + 20 files changed, 242 insertions(+), 175 deletions(-) rename cranelift/src/filetest/{binemit.rs => test_binemit.rs} (100%) create mode 100644 cranelift/src/filetest/test_cat.rs rename cranelift/src/filetest/{compile.rs => test_compile.rs} (100%) rename cranelift/src/filetest/{domtree.rs => test_domtree.rs} (100%) rename cranelift/src/filetest/{legalizer.rs => test_legalizer.rs} (100%) rename cranelift/src/filetest/{licm.rs => test_licm.rs} (100%) rename cranelift/src/filetest/{preopt.rs => test_preopt.rs} (100%) create mode 100644 cranelift/src/filetest/test_print_cfg.rs rename cranelift/src/filetest/{regalloc.rs => test_regalloc.rs} (100%) rename cranelift/src/filetest/{simple_gvn.rs => test_simple_gvn.rs} (100%) rename cranelift/src/filetest/{verifier.rs => test_verifier.rs} (100%) create mode 100644 cranelift/tests/cton-util-test.rs create mode 100644 lib/cretonne/src/cfg_printer.rs diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 297e8c31b3..ee904633f9 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -23,7 +23,7 @@ docopt = "0.8.0" serde = "1.0.8" serde_derive = "1.0.8" num_cpus = "1.5.1" -tempdir="0.3.5" +tempdir = "0.3.5" term = "0.5" [workspace] diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index 2b84b1c19b..07aa8f8e16 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -324,6 +324,22 @@ Test the simple GVN pass. The simple GVN pass is run on each function, and then results are run through filecheck. +`test licm` +----------------- + +Test the LICM pass. + +The LICM pass is run on each function, and then results are run +through filecheck. + +`test preopt` +----------------- + +Test the preopt pass. + +The preopt pass is run on each function, and then results are run +through filecheck. + `test compile` -------------- diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs index 36ae818e2c..3ae9c843d8 100644 --- a/cranelift/src/cat.rs +++ b/cranelift/src/cat.rs @@ -3,12 +3,9 @@ //! Read a sequence of Cretonne IL files and print them again to stdout. This has the effect of //! normalizing formatting and removing comments. -use std::borrow::Cow; -use cretonne::ir::Function; -use cton_reader::{parse_functions, TestCommand}; +use cton_reader::parse_functions; use CommandResult; use utils::read_to_string; -use filetest::subtest::{self, SubTest, Context, Result as STResult}; pub fn run(files: Vec) -> CommandResult { for (i, f) in files.into_iter().enumerate() { @@ -37,34 +34,3 @@ fn cat_one(filename: String) -> CommandResult { Ok(()) } - -/// Object implementing the `test cat` sub-test. -/// -/// This command is used for testing the parser and function printer. It simply parses a function -/// and prints it out again. -/// -/// The result is verified by filecheck. -struct TestCat; - -pub fn subtest(parsed: &TestCommand) -> STResult> { - assert_eq!(parsed.command, "cat"); - if !parsed.options.is_empty() { - Err(format!("No options allowed on {}", parsed)) - } else { - Ok(Box::new(TestCat)) - } -} - -impl SubTest for TestCat { - fn name(&self) -> Cow { - Cow::from("cat") - } - - fn needs_verifier(&self) -> bool { - false - } - - fn run(&self, func: Cow, context: &Context) -> STResult<()> { - subtest::run_filecheck(&func.display(context.isa).to_string(), context) - } -} diff --git a/cranelift/src/filetest/mod.rs b/cranelift/src/filetest/mod.rs index 5286307811..6d7a5e39d5 100644 --- a/cranelift/src/filetest/mod.rs +++ b/cranelift/src/filetest/mod.rs @@ -7,27 +7,27 @@ use std::path::Path; use std::time; use cton_reader::TestCommand; use CommandResult; -use cat; -use print_cfg; use filetest::runner::TestRunner; -pub mod subtest; - -mod binemit; -mod compile; mod concurrent; -mod domtree; -mod legalizer; -mod licm; -mod preopt; -mod regalloc; mod runner; mod runone; -mod simple_gvn; -mod verifier; +mod subtest; + +mod test_binemit; +mod test_cat; +mod test_compile; +mod test_domtree; +mod test_legalizer; +mod test_licm; +mod test_preopt; +mod test_print_cfg; +mod test_regalloc; +mod test_simple_gvn; +mod test_verifier; /// The result of running the test in a file. -pub type TestResult = Result; +type TestResult = Result; /// Main entry point for `cton-util test`. /// @@ -59,17 +59,17 @@ pub fn run(verbose: bool, files: Vec) -> CommandResult { /// a `.cton` test file. fn new_subtest(parsed: &TestCommand) -> subtest::Result> { match parsed.command { - "binemit" => binemit::subtest(parsed), - "cat" => cat::subtest(parsed), - "compile" => compile::subtest(parsed), - "domtree" => domtree::subtest(parsed), - "legalizer" => legalizer::subtest(parsed), - "licm" => licm::subtest(parsed), - "preopt" => preopt::subtest(parsed), - "print-cfg" => print_cfg::subtest(parsed), - "regalloc" => regalloc::subtest(parsed), - "simple-gvn" => simple_gvn::subtest(parsed), - "verifier" => verifier::subtest(parsed), + "binemit" => test_binemit::subtest(parsed), + "cat" => test_cat::subtest(parsed), + "compile" => test_compile::subtest(parsed), + "domtree" => test_domtree::subtest(parsed), + "legalizer" => test_legalizer::subtest(parsed), + "licm" => test_licm::subtest(parsed), + "preopt" => test_preopt::subtest(parsed), + "print-cfg" => test_print_cfg::subtest(parsed), + "regalloc" => test_regalloc::subtest(parsed), + "simple-gvn" => test_simple_gvn::subtest(parsed), + "verifier" => test_verifier::subtest(parsed), _ => Err(format!("unknown test command '{}'", parsed.command)), } } diff --git a/cranelift/src/filetest/binemit.rs b/cranelift/src/filetest/test_binemit.rs similarity index 100% rename from cranelift/src/filetest/binemit.rs rename to cranelift/src/filetest/test_binemit.rs diff --git a/cranelift/src/filetest/test_cat.rs b/cranelift/src/filetest/test_cat.rs new file mode 100644 index 0000000000..fc4a3cac10 --- /dev/null +++ b/cranelift/src/filetest/test_cat.rs @@ -0,0 +1,37 @@ +//! The `cat` subtest. + +use std::borrow::Cow; +use cretonne::ir::Function; +use cton_reader::TestCommand; +use filetest::subtest::{self, SubTest, Context, Result as STResult}; + +/// Object implementing the `test cat` sub-test. +/// +/// This command is used for testing the parser and function printer. It simply parses a function +/// and prints it out again. +/// +/// The result is verified by filecheck. +struct TestCat; + +pub fn subtest(parsed: &TestCommand) -> STResult> { + assert_eq!(parsed.command, "cat"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestCat)) + } +} + +impl SubTest for TestCat { + fn name(&self) -> Cow { + Cow::from("cat") + } + + fn needs_verifier(&self) -> bool { + false + } + + fn run(&self, func: Cow, context: &Context) -> STResult<()> { + subtest::run_filecheck(&func.display(context.isa).to_string(), context) + } +} diff --git a/cranelift/src/filetest/compile.rs b/cranelift/src/filetest/test_compile.rs similarity index 100% rename from cranelift/src/filetest/compile.rs rename to cranelift/src/filetest/test_compile.rs diff --git a/cranelift/src/filetest/domtree.rs b/cranelift/src/filetest/test_domtree.rs similarity index 100% rename from cranelift/src/filetest/domtree.rs rename to cranelift/src/filetest/test_domtree.rs diff --git a/cranelift/src/filetest/legalizer.rs b/cranelift/src/filetest/test_legalizer.rs similarity index 100% rename from cranelift/src/filetest/legalizer.rs rename to cranelift/src/filetest/test_legalizer.rs diff --git a/cranelift/src/filetest/licm.rs b/cranelift/src/filetest/test_licm.rs similarity index 100% rename from cranelift/src/filetest/licm.rs rename to cranelift/src/filetest/test_licm.rs diff --git a/cranelift/src/filetest/preopt.rs b/cranelift/src/filetest/test_preopt.rs similarity index 100% rename from cranelift/src/filetest/preopt.rs rename to cranelift/src/filetest/test_preopt.rs diff --git a/cranelift/src/filetest/test_print_cfg.rs b/cranelift/src/filetest/test_print_cfg.rs new file mode 100644 index 0000000000..1be9987087 --- /dev/null +++ b/cranelift/src/filetest/test_print_cfg.rs @@ -0,0 +1,37 @@ +//! The `print-cfg` sub-command. +//! +//! Read a series of Cretonne IL files and print their control flow graphs +//! in graphviz format. + +use std::borrow::Cow; + +use cretonne::ir::Function; +use cretonne::cfg_printer::CFGPrinter; +use cton_reader::TestCommand; +use filetest::subtest::{self, SubTest, Context, Result as STResult}; + +/// Object implementing the `test print-cfg` sub-test. +struct TestPrintCfg; + +pub fn subtest(parsed: &TestCommand) -> STResult> { + assert_eq!(parsed.command, "print-cfg"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestPrintCfg)) + } +} + +impl SubTest for TestPrintCfg { + fn name(&self) -> Cow { + Cow::from("print-cfg") + } + + fn needs_verifier(&self) -> bool { + false + } + + fn run(&self, func: Cow, context: &Context) -> STResult<()> { + subtest::run_filecheck(&CFGPrinter::new(&func).to_string(), context) + } +} diff --git a/cranelift/src/filetest/regalloc.rs b/cranelift/src/filetest/test_regalloc.rs similarity index 100% rename from cranelift/src/filetest/regalloc.rs rename to cranelift/src/filetest/test_regalloc.rs diff --git a/cranelift/src/filetest/simple_gvn.rs b/cranelift/src/filetest/test_simple_gvn.rs similarity index 100% rename from cranelift/src/filetest/simple_gvn.rs rename to cranelift/src/filetest/test_simple_gvn.rs diff --git a/cranelift/src/filetest/verifier.rs b/cranelift/src/filetest/test_verifier.rs similarity index 100% rename from cranelift/src/filetest/verifier.rs rename to cranelift/src/filetest/test_verifier.rs diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index df624993ad..08ab439d05 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -3,15 +3,9 @@ //! Read a series of Cretonne IL files and print their control flow graphs //! in graphviz format. -use std::borrow::Cow; -use std::fmt::{Result, Write, Display, Formatter}; - use CommandResult; -use cretonne::flowgraph::ControlFlowGraph; -use cretonne::ir::Function; -use cretonne::ir::instructions::BranchInfo; -use cton_reader::{parse_functions, TestCommand}; -use filetest::subtest::{self, SubTest, Context, Result as STResult}; +use cretonne::cfg_printer::CFGPrinter; +use cton_reader::parse_functions; use utils::read_to_string; pub fn run(files: Vec) -> CommandResult { @@ -24,72 +18,6 @@ pub fn run(files: Vec) -> CommandResult { Ok(()) } -struct CFGPrinter<'a> { - func: &'a Function, - cfg: ControlFlowGraph, -} - -impl<'a> CFGPrinter<'a> { - pub fn new(func: &'a Function) -> CFGPrinter<'a> { - CFGPrinter { - func, - cfg: ControlFlowGraph::with_function(func), - } - } - - /// Write the CFG for this function to `w`. - pub fn write(&self, w: &mut Write) -> Result { - self.header(w)?; - self.ebb_nodes(w)?; - self.cfg_connections(w)?; - writeln!(w, "}}") - } - - fn header(&self, w: &mut Write) -> Result { - writeln!(w, "digraph \"{}\" {{", self.func.name)?; - if let Some(entry) = self.func.layout.entry_block() { - writeln!(w, " {{rank=min; {}}}", entry)?; - } - Ok(()) - } - - fn ebb_nodes(&self, w: &mut Write) -> Result { - for ebb in &self.func.layout { - write!(w, " {} [shape=record, label=\"{{{}", ebb, ebb)?; - // Add all outgoing branch instructions to the label. - for inst in self.func.layout.ebb_insts(ebb) { - let idata = &self.func.dfg[inst]; - match idata.analyze_branch(&self.func.dfg.value_lists) { - BranchInfo::SingleDest(dest, _) => { - write!(w, " | <{}>{} {}", inst, idata.opcode(), dest)? - } - BranchInfo::Table(table) => { - write!(w, " | <{}>{} {}", inst, idata.opcode(), table)? - } - BranchInfo::NotABranch => {} - } - } - writeln!(w, "}}\"]")? - } - Ok(()) - } - - fn cfg_connections(&self, w: &mut Write) -> Result { - for ebb in &self.func.layout { - for (parent, inst) in self.cfg.pred_iter(ebb) { - writeln!(w, " {}:{} -> {}", parent, inst, ebb)?; - } - } - Ok(()) - } -} - -impl<'a> Display for CFGPrinter<'a> { - fn fmt(&self, f: &mut Formatter) -> Result { - self.write(f) - } -} - fn print_cfg(filename: String) -> CommandResult { let buffer = read_to_string(&filename).map_err( |e| format!("{}: {}", filename, e), @@ -107,29 +35,3 @@ fn print_cfg(filename: String) -> CommandResult { Ok(()) } - -/// Object implementing the `test print-cfg` sub-test. -struct TestPrintCfg; - -pub fn subtest(parsed: &TestCommand) -> STResult> { - assert_eq!(parsed.command, "print-cfg"); - if !parsed.options.is_empty() { - Err(format!("No options allowed on {}", parsed)) - } else { - Ok(Box::new(TestPrintCfg)) - } -} - -impl SubTest for TestPrintCfg { - fn name(&self) -> Cow { - Cow::from("print-cfg") - } - - fn needs_verifier(&self) -> bool { - false - } - - fn run(&self, func: Cow, context: &Context) -> STResult<()> { - subtest::run_filecheck(&CFGPrinter::new(&func).to_string(), context) - } -} diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 2402eb7a39..fec95573f4 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -43,21 +43,19 @@ if [ -n "$needcheck" ]; then fi cd "$topdir" -banner "Rust unit tests" -cargo test --all -# Build cton-util for parser testing. -cd "$topdir" -banner "Rust documentation" -echo "open $topdir/target/doc/cretonne/index.html" +# Make sure the code builds in debug mode. +banner "Rust debug build" +cargo build + +# Make sure the code builds in release mode, and run the unit tests. We run +# these in release mode for speed, but note that the top-level Cargo.toml file +# does enable debug assertions in release builds. +banner "Rust release build and unit tests" +cargo test --all --release + +# Make sure the documentation builds. +banner "Rust documentation: $topdir/target/doc/cretonne/index.html" cargo doc -banner "Rust release build" -cargo build --release - -export CTONUTIL="$topdir/target/release/cton-util" - -cd "$topdir" -banner "File tests" -"$CTONUTIL" test filetests docs banner "OK" diff --git a/cranelift/tests/cton-util-test.rs b/cranelift/tests/cton-util-test.rs new file mode 100644 index 0000000000..d9870351d1 --- /dev/null +++ b/cranelift/tests/cton-util-test.rs @@ -0,0 +1,34 @@ +//! Run `cton-util test` on all available testcases. + +use std::process::{Command, Output}; +use std::env; +use std::path::PathBuf; +use std::io::{self, Write}; + +/// Returns the target directory, where we can find build artifacts +/// and such for the current configuration. +fn get_target_dir() -> PathBuf { + let mut path = env::current_exe().unwrap(); + path.pop(); // chop off exe name + path.pop(); // chop off deps name + path +} + +#[test] +fn cton_util_test() { + let mut cmd = Command::new(&get_target_dir().join("cton-util")); + cmd.arg("test"); + + // We have testcases in the following directories: + cmd.arg("filetests"); + cmd.arg("docs"); + + let Output { + status, + stdout, + stderr, + } = cmd.output().unwrap(); + io::stdout().write(&stdout).unwrap(); + io::stderr().write(&stderr).unwrap(); + assert!(status.success(), "failed with exit status {}", status); +} diff --git a/lib/cretonne/src/cfg_printer.rs b/lib/cretonne/src/cfg_printer.rs new file mode 100644 index 0000000000..736d12f0fc --- /dev/null +++ b/lib/cretonne/src/cfg_printer.rs @@ -0,0 +1,76 @@ +//! The `CFGPrinter` utility. + +use std::fmt::{Result, Write, Display, Formatter}; + +use flowgraph::ControlFlowGraph; +use ir::Function; +use ir::instructions::BranchInfo; + +/// A utility for pretty-printing the CFG of a `Function`. +pub struct CFGPrinter<'a> { + func: &'a Function, + cfg: ControlFlowGraph, +} + +/// A utility for pretty-printing the CFG of a `Function`. +impl<'a> CFGPrinter<'a> { + /// Create a new CFGPrinter. + pub fn new(func: &'a Function) -> CFGPrinter<'a> { + CFGPrinter { + func, + cfg: ControlFlowGraph::with_function(func), + } + } + + /// Write the CFG for this function to `w`. + pub fn write(&self, w: &mut Write) -> Result { + self.header(w)?; + self.ebb_nodes(w)?; + self.cfg_connections(w)?; + writeln!(w, "}}") + } + + fn header(&self, w: &mut Write) -> Result { + writeln!(w, "digraph \"{}\" {{", self.func.name)?; + if let Some(entry) = self.func.layout.entry_block() { + writeln!(w, " {{rank=min; {}}}", entry)?; + } + Ok(()) + } + + fn ebb_nodes(&self, w: &mut Write) -> Result { + for ebb in &self.func.layout { + write!(w, " {} [shape=record, label=\"{{{}", ebb, ebb)?; + // Add all outgoing branch instructions to the label. + for inst in self.func.layout.ebb_insts(ebb) { + let idata = &self.func.dfg[inst]; + match idata.analyze_branch(&self.func.dfg.value_lists) { + BranchInfo::SingleDest(dest, _) => { + write!(w, " | <{}>{} {}", inst, idata.opcode(), dest)? + } + BranchInfo::Table(table) => { + write!(w, " | <{}>{} {}", inst, idata.opcode(), table)? + } + BranchInfo::NotABranch => {} + } + } + writeln!(w, "}}\"]")? + } + Ok(()) + } + + fn cfg_connections(&self, w: &mut Write) -> Result { + for ebb in &self.func.layout { + for (parent, inst) in self.cfg.pred_iter(ebb) { + writeln!(w, " {}:{} -> {}", parent, inst, ebb)?; + } + } + Ok(()) + } +} + +impl<'a> Display for CFGPrinter<'a> { + fn fmt(&self, f: &mut Formatter) -> Result { + self.write(f) + } +} diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index ab7bb51f03..b924f5d9e5 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -19,6 +19,7 @@ pub mod entity; pub mod bforest; pub mod binemit; +pub mod cfg_printer; pub mod cursor; pub mod dominator_tree; pub mod flowgraph; From 965b93bd2a7b754cc044c0b8cd38bf7fb9ae2076 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 15 Mar 2018 13:00:29 -0700 Subject: [PATCH 1615/3084] Move the filetest harness into its own crate. This allows us to run the tests via a library call rather than just as a command execution. And, it's a step toward a broader goal, which is to keep the code in the top-level src directory minimal, with important functionality exposed as crates. --- cranelift/Cargo.toml | 2 +- cranelift/docs/testing.rst | 3 + cranelift/src/compile.rs | 3 +- cranelift/src/cton-util.rs | 6 +- cranelift/src/rsfilecheck.rs | 4 ++ cranelift/src/utils.rs | 58 ------------------- cranelift/src/wasm.rs | 3 +- cranelift/test-all.sh | 9 ++- cranelift/tests/cton-util-test.rs | 34 ----------- cranelift/tests/filetests.rs | 7 +++ lib/cretonne/src/lib.rs | 1 + lib/cretonne/src/print_errors.rs | 33 +++++++++++ lib/filetests/Cargo.toml | 18 ++++++ .../filetests/src}/concurrent.rs | 2 +- .../mod.rs => lib/filetests/src/lib.rs | 16 +++-- lib/filetests/src/match_directive.rs | 27 +++++++++ .../filetest => lib/filetests/src}/runner.rs | 11 ++-- .../filetest => lib/filetests/src}/runone.rs | 16 ++++- .../filetest => lib/filetests/src}/subtest.rs | 0 .../filetests/src}/test_binemit.rs | 5 +- .../filetests/src}/test_cat.rs | 2 +- .../filetests/src}/test_compile.rs | 4 +- .../filetests/src}/test_domtree.rs | 6 +- .../filetests/src}/test_legalizer.rs | 4 +- .../filetests/src}/test_licm.rs | 4 +- .../filetests/src}/test_preopt.rs | 4 +- .../filetests/src}/test_print_cfg.rs | 2 +- .../filetests/src}/test_regalloc.rs | 4 +- .../filetests/src}/test_simple_gvn.rs | 4 +- .../filetests/src}/test_verifier.rs | 6 +- lib/wasm/tests/wasm_testsuite.rs | 26 +-------- 31 files changed, 161 insertions(+), 163 deletions(-) delete mode 100644 cranelift/tests/cton-util-test.rs create mode 100644 cranelift/tests/filetests.rs create mode 100644 lib/cretonne/src/print_errors.rs create mode 100644 lib/filetests/Cargo.toml rename {cranelift/src/filetest => lib/filetests/src}/concurrent.rs (99%) rename cranelift/src/filetest/mod.rs => lib/filetests/src/lib.rs (85%) create mode 100644 lib/filetests/src/match_directive.rs rename {cranelift/src/filetest => lib/filetests/src}/runner.rs (98%) rename {cranelift/src/filetest => lib/filetests/src}/runone.rs (91%) rename {cranelift/src/filetest => lib/filetests/src}/subtest.rs (100%) rename {cranelift/src/filetest => lib/filetests/src}/test_binemit.rs (98%) rename {cranelift/src/filetest => lib/filetests/src}/test_cat.rs (92%) rename {cranelift/src/filetest => lib/filetests/src}/test_compile.rs (96%) rename {cranelift/src/filetest => lib/filetests/src}/test_domtree.rs (97%) rename {cranelift/src/filetest => lib/filetests/src}/test_legalizer.rs (93%) rename {cranelift/src/filetest => lib/filetests/src}/test_licm.rs (93%) rename {cranelift/src/filetest => lib/filetests/src}/test_preopt.rs (92%) rename {cranelift/src/filetest => lib/filetests/src}/test_print_cfg.rs (93%) rename {cranelift/src/filetest => lib/filetests/src}/test_regalloc.rs (94%) rename {cranelift/src/filetest => lib/filetests/src}/test_simple_gvn.rs (93%) rename {cranelift/src/filetest => lib/filetests/src}/test_verifier.rs (96%) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index ee904633f9..064a2ee5ab 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -18,11 +18,11 @@ cretonne-reader = { path = "lib/reader", version = "0.3.4" } cretonne-frontend = { path = "lib/frontend", version = "0.3.4" } cretonne-wasm = { path = "lib/wasm", version = "0.3.4" } cretonne-native = { path = "lib/native", version = "0.3.4" } +cretonne-filetests = { path = "lib/filetests", version = "0.3.4" } filecheck = { path = "lib/filecheck" } docopt = "0.8.0" serde = "1.0.8" serde_derive = "1.0.8" -num_cpus = "1.5.1" tempdir = "0.3.5" term = "0.5" diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index 07aa8f8e16..6ddc53ed84 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -136,6 +136,9 @@ This example will run the legalizer test twice. Both runs will have ``opt_level=best``, but they will have different ``is_64bit`` settings. The 32-bit run will also have the RISC-V specific flag ``supports_m`` disabled. +The filetests are run automatically as part of `cargo test`, and they can +also be run manually with the `cton-util test` command. + Filecheck --------- diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 9a9644b1cf..9047f055fe 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -7,8 +7,9 @@ use std::path::PathBuf; use cretonne::Context; use cretonne::settings::FlagsOrIsa; use cretonne::{binemit, ir}; +use cretonne::print_errors::pretty_error; use std::path::Path; -use utils::{pretty_error, read_to_string, parse_sets_and_isa}; +use utils::{read_to_string, parse_sets_and_isa}; struct PrintRelocs { flag_print: bool, diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 9f6891489e..54d7d38d4d 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -1,12 +1,11 @@ -#[macro_use(dbg)] extern crate cretonne; extern crate cton_reader; extern crate cton_wasm; +extern crate cton_filetests; extern crate docopt; #[macro_use] extern crate serde_derive; extern crate filecheck; -extern crate num_cpus; extern crate tempdir; extern crate term; @@ -16,7 +15,6 @@ use std::io::{self, Write}; use std::process; mod utils; -mod filetest; mod cat; mod print_cfg; mod rsfilecheck; @@ -88,7 +86,7 @@ fn cton_util() -> CommandResult { // Find the sub-command to execute. let result = if args.cmd_test { - filetest::run(args.flag_verbose, args.arg_file) + cton_filetests::run(args.flag_verbose, args.arg_file).map(|_time| ()) } else if args.cmd_cat { cat::run(args.arg_file) } else if args.cmd_filecheck { diff --git a/cranelift/src/rsfilecheck.rs b/cranelift/src/rsfilecheck.rs index 46605bb368..57b9bb9e8f 100644 --- a/cranelift/src/rsfilecheck.rs +++ b/cranelift/src/rsfilecheck.rs @@ -1,3 +1,7 @@ +//! The `filecheck` sub-command. +//! +//! This file is named to avoid a name collision with the filecheck crate. + use CommandResult; use utils::read_to_string; use filecheck::{CheckerBuilder, Checker, NO_VARIABLES}; diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index 25c6dbf067..7e45e99436 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -1,13 +1,9 @@ //! Utility functions. -use cretonne::ir::entities::AnyEntity; -use cretonne::{ir, verifier}; -use cretonne::result::CtonError; use cretonne::isa::TargetIsa; use cretonne::settings::{self, FlagsOrIsa}; use cretonne::isa; use cton_reader::{parse_options, Location}; -use std::fmt::Write; use std::fs::File; use std::io::{self, Read}; use std::path::Path; @@ -28,51 +24,6 @@ pub fn read_to_end>(path: P) -> io::Result> { Ok(buffer) } -/// Look for a directive in a comment string. -/// The directive is of the form "foo:" and should follow the leading `;` in the comment: -/// -/// ; dominates: ebb3 ebb4 -/// -/// Return the comment text following the directive. -pub fn match_directive<'a>(comment: &'a str, directive: &str) -> Option<&'a str> { - assert!( - directive.ends_with(':'), - "Directive must include trailing colon" - ); - let text = comment.trim_left_matches(';').trim_left(); - if text.starts_with(directive) { - Some(text[directive.len()..].trim()) - } else { - None - } -} - -/// Pretty-print a verifier error. -pub fn pretty_verifier_error( - func: &ir::Function, - isa: Option<&TargetIsa>, - err: verifier::Error, -) -> String { - let mut msg = err.to_string(); - match err.location { - AnyEntity::Inst(inst) => { - write!(msg, "\n{}: {}\n\n", inst, func.dfg.display_inst(inst, isa)).unwrap() - } - _ => msg.push('\n'), - } - write!(msg, "{}", func.display(isa)).unwrap(); - msg -} - -/// Pretty-print a Cretonne error. -pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CtonError) -> String { - if let CtonError::Verifier(e) = err { - pretty_verifier_error(func, isa, e) - } else { - err.to_string() - } -} - /// Like `FlagsOrIsa`, but holds ownership. pub enum OwnedFlagsOrIsa { Flags(settings::Flags), @@ -119,12 +70,3 @@ pub fn parse_sets_and_isa( Ok(OwnedFlagsOrIsa::Flags(settings::Flags::new(&flag_builder))) } } - -#[test] -fn test_match_directive() { - assert_eq!(match_directive("; foo: bar ", "foo:"), Some("bar")); - assert_eq!(match_directive(" foo:bar", "foo:"), Some("bar")); - assert_eq!(match_directive("foo:bar", "foo:"), Some("bar")); - assert_eq!(match_directive(";x foo: bar", "foo:"), None); - assert_eq!(match_directive(";;; foo: bar", "foo:"), Some("bar")); -} diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index ab6a245d3a..3ab877bf34 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -6,6 +6,7 @@ use cton_wasm::{translate_module, DummyEnvironment, ModuleEnvironment}; use std::path::PathBuf; use cretonne::Context; use cretonne::settings::FlagsOrIsa; +use cretonne::print_errors::{pretty_error, pretty_verifier_error}; use std::fs::File; use std::error::Error; use std::io; @@ -13,7 +14,7 @@ use std::path::Path; use std::process::Command; use tempdir::TempDir; use term; -use utils::{pretty_verifier_error, pretty_error, parse_sets_and_isa, read_to_end}; +use utils::{parse_sets_and_isa, read_to_end}; macro_rules! vprintln { ($x: expr, $($tts:tt)*) => { diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index fec95573f4..7b86495903 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -3,11 +3,10 @@ set -euo pipefail # This is the top-level test script: # -# - Build documentation for Rust code in 'src/tools/target/doc'. -# - Run unit tests for all Rust crates. -# - Make a debug build of all crates. -# - Make a release build of cton-util. -# - Run file-level tests with the release build of cton-util. +# - Make a debug build. +# - Make a release build. +# - Run unit tests for all Rust crates (including the filetests) +# - Build API documentation. # # All tests run by this script should be passing at all times. diff --git a/cranelift/tests/cton-util-test.rs b/cranelift/tests/cton-util-test.rs deleted file mode 100644 index d9870351d1..0000000000 --- a/cranelift/tests/cton-util-test.rs +++ /dev/null @@ -1,34 +0,0 @@ -//! Run `cton-util test` on all available testcases. - -use std::process::{Command, Output}; -use std::env; -use std::path::PathBuf; -use std::io::{self, Write}; - -/// Returns the target directory, where we can find build artifacts -/// and such for the current configuration. -fn get_target_dir() -> PathBuf { - let mut path = env::current_exe().unwrap(); - path.pop(); // chop off exe name - path.pop(); // chop off deps name - path -} - -#[test] -fn cton_util_test() { - let mut cmd = Command::new(&get_target_dir().join("cton-util")); - cmd.arg("test"); - - // We have testcases in the following directories: - cmd.arg("filetests"); - cmd.arg("docs"); - - let Output { - status, - stdout, - stderr, - } = cmd.output().unwrap(); - io::stdout().write(&stdout).unwrap(); - io::stderr().write(&stderr).unwrap(); - assert!(status.success(), "failed with exit status {}", status); -} diff --git a/cranelift/tests/filetests.rs b/cranelift/tests/filetests.rs new file mode 100644 index 0000000000..97fbeb296f --- /dev/null +++ b/cranelift/tests/filetests.rs @@ -0,0 +1,7 @@ +extern crate cton_filetests; + +#[test] +fn filetests() { + // Run all the filetests in the following directories. + cton_filetests::run(false, vec!["filetests".into(), "docs".into()]).expect("test harness"); +} diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index b924f5d9e5..f317ed9f85 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -27,6 +27,7 @@ pub mod ir; pub mod isa; pub mod loop_analysis; pub mod packed_option; +pub mod print_errors; pub mod result; pub mod settings; pub mod timing; diff --git a/lib/cretonne/src/print_errors.rs b/lib/cretonne/src/print_errors.rs new file mode 100644 index 0000000000..abd22f3e6b --- /dev/null +++ b/lib/cretonne/src/print_errors.rs @@ -0,0 +1,33 @@ +//! Utility routines for pretty-printing error messages. + +use ir; +use verifier; +use result::CtonError; +use isa::TargetIsa; +use std::fmt::Write; + +/// Pretty-print a verifier error. +pub fn pretty_verifier_error( + func: &ir::Function, + isa: Option<&TargetIsa>, + err: verifier::Error, +) -> String { + let mut msg = err.to_string(); + match err.location { + ir::entities::AnyEntity::Inst(inst) => { + write!(msg, "\n{}: {}\n\n", inst, func.dfg.display_inst(inst, isa)).unwrap() + } + _ => msg.push('\n'), + } + write!(msg, "{}", func.display(isa)).unwrap(); + msg +} + +/// Pretty-print a Cretonne error. +pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CtonError) -> String { + if let CtonError::Verifier(e) = err { + pretty_verifier_error(func, isa, e) + } else { + err.to_string() + } +} diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml new file mode 100644 index 0000000000..215f947e74 --- /dev/null +++ b/lib/filetests/Cargo.toml @@ -0,0 +1,18 @@ +[package] +name = "cretonne-filetests" +authors = ["The Cretonne Project Developers"] +version = "0.3.4" +description = "Test driver and implementations of the filetest commands" +license = "Apache-2.0" +documentation = "http://cretonne.readthedocs.io/en/latest/testing.html#file-tests" +repository = "https://github.com/Cretonne/cretonne" +publish = false + +[lib] +name = "cton_filetests" + +[dependencies] +cretonne = { path = "../cretonne", version = "0.3.4" } +cretonne-reader = { path = "../reader", version = "0.3.4" } +filecheck = { path = "../filecheck", version = "0.1.0" } +num_cpus = "1.5.1" diff --git a/cranelift/src/filetest/concurrent.rs b/lib/filetests/src/concurrent.rs similarity index 99% rename from cranelift/src/filetest/concurrent.rs rename to lib/filetests/src/concurrent.rs index a651c14071..6a18bb2390 100644 --- a/cranelift/src/filetest/concurrent.rs +++ b/lib/filetests/src/concurrent.rs @@ -11,7 +11,7 @@ use std::sync::{Arc, Mutex}; use std::thread; use std::time::Duration; use num_cpus; -use filetest::{TestResult, runone}; +use {TestResult, runone}; // Request sent to worker threads contains jobid and path. struct Request(usize, PathBuf); diff --git a/cranelift/src/filetest/mod.rs b/lib/filetests/src/lib.rs similarity index 85% rename from cranelift/src/filetest/mod.rs rename to lib/filetests/src/lib.rs index 6d7a5e39d5..36d321358c 100644 --- a/cranelift/src/filetest/mod.rs +++ b/lib/filetests/src/lib.rs @@ -1,18 +1,24 @@ //! File tests. //! -//! This module contains the main driver for `cton-util test` as well as implementations of the -//! available test commands. +//! This crate contains the main test driver as well as implementations of the +//! available filetest commands. + +#[macro_use(dbg)] +extern crate cretonne; +extern crate cton_reader; +extern crate filecheck; +extern crate num_cpus; use std::path::Path; use std::time; use cton_reader::TestCommand; -use CommandResult; -use filetest::runner::TestRunner; +use runner::TestRunner; mod concurrent; mod runner; mod runone; mod subtest; +mod match_directive; mod test_binemit; mod test_cat; @@ -38,7 +44,7 @@ type TestResult = Result; /// Directories are scanned recursively for test cases ending in `.cton`. These test cases are /// executed on background threads. /// -pub fn run(verbose: bool, files: Vec) -> CommandResult { +pub fn run(verbose: bool, files: Vec) -> TestResult { let mut runner = TestRunner::new(verbose); for path in files.iter().map(Path::new) { diff --git a/lib/filetests/src/match_directive.rs b/lib/filetests/src/match_directive.rs new file mode 100644 index 0000000000..9a8d94a4df --- /dev/null +++ b/lib/filetests/src/match_directive.rs @@ -0,0 +1,27 @@ +/// Look for a directive in a comment string. +/// The directive is of the form "foo:" and should follow the leading `;` in the comment: +/// +/// ; dominates: ebb3 ebb4 +/// +/// Return the comment text following the directive. +pub fn match_directive<'a>(comment: &'a str, directive: &str) -> Option<&'a str> { + assert!( + directive.ends_with(':'), + "Directive must include trailing colon" + ); + let text = comment.trim_left_matches(';').trim_left(); + if text.starts_with(directive) { + Some(text[directive.len()..].trim()) + } else { + None + } +} + +#[test] +fn test_match_directive() { + assert_eq!(match_directive("; foo: bar ", "foo:"), Some("bar")); + assert_eq!(match_directive(" foo:bar", "foo:"), Some("bar")); + assert_eq!(match_directive("foo:bar", "foo:"), Some("bar")); + assert_eq!(match_directive(";x foo: bar", "foo:"), None); + assert_eq!(match_directive(";;; foo: bar", "foo:"), Some("bar")); +} diff --git a/cranelift/src/filetest/runner.rs b/lib/filetests/src/runner.rs similarity index 98% rename from cranelift/src/filetest/runner.rs rename to lib/filetests/src/runner.rs index 09ec3a19a9..ff40ed81e2 100644 --- a/cranelift/src/filetest/runner.rs +++ b/lib/filetests/src/runner.rs @@ -7,9 +7,9 @@ use std::error::Error; use std::fmt::{self, Display}; use std::ffi::OsStr; use std::path::{Path, PathBuf}; -use filetest::{TestResult, runone}; -use filetest::concurrent::{ConcurrentRunner, Reply}; -use CommandResult; +use std::time; +use {TestResult, runone}; +use concurrent::{ConcurrentRunner, Reply}; // Timeout in seconds when we're not making progress. const TIMEOUT_PANIC: usize = 10; @@ -323,14 +323,15 @@ impl TestRunner { } /// Scan pushed directories for tests and run them. - pub fn run(&mut self) -> CommandResult { + pub fn run(&mut self) -> TestResult { + let started = time::Instant::now(); self.scan_dirs(); self.schedule_jobs(); self.drain_threads(); self.report_slow_tests(); println!("{} tests", self.tests.len()); match self.errors { - 0 => Ok(()), + 0 => Ok(started.elapsed()), 1 => Err("1 failure".to_string()), n => Err(format!("{} failures", n)), } diff --git a/cranelift/src/filetest/runone.rs b/lib/filetests/src/runone.rs similarity index 91% rename from cranelift/src/filetest/runone.rs rename to lib/filetests/src/runone.rs index 1dc8eb9849..fcc01e4f95 100644 --- a/cranelift/src/filetest/runone.rs +++ b/lib/filetests/src/runone.rs @@ -3,16 +3,26 @@ use std::borrow::Cow; use std::path::Path; use std::time; +use std::io::{self, Read}; +use std::fs; use cretonne::ir::Function; use cretonne::isa::TargetIsa; use cretonne::settings::Flags; use cretonne::timing; use cretonne::verify_function; +use cretonne::print_errors::pretty_verifier_error; use cton_reader::parse_test; use cton_reader::IsaSpec; -use utils::{read_to_string, pretty_verifier_error}; -use filetest::{TestResult, new_subtest}; -use filetest::subtest::{SubTest, Context, Result}; +use {TestResult, new_subtest}; +use subtest::{SubTest, Context, Result}; + +/// Read an entire file into a string. +fn read_to_string>(path: P) -> io::Result { + let mut file = fs::File::open(path)?; + let mut buffer = String::new(); + file.read_to_string(&mut buffer)?; + Ok(buffer) +} /// Load `path` and run the test in it. /// diff --git a/cranelift/src/filetest/subtest.rs b/lib/filetests/src/subtest.rs similarity index 100% rename from cranelift/src/filetest/subtest.rs rename to lib/filetests/src/subtest.rs diff --git a/cranelift/src/filetest/test_binemit.rs b/lib/filetests/src/test_binemit.rs similarity index 98% rename from cranelift/src/filetest/test_binemit.rs rename to lib/filetests/src/test_binemit.rs index 3c52a63b2b..c3284c1423 100644 --- a/cranelift/src/filetest/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -11,9 +11,10 @@ use cretonne::dbg::DisplayList; use cretonne::ir; use cretonne::ir::entities::AnyEntity; use cretonne::binemit::RegDiversions; +use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use filetest::subtest::{SubTest, Context, Result}; -use utils::{match_directive, pretty_error}; +use subtest::{SubTest, Context, Result}; +use match_directive::match_directive; struct TestBinEmit; diff --git a/cranelift/src/filetest/test_cat.rs b/lib/filetests/src/test_cat.rs similarity index 92% rename from cranelift/src/filetest/test_cat.rs rename to lib/filetests/src/test_cat.rs index fc4a3cac10..fbbcbd034b 100644 --- a/cranelift/src/filetest/test_cat.rs +++ b/lib/filetests/src/test_cat.rs @@ -3,7 +3,7 @@ use std::borrow::Cow; use cretonne::ir::Function; use cton_reader::TestCommand; -use filetest::subtest::{self, SubTest, Context, Result as STResult}; +use subtest::{self, SubTest, Context, Result as STResult}; /// Object implementing the `test cat` sub-test. /// diff --git a/cranelift/src/filetest/test_compile.rs b/lib/filetests/src/test_compile.rs similarity index 96% rename from cranelift/src/filetest/test_compile.rs rename to lib/filetests/src/test_compile.rs index 8c8db46591..3772081a16 100644 --- a/cranelift/src/filetest/test_compile.rs +++ b/lib/filetests/src/test_compile.rs @@ -5,11 +5,11 @@ use cretonne::binemit; use cretonne::ir; use cretonne; +use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use filetest::subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{SubTest, Context, Result, run_filecheck}; use std::borrow::Cow; use std::fmt::Write; -use utils::pretty_error; struct TestCompile; diff --git a/cranelift/src/filetest/test_domtree.rs b/lib/filetests/src/test_domtree.rs similarity index 97% rename from cranelift/src/filetest/test_domtree.rs rename to lib/filetests/src/test_domtree.rs index 9c0fe5a841..6eae539644 100644 --- a/cranelift/src/filetest/test_domtree.rs +++ b/lib/filetests/src/test_domtree.rs @@ -2,7 +2,9 @@ //! //! The `test domtree` test command looks for annotations on instructions like this: //! +//! ```cton //! jump ebb3 ; dominates: ebb3 +//! ``` //! //! This annotation means that the jump instruction is expected to be the immediate dominator of //! `ebb3`. @@ -15,12 +17,12 @@ use cretonne::flowgraph::ControlFlowGraph; use cretonne::ir::Function; use cretonne::ir::entities::AnyEntity; use cton_reader::TestCommand; -use filetest::subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{SubTest, Context, Result, run_filecheck}; use std::borrow::{Borrow, Cow}; use std::collections::HashMap; use std::fmt::{self, Write}; use std::result; -use utils::match_directive; +use match_directive::match_directive; struct TestDomtree; diff --git a/cranelift/src/filetest/test_legalizer.rs b/lib/filetests/src/test_legalizer.rs similarity index 93% rename from cranelift/src/filetest/test_legalizer.rs rename to lib/filetests/src/test_legalizer.rs index e79aada799..c8d03e4c2a 100644 --- a/cranelift/src/filetest/test_legalizer.rs +++ b/lib/filetests/src/test_legalizer.rs @@ -6,10 +6,10 @@ use std::borrow::Cow; use cretonne; use cretonne::ir::Function; +use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use filetest::subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{SubTest, Context, Result, run_filecheck}; use std::fmt::Write; -use utils::pretty_error; struct TestLegalizer; diff --git a/cranelift/src/filetest/test_licm.rs b/lib/filetests/src/test_licm.rs similarity index 93% rename from cranelift/src/filetest/test_licm.rs rename to lib/filetests/src/test_licm.rs index 32be450476..f7d4397b5a 100644 --- a/cranelift/src/filetest/test_licm.rs +++ b/lib/filetests/src/test_licm.rs @@ -7,11 +7,11 @@ use cretonne::ir::Function; use cretonne; +use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use filetest::subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{SubTest, Context, Result, run_filecheck}; use std::borrow::Cow; use std::fmt::Write; -use utils::pretty_error; struct TestLICM; diff --git a/cranelift/src/filetest/test_preopt.rs b/lib/filetests/src/test_preopt.rs similarity index 92% rename from cranelift/src/filetest/test_preopt.rs rename to lib/filetests/src/test_preopt.rs index 60d03f8207..c141d28480 100644 --- a/cranelift/src/filetest/test_preopt.rs +++ b/lib/filetests/src/test_preopt.rs @@ -4,11 +4,11 @@ use cretonne::ir::Function; use cretonne; +use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use filetest::subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{SubTest, Context, Result, run_filecheck}; use std::borrow::Cow; use std::fmt::Write; -use utils::pretty_error; struct TestPreopt; diff --git a/cranelift/src/filetest/test_print_cfg.rs b/lib/filetests/src/test_print_cfg.rs similarity index 93% rename from cranelift/src/filetest/test_print_cfg.rs rename to lib/filetests/src/test_print_cfg.rs index 1be9987087..470d293185 100644 --- a/cranelift/src/filetest/test_print_cfg.rs +++ b/lib/filetests/src/test_print_cfg.rs @@ -8,7 +8,7 @@ use std::borrow::Cow; use cretonne::ir::Function; use cretonne::cfg_printer::CFGPrinter; use cton_reader::TestCommand; -use filetest::subtest::{self, SubTest, Context, Result as STResult}; +use subtest::{self, SubTest, Context, Result as STResult}; /// Object implementing the `test print-cfg` sub-test. struct TestPrintCfg; diff --git a/cranelift/src/filetest/test_regalloc.rs b/lib/filetests/src/test_regalloc.rs similarity index 94% rename from cranelift/src/filetest/test_regalloc.rs rename to lib/filetests/src/test_regalloc.rs index 48160796a8..7cd0788928 100644 --- a/cranelift/src/filetest/test_regalloc.rs +++ b/lib/filetests/src/test_regalloc.rs @@ -7,11 +7,11 @@ use cretonne::ir::Function; use cretonne; +use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use filetest::subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{SubTest, Context, Result, run_filecheck}; use std::borrow::Cow; use std::fmt::Write; -use utils::pretty_error; struct TestRegalloc; diff --git a/cranelift/src/filetest/test_simple_gvn.rs b/lib/filetests/src/test_simple_gvn.rs similarity index 93% rename from cranelift/src/filetest/test_simple_gvn.rs rename to lib/filetests/src/test_simple_gvn.rs index a1da0e42a4..f967c3fa91 100644 --- a/cranelift/src/filetest/test_simple_gvn.rs +++ b/lib/filetests/src/test_simple_gvn.rs @@ -7,11 +7,11 @@ use cretonne::ir::Function; use cretonne; +use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use filetest::subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{SubTest, Context, Result, run_filecheck}; use std::borrow::Cow; use std::fmt::Write; -use utils::pretty_error; struct TestSimpleGVN; diff --git a/cranelift/src/filetest/test_verifier.rs b/lib/filetests/src/test_verifier.rs similarity index 96% rename from cranelift/src/filetest/test_verifier.rs rename to lib/filetests/src/test_verifier.rs index 7834fa778d..bb4375b59e 100644 --- a/cranelift/src/filetest/test_verifier.rs +++ b/lib/filetests/src/test_verifier.rs @@ -2,7 +2,9 @@ //! //! The `test verifier` test command looks for annotations on instructions like this: //! +//! ```cton //! jump ebb3 ; error: jump to non-existent EBB +//! ``` //! //! This annotation means that the verifier is expected to given an error for the jump instruction //! containing the substring "jump to non-existent EBB". @@ -11,8 +13,8 @@ use std::borrow::{Borrow, Cow}; use cretonne::verify_function; use cretonne::ir::Function; use cton_reader::TestCommand; -use filetest::subtest::{SubTest, Context, Result}; -use utils::match_directive; +use subtest::{SubTest, Context, Result}; +use match_directive::match_directive; struct TestVerifier; diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs index c551dbf783..824a7606da 100644 --- a/lib/wasm/tests/wasm_testsuite.rs +++ b/lib/wasm/tests/wasm_testsuite.rs @@ -11,11 +11,9 @@ use std::str; use std::io::prelude::*; use std::process::Command; use std::fs; -use cretonne::ir; -use cretonne::ir::entities::AnyEntity; -use cretonne::isa::TargetIsa; use cretonne::settings::{self, Configurable, Flags}; use cretonne::verifier; +use cretonne::print_errors::pretty_verifier_error; use tempdir::TempDir; #[test] @@ -109,25 +107,3 @@ fn handle_module(path: PathBuf, flags: &Flags) { .unwrap(); } } - - -/// Pretty-print a verifier error. -pub fn pretty_verifier_error( - func: &ir::Function, - isa: Option<&TargetIsa>, - err: verifier::Error, -) -> String { - let msg = err.to_string(); - let str1 = match err.location { - AnyEntity::Inst(inst) => { - format!( - "{}\n{}: {}\n\n", - msg, - inst, - func.dfg.display_inst(inst, isa) - ) - } - _ => String::from(format!("{}\n", msg)), - }; - format!("{}{}", str1, func.display(isa)) -} From e889b88d04a64bc12f89d2b4f9ecbac7783a47fb Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 15 Mar 2018 14:56:54 -0700 Subject: [PATCH 1616/3084] Split filecheck out into its own repo. --- cranelift/Cargo.toml | 2 +- cranelift/docs/testing.rst | 3 +- cranelift/publish-all.sh | 6 +- lib/filecheck/Cargo.toml | 14 - lib/filecheck/src/checker.rs | 456 -------------------------- lib/filecheck/src/error.rs | 69 ---- lib/filecheck/src/explain.rs | 202 ------------ lib/filecheck/src/lib.rs | 256 --------------- lib/filecheck/src/pattern.rs | 593 ---------------------------------- lib/filecheck/src/variable.rs | 62 ---- lib/filecheck/tests/basic.rs | 402 ----------------------- lib/filetests/Cargo.toml | 2 +- 12 files changed, 4 insertions(+), 2063 deletions(-) delete mode 100644 lib/filecheck/Cargo.toml delete mode 100644 lib/filecheck/src/checker.rs delete mode 100644 lib/filecheck/src/error.rs delete mode 100644 lib/filecheck/src/explain.rs delete mode 100644 lib/filecheck/src/lib.rs delete mode 100644 lib/filecheck/src/pattern.rs delete mode 100644 lib/filecheck/src/variable.rs delete mode 100644 lib/filecheck/tests/basic.rs diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 064a2ee5ab..b9acc0606c 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -19,7 +19,7 @@ cretonne-frontend = { path = "lib/frontend", version = "0.3.4" } cretonne-wasm = { path = "lib/wasm", version = "0.3.4" } cretonne-native = { path = "lib/native", version = "0.3.4" } cretonne-filetests = { path = "lib/filetests", version = "0.3.4" } -filecheck = { path = "lib/filecheck" } +filecheck = "0.2.1" docopt = "0.8.0" serde = "1.0.8" serde_derive = "1.0.8" diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index 6ddc53ed84..320cc37389 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -144,8 +144,7 @@ Filecheck Many of the test commands described below use *filecheck* to verify their output. Filecheck is a Rust implementation of the LLVM tool of the same name. -See the :file:`lib/filecheck` `documentation `_ for -details of its syntax. +See the `documentation `_ for details of its syntax. Comments in :file:`.cton` files are associated with the entity they follow. This typically means an instruction or the whole function. Those tests that diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 7d8b7f7f3c..c5cd6bfca3 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,6 @@ cd $(dirname "$0") topdir="$(pwd)" # All the cretonne-* crates have the same version number -# The filecheck crate version is managed independently. version="0.3.4" # Update all of the Cargo.toml files. @@ -12,9 +11,6 @@ version="0.3.4" # The main Cargo.toml in the top-level directory is the cretonne-tools crate which we don't publish. echo "Updating crate versions to $version" for crate in . lib/*; do - if [ "$crate" = "lib/filecheck" ]; then - continue - fi # Update the version number of this crate to $version. sed -i.bk -e "s/^version = .*/version = \"$version\"/" "$crate/Cargo.toml" # Update the required version number of any cretonne* dependencies. @@ -31,7 +27,7 @@ cargo update echo git commit -a -m "\"Bump version to $version"\" echo git push -for crate in filecheck cretonne frontend native reader wasm; do +for crate in cretonne frontend native reader wasm; do echo cargo publish --manifest-path "lib/$crate/Cargo.toml" done echo diff --git a/lib/filecheck/Cargo.toml b/lib/filecheck/Cargo.toml deleted file mode 100644 index aec22b0f51..0000000000 --- a/lib/filecheck/Cargo.toml +++ /dev/null @@ -1,14 +0,0 @@ -[package] -authors = ["The Cretonne Project Developers"] -name = "filecheck" -version = "0.1.0" -description = "Library for matching test outputs against filecheck directives" -license = "Apache-2.0" -repository = "https://github.com/Cretonne/cretonne" -documentation = "https://docs.rs/filecheck" - -[lib] -name = "filecheck" - -[dependencies] -regex = "0.2.6" diff --git a/lib/filecheck/src/checker.rs b/lib/filecheck/src/checker.rs deleted file mode 100644 index b3a68fba4c..0000000000 --- a/lib/filecheck/src/checker.rs +++ /dev/null @@ -1,456 +0,0 @@ -use error::{Error, Result}; -use variable::{VariableMap, Value, varname_prefix}; -use pattern::Pattern; -use regex::{Regex, Captures}; -use std::borrow::Cow; -use std::collections::HashMap; -use std::cmp::max; -use std::fmt::{self, Display, Formatter}; -use std::mem; -use MatchRange; -use explain::{Recorder, Explainer}; - -// The different kinds of directives we support. -enum Directive { - Check(Pattern), - SameLn(Pattern), - NextLn(Pattern), - Unordered(Pattern), - Not(Pattern), - Regex(String, String), -} - -// Regular expression matching a directive. -// The match groups are: -// -// 1. Keyword. -// 2. Rest of line / pattern. -// -const DIRECTIVE_RX: &str = r"\b(check|sameln|nextln|unordered|not|regex):\s+(.*)"; - -impl Directive { - /// Create a new directive from a `DIRECTIVE_RX` match. - fn new(caps: Captures) -> Result { - let cmd = caps.get(1).map(|m| m.as_str()).expect("group 1 must match"); - let rest = caps.get(2).map(|m| m.as_str()).expect("group 2 must match"); - - if cmd == "regex" { - return Directive::regex(rest); - } - - // All other commands are followed by a pattern. - let pat = rest.parse()?; - - match cmd { - "check" => Ok(Directive::Check(pat)), - "sameln" => Ok(Directive::SameLn(pat)), - "nextln" => Ok(Directive::NextLn(pat)), - "unordered" => Ok(Directive::Unordered(pat)), - "not" => { - if !pat.defs().is_empty() { - let msg = format!( - "can't define variables '$({}=...' in not: {}", - pat.defs()[0], - rest - ); - Err(Error::DuplicateDef(msg)) - } else { - Ok(Directive::Not(pat)) - } - } - _ => panic!("unexpected command {} in regex match", cmd), - } - } - - /// Create a `regex:` directive from a `VAR=...` string. - fn regex(rest: &str) -> Result { - let varlen = varname_prefix(rest); - if varlen == 0 { - return Err(Error::Syntax( - format!("invalid variable name in regex: {}", rest), - )); - } - let var = rest[0..varlen].to_string(); - if !rest[varlen..].starts_with('=') { - return Err(Error::Syntax(format!( - "expected '=' after variable '{}' in regex: {}", - var, - rest - ))); - } - // Ignore trailing white space in the regex, including CR. - Ok(Directive::Regex( - var, - rest[varlen + 1..].trim_right().to_string(), - )) - } -} - - -/// Builder for constructing a `Checker` instance. -pub struct CheckerBuilder { - directives: Vec, - linerx: Regex, -} - -impl CheckerBuilder { - /// Create a new, blank `CheckerBuilder`. - pub fn new() -> Self { - Self { - directives: Vec::new(), - linerx: Regex::new(DIRECTIVE_RX).unwrap(), - } - } - - /// Add a potential directive line. - /// - /// Returns true if this is a directive with one of the known prefixes. - /// Returns false if no known directive was found. - /// Returns an error if there is a problem with the directive. - pub fn directive(&mut self, l: &str) -> Result { - match self.linerx.captures(l) { - Some(caps) => { - self.directives.push(Directive::new(caps)?); - Ok(true) - } - None => Ok(false), - } - } - - /// Add multiple directives. - /// - /// The text is split into lines that are added individually as potential directives. - /// This method can be used to parse a whole test file containing multiple directives. - pub fn text(&mut self, t: &str) -> Result<&mut Self> { - for caps in self.linerx.captures_iter(t) { - self.directives.push(Directive::new(caps)?); - } - Ok(self) - } - - /// Get the finished `Checker`. - pub fn finish(&mut self) -> Checker { - // Move directives into the new checker, leaving `self.directives` empty and ready for - // building a new checker. - let new_directives = mem::replace(&mut self.directives, Vec::new()); - Checker::new(new_directives) - } -} - -/// Verify a list of directives against a test input. -/// -/// Use a `CheckerBuilder` to construct a `Checker`. Then use the `test` method to verify the list -/// of directives against a test input. -pub struct Checker { - directives: Vec, -} - -impl Checker { - fn new(directives: Vec) -> Self { - Self { directives: directives } - } - - /// An empty checker contains no directives, and will match any input string. - pub fn is_empty(&self) -> bool { - self.directives.is_empty() - } - - /// Verify directives against the input text. - /// - /// This returns `true` if the text matches all the directives, `false` if it doesn't. - /// An error is only returned if there is a problem with the directives. - pub fn check(&self, text: &str, vars: &VariableMap) -> Result { - self.run(text, vars, &mut ()) - } - - /// Explain how directives are matched against the input text. - pub fn explain(&self, text: &str, vars: &VariableMap) -> Result<(bool, String)> { - let mut expl = Explainer::new(text); - let success = self.run(text, vars, &mut expl)?; - expl.finish(); - Ok((success, expl.to_string())) - } - - fn run(&self, text: &str, vars: &VariableMap, recorder: &mut Recorder) -> Result { - let mut state = State::new(text, vars, recorder); - - // For each pending `not:` check, store (begin-offset, regex). - let mut nots = Vec::new(); - - for (dct_idx, dct) in self.directives.iter().enumerate() { - let (pat, range) = match *dct { - Directive::Check(ref pat) => (pat, state.check()), - Directive::SameLn(ref pat) => (pat, state.sameln()), - Directive::NextLn(ref pat) => (pat, state.nextln()), - Directive::Unordered(ref pat) => (pat, state.unordered(pat)), - Directive::Not(ref pat) => { - // Resolve `not:` directives immediately to get the right variable values, but - // don't match it until we know the end of the range. - // - // The `not:` directives test the same range as `unordered:` directives. In - // particular, if they refer to defined variables, their range is restricted to - // the text following the match that defined the variable. - nots.push((dct_idx, state.unordered_begin(pat), pat.resolve(&state)?)); - continue; - } - Directive::Regex(ref var, ref rx) => { - state.vars.insert( - var.clone(), - VarDef { - value: Value::Regex(Cow::Borrowed(rx)), - offset: 0, - }, - ); - continue; - } - }; - // Check if `pat` matches in `range`. - state.recorder.directive(dct_idx); - if let Some((match_begin, match_end)) = state.match_positive(pat, range)? { - if let Directive::Unordered(_) = *dct { - // This was an unordered match. - // Keep track of the largest matched position, but leave `last_ordered` alone. - state.max_match = max(state.max_match, match_end); - } else { - // Ordered match. - state.last_ordered = match_end; - state.max_match = match_end; - - // Verify any pending `not:` directives now that we know their range. - for (not_idx, not_begin, rx) in nots.drain(..) { - state.recorder.directive(not_idx); - if let Some(mat) = rx.find(&text[not_begin..match_begin]) { - // Matched `not:` pattern. - state.recorder.matched_not(rx.as_str(), ( - not_begin + mat.start(), - not_begin + mat.end(), - )); - return Ok(false); - } else { - state.recorder.missed_not( - rx.as_str(), - (not_begin, match_begin), - ); - } - } - } - } else { - // No match! - return Ok(false); - } - } - - // Verify any pending `not:` directives after the last ordered directive. - for (not_idx, not_begin, rx) in nots.drain(..) { - state.recorder.directive(not_idx); - if rx.find(&text[not_begin..]).is_some() { - // Matched `not:` pattern. - // TODO: Use matched range for an error message. - return Ok(false); - } - } - - Ok(true) - } -} - -/// A local definition of a variable. -pub struct VarDef<'a> { - /// The value given to the variable. - value: Value<'a>, - /// Offset in input text from where the variable is available. - offset: usize, -} - -struct State<'a> { - text: &'a str, - env_vars: &'a VariableMap, - recorder: &'a mut Recorder, - - vars: HashMap>, - // Offset after the last ordered match. This does not include recent unordered matches. - last_ordered: usize, - // Largest offset following a positive match, including unordered matches. - max_match: usize, -} - -impl<'a> State<'a> { - fn new(text: &'a str, env_vars: &'a VariableMap, recorder: &'a mut Recorder) -> State<'a> { - State { - text, - env_vars, - recorder, - vars: HashMap::new(), - last_ordered: 0, - max_match: 0, - } - } - - // Get the offset following the match that defined `var`, or 0 if var is an environment - // variable or unknown. - fn def_offset(&self, var: &str) -> usize { - self.vars - .get(var) - .map(|&VarDef { offset, .. }| offset) - .unwrap_or(0) - } - - // Get the offset of the beginning of the next line after `pos`. - fn bol(&self, pos: usize) -> usize { - if let Some(offset) = self.text[pos..].find('\n') { - pos + offset + 1 - } else { - self.text.len() - } - } - - // Get the range in text to be matched by a `check:`. - fn check(&self) -> MatchRange { - (self.max_match, self.text.len()) - } - - // Get the range in text to be matched by a `sameln:`. - fn sameln(&self) -> MatchRange { - let b = self.max_match; - let e = self.bol(b); - (b, e) - } - - // Get the range in text to be matched by a `nextln:`. - fn nextln(&self) -> MatchRange { - let b = self.bol(self.max_match); - let e = self.bol(b); - (b, e) - } - - // Get the beginning of the range in text to be matched by a `unordered:` or `not:` directive. - // The unordered directive must match after the directives that define the variables used. - fn unordered_begin(&self, pat: &Pattern) -> usize { - pat.parts() - .iter() - .filter_map(|part| part.ref_var()) - .map(|var| self.def_offset(var)) - .fold(self.last_ordered, max) - } - - // Get the range in text to be matched by a `unordered:` directive. - fn unordered(&self, pat: &Pattern) -> MatchRange { - (self.unordered_begin(pat), self.text.len()) - } - - // Search for `pat` in `range`, return the range matched. - // After a positive match, update variable definitions, if any. - fn match_positive(&mut self, pat: &Pattern, range: MatchRange) -> Result> { - let rx = pat.resolve(self)?; - let txt = &self.text[range.0..range.1]; - let defs = pat.defs(); - let matched_range = if defs.is_empty() { - // Pattern defines no variables. Fastest search is `find`. - rx.find(txt) - } else { - // We need the captures to define variables. - rx.captures(txt).map(|caps| { - let matched_range = caps.get(0).expect("whole expression must match"); - for var in defs { - let txtval = caps.name(var).map(|mat| mat.as_str()).unwrap_or(""); - self.recorder.defined_var(var, txtval); - let vardef = VarDef { - value: Value::Text(Cow::Borrowed(txtval)), - // This offset is the end of the whole matched pattern, not just the text - // defining the variable. - offset: range.0 + matched_range.end(), - }; - self.vars.insert(var.clone(), vardef); - } - matched_range - }) - }; - Ok(if let Some(mat) = matched_range { - let r = (range.0 + mat.start(), range.0 + mat.end()); - self.recorder.matched_check(rx.as_str(), r); - Some(r) - } else { - self.recorder.missed_check(rx.as_str(), range); - None - }) - } -} - -impl<'a> VariableMap for State<'a> { - fn lookup(&self, varname: &str) -> Option { - // First look for a local define. - if let Some(&VarDef { ref value, .. }) = self.vars.get(varname) { - Some(value.clone()) - } else { - // No local, maybe an environment variable? - self.env_vars.lookup(varname) - } - } -} - -impl Display for Directive { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - use self::Directive::*; - match *self { - Check(ref pat) => writeln!(f, "check: {}", pat), - SameLn(ref pat) => writeln!(f, "sameln: {}", pat), - NextLn(ref pat) => writeln!(f, "nextln: {}", pat), - Unordered(ref pat) => writeln!(f, "unordered: {}", pat), - Not(ref pat) => writeln!(f, "not: {}", pat), - Regex(ref var, ref rx) => writeln!(f, "regex: {}={}", var, rx), - } - } -} - -impl Display for Checker { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - for (idx, dir) in self.directives.iter().enumerate() { - write!(f, "#{} {}", idx, dir)?; - } - Ok(()) - } -} - -#[cfg(test)] -mod tests { - use super::CheckerBuilder; - use error::Error; - - fn e2s(e: Error) -> String { - e.to_string() - } - - #[test] - fn directive() { - let mut b = CheckerBuilder::new(); - - assert_eq!(b.directive("not here: more text").map_err(e2s), Ok(false)); - assert_eq!( - b.directive("not here: regex: X=more text").map_err(e2s), - Ok(true) - ); - assert_eq!( - b.directive("regex: X = tommy").map_err(e2s), - Err( - "expected '=' after variable 'X' in regex: X = tommy".to_string(), - ) - ); - assert_eq!( - b.directive("[arm]not: patt $x $(y) here").map_err(e2s), - Ok(true) - ); - assert_eq!( - b.directive("[x86]sameln: $x $(y=[^]]*) there").map_err(e2s), - Ok(true) - ); - // Windows line ending sneaking in. - assert_eq!(b.directive("regex: Y=foo\r").map_err(e2s), Ok(true)); - - let c = b.finish(); - assert_eq!( - c.to_string(), - "#0 regex: X=more text\n#1 not: patt $(x) $(y) here\n#2 sameln: $(x) \ - $(y=[^]]*) there\n#3 regex: Y=foo\n" - ); - } -} diff --git a/lib/filecheck/src/error.rs b/lib/filecheck/src/error.rs deleted file mode 100644 index f15cb17124..0000000000 --- a/lib/filecheck/src/error.rs +++ /dev/null @@ -1,69 +0,0 @@ -use std::result; -use std::convert::From; -use std::error::Error as StdError; -use std::fmt; -use regex; - -/// A result from the filecheck library. -pub type Result = result::Result; - -/// A filecheck error. -#[derive(Debug)] -pub enum Error { - /// A syntax error in a check line. - Syntax(String), - /// A check refers to an undefined variable. - /// - /// The pattern contains `$foo` where the `foo` variable has not yet been defined. - /// Use `$$` to match a literal dollar sign. - UndefVariable(String), - /// A pattern contains a back-reference to a variable that was defined in the same pattern. - /// - /// For example, `check: Hello $(world=.*) $world`. Backreferences are not supported. Often the - /// desired effect can be achieved with the `sameln` check: - /// - /// ```text - /// check: Hello $(world=[^ ]*) - /// sameln: $world - /// ``` - Backref(String), - /// A pattern contains multiple definitions of the same variable. - DuplicateDef(String), - /// An error in a regular expression. - /// - /// Use `cause()` to get the underlying `Regex` library error. - Regex(regex::Error), -} - -impl StdError for Error { - fn description(&self) -> &str { - use Error::*; - match *self { - Syntax(ref s) | - UndefVariable(ref s) | - Backref(ref s) | - DuplicateDef(ref s) => s, - Regex(ref err) => err.description(), - } - } - - fn cause(&self) -> Option<&StdError> { - use Error::*; - match *self { - Regex(ref err) => Some(err), - _ => None, - } - } -} - -impl fmt::Display for Error { - fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result { - write!(fmt, "{}", self.description()) - } -} - -impl From for Error { - fn from(e: regex::Error) -> Error { - Error::Regex(e) - } -} diff --git a/lib/filecheck/src/explain.rs b/lib/filecheck/src/explain.rs deleted file mode 100644 index 137d360f89..0000000000 --- a/lib/filecheck/src/explain.rs +++ /dev/null @@ -1,202 +0,0 @@ -//! Explaining how *filecheck* matched or failed to match a file. - -use MatchRange; -use std::fmt::{self, Display, Formatter}; -use std::cmp::min; - -/// Record events during matching. -pub trait Recorder { - /// Set the directive we're talking about now. - fn directive(&mut self, dct: usize); - - /// Matched a positive check directive (check/sameln/nextln/unordered). - fn matched_check(&mut self, regex: &str, matched: MatchRange); - - /// Matched a `not:` directive. This means the match will fail. - fn matched_not(&mut self, regex: &str, matched: MatchRange); - - /// Missed a positive check directive. The range given is the range searched for a match. - fn missed_check(&mut self, regex: &str, searched: MatchRange); - - /// Missed `not:` directive (as intended). - fn missed_not(&mut self, regex: &str, searched: MatchRange); - - /// The directive defined a variable. - fn defined_var(&mut self, varname: &str, value: &str); -} - -/// The null recorder just doesn't listen to anything you say. -impl Recorder for () { - fn directive(&mut self, _: usize) {} - fn matched_check(&mut self, _: &str, _: MatchRange) {} - fn matched_not(&mut self, _: &str, _: MatchRange) {} - fn defined_var(&mut self, _: &str, _: &str) {} - fn missed_check(&mut self, _: &str, _: MatchRange) {} - fn missed_not(&mut self, _: &str, _: MatchRange) {} -} - -struct Match { - directive: usize, - is_match: bool, - is_not: bool, - regex: String, - range: MatchRange, -} - -struct VarDef { - directive: usize, - varname: String, - value: String, -} - -/// Record an explanation for the matching process, success or failure. -pub struct Explainer<'a> { - text: &'a str, - directive: usize, - matches: Vec, - vardefs: Vec, -} - -impl<'a> Explainer<'a> { - pub fn new(text: &'a str) -> Explainer { - Explainer { - text, - directive: 0, - matches: Vec::new(), - vardefs: Vec::new(), - } - } - - /// Finish up after recording all events in a match. - pub fn finish(&mut self) { - self.matches.sort_by_key(|m| (m.range, m.directive)); - self.vardefs.sort_by_key(|v| v.directive); - } -} - -impl<'a> Display for Explainer<'a> { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - // Offset of beginning of the last line printed. - let mut curln = 0; - // Offset of beginning of the next line to be printed. - let mut nextln = 0; - - for m in &self.matches { - // Emit lines until m.range.0 is visible. - while nextln <= m.range.0 && nextln < self.text.len() { - let newln = self.text[nextln..] - .find('\n') - .map(|d| nextln + d + 1) - .unwrap_or(self.text.len()); - assert!(newln > nextln); - writeln!(f, "> {}", &self.text[nextln..newln - 1])?; - curln = nextln; - nextln = newln; - } - - // Emit ~~~ under the part of the match in curln. - if m.is_match { - write!(f, " ")?; - let mend = min(m.range.1, nextln - 1); - for pos in curln..mend { - if pos < m.range.0 { - write!(f, " ") - } else if pos == m.range.0 { - write!(f, "^") - } else { - write!(f, "~") - }?; - } - writeln!(f, "")?; - } - - // Emit the match message itself. - writeln!( - f, - "{} #{}{}: {}", - if m.is_match { "Matched" } else { "Missed" }, - m.directive, - if m.is_not { " not" } else { "" }, - m.regex - )?; - - // Emit any variable definitions. - if let Ok(found) = self.vardefs.binary_search_by_key( - &m.directive, - |v| v.directive, - ) - { - let mut first = found; - while first > 0 && self.vardefs[first - 1].directive == m.directive { - first -= 1; - } - for d in &self.vardefs[first..] { - if d.directive != m.directive { - break; - } - writeln!(f, "Define {}={}", d.varname, d.value)?; - } - } - } - - // Emit trailing lines. - for line in self.text[nextln..].lines() { - writeln!(f, "> {}", line)?; - } - Ok(()) - } -} - -impl<'a> Recorder for Explainer<'a> { - fn directive(&mut self, dct: usize) { - self.directive = dct; - } - - fn matched_check(&mut self, regex: &str, matched: MatchRange) { - self.matches.push(Match { - directive: self.directive, - is_match: true, - is_not: false, - regex: regex.to_owned(), - range: matched, - }); - } - - fn matched_not(&mut self, regex: &str, matched: MatchRange) { - self.matches.push(Match { - directive: self.directive, - is_match: true, - is_not: true, - regex: regex.to_owned(), - range: matched, - }); - } - - fn missed_check(&mut self, regex: &str, searched: MatchRange) { - self.matches.push(Match { - directive: self.directive, - is_match: false, - is_not: false, - regex: regex.to_owned(), - range: searched, - }); - } - - fn missed_not(&mut self, regex: &str, searched: MatchRange) { - self.matches.push(Match { - directive: self.directive, - is_match: false, - is_not: true, - regex: regex.to_owned(), - range: searched, - }); - } - - fn defined_var(&mut self, varname: &str, value: &str) { - self.vardefs.push(VarDef { - directive: self.directive, - varname: varname.to_owned(), - value: value.to_owned(), - }); - } -} diff --git a/lib/filecheck/src/lib.rs b/lib/filecheck/src/lib.rs deleted file mode 100644 index 4dec7a9b65..0000000000 --- a/lib/filecheck/src/lib.rs +++ /dev/null @@ -1,256 +0,0 @@ -//! This crate provides a text pattern matching library with functionality similar to the LLVM -//! project's [FileCheck command](https://llvm.org/docs/CommandGuide/FileCheck.html). -//! -//! A list of directives is typically extracted from a file containing a test case. The test case -//! is then run through the program under test, and its output matched against the directives. -//! -//! See the [`CheckerBuilder`](struct.CheckerBuilder.html) and [`Checker`](struct.Checker.html) -//! types for the main library API. -//! -//! # Directives -//! -//! These are the directives recognized by *filecheck*: -//! -//! 

-//! check: <pattern>
-//! sameln: <pattern>
-//! nextln: <pattern>
-//! unordered: <pattern>
-//! not: <pattern>
-//! regex: <variable>=<regex>
-//!
-//! -//! Each directive is described in more detail below. -//! -//! ## Example -//! -//! The Rust program below prints the primes less than 100. It has *filecheck* directives embedded -//! in comments: -//! -//! ```rust -//! fn is_prime(x: u32) -> bool { -//! (2..x).all(|d| x % d != 0) -//! } -//! -//! // Check that we get the primes and nothing else: -//! // regex: NUM=\d+ -//! // not: $NUM -//! // check: 2 -//! // nextln: 3 -//! // check: 89 -//! // nextln: 97 -//! // not: $NUM -//! fn main() { -//! for p in (2..10).filter(|&x| is_prime(x)) { -//! println!("{}", p); -//! } -//! } -//! ``` -//! -//! A test driver compiles and runs the program, then pipes the output through *filecheck*: -//! -//! ```sh -//! $ rustc primes.rs -//! $ ./primes | cton-util filecheck -v -//! #0 regex: NUM=\d+ -//! #1 not: $NUM -//! #2 check: 2 -//! #3 nextln: 3 -//! #4 check: 89 -//! #5 nextln: 97 -//! #6 not: $NUM -//! no match #1: \d+ -//! > 2 -//! ~ -//! match #2: \b2\b -//! > 3 -//! ~ -//! match #3: \b3\b -//! > 5 -//! > 7 -//! ... -//! > 79 -//! > 83 -//! > 89 -//! ~~ -//! match #4: \b89\b -//! > 97 -//! ~~ -//! match #5: \b97\b -//! no match #6: \d+ -//! OK -//! ``` -//! -//! ## The `check:` directive -//! -//! Match patterns non-overlapping and in order: -//! -//! ```sh -//! #0 check: one -//! #1 check: two -//! ``` -//! -//! These directives will match the string `"one two"`, but not `"two one"`. The second directive -//! must match after the first one, and it can't overlap. -//! -//! ## The `sameln:` directive -//! -//! Match a pattern in the same line as the previous match. -//! -//! ```sh -//! #0 check: one -//! #1 sameln: two -//! ``` -//! -//! These directives will match the string `"one two"`, but not `"one\ntwo"`. The second match must -//! be in the same line as the first. Like the `check:` directive, the match must also follow the -//! first match, so `"two one" would not be matched. -//! -//! If there is no previous match, `sameln:` matches on the first line of the input. -//! -//! ## The `nextln:` directive -//! -//! Match a pattern in the next line after the previous match. -//! -//! ```sh -//! #0 check: one -//! #1 nextln: two -//! ``` -//! -//! These directives will match the string `"one\ntwo"`, but not `"one two"` or `"one\n\ntwo"`. -//! -//! If there is no previous match, `nextln:` matches on the second line of the input as if there -//! were a previous match on the first line. -//! -//! ## The `unordered:` directive -//! -//! Match patterns in any order, and possibly overlapping each other. -//! -//! ```sh -//! #0 unordered: one -//! #1 unordered: two -//! ``` -//! -//! These directives will match the string `"one two"` *and* the string `"two one"`. -//! -//! When a normal ordered match is inserted into a sequence of `unordered:` directives, it acts as -//! a barrier: -//! -//! ```sh -//! #0 unordered: one -//! #1 unordered: two -//! #2 check: three -//! #3 unordered: four -//! #4 unordered: five -//! ``` -//! -//! These directives will match `"two one three four five"`, but not `"two three one four five"`. -//! The `unordered:` matches are not allowed to cross the ordered `check:` directive. -//! -//! When `unordered:` matches define and use variables, a topological order is enforced. This means -//! that a match referencing a variable must follow the match where the variable was defined: -//! -//! ```sh -//! #0 regex: V=\bv\d+\b -//! #1 unordered: $(va=$V) = load -//! #2 unordered: $(vb=$V) = iadd $va -//! #3 unordered: $(vc=$V) = load -//! #4 unordered: iadd $va, $vc -//! ``` -//! -//! In the above directives, #2 must match after #1, and #4 must match after both #1 and #3, but -//! otherwise they can match in any order. -//! -//! ## The `not:` directive -//! -//! Check that a pattern *does not* appear between matches. -//! -//! ```sh -//! #0 check: one -//! #1 not: two -//! #2 check: three -//! ``` -//! -//! The directives above will match `"one five three"`, but not `"one two three"`. -//! -//! The pattern in a `not:` directive can't define any variables. Since it never matches anything, -//! the variables would not get a value. -//! -//! ## The `regex:` directive -//! -//! Define a shorthand name for a regular expression. -//! -//! ```sh -//! #0 regex: ID=\b[_a-zA-Z][_0-9a-zA-Z]*\b -//! #1 check: $ID + $ID -//! ``` -//! -//! The `regex:` directive gives a name to a regular expression which can then be used as part of a -//! pattern to match. Patterns are otherwise just plain text strings to match, so this is not -//! simple macro expansion. -//! -//! See [the Rust regex crate](../regex/index.html#syntax) for the regular expression syntax. -//! -//! # Patterns and variables -//! -//! Patterns are plain text strings to be matched in the input file. The dollar sign is used as an -//! escape character to expand variables. The following escape sequences are recognized: -//! -//! 
-//! $$                Match single dollar sign.
-//! $()               Match the empty string.
-//! $(=<regex>)       Match regular expression <regex>.
-//! $<var>            Match contents of variable <var>.
-//! $(<var>)          Match contents of variable <var>.
-//! $(<var>=<regex>)  Match <regex>, then
-//!                   define <var> as the matched text.
-//! $(<var>=$<rxvar>) Match regex in <rxvar>, then
-//!                   define <var> as the matched text.
-//!
-//! -//! Variables can contain either plain text or regular expressions. Plain text variables are -//! defined with the `$(var=...)` syntax in a previous directive. They match the same text again. -//! Backreferences within the same pattern are not allowed. When a variable is defined in a -//! pattern, it can't be referenced again in the same pattern. -//! -//! Regular expression variables are defined with the `regex:` directive. They match the regular -//! expression each time they are used, so the matches don't need to be identical. -//! -//! ## Word boundaries -//! -//! If a pattern begins or ends with a (plain text) letter or number, it will only match on a word -//! boundary. Use the `$()` empty string match to prevent this: -//! -//! ```sh -//! check: one$() -//! ``` -//! -//! This will match `"one"` and `"onetwo"`, but not `"zeroone"`. -//! -//! The empty match syntax can also be used to require leading or trailing whitespace: -//! -//! ```sh -//! check: one, $() -//! ``` -//! -//! This will match `"one, two"` , but not `"one,two"`. Without the `$()`, trailing whitespace -//! would be trimmed from the pattern. - -#![deny(missing_docs, - trivial_numeric_casts, - unused_extern_crates)] - -pub use error::{Error, Result}; -pub use variable::{VariableMap, Value, NO_VARIABLES}; -pub use checker::{Checker, CheckerBuilder}; - -extern crate regex; - -mod error; -mod variable; -mod pattern; -mod checker; -mod explain; - -/// The range of a match in the input text. -pub type MatchRange = (usize, usize); diff --git a/lib/filecheck/src/pattern.rs b/lib/filecheck/src/pattern.rs deleted file mode 100644 index 97977b191c..0000000000 --- a/lib/filecheck/src/pattern.rs +++ /dev/null @@ -1,593 +0,0 @@ -//! Pattern matching for a single directive. - -use error::{Error, Result}; -use variable::{varname_prefix, VariableMap, Value}; -use std::str::FromStr; -use std::fmt::{self, Display, Formatter, Write}; -use regex::{Regex, RegexBuilder, escape}; - -/// A pattern to match as specified in a directive. -/// -/// Each pattern is broken into a sequence of parts that must match in order. The kinds of parts -/// are: -/// -/// 1. Plain text match. -/// 2. Variable match, `$FOO` or `$(FOO)`. The variable `FOO` may expand to plain text or a regex. -/// 3. Variable definition from literal regex, `$(foo=.*)`. Match the regex and assign matching text -/// to variable `foo`. -/// 4. Variable definition from regex variable, `$(foo=$RX)`. Lookup variable `RX` which should -/// expand to a regex, match the regex, and assign matching text to variable `foo`. -/// -pub struct Pattern { - parts: Vec, - // Variables defined by this pattern. - defs: Vec, -} - -/// One atomic part of a pattern. -#[derive(Debug, PartialEq, Eq)] -pub enum Part { - /// Match a plain string. - Text(String), - /// Match a regular expression. The regex has already been wrapped in a non-capturing group if - /// necessary, so it is safe to concatenate. - Regex(String), - /// Match the contents of a variable, which can be plain text or regex. - Var(String), - /// Match literal regex, then assign match to variable. - /// The regex has already been wrapped in a named capture group. - DefLit { def: usize, regex: String }, - /// Lookup variable `var`, match resulting regex, assign matching text to variable `defs[def]`. - DefVar { def: usize, var: String }, -} - -impl Part { - /// Get the variable referenced by this part, if any. - pub fn ref_var(&self) -> Option<&str> { - match *self { - Part::Var(ref var) | - Part::DefVar { ref var, .. } => Some(var), - _ => None, - } - } -} - -impl Pattern { - /// Create a new blank pattern. Use the `FromStr` trait to generate Patterns with content. - fn new() -> Self { - Self { - parts: Vec::new(), - defs: Vec::new(), - } - } - - /// Check if the variable `v` is defined by this pattern. - pub fn defines_var(&self, v: &str) -> bool { - self.defs.iter().any(|d| d == v) - } - - /// Add a definition of a new variable. - /// Return the allocated def number. - fn add_def(&mut self, v: &str) -> Result { - if self.defines_var(v) { - Err(Error::DuplicateDef( - format!("duplicate definition of ${} in same pattern", v), - )) - } else { - let idx = self.defs.len(); - self.defs.push(v.to_string()); - Ok(idx) - } - } - - /// Parse a `Part` from a prefix of `s`. - /// Return the part and the number of bytes consumed from `s`. - /// Adds defined variables to `self.defs`. - fn parse_part(&mut self, s: &str) -> Result<(Part, usize)> { - let dollar = s.find('$'); - if dollar != Some(0) { - // String doesn't begin with a dollar sign, so match plain text up to the dollar sign. - let end = dollar.unwrap_or(s.len()); - return Ok((Part::Text(s[0..end].to_string()), end)); - } - - // String starts with a dollar sign. Look for these possibilities: - // - // 1. `$$`. - // 2. `$var`. - // 3. `$(var)`. - // 4. `$(var=regex)`. Where `regex` is a regular expression possibly containing matching - // braces. - // 5. `$(var=$VAR)`. - - // A doubled dollar sign matches a single dollar sign. - if s.starts_with("$$") { - return Ok((Part::Text("$".to_string()), 2)); - } - - // Look for `$var`. - let varname_end = 1 + varname_prefix(&s[1..]); - if varname_end != 1 { - return Ok((Part::Var(s[1..varname_end].to_string()), varname_end)); - } - - // All remaining possibilities start with `$(`. - if s.len() < 2 || !s.starts_with("$(") { - return Err(Error::Syntax( - "pattern syntax error, use $$ to match a single $" - .to_string(), - )); - } - - // Match the variable name, allowing for an empty varname in `$()`, or `$(=...)`. - let varname_end = 2 + varname_prefix(&s[2..]); - let varname = s[2..varname_end].to_string(); - - match s[varname_end..].chars().next() { - None => { - return Err(Error::Syntax(format!("unterminated $({}...", varname))); - } - Some(')') => { - let part = if varname.is_empty() { - // Match `$()`, turn it into an empty text match. - Part::Text(varname) - } else { - // Match `$(var)`. - Part::Var(varname) - }; - return Ok((part, varname_end + 1)); - } - Some('=') => { - // Variable definition. Fall through. - } - Some(ch) => { - return Err(Error::Syntax( - format!("syntax error in $({}... '{}'", varname, ch), - )); - } - } - - // This is a variable definition of the form `$(var=...`. - - // Allocate a definition index. - let def = if varname.is_empty() { - None - } else { - Some(self.add_def(&varname)?) - }; - - // Match `$(var=$PAT)`. - if s[varname_end + 1..].starts_with('$') { - let refname_begin = varname_end + 2; - let refname_end = refname_begin + varname_prefix(&s[refname_begin..]); - if refname_begin == refname_end { - return Err(Error::Syntax( - format!("expected variable name in $({}=$...", varname), - )); - } - if !s[refname_end..].starts_with(')') { - return Err(Error::Syntax(format!( - "expected ')' after $({}=${}...", - varname, - &s[refname_begin..refname_end] - ))); - } - let refname = s[refname_begin..refname_end].to_string(); - return if let Some(defidx) = def { - Ok(( - Part::DefVar { - def: defidx, - var: refname, - }, - refname_end + 1, - )) - } else { - Err(Error::Syntax( - format!("expected variable name in $(=${})", refname), - )) - }; - } - - // Last case: `$(var=...)` where `...` is a regular expression, possibly containing matched - // parentheses. - let rx_begin = varname_end + 1; - let rx_end = rx_begin + regex_prefix(&s[rx_begin..]); - if s[rx_end..].starts_with(')') { - let part = if let Some(defidx) = def { - // Wrap the regex in a named capture group. - Part::DefLit { - def: defidx, - regex: format!("(?P<{}>{})", varname, &s[rx_begin..rx_end]), - } - } else { - // When the varname is empty just match the regex, don't capture any variables. - // This is `$(=[a-z])`. - // Wrap the regex in a non-capturing group to make it concatenation-safe. - Part::Regex(format!("(?:{})", &s[rx_begin..rx_end])) - }; - Ok((part, rx_end + 1)) - } else { - Err(Error::Syntax(format!( - "missing ')' after regex in $({}={}", - varname, - &s[rx_begin..rx_end] - ))) - } - } -} - -/// Compute the length of a regular expression terminated by `)` or `}`. -/// Handle nested and escaped parentheses in the rx, but don't actually parse it. -/// Return the position of the terminating brace or the length of the string. -fn regex_prefix(s: &str) -> usize { - // The previous char was a backslash. - let mut escape = false; - // State around parsing charsets. - enum State { - Normal, // Outside any charset. - Curly, // Inside curly braces. - CSFirst, // Immediately after opening `[`. - CSNeg, // Immediately after `[^`. - CSBody, // Inside `[...`. - } - let mut state = State::Normal; - - // Current nesting level of parens. - let mut nest = 0usize; - - for (idx, ch) in s.char_indices() { - if escape { - escape = false; - continue; - } else if ch == '\\' { - escape = true; - continue; - } - match state { - State::Normal => { - match ch { - '[' => state = State::CSFirst, - '{' => state = State::Curly, - '(' => nest += 1, - ')' if nest > 0 => nest -= 1, - ')' | '}' => return idx, - _ => {} - } - } - State::Curly => { - if ch == '}' { - state = State::Normal; - } - } - State::CSFirst => { - state = match ch { - '^' => State::CSNeg, - _ => State::CSBody, - } - } - State::CSNeg => state = State::CSBody, - State::CSBody => { - if ch == ']' { - state = State::Normal; - } - } - } - } - s.len() -} - -impl FromStr for Pattern { - type Err = Error; - - fn from_str(s: &str) -> Result { - // Always remove leading and trailing whitespace. - // Use `$()` to actually include that in a match. - let s = s.trim(); - let mut pat = Pattern::new(); - let mut pos = 0; - while pos < s.len() { - let (part, len) = pat.parse_part(&s[pos..])?; - if let Some(v) = part.ref_var() { - if pat.defines_var(v) { - return Err(Error::Backref(format!( - "unsupported back-reference to '${}' \ - defined in same pattern", - v - ))); - } - } - pat.parts.push(part); - pos += len; - } - Ok(pat) - } -} - -impl Pattern { - /// Get a list of parts in this pattern. - pub fn parts(&self) -> &[Part] { - &self.parts - } - - /// Get a list of variable names defined when this pattern matches. - pub fn defs(&self) -> &[String] { - &self.defs - } - - /// Resolve all variable references in this pattern, turning it into a regular expression. - pub fn resolve(&self, vmap: &VariableMap) -> Result { - let mut out = String::new(); - - // Add a word boundary check `\b` to the beginning of the regex, but only if the first part - // is a plain text match that starts with a word character. - // - // This behavior can be disabled by starting the pattern with `$()`. - if let Some(&Part::Text(ref s)) = self.parts.first() { - if s.starts_with(char::is_alphanumeric) { - out.push_str(r"\b"); - } - } - - for part in &self.parts { - match *part { - Part::Text(ref s) => { - out.push_str(&escape(s)); - } - Part::Regex(ref rx) => out.push_str(rx), - Part::Var(ref var) => { - // Resolve the variable. We can handle a plain text expansion. - match vmap.lookup(var) { - None => { - return Err(Error::UndefVariable(format!("undefined variable ${}", var))) - } - Some(Value::Text(s)) => out.push_str(&escape(&s)), - // Wrap regex in non-capturing group for safe concatenation. - Some(Value::Regex(rx)) => write!(out, "(?:{})", rx).unwrap(), - } - } - Part::DefLit { ref regex, .. } => out.push_str(regex), - Part::DefVar { def, ref var } => { - // Wrap regex in a named capture group. - write!(out, "(?P<{}>", self.defs[def]).unwrap(); - match vmap.lookup(var) { - None => { - return Err(Error::UndefVariable(format!("undefined variable ${}", var))) - } - Some(Value::Text(s)) => write!(out, "{})", escape(&s[..])).unwrap(), - Some(Value::Regex(rx)) => write!(out, "{})", rx).unwrap(), - } - } - } - - } - - // Add a word boundary check `\b` to the end of the regex, but only if the final part - // is a plain text match that ends with a word character. - // - // This behavior can be disabled by ending the pattern with `$()`. - if let Some(&Part::Text(ref s)) = self.parts.last() { - if s.ends_with(char::is_alphanumeric) { - out.push_str(r"\b"); - } - } - - Ok(RegexBuilder::new(&out).multi_line(true).build()?) - } -} - -impl Display for Pattern { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - for part in &self.parts { - use self::Part::*; - match *part { - Text(ref txt) if txt == "" => write!(f, "$()"), - Text(ref txt) if txt == "$" => write!(f, "$$"), - Text(ref txt) => write!(f, "{}", txt), - Regex(ref rx) => write!(f, "$(={})", rx), - Var(ref var) => write!(f, "$({})", var), - DefLit { def, ref regex } => { - let defvar = &self.defs[def]; - // (?P...). - let litrx = ®ex[5 + defvar.len()..regex.len() - 1]; - write!(f, "$({}={})", defvar, litrx) - } - DefVar { def, ref var } => write!(f, "$({}=${})", self.defs[def], var), - }?; - } - Ok(()) - } -} - -#[cfg(test)] -mod tests { - #[test] - fn regex() { - use super::regex_prefix; - - assert_eq!(regex_prefix(""), 0); - assert_eq!(regex_prefix(")"), 0); - assert_eq!(regex_prefix(")c"), 0); - assert_eq!(regex_prefix("x"), 1); - assert_eq!(regex_prefix("x)x"), 1); - - assert_eq!(regex_prefix("x(c))x"), 4); - assert_eq!(regex_prefix("()x(c))x"), 6); - assert_eq!(regex_prefix("()x(c)"), 6); - - assert_eq!(regex_prefix("x([)]))x"), 6); - assert_eq!(regex_prefix("x[)])x"), 4); - assert_eq!(regex_prefix("x[^)])x"), 5); - assert_eq!(regex_prefix("x[^])x"), 6); - } - - #[test] - fn part() { - use super::{Pattern, Part}; - let mut pat = Pattern::new(); - - // This is dubious, should we panic instead? - assert_eq!(pat.parse_part("").unwrap(), (Part::Text("".to_string()), 0)); - - assert_eq!( - pat.parse_part("x").unwrap(), - (Part::Text("x".to_string()), 1) - ); - assert_eq!(pat.parse_part("x2").unwrap(), ( - Part::Text("x2".to_string()), - 2, - )); - assert_eq!(pat.parse_part("x$").unwrap(), ( - Part::Text("x".to_string()), - 1, - )); - assert_eq!(pat.parse_part("x$$").unwrap(), ( - Part::Text("x".to_string()), - 1, - )); - - assert_eq!( - pat.parse_part("$").unwrap_err().to_string(), - "pattern syntax error, use $$ to match a single $" - ); - - assert_eq!(pat.parse_part("$$").unwrap(), ( - Part::Text("$".to_string()), - 2, - )); - assert_eq!(pat.parse_part("$$ ").unwrap(), ( - Part::Text("$".to_string()), - 2, - )); - - assert_eq!( - pat.parse_part("$0").unwrap(), - (Part::Var("0".to_string()), 2) - ); - assert_eq!(pat.parse_part("$xx=").unwrap(), ( - Part::Var("xx".to_string()), - 3, - )); - assert_eq!(pat.parse_part("$xx$").unwrap(), ( - Part::Var("xx".to_string()), - 3, - )); - - assert_eq!(pat.parse_part("$(0)").unwrap(), ( - Part::Var("0".to_string()), - 4, - )); - assert_eq!(pat.parse_part("$()").unwrap(), ( - Part::Text("".to_string()), - 3, - )); - - assert_eq!( - pat.parse_part("$(0").unwrap_err().to_string(), - ("unterminated $(0...") - ); - assert_eq!( - pat.parse_part("$(foo:").unwrap_err().to_string(), - ("syntax error in $(foo... ':'") - ); - assert_eq!( - pat.parse_part("$(foo =").unwrap_err().to_string(), - ("syntax error in $(foo... ' '") - ); - assert_eq!( - pat.parse_part("$(eo0=$bar").unwrap_err().to_string(), - ("expected ')' after $(eo0=$bar...") - ); - assert_eq!( - pat.parse_part("$(eo1=$bar}").unwrap_err().to_string(), - ("expected ')' after $(eo1=$bar...") - ); - assert_eq!( - pat.parse_part("$(eo2=$)").unwrap_err().to_string(), - ("expected variable name in $(eo2=$...") - ); - assert_eq!( - pat.parse_part("$(eo3=$-)").unwrap_err().to_string(), - ("expected variable name in $(eo3=$...") - ); - } - - #[test] - fn partdefs() { - use super::{Pattern, Part}; - let mut pat = Pattern::new(); - - assert_eq!(pat.parse_part("$(foo=$bar)").unwrap(), ( - Part::DefVar { - def: 0, - var: "bar".to_string(), - }, - 11, - )); - assert_eq!( - pat.parse_part("$(foo=$bar)").unwrap_err().to_string(), - "duplicate definition of $foo in same pattern" - ); - - assert_eq!(pat.parse_part("$(fxo=$bar)x").unwrap(), ( - Part::DefVar { - def: 1, - var: "bar".to_string(), - }, - 11, - )); - - assert_eq!(pat.parse_part("$(fo2=[a-z])").unwrap(), ( - Part::DefLit { - def: 2, - regex: "(?P[a-z])".to_string(), - }, - 12, - )); - assert_eq!(pat.parse_part("$(fo3=[a-)])").unwrap(), ( - Part::DefLit { - def: 3, - regex: "(?P[a-)])".to_string(), - }, - 12, - )); - assert_eq!(pat.parse_part("$(fo4=)").unwrap(), ( - Part::DefLit { - def: 4, - regex: "(?P)".to_string(), - }, - 7, - )); - - assert_eq!(pat.parse_part("$(=.*)").unwrap(), ( - Part::Regex( - "(?:.*)".to_string(), - ), - 6, - )); - - assert_eq!(pat.parse_part("$(=)").unwrap(), ( - Part::Regex( - "(?:)".to_string(), - ), - 4, - )); - assert_eq!(pat.parse_part("$()").unwrap(), ( - Part::Text("".to_string()), - 3, - )); - } - - #[test] - fn pattern() { - use super::Pattern; - - let p: Pattern = " Hello world! ".parse().unwrap(); - assert_eq!(format!("{:?}", p.parts), "[Text(\"Hello world!\")]"); - - let p: Pattern = " $foo=$(bar) ".parse().unwrap(); - assert_eq!( - format!("{:?}", p.parts), - "[Var(\"foo\"), Text(\"=\"), Var(\"bar\")]" - ); - } -} diff --git a/lib/filecheck/src/variable.rs b/lib/filecheck/src/variable.rs deleted file mode 100644 index 5977f06354..0000000000 --- a/lib/filecheck/src/variable.rs +++ /dev/null @@ -1,62 +0,0 @@ -use std::borrow::Cow; - -/// A variable name is one or more ASCII alphanumerical characters, including underscore. -/// Note that numerical variable names like `$45` are allowed too. -/// -/// Try to parse a variable name from the beginning of `s`. -/// Return the index of the character following the varname. -/// This returns 0 if `s` doesn't have a prefix that is a variable name. -pub fn varname_prefix(s: &str) -> usize { - for (idx, ch) in s.char_indices() { - match ch { - 'a'...'z' | 'A'...'Z' | '0'...'9' | '_' => {} - _ => return idx, - } - } - s.len() -} - -/// A variable can contain either a regular expression or plain text. -#[derive(Debug, Clone, PartialEq, Eq)] -pub enum Value<'a> { - /// Verbatim text. - Text(Cow<'a, str>), - /// Regular expression. - Regex(Cow<'a, str>), -} - -/// Resolve variables by name. -pub trait VariableMap { - /// Get the value of the variable `varname`, or return `None` for an unknown variable name. - fn lookup(&self, varname: &str) -> Option; -} - -impl VariableMap for () { - fn lookup(&self, _: &str) -> Option { - None - } -} - -/// An empty variable map. -pub const NO_VARIABLES: &'static VariableMap = &(); - -#[cfg(test)] -mod tests { - #[test] - fn varname() { - use super::varname_prefix; - - assert_eq!(varname_prefix(""), 0); - assert_eq!(varname_prefix("\0"), 0); - assert_eq!(varname_prefix("_"), 1); - assert_eq!(varname_prefix("0"), 1); - assert_eq!(varname_prefix("01"), 2); - assert_eq!(varname_prefix("b"), 1); - assert_eq!(varname_prefix("C"), 1); - assert_eq!(varname_prefix("."), 0); - assert_eq!(varname_prefix(".s"), 0); - assert_eq!(varname_prefix("0."), 1); - assert_eq!(varname_prefix("01="), 2); - assert_eq!(varname_prefix("0a)"), 2); - } -} diff --git a/lib/filecheck/tests/basic.rs b/lib/filecheck/tests/basic.rs deleted file mode 100644 index 1ccc5991a5..0000000000 --- a/lib/filecheck/tests/basic.rs +++ /dev/null @@ -1,402 +0,0 @@ -extern crate filecheck; - -use filecheck::{CheckerBuilder, NO_VARIABLES, Error as FcError}; - -fn e2s(e: FcError) -> String { - e.to_string() -} - -#[test] -fn empty() { - let c = CheckerBuilder::new().finish(); - assert!(c.is_empty()); - - // An empty checker matches anything. - assert_eq!(c.check("", NO_VARIABLES).map_err(e2s), Ok(true)); - assert_eq!(c.check("hello", NO_VARIABLES).map_err(e2s), Ok(true)); -} - -#[test] -fn no_directives() { - let c = CheckerBuilder::new().text("nothing here").unwrap().finish(); - assert!(c.is_empty()); - - // An empty checker matches anything. - assert_eq!(c.check("", NO_VARIABLES).map_err(e2s), Ok(true)); - assert_eq!(c.check("hello", NO_VARIABLES).map_err(e2s), Ok(true)); -} - -#[test] -fn no_matches() { - let c = CheckerBuilder::new() - .text("regex: FOO=bar") - .unwrap() - .finish(); - assert!(!c.is_empty()); - - // An empty checker matches anything. - assert_eq!(c.check("", NO_VARIABLES).map_err(e2s), Ok(true)); - assert_eq!(c.check("hello", NO_VARIABLES).map_err(e2s), Ok(true)); -} - -#[test] -fn simple() { - let c = CheckerBuilder::new() - .text( - " - check: one - check: two - ", - ) - .unwrap() - .finish(); - - let t = " - zero - one - and a half - two - three - "; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); - - let t = " - zero - and a half - two - one - three - "; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); -} - -#[test] -fn sameln() { - let c = CheckerBuilder::new() - .text( - " - check: one - sameln: two - ", - ) - .unwrap() - .finish(); - - let t = " - zero - one - and a half - two - three - "; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); - - let t = " - zero - one - two - three - "; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); - - let t = " - zero - one two - three - "; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); -} - -#[test] -fn nextln() { - let c = CheckerBuilder::new() - .text( - " - check: one - nextln: two - ", - ) - .unwrap() - .finish(); - - let t = " - zero - one - and a half - two - three - "; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); - - let t = " - zero - one - two - three - "; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); - - let t = " - zero - one two - three - "; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); - - let t = " - zero - one - two"; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); -} - -#[test] -fn leading_nextln() { - // A leading nextln directive should match from line 2. - // This is somewhat arbitrary, but consistent with a preceding 'check: $()' directive. - let c = CheckerBuilder::new() - .text( - " - nextln: one - nextln: two - ", - ) - .unwrap() - .finish(); - - let t = "zero - one - two - three - "; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); - - let t = "one - two - three - "; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); -} - -#[test] -fn leading_sameln() { - // A leading sameln directive should match from line 1. - let c = CheckerBuilder::new() - .text( - " - sameln: one - sameln: two - ", - ) - .unwrap() - .finish(); - - let t = "zero - one two three - "; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); - - let t = "zero one two three"; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); - - let t = "zero one - two three"; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); -} - -#[test] -fn not() { - let c = CheckerBuilder::new() - .text( - " - check: one$() - not: $()eat$() - check: $()two - ", - ) - .unwrap() - .finish(); - - let t = "onetwo"; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); - - let t = "one eat two"; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); - - let t = "oneeattwo"; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); - - let t = "oneatwo"; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); -} - -#[test] -fn notnot() { - let c = CheckerBuilder::new() - .text( - " - check: one$() - not: $()eat$() - not: half - check: $()two - ", - ) - .unwrap() - .finish(); - - let t = "onetwo"; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); - - let t = "one eat two"; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); - - let t = "one half two"; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); - - let t = "oneeattwo"; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); - - // The `not: half` pattern only matches whole words, but the bracketing matches are considered - // word boundaries, so it does match in this case. - let t = "onehalftwo"; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(false)); - - let t = "oneatwo"; - assert_eq!(c.check(t, NO_VARIABLES).map_err(e2s), Ok(true)); -} - -#[test] -fn unordered() { - let c = CheckerBuilder::new() - .text( - " - check: one - unordered: two - unordered: three - check: four - ", - ) - .unwrap() - .finish(); - - assert_eq!( - c.check("one two three four", NO_VARIABLES).map_err(e2s), - Ok(true) - ); - assert_eq!( - c.check("one three two four", NO_VARIABLES).map_err(e2s), - Ok(true) - ); - - assert_eq!( - c.check("one two four three four", NO_VARIABLES).map_err( - e2s, - ), - Ok(true) - ); - assert_eq!( - c.check("one three four two four", NO_VARIABLES).map_err( - e2s, - ), - Ok(true) - ); - - assert_eq!( - c.check("one two four three", NO_VARIABLES).map_err(e2s), - Ok(false) - ); - assert_eq!( - c.check("one three four two", NO_VARIABLES).map_err(e2s), - Ok(false) - ); -} - -#[test] -fn leading_unordered() { - let c = CheckerBuilder::new() - .text( - " - unordered: two - unordered: three - check: four - ", - ) - .unwrap() - .finish(); - - assert_eq!( - c.check("one two three four", NO_VARIABLES).map_err(e2s), - Ok(true) - ); - assert_eq!( - c.check("one three two four", NO_VARIABLES).map_err(e2s), - Ok(true) - ); - - assert_eq!( - c.check("one two four three four", NO_VARIABLES).map_err( - e2s, - ), - Ok(true) - ); - assert_eq!( - c.check("one three four two four", NO_VARIABLES).map_err( - e2s, - ), - Ok(true) - ); - - assert_eq!( - c.check("one two four three", NO_VARIABLES).map_err(e2s), - Ok(false) - ); - assert_eq!( - c.check("one three four two", NO_VARIABLES).map_err(e2s), - Ok(false) - ); -} - -#[test] -fn trailing_unordered() { - let c = CheckerBuilder::new() - .text( - " - check: one - unordered: two - unordered: three - ", - ) - .unwrap() - .finish(); - - assert_eq!( - c.check("one two three four", NO_VARIABLES).map_err(e2s), - Ok(true) - ); - assert_eq!( - c.check("one three two four", NO_VARIABLES).map_err(e2s), - Ok(true) - ); - - assert_eq!( - c.check("one two four three four", NO_VARIABLES).map_err( - e2s, - ), - Ok(true) - ); - assert_eq!( - c.check("one three four two four", NO_VARIABLES).map_err( - e2s, - ), - Ok(true) - ); - - assert_eq!( - c.check("one two four three", NO_VARIABLES).map_err(e2s), - Ok(true) - ); - assert_eq!( - c.check("one three four two", NO_VARIABLES).map_err(e2s), - Ok(true) - ); -} diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 215f947e74..4ef875ea2c 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -14,5 +14,5 @@ name = "cton_filetests" [dependencies] cretonne = { path = "../cretonne", version = "0.3.4" } cretonne-reader = { path = "../reader", version = "0.3.4" } -filecheck = { path = "../filecheck", version = "0.1.0" } +filecheck = "0.2.0" num_cpus = "1.5.1" From 4dbafb45c7121686059dee66125911d7c0f2af21 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 15 Mar 2018 21:09:59 -0700 Subject: [PATCH 1617/3084] Update to num_cpus 1.8, and other updates. --- cranelift/Cargo.toml | 2 +- lib/filetests/Cargo.toml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index b9acc0606c..c7e9f6fef4 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -24,7 +24,7 @@ docopt = "0.8.0" serde = "1.0.8" serde_derive = "1.0.8" tempdir = "0.3.5" -term = "0.5" +term = "0.5.1" [workspace] diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 4ef875ea2c..c1dd254902 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -14,5 +14,5 @@ name = "cton_filetests" [dependencies] cretonne = { path = "../cretonne", version = "0.3.4" } cretonne-reader = { path = "../reader", version = "0.3.4" } -filecheck = "0.2.0" -num_cpus = "1.5.1" +filecheck = "0.2.1" +num_cpus = "1.8.0" From e2f3079d8b04c59c6a85e74f080e1954fc3cba0d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 15 Mar 2018 21:37:08 -0700 Subject: [PATCH 1618/3084] Tidy up redundant commands in test-all.sh. --- cranelift/test-all.sh | 2 -- 1 file changed, 2 deletions(-) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 7b86495903..2b4e2eba01 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -41,8 +41,6 @@ if [ -n "$needcheck" ]; then touch $tsfile || echo no target directory fi -cd "$topdir" - # Make sure the code builds in debug mode. banner "Rust debug build" cargo build From 5e5fcefdfd3213fddfd399b9f6c005c196bba55e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 15 Mar 2018 21:37:31 -0700 Subject: [PATCH 1619/3084] Add Travis builds with Rust nightly. See [here](https://docs.travis-ci.com/user/languages/rust/) for details. --- .travis.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.travis.yml b/.travis.yml index ce1a269955..301d7871d0 100644 --- a/.travis.yml +++ b/.travis.yml @@ -2,6 +2,10 @@ language: rust rust: - stable - beta + - nightly +matrix: + allow_failures: + - rust: nightly dist: trusty sudo: false addons: From e6db8e278c79451355acdd604f5d7cbb20347c10 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 7 Mar 2018 21:32:47 -0800 Subject: [PATCH 1620/3084] Rename one more "local" to "explicit_slot". --- misc/vim/syntax/cton.vim | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/misc/vim/syntax/cton.vim b/misc/vim/syntax/cton.vim index 6ccacae7fc..4862c38ec8 100644 --- a/misc/vim/syntax/cton.vim +++ b/misc/vim/syntax/cton.vim @@ -14,7 +14,7 @@ endif syn spell notoplevel syn keyword ctonHeader test isa set -syn keyword ctonDecl function jump_table incoming_arg outgoing_arg spill_slot local emergency_slot +syn keyword ctonDecl function jump_table incoming_arg outgoing_arg spill_slot explicit_slot emergency_slot syn keyword ctonFilecheck check sameln nextln unordered not regex contained syn match ctonType /\<\([bif]\d\+\(x\d\+\)\?\)\|[if]flags\>/ From 99f7cb5b8d0366ab848cf002061c1df2d8fc0a1e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 7 Mar 2018 21:32:56 -0800 Subject: [PATCH 1621/3084] Use debug_assert_ne rather than debug_assert with a !=. https://github.com/rust-lang-nursery/rust-clippy/wiki#should_assert_eq --- lib/cretonne/src/preopt.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/preopt.rs b/lib/cretonne/src/preopt.rs index 8e4bef8da3..a232728efd 100644 --- a/lib/cretonne/src/preopt.rs +++ b/lib/cretonne/src/preopt.rs @@ -223,7 +223,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso let t3 = pos.ins().iadd(t2, q1); // I never found any case where shiftBy == 1 here. // So there's no attempt to fold out a zero shift. - debug_assert!(shiftBy != 1); + debug_assert_ne!(shiftBy, 1); qf = pos.ins().ushr_imm(t3, (shiftBy - 1) as i64); } else { debug_assert!(shiftBy >= 0 && shiftBy <= 31); @@ -295,7 +295,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso let t3 = pos.ins().iadd(t2, q1); // I never found any case where shiftBy == 1 here. // So there's no attempt to fold out a zero shift. - debug_assert!(shiftBy != 1); + debug_assert_ne!(shiftBy, 1); qf = pos.ins().ushr_imm(t3, (shiftBy - 1) as i64); } else { debug_assert!(shiftBy >= 0 && shiftBy <= 63); From 06fe3b654fb9adfdc8f00806e945e3fe75a3c1d3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 15 Mar 2018 22:38:46 -0700 Subject: [PATCH 1622/3084] Avoid match expressions with reference patterns. https://github.com/rust-lang-nursery/rust-clippy/wiki#match_ref_pats --- lib/cretonne/src/bforest/path.rs | 44 ++++++++++----------- lib/cretonne/src/bforest/pool.rs | 8 ++-- lib/cretonne/src/preopt.rs | 66 ++++++++++++++++---------------- lib/cretonne/src/settings.rs | 4 +- 4 files changed, 61 insertions(+), 61 deletions(-) diff --git a/lib/cretonne/src/bforest/path.rs b/lib/cretonne/src/bforest/path.rs index 1add8f6e63..e52c8e897e 100644 --- a/lib/cretonne/src/bforest/path.rs +++ b/lib/cretonne/src/bforest/path.rs @@ -55,8 +55,8 @@ impl Path { for level in 0.. { self.size = level + 1; self.node[level] = node; - match &pool[node] { - &NodeData::Inner { size, keys, tree } => { + match pool[node] { + NodeData::Inner { size, keys, tree } => { // Invariant: `tree[i]` contains keys smaller than // `keys[i]`, greater or equal to `keys[i-1]`. let i = match comp.search(key, &keys[0..size.into()]) { @@ -68,7 +68,7 @@ impl Path { self.entry[level] = i as u8; node = tree[i]; } - &NodeData::Leaf { size, keys, vals } => { + NodeData::Leaf { size, keys, vals } => { // For a leaf we want either the found key or an insert position. return match comp.search(key, &keys.borrow()[0..size.into()]) { Ok(i) => { @@ -81,7 +81,7 @@ impl Path { } }; } - &NodeData::Free { .. } => panic!("Free {} reached from {}", node, root), + NodeData::Free { .. } => panic!("Free {} reached from {}", node, root), } } unreachable!(); @@ -94,10 +94,10 @@ impl Path { self.size = level + 1; self.node[level] = node; self.entry[level] = 0; - match &pool[node] { - &NodeData::Inner { tree, .. } => node = tree[0], - &NodeData::Leaf { keys, vals, .. } => return (keys.borrow()[0], vals.borrow()[0]), - &NodeData::Free { .. } => panic!("Free {} reached from {}", node, root), + match pool[node] { + NodeData::Inner { tree, .. } => node = tree[0], + NodeData::Leaf { keys, vals, .. } => return (keys.borrow()[0], vals.borrow()[0]), + NodeData::Free { .. } => panic!("Free {} reached from {}", node, root), } } unreachable!(); @@ -205,17 +205,17 @@ impl Path { let mut node = root; for l in level.. { self.node[l] = node; - match &pool[node] { - &NodeData::Inner { size, ref tree, .. } => { + match pool[node] { + NodeData::Inner { size, ref tree, .. } => { self.entry[l] = size; node = tree[usize::from(size)]; } - &NodeData::Leaf { size, .. } => { + NodeData::Leaf { size, .. } => { self.entry[l] = size - 1; self.size = l + 1; break; } - &NodeData::Free { .. } => panic!("Free {} reached from {}", node, root), + NodeData::Free { .. } => panic!("Free {} reached from {}", node, root), } } node @@ -405,8 +405,8 @@ impl Path { let crit_key = pool[self.leaf_node()].leaf_crit_key(); let crit_node = self.node[crit_level]; - match &mut pool[crit_node] { - &mut NodeData::Inner { size, ref mut keys, .. } => { + match pool[crit_node] { + NodeData::Inner { size, ref mut keys, .. } => { debug_assert!(crit_kidx < size); keys[usize::from(crit_kidx)] = crit_key; } @@ -581,8 +581,8 @@ impl Path { /// /// Returns `None` if the current node is a right-most node so no right sibling exists. fn right_sibling_branch_level(&self, level: usize, pool: &NodePool) -> Option { - (0..level).rposition(|l| match &pool[self.node[l]] { - &NodeData::Inner { size, .. } => self.entry[l] < size, + (0..level).rposition(|l| match pool[self.node[l]] { + NodeData::Inner { size, .. } => self.entry[l] < size, _ => panic!("Expected inner node"), }) } @@ -622,8 +622,8 @@ impl Path { let bl = self.right_sibling_branch_level(level, pool).expect( "No right sibling exists", ); - match &mut pool[self.node[bl]] { - &mut NodeData::Inner { ref mut keys, .. } => { + match pool[self.node[bl]] { + NodeData::Inner { ref mut keys, .. } => { keys[usize::from(self.entry[bl])] = crit_key; } _ => panic!("Expected inner node"), @@ -647,8 +647,8 @@ impl Path { /// Check the internal consistency of this path. pub fn verify(&self, pool: &NodePool) { for level in 0..self.size { - match &pool[self.node[level]] { - &NodeData::Inner { size, tree, .. } => { + match pool[self.node[level]] { + NodeData::Inner { size, tree, .. } => { assert!( level < self.size - 1, "Expected leaf node at level {}", @@ -668,7 +668,7 @@ impl Path { level ); } - &NodeData::Leaf { size, .. } => { + NodeData::Leaf { size, .. } => { assert_eq!(level, self.size - 1, "Expected inner node"); assert!( self.entry[level] <= size, @@ -677,7 +677,7 @@ impl Path { size, ); } - &NodeData::Free { .. } => { + NodeData::Free { .. } => { panic!("Free {} in path", self.node[level]); } } diff --git a/lib/cretonne/src/bforest/pool.rs b/lib/cretonne/src/bforest/pool.rs index 37801ee4ab..1ce9e83279 100644 --- a/lib/cretonne/src/bforest/pool.rs +++ b/lib/cretonne/src/bforest/pool.rs @@ -104,8 +104,8 @@ impl NodePool { ); let mut lower = lkey; - match &self[node] { - &NodeData::Inner { size, keys, tree } => { + match self[node] { + NodeData::Inner { size, keys, tree } => { let size = size as usize; let capacity = tree.len(); let keys = &keys[0..size]; @@ -147,7 +147,7 @@ impl NodePool { lower = upper; } } - &NodeData::Leaf { size, keys, .. } => { + NodeData::Leaf { size, keys, .. } => { let size = size as usize; let capacity = keys.borrow().len(); let keys = &keys.borrow()[0..size]; @@ -190,7 +190,7 @@ impl NodePool { lower = upper; } } - &NodeData::Free { .. } => panic!("Free {} reached", node), + NodeData::Free { .. } => panic!("Free {} reached", node), } } } diff --git a/lib/cretonne/src/preopt.rs b/lib/cretonne/src/preopt.rs index a232728efd..f5e8b3a7fe 100644 --- a/lib/cretonne/src/preopt.rs +++ b/lib/cretonne/src/preopt.rs @@ -170,18 +170,18 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso DivRemByConstInfo::RemS64(_, _) => true, }; - match divrem_info { + match *divrem_info { // -------------------- U32 -------------------- // U32 div, rem by zero: ignore - &DivRemByConstInfo::DivU32(_n1, 0) | - &DivRemByConstInfo::RemU32(_n1, 0) => {} + DivRemByConstInfo::DivU32(_n1, 0) | + DivRemByConstInfo::RemU32(_n1, 0) => {} // U32 div by 1: identity // U32 rem by 1: zero - &DivRemByConstInfo::DivU32(n1, 1) | - &DivRemByConstInfo::RemU32(n1, 1) => { + DivRemByConstInfo::DivU32(n1, 1) | + DivRemByConstInfo::RemU32(n1, 1) => { if isRem { pos.func.dfg.replace(inst).iconst(I32, 0); } else { @@ -190,8 +190,8 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } // U32 div, rem by a power-of-2 - &DivRemByConstInfo::DivU32(n1, d) | - &DivRemByConstInfo::RemU32(n1, d) if d.is_power_of_two() => { + DivRemByConstInfo::DivU32(n1, d) | + DivRemByConstInfo::RemU32(n1, d) if d.is_power_of_two() => { debug_assert!(d >= 2); // compute k where d == 2^k let k = d.trailing_zeros(); @@ -205,8 +205,8 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } // U32 div, rem by non-power-of-2 - &DivRemByConstInfo::DivU32(n1, d) | - &DivRemByConstInfo::RemU32(n1, d) => { + DivRemByConstInfo::DivU32(n1, d) | + DivRemByConstInfo::RemU32(n1, d) => { debug_assert!(d >= 3); let MU32 { mulBy, @@ -247,13 +247,13 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso // -------------------- U64 -------------------- // U64 div, rem by zero: ignore - &DivRemByConstInfo::DivU64(_n1, 0) | - &DivRemByConstInfo::RemU64(_n1, 0) => {} + DivRemByConstInfo::DivU64(_n1, 0) | + DivRemByConstInfo::RemU64(_n1, 0) => {} // U64 div by 1: identity // U64 rem by 1: zero - &DivRemByConstInfo::DivU64(n1, 1) | - &DivRemByConstInfo::RemU64(n1, 1) => { + DivRemByConstInfo::DivU64(n1, 1) | + DivRemByConstInfo::RemU64(n1, 1) => { if isRem { pos.func.dfg.replace(inst).iconst(I64, 0); } else { @@ -262,8 +262,8 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } // U64 div, rem by a power-of-2 - &DivRemByConstInfo::DivU64(n1, d) | - &DivRemByConstInfo::RemU64(n1, d) if d.is_power_of_two() => { + DivRemByConstInfo::DivU64(n1, d) | + DivRemByConstInfo::RemU64(n1, d) if d.is_power_of_two() => { debug_assert!(d >= 2); // compute k where d == 2^k let k = d.trailing_zeros(); @@ -277,8 +277,8 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } // U64 div, rem by non-power-of-2 - &DivRemByConstInfo::DivU64(n1, d) | - &DivRemByConstInfo::RemU64(n1, d) => { + DivRemByConstInfo::DivU64(n1, d) | + DivRemByConstInfo::RemU64(n1, d) => { debug_assert!(d >= 3); let MU64 { mulBy, @@ -319,15 +319,15 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso // -------------------- S32 -------------------- // S32 div, rem by zero or -1: ignore - &DivRemByConstInfo::DivS32(_n1, -1) | - &DivRemByConstInfo::RemS32(_n1, -1) | - &DivRemByConstInfo::DivS32(_n1, 0) | - &DivRemByConstInfo::RemS32(_n1, 0) => {} + DivRemByConstInfo::DivS32(_n1, -1) | + DivRemByConstInfo::RemS32(_n1, -1) | + DivRemByConstInfo::DivS32(_n1, 0) | + DivRemByConstInfo::RemS32(_n1, 0) => {} // S32 div by 1: identity // S32 rem by 1: zero - &DivRemByConstInfo::DivS32(n1, 1) | - &DivRemByConstInfo::RemS32(n1, 1) => { + DivRemByConstInfo::DivS32(n1, 1) | + DivRemByConstInfo::RemS32(n1, 1) => { if isRem { pos.func.dfg.replace(inst).iconst(I32, 0); } else { @@ -335,8 +335,8 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } } - &DivRemByConstInfo::DivS32(n1, d) | - &DivRemByConstInfo::RemS32(n1, d) => { + DivRemByConstInfo::DivS32(n1, d) | + DivRemByConstInfo::RemS32(n1, d) => { if let Some((isNeg, k)) = isPowerOf2_S32(d) { // k can be 31 only in the case that d is -2^31. debug_assert!(k >= 1 && k <= 31); @@ -396,15 +396,15 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso // -------------------- S64 -------------------- // S64 div, rem by zero or -1: ignore - &DivRemByConstInfo::DivS64(_n1, -1) | - &DivRemByConstInfo::RemS64(_n1, -1) | - &DivRemByConstInfo::DivS64(_n1, 0) | - &DivRemByConstInfo::RemS64(_n1, 0) => {} + DivRemByConstInfo::DivS64(_n1, -1) | + DivRemByConstInfo::RemS64(_n1, -1) | + DivRemByConstInfo::DivS64(_n1, 0) | + DivRemByConstInfo::RemS64(_n1, 0) => {} // S64 div by 1: identity // S64 rem by 1: zero - &DivRemByConstInfo::DivS64(n1, 1) | - &DivRemByConstInfo::RemS64(n1, 1) => { + DivRemByConstInfo::DivS64(n1, 1) | + DivRemByConstInfo::RemS64(n1, 1) => { if isRem { pos.func.dfg.replace(inst).iconst(I64, 0); } else { @@ -412,8 +412,8 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } } - &DivRemByConstInfo::DivS64(n1, d) | - &DivRemByConstInfo::RemS64(n1, d) => { + DivRemByConstInfo::DivS64(n1, d) | + DivRemByConstInfo::RemS64(n1, d) => { if let Some((isNeg, k)) = isPowerOf2_S64(d) { // k can be 63 only in the case that d is -2^63. debug_assert!(k >= 1 && k <= 63); diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index a87ceda695..2cfce65957 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -302,8 +302,8 @@ pub mod detail { /// Check if a detail is a Detail::Preset. Useful because the Descriptor /// offset field has a different meaning when the detail is a preset. pub fn is_preset(&self) -> bool { - match self { - &Detail::Preset => true, + match *self { + Detail::Preset => true, _ => false, } } From a4c51d66a8f353368665498d248e842f9681124a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 12 Mar 2018 14:06:15 -0700 Subject: [PATCH 1623/3084] Remove uses of `println!` in tests. In this case, it's a little nicer to just use more assertions, which will print their line number indicating how far the test got, when they fail. And this allows the tests to be run in no_std configurations. --- lib/cretonne/src/divconst_magic_numbers.rs | 23 ++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/src/divconst_magic_numbers.rs b/lib/cretonne/src/divconst_magic_numbers.rs index b046e22411..f189e23d1a 100644 --- a/lib/cretonne/src/divconst_magic_numbers.rs +++ b/lib/cretonne/src/divconst_magic_numbers.rs @@ -480,56 +480,63 @@ mod tests { // don't panic with integer wraparounds, especially at boundary // cases for their arguments. The actual results are thrown away. let mut total: u64 = 0; - println!("Testing UP magicU32"); + // Testing UP magicU32 for x in 2..(200 * 1000u32) { let m = magicU32(x); total = total ^ (m.mulBy as u64); total = total + (m.shiftBy as u64); total = total - (if m.doAdd { 123 } else { 456 }); } - println!("Testing DOWN magicU32"); + assert_eq!(total, 1747815691); + // Testing DOWN magicU32 for x in 0..(200 * 1000u32) { let m = magicU32(0xFFFF_FFFFu32 - x); total = total ^ (m.mulBy as u64); total = total + (m.shiftBy as u64); total = total - (if m.doAdd { 123 } else { 456 }); } + assert_eq!(total, 2210292772); - println!("Testing UP magicU64"); + // Testing UP magicU64 for x in 2..(200 * 1000u64) { let m = magicU64(x); total = total ^ m.mulBy; total = total + (m.shiftBy as u64); total = total - (if m.doAdd { 123 } else { 456 }); } - println!("Testing DOWN magicU64"); + assert_eq!(total, 7430004084791260605); + // Testing DOWN magicU64 for x in 0..(200 * 1000u64) { let m = magicU64(0xFFFF_FFFF_FFFF_FFFFu64 - x); total = total ^ m.mulBy; total = total + (m.shiftBy as u64); total = total - (if m.doAdd { 123 } else { 456 }); } + assert_eq!(total, 7547519887519825919); - println!("Testing UP magicS32"); + // Testing UP magicS32 for x in 0..(200 * 1000i32) { let m = magicS32(-0x8000_0000i32 + x); total = total ^ (m.mulBy as u64); total = total + (m.shiftBy as u64); } - println!("Testing DOWN magicS32"); + assert_eq!(total, 10899224186731671235); + // Testing DOWN magicS32 for x in 0..(200 * 1000i32) { let m = magicS32(0x7FFF_FFFFi32 - x); total = total ^ (m.mulBy as u64); total = total + (m.shiftBy as u64); } + assert_eq!(total, 7547519887517897369); - println!("Testing UP magicS64"); + // Testing UP magicS64 for x in 0..(200 * 1000i64) { let m = magicS64(-0x8000_0000_0000_0000i64 + x); total = total ^ (m.mulBy as u64); total = total + (m.shiftBy as u64); } - println!("Testing DOWN magicS64"); + assert_eq!(total, 8029756891368555163); + // Testing DOWN magicS64 for x in 0..(200 * 1000i64) { let m = magicS64(0x7FFF_FFFF_FFFF_FFFFi64 - x); total = total ^ (m.mulBy as u64); From 5e21ac1a307a088913fc83066cb27d0758b4ff8c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 16 Mar 2018 16:01:15 -0700 Subject: [PATCH 1624/3084] Use whitespace more consistently in Cargo.toml files. --- lib/cretonne/Cargo.toml | 2 +- lib/wasm/Cargo.toml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index adcaa28f76..96385da3e8 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -7,7 +7,7 @@ license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/Cretonne/cretonne" readme = "README.md" -keywords = [ "compile", "compiler", "jit" ] +keywords = ["compile", "compiler", "jit"] build = "build.rs" [lib] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 140ab69cfe..536c31daa6 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -6,7 +6,7 @@ description = "Translator from WebAssembly to Cretonne IL" repository = "https://github.com/Cretonne/cretonne" license = "Apache-2.0" readme = "README.md" -keywords = [ "webassembly", "wasm" ] +keywords = ["webassembly", "wasm"] [lib] name = "cton_wasm" From 921cea284532b2cf23816d1400a618f112197273 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 16 Mar 2018 16:12:56 -0700 Subject: [PATCH 1625/3084] Bump version to 0.4.0 --- cranelift/Cargo.toml | 14 +++++++------- cranelift/publish-all.sh | 2 +- lib/cretonne/Cargo.toml | 2 +- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/wasm/Cargo.toml | 6 +++--- 8 files changed, 21 insertions(+), 21 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index c7e9f6fef4..a6d363dcdb 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.3.4" +version = "0.4.0" description = "Binaries for testing the Cretonne library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,12 +13,12 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne = { path = "lib/cretonne", version = "0.3.4" } -cretonne-reader = { path = "lib/reader", version = "0.3.4" } -cretonne-frontend = { path = "lib/frontend", version = "0.3.4" } -cretonne-wasm = { path = "lib/wasm", version = "0.3.4" } -cretonne-native = { path = "lib/native", version = "0.3.4" } -cretonne-filetests = { path = "lib/filetests", version = "0.3.4" } +cretonne = { path = "lib/cretonne", version = "0.4.0" } +cretonne-reader = { path = "lib/reader", version = "0.4.0" } +cretonne-frontend = { path = "lib/frontend", version = "0.4.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.4.0" } +cretonne-native = { path = "lib/native", version = "0.4.0" } +cretonne-filetests = { path = "lib/filetests", version = "0.4.0" } filecheck = "0.2.1" docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index c5cd6bfca3..6b23f71064 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ cd $(dirname "$0") topdir="$(pwd)" # All the cretonne-* crates have the same version number -version="0.3.4" +version="0.4.0" # Update all of the Cargo.toml files. # diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 96385da3e8..d603a9597e 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.3.4" +version = "0.4.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index c1dd254902..f3ecf521c0 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.3.4" +version = "0.4.0" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "http://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -12,7 +12,7 @@ publish = false name = "cton_filetests" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.4" } -cretonne-reader = { path = "../reader", version = "0.3.4" } +cretonne = { path = "../cretonne", version = "0.4.0" } +cretonne-reader = { path = "../reader", version = "0.4.0" } filecheck = "0.2.1" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 368b6f058e..bd5455d9d3 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.3.4" +version = "0.4.0" description = "Cretonne IL builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,7 +12,7 @@ readme = "README.md" name = "cton_frontend" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.4" } +cretonne = { path = "../cretonne", version = "0.4.0" } [badges] maintenance = { status = "experimental" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 89676fc1b3..9a2ba0eea7 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.3.4" +version = "0.4.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -11,7 +11,7 @@ readme = "README.md" name = "cton_native" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.4" } +cretonne = { path = "../cretonne", version = "0.4.0" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index bd26dc0f7c..1841357bc4 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.3.4" +version = "0.4.0" description = "Cretonne textual IL reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,7 +12,7 @@ readme = "README.md" name = "cton_reader" [dependencies] -cretonne = { path = "../cretonne", version = "0.3.4" } +cretonne = { path = "../cretonne", version = "0.4.0" } [badges] maintenance = { status = "experimental" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 536c31daa6..df978cb4d0 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.3.4" +version = "0.4.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IL" repository = "https://github.com/Cretonne/cretonne" @@ -13,8 +13,8 @@ name = "cton_wasm" [dependencies] wasmparser = "0.15.1" -cretonne = { path = "../cretonne", version = "0.3.4" } -cretonne-frontend = { path = "../frontend", version = "0.3.4" } +cretonne = { path = "../cretonne", version = "0.4.0" } +cretonne-frontend = { path = "../frontend", version = "0.4.0" } [dev-dependencies] tempdir = "0.3.5" From 9a49bc2ec97a2ed7b536487d42b61139322d41fb Mon Sep 17 00:00:00 2001 From: Afnan Enayet Date: Sun, 18 Mar 2018 13:50:51 -0700 Subject: [PATCH 1626/3084] Rename `I32` -> `X86_32` and `I64` -> `X86_64` (#271) * Rename `I32` -> `X86_32` and `I64` -> `X86_64` * Format file to pass flake8 tests * Fix comment so lines are under 80 char limit * Remove trailing whitespace from comment * Renamed `enc_i64` to `enc_x86_64` as per suggestion from PR --- lib/cretonne/meta/isa/intel/defs.py | 4 +- lib/cretonne/meta/isa/intel/encodings.py | 306 +++++++++++------------ 2 files changed, 155 insertions(+), 155 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/defs.py b/lib/cretonne/meta/isa/intel/defs.py index d5bb0b5a1f..b6a4d37206 100644 --- a/lib/cretonne/meta/isa/intel/defs.py +++ b/lib/cretonne/meta/isa/intel/defs.py @@ -12,8 +12,8 @@ from base.immediates import floatcc ISA = TargetISA('intel', [base.instructions.GROUP, x86.GROUP]) # CPU modes for 32-bit and 64-bit operation. -I64 = CPUMode('I64', ISA) -I32 = CPUMode('I32', ISA) +X86_64 = CPUMode('I64', ISA) +X86_32 = CPUMode('I32', ISA) # The set of floating point condition codes that are directly supported. # Other condition codes need to be reversed or expressed as two tests. diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 162caccf65..995149794a 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -5,7 +5,7 @@ from __future__ import absolute_import from cdsl.predicates import IsUnsignedInt, Not, And from base import instructions as base from base.formats import UnaryImm -from .defs import I32, I64 +from .defs import X86_64, X86_32 from . import recipes as r from . import settings as cfg from . import instructions as x86 @@ -22,83 +22,83 @@ except ImportError: pass -I32.legalize_monomorphic(expand_flags) -I32.legalize_type( - default=narrow, - b1=expand_flags, - i32=intel_expand, - f32=intel_expand, - f64=intel_expand) +X86_32.legalize_monomorphic(expand_flags) +X86_32.legalize_type( + default=narrow, + b1=expand_flags, + i32=intel_expand, + f32=intel_expand, + f64=intel_expand) -I64.legalize_monomorphic(expand_flags) -I64.legalize_type( - default=narrow, - b1=expand_flags, - i32=intel_expand, - i64=intel_expand, - f32=intel_expand, - f64=intel_expand) +X86_64.legalize_monomorphic(expand_flags) +X86_64.legalize_type( + default=narrow, + b1=expand_flags, + i32=intel_expand, + i64=intel_expand, + f32=intel_expand, + f64=intel_expand) # # Helper functions for generating encodings. # -def enc_i64(inst, recipe, *args, **kwargs): +def enc_x86_64(inst, recipe, *args, **kwargs): # type: (MaybeBoundInst, r.TailRecipe, *int, **int) -> None """ - Add encodings for `inst` to I64 with and without a REX prefix. + Add encodings for `inst` to X86_64 with and without a REX prefix. """ - I64.enc(inst, *recipe.rex(*args, **kwargs)) - I64.enc(inst, *recipe(*args, **kwargs)) + X86_64.enc(inst, *recipe.rex(*args, **kwargs)) + X86_64.enc(inst, *recipe(*args, **kwargs)) def enc_both(inst, recipe, *args, **kwargs): # type: (MaybeBoundInst, r.TailRecipe, *int, **Any) -> None """ - Add encodings for `inst` to both I32 and I64. + Add encodings for `inst` to both X86_32 and X86_64. """ - I32.enc(inst, *recipe(*args, **kwargs)) - enc_i64(inst, recipe, *args, **kwargs) + X86_32.enc(inst, *recipe(*args, **kwargs)) + enc_x86_64(inst, recipe, *args, **kwargs) def enc_i32_i64(inst, recipe, *args, **kwargs): # type: (MaybeBoundInst, r.TailRecipe, *int, **int) -> None """ - Add encodings for `inst.i32` to I32. - Add encodings for `inst.i32` to I64 with and without REX. - Add encodings for `inst.i64` to I64 with a REX.W prefix. + Add encodings for `inst.i32` to X86_32. + Add encodings for `inst.i32` to X86_64 with and without REX. + Add encodings for `inst.i64` to X86_64 with a REX.W prefix. """ - I32.enc(inst.i32, *recipe(*args, **kwargs)) + X86_32.enc(inst.i32, *recipe(*args, **kwargs)) # REX-less encoding must come after REX encoding so we don't use it by # default. Otherwise reg-alloc would never use r8 and up. - I64.enc(inst.i32, *recipe.rex(*args, **kwargs)) - I64.enc(inst.i32, *recipe(*args, **kwargs)) + X86_64.enc(inst.i32, *recipe.rex(*args, **kwargs)) + X86_64.enc(inst.i32, *recipe(*args, **kwargs)) - I64.enc(inst.i64, *recipe.rex(*args, w=1, **kwargs)) + X86_64.enc(inst.i64, *recipe.rex(*args, w=1, **kwargs)) def enc_i32_i64_ld_st(inst, w_bit, recipe, *args, **kwargs): # type: (MaybeBoundInst, bool, r.TailRecipe, *int, **int) -> None """ - Add encodings for `inst.i32` to I32. - Add encodings for `inst.i32` to I64 with and without REX. - Add encodings for `inst.i64` to I64 with a REX prefix, using the `w_bit` + Add encodings for `inst.i32` to X86_32. + Add encodings for `inst.i32` to X86_64 with and without REX. + Add encodings for `inst.i64` to X86_64 with a REX prefix, using the `w_bit` argument to determine whether or not to set the REX.W bit. """ - I32.enc(inst.i32.any, *recipe(*args, **kwargs)) + X86_32.enc(inst.i32.any, *recipe(*args, **kwargs)) # REX-less encoding must come after REX encoding so we don't use it by # default. Otherwise reg-alloc would never use r8 and up. - I64.enc(inst.i32.any, *recipe.rex(*args, **kwargs)) - I64.enc(inst.i32.any, *recipe(*args, **kwargs)) + X86_64.enc(inst.i32.any, *recipe.rex(*args, **kwargs)) + X86_64.enc(inst.i32.any, *recipe(*args, **kwargs)) if w_bit: - I64.enc(inst.i64.any, *recipe.rex(*args, w=1, **kwargs)) + X86_64.enc(inst.i64.any, *recipe.rex(*args, w=1, **kwargs)) else: - I64.enc(inst.i64.any, *recipe.rex(*args, **kwargs)) - I64.enc(inst.i64.any, *recipe(*args, **kwargs)) + X86_64.enc(inst.i64.any, *recipe.rex(*args, **kwargs)) + X86_64.enc(inst.i64.any, *recipe(*args, **kwargs)) for inst, opc in [ @@ -141,19 +141,19 @@ for inst, rrr in [ # band_imm.i32. Can even use the single-byte immediate for 0xffff_ffXX masks. # Immediate constants. -I32.enc(base.iconst.i32, *r.puid(0xb8)) +X86_32.enc(base.iconst.i32, *r.puid(0xb8)) -I64.enc(base.iconst.i32, *r.puid.rex(0xb8)) -I64.enc(base.iconst.i32, *r.puid(0xb8)) +X86_64.enc(base.iconst.i32, *r.puid.rex(0xb8)) +X86_64.enc(base.iconst.i32, *r.puid(0xb8)) # The 32-bit immediate movl also zero-extends to 64 bits. -I64.enc(base.iconst.i64, *r.puid.rex(0xb8), - instp=IsUnsignedInt(UnaryImm.imm, 32)) -I64.enc(base.iconst.i64, *r.puid(0xb8), - instp=IsUnsignedInt(UnaryImm.imm, 32)) +X86_64.enc(base.iconst.i64, *r.puid.rex(0xb8), + instp=IsUnsignedInt(UnaryImm.imm, 32)) +X86_64.enc(base.iconst.i64, *r.puid(0xb8), + instp=IsUnsignedInt(UnaryImm.imm, 32)) # Sign-extended 32-bit immediate. -I64.enc(base.iconst.i64, *r.uid.rex(0xc7, rrr=0, w=1)) +X86_64.enc(base.iconst.i64, *r.uid.rex(0xc7, rrr=0, w=1)) # Finally, the 0xb8 opcode takes an 8-byte immediate with a REX.W prefix. -I64.enc(base.iconst.i64, *r.puiq.rex(0xb8, w=1)) +X86_64.enc(base.iconst.i64, *r.puiq.rex(0xb8, w=1)) # Shifts and rotates. # Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit @@ -164,38 +164,38 @@ for inst, rrr in [ (base.ishl, 4), (base.ushr, 5), (base.sshr, 7)]: - I32.enc(inst.i32.any, *r.rc(0xd3, rrr=rrr)) - I64.enc(inst.i64.any, *r.rc.rex(0xd3, rrr=rrr, w=1)) - I64.enc(inst.i32.any, *r.rc.rex(0xd3, rrr=rrr)) - I64.enc(inst.i32.any, *r.rc(0xd3, rrr=rrr)) + X86_32.enc(inst.i32.any, *r.rc(0xd3, rrr=rrr)) + X86_64.enc(inst.i64.any, *r.rc.rex(0xd3, rrr=rrr, w=1)) + X86_64.enc(inst.i32.any, *r.rc.rex(0xd3, rrr=rrr)) + X86_64.enc(inst.i32.any, *r.rc(0xd3, rrr=rrr)) # Population count. -I32.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) -I64.enc(base.popcnt.i64, *r.urm.rex(0xf3, 0x0f, 0xb8, w=1), - isap=cfg.use_popcnt) -I64.enc(base.popcnt.i32, *r.urm.rex(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) -I64.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) +X86_32.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) +X86_64.enc(base.popcnt.i64, *r.urm.rex(0xf3, 0x0f, 0xb8, w=1), + isap=cfg.use_popcnt) +X86_64.enc(base.popcnt.i32, *r.urm.rex(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) +X86_64.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) # Count leading zero bits. -I32.enc(base.clz.i32, *r.urm(0xf3, 0x0f, 0xbd), isap=cfg.use_lzcnt) -I64.enc(base.clz.i64, *r.urm.rex(0xf3, 0x0f, 0xbd, w=1), - isap=cfg.use_lzcnt) -I64.enc(base.clz.i32, *r.urm.rex(0xf3, 0x0f, 0xbd), isap=cfg.use_lzcnt) -I64.enc(base.clz.i32, *r.urm(0xf3, 0x0f, 0xbd), isap=cfg.use_lzcnt) +X86_32.enc(base.clz.i32, *r.urm(0xf3, 0x0f, 0xbd), isap=cfg.use_lzcnt) +X86_64.enc(base.clz.i64, *r.urm.rex(0xf3, 0x0f, 0xbd, w=1), + isap=cfg.use_lzcnt) +X86_64.enc(base.clz.i32, *r.urm.rex(0xf3, 0x0f, 0xbd), isap=cfg.use_lzcnt) +X86_64.enc(base.clz.i32, *r.urm(0xf3, 0x0f, 0xbd), isap=cfg.use_lzcnt) # Count trailing zero bits. -I32.enc(base.ctz.i32, *r.urm(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) -I64.enc(base.ctz.i64, *r.urm.rex(0xf3, 0x0f, 0xbc, w=1), - isap=cfg.use_bmi1) -I64.enc(base.ctz.i32, *r.urm.rex(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) -I64.enc(base.ctz.i32, *r.urm(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) +X86_32.enc(base.ctz.i32, *r.urm(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) +X86_64.enc(base.ctz.i64, *r.urm.rex(0xf3, 0x0f, 0xbc, w=1), + isap=cfg.use_bmi1) +X86_64.enc(base.ctz.i32, *r.urm.rex(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) +X86_64.enc(base.ctz.i32, *r.urm(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) # # Loads and stores. # for recipe in [r.st, r.stDisp8, r.stDisp32]: enc_i32_i64_ld_st(base.store, True, recipe, 0x89) - enc_i64(base.istore32.i64.any, recipe, 0x89) + enc_x86_64(base.istore32.i64.any, recipe, 0x89) enc_i32_i64_ld_st(base.istore16, False, recipe, 0x66, 0x89) # Byte stores are more complicated because the registers they can address @@ -203,7 +203,7 @@ for recipe in [r.st, r.stDisp8, r.stDisp32]: # the corresponding st* recipes when a REX prefix is applied. for recipe in [r.st_abcd, r.stDisp8_abcd, r.stDisp32_abcd]: enc_both(base.istore8.i32.any, recipe, 0x88) - enc_i64(base.istore8.i64.any, recipe, 0x88) + enc_x86_64(base.istore8.i64.any, recipe, 0x88) enc_i32_i64(base.spill, r.spSib32, 0x89) enc_i32_i64(base.regspill, r.rsp32, 0x89) @@ -216,8 +216,8 @@ enc_both(base.regspill.b1, r.rsp32, 0x89) for recipe in [r.ld, r.ldDisp8, r.ldDisp32]: enc_i32_i64_ld_st(base.load, True, recipe, 0x8b) - enc_i64(base.uload32.i64, recipe, 0x8b) - I64.enc(base.sload32.i64, *recipe.rex(0x63, w=1)) + enc_x86_64(base.uload32.i64, recipe, 0x8b) + X86_64.enc(base.sload32.i64, *recipe.rex(0x63, w=1)) enc_i32_i64_ld_st(base.uload16, True, recipe, 0x0f, 0xb7) enc_i32_i64_ld_st(base.sload16, True, recipe, 0x0f, 0xbf) enc_i32_i64_ld_st(base.uload8, True, recipe, 0x0f, 0xb6) @@ -231,21 +231,21 @@ enc_both(base.fill.b1, r.fiSib32, 0x8b) enc_both(base.regfill.b1, r.rfi32, 0x8b) # Push and Pop -I32.enc(x86.push.i32, *r.pushq(0x50)) -enc_i64(x86.push.i64, r.pushq, 0x50) +X86_32.enc(x86.push.i32, *r.pushq(0x50)) +enc_x86_64(x86.push.i64, r.pushq, 0x50) -I32.enc(x86.pop.i32, *r.popq(0x58)) -enc_i64(x86.pop.i64, r.popq, 0x58) +X86_32.enc(x86.pop.i32, *r.popq(0x58)) +enc_x86_64(x86.pop.i64, r.popq, 0x58) # Copy Special -I64.enc(base.copy_special, *r.copysp.rex(0x89, w=1)) -I32.enc(base.copy_special, *r.copysp(0x89)) +X86_64.enc(base.copy_special, *r.copysp.rex(0x89, w=1)) +X86_32.enc(base.copy_special, *r.copysp(0x89)) # Adjust SP Imm -I32.enc(base.adjust_sp_imm, *r.adjustsp8(0x83)) -I32.enc(base.adjust_sp_imm, *r.adjustsp32(0x81)) -I64.enc(base.adjust_sp_imm, *r.adjustsp8.rex(0x83, w=1)) -I64.enc(base.adjust_sp_imm, *r.adjustsp32.rex(0x81, w=1)) +X86_32.enc(base.adjust_sp_imm, *r.adjustsp8(0x83)) +X86_32.enc(base.adjust_sp_imm, *r.adjustsp32(0x81)) +X86_64.enc(base.adjust_sp_imm, *r.adjustsp8.rex(0x83, w=1)) +X86_64.enc(base.adjust_sp_imm, *r.adjustsp32.rex(0x81, w=1)) # # Float loads and stores. @@ -281,43 +281,43 @@ enc_both(base.regspill.f64, r.frsp32, 0x66, 0x0f, 0xd6) # Function addresses. # -I32.enc(base.func_addr.i32, *r.fnaddr4(0xb8), - isap=Not(allones_funcaddrs)) -I64.enc(base.func_addr.i64, *r.fnaddr8.rex(0xb8, w=1), - isap=And(Not(allones_funcaddrs), Not(is_pic))) +X86_32.enc(base.func_addr.i32, *r.fnaddr4(0xb8), + isap=Not(allones_funcaddrs)) +X86_64.enc(base.func_addr.i64, *r.fnaddr8.rex(0xb8, w=1), + isap=And(Not(allones_funcaddrs), Not(is_pic))) -I32.enc(base.func_addr.i32, *r.allones_fnaddr4(0xb8), - isap=allones_funcaddrs) -I64.enc(base.func_addr.i64, *r.allones_fnaddr8.rex(0xb8, w=1), - isap=And(allones_funcaddrs, Not(is_pic))) +X86_32.enc(base.func_addr.i32, *r.allones_fnaddr4(0xb8), + isap=allones_funcaddrs) +X86_64.enc(base.func_addr.i64, *r.allones_fnaddr8.rex(0xb8, w=1), + isap=And(allones_funcaddrs, Not(is_pic))) -I64.enc(base.func_addr.i64, *r.got_fnaddr8.rex(0x8b, w=1), - isap=is_pic) +X86_64.enc(base.func_addr.i64, *r.got_fnaddr8.rex(0x8b, w=1), + isap=is_pic) # # Global addresses. # -I32.enc(base.globalsym_addr.i32, *r.gvaddr4(0xb8)) -I64.enc(base.globalsym_addr.i64, *r.gvaddr8.rex(0xb8, w=1), - isap=Not(is_pic)) +X86_32.enc(base.globalsym_addr.i32, *r.gvaddr4(0xb8)) +X86_64.enc(base.globalsym_addr.i64, *r.gvaddr8.rex(0xb8, w=1), + isap=Not(is_pic)) -I64.enc(base.globalsym_addr.i64, *r.got_gvaddr8.rex(0x8b, w=1), - isap=is_pic) +X86_64.enc(base.globalsym_addr.i64, *r.got_gvaddr8.rex(0x8b, w=1), + isap=is_pic) # # Call/return # -I32.enc(base.call, *r.call_id(0xe8)) -I64.enc(base.call, *r.call_id(0xe8), isap=Not(is_pic)) -I64.enc(base.call, *r.call_plt_id(0xe8), isap=is_pic) +X86_32.enc(base.call, *r.call_id(0xe8)) +X86_64.enc(base.call, *r.call_id(0xe8), isap=Not(is_pic)) +X86_64.enc(base.call, *r.call_plt_id(0xe8), isap=is_pic) -I32.enc(base.call_indirect.i32, *r.call_r(0xff, rrr=2)) -I64.enc(base.call_indirect.i64, *r.call_r.rex(0xff, rrr=2)) -I64.enc(base.call_indirect.i64, *r.call_r(0xff, rrr=2)) +X86_32.enc(base.call_indirect.i32, *r.call_r(0xff, rrr=2)) +X86_64.enc(base.call_indirect.i64, *r.call_r.rex(0xff, rrr=2)) +X86_64.enc(base.call_indirect.i64, *r.call_r(0xff, rrr=2)) -I32.enc(base.x_return, *r.ret(0xc3)) -I64.enc(base.x_return, *r.ret(0xc3)) +X86_32.enc(base.x_return, *r.ret(0xc3)) +X86_64.enc(base.x_return, *r.ret(0xc3)) # # Branches @@ -341,10 +341,10 @@ enc_i32_i64(base.brnz, r.tjccd, 0x85) # Branch on a b1 value in a register only looks at the low 8 bits. See also # bint encodings below. # -# Start with the worst-case encoding for I32 only. The register allocator can't -# handle a branch with an ABCD-constrained operand. -I32.enc(base.brz.b1, *r.t8jccd_long(0x84)) -I32.enc(base.brnz.b1, *r.t8jccd_long(0x85)) +# Start with the worst-case encoding for X86_32 only. The register allocator +# can't handle a branch with an ABCD-constrained operand. +X86_32.enc(base.brz.b1, *r.t8jccd_long(0x84)) +X86_32.enc(base.brnz.b1, *r.t8jccd_long(0x85)) enc_both(base.brz.b1, r.t8jccb_abcd, 0x74) enc_both(base.brz.b1, r.t8jccd_abcd, 0x84) @@ -354,14 +354,14 @@ enc_both(base.brnz.b1, r.t8jccd_abcd, 0x85) # # Trap as ud2 # -I32.enc(base.trap, *r.trap(0x0f, 0x0b)) -I64.enc(base.trap, *r.trap(0x0f, 0x0b)) +X86_32.enc(base.trap, *r.trap(0x0f, 0x0b)) +X86_64.enc(base.trap, *r.trap(0x0f, 0x0b)) # Using a standard EncRecipe, not the TailRecipe. -I32.enc(base.trapif, r.trapif, 0) -I64.enc(base.trapif, r.trapif, 0) -I32.enc(base.trapff, r.trapff, 0) -I64.enc(base.trapff, r.trapff, 0) +X86_32.enc(base.trapif, r.trapif, 0) +X86_64.enc(base.trapif, r.trapif, 0) +X86_32.enc(base.trapff, r.trapff, 0) +X86_64.enc(base.trapff, r.trapff, 0) # # Comparisons @@ -372,8 +372,8 @@ enc_i32_i64(base.ifcmp_imm, r.rcmpib, 0x83, rrr=7) enc_i32_i64(base.ifcmp_imm, r.rcmpid, 0x81, rrr=7) # TODO: We could special-case ifcmp_imm(x, 0) to TEST(x, x). -I32.enc(base.ifcmp_sp.i32, *r.rcmp_sp(0x39)) -I64.enc(base.ifcmp_sp.i64, *r.rcmp_sp.rex(0x39, w=1)) +X86_32.enc(base.ifcmp_sp.i32, *r.rcmp_sp(0x39)) +X86_64.enc(base.ifcmp_sp.i64, *r.rcmp_sp.rex(0x39, w=1)) # # Convert flags to bool. @@ -398,66 +398,66 @@ enc_i32_i64(x86.bsr, r.bsf_and_bsr, 0x0F, 0xBD) # # This assumes that b1 is represented as an 8-bit low register with the value 0 # or 1. -I32.enc(base.bint.i32.b1, *r.urm_abcd(0x0f, 0xb6)) -I64.enc(base.bint.i64.b1, *r.urm.rex(0x0f, 0xb6)) # zext to i64 implicit. -I64.enc(base.bint.i64.b1, *r.urm_abcd(0x0f, 0xb6)) # zext to i64 implicit. -I64.enc(base.bint.i32.b1, *r.urm.rex(0x0f, 0xb6)) -I64.enc(base.bint.i32.b1, *r.urm_abcd(0x0f, 0xb6)) +X86_32.enc(base.bint.i32.b1, *r.urm_abcd(0x0f, 0xb6)) +X86_64.enc(base.bint.i64.b1, *r.urm.rex(0x0f, 0xb6)) # zext to i64 implicit. +X86_64.enc(base.bint.i64.b1, *r.urm_abcd(0x0f, 0xb6)) # zext to i64 implicit. +X86_64.enc(base.bint.i32.b1, *r.urm.rex(0x0f, 0xb6)) +X86_64.enc(base.bint.i32.b1, *r.urm_abcd(0x0f, 0xb6)) # Numerical conversions. # Reducing an integer is a no-op. -I32.enc(base.ireduce.i8.i32, r.null, 0) -I32.enc(base.ireduce.i16.i32, r.null, 0) -I64.enc(base.ireduce.i8.i32, r.null, 0) -I64.enc(base.ireduce.i16.i32, r.null, 0) -I64.enc(base.ireduce.i8.i64, r.null, 0) -I64.enc(base.ireduce.i16.i64, r.null, 0) -I64.enc(base.ireduce.i32.i64, r.null, 0) +X86_32.enc(base.ireduce.i8.i32, r.null, 0) +X86_32.enc(base.ireduce.i16.i32, r.null, 0) +X86_64.enc(base.ireduce.i8.i32, r.null, 0) +X86_64.enc(base.ireduce.i16.i32, r.null, 0) +X86_64.enc(base.ireduce.i8.i64, r.null, 0) +X86_64.enc(base.ireduce.i16.i64, r.null, 0) +X86_64.enc(base.ireduce.i32.i64, r.null, 0) # TODO: Add encodings for cbw, cwde, cdqe, which are sign-extending # instructions for %al/%ax/%eax to %ax/%eax/%rax. # movsbl -I32.enc(base.sextend.i32.i8, *r.urm(0x0f, 0xbe)) -I64.enc(base.sextend.i32.i8, *r.urm.rex(0x0f, 0xbe)) -I64.enc(base.sextend.i32.i8, *r.urm(0x0f, 0xbe)) +X86_32.enc(base.sextend.i32.i8, *r.urm(0x0f, 0xbe)) +X86_64.enc(base.sextend.i32.i8, *r.urm.rex(0x0f, 0xbe)) +X86_64.enc(base.sextend.i32.i8, *r.urm(0x0f, 0xbe)) # movswl -I32.enc(base.sextend.i32.i16, *r.urm(0x0f, 0xbf)) -I64.enc(base.sextend.i32.i16, *r.urm.rex(0x0f, 0xbf)) -I64.enc(base.sextend.i32.i16, *r.urm(0x0f, 0xbf)) +X86_32.enc(base.sextend.i32.i16, *r.urm(0x0f, 0xbf)) +X86_64.enc(base.sextend.i32.i16, *r.urm.rex(0x0f, 0xbf)) +X86_64.enc(base.sextend.i32.i16, *r.urm(0x0f, 0xbf)) # movsbq -I64.enc(base.sextend.i64.i8, *r.urm.rex(0x0f, 0xbe, w=1)) +X86_64.enc(base.sextend.i64.i8, *r.urm.rex(0x0f, 0xbe, w=1)) # movswq -I64.enc(base.sextend.i64.i16, *r.urm.rex(0x0f, 0xbf, w=1)) +X86_64.enc(base.sextend.i64.i16, *r.urm.rex(0x0f, 0xbf, w=1)) # movslq -I64.enc(base.sextend.i64.i32, *r.urm.rex(0x63, w=1)) +X86_64.enc(base.sextend.i64.i32, *r.urm.rex(0x63, w=1)) # movzbl -I32.enc(base.uextend.i32.i8, *r.urm(0x0f, 0xb6)) -I64.enc(base.uextend.i32.i8, *r.urm.rex(0x0f, 0xb6)) -I64.enc(base.uextend.i32.i8, *r.urm(0x0f, 0xb6)) +X86_32.enc(base.uextend.i32.i8, *r.urm(0x0f, 0xb6)) +X86_64.enc(base.uextend.i32.i8, *r.urm.rex(0x0f, 0xb6)) +X86_64.enc(base.uextend.i32.i8, *r.urm(0x0f, 0xb6)) # movzwl -I32.enc(base.uextend.i32.i16, *r.urm(0x0f, 0xb7)) -I64.enc(base.uextend.i32.i16, *r.urm.rex(0x0f, 0xb7)) -I64.enc(base.uextend.i32.i16, *r.urm(0x0f, 0xb7)) +X86_32.enc(base.uextend.i32.i16, *r.urm(0x0f, 0xb7)) +X86_64.enc(base.uextend.i32.i16, *r.urm.rex(0x0f, 0xb7)) +X86_64.enc(base.uextend.i32.i16, *r.urm(0x0f, 0xb7)) # movzbq, encoded as movzbl because it's equivalent and shorter -I64.enc(base.uextend.i64.i8, *r.urm.rex(0x0f, 0xb6)) -I64.enc(base.uextend.i64.i8, *r.urm(0x0f, 0xb6)) +X86_64.enc(base.uextend.i64.i8, *r.urm.rex(0x0f, 0xb6)) +X86_64.enc(base.uextend.i64.i8, *r.urm(0x0f, 0xb6)) # movzwq, encoded as movzwl because it's equivalent and shorter -I64.enc(base.uextend.i64.i16, *r.urm.rex(0x0f, 0xb7)) -I64.enc(base.uextend.i64.i16, *r.urm(0x0f, 0xb7)) +X86_64.enc(base.uextend.i64.i16, *r.urm.rex(0x0f, 0xb7)) +X86_64.enc(base.uextend.i64.i16, *r.urm(0x0f, 0xb7)) # A 32-bit register copy clears the high 32 bits. -I64.enc(base.uextend.i64.i32, *r.umr.rex(0x89)) -I64.enc(base.uextend.i64.i32, *r.umr(0x89)) +X86_64.enc(base.uextend.i64.i32, *r.umr.rex(0x89)) +X86_64.enc(base.uextend.i64.i32, *r.umr(0x89)) # @@ -469,8 +469,8 @@ enc_both(base.bitcast.f32.i32, r.frurm, 0x66, 0x0f, 0x6e) enc_both(base.bitcast.i32.f32, r.rfumr, 0x66, 0x0f, 0x7e) # movq -I64.enc(base.bitcast.f64.i64, *r.frurm.rex(0x66, 0x0f, 0x6e, w=1)) -I64.enc(base.bitcast.i64.f64, *r.rfumr.rex(0x66, 0x0f, 0x7e, w=1)) +X86_64.enc(base.bitcast.f64.i64, *r.frurm.rex(0x66, 0x0f, 0x6e, w=1)) +X86_64.enc(base.bitcast.i64.f64, *r.rfumr.rex(0x66, 0x0f, 0x7e, w=1)) # movaps enc_both(base.copy.f32, r.furm, 0x0f, 0x28) @@ -492,11 +492,11 @@ enc_both(base.fdemote.f32.f64, r.furm, 0xf2, 0x0f, 0x5a) # cvttss2si enc_both(x86.cvtt2si.i32.f32, r.rfurm, 0xf3, 0x0f, 0x2c) -I64.enc(x86.cvtt2si.i64.f32, *r.rfurm.rex(0xf3, 0x0f, 0x2c, w=1)) +X86_64.enc(x86.cvtt2si.i64.f32, *r.rfurm.rex(0xf3, 0x0f, 0x2c, w=1)) # cvttsd2si enc_both(x86.cvtt2si.i32.f64, r.rfurm, 0xf2, 0x0f, 0x2c) -I64.enc(x86.cvtt2si.i64.f64, *r.rfurm.rex(0xf2, 0x0f, 0x2c, w=1)) +X86_64.enc(x86.cvtt2si.i64.f64, *r.rfurm.rex(0xf2, 0x0f, 0x2c, w=1)) # Exact square roots. enc_both(base.sqrt.f32, r.furm, 0xf3, 0x0f, 0x51) From c161b0d1030e583c96b494890c51111c86babbf2 Mon Sep 17 00:00:00 2001 From: Sergey Pepyakin Date: Sun, 18 Mar 2018 23:53:02 +0300 Subject: [PATCH 1627/3084] Fix check-rustfmt.sh for macOS (#273) There are two cases: 1. It seems that grep on macOS exits as soon as it finds the first match. This makes cargo unhappy and it prints message like "failed printing to stdout: Broken pipe (os error 32)". The solution is to fully consume the output from cargo. I choose to use tee for this task. 2. When in a strict mode, bash complains that $1 is not defined (when it's actually not defined in case of omitting --install). The solution is to apply bash substitution magic: when $1 is undefined or set to null substitute it with empty string. --- check-rustfmt.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/check-rustfmt.sh b/check-rustfmt.sh index 1983493342..9a49e8bac5 100755 --- a/check-rustfmt.sh +++ b/check-rustfmt.sh @@ -22,11 +22,11 @@ set -euo pipefail # operation, however that doesn't appear to be possible through "cargo fmt"). VERS="0.9.0" -if cargo install --list | grep -q "^rustfmt v$VERS"; then +if cargo install --list | tee /dev/null | grep -q "^rustfmt v$VERS"; then exit 0 fi -if [ "$1" != "--install" ]; then +if [[ ${1:-""} != "--install" ]]; then echo "********************************************************************" echo "* Please install rustfmt v$VERS to verify formatting. *" echo "* If a newer version of rustfmt is available, update this script. *" From 832d9d9a0df56573e1ed53d73ea8c7d67aa34fb1 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sun, 18 Mar 2018 13:45:41 -0700 Subject: [PATCH 1628/3084] Minor code simplification. --- lib/wasm/src/code_translator.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 5535a1555e..aab015c536 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -186,8 +186,8 @@ pub fn translate_operator( } Operator::End => { let frame = state.control_stack.pop().unwrap(); - let return_count = frame.num_return_values(); if !builder.is_unreachable() || !builder.is_pristine() { + let return_count = frame.num_return_values(); builder.ins().jump( frame.following_code(), state.peekn(return_count), From 492fa1283c4d5b10901623a2810d28d507a2f046 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sun, 18 Mar 2018 13:46:07 -0700 Subject: [PATCH 1629/3084] Define an "interrupt" trap code. This is a trap code for interrupting the running code, to allow timeouts and safepoints to be implemented. It is resumable. --- lib/cretonne/src/ir/trapcode.rs | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/cretonne/src/ir/trapcode.rs b/lib/cretonne/src/ir/trapcode.rs index 2374c206f3..68b836f309 100644 --- a/lib/cretonne/src/ir/trapcode.rs +++ b/lib/cretonne/src/ir/trapcode.rs @@ -38,6 +38,10 @@ pub enum TrapCode { /// Failed float-to-int conversion. BadConversionToInteger, + /// Execution has potentially run too long and may be interrupted. + /// This trap is resumable. + Interrupt, + /// A user-defined trap code. User(u16), } @@ -54,6 +58,7 @@ impl Display for TrapCode { IntegerOverflow => "int_ovf", IntegerDivisionByZero => "int_divz", BadConversionToInteger => "bad_toint", + Interrupt => "interrupt", User(x) => return write!(f, "user{}", x), }; f.write_str(identifier) @@ -74,6 +79,7 @@ impl FromStr for TrapCode { "int_ovf" => Ok(IntegerOverflow), "int_divz" => Ok(IntegerDivisionByZero), "bad_toint" => Ok(BadConversionToInteger), + "interrupt" => Ok(Interrupt), _ if s.starts_with("user") => s[4..].parse().map(User).map_err(|_| ()), _ => Err(()), } From d99b43e4b7aae1fde986c83055de99356981c363 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sun, 18 Mar 2018 13:47:17 -0700 Subject: [PATCH 1630/3084] Add a hook to the wasm FuncEnvironment for emitting loop headers. This will allow wasm implementations that wish to insert code into every loop, for example to insert an interrupt check or a safepoint. do so without relying on asynchronous signals. --- lib/wasm/src/code_translator.rs | 1 + lib/wasm/src/environ/spec.rs | 8 ++++++++ 2 files changed, 9 insertions(+) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index aab015c536..bb675cb1c7 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -137,6 +137,7 @@ pub fn translate_operator( builder.ins().jump(loop_body, &[]); state.push_loop(loop_body, next, num_return_values(ty)); builder.switch_to_block(loop_body); + environ.translate_loop_header(builder.cursor()); } Operator::If { ty } => { let val = state.pop1(); diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 732438a573..7f73c3fc16 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -144,6 +144,14 @@ pub trait FuncEnvironment { index: MemoryIndex, heap: ir::Heap, ) -> ir::Value; + + /// Emit code at the beginning of every wasm loop. + /// + /// This can be used to insert explicit interrupt or safepoint checking at + /// the beginnings of loops. + fn translate_loop_header(&mut self, _pos: FuncCursor) { + // By default, don't emit anything. + } } /// An object satisfying the `ModuleEnvironment` trait can be passed as argument to the From 2d1f9f874c23eab253d4048da2a2317b49df3150 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sun, 18 Mar 2018 14:56:30 -0700 Subject: [PATCH 1631/3084] Bump version to 0.4.1 --- cranelift/Cargo.toml | 14 +++++++------- cranelift/publish-all.sh | 2 +- lib/cretonne/Cargo.toml | 2 +- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/wasm/Cargo.toml | 6 +++--- 8 files changed, 21 insertions(+), 21 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index a6d363dcdb..947b50b44a 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.4.0" +version = "0.4.1" description = "Binaries for testing the Cretonne library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,12 +13,12 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne = { path = "lib/cretonne", version = "0.4.0" } -cretonne-reader = { path = "lib/reader", version = "0.4.0" } -cretonne-frontend = { path = "lib/frontend", version = "0.4.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.4.0" } -cretonne-native = { path = "lib/native", version = "0.4.0" } -cretonne-filetests = { path = "lib/filetests", version = "0.4.0" } +cretonne = { path = "lib/cretonne", version = "0.4.1" } +cretonne-reader = { path = "lib/reader", version = "0.4.1" } +cretonne-frontend = { path = "lib/frontend", version = "0.4.1" } +cretonne-wasm = { path = "lib/wasm", version = "0.4.1" } +cretonne-native = { path = "lib/native", version = "0.4.1" } +cretonne-filetests = { path = "lib/filetests", version = "0.4.1" } filecheck = "0.2.1" docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 6b23f71064..a29f308929 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ cd $(dirname "$0") topdir="$(pwd)" # All the cretonne-* crates have the same version number -version="0.4.0" +version="0.4.1" # Update all of the Cargo.toml files. # diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index d603a9597e..66cd819715 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.4.0" +version = "0.4.1" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index f3ecf521c0..093b51b323 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.4.0" +version = "0.4.1" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "http://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -12,7 +12,7 @@ publish = false name = "cton_filetests" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.0" } -cretonne-reader = { path = "../reader", version = "0.4.0" } +cretonne = { path = "../cretonne", version = "0.4.1" } +cretonne-reader = { path = "../reader", version = "0.4.1" } filecheck = "0.2.1" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index bd5455d9d3..40f1833c71 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.4.0" +version = "0.4.1" description = "Cretonne IL builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,7 +12,7 @@ readme = "README.md" name = "cton_frontend" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.0" } +cretonne = { path = "../cretonne", version = "0.4.1" } [badges] maintenance = { status = "experimental" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 9a2ba0eea7..31dbdb0708 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.4.0" +version = "0.4.1" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -11,7 +11,7 @@ readme = "README.md" name = "cton_native" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.0" } +cretonne = { path = "../cretonne", version = "0.4.1" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 1841357bc4..80fab0f499 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.4.0" +version = "0.4.1" description = "Cretonne textual IL reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,7 +12,7 @@ readme = "README.md" name = "cton_reader" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.0" } +cretonne = { path = "../cretonne", version = "0.4.1" } [badges] maintenance = { status = "experimental" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index df978cb4d0..0fcd2210f5 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.4.0" +version = "0.4.1" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IL" repository = "https://github.com/Cretonne/cretonne" @@ -13,8 +13,8 @@ name = "cton_wasm" [dependencies] wasmparser = "0.15.1" -cretonne = { path = "../cretonne", version = "0.4.0" } -cretonne-frontend = { path = "../frontend", version = "0.4.0" } +cretonne = { path = "../cretonne", version = "0.4.1" } +cretonne-frontend = { path = "../frontend", version = "0.4.1" } [dev-dependencies] tempdir = "0.3.5" From c333a52e3e21dbbf1b6b87ef4991e9d8a309d561 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 13 Mar 2018 05:56:59 -0700 Subject: [PATCH 1632/3084] Factor out fcmp and icmp translation into helper functions. --- lib/wasm/src/code_translator.rs | 108 +++++++++++--------------------- 1 file changed, 36 insertions(+), 72 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index bb675cb1c7..10c0f7fdd2 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -765,101 +765,45 @@ pub fn translate_operator( } /**************************** Comparison Operators **********************************/ Operator::I32LtS | Operator::I64LtS => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().icmp(IntCC::SignedLessThan, arg1, arg2); - state.push1(builder.ins().bint(I32, val)); + translate_icmp(IntCC::SignedLessThan, builder, state) } Operator::I32LtU | Operator::I64LtU => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().icmp(IntCC::UnsignedLessThan, arg1, arg2); - state.push1(builder.ins().bint(I32, val)); + translate_icmp(IntCC::UnsignedLessThan, builder, state) } Operator::I32LeS | Operator::I64LeS => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().icmp(IntCC::SignedLessThanOrEqual, arg1, arg2); - state.push1(builder.ins().bint(I32, val)); + translate_icmp(IntCC::SignedLessThanOrEqual, builder, state) } Operator::I32LeU | Operator::I64LeU => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().icmp( - IntCC::UnsignedLessThanOrEqual, - arg1, - arg2, - ); - state.push1(builder.ins().bint(I32, val)); + translate_icmp(IntCC::UnsignedLessThanOrEqual, builder, state) } Operator::I32GtS | Operator::I64GtS => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().icmp(IntCC::SignedGreaterThan, arg1, arg2); - state.push1(builder.ins().bint(I32, val)); + translate_icmp(IntCC::SignedGreaterThan, builder, state) } Operator::I32GtU | Operator::I64GtU => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().icmp(IntCC::UnsignedGreaterThan, arg1, arg2); - state.push1(builder.ins().bint(I32, val)); + translate_icmp(IntCC::UnsignedGreaterThan, builder, state) } Operator::I32GeS | Operator::I64GeS => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().icmp( - IntCC::SignedGreaterThanOrEqual, - arg1, - arg2, - ); - state.push1(builder.ins().bint(I32, val)); + translate_icmp(IntCC::SignedGreaterThanOrEqual, builder, state) } Operator::I32GeU | Operator::I64GeU => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().icmp( - IntCC::UnsignedGreaterThanOrEqual, - arg1, - arg2, - ); - state.push1(builder.ins().bint(I32, val)); + translate_icmp(IntCC::UnsignedGreaterThanOrEqual, builder, state) } Operator::I32Eqz | Operator::I64Eqz => { let arg = state.pop1(); let val = builder.ins().icmp_imm(IntCC::Equal, arg, 0); state.push1(builder.ins().bint(I32, val)); } - Operator::I32Eq | Operator::I64Eq => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().icmp(IntCC::Equal, arg1, arg2); - state.push1(builder.ins().bint(I32, val)); - } - Operator::F32Eq | Operator::F64Eq => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().fcmp(FloatCC::Equal, arg1, arg2); - state.push1(builder.ins().bint(I32, val)); - } - Operator::I32Ne | Operator::I64Ne => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().icmp(IntCC::NotEqual, arg1, arg2); - state.push1(builder.ins().bint(I32, val)); - } - Operator::F32Ne | Operator::F64Ne => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().fcmp(FloatCC::NotEqual, arg1, arg2); - state.push1(builder.ins().bint(I32, val)); - } - Operator::F32Gt | Operator::F64Gt => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().fcmp(FloatCC::GreaterThan, arg1, arg2); - state.push1(builder.ins().bint(I32, val)); - } + Operator::I32Eq | Operator::I64Eq => translate_icmp(IntCC::Equal, builder, state), + Operator::F32Eq | Operator::F64Eq => translate_fcmp(FloatCC::Equal, builder, state), + Operator::I32Ne | Operator::I64Ne => translate_icmp(IntCC::NotEqual, builder, state), + Operator::F32Ne | Operator::F64Ne => translate_fcmp(FloatCC::NotEqual, builder, state), + Operator::F32Gt | Operator::F64Gt => translate_fcmp(FloatCC::GreaterThan, builder, state), Operator::F32Ge | Operator::F64Ge => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().fcmp(FloatCC::GreaterThanOrEqual, arg1, arg2); - state.push1(builder.ins().bint(I32, val)); - } - Operator::F32Lt | Operator::F64Lt => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().fcmp(FloatCC::LessThan, arg1, arg2); - state.push1(builder.ins().bint(I32, val)); + translate_fcmp(FloatCC::GreaterThanOrEqual, builder, state) } + Operator::F32Lt | Operator::F64Lt => translate_fcmp(FloatCC::LessThan, builder, state), Operator::F32Le | Operator::F64Le => { - let (arg1, arg2) = state.pop2(); - let val = builder.ins().fcmp(FloatCC::LessThanOrEqual, arg1, arg2); - state.push1(builder.ins().bint(I32, val)); + translate_fcmp(FloatCC::LessThanOrEqual, builder, state) } Operator::Wake { .. } | Operator::I32Wait { .. } | @@ -1101,3 +1045,23 @@ fn translate_store( base, ); } + +fn translate_icmp( + cc: IntCC, + builder: &mut FunctionBuilder, + state: &mut TranslationState, +) { + let (arg0, arg1) = state.pop2(); + let val = builder.ins().icmp(cc, arg0, arg1); + state.push1(builder.ins().bint(I32, val)); +} + +fn translate_fcmp( + cc: FloatCC, + builder: &mut FunctionBuilder, + state: &mut TranslationState, +) { + let (arg0, arg1) = state.pop2(); + let val = builder.ins().fcmp(cc, arg0, arg1); + state.push1(builder.ins().bint(I32, val)); +} From f6b8cf86a566969544762019e96a5238f3ae89a0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 13 Mar 2018 07:04:38 -0700 Subject: [PATCH 1633/3084] Factor out br_if translation into helper functions. --- lib/wasm/src/code_translator.rs | 53 ++++++++++++++++++++------------- 1 file changed, 32 insertions(+), 21 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 10c0f7fdd2..cccbae1368 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -247,27 +247,7 @@ pub fn translate_operator( state.popn(return_count); state.reachable = false; } - Operator::BrIf { relative_depth } => { - let val = state.pop1(); - let i = state.control_stack.len() - 1 - (relative_depth as usize); - let (return_count, br_destination) = { - let frame = &mut state.control_stack[i]; - // The values returned by the branch are still available for the reachable - // code that comes after it - frame.set_branched_to_exit(); - let return_count = if frame.is_loop() { - 0 - } else { - frame.num_return_values() - }; - (return_count, frame.br_destination()) - }; - builder.ins().brnz( - val, - br_destination, - state.peekn(return_count), - ); - } + Operator::BrIf { relative_depth } => translate_br_if(relative_depth, builder, state), Operator::BrTable { table } => { let (depths, default) = table.read_table(); let mut min_depth = default; @@ -1065,3 +1045,34 @@ fn translate_fcmp( let val = builder.ins().fcmp(cc, arg0, arg1); state.push1(builder.ins().bint(I32, val)); } + +fn translate_br_if( + relative_depth: u32, + builder: &mut FunctionBuilder, + state: &mut TranslationState, +) { + let val = state.pop1(); + let (br_destination, inputs) = translate_br_if_args(relative_depth, state); + builder.ins().brnz(val, br_destination, inputs); +} + +fn translate_br_if_args<'state>( + relative_depth: u32, + state: &'state mut TranslationState, +) -> (ir::Ebb, &'state [ir::Value]) { + let i = state.control_stack.len() - 1 - (relative_depth as usize); + let (return_count, br_destination) = { + let frame = &mut state.control_stack[i]; + // The values returned by the branch are still available for the reachable + // code that comes after it + frame.set_branched_to_exit(); + let return_count = if frame.is_loop() { + 0 + } else { + frame.num_return_values() + }; + (return_count, frame.br_destination()) + }; + let inputs = state.peekn(return_count); + (br_destination, inputs) +} From ca4582ae8231b139f6bc2bdf32fe0f1d3d0d7b99 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 19 Mar 2018 09:13:47 -0700 Subject: [PATCH 1634/3084] Rename the recipes for x86 spill/fill instructions. Both "sp" and "fi" have multiple meanings in this context, so use slightly longer but less ambiguous names. --- lib/cretonne/meta/isa/intel/encodings.py | 32 +++++++++---------- lib/cretonne/meta/isa/intel/recipes.py | 40 ++++++++++++++---------- 2 files changed, 40 insertions(+), 32 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 995149794a..489e61be9f 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -205,14 +205,14 @@ for recipe in [r.st_abcd, r.stDisp8_abcd, r.stDisp32_abcd]: enc_both(base.istore8.i32.any, recipe, 0x88) enc_x86_64(base.istore8.i64.any, recipe, 0x88) -enc_i32_i64(base.spill, r.spSib32, 0x89) -enc_i32_i64(base.regspill, r.rsp32, 0x89) +enc_i32_i64(base.spill, r.spillSib32, 0x89) +enc_i32_i64(base.regspill, r.regspill32, 0x89) # Use a 32-bit write for spilling `b1` to avoid constraining the permitted # registers. # See MIN_SPILL_SLOT_SIZE which makes this safe. -enc_both(base.spill.b1, r.spSib32, 0x89) -enc_both(base.regspill.b1, r.rsp32, 0x89) +enc_both(base.spill.b1, r.spillSib32, 0x89) +enc_both(base.regspill.b1, r.regspill32, 0x89) for recipe in [r.ld, r.ldDisp8, r.ldDisp32]: enc_i32_i64_ld_st(base.load, True, recipe, 0x8b) @@ -223,12 +223,12 @@ for recipe in [r.ld, r.ldDisp8, r.ldDisp32]: enc_i32_i64_ld_st(base.uload8, True, recipe, 0x0f, 0xb6) enc_i32_i64_ld_st(base.sload8, True, recipe, 0x0f, 0xbe) -enc_i32_i64(base.fill, r.fiSib32, 0x8b) -enc_i32_i64(base.regfill, r.rfi32, 0x8b) +enc_i32_i64(base.fill, r.fillSib32, 0x8b) +enc_i32_i64(base.regfill, r.regfill32, 0x8b) # Load 32 bits from `b1` spill slots. See `spill.b1` above. -enc_both(base.fill.b1, r.fiSib32, 0x8b) -enc_both(base.regfill.b1, r.rfi32, 0x8b) +enc_both(base.fill.b1, r.fillSib32, 0x8b) +enc_both(base.regfill.b1, r.regfill32, 0x8b) # Push and Pop X86_32.enc(x86.push.i32, *r.pushq(0x50)) @@ -267,15 +267,15 @@ enc_both(base.store.f64.any, r.fst, 0x66, 0x0f, 0xd6) enc_both(base.store.f64.any, r.fstDisp8, 0x66, 0x0f, 0xd6) enc_both(base.store.f64.any, r.fstDisp32, 0x66, 0x0f, 0xd6) -enc_both(base.fill.f32, r.ffiSib32, 0x66, 0x0f, 0x6e) -enc_both(base.regfill.f32, r.frfi32, 0x66, 0x0f, 0x6e) -enc_both(base.fill.f64, r.ffiSib32, 0xf3, 0x0f, 0x7e) -enc_both(base.regfill.f64, r.frfi32, 0xf3, 0x0f, 0x7e) +enc_both(base.fill.f32, r.ffillSib32, 0x66, 0x0f, 0x6e) +enc_both(base.regfill.f32, r.fregfill32, 0x66, 0x0f, 0x6e) +enc_both(base.fill.f64, r.ffillSib32, 0xf3, 0x0f, 0x7e) +enc_both(base.regfill.f64, r.fregfill32, 0xf3, 0x0f, 0x7e) -enc_both(base.spill.f32, r.fspSib32, 0x66, 0x0f, 0x7e) -enc_both(base.regspill.f32, r.frsp32, 0x66, 0x0f, 0x7e) -enc_both(base.spill.f64, r.fspSib32, 0x66, 0x0f, 0xd6) -enc_both(base.regspill.f64, r.frsp32, 0x66, 0x0f, 0xd6) +enc_both(base.spill.f32, r.fspillSib32, 0x66, 0x0f, 0x7e) +enc_both(base.regspill.f32, r.fregspill32, 0x66, 0x0f, 0x7e) +enc_both(base.spill.f64, r.fspillSib32, 0x66, 0x0f, 0xd6) +enc_both(base.regspill.f64, r.fregspill32, 0x66, 0x0f, 0xd6) # # Function addresses. diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 9d03d02053..9d015a25bb 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -756,8 +756,8 @@ fstDisp32 = TailRecipe( ''') # Unary spill with SIB and 32-bit displacement. -spSib32 = TailRecipe( - 'spSib32', Unary, size=6, ins=GPR, outs=StackGPR32, +spillSib32 = TailRecipe( + 'spillSib32', Unary, size=6, ins=GPR, outs=StackGPR32, clobbers_flags=False, emit=''' let base = stk_base(out_stk0.base); @@ -766,8 +766,10 @@ spSib32 = TailRecipe( sib_noindex(base, sink); sink.put4(out_stk0.offset as u32); ''') -fspSib32 = TailRecipe( - 'fspSib32', Unary, size=6, ins=FPR, outs=StackFPR32, + +# Like spillSib32, but targeting an FPR rather than a GPR. +fspillSib32 = TailRecipe( + 'fspillSib32', Unary, size=6, ins=FPR, outs=StackFPR32, clobbers_flags=False, emit=''' let base = stk_base(out_stk0.base); @@ -778,8 +780,8 @@ fspSib32 = TailRecipe( ''') # Regspill using RSP-relative addressing. -rsp32 = TailRecipe( - 'rsp32', RegSpill, size=6, ins=GPR, outs=(), +regspill32 = TailRecipe( + 'regspill32', RegSpill, size=6, ins=GPR, outs=(), clobbers_flags=False, emit=''' let dst = StackRef::sp(dst, &func.stack_slots); @@ -789,8 +791,10 @@ rsp32 = TailRecipe( sib_noindex(base, sink); sink.put4(dst.offset as u32); ''') -frsp32 = TailRecipe( - 'frsp32', RegSpill, size=6, ins=FPR, outs=(), + +# Like regspill32, but targeting an FPR rather than a GPR. +fregspill32 = TailRecipe( + 'fregspill32', RegSpill, size=6, ins=FPR, outs=(), clobbers_flags=False, emit=''' let dst = StackRef::sp(dst, &func.stack_slots); @@ -874,8 +878,8 @@ fldDisp32 = TailRecipe( ''') # Unary fill with SIB and 32-bit displacement. -fiSib32 = TailRecipe( - 'fiSib32', Unary, size=6, ins=StackGPR32, outs=GPR, +fillSib32 = TailRecipe( + 'fillSib32', Unary, size=6, ins=StackGPR32, outs=GPR, clobbers_flags=False, emit=''' let base = stk_base(in_stk0.base); @@ -884,8 +888,10 @@ fiSib32 = TailRecipe( sib_noindex(base, sink); sink.put4(in_stk0.offset as u32); ''') -ffiSib32 = TailRecipe( - 'ffiSib32', Unary, size=6, ins=StackFPR32, outs=FPR, + +# Like fillSib32, but targeting an FPR rather than a GPR. +ffillSib32 = TailRecipe( + 'ffillSib32', Unary, size=6, ins=StackFPR32, outs=FPR, clobbers_flags=False, emit=''' let base = stk_base(in_stk0.base); @@ -896,8 +902,8 @@ ffiSib32 = TailRecipe( ''') # Regfill with RSP-relative 32-bit displacement. -rfi32 = TailRecipe( - 'rfi32', RegFill, size=6, ins=StackGPR32, outs=(), +regfill32 = TailRecipe( + 'regfill32', RegFill, size=6, ins=StackGPR32, outs=(), clobbers_flags=False, emit=''' let src = StackRef::sp(src, &func.stack_slots); @@ -907,8 +913,10 @@ rfi32 = TailRecipe( sib_noindex(base, sink); sink.put4(src.offset as u32); ''') -frfi32 = TailRecipe( - 'frfi32', RegFill, size=6, ins=StackFPR32, outs=(), + +# Like regfill32, but targeting an FPR rather than a GPR. +fregfill32 = TailRecipe( + 'fregfill32', RegFill, size=6, ins=StackFPR32, outs=(), clobbers_flags=False, emit=''' let src = StackRef::sp(src, &func.stack_slots); From da4cf27780468b39f0983b65d074191263c8da4e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 20 Mar 2018 12:04:17 -0700 Subject: [PATCH 1635/3084] Update to filecheck 0.3.0. --- lib/filetests/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 093b51b323..0e37c1ef8d 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -14,5 +14,5 @@ name = "cton_filetests" [dependencies] cretonne = { path = "../cretonne", version = "0.4.1" } cretonne-reader = { path = "../reader", version = "0.4.1" } -filecheck = "0.2.1" +filecheck = "0.3.0" num_cpus = "1.8.0" From 2b3df1a506399e21dcdcc72bb835e66df7be3a74 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 20 Mar 2018 12:15:59 -0700 Subject: [PATCH 1636/3084] Add `use` declarations for `std` features. Merge the `use` parts of the `no_std` branch. This reduces the diffs between master and the `no_std` branch, making it easier to maintain. Most of these changes are derived from patches by @lachlansneff in https://github.com/Cretonne/cretonne/tree/no_std. --- lib/cretonne/src/abi.rs | 1 + lib/cretonne/src/bforest/map.rs | 1 + lib/cretonne/src/bforest/node.rs | 1 + lib/cretonne/src/bforest/pool.rs | 1 + lib/cretonne/src/bforest/set.rs | 1 + lib/cretonne/src/dominator_tree.rs | 2 +- lib/cretonne/src/entity/list.rs | 1 + lib/cretonne/src/entity/map.rs | 1 + lib/cretonne/src/entity/primary.rs | 1 + lib/cretonne/src/entity/set.rs | 1 + lib/cretonne/src/entity/sparse.rs | 1 + lib/cretonne/src/flowgraph.rs | 1 + lib/cretonne/src/ir/condcodes.rs | 1 + lib/cretonne/src/ir/dfg.rs | 1 + lib/cretonne/src/ir/entities.rs | 1 + lib/cretonne/src/ir/extfunc.rs | 2 ++ lib/cretonne/src/ir/extname.rs | 1 + lib/cretonne/src/ir/immediates.rs | 1 + lib/cretonne/src/ir/instructions.rs | 2 ++ lib/cretonne/src/ir/jumptable.rs | 3 +++ lib/cretonne/src/ir/layout.rs | 1 + lib/cretonne/src/ir/libcall.rs | 1 + lib/cretonne/src/ir/progpoint.rs | 1 + lib/cretonne/src/ir/sourceloc.rs | 1 + lib/cretonne/src/ir/stackslot.rs | 2 ++ lib/cretonne/src/ir/trapcode.rs | 1 + lib/cretonne/src/ir/types.rs | 1 + lib/cretonne/src/isa/arm32/mod.rs | 1 + lib/cretonne/src/isa/arm32/registers.rs | 1 + lib/cretonne/src/isa/arm64/mod.rs | 1 + lib/cretonne/src/isa/arm64/registers.rs | 1 + lib/cretonne/src/isa/intel/mod.rs | 2 +- lib/cretonne/src/isa/intel/registers.rs | 1 + lib/cretonne/src/isa/mod.rs | 1 + lib/cretonne/src/isa/riscv/mod.rs | 2 ++ lib/cretonne/src/isa/riscv/registers.rs | 1 + lib/cretonne/src/isa/riscv/settings.rs | 1 + lib/cretonne/src/iterators.rs | 2 ++ lib/cretonne/src/legalizer/boundary.rs | 1 + lib/cretonne/src/legalizer/split.rs | 1 + lib/cretonne/src/licm.rs | 1 + lib/cretonne/src/loop_analysis.rs | 2 ++ lib/cretonne/src/partition_slice.rs | 1 + lib/cretonne/src/regalloc/allocatable_set.rs | 1 + lib/cretonne/src/regalloc/coalescing.rs | 1 + lib/cretonne/src/regalloc/diversion.rs | 1 + lib/cretonne/src/regalloc/live_value_tracker.rs | 1 + lib/cretonne/src/regalloc/liveness.rs | 1 + lib/cretonne/src/regalloc/liverange.rs | 1 + lib/cretonne/src/regalloc/pressure.rs | 1 + lib/cretonne/src/regalloc/reload.rs | 1 + lib/cretonne/src/regalloc/solver.rs | 2 ++ lib/cretonne/src/regalloc/spilling.rs | 1 + lib/cretonne/src/regalloc/virtregs.rs | 1 + lib/cretonne/src/settings.rs | 2 ++ lib/cretonne/src/simple_gvn.rs | 1 + lib/cretonne/src/topo_order.rs | 1 + lib/cretonne/src/verifier/mod.rs | 2 ++ lib/cretonne/src/write.rs | 2 ++ lib/frontend/src/ssa.rs | 1 + lib/wasm/src/code_translator.rs | 1 + lib/wasm/src/environ/dummy.rs | 2 ++ lib/wasm/src/environ/spec.rs | 2 ++ lib/wasm/src/module_translator.rs | 2 ++ lib/wasm/src/sections_translator.rs | 2 ++ lib/wasm/src/state.rs | 1 + 66 files changed, 82 insertions(+), 2 deletions(-) diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index 801d4bfe9e..b69821e657 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -5,6 +5,7 @@ use ir::{ArgumentLoc, AbiParam, ArgumentExtension, Type}; use std::cmp::Ordering; +use std::vec::Vec; /// Legalization action to perform on a single argument or return value when converting a /// signature. diff --git a/lib/cretonne/src/bforest/map.rs b/lib/cretonne/src/bforest/map.rs index 5aa19bceda..edf301aa4c 100644 --- a/lib/cretonne/src/bforest/map.rs +++ b/lib/cretonne/src/bforest/map.rs @@ -423,6 +423,7 @@ where #[cfg(test)] mod test { use std::mem; + use std::vec::Vec; use super::*; use super::super::NodeData; diff --git a/lib/cretonne/src/bforest/node.rs b/lib/cretonne/src/bforest/node.rs index 3c6b8c8774..d5c46f54bd 100644 --- a/lib/cretonne/src/bforest/node.rs +++ b/lib/cretonne/src/bforest/node.rs @@ -580,6 +580,7 @@ where #[cfg(test)] mod test { use std::mem; + use std::string::ToString; use super::*; // Forest impl for a set implementation. diff --git a/lib/cretonne/src/bforest/pool.rs b/lib/cretonne/src/bforest/pool.rs index 1ce9e83279..481271c8db 100644 --- a/lib/cretonne/src/bforest/pool.rs +++ b/lib/cretonne/src/bforest/pool.rs @@ -78,6 +78,7 @@ impl NodePool { { use std::borrow::Borrow; use std::cmp::Ordering; + use std::vec::Vec; use super::Comparator; use entity::SparseSet; diff --git a/lib/cretonne/src/bforest/set.rs b/lib/cretonne/src/bforest/set.rs index 6992265773..dd7f63ca17 100644 --- a/lib/cretonne/src/bforest/set.rs +++ b/lib/cretonne/src/bforest/set.rs @@ -351,6 +351,7 @@ where #[cfg(test)] mod test { use std::mem; + use std::vec::Vec; use super::*; use super::super::NodeData; diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index e643a42592..823469d081 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -8,8 +8,8 @@ use packed_option::PackedOption; use std::cmp; use std::mem; use timing; - use std::cmp::Ordering; +use std::vec::Vec; // RPO numbers are not first assigned in a contiguous way but as multiples of STRIDE, to leave // room for modifications of the dominator tree. diff --git a/lib/cretonne/src/entity/list.rs b/lib/cretonne/src/entity/list.rs index 6c963bc377..ad03dc6c85 100644 --- a/lib/cretonne/src/entity/list.rs +++ b/lib/cretonne/src/entity/list.rs @@ -3,6 +3,7 @@ use entity::EntityRef; use std::hash::{Hash, Hasher}; use std::marker::PhantomData; use std::mem; +use std::vec::Vec; /// A small list of entity references allocated from a pool. /// diff --git a/lib/cretonne/src/entity/map.rs b/lib/cretonne/src/entity/map.rs index 9622f93485..f251c2f831 100644 --- a/lib/cretonne/src/entity/map.rs +++ b/lib/cretonne/src/entity/map.rs @@ -3,6 +3,7 @@ use entity::{EntityRef, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; +use std::vec::Vec; /// A mapping `K -> V` for densely indexed entity references. /// diff --git a/lib/cretonne/src/entity/primary.rs b/lib/cretonne/src/entity/primary.rs index 137320f3e5..c06b818355 100644 --- a/lib/cretonne/src/entity/primary.rs +++ b/lib/cretonne/src/entity/primary.rs @@ -2,6 +2,7 @@ use entity::{EntityRef, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; +use std::vec::Vec; /// A primary mapping `K -> V` allocating dense entity references. /// diff --git a/lib/cretonne/src/entity/set.rs b/lib/cretonne/src/entity/set.rs index 82dc1384ee..e4acf47723 100644 --- a/lib/cretonne/src/entity/set.rs +++ b/lib/cretonne/src/entity/set.rs @@ -2,6 +2,7 @@ use entity::{EntityRef, Keys}; use std::marker::PhantomData; +use std::vec::Vec; /// A set of `K` for densely indexed entity references. /// diff --git a/lib/cretonne/src/entity/sparse.rs b/lib/cretonne/src/entity/sparse.rs index 82e1b49546..488fd55393 100644 --- a/lib/cretonne/src/entity/sparse.rs +++ b/lib/cretonne/src/entity/sparse.rs @@ -11,6 +11,7 @@ use entity::{EntityRef, EntityMap}; use std::mem; use std::slice; use std::u32; +use std::vec::Vec; /// Trait for extracting keys from values stored in a `SparseMap`. /// diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index fbc74b79d0..dce60162a4 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -204,6 +204,7 @@ mod tests { use super::*; use cursor::{Cursor, FuncCursor}; use ir::{Function, InstBuilder, types}; + use std::vec::Vec; #[test] fn empty() { diff --git a/lib/cretonne/src/ir/condcodes.rs b/lib/cretonne/src/ir/condcodes.rs index 4bbf82cb6f..11c438b10a 100644 --- a/lib/cretonne/src/ir/condcodes.rs +++ b/lib/cretonne/src/ir/condcodes.rs @@ -266,6 +266,7 @@ impl FromStr for FloatCC { #[cfg(test)] mod tests { use super::*; + use std::string::ToString; static INT_ALL: [IntCC; 10] = [ IntCC::Equal, diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index ccd08f1bb5..344483a6d9 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -944,6 +944,7 @@ mod tests { use cursor::{Cursor, FuncCursor}; use ir::types; use ir::{Function, Opcode, InstructionData, TrapCode}; + use std::string::ToString; #[test] fn make_inst() { diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index a9e6705d98..1a4e772be4 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -262,6 +262,7 @@ impl From for AnyEntity { mod tests { use super::*; use std::u32; + use std::string::ToString; #[test] fn value_with_number() { diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index b3c3a274e7..f8fe767970 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -10,6 +10,7 @@ use isa::{RegInfo, RegUnit}; use std::cmp; use std::fmt; use std::str::FromStr; +use std::vec::Vec; /// Function signature. /// @@ -378,6 +379,7 @@ impl FromStr for CallConv { mod tests { use super::*; use ir::types::{I32, F32, B8}; + use std::string::ToString; #[test] fn argument_type() { diff --git a/lib/cretonne/src/ir/extname.rs b/lib/cretonne/src/ir/extname.rs index f58f8d24ec..b7bcc7d94f 100644 --- a/lib/cretonne/src/ir/extname.rs +++ b/lib/cretonne/src/ir/extname.rs @@ -122,6 +122,7 @@ impl FromStr for ExternalName { mod tests { use super::ExternalName; use ir::LibCall; + use std::string::ToString; #[test] fn display_testcase() { diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 69c9aab51a..2e6339cc01 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -652,6 +652,7 @@ mod tests { use std::{f32, f64}; use std::str::FromStr; use std::fmt::Display; + use std::string::ToString; #[test] fn format_imm64() { diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 358c63d942..68ee656e85 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -9,6 +9,7 @@ use std::fmt::{self, Display, Formatter}; use std::str::FromStr; use std::ops::{Deref, DerefMut}; +use std::vec::Vec; use ir; use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef}; @@ -537,6 +538,7 @@ pub enum ResolvedConstraint { #[cfg(test)] mod tests { use super::*; + use std::string::ToString; #[test] fn opcodes() { diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index 2d09aab4da..1c7bf79157 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -8,6 +8,7 @@ use ir::entities::Ebb; use std::iter; use std::slice; use std::fmt::{self, Display, Formatter}; +use std::vec::Vec; /// Contents of a jump table. /// @@ -141,6 +142,8 @@ mod tests { use super::JumpTableData; use ir::Ebb; use entity::EntityRef; + use std::vec::Vec; + use std::string::ToString; #[test] fn empty() { diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index de117b3ed8..0e6e010e7c 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -743,6 +743,7 @@ mod tests { use entity::EntityRef; use ir::{Ebb, Inst, ProgramOrder, SourceLoc}; use std::cmp::Ordering; + use std::vec::Vec; struct LayoutCursor<'f> { /// Borrowed function layout. Public so it can be re-borrowed from this cursor. diff --git a/lib/cretonne/src/ir/libcall.rs b/lib/cretonne/src/ir/libcall.rs index e8c848f71d..85133611be 100644 --- a/lib/cretonne/src/ir/libcall.rs +++ b/lib/cretonne/src/ir/libcall.rs @@ -100,6 +100,7 @@ impl LibCall { #[cfg(test)] mod test { use super::*; + use std::string::ToString; #[test] fn display() { diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index b7b2994c3b..ce5b108e54 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -148,6 +148,7 @@ mod tests { use super::*; use entity::EntityRef; use ir::{Inst, Ebb}; + use std::string::ToString; #[test] fn convert() { diff --git a/lib/cretonne/src/ir/sourceloc.rs b/lib/cretonne/src/ir/sourceloc.rs index ffcf0db943..36e5247488 100644 --- a/lib/cretonne/src/ir/sourceloc.rs +++ b/lib/cretonne/src/ir/sourceloc.rs @@ -51,6 +51,7 @@ impl fmt::Display for SourceLoc { #[cfg(test)] mod tests { use ir::SourceLoc; + use std::string::ToString; #[test] fn display() { diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index ebae8e2391..5f73fb4b37 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -10,6 +10,7 @@ use std::cmp; use std::fmt; use std::ops::{Index, IndexMut}; use std::str::FromStr; +use std::vec::Vec; /// The size of an object on the stack, or the size of a stack frame. /// @@ -319,6 +320,7 @@ mod tests { use ir::Function; use ir::types; use super::*; + use std::string::ToString; #[test] fn stack_slot() { diff --git a/lib/cretonne/src/ir/trapcode.rs b/lib/cretonne/src/ir/trapcode.rs index 68b836f309..fe712b7ffa 100644 --- a/lib/cretonne/src/ir/trapcode.rs +++ b/lib/cretonne/src/ir/trapcode.rs @@ -89,6 +89,7 @@ impl FromStr for TrapCode { #[cfg(test)] mod tests { use super::*; + use std::string::ToString; // Everything but user-defined codes. const CODES: [TrapCode; 8] = [ diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index f8774250ec..07cf97aefa 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -324,6 +324,7 @@ impl Default for Type { #[cfg(test)] mod tests { use super::*; + use std::string::ToString; #[test] fn basic_scalars() { diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index c3f295d4fb..02e0d5f1af 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -14,6 +14,7 @@ use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; use std::fmt; +use std::boxed::Box; #[allow(dead_code)] struct Isa { diff --git a/lib/cretonne/src/isa/arm32/registers.rs b/lib/cretonne/src/isa/arm32/registers.rs index 7c6ac406e1..e2c4813bdd 100644 --- a/lib/cretonne/src/isa/arm32/registers.rs +++ b/lib/cretonne/src/isa/arm32/registers.rs @@ -8,6 +8,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-arm32.rs")); mod tests { use super::{INFO, GPR, S, D}; use isa::RegUnit; + use std::string::{String, ToString}; #[test] fn unit_encodings() { diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index d0644ae2f6..8239cb01e9 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -14,6 +14,7 @@ use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; use std::fmt; +use std::boxed::Box; #[allow(dead_code)] struct Isa { diff --git a/lib/cretonne/src/isa/arm64/registers.rs b/lib/cretonne/src/isa/arm64/registers.rs index 62311aaebe..2c1d85e091 100644 --- a/lib/cretonne/src/isa/arm64/registers.rs +++ b/lib/cretonne/src/isa/arm64/registers.rs @@ -8,6 +8,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-arm64.rs")); mod tests { use super::INFO; use isa::RegUnit; + use std::string::{String, ToString}; #[test] fn unit_encodings() { diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 3f02890a82..578dac2d16 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -16,7 +16,7 @@ use regalloc; use result; use timing; use std::fmt; - +use std::boxed::Box; #[allow(dead_code)] struct Isa { diff --git a/lib/cretonne/src/isa/intel/registers.rs b/lib/cretonne/src/isa/intel/registers.rs index c2b96f77d3..c972c10a13 100644 --- a/lib/cretonne/src/isa/intel/registers.rs +++ b/lib/cretonne/src/isa/intel/registers.rs @@ -8,6 +8,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-intel.rs")); mod tests { use super::*; use isa::RegUnit; + use std::string::{String, ToString}; #[test] fn unit_encodings() { diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 28b297860c..aefe93a05d 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -54,6 +54,7 @@ use result; use timing; use isa::enc_tables::Encodings; use std::fmt; +use std::boxed::Box; #[cfg(build_riscv)] mod riscv; diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 2fd0d5cdb9..6fc976686b 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -14,6 +14,7 @@ use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; use ir; use regalloc; use std::fmt; +use std::boxed::Box; #[allow(dead_code)] struct Isa { @@ -116,6 +117,7 @@ mod tests { use isa; use ir::{DataFlowGraph, InstructionData, Opcode}; use ir::{types, immediates}; + use std::string::{String, ToString}; fn encstr(isa: &isa::TargetIsa, enc: Result) -> String { match enc { diff --git a/lib/cretonne/src/isa/riscv/registers.rs b/lib/cretonne/src/isa/riscv/registers.rs index 3cce8c4988..e2073899b6 100644 --- a/lib/cretonne/src/isa/riscv/registers.rs +++ b/lib/cretonne/src/isa/riscv/registers.rs @@ -8,6 +8,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-riscv.rs")); mod tests { use super::{INFO, GPR, FPR}; use isa::RegUnit; + use std::string::{String, ToString}; #[test] fn unit_encodings() { diff --git a/lib/cretonne/src/isa/riscv/settings.rs b/lib/cretonne/src/isa/riscv/settings.rs index 2aa78fa853..2f0f6822a9 100644 --- a/lib/cretonne/src/isa/riscv/settings.rs +++ b/lib/cretonne/src/isa/riscv/settings.rs @@ -12,6 +12,7 @@ include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs")); mod tests { use super::{builder, Flags}; use settings::{self, Configurable}; + use std::string::ToString; #[test] fn display_default() { diff --git a/lib/cretonne/src/iterators.rs b/lib/cretonne/src/iterators.rs index 08866717e8..0524343028 100644 --- a/lib/cretonne/src/iterators.rs +++ b/lib/cretonne/src/iterators.rs @@ -48,6 +48,8 @@ where #[cfg(test)] mod tests { + use std::vec::Vec; + #[test] fn adjpairs() { use super::IteratorExtras; diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 29af50b3ad..0c5479650c 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -25,6 +25,7 @@ use ir::{Function, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, Signature use ir::instructions::CallInfo; use isa::TargetIsa; use legalizer::split::{isplit, vsplit}; +use std::vec::Vec; /// Legalize all the function signatures in `func`. /// diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index aafc050c26..cdc60974e0 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -68,6 +68,7 @@ use cursor::{Cursor, CursorPosition, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::{self, Ebb, Inst, Value, Type, Opcode, ValueDef, InstructionData, InstBuilder}; use std::iter; +use std::vec::Vec; /// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values /// if possible. diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 1eb035d9bf..e767fc838b 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -8,6 +8,7 @@ use dominator_tree::DominatorTree; use entity::{EntityList, ListPool}; use loop_analysis::{Loop, LoopAnalysis}; use timing; +use std::vec::Vec; /// Performs the LICM pass by detecting loops within the CFG and moving /// loop-invariant instructions out of them. diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index f6dee54d71..431496b6e4 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -8,6 +8,7 @@ use flowgraph::ControlFlowGraph; use ir::{Function, Ebb, Layout}; use packed_option::PackedOption; use timing; +use std::vec::Vec; /// A opaque reference to a code loop. #[derive(Copy, Clone, PartialEq, Eq, Hash)] @@ -230,6 +231,7 @@ mod test { use loop_analysis::{Loop, LoopAnalysis}; use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; + use std::vec::Vec; #[test] fn nested_loops_detection() { diff --git a/lib/cretonne/src/partition_slice.rs b/lib/cretonne/src/partition_slice.rs index 6f106e5bfc..7a94a9fe0d 100644 --- a/lib/cretonne/src/partition_slice.rs +++ b/lib/cretonne/src/partition_slice.rs @@ -33,6 +33,7 @@ where #[cfg(test)] mod tests { use super::partition_slice; + use std::vec::Vec; fn check(x: &[u32], want: &[u32]) { assert_eq!(x.len(), want.len()); diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index cc72066b5d..e0abf76d37 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -221,6 +221,7 @@ impl fmt::Display for AllocatableSet { mod tests { use super::*; use isa::registers::{RegClass, RegClassData}; + use std::vec::Vec; // Register classes for testing. const GPR: RegClass = &RegClassData { diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 3b9d6d03bb..2437fd1a03 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -18,6 +18,7 @@ use std::cmp; use std::iter; use std::fmt; use std::slice; +use std::vec::Vec; use isa::{TargetIsa, EncInfo}; use timing; diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index 588ad040a5..16eb0e9b50 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -11,6 +11,7 @@ use ir::{Value, ValueLoc, ValueLocations, StackSlot}; use ir::{InstructionData, Opcode}; use isa::{RegUnit, RegInfo}; use std::fmt; +use std::vec::Vec; /// A diversion of a value from its original location to a new register or stack location. /// diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index f53b5ec8bd..bedd0480e5 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -13,6 +13,7 @@ use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; use regalloc::liverange::LiveRange; use std::collections::HashMap; +use std::vec::Vec; type ValueList = EntityList; diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 11e7d8cc3d..9a76cb5899 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -184,6 +184,7 @@ use regalloc::affinity::Affinity; use regalloc::liverange::{LiveRange, LiveRangeForest, LiveRangeContext}; use std::mem; use std::ops::Index; +use std::vec::Vec; use timing; /// A set of live ranges, indexed by value number. diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 9462d32ad0..dbde52f05d 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -463,6 +463,7 @@ mod tests { use entity::EntityRef; use ir::{ProgramOrder, ExpandedProgramPoint}; use std::cmp::Ordering; + use std::vec::Vec; // Dummy program order which simply compares indexes. // It is assumed that EBBs have indexes that are multiples of 10, and instructions have indexes diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index c71037efdd..67e0e99b70 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -273,6 +273,7 @@ mod tests { use regalloc::AllocatableSet; use std::borrow::Borrow; use super::Pressure; + use std::boxed::Box; // Make an arm32 `TargetIsa`, if possible. fn arm32() -> Option> { diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 2967081807..63e7354453 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -21,6 +21,7 @@ use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; use timing; use topo_order::TopoOrder; +use std::vec::Vec; /// Reusable data structures for the reload pass. pub struct Reload { diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 0e68099a67..7824444eec 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -108,6 +108,7 @@ use std::fmt; use std::mem; use super::AllocatableSet; use std::u16; +use std::vec::Vec; /// A variable in the constraint problem. /// @@ -1162,6 +1163,7 @@ mod tests { use isa::{TargetIsa, RegClass, RegUnit, RegInfo}; use regalloc::AllocatableSet; use super::{Solver, Move}; + use std::boxed::Box; // Make an arm32 `TargetIsa`, if possible. fn arm32() -> Option> { diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 15739cd260..3c3f805cbf 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -26,6 +26,7 @@ use regalloc::liveness::Liveness; use regalloc::pressure::Pressure; use regalloc::virtregs::VirtRegs; use std::fmt; +use std::vec::Vec; use timing; use topo_order::TopoOrder; diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index fca213a62b..097a66cccd 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -21,6 +21,7 @@ use packed_option::PackedOption; use ref_slice::ref_slice; use std::cmp::Ordering; use std::fmt; +use std::vec::Vec; /// A virtual register reference. #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 2cfce65957..20ba99e0d9 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -24,6 +24,7 @@ use constant_hash::{probe, simple_hash}; use isa::TargetIsa; use std::fmt; use std::result; +use std::vec::Vec; /// A string-based configurator for settings groups. /// @@ -347,6 +348,7 @@ mod tests { use super::{builder, Flags}; use super::Error::*; use super::Configurable; + use std::string::ToString; #[test] fn display_default() { diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index 8642122593..dd0fb3b1aa 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -6,6 +6,7 @@ use dominator_tree::DominatorTree; use ir::{InstructionData, Function, Inst, Opcode, Type}; use scoped_hash_map::ScopedHashMap; use timing; +use std::vec::Vec; /// Test whether the given opcode is unsafe to even consider for GVN. fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { diff --git a/lib/cretonne/src/topo_order.rs b/lib/cretonne/src/topo_order.rs index cd783c928a..373ab69e0d 100644 --- a/lib/cretonne/src/topo_order.rs +++ b/lib/cretonne/src/topo_order.rs @@ -3,6 +3,7 @@ use entity::SparseSet; use dominator_tree::DominatorTree; use ir::{Ebb, Layout}; +use std::vec::Vec; /// Present EBBs in a topological order such that all dominating EBBs are guaranteed to be visited /// before the current EBB. diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 33fd3582cb..26eb9a53d9 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -73,6 +73,8 @@ use std::collections::BTreeSet; use std::error as std_error; use std::fmt::{self, Display, Formatter, Write}; use std::result; +use std::vec::Vec; +use std::string::String; use timing; pub use self::cssa::verify_cssa; diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 02f0fa3ab2..50acbbf553 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -9,6 +9,7 @@ use isa::{TargetIsa, RegInfo}; use std::fmt::{self, Result, Error, Write}; use std::result; use packed_option::ReservedValue; +use std::string::String; /// Write `func` to `w` as equivalent text. /// Use `isa` to emit ISA-dependent annotations. @@ -466,6 +467,7 @@ impl<'a> fmt::Display for DisplayValues<'a> { mod tests { use ir::{Function, ExternalName, StackSlotData, StackSlotKind}; use ir::types; + use std::string::ToString; #[test] fn basic() { diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 5b974799bf..bd4a8d89fd 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -15,6 +15,7 @@ use std::u32; use cretonne::ir::types::{F32, F64}; use cretonne::ir::immediates::{Ieee32, Ieee64}; use std::mem; +use std::vec::Vec; /// Structure containing the data relevant the construction of SSA for a given function. /// diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index cccbae1368..940725600a 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -34,6 +34,7 @@ use state::{TranslationState, ControlStackFrame}; use std::collections::{HashMap, hash_map}; use environ::{FuncEnvironment, GlobalValue}; use std::{i32, u32}; +use std::vec::Vec; /// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted /// a return. diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 5557d842f6..de941f3b9a 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -10,6 +10,8 @@ use cretonne::cursor::FuncCursor; use cretonne::settings; use wasmparser; use std::error::Error; +use std::vec::Vec; +use std::string::String; /// Compute a `ir::ExternalName` for a given wasm function index. fn get_func_name(func_index: FunctionIndex) -> ir::ExternalName { diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 7f73c3fc16..41583c1185 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -5,6 +5,8 @@ use cretonne::cursor::FuncCursor; use cretonne::settings::Flags; use translation_utils::{SignatureIndex, FunctionIndex, TableIndex, GlobalIndex, MemoryIndex, Global, Table, Memory}; +use std::vec::Vec; +use std::string::String; /// The value of a WebAssembly global variable. #[derive(Clone, Copy)] diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index df640cbb49..bb74db4276 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -8,6 +8,8 @@ use sections_translator::{SectionParsingError, parse_function_signatures, parse_ parse_elements_section, parse_data_section}; use environ::ModuleEnvironment; +use std::string::String; + /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IL /// [`Function`](../cretonne/ir/function/struct.Function.html). /// Returns the functions and also the mappings for imported functions and signature between the diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 359510547a..1bf7c9441c 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -16,6 +16,8 @@ use wasmparser::{Parser, ParserState, FuncType, ImportSectionEntryType, External use wasmparser; use std::str::from_utf8; use environ::ModuleEnvironment; +use std::vec::Vec; +use std::string::String; pub enum SectionParsingError { WrongSectionContent(String), diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index 1943f31326..5ced6d6056 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -7,6 +7,7 @@ use cretonne::ir::{self, Ebb, Inst, Value}; use environ::{FuncEnvironment, GlobalValue}; use std::collections::HashMap; use translation_utils::{GlobalIndex, MemoryIndex, SignatureIndex, FunctionIndex}; +use std::vec::Vec; /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following /// fields: From 03ee00762467ab1adb6924e595f218d4e9a3b951 Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Thu, 22 Mar 2018 13:10:41 -0700 Subject: [PATCH 1637/3084] Use clippy (#276) * cton-util: fix some clippy unnecessary pass-by-value warnings * clippy: ignore too many arguments / cyclomatic complexity in module since these functions are taking args coming from the command line, i dont think this is actually a valid lint, morally the arguments are all from one structure * cton-util: take care of remaining clippy warnings * cton-reader: fix all non-suspicious clippy warnings * cton-reader: disable clippy at site of suspicious lint * cton-frontend: disable clippy at the site of an invalid lint * cton-frontend: fix clippy warnings, or ignore benign ones * clippy: ignore the camelcase word WebAssembly in docs * cton-wasm: fix clippy complaints or ignore benign ones * cton-wasm tests: fix clippy complaints * cretonne: starting point turns off all clippy warnings * cretonne: clippy fixes, or lower allow() to source of problem * cretonne: more clippy fixes * cretonne: fix or disable needless_lifetimes lint this linter is buggy when the declared lifetime is used for another type constraint. * cretonne: fix clippy complaint about Pass::NoPass * rustfmt * fix prev minor api changes clippy suggested * add clippy to test-all * cton-filetests: clippy fixes * simplify clippy reporting in test-all * cretonne: document clippy allows better * cretonne: fix some more clippy lints * cretonne: fix clippy lints (mostly doc comments) * cretonne: allow all needless_lifetimes clippy warnings remove overrides at the false positives * rustfmt --- check-clippy.sh | 10 +++ clippy-all.sh | 7 ++ cranelift/clippy.toml | 1 + cranelift/src/cat.rs | 4 +- cranelift/src/compile.rs | 10 +-- cranelift/src/cton-util.rs | 19 +++-- cranelift/src/print_cfg.rs | 6 +- cranelift/src/rsfilecheck.rs | 2 +- cranelift/src/utils.rs | 5 +- cranelift/src/wasm.rs | 58 +++++++------- cranelift/test-all.sh | 8 ++ cranelift/tests/filetests.rs | 2 +- lib/cretonne/meta/gen_settings.py | 2 +- lib/cretonne/src/bforest/path.rs | 9 +-- lib/cretonne/src/bforest/pool.rs | 1 + lib/cretonne/src/binemit/relaxation.rs | 15 ++-- lib/cretonne/src/context.rs | 4 +- lib/cretonne/src/cursor.rs | 3 +- lib/cretonne/src/ir/extname.rs | 2 +- lib/cretonne/src/ir/layout.rs | 4 +- lib/cretonne/src/lib.rs | 33 +++++++- lib/cretonne/src/licm.rs | 1 + lib/cretonne/src/partition_slice.rs | 2 +- lib/cretonne/src/predicates.rs | 2 +- lib/cretonne/src/preopt.rs | 6 +- lib/cretonne/src/print_errors.rs | 4 +- lib/cretonne/src/ref_slice.rs | 2 +- lib/cretonne/src/regalloc/allocatable_set.rs | 6 +- lib/cretonne/src/regalloc/liverange.rs | 38 +++++----- lib/cretonne/src/regalloc/reload.rs | 14 ++-- lib/cretonne/src/regalloc/solver.rs | 1 + lib/cretonne/src/regalloc/virtregs.rs | 8 +- lib/cretonne/src/scoped_hash_map.rs | 2 +- lib/cretonne/src/timing.rs | 12 +-- lib/cretonne/src/verifier/liveness.rs | 34 ++++----- lib/cretonne/src/verifier/mod.rs | 79 ++++++++++---------- lib/filetests/src/concurrent.rs | 2 +- lib/filetests/src/lib.rs | 7 +- lib/filetests/src/runner.rs | 8 +- lib/filetests/src/runone.rs | 2 +- lib/filetests/src/subtest.rs | 2 +- lib/frontend/src/frontend.rs | 4 +- lib/frontend/src/lib.rs | 3 + lib/reader/src/lexer.rs | 2 +- lib/reader/src/lib.rs | 2 +- lib/reader/src/parser.rs | 32 ++++---- lib/reader/src/sourcemap.rs | 2 +- lib/wasm/src/code_translator.rs | 57 +++++++------- lib/wasm/src/environ/dummy.rs | 2 +- lib/wasm/src/lib.rs | 4 + lib/wasm/tests/wasm_testsuite.rs | 10 +-- 51 files changed, 310 insertions(+), 245 deletions(-) create mode 100755 check-clippy.sh create mode 100755 clippy-all.sh create mode 100644 cranelift/clippy.toml diff --git a/check-clippy.sh b/check-clippy.sh new file mode 100755 index 0000000000..20ac595851 --- /dev/null +++ b/check-clippy.sh @@ -0,0 +1,10 @@ +#!/bin/bash +set -euo pipefail + +# Usage: check-clippy.sh [--install] + +if cargo install --list | tee /dev/null | grep -q "^clippy v0"; then + exit 0 +else + exit 1 +fi diff --git a/clippy-all.sh b/clippy-all.sh new file mode 100755 index 0000000000..6ec2fd1411 --- /dev/null +++ b/clippy-all.sh @@ -0,0 +1,7 @@ +#!/bin/bash +set -euo pipefail + +# Check all sources with clippy. +# In the cton-util crate (root dir) clippy will only work with nightly cargo - +# there is a bug where it will reject the commands passed to it by cargo 0.25.0 +cargo +nightly clippy --all diff --git a/cranelift/clippy.toml b/cranelift/clippy.toml new file mode 100644 index 0000000000..caabf12b77 --- /dev/null +++ b/cranelift/clippy.toml @@ -0,0 +1 @@ +doc-valid-idents = [ "WebAssembly", "NaN", "SetCC" ] diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs index 3ae9c843d8..4d3d73ac2c 100644 --- a/cranelift/src/cat.rs +++ b/cranelift/src/cat.rs @@ -7,7 +7,7 @@ use cton_reader::parse_functions; use CommandResult; use utils::read_to_string; -pub fn run(files: Vec) -> CommandResult { +pub fn run(files: &[String]) -> CommandResult { for (i, f) in files.into_iter().enumerate() { if i != 0 { println!(); @@ -17,7 +17,7 @@ pub fn run(files: Vec) -> CommandResult { Ok(()) } -fn cat_one(filename: String) -> CommandResult { +fn cat_one(filename: &str) -> CommandResult { let buffer = read_to_string(&filename).map_err( |e| format!("{}: {}", filename, e), )?; diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 9047f055fe..a1cc244f5d 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -49,23 +49,23 @@ impl binemit::RelocSink for PrintRelocs { pub fn run( files: Vec, flag_print: bool, - flag_set: Vec, - flag_isa: String, + flag_set: &[String], + flag_isa: &str, ) -> Result<(), String> { let parsed = parse_sets_and_isa(flag_set, flag_isa)?; for filename in files { let path = Path::new(&filename); let name = String::from(path.as_os_str().to_string_lossy()); - handle_module(flag_print, path.to_path_buf(), name, parsed.as_fisa())?; + handle_module(flag_print, &path.to_path_buf(), &name, parsed.as_fisa())?; } Ok(()) } fn handle_module( flag_print: bool, - path: PathBuf, - name: String, + path: &PathBuf, + name: &str, fisa: FlagsOrIsa, ) -> Result<(), String> { let buffer = read_to_string(&path).map_err( diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 54d7d38d4d..216eae458b 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -86,15 +86,20 @@ fn cton_util() -> CommandResult { // Find the sub-command to execute. let result = if args.cmd_test { - cton_filetests::run(args.flag_verbose, args.arg_file).map(|_time| ()) + cton_filetests::run(args.flag_verbose, &args.arg_file).map(|_time| ()) } else if args.cmd_cat { - cat::run(args.arg_file) + cat::run(&args.arg_file) } else if args.cmd_filecheck { - rsfilecheck::run(args.arg_file, args.flag_verbose) + rsfilecheck::run(&args.arg_file, args.flag_verbose) } else if args.cmd_print_cfg { - print_cfg::run(args.arg_file) + print_cfg::run(&args.arg_file) } else if args.cmd_compile { - compile::run(args.arg_file, args.flag_print, args.flag_set, args.flag_isa) + compile::run( + args.arg_file, + args.flag_print, + &args.flag_set, + &args.flag_isa, + ) } else if args.cmd_wasm { wasm::run( args.arg_file, @@ -102,8 +107,8 @@ fn cton_util() -> CommandResult { args.flag_just_decode, args.flag_check_translation, args.flag_print, - args.flag_set, - args.flag_isa, + &args.flag_set, + &args.flag_isa, args.flag_print_size, ) } else { diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index 08ab439d05..65242cc8f8 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -8,7 +8,7 @@ use cretonne::cfg_printer::CFGPrinter; use cton_reader::parse_functions; use utils::read_to_string; -pub fn run(files: Vec) -> CommandResult { +pub fn run(files: &[String]) -> CommandResult { for (i, f) in files.into_iter().enumerate() { if i != 0 { println!(); @@ -18,8 +18,8 @@ pub fn run(files: Vec) -> CommandResult { Ok(()) } -fn print_cfg(filename: String) -> CommandResult { - let buffer = read_to_string(&filename).map_err( +fn print_cfg(filename: &str) -> CommandResult { + let buffer = read_to_string(filename).map_err( |e| format!("{}: {}", filename, e), )?; let items = parse_functions(&buffer).map_err( diff --git a/cranelift/src/rsfilecheck.rs b/cranelift/src/rsfilecheck.rs index 57b9bb9e8f..58b9cb9e53 100644 --- a/cranelift/src/rsfilecheck.rs +++ b/cranelift/src/rsfilecheck.rs @@ -7,7 +7,7 @@ use utils::read_to_string; use filecheck::{CheckerBuilder, Checker, NO_VARIABLES}; use std::io::{self, Read}; -pub fn run(files: Vec, verbose: bool) -> CommandResult { +pub fn run(files: &[String], verbose: bool) -> CommandResult { if files.is_empty() { return Err("No check files".to_string()); } diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index 7e45e99436..dc4f501893 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -41,10 +41,7 @@ impl OwnedFlagsOrIsa { } /// Parse "set" and "isa" commands. -pub fn parse_sets_and_isa( - flag_set: Vec, - flag_isa: String, -) -> Result { +pub fn parse_sets_and_isa(flag_set: &[String], flag_isa: &str) -> Result { let mut flag_builder = settings::builder(); parse_options( flag_set.iter().map(|x| x.as_str()), diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 3ab877bf34..efd93df77f 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -1,6 +1,7 @@ //! CLI tool to use the functions provided by the [cretonne-wasm](../cton_wasm/index.html) crate. //! //! Reads Wasm binary files, translates the functions' code to Cretonne IL. +#![cfg_attr(feature = "cargo-clippy", allow(too_many_arguments, cyclomatic_complexity))] use cton_wasm::{translate_module, DummyEnvironment, ModuleEnvironment}; use std::path::PathBuf; @@ -38,8 +39,8 @@ pub fn run( flag_just_decode: bool, flag_check_translation: bool, flag_print: bool, - flag_set: Vec, - flag_isa: String, + flag_set: &[String], + flag_isa: &str, flag_print_size: bool, ) -> Result<(), String> { let parsed = parse_sets_and_isa(flag_set, flag_isa)?; @@ -53,8 +54,8 @@ pub fn run( flag_check_translation, flag_print, flag_print_size, - path.to_path_buf(), - name, + &path.to_path_buf(), + &name, parsed.as_fisa(), )?; } @@ -67,8 +68,8 @@ fn handle_module( flag_check_translation: bool, flag_print: bool, flag_print_size: bool, - path: PathBuf, - name: String, + path: &PathBuf, + name: &str, fisa: FlagsOrIsa, ) -> Result<(), String> { let mut terminal = term::stdout().unwrap(); @@ -153,29 +154,27 @@ fn handle_module( context.func = func.clone(); if flag_check_translation { context.verify(fisa).map_err(|err| { - pretty_verifier_error(&context.func, fisa.isa, err) + pretty_verifier_error(&context.func, fisa.isa, &err) })?; - } else { - if let Some(isa) = fisa.isa { - let compiled_size = context.compile(isa).map_err(|err| { - pretty_error(&context.func, fisa.isa, err) - })?; - if flag_print_size { - println!( - "Function #{} code size: {} bytes", - func_index, - compiled_size - ); - total_module_code_size += compiled_size; - println!( - "Function #{} bytecode size: {} bytes", - func_index, - dummy_environ.func_bytecode_sizes[func_index] - ); - } - } else { - return Err(String::from("compilation requires a target isa")); + } else if let Some(isa) = fisa.isa { + let compiled_size = context.compile(isa).map_err(|err| { + pretty_error(&context.func, fisa.isa, err) + })?; + if flag_print_size { + println!( + "Function #{} code size: {} bytes", + func_index, + compiled_size + ); + total_module_code_size += compiled_size; + println!( + "Function #{} bytecode size: {} bytes", + func_index, + dummy_environ.func_bytecode_sizes[func_index] + ); } + } else { + return Err(String::from("compilation requires a target isa")); } if flag_print { vprintln!(flag_verbose, ""); @@ -194,10 +193,7 @@ fn handle_module( if !flag_check_translation && flag_print_size { println!("Total module code size: {} bytes", total_module_code_size); - let total_bytecode_size = dummy_environ.func_bytecode_sizes.iter().fold( - 0, - |sum, x| sum + x, - ); + let total_bytecode_size: usize = dummy_environ.func_bytecode_sizes.iter().sum(); println!("Total module bytecode size: {} bytes", total_bytecode_size); } diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 2b4e2eba01..572bf5de7b 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -55,4 +55,12 @@ cargo test --all --release banner "Rust documentation: $topdir/target/doc/cretonne/index.html" cargo doc +# Run clippy if we have it. +banner "Rust linter" +if $topdir/check-clippy.sh; then + $topdir/clippy-all.sh --write-mode=diff +else + echo "\`cargo +nightly install clippy\` for optional rust linting" +fi + banner "OK" diff --git a/cranelift/tests/filetests.rs b/cranelift/tests/filetests.rs index 97fbeb296f..6b5e392c76 100644 --- a/cranelift/tests/filetests.rs +++ b/cranelift/tests/filetests.rs @@ -3,5 +3,5 @@ extern crate cton_filetests; #[test] fn filetests() { // Run all the filetests in the following directories. - cton_filetests::run(false, vec!["filetests".into(), "docs".into()]).expect("test harness"); + cton_filetests::run(false, &["filetests".into(), "docs".into()]).expect("test harness"); } diff --git a/lib/cretonne/meta/gen_settings.py b/lib/cretonne/meta/gen_settings.py index b5f0de3488..373c9cb1a9 100644 --- a/lib/cretonne/meta/gen_settings.py +++ b/lib/cretonne/meta/gen_settings.py @@ -28,7 +28,7 @@ def gen_enum_types(sgrp, fmt): if not isinstance(setting, EnumSetting): continue ty = camel_case(setting.name) - fmt.doc_comment('Values for {}.'.format(setting)) + fmt.doc_comment('Values for `{}`.'.format(setting)) fmt.line('#[derive(Debug, PartialEq, Eq)]') with fmt.indented('pub enum {} {{'.format(ty), '}'): for v in setting.values: diff --git a/lib/cretonne/src/bforest/path.rs b/lib/cretonne/src/bforest/path.rs index e52c8e897e..34f165d86c 100644 --- a/lib/cretonne/src/bforest/path.rs +++ b/lib/cretonne/src/bforest/path.rs @@ -437,7 +437,7 @@ impl Path { // Discard the root node if it has shrunk to a single sub-tree. let mut ns = 0; - while let &NodeData::Inner { size: 0, ref tree, .. } = &pool[self.node[ns]] { + while let NodeData::Inner { size: 0, ref tree, .. } = pool[self.node[ns]] { ns += 1; self.node[ns] = tree[0]; } @@ -529,12 +529,11 @@ impl Path { // current entry[level] was one off the end of the node, it will now point at a proper // entry. debug_assert!(usize::from(self.entry[level]) < pool[self.node[level]].entries()); - } else { + + } else if usize::from(self.entry[level]) >= pool[self.node[level]].entries() { // There's no right sibling at this level, so the node can't be rebalanced. // Check if we are in an off-the-end position. - if usize::from(self.entry[level]) >= pool[self.node[level]].entries() { - self.size = 0; - } + self.size = 0; } } diff --git a/lib/cretonne/src/bforest/pool.rs b/lib/cretonne/src/bforest/pool.rs index 481271c8db..54a1f81cef 100644 --- a/lib/cretonne/src/bforest/pool.rs +++ b/lib/cretonne/src/bforest/pool.rs @@ -57,6 +57,7 @@ impl NodePool { pub fn free_tree(&mut self, node: Node) { if let NodeData::Inner { size, tree, .. } = self[node] { // Note that we have to capture `tree` by value to avoid borrow checker trouble. + #[cfg_attr(feature = "cargo-clippy", allow(needless_range_loop))] for i in 0..usize::from(size + 1) { // Recursively free sub-trees. This recursion can never be deeper than `MAX_PATH`, // and since most trees have less than a handful of nodes, it is worthwhile to diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index b0bf6196c6..24a78ceebd 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -76,14 +76,13 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> Result(&self, isa: &TargetIsa) -> verifier::Result { + pub fn verify_locations(&self, isa: &TargetIsa) -> verifier::Result { verifier::verify_locations(isa, &self.func, None) } /// Run the locations verifier only if the `enable_verifier` setting is true. - pub fn verify_locations_if<'a>(&self, isa: &TargetIsa) -> CtonResult { + pub fn verify_locations_if(&self, isa: &TargetIsa) -> CtonResult { if isa.flags().enable_verifier() { self.verify_locations(isa).map_err(Into::into) } else { diff --git a/lib/cretonne/src/cursor.rs b/lib/cretonne/src/cursor.rs index ed2528d5be..42f5e1e741 100644 --- a/lib/cretonne/src/cursor.rs +++ b/lib/cretonne/src/cursor.rs @@ -744,8 +744,9 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut EncCursor<'f> { if !self.srcloc.is_default() { self.func.srclocs[inst] = self.srcloc; } - // Assign an encoding. + // XXX Is there a way to describe this error to the user? + #[cfg_attr(feature = "cargo-clippy", allow(match_wild_err_arm))] match self.isa.encode( &self.func.dfg, &self.func.dfg[inst], diff --git a/lib/cretonne/src/ir/extname.rs b/lib/cretonne/src/ir/extname.rs index b7bcc7d94f..b37e40086c 100644 --- a/lib/cretonne/src/ir/extname.rs +++ b/lib/cretonne/src/ir/extname.rs @@ -16,7 +16,7 @@ const TESTCASE_NAME_LENGTH: usize = 16; /// to keep track of a sy mbol table. /// /// External names are primarily used as keys by code using Cretonne to map -/// from a cretonne::ir::FuncRef or similar to additional associated data. +/// from a `cretonne::ir::FuncRef` or similar to additional associated data. /// /// External names can also serve as a primitive testing and debugging tool. /// In particular, many `.cton` test files use function names to identify diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 0e6e010e7c..2f4c378edd 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -428,7 +428,7 @@ impl Layout { } /// Return an iterator over all EBBs in layout order. - pub fn ebbs<'f>(&'f self) -> Ebbs<'f> { + pub fn ebbs(&self) -> Ebbs { Ebbs { layout: self, next: self.first_ebb, @@ -611,7 +611,7 @@ impl Layout { } /// Iterate over the instructions in `ebb` in layout order. - pub fn ebb_insts<'f>(&'f self, ebb: Ebb) -> Insts<'f> { + pub fn ebb_insts(&self, ebb: Ebb) -> Insts { Insts { layout: self, head: self.ebbs[ebb].first_inst.into(), diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index f317ed9f85..0f731b9620 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -4,13 +4,44 @@ trivial_numeric_casts, unused_extern_crates)] +#![cfg_attr(feature="clippy", + plugin(clippy(conf_file="../../clippy.toml")))] + +#![cfg_attr(feature="cargo-clippy", allow( +// Rustfmt 0.9.0 is at odds with this lint: + block_in_if_condition_stmt, +// Produces only a false positive: + while_let_loop, +// Produces many false positives, but did produce some valid lints, now fixed: + needless_lifetimes, +// Generated code makes some style transgressions, but readability doesn't suffer much: + many_single_char_names, + identity_op, + needless_borrow, + cast_lossless, + unreadable_literal, + assign_op_pattern, + empty_line_after_outer_attr, +// Hard to avoid in generated code: + cyclomatic_complexity, + too_many_arguments, +// Code generator doesn't have a way to collapse identical arms: + match_same_arms, +// These are relatively minor style issues, but would be easy to fix: + new_without_default, + new_without_default_derive, + should_implement_trait, + redundant_field_names, + useless_let_if_seq, + len_without_is_empty))] + pub use context::Context; pub use legalizer::legalize_function; pub use verifier::verify_function; pub use write::write_function; /// Version number of the cretonne crate. -pub const VERSION: &'static str = env!("CARGO_PKG_VERSION"); +pub const VERSION: &str = env!("CARGO_PKG_VERSION"); #[macro_use] pub mod dbg; diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index e767fc838b..7887a39e4e 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -150,6 +150,7 @@ fn remove_loop_invariant_instructions( loop_values.insert(*val); } pos.goto_top(*ebb); + #[cfg_attr(feature = "cargo-clippy", allow(block_in_if_condition_stmt))] while let Some(inst) = pos.next_inst() { if pos.func.dfg.has_results(inst) && pos.func.dfg.inst_args(inst).into_iter().all(|arg| { diff --git a/lib/cretonne/src/partition_slice.rs b/lib/cretonne/src/partition_slice.rs index 7a94a9fe0d..dd0aff5da6 100644 --- a/lib/cretonne/src/partition_slice.rs +++ b/lib/cretonne/src/partition_slice.rs @@ -6,7 +6,7 @@ /// The order of elements is not preserved, unless the slice is already partitioned. /// /// Returns the number of elements where `p(t)` is true. -pub fn partition_slice<'a, T: 'a, F>(s: &'a mut [T], mut p: F) -> usize +pub fn partition_slice(s: &mut [T], mut p: F) -> usize where F: FnMut(&T) -> bool, { diff --git a/lib/cretonne/src/predicates.rs b/lib/cretonne/src/predicates.rs index 63d2e79af3..618369291d 100644 --- a/lib/cretonne/src/predicates.rs +++ b/lib/cretonne/src/predicates.rs @@ -7,7 +7,7 @@ //! bound is implemented by all the native integer types as well as `Imm64`. //! //! Some of these predicates may be unused in certain ISA configurations, so we suppress the -//! dead_code warning. +//! dead code warning. /// Check that `x` is the same as `y`. #[allow(dead_code)] diff --git a/lib/cretonne/src/preopt.rs b/lib/cretonne/src/preopt.rs index f5e8b3a7fe..698631316e 100644 --- a/lib/cretonne/src/preopt.rs +++ b/lib/cretonne/src/preopt.rs @@ -114,7 +114,7 @@ fn package_up_divrem_info( fn get_div_info(inst: Inst, dfg: &DataFlowGraph) -> Option { let idata: &InstructionData = &dfg[inst]; - if let &InstructionData::BinaryImm { opcode, arg, imm } = idata { + if let InstructionData::BinaryImm { opcode, arg, imm } = *idata { let (isSigned, isRem) = match opcode { Opcode::UdivImm => (false, false), Opcode::UremImm => (false, true), @@ -132,7 +132,7 @@ fn get_div_info(inst: Inst, dfg: &DataFlowGraph) -> Option { // that some previous constant propagation pass has pushed all such // immediates to their use points, creating BinaryImm instructions // instead? For now we take the conservative approach. - if let &InstructionData::Binary { opcode, args } = idata { + if let InstructionData::Binary { opcode, args } = *idata { let (isSigned, isRem) = match opcode { Opcode::Udiv => (false, false), Opcode::Urem => (false, true), @@ -484,7 +484,7 @@ fn get_const(value: Value, dfg: &DataFlowGraph) -> Option { match dfg.value_def(value) { ValueDef::Result(definingInst, resultNo) => { let definingIData: &InstructionData = &dfg[definingInst]; - if let &InstructionData::UnaryImm { opcode, imm } = definingIData { + if let InstructionData::UnaryImm { opcode, imm } = *definingIData { if opcode == Opcode::Iconst && resultNo == 0 { return Some(imm.into()); } diff --git a/lib/cretonne/src/print_errors.rs b/lib/cretonne/src/print_errors.rs index abd22f3e6b..1bc61cfb59 100644 --- a/lib/cretonne/src/print_errors.rs +++ b/lib/cretonne/src/print_errors.rs @@ -10,7 +10,7 @@ use std::fmt::Write; pub fn pretty_verifier_error( func: &ir::Function, isa: Option<&TargetIsa>, - err: verifier::Error, + err: &verifier::Error, ) -> String { let mut msg = err.to_string(); match err.location { @@ -26,7 +26,7 @@ pub fn pretty_verifier_error( /// Pretty-print a Cretonne error. pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CtonError) -> String { if let CtonError::Verifier(e) = err { - pretty_verifier_error(func, isa, e) + pretty_verifier_error(func, isa, &e) } else { err.to_string() } diff --git a/lib/cretonne/src/ref_slice.rs b/lib/cretonne/src/ref_slice.rs index b9dbd55dd1..251b601d00 100644 --- a/lib/cretonne/src/ref_slice.rs +++ b/lib/cretonne/src/ref_slice.rs @@ -1,6 +1,6 @@ //! Functions for converting a reference into a singleton slice. //! -//! See also the ref_slice crate on crates.io. +//! See also the [`ref_slice` crate](https://crates.io/crates/ref_slice). //! //! We define the functions here to avoid external dependencies, and to ensure that they are //! inlined in this crate. diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index e0abf76d37..8c40809310 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -197,9 +197,9 @@ impl<'a> fmt::Display for DisplayAllocatableSet<'a> { "{}", bank.names .get(offset as usize) - .and_then(|name| name.chars().skip(1).next()) - .unwrap_or( - char::from_digit(u32::from(offset % 10), 10).unwrap(), + .and_then(|name| name.chars().nth(1)) + .unwrap_or_else( + || char::from_digit(u32::from(offset % 10), 10).unwrap(), ) )?; } diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index dbde52f05d..2d66b010f7 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -276,33 +276,31 @@ impl GenLiveRange { } else { return first_time_livein; } - } else { + } else if let Some((_, end)) = c.prev() { // There's no interval beginning at `ebb`, but we could still be live-in at `ebb` with // a coalesced interval that begins before and ends after. - if let Some((_, end)) = c.prev() { - if order.cmp(end, ebb) == Ordering::Greater { - // Yep, the previous interval overlaps `ebb`. - first_time_livein = false; - if order.cmp(end, to) == Ordering::Less { - *c.value_mut().unwrap() = to; - } else { - return first_time_livein; - } + if order.cmp(end, ebb) == Ordering::Greater { + // Yep, the previous interval overlaps `ebb`. + first_time_livein = false; + if order.cmp(end, to) == Ordering::Less { + *c.value_mut().unwrap() = to; } else { - first_time_livein = true; - // The current interval does not overlap `ebb`, but it may still be possible to - // coalesce with it. - if order.is_ebb_gap(end, ebb) { - *c.value_mut().unwrap() = to; - } else { - c.insert(ebb, to); - } + return first_time_livein; } } else { - // There is no existing interval before `ebb`. first_time_livein = true; - c.insert(ebb, to); + // The current interval does not overlap `ebb`, but it may still be possible to + // coalesce with it. + if order.is_ebb_gap(end, ebb) { + *c.value_mut().unwrap() = to; + } else { + c.insert(ebb, to); + } } + } else { + // There is no existing interval before `ebb`. + first_time_livein = true; + c.insert(ebb, to); } // Now `c` to left pointing at an interval that ends in `to`. diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 63e7354453..106830bc19 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -306,14 +306,12 @@ impl<'a> Context<'a> { let args = self.cur.func.dfg.inst_args(inst); for (argidx, (op, &arg)) in constraints.ins.iter().zip(args).enumerate() { - if op.kind != ConstraintKind::Stack { - if self.liveness[arg].affinity.is_stack() { - self.candidates.push(ReloadCandidate { - argidx, - value: arg, - regclass: op.regclass, - }) - } + if op.kind != ConstraintKind::Stack && self.liveness[arg].affinity.is_stack() { + self.candidates.push(ReloadCandidate { + argidx, + value: arg, + regclass: op.regclass, + }) } } diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 7824444eec..3559623ae4 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -299,6 +299,7 @@ impl Move { } /// Get the "from" register and register class, if possible. + #[cfg_attr(feature = "cargo-clippy", allow(wrong_self_convention))] fn from_reg(&self) -> Option<(RegClass, RegUnit)> { match *self { Move::Reg { rc, from, .. } | diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index 097a66cccd..8069a0b69b 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -101,8 +101,10 @@ impl VirtRegs { where 'a: 'b, { - self.get(*value).map(|vr| self.values(vr)).unwrap_or( - ref_slice(value), + self.get(*value).map(|vr| self.values(vr)).unwrap_or_else( + || { + ref_slice(value) + }, ) } @@ -371,7 +373,7 @@ impl VirtRegs { let vreg = self.get(leader).unwrap_or_else(|| { // Allocate a vreg for `leader`, but leave it empty. let vr = self.alloc(); - if let &mut Some(ref mut vec) = &mut new_vregs { + if let Some(ref mut vec) = new_vregs { vec.push(vr); } self.value_vregs[leader] = vr.into(); diff --git a/lib/cretonne/src/scoped_hash_map.rs b/lib/cretonne/src/scoped_hash_map.rs index d5acf2ecb6..ded42672ab 100644 --- a/lib/cretonne/src/scoped_hash_map.rs +++ b/lib/cretonne/src/scoped_hash_map.rs @@ -1,4 +1,4 @@ -//! ScopedHashMap +//! `ScopedHashMap` //! //! This module defines a struct `ScopedHashMap` which defines a `HashMap`-like //! container that has a concept of scopes that can be entered and exited, such that diff --git a/lib/cretonne/src/timing.rs b/lib/cretonne/src/timing.rs index 7a7572f5c7..647e0a07aa 100644 --- a/lib/cretonne/src/timing.rs +++ b/lib/cretonne/src/timing.rs @@ -11,7 +11,7 @@ pub use self::details::{TimingToken, PassTimes, take_current, add_to_current}; // // This macro defines: // -// - A C-style enum containing all the pass names and a `NoPass` variant. +// - A C-style enum containing all the pass names and a `None` variant. // - A usize constant with the number of defined passes. // - A const array of pass descriptions. // - A public function per pass used to start the timing of that pass. @@ -21,9 +21,9 @@ macro_rules! define_passes { } => { #[allow(non_camel_case_types)] #[derive(Clone, Copy, Debug, PartialEq, Eq)] - enum $enum { $($pass,)+ NoPass } + enum $enum { $($pass,)+ None} - const $num_passes: usize = $enum::NoPass as usize; + const $num_passes: usize = $enum::None as usize; const $descriptions: [&str; $num_passes] = [ $($desc),+ ]; @@ -164,7 +164,7 @@ mod details { /// Information about passes in a single thread. thread_local!{ - static CURRENT_PASS: Cell = Cell::new(Pass::NoPass); + static CURRENT_PASS: Cell = Cell::new(Pass::None); static PASS_TIME: RefCell = RefCell::new(Default::default()); } @@ -204,7 +204,7 @@ mod details { } /// Add `timings` to the accumulated timings for the current thread. - pub fn add_to_current(times: PassTimes) { + pub fn add_to_current(times: &PassTimes) { PASS_TIME.with(|rc| for (a, b) in rc.borrow_mut().pass.iter_mut().zip( ×.pass, ) @@ -221,7 +221,7 @@ mod test { #[test] fn display() { - assert_eq!(Pass::NoPass.to_string(), ""); + assert_eq!(Pass::None.to_string(), ""); assert_eq!(Pass::regalloc.to_string(), "Register allocation"); } } diff --git a/lib/cretonne/src/verifier/liveness.rs b/lib/cretonne/src/verifier/liveness.rs index 95691625c8..8f971e2d05 100644 --- a/lib/cretonne/src/verifier/liveness.rs +++ b/lib/cretonne/src/verifier/liveness.rs @@ -86,16 +86,14 @@ impl<'a> LivenessVerifier<'a> { self.isa.encoding_info().display(encoding) ); } - } else { + } else if !lr.affinity.is_none() { // A non-encoded instruction can only define ghost values. - if !lr.affinity.is_none() { - return err!( - inst, - "{} is a real {} value defined by a ghost instruction", - val, - lr.affinity.display(&self.isa.register_info()) - ); - } + return err!( + inst, + "{} is a real {} value defined by a ghost instruction", + val, + lr.affinity.display(&self.isa.register_info()) + ); } } @@ -109,16 +107,14 @@ impl<'a> LivenessVerifier<'a> { return err!(inst, "{} is not live at this use", val); } - if encoding.is_legal() { - // A legal instruction is not allowed to depend on ghost values. - if lr.affinity.is_none() { - return err!( - inst, - "{} is a ghost value used by a real [{}] instruction", - val, - self.isa.encoding_info().display(encoding) - ); - } + // A legal instruction is not allowed to depend on ghost values. + if encoding.is_legal() && lr.affinity.is_none() { + return err!( + inst, + "{} is a ghost value used by a real [{}] instruction", + val, + self.isa.encoding_info().display(encoding) + ); } } } diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 26eb9a53d9..b19510a485 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -34,7 +34,7 @@ //! For polymorphic opcodes, determine the controlling type variable first. //! - Branches and jumps must pass arguments to destination EBBs that match the //! expected types exactly. The number of arguments must match. -//! - All EBBs in a jump_table must take no arguments. +//! - All EBBs in a jump table must take no arguments. //! - Function calls are type checked against their signature. //! - The entry block must take arguments that match the signature of the current //! function. @@ -871,50 +871,47 @@ impl<'a> Verifier<'a> { // Check special-purpose type constraints that can't be expressed in the normal opcode // constraints. fn typecheck_special(&self, inst: Inst, ctrl_type: Type) -> Result { - match self.func.dfg[inst] { - ir::InstructionData::Unary { opcode, arg } => { - let arg_type = self.func.dfg.value_type(arg); - match opcode { - Opcode::Bextend | Opcode::Uextend | Opcode::Sextend | Opcode::Fpromote => { - if arg_type.lane_count() != ctrl_type.lane_count() { - return err!( - inst, - "input {} and output {} must have same number of lanes", - arg_type, - ctrl_type - ); - } - if arg_type.lane_bits() >= ctrl_type.lane_bits() { - return err!( - inst, - "input {} must be smaller than output {}", - arg_type, - ctrl_type - ); - } + if let ir::InstructionData::Unary { opcode, arg } = self.func.dfg[inst] { + let arg_type = self.func.dfg.value_type(arg); + match opcode { + Opcode::Bextend | Opcode::Uextend | Opcode::Sextend | Opcode::Fpromote => { + if arg_type.lane_count() != ctrl_type.lane_count() { + return err!( + inst, + "input {} and output {} must have same number of lanes", + arg_type, + ctrl_type + ); } - Opcode::Breduce | Opcode::Ireduce | Opcode::Fdemote => { - if arg_type.lane_count() != ctrl_type.lane_count() { - return err!( - inst, - "input {} and output {} must have same number of lanes", - arg_type, - ctrl_type - ); - } - if arg_type.lane_bits() <= ctrl_type.lane_bits() { - return err!( - inst, - "input {} must be larger than output {}", - arg_type, - ctrl_type - ); - } + if arg_type.lane_bits() >= ctrl_type.lane_bits() { + return err!( + inst, + "input {} must be smaller than output {}", + arg_type, + ctrl_type + ); } - _ => {} } + Opcode::Breduce | Opcode::Ireduce | Opcode::Fdemote => { + if arg_type.lane_count() != ctrl_type.lane_count() { + return err!( + inst, + "input {} and output {} must have same number of lanes", + arg_type, + ctrl_type + ); + } + if arg_type.lane_bits() <= ctrl_type.lane_bits() { + return err!( + inst, + "input {} must be larger than output {}", + arg_type, + ctrl_type + ); + } + } + _ => {} } - _ => {} } Ok(()) } diff --git a/lib/filetests/src/concurrent.rs b/lib/filetests/src/concurrent.rs index 6a18bb2390..1ddf643258 100644 --- a/lib/filetests/src/concurrent.rs +++ b/lib/filetests/src/concurrent.rs @@ -70,7 +70,7 @@ impl ConcurrentRunner { assert!(self.request_tx.is_none(), "must shutdown before join"); for h in self.handles.drain(..) { match h.join() { - Ok(t) => timing::add_to_current(t), + Ok(t) => timing::add_to_current(&t), Err(e) => println!("worker panicked: {:?}", e), } } diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs index 36d321358c..a6a43d4437 100644 --- a/lib/filetests/src/lib.rs +++ b/lib/filetests/src/lib.rs @@ -3,6 +3,11 @@ //! This crate contains the main test driver as well as implementations of the //! available filetest commands. +#![cfg_attr(feature="cargo-clippy", allow( + type_complexity, +// Rustfmt 0.9.0 is at odds with this lint: + block_in_if_condition_stmt))] + #[macro_use(dbg)] extern crate cretonne; extern crate cton_reader; @@ -44,7 +49,7 @@ type TestResult = Result; /// Directories are scanned recursively for test cases ending in `.cton`. These test cases are /// executed on background threads. /// -pub fn run(verbose: bool, files: Vec) -> TestResult { +pub fn run(verbose: bool, files: &[String]) -> TestResult { let mut runner = TestRunner::new(verbose); for path in files.iter().map(Path::new) { diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs index ff40ed81e2..49078cff8e 100644 --- a/lib/filetests/src/runner.rs +++ b/lib/filetests/src/runner.rs @@ -45,7 +45,7 @@ impl Display for QueueEntry { f, "{}.{:03} {}", dur.as_secs(), - dur.subsec_nanos() / 1000000, + dur.subsec_nanos() / 1_000_000, p ) } @@ -135,7 +135,7 @@ impl TestRunner { // This lets us skip spurious extensionless files without statting everything // needlessly. if !dir.is_file() { - self.path_error(dir, err); + self.path_error(&dir, &err); } } Ok(entries) => { @@ -149,7 +149,7 @@ impl TestRunner { // libstd/sys/unix/fs.rs seems to suggest that breaking now would // be a good idea, or the iterator could keep returning the same // error forever. - self.path_error(dir, err); + self.path_error(&dir, &err); break; } Ok(entry) => { @@ -172,7 +172,7 @@ impl TestRunner { } /// Report an error related to a path. - fn path_error(&mut self, path: PathBuf, err: E) { + fn path_error(&mut self, path: &PathBuf, err: &E) { self.errors += 1; println!("{}: {}", path.to_string_lossy(), err); } diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs index fcc01e4f95..dab44ff920 100644 --- a/lib/filetests/src/runone.rs +++ b/lib/filetests/src/runone.rs @@ -132,7 +132,7 @@ fn run_one_test<'a>( if !context.verified && test.needs_verifier() { verify_function(&func, context.flags_or_isa()).map_err( |e| { - pretty_verifier_error(&func, isa, e) + pretty_verifier_error(&func, isa, &e) }, )?; context.verified = true; diff --git a/lib/filetests/src/subtest.rs b/lib/filetests/src/subtest.rs index a9cf657d8e..39812ed228 100644 --- a/lib/filetests/src/subtest.rs +++ b/lib/filetests/src/subtest.rs @@ -1,4 +1,4 @@ -//! SubTest trait. +//! `SubTest` trait. use std::result; use std::borrow::Cow; diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index de89698f15..8cd2c5cc82 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -400,7 +400,7 @@ where /// /// This can be used to insert SSA code that doesn't need to access locals and that doesn't /// need to know about `FunctionBuilder` at all. - pub fn cursor<'f>(&'f mut self) -> FuncCursor<'f> { + pub fn cursor(&mut self) -> FuncCursor { self.ensure_inserted_ebb(); FuncCursor::new(self.func) .with_srcloc(self.srcloc) @@ -548,6 +548,8 @@ where /// Returns a displayable object for the function as it is. /// /// Useful for debug purposes. Use it with `None` for standard printing. + // Clippy thinks the lifetime that follows is needless, but rustc needs it + #[cfg_attr(feature = "cargo-clippy", allow(needless_lifetimes))] pub fn display<'b, I: Into>>(&'b self, isa: I) -> DisplayFunction { self.func.display(isa) } diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index f5b4514e56..1f2c2e0db8 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -131,6 +131,9 @@ trivial_numeric_casts, unused_extern_crates)] +#![cfg_attr(feature="cargo-clippy", + allow(new_without_default, redundant_field_names))] + extern crate cretonne; pub use frontend::{FunctionBuilderContext, FunctionBuilder}; diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index ca315d3b58..4b1a952d1b 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -53,7 +53,7 @@ pub struct LocatedToken<'a> { } /// Wrap up a `Token` with the given location. -fn token<'a>(token: Token<'a>, loc: Location) -> Result, LocatedError> { +fn token(token: Token, loc: Location) -> Result { Ok(LocatedToken { token, location: loc, diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index 85ea048950..d6c23cfa81 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -1,6 +1,6 @@ //! Cretonne file reader library. //! -//! The cton_reader library supports reading .cton files. This functionality is needed for testing +//! The `cton_reader` library supports reading .cton files. This functionality is needed for testing //! Cretonne, but is not essential for a JIT compiler. #![deny(missing_docs, diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index a9568942ed..2e320b31a7 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -37,7 +37,7 @@ pub fn parse_functions(text: &str) -> Result> { /// Parse the entire `text` as a test case file. /// /// The returned `TestFile` contains direct references to substrings of `text`. -pub fn parse_test<'a>(text: &'a str) -> Result> { +pub fn parse_test(text: &str) -> Result { let _tt = timing::parse_text(); let mut parser = Parser::new(text); // Gather the preamble comments. @@ -277,6 +277,10 @@ impl<'a> Parser<'a> { // Get the current lookahead token, after making sure there is one. fn token(&mut self) -> Option> { + // clippy says self.lookahead is immutable so this loop is either infinite or never + // running. I don't think this is true - self.lookahead is mutated in the loop body - so + // maybe this is a clippy bug? Either way, disable clippy for this. + #[cfg_attr(feature = "cargo-clippy", allow(while_immutable_condition))] while self.lookahead == None { match self.lex.next() { Some(Ok(lexer::LocatedToken { token, location })) => { @@ -957,7 +961,7 @@ impl<'a> Parser<'a> { isa.register_info() .parse_regunit(name) .map(ArgumentLoc::Reg) - .ok_or(self.error("invalid register name")) + .ok_or_else(|| self.error("invalid register name")) } else { err!(self.loc, "argument location requires exactly one isa") } @@ -1392,12 +1396,12 @@ impl<'a> Parser<'a> { match self.token() { Some(Token::Arrow) => { self.consume(); - self.parse_value_alias(results, ctx)?; + self.parse_value_alias(&results, ctx)?; } Some(Token::Equal) => { self.consume(); self.parse_instruction( - results, + &results, srcloc, encoding, result_locations, @@ -1408,7 +1412,7 @@ impl<'a> Parser<'a> { _ if !results.is_empty() => return err!(self.loc, "expected -> or ="), _ => { self.parse_instruction( - results, + &results, srcloc, encoding, result_locations, @@ -1512,7 +1516,7 @@ impl<'a> Parser<'a> { isa.register_info() .parse_regunit(name) .map(ValueLoc::Reg) - .ok_or(self.error("invalid register value location")) + .ok_or_else(|| self.error("invalid register value location")) } else { err!(self.loc, "value location requires exactly one isa") } @@ -1601,7 +1605,7 @@ impl<'a> Parser<'a> { // // value_alias ::= [inst-results] "->" Value(v) // - fn parse_value_alias(&mut self, results: Vec, ctx: &mut Context) -> Result<()> { + fn parse_value_alias(&mut self, results: &[Value], ctx: &mut Context) -> Result<()> { if results.len() != 1 { return err!(self.loc, "wrong number of aliases"); } @@ -1621,7 +1625,7 @@ impl<'a> Parser<'a> { // fn parse_instruction( &mut self, - results: Vec, + results: &[Value], srcloc: ir::SourceLoc, encoding: Option, result_locations: Option>, @@ -1629,7 +1633,7 @@ impl<'a> Parser<'a> { ebb: Ebb, ) -> Result<()> { // Define the result values. - for val in &results { + for val in results { ctx.map.def_value(*val, &self.loc)?; } @@ -1674,7 +1678,7 @@ impl<'a> Parser<'a> { let num_results = ctx.function.dfg.make_inst_results_for_parser( inst, ctrl_typevar, - &results, + results, ); ctx.function.layout.append_inst(inst, ebb); ctx.map.def_entity(inst.into(), &opcode_loc).expect( @@ -1784,11 +1788,9 @@ impl<'a> Parser<'a> { opcode ); } - } else { - // Treat it as a syntax error to speficy a typevar on a non-polymorphic opcode. - if ctrl_type != VOID { - return err!(self.loc, "{} does not take a typevar", opcode); - } + // Treat it as a syntax error to speficy a typevar on a non-polymorphic opcode. + } else if ctrl_type != VOID { + return err!(self.loc, "{} does not take a typevar", opcode); } Ok(ctrl_type) diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 704c097855..36fb954fd9 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -13,7 +13,7 @@ use lexer::split_entity_name; use std::collections::HashMap; /// Mapping from entity names to source locations. -#[derive(Debug)] +#[derive(Debug, Default)] pub struct SourceMap { // Store locations for entities, including instructions. locations: HashMap, diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 940725600a..42d232e6f5 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -36,6 +36,8 @@ use environ::{FuncEnvironment, GlobalValue}; use std::{i32, u32}; use std::vec::Vec; +// Clippy warns about "flags: _" but its important to document that the flags field is ignored +#[cfg_attr(feature = "cargo-clippy", allow(unneeded_field_pattern))] /// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted /// a return. pub fn translate_operator( @@ -45,7 +47,7 @@ pub fn translate_operator( environ: &mut FE, ) { if !state.reachable { - return translate_unreachable_operator(op, builder, state); + return translate_unreachable_operator(&op, builder, state); } // This big match treats all Wasm code operators. @@ -198,9 +200,8 @@ pub fn translate_operator( builder.switch_to_block(frame.following_code()); builder.seal_block(frame.following_code()); // If it is a loop we also have to seal the body loop block - match frame { - ControlStackFrame::Loop { header, .. } => builder.seal_block(header), - _ => {} + if let ControlStackFrame::Loop { header, .. } = frame { + builder.seal_block(header) } state.stack.truncate(frame.original_stack_size()); state.stack.extend_from_slice( @@ -857,15 +858,17 @@ pub fn translate_operator( } } +// Clippy warns us of some fields we are deliberately ignoring +#[cfg_attr(feature = "cargo-clippy", allow(unneeded_field_pattern))] /// Deals with a Wasm instruction located in an unreachable portion of the code. Most of them /// are dropped but special ones like `End` or `Else` signal the potential end of the unreachable /// portion so the translation state muts be updated accordingly. fn translate_unreachable_operator( - op: Operator, + op: &Operator, builder: &mut FunctionBuilder, state: &mut TranslationState, ) { - match op { + match *op { Operator::If { ty: _ } => { // Push a placeholder control stack entry. The if isn't reachable, // so we don't have any branches anywhere. @@ -877,27 +880,25 @@ fn translate_unreachable_operator( } Operator::Else => { let i = state.control_stack.len() - 1; - match state.control_stack[i] { - ControlStackFrame::If { - branch_inst, - ref mut reachable_from_top, - .. - } => { - if *reachable_from_top { - // We have a branch from the top of the if to the else. - state.reachable = true; - // And because there's an else, there can no longer be a - // branch from the top directly to the end. - *reachable_from_top = false; + if let ControlStackFrame::If { + branch_inst, + ref mut reachable_from_top, + .. + } = state.control_stack[i] + { + if *reachable_from_top { + // We have a branch from the top of the if to the else. + state.reachable = true; + // And because there's an else, there can no longer be a + // branch from the top directly to the end. + *reachable_from_top = false; - // We change the target of the branch instruction - let else_ebb = builder.create_ebb(); - builder.change_jump_destination(branch_inst, else_ebb); - builder.seal_block(else_ebb); - builder.switch_to_block(else_ebb); - } + // We change the target of the branch instruction + let else_ebb = builder.create_ebb(); + builder.change_jump_destination(branch_inst, else_ebb); + builder.seal_block(else_ebb); + builder.switch_to_block(else_ebb); } - _ => {} } } Operator::End => { @@ -1057,10 +1058,10 @@ fn translate_br_if( builder.ins().brnz(val, br_destination, inputs); } -fn translate_br_if_args<'state>( +fn translate_br_if_args( relative_depth: u32, - state: &'state mut TranslationState, -) -> (ir::Ebb, &'state [ir::Value]) { + state: &mut TranslationState, +) -> (ir::Ebb, &[ir::Value]) { let i = state.control_stack.len() - 1 - (relative_depth as usize); let (return_count, br_destination) = { let frame = &mut state.control_stack[i]; diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index de941f3b9a..c99afd32d9 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -121,7 +121,7 @@ impl DummyEnvironment { } } -/// The FuncEnvironment implementation for use by the `DummyEnvironment`. +/// The `FuncEnvironment` implementation for use by the `DummyEnvironment`. pub struct DummyFuncEnvironment<'dummy_environment> { pub mod_info: &'dummy_environment DummyModuleInfo, } diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 6e9a6a365a..f2a1adb7cb 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -12,6 +12,10 @@ #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![cfg_attr(feature="clippy", + plugin(clippy(conf_file="../../clippy.toml")))] +#![cfg_attr(feature="cargo-clippy", + allow(new_without_default, redundant_field_names))] extern crate wasmparser; extern crate cton_frontend; diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs index 824a7606da..5d969c9cc2 100644 --- a/lib/wasm/tests/wasm_testsuite.rs +++ b/lib/wasm/tests/wasm_testsuite.rs @@ -25,7 +25,7 @@ fn testsuite() { // Ignore files starting with `.`, which could be editor temporary files if let Some(stem) = p.path().file_stem() { if let Some(stemstr) = stem.to_str() { - return !stemstr.starts_with("."); + return !stemstr.starts_with('.'); } } false @@ -35,7 +35,7 @@ fn testsuite() { let flags = Flags::new(&settings::builder()); for path in paths { let path = path.path(); - handle_module(path, &flags); + handle_module(&path, &flags); } } @@ -44,7 +44,7 @@ fn return_at_end() { let mut flag_builder = settings::builder(); flag_builder.enable("return_at_end").unwrap(); let flags = Flags::new(&flag_builder); - handle_module(PathBuf::from("../../wasmtests/return_at_end.wat"), &flags); + handle_module(&PathBuf::from("../../wasmtests/return_at_end.wat"), &flags); } fn read_wasm_file(path: PathBuf) -> Result, io::Error> { @@ -54,7 +54,7 @@ fn read_wasm_file(path: PathBuf) -> Result, io::Error> { Ok(buf) } -fn handle_module(path: PathBuf, flags: &Flags) { +fn handle_module(path: &PathBuf, flags: &Flags) { let data = match path.extension() { None => { panic!("the file extension is not wasm or wat"); @@ -103,7 +103,7 @@ fn handle_module(path: PathBuf, flags: &Flags) { translate_module(&data, &mut dummy_environ).unwrap(); for func in &dummy_environ.info.function_bodies { verifier::verify_function(func, flags) - .map_err(|err| panic!(pretty_verifier_error(func, None, err))) + .map_err(|err| panic!(pretty_verifier_error(func, None, &err))) .unwrap(); } } From e78735752024d73655e8538e165e735e19c41903 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 20 Mar 2018 15:22:22 -0700 Subject: [PATCH 1638/3084] Wrap build.py's contents in a main function. This prevents its local variables from becoming global variables. --- lib/cretonne/meta/build.py | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/lib/cretonne/meta/build.py b/lib/cretonne/meta/build.py index a94b9ce706..8c83300604 100644 --- a/lib/cretonne/meta/build.py +++ b/lib/cretonne/meta/build.py @@ -14,19 +14,27 @@ import gen_legalizer import gen_registers import gen_binemit -parser = argparse.ArgumentParser(description='Generate sources for Cretonne.') -parser.add_argument('--out-dir', help='set output directory') -args = parser.parse_args() -out_dir = args.out_dir +def main(): + # type: () -> None + parser = argparse.ArgumentParser( + description='Generate sources for Cretonne.') + parser.add_argument('--out-dir', help='set output directory') -isas = isa.all_isas() + args = parser.parse_args() + out_dir = args.out_dir -gen_types.generate(out_dir) -gen_instr.generate(isas, out_dir) -gen_settings.generate(isas, out_dir) -gen_encoding.generate(isas, out_dir) -gen_legalizer.generate(isas, out_dir) -gen_registers.generate(isas, out_dir) -gen_binemit.generate(isas, out_dir) -gen_build_deps.generate() + isas = isa.all_isas() + + gen_types.generate(out_dir) + gen_instr.generate(isas, out_dir) + gen_settings.generate(isas, out_dir) + gen_encoding.generate(isas, out_dir) + gen_legalizer.generate(isas, out_dir) + gen_registers.generate(isas, out_dir) + gen_binemit.generate(isas, out_dir) + gen_build_deps.generate() + + +if __name__ == "__main__": + main() From 3ec7918ba1abb1cb05a2706d065f4fe1d8c3e058 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 20 Mar 2018 15:25:40 -0700 Subject: [PATCH 1639/3084] Tidy up a redundant import. --- lib/cretonne/meta/cdsl/isa.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/cretonne/meta/cdsl/isa.py index 064a353117..77b071636a 100644 --- a/lib/cretonne/meta/cdsl/isa.py +++ b/lib/cretonne/meta/cdsl/isa.py @@ -12,7 +12,7 @@ from .instructions import InstructionGroup try: from typing import Tuple, Union, Any, Iterable, Sequence, List, Set, Dict, TYPE_CHECKING # noqa if TYPE_CHECKING: - from .instructions import MaybeBoundInst, InstructionGroup, InstructionFormat # noqa + from .instructions import MaybeBoundInst, InstructionFormat # noqa from .predicates import PredNode, PredKey # noqa from .settings import SettingGroup # noqa from .registers import RegBank # noqa From af8ac8f8caa6a1b7b6fa070bd14d899e0f93d848 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 20 Mar 2018 17:37:09 -0700 Subject: [PATCH 1640/3084] Avoid calling `analyze_branch()` when the `BranchInfo` is not needed. It's faster to just call `opcode().is_branch()`. --- lib/cretonne/src/regalloc/live_value_tracker.rs | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index bedd0480e5..5cb3a6f694 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -6,7 +6,6 @@ use dominator_tree::DominatorTree; use entity::{EntityList, ListPool}; -use ir::instructions::BranchInfo; use ir::{Inst, Ebb, Value, DataFlowGraph, Layout, ExpandedProgramPoint}; use partition_slice::partition_slice; use regalloc::affinity::Affinity; @@ -261,9 +260,8 @@ impl LiveValueTracker { ) -> (&[LiveValue], &[LiveValue], &[LiveValue]) { // Save a copy of the live values before any branches or jumps that could be somebody's // immediate dominator. - match dfg.analyze_branch(inst) { - BranchInfo::NotABranch => {} - _ => self.save_idom_live_set(inst), + if dfg[inst].opcode().is_branch() { + self.save_idom_live_set(inst); } // Move killed values to the end of the vector. From fc7b0a7e510606a16ed5c8da3f6be3d85902683d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 22 Mar 2018 13:43:06 -0700 Subject: [PATCH 1641/3084] Rename the `no_std` feature to `core`. See https://github.com/yurydelendik/wasmparser.rs/pull/49#issuecomment-375436225 for more details. --- README.rst | 22 +++++++++++----------- cranelift/test-no_std.sh | 4 ++-- lib/cretonne/Cargo.toml | 4 ++-- lib/cretonne/src/dbg.rs | 30 +++++++++++++++--------------- lib/cretonne/src/lib.rs | 8 ++++---- lib/cretonne/src/timing.rs | 4 ++-- lib/cretonne/src/verifier/mod.rs | 4 ++-- lib/frontend/Cargo.toml | 2 +- lib/frontend/src/ssa.rs | 12 ++++++------ lib/native/Cargo.toml | 2 +- lib/wasm/Cargo.toml | 2 +- lib/wasm/src/lib.rs | 2 +- 12 files changed, 48 insertions(+), 48 deletions(-) diff --git a/README.rst b/README.rst index 51ee362d10..8368aed68f 100644 --- a/README.rst +++ b/README.rst @@ -62,14 +62,14 @@ Building with `no_std` To build cretonne without libstd, disable the `std` feature on `lib/cretonne`, `lib/frontend`, `lib/native`, and `lib/wasm`, which is otherwise enabled by -default, and enable the `no_std` feature. +default, and enable the `core` feature. For example, to build `cretonne`: .. code-block:: sh cd lib/cretonne - cargo build --no-default-features --features no_std + cargo build --no-default-features --features core Or, when using `cretonne` as a dependency (in Cargo.toml): @@ -78,23 +78,23 @@ Or, when using `cretonne` as a dependency (in Cargo.toml): [dependency.cretonne] ... default-features = false - features = ["no_std"] + features = ["core"] -`no_std` is currently "best effort". We won't try to break it, and we'll -accept patches fixing problems, however we don't expect all developers to -build and test with `no_std` when submitting patches. Accordingly, the +`no_std` support is currently "best effort". We won't try to break it, and +we'll accept patches fixing problems, however we don't expect all developers to +build and test `no_std` when submitting patches. Accordingly, the `./test-all.sh` script does not test `no_std`. There is a separate `./test-no_std.sh` script that tests the `no_std` -feature in packages which support it. +support in packages which support it. It's important to note that cretonne still needs liballoc to compile. Thus, whatever environment is used must implement an allocator. -Also, to allow the use of HashMaps in `no_std` mode, an external crate -called `hashmap_core` is pulled in (only in `no_std` builds). This -is mostly the same as `std::collections::HashMap`, except that it doesn't -have DOS protection. Just something to think about. +Also, to allow the use of HashMaps with `no_std`, an external crate called +`hashmap_core` is pulled in (via the `core` feature). This is mostly the same +as `std::collections::HashMap`, except that it doesn't have DOS protection. +Just something to think about. Building the documentation -------------------------- diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh index d3edbb921f..c2857ff756 100755 --- a/cranelift/test-no_std.sh +++ b/cranelift/test-no_std.sh @@ -21,8 +21,8 @@ for LIB in $LIBS do banner "Rust unit tests in $LIB" cd "lib/$LIB" - cargo test --no-default-features --features no_std - cargo test --features no_std + cargo test --no-default-features --features core + cargo test --features core cd "$topdir" done diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 89340e0a1a..01b7e2ab70 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -26,9 +26,9 @@ version = "0.1.1" optional = true [features] -# The "std" feature enables use of libstd. The "no_std" feature enables use +# The "std" feature enables use of libstd. The "core" feature enables use # of some minimal std-like replacement libraries. At least one of these two # features to be enabled. default = ["std"] std = [] -no_std = ["hashmap_core"] +core = ["hashmap_core"] diff --git a/lib/cretonne/src/dbg.rs b/lib/cretonne/src/dbg.rs index 8f949f00e1..6c51bd23e5 100644 --- a/lib/cretonne/src/dbg.rs +++ b/lib/cretonne/src/dbg.rs @@ -8,23 +8,23 @@ /// /// The output will appear in files named `cretonne.dbg.*`, where the suffix is named after the /// thread doing the logging. -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] use std::cell::RefCell; -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] use std::env; -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] use std::ffi::OsStr; use std::fmt; -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] use std::fs::File; -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] use std::io::{self, Write}; -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] use std::sync::atomic; -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] use std::thread; -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] static STATE: atomic::AtomicIsize = atomic::ATOMIC_ISIZE_INIT; /// Is debug tracing enabled? @@ -33,7 +33,7 @@ static STATE: atomic::AtomicIsize = atomic::ATOMIC_ISIZE_INIT; /// other than `0`. /// /// This inline function turns into a constant `false` when debug assertions are disabled. -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] #[inline] pub fn enabled() -> bool { if cfg!(debug_assertions) { @@ -47,14 +47,14 @@ pub fn enabled() -> bool { } /// Does nothing -#[cfg(feature = "no_std")] +#[cfg(not(feature = "std"))] #[inline] pub fn enabled() -> bool { false } /// Initialize `STATE` from the environment variable. -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] fn initialize() -> bool { let enable = match env::var_os("CRETONNE_DBG") { Some(s) => s != OsStr::new("0"), @@ -70,7 +70,7 @@ fn initialize() -> bool { enable } -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] thread_local! { static WRITER : RefCell> = RefCell::new(open_file()); } @@ -78,7 +78,7 @@ thread_local! { /// Write a line with the given format arguments. /// /// This is for use by the `dbg!` macro. -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] pub fn writeln_with_format_args(args: fmt::Arguments) -> io::Result<()> { WRITER.with(|rc| { let mut w = rc.borrow_mut(); @@ -88,7 +88,7 @@ pub fn writeln_with_format_args(args: fmt::Arguments) -> io::Result<()> { } /// Open the tracing file for the current thread. -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] fn open_file() -> io::BufWriter { let curthread = thread::current(); let tmpstr; @@ -116,7 +116,7 @@ macro_rules! dbg { if $crate::dbg::enabled() { // Drop the error result so we don't get compiler errors for ignoring it. // What are you going to do, log the error? - #[cfg(not(feature = "no_std"))] + #[cfg(feature = "std")] $crate::dbg::writeln_with_format_args(format_args!($($arg)+)).ok(); } } diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 29643d552f..8511951b50 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -4,13 +4,13 @@ trivial_numeric_casts, unused_extern_crates)] -// Turns on alloc feature if no_std +// Turns on no_std and alloc features if std is not available. #![cfg_attr(not(feature = "std"), no_std)] #![cfg_attr(not(feature = "std"), feature(alloc))] -// Include the `hashmap_core` crate if no_std +// Include the `hashmap_core` crate if std is not available. #[allow(unused_extern_crates)] -#[cfg(feature = "no_std")] +#[cfg(not(feature = "std"))] extern crate hashmap_core; #[cfg(not(feature = "std"))] #[macro_use] @@ -66,7 +66,7 @@ mod topo_order; mod unreachable_code; mod write; -/// This replaces `std` in builds with no_std. +/// This replaces `std` in builds with `core`. #[cfg(not(feature = "std"))] mod std { pub use core::*; diff --git a/lib/cretonne/src/timing.rs b/lib/cretonne/src/timing.rs index a70ed5fb21..bbe2fce07c 100644 --- a/lib/cretonne/src/timing.rs +++ b/lib/cretonne/src/timing.rs @@ -93,7 +93,7 @@ impl fmt::Display for Pass { /// This whole module can be gated on a `cfg` feature to provide a dummy implementation for /// performance-sensitive builds or restricted environments. The dummy implementation must provide /// `TimingToken` and `PassTimes` types and `take_current`, `add_to_current`, and `start_pass` funcs -#[cfg(not(feature = "no_std"))] +#[cfg(feature = "std")] mod details { use super::{Pass, NUM_PASSES, DESCRIPTIONS}; use std::cell::{Cell, RefCell}; @@ -216,7 +216,7 @@ mod details { } /// Dummy `debug` implementation -#[cfg(feature = "no_std")] +#[cfg(not(feature = "std"))] mod details { use super::Pass; /// Dummy `TimingToken` diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index d8c2cb6dc3..b2ad18c8e8 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -1125,9 +1125,9 @@ mod tests { Ok(_) => { panic!("Expected an error!") }, Err(Error { message, .. } ) => { if !message.contains($msg) { - #[cfg(not(feature = "no_std"))] + #[cfg(feature = "std")] panic!(format!("'{}' did not contain the substring '{}'", message, $msg)); - #[cfg(feature = "no_std")] + #[cfg(not(feature = "std"))] panic!("error message did not contain the expected substring"); } } diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index cdbabdda79..75822af7ce 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -17,4 +17,4 @@ cretonne = { path = "../cretonne", version = "0.3.4", default-features = false } [features] default = ["std"] std = ["cretonne/std"] -no_std = ["cretonne/no_std"] +core = ["cretonne/core"] diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index cf44371983..37e4fd8751 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -1026,9 +1026,9 @@ mod tests { match verify_function(&func, &flags) { Ok(()) => {} Err(err) => { - #[cfg(not(feature = "no_std"))] + #[cfg(feature = "std")] panic!(err.message); - #[cfg(feature = "no_std")] + #[cfg(not(feature = "std"))] panic!("function failed to verify"); } } @@ -1205,9 +1205,9 @@ mod tests { match verify_function(&func, &flags) { Ok(()) => {} Err(err) => { - #[cfg(not(feature = "no_std"))] + #[cfg(feature = "std")] panic!(err.message); - #[cfg(feature = "no_std")] + #[cfg(not(feature = "std"))] panic!("function failed to verify"); } } @@ -1256,9 +1256,9 @@ mod tests { match verify_function(&func, &flags) { Ok(()) => {} Err(err) => { - #[cfg(not(feature = "no_std"))] + #[cfg(feature = "std")] panic!(err.message); - #[cfg(feature = "no_std")] + #[cfg(not(feature = "std"))] panic!("function failed to verify"); } } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index e89b0f02fe..817f12c73d 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -19,4 +19,4 @@ raw-cpuid = "3.0.0" [features] default = ["std"] std = ["cretonne/std"] -no_std = ["cretonne/no_std"] +core = ["cretonne/core"] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 3f35ee5c96..27177521ee 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -26,4 +26,4 @@ tempdir = "0.3.5" [features] default = ["std"] std = ["cretonne/std", "cretonne-frontend/std"] -no_std = ["hashmap_core", "cretonne/no_std", "cretonne-frontend/no_std"] +core = ["hashmap_core", "cretonne/core", "cretonne-frontend/core"] diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index a6492b5d66..d791a14846 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -21,7 +21,7 @@ extern crate alloc; #[allow(unused_extern_crates)] -#[cfg(feature = "no_std")] +#[cfg(not(feature = "std"))] extern crate hashmap_core; extern crate wasmparser; From 9602b78320be2ca588f710510d5b7805ec37530a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 22 Mar 2018 16:55:18 -0700 Subject: [PATCH 1642/3084] Disable the LICM pass for now. There appear to be underlying problems with the way Cretonne handles value aliases, which are causing problems for LICM. Disable LICM until we have a chance to fix the underlying issues. Fixes #275. --- lib/cretonne/src/context.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 5c2c884fdf..c6018dcd9e 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -92,8 +92,10 @@ impl Context { self.legalize(isa)?; if isa.flags().opt_level() == OptLevel::Best { self.compute_domtree(); + /* TODO: Re-enable LICM. self.compute_loop_analysis(); self.licm(isa)?; + */ self.simple_gvn(isa)?; } self.compute_domtree(); From aa73de8ca160426220de508d6901e05483770616 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 23 Mar 2018 03:46:42 -0700 Subject: [PATCH 1643/3084] Make code that tests for specific opcode families more consistent. --- lib/cretonne/src/legalizer/mod.rs | 26 +++++++++++++------------- lib/cretonne/src/verifier/locations.rs | 8 ++++---- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 51198d05a0..ae40c42581 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -56,19 +56,19 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is let opcode = pos.func.dfg[inst].opcode(); // Check for ABI boundaries that need to be converted to the legalized signature. - if opcode.is_call() && boundary::handle_call_abi(inst, pos.func, cfg) { - // Go back and legalize the inserted argument conversion instructions. - pos.set_position(prev_pos); - continue; - } - - if opcode.is_return() && boundary::handle_return_abi(inst, pos.func, cfg) { - // Go back and legalize the inserted return value conversion instructions. - pos.set_position(prev_pos); - continue; - } - - if opcode.is_branch() { + if opcode.is_call() { + if boundary::handle_call_abi(inst, pos.func, cfg) { + // Go back and legalize the inserted argument conversion instructions. + pos.set_position(prev_pos); + continue; + } + } else if opcode.is_return() { + if boundary::handle_return_abi(inst, pos.func, cfg) { + // Go back and legalize the inserted return value conversion instructions. + pos.set_position(prev_pos); + continue; + } + } else if opcode.is_branch() { split::simplify_branch_arguments(&mut pos.func.dfg, inst); } diff --git a/lib/cretonne/src/verifier/locations.rs b/lib/cretonne/src/verifier/locations.rs index ae3c5cfb24..f2ad816924 100644 --- a/lib/cretonne/src/verifier/locations.rs +++ b/lib/cretonne/src/verifier/locations.rs @@ -69,10 +69,10 @@ impl<'a> LocationVerifier<'a> { let opcode = dfg[inst].opcode(); if opcode.is_return() { self.check_return_abi(inst, &divert)?; - } - - if opcode.is_branch() && !divert.is_empty() { - self.check_cfg_edges(inst, &divert)?; + } else if opcode.is_branch() { + if !divert.is_empty() { + self.check_cfg_edges(inst, &divert)?; + } } self.update_diversions(inst, &mut divert)?; From 51eea8b89dd715763ccd65fc722c16f0be8ef1c0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 23 Mar 2018 12:46:23 -0700 Subject: [PATCH 1644/3084] Add some comments noting unimplemented features. While use of these features will trigger an `unimplemented!()`, it is nice to let users know in advance about features which aren't yet implemented. --- lib/cretonne/src/ir/heap.rs | 2 ++ lib/cretonne/src/isa/stack.rs | 2 ++ 2 files changed, 4 insertions(+) diff --git a/lib/cretonne/src/ir/heap.rs b/lib/cretonne/src/ir/heap.rs index 7920ec344a..e2a5366a1a 100644 --- a/lib/cretonne/src/ir/heap.rs +++ b/lib/cretonne/src/ir/heap.rs @@ -25,6 +25,8 @@ pub struct HeapData { #[derive(Clone)] pub enum HeapBase { /// The heap base lives in a reserved register. + /// + /// This feature is not yet implemented. ReservedReg, /// The heap base is in a global variable. diff --git a/lib/cretonne/src/isa/stack.rs b/lib/cretonne/src/isa/stack.rs index e8a9b6981b..9e09a52808 100644 --- a/lib/cretonne/src/isa/stack.rs +++ b/lib/cretonne/src/isa/stack.rs @@ -68,6 +68,8 @@ pub enum StackBase { FP = 1, /// Use an explicit zone pointer in a general-purpose register. + /// + /// This feature is not yet implemented. Zone = 2, } From 80d2c5d9bff736c67ab3823505ae7b64f61a67fd Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Mon, 26 Mar 2018 16:48:20 -0700 Subject: [PATCH 1645/3084] Implement shift-immediate encodings for x86 (#283) * add x86 encodings for shift-immediate instructions implements encodings for ishl_imm, sshr_imm, and ushr_imm. uses 8-bit immediates. added tests for the encodings to intel/binary64.cton. Canonical versions come from llvm-mc. * translate test to use shift-immediates * shift immediate encodings: use enc_i32_i64 and note why the regular shift encodings cant use it above * add additional encoding tests for shift immediates this covers 32 bit mode, and 64 bit operations in 64 bit mode. --- .../isa/intel/baseline_clz_ctz_popcount.cton | 30 +++++++------------ cranelift/filetests/isa/intel/binary32.cton | 7 +++++ cranelift/filetests/isa/intel/binary64.cton | 30 +++++++++++++++++++ .../isa/intel/legalize-div-traps.cton | 4 +-- .../filetests/isa/intel/legalize-div.cton | 4 +-- lib/cretonne/meta/isa/intel/encodings.py | 8 +++++ 6 files changed, 59 insertions(+), 24 deletions(-) diff --git a/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount.cton b/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount.cton index 62a793de60..b87815aa12 100644 --- a/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount.cton +++ b/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount.cton @@ -47,28 +47,23 @@ ebb1(v20: i32): function %i64_popcount(i64) -> i64 { ebb0(v30: i64): v31 = popcnt v30; - ; check: iconst.i32 - ; check: ushr + ; check: ushr_imm ; check: iconst.i64 ; check: band ; check: isub - ; check: iconst.i32 - ; check: ushr + ; check: ushr_imm ; check: band ; check: isub - ; check: iconst.i32 - ; check: ushr + ; check: ushr_imm ; check: band ; check: isub - ; check: iconst.i32 - ; check: ushr + ; check: ushr_imm ; check: iadd ; check: iconst.i64 ; check: band ; check: iconst.i64 ; check: imul - ; check: iconst.i32 - ; check: ushr + ; check: ushr_imm return v31; } @@ -78,27 +73,22 @@ ebb0(v30: i64): function %i32_popcount(i32) -> i32 { ebb0(v40: i32): v41 = popcnt v40; - ; check: iconst.i32 - ; check: ushr + ; check: ushr_imm ; check: iconst.i32 ; check: band ; check: isub - ; check: iconst.i32 - ; check: ushr + ; check: ushr_imm ; check: band ; check: isub - ; check: iconst.i32 - ; check: ushr + ; check: ushr_imm ; check: band ; check: isub - ; check: iconst.i32 - ; check: ushr + ; check: ushr_imm ; check: iadd ; check: iconst.i32 ; check: band ; check: iconst.i32 ; check: imul - ; check: iconst.i32 - ; check: ushr + ; check: ushr_imm return v41; } diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index df98a86515..56301607cc 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -403,6 +403,13 @@ ebb0: ; asm: addl $-2147483648, %esp adjust_sp_imm -2147483648 ; bin: 81 c4 80000000 + ; Shift immediates + ; asm: shll $2, %esi + [-,%rsi] v513 = ishl_imm v2, 2 ; bin: c1 e6 02 + ; asm: sarl $5, %esi + [-,%rsi] v514 = sshr_imm v2, 5 ; bin: c1 fe 05 + ; asm: shrl $8, %esi + [-,%rsi] v515 = ushr_imm v2, 8 ; bin: c1 ee 08 ; asm: testl %ecx, %ecx ; asm: je ebb1 diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 9eca950ee6..5741f18041 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -529,6 +529,21 @@ ebb0: ; asm: addq $-2147483648, %rsp adjust_sp_imm -2147483648 ; bin: 48 81 c4 80000000 + ; Shift immediates + ; asm: shlq $12, %rsi + [-,%rsi] v515 = ishl_imm v2, 12 ; bin: 48 c1 e6 0c + ; asm: shlq $13, %r8 + [-,%r8] v516 = ishl_imm v4, 13 ; bin: 49 c1 e0 0d + ; asm: sarq $32, %rsi + [-,%rsi] v517 = sshr_imm v2, 32 ; bin: 48 c1 fe 20 + ; asm: sarq $33, %r8 + [-,%r8] v518 = sshr_imm v4, 33 ; bin: 49 c1 f8 21 + ; asm: shrl $62, %rsi + [-,%rsi] v519 = ushr_imm v2, 62 ; bin: 48 c1 ee 3e + ; asm: shrl $63, %r8 + [-,%r8] v520 = ushr_imm v4, 63 ; bin: 49 c1 e8 3f + + ; asm: testq %rcx, %rcx ; asm: je ebb1 brz v1, ebb1 ; bin: 48 85 c9 74 1b @@ -653,6 +668,7 @@ ebb1: ; asm: cmpq $10000, %r10 [-,%eflags] v525 = ifcmp_imm v2, 10000 ; bin: 49 81 fa 00002710 + return } @@ -1053,6 +1069,19 @@ ebb0: ; asm: cmpl $10000, %r10d [-,%eflags] v525 = ifcmp_imm v3, 10000 ; bin: 41 81 fa 00002710 + ; asm: shll $2, %esi + [-,%rsi] v526 = ishl_imm v2, 2 ; bin: c1 e6 02 + ; asm: shll $12, %r10d + [-,%r10] v527 = ishl_imm v3, 12 ; bin: 41 c1 e2 0c + ; asm: sarl $5, %esi + [-,%rsi] v529 = sshr_imm v2, 5 ; bin: c1 fe 05 + ; asm: sarl $32, %r10d + [-,%r10] v530 = sshr_imm v3, 32 ; bin: 41 c1 fa 20 + ; asm: shrl $8, %esi + [-,%rsi] v532 = ushr_imm v2, 8 ; bin: c1 ee 08 + ; asm: shrl $31, %r10d + [-,%r10] v533 = ushr_imm v3, 31 ; bin: 41 c1 ea 1f + ; asm: testl %ecx, %ecx ; asm: je ebb1x brz v1, ebb1 ; bin: 85 c9 74 18 @@ -1082,6 +1111,7 @@ ebb1: ; asm: ebb2x: ebb2: jump ebb1 ; bin: eb fd + } ; Tests for i32/i8 conversion instructions. diff --git a/cranelift/filetests/isa/intel/legalize-div-traps.cton b/cranelift/filetests/isa/intel/legalize-div-traps.cton index fa070fba1a..3869e66325 100644 --- a/cranelift/filetests/isa/intel/legalize-div-traps.cton +++ b/cranelift/filetests/isa/intel/legalize-div-traps.cton @@ -40,7 +40,7 @@ ebb0(v0: i64, v1: i64): ; nextln: brif eq $fm1, $(m1=$EBB) ; nextln: $(fz=$V) = ifcmp_imm v1, 0 ; nextln: trapif eq $fz, int_divz - ; check: $(hi=$V) = sshr + ; check: $(hi=$V) = sshr_imm ; nextln: $(q=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1 ; nextln: jump $(done=$EBB)($q) ; check: $m1: @@ -60,7 +60,7 @@ ebb0(v0: i64, v1: i64): v2 = srem v0, v1 ; nextln: $(fm1=$V) = ifcmp_imm v1, -1 ; nextln: brif eq $fm1, $(m1=$EBB) - ; check: $(hi=$V) = sshr + ; check: $(hi=$V) = sshr_imm ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1 ; nextln: jump $(done=$EBB)($r) ; check: $m1: diff --git a/cranelift/filetests/isa/intel/legalize-div.cton b/cranelift/filetests/isa/intel/legalize-div.cton index ec3f8ec5d3..d6179b2611 100644 --- a/cranelift/filetests/isa/intel/legalize-div.cton +++ b/cranelift/filetests/isa/intel/legalize-div.cton @@ -32,7 +32,7 @@ function %sdiv(i64, i64) -> i64 { ebb0(v0: i64, v1: i64): ; check: ebb0( v2 = sdiv v0, v1 - ; check: $(hi=$V) = sshr + ; check: $(hi=$V) = sshr_imm ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1 return v2 ; nextln: return $d @@ -46,7 +46,7 @@ ebb0(v0: i64, v1: i64): v2 = srem v0, v1 ; nextln: $(fm1=$V) = ifcmp_imm v1, -1 ; nextln: brif eq $fm1, $(m1=$EBB) - ; check: $(hi=$V) = sshr + ; check: $(hi=$V) = sshr_imm ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1 ; nextln: jump $(done=$EBB)($r) ; check: $m1: diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 489e61be9f..afb544cb9a 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -164,11 +164,19 @@ for inst, rrr in [ (base.ishl, 4), (base.ushr, 5), (base.sshr, 7)]: + # Cannot use enc_i32_i64 for this pattern because instructions require + # .any suffix. X86_32.enc(inst.i32.any, *r.rc(0xd3, rrr=rrr)) X86_64.enc(inst.i64.any, *r.rc.rex(0xd3, rrr=rrr, w=1)) X86_64.enc(inst.i32.any, *r.rc.rex(0xd3, rrr=rrr)) X86_64.enc(inst.i32.any, *r.rc(0xd3, rrr=rrr)) +for inst, rrr in [ + (base.ishl_imm, 4), + (base.ushr_imm, 5), + (base.sshr_imm, 7)]: + enc_i32_i64(inst, r.rib, 0xc1, rrr=rrr) + # Population count. X86_32.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) X86_64.enc(base.popcnt.i64, *r.urm.rex(0xf3, 0x0f, 0xb8, w=1), From bcd3309c155667afa91ad10c70b78046034a2b97 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 23 Mar 2018 13:10:28 -0700 Subject: [PATCH 1646/3084] Mention ``reserved_reg`` being unimplemented in the docs too. --- cranelift/docs/langref.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 3a855c32b4..fb62111d0a 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -654,6 +654,8 @@ trap when accessed. address space reserved for the heap, not including the guard pages. :arg GuardBytes: Size of the guard pages in bytes. +The ``reserved_reg`` option is not yet implemented. + Dynamic heaps ~~~~~~~~~~~~~ @@ -672,6 +674,8 @@ is resized. The bound of a dynamic heap is stored in a global variable. :arg BoundGV: Global variable containing the current heap bound in bytes. :arg GuardBytes: Size of the guard pages in bytes. +The ``reserved_reg`` option is not yet implemented. + Heap examples ~~~~~~~~~~~~~ From 685cde98a4ef43e3ac43321887e40102655f11bb Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 23 Mar 2018 13:34:40 -0700 Subject: [PATCH 1647/3084] Mark loads from globals `aligned` and `notrap`. Mark loads from globals generated by cton_wasm or by legalization as `aligned` and `notrap`, since memory for these globals should be allocated by the runtime environment for that purpose. This reduces the number of potentially trapping instructions, which can reduce the amount of metadata required by embedding environments. --- cranelift/docs/langref.rst | 6 ++++++ cranelift/filetests/isa/intel/legalize-memory.cton | 6 +++--- lib/cretonne/meta/base/instructions.py | 3 +++ lib/cretonne/src/ir/globalvar.rs | 3 ++- lib/cretonne/src/ir/heap.rs | 6 ++++-- lib/cretonne/src/legalizer/globalvar.rs | 7 +++++-- lib/cretonne/src/legalizer/heap.rs | 12 ++++++++++-- lib/wasm/src/code_translator.rs | 14 ++++++++++---- lib/wasm/src/environ/dummy.rs | 5 ++++- 9 files changed, 47 insertions(+), 15 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index fb62111d0a..fb716257bd 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -654,6 +654,9 @@ trap when accessed. address space reserved for the heap, not including the guard pages. :arg GuardBytes: Size of the guard pages in bytes. +When the base is a global variable, it must be :term:`accessible` and naturally +aligned for a pointer value. + The ``reserved_reg`` option is not yet implemented. Dynamic heaps @@ -674,6 +677,9 @@ is resized. The bound of a dynamic heap is stored in a global variable. :arg BoundGV: Global variable containing the current heap bound in bytes. :arg GuardBytes: Size of the guard pages in bytes. +When the base is a global variable, it must be :term:`accessible` and naturally +aligned for a pointer value. + The ``reserved_reg`` option is not yet implemented. Heap examples diff --git a/cranelift/filetests/isa/intel/legalize-memory.cton b/cranelift/filetests/isa/intel/legalize-memory.cton index 5b7246a2e3..1ba979deb1 100644 --- a/cranelift/filetests/isa/intel/legalize-memory.cton +++ b/cranelift/filetests/isa/intel/legalize-memory.cton @@ -23,7 +23,7 @@ function %deref(i64 vmctx) -> i64 { ebb1(v1: i64): v2 = global_addr.i64 gv2 ; check: $(a1=$V) = iadd_imm v1, -16 - ; check: $(p1=$V) = load.i64 $a1 + ; check: $(p1=$V) = load.i64 notrap aligned $a1 ; check: v2 = iadd_imm $p1, 32 return v2 ; check: return v2 @@ -55,7 +55,7 @@ ebb0(v0: i32, v999: i64): ; Checks here are assuming that no pipehole opts fold the load offsets. ; nextln: $(xoff=$V) = uextend.i64 v0 ; nextln: $(haddr=$V) = iadd_imm v999, 64 - ; nextln: $(hbase=$V) = load.i64 $haddr + ; nextln: $(hbase=$V) = load.i64 notrap aligned $haddr ; nextln: v1 = iadd $hbase, $xoff v2 = load.f32 v1+16 ; nextln: v2 = load.f32 v1+16 @@ -103,7 +103,7 @@ ebb0(v0: i32, v999: i64): ; Checks here are assuming that no pipehole opts fold the load offsets. ; nextln: $(xoff=$V) = uextend.i64 v0 ; nextln: $(haddr=$V) = iadd_imm.i64 v999, 64 - ; nextln: $(hbase=$V) = load.i64 $haddr + ; nextln: $(hbase=$V) = load.i64 notrap aligned $haddr ; nextln: v1 = iadd $hbase, $xoff v2 = load.f32 v1+0x7fff_ffff ; nextln: v2 = load.f32 v1+0x7fff_ffff diff --git a/lib/cretonne/meta/base/instructions.py b/lib/cretonne/meta/base/instructions.py index 8ba1aa65fa..c54f3ef570 100644 --- a/lib/cretonne/meta/base/instructions.py +++ b/lib/cretonne/meta/base/instructions.py @@ -588,6 +588,9 @@ stack_check = Instruction( Read the stack limit from ``GV`` and compare it to the stack pointer. If the stack pointer has reached or exceeded the limit, generate a trap with a ``stk_ovf`` code. + + The global variable must be accessible and naturally aligned for a + pointer-sized value. """, ins=GV, can_trap=True) diff --git a/lib/cretonne/src/ir/globalvar.rs b/lib/cretonne/src/ir/globalvar.rs index 78e1c6cb98..c8fc3b8ce1 100644 --- a/lib/cretonne/src/ir/globalvar.rs +++ b/lib/cretonne/src/ir/globalvar.rs @@ -17,7 +17,8 @@ pub enum GlobalVarData { /// Variable is part of a struct pointed to by another global variable. /// /// The `base` global variable is assumed to contain a pointer to a struct. This global - /// variable lives at an offset into the struct. + /// variable lives at an offset into the struct. The memory must be accessible, and + /// naturally aligned to hold a pointer value. Deref { /// The base pointer global variable. base: GlobalVar, diff --git a/lib/cretonne/src/ir/heap.rs b/lib/cretonne/src/ir/heap.rs index e2a5366a1a..b41565b25b 100644 --- a/lib/cretonne/src/ir/heap.rs +++ b/lib/cretonne/src/ir/heap.rs @@ -29,7 +29,8 @@ pub enum HeapBase { /// This feature is not yet implemented. ReservedReg, - /// The heap base is in a global variable. + /// The heap base is in a global variable. The variable must be accessible and naturally + /// aligned for a pointer. GlobalVar(GlobalVar), } @@ -38,7 +39,8 @@ pub enum HeapBase { pub enum HeapStyle { /// A dynamic heap can be relocated to a different base address when it is grown. Dynamic { - /// Global variable holding the current bound of the heap in bytes. + /// Global variable holding the current bound of the heap in bytes. It is + /// required to be accessible and naturally aligned for a pointer-sized integer. bound_gv: GlobalVar, }, diff --git a/lib/cretonne/src/legalizer/globalvar.rs b/lib/cretonne/src/legalizer/globalvar.rs index 5f6427822b..901b0da416 100644 --- a/lib/cretonne/src/legalizer/globalvar.rs +++ b/lib/cretonne/src/legalizer/globalvar.rs @@ -52,8 +52,11 @@ fn deref_addr(inst: ir::Inst, func: &mut ir::Function, base: ir::GlobalVar, offs pos.use_srcloc(inst); let base_addr = pos.ins().global_addr(ptr_ty, base); - // TODO: We could probably set both `notrap` and `aligned` on this load instruction. - let base_ptr = pos.ins().load(ptr_ty, ir::MemFlags::new(), base_addr, 0); + let mut mflags = ir::MemFlags::new(); + // Deref globals are required to be accessible and aligned. + mflags.set_notrap(); + mflags.set_aligned(); + let base_ptr = pos.ins().load(ptr_ty, mflags, base_addr, 0); pos.func.dfg.replace(inst).iadd_imm(base_ptr, offset); } diff --git a/lib/cretonne/src/legalizer/heap.rs b/lib/cretonne/src/legalizer/heap.rs index 37ce226351..0d6e7d385b 100644 --- a/lib/cretonne/src/legalizer/heap.rs +++ b/lib/cretonne/src/legalizer/heap.rs @@ -58,7 +58,11 @@ fn dynamic_addr( // Start with the bounds check. Trap if `offset + size > bound`. let bound_addr = pos.ins().global_addr(addr_ty, bound_gv); - let bound = pos.ins().load(offset_ty, MemFlags::new(), bound_addr, 0); + let mut mflags = MemFlags::new(); + // The bound variable is requied to be accessible and aligned. + mflags.set_notrap(); + mflags.set_aligned(); + let bound = pos.ins().load(offset_ty, mflags, bound_addr, 0); let oob; if size == 1 { @@ -175,7 +179,11 @@ fn offset_addr( ir::HeapBase::ReservedReg => unimplemented!(), ir::HeapBase::GlobalVar(base_gv) => { let base_addr = pos.ins().global_addr(addr_ty, base_gv); - let base = pos.ins().load(addr_ty, MemFlags::new(), base_addr, 0); + let mut mflags = MemFlags::new(); + // The base address variable is requied to be accessible and aligned. + mflags.set_notrap(); + mflags.set_aligned(); + let base = pos.ins().load(addr_ty, mflags, base_addr, 0); pos.func.dfg.replace(inst).iadd(base, offset); } } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 42d232e6f5..69111dc8e8 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -75,8 +75,9 @@ pub fn translate_operator( GlobalValue::Const(val) => val, GlobalValue::Memory { gv, ty } => { let addr = builder.ins().global_addr(environ.native_pointer(), gv); - // TODO: It is likely safe to set `aligned notrap` flags on a global load. - let flags = ir::MemFlags::new(); + let mut flags = ir::MemFlags::new(); + flags.set_notrap(); + flags.set_aligned(); builder.ins().load(ty, flags, addr, 0) } }; @@ -87,8 +88,9 @@ pub fn translate_operator( GlobalValue::Const(_) => panic!("global #{} is a constant", global_index), GlobalValue::Memory { gv, .. } => { let addr = builder.ins().global_addr(environ.native_pointer(), gv); - // TODO: It is likely safe to set `aligned notrap` flags on a global store. - let flags = ir::MemFlags::new(); + let mut flags = ir::MemFlags::new(); + flags.set_notrap(); + flags.set_aligned(); let val = state.pop1(); builder.ins().store(flags, val, addr, 0); } @@ -992,6 +994,9 @@ fn translate_load( // We don't yet support multiple linear memories. let heap = state.get_heap(builder.func, 0, environ); let (base, offset) = get_heap_addr(heap, addr32, offset, environ.native_pointer(), builder); + // Note that we don't set `is_aligned` here, even if the load instruction's + // alignment immediate says it's aligned, because WebAssembly's immediate + // field is just a hint, while Cretonne's aligned flag needs a guarantee. let flags = MemFlags::new(); let (load, dfg) = builder.ins().Load( opcode, @@ -1017,6 +1022,7 @@ fn translate_store( // We don't yet support multiple linear memories. let heap = state.get_heap(builder.func, 0, environ); let (base, offset) = get_heap_addr(heap, addr32, offset, environ.native_pointer(), builder); + // See the comments in `translate_load` about the flags. let flags = MemFlags::new(); builder.ins().Store( opcode, diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index c99afd32d9..bb21b7536a 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -209,7 +209,10 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ let ext = pos.ins().uextend(I64, callee); pos.ins().imul_imm(ext, 4) }; - let func_ptr = pos.ins().load(ptr, ir::MemFlags::new(), callee_offset, 0); + let mut mflags = ir::MemFlags::new(); + mflags.set_notrap(); + mflags.set_aligned(); + let func_ptr = pos.ins().load(ptr, mflags, callee_offset, 0); // Build a value list for the indirect call instruction containing the callee, call_args, // and the vmctx parameter. From 14faef7374141fe619fc7ad600255fb181a94aa9 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 26 Mar 2018 08:18:32 -0700 Subject: [PATCH 1648/3084] Derive Debug for LiveValue. --- lib/cretonne/src/regalloc/live_value_tracker.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index 5cb3a6f694..ee1f58944b 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -32,6 +32,7 @@ pub struct LiveValueTracker { } /// Information about a value that is live at the current program point. +#[derive(Debug)] pub struct LiveValue { /// The live value. pub value: Value, From ffe89cdc0a55fbd5611093d34f1633890b4f21f7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 26 Mar 2018 21:34:15 -0700 Subject: [PATCH 1649/3084] Rename %eflags to %rflags. EFLAGS is a subregister of RFLAGS. For consistency with GPRs where we use the 64-bit names to refer to the registers, use the 64-bit name for RFLAGS as well. --- .../baseline_clz_ctz_popcount_encoding.cton | 24 ++++++------- .../filetests/isa/intel/binary32-float.cton | 14 ++++---- cranelift/filetests/isa/intel/binary32.cton | 16 ++++----- .../filetests/isa/intel/binary64-float.cton | 14 ++++---- cranelift/filetests/isa/intel/binary64.cton | 28 +++++++-------- lib/cretonne/meta/isa/intel/recipes.py | 36 +++++++++---------- lib/cretonne/meta/isa/intel/registers.py | 2 +- lib/cretonne/src/isa/intel/binemit.rs | 2 +- 8 files changed, 68 insertions(+), 68 deletions(-) diff --git a/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton b/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton index 0b7003449d..2e552b7145 100644 --- a/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton +++ b/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton @@ -15,56 +15,56 @@ ebb0: [-,%r11] v10 = iconst.i64 0x1234 ; asm: bsfq %r11, %rcx - [-,%rcx,%eflags] v11, v12 = x86_bsf v10 ; bin: 49 0f bc cb + [-,%rcx,%rflags] v11, v12 = x86_bsf v10 ; bin: 49 0f bc cb [-,%rdx] v14 = iconst.i64 0x5678 ; asm: bsfq %rdx, %r12 - [-,%r12,%eflags] v15, v16 = x86_bsf v14 ; bin: 4c 0f bc e2 + [-,%r12,%rflags] v15, v16 = x86_bsf v14 ; bin: 4c 0f bc e2 ; asm: bsfq %rdx, %rdi - [-,%rdi,%eflags] v17, v18 = x86_bsf v14 ; bin: 48 0f bc fa + [-,%rdi,%rflags] v17, v18 = x86_bsf v14 ; bin: 48 0f bc fa ; 32-bit wide bsf [-,%r11] v20 = iconst.i32 0x1234 ; asm: bsfl %r11d, %ecx - [-,%rcx,%eflags] v21, v22 = x86_bsf v20 ; bin: 41 0f bc cb + [-,%rcx,%rflags] v21, v22 = x86_bsf v20 ; bin: 41 0f bc cb [-,%rdx] v24 = iconst.i32 0x5678 ; asm: bsfl %edx, %r12d - [-,%r12,%eflags] v25, v26 = x86_bsf v24 ; bin: 44 0f bc e2 + [-,%r12,%rflags] v25, v26 = x86_bsf v24 ; bin: 44 0f bc e2 ; asm: bsfl %edx, %esi - [-,%rsi,%eflags] v27, v28 = x86_bsf v24 ; bin: 0f bc f2 + [-,%rsi,%rflags] v27, v28 = x86_bsf v24 ; bin: 0f bc f2 ; 64-bit wide bsr [-,%r11] v30 = iconst.i64 0x1234 ; asm: bsrq %r11, %rcx - [-,%rcx,%eflags] v31, v32 = x86_bsr v30 ; bin: 49 0f bd cb + [-,%rcx,%rflags] v31, v32 = x86_bsr v30 ; bin: 49 0f bd cb [-,%rdx] v34 = iconst.i64 0x5678 ; asm: bsrq %rdx, %r12 - [-,%r12,%eflags] v35, v36 = x86_bsr v34 ; bin: 4c 0f bd e2 + [-,%r12,%rflags] v35, v36 = x86_bsr v34 ; bin: 4c 0f bd e2 ; asm: bsrq %rdx, %rdi - [-,%rdi,%eflags] v37, v38 = x86_bsr v34 ; bin: 48 0f bd fa + [-,%rdi,%rflags] v37, v38 = x86_bsr v34 ; bin: 48 0f bd fa ; 32-bit wide bsr [-,%r11] v40 = iconst.i32 0x1234 ; asm: bsrl %r11d, %ecx - [-,%rcx,%eflags] v41, v42 = x86_bsr v40 ; bin: 41 0f bd cb + [-,%rcx,%rflags] v41, v42 = x86_bsr v40 ; bin: 41 0f bd cb [-,%rdx] v44 = iconst.i32 0x5678 ; asm: bsrl %edx, %r12d - [-,%r12,%eflags] v45, v46 = x86_bsr v44 ; bin: 44 0f bd e2 + [-,%r12,%rflags] v45, v46 = x86_bsr v44 ; bin: 44 0f bd e2 ; asm: bsrl %edx, %esi - [-,%rsi,%eflags] v47, v48 = x86_bsr v44 ; bin: 0f bd f2 + [-,%rsi,%rflags] v47, v48 = x86_bsr v44 ; bin: 0f bd f2 ; 64-bit wide cmov diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 8f60cf20be..94db28d26e 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -221,11 +221,11 @@ ebb0: [-,%rdx] v307 = fcmp ule v11, v10 ; bin: 0f 2e d5 0f 96 c2 ; asm: ucomiss %xmm2, %xmm5 - [-,%eflags] v310 = ffcmp v10, v11 ; bin: 0f 2e ea + [-,%rflags] v310 = ffcmp v10, v11 ; bin: 0f 2e ea ; asm: ucomiss %xmm2, %xmm5 - [-,%eflags] v311 = ffcmp v11, v10 ; bin: 0f 2e d5 + [-,%rflags] v311 = ffcmp v11, v10 ; bin: 0f 2e d5 ; asm: ucomiss %xmm5, %xmm5 - [-,%eflags] v312 = ffcmp v10, v10 ; bin: 0f 2e ed + [-,%rflags] v312 = ffcmp v10, v10 ; bin: 0f 2e ed return } @@ -436,11 +436,11 @@ ebb0: [-,%rdx] v307 = fcmp ule v11, v10 ; bin: 66 0f 2e d5 0f 96 c2 ; asm: ucomisd %xmm2, %xmm5 - [-,%eflags] v310 = ffcmp v10, v11 ; bin: 66 0f 2e ea + [-,%rflags] v310 = ffcmp v10, v11 ; bin: 66 0f 2e ea ; asm: ucomisd %xmm2, %xmm5 - [-,%eflags] v311 = ffcmp v11, v10 ; bin: 66 0f 2e d5 + [-,%rflags] v311 = ffcmp v11, v10 ; bin: 66 0f 2e d5 ; asm: ucomisd %xmm5, %xmm5 - [-,%eflags] v312 = ffcmp v10, v10 ; bin: 66 0f 2e ed + [-,%rflags] v312 = ffcmp v10, v10 ; bin: 66 0f 2e ed return } @@ -448,7 +448,7 @@ ebb0: function %cpuflags_float(f32 [%xmm0]) { ebb0(v0: f32 [%xmm0]): ; asm: ucomiss %xmm0, %xmm0 - [-,%eflags] v1 = ffcmp v0, v0 ; bin: 0f 2e c0 + [-,%rflags] v1 = ffcmp v0, v0 ; bin: 0f 2e c0 jump ebb1 diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 56301607cc..771bb19af2 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -473,9 +473,9 @@ ebb0: ebb1: ; asm: cmpl %esi, %ecx - [-,%eflags] v10 = ifcmp v1, v2 ; bin: 39 f1 + [-,%rflags] v10 = ifcmp v1, v2 ; bin: 39 f1 ; asm: cmpl %ecx, %esi - [-,%eflags] v11 = ifcmp v2, v1 ; bin: 39 ce + [-,%rflags] v11 = ifcmp v2, v1 ; bin: 39 ce ; asm: je ebb1 brif eq v11, ebb1 ; bin: 74 fa @@ -543,19 +543,19 @@ ebb1: ; Stack check. ; asm: cmpl %esp, %ecx - [-,%eflags] v40 = ifcmp_sp v1 ; bin: 39 e1 + [-,%rflags] v40 = ifcmp_sp v1 ; bin: 39 e1 ; asm: cmpl %esp, %esi - [-,%eflags] v41 = ifcmp_sp v2 ; bin: 39 e6 + [-,%rflags] v41 = ifcmp_sp v2 ; bin: 39 e6 ; asm: cmpl $-100, %ecx - [-,%eflags] v42 = ifcmp_imm v1, -100 ; bin: 83 f9 9c + [-,%rflags] v42 = ifcmp_imm v1, -100 ; bin: 83 f9 9c ; asm: cmpl $100, %esi - [-,%eflags] v43 = ifcmp_imm v2, 100 ; bin: 83 fe 64 + [-,%rflags] v43 = ifcmp_imm v2, 100 ; bin: 83 fe 64 ; asm: cmpl $-10000, %ecx - [-,%eflags] v44 = ifcmp_imm v1, -10000 ; bin: 81 f9 ffffd8f0 + [-,%rflags] v44 = ifcmp_imm v1, -10000 ; bin: 81 f9 ffffd8f0 ; asm: cmpl $10000, %esi - [-,%eflags] v45 = ifcmp_imm v2, 10000 ; bin: 81 fe 00002710 + [-,%rflags] v45 = ifcmp_imm v2, 10000 ; bin: 81 fe 00002710 return } diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index 69bfeb6c36..b362b7eb4a 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -235,11 +235,11 @@ ebb0: [-,%rdx] v307 = fcmp ule v11, v10 ; bin: 44 0f 2e d5 0f 96 c2 ; asm: ucomiss %xmm10, %xmm5 - [-,%eflags] v310 = ffcmp v10, v11 ; bin: 41 0f 2e ea + [-,%rflags] v310 = ffcmp v10, v11 ; bin: 41 0f 2e ea ; asm: ucomiss %xmm10, %xmm5 - [-,%eflags] v311 = ffcmp v11, v10 ; bin: 44 0f 2e d5 + [-,%rflags] v311 = ffcmp v11, v10 ; bin: 44 0f 2e d5 ; asm: ucomiss %xmm5, %xmm5 - [-,%eflags] v312 = ffcmp v10, v10 ; bin: 0f 2e ed + [-,%rflags] v312 = ffcmp v10, v10 ; bin: 0f 2e ed return } @@ -470,11 +470,11 @@ ebb0: [-,%rdx] v307 = fcmp ule v11, v10 ; bin: 66 44 0f 2e d5 0f 96 c2 ; asm: ucomisd %xmm10, %xmm5 - [-,%eflags] v310 = ffcmp v10, v11 ; bin: 66 41 0f 2e ea + [-,%rflags] v310 = ffcmp v10, v11 ; bin: 66 41 0f 2e ea ; asm: ucomisd %xmm10, %xmm5 - [-,%eflags] v311 = ffcmp v11, v10 ; bin: 66 44 0f 2e d5 + [-,%rflags] v311 = ffcmp v11, v10 ; bin: 66 44 0f 2e d5 ; asm: ucomisd %xmm5, %xmm5 - [-,%eflags] v312 = ffcmp v10, v10 ; bin: 66 0f 2e ed + [-,%rflags] v312 = ffcmp v10, v10 ; bin: 66 0f 2e ed return } @@ -482,7 +482,7 @@ ebb0: function %cpuflags_float(f32 [%xmm0]) { ebb0(v0: f32 [%xmm0]): ; asm: ucomiss %xmm0, %xmm0 - [-,%eflags] v1 = ffcmp v0, v0 ; bin: 0f 2e c0 + [-,%rflags] v1 = ffcmp v0, v0 ; bin: 0f 2e c0 jump ebb1 diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 5741f18041..3c35bd2a0c 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -584,9 +584,9 @@ ebb0: ebb1: ; asm: cmpq %r10, %rcx - [-,%eflags] v10 = ifcmp v1, v2 ; bin: 4c 39 d1 + [-,%rflags] v10 = ifcmp v1, v2 ; bin: 4c 39 d1 ; asm: cmpq %rcx, %r10 - [-,%eflags] v11 = ifcmp v2, v1 ; bin: 49 39 ca + [-,%rflags] v11 = ifcmp v2, v1 ; bin: 49 39 ca ; asm: je ebb1 brif eq v11, ebb1 ; bin: 74 f8 @@ -654,19 +654,19 @@ ebb1: ; Stack check. ; asm: cmpq %rsp, %rcx - [-,%eflags] v40 = ifcmp_sp v1 ; bin: 48 39 e1 + [-,%rflags] v40 = ifcmp_sp v1 ; bin: 48 39 e1 ; asm: cmpq %rsp, %r10 - [-,%eflags] v41 = ifcmp_sp v2 ; bin: 49 39 e2 + [-,%rflags] v41 = ifcmp_sp v2 ; bin: 49 39 e2 ; asm: cmpq $-100, %rcx - [-,%eflags] v522 = ifcmp_imm v1, -100 ; bin: 48 83 f9 9c + [-,%rflags] v522 = ifcmp_imm v1, -100 ; bin: 48 83 f9 9c ; asm: cmpq $100, %r10 - [-,%eflags] v523 = ifcmp_imm v2, 100 ; bin: 49 83 fa 64 + [-,%rflags] v523 = ifcmp_imm v2, 100 ; bin: 49 83 fa 64 ; asm: cmpq $-10000, %rcx - [-,%eflags] v524 = ifcmp_imm v1, -10000 ; bin: 48 81 f9 ffffd8f0 + [-,%rflags] v524 = ifcmp_imm v1, -10000 ; bin: 48 81 f9 ffffd8f0 ; asm: cmpq $10000, %r10 - [-,%eflags] v525 = ifcmp_imm v2, 10000 ; bin: 49 81 fa 00002710 + [-,%rflags] v525 = ifcmp_imm v2, 10000 ; bin: 49 81 fa 00002710 return @@ -1055,19 +1055,19 @@ ebb0: regfill v1, ss1 -> %rcx ; bin: 8b 8c 24 00000408 ; asm: cmpl %esi, %ecx - [-,%eflags] v520 = ifcmp v1, v2 ; bin: 39 f1 + [-,%rflags] v520 = ifcmp v1, v2 ; bin: 39 f1 ; asm: cmpl %r10d, %esi - [-,%eflags] v521 = ifcmp v2, v3 ; bin: 44 39 d6 + [-,%rflags] v521 = ifcmp v2, v3 ; bin: 44 39 d6 ; asm: cmpl $-100, %ecx - [-,%eflags] v522 = ifcmp_imm v1, -100 ; bin: 83 f9 9c + [-,%rflags] v522 = ifcmp_imm v1, -100 ; bin: 83 f9 9c ; asm: cmpl $100, %r10d - [-,%eflags] v523 = ifcmp_imm v3, 100 ; bin: 41 83 fa 64 + [-,%rflags] v523 = ifcmp_imm v3, 100 ; bin: 41 83 fa 64 ; asm: cmpl $-10000, %ecx - [-,%eflags] v524 = ifcmp_imm v1, -10000 ; bin: 81 f9 ffffd8f0 + [-,%rflags] v524 = ifcmp_imm v1, -10000 ; bin: 81 f9 ffffd8f0 ; asm: cmpl $10000, %r10d - [-,%eflags] v525 = ifcmp_imm v3, 10000 ; bin: 41 81 fa 00002710 + [-,%rflags] v525 = ifcmp_imm v3, 10000 ; bin: 41 81 fa 00002710 ; asm: shll $2, %esi [-,%rsi] v526 = ishl_imm v2, 2 ; bin: c1 e6 02 diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 9d015a25bb..5c37677f14 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -281,7 +281,7 @@ trap = TailRecipe( # Macro: conditional jump over a ud2. trapif = EncRecipe( - 'trapif', IntCondTrap, size=4, ins=FLAG.eflags, outs=(), + 'trapif', IntCondTrap, size=4, ins=FLAG.rflags, outs=(), clobbers_flags=False, emit=''' // Jump over a 2-byte ud2. @@ -293,7 +293,7 @@ trapif = EncRecipe( ''') trapff = EncRecipe( - 'trapff', FloatCondTrap, size=4, ins=FLAG.eflags, outs=(), + 'trapff', FloatCondTrap, size=4, ins=FLAG.rflags, outs=(), clobbers_flags=False, instp=floatccs(FloatCondTrap), emit=''' @@ -985,7 +985,7 @@ jmpd = TailRecipe( ''') brib = TailRecipe( - 'brib', BranchInt, size=1, ins=FLAG.eflags, outs=(), + 'brib', BranchInt, size=1, ins=FLAG.rflags, outs=(), branch_range=8, clobbers_flags=False, emit=''' @@ -994,7 +994,7 @@ brib = TailRecipe( ''') brid = TailRecipe( - 'brid', BranchInt, size=4, ins=FLAG.eflags, outs=(), + 'brid', BranchInt, size=4, ins=FLAG.rflags, outs=(), branch_range=32, clobbers_flags=False, emit=''' @@ -1003,7 +1003,7 @@ brid = TailRecipe( ''') brfb = TailRecipe( - 'brfb', BranchFloat, size=1, ins=FLAG.eflags, outs=(), + 'brfb', BranchFloat, size=1, ins=FLAG.rflags, outs=(), branch_range=8, clobbers_flags=False, instp=floatccs(BranchFloat), @@ -1013,7 +1013,7 @@ brfb = TailRecipe( ''') brfd = TailRecipe( - 'brfd', BranchFloat, size=4, ins=FLAG.eflags, outs=(), + 'brfd', BranchFloat, size=4, ins=FLAG.rflags, outs=(), branch_range=32, clobbers_flags=False, instp=floatccs(BranchFloat), @@ -1033,7 +1033,7 @@ brfd = TailRecipe( # seti = TailRecipe( - 'seti', IntCond, size=1, ins=FLAG.eflags, outs=GPR, + 'seti', IntCond, size=1, ins=FLAG.rflags, outs=GPR, requires_prefix=True, clobbers_flags=False, emit=''' @@ -1041,7 +1041,7 @@ seti = TailRecipe( modrm_r_bits(out_reg0, bits, sink); ''') seti_abcd = TailRecipe( - 'seti_abcd', IntCond, size=1, ins=FLAG.eflags, outs=ABCD, + 'seti_abcd', IntCond, size=1, ins=FLAG.rflags, outs=ABCD, when_prefixed=seti, clobbers_flags=False, emit=''' @@ -1050,7 +1050,7 @@ seti_abcd = TailRecipe( ''') setf = TailRecipe( - 'setf', FloatCond, size=1, ins=FLAG.eflags, outs=GPR, + 'setf', FloatCond, size=1, ins=FLAG.rflags, outs=GPR, requires_prefix=True, clobbers_flags=False, emit=''' @@ -1058,7 +1058,7 @@ setf = TailRecipe( modrm_r_bits(out_reg0, bits, sink); ''') setf_abcd = TailRecipe( - 'setf_abcd', FloatCond, size=1, ins=FLAG.eflags, outs=ABCD, + 'setf_abcd', FloatCond, size=1, ins=FLAG.rflags, outs=ABCD, when_prefixed=setf, clobbers_flags=False, emit=''' @@ -1072,7 +1072,7 @@ setf_abcd = TailRecipe( # 1 byte, modrm(r,r), is after the opcode # cmov = TailRecipe( - 'cmov', IntSelect, size=1, ins=(FLAG.eflags, GPR, GPR), outs=2, + 'cmov', IntSelect, size=1, ins=(FLAG.rflags, GPR, GPR), outs=2, requires_prefix=False, clobbers_flags=False, emit=''' @@ -1084,7 +1084,7 @@ cmov = TailRecipe( # Bit scan forwards and reverse # bsf_and_bsr = TailRecipe( - 'bsf_and_bsr', Unary, size=1, ins=GPR, outs=(GPR, FLAG.eflags), + 'bsf_and_bsr', Unary, size=1, ins=GPR, outs=(GPR, FLAG.rflags), requires_prefix=False, clobbers_flags=True, emit=''' @@ -1098,7 +1098,7 @@ bsf_and_bsr = TailRecipe( # XX /r, MR form. Compare two GPR registers and set flags. rcmp = TailRecipe( - 'rcmp', Binary, size=1, ins=(GPR, GPR), outs=FLAG.eflags, + 'rcmp', Binary, size=1, ins=(GPR, GPR), outs=FLAG.rflags, emit=''' PUT_OP(bits, rex2(in_reg0, in_reg1), sink); modrm_rr(in_reg0, in_reg1, sink); @@ -1106,7 +1106,7 @@ rcmp = TailRecipe( # XX /r, RM form. Compare two FPR registers and set flags. fcmp = TailRecipe( - 'fcmp', Binary, size=1, ins=(FPR, FPR), outs=FLAG.eflags, + 'fcmp', Binary, size=1, ins=(FPR, FPR), outs=FLAG.rflags, emit=''' PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_rr(in_reg1, in_reg0, sink); @@ -1114,7 +1114,7 @@ fcmp = TailRecipe( # XX /n, MI form with imm8. rcmpib = TailRecipe( - 'rcmpib', BinaryImm, size=2, ins=GPR, outs=FLAG.eflags, + 'rcmpib', BinaryImm, size=2, ins=GPR, outs=FLAG.rflags, instp=IsSignedInt(BinaryImm.imm, 8), emit=''' PUT_OP(bits, rex1(in_reg0), sink); @@ -1125,7 +1125,7 @@ rcmpib = TailRecipe( # XX /n, MI form with imm32. rcmpid = TailRecipe( - 'rcmpid', BinaryImm, size=5, ins=GPR, outs=FLAG.eflags, + 'rcmpid', BinaryImm, size=5, ins=GPR, outs=FLAG.rflags, instp=IsSignedInt(BinaryImm.imm, 32), emit=''' PUT_OP(bits, rex1(in_reg0), sink); @@ -1136,7 +1136,7 @@ rcmpid = TailRecipe( # Same as rcmp, but second operand is the stack pointer. rcmp_sp = TailRecipe( - 'rcmp_sp', Unary, size=1, ins=GPR, outs=FLAG.eflags, + 'rcmp_sp', Unary, size=1, ins=GPR, outs=FLAG.rflags, emit=''' PUT_OP(bits, rex2(in_reg0, RU::rsp.into()), sink); modrm_rr(in_reg0, RU::rsp.into(), sink); @@ -1302,7 +1302,7 @@ icscc = TailRecipe( # Same thing for floating point. # -# The ucomiss/ucomisd instructions set the EFLAGS bits CF/PF/CF like this: +# The ucomiss/ucomisd instructions set the FLAGS bits CF/PF/CF like this: # # ZPC OSA # UN 111 000 diff --git a/lib/cretonne/meta/isa/intel/registers.py b/lib/cretonne/meta/isa/intel/registers.py index 354001a102..a4832640cd 100644 --- a/lib/cretonne/meta/isa/intel/registers.py +++ b/lib/cretonne/meta/isa/intel/registers.py @@ -43,7 +43,7 @@ FlagRegs = RegBank( 'Flag registers', units=1, pressure_tracking=False, - names=['eflags']) + names=['rflags']) GPR = RegClass(IntRegs) # Certain types of deref encodings cannot be used with all registers. diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 2d045a0882..ddd3405e18 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -257,7 +257,7 @@ fn icc2opc(cond: IntCC) -> u16 { /// Get the low 4 bits of an opcode for a floating point condition code. /// -/// The ucomiss/ucomisd instructions set the EFLAGS bits CF/PF/CF like this: +/// The ucomiss/ucomisd instructions set the FLAGS bits CF/PF/CF like this: /// /// ZPC OSA /// UN 111 000 From a661a8a9bb77cad40a8d86e224fd5089470aa3a3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 26 Mar 2018 21:54:36 -0700 Subject: [PATCH 1650/3084] Factor out common ways to call `encode` from a dfg or func. --- lib/cretonne/src/ir/dfg.rs | 8 +++++++- lib/cretonne/src/ir/function.rs | 9 ++++++++- lib/cretonne/src/legalizer/mod.rs | 8 ++------ lib/cretonne/src/verifier/mod.rs | 6 +----- 4 files changed, 18 insertions(+), 13 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 344483a6d9..d23606ee13 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -1,7 +1,7 @@ //! Data flow graph tracking Instructions, Values, and EBBs. use entity::{PrimaryMap, EntityMap}; -use isa::TargetIsa; +use isa::{TargetIsa, Encoding, Legalize}; use ir; use ir::builder::ReplaceBuilder; use ir::extfunc::ExtFuncData; @@ -645,6 +645,12 @@ impl DataFlowGraph { self.value_type(self.first_result(inst)) } } + + /// Wrapper around `TargetIsa::encode` for encoding an existing instruction + /// in the `DataFlowGraph`. + pub fn encode(&self, inst: Inst, isa: &TargetIsa) -> Result { + isa.encode(&self, &self[inst], self.ctrl_typevar(inst)) + } } /// Allow immutable access to instructions via indexing. diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 91ee7eca75..8ee332526f 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -10,7 +10,7 @@ use ir::{ExternalName, CallConv, Signature, DataFlowGraph, Layout}; use ir::{InstEncodings, ValueLocations, JumpTables, StackSlots, EbbOffsets, SourceLocs}; use ir::{Ebb, JumpTableData, JumpTable, StackSlotData, StackSlot, SigRef, ExtFuncData, FuncRef, GlobalVarData, GlobalVar, HeapData, Heap}; -use isa::{TargetIsa, EncInfo}; +use isa::{TargetIsa, EncInfo, Legalize}; use std::fmt; use write::write_function; @@ -176,6 +176,13 @@ impl Function { iter: self.layout.ebb_insts(ebb), } } + + /// Wrapper around `DataFlowGraph::encode` which assigns `inst` the resulting encoding. + pub fn update_encoding(&mut self, inst: ir::Inst, isa: &TargetIsa) -> Result<(), Legalize> { + self.dfg.encode(inst, isa).map( + |e| { self.encodings[inst] = e; }, + ) + } } /// Wrapper type capable of displaying a `Function` with correct ISA annotations. diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index ae40c42581..68f24ee0da 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -72,12 +72,8 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is split::simplify_branch_arguments(&mut pos.func.dfg, inst); } - match isa.encode( - &pos.func.dfg, - &pos.func.dfg[inst], - pos.func.dfg.ctrl_typevar(inst), - ) { - Ok(encoding) => pos.func.encodings[inst] = encoding, + match pos.func.update_encoding(inst, isa) { + Ok(()) => {} Err(action) => { // We should transform the instruction into legal equivalents. let changed = action(inst, pos.func, cfg, isa); diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index b19510a485..8b6eb5e972 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -1058,11 +1058,7 @@ impl<'a> Verifier<'a> { if let Some(text) = needs_enc { // This instruction needs an encoding, so generate an error. // Provide the ISA default encoding as a hint. - match isa.encode( - &self.func.dfg, - &self.func.dfg[inst], - self.func.dfg.ctrl_typevar(inst), - ) { + match self.func.dfg.encode(inst, isa) { Ok(enc) => { return err!( inst, From 3b0a9b9ecfdbf4a39806dbfc554826c3e3383c77 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 26 Mar 2018 21:58:46 -0700 Subject: [PATCH 1651/3084] Remove an unused argument. --- lib/cretonne/src/context.rs | 2 +- lib/cretonne/src/simple_gvn.rs | 4 +--- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index c6018dcd9e..8358f21545 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -191,7 +191,7 @@ impl Context { /// Perform simple GVN on the function. pub fn simple_gvn<'a, FOI: Into>>(&mut self, fisa: FOI) -> CtonResult { - do_simple_gvn(&mut self.func, &mut self.cfg, &mut self.domtree); + do_simple_gvn(&mut self.func, &mut self.domtree); self.verify_if(fisa) } diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index dd0fb3b1aa..1cd82e4b2c 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -1,7 +1,6 @@ //! A simple GVN pass. use cursor::{Cursor, FuncCursor}; -use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; use ir::{InstructionData, Function, Inst, Opcode, Type}; use scoped_hash_map::ScopedHashMap; @@ -17,9 +16,8 @@ fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { /// Perform simple GVN on `func`. /// -pub fn do_simple_gvn(func: &mut Function, cfg: &mut ControlFlowGraph, domtree: &mut DominatorTree) { +pub fn do_simple_gvn(func: &mut Function, domtree: &mut DominatorTree) { let _tt = timing::gvn(); - debug_assert!(cfg.is_valid()); debug_assert!(domtree.is_valid()); let mut visible_values: ScopedHashMap<(InstructionData, Type), Inst> = ScopedHashMap::new(); From 79f02e42dd98a09a1e2cd6e05b7a97c1c273fa15 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 27 Mar 2018 11:21:37 -0700 Subject: [PATCH 1652/3084] Use movss/movsd rather than movd/movq for floating-point loads and stores. While there may be CPUs that have a domain crossing penalty here, this also helps the generated code look more like the code produced by other compilers. --- .../filetests/isa/intel/binary32-float.cton | 144 ++++++++-------- .../filetests/isa/intel/binary64-float.cton | 160 +++++++++--------- lib/cretonne/meta/isa/intel/encodings.py | 40 ++--- 3 files changed, 172 insertions(+), 172 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 94db28d26e..95e132cf6e 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -147,48 +147,48 @@ ebb0: ; Load/Store - ; asm: movd (%ecx), %xmm5 - [-,%xmm5] v100 = load.f32 v0 ; bin: 66 0f 6e 29 - ; asm: movd (%esi), %xmm2 - [-,%xmm2] v101 = load.f32 v1 ; bin: 66 0f 6e 16 - ; asm: movd 50(%ecx), %xmm5 - [-,%xmm5] v110 = load.f32 v0+50 ; bin: 66 0f 6e 69 32 - ; asm: movd -50(%esi), %xmm2 - [-,%xmm2] v111 = load.f32 v1-50 ; bin: 66 0f 6e 56 ce - ; asm: movd 10000(%ecx), %xmm5 - [-,%xmm5] v120 = load.f32 v0+10000 ; bin: 66 0f 6e a9 00002710 - ; asm: movd -10000(%esi), %xmm2 - [-,%xmm2] v121 = load.f32 v1-10000 ; bin: 66 0f 6e 96 ffffd8f0 + ; asm: movss (%ecx), %xmm5 + [-,%xmm5] v100 = load.f32 v0 ; bin: f3 0f 10 29 + ; asm: movss (%esi), %xmm2 + [-,%xmm2] v101 = load.f32 v1 ; bin: f3 0f 10 16 + ; asm: movss 50(%ecx), %xmm5 + [-,%xmm5] v110 = load.f32 v0+50 ; bin: f3 0f 10 69 32 + ; asm: movss -50(%esi), %xmm2 + [-,%xmm2] v111 = load.f32 v1-50 ; bin: f3 0f 10 56 ce + ; asm: movss 10000(%ecx), %xmm5 + [-,%xmm5] v120 = load.f32 v0+10000 ; bin: f3 0f 10 a9 00002710 + ; asm: movss -10000(%esi), %xmm2 + [-,%xmm2] v121 = load.f32 v1-10000 ; bin: f3 0f 10 96 ffffd8f0 - ; asm: movd %xmm5, (%ecx) - [-] store.f32 v100, v0 ; bin: 66 0f 7e 29 - ; asm: movd %xmm2, (%esi) - [-] store.f32 v101, v1 ; bin: 66 0f 7e 16 - ; asm: movd %xmm5, 50(%ecx) - [-] store.f32 v100, v0+50 ; bin: 66 0f 7e 69 32 - ; asm: movd %xmm2, -50(%esi) - [-] store.f32 v101, v1-50 ; bin: 66 0f 7e 56 ce - ; asm: movd %xmm5, 10000(%ecx) - [-] store.f32 v100, v0+10000 ; bin: 66 0f 7e a9 00002710 - ; asm: movd %xmm2, -10000(%esi) - [-] store.f32 v101, v1-10000 ; bin: 66 0f 7e 96 ffffd8f0 + ; asm: movss %xmm5, (%ecx) + [-] store.f32 v100, v0 ; bin: f3 0f 11 29 + ; asm: movss %xmm2, (%esi) + [-] store.f32 v101, v1 ; bin: f3 0f 11 16 + ; asm: movss %xmm5, 50(%ecx) + [-] store.f32 v100, v0+50 ; bin: f3 0f 11 69 32 + ; asm: movss %xmm2, -50(%esi) + [-] store.f32 v101, v1-50 ; bin: f3 0f 11 56 ce + ; asm: movss %xmm5, 10000(%ecx) + [-] store.f32 v100, v0+10000 ; bin: f3 0f 11 a9 00002710 + ; asm: movss %xmm2, -10000(%esi) + [-] store.f32 v101, v1-10000 ; bin: f3 0f 11 96 ffffd8f0 ; Spill / Fill. - ; asm: movd %xmm5, 1032(%esp) - [-,ss1] v200 = spill v100 ; bin: 66 0f 7e ac 24 00000408 - ; asm: movd %xmm2, 1032(%esp) - [-,ss1] v201 = spill v101 ; bin: 66 0f 7e 94 24 00000408 + ; asm: movss %xmm5, 1032(%esp) + [-,ss1] v200 = spill v100 ; bin: f3 0f 11 ac 24 00000408 + ; asm: movss %xmm2, 1032(%esp) + [-,ss1] v201 = spill v101 ; bin: f3 0f 11 94 24 00000408 - ; asm: movd 1032(%esp), %xmm5 - [-,%xmm5] v210 = fill v200 ; bin: 66 0f 6e ac 24 00000408 - ; asm: movd 1032(%esp), %xmm2 - [-,%xmm2] v211 = fill v201 ; bin: 66 0f 6e 94 24 00000408 + ; asm: movss 1032(%esp), %xmm5 + [-,%xmm5] v210 = fill v200 ; bin: f3 0f 10 ac 24 00000408 + ; asm: movss 1032(%esp), %xmm2 + [-,%xmm2] v211 = fill v201 ; bin: f3 0f 10 94 24 00000408 - ; asm: movd %xmm5, 1032(%rsp) - regspill v100, %xmm5 -> ss1 ; bin: 66 0f 7e ac 24 00000408 - ; asm: movd 1032(%rsp), %xmm5 - regfill v100, ss1 -> %xmm5 ; bin: 66 0f 6e ac 24 00000408 + ; asm: movss %xmm5, 1032(%rsp) + regspill v100, %xmm5 -> ss1 ; bin: f3 0f 11 ac 24 00000408 + ; asm: movss 1032(%rsp), %xmm5 + regfill v100, ss1 -> %xmm5 ; bin: f3 0f 10 ac 24 00000408 ; Comparisons. ; @@ -362,48 +362,48 @@ ebb0: ; Load/Store - ; asm: movq (%ecx), %xmm5 - [-,%xmm5] v100 = load.f64 v0 ; bin: f3 0f 7e 29 - ; asm: movq (%esi), %xmm2 - [-,%xmm2] v101 = load.f64 v1 ; bin: f3 0f 7e 16 - ; asm: movq 50(%ecx), %xmm5 - [-,%xmm5] v110 = load.f64 v0+50 ; bin: f3 0f 7e 69 32 - ; asm: movq -50(%esi), %xmm2 - [-,%xmm2] v111 = load.f64 v1-50 ; bin: f3 0f 7e 56 ce - ; asm: movq 10000(%ecx), %xmm5 - [-,%xmm5] v120 = load.f64 v0+10000 ; bin: f3 0f 7e a9 00002710 - ; asm: movq -10000(%esi), %xmm2 - [-,%xmm2] v121 = load.f64 v1-10000 ; bin: f3 0f 7e 96 ffffd8f0 + ; asm: movsd (%ecx), %xmm5 + [-,%xmm5] v100 = load.f64 v0 ; bin: f2 0f 10 29 + ; asm: movsd (%esi), %xmm2 + [-,%xmm2] v101 = load.f64 v1 ; bin: f2 0f 10 16 + ; asm: movsd 50(%ecx), %xmm5 + [-,%xmm5] v110 = load.f64 v0+50 ; bin: f2 0f 10 69 32 + ; asm: movsd -50(%esi), %xmm2 + [-,%xmm2] v111 = load.f64 v1-50 ; bin: f2 0f 10 56 ce + ; asm: movsd 10000(%ecx), %xmm5 + [-,%xmm5] v120 = load.f64 v0+10000 ; bin: f2 0f 10 a9 00002710 + ; asm: movsd -10000(%esi), %xmm2 + [-,%xmm2] v121 = load.f64 v1-10000 ; bin: f2 0f 10 96 ffffd8f0 - ; asm: movq %xmm5, (%ecx) - [-] store.f64 v100, v0 ; bin: 66 0f d6 29 - ; asm: movq %xmm2, (%esi) - [-] store.f64 v101, v1 ; bin: 66 0f d6 16 - ; asm: movq %xmm5, 50(%ecx) - [-] store.f64 v100, v0+50 ; bin: 66 0f d6 69 32 - ; asm: movq %xmm2, -50(%esi) - [-] store.f64 v101, v1-50 ; bin: 66 0f d6 56 ce - ; asm: movq %xmm5, 10000(%ecx) - [-] store.f64 v100, v0+10000 ; bin: 66 0f d6 a9 00002710 - ; asm: movq %xmm2, -10000(%esi) - [-] store.f64 v101, v1-10000 ; bin: 66 0f d6 96 ffffd8f0 + ; asm: movsd %xmm5, (%ecx) + [-] store.f64 v100, v0 ; bin: f2 0f 11 29 + ; asm: movsd %xmm2, (%esi) + [-] store.f64 v101, v1 ; bin: f2 0f 11 16 + ; asm: movsd %xmm5, 50(%ecx) + [-] store.f64 v100, v0+50 ; bin: f2 0f 11 69 32 + ; asm: movsd %xmm2, -50(%esi) + [-] store.f64 v101, v1-50 ; bin: f2 0f 11 56 ce + ; asm: movsd %xmm5, 10000(%ecx) + [-] store.f64 v100, v0+10000 ; bin: f2 0f 11 a9 00002710 + ; asm: movsd %xmm2, -10000(%esi) + [-] store.f64 v101, v1-10000 ; bin: f2 0f 11 96 ffffd8f0 ; Spill / Fill. - ; asm: movq %xmm5, 1032(%esp) - [-,ss1] v200 = spill v100 ; bin: 66 0f d6 ac 24 00000408 - ; asm: movq %xmm2, 1032(%esp) - [-,ss1] v201 = spill v101 ; bin: 66 0f d6 94 24 00000408 + ; asm: movsd %xmm5, 1032(%esp) + [-,ss1] v200 = spill v100 ; bin: f2 0f 11 ac 24 00000408 + ; asm: movsd %xmm2, 1032(%esp) + [-,ss1] v201 = spill v101 ; bin: f2 0f 11 94 24 00000408 - ; asm: movq 1032(%esp), %xmm5 - [-,%xmm5] v210 = fill v200 ; bin: f3 0f 7e ac 24 00000408 - ; asm: movq 1032(%esp), %xmm2 - [-,%xmm2] v211 = fill v201 ; bin: f3 0f 7e 94 24 00000408 + ; asm: movsd 1032(%esp), %xmm5 + [-,%xmm5] v210 = fill v200 ; bin: f2 0f 10 ac 24 00000408 + ; asm: movsd 1032(%esp), %xmm2 + [-,%xmm2] v211 = fill v201 ; bin: f2 0f 10 94 24 00000408 - ; asm: movq %xmm5, 1032(%rsp) - regspill v100, %xmm5 -> ss1 ; bin: 66 0f d6 ac 24 00000408 - ; asm: movq 1032(%rsp), %xmm5 - regfill v100, ss1 -> %xmm5 ; bin: f3 0f 7e ac 24 00000408 + ; asm: movsd %xmm5, 1032(%rsp) + regspill v100, %xmm5 -> ss1 ; bin: f2 0f 11 ac 24 00000408 + ; asm: movsd 1032(%rsp), %xmm5 + regfill v100, ss1 -> %xmm5 ; bin: f2 0f 10 ac 24 00000408 ; Comparisons. ; diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index b362b7eb4a..47db5b197f 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -157,52 +157,52 @@ ebb0: ; Load/Store - ; asm: movd (%r14), %xmm5 - [-,%xmm5] v100 = load.f32 v3 ; bin: 66 41 0f 6e 2e - ; asm: movd (%rax), %xmm10 - [-,%xmm10] v101 = load.f32 v2 ; bin: 66 44 0f 6e 10 - ; asm: movd 50(%r14), %xmm5 - [-,%xmm5] v110 = load.f32 v3+50 ; bin: 66 41 0f 6e 6e 32 - ; asm: movd -50(%rax), %xmm10 - [-,%xmm10] v111 = load.f32 v2-50 ; bin: 66 44 0f 6e 50 ce - ; asm: movd 10000(%r14), %xmm5 - [-,%xmm5] v120 = load.f32 v3+10000 ; bin: 66 41 0f 6e ae 00002710 - ; asm: movd -10000(%rax), %xmm10 - [-,%xmm10] v121 = load.f32 v2-10000 ; bin: 66 44 0f 6e 90 ffffd8f0 + ; asm: movss (%r14), %xmm5 + [-,%xmm5] v100 = load.f32 v3 ; bin: f3 41 0f 10 2e + ; asm: movss (%rax), %xmm10 + [-,%xmm10] v101 = load.f32 v2 ; bin: f3 44 0f 10 10 + ; asm: movss 50(%r14), %xmm5 + [-,%xmm5] v110 = load.f32 v3+50 ; bin: f3 41 0f 10 6e 32 + ; asm: movss -50(%rax), %xmm10 + [-,%xmm10] v111 = load.f32 v2-50 ; bin: f3 44 0f 10 50 ce + ; asm: movss 10000(%r14), %xmm5 + [-,%xmm5] v120 = load.f32 v3+10000 ; bin: f3 41 0f 10 ae 00002710 + ; asm: movss -10000(%rax), %xmm10 + [-,%xmm10] v121 = load.f32 v2-10000 ; bin: f3 44 0f 10 90 ffffd8f0 - ; asm: movd %xmm5, (%r14) - [-] store.f32 v100, v3 ; bin: 66 41 0f 7e 2e - ; asm: movd %xmm10, (%rax) - [-] store.f32 v101, v2 ; bin: 66 44 0f 7e 10 - ; asm: movd %xmm5, (%r13) - [-] store.f32 v100, v4 ; bin: 66 41 0f 7e 6d 00 - ; asm: movd %xmm10, (%r13) - [-] store.f32 v101, v4 ; bin: 66 45 0f 7e 55 00 - ; asm: movd %xmm5, 50(%r14) - [-] store.f32 v100, v3+50 ; bin: 66 41 0f 7e 6e 32 - ; asm: movd %xmm10, -50(%rax) - [-] store.f32 v101, v2-50 ; bin: 66 44 0f 7e 50 ce - ; asm: movd %xmm5, 10000(%r14) - [-] store.f32 v100, v3+10000 ; bin: 66 41 0f 7e ae 00002710 - ; asm: movd %xmm10, -10000(%rax) - [-] store.f32 v101, v2-10000 ; bin: 66 44 0f 7e 90 ffffd8f0 + ; asm: movss %xmm5, (%r14) + [-] store.f32 v100, v3 ; bin: f3 41 0f 11 2e + ; asm: movss %xmm10, (%rax) + [-] store.f32 v101, v2 ; bin: f3 44 0f 11 10 + ; asm: movss %xmm5, (%r13) + [-] store.f32 v100, v4 ; bin: f3 41 0f 11 6d 00 + ; asm: movss %xmm10, (%r13) + [-] store.f32 v101, v4 ; bin: f3 45 0f 11 55 00 + ; asm: movss %xmm5, 50(%r14) + [-] store.f32 v100, v3+50 ; bin: f3 41 0f 11 6e 32 + ; asm: movss %xmm10, -50(%rax) + [-] store.f32 v101, v2-50 ; bin: f3 44 0f 11 50 ce + ; asm: movss %xmm5, 10000(%r14) + [-] store.f32 v100, v3+10000 ; bin: f3 41 0f 11 ae 00002710 + ; asm: movss %xmm10, -10000(%rax) + [-] store.f32 v101, v2-10000 ; bin: f3 44 0f 11 90 ffffd8f0 ; Spill / Fill. - ; asm: movd %xmm5, 1032(%rsp) - [-,ss1] v200 = spill v100 ; bin: 66 0f 7e ac 24 00000408 - ; asm: movd %xmm10, 1032(%rsp) - [-,ss1] v201 = spill v101 ; bin: 66 44 0f 7e 94 24 00000408 + ; asm: movss %xmm5, 1032(%rsp) + [-,ss1] v200 = spill v100 ; bin: f3 0f 11 ac 24 00000408 + ; asm: movss %xmm10, 1032(%rsp) + [-,ss1] v201 = spill v101 ; bin: f3 44 0f 11 94 24 00000408 - ; asm: movd 1032(%rsp), %xmm5 - [-,%xmm5] v210 = fill v200 ; bin: 66 0f 6e ac 24 00000408 - ; asm: movd 1032(%rsp), %xmm10 - [-,%xmm10] v211 = fill v201 ; bin: 66 44 0f 6e 94 24 00000408 + ; asm: movss 1032(%rsp), %xmm5 + [-,%xmm5] v210 = fill v200 ; bin: f3 0f 10 ac 24 00000408 + ; asm: movss 1032(%rsp), %xmm10 + [-,%xmm10] v211 = fill v201 ; bin: f3 44 0f 10 94 24 00000408 - ; asm: movd %xmm5, 1032(%rsp) - regspill v100, %xmm5 -> ss1 ; bin: 66 0f 7e ac 24 00000408 - ; asm: movd 1032(%rsp), %xmm5 - regfill v100, ss1 -> %xmm5 ; bin: 66 0f 6e ac 24 00000408 + ; asm: movss %xmm5, 1032(%rsp) + regspill v100, %xmm5 -> ss1 ; bin: f3 0f 11 ac 24 00000408 + ; asm: movss 1032(%rsp), %xmm5 + regfill v100, ss1 -> %xmm5 ; bin: f3 0f 10 ac 24 00000408 ; Comparisons. ; @@ -392,52 +392,52 @@ ebb0: ; Load/Store - ; asm: movq (%r14), %xmm5 - [-,%xmm5] v100 = load.f64 v3 ; bin: f3 41 0f 7e 2e - ; asm: movq (%rax), %xmm10 - [-,%xmm10] v101 = load.f64 v2 ; bin: f3 44 0f 7e 10 - ; asm: movq 50(%r14), %xmm5 - [-,%xmm5] v110 = load.f64 v3+50 ; bin: f3 41 0f 7e 6e 32 - ; asm: movq -50(%rax), %xmm10 - [-,%xmm10] v111 = load.f64 v2-50 ; bin: f3 44 0f 7e 50 ce - ; asm: movq 10000(%r14), %xmm5 - [-,%xmm5] v120 = load.f64 v3+10000 ; bin: f3 41 0f 7e ae 00002710 - ; asm: movq -10000(%rax), %xmm10 - [-,%xmm10] v121 = load.f64 v2-10000 ; bin: f3 44 0f 7e 90 ffffd8f0 + ; asm: movsd (%r14), %xmm5 + [-,%xmm5] v100 = load.f64 v3 ; bin: f2 41 0f 10 2e + ; asm: movsd (%rax), %xmm10 + [-,%xmm10] v101 = load.f64 v2 ; bin: f2 44 0f 10 10 + ; asm: movsd 50(%r14), %xmm5 + [-,%xmm5] v110 = load.f64 v3+50 ; bin: f2 41 0f 10 6e 32 + ; asm: movsd -50(%rax), %xmm10 + [-,%xmm10] v111 = load.f64 v2-50 ; bin: f2 44 0f 10 50 ce + ; asm: movsd 10000(%r14), %xmm5 + [-,%xmm5] v120 = load.f64 v3+10000 ; bin: f2 41 0f 10 ae 00002710 + ; asm: movsd -10000(%rax), %xmm10 + [-,%xmm10] v121 = load.f64 v2-10000 ; bin: f2 44 0f 10 90 ffffd8f0 - ; asm: movq %xmm5, (%r14) - [-] store.f64 v100, v3 ; bin: 66 41 0f d6 2e - ; asm: movq %xmm10, (%rax) - [-] store.f64 v101, v2 ; bin: 66 44 0f d6 10 - ; asm: movq %xmm5, (%r13) - [-] store.f64 v100, v4 ; bin: 66 41 0f d6 6d 00 - ; asm: movq %xmm10, (%r13) - [-] store.f64 v101, v4 ; bin: 66 45 0f d6 55 00 - ; asm: movq %xmm5, 50(%r14) - [-] store.f64 v100, v3+50 ; bin: 66 41 0f d6 6e 32 - ; asm: movq %xmm10, -50(%rax) - [-] store.f64 v101, v2-50 ; bin: 66 44 0f d6 50 ce - ; asm: movq %xmm5, 10000(%r14) - [-] store.f64 v100, v3+10000 ; bin: 66 41 0f d6 ae 00002710 - ; asm: movq %xmm10, -10000(%rax) - [-] store.f64 v101, v2-10000 ; bin: 66 44 0f d6 90 ffffd8f0 + ; asm: movsd %xmm5, (%r14) + [-] store.f64 v100, v3 ; bin: f2 41 0f 11 2e + ; asm: movsd %xmm10, (%rax) + [-] store.f64 v101, v2 ; bin: f2 44 0f 11 10 + ; asm: movsd %xmm5, (%r13) + [-] store.f64 v100, v4 ; bin: f2 41 0f 11 6d 00 + ; asm: movsd %xmm10, (%r13) + [-] store.f64 v101, v4 ; bin: f2 45 0f 11 55 00 + ; asm: movsd %xmm5, 50(%r14) + [-] store.f64 v100, v3+50 ; bin: f2 41 0f 11 6e 32 + ; asm: movsd %xmm10, -50(%rax) + [-] store.f64 v101, v2-50 ; bin: f2 44 0f 11 50 ce + ; asm: movsd %xmm5, 10000(%r14) + [-] store.f64 v100, v3+10000 ; bin: f2 41 0f 11 ae 00002710 + ; asm: movsd %xmm10, -10000(%rax) + [-] store.f64 v101, v2-10000 ; bin: f2 44 0f 11 90 ffffd8f0 ; Spill / Fill. - ; asm: movq %xmm5, 1032(%rsp) - [-,ss1] v200 = spill v100 ; bin: 66 0f d6 ac 24 00000408 - ; asm: movq %xmm10, 1032(%rsp) - [-,ss1] v201 = spill v101 ; bin: 66 44 0f d6 94 24 00000408 + ; asm: movsd %xmm5, 1032(%rsp) + [-,ss1] v200 = spill v100 ; bin: f2 0f 11 ac 24 00000408 + ; asm: movsd %xmm10, 1032(%rsp) + [-,ss1] v201 = spill v101 ; bin: f2 44 0f 11 94 24 00000408 - ; asm: movq 1032(%rsp), %xmm5 - [-,%xmm5] v210 = fill v200 ; bin: f3 0f 7e ac 24 00000408 - ; asm: movq 1032(%rsp), %xmm10 - [-,%xmm10] v211 = fill v201 ; bin: f3 44 0f 7e 94 24 00000408 + ; asm: movsd 1032(%rsp), %xmm5 + [-,%xmm5] v210 = fill v200 ; bin: f2 0f 10 ac 24 00000408 + ; asm: movsd 1032(%rsp), %xmm10 + [-,%xmm10] v211 = fill v201 ; bin: f2 44 0f 10 94 24 00000408 - ; asm: movq %xmm5, 1032(%rsp) - regspill v100, %xmm5 -> ss1 ; bin: 66 0f d6 ac 24 00000408 - ; asm: movq 1032(%rsp), %xmm5 - regfill v100, ss1 -> %xmm5 ; bin: f3 0f 7e ac 24 00000408 + ; asm: movsd %xmm5, 1032(%rsp) + regspill v100, %xmm5 -> ss1 ; bin: f2 0f 11 ac 24 00000408 + ; asm: movsd 1032(%rsp), %xmm5 + regfill v100, ss1 -> %xmm5 ; bin: f2 0f 10 ac 24 00000408 ; Comparisons. ; diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index afb544cb9a..fca2ce8699 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -259,31 +259,31 @@ X86_64.enc(base.adjust_sp_imm, *r.adjustsp32.rex(0x81, w=1)) # Float loads and stores. # -enc_both(base.load.f32.any, r.fld, 0x66, 0x0f, 0x6e) -enc_both(base.load.f32.any, r.fldDisp8, 0x66, 0x0f, 0x6e) -enc_both(base.load.f32.any, r.fldDisp32, 0x66, 0x0f, 0x6e) +enc_both(base.load.f32.any, r.fld, 0xf3, 0x0f, 0x10) +enc_both(base.load.f32.any, r.fldDisp8, 0xf3, 0x0f, 0x10) +enc_both(base.load.f32.any, r.fldDisp32, 0xf3, 0x0f, 0x10) -enc_both(base.load.f64.any, r.fld, 0xf3, 0x0f, 0x7e) -enc_both(base.load.f64.any, r.fldDisp8, 0xf3, 0x0f, 0x7e) -enc_both(base.load.f64.any, r.fldDisp32, 0xf3, 0x0f, 0x7e) +enc_both(base.load.f64.any, r.fld, 0xf2, 0x0f, 0x10) +enc_both(base.load.f64.any, r.fldDisp8, 0xf2, 0x0f, 0x10) +enc_both(base.load.f64.any, r.fldDisp32, 0xf2, 0x0f, 0x10) -enc_both(base.store.f32.any, r.fst, 0x66, 0x0f, 0x7e) -enc_both(base.store.f32.any, r.fstDisp8, 0x66, 0x0f, 0x7e) -enc_both(base.store.f32.any, r.fstDisp32, 0x66, 0x0f, 0x7e) +enc_both(base.store.f32.any, r.fst, 0xf3, 0x0f, 0x11) +enc_both(base.store.f32.any, r.fstDisp8, 0xf3, 0x0f, 0x11) +enc_both(base.store.f32.any, r.fstDisp32, 0xf3, 0x0f, 0x11) -enc_both(base.store.f64.any, r.fst, 0x66, 0x0f, 0xd6) -enc_both(base.store.f64.any, r.fstDisp8, 0x66, 0x0f, 0xd6) -enc_both(base.store.f64.any, r.fstDisp32, 0x66, 0x0f, 0xd6) +enc_both(base.store.f64.any, r.fst, 0xf2, 0x0f, 0x11) +enc_both(base.store.f64.any, r.fstDisp8, 0xf2, 0x0f, 0x11) +enc_both(base.store.f64.any, r.fstDisp32, 0xf2, 0x0f, 0x11) -enc_both(base.fill.f32, r.ffillSib32, 0x66, 0x0f, 0x6e) -enc_both(base.regfill.f32, r.fregfill32, 0x66, 0x0f, 0x6e) -enc_both(base.fill.f64, r.ffillSib32, 0xf3, 0x0f, 0x7e) -enc_both(base.regfill.f64, r.fregfill32, 0xf3, 0x0f, 0x7e) +enc_both(base.fill.f32, r.ffillSib32, 0xf3, 0x0f, 0x10) +enc_both(base.regfill.f32, r.fregfill32, 0xf3, 0x0f, 0x10) +enc_both(base.fill.f64, r.ffillSib32, 0xf2, 0x0f, 0x10) +enc_both(base.regfill.f64, r.fregfill32, 0xf2, 0x0f, 0x10) -enc_both(base.spill.f32, r.fspillSib32, 0x66, 0x0f, 0x7e) -enc_both(base.regspill.f32, r.fregspill32, 0x66, 0x0f, 0x7e) -enc_both(base.spill.f64, r.fspillSib32, 0x66, 0x0f, 0xd6) -enc_both(base.regspill.f64, r.fregspill32, 0x66, 0x0f, 0xd6) +enc_both(base.spill.f32, r.fspillSib32, 0xf3, 0x0f, 0x11) +enc_both(base.regspill.f32, r.fregspill32, 0xf3, 0x0f, 0x11) +enc_both(base.spill.f64, r.fspillSib32, 0xf2, 0x0f, 0x11) +enc_both(base.regspill.f64, r.fregspill32, 0xf2, 0x0f, 0x11) # # Function addresses. From 0b5bb313cb1c2f552141390ae8e7a649a56a957d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 27 Mar 2018 14:12:06 -0700 Subject: [PATCH 1653/3084] Mark CondCode's functions #[must_use]. It's easy to forget whether they mutate the value in place or return a new value. Marking them #[must_use] will catch cases where they are used incorrectly. --- lib/cretonne/src/ir/condcodes.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/cretonne/src/ir/condcodes.rs b/lib/cretonne/src/ir/condcodes.rs index 11c438b10a..61967a5871 100644 --- a/lib/cretonne/src/ir/condcodes.rs +++ b/lib/cretonne/src/ir/condcodes.rs @@ -13,12 +13,14 @@ pub trait CondCode: Copy { /// /// The inverse condition code produces the opposite result for all comparisons. /// That is, `cmp CC, x, y` is true if and only if `cmp CC.inverse(), x, y` is false. + #[must_use] fn inverse(self) -> Self; /// Get the reversed condition code for `self`. /// /// The reversed condition code produces the same result as swapping `x` and `y` in the /// comparison. That is, `cmp CC, x, y` is the same as `cmp CC.reverse(), y, x`. + #[must_use] fn reverse(self) -> Self; } From 7cce4be96a6021c38a50cfd26d285f69a8a7b99c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 27 Mar 2018 21:11:24 -0700 Subject: [PATCH 1654/3084] Fix an incorrect index in "cton-util wasm -s". --- cranelift/src/wasm.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index efd93df77f..5f986ee5f2 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -170,7 +170,7 @@ fn handle_module( println!( "Function #{} bytecode size: {} bytes", func_index, - dummy_environ.func_bytecode_sizes[func_index] + dummy_environ.func_bytecode_sizes[def_index] ); } } else { From 7b51edd2858db121f582db6475b0f89403e54942 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Mar 2018 08:58:58 -0700 Subject: [PATCH 1655/3084] Fix spelling in a comment to be consistent with the code. --- lib/cretonne/src/regalloc/coalescing.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index 2437fd1a03..f23d55433e 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -907,7 +907,7 @@ impl VirtualCopies { self.filter.clear(); } - /// Initialise virtual copies from the (interfering) values in a union-find virtual register + /// Initialize virtual copies from the (interfering) values in a union-find virtual register /// that is going to be broken up and reassembled iteratively. /// /// The values are assumed to be in domtree pre-order. From 8d560cf8ba38b57a42460431d47ffdb08cf95162 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Mar 2018 11:50:19 -0700 Subject: [PATCH 1656/3084] Fix Rust syntax in generated code. This code is not currently emitted, though it will be when there are more legalization rules. --- lib/cretonne/meta/gen_legalizer.py | 13 +++++++------ lib/cretonne/meta/test_gen_legalizer.py | 4 ++-- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 04c48ade51..9051337238 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -103,18 +103,19 @@ def emit_runtime_typecheck(check, fmt, type_sets): base_exp = build_derived_expr(tv.base) if (tv.derived_func == TypeVar.LANEOF): - return "{}.map(|t: Type| -> t.lane_type())".format(base_exp) + return "{}.map(|t: ir::Type| t.lane_type())".format(base_exp) elif (tv.derived_func == TypeVar.ASBOOL): - return "{}.map(|t: Type| -> t.as_bool())".format(base_exp) + return "{}.map(|t: ir::Type| t.as_bool())".format(base_exp) elif (tv.derived_func == TypeVar.HALFWIDTH): - return "{}.and_then(|t: Type| -> t.half_width())".format(base_exp) + return "{}.and_then(|t: ir::Type| t.half_width())".format(base_exp) elif (tv.derived_func == TypeVar.DOUBLEWIDTH): - return "{}.and_then(|t: Type| -> t.double_width())"\ + return "{}.and_then(|t: ir::Type| t.double_width())"\ .format(base_exp) elif (tv.derived_func == TypeVar.HALFVECTOR): - return "{}.and_then(|t: Type| -> t.half_vector())".format(base_exp) + return "{}.and_then(|t: ir::Type| t.half_vector())"\ + .format(base_exp) elif (tv.derived_func == TypeVar.DOUBLEVECTOR): - return "{}.and_then(|t: Type| -> t.by(2))".format(base_exp) + return "{}.and_then(|t: ir::Type| t.by(2))".format(base_exp) else: assert False, "Unknown derived function {}".format(tv.derived_func) diff --git a/lib/cretonne/meta/test_gen_legalizer.py b/lib/cretonne/meta/test_gen_legalizer.py index 793555a42c..3882bd0bf9 100644 --- a/lib/cretonne/meta/test_gen_legalizer.py +++ b/lib/cretonne/meta/test_gen_legalizer.py @@ -148,9 +148,9 @@ class TestRuntimeChecks(TestCase): self.v5 << vselect(self.v1, self.v3, self.v4), ) x = XForm(r, r) - tv2_exp = 'Some({}).map(|t: Type| -> t.as_bool())'\ + tv2_exp = 'Some({}).map(|t: ir::Type| t.as_bool())'\ .format(self.v2.get_typevar().name) - tv3_exp = 'Some({}).map(|t: Type| -> t.as_bool())'\ + tv3_exp = 'Some({}).map(|t: ir::Type| t.as_bool())'\ .format(self.v3.get_typevar().name) self.check_yo_check( From c3f044ff46c500969fd46926c441e723c78034c6 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Mar 2018 13:44:16 -0700 Subject: [PATCH 1657/3084] Note that the "widen" legalization group is not yet implemented. --- lib/cretonne/meta/base/legalize.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index ecc8d5a436..c2392269df 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -41,6 +41,8 @@ widen = XFormGroup('widen', """ The transformations in the 'widen' group work by expressing instructions in terms of larger types. + + This group is not yet implemented. """) expand = XFormGroup('expand', """ From 23ab07b54eddcfc7ea5b623e5eb7f5aacb160bf3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Mar 2018 14:11:16 -0700 Subject: [PATCH 1658/3084] Support legalizing bconst instructions on x86. --- cranelift/filetests/isa/intel/binary32.cton | 3 +++ cranelift/filetests/isa/intel/binary64.cton | 5 +++++ lib/cretonne/meta/isa/intel/encodings.py | 3 +++ lib/cretonne/meta/isa/intel/recipes.py | 14 +++++++++++++- 4 files changed, 24 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 771bb19af2..04f492c354 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -25,6 +25,9 @@ ebb0: ; asm: movl $2, %esi [-,%rsi] v2 = iconst.i32 2 ; bin: be 00000002 + ; asm: movb $1, %cl + [-,%rcx] v9007 = bconst.b1 true ; bin: b9 00000001 + ; Integer Register-Register Operations. ; asm: addl %esi, %ecx diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 3c35bd2a0c..f7aab2f8f4 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -38,6 +38,11 @@ ebb0: ; asm: movq $0xffffffff88001122, %r14 # 32-bit sign-extended constant. [-,%r14] v5 = iconst.i64 0xffff_ffff_8800_1122 ; bin: 49 c7 c6 88001122 + ; asm: movb $1, %cl + [-,%rcx] v9007 = bconst.b1 true ; bin: b9 00000001 + ; asm: movb $1, %sil + [-,%r10] v9008 = bconst.b1 true ; bin: 41 ba 00000001 + ; Integer Register-Register Operations. ; asm: addq %rsi, %rcx diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index fca2ce8699..72ef7a362b 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -155,6 +155,9 @@ X86_64.enc(base.iconst.i64, *r.uid.rex(0xc7, rrr=0, w=1)) # Finally, the 0xb8 opcode takes an 8-byte immediate with a REX.W prefix. X86_64.enc(base.iconst.i64, *r.puiq.rex(0xb8, w=1)) +# bool constants. +enc_both(base.bconst.b1, r.puid_bool, 0xb8) + # Shifts and rotates. # Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit # and 16-bit shifts would need explicit masking. diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 5c37677f14..2406ef6ade 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -5,7 +5,8 @@ from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt, IsEqual, Or from cdsl.registers import RegClass -from base.formats import Unary, UnaryImm, Binary, BinaryImm, MultiAry, NullAry +from base.formats import Unary, UnaryImm, UnaryBool, Binary, BinaryImm +from base.formats import MultiAry, NullAry from base.formats import Trap, Call, IndirectCall, Store, Load from base.formats import IntCompare, FloatCompare, IntCond, FloatCond from base.formats import IntSelect, IntCondTrap, FloatCondTrap @@ -506,6 +507,17 @@ puid = TailRecipe( sink.put4(imm as u32); ''') +# XX+rd id unary with bool immediate. Note no recipe predicate. +puid_bool = TailRecipe( + 'puid_bool', UnaryBool, size=4, ins=(), outs=GPR, + emit=''' + // The destination register is encoded in the low bits of the opcode. + // No ModR/M. + PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); + let imm: u32 = if imm.into() { 1 } else { 0 }; + sink.put4(imm); + ''') + # XX+rd iq unary with 64-bit immediate. puiq = TailRecipe( 'puiq', UnaryImm, size=8, ins=(), outs=GPR, From db2be8ee01613d18ed73f8a96c506277b8766d95 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Mar 2018 20:09:49 -0700 Subject: [PATCH 1659/3084] Verifier: Diagnose an instruction using its own result values. --- .../verifier/defs_dominates_uses.cton | 16 ++++++ lib/cretonne/src/verifier/mod.rs | 56 ++++++++++++++++--- 2 files changed, 65 insertions(+), 7 deletions(-) create mode 100644 cranelift/filetests/verifier/defs_dominates_uses.cton diff --git a/cranelift/filetests/verifier/defs_dominates_uses.cton b/cranelift/filetests/verifier/defs_dominates_uses.cton new file mode 100644 index 0000000000..1bc3819726 --- /dev/null +++ b/cranelift/filetests/verifier/defs_dominates_uses.cton @@ -0,0 +1,16 @@ +test verifier + +; Test verification that uses properly dominate defs. + +function %non_dominating(i32) -> i32 native { +ebb0(v0: i32): + v1 = iadd.i32 v2, v0 ; error: uses value from non-dominating + v2 = iadd.i32 v1, v0 + return v2 +} + +function %inst_uses_its_own_values(i32) -> i32 native { +ebb0(v0: i32): + v1 = iadd.i32 v1, v0 ; error: uses value from itself + return v1 +} diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 8b6eb5e972..e916094bbb 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -268,7 +268,7 @@ impl<'a> Verifier<'a> { use ir::instructions::InstructionData::*; for &arg in self.func.dfg.inst_args(inst) { - self.verify_value(inst, arg)?; + self.verify_inst_arg(inst, arg)?; // All used values must be attached to something. let original = self.func.dfg.resolve_aliases(arg); @@ -278,7 +278,7 @@ impl<'a> Verifier<'a> { } for &res in self.func.dfg.inst_results(inst) { - self.verify_value(inst, res)?; + self.verify_inst_result(inst, res)?; } match self.func.dfg[inst] { @@ -446,8 +446,16 @@ impl<'a> Verifier<'a> { fn verify_value(&self, loc_inst: Inst, v: Value) -> Result { let dfg = &self.func.dfg; if !dfg.value_is_valid(v) { - return err!(loc_inst, "invalid value reference {}", v); + err!(loc_inst, "invalid value reference {}", v) + } else { + Ok(()) } + } + + fn verify_inst_arg(&self, loc_inst: Inst, v: Value) -> Result { + self.verify_value(loc_inst, v)?; + + let dfg = &self.func.dfg; let loc_ebb = self.func.layout.pp_ebb(loc_inst); let is_reachable = self.expected_domtree.is_reachable(loc_ebb); @@ -473,14 +481,23 @@ impl<'a> Verifier<'a> { ); } // Defining instruction dominates the instruction that uses the value. - if is_reachable && - !self.expected_domtree.dominates( + if is_reachable { + if !self.expected_domtree.dominates( def_inst, loc_inst, &self.func.layout, ) - { - return err!(loc_inst, "uses value from non-dominating {}", def_inst); + { + return err!(loc_inst, "uses value from non-dominating {}", def_inst); + } + if def_inst == loc_inst { + return err!( + loc_inst, + "uses value from itself {}, {}", + def_inst, + loc_inst + ); + } } } ValueDef::Param(ebb, _) => { @@ -512,6 +529,31 @@ impl<'a> Verifier<'a> { Ok(()) } + fn verify_inst_result(&self, loc_inst: Inst, v: Value) -> Result { + self.verify_value(loc_inst, v)?; + + match self.func.dfg.value_def(v) { + ValueDef::Result(def_inst, _) => { + if def_inst != loc_inst { + err!( + loc_inst, + "instruction result {} is not defined by the instruction", + v + ) + } else { + Ok(()) + } + } + ValueDef::Param(_, _) => { + err!( + loc_inst, + "instruction result {} is not defined by the instruction", + v + ) + } + } + } + fn domtree_integrity(&self, domtree: &DominatorTree) -> Result { // We consider two `DominatorTree`s to be equal if they return the same immediate // dominator for each EBB. Therefore the current domtree is valid if it matches the freshly From e5ec7242cc0d42af77fde124d4f269c720a37533 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Mar 2018 13:20:11 -0700 Subject: [PATCH 1660/3084] Fix handling of value aliases, and re-enable LICM. Value aliases aren't instructions, so they don't have a location in the CFG, so it's not meaningful to query whether a value alias is defined within a loop. --- cranelift/filetests/licm/reject.cton | 81 ++++++++++++++++++++++++++++ lib/cretonne/src/context.rs | 2 - lib/cretonne/src/licm.rs | 45 +++++++++++----- 3 files changed, 113 insertions(+), 15 deletions(-) create mode 100644 cranelift/filetests/licm/reject.cton diff --git a/cranelift/filetests/licm/reject.cton b/cranelift/filetests/licm/reject.cton new file mode 100644 index 0000000000..052ae2b6f2 --- /dev/null +++ b/cranelift/filetests/licm/reject.cton @@ -0,0 +1,81 @@ +test licm + +function %other_side_effects(i32) -> i32 { + +ebb0(v0: i32): + jump ebb1(v0) + +ebb1(v1: i32): + regmove.i32 v0, %10 -> %20 +; check: ebb1(v1: i32): +; check: regmove.i32 v0, %10 -> %20 + v2 = iconst.i32 1 + brz v1, ebb2(v1) + v5 = isub v1, v2 + jump ebb1(v5) + +ebb2(v6: i32): + return v6 + +} + +function %cpu_flags(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + jump ebb1(v0, v1) + +ebb1(v2: i32, v3: i32): + v4 = ifcmp.i32 v0, v1 + v5 = selectif.i32 eq v4, v2, v3 +; check: ebb1(v2: i32, v3: i32): +; check: ifcmp.i32 v0, v1 +; check: v5 = selectif.i32 eq v4, v2, v3 + v8 = iconst.i32 1 + brz v1, ebb2(v1) + v9 = isub v1, v8 + v10 = iadd v1, v8 + jump ebb1(v9, v10) + +ebb2(v6: i32): + return v6 +} + +function %spill(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 = spill.i32 v0 + jump ebb1(v0, v1) + +ebb1(v3: i32, v4: i32): + v5 = spill.i32 v1 + v6 = fill.i32 v2 + v7 = fill.i32 v5 +; check: ebb1(v3: i32, v4: i32): +; check: v5 = spill.i32 v1 +; check: v6 = fill.i32 v2 +; check: v7 = fill v5 + brz v1, ebb2(v1) + v9 = isub v1, v4 + jump ebb1(v9, v3) + +ebb2(v10: i32): + return v10 +} + +function %non_invariant_aliases(i32) -> i32 { + +ebb0(v0: i32): + jump ebb1(v0) + +ebb1(v1: i32): + v8 -> v1 + v9 -> v1 + v2 = iadd v8, v9 +; check: ebb1(v1: i32): +; check: v2 = iadd v8, v9 + brz v1, ebb2(v1) + v5 = isub v1, v2 + jump ebb1(v5) + +ebb2(v6: i32): + return v6 + +} diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 8358f21545..fba2cd2cc3 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -92,10 +92,8 @@ impl Context { self.legalize(isa)?; if isa.flags().opt_level() == OptLevel::Best { self.compute_domtree(); - /* TODO: Re-enable LICM. self.compute_loop_analysis(); self.licm(isa)?; - */ self.simple_gvn(isa)?; } self.compute_domtree(); diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 7887a39e4e..f0327296a1 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -1,7 +1,7 @@ //! A Loop Invariant Code Motion optimization pass use cursor::{Cursor, FuncCursor}; -use ir::{Function, Ebb, Inst, Value, Type, InstBuilder, Layout}; +use ir::{Function, Ebb, Inst, Value, Type, InstBuilder, Layout, Opcode, DataFlowGraph}; use flowgraph::ControlFlowGraph; use std::collections::HashSet; use dominator_tree::DominatorTree; @@ -27,10 +27,10 @@ pub fn do_licm( for lp in loop_analysis.loops() { // For each loop that we want to optimize we determine the set of loop-invariant // instructions - let invariant_inst = remove_loop_invariant_instructions(lp, func, cfg, loop_analysis); + let invariant_insts = remove_loop_invariant_instructions(lp, func, cfg, loop_analysis); // Then we create the loop's pre-header and fill it with the invariant instructions // Then we remove the invariant instructions from the loop body - if !invariant_inst.is_empty() { + if !invariant_insts.is_empty() { // If the loop has a natural pre-header we use it, otherwise we create it. let mut pos; match has_pre_header(&func.layout, cfg, domtree, loop_analysis.loop_header(lp)) { @@ -47,7 +47,7 @@ pub fn do_licm( }; // The last instruction of the pre-header is the termination instruction (usually // a jump) so we need to insert just before this. - for inst in invariant_inst { + for inst in invariant_insts { pos.insert_inst(inst); } } @@ -131,6 +131,29 @@ fn change_branch_jump_destination(inst: Inst, new_ebb: Ebb, func: &mut Function) } } +/// Test whether the given opcode is unsafe to even consider for LICM. +fn trivially_unsafe_for_licm(opcode: Opcode) -> bool { + opcode.can_load() || opcode.can_store() || opcode.is_call() || opcode.is_branch() || + opcode.is_terminator() || opcode.is_return() || + opcode.can_trap() || opcode.other_side_effects() || opcode.writes_cpu_flags() +} + +/// Test whether the given instruction is loop-invariant. +fn is_loop_invariant(inst: Inst, dfg: &DataFlowGraph, loop_values: &HashSet) -> bool { + if trivially_unsafe_for_licm(dfg[inst].opcode()) { + return false; + } + + let inst_args = dfg.inst_args(inst); + for arg in inst_args { + let arg = dfg.resolve_aliases(*arg); + if loop_values.contains(&arg) { + return false; + } + } + return true; +} + // Traverses a loop in reverse post-order from a header EBB and identify loop-invariant // instructions. These loop-invariant instructions are then removed from the code and returned // (in reverse post-order) for later use. @@ -141,7 +164,7 @@ fn remove_loop_invariant_instructions( loop_analysis: &LoopAnalysis, ) -> Vec { let mut loop_values: HashSet = HashSet::new(); - let mut invariant_inst: Vec = Vec::new(); + let mut invariant_insts: Vec = Vec::new(); let mut pos = FuncCursor::new(func); // We traverse the loop EBB in reverse post-order. for ebb in postorder_ebbs_loop(loop_analysis, cfg, lp).iter().rev() { @@ -151,15 +174,11 @@ fn remove_loop_invariant_instructions( } pos.goto_top(*ebb); #[cfg_attr(feature = "cargo-clippy", allow(block_in_if_condition_stmt))] - while let Some(inst) = pos.next_inst() { - if pos.func.dfg.has_results(inst) && - pos.func.dfg.inst_args(inst).into_iter().all(|arg| { - !loop_values.contains(arg) - }) - { + 'next_inst: while let Some(inst) = pos.next_inst() { + if is_loop_invariant(inst, &pos.func.dfg, &loop_values) { // If all the instruction's argument are defined outside the loop // then this instruction is loop-invariant - invariant_inst.push(inst); + invariant_insts.push(inst); // We remove it from the loop pos.remove_inst_and_step_back(); } else { @@ -171,7 +190,7 @@ fn remove_loop_invariant_instructions( } } } - invariant_inst + invariant_insts } /// Return ebbs from a loop in post-order, starting from an entry point in the block. From 592db1de7a2898be5fd3017db88378a0d427c327 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Mar 2018 13:28:59 -0700 Subject: [PATCH 1661/3084] Tighten up the parser's verification of value aliases. This prevents uses of undefined values from passsing through unnoticed, and ensures that all aliases are ultimately resolved, regardless of where they are defined. --- lib/cretonne/src/ir/dfg.rs | 70 ++++++++++++++++++++++++++++++++++---- lib/reader/src/parser.rs | 49 ++++++++++++++++++++++++++ 2 files changed, 112 insertions(+), 7 deletions(-) diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index d23606ee13..b8976b7267 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -121,8 +121,9 @@ impl DataFlowGraph { /// Resolve value aliases. /// -/// Find the original SSA value that `value` aliases. -fn resolve_aliases(values: &PrimaryMap, value: Value) -> Value { +/// Find the original SSA value that `value` aliases, or None if an +/// alias cycle is detected. +fn maybe_resolve_aliases(values: &PrimaryMap, value: Value) -> Option { let mut v = value; // Note that values may be empty here. @@ -130,10 +131,22 @@ fn resolve_aliases(values: &PrimaryMap, value: Value) -> Value if let ValueData::Alias { original, .. } = values[v] { v = original; } else { - return v; + return Some(v); } } - panic!("Value alias loop detected for {}", value); + + None +} + +/// Resolve value aliases. +/// +/// Find the original SSA value that `value` aliases. +fn resolve_aliases(values: &PrimaryMap, value: Value) -> Value { + if let Some(v) = maybe_resolve_aliases(values, value) { + v + } else { + panic!("Value alias loop detected for {}", value); + } } /// Handling values. @@ -238,6 +251,7 @@ impl DataFlowGraph { self.value_type(dest), ty ); + debug_assert_ne!(ty, types::VOID); self.values[dest] = ValueData::Alias { ty, original }; } @@ -282,6 +296,7 @@ impl DataFlowGraph { self.value_type(dest), ty ); + debug_assert_ne!(ty, types::VOID); self.values[dest] = ValueData::Alias { ty, original }; } @@ -865,8 +880,9 @@ impl DataFlowGraph { /// to create invalid values for index padding which may be reassigned later. #[cold] fn set_value_type_for_parser(&mut self, v: Value, t: Type) { - assert!( - self.value_type(v) == types::VOID, + assert_eq!( + self.value_type(v), + types::VOID, "this function is only for assigning types to previously invalid values" ); match self.values[v] { @@ -926,12 +942,38 @@ impl DataFlowGraph { /// aliases with specific values. #[cold] pub fn make_value_alias_for_parser(&mut self, src: Value, dest: Value) { - let ty = self.value_type(src); + assert_ne!(src, Value::reserved_value()); + assert_ne!(dest, Value::reserved_value()); + let ty = if self.values.is_valid(src) { + self.value_type(src) + } else { + // As a special case, if we can't resolve the aliasee yet, use VOID + // temporarily. It will be resolved later in parsing. + types::VOID + }; let data = ValueData::Alias { ty, original: src }; self.values[dest] = data; } + /// Compute the type of an alias. This is only for use in the parser. + /// Returns false if an alias cycle was encountered. + #[cold] + pub fn set_alias_type_for_parser(&mut self, v: Value) -> bool { + if let Some(resolved) = maybe_resolve_aliases(&self.values, v) { + let old_ty = self.value_type(v); + let new_ty = self.value_type(resolved); + if old_ty == types::VOID { + self.set_value_type_for_parser(v, new_ty); + } else { + assert_eq!(old_ty, new_ty); + } + true + } else { + false + } + } + /// Create an invalid value, to pad the index space. This is only for use by /// the parser to pad out the value index space. #[cold] @@ -942,6 +984,20 @@ impl DataFlowGraph { }; self.make_value(data); } + + /// Check if a value reference is valid, while being aware of aliases which + /// may be unresolved while parsing. + #[cold] + pub fn value_is_valid_for_parser(&self, v: Value) -> bool { + if !self.value_is_valid(v) { + return false; + } + if let ValueData::Alias { ty, .. } = self.values[v] { + ty != types::VOID + } else { + true + } + } } #[cfg(test)] diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 2e320b31a7..384de9381b 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -86,6 +86,9 @@ struct Context<'a> { function: Function, map: SourceMap, + // Aliases to resolve once value definitions are known. + aliases: Vec, + // Reference to the unique_isa for things like parsing ISA-specific instruction encoding // information. This is only `Some` if exactly one set of `isa` directives were found in the // prologue (it is valid to have directives for multiple different ISAs, but in that case we @@ -99,6 +102,7 @@ impl<'a> Context<'a> { function: f, map: SourceMap::new(), unique_isa, + aliases: Vec::new(), } } @@ -1341,6 +1345,30 @@ impl<'a> Parser<'a> { while self.token() != Some(Token::RBrace) { self.parse_extended_basic_block(ctx)?; } + + // Now that we've seen all defined values in the function, ensure that + // all references refer to a definition. + for ebb in &ctx.function.layout { + for inst in ctx.function.layout.ebb_insts(ebb) { + for value in ctx.function.dfg.inst_args(inst) { + if !ctx.map.contains_value(*value) { + return err!( + ctx.map.location(AnyEntity::Inst(inst)).unwrap(), + "undefined operand value {}", + value + ); + } + } + } + } + + for alias in &ctx.aliases { + if !ctx.function.dfg.set_alias_type_for_parser(*alias) { + let loc = ctx.map.location(AnyEntity::Value(*alias)).unwrap(); + return err!(loc, "alias cycle involving {}", alias); + } + } + Ok(()) } @@ -1616,6 +1644,7 @@ impl<'a> Parser<'a> { results[0], ); ctx.map.def_value(results[0], &self.loc)?; + ctx.aliases.push(results[0]); Ok(()) } @@ -1758,6 +1787,26 @@ impl<'a> Parser<'a> { let ctrl_src_value = inst_data .typevar_operand(&ctx.function.dfg.value_lists) .expect("Constraints <-> Format inconsistency"); + if !ctx.map.contains_value(ctrl_src_value) { + return err!( + self.loc, + "type variable required for polymorphic opcode, e.g. '{}.{}'; \ + can't infer from {} which is not yet defined", + opcode, + constraints.ctrl_typeset().unwrap().example(), + ctrl_src_value + ); + } + if !ctx.function.dfg.value_is_valid_for_parser(ctrl_src_value) { + return err!( + self.loc, + "type variable required for polymorphic opcode, e.g. '{}.{}'; \ + can't infer from {} which is not yet resolved", + opcode, + constraints.ctrl_typeset().unwrap().example(), + ctrl_src_value + ); + } ctx.function.dfg.value_type(ctrl_src_value) } else if constraints.is_polymorphic() { // This opcode does not support type inference, so the explicit type From a297465c25f0e415061ad24f52cc6240a55248eb Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Mar 2018 13:29:17 -0700 Subject: [PATCH 1662/3084] Tidy up comment formatting. Convert several normal comments to documentation comments, and make separator comments consistent with other files. --- lib/cretonne/src/divconst_magic_numbers.rs | 18 +++++----- lib/cretonne/src/preopt.rs | 39 ++++++++++------------ lib/cretonne/src/write.rs | 12 ++----- 3 files changed, 29 insertions(+), 40 deletions(-) diff --git a/lib/cretonne/src/divconst_magic_numbers.rs b/lib/cretonne/src/divconst_magic_numbers.rs index f189e23d1a..01b5066b28 100644 --- a/lib/cretonne/src/divconst_magic_numbers.rs +++ b/lib/cretonne/src/divconst_magic_numbers.rs @@ -1,17 +1,15 @@ //! Compute "magic numbers" for division-by-constants transformations. +//! +//! Math helpers for division by (non-power-of-2) constants. This is based +//! on the presentation in "Hacker's Delight" by Henry Warren, 2003. There +//! are four cases: {unsigned, signed} x {32 bit, 64 bit}. The word size +//! makes little difference, but the signed-vs-unsigned aspect has a large +//! effect. Therefore everything is presented in the order U32 U64 S32 S64 +//! so as to emphasise the similarity of the U32 and U64 cases and the S32 +//! and S64 cases. #![allow(non_snake_case)] -//---------------------------------------------------------------------- -// -// Math helpers for division by (non-power-of-2) constants. This is based -// on the presentation in "Hacker's Delight" by Henry Warren, 2003. There -// are four cases: {unsigned, signed} x {32 bit, 64 bit}. The word size -// makes little difference, but the signed-vs-unsigned aspect has a large -// effect. Therefore everything is presented in the order U32 U64 S32 S64 -// so as to emphasise the similarity of the U32 and U64 cases and the S32 -// and S64 cases. - // Structures to hold the "magic numbers" computed. #[derive(PartialEq, Debug)] diff --git a/lib/cretonne/src/preopt.rs b/lib/cretonne/src/preopt.rs index 698631316e..9adce51369 100644 --- a/lib/cretonne/src/preopt.rs +++ b/lib/cretonne/src/preopt.rs @@ -19,8 +19,8 @@ use timing; // Simple math helpers -// if `x` is a power of two, or the negation thereof, return the power along -// with a boolean that indicates whether `x` is negative. Else return None. +/// if `x` is a power of two, or the negation thereof, return the power along +/// with a boolean that indicates whether `x` is negative. Else return None. #[inline] fn isPowerOf2_S32(x: i32) -> Option<(bool, u32)> { // We have to special-case this because abs(x) isn't representable. @@ -34,7 +34,7 @@ fn isPowerOf2_S32(x: i32) -> Option<(bool, u32)> { None } -// Same comments as for isPowerOf2_S64 apply. +/// Same comments as for isPowerOf2_S64 apply. #[inline] fn isPowerOf2_S64(x: i64) -> Option<(bool, u32)> { // We have to special-case this because abs(x) isn't representable. @@ -60,9 +60,9 @@ enum DivRemByConstInfo { RemS64(Value, i64), } -// Possibly create a DivRemByConstInfo from the given components, by -// figuring out which, if any, of the 8 cases apply, and also taking care to -// sanity-check the immediate. +/// Possibly create a DivRemByConstInfo from the given components, by +/// figuring out which, if any, of the 8 cases apply, and also taking care to +/// sanity-check the immediate. fn package_up_divrem_info( argL: Value, argL_ty: Type, @@ -108,9 +108,9 @@ fn package_up_divrem_info( None } -// Examine `idata` to see if it is a div or rem by a constant, and if so -// return the operands, signedness, operation size and div-vs-rem-ness in a -// handy bundle. +/// Examine `idata` to see if it is a div or rem by a constant, and if so +/// return the operands, signedness, operation size and div-vs-rem-ness in a +/// handy bundle. fn get_div_info(inst: Inst, dfg: &DataFlowGraph) -> Option { let idata: &InstructionData = &dfg[inst]; @@ -152,12 +152,12 @@ fn get_div_info(inst: Inst, dfg: &DataFlowGraph) -> Option { None } -// Actually do the transformation given a bundle containing the relevant -// information. `divrem_info` describes a div or rem by a constant, that -// `pos` currently points at, and `inst` is the associated instruction. -// `inst` is replaced by a sequence of other operations that calculate the -// same result. Note that there are various `divrem_info` cases where we -// cannot do any transformation, in which case `inst` is left unchanged. +/// Actually do the transformation given a bundle containing the relevant +/// information. `divrem_info` describes a div or rem by a constant, that +/// `pos` currently points at, and `inst` is the associated instruction. +/// `inst` is replaced by a sequence of other operations that calculate the +/// same result. Note that there are various `divrem_info` cases where we +/// cannot do any transformation, in which case `inst` is left unchanged. fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCursor, inst: Inst) { let isRem = match *divrem_info { DivRemByConstInfo::DivU32(_, _) | @@ -478,8 +478,8 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso // // General pattern-match helpers. -// Find out if `value` actually resolves to a constant, and if so what its -// value is. +/// Find out if `value` actually resolves to a constant, and if so what its +/// value is. fn get_const(value: Value, dfg: &DataFlowGraph) -> Option { match dfg.value_def(value) { ValueDef::Result(definingInst, resultNo) => { @@ -496,10 +496,7 @@ fn get_const(value: Value, dfg: &DataFlowGraph) -> Option { } -//---------------------------------------------------------------------- -// -// The main pre-opt pass. - +/// The main pre-opt pass. pub fn do_preopt(func: &mut Function) { let _tt = timing::preopt(); let mut pos = FuncCursor::new(func); diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 50acbbf553..bbe8375b89 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -30,11 +30,9 @@ pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) - writeln!(w, "}}") } -// ====--------------------------------------------------------------------------------------====// +//---------------------------------------------------------------------- // // Function spec. -// -// ====--------------------------------------------------------------------------------------====// fn write_spec(w: &mut Write, func: &Function, regs: Option<&RegInfo>) -> Result { write!(w, "function {}{}", func.name, func.signature.display(regs)) @@ -90,11 +88,9 @@ fn write_preamble( Ok(any) } -// ====--------------------------------------------------------------------------------------====// +//---------------------------------------------------------------------- // // Basic blocks -// -// ====--------------------------------------------------------------------------------------====// pub fn write_arg(w: &mut Write, func: &Function, regs: Option<&RegInfo>, arg: Value) -> Result { write!(w, "{}: {}", arg, func.dfg.value_type(arg))?; @@ -158,11 +154,9 @@ pub fn write_ebb(w: &mut Write, func: &Function, isa: Option<&TargetIsa>, ebb: E } -// ====--------------------------------------------------------------------------------------====// +//---------------------------------------------------------------------- // // Instructions -// -// ====--------------------------------------------------------------------------------------====// // Should `inst` be printed with a type suffix? // From 57cd69d8b40f0e6184e6a7388e64e9a33601184f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Mar 2018 14:15:01 -0700 Subject: [PATCH 1663/3084] Say "IR" instead of "IL". While the specifics of these terms are debatable, "IR" generally isn't incorrect in this context, and is the more widely recognized term at this time. See also the discussion in #267. Fixes #267. --- README.rst | 2 +- cranelift/docs/compare-llvm.rst | 20 +++++++-------- cranelift/docs/cton_domain.py | 16 ++++++------ cranelift/docs/langref.rst | 34 ++++++++++--------------- cranelift/docs/testing.rst | 8 +++--- cranelift/src/cat.rs | 2 +- cranelift/src/compile.rs | 4 +-- cranelift/src/cton-util.rs | 6 ++--- cranelift/src/print_cfg.rs | 2 +- cranelift/src/wasm.rs | 2 +- lib/cretonne/README.md | 2 +- lib/cretonne/meta/base/formats.py | 2 +- lib/cretonne/meta/base/settings.py | 4 +-- lib/cretonne/meta/cdsl/ast.py | 2 +- lib/cretonne/src/binemit/memorysink.rs | 2 +- lib/cretonne/src/ir/entities.rs | 6 ++--- lib/cretonne/src/ir/function.rs | 2 +- lib/cretonne/src/ir/instructions.rs | 2 +- lib/cretonne/src/ir/layout.rs | 3 +-- lib/cretonne/src/ir/libcall.rs | 2 +- lib/cretonne/src/ir/mod.rs | 2 +- lib/cretonne/src/ir/progpoint.rs | 2 +- lib/cretonne/src/ir/sourceloc.rs | 2 +- lib/cretonne/src/legalizer/globalvar.rs | 2 +- lib/cretonne/src/regalloc/diversion.rs | 2 +- lib/cretonne/src/result.rs | 4 +-- lib/cretonne/src/timing.rs | 4 +-- lib/cretonne/src/write.rs | 7 +++-- lib/filetests/src/test_legalizer.rs | 2 +- lib/filetests/src/test_print_cfg.rs | 2 +- lib/filetests/src/test_verifier.rs | 2 +- lib/frontend/Cargo.toml | 2 +- lib/frontend/README.md | 4 +-- lib/frontend/src/frontend.rs | 24 ++++++++--------- lib/frontend/src/lib.rs | 10 ++++---- lib/frontend/src/ssa.rs | 2 +- lib/reader/Cargo.toml | 2 +- lib/reader/src/parser.rs | 2 +- lib/wasm/Cargo.toml | 2 +- lib/wasm/README.md | 3 +-- lib/wasm/src/code_translator.rs | 4 +-- lib/wasm/src/environ/spec.rs | 2 +- lib/wasm/src/func_translator.rs | 8 +++--- lib/wasm/src/lib.rs | 4 +-- lib/wasm/src/module_translator.rs | 2 +- 45 files changed, 106 insertions(+), 119 deletions(-) diff --git a/README.rst b/README.rst index 12b889f56d..41e6a9284d 100644 --- a/README.rst +++ b/README.rst @@ -3,7 +3,7 @@ Cretonne Code Generator ======================= Cretonne is a low-level retargetable code generator. It translates a `target-independent -intermediate language `_ into executable +intermediate representation `_ into executable machine code. *This is a work in progress that is not yet functional.* diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst index 0180f6cc92..4a622dbeda 100644 --- a/cranelift/docs/compare-llvm.rst +++ b/cranelift/docs/compare-llvm.rst @@ -16,8 +16,8 @@ highlighting some of the differences and similarities. Both projects: - Use an ISA-agnostic input language in order to mostly abstract away the differences between target instruction set architectures. - Depend extensively on SSA form. -- Have both textual and in-memory forms of their primary intermediate language. - (LLVM also has a binary bitcode format; Cretonne doesn't.) +- Have both textual and in-memory forms of their primary intermediate + representation. (LLVM also has a binary bitcode format; Cretonne doesn't.) - Can target multiple ISAs. - Can cross-compile by default without rebuilding the code generator. @@ -41,8 +41,8 @@ LLVM uses multiple intermediate representations as it translates a program to binary machine code: `LLVM IR `_ - This is the primary intermediate language which has textual, binary, and - in-memory representations. It serves two main purposes: + This is the primary intermediate representation which has textual, binary, and + in-memory forms. It serves two main purposes: - An ISA-agnostic, stable(ish) input language that front ends can generate easily. @@ -89,9 +89,9 @@ representation. Some target ISAs have a fast instruction selector that can translate simple code directly to MachineInstrs, bypassing SelectionDAG when possible. -:doc:`Cretonne ` uses a single intermediate language to cover these -levels of abstraction. This is possible in part because of Cretonne's smaller -scope. +:doc:`Cretonne ` uses a single intermediate representation to cover +these levels of abstraction. This is possible in part because of Cretonne's +smaller scope. - Cretonne does not provide assemblers and disassemblers, so it is not necessary to be able to represent every weird instruction in an ISA. Only @@ -102,7 +102,7 @@ scope. - SSA form is preserved throughout. After register allocation, each SSA value is annotated with an assigned ISA register or stack slot. -The Cretonne intermediate language is similar to LLVM IR, but at a slightly +The Cretonne intermediate representation is similar to LLVM IR, but at a slightly lower level of abstraction. Program structure @@ -112,12 +112,12 @@ In LLVM IR, the largest representable unit is the *module* which corresponds more or less to a C translation unit. It is a collection of functions and global variables that may contain references to external symbols too. -In Cretonne IL, the largest representable unit is the *function*. This is so +In Cretonne IR, the largest representable unit is the *function*. This is so that functions can easily be compiled in parallel without worrying about references to shared data structures. Cretonne does not have any inter-procedural optimizations like inlining. -An LLVM IR function is a graph of *basic blocks*. A Cretonne IL function is a +An LLVM IR function is a graph of *basic blocks*. A Cretonne IR function is a graph of *extended basic blocks* that may contain internal branch instructions. The main difference is that an LLVM conditional branch instruction has two target basic blocks---a true and a false edge. A Cretonne branch instruction diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index a03acf1160..2d6e45eeed 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- # -# Sphinx domain for documenting compiler intermediate languages. +# Sphinx domain for documenting compiler intermediate representations. # # This defines a 'cton' Sphinx domain with the following directives and roles: # @@ -29,10 +29,10 @@ import sphinx.ext.autodoc class CtonObject(ObjectDescription): """ - Any kind of Cretonne IL object. + Any kind of Cretonne IR object. This is a shared base class for the different kinds of indexable objects - in the Cretonne IL reference. + in the Cretonne IR reference. """ option_spec = { 'noindex': directives.flag, @@ -98,7 +98,7 @@ def parse_type(name, signode): class CtonType(CtonObject): - """A Cretonne IL type description.""" + """A Cretonne IR type description.""" def handle_signature(self, sig, signode): """ @@ -112,7 +112,7 @@ class CtonType(CtonObject): return name def get_index_text(self, name): - return name + ' (IL type)' + return name + ' (IR type)' sep_equal = re.compile('\s*=\s*') @@ -127,7 +127,7 @@ def parse_params(s, signode): class CtonInst(CtonObject): - """A Cretonne IL instruction.""" + """A Cretonne IR instruction.""" doc_field_types = [ TypedField('argument', label=l_('Arguments'), @@ -176,11 +176,11 @@ class CtonInst(CtonObject): class CtonInstGroup(CtonObject): - """A Cretonne IL instruction group.""" + """A Cretonne IR instruction group.""" class CretonneDomain(Domain): - """Cretonne domain for intermediate language objects.""" + """Cretonne domain for IR objects.""" name = 'cton' label = 'Cretonne' diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index fb716257bd..162248c83e 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -5,19 +5,19 @@ Cretonne Language Reference .. default-domain:: cton .. highlight:: cton -The Cretonne intermediate language (:term:`IL`) has two equivalent -representations: an *in-memory data structure* that the code generator library -is using, and a *text format* which is used for test cases and debug output. -Files containing Cretonne textual IL have the ``.cton`` filename extension. +The Cretonne intermediate representation (:term:`IR`) has two primary forms: +an *in-memory data structure* that the code generator library is using, and a +*text format* which is used for test cases and debug output. +Files containing Cretonne textual IR have the ``.cton`` filename extension. -This reference uses the text format to describe IL semantics but glosses over +This reference uses the text format to describe IR semantics but glosses over the finer details of the lexical and syntactic structure of the format. Overall structure ================= -Cretonne compiles functions independently. A ``.cton`` IL file may contain +Cretonne compiles functions independently. A ``.cton`` IR file may contain multiple functions, and the programmatic API can create multiple function handles at the same time, but the functions don't share any data or reference each other directly. @@ -27,7 +27,7 @@ This is a simple C function that computes the average of an array of floats: .. literalinclude:: example.c :language: c -Here is the same function compiled into Cretonne IL: +Here is the same function compiled into Cretonne IR: .. literalinclude:: example.cton :language: cton @@ -77,7 +77,7 @@ variable value for the next iteration. The `cton_frontend` crate contains utilities for translating from programs containing multiple assignments to the same variables into SSA form for -Cretonne :term:`IL`. +Cretonne :term:`IR`. Such variables can also be presented to Cretonne as :term:`stack slot`\s. Stack slots are accessed with the :inst:`stack_store` and :inst:`stack_load` @@ -303,7 +303,7 @@ indicate the different kinds of immediate operands on an instruction. A floating point condition code. See the :inst:`fcmp` instruction for details. The two IEEE floating point immediate types :type:`ieee32` and :type:`ieee64` -are displayed as hexadecimal floating point literals in the textual :term:`IL` +are displayed as hexadecimal floating point literals in the textual :term:`IR` format. Decimal floating point literals are not allowed because some computer systems can round differently when converting to binary. The hexadecimal floating point format is mostly the same as the one used by C99, but extended @@ -563,7 +563,7 @@ runtime data structures. alignment for storing a pointer. Chains of ``deref`` global variables are possible, but cycles are not - allowed. They will be caught by the IL verifier. + allowed. They will be caught by the IR verifier. :arg BaseGV: Global variable containing the base pointer. :arg Offset: Byte offset from the loaded base pointer to the global @@ -1154,19 +1154,11 @@ Glossary The extended basic blocks which contain all the executable code in a function. The function body follows the function preamble. - intermediate language - IL - The language used to describe functions to Cretonne. This reference - describes the syntax and semantics of the Cretonne IL. The IL has two - forms: Textual and an in-memory intermediate representation - (:term:`IR`). - intermediate representation IR - The in-memory representation of :term:`IL`. The data structures - Cretonne uses to represent a program internally are called the - intermediate representation. Cretonne's IR can be converted to text - losslessly. + The language used to describe functions to Cretonne. This reference + describes the syntax and semantics of Cretonne IR. The IR has two + forms: Textual, and an in-memory data structure. stack slot A fixed size memory allocation in the current function's activation diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index 320cc37389..c14a74aeba 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -89,7 +89,7 @@ easier to provide substantial input functions for the compiler tests. File tests are :file:`*.cton` files in the :file:`filetests/` directory hierarchy. Each file has a header describing what to test followed by a number -of input functions in the :doc:`Cretonne textual intermediate language +of input functions in the :doc:`Cretonne textual intermediate representation `: .. productionlist:: @@ -166,7 +166,7 @@ Cretonne's tests don't need this. ---------- This is one of the simplest file tests, used for testing the conversion to and -from textual IL. The ``test cat`` command simply parses each function and +from textual IR. The ``test cat`` command simply parses each function and converts it back to text again. The text of each function is then matched against the associated filecheck directives. @@ -188,7 +188,7 @@ Example:: `test verifier` --------------- -Run each function through the IL verifier and check that it produces the +Run each function through the IR verifier and check that it produces the expected error messages. Expected error messages are indicated with an ``error:`` directive *on the @@ -351,4 +351,4 @@ Each function is passed through the full ``Context::compile()`` function which is normally used to compile code. This type of test often depends on assertions or verifier errors, but it is also possible to use filecheck directives which will be matched against the final form of the -Cretonne IL right before binary machine code emission. +Cretonne IR right before binary machine code emission. diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs index 4d3d73ac2c..0fc5541db9 100644 --- a/cranelift/src/cat.rs +++ b/cranelift/src/cat.rs @@ -1,6 +1,6 @@ //! The `cat` sub-command. //! -//! Read a sequence of Cretonne IL files and print them again to stdout. This has the effect of +//! Read a sequence of Cretonne IR files and print them again to stdout. This has the effect of //! normalizing formatting and removing comments. use cton_reader::parse_functions; diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index a1cc244f5d..c21b989149 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -1,6 +1,4 @@ -//! CLI tool to compile cretonne IL into native code. -//! -//! Reads IR files into Cretonne IL and compiles it. +//! CLI tool to read Cretonne IR files and compile them into native code. use cton_reader::parse_test; use std::path::PathBuf; diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 216eae458b..4dd9865d3d 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -38,12 +38,12 @@ Options: -T, --time-passes print pass timing report -t, --just-decode - just decode WebAssembly to Cretonne IL + just decode WebAssembly to Cretonne IR -s, --print-size prints generated code size -c, --check-translation - just checks the correctness of Cretonne IL translated from WebAssembly - -p, --print print the resulting Cretonne IL + just checks the correctness of Cretonne IR translated from WebAssembly + -p, --print print the resulting Cretonne IR -h, --help print this help message --set= configure Cretonne settings --isa= specify the Cretonne ISA diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index 65242cc8f8..846ee3e8fa 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -1,6 +1,6 @@ //! The `print-cfg` sub-command. //! -//! Read a series of Cretonne IL files and print their control flow graphs +//! Read a series of Cretonne IR files and print their control flow graphs //! in graphviz format. use CommandResult; diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 5f986ee5f2..918deffd5a 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -1,6 +1,6 @@ //! CLI tool to use the functions provided by the [cretonne-wasm](../cton_wasm/index.html) crate. //! -//! Reads Wasm binary files, translates the functions' code to Cretonne IL. +//! Reads Wasm binary files, translates the functions' code to Cretonne IR. #![cfg_attr(feature = "cargo-clippy", allow(too_many_arguments, cyclomatic_complexity))] use cton_wasm::{translate_module, DummyEnvironment, ModuleEnvironment}; diff --git a/lib/cretonne/README.md b/lib/cretonne/README.md index 73b2806f5d..c12dfbf5a9 100644 --- a/lib/cretonne/README.md +++ b/lib/cretonne/README.md @@ -1,2 +1,2 @@ This crate contains the core Cretonne code generator. It translates code from an -intermediate language into executable machine code. +intermediate representation into executable machine code. diff --git a/lib/cretonne/meta/base/formats.py b/lib/cretonne/meta/base/formats.py index b50824581b..595f09b9fa 100644 --- a/lib/cretonne/meta/base/formats.py +++ b/lib/cretonne/meta/base/formats.py @@ -2,7 +2,7 @@ The cretonne.formats defines all instruction formats. Every instruction format has a corresponding `InstructionData` variant in the -Rust representation of cretonne IL, so all instruction formats must be defined +Rust representation of Cretonne IR, so all instruction formats must be defined in this module. """ from __future__ import absolute_import diff --git a/lib/cretonne/meta/base/settings.py b/lib/cretonne/meta/base/settings.py index e8785fbc08..3bd90f107b 100644 --- a/lib/cretonne/meta/base/settings.py +++ b/lib/cretonne/meta/base/settings.py @@ -20,10 +20,10 @@ opt_level = EnumSetting( enable_verifier = BoolSetting( """ - Run the Cretonne IL verifier at strategic times during compilation. + Run the Cretonne IR verifier at strategic times during compilation. This makes compilation slower but catches many bugs. The verifier is - disabled by default, except when reading Cretonne IL from a text file. + disabled by default, except when reading Cretonne IR from a text file. """, default=True) diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/cretonne/meta/cdsl/ast.py index 38d633e17d..5eefe222f8 100644 --- a/lib/cretonne/meta/cdsl/ast.py +++ b/lib/cretonne/meta/cdsl/ast.py @@ -559,7 +559,7 @@ class Enumerator(Literal): is an AST leaf node representing one of the values. :param kind: The enumerated `ImmediateKind` containing the value. - :param value: The textual IL representation of the value. + :param value: The textual IR representation of the value. `Enumerator` nodes are not usually created directly. They are created by using the dot syntax on immediate kinds: `intcc.ult`. diff --git a/lib/cretonne/src/binemit/memorysink.rs b/lib/cretonne/src/binemit/memorysink.rs index 1d6758afe6..506ea8b5a4 100644 --- a/lib/cretonne/src/binemit/memorysink.rs +++ b/lib/cretonne/src/binemit/memorysink.rs @@ -20,7 +20,7 @@ use std::ptr::write_unaligned; /// A `CodeSink` that writes binary machine code directly into memory. /// -/// A `MemoryCodeSink` object should be used when emitting a Cretonne IL function into executable +/// A `MemoryCodeSink` object should be used when emitting a Cretonne IR function into executable /// memory. It writes machine code directly to a raw pointer without any bounds checking, so make /// sure to allocate enough memory for the whole function. The number of bytes required is returned /// by the `Context::compile()` function. diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 1a4e772be4..0727721d6a 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -1,6 +1,6 @@ -//! IL entity references. +//! Cretonne IR entity references. //! -//! Instructions in Cretonne IL need to reference other entities in the function. This can be other +//! Instructions in Cretonne IR need to reference other entities in the function. This can be other //! parts of the function like extended basic blocks or stack slots, or it can be external entities //! that are declared in the function preamble in the text format. //! @@ -16,7 +16,7 @@ //! data structures use the `PackedOption` representation, while function arguments and //! return values prefer the more Rust-like `Option` variant. //! -//! The entity references all implement the `Display` trait in a way that matches the textual IL +//! The entity references all implement the `Display` trait in a way that matches the textual IR //! format. use std::fmt; diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 8ee332526f..e52bb4c96f 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -55,7 +55,7 @@ pub struct Function { /// /// This information is only transiently available after the `binemit::relax_branches` function /// computes it, and it can easily be recomputed by calling that function. It is not included - /// in the textual IL format. + /// in the textual IR format. pub offsets: EbbOffsets, /// Source locations. diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 68ee656e85..bb09e24834 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -1,7 +1,7 @@ //! Instruction formats and opcodes. //! //! The `instructions` module contains definitions for instruction formats, opcodes, and the -//! in-memory representation of IL instructions. +//! in-memory representation of IR instructions. //! //! A large part of this module is auto-generated from the instruction descriptions in the meta //! directory. diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 2f4c378edd..c19cf6dd37 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -71,8 +71,7 @@ impl Layout { // within an EBB. The instruction sequence numbers are all between the sequence number of their // containing EBB and the following EBB. // -// The result is that sequence numbers work like BASIC line numbers for the textual representation -// of the IL. +// The result is that sequence numbers work like BASIC line numbers for the textual form of the IR. type SequenceNumber = u32; // Initial stride assigned to new sequence numbers. diff --git a/lib/cretonne/src/ir/libcall.rs b/lib/cretonne/src/ir/libcall.rs index 85133611be..6434197247 100644 --- a/lib/cretonne/src/ir/libcall.rs +++ b/lib/cretonne/src/ir/libcall.rs @@ -6,7 +6,7 @@ use std::str::FromStr; /// The name of a runtime library routine. /// -/// Runtime library calls are generated for Cretonne IL instructions that don't have an equivalent +/// Runtime library calls are generated for Cretonne IR instructions that don't have an equivalent /// ISA instruction or an easy macro expansion. A `LibCall` is used as a well-known name to refer to /// the runtime library routine. This way, Cretonne doesn't have to know about the naming /// convention in the embedding VM's runtime library. diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 2144ad6811..4f4e671f30 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -1,4 +1,4 @@ -//! Representation of Cretonne IL functions. +//! Representation of Cretonne IR functions. pub mod types; pub mod entities; diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index ce5b108e54..8fea3193c6 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -12,7 +12,7 @@ use std::cmp; /// 1. An instruction or /// 2. An EBB header. /// -/// This corresponds more or less to the lines in the textual representation of Cretonne IL. +/// This corresponds more or less to the lines in the textual form of Cretonne IR. #[derive(PartialEq, Eq, Clone, Copy)] pub struct ProgramPoint(u32); diff --git a/lib/cretonne/src/ir/sourceloc.rs b/lib/cretonne/src/ir/sourceloc.rs index 36e5247488..768bbf99d5 100644 --- a/lib/cretonne/src/ir/sourceloc.rs +++ b/lib/cretonne/src/ir/sourceloc.rs @@ -7,7 +7,7 @@ use std::fmt; /// A source location. /// -/// This is an opaque 32-bit number attached to each Cretonne IL instruction. Cretonne does not +/// This is an opaque 32-bit number attached to each Cretonne IR instruction. Cretonne does not /// interpret source locations in any way, they are simply preserved from the input to the output. /// /// The default source location uses the all-ones bit pattern `!0`. It is used for instructions diff --git a/lib/cretonne/src/legalizer/globalvar.rs b/lib/cretonne/src/legalizer/globalvar.rs index 901b0da416..27494bfd87 100644 --- a/lib/cretonne/src/legalizer/globalvar.rs +++ b/lib/cretonne/src/legalizer/globalvar.rs @@ -45,7 +45,7 @@ fn vmctx_addr(inst: ir::Inst, func: &mut ir::Function, offset: i64) { /// Expand a `global_addr` instruction for a deref global. fn deref_addr(inst: ir::Inst, func: &mut ir::Function, base: ir::GlobalVar, offset: i64) { // We need to load a pointer from the `base` global variable, so insert a new `global_addr` - // instruction. This depends on the iterative legalization loop. Note that the IL verifier + // instruction. This depends on the iterative legalization loop. Note that the IR verifier // detects any cycles in the `deref` globals. let ptr_ty = func.dfg.value_type(func.dfg.first_result(inst)); let mut pos = FuncCursor::new(func).at_inst(inst); diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index 16eb0e9b50..68fd04754e 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -15,7 +15,7 @@ use std::vec::Vec; /// A diversion of a value from its original location to a new register or stack location. /// -/// In IL, a diversion is represented by a `regmove` instruction, possibly a chain of them for the +/// In IR, a diversion is represented by a `regmove` instruction, possibly a chain of them for the /// same value. /// /// When tracking diversions, the `from` field is the original assigned value location, and `to` is diff --git a/lib/cretonne/src/result.rs b/lib/cretonne/src/result.rs index aa209a0845..bd2ae5a101 100644 --- a/lib/cretonne/src/result.rs +++ b/lib/cretonne/src/result.rs @@ -15,9 +15,9 @@ pub enum CtonError { /// code. This should never happen for validated WebAssembly code. InvalidInput, - /// An IL verifier error. + /// An IR verifier error. /// - /// This always represents a bug, either in the code that generated IL for Cretonne, or a bug + /// This always represents a bug, either in the code that generated IR for Cretonne, or a bug /// in Cretonne itself. Verifier(verifier::Error), diff --git a/lib/cretonne/src/timing.rs b/lib/cretonne/src/timing.rs index 647e0a07aa..51cdd85bfa 100644 --- a/lib/cretonne/src/timing.rs +++ b/lib/cretonne/src/timing.rs @@ -41,11 +41,11 @@ define_passes!{ Pass, NUM_PASSES, DESCRIPTIONS; process_file: "Processing test file", - parse_text: "Parsing textual Cretonne IL", + parse_text: "Parsing textual Cretonne IR", wasm_translate_module: "Translate WASM module", wasm_translate_function: "Translate WASM function", - verifier: "Verify Cretonne IL", + verifier: "Verify Cretonne IR", verify_cssa: "Verify CSSA", verify_liveness: "Verify live ranges", verify_locations: "Verify value locations", diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index bbe8375b89..1081a87d77 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -1,8 +1,7 @@ -//! Converting Cretonne IL to text. +//! Converting Cretonne IR to text. //! -//! The `write` module provides the `write_function` function which converts an IL `Function` to an -//! equivalent textual representation. This textual representation can be read back by the -//! `cretonne-reader` crate. +//! The `write` module provides the `write_function` function which converts an IR `Function` to an +//! equivalent textual form. This textual form can be read back by the `cretonne-reader` crate. use ir::{Function, DataFlowGraph, Ebb, Inst, Value, ValueDef, Type, SigRef}; use isa::{TargetIsa, RegInfo}; diff --git a/lib/filetests/src/test_legalizer.rs b/lib/filetests/src/test_legalizer.rs index c8d03e4c2a..08f6c53ae4 100644 --- a/lib/filetests/src/test_legalizer.rs +++ b/lib/filetests/src/test_legalizer.rs @@ -1,4 +1,4 @@ -//! Test command for checking the IL legalizer. +//! Test command for checking the IR legalizer. //! //! The `test legalizer` test command runs each function through `legalize_function()` and sends //! the result to filecheck. diff --git a/lib/filetests/src/test_print_cfg.rs b/lib/filetests/src/test_print_cfg.rs index 470d293185..b043dc58b2 100644 --- a/lib/filetests/src/test_print_cfg.rs +++ b/lib/filetests/src/test_print_cfg.rs @@ -1,6 +1,6 @@ //! The `print-cfg` sub-command. //! -//! Read a series of Cretonne IL files and print their control flow graphs +//! Read a series of Cretonne IR files and print their control flow graphs //! in graphviz format. use std::borrow::Cow; diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs index bb4375b59e..6db9c681ce 100644 --- a/lib/filetests/src/test_verifier.rs +++ b/lib/filetests/src/test_verifier.rs @@ -1,4 +1,4 @@ -//! Test command for checking the IL verifier. +//! Test command for checking the IR verifier. //! //! The `test verifier` test command looks for annotations on instructions like this: //! diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 40f1833c71..0bef102c66 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -2,7 +2,7 @@ authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" version = "0.4.1" -description = "Cretonne IL builder helper" +description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/Cretonne/cretonne" diff --git a/lib/frontend/README.md b/lib/frontend/README.md index 53248b68ab..cca2b12ab5 100644 --- a/lib/frontend/README.md +++ b/lib/frontend/README.md @@ -1,5 +1,5 @@ This crate provides a straightforward way to create a -[Cretonne](https://crates.io/crates/cretonne) IL function and fill it with +[Cretonne](https://crates.io/crates/cretonne) IR function and fill it with instructions translated from another language. It contains an SSA construction module that provides convenient methods for translating non-SSA variables into -SSA Cretonne IL values via `use_var` and `def_var` calls. +SSA Cretonne IR values via `use_var` and `def_var` calls. diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 8cd2c5cc82..2706846124 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -1,4 +1,4 @@ -//! A frontend for building Cretonne IL from other languages. +//! A frontend for building Cretonne IR from other languages. use cretonne::cursor::{Cursor, FuncCursor}; use cretonne::ir; use cretonne::ir::{Ebb, Type, Value, Function, Inst, JumpTable, StackSlot, JumpTableData, @@ -10,7 +10,7 @@ use ssa::{SSABuilder, SideEffects, Block}; use cretonne::entity::{EntityRef, EntityMap, EntitySet}; use cretonne::packed_option::PackedOption; -/// Structure used for translating a series of functions into Cretonne IL. +/// Structure used for translating a series of functions into Cretonne IR. /// /// In order to reduce memory reallocations when compiling multiple functions, /// `FunctionBuilderContext` holds various data structures which are cleared between @@ -29,7 +29,7 @@ where } -/// Temporary object used to build a single Cretonne IL `Function`. +/// Temporary object used to build a single Cretonne IR `Function`. pub struct FunctionBuilder<'a, Variable: 'a> where Variable: EntityRef, @@ -103,7 +103,7 @@ where } /// Implementation of the [`InstBuilder`](../cretonne/ir/builder/trait.InstBuilder.html) that has -/// one convenience method per Cretonne IL instruction. +/// one convenience method per Cretonne IR instruction. pub struct FuncInstBuilder<'short, 'long: 'short, Variable: 'long> where Variable: EntityRef, @@ -191,7 +191,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short } } -/// This module allows you to create a function in Cretonne IL in a straightforward way, hiding +/// This module allows you to create a function in Cretonne IR in a straightforward way, hiding /// all the complexity of its internal representation. /// /// The module is parametrized by one type which is the representation of variables in your @@ -203,10 +203,10 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short /// - the last instruction of each block is a terminator instruction which has no natural successor, /// and those instructions can only appear at the end of extended blocks. /// -/// The parameters of Cretonne IL instructions are Cretonne IL values, which can only be created -/// as results of other Cretonne IL instructions. To be able to create variables redefined multiple +/// The parameters of Cretonne IR instructions are Cretonne IR values, which can only be created +/// as results of other Cretonne IR instructions. To be able to create variables redefined multiple /// times in your program, use the `def_var` and `use_var` command, that will maintain the -/// correspondence between your variables and Cretonne IL SSA values. +/// correspondence between your variables and Cretonne IR SSA values. /// /// The first block for which you call `switch_to_block` will be assumed to be the beginning of /// the function. @@ -220,7 +220,7 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short /// /// # Errors /// -/// The functions below will panic in debug mode whenever you try to modify the Cretonne IL +/// The functions below will panic in debug mode whenever you try to modify the Cretonne IR /// function in a way that violate the coherence of the code. For instance: switching to a new /// `Ebb` when you haven't filled the current one with a terminator instruction, inserting a /// return instruction with arguments that don't match the function's signature. @@ -311,7 +311,7 @@ where self.func_ctx.types[var] = ty; } - /// Returns the Cretonne IL value corresponding to the utilization at the current program + /// Returns the Cretonne IR value corresponding to the utilization at the current program /// position of a previously defined user variable. pub fn use_var(&mut self, var: Variable) -> Value { let ty = *self.func_ctx.types.get(var).expect( @@ -462,8 +462,8 @@ where } /// All the functions documented in the previous block are write-only and help you build a valid -/// Cretonne IL functions via multiple debug asserts. However, you might need to improve the -/// performance of your translation perform more complex transformations to your Cretonne IL +/// Cretonne IR functions via multiple debug asserts. However, you might need to improve the +/// performance of your translation perform more complex transformations to your Cretonne IR /// function. The functions below help you inspect the function you're creating and modify it /// in ways that can be unsafe if used incorrectly. impl<'a, Variable> FunctionBuilder<'a, Variable> diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 1f2c2e0db8..c7defce3bb 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -1,15 +1,15 @@ -//! Cretonne IL builder library. +//! Cretonne IR builder library. //! -//! Provides a straightforward way to create a Cretonne IL function and fill it with instructions +//! Provides a straightforward way to create a Cretonne IR function and fill it with instructions //! translated from another language. Contains an SSA construction module that lets you translate -//! your non-SSA variables into SSA Cretonne IL values via `use_var` and `def_var` calls. +//! your non-SSA variables into SSA Cretonne IR values via `use_var` and `def_var` calls. //! //! To get started, create an [`FunctionBuilderContext`](struct.FunctionBuilderContext.html) and //! pass it as an argument to a [`FunctionBuilder`](struct.FunctionBuilder.html). //! //! # Example //! -//! Here is a pseudo-program we want to transform into Cretonne IL: +//! Here is a pseudo-program we want to transform into Cretonne IR: //! //! ```cton //! function(x) { @@ -29,7 +29,7 @@ //! } //! ``` //! -//! Here is how you build the corresponding Cretonne IL function using `FunctionBuilderContext`: +//! Here is how you build the corresponding Cretonne IR function using `FunctionBuilderContext`: //! //! ```rust //! extern crate cretonne; diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index bd4a8d89fd..af0d35695e 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -231,7 +231,7 @@ fn emit_zero(ty: Type, mut cur: FuncCursor) -> Value { } } /// The following methods are the API of the SSA builder. Here is how it should be used when -/// translating to Cretonne IL: +/// translating to Cretonne IR: /// /// - for each sequence of contiguous instructions (with no branches), create a corresponding /// basic block with `declare_ebb_body_block` or `declare_ebb_header_block` depending on the diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 80fab0f499..d8259a2027 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -2,7 +2,7 @@ authors = ["The Cretonne Project Developers"] name = "cretonne-reader" version = "0.4.1" -description = "Cretonne textual IL reader" +description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/Cretonne/cretonne" diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 384de9381b..b165756566 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1782,7 +1782,7 @@ impl<'a> Parser<'a> { // explicit type specified. Look up `ctrl_value` to see if it was defined // already. // TBD: If it is defined in another block, the type should have been - // specified explicitly. It is unfortunate that the correctness of IL + // specified explicitly. It is unfortunate that the correctness of IR // depends on the layout of the blocks. let ctrl_src_value = inst_data .typevar_operand(&ctx.function.dfg.value_lists) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 0fcd2210f5..b532f29c55 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -2,7 +2,7 @@ name = "cretonne-wasm" version = "0.4.1" authors = ["The Cretonne Project Developers"] -description = "Translator from WebAssembly to Cretonne IL" +description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/Cretonne/cretonne" license = "Apache-2.0" readme = "README.md" diff --git a/lib/wasm/README.md b/lib/wasm/README.md index f12f00d46a..946a41f26d 100644 --- a/lib/wasm/README.md +++ b/lib/wasm/README.md @@ -1,3 +1,2 @@ This crate performs the translation from a wasm module in binary format to the -in-memory representation of the [Cretonne](https://crates.io/crates/cretonne) -IL. +in-memory form of the [Cretonne](https://crates.io/crates/cretonne) IR. diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 69111dc8e8..61b7118636 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -1,5 +1,5 @@ //! This module contains the bulk of the interesting code performing the translation between -//! WebAssembly and Cretonne IL. +//! WebAssembly and Cretonne IR. //! //! The translation is done in one pass, opcode by opcode. Two main data structures are used during //! code translations: the value stack and the control stack. The value stack mimics the execution @@ -38,7 +38,7 @@ use std::vec::Vec; // Clippy warns about "flags: _" but its important to document that the flags field is ignored #[cfg_attr(feature = "cargo-clippy", allow(unneeded_field_pattern))] -/// Translates wasm operators into Cretonne IL instructions. Returns `true` if it inserted +/// Translates wasm operators into Cretonne IR instructions. Returns `true` if it inserted /// a return. pub fn translate_operator( op: Operator, diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 41583c1185..233188364a 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -26,7 +26,7 @@ pub enum GlobalValue { /// Environment affecting the translation of a single WebAssembly function. /// /// A `FuncEnvironment` trait object is required to translate a WebAssembly function to Cretonne -/// IL. The function environment provides information about the WebAssembly module as well as the +/// IR. The function environment provides information about the WebAssembly module as well as the /// runtime environment. pub trait FuncEnvironment { /// Get the flags for the current compilation. diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 5ac69b07b8..dbb1fc9bcc 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -1,7 +1,7 @@ -//! Stand-alone WebAssembly to Cretonne IL translator. +//! Stand-alone WebAssembly to Cretonne IR translator. //! //! This module defines the `FuncTranslator` type which can translate a single WebAssembly -//! function to Cretonne IL guided by a `FuncEnvironment` which provides information about the +//! function to Cretonne IR guided by a `FuncEnvironment` which provides information about the //! WebAssembly module and the runtime environment. use code_translator::translate_operator; @@ -14,9 +14,9 @@ use environ::FuncEnvironment; use state::TranslationState; use wasmparser::{self, BinaryReader}; -/// WebAssembly to Cretonne IL function translator. +/// WebAssembly to Cretonne IR function translator. /// -/// A `FuncTranslator` is used to translate a binary WebAssembly function into Cretonne IL guided +/// A `FuncTranslator` is used to translate a binary WebAssembly function into Cretonne IR guided /// by a `FuncEnvironment` object. A single translator instance can be reused to translate multiple /// functions which will reduce heap allocation traffic. pub struct FuncTranslator { diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index f2a1adb7cb..cc35a19fbc 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -1,5 +1,5 @@ -//! Performs the translation from a wasm module in binary format to the in-memory representation -//! of the Cretonne IL. More particularly, it translates the code of all the functions bodies and +//! Performs translation from a wasm module in binary format to the in-memory form +//! of Cretonne IR. More particularly, it translates the code of all the functions bodies and //! interacts with an environment implementing the //! [`ModuleEnvironment`](trait.ModuleEnvironment.html) //! trait to deal with tables, globals and linear memory. diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index bb74db4276..171f720e12 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -10,7 +10,7 @@ use environ::ModuleEnvironment; use std::string::String; -/// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IL +/// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IR /// [`Function`](../cretonne/ir/function/struct.Function.html). /// Returns the functions and also the mappings for imported functions and signature between the /// indexes in the wasm module and the indexes inside each functions. From d566faa8fb1f626ef89eb351da3ec9a448c7736c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Mar 2018 16:16:35 -0700 Subject: [PATCH 1664/3084] Disable preopt at opt_level=fastest. --- lib/cretonne/src/context.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index fba2cd2cc3..54fce4b84a 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -88,7 +88,9 @@ impl Context { self.verify_if(isa)?; self.compute_cfg(); - self.preopt(isa)?; + if isa.flags().opt_level() != OptLevel::Fastest { + self.preopt(isa)?; + } self.legalize(isa)?; if isa.flags().opt_level() == OptLevel::Best { self.compute_domtree(); From 951ff11f85300df92e2c8a45d91bdf7923675c09 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Wed, 28 Mar 2018 22:48:03 -0700 Subject: [PATCH 1665/3084] [WIP] Add a Trap sink to code generation (#279) * First draft of TrapSink implementation. * Add trap sink calls to 'trapif' and 'trapff' recipes. * Add SourceLoc to trap sink calls, and add trap sink calls to all loads and stores. * Add IntegerDivisionByZero trap to div recipe. * Only emit load/store traps if 'notrap' flag is not set on the instruction. * Update filetest machinery to add new trap sink functionality. * Update filetests to include traps in output. * Add a few more trap outputs to filetests. * Add trap output to CLI tool. --- .../filetests/isa/intel/binary32-float.cton | 64 ++--- cranelift/filetests/isa/intel/binary32.cton | 124 ++++----- .../filetests/isa/intel/binary64-float.cton | 72 ++--- cranelift/filetests/isa/intel/binary64.cton | 246 +++++++++--------- cranelift/src/compile.rs | 15 +- lib/cretonne/meta/isa/intel/recipes.py | 53 +++- lib/cretonne/src/binemit/memorysink.rs | 21 +- lib/cretonne/src/binemit/mod.rs | 7 +- lib/cretonne/src/context.rs | 12 +- lib/cretonne/src/isa/intel/binemit.rs | 2 +- lib/filetests/src/test_binemit.rs | 4 + lib/filetests/src/test_compile.rs | 1 + 12 files changed, 358 insertions(+), 263 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/intel/binary32-float.cton index 95e132cf6e..4477c4d8d3 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/intel/binary32-float.cton @@ -148,30 +148,30 @@ ebb0: ; Load/Store ; asm: movss (%ecx), %xmm5 - [-,%xmm5] v100 = load.f32 v0 ; bin: f3 0f 10 29 + [-,%xmm5] v100 = load.f32 v0 ; bin: heap_oob f3 0f 10 29 ; asm: movss (%esi), %xmm2 - [-,%xmm2] v101 = load.f32 v1 ; bin: f3 0f 10 16 + [-,%xmm2] v101 = load.f32 v1 ; bin: heap_oob f3 0f 10 16 ; asm: movss 50(%ecx), %xmm5 - [-,%xmm5] v110 = load.f32 v0+50 ; bin: f3 0f 10 69 32 + [-,%xmm5] v110 = load.f32 v0+50 ; bin: heap_oob f3 0f 10 69 32 ; asm: movss -50(%esi), %xmm2 - [-,%xmm2] v111 = load.f32 v1-50 ; bin: f3 0f 10 56 ce + [-,%xmm2] v111 = load.f32 v1-50 ; bin: heap_oob f3 0f 10 56 ce ; asm: movss 10000(%ecx), %xmm5 - [-,%xmm5] v120 = load.f32 v0+10000 ; bin: f3 0f 10 a9 00002710 + [-,%xmm5] v120 = load.f32 v0+10000 ; bin: heap_oob f3 0f 10 a9 00002710 ; asm: movss -10000(%esi), %xmm2 - [-,%xmm2] v121 = load.f32 v1-10000 ; bin: f3 0f 10 96 ffffd8f0 + [-,%xmm2] v121 = load.f32 v1-10000 ; bin: heap_oob f3 0f 10 96 ffffd8f0 ; asm: movss %xmm5, (%ecx) - [-] store.f32 v100, v0 ; bin: f3 0f 11 29 + [-] store.f32 v100, v0 ; bin: heap_oob f3 0f 11 29 ; asm: movss %xmm2, (%esi) - [-] store.f32 v101, v1 ; bin: f3 0f 11 16 + [-] store.f32 v101, v1 ; bin: heap_oob f3 0f 11 16 ; asm: movss %xmm5, 50(%ecx) - [-] store.f32 v100, v0+50 ; bin: f3 0f 11 69 32 + [-] store.f32 v100, v0+50 ; bin: heap_oob f3 0f 11 69 32 ; asm: movss %xmm2, -50(%esi) - [-] store.f32 v101, v1-50 ; bin: f3 0f 11 56 ce + [-] store.f32 v101, v1-50 ; bin: heap_oob f3 0f 11 56 ce ; asm: movss %xmm5, 10000(%ecx) - [-] store.f32 v100, v0+10000 ; bin: f3 0f 11 a9 00002710 + [-] store.f32 v100, v0+10000 ; bin: heap_oob f3 0f 11 a9 00002710 ; asm: movss %xmm2, -10000(%esi) - [-] store.f32 v101, v1-10000 ; bin: f3 0f 11 96 ffffd8f0 + [-] store.f32 v101, v1-10000 ; bin: heap_oob f3 0f 11 96 ffffd8f0 ; Spill / Fill. @@ -363,30 +363,30 @@ ebb0: ; Load/Store ; asm: movsd (%ecx), %xmm5 - [-,%xmm5] v100 = load.f64 v0 ; bin: f2 0f 10 29 + [-,%xmm5] v100 = load.f64 v0 ; bin: heap_oob f2 0f 10 29 ; asm: movsd (%esi), %xmm2 - [-,%xmm2] v101 = load.f64 v1 ; bin: f2 0f 10 16 + [-,%xmm2] v101 = load.f64 v1 ; bin: heap_oob f2 0f 10 16 ; asm: movsd 50(%ecx), %xmm5 - [-,%xmm5] v110 = load.f64 v0+50 ; bin: f2 0f 10 69 32 + [-,%xmm5] v110 = load.f64 v0+50 ; bin: heap_oob f2 0f 10 69 32 ; asm: movsd -50(%esi), %xmm2 - [-,%xmm2] v111 = load.f64 v1-50 ; bin: f2 0f 10 56 ce + [-,%xmm2] v111 = load.f64 v1-50 ; bin: heap_oob f2 0f 10 56 ce ; asm: movsd 10000(%ecx), %xmm5 - [-,%xmm5] v120 = load.f64 v0+10000 ; bin: f2 0f 10 a9 00002710 + [-,%xmm5] v120 = load.f64 v0+10000 ; bin: heap_oob f2 0f 10 a9 00002710 ; asm: movsd -10000(%esi), %xmm2 - [-,%xmm2] v121 = load.f64 v1-10000 ; bin: f2 0f 10 96 ffffd8f0 + [-,%xmm2] v121 = load.f64 v1-10000 ; bin: heap_oob f2 0f 10 96 ffffd8f0 ; asm: movsd %xmm5, (%ecx) - [-] store.f64 v100, v0 ; bin: f2 0f 11 29 + [-] store.f64 v100, v0 ; bin: heap_oob f2 0f 11 29 ; asm: movsd %xmm2, (%esi) - [-] store.f64 v101, v1 ; bin: f2 0f 11 16 + [-] store.f64 v101, v1 ; bin: heap_oob f2 0f 11 16 ; asm: movsd %xmm5, 50(%ecx) - [-] store.f64 v100, v0+50 ; bin: f2 0f 11 69 32 + [-] store.f64 v100, v0+50 ; bin: heap_oob f2 0f 11 69 32 ; asm: movsd %xmm2, -50(%esi) - [-] store.f64 v101, v1-50 ; bin: f2 0f 11 56 ce + [-] store.f64 v101, v1-50 ; bin: heap_oob f2 0f 11 56 ce ; asm: movsd %xmm5, 10000(%ecx) - [-] store.f64 v100, v0+10000 ; bin: f2 0f 11 a9 00002710 + [-] store.f64 v100, v0+10000 ; bin: heap_oob f2 0f 11 a9 00002710 ; asm: movsd %xmm2, -10000(%esi) - [-] store.f64 v101, v1-10000 ; bin: f2 0f 11 96 ffffd8f0 + [-] store.f64 v101, v1-10000 ; bin: heap_oob f2 0f 11 96 ffffd8f0 ; Spill / Fill. @@ -471,21 +471,21 @@ ebb1: brff ule v1, ebb1 ; bin: 76 f0 ; asm: jp .+4; ud2 - trapff ord v1, user0 ; bin: 7a 02 0f 0b + trapff ord v1, user0 ; bin: 7a 02 user0 0f 0b ; asm: jnp .+4; ud2 - trapff uno v1, user0 ; bin: 7b 02 0f 0b + trapff uno v1, user0 ; bin: 7b 02 user0 0f 0b ; asm: je .+4; ud2 - trapff one v1, user0 ; bin: 74 02 0f 0b + trapff one v1, user0 ; bin: 74 02 user0 0f 0b ; asm: jne .+4; ud2 - trapff ueq v1, user0 ; bin: 75 02 0f 0b + trapff ueq v1, user0 ; bin: 75 02 user0 0f 0b ; asm: jna .+4; ud2 - trapff gt v1, user0 ; bin: 76 02 0f 0b + trapff gt v1, user0 ; bin: 76 02 user0 0f 0b ; asm: jnae .+4; ud2 - trapff ge v1, user0 ; bin: 72 02 0f 0b + trapff ge v1, user0 ; bin: 72 02 user0 0f 0b ; asm: jnb .+4; ud2 - trapff ult v1, user0 ; bin: 73 02 0f 0b + trapff ult v1, user0 ; bin: 73 02 user0 0f 0b ; asm: jnbe .+4; ud2 - trapff ule v1, user0 ; bin: 77 02 0f 0b + trapff ule v1, user0 ; bin: 77 02 user0 0f 0b ; asm: setnp %bl [-,%rbx] v10 = trueff ord v1 ; bin: 0f 9b c3 diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 04f492c354..a3d0723be8 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -128,13 +128,13 @@ ebb0: ; asm: movl $2, %edx [-,%rdx] v53 = iconst.i32 2 ; bin: ba 00000002 ; asm: idivl %ecx - [-,%rax,%rdx] v54, v55 = x86_sdivmodx v52, v53, v1 ; bin: f7 f9 + [-,%rax,%rdx] v54, v55 = x86_sdivmodx v52, v53, v1 ; bin: int_divz f7 f9 ; asm: idivl %esi - [-,%rax,%rdx] v56, v57 = x86_sdivmodx v52, v53, v2 ; bin: f7 fe + [-,%rax,%rdx] v56, v57 = x86_sdivmodx v52, v53, v2 ; bin: int_divz f7 fe ; asm: divl %ecx - [-,%rax,%rdx] v58, v59 = x86_udivmodx v52, v53, v1 ; bin: f7 f1 + [-,%rax,%rdx] v58, v59 = x86_udivmodx v52, v53, v1 ; bin: int_divz f7 f1 ; asm: divl %esi - [-,%rax,%rdx] v60, v61 = x86_udivmodx v52, v53, v2 ; bin: f7 f6 + [-,%rax,%rdx] v60, v61 = x86_udivmodx v52, v53, v2 ; bin: int_divz f7 f6 ; Register copies. @@ -155,105 +155,105 @@ ebb0: ; Register indirect addressing with no displacement. ; asm: movl %ecx, (%esi) - store v1, v2 ; bin: 89 0e + store v1, v2 ; bin: heap_oob 89 0e ; asm: movl %esi, (%ecx) - store v2, v1 ; bin: 89 31 + store v2, v1 ; bin: heap_oob 89 31 ; asm: movw %cx, (%esi) - istore16 v1, v2 ; bin: 66 89 0e + istore16 v1, v2 ; bin: heap_oob 66 89 0e ; asm: movw %si, (%ecx) - istore16 v2, v1 ; bin: 66 89 31 + istore16 v2, v1 ; bin: heap_oob 66 89 31 ; asm: movb %cl, (%esi) - istore8 v1, v2 ; bin: 88 0e + istore8 v1, v2 ; bin: heap_oob 88 0e ; Can't store %sil in 32-bit mode (needs REX prefix). ; asm: movl (%ecx), %edi - [-,%rdi] v100 = load.i32 v1 ; bin: 8b 39 + [-,%rdi] v100 = load.i32 v1 ; bin: heap_oob 8b 39 ; asm: movl (%esi), %edx - [-,%rdx] v101 = load.i32 v2 ; bin: 8b 16 + [-,%rdx] v101 = load.i32 v2 ; bin: heap_oob 8b 16 ; asm: movzwl (%ecx), %edi - [-,%rdi] v102 = uload16.i32 v1 ; bin: 0f b7 39 + [-,%rdi] v102 = uload16.i32 v1 ; bin: heap_oob 0f b7 39 ; asm: movzwl (%esi), %edx - [-,%rdx] v103 = uload16.i32 v2 ; bin: 0f b7 16 + [-,%rdx] v103 = uload16.i32 v2 ; bin: heap_oob 0f b7 16 ; asm: movswl (%ecx), %edi - [-,%rdi] v104 = sload16.i32 v1 ; bin: 0f bf 39 + [-,%rdi] v104 = sload16.i32 v1 ; bin: heap_oob 0f bf 39 ; asm: movswl (%esi), %edx - [-,%rdx] v105 = sload16.i32 v2 ; bin: 0f bf 16 + [-,%rdx] v105 = sload16.i32 v2 ; bin: heap_oob 0f bf 16 ; asm: movzbl (%ecx), %edi - [-,%rdi] v106 = uload8.i32 v1 ; bin: 0f b6 39 + [-,%rdi] v106 = uload8.i32 v1 ; bin: heap_oob 0f b6 39 ; asm: movzbl (%esi), %edx - [-,%rdx] v107 = uload8.i32 v2 ; bin: 0f b6 16 + [-,%rdx] v107 = uload8.i32 v2 ; bin: heap_oob 0f b6 16 ; asm: movsbl (%ecx), %edi - [-,%rdi] v108 = sload8.i32 v1 ; bin: 0f be 39 + [-,%rdi] v108 = sload8.i32 v1 ; bin: heap_oob 0f be 39 ; asm: movsbl (%esi), %edx - [-,%rdx] v109 = sload8.i32 v2 ; bin: 0f be 16 + [-,%rdx] v109 = sload8.i32 v2 ; bin: heap_oob 0f be 16 ; Register-indirect with 8-bit signed displacement. ; asm: movl %ecx, 100(%esi) - store v1, v2+100 ; bin: 89 4e 64 + store v1, v2+100 ; bin: heap_oob 89 4e 64 ; asm: movl %esi, -100(%ecx) - store v2, v1-100 ; bin: 89 71 9c + store v2, v1-100 ; bin: heap_oob 89 71 9c ; asm: movw %cx, 100(%esi) - istore16 v1, v2+100 ; bin: 66 89 4e 64 + istore16 v1, v2+100 ; bin: heap_oob 66 89 4e 64 ; asm: movw %si, -100(%ecx) - istore16 v2, v1-100 ; bin: 66 89 71 9c + istore16 v2, v1-100 ; bin: heap_oob 66 89 71 9c ; asm: movb %cl, 100(%esi) - istore8 v1, v2+100 ; bin: 88 4e 64 + istore8 v1, v2+100 ; bin: heap_oob 88 4e 64 ; asm: movl 50(%ecx), %edi - [-,%rdi] v110 = load.i32 v1+50 ; bin: 8b 79 32 + [-,%rdi] v110 = load.i32 v1+50 ; bin: heap_oob 8b 79 32 ; asm: movl -50(%esi), %edx - [-,%rdx] v111 = load.i32 v2-50 ; bin: 8b 56 ce + [-,%rdx] v111 = load.i32 v2-50 ; bin: heap_oob 8b 56 ce ; asm: movzwl 50(%ecx), %edi - [-,%rdi] v112 = uload16.i32 v1+50 ; bin: 0f b7 79 32 + [-,%rdi] v112 = uload16.i32 v1+50 ; bin: heap_oob 0f b7 79 32 ; asm: movzwl -50(%esi), %edx - [-,%rdx] v113 = uload16.i32 v2-50 ; bin: 0f b7 56 ce + [-,%rdx] v113 = uload16.i32 v2-50 ; bin: heap_oob 0f b7 56 ce ; asm: movswl 50(%ecx), %edi - [-,%rdi] v114 = sload16.i32 v1+50 ; bin: 0f bf 79 32 + [-,%rdi] v114 = sload16.i32 v1+50 ; bin: heap_oob 0f bf 79 32 ; asm: movswl -50(%esi), %edx - [-,%rdx] v115 = sload16.i32 v2-50 ; bin: 0f bf 56 ce + [-,%rdx] v115 = sload16.i32 v2-50 ; bin: heap_oob 0f bf 56 ce ; asm: movzbl 50(%ecx), %edi - [-,%rdi] v116 = uload8.i32 v1+50 ; bin: 0f b6 79 32 + [-,%rdi] v116 = uload8.i32 v1+50 ; bin: heap_oob 0f b6 79 32 ; asm: movzbl -50(%esi), %edx - [-,%rdx] v117 = uload8.i32 v2-50 ; bin: 0f b6 56 ce + [-,%rdx] v117 = uload8.i32 v2-50 ; bin: heap_oob 0f b6 56 ce ; asm: movsbl 50(%ecx), %edi - [-,%rdi] v118 = sload8.i32 v1+50 ; bin: 0f be 79 32 + [-,%rdi] v118 = sload8.i32 v1+50 ; bin: heap_oob 0f be 79 32 ; asm: movsbl -50(%esi), %edx - [-,%rdx] v119 = sload8.i32 v2-50 ; bin: 0f be 56 ce + [-,%rdx] v119 = sload8.i32 v2-50 ; bin: heap_oob 0f be 56 ce ; Register-indirect with 32-bit signed displacement. ; asm: movl %ecx, 10000(%esi) - store v1, v2+10000 ; bin: 89 8e 00002710 + store v1, v2+10000 ; bin: heap_oob 89 8e 00002710 ; asm: movl %esi, -10000(%ecx) - store v2, v1-10000 ; bin: 89 b1 ffffd8f0 + store v2, v1-10000 ; bin: heap_oob 89 b1 ffffd8f0 ; asm: movw %cx, 10000(%esi) - istore16 v1, v2+10000 ; bin: 66 89 8e 00002710 + istore16 v1, v2+10000 ; bin: heap_oob 66 89 8e 00002710 ; asm: movw %si, -10000(%ecx) - istore16 v2, v1-10000 ; bin: 66 89 b1 ffffd8f0 + istore16 v2, v1-10000 ; bin: heap_oob 66 89 b1 ffffd8f0 ; asm: movb %cl, 10000(%esi) - istore8 v1, v2+10000 ; bin: 88 8e 00002710 + istore8 v1, v2+10000 ; bin: heap_oob 88 8e 00002710 ; asm: movl 50000(%ecx), %edi - [-,%rdi] v120 = load.i32 v1+50000 ; bin: 8b b9 0000c350 + [-,%rdi] v120 = load.i32 v1+50000 ; bin: heap_oob 8b b9 0000c350 ; asm: movl -50000(%esi), %edx - [-,%rdx] v121 = load.i32 v2-50000 ; bin: 8b 96 ffff3cb0 + [-,%rdx] v121 = load.i32 v2-50000 ; bin: heap_oob 8b 96 ffff3cb0 ; asm: movzwl 50000(%ecx), %edi - [-,%rdi] v122 = uload16.i32 v1+50000 ; bin: 0f b7 b9 0000c350 + [-,%rdi] v122 = uload16.i32 v1+50000 ; bin: heap_oob 0f b7 b9 0000c350 ; asm: movzwl -50000(%esi), %edx - [-,%rdx] v123 = uload16.i32 v2-50000 ; bin: 0f b7 96 ffff3cb0 + [-,%rdx] v123 = uload16.i32 v2-50000 ; bin: heap_oob 0f b7 96 ffff3cb0 ; asm: movswl 50000(%ecx), %edi - [-,%rdi] v124 = sload16.i32 v1+50000 ; bin: 0f bf b9 0000c350 + [-,%rdi] v124 = sload16.i32 v1+50000 ; bin: heap_oob 0f bf b9 0000c350 ; asm: movswl -50000(%esi), %edx - [-,%rdx] v125 = sload16.i32 v2-50000 ; bin: 0f bf 96 ffff3cb0 + [-,%rdx] v125 = sload16.i32 v2-50000 ; bin: heap_oob 0f bf 96 ffff3cb0 ; asm: movzbl 50000(%ecx), %edi - [-,%rdi] v126 = uload8.i32 v1+50000 ; bin: 0f b6 b9 0000c350 + [-,%rdi] v126 = uload8.i32 v1+50000 ; bin: heap_oob 0f b6 b9 0000c350 ; asm: movzbl -50000(%esi), %edx - [-,%rdx] v127 = uload8.i32 v2-50000 ; bin: 0f b6 96 ffff3cb0 + [-,%rdx] v127 = uload8.i32 v2-50000 ; bin: heap_oob 0f b6 96 ffff3cb0 ; asm: movsbl 50000(%ecx), %edi - [-,%rdi] v128 = sload8.i32 v1+50000 ; bin: 0f be b9 0000c350 + [-,%rdi] v128 = sload8.i32 v1+50000 ; bin: heap_oob 0f be b9 0000c350 ; asm: movsbl -50000(%esi), %edx - [-,%rdx] v129 = sload8.i32 v2-50000 ; bin: 0f be 96 ffff3cb0 + [-,%rdx] v129 = sload8.i32 v2-50000 ; bin: heap_oob 0f be 96 ffff3cb0 ; Bit-counting instructions. @@ -437,7 +437,7 @@ ebb1: ; asm: ebb2: ebb2: - trap user0 ; bin: 0f 0b + trap user0 ; bin: user0 0f 0b } ; Special branch encodings only for I32 mode. @@ -524,25 +524,25 @@ ebb1: ; The trapif instructions are encoded as macros: a conditional jump over a ud2. ; asm: jne .+4; ud2 - trapif eq v11, user0 ; bin: 75 02 0f 0b + trapif eq v11, user0 ; bin: 75 02 user0 0f 0b ; asm: je .+4; ud2 - trapif ne v11, user0 ; bin: 74 02 0f 0b + trapif ne v11, user0 ; bin: 74 02 user0 0f 0b ; asm: jnl .+4; ud2 - trapif slt v11, user0 ; bin: 7d 02 0f 0b + trapif slt v11, user0 ; bin: 7d 02 user0 0f 0b ; asm: jnge .+4; ud2 - trapif sge v11, user0 ; bin: 7c 02 0f 0b + trapif sge v11, user0 ; bin: 7c 02 user0 0f 0b ; asm: jng .+4; ud2 - trapif sgt v11, user0 ; bin: 7e 02 0f 0b + trapif sgt v11, user0 ; bin: 7e 02 user0 0f 0b ; asm: jnle .+4; ud2 - trapif sle v11, user0 ; bin: 7f 02 0f 0b + trapif sle v11, user0 ; bin: 7f 02 user0 0f 0b ; asm: jnb .+4; ud2 - trapif ult v11, user0 ; bin: 73 02 0f 0b + trapif ult v11, user0 ; bin: 73 02 user0 0f 0b ; asm: jnae .+4; ud2 - trapif uge v11, user0 ; bin: 72 02 0f 0b + trapif uge v11, user0 ; bin: 72 02 user0 0f 0b ; asm: jna .+4; ud2 - trapif ugt v11, user0 ; bin: 76 02 0f 0b + trapif ugt v11, user0 ; bin: 76 02 user0 0f 0b ; asm: jnbe .+4; ud2 - trapif ule v11, user0 ; bin: 77 02 0f 0b + trapif ule v11, user0 ; bin: 77 02 user0 0f 0b ; Stack check. ; asm: cmpl %esp, %ecx @@ -576,7 +576,7 @@ ebb0: ; asm: movzbl %cl, %esi [-,%rsi] v30 = uextend.i32 v11 ; bin: 0f b6 f1 - trap user0 ; bin: 0f 0b + trap user0 ; bin: user0 0f 0b } ; Tests for i32/i16 conversion instructions. @@ -592,5 +592,5 @@ ebb0: ; asm: movzwl %cx, %esi [-,%rsi] v30 = uextend.i32 v11 ; bin: 0f b7 f1 - trap user0 ; bin: 0f 0b + trap user0 ; bin: user0 0f 0b } diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/intel/binary64-float.cton index 47db5b197f..7053a1809e 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/intel/binary64-float.cton @@ -158,34 +158,34 @@ ebb0: ; Load/Store ; asm: movss (%r14), %xmm5 - [-,%xmm5] v100 = load.f32 v3 ; bin: f3 41 0f 10 2e + [-,%xmm5] v100 = load.f32 v3 ; bin: heap_oob f3 41 0f 10 2e ; asm: movss (%rax), %xmm10 - [-,%xmm10] v101 = load.f32 v2 ; bin: f3 44 0f 10 10 + [-,%xmm10] v101 = load.f32 v2 ; bin: heap_oob f3 44 0f 10 10 ; asm: movss 50(%r14), %xmm5 - [-,%xmm5] v110 = load.f32 v3+50 ; bin: f3 41 0f 10 6e 32 + [-,%xmm5] v110 = load.f32 v3+50 ; bin: heap_oob f3 41 0f 10 6e 32 ; asm: movss -50(%rax), %xmm10 - [-,%xmm10] v111 = load.f32 v2-50 ; bin: f3 44 0f 10 50 ce + [-,%xmm10] v111 = load.f32 v2-50 ; bin: heap_oob f3 44 0f 10 50 ce ; asm: movss 10000(%r14), %xmm5 - [-,%xmm5] v120 = load.f32 v3+10000 ; bin: f3 41 0f 10 ae 00002710 + [-,%xmm5] v120 = load.f32 v3+10000 ; bin: heap_oob f3 41 0f 10 ae 00002710 ; asm: movss -10000(%rax), %xmm10 - [-,%xmm10] v121 = load.f32 v2-10000 ; bin: f3 44 0f 10 90 ffffd8f0 + [-,%xmm10] v121 = load.f32 v2-10000 ; bin: heap_oob f3 44 0f 10 90 ffffd8f0 ; asm: movss %xmm5, (%r14) - [-] store.f32 v100, v3 ; bin: f3 41 0f 11 2e + [-] store.f32 v100, v3 ; bin: heap_oob f3 41 0f 11 2e ; asm: movss %xmm10, (%rax) - [-] store.f32 v101, v2 ; bin: f3 44 0f 11 10 + [-] store.f32 v101, v2 ; bin: heap_oob f3 44 0f 11 10 ; asm: movss %xmm5, (%r13) - [-] store.f32 v100, v4 ; bin: f3 41 0f 11 6d 00 + [-] store.f32 v100, v4 ; bin: heap_oob f3 41 0f 11 6d 00 ; asm: movss %xmm10, (%r13) - [-] store.f32 v101, v4 ; bin: f3 45 0f 11 55 00 + [-] store.f32 v101, v4 ; bin: heap_oob f3 45 0f 11 55 00 ; asm: movss %xmm5, 50(%r14) - [-] store.f32 v100, v3+50 ; bin: f3 41 0f 11 6e 32 + [-] store.f32 v100, v3+50 ; bin: heap_oob f3 41 0f 11 6e 32 ; asm: movss %xmm10, -50(%rax) - [-] store.f32 v101, v2-50 ; bin: f3 44 0f 11 50 ce + [-] store.f32 v101, v2-50 ; bin: heap_oob f3 44 0f 11 50 ce ; asm: movss %xmm5, 10000(%r14) - [-] store.f32 v100, v3+10000 ; bin: f3 41 0f 11 ae 00002710 + [-] store.f32 v100, v3+10000 ; bin: heap_oob f3 41 0f 11 ae 00002710 ; asm: movss %xmm10, -10000(%rax) - [-] store.f32 v101, v2-10000 ; bin: f3 44 0f 11 90 ffffd8f0 + [-] store.f32 v101, v2-10000 ; bin: heap_oob f3 44 0f 11 90 ffffd8f0 ; Spill / Fill. @@ -393,34 +393,34 @@ ebb0: ; Load/Store ; asm: movsd (%r14), %xmm5 - [-,%xmm5] v100 = load.f64 v3 ; bin: f2 41 0f 10 2e + [-,%xmm5] v100 = load.f64 v3 ; bin: heap_oob f2 41 0f 10 2e ; asm: movsd (%rax), %xmm10 - [-,%xmm10] v101 = load.f64 v2 ; bin: f2 44 0f 10 10 + [-,%xmm10] v101 = load.f64 v2 ; bin: heap_oob f2 44 0f 10 10 ; asm: movsd 50(%r14), %xmm5 - [-,%xmm5] v110 = load.f64 v3+50 ; bin: f2 41 0f 10 6e 32 + [-,%xmm5] v110 = load.f64 v3+50 ; bin: heap_oob f2 41 0f 10 6e 32 ; asm: movsd -50(%rax), %xmm10 - [-,%xmm10] v111 = load.f64 v2-50 ; bin: f2 44 0f 10 50 ce + [-,%xmm10] v111 = load.f64 v2-50 ; bin: heap_oob f2 44 0f 10 50 ce ; asm: movsd 10000(%r14), %xmm5 - [-,%xmm5] v120 = load.f64 v3+10000 ; bin: f2 41 0f 10 ae 00002710 + [-,%xmm5] v120 = load.f64 v3+10000 ; bin: heap_oob f2 41 0f 10 ae 00002710 ; asm: movsd -10000(%rax), %xmm10 - [-,%xmm10] v121 = load.f64 v2-10000 ; bin: f2 44 0f 10 90 ffffd8f0 + [-,%xmm10] v121 = load.f64 v2-10000 ; bin: heap_oob f2 44 0f 10 90 ffffd8f0 ; asm: movsd %xmm5, (%r14) - [-] store.f64 v100, v3 ; bin: f2 41 0f 11 2e + [-] store.f64 v100, v3 ; bin: heap_oob f2 41 0f 11 2e ; asm: movsd %xmm10, (%rax) - [-] store.f64 v101, v2 ; bin: f2 44 0f 11 10 + [-] store.f64 v101, v2 ; bin: heap_oob f2 44 0f 11 10 ; asm: movsd %xmm5, (%r13) - [-] store.f64 v100, v4 ; bin: f2 41 0f 11 6d 00 + [-] store.f64 v100, v4 ; bin: heap_oob f2 41 0f 11 6d 00 ; asm: movsd %xmm10, (%r13) - [-] store.f64 v101, v4 ; bin: f2 45 0f 11 55 00 + [-] store.f64 v101, v4 ; bin: heap_oob f2 45 0f 11 55 00 ; asm: movsd %xmm5, 50(%r14) - [-] store.f64 v100, v3+50 ; bin: f2 41 0f 11 6e 32 + [-] store.f64 v100, v3+50 ; bin: heap_oob f2 41 0f 11 6e 32 ; asm: movsd %xmm10, -50(%rax) - [-] store.f64 v101, v2-50 ; bin: f2 44 0f 11 50 ce + [-] store.f64 v101, v2-50 ; bin: heap_oob f2 44 0f 11 50 ce ; asm: movsd %xmm5, 10000(%r14) - [-] store.f64 v100, v3+10000 ; bin: f2 41 0f 11 ae 00002710 + [-] store.f64 v100, v3+10000 ; bin: heap_oob f2 41 0f 11 ae 00002710 ; asm: movsd %xmm10, -10000(%rax) - [-] store.f64 v101, v2-10000 ; bin: f2 44 0f 11 90 ffffd8f0 + [-] store.f64 v101, v2-10000 ; bin: heap_oob f2 44 0f 11 90 ffffd8f0 ; Spill / Fill. @@ -505,21 +505,21 @@ ebb1: brff ule v1, ebb1 ; bin: 76 f0 ; asm: jp .+4; ud2 - trapff ord v1, user0 ; bin: 7a 02 0f 0b + trapff ord v1, user0 ; bin: 7a 02 user0 0f 0b ; asm: jnp .+4; ud2 - trapff uno v1, user0 ; bin: 7b 02 0f 0b + trapff uno v1, user0 ; bin: 7b 02 user0 0f 0b ; asm: je .+4; ud2 - trapff one v1, user0 ; bin: 74 02 0f 0b + trapff one v1, user0 ; bin: 74 02 user0 0f 0b ; asm: jne .+4; ud2 - trapff ueq v1, user0 ; bin: 75 02 0f 0b + trapff ueq v1, user0 ; bin: 75 02 user0 0f 0b ; asm: jna .+4; ud2 - trapff gt v1, user0 ; bin: 76 02 0f 0b + trapff gt v1, user0 ; bin: 76 02 user0 0f 0b ; asm: jnae .+4; ud2 - trapff ge v1, user0 ; bin: 72 02 0f 0b + trapff ge v1, user0 ; bin: 72 02 user0 0f 0b ; asm: jnb .+4; ud2 - trapff ult v1, user0 ; bin: 73 02 0f 0b + trapff ult v1, user0 ; bin: 73 02 user0 0f 0b ; asm: jnbe .+4; ud2 - trapff ule v1, user0 ; bin: 77 02 0f 0b + trapff ule v1, user0 ; bin: 77 02 user0 0f 0b ; asm: setnp %bl [-,%rbx] v10 = trueff ord v1 ; bin: 0f 9b c3 diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index f7aab2f8f4..1444c1e062 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -175,146 +175,146 @@ ebb0: ; Register indirect addressing with no displacement. ; asm: movq %rcx, (%r10) - store v1, v3 ; bin: 49 89 0a + store v1, v3 ; bin: heap_oob 49 89 0a ; asm: movq %r10, (%rcx) - store v3, v1 ; bin: 4c 89 11 + store v3, v1 ; bin: heap_oob 4c 89 11 ; asm: movl %ecx, (%r10) - istore32 v1, v3 ; bin: 41 89 0a + istore32 v1, v3 ; bin: heap_oob 41 89 0a ; asm: movl %r10d, (%rcx) - istore32 v3, v1 ; bin: 44 89 11 + istore32 v3, v1 ; bin: heap_oob 44 89 11 ; asm: movw %cx, (%r10) - istore16 v1, v3 ; bin: 66 41 89 0a + istore16 v1, v3 ; bin: heap_oob 66 41 89 0a ; asm: movw %r10w, (%rcx) - istore16 v3, v1 ; bin: 66 44 89 11 + istore16 v3, v1 ; bin: heap_oob 66 44 89 11 ; asm: movb %cl, (%r10) - istore8 v1, v3 ; bin: 41 88 0a + istore8 v1, v3 ; bin: heap_oob 41 88 0a ; asm: movb %r10b, (%rcx) - istore8 v3, v1 ; bin: 44 88 11 + istore8 v3, v1 ; bin: heap_oob 44 88 11 ; asm: movq (%rcx), %r14 - [-,%r14] v120 = load.i64 v1 ; bin: 4c 8b 31 + [-,%r14] v120 = load.i64 v1 ; bin: heap_oob 4c 8b 31 ; asm: movq (%r10), %rdx - [-,%rdx] v121 = load.i64 v3 ; bin: 49 8b 12 + [-,%rdx] v121 = load.i64 v3 ; bin: heap_oob 49 8b 12 ; asm: movl (%rcx), %r14d - [-,%r14] v122 = uload32.i64 v1 ; bin: 44 8b 31 + [-,%r14] v122 = uload32.i64 v1 ; bin: heap_oob 44 8b 31 ; asm: movl (%r10), %edx - [-,%rdx] v123 = uload32.i64 v3 ; bin: 41 8b 12 + [-,%rdx] v123 = uload32.i64 v3 ; bin: heap_oob 41 8b 12 ; asm: movslq (%rcx), %r14 - [-,%r14] v124 = sload32.i64 v1 ; bin: 4c 63 31 + [-,%r14] v124 = sload32.i64 v1 ; bin: heap_oob 4c 63 31 ; asm: movslq (%r10), %rdx - [-,%rdx] v125 = sload32.i64 v3 ; bin: 49 63 12 + [-,%rdx] v125 = sload32.i64 v3 ; bin: heap_oob 49 63 12 ; asm: movzwq (%rcx), %r14 - [-,%r14] v126 = uload16.i64 v1 ; bin: 4c 0f b7 31 + [-,%r14] v126 = uload16.i64 v1 ; bin: heap_oob 4c 0f b7 31 ; asm: movzwq (%r10), %rdx - [-,%rdx] v127 = uload16.i64 v3 ; bin: 49 0f b7 12 + [-,%rdx] v127 = uload16.i64 v3 ; bin: heap_oob 49 0f b7 12 ; asm: movswq (%rcx), %r14 - [-,%r14] v128 = sload16.i64 v1 ; bin: 4c 0f bf 31 + [-,%r14] v128 = sload16.i64 v1 ; bin: heap_oob 4c 0f bf 31 ; asm: movswq (%r10), %rdx - [-,%rdx] v129 = sload16.i64 v3 ; bin: 49 0f bf 12 + [-,%rdx] v129 = sload16.i64 v3 ; bin: heap_oob 49 0f bf 12 ; asm: movzbq (%rcx), %r14 - [-,%r14] v130 = uload8.i64 v1 ; bin: 4c 0f b6 31 + [-,%r14] v130 = uload8.i64 v1 ; bin: heap_oob 4c 0f b6 31 ; asm: movzbq (%r10), %rdx - [-,%rdx] v131 = uload8.i64 v3 ; bin: 49 0f b6 12 + [-,%rdx] v131 = uload8.i64 v3 ; bin: heap_oob 49 0f b6 12 ; asm: movsbq (%rcx), %r14 - [-,%r14] v132 = sload8.i64 v1 ; bin: 4c 0f be 31 + [-,%r14] v132 = sload8.i64 v1 ; bin: heap_oob 4c 0f be 31 ; asm: movsbq (%r10), %rdx - [-,%rdx] v133 = sload8.i64 v3 ; bin: 49 0f be 12 + [-,%rdx] v133 = sload8.i64 v3 ; bin: heap_oob 49 0f be 12 ; Register-indirect with 8-bit signed displacement. ; asm: movq %rcx, 100(%r10) - store v1, v3+100 ; bin: 49 89 4a 64 + store v1, v3+100 ; bin: heap_oob 49 89 4a 64 ; asm: movq %r10, -100(%rcx) - store v3, v1-100 ; bin: 4c 89 51 9c + store v3, v1-100 ; bin: heap_oob 4c 89 51 9c ; asm: movl %ecx, 100(%r10) - istore32 v1, v3+100 ; bin: 41 89 4a 64 + istore32 v1, v3+100 ; bin: heap_oob 41 89 4a 64 ; asm: movl %r10d, -100(%rcx) - istore32 v3, v1-100 ; bin: 44 89 51 9c + istore32 v3, v1-100 ; bin: heap_oob 44 89 51 9c ; asm: movw %cx, 100(%r10) - istore16 v1, v3+100 ; bin: 66 41 89 4a 64 + istore16 v1, v3+100 ; bin: heap_oob 66 41 89 4a 64 ; asm: movw %r10w, -100(%rcx) - istore16 v3, v1-100 ; bin: 66 44 89 51 9c + istore16 v3, v1-100 ; bin: heap_oob 66 44 89 51 9c ; asm: movb %cl, 100(%r10) - istore8 v1, v3+100 ; bin: 41 88 4a 64 + istore8 v1, v3+100 ; bin: heap_oob 41 88 4a 64 ; asm: movb %r10b, 100(%rcx) - istore8 v3, v1+100 ; bin: 44 88 51 64 + istore8 v3, v1+100 ; bin: heap_oob 44 88 51 64 ; asm: movq 50(%rcx), %r10 - [-,%r10] v140 = load.i64 v1+50 ; bin: 4c 8b 51 32 + [-,%r10] v140 = load.i64 v1+50 ; bin: heap_oob 4c 8b 51 32 ; asm: movq -50(%r10), %rdx - [-,%rdx] v141 = load.i64 v3-50 ; bin: 49 8b 52 ce + [-,%rdx] v141 = load.i64 v3-50 ; bin: heap_oob 49 8b 52 ce ; asm: movl 50(%rcx), %edi - [-,%rdi] v142 = uload32.i64 v1+50 ; bin: 8b 79 32 + [-,%rdi] v142 = uload32.i64 v1+50 ; bin: heap_oob 8b 79 32 ; asm: movl -50(%rsi), %edx - [-,%rdx] v143 = uload32.i64 v2-50 ; bin: 8b 56 ce + [-,%rdx] v143 = uload32.i64 v2-50 ; bin: heap_oob 8b 56 ce ; asm: movslq 50(%rcx), %rdi - [-,%rdi] v144 = sload32.i64 v1+50 ; bin: 48 63 79 32 + [-,%rdi] v144 = sload32.i64 v1+50 ; bin: heap_oob 48 63 79 32 ; asm: movslq -50(%rsi), %rdx - [-,%rdx] v145 = sload32.i64 v2-50 ; bin: 48 63 56 ce + [-,%rdx] v145 = sload32.i64 v2-50 ; bin: heap_oob 48 63 56 ce ; asm: movzwq 50(%rcx), %rdi - [-,%rdi] v146 = uload16.i64 v1+50 ; bin: 48 0f b7 79 32 + [-,%rdi] v146 = uload16.i64 v1+50 ; bin: heap_oob 48 0f b7 79 32 ; asm: movzwq -50(%rsi), %rdx - [-,%rdx] v147 = uload16.i64 v2-50 ; bin: 48 0f b7 56 ce + [-,%rdx] v147 = uload16.i64 v2-50 ; bin: heap_oob 48 0f b7 56 ce ; asm: movswq 50(%rcx), %rdi - [-,%rdi] v148 = sload16.i64 v1+50 ; bin: 48 0f bf 79 32 + [-,%rdi] v148 = sload16.i64 v1+50 ; bin: heap_oob 48 0f bf 79 32 ; asm: movswq -50(%rsi), %rdx - [-,%rdx] v149 = sload16.i64 v2-50 ; bin: 48 0f bf 56 ce + [-,%rdx] v149 = sload16.i64 v2-50 ; bin: heap_oob 48 0f bf 56 ce ; asm: movzbq 50(%rcx), %rdi - [-,%rdi] v150 = uload8.i64 v1+50 ; bin: 48 0f b6 79 32 + [-,%rdi] v150 = uload8.i64 v1+50 ; bin: heap_oob 48 0f b6 79 32 ; asm: movzbq -50(%rsi), %rdx - [-,%rdx] v151 = uload8.i64 v2-50 ; bin: 48 0f b6 56 ce + [-,%rdx] v151 = uload8.i64 v2-50 ; bin: heap_oob 48 0f b6 56 ce ; asm: movsbq 50(%rcx), %rdi - [-,%rdi] v152 = sload8.i64 v1+50 ; bin: 48 0f be 79 32 + [-,%rdi] v152 = sload8.i64 v1+50 ; bin: heap_oob 48 0f be 79 32 ; asm: movsbq -50(%rsi), %rdx - [-,%rdx] v153 = sload8.i64 v2-50 ; bin: 48 0f be 56 ce + [-,%rdx] v153 = sload8.i64 v2-50 ; bin: heap_oob 48 0f be 56 ce ; Register-indirect with 32-bit signed displacement. ; asm: movq %rcx, 10000(%r10) - store v1, v3+10000 ; bin: 49 89 8a 00002710 + store v1, v3+10000 ; bin: heap_oob 49 89 8a 00002710 ; asm: movq %r10, -10000(%rcx) - store v3, v1-10000 ; bin: 4c 89 91 ffffd8f0 + store v3, v1-10000 ; bin: heap_oob 4c 89 91 ffffd8f0 ; asm: movl %ecx, 10000(%rsi) - istore32 v1, v2+10000 ; bin: 89 8e 00002710 + istore32 v1, v2+10000 ; bin: heap_oob 89 8e 00002710 ; asm: movl %esi, -10000(%rcx) - istore32 v2, v1-10000 ; bin: 89 b1 ffffd8f0 + istore32 v2, v1-10000 ; bin: heap_oob 89 b1 ffffd8f0 ; asm: movw %cx, 10000(%rsi) - istore16 v1, v2+10000 ; bin: 66 89 8e 00002710 + istore16 v1, v2+10000 ; bin: heap_oob 66 89 8e 00002710 ; asm: movw %si, -10000(%rcx) - istore16 v2, v1-10000 ; bin: 66 89 b1 ffffd8f0 + istore16 v2, v1-10000 ; bin: heap_oob 66 89 b1 ffffd8f0 ; asm: movb %cl, 10000(%rsi) - istore8 v1, v2+10000 ; bin: 88 8e 00002710 + istore8 v1, v2+10000 ; bin: heap_oob 88 8e 00002710 ; asm: movb %sil, 10000(%rcx) - istore8 v2, v1+10000 ; bin: 40 88 b1 00002710 + istore8 v2, v1+10000 ; bin: heap_oob 40 88 b1 00002710 ; asm: movq 50000(%rcx), %r10 - [-,%r10] v160 = load.i64 v1+50000 ; bin: 4c 8b 91 0000c350 + [-,%r10] v160 = load.i64 v1+50000 ; bin: heap_oob 4c 8b 91 0000c350 ; asm: movq -50000(%r10), %rdx - [-,%rdx] v161 = load.i64 v3-50000 ; bin: 49 8b 92 ffff3cb0 + [-,%rdx] v161 = load.i64 v3-50000 ; bin: heap_oob 49 8b 92 ffff3cb0 ; asm: movl 50000(%rcx), %edi - [-,%rdi] v162 = uload32.i64 v1+50000 ; bin: 8b b9 0000c350 + [-,%rdi] v162 = uload32.i64 v1+50000 ; bin: heap_oob 8b b9 0000c350 ; asm: movl -50000(%rsi), %edx - [-,%rdx] v163 = uload32.i64 v2-50000 ; bin: 8b 96 ffff3cb0 + [-,%rdx] v163 = uload32.i64 v2-50000 ; bin: heap_oob 8b 96 ffff3cb0 ; asm: movslq 50000(%rcx), %rdi - [-,%rdi] v164 = sload32.i64 v1+50000 ; bin: 48 63 b9 0000c350 + [-,%rdi] v164 = sload32.i64 v1+50000 ; bin: heap_oob 48 63 b9 0000c350 ; asm: movslq -50000(%rsi), %rdx - [-,%rdx] v165 = sload32.i64 v2-50000 ; bin: 48 63 96 ffff3cb0 + [-,%rdx] v165 = sload32.i64 v2-50000 ; bin: heap_oob 48 63 96 ffff3cb0 ; asm: movzwq 50000(%rcx), %rdi - [-,%rdi] v166 = uload16.i64 v1+50000 ; bin: 48 0f b7 b9 0000c350 + [-,%rdi] v166 = uload16.i64 v1+50000 ; bin: heap_oob 48 0f b7 b9 0000c350 ; asm: movzwq -50000(%rsi), %rdx - [-,%rdx] v167 = uload16.i64 v2-50000 ; bin: 48 0f b7 96 ffff3cb0 + [-,%rdx] v167 = uload16.i64 v2-50000 ; bin: heap_oob 48 0f b7 96 ffff3cb0 ; asm: movswq 50000(%rcx), %rdi - [-,%rdi] v168 = sload16.i64 v1+50000 ; bin: 48 0f bf b9 0000c350 + [-,%rdi] v168 = sload16.i64 v1+50000 ; bin: heap_oob 48 0f bf b9 0000c350 ; asm: movswq -50000(%rsi), %rdx - [-,%rdx] v169 = sload16.i64 v2-50000 ; bin: 48 0f bf 96 ffff3cb0 + [-,%rdx] v169 = sload16.i64 v2-50000 ; bin: heap_oob 48 0f bf 96 ffff3cb0 ; asm: movzbq 50000(%rcx), %rdi - [-,%rdi] v170 = uload8.i64 v1+50000 ; bin: 48 0f b6 b9 0000c350 + [-,%rdi] v170 = uload8.i64 v1+50000 ; bin: heap_oob 48 0f b6 b9 0000c350 ; asm: movzbq -50000(%rsi), %rdx - [-,%rdx] v171 = uload8.i64 v2-50000 ; bin: 48 0f b6 96 ffff3cb0 + [-,%rdx] v171 = uload8.i64 v2-50000 ; bin: heap_oob 48 0f b6 96 ffff3cb0 ; asm: movsbq 50000(%rcx), %rdi - [-,%rdi] v172 = sload8.i64 v1+50000 ; bin: 48 0f be b9 0000c350 + [-,%rdi] v172 = sload8.i64 v1+50000 ; bin: heap_oob 48 0f be b9 0000c350 ; asm: movsbq -50000(%rsi), %rdx - [-,%rdx] v173 = sload8.i64 v2-50000 ; bin: 48 0f be 96 ffff3cb0 + [-,%rdx] v173 = sload8.i64 v2-50000 ; bin: heap_oob 48 0f be 96 ffff3cb0 ; More arithmetic. @@ -329,17 +329,17 @@ ebb0: [-,%rax] v190 = iconst.i64 1 [-,%rdx] v191 = iconst.i64 2 ; asm: idivq %rcx - [-,%rax,%rdx] v192, v193 = x86_sdivmodx v190, v191, v1 ; bin: 48 f7 f9 + [-,%rax,%rdx] v192, v193 = x86_sdivmodx v190, v191, v1 ; bin: int_divz 48 f7 f9 ; asm: idivq %rsi - [-,%rax,%rdx] v194, v195 = x86_sdivmodx v190, v191, v2 ; bin: 48 f7 fe + [-,%rax,%rdx] v194, v195 = x86_sdivmodx v190, v191, v2 ; bin: int_divz 48 f7 fe ; asm: idivq %r10 - [-,%rax,%rdx] v196, v197 = x86_sdivmodx v190, v191, v3 ; bin: 49 f7 fa + [-,%rax,%rdx] v196, v197 = x86_sdivmodx v190, v191, v3 ; bin: int_divz 49 f7 fa ; asm: divq %rcx - [-,%rax,%rdx] v198, v199 = x86_udivmodx v190, v191, v1 ; bin: 48 f7 f1 + [-,%rax,%rdx] v198, v199 = x86_udivmodx v190, v191, v1 ; bin: int_divz 48 f7 f1 ; asm: divq %rsi - [-,%rax,%rdx] v200, v201 = x86_udivmodx v190, v191, v2 ; bin: 48 f7 f6 + [-,%rax,%rdx] v200, v201 = x86_udivmodx v190, v191, v2 ; bin: int_divz 48 f7 f6 ; asm: divq %r10 - [-,%rax,%rdx] v202, v203 = x86_udivmodx v190, v191, v3 ; bin: 49 f7 f2 + [-,%rax,%rdx] v202, v203 = x86_udivmodx v190, v191, v3 ; bin: int_divz 49 f7 f2 ; double-length multiply instructions, 64 bit [-,%rax] v1001 = iconst.i64 1 @@ -637,25 +637,25 @@ ebb1: ; The trapif instructions are encoded as macros: a conditional jump over a ud2. ; asm: jne .+4; ud2 - trapif eq v11, user0 ; bin: 75 02 0f 0b + trapif eq v11, user0 ; bin: 75 02 user0 0f 0b ; asm: je .+4; ud2 - trapif ne v11, user0 ; bin: 74 02 0f 0b + trapif ne v11, user0 ; bin: 74 02 user0 0f 0b ; asm: jnl .+4; ud2 - trapif slt v11, user0 ; bin: 7d 02 0f 0b + trapif slt v11, user0 ; bin: 7d 02 user0 0f 0b ; asm: jnge .+4; ud2 - trapif sge v11, user0 ; bin: 7c 02 0f 0b + trapif sge v11, user0 ; bin: 7c 02 user0 0f 0b ; asm: jng .+4; ud2 - trapif sgt v11, user0 ; bin: 7e 02 0f 0b + trapif sgt v11, user0 ; bin: 7e 02 user0 0f 0b ; asm: jnle .+4; ud2 - trapif sle v11, user0 ; bin: 7f 02 0f 0b + trapif sle v11, user0 ; bin: 7f 02 user0 0f 0b ; asm: jnb .+4; ud2 - trapif ult v11, user0 ; bin: 73 02 0f 0b + trapif ult v11, user0 ; bin: 73 02 user0 0f 0b ; asm: jnae .+4; ud2 - trapif uge v11, user0 ; bin: 72 02 0f 0b + trapif uge v11, user0 ; bin: 72 02 user0 0f 0b ; asm: jna .+4; ud2 - trapif ugt v11, user0 ; bin: 76 02 0f 0b + trapif ugt v11, user0 ; bin: 76 02 user0 0f 0b ; asm: jnbe .+4; ud2 - trapif ule v11, user0 ; bin: 77 02 0f 0b + trapif ule v11, user0 ; bin: 77 02 user0 0f 0b ; Stack check. ; asm: cmpq %rsp, %rcx @@ -729,71 +729,71 @@ ebb0: ; Register indirect addressing with no displacement. ; asm: movl (%rcx), %edi - [-,%rdi] v10 = load.i32 v1 ; bin: 8b 39 + [-,%rdi] v10 = load.i32 v1 ; bin: heap_oob 8b 39 ; asm: movl (%rsi), %edx - [-,%rdx] v11 = load.i32 v2 ; bin: 8b 16 + [-,%rdx] v11 = load.i32 v2 ; bin: heap_oob 8b 16 ; asm: movzwl (%rcx), %edi - [-,%rdi] v12 = uload16.i32 v1 ; bin: 0f b7 39 + [-,%rdi] v12 = uload16.i32 v1 ; bin: heap_oob 0f b7 39 ; asm: movzwl (%rsi), %edx - [-,%rdx] v13 = uload16.i32 v2 ; bin: 0f b7 16 + [-,%rdx] v13 = uload16.i32 v2 ; bin: heap_oob 0f b7 16 ; asm: movswl (%rcx), %edi - [-,%rdi] v14 = sload16.i32 v1 ; bin: 0f bf 39 + [-,%rdi] v14 = sload16.i32 v1 ; bin: heap_oob 0f bf 39 ; asm: movswl (%rsi), %edx - [-,%rdx] v15 = sload16.i32 v2 ; bin: 0f bf 16 + [-,%rdx] v15 = sload16.i32 v2 ; bin: heap_oob 0f bf 16 ; asm: movzbl (%rcx), %edi - [-,%rdi] v16 = uload8.i32 v1 ; bin: 0f b6 39 + [-,%rdi] v16 = uload8.i32 v1 ; bin: heap_oob 0f b6 39 ; asm: movzbl (%rsi), %edx - [-,%rdx] v17 = uload8.i32 v2 ; bin: 0f b6 16 + [-,%rdx] v17 = uload8.i32 v2 ; bin: heap_oob 0f b6 16 ; asm: movsbl (%rcx), %edi - [-,%rdi] v18 = sload8.i32 v1 ; bin: 0f be 39 + [-,%rdi] v18 = sload8.i32 v1 ; bin: heap_oob 0f be 39 ; asm: movsbl (%rsi), %edx - [-,%rdx] v19 = sload8.i32 v2 ; bin: 0f be 16 + [-,%rdx] v19 = sload8.i32 v2 ; bin: heap_oob 0f be 16 ; Register-indirect with 8-bit signed displacement. ; asm: movl 50(%rcx), %edi - [-,%rdi] v20 = load.i32 v1+50 ; bin: 8b 79 32 + [-,%rdi] v20 = load.i32 v1+50 ; bin: heap_oob 8b 79 32 ; asm: movl -50(%rsi), %edx - [-,%rdx] v21 = load.i32 v2-50 ; bin: 8b 56 ce + [-,%rdx] v21 = load.i32 v2-50 ; bin: heap_oob 8b 56 ce ; asm: movzwl 50(%rcx), %edi - [-,%rdi] v22 = uload16.i32 v1+50 ; bin: 0f b7 79 32 + [-,%rdi] v22 = uload16.i32 v1+50 ; bin: heap_oob 0f b7 79 32 ; asm: movzwl -50(%rsi), %edx - [-,%rdx] v23 = uload16.i32 v2-50 ; bin: 0f b7 56 ce + [-,%rdx] v23 = uload16.i32 v2-50 ; bin: heap_oob 0f b7 56 ce ; asm: movswl 50(%rcx), %edi - [-,%rdi] v24 = sload16.i32 v1+50 ; bin: 0f bf 79 32 + [-,%rdi] v24 = sload16.i32 v1+50 ; bin: heap_oob 0f bf 79 32 ; asm: movswl -50(%rsi), %edx - [-,%rdx] v25 = sload16.i32 v2-50 ; bin: 0f bf 56 ce + [-,%rdx] v25 = sload16.i32 v2-50 ; bin: heap_oob 0f bf 56 ce ; asm: movzbl 50(%rcx), %edi - [-,%rdi] v26 = uload8.i32 v1+50 ; bin: 0f b6 79 32 + [-,%rdi] v26 = uload8.i32 v1+50 ; bin: heap_oob 0f b6 79 32 ; asm: movzbl -50(%rsi), %edx - [-,%rdx] v27 = uload8.i32 v2-50 ; bin: 0f b6 56 ce + [-,%rdx] v27 = uload8.i32 v2-50 ; bin: heap_oob 0f b6 56 ce ; asm: movsbl 50(%rcx), %edi - [-,%rdi] v28 = sload8.i32 v1+50 ; bin: 0f be 79 32 + [-,%rdi] v28 = sload8.i32 v1+50 ; bin: heap_oob 0f be 79 32 ; asm: movsbl -50(%rsi), %edx - [-,%rdx] v29 = sload8.i32 v2-50 ; bin: 0f be 56 ce + [-,%rdx] v29 = sload8.i32 v2-50 ; bin: heap_oob 0f be 56 ce ; Register-indirect with 32-bit signed displacement. ; asm: movl 50000(%rcx), %edi - [-,%rdi] v30 = load.i32 v1+50000 ; bin: 8b b9 0000c350 + [-,%rdi] v30 = load.i32 v1+50000 ; bin: heap_oob 8b b9 0000c350 ; asm: movl -50000(%rsi), %edx - [-,%rdx] v31 = load.i32 v2-50000 ; bin: 8b 96 ffff3cb0 + [-,%rdx] v31 = load.i32 v2-50000 ; bin: heap_oob 8b 96 ffff3cb0 ; asm: movzwl 50000(%rcx), %edi - [-,%rdi] v32 = uload16.i32 v1+50000 ; bin: 0f b7 b9 0000c350 + [-,%rdi] v32 = uload16.i32 v1+50000 ; bin: heap_oob 0f b7 b9 0000c350 ; asm: movzwl -50000(%rsi), %edx - [-,%rdx] v33 = uload16.i32 v2-50000 ; bin: 0f b7 96 ffff3cb0 + [-,%rdx] v33 = uload16.i32 v2-50000 ; bin: heap_oob 0f b7 96 ffff3cb0 ; asm: movswl 50000(%rcx), %edi - [-,%rdi] v34 = sload16.i32 v1+50000 ; bin: 0f bf b9 0000c350 + [-,%rdi] v34 = sload16.i32 v1+50000 ; bin: heap_oob 0f bf b9 0000c350 ; asm: movswl -50000(%rsi), %edx - [-,%rdx] v35 = sload16.i32 v2-50000 ; bin: 0f bf 96 ffff3cb0 + [-,%rdx] v35 = sload16.i32 v2-50000 ; bin: heap_oob 0f bf 96 ffff3cb0 ; asm: movzbl 50000(%rcx), %edi - [-,%rdi] v36 = uload8.i32 v1+50000 ; bin: 0f b6 b9 0000c350 + [-,%rdi] v36 = uload8.i32 v1+50000 ; bin: heap_oob 0f b6 b9 0000c350 ; asm: movzbl -50000(%rsi), %edx - [-,%rdx] v37 = uload8.i32 v2-50000 ; bin: 0f b6 96 ffff3cb0 + [-,%rdx] v37 = uload8.i32 v2-50000 ; bin: heap_oob 0f b6 96 ffff3cb0 ; asm: movsbl 50000(%rcx), %edi - [-,%rdi] v38 = sload8.i32 v1+50000 ; bin: 0f be b9 0000c350 + [-,%rdi] v38 = sload8.i32 v1+50000 ; bin: heap_oob 0f be b9 0000c350 ; asm: movsbl -50000(%rsi), %edx - [-,%rdx] v39 = sload8.i32 v2-50000 ; bin: 0f be 96 ffff3cb0 + [-,%rdx] v39 = sload8.i32 v2-50000 ; bin: heap_oob 0f be 96 ffff3cb0 ; Integer Register-Register Operations. @@ -924,17 +924,17 @@ ebb0: [-,%rax] v160 = iconst.i32 1 [-,%rdx] v161 = iconst.i32 2 ; asm: idivl %ecx - [-,%rax,%rdx] v162, v163 = x86_sdivmodx v160, v161, v1 ; bin: f7 f9 + [-,%rax,%rdx] v162, v163 = x86_sdivmodx v160, v161, v1 ; bin: int_divz f7 f9 ; asm: idivl %esi - [-,%rax,%rdx] v164, v165 = x86_sdivmodx v160, v161, v2 ; bin: f7 fe + [-,%rax,%rdx] v164, v165 = x86_sdivmodx v160, v161, v2 ; bin: int_divz f7 fe ; asm: idivl %r10d - [-,%rax,%rdx] v166, v167 = x86_sdivmodx v160, v161, v3 ; bin: 41 f7 fa + [-,%rax,%rdx] v166, v167 = x86_sdivmodx v160, v161, v3 ; bin: int_divz 41 f7 fa ; asm: divl %ecx - [-,%rax,%rdx] v168, v169 = x86_udivmodx v160, v161, v1 ; bin: f7 f1 + [-,%rax,%rdx] v168, v169 = x86_udivmodx v160, v161, v1 ; bin: int_divz f7 f1 ; asm: divl %esi - [-,%rax,%rdx] v170, v171 = x86_udivmodx v160, v161, v2 ; bin: f7 f6 + [-,%rax,%rdx] v170, v171 = x86_udivmodx v160, v161, v2 ; bin: int_divz f7 f6 ; asm: divl %r10d - [-,%rax,%rdx] v172, v173 = x86_udivmodx v160, v161, v3 ; bin: 41 f7 f2 + [-,%rax,%rdx] v172, v173 = x86_udivmodx v160, v161, v3 ; bin: int_divz 41 f7 f2 ; Bit-counting instructions. @@ -1144,7 +1144,7 @@ ebb0: ; asm: movzbl %r10b, %ecx [-,%rcx] v32 = uextend.i32 v13 ; bin: 41 0f b6 ca - trap user0 ; bin: 0f 0b + trap user0 ; bin: user0 0f 0b } ; Tests for i32/i16 conversion instructions. @@ -1172,7 +1172,7 @@ ebb0: ; asm: movzwl %r10w, %ecx [-,%rcx] v32 = uextend.i32 v13 ; bin: 41 0f b7 ca - trap user0 ; bin: 0f 0b + trap user0 ; bin: user0 0f 0b } ; Tests for i64/i8 conversion instructions. @@ -1200,7 +1200,7 @@ ebb0: ; asm: movzbl %r10b, %ecx [-,%rcx] v32 = uextend.i64 v13 ; bin: 41 0f b6 ca - trap user0 ; bin: 0f 0b + trap user0 ; bin: user0 0f 0b } ; Tests for i64/i16 conversion instructions. @@ -1228,7 +1228,7 @@ ebb0: ; asm: movzwl %r10w, %ecx [-,%rcx] v32 = uextend.i64 v13 ; bin: 41 0f b7 ca - trap user0 ; bin: 0f 0b + trap user0 ; bin: user0 0f 0b } ; Tests for i64/i32 conversion instructions. @@ -1256,5 +1256,5 @@ ebb0: ; asm: movl %r10d, %ecx [-,%rcx] v32 = uextend.i64 v13 ; bin: 44 89 d1 - trap user0 ; bin: 0f 0b + trap user0 ; bin: user0 0f 0b } diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index c21b989149..ccc049fa70 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -44,6 +44,18 @@ impl binemit::RelocSink for PrintRelocs { } } +struct PrintTraps { + flag_print: bool, +} + +impl binemit::TrapSink for PrintTraps { + fn trap(&mut self, offset: binemit::CodeOffset, _srcloc: ir::SourceLoc, code: ir::TrapCode) { + if self.flag_print { + println!("trap: {} at {}", code, offset); + } + } +} + pub fn run( files: Vec, flag_print: bool, @@ -94,8 +106,9 @@ fn handle_module( // Encode the result as machine code. let mut mem = Vec::new(); let mut relocs = PrintRelocs { flag_print }; + let mut traps = PrintTraps { flag_print }; mem.resize(size as usize, 0); - context.emit_to_memory(mem.as_mut_ptr(), &mut relocs, &*isa); + context.emit_to_memory(mem.as_mut_ptr(), &mut relocs, &mut traps, &*isa); if flag_print { print!(".byte "); diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 2406ef6ade..bb3db3cba1 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -278,7 +278,10 @@ null = EncRecipe('null', Unary, size=0, ins=GPR, outs=0, emit='') # XX opcode, no ModR/M. trap = TailRecipe( 'trap', Trap, size=0, ins=(), outs=(), - emit='PUT_OP(bits, BASE_REX, sink);') + emit=''' + sink.trap(code, func.srclocs[inst]); + PUT_OP(bits, BASE_REX, sink); + ''') # Macro: conditional jump over a ud2. trapif = EncRecipe( @@ -289,6 +292,7 @@ trapif = EncRecipe( sink.put1(0x70 | (icc2opc(cond.inverse()) as u8)); sink.put1(2); // ud2. + sink.trap(code, func.srclocs[inst]); sink.put1(0x0f); sink.put1(0x0b); ''') @@ -302,6 +306,7 @@ trapff = EncRecipe( sink.put1(0x70 | (fcc2opc(cond.inverse()) as u8)); sink.put1(2); // ud2. + sink.trap(code, func.srclocs[inst]); sink.put1(0x0f); sink.put1(0x0b); ''') @@ -450,6 +455,7 @@ div = TailRecipe( 'div', Ternary, size=1, ins=(GPR.rax, GPR.rdx, GPR), outs=(GPR.rax, GPR.rdx), emit=''' + sink.trap(TrapCode::IntegerDivisionByZero, func.srclocs[inst]); PUT_OP(bits, rex1(in_reg2), sink); modrm_r_bits(in_reg2, bits, sink); ''') @@ -678,6 +684,9 @@ st = TailRecipe( instp=IsEqual(Store.offset, 0), clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_rm(in_reg1, in_reg0, sink); ''') @@ -690,6 +699,9 @@ st_abcd = TailRecipe( when_prefixed=st, clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_rm(in_reg1, in_reg0, sink); ''') @@ -700,6 +712,9 @@ fst = TailRecipe( instp=IsEqual(Store.offset, 0), clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_rm(in_reg1, in_reg0, sink); ''') @@ -710,6 +725,9 @@ stDisp8 = TailRecipe( instp=IsSignedInt(Store.offset, 8), clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp8(in_reg1, in_reg0, sink); let offset: i32 = offset.into(); @@ -721,6 +739,9 @@ stDisp8_abcd = TailRecipe( when_prefixed=stDisp8, clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp8(in_reg1, in_reg0, sink); let offset: i32 = offset.into(); @@ -731,6 +752,9 @@ fstDisp8 = TailRecipe( instp=IsSignedInt(Store.offset, 8), clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp8(in_reg1, in_reg0, sink); let offset: i32 = offset.into(); @@ -742,6 +766,9 @@ stDisp32 = TailRecipe( 'stDisp32', Store, size=5, ins=(GPR, GPR_DEREF_SAFE), outs=(), clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp32(in_reg1, in_reg0, sink); let offset: i32 = offset.into(); @@ -752,6 +779,9 @@ stDisp32_abcd = TailRecipe( when_prefixed=stDisp32, clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp32(in_reg1, in_reg0, sink); let offset: i32 = offset.into(); @@ -761,6 +791,9 @@ fstDisp32 = TailRecipe( 'fstDisp32', Store, size=5, ins=(FPR, GPR_DEREF_SAFE), outs=(), clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_disp32(in_reg1, in_reg0, sink); let offset: i32 = offset.into(); @@ -827,6 +860,9 @@ ld = TailRecipe( instp=IsEqual(Load.offset, 0), clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_rm(in_reg0, out_reg0, sink); ''') @@ -837,6 +873,9 @@ fld = TailRecipe( instp=IsEqual(Load.offset, 0), clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_rm(in_reg0, out_reg0, sink); ''') @@ -847,6 +886,9 @@ ldDisp8 = TailRecipe( instp=IsSignedInt(Load.offset, 8), clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_disp8(in_reg0, out_reg0, sink); let offset: i32 = offset.into(); @@ -859,6 +901,9 @@ fldDisp8 = TailRecipe( instp=IsSignedInt(Load.offset, 8), clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_disp8(in_reg0, out_reg0, sink); let offset: i32 = offset.into(); @@ -871,6 +916,9 @@ ldDisp32 = TailRecipe( instp=IsSignedInt(Load.offset, 32), clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_disp32(in_reg0, out_reg0, sink); let offset: i32 = offset.into(); @@ -883,6 +931,9 @@ fldDisp32 = TailRecipe( instp=IsSignedInt(Load.offset, 32), clobbers_flags=False, emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_disp32(in_reg0, out_reg0, sink); let offset: i32 = offset.into(); diff --git a/lib/cretonne/src/binemit/memorysink.rs b/lib/cretonne/src/binemit/memorysink.rs index 506ea8b5a4..d4b07319ab 100644 --- a/lib/cretonne/src/binemit/memorysink.rs +++ b/lib/cretonne/src/binemit/memorysink.rs @@ -14,7 +14,7 @@ //! relocations to a `RelocSink` trait object. Relocations are less frequent than the //! `CodeSink::put*` methods, so the performance impact of the virtual callbacks is less severe. -use ir::{ExternalName, JumpTable}; +use ir::{ExternalName, JumpTable, TrapCode, SourceLoc}; use super::{CodeSink, CodeOffset, Reloc, Addend}; use std::ptr::write_unaligned; @@ -33,15 +33,21 @@ pub struct MemoryCodeSink<'a> { data: *mut u8, offset: isize, relocs: &'a mut RelocSink, + traps: &'a mut TrapSink, } impl<'a> MemoryCodeSink<'a> { /// Create a new memory code sink that writes a function to the memory pointed to by `data`. - pub fn new(data: *mut u8, relocs: &mut RelocSink) -> MemoryCodeSink { + pub fn new<'sink>( + data: *mut u8, + relocs: &'sink mut RelocSink, + traps: &'sink mut TrapSink, + ) -> MemoryCodeSink<'sink> { MemoryCodeSink { data, offset: 0, relocs, + traps, } } } @@ -58,6 +64,12 @@ pub trait RelocSink { fn reloc_jt(&mut self, CodeOffset, Reloc, JumpTable); } +/// A trait for receiving trap codes and offsets. +pub trait TrapSink { + /// Add trap information for a specific offset. + fn trap(&mut self, CodeOffset, SourceLoc, TrapCode); +} + impl<'a> CodeSink for MemoryCodeSink<'a> { fn offset(&self) -> CodeOffset { self.offset as CodeOffset @@ -105,4 +117,9 @@ impl<'a> CodeSink for MemoryCodeSink<'a> { let ofs = self.offset(); self.relocs.reloc_jt(ofs, rel, jt); } + + fn trap(&mut self, code: TrapCode, srcloc: SourceLoc) { + let ofs = self.offset(); + self.traps.trap(ofs, srcloc, code); + } } diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index bca1f6fa4d..8f6f3c2924 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -8,9 +8,9 @@ mod memorysink; pub use regalloc::RegDiversions; pub use self::relaxation::relax_branches; -pub use self::memorysink::{MemoryCodeSink, RelocSink}; +pub use self::memorysink::{MemoryCodeSink, RelocSink, TrapSink}; -use ir::{ExternalName, JumpTable, Function, Inst}; +use ir::{ExternalName, JumpTable, Function, Inst, TrapCode, SourceLoc}; use std::fmt; /// Offset in bytes from the beginning of the function. @@ -86,6 +86,9 @@ pub trait CodeSink { /// Add a relocation referencing a jump table. fn reloc_jt(&mut self, Reloc, JumpTable); + + /// Add trap information for the current offset. + fn trap(&mut self, TrapCode, SourceLoc); } /// Report a bad encoding error. diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 54fce4b84a..92ea61c643 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -9,7 +9,7 @@ //! contexts concurrently. Typically, you would have one context per compilation thread and only a //! single ISA instance. -use binemit::{CodeOffset, relax_branches, MemoryCodeSink, RelocSink}; +use binemit::{CodeOffset, relax_branches, MemoryCodeSink, RelocSink, TrapSink}; use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir::Function; @@ -111,9 +111,15 @@ impl Context { /// code is returned by `compile` above. /// /// The machine code is not relocated. Instead, any relocations are emitted into `relocs`. - pub fn emit_to_memory(&self, mem: *mut u8, relocs: &mut RelocSink, isa: &TargetIsa) { + pub fn emit_to_memory( + &self, + mem: *mut u8, + relocs: &mut RelocSink, + traps: &mut TrapSink, + isa: &TargetIsa, + ) { let _tt = timing::binemit(); - isa.emit_function(&self.func, &mut MemoryCodeSink::new(mem, relocs)); + isa.emit_function(&self.func, &mut MemoryCodeSink::new(mem, relocs, traps)); } /// Run the verifier on the function. diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index ddd3405e18..5f2f9512ce 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -1,7 +1,7 @@ //! Emitting binary Intel machine code. use binemit::{CodeSink, Reloc, bad_encoding}; -use ir::{Function, Inst, Ebb, InstructionData, Opcode}; +use ir::{Function, Inst, Ebb, InstructionData, Opcode, TrapCode}; use ir::condcodes::{CondCode, IntCC, FloatCC}; use isa::{RegUnit, StackRef, StackBase, StackBaseMask}; use regalloc::RegDiversions; diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index c3284c1423..12a83c8356 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -102,6 +102,10 @@ impl binemit::CodeSink for TextSink { fn reloc_jt(&mut self, reloc: binemit::Reloc, jt: ir::JumpTable) { write!(self.text, "{}({}) ", reloc, jt).unwrap(); } + + fn trap(&mut self, code: ir::TrapCode, _srcloc: ir::SourceLoc) { + write!(self.text, "{} ", code).unwrap(); + } } impl SubTest for TestBinEmit { diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs index 3772081a16..5e7a1b6109 100644 --- a/lib/filetests/src/test_compile.rs +++ b/lib/filetests/src/test_compile.rs @@ -111,4 +111,5 @@ impl binemit::CodeSink for SizeSink { ) { } fn reloc_jt(&mut self, _reloc: binemit::Reloc, _jt: ir::JumpTable) {} + fn trap(&mut self, _code: ir::TrapCode, _srcloc: ir::SourceLoc) {} } From 68b2040ba25f8b206c4af6fcf27662d83fc4beed Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Mar 2018 16:20:46 -0700 Subject: [PATCH 1666/3084] Document that opcodes.rs contains `InstructionData`. --- lib/cretonne/src/ir/instructions.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index bb09e24834..b25e5a069b 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -31,6 +31,7 @@ pub type ValueListPool = entity::ListPool; // Include code generated by `lib/cretonne/meta/gen_instr.py`. This file contains: // // - The `pub enum InstructionFormat` enum with all the instruction formats. +// - The `pub enum InstructionData` enum with all the instruction data fields. // - The `pub enum Opcode` definition with all known opcodes, // - The `const OPCODE_FORMAT: [InstructionFormat; N]` table. // - The private `fn opcode_name(Opcode) -> &'static str` function, and From 4af95e37a60a84db3ee19a45688ff956905ca8a1 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Mar 2018 16:22:05 -0700 Subject: [PATCH 1667/3084] Convert regular comments to documentation comments. --- lib/cretonne/src/dominator_tree.rs | 30 ++++++++--------- lib/cretonne/src/ir/dfg.rs | 24 +++++++------- lib/cretonne/src/ir/immediates.rs | 46 +++++++++++++-------------- lib/cretonne/src/ir/layout.rs | 46 +++++++++++++-------------- lib/cretonne/src/ir/types.rs | 4 +-- lib/cretonne/src/regalloc/spilling.rs | 4 +-- lib/filetests/src/concurrent.rs | 12 +++---- lib/filetests/src/runner.rs | 4 +-- lib/filetests/src/test_binemit.rs | 2 +- lib/filetests/src/test_compile.rs | 2 +- lib/frontend/src/ssa.rs | 12 +++---- lib/reader/src/parser.rs | 22 ++++++------- lib/wasm/src/code_translator.rs | 6 ++-- 13 files changed, 107 insertions(+), 107 deletions(-) diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 823469d081..57f55781a3 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -11,28 +11,28 @@ use timing; use std::cmp::Ordering; use std::vec::Vec; -// RPO numbers are not first assigned in a contiguous way but as multiples of STRIDE, to leave -// room for modifications of the dominator tree. +/// RPO numbers are not first assigned in a contiguous way but as multiples of STRIDE, to leave +/// room for modifications of the dominator tree. const STRIDE: u32 = 4; -// Special RPO numbers used during `compute_postorder`. +/// Special RPO numbers used during `compute_postorder`. const DONE: u32 = 1; const SEEN: u32 = 2; -// Dominator tree node. We keep one of these per EBB. +/// Dominator tree node. We keep one of these per EBB. #[derive(Clone, Default)] struct DomNode { - // Number of this node in a reverse post-order traversal of the CFG, starting from 1. - // This number is monotonic in the reverse postorder but not contiguous, since we leave - // holes for later localized modifications of the dominator tree. - // Unreachable nodes get number 0, all others are positive. + /// Number of this node in a reverse post-order traversal of the CFG, starting from 1. + /// This number is monotonic in the reverse postorder but not contiguous, since we leave + /// holes for later localized modifications of the dominator tree. + /// Unreachable nodes get number 0, all others are positive. rpo_number: u32, - // The immediate dominator of this EBB, represented as the branch or jump instruction at the - // end of the dominating basic block. - // - // This is `None` for unreachable blocks and the entry block which doesn't have an immediate - // dominator. + /// The immediate dominator of this EBB, represented as the branch or jump instruction at the + /// end of the dominating basic block. + /// + /// This is `None` for unreachable blocks and the entry block which doesn't have an immediate + /// dominator. idom: PackedOption, } @@ -40,10 +40,10 @@ struct DomNode { pub struct DominatorTree { nodes: EntityMap, - // CFG post-order of all reachable EBBs. + /// CFG post-order of all reachable EBBs. postorder: Vec, - // Scratch memory used by `compute_postorder()`. + /// Scratch memory used by `compute_postorder()`. stack: Vec, valid: bool, diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index b8976b7267..d8cb9b2eb1 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -348,18 +348,18 @@ impl ValueDef { } } -// Internal table storage for extended values. +/// Internal table storage for extended values. #[derive(Clone, Debug)] enum ValueData { - // Value is defined by an instruction. + /// Value is defined by an instruction. Inst { ty: Type, num: u16, inst: Inst }, - // Value is an EBB parameter. + /// Value is an EBB parameter. Param { ty: Type, num: u16, ebb: Ebb }, - // Value is an alias of another value. - // An alias value can't be linked as an instruction result or EBB parameter. It is used as a - // placeholder when the original instruction or EBB has been rewritten or modified. + /// Value is an alias of another value. + /// An alias value can't be linked as an instruction result or EBB parameter. It is used as a + /// placeholder when the original instruction or EBB has been rewritten or modified. Alias { ty: Type, original: Value }, } @@ -829,14 +829,14 @@ impl DataFlowGraph { } } -// Contents of an extended basic block. -// -// Parameters on an extended basic block are values that dominate everything in the EBB. All -// branches to this EBB must provide matching arguments, and the arguments to the entry EBB must -// match the function arguments. +/// Contents of an extended basic block. +/// +/// Parameters on an extended basic block are values that dominate everything in the EBB. All +/// branches to this EBB must provide matching arguments, and the arguments to the entry EBB must +/// match the function arguments. #[derive(Clone)] struct EbbData { - // List of parameters to this EBB. + /// List of parameters to this EBB. params: ValueList, } diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 2e6339cc01..b742c0032d 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -35,12 +35,12 @@ impl From for Imm64 { } } -// Hexadecimal with a multiple of 4 digits and group separators: -// -// 0xfff0 -// 0x0001_ffff -// 0xffff_ffff_fff8_4400 -// +/// Hexadecimal with a multiple of 4 digits and group separators: +/// +/// 0xfff0 +/// 0x0001_ffff +/// 0xffff_ffff_fff8_4400 +/// fn write_hex(x: i64, f: &mut Formatter) -> fmt::Result { let mut pos = (64 - x.leading_zeros() - 1) & 0xf0; write!(f, "0x{:04x}", (x >> pos) & 0xffff)?; @@ -280,16 +280,16 @@ pub struct Ieee32(u32); #[derive(Copy, Clone, Debug, Eq, PartialEq, Hash)] pub struct Ieee64(u64); -// Format a floating point number in a way that is reasonably human-readable, and that can be -// converted back to binary without any rounding issues. The hexadecimal formatting of normal and -// subnormal numbers is compatible with C99 and the `printf "%a"` format specifier. The NaN and Inf -// formats are not supported by C99. -// -// The encoding parameters are: -// -// w - exponent field width in bits -// t - trailing significand field width in bits -// +/// Format a floating point number in a way that is reasonably human-readable, and that can be +/// converted back to binary without any rounding issues. The hexadecimal formatting of normal and +/// subnormal numbers is compatible with C99 and the `printf "%a"` format specifier. The NaN and Inf +/// formats are not supported by C99. +/// +/// The encoding parameters are: +/// +/// w - exponent field width in bits +/// t - trailing significand field width in bits +/// fn format_float(bits: u64, w: u8, t: u8, f: &mut Formatter) -> fmt::Result { debug_assert!(w > 0 && w <= 16, "Invalid exponent range"); debug_assert!(1 + w + t <= 64, "Too large IEEE format for u64"); @@ -358,13 +358,13 @@ fn format_float(bits: u64, w: u8, t: u8, f: &mut Formatter) -> fmt::Result { } } -// Parse a float using the same format as `format_float` above. -// -// The encoding parameters are: -// -// w - exponent field width in bits -// t - trailing significand field width in bits -// +/// Parse a float using the same format as `format_float` above. +/// +/// The encoding parameters are: +/// +/// w - exponent field width in bits +/// t - trailing significand field width in bits +/// fn parse_float(s: &str, w: u8, t: u8) -> Result { debug_assert!(w > 0 && w <= 16, "Invalid exponent range"); debug_assert!(1 + w + t <= 64, "Too large IEEE format for u64"); diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index c19cf6dd37..af0c99e0fe 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -26,18 +26,18 @@ use timing; /// #[derive(Clone)] pub struct Layout { - // Linked list nodes for the layout order of EBBs Forms a doubly linked list, terminated in - // both ends by `None`. + /// Linked list nodes for the layout order of EBBs Forms a doubly linked list, terminated in + /// both ends by `None`. ebbs: EntityMap, - // Linked list nodes for the layout order of instructions. Forms a double linked list per EBB, - // terminated in both ends by `None`. + /// Linked list nodes for the layout order of instructions. Forms a double linked list per EBB, + /// terminated in both ends by `None`. insts: EntityMap, - // First EBB in the layout order, or `None` when no EBBs have been laid out. + /// First EBB in the layout order, or `None` when no EBBs have been laid out. first_ebb: Option, - // Last EBB in the layout order, or `None` when no EBBs have been laid out. + /// Last EBB in the layout order, or `None` when no EBBs have been laid out. last_ebb: Option, } @@ -61,31 +61,31 @@ impl Layout { } } -// Sequence numbers. -// -// All instructions and EBBs are given a sequence number that can be used to quickly determine -// their relative position in the layout. The sequence numbers are not contiguous, but are assigned -// like line numbers in BASIC: 10, 20, 30, ... -// -// The EBB sequence numbers are strictly increasing, and so are the instruction sequence numbers -// within an EBB. The instruction sequence numbers are all between the sequence number of their -// containing EBB and the following EBB. -// -// The result is that sequence numbers work like BASIC line numbers for the textual form of the IR. +/// Sequence numbers. +/// +/// All instructions and EBBs are given a sequence number that can be used to quickly determine +/// their relative position in the layout. The sequence numbers are not contiguous, but are assigned +/// like line numbers in BASIC: 10, 20, 30, ... +/// +/// The EBB sequence numbers are strictly increasing, and so are the instruction sequence numbers +/// within an EBB. The instruction sequence numbers are all between the sequence number of their +/// containing EBB and the following EBB. +/// +/// The result is that sequence numbers work like BASIC line numbers for the textual form of the IR. type SequenceNumber = u32; -// Initial stride assigned to new sequence numbers. +/// Initial stride assigned to new sequence numbers. const MAJOR_STRIDE: SequenceNumber = 10; -// Secondary stride used when renumbering locally. +/// Secondary stride used when renumbering locally. const MINOR_STRIDE: SequenceNumber = 2; -// Limit on the sequence number range we'll renumber locally. If this limit is exceeded, we'll -// switch to a full function renumbering. +/// Limit on the sequence number range we'll renumber locally. If this limit is exceeded, we'll +/// switch to a full function renumbering. const LOCAL_LIMIT: SequenceNumber = 100 * MINOR_STRIDE; -// Compute the midpoint between `a` and `b`. -// Return `None` if the midpoint would be equal to either. +/// Compute the midpoint between `a` and `b`. +/// Return `None` if the midpoint would be equal to either. fn midpoint(a: SequenceNumber, b: SequenceNumber) -> Option { debug_assert!(a < b); // Avoid integer overflow. diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 07cf97aefa..29a37f0cbf 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -24,10 +24,10 @@ pub struct Type(u8); /// a SIMD vector. pub const VOID: Type = Type(0); -// Start of the lane types. See also `meta/cdsl.types.py`. +/// Start of the lane types. See also `meta/cdsl.types.py`. const LANE_BASE: u8 = 0x70; -// Start of the 2-lane vector types. +/// Start of the 2-lane vector types. const VECTOR_BASE: u8 = LANE_BASE + 16; // Include code generated by `lib/cretonne/meta/gen_types.py`. This file contains constant diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index 3c3f805cbf..f616ea3b74 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -548,8 +548,8 @@ impl<'a> Context<'a> { } } -// Struct representing a register use of a value. -// Used to detect multiple uses of the same value with incompatible register constraints. +/// Struct representing a register use of a value. +/// Used to detect multiple uses of the same value with incompatible register constraints. #[derive(Clone, Copy)] struct RegUse { value: Value, diff --git a/lib/filetests/src/concurrent.rs b/lib/filetests/src/concurrent.rs index 1ddf643258..1db5aa6832 100644 --- a/lib/filetests/src/concurrent.rs +++ b/lib/filetests/src/concurrent.rs @@ -13,7 +13,7 @@ use std::time::Duration; use num_cpus; use {TestResult, runone}; -// Request sent to worker threads contains jobid and path. +/// Request sent to worker threads contains jobid and path. struct Request(usize, PathBuf); /// Reply from worker thread, @@ -25,13 +25,13 @@ pub enum Reply { /// Manage threads that run test jobs concurrently. pub struct ConcurrentRunner { - // Channel for sending requests to the worker threads. - // The workers are sharing the receiver with an `Arc>`. - // This is `None` when shutting down. + /// Channel for sending requests to the worker threads. + /// The workers are sharing the receiver with an `Arc>`. + /// This is `None` when shutting down. request_tx: Option>, - // Channel for receiving replies from the workers. - // Workers have their own `Sender`. + /// Channel for receiving replies from the workers. + /// Workers have their own `Sender`. reply_rx: Receiver, handles: Vec>, diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs index 49078cff8e..d2d2db55c7 100644 --- a/lib/filetests/src/runner.rs +++ b/lib/filetests/src/runner.rs @@ -11,10 +11,10 @@ use std::time; use {TestResult, runone}; use concurrent::{ConcurrentRunner, Reply}; -// Timeout in seconds when we're not making progress. +/// Timeout in seconds when we're not making progress. const TIMEOUT_PANIC: usize = 10; -// Timeout for reporting slow tests without panicking. +/// Timeout for reporting slow tests without panicking. const TIMEOUT_SLOW: usize = 3; struct QueueEntry { diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index 12a83c8356..c22f1ed386 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -27,7 +27,7 @@ pub fn subtest(parsed: &TestCommand) -> Result> { } } -// Code sink that generates text. +/// Code sink that generates text. struct TextSink { offset: binemit::CodeOffset, text: String, diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs index 5e7a1b6109..af2343f765 100644 --- a/lib/filetests/src/test_compile.rs +++ b/lib/filetests/src/test_compile.rs @@ -76,7 +76,7 @@ impl SubTest for TestCompile { } } -// Code sink that simply counts bytes. +/// Code sink that simply counts bytes. struct SizeSink { offset: binemit::CodeOffset, } diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index af0d35695e..6fa59e784d 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -77,12 +77,12 @@ impl SideEffects { } } -// Describes the current position of a basic block in the control flow graph. +/// Describes the current position of a basic block in the control flow graph. enum BlockData { - // A block at the top of an `Ebb`. + /// A block at the top of an `Ebb`. EbbHeader(EbbHeaderBlockData), - // A block inside an `Ebb` with an unique other block as its predecessor. - // The block is implicitly sealed at creation. + /// A block inside an `Ebb` with an unique other block as its predecessor. + /// The block is implicitly sealed at creation. EbbBody { predecessor: Block }, } @@ -179,7 +179,7 @@ where } } -// Small enum used for clarity in some functions. +/// Small enum used for clarity in some functions. #[derive(Debug)] enum ZeroOneOrMore { Zero(), @@ -194,7 +194,7 @@ enum UseVarCases { SealedMultiplePredecessors(Value, Ebb), } -// States for the `use_var`/`predecessors_lookup` state machine. +/// States for the `use_var`/`predecessors_lookup` state machine. enum Call { UseVar(Block), FinishSealedOnePredecessor(Block), diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index b165756566..f0fa723594 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -65,34 +65,34 @@ pub struct Parser<'a> { lex_error: Option, - // Current lookahead token. + /// Current lookahead token. lookahead: Option>, - // Location of lookahead. + /// Location of lookahead. loc: Location, - // Are we gathering any comments that we encounter? + /// Are we gathering any comments that we encounter? gathering_comments: bool, - // The gathered comments; claim them with `claim_gathered_comments`. + /// The gathered comments; claim them with `claim_gathered_comments`. gathered_comments: Vec<&'a str>, - // Comments collected so far. + /// Comments collected so far. comments: Vec>, } -// Context for resolving references when parsing a single function. +/// Context for resolving references when parsing a single function. struct Context<'a> { function: Function, map: SourceMap, - // Aliases to resolve once value definitions are known. + /// Aliases to resolve once value definitions are known. aliases: Vec, - // Reference to the unique_isa for things like parsing ISA-specific instruction encoding - // information. This is only `Some` if exactly one set of `isa` directives were found in the - // prologue (it is valid to have directives for multiple different ISAs, but in that case we - // couldn't know which ISA the provided encodings are intended for) + /// Reference to the unique_isa for things like parsing ISA-specific instruction encoding + /// information. This is only `Some` if exactly one set of `isa` directives were found in the + /// prologue (it is valid to have directives for multiple different ISAs, but in that case we + /// couldn't know which ISA the provided encodings are intended for) unique_isa: Option<&'a TargetIsa>, } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 61b7118636..97f67d6c48 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -944,7 +944,7 @@ fn translate_unreachable_operator( } } -// Get the address+offset to use for a heap access. +/// Get the address+offset to use for a heap access. fn get_heap_addr( heap: ir::Heap, addr32: ir::Value, @@ -981,7 +981,7 @@ fn get_heap_addr( } } -// Translate a load instruction. +/// Translate a load instruction. fn translate_load( offset: u32, opcode: ir::Opcode, @@ -1008,7 +1008,7 @@ fn translate_load( state.push1(dfg.first_result(load)); } -// Translate a store instruction. +/// Translate a store instruction. fn translate_store( offset: u32, opcode: ir::Opcode, From 5377092e5b851747fed03d385a666789664c509b Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 28 Mar 2018 17:15:19 -0700 Subject: [PATCH 1668/3084] Use `#[cold]` rather than `#[inline(never)]`. This gives optimizers more information about likely branch probabilities. --- lib/cretonne/src/binemit/mod.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 8f6f3c2924..6d2099ceba 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -92,7 +92,7 @@ pub trait CodeSink { } /// Report a bad encoding error. -#[inline(never)] +#[cold] pub fn bad_encoding(func: &Function, inst: Inst) -> ! { panic!( "Bad encoding {} for {}", From 6606b88136dc8706d5b9bc32c3f0cae1b9485389 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 30 Mar 2018 12:30:07 -0700 Subject: [PATCH 1669/3084] Optimize immediates and compare and branch sequences (#286) * Add a pre-opt optimization to change constants into immediates. This converts 'iadd' + 'iconst' into 'iadd_imm', and so on. * Optimize away redundant `bint` instructions. Cretonne has a concept of "Testable" values, which can be either boolean or integer. When the an instruction needing a "Testable" value receives the result of a `bint`, converting boolean to integer, eliminate the `bint`, as it's redundant. * Postopt: Optimize using CPU flags. This introduces a post-legalization optimization pass which converts compare+branch sequences to use flags values on CPUs which support it. * Define a form of x86's `urm` that doesn't clobber FLAGS. movzbl/movsbl/etc. don't clobber FLAGS; define a form of the `urm` recipe that represents this. * Implement a DCE pass. This pass deletes instructions with no side effects and no results that are used. * Clarify ambiguity about "32-bit" and "64-bit" in comments. * Add x86 encodings for icmp_imm. * Add a testcase for postopt CPU flags optimization. This covers the basic functionality of transforming compare+branch sequences to use CPU flags. * Pattern-match irsub_imm in preopt. --- cranelift/docs/testing.rst | 16 ++ cranelift/filetests/dce/basic.cton | 46 ++++ cranelift/filetests/isa/intel/binary32.cton | 2 +- cranelift/filetests/isa/intel/binary64.cton | 18 +- cranelift/filetests/postopt/basic.cton | 100 +++++++++ cranelift/filetests/preopt/simplify.cton | 80 +++++++ .../filetests/regalloc/coloring-227.cton | 2 +- cranelift/filetests/verifier/flags.cton | 76 +++---- lib/cretonne/meta/isa/intel/encodings.py | 52 +++-- lib/cretonne/meta/isa/intel/recipes.py | 77 ++++++- lib/cretonne/src/context.rs | 20 ++ lib/cretonne/src/dce.rs | 68 ++++++ lib/cretonne/src/ir/immediates.rs | 5 + lib/cretonne/src/isa/intel/mod.rs | 4 + lib/cretonne/src/isa/mod.rs | 5 + lib/cretonne/src/lib.rs | 2 + lib/cretonne/src/postopt.rs | 211 ++++++++++++++++++ lib/cretonne/src/preopt.rs | 137 ++++++++---- lib/cretonne/src/timing.rs | 2 + lib/filetests/src/lib.rs | 4 + lib/filetests/src/test_dce.rs | 53 +++++ lib/filetests/src/test_postopt.rs | 50 +++++ 22 files changed, 921 insertions(+), 109 deletions(-) create mode 100644 cranelift/filetests/dce/basic.cton create mode 100644 cranelift/filetests/postopt/basic.cton create mode 100644 cranelift/filetests/preopt/simplify.cton create mode 100644 lib/cretonne/src/dce.rs create mode 100644 lib/cretonne/src/postopt.rs create mode 100644 lib/filetests/src/test_dce.rs create mode 100644 lib/filetests/src/test_postopt.rs diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index c14a74aeba..db7efb1da1 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -334,6 +334,14 @@ Test the LICM pass. The LICM pass is run on each function, and then results are run through filecheck. +`test dce` +----------------- + +Test the DCE pass. + +The DCE pass is run on each function, and then results are run +through filecheck. + `test preopt` ----------------- @@ -342,6 +350,14 @@ Test the preopt pass. The preopt pass is run on each function, and then results are run through filecheck. +`test postopt` +----------------- + +Test the postopt pass. + +The postopt pass is run on each function, and then results are run +through filecheck. + `test compile` -------------- diff --git a/cranelift/filetests/dce/basic.cton b/cranelift/filetests/dce/basic.cton new file mode 100644 index 0000000000..436b4aee19 --- /dev/null +++ b/cranelift/filetests/dce/basic.cton @@ -0,0 +1,46 @@ +test dce + +function %simple() -> i32 { +ebb0: + v2 = iconst.i32 2 + v3 = iconst.i32 3 + return v3 +} +; sameln: function %simple +; nextln: ebb0: +; nextln: v3 = iconst.i32 3 +; nextln: return v3 +; nextln: } + +function %some_branching(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v3 = iconst.i32 70 + v4 = iconst.i32 71 + v5 = iconst.i32 72 + v8 = iconst.i32 73 + brz v0, ebb1 + jump ebb2(v8) + +ebb1: + v2 = iadd v0, v3 + return v0 + +ebb2(v9: i32): + v6 = iadd v1, v4 + v7 = iadd v6, v9 + return v7 +} +; sameln: function %some_branching +; nextln: ebb0(v0: i32, v1: i32): +; nextln: v4 = iconst.i32 71 +; nextln: v8 = iconst.i32 73 +; nextln: brz v0, ebb1 +; nextln: jump ebb2(v8) +; nextln: +; nextln: ebb1: +; nextln: return v0 +; nextln: +; nextln: ebb2(v9: i32): +; nextln: v6 = iadd.i32 v1, v4 +; nextln: v7 = iadd v6, v9 +; nextln: return v7 diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index a3d0723be8..2567fbe88c 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -1,4 +1,4 @@ -; binary emission of 32-bit code. +; binary emission of x86-32 code. test binemit set is_compressed isa intel haswell diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 1444c1e062..1a47dc1cba 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -1,4 +1,4 @@ -; binary emission of 64-bit code. +; binary emission of x86-64 code. test binemit set is_64bit set is_compressed @@ -458,6 +458,14 @@ ebb0: ; asm: setbe %dl [-,%rdx] v319 = icmp ule v2, v3 ; bin: 4c 39 d6 0f 96 c2 + ; asm: cmpq $37, %rcx + ; asm: setl %bl + [-,%rbx] v320 = icmp_imm slt v1, 37 ; bin: 48 83 f9 25 0f 9c c3 + + ; asm: cmpq $100000, %rcx + ; asm: setl %bl + [-,%rbx] v321 = icmp_imm slt v1, 100000 ; bin: 48 81 f9 000186a0 0f 9c c3 + ; Bool-to-int conversions. ; asm: movzbq %bl, %rcx @@ -1031,6 +1039,14 @@ ebb0: ; asm: setbe %dl [-,%rdx] v319 = icmp ule v2, v3 ; bin: 44 39 d6 0f 96 c2 + ; asm: cmpl $37, %ecx + ; asm: setl %bl + [-,%rbx] v320 = icmp_imm slt v1, 37 ; bin: 83 f9 25 0f 9c c3 + + ; asm: cmpq $100000, %ecx + ; asm: setl %bl + [-,%rbx] v321 = icmp_imm slt v1, 100000 ; bin: 81 f9 000186a0 0f 9c c3 + ; Bool-to-int conversions. ; asm: movzbl %bl, %ecx diff --git a/cranelift/filetests/postopt/basic.cton b/cranelift/filetests/postopt/basic.cton new file mode 100644 index 0000000000..218d4dfee4 --- /dev/null +++ b/cranelift/filetests/postopt/basic.cton @@ -0,0 +1,100 @@ +test postopt +isa intel + +; Test that compare+branch sequences are folded effectively on x86. + +function %br_icmp(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): +[Op1icscc#39,%rdx] v2 = icmp slt v0, v1 +[Op1t8jccd_long#85] brnz v2, ebb1 +[Op1ret#c3] return v1 + +ebb1: +[Op1puid#b8,%rax] v8 = iconst.i32 3 +[Op1ret#c3] return v8 +} +; sameln: function %br_icmp +; nextln: ebb0(v0: i32, v1: i32): +; nextln: v9 = ifcmp v0, v1 +; nextln: v2 = trueif slt v9 +; nextln: brif slt v9, ebb1 +; nextln: return v1 +; nextln: +; nextln: ebb1: +; nextln: v8 = iconst.i32 3 +; nextln: return v8 +; nextln: } + +; Use brz instead of brnz, so the condition is inverted. + +function %br_icmp_inverse(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): +[Op1icscc#39,%rdx] v2 = icmp slt v0, v1 +[Op1t8jccd_long#84] brz v2, ebb1 +[Op1ret#c3] return v1 + +ebb1: +[Op1puid#b8,%rax] v8 = iconst.i32 3 +[Op1ret#c3] return v8 +} +; sameln: function %br_icmp_inverse +; nextln: ebb0(v0: i32, v1: i32): +; nextln: v9 = ifcmp v0, v1 +; nextln: v2 = trueif slt v9 +; nextln: brif sge v9, ebb1 +; nextln: return v1 +; nextln: +; nextln: ebb1: +; nextln: v8 = iconst.i32 3 +; nextln: return v8 +; nextln: } + +; Use icmp_imm instead of icmp. + +function %br_icmp_imm(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): +[Op1icsccib#7083] v2 = icmp_imm slt v0, 2 +[Op1t8jccd_long#84] brz v2, ebb1 +[Op1ret#c3] return v1 + +ebb1: +[Op1puid#b8,%rax] v8 = iconst.i32 3 +[Op1ret#c3] return v8 +} +; sameln: function %br_icmp_imm +; nextln: ebb0(v0: i32, v1: i32): +; nextln: v9 = ifcmp_imm v0, 2 +; nextln: v2 = trueif slt v9 +; nextln: brif sge v9, ebb1 +; nextln: return v1 +; nextln: +; nextln: ebb1: +; nextln: v8 = iconst.i32 3 +; nextln: return v8 +; nextln: } + +; Use fcmp instead of icmp. + +function %br_fcmp(f32, f32) -> f32 { +ebb0(v0: f32, v1: f32): +[Op2fcscc#42e,%rdx] v2 = fcmp gt v0, v1 +[Op1t8jccd_long#84] brz v2, ebb1 +[Op1ret#c3] return v1 + +ebb1: +[Op1puid#b8,%rax] v18 = iconst.i32 0x40a8_0000 +[Mp2frurm#56e,%xmm0] v8 = bitcast.f32 v18 +[Op1ret#c3] return v8 +} +; sameln: function %br_fcmp +; nextln: ebb0(v0: f32, v1: f32): +; nextln: v19 = ffcmp v0, v1 +; nextln: v2 = trueff gt v19 +; nextln: brff ule v19, ebb1 +; nextln: return v1 +; nextln: +; nextln: ebb1: +; nextln: v18 = iconst.i32 0x40a8_0000 +; nextln: v8 = bitcast.f32 v18 +; nextln: return v8 +; nextln: } diff --git a/cranelift/filetests/preopt/simplify.cton b/cranelift/filetests/preopt/simplify.cton new file mode 100644 index 0000000000..a2e67caf31 --- /dev/null +++ b/cranelift/filetests/preopt/simplify.cton @@ -0,0 +1,80 @@ +test preopt +isa intel + +function %iadd_imm(i32) -> i32 { +ebb0(v0: i32): + v1 = iconst.i32 2 + v2 = iadd v0, v1 + return v2 +} +; sameln: function %iadd_imm +; nextln: ebb0(v0: i32): +; nextln: v1 = iconst.i32 2 +; nextln: v2 = iadd_imm v0, 2 +; nextln: return v2 +; nextln: } + +function %isub_imm(i32) -> i32 { +ebb0(v0: i32): + v1 = iconst.i32 2 + v2 = isub v0, v1 + return v2 +} +; sameln: function %isub_imm +; nextln: ebb0(v0: i32): +; nextln: v1 = iconst.i32 2 +; nextln: v2 = iadd_imm v0, -2 +; nextln: return v2 +; nextln: } + +function %icmp_imm(i32) -> i32 { +ebb0(v0: i32): + v1 = iconst.i32 2 + v2 = icmp slt v0, v1 + v3 = bint.i32 v2 + return v3 +} +; sameln: function %icmp_imm +; nextln: ebb0(v0: i32): +; nextln: v1 = iconst.i32 2 +; nextln: v2 = icmp_imm slt v0, 2 +; nextln: v3 = bint.i32 v2 +; nextln: return v3 +; nextln: } + +function %brz_bint(i32) { +ebb0(v0: i32): + v3 = icmp_imm slt v0, 0 + v1 = bint.i32 v3 + v2 = select v1, v1, v1 + trapz v1, user0 + brz v1, ebb1 + jump ebb2 + +ebb1: + return + +ebb2: + return +} +; sameln: function %brz_bint +; nextln: (v0: i32): +; nextln: v3 = icmp_imm slt v0, 0 +; nextln: v1 = bint.i32 v3 +; nextln: v2 = select v3, v1, v1 +; nextln: trapz v3, user0 +; nextln: brz v3, ebb1 +; nextln: jump ebb2 + +function %irsub_imm(i32) -> i32 { +ebb0(v0: i32): + v1 = iconst.i32 2 + v2 = isub v1, v0 + return v2 +} +; sameln: function %irsub_imm +; nextln: ebb0(v0: i32): +; nextln: v1 = iconst.i32 2 +; nextln: v2 = irsub_imm v1, 2 +; nextln: return v2 +; nextln: } diff --git a/cranelift/filetests/regalloc/coloring-227.cton b/cranelift/filetests/regalloc/coloring-227.cton index 07681a7509..7e2dac2e50 100644 --- a/cranelift/filetests/regalloc/coloring-227.cton +++ b/cranelift/filetests/regalloc/coloring-227.cton @@ -21,7 +21,7 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) @0011 [RexOp1puid#b8] v9 = iconst.i32 0 @0015 [RexOp1puid#b8] v11 = iconst.i32 0 @0017 [RexOp1icscc#39] v12 = icmp.i32 eq v15, v11 -@0017 [RexOp2urm#4b6] v13 = bint.i32 v12 +@0017 [RexOp2urm_noflags#4b6] v13 = bint.i32 v12 @001a [RexOp1rr#21] v14 = band v9, v13 @001b [RexOp1tjccb#75] brnz v14, ebb6 @001d [RexOp1jmpb#eb] jump ebb7 diff --git a/cranelift/filetests/verifier/flags.cton b/cranelift/filetests/verifier/flags.cton index dff01244e2..b4a01621c4 100644 --- a/cranelift/filetests/verifier/flags.cton +++ b/cranelift/filetests/verifier/flags.cton @@ -4,65 +4,65 @@ isa intel ; Simple, correct use of CPU flags. function %simple(i32) -> i32 { ebb0(v0: i32): - [Op1rcmp#39] v1 = ifcmp v0, v0 - [Op2seti_abcd#490] v2 = trueif ugt v1 - [Op2urm_abcd#4b6] v3 = bint.i32 v2 - [Op1ret#c3] return v3 + [Op1rcmp#39] v1 = ifcmp v0, v0 + [Op2seti_abcd#490] v2 = trueif ugt v1 + [Op2urm_noflags_abcd#4b6] v3 = bint.i32 v2 + [Op1ret#c3] return v3 } ; Overlapping flag values of different types. function %overlap(i32, f32) -> i32 { ebb0(v0: i32, v1: f32): - [Op1rcmp#39] v2 = ifcmp v0, v0 - [Op2fcmp#42e] v3 = ffcmp v1, v1 - [Op2setf_abcd#490] v4 = trueff gt v3 ; error: conflicting live CPU flags: v2 and v3 - [Op2seti_abcd#490] v5 = trueif ugt v2 - [Op1rr#21] v6 = band v4, v5 - [Op2urm_abcd#4b6] v7 = bint.i32 v6 - [Op1ret#c3] return v7 + [Op1rcmp#39] v2 = ifcmp v0, v0 + [Op2fcmp#42e] v3 = ffcmp v1, v1 + [Op2setf_abcd#490] v4 = trueff gt v3 ; error: conflicting live CPU flags: v2 and v3 + [Op2seti_abcd#490] v5 = trueif ugt v2 + [Op1rr#21] v6 = band v4, v5 + [Op2urm_noflags_abcd#4b6] v7 = bint.i32 v6 + [Op1ret#c3] return v7 } ; CPU flags clobbered by arithmetic. function %clobbered(i32) -> i32 { ebb0(v0: i32): - [Op1rcmp#39] v1 = ifcmp v0, v0 - [Op1rr#01] v2 = iadd v0, v0 ; error: encoding clobbers live CPU flags in v1 - [Op2seti_abcd#490] v3 = trueif ugt v1 - [Op2urm_abcd#4b6] v4 = bint.i32 v3 - [Op1ret#c3] return v4 + [Op1rcmp#39] v1 = ifcmp v0, v0 + [Op1rr#01] v2 = iadd v0, v0 ; error: encoding clobbers live CPU flags in v1 + [Op2seti_abcd#490] v3 = trueif ugt v1 + [Op2urm_noflags_abcd#4b6] v4 = bint.i32 v3 + [Op1ret#c3] return v4 } ; CPU flags not clobbered by load. function %live_across_load(i32) -> i32 { ebb0(v0: i32): - [Op1rcmp#39] v1 = ifcmp v0, v0 - [Op1ld#8b] v2 = load.i32 v0 - [Op2seti_abcd#490] v3 = trueif ugt v1 - [Op2urm_abcd#4b6] v4 = bint.i32 v3 - [Op1ret#c3] return v4 + [Op1rcmp#39] v1 = ifcmp v0, v0 + [Op1ld#8b] v2 = load.i32 v0 + [Op2seti_abcd#490] v3 = trueif ugt v1 + [Op2urm_noflags_abcd#4b6] v4 = bint.i32 v3 + [Op1ret#c3] return v4 } ; Correct use of CPU flags across EBB. function %live_across_ebb(i32) -> i32 { - ebb0(v0: i32): - [Op1rcmp#39] v1 = ifcmp v0, v0 - [Op1jmpb#eb] jump ebb1 - ebb1: - [Op2seti_abcd#490] v2 = trueif ugt v1 - [Op2urm_abcd#4b6] v3 = bint.i32 v2 - [Op1ret#c3] return v3 + ebb0(v0: i32): + [Op1rcmp#39] v1 = ifcmp v0, v0 + [Op1jmpb#eb] jump ebb1 + ebb1: + [Op2seti_abcd#490] v2 = trueif ugt v1 + [Op2urm_noflags_abcd#4b6] v3 = bint.i32 v2 + [Op1ret#c3] return v3 } function %live_across_ebb_backwards(i32) -> i32 { - ebb0(v0: i32): - [Op1jmpb#eb] jump ebb2 - ebb1: - [Op2seti_abcd#490] v2 = trueif ugt v1 - [Op2urm_abcd#4b6] v3 = bint.i32 v2 - [Op1ret#c3] return v3 - ebb2: - [Op1rcmp#39] v1 = ifcmp v0, v0 - [Op1jmpb#eb] jump ebb1 + ebb0(v0: i32): + [Op1jmpb#eb] jump ebb2 + ebb1: + [Op2seti_abcd#490] v2 = trueif ugt v1 + [Op2urm_noflags_abcd#4b6] v3 = bint.i32 v2 + [Op1ret#c3] return v3 + ebb2: + [Op1rcmp#39] v1 = ifcmp v0, v0 + [Op1jmpb#eb] jump ebb1 } ; Flags live into loop. @@ -73,4 +73,4 @@ function %live_into_loop(i32) -> i32 { ebb1: [Op2seti_abcd#490] v2 = trueif ugt v1 [Op1jmpb#eb] jump ebb1 -} \ No newline at end of file +} diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 72ef7a362b..0fc507bac5 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -378,6 +378,8 @@ X86_64.enc(base.trapff, r.trapff, 0) # Comparisons # enc_i32_i64(base.icmp, r.icscc, 0x39) +enc_i32_i64(base.icmp_imm, r.icsccib, 0x83, rrr=7) +enc_i32_i64(base.icmp_imm, r.icsccid, 0x81, rrr=7) enc_i32_i64(base.ifcmp, r.rcmp, 0x39) enc_i32_i64(base.ifcmp_imm, r.rcmpib, 0x83, rrr=7) enc_i32_i64(base.ifcmp_imm, r.rcmpid, 0x81, rrr=7) @@ -409,11 +411,13 @@ enc_i32_i64(x86.bsr, r.bsf_and_bsr, 0x0F, 0xBD) # # This assumes that b1 is represented as an 8-bit low register with the value 0 # or 1. -X86_32.enc(base.bint.i32.b1, *r.urm_abcd(0x0f, 0xb6)) -X86_64.enc(base.bint.i64.b1, *r.urm.rex(0x0f, 0xb6)) # zext to i64 implicit. -X86_64.enc(base.bint.i64.b1, *r.urm_abcd(0x0f, 0xb6)) # zext to i64 implicit. -X86_64.enc(base.bint.i32.b1, *r.urm.rex(0x0f, 0xb6)) -X86_64.enc(base.bint.i32.b1, *r.urm_abcd(0x0f, 0xb6)) +# +# Encode movzbq as movzbl, because it's equivalent and shorter. +X86_32.enc(base.bint.i32.b1, *r.urm_noflags_abcd(0x0f, 0xb6)) +X86_64.enc(base.bint.i64.b1, *r.urm_noflags.rex(0x0f, 0xb6)) +X86_64.enc(base.bint.i64.b1, *r.urm_noflags_abcd(0x0f, 0xb6)) +X86_64.enc(base.bint.i32.b1, *r.urm_noflags.rex(0x0f, 0xb6)) +X86_64.enc(base.bint.i32.b1, *r.urm_noflags_abcd(0x0f, 0xb6)) # Numerical conversions. @@ -430,41 +434,41 @@ X86_64.enc(base.ireduce.i32.i64, r.null, 0) # instructions for %al/%ax/%eax to %ax/%eax/%rax. # movsbl -X86_32.enc(base.sextend.i32.i8, *r.urm(0x0f, 0xbe)) -X86_64.enc(base.sextend.i32.i8, *r.urm.rex(0x0f, 0xbe)) -X86_64.enc(base.sextend.i32.i8, *r.urm(0x0f, 0xbe)) +X86_32.enc(base.sextend.i32.i8, *r.urm_noflags(0x0f, 0xbe)) +X86_64.enc(base.sextend.i32.i8, *r.urm_noflags.rex(0x0f, 0xbe)) +X86_64.enc(base.sextend.i32.i8, *r.urm_noflags(0x0f, 0xbe)) # movswl -X86_32.enc(base.sextend.i32.i16, *r.urm(0x0f, 0xbf)) -X86_64.enc(base.sextend.i32.i16, *r.urm.rex(0x0f, 0xbf)) -X86_64.enc(base.sextend.i32.i16, *r.urm(0x0f, 0xbf)) +X86_32.enc(base.sextend.i32.i16, *r.urm_noflags(0x0f, 0xbf)) +X86_64.enc(base.sextend.i32.i16, *r.urm_noflags.rex(0x0f, 0xbf)) +X86_64.enc(base.sextend.i32.i16, *r.urm_noflags(0x0f, 0xbf)) # movsbq -X86_64.enc(base.sextend.i64.i8, *r.urm.rex(0x0f, 0xbe, w=1)) +X86_64.enc(base.sextend.i64.i8, *r.urm_noflags.rex(0x0f, 0xbe, w=1)) # movswq -X86_64.enc(base.sextend.i64.i16, *r.urm.rex(0x0f, 0xbf, w=1)) +X86_64.enc(base.sextend.i64.i16, *r.urm_noflags.rex(0x0f, 0xbf, w=1)) # movslq -X86_64.enc(base.sextend.i64.i32, *r.urm.rex(0x63, w=1)) +X86_64.enc(base.sextend.i64.i32, *r.urm_noflags.rex(0x63, w=1)) # movzbl -X86_32.enc(base.uextend.i32.i8, *r.urm(0x0f, 0xb6)) -X86_64.enc(base.uextend.i32.i8, *r.urm.rex(0x0f, 0xb6)) -X86_64.enc(base.uextend.i32.i8, *r.urm(0x0f, 0xb6)) +X86_32.enc(base.uextend.i32.i8, *r.urm_noflags(0x0f, 0xb6)) +X86_64.enc(base.uextend.i32.i8, *r.urm_noflags.rex(0x0f, 0xb6)) +X86_64.enc(base.uextend.i32.i8, *r.urm_noflags(0x0f, 0xb6)) # movzwl -X86_32.enc(base.uextend.i32.i16, *r.urm(0x0f, 0xb7)) -X86_64.enc(base.uextend.i32.i16, *r.urm.rex(0x0f, 0xb7)) -X86_64.enc(base.uextend.i32.i16, *r.urm(0x0f, 0xb7)) +X86_32.enc(base.uextend.i32.i16, *r.urm_noflags(0x0f, 0xb7)) +X86_64.enc(base.uextend.i32.i16, *r.urm_noflags.rex(0x0f, 0xb7)) +X86_64.enc(base.uextend.i32.i16, *r.urm_noflags(0x0f, 0xb7)) # movzbq, encoded as movzbl because it's equivalent and shorter -X86_64.enc(base.uextend.i64.i8, *r.urm.rex(0x0f, 0xb6)) -X86_64.enc(base.uextend.i64.i8, *r.urm(0x0f, 0xb6)) +X86_64.enc(base.uextend.i64.i8, *r.urm_noflags.rex(0x0f, 0xb6)) +X86_64.enc(base.uextend.i64.i8, *r.urm_noflags(0x0f, 0xb6)) # movzwq, encoded as movzwl because it's equivalent and shorter -X86_64.enc(base.uextend.i64.i16, *r.urm.rex(0x0f, 0xb7)) -X86_64.enc(base.uextend.i64.i16, *r.urm(0x0f, 0xb7)) +X86_64.enc(base.uextend.i64.i16, *r.urm_noflags.rex(0x0f, 0xb7)) +X86_64.enc(base.uextend.i64.i16, *r.urm_noflags(0x0f, 0xb7)) # A 32-bit register copy clears the high 32 bits. X86_64.enc(base.uextend.i64.i32, *r.umr.rex(0x89)) diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index bb3db3cba1..54df437c67 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -8,7 +8,8 @@ from cdsl.registers import RegClass from base.formats import Unary, UnaryImm, UnaryBool, Binary, BinaryImm from base.formats import MultiAry, NullAry from base.formats import Trap, Call, IndirectCall, Store, Load -from base.formats import IntCompare, FloatCompare, IntCond, FloatCond +from base.formats import IntCompare, IntCompareImm, FloatCompare +from base.formats import IntCond, FloatCond from base.formats import IntSelect, IntCondTrap, FloatCondTrap from base.formats import Jump, Branch, BranchInt, BranchFloat from base.formats import Ternary, FuncAddr, UnaryGlobalVar @@ -364,7 +365,7 @@ rfumr = TailRecipe( ''') # XX /r, but for a unary operator with separate input/output register. -# RM form. +# RM form. Clobbers FLAGS. urm = TailRecipe( 'urm', Unary, size=1, ins=GPR, outs=GPR, emit=''' @@ -372,10 +373,19 @@ urm = TailRecipe( modrm_rr(in_reg0, out_reg0, sink); ''') -# XX /r. Same as urm, but input limited to ABCD. -urm_abcd = TailRecipe( - 'urm_abcd', Unary, size=1, ins=ABCD, outs=GPR, - when_prefixed=urm, +# XX /r. Same as urm, but doesn't clobber FLAGS. +urm_noflags = TailRecipe( + 'urm_noflags', Unary, size=1, ins=GPR, outs=GPR, + clobbers_flags=False, + emit=''' + PUT_OP(bits, rex2(in_reg0, out_reg0), sink); + modrm_rr(in_reg0, out_reg0, sink); + ''') + +# XX /r. Same as urm_noflags, but input limited to ABCD. +urm_noflags_abcd = TailRecipe( + 'urm_noflags_abcd', Unary, size=1, ins=ABCD, outs=GPR, + when_prefixed=urm_noflags, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_rr(in_reg0, out_reg0, sink); @@ -1360,6 +1370,61 @@ icscc = TailRecipe( modrm_rr(out_reg0, 0, sink); ''') +icsccib = TailRecipe( + 'icsccib', IntCompareImm, size=2 + 3, ins=GPR, outs=ABCD, + instp=IsSignedInt(IntCompareImm.imm, 8), + emit=''' + // Comparison instruction. + PUT_OP(bits, rex1(in_reg0), sink); + modrm_r_bits(in_reg0, bits, sink); + let imm: i64 = imm.into(); + sink.put1(imm as u8); + // `setCC` instruction, no REX. + use ir::condcodes::IntCC::*; + let setcc = match cond { + Equal => 0x94, + NotEqual => 0x95, + SignedLessThan => 0x9c, + SignedGreaterThanOrEqual => 0x9d, + SignedGreaterThan => 0x9f, + SignedLessThanOrEqual => 0x9e, + UnsignedLessThan => 0x92, + UnsignedGreaterThanOrEqual => 0x93, + UnsignedGreaterThan => 0x97, + UnsignedLessThanOrEqual => 0x96, + }; + sink.put1(0x0f); + sink.put1(setcc); + modrm_rr(out_reg0, 0, sink); + ''') + +icsccid = TailRecipe( + 'icsccid', IntCompareImm, size=5 + 3, ins=GPR, outs=ABCD, + instp=IsSignedInt(IntCompareImm.imm, 32), + emit=''' + // Comparison instruction. + PUT_OP(bits, rex1(in_reg0), sink); + modrm_r_bits(in_reg0, bits, sink); + let imm: i64 = imm.into(); + sink.put4(imm as u32); + // `setCC` instruction, no REX. + use ir::condcodes::IntCC::*; + let setcc = match cond { + Equal => 0x94, + NotEqual => 0x95, + SignedLessThan => 0x9c, + SignedGreaterThanOrEqual => 0x9d, + SignedGreaterThan => 0x9f, + SignedLessThanOrEqual => 0x9e, + UnsignedLessThan => 0x92, + UnsignedGreaterThanOrEqual => 0x93, + UnsignedGreaterThan => 0x97, + UnsignedLessThanOrEqual => 0x96, + }; + sink.put1(0x0f); + sink.put1(setcc); + modrm_rr(out_reg0, 0, sink); + ''') # Make a FloatCompare instruction predicate with the supported condition codes. diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 92ea61c643..2441f19d02 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -21,9 +21,11 @@ use result::{CtonError, CtonResult}; use settings::{FlagsOrIsa, OptLevel}; use unreachable_code::eliminate_unreachable_code; use verifier; +use dce::do_dce; use simple_gvn::do_simple_gvn; use licm::do_licm; use preopt::do_preopt; +use postopt::do_postopt; use timing; /// Persistent data structures and compilation pipeline. @@ -92,6 +94,9 @@ impl Context { self.preopt(isa)?; } self.legalize(isa)?; + if isa.flags().opt_level() != OptLevel::Fastest { + self.postopt(isa)?; + } if isa.flags().opt_level() == OptLevel::Best { self.compute_domtree(); self.compute_loop_analysis(); @@ -100,6 +105,7 @@ impl Context { } self.compute_domtree(); self.eliminate_unreachable_code(isa)?; + self.dce(isa)?; self.regalloc(isa)?; self.prologue_epilogue(isa)?; self.relax_branches(isa) @@ -153,6 +159,13 @@ impl Context { } } + /// Perform dead-code elimination on the function. + pub fn dce<'a, FOI: Into>>(&mut self, fisa: FOI) -> CtonResult { + do_dce(&mut self.func, &mut self.domtree); + self.verify_if(fisa)?; + Ok(()) + } + /// Perform pre-legalization rewrites on the function. pub fn preopt(&mut self, isa: &TargetIsa) -> CtonResult { do_preopt(&mut self.func); @@ -170,6 +183,13 @@ impl Context { self.verify_if(isa) } + /// Perform post-legalization rewrites on the function. + pub fn postopt(&mut self, isa: &TargetIsa) -> CtonResult { + do_postopt(&mut self.func, isa); + self.verify_if(isa)?; + Ok(()) + } + /// Compute the control flow graph. pub fn compute_cfg(&mut self) { self.cfg.compute(&self.func) diff --git a/lib/cretonne/src/dce.rs b/lib/cretonne/src/dce.rs new file mode 100644 index 0000000000..6e34bad74f --- /dev/null +++ b/lib/cretonne/src/dce.rs @@ -0,0 +1,68 @@ +//! A Dead-Code Elimination (DCE) pass. +//! +//! Dead code here means instructions that have no side effects and have no +//! result values used by other instructions. + +use cursor::{Cursor, FuncCursor}; +use dominator_tree::DominatorTree; +use entity::EntityRef; +use ir::{Function, Inst, Opcode, DataFlowGraph}; +use ir::instructions::InstructionData; +use timing; +use std::vec::Vec; + +/// Test whether the given opcode is unsafe to even consider for DCE. +fn trivially_unsafe_for_dce(opcode: Opcode) -> bool { + opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || + opcode.is_return() || opcode.can_trap() || opcode.other_side_effects() || + opcode.can_store() +} + +/// Preserve instructions with used result values. +fn any_inst_results_used(inst: Inst, live: &[bool], dfg: &DataFlowGraph) -> bool { + dfg.inst_results(inst).iter().any(|v| live[v.index()]) +} + +/// Load instructions without the `notrap` flag are defined to trap when +/// operating on inaccessible memory, so we can't DCE them even if the +/// loaded value is unused. +fn is_load_with_defined_trapping(opcode: Opcode, data: &InstructionData) -> bool { + if !opcode.can_load() { + return false; + } + match *data { + InstructionData::StackLoad { .. } => false, + InstructionData::Load { flags, .. } => !flags.notrap(), + _ => true, + } +} + +/// Perform DCE on `func`. +pub fn do_dce(func: &mut Function, domtree: &mut DominatorTree) { + let _tt = timing::dce(); + debug_assert!(domtree.is_valid()); + + let mut live = Vec::with_capacity(func.dfg.num_values()); + live.resize(func.dfg.num_values(), false); + + for &ebb in domtree.cfg_postorder().iter() { + let mut pos = FuncCursor::new(func).at_bottom(ebb); + while let Some(inst) = pos.prev_inst() { + { + let data = &pos.func.dfg[inst]; + let opcode = data.opcode(); + if trivially_unsafe_for_dce(opcode) || + is_load_with_defined_trapping(opcode, &data) || + any_inst_results_used(inst, &live, &pos.func.dfg) + { + for arg in pos.func.dfg.inst_args(inst) { + let v = pos.func.dfg.resolve_aliases(*arg); + live[v.index()] = true; + } + continue; + } + } + pos.remove_inst(); + } + } +} diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index b742c0032d..6026cecf26 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -21,6 +21,11 @@ impl Imm64 { pub fn new(x: i64) -> Imm64 { Imm64(x) } + + /// Return self negated. + pub fn wrapping_neg(self) -> Imm64 { + Imm64(self.0.wrapping_neg()) + } } impl Into for Imm64 { diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 578dac2d16..04ff149654 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -58,6 +58,10 @@ impl TargetIsa for Isa { &self.shared_flags } + fn uses_cpu_flags(&self) -> bool { + true + } + fn register_info(&self) -> RegInfo { registers::INFO.clone() } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index aefe93a05d..1a7714fc76 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -158,6 +158,11 @@ pub trait TargetIsa: fmt::Display { /// Get the ISA-independent flags that were used to make this trait object. fn flags(&self) -> &settings::Flags; + /// Does the CPU implement scalar comparisons using a CPU flags register? + fn uses_cpu_flags(&self) -> bool { + false + } + /// Get a data structure describing the registers in this ISA. fn register_info(&self) -> RegInfo; diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 0f731b9620..cde427df9f 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -68,11 +68,13 @@ mod abi; mod bitset; mod constant_hash; mod context; +mod dce; mod divconst_magic_numbers; mod iterators; mod legalizer; mod licm; mod partition_slice; +mod postopt; mod predicates; mod preopt; mod ref_slice; diff --git a/lib/cretonne/src/postopt.rs b/lib/cretonne/src/postopt.rs new file mode 100644 index 0000000000..a95baefd3a --- /dev/null +++ b/lib/cretonne/src/postopt.rs @@ -0,0 +1,211 @@ +//! A post-legalization rewriting pass. + +#![allow(non_snake_case)] + +use cursor::{Cursor, EncCursor}; +use ir::dfg::ValueDef; +use ir::{Function, InstructionData, Value, InstBuilder, Ebb, Inst}; +use ir::condcodes::{CondCode, IntCC, FloatCC}; +use ir::instructions::{Opcode, ValueList}; +use ir::immediates::Imm64; +use isa::TargetIsa; +use timing; + +/// Information collected about a compare+branch sequence. +struct CmpBrInfo { + /// The branch instruction. + br_inst: Inst, + /// The icmp, icmp_imm, or fcmp instruction. + cmp_inst: Inst, + /// The destination of the branch. + destination: Ebb, + /// The arguments of the branch. + args: ValueList, + /// The first argument to the comparison. The second is in the `kind` field. + cmp_arg: Value, + /// If the branch is `brz` rather than `brnz`, we need to invert the condition + /// before the branch. + invert_branch_cond: bool, + /// The kind of comparison, and the second argument. + kind: CmpBrKind, +} + +enum CmpBrKind { + Icmp { cond: IntCC, arg: Value }, + IcmpImm { cond: IntCC, imm: Imm64 }, + Fcmp { cond: FloatCC, arg: Value }, +} + +/// Optimize comparisons to use flags values, to avoid materializing conditions +/// in integer registers. +/// +/// For example, optimize icmp/fcmp brz/brnz sequences into ifcmp/ffcmp brif/brff +/// sequences. +fn optimize_cpu_flags( + pos: &mut EncCursor, + inst: Inst, + last_flags_clobber: Option, + isa: &TargetIsa, +) { + // Look for compare and branch patterns. + // This code could be considerably simplified with non-lexical lifetimes. + let info = match pos.func.dfg[inst] { + InstructionData::Branch { + opcode, + destination, + ref args, + } => { + let first_arg = args.first(&pos.func.dfg.value_lists).unwrap(); + let invert_branch_cond = match opcode { + Opcode::Brz => true, + Opcode::Brnz => false, + _ => panic!(), + }; + if let ValueDef::Result(cond_inst, _) = pos.func.dfg.value_def(first_arg) { + match pos.func.dfg[cond_inst] { + InstructionData::IntCompare { + cond, + args: cmp_args, + .. + } => { + CmpBrInfo { + br_inst: inst, + cmp_inst: cond_inst, + destination, + args: args.clone(), + cmp_arg: cmp_args[0], + invert_branch_cond, + kind: CmpBrKind::Icmp { + cond, + arg: cmp_args[1], + }, + } + } + InstructionData::IntCompareImm { + cond, + arg: cmp_arg, + imm: cmp_imm, + .. + } => { + CmpBrInfo { + br_inst: inst, + cmp_inst: cond_inst, + destination, + args: args.clone(), + cmp_arg, + invert_branch_cond, + kind: CmpBrKind::IcmpImm { cond, imm: cmp_imm }, + } + } + InstructionData::FloatCompare { + cond, + args: cmp_args, + .. + } => { + CmpBrInfo { + br_inst: inst, + cmp_inst: cond_inst, + destination, + args: args.clone(), + cmp_arg: cmp_args[0], + invert_branch_cond, + kind: CmpBrKind::Fcmp { + cond, + arg: cmp_args[1], + }, + } + } + _ => return, + } + } else { + return; + } + } + // TODO: trapif, trueif, selectif, and their ff counterparts. + _ => return, + }; + + // If any instructions clobber the flags between the comparison and the branch, + // don't optimize them. + if last_flags_clobber != Some(info.cmp_inst) { + return; + } + + // We found a compare+branch pattern. Transform it to use flags. + let args = info.args.as_slice(&pos.func.dfg.value_lists)[1..].to_vec(); + pos.goto_inst(info.cmp_inst); + match info.kind { + CmpBrKind::Icmp { mut cond, arg } => { + let flags = pos.ins().ifcmp(info.cmp_arg, arg); + pos.func.dfg.replace(info.cmp_inst).trueif(cond, flags); + if info.invert_branch_cond { + cond = cond.inverse(); + } + pos.func.dfg.replace(info.br_inst).brif( + cond, + flags, + info.destination, + &args, + ); + } + CmpBrKind::IcmpImm { mut cond, imm } => { + let flags = pos.ins().ifcmp_imm(info.cmp_arg, imm); + pos.func.dfg.replace(info.cmp_inst).trueif(cond, flags); + if info.invert_branch_cond { + cond = cond.inverse(); + } + pos.func.dfg.replace(info.br_inst).brif( + cond, + flags, + info.destination, + &args, + ); + } + CmpBrKind::Fcmp { mut cond, arg } => { + let flags = pos.ins().ffcmp(info.cmp_arg, arg); + pos.func.dfg.replace(info.cmp_inst).trueff(cond, flags); + if info.invert_branch_cond { + cond = cond.inverse(); + } + pos.func.dfg.replace(info.br_inst).brff( + cond, + flags, + info.destination, + &args, + ); + } + } + pos.func.update_encoding(info.cmp_inst, isa).is_ok(); + pos.func.update_encoding(info.br_inst, isa).is_ok(); +} + + +//---------------------------------------------------------------------- +// +// The main post-opt pass. + +pub fn do_postopt(func: &mut Function, isa: &TargetIsa) { + let _tt = timing::postopt(); + let mut pos = EncCursor::new(func, isa); + while let Some(_ebb) = pos.next_ebb() { + let mut last_flags_clobber = None; + while let Some(inst) = pos.next_inst() { + if isa.uses_cpu_flags() { + // Optimize instructions to make use of flags. + optimize_cpu_flags(&mut pos, inst, last_flags_clobber, isa); + + // Track the most recent seen instruction that clobbers the flags. + if let Some(constraints) = + isa.encoding_info().operand_constraints( + pos.func.encodings[inst], + ) + { + if constraints.clobbers_flags { + last_flags_clobber = Some(inst) + } + } + + } + } + } +} diff --git a/lib/cretonne/src/preopt.rs b/lib/cretonne/src/preopt.rs index 9adce51369..34dab71c4a 100644 --- a/lib/cretonne/src/preopt.rs +++ b/lib/cretonne/src/preopt.rs @@ -127,28 +127,6 @@ fn get_div_info(inst: Inst, dfg: &DataFlowGraph) -> Option { return package_up_divrem_info(arg, argL_ty, imm.into(), isSigned, isRem); } - // TODO: should we actually bother to do this (that is, manually match - // the case that the second argument is an iconst)? Or should we assume - // that some previous constant propagation pass has pushed all such - // immediates to their use points, creating BinaryImm instructions - // instead? For now we take the conservative approach. - if let InstructionData::Binary { opcode, args } = *idata { - let (isSigned, isRem) = match opcode { - Opcode::Udiv => (false, false), - Opcode::Urem => (false, true), - Opcode::Sdiv => (true, false), - Opcode::Srem => (true, true), - _other => return None, - }; - let argR: Value = args[1]; - if let Some(simm64) = get_const(argR, dfg) { - let argL: Value = args[0]; - // Pull the operation size (type) from the left arg - let argL_ty = dfg.value_type(argL); - return package_up_divrem_info(argL, argL_ty, simm64, isSigned, isRem); - } - } - None } @@ -473,25 +451,106 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } } - -//---------------------------------------------------------------------- -// -// General pattern-match helpers. - -/// Find out if `value` actually resolves to a constant, and if so what its -/// value is. -fn get_const(value: Value, dfg: &DataFlowGraph) -> Option { - match dfg.value_def(value) { - ValueDef::Result(definingInst, resultNo) => { - let definingIData: &InstructionData = &dfg[definingInst]; - if let InstructionData::UnaryImm { opcode, imm } = *definingIData { - if opcode == Opcode::Iconst && resultNo == 0 { - return Some(imm.into()); +/// Apply basic simplifications. +/// +/// This folds constants with arithmetic to form `_imm` instructions, and other +/// minor simplifications. +fn simplify(pos: &mut FuncCursor, inst: Inst) { + match pos.func.dfg[inst] { + InstructionData::Binary { opcode, args } => { + if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(args[1]) { + if let InstructionData::UnaryImm { + opcode: Opcode::Iconst, + mut imm, + } = pos.func.dfg[iconst_inst] + { + let new_opcode = match opcode { + Opcode::Iadd => Opcode::IaddImm, + Opcode::Imul => Opcode::ImulImm, + Opcode::Sdiv => Opcode::SdivImm, + Opcode::Udiv => Opcode::UdivImm, + Opcode::Srem => Opcode::SremImm, + Opcode::Urem => Opcode::UremImm, + Opcode::Band => Opcode::BandImm, + Opcode::Bor => Opcode::BorImm, + Opcode::Bxor => Opcode::BxorImm, + Opcode::Rotl => Opcode::RotlImm, + Opcode::Rotr => Opcode::RotrImm, + Opcode::Ishl => Opcode::IshlImm, + Opcode::Ushr => Opcode::UshrImm, + Opcode::Sshr => Opcode::SshrImm, + Opcode::Isub => { + imm = imm.wrapping_neg(); + Opcode::IaddImm + } + _ => return, + }; + let ty = pos.func.dfg.ctrl_typevar(inst); + pos.func.dfg.replace(inst).BinaryImm( + new_opcode, + ty, + imm, + args[0], + ); + } + } else if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(args[0]) { + if let InstructionData::UnaryImm { + opcode: Opcode::Iconst, + mut imm, + } = pos.func.dfg[iconst_inst] + { + let new_opcode = match opcode { + Opcode::Isub => Opcode::IrsubImm, + _ => return, + }; + let ty = pos.func.dfg.ctrl_typevar(inst); + pos.func.dfg.replace(inst).BinaryImm( + new_opcode, + ty, + imm, + args[0], + ); } } - None } - ValueDef::Param(_definingEbb, _paramNo) => None, + InstructionData::IntCompare { opcode, cond, args } => { + debug_assert_eq!(opcode, Opcode::Icmp); + if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(args[1]) { + if let InstructionData::UnaryImm { + opcode: Opcode::Iconst, + imm, + } = pos.func.dfg[iconst_inst] + { + pos.func.dfg.replace(inst).icmp_imm(cond, args[0], imm); + } + } + } + InstructionData::CondTrap { .. } | + InstructionData::Branch { .. } | + InstructionData::Ternary { opcode: Opcode::Select, .. } => { + // Fold away a redundant `bint`. + let maybe = { + let args = pos.func.dfg.inst_args(inst); + if let ValueDef::Result(def_inst, _) = pos.func.dfg.value_def(args[0]) { + if let InstructionData::Unary { + opcode: Opcode::Bint, + arg: bool_val, + } = pos.func.dfg[def_inst] + { + Some(bool_val) + } else { + None + } + } else { + None + } + }; + if let Some(bool_val) = maybe { + let args = pos.func.dfg.inst_args_mut(inst); + args[0] = bool_val; + } + } + _ => {} } } @@ -503,6 +562,8 @@ pub fn do_preopt(func: &mut Function) { while let Some(_ebb) = pos.next_ebb() { while let Some(inst) = pos.next_inst() { + // Apply basic simplifications. + simplify(&mut pos, inst); //-- BEGIN -- division by constants ---------------- diff --git a/lib/cretonne/src/timing.rs b/lib/cretonne/src/timing.rs index 51cdd85bfa..2626445921 100644 --- a/lib/cretonne/src/timing.rs +++ b/lib/cretonne/src/timing.rs @@ -55,7 +55,9 @@ define_passes!{ flowgraph: "Control flow graph", domtree: "Dominator tree", loop_analysis: "Loop analysis", + postopt: "Post-legalization rewriting", preopt: "Pre-legalization rewriting", + dce: "Dead code elimination", legalize: "Legalization", gvn: "Global value numbering", licm: "Loop invariant code motion", diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs index a6a43d4437..66bb7023ec 100644 --- a/lib/filetests/src/lib.rs +++ b/lib/filetests/src/lib.rs @@ -28,9 +28,11 @@ mod match_directive; mod test_binemit; mod test_cat; mod test_compile; +mod test_dce; mod test_domtree; mod test_legalizer; mod test_licm; +mod test_postopt; mod test_preopt; mod test_print_cfg; mod test_regalloc; @@ -73,9 +75,11 @@ fn new_subtest(parsed: &TestCommand) -> subtest::Result> { "binemit" => test_binemit::subtest(parsed), "cat" => test_cat::subtest(parsed), "compile" => test_compile::subtest(parsed), + "dce" => test_dce::subtest(parsed), "domtree" => test_domtree::subtest(parsed), "legalizer" => test_legalizer::subtest(parsed), "licm" => test_licm::subtest(parsed), + "postopt" => test_postopt::subtest(parsed), "preopt" => test_preopt::subtest(parsed), "print-cfg" => test_print_cfg::subtest(parsed), "regalloc" => test_regalloc::subtest(parsed), diff --git a/lib/filetests/src/test_dce.rs b/lib/filetests/src/test_dce.rs new file mode 100644 index 0000000000..5a57d83e3e --- /dev/null +++ b/lib/filetests/src/test_dce.rs @@ -0,0 +1,53 @@ +//! Test command for testing the DCE pass. +//! +//! The `dce` test command runs each function through the DCE pass after ensuring +//! that all instructions are legal for the target. +//! +//! The resulting function is sent to `filecheck`. + +use cretonne::ir::Function; +use cretonne; +use cretonne::print_errors::pretty_error; +use cton_reader::TestCommand; +use subtest::{SubTest, Context, Result, run_filecheck}; +use std::borrow::Cow; +use std::fmt::Write; + +struct TestDCE; + +pub fn subtest(parsed: &TestCommand) -> Result> { + assert_eq!(parsed.command, "dce"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestDCE)) + } +} + +impl SubTest for TestDCE { + fn name(&self) -> Cow { + Cow::from("dce") + } + + fn is_mutating(&self) -> bool { + true + } + + fn run(&self, func: Cow, context: &Context) -> Result<()> { + // Create a compilation context, and drop in the function. + let mut comp_ctx = cretonne::Context::new(); + comp_ctx.func = func.into_owned(); + + comp_ctx.flowgraph(); + comp_ctx.compute_loop_analysis(); + comp_ctx.dce(context.flags_or_isa()).map_err(|e| { + pretty_error(&comp_ctx.func, context.isa, Into::into(e)) + })?; + + let mut text = String::new(); + write!(&mut text, "{}", &comp_ctx.func).map_err( + |e| e.to_string(), + )?; + run_filecheck(&text, context) + } +} diff --git a/lib/filetests/src/test_postopt.rs b/lib/filetests/src/test_postopt.rs new file mode 100644 index 0000000000..34ffb3f462 --- /dev/null +++ b/lib/filetests/src/test_postopt.rs @@ -0,0 +1,50 @@ +//! Test command for testing the postopt pass. +//! +//! The resulting function is sent to `filecheck`. + +use cretonne::ir::Function; +use cretonne; +use cretonne::print_errors::pretty_error; +use cton_reader::TestCommand; +use subtest::{SubTest, Context, Result, run_filecheck}; +use std::borrow::Cow; +use std::fmt::Write; + +struct TestPostopt; + +pub fn subtest(parsed: &TestCommand) -> Result> { + assert_eq!(parsed.command, "postopt"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestPostopt)) + } +} + +impl SubTest for TestPostopt { + fn name(&self) -> Cow { + Cow::from("postopt") + } + + fn is_mutating(&self) -> bool { + true + } + + fn run(&self, func: Cow, context: &Context) -> Result<()> { + // Create a compilation context, and drop in the function. + let mut comp_ctx = cretonne::Context::new(); + comp_ctx.func = func.into_owned(); + let isa = context.isa.expect("postopt needs an ISA"); + + comp_ctx.flowgraph(); + comp_ctx.postopt(isa).map_err(|e| { + pretty_error(&comp_ctx.func, context.isa, Into::into(e)) + })?; + + let mut text = String::new(); + write!(&mut text, "{}", &comp_ctx.func).map_err( + |e| e.to_string(), + )?; + run_filecheck(&text, context) + } +} From 9e4ab7dc868346af30e7b00d850c3d3b5197f9c8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 30 Mar 2018 12:32:14 -0700 Subject: [PATCH 1670/3084] Rename CallConv::Native to CallConv::SystemV. (#291) To keep cross-compiling straightforward, Cretonne shouldn't have any behavior that depends on the host. This renames the "Native" calling convention to "SystemV", which has a defined meaning for each target, so that it's clear that the calling convention doesn't change depending on what host Cretonne is running on. --- cranelift/docs/callex.cton | 2 +- cranelift/docs/example.cton | 2 +- cranelift/docs/langref.rst | 6 +-- cranelift/filetests/domtree/loops.cton | 2 +- cranelift/filetests/domtree/loops2.cton | 2 +- cranelift/filetests/isa/intel/abi-bool.cton | 2 +- cranelift/filetests/isa/intel/abi32.cton | 12 +++--- cranelift/filetests/isa/intel/abi64.cton | 12 +++--- .../filetests/isa/intel/legalize-libcall.cton | 4 +- .../isa/intel/prologue-epilogue.cton | 2 +- cranelift/filetests/isa/riscv/abi-e.cton | 4 +- cranelift/filetests/isa/riscv/abi.cton | 24 +++++------ .../filetests/isa/riscv/legalize-abi.cton | 4 +- .../filetests/isa/riscv/parse-encoding.cton | 30 +++++++------- cranelift/filetests/licm/complex.cton | 2 +- cranelift/filetests/parser/branch.cton | 12 +++--- cranelift/filetests/parser/call.cton | 10 ++--- .../parser/instruction_encoding.cton | 2 +- cranelift/filetests/parser/keywords.cton | 2 +- cranelift/filetests/parser/rewrite.cton | 4 +- cranelift/filetests/parser/tiny.cton | 28 ++++++------- cranelift/filetests/regalloc/coalesce.cton | 2 +- .../filetests/regalloc/coalescing-207.cton | 12 +++--- .../filetests/regalloc/coalescing-216.cton | 2 +- .../filetests/regalloc/coloring-227.cton | 2 +- cranelift/filetests/regalloc/ghost-param.cton | 2 +- .../filetests/regalloc/global-fixed.cton | 2 +- .../filetests/regalloc/intel-regres.cton | 2 +- .../regalloc/output-interference.cton | 2 +- cranelift/filetests/regalloc/reload-208.cton | 6 +-- cranelift/filetests/regalloc/reload.cton | 2 +- .../filetests/regalloc/schedule-moves.cton | 4 +- .../filetests/regalloc/spill-noregs.cton | 2 +- cranelift/filetests/regalloc/spill.cton | 4 +- .../verifier/defs_dominates_uses.cton | 4 +- lib/cretonne/src/ir/extfunc.rs | 13 +++--- lib/cretonne/src/ir/function.rs | 6 +-- lib/cretonne/src/isa/intel/abi.rs | 16 ++++---- lib/cretonne/src/legalizer/libcall.rs | 4 +- lib/cretonne/src/write.rs | 12 +++--- lib/frontend/src/frontend.rs | 2 +- lib/frontend/src/lib.rs | 2 +- lib/reader/src/parser.rs | 40 +++++++++---------- lib/wasm/src/sections_translator.rs | 2 +- 44 files changed, 157 insertions(+), 156 deletions(-) diff --git a/cranelift/docs/callex.cton b/cranelift/docs/callex.cton index 6690e78f90..837f9ea6e7 100644 --- a/cranelift/docs/callex.cton +++ b/cranelift/docs/callex.cton @@ -1,6 +1,6 @@ test verifier -function %gcd(i32 uext, i32 uext) -> i32 uext native { +function %gcd(i32 uext, i32 uext) -> i32 uext system_v { fn1 = function %divmod(i32 uext, i32 uext) -> i32 uext, i32 uext ebb1(v1: i32, v2: i32): diff --git a/cranelift/docs/example.cton b/cranelift/docs/example.cton index a36fef3238..09e0c9cb5f 100644 --- a/cranelift/docs/example.cton +++ b/cranelift/docs/example.cton @@ -1,6 +1,6 @@ test verifier -function %average(i32, i32) -> f32 native { +function %average(i32, i32) -> f32 system_v { ss1 = explicit_slot 8 ; Stack slot for ``sum``. ebb1(v1: i32, v2: i32): diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 162248c83e..edcc039c86 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -400,11 +400,11 @@ convention: param : type [paramext] [paramspecial] paramext : "uext" | "sext" paramspecial : "sret" | "link" | "fp" | "csr" | "vmctx" - callconv : "native" | "spiderwasm" + callconv : "system_v" | "spiderwasm" Parameters and return values have flags whose meaning is mostly target -dependent. They make it possible to call native functions on the target -platform. When calling other Cretonne functions, the flags are not necessary. +dependent. These flags support interfacing with code produced by other +compilers. Functions that are called directly must be declared in the :term:`function preamble`: diff --git a/cranelift/filetests/domtree/loops.cton b/cranelift/filetests/domtree/loops.cton index 70c62d4130..fda2984e94 100644 --- a/cranelift/filetests/domtree/loops.cton +++ b/cranelift/filetests/domtree/loops.cton @@ -59,7 +59,7 @@ function %test(i32) { ; nextln: ebb5: ; nextln: } -function %loop2(i32) native { +function %loop2(i32) system_v { ebb0(v0: i32): brz v0, ebb1 ; dominates: ebb1 ebb3 ebb4 ebb5 jump ebb2 ; dominates: ebb2 diff --git a/cranelift/filetests/domtree/loops2.cton b/cranelift/filetests/domtree/loops2.cton index 029cea922a..80e00ca278 100644 --- a/cranelift/filetests/domtree/loops2.cton +++ b/cranelift/filetests/domtree/loops2.cton @@ -43,7 +43,7 @@ function %loop1(i32) { ; nextln: ebb9: ; nextln: } -function %loop2(i32) native { +function %loop2(i32) system_v { ebb0(v0: i32): brz v0, ebb1 ; dominates: ebb1 ebb3 ebb4 ebb5 jump ebb2 ; dominates: ebb2 diff --git a/cranelift/filetests/isa/intel/abi-bool.cton b/cranelift/filetests/isa/intel/abi-bool.cton index bc1a1ba1d5..6a53cdbb3c 100644 --- a/cranelift/filetests/isa/intel/abi-bool.cton +++ b/cranelift/filetests/isa/intel/abi-bool.cton @@ -2,7 +2,7 @@ test compile set is_64bit=1 isa intel haswell -function %foo(i64, i64, i64, i32) -> b1 native { +function %foo(i64, i64, i64, i32) -> b1 system_v { ebb3(v0: i64, v1: i64, v2: i64, v3: i32): v5 = icmp ne v2, v2 v8 = iconst.i64 0 diff --git a/cranelift/filetests/isa/intel/abi32.cton b/cranelift/filetests/isa/intel/abi32.cton index c7e399960c..b949af16f7 100644 --- a/cranelift/filetests/isa/intel/abi32.cton +++ b/cranelift/filetests/isa/intel/abi32.cton @@ -5,14 +5,14 @@ isa intel ; regex: V=v\d+ function %f() { - sig0 = (i32) -> i32 native - ; check: sig0 = (i32 [0]) -> i32 [%rax] native + sig0 = (i32) -> i32 system_v + ; check: sig0 = (i32 [0]) -> i32 [%rax] system_v - sig1 = (i64) -> b1 native - ; check: sig1 = (i32 [0], i32 [4]) -> b1 [%rax] native + sig1 = (i64) -> b1 system_v + ; check: sig1 = (i32 [0], i32 [4]) -> b1 [%rax] system_v - sig2 = (f32, i64) -> f64 native - ; check: sig2 = (f32 [0], i32 [4], i32 [8]) -> f64 [%xmm0] native + sig2 = (f32, i64) -> f64 system_v + ; check: sig2 = (f32 [0], i32 [4], i32 [8]) -> f64 [%xmm0] system_v ebb0: return diff --git a/cranelift/filetests/isa/intel/abi64.cton b/cranelift/filetests/isa/intel/abi64.cton index d56a31d479..8aa6a49b0f 100644 --- a/cranelift/filetests/isa/intel/abi64.cton +++ b/cranelift/filetests/isa/intel/abi64.cton @@ -6,14 +6,14 @@ isa intel ; regex: V=v\d+ function %f() { - sig0 = (i32) -> i32 native - ; check: sig0 = (i32 [%rdi]) -> i32 [%rax] native + sig0 = (i32) -> i32 system_v + ; check: sig0 = (i32 [%rdi]) -> i32 [%rax] system_v - sig1 = (i64) -> b1 native - ; check: sig1 = (i64 [%rdi]) -> b1 [%rax] native + sig1 = (i64) -> b1 system_v + ; check: sig1 = (i64 [%rdi]) -> b1 [%rax] system_v - sig2 = (f32, i64) -> f64 native - ; check: sig2 = (f32 [%xmm0], i64 [%rdi]) -> f64 [%xmm0] native + sig2 = (f32, i64) -> f64 system_v + ; check: sig2 = (f32 [%xmm0], i64 [%rdi]) -> f64 [%xmm0] system_v ebb0: return diff --git a/cranelift/filetests/isa/intel/legalize-libcall.cton b/cranelift/filetests/isa/intel/legalize-libcall.cton index 562efb6beb..c15587fc7f 100644 --- a/cranelift/filetests/isa/intel/legalize-libcall.cton +++ b/cranelift/filetests/isa/intel/legalize-libcall.cton @@ -9,7 +9,7 @@ ebb0(v0: f32): v1 = floor v0 return v1 } -; check: function %floor(f32 [%xmm0]) -> f32 [%xmm0] native { -; check: sig0 = (f32) -> f32 native +; check: function %floor(f32 [%xmm0]) -> f32 [%xmm0] system_v { +; check: sig0 = (f32) -> f32 system_v ; check: fn0 = sig0 %FloorF32 ; check: v1 = call fn0(v0) diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/intel/prologue-epilogue.cton index fc53f72008..cd774fe6bd 100644 --- a/cranelift/filetests/isa/intel/prologue-epilogue.cton +++ b/cranelift/filetests/isa/intel/prologue-epilogue.cton @@ -9,7 +9,7 @@ ebb0: return } -; check: function %foo(i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] native { +; check: function %foo(i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] system_v { ; nextln: ss0 = explicit_slot 168, offset -224 ; nextln: ss1 = incoming_arg 56, offset -56 ; check: ebb0(v0: i64 [%rbp], v1: i64 [%rbx], v2: i64 [%r12], v3: i64 [%r13], v4: i64 [%r14], v5: i64 [%r15]): diff --git a/cranelift/filetests/isa/riscv/abi-e.cton b/cranelift/filetests/isa/riscv/abi-e.cton index df06402283..50100ebf4e 100644 --- a/cranelift/filetests/isa/riscv/abi-e.cton +++ b/cranelift/filetests/isa/riscv/abi-e.cton @@ -7,8 +7,8 @@ isa riscv enable_e function %f() { ; Spilling into the stack args after %x15 since %16 and up are not ; available in RV32E. - sig0 = (i64, i64, i64, i64) -> i64 native - ; check: sig0 = (i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [0], i32 [4]) -> i32 [%x10], i32 [%x11] native + sig0 = (i64, i64, i64, i64) -> i64 system_v + ; check: sig0 = (i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [0], i32 [4]) -> i32 [%x10], i32 [%x11] system_v ebb0: return } diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.cton index c57c09fd97..12371ac537 100644 --- a/cranelift/filetests/isa/riscv/abi.cton +++ b/cranelift/filetests/isa/riscv/abi.cton @@ -5,27 +5,27 @@ isa riscv ; regex: V=v\d+ function %f() { - sig0 = (i32) -> i32 native - ; check: sig0 = (i32 [%x10]) -> i32 [%x10] native + sig0 = (i32) -> i32 system_v + ; check: sig0 = (i32 [%x10]) -> i32 [%x10] system_v - sig1 = (i64) -> b1 native - ; check: sig1 = (i32 [%x10], i32 [%x11]) -> b1 [%x10] native + sig1 = (i64) -> b1 system_v + ; check: sig1 = (i32 [%x10], i32 [%x11]) -> b1 [%x10] system_v ; The i64 argument must go in an even-odd register pair. - sig2 = (f32, i64) -> f64 native - ; check: sig2 = (f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] native + sig2 = (f32, i64) -> f64 system_v + ; check: sig2 = (f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] system_v ; Spilling into the stack args. - sig3 = (f64, f64, f64, f64, f64, f64, f64, i64) -> f64 native - ; check: sig3 = (f64 [%f10], f64 [%f11], f64 [%f12], f64 [%f13], f64 [%f14], f64 [%f15], f64 [%f16], i32 [0], i32 [4]) -> f64 [%f10] native + sig3 = (f64, f64, f64, f64, f64, f64, f64, i64) -> f64 system_v + ; check: sig3 = (f64 [%f10], f64 [%f11], f64 [%f12], f64 [%f13], f64 [%f14], f64 [%f15], f64 [%f16], i32 [0], i32 [4]) -> f64 [%f10] system_v ; Splitting vectors. - sig4 = (i32x4) native - ; check: sig4 = (i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13]) native + sig4 = (i32x4) system_v + ; check: sig4 = (i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13]) system_v ; Splitting vectors, then splitting ints. - sig5 = (i64x4) native - ; check: sig5 = (i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [%x16], i32 [%x17]) native + sig5 = (i64x4) system_v + ; check: sig5 = (i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [%x16], i32 [%x17]) system_v ebb0: return diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index 3234407052..44865120ea 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -106,7 +106,7 @@ ebb0(v0: i64x4): } function %indirect(i32) { - sig1 = () native + sig1 = () system_v ebb0(v0: i32): call_indirect sig1, v0() return @@ -114,7 +114,7 @@ ebb0(v0: i32): ; The first argument to call_indirect doesn't get altered. function %indirect_arg(i32, f32x2) { - sig1 = (f32x2) native + sig1 = (f32x2) system_v ebb0(v0: i32, v1: f32x2): call_indirect sig1, v0(v1) ; check: call_indirect sig1, v0($V, $V) diff --git a/cranelift/filetests/isa/riscv/parse-encoding.cton b/cranelift/filetests/isa/riscv/parse-encoding.cton index e955ad296a..0fc0879f38 100644 --- a/cranelift/filetests/isa/riscv/parse-encoding.cton +++ b/cranelift/filetests/isa/riscv/parse-encoding.cton @@ -3,32 +3,32 @@ test legalizer isa riscv function %parse_encoding(i32 [%x5]) -> i32 [%x10] { - ; check: function %parse_encoding(i32 [%x5], i32 link [%x1]) -> i32 [%x10], i32 link [%x1] native { + ; check: function %parse_encoding(i32 [%x5], i32 link [%x1]) -> i32 [%x10], i32 link [%x1] system_v { - sig0 = (i32 [%x10]) -> i32 [%x10] native - ; check: sig0 = (i32 [%x10]) -> i32 [%x10] native + sig0 = (i32 [%x10]) -> i32 [%x10] system_v + ; check: sig0 = (i32 [%x10]) -> i32 [%x10] system_v - sig1 = (i32 [%x10], i32 [%x11]) -> b1 [%x10] native - ; check: sig1 = (i32 [%x10], i32 [%x11]) -> b1 [%x10] native + sig1 = (i32 [%x10], i32 [%x11]) -> b1 [%x10] system_v + ; check: sig1 = (i32 [%x10], i32 [%x11]) -> b1 [%x10] system_v - sig2 = (f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] native - ; check: sig2 = (f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] native + sig2 = (f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] system_v + ; check: sig2 = (f32 [%f10], i32 [%x12], i32 [%x13]) -> f64 [%f10] system_v ; Arguments on stack where not necessary - sig3 = (f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] native - ; check: sig3 = (f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] native + sig3 = (f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] system_v + ; check: sig3 = (f64 [%f10], i32 [0], i32 [4]) -> f64 [%f10] system_v ; Stack argument before register argument - sig4 = (f32 [72], i32 [%x10]) native - ; check: sig4 = (f32 [72], i32 [%x10]) native + sig4 = (f32 [72], i32 [%x10]) system_v + ; check: sig4 = (f32 [72], i32 [%x10]) system_v ; Return value on stack - sig5 = () -> f32 [0] native - ; check: sig5 = () -> f32 [0] native + sig5 = () -> f32 [0] system_v + ; check: sig5 = () -> f32 [0] system_v ; function + signature - fn0 = function %bar(i32 [%x10]) -> b1 [%x10] native - ; check: sig6 = (i32 [%x10]) -> b1 [%x10] native + fn0 = function %bar(i32 [%x10]) -> b1 [%x10] system_v + ; check: sig6 = (i32 [%x10]) -> b1 [%x10] system_v ; nextln: fn0 = sig6 %bar ebb0(v0: i32): diff --git a/cranelift/filetests/licm/complex.cton b/cranelift/filetests/licm/complex.cton index e9012b1a4f..3b339574a8 100644 --- a/cranelift/filetests/licm/complex.cton +++ b/cranelift/filetests/licm/complex.cton @@ -1,6 +1,6 @@ test licm -function %complex(i32) -> i32 native { +function %complex(i32) -> i32 system_v { ebb0(v0: i32): jump ebb1(v0) diff --git a/cranelift/filetests/parser/branch.cton b/cranelift/filetests/parser/branch.cton index 868b92d024..4e35147a6b 100644 --- a/cranelift/filetests/parser/branch.cton +++ b/cranelift/filetests/parser/branch.cton @@ -9,7 +9,7 @@ ebb0: ebb1: jump ebb0() } -; sameln: function %minimal() native { +; sameln: function %minimal() system_v { ; nextln: ebb0: ; nextln: jump ebb1 ; nextln: @@ -25,7 +25,7 @@ ebb0(v90: i32): ebb1(v91: i32): jump ebb0(v91) } -; sameln: function %onearg(i32) native { +; sameln: function %onearg(i32) system_v { ; nextln: ebb0(v90: i32): ; nextln: jump ebb1(v90) ; nextln: @@ -41,7 +41,7 @@ ebb0(v90: i32, v91: f32): ebb1(v92: i32, v93: f32): jump ebb0(v92, v93) } -; sameln: function %twoargs(i32, f32) native { +; sameln: function %twoargs(i32, f32) system_v { ; nextln: ebb0(v90: i32, v91: f32): ; nextln: jump ebb1(v90, v91) ; nextln: @@ -57,7 +57,7 @@ ebb0(v90: i32): ebb1: brnz v90, ebb1() } -; sameln: function %minimal(i32) native { +; sameln: function %minimal(i32) system_v { ; nextln: ebb0(v90: i32): ; nextln: brz v90, ebb1 ; nextln: @@ -72,7 +72,7 @@ ebb0(v90: i32, v91: f32): ebb1(v92: i32, v93: f32): brnz v90, ebb0(v92, v93) } -; sameln: function %twoargs(i32, f32) native { +; sameln: function %twoargs(i32, f32) system_v { ; nextln: ebb0(v90: i32, v91: f32): ; nextln: brz v90, ebb1(v90, v91) ; nextln: @@ -94,7 +94,7 @@ ebb30: ebb40: trap user4 } -; sameln: function %jumptable(i32) native { +; sameln: function %jumptable(i32) system_v { ; check: jt2 = jump_table 0, 0, ebb10, ebb40, ebb20, ebb30 ; check: jt200 = jump_table 0 ; check: ebb10(v3: i32): diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton index 77543800d0..3413696caf 100644 --- a/cranelift/filetests/parser/call.cton +++ b/cranelift/filetests/parser/call.cton @@ -5,7 +5,7 @@ function %mini() { ebb1: return } -; sameln: function %mini() native { +; sameln: function %mini() system_v { ; nextln: ebb1: ; nextln: return ; nextln: } @@ -29,10 +29,10 @@ function %signatures() { fn5 = sig11 %foo fn8 = function %bar(i32) -> b1 } -; sameln: function %signatures() native { -; check: sig10 = () native +; sameln: function %signatures() system_v { +; check: sig10 = () system_v ; check: sig11 = (i32, f64) -> i32, b1 spiderwasm -; check: sig12 = (i32) -> b1 native +; check: sig12 = (i32) -> b1 system_v ; not: fn0 ; check: fn5 = sig11 %foo ; check: fn8 = sig12 %bar @@ -88,7 +88,7 @@ function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 ebb0(v1: i32, v2: i32, v3: i32, v4: i32): return v4, v2, v3, v1 } -; check: function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret native { +; check: function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret system_v { ; check: ebb0(v1: i32, v2: i32, v3: i32, v4: i32): ; check: return v4, v2, v3, v1 ; check: } diff --git a/cranelift/filetests/parser/instruction_encoding.cton b/cranelift/filetests/parser/instruction_encoding.cton index a16c24a364..e0a2de789a 100644 --- a/cranelift/filetests/parser/instruction_encoding.cton +++ b/cranelift/filetests/parser/instruction_encoding.cton @@ -13,7 +13,7 @@ ebb1(v0: i32 [%x8], v1: i32): @55 v9 = iadd v8, v7 @a5 [Iret#5] return v0, v8 } -; sameln: function %foo(i32, i32) native { +; sameln: function %foo(i32, i32) system_v { ; nextln: ebb1(v0: i32 [%x8], v1: i32): ; nextln: [-,-]$WS v2 = iadd v0, v1 ; nextln: [-]$WS trap heap_oob diff --git a/cranelift/filetests/parser/keywords.cton b/cranelift/filetests/parser/keywords.cton index a4b894574e..aaf8403c0a 100644 --- a/cranelift/filetests/parser/keywords.cton +++ b/cranelift/filetests/parser/keywords.cton @@ -2,4 +2,4 @@ test cat ; 'function' is not a keyword, and can be used as the name of a function too. function %function() {} -; check: function %function() native +; check: function %function() system_v diff --git a/cranelift/filetests/parser/rewrite.cton b/cranelift/filetests/parser/rewrite.cton index a62157082a..679f00d2bc 100644 --- a/cranelift/filetests/parser/rewrite.cton +++ b/cranelift/filetests/parser/rewrite.cton @@ -9,7 +9,7 @@ ebb100(v20: i32): v9200 = f64const 0x4.0p0 trap user4 } -; sameln: function %defs() native { +; sameln: function %defs() system_v { ; nextln: ebb100(v20: i32): ; nextln: v1000 = iconst.i32x8 5 ; nextln: v9200 = f64const 0x1.0000000000000p2 @@ -23,7 +23,7 @@ ebb100(v20: i32): v200 = iadd v20, v1000 jump ebb100(v1000) } -; sameln: function %use_value() native { +; sameln: function %use_value() system_v { ; nextln: ebb100(v20: i32): ; nextln: v1000 = iadd_imm v20, 5 ; nextln: v200 = iadd v20, v1000 diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 0c619565a0..e800d44c26 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -5,7 +5,7 @@ function %minimal() { ebb0: trap user0 } -; sameln: function %minimal() native { +; sameln: function %minimal() system_v { ; nextln: ebb0: ; nextln: trap user0 ; nextln: } @@ -18,7 +18,7 @@ ebb0: v1 = iconst.i8 6 v2 = ishl v0, v1 } -; sameln: function %ivalues() native { +; sameln: function %ivalues() system_v { ; nextln: ebb0: ; nextln: v0 = iconst.i32 2 ; nextln: v1 = iconst.i8 6 @@ -34,7 +34,7 @@ ebb0: v2 = bextend.b32 v1 v3 = bxor v0, v2 } -; sameln: function %bvalues() native { +; sameln: function %bvalues() system_v { ; nextln: ebb0: ; nextln: v0 = bconst.b32 true ; nextln: v1 = bconst.b8 false @@ -47,17 +47,17 @@ function %select() { ebb0(v90: i32, v91: i32, v92: b1): v0 = select v92, v90, v91 } -; sameln: function %select() native { +; sameln: function %select() system_v { ; nextln: ebb0(v90: i32, v91: i32, v92: b1): ; nextln: v0 = select v92, v90, v91 ; nextln: } ; Polymorphic instruction controlled by third operand. -function %selectif() native { +function %selectif() system_v { ebb0(v95: i32, v96: i32, v97: b1): v98 = selectif.i32 eq v97, v95, v96 } -; sameln: function %selectif() native { +; sameln: function %selectif() system_v { ; nextln: ebb0(v95: i32, v96: i32, v97: b1): ; nextln: v98 = selectif.i32 eq v97, v95, v96 ; nextln: } @@ -69,7 +69,7 @@ ebb0: v1 = extractlane v0, 3 v2 = insertlane v0, 1, v1 } -; sameln: function %lanes() native { +; sameln: function %lanes() system_v { ; nextln: ebb0: ; nextln: v0 = iconst.i32x4 2 ; nextln: v1 = extractlane v0, 3 @@ -85,7 +85,7 @@ ebb0(v90: i32, v91: i32): v3 = irsub_imm v91, 45 br_icmp eq v90, v91, ebb0(v91, v90) } -; sameln: function %icmp(i32, i32) native { +; sameln: function %icmp(i32, i32) system_v { ; nextln: ebb0(v90: i32, v91: i32): ; nextln: v0 = icmp eq v90, v91 ; nextln: v1 = icmp ult v90, v91 @@ -101,7 +101,7 @@ ebb0(v90: f32, v91: f32): v1 = fcmp uno v90, v91 v2 = fcmp lt v90, v91 } -; sameln: function %fcmp(f32, f32) native { +; sameln: function %fcmp(f32, f32) system_v { ; nextln: ebb0(v90: f32, v91: f32): ; nextln: v0 = fcmp eq v90, v91 ; nextln: v1 = fcmp uno v90, v91 @@ -115,7 +115,7 @@ ebb0(v90: i32, v91: f32): v0 = bitcast.i8x4 v90 v1 = bitcast.i32 v91 } -; sameln: function %bitcast(i32, f32) native { +; sameln: function %bitcast(i32, f32) system_v { ; nextln: ebb0(v90: i32, v91: f32): ; nextln: v0 = bitcast.i8x4 v90 ; nextln: v1 = bitcast.i32 v91 @@ -135,7 +135,7 @@ ebb0: stack_store v1, ss10+2 stack_store v2, ss2 } -; sameln: function %stack() native { +; sameln: function %stack() system_v { ; check: ss2 = explicit_slot 4 ; check: ss3 = incoming_arg 4, offset 8 ; check: ss4 = outgoing_arg 4 @@ -162,7 +162,7 @@ ebb0(v1: i32): store aligned v3, v1+12 store notrap aligned v3, v1-12 } -; sameln: function %memory(i32) native { +; sameln: function %memory(i32) system_v { ; nextln: ebb0(v1: i32): ; nextln: v2 = load.i64 v1 ; nextln: v3 = load.i64 aligned v1 @@ -187,7 +187,7 @@ ebb0(v1: i32): regfill v1, ss0 -> %10 return } -; sameln: function %diversion(i32) native { +; sameln: function %diversion(i32) system_v { ; nextln: ss0 = spill_slot 4 ; check: ebb0(v1: i32): ; nextln: regmove v1, %10 -> %20 @@ -204,7 +204,7 @@ ebb0: copy_special %20 -> %10 return } -; sameln: function %copy_special() native { +; sameln: function %copy_special() system_v { ; nextln: ebb0: ; nextln: copy_special %10 -> %20 ; nextln: copy_special %20 -> %10 diff --git a/cranelift/filetests/regalloc/coalesce.cton b/cranelift/filetests/regalloc/coalesce.cton index b2700b01dd..04782fc1fc 100644 --- a/cranelift/filetests/regalloc/coalesce.cton +++ b/cranelift/filetests/regalloc/coalesce.cton @@ -109,7 +109,7 @@ ebb1(v10: i32): return v11 } -function %gvn_unremovable_phi(i32) native { +function %gvn_unremovable_phi(i32) system_v { ebb0(v0: i32): v2 = iconst.i32 0 jump ebb2(v2, v0) diff --git a/cranelift/filetests/regalloc/coalescing-207.cton b/cranelift/filetests/regalloc/coalescing-207.cton index 8641c87354..c4d348e10b 100644 --- a/cranelift/filetests/regalloc/coalescing-207.cton +++ b/cranelift/filetests/regalloc/coalescing-207.cton @@ -5,12 +5,12 @@ isa intel haswell ; Reported as https://github.com/Cretonne/cretonne/issues/207 ; ; The coalescer creates a virtual register with two interfering values. -function %pr207(i64 vmctx, i32, i32) -> i32 native { +function %pr207(i64 vmctx, i32, i32) -> i32 system_v { gv0 = vmctx-8 heap0 = static gv0, min 0, bound 0x5000, guard 0x0040_0000 - sig0 = (i64 vmctx, i32, i32) -> i32 native - sig1 = (i64 vmctx, i32, i32, i32) -> i32 native - sig2 = (i64 vmctx, i32, i32, i32) -> i32 native + sig0 = (i64 vmctx, i32, i32) -> i32 system_v + sig1 = (i64 vmctx, i32, i32, i32) -> i32 system_v + sig2 = (i64 vmctx, i32, i32, i32) -> i32 system_v fn0 = sig0 u0:2 fn1 = sig1 u0:0 fn2 = sig2 u0:1 @@ -1034,10 +1034,10 @@ ebb92(v767: i32): } ; Same problem from musl.wasm. -function %musl(f64 [%xmm0], i64 vmctx [%rdi]) -> f64 [%xmm0] native { +function %musl(f64 [%xmm0], i64 vmctx [%rdi]) -> f64 [%xmm0] system_v { gv0 = vmctx heap0 = static gv0, min 0, bound 0x0001_0000_0000, guard 0x8000_0000 - sig0 = (f64 [%xmm0], i32 [%rdi], i64 vmctx [%rsi]) -> f64 [%xmm0] native + sig0 = (f64 [%xmm0], i32 [%rdi], i64 vmctx [%rsi]) -> f64 [%xmm0] system_v fn0 = sig0 u0:517 ebb0(v0: f64, v1: i64): diff --git a/cranelift/filetests/regalloc/coalescing-216.cton b/cranelift/filetests/regalloc/coalescing-216.cton index de44d53d5b..a6cf7b3b42 100644 --- a/cranelift/filetests/regalloc/coalescing-216.cton +++ b/cranelift/filetests/regalloc/coalescing-216.cton @@ -5,7 +5,7 @@ isa intel haswell ; Reported as https://github.com/Cretonne/cretonne/issues/216 from the Binaryen fuzzer. ; ; The (old) coalescer creates a virtual register with two identical values. -function %pr216(i32 [%rdi], i64 vmctx [%rsi]) -> i64 [%rax] native { +function %pr216(i32 [%rdi], i64 vmctx [%rsi]) -> i64 [%rax] system_v { ebb0(v0: i32, v1: i64): v3 = iconst.i64 0 v5 = iconst.i32 0 diff --git a/cranelift/filetests/regalloc/coloring-227.cton b/cranelift/filetests/regalloc/coloring-227.cton index 7e2dac2e50..accfb3528b 100644 --- a/cranelift/filetests/regalloc/coloring-227.cton +++ b/cranelift/filetests/regalloc/coloring-227.cton @@ -2,7 +2,7 @@ test regalloc set is_64bit isa intel haswell -function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) native { +function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) system_v { gv0 = vmctx heap0 = static gv0, min 0, bound 0x0001_0000_0000, guard 0x8000_0000 diff --git a/cranelift/filetests/regalloc/ghost-param.cton b/cranelift/filetests/regalloc/ghost-param.cton index 8b0ba6c2ab..c4ce229360 100644 --- a/cranelift/filetests/regalloc/ghost-param.cton +++ b/cranelift/filetests/regalloc/ghost-param.cton @@ -9,7 +9,7 @@ isa intel haswell ; ; Test case by binaryen fuzzer! -function %pr215(i64 vmctx [%rdi]) native { +function %pr215(i64 vmctx [%rdi]) system_v { ebb0(v0: i64): v10 = iconst.i64 0 v1 = bitcast.f64 v10 diff --git a/cranelift/filetests/regalloc/global-fixed.cton b/cranelift/filetests/regalloc/global-fixed.cton index a332311b6e..14b9d8fecc 100644 --- a/cranelift/filetests/regalloc/global-fixed.cton +++ b/cranelift/filetests/regalloc/global-fixed.cton @@ -2,7 +2,7 @@ test regalloc set is_64bit=1 isa intel haswell -function %foo() native { +function %foo() system_v { ebb4: v3 = iconst.i32 0 jump ebb3 diff --git a/cranelift/filetests/regalloc/intel-regres.cton b/cranelift/filetests/regalloc/intel-regres.cton index 50384d6349..0f76e8f2d0 100644 --- a/cranelift/filetests/regalloc/intel-regres.cton +++ b/cranelift/filetests/regalloc/intel-regres.cton @@ -11,7 +11,7 @@ isa intel ; This ended up confusong the constraint solver which had not made a record of ; the fixed register assignment for v9 since it was already in the correct ; register. -function %pr147(i32) -> i32 native { +function %pr147(i32) -> i32 system_v { ebb0(v0: i32): v1 = iconst.i32 0 v2 = iconst.i32 1 diff --git a/cranelift/filetests/regalloc/output-interference.cton b/cranelift/filetests/regalloc/output-interference.cton index 666c13c0d7..4a5b90c856 100644 --- a/cranelift/filetests/regalloc/output-interference.cton +++ b/cranelift/filetests/regalloc/output-interference.cton @@ -2,7 +2,7 @@ test regalloc set is_64bit=1 isa intel haswell -function %test(i64) -> i64 native { +function %test(i64) -> i64 system_v { ebb0(v0: i64): v2 = iconst.i64 12 ; This division clobbers two of its fixed input registers on Intel. diff --git a/cranelift/filetests/regalloc/reload-208.cton b/cranelift/filetests/regalloc/reload-208.cton index 1897a03c5d..70dc1c4523 100644 --- a/cranelift/filetests/regalloc/reload-208.cton +++ b/cranelift/filetests/regalloc/reload-208.cton @@ -11,11 +11,11 @@ isa intel haswell ; ; The problem was the reload pass rewriting EBB arguments on "brnz v9, ebb3(v9)" -function %pr208(i64 vmctx [%rdi]) native { +function %pr208(i64 vmctx [%rdi]) system_v { gv0 = vmctx-8 heap0 = static gv0, min 0, bound 0x5000, guard 0x0040_0000 - sig0 = (i64 vmctx [%rdi]) -> i32 [%rax] native - sig1 = (i64 vmctx [%rdi], i32 [%rsi]) native + sig0 = (i64 vmctx [%rdi]) -> i32 [%rax] system_v + sig1 = (i64 vmctx [%rdi], i32 [%rsi]) system_v fn0 = sig0 u0:1 fn1 = sig1 u0:3 diff --git a/cranelift/filetests/regalloc/reload.cton b/cranelift/filetests/regalloc/reload.cton index 68a6e8a149..5e62db3213 100644 --- a/cranelift/filetests/regalloc/reload.cton +++ b/cranelift/filetests/regalloc/reload.cton @@ -5,7 +5,7 @@ isa riscv enable_e ; Check that we can handle a function return value that got spilled. function %spill_return() -> i32 { - fn0 = function %foo() -> i32 native + fn0 = function %foo() -> i32 system_v ebb0: v0 = call fn0() diff --git a/cranelift/filetests/regalloc/schedule-moves.cton b/cranelift/filetests/regalloc/schedule-moves.cton index e3bae45be3..6ba78db893 100644 --- a/cranelift/filetests/regalloc/schedule-moves.cton +++ b/cranelift/filetests/regalloc/schedule-moves.cton @@ -1,7 +1,7 @@ test regalloc isa intel haswell -function %pr165() native { +function %pr165() system_v { ebb0: v0 = iconst.i32 0x0102_0304 v1 = iconst.i32 0x1102_0304 @@ -19,7 +19,7 @@ ebb0: ; Same as above, but use so many registers that spilling is required. ; Note: This is also a candidate for using xchg instructions. -function %emergency_spill() native { +function %emergency_spill() system_v { ebb0: v0 = iconst.i32 0x0102_0304 v1 = iconst.i32 0x1102_0304 diff --git a/cranelift/filetests/regalloc/spill-noregs.cton b/cranelift/filetests/regalloc/spill-noregs.cton index de4fa618bf..733606b828 100644 --- a/cranelift/filetests/regalloc/spill-noregs.cton +++ b/cranelift/filetests/regalloc/spill-noregs.cton @@ -13,7 +13,7 @@ isa intel ; ; The spiller was not releasing register pressure for dead EBB parameters. -function %pr223(i32 [%rdi], i64 vmctx [%rsi]) -> i64 [%rax] native { +function %pr223(i32 [%rdi], i64 vmctx [%rsi]) -> i64 [%rax] system_v { ebb0(v0: i32, v1: i64): v2 = iconst.i32 0 v3 = iconst.i64 0 diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index 8d18baa525..fb822e1a6f 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -93,7 +93,7 @@ ebb0(v0: i32): ; The same value used as indirect callee and argument. function %doubleuse_icall1(i32) { - sig0 = (i32) native + sig0 = (i32) system_v ebb0(v0: i32): ; not:copy call_indirect sig0, v0(v0) @@ -102,7 +102,7 @@ ebb0(v0: i32): ; The same value used as indirect callee and two arguments. function %doubleuse_icall2(i32) { - sig0 = (i32, i32) native + sig0 = (i32, i32) system_v ebb0(v0: i32): ; check: $(c=$V) = copy v0 call_indirect sig0, v0(v0, v0) diff --git a/cranelift/filetests/verifier/defs_dominates_uses.cton b/cranelift/filetests/verifier/defs_dominates_uses.cton index 1bc3819726..ba3dc98eb4 100644 --- a/cranelift/filetests/verifier/defs_dominates_uses.cton +++ b/cranelift/filetests/verifier/defs_dominates_uses.cton @@ -2,14 +2,14 @@ test verifier ; Test verification that uses properly dominate defs. -function %non_dominating(i32) -> i32 native { +function %non_dominating(i32) -> i32 system_v { ebb0(v0: i32): v1 = iadd.i32 v2, v0 ; error: uses value from non-dominating v2 = iadd.i32 v1, v0 return v2 } -function %inst_uses_its_own_values(i32) -> i32 native { +function %inst_uses_its_own_values(i32) -> i32 system_v { ebb0(v0: i32): v1 = iadd.i32 v1, v0 ; error: uses value from itself return v1 diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index f8fe767970..08fa28cf5c 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -343,10 +343,11 @@ impl fmt::Display for ExtFuncData { /// determined by a `(TargetIsa, CallConv)` tuple. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum CallConv { - /// The C calling convention. + /// The System V-style calling convention. /// - /// This is the native calling convention that a C compiler would use on the platform. - Native, + /// This is the System V-style calling convention that a C compiler would + /// use on many platforms. + SystemV, /// A JIT-compiled WebAssembly function in the SpiderMonkey VM. SpiderWASM, @@ -356,7 +357,7 @@ impl fmt::Display for CallConv { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { use self::CallConv::*; f.write_str(match *self { - Native => "native", + SystemV => "system_v", SpiderWASM => "spiderwasm", }) } @@ -368,7 +369,7 @@ impl FromStr for CallConv { fn from_str(s: &str) -> Result { use self::CallConv::*; match s { - "native" => Ok(Native), + "system_v" => Ok(SystemV), "spiderwasm" => Ok(SpiderWASM), _ => Err(()), } @@ -410,7 +411,7 @@ mod tests { #[test] fn call_conv() { - for &cc in &[CallConv::Native, CallConv::SpiderWASM] { + for &cc in &[CallConv::SystemV, CallConv::SpiderWASM] { assert_eq!(Ok(cc), cc.to_string().parse()) } } diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index e52bb4c96f..d122e73360 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -86,7 +86,7 @@ impl Function { /// Clear all data structures in this function. pub fn clear(&mut self) { - self.signature.clear(ir::CallConv::Native); + self.signature.clear(ir::CallConv::SystemV); self.stack_slots.clear(); self.global_vars.clear(); self.heaps.clear(); @@ -99,9 +99,9 @@ impl Function { self.srclocs.clear(); } - /// Create a new empty, anonymous function with a native calling convention. + /// Create a new empty, anonymous function with a SystemV calling convention. pub fn new() -> Self { - Self::with_name_signature(ExternalName::default(), Signature::new(CallConv::Native)) + Self::with_name_signature(ExternalName::default(), Signature::new(CallConv::SystemV)) } /// Creates a jump table in the function, to be used by `br_table` instructions. diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 0a14566b50..e2c79e0377 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -171,7 +171,7 @@ pub fn callee_saved_registers(flags: &shared_settings::Flags) -> &'static [RU] { pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { match func.signature.call_conv { - ir::CallConv::Native => native_prologue_epilogue(func, isa), + ir::CallConv::SystemV => system_v_prologue_epilogue(func, isa), ir::CallConv::SpiderWASM => spiderwasm_prologue_epilogue(func, isa), } } @@ -194,7 +194,7 @@ pub fn spiderwasm_prologue_epilogue( } /// Insert a System V-compatible prologue and epilogue. -pub fn native_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { +pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { // The original 32-bit x86 ELF ABI had a 4-byte aligned stack pointer, but // newer versions use a 16-byte aligned stack pointer. let stack_align = 16; @@ -242,17 +242,17 @@ pub fn native_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> res // Set up the cursor and insert the prologue let entry_ebb = func.layout.entry_block().expect("missing entry block"); let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_ebb); - insert_native_prologue(&mut pos, local_stack_size, csr_type, csrs); + insert_system_v_prologue(&mut pos, local_stack_size, csr_type, csrs); // Reset the cursor and insert the epilogue let mut pos = pos.at_position(CursorPosition::Nowhere); - insert_native_epilogues(&mut pos, local_stack_size, csr_type, csrs); + insert_system_v_epilogues(&mut pos, local_stack_size, csr_type, csrs); Ok(()) } /// Insert the prologue for a given function. -fn insert_native_prologue( +fn insert_system_v_prologue( pos: &mut EncCursor, stack_size: i64, csr_type: ir::types::Type, @@ -286,7 +286,7 @@ fn insert_native_prologue( } /// Find all `return` instructions and insert epilogues before them. -fn insert_native_epilogues( +fn insert_system_v_epilogues( pos: &mut EncCursor, stack_size: i64, csr_type: ir::types::Type, @@ -296,14 +296,14 @@ fn insert_native_epilogues( pos.goto_last_inst(ebb); if let Some(inst) = pos.current_inst() { if pos.func.dfg[inst].opcode().is_return() { - insert_native_epilogue(inst, stack_size, pos, csr_type, csrs); + insert_system_v_epilogue(inst, stack_size, pos, csr_type, csrs); } } } } /// Insert an epilogue given a specific `return` instruction. -fn insert_native_epilogue( +fn insert_system_v_epilogue( inst: ir::Inst, stack_size: i64, pos: &mut EncCursor, diff --git a/lib/cretonne/src/legalizer/libcall.rs b/lib/cretonne/src/legalizer/libcall.rs index 156a05b813..c4d063bd3c 100644 --- a/lib/cretonne/src/legalizer/libcall.rs +++ b/lib/cretonne/src/legalizer/libcall.rs @@ -44,8 +44,8 @@ fn find_funcref(libcall: ir::LibCall, func: &ir::Function) -> Option ir::FuncRef { - // Start with a native calling convention. We'll give the ISA a chance to change it. - let mut sig = ir::Signature::new(ir::CallConv::Native); + // Start with a system_v calling convention. We'll give the ISA a chance to change it. + let mut sig = ir::Signature::new(ir::CallConv::SystemV); for &v in func.dfg.inst_args(inst) { sig.params.push(ir::AbiParam::new(func.dfg.value_type(v))); } diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 1081a87d77..f6e7d90ff2 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -465,34 +465,34 @@ mod tests { #[test] fn basic() { let mut f = Function::new(); - assert_eq!(f.to_string(), "function u0:0() native {\n}\n"); + assert_eq!(f.to_string(), "function u0:0() system_v {\n}\n"); f.name = ExternalName::testcase("foo"); - assert_eq!(f.to_string(), "function %foo() native {\n}\n"); + assert_eq!(f.to_string(), "function %foo() system_v {\n}\n"); f.create_stack_slot(StackSlotData::new(StackSlotKind::ExplicitSlot, 4)); assert_eq!( f.to_string(), - "function %foo() native {\n ss0 = explicit_slot 4\n}\n" + "function %foo() system_v {\n ss0 = explicit_slot 4\n}\n" ); let ebb = f.dfg.make_ebb(); f.layout.append_ebb(ebb); assert_eq!( f.to_string(), - "function %foo() native {\n ss0 = explicit_slot 4\n\nebb0:\n}\n" + "function %foo() system_v {\n ss0 = explicit_slot 4\n\nebb0:\n}\n" ); f.dfg.append_ebb_param(ebb, types::I8); assert_eq!( f.to_string(), - "function %foo() native {\n ss0 = explicit_slot 4\n\nebb0(v0: i8):\n}\n" + "function %foo() system_v {\n ss0 = explicit_slot 4\n\nebb0(v0: i8):\n}\n" ); f.dfg.append_ebb_param(ebb, types::F32.by(4).unwrap()); assert_eq!( f.to_string(), - "function %foo() native {\n ss0 = explicit_slot 4\n\nebb0(v0: i8, v1: f32x4):\n}\n" + "function %foo() system_v {\n ss0 = explicit_slot 4\n\nebb0(v0: i8, v1: f32x4):\n}\n" ); } } diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 2706846124..bd01ce1893 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -600,7 +600,7 @@ mod tests { use Variable; fn sample_function(lazy_seal: bool) { - let mut sig = Signature::new(CallConv::Native); + let mut sig = Signature::new(CallConv::SystemV); sig.returns.push(AbiParam::new(I32)); sig.params.push(AbiParam::new(I32)); diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index c7defce3bb..628195428a 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -43,7 +43,7 @@ //! use cretonne::verifier::verify_function; //! //! fn main() { -//! let mut sig = Signature::new(CallConv::Native); +//! let mut sig = Signature::new(CallConv::SystemV); //! sig.returns.push(AbiParam::new(I32)); //! sig.params.push(AbiParam::new(I32)); //! let mut fn_builder_ctx = FunctionBuilderContext::::new(); diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index f0fa723594..c9da49d9ee 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -187,7 +187,7 @@ impl<'a> Context<'a> { fn add_sig(&mut self, sig: SigRef, data: Signature, loc: &Location) -> Result<()> { while self.function.dfg.signatures.next_key().index() <= sig.index() { self.function.import_signature( - Signature::new(CallConv::Native), + Signature::new(CallConv::SystemV), ); } self.function.dfg.signatures[sig] = data; @@ -871,8 +871,8 @@ impl<'a> Parser<'a> { // signature ::= * "(" [paramlist] ")" ["->" retlist] [callconv] // fn parse_signature(&mut self, unique_isa: Option<&TargetIsa>) -> Result { - // Calling convention defaults to `native`, but can be changed. - let mut sig = Signature::new(CallConv::Native); + // Calling convention defaults to `system_v`, but can be changed. + let mut sig = Signature::new(CallConv::SystemV); self.match_token( Token::LPar, @@ -2429,7 +2429,7 @@ mod tests { #[test] fn aliases() { let (func, details) = Parser::new( - "function %qux() native { + "function %qux() system_v { ebb0: v4 = iconst.i8 6 v3 -> v4 @@ -2453,10 +2453,10 @@ mod tests { #[test] fn signature() { - let sig = Parser::new("()native").parse_signature(None).unwrap(); + let sig = Parser::new("()system_v").parse_signature(None).unwrap(); assert_eq!(sig.params.len(), 0); assert_eq!(sig.returns.len(), 0); - assert_eq!(sig.call_conv, CallConv::Native); + assert_eq!(sig.call_conv, CallConv::SystemV); let sig2 = Parser::new("(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 spiderwasm") .parse_signature(None) @@ -2470,7 +2470,7 @@ mod tests { // Old-style signature without a calling convention. assert_eq!( Parser::new("()").parse_signature(None).unwrap().to_string(), - "() native" + "() system_v" ); assert_eq!( Parser::new("() notacc") @@ -2507,7 +2507,7 @@ mod tests { #[test] fn stack_slot_decl() { let (func, _) = Parser::new( - "function %foo() native { + "function %foo() system_v { ss3 = incoming_arg 13 ss1 = spill_slot 1 }", @@ -2530,7 +2530,7 @@ mod tests { // Catch duplicate definitions. assert_eq!( Parser::new( - "function %bar() native { + "function %bar() system_v { ss1 = spill_slot 13 ss1 = spill_slot 1 }", @@ -2544,7 +2544,7 @@ mod tests { #[test] fn ebb_header() { let (func, _) = Parser::new( - "function %ebbs() native { + "function %ebbs() system_v { ebb0: ebb4(v3: i32): }", @@ -2567,7 +2567,7 @@ mod tests { fn comments() { let (func, Details { comments, .. }) = Parser::new( "; before - function %comment() native { ; decl + function %comment() system_v { ; decl ss10 = outgoing_arg 13 ; stackslot. ; Still stackslot. jt10 = jump_table ebb0 @@ -2610,7 +2610,7 @@ mod tests { test verify set enable_float=false ; still preamble - function %comment() native {}", + function %comment() system_v {}", ).unwrap(); assert_eq!(tf.commands.len(), 2); assert_eq!(tf.commands[0].command, "cfg"); @@ -2635,7 +2635,7 @@ mod tests { assert!( parse_test( "isa - function %foo() native {}", + function %foo() system_v {}", ).is_err() ); @@ -2643,14 +2643,14 @@ mod tests { parse_test( "isa riscv set enable_float=false - function %foo() native {}", + function %foo() system_v {}", ).is_err() ); match parse_test( "set enable_float=false isa riscv - function %foo() native {}", + function %foo() system_v {}", ).unwrap() .isa_spec { IsaSpec::None(_) => panic!("Expected some ISA"), @@ -2665,7 +2665,7 @@ mod tests { fn user_function_name() { // Valid characters in the name: let func = Parser::new( - "function u1:2() native { + "function u1:2() system_v { ebb0: trap int_divz }", @@ -2676,7 +2676,7 @@ mod tests { // Invalid characters in the name: let mut parser = Parser::new( - "function u123:abc() native { + "function u123:abc() system_v { ebb0: trap stk_ovf }", @@ -2685,7 +2685,7 @@ mod tests { // Incomplete function names should not be valid: let mut parser = Parser::new( - "function u() native { + "function u() system_v { ebb0: trap int_ovf }", @@ -2693,7 +2693,7 @@ mod tests { assert!(parser.parse_function(None).is_err()); let mut parser = Parser::new( - "function u0() native { + "function u0() system_v { ebb0: trap int_ovf }", @@ -2701,7 +2701,7 @@ mod tests { assert!(parser.parse_function(None).is_err()); let mut parser = Parser::new( - "function u0:() native { + "function u0:() system_v { ebb0: trap int_ovf }", diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 1bf7c9441c..05f655ba82 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -36,7 +36,7 @@ pub fn parse_function_signatures( ref params, ref returns, }) => { - let mut sig = Signature::new(CallConv::Native); + let mut sig = Signature::new(CallConv::SystemV); sig.params.extend(params.iter().map(|ty| { let cret_arg: cretonne::ir::Type = type_to_type(ty).expect( "only numeric types are supported in function signatures", From 72b7a4b3efdfcf2cf87286347847447418bea80e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 30 Mar 2018 11:33:44 -0700 Subject: [PATCH 1671/3084] Add `iter()` and `values()` functions to `PrimaryMap` and `EntityMap`. `iter()` iterates over both keys and values, while `values()` iterates over just values. Also add `_mut()` versions. These replace the otherwise common idiom of iterating with `keys()` and using indexing to get the values, allowing for simpler code. --- lib/cretonne/src/entity/iter.rs | 109 +++++++++++++++++++++++++ lib/cretonne/src/entity/map.rs | 23 +++++- lib/cretonne/src/entity/mod.rs | 2 + lib/cretonne/src/entity/primary.rs | 100 +++++++++++++++++++++-- lib/cretonne/src/ir/stackslot.rs | 23 +++++- lib/cretonne/src/legalizer/boundary.rs | 6 +- lib/cretonne/src/legalizer/libcall.rs | 4 +- lib/cretonne/src/stack_layout.rs | 10 +-- lib/cretonne/src/write.rs | 28 +++---- lib/filetests/src/test_binemit.rs | 4 +- lib/frontend/src/frontend.rs | 8 +- 11 files changed, 274 insertions(+), 43 deletions(-) create mode 100644 lib/cretonne/src/entity/iter.rs diff --git a/lib/cretonne/src/entity/iter.rs b/lib/cretonne/src/entity/iter.rs new file mode 100644 index 0000000000..156021ee86 --- /dev/null +++ b/lib/cretonne/src/entity/iter.rs @@ -0,0 +1,109 @@ +//! A double-ended iterator over entity references and entities. + +use entity::EntityRef; +use std::marker::PhantomData; +use std::slice; + +/// Iterate over all keys in order. +pub struct Iter<'a, K: EntityRef, V> +where + V: 'a, +{ + pos: usize, + iter: slice::Iter<'a, V>, + unused: PhantomData, +} + +impl<'a, K: EntityRef, V> Iter<'a, K, V> { + /// Create an `Iter` iterator that visits the `PrimaryMap` keys and values + /// of `iter`. + pub fn new(key: K, iter: slice::Iter<'a, V>) -> Self { + Self { + pos: key.index(), + iter, + unused: PhantomData, + } + } +} + +impl<'a, K: EntityRef, V> Iterator for Iter<'a, K, V> { + type Item = (K, &'a V); + + fn next(&mut self) -> Option { + if let Some(next) = self.iter.next() { + let pos = self.pos; + self.pos += 1; + Some((K::new(pos), next)) + } else { + None + } + } + + fn size_hint(&self) -> (usize, Option) { + self.iter.size_hint() + } +} + +impl<'a, K: EntityRef, V> DoubleEndedIterator for Iter<'a, K, V> { + fn next_back(&mut self) -> Option { + if let Some(next_back) = self.iter.next_back() { + Some((K::new(self.pos), next_back)) + } else { + None + } + } +} + +impl<'a, K: EntityRef, V> ExactSizeIterator for Iter<'a, K, V> {} + +/// Iterate over all keys in order. +pub struct IterMut<'a, K: EntityRef, V> +where + V: 'a, +{ + pos: usize, + iter: slice::IterMut<'a, V>, + unused: PhantomData, +} + +impl<'a, K: EntityRef, V> IterMut<'a, K, V> { + /// Create an `IterMut` iterator that visits the `PrimaryMap` keys and values + /// of `iter`. + pub fn new(key: K, iter: slice::IterMut<'a, V>) -> Self { + Self { + pos: key.index(), + iter, + unused: PhantomData, + } + } +} + +impl<'a, K: EntityRef, V> Iterator for IterMut<'a, K, V> { + type Item = (K, &'a mut V); + + fn next(&mut self) -> Option { + if let Some(next) = self.iter.next() { + let pos = self.pos; + self.pos += 1; + Some((K::new(pos), next)) + } else { + None + } + } + + fn size_hint(&self) -> (usize, Option) { + self.iter.size_hint() + } +} + +impl<'a, K: EntityRef, V> DoubleEndedIterator for IterMut<'a, K, V> { + fn next_back(&mut self) -> Option { + if let Some(next_back) = self.iter.next_back() { + Some((K::new(self.pos), next_back)) + } else { + None + } + } +} + +impl<'a, K: EntityRef, V> ExactSizeIterator for IterMut<'a, K, V> {} diff --git a/lib/cretonne/src/entity/map.rs b/lib/cretonne/src/entity/map.rs index f251c2f831..3aac22b44f 100644 --- a/lib/cretonne/src/entity/map.rs +++ b/lib/cretonne/src/entity/map.rs @@ -1,9 +1,10 @@ //! Densely numbered entity references as mapping keys. -use entity::{EntityRef, Keys}; +use entity::{EntityRef, Keys, Iter, IterMut}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; use std::vec::Vec; +use std::slice; /// A mapping `K -> V` for densely indexed entity references. /// @@ -68,11 +69,31 @@ where self.elems.clear() } + /// Iterate over all the keys and values in this map. + pub fn iter(&self) -> Iter { + Iter::new(K::new(0), self.elems.iter()) + } + + /// Iterate over all the keys and values in this map, mutable edition. + pub fn iter_mut(&mut self) -> IterMut { + IterMut::new(K::new(0), self.elems.iter_mut()) + } + /// Iterate over all the keys in this map. pub fn keys(&self) -> Keys { Keys::new(self.elems.len()) } + /// Iterate over all the keys in this map. + pub fn values(&self) -> slice::Iter { + self.elems.iter() + } + + /// Iterate over all the keys in this map, mutable edition. + pub fn values_mut(&mut self) -> slice::IterMut { + self.elems.iter_mut() + } + /// Resize the map to have `n` entries by adding default entries as needed. pub fn resize(&mut self, n: usize) { self.elems.resize(n, self.default.clone()); diff --git a/lib/cretonne/src/entity/mod.rs b/lib/cretonne/src/entity/mod.rs index 6457e47f56..c4004f126d 100644 --- a/lib/cretonne/src/entity/mod.rs +++ b/lib/cretonne/src/entity/mod.rs @@ -30,6 +30,7 @@ //! `Vec`. mod keys; +mod iter; mod list; mod map; mod primary; @@ -37,6 +38,7 @@ mod sparse; mod set; pub use self::keys::Keys; +pub use self::iter::{Iter, IterMut}; pub use self::list::{EntityList, ListPool}; pub use self::map::EntityMap; pub use self::primary::PrimaryMap; diff --git a/lib/cretonne/src/entity/primary.rs b/lib/cretonne/src/entity/primary.rs index c06b818355..8671bdf809 100644 --- a/lib/cretonne/src/entity/primary.rs +++ b/lib/cretonne/src/entity/primary.rs @@ -1,8 +1,9 @@ //! Densely numbered entity references as mapping keys. -use entity::{EntityRef, Keys}; +use entity::{EntityRef, Keys, Iter, IterMut}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; use std::vec::Vec; +use std::slice; /// A primary mapping `K -> V` allocating dense entity references. /// @@ -59,6 +60,26 @@ where Keys::new(self.elems.len()) } + /// Iterate over all the values in this map. + pub fn values(&self) -> slice::Iter { + self.elems.iter() + } + + /// Iterate over all the values in this map, mutable edition. + pub fn values_mut(&mut self) -> slice::IterMut { + self.elems.iter_mut() + } + + /// Iterate over all the keys and values in this map. + pub fn iter(&self) -> Iter { + Iter::new(K::new(0), self.elems.iter()) + } + + /// Iterate over all the keys and values in this map, mutable edition. + pub fn iter_mut(&mut self) -> IterMut { + IterMut::new(K::new(0), self.elems.iter_mut()) + } + /// Remove all entries from this map. pub fn clear(&mut self) { self.elems.clear() @@ -133,13 +154,80 @@ mod tests { #[test] fn push() { let mut m = PrimaryMap::new(); - let k1: E = m.push(12); - let k2 = m.push(33); + let k0: E = m.push(12); + let k1 = m.push(33); - assert_eq!(m[k1], 12); - assert_eq!(m[k2], 33); + assert_eq!(m[k0], 12); + assert_eq!(m[k1], 33); let v: Vec = m.keys().collect(); - assert_eq!(v, [k1, k2]); + assert_eq!(v, [k0, k1]); + } + + #[test] + fn iter() { + let mut m: PrimaryMap = PrimaryMap::new(); + m.push(12); + m.push(33); + + let mut i = 0; + for (key, value) in m.iter() { + assert_eq!(key.index(), i); + match i { + 0 => assert_eq!(*value, 12), + 1 => assert_eq!(*value, 33), + _ => panic!(), + } + i += 1; + } + i = 0; + for (key_mut, value_mut) in m.iter_mut() { + assert_eq!(key_mut.index(), i); + match i { + 0 => assert_eq!(*value_mut, 12), + 1 => assert_eq!(*value_mut, 33), + _ => panic!(), + } + i += 1; + } + } + + #[test] + fn keys() { + let mut m: PrimaryMap = PrimaryMap::new(); + m.push(12); + m.push(33); + + let mut i = 0; + for key in m.keys() { + assert_eq!(key.index(), i); + i += 1; + } + } + + #[test] + fn values() { + let mut m: PrimaryMap = PrimaryMap::new(); + m.push(12); + m.push(33); + + let mut i = 0; + for value in m.values() { + match i { + 0 => assert_eq!(*value, 12), + 1 => assert_eq!(*value, 33), + _ => panic!(), + } + i += 1; + } + i = 0; + for value_mut in m.values_mut() { + match i { + 0 => assert_eq!(*value_mut, 12), + 1 => assert_eq!(*value_mut, 33), + _ => panic!(), + } + i += 1; + } } } diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index 5f73fb4b37..bc87e11a82 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -3,11 +3,12 @@ //! The `StackSlotData` struct keeps track of a single stack slot in a function. //! -use entity::{PrimaryMap, Keys}; +use entity::{PrimaryMap, Keys, Iter, IterMut}; use ir::{Type, StackSlot}; use packed_option::PackedOption; use std::cmp; use std::fmt; +use std::slice; use std::ops::{Index, IndexMut}; use std::str::FromStr; use std::vec::Vec; @@ -208,6 +209,26 @@ impl StackSlots { self.slots[ss].offset = Some(offset); } + /// Get an iterator over all the stack slot keys. + pub fn iter(&self) -> Iter { + self.slots.iter() + } + + /// Get an iterator over all the stack slot keys, mutable edition. + pub fn iter_mut(&mut self) -> IterMut { + self.slots.iter_mut() + } + + /// Get an iterator over all the stack slot records. + pub fn values(&self) -> slice::Iter { + self.slots.values() + } + + /// Get an iterator over all the stack slot records, mutable edition. + pub fn values_mut(&mut self) -> slice::IterMut { + self.slots.values_mut() + } + /// Get an iterator over all the stack slot keys. pub fn keys(&self) -> Keys { self.slots.keys() diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 0c5479650c..e1e4a17956 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -35,9 +35,9 @@ use std::vec::Vec; pub fn legalize_signatures(func: &mut Function, isa: &TargetIsa) { isa.legalize_signature(&mut func.signature, true); func.signature.compute_argument_bytes(); - for sig in func.dfg.signatures.keys() { - isa.legalize_signature(&mut func.dfg.signatures[sig], false); - func.dfg.signatures[sig].compute_argument_bytes(); + for sig_data in func.dfg.signatures.values_mut() { + isa.legalize_signature(sig_data, false); + sig_data.compute_argument_bytes(); } if let Some(entry) = func.layout.entry_block() { diff --git a/lib/cretonne/src/legalizer/libcall.rs b/lib/cretonne/src/legalizer/libcall.rs index c4d063bd3c..2bb0f1292c 100644 --- a/lib/cretonne/src/legalizer/libcall.rs +++ b/lib/cretonne/src/legalizer/libcall.rs @@ -29,8 +29,8 @@ pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function) -> bool { fn find_funcref(libcall: ir::LibCall, func: &ir::Function) -> Option { // We're assuming that all libcall function decls are at the end. // If we get this wrong, worst case we'll have duplicate libcall decls which is harmless. - for fref in func.dfg.ext_funcs.keys().rev() { - match func.dfg.ext_funcs[fref].name { + for (fref, func_data) in func.dfg.ext_funcs.iter().rev() { + match func_data.name { ir::ExternalName::LibCall(lc) => { if lc == libcall { return Some(fref); diff --git a/lib/cretonne/src/stack_layout.rs b/lib/cretonne/src/stack_layout.rs index f1cf35f150..0cf6e77527 100644 --- a/lib/cretonne/src/stack_layout.rs +++ b/lib/cretonne/src/stack_layout.rs @@ -39,9 +39,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result max_size { return Err(CtonError::ImplLimitExceeded); } @@ -72,9 +70,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result Result result::Result { let mut any = false; - for ss in func.stack_slots.keys() { + for (ss, slot) in func.stack_slots.iter() { any = true; - writeln!(w, " {} = {}", ss, func.stack_slots[ss])?; + writeln!(w, " {} = {}", ss, slot)?; } - for gv in func.global_vars.keys() { + for (gv, gv_data) in func.global_vars.iter() { any = true; - writeln!(w, " {} = {}", gv, func.global_vars[gv])?; + writeln!(w, " {} = {}", gv, gv_data)?; } - for heap in func.heaps.keys() { + for (heap, heap_data) in func.heaps.iter() { any = true; - writeln!(w, " {} = {}", heap, func.heaps[heap])?; + writeln!(w, " {} = {}", heap, heap_data)?; } // Write out all signatures before functions since function declarations can refer to // signatures. - for sig in func.dfg.signatures.keys() { + for (sig, sig_data) in func.dfg.signatures.iter() { any = true; - writeln!( - w, - " {} = {}", - sig, - func.dfg.signatures[sig].display(regs) - )?; + writeln!(w, " {} = {}", sig, sig_data.display(regs))?; } - for fnref in func.dfg.ext_funcs.keys() { + for (fnref, ext_func) in func.dfg.ext_funcs.iter() { any = true; - let ext_func = &func.dfg.ext_funcs[fnref]; if ext_func.signature != SigRef::reserved_value() { writeln!(w, " {} = {}", fnref, ext_func)?; } } - for jt in func.jump_tables.keys() { + for (jt, jt_data) in func.jump_tables.iter() { any = true; - writeln!(w, " {} = {}", jt, func.jump_tables[jt])?; + writeln!(w, " {} = {}", jt, jt_data)?; } Ok(any) diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index c22f1ed386..81e48f84fe 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -130,8 +130,8 @@ impl SubTest for TestBinEmit { // Fix the stack frame layout so we can test spill/fill encodings. let min_offset = func.stack_slots - .keys() - .map(|ss| func.stack_slots[ss].offset.unwrap()) + .values() + .map(|slot| slot.offset.unwrap()) .min(); func.stack_slots.frame_size = min_offset.map(|off| (-off) as u32); diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index bd01ce1893..37d2fe0206 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -439,14 +439,14 @@ where pub fn finalize(&mut self) { // Check that all the `Ebb`s are filled and sealed. debug_assert!( - self.func_ctx.ebbs.keys().all(|ebb| { - self.func_ctx.ebbs[ebb].pristine || self.func_ctx.ssa.is_sealed(ebb) + self.func_ctx.ebbs.iter().all(|(ebb, ebb_data)| { + ebb_data.pristine || self.func_ctx.ssa.is_sealed(ebb) }), "all blocks should be sealed before dropping a FunctionBuilder" ); debug_assert!( - self.func_ctx.ebbs.keys().all(|ebb| { - self.func_ctx.ebbs[ebb].pristine || self.func_ctx.ebbs[ebb].filled + self.func_ctx.ebbs.values().all(|ebb_data| { + ebb_data.pristine || ebb_data.filled }), "all blocks should be filled before dropping a FunctionBuilder" ); From d7c66a8ec183fc16773ca66b6a308aeb2e9828e4 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 27 Mar 2018 12:22:02 -0700 Subject: [PATCH 1672/3084] Don't run the DCE pass when optimization is disabled. The main purpose of the DCE pass is to clean up dead code left behind by the optimizer, so it's not valuable to run it when the optimizer isn't being run. --- lib/cretonne/src/context.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 2441f19d02..5324248142 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -105,7 +105,9 @@ impl Context { } self.compute_domtree(); self.eliminate_unreachable_code(isa)?; - self.dce(isa)?; + if isa.flags().opt_level() != OptLevel::Fastest { + self.dce(isa)?; + } self.regalloc(isa)?; self.prologue_epilogue(isa)?; self.relax_branches(isa) From 8d5fecd324fb935d75013bd11b72e8bdb63a3502 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 30 Mar 2018 13:17:15 -0700 Subject: [PATCH 1673/3084] Format with stable rustfmt-preview, then with rustfmt-0.9 again. --- cranelift/src/compile.rs | 2 +- cranelift/src/cton-util.rs | 6 +- cranelift/src/rsfilecheck.rs | 2 +- lib/cretonne/build.rs | 1 - lib/cretonne/src/abi.rs | 2 +- lib/cretonne/src/bforest/map.rs | 2 +- lib/cretonne/src/bforest/mod.rs | 4 +- lib/cretonne/src/bforest/node.rs | 2 +- lib/cretonne/src/bforest/path.rs | 6 +- lib/cretonne/src/bforest/set.rs | 2 +- lib/cretonne/src/binemit/memorysink.rs | 4 +- lib/cretonne/src/binemit/mod.rs | 2 +- lib/cretonne/src/binemit/relaxation.rs | 2 +- lib/cretonne/src/bitset.rs | 4 +- lib/cretonne/src/cfg_printer.rs | 2 +- lib/cretonne/src/constant_hash.rs | 1 - lib/cretonne/src/context.rs | 2 +- lib/cretonne/src/cursor.rs | 1 - lib/cretonne/src/dce.rs | 2 +- lib/cretonne/src/divconst_magic_numbers.rs | 4 +- lib/cretonne/src/dominator_tree.rs | 17 ++-- lib/cretonne/src/entity/list.rs | 2 +- lib/cretonne/src/entity/map.rs | 2 +- lib/cretonne/src/entity/mod.rs | 2 +- lib/cretonne/src/entity/primary.rs | 2 +- lib/cretonne/src/entity/sparse.rs | 2 +- lib/cretonne/src/flowgraph.rs | 4 +- lib/cretonne/src/ir/builder.rs | 9 +- lib/cretonne/src/ir/dfg.rs | 12 +-- lib/cretonne/src/ir/extfunc.rs | 4 +- lib/cretonne/src/ir/function.rs | 12 +-- lib/cretonne/src/ir/immediates.rs | 3 - lib/cretonne/src/ir/instructions.rs | 20 ++-- lib/cretonne/src/ir/layout.rs | 5 +- lib/cretonne/src/ir/mod.rs | 20 ++-- lib/cretonne/src/ir/progpoint.rs | 2 +- lib/cretonne/src/ir/stackslot.rs | 4 +- lib/cretonne/src/ir/types.rs | 2 +- lib/cretonne/src/isa/arm32/abi.rs | 2 +- lib/cretonne/src/isa/arm32/binemit.rs | 2 +- lib/cretonne/src/isa/arm32/mod.rs | 4 +- lib/cretonne/src/isa/arm32/registers.rs | 2 +- lib/cretonne/src/isa/arm64/abi.rs | 2 +- lib/cretonne/src/isa/arm64/binemit.rs | 2 +- lib/cretonne/src/isa/arm64/mod.rs | 4 +- lib/cretonne/src/isa/constraints.rs | 3 +- lib/cretonne/src/isa/enc_tables.rs | 4 +- lib/cretonne/src/isa/encoding.rs | 2 +- lib/cretonne/src/isa/intel/abi.rs | 11 +-- lib/cretonne/src/isa/intel/binemit.rs | 8 +- lib/cretonne/src/isa/intel/enc_tables.rs | 6 +- lib/cretonne/src/isa/intel/mod.rs | 4 +- lib/cretonne/src/isa/mod.rs | 8 +- lib/cretonne/src/isa/riscv/abi.rs | 6 +- lib/cretonne/src/isa/riscv/binemit.rs | 4 +- lib/cretonne/src/isa/riscv/mod.rs | 6 +- lib/cretonne/src/isa/riscv/registers.rs | 2 +- lib/cretonne/src/isa/riscv/settings.rs | 12 +-- lib/cretonne/src/isa/stack.rs | 2 +- lib/cretonne/src/legalizer/boundary.rs | 4 +- lib/cretonne/src/legalizer/mod.rs | 1 - lib/cretonne/src/legalizer/split.rs | 3 +- lib/cretonne/src/lib.rs | 9 +- lib/cretonne/src/licm.rs | 3 +- lib/cretonne/src/loop_analysis.rs | 9 +- lib/cretonne/src/postopt.rs | 78 +++++++--------- lib/cretonne/src/preopt.rs | 11 +-- lib/cretonne/src/regalloc/affinity.rs | 2 +- lib/cretonne/src/regalloc/allocatable_set.rs | 2 +- lib/cretonne/src/regalloc/coalescing.rs | 5 +- lib/cretonne/src/regalloc/coloring.rs | 13 +-- lib/cretonne/src/regalloc/context.rs | 3 +- lib/cretonne/src/regalloc/diversion.rs | 4 +- .../src/regalloc/live_value_tracker.rs | 2 +- lib/cretonne/src/regalloc/liveness.rs | 7 +- lib/cretonne/src/regalloc/liverange.rs | 7 +- lib/cretonne/src/regalloc/pressure.rs | 6 +- lib/cretonne/src/regalloc/reload.rs | 6 +- lib/cretonne/src/regalloc/solver.rs | 7 +- lib/cretonne/src/regalloc/spilling.rs | 16 ++-- lib/cretonne/src/regalloc/virtregs.rs | 4 +- lib/cretonne/src/scoped_hash_map.rs | 2 +- lib/cretonne/src/settings.rs | 24 ++--- lib/cretonne/src/simple_gvn.rs | 2 +- lib/cretonne/src/stack_layout.rs | 6 +- lib/cretonne/src/timing.rs | 7 +- lib/cretonne/src/verifier/cssa.rs | 2 +- lib/cretonne/src/verifier/flags.rs | 2 +- lib/cretonne/src/verifier/liveness.rs | 2 +- lib/cretonne/src/verifier/mod.rs | 9 +- lib/cretonne/src/write.rs | 9 +- lib/filetests/src/concurrent.rs | 4 +- lib/filetests/src/runner.rs | 4 +- lib/filetests/src/runone.rs | 5 +- lib/filetests/src/subtest.rs | 4 +- lib/filetests/src/test_binemit.rs | 30 ++---- lib/filetests/src/test_cat.rs | 2 +- lib/filetests/src/test_compile.rs | 2 +- lib/filetests/src/test_dce.rs | 2 +- lib/filetests/src/test_domtree.rs | 8 +- lib/filetests/src/test_legalizer.rs | 2 +- lib/filetests/src/test_licm.rs | 2 +- lib/filetests/src/test_postopt.rs | 2 +- lib/filetests/src/test_preopt.rs | 2 +- lib/filetests/src/test_print_cfg.rs | 2 +- lib/filetests/src/test_regalloc.rs | 2 +- lib/filetests/src/test_simple_gvn.rs | 2 +- lib/filetests/src/test_verifier.rs | 2 +- lib/frontend/src/frontend.rs | 33 +++---- lib/frontend/src/lib.rs | 10 +- lib/frontend/src/ssa.rs | 7 +- lib/native/src/lib.rs | 4 +- lib/reader/src/isaspec.rs | 4 +- lib/reader/src/lexer.rs | 6 +- lib/reader/src/lib.rs | 10 +- lib/reader/src/parser.rs | 91 ++++++++----------- lib/reader/src/sourcemap.rs | 4 +- lib/wasm/src/code_translator.rs | 14 +-- lib/wasm/src/environ/dummy.rs | 4 +- lib/wasm/src/environ/mod.rs | 2 +- lib/wasm/src/environ/spec.rs | 4 +- lib/wasm/src/func_translator.rs | 6 +- lib/wasm/src/lib.rs | 20 ++-- lib/wasm/src/module_translator.rs | 10 +- lib/wasm/src/sections_translator.rs | 10 +- lib/wasm/src/state.rs | 2 +- lib/wasm/tests/wasm_testsuite.rs | 2 +- 127 files changed, 381 insertions(+), 479 deletions(-) diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index ccc049fa70..f2272494cf 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -7,7 +7,7 @@ use cretonne::settings::FlagsOrIsa; use cretonne::{binemit, ir}; use cretonne::print_errors::pretty_error; use std::path::Path; -use utils::{read_to_string, parse_sets_and_isa}; +use utils::{parse_sets_and_isa, read_to_string}; struct PrintRelocs { flag_print: bool, diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 4dd9865d3d..42887640df 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -1,15 +1,15 @@ extern crate cretonne; +extern crate cton_filetests; extern crate cton_reader; extern crate cton_wasm; -extern crate cton_filetests; extern crate docopt; +extern crate filecheck; #[macro_use] extern crate serde_derive; -extern crate filecheck; extern crate tempdir; extern crate term; -use cretonne::{VERSION, timing}; +use cretonne::{timing, VERSION}; use docopt::Docopt; use std::io::{self, Write}; use std::process; diff --git a/cranelift/src/rsfilecheck.rs b/cranelift/src/rsfilecheck.rs index 58b9cb9e53..f71571b513 100644 --- a/cranelift/src/rsfilecheck.rs +++ b/cranelift/src/rsfilecheck.rs @@ -4,7 +4,7 @@ use CommandResult; use utils::read_to_string; -use filecheck::{CheckerBuilder, Checker, NO_VARIABLES}; +use filecheck::{Checker, CheckerBuilder, NO_VARIABLES}; use std::io::{self, Read}; pub fn run(files: &[String], verbose: bool) -> CommandResult { diff --git a/lib/cretonne/build.rs b/lib/cretonne/build.rs index 69be14de94..e13e9349a5 100644 --- a/lib/cretonne/build.rs +++ b/lib/cretonne/build.rs @@ -18,7 +18,6 @@ // The build script expects to be run from the directory where this build.rs file lives. The // current directory is used to find the sources. - use std::env; use std::process; diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index b69821e657..2ba3f7b7a1 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -3,7 +3,7 @@ //! This module provides functions and data structures that are useful for implementing the //! `TargetIsa::legalize_signature()` method. -use ir::{ArgumentLoc, AbiParam, ArgumentExtension, Type}; +use ir::{AbiParam, ArgumentExtension, ArgumentLoc, Type}; use std::cmp::Ordering; use std::vec::Vec; diff --git a/lib/cretonne/src/bforest/map.rs b/lib/cretonne/src/bforest/map.rs index edf301aa4c..23b4625555 100644 --- a/lib/cretonne/src/bforest/map.rs +++ b/lib/cretonne/src/bforest/map.rs @@ -2,7 +2,7 @@ use packed_option::PackedOption; use std::marker::PhantomData; -use super::{INNER_SIZE, Comparator, Forest, NodePool, Node, NodeData, Path}; +use super::{Comparator, Forest, Node, NodeData, NodePool, Path, INNER_SIZE}; /// Tag type defining forest types for a map. struct MapTypes(PhantomData<(K, V, C)>); diff --git a/lib/cretonne/src/bforest/mod.rs b/lib/cretonne/src/bforest/mod.rs index ff1b94cc0f..bc22cf88c0 100644 --- a/lib/cretonne/src/bforest/mod.rs +++ b/lib/cretonne/src/bforest/mod.rs @@ -22,8 +22,8 @@ mod path; mod pool; mod set; -pub use self::map::{MapForest, Map, MapCursor, MapIter}; -pub use self::set::{SetForest, Set, SetCursor, SetIter}; +pub use self::map::{Map, MapCursor, MapForest, MapIter}; +pub use self::set::{Set, SetCursor, SetForest, SetIter}; use self::node::NodeData; use self::path::Path; diff --git a/lib/cretonne/src/bforest/node.rs b/lib/cretonne/src/bforest/node.rs index d5c46f54bd..61fb6e994d 100644 --- a/lib/cretonne/src/bforest/node.rs +++ b/lib/cretonne/src/bforest/node.rs @@ -2,7 +2,7 @@ use std::borrow::{Borrow, BorrowMut}; use std::fmt; -use super::{Forest, Node, INNER_SIZE, SetValue, slice_insert, slice_shift}; +use super::{slice_insert, slice_shift, Forest, Node, SetValue, INNER_SIZE}; /// B+-tree node. /// diff --git a/lib/cretonne/src/bforest/path.rs b/lib/cretonne/src/bforest/path.rs index 34f165d86c..a2d15173a3 100644 --- a/lib/cretonne/src/bforest/path.rs +++ b/lib/cretonne/src/bforest/path.rs @@ -2,7 +2,7 @@ use std::borrow::Borrow; use std::marker::PhantomData; -use super::{Forest, Node, NodeData, NodePool, MAX_PATH, Comparator, slice_insert, slice_shift}; +use super::{slice_insert, slice_shift, Comparator, Forest, Node, NodeData, NodePool, MAX_PATH}; use super::node::Removed; #[cfg(test)] @@ -414,7 +414,6 @@ impl Path { } } - /// Given that the current leaf node is in an unhealthy (underflowed or even empty) status, /// balance it with sibling nodes. /// @@ -529,7 +528,6 @@ impl Path { // current entry[level] was one off the end of the node, it will now point at a proper // entry. debug_assert!(usize::from(self.entry[level]) < pool[self.node[level]].entries()); - } else if usize::from(self.entry[level]) >= pool[self.node[level]].entries() { // There's no right sibling at this level, so the node can't be rebalanced. // Check if we are in an off-the-end position. @@ -703,7 +701,7 @@ impl fmt::Display for Path { mod test { use std::cmp::Ordering; use super::*; - use super::super::{Forest, NodePool, NodeData}; + use super::super::{Forest, NodeData, NodePool}; struct TC(); diff --git a/lib/cretonne/src/bforest/set.rs b/lib/cretonne/src/bforest/set.rs index dd7f63ca17..374c26ad2b 100644 --- a/lib/cretonne/src/bforest/set.rs +++ b/lib/cretonne/src/bforest/set.rs @@ -2,7 +2,7 @@ use packed_option::PackedOption; use std::marker::PhantomData; -use super::{INNER_SIZE, Comparator, Forest, NodePool, Node, NodeData, Path, SetValue}; +use super::{Comparator, Forest, Node, NodeData, NodePool, Path, SetValue, INNER_SIZE}; /// Tag type defining forest types for a set. struct SetTypes(PhantomData<(K, C)>); diff --git a/lib/cretonne/src/binemit/memorysink.rs b/lib/cretonne/src/binemit/memorysink.rs index d4b07319ab..5a9fa8a298 100644 --- a/lib/cretonne/src/binemit/memorysink.rs +++ b/lib/cretonne/src/binemit/memorysink.rs @@ -14,8 +14,8 @@ //! relocations to a `RelocSink` trait object. Relocations are less frequent than the //! `CodeSink::put*` methods, so the performance impact of the virtual callbacks is less severe. -use ir::{ExternalName, JumpTable, TrapCode, SourceLoc}; -use super::{CodeSink, CodeOffset, Reloc, Addend}; +use ir::{ExternalName, JumpTable, SourceLoc, TrapCode}; +use super::{Addend, CodeOffset, CodeSink, Reloc}; use std::ptr::write_unaligned; /// A `CodeSink` that writes binary machine code directly into memory. diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 6d2099ceba..20ff429756 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -10,7 +10,7 @@ pub use regalloc::RegDiversions; pub use self::relaxation::relax_branches; pub use self::memorysink::{MemoryCodeSink, RelocSink, TrapSink}; -use ir::{ExternalName, JumpTable, Function, Inst, TrapCode, SourceLoc}; +use ir::{ExternalName, Function, Inst, JumpTable, SourceLoc, TrapCode}; use std::fmt; /// Offset in bytes from the beginning of the function. diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index 24a78ceebd..8123c5b92f 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -30,7 +30,7 @@ use binemit::CodeOffset; use cursor::{Cursor, FuncCursor}; use ir::{Function, InstructionData, Opcode}; -use isa::{TargetIsa, EncInfo}; +use isa::{EncInfo, TargetIsa}; use iterators::IteratorExtras; use result::CtonError; diff --git a/lib/cretonne/src/bitset.rs b/lib/cretonne/src/bitset.rs index f212790be0..f0a7271957 100644 --- a/lib/cretonne/src/bitset.rs +++ b/lib/cretonne/src/bitset.rs @@ -6,8 +6,8 @@ //! If you would like to add support for larger bitsets in the future, you need to change the trait //! bound Into and the u32 in the implementation of `max_bits()`. use std::mem::size_of; -use std::ops::{Shl, BitOr, Sub, Add}; -use std::convert::{Into, From}; +use std::ops::{Add, BitOr, Shl, Sub}; +use std::convert::{From, Into}; /// A small bitset built on a single primitive integer type #[derive(Clone, Copy, Debug, PartialEq, Eq)] diff --git a/lib/cretonne/src/cfg_printer.rs b/lib/cretonne/src/cfg_printer.rs index 736d12f0fc..46b8754e3e 100644 --- a/lib/cretonne/src/cfg_printer.rs +++ b/lib/cretonne/src/cfg_printer.rs @@ -1,6 +1,6 @@ //! The `CFGPrinter` utility. -use std::fmt::{Result, Write, Display, Formatter}; +use std::fmt::{Display, Formatter, Result, Write}; use flowgraph::ControlFlowGraph; use ir::Function; diff --git a/lib/cretonne/src/constant_hash.rs b/lib/cretonne/src/constant_hash.rs index d049d8e752..0dd2895ad3 100644 --- a/lib/cretonne/src/constant_hash.rs +++ b/lib/cretonne/src/constant_hash.rs @@ -18,7 +18,6 @@ pub trait Table { fn key(&self, idx: usize) -> Option; } - /// Look for `key` in `table`. /// /// The provided `hash` value must have been computed from `key` using the same hash function that diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index 5324248142..c7841f9366 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -9,7 +9,7 @@ //! contexts concurrently. Typically, you would have one context per compilation thread and only a //! single ISA instance. -use binemit::{CodeOffset, relax_branches, MemoryCodeSink, RelocSink, TrapSink}; +use binemit::{relax_branches, CodeOffset, MemoryCodeSink, RelocSink, TrapSink}; use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir::Function; diff --git a/lib/cretonne/src/cursor.rs b/lib/cretonne/src/cursor.rs index 42f5e1e741..082889a693 100644 --- a/lib/cretonne/src/cursor.rs +++ b/lib/cretonne/src/cursor.rs @@ -637,7 +637,6 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut FuncCursor<'f> { } } - /// Encoding cursor. /// /// An `EncCursor` can be used to insert instructions that are immediately assigned an encoding. diff --git a/lib/cretonne/src/dce.rs b/lib/cretonne/src/dce.rs index 6e34bad74f..b78d260ecb 100644 --- a/lib/cretonne/src/dce.rs +++ b/lib/cretonne/src/dce.rs @@ -6,7 +6,7 @@ use cursor::{Cursor, FuncCursor}; use dominator_tree::DominatorTree; use entity::EntityRef; -use ir::{Function, Inst, Opcode, DataFlowGraph}; +use ir::{DataFlowGraph, Function, Inst, Opcode}; use ir::instructions::InstructionData; use timing; use std::vec::Vec; diff --git a/lib/cretonne/src/divconst_magic_numbers.rs b/lib/cretonne/src/divconst_magic_numbers.rs index 01b5066b28..80a59aed8b 100644 --- a/lib/cretonne/src/divconst_magic_numbers.rs +++ b/lib/cretonne/src/divconst_magic_numbers.rs @@ -220,8 +220,8 @@ pub fn magicS64(d: i64) -> MS64 { #[cfg(test)] mod tests { - use super::{magicU32, magicU64, magicS32, magicS64}; - use super::{MU32, MU64, MS32, MS64}; + use super::{magicS32, magicS64, magicU32, magicU64}; + use super::{MS32, MS64, MU32, MU64}; fn mkMU32(mulBy: u32, doAdd: bool, shiftBy: i32) -> MU32 { MU32 { diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 57f55781a3..574697c1ba 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -1,8 +1,8 @@ //! A Dominator Tree represented as mappings of Ebbs to their immediate dominator. use entity::EntityMap; -use flowgraph::{ControlFlowGraph, BasicBlock}; -use ir::{Ebb, Inst, Value, Function, Layout, ProgramOrder, ExpandedProgramPoint}; +use flowgraph::{BasicBlock, ControlFlowGraph}; +use ir::{Ebb, ExpandedProgramPoint, Function, Inst, Layout, ProgramOrder, Value}; use ir::instructions::BranchInfo; use packed_option::PackedOption; use std::cmp; @@ -144,12 +144,12 @@ impl DominatorTree { { let (mut ebb_b, mut inst_b) = match b.into() { ExpandedProgramPoint::Ebb(ebb) => (ebb, None), - ExpandedProgramPoint::Inst(inst) => { - ( - layout.inst_ebb(inst).expect("Instruction not in layout."), - Some(inst), - ) - } + ExpandedProgramPoint::Inst(inst) => ( + layout.inst_ebb(inst).expect( + "Instruction not in layout.", + ), + Some(inst), + ), }; let rpo_a = self.nodes[a].rpo_number; @@ -460,7 +460,6 @@ impl DominatorTree { rpo_number: new_ebb_rpo, idom: Some(split_jump_inst).into(), }; - } // Insert new_ebb just after ebb in the RPO. This function checks diff --git a/lib/cretonne/src/entity/list.rs b/lib/cretonne/src/entity/list.rs index ad03dc6c85..2c743e169b 100644 --- a/lib/cretonne/src/entity/list.rs +++ b/lib/cretonne/src/entity/list.rs @@ -480,7 +480,7 @@ impl EntityList { #[cfg(test)] mod tests { use super::*; - use super::{sclass_size, sclass_for_length}; + use super::{sclass_for_length, sclass_size}; use ir::Inst; use entity::EntityRef; diff --git a/lib/cretonne/src/entity/map.rs b/lib/cretonne/src/entity/map.rs index 3aac22b44f..88b355585b 100644 --- a/lib/cretonne/src/entity/map.rs +++ b/lib/cretonne/src/entity/map.rs @@ -1,6 +1,6 @@ //! Densely numbered entity references as mapping keys. -use entity::{EntityRef, Keys, Iter, IterMut}; +use entity::{EntityRef, Iter, IterMut, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; use std::vec::Vec; diff --git a/lib/cretonne/src/entity/mod.rs b/lib/cretonne/src/entity/mod.rs index c4004f126d..9aeab1449d 100644 --- a/lib/cretonne/src/entity/mod.rs +++ b/lib/cretonne/src/entity/mod.rs @@ -43,7 +43,7 @@ pub use self::list::{EntityList, ListPool}; pub use self::map::EntityMap; pub use self::primary::PrimaryMap; pub use self::set::EntitySet; -pub use self::sparse::{SparseSet, SparseMap, SparseMapValue}; +pub use self::sparse::{SparseMap, SparseMapValue, SparseSet}; /// A type wrapping a small integer index should implement `EntityRef` so it can be used as the key /// of an `EntityMap` or `SparseMap`. diff --git a/lib/cretonne/src/entity/primary.rs b/lib/cretonne/src/entity/primary.rs index 8671bdf809..a442928d77 100644 --- a/lib/cretonne/src/entity/primary.rs +++ b/lib/cretonne/src/entity/primary.rs @@ -1,5 +1,5 @@ //! Densely numbered entity references as mapping keys. -use entity::{EntityRef, Keys, Iter, IterMut}; +use entity::{EntityRef, Iter, IterMut, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; use std::vec::Vec; diff --git a/lib/cretonne/src/entity/sparse.rs b/lib/cretonne/src/entity/sparse.rs index 488fd55393..6894c89498 100644 --- a/lib/cretonne/src/entity/sparse.rs +++ b/lib/cretonne/src/entity/sparse.rs @@ -7,7 +7,7 @@ //! > Briggs, Torczon, *An efficient representation for sparse sets*, //! ACM Letters on Programming Languages and Systems, Volume 2, Issue 1-4, March-Dec. 1993. -use entity::{EntityRef, EntityMap}; +use entity::{EntityMap, EntityRef}; use std::mem; use std::slice; use std::u32; diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index dce60162a4..3ed295d6f0 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -24,7 +24,7 @@ //! and `(Ebb0, jmp Ebb2)` respectively. use bforest; -use ir::{Function, Inst, Ebb}; +use ir::{Ebb, Function, Inst}; use ir::instructions::BranchInfo; use entity::EntityMap; use std::mem; @@ -203,7 +203,7 @@ pub type SuccIter<'a> = bforest::SetIter<'a, Ebb, ()>; mod tests { use super::*; use cursor::{Cursor, FuncCursor}; - use ir::{Function, InstBuilder, types}; + use ir::{types, Function, InstBuilder}; use std::vec::Vec; #[test] diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 7e1480186d..6308f3e1bd 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -5,8 +5,8 @@ use ir; use ir::types; -use ir::{InstructionData, DataFlowGraph}; -use ir::{Opcode, Type, Inst, Value}; +use ir::{DataFlowGraph, InstructionData}; +use ir::{Inst, Opcode, Type, Value}; use isa; /// Base trait for instruction builders. @@ -145,8 +145,9 @@ where } impl<'f, IIB, Array> InstBuilderBase<'f> for InsertReuseBuilder<'f, IIB, Array> - where IIB: InstInserterBase<'f>, - Array: AsRef<[Option]> +where + IIB: InstInserterBase<'f>, + Array: AsRef<[Option]>, { fn data_flow_graph(&self) -> &DataFlowGraph { self.inserter.data_flow_graph() diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index d8cb9b2eb1..48faf3b5ac 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -1,13 +1,13 @@ //! Data flow graph tracking Instructions, Values, and EBBs. -use entity::{PrimaryMap, EntityMap}; -use isa::{TargetIsa, Encoding, Legalize}; +use entity::{EntityMap, PrimaryMap}; +use isa::{Encoding, Legalize, TargetIsa}; use ir; use ir::builder::ReplaceBuilder; use ir::extfunc::ExtFuncData; -use ir::instructions::{InstructionData, CallInfo, BranchInfo}; +use ir::instructions::{BranchInfo, CallInfo, InstructionData}; use ir::types; -use ir::{Ebb, Inst, Value, Type, SigRef, Signature, FuncRef, ValueList, ValueListPool}; +use ir::{Ebb, FuncRef, Inst, SigRef, Signature, Type, Value, ValueList, ValueListPool}; use packed_option::ReservedValue; use write::write_operands; use std::fmt; @@ -775,7 +775,6 @@ impl DataFlowGraph { } } - /// Append an existing value to `ebb`'s parameters. /// /// The appended value can't already be attached to something else. @@ -863,7 +862,6 @@ impl<'a> fmt::Display for DisplayInst<'a> { write!(f, " = ")?; } - let typevar = dfg.ctrl_typevar(inst); if typevar.is_void() { write!(f, "{}", dfg[inst].opcode())?; @@ -1005,7 +1003,7 @@ mod tests { use super::*; use cursor::{Cursor, FuncCursor}; use ir::types; - use ir::{Function, Opcode, InstructionData, TrapCode}; + use ir::{Function, InstructionData, Opcode, TrapCode}; use std::string::ToString; #[test] diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 08fa28cf5c..45c44c952a 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -5,7 +5,7 @@ //! //! This module declares the data types used to represent external functions and call signatures. -use ir::{Type, ExternalName, SigRef, ArgumentLoc}; +use ir::{ArgumentLoc, ExternalName, SigRef, Type}; use isa::{RegInfo, RegUnit}; use std::cmp; use std::fmt; @@ -379,7 +379,7 @@ impl FromStr for CallConv { #[cfg(test)] mod tests { use super::*; - use ir::types::{I32, F32, B8}; + use ir::types::{B8, F32, I32}; use std::string::ToString; #[test] diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index d122e73360..5010e0caae 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -4,13 +4,13 @@ //! instructions. use binemit::CodeOffset; -use entity::{PrimaryMap, EntityMap}; +use entity::{EntityMap, PrimaryMap}; use ir; -use ir::{ExternalName, CallConv, Signature, DataFlowGraph, Layout}; -use ir::{InstEncodings, ValueLocations, JumpTables, StackSlots, EbbOffsets, SourceLocs}; -use ir::{Ebb, JumpTableData, JumpTable, StackSlotData, StackSlot, SigRef, ExtFuncData, FuncRef, - GlobalVarData, GlobalVar, HeapData, Heap}; -use isa::{TargetIsa, EncInfo, Legalize}; +use ir::{CallConv, DataFlowGraph, ExternalName, Layout, Signature}; +use ir::{EbbOffsets, InstEncodings, JumpTables, SourceLocs, StackSlots, ValueLocations}; +use ir::{Ebb, ExtFuncData, FuncRef, GlobalVar, GlobalVarData, Heap, HeapData, JumpTable, + JumpTableData, SigRef, StackSlot, StackSlotData}; +use isa::{EncInfo, Legalize, TargetIsa}; use std::fmt; use write::write_function; diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 6026cecf26..3dd09bd0bd 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -184,7 +184,6 @@ impl Display for Uimm32 { } else { write_hex(i64::from(self.0), f) } - } } @@ -249,7 +248,6 @@ impl Display for Offset32 { } else { write_hex(val, f) } - } } @@ -461,7 +459,6 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { None => return Err("Invalid character"), } } - } } diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index b25e5a069b..94293b517b 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -12,7 +12,7 @@ use std::ops::{Deref, DerefMut}; use std::vec::Vec; use ir; -use ir::{Value, Type, Ebb, JumpTable, SigRef, FuncRef}; +use ir::{Ebb, FuncRef, JumpTable, SigRef, Type, Value}; use ir::types; use isa; @@ -73,7 +73,7 @@ impl FromStr for Opcode { /// Parse an Opcode name from a string. fn from_str(s: &str) -> Result { - use constant_hash::{Table, simple_hash, probe}; + use constant_hash::{probe, simple_hash, Table}; impl<'a> Table<&'a str> for [Option] { fn len(&self) -> usize { @@ -512,16 +512,12 @@ impl OperandConstraint { LaneOf => Bound(ctrl_type.lane_type()), AsBool => Bound(ctrl_type.as_bool()), HalfWidth => Bound(ctrl_type.half_width().expect("invalid type for half_width")), - DoubleWidth => { - Bound(ctrl_type.double_width().expect( - "invalid type for double_width", - )) - } - HalfVector => { - Bound(ctrl_type.half_vector().expect( - "invalid type for half_vector", - )) - } + DoubleWidth => Bound(ctrl_type.double_width().expect( + "invalid type for double_width", + )), + HalfVector => Bound(ctrl_type.half_vector().expect( + "invalid type for half_vector", + )), DoubleVector => Bound(ctrl_type.by(2).expect("invalid type for double_vector")), } } diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index af0c99e0fe..78db62530b 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -5,10 +5,10 @@ use entity::EntityMap; use ir::{Ebb, Inst}; -use ir::progpoint::{ProgramOrder, ExpandedProgramPoint}; +use ir::progpoint::{ExpandedProgramPoint, ProgramOrder}; use packed_option::PackedOption; use std::cmp; -use std::iter::{Iterator, IntoIterator}; +use std::iter::{IntoIterator, Iterator}; use timing; /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not @@ -734,7 +734,6 @@ impl<'f> DoubleEndedIterator for Insts<'f> { } } - #[cfg(test)] mod tests { use cursor::{Cursor, CursorPosition}; diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 4f4e671f30..0d805cb3de 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -22,29 +22,29 @@ mod sourceloc; mod trapcode; mod valueloc; -pub use ir::builder::{InstBuilder, InstBuilderBase, InstInserterBase, InsertBuilder}; +pub use ir::builder::{InsertBuilder, InstBuilder, InstBuilderBase, InstInserterBase}; pub use ir::dfg::{DataFlowGraph, ValueDef}; -pub use ir::entities::{Ebb, Inst, Value, StackSlot, GlobalVar, JumpTable, FuncRef, SigRef, Heap}; -pub use ir::extfunc::{Signature, CallConv, AbiParam, ArgumentExtension, ArgumentPurpose, - ExtFuncData}; +pub use ir::entities::{Ebb, FuncRef, GlobalVar, Heap, Inst, JumpTable, SigRef, StackSlot, Value}; +pub use ir::extfunc::{AbiParam, ArgumentExtension, ArgumentPurpose, CallConv, ExtFuncData, + Signature}; pub use ir::extname::ExternalName; pub use ir::function::Function; pub use ir::globalvar::GlobalVarData; -pub use ir::heap::{HeapData, HeapStyle, HeapBase}; -pub use ir::instructions::{Opcode, InstructionData, VariableArgs, ValueList, ValueListPool}; +pub use ir::heap::{HeapBase, HeapData, HeapStyle}; +pub use ir::instructions::{InstructionData, Opcode, ValueList, ValueListPool, VariableArgs}; pub use ir::jumptable::JumpTableData; pub use ir::layout::Layout; pub use ir::libcall::LibCall; pub use ir::memflags::MemFlags; -pub use ir::progpoint::{ProgramPoint, ProgramOrder, ExpandedProgramPoint}; +pub use ir::progpoint::{ExpandedProgramPoint, ProgramOrder, ProgramPoint}; pub use ir::sourceloc::SourceLoc; -pub use ir::stackslot::{StackSlots, StackSlotKind, StackSlotData}; +pub use ir::stackslot::{StackSlotData, StackSlotKind, StackSlots}; pub use ir::trapcode::TrapCode; pub use ir::types::Type; -pub use ir::valueloc::{ValueLoc, ArgumentLoc}; +pub use ir::valueloc::{ArgumentLoc, ValueLoc}; use binemit; -use entity::{PrimaryMap, EntityMap}; +use entity::{EntityMap, PrimaryMap}; use isa; /// Map of value locations. diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index 8fea3193c6..6eab7ec743 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -147,7 +147,7 @@ pub trait ProgramOrder { mod tests { use super::*; use entity::EntityRef; - use ir::{Inst, Ebb}; + use ir::{Ebb, Inst}; use std::string::ToString; #[test] diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index bc87e11a82..fd484915fd 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -3,8 +3,8 @@ //! The `StackSlotData` struct keeps track of a single stack slot in a function. //! -use entity::{PrimaryMap, Keys, Iter, IterMut}; -use ir::{Type, StackSlot}; +use entity::{Iter, IterMut, Keys, PrimaryMap}; +use ir::{StackSlot, Type}; use packed_option::PackedOption; use std::cmp; use std::fmt; diff --git a/lib/cretonne/src/ir/types.rs b/lib/cretonne/src/ir/types.rs index 29a37f0cbf..496312b1c4 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/cretonne/src/ir/types.rs @@ -1,7 +1,7 @@ //! Common types for the Cretonne code generator. use std::default::Default; -use std::fmt::{self, Display, Debug, Formatter}; +use std::fmt::{self, Debug, Display, Formatter}; /// The type of an SSA value. /// diff --git a/lib/cretonne/src/isa/arm32/abi.rs b/lib/cretonne/src/isa/arm32/abi.rs index 47efbc97bf..020a525ac5 100644 --- a/lib/cretonne/src/isa/arm32/abi.rs +++ b/lib/cretonne/src/isa/arm32/abi.rs @@ -4,7 +4,7 @@ use ir; use isa::RegClass; use regalloc::AllocatableSet; use settings as shared_settings; -use super::registers::{S, D, Q, GPR}; +use super::registers::{D, GPR, Q, S}; /// Legalize `sig`. pub fn legalize_signature( diff --git a/lib/cretonne/src/isa/arm32/binemit.rs b/lib/cretonne/src/isa/arm32/binemit.rs index 78fa2fd657..9f50e63993 100644 --- a/lib/cretonne/src/isa/arm32/binemit.rs +++ b/lib/cretonne/src/isa/arm32/binemit.rs @@ -1,6 +1,6 @@ //! Emitting binary ARM32 machine code. -use binemit::{CodeSink, bad_encoding}; +use binemit::{bad_encoding, CodeSink}; use ir::{Function, Inst}; use regalloc::RegDiversions; diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 02e0d5f1af..11271fe3d9 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -6,11 +6,11 @@ mod binemit; mod enc_tables; mod registers; -use binemit::{CodeSink, MemoryCodeSink, emit_function}; +use binemit::{emit_function, CodeSink, MemoryCodeSink}; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; +use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use ir; use regalloc; use std::fmt; diff --git a/lib/cretonne/src/isa/arm32/registers.rs b/lib/cretonne/src/isa/arm32/registers.rs index e2c4813bdd..305890b29e 100644 --- a/lib/cretonne/src/isa/arm32/registers.rs +++ b/lib/cretonne/src/isa/arm32/registers.rs @@ -6,7 +6,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-arm32.rs")); #[cfg(test)] mod tests { - use super::{INFO, GPR, S, D}; + use super::{D, GPR, INFO, S}; use isa::RegUnit; use std::string::{String, ToString}; diff --git a/lib/cretonne/src/isa/arm64/abi.rs b/lib/cretonne/src/isa/arm64/abi.rs index 14af59ab18..9dba70c42f 100644 --- a/lib/cretonne/src/isa/arm64/abi.rs +++ b/lib/cretonne/src/isa/arm64/abi.rs @@ -4,7 +4,7 @@ use ir; use isa::RegClass; use regalloc::AllocatableSet; use settings as shared_settings; -use super::registers::{GPR, FPR}; +use super::registers::{FPR, GPR}; /// Legalize `sig`. pub fn legalize_signature( diff --git a/lib/cretonne/src/isa/arm64/binemit.rs b/lib/cretonne/src/isa/arm64/binemit.rs index a4b79cda55..dd0654067e 100644 --- a/lib/cretonne/src/isa/arm64/binemit.rs +++ b/lib/cretonne/src/isa/arm64/binemit.rs @@ -1,6 +1,6 @@ //! Emitting binary ARM64 machine code. -use binemit::{CodeSink, bad_encoding}; +use binemit::{bad_encoding, CodeSink}; use ir::{Function, Inst}; use regalloc::RegDiversions; diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 8239cb01e9..485b008574 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -6,11 +6,11 @@ mod binemit; mod enc_tables; mod registers; -use binemit::{CodeSink, MemoryCodeSink, emit_function}; +use binemit::{emit_function, CodeSink, MemoryCodeSink}; use super::super::settings as shared_settings; use isa::enc_tables::{lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; +use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use ir; use regalloc; use std::fmt; diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index f831c0e187..4ab22a6026 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -9,7 +9,7 @@ use binemit::CodeOffset; use isa::{RegClass, RegUnit}; -use ir::{Function, ValueLoc, Inst}; +use ir::{Function, Inst, ValueLoc}; use regalloc::RegDiversions; /// Register constraint for a single value operand or instruction result. @@ -205,6 +205,5 @@ mod tests { // Backward limit assert!(t1.contains(1000, 748)); assert!(!t1.contains(1000, 746)); - } } diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index 7453b7c044..c987606fa6 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -3,8 +3,8 @@ //! This module contains types and functions for working with the encoding tables generated by //! `lib/cretonne/meta/gen_encoding.py`. -use constant_hash::{Table, probe}; -use ir::{Type, Opcode, DataFlowGraph, InstructionData}; +use constant_hash::{probe, Table}; +use ir::{DataFlowGraph, InstructionData, Opcode, Type}; use isa::{Encoding, Legalize}; use settings::PredicateView; use std::ops::Range; diff --git a/lib/cretonne/src/isa/encoding.rs b/lib/cretonne/src/isa/encoding.rs index b82137a6b8..1b850d5c64 100644 --- a/lib/cretonne/src/isa/encoding.rs +++ b/lib/cretonne/src/isa/encoding.rs @@ -1,7 +1,7 @@ //! The `Encoding` struct. use binemit::CodeOffset; -use isa::constraints::{RecipeConstraints, BranchRange}; +use isa::constraints::{BranchRange, RecipeConstraints}; use std::fmt; /// Bits needed to encode an instruction as binary machine code. diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index e2c79e0377..46ad7679e6 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -4,17 +4,16 @@ use ir; use isa::{RegClass, RegUnit, TargetIsa}; use regalloc::AllocatableSet; use settings as shared_settings; -use super::registers::{GPR, FPR, RU}; -use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; -use ir::{AbiParam, ArgumentPurpose, ArgumentLoc, ArgumentExtension, CallConv, InstBuilder}; -use ir::stackslot::{StackSize, StackOffset}; +use super::registers::{FPR, GPR, RU}; +use abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion}; +use ir::{AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, CallConv, InstBuilder}; +use ir::stackslot::{StackOffset, StackSize}; use ir::immediates::Imm64; use stack_layout::layout_stack; use std::i32; -use cursor::{Cursor, EncCursor, CursorPosition}; +use cursor::{Cursor, CursorPosition, EncCursor}; use result; - /// Argument registers for x86-64 static ARG_GPRS: [RU; 6] = [RU::rdi, RU::rsi, RU::rdx, RU::rcx, RU::r8, RU::r9]; diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 5f2f9512ce..5ec1d7175f 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -1,9 +1,9 @@ //! Emitting binary Intel machine code. -use binemit::{CodeSink, Reloc, bad_encoding}; -use ir::{Function, Inst, Ebb, InstructionData, Opcode, TrapCode}; -use ir::condcodes::{CondCode, IntCC, FloatCC}; -use isa::{RegUnit, StackRef, StackBase, StackBaseMask}; +use binemit::{bad_encoding, CodeSink, Reloc}; +use ir::{Ebb, Function, Inst, InstructionData, Opcode, TrapCode}; +use ir::condcodes::{CondCode, FloatCC, IntCC}; +use isa::{RegUnit, StackBase, StackBaseMask, StackRef}; use regalloc::RegDiversions; use super::registers::RU; diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index d3f62adf7c..b8461c0008 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -22,7 +22,6 @@ fn expand_sdivrem( cfg: &mut ControlFlowGraph, isa: &isa::TargetIsa, ) { - let (x, y, is_srem) = match func.dfg[inst] { ir::InstructionData::Binary { opcode: ir::Opcode::Sdiv, @@ -113,7 +112,6 @@ fn expand_udivrem( _cfg: &mut ControlFlowGraph, isa: &isa::TargetIsa, ) { - let (x, y, is_urem) = match func.dfg[inst] { ir::InstructionData::Binary { opcode: ir::Opcode::Udiv, @@ -324,7 +322,7 @@ fn expand_fcvt_to_sint( cfg: &mut ControlFlowGraph, _isa: &isa::TargetIsa, ) { - use ir::condcodes::{IntCC, FloatCC}; + use ir::condcodes::{FloatCC, IntCC}; use ir::immediates::{Ieee32, Ieee64}; let x; @@ -423,7 +421,7 @@ fn expand_fcvt_to_uint( cfg: &mut ControlFlowGraph, _isa: &isa::TargetIsa, ) { - use ir::condcodes::{IntCC, FloatCC}; + use ir::condcodes::{FloatCC, IntCC}; use ir::immediates::{Ieee32, Ieee64}; let x; diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 04ff149654..aa6f2a0a01 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -6,11 +6,11 @@ mod binemit; mod enc_tables; mod registers; -use binemit::{CodeSink, MemoryCodeSink, emit_function}; +use binemit::{emit_function, CodeSink, MemoryCodeSink}; use super::super::settings as shared_settings; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; +use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use ir; use regalloc; use result; diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 1a7714fc76..8c141126af 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -40,9 +40,9 @@ //! The configured target ISA trait object is a `Box` which can be used for multiple //! concurrent function compilations. -pub use isa::constraints::{RecipeConstraints, OperandConstraint, ConstraintKind, BranchRange}; -pub use isa::encoding::{Encoding, EncInfo}; -pub use isa::registers::{RegInfo, RegUnit, RegClass, RegClassIndex, regs_overlap}; +pub use isa::constraints::{BranchRange, ConstraintKind, OperandConstraint, RecipeConstraints}; +pub use isa::encoding::{EncInfo, Encoding}; +pub use isa::registers::{regs_overlap, RegClass, RegClassIndex, RegInfo, RegUnit}; pub use isa::stack::{StackBase, StackBaseMask, StackRef}; use binemit; @@ -249,7 +249,7 @@ pub trait TargetIsa: fmt::Display { let _tt = timing::prologue_epilogue(); // This default implementation is unlikely to be good enough. use stack_layout::layout_stack; - use ir::stackslot::{StackSize, StackOffset}; + use ir::stackslot::{StackOffset, StackSize}; let word_size = if self.flags().is_64bit() { 8 } else { 4 }; diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index 1db377b0ef..4eff90c954 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -5,12 +5,12 @@ //! //! This doesn't support the soft-float ABI at the moment. -use abi::{ArgAction, ValueConversion, ArgAssigner, legalize_args}; -use ir::{self, Type, AbiParam, ArgumentLoc, ArgumentExtension, ArgumentPurpose}; +use abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion}; +use ir::{self, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, Type}; use isa::RegClass; use regalloc::AllocatableSet; use settings as shared_settings; -use super::registers::{GPR, FPR}; +use super::registers::{FPR, GPR}; use super::settings; use std::i32; diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/cretonne/src/isa/riscv/binemit.rs index 8cc1ef17d1..120324d1be 100644 --- a/lib/cretonne/src/isa/riscv/binemit.rs +++ b/lib/cretonne/src/isa/riscv/binemit.rs @@ -1,8 +1,8 @@ //! Emitting binary RISC-V machine code. -use binemit::{CodeSink, Reloc, bad_encoding}; +use binemit::{bad_encoding, CodeSink, Reloc}; use ir::{Function, Inst, InstructionData}; -use isa::{RegUnit, StackRef, StackBaseMask}; +use isa::{RegUnit, StackBaseMask, StackRef}; use predicates::is_signed_int; use regalloc::RegDiversions; use std::u32; diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 6fc976686b..63a3c01088 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -7,10 +7,10 @@ mod enc_tables; mod registers; use super::super::settings as shared_settings; -use binemit::{CodeSink, MemoryCodeSink, emit_function}; +use binemit::{emit_function, CodeSink, MemoryCodeSink}; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; -use isa::{TargetIsa, RegInfo, RegClass, EncInfo}; +use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use ir; use regalloc; use std::fmt; @@ -116,7 +116,7 @@ mod tests { use settings::{self, Configurable}; use isa; use ir::{DataFlowGraph, InstructionData, Opcode}; - use ir::{types, immediates}; + use ir::{immediates, types}; use std::string::{String, ToString}; fn encstr(isa: &isa::TargetIsa, enc: Result) -> String { diff --git a/lib/cretonne/src/isa/riscv/registers.rs b/lib/cretonne/src/isa/riscv/registers.rs index e2073899b6..d6254fe38e 100644 --- a/lib/cretonne/src/isa/riscv/registers.rs +++ b/lib/cretonne/src/isa/riscv/registers.rs @@ -6,7 +6,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-riscv.rs")); #[cfg(test)] mod tests { - use super::{INFO, GPR, FPR}; + use super::{FPR, GPR, INFO}; use isa::RegUnit; use std::string::{String, ToString}; diff --git a/lib/cretonne/src/isa/riscv/settings.rs b/lib/cretonne/src/isa/riscv/settings.rs index 2f0f6822a9..27cba43f2f 100644 --- a/lib/cretonne/src/isa/riscv/settings.rs +++ b/lib/cretonne/src/isa/riscv/settings.rs @@ -22,12 +22,12 @@ mod tests { assert_eq!( f.to_string(), "[riscv]\n\ - supports_m = false\n\ - supports_a = false\n\ - supports_f = false\n\ - supports_d = false\n\ - enable_m = true\n\ - enable_e = false\n" + supports_m = false\n\ + supports_a = false\n\ + supports_f = false\n\ + supports_d = false\n\ + enable_m = true\n\ + enable_e = false\n" ); // Predicates are not part of the Display output. assert_eq!(f.full_float(), false); diff --git a/lib/cretonne/src/isa/stack.rs b/lib/cretonne/src/isa/stack.rs index 9e09a52808..7eac43b62c 100644 --- a/lib/cretonne/src/isa/stack.rs +++ b/lib/cretonne/src/isa/stack.rs @@ -4,7 +4,7 @@ //! defined in this module expresses the low-level details of accessing a stack slot from an //! encoded instruction. -use ir::stackslot::{StackSlots, StackOffset, StackSlotKind}; +use ir::stackslot::{StackOffset, StackSlotKind, StackSlots}; use ir::StackSlot; /// A method for referencing a stack slot in the current stack frame. diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index e1e4a17956..6cad2775e5 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -20,8 +20,8 @@ use abi::{legalize_abi_value, ValueConversion}; use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; -use ir::{Function, DataFlowGraph, Inst, InstBuilder, Ebb, Type, Value, Signature, SigRef, - AbiParam, ArgumentPurpose, ArgumentLoc, ValueLoc}; +use ir::{AbiParam, ArgumentLoc, ArgumentPurpose, DataFlowGraph, Ebb, Function, Inst, InstBuilder, + SigRef, Signature, Type, Value, ValueLoc}; use ir::instructions::CallInfo; use isa::TargetIsa; use legalizer::split::{isplit, vsplit}; diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 68f24ee0da..d434d6b6e5 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -235,7 +235,6 @@ fn expand_select( cfg.recompute_ebb(pos.func, old_ebb); } - /// Expand illegal `f32const` and `f64const` instructions. fn expand_fconst( inst: ir::Inst, diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/cretonne/src/legalizer/split.rs index cdc60974e0..8f82166db8 100644 --- a/lib/cretonne/src/legalizer/split.rs +++ b/lib/cretonne/src/legalizer/split.rs @@ -66,7 +66,7 @@ use cursor::{Cursor, CursorPosition, FuncCursor}; use flowgraph::ControlFlowGraph; -use ir::{self, Ebb, Inst, Value, Type, Opcode, ValueDef, InstructionData, InstBuilder}; +use ir::{self, Ebb, Inst, InstBuilder, InstructionData, Opcode, Type, Value, ValueDef}; use std::iter; use std::vec::Vec; @@ -229,7 +229,6 @@ fn split_value( let hi = pos.func.dfg.append_ebb_param(ebb, split_type); reuse = Some((lo, hi)); - // Now the original value is dangling. Insert a concatenation instruction that can // compute it from the two new parameters. This also serves as a record of what we // did so a future call to this function doesn't have to redo the work. diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index cde427df9f..99246defe3 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -1,12 +1,7 @@ //! Cretonne code generation library. -#![deny(missing_docs, - trivial_numeric_casts, - unused_extern_crates)] - -#![cfg_attr(feature="clippy", - plugin(clippy(conf_file="../../clippy.toml")))] - +#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature="cargo-clippy", allow( // Rustfmt 0.9.0 is at odds with this lint: block_in_if_condition_stmt, diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index f0327296a1..546fc01bd8 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -1,7 +1,7 @@ //! A Loop Invariant Code Motion optimization pass use cursor::{Cursor, FuncCursor}; -use ir::{Function, Ebb, Inst, Value, Type, InstBuilder, Layout, Opcode, DataFlowGraph}; +use ir::{DataFlowGraph, Ebb, Function, Inst, InstBuilder, Layout, Opcode, Type, Value}; use flowgraph::ControlFlowGraph; use std::collections::HashSet; use dominator_tree::DominatorTree; @@ -121,7 +121,6 @@ fn has_pre_header( result } - // Change the destination of a jump or branch instruction. Does nothing if called with a non-jump // or non-branch instruction. fn change_branch_jump_destination(inst: Inst, new_ebb: Ebb, func: &mut Function) { diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index 431496b6e4..89d2939ac9 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -2,10 +2,10 @@ //! and parent in the loop tree. use dominator_tree::DominatorTree; -use entity::{PrimaryMap, Keys}; +use entity::{Keys, PrimaryMap}; use entity::EntityMap; use flowgraph::ControlFlowGraph; -use ir::{Function, Ebb, Layout}; +use ir::{Ebb, Function, Layout}; use packed_option::PackedOption; use timing; use std::vec::Vec; @@ -218,7 +218,6 @@ impl LoopAnalysis { } } } - } } } @@ -227,7 +226,7 @@ impl LoopAnalysis { mod test { use cursor::{Cursor, FuncCursor}; - use ir::{Function, InstBuilder, types}; + use ir::{types, Function, InstBuilder}; use loop_analysis::{Loop, LoopAnalysis}; use flowgraph::ControlFlowGraph; use dominator_tree::DominatorTree; @@ -257,7 +256,6 @@ mod test { cur.insert_ebb(ebb3); cur.ins().brnz(cond, ebb0, &[]); - } let mut loop_analysis = LoopAnalysis::new(); @@ -317,7 +315,6 @@ mod test { cur.insert_ebb(ebb5); cur.ins().brnz(cond, ebb0, &[]); - } let mut loop_analysis = LoopAnalysis::new(); diff --git a/lib/cretonne/src/postopt.rs b/lib/cretonne/src/postopt.rs index a95baefd3a..efe85d40d7 100644 --- a/lib/cretonne/src/postopt.rs +++ b/lib/cretonne/src/postopt.rs @@ -4,8 +4,8 @@ use cursor::{Cursor, EncCursor}; use ir::dfg::ValueDef; -use ir::{Function, InstructionData, Value, InstBuilder, Ebb, Inst}; -use ir::condcodes::{CondCode, IntCC, FloatCC}; +use ir::{Ebb, Function, Inst, InstBuilder, InstructionData, Value}; +use ir::condcodes::{CondCode, FloatCC, IntCC}; use ir::instructions::{Opcode, ValueList}; use ir::immediates::Imm64; use isa::TargetIsa; @@ -67,54 +67,48 @@ fn optimize_cpu_flags( cond, args: cmp_args, .. - } => { - CmpBrInfo { - br_inst: inst, - cmp_inst: cond_inst, - destination, - args: args.clone(), - cmp_arg: cmp_args[0], - invert_branch_cond, - kind: CmpBrKind::Icmp { - cond, - arg: cmp_args[1], - }, - } - } + } => CmpBrInfo { + br_inst: inst, + cmp_inst: cond_inst, + destination, + args: args.clone(), + cmp_arg: cmp_args[0], + invert_branch_cond, + kind: CmpBrKind::Icmp { + cond, + arg: cmp_args[1], + }, + }, InstructionData::IntCompareImm { cond, arg: cmp_arg, imm: cmp_imm, .. - } => { - CmpBrInfo { - br_inst: inst, - cmp_inst: cond_inst, - destination, - args: args.clone(), - cmp_arg, - invert_branch_cond, - kind: CmpBrKind::IcmpImm { cond, imm: cmp_imm }, - } - } + } => CmpBrInfo { + br_inst: inst, + cmp_inst: cond_inst, + destination, + args: args.clone(), + cmp_arg, + invert_branch_cond, + kind: CmpBrKind::IcmpImm { cond, imm: cmp_imm }, + }, InstructionData::FloatCompare { cond, args: cmp_args, .. - } => { - CmpBrInfo { - br_inst: inst, - cmp_inst: cond_inst, - destination, - args: args.clone(), - cmp_arg: cmp_args[0], - invert_branch_cond, - kind: CmpBrKind::Fcmp { - cond, - arg: cmp_args[1], - }, - } - } + } => CmpBrInfo { + br_inst: inst, + cmp_inst: cond_inst, + destination, + args: args.clone(), + cmp_arg: cmp_args[0], + invert_branch_cond, + kind: CmpBrKind::Fcmp { + cond, + arg: cmp_args[1], + }, + }, _ => return, } } else { @@ -179,7 +173,6 @@ fn optimize_cpu_flags( pos.func.update_encoding(info.br_inst, isa).is_ok(); } - //---------------------------------------------------------------------- // // The main post-opt pass. @@ -204,7 +197,6 @@ pub fn do_postopt(func: &mut Function, isa: &TargetIsa) { last_flags_clobber = Some(inst) } } - } } } diff --git a/lib/cretonne/src/preopt.rs b/lib/cretonne/src/preopt.rs index 34dab71c4a..81e798e987 100644 --- a/lib/cretonne/src/preopt.rs +++ b/lib/cretonne/src/preopt.rs @@ -4,15 +4,14 @@ use cursor::{Cursor, FuncCursor}; use ir::dfg::ValueDef; -use ir::{Function, InstructionData, Value, DataFlowGraph, InstBuilder, Type}; +use ir::{DataFlowGraph, Function, InstBuilder, InstructionData, Type, Value}; use ir::Inst; use ir::types::{I32, I64}; use ir::instructions::Opcode; -use divconst_magic_numbers::{MU32, MU64, MS32, MS64}; -use divconst_magic_numbers::{magicU32, magicU64, magicS32, magicS64}; +use divconst_magic_numbers::{MS32, MS64, MU32, MU64}; +use divconst_magic_numbers::{magicS32, magicS64, magicU32, magicU64}; use timing; - //---------------------------------------------------------------------- // // Pattern-match helpers and transformation for div and rem by constants. @@ -149,7 +148,6 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso }; match *divrem_info { - // -------------------- U32 -------------------- // U32 div, rem by zero: ignore @@ -447,7 +445,6 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } } } - } } @@ -554,13 +551,11 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) { } } - /// The main pre-opt pass. pub fn do_preopt(func: &mut Function) { let _tt = timing::preopt(); let mut pos = FuncCursor::new(func); while let Some(_ebb) = pos.next_ebb() { - while let Some(inst) = pos.next_inst() { // Apply basic simplifications. simplify(&mut pos, inst); diff --git a/lib/cretonne/src/regalloc/affinity.rs b/lib/cretonne/src/regalloc/affinity.rs index 7d85ae8721..0e49ac5f0b 100644 --- a/lib/cretonne/src/regalloc/affinity.rs +++ b/lib/cretonne/src/regalloc/affinity.rs @@ -10,7 +10,7 @@ use std::fmt; use ir::{AbiParam, ArgumentLoc}; -use isa::{TargetIsa, RegInfo, RegClassIndex, OperandConstraint, ConstraintKind}; +use isa::{ConstraintKind, OperandConstraint, RegClassIndex, RegInfo, TargetIsa}; /// Preferred register allocation for an SSA value. #[derive(Clone, Copy, Debug)] diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/allocatable_set.rs index 8c40809310..63b7a297d7 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/allocatable_set.rs @@ -5,7 +5,7 @@ //! "register unit" abstraction. Every register contains one or more register units. Registers that //! share a register unit can't be in use at the same time. -use isa::registers::{RegInfo, RegUnit, RegUnitMask, RegClass}; +use isa::registers::{RegClass, RegInfo, RegUnit, RegUnitMask}; use std::char; use std::fmt; use std::iter::ExactSizeIterator; diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index f23d55433e..b8a7270d59 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -10,7 +10,7 @@ use dbg::DisplayList; use dominator_tree::{DominatorTree, DominatorTreePreorder}; use flowgraph::ControlFlowGraph; use ir::{self, InstBuilder, ProgramOrder}; -use ir::{Function, Ebb, Inst, Value, ExpandedProgramPoint}; +use ir::{Ebb, ExpandedProgramPoint, Function, Inst, Value}; use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; use regalloc::virtregs::{VirtReg, VirtRegs}; @@ -19,7 +19,7 @@ use std::iter; use std::fmt; use std::slice; use std::vec::Vec; -use isa::{TargetIsa, EncInfo}; +use isa::{EncInfo, TargetIsa}; use timing; // # Implementation @@ -92,7 +92,6 @@ impl Coalescing { predecessors: Vec::new(), backedges: Vec::new(), } - } /// Clear all data structures in this coalescing pass. diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index d7ab5dd0f3..ad5952885d 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -44,10 +44,10 @@ use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; -use ir::{Ebb, Inst, Value, Function, Layout, ValueLoc, SigRef}; -use ir::{InstBuilder, AbiParam, ArgumentLoc, ValueDef}; -use isa::{RegUnit, RegClass, RegInfo, regs_overlap}; -use isa::{TargetIsa, EncInfo, RecipeConstraints, OperandConstraint, ConstraintKind}; +use ir::{Ebb, Function, Inst, Layout, SigRef, Value, ValueLoc}; +use ir::{AbiParam, ArgumentLoc, InstBuilder, ValueDef}; +use isa::{regs_overlap, RegClass, RegInfo, RegUnit}; +use isa::{ConstraintKind, EncInfo, OperandConstraint, RecipeConstraints, TargetIsa}; use packed_option::PackedOption; use regalloc::RegDiversions; use regalloc::affinity::Affinity; @@ -59,7 +59,6 @@ use regalloc::solver::{Solver, SolverError}; use std::mem; use timing; - /// Data structures for the coloring pass. /// /// These are scratch space data structures that can be reused between invocations. @@ -268,7 +267,6 @@ impl<'a> Context<'a> { abi.display(&self.reginfo) ); } - } // The spiller will have assigned an incoming stack slot already. Affinity::Stack => debug_assert!(abi.location.is_stack()), @@ -426,7 +424,6 @@ impl<'a> Context<'a> { self.iterate_solution(throughs, ®s.global, &mut replace_global_defines) }); - // The solution and/or fixed input constraints may require us to shuffle the set of live // registers around. self.shuffle_inputs(&mut regs.input); @@ -722,7 +719,6 @@ impl<'a> Context<'a> { ConstraintKind::Reg | ConstraintKind::Tied(_) | ConstraintKind::Stack => {} - } } } @@ -869,7 +865,6 @@ impl<'a> Context<'a> { self.solver.clear_all_global_flags(); } }; - } } diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/cretonne/src/regalloc/context.rs index a8284a9579..cb29963e89 100644 --- a/lib/cretonne/src/regalloc/context.rs +++ b/lib/cretonne/src/regalloc/context.rs @@ -18,7 +18,7 @@ use regalloc::virtregs::VirtRegs; use result::CtonResult; use timing; use topo_order::TopoOrder; -use verifier::{verify_context, verify_liveness, verify_cssa, verify_locations}; +use verifier::{verify_context, verify_cssa, verify_liveness, verify_locations}; /// Persistent memory allocations for register allocation. pub struct Context { @@ -106,7 +106,6 @@ impl Context { verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?; } - // Pass: Spilling. self.spilling.run( isa, diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index 68fd04754e..d0b7e4db91 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -7,9 +7,9 @@ //! These register diversions are local to an EBB. No values can be diverted when entering a new //! EBB. -use ir::{Value, ValueLoc, ValueLocations, StackSlot}; +use ir::{StackSlot, Value, ValueLoc, ValueLocations}; use ir::{InstructionData, Opcode}; -use isa::{RegUnit, RegInfo}; +use isa::{RegInfo, RegUnit}; use std::fmt; use std::vec::Vec; diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/cretonne/src/regalloc/live_value_tracker.rs index ee1f58944b..27a76deb5c 100644 --- a/lib/cretonne/src/regalloc/live_value_tracker.rs +++ b/lib/cretonne/src/regalloc/live_value_tracker.rs @@ -6,7 +6,7 @@ use dominator_tree::DominatorTree; use entity::{EntityList, ListPool}; -use ir::{Inst, Ebb, Value, DataFlowGraph, Layout, ExpandedProgramPoint}; +use ir::{DataFlowGraph, Ebb, ExpandedProgramPoint, Inst, Layout, Value}; use partition_slice::partition_slice; use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/cretonne/src/regalloc/liveness.rs index 9a76cb5899..1b82d98bf8 100644 --- a/lib/cretonne/src/regalloc/liveness.rs +++ b/lib/cretonne/src/regalloc/liveness.rs @@ -178,10 +178,10 @@ use entity::SparseMap; use flowgraph::ControlFlowGraph; use ir::dfg::ValueDef; -use ir::{Function, Value, Inst, Ebb, Layout, ProgramPoint}; -use isa::{TargetIsa, EncInfo}; +use ir::{Ebb, Function, Inst, Layout, ProgramPoint, Value}; +use isa::{EncInfo, TargetIsa}; use regalloc::affinity::Affinity; -use regalloc::liverange::{LiveRange, LiveRangeForest, LiveRangeContext}; +use regalloc::liverange::{LiveRange, LiveRangeContext, LiveRangeForest}; use std::mem; use std::ops::Index; use std::vec::Vec; @@ -378,7 +378,6 @@ impl Liveness { mem::replace(&mut lr.affinity, Affinity::Stack) } - /// Compute the live ranges of all SSA values used in `func`. /// This clears out any existing analysis stored in this data structure. pub fn compute(&mut self, isa: &TargetIsa, func: &mut Function, cfg: &ControlFlowGraph) { diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 2d66b010f7..79526c613d 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -109,7 +109,7 @@ use bforest; use entity::SparseMapValue; -use ir::{Inst, Ebb, Value, Layout, ProgramPoint, ExpandedProgramPoint, ProgramOrder}; +use ir::{Ebb, ExpandedProgramPoint, Inst, Layout, ProgramOrder, ProgramPoint, Value}; use regalloc::affinity::Affinity; use std::cmp::Ordering; @@ -457,9 +457,9 @@ impl SparseMapValue for GenLiveRange { mod tests { use super::{GenLiveRange, LiveRangeContext}; use bforest; - use ir::{Inst, Ebb, Value}; + use ir::{Ebb, Inst, Value}; use entity::EntityRef; - use ir::{ProgramOrder, ExpandedProgramPoint}; + use ir::{ExpandedProgramPoint, ProgramOrder}; use std::cmp::Ordering; use std::vec::Vec; @@ -543,7 +543,6 @@ mod tests { // Save for next round. prev_end = Some(end); } - } } diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index 67e0e99b70..6bbe2b75a5 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -36,7 +36,7 @@ // Remove once we're using the pressure tracker. #![allow(dead_code)] -use isa::registers::{RegInfo, MAX_TRACKED_TOPRCS, RegClass, RegClassMask}; +use isa::registers::{RegClass, RegClassMask, RegInfo, MAX_TRACKED_TOPRCS}; use regalloc::AllocatableSet; use std::cmp::min; use std::fmt; @@ -135,7 +135,7 @@ impl Pressure { /// `can_take()` to check again. fn check_avail(&self, rc: RegClass) -> RegClassMask { let entry = match self.toprc.get(rc.toprc as usize) { - None => return 0, // Not a pressure tracked bank. + None => return 0, // Not a pressure tracked bank. Some(e) => e, }; let mask = 1 << rc.toprc; @@ -269,7 +269,7 @@ impl fmt::Display for Pressure { #[cfg(test)] #[cfg(build_arm32)] mod tests { - use isa::{TargetIsa, RegClass}; + use isa::{RegClass, TargetIsa}; use regalloc::AllocatableSet; use std::borrow::Borrow; use super::Pressure; diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 106830bc19..2ce9a04f75 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -12,10 +12,10 @@ use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; use entity::{SparseMap, SparseMapValue}; -use ir::{Ebb, Inst, Value, Function}; -use ir::{InstBuilder, AbiParam, ArgumentLoc}; +use ir::{Ebb, Function, Inst, Value}; +use ir::{AbiParam, ArgumentLoc, InstBuilder}; use isa::RegClass; -use isa::{TargetIsa, Encoding, EncInfo, RecipeConstraints, ConstraintKind}; +use isa::{ConstraintKind, EncInfo, Encoding, RecipeConstraints, TargetIsa}; use regalloc::affinity::Affinity; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 3559623ae4..92b7a62679 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -350,7 +350,6 @@ impl Move { } } - /// Get the value being moved. fn value(&self) -> Value { match *self { @@ -1161,9 +1160,9 @@ impl fmt::Display for Solver { mod tests { use entity::EntityRef; use ir::Value; - use isa::{TargetIsa, RegClass, RegUnit, RegInfo}; + use isa::{RegClass, RegInfo, RegUnit, TargetIsa}; use regalloc::AllocatableSet; - use super::{Solver, Move}; + use super::{Move, Solver}; use std::boxed::Box; // Make an arm32 `TargetIsa`, if possible. @@ -1396,7 +1395,7 @@ mod tests { mov(v15, gpr, r5, r3), mov(v14, gpr, r4, r5), mov(v13, gpr, r1, r4), - fill(v10, gpr, 0, r1), // Finally complete cycle 1. + fill(v10, gpr, 0, r1) // Finally complete cycle 1. ] ); } diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/cretonne/src/regalloc/spilling.rs index f616ea3b74..5a695eeb8b 100644 --- a/lib/cretonne/src/regalloc/spilling.rs +++ b/lib/cretonne/src/regalloc/spilling.rs @@ -17,9 +17,9 @@ use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; -use ir::{InstBuilder, Function, Ebb, Inst, Value, ValueLoc, SigRef}; -use isa::registers::{RegClassMask, RegClassIndex}; -use isa::{TargetIsa, RegInfo, EncInfo, RecipeConstraints, ConstraintKind}; +use ir::{Ebb, Function, Inst, InstBuilder, SigRef, Value, ValueLoc}; +use isa::registers::{RegClassIndex, RegClassMask}; +use isa::{ConstraintKind, EncInfo, RecipeConstraints, RegInfo, TargetIsa}; use regalloc::affinity::Affinity; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; @@ -359,12 +359,10 @@ impl<'a> Context<'a> { if abi.location.is_reg() { let (rci, spilled) = match self.liveness[arg].affinity { Affinity::Reg(rci) => (rci, false), - Affinity::Stack => { - ( - self.cur.isa.regclass_for_abi_type(abi.value_type).into(), - true, - ) - } + Affinity::Stack => ( + self.cur.isa.regclass_for_abi_type(abi.value_type).into(), + true, + ), Affinity::None => panic!("Missing affinity for {}", arg), }; let mut reguse = RegUse::new(arg, fixed_args + idx, rci); diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index 8069a0b69b..4583effe4f 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -14,9 +14,9 @@ use dbg::DisplayList; use dominator_tree::DominatorTreePreorder; use entity::{EntityList, ListPool}; -use entity::{PrimaryMap, EntityMap, Keys}; +use entity::{EntityMap, Keys, PrimaryMap}; use entity::EntityRef; -use ir::{Value, Function}; +use ir::{Function, Value}; use packed_option::PackedOption; use ref_slice::ref_slice; use std::cmp::Ordering; diff --git a/lib/cretonne/src/scoped_hash_map.rs b/lib/cretonne/src/scoped_hash_map.rs index ded42672ab..4acc7087f0 100644 --- a/lib/cretonne/src/scoped_hash_map.rs +++ b/lib/cretonne/src/scoped_hash_map.rs @@ -4,7 +4,7 @@ //! container that has a concept of scopes that can be entered and exited, such that //! values inserted while inside a scope aren't visible outside the scope. -use std::collections::{HashMap, hash_map}; +use std::collections::{hash_map, HashMap}; use std::hash::Hash; use std::mem; diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index 20ba99e0d9..ed1d8cd168 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -357,18 +357,18 @@ mod tests { assert_eq!( f.to_string(), "[shared]\n\ - opt_level = \"default\"\n\ - enable_verifier = true\n\ - is_64bit = false\n\ - is_pic = false\n\ - return_at_end = false\n\ - avoid_div_traps = false\n\ - is_compressed = false\n\ - enable_float = true\n\ - enable_simd = true\n\ - enable_atomics = true\n\ - spiderwasm_prologue_words = 0\n\ - allones_funcaddrs = false\n" + opt_level = \"default\"\n\ + enable_verifier = true\n\ + is_64bit = false\n\ + is_pic = false\n\ + return_at_end = false\n\ + avoid_div_traps = false\n\ + is_compressed = false\n\ + enable_float = true\n\ + enable_simd = true\n\ + enable_atomics = true\n\ + spiderwasm_prologue_words = 0\n\ + allones_funcaddrs = false\n" ); assert_eq!(f.opt_level(), super::OptLevel::Default); assert_eq!(f.enable_simd(), true); diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index 1cd82e4b2c..073101cd9c 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -2,7 +2,7 @@ use cursor::{Cursor, FuncCursor}; use dominator_tree::DominatorTree; -use ir::{InstructionData, Function, Inst, Opcode, Type}; +use ir::{Function, Inst, InstructionData, Opcode, Type}; use scoped_hash_map::ScopedHashMap; use timing; use std::vec::Vec; diff --git a/lib/cretonne/src/stack_layout.rs b/lib/cretonne/src/stack_layout.rs index 0cf6e77527..9d723791d7 100644 --- a/lib/cretonne/src/stack_layout.rs +++ b/lib/cretonne/src/stack_layout.rs @@ -1,9 +1,9 @@ //! Computing stack layout. use ir::StackSlots; -use ir::stackslot::{StackSize, StackOffset, StackSlotKind}; +use ir::stackslot::{StackOffset, StackSize, StackSlotKind}; use result::CtonError; -use std::cmp::{min, max}; +use std::cmp::{max, min}; /// Compute the stack frame layout. /// @@ -110,7 +110,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result Verifier<'a> { } fn ebb_integrity(&self, ebb: Ebb, inst: Inst) -> Result { - let is_terminator = self.func.dfg[inst].opcode().is_terminator(); let is_last_inst = self.func.layout.last_inst(ebb) == Some(inst); @@ -1155,7 +1154,7 @@ impl<'a> Verifier<'a> { #[cfg(test)] mod tests { - use super::{Verifier, Error}; + use super::{Error, Verifier}; use ir::Function; use ir::instructions::{InstructionData, Opcode}; use entity::EntityList; diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 87316c8ca4..dee3ab0346 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -3,9 +3,9 @@ //! The `write` module provides the `write_function` function which converts an IR `Function` to an //! equivalent textual form. This textual form can be read back by the `cretonne-reader` crate. -use ir::{Function, DataFlowGraph, Ebb, Inst, Value, ValueDef, Type, SigRef}; -use isa::{TargetIsa, RegInfo}; -use std::fmt::{self, Result, Error, Write}; +use ir::{DataFlowGraph, Ebb, Function, Inst, SigRef, Type, Value, ValueDef}; +use isa::{RegInfo, TargetIsa}; +use std::fmt::{self, Error, Result, Write}; use std::result; use packed_option::ReservedValue; use std::string::String; @@ -146,7 +146,6 @@ pub fn write_ebb(w: &mut Write, func: &Function, isa: Option<&TargetIsa>, ebb: E Ok(()) } - //---------------------------------------------------------------------- // // Instructions @@ -452,7 +451,7 @@ impl<'a> fmt::Display for DisplayValues<'a> { #[cfg(test)] mod tests { - use ir::{Function, ExternalName, StackSlotData, StackSlotKind}; + use ir::{ExternalName, Function, StackSlotData, StackSlotKind}; use ir::types; use std::string::ToString; diff --git a/lib/filetests/src/concurrent.rs b/lib/filetests/src/concurrent.rs index 1db5aa6832..a93d39cddf 100644 --- a/lib/filetests/src/concurrent.rs +++ b/lib/filetests/src/concurrent.rs @@ -6,12 +6,12 @@ use cretonne::timing; use std::panic::catch_unwind; use std::path::{Path, PathBuf}; -use std::sync::mpsc::{channel, Sender, Receiver}; +use std::sync::mpsc::{channel, Receiver, Sender}; use std::sync::{Arc, Mutex}; use std::thread; use std::time::Duration; use num_cpus; -use {TestResult, runone}; +use {runone, TestResult}; /// Request sent to worker threads contains jobid and path. struct Request(usize, PathBuf); diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs index d2d2db55c7..f9e8fb3638 100644 --- a/lib/filetests/src/runner.rs +++ b/lib/filetests/src/runner.rs @@ -8,7 +8,7 @@ use std::fmt::{self, Display}; use std::ffi::OsStr; use std::path::{Path, PathBuf}; use std::time; -use {TestResult, runone}; +use {runone, TestResult}; use concurrent::{ConcurrentRunner, Reply}; /// Timeout in seconds when we're not making progress. @@ -302,7 +302,6 @@ impl TestRunner { // Inter-quartile range. let iqr = q3 - q1; - // Cut-off for what we consider a 'slow' test: 3 IQR from the 75% quartile. // // Q3 + 1.5 IQR are the data points that would be plotted as outliers outside a box plot, @@ -319,7 +318,6 @@ impl TestRunner { { println!("slow: {}", t) } - } /// Scan pushed directories for tests and run them. diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs index dab44ff920..4b3033ea2e 100644 --- a/lib/filetests/src/runone.rs +++ b/lib/filetests/src/runone.rs @@ -13,8 +13,8 @@ use cretonne::verify_function; use cretonne::print_errors::pretty_verifier_error; use cton_reader::parse_test; use cton_reader::IsaSpec; -use {TestResult, new_subtest}; -use subtest::{SubTest, Context, Result}; +use {new_subtest, TestResult}; +use subtest::{Context, Result, SubTest}; /// Read an entire file into a string. fn read_to_string>(path: P) -> io::Result { @@ -82,7 +82,6 @@ pub fn run(path: &Path) -> TestResult { run_one_test(last_tuple, Cow::Owned(func), &mut context)?; } - Ok(started.elapsed()) } diff --git a/lib/filetests/src/subtest.rs b/lib/filetests/src/subtest.rs index 39812ed228..4395780021 100644 --- a/lib/filetests/src/subtest.rs +++ b/lib/filetests/src/subtest.rs @@ -5,8 +5,8 @@ use std::borrow::Cow; use cretonne::ir::Function; use cretonne::isa::TargetIsa; use cretonne::settings::{Flags, FlagsOrIsa}; -use cton_reader::{Details, Comment}; -use filecheck::{CheckerBuilder, Checker, NO_VARIABLES}; +use cton_reader::{Comment, Details}; +use filecheck::{Checker, CheckerBuilder, NO_VARIABLES}; pub type Result = result::Result; diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index 81e48f84fe..db3d9be634 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -13,7 +13,7 @@ use cretonne::ir::entities::AnyEntity; use cretonne::binemit::RegDiversions; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{SubTest, Context, Result}; +use subtest::{Context, Result, SubTest}; use match_directive::match_directive; struct TestBinEmit; @@ -43,8 +43,6 @@ impl TextSink { } } - - impl binemit::CodeSink for TextSink { fn offset(&self) -> binemit::CodeOffset { self.offset @@ -80,23 +78,11 @@ impl binemit::CodeSink for TextSink { name: &ir::ExternalName, addend: binemit::Addend, ) { - write!( - self.text, - "{}({}", - reloc, - name, - ).unwrap(); + write!(self.text, "{}({}", reloc, name,).unwrap(); if addend != 0 { - write!( - self.text, - "{:+}", - addend, - ).unwrap(); + write!(self.text, "{:+}", addend,).unwrap(); } - write!( - self.text, - ") ", - ).unwrap(); + write!(self.text, ") ",).unwrap(); } fn reloc_jt(&mut self, reloc: binemit::Reloc, jt: ir::JumpTable) { @@ -278,10 +264,10 @@ impl SubTest for TestBinEmit { )); } return Err(format!( - "No matching encodings for {} in {}", - func.dfg.display_inst(inst, isa), - DisplayList(&encodings), - )); + "No matching encodings for {} in {}", + func.dfg.display_inst(inst, isa), + DisplayList(&encodings), + )); } let have = sink.text.trim(); if have != want { diff --git a/lib/filetests/src/test_cat.rs b/lib/filetests/src/test_cat.rs index fbbcbd034b..42b18b3625 100644 --- a/lib/filetests/src/test_cat.rs +++ b/lib/filetests/src/test_cat.rs @@ -3,7 +3,7 @@ use std::borrow::Cow; use cretonne::ir::Function; use cton_reader::TestCommand; -use subtest::{self, SubTest, Context, Result as STResult}; +use subtest::{self, Context, Result as STResult, SubTest}; /// Object implementing the `test cat` sub-test. /// diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs index af2343f765..5217f7ebb6 100644 --- a/lib/filetests/src/test_compile.rs +++ b/lib/filetests/src/test_compile.rs @@ -7,7 +7,7 @@ use cretonne::ir; use cretonne; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; diff --git a/lib/filetests/src/test_dce.rs b/lib/filetests/src/test_dce.rs index 5a57d83e3e..4e826c9abc 100644 --- a/lib/filetests/src/test_dce.rs +++ b/lib/filetests/src/test_dce.rs @@ -9,7 +9,7 @@ use cretonne::ir::Function; use cretonne; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; diff --git a/lib/filetests/src/test_domtree.rs b/lib/filetests/src/test_domtree.rs index 6eae539644..fed38d9d50 100644 --- a/lib/filetests/src/test_domtree.rs +++ b/lib/filetests/src/test_domtree.rs @@ -17,7 +17,7 @@ use cretonne::flowgraph::ControlFlowGraph; use cretonne::ir::Function; use cretonne::ir::entities::AnyEntity; use cton_reader::TestCommand; -use subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::{Borrow, Cow}; use std::collections::HashMap; use std::fmt::{self, Write}; @@ -76,7 +76,7 @@ impl SubTest for TestDomtree { Some(got_inst) if got_inst != inst => { return Err(format!( "mismatching idoms for {}:\n\ - want: {}, got: {}", + want: {}, got: {}", src_ebb, inst, got_inst @@ -85,7 +85,7 @@ impl SubTest for TestDomtree { None => { return Err(format!( "mismatching idoms for {}:\n\ - want: {}, got: unreachable", + want: {}, got: unreachable", src_ebb, inst )); @@ -105,7 +105,7 @@ impl SubTest for TestDomtree { if let Some(got_inst) = domtree.idom(ebb) { return Err(format!( "mismatching idoms for renumbered {}:\n\ - want: unrechable, got: {}", + want: unrechable, got: {}", ebb, got_inst )); diff --git a/lib/filetests/src/test_legalizer.rs b/lib/filetests/src/test_legalizer.rs index 08f6c53ae4..76bd9f491b 100644 --- a/lib/filetests/src/test_legalizer.rs +++ b/lib/filetests/src/test_legalizer.rs @@ -8,7 +8,7 @@ use cretonne; use cretonne::ir::Function; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{run_filecheck, Context, Result, SubTest}; use std::fmt::Write; struct TestLegalizer; diff --git a/lib/filetests/src/test_licm.rs b/lib/filetests/src/test_licm.rs index f7d4397b5a..bed36b44c2 100644 --- a/lib/filetests/src/test_licm.rs +++ b/lib/filetests/src/test_licm.rs @@ -9,7 +9,7 @@ use cretonne::ir::Function; use cretonne; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; diff --git a/lib/filetests/src/test_postopt.rs b/lib/filetests/src/test_postopt.rs index 34ffb3f462..ad26a06c30 100644 --- a/lib/filetests/src/test_postopt.rs +++ b/lib/filetests/src/test_postopt.rs @@ -6,7 +6,7 @@ use cretonne::ir::Function; use cretonne; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; diff --git a/lib/filetests/src/test_preopt.rs b/lib/filetests/src/test_preopt.rs index c141d28480..8a30f0feb3 100644 --- a/lib/filetests/src/test_preopt.rs +++ b/lib/filetests/src/test_preopt.rs @@ -6,7 +6,7 @@ use cretonne::ir::Function; use cretonne; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; diff --git a/lib/filetests/src/test_print_cfg.rs b/lib/filetests/src/test_print_cfg.rs index b043dc58b2..4c0b3250d2 100644 --- a/lib/filetests/src/test_print_cfg.rs +++ b/lib/filetests/src/test_print_cfg.rs @@ -8,7 +8,7 @@ use std::borrow::Cow; use cretonne::ir::Function; use cretonne::cfg_printer::CFGPrinter; use cton_reader::TestCommand; -use subtest::{self, SubTest, Context, Result as STResult}; +use subtest::{self, Context, Result as STResult, SubTest}; /// Object implementing the `test print-cfg` sub-test. struct TestPrintCfg; diff --git a/lib/filetests/src/test_regalloc.rs b/lib/filetests/src/test_regalloc.rs index 7cd0788928..e94a142dcc 100644 --- a/lib/filetests/src/test_regalloc.rs +++ b/lib/filetests/src/test_regalloc.rs @@ -9,7 +9,7 @@ use cretonne::ir::Function; use cretonne; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; diff --git a/lib/filetests/src/test_simple_gvn.rs b/lib/filetests/src/test_simple_gvn.rs index f967c3fa91..04fcb078dd 100644 --- a/lib/filetests/src/test_simple_gvn.rs +++ b/lib/filetests/src/test_simple_gvn.rs @@ -9,7 +9,7 @@ use cretonne::ir::Function; use cretonne; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{SubTest, Context, Result, run_filecheck}; +use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs index 6db9c681ce..906b279408 100644 --- a/lib/filetests/src/test_verifier.rs +++ b/lib/filetests/src/test_verifier.rs @@ -13,7 +13,7 @@ use std::borrow::{Borrow, Cow}; use cretonne::verify_function; use cretonne::ir::Function; use cton_reader::TestCommand; -use subtest::{SubTest, Context, Result}; +use subtest::{Context, Result, SubTest}; use match_directive::match_directive; struct TestVerifier; diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 37d2fe0206..00e5de53f1 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -1,13 +1,13 @@ //! A frontend for building Cretonne IR from other languages. use cretonne::cursor::{Cursor, FuncCursor}; use cretonne::ir; -use cretonne::ir::{Ebb, Type, Value, Function, Inst, JumpTable, StackSlot, JumpTableData, - StackSlotData, DataFlowGraph, InstructionData, ExtFuncData, FuncRef, SigRef, - Signature, InstBuilderBase, GlobalVarData, GlobalVar, HeapData, Heap}; +use cretonne::ir::{DataFlowGraph, Ebb, ExtFuncData, FuncRef, Function, GlobalVar, GlobalVarData, + Heap, HeapData, Inst, InstBuilderBase, InstructionData, JumpTable, + JumpTableData, SigRef, Signature, StackSlot, StackSlotData, Type, Value}; use cretonne::ir::function::DisplayFunction; use cretonne::isa::TargetIsa; -use ssa::{SSABuilder, SideEffects, Block}; -use cretonne::entity::{EntityRef, EntityMap, EntitySet}; +use ssa::{Block, SSABuilder, SideEffects}; +use cretonne::entity::{EntityMap, EntityRef, EntitySet}; use cretonne::packed_option::PackedOption; /// Structure used for translating a series of functions into Cretonne IR. @@ -28,7 +28,6 @@ where types: EntityMap, } - /// Temporary object used to build a single Cretonne IR `Function`. pub struct FunctionBuilder<'a, Variable: 'a> where @@ -125,7 +124,8 @@ where } impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long, Variable> - where Variable: EntityRef +where + Variable: EntityRef, { fn data_flow_graph(&self) -> &DataFlowGraph { &self.builder.func.dfg @@ -165,13 +165,14 @@ impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short // multiple times, so we must deduplicate. let mut unique = EntitySet::::new(); for dest_ebb in self.builder - .func - .jump_tables - .get(table) - .expect("you are referencing an undeclared jump table") - .entries() - .map(|(_, ebb)| ebb) - .filter(|dest_ebb| unique.insert(*dest_ebb)) { + .func + .jump_tables + .get(table) + .expect("you are referencing an undeclared jump table") + .entries() + .map(|(_, ebb)| ebb) + .filter(|dest_ebb| unique.insert(*dest_ebb)) + { self.builder.func_ctx.ssa.declare_ebb_predecessor( dest_ebb, self.builder.position.basic_block.unwrap(), @@ -592,9 +593,9 @@ where mod tests { use cretonne::entity::EntityRef; - use cretonne::ir::{ExternalName, Function, CallConv, Signature, AbiParam, InstBuilder}; + use cretonne::ir::{AbiParam, CallConv, ExternalName, Function, InstBuilder, Signature}; use cretonne::ir::types::*; - use frontend::{FunctionBuilderContext, FunctionBuilder}; + use frontend::{FunctionBuilder, FunctionBuilderContext}; use cretonne::verifier::verify_function; use cretonne::settings; use Variable; diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 628195428a..329b6afb50 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -127,16 +127,12 @@ //! } //! ``` -#![deny(missing_docs, - trivial_numeric_casts, - unused_extern_crates)] - -#![cfg_attr(feature="cargo-clippy", - allow(new_without_default, redundant_field_names))] +#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, redundant_field_names))] extern crate cretonne; -pub use frontend::{FunctionBuilderContext, FunctionBuilder}; +pub use frontend::{FunctionBuilder, FunctionBuilderContext}; pub use variable::Variable; mod frontend; diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 6fa59e784d..96085c38f8 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -6,9 +6,9 @@ //! Lecture Notes in Computer Science, vol 7791. Springer, Berlin, Heidelberg use cretonne::cursor::{Cursor, FuncCursor}; -use cretonne::ir::{Ebb, Value, Inst, Type, Function, InstBuilder}; +use cretonne::ir::{Ebb, Function, Inst, InstBuilder, Type, Value}; use cretonne::ir::instructions::BranchInfo; -use cretonne::entity::{EntityRef, PrimaryMap, EntityMap}; +use cretonne::entity::{EntityMap, EntityRef, PrimaryMap}; use cretonne::packed_option::PackedOption; use cretonne::packed_option::ReservedValue; use std::u32; @@ -715,7 +715,7 @@ where mod tests { use cretonne::cursor::{Cursor, FuncCursor}; use cretonne::entity::EntityRef; - use cretonne::ir::{Function, InstBuilder, Inst, JumpTableData, Opcode}; + use cretonne::ir::{Function, Inst, InstBuilder, JumpTableData, Opcode}; use cretonne::ir::types::*; use cretonne::verify_function; use cretonne::ir::instructions::BranchInfo; @@ -960,7 +960,6 @@ mod tests { assert_eq!(func.dfg.ebb_params(ebb1)[0], z2); assert_eq!(func.dfg.ebb_params(ebb1)[1], y3); assert_eq!(func.dfg.resolve_aliases(x3), x1); - } #[test] diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index ea828ea252..15f3e69350 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -1,9 +1,7 @@ //! Performs autodetection of the host for the purposes of running //! Cretonne to generate code to run on the same machine. -#![deny(missing_docs, - trivial_numeric_casts, - unused_extern_crates)] +#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] extern crate cretonne; diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs index 2b1e8fae6d..6cc29a81a1 100644 --- a/lib/reader/src/isaspec.rs +++ b/lib/reader/src/isaspec.rs @@ -6,9 +6,9 @@ //! If a test case file contains `isa` commands, the tests will only be run against the specified //! ISAs. If the file contains no `isa` commands, the tests will be run against all supported ISAs. -use cretonne::settings::{Flags, Configurable, Error as SetError}; +use cretonne::settings::{Configurable, Error as SetError, Flags}; use cretonne::isa::TargetIsa; -use error::{Result, Location}; +use error::{Location, Result}; use testcommand::TestOption; /// The ISA specifications in a `.cton` file. diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 4b1a952d1b..c55bbf1dd4 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -5,7 +5,7 @@ use std::u16; #[allow(unused_imports)] use std::ascii::AsciiExt; use cretonne::ir::types; -use cretonne::ir::{Value, Ebb}; +use cretonne::ir::{Ebb, Value}; use error::Location; /// A Token returned from the `Lexer`. @@ -458,7 +458,7 @@ mod tests { use super::trailing_digits; use super::*; use cretonne::ir::types; - use cretonne::ir::{Value, Ebb}; + use cretonne::ir::{Ebb, Value}; use error::Location; #[test] @@ -556,7 +556,7 @@ mod tests { fn lex_identifiers() { let mut lex = Lexer::new( "v0 v00 vx01 ebb1234567890 ebb5234567890 v1x vx1 vxvx4 \ - function0 function b1 i32x4 f32x5 \ + function0 function b1 i32x4 f32x5 \ iflags fflags iflagss", ); assert_eq!( diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index d6c23cfa81..58fe21b5ce 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -3,17 +3,15 @@ //! The `cton_reader` library supports reading .cton files. This functionality is needed for testing //! Cretonne, but is not essential for a JIT compiler. -#![deny(missing_docs, - trivial_numeric_casts, - unused_extern_crates)] +#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] extern crate cretonne; -pub use error::{Location, Result, Error}; +pub use error::{Error, Location, Result}; pub use parser::{parse_functions, parse_test}; pub use testcommand::{TestCommand, TestOption}; -pub use testfile::{TestFile, Details, Comment}; -pub use isaspec::{IsaSpec, parse_options}; +pub use testfile::{Comment, Details, TestFile}; +pub use isaspec::{parse_options, IsaSpec}; pub use sourcemap::SourceMap; mod error; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index c9da49d9ee..48e2e2745b 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -3,22 +3,21 @@ use std::str::FromStr; use std::{u16, u32}; use std::mem; -use cretonne::ir::{Function, Ebb, Opcode, Value, Type, ExternalName, CallConv, StackSlotData, - StackSlotKind, JumpTable, JumpTableData, Signature, AbiParam, - ArgumentExtension, ExtFuncData, SigRef, FuncRef, StackSlot, ValueLoc, - ArgumentLoc, MemFlags, GlobalVar, GlobalVarData, Heap, HeapData, HeapStyle, - HeapBase}; +use cretonne::ir::{AbiParam, ArgumentExtension, ArgumentLoc, CallConv, Ebb, ExtFuncData, + ExternalName, FuncRef, Function, GlobalVar, GlobalVarData, Heap, HeapBase, + HeapData, HeapStyle, JumpTable, JumpTableData, MemFlags, Opcode, SigRef, + Signature, StackSlot, StackSlotData, StackSlotKind, Type, Value, ValueLoc}; use cretonne::ir; use cretonne::ir::types::VOID; -use cretonne::ir::immediates::{Imm64, Uimm32, Offset32, Ieee32, Ieee64}; +use cretonne::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32}; use cretonne::ir::entities::AnyEntity; -use cretonne::ir::instructions::{InstructionFormat, InstructionData, VariableArgs}; -use cretonne::isa::{self, TargetIsa, Encoding, RegUnit}; +use cretonne::ir::instructions::{InstructionData, InstructionFormat, VariableArgs}; +use cretonne::isa::{self, Encoding, RegUnit, TargetIsa}; use cretonne::{settings, timing}; use cretonne::entity::EntityRef; use cretonne::packed_option::ReservedValue; -use testfile::{TestFile, Details, Comment}; -use error::{Location, Error, Result}; +use testfile::{Comment, Details, TestFile}; +use error::{Error, Location, Result}; use lexer::{self, Lexer, Token}; use testcommand::TestCommand; use isaspec; @@ -1189,12 +1188,12 @@ impl<'a> Parser<'a> { } "bound" => { data.style = match style_name { - "dynamic" => { - HeapStyle::Dynamic { bound_gv: self.match_gv("expected gv bound")? } - } - "static" => { - HeapStyle::Static { bound: self.match_imm64("expected integer bound")? } - } + "dynamic" => HeapStyle::Dynamic { + bound_gv: self.match_gv("expected gv bound")?, + }, + "static" => HeapStyle::Static { + bound: self.match_imm64("expected integer bound")?, + }, t => return err!(self.loc, "unknown heap style '{}'", t), }; } @@ -1736,7 +1735,7 @@ impl<'a> Parser<'a> { return err!( self.loc, "instruction produces {} result values, but {} locations were \ - specified", + specified", results.len(), result_locations.len() ); @@ -1791,7 +1790,7 @@ impl<'a> Parser<'a> { return err!( self.loc, "type variable required for polymorphic opcode, e.g. '{}.{}'; \ - can't infer from {} which is not yet defined", + can't infer from {} which is not yet defined", opcode, constraints.ctrl_typeset().unwrap().example(), ctrl_src_value @@ -1801,7 +1800,7 @@ impl<'a> Parser<'a> { return err!( self.loc, "type variable required for polymorphic opcode, e.g. '{}.{}'; \ - can't infer from {} which is not yet resolved", + can't infer from {} which is not yet resolved", opcode, constraints.ctrl_typeset().unwrap().example(), ctrl_src_value @@ -1890,36 +1889,26 @@ impl<'a> Parser<'a> { opcode: Opcode, ) -> Result { let idata = match opcode.format() { - InstructionFormat::Unary => { - InstructionData::Unary { - opcode, - arg: self.match_value("expected SSA value operand")?, - } - } - InstructionFormat::UnaryImm => { - InstructionData::UnaryImm { - opcode, - imm: self.match_imm64("expected immediate integer operand")?, - } - } - InstructionFormat::UnaryIeee32 => { - InstructionData::UnaryIeee32 { - opcode, - imm: self.match_ieee32("expected immediate 32-bit float operand")?, - } - } - InstructionFormat::UnaryIeee64 => { - InstructionData::UnaryIeee64 { - opcode, - imm: self.match_ieee64("expected immediate 64-bit float operand")?, - } - } - InstructionFormat::UnaryBool => { - InstructionData::UnaryBool { - opcode, - imm: self.match_bool("expected immediate boolean operand")?, - } - } + InstructionFormat::Unary => InstructionData::Unary { + opcode, + arg: self.match_value("expected SSA value operand")?, + }, + InstructionFormat::UnaryImm => InstructionData::UnaryImm { + opcode, + imm: self.match_imm64("expected immediate integer operand")?, + }, + InstructionFormat::UnaryIeee32 => InstructionData::UnaryIeee32 { + opcode, + imm: self.match_ieee32("expected immediate 32-bit float operand")?, + }, + InstructionFormat::UnaryIeee64 => InstructionData::UnaryIeee64 { + opcode, + imm: self.match_ieee64("expected immediate 64-bit float operand")?, + }, + InstructionFormat::UnaryBool => InstructionData::UnaryBool { + opcode, + imm: self.match_bool("expected immediate boolean operand")?, + }, InstructionFormat::UnaryGlobalVar => { let gv = self.match_gv("expected global variable")?; ctx.check_gv(gv, &self.loc)?; @@ -2406,11 +2395,11 @@ impl<'a> Parser<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne::ir::{CallConv, ArgumentExtension, ArgumentPurpose}; + use cretonne::ir::{ArgumentExtension, ArgumentPurpose, CallConv}; use cretonne::ir::types; use cretonne::ir::StackSlotKind; use cretonne::ir::entities::AnyEntity; - use testfile::{Details, Comment}; + use testfile::{Comment, Details}; use isaspec::IsaSpec; use error::Error; diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 36fb954fd9..b30b172768 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -7,8 +7,8 @@ //! to parser clients. use cretonne::ir::entities::AnyEntity; -use cretonne::ir::{StackSlot, GlobalVar, Heap, JumpTable, Ebb, Value, SigRef, FuncRef}; -use error::{Result, Location}; +use cretonne::ir::{Ebb, FuncRef, GlobalVar, Heap, JumpTable, SigRef, StackSlot, Value}; +use error::{Location, Result}; use lexer::split_entity_name; use std::collections::HashMap; diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 97f67d6c48..82fec54e5b 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -22,16 +22,16 @@ //! //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as //! argument. -use cretonne::ir::{self, InstBuilder, MemFlags, JumpTableData}; +use cretonne::ir::{self, InstBuilder, JumpTableData, MemFlags}; use cretonne::ir::types::*; -use cretonne::ir::condcodes::{IntCC, FloatCC}; +use cretonne::ir::condcodes::{FloatCC, IntCC}; use cretonne::packed_option::ReservedValue; use cton_frontend::{FunctionBuilder, Variable}; -use wasmparser::{Operator, MemoryImmediate}; -use translation_utils::{f32_translation, f64_translation, type_to_type, num_return_values}; -use translation_utils::{TableIndex, SignatureIndex, FunctionIndex, MemoryIndex}; -use state::{TranslationState, ControlStackFrame}; -use std::collections::{HashMap, hash_map}; +use wasmparser::{MemoryImmediate, Operator}; +use translation_utils::{num_return_values, type_to_type, f32_translation, f64_translation}; +use translation_utils::{FunctionIndex, MemoryIndex, SignatureIndex, TableIndex}; +use state::{ControlStackFrame, TranslationState}; +use std::collections::{hash_map, HashMap}; use environ::{FuncEnvironment, GlobalValue}; use std::{i32, u32}; use std::vec::Vec; diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index bb21b7536a..b24499bde0 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -1,8 +1,8 @@ //! "Dummy" environment for testing wasm translation. use environ::{FuncEnvironment, GlobalValue, ModuleEnvironment}; -use translation_utils::{Global, Memory, Table, GlobalIndex, TableIndex, SignatureIndex, - FunctionIndex, MemoryIndex}; +use translation_utils::{FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, + Table, TableIndex}; use func_translator::FuncTranslator; use cretonne::ir::{self, InstBuilder}; use cretonne::ir::types::*; diff --git a/lib/wasm/src/environ/mod.rs b/lib/wasm/src/environ/mod.rs index f45dbd3161..57c9f593b2 100644 --- a/lib/wasm/src/environ/mod.rs +++ b/lib/wasm/src/environ/mod.rs @@ -3,5 +3,5 @@ mod spec; mod dummy; -pub use environ::spec::{ModuleEnvironment, FuncEnvironment, GlobalValue}; +pub use environ::spec::{FuncEnvironment, GlobalValue, ModuleEnvironment}; pub use environ::dummy::DummyEnvironment; diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 233188364a..882ca1a30f 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -3,8 +3,8 @@ use cretonne::ir::{self, InstBuilder}; use cretonne::cursor::FuncCursor; use cretonne::settings::Flags; -use translation_utils::{SignatureIndex, FunctionIndex, TableIndex, GlobalIndex, MemoryIndex, - Global, Table, Memory}; +use translation_utils::{FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, + Table, TableIndex}; use std::vec::Vec; use std::string::String; diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index dbb1fc9bcc..159d8120d1 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -6,10 +6,10 @@ use code_translator::translate_operator; use cretonne::entity::EntityRef; -use cretonne::ir::{self, InstBuilder, Ebb}; -use cretonne::result::{CtonResult, CtonError}; +use cretonne::ir::{self, Ebb, InstBuilder}; +use cretonne::result::{CtonError, CtonResult}; use cretonne::timing; -use cton_frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; +use cton_frontend::{FunctionBuilder, FunctionBuilderContext, Variable}; use environ::FuncEnvironment; use state::TranslationState; use wasmparser::{self, BinaryReader}; diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index cc35a19fbc..6f0ed8fd70 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -9,18 +9,14 @@ //! //! The main function of this module is [`translate_module`](fn.translate_module.html). -#![deny(missing_docs, - trivial_numeric_casts, - unused_extern_crates)] -#![cfg_attr(feature="clippy", - plugin(clippy(conf_file="../../clippy.toml")))] -#![cfg_attr(feature="cargo-clippy", - allow(new_without_default, redundant_field_names))] +#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, redundant_field_names))] -extern crate wasmparser; -extern crate cton_frontend; #[macro_use(dbg)] extern crate cretonne; +extern crate cton_frontend; +extern crate wasmparser; mod code_translator; mod func_translator; @@ -32,6 +28,6 @@ mod translation_utils; pub use func_translator::FuncTranslator; pub use module_translator::translate_module; -pub use environ::{FuncEnvironment, ModuleEnvironment, DummyEnvironment, GlobalValue}; -pub use translation_utils::{FunctionIndex, GlobalIndex, TableIndex, MemoryIndex, SignatureIndex, - Global, GlobalInit, Table, Memory}; +pub use environ::{DummyEnvironment, FuncEnvironment, GlobalValue, ModuleEnvironment}; +pub use translation_utils::{FunctionIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, + SignatureIndex, Table, TableIndex}; diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 171f720e12..e345e5ef3c 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -1,11 +1,11 @@ //! Translation skeleton that traverses the whole WebAssembly module and call helper functions //! to deal with each part of it. use cretonne::timing; -use wasmparser::{ParserState, SectionCode, ParserInput, Parser, WasmDecoder, BinaryReaderError}; -use sections_translator::{SectionParsingError, parse_function_signatures, parse_import_section, - parse_function_section, parse_export_section, parse_start_section, - parse_memory_section, parse_global_section, parse_table_section, - parse_elements_section, parse_data_section}; +use wasmparser::{BinaryReaderError, Parser, ParserInput, ParserState, SectionCode, WasmDecoder}; +use sections_translator::{parse_data_section, parse_elements_section, parse_export_section, + parse_function_section, parse_function_signatures, parse_global_section, + parse_import_section, parse_memory_section, parse_start_section, + parse_table_section, SectionParsingError}; use environ::ModuleEnvironment; use std::string::String; diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 05f655ba82..4792c3f450 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -7,12 +7,12 @@ //! The special case of the initialize expressions for table elements offsets or global variables //! is handled, according to the semantics of WebAssembly, to only specific expressions that are //! interpreted on the fly. -use translation_utils::{type_to_type, TableIndex, FunctionIndex, GlobalIndex, SignatureIndex, - MemoryIndex, Global, GlobalInit, Table, TableElementType, Memory}; -use cretonne::ir::{Signature, AbiParam, CallConv}; +use translation_utils::{type_to_type, FunctionIndex, Global, GlobalIndex, GlobalInit, Memory, + MemoryIndex, SignatureIndex, Table, TableElementType, TableIndex}; +use cretonne::ir::{AbiParam, CallConv, Signature}; use cretonne; -use wasmparser::{Parser, ParserState, FuncType, ImportSectionEntryType, ExternalKind, WasmDecoder, - MemoryType, Operator}; +use wasmparser::{ExternalKind, FuncType, ImportSectionEntryType, MemoryType, Operator, Parser, + ParserState, WasmDecoder}; use wasmparser; use std::str::from_utf8; use environ::ModuleEnvironment; diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index 5ced6d6056..8d1d86fe80 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -6,7 +6,7 @@ use cretonne::ir::{self, Ebb, Inst, Value}; use environ::{FuncEnvironment, GlobalValue}; use std::collections::HashMap; -use translation_utils::{GlobalIndex, MemoryIndex, SignatureIndex, FunctionIndex}; +use translation_utils::{FunctionIndex, GlobalIndex, MemoryIndex, SignatureIndex}; use std::vec::Vec; /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs index 5d969c9cc2..1bd6ecf3f9 100644 --- a/lib/wasm/tests/wasm_testsuite.rs +++ b/lib/wasm/tests/wasm_testsuite.rs @@ -1,5 +1,5 @@ -extern crate cton_wasm; extern crate cretonne; +extern crate cton_wasm; extern crate tempdir; use cton_wasm::{translate_module, DummyEnvironment}; From c50675deb8125f2bf4fea187765443c5e8f88885 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 30 Mar 2018 13:24:58 -0700 Subject: [PATCH 1674/3084] Format with nightly rustfmt-preview, then with rustfmt-0.9 again. --- cranelift/src/cat.rs | 2 +- cranelift/src/compile.rs | 6 ++-- cranelift/src/cton-util.rs | 4 +-- cranelift/src/rsfilecheck.rs | 2 +- cranelift/src/utils.rs | 2 +- cranelift/src/wasm.rs | 8 ++--- lib/cretonne/src/abi.rs | 2 +- lib/cretonne/src/bforest/map.rs | 6 ++-- lib/cretonne/src/bforest/node.rs | 4 +-- lib/cretonne/src/bforest/path.rs | 8 ++--- lib/cretonne/src/bforest/pool.rs | 6 ++-- lib/cretonne/src/bforest/set.rs | 6 ++-- lib/cretonne/src/binemit/memorysink.rs | 2 +- lib/cretonne/src/binemit/mod.rs | 6 ++-- lib/cretonne/src/bitset.rs | 2 +- lib/cretonne/src/context.rs | 14 ++++---- lib/cretonne/src/dbg.rs | 1 - lib/cretonne/src/dce.rs | 4 +-- lib/cretonne/src/divconst_magic_numbers.rs | 2 +- lib/cretonne/src/dominator_tree.rs | 8 ++--- lib/cretonne/src/entity/list.rs | 2 +- lib/cretonne/src/entity/map.rs | 2 +- lib/cretonne/src/entity/mod.rs | 8 ++--- lib/cretonne/src/entity/primary.rs | 2 +- lib/cretonne/src/flowgraph.rs | 4 +-- lib/cretonne/src/ir/builder.rs | 4 +-- lib/cretonne/src/ir/dfg.rs | 4 +-- lib/cretonne/src/ir/entities.rs | 4 +-- lib/cretonne/src/ir/function.rs | 2 +- lib/cretonne/src/ir/globalvar.rs | 2 +- lib/cretonne/src/ir/heap.rs | 2 +- lib/cretonne/src/ir/immediates.rs | 4 +-- lib/cretonne/src/ir/instructions.rs | 6 ++-- lib/cretonne/src/ir/jumptable.rs | 8 ++--- lib/cretonne/src/ir/layout.rs | 4 +-- lib/cretonne/src/ir/mod.rs | 20 ++++++------ lib/cretonne/src/ir/progpoint.rs | 2 +- lib/cretonne/src/ir/stackslot.rs | 4 +-- lib/cretonne/src/ir/valueloc.rs | 2 +- lib/cretonne/src/isa/arm32/abi.rs | 2 +- lib/cretonne/src/isa/arm32/mod.rs | 12 +++---- lib/cretonne/src/isa/arm64/abi.rs | 2 +- lib/cretonne/src/isa/arm64/mod.rs | 12 +++---- lib/cretonne/src/isa/constraints.rs | 2 +- lib/cretonne/src/isa/intel/abi.rs | 16 ++++----- lib/cretonne/src/isa/intel/binemit.rs | 4 +-- lib/cretonne/src/isa/intel/enc_tables.rs | 6 ++-- lib/cretonne/src/isa/intel/mod.rs | 14 ++++---- lib/cretonne/src/isa/mod.rs | 38 ++++++++++------------ lib/cretonne/src/isa/riscv/abi.rs | 4 +-- lib/cretonne/src/isa/riscv/enc_tables.rs | 2 +- lib/cretonne/src/isa/riscv/mod.rs | 14 ++++---- lib/cretonne/src/isa/stack.rs | 2 +- lib/cretonne/src/legalizer/boundary.rs | 2 +- lib/cretonne/src/legalizer/heap.rs | 2 +- lib/cretonne/src/legalizer/mod.rs | 2 +- lib/cretonne/src/licm.rs | 8 ++--- lib/cretonne/src/loop_analysis.rs | 8 ++--- lib/cretonne/src/postopt.rs | 6 ++-- lib/cretonne/src/preopt.rs | 10 +++--- lib/cretonne/src/print_errors.rs | 4 +-- lib/cretonne/src/regalloc/affinity.rs | 2 +- lib/cretonne/src/regalloc/coalescing.rs | 4 +-- lib/cretonne/src/regalloc/coloring.rs | 4 +-- lib/cretonne/src/regalloc/diversion.rs | 4 +-- lib/cretonne/src/regalloc/liverange.rs | 2 +- lib/cretonne/src/regalloc/mod.rs | 6 ++-- lib/cretonne/src/regalloc/pressure.rs | 4 +-- lib/cretonne/src/regalloc/reload.rs | 4 +-- lib/cretonne/src/regalloc/solver.rs | 6 ++-- lib/cretonne/src/regalloc/virtregs.rs | 2 +- lib/cretonne/src/result.rs | 2 +- lib/cretonne/src/settings.rs | 6 ++-- lib/cretonne/src/simple_gvn.rs | 2 +- lib/cretonne/src/stack_layout.rs | 4 +-- lib/cretonne/src/topo_order.rs | 6 ++-- lib/cretonne/src/verifier/flags.rs | 4 +-- lib/cretonne/src/verifier/liveness.rs | 4 +-- lib/cretonne/src/verifier/locations.rs | 2 +- lib/cretonne/src/verifier/mod.rs | 21 +++++++----- lib/cretonne/src/write.rs | 4 +-- lib/filetests/src/concurrent.rs | 2 +- lib/filetests/src/lib.rs | 6 ++-- lib/filetests/src/runner.rs | 4 +-- lib/filetests/src/runone.rs | 16 ++++----- lib/filetests/src/subtest.rs | 4 +-- lib/filetests/src/test_binemit.rs | 10 +++--- lib/filetests/src/test_cat.rs | 2 +- lib/filetests/src/test_compile.rs | 4 +-- lib/filetests/src/test_dce.rs | 4 +-- lib/filetests/src/test_domtree.rs | 4 +-- lib/filetests/src/test_legalizer.rs | 4 +-- lib/filetests/src/test_licm.rs | 4 +-- lib/filetests/src/test_postopt.rs | 4 +-- lib/filetests/src/test_preopt.rs | 4 +-- lib/filetests/src/test_print_cfg.rs | 2 +- lib/filetests/src/test_regalloc.rs | 4 +-- lib/filetests/src/test_simple_gvn.rs | 4 +-- lib/filetests/src/test_verifier.rs | 6 ++-- lib/frontend/src/frontend.rs | 18 +++++----- lib/frontend/src/ssa.rs | 18 +++++----- lib/reader/src/isaspec.rs | 2 +- lib/reader/src/lexer.rs | 8 ++--- lib/reader/src/lib.rs | 10 +++--- lib/reader/src/parser.rs | 34 +++++++++---------- lib/reader/src/testfile.rs | 4 +-- lib/wasm/src/code_translator.rs | 14 ++++---- lib/wasm/src/environ/dummy.rs | 16 ++++----- lib/wasm/src/environ/mod.rs | 4 +-- lib/wasm/src/environ/spec.rs | 6 ++-- lib/wasm/src/func_translator.rs | 6 ++-- lib/wasm/src/lib.rs | 4 +-- lib/wasm/src/module_translator.rs | 4 +-- lib/wasm/src/sections_translator.rs | 14 ++++---- lib/wasm/src/state.rs | 2 +- lib/wasm/src/translation_utils.rs | 2 +- lib/wasm/tests/wasm_testsuite.rs | 20 ++++++------ 117 files changed, 360 insertions(+), 360 deletions(-) diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs index 0fc5541db9..e82af42624 100644 --- a/cranelift/src/cat.rs +++ b/cranelift/src/cat.rs @@ -3,8 +3,8 @@ //! Read a sequence of Cretonne IR files and print them again to stdout. This has the effect of //! normalizing formatting and removing comments. -use cton_reader::parse_functions; use CommandResult; +use cton_reader::parse_functions; use utils::read_to_string; pub fn run(files: &[String]) -> CommandResult { diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index f2272494cf..6bab77e0d2 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -1,12 +1,12 @@ //! CLI tool to read Cretonne IR files and compile them into native code. -use cton_reader::parse_test; -use std::path::PathBuf; use cretonne::Context; +use cretonne::print_errors::pretty_error; use cretonne::settings::FlagsOrIsa; use cretonne::{binemit, ir}; -use cretonne::print_errors::pretty_error; +use cton_reader::parse_test; use std::path::Path; +use std::path::PathBuf; use utils::{parse_sets_and_isa, read_to_string}; struct PrintRelocs { diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 42887640df..3968030709 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -14,12 +14,12 @@ use docopt::Docopt; use std::io::{self, Write}; use std::process; -mod utils; mod cat; +mod compile; mod print_cfg; mod rsfilecheck; +mod utils; mod wasm; -mod compile; const USAGE: &str = " Cretonne code generator utility diff --git a/cranelift/src/rsfilecheck.rs b/cranelift/src/rsfilecheck.rs index f71571b513..6269ed226b 100644 --- a/cranelift/src/rsfilecheck.rs +++ b/cranelift/src/rsfilecheck.rs @@ -3,9 +3,9 @@ //! This file is named to avoid a name collision with the filecheck crate. use CommandResult; -use utils::read_to_string; use filecheck::{Checker, CheckerBuilder, NO_VARIABLES}; use std::io::{self, Read}; +use utils::read_to_string; pub fn run(files: &[String], verbose: bool) -> CommandResult { if files.is_empty() { diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index dc4f501893..8a2994a7ab 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -1,8 +1,8 @@ //! Utility functions. +use cretonne::isa; use cretonne::isa::TargetIsa; use cretonne::settings::{self, FlagsOrIsa}; -use cretonne::isa; use cton_reader::{parse_options, Location}; use std::fs::File; use std::io::{self, Read}; diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 918deffd5a..029c7ad5c0 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -3,15 +3,15 @@ //! Reads Wasm binary files, translates the functions' code to Cretonne IR. #![cfg_attr(feature = "cargo-clippy", allow(too_many_arguments, cyclomatic_complexity))] -use cton_wasm::{translate_module, DummyEnvironment, ModuleEnvironment}; -use std::path::PathBuf; use cretonne::Context; -use cretonne::settings::FlagsOrIsa; use cretonne::print_errors::{pretty_error, pretty_verifier_error}; -use std::fs::File; +use cretonne::settings::FlagsOrIsa; +use cton_wasm::{translate_module, DummyEnvironment, ModuleEnvironment}; use std::error::Error; +use std::fs::File; use std::io; use std::path::Path; +use std::path::PathBuf; use std::process::Command; use tempdir::TempDir; use term; diff --git a/lib/cretonne/src/abi.rs b/lib/cretonne/src/abi.rs index 2ba3f7b7a1..88a8a424a6 100644 --- a/lib/cretonne/src/abi.rs +++ b/lib/cretonne/src/abi.rs @@ -186,8 +186,8 @@ pub fn legalize_abi_value(have: Type, arg: &AbiParam) -> ValueConversion { #[cfg(test)] mod tests { use super::*; - use ir::types; use ir::AbiParam; + use ir::types; #[test] fn legalize() { diff --git a/lib/cretonne/src/bforest/map.rs b/lib/cretonne/src/bforest/map.rs index 23b4625555..6f8bedc225 100644 --- a/lib/cretonne/src/bforest/map.rs +++ b/lib/cretonne/src/bforest/map.rs @@ -1,8 +1,8 @@ //! Forest of maps. +use super::{Comparator, Forest, Node, NodeData, NodePool, Path, INNER_SIZE}; use packed_option::PackedOption; use std::marker::PhantomData; -use super::{Comparator, Forest, Node, NodeData, NodePool, Path, INNER_SIZE}; /// Tag type defining forest types for a map. struct MapTypes(PhantomData<(K, V, C)>); @@ -422,10 +422,10 @@ where #[cfg(test)] mod test { + use super::super::NodeData; + use super::*; use std::mem; use std::vec::Vec; - use super::*; - use super::super::NodeData; #[test] fn node_size() { diff --git a/lib/cretonne/src/bforest/node.rs b/lib/cretonne/src/bforest/node.rs index 61fb6e994d..f4a85613b5 100644 --- a/lib/cretonne/src/bforest/node.rs +++ b/lib/cretonne/src/bforest/node.rs @@ -1,8 +1,8 @@ //! B+-tree nodes. +use super::{slice_insert, slice_shift, Forest, Node, SetValue, INNER_SIZE}; use std::borrow::{Borrow, BorrowMut}; use std::fmt; -use super::{slice_insert, slice_shift, Forest, Node, SetValue, INNER_SIZE}; /// B+-tree node. /// @@ -579,9 +579,9 @@ where #[cfg(test)] mod test { + use super::*; use std::mem; use std::string::ToString; - use super::*; // Forest impl for a set implementation. struct TF(); diff --git a/lib/cretonne/src/bforest/path.rs b/lib/cretonne/src/bforest/path.rs index a2d15173a3..d011455571 100644 --- a/lib/cretonne/src/bforest/path.rs +++ b/lib/cretonne/src/bforest/path.rs @@ -1,9 +1,9 @@ //! A path from the root of a B+-tree to a leaf node. +use super::node::Removed; +use super::{slice_insert, slice_shift, Comparator, Forest, Node, NodeData, NodePool, MAX_PATH}; use std::borrow::Borrow; use std::marker::PhantomData; -use super::{slice_insert, slice_shift, Comparator, Forest, Node, NodeData, NodePool, MAX_PATH}; -use super::node::Removed; #[cfg(test)] use std::fmt; @@ -699,9 +699,9 @@ impl fmt::Display for Path { #[cfg(test)] mod test { - use std::cmp::Ordering; - use super::*; use super::super::{Forest, NodeData, NodePool}; + use super::*; + use std::cmp::Ordering; struct TC(); diff --git a/lib/cretonne/src/bforest/pool.rs b/lib/cretonne/src/bforest/pool.rs index 54a1f81cef..0e312f9380 100644 --- a/lib/cretonne/src/bforest/pool.rs +++ b/lib/cretonne/src/bforest/pool.rs @@ -1,8 +1,8 @@ //! B+-tree node pool. +use super::{Forest, Node, NodeData}; use entity::PrimaryMap; use std::ops::{Index, IndexMut}; -use super::{Forest, Node, NodeData}; /// A pool of nodes, including a free list. pub(super) struct NodePool { @@ -77,11 +77,11 @@ impl NodePool { NodeData: ::std::fmt::Display, F::Key: ::std::fmt::Display, { + use super::Comparator; + use entity::SparseSet; use std::borrow::Borrow; use std::cmp::Ordering; use std::vec::Vec; - use super::Comparator; - use entity::SparseSet; // The root node can't be an inner node with just a single sub-tree. It should have been // pruned. diff --git a/lib/cretonne/src/bforest/set.rs b/lib/cretonne/src/bforest/set.rs index 374c26ad2b..fbd0dbfca1 100644 --- a/lib/cretonne/src/bforest/set.rs +++ b/lib/cretonne/src/bforest/set.rs @@ -1,8 +1,8 @@ //! Forest of sets. +use super::{Comparator, Forest, Node, NodeData, NodePool, Path, SetValue, INNER_SIZE}; use packed_option::PackedOption; use std::marker::PhantomData; -use super::{Comparator, Forest, Node, NodeData, NodePool, Path, SetValue, INNER_SIZE}; /// Tag type defining forest types for a set. struct SetTypes(PhantomData<(K, C)>); @@ -350,10 +350,10 @@ where #[cfg(test)] mod test { + use super::super::NodeData; + use super::*; use std::mem; use std::vec::Vec; - use super::*; - use super::super::NodeData; #[test] fn node_size() { diff --git a/lib/cretonne/src/binemit/memorysink.rs b/lib/cretonne/src/binemit/memorysink.rs index 5a9fa8a298..bed34cd1ac 100644 --- a/lib/cretonne/src/binemit/memorysink.rs +++ b/lib/cretonne/src/binemit/memorysink.rs @@ -14,8 +14,8 @@ //! relocations to a `RelocSink` trait object. Relocations are less frequent than the //! `CodeSink::put*` methods, so the performance impact of the virtual callbacks is less severe. -use ir::{ExternalName, JumpTable, SourceLoc, TrapCode}; use super::{Addend, CodeOffset, CodeSink, Reloc}; +use ir::{ExternalName, JumpTable, SourceLoc, TrapCode}; use std::ptr::write_unaligned; /// A `CodeSink` that writes binary machine code directly into memory. diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 20ff429756..ea31abaae5 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -3,12 +3,12 @@ //! The `binemit` module contains code for translating Cretonne's intermediate representation into //! binary machine code. -mod relaxation; mod memorysink; +mod relaxation; -pub use regalloc::RegDiversions; -pub use self::relaxation::relax_branches; pub use self::memorysink::{MemoryCodeSink, RelocSink, TrapSink}; +pub use self::relaxation::relax_branches; +pub use regalloc::RegDiversions; use ir::{ExternalName, Function, Inst, JumpTable, SourceLoc, TrapCode}; use std::fmt; diff --git a/lib/cretonne/src/bitset.rs b/lib/cretonne/src/bitset.rs index f0a7271957..160c1c3e09 100644 --- a/lib/cretonne/src/bitset.rs +++ b/lib/cretonne/src/bitset.rs @@ -5,9 +5,9 @@ //! //! If you would like to add support for larger bitsets in the future, you need to change the trait //! bound Into and the u32 in the implementation of `max_bits()`. +use std::convert::{From, Into}; use std::mem::size_of; use std::ops::{Add, BitOr, Shl, Sub}; -use std::convert::{From, Into}; /// A small bitset built on a single primitive integer type #[derive(Clone, Copy, Debug, PartialEq, Eq)] diff --git a/lib/cretonne/src/context.rs b/lib/cretonne/src/context.rs index c7841f9366..ad31e9025b 100644 --- a/lib/cretonne/src/context.rs +++ b/lib/cretonne/src/context.rs @@ -10,23 +10,23 @@ //! single ISA instance. use binemit::{relax_branches, CodeOffset, MemoryCodeSink, RelocSink, TrapSink}; +use dce::do_dce; use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; use ir::Function; -use loop_analysis::LoopAnalysis; use isa::TargetIsa; use legalize_function; +use licm::do_licm; +use loop_analysis::LoopAnalysis; +use postopt::do_postopt; +use preopt::do_preopt; use regalloc; use result::{CtonError, CtonResult}; use settings::{FlagsOrIsa, OptLevel}; +use simple_gvn::do_simple_gvn; +use timing; use unreachable_code::eliminate_unreachable_code; use verifier; -use dce::do_dce; -use simple_gvn::do_simple_gvn; -use licm::do_licm; -use preopt::do_preopt; -use postopt::do_postopt; -use timing; /// Persistent data structures and compilation pipeline. pub struct Context { diff --git a/lib/cretonne/src/dbg.rs b/lib/cretonne/src/dbg.rs index 5acadd2098..0d9f77b76d 100644 --- a/lib/cretonne/src/dbg.rs +++ b/lib/cretonne/src/dbg.rs @@ -8,7 +8,6 @@ /// /// The output will appear in files named `cretonne.dbg.*`, where the suffix is named after the /// thread doing the logging. - use std::cell::RefCell; use std::env; use std::ffi::OsStr; diff --git a/lib/cretonne/src/dce.rs b/lib/cretonne/src/dce.rs index b78d260ecb..895b922dd7 100644 --- a/lib/cretonne/src/dce.rs +++ b/lib/cretonne/src/dce.rs @@ -6,10 +6,10 @@ use cursor::{Cursor, FuncCursor}; use dominator_tree::DominatorTree; use entity::EntityRef; -use ir::{DataFlowGraph, Function, Inst, Opcode}; use ir::instructions::InstructionData; -use timing; +use ir::{DataFlowGraph, Function, Inst, Opcode}; use std::vec::Vec; +use timing; /// Test whether the given opcode is unsafe to even consider for DCE. fn trivially_unsafe_for_dce(opcode: Opcode) -> bool { diff --git a/lib/cretonne/src/divconst_magic_numbers.rs b/lib/cretonne/src/divconst_magic_numbers.rs index 80a59aed8b..8a848711a4 100644 --- a/lib/cretonne/src/divconst_magic_numbers.rs +++ b/lib/cretonne/src/divconst_magic_numbers.rs @@ -220,8 +220,8 @@ pub fn magicS64(d: i64) -> MS64 { #[cfg(test)] mod tests { - use super::{magicS32, magicS64, magicU32, magicU64}; use super::{MS32, MS64, MU32, MU64}; + use super::{magicS32, magicS64, magicU32, magicU64}; fn mkMU32(mulBy: u32, doAdd: bool, shiftBy: i32) -> MU32 { MU32 { diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/cretonne/src/dominator_tree.rs index 574697c1ba..60ef1e6e42 100644 --- a/lib/cretonne/src/dominator_tree.rs +++ b/lib/cretonne/src/dominator_tree.rs @@ -2,14 +2,14 @@ use entity::EntityMap; use flowgraph::{BasicBlock, ControlFlowGraph}; -use ir::{Ebb, ExpandedProgramPoint, Function, Inst, Layout, ProgramOrder, Value}; use ir::instructions::BranchInfo; +use ir::{Ebb, ExpandedProgramPoint, Function, Inst, Layout, ProgramOrder, Value}; use packed_option::PackedOption; use std::cmp; -use std::mem; -use timing; use std::cmp::Ordering; +use std::mem; use std::vec::Vec; +use timing; /// RPO numbers are not first assigned in a contiguous way but as multiples of STRIDE, to leave /// room for modifications of the dominator tree. @@ -666,12 +666,12 @@ impl DominatorTreePreorder { #[cfg(test)] mod test { + use super::*; use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::types::*; use ir::{Function, InstBuilder, TrapCode}; use settings; - use super::*; use verifier::verify_context; #[test] diff --git a/lib/cretonne/src/entity/list.rs b/lib/cretonne/src/entity/list.rs index 2c743e169b..545b1e146b 100644 --- a/lib/cretonne/src/entity/list.rs +++ b/lib/cretonne/src/entity/list.rs @@ -481,8 +481,8 @@ impl EntityList { mod tests { use super::*; use super::{sclass_for_length, sclass_size}; - use ir::Inst; use entity::EntityRef; + use ir::Inst; #[test] fn size_classes() { diff --git a/lib/cretonne/src/entity/map.rs b/lib/cretonne/src/entity/map.rs index 88b355585b..ff402b104f 100644 --- a/lib/cretonne/src/entity/map.rs +++ b/lib/cretonne/src/entity/map.rs @@ -3,8 +3,8 @@ use entity::{EntityRef, Iter, IterMut, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; -use std::vec::Vec; use std::slice; +use std::vec::Vec; /// A mapping `K -> V` for densely indexed entity references. /// diff --git a/lib/cretonne/src/entity/mod.rs b/lib/cretonne/src/entity/mod.rs index 9aeab1449d..88c0977cf2 100644 --- a/lib/cretonne/src/entity/mod.rs +++ b/lib/cretonne/src/entity/mod.rs @@ -29,16 +29,16 @@ //! references allocated from an associated memory pool. It has a much smaller footprint than //! `Vec`. -mod keys; mod iter; +mod keys; mod list; mod map; mod primary; -mod sparse; mod set; +mod sparse; -pub use self::keys::Keys; pub use self::iter::{Iter, IterMut}; +pub use self::keys::Keys; pub use self::list::{EntityList, ListPool}; pub use self::map::EntityMap; pub use self::primary::PrimaryMap; @@ -95,5 +95,5 @@ macro_rules! entity_impl { (self as &::std::fmt::Display).fmt(f) } } - } + }; } diff --git a/lib/cretonne/src/entity/primary.rs b/lib/cretonne/src/entity/primary.rs index a442928d77..5d690a7553 100644 --- a/lib/cretonne/src/entity/primary.rs +++ b/lib/cretonne/src/entity/primary.rs @@ -2,8 +2,8 @@ use entity::{EntityRef, Iter, IterMut, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; -use std::vec::Vec; use std::slice; +use std::vec::Vec; /// A primary mapping `K -> V` allocating dense entity references. /// diff --git a/lib/cretonne/src/flowgraph.rs b/lib/cretonne/src/flowgraph.rs index 3ed295d6f0..66f377ce35 100644 --- a/lib/cretonne/src/flowgraph.rs +++ b/lib/cretonne/src/flowgraph.rs @@ -24,9 +24,9 @@ //! and `(Ebb0, jmp Ebb2)` respectively. use bforest; -use ir::{Ebb, Function, Inst}; -use ir::instructions::BranchInfo; use entity::EntityMap; +use ir::instructions::BranchInfo; +use ir::{Ebb, Function, Inst}; use std::mem; use timing; diff --git a/lib/cretonne/src/ir/builder.rs b/lib/cretonne/src/ir/builder.rs index 6308f3e1bd..34f56b9439 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/cretonne/src/ir/builder.rs @@ -216,9 +216,9 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> { #[cfg(test)] mod tests { use cursor::{Cursor, FuncCursor}; - use ir::{Function, InstBuilder, ValueDef}; - use ir::types::*; use ir::condcodes::*; + use ir::types::*; + use ir::{Function, InstBuilder, ValueDef}; #[test] fn types() { diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 48faf3b5ac..24f18ff332 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -1,20 +1,20 @@ //! Data flow graph tracking Instructions, Values, and EBBs. use entity::{EntityMap, PrimaryMap}; -use isa::{Encoding, Legalize, TargetIsa}; use ir; use ir::builder::ReplaceBuilder; use ir::extfunc::ExtFuncData; use ir::instructions::{BranchInfo, CallInfo, InstructionData}; use ir::types; use ir::{Ebb, FuncRef, Inst, SigRef, Signature, Type, Value, ValueList, ValueListPool}; +use isa::{Encoding, Legalize, TargetIsa}; use packed_option::ReservedValue; -use write::write_operands; use std::fmt; use std::iter; use std::mem; use std::ops::{Index, IndexMut}; use std::u16; +use write::write_operands; /// A data flow graph defines all instructions and extended basic blocks in a function as well as /// the data flow dependencies between them. The DFG also tracks values which can be either diff --git a/lib/cretonne/src/ir/entities.rs b/lib/cretonne/src/ir/entities.rs index 0727721d6a..d0c627952b 100644 --- a/lib/cretonne/src/ir/entities.rs +++ b/lib/cretonne/src/ir/entities.rs @@ -261,8 +261,8 @@ impl From for AnyEntity { #[cfg(test)] mod tests { use super::*; - use std::u32; use std::string::ToString; + use std::u32; #[test] fn value_with_number() { @@ -275,8 +275,8 @@ mod tests { #[test] fn memory() { - use std::mem; use packed_option::PackedOption; + use std::mem; // This is the whole point of `PackedOption`. assert_eq!( mem::size_of::(), diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 5010e0caae..19baa16b2b 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -7,9 +7,9 @@ use binemit::CodeOffset; use entity::{EntityMap, PrimaryMap}; use ir; use ir::{CallConv, DataFlowGraph, ExternalName, Layout, Signature}; -use ir::{EbbOffsets, InstEncodings, JumpTables, SourceLocs, StackSlots, ValueLocations}; use ir::{Ebb, ExtFuncData, FuncRef, GlobalVar, GlobalVarData, Heap, HeapData, JumpTable, JumpTableData, SigRef, StackSlot, StackSlotData}; +use ir::{EbbOffsets, InstEncodings, JumpTables, SourceLocs, StackSlots, ValueLocations}; use isa::{EncInfo, Legalize, TargetIsa}; use std::fmt; use write::write_function; diff --git a/lib/cretonne/src/ir/globalvar.rs b/lib/cretonne/src/ir/globalvar.rs index c8fc3b8ce1..228f691d84 100644 --- a/lib/cretonne/src/ir/globalvar.rs +++ b/lib/cretonne/src/ir/globalvar.rs @@ -1,7 +1,7 @@ //! Global variables. -use ir::{ExternalName, GlobalVar}; use ir::immediates::Offset32; +use ir::{ExternalName, GlobalVar}; use std::fmt; /// Information about a global variable declaration. diff --git a/lib/cretonne/src/ir/heap.rs b/lib/cretonne/src/ir/heap.rs index b41565b25b..01beb15b35 100644 --- a/lib/cretonne/src/ir/heap.rs +++ b/lib/cretonne/src/ir/heap.rs @@ -1,7 +1,7 @@ //! Heaps. -use ir::immediates::Imm64; use ir::GlobalVar; +use ir::immediates::Imm64; use std::fmt; /// Information about a heap declaration. diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/cretonne/src/ir/immediates.rs index 3dd09bd0bd..d56e963bb1 100644 --- a/lib/cretonne/src/ir/immediates.rs +++ b/lib/cretonne/src/ir/immediates.rs @@ -651,10 +651,10 @@ impl FromStr for Ieee64 { #[cfg(test)] mod tests { use super::*; - use std::{f32, f64}; - use std::str::FromStr; use std::fmt::Display; + use std::str::FromStr; use std::string::ToString; + use std::{f32, f64}; #[test] fn format_imm64() { diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/cretonne/src/ir/instructions.rs index 94293b517b..aa6a35df3f 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/cretonne/src/ir/instructions.rs @@ -7,17 +7,17 @@ //! directory. use std::fmt::{self, Display, Formatter}; -use std::str::FromStr; use std::ops::{Deref, DerefMut}; +use std::str::FromStr; use std::vec::Vec; use ir; -use ir::{Ebb, FuncRef, JumpTable, SigRef, Type, Value}; use ir::types; +use ir::{Ebb, FuncRef, JumpTable, SigRef, Type, Value}; use isa; -use entity; use bitset::BitSet; +use entity; use ref_slice::{ref_slice, ref_slice_mut}; /// Some instructions use an external list of argument values because there is not enough space in diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/cretonne/src/ir/jumptable.rs index 1c7bf79157..90b18b1bfe 100644 --- a/lib/cretonne/src/ir/jumptable.rs +++ b/lib/cretonne/src/ir/jumptable.rs @@ -3,11 +3,11 @@ //! Jump tables are declared in the preamble and assigned an `ir::entities::JumpTable` reference. //! The actual table of destinations is stored in a `JumpTableData` struct defined in this module. -use packed_option::PackedOption; use ir::entities::Ebb; +use packed_option::PackedOption; +use std::fmt::{self, Display, Formatter}; use std::iter; use std::slice; -use std::fmt::{self, Display, Formatter}; use std::vec::Vec; /// Contents of a jump table. @@ -140,10 +140,10 @@ impl Display for JumpTableData { #[cfg(test)] mod tests { use super::JumpTableData; - use ir::Ebb; use entity::EntityRef; - use std::vec::Vec; + use ir::Ebb; use std::string::ToString; + use std::vec::Vec; #[test] fn empty() { diff --git a/lib/cretonne/src/ir/layout.rs b/lib/cretonne/src/ir/layout.rs index 78db62530b..2e3ac89903 100644 --- a/lib/cretonne/src/ir/layout.rs +++ b/lib/cretonne/src/ir/layout.rs @@ -4,8 +4,8 @@ //! determined by the `Layout` data structure defined in this module. use entity::EntityMap; -use ir::{Ebb, Inst}; use ir::progpoint::{ExpandedProgramPoint, ProgramOrder}; +use ir::{Ebb, Inst}; use packed_option::PackedOption; use std::cmp; use std::iter::{IntoIterator, Iterator}; @@ -736,8 +736,8 @@ impl<'f> DoubleEndedIterator for Insts<'f> { #[cfg(test)] mod tests { - use cursor::{Cursor, CursorPosition}; use super::Layout; + use cursor::{Cursor, CursorPosition}; use entity::EntityRef; use ir::{Ebb, Inst, ProgramOrder, SourceLoc}; use std::cmp::Ordering; diff --git a/lib/cretonne/src/ir/mod.rs b/lib/cretonne/src/ir/mod.rs index 0d805cb3de..ed9771ccad 100644 --- a/lib/cretonne/src/ir/mod.rs +++ b/lib/cretonne/src/ir/mod.rs @@ -1,25 +1,25 @@ //! Representation of Cretonne IR functions. -pub mod types; -pub mod entities; -pub mod condcodes; -pub mod immediates; -pub mod instructions; -pub mod stackslot; -pub mod jumptable; -pub mod dfg; -pub mod layout; -pub mod function; mod builder; +pub mod condcodes; +pub mod dfg; +pub mod entities; mod extfunc; mod extname; +pub mod function; mod globalvar; mod heap; +pub mod immediates; +pub mod instructions; +pub mod jumptable; +pub mod layout; mod libcall; mod memflags; mod progpoint; mod sourceloc; +pub mod stackslot; mod trapcode; +pub mod types; mod valueloc; pub use ir::builder::{InsertBuilder, InstBuilder, InstBuilderBase, InstInserterBase}; diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/cretonne/src/ir/progpoint.rs index 6eab7ec743..4a0785aef4 100644 --- a/lib/cretonne/src/ir/progpoint.rs +++ b/lib/cretonne/src/ir/progpoint.rs @@ -2,9 +2,9 @@ use entity::EntityRef; use ir::{Ebb, Inst, ValueDef}; +use std::cmp; use std::fmt; use std::u32; -use std::cmp; /// A `ProgramPoint` represents a position in a function where the live range of an SSA value can /// begin or end. It can be either: diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index fd484915fd..9dfcfa7d21 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -8,8 +8,8 @@ use ir::{StackSlot, Type}; use packed_option::PackedOption; use std::cmp; use std::fmt; -use std::slice; use std::ops::{Index, IndexMut}; +use std::slice; use std::str::FromStr; use std::vec::Vec; @@ -338,9 +338,9 @@ impl StackSlots { #[cfg(test)] mod tests { + use super::*; use ir::Function; use ir::types; - use super::*; use std::string::ToString; #[test] diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/cretonne/src/ir/valueloc.rs index 1e3cf20f87..aecf23d0bb 100644 --- a/lib/cretonne/src/ir/valueloc.rs +++ b/lib/cretonne/src/ir/valueloc.rs @@ -3,8 +3,8 @@ //! The register allocator assigns every SSA value to either a register or a stack slot. This //! assignment is represented by a `ValueLoc` object. -use isa::{RegInfo, RegUnit}; use ir::StackSlot; +use isa::{RegInfo, RegUnit}; use std::fmt; /// Value location. diff --git a/lib/cretonne/src/isa/arm32/abi.rs b/lib/cretonne/src/isa/arm32/abi.rs index 020a525ac5..528a396570 100644 --- a/lib/cretonne/src/isa/arm32/abi.rs +++ b/lib/cretonne/src/isa/arm32/abi.rs @@ -1,10 +1,10 @@ //! ARM ABI implementation. +use super::registers::{D, GPR, Q, S}; use ir; use isa::RegClass; use regalloc::AllocatableSet; use settings as shared_settings; -use super::registers::{D, GPR, Q, S}; /// Legalize `sig`. pub fn legalize_signature( diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 11271fe3d9..350764a436 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -1,20 +1,20 @@ //! ARM 32-bit Instruction Set Architecture. -pub mod settings; mod abi; mod binemit; mod enc_tables; mod registers; +pub mod settings; -use binemit::{emit_function, CodeSink, MemoryCodeSink}; use super::super::settings as shared_settings; -use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; -use isa::Builder as IsaBuilder; -use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; +use binemit::{emit_function, CodeSink, MemoryCodeSink}; use ir; +use isa::Builder as IsaBuilder; +use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; +use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use regalloc; -use std::fmt; use std::boxed::Box; +use std::fmt; #[allow(dead_code)] struct Isa { diff --git a/lib/cretonne/src/isa/arm64/abi.rs b/lib/cretonne/src/isa/arm64/abi.rs index 9dba70c42f..0540746afa 100644 --- a/lib/cretonne/src/isa/arm64/abi.rs +++ b/lib/cretonne/src/isa/arm64/abi.rs @@ -1,10 +1,10 @@ //! ARM 64 ABI implementation. +use super::registers::{FPR, GPR}; use ir; use isa::RegClass; use regalloc::AllocatableSet; use settings as shared_settings; -use super::registers::{FPR, GPR}; /// Legalize `sig`. pub fn legalize_signature( diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 485b008574..3b63d56d54 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -1,20 +1,20 @@ //! ARM 64-bit Instruction Set Architecture. -pub mod settings; mod abi; mod binemit; mod enc_tables; mod registers; +pub mod settings; -use binemit::{emit_function, CodeSink, MemoryCodeSink}; use super::super::settings as shared_settings; -use isa::enc_tables::{lookup_enclist, Encodings}; -use isa::Builder as IsaBuilder; -use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; +use binemit::{emit_function, CodeSink, MemoryCodeSink}; use ir; +use isa::Builder as IsaBuilder; +use isa::enc_tables::{lookup_enclist, Encodings}; +use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use regalloc; -use std::fmt; use std::boxed::Box; +use std::fmt; #[allow(dead_code)] struct Isa { diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index 4ab22a6026..b08f635ba4 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -8,8 +8,8 @@ //! are satisfied. use binemit::CodeOffset; -use isa::{RegClass, RegUnit}; use ir::{Function, Inst, ValueLoc}; +use isa::{RegClass, RegUnit}; use regalloc::RegDiversions; /// Register constraint for a single value operand or instruction result. diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index 46ad7679e6..e9019291ef 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -1,18 +1,18 @@ //! Intel ABI implementation. -use ir; -use isa::{RegClass, RegUnit, TargetIsa}; -use regalloc::AllocatableSet; -use settings as shared_settings; use super::registers::{FPR, GPR, RU}; use abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion}; -use ir::{AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, CallConv, InstBuilder}; -use ir::stackslot::{StackOffset, StackSize}; +use cursor::{Cursor, CursorPosition, EncCursor}; +use ir; use ir::immediates::Imm64; +use ir::stackslot::{StackOffset, StackSize}; +use ir::{AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, CallConv, InstBuilder}; +use isa::{RegClass, RegUnit, TargetIsa}; +use regalloc::AllocatableSet; +use result; +use settings as shared_settings; use stack_layout::layout_stack; use std::i32; -use cursor::{Cursor, CursorPosition, EncCursor}; -use result; /// Argument registers for x86-64 static ARG_GPRS: [RU; 6] = [RU::rdi, RU::rsi, RU::rdx, RU::rcx, RU::r8, RU::r9]; diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/intel/binemit.rs index 5ec1d7175f..da14f80d29 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/intel/binemit.rs @@ -1,11 +1,11 @@ //! Emitting binary Intel machine code. +use super::registers::RU; use binemit::{bad_encoding, CodeSink, Reloc}; -use ir::{Ebb, Function, Inst, InstructionData, Opcode, TrapCode}; use ir::condcodes::{CondCode, FloatCC, IntCC}; +use ir::{Ebb, Function, Inst, InstructionData, Opcode, TrapCode}; use isa::{RegUnit, StackBase, StackBaseMask, StackRef}; use regalloc::RegDiversions; -use super::registers::RU; include!(concat!(env!("OUT_DIR"), "/binemit-intel.rs")); diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/intel/enc_tables.rs index b8461c0008..d847eb6d53 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/intel/enc_tables.rs @@ -1,16 +1,16 @@ //! Encoding tables for Intel ISAs. +use super::registers::*; use bitset::BitSet; use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; -use ir::{self, InstBuilder}; use ir::condcodes::IntCC; +use ir::{self, InstBuilder}; +use isa; use isa::constraints::*; use isa::enc_tables::*; use isa::encoding::RecipeSizing; -use isa; use predicates; -use super::registers::*; include!(concat!(env!("OUT_DIR"), "/encoding-intel.rs")); include!(concat!(env!("OUT_DIR"), "/legalize-intel.rs")); diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index aa6f2a0a01..91d8325a6b 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -1,22 +1,22 @@ //! Intel Instruction Set Architectures. -pub mod settings; mod abi; mod binemit; mod enc_tables; mod registers; +pub mod settings; -use binemit::{emit_function, CodeSink, MemoryCodeSink}; use super::super::settings as shared_settings; -use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; -use isa::Builder as IsaBuilder; -use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; +use binemit::{emit_function, CodeSink, MemoryCodeSink}; use ir; +use isa::Builder as IsaBuilder; +use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; +use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use regalloc; use result; -use timing; -use std::fmt; use std::boxed::Box; +use std::fmt; +use timing; #[allow(dead_code)] struct Isa { diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 8c141126af..39d7d4eb60 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -47,14 +47,14 @@ pub use isa::stack::{StackBase, StackBaseMask, StackRef}; use binemit; use flowgraph; -use settings; use ir; +use isa::enc_tables::Encodings; use regalloc; use result; -use timing; -use isa::enc_tables::Encodings; -use std::fmt; +use settings; use std::boxed::Box; +use std::fmt; +use timing; #[cfg(build_riscv)] mod riscv; @@ -68,28 +68,26 @@ mod arm32; #[cfg(build_arm64)] mod arm64; -pub mod registers; -mod encoding; -mod enc_tables; mod constraints; +mod enc_tables; +mod encoding; +pub mod registers; mod stack; /// Returns a builder that can create a corresponding `TargetIsa` /// or `Err(LookupError::Unsupported)` if not enabled. macro_rules! isa_builder { - ($module:ident, $name:ident) => { - { - #[cfg($name)] - fn $name() -> Result { - Ok($module::isa_builder()) - }; - #[cfg(not($name))] - fn $name() -> Result { - Err(LookupError::Unsupported) - } - $name() + ($module:ident, $name:ident) => {{ + #[cfg($name)] + fn $name() -> Result { + Ok($module::isa_builder()) + }; + #[cfg(not($name))] + fn $name() -> Result { + Err(LookupError::Unsupported) } - }; + $name() + }}; } /// Look for a supported ISA with the given `name`. @@ -248,8 +246,8 @@ pub trait TargetIsa: fmt::Display { fn prologue_epilogue(&self, func: &mut ir::Function) -> result::CtonResult { let _tt = timing::prologue_epilogue(); // This default implementation is unlikely to be good enough. - use stack_layout::layout_stack; use ir::stackslot::{StackOffset, StackSize}; + use stack_layout::layout_stack; let word_size = if self.flags().is_64bit() { 8 } else { 4 }; diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index 4eff90c954..0782fa5e53 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -5,13 +5,13 @@ //! //! This doesn't support the soft-float ABI at the moment. +use super::registers::{FPR, GPR}; +use super::settings; use abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion}; use ir::{self, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, Type}; use isa::RegClass; use regalloc::AllocatableSet; use settings as shared_settings; -use super::registers::{FPR, GPR}; -use super::settings; use std::i32; struct Args { diff --git a/lib/cretonne/src/isa/riscv/enc_tables.rs b/lib/cretonne/src/isa/riscv/enc_tables.rs index 1f0f2e132d..46b2458b49 100644 --- a/lib/cretonne/src/isa/riscv/enc_tables.rs +++ b/lib/cretonne/src/isa/riscv/enc_tables.rs @@ -1,12 +1,12 @@ //! Encoding tables for RISC-V. +use super::registers::*; use ir; use isa; use isa::constraints::*; use isa::enc_tables::*; use isa::encoding::RecipeSizing; use predicates; -use super::registers::*; // Include the generated encoding tables: // - `LEVEL1_RV32` diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 63a3c01088..9c3a498d5e 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -1,20 +1,20 @@ //! RISC-V Instruction Set Architecture. -pub mod settings; mod abi; mod binemit; mod enc_tables; mod registers; +pub mod settings; use super::super::settings as shared_settings; use binemit::{emit_function, CodeSink, MemoryCodeSink}; -use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; -use isa::Builder as IsaBuilder; -use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use ir; +use isa::Builder as IsaBuilder; +use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; +use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use regalloc; -use std::fmt; use std::boxed::Box; +use std::fmt; #[allow(dead_code)] struct Isa { @@ -113,10 +113,10 @@ impl TargetIsa for Isa { #[cfg(test)] mod tests { - use settings::{self, Configurable}; - use isa; use ir::{DataFlowGraph, InstructionData, Opcode}; use ir::{immediates, types}; + use isa; + use settings::{self, Configurable}; use std::string::{String, ToString}; fn encstr(isa: &isa::TargetIsa, enc: Result) -> String { diff --git a/lib/cretonne/src/isa/stack.rs b/lib/cretonne/src/isa/stack.rs index 7eac43b62c..0db6279d3f 100644 --- a/lib/cretonne/src/isa/stack.rs +++ b/lib/cretonne/src/isa/stack.rs @@ -4,8 +4,8 @@ //! defined in this module expresses the low-level details of accessing a stack slot from an //! encoded instruction. -use ir::stackslot::{StackOffset, StackSlotKind, StackSlots}; use ir::StackSlot; +use ir::stackslot::{StackOffset, StackSlotKind, StackSlots}; /// A method for referencing a stack slot in the current stack frame. /// diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/cretonne/src/legalizer/boundary.rs index 6cad2775e5..61f07884ca 100644 --- a/lib/cretonne/src/legalizer/boundary.rs +++ b/lib/cretonne/src/legalizer/boundary.rs @@ -20,9 +20,9 @@ use abi::{legalize_abi_value, ValueConversion}; use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; +use ir::instructions::CallInfo; use ir::{AbiParam, ArgumentLoc, ArgumentPurpose, DataFlowGraph, Ebb, Function, Inst, InstBuilder, SigRef, Signature, Type, Value, ValueLoc}; -use ir::instructions::CallInfo; use isa::TargetIsa; use legalizer::split::{isplit, vsplit}; use std::vec::Vec; diff --git a/lib/cretonne/src/legalizer/heap.rs b/lib/cretonne/src/legalizer/heap.rs index 0d6e7d385b..b3a595833f 100644 --- a/lib/cretonne/src/legalizer/heap.rs +++ b/lib/cretonne/src/legalizer/heap.rs @@ -5,8 +5,8 @@ use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; -use ir::{self, InstBuilder, MemFlags}; use ir::condcodes::IntCC; +use ir::{self, InstBuilder, MemFlags}; use isa::TargetIsa; /// Expand a `heap_addr` instruction according to the definition of the heap. diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index d434d6b6e5..23b3f56c89 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -13,11 +13,11 @@ //! The legalizer does not deal with register allocation constraints. These constraints are derived //! from the encoding recipes, and solved later by the register allocator. +use bitset::BitSet; use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::{self, InstBuilder}; use isa::TargetIsa; -use bitset::BitSet; use timing; mod boundary; diff --git a/lib/cretonne/src/licm.rs b/lib/cretonne/src/licm.rs index 546fc01bd8..b5a8f5ed68 100644 --- a/lib/cretonne/src/licm.rs +++ b/lib/cretonne/src/licm.rs @@ -1,14 +1,14 @@ //! A Loop Invariant Code Motion optimization pass use cursor::{Cursor, FuncCursor}; -use ir::{DataFlowGraph, Ebb, Function, Inst, InstBuilder, Layout, Opcode, Type, Value}; -use flowgraph::ControlFlowGraph; -use std::collections::HashSet; use dominator_tree::DominatorTree; use entity::{EntityList, ListPool}; +use flowgraph::ControlFlowGraph; +use ir::{DataFlowGraph, Ebb, Function, Inst, InstBuilder, Layout, Opcode, Type, Value}; use loop_analysis::{Loop, LoopAnalysis}; -use timing; +use std::collections::HashSet; use std::vec::Vec; +use timing; /// Performs the LICM pass by detecting loops within the CFG and moving /// loop-invariant instructions out of them. diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/cretonne/src/loop_analysis.rs index 89d2939ac9..e75f5f26fe 100644 --- a/lib/cretonne/src/loop_analysis.rs +++ b/lib/cretonne/src/loop_analysis.rs @@ -2,13 +2,13 @@ //! and parent in the loop tree. use dominator_tree::DominatorTree; -use entity::{Keys, PrimaryMap}; use entity::EntityMap; +use entity::{Keys, PrimaryMap}; use flowgraph::ControlFlowGraph; use ir::{Ebb, Function, Layout}; use packed_option::PackedOption; -use timing; use std::vec::Vec; +use timing; /// A opaque reference to a code loop. #[derive(Copy, Clone, PartialEq, Eq, Hash)] @@ -226,10 +226,10 @@ impl LoopAnalysis { mod test { use cursor::{Cursor, FuncCursor}; + use dominator_tree::DominatorTree; + use flowgraph::ControlFlowGraph; use ir::{types, Function, InstBuilder}; use loop_analysis::{Loop, LoopAnalysis}; - use flowgraph::ControlFlowGraph; - use dominator_tree::DominatorTree; use std::vec::Vec; #[test] diff --git a/lib/cretonne/src/postopt.rs b/lib/cretonne/src/postopt.rs index efe85d40d7..18f467a68a 100644 --- a/lib/cretonne/src/postopt.rs +++ b/lib/cretonne/src/postopt.rs @@ -3,11 +3,11 @@ #![allow(non_snake_case)] use cursor::{Cursor, EncCursor}; -use ir::dfg::ValueDef; -use ir::{Ebb, Function, Inst, InstBuilder, InstructionData, Value}; use ir::condcodes::{CondCode, FloatCC, IntCC}; -use ir::instructions::{Opcode, ValueList}; +use ir::dfg::ValueDef; use ir::immediates::Imm64; +use ir::instructions::{Opcode, ValueList}; +use ir::{Ebb, Function, Inst, InstBuilder, InstructionData, Value}; use isa::TargetIsa; use timing; diff --git a/lib/cretonne/src/preopt.rs b/lib/cretonne/src/preopt.rs index 81e798e987..bdd1c445eb 100644 --- a/lib/cretonne/src/preopt.rs +++ b/lib/cretonne/src/preopt.rs @@ -3,13 +3,13 @@ #![allow(non_snake_case)] use cursor::{Cursor, FuncCursor}; -use ir::dfg::ValueDef; -use ir::{DataFlowGraph, Function, InstBuilder, InstructionData, Type, Value}; -use ir::Inst; -use ir::types::{I32, I64}; -use ir::instructions::Opcode; use divconst_magic_numbers::{MS32, MS64, MU32, MU64}; use divconst_magic_numbers::{magicS32, magicS64, magicU32, magicU64}; +use ir::Inst; +use ir::dfg::ValueDef; +use ir::instructions::Opcode; +use ir::types::{I32, I64}; +use ir::{DataFlowGraph, Function, InstBuilder, InstructionData, Type, Value}; use timing; //---------------------------------------------------------------------- diff --git a/lib/cretonne/src/print_errors.rs b/lib/cretonne/src/print_errors.rs index 1bc61cfb59..917f860ce9 100644 --- a/lib/cretonne/src/print_errors.rs +++ b/lib/cretonne/src/print_errors.rs @@ -1,10 +1,10 @@ //! Utility routines for pretty-printing error messages. use ir; -use verifier; -use result::CtonError; use isa::TargetIsa; +use result::CtonError; use std::fmt::Write; +use verifier; /// Pretty-print a verifier error. pub fn pretty_verifier_error( diff --git a/lib/cretonne/src/regalloc/affinity.rs b/lib/cretonne/src/regalloc/affinity.rs index 0e49ac5f0b..e0078579a7 100644 --- a/lib/cretonne/src/regalloc/affinity.rs +++ b/lib/cretonne/src/regalloc/affinity.rs @@ -8,9 +8,9 @@ //! subclass. This is just a hint, and the register allocator is allowed to pick a register from a //! larger register class instead. -use std::fmt; use ir::{AbiParam, ArgumentLoc}; use isa::{ConstraintKind, OperandConstraint, RegClassIndex, RegInfo, TargetIsa}; +use std::fmt; /// Preferred register allocation for an SSA value. #[derive(Clone, Copy, Debug)] diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/cretonne/src/regalloc/coalescing.rs index b8a7270d59..b77488fec7 100644 --- a/lib/cretonne/src/regalloc/coalescing.rs +++ b/lib/cretonne/src/regalloc/coalescing.rs @@ -11,15 +11,15 @@ use dominator_tree::{DominatorTree, DominatorTreePreorder}; use flowgraph::ControlFlowGraph; use ir::{self, InstBuilder, ProgramOrder}; use ir::{Ebb, ExpandedProgramPoint, Function, Inst, Value}; +use isa::{EncInfo, TargetIsa}; use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; use regalloc::virtregs::{VirtReg, VirtRegs}; use std::cmp; -use std::iter; use std::fmt; +use std::iter; use std::slice; use std::vec::Vec; -use isa::{EncInfo, TargetIsa}; use timing; // # Implementation diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index ad5952885d..b9b1512220 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -44,10 +44,10 @@ use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; -use ir::{Ebb, Function, Inst, Layout, SigRef, Value, ValueLoc}; use ir::{AbiParam, ArgumentLoc, InstBuilder, ValueDef}; -use isa::{regs_overlap, RegClass, RegInfo, RegUnit}; +use ir::{Ebb, Function, Inst, Layout, SigRef, Value, ValueLoc}; use isa::{ConstraintKind, EncInfo, OperandConstraint, RecipeConstraints, TargetIsa}; +use isa::{regs_overlap, RegClass, RegInfo, RegUnit}; use packed_option::PackedOption; use regalloc::RegDiversions; use regalloc::affinity::Affinity; diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/cretonne/src/regalloc/diversion.rs index d0b7e4db91..b848a879c9 100644 --- a/lib/cretonne/src/regalloc/diversion.rs +++ b/lib/cretonne/src/regalloc/diversion.rs @@ -7,8 +7,8 @@ //! These register diversions are local to an EBB. No values can be diverted when entering a new //! EBB. -use ir::{StackSlot, Value, ValueLoc, ValueLocations}; use ir::{InstructionData, Opcode}; +use ir::{StackSlot, Value, ValueLoc, ValueLocations}; use isa::{RegInfo, RegUnit}; use std::fmt; use std::vec::Vec; @@ -187,8 +187,8 @@ impl<'a> fmt::Display for DisplayDiversions<'a> { #[cfg(test)] mod tests { use super::*; - use ir::Value; use entity::EntityRef; + use ir::Value; #[test] fn inserts() { diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/cretonne/src/regalloc/liverange.rs index 79526c613d..18118e706d 100644 --- a/lib/cretonne/src/regalloc/liverange.rs +++ b/lib/cretonne/src/regalloc/liverange.rs @@ -457,8 +457,8 @@ impl SparseMapValue for GenLiveRange { mod tests { use super::{GenLiveRange, LiveRangeContext}; use bforest; - use ir::{Ebb, Inst, Value}; use entity::EntityRef; + use ir::{Ebb, Inst, Value}; use ir::{ExpandedProgramPoint, ProgramOrder}; use std::cmp::Ordering; use std::vec::Vec; diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index 6ae93c67b6..6868c0a235 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -2,11 +2,11 @@ //! //! This module contains data structures and algorithms used for register allocation. -pub mod liverange; -pub mod liveness; pub mod allocatable_set; -pub mod live_value_tracker; pub mod coloring; +pub mod live_value_tracker; +pub mod liveness; +pub mod liverange; pub mod virtregs; mod affinity; diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index 6bbe2b75a5..14978f48b8 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -269,16 +269,16 @@ impl fmt::Display for Pressure { #[cfg(test)] #[cfg(build_arm32)] mod tests { + use super::Pressure; use isa::{RegClass, TargetIsa}; use regalloc::AllocatableSet; use std::borrow::Borrow; - use super::Pressure; use std::boxed::Box; // Make an arm32 `TargetIsa`, if possible. fn arm32() -> Option> { - use settings; use isa; + use settings; let shared_builder = settings::builder(); let shared_flags = settings::Flags::new(&shared_builder); diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/cretonne/src/regalloc/reload.rs index 2ce9a04f75..df527fdbca 100644 --- a/lib/cretonne/src/regalloc/reload.rs +++ b/lib/cretonne/src/regalloc/reload.rs @@ -12,16 +12,16 @@ use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; use entity::{SparseMap, SparseMapValue}; -use ir::{Ebb, Function, Inst, Value}; use ir::{AbiParam, ArgumentLoc, InstBuilder}; +use ir::{Ebb, Function, Inst, Value}; use isa::RegClass; use isa::{ConstraintKind, EncInfo, Encoding, RecipeConstraints, TargetIsa}; use regalloc::affinity::Affinity; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; +use std::vec::Vec; use timing; use topo_order::TopoOrder; -use std::vec::Vec; /// Reusable data structures for the reload pass. pub struct Reload { diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 92b7a62679..4bac41aa99 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -98,6 +98,7 @@ //! appropriate candidate among the set of live register values, add it as a variable and start //! over. +use super::AllocatableSet; use dbg::DisplayList; use entity::{SparseMap, SparseMapValue}; use ir::Value; @@ -106,7 +107,6 @@ use regalloc::allocatable_set::RegSetIter; use std::cmp; use std::fmt; use std::mem; -use super::AllocatableSet; use std::u16; use std::vec::Vec; @@ -1158,17 +1158,17 @@ impl fmt::Display for Solver { #[cfg(test)] #[cfg(build_arm32)] mod tests { + use super::{Move, Solver}; use entity::EntityRef; use ir::Value; use isa::{RegClass, RegInfo, RegUnit, TargetIsa}; use regalloc::AllocatableSet; - use super::{Move, Solver}; use std::boxed::Box; // Make an arm32 `TargetIsa`, if possible. fn arm32() -> Option> { - use settings; use isa; + use settings; let shared_builder = settings::builder(); let shared_flags = settings::Flags::new(&shared_builder); diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/cretonne/src/regalloc/virtregs.rs index 4583effe4f..215e5efac2 100644 --- a/lib/cretonne/src/regalloc/virtregs.rs +++ b/lib/cretonne/src/regalloc/virtregs.rs @@ -13,9 +13,9 @@ use dbg::DisplayList; use dominator_tree::DominatorTreePreorder; +use entity::EntityRef; use entity::{EntityList, ListPool}; use entity::{EntityMap, Keys, PrimaryMap}; -use entity::EntityRef; use ir::{Function, Value}; use packed_option::PackedOption; use ref_slice::ref_slice; diff --git a/lib/cretonne/src/result.rs b/lib/cretonne/src/result.rs index bd2ae5a101..04e248dcbb 100644 --- a/lib/cretonne/src/result.rs +++ b/lib/cretonne/src/result.rs @@ -1,8 +1,8 @@ //! Result and error types representing the outcome of compiling a function. -use verifier; use std::error::Error as StdError; use std::fmt; +use verifier; /// A compilation error. /// diff --git a/lib/cretonne/src/settings.rs b/lib/cretonne/src/settings.rs index ed1d8cd168..aee441f3ca 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/cretonne/src/settings.rs @@ -190,8 +190,8 @@ impl<'a> PredicateView<'a> { /// This module holds definitions that need to be public so the can be instantiated by generated /// code in other modules. pub mod detail { - use std::fmt; use constant_hash; + use std::fmt; /// An instruction group template. pub struct Template { @@ -345,9 +345,9 @@ impl<'a> From<&'a TargetIsa> for FlagsOrIsa<'a> { #[cfg(test)] mod tests { - use super::{builder, Flags}; - use super::Error::*; use super::Configurable; + use super::Error::*; + use super::{builder, Flags}; use std::string::ToString; #[test] diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/cretonne/src/simple_gvn.rs index 073101cd9c..ae8dd387d6 100644 --- a/lib/cretonne/src/simple_gvn.rs +++ b/lib/cretonne/src/simple_gvn.rs @@ -4,8 +4,8 @@ use cursor::{Cursor, FuncCursor}; use dominator_tree::DominatorTree; use ir::{Function, Inst, InstructionData, Opcode, Type}; use scoped_hash_map::ScopedHashMap; -use timing; use std::vec::Vec; +use timing; /// Test whether the given opcode is unsafe to even consider for GVN. fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { diff --git a/lib/cretonne/src/stack_layout.rs b/lib/cretonne/src/stack_layout.rs index 9d723791d7..3b07ee154a 100644 --- a/lib/cretonne/src/stack_layout.rs +++ b/lib/cretonne/src/stack_layout.rs @@ -110,10 +110,10 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result Verifier<'a> { #[cfg(test)] mod tests { use super::{Error, Verifier}; + use entity::EntityList; use ir::Function; use ir::instructions::{InstructionData, Opcode}; - use entity::EntityList; use settings; macro_rules! assert_err_with_msg { - ($e:expr, $msg:expr) => ( + ($e:expr, $msg:expr) => { match $e { - Ok(_) => { panic!("Expected an error!") }, - Err(Error { message, .. } ) => { + Ok(_) => panic!("Expected an error!"), + Err(Error { message, .. }) => { if !message.contains($msg) { - panic!(format!("'{}' did not contain the substring '{}'", message, $msg)); + panic!(format!( + "'{}' did not contain the substring '{}'", + message, $msg + )); } } } - ) + }; } #[test] diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index dee3ab0346..2ed8da87e9 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -5,9 +5,9 @@ use ir::{DataFlowGraph, Ebb, Function, Inst, SigRef, Type, Value, ValueDef}; use isa::{RegInfo, TargetIsa}; +use packed_option::ReservedValue; use std::fmt::{self, Error, Result, Write}; use std::result; -use packed_option::ReservedValue; use std::string::String; /// Write `func` to `w` as equivalent text. @@ -451,8 +451,8 @@ impl<'a> fmt::Display for DisplayValues<'a> { #[cfg(test)] mod tests { - use ir::{ExternalName, Function, StackSlotData, StackSlotKind}; use ir::types; + use ir::{ExternalName, Function, StackSlotData, StackSlotKind}; use std::string::ToString; #[test] diff --git a/lib/filetests/src/concurrent.rs b/lib/filetests/src/concurrent.rs index a93d39cddf..7ca4b01d89 100644 --- a/lib/filetests/src/concurrent.rs +++ b/lib/filetests/src/concurrent.rs @@ -4,13 +4,13 @@ //! concurrently. use cretonne::timing; +use num_cpus; use std::panic::catch_unwind; use std::path::{Path, PathBuf}; use std::sync::mpsc::{channel, Receiver, Sender}; use std::sync::{Arc, Mutex}; use std::thread; use std::time::Duration; -use num_cpus; use {runone, TestResult}; /// Request sent to worker threads contains jobid and path. diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs index 66bb7023ec..718502eb53 100644 --- a/lib/filetests/src/lib.rs +++ b/lib/filetests/src/lib.rs @@ -14,16 +14,16 @@ extern crate cton_reader; extern crate filecheck; extern crate num_cpus; -use std::path::Path; -use std::time; use cton_reader::TestCommand; use runner::TestRunner; +use std::path::Path; +use std::time; mod concurrent; +mod match_directive; mod runner; mod runone; mod subtest; -mod match_directive; mod test_binemit; mod test_cat; diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs index f9e8fb3638..d4865b92e9 100644 --- a/lib/filetests/src/runner.rs +++ b/lib/filetests/src/runner.rs @@ -3,13 +3,13 @@ //! This module implements the `TestRunner` struct which manages executing tests as well as //! scanning directories for tests. +use concurrent::{ConcurrentRunner, Reply}; use std::error::Error; -use std::fmt::{self, Display}; use std::ffi::OsStr; +use std::fmt::{self, Display}; use std::path::{Path, PathBuf}; use std::time; use {runone, TestResult}; -use concurrent::{ConcurrentRunner, Reply}; /// Timeout in seconds when we're not making progress. const TIMEOUT_PANIC: usize = 10; diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs index 4b3033ea2e..8cdec2d626 100644 --- a/lib/filetests/src/runone.rs +++ b/lib/filetests/src/runone.rs @@ -1,20 +1,20 @@ //! Run the tests in a single test file. -use std::borrow::Cow; -use std::path::Path; -use std::time; -use std::io::{self, Read}; -use std::fs; use cretonne::ir::Function; use cretonne::isa::TargetIsa; +use cretonne::print_errors::pretty_verifier_error; use cretonne::settings::Flags; use cretonne::timing; use cretonne::verify_function; -use cretonne::print_errors::pretty_verifier_error; -use cton_reader::parse_test; use cton_reader::IsaSpec; -use {new_subtest, TestResult}; +use cton_reader::parse_test; +use std::borrow::Cow; +use std::fs; +use std::io::{self, Read}; +use std::path::Path; +use std::time; use subtest::{Context, Result, SubTest}; +use {new_subtest, TestResult}; /// Read an entire file into a string. fn read_to_string>(path: P) -> io::Result { diff --git a/lib/filetests/src/subtest.rs b/lib/filetests/src/subtest.rs index 4395780021..ba94f87e78 100644 --- a/lib/filetests/src/subtest.rs +++ b/lib/filetests/src/subtest.rs @@ -1,12 +1,12 @@ //! `SubTest` trait. -use std::result; -use std::borrow::Cow; use cretonne::ir::Function; use cretonne::isa::TargetIsa; use cretonne::settings::{Flags, FlagsOrIsa}; use cton_reader::{Comment, Details}; use filecheck::{Checker, CheckerBuilder, NO_VARIABLES}; +use std::borrow::Cow; +use std::result; pub type Result = result::Result; diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index db3d9be634..775e014600 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -3,18 +3,18 @@ //! The `binemit` test command generates binary machine code for every instruction in the input //! functions and compares the results to the expected output. -use std::borrow::Cow; -use std::collections::HashMap; -use std::fmt::Write; use cretonne::binemit; +use cretonne::binemit::RegDiversions; use cretonne::dbg::DisplayList; use cretonne::ir; use cretonne::ir::entities::AnyEntity; -use cretonne::binemit::RegDiversions; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{Context, Result, SubTest}; use match_directive::match_directive; +use std::borrow::Cow; +use std::collections::HashMap; +use std::fmt::Write; +use subtest::{Context, Result, SubTest}; struct TestBinEmit; diff --git a/lib/filetests/src/test_cat.rs b/lib/filetests/src/test_cat.rs index 42b18b3625..f8f145834c 100644 --- a/lib/filetests/src/test_cat.rs +++ b/lib/filetests/src/test_cat.rs @@ -1,8 +1,8 @@ //! The `cat` subtest. -use std::borrow::Cow; use cretonne::ir::Function; use cton_reader::TestCommand; +use std::borrow::Cow; use subtest::{self, Context, Result as STResult, SubTest}; /// Object implementing the `test cat` sub-test. diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs index 5217f7ebb6..a924eee799 100644 --- a/lib/filetests/src/test_compile.rs +++ b/lib/filetests/src/test_compile.rs @@ -2,14 +2,14 @@ //! //! The `compile` test command runs each function through the full code generator pipeline +use cretonne; use cretonne::binemit; use cretonne::ir; -use cretonne; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; +use subtest::{run_filecheck, Context, Result, SubTest}; struct TestCompile; diff --git a/lib/filetests/src/test_dce.rs b/lib/filetests/src/test_dce.rs index 4e826c9abc..d96214182b 100644 --- a/lib/filetests/src/test_dce.rs +++ b/lib/filetests/src/test_dce.rs @@ -5,13 +5,13 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne::ir::Function; use cretonne; +use cretonne::ir::Function; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; +use subtest::{run_filecheck, Context, Result, SubTest}; struct TestDCE; diff --git a/lib/filetests/src/test_domtree.rs b/lib/filetests/src/test_domtree.rs index fed38d9d50..da5a926f3d 100644 --- a/lib/filetests/src/test_domtree.rs +++ b/lib/filetests/src/test_domtree.rs @@ -17,12 +17,12 @@ use cretonne::flowgraph::ControlFlowGraph; use cretonne::ir::Function; use cretonne::ir::entities::AnyEntity; use cton_reader::TestCommand; -use subtest::{run_filecheck, Context, Result, SubTest}; +use match_directive::match_directive; use std::borrow::{Borrow, Cow}; use std::collections::HashMap; use std::fmt::{self, Write}; use std::result; -use match_directive::match_directive; +use subtest::{run_filecheck, Context, Result, SubTest}; struct TestDomtree; diff --git a/lib/filetests/src/test_legalizer.rs b/lib/filetests/src/test_legalizer.rs index 76bd9f491b..0794081e51 100644 --- a/lib/filetests/src/test_legalizer.rs +++ b/lib/filetests/src/test_legalizer.rs @@ -3,13 +3,13 @@ //! The `test legalizer` test command runs each function through `legalize_function()` and sends //! the result to filecheck. -use std::borrow::Cow; use cretonne; use cretonne::ir::Function; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{run_filecheck, Context, Result, SubTest}; +use std::borrow::Cow; use std::fmt::Write; +use subtest::{run_filecheck, Context, Result, SubTest}; struct TestLegalizer; diff --git a/lib/filetests/src/test_licm.rs b/lib/filetests/src/test_licm.rs index bed36b44c2..c59dee1cba 100644 --- a/lib/filetests/src/test_licm.rs +++ b/lib/filetests/src/test_licm.rs @@ -5,13 +5,13 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne::ir::Function; use cretonne; +use cretonne::ir::Function; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; +use subtest::{run_filecheck, Context, Result, SubTest}; struct TestLICM; diff --git a/lib/filetests/src/test_postopt.rs b/lib/filetests/src/test_postopt.rs index ad26a06c30..c89fb33ea5 100644 --- a/lib/filetests/src/test_postopt.rs +++ b/lib/filetests/src/test_postopt.rs @@ -2,13 +2,13 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne::ir::Function; use cretonne; +use cretonne::ir::Function; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; +use subtest::{run_filecheck, Context, Result, SubTest}; struct TestPostopt; diff --git a/lib/filetests/src/test_preopt.rs b/lib/filetests/src/test_preopt.rs index 8a30f0feb3..e2fc9819c9 100644 --- a/lib/filetests/src/test_preopt.rs +++ b/lib/filetests/src/test_preopt.rs @@ -2,13 +2,13 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne::ir::Function; use cretonne; +use cretonne::ir::Function; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; +use subtest::{run_filecheck, Context, Result, SubTest}; struct TestPreopt; diff --git a/lib/filetests/src/test_print_cfg.rs b/lib/filetests/src/test_print_cfg.rs index 4c0b3250d2..7092126cd7 100644 --- a/lib/filetests/src/test_print_cfg.rs +++ b/lib/filetests/src/test_print_cfg.rs @@ -5,8 +5,8 @@ use std::borrow::Cow; -use cretonne::ir::Function; use cretonne::cfg_printer::CFGPrinter; +use cretonne::ir::Function; use cton_reader::TestCommand; use subtest::{self, Context, Result as STResult, SubTest}; diff --git a/lib/filetests/src/test_regalloc.rs b/lib/filetests/src/test_regalloc.rs index e94a142dcc..b2ab98c3a4 100644 --- a/lib/filetests/src/test_regalloc.rs +++ b/lib/filetests/src/test_regalloc.rs @@ -5,13 +5,13 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne::ir::Function; use cretonne; +use cretonne::ir::Function; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; +use subtest::{run_filecheck, Context, Result, SubTest}; struct TestRegalloc; diff --git a/lib/filetests/src/test_simple_gvn.rs b/lib/filetests/src/test_simple_gvn.rs index 04fcb078dd..62f5a2adae 100644 --- a/lib/filetests/src/test_simple_gvn.rs +++ b/lib/filetests/src/test_simple_gvn.rs @@ -5,13 +5,13 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne::ir::Function; use cretonne; +use cretonne::ir::Function; use cretonne::print_errors::pretty_error; use cton_reader::TestCommand; -use subtest::{run_filecheck, Context, Result, SubTest}; use std::borrow::Cow; use std::fmt::Write; +use subtest::{run_filecheck, Context, Result, SubTest}; struct TestSimpleGVN; diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs index 906b279408..a4ec799ee3 100644 --- a/lib/filetests/src/test_verifier.rs +++ b/lib/filetests/src/test_verifier.rs @@ -9,12 +9,12 @@ //! This annotation means that the verifier is expected to given an error for the jump instruction //! containing the substring "jump to non-existent EBB". -use std::borrow::{Borrow, Cow}; -use cretonne::verify_function; use cretonne::ir::Function; +use cretonne::verify_function; use cton_reader::TestCommand; -use subtest::{Context, Result, SubTest}; use match_directive::match_directive; +use std::borrow::{Borrow, Cow}; +use subtest::{Context, Result, SubTest}; struct TestVerifier; diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 00e5de53f1..4396c4edac 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -1,14 +1,14 @@ //! A frontend for building Cretonne IR from other languages. use cretonne::cursor::{Cursor, FuncCursor}; +use cretonne::entity::{EntityMap, EntityRef, EntitySet}; use cretonne::ir; +use cretonne::ir::function::DisplayFunction; use cretonne::ir::{DataFlowGraph, Ebb, ExtFuncData, FuncRef, Function, GlobalVar, GlobalVarData, Heap, HeapData, Inst, InstBuilderBase, InstructionData, JumpTable, JumpTableData, SigRef, Signature, StackSlot, StackSlotData, Type, Value}; -use cretonne::ir::function::DisplayFunction; use cretonne::isa::TargetIsa; -use ssa::{Block, SSABuilder, SideEffects}; -use cretonne::entity::{EntityMap, EntityRef, EntitySet}; use cretonne::packed_option::PackedOption; +use ssa::{Block, SSABuilder, SideEffects}; /// Structure used for translating a series of functions into Cretonne IR. /// @@ -592,13 +592,13 @@ where #[cfg(test)] mod tests { - use cretonne::entity::EntityRef; - use cretonne::ir::{AbiParam, CallConv, ExternalName, Function, InstBuilder, Signature}; - use cretonne::ir::types::*; - use frontend::{FunctionBuilder, FunctionBuilderContext}; - use cretonne::verifier::verify_function; - use cretonne::settings; use Variable; + use cretonne::entity::EntityRef; + use cretonne::ir::types::*; + use cretonne::ir::{AbiParam, CallConv, ExternalName, Function, InstBuilder, Signature}; + use cretonne::settings; + use cretonne::verifier::verify_function; + use frontend::{FunctionBuilder, FunctionBuilderContext}; fn sample_function(lazy_seal: bool) { let mut sig = Signature::new(CallConv::SystemV); diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 96085c38f8..82e9242b4f 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -6,15 +6,15 @@ //! Lecture Notes in Computer Science, vol 7791. Springer, Berlin, Heidelberg use cretonne::cursor::{Cursor, FuncCursor}; -use cretonne::ir::{Ebb, Function, Inst, InstBuilder, Type, Value}; -use cretonne::ir::instructions::BranchInfo; use cretonne::entity::{EntityMap, EntityRef, PrimaryMap}; +use cretonne::ir::immediates::{Ieee32, Ieee64}; +use cretonne::ir::instructions::BranchInfo; +use cretonne::ir::types::{F32, F64}; +use cretonne::ir::{Ebb, Function, Inst, InstBuilder, Type, Value}; use cretonne::packed_option::PackedOption; use cretonne::packed_option::ReservedValue; -use std::u32; -use cretonne::ir::types::{F32, F64}; -use cretonne::ir::immediates::{Ieee32, Ieee64}; use std::mem; +use std::u32; use std::vec::Vec; /// Structure containing the data relevant the construction of SSA for a given function. @@ -713,15 +713,15 @@ where #[cfg(test)] mod tests { + use Variable; use cretonne::cursor::{Cursor, FuncCursor}; use cretonne::entity::EntityRef; - use cretonne::ir::{Function, Inst, InstBuilder, JumpTableData, Opcode}; - use cretonne::ir::types::*; - use cretonne::verify_function; use cretonne::ir::instructions::BranchInfo; + use cretonne::ir::types::*; + use cretonne::ir::{Function, Inst, InstBuilder, JumpTableData, Opcode}; use cretonne::settings; + use cretonne::verify_function; use ssa::SSABuilder; - use Variable; #[test] fn simple_block() { diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs index 6cc29a81a1..e87b2b6771 100644 --- a/lib/reader/src/isaspec.rs +++ b/lib/reader/src/isaspec.rs @@ -6,8 +6,8 @@ //! If a test case file contains `isa` commands, the tests will only be run against the specified //! ISAs. If the file contains no `isa` commands, the tests will be run against all supported ISAs. -use cretonne::settings::{Configurable, Error as SetError, Flags}; use cretonne::isa::TargetIsa; +use cretonne::settings::{Configurable, Error as SetError, Flags}; use error::{Location, Result}; use testcommand::TestOption; diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index c55bbf1dd4..a2fde86b5a 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -1,12 +1,12 @@ //! Lexical analysis for .cton files. -use std::str::CharIndices; -use std::u16; -#[allow(unused_imports)] -use std::ascii::AsciiExt; use cretonne::ir::types; use cretonne::ir::{Ebb, Value}; use error::Location; +#[allow(unused_imports)] +use std::ascii::AsciiExt; +use std::str::CharIndices; +use std::u16; /// A Token returned from the `Lexer`. /// diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index 58fe21b5ce..e012876c1c 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -8,16 +8,16 @@ extern crate cretonne; pub use error::{Error, Location, Result}; +pub use isaspec::{parse_options, IsaSpec}; pub use parser::{parse_functions, parse_test}; +pub use sourcemap::SourceMap; pub use testcommand::{TestCommand, TestOption}; pub use testfile::{Comment, Details, TestFile}; -pub use isaspec::{parse_options, IsaSpec}; -pub use sourcemap::SourceMap; mod error; +mod isaspec; mod lexer; mod parser; -mod testcommand; -mod isaspec; -mod testfile; mod sourcemap; +mod testcommand; +mod testfile; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 48e2e2745b..f23f644a0e 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1,27 +1,27 @@ //! Parser for .cton files. -use std::str::FromStr; -use std::{u16, u32}; -use std::mem; +use cretonne::entity::EntityRef; +use cretonne::ir; +use cretonne::ir::entities::AnyEntity; +use cretonne::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32}; +use cretonne::ir::instructions::{InstructionData, InstructionFormat, VariableArgs}; +use cretonne::ir::types::VOID; use cretonne::ir::{AbiParam, ArgumentExtension, ArgumentLoc, CallConv, Ebb, ExtFuncData, ExternalName, FuncRef, Function, GlobalVar, GlobalVarData, Heap, HeapBase, HeapData, HeapStyle, JumpTable, JumpTableData, MemFlags, Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind, Type, Value, ValueLoc}; -use cretonne::ir; -use cretonne::ir::types::VOID; -use cretonne::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32}; -use cretonne::ir::entities::AnyEntity; -use cretonne::ir::instructions::{InstructionData, InstructionFormat, VariableArgs}; use cretonne::isa::{self, Encoding, RegUnit, TargetIsa}; -use cretonne::{settings, timing}; -use cretonne::entity::EntityRef; use cretonne::packed_option::ReservedValue; -use testfile::{Comment, Details, TestFile}; +use cretonne::{settings, timing}; use error::{Error, Location, Result}; -use lexer::{self, Lexer, Token}; -use testcommand::TestCommand; use isaspec; +use lexer::{self, Lexer, Token}; use sourcemap::SourceMap; +use std::mem; +use std::str::FromStr; +use std::{u16, u32}; +use testcommand::TestCommand; +use testfile::{Comment, Details, TestFile}; /// Parse the entire `text` into a list of functions. /// @@ -2395,13 +2395,13 @@ impl<'a> Parser<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne::ir::{ArgumentExtension, ArgumentPurpose, CallConv}; - use cretonne::ir::types; use cretonne::ir::StackSlotKind; use cretonne::ir::entities::AnyEntity; - use testfile::{Comment, Details}; - use isaspec::IsaSpec; + use cretonne::ir::types; + use cretonne::ir::{ArgumentExtension, ArgumentPurpose, CallConv}; use error::Error; + use isaspec::IsaSpec; + use testfile::{Comment, Details}; #[test] fn argument_type() { diff --git a/lib/reader/src/testfile.rs b/lib/reader/src/testfile.rs index 8ec5271a07..7218f8cb72 100644 --- a/lib/reader/src/testfile.rs +++ b/lib/reader/src/testfile.rs @@ -6,10 +6,10 @@ use cretonne::ir::Function; use cretonne::ir::entities::AnyEntity; -use testcommand::TestCommand; +use error::Location; use isaspec::IsaSpec; use sourcemap::SourceMap; -use error::Location; +use testcommand::TestCommand; /// A parsed test case. /// diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 82fec54e5b..9dbace8556 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -22,19 +22,19 @@ //! //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as //! argument. -use cretonne::ir::{self, InstBuilder, JumpTableData, MemFlags}; -use cretonne::ir::types::*; use cretonne::ir::condcodes::{FloatCC, IntCC}; +use cretonne::ir::types::*; +use cretonne::ir::{self, InstBuilder, JumpTableData, MemFlags}; use cretonne::packed_option::ReservedValue; use cton_frontend::{FunctionBuilder, Variable}; -use wasmparser::{MemoryImmediate, Operator}; -use translation_utils::{num_return_values, type_to_type, f32_translation, f64_translation}; -use translation_utils::{FunctionIndex, MemoryIndex, SignatureIndex, TableIndex}; +use environ::{FuncEnvironment, GlobalValue}; use state::{ControlStackFrame, TranslationState}; use std::collections::{hash_map, HashMap}; -use environ::{FuncEnvironment, GlobalValue}; -use std::{i32, u32}; use std::vec::Vec; +use std::{i32, u32}; +use translation_utils::{FunctionIndex, MemoryIndex, SignatureIndex, TableIndex}; +use translation_utils::{num_return_values, type_to_type, f32_translation, f64_translation}; +use wasmparser::{MemoryImmediate, Operator}; // Clippy warns about "flags: _" but its important to document that the flags field is ignored #[cfg_attr(feature = "cargo-clippy", allow(unneeded_field_pattern))] diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index b24499bde0..a71dddda3d 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -1,17 +1,17 @@ //! "Dummy" environment for testing wasm translation. +use cretonne::cursor::FuncCursor; +use cretonne::ir::types::*; +use cretonne::ir::{self, InstBuilder}; +use cretonne::settings; use environ::{FuncEnvironment, GlobalValue, ModuleEnvironment}; +use func_translator::FuncTranslator; +use std::error::Error; +use std::string::String; +use std::vec::Vec; use translation_utils::{FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex}; -use func_translator::FuncTranslator; -use cretonne::ir::{self, InstBuilder}; -use cretonne::ir::types::*; -use cretonne::cursor::FuncCursor; -use cretonne::settings; use wasmparser; -use std::error::Error; -use std::vec::Vec; -use std::string::String; /// Compute a `ir::ExternalName` for a given wasm function index. fn get_func_name(func_index: FunctionIndex) -> ir::ExternalName { diff --git a/lib/wasm/src/environ/mod.rs b/lib/wasm/src/environ/mod.rs index 57c9f593b2..e89995d8f8 100644 --- a/lib/wasm/src/environ/mod.rs +++ b/lib/wasm/src/environ/mod.rs @@ -1,7 +1,7 @@ //! Support for configurable wasm translation. -mod spec; mod dummy; +mod spec; -pub use environ::spec::{FuncEnvironment, GlobalValue, ModuleEnvironment}; pub use environ::dummy::DummyEnvironment; +pub use environ::spec::{FuncEnvironment, GlobalValue, ModuleEnvironment}; diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 882ca1a30f..8c313f0ef3 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -1,12 +1,12 @@ //! All the runtime support necessary for the wasm to cretonne translation is formalized by the //! traits `FunctionEnvironment` and `ModuleEnvironment`. -use cretonne::ir::{self, InstBuilder}; use cretonne::cursor::FuncCursor; +use cretonne::ir::{self, InstBuilder}; use cretonne::settings::Flags; +use std::string::String; +use std::vec::Vec; use translation_utils::{FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex}; -use std::vec::Vec; -use std::string::String; /// The value of a WebAssembly global variable. #[derive(Clone, Copy)] diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 159d8120d1..c89fa21266 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -232,10 +232,10 @@ fn cur_srcloc(reader: &BinaryReader) -> ir::SourceLoc { #[cfg(test)] mod tests { - use cretonne::{ir, Context}; - use cretonne::ir::types::I32; - use environ::{DummyEnvironment, FuncEnvironment}; use super::FuncTranslator; + use cretonne::ir::types::I32; + use cretonne::{ir, Context}; + use environ::{DummyEnvironment, FuncEnvironment}; #[test] fn small1() { diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 6f0ed8fd70..6ed76be8e3 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -19,15 +19,15 @@ extern crate cton_frontend; extern crate wasmparser; mod code_translator; +mod environ; mod func_translator; mod module_translator; -mod environ; mod sections_translator; mod state; mod translation_utils; +pub use environ::{DummyEnvironment, FuncEnvironment, GlobalValue, ModuleEnvironment}; pub use func_translator::FuncTranslator; pub use module_translator::translate_module; -pub use environ::{DummyEnvironment, FuncEnvironment, GlobalValue, ModuleEnvironment}; pub use translation_utils::{FunctionIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex, Table, TableIndex}; diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index e345e5ef3c..e41712a6d6 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -1,12 +1,12 @@ //! Translation skeleton that traverses the whole WebAssembly module and call helper functions //! to deal with each part of it. use cretonne::timing; -use wasmparser::{BinaryReaderError, Parser, ParserInput, ParserState, SectionCode, WasmDecoder}; +use environ::ModuleEnvironment; use sections_translator::{parse_data_section, parse_elements_section, parse_export_section, parse_function_section, parse_function_signatures, parse_global_section, parse_import_section, parse_memory_section, parse_start_section, parse_table_section, SectionParsingError}; -use environ::ModuleEnvironment; +use wasmparser::{BinaryReaderError, Parser, ParserInput, ParserState, SectionCode, WasmDecoder}; use std::string::String; diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 4792c3f450..ce5ed05995 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -7,17 +7,17 @@ //! The special case of the initialize expressions for table elements offsets or global variables //! is handled, according to the semantics of WebAssembly, to only specific expressions that are //! interpreted on the fly. +use cretonne; +use cretonne::ir::{AbiParam, CallConv, Signature}; +use environ::ModuleEnvironment; +use std::str::from_utf8; +use std::string::String; +use std::vec::Vec; use translation_utils::{type_to_type, FunctionIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex, Table, TableElementType, TableIndex}; -use cretonne::ir::{AbiParam, CallConv, Signature}; -use cretonne; +use wasmparser; use wasmparser::{ExternalKind, FuncType, ImportSectionEntryType, MemoryType, Operator, Parser, ParserState, WasmDecoder}; -use wasmparser; -use std::str::from_utf8; -use environ::ModuleEnvironment; -use std::vec::Vec; -use std::string::String; pub enum SectionParsingError { WrongSectionContent(String), diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index 8d1d86fe80..f95619d52b 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -6,8 +6,8 @@ use cretonne::ir::{self, Ebb, Inst, Value}; use environ::{FuncEnvironment, GlobalValue}; use std::collections::HashMap; -use translation_utils::{FunctionIndex, GlobalIndex, MemoryIndex, SignatureIndex}; use std::vec::Vec; +use translation_utils::{FunctionIndex, GlobalIndex, MemoryIndex, SignatureIndex}; /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following /// fields: diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 44c03fb6dd..bac50be7a6 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -1,7 +1,7 @@ //! Helper functions and structures for the translation. -use wasmparser; use cretonne; use std::u32; +use wasmparser; /// Index of a function (imported or defined) inside the WebAssembly module. pub type FunctionIndex = usize; diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs index 1bd6ecf3f9..273818f470 100644 --- a/lib/wasm/tests/wasm_testsuite.rs +++ b/lib/wasm/tests/wasm_testsuite.rs @@ -2,18 +2,18 @@ extern crate cretonne; extern crate cton_wasm; extern crate tempdir; -use cton_wasm::{translate_module, DummyEnvironment}; -use std::path::PathBuf; -use std::fs::File; -use std::error::Error; -use std::io; -use std::str; -use std::io::prelude::*; -use std::process::Command; -use std::fs; +use cretonne::print_errors::pretty_verifier_error; use cretonne::settings::{self, Configurable, Flags}; use cretonne::verifier; -use cretonne::print_errors::pretty_verifier_error; +use cton_wasm::{translate_module, DummyEnvironment}; +use std::error::Error; +use std::fs; +use std::fs::File; +use std::io; +use std::io::prelude::*; +use std::path::PathBuf; +use std::process::Command; +use std::str; use tempdir::TempDir; #[test] From b523b69c1675dc918931763aea0dc9713aff468d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 30 Mar 2018 13:38:30 -0700 Subject: [PATCH 1675/3084] Make bash function syntax consistent with other scripts in the repo. --- lib/cretonne/meta/check.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/meta/check.sh b/lib/cretonne/meta/check.sh index aa0e88ce1e..fa86bf0804 100755 --- a/lib/cretonne/meta/check.sh +++ b/lib/cretonne/meta/check.sh @@ -2,7 +2,7 @@ set -euo pipefail cd $(dirname "$0") -runif() { +function runif() { if command -v "$1" > /dev/null; then echo " === $1 ===" "$@" From 95c09bbfdef979b74c847729d85c193e1ffddce7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 30 Mar 2018 17:09:21 -0700 Subject: [PATCH 1676/3084] Add docs/_build to .gitignore. It's generated by running "make html" or other make command in the docs directory. --- cranelift/.gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/cranelift/.gitignore b/cranelift/.gitignore index a481ff48e8..041b7ad3a8 100644 --- a/cranelift/.gitignore +++ b/cranelift/.gitignore @@ -10,3 +10,4 @@ Cargo.lock cretonne.dbg* .mypy_cache rusty-tags.* +docs/_build From e107793b68762d6ffed2c46ec931c7ef37479dcc Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 30 Mar 2018 21:00:37 -0700 Subject: [PATCH 1677/3084] Pre-opt: Use the correct operand in the irsub_imm pattern. --- cranelift/filetests/preopt/simplify.cton | 2 +- lib/cretonne/src/preopt.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/cranelift/filetests/preopt/simplify.cton b/cranelift/filetests/preopt/simplify.cton index a2e67caf31..020e2f0108 100644 --- a/cranelift/filetests/preopt/simplify.cton +++ b/cranelift/filetests/preopt/simplify.cton @@ -75,6 +75,6 @@ ebb0(v0: i32): ; sameln: function %irsub_imm ; nextln: ebb0(v0: i32): ; nextln: v1 = iconst.i32 2 -; nextln: v2 = irsub_imm v1, 2 +; nextln: v2 = irsub_imm v0, 2 ; nextln: return v2 ; nextln: } diff --git a/lib/cretonne/src/preopt.rs b/lib/cretonne/src/preopt.rs index bdd1c445eb..0a1c13d6ea 100644 --- a/lib/cretonne/src/preopt.rs +++ b/lib/cretonne/src/preopt.rs @@ -505,7 +505,7 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) { new_opcode, ty, imm, - args[0], + args[1], ); } } From 0948ca9963de7acbbc85aaadde21bfb4411c8549 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Apr 2018 10:12:03 -0700 Subject: [PATCH 1678/3084] Add a status summary to the README. (#290) * Add a status summary to the README. This adds a brief blurb about Cretonne's current status, so that people looking at Cretonne have an idea of what to expect. Also remove the "not yet functional" disclaimer, as Cretonne is functional for some use cases now, and the new Status section explains the current status. --- README.rst | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/README.rst b/README.rst index 41e6a9284d..eae5f507ba 100644 --- a/README.rst +++ b/README.rst @@ -6,8 +6,6 @@ Cretonne is a low-level retargetable code generator. It translates a `target-ind intermediate representation `_ into executable machine code. -*This is a work in progress that is not yet functional.* - .. image:: https://readthedocs.org/projects/cretonne/badge/?version=latest :target: https://cretonne.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status @@ -23,6 +21,37 @@ machine code. For more information, see `the documentation `_. +Status +------ + +Cretonne currently supports enough functionality to run a wide variety of +programs, including all the functionality needed to execute WebAssembly MVP +functions, although it needs to be used within an external WebAssembly +embedding to be part of a complete WebAssembly implementation. + +The x86-64 backend is currently the most complete and stable; other +architectures are in various stages of development. Cretonne currently supports +the System V AMD64 ABI calling convention used on many platforms, but does not +yet support the Windows x64 calling convention. The performance of code +produced by Cretonne is not yet impressive, though we have plans to fix that. + +The core codegen crates have minimal dependencies, and do not require any host +floating-point support. Support for `no_std` mode in the core codegen crates is +`in development `_. + +Cretonne does not yet perform mitigations for Spectre or related security +issues, though it may do so in the future. It does not currently make any +security-relevant instruction timing guarantees. It has seen a fair amount +of testing and fuzzing, although more work is needed before it would be +ready for a production use case. + +Cretonne's APIs are not yet stable. + +Cretonne currently supports Rust 1.22.1 and later. We intend to always support +the latest *stable* Rust. And, we currently support the version of Rust in the +latest Ubuntu LTS, although whether we will always do so is not yet determined. +Cretonne requires Python 2.7 or Python 3 to build. + Planned uses ------------ From 775c674b38e8402160d43011e999ec3e89fb3d76 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Tue, 3 Apr 2018 14:44:12 -0700 Subject: [PATCH 1679/3084] Only save callee-saved registers that are used (#293) * Only save callee-saved registers that are actually being used. * Rename AllocatableSet to RegisterSet * Style cleanup and small renames for readability. * Adjust x86 prologue-epilogue test to account for callee-saved register optimization. * Add more tests for prologue-epilogue optimizations. --- .../isa/intel/prologue-epilogue.cton | 214 ++++++++++++++++-- lib/cretonne/src/isa/arm32/abi.rs | 4 +- lib/cretonne/src/isa/arm32/mod.rs | 2 +- lib/cretonne/src/isa/arm64/abi.rs | 4 +- lib/cretonne/src/isa/arm64/mod.rs | 2 +- lib/cretonne/src/isa/intel/abi.rs | 64 ++++-- lib/cretonne/src/isa/intel/mod.rs | 2 +- lib/cretonne/src/isa/mod.rs | 2 +- lib/cretonne/src/isa/riscv/abi.rs | 6 +- lib/cretonne/src/isa/riscv/mod.rs | 2 +- lib/cretonne/src/regalloc/coloring.rs | 22 +- lib/cretonne/src/regalloc/mod.rs | 4 +- lib/cretonne/src/regalloc/pressure.rs | 10 +- .../{allocatable_set.rs => register_set.rs} | 37 +-- lib/cretonne/src/regalloc/solver.rs | 58 ++--- 15 files changed, 310 insertions(+), 123 deletions(-) rename lib/cretonne/src/regalloc/{allocatable_set.rs => register_set.rs} (90%) diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/intel/prologue-epilogue.cton index cd774fe6bd..416f90df63 100644 --- a/cranelift/filetests/isa/intel/prologue-epilogue.cton +++ b/cranelift/filetests/isa/intel/prologue-epilogue.cton @@ -3,30 +3,206 @@ set is_64bit set is_compressed isa intel haswell -function %foo() { +; An empty function. + +function %empty() { +ebb0: + return +} + +; check: function %empty(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; nextln: ss0 = incoming_arg 16, offset -16 +; nextln: +; nextln: ebb0(v0: i64 [%rbp]): +; nextln: x86_push v0 +; nextln: copy_special %rsp -> %rbp +; nextln: v1 = x86_pop.i64 +; nextln: return v1 +; nextln: } + +; A function with a single stack slot. + +function %one_stack_slot() { ss0 = explicit_slot 168 ebb0: return } -; check: function %foo(i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] system_v { -; nextln: ss0 = explicit_slot 168, offset -224 -; nextln: ss1 = incoming_arg 56, offset -56 -; check: ebb0(v0: i64 [%rbp], v1: i64 [%rbx], v2: i64 [%r12], v3: i64 [%r13], v4: i64 [%r14], v5: i64 [%r15]): +; check: function %one_stack_slot(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; nextln: ss0 = explicit_slot 168, offset -184 +; nextln: ss1 = incoming_arg 16, offset -16 +; nextln: +; nextln: ebb0(v0: i64 [%rbp]): ; nextln: x86_push v0 ; nextln: copy_special %rsp -> %rbp -; nextln: x86_push v1 -; nextln: x86_push v2 -; nextln: x86_push v3 -; nextln: x86_push v4 -; nextln: x86_push v5 -; nextln: adjust_sp_imm -168 -; nextln: adjust_sp_imm 168 -; nextln: v11 = x86_pop.i64 -; nextln: v10 = x86_pop.i64 -; nextln: v9 = x86_pop.i64 -; nextln: v8 = x86_pop.i64 -; nextln: v7 = x86_pop.i64 -; nextln: v6 = x86_pop.i64 -; nextln: return v6, v7, v8, v9, v10, v11 +; nextln: adjust_sp_imm -176 +; nextln: adjust_sp_imm 176 +; nextln: v1 = x86_pop.i64 +; nextln: return v1 +; nextln: } + +; A function performing a call. + +function %call() { + fn0 = function %foo() + +ebb0: + call fn0() + return +} + +; check: function %call(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; nextln: ss0 = incoming_arg 16, offset -16 +; nextln: sig0 = () system_v +; nextln: fn0 = sig0 %foo +; nextln: +; nextln: ebb0(v0: i64 [%rbp]): +; nextln: x86_push v0 +; nextln: copy_special %rsp -> %rbp +; nextln: call fn0() +; nextln: v1 = x86_pop.i64 +; nextln: return v1 +; nextln: } + +; A function that uses a lot of registers but doesn't quite need to spill. + +function %no_spill(i64, i64) { +ebb0(v0: i64, v1: i64): + v2 = load.i32 v0+0 + v3 = load.i32 v0+8 + v4 = load.i32 v0+16 + v5 = load.i32 v0+24 + v6 = load.i32 v0+32 + v7 = load.i32 v0+40 + v8 = load.i32 v0+48 + v9 = load.i32 v0+56 + v10 = load.i32 v0+64 + v11 = load.i32 v0+72 + v12 = load.i32 v0+80 + v13 = load.i32 v0+88 + v14 = load.i32 v0+96 + store.i32 v2, v1+0 + store.i32 v3, v1+8 + store.i32 v4, v1+16 + store.i32 v5, v1+24 + store.i32 v6, v1+32 + store.i32 v7, v1+40 + store.i32 v8, v1+48 + store.i32 v9, v1+56 + store.i32 v10, v1+64 + store.i32 v11, v1+72 + store.i32 v12, v1+80 + store.i32 v13, v1+88 + store.i32 v14, v1+96 + return +} + +; check: function %no_spill(i64 [%rdi], i64 [%rsi], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] system_v { +; nextln: ss0 = incoming_arg 56, offset -56 +; nextln: +; nextln: ebb0(v0: i64 [%rdi], v1: i64 [%rsi], v15: i64 [%rbp], v16: i64 [%rbx], v17: i64 [%r12], v18: i64 [%r13], v19: i64 [%r14], v20: i64 [%r15]): +; nextln: x86_push v15 +; nextln: copy_special %rsp -> %rbp +; nextln: x86_push v16 +; nextln: x86_push v17 +; nextln: x86_push v18 +; nextln: x86_push v19 +; nextln: x86_push v20 +; nextln: adjust_sp_imm -8 +; nextln: v2 = load.i32 v0 +; nextln: v3 = load.i32 v0+8 +; nextln: v4 = load.i32 v0+16 +; nextln: v5 = load.i32 v0+24 +; nextln: v6 = load.i32 v0+32 +; nextln: v7 = load.i32 v0+40 +; nextln: v8 = load.i32 v0+48 +; nextln: v9 = load.i32 v0+56 +; nextln: v10 = load.i32 v0+64 +; nextln: v11 = load.i32 v0+72 +; nextln: v12 = load.i32 v0+80 +; nextln: v13 = load.i32 v0+88 +; nextln: v14 = load.i32 v0+96 +; nextln: store v2, v1 +; nextln: store v3, v1+8 +; nextln: store v4, v1+16 +; nextln: store v5, v1+24 +; nextln: store v6, v1+32 +; nextln: store v7, v1+40 +; nextln: store v8, v1+48 +; nextln: store v9, v1+56 +; nextln: store v10, v1+64 +; nextln: store v11, v1+72 +; nextln: store v12, v1+80 +; nextln: store v13, v1+88 +; nextln: store v14, v1+96 +; nextln: adjust_sp_imm 8 +; nextln: v26 = x86_pop.i64 +; nextln: v25 = x86_pop.i64 +; nextln: v24 = x86_pop.i64 +; nextln: v23 = x86_pop.i64 +; nextln: v22 = x86_pop.i64 +; nextln: v21 = x86_pop.i64 +; nextln: return v21, v22, v23, v24, v25, v26 +; nextln: } + +; This function requires too many registers and must spill. + +function %yes_spill(i64, i64) { +ebb0(v0: i64, v1: i64): + v2 = load.i32 v0+0 + v3 = load.i32 v0+8 + v4 = load.i32 v0+16 + v5 = load.i32 v0+24 + v6 = load.i32 v0+32 + v7 = load.i32 v0+40 + v8 = load.i32 v0+48 + v9 = load.i32 v0+56 + v10 = load.i32 v0+64 + v11 = load.i32 v0+72 + v12 = load.i32 v0+80 + v13 = load.i32 v0+88 + v14 = load.i32 v0+96 + v15 = load.i32 v0+104 + store.i32 v2, v1+0 + store.i32 v3, v1+8 + store.i32 v4, v1+16 + store.i32 v5, v1+24 + store.i32 v6, v1+32 + store.i32 v7, v1+40 + store.i32 v8, v1+48 + store.i32 v9, v1+56 + store.i32 v10, v1+64 + store.i32 v11, v1+72 + store.i32 v12, v1+80 + store.i32 v13, v1+88 + store.i32 v14, v1+96 + store.i32 v15, v1+104 + return +} + +; check: function %yes_spill(i64 [%rdi], i64 [%rsi], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] system_v { +; check: ss0 = spill_slot + +; check: ebb0(v16: i64 [%rdi], v17: i64 [%rsi], v48: i64 [%rbp], v49: i64 [%rbx], v50: i64 [%r12], v51: i64 [%r13], v52: i64 [%r14], v53: i64 [%r15]): +; nextln: x86_push v48 +; nextln: copy_special %rsp -> %rbp +; nextln: x86_push v49 +; nextln: x86_push v50 +; nextln: x86_push v51 +; nextln: x86_push v52 +; nextln: x86_push v53 +; nextln: adjust_sp_imm + +; check: spill + +; check: fill + +; check: adjust_sp_imm +; nextln: v59 = x86_pop.i64 +; nextln: v58 = x86_pop.i64 +; nextln: v57 = x86_pop.i64 +; nextln: v56 = x86_pop.i64 +; nextln: v55 = x86_pop.i64 +; nextln: v54 = x86_pop.i64 +; nextln: return v54, v55, v56, v57, v58, v59 ; nextln: } diff --git a/lib/cretonne/src/isa/arm32/abi.rs b/lib/cretonne/src/isa/arm32/abi.rs index 528a396570..1305685825 100644 --- a/lib/cretonne/src/isa/arm32/abi.rs +++ b/lib/cretonne/src/isa/arm32/abi.rs @@ -3,7 +3,7 @@ use super::registers::{D, GPR, Q, S}; use ir; use isa::RegClass; -use regalloc::AllocatableSet; +use regalloc::RegisterSet; use settings as shared_settings; /// Legalize `sig`. @@ -30,6 +30,6 @@ pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { } /// Get the set of allocatable registers for `func`. -pub fn allocatable_registers(_func: &ir::Function) -> AllocatableSet { +pub fn allocatable_registers(_func: &ir::Function) -> RegisterSet { unimplemented!() } diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index 350764a436..dbef9aaf8a 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -92,7 +92,7 @@ impl TargetIsa for Isa { abi::regclass_for_abi_type(ty) } - fn allocatable_registers(&self, func: &ir::Function) -> regalloc::AllocatableSet { + fn allocatable_registers(&self, func: &ir::Function) -> regalloc::RegisterSet { abi::allocatable_registers(func) } diff --git a/lib/cretonne/src/isa/arm64/abi.rs b/lib/cretonne/src/isa/arm64/abi.rs index 0540746afa..4bd9aad7be 100644 --- a/lib/cretonne/src/isa/arm64/abi.rs +++ b/lib/cretonne/src/isa/arm64/abi.rs @@ -3,7 +3,7 @@ use super::registers::{FPR, GPR}; use ir; use isa::RegClass; -use regalloc::AllocatableSet; +use regalloc::RegisterSet; use settings as shared_settings; /// Legalize `sig`. @@ -21,6 +21,6 @@ pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { } /// Get the set of allocatable registers for `func`. -pub fn allocatable_registers(_func: &ir::Function) -> AllocatableSet { +pub fn allocatable_registers(_func: &ir::Function) -> RegisterSet { unimplemented!() } diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index 3b63d56d54..eeddec74fd 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -85,7 +85,7 @@ impl TargetIsa for Isa { abi::regclass_for_abi_type(ty) } - fn allocatable_registers(&self, func: &ir::Function) -> regalloc::AllocatableSet { + fn allocatable_registers(&self, func: &ir::Function) -> regalloc::RegisterSet { abi::allocatable_registers(func) } diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index e9019291ef..f1abd1cb42 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -6,9 +6,10 @@ use cursor::{Cursor, CursorPosition, EncCursor}; use ir; use ir::immediates::Imm64; use ir::stackslot::{StackOffset, StackSize}; -use ir::{AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, CallConv, InstBuilder}; +use ir::{AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, CallConv, InstBuilder, + ValueLoc}; use isa::{RegClass, RegUnit, TargetIsa}; -use regalloc::AllocatableSet; +use regalloc::RegisterSet; use result; use settings as shared_settings; use stack_layout::layout_stack; @@ -140,11 +141,8 @@ pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { } /// Get the set of allocatable registers for `func`. -pub fn allocatable_registers( - _func: &ir::Function, - flags: &shared_settings::Flags, -) -> AllocatableSet { - let mut regs = AllocatableSet::new(); +pub fn allocatable_registers(_func: &ir::Function, flags: &shared_settings::Flags) -> RegisterSet { + let mut regs = RegisterSet::new(); regs.take(GPR, RU::rsp as RegUnit); regs.take(GPR, RU::rbp as RegUnit); @@ -160,7 +158,7 @@ pub fn allocatable_registers( } /// Get the set of callee-saved registers. -pub fn callee_saved_registers(flags: &shared_settings::Flags) -> &'static [RU] { +fn callee_saved_gprs(flags: &shared_settings::Flags) -> &'static [RU] { if flags.is_64bit() { &[RU::rbx, RU::r12, RU::r13, RU::r14, RU::r15] } else { @@ -168,6 +166,28 @@ pub fn callee_saved_registers(flags: &shared_settings::Flags) -> &'static [RU] { } } +fn callee_saved_gprs_used(flags: &shared_settings::Flags, func: &ir::Function) -> RegisterSet { + let mut all_callee_saved = RegisterSet::empty(); + for reg in callee_saved_gprs(flags) { + all_callee_saved.free(GPR, *reg as RegUnit); + } + + let mut used = RegisterSet::empty(); + for value_loc in func.locations.values() { + // Note that `value_loc` here contains only a single unit of a potentially multi-unit + // register. We don't use registers that overlap each other in the x86 ISA, but in others + // we do. So this should not be blindly reused. + if let ValueLoc::Reg(ru) = *value_loc { + if !used.is_avail(GPR, ru) { + used.free(GPR, ru); + } + } + } + + used.intersect(&all_callee_saved); + return used; +} + pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { match func.signature.call_conv { ir::CallConv::SystemV => system_v_prologue_epilogue(func, isa), @@ -203,7 +223,8 @@ pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r } else { ir::types::I32 }; - let csrs = callee_saved_registers(isa.flags()); + + let csrs = callee_saved_gprs_used(isa.flags(), func); // The reserved stack area is composed of: // return address + frame pointer + all callee-saved registers @@ -212,7 +233,7 @@ pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r // instruction. Each of the others we will then push explicitly. Then we // will adjust the stack pointer to make room for the rest of the required // space for this frame. - let csr_stack_size = ((csrs.len() + 2) * word_size as usize) as i32; + let csr_stack_size = ((csrs.iter(GPR).len() + 2) * word_size as usize) as i32; func.create_stack_slot(ir::StackSlotData { kind: ir::StackSlotKind::IncomingArg, size: csr_stack_size as u32, @@ -231,9 +252,8 @@ pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r func.signature.params.push(fp_arg); func.signature.returns.push(fp_arg); - for csr in csrs.iter() { - let csr_arg = - ir::AbiParam::special_reg(csr_type, ir::ArgumentPurpose::CalleeSaved, *csr as RegUnit); + for csr in csrs.iter(GPR) { + let csr_arg = ir::AbiParam::special_reg(csr_type, ir::ArgumentPurpose::CalleeSaved, csr); func.signature.params.push(csr_arg); func.signature.returns.push(csr_arg); } @@ -241,11 +261,11 @@ pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r // Set up the cursor and insert the prologue let entry_ebb = func.layout.entry_block().expect("missing entry block"); let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_ebb); - insert_system_v_prologue(&mut pos, local_stack_size, csr_type, csrs); + insert_system_v_prologue(&mut pos, local_stack_size, csr_type, &csrs); // Reset the cursor and insert the epilogue let mut pos = pos.at_position(CursorPosition::Nowhere); - insert_system_v_epilogues(&mut pos, local_stack_size, csr_type, csrs); + insert_system_v_epilogues(&mut pos, local_stack_size, csr_type, &csrs); Ok(()) } @@ -255,7 +275,7 @@ fn insert_system_v_prologue( pos: &mut EncCursor, stack_size: i64, csr_type: ir::types::Type, - csrs: &'static [RU], + csrs: &RegisterSet, ) { // Append param to entry EBB let ebb = pos.current_ebb().expect("missing ebb under cursor"); @@ -268,12 +288,12 @@ fn insert_system_v_prologue( RU::rbp as RegUnit, ); - for reg in csrs.iter() { + for reg in csrs.iter(GPR) { // Append param to entry EBB let csr_arg = pos.func.dfg.append_ebb_param(ebb, csr_type); // Assign it a location - pos.func.locations[csr_arg] = ir::ValueLoc::Reg(*reg as RegUnit); + pos.func.locations[csr_arg] = ir::ValueLoc::Reg(reg); // Remember it so we can push it momentarily pos.ins().x86_push(csr_arg); @@ -289,7 +309,7 @@ fn insert_system_v_epilogues( pos: &mut EncCursor, stack_size: i64, csr_type: ir::types::Type, - csrs: &'static [RU], + csrs: &RegisterSet, ) { while let Some(ebb) = pos.next_ebb() { pos.goto_last_inst(ebb); @@ -307,7 +327,7 @@ fn insert_system_v_epilogue( stack_size: i64, pos: &mut EncCursor, csr_type: ir::types::Type, - csrs: &'static [RU], + csrs: &RegisterSet, ) { if stack_size > 0 { pos.ins().adjust_sp_imm(Imm64::new(stack_size)); @@ -321,11 +341,11 @@ fn insert_system_v_epilogue( pos.func.locations[fp_ret] = ir::ValueLoc::Reg(RU::rbp as RegUnit); pos.func.dfg.append_inst_arg(inst, fp_ret); - for reg in csrs.iter() { + for reg in csrs.iter(GPR) { let csr_ret = pos.ins().x86_pop(csr_type); pos.prev_inst(); - pos.func.locations[csr_ret] = ir::ValueLoc::Reg(*reg as RegUnit); + pos.func.locations[csr_ret] = ir::ValueLoc::Reg(reg); pos.func.dfg.append_inst_arg(inst, csr_ret); } } diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/intel/mod.rs index 91d8325a6b..02cccc4f54 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/intel/mod.rs @@ -98,7 +98,7 @@ impl TargetIsa for Isa { abi::regclass_for_abi_type(ty) } - fn allocatable_registers(&self, func: &ir::Function) -> regalloc::AllocatableSet { + fn allocatable_registers(&self, func: &ir::Function) -> regalloc::RegisterSet { abi::allocatable_registers(func, &self.shared_flags) } diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 39d7d4eb60..71537a0487 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -238,7 +238,7 @@ pub trait TargetIsa: fmt::Display { /// /// This set excludes reserved registers like the stack pointer and other special-purpose /// registers. - fn allocatable_registers(&self, func: &ir::Function) -> regalloc::AllocatableSet; + fn allocatable_registers(&self, func: &ir::Function) -> regalloc::RegisterSet; /// Compute the stack layout and insert prologue and epilogue code into `func`. /// diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/cretonne/src/isa/riscv/abi.rs index 0782fa5e53..96e532e827 100644 --- a/lib/cretonne/src/isa/riscv/abi.rs +++ b/lib/cretonne/src/isa/riscv/abi.rs @@ -10,7 +10,7 @@ use super::settings; use abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion}; use ir::{self, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, Type}; use isa::RegClass; -use regalloc::AllocatableSet; +use regalloc::RegisterSet; use settings as shared_settings; use std::i32; @@ -120,8 +120,8 @@ pub fn regclass_for_abi_type(ty: Type) -> RegClass { if ty.is_float() { FPR } else { GPR } } -pub fn allocatable_registers(_func: &ir::Function, isa_flags: &settings::Flags) -> AllocatableSet { - let mut regs = AllocatableSet::new(); +pub fn allocatable_registers(_func: &ir::Function, isa_flags: &settings::Flags) -> RegisterSet { + let mut regs = RegisterSet::new(); regs.take(GPR, GPR.unit(0)); // Hard-wired 0. // %x1 is the link register which is available for allocation. regs.take(GPR, GPR.unit(2)); // Stack pointer. diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index 9c3a498d5e..c81d3baceb 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -92,7 +92,7 @@ impl TargetIsa for Isa { abi::regclass_for_abi_type(ty) } - fn allocatable_registers(&self, func: &ir::Function) -> regalloc::AllocatableSet { + fn allocatable_registers(&self, func: &ir::Function) -> regalloc::RegisterSet { abi::allocatable_registers(func, &self.isa_flags) } diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/cretonne/src/regalloc/coloring.rs index b9b1512220..4ddda53e16 100644 --- a/lib/cretonne/src/regalloc/coloring.rs +++ b/lib/cretonne/src/regalloc/coloring.rs @@ -51,7 +51,7 @@ use isa::{regs_overlap, RegClass, RegInfo, RegUnit}; use packed_option::PackedOption; use regalloc::RegDiversions; use regalloc::affinity::Affinity; -use regalloc::allocatable_set::AllocatableSet; +use regalloc::register_set::RegisterSet; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; use regalloc::liverange::{LiveRange, LiveRangeContext}; @@ -96,7 +96,7 @@ struct Context<'a> { // Pristine set of registers that the allocator can use. // This set remains immutable, we make clones. - usable_regs: AllocatableSet, + usable_regs: RegisterSet, } impl Coloring { @@ -699,7 +699,7 @@ impl<'a> Context<'a> { defs: &[LiveValue], throughs: &[LiveValue], replace_global_defines: &mut bool, - global_regs: &AllocatableSet, + global_regs: &RegisterSet, ) { for (op, lv) in constraints.iter().zip(defs) { match op.kind { @@ -732,7 +732,7 @@ impl<'a> Context<'a> { defs: &[LiveValue], throughs: &[LiveValue], replace_global_defines: &mut bool, - global_regs: &AllocatableSet, + global_regs: &RegisterSet, ) { // It's technically possible for a call instruction to have fixed results before the // variable list of results, but we have no known instances of that. @@ -797,7 +797,7 @@ impl<'a> Context<'a> { constraints: &[OperandConstraint], defs: &[LiveValue], replace_global_defines: &mut bool, - global_regs: &AllocatableSet, + global_regs: &RegisterSet, ) { for (op, lv) in constraints.iter().zip(defs) { match op.kind { @@ -843,9 +843,9 @@ impl<'a> Context<'a> { fn iterate_solution( &mut self, throughs: &[LiveValue], - global_regs: &AllocatableSet, + global_regs: &RegisterSet, replace_global_defines: &mut bool, - ) -> AllocatableSet { + ) -> RegisterSet { // Make sure `try_add_var()` below doesn't create a variable with too loose constraints. self.program_complete_input_constraints(); @@ -923,7 +923,7 @@ impl<'a> Context<'a> { /// inserted before. /// /// The solver needs to be reminded of the available registers before any moves are inserted. - fn shuffle_inputs(&mut self, regs: &mut AllocatableSet) { + fn shuffle_inputs(&mut self, regs: &mut RegisterSet) { use regalloc::solver::Move::*; let spills = self.solver.schedule_moves(regs); @@ -1114,19 +1114,19 @@ fn program_input_abi( struct AvailableRegs { /// The exact set of registers available on the input side of the current instruction. This /// takes into account register diversions, and it includes both local and global live ranges. - input: AllocatableSet, + input: RegisterSet, /// Registers available for allocating globally live values. This set ignores any local values, /// and it does not account for register diversions. /// /// Global values must be allocated out of this set because conflicts with other global values /// can't be resolved with local diversions. - global: AllocatableSet, + global: RegisterSet, } impl AvailableRegs { /// Initialize both the input and global sets from `regs`. - pub fn new(regs: &AllocatableSet) -> AvailableRegs { + pub fn new(regs: &RegisterSet) -> AvailableRegs { AvailableRegs { input: regs.clone(), global: regs.clone(), diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/cretonne/src/regalloc/mod.rs index 6868c0a235..1444874b83 100644 --- a/lib/cretonne/src/regalloc/mod.rs +++ b/lib/cretonne/src/regalloc/mod.rs @@ -2,7 +2,7 @@ //! //! This module contains data structures and algorithms used for register allocation. -pub mod allocatable_set; +pub mod register_set; pub mod coloring; pub mod live_value_tracker; pub mod liveness; @@ -18,6 +18,6 @@ mod reload; mod solver; mod spilling; -pub use self::allocatable_set::AllocatableSet; +pub use self::register_set::RegisterSet; pub use self::context::Context; pub use self::diversion::RegDiversions; diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/cretonne/src/regalloc/pressure.rs index 14978f48b8..9f7d64365d 100644 --- a/lib/cretonne/src/regalloc/pressure.rs +++ b/lib/cretonne/src/regalloc/pressure.rs @@ -37,7 +37,7 @@ #![allow(dead_code)] use isa::registers::{RegClass, RegClassMask, RegInfo, MAX_TRACKED_TOPRCS}; -use regalloc::AllocatableSet; +use regalloc::RegisterSet; use std::cmp::min; use std::fmt; use std::iter::ExactSizeIterator; @@ -81,7 +81,7 @@ pub struct Pressure { impl Pressure { /// Create a new register pressure tracker. - pub fn new(reginfo: &RegInfo, usable: &AllocatableSet) -> Pressure { + pub fn new(reginfo: &RegInfo, usable: &RegisterSet) -> Pressure { let mut p = Pressure { aliased: 0, toprc: Default::default(), @@ -271,7 +271,7 @@ impl fmt::Display for Pressure { mod tests { use super::Pressure; use isa::{RegClass, TargetIsa}; - use regalloc::AllocatableSet; + use regalloc::RegisterSet; use std::borrow::Borrow; use std::boxed::Box; @@ -302,7 +302,7 @@ mod tests { let gpr = rc_by_name(isa, "GPR"); let s = rc_by_name(isa, "S"); let reginfo = isa.register_info(); - let regs = AllocatableSet::new(); + let regs = RegisterSet::new(); let mut pressure = Pressure::new(®info, ®s); let mut count = 0; @@ -331,7 +331,7 @@ mod tests { let d = rc_by_name(isa, "D"); let q = rc_by_name(isa, "Q"); let reginfo = isa.register_info(); - let regs = AllocatableSet::new(); + let regs = RegisterSet::new(); let mut pressure = Pressure::new(®info, ®s); assert_eq!(pressure.check_avail(s), 0); diff --git a/lib/cretonne/src/regalloc/allocatable_set.rs b/lib/cretonne/src/regalloc/register_set.rs similarity index 90% rename from lib/cretonne/src/regalloc/allocatable_set.rs rename to lib/cretonne/src/regalloc/register_set.rs index 63b7a297d7..f2f109ecd9 100644 --- a/lib/cretonne/src/regalloc/allocatable_set.rs +++ b/lib/cretonne/src/regalloc/register_set.rs @@ -13,7 +13,7 @@ use std::mem::size_of_val; /// Set of registers available for allocation. #[derive(Clone)] -pub struct AllocatableSet { +pub struct RegisterSet { avail: RegUnitMask, } @@ -32,7 +32,7 @@ fn bitmask(rc: RegClass, reg: RegUnit) -> (usize, u32) { (word_index, reg_bits) } -impl AllocatableSet { +impl RegisterSet { /// Create a new register set with all registers available. /// /// Note that this includes *all* registers. Query the `TargetIsa` object to get a set of @@ -41,6 +41,11 @@ impl AllocatableSet { Self { avail: [!0; 3] } } + /// Create a new register set with no registers available. + pub fn empty() -> Self { + Self { avail: [0; 3] } + } + /// Returns `true` if the specified register is available. pub fn is_avail(&self, rc: RegClass, reg: RegUnit) -> bool { let (idx, bits) = bitmask(rc, reg); @@ -62,7 +67,7 @@ impl AllocatableSet { self.avail[idx] &= !bits; } - /// Make `reg` available for allocation again. + /// Return `reg` and all of its register units to the set of available registers. pub fn free(&mut self, rc: RegClass, reg: RegUnit) { let (idx, bits) = bitmask(rc, reg); debug_assert!( @@ -98,15 +103,15 @@ impl AllocatableSet { /// of `other`. /// /// This assumes that unused bits are 1. - pub fn interferes_with(&self, other: &AllocatableSet) -> bool { + pub fn interferes_with(&self, other: &RegisterSet) -> bool { self.avail.iter().zip(&other.avail).any( |(&x, &y)| (x | y) != !0, ) } - /// Intersect this set of allocatable registers with `other`. This has the effect of removing - /// any register units from this set that are not in `other`. - pub fn intersect(&mut self, other: &AllocatableSet) { + /// Intersect this set of registers with `other`. This has the effect of removing any register + /// units from this set that are not in `other`. + pub fn intersect(&mut self, other: &RegisterSet) { for (x, &y) in self.avail.iter_mut().zip(&other.avail) { *x &= y; } @@ -114,8 +119,8 @@ impl AllocatableSet { /// Return an object that can display this register set, using the register info from the /// target ISA. - pub fn display<'a, R: Into>>(&self, regs: R) -> DisplayAllocatableSet<'a> { - DisplayAllocatableSet(self.clone(), regs.into()) + pub fn display<'a, R: Into>>(&self, regs: R) -> DisplayRegisterSet<'a> { + DisplayRegisterSet(self.clone(), regs.into()) } } @@ -157,10 +162,10 @@ impl Iterator for RegSetIter { impl ExactSizeIterator for RegSetIter {} -/// Displaying an `AllocatableSet` correctly requires the associated `RegInfo` from the target ISA. -pub struct DisplayAllocatableSet<'a>(AllocatableSet, Option<&'a RegInfo>); +/// Displaying an `RegisterSet` correctly requires the associated `RegInfo` from the target ISA. +pub struct DisplayRegisterSet<'a>(RegisterSet, Option<&'a RegInfo>); -impl<'a> fmt::Display for DisplayAllocatableSet<'a> { +impl<'a> fmt::Display for DisplayRegisterSet<'a> { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "[")?; match self.1 { @@ -211,7 +216,7 @@ impl<'a> fmt::Display for DisplayAllocatableSet<'a> { } } -impl fmt::Display for AllocatableSet { +impl fmt::Display for RegisterSet { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { self.display(None).fmt(f) } @@ -255,7 +260,7 @@ mod tests { #[test] fn put_and_take() { - let mut regs = AllocatableSet::new(); + let mut regs = RegisterSet::new(); // `GPR` has units 28-36. assert_eq!(regs.iter(GPR).len(), 8); @@ -302,8 +307,8 @@ mod tests { #[test] fn interference() { - let mut regs1 = AllocatableSet::new(); - let mut regs2 = AllocatableSet::new(); + let mut regs1 = RegisterSet::new(); + let mut regs2 = RegisterSet::new(); assert!(!regs1.interferes_with(®s2)); regs1.take(&GPR, 32); diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/cretonne/src/regalloc/solver.rs index 4bac41aa99..218f0a2600 100644 --- a/lib/cretonne/src/regalloc/solver.rs +++ b/lib/cretonne/src/regalloc/solver.rs @@ -98,12 +98,12 @@ //! appropriate candidate among the set of live register values, add it as a variable and start //! over. -use super::AllocatableSet; +use super::RegisterSet; use dbg::DisplayList; use entity::{SparseMap, SparseMapValue}; use ir::Value; use isa::{RegClass, RegUnit}; -use regalloc::allocatable_set::RegSetIter; +use regalloc::register_set::RegSetIter; use std::cmp; use std::fmt; use std::mem; @@ -184,12 +184,7 @@ impl Variable { /// Get an iterator over possible register choices, given the available registers on the input /// and output sides as well as the available global register set. - fn iter( - &self, - iregs: &AllocatableSet, - oregs: &AllocatableSet, - gregs: &AllocatableSet, - ) -> RegSetIter { + fn iter(&self, iregs: &RegisterSet, oregs: &RegisterSet, gregs: &RegisterSet) -> RegSetIter { if !self.is_output { debug_assert!(!self.is_global, "Global implies output"); debug_assert!(self.is_input, "Missing interference set"); @@ -476,7 +471,7 @@ pub struct Solver { /// - The 'to' registers of fixed input reassignments are marked as unavailable. /// - Input-side variables are marked as available. /// - regs_in: AllocatableSet, + regs_in: RegisterSet, /// Available registers on the output side of the instruction / fixed input scratch space. /// @@ -490,7 +485,7 @@ pub struct Solver { /// - Fixed output assignments are marked as unavailable. /// - Live-through variables are marked as available. /// - regs_out: AllocatableSet, + regs_out: RegisterSet, /// List of register moves scheduled to avoid conflicts. /// @@ -509,8 +504,8 @@ impl Solver { assignments: SparseMap::new(), vars: Vec::new(), inputs_done: false, - regs_in: AllocatableSet::new(), - regs_out: AllocatableSet::new(), + regs_in: RegisterSet::new(), + regs_out: RegisterSet::new(), moves: Vec::new(), fills: Vec::new(), } @@ -521,8 +516,8 @@ impl Solver { self.assignments.clear(); self.vars.clear(); self.inputs_done = false; - self.regs_in = AllocatableSet::new(); - self.regs_out = AllocatableSet::new(); + self.regs_in = RegisterSet::new(); + self.regs_out = RegisterSet::new(); self.moves.clear(); self.fills.clear(); } @@ -531,13 +526,13 @@ impl Solver { /// allocatable registers. /// /// The `regs` set is the allocatable registers before any reassignments are applied. - pub fn reset(&mut self, regs: &AllocatableSet) { + pub fn reset(&mut self, regs: &RegisterSet) { self.assignments.clear(); self.vars.clear(); self.inputs_done = false; self.regs_in = regs.clone(); // Used for tracking fixed input assignments while `!inputs_done`: - self.regs_out = AllocatableSet::new(); + self.regs_out = RegisterSet::new(); self.moves.clear(); self.fills.clear(); } @@ -870,10 +865,7 @@ impl Solver { /// always trivial. /// /// Returns `Ok(regs)` if a solution was found. - pub fn quick_solve( - &mut self, - global_regs: &AllocatableSet, - ) -> Result { + pub fn quick_solve(&mut self, global_regs: &RegisterSet) -> Result { self.find_solution(global_regs) } @@ -884,10 +876,7 @@ impl Solver { /// This may return an error with a register class that has run out of registers. If registers /// can be freed up in the starving class, this method can be called again after adding /// variables for the freed registers. - pub fn real_solve( - &mut self, - global_regs: &AllocatableSet, - ) -> Result { + pub fn real_solve(&mut self, global_regs: &RegisterSet) -> Result { // Compute domain sizes for all the variables given the current register sets. for v in &mut self.vars { let d = v.iter(&self.regs_in, &self.regs_out, global_regs).len(); @@ -933,10 +922,7 @@ impl Solver { /// If a solution was found, returns `Ok(regs)` with the set of available registers on the /// output side after the solution. If no solution could be found, returns `Err(rc)` with the /// constraint register class that needs more available registers. - fn find_solution( - &mut self, - global_regs: &AllocatableSet, - ) -> Result { + fn find_solution(&mut self, global_regs: &RegisterSet) -> Result { // Available registers on the input and output sides respectively. let mut iregs = self.regs_in.clone(); let mut oregs = self.regs_out.clone(); @@ -1025,7 +1011,7 @@ impl Solver { /// a register. /// /// Returns the number of spills that had to be emitted. - pub fn schedule_moves(&mut self, regs: &AllocatableSet) -> usize { + pub fn schedule_moves(&mut self, regs: &RegisterSet) -> usize { self.collect_moves(); debug_assert!(self.fills.is_empty()); @@ -1162,7 +1148,7 @@ mod tests { use entity::EntityRef; use ir::Value; use isa::{RegClass, RegInfo, RegUnit, TargetIsa}; - use regalloc::AllocatableSet; + use regalloc::RegisterSet; use std::boxed::Box; // Make an arm32 `TargetIsa`, if possible. @@ -1219,8 +1205,8 @@ mod tests { let r0 = gpr.unit(0); let r1 = gpr.unit(1); let r2 = gpr.unit(2); - let gregs = AllocatableSet::new(); - let mut regs = AllocatableSet::new(); + let gregs = RegisterSet::new(); + let mut regs = RegisterSet::new(); let mut solver = Solver::new(); let v10 = Value::new(10); let v11 = Value::new(11); @@ -1277,8 +1263,8 @@ mod tests { let s1 = s.unit(1); let s2 = s.unit(2); let s3 = s.unit(3); - let gregs = AllocatableSet::new(); - let mut regs = AllocatableSet::new(); + let gregs = RegisterSet::new(); + let mut regs = RegisterSet::new(); let mut solver = Solver::new(); let v10 = Value::new(10); let v11 = Value::new(11); @@ -1337,8 +1323,8 @@ mod tests { let r3 = gpr.unit(3); let r4 = gpr.unit(4); let r5 = gpr.unit(5); - let gregs = AllocatableSet::new(); - let mut regs = AllocatableSet::new(); + let gregs = RegisterSet::new(); + let mut regs = RegisterSet::new(); let mut solver = Solver::new(); let v10 = Value::new(10); let v11 = Value::new(11); From e2681a63034a508091f4970fba3d9837046abcfd Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Apr 2018 15:48:14 -0700 Subject: [PATCH 1680/3084] Tidy up an unneeded `mut`. --- lib/cretonne/src/preopt.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cretonne/src/preopt.rs b/lib/cretonne/src/preopt.rs index 0a1c13d6ea..cbe7484fd0 100644 --- a/lib/cretonne/src/preopt.rs +++ b/lib/cretonne/src/preopt.rs @@ -493,7 +493,7 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) { } else if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(args[0]) { if let InstructionData::UnaryImm { opcode: Opcode::Iconst, - mut imm, + imm, } = pos.func.dfg[iconst_inst] { let new_opcode = match opcode { From c47fa576b297b0eea6c1f0adb30c1673474ff099 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Apr 2018 16:00:29 -0700 Subject: [PATCH 1681/3084] Convert some http links to https. --- cranelift/docs/compare-llvm.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst index 4a622dbeda..971585cebb 100644 --- a/cranelift/docs/compare-llvm.rst +++ b/cranelift/docs/compare-llvm.rst @@ -6,8 +6,8 @@ Cretonne compared to LLVM a set of C++ libraries. It can be used to build both JIT compilers and static compilers like `Clang `_, and it is deservedly very popular. `Chris Lattner's chapter about LLVM -`_ in the `Architecture of Open Source -Applications `_ book gives an excellent +`_ in the `Architecture of Open Source +Applications `_ book gives an excellent overview of the architecture and design of LLVM. Cretonne and LLVM are superficially similar projects, so it is worth From 781d3ee3ffe527409b56f645b581301b81b0d63f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Apr 2018 16:04:54 -0700 Subject: [PATCH 1682/3084] Bump version to 0.4.2 --- cranelift/Cargo.toml | 14 +++++++------- cranelift/publish-all.sh | 2 +- lib/cretonne/Cargo.toml | 2 +- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/wasm/Cargo.toml | 6 +++--- 8 files changed, 21 insertions(+), 21 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 947b50b44a..73fab18586 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.4.1" +version = "0.4.2" description = "Binaries for testing the Cretonne library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,12 +13,12 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne = { path = "lib/cretonne", version = "0.4.1" } -cretonne-reader = { path = "lib/reader", version = "0.4.1" } -cretonne-frontend = { path = "lib/frontend", version = "0.4.1" } -cretonne-wasm = { path = "lib/wasm", version = "0.4.1" } -cretonne-native = { path = "lib/native", version = "0.4.1" } -cretonne-filetests = { path = "lib/filetests", version = "0.4.1" } +cretonne = { path = "lib/cretonne", version = "0.4.2" } +cretonne-reader = { path = "lib/reader", version = "0.4.2" } +cretonne-frontend = { path = "lib/frontend", version = "0.4.2" } +cretonne-wasm = { path = "lib/wasm", version = "0.4.2" } +cretonne-native = { path = "lib/native", version = "0.4.2" } +cretonne-filetests = { path = "lib/filetests", version = "0.4.2" } filecheck = "0.2.1" docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index a29f308929..4ac7e2bcff 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ cd $(dirname "$0") topdir="$(pwd)" # All the cretonne-* crates have the same version number -version="0.4.1" +version="0.4.2" # Update all of the Cargo.toml files. # diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 66cd819715..cf12060989 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.4.1" +version = "0.4.2" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 0e37c1ef8d..48740ece84 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.4.1" +version = "0.4.2" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "http://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -12,7 +12,7 @@ publish = false name = "cton_filetests" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.1" } -cretonne-reader = { path = "../reader", version = "0.4.1" } +cretonne = { path = "../cretonne", version = "0.4.2" } +cretonne-reader = { path = "../reader", version = "0.4.2" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 0bef102c66..f63f4e7daf 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.4.1" +version = "0.4.2" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,7 +12,7 @@ readme = "README.md" name = "cton_frontend" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.1" } +cretonne = { path = "../cretonne", version = "0.4.2" } [badges] maintenance = { status = "experimental" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 31dbdb0708..3d959a700e 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.4.1" +version = "0.4.2" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -11,7 +11,7 @@ readme = "README.md" name = "cton_native" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.1" } +cretonne = { path = "../cretonne", version = "0.4.2" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index d8259a2027..a8c3c1ffd1 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.4.1" +version = "0.4.2" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,7 +12,7 @@ readme = "README.md" name = "cton_reader" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.1" } +cretonne = { path = "../cretonne", version = "0.4.2" } [badges] maintenance = { status = "experimental" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index b532f29c55..c8b9c9f6f5 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.4.1" +version = "0.4.2" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/Cretonne/cretonne" @@ -13,8 +13,8 @@ name = "cton_wasm" [dependencies] wasmparser = "0.15.1" -cretonne = { path = "../cretonne", version = "0.4.1" } -cretonne-frontend = { path = "../frontend", version = "0.4.1" } +cretonne = { path = "../cretonne", version = "0.4.2" } +cretonne-frontend = { path = "../frontend", version = "0.4.2" } [dev-dependencies] tempdir = "0.3.5" From b0d414731c89e1ea28c0c4db78ba5bd0d600bb53 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 9 Apr 2018 06:15:33 -0700 Subject: [PATCH 1683/3084] The addend for a PCRel4 reloc should be -4 too. --- cranelift/filetests/isa/intel/binary32.cton | 2 +- cranelift/filetests/isa/intel/binary64.cton | 2 +- lib/cretonne/meta/isa/intel/recipes.py | 4 +++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/intel/binary32.cton index 2567fbe88c..c6bff52a64 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/intel/binary32.cton @@ -352,7 +352,7 @@ ebb0: [-,%rsi] v351 = bint.i32 v301 ; bin: 0f b6 f2 ; asm: call foo - call fn0() ; bin: e8 PCRel4(%foo) 00000000 + call fn0() ; bin: e8 PCRel4(%foo-4) 00000000 ; asm: movl $0, %ecx [-,%rcx] v400 = func_addr.i32 fn0 ; bin: b9 Abs4(%foo) 00000000 diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 1a47dc1cba..859c7d0d12 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -474,7 +474,7 @@ ebb0: [-,%rsi] v351 = bint.i64 v301 ; bin: 0f b6 f2 ; asm: call foo - call fn0() ; bin: e8 PCRel4(%foo) 00000000 + call fn0() ; bin: e8 PCRel4(%foo-4) 00000000 ; asm: movabsq $0, %rcx [-,%rcx] v400 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(%foo) 0000000000000000 diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/intel/recipes.py index 54df437c67..5f36e20291 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/intel/recipes.py @@ -1007,9 +1007,11 @@ call_id = TailRecipe( 'call_id', Call, size=4, ins=(), outs=(), emit=''' PUT_OP(bits, BASE_REX, sink); + // The addend adjusts for the difference between the end of the + // instruction and the beginning of the immediate field. sink.reloc_external(Reloc::IntelPCRel4, &func.dfg.ext_funcs[func_ref].name, - 0); + -4); sink.put4(0); ''') From 2703b8ce6f962e59b99d1548040624b073319c4a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 9 Apr 2018 10:33:00 -0700 Subject: [PATCH 1684/3084] The current x86-32 encodings for symbolic addresses are non-PIC. --- lib/cretonne/meta/isa/intel/encodings.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 0fc507bac5..4d4fa12ced 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -293,12 +293,12 @@ enc_both(base.regspill.f64, r.fregspill32, 0xf2, 0x0f, 0x11) # X86_32.enc(base.func_addr.i32, *r.fnaddr4(0xb8), - isap=Not(allones_funcaddrs)) + isap=And(Not(allones_funcaddrs), Not(is_pic))) X86_64.enc(base.func_addr.i64, *r.fnaddr8.rex(0xb8, w=1), isap=And(Not(allones_funcaddrs), Not(is_pic))) X86_32.enc(base.func_addr.i32, *r.allones_fnaddr4(0xb8), - isap=allones_funcaddrs) + isap=And(allones_funcaddrs, Not(is_pic))) X86_64.enc(base.func_addr.i64, *r.allones_fnaddr8.rex(0xb8, w=1), isap=And(allones_funcaddrs, Not(is_pic))) @@ -309,7 +309,8 @@ X86_64.enc(base.func_addr.i64, *r.got_fnaddr8.rex(0x8b, w=1), # Global addresses. # -X86_32.enc(base.globalsym_addr.i32, *r.gvaddr4(0xb8)) +X86_32.enc(base.globalsym_addr.i32, *r.gvaddr4(0xb8), + isap=Not(is_pic)) X86_64.enc(base.globalsym_addr.i64, *r.gvaddr8.rex(0xb8, w=1), isap=Not(is_pic)) From eab57c0a408a2726f887a2aab2d4873b3c03e655 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 9 Apr 2018 14:53:00 -0700 Subject: [PATCH 1685/3084] Use large-model addressing for calls when in non-PIC mode. The main use for non-PIC code at present is JIT code, and JIT code can live anywhere in memory and reference other symbols defined anywhere in memory, so it needs to use the "large" code model. func_addr and globalsym_addr instructions were already using `movabs` to support arbitrary 64-bit addresses, so this just makes calls be legalized to support arbitrary 64-bit addresses also. --- cranelift/filetests/isa/intel/binary64.cton | 7 ++- .../filetests/isa/intel/legalize-libcall.cton | 1 + .../isa/intel/prologue-epilogue.cton | 1 + lib/cretonne/meta/base/legalize.py | 3 + lib/cretonne/meta/isa/intel/encodings.py | 1 - lib/cretonne/src/legalizer/call.rs | 60 +++++++++++++++++++ lib/cretonne/src/legalizer/mod.rs | 2 + 7 files changed, 72 insertions(+), 3 deletions(-) create mode 100644 lib/cretonne/src/legalizer/call.rs diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/intel/binary64.cton index 859c7d0d12..d2fab8e19d 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/intel/binary64.cton @@ -473,8 +473,11 @@ ebb0: ; asm: movzbq %dl, %rsi [-,%rsi] v351 = bint.i64 v301 ; bin: 0f b6 f2 - ; asm: call foo - call fn0() ; bin: e8 PCRel4(%foo-4) 00000000 + ; TODO: x86-64 can't encode a direct call to an arbitrary 64-bit address in + ; a single instruction. When we add a concept of colocated definitions, this + ; test can be re-enabled. + ; disabled: asm: call foo + ; disabled: call fn0() ; bin: e8 PCRel4(%foo-4) 00000000 ; asm: movabsq $0, %rcx [-,%rcx] v400 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(%foo) 0000000000000000 diff --git a/cranelift/filetests/isa/intel/legalize-libcall.cton b/cranelift/filetests/isa/intel/legalize-libcall.cton index c15587fc7f..239b3e5a57 100644 --- a/cranelift/filetests/isa/intel/legalize-libcall.cton +++ b/cranelift/filetests/isa/intel/legalize-libcall.cton @@ -2,6 +2,7 @@ test legalizer ; Pre-SSE 4.1, we need to use runtime library calls for floating point rounding operations. set is_64bit +set is_pic isa intel function %floor(f32) -> f32 { diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/intel/prologue-epilogue.cton index 416f90df63..0eadcd27d7 100644 --- a/cranelift/filetests/isa/intel/prologue-epilogue.cton +++ b/cranelift/filetests/isa/intel/prologue-epilogue.cton @@ -1,6 +1,7 @@ test compile set is_64bit set is_compressed +set is_pic isa intel haswell ; An empty function. diff --git a/lib/cretonne/meta/base/legalize.py b/lib/cretonne/meta/base/legalize.py index c2392269df..95a58987d3 100644 --- a/lib/cretonne/meta/base/legalize.py +++ b/lib/cretonne/meta/base/legalize.py @@ -65,6 +65,9 @@ expand_flags = XFormGroup('expand_flags', """ expand.custom_legalize(insts.global_addr, 'expand_global_addr') expand.custom_legalize(insts.heap_addr, 'expand_heap_addr') +# Custom expansions for calls. +expand.custom_legalize(insts.call, 'expand_call') + # Custom expansions that need to change the CFG. # TODO: Add sufficient XForm syntax that we don't need to hand-code these. expand.custom_legalize(insts.trapz, 'expand_cond_trap') diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/intel/encodings.py index 4d4fa12ced..c7833d04bc 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/intel/encodings.py @@ -321,7 +321,6 @@ X86_64.enc(base.globalsym_addr.i64, *r.got_gvaddr8.rex(0x8b, w=1), # Call/return # X86_32.enc(base.call, *r.call_id(0xe8)) -X86_64.enc(base.call, *r.call_id(0xe8), isap=Not(is_pic)) X86_64.enc(base.call, *r.call_plt_id(0xe8), isap=is_pic) X86_32.enc(base.call_indirect.i32, *r.call_r(0xff, rrr=2)) diff --git a/lib/cretonne/src/legalizer/call.rs b/lib/cretonne/src/legalizer/call.rs new file mode 100644 index 0000000000..d1d8bcba08 --- /dev/null +++ b/lib/cretonne/src/legalizer/call.rs @@ -0,0 +1,60 @@ +//! Legalization of calls. +//! +//! This module exports the `expand_call` function which transforms a `call` +//! instruction into `func_addr` and `call_indirect` instructions. + +use cursor::{Cursor, FuncCursor}; +use flowgraph::ControlFlowGraph; +use ir::{self, InstBuilder}; +use isa::TargetIsa; + +/// Expand a `call` instruction. +pub fn expand_call( + inst: ir::Inst, + func: &mut ir::Function, + _cfg: &mut ControlFlowGraph, + isa: &TargetIsa, +) { + // Unpack the instruction. + let (func_ref, old_args) = match func.dfg[inst] { + ir::InstructionData::Call { + opcode, + ref args, + func_ref, + } => { + debug_assert_eq!(opcode, ir::Opcode::Call); + (func_ref, args.clone()) + } + _ => panic!("Wanted call: {}", func.dfg.display_inst(inst, None)), + }; + + let ptr_ty = if isa.flags().is_64bit() { + ir::types::I64 + } else { + ir::types::I32 + }; + + let sig = func.dfg.ext_funcs[func_ref].signature; + + let callee = { + let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); + pos.ins().func_addr(ptr_ty, func_ref) + }; + + let mut new_args = ir::ValueList::default(); + new_args.push(callee, &mut func.dfg.value_lists); + for i in 0..old_args.len(&func.dfg.value_lists) { + new_args.push( + old_args.as_slice(&func.dfg.value_lists)[i], + &mut func.dfg.value_lists, + ); + } + + func.dfg.replace(inst).IndirectCall( + ir::Opcode::CallIndirect, + ptr_ty, + sig, + new_args, + ); +} diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/cretonne/src/legalizer/mod.rs index 23b3f56c89..4486d5df4e 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/cretonne/src/legalizer/mod.rs @@ -21,6 +21,7 @@ use isa::TargetIsa; use timing; mod boundary; +mod call; mod globalvar; mod heap; mod libcall; @@ -28,6 +29,7 @@ mod split; use self::globalvar::expand_global_addr; use self::heap::expand_heap_addr; +use self::call::expand_call; use self::libcall::expand_as_libcall; /// Legalize `func` for `isa`. From 18b2f1215096eec0cd7f1d8f813bd55bfce724c3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 9 Apr 2018 22:33:54 -0700 Subject: [PATCH 1686/3084] Save/restore callee-saved registers used in regmove/regfill. The regmove and regfill instructions temporarily divert a value's location, and these temporary diversions are not reflected in `func.locations`. For now, make an extra scan through the instructions of the function to find any regmove or regfill instructions in order to find all used callee-saved registers. This fixes #296. --- .../isa/intel/prologue-epilogue.cton | 22 +++++++++++++++++++ lib/cretonne/src/isa/intel/abi.rs | 20 +++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/intel/prologue-epilogue.cton index 0eadcd27d7..ef42836a5a 100644 --- a/cranelift/filetests/isa/intel/prologue-epilogue.cton +++ b/cranelift/filetests/isa/intel/prologue-epilogue.cton @@ -207,3 +207,25 @@ ebb0(v0: i64, v1: i64): ; nextln: v54 = x86_pop.i64 ; nextln: return v54, v55, v56, v57, v58, v59 ; nextln: } + +; A function which uses diverted registers. + +function %divert(i32) -> i32 system_v { +ebb0(v0: i32): + v2 = iconst.i32 0 + v3 = iconst.i32 1 + jump ebb3(v0, v3, v2) + +ebb3(v4: i32, v5: i32, v6: i32): + brz v4, ebb4 + v7 = iadd v5, v6 + v8 = iadd_imm v4, -1 + jump ebb3(v8, v7, v5) + +ebb4: + return v5 +} + +; check: function %divert +; check: regmove v5, %rcx -> %rbx +; check: [RexOp1popq#58,%rbx] v15 = x86_pop.i64 diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/intel/abi.rs index f1abd1cb42..488f8a3d49 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/intel/abi.rs @@ -184,6 +184,26 @@ fn callee_saved_gprs_used(flags: &shared_settings::Flags, func: &ir::Function) - } } + // regmove and regfill instructions may temporarily divert values into other registers, + // and these are not reflected in `func.locations`. Scan the function for such instructions + // and note which callee-saved registers they use. + // + // TODO: Consider re-evaluating how regmove/regfill/regspill work and whether it's possible + // to avoid this step. + for ebb in &func.layout { + for inst in func.layout.ebb_insts(ebb) { + match func.dfg[inst] { + ir::instructions::InstructionData::RegMove { dst, .. } | + ir::instructions::InstructionData::RegFill { dst, .. } => { + if !used.is_avail(GPR, dst) { + used.free(GPR, dst); + } + } + _ => (), + } + } + } + used.intersect(&all_callee_saved); return used; } From 9e17e62d68e9e453893fbebdf75cf51dcc085adb Mon Sep 17 00:00:00 2001 From: Amanieu d'Antras Date: Tue, 10 Apr 2018 19:06:49 +0100 Subject: [PATCH 1687/3084] Move entity into a separate crate (#297) --- lib/cretonne/Cargo.toml | 1 + lib/cretonne/src/lib.rs | 8 ++- lib/entity/Cargo.toml | 21 ++++++ lib/entity/README.md | 2 + .../src/entity => entity/src}/iter.rs | 2 +- .../src/entity => entity/src}/keys.rs | 2 +- .../src/entity/mod.rs => entity/src/lib.rs} | 69 ++++++++++++------- .../src/entity => entity/src}/list.rs | 10 ++- .../src/entity => entity/src}/map.rs | 2 +- lib/{cretonne => entity}/src/packed_option.rs | 0 .../src/entity => entity/src}/primary.rs | 2 +- .../src/entity => entity/src}/set.rs | 2 +- .../src/entity => entity/src}/sparse.rs | 10 ++- 13 files changed, 94 insertions(+), 37 deletions(-) create mode 100644 lib/entity/Cargo.toml create mode 100644 lib/entity/README.md rename lib/{cretonne/src/entity => entity/src}/iter.rs (99%) rename lib/{cretonne/src/entity => entity/src}/keys.rs (98%) rename lib/{cretonne/src/entity/mod.rs => entity/src/lib.rs} (71%) rename lib/{cretonne/src/entity => entity/src}/list.rs (99%) rename lib/{cretonne/src/entity => entity/src}/map.rs (98%) rename lib/{cretonne => entity}/src/packed_option.rs (100%) rename lib/{cretonne/src/entity => entity/src}/primary.rs (99%) rename lib/{cretonne/src/entity => entity/src}/set.rs (99%) rename lib/{cretonne/src/entity => entity/src}/sparse.rs (98%) diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index cf12060989..4540a783a6 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -14,6 +14,7 @@ build = "build.rs" name = "cretonne" [dependencies] +cretonne-entity = { path = "../entity" } # It is a goal of the cretonne crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 99246defe3..996bb865bd 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -39,9 +39,10 @@ pub use write::write_function; pub const VERSION: &str = env!("CARGO_PKG_VERSION"); #[macro_use] -pub mod dbg; +pub extern crate cton_entity as entity; + #[macro_use] -pub mod entity; +pub mod dbg; pub mod bforest; pub mod binemit; @@ -52,13 +53,14 @@ pub mod flowgraph; pub mod ir; pub mod isa; pub mod loop_analysis; -pub mod packed_option; pub mod print_errors; pub mod result; pub mod settings; pub mod timing; pub mod verifier; +pub use entity::packed_option; + mod abi; mod bitset; mod constant_hash; diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml new file mode 100644 index 0000000000..c86962c809 --- /dev/null +++ b/lib/entity/Cargo.toml @@ -0,0 +1,21 @@ +[package] +authors = ["The Cretonne Project Developers"] +name = "cretonne-entity" +version = "0.4.2" +description = "Data structures using entity references as mapping keys" +license = "Apache-2.0" +documentation = "https://cretonne.readthedocs.io/" +repository = "https://github.com/Cretonne/cretonne" +readme = "README.md" +keywords = ["entity", "set", "map"] + +[lib] +name = "cton_entity" + +[features] +default = ["std"] +std = [] + +[badges] +maintenance = { status = "experimental" } +travis-ci = { repository = "Cretonne/cretonne" } diff --git a/lib/entity/README.md b/lib/entity/README.md new file mode 100644 index 0000000000..abe3b6210c --- /dev/null +++ b/lib/entity/README.md @@ -0,0 +1,2 @@ +This crate contains array-based data structures used by the core Cretonne code +generator which use densely numbered entity references as mapping keys. diff --git a/lib/cretonne/src/entity/iter.rs b/lib/entity/src/iter.rs similarity index 99% rename from lib/cretonne/src/entity/iter.rs rename to lib/entity/src/iter.rs index 156021ee86..9eeba6c7a6 100644 --- a/lib/cretonne/src/entity/iter.rs +++ b/lib/entity/src/iter.rs @@ -1,6 +1,6 @@ //! A double-ended iterator over entity references and entities. -use entity::EntityRef; +use EntityRef; use std::marker::PhantomData; use std::slice; diff --git a/lib/cretonne/src/entity/keys.rs b/lib/entity/src/keys.rs similarity index 98% rename from lib/cretonne/src/entity/keys.rs rename to lib/entity/src/keys.rs index 3ad5b660b6..52481097cb 100644 --- a/lib/cretonne/src/entity/keys.rs +++ b/lib/entity/src/keys.rs @@ -1,6 +1,6 @@ //! A double-ended iterator over entity references. -use entity::EntityRef; +use EntityRef; use std::marker::PhantomData; /// Iterate over all keys in order. diff --git a/lib/cretonne/src/entity/mod.rs b/lib/entity/src/lib.rs similarity index 71% rename from lib/cretonne/src/entity/mod.rs rename to lib/entity/src/lib.rs index 88c0977cf2..00b6412c0d 100644 --- a/lib/cretonne/src/entity/mod.rs +++ b/lib/entity/src/lib.rs @@ -1,6 +1,6 @@ //! Array-based data structures using densely numbered entity references as mapping keys. //! -//! This module defines a number of data structures based on arrays. The arrays are not indexed by +//! This crate defines a number of data structures based on arrays. The arrays are not indexed by //! `usize` as usual, but by *entity references* which are integers wrapped in new-types. This has //! a couple advantages: //! @@ -29,21 +29,26 @@ //! references allocated from an associated memory pool. It has a much smaller footprint than //! `Vec`. -mod iter; -mod keys; -mod list; -mod map; -mod primary; -mod set; -mod sparse; +#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", + allow(new_without_default, new_without_default_derive, redundant_field_names))] -pub use self::iter::{Iter, IterMut}; -pub use self::keys::Keys; -pub use self::list::{EntityList, ListPool}; -pub use self::map::EntityMap; -pub use self::primary::PrimaryMap; -pub use self::set::EntitySet; -pub use self::sparse::{SparseMap, SparseMapValue, SparseSet}; +// Turns on no_std and alloc features if std is not available. +#![cfg_attr(not(feature = "std"), no_std)] +#![cfg_attr(not(feature = "std"), feature(alloc))] + +/// This replaces `std` in builds with `core`. +#[cfg(not(feature = "std"))] +mod std { + extern crate alloc; + pub use self::alloc::{boxed, string, vec}; + pub use core::*; +} + +// Re-export core so that the macros works with both std and no_std crates +#[doc(hidden)] +pub extern crate core as __core; /// A type wrapping a small integer index should implement `EntityRef` so it can be used as the key /// of an `EntityMap` or `SparseMap`. @@ -61,9 +66,9 @@ pub trait EntityRef: Copy + Eq { macro_rules! entity_impl { // Basic traits. ($entity:ident) => { - impl $crate::entity::EntityRef for $entity { + impl $crate::EntityRef for $entity { fn new(index: usize) -> Self { - debug_assert!(index < (::std::u32::MAX as usize)); + debug_assert!(index < ($crate::__core::u32::MAX as usize)); $entity(index as u32) } @@ -74,7 +79,7 @@ macro_rules! entity_impl { impl $crate::packed_option::ReservedValue for $entity { fn reserved_value() -> $entity { - $entity(::std::u32::MAX) + $entity($crate::__core::u32::MAX) } } }; @@ -84,16 +89,34 @@ macro_rules! entity_impl { ($entity:ident, $display_prefix:expr) => { entity_impl!($entity); - impl ::std::fmt::Display for $entity { - fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result { + impl $crate::__core::fmt::Display for $entity { + fn fmt(&self, f: &mut $crate::__core::fmt::Formatter) -> $crate::__core::fmt::Result { write!(f, "{}{}", $display_prefix, self.0) } } - impl ::std::fmt::Debug for $entity { - fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result { - (self as &::std::fmt::Display).fmt(f) + impl $crate::__core::fmt::Debug for $entity { + fn fmt(&self, f: &mut $crate::__core::fmt::Formatter) -> $crate::__core::fmt::Result { + (self as &$crate::__core::fmt::Display).fmt(f) } } }; } + +pub mod packed_option; + +mod iter; +mod keys; +mod list; +mod map; +mod primary; +mod set; +mod sparse; + +pub use self::iter::{Iter, IterMut}; +pub use self::keys::Keys; +pub use self::list::{EntityList, ListPool}; +pub use self::map::EntityMap; +pub use self::primary::PrimaryMap; +pub use self::set::EntitySet; +pub use self::sparse::{SparseMap, SparseMapValue, SparseSet}; diff --git a/lib/cretonne/src/entity/list.rs b/lib/entity/src/list.rs similarity index 99% rename from lib/cretonne/src/entity/list.rs rename to lib/entity/src/list.rs index 545b1e146b..803504bc76 100644 --- a/lib/cretonne/src/entity/list.rs +++ b/lib/entity/src/list.rs @@ -1,5 +1,5 @@ //! Small lists of entity references. -use entity::EntityRef; +use EntityRef; use std::hash::{Hash, Hasher}; use std::marker::PhantomData; use std::mem; @@ -481,8 +481,12 @@ impl EntityList { mod tests { use super::*; use super::{sclass_for_length, sclass_size}; - use entity::EntityRef; - use ir::Inst; + use EntityRef; + + /// An opaque reference to an instruction in a function. + #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] + pub struct Inst(u32); + entity_impl!(Inst, "inst"); #[test] fn size_classes() { diff --git a/lib/cretonne/src/entity/map.rs b/lib/entity/src/map.rs similarity index 98% rename from lib/cretonne/src/entity/map.rs rename to lib/entity/src/map.rs index ff402b104f..8256ba9c90 100644 --- a/lib/cretonne/src/entity/map.rs +++ b/lib/entity/src/map.rs @@ -1,6 +1,6 @@ //! Densely numbered entity references as mapping keys. -use entity::{EntityRef, Iter, IterMut, Keys}; +use {EntityRef, Iter, IterMut, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; use std::slice; diff --git a/lib/cretonne/src/packed_option.rs b/lib/entity/src/packed_option.rs similarity index 100% rename from lib/cretonne/src/packed_option.rs rename to lib/entity/src/packed_option.rs diff --git a/lib/cretonne/src/entity/primary.rs b/lib/entity/src/primary.rs similarity index 99% rename from lib/cretonne/src/entity/primary.rs rename to lib/entity/src/primary.rs index 5d690a7553..805ccc6866 100644 --- a/lib/cretonne/src/entity/primary.rs +++ b/lib/entity/src/primary.rs @@ -1,5 +1,5 @@ //! Densely numbered entity references as mapping keys. -use entity::{EntityRef, Iter, IterMut, Keys}; +use {EntityRef, Iter, IterMut, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; use std::slice; diff --git a/lib/cretonne/src/entity/set.rs b/lib/entity/src/set.rs similarity index 99% rename from lib/cretonne/src/entity/set.rs rename to lib/entity/src/set.rs index e4acf47723..392813401c 100644 --- a/lib/cretonne/src/entity/set.rs +++ b/lib/entity/src/set.rs @@ -1,6 +1,6 @@ //! Densely numbered entity references as set keys. -use entity::{EntityRef, Keys}; +use {EntityRef, Keys}; use std::marker::PhantomData; use std::vec::Vec; diff --git a/lib/cretonne/src/entity/sparse.rs b/lib/entity/src/sparse.rs similarity index 98% rename from lib/cretonne/src/entity/sparse.rs rename to lib/entity/src/sparse.rs index 6894c89498..95d0bba41b 100644 --- a/lib/cretonne/src/entity/sparse.rs +++ b/lib/entity/src/sparse.rs @@ -7,7 +7,7 @@ //! > Briggs, Torczon, *An efficient representation for sparse sets*, //! ACM Letters on Programming Languages and Systems, Volume 2, Issue 1-4, March-Dec. 1993. -use entity::{EntityMap, EntityRef}; +use {EntityMap, EntityRef}; use std::mem; use std::slice; use std::u32; @@ -229,8 +229,12 @@ pub type SparseSet = SparseMap; #[cfg(test)] mod tests { use super::*; - use entity::EntityRef; - use ir::Inst; + use EntityRef; + + /// An opaque reference to an instruction in a function. + #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] + pub struct Inst(u32); + entity_impl!(Inst, "inst"); // Mock key-value object for testing. #[derive(PartialEq, Eq, Debug)] From 1c760ab179cfc2965ab416aedf4d99076ef75b07 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 12 Apr 2018 15:23:39 -0700 Subject: [PATCH 1688/3084] Rename intel to x86. x86 is the more accurate name, as there are non-Intel x86 implementations. Fixes #263. --- cranelift/docs/compare-llvm.rst | 2 +- cranelift/docs/langref.rst | 24 +++++----- cranelift/docs/metaref.rst | 4 +- cranelift/docs/regalloc.rst | 4 +- .../isa/{intel => x86}/abi-bool.cton | 2 +- .../filetests/isa/{intel => x86}/abi32.cton | 2 +- .../filetests/isa/{intel => x86}/abi64.cton | 2 +- .../{intel => x86}/allones_funcaddrs32.cton | 4 +- .../{intel => x86}/allones_funcaddrs64.cton | 4 +- .../baseline_clz_ctz_popcount.cton | 2 +- .../baseline_clz_ctz_popcount_encoding.cton | 4 +- .../isa/{intel => x86}/binary32-float.cton | 4 +- .../isa/{intel => x86}/binary32.cton | 4 +- .../isa/{intel => x86}/binary64-float.cton | 4 +- .../isa/{intel => x86}/binary64-pic.cton | 4 +- .../isa/{intel => x86}/binary64.cton | 4 +- .../isa/{intel => x86}/legalize-custom.cton | 4 +- .../{intel => x86}/legalize-div-traps.cton | 2 +- .../isa/{intel => x86}/legalize-div.cton | 2 +- .../isa/{intel => x86}/legalize-libcall.cton | 2 +- .../isa/{intel => x86}/legalize-memory.cton | 2 +- .../isa/{intel => x86}/legalize-mulhi.cton | 2 +- .../isa/{intel => x86}/prologue-epilogue.cton | 2 +- cranelift/filetests/postopt/basic.cton | 2 +- .../preopt/div_by_const_indirect.cton | 2 +- .../preopt/div_by_const_non_power_of_2.cton | 2 +- .../preopt/div_by_const_power_of_2.cton | 2 +- .../preopt/rem_by_const_non_power_of_2.cton | 2 +- .../preopt/rem_by_const_power_of_2.cton | 2 +- cranelift/filetests/preopt/simplify.cton | 2 +- cranelift/filetests/regalloc/aliases.cton | 2 +- .../filetests/regalloc/coalescing-207.cton | 2 +- .../filetests/regalloc/coalescing-216.cton | 2 +- .../filetests/regalloc/coloring-227.cton | 2 +- cranelift/filetests/regalloc/constraints.cton | 2 +- cranelift/filetests/regalloc/ghost-param.cton | 2 +- .../regalloc/global-constraints.cton | 4 +- .../filetests/regalloc/global-fixed.cton | 2 +- cranelift/filetests/regalloc/iterate.cton | 2 +- .../filetests/regalloc/multi-constraints.cton | 4 +- .../regalloc/output-interference.cton | 4 +- cranelift/filetests/regalloc/reload-208.cton | 2 +- .../filetests/regalloc/schedule-moves.cton | 2 +- .../filetests/regalloc/spill-noregs.cton | 2 +- .../filetests/regalloc/unreachable_code.cton | 2 +- .../{intel-regres.cton => x86-regres.cton} | 2 +- cranelift/filetests/verifier/flags.cton | 2 +- cranelift/filetests/wasm/control.cton | 4 +- cranelift/filetests/wasm/conversions.cton | 2 +- cranelift/filetests/wasm/f32-arith.cton | 4 +- cranelift/filetests/wasm/f32-compares.cton | 4 +- cranelift/filetests/wasm/f32-memory64.cton | 2 +- cranelift/filetests/wasm/f64-arith.cton | 2 +- cranelift/filetests/wasm/f64-compares.cton | 4 +- cranelift/filetests/wasm/f64-memory64.cton | 2 +- cranelift/filetests/wasm/i32-arith.cton | 4 +- cranelift/filetests/wasm/i32-compares.cton | 4 +- cranelift/filetests/wasm/i32-memory64.cton | 2 +- cranelift/filetests/wasm/i64-arith.cton | 2 +- cranelift/filetests/wasm/i64-compares.cton | 2 +- cranelift/filetests/wasm/i64-memory64.cton | 2 +- cranelift/filetests/wasm/select.cton | 4 +- lib/cretonne/build.rs | 8 ++-- lib/cretonne/meta/base/settings.py | 2 +- lib/cretonne/meta/isa/__init__.py | 4 +- lib/cretonne/meta/isa/intel/__init__.py | 23 --------- lib/cretonne/meta/isa/x86/__init__.py | 21 ++++++++ lib/cretonne/meta/isa/{intel => x86}/defs.py | 4 +- .../meta/isa/{intel => x86}/encodings.py | 18 +++---- .../meta/isa/{intel => x86}/instructions.py | 10 ++-- .../meta/isa/{intel => x86}/legalize.py | 48 +++++++++---------- .../meta/isa/{intel => x86}/recipes.py | 28 +++++------ .../meta/isa/{intel => x86}/registers.py | 6 +-- .../meta/isa/{intel => x86}/settings.py | 6 +-- lib/cretonne/src/binemit/mod.rs | 30 ++++++------ lib/cretonne/src/binemit/relaxation.rs | 2 +- lib/cretonne/src/ir/stackslot.rs | 5 +- lib/cretonne/src/isa/constraints.rs | 2 +- lib/cretonne/src/isa/mod.rs | 6 +-- lib/cretonne/src/isa/{intel => x86}/abi.rs | 2 +- .../src/isa/{intel => x86}/binemit.rs | 4 +- .../src/isa/{intel => x86}/enc_tables.rs | 10 ++-- lib/cretonne/src/isa/{intel => x86}/mod.rs | 6 +-- .../src/isa/{intel => x86}/registers.rs | 4 +- .../src/isa/{intel => x86}/settings.rs | 6 +-- lib/cretonne/src/regalloc/register_set.rs | 2 +- lib/native/src/lib.rs | 2 +- 87 files changed, 222 insertions(+), 225 deletions(-) rename cranelift/filetests/isa/{intel => x86}/abi-bool.cton (95%) rename cranelift/filetests/isa/{intel => x86}/abi32.cton (97%) rename cranelift/filetests/isa/{intel => x86}/abi64.cton (99%) rename cranelift/filetests/isa/{intel => x86}/allones_funcaddrs32.cton (81%) rename cranelift/filetests/isa/{intel => x86}/allones_funcaddrs64.cton (84%) rename cranelift/filetests/isa/{intel => x86}/baseline_clz_ctz_popcount.cton (98%) rename cranelift/filetests/isa/{intel => x86}/baseline_clz_ctz_popcount_encoding.cton (94%) rename cranelift/filetests/isa/{intel => x86}/binary32-float.cton (99%) rename cranelift/filetests/isa/{intel => x86}/binary32.cton (99%) rename cranelift/filetests/isa/{intel => x86}/binary64-float.cton (99%) rename cranelift/filetests/isa/{intel => x86}/binary64-pic.cton (92%) rename cranelift/filetests/isa/{intel => x86}/binary64.cton (99%) rename cranelift/filetests/isa/{intel => x86}/legalize-custom.cton (98%) rename cranelift/filetests/isa/{intel => x86}/legalize-div-traps.cton (99%) rename cranelift/filetests/isa/{intel => x86}/legalize-div.cton (99%) rename cranelift/filetests/isa/{intel => x86}/legalize-libcall.cton (97%) rename cranelift/filetests/isa/{intel => x86}/legalize-memory.cton (99%) rename cranelift/filetests/isa/{intel => x86}/legalize-mulhi.cton (97%) rename cranelift/filetests/isa/{intel => x86}/prologue-epilogue.cton (99%) rename cranelift/filetests/regalloc/{intel-regres.cton => x86-regres.cton} (99%) delete mode 100644 lib/cretonne/meta/isa/intel/__init__.py create mode 100644 lib/cretonne/meta/isa/x86/__init__.py rename lib/cretonne/meta/isa/{intel => x86}/defs.py (88%) rename lib/cretonne/meta/isa/{intel => x86}/encodings.py (98%) rename lib/cretonne/meta/isa/{intel => x86}/instructions.py (95%) rename lib/cretonne/meta/isa/{intel => x86}/legalize.py (85%) rename lib/cretonne/meta/isa/{intel => x86}/recipes.py (98%) rename lib/cretonne/meta/isa/{intel => x86}/registers.py (94%) rename lib/cretonne/meta/isa/{intel => x86}/settings.py (92%) rename lib/cretonne/src/isa/{intel => x86}/abi.rs (99%) rename lib/cretonne/src/isa/{intel => x86}/binemit.rs (99%) rename lib/cretonne/src/isa/{intel => x86}/enc_tables.rs (98%) rename lib/cretonne/src/isa/{intel => x86}/mod.rs (96%) rename lib/cretonne/src/isa/{intel => x86}/registers.rs (95%) rename lib/cretonne/src/isa/{intel => x86}/settings.rs (92%) diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst index 971585cebb..abc47caaa3 100644 --- a/cranelift/docs/compare-llvm.rst +++ b/cranelift/docs/compare-llvm.rst @@ -174,7 +174,7 @@ is emitted, there are opcodes for every native instruction that can be generated. There is a lot of overlap between different ISAs, so for example the :cton:inst:`iadd_imm` instruction is used by every ISA that can add an immediate integer to a register. A simple RISC ISA like RISC-V can be defined -with only shared instructions, while an Intel ISA needs a number of specific +with only shared instructions, while x86 needs a number of specific instructions to model addressing modes. Undefined behavior diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index edcc039c86..14f44bab53 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -998,20 +998,20 @@ ISA-specific instructions Target ISAs can define supplemental instructions that do not make sense to support generally. -Intel +x86 ----- -Instructions that can only be used by the Intel target ISA. +Instructions that can only be used by the x86 target ISA. -.. autoinst:: isa.intel.instructions.sdivmodx -.. autoinst:: isa.intel.instructions.udivmodx -.. autoinst:: isa.intel.instructions.cvtt2si -.. autoinst:: isa.intel.instructions.fmin -.. autoinst:: isa.intel.instructions.fmax -.. autoinst:: isa.intel.instructions.bsf -.. autoinst:: isa.intel.instructions.bsr -.. autoinst:: isa.intel.instructions.push -.. autoinst:: isa.intel.instructions.pop +.. autoinst:: isa.x86.instructions.sdivmodx +.. autoinst:: isa.x86.instructions.udivmodx +.. autoinst:: isa.x86.instructions.cvtt2si +.. autoinst:: isa.x86.instructions.fmin +.. autoinst:: isa.x86.instructions.fmax +.. autoinst:: isa.x86.instructions.bsf +.. autoinst:: isa.x86.instructions.bsr +.. autoinst:: isa.x86.instructions.push +.. autoinst:: isa.x86.instructions.pop Instruction groups ================== @@ -1023,7 +1023,7 @@ group. Target ISAs may define further instructions in their own instruction groups: -.. autoinstgroup:: isa.intel.instructions.GROUP +.. autoinstgroup:: isa.x86.instructions.GROUP Implementation limits ===================== diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 6f23647b0a..d0b78d3a07 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -400,7 +400,7 @@ Fixed register operands ----------------------- Some instructions use hard-coded input and output registers for some value -operands. An example is the ``pblendvb`` Intel SSE instruction which takes one +operands. An example is the ``pblendvb`` x86 SSE instruction which takes one of its three value operands in the hard-coded ``%xmm0`` register:: XMM0 = FPR[0] @@ -439,7 +439,7 @@ The definitions for each supported target live in a package under :members: .. automodule:: isa.riscv -.. automodule:: isa.intel +.. automodule:: isa.x86 .. automodule:: isa.arm32 .. automodule:: isa.arm64 diff --git a/cranelift/docs/regalloc.rst b/cranelift/docs/regalloc.rst index 74bd6e8575..a87753a34f 100644 --- a/cranelift/docs/regalloc.rst +++ b/cranelift/docs/regalloc.rst @@ -79,7 +79,7 @@ Different register banks Instructions with fixed operands Some instructions use a fixed register for an operand. This happens on the - Intel ISAs: + x86 ISAs: - Dynamic shift and rotate instructions take the shift amount in CL. - Division instructions use RAX and RDX for both input and output operands. @@ -109,7 +109,7 @@ ABI boundaries Aliasing registers Different registers sometimes share the same bits in the register bank. This can make it difficult to measure register pressure. For example, the - Intel registers RAX, EAX, AX, AL, and AH overlap. + x86 registers RAX, EAX, AX, AL, and AH overlap. If only one of the aliasing registers can be used at a time, the aliasing doesn't cause problems since the registers can simply be counted as one diff --git a/cranelift/filetests/isa/intel/abi-bool.cton b/cranelift/filetests/isa/x86/abi-bool.cton similarity index 95% rename from cranelift/filetests/isa/intel/abi-bool.cton rename to cranelift/filetests/isa/x86/abi-bool.cton index 6a53cdbb3c..48ac976a84 100644 --- a/cranelift/filetests/isa/intel/abi-bool.cton +++ b/cranelift/filetests/isa/x86/abi-bool.cton @@ -1,6 +1,6 @@ test compile set is_64bit=1 -isa intel haswell +isa x86 haswell function %foo(i64, i64, i64, i32) -> b1 system_v { ebb3(v0: i64, v1: i64, v2: i64, v3: i32): diff --git a/cranelift/filetests/isa/intel/abi32.cton b/cranelift/filetests/isa/x86/abi32.cton similarity index 97% rename from cranelift/filetests/isa/intel/abi32.cton rename to cranelift/filetests/isa/x86/abi32.cton index b949af16f7..04b8d2ec6f 100644 --- a/cranelift/filetests/isa/intel/abi32.cton +++ b/cranelift/filetests/isa/x86/abi32.cton @@ -1,6 +1,6 @@ ; Test the legalization of function signatures. test legalizer -isa intel +isa x86 ; regex: V=v\d+ diff --git a/cranelift/filetests/isa/intel/abi64.cton b/cranelift/filetests/isa/x86/abi64.cton similarity index 99% rename from cranelift/filetests/isa/intel/abi64.cton rename to cranelift/filetests/isa/x86/abi64.cton index 8aa6a49b0f..2cb63c0dab 100644 --- a/cranelift/filetests/isa/intel/abi64.cton +++ b/cranelift/filetests/isa/x86/abi64.cton @@ -1,7 +1,7 @@ ; Test the legalization of function signatures. test legalizer set is_64bit -isa intel +isa x86 ; regex: V=v\d+ diff --git a/cranelift/filetests/isa/intel/allones_funcaddrs32.cton b/cranelift/filetests/isa/x86/allones_funcaddrs32.cton similarity index 81% rename from cranelift/filetests/isa/intel/allones_funcaddrs32.cton rename to cranelift/filetests/isa/x86/allones_funcaddrs32.cton index 90af70fe5b..d549c8d5e3 100644 --- a/cranelift/filetests/isa/intel/allones_funcaddrs32.cton +++ b/cranelift/filetests/isa/x86/allones_funcaddrs32.cton @@ -2,11 +2,11 @@ test binemit set is_compressed set allones_funcaddrs -isa intel haswell +isa x86 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/allones_funcaddrs32.cton | llvm-mc -show-encoding -triple=i386 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/allones_funcaddrs32.cton | llvm-mc -show-encoding -triple=i386 ; ; Tests from binary32.cton affected by allones_funcaddrs. diff --git a/cranelift/filetests/isa/intel/allones_funcaddrs64.cton b/cranelift/filetests/isa/x86/allones_funcaddrs64.cton similarity index 84% rename from cranelift/filetests/isa/intel/allones_funcaddrs64.cton rename to cranelift/filetests/isa/x86/allones_funcaddrs64.cton index 7be06c3963..478c875e3d 100644 --- a/cranelift/filetests/isa/intel/allones_funcaddrs64.cton +++ b/cranelift/filetests/isa/x86/allones_funcaddrs64.cton @@ -3,11 +3,11 @@ test binemit set is_64bit set is_compressed set allones_funcaddrs -isa intel haswell +isa x86 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/allones_funcaddrs64.cton | llvm-mc -show-encoding -triple=x86_64 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/allones_funcaddrs64.cton | llvm-mc -show-encoding -triple=x86_64 ; ; Tests from binary64.cton affected by allones_funcaddrs. diff --git a/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount.cton b/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount.cton similarity index 98% rename from cranelift/filetests/isa/intel/baseline_clz_ctz_popcount.cton rename to cranelift/filetests/isa/x86/baseline_clz_ctz_popcount.cton index b87815aa12..c79de42317 100644 --- a/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount.cton +++ b/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount.cton @@ -1,7 +1,7 @@ test compile set is_64bit -isa intel baseline +isa x86 baseline ; clz/ctz on 64 bit operands diff --git a/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton b/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton similarity index 94% rename from cranelift/filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton rename to cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton index 2e552b7145..f69efd573a 100644 --- a/cranelift/filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton +++ b/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton @@ -2,11 +2,11 @@ test binemit set is_64bit set is_compressed -isa intel baseline +isa x86 baseline ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/baseline_clz_ctz_popcount_encoding.cton | llvm-mc -show-encoding -triple=x86_64 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton | llvm-mc -show-encoding -triple=x86_64 ; function %Foo() { diff --git a/cranelift/filetests/isa/intel/binary32-float.cton b/cranelift/filetests/isa/x86/binary32-float.cton similarity index 99% rename from cranelift/filetests/isa/intel/binary32-float.cton rename to cranelift/filetests/isa/x86/binary32-float.cton index 4477c4d8d3..a94fff71c1 100644 --- a/cranelift/filetests/isa/intel/binary32-float.cton +++ b/cranelift/filetests/isa/x86/binary32-float.cton @@ -1,10 +1,10 @@ ; Binary emission of 32-bit floating point code. test binemit -isa intel haswell +isa x86 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/binary32-float.cton | llvm-mc -show-encoding -triple=i386 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary32-float.cton | llvm-mc -show-encoding -triple=i386 ; function %F32() { diff --git a/cranelift/filetests/isa/intel/binary32.cton b/cranelift/filetests/isa/x86/binary32.cton similarity index 99% rename from cranelift/filetests/isa/intel/binary32.cton rename to cranelift/filetests/isa/x86/binary32.cton index c6bff52a64..342af80140 100644 --- a/cranelift/filetests/isa/intel/binary32.cton +++ b/cranelift/filetests/isa/x86/binary32.cton @@ -1,11 +1,11 @@ ; binary emission of x86-32 code. test binemit set is_compressed -isa intel haswell +isa x86 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/binary32.cton | llvm-mc -show-encoding -triple=i386 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary32.cton | llvm-mc -show-encoding -triple=i386 ; function %I32() { diff --git a/cranelift/filetests/isa/intel/binary64-float.cton b/cranelift/filetests/isa/x86/binary64-float.cton similarity index 99% rename from cranelift/filetests/isa/intel/binary64-float.cton rename to cranelift/filetests/isa/x86/binary64-float.cton index 7053a1809e..df910941cd 100644 --- a/cranelift/filetests/isa/intel/binary64-float.cton +++ b/cranelift/filetests/isa/x86/binary64-float.cton @@ -2,11 +2,11 @@ test binemit set is_64bit set is_compressed -isa intel haswell +isa x86 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/binary64-float.cton | llvm-mc -show-encoding -triple=x86_64 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary64-float.cton | llvm-mc -show-encoding -triple=x86_64 ; function %F32() { diff --git a/cranelift/filetests/isa/intel/binary64-pic.cton b/cranelift/filetests/isa/x86/binary64-pic.cton similarity index 92% rename from cranelift/filetests/isa/intel/binary64-pic.cton rename to cranelift/filetests/isa/x86/binary64-pic.cton index 3129bd9563..e1cd79ac71 100644 --- a/cranelift/filetests/isa/intel/binary64-pic.cton +++ b/cranelift/filetests/isa/x86/binary64-pic.cton @@ -3,11 +3,11 @@ test binemit set is_64bit set is_compressed set is_pic -isa intel haswell +isa x86 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/binary64-pic.cton | llvm-mc -show-encoding -triple=x86_64 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary64-pic.cton | llvm-mc -show-encoding -triple=x86_64 ; ; Tests for i64 instructions. diff --git a/cranelift/filetests/isa/intel/binary64.cton b/cranelift/filetests/isa/x86/binary64.cton similarity index 99% rename from cranelift/filetests/isa/intel/binary64.cton rename to cranelift/filetests/isa/x86/binary64.cton index d2fab8e19d..37f026f2ab 100644 --- a/cranelift/filetests/isa/intel/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.cton @@ -2,11 +2,11 @@ test binemit set is_64bit set is_compressed -isa intel haswell +isa x86 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/intel/binary64.cton | llvm-mc -show-encoding -triple=x86_64 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary64.cton | llvm-mc -show-encoding -triple=x86_64 ; ; Tests for i64 instructions. diff --git a/cranelift/filetests/isa/intel/legalize-custom.cton b/cranelift/filetests/isa/x86/legalize-custom.cton similarity index 98% rename from cranelift/filetests/isa/intel/legalize-custom.cton rename to cranelift/filetests/isa/x86/legalize-custom.cton index 5cb5ac0462..8f719056cd 100644 --- a/cranelift/filetests/isa/intel/legalize-custom.cton +++ b/cranelift/filetests/isa/x86/legalize-custom.cton @@ -1,8 +1,8 @@ ; Test the custom legalizations. test legalizer -isa intel +isa x86 set is_64bit -isa intel +isa x86 ; regex: V=v\d+ ; regex: EBB=ebb\d+ diff --git a/cranelift/filetests/isa/intel/legalize-div-traps.cton b/cranelift/filetests/isa/x86/legalize-div-traps.cton similarity index 99% rename from cranelift/filetests/isa/intel/legalize-div-traps.cton rename to cranelift/filetests/isa/x86/legalize-div-traps.cton index 3869e66325..4d7beb59b8 100644 --- a/cranelift/filetests/isa/intel/legalize-div-traps.cton +++ b/cranelift/filetests/isa/x86/legalize-div-traps.cton @@ -3,7 +3,7 @@ test legalizer set is_64bit ; See also legalize-div.cton. set avoid_div_traps=1 -isa intel +isa x86 ; regex: V=v\d+ ; regex: EBB=ebb\d+ diff --git a/cranelift/filetests/isa/intel/legalize-div.cton b/cranelift/filetests/isa/x86/legalize-div.cton similarity index 99% rename from cranelift/filetests/isa/intel/legalize-div.cton rename to cranelift/filetests/isa/x86/legalize-div.cton index d6179b2611..3777be659e 100644 --- a/cranelift/filetests/isa/intel/legalize-div.cton +++ b/cranelift/filetests/isa/x86/legalize-div.cton @@ -3,7 +3,7 @@ test legalizer set is_64bit ; See also legalize-div-traps.cton. set avoid_div_traps=0 -isa intel +isa x86 ; regex: V=v\d+ ; regex: EBB=ebb\d+ diff --git a/cranelift/filetests/isa/intel/legalize-libcall.cton b/cranelift/filetests/isa/x86/legalize-libcall.cton similarity index 97% rename from cranelift/filetests/isa/intel/legalize-libcall.cton rename to cranelift/filetests/isa/x86/legalize-libcall.cton index 239b3e5a57..bbe332170e 100644 --- a/cranelift/filetests/isa/intel/legalize-libcall.cton +++ b/cranelift/filetests/isa/x86/legalize-libcall.cton @@ -3,7 +3,7 @@ test legalizer ; Pre-SSE 4.1, we need to use runtime library calls for floating point rounding operations. set is_64bit set is_pic -isa intel +isa x86 function %floor(f32) -> f32 { ebb0(v0: f32): diff --git a/cranelift/filetests/isa/intel/legalize-memory.cton b/cranelift/filetests/isa/x86/legalize-memory.cton similarity index 99% rename from cranelift/filetests/isa/intel/legalize-memory.cton rename to cranelift/filetests/isa/x86/legalize-memory.cton index 1ba979deb1..9cb8661873 100644 --- a/cranelift/filetests/isa/intel/legalize-memory.cton +++ b/cranelift/filetests/isa/x86/legalize-memory.cton @@ -1,7 +1,7 @@ ; Test the legalization of memory objects. test legalizer set is_64bit -isa intel +isa x86 ; regex: V=v\d+ ; regex: EBB=ebb\d+ diff --git a/cranelift/filetests/isa/intel/legalize-mulhi.cton b/cranelift/filetests/isa/x86/legalize-mulhi.cton similarity index 97% rename from cranelift/filetests/isa/intel/legalize-mulhi.cton rename to cranelift/filetests/isa/x86/legalize-mulhi.cton index 673a19db3b..a588f25d4e 100644 --- a/cranelift/filetests/isa/intel/legalize-mulhi.cton +++ b/cranelift/filetests/isa/x86/legalize-mulhi.cton @@ -1,7 +1,7 @@ test compile set is_64bit -isa intel baseline +isa x86 baseline ; umulhi/smulhi on 64 bit operands diff --git a/cranelift/filetests/isa/intel/prologue-epilogue.cton b/cranelift/filetests/isa/x86/prologue-epilogue.cton similarity index 99% rename from cranelift/filetests/isa/intel/prologue-epilogue.cton rename to cranelift/filetests/isa/x86/prologue-epilogue.cton index ef42836a5a..eee6917b1b 100644 --- a/cranelift/filetests/isa/intel/prologue-epilogue.cton +++ b/cranelift/filetests/isa/x86/prologue-epilogue.cton @@ -2,7 +2,7 @@ test compile set is_64bit set is_compressed set is_pic -isa intel haswell +isa x86 haswell ; An empty function. diff --git a/cranelift/filetests/postopt/basic.cton b/cranelift/filetests/postopt/basic.cton index 218d4dfee4..48b6b66007 100644 --- a/cranelift/filetests/postopt/basic.cton +++ b/cranelift/filetests/postopt/basic.cton @@ -1,5 +1,5 @@ test postopt -isa intel +isa x86 ; Test that compare+branch sequences are folded effectively on x86. diff --git a/cranelift/filetests/preopt/div_by_const_indirect.cton b/cranelift/filetests/preopt/div_by_const_indirect.cton index ccc83cd49b..5d3f9475fc 100644 --- a/cranelift/filetests/preopt/div_by_const_indirect.cton +++ b/cranelift/filetests/preopt/div_by_const_indirect.cton @@ -1,6 +1,6 @@ test preopt -isa intel baseline +isa x86 baseline ; Cases where the denominator is created by an iconst diff --git a/cranelift/filetests/preopt/div_by_const_non_power_of_2.cton b/cranelift/filetests/preopt/div_by_const_non_power_of_2.cton index 18811fcd82..fa3f7fa96a 100644 --- a/cranelift/filetests/preopt/div_by_const_non_power_of_2.cton +++ b/cranelift/filetests/preopt/div_by_const_non_power_of_2.cton @@ -1,6 +1,6 @@ test preopt -isa intel baseline +isa x86 baseline ; -------- U32 -------- diff --git a/cranelift/filetests/preopt/div_by_const_power_of_2.cton b/cranelift/filetests/preopt/div_by_const_power_of_2.cton index dc51c5395d..2318aa1f81 100644 --- a/cranelift/filetests/preopt/div_by_const_power_of_2.cton +++ b/cranelift/filetests/preopt/div_by_const_power_of_2.cton @@ -1,6 +1,6 @@ test preopt -isa intel baseline +isa x86 baseline ; -------- U32 -------- diff --git a/cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton b/cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton index c142a16359..3d4b986213 100644 --- a/cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton +++ b/cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton @@ -1,6 +1,6 @@ test preopt -isa intel baseline +isa x86 baseline ; -------- U32 -------- diff --git a/cranelift/filetests/preopt/rem_by_const_power_of_2.cton b/cranelift/filetests/preopt/rem_by_const_power_of_2.cton index 931623d2e7..7b537fdbb4 100644 --- a/cranelift/filetests/preopt/rem_by_const_power_of_2.cton +++ b/cranelift/filetests/preopt/rem_by_const_power_of_2.cton @@ -1,6 +1,6 @@ test preopt -isa intel baseline +isa x86 baseline ; -------- U32 -------- diff --git a/cranelift/filetests/preopt/simplify.cton b/cranelift/filetests/preopt/simplify.cton index 020e2f0108..32f19f4f03 100644 --- a/cranelift/filetests/preopt/simplify.cton +++ b/cranelift/filetests/preopt/simplify.cton @@ -1,5 +1,5 @@ test preopt -isa intel +isa x86 function %iadd_imm(i32) -> i32 { ebb0(v0: i32): diff --git a/cranelift/filetests/regalloc/aliases.cton b/cranelift/filetests/regalloc/aliases.cton index a16fe7231b..096ba99c2a 100644 --- a/cranelift/filetests/regalloc/aliases.cton +++ b/cranelift/filetests/regalloc/aliases.cton @@ -1,6 +1,6 @@ test regalloc set is_64bit -isa intel haswell +isa x86 haswell function %value_aliases(i32, f32, i64 vmctx) spiderwasm { gv0 = vmctx diff --git a/cranelift/filetests/regalloc/coalescing-207.cton b/cranelift/filetests/regalloc/coalescing-207.cton index c4d348e10b..9066bd7c4f 100644 --- a/cranelift/filetests/regalloc/coalescing-207.cton +++ b/cranelift/filetests/regalloc/coalescing-207.cton @@ -1,6 +1,6 @@ test regalloc set is_64bit -isa intel haswell +isa x86 haswell ; Reported as https://github.com/Cretonne/cretonne/issues/207 ; diff --git a/cranelift/filetests/regalloc/coalescing-216.cton b/cranelift/filetests/regalloc/coalescing-216.cton index a6cf7b3b42..1138e2578a 100644 --- a/cranelift/filetests/regalloc/coalescing-216.cton +++ b/cranelift/filetests/regalloc/coalescing-216.cton @@ -1,6 +1,6 @@ test regalloc set is_64bit -isa intel haswell +isa x86 haswell ; Reported as https://github.com/Cretonne/cretonne/issues/216 from the Binaryen fuzzer. ; diff --git a/cranelift/filetests/regalloc/coloring-227.cton b/cranelift/filetests/regalloc/coloring-227.cton index accfb3528b..8144bba62e 100644 --- a/cranelift/filetests/regalloc/coloring-227.cton +++ b/cranelift/filetests/regalloc/coloring-227.cton @@ -1,6 +1,6 @@ test regalloc set is_64bit -isa intel haswell +isa x86 haswell function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) system_v { gv0 = vmctx diff --git a/cranelift/filetests/regalloc/constraints.cton b/cranelift/filetests/regalloc/constraints.cton index bda37df149..e4db3bf9da 100644 --- a/cranelift/filetests/regalloc/constraints.cton +++ b/cranelift/filetests/regalloc/constraints.cton @@ -1,5 +1,5 @@ test regalloc -isa intel +isa x86 ; regex: V=v\d+ ; regex: REG=%r([abcd]x|[sd]i) diff --git a/cranelift/filetests/regalloc/ghost-param.cton b/cranelift/filetests/regalloc/ghost-param.cton index c4ce229360..2064231318 100644 --- a/cranelift/filetests/regalloc/ghost-param.cton +++ b/cranelift/filetests/regalloc/ghost-param.cton @@ -1,6 +1,6 @@ test regalloc set is_64bit -isa intel haswell +isa x86 haswell ; This test case would create an EBB parameter that was a ghost value. ; The coalescer would insert a copy of the ghost value, leading to verifier errors. diff --git a/cranelift/filetests/regalloc/global-constraints.cton b/cranelift/filetests/regalloc/global-constraints.cton index b2a3fc0a7e..4975adeac5 100644 --- a/cranelift/filetests/regalloc/global-constraints.cton +++ b/cranelift/filetests/regalloc/global-constraints.cton @@ -1,11 +1,11 @@ test regalloc -isa intel +isa x86 ; This test covers the troubles when values with global live ranges are defined ; by instructions with constrained register classes. ; ; The icmp_imm instrutions write their b1 result to the ABCD register class on -; 32-bit Intel. So if we define 5 live values, they can't all fit. +; 32-bit x86. So if we define 5 live values, they can't all fit. function %global_constraints(i32) { ebb0(v0: i32): v1 = icmp_imm eq v0, 1 diff --git a/cranelift/filetests/regalloc/global-fixed.cton b/cranelift/filetests/regalloc/global-fixed.cton index 14b9d8fecc..61d4928806 100644 --- a/cranelift/filetests/regalloc/global-fixed.cton +++ b/cranelift/filetests/regalloc/global-fixed.cton @@ -1,6 +1,6 @@ test regalloc set is_64bit=1 -isa intel haswell +isa x86 haswell function %foo() system_v { ebb4: diff --git a/cranelift/filetests/regalloc/iterate.cton b/cranelift/filetests/regalloc/iterate.cton index 2c585e941d..acb97a31af 100644 --- a/cranelift/filetests/regalloc/iterate.cton +++ b/cranelift/filetests/regalloc/iterate.cton @@ -1,6 +1,6 @@ test regalloc set is_64bit -isa intel haswell +isa x86 haswell function u0:9(i64 [%rdi], f32 [%xmm0], f64 [%xmm1], i32 [%rsi], i32 [%rdx], i64 vmctx [%r14]) -> i64 [%rax] spiderwasm { ebb0(v0: i64, v1: f32, v2: f64, v3: i32, v4: i32, v5: i64): diff --git a/cranelift/filetests/regalloc/multi-constraints.cton b/cranelift/filetests/regalloc/multi-constraints.cton index afbbf6fc2b..590d93eb38 100644 --- a/cranelift/filetests/regalloc/multi-constraints.cton +++ b/cranelift/filetests/regalloc/multi-constraints.cton @@ -1,10 +1,10 @@ test regalloc set is_64bit -isa intel haswell +isa x86 haswell ; Test combinations of constraints. ; -; The Intel ushr instruction requires its second operand to be passed in %rcx and its output is +; The x86 ushr instruction requires its second operand to be passed in %rcx and its output is ; tied to the first input operand. ; ; If we pass the same value to both operands, both constraints must be satisfied. diff --git a/cranelift/filetests/regalloc/output-interference.cton b/cranelift/filetests/regalloc/output-interference.cton index 4a5b90c856..ac2bb537c7 100644 --- a/cranelift/filetests/regalloc/output-interference.cton +++ b/cranelift/filetests/regalloc/output-interference.cton @@ -1,11 +1,11 @@ test regalloc set is_64bit=1 -isa intel haswell +isa x86 haswell function %test(i64) -> i64 system_v { ebb0(v0: i64): v2 = iconst.i64 12 - ; This division clobbers two of its fixed input registers on Intel. + ; This division clobbers two of its fixed input registers on x86. ; These are FixedTied constraints that the spiller needs to resolve. v5 = udiv v0, v2 v6 = iconst.i64 13 diff --git a/cranelift/filetests/regalloc/reload-208.cton b/cranelift/filetests/regalloc/reload-208.cton index 70dc1c4523..8ccd3e6ef3 100644 --- a/cranelift/filetests/regalloc/reload-208.cton +++ b/cranelift/filetests/regalloc/reload-208.cton @@ -1,6 +1,6 @@ test regalloc set is_64bit -isa intel haswell +isa x86 haswell ; regex: V=v\d+ diff --git a/cranelift/filetests/regalloc/schedule-moves.cton b/cranelift/filetests/regalloc/schedule-moves.cton index 6ba78db893..946bcc9cf3 100644 --- a/cranelift/filetests/regalloc/schedule-moves.cton +++ b/cranelift/filetests/regalloc/schedule-moves.cton @@ -1,5 +1,5 @@ test regalloc -isa intel haswell +isa x86 haswell function %pr165() system_v { ebb0: diff --git a/cranelift/filetests/regalloc/spill-noregs.cton b/cranelift/filetests/regalloc/spill-noregs.cton index 733606b828..65f1da84c8 100644 --- a/cranelift/filetests/regalloc/spill-noregs.cton +++ b/cranelift/filetests/regalloc/spill-noregs.cton @@ -1,6 +1,6 @@ test regalloc set is_64bit -isa intel +isa x86 ; Test case found by the Binaryen fuzzer. ; diff --git a/cranelift/filetests/regalloc/unreachable_code.cton b/cranelift/filetests/regalloc/unreachable_code.cton index ac667e7721..4b4896f4a8 100644 --- a/cranelift/filetests/regalloc/unreachable_code.cton +++ b/cranelift/filetests/regalloc/unreachable_code.cton @@ -2,7 +2,7 @@ test compile set is_64bit -isa intel haswell +isa x86 haswell ; This function contains unreachable blocks which trip up the register ; allocator if they don't get cleared out. diff --git a/cranelift/filetests/regalloc/intel-regres.cton b/cranelift/filetests/regalloc/x86-regres.cton similarity index 99% rename from cranelift/filetests/regalloc/intel-regres.cton rename to cranelift/filetests/regalloc/x86-regres.cton index 0f76e8f2d0..e396fb1022 100644 --- a/cranelift/filetests/regalloc/intel-regres.cton +++ b/cranelift/filetests/regalloc/x86-regres.cton @@ -1,6 +1,6 @@ test regalloc -isa intel +isa x86 ; regex: V=v\d+ diff --git a/cranelift/filetests/verifier/flags.cton b/cranelift/filetests/verifier/flags.cton index b4a01621c4..55d9bdf4e1 100644 --- a/cranelift/filetests/verifier/flags.cton +++ b/cranelift/filetests/verifier/flags.cton @@ -1,5 +1,5 @@ test verifier -isa intel +isa x86 ; Simple, correct use of CPU flags. function %simple(i32) -> i32 { diff --git a/cranelift/filetests/wasm/control.cton b/cranelift/filetests/wasm/control.cton index 1d12aeff3d..30d4240490 100644 --- a/cranelift/filetests/wasm/control.cton +++ b/cranelift/filetests/wasm/control.cton @@ -2,10 +2,10 @@ test compile set is_64bit=0 -isa intel haswell +isa x86 haswell set is_64bit=1 -isa intel haswell +isa x86 haswell function %br_if(i32) -> i32 { ebb0(v0: i32): diff --git a/cranelift/filetests/wasm/conversions.cton b/cranelift/filetests/wasm/conversions.cton index 4ce70ee583..5ca8072067 100644 --- a/cranelift/filetests/wasm/conversions.cton +++ b/cranelift/filetests/wasm/conversions.cton @@ -2,7 +2,7 @@ test compile set is_64bit=1 -isa intel haswell +isa x86 haswell function %i32_wrap_i64(i64) -> i32 { ebb0(v0: i64): diff --git a/cranelift/filetests/wasm/f32-arith.cton b/cranelift/filetests/wasm/f32-arith.cton index 331f9c7fad..d21316d293 100644 --- a/cranelift/filetests/wasm/f32-arith.cton +++ b/cranelift/filetests/wasm/f32-arith.cton @@ -2,10 +2,10 @@ test compile set is_64bit=0 -isa intel haswell +isa x86 haswell set is_64bit=1 -isa intel haswell +isa x86 haswell ; Constants. diff --git a/cranelift/filetests/wasm/f32-compares.cton b/cranelift/filetests/wasm/f32-compares.cton index 560b86ebcb..0cdee58ef3 100644 --- a/cranelift/filetests/wasm/f32-compares.cton +++ b/cranelift/filetests/wasm/f32-compares.cton @@ -2,10 +2,10 @@ test compile set is_64bit=0 -isa intel haswell +isa x86 haswell set is_64bit=1 -isa intel haswell +isa x86 haswell function %f32_eq(f32, f32) -> i32 { ebb0(v0: f32, v1: f32): diff --git a/cranelift/filetests/wasm/f32-memory64.cton b/cranelift/filetests/wasm/f32-memory64.cton index 7125e66d3b..a6e0d54af7 100644 --- a/cranelift/filetests/wasm/f32-memory64.cton +++ b/cranelift/filetests/wasm/f32-memory64.cton @@ -4,7 +4,7 @@ test compile ; We only test on 64-bit since the heap_addr instructions and vmctx parameters ; explicitly mention the pointer width. set is_64bit=1 -isa intel haswell +isa x86 haswell function %f32_load(i32, i64 vmctx) -> f32 { gv0 = vmctx diff --git a/cranelift/filetests/wasm/f64-arith.cton b/cranelift/filetests/wasm/f64-arith.cton index 4f39fb6a5f..7b870ab2cb 100644 --- a/cranelift/filetests/wasm/f64-arith.cton +++ b/cranelift/filetests/wasm/f64-arith.cton @@ -2,7 +2,7 @@ test compile set is_64bit=1 -isa intel haswell +isa x86 haswell ; Constants. diff --git a/cranelift/filetests/wasm/f64-compares.cton b/cranelift/filetests/wasm/f64-compares.cton index 78a260ef27..ff1b9526d0 100644 --- a/cranelift/filetests/wasm/f64-compares.cton +++ b/cranelift/filetests/wasm/f64-compares.cton @@ -2,10 +2,10 @@ test compile set is_64bit=0 -isa intel haswell +isa x86 haswell set is_64bit=1 -isa intel haswell +isa x86 haswell function %f64_eq(f64, f64) -> i32 { ebb0(v0: f64, v1: f64): diff --git a/cranelift/filetests/wasm/f64-memory64.cton b/cranelift/filetests/wasm/f64-memory64.cton index 1f61749e51..0a15e69a37 100644 --- a/cranelift/filetests/wasm/f64-memory64.cton +++ b/cranelift/filetests/wasm/f64-memory64.cton @@ -4,7 +4,7 @@ test compile ; We only test on 64-bit since the heap_addr instructions and vmctx parameters ; explicitly mention the pointer width. set is_64bit=1 -isa intel haswell +isa x86 haswell function %f64_load(i32, i64 vmctx) -> f64 { gv0 = vmctx diff --git a/cranelift/filetests/wasm/i32-arith.cton b/cranelift/filetests/wasm/i32-arith.cton index fe9cf19883..7fb388054f 100644 --- a/cranelift/filetests/wasm/i32-arith.cton +++ b/cranelift/filetests/wasm/i32-arith.cton @@ -2,10 +2,10 @@ test compile set is_64bit=0 -isa intel haswell +isa x86 haswell set is_64bit=1 -isa intel haswell +isa x86 haswell ; Constants. diff --git a/cranelift/filetests/wasm/i32-compares.cton b/cranelift/filetests/wasm/i32-compares.cton index 228258d279..a36df1b28a 100644 --- a/cranelift/filetests/wasm/i32-compares.cton +++ b/cranelift/filetests/wasm/i32-compares.cton @@ -2,10 +2,10 @@ test compile set is_64bit=0 -isa intel haswell +isa x86 haswell set is_64bit=1 -isa intel haswell +isa x86 haswell function %i32_eqz(i32) -> i32 { ebb0(v0: i32): diff --git a/cranelift/filetests/wasm/i32-memory64.cton b/cranelift/filetests/wasm/i32-memory64.cton index 0fbffa4fb8..319609e9d3 100644 --- a/cranelift/filetests/wasm/i32-memory64.cton +++ b/cranelift/filetests/wasm/i32-memory64.cton @@ -4,7 +4,7 @@ test compile ; We only test on 64-bit since the heap_addr instructions and vmctx parameters ; explicitly mention the pointer width. set is_64bit=1 -isa intel haswell +isa x86 haswell function %i32_load(i32, i64 vmctx) -> i32 { gv0 = vmctx diff --git a/cranelift/filetests/wasm/i64-arith.cton b/cranelift/filetests/wasm/i64-arith.cton index 4e8cdc06df..6cf6f2e4a2 100644 --- a/cranelift/filetests/wasm/i64-arith.cton +++ b/cranelift/filetests/wasm/i64-arith.cton @@ -2,7 +2,7 @@ test compile set is_64bit=1 -isa intel haswell +isa x86 haswell ; Constants. diff --git a/cranelift/filetests/wasm/i64-compares.cton b/cranelift/filetests/wasm/i64-compares.cton index 3406463f0d..4679268018 100644 --- a/cranelift/filetests/wasm/i64-compares.cton +++ b/cranelift/filetests/wasm/i64-compares.cton @@ -2,7 +2,7 @@ test compile set is_64bit=1 -isa intel haswell +isa x86 haswell function %i64_eqz(i64) -> i32 { ebb0(v0: i64): diff --git a/cranelift/filetests/wasm/i64-memory64.cton b/cranelift/filetests/wasm/i64-memory64.cton index bc44a2bbea..7c9836dbf3 100644 --- a/cranelift/filetests/wasm/i64-memory64.cton +++ b/cranelift/filetests/wasm/i64-memory64.cton @@ -4,7 +4,7 @@ test compile ; We only test on 64-bit since the heap_addr instructions and vmctx parameters ; explicitly mention the pointer width. set is_64bit=1 -isa intel haswell +isa x86 haswell function %i64_load(i32, i64 vmctx) -> i64 { gv0 = vmctx diff --git a/cranelift/filetests/wasm/select.cton b/cranelift/filetests/wasm/select.cton index fc08c92a8a..7f1c26e724 100644 --- a/cranelift/filetests/wasm/select.cton +++ b/cranelift/filetests/wasm/select.cton @@ -2,10 +2,10 @@ test compile set is_64bit=0 -isa intel haswell +isa x86 haswell set is_64bit=1 -isa intel haswell +isa x86 haswell function %select_i32(i32, i32, i32) -> i32 { ebb0(v0: i32, v1: i32, v2: i32): diff --git a/lib/cretonne/build.rs b/lib/cretonne/build.rs index e13e9349a5..8031a71c5a 100644 --- a/lib/cretonne/build.rs +++ b/lib/cretonne/build.rs @@ -77,7 +77,7 @@ fn main() { #[derive(Copy, Clone)] enum Isa { Riscv, - Intel, + X86, Arm32, Arm64, } @@ -103,14 +103,14 @@ impl Isa { /// Returns all supported isa targets. fn all() -> [Isa; 4] { - [Isa::Riscv, Isa::Intel, Isa::Arm32, Isa::Arm64] + [Isa::Riscv, Isa::X86, Isa::Arm32, Isa::Arm64] } /// Returns name of the isa target. fn name(&self) -> &'static str { match *self { Isa::Riscv => "riscv", - Isa::Intel => "intel", + Isa::X86 => "x86", Isa::Arm32 => "arm32", Isa::Arm64 => "arm64", } @@ -120,7 +120,7 @@ impl Isa { fn is_arch_applicable(&self, arch: &str) -> bool { match *self { Isa::Riscv => arch == "riscv", - Isa::Intel => ["x86_64", "i386", "i586", "i686"].contains(&arch), + Isa::X86 => ["x86_64", "i386", "i586", "i686"].contains(&arch), Isa::Arm32 => arch.starts_with("arm") || arch.starts_with("thumb"), Isa::Arm64 => arch == "aarch64", } diff --git a/lib/cretonne/meta/base/settings.py b/lib/cretonne/meta/base/settings.py index 3bd90f107b..a59904a572 100644 --- a/lib/cretonne/meta/base/settings.py +++ b/lib/cretonne/meta/base/settings.py @@ -85,7 +85,7 @@ spiderwasm_prologue_words = NumSetting( This setting configures the number of pointer-sized words pushed on the stack when the Cretonne-generated code is entered. This includes the - pushed return address on Intel ISAs. + pushed return address on x86. """) # diff --git a/lib/cretonne/meta/isa/__init__.py b/lib/cretonne/meta/isa/__init__.py index bad91c5c90..1add6e033f 100644 --- a/lib/cretonne/meta/isa/__init__.py +++ b/lib/cretonne/meta/isa/__init__.py @@ -7,7 +7,7 @@ architecture supported by Cretonne. """ from __future__ import absolute_import from cdsl.isa import TargetISA # noqa -from . import riscv, intel, arm32, arm64 +from . import riscv, x86, arm32, arm64 try: from typing import List # noqa @@ -21,4 +21,4 @@ def all_isas(): Get a list of all the supported target ISAs. Each target ISA is represented as a :py:class:`cretonne.TargetISA` instance. """ - return [riscv.ISA, intel.ISA, arm32.ISA, arm64.ISA] + return [riscv.ISA, x86.ISA, arm32.ISA, arm64.ISA] diff --git a/lib/cretonne/meta/isa/intel/__init__.py b/lib/cretonne/meta/isa/intel/__init__.py deleted file mode 100644 index 0828d790aa..0000000000 --- a/lib/cretonne/meta/isa/intel/__init__.py +++ /dev/null @@ -1,23 +0,0 @@ -""" -Intel Target Architecture -------------------------- - -This target ISA generates code for Intel CPUs with two separate CPU modes: - -`I32` - IA-32 architecture, also known as 'x86'. Generates code for the Intel 386 - and later processors in 32-bit mode. -`I64` - Intel 64 architecture, also known as 'x86-64, 'x64', and 'amd64'. Intel and - AMD CPUs running in 64-bit mode. - -Floating point is supported only on CPUs with support for SSE2 or later. There -is no x87 floating point support. -""" - -from __future__ import absolute_import -from . import defs -from . import encodings, settings, registers # noqa - -# Re-export the primary target ISA definition. -ISA = defs.ISA.finish() diff --git a/lib/cretonne/meta/isa/x86/__init__.py b/lib/cretonne/meta/isa/x86/__init__.py new file mode 100644 index 0000000000..d87b95964a --- /dev/null +++ b/lib/cretonne/meta/isa/x86/__init__.py @@ -0,0 +1,21 @@ +""" +x86 Target Architecture +------------------------- + +This target ISA generates code for x86 CPUs with two separate CPU modes: + +`I32` + 32-bit x86 architecture, also known as 'IA-32', also sometimes referred + to as 'i386', however note that Cretonne depends on instructions not + in the original `i386`, such as SSE2, CMOVcc, and UD2. + +`I64` + x86-64 architecture, also known as 'AMD64`, `Intel 64`, and 'x64'. +""" + +from __future__ import absolute_import +from . import defs +from . import encodings, settings, registers # noqa + +# Re-export the primary target ISA definition. +ISA = defs.ISA.finish() diff --git a/lib/cretonne/meta/isa/intel/defs.py b/lib/cretonne/meta/isa/x86/defs.py similarity index 88% rename from lib/cretonne/meta/isa/intel/defs.py rename to lib/cretonne/meta/isa/x86/defs.py index b6a4d37206..32c54ab620 100644 --- a/lib/cretonne/meta/isa/intel/defs.py +++ b/lib/cretonne/meta/isa/x86/defs.py @@ -1,5 +1,5 @@ """ -Intel definitions. +x86 definitions. Commonly used definitions. """ @@ -9,7 +9,7 @@ import base.instructions from . import instructions as x86 from base.immediates import floatcc -ISA = TargetISA('intel', [base.instructions.GROUP, x86.GROUP]) +ISA = TargetISA('x86', [base.instructions.GROUP, x86.GROUP]) # CPU modes for 32-bit and 64-bit operation. X86_64 = CPUMode('I64', ISA) diff --git a/lib/cretonne/meta/isa/intel/encodings.py b/lib/cretonne/meta/isa/x86/encodings.py similarity index 98% rename from lib/cretonne/meta/isa/intel/encodings.py rename to lib/cretonne/meta/isa/x86/encodings.py index c7833d04bc..915e75436d 100644 --- a/lib/cretonne/meta/isa/intel/encodings.py +++ b/lib/cretonne/meta/isa/x86/encodings.py @@ -1,5 +1,5 @@ """ -Intel Encodings. +x86 Encodings. """ from __future__ import absolute_import from cdsl.predicates import IsUnsignedInt, Not, And @@ -9,7 +9,7 @@ from .defs import X86_64, X86_32 from . import recipes as r from . import settings as cfg from . import instructions as x86 -from .legalize import intel_expand +from .legalize import x86_expand from base.legalize import narrow, expand_flags from base.settings import allones_funcaddrs, is_pic from .settings import use_sse41 @@ -26,18 +26,18 @@ X86_32.legalize_monomorphic(expand_flags) X86_32.legalize_type( default=narrow, b1=expand_flags, - i32=intel_expand, - f32=intel_expand, - f64=intel_expand) + i32=x86_expand, + f32=x86_expand, + f64=x86_expand) X86_64.legalize_monomorphic(expand_flags) X86_64.legalize_type( default=narrow, b1=expand_flags, - i32=intel_expand, - i64=intel_expand, - f32=intel_expand, - f64=intel_expand) + i32=x86_expand, + i64=x86_expand, + f32=x86_expand, + f64=x86_expand) # diff --git a/lib/cretonne/meta/isa/intel/instructions.py b/lib/cretonne/meta/isa/x86/instructions.py similarity index 95% rename from lib/cretonne/meta/isa/intel/instructions.py rename to lib/cretonne/meta/isa/x86/instructions.py index 1fe75ff984..6adc2ad689 100644 --- a/lib/cretonne/meta/isa/intel/instructions.py +++ b/lib/cretonne/meta/isa/x86/instructions.py @@ -1,7 +1,7 @@ """ -Supplementary instruction definitions for Intel. +Supplementary instruction definitions for x86. -This module defines additional instructions that are useful only to the Intel +This module defines additional instructions that are useful only to the x86 target ISA. """ @@ -11,7 +11,7 @@ from cdsl.typevar import TypeVar from cdsl.instructions import Instruction, InstructionGroup -GROUP = InstructionGroup("x86", "Intel-specific instruction set") +GROUP = InstructionGroup("x86", "x86-specific instruction set") iWord = TypeVar('iWord', 'A scalar integer machine word', ints=(32, 64)) @@ -98,7 +98,7 @@ y = Operand('y', Float) fmin = Instruction( 'x86_fmin', r""" - Floating point minimum with Intel semantics. + Floating point minimum with x86 semantics. This is equivalent to the C ternary operator `x < y ? x : y` which differs from :inst:`fmin` when either operand is NaN or when comparing @@ -111,7 +111,7 @@ fmin = Instruction( fmax = Instruction( 'x86_fmax', r""" - Floating point maximum with Intel semantics. + Floating point maximum with x86 semantics. This is equivalent to the C ternary operator `x > y ? x : y` which differs from :inst:`fmax` when either operand is NaN or when comparing diff --git a/lib/cretonne/meta/isa/intel/legalize.py b/lib/cretonne/meta/isa/x86/legalize.py similarity index 85% rename from lib/cretonne/meta/isa/intel/legalize.py rename to lib/cretonne/meta/isa/x86/legalize.py index 5806bb9284..6da7d53340 100644 --- a/lib/cretonne/meta/isa/intel/legalize.py +++ b/lib/cretonne/meta/isa/x86/legalize.py @@ -1,5 +1,5 @@ """ -Custom legalization patterns for Intel. +Custom legalization patterns for x86. """ from __future__ import absolute_import from cdsl.ast import Var @@ -10,12 +10,12 @@ from base import instructions as insts from . import instructions as x86 from .defs import ISA -intel_expand = XFormGroup( - 'intel_expand', +x86_expand = XFormGroup( + 'x86_expand', """ Legalize instructions by expansion. - Use Intel-specific instructions if needed. + Use x86-specific instructions if needed. """, isa=ISA, chain=shared.expand_flags) @@ -32,23 +32,23 @@ a2 = Var('a2') # # The srem expansion requires custom code because srem INT_MIN, -1 is not # allowed to trap. The other ops need to check avoid_div_traps. -intel_expand.custom_legalize(insts.sdiv, 'expand_sdivrem') -intel_expand.custom_legalize(insts.srem, 'expand_sdivrem') -intel_expand.custom_legalize(insts.udiv, 'expand_udivrem') -intel_expand.custom_legalize(insts.urem, 'expand_udivrem') +x86_expand.custom_legalize(insts.sdiv, 'expand_sdivrem') +x86_expand.custom_legalize(insts.srem, 'expand_sdivrem') +x86_expand.custom_legalize(insts.udiv, 'expand_udivrem') +x86_expand.custom_legalize(insts.urem, 'expand_udivrem') # # Double length (widening) multiplication # resLo = Var('resLo') resHi = Var('resHi') -intel_expand.legalize( +x86_expand.legalize( resHi << insts.umulhi(x, y), Rtl( (resLo, resHi) << x86.umulx(x, y) )) -intel_expand.legalize( +x86_expand.legalize( resHi << insts.smulhi(x, y), Rtl( (resLo, resHi) << x86.smulx(x, y) @@ -61,14 +61,14 @@ intel_expand.legalize( # patterns. # Equality needs an explicit `ord` test which checks the parity bit. -intel_expand.legalize( +x86_expand.legalize( a << insts.fcmp(floatcc.eq, x, y), Rtl( a1 << insts.fcmp(floatcc.ord, x, y), a2 << insts.fcmp(floatcc.ueq, x, y), a << insts.band(a1, a2) )) -intel_expand.legalize( +x86_expand.legalize( a << insts.fcmp(floatcc.ne, x, y), Rtl( a1 << insts.fcmp(floatcc.uno, x, y), @@ -82,21 +82,21 @@ for cc, rev_cc in [ (floatcc.le, floatcc.ge), (floatcc.ugt, floatcc.ult), (floatcc.uge, floatcc.ule)]: - intel_expand.legalize( + x86_expand.legalize( a << insts.fcmp(cc, x, y), Rtl( a << insts.fcmp(rev_cc, y, x) )) # We need to modify the CFG for min/max legalization. -intel_expand.custom_legalize(insts.fmin, 'expand_minmax') -intel_expand.custom_legalize(insts.fmax, 'expand_minmax') +x86_expand.custom_legalize(insts.fmin, 'expand_minmax') +x86_expand.custom_legalize(insts.fmax, 'expand_minmax') # Conversions from unsigned need special handling. -intel_expand.custom_legalize(insts.fcvt_from_uint, 'expand_fcvt_from_uint') +x86_expand.custom_legalize(insts.fcvt_from_uint, 'expand_fcvt_from_uint') # Conversions from float to int can trap. -intel_expand.custom_legalize(insts.fcvt_to_sint, 'expand_fcvt_to_sint') -intel_expand.custom_legalize(insts.fcvt_to_uint, 'expand_fcvt_to_uint') +x86_expand.custom_legalize(insts.fcvt_to_sint, 'expand_fcvt_to_sint') +x86_expand.custom_legalize(insts.fcvt_to_uint, 'expand_fcvt_to_uint') # Count leading and trailing zeroes, for baseline x86_64 c_minus_one = Var('c_minus_one') @@ -108,7 +108,7 @@ index1 = Var('index1') r2flags = Var('r2flags') index2 = Var('index2') -intel_expand.legalize( +x86_expand.legalize( a << insts.clz.i64(x), Rtl( c_minus_one << insts.iconst(imm64(-1)), @@ -118,7 +118,7 @@ intel_expand.legalize( a << insts.isub(c_sixty_three, index2), )) -intel_expand.legalize( +x86_expand.legalize( a << insts.clz.i32(x), Rtl( c_minus_one << insts.iconst(imm64(-1)), @@ -128,7 +128,7 @@ intel_expand.legalize( a << insts.isub(c_thirty_one, index2), )) -intel_expand.legalize( +x86_expand.legalize( a << insts.ctz.i64(x), Rtl( c_sixty_four << insts.iconst(imm64(64)), @@ -136,7 +136,7 @@ intel_expand.legalize( a << insts.selectif(intcc.eq, r2flags, c_sixty_four, index1), )) -intel_expand.legalize( +x86_expand.legalize( a << insts.ctz.i32(x), Rtl( c_thirty_two << insts.iconst(imm64(32)), @@ -164,7 +164,7 @@ qv16 = Var('qv16') qc77 = Var('qc77') qc0F = Var('qc0F') qc01 = Var('qc01') -intel_expand.legalize( +x86_expand.legalize( qv16 << insts.popcnt.i64(qv1), Rtl( qv3 << insts.ushr_imm(qv1, imm64(1)), @@ -204,7 +204,7 @@ lv16 = Var('lv16') lc77 = Var('lc77') lc0F = Var('lc0F') lc01 = Var('lc01') -intel_expand.legalize( +x86_expand.legalize( lv16 << insts.popcnt.i32(lv1), Rtl( lv3 << insts.ushr_imm(lv1, imm64(1)), diff --git a/lib/cretonne/meta/isa/intel/recipes.py b/lib/cretonne/meta/isa/x86/recipes.py similarity index 98% rename from lib/cretonne/meta/isa/intel/recipes.py rename to lib/cretonne/meta/isa/x86/recipes.py index 5f36e20291..af1cab5792 100644 --- a/lib/cretonne/meta/isa/intel/recipes.py +++ b/lib/cretonne/meta/isa/x86/recipes.py @@ -1,5 +1,5 @@ """ -Intel Encoding recipes. +x86 Encoding recipes. """ from __future__ import absolute_import from cdsl.isa import EncRecipe @@ -31,7 +31,7 @@ except ImportError: # Opcode representation. # # Cretonne requires each recipe to have a single encoding size in bytes, and -# Intel opcodes are variable length, so we use separate recipes for different +# x86 opcodes are variable length, so we use separate recipes for different # styles of opcodes and prefixes. The opcode format is indicated by the recipe # name prefix: @@ -124,7 +124,7 @@ class TailRecipe: """ Generate encoding recipes on demand. - Intel encodings are somewhat orthogonal with the opcode representation on + x86 encodings are somewhat orthogonal with the opcode representation on one side and the ModR/M, SIB and immediate fields on the other side. A `TailRecipe` represents the part of an encoding that follow the opcode. @@ -144,7 +144,7 @@ class TailRecipe: The `emit` parameter contains Rust code to actually emit an encoding, like `EncRecipe` does it. Additionally, the text `PUT_OP` is substituted with - the proper `put_*` function from the `intel/binemit.rs` module. + the proper `put_*` function from the `x86/binemit.rs` module. """ def __init__( @@ -590,7 +590,7 @@ fnaddr4 = TailRecipe( 'fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(Reloc::IntelAbs4, + sink.reloc_external(Reloc::X86Abs4, &func.dfg.ext_funcs[func_ref].name, 0); sink.put4(0); @@ -601,7 +601,7 @@ fnaddr8 = TailRecipe( 'fnaddr8', FuncAddr, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(Reloc::IntelAbs8, + sink.reloc_external(Reloc::X86Abs8, &func.dfg.ext_funcs[func_ref].name, 0); sink.put8(0); @@ -612,7 +612,7 @@ allones_fnaddr4 = TailRecipe( 'allones_fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(Reloc::IntelAbs4, + sink.reloc_external(Reloc::X86Abs4, &func.dfg.ext_funcs[func_ref].name, 0); // Write the immediate as `!0` for the benefit of BaldrMonkey. @@ -624,7 +624,7 @@ allones_fnaddr8 = TailRecipe( 'allones_fnaddr8', FuncAddr, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(Reloc::IntelAbs8, + sink.reloc_external(Reloc::X86Abs8, &func.dfg.ext_funcs[func_ref].name, 0); // Write the immediate as `!0` for the benefit of BaldrMonkey. @@ -640,7 +640,7 @@ got_fnaddr8 = TailRecipe( modrm_riprel(out_reg0, sink); // The addend adjusts for the difference between the end of the // instruction and the beginning of the immediate field. - sink.reloc_external(Reloc::IntelGOTPCRel4, + sink.reloc_external(Reloc::X86GOTPCRel4, &func.dfg.ext_funcs[func_ref].name, -4); sink.put4(0); @@ -652,7 +652,7 @@ gvaddr4 = TailRecipe( 'gvaddr4', UnaryGlobalVar, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(Reloc::IntelAbs4, + sink.reloc_external(Reloc::X86Abs4, &func.global_vars[global_var].symbol_name(), 0); sink.put4(0); @@ -663,7 +663,7 @@ gvaddr8 = TailRecipe( 'gvaddr8', UnaryGlobalVar, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(Reloc::IntelAbs8, + sink.reloc_external(Reloc::X86Abs8, &func.global_vars[global_var].symbol_name(), 0); sink.put8(0); @@ -677,7 +677,7 @@ got_gvaddr8 = TailRecipe( modrm_rm(5, out_reg0, sink); // The addend adjusts for the difference between the end of the // instruction and the beginning of the immediate field. - sink.reloc_external(Reloc::IntelGOTPCRel4, + sink.reloc_external(Reloc::X86GOTPCRel4, &func.global_vars[global_var].symbol_name(), -4); sink.put4(0); @@ -1009,7 +1009,7 @@ call_id = TailRecipe( PUT_OP(bits, BASE_REX, sink); // The addend adjusts for the difference between the end of the // instruction and the beginning of the immediate field. - sink.reloc_external(Reloc::IntelPCRel4, + sink.reloc_external(Reloc::X86PCRel4, &func.dfg.ext_funcs[func_ref].name, -4); sink.put4(0); @@ -1019,7 +1019,7 @@ call_plt_id = TailRecipe( 'call_plt_id', Call, size=4, ins=(), outs=(), emit=''' PUT_OP(bits, BASE_REX, sink); - sink.reloc_external(Reloc::IntelPLTRel4, + sink.reloc_external(Reloc::X86PLTRel4, &func.dfg.ext_funcs[func_ref].name, -4); sink.put4(0); diff --git a/lib/cretonne/meta/isa/intel/registers.py b/lib/cretonne/meta/isa/x86/registers.py similarity index 94% rename from lib/cretonne/meta/isa/intel/registers.py rename to lib/cretonne/meta/isa/x86/registers.py index a4832640cd..f463b93a46 100644 --- a/lib/cretonne/meta/isa/intel/registers.py +++ b/lib/cretonne/meta/isa/x86/registers.py @@ -1,9 +1,9 @@ """ -Intel register banks. +x86 register banks. While the floating-point registers are straight-forward, the general purpose -register bank has a few quirks on Intel architectures. We have these encodings -of the 8-bit registers: +register bank has a few quirks on x86. We have these encodings of the 8-bit +registers: I32 I64 | 16b 32b 64b 000 AL AL | AX EAX RAX diff --git a/lib/cretonne/meta/isa/intel/settings.py b/lib/cretonne/meta/isa/x86/settings.py similarity index 92% rename from lib/cretonne/meta/isa/intel/settings.py rename to lib/cretonne/meta/isa/x86/settings.py index c62012e0c1..70da674fc0 100644 --- a/lib/cretonne/meta/isa/intel/settings.py +++ b/lib/cretonne/meta/isa/x86/settings.py @@ -1,5 +1,5 @@ """ -Intel settings. +x86 settings. """ from __future__ import absolute_import from cdsl.settings import SettingGroup, BoolSetting, Preset @@ -7,7 +7,7 @@ from cdsl.predicates import And import base.settings as shared from .defs import ISA -ISA.settings = SettingGroup('intel', parent=shared.group) +ISA.settings = SettingGroup('x86', parent=shared.group) # The has_* settings here correspond to CPUID bits. @@ -35,7 +35,7 @@ use_popcnt = And(has_popcnt, has_sse42) use_bmi1 = And(has_bmi1) use_lzcnt = And(has_lzcnt) -# Presets corresponding to Intel CPUs. +# Presets corresponding to x86 CPUs. baseline = Preset() nehalem = Preset( diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index ea31abaae5..7caca7abab 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -25,16 +25,16 @@ pub type Addend = i64; /// Relocation kinds for every ISA #[derive(Debug)] pub enum Reloc { - /// Intel PC-relative 4-byte - IntelPCRel4, - /// Intel absolute 4-byte - IntelAbs4, - /// Intel absolute 8-byte - IntelAbs8, - /// Intel GOT PC-relative 4-byte - IntelGOTPCRel4, - /// Intel PLT-relative 4-byte - IntelPLTRel4, + /// x86 PC-relative 4-byte + X86PCRel4, + /// x86 absolute 4-byte + X86Abs4, + /// x86 absolute 8-byte + X86Abs8, + /// x86 GOT PC-relative 4-byte + X86GOTPCRel4, + /// x86 PLT-relative 4-byte + X86PLTRel4, /// Arm32 call target Arm32Call, /// Arm64 call target @@ -48,11 +48,11 @@ impl fmt::Display for Reloc { /// already unambigious, e.g. cton syntax with isa specified. In other contexts, use Debug. fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match *self { - Reloc::IntelPCRel4 => write!(f, "{}", "PCRel4"), - Reloc::IntelAbs4 => write!(f, "{}", "Abs4"), - Reloc::IntelAbs8 => write!(f, "{}", "Abs8"), - Reloc::IntelGOTPCRel4 => write!(f, "{}", "GOTPCRel4"), - Reloc::IntelPLTRel4 => write!(f, "{}", "PLTRel4"), + Reloc::X86PCRel4 => write!(f, "{}", "PCRel4"), + Reloc::X86Abs4 => write!(f, "{}", "Abs4"), + Reloc::X86Abs8 => write!(f, "{}", "Abs8"), + Reloc::X86GOTPCRel4 => write!(f, "{}", "GOTPCRel4"), + Reloc::X86PLTRel4 => write!(f, "{}", "PLTRel4"), Reloc::Arm32Call | Reloc::Arm64Call | Reloc::RiscvCall => write!(f, "{}", "Call"), } } diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index 8123c5b92f..44a6611c3c 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -10,7 +10,7 @@ //! //! Branch relaxation is the process of ensuring that all branches in the function have enough //! range to encode their destination. It is common to have multiple branch encodings in an ISA. -//! For example, Intel branches can have either an 8-bit or a 32-bit displacement. +//! For example, x86 branches can have either an 8-bit or a 32-bit displacement. //! //! On RISC architectures, it can happen that conditional branches have a shorter range than //! unconditional branches: diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/cretonne/src/ir/stackslot.rs index 9dfcfa7d21..bdf2c95825 100644 --- a/lib/cretonne/src/ir/stackslot.rs +++ b/lib/cretonne/src/ir/stackslot.rs @@ -107,9 +107,8 @@ pub struct StackSlotData { /// Offset of stack slot relative to the stack pointer in the caller. /// - /// On Intel ISAs, the base address is the stack pointer *before* the return address was - /// pushed. On RISC ISAs, the base address is the value of the stack pointer on entry to the - /// function. + /// On x86, the base address is the stack pointer *before* the return address was pushed. On + /// RISC ISAs, the base address is the value of the stack pointer on entry to the function. /// /// For `OutgoingArg` stack slots, the offset is relative to the current function's stack /// pointer immediately before the call. diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/cretonne/src/isa/constraints.rs index b08f635ba4..b9a4838a9d 100644 --- a/lib/cretonne/src/isa/constraints.rs +++ b/lib/cretonne/src/isa/constraints.rs @@ -158,7 +158,7 @@ impl RecipeConstraints { /// The origin depends on the ISA and the specific instruction: /// /// - RISC-V and ARM Aarch64 use the address of the branch instruction, `origin = 0`. -/// - Intel uses the address of the instruction following the branch, `origin = 2` for a 2-byte +/// - x86 uses the address of the instruction following the branch, `origin = 2` for a 2-byte /// branch instruction. /// - ARM's A32 encoding uses the address of the branch instruction + 8 bytes, `origin = 8`. #[derive(Clone, Copy, Debug)] diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 71537a0487..5ee3359d50 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -59,8 +59,8 @@ use timing; #[cfg(build_riscv)] mod riscv; -#[cfg(build_intel)] -mod intel; +#[cfg(build_x86)] +mod x86; #[cfg(build_arm32)] mod arm32; @@ -95,7 +95,7 @@ macro_rules! isa_builder { pub fn lookup(name: &str) -> Result { match name { "riscv" => isa_builder!(riscv, build_riscv), - "intel" => isa_builder!(intel, build_intel), + "x86" => isa_builder!(x86, build_x86), "arm32" => isa_builder!(arm32, build_arm32), "arm64" => isa_builder!(arm64, build_arm64), _ => Err(LookupError::Unknown), diff --git a/lib/cretonne/src/isa/intel/abi.rs b/lib/cretonne/src/isa/x86/abi.rs similarity index 99% rename from lib/cretonne/src/isa/intel/abi.rs rename to lib/cretonne/src/isa/x86/abi.rs index 488f8a3d49..16cea926b8 100644 --- a/lib/cretonne/src/isa/intel/abi.rs +++ b/lib/cretonne/src/isa/x86/abi.rs @@ -1,4 +1,4 @@ -//! Intel ABI implementation. +//! x86 ABI implementation. use super::registers::{FPR, GPR, RU}; use abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion}; diff --git a/lib/cretonne/src/isa/intel/binemit.rs b/lib/cretonne/src/isa/x86/binemit.rs similarity index 99% rename from lib/cretonne/src/isa/intel/binemit.rs rename to lib/cretonne/src/isa/x86/binemit.rs index da14f80d29..3a7c14d7be 100644 --- a/lib/cretonne/src/isa/intel/binemit.rs +++ b/lib/cretonne/src/isa/x86/binemit.rs @@ -1,4 +1,4 @@ -//! Emitting binary Intel machine code. +//! Emitting binary x86 machine code. use super::registers::RU; use binemit::{bad_encoding, CodeSink, Reloc}; @@ -7,7 +7,7 @@ use ir::{Ebb, Function, Inst, InstructionData, Opcode, TrapCode}; use isa::{RegUnit, StackBase, StackBaseMask, StackRef}; use regalloc::RegDiversions; -include!(concat!(env!("OUT_DIR"), "/binemit-intel.rs")); +include!(concat!(env!("OUT_DIR"), "/binemit-x86.rs")); // Convert a stack base to the corresponding register. fn stk_base(base: StackBase) -> RegUnit { diff --git a/lib/cretonne/src/isa/intel/enc_tables.rs b/lib/cretonne/src/isa/x86/enc_tables.rs similarity index 98% rename from lib/cretonne/src/isa/intel/enc_tables.rs rename to lib/cretonne/src/isa/x86/enc_tables.rs index d847eb6d53..01c9ede2f0 100644 --- a/lib/cretonne/src/isa/intel/enc_tables.rs +++ b/lib/cretonne/src/isa/x86/enc_tables.rs @@ -1,4 +1,4 @@ -//! Encoding tables for Intel ISAs. +//! Encoding tables for x86 ISAs. use super::registers::*; use bitset::BitSet; @@ -12,8 +12,8 @@ use isa::enc_tables::*; use isa::encoding::RecipeSizing; use predicates; -include!(concat!(env!("OUT_DIR"), "/encoding-intel.rs")); -include!(concat!(env!("OUT_DIR"), "/legalize-intel.rs")); +include!(concat!(env!("OUT_DIR"), "/encoding-x86.rs")); +include!(concat!(env!("OUT_DIR"), "/legalize-x86.rs")); /// Expand the `sdiv` and `srem` instructions using `x86_sdivmodx`. fn expand_sdivrem( @@ -147,7 +147,7 @@ fn expand_udivrem( pos.remove_inst(); } -/// Expand the `fmin` and `fmax` instructions using the Intel `x86_fmin` and `x86_fmax` +/// Expand the `fmin` and `fmax` instructions using the x86 `x86_fmin` and `x86_fmax` /// instructions. fn expand_minmax( inst: ir::Inst, @@ -241,7 +241,7 @@ fn expand_minmax( cfg.recompute_ebb(pos.func, done); } -/// Intel has no unsigned-to-float conversions. We handle the easy case of zero-extending i32 to +/// x86 has no unsigned-to-float conversions. We handle the easy case of zero-extending i32 to /// i64 with a pattern, the rest needs more code. fn expand_fcvt_from_uint( inst: ir::Inst, diff --git a/lib/cretonne/src/isa/intel/mod.rs b/lib/cretonne/src/isa/x86/mod.rs similarity index 96% rename from lib/cretonne/src/isa/intel/mod.rs rename to lib/cretonne/src/isa/x86/mod.rs index 02cccc4f54..a272420421 100644 --- a/lib/cretonne/src/isa/intel/mod.rs +++ b/lib/cretonne/src/isa/x86/mod.rs @@ -1,4 +1,4 @@ -//! Intel Instruction Set Architectures. +//! x86 Instruction Set Architectures. mod abi; mod binemit; @@ -25,7 +25,7 @@ struct Isa { cpumode: &'static [shared_enc_tables::Level1Entry], } -/// Get an ISA builder for creating Intel targets. +/// Get an ISA builder for creating x86 targets. pub fn isa_builder() -> IsaBuilder { IsaBuilder { setup: settings::builder(), @@ -51,7 +51,7 @@ fn isa_constructor( impl TargetIsa for Isa { fn name(&self) -> &'static str { - "intel" + "x86" } fn flags(&self) -> &shared_settings::Flags { diff --git a/lib/cretonne/src/isa/intel/registers.rs b/lib/cretonne/src/isa/x86/registers.rs similarity index 95% rename from lib/cretonne/src/isa/intel/registers.rs rename to lib/cretonne/src/isa/x86/registers.rs index c972c10a13..5ca0c4912c 100644 --- a/lib/cretonne/src/isa/intel/registers.rs +++ b/lib/cretonne/src/isa/x86/registers.rs @@ -1,8 +1,8 @@ -//! Intel register descriptions. +//! x86 register descriptions. use isa::registers::{RegBank, RegClass, RegClassData, RegInfo, RegUnit}; -include!(concat!(env!("OUT_DIR"), "/registers-intel.rs")); +include!(concat!(env!("OUT_DIR"), "/registers-x86.rs")); #[cfg(test)] mod tests { diff --git a/lib/cretonne/src/isa/intel/settings.rs b/lib/cretonne/src/isa/x86/settings.rs similarity index 92% rename from lib/cretonne/src/isa/intel/settings.rs rename to lib/cretonne/src/isa/x86/settings.rs index 147af4c2fa..8bcf75249b 100644 --- a/lib/cretonne/src/isa/intel/settings.rs +++ b/lib/cretonne/src/isa/x86/settings.rs @@ -1,12 +1,12 @@ -//! Intel Settings. +//! x86 Settings. use settings::{self, detail, Builder}; use std::fmt; // Include code generated by `lib/cretonne/meta/gen_settings.py`. This file contains a public // `Flags` struct with an impl for all of the settings defined in -// `lib/cretonne/meta/isa/intel/settings.py`. -include!(concat!(env!("OUT_DIR"), "/settings-intel.rs")); +// `lib/cretonne/meta/isa/x86/settings.py`. +include!(concat!(env!("OUT_DIR"), "/settings-x86.rs")); #[cfg(test)] mod tests { diff --git a/lib/cretonne/src/regalloc/register_set.rs b/lib/cretonne/src/regalloc/register_set.rs index f2f109ecd9..65414dbd9b 100644 --- a/lib/cretonne/src/regalloc/register_set.rs +++ b/lib/cretonne/src/regalloc/register_set.rs @@ -196,7 +196,7 @@ impl<'a> fmt::Display for DisplayRegisterSet<'a> { } // Display individual registers as either the second letter of their // name or the last digit of their number. - // This works for Intel (rax, rbx, ...) and for numbered regs. + // This works for x86 (rax, rbx, ...) and for numbered regs. write!( f, "{}", diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index 15f3e69350..971abdc4f5 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -27,7 +27,7 @@ pub fn builders() -> Result<(settings::Builder, isa::Builder), &'static str> { } let name = if cfg!(any(target_arch = "x86", target_arch = "x86_64")) { - "intel" + "x86" } else if cfg!(target_arch = "arm") { "arm32" } else if cfg!(target_arch = "aarch64") { From 04746270b3254776f5a716e87513964a06b2abbd Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 13 Apr 2018 09:11:14 -0700 Subject: [PATCH 1689/3084] Rename X86Abs4/X86Abs8 to Abs4/Abs8. These relocation codes are for simple absolute addresses and aren't architecture-specific. --- lib/cretonne/meta/isa/x86/recipes.py | 12 ++++++------ lib/cretonne/src/binemit/mod.rs | 12 ++++++------ 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/lib/cretonne/meta/isa/x86/recipes.py b/lib/cretonne/meta/isa/x86/recipes.py index af1cab5792..42ba7c3852 100644 --- a/lib/cretonne/meta/isa/x86/recipes.py +++ b/lib/cretonne/meta/isa/x86/recipes.py @@ -590,7 +590,7 @@ fnaddr4 = TailRecipe( 'fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(Reloc::X86Abs4, + sink.reloc_external(Reloc::Abs4, &func.dfg.ext_funcs[func_ref].name, 0); sink.put4(0); @@ -601,7 +601,7 @@ fnaddr8 = TailRecipe( 'fnaddr8', FuncAddr, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(Reloc::X86Abs8, + sink.reloc_external(Reloc::Abs8, &func.dfg.ext_funcs[func_ref].name, 0); sink.put8(0); @@ -612,7 +612,7 @@ allones_fnaddr4 = TailRecipe( 'allones_fnaddr4', FuncAddr, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(Reloc::X86Abs4, + sink.reloc_external(Reloc::Abs4, &func.dfg.ext_funcs[func_ref].name, 0); // Write the immediate as `!0` for the benefit of BaldrMonkey. @@ -624,7 +624,7 @@ allones_fnaddr8 = TailRecipe( 'allones_fnaddr8', FuncAddr, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(Reloc::X86Abs8, + sink.reloc_external(Reloc::Abs8, &func.dfg.ext_funcs[func_ref].name, 0); // Write the immediate as `!0` for the benefit of BaldrMonkey. @@ -652,7 +652,7 @@ gvaddr4 = TailRecipe( 'gvaddr4', UnaryGlobalVar, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(Reloc::X86Abs4, + sink.reloc_external(Reloc::Abs4, &func.global_vars[global_var].symbol_name(), 0); sink.put4(0); @@ -663,7 +663,7 @@ gvaddr8 = TailRecipe( 'gvaddr8', UnaryGlobalVar, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - sink.reloc_external(Reloc::X86Abs8, + sink.reloc_external(Reloc::Abs8, &func.global_vars[global_var].symbol_name(), 0); sink.put8(0); diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/cretonne/src/binemit/mod.rs index 7caca7abab..9c78e4f784 100644 --- a/lib/cretonne/src/binemit/mod.rs +++ b/lib/cretonne/src/binemit/mod.rs @@ -25,12 +25,12 @@ pub type Addend = i64; /// Relocation kinds for every ISA #[derive(Debug)] pub enum Reloc { + /// absolute 4-byte + Abs4, + /// absolute 8-byte + Abs8, /// x86 PC-relative 4-byte X86PCRel4, - /// x86 absolute 4-byte - X86Abs4, - /// x86 absolute 8-byte - X86Abs8, /// x86 GOT PC-relative 4-byte X86GOTPCRel4, /// x86 PLT-relative 4-byte @@ -48,9 +48,9 @@ impl fmt::Display for Reloc { /// already unambigious, e.g. cton syntax with isa specified. In other contexts, use Debug. fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match *self { + Reloc::Abs4 => write!(f, "{}", "Abs4"), + Reloc::Abs8 => write!(f, "{}", "Abs8"), Reloc::X86PCRel4 => write!(f, "{}", "PCRel4"), - Reloc::X86Abs4 => write!(f, "{}", "Abs4"), - Reloc::X86Abs8 => write!(f, "{}", "Abs8"), Reloc::X86GOTPCRel4 => write!(f, "{}", "GOTPCRel4"), Reloc::X86PLTRel4 => write!(f, "{}", "PLTRel4"), Reloc::Arm32Call | Reloc::Arm64Call | Reloc::RiscvCall => write!(f, "{}", "Call"), From 25d2ab4c00d5e68cde6d46f529fbbfdec85444e8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 13 Apr 2018 09:58:36 -0700 Subject: [PATCH 1690/3084] Change the Travis config to allow failures in rust beta. Rustc beta and nightly crash when compiling Cretonne. I've filed https://github.com/rust-lang/rust/issues/49528 to track this upstream. For now, add Beta to the allow-failures list to temporarily work around this. --- .travis.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 301d7871d0..83977c9a89 100644 --- a/.travis.yml +++ b/.travis.yml @@ -5,7 +5,9 @@ rust: - nightly matrix: allow_failures: - - rust: nightly + - rust: + - beta + - nightly dist: trusty sudo: false addons: From 98a4e3223676ce25e1bd9b5725dc0e56780dbcae Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 13 Apr 2018 10:24:58 -0700 Subject: [PATCH 1691/3084] Try a different syntax for allowing failures on rust beta. --- .travis.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.travis.yml b/.travis.yml index 83977c9a89..04e455a285 100644 --- a/.travis.yml +++ b/.travis.yml @@ -5,9 +5,8 @@ rust: - nightly matrix: allow_failures: - - rust: - - beta - - nightly + - rust: beta + - rust: nightly dist: trusty sudo: false addons: From 645fa3e858180a4a81140a8484e9d3925493a35f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 13 Apr 2018 13:48:16 -0700 Subject: [PATCH 1692/3084] Minor code cleanup. --- lib/cretonne/src/ir/function.rs | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index 19baa16b2b..f7dee1b918 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -179,9 +179,7 @@ impl Function { /// Wrapper around `DataFlowGraph::encode` which assigns `inst` the resulting encoding. pub fn update_encoding(&mut self, inst: ir::Inst, isa: &TargetIsa) -> Result<(), Legalize> { - self.dfg.encode(inst, isa).map( - |e| { self.encodings[inst] = e; }, - ) + self.dfg.encode(inst, isa).map(|e| self.encodings[inst] = e) } } From 0e57f3d0ead1626b4eb1309224125ac038284d19 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 13 Apr 2018 15:00:09 -0700 Subject: [PATCH 1693/3084] Add a "colocated" flag to symbol references. (#298) This adds a "colocated" flag to function and symbolic global variables which indicates that they are defined along with the current function, so they can use PC-relative addressing. This also changes the function decl syntax; the name now always precedes the signature, and the "function" keyword is no longer included. --- cranelift/docs/callex.cton | 2 +- cranelift/docs/langref.rst | 15 ++-- cranelift/filetests/isa/riscv/binary32.cton | 2 +- .../filetests/isa/riscv/legalize-abi.cton | 12 ++-- .../filetests/isa/riscv/parse-encoding.cton | 4 +- .../filetests/isa/riscv/verify-encoding.cton | 4 +- cranelift/filetests/isa/x86/abi64.cton | 2 +- .../isa/x86/allones_funcaddrs32.cton | 2 +- .../isa/x86/allones_funcaddrs64.cton | 2 +- cranelift/filetests/isa/x86/binary32.cton | 2 +- cranelift/filetests/isa/x86/binary64-pic.cton | 14 +++- cranelift/filetests/isa/x86/binary64.cton | 10 +-- .../filetests/isa/x86/legalize-libcall.cton | 2 +- .../filetests/isa/x86/prologue-epilogue.cton | 4 +- cranelift/filetests/parser/call.cton | 16 ++--- cranelift/filetests/regalloc/basic.cton | 4 +- .../filetests/regalloc/coalescing-207.cton | 8 +-- cranelift/filetests/regalloc/reload-208.cton | 4 +- cranelift/filetests/regalloc/reload.cton | 2 +- cranelift/filetests/regalloc/spill.cton | 4 +- cranelift/filetests/verifier/type_check.cton | 4 +- lib/cretonne/meta/base/predicates.py | 35 +++++++++ lib/cretonne/meta/cdsl/predicates.py | 8 +-- lib/cretonne/meta/gen_encoding.py | 16 ++--- lib/cretonne/meta/gen_legalizer.py | 6 +- lib/cretonne/meta/isa/x86/encodings.py | 25 ++++++- lib/cretonne/meta/isa/x86/recipes.py | 29 ++++++++ lib/cretonne/src/binemit/relaxation.rs | 2 +- lib/cretonne/src/cursor.rs | 2 +- lib/cretonne/src/ir/dfg.rs | 8 +-- lib/cretonne/src/ir/extfunc.rs | 9 ++- lib/cretonne/src/ir/function.rs | 12 +++- lib/cretonne/src/ir/globalvar.rs | 17 ++++- lib/cretonne/src/isa/arm32/mod.rs | 4 +- lib/cretonne/src/isa/arm64/mod.rs | 4 +- lib/cretonne/src/isa/enc_tables.rs | 16 ++--- lib/cretonne/src/isa/mod.rs | 6 +- lib/cretonne/src/isa/riscv/mod.rs | 47 ++++++------ lib/cretonne/src/isa/x86/mod.rs | 4 +- lib/cretonne/src/legalizer/libcall.rs | 2 + lib/cretonne/src/predicates.rs | 15 ++++ lib/cretonne/src/verifier/mod.rs | 6 +- lib/cretonne/src/write.rs | 3 +- lib/filetests/src/test_binemit.rs | 4 +- lib/reader/src/parser.rs | 71 ++++++++++--------- lib/wasm/src/environ/dummy.rs | 6 +- 46 files changed, 312 insertions(+), 164 deletions(-) create mode 100644 lib/cretonne/meta/base/predicates.py diff --git a/cranelift/docs/callex.cton b/cranelift/docs/callex.cton index 837f9ea6e7..26abda55b4 100644 --- a/cranelift/docs/callex.cton +++ b/cranelift/docs/callex.cton @@ -1,7 +1,7 @@ test verifier function %gcd(i32 uext, i32 uext) -> i32 uext system_v { - fn1 = function %divmod(i32 uext, i32 uext) -> i32 uext, i32 uext + fn1 = %divmod(i32 uext, i32 uext) -> i32 uext, i32 uext ebb1(v1: i32, v2: i32): brz v2, ebb2 diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 14f44bab53..070b27ee37 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -48,8 +48,7 @@ A ``.cton`` file consists of a sequence of independent function definitions: .. productionlist:: function_list : { function } - function : function_spec "{" preamble function_body "}" - function_spec : "function" function_name signature + function : "function" function_name signature "{" preamble function_body "}" preamble : { preamble_decl } function_body : { extended_basic_block } @@ -409,10 +408,14 @@ compilers. Functions that are called directly must be declared in the :term:`function preamble`: -.. inst:: FN = function NAME signature +.. inst:: FN = [colocated] NAME signature Declare a function so it can be called directly. + If the colocated keyword is present, the symbol's definition will be + defined along with the current function, such that it can use more + efficient addressing. + :arg NAME: Name of the function, passed to the linker for resolution. :arg signature: Function signature. See below. :result FN: A function identifier that can be used with :inst:`call`. @@ -570,13 +573,17 @@ runtime data structures. variable. :result GV: Global variable. -.. inst:: GV = globalsym name +.. inst:: GV = [colocated] globalsym name Declare a global variable at a symbolic address. The address of GV is symbolic and will be assigned a relocation, so that it can be resolved by a later linking phase. + If the colocated keyword is present, the symbol's definition will be + defined along with the current function, such that it can use more + efficient addressing. + :arg name: External name. :result GV: Global variable. diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 459d9d7419..7837672bcc 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -4,7 +4,7 @@ isa riscv function %RV32I(i32 link [%x1]) -> i32 link [%x1] { sig0 = () - fn0 = function %foo() + fn0 = %foo() ebb0(v9999: i32): [-,%x10] v1 = iconst.i32 1 diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index 44865120ea..df60b91f81 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -17,8 +17,8 @@ ebb0(v0: i64): } function %split_call_arg(i32) { - fn1 = function %foo(i64) - fn2 = function %foo(i32, i64) + fn1 = %foo(i64) + fn2 = %foo(i32, i64) ebb0(v0: i32): v1 = uextend.i64 v0 call fn1(v1) @@ -30,7 +30,7 @@ ebb0(v0: i32): } function %split_ret_val() { - fn1 = function %foo() -> i64 + fn1 = %foo() -> i64 ebb0: v1 = call fn1() ; check: ebb0($(link=$V): i32): @@ -45,7 +45,7 @@ ebb1(v10: i64): ; First return value is fine, second one is expanded. function %split_ret_val2() { - fn1 = function %foo() -> i32, i64 + fn1 = %foo() -> i32, i64 ebb0: v1, v2 = call fn1() ; check: ebb0($(link=$V): i32): @@ -70,7 +70,7 @@ ebb0(v1: i8, v2: i8, v3: i8): ; Function produces single return value, still need to copy. function %ext_ret_val() { - fn1 = function %foo() -> i8 sext + fn1 = %foo() -> i8 sext ebb0: v1 = call fn1() ; check: ebb0($V: i32): @@ -124,7 +124,7 @@ ebb0(v0: i32, v1: f32x2): ; Call a function that takes arguments on the stack. function %stack_args(i32) { ; check: $(ss0=$SS) = outgoing_arg 4 - fn1 = function %foo(i64, i64, i64, i64, i32) + fn1 = %foo(i64, i64, i64, i64, i32) ebb0(v0: i32): v1 = iconst.i64 1 call fn1(v1, v1, v1, v1, v0) diff --git a/cranelift/filetests/isa/riscv/parse-encoding.cton b/cranelift/filetests/isa/riscv/parse-encoding.cton index 0fc0879f38..b8220b1688 100644 --- a/cranelift/filetests/isa/riscv/parse-encoding.cton +++ b/cranelift/filetests/isa/riscv/parse-encoding.cton @@ -27,9 +27,9 @@ function %parse_encoding(i32 [%x5]) -> i32 [%x10] { ; check: sig5 = () -> f32 [0] system_v ; function + signature - fn0 = function %bar(i32 [%x10]) -> b1 [%x10] system_v + fn0 = %bar(i32 [%x10]) -> b1 [%x10] system_v ; check: sig6 = (i32 [%x10]) -> b1 [%x10] system_v - ; nextln: fn0 = sig6 %bar + ; nextln: fn0 = %bar sig6 ebb0(v0: i32): return v0 diff --git a/cranelift/filetests/isa/riscv/verify-encoding.cton b/cranelift/filetests/isa/riscv/verify-encoding.cton index 725b9e5744..1aaedf1c89 100644 --- a/cranelift/filetests/isa/riscv/verify-encoding.cton +++ b/cranelift/filetests/isa/riscv/verify-encoding.cton @@ -2,7 +2,7 @@ test verifier isa riscv function %RV32I(i32 link [%x1]) -> i32 link [%x1] { - fn0 = function %foo() + fn0 = %foo() ebb0(v9999: i32): ; iconst.i32 needs legalizing, so it should throw a @@ -11,7 +11,7 @@ ebb0(v9999: i32): } function %RV32I(i32 link [%x1]) -> i32 link [%x1] { - fn0 = function %foo() + fn0 = %foo() ebb0(v9999: i32): v1 = iconst.i32 1 diff --git a/cranelift/filetests/isa/x86/abi64.cton b/cranelift/filetests/isa/x86/abi64.cton index 2cb63c0dab..40321f90fb 100644 --- a/cranelift/filetests/isa/x86/abi64.cton +++ b/cranelift/filetests/isa/x86/abi64.cton @@ -21,7 +21,7 @@ ebb0: function %pass_stack_int64(i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) spiderwasm { sig0 = (i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) spiderwasm - fn0 = sig0 u0:0 + fn0 = u0:0 sig0 ebb0(v0: i64, v1: i64, v2: i64, v3: i64, v4: i64, v5: i64, v6: i64, v7: i64, v8: i64, v9: i64, v10: i64, v11: i64, v12: i64, v13: i64, v14: i64, v15: i64, v16: i64, v17: i64, v18: i64, v19: i64, v20: i64): call fn0(v0, v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16, v17, v18, v19, v20) diff --git a/cranelift/filetests/isa/x86/allones_funcaddrs32.cton b/cranelift/filetests/isa/x86/allones_funcaddrs32.cton index d549c8d5e3..c4c078470b 100644 --- a/cranelift/filetests/isa/x86/allones_funcaddrs32.cton +++ b/cranelift/filetests/isa/x86/allones_funcaddrs32.cton @@ -12,7 +12,7 @@ isa x86 haswell ; Tests from binary32.cton affected by allones_funcaddrs. function %I32() { sig0 = () - fn0 = function %foo() + fn0 = %foo() ebb0: diff --git a/cranelift/filetests/isa/x86/allones_funcaddrs64.cton b/cranelift/filetests/isa/x86/allones_funcaddrs64.cton index 478c875e3d..ca3cf1ab69 100644 --- a/cranelift/filetests/isa/x86/allones_funcaddrs64.cton +++ b/cranelift/filetests/isa/x86/allones_funcaddrs64.cton @@ -13,7 +13,7 @@ isa x86 haswell ; Tests from binary64.cton affected by allones_funcaddrs. function %I64() { sig0 = () - fn0 = function %foo() + fn0 = %foo() ebb0: diff --git a/cranelift/filetests/isa/x86/binary32.cton b/cranelift/filetests/isa/x86/binary32.cton index 342af80140..842f941779 100644 --- a/cranelift/filetests/isa/x86/binary32.cton +++ b/cranelift/filetests/isa/x86/binary32.cton @@ -10,7 +10,7 @@ isa x86 haswell function %I32() { sig0 = () - fn0 = function %foo() + fn0 = %foo() gv0 = globalsym %some_gv diff --git a/cranelift/filetests/isa/x86/binary64-pic.cton b/cranelift/filetests/isa/x86/binary64-pic.cton index e1cd79ac71..f3b838795a 100644 --- a/cranelift/filetests/isa/x86/binary64-pic.cton +++ b/cranelift/filetests/isa/x86/binary64-pic.cton @@ -13,9 +13,11 @@ isa x86 haswell ; Tests for i64 instructions. function %I64() { sig0 = () - fn0 = function %foo() + fn0 = %foo() + fn1 = colocated %bar() gv0 = globalsym %some_gv + gv1 = globalsym colocated %some_gv ; Use incoming_arg stack slots because they won't be relocated by the frame ; layout. @@ -29,6 +31,9 @@ ebb0: ; asm: call foo@PLT call fn0() ; bin: e8 PLTRel4(%foo-4) 00000000 + ; asm: call foo + call fn1() ; bin: e8 PCRel4(%bar-4) 00000000 + ; asm: mov 0x0(%rip), %rax [-,%rax] v0 = func_addr.i64 fn0 ; bin: 48 8b 05 GOTPCRel4(%foo-4) 00000000 ; asm: mov 0x0(%rip), %rsi @@ -50,5 +55,12 @@ ebb0: ; asm: mov 0x0(%rip), %r10 [-,%r10] v5 = globalsym_addr.i64 gv0 ; bin: 4c 8b 15 GOTPCRel4(%some_gv-4) 00000000 + ; asm: lea 0x0(%rip), %rcx + [-,%rcx] v6 = globalsym_addr.i64 gv1 ; bin: 48 8d 0d PCRel4(%some_gv-4) 00000000 + ; asm: lea 0x0(%rip), %rsi + [-,%rsi] v7 = globalsym_addr.i64 gv1 ; bin: 48 8d 35 PCRel4(%some_gv-4) 00000000 + ; asm: lea 0x0(%rip), %r10 + [-,%r10] v8 = globalsym_addr.i64 gv1 ; bin: 4c 8d 15 PCRel4(%some_gv-4) 00000000 + return } diff --git a/cranelift/filetests/isa/x86/binary64.cton b/cranelift/filetests/isa/x86/binary64.cton index 37f026f2ab..6ecc086231 100644 --- a/cranelift/filetests/isa/x86/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.cton @@ -12,7 +12,7 @@ isa x86 haswell ; Tests for i64 instructions. function %I64() { sig0 = () - fn0 = function %foo() + fn0 = %foo() gv0 = globalsym %some_gv @@ -473,12 +473,6 @@ ebb0: ; asm: movzbq %dl, %rsi [-,%rsi] v351 = bint.i64 v301 ; bin: 0f b6 f2 - ; TODO: x86-64 can't encode a direct call to an arbitrary 64-bit address in - ; a single instruction. When we add a concept of colocated definitions, this - ; test can be re-enabled. - ; disabled: asm: call foo - ; disabled: call fn0() ; bin: e8 PCRel4(%foo-4) 00000000 - ; asm: movabsq $0, %rcx [-,%rcx] v400 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(%foo) 0000000000000000 ; asm: movabsq $0, %rsi @@ -713,7 +707,7 @@ ebb0: ; be done by an instruction shrinking pass. function %I32() { sig0 = () - fn0 = function %foo() + fn0 = %foo() ss0 = incoming_arg 8, offset 0 ss1 = incoming_arg 1024, offset -1024 diff --git a/cranelift/filetests/isa/x86/legalize-libcall.cton b/cranelift/filetests/isa/x86/legalize-libcall.cton index bbe332170e..70d392e6e4 100644 --- a/cranelift/filetests/isa/x86/legalize-libcall.cton +++ b/cranelift/filetests/isa/x86/legalize-libcall.cton @@ -12,5 +12,5 @@ ebb0(v0: f32): } ; check: function %floor(f32 [%xmm0]) -> f32 [%xmm0] system_v { ; check: sig0 = (f32) -> f32 system_v -; check: fn0 = sig0 %FloorF32 +; check: fn0 = %FloorF32 sig0 ; check: v1 = call fn0(v0) diff --git a/cranelift/filetests/isa/x86/prologue-epilogue.cton b/cranelift/filetests/isa/x86/prologue-epilogue.cton index eee6917b1b..9ed37274ca 100644 --- a/cranelift/filetests/isa/x86/prologue-epilogue.cton +++ b/cranelift/filetests/isa/x86/prologue-epilogue.cton @@ -45,7 +45,7 @@ ebb0: ; A function performing a call. function %call() { - fn0 = function %foo() + fn0 = %foo() ebb0: call fn0() @@ -55,7 +55,7 @@ ebb0: ; check: function %call(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { ; nextln: ss0 = incoming_arg 16, offset -16 ; nextln: sig0 = () system_v -; nextln: fn0 = sig0 %foo +; nextln: fn0 = %foo sig0 ; nextln: ; nextln: ebb0(v0: i64 [%rbp]): ; nextln: x86_push v0 diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton index 3413696caf..9d1e8def02 100644 --- a/cranelift/filetests/parser/call.cton +++ b/cranelift/filetests/parser/call.cton @@ -26,22 +26,22 @@ ebb1: function %signatures() { sig10 = () sig11 = (i32, f64) -> i32, b1 spiderwasm - fn5 = sig11 %foo - fn8 = function %bar(i32) -> b1 + fn5 = %foo sig11 + fn8 = %bar(i32) -> b1 } ; sameln: function %signatures() system_v { ; check: sig10 = () system_v ; check: sig11 = (i32, f64) -> i32, b1 spiderwasm ; check: sig12 = (i32) -> b1 system_v ; not: fn0 -; check: fn5 = sig11 %foo -; check: fn8 = sig12 %bar +; check: fn5 = %foo sig11 +; check: fn8 = %bar sig12 ; check: } function %direct() { - fn0 = function %none() - fn1 = function %one() -> i32 - fn2 = function %two() -> i32, f32 + fn0 = %none() + fn1 = %one() -> i32 + fn2 = %two() -> i32, f32 ebb0: call fn0() @@ -72,7 +72,7 @@ ebb0(v0: i64): function %long_call() { sig0 = () - fn0 = sig0 %none + fn0 = %none sig0 ebb0: v0 = func_addr.i32 fn0 diff --git a/cranelift/filetests/regalloc/basic.cton b/cranelift/filetests/regalloc/basic.cton index b9399f464d..c24f87c1f9 100644 --- a/cranelift/filetests/regalloc/basic.cton +++ b/cranelift/filetests/regalloc/basic.cton @@ -53,7 +53,7 @@ ebb1(v10: i32): ; Pass an EBB argument as a function argument. function %callebb(i32, i32) -> i32 { - fn0 = function %foo(i32) -> i32 + fn0 = %foo(i32) -> i32 ebb0(v1: i32, v2: i32): brnz v1, ebb1(v1) @@ -66,7 +66,7 @@ ebb1(v10: i32): ; Pass an EBB argument as a jump argument. function %jumpebb(i32, i32) -> i32 { - fn0 = function %foo(i32) -> i32 + fn0 = %foo(i32) -> i32 ebb0(v1: i32, v2: i32): brnz v1, ebb1(v1, v2) diff --git a/cranelift/filetests/regalloc/coalescing-207.cton b/cranelift/filetests/regalloc/coalescing-207.cton index 9066bd7c4f..0deaa454f4 100644 --- a/cranelift/filetests/regalloc/coalescing-207.cton +++ b/cranelift/filetests/regalloc/coalescing-207.cton @@ -11,9 +11,9 @@ function %pr207(i64 vmctx, i32, i32) -> i32 system_v { sig0 = (i64 vmctx, i32, i32) -> i32 system_v sig1 = (i64 vmctx, i32, i32, i32) -> i32 system_v sig2 = (i64 vmctx, i32, i32, i32) -> i32 system_v - fn0 = sig0 u0:2 - fn1 = sig1 u0:0 - fn2 = sig2 u0:1 + fn0 = u0:2 sig0 + fn1 = u0:0 sig1 + fn2 = u0:1 sig2 ebb0(v0: i64, v1: i32, v2: i32): v3 = iconst.i32 0 @@ -1038,7 +1038,7 @@ function %musl(f64 [%xmm0], i64 vmctx [%rdi]) -> f64 [%xmm0] system_v { gv0 = vmctx heap0 = static gv0, min 0, bound 0x0001_0000_0000, guard 0x8000_0000 sig0 = (f64 [%xmm0], i32 [%rdi], i64 vmctx [%rsi]) -> f64 [%xmm0] system_v - fn0 = sig0 u0:517 + fn0 = u0:517 sig0 ebb0(v0: f64, v1: i64): v3 = iconst.i64 0 diff --git a/cranelift/filetests/regalloc/reload-208.cton b/cranelift/filetests/regalloc/reload-208.cton index 8ccd3e6ef3..bb775acebb 100644 --- a/cranelift/filetests/regalloc/reload-208.cton +++ b/cranelift/filetests/regalloc/reload-208.cton @@ -16,8 +16,8 @@ function %pr208(i64 vmctx [%rdi]) system_v { heap0 = static gv0, min 0, bound 0x5000, guard 0x0040_0000 sig0 = (i64 vmctx [%rdi]) -> i32 [%rax] system_v sig1 = (i64 vmctx [%rdi], i32 [%rsi]) system_v - fn0 = sig0 u0:1 - fn1 = sig1 u0:3 + fn0 = u0:1 sig0 + fn1 = u0:3 sig1 ebb0(v0: i64): v1 = iconst.i32 0 diff --git a/cranelift/filetests/regalloc/reload.cton b/cranelift/filetests/regalloc/reload.cton index 5e62db3213..d0a9646808 100644 --- a/cranelift/filetests/regalloc/reload.cton +++ b/cranelift/filetests/regalloc/reload.cton @@ -5,7 +5,7 @@ isa riscv enable_e ; Check that we can handle a function return value that got spilled. function %spill_return() -> i32 { - fn0 = function %foo() -> i32 system_v + fn0 = %foo() -> i32 system_v ebb0: v0 = call fn0() diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index fb822e1a6f..7205d6a851 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -70,7 +70,7 @@ ebb0(v1: i32): ; All values live across a call must be spilled function %across_call(i32) { - fn0 = function %foo(i32) + fn0 = %foo(i32) ebb0(v1: i32): ; check: v1 = spill call fn0(v1) @@ -83,7 +83,7 @@ ebb0(v1: i32): ; The same value used for two function arguments. function %doubleuse(i32) { - fn0 = function %xx(i32, i32) + fn0 = %xx(i32, i32) ebb0(v0: i32): ; check: $(c=$V) = copy v0 call fn0(v0, v0) diff --git a/cranelift/filetests/verifier/type_check.cton b/cranelift/filetests/verifier/type_check.cton index 4bd386efbf..8783c4aced 100644 --- a/cranelift/filetests/verifier/type_check.cton +++ b/cranelift/filetests/verifier/type_check.cton @@ -42,14 +42,14 @@ function %type_mismatch_controlling_variable() { } function %fn_call_too_few_args() { - fn2 = function %great_fn(i32, f32) + fn2 = %great_fn(i32, f32) ebb0: call fn2() ; error: mismatched argument count, got 0, expected 2 return } function %fn_call_too_many_args() { - fn5 = function %best_fn() + fn5 = %best_fn() ebb0: v0 = iconst.i64 56 v1 = f32const 0.0 diff --git a/lib/cretonne/meta/base/predicates.py b/lib/cretonne/meta/base/predicates.py new file mode 100644 index 0000000000..1a6b4c2c75 --- /dev/null +++ b/lib/cretonne/meta/base/predicates.py @@ -0,0 +1,35 @@ +""" +Cretonne predicates that consider `Function` fields. +""" +from cdsl.predicates import FieldPredicate +from .formats import UnaryGlobalVar + +try: + from typing import TYPE_CHECKING + if TYPE_CHECKING: + from cdsl.formats import FormatField # noqa +except ImportError: + pass + + +class IsColocatedFunc(FieldPredicate): + """ + An instruction predicate that checks the referenced function is colocated. + """ + + def __init__(self, field): + # type: (FormatField) -> None + super(IsColocatedFunc, self).__init__( + field, 'is_colocated_func', ('func',)) + + +class IsColocatedData(FieldPredicate): + """ + An instruction predicate that checks the referenced data object is + colocated. + """ + + def __init__(self): + # type: () -> None + super(IsColocatedData, self).__init__( + UnaryGlobalVar.global_var, 'is_colocated_data', ('func',)) diff --git a/lib/cretonne/meta/cdsl/predicates.py b/lib/cretonne/meta/cdsl/predicates.py index 9a28fd9945..13957c5d02 100644 --- a/lib/cretonne/meta/cdsl/predicates.py +++ b/lib/cretonne/meta/cdsl/predicates.py @@ -370,9 +370,9 @@ class TypePredicate(object): """ Return Rust code for evaluating this predicate. - It is assumed that the context has `dfg` and `args` variables. + It is assumed that the context has `func` and `args` variables. """ - return 'dfg.value_type(args[{}]) == {}'.format( + return 'func.dfg.value_type(args[{}]) == {}'.format( self.value_arg, self.value_type.rust_name()) @@ -409,7 +409,7 @@ class CtrlTypePredicate(object): """ Return Rust code for evaluating this predicate. - It is assumed that the context has `dfg` and `inst` variables. + It is assumed that the context has `func` and `inst` variables. """ - return 'dfg.ctrl_typevar(inst) == {}'.format( + return 'func.dfg.ctrl_typevar(inst) == {}'.format( self.value_type.rust_name()) diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/cretonne/meta/gen_encoding.py index 61ceb8d498..e3915100a2 100644 --- a/lib/cretonne/meta/gen_encoding.py +++ b/lib/cretonne/meta/gen_encoding.py @@ -74,7 +74,7 @@ except ImportError: pass -def emit_instp(instp, fmt, has_dfg=False): +def emit_instp(instp, fmt, has_func=False): # type: (PredNode, srcgen.Formatter, bool) -> None """ Emit code for matching an instruction predicate against an @@ -87,7 +87,7 @@ def emit_instp(instp, fmt, has_dfg=False): # Deal with pure type check predicates which apply to any instruction. if iform == instruction_context: - fmt.line('let args = inst.arguments(&dfg.value_lists);') + fmt.line('let args = inst.arguments(&func.dfg.value_lists);') fmt.line(instp.rust_predicate(0)) return @@ -114,11 +114,11 @@ def emit_instp(instp, fmt, has_dfg=False): .format(iform.name, fields), '}'): if has_type_check: # We could implement this if we need to. - assert has_dfg, "Recipe predicates can't check type variables." - fmt.line('let args = inst.arguments(&dfg.value_lists);') - elif has_dfg: + assert has_func, "Recipe predicates can't check type variables." + fmt.line('let args = inst.arguments(&func.dfg.value_lists);') + elif has_func: # Silence dead argument warning. - fmt.line('let _ = dfg;') + fmt.line('let _ = func;') fmt.format('return {};', instp.rust_predicate(0)) fmt.line('unreachable!();') @@ -132,9 +132,9 @@ def emit_inst_predicates(instps, fmt): for instp, number in instps.items(): name = 'inst_predicate_{}'.format(number) with fmt.indented( - 'fn {}(dfg: &ir::DataFlowGraph, inst: &ir::InstructionData)' + 'fn {}(func: &ir::Function, inst: &ir::InstructionData)' '-> bool {{'.format(name), '}'): - emit_instp(instp, fmt, has_dfg=True) + emit_instp(instp, fmt, has_func=True) # Generate the static table. with fmt.indented( diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/cretonne/meta/gen_legalizer.py index 9051337238..104574b7b6 100644 --- a/lib/cretonne/meta/gen_legalizer.py +++ b/lib/cretonne/meta/gen_legalizer.py @@ -185,9 +185,9 @@ def unwrap_inst(iref, node, fmt): fmt.line('ref args,') fmt.line('..') fmt.outdented_line('} = pos.func.dfg[inst] {') - fmt.line('let dfg = &pos.func.dfg;') + fmt.line('let func = &pos.func;') if iform.has_value_list: - fmt.line('let args = args.as_slice(&dfg.value_lists);') + fmt.line('let args = args.as_slice(&func.dfg.value_lists);') elif nvops == 1: fmt.line('let args = [arg];') # Generate the values for the tuple. @@ -198,7 +198,7 @@ def unwrap_inst(iref, node, fmt): fmt.format('{},', iform.imm_fields[n].member) elif op.is_value(): n = expr.inst.value_opnums.index(opnum) - fmt.format('dfg.resolve_aliases(args[{}]),', n) + fmt.format('func.dfg.resolve_aliases(args[{}]),', n) # Evaluate the instruction predicate, if any. instp = expr.inst_predicate_with_ctrl_typevar() fmt.line(instp.rust_predicate(0) if instp else 'true') diff --git a/lib/cretonne/meta/isa/x86/encodings.py b/lib/cretonne/meta/isa/x86/encodings.py index 915e75436d..c120fe1a58 100644 --- a/lib/cretonne/meta/isa/x86/encodings.py +++ b/lib/cretonne/meta/isa/x86/encodings.py @@ -3,8 +3,9 @@ x86 Encodings. """ from __future__ import absolute_import from cdsl.predicates import IsUnsignedInt, Not, And +from base.predicates import IsColocatedFunc, IsColocatedData from base import instructions as base -from base.formats import UnaryImm +from base.formats import UnaryImm, FuncAddr, Call from .defs import X86_64, X86_32 from . import recipes as r from . import settings as cfg @@ -292,16 +293,22 @@ enc_both(base.regspill.f64, r.fregspill32, 0xf2, 0x0f, 0x11) # Function addresses. # +# Non-PIC, all-ones funcaddresses. X86_32.enc(base.func_addr.i32, *r.fnaddr4(0xb8), isap=And(Not(allones_funcaddrs), Not(is_pic))) X86_64.enc(base.func_addr.i64, *r.fnaddr8.rex(0xb8, w=1), isap=And(Not(allones_funcaddrs), Not(is_pic))) +# Non-PIC, all-zeros funcaddresses. X86_32.enc(base.func_addr.i32, *r.allones_fnaddr4(0xb8), isap=And(allones_funcaddrs, Not(is_pic))) X86_64.enc(base.func_addr.i64, *r.allones_fnaddr8.rex(0xb8, w=1), isap=And(allones_funcaddrs, Not(is_pic))) +# PIC +X86_64.enc(base.func_addr.i64, *r.pcrel_fnaddr8.rex(0x8d, w=1), + isap=is_pic, + instp=IsColocatedFunc(FuncAddr.func_ref)) X86_64.enc(base.func_addr.i64, *r.got_fnaddr8.rex(0x8b, w=1), isap=is_pic) @@ -309,18 +316,34 @@ X86_64.enc(base.func_addr.i64, *r.got_fnaddr8.rex(0x8b, w=1), # Global addresses. # +# Non-PIC X86_32.enc(base.globalsym_addr.i32, *r.gvaddr4(0xb8), isap=Not(is_pic)) X86_64.enc(base.globalsym_addr.i64, *r.gvaddr8.rex(0xb8, w=1), isap=Not(is_pic)) +# PIC, colocated +X86_64.enc(base.globalsym_addr.i64, *r.pcrel_gvaddr8.rex(0x8d, w=1), + isap=is_pic, + instp=IsColocatedData()) + +# PIC, non-colocated X86_64.enc(base.globalsym_addr.i64, *r.got_gvaddr8.rex(0x8b, w=1), isap=is_pic) # # Call/return # + +# 32-bit, both PIC and non-PIC. X86_32.enc(base.call, *r.call_id(0xe8)) + +# 64-bit, PIC, colocated and non-colocated. There is no 64-bit non-PIC, since +# non-PIC is currently using the large model, which requires calls be lowered +# to func_addr+call_indirect. +X86_64.enc(base.call, *r.call_id(0xe8), + isap=is_pic, + instp=IsColocatedFunc(Call.func_ref)) X86_64.enc(base.call, *r.call_plt_id(0xe8), isap=is_pic) X86_32.enc(base.call_indirect.i32, *r.call_r(0xff, rrr=2)) diff --git a/lib/cretonne/meta/isa/x86/recipes.py b/lib/cretonne/meta/isa/x86/recipes.py index 42ba7c3852..5d0b045ead 100644 --- a/lib/cretonne/meta/isa/x86/recipes.py +++ b/lib/cretonne/meta/isa/x86/recipes.py @@ -631,6 +631,21 @@ allones_fnaddr8 = TailRecipe( sink.put8(!0); ''') +pcrel_fnaddr8 = TailRecipe( + 'pcrel_fnaddr8', FuncAddr, size=5, ins=(), outs=GPR, + # rex2 gets passed 0 for r/m register because the upper bit of + # r/m doesnt get decoded when in rip-relative addressing mode. + emit=''' + PUT_OP(bits, rex2(0, out_reg0), sink); + modrm_riprel(out_reg0, sink); + // The addend adjusts for the difference between the end of the + // instruction and the beginning of the immediate field. + sink.reloc_external(Reloc::X86PCRel4, + &func.dfg.ext_funcs[func_ref].name, + -4); + sink.put4(0); + ''') + got_fnaddr8 = TailRecipe( 'got_fnaddr8', FuncAddr, size=5, ins=(), outs=GPR, # rex2 gets passed 0 for r/m register because the upper bit of @@ -669,6 +684,20 @@ gvaddr8 = TailRecipe( sink.put8(0); ''') +# XX+rd iq with PCRel4 globalsym relocation. +pcrel_gvaddr8 = TailRecipe( + 'pcrel_gvaddr8', UnaryGlobalVar, size=5, ins=(), outs=GPR, + emit=''' + PUT_OP(bits, rex2(0, out_reg0), sink); + modrm_rm(5, out_reg0, sink); + // The addend adjusts for the difference between the end of the + // instruction and the beginning of the immediate field. + sink.reloc_external(Reloc::X86PCRel4, + &func.global_vars[global_var].symbol_name(), + -4); + sink.put4(0); + ''') + # XX+rd iq with Abs8 globalsym relocation. got_gvaddr8 = TailRecipe( 'got_gvaddr8', UnaryGlobalVar, size=5, ins=(), outs=GPR, diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/cretonne/src/binemit/relaxation.rs index 44a6611c3c..93c6afedec 100644 --- a/lib/cretonne/src/binemit/relaxation.rs +++ b/lib/cretonne/src/binemit/relaxation.rs @@ -148,7 +148,7 @@ fn relax_branch( // Pick the first encoding that can handle the branch range. let dfg = &cur.func.dfg; let ctrl_type = dfg.ctrl_typevar(inst); - if let Some(enc) = isa.legal_encodings(dfg, &dfg[inst], ctrl_type).find( + if let Some(enc) = isa.legal_encodings(cur.func, &dfg[inst], ctrl_type).find( |&enc| { let range = encinfo.branch_range(enc).expect("Branch with no range"); if !range.contains(offset, dest_offset) { diff --git a/lib/cretonne/src/cursor.rs b/lib/cretonne/src/cursor.rs index 082889a693..865530341d 100644 --- a/lib/cretonne/src/cursor.rs +++ b/lib/cretonne/src/cursor.rs @@ -747,7 +747,7 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut EncCursor<'f> { // XXX Is there a way to describe this error to the user? #[cfg_attr(feature = "cargo-clippy", allow(match_wild_err_arm))] match self.isa.encode( - &self.func.dfg, + &self.func, &self.func.dfg[inst], ctrl_typevar, ) { diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/cretonne/src/ir/dfg.rs index 24f18ff332..c2f07beb25 100644 --- a/lib/cretonne/src/ir/dfg.rs +++ b/lib/cretonne/src/ir/dfg.rs @@ -7,7 +7,7 @@ use ir::extfunc::ExtFuncData; use ir::instructions::{BranchInfo, CallInfo, InstructionData}; use ir::types; use ir::{Ebb, FuncRef, Inst, SigRef, Signature, Type, Value, ValueList, ValueListPool}; -use isa::{Encoding, Legalize, TargetIsa}; +use isa::TargetIsa; use packed_option::ReservedValue; use std::fmt; use std::iter; @@ -660,12 +660,6 @@ impl DataFlowGraph { self.value_type(self.first_result(inst)) } } - - /// Wrapper around `TargetIsa::encode` for encoding an existing instruction - /// in the `DataFlowGraph`. - pub fn encode(&self, inst: Inst, isa: &TargetIsa) -> Result { - isa.encode(&self, &self[inst], self.ctrl_typevar(inst)) - } } /// Allow immutable access to instructions via indexing. diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/cretonne/src/ir/extfunc.rs index 45c44c952a..474b463727 100644 --- a/lib/cretonne/src/ir/extfunc.rs +++ b/lib/cretonne/src/ir/extfunc.rs @@ -327,11 +327,18 @@ pub struct ExtFuncData { pub name: ExternalName, /// Call signature of function. pub signature: SigRef, + /// Will this function be defined nearby, such that it will always be a certain distance away, + /// after linking? If so, references to it can avoid going through a GOT or PLT. Note that + /// symbols meant to be preemptible cannot be considered colocated. + pub colocated: bool, } impl fmt::Display for ExtFuncData { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - write!(f, "{} {}", self.signature, self.name) + if self.colocated { + write!(f, "colocated ")?; + } + write!(f, "{} {}", self.name, self.signature) } } diff --git a/lib/cretonne/src/ir/function.rs b/lib/cretonne/src/ir/function.rs index f7dee1b918..5e70ff3780 100644 --- a/lib/cretonne/src/ir/function.rs +++ b/lib/cretonne/src/ir/function.rs @@ -10,7 +10,7 @@ use ir::{CallConv, DataFlowGraph, ExternalName, Layout, Signature}; use ir::{Ebb, ExtFuncData, FuncRef, GlobalVar, GlobalVarData, Heap, HeapData, JumpTable, JumpTableData, SigRef, StackSlot, StackSlotData}; use ir::{EbbOffsets, InstEncodings, JumpTables, SourceLocs, StackSlots, ValueLocations}; -use isa::{EncInfo, Legalize, TargetIsa}; +use isa::{EncInfo, Legalize, TargetIsa, Encoding}; use std::fmt; use write::write_function; @@ -177,9 +177,15 @@ impl Function { } } - /// Wrapper around `DataFlowGraph::encode` which assigns `inst` the resulting encoding. + /// Wrapper around `encode` which assigns `inst` the resulting encoding. pub fn update_encoding(&mut self, inst: ir::Inst, isa: &TargetIsa) -> Result<(), Legalize> { - self.dfg.encode(inst, isa).map(|e| self.encodings[inst] = e) + self.encode(inst, isa).map(|e| self.encodings[inst] = e) + } + + /// Wrapper around `TargetIsa::encode` for encoding an existing instruction + /// in the `Function`. + pub fn encode(&self, inst: ir::Inst, isa: &TargetIsa) -> Result { + isa.encode(&self, &self.dfg[inst], self.dfg.ctrl_typevar(inst)) } } diff --git a/lib/cretonne/src/ir/globalvar.rs b/lib/cretonne/src/ir/globalvar.rs index 228f691d84..ad75496362 100644 --- a/lib/cretonne/src/ir/globalvar.rs +++ b/lib/cretonne/src/ir/globalvar.rs @@ -33,6 +33,11 @@ pub enum GlobalVarData { Sym { /// The symbolic name. name: ExternalName, + + /// Will this variable be defined nearby, such that it will always be a certain distance + /// away, after linking? If so, references to it can avoid going through a GOT. Note that + /// symbols meant to be preemptible cannot be colocated. + colocated: bool, }, } @@ -40,7 +45,7 @@ impl GlobalVarData { /// Assume that `self` is an `GlobalVarData::Sym` and return its name. pub fn symbol_name(&self) -> &ExternalName { match *self { - GlobalVarData::Sym { ref name } => name, + GlobalVarData::Sym { ref name, .. } => name, _ => panic!("only symbols have names"), } } @@ -51,7 +56,15 @@ impl fmt::Display for GlobalVarData { match *self { GlobalVarData::VmCtx { offset } => write!(f, "vmctx{}", offset), GlobalVarData::Deref { base, offset } => write!(f, "deref({}){}", base, offset), - GlobalVarData::Sym { ref name } => write!(f, "globalsym {}", name), + GlobalVarData::Sym { + ref name, + colocated, + } => { + if colocated { + write!(f, "colocated ")?; + } + write!(f, "globalsym {}", name) + } } } } diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/cretonne/src/isa/arm32/mod.rs index dbef9aaf8a..967c21a1ba 100644 --- a/lib/cretonne/src/isa/arm32/mod.rs +++ b/lib/cretonne/src/isa/arm32/mod.rs @@ -66,14 +66,14 @@ impl TargetIsa for Isa { fn legal_encodings<'a>( &'a self, - dfg: &'a ir::DataFlowGraph, + func: &'a ir::Function, inst: &'a ir::InstructionData, ctrl_typevar: ir::Type, ) -> Encodings<'a> { lookup_enclist( ctrl_typevar, inst, - dfg, + func, self.cpumode, &enc_tables::LEVEL2[..], &enc_tables::ENCLISTS[..], diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/cretonne/src/isa/arm64/mod.rs index eeddec74fd..27c95f51b3 100644 --- a/lib/cretonne/src/isa/arm64/mod.rs +++ b/lib/cretonne/src/isa/arm64/mod.rs @@ -59,14 +59,14 @@ impl TargetIsa for Isa { fn legal_encodings<'a>( &'a self, - dfg: &'a ir::DataFlowGraph, + func: &'a ir::Function, inst: &'a ir::InstructionData, ctrl_typevar: ir::Type, ) -> Encodings<'a> { lookup_enclist( ctrl_typevar, inst, - dfg, + func, &enc_tables::LEVEL1_A64[..], &enc_tables::LEVEL2[..], &enc_tables::ENCLISTS[..], diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/cretonne/src/isa/enc_tables.rs index c987606fa6..b4c9bd86ba 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/cretonne/src/isa/enc_tables.rs @@ -4,7 +4,7 @@ //! `lib/cretonne/meta/gen_encoding.py`. use constant_hash::{probe, Table}; -use ir::{DataFlowGraph, InstructionData, Opcode, Type}; +use ir::{Function, InstructionData, Opcode, Type}; use isa::{Encoding, Legalize}; use settings::PredicateView; use std::ops::Range; @@ -20,7 +20,7 @@ pub type RecipePredicate = Option bool>; /// /// This is a predicate function that needs to be tested in addition to the recipe predicate. It /// can't depend on ISA settings. -pub type InstPredicate = fn(&DataFlowGraph, &InstructionData) -> bool; +pub type InstPredicate = fn(&Function, &InstructionData) -> bool; /// Legalization action to perform when no encoding can be found for an instruction. /// @@ -106,7 +106,7 @@ impl + Copy> Table for [Level2Entry] { pub fn lookup_enclist<'a, OffT1, OffT2>( ctrl_typevar: Type, inst: &'a InstructionData, - dfg: &'a DataFlowGraph, + func: &'a Function, level1_table: &'static [Level1Entry], level2_table: &'static [Level2Entry], enclist: &'static [EncListEntry], @@ -150,7 +150,7 @@ where offset, legalize, inst, - dfg, + func, enclist, legalize_actions, recipe_preds, @@ -177,7 +177,7 @@ pub struct Encodings<'a> { // Legalization code to use of no encoding is found. legalize: LegalizeCode, inst: &'a InstructionData, - dfg: &'a DataFlowGraph, + func: &'a Function, enclist: &'static [EncListEntry], legalize_actions: &'static [Legalize], recipe_preds: &'static [RecipePredicate], @@ -195,7 +195,7 @@ impl<'a> Encodings<'a> { offset: usize, legalize: LegalizeCode, inst: &'a InstructionData, - dfg: &'a DataFlowGraph, + func: &'a Function, enclist: &'static [EncListEntry], legalize_actions: &'static [Legalize], recipe_preds: &'static [RecipePredicate], @@ -205,7 +205,7 @@ impl<'a> Encodings<'a> { Encodings { offset, inst, - dfg, + func, legalize, isa_preds, recipe_preds, @@ -236,7 +236,7 @@ impl<'a> Encodings<'a> { /// Check an instruction or isa predicate. fn check_pred(&self, pred: usize) -> bool { if let Some(&p) = self.inst_preds.get(pred) { - p(self.dfg, self.inst) + p(self.func, self.inst) } else { let pred = pred - self.inst_preds.len(); self.isa_preds.test(pred) diff --git a/lib/cretonne/src/isa/mod.rs b/lib/cretonne/src/isa/mod.rs index 5ee3359d50..d851d8d9a0 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/cretonne/src/isa/mod.rs @@ -167,7 +167,7 @@ pub trait TargetIsa: fmt::Display { /// Returns an iterartor over legal encodings for the instruction. fn legal_encodings<'a>( &'a self, - dfg: &'a ir::DataFlowGraph, + func: &'a ir::Function, inst: &'a ir::InstructionData, ctrl_typevar: ir::Type, ) -> Encodings<'a>; @@ -180,11 +180,11 @@ pub trait TargetIsa: fmt::Display { /// This is also the main entry point for determining if an instruction is legal. fn encode( &self, - dfg: &ir::DataFlowGraph, + func: &ir::Function, inst: &ir::InstructionData, ctrl_typevar: ir::Type, ) -> Result { - let mut iter = self.legal_encodings(dfg, inst, ctrl_typevar); + let mut iter = self.legal_encodings(func, inst, ctrl_typevar); iter.next().ok_or_else(|| iter.legalize()) } diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/cretonne/src/isa/riscv/mod.rs index c81d3baceb..af993aad0d 100644 --- a/lib/cretonne/src/isa/riscv/mod.rs +++ b/lib/cretonne/src/isa/riscv/mod.rs @@ -66,14 +66,14 @@ impl TargetIsa for Isa { fn legal_encodings<'a>( &'a self, - dfg: &'a ir::DataFlowGraph, + func: &'a ir::Function, inst: &'a ir::InstructionData, ctrl_typevar: ir::Type, ) -> Encodings<'a> { lookup_enclist( ctrl_typevar, inst, - dfg, + func, self.cpumode, &enc_tables::LEVEL2[..], &enc_tables::ENCLISTS[..], @@ -113,7 +113,7 @@ impl TargetIsa for Isa { #[cfg(test)] mod tests { - use ir::{DataFlowGraph, InstructionData, Opcode}; + use ir::{Function, InstructionData, Opcode}; use ir::{immediates, types}; use isa; use settings::{self, Configurable}; @@ -133,10 +133,10 @@ mod tests { let shared_flags = settings::Flags::new(&shared_builder); let isa = isa::lookup("riscv").unwrap().finish(shared_flags); - let mut dfg = DataFlowGraph::new(); - let ebb = dfg.make_ebb(); - let arg64 = dfg.append_ebb_param(ebb, types::I64); - let arg32 = dfg.append_ebb_param(ebb, types::I32); + let mut func = Function::new(); + let ebb = func.dfg.make_ebb(); + let arg64 = func.dfg.append_ebb_param(ebb, types::I64); + let arg32 = func.dfg.append_ebb_param(ebb, types::I32); // Try to encode iadd_imm.i64 v1, -10. let inst64 = InstructionData::BinaryImm { @@ -147,7 +147,7 @@ mod tests { // ADDI is I/0b00100 assert_eq!( - encstr(&*isa, isa.encode(&dfg, &inst64, types::I64)), + encstr(&*isa, isa.encode(&func, &inst64, types::I64)), "Ii#04" ); @@ -159,7 +159,7 @@ mod tests { }; // Immediate is out of range for ADDI. - assert!(isa.encode(&dfg, &inst64_large, types::I64).is_err()); + assert!(isa.encode(&func, &inst64_large, types::I64).is_err()); // Create an iadd_imm.i32 which is encodable in RV64. let inst32 = InstructionData::BinaryImm { @@ -170,7 +170,7 @@ mod tests { // ADDIW is I/0b00110 assert_eq!( - encstr(&*isa, isa.encode(&dfg, &inst32, types::I32)), + encstr(&*isa, isa.encode(&func, &inst32, types::I32)), "Ii#06" ); } @@ -183,10 +183,10 @@ mod tests { let shared_flags = settings::Flags::new(&shared_builder); let isa = isa::lookup("riscv").unwrap().finish(shared_flags); - let mut dfg = DataFlowGraph::new(); - let ebb = dfg.make_ebb(); - let arg64 = dfg.append_ebb_param(ebb, types::I64); - let arg32 = dfg.append_ebb_param(ebb, types::I32); + let mut func = Function::new(); + let ebb = func.dfg.make_ebb(); + let arg64 = func.dfg.append_ebb_param(ebb, types::I64); + let arg32 = func.dfg.append_ebb_param(ebb, types::I32); // Try to encode iadd_imm.i64 v1, -10. let inst64 = InstructionData::BinaryImm { @@ -196,7 +196,7 @@ mod tests { }; // In 32-bit mode, an i64 bit add should be narrowed. - assert!(isa.encode(&dfg, &inst64, types::I64).is_err()); + assert!(isa.encode(&func, &inst64, types::I64).is_err()); // Try to encode iadd_imm.i64 v1, -10000. let inst64_large = InstructionData::BinaryImm { @@ -206,7 +206,7 @@ mod tests { }; // In 32-bit mode, an i64 bit add should be narrowed. - assert!(isa.encode(&dfg, &inst64_large, types::I64).is_err()); + assert!(isa.encode(&func, &inst64_large, types::I64).is_err()); // Create an iadd_imm.i32 which is encodable in RV32. let inst32 = InstructionData::BinaryImm { @@ -217,7 +217,7 @@ mod tests { // ADDI is I/0b00100 assert_eq!( - encstr(&*isa, isa.encode(&dfg, &inst32, types::I32)), + encstr(&*isa, isa.encode(&func, &inst32, types::I32)), "Ii#04" ); @@ -227,7 +227,7 @@ mod tests { args: [arg32, arg32], }; - assert!(isa.encode(&dfg, &mul32, types::I32).is_err()); + assert!(isa.encode(&func, &mul32, types::I32).is_err()); } #[test] @@ -243,16 +243,19 @@ mod tests { let isa = isa_builder.finish(shared_flags); - let mut dfg = DataFlowGraph::new(); - let ebb = dfg.make_ebb(); - let arg32 = dfg.append_ebb_param(ebb, types::I32); + let mut func = Function::new(); + let ebb = func.dfg.make_ebb(); + let arg32 = func.dfg.append_ebb_param(ebb, types::I32); // Create an imul.i32 which is encodable in RV32M. let mul32 = InstructionData::Binary { opcode: Opcode::Imul, args: [arg32, arg32], }; - assert_eq!(encstr(&*isa, isa.encode(&dfg, &mul32, types::I32)), "R#10c"); + assert_eq!( + encstr(&*isa, isa.encode(&func, &mul32, types::I32)), + "R#10c" + ); } } diff --git a/lib/cretonne/src/isa/x86/mod.rs b/lib/cretonne/src/isa/x86/mod.rs index a272420421..6bbde318a5 100644 --- a/lib/cretonne/src/isa/x86/mod.rs +++ b/lib/cretonne/src/isa/x86/mod.rs @@ -72,14 +72,14 @@ impl TargetIsa for Isa { fn legal_encodings<'a>( &'a self, - dfg: &'a ir::DataFlowGraph, + func: &'a ir::Function, inst: &'a ir::InstructionData, ctrl_typevar: ir::Type, ) -> Encodings<'a> { lookup_enclist( ctrl_typevar, inst, - dfg, + func, self.cpumode, &enc_tables::LEVEL2[..], &enc_tables::ENCLISTS[..], diff --git a/lib/cretonne/src/legalizer/libcall.rs b/lib/cretonne/src/legalizer/libcall.rs index 2bb0f1292c..dafc353903 100644 --- a/lib/cretonne/src/legalizer/libcall.rs +++ b/lib/cretonne/src/legalizer/libcall.rs @@ -54,8 +54,10 @@ fn make_funcref(libcall: ir::LibCall, inst: ir::Inst, func: &mut ir::Function) - } let sigref = func.import_signature(sig); + // TODO: Can libcalls be colocated in some circumstances? func.import_function(ir::ExtFuncData { name: ir::ExternalName::LibCall(libcall), signature: sigref, + colocated: false, }) } diff --git a/lib/cretonne/src/predicates.rs b/lib/cretonne/src/predicates.rs index 618369291d..333ca8eb31 100644 --- a/lib/cretonne/src/predicates.rs +++ b/lib/cretonne/src/predicates.rs @@ -9,6 +9,8 @@ //! Some of these predicates may be unused in certain ISA configurations, so we suppress the //! dead code warning. +use ir; + /// Check that `x` is the same as `y`. #[allow(dead_code)] pub fn is_equal + Copy>(x: T, y: O) -> bool { @@ -31,6 +33,19 @@ pub fn is_unsigned_int>(x: T, wd: u8, sc: u8) -> bool { u == (u & m) } +#[allow(dead_code)] +pub fn is_colocated_func(func_ref: ir::FuncRef, func: &ir::Function) -> bool { + func.dfg.ext_funcs[func_ref].colocated +} + +#[allow(dead_code)] +pub fn is_colocated_data(global_var: ir::GlobalVar, func: &ir::Function) -> bool { + match func.global_vars[global_var] { + ir::GlobalVarData::Sym { colocated, .. } => colocated, + _ => panic!("is_colocated_data only makes sense for data with symbolic addresses"), + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/cretonne/src/verifier/mod.rs index 4000e5883f..abda326c03 100644 --- a/lib/cretonne/src/verifier/mod.rs +++ b/lib/cretonne/src/verifier/mod.rs @@ -1025,7 +1025,7 @@ impl<'a> Verifier<'a> { let encoding = self.func.encodings[inst]; if encoding.is_legal() { let mut encodings = isa.legal_encodings( - &self.func.dfg, + &self.func, &self.func.dfg[inst], self.func.dfg.ctrl_typevar(inst), ).peekable(); @@ -1045,7 +1045,7 @@ impl<'a> Verifier<'a> { let mut multiple_encodings = false; for enc in isa.legal_encodings( - &self.func.dfg, + &self.func, &self.func.dfg[inst], self.func.dfg.ctrl_typevar(inst), ) @@ -1099,7 +1099,7 @@ impl<'a> Verifier<'a> { if let Some(text) = needs_enc { // This instruction needs an encoding, so generate an error. // Provide the ISA default encoding as a hint. - match self.func.dfg.encode(inst, isa) { + match self.func.encode(inst, isa) { Ok(enc) => { return err!( inst, diff --git a/lib/cretonne/src/write.rs b/lib/cretonne/src/write.rs index 2ed8da87e9..22d78bdb82 100644 --- a/lib/cretonne/src/write.rs +++ b/lib/cretonne/src/write.rs @@ -16,6 +16,7 @@ pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) - let regs = isa.map(TargetIsa::register_info); let regs = regs.as_ref(); + write!(w, "function ")?; write_spec(w, func, regs)?; writeln!(w, " {{")?; let mut any = write_preamble(w, func, regs)?; @@ -34,7 +35,7 @@ pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) - // Function spec. fn write_spec(w: &mut Write, func: &Function, regs: Option<&RegInfo>) -> Result { - write!(w, "function {}{}", func.name, func.signature.display(regs)) + write!(w, "{}{}", func.name, func.signature.display(regs)) } fn write_preamble( diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index 775e014600..ce2e97e888 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -133,7 +133,7 @@ impl SubTest for TestBinEmit { // constraints. if let Some(enc) = { let mut legal_encodings = isa.legal_encodings( - &func.dfg, + &func, &func.dfg[inst], func.dfg.ctrl_typevar(inst), ).filter(|e| { @@ -251,7 +251,7 @@ impl SubTest for TestBinEmit { // Do any encodings exist? let encodings = isa.legal_encodings( - &func.dfg, + &func, &func.dfg[inst], func.dfg.ctrl_typevar(inst), ).map(|e| encinfo.display(e)) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index f23f644a0e..66a66ac18c 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -144,6 +144,7 @@ impl<'a> Context<'a> { while self.function.global_vars.next_key().index() <= gv.index() { self.function.create_global_var(GlobalVarData::Sym { name: ExternalName::testcase(""), + colocated: false, }); } self.function.global_vars[gv] = data; @@ -208,6 +209,7 @@ impl<'a> Context<'a> { self.function.import_function(ExtFuncData { name: ExternalName::testcase(""), signature: SigRef::reserved_value(), + colocated: false, }); } self.function.dfg.ext_funcs[fn_] = data; @@ -760,7 +762,7 @@ impl<'a> Parser<'a> { // Parse a whole function definition. // - // function ::= * function-spec "{" preamble function-body "}" + // function ::= * "function" name signature "{" preamble function-body "}" // fn parse_function( &mut self, @@ -772,10 +774,19 @@ impl<'a> Parser<'a> { debug_assert!(self.comments.is_empty()); self.start_gathering_comments(); - let (location, name, sig) = self.parse_function_spec(unique_isa)?; + self.match_identifier("function", "expected 'function'")?; + + let location = self.loc; + + // function ::= "function" * name signature "{" preamble function-body "}" + let name = self.parse_external_name()?; + + // function ::= "function" name * signature "{" preamble function-body "}" + let sig = self.parse_signature(unique_isa)?; + let mut ctx = Context::new(Function::with_name_signature(name, sig), unique_isa); - // function ::= function-spec * "{" preamble function-body "}" + // function ::= "function" name signature * "{" preamble function-body "}" self.match_token( Token::LBrace, "expected '{' before function body", @@ -784,11 +795,11 @@ impl<'a> Parser<'a> { self.token(); self.claim_gathered_comments(AnyEntity::Function); - // function ::= function-spec "{" * preamble function-body "}" + // function ::= "function" name signature "{" * preamble function-body "}" self.parse_preamble(&mut ctx)?; - // function ::= function-spec "{" preamble * function-body "}" + // function ::= "function" name signature "{" preamble * function-body "}" self.parse_function_body(&mut ctx)?; - // function ::= function-spec "{" preamble function-body * "}" + // function ::= "function" name signature "{" preamble function-body * "}" self.match_token( Token::RBrace, "expected '}' after function body", @@ -808,29 +819,9 @@ impl<'a> Parser<'a> { Ok((ctx.function, details)) } - // Parse a function spec. - // - // function-spec ::= * "function" name signature - // - fn parse_function_spec( - &mut self, - unique_isa: Option<&TargetIsa>, - ) -> Result<(Location, ExternalName, Signature)> { - self.match_identifier("function", "expected 'function'")?; - let location = self.loc; - - // function-spec ::= "function" * name signature - let name = self.parse_external_name()?; - - // function-spec ::= "function" name * signature - let sig = self.parse_signature(unique_isa)?; - - Ok((location, name, sig)) - } - // Parse an external name. // - // For example, in a function spec, the parser would be in this state: + // For example, in a function decl, the parser would be in this state: // // function ::= "function" * name signature { ... } // @@ -1095,7 +1086,7 @@ impl<'a> Parser<'a> { // global-var-decl ::= * GlobalVar(gv) "=" global-var-desc // global-var-desc ::= "vmctx" offset32 // | "deref" "(" GlobalVar(base) ")" offset32 - // | "globalsym" name + // | globalsym ["colocated"] name // fn parse_global_var_decl(&mut self) -> Result<(GlobalVar, GlobalVarData)> { let gv = self.match_gv("expected global variable number: gv«n»")?; @@ -1124,8 +1115,9 @@ impl<'a> Parser<'a> { GlobalVarData::Deref { base, offset } } "globalsym" => { + let colocated = self.optional(Token::Identifier("colocated")); let name = self.parse_external_name()?; - GlobalVarData::Sym { name } + GlobalVarData::Sym { name, colocated } } other => return err!(self.loc, "Unknown global variable kind '{}'", other), }; @@ -1237,8 +1229,8 @@ impl<'a> Parser<'a> { // // Two variants: // - // function-decl ::= FuncRef(fnref) "=" function-spec - // FuncRef(fnref) "=" SigRef(sig) name + // function-decl ::= FuncRef(fnref) "=" ["colocated"]" name function-decl-sig + // function-decl-sig ::= SigRef(sig) | signature // // The first variant allocates a new signature reference. The second references an existing // signature which must be declared first. @@ -1250,9 +1242,19 @@ impl<'a> Parser<'a> { "expected '=' in function decl", )?; + let loc = self.loc; + + // function-decl ::= FuncRef(fnref) "=" * ["colocated"] name function-decl-sig + let colocated = self.optional(Token::Identifier("colocated")); + + // function-decl ::= FuncRef(fnref) "=" ["colocated"] * name function-decl-sig + let name = self.parse_external_name()?; + + // function-decl ::= FuncRef(fnref) "=" ["colocated"] name * function-decl-sig let data = match self.token() { - Some(Token::Identifier("function")) => { - let (loc, name, sig) = self.parse_function_spec(ctx.unique_isa)?; + Some(Token::LPar) => { + // function-decl ::= FuncRef(fnref) "=" ["colocated"] name * signature + let sig = self.parse_signature(ctx.unique_isa)?; let sigref = ctx.function.import_signature(sig); ctx.map.def_entity(sigref.into(), &loc).expect( "duplicate SigRef entities created", @@ -1260,6 +1262,7 @@ impl<'a> Parser<'a> { ExtFuncData { name, signature: sigref, + colocated, } } Some(Token::SigRef(sig_src)) => { @@ -1271,10 +1274,10 @@ impl<'a> Parser<'a> { }; ctx.check_sig(sig, &self.loc)?; self.consume(); - let name = self.parse_external_name()?; ExtFuncData { name, signature: sig, + colocated, } } _ => return err!(self.loc, "expected 'function' or sig«n» in function decl"), diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index a71dddda3d..2bac8ed783 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -182,7 +182,11 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ // And maybe attempt some signature de-duplication. let signature = func.import_signature(self.vmctx_sig(sigidx)); let name = get_func_name(index); - func.import_function(ir::ExtFuncData { name, signature }) + func.import_function(ir::ExtFuncData { + name, + signature, + colocated: false, + }) } fn translate_call_indirect( From a4523cf0b8143b37f7498e852f9ca10fa3d2746a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 16 Apr 2018 14:48:01 -0700 Subject: [PATCH 1694/3084] Bump version to 0.4.3 --- cranelift/Cargo.toml | 14 +++++++------- cranelift/publish-all.sh | 2 +- lib/cretonne/Cargo.toml | 2 +- lib/entity/Cargo.toml | 2 +- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/wasm/Cargo.toml | 6 +++--- 9 files changed, 22 insertions(+), 22 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 73fab18586..c2a5924451 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.4.2" +version = "0.4.3" description = "Binaries for testing the Cretonne library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,12 +13,12 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne = { path = "lib/cretonne", version = "0.4.2" } -cretonne-reader = { path = "lib/reader", version = "0.4.2" } -cretonne-frontend = { path = "lib/frontend", version = "0.4.2" } -cretonne-wasm = { path = "lib/wasm", version = "0.4.2" } -cretonne-native = { path = "lib/native", version = "0.4.2" } -cretonne-filetests = { path = "lib/filetests", version = "0.4.2" } +cretonne = { path = "lib/cretonne", version = "0.4.3" } +cretonne-reader = { path = "lib/reader", version = "0.4.3" } +cretonne-frontend = { path = "lib/frontend", version = "0.4.3" } +cretonne-wasm = { path = "lib/wasm", version = "0.4.3" } +cretonne-native = { path = "lib/native", version = "0.4.3" } +cretonne-filetests = { path = "lib/filetests", version = "0.4.3" } filecheck = "0.2.1" docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 4ac7e2bcff..c4ef885927 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ cd $(dirname "$0") topdir="$(pwd)" # All the cretonne-* crates have the same version number -version="0.4.2" +version="0.4.3" # Update all of the Cargo.toml files. # diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 4540a783a6..eaa91faccc 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.4.2" +version = "0.4.3" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index c86962c809..eef8abb4f8 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-entity" -version = "0.4.2" +version = "0.4.3" description = "Data structures using entity references as mapping keys" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 48740ece84..97cd62acde 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.4.2" +version = "0.4.3" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "http://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -12,7 +12,7 @@ publish = false name = "cton_filetests" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.2" } -cretonne-reader = { path = "../reader", version = "0.4.2" } +cretonne = { path = "../cretonne", version = "0.4.3" } +cretonne-reader = { path = "../reader", version = "0.4.3" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index f63f4e7daf..5c6a6d9832 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.4.2" +version = "0.4.3" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,7 +12,7 @@ readme = "README.md" name = "cton_frontend" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.2" } +cretonne = { path = "../cretonne", version = "0.4.3" } [badges] maintenance = { status = "experimental" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 3d959a700e..9758eda9db 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.4.2" +version = "0.4.3" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -11,7 +11,7 @@ readme = "README.md" name = "cton_native" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.2" } +cretonne = { path = "../cretonne", version = "0.4.3" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index a8c3c1ffd1..7ec20d899c 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.4.2" +version = "0.4.3" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,7 +12,7 @@ readme = "README.md" name = "cton_reader" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.2" } +cretonne = { path = "../cretonne", version = "0.4.3" } [badges] maintenance = { status = "experimental" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index c8b9c9f6f5..1733219e07 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.4.2" +version = "0.4.3" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/Cretonne/cretonne" @@ -13,8 +13,8 @@ name = "cton_wasm" [dependencies] wasmparser = "0.15.1" -cretonne = { path = "../cretonne", version = "0.4.2" } -cretonne-frontend = { path = "../frontend", version = "0.4.2" } +cretonne = { path = "../cretonne", version = "0.4.3" } +cretonne-frontend = { path = "../frontend", version = "0.4.3" } [dev-dependencies] tempdir = "0.3.5" From 5977f2e45e0d3f22f1891dc74d6ff115e170bbc7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 16 Apr 2018 14:50:41 -0700 Subject: [PATCH 1695/3084] Update packaging metadata for cretonne-entity. --- cranelift/publish-all.sh | 2 +- lib/cretonne/Cargo.toml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index c4ef885927..88eab7841d 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -27,7 +27,7 @@ cargo update echo git commit -a -m "\"Bump version to $version"\" echo git push -for crate in cretonne frontend native reader wasm; do +for crate in entity cretonne frontend native reader wasm ; do echo cargo publish --manifest-path "lib/$crate/Cargo.toml" done echo diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index eaa91faccc..101e2423bb 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -14,7 +14,7 @@ build = "build.rs" name = "cretonne" [dependencies] -cretonne-entity = { path = "../entity" } +cretonne-entity = { path = "../entity", version = "0.4.3" } # It is a goal of the cretonne crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be From 56f11e76b4e22fb130ece19e703e7d81a90eb9e4 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 16 Apr 2018 16:27:27 -0700 Subject: [PATCH 1696/3084] Use PC-relative encodings for colocated functions on non-PIC. Colocated functions are expected to be defined within the PC-relative immediate range on x86-64, so allow this addressing for non-PIC as well as PIC. --- cranelift/filetests/isa/x86/binary64-pic.cton | 34 ++++++++++++---- cranelift/filetests/isa/x86/binary64.cton | 40 ++++++++++++++----- lib/cretonne/meta/isa/x86/encodings.py | 16 +++++--- 3 files changed, 67 insertions(+), 23 deletions(-) diff --git a/cranelift/filetests/isa/x86/binary64-pic.cton b/cranelift/filetests/isa/x86/binary64-pic.cton index f3b838795a..cc592e0188 100644 --- a/cranelift/filetests/isa/x86/binary64-pic.cton +++ b/cranelift/filetests/isa/x86/binary64-pic.cton @@ -28,18 +28,17 @@ function %I64() { ebb0: - ; asm: call foo@PLT - call fn0() ; bin: e8 PLTRel4(%foo-4) 00000000 + ; Colocated functions. ; asm: call foo call fn1() ; bin: e8 PCRel4(%bar-4) 00000000 - ; asm: mov 0x0(%rip), %rax - [-,%rax] v0 = func_addr.i64 fn0 ; bin: 48 8b 05 GOTPCRel4(%foo-4) 00000000 - ; asm: mov 0x0(%rip), %rsi - [-,%rsi] v1 = func_addr.i64 fn0 ; bin: 48 8b 35 GOTPCRel4(%foo-4) 00000000 - ; asm: mov 0x0(%rip), %r10 - [-,%r10] v2 = func_addr.i64 fn0 ; bin: 4c 8b 15 GOTPCRel4(%foo-4) 00000000 + ; asm: lea 0x0(%rip), %rax + [-,%rax] v0 = func_addr.i64 fn1 ; bin: 48 8d 05 PCRel4(%bar-4) 00000000 + ; asm: lea 0x0(%rip), %rsi + [-,%rsi] v1 = func_addr.i64 fn1 ; bin: 48 8d 35 PCRel4(%bar-4) 00000000 + ; asm: lea 0x0(%rip), %r10 + [-,%r10] v2 = func_addr.i64 fn1 ; bin: 4c 8d 15 PCRel4(%bar-4) 00000000 ; asm: call *%rax call_indirect sig0, v0() ; bin: ff d0 @@ -48,6 +47,25 @@ ebb0: ; asm: call *%r10 call_indirect sig0, v2() ; bin: 41 ff d2 + ; Non-colocated functions. + + ; asm: call foo@PLT + call fn0() ; bin: e8 PLTRel4(%foo-4) 00000000 + + ; asm: mov 0x0(%rip), %rax + [-,%rax] v100 = func_addr.i64 fn0 ; bin: 48 8b 05 GOTPCRel4(%foo-4) 00000000 + ; asm: mov 0x0(%rip), %rsi + [-,%rsi] v101 = func_addr.i64 fn0 ; bin: 48 8b 35 GOTPCRel4(%foo-4) 00000000 + ; asm: mov 0x0(%rip), %r10 + [-,%r10] v102 = func_addr.i64 fn0 ; bin: 4c 8b 15 GOTPCRel4(%foo-4) 00000000 + + ; asm: call *%rax + call_indirect sig0, v100() ; bin: ff d0 + ; asm: call *%rsi + call_indirect sig0, v101() ; bin: ff d6 + ; asm: call *%r10 + call_indirect sig0, v102() ; bin: 41 ff d2 + ; asm: mov 0x0(%rip), %rcx [-,%rcx] v3 = globalsym_addr.i64 gv0 ; bin: 48 8b 0d GOTPCRel4(%some_gv-4) 00000000 ; asm: mov 0x0(%rip), %rsi diff --git a/cranelift/filetests/isa/x86/binary64.cton b/cranelift/filetests/isa/x86/binary64.cton index 6ecc086231..faf4fe99ac 100644 --- a/cranelift/filetests/isa/x86/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.cton @@ -13,6 +13,7 @@ isa x86 haswell function %I64() { sig0 = () fn0 = %foo() + fn1 = colocated %bar() gv0 = globalsym %some_gv @@ -473,12 +474,17 @@ ebb0: ; asm: movzbq %dl, %rsi [-,%rsi] v351 = bint.i64 v301 ; bin: 0f b6 f2 - ; asm: movabsq $0, %rcx - [-,%rcx] v400 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(%foo) 0000000000000000 - ; asm: movabsq $0, %rsi - [-,%rsi] v401 = func_addr.i64 fn0 ; bin: 48 be Abs8(%foo) 0000000000000000 - ; asm: movabsq $0, %r10 - [-,%r10] v402 = func_addr.i64 fn0 ; bin: 49 ba Abs8(%foo) 0000000000000000 + ; Colocated functions. + + ; asm: call bar + ; call fn1() ; bin: e8 PCRel4(%bar-4) 00000000 + + ; asm: lea 0x0(%rip), %rcx + [-,%rcx] v400 = func_addr.i64 fn1 ; bin: 48 8d 0d PCRel4(%bar-4) 00000000 + ; asm: lea 0x0(%rip), %rsi + [-,%rsi] v401 = func_addr.i64 fn1 ; bin: 48 8d 35 PCRel4(%bar-4) 00000000 + ; asm: lea 0x0(%rip), %r10 + [-,%r10] v402 = func_addr.i64 fn1 ; bin: 4c 8d 15 PCRel4(%bar-4) 00000000 ; asm: call *%rcx call_indirect sig0, v400() ; bin: ff d1 @@ -487,6 +493,22 @@ ebb0: ; asm: call *%r10 call_indirect sig0, v402() ; bin: 41 ff d2 + ; Non-colocated functions. Note that there is no non-colocated non-PIC call. + + ; asm: movabsq $0, %rcx + [-,%rcx] v410 = func_addr.i64 fn0 ; bin: 48 b9 Abs8(%foo) 0000000000000000 + ; asm: movabsq $0, %rsi + [-,%rsi] v411 = func_addr.i64 fn0 ; bin: 48 be Abs8(%foo) 0000000000000000 + ; asm: movabsq $0, %r10 + [-,%r10] v412 = func_addr.i64 fn0 ; bin: 49 ba Abs8(%foo) 0000000000000000 + + ; asm: call *%rcx + call_indirect sig0, v410() ; bin: ff d1 + ; asm: call *%rsi + call_indirect sig0, v411() ; bin: ff d6 + ; asm: call *%r10 + call_indirect sig0, v412() ; bin: 41 ff d2 + ; asm: movabsq $-1, %rcx [-,%rcx] v450 = globalsym_addr.i64 gv0 ; bin: 48 b9 Abs8(%some_gv) 0000000000000000 ; asm: movabsq $-1, %rsi @@ -548,9 +570,9 @@ ebb0: [-,%rsi] v517 = sshr_imm v2, 32 ; bin: 48 c1 fe 20 ; asm: sarq $33, %r8 [-,%r8] v518 = sshr_imm v4, 33 ; bin: 49 c1 f8 21 - ; asm: shrl $62, %rsi + ; asm: shrq $62, %rsi [-,%rsi] v519 = ushr_imm v2, 62 ; bin: 48 c1 ee 3e - ; asm: shrl $63, %r8 + ; asm: shrq $63, %r8 [-,%r8] v520 = ushr_imm v4, 63 ; bin: 49 c1 e8 3f @@ -1040,7 +1062,7 @@ ebb0: ; asm: setl %bl [-,%rbx] v320 = icmp_imm slt v1, 37 ; bin: 83 f9 25 0f 9c c3 - ; asm: cmpq $100000, %ecx + ; asm: cmpl $100000, %ecx ; asm: setl %bl [-,%rbx] v321 = icmp_imm slt v1, 100000 ; bin: 81 f9 000186a0 0f 9c c3 diff --git a/lib/cretonne/meta/isa/x86/encodings.py b/lib/cretonne/meta/isa/x86/encodings.py index c120fe1a58..5ab6636a44 100644 --- a/lib/cretonne/meta/isa/x86/encodings.py +++ b/lib/cretonne/meta/isa/x86/encodings.py @@ -305,10 +305,12 @@ X86_32.enc(base.func_addr.i32, *r.allones_fnaddr4(0xb8), X86_64.enc(base.func_addr.i64, *r.allones_fnaddr8.rex(0xb8, w=1), isap=And(allones_funcaddrs, Not(is_pic))) -# PIC +# 64-bit, colocated, both PIC and non-PIC. Use the lea instruction's +# pc-relative field. X86_64.enc(base.func_addr.i64, *r.pcrel_fnaddr8.rex(0x8d, w=1), - isap=is_pic, instp=IsColocatedFunc(FuncAddr.func_ref)) + +# 64-bit, non-colocated, PIC. X86_64.enc(base.func_addr.i64, *r.got_fnaddr8.rex(0x8b, w=1), isap=is_pic) @@ -338,12 +340,14 @@ X86_64.enc(base.globalsym_addr.i64, *r.got_gvaddr8.rex(0x8b, w=1), # 32-bit, both PIC and non-PIC. X86_32.enc(base.call, *r.call_id(0xe8)) -# 64-bit, PIC, colocated and non-colocated. There is no 64-bit non-PIC, since -# non-PIC is currently using the large model, which requires calls be lowered -# to func_addr+call_indirect. +# 64-bit, colocated, both PIC and non-PIC. Use the call instruction's +# pc-relative field. X86_64.enc(base.call, *r.call_id(0xe8), - isap=is_pic, instp=IsColocatedFunc(Call.func_ref)) + +# 64-bit, non-colocated, PIC. There is no 64-bit non-colocated non-PIC version, +# since non-PIC is currently using the large model, which requires calls be +# lowered to func_addr+call_indirect. X86_64.enc(base.call, *r.call_plt_id(0xe8), isap=is_pic) X86_32.enc(base.call_indirect.i32, *r.call_r(0xff, rrr=2)) From f552c8768e96e9ea2b687660ae5f09654fd6b322 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 16 Apr 2018 16:36:52 -0700 Subject: [PATCH 1697/3084] Bump version to 0.4.4 --- cranelift/Cargo.toml | 14 +++++++------- cranelift/publish-all.sh | 2 +- lib/cretonne/Cargo.toml | 4 ++-- lib/entity/Cargo.toml | 2 +- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/wasm/Cargo.toml | 6 +++--- 9 files changed, 23 insertions(+), 23 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index c2a5924451..c5ab163829 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.4.3" +version = "0.4.4" description = "Binaries for testing the Cretonne library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,12 +13,12 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne = { path = "lib/cretonne", version = "0.4.3" } -cretonne-reader = { path = "lib/reader", version = "0.4.3" } -cretonne-frontend = { path = "lib/frontend", version = "0.4.3" } -cretonne-wasm = { path = "lib/wasm", version = "0.4.3" } -cretonne-native = { path = "lib/native", version = "0.4.3" } -cretonne-filetests = { path = "lib/filetests", version = "0.4.3" } +cretonne = { path = "lib/cretonne", version = "0.4.4" } +cretonne-reader = { path = "lib/reader", version = "0.4.4" } +cretonne-frontend = { path = "lib/frontend", version = "0.4.4" } +cretonne-wasm = { path = "lib/wasm", version = "0.4.4" } +cretonne-native = { path = "lib/native", version = "0.4.4" } +cretonne-filetests = { path = "lib/filetests", version = "0.4.4" } filecheck = "0.2.1" docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 88eab7841d..c98d0da07a 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ cd $(dirname "$0") topdir="$(pwd)" # All the cretonne-* crates have the same version number -version="0.4.3" +version="0.4.4" # Update all of the Cargo.toml files. # diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 101e2423bb..1b93158451 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.4.3" +version = "0.4.4" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -14,7 +14,7 @@ build = "build.rs" name = "cretonne" [dependencies] -cretonne-entity = { path = "../entity", version = "0.4.3" } +cretonne-entity = { path = "../entity", version = "0.4.4" } # It is a goal of the cretonne crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index eef8abb4f8..726f85e46b 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-entity" -version = "0.4.3" +version = "0.4.4" description = "Data structures using entity references as mapping keys" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 97cd62acde..0c7b7d5739 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.4.3" +version = "0.4.4" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "http://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -12,7 +12,7 @@ publish = false name = "cton_filetests" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.3" } -cretonne-reader = { path = "../reader", version = "0.4.3" } +cretonne = { path = "../cretonne", version = "0.4.4" } +cretonne-reader = { path = "../reader", version = "0.4.4" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 5c6a6d9832..3f35168f75 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.4.3" +version = "0.4.4" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,7 +12,7 @@ readme = "README.md" name = "cton_frontend" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.3" } +cretonne = { path = "../cretonne", version = "0.4.4" } [badges] maintenance = { status = "experimental" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 9758eda9db..4091d5aaa5 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.4.3" +version = "0.4.4" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -11,7 +11,7 @@ readme = "README.md" name = "cton_native" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.3" } +cretonne = { path = "../cretonne", version = "0.4.4" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 7ec20d899c..16f097bccc 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.4.3" +version = "0.4.4" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -12,7 +12,7 @@ readme = "README.md" name = "cton_reader" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.3" } +cretonne = { path = "../cretonne", version = "0.4.4" } [badges] maintenance = { status = "experimental" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 1733219e07..16af0765ac 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.4.3" +version = "0.4.4" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/Cretonne/cretonne" @@ -13,8 +13,8 @@ name = "cton_wasm" [dependencies] wasmparser = "0.15.1" -cretonne = { path = "../cretonne", version = "0.4.3" } -cretonne-frontend = { path = "../frontend", version = "0.4.3" } +cretonne = { path = "../cretonne", version = "0.4.4" } +cretonne-frontend = { path = "../frontend", version = "0.4.4" } [dev-dependencies] tempdir = "0.3.5" From 7767186dd05e06d46216cc328caa2fa6a8b03242 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 07:55:59 -0700 Subject: [PATCH 1698/3084] Rename 'cton_*' library names to match the 'cretonne_*' crate names. This renames `cton_frontend` to `cretonne_frontend` and so on. This fixes the first part of #287. --- cranelift/docs/langref.rst | 2 +- cranelift/src/cat.rs | 2 +- cranelift/src/compile.rs | 2 +- cranelift/src/cton-util.rs | 8 ++++---- cranelift/src/print_cfg.rs | 2 +- cranelift/src/utils.rs | 2 +- cranelift/src/wasm.rs | 5 +++-- cranelift/tests/filetests.rs | 4 ++-- lib/cretonne/Cargo.toml | 3 --- lib/cretonne/src/lib.rs | 2 +- lib/entity/Cargo.toml | 3 --- lib/filetests/Cargo.toml | 3 --- lib/filetests/src/lib.rs | 4 ++-- lib/filetests/src/runone.rs | 4 ++-- lib/filetests/src/subtest.rs | 2 +- lib/filetests/src/test_binemit.rs | 2 +- lib/filetests/src/test_cat.rs | 2 +- lib/filetests/src/test_compile.rs | 2 +- lib/filetests/src/test_dce.rs | 2 +- lib/filetests/src/test_domtree.rs | 2 +- lib/filetests/src/test_legalizer.rs | 2 +- lib/filetests/src/test_licm.rs | 2 +- lib/filetests/src/test_postopt.rs | 2 +- lib/filetests/src/test_preopt.rs | 2 +- lib/filetests/src/test_print_cfg.rs | 2 +- lib/filetests/src/test_regalloc.rs | 2 +- lib/filetests/src/test_simple_gvn.rs | 2 +- lib/filetests/src/test_verifier.rs | 2 +- lib/frontend/Cargo.toml | 3 --- lib/frontend/src/lib.rs | 4 ++-- lib/native/Cargo.toml | 3 --- lib/reader/Cargo.toml | 3 --- lib/reader/src/lib.rs | 4 ++-- lib/wasm/Cargo.toml | 3 --- lib/wasm/src/code_translator.rs | 2 +- lib/wasm/src/func_translator.rs | 2 +- lib/wasm/src/lib.rs | 2 +- lib/wasm/tests/wasm_testsuite.rs | 4 ++-- 38 files changed, 42 insertions(+), 62 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 070b27ee37..33fc1f0694 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -74,7 +74,7 @@ SSA values: In the entry block, ``v4`` is the initial value. In the loop block variable during each iteration. Finally, ``v12`` is computed as the induction variable value for the next iteration. -The `cton_frontend` crate contains utilities for translating from programs +The `cretonne_frontend` crate contains utilities for translating from programs containing multiple assignments to the same variables into SSA form for Cretonne :term:`IR`. diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs index e82af42624..c4890ffb24 100644 --- a/cranelift/src/cat.rs +++ b/cranelift/src/cat.rs @@ -4,7 +4,7 @@ //! normalizing formatting and removing comments. use CommandResult; -use cton_reader::parse_functions; +use cretonne_reader::parse_functions; use utils::read_to_string; pub fn run(files: &[String]) -> CommandResult { diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 6bab77e0d2..722f5b83c7 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -4,7 +4,7 @@ use cretonne::Context; use cretonne::print_errors::pretty_error; use cretonne::settings::FlagsOrIsa; use cretonne::{binemit, ir}; -use cton_reader::parse_test; +use cretonne_reader::parse_test; use std::path::Path; use std::path::PathBuf; use utils::{parse_sets_and_isa, read_to_string}; diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 3968030709..7ab78d649a 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -1,7 +1,7 @@ extern crate cretonne; -extern crate cton_filetests; -extern crate cton_reader; -extern crate cton_wasm; +extern crate cretonne_filetests; +extern crate cretonne_reader; +extern crate cretonne_wasm; extern crate docopt; extern crate filecheck; #[macro_use] @@ -86,7 +86,7 @@ fn cton_util() -> CommandResult { // Find the sub-command to execute. let result = if args.cmd_test { - cton_filetests::run(args.flag_verbose, &args.arg_file).map(|_time| ()) + cretonne_filetests::run(args.flag_verbose, &args.arg_file).map(|_time| ()) } else if args.cmd_cat { cat::run(&args.arg_file) } else if args.cmd_filecheck { diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index 846ee3e8fa..8dbd4324fc 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -5,7 +5,7 @@ use CommandResult; use cretonne::cfg_printer::CFGPrinter; -use cton_reader::parse_functions; +use cretonne_reader::parse_functions; use utils::read_to_string; pub fn run(files: &[String]) -> CommandResult { diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index 8a2994a7ab..fe43be14ab 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -3,7 +3,7 @@ use cretonne::isa; use cretonne::isa::TargetIsa; use cretonne::settings::{self, FlagsOrIsa}; -use cton_reader::{parse_options, Location}; +use cretonne_reader::{parse_options, Location}; use std::fs::File; use std::io::{self, Read}; use std::path::Path; diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 029c7ad5c0..1cbe2c754b 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -1,4 +1,5 @@ -//! CLI tool to use the functions provided by the [cretonne-wasm](../cton_wasm/index.html) crate. +//! CLI tool to use the functions provided by the [cretonne-wasm](../cretonne_wasm/index.html) +//! crate. //! //! Reads Wasm binary files, translates the functions' code to Cretonne IR. #![cfg_attr(feature = "cargo-clippy", allow(too_many_arguments, cyclomatic_complexity))] @@ -6,7 +7,7 @@ use cretonne::Context; use cretonne::print_errors::{pretty_error, pretty_verifier_error}; use cretonne::settings::FlagsOrIsa; -use cton_wasm::{translate_module, DummyEnvironment, ModuleEnvironment}; +use cretonne_wasm::{translate_module, DummyEnvironment, ModuleEnvironment}; use std::error::Error; use std::fs::File; use std::io; diff --git a/cranelift/tests/filetests.rs b/cranelift/tests/filetests.rs index 6b5e392c76..d99f168b61 100644 --- a/cranelift/tests/filetests.rs +++ b/cranelift/tests/filetests.rs @@ -1,7 +1,7 @@ -extern crate cton_filetests; +extern crate cretonne_filetests; #[test] fn filetests() { // Run all the filetests in the following directories. - cton_filetests::run(false, &["filetests".into(), "docs".into()]).expect("test harness"); + cretonne_filetests::run(false, &["filetests".into(), "docs".into()]).expect("test harness"); } diff --git a/lib/cretonne/Cargo.toml b/lib/cretonne/Cargo.toml index 1b93158451..d252fec291 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/cretonne/Cargo.toml @@ -10,9 +10,6 @@ readme = "README.md" keywords = ["compile", "compiler", "jit"] build = "build.rs" -[lib] -name = "cretonne" - [dependencies] cretonne-entity = { path = "../entity", version = "0.4.4" } # It is a goal of the cretonne crate to have minimal external dependencies. diff --git a/lib/cretonne/src/lib.rs b/lib/cretonne/src/lib.rs index 996bb865bd..8bc88ab420 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/cretonne/src/lib.rs @@ -39,7 +39,7 @@ pub use write::write_function; pub const VERSION: &str = env!("CARGO_PKG_VERSION"); #[macro_use] -pub extern crate cton_entity as entity; +pub extern crate cretonne_entity as entity; #[macro_use] pub mod dbg; diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index 726f85e46b..fa862aef94 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -9,9 +9,6 @@ repository = "https://github.com/Cretonne/cretonne" readme = "README.md" keywords = ["entity", "set", "map"] -[lib] -name = "cton_entity" - [features] default = ["std"] std = [] diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 0c7b7d5739..e0187f3f75 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -8,9 +8,6 @@ documentation = "http://cretonne.readthedocs.io/en/latest/testing.html#file-test repository = "https://github.com/Cretonne/cretonne" publish = false -[lib] -name = "cton_filetests" - [dependencies] cretonne = { path = "../cretonne", version = "0.4.4" } cretonne-reader = { path = "../reader", version = "0.4.4" } diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs index 718502eb53..d58f15fdc0 100644 --- a/lib/filetests/src/lib.rs +++ b/lib/filetests/src/lib.rs @@ -10,11 +10,11 @@ #[macro_use(dbg)] extern crate cretonne; -extern crate cton_reader; +extern crate cretonne_reader; extern crate filecheck; extern crate num_cpus; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use runner::TestRunner; use std::path::Path; use std::time; diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs index 8cdec2d626..dee5bfc2c9 100644 --- a/lib/filetests/src/runone.rs +++ b/lib/filetests/src/runone.rs @@ -6,8 +6,8 @@ use cretonne::print_errors::pretty_verifier_error; use cretonne::settings::Flags; use cretonne::timing; use cretonne::verify_function; -use cton_reader::IsaSpec; -use cton_reader::parse_test; +use cretonne_reader::IsaSpec; +use cretonne_reader::parse_test; use std::borrow::Cow; use std::fs; use std::io::{self, Read}; diff --git a/lib/filetests/src/subtest.rs b/lib/filetests/src/subtest.rs index ba94f87e78..4ae71494d7 100644 --- a/lib/filetests/src/subtest.rs +++ b/lib/filetests/src/subtest.rs @@ -3,7 +3,7 @@ use cretonne::ir::Function; use cretonne::isa::TargetIsa; use cretonne::settings::{Flags, FlagsOrIsa}; -use cton_reader::{Comment, Details}; +use cretonne_reader::{Comment, Details}; use filecheck::{Checker, CheckerBuilder, NO_VARIABLES}; use std::borrow::Cow; use std::result; diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index ce2e97e888..620eb7e9f3 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -9,7 +9,7 @@ use cretonne::dbg::DisplayList; use cretonne::ir; use cretonne::ir::entities::AnyEntity; use cretonne::print_errors::pretty_error; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use match_directive::match_directive; use std::borrow::Cow; use std::collections::HashMap; diff --git a/lib/filetests/src/test_cat.rs b/lib/filetests/src/test_cat.rs index f8f145834c..3d04fcae3e 100644 --- a/lib/filetests/src/test_cat.rs +++ b/lib/filetests/src/test_cat.rs @@ -1,7 +1,7 @@ //! The `cat` subtest. use cretonne::ir::Function; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use std::borrow::Cow; use subtest::{self, Context, Result as STResult, SubTest}; diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs index a924eee799..a5af1f54e7 100644 --- a/lib/filetests/src/test_compile.rs +++ b/lib/filetests/src/test_compile.rs @@ -6,7 +6,7 @@ use cretonne; use cretonne::binemit; use cretonne::ir; use cretonne::print_errors::pretty_error; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; diff --git a/lib/filetests/src/test_dce.rs b/lib/filetests/src/test_dce.rs index d96214182b..4d79cea1ce 100644 --- a/lib/filetests/src/test_dce.rs +++ b/lib/filetests/src/test_dce.rs @@ -8,7 +8,7 @@ use cretonne; use cretonne::ir::Function; use cretonne::print_errors::pretty_error; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; diff --git a/lib/filetests/src/test_domtree.rs b/lib/filetests/src/test_domtree.rs index da5a926f3d..9862f3f4fe 100644 --- a/lib/filetests/src/test_domtree.rs +++ b/lib/filetests/src/test_domtree.rs @@ -16,7 +16,7 @@ use cretonne::dominator_tree::{DominatorTree, DominatorTreePreorder}; use cretonne::flowgraph::ControlFlowGraph; use cretonne::ir::Function; use cretonne::ir::entities::AnyEntity; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use match_directive::match_directive; use std::borrow::{Borrow, Cow}; use std::collections::HashMap; diff --git a/lib/filetests/src/test_legalizer.rs b/lib/filetests/src/test_legalizer.rs index 0794081e51..af288bcc8d 100644 --- a/lib/filetests/src/test_legalizer.rs +++ b/lib/filetests/src/test_legalizer.rs @@ -6,7 +6,7 @@ use cretonne; use cretonne::ir::Function; use cretonne::print_errors::pretty_error; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; diff --git a/lib/filetests/src/test_licm.rs b/lib/filetests/src/test_licm.rs index c59dee1cba..435b5ab1bc 100644 --- a/lib/filetests/src/test_licm.rs +++ b/lib/filetests/src/test_licm.rs @@ -8,7 +8,7 @@ use cretonne; use cretonne::ir::Function; use cretonne::print_errors::pretty_error; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; diff --git a/lib/filetests/src/test_postopt.rs b/lib/filetests/src/test_postopt.rs index c89fb33ea5..b575a36096 100644 --- a/lib/filetests/src/test_postopt.rs +++ b/lib/filetests/src/test_postopt.rs @@ -5,7 +5,7 @@ use cretonne; use cretonne::ir::Function; use cretonne::print_errors::pretty_error; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; diff --git a/lib/filetests/src/test_preopt.rs b/lib/filetests/src/test_preopt.rs index e2fc9819c9..f58d216a26 100644 --- a/lib/filetests/src/test_preopt.rs +++ b/lib/filetests/src/test_preopt.rs @@ -5,7 +5,7 @@ use cretonne; use cretonne::ir::Function; use cretonne::print_errors::pretty_error; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; diff --git a/lib/filetests/src/test_print_cfg.rs b/lib/filetests/src/test_print_cfg.rs index 7092126cd7..f6388b70fe 100644 --- a/lib/filetests/src/test_print_cfg.rs +++ b/lib/filetests/src/test_print_cfg.rs @@ -7,7 +7,7 @@ use std::borrow::Cow; use cretonne::cfg_printer::CFGPrinter; use cretonne::ir::Function; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use subtest::{self, Context, Result as STResult, SubTest}; /// Object implementing the `test print-cfg` sub-test. diff --git a/lib/filetests/src/test_regalloc.rs b/lib/filetests/src/test_regalloc.rs index b2ab98c3a4..36f3ac169f 100644 --- a/lib/filetests/src/test_regalloc.rs +++ b/lib/filetests/src/test_regalloc.rs @@ -8,7 +8,7 @@ use cretonne; use cretonne::ir::Function; use cretonne::print_errors::pretty_error; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; diff --git a/lib/filetests/src/test_simple_gvn.rs b/lib/filetests/src/test_simple_gvn.rs index 62f5a2adae..b60379cff1 100644 --- a/lib/filetests/src/test_simple_gvn.rs +++ b/lib/filetests/src/test_simple_gvn.rs @@ -8,7 +8,7 @@ use cretonne; use cretonne::ir::Function; use cretonne::print_errors::pretty_error; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs index a4ec799ee3..0243469630 100644 --- a/lib/filetests/src/test_verifier.rs +++ b/lib/filetests/src/test_verifier.rs @@ -11,7 +11,7 @@ use cretonne::ir::Function; use cretonne::verify_function; -use cton_reader::TestCommand; +use cretonne_reader::TestCommand; use match_directive::match_directive; use std::borrow::{Borrow, Cow}; use subtest::{Context, Result, SubTest}; diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 3f35168f75..5dc031926b 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -8,9 +8,6 @@ documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/Cretonne/cretonne" readme = "README.md" -[lib] -name = "cton_frontend" - [dependencies] cretonne = { path = "../cretonne", version = "0.4.4" } diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 329b6afb50..369a4cf13d 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -33,13 +33,13 @@ //! //! ```rust //! extern crate cretonne; -//! extern crate cton_frontend; +//! extern crate cretonne_frontend; //! //! use cretonne::entity::EntityRef; //! use cretonne::ir::{ExternalName, CallConv, Function, Signature, AbiParam, InstBuilder}; //! use cretonne::ir::types::*; //! use cretonne::settings; -//! use cton_frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; +//! use cretonne_frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; //! use cretonne::verifier::verify_function; //! //! fn main() { diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 4091d5aaa5..3c80e45d15 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -7,9 +7,6 @@ repository = "https://github.com/Cretonne/cretonne" license = "Apache-2.0" readme = "README.md" -[lib] -name = "cton_native" - [dependencies] cretonne = { path = "../cretonne", version = "0.4.4" } diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 16f097bccc..d3a5213cdd 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -8,9 +8,6 @@ documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/Cretonne/cretonne" readme = "README.md" -[lib] -name = "cton_reader" - [dependencies] cretonne = { path = "../cretonne", version = "0.4.4" } diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index e012876c1c..8b4b831042 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -1,7 +1,7 @@ //! Cretonne file reader library. //! -//! The `cton_reader` library supports reading .cton files. This functionality is needed for testing -//! Cretonne, but is not essential for a JIT compiler. +//! The `cretonne_reader` library supports reading .cton files. This functionality is needed for +//! testing Cretonne, but is not essential for a JIT compiler. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 16af0765ac..f9c60e7c92 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -8,9 +8,6 @@ license = "Apache-2.0" readme = "README.md" keywords = ["webassembly", "wasm"] -[lib] -name = "cton_wasm" - [dependencies] wasmparser = "0.15.1" cretonne = { path = "../cretonne", version = "0.4.4" } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 9dbace8556..945cedc1de 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -26,7 +26,7 @@ use cretonne::ir::condcodes::{FloatCC, IntCC}; use cretonne::ir::types::*; use cretonne::ir::{self, InstBuilder, JumpTableData, MemFlags}; use cretonne::packed_option::ReservedValue; -use cton_frontend::{FunctionBuilder, Variable}; +use cretonne_frontend::{FunctionBuilder, Variable}; use environ::{FuncEnvironment, GlobalValue}; use state::{ControlStackFrame, TranslationState}; use std::collections::{hash_map, HashMap}; diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index c89fa21266..537aeed0c2 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -9,7 +9,7 @@ use cretonne::entity::EntityRef; use cretonne::ir::{self, Ebb, InstBuilder}; use cretonne::result::{CtonError, CtonResult}; use cretonne::timing; -use cton_frontend::{FunctionBuilder, FunctionBuilderContext, Variable}; +use cretonne_frontend::{FunctionBuilder, FunctionBuilderContext, Variable}; use environ::FuncEnvironment; use state::TranslationState; use wasmparser::{self, BinaryReader}; diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 6ed76be8e3..197d228eab 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -15,7 +15,7 @@ #[macro_use(dbg)] extern crate cretonne; -extern crate cton_frontend; +extern crate cretonne_frontend; extern crate wasmparser; mod code_translator; diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs index 273818f470..fac93e6012 100644 --- a/lib/wasm/tests/wasm_testsuite.rs +++ b/lib/wasm/tests/wasm_testsuite.rs @@ -1,11 +1,11 @@ extern crate cretonne; -extern crate cton_wasm; +extern crate cretonne_wasm; extern crate tempdir; use cretonne::print_errors::pretty_verifier_error; use cretonne::settings::{self, Configurable, Flags}; use cretonne::verifier; -use cton_wasm::{translate_module, DummyEnvironment}; +use cretonne_wasm::{translate_module, DummyEnvironment}; use std::error::Error; use std::fs; use std::fs::File; From 24fa169e1f90b75318409050ba3b56ca57caead8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 08:48:02 -0700 Subject: [PATCH 1699/3084] Rename the 'cretonne' crate to 'cretonne-codegen'. This fixes the next part of #287. --- cranelift/Cargo.toml | 6 +-- cranelift/docs/Makefile | 2 +- cranelift/docs/conf.py | 2 +- cranelift/docs/metaref.rst | 10 ++-- cranelift/docs/testing.rst | 10 ++-- .../filetests/regalloc/spill-noregs.cton | 2 +- cranelift/publish-all.sh | 2 +- cranelift/src/compile.rs | 8 +-- cranelift/src/cton-util.rs | 4 +- cranelift/src/print_cfg.rs | 2 +- cranelift/src/utils.rs | 6 +-- cranelift/src/wasm.rs | 6 +-- cranelift/test-all.sh | 4 +- lib/{cretonne => codegen}/Cargo.toml | 4 +- lib/{cretonne => codegen}/README.md | 0 lib/{cretonne => codegen}/build.rs | 4 +- .../meta/base/__init__.py | 0 .../meta/base/entities.py | 0 .../meta/base/formats.py | 0 .../meta/base/immediates.py | 0 .../meta/base/instructions.py | 0 .../meta/base/legalize.py | 0 .../meta/base/predicates.py | 0 .../meta/base/semantics.py | 0 .../meta/base/settings.py | 0 lib/{cretonne => codegen}/meta/base/types.py | 0 lib/{cretonne => codegen}/meta/build.py | 2 +- .../meta/cdsl/__init__.py | 0 lib/{cretonne => codegen}/meta/cdsl/ast.py | 0 .../meta/cdsl/formats.py | 0 .../meta/cdsl/instructions.py | 0 lib/{cretonne => codegen}/meta/cdsl/isa.py | 0 .../meta/cdsl/operands.py | 0 .../meta/cdsl/predicates.py | 0 .../meta/cdsl/registers.py | 0 .../meta/cdsl/settings.py | 0 .../meta/cdsl/test_ast.py | 0 .../meta/cdsl/test_package.py | 0 .../meta/cdsl/test_ti.py | 0 .../meta/cdsl/test_typevar.py | 0 .../meta/cdsl/test_xform.py | 0 lib/{cretonne => codegen}/meta/cdsl/ti.py | 0 lib/{cretonne => codegen}/meta/cdsl/types.py | 0 .../meta/cdsl/typevar.py | 0 lib/{cretonne => codegen}/meta/cdsl/xform.py | 0 lib/{cretonne => codegen}/meta/check.sh | 0 .../meta/constant_hash.py | 0 lib/{cretonne => codegen}/meta/gen_binemit.py | 0 .../meta/gen_build_deps.py | 2 +- .../meta/gen_encoding.py | 0 lib/{cretonne => codegen}/meta/gen_instr.py | 0 .../meta/gen_legalizer.py | 0 .../meta/gen_registers.py | 0 .../meta/gen_settings.py | 0 lib/{cretonne => codegen}/meta/gen_types.py | 2 +- .../meta/isa/__init__.py | 0 .../meta/isa/arm32/__init__.py | 0 .../meta/isa/arm32/defs.py | 0 .../meta/isa/arm32/registers.py | 0 .../meta/isa/arm32/settings.py | 0 .../meta/isa/arm64/__init__.py | 0 .../meta/isa/arm64/defs.py | 0 .../meta/isa/arm64/registers.py | 0 .../meta/isa/arm64/settings.py | 0 .../meta/isa/riscv/__init__.py | 0 .../meta/isa/riscv/defs.py | 0 .../meta/isa/riscv/encodings.py | 0 .../meta/isa/riscv/recipes.py | 0 .../meta/isa/riscv/registers.py | 0 .../meta/isa/riscv/settings.py | 0 .../meta/isa/x86/__init__.py | 0 .../meta/isa/x86/defs.py | 0 .../meta/isa/x86/encodings.py | 0 .../meta/isa/x86/instructions.py | 0 .../meta/isa/x86/legalize.py | 0 .../meta/isa/x86/recipes.py | 0 .../meta/isa/x86/registers.py | 0 .../meta/isa/x86/settings.py | 0 lib/{cretonne => codegen}/meta/mypy.ini | 0 .../meta/semantics/__init__.py | 0 .../meta/semantics/elaborate.py | 0 .../meta/semantics/macros.py | 0 .../meta/semantics/primitives.py | 0 .../meta/semantics/smtlib.py | 0 .../meta/semantics/test_elaborate.py | 0 lib/{cretonne => codegen}/meta/srcgen.py | 0 .../meta/stubs/z3/__init__.pyi | 0 .../meta/stubs/z3/z3core.pyi | 0 .../meta/stubs/z3/z3types.pyi | 0 .../meta/test_constant_hash.py | 0 .../meta/test_gen_legalizer.py | 0 lib/{cretonne => codegen}/meta/test_srcgen.py | 0 .../meta/unique_table.py | 0 lib/{cretonne => codegen}/src/abi.rs | 0 lib/{cretonne => codegen}/src/bforest/map.rs | 0 lib/{cretonne => codegen}/src/bforest/mod.rs | 0 lib/{cretonne => codegen}/src/bforest/node.rs | 0 lib/{cretonne => codegen}/src/bforest/path.rs | 0 lib/{cretonne => codegen}/src/bforest/pool.rs | 0 lib/{cretonne => codegen}/src/bforest/set.rs | 0 .../src/binemit/memorysink.rs | 0 lib/{cretonne => codegen}/src/binemit/mod.rs | 0 .../src/binemit/relaxation.rs | 0 lib/{cretonne => codegen}/src/bitset.rs | 0 lib/{cretonne => codegen}/src/cfg_printer.rs | 0 .../src/constant_hash.rs | 4 +- lib/{cretonne => codegen}/src/context.rs | 0 lib/{cretonne => codegen}/src/cursor.rs | 52 +++++++++---------- lib/{cretonne => codegen}/src/dbg.rs | 0 lib/{cretonne => codegen}/src/dce.rs | 0 .../src/divconst_magic_numbers.rs | 0 .../src/dominator_tree.rs | 0 lib/{cretonne => codegen}/src/flowgraph.rs | 0 lib/{cretonne => codegen}/src/ir/builder.rs | 2 +- lib/{cretonne => codegen}/src/ir/condcodes.rs | 0 lib/{cretonne => codegen}/src/ir/dfg.rs | 0 lib/{cretonne => codegen}/src/ir/entities.rs | 0 lib/{cretonne => codegen}/src/ir/extfunc.rs | 0 lib/{cretonne => codegen}/src/ir/extname.rs | 7 +-- lib/{cretonne => codegen}/src/ir/function.rs | 0 lib/{cretonne => codegen}/src/ir/globalvar.rs | 0 lib/{cretonne => codegen}/src/ir/heap.rs | 0 .../src/ir/immediates.rs | 0 .../src/ir/instructions.rs | 2 +- lib/{cretonne => codegen}/src/ir/jumptable.rs | 0 lib/{cretonne => codegen}/src/ir/layout.rs | 0 lib/{cretonne => codegen}/src/ir/libcall.rs | 0 lib/{cretonne => codegen}/src/ir/memflags.rs | 0 lib/{cretonne => codegen}/src/ir/mod.rs | 0 lib/{cretonne => codegen}/src/ir/progpoint.rs | 0 lib/{cretonne => codegen}/src/ir/sourceloc.rs | 0 lib/{cretonne => codegen}/src/ir/stackslot.rs | 0 lib/{cretonne => codegen}/src/ir/trapcode.rs | 0 lib/{cretonne => codegen}/src/ir/types.rs | 2 +- lib/{cretonne => codegen}/src/ir/valueloc.rs | 0 .../src/isa/arm32/abi.rs | 0 .../src/isa/arm32/binemit.rs | 0 .../src/isa/arm32/enc_tables.rs | 0 .../src/isa/arm32/mod.rs | 0 .../src/isa/arm32/registers.rs | 0 .../src/isa/arm32/settings.rs | 4 +- .../src/isa/arm64/abi.rs | 0 .../src/isa/arm64/binemit.rs | 0 .../src/isa/arm64/enc_tables.rs | 0 .../src/isa/arm64/mod.rs | 0 .../src/isa/arm64/registers.rs | 0 .../src/isa/arm64/settings.rs | 4 +- .../src/isa/constraints.rs | 0 .../src/isa/enc_tables.rs | 2 +- lib/{cretonne => codegen}/src/isa/encoding.rs | 0 lib/{cretonne => codegen}/src/isa/mod.rs | 4 +- .../src/isa/registers.rs | 0 .../src/isa/riscv/abi.rs | 0 .../src/isa/riscv/binemit.rs | 0 .../src/isa/riscv/enc_tables.rs | 0 .../src/isa/riscv/mod.rs | 0 .../src/isa/riscv/registers.rs | 0 .../src/isa/riscv/settings.rs | 4 +- lib/{cretonne => codegen}/src/isa/stack.rs | 0 lib/{cretonne => codegen}/src/isa/x86/abi.rs | 0 .../src/isa/x86/binemit.rs | 0 .../src/isa/x86/enc_tables.rs | 0 lib/{cretonne => codegen}/src/isa/x86/mod.rs | 0 .../src/isa/x86/registers.rs | 0 .../src/isa/x86/settings.rs | 4 +- lib/{cretonne => codegen}/src/iterators.rs | 0 .../src/legalizer/boundary.rs | 0 .../src/legalizer/call.rs | 0 .../src/legalizer/globalvar.rs | 0 .../src/legalizer/heap.rs | 0 .../src/legalizer/libcall.rs | 0 .../src/legalizer/mod.rs | 2 +- .../src/legalizer/split.rs | 0 lib/{cretonne => codegen}/src/lib.rs | 2 +- lib/{cretonne => codegen}/src/licm.rs | 0 .../src/loop_analysis.rs | 0 .../src/partition_slice.rs | 0 lib/{cretonne => codegen}/src/postopt.rs | 0 lib/{cretonne => codegen}/src/predicates.rs | 2 +- lib/{cretonne => codegen}/src/preopt.rs | 0 lib/{cretonne => codegen}/src/print_errors.rs | 0 lib/{cretonne => codegen}/src/ref_slice.rs | 0 .../src/regalloc/affinity.rs | 0 .../src/regalloc/coalescing.rs | 0 .../src/regalloc/coloring.rs | 0 .../src/regalloc/context.rs | 0 .../src/regalloc/diversion.rs | 0 .../src/regalloc/live_value_tracker.rs | 0 .../src/regalloc/liveness.rs | 0 .../src/regalloc/liverange.rs | 0 lib/{cretonne => codegen}/src/regalloc/mod.rs | 0 .../src/regalloc/pressure.rs | 0 .../src/regalloc/register_set.rs | 0 .../src/regalloc/reload.rs | 0 .../src/regalloc/solver.rs | 0 .../src/regalloc/spilling.rs | 0 .../src/regalloc/virtregs.rs | 0 lib/{cretonne => codegen}/src/result.rs | 0 .../src/scoped_hash_map.rs | 0 lib/{cretonne => codegen}/src/settings.rs | 4 +- lib/{cretonne => codegen}/src/simple_gvn.rs | 0 lib/{cretonne => codegen}/src/stack_layout.rs | 0 lib/{cretonne => codegen}/src/timing.rs | 0 lib/{cretonne => codegen}/src/topo_order.rs | 0 .../src/unreachable_code.rs | 0 .../src/verifier/cssa.rs | 0 .../src/verifier/flags.rs | 0 .../src/verifier/liveness.rs | 0 .../src/verifier/locations.rs | 0 lib/{cretonne => codegen}/src/verifier/mod.rs | 0 lib/{cretonne => codegen}/src/write.rs | 0 lib/filetests/Cargo.toml | 2 +- lib/filetests/src/concurrent.rs | 2 +- lib/filetests/src/lib.rs | 2 +- lib/filetests/src/runone.rs | 12 ++--- lib/filetests/src/subtest.rs | 6 +-- lib/filetests/src/test_binemit.rs | 12 ++--- lib/filetests/src/test_cat.rs | 2 +- lib/filetests/src/test_compile.rs | 9 ++-- lib/filetests/src/test_dce.rs | 8 +-- lib/filetests/src/test_domtree.rs | 8 +-- lib/filetests/src/test_legalizer.rs | 8 +-- lib/filetests/src/test_licm.rs | 8 +-- lib/filetests/src/test_postopt.rs | 8 +-- lib/filetests/src/test_preopt.rs | 8 +-- lib/filetests/src/test_print_cfg.rs | 4 +- lib/filetests/src/test_regalloc.rs | 8 +-- lib/filetests/src/test_simple_gvn.rs | 8 +-- lib/filetests/src/test_verifier.rs | 4 +- lib/frontend/Cargo.toml | 2 +- lib/frontend/src/frontend.rs | 33 ++++++------ lib/frontend/src/lib.rs | 14 ++--- lib/frontend/src/ssa.rs | 30 +++++------ lib/frontend/src/variable.rs | 2 +- lib/native/Cargo.toml | 2 +- lib/native/src/lib.rs | 6 +-- lib/reader/Cargo.toml | 2 +- lib/reader/src/isaspec.rs | 4 +- lib/reader/src/lexer.rs | 8 +-- lib/reader/src/lib.rs | 2 +- lib/reader/src/parser.rs | 35 +++++++------ lib/reader/src/sourcemap.rs | 4 +- lib/reader/src/testfile.rs | 4 +- lib/wasm/Cargo.toml | 2 +- lib/wasm/src/code_translator.rs | 8 +-- lib/wasm/src/environ/dummy.rs | 8 +-- lib/wasm/src/environ/spec.rs | 6 +-- lib/wasm/src/func_translator.rs | 12 ++--- lib/wasm/src/lib.rs | 2 +- lib/wasm/src/module_translator.rs | 4 +- lib/wasm/src/sections_translator.rs | 7 ++- lib/wasm/src/state.rs | 2 +- lib/wasm/src/translation_utils.rs | 24 ++++----- lib/wasm/tests/wasm_testsuite.rs | 8 +-- 254 files changed, 265 insertions(+), 264 deletions(-) rename lib/{cretonne => codegen}/Cargo.toml (86%) rename lib/{cretonne => codegen}/README.md (100%) rename lib/{cretonne => codegen}/build.rs (96%) rename lib/{cretonne => codegen}/meta/base/__init__.py (100%) rename lib/{cretonne => codegen}/meta/base/entities.py (100%) rename lib/{cretonne => codegen}/meta/base/formats.py (100%) rename lib/{cretonne => codegen}/meta/base/immediates.py (100%) rename lib/{cretonne => codegen}/meta/base/instructions.py (100%) rename lib/{cretonne => codegen}/meta/base/legalize.py (100%) rename lib/{cretonne => codegen}/meta/base/predicates.py (100%) rename lib/{cretonne => codegen}/meta/base/semantics.py (100%) rename lib/{cretonne => codegen}/meta/base/settings.py (100%) rename lib/{cretonne => codegen}/meta/base/types.py (100%) rename lib/{cretonne => codegen}/meta/build.py (92%) rename lib/{cretonne => codegen}/meta/cdsl/__init__.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/ast.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/formats.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/instructions.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/isa.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/operands.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/predicates.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/registers.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/settings.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/test_ast.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/test_package.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/test_ti.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/test_typevar.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/test_xform.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/ti.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/types.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/typevar.py (100%) rename lib/{cretonne => codegen}/meta/cdsl/xform.py (100%) rename lib/{cretonne => codegen}/meta/check.sh (100%) rename lib/{cretonne => codegen}/meta/constant_hash.py (100%) rename lib/{cretonne => codegen}/meta/gen_binemit.py (100%) rename lib/{cretonne => codegen}/meta/gen_build_deps.py (94%) rename lib/{cretonne => codegen}/meta/gen_encoding.py (100%) rename lib/{cretonne => codegen}/meta/gen_instr.py (100%) rename lib/{cretonne => codegen}/meta/gen_legalizer.py (100%) rename lib/{cretonne => codegen}/meta/gen_registers.py (100%) rename lib/{cretonne => codegen}/meta/gen_settings.py (100%) rename lib/{cretonne => codegen}/meta/gen_types.py (95%) rename lib/{cretonne => codegen}/meta/isa/__init__.py (100%) rename lib/{cretonne => codegen}/meta/isa/arm32/__init__.py (100%) rename lib/{cretonne => codegen}/meta/isa/arm32/defs.py (100%) rename lib/{cretonne => codegen}/meta/isa/arm32/registers.py (100%) rename lib/{cretonne => codegen}/meta/isa/arm32/settings.py (100%) rename lib/{cretonne => codegen}/meta/isa/arm64/__init__.py (100%) rename lib/{cretonne => codegen}/meta/isa/arm64/defs.py (100%) rename lib/{cretonne => codegen}/meta/isa/arm64/registers.py (100%) rename lib/{cretonne => codegen}/meta/isa/arm64/settings.py (100%) rename lib/{cretonne => codegen}/meta/isa/riscv/__init__.py (100%) rename lib/{cretonne => codegen}/meta/isa/riscv/defs.py (100%) rename lib/{cretonne => codegen}/meta/isa/riscv/encodings.py (100%) rename lib/{cretonne => codegen}/meta/isa/riscv/recipes.py (100%) rename lib/{cretonne => codegen}/meta/isa/riscv/registers.py (100%) rename lib/{cretonne => codegen}/meta/isa/riscv/settings.py (100%) rename lib/{cretonne => codegen}/meta/isa/x86/__init__.py (100%) rename lib/{cretonne => codegen}/meta/isa/x86/defs.py (100%) rename lib/{cretonne => codegen}/meta/isa/x86/encodings.py (100%) rename lib/{cretonne => codegen}/meta/isa/x86/instructions.py (100%) rename lib/{cretonne => codegen}/meta/isa/x86/legalize.py (100%) rename lib/{cretonne => codegen}/meta/isa/x86/recipes.py (100%) rename lib/{cretonne => codegen}/meta/isa/x86/registers.py (100%) rename lib/{cretonne => codegen}/meta/isa/x86/settings.py (100%) rename lib/{cretonne => codegen}/meta/mypy.ini (100%) rename lib/{cretonne => codegen}/meta/semantics/__init__.py (100%) rename lib/{cretonne => codegen}/meta/semantics/elaborate.py (100%) rename lib/{cretonne => codegen}/meta/semantics/macros.py (100%) rename lib/{cretonne => codegen}/meta/semantics/primitives.py (100%) rename lib/{cretonne => codegen}/meta/semantics/smtlib.py (100%) rename lib/{cretonne => codegen}/meta/semantics/test_elaborate.py (100%) rename lib/{cretonne => codegen}/meta/srcgen.py (100%) rename lib/{cretonne => codegen}/meta/stubs/z3/__init__.pyi (100%) rename lib/{cretonne => codegen}/meta/stubs/z3/z3core.pyi (100%) rename lib/{cretonne => codegen}/meta/stubs/z3/z3types.pyi (100%) rename lib/{cretonne => codegen}/meta/test_constant_hash.py (100%) rename lib/{cretonne => codegen}/meta/test_gen_legalizer.py (100%) rename lib/{cretonne => codegen}/meta/test_srcgen.py (100%) rename lib/{cretonne => codegen}/meta/unique_table.py (100%) rename lib/{cretonne => codegen}/src/abi.rs (100%) rename lib/{cretonne => codegen}/src/bforest/map.rs (100%) rename lib/{cretonne => codegen}/src/bforest/mod.rs (100%) rename lib/{cretonne => codegen}/src/bforest/node.rs (100%) rename lib/{cretonne => codegen}/src/bforest/path.rs (100%) rename lib/{cretonne => codegen}/src/bforest/pool.rs (100%) rename lib/{cretonne => codegen}/src/bforest/set.rs (100%) rename lib/{cretonne => codegen}/src/binemit/memorysink.rs (100%) rename lib/{cretonne => codegen}/src/binemit/mod.rs (100%) rename lib/{cretonne => codegen}/src/binemit/relaxation.rs (100%) rename lib/{cretonne => codegen}/src/bitset.rs (100%) rename lib/{cretonne => codegen}/src/cfg_printer.rs (100%) rename lib/{cretonne => codegen}/src/constant_hash.rs (93%) rename lib/{cretonne => codegen}/src/context.rs (100%) rename lib/{cretonne => codegen}/src/cursor.rs (93%) rename lib/{cretonne => codegen}/src/dbg.rs (100%) rename lib/{cretonne => codegen}/src/dce.rs (100%) rename lib/{cretonne => codegen}/src/divconst_magic_numbers.rs (100%) rename lib/{cretonne => codegen}/src/dominator_tree.rs (100%) rename lib/{cretonne => codegen}/src/flowgraph.rs (100%) rename lib/{cretonne => codegen}/src/ir/builder.rs (99%) rename lib/{cretonne => codegen}/src/ir/condcodes.rs (100%) rename lib/{cretonne => codegen}/src/ir/dfg.rs (100%) rename lib/{cretonne => codegen}/src/ir/entities.rs (100%) rename lib/{cretonne => codegen}/src/ir/extfunc.rs (100%) rename lib/{cretonne => codegen}/src/ir/extname.rs (96%) rename lib/{cretonne => codegen}/src/ir/function.rs (100%) rename lib/{cretonne => codegen}/src/ir/globalvar.rs (100%) rename lib/{cretonne => codegen}/src/ir/heap.rs (100%) rename lib/{cretonne => codegen}/src/ir/immediates.rs (100%) rename lib/{cretonne => codegen}/src/ir/instructions.rs (99%) rename lib/{cretonne => codegen}/src/ir/jumptable.rs (100%) rename lib/{cretonne => codegen}/src/ir/layout.rs (100%) rename lib/{cretonne => codegen}/src/ir/libcall.rs (100%) rename lib/{cretonne => codegen}/src/ir/memflags.rs (100%) rename lib/{cretonne => codegen}/src/ir/mod.rs (100%) rename lib/{cretonne => codegen}/src/ir/progpoint.rs (100%) rename lib/{cretonne => codegen}/src/ir/sourceloc.rs (100%) rename lib/{cretonne => codegen}/src/ir/stackslot.rs (100%) rename lib/{cretonne => codegen}/src/ir/trapcode.rs (100%) rename lib/{cretonne => codegen}/src/ir/types.rs (99%) rename lib/{cretonne => codegen}/src/ir/valueloc.rs (100%) rename lib/{cretonne => codegen}/src/isa/arm32/abi.rs (100%) rename lib/{cretonne => codegen}/src/isa/arm32/binemit.rs (100%) rename lib/{cretonne => codegen}/src/isa/arm32/enc_tables.rs (100%) rename lib/{cretonne => codegen}/src/isa/arm32/mod.rs (100%) rename lib/{cretonne => codegen}/src/isa/arm32/registers.rs (100%) rename lib/{cretonne => codegen}/src/isa/arm32/settings.rs (58%) rename lib/{cretonne => codegen}/src/isa/arm64/abi.rs (100%) rename lib/{cretonne => codegen}/src/isa/arm64/binemit.rs (100%) rename lib/{cretonne => codegen}/src/isa/arm64/enc_tables.rs (100%) rename lib/{cretonne => codegen}/src/isa/arm64/mod.rs (100%) rename lib/{cretonne => codegen}/src/isa/arm64/registers.rs (100%) rename lib/{cretonne => codegen}/src/isa/arm64/settings.rs (58%) rename lib/{cretonne => codegen}/src/isa/constraints.rs (100%) rename lib/{cretonne => codegen}/src/isa/enc_tables.rs (99%) rename lib/{cretonne => codegen}/src/isa/encoding.rs (100%) rename lib/{cretonne => codegen}/src/isa/mod.rs (99%) rename lib/{cretonne => codegen}/src/isa/registers.rs (100%) rename lib/{cretonne => codegen}/src/isa/riscv/abi.rs (100%) rename lib/{cretonne => codegen}/src/isa/riscv/binemit.rs (100%) rename lib/{cretonne => codegen}/src/isa/riscv/enc_tables.rs (100%) rename lib/{cretonne => codegen}/src/isa/riscv/mod.rs (100%) rename lib/{cretonne => codegen}/src/isa/riscv/registers.rs (100%) rename lib/{cretonne => codegen}/src/isa/riscv/settings.rs (91%) rename lib/{cretonne => codegen}/src/isa/stack.rs (100%) rename lib/{cretonne => codegen}/src/isa/x86/abi.rs (100%) rename lib/{cretonne => codegen}/src/isa/x86/binemit.rs (100%) rename lib/{cretonne => codegen}/src/isa/x86/enc_tables.rs (100%) rename lib/{cretonne => codegen}/src/isa/x86/mod.rs (100%) rename lib/{cretonne => codegen}/src/isa/x86/registers.rs (100%) rename lib/{cretonne => codegen}/src/isa/x86/settings.rs (91%) rename lib/{cretonne => codegen}/src/iterators.rs (100%) rename lib/{cretonne => codegen}/src/legalizer/boundary.rs (100%) rename lib/{cretonne => codegen}/src/legalizer/call.rs (100%) rename lib/{cretonne => codegen}/src/legalizer/globalvar.rs (100%) rename lib/{cretonne => codegen}/src/legalizer/heap.rs (100%) rename lib/{cretonne => codegen}/src/legalizer/libcall.rs (100%) rename lib/{cretonne => codegen}/src/legalizer/mod.rs (99%) rename lib/{cretonne => codegen}/src/legalizer/split.rs (100%) rename lib/{cretonne => codegen}/src/lib.rs (97%) rename lib/{cretonne => codegen}/src/licm.rs (100%) rename lib/{cretonne => codegen}/src/loop_analysis.rs (100%) rename lib/{cretonne => codegen}/src/partition_slice.rs (100%) rename lib/{cretonne => codegen}/src/postopt.rs (100%) rename lib/{cretonne => codegen}/src/predicates.rs (98%) rename lib/{cretonne => codegen}/src/preopt.rs (100%) rename lib/{cretonne => codegen}/src/print_errors.rs (100%) rename lib/{cretonne => codegen}/src/ref_slice.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/affinity.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/coalescing.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/coloring.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/context.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/diversion.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/live_value_tracker.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/liveness.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/liverange.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/mod.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/pressure.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/register_set.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/reload.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/solver.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/spilling.rs (100%) rename lib/{cretonne => codegen}/src/regalloc/virtregs.rs (100%) rename lib/{cretonne => codegen}/src/result.rs (100%) rename lib/{cretonne => codegen}/src/scoped_hash_map.rs (100%) rename lib/{cretonne => codegen}/src/settings.rs (98%) rename lib/{cretonne => codegen}/src/simple_gvn.rs (100%) rename lib/{cretonne => codegen}/src/stack_layout.rs (100%) rename lib/{cretonne => codegen}/src/timing.rs (100%) rename lib/{cretonne => codegen}/src/topo_order.rs (100%) rename lib/{cretonne => codegen}/src/unreachable_code.rs (100%) rename lib/{cretonne => codegen}/src/verifier/cssa.rs (100%) rename lib/{cretonne => codegen}/src/verifier/flags.rs (100%) rename lib/{cretonne => codegen}/src/verifier/liveness.rs (100%) rename lib/{cretonne => codegen}/src/verifier/locations.rs (100%) rename lib/{cretonne => codegen}/src/verifier/mod.rs (100%) rename lib/{cretonne => codegen}/src/write.rs (100%) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index c5ab163829..62b485ae50 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -2,7 +2,7 @@ name = "cretonne-tools" authors = ["The Cretonne Project Developers"] version = "0.4.4" -description = "Binaries for testing the Cretonne library" +description = "Binaries for testing the Cretonne libraries" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/Cretonne/cretonne" @@ -13,7 +13,7 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne = { path = "lib/cretonne", version = "0.4.4" } +cretonne-codegen = { path = "lib/codegen", version = "0.4.4" } cretonne-reader = { path = "lib/reader", version = "0.4.4" } cretonne-frontend = { path = "lib/frontend", version = "0.4.4" } cretonne-wasm = { path = "lib/wasm", version = "0.4.4" } @@ -30,7 +30,7 @@ term = "0.5.1" # Enable debug assertions and parallel compilation when building cretonne-tools # since they are for testing and development mostly. This doesn't affect the -# flags used to build the Cretonne crate when used as a dependency. +# flags used to build the cretonne-* crates when used as a dependency. [profile.release] opt-level = 2 debug-assertions = true diff --git a/cranelift/docs/Makefile b/cranelift/docs/Makefile index 335779f1f3..6ac763bdd3 100644 --- a/cranelift/docs/Makefile +++ b/cranelift/docs/Makefile @@ -14,7 +14,7 @@ help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) autohtml: html - $(SPHINXABUILD) -z ../lib/cretonne/meta --ignore '.*' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html + $(SPHINXABUILD) -z ../lib/codegen/meta --ignore '.*' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html .PHONY: help Makefile diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py index 20cfd32f91..b2c1f14f97 100644 --- a/cranelift/docs/conf.py +++ b/cranelift/docs/conf.py @@ -23,7 +23,7 @@ sys.path.insert(0, os.path.abspath('.')) # Also add the meta directory to sys.path so autodoc can find the Cretonne meta # language definitions. -sys.path.insert(0, os.path.abspath('../lib/cretonne/meta')) +sys.path.insert(0, os.path.abspath('../lib/codegen/meta')) # -- General configuration ------------------------------------------------ diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index d0b78d3a07..134628a8b1 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -11,7 +11,7 @@ domain specific language embedded in Python. This document describes the Python modules that form the embedded DSL. The meta language descriptions are Python modules under the -:file:`lib/cretonne/meta` directory. The descriptions are processed in two +:file:`lib/codegen/meta` directory. The descriptions are processed in two steps: 1. The Python modules are imported. This has the effect of building static data @@ -23,8 +23,8 @@ steps: constant tables. The main driver for this source code generation process is the -:file:`lib/cretonne/meta/build.py` script which is invoked as part of the build -process if anything in the :file:`lib/cretonne/meta` directory has changed +:file:`lib/codegen/meta/build.py` script which is invoked as part of the build +process if anything in the :file:`lib/codegen/meta` directory has changed since the last build. @@ -38,7 +38,7 @@ of code generation. Each setting is defined in the meta language so a compact and consistent Rust representation can be generated. Shared settings are defined in the :mod:`base.settings` module. Some settings are specific to a target ISA, and defined in a :file:`settings.py` module under the appropriate -:file:`lib/cretonne/meta/isa/*` directory. +:file:`lib/codegen/meta/isa/*` directory. Settings can take boolean on/off values, small numbers, or explicitly enumerated symbolic values. Each type is represented by a sub-class of :class:`Setting`: @@ -433,7 +433,7 @@ architectures. Each ISA is represented by a :py:class:`cdsl.isa.TargetISA` insta .. autoclass:: TargetISA The definitions for each supported target live in a package under -:file:`lib/cretonne/meta/isa`. +:file:`lib/codegen/meta/isa`. .. automodule:: isa :members: diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index db7efb1da1..b2ea827075 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -51,7 +51,7 @@ tested:: //! //! # Example //! ``` - //! use cretonne::settings::{self, Configurable}; + //! use cretonne_codegen::settings::{self, Configurable}; //! //! let mut b = settings::builder(); //! b.set("opt_level", "fastest"); @@ -73,9 +73,9 @@ test. These tests are usually found in the :file:`tests` top-level directory where they have access to all the crates in the Cretonne repository. The -:file:`lib/cretonne` and :file:`lib/reader` crates have no external +:file:`lib/codegen` and :file:`lib/reader` crates have no external dependencies, which can make testing tedious. Integration tests that don't need -to depend on other crates can be placed in :file:`lib/cretonne/tests` and +to depend on other crates can be placed in :file:`lib/codegen/tests` and :file:`lib/reader/tests`. File tests @@ -109,7 +109,7 @@ header: isa_spec : "isa" isa_name { `option` } "\n" The options given on the ``isa`` line modify the ISA-specific settings defined in -:file:`lib/cretonne/meta/isa/*/settings.py`. +:file:`lib/codegen/meta/isa/*/settings.py`. All types of tests allow shared Cretonne settings to be modified: @@ -119,7 +119,7 @@ All types of tests allow shared Cretonne settings to be modified: option : flag | setting "=" value The shared settings available for all target ISAs are defined in -:file:`lib/cretonne/meta/base/settings.py`. +:file:`lib/codegen/meta/base/settings.py`. The ``set`` lines apply settings cumulatively:: diff --git a/cranelift/filetests/regalloc/spill-noregs.cton b/cranelift/filetests/regalloc/spill-noregs.cton index 65f1da84c8..ffb9f041c1 100644 --- a/cranelift/filetests/regalloc/spill-noregs.cton +++ b/cranelift/filetests/regalloc/spill-noregs.cton @@ -6,7 +6,7 @@ isa x86 ; ; The spiller panics with a ; 'Ran out of GPR registers when inserting copy before v68 = icmp.i32 eq v66, v67', -; lib/cretonne/src/regalloc/spilling.rs:425:28 message. +; lib/codegen/src/regalloc/spilling.rs:425:28 message. ; ; The process_reg_uses() function is trying to insert a copy before the icmp instruction in ebb4 ; and runs out of registers to spill. Note that ebb7 has a lot of dead parameter values. diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index c98d0da07a..3b17f7cf1a 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -27,7 +27,7 @@ cargo update echo git commit -a -m "\"Bump version to $version"\" echo git push -for crate in entity cretonne frontend native reader wasm ; do +for crate in entity codegen frontend native reader wasm ; do echo cargo publish --manifest-path "lib/$crate/Cargo.toml" done echo diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 722f5b83c7..ae8cd1e30c 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -1,9 +1,9 @@ //! CLI tool to read Cretonne IR files and compile them into native code. -use cretonne::Context; -use cretonne::print_errors::pretty_error; -use cretonne::settings::FlagsOrIsa; -use cretonne::{binemit, ir}; +use cretonne_codegen::Context; +use cretonne_codegen::print_errors::pretty_error; +use cretonne_codegen::settings::FlagsOrIsa; +use cretonne_codegen::{binemit, ir}; use cretonne_reader::parse_test; use std::path::Path; use std::path::PathBuf; diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 7ab78d649a..20743d923b 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -1,4 +1,4 @@ -extern crate cretonne; +extern crate cretonne_codegen; extern crate cretonne_filetests; extern crate cretonne_reader; extern crate cretonne_wasm; @@ -9,7 +9,7 @@ extern crate serde_derive; extern crate tempdir; extern crate term; -use cretonne::{timing, VERSION}; +use cretonne_codegen::{timing, VERSION}; use docopt::Docopt; use std::io::{self, Write}; use std::process; diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index 8dbd4324fc..f078c7ee54 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -4,7 +4,7 @@ //! in graphviz format. use CommandResult; -use cretonne::cfg_printer::CFGPrinter; +use cretonne_codegen::cfg_printer::CFGPrinter; use cretonne_reader::parse_functions; use utils::read_to_string; diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index fe43be14ab..4d54cf118a 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -1,8 +1,8 @@ //! Utility functions. -use cretonne::isa; -use cretonne::isa::TargetIsa; -use cretonne::settings::{self, FlagsOrIsa}; +use cretonne_codegen::isa; +use cretonne_codegen::isa::TargetIsa; +use cretonne_codegen::settings::{self, FlagsOrIsa}; use cretonne_reader::{parse_options, Location}; use std::fs::File; use std::io::{self, Read}; diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 1cbe2c754b..5ef8e21f2e 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -4,9 +4,9 @@ //! Reads Wasm binary files, translates the functions' code to Cretonne IR. #![cfg_attr(feature = "cargo-clippy", allow(too_many_arguments, cyclomatic_complexity))] -use cretonne::Context; -use cretonne::print_errors::{pretty_error, pretty_verifier_error}; -use cretonne::settings::FlagsOrIsa; +use cretonne_codegen::Context; +use cretonne_codegen::print_errors::{pretty_error, pretty_verifier_error}; +use cretonne_codegen::settings::FlagsOrIsa; use cretonne_wasm::{translate_module, DummyEnvironment, ModuleEnvironment}; use std::error::Error; use std::fs::File; diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 572bf5de7b..ba3faf70b0 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -31,13 +31,13 @@ fi # Check if any Python files have changed since we last checked them. tsfile=$topdir/target/meta-checked if [ -f $tsfile ]; then - needcheck=$(find $topdir/lib/cretonne/meta -name '*.py' -newer $tsfile) + needcheck=$(find $topdir/lib/codegen/meta -name '*.py' -newer $tsfile) else needcheck=yes fi if [ -n "$needcheck" ]; then banner "$(python --version 2>&1), $(python3 --version 2>&1)" - $topdir/lib/cretonne/meta/check.sh + $topdir/lib/codegen/meta/check.sh touch $tsfile || echo no target directory fi diff --git a/lib/cretonne/Cargo.toml b/lib/codegen/Cargo.toml similarity index 86% rename from lib/cretonne/Cargo.toml rename to lib/codegen/Cargo.toml index d252fec291..6a5027f0f7 100644 --- a/lib/cretonne/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -1,6 +1,6 @@ [package] authors = ["The Cretonne Project Developers"] -name = "cretonne" +name = "cretonne-codegen" version = "0.4.4" description = "Low-level code generator library" license = "Apache-2.0" @@ -12,7 +12,7 @@ build = "build.rs" [dependencies] cretonne-entity = { path = "../entity", version = "0.4.4" } -# It is a goal of the cretonne crate to have minimal external dependencies. +# It is a goal of the cretonne-codegen crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be # accomodated in `tests`. diff --git a/lib/cretonne/README.md b/lib/codegen/README.md similarity index 100% rename from lib/cretonne/README.md rename to lib/codegen/README.md diff --git a/lib/cretonne/build.rs b/lib/codegen/build.rs similarity index 96% rename from lib/cretonne/build.rs rename to lib/codegen/build.rs index 8031a71c5a..69100ef00f 100644 --- a/lib/cretonne/build.rs +++ b/lib/codegen/build.rs @@ -1,7 +1,7 @@ // Build script. // -// This program is run by Cargo when building lib/cretonne. It is used to generate Rust code from -// the language definitions in the lib/cretonne/meta directory. +// This program is run by Cargo when building lib/codegen. It is used to generate Rust code from +// the language definitions in the lib/codegen/meta directory. // // Environment: // diff --git a/lib/cretonne/meta/base/__init__.py b/lib/codegen/meta/base/__init__.py similarity index 100% rename from lib/cretonne/meta/base/__init__.py rename to lib/codegen/meta/base/__init__.py diff --git a/lib/cretonne/meta/base/entities.py b/lib/codegen/meta/base/entities.py similarity index 100% rename from lib/cretonne/meta/base/entities.py rename to lib/codegen/meta/base/entities.py diff --git a/lib/cretonne/meta/base/formats.py b/lib/codegen/meta/base/formats.py similarity index 100% rename from lib/cretonne/meta/base/formats.py rename to lib/codegen/meta/base/formats.py diff --git a/lib/cretonne/meta/base/immediates.py b/lib/codegen/meta/base/immediates.py similarity index 100% rename from lib/cretonne/meta/base/immediates.py rename to lib/codegen/meta/base/immediates.py diff --git a/lib/cretonne/meta/base/instructions.py b/lib/codegen/meta/base/instructions.py similarity index 100% rename from lib/cretonne/meta/base/instructions.py rename to lib/codegen/meta/base/instructions.py diff --git a/lib/cretonne/meta/base/legalize.py b/lib/codegen/meta/base/legalize.py similarity index 100% rename from lib/cretonne/meta/base/legalize.py rename to lib/codegen/meta/base/legalize.py diff --git a/lib/cretonne/meta/base/predicates.py b/lib/codegen/meta/base/predicates.py similarity index 100% rename from lib/cretonne/meta/base/predicates.py rename to lib/codegen/meta/base/predicates.py diff --git a/lib/cretonne/meta/base/semantics.py b/lib/codegen/meta/base/semantics.py similarity index 100% rename from lib/cretonne/meta/base/semantics.py rename to lib/codegen/meta/base/semantics.py diff --git a/lib/cretonne/meta/base/settings.py b/lib/codegen/meta/base/settings.py similarity index 100% rename from lib/cretonne/meta/base/settings.py rename to lib/codegen/meta/base/settings.py diff --git a/lib/cretonne/meta/base/types.py b/lib/codegen/meta/base/types.py similarity index 100% rename from lib/cretonne/meta/base/types.py rename to lib/codegen/meta/base/types.py diff --git a/lib/cretonne/meta/build.py b/lib/codegen/meta/build.py similarity index 92% rename from lib/cretonne/meta/build.py rename to lib/codegen/meta/build.py index 8c83300604..d7ad4cdba2 100644 --- a/lib/cretonne/meta/build.py +++ b/lib/codegen/meta/build.py @@ -1,6 +1,6 @@ # Second-level build script. # -# This script is run from lib/cretonne/build.rs to generate Rust files. +# This script is run from lib/codegen/build.rs to generate Rust files. from __future__ import absolute_import import argparse diff --git a/lib/cretonne/meta/cdsl/__init__.py b/lib/codegen/meta/cdsl/__init__.py similarity index 100% rename from lib/cretonne/meta/cdsl/__init__.py rename to lib/codegen/meta/cdsl/__init__.py diff --git a/lib/cretonne/meta/cdsl/ast.py b/lib/codegen/meta/cdsl/ast.py similarity index 100% rename from lib/cretonne/meta/cdsl/ast.py rename to lib/codegen/meta/cdsl/ast.py diff --git a/lib/cretonne/meta/cdsl/formats.py b/lib/codegen/meta/cdsl/formats.py similarity index 100% rename from lib/cretonne/meta/cdsl/formats.py rename to lib/codegen/meta/cdsl/formats.py diff --git a/lib/cretonne/meta/cdsl/instructions.py b/lib/codegen/meta/cdsl/instructions.py similarity index 100% rename from lib/cretonne/meta/cdsl/instructions.py rename to lib/codegen/meta/cdsl/instructions.py diff --git a/lib/cretonne/meta/cdsl/isa.py b/lib/codegen/meta/cdsl/isa.py similarity index 100% rename from lib/cretonne/meta/cdsl/isa.py rename to lib/codegen/meta/cdsl/isa.py diff --git a/lib/cretonne/meta/cdsl/operands.py b/lib/codegen/meta/cdsl/operands.py similarity index 100% rename from lib/cretonne/meta/cdsl/operands.py rename to lib/codegen/meta/cdsl/operands.py diff --git a/lib/cretonne/meta/cdsl/predicates.py b/lib/codegen/meta/cdsl/predicates.py similarity index 100% rename from lib/cretonne/meta/cdsl/predicates.py rename to lib/codegen/meta/cdsl/predicates.py diff --git a/lib/cretonne/meta/cdsl/registers.py b/lib/codegen/meta/cdsl/registers.py similarity index 100% rename from lib/cretonne/meta/cdsl/registers.py rename to lib/codegen/meta/cdsl/registers.py diff --git a/lib/cretonne/meta/cdsl/settings.py b/lib/codegen/meta/cdsl/settings.py similarity index 100% rename from lib/cretonne/meta/cdsl/settings.py rename to lib/codegen/meta/cdsl/settings.py diff --git a/lib/cretonne/meta/cdsl/test_ast.py b/lib/codegen/meta/cdsl/test_ast.py similarity index 100% rename from lib/cretonne/meta/cdsl/test_ast.py rename to lib/codegen/meta/cdsl/test_ast.py diff --git a/lib/cretonne/meta/cdsl/test_package.py b/lib/codegen/meta/cdsl/test_package.py similarity index 100% rename from lib/cretonne/meta/cdsl/test_package.py rename to lib/codegen/meta/cdsl/test_package.py diff --git a/lib/cretonne/meta/cdsl/test_ti.py b/lib/codegen/meta/cdsl/test_ti.py similarity index 100% rename from lib/cretonne/meta/cdsl/test_ti.py rename to lib/codegen/meta/cdsl/test_ti.py diff --git a/lib/cretonne/meta/cdsl/test_typevar.py b/lib/codegen/meta/cdsl/test_typevar.py similarity index 100% rename from lib/cretonne/meta/cdsl/test_typevar.py rename to lib/codegen/meta/cdsl/test_typevar.py diff --git a/lib/cretonne/meta/cdsl/test_xform.py b/lib/codegen/meta/cdsl/test_xform.py similarity index 100% rename from lib/cretonne/meta/cdsl/test_xform.py rename to lib/codegen/meta/cdsl/test_xform.py diff --git a/lib/cretonne/meta/cdsl/ti.py b/lib/codegen/meta/cdsl/ti.py similarity index 100% rename from lib/cretonne/meta/cdsl/ti.py rename to lib/codegen/meta/cdsl/ti.py diff --git a/lib/cretonne/meta/cdsl/types.py b/lib/codegen/meta/cdsl/types.py similarity index 100% rename from lib/cretonne/meta/cdsl/types.py rename to lib/codegen/meta/cdsl/types.py diff --git a/lib/cretonne/meta/cdsl/typevar.py b/lib/codegen/meta/cdsl/typevar.py similarity index 100% rename from lib/cretonne/meta/cdsl/typevar.py rename to lib/codegen/meta/cdsl/typevar.py diff --git a/lib/cretonne/meta/cdsl/xform.py b/lib/codegen/meta/cdsl/xform.py similarity index 100% rename from lib/cretonne/meta/cdsl/xform.py rename to lib/codegen/meta/cdsl/xform.py diff --git a/lib/cretonne/meta/check.sh b/lib/codegen/meta/check.sh similarity index 100% rename from lib/cretonne/meta/check.sh rename to lib/codegen/meta/check.sh diff --git a/lib/cretonne/meta/constant_hash.py b/lib/codegen/meta/constant_hash.py similarity index 100% rename from lib/cretonne/meta/constant_hash.py rename to lib/codegen/meta/constant_hash.py diff --git a/lib/cretonne/meta/gen_binemit.py b/lib/codegen/meta/gen_binemit.py similarity index 100% rename from lib/cretonne/meta/gen_binemit.py rename to lib/codegen/meta/gen_binemit.py diff --git a/lib/cretonne/meta/gen_build_deps.py b/lib/codegen/meta/gen_build_deps.py similarity index 94% rename from lib/cretonne/meta/gen_build_deps.py rename to lib/codegen/meta/gen_build_deps.py index 5e1419284c..03d9ee72d0 100644 --- a/lib/cretonne/meta/gen_build_deps.py +++ b/lib/codegen/meta/gen_build_deps.py @@ -1,7 +1,7 @@ """ Generate build dependencies for Cargo. -The `build.py` script is invoked by cargo when building lib/cretonne to +The `build.py` script is invoked by cargo when building lib/codegen to generate Rust code from the instruction descriptions. Cargo needs to know when it is necessary to rerun the build script. diff --git a/lib/cretonne/meta/gen_encoding.py b/lib/codegen/meta/gen_encoding.py similarity index 100% rename from lib/cretonne/meta/gen_encoding.py rename to lib/codegen/meta/gen_encoding.py diff --git a/lib/cretonne/meta/gen_instr.py b/lib/codegen/meta/gen_instr.py similarity index 100% rename from lib/cretonne/meta/gen_instr.py rename to lib/codegen/meta/gen_instr.py diff --git a/lib/cretonne/meta/gen_legalizer.py b/lib/codegen/meta/gen_legalizer.py similarity index 100% rename from lib/cretonne/meta/gen_legalizer.py rename to lib/codegen/meta/gen_legalizer.py diff --git a/lib/cretonne/meta/gen_registers.py b/lib/codegen/meta/gen_registers.py similarity index 100% rename from lib/cretonne/meta/gen_registers.py rename to lib/codegen/meta/gen_registers.py diff --git a/lib/cretonne/meta/gen_settings.py b/lib/codegen/meta/gen_settings.py similarity index 100% rename from lib/cretonne/meta/gen_settings.py rename to lib/codegen/meta/gen_settings.py diff --git a/lib/cretonne/meta/gen_types.py b/lib/codegen/meta/gen_types.py similarity index 95% rename from lib/cretonne/meta/gen_types.py rename to lib/codegen/meta/gen_types.py index bc912e78f7..4199efa51f 100644 --- a/lib/cretonne/meta/gen_types.py +++ b/lib/codegen/meta/gen_types.py @@ -2,7 +2,7 @@ Generate sources with type info. This generates a `types.rs` file which is included in -`lib/cretonne/ir/types.rs`. The file provides constant definitions for the most +`lib/codegen/ir/types.rs`. The file provides constant definitions for the most commonly used types, including all of the scalar types. This ensures that Python and Rust use the same type numbering. diff --git a/lib/cretonne/meta/isa/__init__.py b/lib/codegen/meta/isa/__init__.py similarity index 100% rename from lib/cretonne/meta/isa/__init__.py rename to lib/codegen/meta/isa/__init__.py diff --git a/lib/cretonne/meta/isa/arm32/__init__.py b/lib/codegen/meta/isa/arm32/__init__.py similarity index 100% rename from lib/cretonne/meta/isa/arm32/__init__.py rename to lib/codegen/meta/isa/arm32/__init__.py diff --git a/lib/cretonne/meta/isa/arm32/defs.py b/lib/codegen/meta/isa/arm32/defs.py similarity index 100% rename from lib/cretonne/meta/isa/arm32/defs.py rename to lib/codegen/meta/isa/arm32/defs.py diff --git a/lib/cretonne/meta/isa/arm32/registers.py b/lib/codegen/meta/isa/arm32/registers.py similarity index 100% rename from lib/cretonne/meta/isa/arm32/registers.py rename to lib/codegen/meta/isa/arm32/registers.py diff --git a/lib/cretonne/meta/isa/arm32/settings.py b/lib/codegen/meta/isa/arm32/settings.py similarity index 100% rename from lib/cretonne/meta/isa/arm32/settings.py rename to lib/codegen/meta/isa/arm32/settings.py diff --git a/lib/cretonne/meta/isa/arm64/__init__.py b/lib/codegen/meta/isa/arm64/__init__.py similarity index 100% rename from lib/cretonne/meta/isa/arm64/__init__.py rename to lib/codegen/meta/isa/arm64/__init__.py diff --git a/lib/cretonne/meta/isa/arm64/defs.py b/lib/codegen/meta/isa/arm64/defs.py similarity index 100% rename from lib/cretonne/meta/isa/arm64/defs.py rename to lib/codegen/meta/isa/arm64/defs.py diff --git a/lib/cretonne/meta/isa/arm64/registers.py b/lib/codegen/meta/isa/arm64/registers.py similarity index 100% rename from lib/cretonne/meta/isa/arm64/registers.py rename to lib/codegen/meta/isa/arm64/registers.py diff --git a/lib/cretonne/meta/isa/arm64/settings.py b/lib/codegen/meta/isa/arm64/settings.py similarity index 100% rename from lib/cretonne/meta/isa/arm64/settings.py rename to lib/codegen/meta/isa/arm64/settings.py diff --git a/lib/cretonne/meta/isa/riscv/__init__.py b/lib/codegen/meta/isa/riscv/__init__.py similarity index 100% rename from lib/cretonne/meta/isa/riscv/__init__.py rename to lib/codegen/meta/isa/riscv/__init__.py diff --git a/lib/cretonne/meta/isa/riscv/defs.py b/lib/codegen/meta/isa/riscv/defs.py similarity index 100% rename from lib/cretonne/meta/isa/riscv/defs.py rename to lib/codegen/meta/isa/riscv/defs.py diff --git a/lib/cretonne/meta/isa/riscv/encodings.py b/lib/codegen/meta/isa/riscv/encodings.py similarity index 100% rename from lib/cretonne/meta/isa/riscv/encodings.py rename to lib/codegen/meta/isa/riscv/encodings.py diff --git a/lib/cretonne/meta/isa/riscv/recipes.py b/lib/codegen/meta/isa/riscv/recipes.py similarity index 100% rename from lib/cretonne/meta/isa/riscv/recipes.py rename to lib/codegen/meta/isa/riscv/recipes.py diff --git a/lib/cretonne/meta/isa/riscv/registers.py b/lib/codegen/meta/isa/riscv/registers.py similarity index 100% rename from lib/cretonne/meta/isa/riscv/registers.py rename to lib/codegen/meta/isa/riscv/registers.py diff --git a/lib/cretonne/meta/isa/riscv/settings.py b/lib/codegen/meta/isa/riscv/settings.py similarity index 100% rename from lib/cretonne/meta/isa/riscv/settings.py rename to lib/codegen/meta/isa/riscv/settings.py diff --git a/lib/cretonne/meta/isa/x86/__init__.py b/lib/codegen/meta/isa/x86/__init__.py similarity index 100% rename from lib/cretonne/meta/isa/x86/__init__.py rename to lib/codegen/meta/isa/x86/__init__.py diff --git a/lib/cretonne/meta/isa/x86/defs.py b/lib/codegen/meta/isa/x86/defs.py similarity index 100% rename from lib/cretonne/meta/isa/x86/defs.py rename to lib/codegen/meta/isa/x86/defs.py diff --git a/lib/cretonne/meta/isa/x86/encodings.py b/lib/codegen/meta/isa/x86/encodings.py similarity index 100% rename from lib/cretonne/meta/isa/x86/encodings.py rename to lib/codegen/meta/isa/x86/encodings.py diff --git a/lib/cretonne/meta/isa/x86/instructions.py b/lib/codegen/meta/isa/x86/instructions.py similarity index 100% rename from lib/cretonne/meta/isa/x86/instructions.py rename to lib/codegen/meta/isa/x86/instructions.py diff --git a/lib/cretonne/meta/isa/x86/legalize.py b/lib/codegen/meta/isa/x86/legalize.py similarity index 100% rename from lib/cretonne/meta/isa/x86/legalize.py rename to lib/codegen/meta/isa/x86/legalize.py diff --git a/lib/cretonne/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py similarity index 100% rename from lib/cretonne/meta/isa/x86/recipes.py rename to lib/codegen/meta/isa/x86/recipes.py diff --git a/lib/cretonne/meta/isa/x86/registers.py b/lib/codegen/meta/isa/x86/registers.py similarity index 100% rename from lib/cretonne/meta/isa/x86/registers.py rename to lib/codegen/meta/isa/x86/registers.py diff --git a/lib/cretonne/meta/isa/x86/settings.py b/lib/codegen/meta/isa/x86/settings.py similarity index 100% rename from lib/cretonne/meta/isa/x86/settings.py rename to lib/codegen/meta/isa/x86/settings.py diff --git a/lib/cretonne/meta/mypy.ini b/lib/codegen/meta/mypy.ini similarity index 100% rename from lib/cretonne/meta/mypy.ini rename to lib/codegen/meta/mypy.ini diff --git a/lib/cretonne/meta/semantics/__init__.py b/lib/codegen/meta/semantics/__init__.py similarity index 100% rename from lib/cretonne/meta/semantics/__init__.py rename to lib/codegen/meta/semantics/__init__.py diff --git a/lib/cretonne/meta/semantics/elaborate.py b/lib/codegen/meta/semantics/elaborate.py similarity index 100% rename from lib/cretonne/meta/semantics/elaborate.py rename to lib/codegen/meta/semantics/elaborate.py diff --git a/lib/cretonne/meta/semantics/macros.py b/lib/codegen/meta/semantics/macros.py similarity index 100% rename from lib/cretonne/meta/semantics/macros.py rename to lib/codegen/meta/semantics/macros.py diff --git a/lib/cretonne/meta/semantics/primitives.py b/lib/codegen/meta/semantics/primitives.py similarity index 100% rename from lib/cretonne/meta/semantics/primitives.py rename to lib/codegen/meta/semantics/primitives.py diff --git a/lib/cretonne/meta/semantics/smtlib.py b/lib/codegen/meta/semantics/smtlib.py similarity index 100% rename from lib/cretonne/meta/semantics/smtlib.py rename to lib/codegen/meta/semantics/smtlib.py diff --git a/lib/cretonne/meta/semantics/test_elaborate.py b/lib/codegen/meta/semantics/test_elaborate.py similarity index 100% rename from lib/cretonne/meta/semantics/test_elaborate.py rename to lib/codegen/meta/semantics/test_elaborate.py diff --git a/lib/cretonne/meta/srcgen.py b/lib/codegen/meta/srcgen.py similarity index 100% rename from lib/cretonne/meta/srcgen.py rename to lib/codegen/meta/srcgen.py diff --git a/lib/cretonne/meta/stubs/z3/__init__.pyi b/lib/codegen/meta/stubs/z3/__init__.pyi similarity index 100% rename from lib/cretonne/meta/stubs/z3/__init__.pyi rename to lib/codegen/meta/stubs/z3/__init__.pyi diff --git a/lib/cretonne/meta/stubs/z3/z3core.pyi b/lib/codegen/meta/stubs/z3/z3core.pyi similarity index 100% rename from lib/cretonne/meta/stubs/z3/z3core.pyi rename to lib/codegen/meta/stubs/z3/z3core.pyi diff --git a/lib/cretonne/meta/stubs/z3/z3types.pyi b/lib/codegen/meta/stubs/z3/z3types.pyi similarity index 100% rename from lib/cretonne/meta/stubs/z3/z3types.pyi rename to lib/codegen/meta/stubs/z3/z3types.pyi diff --git a/lib/cretonne/meta/test_constant_hash.py b/lib/codegen/meta/test_constant_hash.py similarity index 100% rename from lib/cretonne/meta/test_constant_hash.py rename to lib/codegen/meta/test_constant_hash.py diff --git a/lib/cretonne/meta/test_gen_legalizer.py b/lib/codegen/meta/test_gen_legalizer.py similarity index 100% rename from lib/cretonne/meta/test_gen_legalizer.py rename to lib/codegen/meta/test_gen_legalizer.py diff --git a/lib/cretonne/meta/test_srcgen.py b/lib/codegen/meta/test_srcgen.py similarity index 100% rename from lib/cretonne/meta/test_srcgen.py rename to lib/codegen/meta/test_srcgen.py diff --git a/lib/cretonne/meta/unique_table.py b/lib/codegen/meta/unique_table.py similarity index 100% rename from lib/cretonne/meta/unique_table.py rename to lib/codegen/meta/unique_table.py diff --git a/lib/cretonne/src/abi.rs b/lib/codegen/src/abi.rs similarity index 100% rename from lib/cretonne/src/abi.rs rename to lib/codegen/src/abi.rs diff --git a/lib/cretonne/src/bforest/map.rs b/lib/codegen/src/bforest/map.rs similarity index 100% rename from lib/cretonne/src/bforest/map.rs rename to lib/codegen/src/bforest/map.rs diff --git a/lib/cretonne/src/bforest/mod.rs b/lib/codegen/src/bforest/mod.rs similarity index 100% rename from lib/cretonne/src/bforest/mod.rs rename to lib/codegen/src/bforest/mod.rs diff --git a/lib/cretonne/src/bforest/node.rs b/lib/codegen/src/bforest/node.rs similarity index 100% rename from lib/cretonne/src/bforest/node.rs rename to lib/codegen/src/bforest/node.rs diff --git a/lib/cretonne/src/bforest/path.rs b/lib/codegen/src/bforest/path.rs similarity index 100% rename from lib/cretonne/src/bforest/path.rs rename to lib/codegen/src/bforest/path.rs diff --git a/lib/cretonne/src/bforest/pool.rs b/lib/codegen/src/bforest/pool.rs similarity index 100% rename from lib/cretonne/src/bforest/pool.rs rename to lib/codegen/src/bforest/pool.rs diff --git a/lib/cretonne/src/bforest/set.rs b/lib/codegen/src/bforest/set.rs similarity index 100% rename from lib/cretonne/src/bforest/set.rs rename to lib/codegen/src/bforest/set.rs diff --git a/lib/cretonne/src/binemit/memorysink.rs b/lib/codegen/src/binemit/memorysink.rs similarity index 100% rename from lib/cretonne/src/binemit/memorysink.rs rename to lib/codegen/src/binemit/memorysink.rs diff --git a/lib/cretonne/src/binemit/mod.rs b/lib/codegen/src/binemit/mod.rs similarity index 100% rename from lib/cretonne/src/binemit/mod.rs rename to lib/codegen/src/binemit/mod.rs diff --git a/lib/cretonne/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs similarity index 100% rename from lib/cretonne/src/binemit/relaxation.rs rename to lib/codegen/src/binemit/relaxation.rs diff --git a/lib/cretonne/src/bitset.rs b/lib/codegen/src/bitset.rs similarity index 100% rename from lib/cretonne/src/bitset.rs rename to lib/codegen/src/bitset.rs diff --git a/lib/cretonne/src/cfg_printer.rs b/lib/codegen/src/cfg_printer.rs similarity index 100% rename from lib/cretonne/src/cfg_printer.rs rename to lib/codegen/src/cfg_printer.rs diff --git a/lib/cretonne/src/constant_hash.rs b/lib/codegen/src/constant_hash.rs similarity index 93% rename from lib/cretonne/src/constant_hash.rs rename to lib/codegen/src/constant_hash.rs index 0dd2895ad3..2bfeaf58c3 100644 --- a/lib/cretonne/src/constant_hash.rs +++ b/lib/codegen/src/constant_hash.rs @@ -1,6 +1,6 @@ //! Runtime support for precomputed constant hash tables. //! -//! The `lib/cretonne/meta/constant_hash.py` Python module can generate constant hash tables using +//! The `lib/codegen/meta/constant_hash.py` Python module can generate constant hash tables using //! open addressing and quadratic probing. The hash tables are arrays that are guaranteed to: //! //! - Have a power-of-two size. @@ -56,7 +56,7 @@ pub fn probe + ?Sized>( } /// A primitive hash function for matching opcodes. -/// Must match `lib/cretonne/meta/constant_hash.py`. +/// Must match `lib/codegen/meta/constant_hash.py`. pub fn simple_hash(s: &str) -> usize { let mut h: u32 = 5381; for c in s.chars() { diff --git a/lib/cretonne/src/context.rs b/lib/codegen/src/context.rs similarity index 100% rename from lib/cretonne/src/context.rs rename to lib/codegen/src/context.rs diff --git a/lib/cretonne/src/cursor.rs b/lib/codegen/src/cursor.rs similarity index 93% rename from lib/cretonne/src/cursor.rs rename to lib/codegen/src/cursor.rs index 865530341d..6842d8b348 100644 --- a/lib/cretonne/src/cursor.rs +++ b/lib/codegen/src/cursor.rs @@ -46,8 +46,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne::ir::{Function, Ebb, SourceLoc}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// # use cretonne_codegen::ir::{Function, Ebb, SourceLoc}; + /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, srcloc: SourceLoc) { /// let mut pos = FuncCursor::new(func).with_srcloc(srcloc); /// @@ -76,8 +76,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; + /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, inst: Inst) { /// let mut pos = FuncCursor::new(func).at_inst(inst); /// @@ -99,8 +99,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; + /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { /// let mut pos = FuncCursor::new(func).at_first_insertion_point(ebb); /// @@ -120,8 +120,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; + /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { /// let mut pos = FuncCursor::new(func).at_first_inst(ebb); /// @@ -141,8 +141,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; + /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { /// let mut pos = FuncCursor::new(func).at_last_inst(ebb); /// @@ -162,8 +162,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; + /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, inst: Inst) { /// let mut pos = FuncCursor::new(func).after_inst(inst); /// @@ -183,8 +183,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; + /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { /// let mut pos = FuncCursor::new(func).at_top(ebb); /// @@ -204,8 +204,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne::ir::{Function, Ebb, Inst}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; + /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { /// let mut pos = FuncCursor::new(func).at_bottom(ebb); /// @@ -309,8 +309,8 @@ pub trait Cursor { /// The `next_ebb()` method is intended for iterating over the EBBs in layout order: /// /// ``` - /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// # use cretonne_codegen::ir::{Function, Ebb}; + /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function) { /// let mut cursor = FuncCursor::new(func); /// while let Some(ebb) = cursor.next_ebb() { @@ -342,8 +342,8 @@ pub trait Cursor { /// The `prev_ebb()` method is intended for iterating over the EBBs in backwards layout order: /// /// ``` - /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// # use cretonne_codegen::ir::{Function, Ebb}; + /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function) { /// let mut cursor = FuncCursor::new(func); /// while let Some(ebb) = cursor.prev_ebb() { @@ -379,8 +379,8 @@ pub trait Cursor { /// this: /// /// ``` - /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// # use cretonne_codegen::ir::{Function, Ebb}; + /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_ebb(func: &mut Function, ebb: Ebb) { /// let mut cursor = FuncCursor::new(func).at_top(ebb); /// while let Some(inst) = cursor.next_inst() { @@ -393,8 +393,8 @@ pub trait Cursor { /// Iterating over all the instructions in a function looks like this: /// /// ``` - /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// # use cretonne_codegen::ir::{Function, Ebb}; + /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function) { /// let mut cursor = FuncCursor::new(func); /// while let Some(ebb) = cursor.next_ebb() { @@ -447,8 +447,8 @@ pub trait Cursor { /// EBB like this: /// /// ``` - /// # use cretonne::ir::{Function, Ebb}; - /// # use cretonne::cursor::{Cursor, FuncCursor}; + /// # use cretonne_codegen::ir::{Function, Ebb}; + /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_ebb(func: &mut Function, ebb: Ebb) { /// let mut cursor = FuncCursor::new(func).at_bottom(ebb); /// while let Some(inst) = cursor.prev_inst() { diff --git a/lib/cretonne/src/dbg.rs b/lib/codegen/src/dbg.rs similarity index 100% rename from lib/cretonne/src/dbg.rs rename to lib/codegen/src/dbg.rs diff --git a/lib/cretonne/src/dce.rs b/lib/codegen/src/dce.rs similarity index 100% rename from lib/cretonne/src/dce.rs rename to lib/codegen/src/dce.rs diff --git a/lib/cretonne/src/divconst_magic_numbers.rs b/lib/codegen/src/divconst_magic_numbers.rs similarity index 100% rename from lib/cretonne/src/divconst_magic_numbers.rs rename to lib/codegen/src/divconst_magic_numbers.rs diff --git a/lib/cretonne/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs similarity index 100% rename from lib/cretonne/src/dominator_tree.rs rename to lib/codegen/src/dominator_tree.rs diff --git a/lib/cretonne/src/flowgraph.rs b/lib/codegen/src/flowgraph.rs similarity index 100% rename from lib/cretonne/src/flowgraph.rs rename to lib/codegen/src/flowgraph.rs diff --git a/lib/cretonne/src/ir/builder.rs b/lib/codegen/src/ir/builder.rs similarity index 99% rename from lib/cretonne/src/ir/builder.rs rename to lib/codegen/src/ir/builder.rs index 34f56b9439..e27cda0bcd 100644 --- a/lib/cretonne/src/ir/builder.rs +++ b/lib/codegen/src/ir/builder.rs @@ -32,7 +32,7 @@ pub trait InstBuilderBase<'f>: Sized { fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'f mut DataFlowGraph); } -// Include trait code generated by `lib/cretonne/meta/gen_instr.py`. +// Include trait code generated by `lib/codegen/meta/gen_instr.py`. // // This file defines the `InstBuilder` trait as an extension of `InstBuilderBase` with methods per // instruction format and per opcode. diff --git a/lib/cretonne/src/ir/condcodes.rs b/lib/codegen/src/ir/condcodes.rs similarity index 100% rename from lib/cretonne/src/ir/condcodes.rs rename to lib/codegen/src/ir/condcodes.rs diff --git a/lib/cretonne/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs similarity index 100% rename from lib/cretonne/src/ir/dfg.rs rename to lib/codegen/src/ir/dfg.rs diff --git a/lib/cretonne/src/ir/entities.rs b/lib/codegen/src/ir/entities.rs similarity index 100% rename from lib/cretonne/src/ir/entities.rs rename to lib/codegen/src/ir/entities.rs diff --git a/lib/cretonne/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs similarity index 100% rename from lib/cretonne/src/ir/extfunc.rs rename to lib/codegen/src/ir/extfunc.rs diff --git a/lib/cretonne/src/ir/extname.rs b/lib/codegen/src/ir/extname.rs similarity index 96% rename from lib/cretonne/src/ir/extname.rs rename to lib/codegen/src/ir/extname.rs index b37e40086c..f3a69f7e7a 100644 --- a/lib/cretonne/src/ir/extname.rs +++ b/lib/codegen/src/ir/extname.rs @@ -16,7 +16,8 @@ const TESTCASE_NAME_LENGTH: usize = 16; /// to keep track of a sy mbol table. /// /// External names are primarily used as keys by code using Cretonne to map -/// from a `cretonne::ir::FuncRef` or similar to additional associated data. +/// from a `cretonne_codegen::ir::FuncRef` or similar to additional associated +/// data. /// /// External names can also serve as a primitive testing and debugging tool. /// In particular, many `.cton` test files use function names to identify @@ -50,7 +51,7 @@ impl ExternalName { /// # Examples /// /// ```rust - /// # use cretonne::ir::ExternalName; + /// # use cretonne_codegen::ir::ExternalName; /// // Create `ExternalName` from a string. /// let name = ExternalName::testcase("hello"); /// assert_eq!(name.to_string(), "%hello"); @@ -71,7 +72,7 @@ impl ExternalName { /// /// # Examples /// ```rust - /// # use cretonne::ir::ExternalName; + /// # use cretonne_codegen::ir::ExternalName; /// // Create `ExternalName` from integer indicies /// let name = ExternalName::user(123, 456); /// assert_eq!(name.to_string(), "u123:456"); diff --git a/lib/cretonne/src/ir/function.rs b/lib/codegen/src/ir/function.rs similarity index 100% rename from lib/cretonne/src/ir/function.rs rename to lib/codegen/src/ir/function.rs diff --git a/lib/cretonne/src/ir/globalvar.rs b/lib/codegen/src/ir/globalvar.rs similarity index 100% rename from lib/cretonne/src/ir/globalvar.rs rename to lib/codegen/src/ir/globalvar.rs diff --git a/lib/cretonne/src/ir/heap.rs b/lib/codegen/src/ir/heap.rs similarity index 100% rename from lib/cretonne/src/ir/heap.rs rename to lib/codegen/src/ir/heap.rs diff --git a/lib/cretonne/src/ir/immediates.rs b/lib/codegen/src/ir/immediates.rs similarity index 100% rename from lib/cretonne/src/ir/immediates.rs rename to lib/codegen/src/ir/immediates.rs diff --git a/lib/cretonne/src/ir/instructions.rs b/lib/codegen/src/ir/instructions.rs similarity index 99% rename from lib/cretonne/src/ir/instructions.rs rename to lib/codegen/src/ir/instructions.rs index aa6a35df3f..ff415af22b 100644 --- a/lib/cretonne/src/ir/instructions.rs +++ b/lib/codegen/src/ir/instructions.rs @@ -28,7 +28,7 @@ pub type ValueList = entity::EntityList; /// Memory pool for holding value lists. See `ValueList`. pub type ValueListPool = entity::ListPool; -// Include code generated by `lib/cretonne/meta/gen_instr.py`. This file contains: +// Include code generated by `lib/codegen/meta/gen_instr.py`. This file contains: // // - The `pub enum InstructionFormat` enum with all the instruction formats. // - The `pub enum InstructionData` enum with all the instruction data fields. diff --git a/lib/cretonne/src/ir/jumptable.rs b/lib/codegen/src/ir/jumptable.rs similarity index 100% rename from lib/cretonne/src/ir/jumptable.rs rename to lib/codegen/src/ir/jumptable.rs diff --git a/lib/cretonne/src/ir/layout.rs b/lib/codegen/src/ir/layout.rs similarity index 100% rename from lib/cretonne/src/ir/layout.rs rename to lib/codegen/src/ir/layout.rs diff --git a/lib/cretonne/src/ir/libcall.rs b/lib/codegen/src/ir/libcall.rs similarity index 100% rename from lib/cretonne/src/ir/libcall.rs rename to lib/codegen/src/ir/libcall.rs diff --git a/lib/cretonne/src/ir/memflags.rs b/lib/codegen/src/ir/memflags.rs similarity index 100% rename from lib/cretonne/src/ir/memflags.rs rename to lib/codegen/src/ir/memflags.rs diff --git a/lib/cretonne/src/ir/mod.rs b/lib/codegen/src/ir/mod.rs similarity index 100% rename from lib/cretonne/src/ir/mod.rs rename to lib/codegen/src/ir/mod.rs diff --git a/lib/cretonne/src/ir/progpoint.rs b/lib/codegen/src/ir/progpoint.rs similarity index 100% rename from lib/cretonne/src/ir/progpoint.rs rename to lib/codegen/src/ir/progpoint.rs diff --git a/lib/cretonne/src/ir/sourceloc.rs b/lib/codegen/src/ir/sourceloc.rs similarity index 100% rename from lib/cretonne/src/ir/sourceloc.rs rename to lib/codegen/src/ir/sourceloc.rs diff --git a/lib/cretonne/src/ir/stackslot.rs b/lib/codegen/src/ir/stackslot.rs similarity index 100% rename from lib/cretonne/src/ir/stackslot.rs rename to lib/codegen/src/ir/stackslot.rs diff --git a/lib/cretonne/src/ir/trapcode.rs b/lib/codegen/src/ir/trapcode.rs similarity index 100% rename from lib/cretonne/src/ir/trapcode.rs rename to lib/codegen/src/ir/trapcode.rs diff --git a/lib/cretonne/src/ir/types.rs b/lib/codegen/src/ir/types.rs similarity index 99% rename from lib/cretonne/src/ir/types.rs rename to lib/codegen/src/ir/types.rs index 496312b1c4..3dc71454de 100644 --- a/lib/cretonne/src/ir/types.rs +++ b/lib/codegen/src/ir/types.rs @@ -30,7 +30,7 @@ const LANE_BASE: u8 = 0x70; /// Start of the 2-lane vector types. const VECTOR_BASE: u8 = LANE_BASE + 16; -// Include code generated by `lib/cretonne/meta/gen_types.py`. This file contains constant +// Include code generated by `lib/codegen/meta/gen_types.py`. This file contains constant // definitions for all the scalar types as well as common vector types for 64, 128, 256, and // 512-bit SIMD vectors. include!(concat!(env!("OUT_DIR"), "/types.rs")); diff --git a/lib/cretonne/src/ir/valueloc.rs b/lib/codegen/src/ir/valueloc.rs similarity index 100% rename from lib/cretonne/src/ir/valueloc.rs rename to lib/codegen/src/ir/valueloc.rs diff --git a/lib/cretonne/src/isa/arm32/abi.rs b/lib/codegen/src/isa/arm32/abi.rs similarity index 100% rename from lib/cretonne/src/isa/arm32/abi.rs rename to lib/codegen/src/isa/arm32/abi.rs diff --git a/lib/cretonne/src/isa/arm32/binemit.rs b/lib/codegen/src/isa/arm32/binemit.rs similarity index 100% rename from lib/cretonne/src/isa/arm32/binemit.rs rename to lib/codegen/src/isa/arm32/binemit.rs diff --git a/lib/cretonne/src/isa/arm32/enc_tables.rs b/lib/codegen/src/isa/arm32/enc_tables.rs similarity index 100% rename from lib/cretonne/src/isa/arm32/enc_tables.rs rename to lib/codegen/src/isa/arm32/enc_tables.rs diff --git a/lib/cretonne/src/isa/arm32/mod.rs b/lib/codegen/src/isa/arm32/mod.rs similarity index 100% rename from lib/cretonne/src/isa/arm32/mod.rs rename to lib/codegen/src/isa/arm32/mod.rs diff --git a/lib/cretonne/src/isa/arm32/registers.rs b/lib/codegen/src/isa/arm32/registers.rs similarity index 100% rename from lib/cretonne/src/isa/arm32/registers.rs rename to lib/codegen/src/isa/arm32/registers.rs diff --git a/lib/cretonne/src/isa/arm32/settings.rs b/lib/codegen/src/isa/arm32/settings.rs similarity index 58% rename from lib/cretonne/src/isa/arm32/settings.rs rename to lib/codegen/src/isa/arm32/settings.rs index b502deee40..ed7dd01281 100644 --- a/lib/cretonne/src/isa/arm32/settings.rs +++ b/lib/codegen/src/isa/arm32/settings.rs @@ -3,7 +3,7 @@ use settings::{self, detail, Builder}; use std::fmt; -// Include code generated by `lib/cretonne/meta/gen_settings.py`. This file contains a public +// Include code generated by `lib/codegen/meta/gen_settings.py`. This file contains a public // `Flags` struct with an impl for all of the settings defined in -// `lib/cretonne/meta/isa/arm32/settings.py`. +// `lib/codegen/meta/isa/arm32/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings-arm32.rs")); diff --git a/lib/cretonne/src/isa/arm64/abi.rs b/lib/codegen/src/isa/arm64/abi.rs similarity index 100% rename from lib/cretonne/src/isa/arm64/abi.rs rename to lib/codegen/src/isa/arm64/abi.rs diff --git a/lib/cretonne/src/isa/arm64/binemit.rs b/lib/codegen/src/isa/arm64/binemit.rs similarity index 100% rename from lib/cretonne/src/isa/arm64/binemit.rs rename to lib/codegen/src/isa/arm64/binemit.rs diff --git a/lib/cretonne/src/isa/arm64/enc_tables.rs b/lib/codegen/src/isa/arm64/enc_tables.rs similarity index 100% rename from lib/cretonne/src/isa/arm64/enc_tables.rs rename to lib/codegen/src/isa/arm64/enc_tables.rs diff --git a/lib/cretonne/src/isa/arm64/mod.rs b/lib/codegen/src/isa/arm64/mod.rs similarity index 100% rename from lib/cretonne/src/isa/arm64/mod.rs rename to lib/codegen/src/isa/arm64/mod.rs diff --git a/lib/cretonne/src/isa/arm64/registers.rs b/lib/codegen/src/isa/arm64/registers.rs similarity index 100% rename from lib/cretonne/src/isa/arm64/registers.rs rename to lib/codegen/src/isa/arm64/registers.rs diff --git a/lib/cretonne/src/isa/arm64/settings.rs b/lib/codegen/src/isa/arm64/settings.rs similarity index 58% rename from lib/cretonne/src/isa/arm64/settings.rs rename to lib/codegen/src/isa/arm64/settings.rs index b575168361..cdaa129b2d 100644 --- a/lib/cretonne/src/isa/arm64/settings.rs +++ b/lib/codegen/src/isa/arm64/settings.rs @@ -3,7 +3,7 @@ use settings::{self, detail, Builder}; use std::fmt; -// Include code generated by `lib/cretonne/meta/gen_settings.py`. This file contains a public +// Include code generated by `lib/codegen/meta/gen_settings.py`. This file contains a public // `Flags` struct with an impl for all of the settings defined in -// `lib/cretonne/meta/isa/arm64/settings.py`. +// `lib/codegen/meta/isa/arm64/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings-arm64.rs")); diff --git a/lib/cretonne/src/isa/constraints.rs b/lib/codegen/src/isa/constraints.rs similarity index 100% rename from lib/cretonne/src/isa/constraints.rs rename to lib/codegen/src/isa/constraints.rs diff --git a/lib/cretonne/src/isa/enc_tables.rs b/lib/codegen/src/isa/enc_tables.rs similarity index 99% rename from lib/cretonne/src/isa/enc_tables.rs rename to lib/codegen/src/isa/enc_tables.rs index b4c9bd86ba..b3deb85de1 100644 --- a/lib/cretonne/src/isa/enc_tables.rs +++ b/lib/codegen/src/isa/enc_tables.rs @@ -1,7 +1,7 @@ //! Support types for generated encoding tables. //! //! This module contains types and functions for working with the encoding tables generated by -//! `lib/cretonne/meta/gen_encoding.py`. +//! `lib/codegen/meta/gen_encoding.py`. use constant_hash::{probe, Table}; use ir::{Function, InstructionData, Opcode, Type}; diff --git a/lib/cretonne/src/isa/encoding.rs b/lib/codegen/src/isa/encoding.rs similarity index 100% rename from lib/cretonne/src/isa/encoding.rs rename to lib/codegen/src/isa/encoding.rs diff --git a/lib/cretonne/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs similarity index 99% rename from lib/cretonne/src/isa/mod.rs rename to lib/codegen/src/isa/mod.rs index d851d8d9a0..a8d8616858 100644 --- a/lib/cretonne/src/isa/mod.rs +++ b/lib/codegen/src/isa/mod.rs @@ -20,8 +20,8 @@ //! appropriate for the requested ISA: //! //! ``` -//! use cretonne::settings::{self, Configurable}; -//! use cretonne::isa; +//! use cretonne_codegen::settings::{self, Configurable}; +//! use cretonne_codegen::isa; //! //! let shared_builder = settings::builder(); //! let shared_flags = settings::Flags::new(&shared_builder); diff --git a/lib/cretonne/src/isa/registers.rs b/lib/codegen/src/isa/registers.rs similarity index 100% rename from lib/cretonne/src/isa/registers.rs rename to lib/codegen/src/isa/registers.rs diff --git a/lib/cretonne/src/isa/riscv/abi.rs b/lib/codegen/src/isa/riscv/abi.rs similarity index 100% rename from lib/cretonne/src/isa/riscv/abi.rs rename to lib/codegen/src/isa/riscv/abi.rs diff --git a/lib/cretonne/src/isa/riscv/binemit.rs b/lib/codegen/src/isa/riscv/binemit.rs similarity index 100% rename from lib/cretonne/src/isa/riscv/binemit.rs rename to lib/codegen/src/isa/riscv/binemit.rs diff --git a/lib/cretonne/src/isa/riscv/enc_tables.rs b/lib/codegen/src/isa/riscv/enc_tables.rs similarity index 100% rename from lib/cretonne/src/isa/riscv/enc_tables.rs rename to lib/codegen/src/isa/riscv/enc_tables.rs diff --git a/lib/cretonne/src/isa/riscv/mod.rs b/lib/codegen/src/isa/riscv/mod.rs similarity index 100% rename from lib/cretonne/src/isa/riscv/mod.rs rename to lib/codegen/src/isa/riscv/mod.rs diff --git a/lib/cretonne/src/isa/riscv/registers.rs b/lib/codegen/src/isa/riscv/registers.rs similarity index 100% rename from lib/cretonne/src/isa/riscv/registers.rs rename to lib/codegen/src/isa/riscv/registers.rs diff --git a/lib/cretonne/src/isa/riscv/settings.rs b/lib/codegen/src/isa/riscv/settings.rs similarity index 91% rename from lib/cretonne/src/isa/riscv/settings.rs rename to lib/codegen/src/isa/riscv/settings.rs index 27cba43f2f..4a127b578a 100644 --- a/lib/cretonne/src/isa/riscv/settings.rs +++ b/lib/codegen/src/isa/riscv/settings.rs @@ -3,9 +3,9 @@ use settings::{self, detail, Builder}; use std::fmt; -// Include code generated by `lib/cretonne/meta/gen_settings.py`. This file contains a public +// Include code generated by `lib/codegen/meta/gen_settings.py`. This file contains a public // `Flags` struct with an impl for all of the settings defined in -// `lib/cretonne/meta/isa/riscv/settings.py`. +// `lib/codegen/meta/isa/riscv/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs")); #[cfg(test)] diff --git a/lib/cretonne/src/isa/stack.rs b/lib/codegen/src/isa/stack.rs similarity index 100% rename from lib/cretonne/src/isa/stack.rs rename to lib/codegen/src/isa/stack.rs diff --git a/lib/cretonne/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs similarity index 100% rename from lib/cretonne/src/isa/x86/abi.rs rename to lib/codegen/src/isa/x86/abi.rs diff --git a/lib/cretonne/src/isa/x86/binemit.rs b/lib/codegen/src/isa/x86/binemit.rs similarity index 100% rename from lib/cretonne/src/isa/x86/binemit.rs rename to lib/codegen/src/isa/x86/binemit.rs diff --git a/lib/cretonne/src/isa/x86/enc_tables.rs b/lib/codegen/src/isa/x86/enc_tables.rs similarity index 100% rename from lib/cretonne/src/isa/x86/enc_tables.rs rename to lib/codegen/src/isa/x86/enc_tables.rs diff --git a/lib/cretonne/src/isa/x86/mod.rs b/lib/codegen/src/isa/x86/mod.rs similarity index 100% rename from lib/cretonne/src/isa/x86/mod.rs rename to lib/codegen/src/isa/x86/mod.rs diff --git a/lib/cretonne/src/isa/x86/registers.rs b/lib/codegen/src/isa/x86/registers.rs similarity index 100% rename from lib/cretonne/src/isa/x86/registers.rs rename to lib/codegen/src/isa/x86/registers.rs diff --git a/lib/cretonne/src/isa/x86/settings.rs b/lib/codegen/src/isa/x86/settings.rs similarity index 91% rename from lib/cretonne/src/isa/x86/settings.rs rename to lib/codegen/src/isa/x86/settings.rs index 8bcf75249b..379f0db493 100644 --- a/lib/cretonne/src/isa/x86/settings.rs +++ b/lib/codegen/src/isa/x86/settings.rs @@ -3,9 +3,9 @@ use settings::{self, detail, Builder}; use std::fmt; -// Include code generated by `lib/cretonne/meta/gen_settings.py`. This file contains a public +// Include code generated by `lib/codegen/meta/gen_settings.py`. This file contains a public // `Flags` struct with an impl for all of the settings defined in -// `lib/cretonne/meta/isa/x86/settings.py`. +// `lib/codegen/meta/isa/x86/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings-x86.rs")); #[cfg(test)] diff --git a/lib/cretonne/src/iterators.rs b/lib/codegen/src/iterators.rs similarity index 100% rename from lib/cretonne/src/iterators.rs rename to lib/codegen/src/iterators.rs diff --git a/lib/cretonne/src/legalizer/boundary.rs b/lib/codegen/src/legalizer/boundary.rs similarity index 100% rename from lib/cretonne/src/legalizer/boundary.rs rename to lib/codegen/src/legalizer/boundary.rs diff --git a/lib/cretonne/src/legalizer/call.rs b/lib/codegen/src/legalizer/call.rs similarity index 100% rename from lib/cretonne/src/legalizer/call.rs rename to lib/codegen/src/legalizer/call.rs diff --git a/lib/cretonne/src/legalizer/globalvar.rs b/lib/codegen/src/legalizer/globalvar.rs similarity index 100% rename from lib/cretonne/src/legalizer/globalvar.rs rename to lib/codegen/src/legalizer/globalvar.rs diff --git a/lib/cretonne/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs similarity index 100% rename from lib/cretonne/src/legalizer/heap.rs rename to lib/codegen/src/legalizer/heap.rs diff --git a/lib/cretonne/src/legalizer/libcall.rs b/lib/codegen/src/legalizer/libcall.rs similarity index 100% rename from lib/cretonne/src/legalizer/libcall.rs rename to lib/codegen/src/legalizer/libcall.rs diff --git a/lib/cretonne/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs similarity index 99% rename from lib/cretonne/src/legalizer/mod.rs rename to lib/codegen/src/legalizer/mod.rs index 4486d5df4e..f4f9389390 100644 --- a/lib/cretonne/src/legalizer/mod.rs +++ b/lib/codegen/src/legalizer/mod.rs @@ -105,7 +105,7 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is } // Include legalization patterns that were generated by `gen_legalizer.py` from the `XForms` in -// `lib/cretonne/meta/base/legalize.py`. +// `lib/codegen/meta/base/legalize.py`. // // Concretely, this defines private functions `narrow()`, and `expand()`. include!(concat!(env!("OUT_DIR"), "/legalizer.rs")); diff --git a/lib/cretonne/src/legalizer/split.rs b/lib/codegen/src/legalizer/split.rs similarity index 100% rename from lib/cretonne/src/legalizer/split.rs rename to lib/codegen/src/legalizer/split.rs diff --git a/lib/cretonne/src/lib.rs b/lib/codegen/src/lib.rs similarity index 97% rename from lib/cretonne/src/lib.rs rename to lib/codegen/src/lib.rs index 8bc88ab420..8860cb2e69 100644 --- a/lib/cretonne/src/lib.rs +++ b/lib/codegen/src/lib.rs @@ -35,7 +35,7 @@ pub use legalizer::legalize_function; pub use verifier::verify_function; pub use write::write_function; -/// Version number of the cretonne crate. +/// Version number of the cretonne-codegen crate. pub const VERSION: &str = env!("CARGO_PKG_VERSION"); #[macro_use] diff --git a/lib/cretonne/src/licm.rs b/lib/codegen/src/licm.rs similarity index 100% rename from lib/cretonne/src/licm.rs rename to lib/codegen/src/licm.rs diff --git a/lib/cretonne/src/loop_analysis.rs b/lib/codegen/src/loop_analysis.rs similarity index 100% rename from lib/cretonne/src/loop_analysis.rs rename to lib/codegen/src/loop_analysis.rs diff --git a/lib/cretonne/src/partition_slice.rs b/lib/codegen/src/partition_slice.rs similarity index 100% rename from lib/cretonne/src/partition_slice.rs rename to lib/codegen/src/partition_slice.rs diff --git a/lib/cretonne/src/postopt.rs b/lib/codegen/src/postopt.rs similarity index 100% rename from lib/cretonne/src/postopt.rs rename to lib/codegen/src/postopt.rs diff --git a/lib/cretonne/src/predicates.rs b/lib/codegen/src/predicates.rs similarity index 98% rename from lib/cretonne/src/predicates.rs rename to lib/codegen/src/predicates.rs index 333ca8eb31..e35e7ad179 100644 --- a/lib/cretonne/src/predicates.rs +++ b/lib/codegen/src/predicates.rs @@ -1,7 +1,7 @@ //! Predicate functions for testing instruction fields. //! //! This module defines functions that are used by the instruction predicates defined by -//! `lib/cretonne/meta/cdsl/predicates.py` classes. +//! `lib/codegen/meta/cdsl/predicates.py` classes. //! //! The predicates the operate on integer fields use `Into` as a shared trait bound. This //! bound is implemented by all the native integer types as well as `Imm64`. diff --git a/lib/cretonne/src/preopt.rs b/lib/codegen/src/preopt.rs similarity index 100% rename from lib/cretonne/src/preopt.rs rename to lib/codegen/src/preopt.rs diff --git a/lib/cretonne/src/print_errors.rs b/lib/codegen/src/print_errors.rs similarity index 100% rename from lib/cretonne/src/print_errors.rs rename to lib/codegen/src/print_errors.rs diff --git a/lib/cretonne/src/ref_slice.rs b/lib/codegen/src/ref_slice.rs similarity index 100% rename from lib/cretonne/src/ref_slice.rs rename to lib/codegen/src/ref_slice.rs diff --git a/lib/cretonne/src/regalloc/affinity.rs b/lib/codegen/src/regalloc/affinity.rs similarity index 100% rename from lib/cretonne/src/regalloc/affinity.rs rename to lib/codegen/src/regalloc/affinity.rs diff --git a/lib/cretonne/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs similarity index 100% rename from lib/cretonne/src/regalloc/coalescing.rs rename to lib/codegen/src/regalloc/coalescing.rs diff --git a/lib/cretonne/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs similarity index 100% rename from lib/cretonne/src/regalloc/coloring.rs rename to lib/codegen/src/regalloc/coloring.rs diff --git a/lib/cretonne/src/regalloc/context.rs b/lib/codegen/src/regalloc/context.rs similarity index 100% rename from lib/cretonne/src/regalloc/context.rs rename to lib/codegen/src/regalloc/context.rs diff --git a/lib/cretonne/src/regalloc/diversion.rs b/lib/codegen/src/regalloc/diversion.rs similarity index 100% rename from lib/cretonne/src/regalloc/diversion.rs rename to lib/codegen/src/regalloc/diversion.rs diff --git a/lib/cretonne/src/regalloc/live_value_tracker.rs b/lib/codegen/src/regalloc/live_value_tracker.rs similarity index 100% rename from lib/cretonne/src/regalloc/live_value_tracker.rs rename to lib/codegen/src/regalloc/live_value_tracker.rs diff --git a/lib/cretonne/src/regalloc/liveness.rs b/lib/codegen/src/regalloc/liveness.rs similarity index 100% rename from lib/cretonne/src/regalloc/liveness.rs rename to lib/codegen/src/regalloc/liveness.rs diff --git a/lib/cretonne/src/regalloc/liverange.rs b/lib/codegen/src/regalloc/liverange.rs similarity index 100% rename from lib/cretonne/src/regalloc/liverange.rs rename to lib/codegen/src/regalloc/liverange.rs diff --git a/lib/cretonne/src/regalloc/mod.rs b/lib/codegen/src/regalloc/mod.rs similarity index 100% rename from lib/cretonne/src/regalloc/mod.rs rename to lib/codegen/src/regalloc/mod.rs diff --git a/lib/cretonne/src/regalloc/pressure.rs b/lib/codegen/src/regalloc/pressure.rs similarity index 100% rename from lib/cretonne/src/regalloc/pressure.rs rename to lib/codegen/src/regalloc/pressure.rs diff --git a/lib/cretonne/src/regalloc/register_set.rs b/lib/codegen/src/regalloc/register_set.rs similarity index 100% rename from lib/cretonne/src/regalloc/register_set.rs rename to lib/codegen/src/regalloc/register_set.rs diff --git a/lib/cretonne/src/regalloc/reload.rs b/lib/codegen/src/regalloc/reload.rs similarity index 100% rename from lib/cretonne/src/regalloc/reload.rs rename to lib/codegen/src/regalloc/reload.rs diff --git a/lib/cretonne/src/regalloc/solver.rs b/lib/codegen/src/regalloc/solver.rs similarity index 100% rename from lib/cretonne/src/regalloc/solver.rs rename to lib/codegen/src/regalloc/solver.rs diff --git a/lib/cretonne/src/regalloc/spilling.rs b/lib/codegen/src/regalloc/spilling.rs similarity index 100% rename from lib/cretonne/src/regalloc/spilling.rs rename to lib/codegen/src/regalloc/spilling.rs diff --git a/lib/cretonne/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs similarity index 100% rename from lib/cretonne/src/regalloc/virtregs.rs rename to lib/codegen/src/regalloc/virtregs.rs diff --git a/lib/cretonne/src/result.rs b/lib/codegen/src/result.rs similarity index 100% rename from lib/cretonne/src/result.rs rename to lib/codegen/src/result.rs diff --git a/lib/cretonne/src/scoped_hash_map.rs b/lib/codegen/src/scoped_hash_map.rs similarity index 100% rename from lib/cretonne/src/scoped_hash_map.rs rename to lib/codegen/src/scoped_hash_map.rs diff --git a/lib/cretonne/src/settings.rs b/lib/codegen/src/settings.rs similarity index 98% rename from lib/cretonne/src/settings.rs rename to lib/codegen/src/settings.rs index aee441f3ca..6db7ef402e 100644 --- a/lib/cretonne/src/settings.rs +++ b/lib/codegen/src/settings.rs @@ -11,7 +11,7 @@ //! //! # Example //! ``` -//! use cretonne::settings::{self, Configurable}; +//! use cretonne_codegen::settings::{self, Configurable}; //! //! let mut b = settings::builder(); //! b.set("opt_level", "fastest"); @@ -312,7 +312,7 @@ pub mod detail { } // Include code generated by `meta/gen_settings.py`. This file contains a public `Flags` struct -// with an impl for all of the settings defined in `lib/cretonne/meta/base/settings.py`. +// with an impl for all of the settings defined in `lib/codegen/meta/base/settings.py`. include!(concat!(env!("OUT_DIR"), "/settings.rs")); /// Wrapper containing flags and optionally a `TargetIsa` trait object. diff --git a/lib/cretonne/src/simple_gvn.rs b/lib/codegen/src/simple_gvn.rs similarity index 100% rename from lib/cretonne/src/simple_gvn.rs rename to lib/codegen/src/simple_gvn.rs diff --git a/lib/cretonne/src/stack_layout.rs b/lib/codegen/src/stack_layout.rs similarity index 100% rename from lib/cretonne/src/stack_layout.rs rename to lib/codegen/src/stack_layout.rs diff --git a/lib/cretonne/src/timing.rs b/lib/codegen/src/timing.rs similarity index 100% rename from lib/cretonne/src/timing.rs rename to lib/codegen/src/timing.rs diff --git a/lib/cretonne/src/topo_order.rs b/lib/codegen/src/topo_order.rs similarity index 100% rename from lib/cretonne/src/topo_order.rs rename to lib/codegen/src/topo_order.rs diff --git a/lib/cretonne/src/unreachable_code.rs b/lib/codegen/src/unreachable_code.rs similarity index 100% rename from lib/cretonne/src/unreachable_code.rs rename to lib/codegen/src/unreachable_code.rs diff --git a/lib/cretonne/src/verifier/cssa.rs b/lib/codegen/src/verifier/cssa.rs similarity index 100% rename from lib/cretonne/src/verifier/cssa.rs rename to lib/codegen/src/verifier/cssa.rs diff --git a/lib/cretonne/src/verifier/flags.rs b/lib/codegen/src/verifier/flags.rs similarity index 100% rename from lib/cretonne/src/verifier/flags.rs rename to lib/codegen/src/verifier/flags.rs diff --git a/lib/cretonne/src/verifier/liveness.rs b/lib/codegen/src/verifier/liveness.rs similarity index 100% rename from lib/cretonne/src/verifier/liveness.rs rename to lib/codegen/src/verifier/liveness.rs diff --git a/lib/cretonne/src/verifier/locations.rs b/lib/codegen/src/verifier/locations.rs similarity index 100% rename from lib/cretonne/src/verifier/locations.rs rename to lib/codegen/src/verifier/locations.rs diff --git a/lib/cretonne/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs similarity index 100% rename from lib/cretonne/src/verifier/mod.rs rename to lib/codegen/src/verifier/mod.rs diff --git a/lib/cretonne/src/write.rs b/lib/codegen/src/write.rs similarity index 100% rename from lib/cretonne/src/write.rs rename to lib/codegen/src/write.rs diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index e0187f3f75..984d5e0629 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -9,7 +9,7 @@ repository = "https://github.com/Cretonne/cretonne" publish = false [dependencies] -cretonne = { path = "../cretonne", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.4.4" } cretonne-reader = { path = "../reader", version = "0.4.4" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/filetests/src/concurrent.rs b/lib/filetests/src/concurrent.rs index 7ca4b01d89..ac45624114 100644 --- a/lib/filetests/src/concurrent.rs +++ b/lib/filetests/src/concurrent.rs @@ -3,7 +3,7 @@ //! This module provides the `ConcurrentRunner` struct which uses a pool of threads to run tests //! concurrently. -use cretonne::timing; +use cretonne_codegen::timing; use num_cpus; use std::panic::catch_unwind; use std::path::{Path, PathBuf}; diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs index d58f15fdc0..df7be54fa6 100644 --- a/lib/filetests/src/lib.rs +++ b/lib/filetests/src/lib.rs @@ -9,7 +9,7 @@ block_in_if_condition_stmt))] #[macro_use(dbg)] -extern crate cretonne; +extern crate cretonne_codegen; extern crate cretonne_reader; extern crate filecheck; extern crate num_cpus; diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs index dee5bfc2c9..6d80afffad 100644 --- a/lib/filetests/src/runone.rs +++ b/lib/filetests/src/runone.rs @@ -1,11 +1,11 @@ //! Run the tests in a single test file. -use cretonne::ir::Function; -use cretonne::isa::TargetIsa; -use cretonne::print_errors::pretty_verifier_error; -use cretonne::settings::Flags; -use cretonne::timing; -use cretonne::verify_function; +use cretonne_codegen::ir::Function; +use cretonne_codegen::isa::TargetIsa; +use cretonne_codegen::print_errors::pretty_verifier_error; +use cretonne_codegen::settings::Flags; +use cretonne_codegen::timing; +use cretonne_codegen::verify_function; use cretonne_reader::IsaSpec; use cretonne_reader::parse_test; use std::borrow::Cow; diff --git a/lib/filetests/src/subtest.rs b/lib/filetests/src/subtest.rs index 4ae71494d7..2ed560cb3f 100644 --- a/lib/filetests/src/subtest.rs +++ b/lib/filetests/src/subtest.rs @@ -1,8 +1,8 @@ //! `SubTest` trait. -use cretonne::ir::Function; -use cretonne::isa::TargetIsa; -use cretonne::settings::{Flags, FlagsOrIsa}; +use cretonne_codegen::ir::Function; +use cretonne_codegen::isa::TargetIsa; +use cretonne_codegen::settings::{Flags, FlagsOrIsa}; use cretonne_reader::{Comment, Details}; use filecheck::{Checker, CheckerBuilder, NO_VARIABLES}; use std::borrow::Cow; diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index 620eb7e9f3..86058673e3 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -3,12 +3,12 @@ //! The `binemit` test command generates binary machine code for every instruction in the input //! functions and compares the results to the expected output. -use cretonne::binemit; -use cretonne::binemit::RegDiversions; -use cretonne::dbg::DisplayList; -use cretonne::ir; -use cretonne::ir::entities::AnyEntity; -use cretonne::print_errors::pretty_error; +use cretonne_codegen::binemit; +use cretonne_codegen::binemit::RegDiversions; +use cretonne_codegen::dbg::DisplayList; +use cretonne_codegen::ir; +use cretonne_codegen::ir::entities::AnyEntity; +use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use match_directive::match_directive; use std::borrow::Cow; diff --git a/lib/filetests/src/test_cat.rs b/lib/filetests/src/test_cat.rs index 3d04fcae3e..4fdedf115f 100644 --- a/lib/filetests/src/test_cat.rs +++ b/lib/filetests/src/test_cat.rs @@ -1,6 +1,6 @@ //! The `cat` subtest. -use cretonne::ir::Function; +use cretonne_codegen::ir::Function; use cretonne_reader::TestCommand; use std::borrow::Cow; use subtest::{self, Context, Result as STResult, SubTest}; diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs index a5af1f54e7..c4b737bfc6 100644 --- a/lib/filetests/src/test_compile.rs +++ b/lib/filetests/src/test_compile.rs @@ -2,10 +2,9 @@ //! //! The `compile` test command runs each function through the full code generator pipeline -use cretonne; -use cretonne::binemit; -use cretonne::ir; -use cretonne::print_errors::pretty_error; +use cretonne_codegen; +use cretonne_codegen::{binemit, ir}; +use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; @@ -39,7 +38,7 @@ impl SubTest for TestCompile { let isa = context.isa.expect("compile needs an ISA"); // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne::Context::new(); + let mut comp_ctx = cretonne_codegen::Context::new(); comp_ctx.func = func.into_owned(); let code_size = comp_ctx.compile(isa).map_err(|e| { diff --git a/lib/filetests/src/test_dce.rs b/lib/filetests/src/test_dce.rs index 4d79cea1ce..14d535aca4 100644 --- a/lib/filetests/src/test_dce.rs +++ b/lib/filetests/src/test_dce.rs @@ -5,9 +5,9 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne; -use cretonne::ir::Function; -use cretonne::print_errors::pretty_error; +use cretonne_codegen; +use cretonne_codegen::ir::Function; +use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; @@ -35,7 +35,7 @@ impl SubTest for TestDCE { fn run(&self, func: Cow, context: &Context) -> Result<()> { // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne::Context::new(); + let mut comp_ctx = cretonne_codegen::Context::new(); comp_ctx.func = func.into_owned(); comp_ctx.flowgraph(); diff --git a/lib/filetests/src/test_domtree.rs b/lib/filetests/src/test_domtree.rs index 9862f3f4fe..9803a67d4e 100644 --- a/lib/filetests/src/test_domtree.rs +++ b/lib/filetests/src/test_domtree.rs @@ -12,10 +12,10 @@ //! We verify that the dominator tree annotations are complete and correct. //! -use cretonne::dominator_tree::{DominatorTree, DominatorTreePreorder}; -use cretonne::flowgraph::ControlFlowGraph; -use cretonne::ir::Function; -use cretonne::ir::entities::AnyEntity; +use cretonne_codegen::dominator_tree::{DominatorTree, DominatorTreePreorder}; +use cretonne_codegen::flowgraph::ControlFlowGraph; +use cretonne_codegen::ir::Function; +use cretonne_codegen::ir::entities::AnyEntity; use cretonne_reader::TestCommand; use match_directive::match_directive; use std::borrow::{Borrow, Cow}; diff --git a/lib/filetests/src/test_legalizer.rs b/lib/filetests/src/test_legalizer.rs index af288bcc8d..cb24ba44a5 100644 --- a/lib/filetests/src/test_legalizer.rs +++ b/lib/filetests/src/test_legalizer.rs @@ -3,9 +3,9 @@ //! The `test legalizer` test command runs each function through `legalize_function()` and sends //! the result to filecheck. -use cretonne; -use cretonne::ir::Function; -use cretonne::print_errors::pretty_error; +use cretonne_codegen; +use cretonne_codegen::ir::Function; +use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; @@ -36,7 +36,7 @@ impl SubTest for TestLegalizer { } fn run(&self, func: Cow, context: &Context) -> Result<()> { - let mut comp_ctx = cretonne::Context::new(); + let mut comp_ctx = cretonne_codegen::Context::new(); comp_ctx.func = func.into_owned(); let isa = context.isa.expect("legalizer needs an ISA"); diff --git a/lib/filetests/src/test_licm.rs b/lib/filetests/src/test_licm.rs index 435b5ab1bc..3add99b1ab 100644 --- a/lib/filetests/src/test_licm.rs +++ b/lib/filetests/src/test_licm.rs @@ -5,9 +5,9 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne; -use cretonne::ir::Function; -use cretonne::print_errors::pretty_error; +use cretonne_codegen; +use cretonne_codegen::ir::Function; +use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; @@ -35,7 +35,7 @@ impl SubTest for TestLICM { fn run(&self, func: Cow, context: &Context) -> Result<()> { // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne::Context::new(); + let mut comp_ctx = cretonne_codegen::Context::new(); comp_ctx.func = func.into_owned(); comp_ctx.flowgraph(); diff --git a/lib/filetests/src/test_postopt.rs b/lib/filetests/src/test_postopt.rs index b575a36096..14c0060e6d 100644 --- a/lib/filetests/src/test_postopt.rs +++ b/lib/filetests/src/test_postopt.rs @@ -2,9 +2,9 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne; -use cretonne::ir::Function; -use cretonne::print_errors::pretty_error; +use cretonne_codegen; +use cretonne_codegen::ir::Function; +use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; @@ -32,7 +32,7 @@ impl SubTest for TestPostopt { fn run(&self, func: Cow, context: &Context) -> Result<()> { // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne::Context::new(); + let mut comp_ctx = cretonne_codegen::Context::new(); comp_ctx.func = func.into_owned(); let isa = context.isa.expect("postopt needs an ISA"); diff --git a/lib/filetests/src/test_preopt.rs b/lib/filetests/src/test_preopt.rs index f58d216a26..f4c53ce267 100644 --- a/lib/filetests/src/test_preopt.rs +++ b/lib/filetests/src/test_preopt.rs @@ -2,9 +2,9 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne; -use cretonne::ir::Function; -use cretonne::print_errors::pretty_error; +use cretonne_codegen; +use cretonne_codegen::ir::Function; +use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; @@ -32,7 +32,7 @@ impl SubTest for TestPreopt { fn run(&self, func: Cow, context: &Context) -> Result<()> { // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne::Context::new(); + let mut comp_ctx = cretonne_codegen::Context::new(); comp_ctx.func = func.into_owned(); let isa = context.isa.expect("preopt needs an ISA"); diff --git a/lib/filetests/src/test_print_cfg.rs b/lib/filetests/src/test_print_cfg.rs index f6388b70fe..3f48b1d4e9 100644 --- a/lib/filetests/src/test_print_cfg.rs +++ b/lib/filetests/src/test_print_cfg.rs @@ -5,8 +5,8 @@ use std::borrow::Cow; -use cretonne::cfg_printer::CFGPrinter; -use cretonne::ir::Function; +use cretonne_codegen::cfg_printer::CFGPrinter; +use cretonne_codegen::ir::Function; use cretonne_reader::TestCommand; use subtest::{self, Context, Result as STResult, SubTest}; diff --git a/lib/filetests/src/test_regalloc.rs b/lib/filetests/src/test_regalloc.rs index 36f3ac169f..5010c84ad4 100644 --- a/lib/filetests/src/test_regalloc.rs +++ b/lib/filetests/src/test_regalloc.rs @@ -5,9 +5,9 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne; -use cretonne::ir::Function; -use cretonne::print_errors::pretty_error; +use cretonne_codegen; +use cretonne_codegen::ir::Function; +use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; @@ -41,7 +41,7 @@ impl SubTest for TestRegalloc { let isa = context.isa.expect("register allocator needs an ISA"); // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne::Context::new(); + let mut comp_ctx = cretonne_codegen::Context::new(); comp_ctx.func = func.into_owned(); comp_ctx.compute_cfg(); diff --git a/lib/filetests/src/test_simple_gvn.rs b/lib/filetests/src/test_simple_gvn.rs index b60379cff1..467559eecb 100644 --- a/lib/filetests/src/test_simple_gvn.rs +++ b/lib/filetests/src/test_simple_gvn.rs @@ -5,9 +5,9 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne; -use cretonne::ir::Function; -use cretonne::print_errors::pretty_error; +use cretonne_codegen; +use cretonne_codegen::ir::Function; +use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; @@ -35,7 +35,7 @@ impl SubTest for TestSimpleGVN { fn run(&self, func: Cow, context: &Context) -> Result<()> { // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne::Context::new(); + let mut comp_ctx = cretonne_codegen::Context::new(); comp_ctx.func = func.into_owned(); comp_ctx.flowgraph(); diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs index 0243469630..514e988936 100644 --- a/lib/filetests/src/test_verifier.rs +++ b/lib/filetests/src/test_verifier.rs @@ -9,8 +9,8 @@ //! This annotation means that the verifier is expected to given an error for the jump instruction //! containing the substring "jump to non-existent EBB". -use cretonne::ir::Function; -use cretonne::verify_function; +use cretonne_codegen::ir::Function; +use cretonne_codegen::verify_function; use cretonne_reader::TestCommand; use match_directive::match_directive; use std::borrow::{Borrow, Cow}; diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 5dc031926b..77c54a32f0 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -9,7 +9,7 @@ repository = "https://github.com/Cretonne/cretonne" readme = "README.md" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.4.4" } [badges] maintenance = { status = "experimental" } diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 4396c4edac..12568582e2 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -1,13 +1,14 @@ //! A frontend for building Cretonne IR from other languages. -use cretonne::cursor::{Cursor, FuncCursor}; -use cretonne::entity::{EntityMap, EntityRef, EntitySet}; -use cretonne::ir; -use cretonne::ir::function::DisplayFunction; -use cretonne::ir::{DataFlowGraph, Ebb, ExtFuncData, FuncRef, Function, GlobalVar, GlobalVarData, - Heap, HeapData, Inst, InstBuilderBase, InstructionData, JumpTable, - JumpTableData, SigRef, Signature, StackSlot, StackSlotData, Type, Value}; -use cretonne::isa::TargetIsa; -use cretonne::packed_option::PackedOption; +use cretonne_codegen::cursor::{Cursor, FuncCursor}; +use cretonne_codegen::entity::{EntityMap, EntityRef, EntitySet}; +use cretonne_codegen::ir; +use cretonne_codegen::ir::function::DisplayFunction; +use cretonne_codegen::ir::{DataFlowGraph, Ebb, ExtFuncData, FuncRef, Function, GlobalVar, + GlobalVarData, Heap, HeapData, Inst, InstBuilderBase, InstructionData, + JumpTable, JumpTableData, SigRef, Signature, StackSlot, StackSlotData, + Type, Value}; +use cretonne_codegen::isa::TargetIsa; +use cretonne_codegen::packed_option::PackedOption; use ssa::{Block, SSABuilder, SideEffects}; /// Structure used for translating a series of functions into Cretonne IR. @@ -101,7 +102,7 @@ where } } -/// Implementation of the [`InstBuilder`](../cretonne/ir/builder/trait.InstBuilder.html) that has +/// Implementation of the [`InstBuilder`](../codegen/ir/builder/trait.InstBuilder.html) that has /// one convenience method per Cretonne IR instruction. pub struct FuncInstBuilder<'short, 'long: 'short, Variable: 'long> where @@ -374,7 +375,7 @@ where self.func.create_heap(data) } - /// Returns an object with the [`InstBuilder`](../cretonne/ir/builder/trait.InstBuilder.html) + /// Returns an object with the [`InstBuilder`](../codegen/ir/builder/trait.InstBuilder.html) /// trait that allows to conveniently append an instruction to the current `Ebb` being built. pub fn ins<'short>(&'short mut self) -> FuncInstBuilder<'short, 'a, Variable> { let ebb = self.position.ebb.unwrap(); @@ -593,11 +594,11 @@ where mod tests { use Variable; - use cretonne::entity::EntityRef; - use cretonne::ir::types::*; - use cretonne::ir::{AbiParam, CallConv, ExternalName, Function, InstBuilder, Signature}; - use cretonne::settings; - use cretonne::verifier::verify_function; + use cretonne_codegen::entity::EntityRef; + use cretonne_codegen::ir::types::*; + use cretonne_codegen::ir::{AbiParam, CallConv, ExternalName, Function, InstBuilder, Signature}; + use cretonne_codegen::settings; + use cretonne_codegen::verifier::verify_function; use frontend::{FunctionBuilder, FunctionBuilderContext}; fn sample_function(lazy_seal: bool) { diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 369a4cf13d..8330a062ad 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -32,15 +32,15 @@ //! Here is how you build the corresponding Cretonne IR function using `FunctionBuilderContext`: //! //! ```rust -//! extern crate cretonne; +//! extern crate cretonne_codegen; //! extern crate cretonne_frontend; //! -//! use cretonne::entity::EntityRef; -//! use cretonne::ir::{ExternalName, CallConv, Function, Signature, AbiParam, InstBuilder}; -//! use cretonne::ir::types::*; -//! use cretonne::settings; +//! use cretonne_codegen::entity::EntityRef; +//! use cretonne_codegen::ir::{ExternalName, CallConv, Function, Signature, AbiParam, InstBuilder}; +//! use cretonne_codegen::ir::types::*; +//! use cretonne_codegen::settings; //! use cretonne_frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; -//! use cretonne::verifier::verify_function; +//! use cretonne_codegen::verifier::verify_function; //! //! fn main() { //! let mut sig = Signature::new(CallConv::SystemV); @@ -130,7 +130,7 @@ #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, redundant_field_names))] -extern crate cretonne; +extern crate cretonne_codegen; pub use frontend::{FunctionBuilder, FunctionBuilderContext}; pub use variable::Variable; diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 82e9242b4f..22daff51d2 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -5,14 +5,14 @@ //! In: Jhala R., De Bosschere K. (eds) Compiler Construction. CC 2013. //! Lecture Notes in Computer Science, vol 7791. Springer, Berlin, Heidelberg -use cretonne::cursor::{Cursor, FuncCursor}; -use cretonne::entity::{EntityMap, EntityRef, PrimaryMap}; -use cretonne::ir::immediates::{Ieee32, Ieee64}; -use cretonne::ir::instructions::BranchInfo; -use cretonne::ir::types::{F32, F64}; -use cretonne::ir::{Ebb, Function, Inst, InstBuilder, Type, Value}; -use cretonne::packed_option::PackedOption; -use cretonne::packed_option::ReservedValue; +use cretonne_codegen::cursor::{Cursor, FuncCursor}; +use cretonne_codegen::entity::{EntityMap, EntityRef, PrimaryMap}; +use cretonne_codegen::ir::immediates::{Ieee32, Ieee64}; +use cretonne_codegen::ir::instructions::BranchInfo; +use cretonne_codegen::ir::types::{F32, F64}; +use cretonne_codegen::ir::{Ebb, Function, Inst, InstBuilder, Type, Value}; +use cretonne_codegen::packed_option::PackedOption; +use cretonne_codegen::packed_option::ReservedValue; use std::mem; use std::u32; use std::vec::Vec; @@ -714,13 +714,13 @@ where #[cfg(test)] mod tests { use Variable; - use cretonne::cursor::{Cursor, FuncCursor}; - use cretonne::entity::EntityRef; - use cretonne::ir::instructions::BranchInfo; - use cretonne::ir::types::*; - use cretonne::ir::{Function, Inst, InstBuilder, JumpTableData, Opcode}; - use cretonne::settings; - use cretonne::verify_function; + use cretonne_codegen::cursor::{Cursor, FuncCursor}; + use cretonne_codegen::entity::EntityRef; + use cretonne_codegen::ir::instructions::BranchInfo; + use cretonne_codegen::ir::types::*; + use cretonne_codegen::ir::{Function, Inst, InstBuilder, JumpTableData, Opcode}; + use cretonne_codegen::settings; + use cretonne_codegen::verify_function; use ssa::SSABuilder; #[test] diff --git a/lib/frontend/src/variable.rs b/lib/frontend/src/variable.rs index b4e3c75da2..368d4bfb18 100644 --- a/lib/frontend/src/variable.rs +++ b/lib/frontend/src/variable.rs @@ -5,7 +5,7 @@ //! their own index types to use them directly. Frontends which don't //! can use the `Variable` defined here. -use cretonne::entity::EntityRef; +use cretonne_codegen::entity::EntityRef; use std::u32; ///! An opaque reference to a variable. diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 3c80e45d15..35afe7441f 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -8,7 +8,7 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.4.4" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index 971abdc4f5..c2504b5c38 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -3,13 +3,13 @@ #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] -extern crate cretonne; +extern crate cretonne_codegen; #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] extern crate raw_cpuid; -use cretonne::isa; -use cretonne::settings::{self, Configurable}; +use cretonne_codegen::isa; +use cretonne_codegen::settings::{self, Configurable}; #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] use raw_cpuid::CpuId; diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index d3a5213cdd..49afaf482a 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -9,7 +9,7 @@ repository = "https://github.com/Cretonne/cretonne" readme = "README.md" [dependencies] -cretonne = { path = "../cretonne", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.4.4" } [badges] maintenance = { status = "experimental" } diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs index e87b2b6771..85b08df368 100644 --- a/lib/reader/src/isaspec.rs +++ b/lib/reader/src/isaspec.rs @@ -6,8 +6,8 @@ //! If a test case file contains `isa` commands, the tests will only be run against the specified //! ISAs. If the file contains no `isa` commands, the tests will be run against all supported ISAs. -use cretonne::isa::TargetIsa; -use cretonne::settings::{Configurable, Error as SetError, Flags}; +use cretonne_codegen::isa::TargetIsa; +use cretonne_codegen::settings::{Configurable, Error as SetError, Flags}; use error::{Location, Result}; use testcommand::TestOption; diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index a2fde86b5a..bad7a6369d 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -1,7 +1,7 @@ //! Lexical analysis for .cton files. -use cretonne::ir::types; -use cretonne::ir::{Ebb, Value}; +use cretonne_codegen::ir::types; +use cretonne_codegen::ir::{Ebb, Value}; use error::Location; #[allow(unused_imports)] use std::ascii::AsciiExt; @@ -457,8 +457,8 @@ impl<'a> Lexer<'a> { mod tests { use super::trailing_digits; use super::*; - use cretonne::ir::types; - use cretonne::ir::{Ebb, Value}; + use cretonne_codegen::ir::types; + use cretonne_codegen::ir::{Ebb, Value}; use error::Location; #[test] diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index 8b4b831042..5a51d4199d 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -5,7 +5,7 @@ #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] -extern crate cretonne; +extern crate cretonne_codegen; pub use error::{Error, Location, Result}; pub use isaspec::{parse_options, IsaSpec}; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 66a66ac18c..88d067112b 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1,18 +1,19 @@ //! Parser for .cton files. -use cretonne::entity::EntityRef; -use cretonne::ir; -use cretonne::ir::entities::AnyEntity; -use cretonne::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32}; -use cretonne::ir::instructions::{InstructionData, InstructionFormat, VariableArgs}; -use cretonne::ir::types::VOID; -use cretonne::ir::{AbiParam, ArgumentExtension, ArgumentLoc, CallConv, Ebb, ExtFuncData, - ExternalName, FuncRef, Function, GlobalVar, GlobalVarData, Heap, HeapBase, - HeapData, HeapStyle, JumpTable, JumpTableData, MemFlags, Opcode, SigRef, - Signature, StackSlot, StackSlotData, StackSlotKind, Type, Value, ValueLoc}; -use cretonne::isa::{self, Encoding, RegUnit, TargetIsa}; -use cretonne::packed_option::ReservedValue; -use cretonne::{settings, timing}; +use cretonne_codegen::entity::EntityRef; +use cretonne_codegen::ir; +use cretonne_codegen::ir::entities::AnyEntity; +use cretonne_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32}; +use cretonne_codegen::ir::instructions::{InstructionData, InstructionFormat, VariableArgs}; +use cretonne_codegen::ir::types::VOID; +use cretonne_codegen::ir::{AbiParam, ArgumentExtension, ArgumentLoc, CallConv, Ebb, ExtFuncData, + ExternalName, FuncRef, Function, GlobalVar, GlobalVarData, Heap, + HeapBase, HeapData, HeapStyle, JumpTable, JumpTableData, MemFlags, + Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind, + Type, Value, ValueLoc}; +use cretonne_codegen::isa::{self, Encoding, RegUnit, TargetIsa}; +use cretonne_codegen::packed_option::ReservedValue; +use cretonne_codegen::{settings, timing}; use error::{Error, Location, Result}; use isaspec; use lexer::{self, Lexer, Token}; @@ -2398,10 +2399,10 @@ impl<'a> Parser<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne::ir::StackSlotKind; - use cretonne::ir::entities::AnyEntity; - use cretonne::ir::types; - use cretonne::ir::{ArgumentExtension, ArgumentPurpose, CallConv}; + use cretonne_codegen::ir::StackSlotKind; + use cretonne_codegen::ir::entities::AnyEntity; + use cretonne_codegen::ir::types; + use cretonne_codegen::ir::{ArgumentExtension, ArgumentPurpose, CallConv}; use error::Error; use isaspec::IsaSpec; use testfile::{Comment, Details}; diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index b30b172768..a4bc1954c1 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -6,8 +6,8 @@ //! The `SourceMap` struct defined in this module makes this mapping available //! to parser clients. -use cretonne::ir::entities::AnyEntity; -use cretonne::ir::{Ebb, FuncRef, GlobalVar, Heap, JumpTable, SigRef, StackSlot, Value}; +use cretonne_codegen::ir::entities::AnyEntity; +use cretonne_codegen::ir::{Ebb, FuncRef, GlobalVar, Heap, JumpTable, SigRef, StackSlot, Value}; use error::{Location, Result}; use lexer::split_entity_name; use std::collections::HashMap; diff --git a/lib/reader/src/testfile.rs b/lib/reader/src/testfile.rs index 7218f8cb72..2fae3c2d3d 100644 --- a/lib/reader/src/testfile.rs +++ b/lib/reader/src/testfile.rs @@ -4,8 +4,8 @@ //! file-based test case. //! -use cretonne::ir::Function; -use cretonne::ir::entities::AnyEntity; +use cretonne_codegen::ir::Function; +use cretonne_codegen::ir::entities::AnyEntity; use error::Location; use isaspec::IsaSpec; use sourcemap::SourceMap; diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index f9c60e7c92..8384df98d7 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -10,7 +10,7 @@ keywords = ["webassembly", "wasm"] [dependencies] wasmparser = "0.15.1" -cretonne = { path = "../cretonne", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.4.4" } cretonne-frontend = { path = "../frontend", version = "0.4.4" } [dev-dependencies] diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 945cedc1de..2b48cd50c0 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -22,10 +22,10 @@ //! //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as //! argument. -use cretonne::ir::condcodes::{FloatCC, IntCC}; -use cretonne::ir::types::*; -use cretonne::ir::{self, InstBuilder, JumpTableData, MemFlags}; -use cretonne::packed_option::ReservedValue; +use cretonne_codegen::ir::condcodes::{FloatCC, IntCC}; +use cretonne_codegen::ir::types::*; +use cretonne_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags}; +use cretonne_codegen::packed_option::ReservedValue; use cretonne_frontend::{FunctionBuilder, Variable}; use environ::{FuncEnvironment, GlobalValue}; use state::{ControlStackFrame, TranslationState}; diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 2bac8ed783..598071c325 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -1,9 +1,9 @@ //! "Dummy" environment for testing wasm translation. -use cretonne::cursor::FuncCursor; -use cretonne::ir::types::*; -use cretonne::ir::{self, InstBuilder}; -use cretonne::settings; +use cretonne_codegen::cursor::FuncCursor; +use cretonne_codegen::ir::types::*; +use cretonne_codegen::ir::{self, InstBuilder}; +use cretonne_codegen::settings; use environ::{FuncEnvironment, GlobalValue, ModuleEnvironment}; use func_translator::FuncTranslator; use std::error::Error; diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 8c313f0ef3..51aba3fd7e 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -1,8 +1,8 @@ //! All the runtime support necessary for the wasm to cretonne translation is formalized by the //! traits `FunctionEnvironment` and `ModuleEnvironment`. -use cretonne::cursor::FuncCursor; -use cretonne::ir::{self, InstBuilder}; -use cretonne::settings::Flags; +use cretonne_codegen::cursor::FuncCursor; +use cretonne_codegen::ir::{self, InstBuilder}; +use cretonne_codegen::settings::Flags; use std::string::String; use std::vec::Vec; use translation_utils::{FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 537aeed0c2..4801cccf11 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -5,10 +5,10 @@ //! WebAssembly module and the runtime environment. use code_translator::translate_operator; -use cretonne::entity::EntityRef; -use cretonne::ir::{self, Ebb, InstBuilder}; -use cretonne::result::{CtonError, CtonResult}; -use cretonne::timing; +use cretonne_codegen::entity::EntityRef; +use cretonne_codegen::ir::{self, Ebb, InstBuilder}; +use cretonne_codegen::result::{CtonError, CtonResult}; +use cretonne_codegen::timing; use cretonne_frontend::{FunctionBuilder, FunctionBuilderContext, Variable}; use environ::FuncEnvironment; use state::TranslationState; @@ -233,8 +233,8 @@ fn cur_srcloc(reader: &BinaryReader) -> ir::SourceLoc { #[cfg(test)] mod tests { use super::FuncTranslator; - use cretonne::ir::types::I32; - use cretonne::{ir, Context}; + use cretonne_codegen::ir::types::I32; + use cretonne_codegen::{ir, Context}; use environ::{DummyEnvironment, FuncEnvironment}; #[test] diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 197d228eab..1f0f063e37 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -14,7 +14,7 @@ #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, redundant_field_names))] #[macro_use(dbg)] -extern crate cretonne; +extern crate cretonne_codegen; extern crate cretonne_frontend; extern crate wasmparser; diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index e41712a6d6..f1092651d6 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -1,6 +1,6 @@ //! Translation skeleton that traverses the whole WebAssembly module and call helper functions //! to deal with each part of it. -use cretonne::timing; +use cretonne_codegen::timing; use environ::ModuleEnvironment; use sections_translator::{parse_data_section, parse_elements_section, parse_export_section, parse_function_section, parse_function_signatures, parse_global_section, @@ -11,7 +11,7 @@ use wasmparser::{BinaryReaderError, Parser, ParserInput, ParserState, SectionCod use std::string::String; /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IR -/// [`Function`](../cretonne/ir/function/struct.Function.html). +/// [`Function`](../codegen/ir/function/struct.Function.html). /// Returns the functions and also the mappings for imported functions and signature between the /// indexes in the wasm module and the indexes inside each functions. pub fn translate_module<'data>( diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index ce5ed05995..b744c505e4 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -7,8 +7,7 @@ //! The special case of the initialize expressions for table elements offsets or global variables //! is handled, according to the semantics of WebAssembly, to only specific expressions that are //! interpreted on the fly. -use cretonne; -use cretonne::ir::{AbiParam, CallConv, Signature}; +use cretonne_codegen::ir::{self, AbiParam, CallConv, Signature}; use environ::ModuleEnvironment; use std::str::from_utf8; use std::string::String; @@ -38,13 +37,13 @@ pub fn parse_function_signatures( }) => { let mut sig = Signature::new(CallConv::SystemV); sig.params.extend(params.iter().map(|ty| { - let cret_arg: cretonne::ir::Type = type_to_type(ty).expect( + let cret_arg: ir::Type = type_to_type(ty).expect( "only numeric types are supported in function signatures", ); AbiParam::new(cret_arg) })); sig.returns.extend(returns.iter().map(|ty| { - let cret_arg: cretonne::ir::Type = type_to_type(ty).expect( + let cret_arg: ir::Type = type_to_type(ty).expect( "only numeric types are supported in function signatures", ); AbiParam::new(cret_arg) diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index f95619d52b..c322906c95 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -3,7 +3,7 @@ //! The `TranslationState` struct defined in this module is used to keep track of the WebAssembly //! value and control stacks during the translation of a single function. -use cretonne::ir::{self, Ebb, Inst, Value}; +use cretonne_codegen::ir::{self, Ebb, Inst, Value}; use environ::{FuncEnvironment, GlobalValue}; use std::collections::HashMap; use std::vec::Vec; diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index bac50be7a6..0089c5a75d 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -1,5 +1,5 @@ //! Helper functions and structures for the translation. -use cretonne; +use cretonne_codegen::ir; use std::u32; use wasmparser; @@ -18,7 +18,7 @@ pub type SignatureIndex = usize; #[derive(Debug, Clone, Copy)] pub struct Global { /// The type of the value stored in the global. - pub ty: cretonne::ir::Type, + pub ty: ir::Type, /// A flag indicating whether the value may change at runtime. pub mutability: bool, /// The source of the initial value. @@ -56,7 +56,7 @@ pub struct Table { /// WebAssembly table element. Can be a function or a scalar type. #[derive(Debug, Clone, Copy)] pub enum TableElementType { - Val(cretonne::ir::Type), + Val(ir::Type), Func(), } @@ -72,24 +72,24 @@ pub struct Memory { } /// Helper function translating wasmparser types to Cretonne types when possible. -pub fn type_to_type(ty: &wasmparser::Type) -> Result { +pub fn type_to_type(ty: &wasmparser::Type) -> Result { match *ty { - wasmparser::Type::I32 => Ok(cretonne::ir::types::I32), - wasmparser::Type::I64 => Ok(cretonne::ir::types::I64), - wasmparser::Type::F32 => Ok(cretonne::ir::types::F32), - wasmparser::Type::F64 => Ok(cretonne::ir::types::F64), + wasmparser::Type::I32 => Ok(ir::types::I32), + wasmparser::Type::I64 => Ok(ir::types::I64), + wasmparser::Type::F32 => Ok(ir::types::F32), + wasmparser::Type::F64 => Ok(ir::types::F64), _ => Err(()), } } /// Turns a `wasmparser` `f32` into a `Cretonne` one. -pub fn f32_translation(x: wasmparser::Ieee32) -> cretonne::ir::immediates::Ieee32 { - cretonne::ir::immediates::Ieee32::with_bits(x.bits()) +pub fn f32_translation(x: wasmparser::Ieee32) -> ir::immediates::Ieee32 { + ir::immediates::Ieee32::with_bits(x.bits()) } /// Turns a `wasmparser` `f64` into a `Cretonne` one. -pub fn f64_translation(x: wasmparser::Ieee64) -> cretonne::ir::immediates::Ieee64 { - cretonne::ir::immediates::Ieee64::with_bits(x.bits()) +pub fn f64_translation(x: wasmparser::Ieee64) -> ir::immediates::Ieee64 { + ir::immediates::Ieee64::with_bits(x.bits()) } /// Translate a `wasmparser` type into its `Cretonne` equivalent, when possible diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs index fac93e6012..15feea4b55 100644 --- a/lib/wasm/tests/wasm_testsuite.rs +++ b/lib/wasm/tests/wasm_testsuite.rs @@ -1,10 +1,10 @@ -extern crate cretonne; +extern crate cretonne_codegen; extern crate cretonne_wasm; extern crate tempdir; -use cretonne::print_errors::pretty_verifier_error; -use cretonne::settings::{self, Configurable, Flags}; -use cretonne::verifier; +use cretonne_codegen::print_errors::pretty_verifier_error; +use cretonne_codegen::settings::{self, Configurable, Flags}; +use cretonne_codegen::verifier; use cretonne_wasm::{translate_module, DummyEnvironment}; use std::error::Error; use std::fs; From f43b6aca1a95534e08e7292d81596acdc4531b7e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 08:55:46 -0700 Subject: [PATCH 1700/3084] Use lower-case letters for github URLs. This makes it a little more consistent; now, "cretonne" is never capitalized in identifier, path, or URL contexts. It is capitalized in natural language contexts when referring to the project. --- README.rst | 10 +++++----- cranelift/Cargo.toml | 2 +- cranelift/filetests/regalloc/coalescing-207.cton | 2 +- cranelift/filetests/regalloc/coalescing-216.cton | 2 +- cranelift/filetests/regalloc/reload-208.cton | 2 +- cranelift/publish-all.sh | 2 +- lib/codegen/Cargo.toml | 4 ++-- lib/entity/Cargo.toml | 4 ++-- lib/filetests/Cargo.toml | 2 +- lib/frontend/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/wasm/Cargo.toml | 4 ++-- 13 files changed, 23 insertions(+), 23 deletions(-) diff --git a/README.rst b/README.rst index eae5f507ba..5d7ae4fde1 100644 --- a/README.rst +++ b/README.rst @@ -10,12 +10,12 @@ machine code. :target: https://cretonne.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status -.. image:: https://travis-ci.org/Cretonne/cretonne.svg?branch=master - :target: https://travis-ci.org/Cretonne/cretonne +.. image:: https://travis-ci.org/cretonne/cretonne.svg?branch=master + :target: https://travis-ci.org/cretonne/cretonne :alt: Build Status -.. image:: https://badges.gitter.im/Cretonne/cretonne.png - :target: https://gitter.im/Cretonne/Lobby/~chat +.. image:: https://badges.gitter.im/cretonne/cretonne.png + :target: https://gitter.im/cretonne/Lobby/~chat :alt: Gitter chat For more information, see `the documentation @@ -37,7 +37,7 @@ produced by Cretonne is not yet impressive, though we have plans to fix that. The core codegen crates have minimal dependencies, and do not require any host floating-point support. Support for `no_std` mode in the core codegen crates is -`in development `_. +`in development `_. Cretonne does not yet perform mitigations for Spectre or related security issues, though it may do so in the future. It does not currently make any diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 62b485ae50..b8acb84d62 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -5,7 +5,7 @@ version = "0.4.4" description = "Binaries for testing the Cretonne libraries" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/Cretonne/cretonne" +repository = "https://github.com/cretonne/cretonne" publish = false [[bin]] diff --git a/cranelift/filetests/regalloc/coalescing-207.cton b/cranelift/filetests/regalloc/coalescing-207.cton index 0deaa454f4..d8af887da3 100644 --- a/cranelift/filetests/regalloc/coalescing-207.cton +++ b/cranelift/filetests/regalloc/coalescing-207.cton @@ -2,7 +2,7 @@ test regalloc set is_64bit isa x86 haswell -; Reported as https://github.com/Cretonne/cretonne/issues/207 +; Reported as https://github.com/cretonne/cretonne/issues/207 ; ; The coalescer creates a virtual register with two interfering values. function %pr207(i64 vmctx, i32, i32) -> i32 system_v { diff --git a/cranelift/filetests/regalloc/coalescing-216.cton b/cranelift/filetests/regalloc/coalescing-216.cton index 1138e2578a..fc986f7fec 100644 --- a/cranelift/filetests/regalloc/coalescing-216.cton +++ b/cranelift/filetests/regalloc/coalescing-216.cton @@ -2,7 +2,7 @@ test regalloc set is_64bit isa x86 haswell -; Reported as https://github.com/Cretonne/cretonne/issues/216 from the Binaryen fuzzer. +; Reported as https://github.com/cretonne/cretonne/issues/216 from the Binaryen fuzzer. ; ; The (old) coalescer creates a virtual register with two identical values. function %pr216(i32 [%rdi], i64 vmctx [%rsi]) -> i64 [%rax] system_v { diff --git a/cranelift/filetests/regalloc/reload-208.cton b/cranelift/filetests/regalloc/reload-208.cton index bb775acebb..259c4a169b 100644 --- a/cranelift/filetests/regalloc/reload-208.cton +++ b/cranelift/filetests/regalloc/reload-208.cton @@ -4,7 +4,7 @@ isa x86 haswell ; regex: V=v\d+ -; Filed as https://github.com/Cretonne/cretonne/issues/208 +; Filed as https://github.com/cretonne/cretonne/issues/208 ; ; The verifier complains about a branch argument that is not in the same virtual register as the ; corresponding EBB argument. diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 3b17f7cf1a..27d507f8b4 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -31,4 +31,4 @@ for crate in entity codegen frontend native reader wasm ; do echo cargo publish --manifest-path "lib/$crate/Cargo.toml" done echo -echo Then, go to https://github.com/Cretonne/cretonne/releases/ and define a new release. +echo Then, go to https://github.com/cretonne/cretonne/releases/ and define a new release. diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 6a5027f0f7..87dd016a21 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -5,7 +5,7 @@ version = "0.4.4" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/Cretonne/cretonne" +repository = "https://github.com/cretonne/cretonne" readme = "README.md" keywords = ["compile", "compiler", "jit"] build = "build.rs" @@ -19,4 +19,4 @@ cretonne-entity = { path = "../entity", version = "0.4.4" } [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "Cretonne/cretonne" } +travis-ci = { repository = "cretonne/cretonne" } diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index fa862aef94..d93220fbe8 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -5,7 +5,7 @@ version = "0.4.4" description = "Data structures using entity references as mapping keys" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/Cretonne/cretonne" +repository = "https://github.com/cretonne/cretonne" readme = "README.md" keywords = ["entity", "set", "map"] @@ -15,4 +15,4 @@ std = [] [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "Cretonne/cretonne" } +travis-ci = { repository = "cretonne/cretonne" } diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 984d5e0629..1059a7a12a 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -5,7 +5,7 @@ version = "0.4.4" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "http://cretonne.readthedocs.io/en/latest/testing.html#file-tests" -repository = "https://github.com/Cretonne/cretonne" +repository = "https://github.com/cretonne/cretonne" publish = false [dependencies] diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 77c54a32f0..1c2b86d266 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -5,7 +5,7 @@ version = "0.4.4" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/Cretonne/cretonne" +repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] @@ -13,4 +13,4 @@ cretonne-codegen = { path = "../codegen", version = "0.4.4" } [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "Cretonne/cretonne" } +travis-ci = { repository = "cretonne/cretonne" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 35afe7441f..191e9325b5 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -3,7 +3,7 @@ name = "cretonne-native" version = "0.4.4" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" -repository = "https://github.com/Cretonne/cretonne" +repository = "https://github.com/cretonne/cretonne" license = "Apache-2.0" readme = "README.md" @@ -15,4 +15,4 @@ raw-cpuid = "3.0.0" [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "Cretonne/cretonne" } +travis-ci = { repository = "cretonne/cretonne" } diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 49afaf482a..b321da3e43 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -5,7 +5,7 @@ version = "0.4.4" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/Cretonne/cretonne" +repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] @@ -13,4 +13,4 @@ cretonne-codegen = { path = "../codegen", version = "0.4.4" } [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "Cretonne/cretonne" } +travis-ci = { repository = "cretonne/cretonne" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 8384df98d7..fbe7de7126 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -3,7 +3,7 @@ name = "cretonne-wasm" version = "0.4.4" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" -repository = "https://github.com/Cretonne/cretonne" +repository = "https://github.com/cretonne/cretonne" license = "Apache-2.0" readme = "README.md" keywords = ["webassembly", "wasm"] @@ -18,4 +18,4 @@ tempdir = "0.3.5" [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "Cretonne/cretonne" } +travis-ci = { repository = "cretonne/cretonne" } From 85662fbca97e2408ed4c4370fe9bafab86bcbcb1 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 09:13:33 -0700 Subject: [PATCH 1701/3084] Add an "umbrella" crate, which seeks to provide a convenient interface. This currently pulls in cretonne-codegen and cretonne-frontend and provides a simple prelude interface. This fixes #287. --- cranelift/Cargo.toml | 1 + cranelift/publish-all.sh | 8 +++++++- lib/umbrella/Cargo.toml | 18 ++++++++++++++++++ lib/umbrella/README.md | 3 +++ lib/umbrella/src/lib.rs | 19 +++++++++++++++++++ 5 files changed, 48 insertions(+), 1 deletion(-) create mode 100644 lib/umbrella/Cargo.toml create mode 100644 lib/umbrella/README.md create mode 100644 lib/umbrella/src/lib.rs diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index b8acb84d62..4a28857088 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -19,6 +19,7 @@ cretonne-frontend = { path = "lib/frontend", version = "0.4.4" } cretonne-wasm = { path = "lib/wasm", version = "0.4.4" } cretonne-native = { path = "lib/native", version = "0.4.4" } cretonne-filetests = { path = "lib/filetests", version = "0.4.4" } +cretonne = { path = "lib/umbrella", version = "0.4.4" } filecheck = "0.2.1" docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 27d507f8b4..108d8055c2 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -27,7 +27,13 @@ cargo update echo git commit -a -m "\"Bump version to $version"\" echo git push -for crate in entity codegen frontend native reader wasm ; do +for crate in entity codegen frontend native reader wasm umbrella ; do + if [ "$crate" == "umbrella" ]; then + dir="cretonne" + else + dir="$crate" + fi + echo cargo publish --manifest-path "lib/$crate/Cargo.toml" done echo diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml new file mode 100644 index 0000000000..d1f7f95786 --- /dev/null +++ b/lib/umbrella/Cargo.toml @@ -0,0 +1,18 @@ +[package] +authors = ["The Cretonne Project Developers"] +name = "cretonne" +version = "0.4.4" +description = "Umbrella for commonly-used cretonne crates" +license = "Apache-2.0" +documentation = "https://cretonne.readthedocs.io/" +repository = "https://github.com/cretonne/cretonne" +readme = "README.md" +keywords = ["compile", "compiler", "jit"] + +[dependencies] +cretonne-codegen = { path = "../codegen", version = "0.4.4" } +cretonne-frontend = { path = "../frontend", version = "0.4.4" } + +[badges] +maintenance = { status = "experimental" } +travis-ci = { repository = "cretonne/cretonne" } diff --git a/lib/umbrella/README.md b/lib/umbrella/README.md new file mode 100644 index 0000000000..f9c2ee3fbf --- /dev/null +++ b/lib/umbrella/README.md @@ -0,0 +1,3 @@ +This is an umbrella crate which contains no code of its own, but pulls in +other cretonne library crates to provide a convenient one-line dependency +for common use cases. diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs new file mode 100644 index 0000000000..a70a97e86f --- /dev/null +++ b/lib/umbrella/src/lib.rs @@ -0,0 +1,19 @@ +//! Cretonne umbrella crate, providing a convenient one-line dependency. + +#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] + +pub extern crate cretonne_codegen; +pub extern crate cretonne_frontend; + +/// A prelude providing convenient access to commonly-used cretonne features. Use +/// as `use cretonne::prelude::*`. +pub mod prelude { + pub use cretonne_codegen; + pub use cretonne_codegen::entity::EntityRef; + pub use cretonne_codegen::ir::{AbiParam, InstBuilder, Value, Ebb, Signature, CallConv}; + pub use cretonne_codegen::ir::types; + pub use cretonne_codegen::ir::condcodes::IntCC; + + pub use cretonne_frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; +} From 635d14c2516280829edaa2398dfd28ba61d749e4 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 09:29:45 -0700 Subject: [PATCH 1702/3084] Rename GlobalVarData::VmCtx for consistency with ArgumentPurpose::VMContext. --- lib/codegen/src/ir/globalvar.rs | 4 ++-- lib/codegen/src/legalizer/globalvar.rs | 2 +- lib/reader/src/parser.rs | 2 +- lib/wasm/src/environ/dummy.rs | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/codegen/src/ir/globalvar.rs b/lib/codegen/src/ir/globalvar.rs index ad75496362..399f3b7ae4 100644 --- a/lib/codegen/src/ir/globalvar.rs +++ b/lib/codegen/src/ir/globalvar.rs @@ -9,7 +9,7 @@ use std::fmt; pub enum GlobalVarData { /// Variable is part of the VM context struct, it's address is a constant offset from the VM /// context pointer. - VmCtx { + VMContext { /// Offset from the `vmctx` pointer to this global. offset: Offset32, }, @@ -54,7 +54,7 @@ impl GlobalVarData { impl fmt::Display for GlobalVarData { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match *self { - GlobalVarData::VmCtx { offset } => write!(f, "vmctx{}", offset), + GlobalVarData::VMContext { offset } => write!(f, "vmctx{}", offset), GlobalVarData::Deref { base, offset } => write!(f, "deref({}){}", base, offset), GlobalVarData::Sym { ref name, diff --git a/lib/codegen/src/legalizer/globalvar.rs b/lib/codegen/src/legalizer/globalvar.rs index 27494bfd87..9c6e701cf2 100644 --- a/lib/codegen/src/legalizer/globalvar.rs +++ b/lib/codegen/src/legalizer/globalvar.rs @@ -25,7 +25,7 @@ pub fn expand_global_addr( }; match func.global_vars[gv] { - ir::GlobalVarData::VmCtx { offset } => vmctx_addr(inst, func, offset.into()), + ir::GlobalVarData::VMContext { offset } => vmctx_addr(inst, func, offset.into()), ir::GlobalVarData::Deref { base, offset } => deref_addr(inst, func, base, offset.into()), ir::GlobalVarData::Sym { .. } => globalsym(inst, func, gv), } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 88d067112b..1c767b6759 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1100,7 +1100,7 @@ impl<'a> Parser<'a> { let data = match self.match_any_identifier("expected global variable kind")? { "vmctx" => { let offset = self.optional_offset32()?; - GlobalVarData::VmCtx { offset } + GlobalVarData::VMContext { offset } } "deref" => { self.match_token( diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 598071c325..cdbb43e5d4 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -151,7 +151,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue { // Just create a dummy `vmctx` global. let offset = ((index * 8) as i32 + 8).into(); - let gv = func.create_global_var(ir::GlobalVarData::VmCtx { offset }); + let gv = func.create_global_var(ir::GlobalVarData::VMContext { offset }); GlobalValue::Memory { gv, ty: self.mod_info.globals[index].entity.ty, @@ -160,7 +160,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ fn make_heap(&mut self, func: &mut ir::Function, _index: MemoryIndex) -> ir::Heap { // Create a static heap whose base address is stored at `vmctx+0`. - let gv = func.create_global_var(ir::GlobalVarData::VmCtx { offset: 0.into() }); + let gv = func.create_global_var(ir::GlobalVarData::VMContext { offset: 0.into() }); func.create_heap(ir::HeapData { base: ir::HeapBase::GlobalVar(gv), From e2f6705e284d9a12fa3b0037307b615ebdf70903 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 09:36:39 -0700 Subject: [PATCH 1703/3084] Add a comment about why there's no `is_pie` setting. --- lib/codegen/meta/base/settings.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/codegen/meta/base/settings.py b/lib/codegen/meta/base/settings.py index a59904a572..ff1551c151 100644 --- a/lib/codegen/meta/base/settings.py +++ b/lib/codegen/meta/base/settings.py @@ -29,6 +29,9 @@ enable_verifier = BoolSetting( is_64bit = BoolSetting("Enable 64-bit code generation") +# Note that Cretonne doesn't currently need an is_pie flag, because PIE is just +# PIC where symbols can't be pre-empted, which can be expressed with the +# `colocated` flag on external functions and global variables. is_pic = BoolSetting("Enable Position-Independent Code generation") return_at_end = BoolSetting( From a9edb2841402e12ea6594f51a19a9727ee4303a4 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 09:43:05 -0700 Subject: [PATCH 1704/3084] Rename InstructionFormat::IndirectCall for consistency with Opcode::CallIndirect. --- lib/codegen/meta/base/entities.py | 2 +- lib/codegen/meta/base/formats.py | 2 +- lib/codegen/meta/isa/riscv/recipes.py | 8 ++++---- lib/codegen/meta/isa/x86/recipes.py | 4 ++-- lib/codegen/src/ir/instructions.rs | 2 +- lib/codegen/src/legalizer/call.rs | 2 +- lib/codegen/src/verifier/mod.rs | 2 +- lib/codegen/src/write.rs | 2 +- lib/reader/src/parser.rs | 4 ++-- lib/wasm/src/environ/dummy.rs | 2 +- 10 files changed, 15 insertions(+), 15 deletions(-) diff --git a/lib/codegen/meta/base/entities.py b/lib/codegen/meta/base/entities.py index 614b4d6284..996913471b 100644 --- a/lib/codegen/meta/base/entities.py +++ b/lib/codegen/meta/base/entities.py @@ -20,7 +20,7 @@ stack_slot = EntityRefKind('stack_slot', 'A stack slot.') global_var = EntityRefKind('global_var', 'A global variable.') #: A reference to a function sugnature declared in the function preamble. -#: Tbis is used to provide the call signature in an indirect call instruction. +#: This is used to provide the call signature in a call_indirect instruction. sig_ref = EntityRefKind('sig_ref', 'A function signature.') #: A reference to an external function declared in the function preamble. diff --git a/lib/codegen/meta/base/formats.py b/lib/codegen/meta/base/formats.py index 595f09b9fa..89ef30881f 100644 --- a/lib/codegen/meta/base/formats.py +++ b/lib/codegen/meta/base/formats.py @@ -53,7 +53,7 @@ BranchIcmp = InstructionFormat(intcc, VALUE, VALUE, ebb, VARIABLE_ARGS) BranchTable = InstructionFormat(VALUE, entities.jump_table) Call = InstructionFormat(func_ref, VARIABLE_ARGS) -IndirectCall = InstructionFormat(sig_ref, VALUE, VARIABLE_ARGS) +CallIndirect = InstructionFormat(sig_ref, VALUE, VARIABLE_ARGS) FuncAddr = InstructionFormat(func_ref) Load = InstructionFormat(memflags, VALUE, offset32) diff --git a/lib/codegen/meta/isa/riscv/recipes.py b/lib/codegen/meta/isa/riscv/recipes.py index c657c7de47..170dd914d3 100644 --- a/lib/codegen/meta/isa/riscv/recipes.py +++ b/lib/codegen/meta/isa/riscv/recipes.py @@ -14,7 +14,7 @@ from cdsl.predicates import IsSignedInt from cdsl.registers import Stack from base.formats import Binary, BinaryImm, MultiAry, IntCompare, IntCompareImm from base.formats import Unary, UnaryImm, BranchIcmp, Branch, Jump -from base.formats import Call, IndirectCall, RegMove +from base.formats import Call, CallIndirect, RegMove from .registers import GPR # The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit @@ -140,11 +140,11 @@ Iret = EncRecipe( ); ''') -# I-type encoding for `jalr` as an indirect call. +# I-type encoding for `jalr` as a call_indirect. Icall = EncRecipe( - 'Icall', IndirectCall, size=4, ins=GPR, outs=(), + 'Icall', CallIndirect, size=4, ins=GPR, outs=(), emit=''' - // Indirect instructions are jalr with rd=%x1. + // call_indirect instructions are jalr with rd=%x1. put_i( bits, in_reg0, diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py index 5d0b045ead..ea0da832b7 100644 --- a/lib/codegen/meta/isa/x86/recipes.py +++ b/lib/codegen/meta/isa/x86/recipes.py @@ -7,7 +7,7 @@ from cdsl.predicates import IsSignedInt, IsEqual, Or from cdsl.registers import RegClass from base.formats import Unary, UnaryImm, UnaryBool, Binary, BinaryImm from base.formats import MultiAry, NullAry -from base.formats import Trap, Call, IndirectCall, Store, Load +from base.formats import Trap, Call, CallIndirect, Store, Load from base.formats import IntCompare, IntCompareImm, FloatCompare from base.formats import IntCond, FloatCond from base.formats import IntSelect, IntCondTrap, FloatCondTrap @@ -1055,7 +1055,7 @@ call_plt_id = TailRecipe( ''') call_r = TailRecipe( - 'call_r', IndirectCall, size=1, ins=GPR, outs=(), + 'call_r', CallIndirect, size=1, ins=GPR, outs=(), emit=''' PUT_OP(bits, rex1(in_reg0), sink); modrm_r_bits(in_reg0, bits, sink); diff --git a/lib/codegen/src/ir/instructions.rs b/lib/codegen/src/ir/instructions.rs index ff415af22b..d4ca4a1396 100644 --- a/lib/codegen/src/ir/instructions.rs +++ b/lib/codegen/src/ir/instructions.rs @@ -248,7 +248,7 @@ impl InstructionData { InstructionData::Call { func_ref, ref args, .. } => { CallInfo::Direct(func_ref, args.as_slice(pool)) } - InstructionData::IndirectCall { sig_ref, ref args, .. } => { + InstructionData::CallIndirect { sig_ref, ref args, .. } => { CallInfo::Indirect(sig_ref, &args.as_slice(pool)[1..]) } _ => { diff --git a/lib/codegen/src/legalizer/call.rs b/lib/codegen/src/legalizer/call.rs index d1d8bcba08..ed499f53f3 100644 --- a/lib/codegen/src/legalizer/call.rs +++ b/lib/codegen/src/legalizer/call.rs @@ -51,7 +51,7 @@ pub fn expand_call( ); } - func.dfg.replace(inst).IndirectCall( + func.dfg.replace(inst).CallIndirect( ir::Opcode::CallIndirect, ptr_ty, sig, diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs index abda326c03..692a435545 100644 --- a/lib/codegen/src/verifier/mod.rs +++ b/lib/codegen/src/verifier/mod.rs @@ -319,7 +319,7 @@ impl<'a> Verifier<'a> { self.verify_func_ref(inst, func_ref)?; self.verify_value_list(inst, args)?; } - IndirectCall { sig_ref, ref args, .. } => { + CallIndirect { sig_ref, ref args, .. } => { self.verify_sig_ref(inst, sig_ref)?; self.verify_value_list(inst, args)?; } diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs index 22d78bdb82..073ae22f11 100644 --- a/lib/codegen/src/write.rs +++ b/lib/codegen/src/write.rs @@ -349,7 +349,7 @@ pub fn write_operands( Call { func_ref, ref args, .. } => { write!(w, " {}({})", func_ref, DisplayValues(args.as_slice(pool))) } - IndirectCall { sig_ref, ref args, .. } => { + CallIndirect { sig_ref, ref args, .. } => { let args = args.as_slice(pool); write!( w, diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 1c767b6759..e96e415afc 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -2179,7 +2179,7 @@ impl<'a> Parser<'a> { args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), } } - InstructionFormat::IndirectCall => { + InstructionFormat::CallIndirect => { let sig_ref = self.match_sig("expected signature reference")?; ctx.check_sig(sig_ref, &self.loc)?; self.match_token( @@ -2196,7 +2196,7 @@ impl<'a> Parser<'a> { Token::RPar, "expected ')' after arguments", )?; - InstructionData::IndirectCall { + InstructionData::CallIndirect { opcode, sig_ref, args: args.into_value_list(&[callee], &mut ctx.function.dfg.value_lists), diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index cdbb43e5d4..c2eea8219f 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -226,7 +226,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ args.push(vmctx, &mut pos.func.dfg.value_lists); pos.ins() - .IndirectCall(ir::Opcode::CallIndirect, ir::types::VOID, sig_ref, args) + .CallIndirect(ir::Opcode::CallIndirect, ir::types::VOID, sig_ref, args) .0 } From 76db9f022d3903f6f93a0b641c14cfb621f69c82 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 10:52:36 -0700 Subject: [PATCH 1705/3084] [WIP] Module API (#294) * Initial skeleton. * Add basic faerie support. This adds enough functionality to enable simple .o file writing through faerie. This included adding the functionality to Module to support RelocSink implementations. * Add basic SimpleJIT support. This adds enough functionality to enable a simple program to be jitted and executed. * Make declare_func_in_func take a Function instead of a Context. It only needs the Function, and sometimes it's useful to call it from places that don't have a full Context. * Temporarily disable local and exported global variables in the Faerie backend. Faerie assumes these variables use pc-relative offset instructions, and Cretonne is not yet emitting those instructions. * FaerieBackend depends on PIC. Faerie itself only supports PIC objects for now, so add an assert to Cretonne to check that it's using a PIC target flag. * SimpleJIT support for data objects. * Preliminary faerie support for data objects. * Support for data objects in faerie using the new colocated flag. * Unit tests for DataContext and friends. * Add a Module::consume() function. This consumes the Module and returns the contained Backend, so that users can call Backend-specific functions with it. For example, the Faerie backend has functions to write an object file. * Update the new crates to version 0.4.4. * Make FaerieBackend own its TargetIsa. This simplifies its interface, as it eliminates a lifetime parameter. While we may eventually want to look into allowing multiple clients to share a TargetIsa, it isn't worth the complexity for FaerieBackend right now. * Don't try to protect faerie from multiple declarations. Let faerie decide for itself whether it wants to consider two declarations to be compatible. * Use debug_assert_eq rather than debug_assert with ==. * Fix FaerieRelocSink's reloc_external to handle data object names. * Relax the asserts in get_function_definition and get_data_definition. These functions don't require definable symbols, but they do require that definable symbols be defined. This is needed for the simplejit backend. * Add a function to the faerie backend to retrieve the artifact name. * Sync up with cretonne changes. --- cranelift/Cargo.toml | 3 + lib/codegen/src/binemit/mod.rs | 2 +- lib/codegen/src/ir/extfunc.rs | 10 +- lib/codegen/src/ir/valueloc.rs | 2 +- lib/faerie/Cargo.toml | 20 ++ lib/faerie/README.md | 4 + lib/faerie/src/backend.rs | 298 ++++++++++++++++++ lib/faerie/src/container.rs | 33 ++ lib/faerie/src/lib.rs | 19 ++ lib/faerie/src/target.rs | 18 ++ lib/module/Cargo.toml | 17 ++ lib/module/README.md | 3 + lib/module/src/backend.rs | 97 ++++++ lib/module/src/data_context.rs | 205 +++++++++++++ lib/module/src/lib.rs | 15 + lib/module/src/module.rs | 541 +++++++++++++++++++++++++++++++++ lib/simplejit/Cargo.toml | 21 ++ lib/simplejit/README.md | 4 + lib/simplejit/src/backend.rs | 365 ++++++++++++++++++++++ lib/simplejit/src/lib.rs | 15 + lib/simplejit/src/memory.rs | 108 +++++++ 21 files changed, 1793 insertions(+), 7 deletions(-) create mode 100644 lib/faerie/Cargo.toml create mode 100644 lib/faerie/README.md create mode 100644 lib/faerie/src/backend.rs create mode 100644 lib/faerie/src/container.rs create mode 100644 lib/faerie/src/lib.rs create mode 100644 lib/faerie/src/target.rs create mode 100644 lib/module/Cargo.toml create mode 100644 lib/module/README.md create mode 100644 lib/module/src/backend.rs create mode 100644 lib/module/src/data_context.rs create mode 100644 lib/module/src/lib.rs create mode 100644 lib/module/src/module.rs create mode 100644 lib/simplejit/Cargo.toml create mode 100644 lib/simplejit/README.md create mode 100644 lib/simplejit/src/backend.rs create mode 100644 lib/simplejit/src/lib.rs create mode 100644 lib/simplejit/src/memory.rs diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 4a28857088..4ca0fa7760 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -19,6 +19,9 @@ cretonne-frontend = { path = "lib/frontend", version = "0.4.4" } cretonne-wasm = { path = "lib/wasm", version = "0.4.4" } cretonne-native = { path = "lib/native", version = "0.4.4" } cretonne-filetests = { path = "lib/filetests", version = "0.4.4" } +cretonne-module = { path = "lib/module", version = "0.4.4" } +cretonne-faerie = { path = "lib/faerie", version = "0.4.4" } +cretonne-simplejit = { path = "lib/simplejit", version = "0.4.4" } cretonne = { path = "lib/umbrella", version = "0.4.4" } filecheck = "0.2.1" docopt = "0.8.0" diff --git a/lib/codegen/src/binemit/mod.rs b/lib/codegen/src/binemit/mod.rs index 9c78e4f784..8549b77aad 100644 --- a/lib/codegen/src/binemit/mod.rs +++ b/lib/codegen/src/binemit/mod.rs @@ -23,7 +23,7 @@ pub type CodeOffset = u32; pub type Addend = i64; /// Relocation kinds for every ISA -#[derive(Debug)] +#[derive(Copy, Clone, Debug)] pub enum Reloc { /// absolute 4-byte Abs4, diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs index 474b463727..afb32fee81 100644 --- a/lib/codegen/src/ir/extfunc.rs +++ b/lib/codegen/src/ir/extfunc.rs @@ -19,7 +19,7 @@ use std::vec::Vec; /// /// A signature can optionally include ISA-specific ABI information which specifies exactly how /// arguments and return values are passed. -#[derive(Clone, Debug)] +#[derive(Clone, Debug, PartialEq, Eq, Hash)] pub struct Signature { /// The arguments passed to the function. pub params: Vec, @@ -123,7 +123,7 @@ impl fmt::Display for Signature { /// /// This describes the value type being passed to or from a function along with flags that affect /// how the argument is passed. -#[derive(Copy, Clone, Debug)] +#[derive(Copy, Clone, Debug, PartialEq, Eq, Hash)] pub struct AbiParam { /// Type of the argument value. pub value_type: Type, @@ -225,7 +225,7 @@ impl fmt::Display for AbiParam { /// /// On some architectures, small integer function arguments are extended to the width of a /// general-purpose register. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)] pub enum ArgumentExtension { /// No extension, high bits are indeterminate. None, @@ -242,7 +242,7 @@ pub enum ArgumentExtension { /// frame pointers and callee-saved registers. /// /// The argument purpose is used to indicate any special meaning of an argument or return value. -#[derive(Copy, Clone, PartialEq, Eq, Debug)] +#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)] pub enum ArgumentPurpose { /// A normal user program value passed to or from a function. Normal, @@ -348,7 +348,7 @@ impl fmt::Display for ExtFuncData { /// and how stack frames are managed. Since all of these details depend on both the instruction set /// architecture and possibly the operating system, a function's calling convention is only fully /// determined by a `(TargetIsa, CallConv)` tuple. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] pub enum CallConv { /// The System V-style calling convention. /// diff --git a/lib/codegen/src/ir/valueloc.rs b/lib/codegen/src/ir/valueloc.rs index aecf23d0bb..7f68e09792 100644 --- a/lib/codegen/src/ir/valueloc.rs +++ b/lib/codegen/src/ir/valueloc.rs @@ -95,7 +95,7 @@ impl<'a> fmt::Display for DisplayValueLoc<'a> { /// outgoing arguments. /// - For register arguments, there is usually no difference, but if we ever add support for a /// register-window ISA like SPARC, register arguments would also need to be translated. -#[derive(Copy, Clone, Debug)] +#[derive(Copy, Clone, Debug, Eq, PartialEq, Hash)] pub enum ArgumentLoc { /// This argument has not been assigned to a location yet. Unassigned, diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml new file mode 100644 index 0000000000..32434077f7 --- /dev/null +++ b/lib/faerie/Cargo.toml @@ -0,0 +1,20 @@ +[package] +name = "cretonne-faerie" +version = "0.4.4" +authors = ["The Cretonne Project Developers"] +description = "Emit Cretonne output to native object files with Faerie" +repository = "https://github.com/Cretonne/cretonne" +documentation = "https://cretonne.readthedocs.io/" +license = "Apache-2.0" +readme = "README.md" + +[dependencies] +cretonne-codegen = { path = "../codegen", version = "0.4.4" } +cretonne-module = { path = "../module", version = "0.4.4" } +faerie = "0.1.0" +goblin = "0.0.14" +failure = "0.1.1" + +[badges] +maintenance = { status = "experimental" } +travis-ci = { repository = "Cretonne/cretonne" } diff --git a/lib/faerie/README.md b/lib/faerie/README.md new file mode 100644 index 0000000000..7817ccb21e --- /dev/null +++ b/lib/faerie/README.md @@ -0,0 +1,4 @@ +This crate contains a library that enables +[Cretonne](https://crates.io/crates/cretonne) +to emit native object (".o") files, using the +[Faerie](https://crates.io/crates/faerie) library. diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs new file mode 100644 index 0000000000..a284883927 --- /dev/null +++ b/lib/faerie/src/backend.rs @@ -0,0 +1,298 @@ +//! Defines `FaerieBackend`. + +use container; +use cretonne_codegen::binemit::{Addend, CodeOffset, Reloc, RelocSink, TrapSink}; +use cretonne_codegen::isa::TargetIsa; +use cretonne_codegen::result::CtonError; +use cretonne_codegen::{self, binemit, ir}; +use cretonne_module::{Backend, DataContext, Linkage, ModuleNamespace, Init, DataDescription}; +use faerie; +use failure::Error; +use std::fs::File; +use target; + +pub struct FaerieCompiledFunction {} + +pub struct FaerieCompiledData {} + +/// A `FaerieBackend` implements `Backend` and emits ".o" files using the `faerie` library. +pub struct FaerieBackend { + isa: Box, + artifact: faerie::Artifact, + format: container::Format, +} + +impl FaerieBackend { + /// Create a new `FaerieBackend` using the given Cretonne target. + pub fn new( + isa: Box, + name: String, + format: container::Format, + ) -> Result { + debug_assert!(isa.flags().is_pic(), "faerie requires PIC"); + let faerie_target = target::translate(&*isa)?; + Ok(Self { + isa, + artifact: faerie::Artifact::new(faerie_target, name), + format, + }) + } + + /// Return the name of the output file. This is the name passed into `new`. + pub fn name(&self) -> &str { + &self.artifact.name + } + + /// Call `emit` on the faerie `Artifact`, producing bytes in memory. + pub fn emit(&self) -> Result, Error> { + match self.format { + container::Format::ELF => self.artifact.emit::(), + container::Format::MachO => self.artifact.emit::(), + } + } + + /// Call `write` on the faerie `Artifact`, writing to a file. + pub fn write(&self, sink: File) -> Result<(), Error> { + match self.format { + container::Format::ELF => self.artifact.write::(sink), + container::Format::MachO => self.artifact.write::(sink), + } + } +} + +impl Backend for FaerieBackend { + type CompiledFunction = FaerieCompiledFunction; + type CompiledData = FaerieCompiledData; + + // There's no need to return invidual artifacts; we're writing them into + // the output file instead. + type FinalizedFunction = (); + type FinalizedData = (); + + fn isa(&self) -> &TargetIsa { + &*self.isa + } + + fn declare_function(&mut self, name: &str, linkage: Linkage) { + self.artifact + .declare(name, translate_function_linkage(linkage)) + .expect("inconsistent declarations"); + } + + fn declare_data(&mut self, name: &str, linkage: Linkage, writable: bool) { + self.artifact + .declare(name, translate_data_linkage(linkage, writable)) + .expect("inconsistent declarations"); + } + + fn define_function( + &mut self, + name: &str, + ctx: &cretonne_codegen::Context, + namespace: &ModuleNamespace, + code_size: u32, + ) -> Result { + let mut code: Vec = Vec::with_capacity(code_size as usize); + code.resize(code_size as usize, 0); + + // Non-lexical lifetimes would obviate the braces here. + { + let mut reloc_sink = FaerieRelocSink { + format: self.format, + artifact: &mut self.artifact, + name, + namespace, + }; + let mut trap_sink = FaerieTrapSink {}; + + ctx.emit_to_memory( + code.as_mut_ptr(), + &mut reloc_sink, + &mut trap_sink, + &*self.isa, + ); + } + + self.artifact.define(name, code).expect( + "inconsistent declaration", + ); + Ok(FaerieCompiledFunction {}) + } + + fn define_data( + &mut self, + name: &str, + data_ctx: &DataContext, + namespace: &ModuleNamespace, + ) -> Result { + let &DataDescription { + writable: _writable, + ref init, + ref function_decls, + ref data_decls, + ref function_relocs, + ref data_relocs, + } = data_ctx.description(); + + let size = init.size(); + let mut bytes = Vec::with_capacity(size); + match *init { + Init::Uninitialized => { + panic!("data is not initialized yet"); + } + Init::Zeros { .. } => { + bytes.resize(size, 0); + } + Init::Bytes { ref contents } => { + bytes.extend_from_slice(contents); + } + } + + // TODO: Change the signature of this function to use something other + // than `CtonError`, as `CtonError` can't convey faerie's errors. + for &(offset, id) in function_relocs { + let to = &namespace.get_function_decl(&function_decls[id]).name; + self.artifact + .link(faerie::Link { + from: name, + to, + at: offset as usize, + }) + .map_err(|_e| CtonError::InvalidInput)?; + } + for &(offset, id, addend) in data_relocs { + debug_assert_eq!( + addend, + 0, + "faerie doesn't support addends in data section relocations yet" + ); + let to = &namespace.get_data_decl(&data_decls[id]).name; + self.artifact + .link(faerie::Link { + from: name, + to, + at: offset as usize, + }) + .map_err(|_e| CtonError::InvalidInput)?; + } + + self.artifact.define(name, bytes).expect( + "inconsistent declaration", + ); + Ok(FaerieCompiledData {}) + } + + fn write_data_funcaddr( + &mut self, + _data: &mut FaerieCompiledData, + _offset: usize, + _what: ir::FuncRef, + ) { + unimplemented!() + } + + fn write_data_dataaddr( + &mut self, + _data: &mut FaerieCompiledData, + _offset: usize, + _what: ir::GlobalVar, + _usize: binemit::Addend, + ) { + unimplemented!() + } + + fn finalize_function( + &mut self, + _func: &FaerieCompiledFunction, + _namespace: &ModuleNamespace, + ) { + // Nothing to do. + } + + fn finalize_data(&mut self, _data: &FaerieCompiledData, _namespace: &ModuleNamespace) { + // Nothing to do. + } +} + +fn translate_function_linkage(linkage: Linkage) -> faerie::Decl { + match linkage { + Linkage::Import => faerie::Decl::FunctionImport, + Linkage::Local => faerie::Decl::Function { global: false }, + Linkage::Preemptible | Linkage::Export => faerie::Decl::Function { global: true }, + } +} + +fn translate_data_linkage(linkage: Linkage, writable: bool) -> faerie::Decl { + match linkage { + Linkage::Import => faerie::Decl::DataImport, + Linkage::Local => { + faerie::Decl::Data { + global: false, + writeable: writable, + } + } + Linkage::Export => { + faerie::Decl::Data { + global: true, + writeable: writable, + } + } + Linkage::Preemptible => { + unimplemented!("faerie doesn't support preemptible globals yet"); + } + } +} + +struct FaerieRelocSink<'a> { + format: container::Format, + artifact: &'a mut faerie::Artifact, + name: &'a str, + namespace: &'a ModuleNamespace<'a, FaerieBackend>, +} + +impl<'a> RelocSink for FaerieRelocSink<'a> { + fn reloc_ebb(&mut self, _offset: CodeOffset, _reloc: Reloc, _ebb_offset: CodeOffset) { + unimplemented!(); + } + + fn reloc_external( + &mut self, + offset: CodeOffset, + reloc: Reloc, + name: &ir::ExternalName, + addend: Addend, + ) { + let ref_name = if self.namespace.is_function(name) { + &self.namespace.get_function_decl(name).name + } else { + &self.namespace.get_data_decl(name).name + }; + let addend_i32 = addend as i32; + debug_assert!(addend_i32 as i64 == addend); + let raw_reloc = container::raw_relocation(reloc, self.format); + self.artifact + .link_with( + faerie::Link { + from: self.name, + to: ref_name, + at: offset as usize, + }, + faerie::RelocOverride { + reloc: raw_reloc, + addend: addend_i32, + }, + ) + .expect("faerie relocation error"); + } + + fn reloc_jt(&mut self, _offset: CodeOffset, _reloc: Reloc, _jt: ir::JumpTable) { + unimplemented!(); + } +} + +struct FaerieTrapSink {} + +impl TrapSink for FaerieTrapSink { + // Ignore traps for now. For now, frontends should just avoid generating code that traps. + fn trap(&mut self, _offset: CodeOffset, _srcloc: ir::SourceLoc, _code: ir::TrapCode) {} +} diff --git a/lib/faerie/src/container.rs b/lib/faerie/src/container.rs new file mode 100644 index 0000000000..c78f05878c --- /dev/null +++ b/lib/faerie/src/container.rs @@ -0,0 +1,33 @@ +//! Utilities for working with Faerie container formats. + +use cretonne_codegen::binemit::Reloc; + +/// An object file format. +#[derive(Copy, Clone, Debug, Eq, PartialEq)] +pub enum Format { + /// The ELF object file format. + ELF, + /// The Mach-O object file format. + MachO, +} + +/// Translate from a Cretonne `Reloc` to a raw object-file-format-specific +/// relocation code. +pub fn raw_relocation(reloc: Reloc, format: Format) -> u32 { + match format { + Format::ELF => { + use goblin::elf; + match reloc { + Reloc::Abs4 => elf::reloc::R_X86_64_32, + Reloc::Abs8 => elf::reloc::R_X86_64_64, + Reloc::X86PCRel4 => elf::reloc::R_X86_64_PC32, + // TODO: Get Cretonne to tell us when we can use + // R_X86_64_GOTPCRELX/R_X86_64_REX_GOTPCRELX. + Reloc::X86GOTPCRel4 => elf::reloc::R_X86_64_GOTPCREL, + Reloc::X86PLTRel4 => elf::reloc::R_X86_64_PLT32, + _ => unimplemented!(), + } + } + Format::MachO => unimplemented!(), + } +} diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs new file mode 100644 index 0000000000..dc1dd5a0c3 --- /dev/null +++ b/lib/faerie/src/lib.rs @@ -0,0 +1,19 @@ +//! Top-level lib.rs for `cretonne_faerie`. +//! +//! Users of this module should not have to depend on faerie directly. + +#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] + +extern crate cretonne_codegen; +extern crate cretonne_module; +extern crate faerie; +#[macro_use] +extern crate failure; +extern crate goblin; + +mod backend; +mod container; +mod target; + +pub use backend::FaerieBackend; +pub use container::Format; diff --git a/lib/faerie/src/target.rs b/lib/faerie/src/target.rs new file mode 100644 index 0000000000..e1116dd08d --- /dev/null +++ b/lib/faerie/src/target.rs @@ -0,0 +1,18 @@ +use cretonne_codegen::isa; +use faerie::Target; +use failure::Error; + +/// Translate from a Cretonne `TargetIsa` to a Faerie `Target`. +pub fn translate(isa: &isa::TargetIsa) -> Result { + let name = isa.name(); + match name { + "x86" => Ok(if isa.flags().is_64bit() { + Target::X86_64 + } else { + Target::X86 + }), + "arm32" => Ok(Target::ARMv7), + "arm64" => Ok(Target::ARM64), + _ => Err(format_err!("unsupported isa: {}", name)), + } +} diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml new file mode 100644 index 0000000000..04e7b6e4b6 --- /dev/null +++ b/lib/module/Cargo.toml @@ -0,0 +1,17 @@ +[package] +name = "cretonne-module" +version = "0.4.4" +authors = ["The Cretonne Project Developers"] +description = "Support for linking functions and data with Cretonne" +repository = "https://github.com/Cretonne/cretonne" +documentation = "https://cretonne.readthedocs.io/" +license = "Apache-2.0" +readme = "README.md" + +[dependencies] +cretonne-codegen = { path = "../codegen", version = "0.4.4" } +cretonne-entity = { path = "../entity", version = "0.4.4" } + +[badges] +maintenance = { status = "experimental" } +travis-ci = { repository = "Cretonne/cretonne" } diff --git a/lib/module/README.md b/lib/module/README.md new file mode 100644 index 0000000000..370bde4333 --- /dev/null +++ b/lib/module/README.md @@ -0,0 +1,3 @@ +This crate provides the `Module` trait, which provides an interface for +multiple functions and data to be emitted with +[Cretonne](https://crates.io/crates/cretonne) and then linked together. diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs new file mode 100644 index 0000000000..0d6a6c6672 --- /dev/null +++ b/lib/module/src/backend.rs @@ -0,0 +1,97 @@ +//! Defines the `Backend` trait. + +use DataContext; +use Linkage; +use ModuleNamespace; +use cretonne_codegen::Context; +use cretonne_codegen::isa::TargetIsa; +use cretonne_codegen::result::CtonError; +use cretonne_codegen::{binemit, ir}; +use std::marker; + +/// A `Backend` implements the functionality needed to support a `Module`. +pub trait Backend +where + Self: marker::Sized, +{ + /// The results of compiling a function. + type CompiledFunction; + + /// The results of "compiling" a data object. + type CompiledData; + + /// The completed output artifact for a function, if this is meaningful for + /// the Backend. + type FinalizedFunction; + + /// The completed output artifact for a data object, if this is meaningful for + /// the Backend. + type FinalizedData; + + /// Return the `TargetIsa` to compile for. + fn isa(&self) -> &TargetIsa; + + /// Declare a function. + fn declare_function(&mut self, name: &str, linkage: Linkage); + + /// Declare a data object. + fn declare_data(&mut self, name: &str, linkage: Linkage, writable: bool); + + /// Define a function, producing the function body from the given `Context`. + /// + /// Functions must be declared before being defined. + fn define_function( + &mut self, + name: &str, + ctx: &Context, + namespace: &ModuleNamespace, + code_size: u32, + ) -> Result; + + /// Define a zero-initialized data object of the given size. + /// + /// Data objects must be declared before being defined. + /// + /// TODO: Is CtonError the right error code here? + fn define_data( + &mut self, + name: &str, + data_ctx: &DataContext, + namespace: &ModuleNamespace, + ) -> Result; + + /// Write the address of `what` into the data for `data` at `offset`. `data` must refer to a + /// defined data object. + fn write_data_funcaddr( + &mut self, + data: &mut Self::CompiledData, + offset: usize, + what: ir::FuncRef, + ); + + /// Write the address of `what` plus `addend` into the data for `data` at `offset`. `data` must + /// refer to a defined data object. + fn write_data_dataaddr( + &mut self, + data: &mut Self::CompiledData, + offset: usize, + what: ir::GlobalVar, + addend: binemit::Addend, + ); + + /// Perform all outstanding relocations on the given function. This requires all `Local` + /// and `Export` entities referenced to be defined. + fn finalize_function( + &mut self, + func: &Self::CompiledFunction, + namespace: &ModuleNamespace, + ) -> Self::FinalizedFunction; + + /// Perform all outstanding relocations on the given data object. This requires all + /// `Local` and `Export` entities referenced to be defined. + fn finalize_data( + &mut self, + data: &Self::CompiledData, + namespace: &ModuleNamespace, + ) -> Self::FinalizedData; +} diff --git a/lib/module/src/data_context.rs b/lib/module/src/data_context.rs new file mode 100644 index 0000000000..510b549e90 --- /dev/null +++ b/lib/module/src/data_context.rs @@ -0,0 +1,205 @@ +//! Defines `DataContext`. + +use cretonne_codegen::entity::PrimaryMap; +use cretonne_codegen::binemit::{CodeOffset, Addend}; +use cretonne_codegen::ir; + +/// This specifies how data is to be initialized. +#[derive(PartialEq, Eq, Debug)] +pub enum Init { + /// This indicates that no initialization has been specified yet. + Uninitialized, + /// Initialize the data with all zeros. + Zeros { + /// The size of the data. + size: usize, + }, + /// Initialize the data with the specified contents. + Bytes { + /// The contents, which also implies the size of the data. + contents: Box<[u8]>, + }, +} + +impl Init { + /// Return the size of the data to be initialized. + pub fn size(&self) -> usize { + match *self { + Init::Uninitialized => panic!("data size not initialized yet"), + Init::Zeros { size } => size, + Init::Bytes { ref contents } => contents.len(), + } + } +} + +/// A flag specifying whether data is readonly or may be written to. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub enum Writability { + /// Data is readonly, meaning writes to it will trap. + Readonly, + /// Data is writable. + Writable, +} + +/// A description of a data object. +pub struct DataDescription { + /// Whether the data readonly or writable. + pub writable: Writability, + /// How the data should be initialized. + pub init: Init, + /// External function declarations. + pub function_decls: PrimaryMap, + /// External data object declarations. + pub data_decls: PrimaryMap, + /// Function addresses to write at specified offsets. + pub function_relocs: Vec<(CodeOffset, ir::FuncRef)>, + /// Data addresses to write at specified offsets. + pub data_relocs: Vec<(CodeOffset, ir::GlobalVar, Addend)>, +} + +/// This is to data objects what cretonne_codegen::Context is to functions. +pub struct DataContext { + description: DataDescription, +} + +impl DataContext { + /// Allocate a new context. + pub fn new() -> Self { + Self { + description: DataDescription { + writable: Writability::Readonly, + init: Init::Uninitialized, + function_decls: PrimaryMap::new(), + data_decls: PrimaryMap::new(), + function_relocs: Vec::new(), + data_relocs: Vec::new(), + }, + } + } + + /// Clear all data structures in this context. + pub fn clear(&mut self) { + self.description.writable = Writability::Readonly; + self.description.init = Init::Uninitialized; + self.description.function_decls.clear(); + self.description.data_decls.clear(); + self.description.function_relocs.clear(); + self.description.data_relocs.clear(); + } + + /// Define a zero-initialized object with the given size. + pub fn define_zeroinit(&mut self, size: usize, writable: Writability) { + debug_assert_eq!(self.description.init, Init::Uninitialized); + self.description.writable = writable; + self.description.init = Init::Zeros { size }; + } + + /// Define a zero-initialized object with the given size. + /// + /// TODO: Can we avoid a Box here? + pub fn define(&mut self, contents: Box<[u8]>, writable: Writability) { + debug_assert_eq!(self.description.init, Init::Uninitialized); + self.description.writable = writable; + self.description.init = Init::Bytes { contents }; + } + + /// Declare an external function import. + pub fn import_function(&mut self, name: ir::ExternalName) -> ir::FuncRef { + self.description.function_decls.push(name) + } + + /// Declares a global variable import. + /// + /// TODO: Rename to import_data? + pub fn import_global_var(&mut self, name: ir::ExternalName) -> ir::GlobalVar { + self.description.data_decls.push(name) + } + + /// Write the address of `func` into the data at offset `offset`. + pub fn write_function_addr(&mut self, offset: CodeOffset, func: ir::FuncRef) { + self.description.function_relocs.push((offset, func)) + } + + /// Write the address of `data` into the data at offset `offset`. + pub fn write_data_addr(&mut self, offset: CodeOffset, data: ir::GlobalVar, addend: Addend) { + self.description.data_relocs.push((offset, data, addend)) + } + + /// Reference the initializer data. + pub fn description(&self) -> &DataDescription { + debug_assert!( + self.description.init != Init::Uninitialized, + "data must be initialized first" + ); + &self.description + } +} + +#[cfg(test)] +mod tests { + use {DataContext, Writability, Init}; + use cretonne_codegen::ir; + + #[test] + fn basic_data_context() { + let mut data_ctx = DataContext::new(); + { + let description = data_ctx.description(); + assert_eq!(description.writable, Writability::Readonly); + assert_eq!(description.init, Init::Uninitialized); + assert!(description.function_decls.is_empty()); + assert!(description.data_decls.is_empty()); + assert!(description.function_relocs.is_empty()); + assert!(description.data_relocs.is_empty()); + } + + data_ctx.define_zeroinit(256, Writability::Writable); + + let _func_a = data_ctx.import_function(ir::ExternalName::user(0, 0)); + let func_b = data_ctx.import_function(ir::ExternalName::user(0, 1)); + let func_c = data_ctx.import_function(ir::ExternalName::user(1, 0)); + let _data_a = data_ctx.import_global_var(ir::ExternalName::user(2, 2)); + let data_b = data_ctx.import_global_var(ir::ExternalName::user(2, 3)); + + data_ctx.write_function_addr(8, func_b); + data_ctx.write_function_addr(16, func_c); + data_ctx.write_data_addr(32, data_b, 27); + + { + let description = data_ctx.description(); + assert_eq!(description.writable, Writability::Writable); + assert_eq!(description.init, Init::Zeros { size: 256 }); + assert_eq!(description.function_decls.len(), 3); + assert_eq!(description.data_decls.len(), 2); + assert_eq!(description.function_relocs.len(), 2); + assert_eq!(description.data_relocs.len(), 1); + } + + data_ctx.clear(); + { + let description = data_ctx.description(); + assert_eq!(description.writable, Writability::Readonly); + assert_eq!(description.init, Init::Uninitialized); + assert!(description.function_decls.is_empty()); + assert!(description.data_decls.is_empty()); + assert!(description.function_relocs.is_empty()); + assert!(description.data_relocs.is_empty()); + } + + let contents = vec![33, 34, 35, 36]; + let contents_clone = contents.clone(); + data_ctx.define(contents.into_boxed_slice(), Writability::Readonly); + { + let description = data_ctx.description(); + assert_eq!(description.writable, Writability::Readonly); + assert_eq!( + description.init, + Init::Bytes { contents: contents_clone.into_boxed_slice() } + ); + assert_eq!(description.function_decls.len(), 0); + assert_eq!(description.data_decls.len(), 0); + assert_eq!(description.function_relocs.len(), 0); + assert_eq!(description.data_relocs.len(), 0); + } + } +} diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs new file mode 100644 index 0000000000..ebc44420cc --- /dev/null +++ b/lib/module/src/lib.rs @@ -0,0 +1,15 @@ +//! Top-level lib.rs for `cretonne_module`. + +#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] + +extern crate cretonne_codegen; +#[macro_use] +extern crate cretonne_entity; + +mod backend; +mod data_context; +mod module; + +pub use backend::Backend; +pub use data_context::{DataContext, Writability, DataDescription, Init}; +pub use module::{DataId, FuncId, Linkage, Module, ModuleNamespace}; diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs new file mode 100644 index 0000000000..69e9d7432f --- /dev/null +++ b/lib/module/src/module.rs @@ -0,0 +1,541 @@ +//! Defines `Module` and related types. + +// TODO: Should `ir::Function` really have a `name`? + +// TODO: Factor out `ir::Function`'s `ext_funcs` and `global_vars` into a struct +// shared with `DataContext`? + +use Backend; +use cretonne_codegen::entity::{EntityRef, PrimaryMap}; +use cretonne_codegen::result::{CtonError, CtonResult}; +use cretonne_codegen::{binemit, ir, Context}; +use data_context::DataContext; +use std::collections::HashMap; + +/// A function identifier for use in the `Module` interface. +#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub struct FuncId(u32); +entity_impl!(FuncId, "funcid"); + +/// A data object identifier for use in the `Module` interface. +#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub struct DataId(u32); +entity_impl!(DataId, "dataid"); + +/// Linkage refers to where an entity is defined and who can see it. +#[derive(Copy, Clone, PartialEq, Eq)] +pub enum Linkage { + /// Defined outside of a module. + Import, + /// Defined inside the module, but not visible outside it. + Local, + /// Defined inside the module, visible outside it, and may be preempted. + Preemptible, + /// Defined inside the module, and visible outside it. + Export, +} + +impl Linkage { + fn merge(a: Linkage, b: Linkage) -> Linkage { + match a { + Linkage::Export => Linkage::Export, + Linkage::Preemptible => { + match b { + Linkage::Export => Linkage::Export, + _ => Linkage::Preemptible, + } + } + Linkage::Local => { + match b { + Linkage::Export => Linkage::Export, + Linkage::Preemptible => Linkage::Preemptible, + _ => Linkage::Local, + } + } + Linkage::Import => b, + } + } + + /// Test whether this linkage can have a definition. + pub fn is_definable(&self) -> bool { + match *self { + Linkage::Import => false, + Linkage::Local | Linkage::Preemptible | Linkage::Export => true, + } + } + + /// Test whether this linkage will have a definition that cannot be preempted. + pub fn is_final(&self) -> bool { + match *self { + Linkage::Import | Linkage::Preemptible => false, + Linkage::Local | Linkage::Export => true, + } + } +} + +#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] +enum FuncOrDataId { + Func(FuncId), + Data(DataId), +} + +pub struct FunctionDeclaration { + pub name: String, + pub linkage: Linkage, + pub signature: ir::Signature, +} + +struct ModuleFunction +where + B: Backend, +{ + decl: FunctionDeclaration, + compiled: Option, + finalized: bool, +} + +impl ModuleFunction +where + B: Backend, +{ + fn merge(&mut self, linkage: Linkage) { + self.decl.linkage = Linkage::merge(self.decl.linkage, linkage); + } +} + +pub struct DataDeclaration { + pub name: String, + pub linkage: Linkage, + pub writable: bool, +} + +struct ModuleData +where + B: Backend, +{ + decl: DataDeclaration, + compiled: Option, + finalized: bool, +} + +impl ModuleData +where + B: Backend, +{ + fn merge(&mut self, linkage: Linkage, writable: bool) { + self.decl.linkage = Linkage::merge(self.decl.linkage, linkage); + self.decl.writable = self.decl.writable || writable; + } +} + +struct ModuleContents +where + B: Backend, +{ + functions: PrimaryMap>, + data_objects: PrimaryMap>, +} + +impl ModuleContents +where + B: Backend, +{ + fn get_function_info(&self, name: &ir::ExternalName) -> &ModuleFunction { + if let ir::ExternalName::User { namespace, index } = *name { + debug_assert_eq!(namespace, 0); + let func = FuncId::new(index as usize); + &self.functions[func] + } else { + panic!("unexpected ExternalName kind") + } + } + + /// Get the `DataDeclaration` for the function named by `name`. + fn get_data_info(&self, name: &ir::ExternalName) -> &ModuleData { + if let ir::ExternalName::User { namespace, index } = *name { + debug_assert_eq!(namespace, 1); + let data = DataId::new(index as usize); + &self.data_objects[data] + } else { + panic!("unexpected ExternalName kind") + } + } +} + +/// This provides a view to the state of a module which allows `ir::ExternalName`s to be translated +/// into `FunctionDeclaration`s and `DataDeclaration`s. +pub struct ModuleNamespace<'a, B: 'a> +where + B: Backend, +{ + contents: &'a ModuleContents, +} + +impl<'a, B> ModuleNamespace<'a, B> +where + B: Backend, +{ + /// Get the `FunctionDeclaration` for the function named by `name`. + pub fn get_function_decl(&self, name: &ir::ExternalName) -> &FunctionDeclaration { + &self.contents.get_function_info(name).decl + } + + /// Get the `DataDeclaration` for the function named by `name`. + pub fn get_data_decl(&self, name: &ir::ExternalName) -> &DataDeclaration { + &self.contents.get_data_info(name).decl + } + + /// Get the definition for the function named by `name`, along with its name + /// and signature. + pub fn get_function_definition( + &self, + name: &ir::ExternalName, + ) -> (Option<&B::CompiledFunction>, &str, &ir::Signature) { + let info = self.contents.get_function_info(name); + debug_assert_eq!(info.decl.linkage.is_definable(), info.compiled.is_some()); + ( + info.compiled.as_ref(), + &info.decl.name, + &info.decl.signature, + ) + } + + /// Get the definition for the data object named by `name`, along with its name + /// and writable flag + pub fn get_data_definition( + &self, + name: &ir::ExternalName, + ) -> (Option<&B::CompiledData>, &str, bool) { + let info = self.contents.get_data_info(name); + debug_assert_eq!(info.decl.linkage.is_definable(), info.compiled.is_some()); + (info.compiled.as_ref(), &info.decl.name, info.decl.writable) + } + + /// Return whether `name` names a function, rather than a data object. + pub fn is_function(&self, name: &ir::ExternalName) -> bool { + if let ir::ExternalName::User { namespace, .. } = *name { + namespace == 0 + } else { + panic!("unexpected ExternalName kind") + } + } +} + +/// A `Module` is a utility for collecting functions and data objects, and linking them together. +pub struct Module +where + B: Backend, +{ + names: HashMap, + contents: ModuleContents, + backend: B, +} + +impl Module +where + B: Backend, +{ + /// Create a new `Module`. + pub fn new(backend: B) -> Self { + Self { + names: HashMap::new(), + contents: ModuleContents { + functions: PrimaryMap::new(), + data_objects: PrimaryMap::new(), + }, + backend, + } + } + + /// Return then pointer type for the current target. + pub fn pointer_type(&self) -> ir::types::Type { + if self.backend.isa().flags().is_64bit() { + ir::types::I64 + } else { + ir::types::I32 + } + } + + /// Declare a function in this module. + pub fn declare_function( + &mut self, + name: &str, + linkage: Linkage, + signature: &ir::Signature, + ) -> Result { + // TODO: Can we avoid allocating names so often? + use std::collections::hash_map::Entry::*; + match self.names.entry(name.to_owned()) { + Occupied(entry) => { + match *entry.get() { + FuncOrDataId::Func(id) => { + let existing = &mut self.contents.functions[id]; + existing.merge(linkage); + self.backend.declare_function(name, existing.decl.linkage); + Ok(id) + } + FuncOrDataId::Data(..) => unimplemented!(), + } + } + Vacant(entry) => { + let id = self.contents.functions.push(ModuleFunction { + decl: FunctionDeclaration { + name: name.to_owned(), + linkage, + signature: signature.clone(), + }, + compiled: None, + finalized: false, + }); + entry.insert(FuncOrDataId::Func(id)); + self.backend.declare_function(name, linkage); + Ok(id) + } + } + } + + /// Declare a data object in this module. + pub fn declare_data( + &mut self, + name: &str, + linkage: Linkage, + writable: bool, + ) -> Result { + // TODO: Can we avoid allocating names so often? + use std::collections::hash_map::Entry::*; + match self.names.entry(name.to_owned()) { + Occupied(entry) => { + match *entry.get() { + FuncOrDataId::Data(id) => { + let existing = &mut self.contents.data_objects[id]; + existing.merge(linkage, writable); + self.backend.declare_data( + name, + existing.decl.linkage, + existing.decl.writable, + ); + Ok(id) + } + + FuncOrDataId::Func(..) => unimplemented!(), + } + } + Vacant(entry) => { + let id = self.contents.data_objects.push(ModuleData { + decl: DataDeclaration { + name: name.to_owned(), + linkage, + writable, + }, + compiled: None, + finalized: false, + }); + entry.insert(FuncOrDataId::Data(id)); + self.backend.declare_data(name, linkage, writable); + Ok(id) + } + } + } + + /// Use this when you're building the IR of a function to reference a function. + /// + /// TODO: Coalesce redundant decls and signatures. + /// TODO: Look into ways to reduce the risk of using a FuncRef in the wrong function. + pub fn declare_func_in_func(&self, func: FuncId, in_func: &mut ir::Function) -> ir::FuncRef { + let decl = &self.contents.functions[func].decl; + let signature = in_func.import_signature(decl.signature.clone()); + let colocated = decl.linkage.is_final(); + in_func.import_function(ir::ExtFuncData { + name: ir::ExternalName::user(0, func.index() as u32), + signature, + colocated, + }) + } + + /// Use this when you're building the IR of a function to reference a data object. + /// + /// TODO: Same as above. + pub fn declare_data_in_func(&self, data: DataId, func: &mut ir::Function) -> ir::GlobalVar { + let decl = &self.contents.data_objects[data].decl; + let colocated = decl.linkage.is_final(); + func.create_global_var(ir::GlobalVarData::Sym { + name: ir::ExternalName::user(1, data.index() as u32), + colocated, + }) + } + + /// TODO: Same as above. + pub fn declare_func_in_data(&self, func: FuncId, ctx: &mut DataContext) -> ir::FuncRef { + ctx.import_function(ir::ExternalName::user(0, func.index() as u32)) + } + + /// TODO: Same as above. + pub fn declare_data_in_data(&self, data: DataId, ctx: &mut DataContext) -> ir::GlobalVar { + ctx.import_global_var(ir::ExternalName::user(1, data.index() as u32)) + } + + /// Define a function, producing the function body from the given `Context`. + pub fn define_function(&mut self, func: FuncId, ctx: &mut Context) -> CtonResult { + let compiled = { + let code_size = ctx.compile(self.backend.isa())?; + + let info = &self.contents.functions[func]; + debug_assert!( + info.compiled.is_none(), + "functions can be defined only once" + ); + debug_assert!( + info.decl.linkage.is_definable(), + "imported functions cannot be defined" + ); + Some(self.backend.define_function( + &info.decl.name, + ctx, + &ModuleNamespace:: { + contents: &self.contents, + }, + code_size, + )?) + }; + self.contents.functions[func].compiled = compiled; + Ok(()) + } + + /// Define a function, producing the data contents from the given `DataContext`. + pub fn define_data(&mut self, data: DataId, data_ctx: &DataContext) -> CtonResult { + let compiled = { + let info = &self.contents.data_objects[data]; + debug_assert!( + info.compiled.is_none(), + "functions can be defined only once" + ); + debug_assert!( + info.decl.linkage.is_definable(), + "imported functions cannot be defined" + ); + Some(self.backend.define_data( + &info.decl.name, + data_ctx, + &ModuleNamespace:: { + contents: &self.contents, + }, + )?) + }; + self.contents.data_objects[data].compiled = compiled; + Ok(()) + } + + /// Write the address of `what` into the data for `data` at `offset`. `data` must refer to a + /// defined data object. + pub fn write_data_funcaddr(&mut self, data: DataId, offset: usize, what: ir::FuncRef) { + let info = &mut self.contents.data_objects[data]; + debug_assert!( + info.decl.linkage.is_definable(), + "imported data cannot contain references" + ); + self.backend.write_data_funcaddr( + &mut info.compiled.as_mut().expect( + "`data` must refer to a defined data object", + ), + offset, + what, + ); + } + + /// Write the address of `what` plus `addend` into the data for `data` at `offset`. `data` must + /// refer to a defined data object. + pub fn write_data_dataaddr( + &mut self, + data: DataId, + offset: usize, + what: ir::GlobalVar, + addend: binemit::Addend, + ) { + let info = &mut self.contents.data_objects[data]; + debug_assert!( + info.decl.linkage.is_definable(), + "imported data cannot contain references" + ); + self.backend.write_data_dataaddr( + &mut info.compiled.as_mut().expect( + "`data` must refer to a defined data object", + ), + offset, + what, + addend, + ); + } + + /// Perform all outstanding relocations on the given function. This requires all `Local` + /// and `Export` entities referenced to be defined. + pub fn finalize_function(&mut self, func: FuncId) -> B::FinalizedFunction { + let output = { + let info = &self.contents.functions[func]; + debug_assert!( + info.decl.linkage.is_definable(), + "imported data cannot be finalized" + ); + self.backend.finalize_function( + info.compiled.as_ref().expect( + "function must be compiled before it can be finalized", + ), + &ModuleNamespace:: { contents: &self.contents }, + ) + }; + self.contents.functions[func].finalized = true; + output + } + + /// Perform all outstanding relocations on the given data object. This requires all + /// `Local` and `Export` entities referenced to be defined. + pub fn finalize_data(&mut self, data: DataId) -> B::FinalizedData { + let output = { + let info = &self.contents.data_objects[data]; + debug_assert!( + info.decl.linkage.is_definable(), + "imported data cannot be finalized" + ); + self.backend.finalize_data( + info.compiled.as_ref().expect( + "data object must be compiled before it can be finalized", + ), + &ModuleNamespace:: { contents: &self.contents }, + ) + }; + self.contents.data_objects[data].finalized = true; + output + } + + /// Finalize all functions and data objects. Note that this doesn't return the + /// final artifacts returned from `finalize_function` or `finalize_data`. + pub fn finalize_all(&mut self) { + // TODO: Could we use something like `into_iter()` here? + for info in self.contents.functions.values() { + if info.decl.linkage.is_definable() && !info.finalized { + self.backend.finalize_function( + info.compiled.as_ref().expect( + "function must be compiled before it can be finalized", + ), + &ModuleNamespace:: { contents: &self.contents }, + ); + } + } + for info in self.contents.data_objects.values() { + if info.decl.linkage.is_definable() && !info.finalized { + self.backend.finalize_data( + info.compiled.as_ref().expect( + "data object must be compiled before it can be finalized", + ), + &ModuleNamespace:: { contents: &self.contents }, + ); + } + } + } + + /// Consume the module and return its contained `Backend`. Some `Backend` + /// implementations have additional features not available through the + /// `Module` interface. + pub fn consume(self) -> B { + self.backend + } +} diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml new file mode 100644 index 0000000000..edc87d8b64 --- /dev/null +++ b/lib/simplejit/Cargo.toml @@ -0,0 +1,21 @@ +[package] +name = "cretonne-simplejit" +version = "0.4.4" +authors = ["The Cretonne Project Developers"] +description = "A simple JIT library backed by Cretonne" +repository = "https://github.com/Cretonne/cretonne" +documentation = "https://cretonne.readthedocs.io/" +license = "Apache-2.0" +readme = "README.md" + +[dependencies] +cretonne-codegen = { path = "../codegen", version = "0.4.4" } +cretonne-module = { path = "../module", version = "0.4.4" } +cretonne-native = { path = "../native", version = "0.4.4" } +region = "0.2.0" +libc = "0.2.40" +errno = "0.2.3" + +[badges] +maintenance = { status = "experimental" } +travis-ci = { repository = "Cretonne/cretonne" } diff --git a/lib/simplejit/README.md b/lib/simplejit/README.md new file mode 100644 index 0000000000..8b0aea6d28 --- /dev/null +++ b/lib/simplejit/README.md @@ -0,0 +1,4 @@ +This crate provides a simple JIT library that uses +[Cretonne](https://crates.io/crates/cretonne). + +This crate is extremely experimental. diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs new file mode 100644 index 0000000000..7bc5708556 --- /dev/null +++ b/lib/simplejit/src/backend.rs @@ -0,0 +1,365 @@ +//! Defines `SimpleJITBackend`. + +use cretonne_codegen::binemit::{Addend, CodeOffset, Reloc, RelocSink, TrapSink}; +use cretonne_codegen::isa::TargetIsa; +use cretonne_codegen::result::CtonError; +use cretonne_codegen::{self, ir, settings}; +use cretonne_module::{Backend, DataContext, Linkage, ModuleNamespace, Writability, + DataDescription, Init}; +use cretonne_native; +use std::ffi::CString; +use std::ptr; +use libc; +use memory::Memory; + +/// A record of a relocation to perform. +struct RelocRecord { + offset: CodeOffset, + reloc: Reloc, + name: ir::ExternalName, + addend: Addend, +} + +pub struct SimpleJITCompiledFunction { + code: *mut u8, + size: usize, + relocs: Vec, +} + +pub struct SimpleJITCompiledData { + storage: *mut u8, + size: usize, + relocs: Vec, +} + +/// A `SimpleJITBackend` implements `Backend` and emits code and data into memory where it can be +/// directly called and accessed. +pub struct SimpleJITBackend { + isa: Box, + code_memory: Memory, + readonly_memory: Memory, + writable_memory: Memory, +} + +impl SimpleJITBackend { + /// Create a new `SimpleJITBackend`. + pub fn new() -> Self { + let (flag_builder, isa_builder) = cretonne_native::builders().unwrap_or_else(|_| { + panic!("host machine is not a supported target"); + }); + let isa = isa_builder.finish(settings::Flags::new(&flag_builder)); + Self::with_isa(isa) + } + + /// Create a new `SimpleJITBackend` with an arbitrary target. This is mainly + /// useful for testing. + /// + /// SimpleJIT requires a `TargetIsa` configured for non-PIC. + /// + /// To create a `SimpleJITBackend` for native use, use the `new` constructor + /// instead. + pub fn with_isa(isa: Box) -> Self { + debug_assert!(!isa.flags().is_pic(), "SimpleJIT requires non-PIC code"); + Self { + isa, + code_memory: Memory::new(), + readonly_memory: Memory::new(), + writable_memory: Memory::new(), + } + } +} + +impl<'simple_jit_backend> Backend for SimpleJITBackend { + type CompiledFunction = SimpleJITCompiledFunction; + type CompiledData = SimpleJITCompiledData; + + type FinalizedFunction = *const u8; + type FinalizedData = (*mut u8, usize); + + fn isa(&self) -> &TargetIsa { + &*self.isa + } + + fn declare_function(&mut self, _name: &str, _linkage: Linkage) { + // Nothing to do. + } + + fn declare_data(&mut self, _name: &str, _linkage: Linkage, _writable: bool) { + // Nothing to do. + } + + fn define_function( + &mut self, + _name: &str, + ctx: &cretonne_codegen::Context, + _namespace: &ModuleNamespace, + code_size: u32, + ) -> Result { + let size = code_size as usize; + let ptr = self.code_memory.allocate(size).expect( + "TODO: handle OOM etc.", + ); + let mut reloc_sink = SimpleJITRelocSink::new(); + let mut trap_sink = SimpleJITTrapSink {}; + ctx.emit_to_memory(ptr, &mut reloc_sink, &mut trap_sink, &*self.isa); + + Ok(Self::CompiledFunction { + code: ptr, + size, + relocs: reloc_sink.relocs, + }) + } + + fn define_data( + &mut self, + _name: &str, + data: &DataContext, + _namespace: &ModuleNamespace, + ) -> Result { + let &DataDescription { + writable, + ref init, + ref function_decls, + ref data_decls, + ref function_relocs, + ref data_relocs, + } = data.description(); + + let size = init.size(); + let storage = match writable { + Writability::Readonly => { + self.writable_memory.allocate(size).expect( + "TODO: handle OOM etc.", + ) + } + Writability::Writable => { + self.writable_memory.allocate(size).expect( + "TODO: handle OOM etc.", + ) + } + }; + + match *init { + Init::Uninitialized => { + panic!("data is not initialized yet"); + } + Init::Zeros { .. } => { + unsafe { ptr::write_bytes(storage, 0, size) }; + } + Init::Bytes { ref contents } => { + let src = contents.as_ptr(); + unsafe { ptr::copy_nonoverlapping(src, storage, size) }; + } + } + + let reloc = if self.isa.flags().is_64bit() { + Reloc::Abs8 + } else { + Reloc::Abs4 + }; + let mut relocs = Vec::new(); + for &(offset, id) in function_relocs { + relocs.push(RelocRecord { + reloc, + offset, + name: function_decls[id].clone(), + addend: 0, + }); + } + for &(offset, id, addend) in data_relocs { + relocs.push(RelocRecord { + reloc, + offset, + name: data_decls[id].clone(), + addend, + }); + } + + Ok(Self::CompiledData { + storage, + size, + relocs, + }) + } + + fn write_data_funcaddr( + &mut self, + _data: &mut Self::CompiledData, + _offset: usize, + _what: ir::FuncRef, + ) { + unimplemented!(); + } + + fn write_data_dataaddr( + &mut self, + _data: &mut Self::CompiledData, + _offset: usize, + _what: ir::GlobalVar, + _usize: Addend, + ) { + unimplemented!(); + } + + fn finalize_function( + &mut self, + func: &Self::CompiledFunction, + namespace: &ModuleNamespace, + ) -> Self::FinalizedFunction { + use std::ptr::write_unaligned; + + for &RelocRecord { + reloc, + offset, + ref name, + addend, + } in &func.relocs + { + let ptr = func.code; + debug_assert!((offset as usize) < func.size); + let at = unsafe { ptr.offset(offset as isize) }; + let base = if namespace.is_function(name) { + let (def, name_str, _signature) = namespace.get_function_definition(&name); + match def { + Some(compiled) => compiled.code, + None => lookup_with_dlsym(name_str), + } + } else { + let (def, name_str, _writable) = namespace.get_data_definition(&name); + match def { + Some(compiled) => compiled.storage, + None => lookup_with_dlsym(name_str), + } + }; + // TODO: Handle overflow. + let what = unsafe { base.offset(addend as isize) }; + match reloc { + Reloc::Abs4 => { + // TODO: Handle overflow. + unsafe { write_unaligned(at as *mut u32, what as u32) }; + } + Reloc::Abs8 => { + unsafe { write_unaligned(at as *mut u64, what as u64) }; + } + Reloc::X86PCRel4 => { + // TODO: Handle overflow. + let pcrel = ((what as isize) - (at as isize)) as i32; + unsafe { write_unaligned(at as *mut i32, pcrel) }; + } + Reloc::X86GOTPCRel4 | + Reloc::X86PLTRel4 => panic!("unexpected PIC relocation"), + _ => unimplemented!(), + } + } + + // Now that we're done patching, make the memory executable. + self.code_memory.set_executable(); + func.code + } + + fn finalize_data( + &mut self, + data: &Self::CompiledData, + namespace: &ModuleNamespace, + ) -> Self::FinalizedData { + use std::ptr::write_unaligned; + + for record in &data.relocs { + match *record { + RelocRecord { + reloc, + offset, + ref name, + addend, + } => { + let ptr = data.storage; + debug_assert!((offset as usize) < data.size); + let at = unsafe { ptr.offset(offset as isize) }; + let base = if namespace.is_function(name) { + let (def, name_str, _signature) = namespace.get_function_definition(&name); + match def { + Some(compiled) => compiled.code, + None => lookup_with_dlsym(name_str), + } + } else { + let (def, name_str, _writable) = namespace.get_data_definition(&name); + match def { + Some(compiled) => compiled.storage, + None => lookup_with_dlsym(name_str), + } + }; + // TODO: Handle overflow. + let what = unsafe { base.offset(addend as isize) }; + match reloc { + Reloc::Abs4 => { + // TODO: Handle overflow. + unsafe { write_unaligned(at as *mut u32, what as u32) }; + } + Reloc::Abs8 => { + unsafe { write_unaligned(at as *mut u64, what as u64) }; + } + Reloc::X86PCRel4 | + Reloc::X86GOTPCRel4 | + Reloc::X86PLTRel4 => panic!("unexpected text relocation in data"), + _ => unimplemented!(), + } + } + } + } + + self.readonly_memory.set_readonly(); + (data.storage, data.size) + } +} + +fn lookup_with_dlsym(name: &str) -> *const u8 { + let c_str = CString::new(name).unwrap(); + let c_str_ptr = c_str.as_ptr(); + let sym = unsafe { libc::dlsym(libc::RTLD_DEFAULT, c_str_ptr) }; + if sym.is_null() { + panic!("can't resolve symbol {}", name); + } + sym as *const u8 +} + +struct SimpleJITRelocSink { + pub relocs: Vec, +} + +impl SimpleJITRelocSink { + pub fn new() -> Self { + Self { relocs: Vec::new() } + } +} + +impl RelocSink for SimpleJITRelocSink { + fn reloc_ebb(&mut self, _offset: CodeOffset, _reloc: Reloc, _ebb_offset: CodeOffset) { + unimplemented!(); + } + + fn reloc_external( + &mut self, + offset: CodeOffset, + reloc: Reloc, + name: &ir::ExternalName, + addend: Addend, + ) { + self.relocs.push(RelocRecord { + offset, + reloc, + name: name.clone(), + addend, + }); + } + + fn reloc_jt(&mut self, _offset: CodeOffset, _reloc: Reloc, _jt: ir::JumpTable) { + unimplemented!(); + } +} + +struct SimpleJITTrapSink {} + +impl TrapSink for SimpleJITTrapSink { + // Ignore traps for now. For now, frontends should just avoid generating code that traps. + fn trap(&mut self, _offset: CodeOffset, _srcloc: ir::SourceLoc, _code: ir::TrapCode) {} +} diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs new file mode 100644 index 0000000000..0157466cac --- /dev/null +++ b/lib/simplejit/src/lib.rs @@ -0,0 +1,15 @@ +//! Top-level lib.rs for `cretonne_simplejit`. + +#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] + +extern crate cretonne_codegen; +extern crate cretonne_module; +extern crate cretonne_native; +extern crate errno; +extern crate region; +extern crate libc; + +mod backend; +mod memory; + +pub use backend::SimpleJITBackend; diff --git a/lib/simplejit/src/memory.rs b/lib/simplejit/src/memory.rs new file mode 100644 index 0000000000..235a3976d7 --- /dev/null +++ b/lib/simplejit/src/memory.rs @@ -0,0 +1,108 @@ +use std::mem; +use std::ptr; +use errno; +use libc; +use region; + +struct PtrLen { + ptr: *mut u8, + len: usize, +} + +impl PtrLen { + fn new() -> Self { + Self { + ptr: ptr::null_mut(), + len: 0, + } + } + + fn with_size(size: usize) -> Result { + let page_size = region::page::size(); + let alloc_size = (size + (page_size - 1)) & (page_size - 1); + unsafe { + let mut ptr: *mut libc::c_void = mem::uninitialized(); + let err = libc::posix_memalign(&mut ptr, page_size, alloc_size); + if err == 0 { + Ok(Self { + ptr: mem::transmute(ptr), + len: alloc_size, + }) + } else { + Err(errno::Errno(err).to_string()) + } + } + } +} + +pub struct Memory { + allocations: Vec, + executable: usize, + current: PtrLen, + position: usize, +} + +impl Memory { + pub fn new() -> Self { + Self { + allocations: Vec::new(), + executable: 0, + current: PtrLen::new(), + position: 0, + } + } + + fn finish_current(&mut self) { + self.allocations.push(mem::replace( + &mut self.current, + PtrLen::new(), + )); + self.position = 0; + } + + /// TODO: Use a proper error type. + pub fn allocate(&mut self, size: usize) -> Result<*mut u8, String> { + if size <= self.current.len - self.position { + // TODO: Ensure overflow is not possible. + let ptr = unsafe { self.current.ptr.offset(self.position as isize) }; + self.position += size; + return Ok(ptr); + } + + self.finish_current(); + + // TODO: Allocate more at a time. + self.current = PtrLen::with_size(size)?; + self.position = size; + Ok(self.current.ptr) + } + + pub fn set_executable(&mut self) { + self.finish_current(); + + for &PtrLen { ptr, len } in &self.allocations[self.executable..] { + if len != 0 { + unsafe { + region::protect(ptr, len, region::Protection::Execute) + .expect("unable to make memory executable"); + } + } + } + } + + pub fn set_readonly(&mut self) { + self.finish_current(); + + for &PtrLen { ptr, len } in &self.allocations[self.executable..] { + if len != 0 { + unsafe { + region::protect(ptr, len, region::Protection::Read).expect( + "unable to make memory readonly", + ); + } + } + } + } +} + +// TODO: Implement Drop to unprotect and deallocate the memory? From 1f43ec09f3cea30f3eb4931ec8fdb43064d98fed Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 10:58:33 -0700 Subject: [PATCH 1706/3084] Bump version to 0.5.0 --- cranelift/Cargo.toml | 22 +++++++++++----------- cranelift/publish-all.sh | 2 +- lib/codegen/Cargo.toml | 4 ++-- lib/entity/Cargo.toml | 2 +- lib/faerie/Cargo.toml | 6 +++--- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/module/Cargo.toml | 6 +++--- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/simplejit/Cargo.toml | 8 ++++---- lib/umbrella/Cargo.toml | 6 +++--- lib/wasm/Cargo.toml | 6 +++--- 13 files changed, 40 insertions(+), 40 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 4ca0fa7760..9c4c1608ff 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.4.4" +version = "0.5.0" description = "Binaries for testing the Cretonne libraries" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,16 +13,16 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne-codegen = { path = "lib/codegen", version = "0.4.4" } -cretonne-reader = { path = "lib/reader", version = "0.4.4" } -cretonne-frontend = { path = "lib/frontend", version = "0.4.4" } -cretonne-wasm = { path = "lib/wasm", version = "0.4.4" } -cretonne-native = { path = "lib/native", version = "0.4.4" } -cretonne-filetests = { path = "lib/filetests", version = "0.4.4" } -cretonne-module = { path = "lib/module", version = "0.4.4" } -cretonne-faerie = { path = "lib/faerie", version = "0.4.4" } -cretonne-simplejit = { path = "lib/simplejit", version = "0.4.4" } -cretonne = { path = "lib/umbrella", version = "0.4.4" } +cretonne-codegen = { path = "lib/codegen", version = "0.5.0" } +cretonne-reader = { path = "lib/reader", version = "0.5.0" } +cretonne-frontend = { path = "lib/frontend", version = "0.5.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.5.0" } +cretonne-native = { path = "lib/native", version = "0.5.0" } +cretonne-filetests = { path = "lib/filetests", version = "0.5.0" } +cretonne-module = { path = "lib/module", version = "0.5.0" } +cretonne-faerie = { path = "lib/faerie", version = "0.5.0" } +cretonne-simplejit = { path = "lib/simplejit", version = "0.5.0" } +cretonne = { path = "lib/umbrella", version = "0.5.0" } filecheck = "0.2.1" docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 108d8055c2..a67d27f873 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ cd $(dirname "$0") topdir="$(pwd)" # All the cretonne-* crates have the same version number -version="0.4.4" +version="0.5.0" # Update all of the Cargo.toml files. # diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 87dd016a21..4aaeccee7b 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-codegen" -version = "0.4.4" +version = "0.5.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"] build = "build.rs" [dependencies] -cretonne-entity = { path = "../entity", version = "0.4.4" } +cretonne-entity = { path = "../entity", version = "0.5.0" } # It is a goal of the cretonne-codegen crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index d93220fbe8..c7c6b0dde0 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-entity" -version = "0.4.4" +version = "0.5.0" description = "Data structures using entity references as mapping keys" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index 32434077f7..a0e1effeea 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-faerie" -version = "0.4.4" +version = "0.5.0" authors = ["The Cretonne Project Developers"] description = "Emit Cretonne output to native object files with Faerie" repository = "https://github.com/Cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.4.4" } -cretonne-module = { path = "../module", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.5.0" } +cretonne-module = { path = "../module", version = "0.5.0" } faerie = "0.1.0" goblin = "0.0.14" failure = "0.1.1" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 1059a7a12a..2ab50d0ebd 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.4.4" +version = "0.5.0" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "http://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" publish = false [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.4.4" } -cretonne-reader = { path = "../reader", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.5.0" } +cretonne-reader = { path = "../reader", version = "0.5.0" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 1c2b86d266..03752d5689 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.4.4" +version = "0.5.0" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.5.0" } [badges] maintenance = { status = "experimental" } diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 04e7b6e4b6..f43fde30a7 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-module" -version = "0.4.4" +version = "0.5.0" authors = ["The Cretonne Project Developers"] description = "Support for linking functions and data with Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.4.4" } -cretonne-entity = { path = "../entity", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.5.0" } +cretonne-entity = { path = "../entity", version = "0.5.0" } [badges] maintenance = { status = "experimental" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 191e9325b5..265629044e 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.4.4" +version = "0.5.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -8,7 +8,7 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.5.0" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index b321da3e43..0b984515d9 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.4.4" +version = "0.5.0" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.5.0" } [badges] maintenance = { status = "experimental" } diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index edc87d8b64..699212afc0 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-simplejit" -version = "0.4.4" +version = "0.5.0" authors = ["The Cretonne Project Developers"] description = "A simple JIT library backed by Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -9,9 +9,9 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.4.4" } -cretonne-module = { path = "../module", version = "0.4.4" } -cretonne-native = { path = "../native", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.5.0" } +cretonne-module = { path = "../module", version = "0.5.0" } +cretonne-native = { path = "../native", version = "0.5.0" } region = "0.2.0" libc = "0.2.40" errno = "0.2.3" diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml index d1f7f95786..fc89536394 100644 --- a/lib/umbrella/Cargo.toml +++ b/lib/umbrella/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.4.4" +version = "0.5.0" description = "Umbrella for commonly-used cretonne crates" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -10,8 +10,8 @@ readme = "README.md" keywords = ["compile", "compiler", "jit"] [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.4.4" } -cretonne-frontend = { path = "../frontend", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.5.0" } +cretonne-frontend = { path = "../frontend", version = "0.5.0" } [badges] maintenance = { status = "experimental" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index fbe7de7126..49644f7746 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.4.4" +version = "0.5.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/cretonne/cretonne" @@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"] [dependencies] wasmparser = "0.15.1" -cretonne-codegen = { path = "../codegen", version = "0.4.4" } -cretonne-frontend = { path = "../frontend", version = "0.4.4" } +cretonne-codegen = { path = "../codegen", version = "0.5.0" } +cretonne-frontend = { path = "../frontend", version = "0.5.0" } [dev-dependencies] tempdir = "0.3.5" From 58380f38e8d4953a313a6ddd1c05c95da75a6f62 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 11:22:11 -0700 Subject: [PATCH 1707/3084] Refactor SimpleJITTrapSink/FaerieTrapSink into NullTrapSink. This publishes it for use outside of simpljie/faerie as well. --- cranelift/publish-all.sh | 2 +- lib/codegen/src/binemit/memorysink.rs | 8 ++++++++ lib/codegen/src/binemit/mod.rs | 2 +- lib/faerie/src/backend.rs | 13 ++++--------- lib/simplejit/src/backend.rs | 13 ++++--------- 5 files changed, 18 insertions(+), 20 deletions(-) diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index a67d27f873..ba539f2f26 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -27,7 +27,7 @@ cargo update echo git commit -a -m "\"Bump version to $version"\" echo git push -for crate in entity codegen frontend native reader wasm umbrella ; do +for crate in entity codegen frontend native reader wasm module simplejit faerie umbrella ; do if [ "$crate" == "umbrella" ]; then dir="cretonne" else diff --git a/lib/codegen/src/binemit/memorysink.rs b/lib/codegen/src/binemit/memorysink.rs index bed34cd1ac..34380d2464 100644 --- a/lib/codegen/src/binemit/memorysink.rs +++ b/lib/codegen/src/binemit/memorysink.rs @@ -123,3 +123,11 @@ impl<'a> CodeSink for MemoryCodeSink<'a> { self.traps.trap(ofs, srcloc, code); } } + +/// A `TrapSink` implementation that does nothing, which is convenient when +/// compiling code that does not rely on trapping semantics. +pub struct NullTrapSink {} + +impl TrapSink for NullTrapSink { + fn trap(&mut self, _offset: CodeOffset, _srcloc: SourceLoc, _code: TrapCode) {} +} diff --git a/lib/codegen/src/binemit/mod.rs b/lib/codegen/src/binemit/mod.rs index 8549b77aad..fbe88cc61f 100644 --- a/lib/codegen/src/binemit/mod.rs +++ b/lib/codegen/src/binemit/mod.rs @@ -6,7 +6,7 @@ mod memorysink; mod relaxation; -pub use self::memorysink::{MemoryCodeSink, RelocSink, TrapSink}; +pub use self::memorysink::{MemoryCodeSink, RelocSink, TrapSink, NullTrapSink}; pub use self::relaxation::relax_branches; pub use regalloc::RegDiversions; diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index a284883927..e33fe5e420 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -1,7 +1,7 @@ //! Defines `FaerieBackend`. use container; -use cretonne_codegen::binemit::{Addend, CodeOffset, Reloc, RelocSink, TrapSink}; +use cretonne_codegen::binemit::{Addend, CodeOffset, Reloc, RelocSink, NullTrapSink}; use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::result::CtonError; use cretonne_codegen::{self, binemit, ir}; @@ -103,7 +103,9 @@ impl Backend for FaerieBackend { name, namespace, }; - let mut trap_sink = FaerieTrapSink {}; + // Ignore traps for now. For now, frontends should just avoid generating code + // that traps. + let mut trap_sink = NullTrapSink {}; ctx.emit_to_memory( code.as_mut_ptr(), @@ -289,10 +291,3 @@ impl<'a> RelocSink for FaerieRelocSink<'a> { unimplemented!(); } } - -struct FaerieTrapSink {} - -impl TrapSink for FaerieTrapSink { - // Ignore traps for now. For now, frontends should just avoid generating code that traps. - fn trap(&mut self, _offset: CodeOffset, _srcloc: ir::SourceLoc, _code: ir::TrapCode) {} -} diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs index 7bc5708556..497ac1bb15 100644 --- a/lib/simplejit/src/backend.rs +++ b/lib/simplejit/src/backend.rs @@ -1,6 +1,6 @@ //! Defines `SimpleJITBackend`. -use cretonne_codegen::binemit::{Addend, CodeOffset, Reloc, RelocSink, TrapSink}; +use cretonne_codegen::binemit::{Addend, CodeOffset, Reloc, RelocSink, NullTrapSink}; use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::result::CtonError; use cretonne_codegen::{self, ir, settings}; @@ -100,7 +100,9 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { "TODO: handle OOM etc.", ); let mut reloc_sink = SimpleJITRelocSink::new(); - let mut trap_sink = SimpleJITTrapSink {}; + // Ignore traps for now. For now, frontends should just avoid generating code + // that traps. + let mut trap_sink = NullTrapSink {}; ctx.emit_to_memory(ptr, &mut reloc_sink, &mut trap_sink, &*self.isa); Ok(Self::CompiledFunction { @@ -356,10 +358,3 @@ impl RelocSink for SimpleJITRelocSink { unimplemented!(); } } - -struct SimpleJITTrapSink {} - -impl TrapSink for SimpleJITTrapSink { - // Ignore traps for now. For now, frontends should just avoid generating code that traps. - fn trap(&mut self, _offset: CodeOffset, _srcloc: ir::SourceLoc, _code: ir::TrapCode) {} -} From 5c6cb202d8dbdfede47e3d1464ccbf0c09b38954 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 16:02:10 -0700 Subject: [PATCH 1708/3084] Suppress cast_ptr_alignment clippy errors. These are only used in places that use `write_unaligned`, so it's ok that the pointer might be misaligned. --- lib/codegen/src/binemit/memorysink.rs | 3 +++ lib/simplejit/src/backend.rs | 5 +++++ 2 files changed, 8 insertions(+) diff --git a/lib/codegen/src/binemit/memorysink.rs b/lib/codegen/src/binemit/memorysink.rs index 34380d2464..d25513d27a 100644 --- a/lib/codegen/src/binemit/memorysink.rs +++ b/lib/codegen/src/binemit/memorysink.rs @@ -84,6 +84,7 @@ impl<'a> CodeSink for MemoryCodeSink<'a> { fn put2(&mut self, x: u16) { unsafe { + #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] write_unaligned(self.data.offset(self.offset) as *mut u16, x); } self.offset += 2; @@ -91,6 +92,7 @@ impl<'a> CodeSink for MemoryCodeSink<'a> { fn put4(&mut self, x: u32) { unsafe { + #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] write_unaligned(self.data.offset(self.offset) as *mut u32, x); } self.offset += 4; @@ -98,6 +100,7 @@ impl<'a> CodeSink for MemoryCodeSink<'a> { fn put8(&mut self, x: u64) { unsafe { + #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] write_unaligned(self.data.offset(self.offset) as *mut u64, x); } self.offset += 8; diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs index 497ac1bb15..5576c02e32 100644 --- a/lib/simplejit/src/backend.rs +++ b/lib/simplejit/src/backend.rs @@ -238,14 +238,17 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { match reloc { Reloc::Abs4 => { // TODO: Handle overflow. + #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] unsafe { write_unaligned(at as *mut u32, what as u32) }; } Reloc::Abs8 => { + #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] unsafe { write_unaligned(at as *mut u64, what as u64) }; } Reloc::X86PCRel4 => { // TODO: Handle overflow. let pcrel = ((what as isize) - (at as isize)) as i32; + #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] unsafe { write_unaligned(at as *mut i32, pcrel) }; } Reloc::X86GOTPCRel4 | @@ -295,9 +298,11 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { match reloc { Reloc::Abs4 => { // TODO: Handle overflow. + #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] unsafe { write_unaligned(at as *mut u32, what as u32) }; } Reloc::Abs8 => { + #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] unsafe { write_unaligned(at as *mut u64, what as u64) }; } Reloc::X86PCRel4 | From bf597b7abf0e0f7c6549cfb49dbc20769e71db09 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 2 Apr 2018 08:48:06 -0700 Subject: [PATCH 1709/3084] Enable and fix several more clippy lints. --- cranelift/src/cton-util.rs | 12 +++++++ lib/codegen/meta/gen_instr.py | 2 +- lib/codegen/meta/gen_settings.py | 10 +++--- lib/codegen/meta/isa/x86/recipes.py | 2 +- lib/codegen/src/abi.rs | 4 +-- lib/codegen/src/bforest/map.rs | 8 ++--- lib/codegen/src/bforest/node.rs | 8 ++--- lib/codegen/src/bforest/path.rs | 4 +-- lib/codegen/src/bforest/pool.rs | 4 +-- lib/codegen/src/bforest/set.rs | 8 ++--- lib/codegen/src/binemit/mod.rs | 12 +++---- lib/codegen/src/context.rs | 4 +-- lib/codegen/src/dominator_tree.rs | 4 +-- lib/codegen/src/ir/entities.rs | 34 ++++++++++---------- lib/codegen/src/ir/extfunc.rs | 2 +- lib/codegen/src/ir/extname.rs | 13 +++----- lib/codegen/src/ir/immediates.rs | 40 ++++++++++++------------ lib/codegen/src/ir/instructions.rs | 4 +-- lib/codegen/src/ir/libcall.rs | 2 +- lib/codegen/src/ir/progpoint.rs | 14 ++++----- lib/codegen/src/ir/sourceloc.rs | 2 +- lib/codegen/src/ir/stackslot.rs | 6 ++-- lib/codegen/src/ir/types.rs | 20 ++++++------ lib/codegen/src/isa/encoding.rs | 12 +++---- lib/codegen/src/isa/riscv/abi.rs | 4 +-- lib/codegen/src/isa/stack.rs | 8 ++--- lib/codegen/src/isa/x86/abi.rs | 8 ++--- lib/codegen/src/lib.rs | 11 +++++++ lib/codegen/src/licm.rs | 4 +-- lib/codegen/src/loop_analysis.rs | 6 ++-- lib/codegen/src/regalloc/affinity.rs | 26 +++++++-------- lib/codegen/src/regalloc/coalescing.rs | 12 +++---- lib/codegen/src/regalloc/coloring.rs | 6 ++-- lib/codegen/src/regalloc/diversion.rs | 4 +-- lib/codegen/src/regalloc/liveness.rs | 9 +++--- lib/codegen/src/regalloc/liverange.rs | 4 +-- lib/codegen/src/regalloc/pressure.rs | 4 +-- lib/codegen/src/regalloc/register_set.rs | 4 +-- lib/codegen/src/regalloc/reload.rs | 2 +- lib/codegen/src/regalloc/solver.rs | 10 +++--- lib/codegen/src/regalloc/spilling.rs | 13 ++++---- lib/codegen/src/regalloc/virtregs.rs | 9 +++--- lib/codegen/src/result.rs | 2 +- lib/codegen/src/settings.rs | 4 +-- lib/codegen/src/verifier/flags.rs | 3 +- lib/codegen/src/verifier/liveness.rs | 6 ++-- lib/codegen/src/verifier/mod.rs | 6 ++-- lib/codegen/src/write.rs | 4 +-- lib/entity/src/lib.rs | 15 +++++++-- lib/entity/src/map.rs | 2 +- lib/entity/src/packed_option.rs | 8 ++--- lib/faerie/src/backend.rs | 2 +- lib/faerie/src/lib.rs | 14 +++++++++ lib/filetests/src/lib.rs | 10 ++++++ lib/filetests/src/runone.rs | 2 +- lib/filetests/src/test_domtree.rs | 4 +-- lib/frontend/src/frontend.rs | 4 +-- lib/frontend/src/lib.rs | 13 +++++++- lib/frontend/src/ssa.rs | 2 +- lib/module/src/lib.rs | 14 +++++++++ lib/module/src/module.rs | 2 +- lib/native/src/lib.rs | 14 +++++++++ lib/reader/src/lib.rs | 14 +++++++++ lib/reader/src/testcommand.rs | 2 +- lib/simplejit/src/lib.rs | 14 +++++++++ lib/simplejit/src/memory.rs | 2 +- lib/umbrella/src/lib.rs | 13 ++++++++ lib/wasm/src/code_translator.rs | 4 +-- lib/wasm/src/environ/dummy.rs | 6 ++-- lib/wasm/src/lib.rs | 14 ++++++++- lib/wasm/src/sections_translator.rs | 4 +-- 71 files changed, 360 insertions(+), 219 deletions(-) diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 20743d923b..d69c561c44 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -1,3 +1,15 @@ +#![deny(trivial_numeric_casts, unused_extern_crates)] +#![warn(unused_import_braces, unstable_features)] +#![cfg_attr(feature="cargo-clippy", warn( + float_arithmetic, + mut_mut, + nonminimal_bool, + option_map_unwrap_or, + option_map_unwrap_or_else, + unicode_not_nfc, + use_self, + ))] + extern crate cretonne_codegen; extern crate cretonne_filetests; extern crate cretonne_reader; diff --git a/lib/codegen/meta/gen_instr.py b/lib/codegen/meta/gen_instr.py index 4e0bc6cb78..0b25cc1835 100644 --- a/lib/codegen/meta/gen_instr.py +++ b/lib/codegen/meta/gen_instr.py @@ -47,7 +47,7 @@ def gen_formats(fmt): with fmt.indented( "impl<'a> From<&'a InstructionData> for InstructionFormat {", '}'): with fmt.indented( - "fn from(inst: &'a InstructionData) -> InstructionFormat {", + "fn from(inst: &'a InstructionData) -> Self {", '}'): m = srcgen.Match('*inst') for f in InstructionFormat.all_formats: diff --git a/lib/codegen/meta/gen_settings.py b/lib/codegen/meta/gen_settings.py index 373c9cb1a9..f111c6c415 100644 --- a/lib/codegen/meta/gen_settings.py +++ b/lib/codegen/meta/gen_settings.py @@ -29,7 +29,7 @@ def gen_enum_types(sgrp, fmt): continue ty = camel_case(setting.name) fmt.doc_comment('Values for `{}`.'.format(setting)) - fmt.line('#[derive(Debug, PartialEq, Eq)]') + fmt.line('#[derive(Debug, Copy, Clone, PartialEq, Eq)]') with fmt.indented('pub enum {} {{'.format(ty), '}'): for v in setting.values: fmt.doc_comment('`{}`.'.format(v)) @@ -223,7 +223,7 @@ def gen_display(sgrp, fmt): fmt.line( 'TEMPLATE.format_toml_value(d.detail,' + 'self.bytes[d.offset as usize], f)?;') - fmt.line('writeln!(f, "")?;') + fmt.line('writeln!(f)?;') fmt.line('Ok(())') @@ -241,7 +241,7 @@ def gen_constructor(sgrp, parent, fmt): fmt.doc_comment('Create flags {} settings group.'.format(sgrp.name)) fmt.line('#[allow(unused_variables)]') with fmt.indented( - 'pub fn new({}) -> Flags {{'.format(args), '}'): + 'pub fn new({}) -> Self {{'.format(args), '}'): fmt.line('let bvec = builder.state_for("{}");'.format(sgrp.name)) fmt.line('let mut bytes = [0; {}];'.format(sgrp.byte_size())) fmt.line( @@ -252,12 +252,12 @@ def gen_constructor(sgrp, parent, fmt): # Stop here without predicates. if len(sgrp.predicate_number) == sgrp.boolean_settings: - fmt.line('Flags { bytes: bytes }') + fmt.line('Self { bytes }') return # Now compute the predicates. fmt.line( - 'let mut {} = Flags {{ bytes: bytes }};' + 'let mut {} = Self {{ bytes }};' .format(sgrp.name)) for pred, number in sgrp.predicate_number.items(): diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py index ea0da832b7..0f9a79d6a7 100644 --- a/lib/codegen/meta/isa/x86/recipes.py +++ b/lib/codegen/meta/isa/x86/recipes.py @@ -530,7 +530,7 @@ puid_bool = TailRecipe( // The destination register is encoded in the low bits of the opcode. // No ModR/M. PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); - let imm: u32 = if imm.into() { 1 } else { 0 }; + let imm: u32 = if imm { 1 } else { 0 }; sink.put4(imm); ''') diff --git a/lib/codegen/src/abi.rs b/lib/codegen/src/abi.rs index 88a8a424a6..5f1452159b 100644 --- a/lib/codegen/src/abi.rs +++ b/lib/codegen/src/abi.rs @@ -25,13 +25,13 @@ pub enum ArgAction { } impl From for ArgAction { - fn from(x: ArgumentLoc) -> ArgAction { + fn from(x: ArgumentLoc) -> Self { ArgAction::Assign(x) } } impl From for ArgAction { - fn from(x: ValueConversion) -> ArgAction { + fn from(x: ValueConversion) -> Self { ArgAction::Convert(x) } } diff --git a/lib/codegen/src/bforest/map.rs b/lib/codegen/src/bforest/map.rs index 6f8bedc225..56e4870679 100644 --- a/lib/codegen/src/bforest/map.rs +++ b/lib/codegen/src/bforest/map.rs @@ -45,8 +45,8 @@ where C: Comparator, { /// Create a new empty forest. - pub fn new() -> MapForest { - MapForest { nodes: NodePool::new() } + pub fn new() -> Self { + Self { nodes: NodePool::new() } } /// Clear all maps in the forest. @@ -83,8 +83,8 @@ where C: Comparator, { /// Make an empty map. - pub fn new() -> Map { - Map { + pub fn new() -> Self { + Self { root: None.into(), unused: PhantomData, } diff --git a/lib/codegen/src/bforest/node.rs b/lib/codegen/src/bforest/node.rs index f4a85613b5..d47a16283b 100644 --- a/lib/codegen/src/bforest/node.rs +++ b/lib/codegen/src/bforest/node.rs @@ -73,7 +73,7 @@ impl NodeData { } /// Create an inner node with a single key and two sub-trees. - pub fn inner(left: Node, key: F::Key, right: Node) -> NodeData { + pub fn inner(left: Node, key: F::Key, right: Node) -> Self { // Splat the key and right node to the whole array. // Saves us from inventing a default/reserved value. let mut tree = [right; INNER_SIZE]; @@ -86,7 +86,7 @@ impl NodeData { } /// Create a leaf node with a single key-value pair. - pub fn leaf(key: F::Key, value: F::Value) -> NodeData { + pub fn leaf(key: F::Key, value: F::Value) -> Self { NodeData::Leaf { size: 1, keys: F::splat_key(key), @@ -360,7 +360,7 @@ impl NodeData { /// /// In the first case, `None` is returned. In the second case, the new critical key for the /// right sibling node is returned. - pub fn balance(&mut self, crit_key: F::Key, rhs: &mut NodeData) -> Option { + pub fn balance(&mut self, crit_key: F::Key, rhs: &mut Self) -> Option { match (self, rhs) { (&mut NodeData::Inner { size: ref mut l_size, @@ -514,7 +514,7 @@ pub(super) enum Removed { impl Removed { /// Create a `Removed` status from a size and capacity. - fn new(removed: usize, new_size: usize, capacity: usize) -> Removed { + fn new(removed: usize, new_size: usize, capacity: usize) -> Self { if 2 * new_size >= capacity { if removed == new_size { Removed::Rightmost diff --git a/lib/codegen/src/bforest/path.rs b/lib/codegen/src/bforest/path.rs index d011455571..3ed91ee09a 100644 --- a/lib/codegen/src/bforest/path.rs +++ b/lib/codegen/src/bforest/path.rs @@ -22,8 +22,8 @@ pub(super) struct Path { } impl Default for Path { - fn default() -> Path { - Path { + fn default() -> Self { + Self { size: 0, node: [Node(0); MAX_PATH], entry: [0; MAX_PATH], diff --git a/lib/codegen/src/bforest/pool.rs b/lib/codegen/src/bforest/pool.rs index 0e312f9380..eed9402c84 100644 --- a/lib/codegen/src/bforest/pool.rs +++ b/lib/codegen/src/bforest/pool.rs @@ -12,8 +12,8 @@ pub(super) struct NodePool { impl NodePool { /// Allocate a new empty pool of nodes. - pub fn new() -> NodePool { - NodePool { + pub fn new() -> Self { + Self { nodes: PrimaryMap::new(), freelist: None, } diff --git a/lib/codegen/src/bforest/set.rs b/lib/codegen/src/bforest/set.rs index fbd0dbfca1..d175b1a752 100644 --- a/lib/codegen/src/bforest/set.rs +++ b/lib/codegen/src/bforest/set.rs @@ -42,8 +42,8 @@ where C: Comparator, { /// Create a new empty forest. - pub fn new() -> SetForest { - SetForest { nodes: NodePool::new() } + pub fn new() -> Self { + Self { nodes: NodePool::new() } } /// Clear all sets in the forest. @@ -78,8 +78,8 @@ where C: Comparator, { /// Make an empty set. - pub fn new() -> Set { - Set { + pub fn new() -> Self { + Self { root: None.into(), unused: PhantomData, } diff --git a/lib/codegen/src/binemit/mod.rs b/lib/codegen/src/binemit/mod.rs index fbe88cc61f..9752901ce9 100644 --- a/lib/codegen/src/binemit/mod.rs +++ b/lib/codegen/src/binemit/mod.rs @@ -48,12 +48,12 @@ impl fmt::Display for Reloc { /// already unambigious, e.g. cton syntax with isa specified. In other contexts, use Debug. fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match *self { - Reloc::Abs4 => write!(f, "{}", "Abs4"), - Reloc::Abs8 => write!(f, "{}", "Abs8"), - Reloc::X86PCRel4 => write!(f, "{}", "PCRel4"), - Reloc::X86GOTPCRel4 => write!(f, "{}", "GOTPCRel4"), - Reloc::X86PLTRel4 => write!(f, "{}", "PLTRel4"), - Reloc::Arm32Call | Reloc::Arm64Call | Reloc::RiscvCall => write!(f, "{}", "Call"), + Reloc::Abs4 => write!(f, "Abs4"), + Reloc::Abs8 => write!(f, "Abs8"), + Reloc::X86PCRel4 => write!(f, "PCRel4"), + Reloc::X86GOTPCRel4 => write!(f, "GOTPCRel4"), + Reloc::X86PLTRel4 => write!(f, "PLTRel4"), + Reloc::Arm32Call | Reloc::Arm64Call | Reloc::RiscvCall => write!(f, "Call"), } } } diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index ad31e9025b..7b3e0253e1 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -52,7 +52,7 @@ impl Context { /// The returned instance should be reused for compiling multiple functions in order to avoid /// needless allocator thrashing. pub fn new() -> Self { - Context::for_function(Function::new()) + Self::for_function(Function::new()) } /// Allocate a new compilation context with an existing Function. @@ -61,7 +61,7 @@ impl Context { /// needless allocator thrashing. pub fn for_function(func: Function) -> Self { Self { - func: func, + func, cfg: ControlFlowGraph::new(), domtree: DominatorTree::new(), regalloc: regalloc::Context::new(), diff --git a/lib/codegen/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs index 60ef1e6e42..b717d524e3 100644 --- a/lib/codegen/src/dominator_tree.rs +++ b/lib/codegen/src/dominator_tree.rs @@ -528,8 +528,8 @@ struct ExtraNode { /// Creating and computing the dominator tree pre-order. impl DominatorTreePreorder { /// Create a new blank `DominatorTreePreorder`. - pub fn new() -> DominatorTreePreorder { - DominatorTreePreorder { + pub fn new() -> Self { + Self { nodes: EntityMap::new(), stack: Vec::new(), } diff --git a/lib/codegen/src/ir/entities.rs b/lib/codegen/src/ir/entities.rs index d0c627952b..30c7cd3fb4 100644 --- a/lib/codegen/src/ir/entities.rs +++ b/lib/codegen/src/ir/entities.rs @@ -31,7 +31,7 @@ impl Ebb { /// Create a new EBB reference from its number. This corresponds to the `ebbNN` representation. /// /// This method is for use by the parser. - pub fn with_number(n: u32) -> Option { + pub fn with_number(n: u32) -> Option { if n < u32::MAX { Some(Ebb(n)) } else { None } } } @@ -46,7 +46,7 @@ impl Value { /// This is the number in the `vNN` notation. /// /// This method is for use by the parser. - pub fn with_number(n: u32) -> Option { + pub fn with_number(n: u32) -> Option { if n < u32::MAX / 2 { Some(Value(n)) } else { @@ -69,7 +69,7 @@ impl StackSlot { /// Create a new stack slot reference from its number. /// /// This method is for use by the parser. - pub fn with_number(n: u32) -> Option { + pub fn with_number(n: u32) -> Option { if n < u32::MAX { Some(StackSlot(n)) } else { @@ -87,7 +87,7 @@ impl GlobalVar { /// Create a new global variable reference from its number. /// /// This method is for use by the parser. - pub fn with_number(n: u32) -> Option { + pub fn with_number(n: u32) -> Option { if n < u32::MAX { Some(GlobalVar(n)) } else { @@ -105,7 +105,7 @@ impl JumpTable { /// Create a new jump table reference from its number. /// /// This method is for use by the parser. - pub fn with_number(n: u32) -> Option { + pub fn with_number(n: u32) -> Option { if n < u32::MAX { Some(JumpTable(n)) } else { @@ -123,7 +123,7 @@ impl FuncRef { /// Create a new external function reference from its number. /// /// This method is for use by the parser. - pub fn with_number(n: u32) -> Option { + pub fn with_number(n: u32) -> Option { if n < u32::MAX { Some(FuncRef(n)) } else { None } } } @@ -137,7 +137,7 @@ impl SigRef { /// Create a new function signature reference from its number. /// /// This method is for use by the parser. - pub fn with_number(n: u32) -> Option { + pub fn with_number(n: u32) -> Option { if n < u32::MAX { Some(SigRef(n)) } else { None } } } @@ -151,7 +151,7 @@ impl Heap { /// Create a new heap reference from its number. /// /// This method is for use by the parser. - pub fn with_number(n: u32) -> Option { + pub fn with_number(n: u32) -> Option { if n < u32::MAX { Some(Heap(n)) } else { None } } } @@ -205,55 +205,55 @@ impl fmt::Debug for AnyEntity { } impl From for AnyEntity { - fn from(r: Ebb) -> AnyEntity { + fn from(r: Ebb) -> Self { AnyEntity::Ebb(r) } } impl From for AnyEntity { - fn from(r: Inst) -> AnyEntity { + fn from(r: Inst) -> Self { AnyEntity::Inst(r) } } impl From for AnyEntity { - fn from(r: Value) -> AnyEntity { + fn from(r: Value) -> Self { AnyEntity::Value(r) } } impl From for AnyEntity { - fn from(r: StackSlot) -> AnyEntity { + fn from(r: StackSlot) -> Self { AnyEntity::StackSlot(r) } } impl From for AnyEntity { - fn from(r: GlobalVar) -> AnyEntity { + fn from(r: GlobalVar) -> Self { AnyEntity::GlobalVar(r) } } impl From for AnyEntity { - fn from(r: JumpTable) -> AnyEntity { + fn from(r: JumpTable) -> Self { AnyEntity::JumpTable(r) } } impl From for AnyEntity { - fn from(r: FuncRef) -> AnyEntity { + fn from(r: FuncRef) -> Self { AnyEntity::FuncRef(r) } } impl From for AnyEntity { - fn from(r: SigRef) -> AnyEntity { + fn from(r: SigRef) -> Self { AnyEntity::SigRef(r) } } impl From for AnyEntity { - fn from(r: Heap) -> AnyEntity { + fn from(r: Heap) -> Self { AnyEntity::Heap(r) } } diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs index afb32fee81..01008946e6 100644 --- a/lib/codegen/src/ir/extfunc.rs +++ b/lib/codegen/src/ir/extfunc.rs @@ -304,7 +304,7 @@ impl fmt::Display for ArgumentPurpose { impl FromStr for ArgumentPurpose { type Err = (); - fn from_str(s: &str) -> Result { + fn from_str(s: &str) -> Result { match s { "normal" => Ok(ArgumentPurpose::Normal), "sret" => Ok(ArgumentPurpose::StructReturn), diff --git a/lib/codegen/src/ir/extname.rs b/lib/codegen/src/ir/extname.rs index f3a69f7e7a..dcb3666703 100644 --- a/lib/codegen/src/ir/extname.rs +++ b/lib/codegen/src/ir/extname.rs @@ -56,7 +56,7 @@ impl ExternalName { /// let name = ExternalName::testcase("hello"); /// assert_eq!(name.to_string(), "%hello"); /// ``` - pub fn testcase>(v: T) -> ExternalName { + pub fn testcase>(v: T) -> Self { let vec = v.as_ref(); let len = cmp::min(vec.len(), TESTCASE_NAME_LENGTH); let mut bytes = [0u8; TESTCASE_NAME_LENGTH]; @@ -77,17 +77,14 @@ impl ExternalName { /// let name = ExternalName::user(123, 456); /// assert_eq!(name.to_string(), "u123:456"); /// ``` - pub fn user(namespace: u32, index: u32) -> ExternalName { - ExternalName::User { - namespace: namespace, - index: index, - } + pub fn user(namespace: u32, index: u32) -> Self { + ExternalName::User { namespace, index } } } impl Default for ExternalName { - fn default() -> ExternalName { - ExternalName::user(0, 0) + fn default() -> Self { + Self::user(0, 0) } } diff --git a/lib/codegen/src/ir/immediates.rs b/lib/codegen/src/ir/immediates.rs index d56e963bb1..32b64dabf8 100644 --- a/lib/codegen/src/ir/immediates.rs +++ b/lib/codegen/src/ir/immediates.rs @@ -18,12 +18,12 @@ pub struct Imm64(i64); impl Imm64 { /// Create a new `Imm64` representing the signed number `x`. - pub fn new(x: i64) -> Imm64 { + pub fn new(x: i64) -> Self { Imm64(x) } /// Return self negated. - pub fn wrapping_neg(self) -> Imm64 { + pub fn wrapping_neg(self) -> Self { Imm64(self.0.wrapping_neg()) } } @@ -143,8 +143,8 @@ impl FromStr for Imm64 { type Err = &'static str; // Parse a decimal or hexadecimal `Imm64`, formatted as above. - fn from_str(s: &str) -> Result { - parse_i64(s).map(Imm64::new) + fn from_str(s: &str) -> Result { + parse_i64(s).map(Self::new) } } @@ -191,7 +191,7 @@ impl FromStr for Uimm32 { type Err = &'static str; // Parse a decimal or hexadecimal `Uimm32`, formatted as above. - fn from_str(s: &str) -> Result { + fn from_str(s: &str) -> Result { parse_i64(s).and_then(|x| if 0 <= x && x <= i64::from(u32::MAX) { Ok(Uimm32(x as u32)) } else { @@ -209,7 +209,7 @@ pub struct Offset32(i32); impl Offset32 { /// Create a new `Offset32` representing the signed number `x`. - pub fn new(x: i32) -> Offset32 { + pub fn new(x: i32) -> Self { Offset32(x) } } @@ -255,14 +255,14 @@ impl FromStr for Offset32 { type Err = &'static str; // Parse a decimal or hexadecimal `Offset32`, formatted as above. - fn from_str(s: &str) -> Result { + fn from_str(s: &str) -> Result { if !(s.starts_with('-') || s.starts_with('+')) { return Err("Offset must begin with sign"); } parse_i64(s).and_then(|x| if i64::from(i32::MIN) <= x && x <= i64::from(i32::MAX) { - Ok(Offset32::new(x as i32)) + Ok(Self::new(x as i32)) } else { Err("Offset out of range") }) @@ -524,12 +524,12 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { impl Ieee32 { /// Create a new `Ieee32` containing the bits of `x`. - pub fn with_bits(x: u32) -> Ieee32 { + pub fn with_bits(x: u32) -> Self { Ieee32(x) } /// Create an `Ieee32` number representing `2.0^n`. - pub fn pow2>(n: I) -> Ieee32 { + pub fn pow2>(n: I) -> Self { let n = n.into(); let w = 8; let t = 23; @@ -542,7 +542,7 @@ impl Ieee32 { /// Create an `Ieee32` number representing the greatest negative value /// not convertable from f32 to a signed integer with width n. - pub fn fcvt_to_sint_negative_overflow>(n: I) -> Ieee32 { + pub fn fcvt_to_sint_negative_overflow>(n: I) -> Self { let n = n.into(); debug_assert!(n < 32); debug_assert!(23 + 1 - n < 32); @@ -552,12 +552,12 @@ impl Ieee32 { } /// Return self negated. - pub fn neg(self) -> Ieee32 { + pub fn neg(self) -> Self { Ieee32(self.0 ^ (1 << 31)) } /// Create a new `Ieee32` representing the number `x`. - pub fn with_float(x: f32) -> Ieee32 { + pub fn with_float(x: f32) -> Self { Ieee32(unsafe { mem::transmute(x) }) } @@ -577,7 +577,7 @@ impl Display for Ieee32 { impl FromStr for Ieee32 { type Err = &'static str; - fn from_str(s: &str) -> Result { + fn from_str(s: &str) -> Result { match parse_float(s, 8, 23) { Ok(b) => Ok(Ieee32(b as u32)), Err(s) => Err(s), @@ -587,12 +587,12 @@ impl FromStr for Ieee32 { impl Ieee64 { /// Create a new `Ieee64` containing the bits of `x`. - pub fn with_bits(x: u64) -> Ieee64 { + pub fn with_bits(x: u64) -> Self { Ieee64(x) } /// Create an `Ieee64` number representing `2.0^n`. - pub fn pow2>(n: I) -> Ieee64 { + pub fn pow2>(n: I) -> Self { let n = n.into(); let w = 11; let t = 52; @@ -605,7 +605,7 @@ impl Ieee64 { /// Create an `Ieee64` number representing the greatest negative value /// not convertable from f64 to a signed integer with width n. - pub fn fcvt_to_sint_negative_overflow>(n: I) -> Ieee64 { + pub fn fcvt_to_sint_negative_overflow>(n: I) -> Self { let n = n.into(); debug_assert!(n < 64); debug_assert!(52 + 1 - n < 64); @@ -615,12 +615,12 @@ impl Ieee64 { } /// Return self negated. - pub fn neg(self) -> Ieee64 { + pub fn neg(self) -> Self { Ieee64(self.0 ^ (1 << 63)) } /// Create a new `Ieee64` representing the number `x`. - pub fn with_float(x: f64) -> Ieee64 { + pub fn with_float(x: f64) -> Self { Ieee64(unsafe { mem::transmute(x) }) } @@ -640,7 +640,7 @@ impl Display for Ieee64 { impl FromStr for Ieee64 { type Err = &'static str; - fn from_str(s: &str) -> Result { + fn from_str(s: &str) -> Result { match parse_float(s, 11, 52) { Ok(b) => Ok(Ieee64(b)), Err(s) => Err(s), diff --git a/lib/codegen/src/ir/instructions.rs b/lib/codegen/src/ir/instructions.rs index d4ca4a1396..e1db66099b 100644 --- a/lib/codegen/src/ir/instructions.rs +++ b/lib/codegen/src/ir/instructions.rs @@ -72,7 +72,7 @@ impl FromStr for Opcode { type Err = &'static str; /// Parse an Opcode name from a string. - fn from_str(s: &str) -> Result { + fn from_str(s: &str) -> Result { use constant_hash::{probe, simple_hash, Table}; impl<'a> Table<&'a str> for [Option] { @@ -85,7 +85,7 @@ impl FromStr for Opcode { } } - match probe::<&str, [Option]>(&OPCODE_HASH_TABLE, s, simple_hash(s)) { + match probe::<&str, [Option]>(&OPCODE_HASH_TABLE, s, simple_hash(s)) { Err(_) => Err("Unknown opcode"), // We unwrap here because probe() should have ensured that the entry // at this index is not None. diff --git a/lib/codegen/src/ir/libcall.rs b/lib/codegen/src/ir/libcall.rs index 6434197247..12216522ae 100644 --- a/lib/codegen/src/ir/libcall.rs +++ b/lib/codegen/src/ir/libcall.rs @@ -72,7 +72,7 @@ impl LibCall { /// given opcode and controlling type variable. /// /// Returns `None` if no well-known library routine name exists for that instruction. - pub fn for_inst(opcode: Opcode, ctrl_type: Type) -> Option { + pub fn for_inst(opcode: Opcode, ctrl_type: Type) -> Option { Some(match ctrl_type { types::F32 => { match opcode { diff --git a/lib/codegen/src/ir/progpoint.rs b/lib/codegen/src/ir/progpoint.rs index 4a0785aef4..86c0e591e4 100644 --- a/lib/codegen/src/ir/progpoint.rs +++ b/lib/codegen/src/ir/progpoint.rs @@ -17,7 +17,7 @@ use std::u32; pub struct ProgramPoint(u32); impl From for ProgramPoint { - fn from(inst: Inst) -> ProgramPoint { + fn from(inst: Inst) -> Self { let idx = inst.index(); debug_assert!(idx < (u32::MAX / 2) as usize); ProgramPoint((idx * 2) as u32) @@ -25,7 +25,7 @@ impl From for ProgramPoint { } impl From for ProgramPoint { - fn from(ebb: Ebb) -> ProgramPoint { + fn from(ebb: Ebb) -> Self { let idx = ebb.index(); debug_assert!(idx < (u32::MAX / 2) as usize); ProgramPoint((idx * 2 + 1) as u32) @@ -33,7 +33,7 @@ impl From for ProgramPoint { } impl From for ProgramPoint { - fn from(def: ValueDef) -> ProgramPoint { + fn from(def: ValueDef) -> Self { match def { ValueDef::Result(inst, _) => inst.into(), ValueDef::Param(ebb, _) => ebb.into(), @@ -62,19 +62,19 @@ impl ExpandedProgramPoint { } impl From for ExpandedProgramPoint { - fn from(inst: Inst) -> ExpandedProgramPoint { + fn from(inst: Inst) -> Self { ExpandedProgramPoint::Inst(inst) } } impl From for ExpandedProgramPoint { - fn from(ebb: Ebb) -> ExpandedProgramPoint { + fn from(ebb: Ebb) -> Self { ExpandedProgramPoint::Ebb(ebb) } } impl From for ExpandedProgramPoint { - fn from(def: ValueDef) -> ExpandedProgramPoint { + fn from(def: ValueDef) -> Self { match def { ValueDef::Result(inst, _) => inst.into(), ValueDef::Param(ebb, _) => ebb.into(), @@ -83,7 +83,7 @@ impl From for ExpandedProgramPoint { } impl From for ExpandedProgramPoint { - fn from(pp: ProgramPoint) -> ExpandedProgramPoint { + fn from(pp: ProgramPoint) -> Self { if pp.0 & 1 == 0 { ExpandedProgramPoint::Inst(Inst::new((pp.0 / 2) as usize)) } else { diff --git a/lib/codegen/src/ir/sourceloc.rs b/lib/codegen/src/ir/sourceloc.rs index 768bbf99d5..6de2206c8a 100644 --- a/lib/codegen/src/ir/sourceloc.rs +++ b/lib/codegen/src/ir/sourceloc.rs @@ -17,7 +17,7 @@ pub struct SourceLoc(u32); impl SourceLoc { /// Create a new source location with the given bits. - pub fn new(bits: u32) -> SourceLoc { + pub fn new(bits: u32) -> Self { SourceLoc(bits) } diff --git a/lib/codegen/src/ir/stackslot.rs b/lib/codegen/src/ir/stackslot.rs index bdf2c95825..aecf8abc8f 100644 --- a/lib/codegen/src/ir/stackslot.rs +++ b/lib/codegen/src/ir/stackslot.rs @@ -70,7 +70,7 @@ pub enum StackSlotKind { impl FromStr for StackSlotKind { type Err = (); - fn from_str(s: &str) -> Result { + fn from_str(s: &str) -> Result { use self::StackSlotKind::*; match s { "explicit_slot" => Ok(ExplicitSlot), @@ -117,8 +117,8 @@ pub struct StackSlotData { impl StackSlotData { /// Create a stack slot with the specified byte size. - pub fn new(kind: StackSlotKind, size: StackSize) -> StackSlotData { - StackSlotData { + pub fn new(kind: StackSlotKind, size: StackSize) -> Self { + Self { kind, size, offset: None, diff --git a/lib/codegen/src/ir/types.rs b/lib/codegen/src/ir/types.rs index 3dc71454de..c4dd5ad724 100644 --- a/lib/codegen/src/ir/types.rs +++ b/lib/codegen/src/ir/types.rs @@ -39,7 +39,7 @@ impl Type { /// Get the lane type of this SIMD vector type. /// /// A lane type is the same as a SIMD vector type with one lane, so it returns itself. - pub fn lane_type(self) -> Type { + pub fn lane_type(self) -> Self { if self.0 < VECTOR_BASE { self } else { @@ -72,7 +72,7 @@ impl Type { } /// Get an integer type with the requested number of bits. - pub fn int(bits: u16) -> Option { + pub fn int(bits: u16) -> Option { match bits { 8 => Some(I8), 16 => Some(I16), @@ -83,7 +83,7 @@ impl Type { } /// Get a type with the same number of lanes as `self`, but using `lane` as the lane type. - fn replace_lanes(self, lane: Type) -> Type { + fn replace_lanes(self, lane: Self) -> Self { debug_assert!(lane.is_lane() && !self.is_special()); Type((lane.0 & 0x0f) | (self.0 & 0xf0)) } @@ -93,7 +93,7 @@ impl Type { /// /// Scalar types are treated as vectors with one lane, so they are converted to the multi-bit /// boolean types. - pub fn as_bool_pedantic(self) -> Type { + pub fn as_bool_pedantic(self) -> Self { // Replace the low 4 bits with the boolean version, preserve the high 4 bits. self.replace_lanes(match self.lane_type() { B8 | I8 => B8, @@ -108,7 +108,7 @@ impl Type { /// booleans of the same size. /// /// Scalar types are all converted to `b1` which is usually what you want. - pub fn as_bool(self) -> Type { + pub fn as_bool(self) -> Self { if !self.is_vector() { B1 } else { @@ -118,7 +118,7 @@ impl Type { /// Get a type with the same number of lanes as this type, but with lanes that are half the /// number of bits. - pub fn half_width(self) -> Option { + pub fn half_width(self) -> Option { Some(self.replace_lanes(match self.lane_type() { I16 => I8, I32 => I16, @@ -133,7 +133,7 @@ impl Type { /// Get a type with the same number of lanes as this type, but with lanes that are twice the /// number of bits. - pub fn double_width(self) -> Option { + pub fn double_width(self) -> Option { Some(self.replace_lanes(match self.lane_type() { I8 => I16, I16 => I32, @@ -235,7 +235,7 @@ impl Type { /// /// If this is already a SIMD vector type, this produces a SIMD vector type with `n * /// self.lane_count()` lanes. - pub fn by(self, n: u16) -> Option { + pub fn by(self, n: u16) -> Option { if self.lane_bits() == 0 || !n.is_power_of_two() { return None; } @@ -251,7 +251,7 @@ impl Type { /// Get a SIMD vector with half the number of lanes. /// /// There is no `double_vector()` method. Use `t.by(2)` instead. - pub fn half_vector(self) -> Option { + pub fn half_vector(self) -> Option { if self.is_vector() { Some(Type(self.0 - 0x10)) } else { @@ -268,7 +268,7 @@ impl Type { /// /// 1. `self.lane_count() == other.lane_count()` and /// 2. `self.lane_bits() >= other.lane_bits()` - pub fn wider_or_equal(self, other: Type) -> bool { + pub fn wider_or_equal(self, other: Self) -> bool { self.lane_count() == other.lane_count() && self.lane_bits() >= other.lane_bits() } } diff --git a/lib/codegen/src/isa/encoding.rs b/lib/codegen/src/isa/encoding.rs index 1b850d5c64..c13c4de40d 100644 --- a/lib/codegen/src/isa/encoding.rs +++ b/lib/codegen/src/isa/encoding.rs @@ -18,8 +18,8 @@ pub struct Encoding { impl Encoding { /// Create a new `Encoding` containing `(recipe, bits)`. - pub fn new(recipe: u16, bits: u16) -> Encoding { - Encoding { recipe, bits } + pub fn new(recipe: u16, bits: u16) -> Self { + Self { recipe, bits } } /// Get the recipe number in this encoding. @@ -122,10 +122,10 @@ impl EncInfo { /// /// Returns 0 for illegal encodings. pub fn bytes(&self, enc: Encoding) -> CodeOffset { - self.sizing - .get(enc.recipe()) - .map(|s| CodeOffset::from(s.bytes)) - .unwrap_or(0) + self.sizing.get(enc.recipe()).map_or( + 0, + |s| CodeOffset::from(s.bytes), + ) } /// Get the branch range that is supported by `enc`, if any. diff --git a/lib/codegen/src/isa/riscv/abi.rs b/lib/codegen/src/isa/riscv/abi.rs index 96e532e827..ebc4a67fab 100644 --- a/lib/codegen/src/isa/riscv/abi.rs +++ b/lib/codegen/src/isa/riscv/abi.rs @@ -24,8 +24,8 @@ struct Args { } impl Args { - fn new(bits: u16, enable_e: bool) -> Args { - Args { + fn new(bits: u16, enable_e: bool) -> Self { + Self { pointer_bits: bits, pointer_bytes: u32::from(bits) / 8, pointer_type: Type::int(bits).unwrap(), diff --git a/lib/codegen/src/isa/stack.rs b/lib/codegen/src/isa/stack.rs index 0db6279d3f..91df090cb4 100644 --- a/lib/codegen/src/isa/stack.rs +++ b/lib/codegen/src/isa/stack.rs @@ -23,10 +23,10 @@ pub struct StackRef { impl StackRef { /// Get a reference to the stack slot `ss` using one of the base pointers in `mask`. - pub fn masked(ss: StackSlot, mask: StackBaseMask, frame: &StackSlots) -> Option { + pub fn masked(ss: StackSlot, mask: StackBaseMask, frame: &StackSlots) -> Option { // Try an SP-relative reference. if mask.contains(StackBase::SP) { - return Some(StackRef::sp(ss, frame)); + return Some(Self::sp(ss, frame)); } // No reference possible with this mask. @@ -34,7 +34,7 @@ impl StackRef { } /// Get a reference to `ss` using the stack pointer as a base. - pub fn sp(ss: StackSlot, frame: &StackSlots) -> StackRef { + pub fn sp(ss: StackSlot, frame: &StackSlots) -> Self { let size = frame.frame_size.expect( "Stack layout must be computed before referencing stack slots", ); @@ -48,7 +48,7 @@ impl StackRef { let sp_offset = -(size as StackOffset); slot.offset.unwrap() - sp_offset }; - StackRef { + Self { base: StackBase::SP, offset, } diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs index 16cea926b8..fe64aed5b1 100644 --- a/lib/codegen/src/isa/x86/abi.rs +++ b/lib/codegen/src/isa/x86/abi.rs @@ -34,8 +34,8 @@ struct Args { } impl Args { - fn new(bits: u16, gpr: &'static [RU], fpr_limit: usize, call_conv: CallConv) -> Args { - Args { + fn new(bits: u16, gpr: &'static [RU], fpr_limit: usize, call_conv: CallConv) -> Self { + Self { pointer_bytes: u32::from(bits) / 8, pointer_bits: bits, pointer_type: ir::Type::int(bits).unwrap(), @@ -44,7 +44,7 @@ impl Args { fpr_limit, fpr_used: 0, offset: 0, - call_conv: call_conv, + call_conv, } } } @@ -205,7 +205,7 @@ fn callee_saved_gprs_used(flags: &shared_settings::Flags, func: &ir::Function) - } used.intersect(&all_callee_saved); - return used; + used } pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs index 8860cb2e69..bdb11f5c30 100644 --- a/lib/codegen/src/lib.rs +++ b/lib/codegen/src/lib.rs @@ -1,6 +1,7 @@ //! Cretonne code generation library. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature="cargo-clippy", allow( // Rustfmt 0.9.0 is at odds with this lint: @@ -29,6 +30,16 @@ redundant_field_names, useless_let_if_seq, len_without_is_empty))] +#![cfg_attr(feature="cargo-clippy", warn( + float_arithmetic, + mut_mut, + nonminimal_bool, + option_map_unwrap_or, + option_map_unwrap_or_else, + print_stdout, + unicode_not_nfc, + use_self, + ))] pub use context::Context; pub use legalizer::legalize_function; diff --git a/lib/codegen/src/licm.rs b/lib/codegen/src/licm.rs index b5a8f5ed68..154b222df2 100644 --- a/lib/codegen/src/licm.rs +++ b/lib/codegen/src/licm.rs @@ -150,7 +150,7 @@ fn is_loop_invariant(inst: Inst, dfg: &DataFlowGraph, loop_values: &HashSet) -> LoopData { - LoopData { - header: header, + pub fn new(header: Ebb, parent: Option) -> Self { + Self { + header, parent: parent.into(), } } diff --git a/lib/codegen/src/regalloc/affinity.rs b/lib/codegen/src/regalloc/affinity.rs index e0078579a7..d514cf4dd4 100644 --- a/lib/codegen/src/regalloc/affinity.rs +++ b/lib/codegen/src/regalloc/affinity.rs @@ -19,7 +19,7 @@ pub enum Affinity { /// /// This indicates a value that is not defined or used by any real instructions. It is a ghost /// value that won't appear in the final program. - None, + Unassigned, /// This value should be placed in a spill slot on the stack. Stack, @@ -30,16 +30,16 @@ pub enum Affinity { impl Default for Affinity { fn default() -> Self { - Affinity::None + Affinity::Unassigned } } impl Affinity { /// Create an affinity that satisfies a single constraint. /// - /// This will never create an `Affinity::None`. + /// This will never create an `Affinity::Unassigned`. /// Use the `Default` implementation for that. - pub fn new(constraint: &OperandConstraint) -> Affinity { + pub fn new(constraint: &OperandConstraint) -> Self { if constraint.kind == ConstraintKind::Stack { Affinity::Stack } else { @@ -48,18 +48,18 @@ impl Affinity { } /// Create an affinity that matches an ABI argument for `isa`. - pub fn abi(arg: &AbiParam, isa: &TargetIsa) -> Affinity { + pub fn abi(arg: &AbiParam, isa: &TargetIsa) -> Self { match arg.location { - ArgumentLoc::Unassigned => Affinity::None, + ArgumentLoc::Unassigned => Affinity::Unassigned, ArgumentLoc::Reg(_) => Affinity::Reg(isa.regclass_for_abi_type(arg.value_type).into()), ArgumentLoc::Stack(_) => Affinity::Stack, } } - /// Is this the `None` affinity? - pub fn is_none(self) -> bool { + /// Is this the `Unassigned` affinity? + pub fn is_unassigned(self) -> bool { match self { - Affinity::None => true, + Affinity::Unassigned => true, _ => false, } } @@ -86,15 +86,15 @@ impl Affinity { /// satisfies the constraint. pub fn merge(&mut self, constraint: &OperandConstraint, reg_info: &RegInfo) { match *self { - Affinity::None => *self = Affinity::new(constraint), + Affinity::Unassigned => *self = Self::new(constraint), Affinity::Reg(rc) => { // If the preferred register class is a subclass of the constraint, there's no need // to change anything. if constraint.kind != ConstraintKind::Stack && !constraint.regclass.has_subclass(rc) { - // If the register classes don't overlap, `intersect` returns `None`, and we - // just keep our previous affinity. + // If the register classes don't overlap, `intersect` returns `Unassigned`, and + // we just keep our previous affinity. if let Some(subclass) = constraint.regclass.intersect_index(reg_info.rc(rc)) { // This constraint shrinks our preferred register class. *self = Affinity::Reg(subclass); @@ -118,7 +118,7 @@ pub struct DisplayAffinity<'a>(Affinity, Option<&'a RegInfo>); impl<'a> fmt::Display for DisplayAffinity<'a> { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match self.0 { - Affinity::None => write!(f, "none"), + Affinity::Unassigned => write!(f, "unassigned"), Affinity::Stack => write!(f, "stack"), Affinity::Reg(rci) => { match self.1 { diff --git a/lib/codegen/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs index b77488fec7..096c8546bc 100644 --- a/lib/codegen/src/regalloc/coalescing.rs +++ b/lib/codegen/src/regalloc/coalescing.rs @@ -704,10 +704,10 @@ struct Node { impl Node { /// Create a node representing `value`. - pub fn value(value: Value, set_id: u8, func: &Function) -> Node { + pub fn value(value: Value, set_id: u8, func: &Function) -> Self { let def = func.dfg.value_def(value).pp(); let ebb = func.layout.pp_ebb(def); - Node { + Self { def, ebb, is_vcopy: false, @@ -717,10 +717,10 @@ impl Node { } /// Create a node representing a virtual copy. - pub fn vcopy(branch: Inst, value: Value, set_id: u8, func: &Function) -> Node { + pub fn vcopy(branch: Inst, value: Value, set_id: u8, func: &Function) -> Self { let def = branch.into(); let ebb = func.layout.pp_ebb(def); - Node { + Self { def, ebb, is_vcopy: true, @@ -891,8 +891,8 @@ struct VirtualCopies { impl VirtualCopies { /// Create an empty VirtualCopies struct. - pub fn new() -> VirtualCopies { - VirtualCopies { + pub fn new() -> Self { + Self { params: Vec::new(), branches: Vec::new(), filter: Vec::new(), diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs index 4ddda53e16..e6281920d2 100644 --- a/lib/codegen/src/regalloc/coloring.rs +++ b/lib/codegen/src/regalloc/coloring.rs @@ -272,7 +272,7 @@ impl<'a> Context<'a> { Affinity::Stack => debug_assert!(abi.location.is_stack()), // This is a ghost value, unused in the function. Don't assign it to a location // either. - Affinity::None => {} + Affinity::Unassigned => {} } } @@ -1126,8 +1126,8 @@ struct AvailableRegs { impl AvailableRegs { /// Initialize both the input and global sets from `regs`. - pub fn new(regs: &RegisterSet) -> AvailableRegs { - AvailableRegs { + pub fn new(regs: &RegisterSet) -> Self { + Self { input: regs.clone(), global: regs.clone(), } diff --git a/lib/codegen/src/regalloc/diversion.rs b/lib/codegen/src/regalloc/diversion.rs index b848a879c9..1a747fc7e6 100644 --- a/lib/codegen/src/regalloc/diversion.rs +++ b/lib/codegen/src/regalloc/diversion.rs @@ -32,9 +32,9 @@ pub struct Diversion { impl Diversion { /// Make a new diversion. - pub fn new(value: Value, from: ValueLoc, to: ValueLoc) -> Diversion { + pub fn new(value: Value, from: ValueLoc, to: ValueLoc) -> Self { debug_assert!(from.is_assigned() && to.is_assigned()); - Diversion { value, from, to } + Self { value, from, to } } } diff --git a/lib/codegen/src/regalloc/liveness.rs b/lib/codegen/src/regalloc/liveness.rs index 1b82d98bf8..5d11d255a6 100644 --- a/lib/codegen/src/regalloc/liveness.rs +++ b/lib/codegen/src/regalloc/liveness.rs @@ -179,7 +179,7 @@ use entity::SparseMap; use flowgraph::ControlFlowGraph; use ir::dfg::ValueDef; use ir::{Ebb, Function, Inst, Layout, ProgramPoint, Value}; -use isa::{EncInfo, TargetIsa}; +use isa::{EncInfo, TargetIsa, OperandConstraint}; use regalloc::affinity::Affinity; use regalloc::liverange::{LiveRange, LiveRangeContext, LiveRangeForest}; use std::mem; @@ -413,11 +413,10 @@ impl Liveness { // Iterator of constraints, one per value operand. let encoding = func.encodings[inst]; - let mut operand_constraints = enc_info + let operand_constraint_slice: &[OperandConstraint] = enc_info .operand_constraints(encoding) - .map(|c| c.ins) - .unwrap_or(&[]) - .iter(); + .map_or(&[], |c| c.ins); + let mut operand_constraints = operand_constraint_slice.iter(); for &arg in func.dfg.inst_args(inst) { // Get the live range, create it as a dead range if necessary. diff --git a/lib/codegen/src/regalloc/liverange.rs b/lib/codegen/src/regalloc/liverange.rs index 18118e706d..d9ccc4fac5 100644 --- a/lib/codegen/src/regalloc/liverange.rs +++ b/lib/codegen/src/regalloc/liverange.rs @@ -217,8 +217,8 @@ impl GenLiveRange { /// Create a new live range for `value` defined at `def`. /// /// The live range will be created as dead, but it can be extended with `extend_in_ebb()`. - pub fn new(value: Value, def: ProgramPoint, affinity: Affinity) -> GenLiveRange { - GenLiveRange { + pub fn new(value: Value, def: ProgramPoint, affinity: Affinity) -> Self { + Self { value, affinity, def_begin: def, diff --git a/lib/codegen/src/regalloc/pressure.rs b/lib/codegen/src/regalloc/pressure.rs index 9f7d64365d..2c86050c59 100644 --- a/lib/codegen/src/regalloc/pressure.rs +++ b/lib/codegen/src/regalloc/pressure.rs @@ -81,8 +81,8 @@ pub struct Pressure { impl Pressure { /// Create a new register pressure tracker. - pub fn new(reginfo: &RegInfo, usable: &RegisterSet) -> Pressure { - let mut p = Pressure { + pub fn new(reginfo: &RegInfo, usable: &RegisterSet) -> Self { + let mut p = Self { aliased: 0, toprc: Default::default(), }; diff --git a/lib/codegen/src/regalloc/register_set.rs b/lib/codegen/src/regalloc/register_set.rs index 65414dbd9b..59f312150e 100644 --- a/lib/codegen/src/regalloc/register_set.rs +++ b/lib/codegen/src/regalloc/register_set.rs @@ -103,7 +103,7 @@ impl RegisterSet { /// of `other`. /// /// This assumes that unused bits are 1. - pub fn interferes_with(&self, other: &RegisterSet) -> bool { + pub fn interferes_with(&self, other: &Self) -> bool { self.avail.iter().zip(&other.avail).any( |(&x, &y)| (x | y) != !0, ) @@ -111,7 +111,7 @@ impl RegisterSet { /// Intersect this set of registers with `other`. This has the effect of removing any register /// units from this set that are not in `other`. - pub fn intersect(&mut self, other: &RegisterSet) { + pub fn intersect(&mut self, other: &Self) { for (x, &y) in self.avail.iter_mut().zip(&other.avail) { *x &= y; } diff --git a/lib/codegen/src/regalloc/reload.rs b/lib/codegen/src/regalloc/reload.rs index df527fdbca..9397d1a606 100644 --- a/lib/codegen/src/regalloc/reload.rs +++ b/lib/codegen/src/regalloc/reload.rs @@ -219,7 +219,7 @@ impl<'a> Context<'a> { self.reloads.insert(ReloadedValue { stack: cand.value, - reg: reg, + reg, }); cand.value = reg; diff --git a/lib/codegen/src/regalloc/solver.rs b/lib/codegen/src/regalloc/solver.rs index 218f0a2600..f3c14c9d81 100644 --- a/lib/codegen/src/regalloc/solver.rs +++ b/lib/codegen/src/regalloc/solver.rs @@ -151,8 +151,8 @@ pub struct Variable { } impl Variable { - fn new_live(value: Value, constraint: RegClass, from: RegUnit, is_output: bool) -> Variable { - Variable { + fn new_live(value: Value, constraint: RegClass, from: RegUnit, is_output: bool) -> Self { + Self { value, constraint, from: Some(from), @@ -164,8 +164,8 @@ impl Variable { } } - fn new_def(value: Value, constraint: RegClass, is_global: bool) -> Variable { - Variable { + fn new_def(value: Value, constraint: RegClass, is_global: bool) -> Self { + Self { value, constraint, from: None, @@ -280,7 +280,7 @@ pub enum Move { impl Move { /// Create a register move from an assignment, but not for identity assignments. - fn with_assignment(a: &Assignment) -> Option { + fn with_assignment(a: &Assignment) -> Option { if a.from != a.to { Some(Move::Reg { value: a.value, diff --git a/lib/codegen/src/regalloc/spilling.rs b/lib/codegen/src/regalloc/spilling.rs index 5a695eeb8b..34eb48a031 100644 --- a/lib/codegen/src/regalloc/spilling.rs +++ b/lib/codegen/src/regalloc/spilling.rs @@ -363,7 +363,7 @@ impl<'a> Context<'a> { self.cur.isa.regclass_for_abi_type(abi.value_type).into(), true, ), - Affinity::None => panic!("Missing affinity for {}", arg), + Affinity::Unassigned => panic!("Missing affinity for {}", arg), }; let mut reguse = RegUse::new(arg, fixed_args + idx, rci); reguse.fixed = true; @@ -393,10 +393,9 @@ impl<'a> Context<'a> { } else if ru.fixed { // This is a fixed register use which doesn't necessarily require a copy. // Make a copy only if this is not the first use of the value. - self.reg_uses - .get(i.wrapping_sub(1)) - .map(|ru2| ru2.value == ru.value) - .unwrap_or(false) + self.reg_uses.get(i.wrapping_sub(1)).map_or(false, |ru2| { + ru2.value == ru.value + }) } else { false }; @@ -567,8 +566,8 @@ struct RegUse { } impl RegUse { - fn new(value: Value, idx: usize, rci: RegClassIndex) -> RegUse { - RegUse { + fn new(value: Value, idx: usize, rci: RegClassIndex) -> Self { + Self { value, opidx: idx as u16, rci, diff --git a/lib/codegen/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs index 215e5efac2..58c7c380de 100644 --- a/lib/codegen/src/regalloc/virtregs.rs +++ b/lib/codegen/src/regalloc/virtregs.rs @@ -101,10 +101,9 @@ impl VirtRegs { where 'a: 'b, { - self.get(*value).map(|vr| self.values(vr)).unwrap_or_else( - || { - ref_slice(value) - }, + self.get(*value).map_or_else( + || ref_slice(value), + |vr| self.values(vr), ) } @@ -257,7 +256,7 @@ enum UFEntry { /// A singleton set is the same as a set with rank 0. It contains only the leader value. impl UFEntry { /// Decode a table entry. - fn decode(x: i32) -> UFEntry { + fn decode(x: i32) -> Self { if x < 0 { UFEntry::Link(Value::new((!x) as usize)) } else { diff --git a/lib/codegen/src/result.rs b/lib/codegen/src/result.rs index 04e248dcbb..a762b48e1c 100644 --- a/lib/codegen/src/result.rs +++ b/lib/codegen/src/result.rs @@ -70,7 +70,7 @@ impl StdError for CtonError { } impl From for CtonError { - fn from(e: verifier::Error) -> CtonError { + fn from(e: verifier::Error) -> Self { CtonError::Verifier(e) } } diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs index 6db7ef402e..f4edfc73cf 100644 --- a/lib/codegen/src/settings.rs +++ b/lib/codegen/src/settings.rs @@ -50,8 +50,8 @@ pub struct Builder { impl Builder { /// Create a new builder with defaults and names from the given template. - pub fn new(tmpl: &'static detail::Template) -> Builder { - Builder { + pub fn new(tmpl: &'static detail::Template) -> Self { + Self { template: tmpl, bytes: tmpl.defaults.into(), } diff --git a/lib/codegen/src/verifier/flags.rs b/lib/codegen/src/verifier/flags.rs index 286c169517..582ae525b8 100644 --- a/lib/codegen/src/verifier/flags.rs +++ b/lib/codegen/src/verifier/flags.rs @@ -102,8 +102,7 @@ impl<'a> FlagsVerifier<'a> { if self.encinfo .as_ref() .and_then(|ei| ei.operand_constraints(self.func.encodings[inst])) - .map(|c| c.clobbers_flags) - .unwrap_or(false) && live_val.is_some() + .map_or(false, |c| c.clobbers_flags) && live_val.is_some() { return err!(inst, "encoding clobbers live CPU flags in {}", live); } diff --git a/lib/codegen/src/verifier/liveness.rs b/lib/codegen/src/verifier/liveness.rs index cb57de9a4f..eeb041c88a 100644 --- a/lib/codegen/src/verifier/liveness.rs +++ b/lib/codegen/src/verifier/liveness.rs @@ -78,7 +78,7 @@ impl<'a> LivenessVerifier<'a> { if encoding.is_legal() { // A legal instruction is not allowed to define ghost values. - if lr.affinity.is_none() { + if lr.affinity.is_unassigned() { return err!( inst, "{} is a ghost value defined by a real [{}] instruction", @@ -86,7 +86,7 @@ impl<'a> LivenessVerifier<'a> { self.isa.encoding_info().display(encoding) ); } - } else if !lr.affinity.is_none() { + } else if !lr.affinity.is_unassigned() { // A non-encoded instruction can only define ghost values. return err!( inst, @@ -108,7 +108,7 @@ impl<'a> LivenessVerifier<'a> { } // A legal instruction is not allowed to depend on ghost values. - if encoding.is_legal() && lr.affinity.is_none() { + if encoding.is_legal() && lr.affinity.is_unassigned() { return err!( inst, "{} is a ghost value used by a real [{}] instruction", diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs index 692a435545..28f74699df 100644 --- a/lib/codegen/src/verifier/mod.rs +++ b/lib/codegen/src/verifier/mod.rs @@ -244,9 +244,9 @@ impl<'a> Verifier<'a> { let fixed_results = inst_data.opcode().constraints().fixed_results(); // var_results is 0 if we aren't a call instruction - let var_results = dfg.call_signature(inst) - .map(|sig| dfg.signatures[sig].returns.len()) - .unwrap_or(0); + let var_results = dfg.call_signature(inst).map_or(0, |sig| { + dfg.signatures[sig].returns.len() + }); let total_results = fixed_results + var_results; // All result values for multi-valued instructions are created diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs index 073ae22f11..a8f12d024f 100644 --- a/lib/codegen/src/write.rs +++ b/lib/codegen/src/write.rs @@ -22,7 +22,7 @@ pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) - let mut any = write_preamble(w, func, regs)?; for ebb in &func.layout { if any { - writeln!(w, "")?; + writeln!(w)?; } write_ebb(w, func, isa, ebb)?; any = true; @@ -258,7 +258,7 @@ fn write_instruction( } write_operands(w, &func.dfg, isa, inst)?; - writeln!(w, "") + writeln!(w) } /// Write the operands of `inst` to `w` with a prepended space. diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs index 00b6412c0d..7936495568 100644 --- a/lib/entity/src/lib.rs +++ b/lib/entity/src/lib.rs @@ -30,9 +30,20 @@ //! `Vec`. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature = "cargo-clippy", - allow(new_without_default, new_without_default_derive, redundant_field_names))] + allow(new_without_default, new_without_default_derive))] +#![cfg_attr(feature="cargo-clippy", warn( + float_arithmetic, + mut_mut, + nonminimal_bool, + option_map_unwrap_or, + option_map_unwrap_or_else, + print_stdout, + unicode_not_nfc, + use_self, + ))] // Turns on no_std and alloc features if std is not available. #![cfg_attr(not(feature = "std"), no_std)] @@ -91,7 +102,7 @@ macro_rules! entity_impl { impl $crate::__core::fmt::Display for $entity { fn fmt(&self, f: &mut $crate::__core::fmt::Formatter) -> $crate::__core::fmt::Result { - write!(f, "{}{}", $display_prefix, self.0) + write!(f, concat!($display_prefix, "{}"), self.0) } } diff --git a/lib/entity/src/map.rs b/lib/entity/src/map.rs index 8256ba9c90..8c09cd6486 100644 --- a/lib/entity/src/map.rs +++ b/lib/entity/src/map.rs @@ -49,7 +49,7 @@ where pub fn with_default(default: V) -> Self { Self { elems: Vec::new(), - default: default, + default, unused: PhantomData, } } diff --git a/lib/entity/src/packed_option.rs b/lib/entity/src/packed_option.rs index f92e821f72..869c57bb0d 100644 --- a/lib/entity/src/packed_option.rs +++ b/lib/entity/src/packed_option.rs @@ -28,7 +28,7 @@ impl PackedOption { /// Returns `true` if the packed option is a `Some` value. pub fn is_some(&self) -> bool { - !self.is_none() + self.0 != T::reserved_value() } /// Expand the packed option into a normal `Option`. @@ -62,14 +62,14 @@ impl PackedOption { impl Default for PackedOption { /// Create a default packed option representing `None`. - fn default() -> PackedOption { + fn default() -> Self { PackedOption(T::reserved_value()) } } impl From for PackedOption { /// Convert `t` into a packed `Some(x)`. - fn from(t: T) -> PackedOption { + fn from(t: T) -> Self { debug_assert!( t != T::reserved_value(), "Can't make a PackedOption from the reserved value." @@ -80,7 +80,7 @@ impl From for PackedOption { impl From> for PackedOption { /// Convert an option into its packed equivalent. - fn from(opt: Option) -> PackedOption { + fn from(opt: Option) -> Self { match opt { None => Self::default(), Some(t) => t.into(), diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index e33fe5e420..d0d4898ab4 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -270,7 +270,7 @@ impl<'a> RelocSink for FaerieRelocSink<'a> { &self.namespace.get_data_decl(name).name }; let addend_i32 = addend as i32; - debug_assert!(addend_i32 as i64 == addend); + debug_assert!(i64::from(addend_i32) == addend); let raw_reloc = container::raw_relocation(reloc, self.format); self.artifact .link_with( diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs index dc1dd5a0c3..3cf007357c 100644 --- a/lib/faerie/src/lib.rs +++ b/lib/faerie/src/lib.rs @@ -3,6 +3,20 @@ //! Users of this module should not have to depend on faerie directly. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![warn(unused_import_braces, unstable_features)] +#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", + allow(new_without_default, new_without_default_derive))] +#![cfg_attr(feature="cargo-clippy", warn( + float_arithmetic, + mut_mut, + nonminimal_bool, + option_map_unwrap_or, + option_map_unwrap_or_else, + print_stdout, + unicode_not_nfc, + use_self, + ))] extern crate cretonne_codegen; extern crate cretonne_module; diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs index df7be54fa6..30a2edc68c 100644 --- a/lib/filetests/src/lib.rs +++ b/lib/filetests/src/lib.rs @@ -3,10 +3,20 @@ //! This crate contains the main test driver as well as implementations of the //! available filetest commands. +#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature="cargo-clippy", allow( type_complexity, // Rustfmt 0.9.0 is at odds with this lint: block_in_if_condition_stmt))] +#![cfg_attr(feature="cargo-clippy", warn( + mut_mut, + nonminimal_bool, + option_map_unwrap_or, + option_map_unwrap_or_else, + unicode_not_nfc, + use_self, + ))] #[macro_use(dbg)] extern crate cretonne_codegen; diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs index 6d80afffad..5389cfdb95 100644 --- a/lib/filetests/src/runone.rs +++ b/lib/filetests/src/runone.rs @@ -122,7 +122,7 @@ fn run_one_test<'a>( ) -> Result<()> { let (test, flags, isa) = tuple; let name = format!("{}({})", test.name(), func.name); - dbg!("Test: {} {}", name, isa.map(TargetIsa::name).unwrap_or("-")); + dbg!("Test: {} {}", name, isa.map_or("-", TargetIsa::name)); context.flags = flags; context.isa = isa; diff --git a/lib/filetests/src/test_domtree.rs b/lib/filetests/src/test_domtree.rs index 9803a67d4e..8bf4a36cfe 100644 --- a/lib/filetests/src/test_domtree.rs +++ b/lib/filetests/src/test_domtree.rs @@ -125,7 +125,7 @@ fn filecheck_text(func: &Function, domtree: &DominatorTree) -> result::Result result::Result FunctionBuilder<'a, Variable> { debug_assert!(func_ctx.is_empty()); FunctionBuilder { - func: func, + func, srcloc: Default::default(), - func_ctx: func_ctx, + func_ctx, position: Position::default(), } } diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 8330a062ad..77bb7ff6e6 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -128,7 +128,18 @@ //! ``` #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] -#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, redundant_field_names))] +#![warn(unused_import_braces, unstable_features)] +#![cfg_attr(feature = "cargo-clippy", allow(new_without_default))] +#![cfg_attr(feature="cargo-clippy", warn( + float_arithmetic, + mut_mut, + nonminimal_bool, + option_map_unwrap_or, + option_map_unwrap_or_else, + print_stdout, + unicode_not_nfc, + use_self, + ))] extern crate cretonne_codegen; diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 22daff51d2..1f95316b3c 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -361,7 +361,7 @@ where let block = self.blocks.push(BlockData::EbbHeader(EbbHeaderBlockData { predecessors: Vec::new(), sealed: false, - ebb: ebb, + ebb, undef_variables: Vec::new(), })); self.ebb_headers[ebb] = block.into(); diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs index ebc44420cc..3835f2f0ce 100644 --- a/lib/module/src/lib.rs +++ b/lib/module/src/lib.rs @@ -1,6 +1,20 @@ //! Top-level lib.rs for `cretonne_module`. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![warn(unused_import_braces, unstable_features)] +#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", + allow(new_without_default, new_without_default_derive))] +#![cfg_attr(feature="cargo-clippy", warn( + float_arithmetic, + mut_mut, + nonminimal_bool, + option_map_unwrap_or, + option_map_unwrap_or_else, + print_stdout, + unicode_not_nfc, + use_self, + ))] extern crate cretonne_codegen; #[macro_use] diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index 69e9d7432f..79ef29eba3 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -36,7 +36,7 @@ pub enum Linkage { } impl Linkage { - fn merge(a: Linkage, b: Linkage) -> Linkage { + fn merge(a: Self, b: Self) -> Self { match a { Linkage::Export => Linkage::Export, Linkage::Preemptible => { diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index c2504b5c38..847ec55af1 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -2,6 +2,20 @@ //! Cretonne to generate code to run on the same machine. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![warn(unused_import_braces, unstable_features)] +#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", + allow(new_without_default, new_without_default_derive))] +#![cfg_attr(feature="cargo-clippy", warn( + float_arithmetic, + mut_mut, + nonminimal_bool, + option_map_unwrap_or, + option_map_unwrap_or_else, + print_stdout, + unicode_not_nfc, + use_self, + ))] extern crate cretonne_codegen; diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index 5a51d4199d..3527bbf587 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -4,6 +4,20 @@ //! testing Cretonne, but is not essential for a JIT compiler. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![warn(unused_import_braces, unstable_features)] +#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", + allow(new_without_default, new_without_default_derive))] +#![cfg_attr(feature="cargo-clippy", warn( + float_arithmetic, + mut_mut, + nonminimal_bool, + option_map_unwrap_or, + option_map_unwrap_or_else, + print_stdout, + unicode_not_nfc, + use_self, + ))] extern crate cretonne_codegen; diff --git a/lib/reader/src/testcommand.rs b/lib/reader/src/testcommand.rs index 2b848a9348..32220e006e 100644 --- a/lib/reader/src/testcommand.rs +++ b/lib/reader/src/testcommand.rs @@ -54,7 +54,7 @@ impl<'a> Display for TestCommand<'a> { for opt in &self.options { write!(f, " {}", opt)?; } - writeln!(f, "") + writeln!(f) } } diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs index 0157466cac..1917a3db5e 100644 --- a/lib/simplejit/src/lib.rs +++ b/lib/simplejit/src/lib.rs @@ -1,6 +1,20 @@ //! Top-level lib.rs for `cretonne_simplejit`. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![warn(unused_import_braces, unstable_features)] +#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", + allow(new_without_default, new_without_default_derive))] +#![cfg_attr(feature="cargo-clippy", warn( + float_arithmetic, + mut_mut, + nonminimal_bool, + option_map_unwrap_or, + option_map_unwrap_or_else, + print_stdout, + unicode_not_nfc, + use_self, + ))] extern crate cretonne_codegen; extern crate cretonne_module; diff --git a/lib/simplejit/src/memory.rs b/lib/simplejit/src/memory.rs index 235a3976d7..1ea654e924 100644 --- a/lib/simplejit/src/memory.rs +++ b/lib/simplejit/src/memory.rs @@ -25,7 +25,7 @@ impl PtrLen { let err = libc::posix_memalign(&mut ptr, page_size, alloc_size); if err == 0 { Ok(Self { - ptr: mem::transmute(ptr), + ptr: ptr as *mut u8, len: alloc_size, }) } else { diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs index a70a97e86f..179a33224a 100644 --- a/lib/umbrella/src/lib.rs +++ b/lib/umbrella/src/lib.rs @@ -1,7 +1,20 @@ //! Cretonne umbrella crate, providing a convenient one-line dependency. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", + allow(new_without_default, new_without_default_derive))] +#![cfg_attr(feature="cargo-clippy", warn( + float_arithmetic, + mut_mut, + nonminimal_bool, + option_map_unwrap_or, + option_map_unwrap_or_else, + print_stdout, + unicode_not_nfc, + use_self, + ))] pub extern crate cretonne_codegen; pub extern crate cretonne_frontend; diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 2b48cd50c0..aea622cc34 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -949,7 +949,7 @@ fn get_heap_addr( heap: ir::Heap, addr32: ir::Value, offset: u32, - addr_ty: ir::Type, + addr_ty: Type, builder: &mut FunctionBuilder, ) -> (ir::Value, i32) { use std::cmp::min; @@ -985,7 +985,7 @@ fn get_heap_addr( fn translate_load( offset: u32, opcode: ir::Opcode, - result_ty: ir::Type, + result_ty: Type, builder: &mut FunctionBuilder, state: &mut TranslationState, environ: &mut FE, diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index c2eea8219f..e716927745 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -226,7 +226,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ args.push(vmctx, &mut pos.func.dfg.value_lists); pos.ins() - .CallIndirect(ir::Opcode::CallIndirect, ir::types::VOID, sig_ref, args) + .CallIndirect(ir::Opcode::CallIndirect, VOID, sig_ref, args) .0 } @@ -248,9 +248,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ args.extend(call_args.iter().cloned(), &mut pos.func.dfg.value_lists); args.push(vmctx, &mut pos.func.dfg.value_lists); - pos.ins() - .Call(ir::Opcode::Call, ir::types::VOID, callee, args) - .0 + pos.ins().Call(ir::Opcode::Call, VOID, callee, args).0 } fn translate_grow_memory( diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 1f0f063e37..5a2e24948b 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -10,8 +10,20 @@ //! The main function of this module is [`translate_module`](fn.translate_module.html). #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] +#![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] -#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, redundant_field_names))] +#![cfg_attr(feature = "cargo-clippy", + allow(new_without_default, new_without_default_derive))] +#![cfg_attr(feature="cargo-clippy", warn( + float_arithmetic, + mut_mut, + nonminimal_bool, + option_map_unwrap_or, + option_map_unwrap_or_else, + print_stdout, + unicode_not_nfc, + use_self, + ))] #[macro_use(dbg)] extern crate cretonne_codegen; diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index b744c505e4..c243348d5a 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -241,8 +241,8 @@ pub fn parse_global_section( } let global = Global { ty: type_to_type(&content_type).unwrap(), - mutability: mutability, - initializer: initializer, + mutability, + initializer, }; environ.declare_global(global); match *parser.read() { From d3e76b8a51add5e013f8cea0ca736db1a47a5628 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 22:12:41 -0700 Subject: [PATCH 1710/3084] Add more tests for non-PIC calls on x86-64. --- cranelift/filetests/isa/x86/binary64.cton | 2 +- cranelift/filetests/isa/x86/legalize-call.cton | 15 +++++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) create mode 100644 cranelift/filetests/isa/x86/legalize-call.cton diff --git a/cranelift/filetests/isa/x86/binary64.cton b/cranelift/filetests/isa/x86/binary64.cton index faf4fe99ac..2a945ee17c 100644 --- a/cranelift/filetests/isa/x86/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.cton @@ -477,7 +477,7 @@ ebb0: ; Colocated functions. ; asm: call bar - ; call fn1() ; bin: e8 PCRel4(%bar-4) 00000000 + call fn1() ; bin: e8 PCRel4(%bar-4) 00000000 ; asm: lea 0x0(%rip), %rcx [-,%rcx] v400 = func_addr.i64 fn1 ; bin: 48 8d 0d PCRel4(%bar-4) 00000000 diff --git a/cranelift/filetests/isa/x86/legalize-call.cton b/cranelift/filetests/isa/x86/legalize-call.cton new file mode 100644 index 0000000000..adcb6a6ad1 --- /dev/null +++ b/cranelift/filetests/isa/x86/legalize-call.cton @@ -0,0 +1,15 @@ +; Test legalization of a non-colocated call in 64-bit non-PIC mode. +test legalizer +set is_64bit +set is_compressed +isa x86 haswell + +function %call() { + fn0 = %foo() +ebb0: + call fn0() + return +} + +; check: v0 = func_addr.i64 fn0 +; nextln: call_indirect sig0, v0() From 1ba468b2304bc83a799c3f3876a567d10b377938 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 22:18:30 -0700 Subject: [PATCH 1711/3084] Bump version to 0.5.1 --- cranelift/Cargo.toml | 22 +++++++++++----------- cranelift/publish-all.sh | 2 +- lib/codegen/Cargo.toml | 4 ++-- lib/entity/Cargo.toml | 2 +- lib/faerie/Cargo.toml | 6 +++--- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/module/Cargo.toml | 6 +++--- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/simplejit/Cargo.toml | 8 ++++---- lib/umbrella/Cargo.toml | 6 +++--- lib/wasm/Cargo.toml | 6 +++--- 13 files changed, 40 insertions(+), 40 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 9c4c1608ff..021440c024 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.5.0" +version = "0.5.1" description = "Binaries for testing the Cretonne libraries" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,16 +13,16 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne-codegen = { path = "lib/codegen", version = "0.5.0" } -cretonne-reader = { path = "lib/reader", version = "0.5.0" } -cretonne-frontend = { path = "lib/frontend", version = "0.5.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.5.0" } -cretonne-native = { path = "lib/native", version = "0.5.0" } -cretonne-filetests = { path = "lib/filetests", version = "0.5.0" } -cretonne-module = { path = "lib/module", version = "0.5.0" } -cretonne-faerie = { path = "lib/faerie", version = "0.5.0" } -cretonne-simplejit = { path = "lib/simplejit", version = "0.5.0" } -cretonne = { path = "lib/umbrella", version = "0.5.0" } +cretonne-codegen = { path = "lib/codegen", version = "0.5.1" } +cretonne-reader = { path = "lib/reader", version = "0.5.1" } +cretonne-frontend = { path = "lib/frontend", version = "0.5.1" } +cretonne-wasm = { path = "lib/wasm", version = "0.5.1" } +cretonne-native = { path = "lib/native", version = "0.5.1" } +cretonne-filetests = { path = "lib/filetests", version = "0.5.1" } +cretonne-module = { path = "lib/module", version = "0.5.1" } +cretonne-faerie = { path = "lib/faerie", version = "0.5.1" } +cretonne-simplejit = { path = "lib/simplejit", version = "0.5.1" } +cretonne = { path = "lib/umbrella", version = "0.5.1" } filecheck = "0.2.1" docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index ba539f2f26..06aaab658a 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ cd $(dirname "$0") topdir="$(pwd)" # All the cretonne-* crates have the same version number -version="0.5.0" +version="0.5.1" # Update all of the Cargo.toml files. # diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 4aaeccee7b..18d0b2aedc 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-codegen" -version = "0.5.0" +version = "0.5.1" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"] build = "build.rs" [dependencies] -cretonne-entity = { path = "../entity", version = "0.5.0" } +cretonne-entity = { path = "../entity", version = "0.5.1" } # It is a goal of the cretonne-codegen crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index c7c6b0dde0..c5f1b88c03 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-entity" -version = "0.5.0" +version = "0.5.1" description = "Data structures using entity references as mapping keys" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index a0e1effeea..77477b8f98 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-faerie" -version = "0.5.0" +version = "0.5.1" authors = ["The Cretonne Project Developers"] description = "Emit Cretonne output to native object files with Faerie" repository = "https://github.com/Cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.0" } -cretonne-module = { path = "../module", version = "0.5.0" } +cretonne-codegen = { path = "../codegen", version = "0.5.1" } +cretonne-module = { path = "../module", version = "0.5.1" } faerie = "0.1.0" goblin = "0.0.14" failure = "0.1.1" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 2ab50d0ebd..4e80bfc17e 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.5.0" +version = "0.5.1" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "http://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" publish = false [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.0" } -cretonne-reader = { path = "../reader", version = "0.5.0" } +cretonne-codegen = { path = "../codegen", version = "0.5.1" } +cretonne-reader = { path = "../reader", version = "0.5.1" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 03752d5689..95d4a18786 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.5.0" +version = "0.5.1" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.0" } +cretonne-codegen = { path = "../codegen", version = "0.5.1" } [badges] maintenance = { status = "experimental" } diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index f43fde30a7..42dc6ee916 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-module" -version = "0.5.0" +version = "0.5.1" authors = ["The Cretonne Project Developers"] description = "Support for linking functions and data with Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.0" } -cretonne-entity = { path = "../entity", version = "0.5.0" } +cretonne-codegen = { path = "../codegen", version = "0.5.1" } +cretonne-entity = { path = "../entity", version = "0.5.1" } [badges] maintenance = { status = "experimental" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 265629044e..2909fa8307 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.5.0" +version = "0.5.1" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -8,7 +8,7 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.0" } +cretonne-codegen = { path = "../codegen", version = "0.5.1" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.0.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 0b984515d9..c97cc3a113 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.5.0" +version = "0.5.1" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.0" } +cretonne-codegen = { path = "../codegen", version = "0.5.1" } [badges] maintenance = { status = "experimental" } diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index 699212afc0..0fdc16da32 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-simplejit" -version = "0.5.0" +version = "0.5.1" authors = ["The Cretonne Project Developers"] description = "A simple JIT library backed by Cretonne" repository = "https://github.com/Cretonne/cretonne" @@ -9,9 +9,9 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.0" } -cretonne-module = { path = "../module", version = "0.5.0" } -cretonne-native = { path = "../native", version = "0.5.0" } +cretonne-codegen = { path = "../codegen", version = "0.5.1" } +cretonne-module = { path = "../module", version = "0.5.1" } +cretonne-native = { path = "../native", version = "0.5.1" } region = "0.2.0" libc = "0.2.40" errno = "0.2.3" diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml index fc89536394..f7bdc002cd 100644 --- a/lib/umbrella/Cargo.toml +++ b/lib/umbrella/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.5.0" +version = "0.5.1" description = "Umbrella for commonly-used cretonne crates" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -10,8 +10,8 @@ readme = "README.md" keywords = ["compile", "compiler", "jit"] [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.0" } -cretonne-frontend = { path = "../frontend", version = "0.5.0" } +cretonne-codegen = { path = "../codegen", version = "0.5.1" } +cretonne-frontend = { path = "../frontend", version = "0.5.1" } [badges] maintenance = { status = "experimental" } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 49644f7746..a0809a5360 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.5.0" +version = "0.5.1" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/cretonne/cretonne" @@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"] [dependencies] wasmparser = "0.15.1" -cretonne-codegen = { path = "../codegen", version = "0.5.0" } -cretonne-frontend = { path = "../frontend", version = "0.5.0" } +cretonne-codegen = { path = "../codegen", version = "0.5.1" } +cretonne-frontend = { path = "../frontend", version = "0.5.1" } [dev-dependencies] tempdir = "0.3.5" From 80da1a1e9fc054d1d23d14a069244244d649fdea Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 17 Apr 2018 23:19:10 -0700 Subject: [PATCH 1712/3084] Add crate descriptions for cretonne-{module, faerie, simplejit}. --- cranelift/docs/index.rst | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst index c2e5b6d5ef..60b153e2a1 100644 --- a/cranelift/docs/index.rst +++ b/cranelift/docs/index.rst @@ -34,6 +34,19 @@ Rust Crate Documentation This crate translates from Cretonne IR's text format into Cretonne IR in in-memory data structures. +`cretonne-module `_ + This crate manages compiling multiple functions and data objects + together. + +`cretonne-faerie `_ + This crate provides a faerie-based backend for `cretonne-module`, which + emits native object files using the + `faerie `_ library. + +`cretonne-simplejit `_ + This crate provides a simple JIT backend for `cretonne-module`, which + emits code and data into memory. + Indices and tables ================== From d7e13284b296e8e6c7310eba6d54da3cbf8ce757 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 18 Apr 2018 06:35:47 -0700 Subject: [PATCH 1713/3084] Mark `emit_to_memory` as unsafe, and introduce a safe `emit`. (#281) * Mark emit_to_memory as unsafe, and provide a safe compile_and_emit. Mark `Context::emit_to_memory` and `MemoryCodeSink::new` as unsafe, as `MemoryCodeSink` does not perform bounds checking when writing to memory. Add a `Context::compile_and_emit` function which provides a convenient interface for doing `compile` and `emit_to_memory` in one step, and which can also provide a safe interface, since it allocates memory of the needed size itself. * Mention that `MemoryCodeSink` can't guarantee that the pointer is valid. --- cranelift/src/compile.rs | 17 ++++++------ lib/codegen/src/binemit/memorysink.rs | 5 +++- lib/codegen/src/context.rs | 37 +++++++++++++++++++++++++-- lib/faerie/src/backend.rs | 14 +++++----- lib/simplejit/src/backend.rs | 2 +- 5 files changed, 56 insertions(+), 19 deletions(-) diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index ae8cd1e30c..4b9f77494d 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -96,19 +96,18 @@ fn handle_module( for (func, _) in test_file.functions { let mut context = Context::new(); context.func = func; - let size = context.compile(isa).map_err(|err| { - pretty_error(&context.func, Some(isa), err) - })?; - if flag_print { - println!("{}", context.func.display(isa)); - } - // Encode the result as machine code. + // Compile and encode the result to machine code. let mut mem = Vec::new(); let mut relocs = PrintRelocs { flag_print }; let mut traps = PrintTraps { flag_print }; - mem.resize(size as usize, 0); - context.emit_to_memory(mem.as_mut_ptr(), &mut relocs, &mut traps, &*isa); + context + .compile_and_emit(isa, &mut mem, &mut relocs, &mut traps) + .map_err(|err| pretty_error(&context.func, Some(isa), err))?; + + if flag_print { + println!("{}", context.func.display(isa)); + } if flag_print { print!(".byte "); diff --git a/lib/codegen/src/binemit/memorysink.rs b/lib/codegen/src/binemit/memorysink.rs index d25513d27a..6871dc0942 100644 --- a/lib/codegen/src/binemit/memorysink.rs +++ b/lib/codegen/src/binemit/memorysink.rs @@ -38,7 +38,10 @@ pub struct MemoryCodeSink<'a> { impl<'a> MemoryCodeSink<'a> { /// Create a new memory code sink that writes a function to the memory pointed to by `data`. - pub fn new<'sink>( + /// + /// This function is unsafe since `MemoryCodeSink` does not perform bounds checking on the + /// memory buffer, and it can't guarantee that the `data` pointer is valid. + pub unsafe fn new<'sink>( data: *mut u8, relocs: &'sink mut RelocSink, traps: &'sink mut TrapSink, diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index 7b3e0253e1..95e78ff128 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -78,6 +78,36 @@ impl Context { self.loop_analysis.clear(); } + /// Compile the function, and emit machine code into a `Vec`. + /// + /// Run the function through all the passes necessary to generate code for the target ISA + /// represented by `isa`, as well as the final step of emitting machine code into a + /// `Vec`. The machine code is not relocated. Instead, any relocations are emitted + /// into `relocs`. + /// + /// This function calls `compile` and `emit_to_memory`, taking care to resize `mem` as + /// needed, so it provides a safe interface. + pub fn compile_and_emit( + &mut self, + isa: &TargetIsa, + mem: &mut Vec, + relocs: &mut RelocSink, + traps: &mut TrapSink, + ) -> CtonResult { + let code_size = self.compile(isa)?; + let old_len = mem.len(); + mem.resize(old_len + code_size as usize, 0); + unsafe { + self.emit_to_memory( + isa, + mem.as_mut_ptr().offset(old_len as isize), + relocs, + traps, + ) + }; + Ok(()) + } + /// Compile the function. /// /// Run the function through all the passes necessary to generate code for the target ISA @@ -119,12 +149,15 @@ impl Context { /// code is returned by `compile` above. /// /// The machine code is not relocated. Instead, any relocations are emitted into `relocs`. - pub fn emit_to_memory( + /// + /// This function is unsafe since it does not perform bounds checking on the memory buffer, + /// and it can't guarantee that the `mem` pointer is valid. + pub unsafe fn emit_to_memory( &self, + isa: &TargetIsa, mem: *mut u8, relocs: &mut RelocSink, traps: &mut TrapSink, - isa: &TargetIsa, ) { let _tt = timing::binemit(); isa.emit_function(&self.func, &mut MemoryCodeSink::new(mem, relocs, traps)); diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index d0d4898ab4..37e94e29e0 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -107,12 +107,14 @@ impl Backend for FaerieBackend { // that traps. let mut trap_sink = NullTrapSink {}; - ctx.emit_to_memory( - code.as_mut_ptr(), - &mut reloc_sink, - &mut trap_sink, - &*self.isa, - ); + unsafe { + ctx.emit_to_memory( + &*self.isa, + code.as_mut_ptr(), + &mut reloc_sink, + &mut trap_sink, + ) + }; } self.artifact.define(name, code).expect( diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs index 5576c02e32..15e6892e19 100644 --- a/lib/simplejit/src/backend.rs +++ b/lib/simplejit/src/backend.rs @@ -103,7 +103,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { // Ignore traps for now. For now, frontends should just avoid generating code // that traps. let mut trap_sink = NullTrapSink {}; - ctx.emit_to_memory(ptr, &mut reloc_sink, &mut trap_sink, &*self.isa); + unsafe { ctx.emit_to_memory(&*self.isa, ptr, &mut reloc_sink, &mut trap_sink) }; Ok(Self::CompiledFunction { code: ptr, From fdf34e9d3eb85050bcfe42fc010f36485c0a3818 Mon Sep 17 00:00:00 2001 From: morenzg Date: Tue, 17 Apr 2018 21:07:22 -0400 Subject: [PATCH 1714/3084] Fix up nostd dependencies --- lib/codegen/Cargo.toml | 2 +- lib/wasm/Cargo.toml | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 63a1566007..d362e0ee4e 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"] build = "build.rs" [dependencies] -cretonne-entity = { path = "../entity", version = "0.5.0" } +cretonne-entity = { path = "../entity", version = "0.5.0", default-features = false } # It is a goal of the cretonne-codegen crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index dd053698f0..ddf76afaf5 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -9,7 +9,7 @@ readme = "README.md" keywords = ["webassembly", "wasm"] [dependencies] -wasmparser = "0.15.1" +wasmparser = { git = "https://github.com/yurydelendik/wasmparser.rs", default_features = false } cretonne-codegen = { path = "../codegen", version = "0.5.0", default_features = false } cretonne-frontend = { path = "../frontend", version = "0.5.0", default_features = false } @@ -22,8 +22,8 @@ tempdir = "0.3.5" [features] default = ["std"] -std = ["cretonne-codegen/std", "cretonne-frontend/std"] -core = ["hashmap_core", "cretonne-codegen/core", "cretonne-frontend/core"] +std = ["cretonne-codegen/std", "cretonne-frontend/std", "wasmparser/std"] +core = ["hashmap_core", "cretonne-codegen/core", "cretonne-frontend/core", "wasmparser/core"] [badges] maintenance = { status = "experimental" } From c791a4f8b18c0190907d39dd9593f7454ebf5911 Mon Sep 17 00:00:00 2001 From: morenzg Date: Wed, 18 Apr 2018 12:46:55 -0400 Subject: [PATCH 1715/3084] Update to wasmparser 0.16.0 --- lib/wasm/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index ddf76afaf5..b2ec94fa34 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -9,7 +9,7 @@ readme = "README.md" keywords = ["webassembly", "wasm"] [dependencies] -wasmparser = { git = "https://github.com/yurydelendik/wasmparser.rs", default_features = false } +wasmparser = { version = "0.16.0", default_features = false } cretonne-codegen = { path = "../codegen", version = "0.5.0", default_features = false } cretonne-frontend = { path = "../frontend", version = "0.5.0", default_features = false } From d122d16f79ed2bdc0eb7de0a4b3f7396ad36b543 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 18 Apr 2018 10:22:49 -0700 Subject: [PATCH 1716/3084] Style consistency: don't end error messages with exclamation points. --- lib/codegen/meta/cdsl/isa.py | 2 +- lib/codegen/meta/cdsl/typevar.py | 2 +- lib/codegen/src/verifier/mod.rs | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/codegen/meta/cdsl/isa.py b/lib/codegen/meta/cdsl/isa.py index 77b071636a..bcff050b1b 100644 --- a/lib/codegen/meta/cdsl/isa.py +++ b/lib/codegen/meta/cdsl/isa.py @@ -54,7 +54,7 @@ class TargetISA(object): self._predicates = dict() # type: Dict[PredKey, PredNode] assert InstructionGroup._current is None,\ - "InstructionGroup {} is still open!"\ + "InstructionGroup {} is still open"\ .format(InstructionGroup._current.name) def __str__(self): diff --git a/lib/codegen/meta/cdsl/typevar.py b/lib/codegen/meta/cdsl/typevar.py index 8cda58785e..d58381529f 100644 --- a/lib/codegen/meta/cdsl/typevar.py +++ b/lib/codegen/meta/cdsl/typevar.py @@ -231,7 +231,7 @@ class TypeSet(object): def __hash__(self): # type: () -> int h = hash(self.typeset_key()) - assert h == getattr(self, 'prev_hash', h), "TypeSet changed!" + assert h == getattr(self, 'prev_hash', h), "TypeSet changed" self.prev_hash = h return h diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs index 28f74699df..66baa5fa91 100644 --- a/lib/codegen/src/verifier/mod.rs +++ b/lib/codegen/src/verifier/mod.rs @@ -207,7 +207,7 @@ impl<'a> Verifier<'a> { ); } if is_last_inst && !is_terminator { - return err!(ebb, "block does not end in a terminator instruction!"); + return err!(ebb, "block does not end in a terminator instruction"); } // Instructions belong to the correct ebb. @@ -1163,7 +1163,7 @@ mod tests { macro_rules! assert_err_with_msg { ($e:expr, $msg:expr) => { match $e { - Ok(_) => panic!("Expected an error!"), + Ok(_) => panic!("Expected an error"), Err(Error { message, .. }) => { if !message.contains($msg) { panic!(format!( From eefa1de4dac10d157bce58df0b0265f0709ba59f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 18 Apr 2018 17:02:32 -0700 Subject: [PATCH 1717/3084] Rename some local variables for consistency with the rest of the codebase. --- lib/codegen/src/regalloc/affinity.rs | 4 ++-- lib/codegen/src/regalloc/liveness.rs | 21 ++++++++++----------- 2 files changed, 12 insertions(+), 13 deletions(-) diff --git a/lib/codegen/src/regalloc/affinity.rs b/lib/codegen/src/regalloc/affinity.rs index d514cf4dd4..a8fa423049 100644 --- a/lib/codegen/src/regalloc/affinity.rs +++ b/lib/codegen/src/regalloc/affinity.rs @@ -84,7 +84,7 @@ impl Affinity { /// /// Note that this does not guarantee that the register allocator will pick a register that /// satisfies the constraint. - pub fn merge(&mut self, constraint: &OperandConstraint, reg_info: &RegInfo) { + pub fn merge(&mut self, constraint: &OperandConstraint, reginfo: &RegInfo) { match *self { Affinity::Unassigned => *self = Self::new(constraint), Affinity::Reg(rc) => { @@ -95,7 +95,7 @@ impl Affinity { { // If the register classes don't overlap, `intersect` returns `Unassigned`, and // we just keep our previous affinity. - if let Some(subclass) = constraint.regclass.intersect_index(reg_info.rc(rc)) { + if let Some(subclass) = constraint.regclass.intersect_index(reginfo.rc(rc)) { // This constraint shrinks our preferred register class. *self = Affinity::Reg(subclass); } diff --git a/lib/codegen/src/regalloc/liveness.rs b/lib/codegen/src/regalloc/liveness.rs index 5d11d255a6..faf35f6a1f 100644 --- a/lib/codegen/src/regalloc/liveness.rs +++ b/lib/codegen/src/regalloc/liveness.rs @@ -197,7 +197,7 @@ fn get_or_create<'a>( value: Value, isa: &TargetIsa, func: &Function, - enc_info: &EncInfo, + encinfo: &EncInfo, ) -> &'a mut LiveRange { // It would be better to use `get_mut()` here, but that leads to borrow checker fighting // which can probably only be resolved by non-lexical lifetimes. @@ -211,7 +211,7 @@ fn get_or_create<'a>( def = inst.into(); // Initialize the affinity from the defining instruction's result constraints. // Don't do this for call return values which are always tied to a single register. - affinity = enc_info + affinity = encinfo .operand_constraints(func.encodings[inst]) .and_then(|rc| rc.outs.get(rnum)) .map(Affinity::new) @@ -385,8 +385,8 @@ impl Liveness { self.ranges.clear(); // Get ISA data structures used for computing live range affinities. - let enc_info = isa.encoding_info(); - let reg_info = isa.register_info(); + let encinfo = isa.encoding_info(); + let reginfo = isa.register_info(); // The liveness computation needs to visit all uses, but the order doesn't matter. // TODO: Perhaps this traversal of the function could be combined with a dead code @@ -397,7 +397,7 @@ impl Liveness { // TODO: If these parameters are really dead, we could remove them, except for the // entry block which must match the function signature. for &arg in func.dfg.ebb_params(ebb) { - get_or_create(&mut self.ranges, arg, isa, func, &enc_info); + get_or_create(&mut self.ranges, arg, isa, func, &encinfo); } for inst in func.layout.ebb_insts(ebb) { @@ -408,19 +408,18 @@ impl Liveness { // TODO: When we implement DCE, we can use the absence of a live range to indicate // an unused value. for &def in func.dfg.inst_results(inst) { - get_or_create(&mut self.ranges, def, isa, func, &enc_info); + get_or_create(&mut self.ranges, def, isa, func, &encinfo); } // Iterator of constraints, one per value operand. let encoding = func.encodings[inst]; - let operand_constraint_slice: &[OperandConstraint] = enc_info - .operand_constraints(encoding) - .map_or(&[], |c| c.ins); + let operand_constraint_slice: &[OperandConstraint] = + encinfo.operand_constraints(encoding).map_or(&[], |c| c.ins); let mut operand_constraints = operand_constraint_slice.iter(); for &arg in func.dfg.inst_args(inst) { // Get the live range, create it as a dead range if necessary. - let lr = get_or_create(&mut self.ranges, arg, isa, func, &enc_info); + let lr = get_or_create(&mut self.ranges, arg, isa, func, &encinfo); // Extend the live range to reach this use. extend_to_use( @@ -437,7 +436,7 @@ impl Liveness { // operands described by `operand_constraints`. Variable arguments are either // EBB arguments or call/return ABI arguments. if let Some(constraint) = operand_constraints.next() { - lr.affinity.merge(constraint, ®_info); + lr.affinity.merge(constraint, ®info); } } } From 947177732a0d3d30638f38d6baa96da6808d4380 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 18 Apr 2018 20:59:42 -0700 Subject: [PATCH 1718/3084] Update to filecheck 0.3.0. --- cranelift/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 021440c024..4961a6fef6 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -23,7 +23,7 @@ cretonne-module = { path = "lib/module", version = "0.5.1" } cretonne-faerie = { path = "lib/faerie", version = "0.5.1" } cretonne-simplejit = { path = "lib/simplejit", version = "0.5.1" } cretonne = { path = "lib/umbrella", version = "0.5.1" } -filecheck = "0.2.1" +filecheck = "0.3.0" docopt = "0.8.0" serde = "1.0.8" serde_derive = "1.0.8" From 07576d3ed07d558a72cdc7e85c88e3fab12a1986 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 18 Apr 2018 20:59:51 -0700 Subject: [PATCH 1719/3084] Make test-all.sh run unit tests in debug mode. It turns out that "cargo test --release" doesn't use `[profile.release]`; it uses `[profile.bench]` instead; see [here](https://doc.rust-lang.org/cargo/reference/manifest.html) for details. So the plan to run the tests in optimized mode but with debug-assertions enabled didn't actually work, and we had an actual failing unit test that was hidden because assertions were disabled. So, this makes test-all.sh just run the unit tests in debug mode, as is the norm for Rust packages, and fixes the buggy test. This also removes the `[profile.release]` override from the top-level Cargo.toml file too. We don't need it now that we're not running tests in release mode, and it had confused multiple people because it made Cretonne's in-tree builds different from how Cretonne is built when used as a dependency in other projects. --- cranelift/Cargo.toml | 8 -------- cranelift/test-all.sh | 12 +++++++----- lib/module/src/data_context.rs | 4 ++-- 3 files changed, 9 insertions(+), 15 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 4961a6fef6..543bbcf8a0 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -31,11 +31,3 @@ tempdir = "0.3.5" term = "0.5.1" [workspace] - -# Enable debug assertions and parallel compilation when building cretonne-tools -# since they are for testing and development mostly. This doesn't affect the -# flags used to build the cretonne-* crates when used as a dependency. -[profile.release] -opt-level = 2 -debug-assertions = true -codegen-units = 4 diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index ba3faf70b0..82ec347dac 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -41,15 +41,17 @@ if [ -n "$needcheck" ]; then touch $tsfile || echo no target directory fi +# Make sure the code builds in release mode. +banner "Rust release build" +cargo build --release + # Make sure the code builds in debug mode. banner "Rust debug build" cargo build -# Make sure the code builds in release mode, and run the unit tests. We run -# these in release mode for speed, but note that the top-level Cargo.toml file -# does enable debug assertions in release builds. -banner "Rust release build and unit tests" -cargo test --all --release +# Run the tests. We run these in debug mode so that assertions are enabled. +banner "Rust unit tests" +cargo test --all # Make sure the documentation builds. banner "Rust documentation: $topdir/target/doc/cretonne/index.html" diff --git a/lib/module/src/data_context.rs b/lib/module/src/data_context.rs index 510b549e90..51eac88b79 100644 --- a/lib/module/src/data_context.rs +++ b/lib/module/src/data_context.rs @@ -144,7 +144,7 @@ mod tests { fn basic_data_context() { let mut data_ctx = DataContext::new(); { - let description = data_ctx.description(); + let description = &data_ctx.description; assert_eq!(description.writable, Writability::Readonly); assert_eq!(description.init, Init::Uninitialized); assert!(description.function_decls.is_empty()); @@ -177,7 +177,7 @@ mod tests { data_ctx.clear(); { - let description = data_ctx.description(); + let description = &data_ctx.description; assert_eq!(description.writable, Writability::Readonly); assert_eq!(description.init, Init::Uninitialized); assert!(description.function_decls.is_empty()); From 5bc0e0e188dbb1f15018305f066dbe088d0f79ca Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Apr 2018 09:40:46 -0700 Subject: [PATCH 1720/3084] Add a comment advertising `NullTrapSink`. --- lib/codegen/src/binemit/memorysink.rs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/codegen/src/binemit/memorysink.rs b/lib/codegen/src/binemit/memorysink.rs index 6871dc0942..fa4aacc233 100644 --- a/lib/codegen/src/binemit/memorysink.rs +++ b/lib/codegen/src/binemit/memorysink.rs @@ -68,6 +68,9 @@ pub trait RelocSink { } /// A trait for receiving trap codes and offsets. +/// +/// If you don't need information about possible traps, you can use the +/// [`NullTrapSink`](binemit/trait.TrapSink.html) implementation. pub trait TrapSink { /// Add trap information for a specific offset. fn trap(&mut self, CodeOffset, SourceLoc, TrapCode); From cb37c25d3a3a8194c14fd67d30a1b27804bc4c92 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Apr 2018 10:54:57 -0700 Subject: [PATCH 1721/3084] Introduce Builder and Product types to the Module workflow. This eliminates API confusion and surface area with respect to what state the `Backend` needs to be in at different points. Now, API users will construct a `Builder`, and pass it into the `Module` which uses it to constrct a `Backend`. The `Backend` instance only lives inside the `Module`. And when finished, the `Module` can return a `Product` back to the user providing any outputs it has. --- lib/faerie/src/backend.rs | 105 ++++++++++++++++++++++++----------- lib/faerie/src/lib.rs | 2 +- lib/module/src/backend.rs | 19 ++++++- lib/module/src/module.rs | 14 ++--- lib/simplejit/src/backend.rs | 96 ++++++++++++++++++++------------ lib/simplejit/src/lib.rs | 2 +- 6 files changed, 161 insertions(+), 77 deletions(-) diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index 37e94e29e0..55af088295 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -11,19 +11,18 @@ use failure::Error; use std::fs::File; use target; -pub struct FaerieCompiledFunction {} - -pub struct FaerieCompiledData {} - -/// A `FaerieBackend` implements `Backend` and emits ".o" files using the `faerie` library. -pub struct FaerieBackend { +/// A builder for `FaerieBackend`. +pub struct FaerieBuilder { isa: Box, - artifact: faerie::Artifact, + name: String, format: container::Format, + faerie_target: faerie::Target, } -impl FaerieBackend { - /// Create a new `FaerieBackend` using the given Cretonne target. +impl FaerieBuilder { + /// Create a new `FaerieBuilder` using the given Cretonne target, that + /// can be passed to + /// [`Module::new`](cretonne_module/struct.Module.html#method.new]. pub fn new( isa: Box, name: String, @@ -33,34 +32,27 @@ impl FaerieBackend { let faerie_target = target::translate(&*isa)?; Ok(Self { isa, - artifact: faerie::Artifact::new(faerie_target, name), + name, format, + faerie_target, }) } - - /// Return the name of the output file. This is the name passed into `new`. - pub fn name(&self) -> &str { - &self.artifact.name - } - - /// Call `emit` on the faerie `Artifact`, producing bytes in memory. - pub fn emit(&self) -> Result, Error> { - match self.format { - container::Format::ELF => self.artifact.emit::(), - container::Format::MachO => self.artifact.emit::(), - } - } - - /// Call `write` on the faerie `Artifact`, writing to a file. - pub fn write(&self, sink: File) -> Result<(), Error> { - match self.format { - container::Format::ELF => self.artifact.write::(sink), - container::Format::MachO => self.artifact.write::(sink), - } - } } +/// A `FaerieBackend` implements `Backend` and emits ".o" files using the `faerie` library. +pub struct FaerieBackend { + isa: Box, + artifact: faerie::Artifact, + format: container::Format, +} + +pub struct FaerieCompiledFunction {} + +pub struct FaerieCompiledData {} + impl Backend for FaerieBackend { + type Builder = FaerieBuilder; + type CompiledFunction = FaerieCompiledFunction; type CompiledData = FaerieCompiledData; @@ -69,6 +61,19 @@ impl Backend for FaerieBackend { type FinalizedFunction = (); type FinalizedData = (); + /// The returned value here provides functions for emitting object files + /// to memory and files. + type Product = FaerieProduct; + + /// Create a new `FaerieBackend` using the given Cretonne target. + fn new(builder: FaerieBuilder) -> Self { + Self { + isa: builder.isa, + artifact: faerie::Artifact::new(builder.faerie_target, builder.name), + format: builder.format, + } + } + fn isa(&self) -> &TargetIsa { &*self.isa } @@ -216,6 +221,44 @@ impl Backend for FaerieBackend { fn finalize_data(&mut self, _data: &FaerieCompiledData, _namespace: &ModuleNamespace) { // Nothing to do. } + + fn finish(self) -> FaerieProduct { + FaerieProduct { + artifact: self.artifact, + format: self.format, + } + } +} + +/// This is the output of `Module`'s +/// [`finish`](../cretonne_module/struct.Module.html#method.finish) function. +/// It provides functions for writing out the object file to memory or a file. +pub struct FaerieProduct { + artifact: faerie::Artifact, + format: container::Format, +} + +impl FaerieProduct { + /// Return the name of the output file. This is the name passed into `new`. + pub fn name(&self) -> &str { + &self.artifact.name + } + + /// Call `emit` on the faerie `Artifact`, producing bytes in memory. + pub fn emit(&self) -> Result, Error> { + match self.format { + container::Format::ELF => self.artifact.emit::(), + container::Format::MachO => self.artifact.emit::(), + } + } + + /// Call `write` on the faerie `Artifact`, writing to a file. + pub fn write(&self, sink: File) -> Result<(), Error> { + match self.format { + container::Format::ELF => self.artifact.write::(sink), + container::Format::MachO => self.artifact.write::(sink), + } + } } fn translate_function_linkage(linkage: Linkage) -> faerie::Decl { diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs index 3cf007357c..3a6d0cad28 100644 --- a/lib/faerie/src/lib.rs +++ b/lib/faerie/src/lib.rs @@ -29,5 +29,5 @@ mod backend; mod container; mod target; -pub use backend::FaerieBackend; +pub use backend::{FaerieBuilder, FaerieBackend}; pub use container::Format; diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs index 0d6a6c6672..108b159e85 100644 --- a/lib/module/src/backend.rs +++ b/lib/module/src/backend.rs @@ -14,6 +14,9 @@ pub trait Backend where Self: marker::Sized, { + /// A builder for constructing `Backend` instances. + type Builder; + /// The results of compiling a function. type CompiledFunction; @@ -21,13 +24,21 @@ where type CompiledData; /// The completed output artifact for a function, if this is meaningful for - /// the Backend. + /// the `Backend`. type FinalizedFunction; /// The completed output artifact for a data object, if this is meaningful for - /// the Backend. + /// the `Backend`. type FinalizedData; + /// This is an object returned by `Module`'s + /// [`finish`](struct.Module.html#method.finish) function, + /// if the `Backend` has a purpose for this. + type Product; + + /// Create a new `Backend` instance. + fn new(Self::Builder) -> Self; + /// Return the `TargetIsa` to compile for. fn isa(&self) -> &TargetIsa; @@ -94,4 +105,8 @@ where data: &Self::CompiledData, namespace: &ModuleNamespace, ) -> Self::FinalizedData; + + /// Consume this `Backend` and return a result. Some implementations may + /// provide additional functionality through this result. + fn finish(self) -> Self::Product; } diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index 79ef29eba3..b47189f1d9 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -236,14 +236,14 @@ where B: Backend, { /// Create a new `Module`. - pub fn new(backend: B) -> Self { + pub fn new(backend_builder: B::Builder) -> Self { Self { names: HashMap::new(), contents: ModuleContents { functions: PrimaryMap::new(), data_objects: PrimaryMap::new(), }, - backend, + backend: B::new(backend_builder), } } @@ -532,10 +532,10 @@ where } } - /// Consume the module and return its contained `Backend`. Some `Backend` - /// implementations have additional features not available through the - /// `Module` interface. - pub fn consume(self) -> B { - self.backend + /// Consume the module and return the resulting `Product`. Some `Backend` + /// implementations may provide additional functionality available after + /// a `Module` is complete. + pub fn finish(self) -> B::Product { + self.backend.finish() } } diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs index 15e6892e19..f5e85575ce 100644 --- a/lib/simplejit/src/backend.rs +++ b/lib/simplejit/src/backend.rs @@ -12,6 +12,43 @@ use std::ptr; use libc; use memory::Memory; +/// A builder for `SimpleJITBackend`. +pub struct SimpleJITBuilder { + isa: Box, +} + +impl SimpleJITBuilder { + /// Create a new `SimpleJITBuilder`. + pub fn new() -> Self { + let (flag_builder, isa_builder) = cretonne_native::builders().unwrap_or_else(|_| { + panic!("host machine is not a supported target"); + }); + let isa = isa_builder.finish(settings::Flags::new(&flag_builder)); + Self::with_isa(isa) + } + + /// Create a new `SimpleJITBuilder` with an arbitrary target. This is mainly + /// useful for testing. + /// + /// SimpleJIT requires a `TargetIsa` configured for non-PIC. + /// + /// To create a `SimpleJITBuilder` for native use, use the `new` constructor + /// instead. + pub fn with_isa(isa: Box) -> Self { + debug_assert!(!isa.flags().is_pic(), "SimpleJIT requires non-PIC code"); + Self { isa } + } +} + +/// A `SimpleJITBackend` implements `Backend` and emits code and data into memory where it can be +/// directly called and accessed. +pub struct SimpleJITBackend { + isa: Box, + code_memory: Memory, + readonly_memory: Memory, + writable_memory: Memory, +} + /// A record of a relocation to perform. struct RelocRecord { offset: CodeOffset, @@ -32,49 +69,34 @@ pub struct SimpleJITCompiledData { relocs: Vec, } -/// A `SimpleJITBackend` implements `Backend` and emits code and data into memory where it can be -/// directly called and accessed. -pub struct SimpleJITBackend { - isa: Box, - code_memory: Memory, - readonly_memory: Memory, - writable_memory: Memory, -} +impl<'simple_jit_backend> Backend for SimpleJITBackend { + type Builder = SimpleJITBuilder; + + /// SimpleJIT compiled function and data objects may have outstanding + /// relocations that need to be performed before the memory can be used. + /// These relocations are performed within `finalize_function` and + /// `finalize_data`. + type CompiledFunction = SimpleJITCompiledFunction; + type CompiledData = SimpleJITCompiledData; + + /// SimpleJIT emits code and data into memory, and provides raw pointers + /// to them. + type FinalizedFunction = *const u8; + type FinalizedData = (*mut u8, usize); + + /// SimpleJIT emits code and data into memory as it processes them, so it + /// doesn't need to provide anything after the `Module` is complete. + type Product = (); -impl SimpleJITBackend { /// Create a new `SimpleJITBackend`. - pub fn new() -> Self { - let (flag_builder, isa_builder) = cretonne_native::builders().unwrap_or_else(|_| { - panic!("host machine is not a supported target"); - }); - let isa = isa_builder.finish(settings::Flags::new(&flag_builder)); - Self::with_isa(isa) - } - - /// Create a new `SimpleJITBackend` with an arbitrary target. This is mainly - /// useful for testing. - /// - /// SimpleJIT requires a `TargetIsa` configured for non-PIC. - /// - /// To create a `SimpleJITBackend` for native use, use the `new` constructor - /// instead. - pub fn with_isa(isa: Box) -> Self { - debug_assert!(!isa.flags().is_pic(), "SimpleJIT requires non-PIC code"); + fn new(builder: SimpleJITBuilder) -> Self { Self { - isa, + isa: builder.isa, code_memory: Memory::new(), readonly_memory: Memory::new(), writable_memory: Memory::new(), } } -} - -impl<'simple_jit_backend> Backend for SimpleJITBackend { - type CompiledFunction = SimpleJITCompiledFunction; - type CompiledData = SimpleJITCompiledData; - - type FinalizedFunction = *const u8; - type FinalizedData = (*mut u8, usize); fn isa(&self) -> &TargetIsa { &*self.isa @@ -317,6 +339,10 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { self.readonly_memory.set_readonly(); (data.storage, data.size) } + + /// SimpleJIT emits code and data into memory as it processes them, so it + /// doesn't need to provide anything after the `Module` is complete. + fn finish(self) -> () {} } fn lookup_with_dlsym(name: &str) -> *const u8 { diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs index 1917a3db5e..68a2ca00ea 100644 --- a/lib/simplejit/src/lib.rs +++ b/lib/simplejit/src/lib.rs @@ -26,4 +26,4 @@ extern crate libc; mod backend; mod memory; -pub use backend::SimpleJITBackend; +pub use backend::{SimpleJITBuilder, SimpleJITBackend}; From 8e4511ed6c814e26a37a063df00be880f1d0ebc0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Apr 2018 11:46:49 -0700 Subject: [PATCH 1722/3084] Elaborate on the prelude. Rename the re-exported crates, so that prelude users can type "cretonne::codegen" rather than "cretonne::cretonne_codegen". And, add a few more useful items to the prelude. --- lib/umbrella/src/lib.rs | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs index 179a33224a..f2a93ad230 100644 --- a/lib/umbrella/src/lib.rs +++ b/lib/umbrella/src/lib.rs @@ -16,17 +16,20 @@ use_self, ))] -pub extern crate cretonne_codegen; -pub extern crate cretonne_frontend; +/// Provide these crates, renamed to reduce stutter. +pub extern crate cretonne_codegen as codegen; +pub extern crate cretonne_frontend as frontend; /// A prelude providing convenient access to commonly-used cretonne features. Use /// as `use cretonne::prelude::*`. pub mod prelude { - pub use cretonne_codegen; - pub use cretonne_codegen::entity::EntityRef; - pub use cretonne_codegen::ir::{AbiParam, InstBuilder, Value, Ebb, Signature, CallConv}; - pub use cretonne_codegen::ir::types; - pub use cretonne_codegen::ir::condcodes::IntCC; + pub use codegen; + pub use codegen::entity::EntityRef; + pub use codegen::ir::{AbiParam, InstBuilder, Value, Ebb, Signature, CallConv, Type, + JumpTableData, MemFlags}; + pub use codegen::ir::types; + pub use codegen::ir::condcodes::{IntCC, FloatCC}; + pub use codegen::ir::immediates::{Ieee32, Ieee64}; - pub use cretonne_frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; + pub use frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; } From d9cd94f20de2ca891aec4e6eb0b232b85d836403 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Apr 2018 12:19:21 -0700 Subject: [PATCH 1723/3084] Add settings::Configurable and isa to the prelude. These are useful for API users that customize the TargetIsa. --- lib/umbrella/src/lib.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs index f2a93ad230..de89cd5788 100644 --- a/lib/umbrella/src/lib.rs +++ b/lib/umbrella/src/lib.rs @@ -30,6 +30,8 @@ pub mod prelude { pub use codegen::ir::types; pub use codegen::ir::condcodes::{IntCC, FloatCC}; pub use codegen::ir::immediates::{Ieee32, Ieee64}; + pub use codegen::settings::{self, Configurable}; + pub use codegen::isa; pub use frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; } From 830ee60d28faeef4830e03d85d519a7600151b16 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 18 Apr 2018 17:39:43 -0700 Subject: [PATCH 1724/3084] Add no_std support in module, simplejit, and umbrella. --- README.rst | 29 +++++++++++++++++++---------- cranelift/test-no_std.sh | 7 ++++++- lib/codegen/Cargo.toml | 6 +++--- lib/codegen/src/context.rs | 1 + lib/module/Cargo.toml | 13 +++++++++++-- lib/simplejit/Cargo.toml | 13 +++++++++---- lib/umbrella/Cargo.toml | 9 +++++++-- 7 files changed, 56 insertions(+), 22 deletions(-) diff --git a/README.rst b/README.rst index 28f2250ace..1e519f2117 100644 --- a/README.rst +++ b/README.rst @@ -35,9 +35,9 @@ the System V AMD64 ABI calling convention used on many platforms, but does not yet support the Windows x64 calling convention. The performance of code produced by Cretonne is not yet impressive, though we have plans to fix that. -The core codegen crates have minimal dependencies, and do not require any host -floating-point support. Support for `no_std` mode in the core codegen crates is -`in development `_. +The core codegen crates have minimal dependencies, support +`no_std <#building-with-no-std>` mode, and do not require any host +floating-point support. Cretonne does not yet perform mitigations for Spectre or related security issues, though it may do so in the future. It does not currently make any @@ -93,22 +93,31 @@ installed. Building with `no_std` ---------------------- -To build cretonne without libstd, disable the `std` feature on `lib/cretonne`, -`lib/frontend`, `lib/native`, and `lib/wasm`, which is otherwise enabled by -default, and enable the `core` feature. +The following crates support `no_std`: + - `cretonne-entity` + - `cretonne-codegen` + - `cretonne-frontend` + - `cretonne-native` + - `cretonne-wasm` + - `cretonne-module` + - `cretonne-simplejit` + - `cretonne` -For example, to build `cretonne`: +To use `no_std` mode, disable the `std` feature and enable the `core` feature. +This currently requires nightly rust. + +For example, to build `cretonne-codegen`: .. code-block:: sh - cd lib/cretonne + cd lib/codegen cargo build --no-default-features --features core -Or, when using `cretonne` as a dependency (in Cargo.toml): +Or, when using `cretonne-codegen` as a dependency (in Cargo.toml): .. code-block:: - [dependency.cretonne] + [dependency.cretonne-codegen] ... default-features = false features = ["core"] diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh index 800c712402..66a629eb57 100755 --- a/cranelift/test-no_std.sh +++ b/cranelift/test-no_std.sh @@ -15,14 +15,19 @@ function banner() { } # Test those packages which have no_std support. -LIBS="codegen frontend wasm native" +LIBS="codegen frontend wasm native module simplejit umbrella" cd "$topdir" for LIB in $LIBS do banner "Rust unit tests in $LIB" cd "lib/$LIB" + + # Test with just "core" enabled. cargo test --no-default-features --features core + + # Test with "core" and "std" enabled at the same time. cargo test --features core + cd "$topdir" done diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 2a7854f6c1..0c5a1104ba 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -12,12 +12,12 @@ build = "build.rs" [dependencies] cretonne-entity = { path = "../entity", version = "0.5.1", default-features = false } +failure = { version = "0.1.1", default-features = false, features = ["derive"] } +failure_derive = { version = "0.1.1", default-features = false } # It is a goal of the cretonne-codegen crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be # accomodated in `tests`. -failure = { version = "0.1.1", default-features = false, features = ["derive"] } -failure_derive = { version = "0.1.1", default-features = false } [dependencies.hashmap_core] version = "0.1.1" @@ -28,7 +28,7 @@ optional = true # of some minimal std-like replacement libraries. At least one of these two # features to be enabled. default = ["std"] -std = [] +std = ["cretonne-entity/std"] core = ["hashmap_core"] [badges] diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index 95e78ff128..71b873b653 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -23,6 +23,7 @@ use preopt::do_preopt; use regalloc; use result::{CtonError, CtonResult}; use settings::{FlagsOrIsa, OptLevel}; +use std::vec::Vec; use simple_gvn::do_simple_gvn; use timing; use unreachable_code::eliminate_unreachable_code; diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 42dc6ee916..f9d5728109 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -9,8 +9,17 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.1" } -cretonne-entity = { path = "../entity", version = "0.5.1" } +cretonne-codegen = { path = "../codegen", version = "0.5.1", default-features = false } +cretonne-entity = { path = "../entity", version = "0.5.1", default-features = false } + +[dependencies.hashmap_core] +version = "0.1.1" +optional = true + +[features] +default = ["std"] +std = ["cretonne-codegen/std", "cretonne-entity/std"] +core = ["hashmap_core", "cretonne-codegen/core"] [badges] maintenance = { status = "experimental" } diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index 0fdc16da32..7d5fbb4d23 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -9,13 +9,18 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.1" } -cretonne-module = { path = "../module", version = "0.5.1" } -cretonne-native = { path = "../native", version = "0.5.1" } +cretonne-codegen = { path = "../codegen", version = "0.5.1", default-features = false } +cretonne-module = { path = "../module", version = "0.5.1", default-features = false } +cretonne-native = { path = "../native", version = "0.5.1", default-features = false } region = "0.2.0" -libc = "0.2.40" +libc = { version = "0.2.40", default-features = false } errno = "0.2.3" +[features] +default = ["std"] +std = ["libc/use_std", "cretonne-codegen/std", "cretonne-module/std", "cretonne-native/std"] +core = ["cretonne-codegen/core", "cretonne-module/core", "cretonne-native/core"] + [badges] maintenance = { status = "experimental" } travis-ci = { repository = "Cretonne/cretonne" } diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml index f7bdc002cd..6895955244 100644 --- a/lib/umbrella/Cargo.toml +++ b/lib/umbrella/Cargo.toml @@ -10,8 +10,13 @@ readme = "README.md" keywords = ["compile", "compiler", "jit"] [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.1" } -cretonne-frontend = { path = "../frontend", version = "0.5.1" } +cretonne-codegen = { path = "../codegen", version = "0.5.1", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.5.1", default-features = false } + +[features] +default = ["std"] +std = ["cretonne-codegen/std", "cretonne-frontend/std"] +core = ["cretonne-codegen/core", "cretonne-frontend/core"] [badges] maintenance = { status = "experimental" } From d72706c47845701d181491392466f3cd3a3fe197 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Apr 2018 13:14:12 -0700 Subject: [PATCH 1725/3084] Use `use` declarations rather than '::std::...' names. This is what most of the rest of the codebase does, so this patch just tidies up a few additional places. --- lib/codegen/src/bforest/map.rs | 16 ++++++++++------ lib/codegen/src/bforest/pool.rs | 6 ++++-- lib/codegen/src/bforest/set.rs | 8 ++++++-- lib/codegen/src/dbg.rs | 1 - lib/codegen/src/ir/extname.rs | 3 ++- lib/codegen/src/legalizer/libcall.rs | 3 ++- 6 files changed, 24 insertions(+), 13 deletions(-) diff --git a/lib/codegen/src/bforest/map.rs b/lib/codegen/src/bforest/map.rs index b6a62ef419..7cfa824a43 100644 --- a/lib/codegen/src/bforest/map.rs +++ b/lib/codegen/src/bforest/map.rs @@ -3,6 +3,10 @@ use super::{Comparator, Forest, Node, NodeData, NodePool, Path, INNER_SIZE}; use packed_option::PackedOption; use std::marker::PhantomData; +#[cfg(test)] +use std::fmt; +#[cfg(test)] +use std::string::String; /// Tag type defining forest types for a map. struct MapTypes(PhantomData<(K, V, C)>); @@ -207,14 +211,14 @@ where #[cfg(test)] impl Map where - K: Copy + ::std::fmt::Display, + K: Copy + fmt::Display, V: Copy, C: Comparator, { /// Verify consistency. fn verify(&self, forest: &MapForest, comp: &C) where - NodeData>: ::std::fmt::Display, + NodeData>: fmt::Display, { if let Some(root) = self.root.expand() { forest.nodes.verify_tree(root, comp); @@ -222,7 +226,7 @@ where } /// Get a text version of the path to `key`. - fn tpath(&self, key: K, forest: &MapForest, comp: &C) -> ::std::string::String { + fn tpath(&self, key: K, forest: &MapForest, comp: &C) -> String { use std::string::ToString; match self.root.expand() { None => "map(empty)".to_string(), @@ -406,8 +410,8 @@ where #[cfg(test)] impl<'a, K, V, C> MapCursor<'a, K, V, C> where - K: Copy + ::std::fmt::Display, - V: Copy + ::std::fmt::Display, + K: Copy + fmt::Display, + V: Copy + fmt::Display, C: Comparator, { fn verify(&self) { @@ -416,7 +420,7 @@ where } /// Get a text version of the path to the current position. - fn tpath(&self) -> ::std::string::String { + fn tpath(&self) -> String { use std::string::ToString; self.path.to_string() } diff --git a/lib/codegen/src/bforest/pool.rs b/lib/codegen/src/bforest/pool.rs index eed9402c84..d543991312 100644 --- a/lib/codegen/src/bforest/pool.rs +++ b/lib/codegen/src/bforest/pool.rs @@ -3,6 +3,8 @@ use super::{Forest, Node, NodeData}; use entity::PrimaryMap; use std::ops::{Index, IndexMut}; +#[cfg(test)] +use std::fmt; /// A pool of nodes, including a free list. pub(super) struct NodePool { @@ -74,8 +76,8 @@ impl NodePool { /// Verify the consistency of the tree rooted at `node`. pub fn verify_tree(&self, node: Node, comp: &F::Comparator) where - NodeData: ::std::fmt::Display, - F::Key: ::std::fmt::Display, + NodeData: fmt::Display, + F::Key: fmt::Display, { use super::Comparator; use entity::SparseSet; diff --git a/lib/codegen/src/bforest/set.rs b/lib/codegen/src/bforest/set.rs index 3c479dd876..667cd133e2 100644 --- a/lib/codegen/src/bforest/set.rs +++ b/lib/codegen/src/bforest/set.rs @@ -3,6 +3,10 @@ use super::{Comparator, Forest, Node, NodeData, NodePool, Path, SetValue, INNER_SIZE}; use packed_option::PackedOption; use std::marker::PhantomData; +#[cfg(test)] +use std::fmt; +#[cfg(test)] +use std::string::String; /// Tag type defining forest types for a set. struct SetTypes(PhantomData<(K, C)>); @@ -305,7 +309,7 @@ where #[cfg(test)] impl<'a, K, C> SetCursor<'a, K, C> where - K: Copy + ::std::fmt::Display, + K: Copy + fmt::Display, C: Comparator, { fn verify(&self) { @@ -314,7 +318,7 @@ where } /// Get a text version of the path to the current position. - fn tpath(&self) -> ::std::string::String { + fn tpath(&self) -> String { use std::string::ToString; self.path.to_string() } diff --git a/lib/codegen/src/dbg.rs b/lib/codegen/src/dbg.rs index 6c51bd23e5..a514dd7a05 100644 --- a/lib/codegen/src/dbg.rs +++ b/lib/codegen/src/dbg.rs @@ -116,7 +116,6 @@ macro_rules! dbg { if $crate::dbg::enabled() { // Drop the error result so we don't get compiler errors for ignoring it. // What are you going to do, log the error? - #[cfg(feature = "std")] $crate::dbg::writeln_with_format_args(format_args!($($arg)+)).ok(); } } diff --git a/lib/codegen/src/ir/extname.rs b/lib/codegen/src/ir/extname.rs index dcb3666703..ed723be6fe 100644 --- a/lib/codegen/src/ir/extname.rs +++ b/lib/codegen/src/ir/extname.rs @@ -121,6 +121,7 @@ mod tests { use super::ExternalName; use ir::LibCall; use std::string::ToString; + use std::u32; #[test] fn display_testcase() { @@ -143,7 +144,7 @@ mod tests { assert_eq!(ExternalName::user(0, 0).to_string(), "u0:0"); assert_eq!(ExternalName::user(1, 1).to_string(), "u1:1"); assert_eq!( - ExternalName::user(::std::u32::MAX, ::std::u32::MAX).to_string(), + ExternalName::user(u32::MAX, u32::MAX).to_string(), "u4294967295:4294967295" ); } diff --git a/lib/codegen/src/legalizer/libcall.rs b/lib/codegen/src/legalizer/libcall.rs index 8f69cba18d..17f9174706 100644 --- a/lib/codegen/src/legalizer/libcall.rs +++ b/lib/codegen/src/legalizer/libcall.rs @@ -2,6 +2,7 @@ use ir; use ir::InstBuilder; +use std::vec::Vec; /// Try to expand `inst` as a library call, returning true is successful. pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function) -> bool { @@ -15,7 +16,7 @@ pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function) -> bool { let funcref = find_funcref(libcall, func).unwrap_or_else(|| make_funcref(libcall, inst, func)); // Now we convert `inst` to a call. First save the arguments. - let mut args = vec![]; + let mut args = Vec::new(); args.extend_from_slice(func.dfg.inst_args(inst)); // The replace builder will preserve the instruction result values. func.dfg.replace(inst).call(funcref, &args); From 73c19bbf6da2d5009e93e17b04d16bec2fd4ecc8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Apr 2018 13:18:21 -0700 Subject: [PATCH 1726/3084] Fix the `dbg!` macro for `no_std` mode. Check for the `std` feature outside the macro body rather than inside, so that we get the `std` feature in `cretonne-codegen`, rather than in whatever crate the macro is being expanded in. --- lib/codegen/src/dbg.rs | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/lib/codegen/src/dbg.rs b/lib/codegen/src/dbg.rs index a514dd7a05..8dd47f625d 100644 --- a/lib/codegen/src/dbg.rs +++ b/lib/codegen/src/dbg.rs @@ -110,6 +110,7 @@ fn open_file() -> io::BufWriter { /// Write a line to the debug trace file if tracing is enabled. /// /// Arguments are the same as for `printf!`. +#[cfg(feature = "std")] #[macro_export] macro_rules! dbg { ($($arg:tt)+) => { @@ -121,6 +122,13 @@ macro_rules! dbg { } } +/// `dbg!` isn't supported in `no_std` mode, so expand it into nothing. +#[cfg(not(feature = "std"))] +#[macro_export] +macro_rules! dbg { + ($($arg:tt)+) => {} +} + /// Helper for printing lists. pub struct DisplayList<'a, T>(pub &'a [T]) where From c9a606da69bc9051c7b99e125f96091bb0a32a10 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Apr 2018 13:44:07 -0700 Subject: [PATCH 1727/3084] Use more lower-case letters for github URLs. This is a continuation of 362f8f13e2b8c7dd79b043c491479bb9fc6f69e9. --- lib/faerie/Cargo.toml | 2 +- lib/module/Cargo.toml | 2 +- lib/simplejit/Cargo.toml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index 77477b8f98..1914154f69 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -3,7 +3,7 @@ name = "cretonne-faerie" version = "0.5.1" authors = ["The Cretonne Project Developers"] description = "Emit Cretonne output to native object files with Faerie" -repository = "https://github.com/Cretonne/cretonne" +repository = "https://github.com/cretonne/cretonne" documentation = "https://cretonne.readthedocs.io/" license = "Apache-2.0" readme = "README.md" diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 42dc6ee916..e2b2aa30c5 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -3,7 +3,7 @@ name = "cretonne-module" version = "0.5.1" authors = ["The Cretonne Project Developers"] description = "Support for linking functions and data with Cretonne" -repository = "https://github.com/Cretonne/cretonne" +repository = "https://github.com/cretonne/cretonne" documentation = "https://cretonne.readthedocs.io/" license = "Apache-2.0" readme = "README.md" diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index 0fdc16da32..b45890450d 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -3,7 +3,7 @@ name = "cretonne-simplejit" version = "0.5.1" authors = ["The Cretonne Project Developers"] description = "A simple JIT library backed by Cretonne" -repository = "https://github.com/Cretonne/cretonne" +repository = "https://github.com/cretonne/cretonne" documentation = "https://cretonne.readthedocs.io/" license = "Apache-2.0" readme = "README.md" From fd8df67cfd9ba6faf9195659694f8ca1a1e14c48 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Apr 2018 13:52:27 -0700 Subject: [PATCH 1728/3084] Convert an http link to https. --- lib/filetests/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 4e80bfc17e..1a61c7c1e5 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -4,7 +4,7 @@ authors = ["The Cretonne Project Developers"] version = "0.5.1" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" -documentation = "http://cretonne.readthedocs.io/en/latest/testing.html#file-tests" +documentation = "https://cretonne.readthedocs.io/en/latest/testing.html#file-tests" repository = "https://github.com/cretonne/cretonne" publish = false From 583ae56fd22e0f54cd1c1c9d039ea754870f1c7e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Apr 2018 16:20:01 -0700 Subject: [PATCH 1729/3084] Use opt_level instead of is_compressed for encoding optimizations. Choosing smaller instruction encodings on eg. x86 is an optimization, rather than a useful discrete setting. Use "is_compressed" only for ISAs that have an explicit compression feature that users of the output may to be aware of, such as RISC-V's RVC or ARM's Thumb-2. --- cranelift/filetests/isa/x86/allones_funcaddrs32.cton | 2 +- cranelift/filetests/isa/x86/allones_funcaddrs64.cton | 2 +- .../isa/x86/baseline_clz_ctz_popcount_encoding.cton | 3 +-- cranelift/filetests/isa/x86/binary32.cton | 2 +- cranelift/filetests/isa/x86/binary64-float.cton | 2 +- cranelift/filetests/isa/x86/binary64-pic.cton | 2 +- cranelift/filetests/isa/x86/binary64.cton | 2 +- cranelift/filetests/isa/x86/legalize-call.cton | 2 +- cranelift/filetests/isa/x86/prologue-epilogue.cton | 2 +- lib/filetests/src/test_binemit.rs | 7 ++++--- 10 files changed, 13 insertions(+), 13 deletions(-) diff --git a/cranelift/filetests/isa/x86/allones_funcaddrs32.cton b/cranelift/filetests/isa/x86/allones_funcaddrs32.cton index c4c078470b..cf0036f4d0 100644 --- a/cranelift/filetests/isa/x86/allones_funcaddrs32.cton +++ b/cranelift/filetests/isa/x86/allones_funcaddrs32.cton @@ -1,6 +1,6 @@ ; binary emission of 32-bit code. test binemit -set is_compressed +set opt_level=best set allones_funcaddrs isa x86 haswell diff --git a/cranelift/filetests/isa/x86/allones_funcaddrs64.cton b/cranelift/filetests/isa/x86/allones_funcaddrs64.cton index ca3cf1ab69..3f47c78052 100644 --- a/cranelift/filetests/isa/x86/allones_funcaddrs64.cton +++ b/cranelift/filetests/isa/x86/allones_funcaddrs64.cton @@ -1,7 +1,7 @@ ; binary emission of 64-bit code. test binemit set is_64bit -set is_compressed +set opt_level=best set allones_funcaddrs isa x86 haswell diff --git a/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton b/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton index f69efd573a..72104272e1 100644 --- a/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton +++ b/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton @@ -1,7 +1,6 @@ - test binemit set is_64bit -set is_compressed +set opt_level=best isa x86 baseline ; The binary encodings can be verified with the command: diff --git a/cranelift/filetests/isa/x86/binary32.cton b/cranelift/filetests/isa/x86/binary32.cton index 842f941779..6eb4565f79 100644 --- a/cranelift/filetests/isa/x86/binary32.cton +++ b/cranelift/filetests/isa/x86/binary32.cton @@ -1,6 +1,6 @@ ; binary emission of x86-32 code. test binemit -set is_compressed +set opt_level=best isa x86 haswell ; The binary encodings can be verified with the command: diff --git a/cranelift/filetests/isa/x86/binary64-float.cton b/cranelift/filetests/isa/x86/binary64-float.cton index df910941cd..e9fcc99e63 100644 --- a/cranelift/filetests/isa/x86/binary64-float.cton +++ b/cranelift/filetests/isa/x86/binary64-float.cton @@ -1,7 +1,7 @@ ; Binary emission of 64-bit floating point code. test binemit set is_64bit -set is_compressed +set opt_level=best isa x86 haswell ; The binary encodings can be verified with the command: diff --git a/cranelift/filetests/isa/x86/binary64-pic.cton b/cranelift/filetests/isa/x86/binary64-pic.cton index cc592e0188..ff9cfc42ac 100644 --- a/cranelift/filetests/isa/x86/binary64-pic.cton +++ b/cranelift/filetests/isa/x86/binary64-pic.cton @@ -1,7 +1,7 @@ ; binary emission of 64-bit code. test binemit set is_64bit -set is_compressed +set opt_level=best set is_pic isa x86 haswell diff --git a/cranelift/filetests/isa/x86/binary64.cton b/cranelift/filetests/isa/x86/binary64.cton index 2a945ee17c..6417cd26e2 100644 --- a/cranelift/filetests/isa/x86/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.cton @@ -1,7 +1,7 @@ ; binary emission of x86-64 code. test binemit set is_64bit -set is_compressed +set opt_level=best isa x86 haswell ; The binary encodings can be verified with the command: diff --git a/cranelift/filetests/isa/x86/legalize-call.cton b/cranelift/filetests/isa/x86/legalize-call.cton index adcb6a6ad1..0eaaba0dfa 100644 --- a/cranelift/filetests/isa/x86/legalize-call.cton +++ b/cranelift/filetests/isa/x86/legalize-call.cton @@ -1,7 +1,7 @@ ; Test legalization of a non-colocated call in 64-bit non-PIC mode. test legalizer set is_64bit -set is_compressed +set opt_level=best isa x86 haswell function %call() { diff --git a/cranelift/filetests/isa/x86/prologue-epilogue.cton b/cranelift/filetests/isa/x86/prologue-epilogue.cton index 9ed37274ca..76aa459f94 100644 --- a/cranelift/filetests/isa/x86/prologue-epilogue.cton +++ b/cranelift/filetests/isa/x86/prologue-epilogue.cton @@ -1,6 +1,6 @@ test compile set is_64bit -set is_compressed +set opt_level=best set is_pic isa x86 haswell diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index 86058673e3..a0711414d4 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -9,6 +9,7 @@ use cretonne_codegen::dbg::DisplayList; use cretonne_codegen::ir; use cretonne_codegen::ir::entities::AnyEntity; use cretonne_codegen::print_errors::pretty_error; +use cretonne_codegen::settings::OptLevel; use cretonne_reader::TestCommand; use match_directive::match_directive; use std::borrow::Cow; @@ -121,7 +122,7 @@ impl SubTest for TestBinEmit { .min(); func.stack_slots.frame_size = min_offset.map(|off| (-off) as u32); - let is_compressed = isa.flags().is_compressed(); + let opt_level = isa.flags().opt_level(); // Give an encoding to any instruction that doesn't already have one. let mut divert = RegDiversions::new(); @@ -141,11 +142,11 @@ impl SubTest for TestBinEmit { recipe_constraints.satisfied(inst, &divert, &func) }); - if is_compressed { + if opt_level == OptLevel::Best { // Get the smallest legal encoding legal_encodings.min_by_key(|&e| encinfo.bytes(e)) } else { - // If not using compressed, just use the first encoding. + // If not optimizing, just use the first encoding. legal_encodings.next() } } From bce8af97e32cbfdbcda6654949ace248f51efb88 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Apr 2018 14:03:38 -0700 Subject: [PATCH 1730/3084] Add an instruction shrinking pass. When an instruction has multiple valid encodings, such as with and without a REX prefix on x86-64, Cretonne typically picks the encoding which gives the register allocator the most flexibility, which is typically the longest encoding. This patch adds a pass that runs after register allocation that picks the smallest encoding, working within the constraints of the register allocator's choices. The result is smaller and easier to read encodings. In the future, we may want to merge this pass into the relaxation pass, or possibly fold it into the final encoding step, however for now, a discrete pass will suffice. --- .../filetests/isa/x86/prologue-epilogue.cton | 2 +- cranelift/filetests/isa/x86/shrink.cton | 29 +++++++++++ lib/codegen/src/binemit/mod.rs | 2 + lib/codegen/src/binemit/shrink.rs | 49 +++++++++++++++++++ lib/codegen/src/context.rs | 13 ++++- 5 files changed, 93 insertions(+), 2 deletions(-) create mode 100644 cranelift/filetests/isa/x86/shrink.cton create mode 100644 lib/codegen/src/binemit/shrink.rs diff --git a/cranelift/filetests/isa/x86/prologue-epilogue.cton b/cranelift/filetests/isa/x86/prologue-epilogue.cton index 76aa459f94..07880f272b 100644 --- a/cranelift/filetests/isa/x86/prologue-epilogue.cton +++ b/cranelift/filetests/isa/x86/prologue-epilogue.cton @@ -228,4 +228,4 @@ ebb4: ; check: function %divert ; check: regmove v5, %rcx -> %rbx -; check: [RexOp1popq#58,%rbx] v15 = x86_pop.i64 +; check: [Op1popq#58,%rbx] v15 = x86_pop.i64 diff --git a/cranelift/filetests/isa/x86/shrink.cton b/cranelift/filetests/isa/x86/shrink.cton new file mode 100644 index 0000000000..1dc5fbcd42 --- /dev/null +++ b/cranelift/filetests/isa/x86/shrink.cton @@ -0,0 +1,29 @@ +test binemit +set is_64bit=1 +set opt_level=best +isa x86 + +; Test that instruction shrinking eliminates REX prefixes when possible. + +; The binary encodings can be verified with the command: +; +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/shrink.cton | llvm-mc -show-encoding -triple=x86_64 +; + +function %test_shrinking(i32) -> i32 { +ebb0(v0: i32 [ %rdi ]): + ; asm: movl $0x2,%eax +[-,%rcx] v1 = iconst.i32 2 ; bin: b9 00000002 + ; asm: subl %ecx,%edi +[-,%rdi] v2 = isub v0, v1 ; bin: 29 cf + return v2 +} + +function %test_not_shrinking(i32) -> i32 { +ebb0(v0: i32 [ %r8 ]): + ; asm: movl $0x2,%eax +[-,%rcx] v1 = iconst.i32 2 ; bin: b9 00000002 + ; asm: subl %ecx,%edi +[-,%r8] v2 = isub v0, v1 ; bin: 41 29 c8 + return v2 +} diff --git a/lib/codegen/src/binemit/mod.rs b/lib/codegen/src/binemit/mod.rs index 9752901ce9..653e38c9ed 100644 --- a/lib/codegen/src/binemit/mod.rs +++ b/lib/codegen/src/binemit/mod.rs @@ -5,9 +5,11 @@ mod memorysink; mod relaxation; +mod shrink; pub use self::memorysink::{MemoryCodeSink, RelocSink, TrapSink, NullTrapSink}; pub use self::relaxation::relax_branches; +pub use self::shrink::shrink_instructions; pub use regalloc::RegDiversions; use ir::{ExternalName, Function, Inst, JumpTable, SourceLoc, TrapCode}; diff --git a/lib/codegen/src/binemit/shrink.rs b/lib/codegen/src/binemit/shrink.rs new file mode 100644 index 0000000000..0ae15584cb --- /dev/null +++ b/lib/codegen/src/binemit/shrink.rs @@ -0,0 +1,49 @@ +//! Instruction shrinking. +//! +//! Sometimes there are multiple valid encodings for a given instruction. Cretonne often initially +//! chooses the largest one, because this typically provides the register allocator the most +//! flexibility. However, once register allocation is done, this is no longer important, and we +//! can switch to smaller encodings when possible. + +use ir::Function; +use isa::TargetIsa; +use regalloc::RegDiversions; + +/// Pick the smallest valid encodings for instructions. +pub fn shrink_instructions(func: &mut Function, isa: &TargetIsa) { + let encinfo = isa.encoding_info(); + let mut divert = RegDiversions::new(); + + for ebb in func.layout.ebbs() { + divert.clear(); + for inst in func.layout.ebb_insts(ebb) { + let enc = func.encodings[inst]; + if enc.is_legal() { + let ctrl_type = func.dfg.ctrl_typevar(inst); + + // Pick the last encoding with constraints that are satisfied. + let best_enc = isa.legal_encodings(func, &func.dfg[inst], ctrl_type) + .filter(|e| { + encinfo.constraints[e.recipe()].satisfied(inst, &divert, &func) + }) + .min_by_key(|e| encinfo.bytes(*e)) + .unwrap(); + + if best_enc != enc { + func.encodings[inst] = best_enc; + + dbg!( + "Shrunk [{}] to [{}] in {}, reducing the size from {} to {}", + encinfo.display(enc), + encinfo.display(best_enc), + func.dfg.display_inst(inst, isa), + encinfo.bytes(enc), + encinfo.bytes(best_enc) + ); + } + + } + divert.apply(&func.dfg[inst]); + } + } +} diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index 95e78ff128..42fde1cc4a 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -9,7 +9,7 @@ //! contexts concurrently. Typically, you would have one context per compilation thread and only a //! single ISA instance. -use binemit::{relax_branches, CodeOffset, MemoryCodeSink, RelocSink, TrapSink}; +use binemit::{relax_branches, shrink_instructions, CodeOffset, MemoryCodeSink, RelocSink, TrapSink}; use dce::do_dce; use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; @@ -140,6 +140,9 @@ impl Context { } self.regalloc(isa)?; self.prologue_epilogue(isa)?; + if isa.flags().opt_level() == OptLevel::Best { + self.shrink_instructions(isa)?; + } self.relax_branches(isa) } @@ -294,6 +297,14 @@ impl Context { Ok(()) } + /// Run the instruction shrinking pass. + pub fn shrink_instructions(&mut self, isa: &TargetIsa) -> CtonResult { + shrink_instructions(&mut self.func, isa); + self.verify_if(isa)?; + self.verify_locations_if(isa)?; + Ok(()) + } + /// Run the branch relaxation pass and return the final code size. pub fn relax_branches(&mut self, isa: &TargetIsa) -> Result { let code_size = relax_branches(&mut self.func, isa)?; From 5434f4dafa58dda32c440b762f6e54322ae1ae76 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Apr 2018 17:09:39 -0700 Subject: [PATCH 1731/3084] Update to raw_cpuid 3.1.0. This hopefully fixes MSVC build issues; see #302. --- lib/native/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 2909fa8307..79fc6e7b6c 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -11,7 +11,7 @@ readme = "README.md" cretonne-codegen = { path = "../codegen", version = "0.5.1" } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] -raw-cpuid = "3.0.0" +raw-cpuid = "3.1.0" [badges] maintenance = { status = "experimental" } From 3e4531657c1514d296b210ff586453c4dd4c6d30 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Apr 2018 17:49:48 -0700 Subject: [PATCH 1732/3084] Temporarily disable the shink_instruction pass. It appears some of the instruction encodings are incorrect. Temporarily disable the use of shorter encodings until these are fixed. --- lib/codegen/src/context.rs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index 42fde1cc4a..d9dcf7ba87 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -140,9 +140,12 @@ impl Context { } self.regalloc(isa)?; self.prologue_epilogue(isa)?; + // Temporarily disable the shrink_instructions pass, as it causes miscompiles. + /* if isa.flags().opt_level() == OptLevel::Best { self.shrink_instructions(isa)?; } + */ self.relax_branches(isa) } From cb3c5a1384f253e883486aa418672373242f3016 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 19 Apr 2018 18:57:59 -0700 Subject: [PATCH 1733/3084] Revert the change to this test that accompanied the shrinking pass too. --- cranelift/filetests/isa/x86/prologue-epilogue.cton | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/filetests/isa/x86/prologue-epilogue.cton b/cranelift/filetests/isa/x86/prologue-epilogue.cton index 07880f272b..76aa459f94 100644 --- a/cranelift/filetests/isa/x86/prologue-epilogue.cton +++ b/cranelift/filetests/isa/x86/prologue-epilogue.cton @@ -228,4 +228,4 @@ ebb4: ; check: function %divert ; check: regmove v5, %rcx -> %rbx -; check: [Op1popq#58,%rbx] v15 = x86_pop.i64 +; check: [RexOp1popq#58,%rbx] v15 = x86_pop.i64 From 33e266eeeb149d0d06413af609fe824b27c3cf2b Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Apr 2018 11:04:13 -0700 Subject: [PATCH 1734/3084] Fix missing word. --- lib/codegen/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 0c5a1104ba..8304c50ca4 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -26,7 +26,7 @@ optional = true [features] # The "std" feature enables use of libstd. The "core" feature enables use # of some minimal std-like replacement libraries. At least one of these two -# features to be enabled. +# features need to be enabled. default = ["std"] std = ["cretonne-entity/std"] core = ["hashmap_core"] From 9249080ce8b5d3ed72a34bae584f888602aabcdc Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Apr 2018 11:06:25 -0700 Subject: [PATCH 1735/3084] Fix URL syntax. --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 1e519f2117..59316dff13 100644 --- a/README.rst +++ b/README.rst @@ -36,7 +36,7 @@ yet support the Windows x64 calling convention. The performance of code produced by Cretonne is not yet impressive, though we have plans to fix that. The core codegen crates have minimal dependencies, support -`no_std <#building-with-no-std>` mode, and do not require any host +`no_std <#building-with-no-std>`_ mode, and do not require any host floating-point support. Cretonne does not yet perform mitigations for Spectre or related security From 653c11d580dcacd930f3657c80c9f626170abdef Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Apr 2018 11:11:38 -0700 Subject: [PATCH 1736/3084] Use "set -euo pipefail" in test-no_std.sh. This makes it consistent with other shell scripts in the repo. --- cranelift/test-no_std.sh | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh index 66a629eb57..7d372f3558 100755 --- a/cranelift/test-no_std.sh +++ b/cranelift/test-no_std.sh @@ -1,11 +1,9 @@ #!/bin/bash +set -euo pipefail # This is the test script for testing the no_std configuration of # packages which support it. -# Exit immediately on errors. -set -e - # Repository top-level directory. cd $(dirname "$0") topdir=$(pwd) From dfb24f19345a30c06a106ba88d600af05a0e1da2 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Apr 2018 12:08:01 -0700 Subject: [PATCH 1737/3084] Fix x86 encoding of uextend/sextend from 8-bit inputs. The x86-32 and non-REX encodings of movsbl and movzbl require one of the ABCD registers as input. --- cranelift/filetests/isa/x86/abcd.cton | 13 +++++++++++++ lib/codegen/meta/isa/x86/encodings.py | 9 +++++---- 2 files changed, 18 insertions(+), 4 deletions(-) create mode 100644 cranelift/filetests/isa/x86/abcd.cton diff --git a/cranelift/filetests/isa/x86/abcd.cton b/cranelift/filetests/isa/x86/abcd.cton new file mode 100644 index 0000000000..f0007d795d --- /dev/null +++ b/cranelift/filetests/isa/x86/abcd.cton @@ -0,0 +1,13 @@ +test regalloc +isa x86 + +; %rdi can't be used in a movsbl instruction, so test that the register +; allocator can move it to a register that can be. + +function %test(i32 [%rdi]) -> i32 system_v { +ebb0(v0: i32 [%rdi]): + v1 = ireduce.i8 v0 + v2 = sextend.i32 v1 + return v2 +} +; check: regmove v1, %rdi -> %rax diff --git a/lib/codegen/meta/isa/x86/encodings.py b/lib/codegen/meta/isa/x86/encodings.py index 5ab6636a44..8b3244222b 100644 --- a/lib/codegen/meta/isa/x86/encodings.py +++ b/lib/codegen/meta/isa/x86/encodings.py @@ -128,6 +128,7 @@ enc_i32_i64(base.copy, r.umr, 0x89) enc_both(base.copy.b1, r.umr, 0x89) enc_i32_i64(base.regmove, r.rmov, 0x89) enc_both(base.regmove.b1, r.rmov, 0x89) +enc_both(base.regmove.i8, r.rmov, 0x89) # Immediate instructions with sign-extended 8-bit and 32-bit immediate. for inst, rrr in [ @@ -461,9 +462,9 @@ X86_64.enc(base.ireduce.i32.i64, r.null, 0) # instructions for %al/%ax/%eax to %ax/%eax/%rax. # movsbl -X86_32.enc(base.sextend.i32.i8, *r.urm_noflags(0x0f, 0xbe)) +X86_32.enc(base.sextend.i32.i8, *r.urm_noflags_abcd(0x0f, 0xbe)) X86_64.enc(base.sextend.i32.i8, *r.urm_noflags.rex(0x0f, 0xbe)) -X86_64.enc(base.sextend.i32.i8, *r.urm_noflags(0x0f, 0xbe)) +X86_64.enc(base.sextend.i32.i8, *r.urm_noflags_abcd(0x0f, 0xbe)) # movswl X86_32.enc(base.sextend.i32.i16, *r.urm_noflags(0x0f, 0xbf)) @@ -480,9 +481,9 @@ X86_64.enc(base.sextend.i64.i16, *r.urm_noflags.rex(0x0f, 0xbf, w=1)) X86_64.enc(base.sextend.i64.i32, *r.urm_noflags.rex(0x63, w=1)) # movzbl -X86_32.enc(base.uextend.i32.i8, *r.urm_noflags(0x0f, 0xb6)) +X86_32.enc(base.uextend.i32.i8, *r.urm_noflags_abcd(0x0f, 0xb6)) X86_64.enc(base.uextend.i32.i8, *r.urm_noflags.rex(0x0f, 0xb6)) -X86_64.enc(base.uextend.i32.i8, *r.urm_noflags(0x0f, 0xb6)) +X86_64.enc(base.uextend.i32.i8, *r.urm_noflags_abcd(0x0f, 0xb6)) # movzwl X86_32.enc(base.uextend.i32.i16, *r.urm_noflags(0x0f, 0xb7)) From 7079c72b28e3f2ef53cc046f2adbf82f14c270d8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Apr 2018 16:36:08 -0700 Subject: [PATCH 1738/3084] Add comments to DataContext's import_function and import_global_var. These are the main obvious place where ExternalName is exposed in the Module API, so add comments advising users that they can use Module to call these functions with the appropriate ExternalNames automatically. --- lib/module/src/data_context.rs | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/lib/module/src/data_context.rs b/lib/module/src/data_context.rs index 51eac88b79..9e9f823383 100644 --- a/lib/module/src/data_context.rs +++ b/lib/module/src/data_context.rs @@ -104,6 +104,10 @@ impl DataContext { } /// Declare an external function import. + /// + /// Users of the `Module` API generally should call + /// `Module::declare_func_in_data` instead, as it takes care of generating + /// the appropriate `ExternalName`. pub fn import_function(&mut self, name: ir::ExternalName) -> ir::FuncRef { self.description.function_decls.push(name) } @@ -111,6 +115,10 @@ impl DataContext { /// Declares a global variable import. /// /// TODO: Rename to import_data? + /// + /// Users of the `Module` API generally should call + /// `Module::declare_data_in_data` instead, as it takes care of generating + /// the appropriate `ExternalName`. pub fn import_global_var(&mut self, name: ir::ExternalName) -> ir::GlobalVar { self.description.data_decls.push(name) } From 8a9e4b9cff642b586bbca759ce2d122f233a1457 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Apr 2018 16:39:15 -0700 Subject: [PATCH 1739/3084] Update to faerie 0.2.0. --- lib/faerie/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index 1914154f69..db7b9c2f35 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -11,7 +11,7 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.5.1" } cretonne-module = { path = "../module", version = "0.5.1" } -faerie = "0.1.0" +faerie = "0.2.0" goblin = "0.0.14" failure = "0.1.1" From c5b15c2396dc963ece66e6e24d1f510de16a407a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sun, 22 Apr 2018 21:35:18 -0700 Subject: [PATCH 1740/3084] Refactor calling convention settings. (#304) Add a calling-convention setting to the `Flags` used as part of the `TargetIsa`. This allows Cretonne code that generates calls to use the correct convention, such as when emitting libcalls during legalization or when the wasm frontend is decoding functions. This setting can be overridden per-function. This also adds "fast", "cold", and "fastcall" conventions, with "fast" as the new default. Note that "fast" and "cold" are not intended to be ABI-compatible across Cretonne versions. This will also ensure Windows users will get an `unimplemented!` rather than silent calling-convention mismatches, which reflects the fact that Windows calling conventions are not yet implemented. This also renames SpiderWASM, which isn't camel-case, to Baldrdash, which is, and which is also a more relevant name. --- cranelift/docs/heapex-dyn.cton | 2 +- cranelift/docs/heapex-sm32.cton | 2 +- cranelift/docs/heapex-sm64.cton | 2 +- cranelift/docs/langref.rst | 23 ++++++- .../filetests/isa/riscv/parse-encoding.cton | 2 +- cranelift/filetests/isa/x86/abi64.cton | 4 +- .../filetests/isa/x86/legalize-libcall.cton | 4 +- .../filetests/isa/x86/legalize-memory.cton | 8 +-- .../filetests/isa/x86/prologue-epilogue.cton | 12 ++-- cranelift/filetests/parser/branch.cton | 12 ++-- cranelift/filetests/parser/call.cton | 18 ++--- .../parser/instruction_encoding.cton | 2 +- cranelift/filetests/parser/keywords.cton | 2 +- cranelift/filetests/parser/rewrite.cton | 4 +- cranelift/filetests/parser/tiny.cton | 24 +++---- cranelift/filetests/regalloc/aliases.cton | 2 +- cranelift/filetests/regalloc/iterate.cton | 6 +- .../filetests/regalloc/unreachable_code.cton | 2 +- lib/codegen/meta/base/settings.py | 23 +++++-- lib/codegen/meta/gen_settings.py | 32 ++++++++- lib/codegen/src/ir/extfunc.rs | 65 +++++-------------- lib/codegen/src/ir/function.rs | 9 +-- lib/codegen/src/ir/mod.rs | 3 +- lib/codegen/src/isa/mod.rs | 5 +- lib/codegen/src/isa/x86/abi.rs | 23 +++---- lib/codegen/src/legalizer/libcall.rs | 17 +++-- lib/codegen/src/legalizer/mod.rs | 2 +- lib/codegen/src/settings.rs | 6 +- lib/codegen/src/write.rs | 12 ++-- lib/faerie/src/backend.rs | 4 ++ lib/frontend/src/frontend.rs | 3 +- lib/frontend/src/lib.rs | 4 +- lib/native/src/lib.rs | 11 +++- lib/reader/src/parser.rs | 20 +++--- lib/umbrella/src/lib.rs | 6 +- lib/wasm/src/environ/dummy.rs | 4 ++ lib/wasm/src/environ/spec.rs | 3 + lib/wasm/src/sections_translator.rs | 4 +- 38 files changed, 226 insertions(+), 161 deletions(-) diff --git a/cranelift/docs/heapex-dyn.cton b/cranelift/docs/heapex-dyn.cton index 2be801da86..9f1f2ac078 100644 --- a/cranelift/docs/heapex-dyn.cton +++ b/cranelift/docs/heapex-dyn.cton @@ -1,6 +1,6 @@ test verifier -function %add_members(i32) -> f32 spiderwasm { +function %add_members(i32) -> f32 baldrdash { gv0 = vmctx+64 gv1 = vmctx+72 heap0 = dynamic gv0, min 0x1000, bound gv1, guard 0 diff --git a/cranelift/docs/heapex-sm32.cton b/cranelift/docs/heapex-sm32.cton index 51b0c4cbcb..61014d05f5 100644 --- a/cranelift/docs/heapex-sm32.cton +++ b/cranelift/docs/heapex-sm32.cton @@ -1,6 +1,6 @@ test verifier -function %add_members(i32) -> f32 spiderwasm { +function %add_members(i32) -> f32 baldrdash { gv0 = vmctx+64 heap0 = static gv0, min 0x1000, bound 0x10_0000, guard 0x1000 diff --git a/cranelift/docs/heapex-sm64.cton b/cranelift/docs/heapex-sm64.cton index e29fa9caf5..4755e8790a 100644 --- a/cranelift/docs/heapex-sm64.cton +++ b/cranelift/docs/heapex-sm64.cton @@ -1,6 +1,6 @@ test verifier -function %add_members(i32) -> f32 spiderwasm { +function %add_members(i32) -> f32 baldrdash { gv0 = vmctx+64 heap0 = static gv0, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000 diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 33fc1f0694..8a8baa5b33 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -399,7 +399,28 @@ convention: param : type [paramext] [paramspecial] paramext : "uext" | "sext" paramspecial : "sret" | "link" | "fp" | "csr" | "vmctx" - callconv : "system_v" | "spiderwasm" + callconv : "fast" | "cold" | "system_v" | "fastcall" | "baldrdash" + +A function's calling convention determines exactly how arguments and return +values are passed, and how stack frames are managed. Since all of these details +depend on both the instruction set /// architecture and possibly the operating +system, a function's calling convention is only fully determined by a +`(TargetIsa, CallConv)` tuple. + +========== =========================================== +Name Description +========== =========================================== +fast not-ABI-stable convention for best performance +cold not-ABI-stable convention for infrequently executed code +system_v System V-style convention used on many platforms +fastcall Windows "fastcall" convention, also used for x64 and ARM +baldrdash SpiderMonkey WebAssembly convention +========== =========================================== + +The "not-ABI-stable" conventions do not follow an external specification and +may change between versions of Cretonne. + +The "fastcall" convention is not yet implemented. Parameters and return values have flags whose meaning is mostly target dependent. These flags support interfacing with code produced by other diff --git a/cranelift/filetests/isa/riscv/parse-encoding.cton b/cranelift/filetests/isa/riscv/parse-encoding.cton index b8220b1688..e61f43ef84 100644 --- a/cranelift/filetests/isa/riscv/parse-encoding.cton +++ b/cranelift/filetests/isa/riscv/parse-encoding.cton @@ -3,7 +3,7 @@ test legalizer isa riscv function %parse_encoding(i32 [%x5]) -> i32 [%x10] { - ; check: function %parse_encoding(i32 [%x5], i32 link [%x1]) -> i32 [%x10], i32 link [%x1] system_v { + ; check: function %parse_encoding(i32 [%x5], i32 link [%x1]) -> i32 [%x10], i32 link [%x1] fast { sig0 = (i32 [%x10]) -> i32 [%x10] system_v ; check: sig0 = (i32 [%x10]) -> i32 [%x10] system_v diff --git a/cranelift/filetests/isa/x86/abi64.cton b/cranelift/filetests/isa/x86/abi64.cton index 40321f90fb..121e5cc1a8 100644 --- a/cranelift/filetests/isa/x86/abi64.cton +++ b/cranelift/filetests/isa/x86/abi64.cton @@ -19,8 +19,8 @@ ebb0: return } -function %pass_stack_int64(i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) spiderwasm { - sig0 = (i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) spiderwasm +function %pass_stack_int64(i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) baldrdash { + sig0 = (i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) baldrdash fn0 = u0:0 sig0 ebb0(v0: i64, v1: i64, v2: i64, v3: i64, v4: i64, v5: i64, v6: i64, v7: i64, v8: i64, v9: i64, v10: i64, v11: i64, v12: i64, v13: i64, v14: i64, v15: i64, v16: i64, v17: i64, v18: i64, v19: i64, v20: i64): diff --git a/cranelift/filetests/isa/x86/legalize-libcall.cton b/cranelift/filetests/isa/x86/legalize-libcall.cton index 70d392e6e4..d633bd4e46 100644 --- a/cranelift/filetests/isa/x86/legalize-libcall.cton +++ b/cranelift/filetests/isa/x86/legalize-libcall.cton @@ -10,7 +10,7 @@ ebb0(v0: f32): v1 = floor v0 return v1 } -; check: function %floor(f32 [%xmm0]) -> f32 [%xmm0] system_v { -; check: sig0 = (f32) -> f32 system_v +; check: function %floor(f32 [%xmm0]) -> f32 [%xmm0] fast { +; check: sig0 = (f32) -> f32 fast ; check: fn0 = %FloorF32 sig0 ; check: v1 = call fn0(v0) diff --git a/cranelift/filetests/isa/x86/legalize-memory.cton b/cranelift/filetests/isa/x86/legalize-memory.cton index 9cb8661873..fc77c757e3 100644 --- a/cranelift/filetests/isa/x86/legalize-memory.cton +++ b/cranelift/filetests/isa/x86/legalize-memory.cton @@ -44,7 +44,7 @@ ebb1: ; SpiderMonkey VM-style static 4+2 GB heap. ; This eliminates bounds checks completely for offsets < 2GB. -function %staticheap_sm64(i32, i64 vmctx) -> f32 spiderwasm { +function %staticheap_sm64(i32, i64 vmctx) -> f32 baldrdash { gv0 = vmctx+64 heap0 = static gv0, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000 @@ -65,7 +65,7 @@ ebb0(v0: i32, v999: i64): return v4 } -function %staticheap_static_oob_sm64(i32, i64 vmctx) -> f32 spiderwasm { +function %staticheap_static_oob_sm64(i32, i64 vmctx) -> f32 baldrdash { gv0 = vmctx+64 heap0 = static gv0, min 0x1000, bound 0x1000_0000, guard 0x8000_0000 @@ -88,7 +88,7 @@ ebb0(v0: i32, v999: i64): ; SpiderMonkey VM-style static 4+2 GB heap. ; Offsets >= 2 GB do require a boundscheck. -function %staticheap_sm64(i32, i64 vmctx) -> f32 spiderwasm { +function %staticheap_sm64(i32, i64 vmctx) -> f32 baldrdash { gv0 = vmctx+64 heap0 = static gv0, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000 @@ -112,7 +112,7 @@ ebb0(v0: i32, v999: i64): ; Stack overflow check. ; The stack limit is stored in a pointer-sized global variable. -function %stkchk(i64 vmctx) spiderwasm { +function %stkchk(i64 vmctx) baldrdash { gv0 = vmctx+64 ebb0(v0: i64): diff --git a/cranelift/filetests/isa/x86/prologue-epilogue.cton b/cranelift/filetests/isa/x86/prologue-epilogue.cton index 76aa459f94..bf4bf298b5 100644 --- a/cranelift/filetests/isa/x86/prologue-epilogue.cton +++ b/cranelift/filetests/isa/x86/prologue-epilogue.cton @@ -11,7 +11,7 @@ ebb0: return } -; check: function %empty(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; check: function %empty(i64 fp [%rbp]) -> i64 fp [%rbp] fast { ; nextln: ss0 = incoming_arg 16, offset -16 ; nextln: ; nextln: ebb0(v0: i64 [%rbp]): @@ -29,7 +29,7 @@ ebb0: return } -; check: function %one_stack_slot(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; check: function %one_stack_slot(i64 fp [%rbp]) -> i64 fp [%rbp] fast { ; nextln: ss0 = explicit_slot 168, offset -184 ; nextln: ss1 = incoming_arg 16, offset -16 ; nextln: @@ -52,9 +52,9 @@ ebb0: return } -; check: function %call(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; check: function %call(i64 fp [%rbp]) -> i64 fp [%rbp] fast { ; nextln: ss0 = incoming_arg 16, offset -16 -; nextln: sig0 = () system_v +; nextln: sig0 = () fast ; nextln: fn0 = %foo sig0 ; nextln: ; nextln: ebb0(v0: i64 [%rbp]): @@ -98,7 +98,7 @@ ebb0(v0: i64, v1: i64): return } -; check: function %no_spill(i64 [%rdi], i64 [%rsi], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] system_v { +; check: function %no_spill(i64 [%rdi], i64 [%rsi], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] fast { ; nextln: ss0 = incoming_arg 56, offset -56 ; nextln: ; nextln: ebb0(v0: i64 [%rdi], v1: i64 [%rsi], v15: i64 [%rbp], v16: i64 [%rbx], v17: i64 [%r12], v18: i64 [%r13], v19: i64 [%r14], v20: i64 [%r15]): @@ -181,7 +181,7 @@ ebb0(v0: i64, v1: i64): return } -; check: function %yes_spill(i64 [%rdi], i64 [%rsi], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] system_v { +; check: function %yes_spill(i64 [%rdi], i64 [%rsi], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] fast { ; check: ss0 = spill_slot ; check: ebb0(v16: i64 [%rdi], v17: i64 [%rsi], v48: i64 [%rbp], v49: i64 [%rbx], v50: i64 [%r12], v51: i64 [%r13], v52: i64 [%r14], v53: i64 [%r15]): diff --git a/cranelift/filetests/parser/branch.cton b/cranelift/filetests/parser/branch.cton index 4e35147a6b..2969c8f384 100644 --- a/cranelift/filetests/parser/branch.cton +++ b/cranelift/filetests/parser/branch.cton @@ -9,7 +9,7 @@ ebb0: ebb1: jump ebb0() } -; sameln: function %minimal() system_v { +; sameln: function %minimal() fast { ; nextln: ebb0: ; nextln: jump ebb1 ; nextln: @@ -25,7 +25,7 @@ ebb0(v90: i32): ebb1(v91: i32): jump ebb0(v91) } -; sameln: function %onearg(i32) system_v { +; sameln: function %onearg(i32) fast { ; nextln: ebb0(v90: i32): ; nextln: jump ebb1(v90) ; nextln: @@ -41,7 +41,7 @@ ebb0(v90: i32, v91: f32): ebb1(v92: i32, v93: f32): jump ebb0(v92, v93) } -; sameln: function %twoargs(i32, f32) system_v { +; sameln: function %twoargs(i32, f32) fast { ; nextln: ebb0(v90: i32, v91: f32): ; nextln: jump ebb1(v90, v91) ; nextln: @@ -57,7 +57,7 @@ ebb0(v90: i32): ebb1: brnz v90, ebb1() } -; sameln: function %minimal(i32) system_v { +; sameln: function %minimal(i32) fast { ; nextln: ebb0(v90: i32): ; nextln: brz v90, ebb1 ; nextln: @@ -72,7 +72,7 @@ ebb0(v90: i32, v91: f32): ebb1(v92: i32, v93: f32): brnz v90, ebb0(v92, v93) } -; sameln: function %twoargs(i32, f32) system_v { +; sameln: function %twoargs(i32, f32) fast { ; nextln: ebb0(v90: i32, v91: f32): ; nextln: brz v90, ebb1(v90, v91) ; nextln: @@ -94,7 +94,7 @@ ebb30: ebb40: trap user4 } -; sameln: function %jumptable(i32) system_v { +; sameln: function %jumptable(i32) fast { ; check: jt2 = jump_table 0, 0, ebb10, ebb40, ebb20, ebb30 ; check: jt200 = jump_table 0 ; check: ebb10(v3: i32): diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.cton index 9d1e8def02..c75fa02393 100644 --- a/cranelift/filetests/parser/call.cton +++ b/cranelift/filetests/parser/call.cton @@ -5,18 +5,18 @@ function %mini() { ebb1: return } -; sameln: function %mini() system_v { +; sameln: function %mini() fast { ; nextln: ebb1: ; nextln: return ; nextln: } -function %r1() -> i32, f32 spiderwasm { +function %r1() -> i32, f32 baldrdash { ebb1: v1 = iconst.i32 3 v2 = f32const 0.0 return v1, v2 } -; sameln: function %r1() -> i32, f32 spiderwasm { +; sameln: function %r1() -> i32, f32 baldrdash { ; nextln: ebb1: ; nextln: v1 = iconst.i32 3 ; nextln: v2 = f32const 0.0 @@ -25,14 +25,14 @@ ebb1: function %signatures() { sig10 = () - sig11 = (i32, f64) -> i32, b1 spiderwasm + sig11 = (i32, f64) -> i32, b1 baldrdash fn5 = %foo sig11 fn8 = %bar(i32) -> b1 } -; sameln: function %signatures() system_v { -; check: sig10 = () system_v -; check: sig11 = (i32, f64) -> i32, b1 spiderwasm -; check: sig12 = (i32) -> b1 system_v +; sameln: function %signatures() fast { +; check: sig10 = () fast +; check: sig11 = (i32, f64) -> i32, b1 baldrdash +; check: sig12 = (i32) -> b1 fast ; not: fn0 ; check: fn5 = %foo sig11 ; check: fn8 = %bar sig12 @@ -88,7 +88,7 @@ function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 ebb0(v1: i32, v2: i32, v3: i32, v4: i32): return v4, v2, v3, v1 } -; check: function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret system_v { +; check: function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret fast { ; check: ebb0(v1: i32, v2: i32, v3: i32, v4: i32): ; check: return v4, v2, v3, v1 ; check: } diff --git a/cranelift/filetests/parser/instruction_encoding.cton b/cranelift/filetests/parser/instruction_encoding.cton index e0a2de789a..7d75e4362a 100644 --- a/cranelift/filetests/parser/instruction_encoding.cton +++ b/cranelift/filetests/parser/instruction_encoding.cton @@ -13,7 +13,7 @@ ebb1(v0: i32 [%x8], v1: i32): @55 v9 = iadd v8, v7 @a5 [Iret#5] return v0, v8 } -; sameln: function %foo(i32, i32) system_v { +; sameln: function %foo(i32, i32) fast { ; nextln: ebb1(v0: i32 [%x8], v1: i32): ; nextln: [-,-]$WS v2 = iadd v0, v1 ; nextln: [-]$WS trap heap_oob diff --git a/cranelift/filetests/parser/keywords.cton b/cranelift/filetests/parser/keywords.cton index aaf8403c0a..e079892be7 100644 --- a/cranelift/filetests/parser/keywords.cton +++ b/cranelift/filetests/parser/keywords.cton @@ -2,4 +2,4 @@ test cat ; 'function' is not a keyword, and can be used as the name of a function too. function %function() {} -; check: function %function() system_v +; check: function %function() fast diff --git a/cranelift/filetests/parser/rewrite.cton b/cranelift/filetests/parser/rewrite.cton index 679f00d2bc..c40bd9589b 100644 --- a/cranelift/filetests/parser/rewrite.cton +++ b/cranelift/filetests/parser/rewrite.cton @@ -9,7 +9,7 @@ ebb100(v20: i32): v9200 = f64const 0x4.0p0 trap user4 } -; sameln: function %defs() system_v { +; sameln: function %defs() fast { ; nextln: ebb100(v20: i32): ; nextln: v1000 = iconst.i32x8 5 ; nextln: v9200 = f64const 0x1.0000000000000p2 @@ -23,7 +23,7 @@ ebb100(v20: i32): v200 = iadd v20, v1000 jump ebb100(v1000) } -; sameln: function %use_value() system_v { +; sameln: function %use_value() fast { ; nextln: ebb100(v20: i32): ; nextln: v1000 = iadd_imm v20, 5 ; nextln: v200 = iadd v20, v1000 diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index e800d44c26..2a342a666b 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -5,7 +5,7 @@ function %minimal() { ebb0: trap user0 } -; sameln: function %minimal() system_v { +; sameln: function %minimal() fast { ; nextln: ebb0: ; nextln: trap user0 ; nextln: } @@ -18,7 +18,7 @@ ebb0: v1 = iconst.i8 6 v2 = ishl v0, v1 } -; sameln: function %ivalues() system_v { +; sameln: function %ivalues() fast { ; nextln: ebb0: ; nextln: v0 = iconst.i32 2 ; nextln: v1 = iconst.i8 6 @@ -34,7 +34,7 @@ ebb0: v2 = bextend.b32 v1 v3 = bxor v0, v2 } -; sameln: function %bvalues() system_v { +; sameln: function %bvalues() fast { ; nextln: ebb0: ; nextln: v0 = bconst.b32 true ; nextln: v1 = bconst.b8 false @@ -47,7 +47,7 @@ function %select() { ebb0(v90: i32, v91: i32, v92: b1): v0 = select v92, v90, v91 } -; sameln: function %select() system_v { +; sameln: function %select() fast { ; nextln: ebb0(v90: i32, v91: i32, v92: b1): ; nextln: v0 = select v92, v90, v91 ; nextln: } @@ -69,7 +69,7 @@ ebb0: v1 = extractlane v0, 3 v2 = insertlane v0, 1, v1 } -; sameln: function %lanes() system_v { +; sameln: function %lanes() fast { ; nextln: ebb0: ; nextln: v0 = iconst.i32x4 2 ; nextln: v1 = extractlane v0, 3 @@ -85,7 +85,7 @@ ebb0(v90: i32, v91: i32): v3 = irsub_imm v91, 45 br_icmp eq v90, v91, ebb0(v91, v90) } -; sameln: function %icmp(i32, i32) system_v { +; sameln: function %icmp(i32, i32) fast { ; nextln: ebb0(v90: i32, v91: i32): ; nextln: v0 = icmp eq v90, v91 ; nextln: v1 = icmp ult v90, v91 @@ -101,7 +101,7 @@ ebb0(v90: f32, v91: f32): v1 = fcmp uno v90, v91 v2 = fcmp lt v90, v91 } -; sameln: function %fcmp(f32, f32) system_v { +; sameln: function %fcmp(f32, f32) fast { ; nextln: ebb0(v90: f32, v91: f32): ; nextln: v0 = fcmp eq v90, v91 ; nextln: v1 = fcmp uno v90, v91 @@ -115,7 +115,7 @@ ebb0(v90: i32, v91: f32): v0 = bitcast.i8x4 v90 v1 = bitcast.i32 v91 } -; sameln: function %bitcast(i32, f32) system_v { +; sameln: function %bitcast(i32, f32) fast { ; nextln: ebb0(v90: i32, v91: f32): ; nextln: v0 = bitcast.i8x4 v90 ; nextln: v1 = bitcast.i32 v91 @@ -135,7 +135,7 @@ ebb0: stack_store v1, ss10+2 stack_store v2, ss2 } -; sameln: function %stack() system_v { +; sameln: function %stack() fast { ; check: ss2 = explicit_slot 4 ; check: ss3 = incoming_arg 4, offset 8 ; check: ss4 = outgoing_arg 4 @@ -162,7 +162,7 @@ ebb0(v1: i32): store aligned v3, v1+12 store notrap aligned v3, v1-12 } -; sameln: function %memory(i32) system_v { +; sameln: function %memory(i32) fast { ; nextln: ebb0(v1: i32): ; nextln: v2 = load.i64 v1 ; nextln: v3 = load.i64 aligned v1 @@ -187,7 +187,7 @@ ebb0(v1: i32): regfill v1, ss0 -> %10 return } -; sameln: function %diversion(i32) system_v { +; sameln: function %diversion(i32) fast { ; nextln: ss0 = spill_slot 4 ; check: ebb0(v1: i32): ; nextln: regmove v1, %10 -> %20 @@ -204,7 +204,7 @@ ebb0: copy_special %20 -> %10 return } -; sameln: function %copy_special() system_v { +; sameln: function %copy_special() fast { ; nextln: ebb0: ; nextln: copy_special %10 -> %20 ; nextln: copy_special %20 -> %10 diff --git a/cranelift/filetests/regalloc/aliases.cton b/cranelift/filetests/regalloc/aliases.cton index 096ba99c2a..eb84c77f98 100644 --- a/cranelift/filetests/regalloc/aliases.cton +++ b/cranelift/filetests/regalloc/aliases.cton @@ -2,7 +2,7 @@ test regalloc set is_64bit isa x86 haswell -function %value_aliases(i32, f32, i64 vmctx) spiderwasm { +function %value_aliases(i32, f32, i64 vmctx) baldrdash { gv0 = vmctx heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 diff --git a/cranelift/filetests/regalloc/iterate.cton b/cranelift/filetests/regalloc/iterate.cton index acb97a31af..44e1265a96 100644 --- a/cranelift/filetests/regalloc/iterate.cton +++ b/cranelift/filetests/regalloc/iterate.cton @@ -2,7 +2,7 @@ test regalloc set is_64bit isa x86 haswell -function u0:9(i64 [%rdi], f32 [%xmm0], f64 [%xmm1], i32 [%rsi], i32 [%rdx], i64 vmctx [%r14]) -> i64 [%rax] spiderwasm { +function u0:9(i64 [%rdi], f32 [%xmm0], f64 [%xmm1], i32 [%rsi], i32 [%rdx], i64 vmctx [%r14]) -> i64 [%rax] baldrdash { ebb0(v0: i64, v1: f32, v2: f64, v3: i32, v4: i32, v5: i64): v32 = iconst.i32 0 v6 = bitcast.f32 v32 @@ -104,9 +104,9 @@ ebb1(v31: i64): return v31 } -function u0:26(i64 vmctx [%r14]) -> i64 [%rax] spiderwasm { +function u0:26(i64 vmctx [%r14]) -> i64 [%rax] baldrdash { gv0 = vmctx+48 - sig0 = (i32 [%rdi], i64 [%rsi], i64 vmctx [%r14], i64 sigid [%rbx]) -> i64 [%rax] spiderwasm + sig0 = (i32 [%rdi], i64 [%rsi], i64 vmctx [%r14], i64 sigid [%rbx]) -> i64 [%rax] baldrdash ebb0(v0: i64): v1 = iconst.i32 32 diff --git a/cranelift/filetests/regalloc/unreachable_code.cton b/cranelift/filetests/regalloc/unreachable_code.cton index 4b4896f4a8..34d8c4c164 100644 --- a/cranelift/filetests/regalloc/unreachable_code.cton +++ b/cranelift/filetests/regalloc/unreachable_code.cton @@ -6,7 +6,7 @@ isa x86 haswell ; This function contains unreachable blocks which trip up the register ; allocator if they don't get cleared out. -function %unreachable_blocks(i64 vmctx) -> i32 spiderwasm { +function %unreachable_blocks(i64 vmctx) -> i32 baldrdash { ebb0(v0: i64): v1 = iconst.i32 0 v2 = iconst.i32 0 diff --git a/lib/codegen/meta/base/settings.py b/lib/codegen/meta/base/settings.py index ff1551c151..580f0ee33e 100644 --- a/lib/codegen/meta/base/settings.py +++ b/lib/codegen/meta/base/settings.py @@ -29,6 +29,21 @@ enable_verifier = BoolSetting( is_64bit = BoolSetting("Enable 64-bit code generation") +call_conv = EnumSetting( + """ + Default calling convention: + + - fast: not-ABI-stable convention for best performance + - cold: not-ABI-stable convention for infrequently executed code + - system_v: System V-style convention used on many platforms + - fastcall: Windows "fastcall" convention, also used for x64 and ARM + - baldrdash: SpiderMonkey WebAssembly convention + + The default calling convention may be overridden by individual + functions. + """, + 'fast', 'cold', 'system_v', 'fastcall', 'baldrdash') + # Note that Cretonne doesn't currently need an is_pie flag, because PIE is just # PIC where symbols can't be pre-empted, which can be expressed with the # `colocated` flag on external functions and global variables. @@ -75,13 +90,13 @@ enable_atomics = BoolSetting( default=True) # -# Settings specific to the `spiderwasm` calling convention. +# Settings specific to the `baldrdash` calling convention. # -spiderwasm_prologue_words = NumSetting( +baldrdash_prologue_words = NumSetting( """ - Number of pointer-sized words pushed by the spiderwasm prologue. + Number of pointer-sized words pushed by the baldrdash prologue. - Functions with the `spiderwasm` calling convention don't generate their + Functions with the `baldrdash` calling convention don't generate their own prologue and epilogue. They depend on externally generated code that pushes a fixed number of words in the prologue and restores them in the epilogue. diff --git a/lib/codegen/meta/gen_settings.py b/lib/codegen/meta/gen_settings.py index f111c6c415..5f760b88e5 100644 --- a/lib/codegen/meta/gen_settings.py +++ b/lib/codegen/meta/gen_settings.py @@ -19,6 +19,32 @@ except ImportError: pass +def gen_to_and_from_str(ty, values, fmt): + # type: (str, Tuple[str, ...], srcgen.Formatter) -> None + """ + Emit Display and FromStr implementations for enum settings. + """ + with fmt.indented('impl fmt::Display for {} {{'.format(ty), '}'): + with fmt.indented( + 'fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {', + '}'): + with fmt.indented('f.write_str(match *self {', '})'): + for v in values: + fmt.line('{}::{} => "{}",' + .format(ty, camel_case(v), v)) + + with fmt.indented('impl str::FromStr for {} {{'.format(ty), '}'): + fmt.line('type Err = ();') + with fmt.indented( + 'fn from_str(s: &str) -> result::Result {', + '}'): + with fmt.indented('match s {', '}'): + for v in values: + fmt.line('"{}" => Ok({}::{}),' + .format(v, ty, camel_case(v))) + fmt.line('_ => Err(()),') + + def gen_enum_types(sgrp, fmt): # type: (SettingGroup, srcgen.Formatter) -> None """ @@ -29,12 +55,14 @@ def gen_enum_types(sgrp, fmt): continue ty = camel_case(setting.name) fmt.doc_comment('Values for `{}`.'.format(setting)) - fmt.line('#[derive(Debug, Copy, Clone, PartialEq, Eq)]') + fmt.line('#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]') with fmt.indented('pub enum {} {{'.format(ty), '}'): for v in setting.values: fmt.doc_comment('`{}`.'.format(v)) fmt.line(camel_case(v) + ',') + gen_to_and_from_str(ty, setting.values, fmt) + def gen_getter(setting, sgrp, fmt): # type: (Setting, SettingGroup, srcgen.Formatter) -> None @@ -221,7 +249,7 @@ def gen_display(sgrp, fmt): with fmt.indented('if !d.detail.is_preset() {', '}'): fmt.line('write!(f, "{} = ", d.name)?;') fmt.line( - 'TEMPLATE.format_toml_value(d.detail,' + + 'TEMPLATE.format_toml_value(d.detail, ' + 'self.bytes[d.offset as usize], f)?;') fmt.line('writeln!(f)?;') fmt.line('Ok(())') diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs index 01008946e6..540083321e 100644 --- a/lib/codegen/src/ir/extfunc.rs +++ b/lib/codegen/src/ir/extfunc.rs @@ -7,6 +7,7 @@ use ir::{ArgumentLoc, ExternalName, SigRef, Type}; use isa::{RegInfo, RegUnit}; +use settings::CallConv; use std::cmp; use std::fmt; use std::str::FromStr; @@ -342,47 +343,6 @@ impl fmt::Display for ExtFuncData { } } -/// A Calling convention. -/// -/// A function's calling convention determines exactly how arguments and return values are passed, -/// and how stack frames are managed. Since all of these details depend on both the instruction set -/// architecture and possibly the operating system, a function's calling convention is only fully -/// determined by a `(TargetIsa, CallConv)` tuple. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] -pub enum CallConv { - /// The System V-style calling convention. - /// - /// This is the System V-style calling convention that a C compiler would - /// use on many platforms. - SystemV, - - /// A JIT-compiled WebAssembly function in the SpiderMonkey VM. - SpiderWASM, -} - -impl fmt::Display for CallConv { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - use self::CallConv::*; - f.write_str(match *self { - SystemV => "system_v", - SpiderWASM => "spiderwasm", - }) - } -} - -impl FromStr for CallConv { - type Err = (); - - fn from_str(s: &str) -> Result { - use self::CallConv::*; - match s { - "system_v" => Ok(SystemV), - "spiderwasm" => Ok(SpiderWASM), - _ => Err(()), - } - } -} - #[cfg(test)] mod tests { use super::*; @@ -418,23 +378,30 @@ mod tests { #[test] fn call_conv() { - for &cc in &[CallConv::SystemV, CallConv::SpiderWASM] { + for &cc in &[ + CallConv::Fast, + CallConv::Cold, + CallConv::SystemV, + CallConv::Fastcall, + CallConv::Baldrdash, + ] + { assert_eq!(Ok(cc), cc.to_string().parse()) } } #[test] fn signatures() { - let mut sig = Signature::new(CallConv::SpiderWASM); - assert_eq!(sig.to_string(), "() spiderwasm"); + let mut sig = Signature::new(CallConv::Baldrdash); + assert_eq!(sig.to_string(), "() baldrdash"); sig.params.push(AbiParam::new(I32)); - assert_eq!(sig.to_string(), "(i32) spiderwasm"); + assert_eq!(sig.to_string(), "(i32) baldrdash"); sig.returns.push(AbiParam::new(F32)); - assert_eq!(sig.to_string(), "(i32) -> f32 spiderwasm"); + assert_eq!(sig.to_string(), "(i32) -> f32 baldrdash"); sig.params.push(AbiParam::new(I32.by(4).unwrap())); - assert_eq!(sig.to_string(), "(i32, i32x4) -> f32 spiderwasm"); + assert_eq!(sig.to_string(), "(i32, i32x4) -> f32 baldrdash"); sig.returns.push(AbiParam::new(B8)); - assert_eq!(sig.to_string(), "(i32, i32x4) -> f32, b8 spiderwasm"); + assert_eq!(sig.to_string(), "(i32, i32x4) -> f32, b8 baldrdash"); // Test the offset computation algorithm. assert_eq!(sig.argument_bytes, None); @@ -450,7 +417,7 @@ mod tests { // Writing ABI-annotated signatures. assert_eq!( sig.to_string(), - "(i32 [24], i32x4 [8]) -> f32, b8 spiderwasm" + "(i32 [24], i32x4 [8]) -> f32, b8 baldrdash" ); } } diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs index 5e70ff3780..10350c506c 100644 --- a/lib/codegen/src/ir/function.rs +++ b/lib/codegen/src/ir/function.rs @@ -6,11 +6,12 @@ use binemit::CodeOffset; use entity::{EntityMap, PrimaryMap}; use ir; -use ir::{CallConv, DataFlowGraph, ExternalName, Layout, Signature}; +use ir::{DataFlowGraph, ExternalName, Layout, Signature}; use ir::{Ebb, ExtFuncData, FuncRef, GlobalVar, GlobalVarData, Heap, HeapData, JumpTable, JumpTableData, SigRef, StackSlot, StackSlotData}; use ir::{EbbOffsets, InstEncodings, JumpTables, SourceLocs, StackSlots, ValueLocations}; use isa::{EncInfo, Legalize, TargetIsa, Encoding}; +use settings::CallConv; use std::fmt; use write::write_function; @@ -86,7 +87,7 @@ impl Function { /// Clear all data structures in this function. pub fn clear(&mut self) { - self.signature.clear(ir::CallConv::SystemV); + self.signature.clear(CallConv::Fast); self.stack_slots.clear(); self.global_vars.clear(); self.heaps.clear(); @@ -99,9 +100,9 @@ impl Function { self.srclocs.clear(); } - /// Create a new empty, anonymous function with a SystemV calling convention. + /// Create a new empty, anonymous function with a Fast calling convention. pub fn new() -> Self { - Self::with_name_signature(ExternalName::default(), Signature::new(CallConv::SystemV)) + Self::with_name_signature(ExternalName::default(), Signature::new(CallConv::Fast)) } /// Creates a jump table in the function, to be used by `br_table` instructions. diff --git a/lib/codegen/src/ir/mod.rs b/lib/codegen/src/ir/mod.rs index ed9771ccad..ab14a305aa 100644 --- a/lib/codegen/src/ir/mod.rs +++ b/lib/codegen/src/ir/mod.rs @@ -25,8 +25,7 @@ mod valueloc; pub use ir::builder::{InsertBuilder, InstBuilder, InstBuilderBase, InstInserterBase}; pub use ir::dfg::{DataFlowGraph, ValueDef}; pub use ir::entities::{Ebb, FuncRef, GlobalVar, Heap, Inst, JumpTable, SigRef, StackSlot, Value}; -pub use ir::extfunc::{AbiParam, ArgumentExtension, ArgumentPurpose, CallConv, ExtFuncData, - Signature}; +pub use ir::extfunc::{AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData, Signature}; pub use ir::extname::ExternalName; pub use ir::function::Function; pub use ir::globalvar::GlobalVarData; diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs index a8d8616858..ed626a60a2 100644 --- a/lib/codegen/src/isa/mod.rs +++ b/lib/codegen/src/isa/mod.rs @@ -52,6 +52,7 @@ use isa::enc_tables::Encodings; use regalloc; use result; use settings; +use settings::CallConv; use std::boxed::Box; use std::fmt; use timing; @@ -252,8 +253,8 @@ pub trait TargetIsa: fmt::Display { let word_size = if self.flags().is_64bit() { 8 } else { 4 }; // Account for the SpiderMonkey standard prologue pushes. - if func.signature.call_conv == ir::CallConv::SpiderWASM { - let bytes = StackSize::from(self.flags().spiderwasm_prologue_words()) * word_size; + if func.signature.call_conv == CallConv::Baldrdash { + let bytes = StackSize::from(self.flags().baldrdash_prologue_words()) * word_size; let mut ss = ir::StackSlotData::new(ir::StackSlotKind::IncomingArg, bytes); ss.offset = Some(-(bytes as StackOffset)); func.stack_slots.push(ss); diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs index fe64aed5b1..0e4a43ee57 100644 --- a/lib/codegen/src/isa/x86/abi.rs +++ b/lib/codegen/src/isa/x86/abi.rs @@ -6,12 +6,12 @@ use cursor::{Cursor, CursorPosition, EncCursor}; use ir; use ir::immediates::Imm64; use ir::stackslot::{StackOffset, StackSize}; -use ir::{AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, CallConv, InstBuilder, - ValueLoc}; +use ir::{AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, InstBuilder, ValueLoc}; use isa::{RegClass, RegUnit, TargetIsa}; use regalloc::RegisterSet; use result; use settings as shared_settings; +use settings::CallConv; use stack_layout::layout_stack; use std::i32; @@ -74,7 +74,7 @@ impl ArgAssigner for Args { } // Handle special-purpose arguments. - if ty.is_int() && self.call_conv == CallConv::SpiderWASM { + if ty.is_int() && self.call_conv == CallConv::Baldrdash { match arg.purpose { // This is SpiderMonkey's `WasmTlsReg`. ArgumentPurpose::VMContext => { @@ -210,19 +210,20 @@ fn callee_saved_gprs_used(flags: &shared_settings::Flags, func: &ir::Function) - pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { match func.signature.call_conv { - ir::CallConv::SystemV => system_v_prologue_epilogue(func, isa), - ir::CallConv::SpiderWASM => spiderwasm_prologue_epilogue(func, isa), + // For now, just translate fast and cold as system_v. + CallConv::Fast | CallConv::Cold | CallConv::SystemV => { + system_v_prologue_epilogue(func, isa) + } + CallConv::Fastcall => unimplemented!("Windows calling conventions"), + CallConv::Baldrdash => baldrdash_prologue_epilogue(func, isa), } } -pub fn spiderwasm_prologue_epilogue( - func: &mut ir::Function, - isa: &TargetIsa, -) -> result::CtonResult { - // Spiderwasm on 32-bit x86 always aligns its stack pointer to 16 bytes. +pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { + // Baldrdash on 32-bit x86 always aligns its stack pointer to 16 bytes. let stack_align = 16; let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; - let bytes = StackSize::from(isa.flags().spiderwasm_prologue_words()) * word_size; + let bytes = StackSize::from(isa.flags().baldrdash_prologue_words()) * word_size; let mut ss = ir::StackSlotData::new(ir::StackSlotKind::IncomingArg, bytes); ss.offset = Some(-(bytes as StackOffset)); diff --git a/lib/codegen/src/legalizer/libcall.rs b/lib/codegen/src/legalizer/libcall.rs index 17f9174706..7d058f24bb 100644 --- a/lib/codegen/src/legalizer/libcall.rs +++ b/lib/codegen/src/legalizer/libcall.rs @@ -3,9 +3,10 @@ use ir; use ir::InstBuilder; use std::vec::Vec; +use isa::TargetIsa; /// Try to expand `inst` as a library call, returning true is successful. -pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function) -> bool { +pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function, isa: &TargetIsa) -> bool { // Does the opcode/ctrl_type combo even have a well-known runtime library name. let libcall = match ir::LibCall::for_inst(func.dfg[inst].opcode(), func.dfg.ctrl_typevar(inst)) { @@ -13,7 +14,8 @@ pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function) -> bool { None => return false, }; - let funcref = find_funcref(libcall, func).unwrap_or_else(|| make_funcref(libcall, inst, func)); + let funcref = + find_funcref(libcall, func).unwrap_or_else(|| make_funcref(libcall, inst, func, isa)); // Now we convert `inst` to a call. First save the arguments. let mut args = Vec::new(); @@ -44,9 +46,14 @@ fn find_funcref(libcall: ir::LibCall, func: &ir::Function) -> Option ir::FuncRef { - // Start with a system_v calling convention. We'll give the ISA a chance to change it. - let mut sig = ir::Signature::new(ir::CallConv::SystemV); +fn make_funcref( + libcall: ir::LibCall, + inst: ir::Inst, + func: &mut ir::Function, + isa: &TargetIsa, +) -> ir::FuncRef { + // Start with a fast calling convention. We'll give the ISA a chance to change it. + let mut sig = ir::Signature::new(isa.flags().call_conv()); for &v in func.dfg.inst_args(inst) { sig.params.push(ir::AbiParam::new(func.dfg.value_type(v))); } diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs index f4f9389390..019b40c413 100644 --- a/lib/codegen/src/legalizer/mod.rs +++ b/lib/codegen/src/legalizer/mod.rs @@ -91,7 +91,7 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is // We don't have any pattern expansion for this instruction either. // Try converting it to a library call as a last resort. - if expand_as_libcall(inst, pos.func) { + if expand_as_libcall(inst, pos.func, isa) { pos.set_position(prev_pos); continue; } diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs index f4edfc73cf..e34ee1421f 100644 --- a/lib/codegen/src/settings.rs +++ b/lib/codegen/src/settings.rs @@ -25,6 +25,7 @@ use isa::TargetIsa; use std::fmt; use std::result; use std::vec::Vec; +use std::str; /// A string-based configurator for settings groups. /// @@ -360,6 +361,7 @@ mod tests { opt_level = \"default\"\n\ enable_verifier = true\n\ is_64bit = false\n\ + call_conv = \"fast\"\n\ is_pic = false\n\ return_at_end = false\n\ avoid_div_traps = false\n\ @@ -367,12 +369,12 @@ mod tests { enable_float = true\n\ enable_simd = true\n\ enable_atomics = true\n\ - spiderwasm_prologue_words = 0\n\ + baldrdash_prologue_words = 0\n\ allones_funcaddrs = false\n" ); assert_eq!(f.opt_level(), super::OptLevel::Default); assert_eq!(f.enable_simd(), true); - assert_eq!(f.spiderwasm_prologue_words(), 0); + assert_eq!(f.baldrdash_prologue_words(), 0); } #[test] diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs index a8f12d024f..b3887949c2 100644 --- a/lib/codegen/src/write.rs +++ b/lib/codegen/src/write.rs @@ -459,34 +459,34 @@ mod tests { #[test] fn basic() { let mut f = Function::new(); - assert_eq!(f.to_string(), "function u0:0() system_v {\n}\n"); + assert_eq!(f.to_string(), "function u0:0() fast {\n}\n"); f.name = ExternalName::testcase("foo"); - assert_eq!(f.to_string(), "function %foo() system_v {\n}\n"); + assert_eq!(f.to_string(), "function %foo() fast {\n}\n"); f.create_stack_slot(StackSlotData::new(StackSlotKind::ExplicitSlot, 4)); assert_eq!( f.to_string(), - "function %foo() system_v {\n ss0 = explicit_slot 4\n}\n" + "function %foo() fast {\n ss0 = explicit_slot 4\n}\n" ); let ebb = f.dfg.make_ebb(); f.layout.append_ebb(ebb); assert_eq!( f.to_string(), - "function %foo() system_v {\n ss0 = explicit_slot 4\n\nebb0:\n}\n" + "function %foo() fast {\n ss0 = explicit_slot 4\n\nebb0:\n}\n" ); f.dfg.append_ebb_param(ebb, types::I8); assert_eq!( f.to_string(), - "function %foo() system_v {\n ss0 = explicit_slot 4\n\nebb0(v0: i8):\n}\n" + "function %foo() fast {\n ss0 = explicit_slot 4\n\nebb0(v0: i8):\n}\n" ); f.dfg.append_ebb_param(ebb, types::F32.by(4).unwrap()); assert_eq!( f.to_string(), - "function %foo() system_v {\n ss0 = explicit_slot 4\n\nebb0(v0: i8, v1: f32x4):\n}\n" + "function %foo() fast {\n ss0 = explicit_slot 4\n\nebb0(v0: i8, v1: f32x4):\n}\n" ); } } diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index 55af088295..4835156637 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -23,6 +23,10 @@ impl FaerieBuilder { /// Create a new `FaerieBuilder` using the given Cretonne target, that /// can be passed to /// [`Module::new`](cretonne_module/struct.Module.html#method.new]. + /// + /// Note: To support calls JIT'd functions from Rust or other compiled + /// code, it's necessary for the `call_conv` setting in `isa`'s flags + /// to match the host platform. pub fn new( isa: Box, name: String, diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 70d2d43820..1f2c818f99 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -596,8 +596,9 @@ mod tests { use Variable; use cretonne_codegen::entity::EntityRef; use cretonne_codegen::ir::types::*; - use cretonne_codegen::ir::{AbiParam, CallConv, ExternalName, Function, InstBuilder, Signature}; + use cretonne_codegen::ir::{AbiParam, ExternalName, Function, InstBuilder, Signature}; use cretonne_codegen::settings; + use cretonne_codegen::settings::CallConv; use cretonne_codegen::verifier::verify_function; use frontend::{FunctionBuilder, FunctionBuilderContext}; diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 494fecb414..65d76fd279 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -36,9 +36,9 @@ //! extern crate cretonne_frontend; //! //! use cretonne_codegen::entity::EntityRef; -//! use cretonne_codegen::ir::{ExternalName, CallConv, Function, Signature, AbiParam, InstBuilder}; +//! use cretonne_codegen::ir::{ExternalName, Function, Signature, AbiParam, InstBuilder}; //! use cretonne_codegen::ir::types::*; -//! use cretonne_codegen::settings; +//! use cretonne_codegen::settings::{self, CallConv}; //! use cretonne_frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; //! use cretonne_codegen::verifier::verify_function; //! diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index e84be550e2..ff3d0632c2 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -34,12 +34,21 @@ use raw_cpuid::CpuId; pub fn builders() -> Result<(settings::Builder, isa::Builder), &'static str> { let mut flag_builder = settings::builder(); - // TODO: Add RISC-V support once Rust supports it. + if cfg!(unix) { + flag_builder.set("call_conv", "system_v").unwrap(); + } else if cfg!(windows) { + flag_builder.set("call_conv", "fastcall").unwrap(); + } else { + return Err("unrecognized environment"); + } if cfg!(target_pointer_width = "64") { flag_builder.enable("is_64bit").unwrap(); + } else if !cfg!(target_pointer_width = "32") { + return Err("unrecognized pointer size"); } + // TODO: Add RISC-V support once Rust supports it. let name = if cfg!(any(target_arch = "x86", target_arch = "x86_64")) { "x86" } else if cfg!(target_arch = "arm") { diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index e96e415afc..8ee584075c 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -6,7 +6,7 @@ use cretonne_codegen::ir::entities::AnyEntity; use cretonne_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32}; use cretonne_codegen::ir::instructions::{InstructionData, InstructionFormat, VariableArgs}; use cretonne_codegen::ir::types::VOID; -use cretonne_codegen::ir::{AbiParam, ArgumentExtension, ArgumentLoc, CallConv, Ebb, ExtFuncData, +use cretonne_codegen::ir::{AbiParam, ArgumentExtension, ArgumentLoc, Ebb, ExtFuncData, ExternalName, FuncRef, Function, GlobalVar, GlobalVarData, Heap, HeapBase, HeapData, HeapStyle, JumpTable, JumpTableData, MemFlags, Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind, @@ -14,6 +14,7 @@ use cretonne_codegen::ir::{AbiParam, ArgumentExtension, ArgumentLoc, CallConv, E use cretonne_codegen::isa::{self, Encoding, RegUnit, TargetIsa}; use cretonne_codegen::packed_option::ReservedValue; use cretonne_codegen::{settings, timing}; +use cretonne_codegen::settings::CallConv; use error::{Error, Location, Result}; use isaspec; use lexer::{self, Lexer, Token}; @@ -188,7 +189,7 @@ impl<'a> Context<'a> { fn add_sig(&mut self, sig: SigRef, data: Signature, loc: &Location) -> Result<()> { while self.function.dfg.signatures.next_key().index() <= sig.index() { self.function.import_signature( - Signature::new(CallConv::SystemV), + Signature::new(CallConv::Fast), ); } self.function.dfg.signatures[sig] = data; @@ -862,8 +863,8 @@ impl<'a> Parser<'a> { // signature ::= * "(" [paramlist] ")" ["->" retlist] [callconv] // fn parse_signature(&mut self, unique_isa: Option<&TargetIsa>) -> Result { - // Calling convention defaults to `system_v`, but can be changed. - let mut sig = Signature::new(CallConv::SystemV); + // Calling convention defaults to `fast`, but can be changed. + let mut sig = Signature::new(CallConv::Fast); self.match_token( Token::LPar, @@ -2402,7 +2403,8 @@ mod tests { use cretonne_codegen::ir::StackSlotKind; use cretonne_codegen::ir::entities::AnyEntity; use cretonne_codegen::ir::types; - use cretonne_codegen::ir::{ArgumentExtension, ArgumentPurpose, CallConv}; + use cretonne_codegen::ir::{ArgumentExtension, ArgumentPurpose}; + use cretonne_codegen::settings::CallConv; use error::Error; use isaspec::IsaSpec; use testfile::{Comment, Details}; @@ -2451,19 +2453,19 @@ mod tests { assert_eq!(sig.returns.len(), 0); assert_eq!(sig.call_conv, CallConv::SystemV); - let sig2 = Parser::new("(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 spiderwasm") + let sig2 = Parser::new("(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 baldrdash") .parse_signature(None) .unwrap(); assert_eq!( sig2.to_string(), - "(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 spiderwasm" + "(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 baldrdash" ); - assert_eq!(sig2.call_conv, CallConv::SpiderWASM); + assert_eq!(sig2.call_conv, CallConv::Baldrdash); // Old-style signature without a calling convention. assert_eq!( Parser::new("()").parse_signature(None).unwrap().to_string(), - "() system_v" + "() fast" ); assert_eq!( Parser::new("() notacc") diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs index de89cd5788..f28fb45547 100644 --- a/lib/umbrella/src/lib.rs +++ b/lib/umbrella/src/lib.rs @@ -25,12 +25,12 @@ pub extern crate cretonne_frontend as frontend; pub mod prelude { pub use codegen; pub use codegen::entity::EntityRef; - pub use codegen::ir::{AbiParam, InstBuilder, Value, Ebb, Signature, CallConv, Type, - JumpTableData, MemFlags}; + pub use codegen::ir::{AbiParam, InstBuilder, Value, Ebb, Signature, Type, JumpTableData, + MemFlags}; pub use codegen::ir::types; pub use codegen::ir::condcodes::{IntCC, FloatCC}; pub use codegen::ir::immediates::{Ieee32, Ieee64}; - pub use codegen::settings::{self, Configurable}; + pub use codegen::settings::{self, Configurable, CallConv}; pub use codegen::isa; pub use frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index e04bd5f180..2b1fbfbfc7 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -271,6 +271,10 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ } impl<'data> ModuleEnvironment<'data> for DummyEnvironment { + fn flags(&self) -> &settings::Flags { + &self.info.flags + } + fn get_func_name(&self, func_index: FunctionIndex) -> ir::ExternalName { get_func_name(func_index) } diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 51aba3fd7e..cc9e4eb771 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -160,6 +160,9 @@ pub trait FuncEnvironment { /// [`translate_module`](fn.translate_module.html) function. These methods should not be called /// by the user, they are only for `cretonne-wasm` internal use. pub trait ModuleEnvironment<'data> { + /// Get the flags for the current compilation. + fn flags(&self) -> &Flags; + /// Return the name for the given function index. fn get_func_name(&self, func_index: FunctionIndex) -> ir::ExternalName; diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index c243348d5a..a9ca07a8df 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -7,7 +7,7 @@ //! The special case of the initialize expressions for table elements offsets or global variables //! is handled, according to the semantics of WebAssembly, to only specific expressions that are //! interpreted on the fly. -use cretonne_codegen::ir::{self, AbiParam, CallConv, Signature}; +use cretonne_codegen::ir::{self, AbiParam, Signature}; use environ::ModuleEnvironment; use std::str::from_utf8; use std::string::String; @@ -35,7 +35,7 @@ pub fn parse_function_signatures( ref params, ref returns, }) => { - let mut sig = Signature::new(CallConv::SystemV); + let mut sig = Signature::new(environ.flags().call_conv()); sig.params.extend(params.iter().map(|ty| { let cret_arg: ir::Type = type_to_type(ty).expect( "only numeric types are supported in function signatures", From 3b1d805758dfdc9c8a3c6e6dfc3c00572365d022 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Apr 2018 21:41:45 -0700 Subject: [PATCH 1741/3084] Stack overflow checking with stack probes. This adds a libcall name, a calling convention, and settings for emitting stack probes, and implements them for x86 system_v ABIs. --- cranelift/filetests/isa/x86/binary32.cton | 32 ++++-- cranelift/filetests/isa/x86/binary64.cton | 32 ++++-- .../isa/x86/probestack-adjusts-sp.cton | 29 +++++ .../isa/x86/probestack-disabled.cton | 25 +++++ .../isa/x86/probestack-noncolocated.cton | 28 +++++ .../filetests/isa/x86/probestack-size.cton | 75 +++++++++++++ cranelift/filetests/isa/x86/probestack.cton | 50 +++++++++ .../filetests/isa/x86/prologue-epilogue.cton | 12 +-- cranelift/filetests/postopt/basic.cton | 10 +- .../filetests/regalloc/coloring-227.cton | 34 +++--- .../filetests/regalloc/unreachable_code.cton | 1 + lib/codegen/meta/base/instructions.py | 30 +++++- lib/codegen/meta/base/settings.py | 39 ++++++- lib/codegen/meta/isa/x86/encodings.py | 50 +++++---- lib/codegen/meta/isa/x86/recipes.py | 55 +++++----- lib/codegen/src/ir/libcall.rs | 102 +++++++++++++++++- lib/codegen/src/ir/mod.rs | 2 +- lib/codegen/src/isa/x86/abi.rs | 80 +++++++++++--- lib/codegen/src/legalizer/libcall.rs | 48 +-------- lib/codegen/src/settings.rs | 6 +- 20 files changed, 585 insertions(+), 155 deletions(-) create mode 100644 cranelift/filetests/isa/x86/probestack-adjusts-sp.cton create mode 100644 cranelift/filetests/isa/x86/probestack-disabled.cton create mode 100644 cranelift/filetests/isa/x86/probestack-noncolocated.cton create mode 100644 cranelift/filetests/isa/x86/probestack-size.cton create mode 100644 cranelift/filetests/isa/x86/probestack.cton diff --git a/cranelift/filetests/isa/x86/binary32.cton b/cranelift/filetests/isa/x86/binary32.cton index 6eb4565f79..9ce08fb66d 100644 --- a/cranelift/filetests/isa/x86/binary32.cton +++ b/cranelift/filetests/isa/x86/binary32.cton @@ -392,19 +392,37 @@ ebb0: ; asm: popl %ecx [-,%rcx] v512 = x86_pop.i32 ; bin: 59 - ; Adjust Stack Pointer + ; Adjust Stack Pointer Up ; asm: addl $64, %esp - adjust_sp_imm 64 ; bin: 83 c4 40 + adjust_sp_up_imm 64 ; bin: 83 c4 40 ; asm: addl $-64, %esp - adjust_sp_imm -64 ; bin: 83 c4 c0 + adjust_sp_up_imm -64 ; bin: 83 c4 c0 ; asm: addl $1024, %esp - adjust_sp_imm 1024 ; bin: 81 c4 00000400 + adjust_sp_up_imm 1024 ; bin: 81 c4 00000400 ; asm: addl $-1024, %esp - adjust_sp_imm -1024 ; bin: 81 c4 fffffc00 + adjust_sp_up_imm -1024 ; bin: 81 c4 fffffc00 ; asm: addl $2147483647, %esp - adjust_sp_imm 2147483647 ; bin: 81 c4 7fffffff + adjust_sp_up_imm 2147483647 ; bin: 81 c4 7fffffff ; asm: addl $-2147483648, %esp - adjust_sp_imm -2147483648 ; bin: 81 c4 80000000 + adjust_sp_up_imm -2147483648 ; bin: 81 c4 80000000 + + ; Adjust Stack Pointer Down + ; asm: subl %ecx, %esp + adjust_sp_down v1 ; bin: 29 cc + ; asm: subl %esi, %esp + adjust_sp_down v2 ; bin: 29 f4 + ; asm: addl $64, %esp + adjust_sp_down_imm 64 ; bin: 83 ec 40 + ; asm: addl $-64, %esp + adjust_sp_down_imm -64 ; bin: 83 ec c0 + ; asm: addl $1024, %esp + adjust_sp_down_imm 1024 ; bin: 81 ec 00000400 + ; asm: addl $-1024, %esp + adjust_sp_down_imm -1024 ; bin: 81 ec fffffc00 + ; asm: addl $2147483647, %esp + adjust_sp_down_imm 2147483647 ; bin: 81 ec 7fffffff + ; asm: addl $-2147483648, %esp + adjust_sp_down_imm -2147483648 ; bin: 81 ec 80000000 ; Shift immediates ; asm: shll $2, %esi diff --git a/cranelift/filetests/isa/x86/binary64.cton b/cranelift/filetests/isa/x86/binary64.cton index 6417cd26e2..a9b96a86dc 100644 --- a/cranelift/filetests/isa/x86/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.cton @@ -547,19 +547,37 @@ ebb0: ; asm: popq %r10 [-,%r10] v514 = x86_pop.i64 ; bin: 41 5a - ; Adjust Stack Pointer + ; Adjust Stack Pointer Up ; asm: addq $64, %rsp - adjust_sp_imm 64 ; bin: 48 83 c4 40 + adjust_sp_up_imm 64 ; bin: 48 83 c4 40 ; asm: addq $-64, %rsp - adjust_sp_imm -64 ; bin: 48 83 c4 c0 + adjust_sp_up_imm -64 ; bin: 48 83 c4 c0 ; asm: addq $1024, %rsp - adjust_sp_imm 1024 ; bin: 48 81 c4 00000400 + adjust_sp_up_imm 1024 ; bin: 48 81 c4 00000400 ; asm: addq $-1024, %rsp - adjust_sp_imm -1024 ; bin: 48 81 c4 fffffc00 + adjust_sp_up_imm -1024 ; bin: 48 81 c4 fffffc00 ; asm: addq $2147483647, %rsp - adjust_sp_imm 2147483647 ; bin: 48 81 c4 7fffffff + adjust_sp_up_imm 2147483647 ; bin: 48 81 c4 7fffffff ; asm: addq $-2147483648, %rsp - adjust_sp_imm -2147483648 ; bin: 48 81 c4 80000000 + adjust_sp_up_imm -2147483648 ; bin: 48 81 c4 80000000 + + ; Adjust Stack Pointer Down + ; asm: subq %rcx, %rsp + adjust_sp_down v1 ; bin: 48 29 cc + ; asm: subq %r10, %rsp + adjust_sp_down v3 ; bin: 4c 29 d4 + ; asm: subq $64, %rsp + adjust_sp_down_imm 64 ; bin: 48 83 ec 40 + ; asm: subq $-64, %rsp + adjust_sp_down_imm -64 ; bin: 48 83 ec c0 + ; asm: subq $1024, %rsp + adjust_sp_down_imm 1024 ; bin: 48 81 ec 00000400 + ; asm: subq $-1024, %rsp + adjust_sp_down_imm -1024 ; bin: 48 81 ec fffffc00 + ; asm: subq $2147483647, %rsp + adjust_sp_down_imm 2147483647 ; bin: 48 81 ec 7fffffff + ; asm: subq $-2147483648, %rsp + adjust_sp_down_imm -2147483648 ; bin: 48 81 ec 80000000 ; Shift immediates ; asm: shlq $12, %rsi diff --git a/cranelift/filetests/isa/x86/probestack-adjusts-sp.cton b/cranelift/filetests/isa/x86/probestack-adjusts-sp.cton new file mode 100644 index 0000000000..2204886046 --- /dev/null +++ b/cranelift/filetests/isa/x86/probestack-adjusts-sp.cton @@ -0,0 +1,29 @@ +test compile +set is_64bit=1 +set colocated_libcalls=1 +set probestack_func_adjusts_sp=1 +isa x86 + +; Like %big in probestack.cton, but with the probestack function adjusting +; the stack pointer itself. + +function %big() system_v { + ss0 = explicit_slot 300000 +ebb0: + return +} +; check: function %big(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; nextln: ss0 = explicit_slot 300000, offset -300016 +; nextln: ss1 = incoming_arg 16, offset -16 +; nextln: sig0 = (i64 [%rax]) probestack +; nextln: fn0 = colocated %Probestack sig0 +; nextln: +; nextln: ebb0(v0: i64 [%rbp]): +; nextln: [RexOp1pushq#50] x86_push v0 +; nextln: [RexOp1copysp#8089] copy_special %rsp -> %rbp +; nextln: [RexOp1pu_id#b8,%rax] v1 = iconst.i64 0x0004_93e0 +; nextln: [Op1call_id#e8] call fn0(v1) +; nextln: [RexOp1adjustsp_id#8081] adjust_sp_up_imm 0x0004_93e0 +; nextln: [RexOp1popq#58,%rbp] v2 = x86_pop.i64 +; nextln: [Op1ret#c3] return v2 +; nextln: } diff --git a/cranelift/filetests/isa/x86/probestack-disabled.cton b/cranelift/filetests/isa/x86/probestack-disabled.cton new file mode 100644 index 0000000000..282e088a3a --- /dev/null +++ b/cranelift/filetests/isa/x86/probestack-disabled.cton @@ -0,0 +1,25 @@ +test compile +set is_64bit=1 +set colocated_libcalls=1 +set probestack_enabled=0 +isa x86 + +; Like %big in probestack.cton, but with probes disabled. + +function %big() system_v { + ss0 = explicit_slot 300000 +ebb0: + return +} +; check: function %big(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; nextln: ss0 = explicit_slot 300000, offset -300016 +; nextln: ss1 = incoming_arg 16, offset -16 +; nextln: +; nextln: ebb0(v0: i64 [%rbp]): +; nextln: [RexOp1pushq#50] x86_push v0 +; nextln: [RexOp1copysp#8089] copy_special %rsp -> %rbp +; nextln: [RexOp1adjustsp_id#d081] adjust_sp_down_imm 0x0004_93e0 +; nextln: [RexOp1adjustsp_id#8081] adjust_sp_up_imm 0x0004_93e0 +; nextln: [RexOp1popq#58,%rbp] v1 = x86_pop.i64 +; nextln: [Op1ret#c3] return v1 +; nextln: } diff --git a/cranelift/filetests/isa/x86/probestack-noncolocated.cton b/cranelift/filetests/isa/x86/probestack-noncolocated.cton new file mode 100644 index 0000000000..3248f4a142 --- /dev/null +++ b/cranelift/filetests/isa/x86/probestack-noncolocated.cton @@ -0,0 +1,28 @@ +test compile +set is_64bit=1 +isa x86 + +; Like %big in probestack.cton, but without a colocated libcall. + +function %big() system_v { + ss0 = explicit_slot 300000 +ebb0: + return +} +; check: function %big(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; nextln: ss0 = explicit_slot 300000, offset -300016 +; nextln: ss1 = incoming_arg 16, offset -16 +; nextln: sig0 = (i64 [%rax]) -> i64 [%rax] probestack +; nextln: fn0 = %Probestack sig0 +; nextln: +; nextln: ebb0(v0: i64 [%rbp]): +; nextln: [RexOp1pushq#50] x86_push v0 +; nextln: [RexOp1copysp#8089] copy_special %rsp -> %rbp +; nextln: [RexOp1pu_id#b8,%rax] v1 = iconst.i64 0x0004_93e0 +; nextln: [RexOp1fnaddr8#80b8,%r11] v2 = func_addr.i64 fn0 +; nextln: [RexOp1call_r#20ff,%rax] v3 = call_indirect sig0, v2(v1) +; nextln: [RexOp1adjustsp#8029] adjust_sp_down v3 +; nextln: [RexOp1adjustsp_id#8081] adjust_sp_up_imm 0x0004_93e0 +; nextln: [RexOp1popq#58,%rbp] v4 = x86_pop.i64 +; nextln: [Op1ret#c3] return v4 +; nextln: } diff --git a/cranelift/filetests/isa/x86/probestack-size.cton b/cranelift/filetests/isa/x86/probestack-size.cton new file mode 100644 index 0000000000..d7c92a5aa5 --- /dev/null +++ b/cranelift/filetests/isa/x86/probestack-size.cton @@ -0,0 +1,75 @@ +test compile +set is_64bit=1 +set colocated_libcalls=1 +set probestack_size_log2=13 +isa x86 + +; Like %big in probestack.cton, but now the probestack size is bigger +; and it no longer needs a probe. + +function %big() system_v { + ss0 = explicit_slot 4097 +ebb0: + return +} + +; check: function %big(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; nextln: ss0 = explicit_slot 4097, offset -4113 +; nextln: ss1 = incoming_arg 16, offset -16 +; nextln: +; nextln: ebb0(v0: i64 [%rbp]): +; nextln: [RexOp1pushq#50] x86_push v0 +; nextln: [RexOp1copysp#8089] copy_special %rsp -> %rbp +; nextln: [RexOp1adjustsp_id#d081] adjust_sp_down_imm 4112 +; nextln: [RexOp1adjustsp_id#8081] adjust_sp_up_imm 4112 +; nextln: [RexOp1popq#58,%rbp] v1 = x86_pop.i64 +; nextln: [Op1ret#c3] return v1 +; nextln: } + + +; Like %big; still doesn't need a probe. + +function %bigger() system_v { + ss0 = explicit_slot 8192 +ebb0: + return +} + +; check: function %bigger(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; nextln: ss0 = explicit_slot 8192, offset -8208 +; nextln: ss1 = incoming_arg 16, offset -16 +; nextln: +; nextln: ebb0(v0: i64 [%rbp]): +; nextln: [RexOp1pushq#50] x86_push v0 +; nextln: [RexOp1copysp#8089] copy_special %rsp -> %rbp +; nextln: [RexOp1adjustsp_id#d081] adjust_sp_down_imm 8192 +; nextln: [RexOp1adjustsp_id#8081] adjust_sp_up_imm 8192 +; nextln: [RexOp1popq#58,%rbp] v1 = x86_pop.i64 +; nextln: [Op1ret#c3] return v1 +; nextln: } + + +; Like %bigger; this needs a probe. + +function %biggest() system_v { + ss0 = explicit_slot 8193 +ebb0: + return +} + +; check: function %biggest(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; nextln: ss0 = explicit_slot 8193, offset -8209 +; nextln: ss1 = incoming_arg 16, offset -16 +; nextln: sig0 = (i64 [%rax]) -> i64 [%rax] probestack +; nextln: fn0 = colocated %Probestack sig0 +; nextln: +; nextln: ebb0(v0: i64 [%rbp]): +; nextln: [RexOp1pushq#50] x86_push v0 +; nextln: [RexOp1copysp#8089] copy_special %rsp -> %rbp +; nextln: [RexOp1pu_id#b8,%rax] v1 = iconst.i64 8208 +; nextln: [Op1call_id#e8,%rax] v2 = call fn0(v1) +; nextln: [RexOp1adjustsp#8029] adjust_sp_down v2 +; nextln: [RexOp1adjustsp_id#8081] adjust_sp_up_imm 8208 +; nextln: [RexOp1popq#58,%rbp] v3 = x86_pop.i64 +; nextln: [Op1ret#c3] return v3 +; nextln: } diff --git a/cranelift/filetests/isa/x86/probestack.cton b/cranelift/filetests/isa/x86/probestack.cton new file mode 100644 index 0000000000..8b961da3bd --- /dev/null +++ b/cranelift/filetests/isa/x86/probestack.cton @@ -0,0 +1,50 @@ +test compile +set is_64bit=1 +set colocated_libcalls=1 +isa x86 + +; A function with a big stack frame. This should have a stack probe. + +function %big() system_v { + ss0 = explicit_slot 4097 +ebb0: + return +} +; check: function %big(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; nextln: ss0 = explicit_slot 4097, offset -4113 +; nextln: ss1 = incoming_arg 16, offset -16 +; nextln: sig0 = (i64 [%rax]) -> i64 [%rax] probestack +; nextln: fn0 = colocated %Probestack sig0 +; nextln: +; nextln: ebb0(v0: i64 [%rbp]): +; nextln: [RexOp1pushq#50] x86_push v0 +; nextln: [RexOp1copysp#8089] copy_special %rsp -> %rbp +; nextln: [RexOp1pu_id#b8,%rax] v1 = iconst.i64 4112 +; nextln: [Op1call_id#e8,%rax] v2 = call fn0(v1) +; nextln: [RexOp1adjustsp#8029] adjust_sp_down v2 +; nextln: [RexOp1adjustsp_id#8081] adjust_sp_up_imm 4112 +; nextln: [RexOp1popq#58,%rbp] v3 = x86_pop.i64 +; nextln: [Op1ret#c3] return v3 +; nextln: } + + +; A function with a small enough stack frame. This shouldn't have a stack probe. + +function %small() system_v { + ss0 = explicit_slot 4096 +ebb0: + return +} + +; check: function %small(i64 fp [%rbp]) -> i64 fp [%rbp] system_v { +; nextln: ss0 = explicit_slot 4096, offset -4112 +; nextln: ss1 = incoming_arg 16, offset -16 +; nextln: +; nextln: ebb0(v0: i64 [%rbp]): +; nextln: [RexOp1pushq#50] x86_push v0 +; nextln: [RexOp1copysp#8089] copy_special %rsp -> %rbp +; nextln: [RexOp1adjustsp_id#d081] adjust_sp_down_imm 4096 +; nextln: [RexOp1adjustsp_id#8081] adjust_sp_up_imm 4096 +; nextln: [RexOp1popq#58,%rbp] v1 = x86_pop.i64 +; nextln: [Op1ret#c3] return v1 +; nextln: } diff --git a/cranelift/filetests/isa/x86/prologue-epilogue.cton b/cranelift/filetests/isa/x86/prologue-epilogue.cton index bf4bf298b5..41d2147438 100644 --- a/cranelift/filetests/isa/x86/prologue-epilogue.cton +++ b/cranelift/filetests/isa/x86/prologue-epilogue.cton @@ -36,8 +36,8 @@ ebb0: ; nextln: ebb0(v0: i64 [%rbp]): ; nextln: x86_push v0 ; nextln: copy_special %rsp -> %rbp -; nextln: adjust_sp_imm -176 -; nextln: adjust_sp_imm 176 +; nextln: adjust_sp_down_imm 176 +; nextln: adjust_sp_up_imm 176 ; nextln: v1 = x86_pop.i64 ; nextln: return v1 ; nextln: } @@ -109,7 +109,7 @@ ebb0(v0: i64, v1: i64): ; nextln: x86_push v18 ; nextln: x86_push v19 ; nextln: x86_push v20 -; nextln: adjust_sp_imm -8 +; nextln: adjust_sp_down_imm 8 ; nextln: v2 = load.i32 v0 ; nextln: v3 = load.i32 v0+8 ; nextln: v4 = load.i32 v0+16 @@ -136,7 +136,7 @@ ebb0(v0: i64, v1: i64): ; nextln: store v12, v1+80 ; nextln: store v13, v1+88 ; nextln: store v14, v1+96 -; nextln: adjust_sp_imm 8 +; nextln: adjust_sp_up_imm 8 ; nextln: v26 = x86_pop.i64 ; nextln: v25 = x86_pop.i64 ; nextln: v24 = x86_pop.i64 @@ -192,13 +192,13 @@ ebb0(v0: i64, v1: i64): ; nextln: x86_push v51 ; nextln: x86_push v52 ; nextln: x86_push v53 -; nextln: adjust_sp_imm +; nextln: adjust_sp_down_imm ; check: spill ; check: fill -; check: adjust_sp_imm +; check: adjust_sp_up_imm ; nextln: v59 = x86_pop.i64 ; nextln: v58 = x86_pop.i64 ; nextln: v57 = x86_pop.i64 diff --git a/cranelift/filetests/postopt/basic.cton b/cranelift/filetests/postopt/basic.cton index 48b6b66007..678cecc27c 100644 --- a/cranelift/filetests/postopt/basic.cton +++ b/cranelift/filetests/postopt/basic.cton @@ -10,7 +10,7 @@ ebb0(v0: i32, v1: i32): [Op1ret#c3] return v1 ebb1: -[Op1puid#b8,%rax] v8 = iconst.i32 3 +[Op1pu_id#b8,%rax] v8 = iconst.i32 3 [Op1ret#c3] return v8 } ; sameln: function %br_icmp @@ -34,7 +34,7 @@ ebb0(v0: i32, v1: i32): [Op1ret#c3] return v1 ebb1: -[Op1puid#b8,%rax] v8 = iconst.i32 3 +[Op1pu_id#b8,%rax] v8 = iconst.i32 3 [Op1ret#c3] return v8 } ; sameln: function %br_icmp_inverse @@ -53,12 +53,12 @@ ebb1: function %br_icmp_imm(i32, i32) -> i32 { ebb0(v0: i32, v1: i32): -[Op1icsccib#7083] v2 = icmp_imm slt v0, 2 +[Op1icscc_ib#7083] v2 = icmp_imm slt v0, 2 [Op1t8jccd_long#84] brz v2, ebb1 [Op1ret#c3] return v1 ebb1: -[Op1puid#b8,%rax] v8 = iconst.i32 3 +[Op1pu_id#b8,%rax] v8 = iconst.i32 3 [Op1ret#c3] return v8 } ; sameln: function %br_icmp_imm @@ -82,7 +82,7 @@ ebb0(v0: f32, v1: f32): [Op1ret#c3] return v1 ebb1: -[Op1puid#b8,%rax] v18 = iconst.i32 0x40a8_0000 +[Op1pu_id#b8,%rax] v18 = iconst.i32 0x40a8_0000 [Mp2frurm#56e,%xmm0] v8 = bitcast.f32 v18 [Op1ret#c3] return v8 } diff --git a/cranelift/filetests/regalloc/coloring-227.cton b/cranelift/filetests/regalloc/coloring-227.cton index 8144bba62e..2b327633c7 100644 --- a/cranelift/filetests/regalloc/coloring-227.cton +++ b/cranelift/filetests/regalloc/coloring-227.cton @@ -7,8 +7,8 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) heap0 = static gv0, min 0, bound 0x0001_0000_0000, guard 0x8000_0000 ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i64): -@0001 [RexOp1puid#b8] v5 = iconst.i32 0 -@0003 [RexOp1puid#b8] v6 = iconst.i32 0 +@0001 [RexOp1pu_id#b8] v5 = iconst.i32 0 +@0003 [RexOp1pu_id#b8] v6 = iconst.i32 0 @0005 [RexOp1tjccb#74] brz v6, ebb10 @0007 [RexOp1jmpb#eb] jump ebb3(v5, v5, v5, v5, v5, v5, v0, v1, v2, v3) @@ -16,10 +16,10 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) @000b [RexOp1jmpb#eb] jump ebb6 ebb6: -@000d [RexOp1puid#b8] v8 = iconst.i32 0 +@000d [RexOp1pu_id#b8] v8 = iconst.i32 0 @000f [RexOp1tjccb#75] brnz v8, ebb5 -@0011 [RexOp1puid#b8] v9 = iconst.i32 0 -@0015 [RexOp1puid#b8] v11 = iconst.i32 0 +@0011 [RexOp1pu_id#b8] v9 = iconst.i32 0 +@0015 [RexOp1pu_id#b8] v11 = iconst.i32 0 @0017 [RexOp1icscc#39] v12 = icmp.i32 eq v15, v11 @0017 [RexOp2urm_noflags#4b6] v13 = bint.i32 v12 @001a [RexOp1rr#21] v14 = band v9, v13 @@ -28,11 +28,11 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) ebb7: @0020 [RexOp1tjccb#74] brz.i32 v17, ebb8 -@0022 [RexOp1puid#b8] v18 = iconst.i32 0 +@0022 [RexOp1pu_id#b8] v18 = iconst.i32 0 @0024 [RexOp1tjccb#74] brz v18, ebb9 -@0028 [RexOp1puid#b8] v21 = iconst.i32 0 +@0028 [RexOp1pu_id#b8] v21 = iconst.i32 0 @002a [RexOp1umr#89] v79 = uextend.i64 v5 -@002a [RexOp1rib#8083] v80 = iadd_imm.i64 v4, 0 +@002a [RexOp1r_ib#8083] v80 = iadd_imm.i64 v4, 0 @002a [RexOp1ld#808b] v81 = load.i64 v80 @002a [RexOp1rr#8001] v22 = iadd v81, v79 @002a [RexMp1st#189] istore16 v21, v22 @@ -42,8 +42,8 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) @002e [RexOp1jmpb#eb] jump ebb8 ebb8: -@0033 [RexOp1puid#b8] v27 = iconst.i32 3 -@0035 [RexOp1puid#b8] v28 = iconst.i32 4 +@0033 [RexOp1pu_id#b8] v27 = iconst.i32 3 +@0035 [RexOp1pu_id#b8] v28 = iconst.i32 4 @003b [RexOp1rr#09] v35 = bor.i32 v31, v13 @003c [RexOp1tjccb#75] brnz v35, ebb15(v27) @003c [RexOp1jmpb#eb] jump ebb15(v28) @@ -58,25 +58,25 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) @0045 [RexOp1jmpb#eb] jump ebb2(v40, v47, v54, v61, v68, v75) ebb10: -@0046 [RexOp1puid#b8] v43 = iconst.i32 0 +@0046 [RexOp1pu_id#b8] v43 = iconst.i32 0 @0048 [RexOp1jmpb#eb] jump ebb2(v43, v5, v0, v1, v2, v3) ebb2(v7: i32, v45: i32, v52: i32, v59: i32, v66: i32, v73: i32): -@004c [RexOp1puid#b8] v44 = iconst.i32 0 +@004c [RexOp1pu_id#b8] v44 = iconst.i32 0 @004e [RexOp1tjccb#74] brz v44, ebb12 -@0052 [RexOp1puid#b8] v50 = iconst.i32 11 +@0052 [RexOp1pu_id#b8] v50 = iconst.i32 11 @0054 [RexOp1tjccb#74] brz v50, ebb14 @0058 [RexOp1umr#89] v82 = uextend.i64 v52 -@0058 [RexOp1rib#8083] v83 = iadd_imm.i64 v4, 0 +@0058 [RexOp1r_ib#8083] v83 = iadd_imm.i64 v4, 0 @0058 [RexOp1ld#808b] v84 = load.i64 v83 @0058 [RexOp1rr#8001] v57 = iadd v84, v82 @0058 [RexOp1ld#8b] v58 = load.i32 v57 @005d [RexOp1umr#89] v85 = uextend.i64 v58 -@005d [RexOp1rib#8083] v86 = iadd_imm.i64 v4, 0 +@005d [RexOp1r_ib#8083] v86 = iadd_imm.i64 v4, 0 @005d [RexOp1ld#808b] v87 = load.i64 v86 @005d [RexOp1rr#8001] v64 = iadd v87, v85 @005d [RexOp1st#88] istore8 v59, v64 -@0060 [RexOp1puid#b8] v65 = iconst.i32 0 +@0060 [RexOp1pu_id#b8] v65 = iconst.i32 0 @0062 [RexOp1jmpb#eb] jump ebb13(v65) ebb14: @@ -84,7 +84,7 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) ebb13(v51: i32): @0066 [RexOp1umr#89] v88 = uextend.i64 v45 -@0066 [RexOp1rib#8083] v89 = iadd_imm.i64 v4, 0 +@0066 [RexOp1r_ib#8083] v89 = iadd_imm.i64 v4, 0 @0066 [RexOp1ld#808b] v90 = load.i64 v89 @0066 [RexOp1rr#8001] v71 = iadd v90, v88 @0066 [RexOp1st#89] store v51, v71 diff --git a/cranelift/filetests/regalloc/unreachable_code.cton b/cranelift/filetests/regalloc/unreachable_code.cton index 34d8c4c164..8bfffc22a2 100644 --- a/cranelift/filetests/regalloc/unreachable_code.cton +++ b/cranelift/filetests/regalloc/unreachable_code.cton @@ -2,6 +2,7 @@ test compile set is_64bit +set probestack_enabled=0 isa x86 haswell ; This function contains unreachable blocks which trip up the register diff --git a/lib/codegen/meta/base/instructions.py b/lib/codegen/meta/base/instructions.py index c54f3ef570..8fee97ae42 100644 --- a/lib/codegen/meta/base/instructions.py +++ b/lib/codegen/meta/base/instructions.py @@ -591,12 +591,25 @@ stack_check = Instruction( The global variable must be accessible and naturally aligned for a pointer-sized value. + + `stack_check` is an alternative way to detect stack overflow, when using + a calling convention that doesn't perform stack probes. """, ins=GV, can_trap=True) +delta = Operand('delta', Int) +adjust_sp_down = Instruction( + 'adjust_sp_down', r""" + Subtracts ``delta`` offset value from the stack pointer register. + + This instruction is used to adjust the stack pointer by a dynamic amount. + """, + ins=(delta,), + other_side_effects=True) + StackOffset = Operand('Offset', imm64, 'Offset from current stack pointer') -adjust_sp_imm = Instruction( - 'adjust_sp_imm', r""" +adjust_sp_up_imm = Instruction( + 'adjust_sp_up_imm', r""" Adds ``Offset`` immediate offset value to the stack pointer register. This instruction is used to adjust the stack pointer, primarily in function @@ -606,6 +619,19 @@ adjust_sp_imm = Instruction( ins=(StackOffset,), other_side_effects=True) +StackOffset = Operand('Offset', imm64, 'Offset from current stack pointer') +adjust_sp_down_imm = Instruction( + 'adjust_sp_down_imm', r""" + Subtracts ``Offset`` immediate offset value from the stack pointer + register. + + This instruction is used to adjust the stack pointer, primarily in function + prologues and epilogues. ``Offset`` is constrained to the size of a signed + 32-bit integer. + """, + ins=(StackOffset,), + other_side_effects=True) + f = Operand('f', iflags) ifcmp_sp = Instruction( diff --git a/lib/codegen/meta/base/settings.py b/lib/codegen/meta/base/settings.py index 580f0ee33e..32df3d63b4 100644 --- a/lib/codegen/meta/base/settings.py +++ b/lib/codegen/meta/base/settings.py @@ -38,17 +38,27 @@ call_conv = EnumSetting( - system_v: System V-style convention used on many platforms - fastcall: Windows "fastcall" convention, also used for x64 and ARM - baldrdash: SpiderMonkey WebAssembly convention + - probestack: specialized convention for the probestack function The default calling convention may be overridden by individual functions. """, - 'fast', 'cold', 'system_v', 'fastcall', 'baldrdash') + 'fast', 'cold', 'system_v', 'fastcall', 'baldrdash', 'probestack') # Note that Cretonne doesn't currently need an is_pie flag, because PIE is just # PIC where symbols can't be pre-empted, which can be expressed with the # `colocated` flag on external functions and global variables. is_pic = BoolSetting("Enable Position-Independent Code generation") +colocated_libcalls = BoolSetting( + """ + Use colocated libcalls. + + Generate code that assumes that libcalls can be declared "colocated", + meaning they will be defined along with the current function, such that + they can use more efficient addressing. + """) + return_at_end = BoolSetting( """ Generate functions with at most a single return instruction at the @@ -115,4 +125,31 @@ allones_funcaddrs = BoolSetting( Emit not-yet-relocated function addresses as all-ones bit patterns. """) +# +# Stack probing options. +# +probestack_enabled = BoolSetting( + """ + Enable the use of stack probes, for calling conventions which support + this functionality. + """, + default=True) + +probestack_func_adjusts_sp = BoolSetting( + """ + Set this to true of the stack probe function modifies the stack pointer + itself. + """) + +probestack_size_log2 = NumSetting( + """ + The log2 of the size of the stack guard region. + + Stack frames larger than this size will have stack overflow checked + by calling the probestack function. + + The default is 12, which translates to a size of 4096. + """, + default=12) + group.close(globals()) diff --git a/lib/codegen/meta/isa/x86/encodings.py b/lib/codegen/meta/isa/x86/encodings.py index 8b3244222b..65c28b1191 100644 --- a/lib/codegen/meta/isa/x86/encodings.py +++ b/lib/codegen/meta/isa/x86/encodings.py @@ -136,29 +136,29 @@ for inst, rrr in [ (base.band_imm, 4), (base.bor_imm, 1), (base.bxor_imm, 6)]: - enc_i32_i64(inst, r.rib, 0x83, rrr=rrr) - enc_i32_i64(inst, r.rid, 0x81, rrr=rrr) + enc_i32_i64(inst, r.r_ib, 0x83, rrr=rrr) + enc_i32_i64(inst, r.r_id, 0x81, rrr=rrr) # TODO: band_imm.i64 with an unsigned 32-bit immediate can be encoded as # band_imm.i32. Can even use the single-byte immediate for 0xffff_ffXX masks. # Immediate constants. -X86_32.enc(base.iconst.i32, *r.puid(0xb8)) +X86_32.enc(base.iconst.i32, *r.pu_id(0xb8)) -X86_64.enc(base.iconst.i32, *r.puid.rex(0xb8)) -X86_64.enc(base.iconst.i32, *r.puid(0xb8)) +X86_64.enc(base.iconst.i32, *r.pu_id.rex(0xb8)) +X86_64.enc(base.iconst.i32, *r.pu_id(0xb8)) # The 32-bit immediate movl also zero-extends to 64 bits. -X86_64.enc(base.iconst.i64, *r.puid.rex(0xb8), +X86_64.enc(base.iconst.i64, *r.pu_id.rex(0xb8), instp=IsUnsignedInt(UnaryImm.imm, 32)) -X86_64.enc(base.iconst.i64, *r.puid(0xb8), +X86_64.enc(base.iconst.i64, *r.pu_id(0xb8), instp=IsUnsignedInt(UnaryImm.imm, 32)) # Sign-extended 32-bit immediate. -X86_64.enc(base.iconst.i64, *r.uid.rex(0xc7, rrr=0, w=1)) +X86_64.enc(base.iconst.i64, *r.u_id.rex(0xc7, rrr=0, w=1)) # Finally, the 0xb8 opcode takes an 8-byte immediate with a REX.W prefix. -X86_64.enc(base.iconst.i64, *r.puiq.rex(0xb8, w=1)) +X86_64.enc(base.iconst.i64, *r.pu_iq.rex(0xb8, w=1)) # bool constants. -enc_both(base.bconst.b1, r.puid_bool, 0xb8) +enc_both(base.bconst.b1, r.pu_id_bool, 0xb8) # Shifts and rotates. # Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit @@ -180,7 +180,7 @@ for inst, rrr in [ (base.ishl_imm, 4), (base.ushr_imm, 5), (base.sshr_imm, 7)]: - enc_i32_i64(inst, r.rib, 0xc1, rrr=rrr) + enc_i32_i64(inst, r.r_ib, 0xc1, rrr=rrr) # Population count. X86_32.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt) @@ -254,11 +254,21 @@ enc_x86_64(x86.pop.i64, r.popq, 0x58) X86_64.enc(base.copy_special, *r.copysp.rex(0x89, w=1)) X86_32.enc(base.copy_special, *r.copysp(0x89)) -# Adjust SP Imm -X86_32.enc(base.adjust_sp_imm, *r.adjustsp8(0x83)) -X86_32.enc(base.adjust_sp_imm, *r.adjustsp32(0x81)) -X86_64.enc(base.adjust_sp_imm, *r.adjustsp8.rex(0x83, w=1)) -X86_64.enc(base.adjust_sp_imm, *r.adjustsp32.rex(0x81, w=1)) +# Adjust SP down by a dynamic value (or up, with a negative operand). +X86_32.enc(base.adjust_sp_down.i32, *r.adjustsp(0x29)) +X86_64.enc(base.adjust_sp_down.i64, *r.adjustsp.rex(0x29, w=1)) + +# Adjust SP up by an immediate (or down, with a negative immediate) +X86_32.enc(base.adjust_sp_up_imm, *r.adjustsp_ib(0x83)) +X86_32.enc(base.adjust_sp_up_imm, *r.adjustsp_id(0x81)) +X86_64.enc(base.adjust_sp_up_imm, *r.adjustsp_ib.rex(0x83, w=1)) +X86_64.enc(base.adjust_sp_up_imm, *r.adjustsp_id.rex(0x81, w=1)) + +# Adjust SP down by an immediate (or up, with a negative immediate) +X86_32.enc(base.adjust_sp_down_imm, *r.adjustsp_ib(0x83, rrr=5)) +X86_32.enc(base.adjust_sp_down_imm, *r.adjustsp_id(0x81, rrr=5)) +X86_64.enc(base.adjust_sp_down_imm, *r.adjustsp_ib.rex(0x83, rrr=5, w=1)) +X86_64.enc(base.adjust_sp_down_imm, *r.adjustsp_id.rex(0x81, rrr=5, w=1)) # # Float loads and stores. @@ -406,11 +416,11 @@ X86_64.enc(base.trapff, r.trapff, 0) # Comparisons # enc_i32_i64(base.icmp, r.icscc, 0x39) -enc_i32_i64(base.icmp_imm, r.icsccib, 0x83, rrr=7) -enc_i32_i64(base.icmp_imm, r.icsccid, 0x81, rrr=7) +enc_i32_i64(base.icmp_imm, r.icscc_ib, 0x83, rrr=7) +enc_i32_i64(base.icmp_imm, r.icscc_id, 0x81, rrr=7) enc_i32_i64(base.ifcmp, r.rcmp, 0x39) -enc_i32_i64(base.ifcmp_imm, r.rcmpib, 0x83, rrr=7) -enc_i32_i64(base.ifcmp_imm, r.rcmpid, 0x81, rrr=7) +enc_i32_i64(base.ifcmp_imm, r.rcmp_ib, 0x83, rrr=7) +enc_i32_i64(base.ifcmp_imm, r.rcmp_id, 0x81, rrr=7) # TODO: We could special-case ifcmp_imm(x, 0) to TEST(x, x). X86_32.enc(base.ifcmp_sp.i32, *r.rcmp_sp(0x39)) diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py index 0f9a79d6a7..bfe57d8bcb 100644 --- a/lib/codegen/meta/isa/x86/recipes.py +++ b/lib/codegen/meta/isa/x86/recipes.py @@ -480,8 +480,8 @@ mulx = TailRecipe( ''') # XX /n ib with 8-bit immediate sign-extended. -rib = TailRecipe( - 'rib', BinaryImm, size=2, ins=GPR, outs=0, +r_ib = TailRecipe( + 'r_ib', BinaryImm, size=2, ins=GPR, outs=0, instp=IsSignedInt(BinaryImm.imm, 8), emit=''' PUT_OP(bits, rex1(in_reg0), sink); @@ -491,8 +491,8 @@ rib = TailRecipe( ''') # XX /n id with 32-bit immediate sign-extended. -rid = TailRecipe( - 'rid', BinaryImm, size=5, ins=GPR, outs=0, +r_id = TailRecipe( + 'r_id', BinaryImm, size=5, ins=GPR, outs=0, instp=IsSignedInt(BinaryImm.imm, 32), emit=''' PUT_OP(bits, rex1(in_reg0), sink); @@ -502,8 +502,8 @@ rid = TailRecipe( ''') # XX /n id with 32-bit immediate sign-extended. UnaryImm version. -uid = TailRecipe( - 'uid', UnaryImm, size=5, ins=(), outs=GPR, +u_id = TailRecipe( + 'u_id', UnaryImm, size=5, ins=(), outs=GPR, instp=IsSignedInt(UnaryImm.imm, 32), emit=''' PUT_OP(bits, rex1(out_reg0), sink); @@ -513,8 +513,8 @@ uid = TailRecipe( ''') # XX+rd id unary with 32-bit immediate. Note no recipe predicate. -puid = TailRecipe( - 'puid', UnaryImm, size=4, ins=(), outs=GPR, +pu_id = TailRecipe( + 'pu_id', UnaryImm, size=4, ins=(), outs=GPR, emit=''' // The destination register is encoded in the low bits of the opcode. // No ModR/M. @@ -524,8 +524,8 @@ puid = TailRecipe( ''') # XX+rd id unary with bool immediate. Note no recipe predicate. -puid_bool = TailRecipe( - 'puid_bool', UnaryBool, size=4, ins=(), outs=GPR, +pu_id_bool = TailRecipe( + 'pu_id_bool', UnaryBool, size=4, ins=(), outs=GPR, emit=''' // The destination register is encoded in the low bits of the opcode. // No ModR/M. @@ -535,8 +535,8 @@ puid_bool = TailRecipe( ''') # XX+rd iq unary with 64-bit immediate. -puiq = TailRecipe( - 'puiq', UnaryImm, size=8, ins=(), outs=GPR, +pu_iq = TailRecipe( + 'pu_iq', UnaryImm, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); let imm: i64 = imm.into(); @@ -564,8 +564,15 @@ copysp = TailRecipe( modrm_rr(dst, src, sink); ''') -adjustsp8 = TailRecipe( - 'adjustsp8', UnaryImm, size=2, ins=(), outs=(), +adjustsp = TailRecipe( + 'adjustsp', Unary, size=1, ins=(GPR), outs=(), + emit=''' + PUT_OP(bits, rex2(RU::rsp.into(), in_reg0), sink); + modrm_rr(RU::rsp.into(), in_reg0, sink); + ''') + +adjustsp_ib = TailRecipe( + 'adjustsp_ib', UnaryImm, size=2, ins=(), outs=(), instp=IsSignedInt(UnaryImm.imm, 8), emit=''' PUT_OP(bits, rex1(RU::rsp.into()), sink); @@ -574,8 +581,8 @@ adjustsp8 = TailRecipe( sink.put1(imm as u8); ''') -adjustsp32 = TailRecipe( - 'adjustsp32', UnaryImm, size=5, ins=(), outs=(), +adjustsp_id = TailRecipe( + 'adjustsp_id', UnaryImm, size=5, ins=(), outs=(), instp=IsSignedInt(UnaryImm.imm, 32), emit=''' PUT_OP(bits, rex1(RU::rsp.into()), sink); @@ -1217,8 +1224,8 @@ fcmp = TailRecipe( ''') # XX /n, MI form with imm8. -rcmpib = TailRecipe( - 'rcmpib', BinaryImm, size=2, ins=GPR, outs=FLAG.rflags, +rcmp_ib = TailRecipe( + 'rcmp_ib', BinaryImm, size=2, ins=GPR, outs=FLAG.rflags, instp=IsSignedInt(BinaryImm.imm, 8), emit=''' PUT_OP(bits, rex1(in_reg0), sink); @@ -1228,8 +1235,8 @@ rcmpib = TailRecipe( ''') # XX /n, MI form with imm32. -rcmpid = TailRecipe( - 'rcmpid', BinaryImm, size=5, ins=GPR, outs=FLAG.rflags, +rcmp_id = TailRecipe( + 'rcmp_id', BinaryImm, size=5, ins=GPR, outs=FLAG.rflags, instp=IsSignedInt(BinaryImm.imm, 32), emit=''' PUT_OP(bits, rex1(in_reg0), sink); @@ -1401,8 +1408,8 @@ icscc = TailRecipe( modrm_rr(out_reg0, 0, sink); ''') -icsccib = TailRecipe( - 'icsccib', IntCompareImm, size=2 + 3, ins=GPR, outs=ABCD, +icscc_ib = TailRecipe( + 'icscc_ib', IntCompareImm, size=2 + 3, ins=GPR, outs=ABCD, instp=IsSignedInt(IntCompareImm.imm, 8), emit=''' // Comparison instruction. @@ -1429,8 +1436,8 @@ icsccib = TailRecipe( modrm_rr(out_reg0, 0, sink); ''') -icsccid = TailRecipe( - 'icsccid', IntCompareImm, size=5 + 3, ins=GPR, outs=ABCD, +icscc_id = TailRecipe( + 'icscc_id', IntCompareImm, size=5 + 3, ins=GPR, outs=ABCD, instp=IsSignedInt(IntCompareImm.imm, 32), emit=''' // Comparison instruction. diff --git a/lib/codegen/src/ir/libcall.rs b/lib/codegen/src/ir/libcall.rs index 12216522ae..8b8e2a0599 100644 --- a/lib/codegen/src/ir/libcall.rs +++ b/lib/codegen/src/ir/libcall.rs @@ -1,6 +1,9 @@ //! Naming well-known routines in the runtime library. -use ir::{types, Opcode, Type}; +use ir::{types, Opcode, Type, Inst, Function, FuncRef, ExternalName, Signature, AbiParam, + ExtFuncData, ArgumentPurpose}; +use settings::CallConv; +use isa::{TargetIsa, RegUnit}; use std::fmt; use std::str::FromStr; @@ -14,6 +17,9 @@ use std::str::FromStr; /// This list is likely to grow over time. #[derive(Copy, Clone, Debug, PartialEq, Eq)] pub enum LibCall { + /// probe for stack overflow. These are emitted for functions which need + /// when the `probestack_enabled` setting is true. + Probestack, /// ceil.f32 CeilF32, /// ceil.f64 @@ -32,7 +38,8 @@ pub enum LibCall { NearestF64, } -const NAME: [&str; 8] = [ +const NAME: [&str; 9] = [ + "Probestack", "CeilF32", "CeilF64", "FloorF32", @@ -54,6 +61,7 @@ impl FromStr for LibCall { fn from_str(s: &str) -> Result { match s { + "Probestack" => Ok(LibCall::Probestack), "CeilF32" => Ok(LibCall::CeilF32), "CeilF64" => Ok(LibCall::CeilF64), "FloorF32" => Ok(LibCall::FloorF32), @@ -97,6 +105,96 @@ impl LibCall { } } +/// Get a function reference for `libcall` in `func`, following the signature +/// for `inst`. +/// +/// If there is an existing reference, use it, otherwise make a new one. +pub fn get_libcall_funcref( + libcall: LibCall, + func: &mut Function, + inst: Inst, + isa: &TargetIsa, +) -> FuncRef { + find_funcref(libcall, func).unwrap_or_else(|| make_funcref_for_inst(libcall, func, inst, isa)) +} + +/// Get a function reference for the probestack function in `func`. +/// +/// If there is an existing reference, use it, otherwise make a new one. +pub fn get_probestack_funcref( + func: &mut Function, + reg_type: Type, + arg_reg: RegUnit, + isa: &TargetIsa, +) -> FuncRef { + find_funcref(LibCall::Probestack, func).unwrap_or_else(|| { + make_funcref_for_probestack(func, reg_type, arg_reg, isa) + }) +} + +/// Get the existing function reference for `libcall` in `func` if it exists. +fn find_funcref(libcall: LibCall, func: &Function) -> Option { + // We're assuming that all libcall function decls are at the end. + // If we get this wrong, worst case we'll have duplicate libcall decls which is harmless. + for (fref, func_data) in func.dfg.ext_funcs.iter().rev() { + match func_data.name { + ExternalName::LibCall(lc) => { + if lc == libcall { + return Some(fref); + } + } + _ => break, + } + } + None +} + +/// Create a funcref for `LibCall::Probestack`. +fn make_funcref_for_probestack( + func: &mut Function, + reg_type: Type, + arg_reg: RegUnit, + isa: &TargetIsa, +) -> FuncRef { + let mut sig = Signature::new(CallConv::Probestack); + let rax = AbiParam::special_reg(reg_type, ArgumentPurpose::Normal, arg_reg); + sig.params.push(rax); + if !isa.flags().probestack_func_adjusts_sp() { + sig.returns.push(rax); + } + make_funcref(LibCall::Probestack, func, sig, isa) +} + +/// Create a funcref for `libcall` with a signature matching `inst`. +fn make_funcref_for_inst( + libcall: LibCall, + func: &mut Function, + inst: Inst, + isa: &TargetIsa, +) -> FuncRef { + // Start with a fast calling convention. We'll give the ISA a chance to change it. + let mut sig = Signature::new(isa.flags().call_conv()); + for &v in func.dfg.inst_args(inst) { + sig.params.push(AbiParam::new(func.dfg.value_type(v))); + } + for &v in func.dfg.inst_results(inst) { + sig.returns.push(AbiParam::new(func.dfg.value_type(v))); + } + + make_funcref(libcall, func, sig, isa) +} + +/// Create a funcref for `libcall`. +fn make_funcref(libcall: LibCall, func: &mut Function, sig: Signature, isa: &TargetIsa) -> FuncRef { + let sigref = func.import_signature(sig); + + func.import_function(ExtFuncData { + name: ExternalName::LibCall(libcall), + signature: sigref, + colocated: isa.flags().colocated_libcalls(), + }) +} + #[cfg(test)] mod test { use super::*; diff --git a/lib/codegen/src/ir/mod.rs b/lib/codegen/src/ir/mod.rs index ab14a305aa..3ee7dc9176 100644 --- a/lib/codegen/src/ir/mod.rs +++ b/lib/codegen/src/ir/mod.rs @@ -33,7 +33,7 @@ pub use ir::heap::{HeapBase, HeapData, HeapStyle}; pub use ir::instructions::{InstructionData, Opcode, ValueList, ValueListPool, VariableArgs}; pub use ir::jumptable::JumpTableData; pub use ir::layout::Layout; -pub use ir::libcall::LibCall; +pub use ir::libcall::{LibCall, get_libcall_funcref, get_probestack_funcref}; pub use ir::memflags::MemFlags; pub use ir::progpoint::{ExpandedProgramPoint, ProgramOrder, ProgramPoint}; pub use ir::sourceloc::SourceLoc; diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs index 0e4a43ee57..5ab9febb29 100644 --- a/lib/codegen/src/isa/x86/abi.rs +++ b/lib/codegen/src/isa/x86/abi.rs @@ -6,7 +6,8 @@ use cursor::{Cursor, CursorPosition, EncCursor}; use ir; use ir::immediates::Imm64; use ir::stackslot::{StackOffset, StackSize}; -use ir::{AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, InstBuilder, ValueLoc}; +use ir::{AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, InstBuilder, ValueLoc, + get_probestack_funcref}; use isa::{RegClass, RegUnit, TargetIsa}; use regalloc::RegisterSet; use result; @@ -216,10 +217,16 @@ pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::Ct } CallConv::Fastcall => unimplemented!("Windows calling conventions"), CallConv::Baldrdash => baldrdash_prologue_epilogue(func, isa), + CallConv::Probestack => unimplemented!("probestack calling convention"), } } pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { + debug_assert!( + !isa.flags().probestack_enabled(), + "baldrdash does not expect cretonne to emit stack probes" + ); + // Baldrdash on 32-bit x86 always aligns its stack pointer to 16 bytes. let stack_align = 16; let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; @@ -239,7 +246,7 @@ pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r // newer versions use a 16-byte aligned stack pointer. let stack_align = 16; let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; - let csr_type = if isa.flags().is_64bit() { + let reg_type = if isa.flags().is_64bit() { ir::types::I64 } else { ir::types::I32 @@ -266,7 +273,7 @@ pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r // Add CSRs to function signature let fp_arg = ir::AbiParam::special_reg( - csr_type, + reg_type, ir::ArgumentPurpose::FramePointer, RU::rbp as RegUnit, ); @@ -274,7 +281,7 @@ pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r func.signature.returns.push(fp_arg); for csr in csrs.iter(GPR) { - let csr_arg = ir::AbiParam::special_reg(csr_type, ir::ArgumentPurpose::CalleeSaved, csr); + let csr_arg = ir::AbiParam::special_reg(reg_type, ir::ArgumentPurpose::CalleeSaved, csr); func.signature.params.push(csr_arg); func.signature.returns.push(csr_arg); } @@ -282,11 +289,11 @@ pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r // Set up the cursor and insert the prologue let entry_ebb = func.layout.entry_block().expect("missing entry block"); let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_ebb); - insert_system_v_prologue(&mut pos, local_stack_size, csr_type, &csrs); + insert_system_v_prologue(&mut pos, local_stack_size, reg_type, &csrs, isa); // Reset the cursor and insert the epilogue let mut pos = pos.at_position(CursorPosition::Nowhere); - insert_system_v_epilogues(&mut pos, local_stack_size, csr_type, &csrs); + insert_system_v_epilogues(&mut pos, local_stack_size, reg_type, &csrs); Ok(()) } @@ -295,12 +302,13 @@ pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r fn insert_system_v_prologue( pos: &mut EncCursor, stack_size: i64, - csr_type: ir::types::Type, + reg_type: ir::types::Type, csrs: &RegisterSet, + isa: &TargetIsa, ) { // Append param to entry EBB let ebb = pos.current_ebb().expect("missing ebb under cursor"); - let fp = pos.func.dfg.append_ebb_param(ebb, csr_type); + let fp = pos.func.dfg.append_ebb_param(ebb, reg_type); pos.func.locations[fp] = ir::ValueLoc::Reg(RU::rbp as RegUnit); pos.ins().x86_push(fp); @@ -311,7 +319,7 @@ fn insert_system_v_prologue( for reg in csrs.iter(GPR) { // Append param to entry EBB - let csr_arg = pos.func.dfg.append_ebb_param(ebb, csr_type); + let csr_arg = pos.func.dfg.append_ebb_param(ebb, reg_type); // Assign it a location pos.func.locations[csr_arg] = ir::ValueLoc::Reg(reg); @@ -320,8 +328,48 @@ fn insert_system_v_prologue( pos.ins().x86_push(csr_arg); } + // Allocate stack frame storage. if stack_size > 0 { - pos.ins().adjust_sp_imm(Imm64::new(-stack_size)); + if isa.flags().probestack_enabled() && + stack_size > (1 << isa.flags().probestack_size_log2()) + { + // Emit a stack probe. + let rax = RU::rax as RegUnit; + let rax_val = ir::ValueLoc::Reg(rax); + + // The probestack function expects its input in %rax. + let arg = pos.ins().iconst(reg_type, stack_size); + pos.func.locations[arg] = rax_val; + + // Call the probestack function. + let callee = get_probestack_funcref(pos.func, reg_type, rax, isa); + + // Make the call. + let call = if !isa.flags().is_pic() && isa.flags().is_64bit() && + !pos.func.dfg.ext_funcs[callee].colocated + { + // 64-bit non-PIC non-colocated calls need to be legalized to call_indirect. + // Use r11 as it may be clobbered under all supported calling conventions. + let r11 = RU::r11 as RegUnit; + let sig = pos.func.dfg.ext_funcs[callee].signature; + let addr = pos.ins().func_addr(reg_type, callee); + pos.func.locations[addr] = ir::ValueLoc::Reg(r11); + pos.ins().call_indirect(sig, addr, &[arg]) + } else { + // Otherwise just do a normal call. + pos.ins().call(callee, &[arg]) + }; + + // If the probestack function doesn't adjust sp, do it ourselves. + if !isa.flags().probestack_func_adjusts_sp() { + let result = pos.func.dfg.inst_results(call)[0]; + pos.func.locations[result] = rax_val; + pos.ins().adjust_sp_down(result); + } + } else { + // Simply decrement the stack pointer. + pos.ins().adjust_sp_down_imm(Imm64::new(stack_size)); + } } } @@ -329,14 +377,14 @@ fn insert_system_v_prologue( fn insert_system_v_epilogues( pos: &mut EncCursor, stack_size: i64, - csr_type: ir::types::Type, + reg_type: ir::types::Type, csrs: &RegisterSet, ) { while let Some(ebb) = pos.next_ebb() { pos.goto_last_inst(ebb); if let Some(inst) = pos.current_inst() { if pos.func.dfg[inst].opcode().is_return() { - insert_system_v_epilogue(inst, stack_size, pos, csr_type, csrs); + insert_system_v_epilogue(inst, stack_size, pos, reg_type, csrs); } } } @@ -347,23 +395,23 @@ fn insert_system_v_epilogue( inst: ir::Inst, stack_size: i64, pos: &mut EncCursor, - csr_type: ir::types::Type, + reg_type: ir::types::Type, csrs: &RegisterSet, ) { if stack_size > 0 { - pos.ins().adjust_sp_imm(Imm64::new(stack_size)); + pos.ins().adjust_sp_up_imm(Imm64::new(stack_size)); } // Pop all the callee-saved registers, stepping backward each time to // preserve the correct order. - let fp_ret = pos.ins().x86_pop(csr_type); + let fp_ret = pos.ins().x86_pop(reg_type); pos.prev_inst(); pos.func.locations[fp_ret] = ir::ValueLoc::Reg(RU::rbp as RegUnit); pos.func.dfg.append_inst_arg(inst, fp_ret); for reg in csrs.iter(GPR) { - let csr_ret = pos.ins().x86_pop(csr_type); + let csr_ret = pos.ins().x86_pop(reg_type); pos.prev_inst(); pos.func.locations[csr_ret] = ir::ValueLoc::Reg(reg); diff --git a/lib/codegen/src/legalizer/libcall.rs b/lib/codegen/src/legalizer/libcall.rs index 7d058f24bb..fd7dd17e7e 100644 --- a/lib/codegen/src/legalizer/libcall.rs +++ b/lib/codegen/src/legalizer/libcall.rs @@ -1,7 +1,7 @@ //! Expanding instructions as runtime library calls. use ir; -use ir::InstBuilder; +use ir::{InstBuilder, get_libcall_funcref}; use std::vec::Vec; use isa::TargetIsa; @@ -14,58 +14,14 @@ pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function, isa: &TargetIs None => return false, }; - let funcref = - find_funcref(libcall, func).unwrap_or_else(|| make_funcref(libcall, inst, func, isa)); - // Now we convert `inst` to a call. First save the arguments. let mut args = Vec::new(); args.extend_from_slice(func.dfg.inst_args(inst)); // The replace builder will preserve the instruction result values. + let funcref = get_libcall_funcref(libcall, func, inst, isa); func.dfg.replace(inst).call(funcref, &args); // TODO: ask the ISA to legalize the signature. true } - -/// Get the existing function reference for `libcall` in `func` if it exists. -fn find_funcref(libcall: ir::LibCall, func: &ir::Function) -> Option { - // We're assuming that all libcall function decls are at the end. - // If we get this wrong, worst case we'll have duplicate libcall decls which is harmless. - for (fref, func_data) in func.dfg.ext_funcs.iter().rev() { - match func_data.name { - ir::ExternalName::LibCall(lc) => { - if lc == libcall { - return Some(fref); - } - } - _ => break, - } - } - None -} - -/// Create a funcref for `libcall` with a signature matching `inst`. -fn make_funcref( - libcall: ir::LibCall, - inst: ir::Inst, - func: &mut ir::Function, - isa: &TargetIsa, -) -> ir::FuncRef { - // Start with a fast calling convention. We'll give the ISA a chance to change it. - let mut sig = ir::Signature::new(isa.flags().call_conv()); - for &v in func.dfg.inst_args(inst) { - sig.params.push(ir::AbiParam::new(func.dfg.value_type(v))); - } - for &v in func.dfg.inst_results(inst) { - sig.returns.push(ir::AbiParam::new(func.dfg.value_type(v))); - } - let sigref = func.import_signature(sig); - - // TODO: Can libcalls be colocated in some circumstances? - func.import_function(ir::ExtFuncData { - name: ir::ExternalName::LibCall(libcall), - signature: sigref, - colocated: false, - }) -} diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs index e34ee1421f..dbefb260c5 100644 --- a/lib/codegen/src/settings.rs +++ b/lib/codegen/src/settings.rs @@ -363,6 +363,7 @@ mod tests { is_64bit = false\n\ call_conv = \"fast\"\n\ is_pic = false\n\ + colocated_libcalls = false\n\ return_at_end = false\n\ avoid_div_traps = false\n\ is_compressed = false\n\ @@ -370,7 +371,10 @@ mod tests { enable_simd = true\n\ enable_atomics = true\n\ baldrdash_prologue_words = 0\n\ - allones_funcaddrs = false\n" + allones_funcaddrs = false\n\ + probestack_enabled = true\n\ + probestack_func_adjusts_sp = false\n\ + probestack_size_log2 = 12\n" ); assert_eq!(f.opt_level(), super::OptLevel::Default); assert_eq!(f.enable_simd(), true); From beddcc50f26dd0706ac2082739a55fdd0b9d63cd Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Apr 2018 21:42:09 -0700 Subject: [PATCH 1742/3084] Minor code refactoring. --- lib/codegen/src/legalizer/mod.rs | 93 +++++++++++++++++--------------- 1 file changed, 49 insertions(+), 44 deletions(-) diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs index 019b40c413..9be4975a00 100644 --- a/lib/codegen/src/legalizer/mod.rs +++ b/lib/codegen/src/legalizer/mod.rs @@ -32,6 +32,49 @@ use self::heap::expand_heap_addr; use self::call::expand_call; use self::libcall::expand_as_libcall; +/// Legalize `inst` for `isa`. Return true if any changes to the code were +/// made; return false if the instruction was successfully encoded as is. +fn legalize_inst( + inst: ir::Inst, + pos: &mut FuncCursor, + cfg: &mut ControlFlowGraph, + isa: &TargetIsa, +) -> bool { + let opcode = pos.func.dfg[inst].opcode(); + + // Check for ABI boundaries that need to be converted to the legalized signature. + if opcode.is_call() { + if boundary::handle_call_abi(inst, pos.func, cfg) { + return true; + } + } else if opcode.is_return() { + if boundary::handle_return_abi(inst, pos.func, cfg) { + return true; + } + } else if opcode.is_branch() { + split::simplify_branch_arguments(&mut pos.func.dfg, inst); + } + + match pos.func.update_encoding(inst, isa) { + Ok(()) => false, + Err(action) => { + // We should transform the instruction into legal equivalents. + // If the current instruction was replaced, we need to double back and revisit + // the expanded sequence. This is both to assign encodings and possible to + // expand further. + // There's a risk of infinite looping here if the legalization patterns are + // unsound. Should we attempt to detect that? + if action(inst, pos.func, cfg, isa) { + return true; + } + + // We don't have any pattern expansion for this instruction either. + // Try converting it to a library call as a last resort. + expand_as_libcall(inst, pos.func, isa) + } + } +} + /// Legalize `func` for `isa`. /// /// - Transform any instructions that don't have a legal representation in `isa`. @@ -55,51 +98,13 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is let mut prev_pos = pos.position(); while let Some(inst) = pos.next_inst() { - let opcode = pos.func.dfg[inst].opcode(); - - // Check for ABI boundaries that need to be converted to the legalized signature. - if opcode.is_call() { - if boundary::handle_call_abi(inst, pos.func, cfg) { - // Go back and legalize the inserted argument conversion instructions. - pos.set_position(prev_pos); - continue; - } - } else if opcode.is_return() { - if boundary::handle_return_abi(inst, pos.func, cfg) { - // Go back and legalize the inserted return value conversion instructions. - pos.set_position(prev_pos); - continue; - } - } else if opcode.is_branch() { - split::simplify_branch_arguments(&mut pos.func.dfg, inst); + if legalize_inst(inst, &mut pos, cfg, isa) { + // Go back and legalize the inserted return value conversion instructions. + pos.set_position(prev_pos); + } else { + // Remember this position in case we need to double back. + prev_pos = pos.position(); } - - match pos.func.update_encoding(inst, isa) { - Ok(()) => {} - Err(action) => { - // We should transform the instruction into legal equivalents. - let changed = action(inst, pos.func, cfg, isa); - // If the current instruction was replaced, we need to double back and revisit - // the expanded sequence. This is both to assign encodings and possible to - // expand further. - // There's a risk of infinite looping here if the legalization patterns are - // unsound. Should we attempt to detect that? - if changed { - pos.set_position(prev_pos); - continue; - } - - // We don't have any pattern expansion for this instruction either. - // Try converting it to a library call as a last resort. - if expand_as_libcall(inst, pos.func, isa) { - pos.set_position(prev_pos); - continue; - } - } - } - - // Remember this position in case we need to double back. - prev_pos = pos.position(); } } } From 9f0a35103a5aee7a0a6e0e452d28df9966665e79 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Apr 2018 21:42:13 -0700 Subject: [PATCH 1743/3084] Update a comment. --- lib/codegen/meta/gen_legalizer.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/codegen/meta/gen_legalizer.py b/lib/codegen/meta/gen_legalizer.py index 104574b7b6..2312071173 100644 --- a/lib/codegen/meta/gen_legalizer.py +++ b/lib/codegen/meta/gen_legalizer.py @@ -356,7 +356,7 @@ def gen_xform(xform, fmt, type_sets): def gen_xform_group(xgrp, fmt, type_sets): # type: (XFormGroup, Formatter, UniqueTable) -> None - fmt.doc_comment("Legalize the instruction pointed to by `pos`.") + fmt.doc_comment("Legalize `inst`.") fmt.line('#[allow(unused_variables,unused_assignments,non_snake_case)]') with fmt.indented('pub fn {}('.format(xgrp.name)): fmt.line('inst: ir::Inst,') From 56b3465ed0259948bd5816d51c1c6614c66800fd Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sun, 22 Apr 2018 22:09:31 -0700 Subject: [PATCH 1744/3084] Use more lower-case letters for github URLs. This is a continuation of 362f8f13e2b8c7dd79b043c491479bb9fc6f69e9. --- lib/faerie/Cargo.toml | 2 +- lib/module/Cargo.toml | 2 +- lib/simplejit/Cargo.toml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index db7b9c2f35..95a8fe3cca 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -17,4 +17,4 @@ failure = "0.1.1" [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "Cretonne/cretonne" } +travis-ci = { repository = "cretonne/cretonne" } diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index defbb016d8..71bdb16072 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -23,4 +23,4 @@ core = ["hashmap_core", "cretonne-codegen/core"] [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "Cretonne/cretonne" } +travis-ci = { repository = "cretonne/cretonne" } diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index e914206a74..bf8ea66140 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -23,4 +23,4 @@ core = ["cretonne-codegen/core", "cretonne-module/core", "cretonne-native/core"] [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "Cretonne/cretonne" } +travis-ci = { repository = "cretonne/cretonne" } From 5bcfd47f3fdf4ab243bc6da581a991f028b57b0b Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 20 Apr 2018 08:59:50 -0700 Subject: [PATCH 1745/3084] Remove the non-REX encodings for regmove et al. regmove, regfill, and regspill have immediates which aren't value operands, so they aren't in the set of things that can be described by the existing constraint system. Consequently, constraints saying that the non-REX encodings only support registers that don't need REX prefixes don't work. Fow now, just remove the non-REX encodings, so that they don't get selected when they aren't valid. This fixes the last known issue with instruction shrinking, so it can be re-enabled. --- .../filetests/isa/x86/prologue-epilogue.cton | 2 +- lib/codegen/meta/isa/x86/encodings.py | 22 ++++++++++++++++--- lib/codegen/src/context.rs | 3 --- 3 files changed, 20 insertions(+), 7 deletions(-) diff --git a/cranelift/filetests/isa/x86/prologue-epilogue.cton b/cranelift/filetests/isa/x86/prologue-epilogue.cton index 41d2147438..249f2a9ff9 100644 --- a/cranelift/filetests/isa/x86/prologue-epilogue.cton +++ b/cranelift/filetests/isa/x86/prologue-epilogue.cton @@ -228,4 +228,4 @@ ebb4: ; check: function %divert ; check: regmove v5, %rcx -> %rbx -; check: [RexOp1popq#58,%rbx] v15 = x86_pop.i64 +; check: [Op1popq#58,%rbx] v15 = x86_pop.i64 diff --git a/lib/codegen/meta/isa/x86/encodings.py b/lib/codegen/meta/isa/x86/encodings.py index 65c28b1191..2284a0d70f 100644 --- a/lib/codegen/meta/isa/x86/encodings.py +++ b/lib/codegen/meta/isa/x86/encodings.py @@ -126,7 +126,13 @@ enc_i32_i64(x86.umulx, r.mulx, 0xf7, rrr=4) enc_i32_i64(base.copy, r.umr, 0x89) enc_both(base.copy.b1, r.umr, 0x89) -enc_i32_i64(base.regmove, r.rmov, 0x89) + +# For x86-64, only define REX forms for now, since we can't describe the +# special regunit immediate operands with the current constraint language. +X86_32.enc(base.regmove.i32, *r.rmov(0x89)) +X86_64.enc(base.regmove.i32, *r.rmov.rex(0x89)) +X86_64.enc(base.regmove.i64, *r.rmov.rex(0x89, w=1)) + enc_both(base.regmove.b1, r.rmov, 0x89) enc_both(base.regmove.i8, r.rmov, 0x89) @@ -251,6 +257,8 @@ X86_32.enc(x86.pop.i32, *r.popq(0x58)) enc_x86_64(x86.pop.i64, r.popq, 0x58) # Copy Special +# For x86-64, only define REX forms for now, since we can't describe the +# special regunit immediate operands with the current constraint language. X86_64.enc(base.copy_special, *r.copysp.rex(0x89, w=1)) X86_32.enc(base.copy_special, *r.copysp(0x89)) @@ -528,8 +536,16 @@ X86_64.enc(base.bitcast.i64.f64, *r.rfumr.rex(0x66, 0x0f, 0x7e, w=1)) # movaps enc_both(base.copy.f32, r.furm, 0x0f, 0x28) enc_both(base.copy.f64, r.furm, 0x0f, 0x28) -enc_both(base.regmove.f32, r.frmov, 0x0f, 0x28) -enc_both(base.regmove.f64, r.frmov, 0x0f, 0x28) + +# For x86-64, only define REX forms for now, since we can't describe the +# special regunit immediate operands with the current constraint language. +X86_32.enc(base.regmove.f32, *r.frmov(0x0f, 0x28)) +X86_64.enc(base.regmove.f32, *r.frmov.rex(0x0f, 0x28)) + +# For x86-64, only define REX forms for now, since we can't describe the +# special regunit immediate operands with the current constraint language. +X86_32.enc(base.regmove.f64, *r.frmov(0x0f, 0x28)) +X86_64.enc(base.regmove.f64, *r.frmov.rex(0x0f, 0x28)) # cvtsi2ss enc_i32_i64(base.fcvt_from_sint.f32, r.frurm, 0xf3, 0x0f, 0x2a) diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index 986ae74d85..c2aab76bad 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -141,12 +141,9 @@ impl Context { } self.regalloc(isa)?; self.prologue_epilogue(isa)?; - // Temporarily disable the shrink_instructions pass, as it causes miscompiles. - /* if isa.flags().opt_level() == OptLevel::Best { self.shrink_instructions(isa)?; } - */ self.relax_branches(isa) } From 103a65f171efcf0947ae8b7ab7604bb2022de093 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 23 Apr 2018 10:14:14 -0700 Subject: [PATCH 1746/3084] Mention the prelude in the umbrella README. --- lib/umbrella/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/umbrella/README.md b/lib/umbrella/README.md index f9c2ee3fbf..96320ade2e 100644 --- a/lib/umbrella/README.md +++ b/lib/umbrella/README.md @@ -1,3 +1,3 @@ This is an umbrella crate which contains no code of its own, but pulls in -other cretonne library crates to provide a convenient one-line dependency -for common use cases. +other cretonne library crates to provide a convenient one-line dependency, +and a prelude, for common use cases. From 0ef032cc41635bc6fe5705e800c7c8d69bc282f7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 23 Apr 2018 10:14:32 -0700 Subject: [PATCH 1747/3084] Add more commonly-used names to the prelude. --- lib/umbrella/src/lib.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs index f28fb45547..cc1a3e1697 100644 --- a/lib/umbrella/src/lib.rs +++ b/lib/umbrella/src/lib.rs @@ -26,10 +26,11 @@ pub mod prelude { pub use codegen; pub use codegen::entity::EntityRef; pub use codegen::ir::{AbiParam, InstBuilder, Value, Ebb, Signature, Type, JumpTableData, - MemFlags}; + MemFlags, ExtFuncData, GlobalVarData, StackSlotData, StackSlotKind, + TrapCode}; pub use codegen::ir::types; pub use codegen::ir::condcodes::{IntCC, FloatCC}; - pub use codegen::ir::immediates::{Ieee32, Ieee64}; + pub use codegen::ir::immediates::{Ieee32, Ieee64, Imm64}; pub use codegen::settings::{self, Configurable, CallConv}; pub use codegen::isa; From 834df5290d2f4feb552bfc0359fd4b0c97f72b6c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 23 Apr 2018 10:26:33 -0700 Subject: [PATCH 1748/3084] Add more comments to module.rs. --- lib/module/src/module.rs | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index b47189f1d9..f275dea575 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -79,18 +79,23 @@ enum FuncOrDataId { Data(DataId), } +/// Information about a function which can be called. pub struct FunctionDeclaration { pub name: String, pub linkage: Linkage, pub signature: ir::Signature, } +/// A function belonging to a `Module`. struct ModuleFunction where B: Backend, { + /// The function declaration. decl: FunctionDeclaration, + /// The compiled artifact, once it's available. compiled: Option, + /// A flag indicating whether the function has been finalized. finalized: bool, } @@ -103,18 +108,23 @@ where } } +/// Information about a data object which can be accessed. pub struct DataDeclaration { pub name: String, pub linkage: Linkage, pub writable: bool, } +/// A data object belonging to a `Module`. struct ModuleData where B: Backend, { + /// The data object declaration. decl: DataDeclaration, + /// The "compiled" artifact, once it's available. compiled: Option, + /// A flag indicating whether the data object has been finalized. finalized: bool, } @@ -128,6 +138,7 @@ where } } +/// The functions and data objects belonging to a module. struct ModuleContents where B: Backend, From 229b748f025bd3d6c23c99a9845b59d5a6137613 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 23 Apr 2018 14:06:54 -0700 Subject: [PATCH 1749/3084] Update to wasmparser 0.16.1. --- lib/wasm/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 7c83cf9ed8..5b48088a05 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -9,7 +9,7 @@ readme = "README.md" keywords = ["webassembly", "wasm"] [dependencies] -wasmparser = { version = "0.16.0", default-features = false } +wasmparser = { version = "0.16.1", default-features = false } cretonne-codegen = { path = "../codegen", version = "0.5.1", default-features = false } cretonne-frontend = { path = "../frontend", version = "0.5.1", default-features = false } From e356c742aafaf8efb8702346b951446237c99c5c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 23 Apr 2018 14:35:23 -0700 Subject: [PATCH 1750/3084] Bump version to 0.6.0 --- cranelift/Cargo.toml | 22 +++++++++++----------- cranelift/publish-all.sh | 2 +- lib/codegen/Cargo.toml | 9 +++------ lib/entity/Cargo.toml | 2 +- lib/faerie/Cargo.toml | 6 +++--- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/module/Cargo.toml | 11 ++++------- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/simplejit/Cargo.toml | 8 ++++---- lib/umbrella/Cargo.toml | 6 +++--- lib/wasm/Cargo.toml | 11 ++++------- 13 files changed, 43 insertions(+), 52 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 543bbcf8a0..ded5a8e588 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.5.1" +version = "0.6.0" description = "Binaries for testing the Cretonne libraries" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,16 +13,16 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne-codegen = { path = "lib/codegen", version = "0.5.1" } -cretonne-reader = { path = "lib/reader", version = "0.5.1" } -cretonne-frontend = { path = "lib/frontend", version = "0.5.1" } -cretonne-wasm = { path = "lib/wasm", version = "0.5.1" } -cretonne-native = { path = "lib/native", version = "0.5.1" } -cretonne-filetests = { path = "lib/filetests", version = "0.5.1" } -cretonne-module = { path = "lib/module", version = "0.5.1" } -cretonne-faerie = { path = "lib/faerie", version = "0.5.1" } -cretonne-simplejit = { path = "lib/simplejit", version = "0.5.1" } -cretonne = { path = "lib/umbrella", version = "0.5.1" } +cretonne-codegen = { path = "lib/codegen", version = "0.6.0" } +cretonne-reader = { path = "lib/reader", version = "0.6.0" } +cretonne-frontend = { path = "lib/frontend", version = "0.6.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.6.0" } +cretonne-native = { path = "lib/native", version = "0.6.0" } +cretonne-filetests = { path = "lib/filetests", version = "0.6.0" } +cretonne-module = { path = "lib/module", version = "0.6.0" } +cretonne-faerie = { path = "lib/faerie", version = "0.6.0" } +cretonne-simplejit = { path = "lib/simplejit", version = "0.6.0" } +cretonne = { path = "lib/umbrella", version = "0.6.0" } filecheck = "0.3.0" docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 06aaab658a..1901862857 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ cd $(dirname "$0") topdir="$(pwd)" # All the cretonne-* crates have the same version number -version="0.5.1" +version="0.6.0" # Update all of the Cargo.toml files. # diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 8304c50ca4..b21ea84d51 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-codegen" -version = "0.5.1" +version = "0.6.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -11,18 +11,15 @@ keywords = ["compile", "compiler", "jit"] build = "build.rs" [dependencies] -cretonne-entity = { path = "../entity", version = "0.5.1", default-features = false } +cretonne-entity = { path = "../entity", version = "0.6.0", default-features = false } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } +hashmap_core = { version = "0.1.4", optional = true } # It is a goal of the cretonne-codegen crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be # accomodated in `tests`. -[dependencies.hashmap_core] -version = "0.1.1" -optional = true - [features] # The "std" feature enables use of libstd. The "core" feature enables use # of some minimal std-like replacement libraries. At least one of these two diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index c5f1b88c03..d502b6de0c 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-entity" -version = "0.5.1" +version = "0.6.0" description = "Data structures using entity references as mapping keys" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index 95a8fe3cca..864553520b 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-faerie" -version = "0.5.1" +version = "0.6.0" authors = ["The Cretonne Project Developers"] description = "Emit Cretonne output to native object files with Faerie" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.1" } -cretonne-module = { path = "../module", version = "0.5.1" } +cretonne-codegen = { path = "../codegen", version = "0.6.0" } +cretonne-module = { path = "../module", version = "0.6.0" } faerie = "0.2.0" goblin = "0.0.14" failure = "0.1.1" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 1a61c7c1e5..75d2dca630 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.5.1" +version = "0.6.0" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" publish = false [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.1" } -cretonne-reader = { path = "../reader", version = "0.5.1" } +cretonne-codegen = { path = "../codegen", version = "0.6.0" } +cretonne-reader = { path = "../reader", version = "0.6.0" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 68a4f3b817..16c5e909b6 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.5.1" +version = "0.6.0" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.1", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.6.0", default-features = false } [features] default = ["std"] diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 71bdb16072..2cec673959 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-module" -version = "0.5.1" +version = "0.6.0" authors = ["The Cretonne Project Developers"] description = "Support for linking functions and data with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,12 +9,9 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.1", default-features = false } -cretonne-entity = { path = "../entity", version = "0.5.1", default-features = false } - -[dependencies.hashmap_core] -version = "0.1.1" -optional = true +cretonne-codegen = { path = "../codegen", version = "0.6.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.6.0", default-features = false } +hashmap_core = { version = "0.1.4", optional = true } [features] default = ["std"] diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 38ed8cef48..9bf991f0d8 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.5.1" +version = "0.6.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -8,7 +8,7 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.1", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.6.0", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.1.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index c97cc3a113..cef19b99f4 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.5.1" +version = "0.6.0" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.1" } +cretonne-codegen = { path = "../codegen", version = "0.6.0" } [badges] maintenance = { status = "experimental" } diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index bf8ea66140..fc30b4daee 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-simplejit" -version = "0.5.1" +version = "0.6.0" authors = ["The Cretonne Project Developers"] description = "A simple JIT library backed by Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,9 +9,9 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.1", default-features = false } -cretonne-module = { path = "../module", version = "0.5.1", default-features = false } -cretonne-native = { path = "../native", version = "0.5.1", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.6.0", default-features = false } +cretonne-module = { path = "../module", version = "0.6.0", default-features = false } +cretonne-native = { path = "../native", version = "0.6.0", default-features = false } region = "0.2.0" libc = { version = "0.2.40", default-features = false } errno = "0.2.3" diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml index 6895955244..7c8501a80c 100644 --- a/lib/umbrella/Cargo.toml +++ b/lib/umbrella/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.5.1" +version = "0.6.0" description = "Umbrella for commonly-used cretonne crates" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -10,8 +10,8 @@ readme = "README.md" keywords = ["compile", "compiler", "jit"] [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.5.1", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.5.1", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.6.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.6.0", default-features = false } [features] default = ["std"] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 5b48088a05..f64da1de6a 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.5.1" +version = "0.6.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/cretonne/cretonne" @@ -10,12 +10,9 @@ keywords = ["webassembly", "wasm"] [dependencies] wasmparser = { version = "0.16.1", default-features = false } -cretonne-codegen = { path = "../codegen", version = "0.5.1", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.5.1", default-features = false } - -[dependencies.hashmap_core] -version = "0.1.1" -optional = true +cretonne-codegen = { path = "../codegen", version = "0.6.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.6.0", default-features = false } +hashmap_core = { version = "0.1.4", optional = true } [dev-dependencies] tempdir = "0.3.5" From 8351ba3e3e69f840fbc1258e9effd43c8c59f036 Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Mon, 23 Apr 2018 23:24:02 -0400 Subject: [PATCH 1751/3084] Disassemble compiled binary for debugging (#308) * Use Capstone to disassemble and print code after compilation in cton-util. * Fix rustfmt errors --- cranelift/Cargo.toml | 1 + cranelift/src/compile.rs | 41 +++++++++++++++++++++++++++++++++++++- cranelift/src/cton-util.rs | 1 + 3 files changed, 42 insertions(+), 1 deletion(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index ded5a8e588..dc916f116a 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -29,5 +29,6 @@ serde = "1.0.8" serde_derive = "1.0.8" tempdir = "0.3.5" term = "0.5.1" +capstone = "0.3.1" [workspace] diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 4b9f77494d..def23bf7de 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -1,8 +1,10 @@ //! CLI tool to read Cretonne IR files and compile them into native code. -use cretonne_codegen::Context; +use capstone::prelude::*; +use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::print_errors::pretty_error; use cretonne_codegen::settings::FlagsOrIsa; +use cretonne_codegen::Context; use cretonne_codegen::{binemit, ir}; use cretonne_reader::parse_test; use std::path::Path; @@ -121,8 +123,45 @@ fn handle_module( print!("{}", byte); } println!(); + + let cs = get_disassembler(isa)?; + + println!("\nDisassembly:"); + let insns = cs.disasm_all(&mem, 0x0).unwrap(); + for i in insns.iter() { + println!("{}", i); + } } } Ok(()) } + +fn get_disassembler(isa: &TargetIsa) -> Result { + let cs = match isa.name() { + "riscv" => return Err(String::from("No disassembler for RiscV")), + "x86" => { + if isa.flags().is_64bit() { + Capstone::new() + .x86() + .mode(arch::x86::ArchMode::Mode64) + .build() + } else { + Capstone::new() + .x86() + .mode(arch::x86::ArchMode::Mode32) + .build() + } + } + "arm32" => Capstone::new().arm().mode(arch::arm::ArchMode::Arm).build(), + "arm64" => { + Capstone::new() + .arm64() + .mode(arch::arm64::ArchMode::Arm) + .build() + } + _ => return Err(String::from("Unknown ISA")), + }; + + cs.map_err(|err| err.to_string()) +} diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index d69c561c44..0010034a0a 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -20,6 +20,7 @@ extern crate filecheck; extern crate serde_derive; extern crate tempdir; extern crate term; +extern crate capstone; use cretonne_codegen::{timing, VERSION}; use docopt::Docopt; From 948d5fdeecd600ac657e4a0bb09d9e55501ef630 Mon Sep 17 00:00:00 2001 From: John Rieth Date: Thu, 26 Apr 2018 11:07:34 -0700 Subject: [PATCH 1752/3084] Document that i8 and i16 arithmetic support is incomplete (#299) * Document that i8 and i16 arithmetic support is incomplete --- cranelift/docs/langref.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 8a8baa5b33..cddc104b9f 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -116,6 +116,8 @@ Integer values have a fixed size and can be interpreted as either signed or unsigned. Some instructions will interpret an operand as a signed or unsigned number, others don't care. +The support for i8 and i16 arithmetic is incomplete and use could lead to bugs. + .. autoctontype:: i8 .. autoctontype:: i16 .. autoctontype:: i32 From ee9dcb8367f477b8add9137d94a8741951272b81 Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Thu, 26 Apr 2018 22:02:35 -0700 Subject: [PATCH 1753/3084] Improvements to Modules API (#307) * test-no_std: use cargo +nightly assume folks have rustup set to use stable by default * cretonne-module, -faerie, -simplejit: use new ModuleError enum CtonError is not really appropriate for use in the module system. Instead, create a new enum ModuleError, which implements failure::Fail (works with no_std). Translate existing panics and unimplemented error cases to return ModuleErrors. * cretonne-faerie: export FaerieProduct * cretonne-module: expose FuncOrDataId, and Module::get_name to lookup This is helpful for looking up a name that has already been declared. Also, implement FuncOrDataId -> ExternalName conversion. * cretonne-faerie: depend on faerie 0.3.0 which has bugfix for data relocations * cretonne-module: change InvalidDefinition to InvalidImportDefinition per dan's code review. plus another typo fix * cretonne-faerie: add optional manifest of all traps from codegen * cretonne-module: provide more context in panics * cretonne-faerie: updates to docs * cretonne-faerie: return an Err instead of debug_assert when isa not pic --- cranelift/test-no_std.sh | 4 +- lib/faerie/Cargo.toml | 2 +- lib/faerie/src/backend.rs | 92 ++++++++++++++++------- lib/faerie/src/lib.rs | 4 +- lib/faerie/src/target.rs | 8 +- lib/faerie/src/traps.rs | 64 ++++++++++++++++ lib/module/Cargo.toml | 1 + lib/module/src/backend.rs | 8 +- lib/module/src/data_context.rs | 2 +- lib/module/src/lib.rs | 5 +- lib/module/src/module.rs | 129 +++++++++++++++++++++++++-------- lib/simplejit/src/backend.rs | 7 +- 12 files changed, 252 insertions(+), 74 deletions(-) create mode 100644 lib/faerie/src/traps.rs diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh index 7d372f3558..8c4d6ecfc1 100755 --- a/cranelift/test-no_std.sh +++ b/cranelift/test-no_std.sh @@ -21,10 +21,10 @@ do cd "lib/$LIB" # Test with just "core" enabled. - cargo test --no-default-features --features core + cargo +nightly test --no-default-features --features core # Test with "core" and "std" enabled at the same time. - cargo test --features core + cargo +nightly test --features core cd "$topdir" done diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index 864553520b..696b2ddad3 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -11,7 +11,7 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.6.0" } cretonne-module = { path = "../module", version = "0.6.0" } -faerie = "0.2.0" +faerie = "0.3.0" goblin = "0.0.14" failure = "0.1.1" diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index 4835156637..f8dfe0a4d3 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -3,13 +3,25 @@ use container; use cretonne_codegen::binemit::{Addend, CodeOffset, Reloc, RelocSink, NullTrapSink}; use cretonne_codegen::isa::TargetIsa; -use cretonne_codegen::result::CtonError; use cretonne_codegen::{self, binemit, ir}; -use cretonne_module::{Backend, DataContext, Linkage, ModuleNamespace, Init, DataDescription}; -use faerie; +use cretonne_module::{Backend, DataContext, Linkage, ModuleNamespace, Init, DataDescription, + ModuleError}; use failure::Error; +use faerie; use std::fs::File; use target; +use traps::{FaerieTrapManifest, FaerieTrapSink}; + +#[derive(Debug)] +/// Setting to enable collection of traps. Setting this to `Enabled` in +/// `FaerieBuilder` means that a `FaerieTrapManifest` will be present +/// in the `FaerieProduct`. +pub enum FaerieTrapCollection { + /// `FaerieProduct::trap_manifest` will be `None` + Disabled, + /// `FaerieProduct::trap_manifest` will be `Some` + Enabled, +} /// A builder for `FaerieBackend`. pub struct FaerieBuilder { @@ -17,6 +29,7 @@ pub struct FaerieBuilder { name: String, format: container::Format, faerie_target: faerie::Target, + collect_traps: FaerieTrapCollection, } impl FaerieBuilder { @@ -24,21 +37,28 @@ impl FaerieBuilder { /// can be passed to /// [`Module::new`](cretonne_module/struct.Module.html#method.new]. /// - /// Note: To support calls JIT'd functions from Rust or other compiled - /// code, it's necessary for the `call_conv` setting in `isa`'s flags - /// to match the host platform. + /// Faerie output requires that TargetIsa have PIC (Position Independent Code) enabled. + /// + /// `collect_traps` setting determines whether trap information is collected in a + /// `FaerieTrapManifest` available in the `FaerieProduct`. pub fn new( isa: Box, name: String, format: container::Format, - ) -> Result { - debug_assert!(isa.flags().is_pic(), "faerie requires PIC"); + collect_traps: FaerieTrapCollection, + ) -> Result { + if !isa.flags().is_pic() { + return Err(ModuleError::Backend( + "faerie requires TargetIsa be PIC".to_owned(), + )); + } let faerie_target = target::translate(&*isa)?; Ok(Self { isa, name, format, faerie_target, + collect_traps, }) } } @@ -48,6 +68,7 @@ pub struct FaerieBackend { isa: Box, artifact: faerie::Artifact, format: container::Format, + trap_manifest: Option, } pub struct FaerieCompiledFunction {} @@ -75,6 +96,10 @@ impl Backend for FaerieBackend { isa: builder.isa, artifact: faerie::Artifact::new(builder.faerie_target, builder.name), format: builder.format, + trap_manifest: match builder.collect_traps { + FaerieTrapCollection::Enabled => Some(FaerieTrapManifest::new()), + FaerieTrapCollection::Disabled => None, + }, } } @@ -100,7 +125,7 @@ impl Backend for FaerieBackend { ctx: &cretonne_codegen::Context, namespace: &ModuleNamespace, code_size: u32, - ) -> Result { + ) -> Result { let mut code: Vec = Vec::with_capacity(code_size as usize); code.resize(code_size as usize, 0); @@ -112,18 +137,29 @@ impl Backend for FaerieBackend { name, namespace, }; - // Ignore traps for now. For now, frontends should just avoid generating code - // that traps. - let mut trap_sink = NullTrapSink {}; - unsafe { - ctx.emit_to_memory( - &*self.isa, - code.as_mut_ptr(), - &mut reloc_sink, - &mut trap_sink, - ) - }; + if let Some(ref mut trap_manifest) = self.trap_manifest { + let mut trap_sink = FaerieTrapSink::new(name, code_size); + unsafe { + ctx.emit_to_memory( + &*self.isa, + code.as_mut_ptr(), + &mut reloc_sink, + &mut trap_sink, + ) + }; + trap_manifest.add_sink(trap_sink); + } else { + let mut trap_sink = NullTrapSink {}; + unsafe { + ctx.emit_to_memory( + &*self.isa, + code.as_mut_ptr(), + &mut reloc_sink, + &mut trap_sink, + ) + }; + } } self.artifact.define(name, code).expect( @@ -137,7 +173,7 @@ impl Backend for FaerieBackend { name: &str, data_ctx: &DataContext, namespace: &ModuleNamespace, - ) -> Result { + ) -> Result { let &DataDescription { writable: _writable, ref init, @@ -161,8 +197,6 @@ impl Backend for FaerieBackend { } } - // TODO: Change the signature of this function to use something other - // than `CtonError`, as `CtonError` can't convey faerie's errors. for &(offset, id) in function_relocs { let to = &namespace.get_function_decl(&function_decls[id]).name; self.artifact @@ -171,7 +205,7 @@ impl Backend for FaerieBackend { to, at: offset as usize, }) - .map_err(|_e| CtonError::InvalidInput)?; + .map_err(|e| ModuleError::Backend(format!("{}", e)))?; } for &(offset, id, addend) in data_relocs { debug_assert_eq!( @@ -186,7 +220,7 @@ impl Backend for FaerieBackend { to, at: offset as usize, }) - .map_err(|_e| CtonError::InvalidInput)?; + .map_err(|e| ModuleError::Backend(format!("{}", e)))?; } self.artifact.define(name, bytes).expect( @@ -230,6 +264,7 @@ impl Backend for FaerieBackend { FaerieProduct { artifact: self.artifact, format: self.format, + trap_manifest: self.trap_manifest, } } } @@ -238,7 +273,12 @@ impl Backend for FaerieBackend { /// [`finish`](../cretonne_module/struct.Module.html#method.finish) function. /// It provides functions for writing out the object file to memory or a file. pub struct FaerieProduct { - artifact: faerie::Artifact, + /// Faerie artifact with all functions, data, and links from the module defined + pub artifact: faerie::Artifact, + /// Optional trap manifest. Contains `FaerieTrapManifest` when `FaerieBuilder.collect_traps` is + /// set to `FaerieTrapCollection::Enabled`. + pub trap_manifest: Option, + /// The format that the builder specified for output. format: container::Format, } diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs index 3a6d0cad28..b56a3836f3 100644 --- a/lib/faerie/src/lib.rs +++ b/lib/faerie/src/lib.rs @@ -21,13 +21,13 @@ extern crate cretonne_codegen; extern crate cretonne_module; extern crate faerie; -#[macro_use] extern crate failure; extern crate goblin; mod backend; mod container; mod target; +pub mod traps; -pub use backend::{FaerieBuilder, FaerieBackend}; +pub use backend::{FaerieBuilder, FaerieBackend, FaerieProduct, FaerieTrapCollection}; pub use container::Format; diff --git a/lib/faerie/src/target.rs b/lib/faerie/src/target.rs index e1116dd08d..ac597fe1a9 100644 --- a/lib/faerie/src/target.rs +++ b/lib/faerie/src/target.rs @@ -1,9 +1,9 @@ use cretonne_codegen::isa; +use cretonne_module::ModuleError; use faerie::Target; -use failure::Error; /// Translate from a Cretonne `TargetIsa` to a Faerie `Target`. -pub fn translate(isa: &isa::TargetIsa) -> Result { +pub fn translate(isa: &isa::TargetIsa) -> Result { let name = isa.name(); match name { "x86" => Ok(if isa.flags().is_64bit() { @@ -13,6 +13,8 @@ pub fn translate(isa: &isa::TargetIsa) -> Result { }), "arm32" => Ok(Target::ARMv7), "arm64" => Ok(Target::ARM64), - _ => Err(format_err!("unsupported isa: {}", name)), + _ => Err(ModuleError::Backend( + format!("unsupported faerie isa: {}", name), + )), } } diff --git a/lib/faerie/src/traps.rs b/lib/faerie/src/traps.rs new file mode 100644 index 0000000000..3874583ee7 --- /dev/null +++ b/lib/faerie/src/traps.rs @@ -0,0 +1,64 @@ +//! Faerie trap manifests record every `TrapCode` that cretonne outputs during code generation, +//! for every function in the module. This data may be useful at runtime. + +use cretonne_codegen::{ir, binemit}; + +/// Record of the arguments cretonne passes to `TrapSink::trap` +pub struct FaerieTrapSite { + /// Offset into function + pub offset: binemit::CodeOffset, + /// Source location given to cretonne + pub srcloc: ir::SourceLoc, + /// Trap code, as determined by cretonne + pub code: ir::TrapCode, +} + +/// Record of the trap sites for a given function +pub struct FaerieTrapSink { + /// Name of function + pub name: String, + /// Total code size of function + pub code_size: u32, + /// All trap sites collected in function + pub sites: Vec, +} + + +impl FaerieTrapSink { + /// Create an empty `FaerieTrapSink` + pub fn new(name: &str, code_size: u32) -> Self { + Self { + sites: Vec::new(), + name: name.to_owned(), + code_size, + } + } +} + +impl binemit::TrapSink for FaerieTrapSink { + fn trap(&mut self, offset: binemit::CodeOffset, srcloc: ir::SourceLoc, code: ir::TrapCode) { + self.sites.push(FaerieTrapSite { + offset, + srcloc, + code, + }); + } +} + +/// Collection of all `FaerieTrapSink`s for the module +pub struct FaerieTrapManifest { + /// All `FaerieTrapSink` for the module + pub sinks: Vec, +} + +impl FaerieTrapManifest { + /// Create an empty `FaerieTrapManifest` + pub fn new() -> Self { + Self { sinks: Vec::new() } + } + + /// Put a `FaerieTrapSink` into manifest + pub fn add_sink(&mut self, sink: FaerieTrapSink) { + self.sinks.push(sink); + } +} diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 2cec673959..66ae646775 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -12,6 +12,7 @@ readme = "README.md" cretonne-codegen = { path = "../codegen", version = "0.6.0", default-features = false } cretonne-entity = { path = "../entity", version = "0.6.0", default-features = false } hashmap_core = { version = "0.1.4", optional = true } +failure = "0.1.1" [features] default = ["std"] diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs index 108b159e85..aa6439690d 100644 --- a/lib/module/src/backend.rs +++ b/lib/module/src/backend.rs @@ -3,9 +3,9 @@ use DataContext; use Linkage; use ModuleNamespace; +use ModuleError; use cretonne_codegen::Context; use cretonne_codegen::isa::TargetIsa; -use cretonne_codegen::result::CtonError; use cretonne_codegen::{binemit, ir}; use std::marker; @@ -57,19 +57,17 @@ where ctx: &Context, namespace: &ModuleNamespace, code_size: u32, - ) -> Result; + ) -> Result; /// Define a zero-initialized data object of the given size. /// /// Data objects must be declared before being defined. - /// - /// TODO: Is CtonError the right error code here? fn define_data( &mut self, name: &str, data_ctx: &DataContext, namespace: &ModuleNamespace, - ) -> Result; + ) -> Result; /// Write the address of `what` into the data for `data` at `offset`. `data` must refer to a /// defined data object. diff --git a/lib/module/src/data_context.rs b/lib/module/src/data_context.rs index 9e9f823383..3f9c9f8061 100644 --- a/lib/module/src/data_context.rs +++ b/lib/module/src/data_context.rs @@ -94,7 +94,7 @@ impl DataContext { self.description.init = Init::Zeros { size }; } - /// Define a zero-initialized object with the given size. + /// Define an object initialized with the given contents. /// /// TODO: Can we avoid a Box here? pub fn define(&mut self, contents: Box<[u8]>, writable: Writability) { diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs index 3835f2f0ce..5ada469555 100644 --- a/lib/module/src/lib.rs +++ b/lib/module/src/lib.rs @@ -16,9 +16,12 @@ use_self, ))] +#[macro_use] extern crate cretonne_codegen; #[macro_use] extern crate cretonne_entity; +#[macro_use] +extern crate failure; mod backend; mod data_context; @@ -26,4 +29,4 @@ mod module; pub use backend::Backend; pub use data_context::{DataContext, Writability, DataDescription, Init}; -pub use module::{DataId, FuncId, Linkage, Module, ModuleNamespace}; +pub use module::{DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleNamespace, ModuleError}; diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index f275dea575..c4090404ac 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -7,7 +7,7 @@ use Backend; use cretonne_codegen::entity::{EntityRef, PrimaryMap}; -use cretonne_codegen::result::{CtonError, CtonResult}; +use cretonne_codegen::result::CtonError; use cretonne_codegen::{binemit, ir, Context}; use data_context::DataContext; use std::collections::HashMap; @@ -17,11 +17,32 @@ use std::collections::HashMap; pub struct FuncId(u32); entity_impl!(FuncId, "funcid"); +/// Function identifiers are namespace 0 in `ir::ExternalName` +impl From for ir::ExternalName { + fn from(id: FuncId) -> ir::ExternalName { + ir::ExternalName::User { + namespace: 0, + index: id.0, + } + } +} + /// A data object identifier for use in the `Module` interface. #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] pub struct DataId(u32); entity_impl!(DataId, "dataid"); +/// Data identifiers are namespace 1 in `ir::ExternalName` +impl From for ir::ExternalName { + fn from(id: DataId) -> ir::ExternalName { + ir::ExternalName::User { + namespace: 1, + index: id.0, + } + } +} + + /// Linkage refers to where an entity is defined and who can see it. #[derive(Copy, Clone, PartialEq, Eq)] pub enum Linkage { @@ -73,12 +94,26 @@ impl Linkage { } } -#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] -enum FuncOrDataId { + +/// A declared name may refer to either a function or data declaration +#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)] +pub enum FuncOrDataId { + /// When it's a FuncId Func(FuncId), + /// When it's a DataId Data(DataId), } +/// Mapping to `ir::ExternalName` is trivial based on the `FuncId` and `DataId` mapping. +impl From for ir::ExternalName { + fn from(id: FuncOrDataId) -> ir::ExternalName { + match id { + FuncOrDataId::Func(funcid) => ir::ExternalName::from(funcid), + FuncOrDataId::Data(dataid) => ir::ExternalName::from(dataid), + } + } +} + /// Information about a function which can be called. pub struct FunctionDeclaration { pub name: String, @@ -86,6 +121,29 @@ pub struct FunctionDeclaration { pub signature: ir::Signature, } +/// Error messages for all `Module` and `Backend` methods +#[derive(Fail, Debug)] +pub enum ModuleError { + /// Indicates an identifier was used before it was declared + #[fail(display = "Undeclared identifier: {}", _0)] + Undeclared(String), + /// Indicates an identifier was used contrary to the way it was declared + #[fail(display = "Incompatible declaration of identifier: {}", _0)] + IncompatibleDeclaration(String), + /// Indicates an identifier was defined more than once + #[fail(display = "Duplicate definition of identifier: {}", _0)] + DuplicateDefinition(String), + /// Indicates an identifier was defined, but was declared as an import + #[fail(display = "Invalid to define identifier declared as an import: {}", _0)] + InvalidImportDefinition(String), + /// Wraps a `cretonne-codegen` error + #[fail(display = "Compilation error: {}", _0)] + Compilation(CtonError), + /// Wraps a generic error from a backend + #[fail(display = "Backend error: {}", _0)] + Backend(String), +} + /// A function belonging to a `Module`. struct ModuleFunction where @@ -157,7 +215,7 @@ where let func = FuncId::new(index as usize); &self.functions[func] } else { - panic!("unexpected ExternalName kind") + panic!("unexpected ExternalName kind {}", name) } } @@ -168,7 +226,7 @@ where let data = DataId::new(index as usize); &self.data_objects[data] } else { - panic!("unexpected ExternalName kind") + panic!("unexpected ExternalName kind {}", name) } } } @@ -227,7 +285,7 @@ where if let ir::ExternalName::User { namespace, .. } = *name { namespace == 0 } else { - panic!("unexpected ExternalName kind") + panic!("unexpected ExternalName kind {}", name) } } } @@ -258,6 +316,12 @@ where } } + /// Get the module identifier for a given name, if that name + /// has been declared. + pub fn get_name(&self, name: &str) -> Option { + self.names.get(name).map(|e| *e) + } + /// Return then pointer type for the current target. pub fn pointer_type(&self) -> ir::types::Type { if self.backend.isa().flags().is_64bit() { @@ -273,7 +337,7 @@ where name: &str, linkage: Linkage, signature: &ir::Signature, - ) -> Result { + ) -> Result { // TODO: Can we avoid allocating names so often? use std::collections::hash_map::Entry::*; match self.names.entry(name.to_owned()) { @@ -285,7 +349,9 @@ where self.backend.declare_function(name, existing.decl.linkage); Ok(id) } - FuncOrDataId::Data(..) => unimplemented!(), + FuncOrDataId::Data(..) => Err( + ModuleError::IncompatibleDeclaration(name.to_owned()), + ), } } Vacant(entry) => { @@ -311,7 +377,7 @@ where name: &str, linkage: Linkage, writable: bool, - ) -> Result { + ) -> Result { // TODO: Can we avoid allocating names so often? use std::collections::hash_map::Entry::*; match self.names.entry(name.to_owned()) { @@ -328,7 +394,9 @@ where Ok(id) } - FuncOrDataId::Func(..) => unimplemented!(), + FuncOrDataId::Func(..) => Err( + ModuleError::IncompatibleDeclaration(name.to_owned()), + ), } } Vacant(entry) => { @@ -386,19 +454,24 @@ where } /// Define a function, producing the function body from the given `Context`. - pub fn define_function(&mut self, func: FuncId, ctx: &mut Context) -> CtonResult { + pub fn define_function(&mut self, func: FuncId, ctx: &mut Context) -> Result<(), ModuleError> { let compiled = { - let code_size = ctx.compile(self.backend.isa())?; + let code_size = ctx.compile(self.backend.isa()).map_err(|e| { + dbg!( + "defining function {}: {}", + func, + ctx.func.display(self.backend.isa()) + ); + ModuleError::Compilation(e) + })?; let info = &self.contents.functions[func]; - debug_assert!( - info.compiled.is_none(), - "functions can be defined only once" - ); - debug_assert!( - info.decl.linkage.is_definable(), - "imported functions cannot be defined" - ); + if !info.compiled.is_none() { + return Err(ModuleError::DuplicateDefinition(info.decl.name.clone())); + } + if !info.decl.linkage.is_definable() { + return Err(ModuleError::InvalidImportDefinition(info.decl.name.clone())); + } Some(self.backend.define_function( &info.decl.name, ctx, @@ -413,17 +486,15 @@ where } /// Define a function, producing the data contents from the given `DataContext`. - pub fn define_data(&mut self, data: DataId, data_ctx: &DataContext) -> CtonResult { + pub fn define_data(&mut self, data: DataId, data_ctx: &DataContext) -> Result<(), ModuleError> { let compiled = { let info = &self.contents.data_objects[data]; - debug_assert!( - info.compiled.is_none(), - "functions can be defined only once" - ); - debug_assert!( - info.decl.linkage.is_definable(), - "imported functions cannot be defined" - ); + if !info.compiled.is_none() { + return Err(ModuleError::DuplicateDefinition(info.decl.name.clone())); + } + if !info.decl.linkage.is_definable() { + return Err(ModuleError::InvalidImportDefinition(info.decl.name.clone())); + } Some(self.backend.define_data( &info.decl.name, data_ctx, diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs index f5e85575ce..a73140a1aa 100644 --- a/lib/simplejit/src/backend.rs +++ b/lib/simplejit/src/backend.rs @@ -2,10 +2,9 @@ use cretonne_codegen::binemit::{Addend, CodeOffset, Reloc, RelocSink, NullTrapSink}; use cretonne_codegen::isa::TargetIsa; -use cretonne_codegen::result::CtonError; use cretonne_codegen::{self, ir, settings}; use cretonne_module::{Backend, DataContext, Linkage, ModuleNamespace, Writability, - DataDescription, Init}; + DataDescription, Init, ModuleError}; use cretonne_native; use std::ffi::CString; use std::ptr; @@ -116,7 +115,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { ctx: &cretonne_codegen::Context, _namespace: &ModuleNamespace, code_size: u32, - ) -> Result { + ) -> Result { let size = code_size as usize; let ptr = self.code_memory.allocate(size).expect( "TODO: handle OOM etc.", @@ -139,7 +138,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { _name: &str, data: &DataContext, _namespace: &ModuleNamespace, - ) -> Result { + ) -> Result { let &DataDescription { writable, ref init, From 779114aaede0b701330b824d27bfea2bf1f36106 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 26 Apr 2018 20:43:25 -0700 Subject: [PATCH 1754/3084] Require at least Sphinx 1.4 since the format of index tuples changed. --- cranelift/docs/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py index b2c1f14f97..d9a6dcb6eb 100644 --- a/cranelift/docs/conf.py +++ b/cranelift/docs/conf.py @@ -29,7 +29,7 @@ sys.path.insert(0, os.path.abspath('../lib/codegen/meta')) # If your documentation needs a minimal Sphinx version, state it here. # -# needs_sphinx = '1.0' +needs_sphinx = '1.4' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom From 9e48344f7fc13da47f81adcd35ccadb6a354b72d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 26 Apr 2018 20:51:45 -0700 Subject: [PATCH 1755/3084] Update examples to reflect that Cretonne indices typically start at 0. --- cranelift/docs/callex.cton | 12 ++++----- cranelift/docs/cton_domain.py | 8 +++--- cranelift/docs/example.cton | 46 +++++++++++++++++------------------ cranelift/docs/langref.rst | 8 +++--- cranelift/docs/testing.rst | 18 +++++++------- 5 files changed, 46 insertions(+), 46 deletions(-) diff --git a/cranelift/docs/callex.cton b/cranelift/docs/callex.cton index 26abda55b4..853f5c4091 100644 --- a/cranelift/docs/callex.cton +++ b/cranelift/docs/callex.cton @@ -1,13 +1,13 @@ test verifier function %gcd(i32 uext, i32 uext) -> i32 uext system_v { - fn1 = %divmod(i32 uext, i32 uext) -> i32 uext, i32 uext + fn0 = %divmod(i32 uext, i32 uext) -> i32 uext, i32 uext -ebb1(v1: i32, v2: i32): - brz v2, ebb2 - v3, v4 = call fn1(v1, v2) - return v3 +ebb1(v0: i32, v1: i32): + brz v1, ebb2 + v2, v3 = call fn0(v0, v1) + return v2 ebb2: - return v1 + return v0 } diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/cton_domain.py index 2d6e45eeed..4d28fbdadc 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/cton_domain.py @@ -6,7 +6,7 @@ # # .. cton::type:: type # Document an IR type. -# .. cton:inst:: v1, v2 = inst op1, op2 +# .. cton:inst:: v0, v1 = inst op0, op1 # Document an IR instruction. # from __future__ import absolute_import @@ -146,9 +146,9 @@ class CtonInst(CtonObject): def handle_signature(self, sig, signode): # Look for signatures like # - # v1, v2 = foo op1, op2 - # v1 = foo - # foo op1 + # v0, v1 = foo op0, op1 + # v0 = foo + # foo op0 parts = re.split(sep_equal, sig, 1) if len(parts) == 2: diff --git a/cranelift/docs/example.cton b/cranelift/docs/example.cton index 09e0c9cb5f..2bc5c9cc4c 100644 --- a/cranelift/docs/example.cton +++ b/cranelift/docs/example.cton @@ -1,31 +1,31 @@ test verifier function %average(i32, i32) -> f32 system_v { - ss1 = explicit_slot 8 ; Stack slot for ``sum``. + ss0 = explicit_slot 8 ; Stack slot for ``sum``. -ebb1(v1: i32, v2: i32): - v3 = f64const 0x0.0 - stack_store v3, ss1 - brz v2, ebb3 ; Handle count == 0. - v4 = iconst.i32 0 - jump ebb2(v4) +ebb1(v0: i32, v1: i32): + v2 = f64const 0x0.0 + stack_store v2, ss0 + brz v1, ebb3 ; Handle count == 0. + v3 = iconst.i32 0 + jump ebb2(v3) -ebb2(v5: i32): - v6 = imul_imm v5, 4 - v7 = iadd v1, v6 - v8 = load.f32 v7 ; array[i] - v9 = fpromote.f64 v8 - v10 = stack_load.f64 ss1 - v11 = fadd v9, v10 - stack_store v11, ss1 - v12 = iadd_imm v5, 1 - v13 = icmp ult v12, v2 - brnz v13, ebb2(v12) ; Loop backedge. - v14 = stack_load.f64 ss1 - v15 = fcvt_from_uint.f64 v2 - v16 = fdiv v14, v15 - v17 = fdemote.f32 v16 - return v17 +ebb2(v4: i32): + v5 = imul_imm v4, 4 + v6 = iadd v0, v5 + v7 = load.f32 v6 ; array[i] + v8 = fpromote.f64 v7 + v9 = stack_load.f64 ss0 + v10 = fadd v8, v9 + stack_store v10, ss0 + v11 = iadd_imm v4, 1 + v12 = icmp ult v11, v1 + brnz v12, ebb2(v11) ; Loop backedge. + v13 = stack_load.f64 ss0 + v14 = fcvt_from_uint.f64 v1 + v15 = fdiv v13, v14 + v16 = fdemote.f32 v15 + return v16 ebb3: v100 = f32const +NaN diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index cddc104b9f..177d133ace 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -37,7 +37,7 @@ The first line of a function definition provides the function *name* and the :term:`function signature` which declares the parameter and return types. Then follows the :term:`function preamble` which declares a number of entities that can be referenced inside the function. In the example above, the preamble -declares a single explicit stack slot, ``ss1``. +declares a single explicit stack slot, ``ss0``. After the preamble follows the :term:`function body` which consists of :term:`extended basic block`\s (EBBs), the first of which is the @@ -533,10 +533,10 @@ in :ref:`unrestricted loads and stores `. The :inst:`stack_addr` instruction can be used to macro-expand the stack access instructions before instruction selection:: - v1 = stack_load.f64 ss3, 16 + v0 = stack_load.f64 ss3, 16 ; Expands to: - v9 = stack_addr ss3, 16 - v1 = load.f64 v9 + v1 = stack_addr ss3, 16 + v0 = load.f64 v1 When Cretonne code is running in a sandbox, it can also be necessary to include stack overflow checks in the prologue. diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index b2ea827075..0265419305 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -226,13 +226,13 @@ command:: ; regex: I=\binst\d+\b ; check: label="{ebb0 | <$(BRZ=$I)>brz ebb2 | <$(JUMP=$I)>jump ebb1}"] - ebb0(v1: i32, v2: i32): - brz v2, ebb2 ; unordered: ebb0:$BRZ -> ebb2 - v4 = iconst.i32 0 - jump ebb1(v4) ; unordered: ebb0:$JUMP -> ebb1 + ebb0(v0: i32, v1: i32): + brz v1, ebb2 ; unordered: ebb0:$BRZ -> ebb2 + v2 = iconst.i32 0 + jump ebb1(v2) ; unordered: ebb0:$JUMP -> ebb1 ebb1(v5: i32): - return v1 + return v0 ebb2: v100 = f32const 0.0 @@ -303,10 +303,10 @@ that instruction is compared to the directive:: function %int32() { ebb0: - [-,%x5] v1 = iconst.i32 1 - [-,%x6] v2 = iconst.i32 2 - [R#0c,%x7] v10 = iadd v1, v2 ; bin: 006283b3 - [R#200c,%x8] v11 = isub v1, v2 ; bin: 40628433 + [-,%x5] v0 = iconst.i32 1 + [-,%x6] v1 = iconst.i32 2 + [R#0c,%x7] v10 = iadd v0, v1 ; bin: 006283b3 + [R#200c,%x8] v11 = isub v0, v1 ; bin: 40628433 return } From c40a4ddc53d785424bb3cf2ea3be9b5ce01cb892 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 27 Apr 2018 05:38:53 -0700 Subject: [PATCH 1756/3084] Add functions to Module for initializing Contexts. This allows Module to assign the Context the TargetIsa's default calling convention. --- lib/module/src/module.rs | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index c4090404ac..f3073f8468 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -331,6 +331,25 @@ where } } + /// Create a new `Context` initialized for use with this `Module`. + /// + /// This ensures that the `Context` is initialized with the default calling + /// convention for the `TargetIsa`. + pub fn make_context(&self) -> Context { + let mut ctx = Context::new(); + ctx.func.signature.call_conv = self.backend.isa().flags().call_conv(); + ctx + } + + /// Create a new `Context` initialized for use with this `Module`. + /// + /// This ensures that the `Context` is initialized with the default calling + /// convention for the `TargetIsa`. + pub fn clear_context(&self, ctx: &mut Context) { + ctx.clear(); + ctx.func.signature.call_conv = self.backend.isa().flags().call_conv(); + } + /// Declare a function in this module. pub fn declare_function( &mut self, From 525f01713bd0b6459439255b6d2340600f45a36e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 27 Apr 2018 06:02:18 -0700 Subject: [PATCH 1757/3084] Bump version to 0.7.0 --- cranelift/Cargo.toml | 22 +++++++++++----------- cranelift/publish-all.sh | 2 +- lib/codegen/Cargo.toml | 4 ++-- lib/entity/Cargo.toml | 2 +- lib/faerie/Cargo.toml | 6 +++--- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/module/Cargo.toml | 6 +++--- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/simplejit/Cargo.toml | 8 ++++---- lib/umbrella/Cargo.toml | 6 +++--- lib/wasm/Cargo.toml | 6 +++--- 13 files changed, 40 insertions(+), 40 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index dc916f116a..587b67d986 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.6.0" +version = "0.7.0" description = "Binaries for testing the Cretonne libraries" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,16 +13,16 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne-codegen = { path = "lib/codegen", version = "0.6.0" } -cretonne-reader = { path = "lib/reader", version = "0.6.0" } -cretonne-frontend = { path = "lib/frontend", version = "0.6.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.6.0" } -cretonne-native = { path = "lib/native", version = "0.6.0" } -cretonne-filetests = { path = "lib/filetests", version = "0.6.0" } -cretonne-module = { path = "lib/module", version = "0.6.0" } -cretonne-faerie = { path = "lib/faerie", version = "0.6.0" } -cretonne-simplejit = { path = "lib/simplejit", version = "0.6.0" } -cretonne = { path = "lib/umbrella", version = "0.6.0" } +cretonne-codegen = { path = "lib/codegen", version = "0.7.0" } +cretonne-reader = { path = "lib/reader", version = "0.7.0" } +cretonne-frontend = { path = "lib/frontend", version = "0.7.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.7.0" } +cretonne-native = { path = "lib/native", version = "0.7.0" } +cretonne-filetests = { path = "lib/filetests", version = "0.7.0" } +cretonne-module = { path = "lib/module", version = "0.7.0" } +cretonne-faerie = { path = "lib/faerie", version = "0.7.0" } +cretonne-simplejit = { path = "lib/simplejit", version = "0.7.0" } +cretonne = { path = "lib/umbrella", version = "0.7.0" } filecheck = "0.3.0" docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 1901862857..5e34c290e0 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ cd $(dirname "$0") topdir="$(pwd)" # All the cretonne-* crates have the same version number -version="0.6.0" +version="0.7.0" # Update all of the Cargo.toml files. # diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index b21ea84d51..097a637275 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-codegen" -version = "0.6.0" +version = "0.7.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"] build = "build.rs" [dependencies] -cretonne-entity = { path = "../entity", version = "0.6.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.7.0", default-features = false } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } hashmap_core = { version = "0.1.4", optional = true } diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index d502b6de0c..441d3323db 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-entity" -version = "0.6.0" +version = "0.7.0" description = "Data structures using entity references as mapping keys" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index 696b2ddad3..c1e55d5102 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-faerie" -version = "0.6.0" +version = "0.7.0" authors = ["The Cretonne Project Developers"] description = "Emit Cretonne output to native object files with Faerie" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.6.0" } -cretonne-module = { path = "../module", version = "0.6.0" } +cretonne-codegen = { path = "../codegen", version = "0.7.0" } +cretonne-module = { path = "../module", version = "0.7.0" } faerie = "0.3.0" goblin = "0.0.14" failure = "0.1.1" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 75d2dca630..add93aee0f 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.6.0" +version = "0.7.0" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" publish = false [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.6.0" } -cretonne-reader = { path = "../reader", version = "0.6.0" } +cretonne-codegen = { path = "../codegen", version = "0.7.0" } +cretonne-reader = { path = "../reader", version = "0.7.0" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 16c5e909b6..006b3240b5 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.6.0" +version = "0.7.0" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.6.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.7.0", default-features = false } [features] default = ["std"] diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 66ae646775..747e44666f 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-module" -version = "0.6.0" +version = "0.7.0" authors = ["The Cretonne Project Developers"] description = "Support for linking functions and data with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.6.0", default-features = false } -cretonne-entity = { path = "../entity", version = "0.6.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.7.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.7.0", default-features = false } hashmap_core = { version = "0.1.4", optional = true } failure = "0.1.1" diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 9bf991f0d8..a5de987412 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.6.0" +version = "0.7.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -8,7 +8,7 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.6.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.7.0", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.1.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index cef19b99f4..540d4aac83 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.6.0" +version = "0.7.0" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.6.0" } +cretonne-codegen = { path = "../codegen", version = "0.7.0" } [badges] maintenance = { status = "experimental" } diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index fc30b4daee..81d438deec 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-simplejit" -version = "0.6.0" +version = "0.7.0" authors = ["The Cretonne Project Developers"] description = "A simple JIT library backed by Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,9 +9,9 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.6.0", default-features = false } -cretonne-module = { path = "../module", version = "0.6.0", default-features = false } -cretonne-native = { path = "../native", version = "0.6.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.7.0", default-features = false } +cretonne-module = { path = "../module", version = "0.7.0", default-features = false } +cretonne-native = { path = "../native", version = "0.7.0", default-features = false } region = "0.2.0" libc = { version = "0.2.40", default-features = false } errno = "0.2.3" diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml index 7c8501a80c..b161fbc5ba 100644 --- a/lib/umbrella/Cargo.toml +++ b/lib/umbrella/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.6.0" +version = "0.7.0" description = "Umbrella for commonly-used cretonne crates" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -10,8 +10,8 @@ readme = "README.md" keywords = ["compile", "compiler", "jit"] [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.6.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.6.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.7.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.7.0", default-features = false } [features] default = ["std"] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index f64da1de6a..d180f2764b 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.6.0" +version = "0.7.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/cretonne/cretonne" @@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"] [dependencies] wasmparser = { version = "0.16.1", default-features = false } -cretonne-codegen = { path = "../codegen", version = "0.6.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.6.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.7.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.7.0", default-features = false } hashmap_core = { version = "0.1.4", optional = true } [dev-dependencies] From afd544272282b8df9e553329bfdb771e119b38f3 Mon Sep 17 00:00:00 2001 From: pup Date: Mon, 30 Apr 2018 13:26:21 -0700 Subject: [PATCH 1758/3084] Minor comment typo fix. (#315) --- lib/codegen/src/timing.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs index 82653458c0..017f3a466e 100644 --- a/lib/codegen/src/timing.rs +++ b/lib/codegen/src/timing.rs @@ -122,7 +122,7 @@ mod details { /// Accumulated timing information for a single pass. #[derive(Default)] struct PassTime { - /// Total time spent running this pas including children. + /// Total time spent running this pass including children. total: Duration, /// Time spent running in child passes. From 4942772f909f8dfd83ba9528b4577a04e459a5f1 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 26 Apr 2018 20:09:27 -0700 Subject: [PATCH 1759/3084] Add several more x86 CPU models. --- lib/codegen/meta/isa/x86/settings.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/codegen/meta/isa/x86/settings.py b/lib/codegen/meta/isa/x86/settings.py index 70da674fc0..76aee65ddd 100644 --- a/lib/codegen/meta/isa/x86/settings.py +++ b/lib/codegen/meta/isa/x86/settings.py @@ -38,8 +38,17 @@ use_lzcnt = And(has_lzcnt) # Presets corresponding to x86 CPUs. baseline = Preset() + nehalem = Preset( has_sse3, has_ssse3, has_sse41, has_sse42, has_popcnt) -haswell = Preset(nehalem, has_bmi1, has_lzcnt) +haswell = Preset(nehalem, has_bmi1, has_bmi2, has_lzcnt) +broadwell = Preset(haswell) +skylake = Preset(broadwell) +cannonlake = Preset(skylake) +icelake = Preset(cannonlake) + +znver1 = Preset( + has_sse3, has_ssse3, has_sse41, has_sse42, has_popcnt, + has_bmi1, has_bmi2, has_lzcnt) ISA.settings.close(globals()) From f4fe438baec2fb0a3ef1871846c46bd1dbd04975 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 30 Apr 2018 08:09:31 -0700 Subject: [PATCH 1760/3084] Suppress a warning about AsciiExt being deprecated. We currently support versions of Rust which need this import, so suppress the warning on versions of Rust which deprecated it. --- lib/reader/src/lexer.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index bad7a6369d..f6d89881fb 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -3,7 +3,7 @@ use cretonne_codegen::ir::types; use cretonne_codegen::ir::{Ebb, Value}; use error::Location; -#[allow(unused_imports)] +#[allow(unused_imports, deprecated)] use std::ascii::AsciiExt; use std::str::CharIndices; use std::u16; From b7f38ac8bc16dcddf8c22c754abe064141683b8a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 30 Apr 2018 13:45:57 -0700 Subject: [PATCH 1761/3084] Replace Builder's Vec with a Box<[u8]>. It doesn't need to dynamically grow, and `Box<[u8]>` is smaller. --- lib/codegen/src/settings.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs index dbefb260c5..f238cb601a 100644 --- a/lib/codegen/src/settings.rs +++ b/lib/codegen/src/settings.rs @@ -24,7 +24,7 @@ use constant_hash::{probe, simple_hash}; use isa::TargetIsa; use std::fmt; use std::result; -use std::vec::Vec; +use std::boxed::Box; use std::str; /// A string-based configurator for settings groups. @@ -46,7 +46,7 @@ pub trait Configurable { /// Collect settings values based on a template. pub struct Builder { template: &'static detail::Template, - bytes: Vec, + bytes: Box<[u8]>, } impl Builder { From 94a883abae571ceae7b25d31d54c3c1bafab8cb1 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 27 Apr 2018 21:01:35 -0700 Subject: [PATCH 1762/3084] Make settings::Flags::new consume the Builder. This makes it more clear what the relationship is between the Builder and the resulting Flags. --- cranelift/src/utils.rs | 4 +-- lib/codegen/meta/gen_settings.py | 2 +- lib/codegen/src/dominator_tree.rs | 2 +- lib/codegen/src/isa/arm32/mod.rs | 2 +- lib/codegen/src/isa/arm64/mod.rs | 2 +- lib/codegen/src/isa/mod.rs | 6 ++-- lib/codegen/src/isa/riscv/mod.rs | 8 ++--- lib/codegen/src/isa/riscv/settings.rs | 12 ++++---- lib/codegen/src/isa/x86/mod.rs | 2 +- lib/codegen/src/isa/x86/settings.rs | 42 +++++++++++++-------------- lib/codegen/src/regalloc/pressure.rs | 2 +- lib/codegen/src/regalloc/solver.rs | 2 +- lib/codegen/src/settings.rs | 9 +++--- lib/codegen/src/verifier/mod.rs | 4 +-- lib/frontend/src/frontend.rs | 2 +- lib/frontend/src/lib.rs | 2 +- lib/frontend/src/ssa.rs | 6 ++-- lib/reader/src/parser.rs | 6 ++-- lib/simplejit/src/backend.rs | 2 +- lib/wasm/src/environ/dummy.rs | 2 +- lib/wasm/tests/wasm_testsuite.rs | 4 +-- 21 files changed, 63 insertions(+), 60 deletions(-) diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index 4d54cf118a..ee879d0843 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -61,9 +61,9 @@ pub fn parse_sets_and_isa(flag_set: &[String], flag_isa: &str) -> Result IsaBuilder { fn isa_constructor( shared_flags: shared_settings::Flags, - builder: &shared_settings::Builder, + builder: shared_settings::Builder, ) -> Box { let level1 = if shared_flags.is_compressed() { &enc_tables::LEVEL1_T32[..] diff --git a/lib/codegen/src/isa/arm64/mod.rs b/lib/codegen/src/isa/arm64/mod.rs index 27c95f51b3..0d4f4a5397 100644 --- a/lib/codegen/src/isa/arm64/mod.rs +++ b/lib/codegen/src/isa/arm64/mod.rs @@ -32,7 +32,7 @@ pub fn isa_builder() -> IsaBuilder { fn isa_constructor( shared_flags: shared_settings::Flags, - builder: &shared_settings::Builder, + builder: shared_settings::Builder, ) -> Box { Box::new(Isa { isa_flags: settings::Flags::new(&shared_flags, builder), diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs index ed626a60a2..d94eb37830 100644 --- a/lib/codegen/src/isa/mod.rs +++ b/lib/codegen/src/isa/mod.rs @@ -24,7 +24,7 @@ //! use cretonne_codegen::isa; //! //! let shared_builder = settings::builder(); -//! let shared_flags = settings::Flags::new(&shared_builder); +//! let shared_flags = settings::Flags::new(shared_builder); //! //! match isa::lookup("riscv") { //! Err(_) => { @@ -117,14 +117,14 @@ pub enum LookupError { /// Modify the ISA-specific settings before creating the `TargetIsa` trait object with `finish`. pub struct Builder { setup: settings::Builder, - constructor: fn(settings::Flags, &settings::Builder) -> Box, + constructor: fn(settings::Flags, settings::Builder) -> Box, } impl Builder { /// Combine the ISA-specific settings with the provided ISA-independent settings and allocate a /// fully configured `TargetIsa` trait object. pub fn finish(self, shared_flags: settings::Flags) -> Box { - (self.constructor)(shared_flags, &self.setup) + (self.constructor)(shared_flags, self.setup) } } diff --git a/lib/codegen/src/isa/riscv/mod.rs b/lib/codegen/src/isa/riscv/mod.rs index af993aad0d..bcddc496d0 100644 --- a/lib/codegen/src/isa/riscv/mod.rs +++ b/lib/codegen/src/isa/riscv/mod.rs @@ -33,7 +33,7 @@ pub fn isa_builder() -> IsaBuilder { fn isa_constructor( shared_flags: shared_settings::Flags, - builder: &shared_settings::Builder, + builder: shared_settings::Builder, ) -> Box { let level1 = if shared_flags.is_64bit() { &enc_tables::LEVEL1_RV64[..] @@ -130,7 +130,7 @@ mod tests { fn test_64bitenc() { let mut shared_builder = settings::builder(); shared_builder.enable("is_64bit").unwrap(); - let shared_flags = settings::Flags::new(&shared_builder); + let shared_flags = settings::Flags::new(shared_builder); let isa = isa::lookup("riscv").unwrap().finish(shared_flags); let mut func = Function::new(); @@ -180,7 +180,7 @@ mod tests { fn test_32bitenc() { let mut shared_builder = settings::builder(); shared_builder.set("is_64bit", "false").unwrap(); - let shared_flags = settings::Flags::new(&shared_builder); + let shared_flags = settings::Flags::new(shared_builder); let isa = isa::lookup("riscv").unwrap().finish(shared_flags); let mut func = Function::new(); @@ -234,7 +234,7 @@ mod tests { fn test_rv32m() { let mut shared_builder = settings::builder(); shared_builder.set("is_64bit", "false").unwrap(); - let shared_flags = settings::Flags::new(&shared_builder); + let shared_flags = settings::Flags::new(shared_builder); // Set the supports_m stting which in turn enables the use_m predicate that unlocks // encodings for imul. diff --git a/lib/codegen/src/isa/riscv/settings.rs b/lib/codegen/src/isa/riscv/settings.rs index 4a127b578a..a5bdd6b850 100644 --- a/lib/codegen/src/isa/riscv/settings.rs +++ b/lib/codegen/src/isa/riscv/settings.rs @@ -16,9 +16,9 @@ mod tests { #[test] fn display_default() { - let shared = settings::Flags::new(&settings::builder()); + let shared = settings::Flags::new(settings::builder()); let b = builder(); - let f = Flags::new(&shared, &b); + let f = Flags::new(&shared, b); assert_eq!( f.to_string(), "[riscv]\n\ @@ -35,20 +35,20 @@ mod tests { #[test] fn predicates() { - let shared = settings::Flags::new(&settings::builder()); + let shared = settings::Flags::new(settings::builder()); let mut b = builder(); b.enable("supports_f").unwrap(); b.enable("supports_d").unwrap(); - let f = Flags::new(&shared, &b); + let f = Flags::new(&shared, b); assert_eq!(f.full_float(), true); let mut sb = settings::builder(); sb.set("enable_simd", "false").unwrap(); - let shared = settings::Flags::new(&sb); + let shared = settings::Flags::new(sb); let mut b = builder(); b.enable("supports_f").unwrap(); b.enable("supports_d").unwrap(); - let f = Flags::new(&shared, &b); + let f = Flags::new(&shared, b); assert_eq!(f.full_float(), false); } } diff --git a/lib/codegen/src/isa/x86/mod.rs b/lib/codegen/src/isa/x86/mod.rs index 6bbde318a5..787ec4a992 100644 --- a/lib/codegen/src/isa/x86/mod.rs +++ b/lib/codegen/src/isa/x86/mod.rs @@ -35,7 +35,7 @@ pub fn isa_builder() -> IsaBuilder { fn isa_constructor( shared_flags: shared_settings::Flags, - builder: &shared_settings::Builder, + builder: shared_settings::Builder, ) -> Box { let level1 = if shared_flags.is_64bit() { &enc_tables::LEVEL1_I64[..] diff --git a/lib/codegen/src/isa/x86/settings.rs b/lib/codegen/src/isa/x86/settings.rs index 379f0db493..84d0e96f06 100644 --- a/lib/codegen/src/isa/x86/settings.rs +++ b/lib/codegen/src/isa/x86/settings.rs @@ -15,38 +15,38 @@ mod tests { #[test] fn presets() { - let shared = settings::Flags::new(&settings::builder()); + let shared = settings::Flags::new(settings::builder()); // Nehalem has SSE4.1 but not BMI1. - let mut b1 = builder(); - b1.enable("nehalem").unwrap(); - let f1 = Flags::new(&shared, &b1); - assert_eq!(f1.has_sse41(), true); - assert_eq!(f1.has_bmi1(), false); + let mut b0 = builder(); + b0.enable("nehalem").unwrap(); + let f0 = Flags::new(&shared, b0); + assert_eq!(f0.has_sse41(), true); + assert_eq!(f0.has_bmi1(), false); - let mut b2 = builder(); - b2.enable("haswell").unwrap(); - let f2 = Flags::new(&shared, &b2); - assert_eq!(f2.has_sse41(), true); - assert_eq!(f2.has_bmi1(), true); + let mut b1 = builder(); + b1.enable("haswell").unwrap(); + let f1 = Flags::new(&shared, b1); + assert_eq!(f1.has_sse41(), true); + assert_eq!(f1.has_bmi1(), true); } #[test] fn display_presets() { // Spot check that the flags Display impl does not cause a panic - let shared = settings::Flags::new(&settings::builder()); + let shared = settings::Flags::new(settings::builder()); - let b1 = builder(); - let f1 = Flags::new(&shared, &b1); + let b0 = builder(); + let f0 = Flags::new(&shared, b0); + let _ = format!("{}", f0); + + let mut b1 = builder(); + b1.enable("nehalem").unwrap(); + let f1 = Flags::new(&shared, b1); let _ = format!("{}", f1); let mut b2 = builder(); - b2.enable("nehalem").unwrap(); - let f2 = Flags::new(&shared, &b1); + b2.enable("haswell").unwrap(); + let f2 = Flags::new(&shared, b2); let _ = format!("{}", f2); - - let mut b3 = builder(); - b3.enable("haswell").unwrap(); - let f3 = Flags::new(&shared, &b1); - let _ = format!("{}", f3); } } diff --git a/lib/codegen/src/regalloc/pressure.rs b/lib/codegen/src/regalloc/pressure.rs index 2c86050c59..49eb224e6d 100644 --- a/lib/codegen/src/regalloc/pressure.rs +++ b/lib/codegen/src/regalloc/pressure.rs @@ -281,7 +281,7 @@ mod tests { use settings; let shared_builder = settings::builder(); - let shared_flags = settings::Flags::new(&shared_builder); + let shared_flags = settings::Flags::new(shared_builder); isa::lookup("arm32").ok().map(|b| b.finish(shared_flags)) } diff --git a/lib/codegen/src/regalloc/solver.rs b/lib/codegen/src/regalloc/solver.rs index f3c14c9d81..80c7f862e2 100644 --- a/lib/codegen/src/regalloc/solver.rs +++ b/lib/codegen/src/regalloc/solver.rs @@ -1157,7 +1157,7 @@ mod tests { use settings; let shared_builder = settings::builder(); - let shared_flags = settings::Flags::new(&shared_builder); + let shared_flags = settings::Flags::new(shared_builder); isa::lookup("arm32").ok().map(|b| b.finish(shared_flags)) } diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs index f238cb601a..39416a298f 100644 --- a/lib/codegen/src/settings.rs +++ b/lib/codegen/src/settings.rs @@ -16,7 +16,7 @@ //! let mut b = settings::builder(); //! b.set("opt_level", "fastest"); //! -//! let f = settings::Flags::new(&b); +//! let f = settings::Flags::new(b); //! assert_eq!(f.opt_level(), settings::OptLevel::Fastest); //! ``` @@ -44,6 +44,7 @@ pub trait Configurable { } /// Collect settings values based on a template. +#[derive(Clone)] pub struct Builder { template: &'static detail::Template, bytes: Box<[u8]>, @@ -354,7 +355,7 @@ mod tests { #[test] fn display_default() { let b = builder(); - let f = Flags::new(&b); + let f = Flags::new(b); assert_eq!( f.to_string(), "[shared]\n\ @@ -388,7 +389,7 @@ mod tests { assert_eq!(b.enable("enable_simd"), Ok(())); assert_eq!(b.set("enable_simd", "false"), Ok(())); - let f = Flags::new(&b); + let f = Flags::new(b); assert_eq!(f.enable_simd(), false); } @@ -402,7 +403,7 @@ mod tests { assert_eq!(b.set("opt_level", "best"), Ok(())); assert_eq!(b.set("enable_simd", "0"), Ok(())); - let f = Flags::new(&b); + let f = Flags::new(b); assert_eq!(f.enable_simd(), false); assert_eq!(f.opt_level(), super::OptLevel::Best); } diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs index 6408b61977..6db1f871f7 100644 --- a/lib/codegen/src/verifier/mod.rs +++ b/lib/codegen/src/verifier/mod.rs @@ -1175,7 +1175,7 @@ mod tests { #[test] fn empty() { let func = Function::new(); - let flags = &settings::Flags::new(&settings::builder()); + let flags = &settings::Flags::new(settings::builder()); let verifier = Verifier::new(&func, flags.into()); assert_eq!(verifier.run(), Ok(())); } @@ -1198,7 +1198,7 @@ mod tests { }), ebb0, ); - let flags = &settings::Flags::new(&settings::builder()); + let flags = &settings::Flags::new(settings::builder()); let verifier = Verifier::new(&func, flags.into()); assert_err_with_msg!(verifier.run(), "instruction format"); } diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 1f2c818f99..b2e6385722 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -688,7 +688,7 @@ mod tests { builder.finalize(); } - let flags = settings::Flags::new(&settings::builder()); + let flags = settings::Flags::new(settings::builder()); let res = verify_function(&func, &flags); // println!("{}", func.display(None)); match res { diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 65d76fd279..3e43d04a84 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -117,7 +117,7 @@ //! builder.finalize(); //! } //! -//! let flags = settings::Flags::new(&settings::builder()); +//! let flags = settings::Flags::new(settings::builder()); //! let res = verify_function(&func, &flags); //! println!("{}", func.display(None)); //! match res { diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 9c07730de9..422cb05931 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -1021,7 +1021,7 @@ mod tests { let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1); cur.ins().return_(&[]) }; - let flags = settings::Flags::new(&settings::builder()); + let flags = settings::Flags::new(settings::builder()); match verify_function(&func, &flags) { Ok(()) => {} Err(err) => { @@ -1200,7 +1200,7 @@ mod tests { ssa.declare_ebb_predecessor(ebb1, block2, j); } ssa.seal_ebb_header_block(ebb1, &mut func); - let flags = settings::Flags::new(&settings::builder()); + let flags = settings::Flags::new(settings::builder()); match verify_function(&func, &flags) { Ok(()) => {} Err(err) => { @@ -1251,7 +1251,7 @@ mod tests { } ssa.seal_ebb_header_block(ebb1, &mut func); ssa.seal_ebb_header_block(ebb2, &mut func); - let flags = settings::Flags::new(&settings::builder()); + let flags = settings::Flags::new(settings::builder()); match verify_function(&func, &flags) { Ok(()) => {} Err(err) => { diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 8ee584075c..67e6939910 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -725,14 +725,16 @@ impl<'a> Parser<'a> { isaspec::parse_options(words, &mut isa_builder, &self.loc)?; // Construct a trait object with the aggregate settings. - isas.push(isa_builder.finish(settings::Flags::new(&flag_builder))); + isas.push(isa_builder.finish( + settings::Flags::new(flag_builder.clone()), + )); } _ => break, } } if !seen_isa { // No `isa` commands, but we allow for `set` commands. - Ok(isaspec::IsaSpec::None(settings::Flags::new(&flag_builder))) + Ok(isaspec::IsaSpec::None(settings::Flags::new(flag_builder))) } else if let Some(loc) = last_set_loc { err!( loc, diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs index a73140a1aa..c7785e04a7 100644 --- a/lib/simplejit/src/backend.rs +++ b/lib/simplejit/src/backend.rs @@ -22,7 +22,7 @@ impl SimpleJITBuilder { let (flag_builder, isa_builder) = cretonne_native::builders().unwrap_or_else(|_| { panic!("host machine is not a supported target"); }); - let isa = isa_builder.finish(settings::Flags::new(&flag_builder)); + let isa = isa_builder.finish(settings::Flags::new(flag_builder)); Self::with_isa(isa) } diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 2b1fbfbfc7..68927101cf 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -101,7 +101,7 @@ pub struct DummyEnvironment { impl DummyEnvironment { /// Allocates the data structures with default flags. pub fn default() -> Self { - Self::with_flags(settings::Flags::new(&settings::builder())) + Self::with_flags(settings::Flags::new(settings::builder())) } /// Allocates the data structures with the given flags. diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs index 15feea4b55..1871b4d781 100644 --- a/lib/wasm/tests/wasm_testsuite.rs +++ b/lib/wasm/tests/wasm_testsuite.rs @@ -32,7 +32,7 @@ fn testsuite() { }) .collect(); paths.sort_by_key(|dir| dir.path()); - let flags = Flags::new(&settings::builder()); + let flags = Flags::new(settings::builder()); for path in paths { let path = path.path(); handle_module(&path, &flags); @@ -43,7 +43,7 @@ fn testsuite() { fn return_at_end() { let mut flag_builder = settings::builder(); flag_builder.enable("return_at_end").unwrap(); - let flags = Flags::new(&flag_builder); + let flags = Flags::new(flag_builder); handle_module(&PathBuf::from("../../wasmtests/return_at_end.wat"), &flags); } From 9c87f3ac8727b27cf28fba8b59048efdd9c3f068 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 30 Apr 2018 13:56:29 -0700 Subject: [PATCH 1763/3084] Fix some warnings in no_std builds. The dbg! macro expands to nothing in no_std mode, so variables that are only used for debugging prompt unused variable warnings. Also, allow unstable_features in no_std builds, since they use feature(alloc), which is an unstable feature. --- lib/codegen/src/lib.rs | 3 ++- lib/codegen/src/regalloc/coalescing.rs | 5 +++-- lib/codegen/src/regalloc/coloring.rs | 7 +++++-- lib/entity/src/lib.rs | 3 ++- lib/frontend/src/lib.rs | 3 ++- lib/frontend/src/ssa.rs | 12 ++++++------ lib/wasm/src/lib.rs | 3 ++- 7 files changed, 22 insertions(+), 14 deletions(-) diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs index 8f60d23ebe..faa21d24d7 100644 --- a/lib/codegen/src/lib.rs +++ b/lib/codegen/src/lib.rs @@ -1,7 +1,8 @@ //! Cretonne code generation library. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] -#![warn(unused_import_braces, unstable_features)] +#![warn(unused_import_braces)] +#![cfg_attr(feature = "std", warn(unstable_features))] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature="cargo-clippy", allow( // Rustfmt 0.9.0 is at odds with this lint: diff --git a/lib/codegen/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs index 096c8546bc..2a70208c78 100644 --- a/lib/codegen/src/regalloc/coalescing.rs +++ b/lib/codegen/src/regalloc/coalescing.rs @@ -6,6 +6,7 @@ //! parameter will belong to the same virtual register as the EBB parameter value itself. use cursor::{Cursor, EncCursor}; +#[cfg(feature = "std")] use dbg::DisplayList; use dominator_tree::{DominatorTree, DominatorTreePreorder}; use flowgraph::ControlFlowGraph; @@ -546,8 +547,8 @@ impl<'a> Context<'a> { return false; } - let vreg = self.virtregs.unify(self.values); - dbg!("-> merged into {} = {}", vreg, DisplayList(self.values)); + let _vreg = self.virtregs.unify(self.values); + dbg!("-> merged into {} = {}", _vreg, DisplayList(self.values)); true } diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs index e6281920d2..d080213ade 100644 --- a/lib/codegen/src/regalloc/coloring.rs +++ b/lib/codegen/src/regalloc/coloring.rs @@ -857,8 +857,11 @@ impl<'a> Context<'a> { let added = self.try_add_var(rc, throughs); debug_assert!(added, "Ran out of registers in {}", rc); } - Err(SolverError::Global(value)) => { - dbg!("Not enough global registers for {}, trying as local", value); + Err(SolverError::Global(_value)) => { + dbg!( + "Not enough global registers for {}, trying as local", + _value + ); // We'll clear the `is_global` flag on all solver variables and instead make a // note to replace all global defines with local defines followed by a copy. *replace_global_defines = true; diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs index 7936495568..693e5d4f8c 100644 --- a/lib/entity/src/lib.rs +++ b/lib/entity/src/lib.rs @@ -30,7 +30,8 @@ //! `Vec`. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] -#![warn(unused_import_braces, unstable_features)] +#![warn(unused_import_braces)] +#![cfg_attr(feature = "std", warn(unstable_features))] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 3e43d04a84..b9880c9696 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -128,7 +128,8 @@ //! ``` #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] -#![warn(unused_import_braces, unstable_features)] +#![warn(unused_import_braces)] +#![cfg_attr(feature = "std", warn(unstable_features))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default))] #![cfg_attr(feature="cargo-clippy", warn( float_arithmetic, diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 422cb05931..d13b9f3d75 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -1024,9 +1024,9 @@ mod tests { let flags = settings::Flags::new(settings::builder()); match verify_function(&func, &flags) { Ok(()) => {} - Err(err) => { + Err(_err) => { #[cfg(feature = "std")] - panic!(err.message); + panic!(_err.message); #[cfg(not(feature = "std"))] panic!("function failed to verify"); } @@ -1203,9 +1203,9 @@ mod tests { let flags = settings::Flags::new(settings::builder()); match verify_function(&func, &flags) { Ok(()) => {} - Err(err) => { + Err(_err) => { #[cfg(feature = "std")] - panic!(err.message); + panic!(_err.message); #[cfg(not(feature = "std"))] panic!("function failed to verify"); } @@ -1254,9 +1254,9 @@ mod tests { let flags = settings::Flags::new(settings::builder()); match verify_function(&func, &flags) { Ok(()) => {} - Err(err) => { + Err(_err) => { #[cfg(feature = "std")] - panic!(err.message); + panic!(_err.message); #[cfg(not(feature = "std"))] panic!("function failed to verify"); } diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index b893e3b903..e20e939ca8 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -10,7 +10,8 @@ //! The main function of this module is [`translate_module`](fn.translate_module.html). #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] -#![warn(unused_import_braces, unstable_features)] +#![warn(unused_import_braces)] +#![cfg_attr(feature = "std", warn(unstable_features))] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] From 8fa0e6da99f0052123bae633c118d6990405dc05 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 30 Apr 2018 16:46:14 -0700 Subject: [PATCH 1764/3084] Fix simplejit's memory size computations. @steffengy noticed that the code to round allocation sizes up to the neareset page size was incorrect. It was masking off the low bits rather than the high bits. Also, while here, add more comments to `Memory`'s implementation. --- lib/simplejit/src/memory.rs | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/lib/simplejit/src/memory.rs b/lib/simplejit/src/memory.rs index 1ea654e924..2158352cc8 100644 --- a/lib/simplejit/src/memory.rs +++ b/lib/simplejit/src/memory.rs @@ -4,12 +4,19 @@ use errno; use libc; use region; +/// Round `size` up to the nearest multiple of `page_size`. +fn round_up_to_page_size(size: usize, page_size: usize) -> usize { + (size + (page_size - 1)) & !(page_size - 1) +} + +/// A simple struct consisting of a pointer and length. struct PtrLen { ptr: *mut u8, len: usize, } impl PtrLen { + /// Create a new empty `PtrLen`. fn new() -> Self { Self { ptr: ptr::null_mut(), @@ -17,9 +24,11 @@ impl PtrLen { } } + /// Create a new `PtrLen` pointing to at least `size` bytes of memory, + /// suitably sized and aligned for memory protection. fn with_size(size: usize) -> Result { let page_size = region::page::size(); - let alloc_size = (size + (page_size - 1)) & (page_size - 1); + let alloc_size = round_up_to_page_size(size, page_size); unsafe { let mut ptr: *mut libc::c_void = mem::uninitialized(); let err = libc::posix_memalign(&mut ptr, page_size, alloc_size); @@ -35,6 +44,8 @@ impl PtrLen { } } +/// JIT memory manager. This manages pages of suitably aligned and +/// accessible memory. pub struct Memory { allocations: Vec, executable: usize, @@ -77,6 +88,7 @@ impl Memory { Ok(self.current.ptr) } + /// Set all memory allocated in this `Memory` up to now as executable. pub fn set_executable(&mut self) { self.finish_current(); @@ -90,6 +102,7 @@ impl Memory { } } + /// Set all memory allocated in this `Memory` up to now as readonly. pub fn set_readonly(&mut self) { self.finish_current(); @@ -106,3 +119,16 @@ impl Memory { } // TODO: Implement Drop to unprotect and deallocate the memory? + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_round_up_to_page_size() { + assert_eq!(round_up_to_page_size(0, 4096), 0); + assert_eq!(round_up_to_page_size(1, 4096), 4096); + assert_eq!(round_up_to_page_size(4096, 4096), 4096); + assert_eq!(round_up_to_page_size(4097, 4096), 8192); + } +} From 5c2ada88f5dd01fe36e258400bc612f221002f6f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 1 May 2018 05:37:56 -0700 Subject: [PATCH 1765/3084] Add support for target_os = "nebulet". (#319) --- lib/native/src/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index ff3d0632c2..f5050bea76 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -34,7 +34,7 @@ use raw_cpuid::CpuId; pub fn builders() -> Result<(settings::Builder, isa::Builder), &'static str> { let mut flag_builder = settings::builder(); - if cfg!(unix) { + if cfg!(any(unix, target_os = "nebulet")) { flag_builder.set("call_conv", "system_v").unwrap(); } else if cfg!(windows) { flag_builder.set("call_conv", "fastcall").unwrap(); From b4f9eb5e558c5e26f813d23b6447b599cf549850 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 1 May 2018 19:54:21 -0700 Subject: [PATCH 1766/3084] Bump versino to 0.8.0 --- cranelift/Cargo.toml | 22 +++++++++++----------- cranelift/publish-all.sh | 2 +- lib/codegen/Cargo.toml | 4 ++-- lib/entity/Cargo.toml | 2 +- lib/faerie/Cargo.toml | 6 +++--- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/module/Cargo.toml | 6 +++--- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/simplejit/Cargo.toml | 8 ++++---- lib/umbrella/Cargo.toml | 6 +++--- lib/wasm/Cargo.toml | 6 +++--- 13 files changed, 40 insertions(+), 40 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 587b67d986..3edf07897d 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.7.0" +version = "0.8.0" description = "Binaries for testing the Cretonne libraries" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -13,16 +13,16 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] -cretonne-codegen = { path = "lib/codegen", version = "0.7.0" } -cretonne-reader = { path = "lib/reader", version = "0.7.0" } -cretonne-frontend = { path = "lib/frontend", version = "0.7.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.7.0" } -cretonne-native = { path = "lib/native", version = "0.7.0" } -cretonne-filetests = { path = "lib/filetests", version = "0.7.0" } -cretonne-module = { path = "lib/module", version = "0.7.0" } -cretonne-faerie = { path = "lib/faerie", version = "0.7.0" } -cretonne-simplejit = { path = "lib/simplejit", version = "0.7.0" } -cretonne = { path = "lib/umbrella", version = "0.7.0" } +cretonne-codegen = { path = "lib/codegen", version = "0.8.0" } +cretonne-reader = { path = "lib/reader", version = "0.8.0" } +cretonne-frontend = { path = "lib/frontend", version = "0.8.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.8.0" } +cretonne-native = { path = "lib/native", version = "0.8.0" } +cretonne-filetests = { path = "lib/filetests", version = "0.8.0" } +cretonne-module = { path = "lib/module", version = "0.8.0" } +cretonne-faerie = { path = "lib/faerie", version = "0.8.0" } +cretonne-simplejit = { path = "lib/simplejit", version = "0.8.0" } +cretonne = { path = "lib/umbrella", version = "0.8.0" } filecheck = "0.3.0" docopt = "0.8.0" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 5e34c290e0..8adbff780e 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ cd $(dirname "$0") topdir="$(pwd)" # All the cretonne-* crates have the same version number -version="0.7.0" +version="0.8.0" # Update all of the Cargo.toml files. # diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 097a637275..26bf497347 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-codegen" -version = "0.7.0" +version = "0.8.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"] build = "build.rs" [dependencies] -cretonne-entity = { path = "../entity", version = "0.7.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.8.0", default-features = false } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } hashmap_core = { version = "0.1.4", optional = true } diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index 441d3323db..2ba2db2687 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-entity" -version = "0.7.0" +version = "0.8.0" description = "Data structures using entity references as mapping keys" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index c1e55d5102..f1923ef09e 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-faerie" -version = "0.7.0" +version = "0.8.0" authors = ["The Cretonne Project Developers"] description = "Emit Cretonne output to native object files with Faerie" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.7.0" } -cretonne-module = { path = "../module", version = "0.7.0" } +cretonne-codegen = { path = "../codegen", version = "0.8.0" } +cretonne-module = { path = "../module", version = "0.8.0" } faerie = "0.3.0" goblin = "0.0.14" failure = "0.1.1" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index add93aee0f..171ca4beae 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.7.0" +version = "0.8.0" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" publish = false [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.7.0" } -cretonne-reader = { path = "../reader", version = "0.7.0" } +cretonne-codegen = { path = "../codegen", version = "0.8.0" } +cretonne-reader = { path = "../reader", version = "0.8.0" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 006b3240b5..5dc4d38ca9 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.7.0" +version = "0.8.0" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.7.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } [features] default = ["std"] diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 747e44666f..8708478ad6 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-module" -version = "0.7.0" +version = "0.8.0" authors = ["The Cretonne Project Developers"] description = "Support for linking functions and data with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.7.0", default-features = false } -cretonne-entity = { path = "../entity", version = "0.7.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.8.0", default-features = false } hashmap_core = { version = "0.1.4", optional = true } failure = "0.1.1" diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index a5de987412..c5845b3444 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.7.0" +version = "0.8.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -8,7 +8,7 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.7.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.1.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 540d4aac83..d37ed2bbcf 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.7.0" +version = "0.8.0" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.7.0" } +cretonne-codegen = { path = "../codegen", version = "0.8.0" } [badges] maintenance = { status = "experimental" } diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index 81d438deec..6c87388761 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-simplejit" -version = "0.7.0" +version = "0.8.0" authors = ["The Cretonne Project Developers"] description = "A simple JIT library backed by Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,9 +9,9 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.7.0", default-features = false } -cretonne-module = { path = "../module", version = "0.7.0", default-features = false } -cretonne-native = { path = "../native", version = "0.7.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } +cretonne-module = { path = "../module", version = "0.8.0", default-features = false } +cretonne-native = { path = "../native", version = "0.8.0", default-features = false } region = "0.2.0" libc = { version = "0.2.40", default-features = false } errno = "0.2.3" diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml index b161fbc5ba..cd7e25dfc1 100644 --- a/lib/umbrella/Cargo.toml +++ b/lib/umbrella/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.7.0" +version = "0.8.0" description = "Umbrella for commonly-used cretonne crates" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -10,8 +10,8 @@ readme = "README.md" keywords = ["compile", "compiler", "jit"] [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.7.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.7.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.8.0", default-features = false } [features] default = ["std"] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index d180f2764b..62cfa3f872 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.7.0" +version = "0.8.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/cretonne/cretonne" @@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"] [dependencies] wasmparser = { version = "0.16.1", default-features = false } -cretonne-codegen = { path = "../codegen", version = "0.7.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.7.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.8.0", default-features = false } hashmap_core = { version = "0.1.4", optional = true } [dev-dependencies] From 7e1f157692efe38f171a94336907b7d65f919eef Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 2 May 2018 09:41:34 -0700 Subject: [PATCH 1767/3084] Use %esp rather than %rsp in x86-32 test comments. --- cranelift/filetests/isa/x86/binary32-float.cton | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/cranelift/filetests/isa/x86/binary32-float.cton b/cranelift/filetests/isa/x86/binary32-float.cton index a94fff71c1..ecd84a47d8 100644 --- a/cranelift/filetests/isa/x86/binary32-float.cton +++ b/cranelift/filetests/isa/x86/binary32-float.cton @@ -185,9 +185,9 @@ ebb0: ; asm: movss 1032(%esp), %xmm2 [-,%xmm2] v211 = fill v201 ; bin: f3 0f 10 94 24 00000408 - ; asm: movss %xmm5, 1032(%rsp) + ; asm: movss %xmm5, 1032(%esp) regspill v100, %xmm5 -> ss1 ; bin: f3 0f 11 ac 24 00000408 - ; asm: movss 1032(%rsp), %xmm5 + ; asm: movss 1032(%esp), %xmm5 regfill v100, ss1 -> %xmm5 ; bin: f3 0f 10 ac 24 00000408 ; Comparisons. @@ -400,9 +400,9 @@ ebb0: ; asm: movsd 1032(%esp), %xmm2 [-,%xmm2] v211 = fill v201 ; bin: f2 0f 10 94 24 00000408 - ; asm: movsd %xmm5, 1032(%rsp) + ; asm: movsd %xmm5, 1032(%esp) regspill v100, %xmm5 -> ss1 ; bin: f2 0f 11 ac 24 00000408 - ; asm: movsd 1032(%rsp), %xmm5 + ; asm: movsd 1032(%esp), %xmm5 regfill v100, ss1 -> %xmm5 ; bin: f2 0f 10 ac 24 00000408 ; Comparisons. From 69468915d5b8e7af48d12236537351c55fc74f5a Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Thu, 3 May 2018 06:54:55 -0700 Subject: [PATCH 1768/3084] cretonne-faerie: add a translation mechanism for LibCalls (#321) * cretonne-faerie: add a translation mechanism for LibCalls * cretonne-faerie: docs for libcall_names, rustfmt * cretonne-faerie: switch libcall naming to use a closure * travis: debug mypy version * travis: pin mypy to 0.521 mypy released 0.600 today and even with `--no-strict-optional` flag passed to it (in lib/codegen/meta/check.sh) it fails to infer the type of values that did not need type annotations in the past * faerie-backend: fix swapped nearbyint functions * cretonne-faerie: move boxing out of FaerieBuilder --- .travis.yml | 2 +- lib/faerie/src/backend.rs | 54 +++++++++++++++++++++++++++++++++++---- 2 files changed, 50 insertions(+), 6 deletions(-) diff --git a/.travis.yml b/.travis.yml index 04e455a285..0fe74d37d2 100644 --- a/.travis.yml +++ b/.travis.yml @@ -17,7 +17,7 @@ before_install: - wget https://storage.googleapis.com/wasm-llvm/builds/linux/26619/wasm-toolchain_0.1.26619_amd64.deb - sudo dpkg -i wasm-toolchain_0.1.26619_amd64.deb install: - - pip3 install --user --upgrade mypy flake8 + - pip3 install --user --upgrade mypy==0.521 flake8 - travis_wait ./check-rustfmt.sh --install script: ./test-all.sh cache: diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index f8dfe0a4d3..d338daf89b 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -30,8 +30,10 @@ pub struct FaerieBuilder { format: container::Format, faerie_target: faerie::Target, collect_traps: FaerieTrapCollection, + libcall_names: Box String>, } + impl FaerieBuilder { /// Create a new `FaerieBuilder` using the given Cretonne target, that /// can be passed to @@ -41,11 +43,17 @@ impl FaerieBuilder { /// /// `collect_traps` setting determines whether trap information is collected in a /// `FaerieTrapManifest` available in the `FaerieProduct`. + /// + /// The `libcall_names` function provides a way to translate `cretonne_codegen`'s `ir::LibCall` + /// enum to symbols. LibCalls are inserted in the IR as part of the legalization for certain + /// floating point instructions, and for stack probes. If you don't know what to use for this + /// argument, use `FaerieBuilder::default_libcall_names()`. pub fn new( isa: Box, name: String, format: container::Format, collect_traps: FaerieTrapCollection, + libcall_names: Box String>, ) -> Result { if !isa.flags().is_pic() { return Err(ModuleError::Backend( @@ -59,16 +67,36 @@ impl FaerieBuilder { format, faerie_target, collect_traps, + libcall_names, + }) + } + + /// Default names for `ir::LibCall`s. A function by this name is imported into the object as + /// part of the translation of a `ir::ExternalName::LibCall` variant. Calls to a LibCall should + /// only be inserted into the IR by the `cretonne_codegen` legalizer pass. + pub fn default_libcall_names() -> Box String> { + Box::new(move |libcall| match libcall { + ir::LibCall::Probestack => "__cretonne_probestack".to_owned(), + ir::LibCall::CeilF32 => "ceilf".to_owned(), + ir::LibCall::CeilF64 => "ceil".to_owned(), + ir::LibCall::FloorF32 => "floorf".to_owned(), + ir::LibCall::FloorF64 => "floor".to_owned(), + ir::LibCall::TruncF32 => "truncf".to_owned(), + ir::LibCall::TruncF64 => "trunc".to_owned(), + ir::LibCall::NearestF32 => "nearbyintf".to_owned(), + ir::LibCall::NearestF64 => "nearbyint".to_owned(), }) } } + /// A `FaerieBackend` implements `Backend` and emits ".o" files using the `faerie` library. pub struct FaerieBackend { isa: Box, artifact: faerie::Artifact, format: container::Format, trap_manifest: Option, + libcall_names: Box String>, } pub struct FaerieCompiledFunction {} @@ -100,6 +128,7 @@ impl Backend for FaerieBackend { FaerieTrapCollection::Enabled => Some(FaerieTrapManifest::new()), FaerieTrapCollection::Disabled => None, }, + libcall_names: builder.libcall_names, } } @@ -136,6 +165,7 @@ impl Backend for FaerieBackend { artifact: &mut self.artifact, name, namespace, + libcall_names: &self.libcall_names, }; if let Some(ref mut trap_manifest) = self.trap_manifest { @@ -334,11 +364,13 @@ fn translate_data_linkage(linkage: Linkage, writable: bool) -> faerie::Decl { } } + struct FaerieRelocSink<'a> { format: container::Format, artifact: &'a mut faerie::Artifact, name: &'a str, namespace: &'a ModuleNamespace<'a, FaerieBackend>, + libcall_names: &'a Box String>, } impl<'a> RelocSink for FaerieRelocSink<'a> { @@ -353,10 +385,22 @@ impl<'a> RelocSink for FaerieRelocSink<'a> { name: &ir::ExternalName, addend: Addend, ) { - let ref_name = if self.namespace.is_function(name) { - &self.namespace.get_function_decl(name).name - } else { - &self.namespace.get_data_decl(name).name + let ref_name: String = match name { + &ir::ExternalName::User { .. } => { + if self.namespace.is_function(name) { + self.namespace.get_function_decl(name).name.clone() + } else { + self.namespace.get_data_decl(name).name.clone() + } + } + &ir::ExternalName::LibCall(ref libcall) => { + let sym = (self.libcall_names)(*libcall); + self.artifact + .declare(sym.clone(), faerie::Decl::FunctionImport) + .expect("faerie declaration of libcall"); + sym + } + _ => panic!("invalid ExternalName {}", name), }; let addend_i32 = addend as i32; debug_assert!(i64::from(addend_i32) == addend); @@ -365,7 +409,7 @@ impl<'a> RelocSink for FaerieRelocSink<'a> { .link_with( faerie::Link { from: self.name, - to: ref_name, + to: &ref_name, at: offset as usize, }, faerie::RelocOverride { From 846a71d93b581fcdc43e692b5528b575293286fc Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 3 May 2018 12:12:19 -0700 Subject: [PATCH 1769/3084] Fixes for mypy 0.600 (#324) * Remove the mypy version constraint and set strict_optional to False. * Add type annotations for `ISA` variables. mypy 0.600 seems to require explicit annotations here. * Annotate the ISA variables in the defs.py files too. --- .travis.yml | 3 ++- lib/codegen/meta/isa/arm32/__init__.py | 3 ++- lib/codegen/meta/isa/arm32/defs.py | 2 +- lib/codegen/meta/isa/arm64/__init__.py | 3 ++- lib/codegen/meta/isa/arm64/defs.py | 2 +- lib/codegen/meta/isa/riscv/__init__.py | 3 ++- lib/codegen/meta/isa/riscv/defs.py | 2 +- lib/codegen/meta/isa/x86/__init__.py | 3 ++- lib/codegen/meta/isa/x86/defs.py | 2 +- lib/codegen/meta/mypy.ini | 1 + 10 files changed, 15 insertions(+), 9 deletions(-) diff --git a/.travis.yml b/.travis.yml index 0fe74d37d2..9329904a88 100644 --- a/.travis.yml +++ b/.travis.yml @@ -17,7 +17,8 @@ before_install: - wget https://storage.googleapis.com/wasm-llvm/builds/linux/26619/wasm-toolchain_0.1.26619_amd64.deb - sudo dpkg -i wasm-toolchain_0.1.26619_amd64.deb install: - - pip3 install --user --upgrade mypy==0.521 flake8 + - pip3 install --user --upgrade mypy flake8 + - mypy --version - travis_wait ./check-rustfmt.sh --install script: ./test-all.sh cache: diff --git a/lib/codegen/meta/isa/arm32/__init__.py b/lib/codegen/meta/isa/arm32/__init__.py index 9e0ae5a7e1..06773f8754 100644 --- a/lib/codegen/meta/isa/arm32/__init__.py +++ b/lib/codegen/meta/isa/arm32/__init__.py @@ -9,6 +9,7 @@ This target ISA generates code for ARMv7 and ARMv8 CPUs in 32-bit mode from __future__ import absolute_import from . import defs from . import settings, registers # noqa +from cdsl.isa import TargetISA # noqa # Re-export the primary target ISA definition. -ISA = defs.ISA.finish() +ISA = defs.ISA.finish() # type: TargetISA diff --git a/lib/codegen/meta/isa/arm32/defs.py b/lib/codegen/meta/isa/arm32/defs.py index 6bba598d27..88b8c53db4 100644 --- a/lib/codegen/meta/isa/arm32/defs.py +++ b/lib/codegen/meta/isa/arm32/defs.py @@ -8,7 +8,7 @@ from cdsl.isa import TargetISA, CPUMode import base.instructions from base.legalize import narrow -ISA = TargetISA('arm32', [base.instructions.GROUP]) +ISA = TargetISA('arm32', [base.instructions.GROUP]) # type: TargetISA # CPU modes for 32-bit ARM and Thumb2. A32 = CPUMode('A32', ISA) diff --git a/lib/codegen/meta/isa/arm64/__init__.py b/lib/codegen/meta/isa/arm64/__init__.py index 3dd69feb4b..fb9005c037 100644 --- a/lib/codegen/meta/isa/arm64/__init__.py +++ b/lib/codegen/meta/isa/arm64/__init__.py @@ -8,6 +8,7 @@ ARMv8 CPUs running the Aarch64 architecture. from __future__ import absolute_import from . import defs from . import settings, registers # noqa +from cdsl.isa import TargetISA # noqa # Re-export the primary target ISA definition. -ISA = defs.ISA.finish() +ISA = defs.ISA.finish() # type: TargetISA diff --git a/lib/codegen/meta/isa/arm64/defs.py b/lib/codegen/meta/isa/arm64/defs.py index b1ed79b5d6..0350908f9b 100644 --- a/lib/codegen/meta/isa/arm64/defs.py +++ b/lib/codegen/meta/isa/arm64/defs.py @@ -8,7 +8,7 @@ from cdsl.isa import TargetISA, CPUMode import base.instructions from base.legalize import narrow -ISA = TargetISA('arm64', [base.instructions.GROUP]) +ISA = TargetISA('arm64', [base.instructions.GROUP]) # type: TargetISA A64 = CPUMode('A64', ISA) # TODO: Refine these diff --git a/lib/codegen/meta/isa/riscv/__init__.py b/lib/codegen/meta/isa/riscv/__init__.py index f40086414d..b58dd68ae2 100644 --- a/lib/codegen/meta/isa/riscv/__init__.py +++ b/lib/codegen/meta/isa/riscv/__init__.py @@ -27,6 +27,7 @@ RV32G / RV64G from __future__ import absolute_import from . import defs from . import encodings, settings, registers # noqa +from cdsl.isa import TargetISA # noqa # Re-export the primary target ISA definition. -ISA = defs.ISA.finish() +ISA = defs.ISA.finish() # type: TargetISA diff --git a/lib/codegen/meta/isa/riscv/defs.py b/lib/codegen/meta/isa/riscv/defs.py index 485dbd7ef0..404895c502 100644 --- a/lib/codegen/meta/isa/riscv/defs.py +++ b/lib/codegen/meta/isa/riscv/defs.py @@ -7,7 +7,7 @@ from __future__ import absolute_import from cdsl.isa import TargetISA, CPUMode import base.instructions -ISA = TargetISA('riscv', [base.instructions.GROUP]) +ISA = TargetISA('riscv', [base.instructions.GROUP]) # type: TargetISA # CPU modes for 32-bit and 64-bit operation. RV32 = CPUMode('RV32', ISA) diff --git a/lib/codegen/meta/isa/x86/__init__.py b/lib/codegen/meta/isa/x86/__init__.py index d87b95964a..93e71b316c 100644 --- a/lib/codegen/meta/isa/x86/__init__.py +++ b/lib/codegen/meta/isa/x86/__init__.py @@ -16,6 +16,7 @@ This target ISA generates code for x86 CPUs with two separate CPU modes: from __future__ import absolute_import from . import defs from . import encodings, settings, registers # noqa +from cdsl.isa import TargetISA # noqa # Re-export the primary target ISA definition. -ISA = defs.ISA.finish() +ISA = defs.ISA.finish() # type: TargetISA diff --git a/lib/codegen/meta/isa/x86/defs.py b/lib/codegen/meta/isa/x86/defs.py index 32c54ab620..00ac2bbbf1 100644 --- a/lib/codegen/meta/isa/x86/defs.py +++ b/lib/codegen/meta/isa/x86/defs.py @@ -9,7 +9,7 @@ import base.instructions from . import instructions as x86 from base.immediates import floatcc -ISA = TargetISA('x86', [base.instructions.GROUP, x86.GROUP]) +ISA = TargetISA('x86', [base.instructions.GROUP, x86.GROUP]) # type: TargetISA # CPU modes for 32-bit and 64-bit operation. X86_64 = CPUMode('I64', ISA) diff --git a/lib/codegen/meta/mypy.ini b/lib/codegen/meta/mypy.ini index 7046100b4c..877e4c9ff5 100644 --- a/lib/codegen/meta/mypy.ini +++ b/lib/codegen/meta/mypy.ini @@ -2,3 +2,4 @@ disallow_untyped_defs = True warn_unused_ignores = True warn_return_any = True +strict_optional = False From bb612af37a0912a69f27b2813179e5c03233960a Mon Sep 17 00:00:00 2001 From: Pat Hickey Date: Thu, 3 May 2018 18:09:07 -0700 Subject: [PATCH 1770/3084] x86 recipes: emit StackOverflow trap for all sp-relative loads and stores (#325) * x86 recipes: emit StackOverflow trap for all sp-relative loads and stores * x86 recipes: emit StackOverflow trap for push and pop * x86 binary filetests: add stk_ovf trap annotations --- .../filetests/isa/x86/binary32-float.cton | 24 ++++---- cranelift/filetests/isa/x86/binary32.cton | 22 +++---- .../filetests/isa/x86/binary64-float.cton | 24 ++++---- cranelift/filetests/isa/x86/binary64-pic.cton | 16 ++--- cranelift/filetests/isa/x86/binary64.cton | 58 +++++++++---------- lib/codegen/meta/isa/x86/recipes.py | 13 +++++ 6 files changed, 85 insertions(+), 72 deletions(-) diff --git a/cranelift/filetests/isa/x86/binary32-float.cton b/cranelift/filetests/isa/x86/binary32-float.cton index ecd84a47d8..702972a241 100644 --- a/cranelift/filetests/isa/x86/binary32-float.cton +++ b/cranelift/filetests/isa/x86/binary32-float.cton @@ -176,19 +176,19 @@ ebb0: ; Spill / Fill. ; asm: movss %xmm5, 1032(%esp) - [-,ss1] v200 = spill v100 ; bin: f3 0f 11 ac 24 00000408 + [-,ss1] v200 = spill v100 ; bin: stk_ovf f3 0f 11 ac 24 00000408 ; asm: movss %xmm2, 1032(%esp) - [-,ss1] v201 = spill v101 ; bin: f3 0f 11 94 24 00000408 + [-,ss1] v201 = spill v101 ; bin: stk_ovf f3 0f 11 94 24 00000408 ; asm: movss 1032(%esp), %xmm5 - [-,%xmm5] v210 = fill v200 ; bin: f3 0f 10 ac 24 00000408 + [-,%xmm5] v210 = fill v200 ; bin: stk_ovf f3 0f 10 ac 24 00000408 ; asm: movss 1032(%esp), %xmm2 - [-,%xmm2] v211 = fill v201 ; bin: f3 0f 10 94 24 00000408 + [-,%xmm2] v211 = fill v201 ; bin: stk_ovf f3 0f 10 94 24 00000408 ; asm: movss %xmm5, 1032(%esp) - regspill v100, %xmm5 -> ss1 ; bin: f3 0f 11 ac 24 00000408 + regspill v100, %xmm5 -> ss1 ; bin: stk_ovf f3 0f 11 ac 24 00000408 ; asm: movss 1032(%esp), %xmm5 - regfill v100, ss1 -> %xmm5 ; bin: f3 0f 10 ac 24 00000408 + regfill v100, ss1 -> %xmm5 ; bin: stk_ovf f3 0f 10 ac 24 00000408 ; Comparisons. ; @@ -391,19 +391,19 @@ ebb0: ; Spill / Fill. ; asm: movsd %xmm5, 1032(%esp) - [-,ss1] v200 = spill v100 ; bin: f2 0f 11 ac 24 00000408 + [-,ss1] v200 = spill v100 ; bin: stk_ovf f2 0f 11 ac 24 00000408 ; asm: movsd %xmm2, 1032(%esp) - [-,ss1] v201 = spill v101 ; bin: f2 0f 11 94 24 00000408 + [-,ss1] v201 = spill v101 ; bin: stk_ovf f2 0f 11 94 24 00000408 ; asm: movsd 1032(%esp), %xmm5 - [-,%xmm5] v210 = fill v200 ; bin: f2 0f 10 ac 24 00000408 + [-,%xmm5] v210 = fill v200 ; bin: stk_ovf f2 0f 10 ac 24 00000408 ; asm: movsd 1032(%esp), %xmm2 - [-,%xmm2] v211 = fill v201 ; bin: f2 0f 10 94 24 00000408 + [-,%xmm2] v211 = fill v201 ; bin: stk_ovf f2 0f 10 94 24 00000408 ; asm: movsd %xmm5, 1032(%esp) - regspill v100, %xmm5 -> ss1 ; bin: f2 0f 11 ac 24 00000408 + regspill v100, %xmm5 -> ss1 ; bin: stk_ovf f2 0f 11 ac 24 00000408 ; asm: movsd 1032(%esp), %xmm5 - regfill v100, ss1 -> %xmm5 ; bin: f2 0f 10 ac 24 00000408 + regfill v100, ss1 -> %xmm5 ; bin: stk_ovf f2 0f 10 ac 24 00000408 ; Comparisons. ; diff --git a/cranelift/filetests/isa/x86/binary32.cton b/cranelift/filetests/isa/x86/binary32.cton index 9ce08fb66d..feeabaa637 100644 --- a/cranelift/filetests/isa/x86/binary32.cton +++ b/cranelift/filetests/isa/x86/binary32.cton @@ -352,7 +352,7 @@ ebb0: [-,%rsi] v351 = bint.i32 v301 ; bin: 0f b6 f2 ; asm: call foo - call fn0() ; bin: e8 PCRel4(%foo-4) 00000000 + call fn0() ; bin: stk_ovf e8 PCRel4(%foo-4) 00000000 ; asm: movl $0, %ecx [-,%rcx] v400 = func_addr.i32 fn0 ; bin: b9 Abs4(%foo) 00000000 @@ -360,9 +360,9 @@ ebb0: [-,%rsi] v401 = func_addr.i32 fn0 ; bin: be Abs4(%foo) 00000000 ; asm: call *%ecx - call_indirect sig0, v400() ; bin: ff d1 + call_indirect sig0, v400() ; bin: stk_ovf ff d1 ; asm: call *%esi - call_indirect sig0, v401() ; bin: ff d6 + call_indirect sig0, v401() ; bin: stk_ovf ff d6 ; asm: movl $0, %ecx [-,%rcx] v450 = globalsym_addr.i32 gv0 ; bin: b9 Abs4(%some_gv) 00000000 @@ -372,25 +372,25 @@ ebb0: ; Spill / Fill. ; asm: movl %ecx, 1032(%esp) - [-,ss1] v500 = spill v1 ; bin: 89 8c 24 00000408 + [-,ss1] v500 = spill v1 ; bin: stk_ovf 89 8c 24 00000408 ; asm: movl %esi, 1032(%esp) - [-,ss1] v501 = spill v2 ; bin: 89 b4 24 00000408 + [-,ss1] v501 = spill v2 ; bin: stk_ovf 89 b4 24 00000408 ; asm: movl 1032(%esp), %ecx - [-,%rcx] v510 = fill v500 ; bin: 8b 8c 24 00000408 + [-,%rcx] v510 = fill v500 ; bin: stk_ovf 8b 8c 24 00000408 ; asm: movl 1032(%esp), %esi - [-,%rsi] v511 = fill v501 ; bin: 8b b4 24 00000408 + [-,%rsi] v511 = fill v501 ; bin: stk_ovf 8b b4 24 00000408 ; asm: movl %ecx, 1032(%esp) - regspill v1, %rcx -> ss1 ; bin: 89 8c 24 00000408 + regspill v1, %rcx -> ss1 ; bin: stk_ovf 89 8c 24 00000408 ; asm: movl 1032(%esp), %ecx - regfill v1, ss1 -> %rcx ; bin: 8b 8c 24 00000408 + regfill v1, ss1 -> %rcx ; bin: stk_ovf 8b 8c 24 00000408 ; Push and Pop ; asm: pushl %ecx - x86_push v1 ; bin: 51 + x86_push v1 ; bin: stk_ovf 51 ; asm: popl %ecx - [-,%rcx] v512 = x86_pop.i32 ; bin: 59 + [-,%rcx] v512 = x86_pop.i32 ; bin: stk_ovf 59 ; Adjust Stack Pointer Up ; asm: addl $64, %esp diff --git a/cranelift/filetests/isa/x86/binary64-float.cton b/cranelift/filetests/isa/x86/binary64-float.cton index e9fcc99e63..fc38271485 100644 --- a/cranelift/filetests/isa/x86/binary64-float.cton +++ b/cranelift/filetests/isa/x86/binary64-float.cton @@ -190,19 +190,19 @@ ebb0: ; Spill / Fill. ; asm: movss %xmm5, 1032(%rsp) - [-,ss1] v200 = spill v100 ; bin: f3 0f 11 ac 24 00000408 + [-,ss1] v200 = spill v100 ; bin: stk_ovf f3 0f 11 ac 24 00000408 ; asm: movss %xmm10, 1032(%rsp) - [-,ss1] v201 = spill v101 ; bin: f3 44 0f 11 94 24 00000408 + [-,ss1] v201 = spill v101 ; bin: stk_ovf f3 44 0f 11 94 24 00000408 ; asm: movss 1032(%rsp), %xmm5 - [-,%xmm5] v210 = fill v200 ; bin: f3 0f 10 ac 24 00000408 + [-,%xmm5] v210 = fill v200 ; bin: stk_ovf f3 0f 10 ac 24 00000408 ; asm: movss 1032(%rsp), %xmm10 - [-,%xmm10] v211 = fill v201 ; bin: f3 44 0f 10 94 24 00000408 + [-,%xmm10] v211 = fill v201 ; bin: stk_ovf f3 44 0f 10 94 24 00000408 ; asm: movss %xmm5, 1032(%rsp) - regspill v100, %xmm5 -> ss1 ; bin: f3 0f 11 ac 24 00000408 + regspill v100, %xmm5 -> ss1 ; bin: stk_ovf f3 0f 11 ac 24 00000408 ; asm: movss 1032(%rsp), %xmm5 - regfill v100, ss1 -> %xmm5 ; bin: f3 0f 10 ac 24 00000408 + regfill v100, ss1 -> %xmm5 ; bin: stk_ovf f3 0f 10 ac 24 00000408 ; Comparisons. ; @@ -425,19 +425,19 @@ ebb0: ; Spill / Fill. ; asm: movsd %xmm5, 1032(%rsp) - [-,ss1] v200 = spill v100 ; bin: f2 0f 11 ac 24 00000408 + [-,ss1] v200 = spill v100 ; bin: stk_ovf f2 0f 11 ac 24 00000408 ; asm: movsd %xmm10, 1032(%rsp) - [-,ss1] v201 = spill v101 ; bin: f2 44 0f 11 94 24 00000408 + [-,ss1] v201 = spill v101 ; bin: stk_ovf f2 44 0f 11 94 24 00000408 ; asm: movsd 1032(%rsp), %xmm5 - [-,%xmm5] v210 = fill v200 ; bin: f2 0f 10 ac 24 00000408 + [-,%xmm5] v210 = fill v200 ; bin: stk_ovf f2 0f 10 ac 24 00000408 ; asm: movsd 1032(%rsp), %xmm10 - [-,%xmm10] v211 = fill v201 ; bin: f2 44 0f 10 94 24 00000408 + [-,%xmm10] v211 = fill v201 ; bin: stk_ovf f2 44 0f 10 94 24 00000408 ; asm: movsd %xmm5, 1032(%rsp) - regspill v100, %xmm5 -> ss1 ; bin: f2 0f 11 ac 24 00000408 + regspill v100, %xmm5 -> ss1 ; bin: stk_ovf f2 0f 11 ac 24 00000408 ; asm: movsd 1032(%rsp), %xmm5 - regfill v100, ss1 -> %xmm5 ; bin: f2 0f 10 ac 24 00000408 + regfill v100, ss1 -> %xmm5 ; bin: stk_ovf f2 0f 10 ac 24 00000408 ; Comparisons. ; diff --git a/cranelift/filetests/isa/x86/binary64-pic.cton b/cranelift/filetests/isa/x86/binary64-pic.cton index ff9cfc42ac..8d3b2e54ab 100644 --- a/cranelift/filetests/isa/x86/binary64-pic.cton +++ b/cranelift/filetests/isa/x86/binary64-pic.cton @@ -31,7 +31,7 @@ ebb0: ; Colocated functions. ; asm: call foo - call fn1() ; bin: e8 PCRel4(%bar-4) 00000000 + call fn1() ; bin: stk_ovf e8 PCRel4(%bar-4) 00000000 ; asm: lea 0x0(%rip), %rax [-,%rax] v0 = func_addr.i64 fn1 ; bin: 48 8d 05 PCRel4(%bar-4) 00000000 @@ -41,16 +41,16 @@ ebb0: [-,%r10] v2 = func_addr.i64 fn1 ; bin: 4c 8d 15 PCRel4(%bar-4) 00000000 ; asm: call *%rax - call_indirect sig0, v0() ; bin: ff d0 + call_indirect sig0, v0() ; bin: stk_ovf ff d0 ; asm: call *%rsi - call_indirect sig0, v1() ; bin: ff d6 + call_indirect sig0, v1() ; bin: stk_ovf ff d6 ; asm: call *%r10 - call_indirect sig0, v2() ; bin: 41 ff d2 + call_indirect sig0, v2() ; bin: stk_ovf 41 ff d2 ; Non-colocated functions. ; asm: call foo@PLT - call fn0() ; bin: e8 PLTRel4(%foo-4) 00000000 + call fn0() ; bin: stk_ovf e8 PLTRel4(%foo-4) 00000000 ; asm: mov 0x0(%rip), %rax [-,%rax] v100 = func_addr.i64 fn0 ; bin: 48 8b 05 GOTPCRel4(%foo-4) 00000000 @@ -60,11 +60,11 @@ ebb0: [-,%r10] v102 = func_addr.i64 fn0 ; bin: 4c 8b 15 GOTPCRel4(%foo-4) 00000000 ; asm: call *%rax - call_indirect sig0, v100() ; bin: ff d0 + call_indirect sig0, v100() ; bin: stk_ovf ff d0 ; asm: call *%rsi - call_indirect sig0, v101() ; bin: ff d6 + call_indirect sig0, v101() ; bin: stk_ovf ff d6 ; asm: call *%r10 - call_indirect sig0, v102() ; bin: 41 ff d2 + call_indirect sig0, v102() ; bin: stk_ovf 41 ff d2 ; asm: mov 0x0(%rip), %rcx [-,%rcx] v3 = globalsym_addr.i64 gv0 ; bin: 48 8b 0d GOTPCRel4(%some_gv-4) 00000000 diff --git a/cranelift/filetests/isa/x86/binary64.cton b/cranelift/filetests/isa/x86/binary64.cton index a9b96a86dc..af625fa1bb 100644 --- a/cranelift/filetests/isa/x86/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.cton @@ -477,7 +477,7 @@ ebb0: ; Colocated functions. ; asm: call bar - call fn1() ; bin: e8 PCRel4(%bar-4) 00000000 + call fn1() ; bin: stk_ovf e8 PCRel4(%bar-4) 00000000 ; asm: lea 0x0(%rip), %rcx [-,%rcx] v400 = func_addr.i64 fn1 ; bin: 48 8d 0d PCRel4(%bar-4) 00000000 @@ -487,11 +487,11 @@ ebb0: [-,%r10] v402 = func_addr.i64 fn1 ; bin: 4c 8d 15 PCRel4(%bar-4) 00000000 ; asm: call *%rcx - call_indirect sig0, v400() ; bin: ff d1 + call_indirect sig0, v400() ; bin: stk_ovf ff d1 ; asm: call *%rsi - call_indirect sig0, v401() ; bin: ff d6 + call_indirect sig0, v401() ; bin: stk_ovf ff d6 ; asm: call *%r10 - call_indirect sig0, v402() ; bin: 41 ff d2 + call_indirect sig0, v402() ; bin: stk_ovf 41 ff d2 ; Non-colocated functions. Note that there is no non-colocated non-PIC call. @@ -503,11 +503,11 @@ ebb0: [-,%r10] v412 = func_addr.i64 fn0 ; bin: 49 ba Abs8(%foo) 0000000000000000 ; asm: call *%rcx - call_indirect sig0, v410() ; bin: ff d1 + call_indirect sig0, v410() ; bin: stk_ovf ff d1 ; asm: call *%rsi - call_indirect sig0, v411() ; bin: ff d6 + call_indirect sig0, v411() ; bin: stk_ovf ff d6 ; asm: call *%r10 - call_indirect sig0, v412() ; bin: 41 ff d2 + call_indirect sig0, v412() ; bin: stk_ovf 41 ff d2 ; asm: movabsq $-1, %rcx [-,%rcx] v450 = globalsym_addr.i64 gv0 ; bin: 48 b9 Abs8(%some_gv) 0000000000000000 @@ -519,33 +519,33 @@ ebb0: ; Spill / Fill. ; asm: movq %rcx, 1032(%rsp) - [-,ss1] v500 = spill v1 ; bin: 48 89 8c 24 00000408 + [-,ss1] v500 = spill v1 ; bin: stk_ovf 48 89 8c 24 00000408 ; asm: movq %rsi, 1032(%rsp) - [-,ss1] v501 = spill v2 ; bin: 48 89 b4 24 00000408 + [-,ss1] v501 = spill v2 ; bin: stk_ovf 48 89 b4 24 00000408 ; asm: movq %r10, 1032(%rsp) - [-,ss1] v502 = spill v3 ; bin: 4c 89 94 24 00000408 + [-,ss1] v502 = spill v3 ; bin: stk_ovf 4c 89 94 24 00000408 ; asm: movq 1032(%rsp), %rcx - [-,%rcx] v510 = fill v500 ; bin: 48 8b 8c 24 00000408 + [-,%rcx] v510 = fill v500 ; bin: stk_ovf 48 8b 8c 24 00000408 ; asm: movq 1032(%rsp), %rsi - [-,%rsi] v511 = fill v501 ; bin: 48 8b b4 24 00000408 + [-,%rsi] v511 = fill v501 ; bin: stk_ovf 48 8b b4 24 00000408 ; asm: movq 1032(%rsp), %r10 - [-,%r10] v512 = fill v502 ; bin: 4c 8b 94 24 00000408 + [-,%r10] v512 = fill v502 ; bin: stk_ovf 4c 8b 94 24 00000408 ; asm: movq %rcx, 1032(%rsp) - regspill v1, %rcx -> ss1 ; bin: 48 89 8c 24 00000408 + regspill v1, %rcx -> ss1 ; bin: stk_ovf 48 89 8c 24 00000408 ; asm: movq 1032(%rsp), %rcx - regfill v1, ss1 -> %rcx ; bin: 48 8b 8c 24 00000408 + regfill v1, ss1 -> %rcx ; bin: stk_ovf 48 8b 8c 24 00000408 ; Push and Pop ; asm: pushq %rcx - x86_push v1 ; bin: 51 + x86_push v1 ; bin: stk_ovf 51 ; asm: pushq %r10 - x86_push v3 ; bin: 41 52 + x86_push v3 ; bin: stk_ovf 41 52 ; asm: popq %rcx - [-,%rcx] v513 = x86_pop.i64 ; bin: 59 + [-,%rcx] v513 = x86_pop.i64 ; bin: stk_ovf 59 ; asm: popq %r10 - [-,%r10] v514 = x86_pop.i64 ; bin: 41 5a + [-,%r10] v514 = x86_pop.i64 ; bin: stk_ovf 41 5a ; Adjust Stack Pointer Up ; asm: addq $64, %rsp @@ -732,9 +732,9 @@ ebb0: [-,%rcx] v1 = iconst.i64 1 ; asm: movq %rcx, 8(%rsp) - [-,ss1] v10 = spill v1 ; bin: 48 89 8c 24 00000008 + [-,ss1] v10 = spill v1 ; bin: stk_ovf 48 89 8c 24 00000008 ; asm: movq %rcx, (%rsp) - [-,ss2] v11 = spill v1 ; bin: 48 89 8c 24 00000000 + [-,ss2] v11 = spill v1 ; bin: stk_ovf 48 89 8c 24 00000000 return } @@ -1094,23 +1094,23 @@ ebb0: ; Spill / Fill. ; asm: movl %ecx, 1032(%rsp) - [-,ss1] v500 = spill v1 ; bin: 89 8c 24 00000408 + [-,ss1] v500 = spill v1 ; bin: stk_ovf 89 8c 24 00000408 ; asm: movl %esi, 1032(%rsp) - [-,ss1] v501 = spill v2 ; bin: 89 b4 24 00000408 + [-,ss1] v501 = spill v2 ; bin: stk_ovf 89 b4 24 00000408 ; asm: movl %r10d, 1032(%rsp) - [-,ss1] v502 = spill v3 ; bin: 44 89 94 24 00000408 + [-,ss1] v502 = spill v3 ; bin: stk_ovf 44 89 94 24 00000408 ; asm: movl 1032(%rsp), %ecx - [-,%rcx] v510 = fill v500 ; bin: 8b 8c 24 00000408 + [-,%rcx] v510 = fill v500 ; bin: stk_ovf 8b 8c 24 00000408 ; asm: movl 1032(%rsp), %esi - [-,%rsi] v511 = fill v501 ; bin: 8b b4 24 00000408 + [-,%rsi] v511 = fill v501 ; bin: stk_ovf 8b b4 24 00000408 ; asm: movl 1032(%rsp), %r10d - [-,%r10] v512 = fill v502 ; bin: 44 8b 94 24 00000408 + [-,%r10] v512 = fill v502 ; bin: stk_ovf 44 8b 94 24 00000408 ; asm: movl %ecx, 1032(%rsp) - regspill v1, %rcx -> ss1 ; bin: 89 8c 24 00000408 + regspill v1, %rcx -> ss1 ; bin: stk_ovf 89 8c 24 00000408 ; asm: movl 1032(%rsp), %ecx - regfill v1, ss1 -> %rcx ; bin: 8b 8c 24 00000408 + regfill v1, ss1 -> %rcx ; bin: stk_ovf 8b 8c 24 00000408 ; asm: cmpl %esi, %ecx [-,%rflags] v520 = ifcmp v1, v2 ; bin: 39 f1 diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py index bfe57d8bcb..ccd85c0fef 100644 --- a/lib/codegen/meta/isa/x86/recipes.py +++ b/lib/codegen/meta/isa/x86/recipes.py @@ -546,12 +546,14 @@ pu_iq = TailRecipe( pushq = TailRecipe( 'pushq', Unary, size=0, ins=GPR, outs=(), emit=''' + sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); PUT_OP(bits | (in_reg0 & 7), rex1(in_reg0), sink); ''') popq = TailRecipe( 'popq', NullAry, size=0, ins=(), outs=GPR, emit=''' + sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); ''') @@ -851,6 +853,7 @@ spillSib32 = TailRecipe( 'spillSib32', Unary, size=6, ins=GPR, outs=StackGPR32, clobbers_flags=False, emit=''' + sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); let base = stk_base(out_stk0.base); PUT_OP(bits, rex2(base, in_reg0), sink); modrm_sib_disp32(in_reg0, sink); @@ -863,6 +866,7 @@ fspillSib32 = TailRecipe( 'fspillSib32', Unary, size=6, ins=FPR, outs=StackFPR32, clobbers_flags=False, emit=''' + sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); let base = stk_base(out_stk0.base); PUT_OP(bits, rex2(base, in_reg0), sink); modrm_sib_disp32(in_reg0, sink); @@ -875,6 +879,7 @@ regspill32 = TailRecipe( 'regspill32', RegSpill, size=6, ins=GPR, outs=(), clobbers_flags=False, emit=''' + sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); let dst = StackRef::sp(dst, &func.stack_slots); let base = stk_base(dst.base); PUT_OP(bits, rex2(base, src), sink); @@ -888,6 +893,7 @@ fregspill32 = TailRecipe( 'fregspill32', RegSpill, size=6, ins=FPR, outs=(), clobbers_flags=False, emit=''' + sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); let dst = StackRef::sp(dst, &func.stack_slots); let base = stk_base(dst.base); PUT_OP(bits, rex2(base, src), sink); @@ -991,6 +997,7 @@ fillSib32 = TailRecipe( 'fillSib32', Unary, size=6, ins=StackGPR32, outs=GPR, clobbers_flags=False, emit=''' + sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); let base = stk_base(in_stk0.base); PUT_OP(bits, rex2(base, out_reg0), sink); modrm_sib_disp32(out_reg0, sink); @@ -1003,6 +1010,7 @@ ffillSib32 = TailRecipe( 'ffillSib32', Unary, size=6, ins=StackFPR32, outs=FPR, clobbers_flags=False, emit=''' + sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); let base = stk_base(in_stk0.base); PUT_OP(bits, rex2(base, out_reg0), sink); modrm_sib_disp32(out_reg0, sink); @@ -1015,6 +1023,7 @@ regfill32 = TailRecipe( 'regfill32', RegFill, size=6, ins=StackGPR32, outs=(), clobbers_flags=False, emit=''' + sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); let src = StackRef::sp(src, &func.stack_slots); let base = stk_base(src.base); PUT_OP(bits, rex2(base, dst), sink); @@ -1028,6 +1037,7 @@ fregfill32 = TailRecipe( 'fregfill32', RegFill, size=6, ins=StackFPR32, outs=(), clobbers_flags=False, emit=''' + sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); let src = StackRef::sp(src, &func.stack_slots); let base = stk_base(src.base); PUT_OP(bits, rex2(base, dst), sink); @@ -1042,6 +1052,7 @@ fregfill32 = TailRecipe( call_id = TailRecipe( 'call_id', Call, size=4, ins=(), outs=(), emit=''' + sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); PUT_OP(bits, BASE_REX, sink); // The addend adjusts for the difference between the end of the // instruction and the beginning of the immediate field. @@ -1054,6 +1065,7 @@ call_id = TailRecipe( call_plt_id = TailRecipe( 'call_plt_id', Call, size=4, ins=(), outs=(), emit=''' + sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); PUT_OP(bits, BASE_REX, sink); sink.reloc_external(Reloc::X86PLTRel4, &func.dfg.ext_funcs[func_ref].name, @@ -1064,6 +1076,7 @@ call_plt_id = TailRecipe( call_r = TailRecipe( 'call_r', CallIndirect, size=1, ins=GPR, outs=(), emit=''' + sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); PUT_OP(bits, rex1(in_reg0), sink); modrm_r_bits(in_reg0, bits, sink); ''') From 5b69930e03dc67b65777ba3f98f088ec0fca7d81 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sat, 5 May 2018 11:01:49 -0500 Subject: [PATCH 1771/3084] Legalize bnot using xor with -1. --- cranelift/filetests/isa/riscv/expand-i32.cton | 3 ++- lib/codegen/meta/base/legalize.py | 8 ++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/cranelift/filetests/isa/riscv/expand-i32.cton b/cranelift/filetests/isa/riscv/expand-i32.cton index 890352e72f..30048622ca 100644 --- a/cranelift/filetests/isa/riscv/expand-i32.cton +++ b/cranelift/filetests/isa/riscv/expand-i32.cton @@ -32,7 +32,8 @@ ebb0(v0: i32): function %bitclear(i32, i32) -> i32 { ebb0(v0: i32, v1: i32): v2 = band_not v0, v1 - ; check: bnot + ; check: iconst.i32 -1 + ; check: bxor ; check: band return v2 } diff --git a/lib/codegen/meta/base/legalize.py b/lib/codegen/meta/base/legalize.py index 95a58987d3..0a7e8ea7b3 100644 --- a/lib/codegen/meta/base/legalize.py +++ b/lib/codegen/meta/base/legalize.py @@ -252,6 +252,14 @@ for inst_not, inst in [ a << inst(x, a1) )) +# Expand bnot using xor. +expand.legalize( + a << bnot(x), + Rtl( + y << iconst(imm64(-1)), + a << bxor(x, y) + )) + # Floating-point sign manipulations. for ty, minus_zero in [ (types.f32, f32const(ieee32.bits(0x80000000))), From 09f883182dc0ddb07b6b1a51ea0f88a485bcd993 Mon Sep 17 00:00:00 2001 From: Ty Overby Date: Mon, 7 May 2018 09:10:01 -0700 Subject: [PATCH 1772/3084] document that low bitsize integers don't have complete arithmetic support (#320) * document that low bitsize integers don't have complete arithmetic support --- lib/codegen/meta/cdsl/types.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/codegen/meta/cdsl/types.py b/lib/codegen/meta/cdsl/types.py index 3f2593fb78..c0e989e710 100644 --- a/lib/codegen/meta/cdsl/types.py +++ b/lib/codegen/meta/cdsl/types.py @@ -200,10 +200,15 @@ class IntType(LaneType): def __init__(self, bits): # type: (int) -> None assert bits > 0, 'IntType must have positive number of bits' + warning = "" + if bits < 32: + warning += "\nWARNING: " + warning += "arithmetic on {}bit integers is incomplete".format( + bits) super(IntType, self).__init__( name='i{:d}'.format(bits), membytes=bits // 8, - doc="An integer type with {} bits.".format(bits)) + doc="An integer type with {} bits.{}".format(bits, warning)) self.bits = bits def __repr__(self): From 5aa84a744b738a777badf7c892bf8e388c184ede Mon Sep 17 00:00:00 2001 From: Steffen Butzer Date: Wed, 9 May 2018 20:18:30 +0200 Subject: [PATCH 1773/3084] windows fastcall (x64) call convention (#314) * initial set of work for windows fastcall (x64) call convention - call conventions: rename `fastcall` to `windows_fastcall` - add initial set of filetests - ensure arguments are written after the shadow space/store (offset-wise) The shadow space available before the arguments (range 0..32) is not used as spill space yet. * address review feedback --- .../isa/x86/windows_fastcall_x64.cton | 34 +++++ lib/codegen/meta/base/settings.py | 12 +- lib/codegen/src/ir/extfunc.rs | 2 +- lib/codegen/src/isa/x86/abi.rs | 144 ++++++++++++++++-- lib/native/src/lib.rs | 2 +- lib/simplejit/Cargo.toml | 3 + lib/simplejit/src/backend.rs | 35 +++++ lib/simplejit/src/lib.rs | 3 + lib/simplejit/src/memory.rs | 27 ++++ 9 files changed, 246 insertions(+), 16 deletions(-) create mode 100644 cranelift/filetests/isa/x86/windows_fastcall_x64.cton diff --git a/cranelift/filetests/isa/x86/windows_fastcall_x64.cton b/cranelift/filetests/isa/x86/windows_fastcall_x64.cton new file mode 100644 index 0000000000..204bd5bb70 --- /dev/null +++ b/cranelift/filetests/isa/x86/windows_fastcall_x64.cton @@ -0,0 +1,34 @@ +test compile +set is_64bit +set opt_level=best +set is_pic +isa x86 haswell + +; check if for one arg we use the right register +function %one_arg(i64) windows_fastcall { +ebb0(v0: i64): + return +} +; check: function %one_arg(i64 [%rcx], i64 fp [%rbp]) -> i64 fp [%rbp] windows_fastcall { +; nextln: ss0 = incoming_arg 16, offset -48 + +; check if we still use registers for 4 arguments +function %four_args(i64, i64, i64, i64) windows_fastcall { +ebb0(v0: i64, v1: i64, v2: i64, v3: i64): + return +} +; check: function %four_args(i64 [%rcx], i64 [%rdx], i64 [%r8], i64 [%r9], i64 fp [%rbp]) -> i64 fp [%rbp] windows_fastcall { + +; check if float arguments are passed through XMM registers +function %four_float_args(f64, f64, f64, f64) windows_fastcall { +ebb0(v0: f64, v1: f64, v2: f64, v3: f64): + return +} +; check: function %four_float_args(f64 [%xmm0], f64 [%xmm1], f64 [%xmm2], f64 [%xmm3], i64 fp [%rbp]) -> i64 fp [%rbp] windows_fastcall { + +; check if we use stack space for > 4 arguments +function %five_args(i64, i64, i64, i64, i64) windows_fastcall { +ebb0(v0: i64, v1: i64, v2: i64, v3: i64, v4: i64): + return +} +; check: function %five_args(i64 [%rcx], i64 [%rdx], i64 [%r8], i64 [%r9], i64 [32], i64 fp [%rbp]) -> i64 fp [%rbp] windows_fastcall { diff --git a/lib/codegen/meta/base/settings.py b/lib/codegen/meta/base/settings.py index 32df3d63b4..a5107e5b4c 100644 --- a/lib/codegen/meta/base/settings.py +++ b/lib/codegen/meta/base/settings.py @@ -36,14 +36,22 @@ call_conv = EnumSetting( - fast: not-ABI-stable convention for best performance - cold: not-ABI-stable convention for infrequently executed code - system_v: System V-style convention used on many platforms - - fastcall: Windows "fastcall" convention, also used for x64 and ARM + - windows_fastcall: Windows "fastcall" convention, also used for + x64 and ARM - baldrdash: SpiderMonkey WebAssembly convention - probestack: specialized convention for the probestack function The default calling convention may be overridden by individual functions. """, - 'fast', 'cold', 'system_v', 'fastcall', 'baldrdash', 'probestack') + + 'fast', + 'cold', + 'system_v', + 'windows_fastcall', + 'baldrdash', + 'probestack' +) # Note that Cretonne doesn't currently need an is_pie flag, because PIE is just # PIC where symbols can't be pre-empted, which can be expressed with the diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs index 540083321e..95f86be8fd 100644 --- a/lib/codegen/src/ir/extfunc.rs +++ b/lib/codegen/src/ir/extfunc.rs @@ -382,7 +382,7 @@ mod tests { CallConv::Fast, CallConv::Cold, CallConv::SystemV, - CallConv::Fastcall, + CallConv::WindowsFastcall, CallConv::Baldrdash, ] { diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs index 5ab9febb29..15b7d94654 100644 --- a/lib/codegen/src/isa/x86/abi.rs +++ b/lib/codegen/src/isa/x86/abi.rs @@ -22,6 +22,12 @@ static ARG_GPRS: [RU; 6] = [RU::rdi, RU::rsi, RU::rdx, RU::rcx, RU::r8, RU::r9]; /// Return value registers. static RET_GPRS: [RU; 3] = [RU::rax, RU::rdx, RU::rcx]; +/// Argument registers for x86-64, when using windows fastcall +static ARG_GPRS_WIN_FASTCALL_X64: [RU; 4] = [RU::rcx, RU::rdx, RU::r8, RU::r9]; + +/// Return value registers for x86-64, when using windows fastcall +static RET_GPRS_WIN_FASTCALL_X64: [RU; 1] = [RU::rax]; + struct Args { pointer_bytes: u32, pointer_bits: u16, @@ -36,6 +42,14 @@ struct Args { impl Args { fn new(bits: u16, gpr: &'static [RU], fpr_limit: usize, call_conv: CallConv) -> Self { + let offset = if let CallConv::WindowsFastcall = call_conv { + // [1] "The caller is responsible for allocating space for parameters to the callee, + // and must always allocate sufficient space to store four register parameters" + 32 + } else { + 0 + }; + Self { pointer_bytes: u32::from(bits) / 8, pointer_bits: bits, @@ -44,7 +58,7 @@ impl Args { gpr_used: 0, fpr_limit, fpr_used: 0, - offset: 0, + offset, call_conv, } } @@ -120,7 +134,11 @@ pub fn legalize_signature(sig: &mut ir::Signature, flags: &shared_settings::Flag if flags.is_64bit() { bits = 64; - args = Args::new(bits, &ARG_GPRS, 8, sig.call_conv); + args = if sig.call_conv == CallConv::WindowsFastcall { + Args::new(bits, &ARG_GPRS_WIN_FASTCALL_X64[..], 4, sig.call_conv) + } else { + Args::new(bits, &ARG_GPRS[..], 8, sig.call_conv) + }; } else { bits = 32; args = Args::new(bits, &[], 0, sig.call_conv); @@ -128,7 +146,13 @@ pub fn legalize_signature(sig: &mut ir::Signature, flags: &shared_settings::Flag legalize_args(&mut sig.params, &mut args); - let mut rets = Args::new(bits, &RET_GPRS, 2, sig.call_conv); + let regs = if sig.call_conv == CallConv::WindowsFastcall { + &RET_GPRS_WIN_FASTCALL_X64[..] + } else { + &RET_GPRS[..] + }; + + let mut rets = Args::new(bits, regs, 2, sig.call_conv); legalize_args(&mut sig.returns, &mut rets); } @@ -161,7 +185,24 @@ pub fn allocatable_registers(_func: &ir::Function, flags: &shared_settings::Flag /// Get the set of callee-saved registers. fn callee_saved_gprs(flags: &shared_settings::Flags) -> &'static [RU] { if flags.is_64bit() { - &[RU::rbx, RU::r12, RU::r13, RU::r14, RU::r15] + if flags.call_conv() == CallConv::WindowsFastcall { + // "registers RBX, RBP, RDI, RSI, RSP, R12, R13, R14, R15 are considered nonvolatile + // and must be saved and restored by a function that uses them." + // as per https://msdn.microsoft.com/en-us/library/6t169e9c.aspx + // RSP & RSB are not listed below, since they are restored automatically during + // a function call. If that wasn't the case, function calls (RET) would not work. + &[ + RU::rbx, + RU::rdi, + RU::rsi, + RU::r12, + RU::r13, + RU::r14, + RU::r15, + ] + } else { + &[RU::rbx, RU::r12, RU::r13, RU::r14, RU::r15] + } } else { &[RU::rbx, RU::rsi, RU::rdi] } @@ -215,7 +256,7 @@ pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::Ct CallConv::Fast | CallConv::Cold | CallConv::SystemV => { system_v_prologue_epilogue(func, isa) } - CallConv::Fastcall => unimplemented!("Windows calling conventions"), + CallConv::WindowsFastcall => fastcall_prologue_epilogue(func, isa), CallConv::Baldrdash => baldrdash_prologue_epilogue(func, isa), CallConv::Probestack => unimplemented!("probestack calling convention"), } @@ -240,6 +281,83 @@ pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> Ok(()) } +/// Implementation of the fastcall-based Win64 calling convention described at [1] +/// [1] https://msdn.microsoft.com/en-us/library/ms235286.aspx +pub fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { + if !isa.flags().is_64bit() { + panic!("TODO: windows-fastcall: x86-32 not implemented yet"); + } + + // [1] "The primary exceptions are the stack pointer and malloc or alloca memory, + // which are aligned to 16 bytes in order to aid performance" + let stack_align = 16; + + let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; + let reg_type = if isa.flags().is_64bit() { + ir::types::I64 + } else { + ir::types::I32 + }; + + let csrs = callee_saved_gprs_used(isa.flags(), func); + + // [1] "Space is allocated on the call stack as a shadow store for callees to save" + // This shadow store contains the parameters which are passed through registers (ARG_GPRS) + // and is eventually used by the callee to save & restore the values of the arguments. + // + // [2] https://blogs.msdn.microsoft.com/oldnewthing/20110302-00/?p=11333 + // "Although the x64 calling convention reserves spill space for parameters, + // you don’t have to use them as such" + // + // The reserved stack area is composed of: + // return address + frame pointer + all callee-saved registers + shadow space + // + // Pushing the return address is an implicit function of the `call` + // instruction. Each of the others we will then push explicitly. Then we + // will adjust the stack pointer to make room for the rest of the required + // space for this frame. + const SHADOW_STORE_SIZE: i32 = 32; + let csr_stack_size = ((csrs.iter(GPR).len() + 2) * word_size) as i32; + + // TODO: eventually use the 32 bytes (shadow store) as spill slot. This currently doesn't work + // since cretonne does not support spill slots before incoming args + + func.create_stack_slot(ir::StackSlotData { + kind: ir::StackSlotKind::IncomingArg, + size: csr_stack_size as u32, + offset: Some(-(SHADOW_STORE_SIZE + csr_stack_size)), + }); + + let total_stack_size = layout_stack(&mut func.stack_slots, stack_align)? as i32; + let local_stack_size = i64::from(total_stack_size - csr_stack_size); + + // Add CSRs to function signature + let fp_arg = ir::AbiParam::special_reg( + reg_type, + ir::ArgumentPurpose::FramePointer, + RU::rbp as RegUnit, + ); + func.signature.params.push(fp_arg); + func.signature.returns.push(fp_arg); + + for csr in csrs.iter(GPR) { + let csr_arg = ir::AbiParam::special_reg(reg_type, ir::ArgumentPurpose::CalleeSaved, csr); + func.signature.params.push(csr_arg); + func.signature.returns.push(csr_arg); + } + + // Set up the cursor and insert the prologue + let entry_ebb = func.layout.entry_block().expect("missing entry block"); + let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_ebb); + insert_common_prologue(&mut pos, local_stack_size, reg_type, &csrs, isa); + + // Reset the cursor and insert the epilogue + let mut pos = pos.at_position(CursorPosition::Nowhere); + insert_common_epilogues(&mut pos, local_stack_size, reg_type, &csrs); + + Ok(()) +} + /// Insert a System V-compatible prologue and epilogue. pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { // The original 32-bit x86 ELF ABI had a 4-byte aligned stack pointer, but @@ -261,7 +379,7 @@ pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r // instruction. Each of the others we will then push explicitly. Then we // will adjust the stack pointer to make room for the rest of the required // space for this frame. - let csr_stack_size = ((csrs.iter(GPR).len() + 2) * word_size as usize) as i32; + let csr_stack_size = ((csrs.iter(GPR).len() + 2) * word_size) as i32; func.create_stack_slot(ir::StackSlotData { kind: ir::StackSlotKind::IncomingArg, size: csr_stack_size as u32, @@ -289,17 +407,18 @@ pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r // Set up the cursor and insert the prologue let entry_ebb = func.layout.entry_block().expect("missing entry block"); let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_ebb); - insert_system_v_prologue(&mut pos, local_stack_size, reg_type, &csrs, isa); + insert_common_prologue(&mut pos, local_stack_size, reg_type, &csrs, isa); // Reset the cursor and insert the epilogue let mut pos = pos.at_position(CursorPosition::Nowhere); - insert_system_v_epilogues(&mut pos, local_stack_size, reg_type, &csrs); + insert_common_epilogues(&mut pos, local_stack_size, reg_type, &csrs); Ok(()) } /// Insert the prologue for a given function. -fn insert_system_v_prologue( +/// This is used by common calling conventions such as System V. +fn insert_common_prologue( pos: &mut EncCursor, stack_size: i64, reg_type: ir::types::Type, @@ -374,7 +493,7 @@ fn insert_system_v_prologue( } /// Find all `return` instructions and insert epilogues before them. -fn insert_system_v_epilogues( +fn insert_common_epilogues( pos: &mut EncCursor, stack_size: i64, reg_type: ir::types::Type, @@ -384,14 +503,15 @@ fn insert_system_v_epilogues( pos.goto_last_inst(ebb); if let Some(inst) = pos.current_inst() { if pos.func.dfg[inst].opcode().is_return() { - insert_system_v_epilogue(inst, stack_size, pos, reg_type, csrs); + insert_common_epilogue(inst, stack_size, pos, reg_type, csrs); } } } } /// Insert an epilogue given a specific `return` instruction. -fn insert_system_v_epilogue( +/// This is used by common calling conventions such as System V. +fn insert_common_epilogue( inst: ir::Inst, stack_size: i64, pos: &mut EncCursor, diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index f5050bea76..734c33d1f6 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -37,7 +37,7 @@ pub fn builders() -> Result<(settings::Builder, isa::Builder), &'static str> { if cfg!(any(unix, target_os = "nebulet")) { flag_builder.set("call_conv", "system_v").unwrap(); } else if cfg!(windows) { - flag_builder.set("call_conv", "fastcall").unwrap(); + flag_builder.set("call_conv", "windows_fastcall").unwrap(); } else { return Err("unrecognized environment"); } diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index 6c87388761..d672c1dc63 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -16,6 +16,9 @@ region = "0.2.0" libc = { version = "0.2.40", default-features = false } errno = "0.2.3" +[target.'cfg(target_os = "windows")'.dependencies] +winapi = { version = "0.3", features = ["winbase", "memoryapi"] } + [features] default = ["std"] std = ["libc/use_std", "cretonne-codegen/std", "cretonne-module/std", "cretonne-native/std"] diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs index c7785e04a7..1fcfd8f93e 100644 --- a/lib/simplejit/src/backend.rs +++ b/lib/simplejit/src/backend.rs @@ -9,6 +9,8 @@ use cretonne_native; use std::ffi::CString; use std::ptr; use libc; +#[cfg(windows)] +use winapi; use memory::Memory; /// A builder for `SimpleJITBackend`. @@ -344,6 +346,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { fn finish(self) -> () {} } +#[cfg(not(windows))] fn lookup_with_dlsym(name: &str) -> *const u8 { let c_str = CString::new(name).unwrap(); let c_str_ptr = c_str.as_ptr(); @@ -354,6 +357,38 @@ fn lookup_with_dlsym(name: &str) -> *const u8 { sym as *const u8 } +#[cfg(windows)] +fn lookup_with_dlsym(name: &str) -> *const u8 { + const MSVCRT_DLL: &[u8] = b"msvcrt.dll\0"; + + let c_str = CString::new(name).unwrap(); + let c_str_ptr = c_str.as_ptr(); + + unsafe { + let handles = [ + // try to find the searched symbol in the currently running executable + ptr::null_mut(), + // try to find the searched symbol in local c runtime + winapi::um::libloaderapi::GetModuleHandleA(MSVCRT_DLL.as_ptr() as *const i8), + ]; + + for handle in &handles { + let addr = winapi::um::libloaderapi::GetProcAddress(*handle, c_str_ptr); + if addr.is_null() { + continue; + } + return addr as *const u8; + } + + let msg = if handles[1].is_null() { + "(msvcrt not loaded)" + } else { + "" + }; + panic!("cannot resolve address of symbol {} {}", name, msg); + } +} + struct SimpleJITRelocSink { pub relocs: Vec, } diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs index 68a2ca00ea..92c088cd71 100644 --- a/lib/simplejit/src/lib.rs +++ b/lib/simplejit/src/lib.rs @@ -23,6 +23,9 @@ extern crate errno; extern crate region; extern crate libc; +#[cfg(target_os = "windows")] +extern crate winapi; + mod backend; mod memory; diff --git a/lib/simplejit/src/memory.rs b/lib/simplejit/src/memory.rs index 2158352cc8..3ec2995372 100644 --- a/lib/simplejit/src/memory.rs +++ b/lib/simplejit/src/memory.rs @@ -26,6 +26,7 @@ impl PtrLen { /// Create a new `PtrLen` pointing to at least `size` bytes of memory, /// suitably sized and aligned for memory protection. + #[cfg(not(target_os = "windows"))] fn with_size(size: usize) -> Result { let page_size = region::page::size(); let alloc_size = round_up_to_page_size(size, page_size); @@ -42,6 +43,32 @@ impl PtrLen { } } } + + #[cfg(target_os = "windows")] + fn with_size(size: usize) -> Result { + use winapi::um::memoryapi::VirtualAlloc; + use winapi::um::winnt::{MEM_COMMIT, MEM_RESERVE, PAGE_READWRITE}; + + let page_size = region::page::size(); + + // VirtualAlloc always rounds up to the next multiple of the page size + let ptr = unsafe { + VirtualAlloc( + ptr::null_mut(), + size, + MEM_COMMIT | MEM_RESERVE, + PAGE_READWRITE, + ) + }; + if !ptr.is_null() { + Ok(Self { + ptr: ptr as *mut u8, + len: round_up_to_page_size(size, page_size), + }) + } else { + Err(errno::errno().to_string()) + } + } } /// JIT memory manager. This manages pages of suitably aligned and From f636d795c5367ee7444f34f2b640b2fafa60e10d Mon Sep 17 00:00:00 2001 From: Tyler McMullen Date: Wed, 9 May 2018 12:07:00 -0700 Subject: [PATCH 1774/3084] load_complex and store_complex instructions (#309) * Start adding the load_complex and store_complex instructions. N.b.: The text format is not correct yet. Requires changes to the lexer and parser. I'm not sure why I needed to change the RuntimeError to Exception yet. Will fix. * Get first few encodings of load_complex working. Still needs var args type checking. * Clean up ModRM helper functions in binemit. * Implement 32-bit displace for load_complex * Use encoding helpers instead of doing them all by hand * Initial implementation of store_complex * Parse value list for load/store_complex with + as delimiter. Looks nice. * Add sign/zero-extension and size variants for load_complex. * Add size variants of store_complex. * Add asm helper lines to load/store complex bin tests. * Example of length-checking the instruction ValueList for an encoding. Extremely questionable implementation. * Fix Python linting issues * First draft of postopt pass to fold adds and loads into load_complex. Just simple loads for now. * Optimization pass now works with all types of loads. * Add store+add -> store_complex to postopt pass * Put complex address optimization behind ISA flag. * Add load/store complex for f32 and f64 * Fixes changes to lexer that broke NaN parsing. Abstracts away the repeated checks for whether or not the characters following a + or - are going to be parsed as a number or not. * Fix formatting issues * Fix register restrictions for complex addresses. * Encoding tests for x86-32. * Add documentation for newly added instructions, recipes, and cdsl changes. * Fix python formatting again * Apply value-list length predicates to all LoadComplex and StoreComplex instructions. * Add predicate types to new encoding helpers for mypy. * Import FieldPredicate to satisfy mypy. * Add and fix some "asm" strings in the encoding tests. * Line-up 'bin' comments in x86/binary64 test * Test parsing of offset-less store_complex instruction. * 'sNaN' not 'sNan' * Bounds check the lookup for polymorphic typevar operand. * Fix encodings for istore16_complex. --- cranelift/docs/langref.rst | 5 + .../filetests/isa/x86/binary32-float.cton | 26 ++ cranelift/filetests/isa/x86/binary32.cton | 31 ++ .../filetests/isa/x86/binary64-float.cton | 54 ++++ cranelift/filetests/isa/x86/binary64.cton | 74 +++++ cranelift/filetests/parser/tiny.cton | 8 + .../filetests/postopt/complex_memory_ops.cton | 95 ++++++ lib/codegen/meta/base/formats.py | 2 + lib/codegen/meta/base/instructions.py | 92 ++++++ lib/codegen/meta/base/predicates.py | 11 +- lib/codegen/meta/cdsl/formats.py | 38 ++- lib/codegen/meta/cdsl/instructions.py | 5 +- lib/codegen/meta/gen_binemit.py | 2 +- lib/codegen/meta/gen_encoding.py | 2 +- lib/codegen/meta/isa/x86/encodings.py | 83 +++++- lib/codegen/meta/isa/x86/recipes.py | 272 ++++++++++++++++++ lib/codegen/src/isa/mod.rs | 5 + lib/codegen/src/isa/x86/binemit.rs | 36 ++- lib/codegen/src/isa/x86/mod.rs | 4 + lib/codegen/src/postopt.rs | 160 ++++++++++- lib/codegen/src/predicates.rs | 5 + lib/codegen/src/verifier/mod.rs | 8 +- lib/codegen/src/write.rs | 47 +++ lib/reader/src/lexer.rs | 33 ++- lib/reader/src/parser.rs | 50 +++- 25 files changed, 1127 insertions(+), 21 deletions(-) create mode 100644 cranelift/filetests/postopt/complex_memory_ops.cton diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 177d133ace..2758a1c7b8 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -476,6 +476,11 @@ these instructions is undefined. If it is addressable but not There are also more restricted operations for accessing specific types of memory objects. +Additionally, instructions are provided for handling multi-register addressing. + +.. autoinst:: load_complex +.. autoinst:: store_complex + Memory operation flags ---------------------- diff --git a/cranelift/filetests/isa/x86/binary32-float.cton b/cranelift/filetests/isa/x86/binary32-float.cton index 702972a241..7b1a07853a 100644 --- a/cranelift/filetests/isa/x86/binary32-float.cton +++ b/cranelift/filetests/isa/x86/binary32-float.cton @@ -227,6 +227,32 @@ ebb0: ; asm: ucomiss %xmm5, %xmm5 [-,%rflags] v312 = ffcmp v10, v10 ; bin: 0f 2e ed + ; Load/Store Complex + + [-,%rax] v350 = iconst.i32 1 + [-,%rbx] v351 = iconst.i32 2 + + ; asm: movss (%rax,%rbx,1),%xmm5 + [-,%xmm5] v352 = load_complex.f32 v350+v351 ; bin: heap_oob f3 0f 10 2c 18 + ; asm: movss 0x32(%rax,%rbx,1),%xmm5 + [-,%xmm5] v353 = load_complex.f32 v350+v351+50 ; bin: heap_oob f3 0f 10 6c 18 32 + ; asm: movss -0x32(%rax,%rbx,1),%xmm5 + [-,%xmm5] v354 = load_complex.f32 v350+v351-50 ; bin: heap_oob f3 0f 10 6c 18 ce + ; asm: movss 0x2710(%rax,%rbx,1),%xmm5 + [-,%xmm5] v355 = load_complex.f32 v350+v351+10000 ; bin: heap_oob f3 0f 10 ac 18 00002710 + ; asm: movss -0x2710(%rax,%rbx,1),%xmm5 + [-,%xmm5] v356 = load_complex.f32 v350+v351-10000 ; bin: heap_oob f3 0f 10 ac 18 ffffd8f0 + ; asm: movss %xmm5,(%rax,%rbx,1) + [-] store_complex.f32 v100, v350+v351 ; bin: heap_oob f3 0f 11 2c 18 + ; asm: movss %xmm5,0x32(%rax,%rbx,1) + [-] store_complex.f32 v100, v350+v351+50 ; bin: heap_oob f3 0f 11 6c 18 32 + ; asm: movss %xmm2,-0x32(%rax,%rbx,1) + [-] store_complex.f32 v101, v350+v351-50 ; bin: heap_oob f3 0f 11 54 18 ce + ; asm: movss %xmm5,0x2710(%rax,%rbx,1) + [-] store_complex.f32 v100, v350+v351+10000 ; bin: heap_oob f3 0f 11 ac 18 00002710 + ; asm: movss %xmm2,-0x2710(%rax,%rbx,1) + [-] store_complex.f32 v101, v350+v351-10000 ; bin: heap_oob f3 0f 11 94 18 ffffd8f0 + return } diff --git a/cranelift/filetests/isa/x86/binary32.cton b/cranelift/filetests/isa/x86/binary32.cton index feeabaa637..5ce1b228c9 100644 --- a/cranelift/filetests/isa/x86/binary32.cton +++ b/cranelift/filetests/isa/x86/binary32.cton @@ -432,6 +432,37 @@ ebb0: ; asm: shrl $8, %esi [-,%rsi] v515 = ushr_imm v2, 8 ; bin: c1 ee 08 + ; Load Complex + [-,%rax] v521 = iconst.i32 1 + [-,%rbx] v522 = iconst.i32 1 + ; asm: movl (%eax,%ebx,1), %ecx + [-,%rcx] v526 = load_complex.i32 v521+v522 ; bin: heap_oob 8b 0c 18 + ; asm: movl 1(%eax,%ebx,1), %ecx + [-,%rcx] v528 = load_complex.i32 v521+v522+1 ; bin: heap_oob 8b 4c 18 01 + ; asm: mov 0x100000(%eax,%ebx,1),%ecx + [-,%rcx] v530 = load_complex.i32 v521+v522+0x1000 ; bin: heap_oob 8b 8c 18 00001000 + ; asm: movzbl (%eax,%ebx,1),%ecx + [-,%rcx] v532 = uload8_complex.i32 v521+v522 ; bin: heap_oob 0f b6 0c 18 + ; asm: movsbl (%eax,%ebx,1),%ecx + [-,%rcx] v534 = sload8_complex.i32 v521+v522 ; bin: heap_oob 0f be 0c 18 + ; asm: movzwl (%eax,%ebx,1),%ecx + [-,%rcx] v536 = uload16_complex.i32 v521+v522 ; bin: heap_oob 0f b7 0c 18 + ; asm: movswl (%eax,%ebx,1),%ecx + [-,%rcx] v538 = sload16_complex.i32 v521+v522 ; bin: heap_oob 0f bf 0c 18 + + ; Store Complex + [-,%rcx] v601 = iconst.i32 1 + ; asm: mov %ecx,(%eax,%ebx,1) + store_complex v601, v521+v522 ; bin: heap_oob 89 0c 18 + ; asm: mov %ecx,0x1(%eax,%ebx,1) + store_complex v601, v521+v522+1 ; bin: heap_oob 89 4c 18 01 + ; asm: mov %ecx,0x100000(%eax,%ebx,1) + store_complex v601, v521+v522+0x1000 ; bin: heap_oob 89 8c 18 00001000 + ; asm: mov %cx,(%eax,%ebx,1) + istore16_complex v601, v521+v522 ; bin: heap_oob 66 89 0c 18 + ; asm: mov %cl,(%eax,%ebx,1) + istore8_complex v601, v521+v522 ; bin: heap_oob 88 0c 18 + ; asm: testl %ecx, %ecx ; asm: je ebb1 brz v1, ebb1 ; bin: 85 c9 74 0e diff --git a/cranelift/filetests/isa/x86/binary64-float.cton b/cranelift/filetests/isa/x86/binary64-float.cton index fc38271485..2f17c75ea6 100644 --- a/cranelift/filetests/isa/x86/binary64-float.cton +++ b/cranelift/filetests/isa/x86/binary64-float.cton @@ -241,6 +241,34 @@ ebb0: ; asm: ucomiss %xmm5, %xmm5 [-,%rflags] v312 = ffcmp v10, v10 ; bin: 0f 2e ed + + ; Load/Store Complex + + [-,%rax] v350 = iconst.i64 1 + [-,%rbx] v351 = iconst.i64 2 + + ; asm: movss (%rax,%rbx,1),%xmm5 + [-,%xmm5] v352 = load_complex.f32 v350+v351 ; bin: heap_oob f3 0f 10 2c 18 + ; asm: movss 0x32(%rax,%rbx,1),%xmm5 + [-,%xmm5] v353 = load_complex.f32 v350+v351+50 ; bin: heap_oob f3 0f 10 6c 18 32 + ; asm: movss -0x32(%rax,%rbx,1),%xmm10 + [-,%xmm10] v354 = load_complex.f32 v350+v351-50 ; bin: heap_oob f3 44 0f 10 54 18 ce + ; asm: 0x2710(%rax,%rbx,1),%xmm5 + [-,%xmm5] v355 = load_complex.f32 v350+v351+10000 ; bin: heap_oob f3 0f 10 ac 18 00002710 + ; asm: -0x2710(%rax,%rbx,1),%xmm10 + [-,%xmm10] v356 = load_complex.f32 v350+v351-10000 ; bin: heap_oob f3 44 0f 10 94 18 ffffd8f0 + + ; asm: movsd %xmm5, (%rax,%rbx,1) + [-] store_complex.f32 v100, v350+v351 ; bin: heap_oob f3 0f 11 2c 18 + ; asm: movsd %xmm5, 50(%rax,%rbx,1) + [-] store_complex.f32 v100, v350+v351+50 ; bin: heap_oob f3 0f 11 6c 18 32 + ; asm: movsd %xmm10, -50(%rax,%rbx,1) + [-] store_complex.f32 v101, v350+v351-50 ; bin: heap_oob f3 44 0f 11 54 18 ce + ; asm: movsd %xmm5, 10000(%rax,%rbx,1) + [-] store_complex.f32 v100, v350+v351+10000 ; bin: heap_oob f3 0f 11 ac 18 00002710 + ; asm: movsd %xmm10, -10000(%rax,%rbx,1) + [-] store_complex.f32 v101, v350+v351-10000 ; bin: heap_oob f3 44 0f 11 94 18 ffffd8f0 + return } @@ -476,6 +504,32 @@ ebb0: ; asm: ucomisd %xmm5, %xmm5 [-,%rflags] v312 = ffcmp v10, v10 ; bin: 66 0f 2e ed + ; Load/Store Complex + + [-,%rax] v350 = iconst.i64 1 + [-,%rbx] v351 = iconst.i64 2 + ; asm: movsd (%rax,%rbx,1),%xmm5 + [-,%xmm5] v352 = load_complex.f64 v350+v351 ; bin: heap_oob f2 0f 10 2c 18 + ; asm: movsd 0x32(%rax,%rbx,1),%xmm5 + [-,%xmm5] v353 = load_complex.f64 v350+v351+50 ; bin: heap_oob f2 0f 10 6c 18 32 + ; asm: movsd -0x32(%rax,%rbx,1),%xmm10 + [-,%xmm10] v354 = load_complex.f64 v350+v351-50 ; bin: heap_oob f2 44 0f 10 54 18 ce + ; asm: movsd 0x2710(%rax,%rbx,1),%xmm5 + [-,%xmm5] v355 = load_complex.f64 v350+v351+10000 ; bin: heap_oob f2 0f 10 ac 18 00002710 + ; asm: movsd -0x2710(%rax,%rbx,1),%xmm10 + [-,%xmm10] v356 = load_complex.f64 v350+v351-10000 ; bin: heap_oob f2 44 0f 10 94 18 ffffd8f0 + + ; asm: movsd %xmm5, (%rax,%rbx,1) + [-] store_complex.f64 v100, v350+v351 ; bin: heap_oob f2 0f 11 2c 18 + ; asm: movsd %xmm5, 50(%rax,%rbx,1) + [-] store_complex.f64 v100, v350+v351+50 ; bin: heap_oob f2 0f 11 6c 18 32 + ; asm: movsd %xmm10, -50(%rax,%rbx,1) + [-] store_complex.f64 v101, v350+v351-50 ; bin: heap_oob f2 44 0f 11 54 18 ce + ; asm: movsd %xmm5, 10000(%rax,%rbx,1) + [-] store_complex.f64 v100, v350+v351+10000 ; bin: heap_oob f2 0f 11 ac 18 00002710 + ; asm: movsd %xmm10, -10000(%rax,%rbx,1) + [-] store_complex.f64 v101, v350+v351-10000 ; bin: heap_oob f2 44 0f 11 94 18 ffffd8f0 + return } diff --git a/cranelift/filetests/isa/x86/binary64.cton b/cranelift/filetests/isa/x86/binary64.cton index af625fa1bb..2432736e64 100644 --- a/cranelift/filetests/isa/x86/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.cton @@ -594,6 +594,80 @@ ebb0: [-,%r8] v520 = ushr_imm v4, 63 ; bin: 49 c1 e8 3f + ; Load Complex + [-,%rax] v521 = iconst.i64 1 + [-,%rbx] v522 = iconst.i64 1 + [-,%rdi] v523 = iconst.i32 1 + [-,%rsi] v524 = iconst.i32 1 + ; asm: movq (%rax,%rbx,1), %rcx + [-,%rcx] v525 = load_complex.i64 v521+v522 ; bin: heap_oob 48 8b 0c 18 + ; asm: movl (%rax,%rbx,1), %ecx + [-,%rcx] v526 = load_complex.i32 v521+v522 ; bin: heap_oob 8b 0c 18 + ; asm: movq 1(%rax,%rbx,1), %rcx + [-,%rcx] v527 = load_complex.i64 v521+v522+1 ; bin: heap_oob 48 8b 4c 18 01 + ; asm: movl 1(%rax,%rbx,1), %ecx + [-,%rcx] v528 = load_complex.i32 v521+v522+1 ; bin: heap_oob 8b 4c 18 01 + ; asm: mov 0x100000(%rax,%rbx,1),%rcx + [-,%rcx] v529 = load_complex.i64 v521+v522+0x1000 ; bin: heap_oob 48 8b 8c 18 00001000 + ; asm: mov 0x100000(%rax,%rbx,1),%ecx + [-,%rcx] v530 = load_complex.i32 v521+v522+0x1000 ; bin: heap_oob 8b 8c 18 00001000 + ; asm: movzbq (%rax,%rbx,1),%rcx + [-,%rcx] v531 = uload8_complex.i64 v521+v522 ; bin: heap_oob 48 0f b6 0c 18 + ; asm: movzbl (%rax,%rbx,1),%ecx + [-,%rcx] v532 = uload8_complex.i32 v521+v522 ; bin: heap_oob 0f b6 0c 18 + ; asm: movsbq (%rax,%rbx,1),%rcx + [-,%rcx] v533 = sload8_complex.i64 v521+v522 ; bin: heap_oob 48 0f be 0c 18 + ; asm: movsbl (%rax,%rbx,1),%ecx + [-,%rcx] v534 = sload8_complex.i32 v521+v522 ; bin: heap_oob 0f be 0c 18 + ; asm: movzwq (%rax,%rbx,1),%rcx + [-,%rcx] v535 = uload16_complex.i64 v521+v522 ; bin: heap_oob 48 0f b7 0c 18 + ; asm: movzwl (%rax,%rbx,1),%ecx + [-,%rcx] v536 = uload16_complex.i32 v521+v522 ; bin: heap_oob 0f b7 0c 18 + ; asm: movswq (%rax,%rbx,1),%rcx + [-,%rcx] v537 = sload16_complex.i64 v521+v522 ; bin: heap_oob 48 0f bf 0c 18 + ; asm: movswl (%rax,%rbx,1),%ecx + [-,%rcx] v538 = sload16_complex.i32 v521+v522 ; bin: heap_oob 0f bf 0c 18 + ; asm: mov (%rax,%rbx,1),%ecx + [-,%rcx] v539 = uload32_complex v521+v522 ; bin: heap_oob 8b 0c 18 + ; asm: movslq (%rax,%rbx,1),%rcx + [-,%rcx] v540 = sload32_complex v521+v522 ; bin: heap_oob 48 63 0c 18 + [-,%r13] v550 = iconst.i64 1 + [-,%r14] v551 = iconst.i64 1 + ; asm: mov 0x0(%r13,%r14,1),%r12d + [-,%r12] v552 = load_complex.i32 v550+v551 ; bin: heap_oob 47 8b 64 35 00 + + ; Store Complex + [-,%rcx] v600 = iconst.i64 1 + [-,%rcx] v601 = iconst.i32 1 + [-,%r10] v602 = iconst.i64 1 + [-,%r11] v603 = iconst.i32 1 + ; asm: mov %rcx,(%rax,%rbx,1) + store_complex v600, v521+v522 ; bin: heap_oob 48 89 0c 18 + ; asm: mov %rcx,0x1(%rax,%rbx,1) + store_complex v600, v521+v522+1 ; bin: heap_oob 48 89 4c 18 01 + ; asm: mov %rcx,0x100000(%rax,%rbx,1) + store_complex v600, v521+v522+0x1000 ; bin: heap_oob 48 89 8c 18 00001000 + ; asm: mov %ecx,(%rax,%rbx,1) + store_complex v601, v521+v522 ; bin: heap_oob 89 0c 18 + ; asm: mov %ecx,0x1(%rax,%rbx,1) + store_complex v601, v521+v522+1 ; bin: heap_oob 89 4c 18 01 + ; asm: mov %ecx,0x100000(%rax,%rbx,1) + store_complex v601, v521+v522+0x1000 ; bin: heap_oob 89 8c 18 00001000 + ; asm: mov %ecx,(%rax,%rbx,1) + istore32_complex v600, v521+v522 ; bin: heap_oob 89 0c 18 + ; asm: mov %cx,(%rax,%rbx,1) + istore16_complex v600, v521+v522 ; bin: heap_oob 66 89 0c 18 + ; asm: mov %cx,(%rax,%rbx,1) + istore16_complex v601, v521+v522 ; bin: heap_oob 66 89 0c 18 + ; asm: mov %r10w,(%rax,%rbx,1) + istore16_complex v602, v521+v522 ; bin: heap_oob 66 44 89 14 18 + ; asm: mov %r11w,(%rax,%rbx,1) + istore16_complex v603, v521+v522 ; bin: heap_oob 66 44 89 1c 18 + ; asm: mov %cl,(%rax,%rbx,1) + istore8_complex v600, v521+v522 ; bin: heap_oob 88 0c 18 + ; asm: mov %cl,(%rax,%rbx,1) + istore8_complex v601, v521+v522 ; bin: heap_oob 88 0c 18 + ; asm: testq %rcx, %rcx ; asm: je ebb1 brz v1, ebb1 ; bin: 48 85 c9 74 1b diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.cton index 2a342a666b..49628a27d6 100644 --- a/cranelift/filetests/parser/tiny.cton +++ b/cranelift/filetests/parser/tiny.cton @@ -158,9 +158,13 @@ ebb0(v1: i32): v6 = load.i64 aligned notrap v1 v7 = load.i64 v1-12 v8 = load.i64 notrap v1+0x1_0000 + v9 = load_complex.i64 v1+v2 + v10 = load_complex.i64 v1+v2+0x1 store v2, v1 store aligned v3, v1+12 store notrap aligned v3, v1-12 + store_complex v3, v1+v2 + store_complex v3, v1+v2+0x1 } ; sameln: function %memory(i32) fast { ; nextln: ebb0(v1: i32): @@ -171,9 +175,13 @@ ebb0(v1: i32): ; nextln: v6 = load.i64 notrap aligned v1 ; nextln: v7 = load.i64 v1-12 ; nextln: v8 = load.i64 notrap v1+0x0001_0000 +; nextln: v9 = load_complex.i64 v1+v2 +; nextln: v10 = load_complex.i64 v1+v2+1 ; nextln: store v2, v1 ; nextln: store aligned v3, v1+12 ; nextln: store notrap aligned v3, v1-12 +; nextln: store_complex v3, v1+v2 +; nextln: store_complex v3, v1+v2+1 ; Register diversions. ; This test file has no ISA, so we can unly use register unit numbers. diff --git a/cranelift/filetests/postopt/complex_memory_ops.cton b/cranelift/filetests/postopt/complex_memory_ops.cton new file mode 100644 index 0000000000..8f12e61577 --- /dev/null +++ b/cranelift/filetests/postopt/complex_memory_ops.cton @@ -0,0 +1,95 @@ +test postopt +set is_64bit +isa x86 + +function %dual_loads(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): +[RexOp1rr#8001] v3 = iadd v0, v1 + v4 = load.i64 v3 + v5 = uload8.i64 v3 + v6 = sload8.i64 v3 + v7 = uload16.i64 v3 + v8 = sload16.i64 v3 + v9 = uload32.i64 v3 + v10 = sload32.i64 v3 +[Op1ret#c3] return v10 +} + +; sameln: function %dual_loads +; nextln: ebb0(v0: i64, v1: i64): +; nextln: v3 = iadd v0, v1 +; nextln: v4 = load_complex.i64 v0+v1 +; nextln: v5 = uload8_complex.i64 v0+v1 +; nextln: v6 = sload8_complex.i64 v0+v1 +; nextln: v7 = uload16_complex.i64 v0+v1 +; nextln: v8 = sload16_complex.i64 v0+v1 +; nextln: v9 = uload32_complex v0+v1 +; nextln: v10 = sload32_complex v0+v1 +; nextln: return v10 +; nextln: } + +function %dual_loads2(i64, i64) -> i64 { +ebb0(v0: i64, v1: i64): +[RexOp1rr#8001] v3 = iadd v0, v1 + v4 = load.i64 v3+1 + v5 = uload8.i64 v3+1 + v6 = sload8.i64 v3+1 + v7 = uload16.i64 v3+1 + v8 = sload16.i64 v3+1 + v9 = uload32.i64 v3+1 + v10 = sload32.i64 v3+1 +[Op1ret#c3] return v10 +} + +; sameln: function %dual_loads2 +; nextln: ebb0(v0: i64, v1: i64): +; nextln: v3 = iadd v0, v1 +; nextln: v4 = load_complex.i64 v0+v1+1 +; nextln: v5 = uload8_complex.i64 v0+v1+1 +; nextln: v6 = sload8_complex.i64 v0+v1+1 +; nextln: v7 = uload16_complex.i64 v0+v1+1 +; nextln: v8 = sload16_complex.i64 v0+v1+1 +; nextln: v9 = uload32_complex v0+v1+1 +; nextln: v10 = sload32_complex v0+v1+1 +; nextln: return v10 +; nextln: } + +function %dual_stores(i64, i64, i64) { +ebb0(v0: i64, v1: i64, v2: i64): +[RexOp1rr#8001] v3 = iadd v0, v1 +[RexOp1st#8089] store.i64 v2, v3 +[RexOp1st#88] istore8.i64 v2, v3 +[RexMp1st#189] istore16.i64 v2, v3 +[RexOp1st#89] istore32.i64 v2, v3 +[Op1ret#c3] return +} + +; sameln: function %dual_stores +; nextln: ebb0(v0: i64, v1: i64, v2: i64): +; nextln: v3 = iadd v0, v1 +; nextln: store_complex v2, v0+v1 +; nextln: istore8_complex v2, v0+v1 +; nextln: istore16_complex v2, v0+v1 +; nextln: istore32_complex v2, v0+v1 +; nextln: return +; nextln: } + +function %dual_stores2(i64, i64, i64) { +ebb0(v0: i64, v1: i64, v2: i64): +[RexOp1rr#8001] v3 = iadd v0, v1 +[RexOp1stDisp8#8089] store.i64 v2, v3+1 +[RexOp1stDisp8#88] istore8.i64 v2, v3+1 +[RexMp1stDisp8#189] istore16.i64 v2, v3+1 +[RexOp1stDisp8#89] istore32.i64 v2, v3+1 +[Op1ret#c3] return +} + +; sameln: function %dual_stores2 +; nextln: ebb0(v0: i64, v1: i64, v2: i64): +; nextln: v3 = iadd v0, v1 +; nextln: store_complex v2, v0+v1+1 +; nextln: istore8_complex v2, v0+v1+1 +; nextln: istore16_complex v2, v0+v1+1 +; nextln: istore32_complex v2, v0+v1+1 +; nextln: return +; nextln: } diff --git a/lib/codegen/meta/base/formats.py b/lib/codegen/meta/base/formats.py index 89ef30881f..817ac0a87d 100644 --- a/lib/codegen/meta/base/formats.py +++ b/lib/codegen/meta/base/formats.py @@ -57,7 +57,9 @@ CallIndirect = InstructionFormat(sig_ref, VALUE, VARIABLE_ARGS) FuncAddr = InstructionFormat(func_ref) Load = InstructionFormat(memflags, VALUE, offset32) +LoadComplex = InstructionFormat(memflags, VARIABLE_ARGS, offset32) Store = InstructionFormat(memflags, VALUE, VALUE, offset32) +StoreComplex = InstructionFormat(memflags, VALUE, VARIABLE_ARGS, offset32) StackLoad = InstructionFormat(stack_slot, offset32) StackStore = InstructionFormat(VALUE, stack_slot, offset32) diff --git a/lib/codegen/meta/base/instructions.py b/lib/codegen/meta/base/instructions.py index 8fee97ae42..3107c9ca41 100644 --- a/lib/codegen/meta/base/instructions.py +++ b/lib/codegen/meta/base/instructions.py @@ -246,6 +246,7 @@ x = Operand('x', Mem, doc='Value to be stored') a = Operand('a', Mem, doc='Value loaded') p = Operand('p', iAddr) Flags = Operand('Flags', memflags) +args = Operand('args', VARIABLE_ARGS, doc='Address arguments') load = Instruction( 'load', r""" @@ -256,6 +257,15 @@ load = Instruction( """, ins=(Flags, p, Offset), outs=a, can_load=True) +load_complex = Instruction( + 'load_complex', r""" + Load from memory at ``sum(args) + Offset``. + + This is a polymorphic instruction that can load any value type which + has a memory representation. + """, + ins=(Flags, args, Offset), outs=a, can_load=True) + store = Instruction( 'store', r""" Store ``x`` to memory at ``p + Offset``. @@ -265,6 +275,16 @@ store = Instruction( """, ins=(Flags, x, p, Offset), can_store=True) +store_complex = Instruction( + 'store_complex', r""" + Store ``x`` to memory at ``sum(args) + Offset``. + + This is a polymorphic instruction that can store any value type with a + memory representation. + """, + ins=(Flags, x, args, Offset), can_store=True) + + iExt8 = TypeVar( 'iExt8', 'An integer type with more than 8 bits', ints=(16, 64)) @@ -279,6 +299,14 @@ uload8 = Instruction( """, ins=(Flags, p, Offset), outs=a, can_load=True) +uload8_complex = Instruction( + 'uload8_complex', r""" + Load 8 bits from memory at ``sum(args) + Offset`` and zero-extend. + + This is equivalent to ``load.i8`` followed by ``uextend``. + """, + ins=(Flags, args, Offset), outs=a, can_load=True) + sload8 = Instruction( 'sload8', r""" Load 8 bits from memory at ``p + Offset`` and sign-extend. @@ -287,6 +315,14 @@ sload8 = Instruction( """, ins=(Flags, p, Offset), outs=a, can_load=True) +sload8_complex = Instruction( + 'sload8_complex', r""" + Load 8 bits from memory at ``sum(args) + Offset`` and sign-extend. + + This is equivalent to ``load.i8`` followed by ``uextend``. + """, + ins=(Flags, args, Offset), outs=a, can_load=True) + istore8 = Instruction( 'istore8', r""" Store the low 8 bits of ``x`` to memory at ``p + Offset``. @@ -295,6 +331,14 @@ istore8 = Instruction( """, ins=(Flags, x, p, Offset), can_store=True) +istore8_complex = Instruction( + 'istore8_complex', r""" + Store the low 8 bits of ``x`` to memory at ``sum(args) + Offset``. + + This is equivalent to ``ireduce.i8`` followed by ``store.i8``. + """, + ins=(Flags, x, args, Offset), can_store=True) + iExt16 = TypeVar( 'iExt16', 'An integer type with more than 16 bits', ints=(32, 64)) @@ -309,6 +353,14 @@ uload16 = Instruction( """, ins=(Flags, p, Offset), outs=a, can_load=True) +uload16_complex = Instruction( + 'uload16_complex', r""" + Load 16 bits from memory at ``sum(args) + Offset`` and zero-extend. + + This is equivalent to ``load.i16`` followed by ``uextend``. + """, + ins=(Flags, args, Offset), outs=a, can_load=True) + sload16 = Instruction( 'sload16', r""" Load 16 bits from memory at ``p + Offset`` and sign-extend. @@ -317,6 +369,14 @@ sload16 = Instruction( """, ins=(Flags, p, Offset), outs=a, can_load=True) +sload16_complex = Instruction( + 'sload16_complex', r""" + Load 16 bits from memory at ``sum(args) + Offset`` and sign-extend. + + This is equivalent to ``load.i16`` followed by ``uextend``. + """, + ins=(Flags, args, Offset), outs=a, can_load=True) + istore16 = Instruction( 'istore16', r""" Store the low 16 bits of ``x`` to memory at ``p + Offset``. @@ -325,6 +385,14 @@ istore16 = Instruction( """, ins=(Flags, x, p, Offset), can_store=True) +istore16_complex = Instruction( + 'istore16_complex', r""" + Store the low 16 bits of ``x`` to memory at ``sum(args) + Offset``. + + This is equivalent to ``ireduce.i16`` followed by ``store.i16``. + """, + ins=(Flags, x, args, Offset), can_store=True) + iExt32 = TypeVar( 'iExt32', 'An integer type with more than 32 bits', ints=(64, 64)) @@ -339,6 +407,14 @@ uload32 = Instruction( """, ins=(Flags, p, Offset), outs=a, can_load=True) +uload32_complex = Instruction( + 'uload32_complex', r""" + Load 32 bits from memory at ``sum(args) + Offset`` and zero-extend. + + This is equivalent to ``load.i32`` followed by ``uextend``. + """, + ins=(Flags, args, Offset), outs=a, can_load=True) + sload32 = Instruction( 'sload32', r""" Load 32 bits from memory at ``p + Offset`` and sign-extend. @@ -347,6 +423,14 @@ sload32 = Instruction( """, ins=(Flags, p, Offset), outs=a, can_load=True) +sload32_complex = Instruction( + 'sload32_complex', r""" + Load 32 bits from memory at ``sum(args) + Offset`` and sign-extend. + + This is equivalent to ``load.i32`` followed by ``uextend``. + """, + ins=(Flags, args, Offset), outs=a, can_load=True) + istore32 = Instruction( 'istore32', r""" Store the low 32 bits of ``x`` to memory at ``p + Offset``. @@ -355,6 +439,14 @@ istore32 = Instruction( """, ins=(Flags, x, p, Offset), can_store=True) +istore32_complex = Instruction( + 'istore32_complex', r""" + Store the low 32 bits of ``x`` to memory at ``sum(args) + Offset``. + + This is equivalent to ``ireduce.i32`` followed by ``store.i32``. + """, + ins=(Flags, x, args, Offset), can_store=True) + x = Operand('x', Mem, doc='Value to be stored') a = Operand('a', Mem, doc='Value loaded') Offset = Operand('Offset', offset32, 'In-bounds offset into stack slot') diff --git a/lib/codegen/meta/base/predicates.py b/lib/codegen/meta/base/predicates.py index 1a6b4c2c75..44f135e0d3 100644 --- a/lib/codegen/meta/base/predicates.py +++ b/lib/codegen/meta/base/predicates.py @@ -2,12 +2,12 @@ Cretonne predicates that consider `Function` fields. """ from cdsl.predicates import FieldPredicate -from .formats import UnaryGlobalVar +from .formats import UnaryGlobalVar, InstructionFormat try: from typing import TYPE_CHECKING if TYPE_CHECKING: - from cdsl.formats import FormatField # noqa + from cdsl.formats import InstructionFormat, FormatField # noqa except ImportError: pass @@ -33,3 +33,10 @@ class IsColocatedData(FieldPredicate): # type: () -> None super(IsColocatedData, self).__init__( UnaryGlobalVar.global_var, 'is_colocated_data', ('func',)) + + +class LengthEquals(FieldPredicate): + def __init__(self, iform, num): + # type: (InstructionFormat, int) -> None + super(LengthEquals, self).__init__( + iform.args(), 'has_length_of', (num, 'func')) diff --git a/lib/codegen/meta/cdsl/formats.py b/lib/codegen/meta/cdsl/formats.py index aba83ed7a2..c8dd58fc7f 100644 --- a/lib/codegen/meta/cdsl/formats.py +++ b/lib/codegen/meta/cdsl/formats.py @@ -103,6 +103,19 @@ class InstructionFormat(object): InstructionFormat._registry[sig] = self InstructionFormat.all_formats.append(self) + def args(self): + # type: () -> FormatField + """ + Provides a ValueListField, which is derived from FormatField, + corresponding to the full ValueList of the instruction format. This + is useful for creating predicates for instructions which use variadic + arguments. + """ + + if self.has_value_list: + return ValueListField(self) + return None + def _process_member_names(self, kinds): # type: (Sequence[Union[OperandKind, Tuple[str, OperandKind]]]) -> Iterable[FormatField] # noqa """ @@ -210,7 +223,7 @@ class FormatField(object): This corresponds to a single member of a variant of the `InstructionData` data type. - :param iformat: Parent `InstructionFormat`. + :param iform: Parent `InstructionFormat`. :param immnum: Immediate operand number in parent. :param kind: Immediate Operand kind. :param member: Member name in `InstructionData` variant. @@ -227,6 +240,29 @@ class FormatField(object): # type: () -> str return '{}.{}'.format(self.format.name, self.member) + def rust_destructuring_name(self): + # type: () -> str + return self.member + def rust_name(self): # type: () -> str return self.member + + +class ValueListField(FormatField): + """ + The full value list field of an instruction format. + + This corresponds to all Value-type members of a variant of the + `InstructionData` format, which contains a ValueList. + + :param iform: Parent `InstructionFormat`. + """ + def __init__(self, iform): + # type: (InstructionFormat) -> None + self.format = iform + self.member = "args" + + def rust_destructuring_name(self): + # type: () -> str + return 'ref {}'.format(self.member) diff --git a/lib/codegen/meta/cdsl/instructions.py b/lib/codegen/meta/cdsl/instructions.py index 3ef95c9cf9..f972f82bc7 100644 --- a/lib/codegen/meta/cdsl/instructions.py +++ b/lib/codegen/meta/cdsl/instructions.py @@ -201,9 +201,10 @@ class Instruction(object): # Prefer to use the typevar_operand to infer the controlling typevar. self.use_typevar_operand = False typevar_error = None - if self.format.typevar_operand is not None: + tv_op = self.format.typevar_operand + if tv_op is not None and tv_op < len(self.value_opnums): try: - opnum = self.value_opnums[self.format.typevar_operand] + opnum = self.value_opnums[tv_op] tv = self.ins[opnum].typevar if tv is tv.free_typevar() or tv.singleton_type() is not None: self.other_typevars = self._verify_ctrl_typevar(tv) diff --git a/lib/codegen/meta/gen_binemit.py b/lib/codegen/meta/gen_binemit.py index 5888d89e10..c813d12977 100644 --- a/lib/codegen/meta/gen_binemit.py +++ b/lib/codegen/meta/gen_binemit.py @@ -27,7 +27,7 @@ def gen_recipe(recipe, fmt): nvops = iform.num_value_operands want_args = any(isinstance(i, RegClass) or isinstance(i, Stack) for i in recipe.ins) - assert not want_args or nvops > 0 + assert not want_args or nvops > 0 or iform.has_value_list want_outs = any(isinstance(o, RegClass) or isinstance(o, Stack) for o in recipe.outs) diff --git a/lib/codegen/meta/gen_encoding.py b/lib/codegen/meta/gen_encoding.py index e3915100a2..a81b66dcca 100644 --- a/lib/codegen/meta/gen_encoding.py +++ b/lib/codegen/meta/gen_encoding.py @@ -103,7 +103,7 @@ def emit_instp(instp, fmt, has_func=False): fnames = set() # type: Set[str] for p in leafs: if isinstance(p, FieldPredicate): - fnames.add(p.field.rust_name()) + fnames.add(p.field.rust_destructuring_name()) else: assert isinstance(p, TypePredicate) has_type_check = True diff --git a/lib/codegen/meta/isa/x86/encodings.py b/lib/codegen/meta/isa/x86/encodings.py index 2284a0d70f..1a9260a28d 100644 --- a/lib/codegen/meta/isa/x86/encodings.py +++ b/lib/codegen/meta/isa/x86/encodings.py @@ -3,9 +3,9 @@ x86 Encodings. """ from __future__ import absolute_import from cdsl.predicates import IsUnsignedInt, Not, And -from base.predicates import IsColocatedFunc, IsColocatedData +from base.predicates import IsColocatedFunc, IsColocatedData, LengthEquals from base import instructions as base -from base.formats import UnaryImm, FuncAddr, Call +from base.formats import UnaryImm, FuncAddr, Call, LoadComplex, StoreComplex from .defs import X86_64, X86_32 from . import recipes as r from . import settings as cfg @@ -19,6 +19,7 @@ try: from typing import TYPE_CHECKING, Any # noqa if TYPE_CHECKING: from cdsl.instructions import MaybeBoundInst # noqa + from cdsl.predicates import FieldPredicate # noqa except ImportError: pass @@ -54,6 +55,15 @@ def enc_x86_64(inst, recipe, *args, **kwargs): X86_64.enc(inst, *recipe(*args, **kwargs)) +def enc_x86_64_instp(inst, recipe, instp, *args, **kwargs): + # type: (MaybeBoundInst, r.TailRecipe, FieldPredicate, *int, **int) -> None + """ + Add encodings for `inst` to X86_64 with and without a REX prefix. + """ + X86_64.enc(inst, *recipe.rex(*args, **kwargs), instp=instp) + X86_64.enc(inst, *recipe(*args, **kwargs), instp=instp) + + def enc_both(inst, recipe, *args, **kwargs): # type: (MaybeBoundInst, r.TailRecipe, *int, **Any) -> None """ @@ -63,6 +73,15 @@ def enc_both(inst, recipe, *args, **kwargs): enc_x86_64(inst, recipe, *args, **kwargs) +def enc_both_instp(inst, recipe, instp, *args, **kwargs): + # type: (MaybeBoundInst, r.TailRecipe, FieldPredicate, *int, **Any) -> None + """ + Add encodings for `inst` to both X86_32 and X86_64. + """ + X86_32.enc(inst, *recipe(*args, **kwargs), instp=instp) + enc_x86_64_instp(inst, recipe, instp, *args, **kwargs) + + def enc_i32_i64(inst, recipe, *args, **kwargs): # type: (MaybeBoundInst, r.TailRecipe, *int, **int) -> None """ @@ -80,6 +99,25 @@ def enc_i32_i64(inst, recipe, *args, **kwargs): X86_64.enc(inst.i64, *recipe.rex(*args, w=1, **kwargs)) +def enc_i32_i64_instp(inst, recipe, instp, *args, **kwargs): + # type: (MaybeBoundInst, r.TailRecipe, FieldPredicate, *int, **int) -> None + """ + Add encodings for `inst.i32` to X86_32. + Add encodings for `inst.i32` to X86_64 with and without REX. + Add encodings for `inst.i64` to X86_64 with a REX.W prefix. + + Similar to `enc_i32_i64` but applies `instp` to each encoding. + """ + X86_32.enc(inst.i32, *recipe(*args, **kwargs), instp=instp) + + # REX-less encoding must come after REX encoding so we don't use it by + # default. Otherwise reg-alloc would never use r8 and up. + X86_64.enc(inst.i32, *recipe.rex(*args, **kwargs), instp=instp) + X86_64.enc(inst.i32, *recipe(*args, **kwargs), instp=instp) + + X86_64.enc(inst.i64, *recipe.rex(*args, w=1, **kwargs), instp=instp) + + def enc_i32_i64_ld_st(inst, w_bit, recipe, *args, **kwargs): # type: (MaybeBoundInst, bool, r.TailRecipe, *int, **int) -> None """ @@ -212,6 +250,31 @@ X86_64.enc(base.ctz.i32, *r.urm(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1) # # Loads and stores. # + +ldcomplexp = LengthEquals(LoadComplex, 2) +for recipe in [r.ldWithIndex, r.ldWithIndexDisp8, r.ldWithIndexDisp32]: + enc_i32_i64_instp(base.load_complex, recipe, ldcomplexp, 0x8b) + enc_x86_64_instp(base.uload32_complex, recipe, ldcomplexp, 0x8b) + X86_64.enc(base.sload32_complex, *recipe.rex(0x63, w=1), + instp=ldcomplexp) + enc_i32_i64_instp(base.uload16_complex, recipe, ldcomplexp, 0x0f, 0xb7) + enc_i32_i64_instp(base.sload16_complex, recipe, ldcomplexp, 0x0f, 0xbf) + enc_i32_i64_instp(base.uload8_complex, recipe, ldcomplexp, 0x0f, 0xb6) + enc_i32_i64_instp(base.sload8_complex, recipe, ldcomplexp, 0x0f, 0xbe) + +stcomplexp = LengthEquals(StoreComplex, 3) +for recipe in [r.stWithIndex, r.stWithIndexDisp8, r.stWithIndexDisp32]: + enc_i32_i64_instp(base.store_complex, recipe, stcomplexp, 0x89) + enc_x86_64_instp(base.istore32_complex, recipe, stcomplexp, 0x89) + enc_both_instp(base.istore16_complex.i32, recipe, stcomplexp, 0x66, 0x89) + enc_x86_64_instp(base.istore16_complex.i64, recipe, stcomplexp, 0x66, 0x89) + +for recipe in [r.stWithIndex_abcd, + r.stWithIndexDisp8_abcd, + r.stWithIndexDisp32_abcd]: + enc_both_instp(base.istore8_complex.i32, recipe, stcomplexp, 0x88) + enc_x86_64_instp(base.istore8_complex.i64, recipe, stcomplexp, 0x88) + for recipe in [r.st, r.stDisp8, r.stDisp32]: enc_i32_i64_ld_st(base.store, True, recipe, 0x89) enc_x86_64(base.istore32.i64.any, recipe, 0x89) @@ -286,18 +349,34 @@ enc_both(base.load.f32.any, r.fld, 0xf3, 0x0f, 0x10) enc_both(base.load.f32.any, r.fldDisp8, 0xf3, 0x0f, 0x10) enc_both(base.load.f32.any, r.fldDisp32, 0xf3, 0x0f, 0x10) +enc_both(base.load_complex.f32, r.fldWithIndex, 0xf3, 0x0f, 0x10) +enc_both(base.load_complex.f32, r.fldWithIndexDisp8, 0xf3, 0x0f, 0x10) +enc_both(base.load_complex.f32, r.fldWithIndexDisp32, 0xf3, 0x0f, 0x10) + enc_both(base.load.f64.any, r.fld, 0xf2, 0x0f, 0x10) enc_both(base.load.f64.any, r.fldDisp8, 0xf2, 0x0f, 0x10) enc_both(base.load.f64.any, r.fldDisp32, 0xf2, 0x0f, 0x10) +enc_both(base.load_complex.f64, r.fldWithIndex, 0xf2, 0x0f, 0x10) +enc_both(base.load_complex.f64, r.fldWithIndexDisp8, 0xf2, 0x0f, 0x10) +enc_both(base.load_complex.f64, r.fldWithIndexDisp32, 0xf2, 0x0f, 0x10) + enc_both(base.store.f32.any, r.fst, 0xf3, 0x0f, 0x11) enc_both(base.store.f32.any, r.fstDisp8, 0xf3, 0x0f, 0x11) enc_both(base.store.f32.any, r.fstDisp32, 0xf3, 0x0f, 0x11) +enc_both(base.store_complex.f32, r.fstWithIndex, 0xf3, 0x0f, 0x11) +enc_both(base.store_complex.f32, r.fstWithIndexDisp8, 0xf3, 0x0f, 0x11) +enc_both(base.store_complex.f32, r.fstWithIndexDisp32, 0xf3, 0x0f, 0x11) + enc_both(base.store.f64.any, r.fst, 0xf2, 0x0f, 0x11) enc_both(base.store.f64.any, r.fstDisp8, 0xf2, 0x0f, 0x11) enc_both(base.store.f64.any, r.fstDisp32, 0xf2, 0x0f, 0x11) +enc_both(base.store_complex.f64, r.fstWithIndex, 0xf2, 0x0f, 0x11) +enc_both(base.store_complex.f64, r.fstWithIndexDisp8, 0xf2, 0x0f, 0x11) +enc_both(base.store_complex.f64, r.fstWithIndexDisp32, 0xf2, 0x0f, 0x11) + enc_both(base.fill.f32, r.ffillSib32, 0xf3, 0x0f, 0x10) enc_both(base.regfill.f32, r.fregfill32, 0xf3, 0x0f, 0x10) enc_both(base.fill.f64, r.ffillSib32, 0xf2, 0x0f, 0x10) diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py index ccd85c0fef..d1f4bac0c2 100644 --- a/lib/codegen/meta/isa/x86/recipes.py +++ b/lib/codegen/meta/isa/x86/recipes.py @@ -14,6 +14,7 @@ from base.formats import IntSelect, IntCondTrap, FloatCondTrap from base.formats import Jump, Branch, BranchInt, BranchFloat from base.formats import Ternary, FuncAddr, UnaryGlobalVar from base.formats import RegMove, RegSpill, RegFill, CopySpecial +from base.formats import LoadComplex, StoreComplex from .registers import GPR, ABCD, FPR, GPR_DEREF_SAFE, GPR_ZERO_DEREF_SAFE from .registers import GPR8, FPR8, GPR8_DEREF_SAFE, GPR8_ZERO_DEREF_SAFE, FLAG from .registers import StackGPR32, StackFPR32 @@ -739,6 +740,22 @@ st = TailRecipe( modrm_rm(in_reg1, in_reg0, sink); ''') +# XX /r register-indirect store with index and no offset. +stWithIndex = TailRecipe( + 'stWithIndex', StoreComplex, size=2, + ins=(GPR, GPR_ZERO_DEREF_SAFE, GPR_DEREF_SAFE), + outs=(), + instp=IsEqual(StoreComplex.offset, 0), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink); + modrm_sib(in_reg0, sink); + sib(0, in_reg2, in_reg1, sink); + ''') + # XX /r register-indirect store with no offset. # Only ABCD allowed for stored value. This is for byte stores with no REX. st_abcd = TailRecipe( @@ -754,6 +771,23 @@ st_abcd = TailRecipe( modrm_rm(in_reg1, in_reg0, sink); ''') +# XX /r register-indirect store with index and no offset. +# Only ABCD allowed for stored value. This is for byte stores with no REX. +stWithIndex_abcd = TailRecipe( + 'stWithIndex_abcd', StoreComplex, size=2, + ins=(ABCD, GPR_ZERO_DEREF_SAFE, GPR_DEREF_SAFE), + outs=(), + instp=IsEqual(StoreComplex.offset, 0), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink); + modrm_sib(in_reg0, sink); + sib(0, in_reg2, in_reg1, sink); + ''') + # XX /r register-indirect store of FPR with no offset. fst = TailRecipe( 'fst', Store, size=1, ins=(FPR, GPR_ZERO_DEREF_SAFE), outs=(), @@ -766,6 +800,20 @@ fst = TailRecipe( PUT_OP(bits, rex2(in_reg1, in_reg0), sink); modrm_rm(in_reg1, in_reg0, sink); ''') +# XX /r register-indirect store with index and no offset of FPR. +fstWithIndex = TailRecipe( + 'fstWithIndex', StoreComplex, size=2, + ins=(FPR, GPR_ZERO_DEREF_SAFE, GPR_DEREF_SAFE), outs=(), + instp=IsEqual(StoreComplex.offset, 0), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink); + modrm_sib(in_reg0, sink); + sib(0, in_reg2, in_reg1, sink); + ''') # XX /r register-indirect store with 8-bit offset. stDisp8 = TailRecipe( @@ -781,6 +829,27 @@ stDisp8 = TailRecipe( let offset: i32 = offset.into(); sink.put1(offset as u8); ''') + +# XX /r register-indirect store with index and 8-bit offset. +stWithIndexDisp8 = TailRecipe( + 'stWithIndexDisp8', StoreComplex, size=3, + ins=(GPR, GPR, GPR_DEREF_SAFE), + outs=(), + instp=IsSignedInt(StoreComplex.offset, 8), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink); + modrm_sib_disp8(in_reg0, sink); + sib(0, in_reg2, in_reg1, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + ''') + +# XX /r register-indirect store with 8-bit offset. +# Only ABCD allowed for stored value. This is for byte stores with no REX. stDisp8_abcd = TailRecipe( 'stDisp8_abcd', Store, size=2, ins=(ABCD, GPR), outs=(), instp=IsSignedInt(Store.offset, 8), @@ -795,6 +864,27 @@ stDisp8_abcd = TailRecipe( let offset: i32 = offset.into(); sink.put1(offset as u8); ''') + +# XX /r register-indirect store with index and 8-bit offset. +# Only ABCD allowed for stored value. This is for byte stores with no REX. +stWithIndexDisp8_abcd = TailRecipe( + 'stWithIndexDisp8_abcd', StoreComplex, size=3, + ins=(ABCD, GPR, GPR_DEREF_SAFE), + outs=(), + instp=IsSignedInt(StoreComplex.offset, 8), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink); + modrm_sib_disp8(in_reg0, sink); + sib(0, in_reg2, in_reg1, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + ''') + +# XX /r register-indirect store with 8-bit offset of FPR. fstDisp8 = TailRecipe( 'fstDisp8', Store, size=2, ins=(FPR, GPR_DEREF_SAFE), outs=(), instp=IsSignedInt(Store.offset, 8), @@ -809,6 +899,24 @@ fstDisp8 = TailRecipe( sink.put1(offset as u8); ''') +# XX /r register-indirect store with index and 8-bit offset of FPR. +fstWithIndexDisp8 = TailRecipe( + 'fstWithIndexDisp8', StoreComplex, size=3, + ins=(FPR, GPR, GPR_DEREF_SAFE), + outs=(), + instp=IsSignedInt(StoreComplex.offset, 8), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink); + modrm_sib_disp8(in_reg0, sink); + sib(0, in_reg2, in_reg1, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + ''') + # XX /r register-indirect store with 32-bit offset. stDisp32 = TailRecipe( 'stDisp32', Store, size=5, ins=(GPR, GPR_DEREF_SAFE), outs=(), @@ -822,6 +930,27 @@ stDisp32 = TailRecipe( let offset: i32 = offset.into(); sink.put4(offset as u32); ''') + +# XX /r register-indirect store with index and 32-bit offset. +stWithIndexDisp32 = TailRecipe( + 'stWithIndexDisp32', StoreComplex, size=6, + ins=(GPR, GPR, GPR_DEREF_SAFE), + outs=(), + instp=IsSignedInt(StoreComplex.offset, 32), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink); + modrm_sib_disp32(in_reg0, sink); + sib(0, in_reg2, in_reg1, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + ''') + +# XX /r register-indirect store with 32-bit offset. +# Only ABCD allowed for stored value. This is for byte stores with no REX. stDisp32_abcd = TailRecipe( 'stDisp32_abcd', Store, size=5, ins=(ABCD, GPR), outs=(), when_prefixed=stDisp32, @@ -835,6 +964,27 @@ stDisp32_abcd = TailRecipe( let offset: i32 = offset.into(); sink.put4(offset as u32); ''') + +# XX /r register-indirect store with index and 32-bit offset. +# Only ABCD allowed for stored value. This is for byte stores with no REX. +stWithIndexDisp32_abcd = TailRecipe( + 'stWithIndexDisp32_abcd', StoreComplex, size=6, + ins=(ABCD, GPR, GPR_DEREF_SAFE), + outs=(), + instp=IsSignedInt(StoreComplex.offset, 32), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink); + modrm_sib_disp32(in_reg0, sink); + sib(0, in_reg2, in_reg1, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + ''') + +# XX /r register-indirect store with 32-bit offset of FPR. fstDisp32 = TailRecipe( 'fstDisp32', Store, size=5, ins=(FPR, GPR_DEREF_SAFE), outs=(), clobbers_flags=False, @@ -848,6 +998,24 @@ fstDisp32 = TailRecipe( sink.put4(offset as u32); ''') +# XX /r register-indirect store with index and 32-bit offset of FPR. +fstWithIndexDisp32 = TailRecipe( + 'fstWithIndexDisp32', StoreComplex, size=6, + ins=(FPR, GPR, GPR_DEREF_SAFE), + outs=(), + instp=IsSignedInt(StoreComplex.offset, 32), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink); + modrm_sib_disp32(in_reg0, sink); + sib(0, in_reg2, in_reg1, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + ''') + # Unary spill with SIB and 32-bit displacement. spillSib32 = TailRecipe( 'spillSib32', Unary, size=6, ins=GPR, outs=StackGPR32, @@ -919,6 +1087,22 @@ ld = TailRecipe( modrm_rm(in_reg0, out_reg0, sink); ''') +# XX /r load with index and no offset. +ldWithIndex = TailRecipe( + 'ldWithIndex', LoadComplex, size=2, + ins=(GPR_ZERO_DEREF_SAFE, GPR_DEREF_SAFE), + outs=(GPR), + instp=IsEqual(LoadComplex.offset, 0), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink); + modrm_sib(out_reg0, sink); + sib(0, in_reg1, in_reg0, sink); + ''') + # XX /r float load with no offset. fld = TailRecipe( 'fld', Load, size=1, ins=(GPR_ZERO_DEREF_SAFE), outs=(FPR), @@ -932,6 +1116,22 @@ fld = TailRecipe( modrm_rm(in_reg0, out_reg0, sink); ''') +# XX /r float load with index and no offset. +fldWithIndex = TailRecipe( + 'fldWithIndex', LoadComplex, size=2, + ins=(GPR_ZERO_DEREF_SAFE, GPR_DEREF_SAFE), + outs=(FPR), + instp=IsEqual(LoadComplex.offset, 0), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink); + modrm_sib(out_reg0, sink); + sib(0, in_reg1, in_reg0, sink); + ''') + # XX /r load with 8-bit offset. ldDisp8 = TailRecipe( 'ldDisp8', Load, size=2, ins=(GPR_DEREF_SAFE), outs=(GPR), @@ -947,6 +1147,24 @@ ldDisp8 = TailRecipe( sink.put1(offset as u8); ''') +# XX /r load with index and 8-bit offset. +ldWithIndexDisp8 = TailRecipe( + 'ldWithIndexDisp8', LoadComplex, size=3, + ins=(GPR, GPR_DEREF_SAFE), + outs=(GPR), + instp=IsSignedInt(LoadComplex.offset, 8), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink); + modrm_sib_disp8(out_reg0, sink); + sib(0, in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + ''') + # XX /r float load with 8-bit offset. fldDisp8 = TailRecipe( 'fldDisp8', Load, size=2, ins=(GPR_DEREF_SAFE), outs=(FPR), @@ -962,6 +1180,24 @@ fldDisp8 = TailRecipe( sink.put1(offset as u8); ''') +# XX /r float load with 8-bit offset. +fldWithIndexDisp8 = TailRecipe( + 'fldWithIndexDisp8', LoadComplex, size=3, + ins=(GPR, GPR_DEREF_SAFE), + outs=(FPR), + instp=IsSignedInt(LoadComplex.offset, 8), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink); + modrm_sib_disp8(out_reg0, sink); + sib(0, in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put1(offset as u8); + ''') + # XX /r load with 32-bit offset. ldDisp32 = TailRecipe( 'ldDisp32', Load, size=5, ins=(GPR_DEREF_SAFE), outs=(GPR), @@ -977,6 +1213,24 @@ ldDisp32 = TailRecipe( sink.put4(offset as u32); ''') +# XX /r load with index and 32-bit offset. +ldWithIndexDisp32 = TailRecipe( + 'ldWithIndexDisp32', LoadComplex, size=6, + ins=(GPR, GPR_DEREF_SAFE), + outs=(GPR), + instp=IsSignedInt(LoadComplex.offset, 32), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink); + modrm_sib_disp32(out_reg0, sink); + sib(0, in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + ''') + # XX /r float load with 32-bit offset. fldDisp32 = TailRecipe( 'fldDisp32', Load, size=5, ins=(GPR_DEREF_SAFE), outs=(FPR), @@ -992,6 +1246,24 @@ fldDisp32 = TailRecipe( sink.put4(offset as u32); ''') +# XX /r float load with index and 32-bit offset. +fldWithIndexDisp32 = TailRecipe( + 'fldWithIndexDisp32', LoadComplex, size=6, + ins=(GPR, GPR_DEREF_SAFE), + outs=(FPR), + instp=IsSignedInt(LoadComplex.offset, 32), + clobbers_flags=False, + emit=''' + if !flags.notrap() { + sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]); + } + PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink); + modrm_sib_disp32(out_reg0, sink); + sib(0, in_reg1, in_reg0, sink); + let offset: i32 = offset.into(); + sink.put4(offset as u32); + ''') + # Unary fill with SIB and 32-bit displacement. fillSib32 = TailRecipe( 'fillSib32', Unary, size=6, ins=StackGPR32, outs=GPR, diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs index d94eb37830..7b21e68d28 100644 --- a/lib/codegen/src/isa/mod.rs +++ b/lib/codegen/src/isa/mod.rs @@ -162,6 +162,11 @@ pub trait TargetIsa: fmt::Display { false } + /// Does the CPU implement multi-register addressing? + fn uses_complex_addresses(&self) -> bool { + false + } + /// Get a data structure describing the registers in this ISA. fn register_info(&self) -> RegInfo; diff --git a/lib/codegen/src/isa/x86/binemit.rs b/lib/codegen/src/isa/x86/binemit.rs index 3a7c14d7be..6b9c709bf2 100644 --- a/lib/codegen/src/isa/x86/binemit.rs +++ b/lib/codegen/src/isa/x86/binemit.rs @@ -46,6 +46,18 @@ fn rex2(rm: RegUnit, reg: RegUnit) -> u8 { BASE_REX | b | (r << 2) } +// Create a three-register REX prefix, setting: +// +// REX.B = bit 3 of r/m register, or SIB base register when a SIB byte is present. +// REX.R = bit 3 of reg register. +// REX.X = bit 3 of SIB index register. +fn rex3(rm: RegUnit, reg: RegUnit, index: RegUnit) -> u8 { + let b = ((rm >> 3) & 1) as u8; + let r = ((reg >> 3) & 1) as u8; + let x = ((index >> 3) & 1) as u8; + BASE_REX | b | (x << 1) | (r << 2) +} + // Emit a REX prefix. // // The R, X, and B bits are computed from registers using the functions above. The W bit is @@ -211,7 +223,19 @@ fn modrm_disp32(rm: RegUnit, reg: RegUnit, sink: &mut CS) sink.put1(b); } -/// Emit a mode 10 ModR/M byte indicating that a SIB byte is present. +/// Emit a mode 00 ModR/M with a 100 RM indicating a SIB byte is present. +fn modrm_sib(reg: RegUnit, sink: &mut CS) { + modrm_rm(0b100, reg, sink); +} + +/// Emit a mode 01 ModR/M with a 100 RM indicating a SIB byte and 8-bit +/// displacement are present. +fn modrm_sib_disp8(reg: RegUnit, sink: &mut CS) { + modrm_disp8(0b100, reg, sink); +} + +/// Emit a mode 10 ModR/M with a 100 RM indicating a SIB byte and 32-bit +/// displacement are present. fn modrm_sib_disp32(reg: RegUnit, sink: &mut CS) { modrm_disp32(0b100, reg, sink); } @@ -225,6 +249,16 @@ fn sib_noindex(base: RegUnit, sink: &mut CS) { sink.put1(b); } +fn sib(scale: u8, index: RegUnit, base: RegUnit, sink: &mut CS) { + // SIB SS_III_BBB. + debug_assert_eq!(scale & !0x03, 0, "Scale out of range"); + let scale = scale & 3; + let index = index as u8 & 7; + let base = base as u8 & 7; + let b: u8 = (scale << 6) | (index << 3) | base; + sink.put1(b); +} + /// Get the low 4 bits of an opcode for an integer condition code. /// /// Add this offset to a base opcode for: diff --git a/lib/codegen/src/isa/x86/mod.rs b/lib/codegen/src/isa/x86/mod.rs index 787ec4a992..335d95cd22 100644 --- a/lib/codegen/src/isa/x86/mod.rs +++ b/lib/codegen/src/isa/x86/mod.rs @@ -62,6 +62,10 @@ impl TargetIsa for Isa { true } + fn uses_complex_addresses(&self) -> bool { + true + } + fn register_info(&self) -> RegInfo { registers::INFO.clone() } diff --git a/lib/codegen/src/postopt.rs b/lib/codegen/src/postopt.rs index 18f467a68a..6de918d08d 100644 --- a/lib/codegen/src/postopt.rs +++ b/lib/codegen/src/postopt.rs @@ -5,9 +5,9 @@ use cursor::{Cursor, EncCursor}; use ir::condcodes::{CondCode, FloatCC, IntCC}; use ir::dfg::ValueDef; -use ir::immediates::Imm64; +use ir::immediates::{Imm64, Offset32}; use ir::instructions::{Opcode, ValueList}; -use ir::{Ebb, Function, Inst, InstBuilder, InstructionData, Value}; +use ir::{Ebb, Function, Inst, InstBuilder, InstructionData, Value, Type, MemFlags}; use isa::TargetIsa; use timing; @@ -173,6 +173,158 @@ fn optimize_cpu_flags( pos.func.update_encoding(info.br_inst, isa).is_ok(); } + +struct MemOpInfo { + opcode: Opcode, + inst: Inst, + itype: Type, + arg: Value, + st_arg: Option, + flags: MemFlags, + offset: Offset32, + add_args: Option<[Value; 2]>, +} + +fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa) { + let mut info = match pos.func.dfg[inst] { + InstructionData::Load { + opcode, + arg, + flags, + offset, + } => MemOpInfo { + opcode: opcode, + inst: inst, + itype: pos.func.dfg.ctrl_typevar(inst), + arg: arg, + st_arg: None, + flags: flags, + offset: offset, + add_args: None, + }, + InstructionData::Store { + opcode, + args, + flags, + offset, + } => MemOpInfo { + opcode: opcode, + inst: inst, + itype: pos.func.dfg.ctrl_typevar(inst), + arg: args[1], + st_arg: Some(args[0]), + flags: flags, + offset: offset, + add_args: None, + }, + _ => return, + }; + + if let ValueDef::Result(result_inst, _) = pos.func.dfg.value_def(info.arg) { + match pos.func.dfg[result_inst] { + InstructionData::Binary { opcode, args } if opcode == Opcode::Iadd => { + info.add_args = Some(args.clone()); + } + _ => return, + } + } else { + return; + } + + match info.opcode { + Opcode::Load => { + pos.func.dfg.replace(info.inst).load_complex( + info.itype, + info.flags, + &info.add_args.unwrap(), + info.offset, + ); + } + Opcode::Uload8 => { + pos.func.dfg.replace(info.inst).uload8_complex( + info.itype, + info.flags, + &info.add_args.unwrap(), + info.offset, + ); + } + Opcode::Sload8 => { + pos.func.dfg.replace(info.inst).sload8_complex( + info.itype, + info.flags, + &info.add_args.unwrap(), + info.offset, + ); + } + Opcode::Uload16 => { + pos.func.dfg.replace(info.inst).uload16_complex( + info.itype, + info.flags, + &info.add_args.unwrap(), + info.offset, + ); + } + Opcode::Sload16 => { + pos.func.dfg.replace(info.inst).sload16_complex( + info.itype, + info.flags, + &info.add_args.unwrap(), + info.offset, + ); + } + Opcode::Uload32 => { + pos.func.dfg.replace(info.inst).uload32_complex( + info.flags, + &info.add_args.unwrap(), + info.offset, + ); + } + Opcode::Sload32 => { + pos.func.dfg.replace(info.inst).sload32_complex( + info.flags, + &info.add_args.unwrap(), + info.offset, + ); + } + Opcode::Store => { + pos.func.dfg.replace(info.inst).store_complex( + info.flags, + info.st_arg.unwrap(), + &info.add_args.unwrap(), + info.offset, + ); + } + Opcode::Istore8 => { + pos.func.dfg.replace(info.inst).istore8_complex( + info.flags, + info.st_arg.unwrap(), + &info.add_args.unwrap(), + info.offset, + ); + } + Opcode::Istore16 => { + pos.func.dfg.replace(info.inst).istore16_complex( + info.flags, + info.st_arg.unwrap(), + &info.add_args.unwrap(), + info.offset, + ); + } + Opcode::Istore32 => { + pos.func.dfg.replace(info.inst).istore32_complex( + info.flags, + info.st_arg.unwrap(), + &info.add_args.unwrap(), + info.offset, + ); + } + _ => return, + } + pos.func.update_encoding(info.inst, isa).is_ok(); +} + + + //---------------------------------------------------------------------- // // The main post-opt pass. @@ -198,6 +350,10 @@ pub fn do_postopt(func: &mut Function, isa: &TargetIsa) { } } } + + if isa.uses_complex_addresses() { + optimize_complex_addresses(&mut pos, inst, isa); + } } } } diff --git a/lib/codegen/src/predicates.rs b/lib/codegen/src/predicates.rs index e35e7ad179..1fb700024e 100644 --- a/lib/codegen/src/predicates.rs +++ b/lib/codegen/src/predicates.rs @@ -46,6 +46,11 @@ pub fn is_colocated_data(global_var: ir::GlobalVar, func: &ir::Function) -> bool } } +#[allow(dead_code)] +pub fn has_length_of(value_list: &ir::ValueList, num: usize, func: &ir::Function) -> bool { + value_list.len(&func.dfg.value_lists) == num +} + #[cfg(test)] mod tests { use super::*; diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs index 6db1f871f7..490222272e 100644 --- a/lib/codegen/src/verifier/mod.rs +++ b/lib/codegen/src/verifier/mod.rs @@ -335,6 +335,12 @@ impl<'a> Verifier<'a> { RegFill { src, .. } => { self.verify_stack_slot(inst, src)?; } + LoadComplex { ref args, .. } => { + self.verify_value_list(inst, args)?; + } + StoreComplex { ref args, .. } => { + self.verify_value_list(inst, args)?; + } // Exhaustive list so we can't forget to add new formats Unary { .. } | @@ -1149,8 +1155,8 @@ impl<'a> Verifier<'a> { mod tests { use super::{Error, Verifier}; use entity::EntityList; - use ir::Function; use ir::instructions::{InstructionData, Opcode}; + use ir::Function; use settings; macro_rules! assert_err_with_msg { diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs index b3887949c2..9a5be48b6f 100644 --- a/lib/codegen/src/write.rs +++ b/lib/codegen/src/write.rs @@ -369,12 +369,44 @@ pub fn write_operands( } => write!(w, " {}, {}{}", arg, stack_slot, offset), HeapAddr { heap, arg, imm, .. } => write!(w, " {}, {}, {}", heap, arg, imm), Load { flags, arg, offset, .. } => write!(w, "{} {}{}", flags, arg, offset), + LoadComplex { + flags, + ref args, + offset, + .. + } => { + let args = args.as_slice(pool); + write!( + w, + "{} {}{}", + flags, + DisplayValuesWithDelimiter(&args, '+'), + offset + ) + + } Store { flags, args, offset, .. } => write!(w, "{} {}, {}{}", flags, args[0], args[1], offset), + StoreComplex { + flags, + ref args, + offset, + .. + } => { + let args = args.as_slice(pool); + write!( + w, + "{} {}, {}{}", + flags, + args[0], + DisplayValuesWithDelimiter(&args[1..], '+'), + offset + ) + } RegMove { arg, src, dst, .. } => { if let Some(isa) = isa { let regs = isa.register_info(); @@ -450,6 +482,21 @@ impl<'a> fmt::Display for DisplayValues<'a> { } } +struct DisplayValuesWithDelimiter<'a>(&'a [Value], char); + +impl<'a> fmt::Display for DisplayValuesWithDelimiter<'a> { + fn fmt(&self, f: &mut fmt::Formatter) -> Result { + for (i, val) in self.0.iter().enumerate() { + if i == 0 { + write!(f, "{}", val)?; + } else { + write!(f, "{}{}", self.1, val)?; + } + } + Ok(()) + } +} + #[cfg(test)] mod tests { use ir::types; diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index f6d89881fb..17b99249fb 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -22,6 +22,7 @@ pub enum Token<'a> { LBracket, // '[' RBracket, // ']' Minus, // '-' + Plus, // '+' Comma, // ',' Dot, // '.' Colon, // ':' @@ -169,6 +170,25 @@ impl<'a> Lexer<'a> { self.source[self.pos..].starts_with(prefix) } + // Starting from `lookahead`, are we looking at a number? + fn looking_at_numeric(&self) -> bool { + if let Some(c) = self.lookahead { + if c.is_digit(10) { + return true; + } + match c { + '-' => return true, + '+' => return true, + '.' => return true, + _ => {} + } + if self.looking_at("NaN") || self.looking_at("Inf") || self.looking_at("sNaN") { + return true; + } + } + false + } + // Scan a single-char token. fn scan_char(&mut self, tok: Token<'a>) -> Result, LocatedError> { assert_ne!(self.lookahead, None); @@ -234,16 +254,17 @@ impl<'a> Lexer<'a> { match self.lookahead { Some('-') => { self.next_ch(); - - if let Some(c) = self.lookahead { - // If the next character won't parse as a number, we return Token::Minus - if !c.is_alphanumeric() && c != '.' { - return token(Token::Minus, loc); - } + if !self.looking_at_numeric() { + // If the next characters won't parse as a number, we return Token::Minus + return token(Token::Minus, loc); } } Some('+') => { self.next_ch(); + if !self.looking_at_numeric() { + // If the next characters won't parse as a number, we return Token::Minus + return token(Token::Plus, loc); + } } _ => {} } diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 67e6939910..40af898efc 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -13,8 +13,8 @@ use cretonne_codegen::ir::{AbiParam, ArgumentExtension, ArgumentLoc, Ebb, ExtFun Type, Value, ValueLoc}; use cretonne_codegen::isa::{self, Encoding, RegUnit, TargetIsa}; use cretonne_codegen::packed_option::ReservedValue; -use cretonne_codegen::{settings, timing}; use cretonne_codegen::settings::CallConv; +use cretonne_codegen::{settings, timing}; use error::{Error, Location, Result}; use isaspec; use lexer::{self, Lexer, Token}; @@ -1872,6 +1872,24 @@ impl<'a> Parser<'a> { Ok(args) } + fn parse_value_sequence(&mut self) -> Result { + let mut args = VariableArgs::new(); + + if let Some(Token::Value(v)) = self.token() { + args.push(v); + self.consume(); + } else { + return Ok(args); + } + + while self.optional(Token::Plus) { + args.push(self.match_value("expected value in argument list")?); + } + + Ok(args) + + } + // Parse an optional value list enclosed in parantheses. fn parse_opt_value_list(&mut self) -> Result { if !self.optional(Token::LPar) { @@ -2267,6 +2285,17 @@ impl<'a> Parser<'a> { offset, } } + InstructionFormat::LoadComplex => { + let flags = self.optional_memflags(); + let args = self.parse_value_sequence()?; + let offset = self.optional_offset32()?; + InstructionData::LoadComplex { + opcode, + flags, + args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists), + offset, + } + } InstructionFormat::Store => { let flags = self.optional_memflags(); let arg = self.match_value("expected SSA value operand")?; @@ -2283,6 +2312,23 @@ impl<'a> Parser<'a> { offset, } } + + InstructionFormat::StoreComplex => { + let flags = self.optional_memflags(); + let src = self.match_value("expected SSA value operand")?; + self.match_token( + Token::Comma, + "expected ',' between operands", + )?; + let args = self.parse_value_sequence()?; + let offset = self.optional_offset32()?; + InstructionData::StoreComplex { + opcode, + flags, + args: args.into_value_list(&[src], &mut ctx.function.dfg.value_lists), + offset, + } + } InstructionFormat::RegMove => { let arg = self.match_value("expected SSA value operand")?; self.match_token( @@ -2402,9 +2448,9 @@ impl<'a> Parser<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne_codegen::ir::StackSlotKind; use cretonne_codegen::ir::entities::AnyEntity; use cretonne_codegen::ir::types; + use cretonne_codegen::ir::StackSlotKind; use cretonne_codegen::ir::{ArgumentExtension, ArgumentPurpose}; use cretonne_codegen::settings::CallConv; use error::Error; From b36fc6b75f1ee60b4d0c8b4d58c758389f1cb8e9 Mon Sep 17 00:00:00 2001 From: pup Date: Wed, 9 May 2018 16:11:58 -0400 Subject: [PATCH 1775/3084] Issue 311 - Add a pass to make NaN bits deterministic. (#322) --- lib/codegen/meta/base/settings.py | 10 +++ lib/codegen/src/context.rs | 10 +++ lib/codegen/src/lib.rs | 1 + lib/codegen/src/nan_canonicalization.rs | 81 +++++++++++++++++++++++++ lib/codegen/src/settings.rs | 1 + lib/codegen/src/timing.rs | 2 + 6 files changed, 105 insertions(+) create mode 100644 lib/codegen/src/nan_canonicalization.rs diff --git a/lib/codegen/meta/base/settings.py b/lib/codegen/meta/base/settings.py index a5107e5b4c..5753c35273 100644 --- a/lib/codegen/meta/base/settings.py +++ b/lib/codegen/meta/base/settings.py @@ -99,6 +99,16 @@ enable_float = BoolSetting( """, default=True) +enable_nan_canonicalization = BoolSetting( + """ + Enable NaN canonicalization + + This replaces NaNs with a single canonical value, for users requiring + entirely deterministic WebAssembly computation. This is not required + by the WebAssembly spec, so it is not enabled by default. + """, + default=False) + enable_simd = BoolSetting( """Enable the use of SIMD instructions.""", default=True) diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index c2aab76bad..b383160703 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -18,6 +18,7 @@ use isa::TargetIsa; use legalize_function; use licm::do_licm; use loop_analysis::LoopAnalysis; +use nan_canonicalization::do_nan_canonicalization; use postopt::do_postopt; use preopt::do_preopt; use regalloc; @@ -124,6 +125,9 @@ impl Context { if isa.flags().opt_level() != OptLevel::Fastest { self.preopt(isa)?; } + if isa.flags().enable_nan_canonicalization() { + self.canonicalize_nans(isa)?; + } self.legalize(isa)?; if isa.flags().opt_level() != OptLevel::Fastest { self.postopt(isa)?; @@ -212,6 +216,12 @@ impl Context { Ok(()) } + /// Perform NaN canonicalizing rewrites on the function. + pub fn canonicalize_nans(&mut self, isa: &TargetIsa) -> CtonResult { + do_nan_canonicalization(&mut self.func); + self.verify_if(isa) + } + /// Run the legalizer for `isa` on the function. pub fn legalize(&mut self, isa: &TargetIsa) -> CtonResult { // Legalization invalidates the domtree and loop_analysis by mutating the CFG. diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs index faa21d24d7..7944c852a0 100644 --- a/lib/codegen/src/lib.rs +++ b/lib/codegen/src/lib.rs @@ -90,6 +90,7 @@ pub use entity::packed_option; mod abi; mod bitset; +mod nan_canonicalization; mod constant_hash; mod context; mod dce; diff --git a/lib/codegen/src/nan_canonicalization.rs b/lib/codegen/src/nan_canonicalization.rs new file mode 100644 index 0000000000..20877c3e18 --- /dev/null +++ b/lib/codegen/src/nan_canonicalization.rs @@ -0,0 +1,81 @@ +//! A NaN-canonicalizing rewriting pass. Patch floating point arithmetic +//! instructions that may return a NaN result with a sequence of operations +//! that will replace nondeterministic NaN's with a single canonical NaN value. + +use cursor::{Cursor, FuncCursor}; +use ir::{Function, Inst, InstBuilder, InstructionData, Opcode, Value}; +use ir::condcodes::FloatCC; +use ir::immediates::{Ieee32, Ieee64}; +use ir::types; +use ir::types::Type; +use timing; + +// Canonical 32-bit and 64-bit NaN values. +static CANON_32BIT_NAN: u32 = 0b01111111110000000000000000000000; +static CANON_64BIT_NAN: u64 = 0b0111111111111000000000000000000000000000000000000000000000000000; + +/// Perform the NaN canonicalization pass. +pub fn do_nan_canonicalization(func: &mut Function) { + let _tt = timing::canonicalize_nans(); + let mut pos = FuncCursor::new(func); + while let Some(_ebb) = pos.next_ebb() { + while let Some(inst) = pos.next_inst() { + if is_fp_arith(&mut pos, inst) { + add_nan_canon_seq(&mut pos, inst); + } + } + } +} + +/// Returns true/false based on whether the instruction is a floating-point +/// arithmetic operation. This ignores operations like `fneg`, `fabs`, or +/// `fcopysign` that only operate on the sign bit of a floating point value. +fn is_fp_arith(pos: &mut FuncCursor, inst: Inst) -> bool { + match pos.func.dfg[inst] { + InstructionData::Unary { opcode, .. } => { + opcode == Opcode::Ceil || opcode == Opcode::Floor || opcode == Opcode::Nearest || + opcode == Opcode::Sqrt || opcode == Opcode::Trunc + } + InstructionData::Binary { opcode, .. } => { + opcode == Opcode::Fadd || opcode == Opcode::Fdiv || opcode == Opcode::Fmax || + opcode == Opcode::Fmin || opcode == Opcode::Fmul || + opcode == Opcode::Fsub + } + InstructionData::Ternary { opcode, .. } => opcode == Opcode::Fma, + _ => false, + } +} + +/// Append a sequence of canonicalizing instructions after the given instruction. +fn add_nan_canon_seq(pos: &mut FuncCursor, inst: Inst) { + // Select the instruction result, result type. Replace the instruction + // result and step forward before inserting the canonicalization sequence. + let val = pos.func.dfg.first_result(inst); + let val_type = pos.func.dfg.value_type(val); + let new_res = pos.func.dfg.replace_result(val, val_type); + let _next_inst = pos.next_inst().expect("EBB missing terminator!"); + + // Insert a comparison instruction, to check if `inst_res` is NaN. Select + // the canonical NaN value if `val` is NaN, assign the result to `inst`. + let is_nan = pos.ins().fcmp(FloatCC::NotEqual, new_res, new_res); + let canon_nan = insert_nan_const(pos, val_type); + pos.ins().with_result(val).select( + is_nan, + canon_nan, + new_res, + ); + + pos.prev_inst(); // Step backwards so the pass does not skip instructions. +} + +/// Insert a canonical 32-bit or 64-bit NaN constant at the current position. +fn insert_nan_const(pos: &mut FuncCursor, nan_type: Type) -> Value { + match nan_type { + types::F32 => pos.ins().f32const(Ieee32::with_bits(CANON_32BIT_NAN)), + types::F64 => pos.ins().f64const(Ieee64::with_bits(CANON_64BIT_NAN)), + _ => { + // Panic if the type given was not an IEEE floating point type. + panic!("Could not canonicalize NaN: Unexpected result type found."); + } + } +} diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs index 39416a298f..422eedc342 100644 --- a/lib/codegen/src/settings.rs +++ b/lib/codegen/src/settings.rs @@ -369,6 +369,7 @@ mod tests { avoid_div_traps = false\n\ is_compressed = false\n\ enable_float = true\n\ + enable_nan_canonicalization = false\n\ enable_simd = true\n\ enable_atomics = true\n\ baldrdash_prologue_words = 0\n\ diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs index 017f3a466e..556ca56de1 100644 --- a/lib/codegen/src/timing.rs +++ b/lib/codegen/src/timing.rs @@ -73,6 +73,8 @@ define_passes!{ prologue_epilogue: "Prologue/epilogue insertion", binemit: "Binary machine code emission", layout_renumber: "Layout full renumbering", + + canonicalize_nans: "Canonicalization of NaNs", } impl Pass { From 34b8c00d61b551c8c039913a8f96215b6b9d311f Mon Sep 17 00:00:00 2001 From: pup Date: Sat, 12 May 2018 13:30:15 -0400 Subject: [PATCH 1776/3084] Minor typo fix in `constant_hash` comment. (#329) --- lib/codegen/meta/constant_hash.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/codegen/meta/constant_hash.py b/lib/codegen/meta/constant_hash.py index 9ef19ff098..e282936723 100644 --- a/lib/codegen/meta/constant_hash.py +++ b/lib/codegen/meta/constant_hash.py @@ -2,7 +2,7 @@ Generate constant hash tables. The `constant_hash` module can generate constant pre-populated hash tables. We -don't attempt parfect hashing, but simply generate an open addressed +don't attempt perfect hashing, but simply generate an open addressed quadratically probed hash table. """ from __future__ import absolute_import From e9a0a9977d2f01031db25a53d2cc017c2abb7a07 Mon Sep 17 00:00:00 2001 From: Jordan Danford Date: Sat, 12 May 2018 10:49:07 -0700 Subject: [PATCH 1777/3084] Use SVG version of Gitter icon in README.md (#331) --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 59316dff13..0e30d5aa13 100644 --- a/README.rst +++ b/README.rst @@ -14,7 +14,7 @@ machine code. :target: https://travis-ci.org/cretonne/cretonne :alt: Build Status -.. image:: https://badges.gitter.im/cretonne/cretonne.png +.. image:: https://badges.gitter.im/cretonne/cretonne.svg :target: https://gitter.im/cretonne/Lobby/~chat :alt: Gitter chat From 07c65bab1107bc6bb5f25cfe750412007beb36d6 Mon Sep 17 00:00:00 2001 From: Bruce Mitchener Date: Tue, 15 May 2018 02:27:14 +0700 Subject: [PATCH 1778/3084] Update to docopt 1.0. (#332) --- cranelift/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 3edf07897d..cf3cbe5aa6 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -24,7 +24,7 @@ cretonne-faerie = { path = "lib/faerie", version = "0.8.0" } cretonne-simplejit = { path = "lib/simplejit", version = "0.8.0" } cretonne = { path = "lib/umbrella", version = "0.8.0" } filecheck = "0.3.0" -docopt = "0.8.0" +docopt = "1" serde = "1.0.8" serde_derive = "1.0.8" tempdir = "0.3.5" From 19240397134b4f0ad0c5fc46b3fcb5aaa92a170f Mon Sep 17 00:00:00 2001 From: Jonathan Foote Date: Mon, 14 May 2018 15:31:27 -0400 Subject: [PATCH 1779/3084] cargo fuzz integration (#306) * added wip translate_module fuzzer * use local binaryen-rs fork (with shim) for fuzzing * minor doc cleanup * check fuzzer integration via CI * switch back to upstream binaryen-rs; add forgotten integration test directive --- cranelift/FUZZING.md | 9 ++++++ cranelift/fuzz/Cargo.toml | 29 ++++++++++++++++++ .../ffaefab69523eb11935a9b420d58826c8ea65c4c | Bin 0 -> 2375 bytes cranelift/fuzz/fuzz_translate_module.rs | 15 +++++++++ cranelift/test-all.sh | 15 +++++++++ 5 files changed, 68 insertions(+) create mode 100644 cranelift/FUZZING.md create mode 100644 cranelift/fuzz/Cargo.toml create mode 100644 cranelift/fuzz/corpus/fuzz_translate_module/ffaefab69523eb11935a9b420d58826c8ea65c4c create mode 100644 cranelift/fuzz/fuzz_translate_module.rs diff --git a/cranelift/FUZZING.md b/cranelift/FUZZING.md new file mode 100644 index 0000000000..a542e139d6 --- /dev/null +++ b/cranelift/FUZZING.md @@ -0,0 +1,9 @@ +# Fuzzing + +This document describes how to fuzz cretonne with [`cargo-fuzz`](https://github.com/rust-fuzz/cargo-fuzz). The fuzz targets use `wasm-opt` from [`binaryen-rs`](https://github.com/pepyakin/binaryen-rs) to generate valid WebAssembly modules from the fuzzed input supplied by `cargo-fuzz` (via [libfuzzer](http://llvm.org/docs/LibFuzzer.html)). In this scheme coverage feedback from both cretonne and the `wasm-opt` input generation code is used to inform the fuzzer. + +# Usage + +1. Install all dependencies required to build `binaryen-rs` and `cargo-fuzz` (including `cmake`) +2. Use the rust nightly toolchain (required by `cargo-fuzz`): `rustup override set nightly` +3. Execute the fuzz target: `cargo fuzz run fuzz_translate_module` diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml new file mode 100644 index 0000000000..d48922c1f4 --- /dev/null +++ b/cranelift/fuzz/Cargo.toml @@ -0,0 +1,29 @@ +[package] +name = "cton-wasm-fuzz" +version = "0.0.1" +authors = ["foote@fastly.com"] +publish = false + +[package.metadata] +cargo-fuzz = true + +[dependencies.cargo-fuzz] +version = "*" + +[dependencies.binaryen] +git = "https://github.com/pepyakin/binaryen-rs.git" +version = "*" + +[dependencies.libfuzzer-sys] +git = "https://github.com/rust-fuzz/libfuzzer-sys.git" + +[dependencies.cretonne-wasm] +path = "../lib/wasm" + +# Prevent this from interfering with workspaces +[workspace] +members = ["."] + +[[bin]] +name = "fuzz_translate_module" +path = "fuzz_translate_module.rs" diff --git a/cranelift/fuzz/corpus/fuzz_translate_module/ffaefab69523eb11935a9b420d58826c8ea65c4c b/cranelift/fuzz/corpus/fuzz_translate_module/ffaefab69523eb11935a9b420d58826c8ea65c4c new file mode 100644 index 0000000000000000000000000000000000000000..1fa4159954301b3074cab3e33e46da5f826c0bc5 GIT binary patch literal 2375 zcmdPpVi*OZAwY)^2nB)!5CH}MX#{{?`HsVACb&4v5yaDegp|54q6Q*JJ*vnk4^Ie0 z Date: Wed, 9 May 2018 07:52:18 -1000 Subject: [PATCH 1780/3084] Improve the error message when python isn't installed. --- lib/codegen/build.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs index 69100ef00f..87d4b754ae 100644 --- a/lib/codegen/build.rs +++ b/lib/codegen/build.rs @@ -67,7 +67,9 @@ fn main() { .arg("--out-dir") .arg(out_dir) .status() - .expect("Failed to launch second-level build script"); + .expect( + "Failed to launch second-level build script; is python installed?", + ); if !status.success() { process::exit(status.code().unwrap()); } From f986acfe7c7c39ac69b47fc9b99a31f754b7bf13 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sat, 12 May 2018 05:51:20 -1000 Subject: [PATCH 1781/3084] Correct an assert message, spotted by @MarkSwanson --- lib/module/src/module.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index f3073f8468..409adf97c9 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -574,7 +574,7 @@ where let info = &self.contents.functions[func]; debug_assert!( info.decl.linkage.is_definable(), - "imported data cannot be finalized" + "imported function cannot be finalized" ); self.backend.finalize_function( info.compiled.as_ref().expect( From 4c150907bf13a96b4f0a7ad8728919c4d122c358 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 14 May 2018 20:05:14 -1000 Subject: [PATCH 1782/3084] Issue better error messages in use_var and def_var. Include the name of the variable when diagnosing uses and defs of undeclared variables. And, add an assert to def_var to check that the declared type of a variable matches the value type of the def. With this change, `Variable` implementations must now implement `Debug`. --- lib/frontend/src/frontend.rs | 59 ++++++++++++++++++++++++------------ 1 file changed, 39 insertions(+), 20 deletions(-) diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index b2e6385722..20aac932ce 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -10,6 +10,7 @@ use cretonne_codegen::ir::{DataFlowGraph, Ebb, ExtFuncData, FuncRef, Function, G use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::packed_option::PackedOption; use ssa::{Block, SSABuilder, SideEffects}; +use std::fmt::Debug; /// Structure used for translating a series of functions into Cretonne IR. /// @@ -22,7 +23,7 @@ use ssa::{Block, SSABuilder, SideEffects}; /// use here, `variable::Variable` can be used. pub struct FunctionBuilderContext where - Variable: EntityRef, + Variable: EntityRef + Debug, { ssa: SSABuilder, ebbs: EntityMap, @@ -32,7 +33,7 @@ where /// Temporary object used to build a single Cretonne IR `Function`. pub struct FunctionBuilder<'a, Variable: 'a> where - Variable: EntityRef, + Variable: EntityRef + Debug, { /// The function currently being built. /// This field is public so the function can be re-borrowed. @@ -79,7 +80,7 @@ impl Position { impl FunctionBuilderContext where - Variable: EntityRef, + Variable: EntityRef + Debug, { /// Creates a FunctionBuilderContext structure. The structure is automatically cleared after /// each [`FunctionBuilder`](struct.FunctionBuilder.html) completes translating a function. @@ -106,7 +107,7 @@ where /// one convenience method per Cretonne IR instruction. pub struct FuncInstBuilder<'short, 'long: 'short, Variable: 'long> where - Variable: EntityRef, + Variable: EntityRef + Debug, { builder: &'short mut FunctionBuilder<'long, Variable>, ebb: Ebb, @@ -114,7 +115,7 @@ where impl<'short, 'long, Variable> FuncInstBuilder<'short, 'long, Variable> where - Variable: EntityRef, + Variable: EntityRef + Debug, { fn new<'s, 'l>( builder: &'s mut FunctionBuilder<'l, Variable>, @@ -126,7 +127,7 @@ where impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long, Variable> where - Variable: EntityRef, + Variable: EntityRef + Debug, { fn data_flow_graph(&self) -> &DataFlowGraph { &self.builder.func.dfg @@ -228,7 +229,7 @@ where /// return instruction with arguments that don't match the function's signature. impl<'a, Variable> FunctionBuilder<'a, Variable> where - Variable: EntityRef, + Variable: EntityRef + Debug, { /// Creates a new FunctionBuilder structure that will operate on a `Function` using a /// `FunctionBuilderContext`. @@ -316,22 +317,40 @@ where /// Returns the Cretonne IR value corresponding to the utilization at the current program /// position of a previously defined user variable. pub fn use_var(&mut self, var: Variable) -> Value { - let ty = *self.func_ctx.types.get(var).expect( - "this variable is used but its type has not been declared", - ); - let (val, side_effects) = self.func_ctx.ssa.use_var( - self.func, - var, - ty, - self.position.basic_block.unwrap(), - ); + let (val, side_effects) = { + let ty = *self.func_ctx.types.get(var).unwrap_or_else(|| { + panic!( + "variable {:?} is used but its type has not been declared", + var + ) + }); + self.func_ctx.ssa.use_var( + self.func, + var, + ty, + self.position.basic_block.unwrap(), + ) + }; self.handle_ssa_side_effects(side_effects); val } - /// Register a new definition of a user variable. Panics if the type of the value is not the - /// same as the type registered for the variable. + /// Register a new definition of a user variable. The type of the value must be + /// the same as the type registered for the variable. pub fn def_var(&mut self, var: Variable, val: Value) { + debug_assert_eq!( + *self.func_ctx.types.get(var).unwrap_or_else(|| { + panic!( + "variable {:?} is used but its type has not been declared", + var + ) + }), + self.func.dfg.value_type(val), + "declared type of variable {:?} doesn't match type of value {}", + var, + val + ); + self.func_ctx.ssa.def_var( var, val, @@ -470,7 +489,7 @@ where /// in ways that can be unsafe if used incorrectly. impl<'a, Variable> FunctionBuilder<'a, Variable> where - Variable: EntityRef, + Variable: EntityRef + Debug, { /// Retrieves all the parameters for an `Ebb` currently inferred from the jump instructions /// inserted that target it and the SSA construction. @@ -560,7 +579,7 @@ where // Helper functions impl<'a, Variable> FunctionBuilder<'a, Variable> where - Variable: EntityRef, + Variable: EntityRef + Debug, { fn move_to_next_basic_block(&mut self) { self.position.basic_block = PackedOption::from(self.func_ctx.ssa.declare_ebb_body_block( From 80fdfb23762bbe79800f6e24b87dd2707520ed23 Mon Sep 17 00:00:00 2001 From: Amanieu d'Antras Date: Fri, 18 May 2018 20:54:00 +0100 Subject: [PATCH 1783/3084] Optimize partition_slice (#341) * Generate debug symbols in optimized builds. This allows profiling tools to provide more accurate information, especially details about inlined functions. * Rewrite and optimize partition_slice This improves the performance of the register allocation passes which use LiveValueTracker. --- cranelift/Cargo.toml | 8 +++++ lib/codegen/src/partition_slice.rs | 56 ++++++++++++++++++++---------- 2 files changed, 46 insertions(+), 18 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index cf3cbe5aa6..b807c2fe01 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -32,3 +32,11 @@ term = "0.5.1" capstone = "0.3.1" [workspace] + +# We want debug symbols on release binaries by default since it allows profiling +# tools to give more accurate information. We can always strip them out later if +# necessary. +[profile.release] +debug = true +[profile.bench] +debug = true diff --git a/lib/codegen/src/partition_slice.rs b/lib/codegen/src/partition_slice.rs index dd0aff5da6..b1e0f95f85 100644 --- a/lib/codegen/src/partition_slice.rs +++ b/lib/codegen/src/partition_slice.rs @@ -1,5 +1,7 @@ //! Rearrange the elements in a slice according to a predicate. +use std::mem; + /// Rearrange the elements of the mutable slice `s` such that elements where `p(t)` is true precede /// the elements where `p(t)` is false. /// @@ -10,24 +12,42 @@ pub fn partition_slice(s: &mut [T], mut p: F) -> usize where F: FnMut(&T) -> bool, { - // Count the length of the prefix where `p` returns true. - let mut count = match s.iter().position(|t| !p(t)) { - Some(t) => t, - None => return s.len(), - }; + // The iterator works like a deque which we can pop from both ends. + let mut i = s.iter_mut(); - // Swap remaining `true` elements into place. - // - // This actually preserves the order of the `true` elements, but the `false` elements get - // shuffled. - for i in count + 1..s.len() { - if p(&s[i]) { - s.swap(count, i); - count += 1; - } + // Number of elements for which the predicate is known to be true. + let mut pos = 0; + + loop { + // Find the first element for which the predicate fails. + let head = loop { + match i.next() { + Some(head) => { + if !p(&head) { + break head; + } + } + None => return pos, + } + pos += 1; + }; + + // Find the last element for which the predicate succeeds. + let tail = loop { + match i.next_back() { + Some(tail) => { + if p(&tail) { + break tail; + } + } + None => return pos, + } + }; + + // Swap the two elements into the right order. + mem::swap(head, tail); + pos += 1; } - - count } #[cfg(test)] @@ -70,8 +90,8 @@ mod tests { check(&[1, 2, 3], &[1, 2, 3]); check(&[1, 2, 10], &[10, 2, 1]); // Note: 2, 1 order not required. check(&[1, 10, 2], &[10, 1, 2]); // Note: 1, 2 order not required. - check(&[1, 20, 10], &[20, 10, 1]); - check(&[1, 20, 3, 10], &[20, 10, 3, 1]); + check(&[1, 20, 10], &[10, 20, 1]); // Note: 10, 20 order not required. + check(&[1, 20, 3, 10], &[10, 20, 3, 1]); check(&[20, 3, 10, 1], &[20, 10, 3, 1]); } } From e9d362d90268f5c152f34c5f8f610aece3165439 Mon Sep 17 00:00:00 2001 From: Amanieu d'Antras Date: Fri, 18 May 2018 23:40:08 +0100 Subject: [PATCH 1784/3084] Improve coalescing performance by using a FxHashMap (#340) * Use FxHashMap instead of HashMap for better performance * Replace the binary search in the coalescing pass with a FxHashMap This speeds up coalescing by up to 16% and overall compilation by 9% --- lib/codegen/src/fx.rs | 111 ++++++++++++++++++ lib/codegen/src/lib.rs | 1 + lib/codegen/src/licm.rs | 10 +- lib/codegen/src/regalloc/coalescing.rs | 21 +--- .../src/regalloc/live_value_tracker.rs | 6 +- lib/codegen/src/scoped_hash_map.rs | 7 +- 6 files changed, 130 insertions(+), 26 deletions(-) create mode 100644 lib/codegen/src/fx.rs diff --git a/lib/codegen/src/fx.rs b/lib/codegen/src/fx.rs new file mode 100644 index 0000000000..b59bac27be --- /dev/null +++ b/lib/codegen/src/fx.rs @@ -0,0 +1,111 @@ +// This file is taken from the Rust compiler: src/librustc_data_structures/fx.rs + +// Copyright 2015 The Rust Project Developers. See the COPYRIGHT +// file at the top-level directory of this distribution and at +// http://rust-lang.org/COPYRIGHT. +// +// Licensed under the Apache License, Version 2.0 or the MIT license +// , at your +// option. This file may not be copied, modified, or distributed +// except according to those terms. + +use std::collections::{HashMap, HashSet}; +use std::default::Default; +use std::hash::{BuildHasherDefault, Hash, Hasher}; +use std::ops::BitXor; + +pub type FxHashMap = HashMap>; +pub type FxHashSet = HashSet>; + +#[allow(non_snake_case)] +pub fn FxHashMap() -> FxHashMap { + HashMap::default() +} + +#[allow(non_snake_case)] +pub fn FxHashSet() -> FxHashSet { + HashSet::default() +} + +/// A speedy hash algorithm for use within rustc. The hashmap in liballoc +/// by default uses SipHash which isn't quite as speedy as we want. In the +/// compiler we're not really worried about DOS attempts, so we use a fast +/// non-cryptographic hash. +/// +/// This is the same as the algorithm used by Firefox -- which is a homespun +/// one not based on any widely-known algorithm -- though modified to produce +/// 64-bit hash values instead of 32-bit hash values. It consistently +/// out-performs an FNV-based hash within rustc itself -- the collision rate is +/// similar or slightly worse than FNV, but the speed of the hash function +/// itself is much higher because it works on up to 8 bytes at a time. +pub struct FxHasher { + hash: usize, +} + +#[cfg(target_pointer_width = "32")] +const K: usize = 0x9e3779b9; +#[cfg(target_pointer_width = "64")] +const K: usize = 0x517cc1b727220a95; + +impl Default for FxHasher { + #[inline] + fn default() -> FxHasher { + FxHasher { hash: 0 } + } +} + +impl FxHasher { + #[inline] + fn add_to_hash(&mut self, i: usize) { + self.hash = self.hash.rotate_left(5).bitxor(i).wrapping_mul(K); + } +} + +impl Hasher for FxHasher { + #[inline] + fn write(&mut self, bytes: &[u8]) { + for byte in bytes { + let i = *byte; + self.add_to_hash(i as usize); + } + } + + #[inline] + fn write_u8(&mut self, i: u8) { + self.add_to_hash(i as usize); + } + + #[inline] + fn write_u16(&mut self, i: u16) { + self.add_to_hash(i as usize); + } + + #[inline] + fn write_u32(&mut self, i: u32) { + self.add_to_hash(i as usize); + } + + #[cfg(target_pointer_width = "32")] + #[inline] + fn write_u64(&mut self, i: u64) { + self.add_to_hash(i as usize); + self.add_to_hash((i >> 32) as usize); + } + + #[cfg(target_pointer_width = "64")] + #[inline] + fn write_u64(&mut self, i: u64) { + self.add_to_hash(i as usize); + } + + #[inline] + fn write_usize(&mut self, i: usize) { + self.add_to_hash(i); + } + + #[inline] + fn finish(&self) -> u64 { + self.hash as u64 + } +} diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs index 7944c852a0..05fe7c03df 100644 --- a/lib/codegen/src/lib.rs +++ b/lib/codegen/src/lib.rs @@ -95,6 +95,7 @@ mod constant_hash; mod context; mod dce; mod divconst_magic_numbers; +mod fx; mod iterators; mod legalizer; mod licm; diff --git a/lib/codegen/src/licm.rs b/lib/codegen/src/licm.rs index 154b222df2..f52d192000 100644 --- a/lib/codegen/src/licm.rs +++ b/lib/codegen/src/licm.rs @@ -4,9 +4,9 @@ use cursor::{Cursor, FuncCursor}; use dominator_tree::DominatorTree; use entity::{EntityList, ListPool}; use flowgraph::ControlFlowGraph; +use fx::FxHashSet; use ir::{DataFlowGraph, Ebb, Function, Inst, InstBuilder, Layout, Opcode, Type, Value}; use loop_analysis::{Loop, LoopAnalysis}; -use std::collections::HashSet; use std::vec::Vec; use timing; @@ -138,7 +138,7 @@ fn trivially_unsafe_for_licm(opcode: Opcode) -> bool { } /// Test whether the given instruction is loop-invariant. -fn is_loop_invariant(inst: Inst, dfg: &DataFlowGraph, loop_values: &HashSet) -> bool { +fn is_loop_invariant(inst: Inst, dfg: &DataFlowGraph, loop_values: &FxHashSet) -> bool { if trivially_unsafe_for_licm(dfg[inst].opcode()) { return false; } @@ -162,7 +162,7 @@ fn remove_loop_invariant_instructions( cfg: &ControlFlowGraph, loop_analysis: &LoopAnalysis, ) -> Vec { - let mut loop_values: HashSet = HashSet::new(); + let mut loop_values: FxHashSet = FxHashSet(); let mut invariant_insts: Vec = Vec::new(); let mut pos = FuncCursor::new(func); // We traverse the loop EBB in reverse post-order. @@ -194,8 +194,8 @@ fn remove_loop_invariant_instructions( /// Return ebbs from a loop in post-order, starting from an entry point in the block. fn postorder_ebbs_loop(loop_analysis: &LoopAnalysis, cfg: &ControlFlowGraph, lp: Loop) -> Vec { - let mut grey = HashSet::new(); - let mut black = HashSet::new(); + let mut grey = FxHashSet(); + let mut black = FxHashSet(); let mut stack = vec![loop_analysis.loop_header(lp)]; let mut postorder = Vec::new(); diff --git a/lib/codegen/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs index 2a70208c78..5cd0815e5b 100644 --- a/lib/codegen/src/regalloc/coalescing.rs +++ b/lib/codegen/src/regalloc/coalescing.rs @@ -10,6 +10,7 @@ use cursor::{Cursor, EncCursor}; use dbg::DisplayList; use dominator_tree::{DominatorTree, DominatorTreePreorder}; use flowgraph::ControlFlowGraph; +use fx::FxHashMap; use ir::{self, InstBuilder, ProgramOrder}; use ir::{Ebb, ExpandedProgramPoint, Function, Inst, Value}; use isa::{EncInfo, TargetIsa}; @@ -883,11 +884,9 @@ struct VirtualCopies { // Filter for the currently active node iterator. // - // An (ebb, set_id, num) entry means that branches to `ebb` are active in `set_id` with branch + // An ebb => (set_id, num) entry means that branches to `ebb` are active in `set_id` with branch // argument number `num`. - // - // This is ordered by EBB number for fast binary search. - filter: Vec<(Ebb, u8, usize)>, + filter: FxHashMap, } impl VirtualCopies { @@ -896,7 +895,7 @@ impl VirtualCopies { Self { params: Vec::new(), branches: Vec::new(), - filter: Vec::new(), + filter: FxHashMap(), } } @@ -1010,12 +1009,10 @@ impl VirtualCopies { // Stop once we're outside the bounds of `self.params`. break; } - self.filter.push((ebb, set_id, num)); + self.filter.insert(ebb, (set_id, num)); } } } - // We'll be using `binary_search_by` with the numerical EBB ordering. - self.filter.sort_unstable(); } /// Look up the set_id and argument number for `ebb` in the current filter. @@ -1023,13 +1020,7 @@ impl VirtualCopies { /// Returns `None` if none of the currently active parameters are defined at `ebb`. Otherwise /// returns `(set_id, argnum)` for an active parameter defined at `ebb`. fn lookup(&self, ebb: Ebb) -> Option<(u8, usize)> { - self.filter - .binary_search_by(|&(e, _, _)| e.cmp(&ebb)) - .ok() - .map(|i| { - let t = self.filter[i]; - (t.1, t.2) - }) + self.filter.get(&ebb).map(|t| *t) } /// Get an iterator of dom-forest nodes corresponding to the current filter. diff --git a/lib/codegen/src/regalloc/live_value_tracker.rs b/lib/codegen/src/regalloc/live_value_tracker.rs index 27a76deb5c..12bda20ff4 100644 --- a/lib/codegen/src/regalloc/live_value_tracker.rs +++ b/lib/codegen/src/regalloc/live_value_tracker.rs @@ -6,12 +6,12 @@ use dominator_tree::DominatorTree; use entity::{EntityList, ListPool}; +use fx::FxHashMap; use ir::{DataFlowGraph, Ebb, ExpandedProgramPoint, Inst, Layout, Value}; use partition_slice::partition_slice; use regalloc::affinity::Affinity; use regalloc::liveness::Liveness; use regalloc::liverange::LiveRange; -use std::collections::HashMap; use std::vec::Vec; type ValueList = EntityList; @@ -25,7 +25,7 @@ pub struct LiveValueTracker { /// dominator of an EBB. /// /// This is the set of values that are live *before* the branch. - idom_sets: HashMap, + idom_sets: FxHashMap, /// Memory pool for the live sets. idom_pool: ListPool, @@ -128,7 +128,7 @@ impl LiveValueTracker { pub fn new() -> Self { Self { live: LiveValueVec::new(), - idom_sets: HashMap::new(), + idom_sets: FxHashMap(), idom_pool: ListPool::new(), } } diff --git a/lib/codegen/src/scoped_hash_map.rs b/lib/codegen/src/scoped_hash_map.rs index 4acc7087f0..f7fd992ac0 100644 --- a/lib/codegen/src/scoped_hash_map.rs +++ b/lib/codegen/src/scoped_hash_map.rs @@ -4,7 +4,8 @@ //! container that has a concept of scopes that can be entered and exited, such that //! values inserted while inside a scope aren't visible outside the scope. -use std::collections::{hash_map, HashMap}; +use fx::FxHashMap; +use std::collections::hash_map; use std::hash::Hash; use std::mem; @@ -58,7 +59,7 @@ pub enum Entry<'a, K: 'a, V: 'a> { /// Shadowing, where one scope has entries with the same keys as a containing scope, /// is not supported in this implementation. pub struct ScopedHashMap { - map: HashMap>, + map: FxHashMap>, last_insert: Option, current_depth: usize, } @@ -70,7 +71,7 @@ where /// Creates an empty `ScopedHashMap`. pub fn new() -> Self { Self { - map: HashMap::new(), + map: FxHashMap(), last_insert: None, current_depth: 0, } From 923ea8ada9f68d9e9910ccf9649b14c9b06df67e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 17 May 2018 14:19:42 -0700 Subject: [PATCH 1785/3084] Add assertions to check that postopt encoding succeeds. --- lib/codegen/src/postopt.rs | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/lib/codegen/src/postopt.rs b/lib/codegen/src/postopt.rs index 6de918d08d..50661f4cc1 100644 --- a/lib/codegen/src/postopt.rs +++ b/lib/codegen/src/postopt.rs @@ -169,8 +169,10 @@ fn optimize_cpu_flags( ); } } - pos.func.update_encoding(info.cmp_inst, isa).is_ok(); - pos.func.update_encoding(info.br_inst, isa).is_ok(); + let ok = pos.func.update_encoding(info.cmp_inst, isa).is_ok(); + debug_assert!(ok); + let ok = pos.func.update_encoding(info.br_inst, isa).is_ok(); + debug_assert!(ok); } @@ -320,7 +322,8 @@ fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa) } _ => return, } - pos.func.update_encoding(info.inst, isa).is_ok(); + let ok = pos.func.update_encoding(info.inst, isa).is_ok(); + debug_assert!(ok); } From 89e7d561206f45aca14234f32f94ad672d156715 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 21 May 2018 20:49:19 -0700 Subject: [PATCH 1786/3084] Allow wasm embedders to reject wasm modules with unsupported features. (#345) Define `WasmError` (and an accompanying `WasmResult`) to represent errors translating WebAssembly functions. Make `translate_call` and related functions return `WasmResult`s so that embedders have the flexibility to reject features they don't support. Move `InvalidInput` out of `CtonError` and into `WasmError`, where it's now named `InvalidWebAssembly`, as it's a WebAssembly-specific error condition. Also extend it to preserve the original error message and bytecode offset. --- cranelift/src/wasm.rs | 4 +- lib/codegen/src/result.rs | 7 -- lib/wasm/Cargo.toml | 2 + lib/wasm/src/code_translator.rs | 18 ++--- lib/wasm/src/environ/dummy.rs | 34 +++++---- lib/wasm/src/environ/mod.rs | 2 +- lib/wasm/src/environ/spec.rs | 57 +++++++++++++-- lib/wasm/src/func_translator.rs | 27 +++---- lib/wasm/src/lib.rs | 4 ++ lib/wasm/src/module_translator.rs | 99 ++++++------------------- lib/wasm/src/sections_translator.rs | 108 ++++++++++++++++------------ 11 files changed, 185 insertions(+), 177 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 5ef8e21f2e..4f44afa6f6 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -105,7 +105,9 @@ fn handle_module( } let mut dummy_environ = DummyEnvironment::with_flags(fisa.flags.clone()); - translate_module(&data, &mut dummy_environ)?; + translate_module(&data, &mut dummy_environ).map_err( + |e| e.to_string(), + )?; terminal.fg(term::color::GREEN).unwrap(); vprintln!(flag_verbose, "ok"); diff --git a/lib/codegen/src/result.rs b/lib/codegen/src/result.rs index 47047e2f4e..9d347e351a 100644 --- a/lib/codegen/src/result.rs +++ b/lib/codegen/src/result.rs @@ -7,13 +7,6 @@ use verifier; /// When Cretonne fails to compile a function, it will return one of these error codes. #[derive(Fail, Debug, PartialEq, Eq)] pub enum CtonError { - /// The input is invalid. - /// - /// This error code is used by a WebAssembly translator when it encounters invalid WebAssembly - /// code. This should never happen for validated WebAssembly code. - #[fail(display = "Invalid input code")] - InvalidInput, - /// An IR verifier error. /// /// This always represents a bug, either in the code that generated IR for Cretonne, or a bug diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 62cfa3f872..f1ad6b726c 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -13,6 +13,8 @@ wasmparser = { version = "0.16.1", default-features = false } cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } cretonne-frontend = { path = "../frontend", version = "0.8.0", default-features = false } hashmap_core = { version = "0.1.4", optional = true } +failure = { version = "0.1.1", default-features = false, features = ["derive"] } +failure_derive = { version = "0.1.1", default-features = false } [dev-dependencies] tempdir = "0.3.5" diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index aea622cc34..dbe23c4d15 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -27,7 +27,7 @@ use cretonne_codegen::ir::types::*; use cretonne_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags}; use cretonne_codegen::packed_option::ReservedValue; use cretonne_frontend::{FunctionBuilder, Variable}; -use environ::{FuncEnvironment, GlobalValue}; +use environ::{FuncEnvironment, GlobalValue, WasmResult}; use state::{ControlStackFrame, TranslationState}; use std::collections::{hash_map, HashMap}; use std::vec::Vec; @@ -45,13 +45,13 @@ pub fn translate_operator( builder: &mut FunctionBuilder, state: &mut TranslationState, environ: &mut FE, -) { +) -> WasmResult<()> { if !state.reachable { - return translate_unreachable_operator(&op, builder, state); + return Ok(translate_unreachable_operator(&op, builder, state)); } // This big match treats all Wasm code operators. - match op { + Ok(match op { /********************************** Locals **************************************** * `get_local` and `set_local` are treated as non-SSA variables and will completely * disappear in the Cretonne Code @@ -362,7 +362,7 @@ pub fn translate_operator( function_index as FunctionIndex, fref, state.peekn(num_args), - ); + )?; state.popn(num_args); state.pushn(builder.inst_results(call)); } @@ -378,7 +378,7 @@ pub fn translate_operator( sigref, callee, state.peekn(num_args), - ); + )?; state.popn(num_args); state.pushn(builder.inst_results(call)); } @@ -397,7 +397,7 @@ pub fn translate_operator( heap_index, heap, val, - )) + )?) } Operator::CurrentMemory { reserved } => { let heap_index = reserved as MemoryIndex; @@ -406,7 +406,7 @@ pub fn translate_operator( builder.cursor(), heap_index, heap, - )); + )?); } /******************************* Load instructions *********************************** * Wasm specifies an integer alignment flag but we drop it in Cretonne. @@ -857,7 +857,7 @@ pub fn translate_operator( Operator::I64AtomicRmw32UCmpxchg { .. } => { panic!("proposed thread operators not yet supported"); } - } + }) } // Clippy warns us of some fields we are deliberately ignoring diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 68927101cf..90f1af3b5a 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -4,7 +4,7 @@ use cretonne_codegen::cursor::FuncCursor; use cretonne_codegen::ir::types::*; use cretonne_codegen::ir::{self, InstBuilder}; use cretonne_codegen::settings; -use environ::{FuncEnvironment, GlobalValue, ModuleEnvironment}; +use environ::{FuncEnvironment, GlobalValue, ModuleEnvironment, WasmResult}; use func_translator::FuncTranslator; use std::string::String; use std::vec::Vec; @@ -196,7 +196,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ sig_ref: ir::SigRef, callee: ir::Value, call_args: &[ir::Value], - ) -> ir::Inst { + ) -> WasmResult { // Pass the current function's vmctx parameter on to the callee. let vmctx = pos.func .special_param(ir::ArgumentPurpose::VMContext) @@ -224,9 +224,11 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ args.extend(call_args.iter().cloned(), &mut pos.func.dfg.value_lists); args.push(vmctx, &mut pos.func.dfg.value_lists); - pos.ins() - .CallIndirect(ir::Opcode::CallIndirect, VOID, sig_ref, args) - .0 + Ok( + pos.ins() + .CallIndirect(ir::Opcode::CallIndirect, VOID, sig_ref, args) + .0, + ) } fn translate_call( @@ -235,7 +237,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ _callee_index: FunctionIndex, callee: ir::FuncRef, call_args: &[ir::Value], - ) -> ir::Inst { + ) -> WasmResult { // Pass the current function's vmctx parameter on to the callee. let vmctx = pos.func .special_param(ir::ArgumentPurpose::VMContext) @@ -247,7 +249,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ args.extend(call_args.iter().cloned(), &mut pos.func.dfg.value_lists); args.push(vmctx, &mut pos.func.dfg.value_lists); - pos.ins().Call(ir::Opcode::Call, VOID, callee, args).0 + Ok(pos.ins().Call(ir::Opcode::Call, VOID, callee, args).0) } fn translate_grow_memory( @@ -256,8 +258,8 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ _index: MemoryIndex, _heap: ir::Heap, _val: ir::Value, - ) -> ir::Value { - pos.ins().iconst(I32, -1) + ) -> WasmResult { + Ok(pos.ins().iconst(I32, -1)) } fn translate_current_memory( @@ -265,8 +267,8 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ mut pos: FuncCursor, _index: MemoryIndex, _heap: ir::Heap, - ) -> ir::Value { - pos.ins().iconst(I32, -1) + ) -> WasmResult { + Ok(pos.ins().iconst(I32, -1)) } } @@ -385,7 +387,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment { self.info.start_func = Some(func_index); } - fn define_function_body(&mut self, body_bytes: &'data [u8]) -> Result<(), String> { + fn define_function_body(&mut self, body_bytes: &'data [u8]) -> WasmResult<()> { let func = { let mut func_environ = DummyFuncEnvironment::new(&self.info); let function_index = self.get_num_func_imports() + self.info.function_bodies.len(); @@ -393,9 +395,11 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment { let sig = func_environ.vmctx_sig(self.get_func_type(function_index)); let mut func = ir::Function::with_name_signature(name, sig); let reader = wasmparser::BinaryReader::new(body_bytes); - self.trans - .translate_from_reader(reader, &mut func, &mut func_environ) - .map_err(|e| format!("{}", e))?; + self.trans.translate_from_reader( + reader, + &mut func, + &mut func_environ, + )?; func }; self.func_bytecode_sizes.push(body_bytes.len()); diff --git a/lib/wasm/src/environ/mod.rs b/lib/wasm/src/environ/mod.rs index e89995d8f8..923d65be11 100644 --- a/lib/wasm/src/environ/mod.rs +++ b/lib/wasm/src/environ/mod.rs @@ -4,4 +4,4 @@ mod dummy; mod spec; pub use environ::dummy::DummyEnvironment; -pub use environ::spec::{FuncEnvironment, GlobalValue, ModuleEnvironment}; +pub use environ::spec::{FuncEnvironment, GlobalValue, ModuleEnvironment, WasmError, WasmResult}; diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index cc9e4eb771..6093203ff2 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -3,10 +3,10 @@ use cretonne_codegen::cursor::FuncCursor; use cretonne_codegen::ir::{self, InstBuilder}; use cretonne_codegen::settings::Flags; -use std::string::String; use std::vec::Vec; use translation_utils::{FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex}; +use wasmparser::BinaryReaderError; /// The value of a WebAssembly global variable. #[derive(Clone, Copy)] @@ -23,6 +23,49 @@ pub enum GlobalValue { }, } +/// A WebAssembly translation error. +/// +/// When a WebAssembly function can't be translated, one of these error codes will be returned +/// to describe the failure. +#[derive(Fail, Debug, PartialEq, Eq)] +pub enum WasmError { + /// The input WebAssembly code is invalid. + /// + /// This error code is used by a WebAssembly translator when it encounters invalid WebAssembly + /// code. This should never happen for validated WebAssembly code. + #[fail(display = "Invalid input WebAssembly code at offset {}: {}", _1, _0)] + InvalidWebAssembly { + message: &'static str, + offset: usize, + }, + + /// A feature used by the WebAssembly code is not supported by the embedding environment. + /// + /// Embedding environments may have their own limitations and feature restrictions. + #[fail(display = "Unsupported feature: {}", _0)] + Unsupported(&'static str), + + /// An implementation limit was exceeded. + /// + /// Cretonne can compile very large and complicated functions, but the [implementation has + /// limits][limits] that cause compilation to fail when they are exceeded. + /// + /// [limits]: https://cretonne.readthedocs.io/en/latest/langref.html#implementation-limits + #[fail(display = "Implementation limit exceeded")] + ImplLimitExceeded, +} + +impl WasmError { + /// Convert from a `BinaryReaderError` to a `WasmError`. + pub fn from_binary_reader_error(e: BinaryReaderError) -> Self { + let BinaryReaderError { message, offset } = e; + WasmError::InvalidWebAssembly { message, offset } + } +} + +/// A convenient alias for a `Result` that uses `WasmError` as the error type. +pub type WasmResult = Result; + /// Environment affecting the translation of a single WebAssembly function. /// /// A `FuncEnvironment` trait object is required to translate a WebAssembly function to Cretonne @@ -99,7 +142,7 @@ pub trait FuncEnvironment { sig_ref: ir::SigRef, callee: ir::Value, call_args: &[ir::Value], - ) -> ir::Inst; + ) -> WasmResult; /// Translate a `call` WebAssembly instruction at `pos`. /// @@ -114,8 +157,8 @@ pub trait FuncEnvironment { _callee_index: FunctionIndex, callee: ir::FuncRef, call_args: &[ir::Value], - ) -> ir::Inst { - pos.ins().call(callee, call_args) + ) -> WasmResult { + Ok(pos.ins().call(callee, call_args)) } /// Translate a `grow_memory` WebAssembly instruction. @@ -132,7 +175,7 @@ pub trait FuncEnvironment { index: MemoryIndex, heap: ir::Heap, val: ir::Value, - ) -> ir::Value; + ) -> WasmResult; /// Translates a `current_memory` WebAssembly instruction. /// @@ -145,7 +188,7 @@ pub trait FuncEnvironment { pos: FuncCursor, index: MemoryIndex, heap: ir::Heap, - ) -> ir::Value; + ) -> WasmResult; /// Emit code at the beginning of every wasm loop. /// @@ -229,5 +272,5 @@ pub trait ModuleEnvironment<'data> { fn declare_start_func(&mut self, index: FunctionIndex); /// Provides the contents of a function body. - fn define_function_body(&mut self, body_bytes: &'data [u8]) -> Result<(), String>; + fn define_function_body(&mut self, body_bytes: &'data [u8]) -> WasmResult<()>; } diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 4801cccf11..ed693af4bf 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -7,10 +7,9 @@ use code_translator::translate_operator; use cretonne_codegen::entity::EntityRef; use cretonne_codegen::ir::{self, Ebb, InstBuilder}; -use cretonne_codegen::result::{CtonError, CtonResult}; use cretonne_codegen::timing; use cretonne_frontend::{FunctionBuilder, FunctionBuilderContext, Variable}; -use environ::FuncEnvironment; +use environ::{FuncEnvironment, WasmError, WasmResult}; use state::TranslationState; use wasmparser::{self, BinaryReader}; @@ -56,7 +55,7 @@ impl FuncTranslator { code: &[u8], func: &mut ir::Function, environ: &mut FE, - ) -> CtonResult { + ) -> WasmResult<()> { self.translate_from_reader(BinaryReader::new(code), func, environ) } @@ -66,7 +65,7 @@ impl FuncTranslator { mut reader: BinaryReader, func: &mut ir::Function, environ: &mut FE, - ) -> CtonResult { + ) -> WasmResult<()> { let _tt = timing::wasm_translate_function(); dbg!( "translate({} bytes, {}{})", @@ -134,17 +133,17 @@ fn parse_local_decls( reader: &mut BinaryReader, builder: &mut FunctionBuilder, num_params: usize, -) -> CtonResult { +) -> WasmResult<()> { let mut next_local = num_params; - let local_count = reader.read_local_count().map_err( - |_| CtonError::InvalidInput, - )?; + let local_count = reader.read_local_count().map_err(|e| { + WasmError::from_binary_reader_error(e) + })?; let mut locals_total = 0; for _ in 0..local_count { builder.set_srcloc(cur_srcloc(reader)); - let (count, ty) = reader.read_local_decl(&mut locals_total).map_err(|_| { - CtonError::InvalidInput + let (count, ty) = reader.read_local_decl(&mut locals_total).map_err(|e| { + WasmError::from_binary_reader_error(e) })?; declare_locals(builder, count, ty, &mut next_local); } @@ -189,15 +188,17 @@ fn parse_function_body( builder: &mut FunctionBuilder, state: &mut TranslationState, environ: &mut FE, -) -> CtonResult { +) -> WasmResult<()> { // The control stack is initialized with a single block representing the whole function. debug_assert_eq!(state.control_stack.len(), 1, "State not initialized"); // Keep going until the final `End` operator which pops the outermost block. while !state.control_stack.is_empty() { builder.set_srcloc(cur_srcloc(&reader)); - let op = reader.read_operator().map_err(|_| CtonError::InvalidInput)?; - translate_operator(op, builder, state, environ); + let op = reader.read_operator().map_err(|e| { + WasmError::from_binary_reader_error(e) + })?; + translate_operator(op, builder, state, environ)?; } // The final `End` operator left us in the exit block where we need to manually add a return diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index e20e939ca8..d4caf9177c 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -42,6 +42,10 @@ extern crate cretonne_codegen; extern crate cretonne_frontend; extern crate wasmparser; +extern crate failure; +#[macro_use] +extern crate failure_derive; + mod code_translator; mod environ; mod func_translator; diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index f1092651d6..683517e7f5 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -1,14 +1,12 @@ //! Translation skeleton that traverses the whole WebAssembly module and call helper functions //! to deal with each part of it. use cretonne_codegen::timing; -use environ::ModuleEnvironment; +use environ::{ModuleEnvironment, WasmError, WasmResult}; use sections_translator::{parse_data_section, parse_elements_section, parse_export_section, parse_function_section, parse_function_signatures, parse_global_section, parse_import_section, parse_memory_section, parse_start_section, - parse_table_section, SectionParsingError}; -use wasmparser::{BinaryReaderError, Parser, ParserInput, ParserState, SectionCode, WasmDecoder}; - -use std::string::String; + parse_table_section}; +use wasmparser::{Parser, ParserInput, ParserState, SectionCode, WasmDecoder}; /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IR /// [`Function`](../codegen/ir/function/struct.Function.html). @@ -17,13 +15,13 @@ use std::string::String; pub fn translate_module<'data>( data: &'data [u8], environ: &mut ModuleEnvironment<'data>, -) -> Result<(), String> { +) -> WasmResult<()> { let _tt = timing::wasm_translate_module(); let mut parser = Parser::new(data); match *parser.read() { ParserState::BeginWasm { .. } => {} - ParserState::Error(BinaryReaderError { message, offset }) => { - return Err(format!("at offset {}: {}", offset, message)); + ParserState::Error(e) => { + return Err(WasmError::from_binary_reader_error(e)); } ref s => panic!("modules should begin properly: {:?}", s), } @@ -31,83 +29,38 @@ pub fn translate_module<'data>( loop { match *parser.read_with_input(next_input) { ParserState::BeginSection { code: SectionCode::Type, .. } => { - match parse_function_signatures(&mut parser, environ) { - Ok(()) => (), - Err(SectionParsingError::WrongSectionContent(s)) => { - return Err(format!("wrong content in the type section: {}", s)) - } - }; + parse_function_signatures(&mut parser, environ)?; next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Import, .. } => { - match parse_import_section(&mut parser, environ) { - Ok(()) => {} - Err(SectionParsingError::WrongSectionContent(s)) => { - return Err(format!("wrong content in the import section: {}", s)) - } - } + parse_import_section(&mut parser, environ)?; next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Function, .. } => { - match parse_function_section(&mut parser, environ) { - Ok(()) => {} - Err(SectionParsingError::WrongSectionContent(s)) => { - return Err(format!("wrong content in the function section: {}", s)) - } - } + parse_function_section(&mut parser, environ)?; next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Table, .. } => { - match parse_table_section(&mut parser, environ) { - Ok(()) => (), - Err(SectionParsingError::WrongSectionContent(s)) => { - return Err(format!("wrong content in the table section: {}", s)) - } - } + parse_table_section(&mut parser, environ)?; } ParserState::BeginSection { code: SectionCode::Memory, .. } => { - match parse_memory_section(&mut parser, environ) { - Ok(()) => {} - Err(SectionParsingError::WrongSectionContent(s)) => { - return Err(format!("wrong content in the memory section: {}", s)) - } - } + parse_memory_section(&mut parser, environ)?; next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Global, .. } => { - match parse_global_section(&mut parser, environ) { - Ok(()) => {} - Err(SectionParsingError::WrongSectionContent(s)) => { - return Err(format!("wrong content in the global section: {}", s)) - } - } + parse_global_section(&mut parser, environ)?; next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Export, .. } => { - match parse_export_section(&mut parser, environ) { - Ok(()) => {} - Err(SectionParsingError::WrongSectionContent(s)) => { - return Err(format!("wrong content in the export section: {}", s)) - } - } + parse_export_section(&mut parser, environ)?; next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Start, .. } => { - match parse_start_section(&mut parser, environ) { - Ok(()) => (), - Err(SectionParsingError::WrongSectionContent(s)) => { - return Err(format!("wrong content in the start section: {}", s)) - } - } + parse_start_section(&mut parser, environ)?; next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Element, .. } => { - match parse_elements_section(&mut parser, environ) { - Ok(()) => (), - Err(SectionParsingError::WrongSectionContent(s)) => { - return Err(format!("wrong content in the element section: {}", s)) - } - } + parse_elements_section(&mut parser, environ)?; next_input = ParserInput::Default; } ParserState::BeginSection { code: SectionCode::Code, .. } => { @@ -119,18 +72,14 @@ pub fn translate_module<'data>( } ParserState::EndWasm => return Ok(()), ParserState::BeginSection { code: SectionCode::Data, .. } => { - match parse_data_section(&mut parser, environ) { - Ok(()) => (), - Err(SectionParsingError::WrongSectionContent(s)) => { - return Err(format!("wrong content in the data section: {}", s)) - } - } + parse_data_section(&mut parser, environ)?; } ParserState::BeginSection { code: SectionCode::Custom { .. }, .. } => { // Ignore unknown custom sections. next_input = ParserInput::SkipSection; } - _ => return Err(String::from("wrong content in the preamble")), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + _ => panic!("wrong content in the preamble"), }; } // At this point we've entered the code section @@ -138,25 +87,21 @@ pub fn translate_module<'data>( match *parser.read() { ParserState::BeginFunctionBody { .. } => {} ParserState::EndSection => break, - _ => return Err(String::from("wrong content in code section")), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("wrong content in code section: {:?}", s), } let mut reader = parser.create_binary_reader(); let size = reader.bytes_remaining(); environ.define_function_body( reader.read_bytes(size).map_err(|e| { - format!("at offset {}: {}", e.offset, e.message) + WasmError::from_binary_reader_error(e) })?, )?; } loop { match *parser.read() { ParserState::BeginSection { code: SectionCode::Data, .. } => { - match parse_data_section(&mut parser, environ) { - Ok(()) => (), - Err(SectionParsingError::WrongSectionContent(s)) => { - return Err(format!("wrong content in the data section: {}", s)) - } - } + parse_data_section(&mut parser, environ)?; } ParserState::EndWasm => break, _ => (), diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index a9ca07a8df..222433fafb 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -8,9 +8,8 @@ //! is handled, according to the semantics of WebAssembly, to only specific expressions that are //! interpreted on the fly. use cretonne_codegen::ir::{self, AbiParam, Signature}; -use environ::ModuleEnvironment; +use environ::{ModuleEnvironment, WasmError, WasmResult}; use std::str::from_utf8; -use std::string::String; use std::vec::Vec; use translation_utils::{type_to_type, FunctionIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex, Table, TableElementType, TableIndex}; @@ -18,15 +17,11 @@ use wasmparser; use wasmparser::{ExternalKind, FuncType, ImportSectionEntryType, MemoryType, Operator, Parser, ParserState, WasmDecoder}; -pub enum SectionParsingError { - WrongSectionContent(String), -} - /// Reads the Type Section of the wasm module and returns the corresponding function signatures. pub fn parse_function_signatures( parser: &mut Parser, environ: &mut ModuleEnvironment, -) -> Result<(), SectionParsingError> { +) -> WasmResult<()> { loop { match *parser.read() { ParserState::EndSection => break, @@ -50,7 +45,8 @@ pub fn parse_function_signatures( })); environ.declare_signature(&sig); } - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), } } Ok(()) @@ -60,7 +56,7 @@ pub fn parse_function_signatures( pub fn parse_import_section<'data>( parser: &mut Parser<'data>, environ: &mut ModuleEnvironment<'data>, -) -> Result<(), SectionParsingError> { +) -> WasmResult<()> { loop { match *parser.read() { ParserState::ImportSectionEntry { @@ -110,7 +106,8 @@ pub fn parse_import_section<'data>( }) } ParserState::EndSection => break, - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; } Ok(()) @@ -120,14 +117,15 @@ pub fn parse_import_section<'data>( pub fn parse_function_section( parser: &mut Parser, environ: &mut ModuleEnvironment, -) -> Result<(), SectionParsingError> { +) -> WasmResult<()> { loop { match *parser.read() { ParserState::FunctionSectionEntry(sigindex) => { environ.declare_func_type(sigindex as SignatureIndex); } ParserState::EndSection => break, - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; } Ok(()) @@ -137,7 +135,7 @@ pub fn parse_function_section( pub fn parse_export_section<'data>( parser: &mut Parser<'data>, environ: &mut ModuleEnvironment<'data>, -) -> Result<(), SectionParsingError> { +) -> WasmResult<()> { loop { match *parser.read() { ParserState::ExportSectionEntry { @@ -158,24 +156,23 @@ pub fn parse_export_section<'data>( } } ParserState::EndSection => break, - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; } Ok(()) } /// Retrieves the start function index from the start section -pub fn parse_start_section( - parser: &mut Parser, - environ: &mut ModuleEnvironment, -) -> Result<(), SectionParsingError> { +pub fn parse_start_section(parser: &mut Parser, environ: &mut ModuleEnvironment) -> WasmResult<()> { loop { match *parser.read() { ParserState::StartSectionEntry(index) => { environ.declare_start_func(index as FunctionIndex); } ParserState::EndSection => break, - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; } Ok(()) @@ -185,7 +182,7 @@ pub fn parse_start_section( pub fn parse_memory_section( parser: &mut Parser, environ: &mut ModuleEnvironment, -) -> Result<(), SectionParsingError> { +) -> WasmResult<()> { loop { match *parser.read() { ParserState::MemorySectionEntry(ref ty) => { @@ -196,7 +193,8 @@ pub fn parse_memory_section( }); } ParserState::EndSection => break, - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; } Ok(()) @@ -206,16 +204,18 @@ pub fn parse_memory_section( pub fn parse_global_section( parser: &mut Parser, environ: &mut ModuleEnvironment, -) -> Result<(), SectionParsingError> { +) -> WasmResult<()> { loop { let (content_type, mutability) = match *parser.read() { ParserState::BeginGlobalSectionEntry(ref ty) => (ty.content_type, ty.mutable), ParserState::EndSection => break, - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; match *parser.read() { ParserState::BeginInitExpressionBody => (), - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), } let initializer = match *parser.read() { ParserState::InitExpressionOperator(Operator::I32Const { value }) => { @@ -233,11 +233,13 @@ pub fn parse_global_section( ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => { GlobalInit::GlobalRef(global_index as GlobalIndex) } - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; match *parser.read() { ParserState::EndInitExpressionBody => (), - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), } let global = Global { ty: type_to_type(&content_type).unwrap(), @@ -247,7 +249,8 @@ pub fn parse_global_section( environ.declare_global(global); match *parser.read() { ParserState::EndGlobalSectionEntry => (), - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), } } Ok(()) @@ -256,16 +259,18 @@ pub fn parse_global_section( pub fn parse_data_section<'data>( parser: &mut Parser<'data>, environ: &mut ModuleEnvironment<'data>, -) -> Result<(), SectionParsingError> { +) -> WasmResult<()> { loop { let memory_index = match *parser.read() { ParserState::BeginDataSectionEntry(memory_index) => memory_index, ParserState::EndSection => break, - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; match *parser.read() { ParserState::BeginInitExpressionBody => (), - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; let (base, offset) = match *parser.read() { ParserState::InitExpressionOperator(Operator::I32Const { value }) => { @@ -278,22 +283,26 @@ pub fn parse_data_section<'data>( _ => panic!("should not happen"), } } - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; match *parser.read() { ParserState::EndInitExpressionBody => (), - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; match *parser.read() { ParserState::BeginDataSectionEntryBody(_) => (), - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; let mut running_offset = offset; loop { let data = match *parser.read() { ParserState::DataSectionEntryBodyChunk(data) => data, ParserState::EndDataSectionEntryBody => break, - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; environ.declare_data_initialization( memory_index as MemoryIndex, @@ -305,17 +314,15 @@ pub fn parse_data_section<'data>( } match *parser.read() { ParserState::EndDataSectionEntry => (), - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; } Ok(()) } /// Retrieves the tables from the table section -pub fn parse_table_section( - parser: &mut Parser, - environ: &mut ModuleEnvironment, -) -> Result<(), SectionParsingError> { +pub fn parse_table_section(parser: &mut Parser, environ: &mut ModuleEnvironment) -> WasmResult<()> { loop { match *parser.read() { ParserState::TableSectionEntry(ref table) => { @@ -329,7 +336,8 @@ pub fn parse_table_section( }) } ParserState::EndSection => break, - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; } Ok(()) @@ -339,16 +347,18 @@ pub fn parse_table_section( pub fn parse_elements_section( parser: &mut Parser, environ: &mut ModuleEnvironment, -) -> Result<(), SectionParsingError> { +) -> WasmResult<()> { loop { let table_index = match *parser.read() { ParserState::BeginElementSectionEntry(table_index) => table_index as TableIndex, ParserState::EndSection => break, - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; match *parser.read() { ParserState::BeginInitExpressionBody => (), - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; let (base, offset) = match *parser.read() { ParserState::InitExpressionOperator(Operator::I32Const { value }) => { @@ -361,11 +371,13 @@ pub fn parse_elements_section( _ => panic!("should not happen"), } } - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; match *parser.read() { ParserState::EndInitExpressionBody => (), - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; match *parser.read() { ParserState::ElementSectionEntryBody(ref elements) => { @@ -373,11 +385,13 @@ pub fn parse_elements_section( elements.iter().map(|&x| x as FunctionIndex).collect(); environ.declare_table_elements(table_index, base, offset, elems) } - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; match *parser.read() { ParserState::EndElementSectionEntry => (), - ref s => return Err(SectionParsingError::WrongSectionContent(format!("{:?}", s))), + ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), + ref s => panic!("unexpected section content: {:?}", s), }; } Ok(()) From f835a1b6626c1f077e9306eee8df25e31cb001f3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 22 May 2018 07:23:28 -0700 Subject: [PATCH 1787/3084] Followup to #345 to fix no_std errors. --- lib/wasm/src/lib.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index d4caf9177c..6fe32541e2 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -30,7 +30,6 @@ #![cfg_attr(not(feature = "std"), feature(alloc))] #[cfg(not(feature = "std"))] -#[macro_use] extern crate alloc; #[allow(unused_extern_crates)] @@ -65,6 +64,8 @@ mod std { pub use alloc::vec; pub use alloc::string; pub use core::{u32, i32, str, cmp}; + pub use core::fmt; + pub use core::option; pub mod collections { pub use hashmap_core::{HashMap, map as hash_map}; } From b855184ae18e59107455aea11710c0aa74637d71 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 22 May 2018 09:37:33 -0700 Subject: [PATCH 1788/3084] Make WasmError and WasmResult public types. --- lib/wasm/src/environ/spec.rs | 2 ++ lib/wasm/src/lib.rs | 3 ++- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 6093203ff2..5b26ca785f 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -35,7 +35,9 @@ pub enum WasmError { /// code. This should never happen for validated WebAssembly code. #[fail(display = "Invalid input WebAssembly code at offset {}: {}", _1, _0)] InvalidWebAssembly { + /// A string describing the validation error. message: &'static str, + /// The bytecode offset where the error occurred. offset: usize, }, diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 6fe32541e2..87d1d87ee2 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -53,7 +53,8 @@ mod sections_translator; mod state; mod translation_utils; -pub use environ::{DummyEnvironment, FuncEnvironment, GlobalValue, ModuleEnvironment}; +pub use environ::{DummyEnvironment, FuncEnvironment, GlobalValue, ModuleEnvironment, WasmError, + WasmResult}; pub use func_translator::FuncTranslator; pub use module_translator::translate_module; pub use translation_utils::{FunctionIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, From 4afb28ef598f996fe1afb33fd7b2daf25562b7fb Mon Sep 17 00:00:00 2001 From: Ram Date: Fri, 25 May 2018 01:23:00 +1000 Subject: [PATCH 1789/3084] Use wabt, and make the wasm subcommand optional. (#347) * Use wabt for wasm testing. * Use wabt in cton-util. * Make the wasm subcommand optional. --- README.rst | 5 ---- cranelift/Cargo.toml | 9 ++++-- cranelift/src/cton-util.rs | 31 ++++++++++++++------ cranelift/src/wasm.rs | 26 ++++------------- lib/wasm/Cargo.toml | 2 +- lib/wasm/tests/wasm_testsuite.rs | 49 +++++++------------------------- 6 files changed, 48 insertions(+), 74 deletions(-) diff --git a/README.rst b/README.rst index 0e30d5aa13..24a601cba4 100644 --- a/README.rst +++ b/README.rst @@ -85,11 +85,6 @@ You can then run tests with:: ./test-all.sh -You may need to install the *wat2wasm* tool from the `wabt -`_ project in order to run all of the -WebAssembly tests. Tests requiring wat2wasm are ignored if the tool is not -installed. - Building with `no_std` ---------------------- diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index b807c2fe01..993c96166d 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -13,10 +13,11 @@ name = "cton-util" path = "src/cton-util.rs" [dependencies] +cfg-if = "0.1" cretonne-codegen = { path = "lib/codegen", version = "0.8.0" } cretonne-reader = { path = "lib/reader", version = "0.8.0" } cretonne-frontend = { path = "lib/frontend", version = "0.8.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.8.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.8.0", optional = true } cretonne-native = { path = "lib/native", version = "0.8.0" } cretonne-filetests = { path = "lib/filetests", version = "0.8.0" } cretonne-module = { path = "lib/module", version = "0.8.0" } @@ -27,9 +28,13 @@ filecheck = "0.3.0" docopt = "1" serde = "1.0.8" serde_derive = "1.0.8" -tempdir = "0.3.5" term = "0.5.1" capstone = "0.3.1" +wabt = { version = "0.3", optional = true } + +[features] +default = ["wasm"] +wasm = ["wabt", "cretonne-wasm"] [workspace] diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 0010034a0a..1ae6991dfb 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -1,5 +1,5 @@ -#![deny(trivial_numeric_casts, unused_extern_crates)] -#![warn(unused_import_braces, unstable_features)] +#![deny(trivial_numeric_casts)] +#![warn(unused_import_braces, unstable_features, unused_extern_crates)] #![cfg_attr(feature="cargo-clippy", warn( float_arithmetic, mut_mut, @@ -10,17 +10,25 @@ use_self, ))] +#[macro_use] +extern crate cfg_if; extern crate cretonne_codegen; extern crate cretonne_filetests; extern crate cretonne_reader; -extern crate cretonne_wasm; extern crate docopt; extern crate filecheck; #[macro_use] extern crate serde_derive; -extern crate tempdir; -extern crate term; extern crate capstone; +extern crate term; + +cfg_if! { + if #[cfg(feature = "wasm")] { + extern crate cretonne_wasm; + extern crate wabt; + mod wasm; + } +} use cretonne_codegen::{timing, VERSION}; use docopt::Docopt; @@ -32,7 +40,6 @@ mod compile; mod print_cfg; mod rsfilecheck; mod utils; -mod wasm; const USAGE: &str = " Cretonne code generator utility @@ -114,7 +121,8 @@ fn cton_util() -> CommandResult { &args.flag_isa, ) } else if args.cmd_wasm { - wasm::run( + #[cfg(feature = "wasm")] + let result = wasm::run( args.arg_file, args.flag_verbose, args.flag_just_decode, @@ -123,7 +131,14 @@ fn cton_util() -> CommandResult { &args.flag_set, &args.flag_isa, args.flag_print_size, - ) + ); + + #[cfg(not(feature = "wasm"))] + let result = Err( + "Error: cton-util was compiled without wasm support.".to_owned(), + ); + + result } else { // Debugging / shouldn't happen with proper command line handling above. Err(format!("Unhandled args: {:?}", args)) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 4f44afa6f6..4c1907b6ce 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -9,14 +9,11 @@ use cretonne_codegen::print_errors::{pretty_error, pretty_verifier_error}; use cretonne_codegen::settings::FlagsOrIsa; use cretonne_wasm::{translate_module, DummyEnvironment, ModuleEnvironment}; use std::error::Error; -use std::fs::File; -use std::io; use std::path::Path; use std::path::PathBuf; -use std::process::Command; -use tempdir::TempDir; use term; use utils::{parse_sets_and_isa, read_to_end}; +use wabt::wat2wasm; macro_rules! vprintln { ($x: expr, $($tts:tt)*) => { @@ -85,23 +82,12 @@ fn handle_module( let mut data = read_to_end(path.clone()).map_err(|err| { String::from(err.description()) })?; + if !data.starts_with(&[b'\0', b'a', b's', b'm']) { - let tmp_dir = TempDir::new("cretonne-wasm").unwrap(); - let file_path = tmp_dir.path().join("module.wasm"); - File::create(file_path.clone()).unwrap(); - Command::new("wat2wasm") - .arg(path.clone()) - .arg("-o") - .arg(file_path.to_str().unwrap()) - .output() - .or_else(|e| if let io::ErrorKind::NotFound = e.kind() { - return Err(String::from("wat2wasm not found")); - } else { - return Err(String::from(e.description())); - })?; - data = read_to_end(file_path).map_err( - |err| String::from(err.description()), - )?; + data = match wat2wasm(&data) { + Ok(data) => data, + Err(e) => return Err(String::from(e.description())), + }; } let mut dummy_environ = DummyEnvironment::with_flags(fisa.flags.clone()); diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index f1ad6b726c..3882a850cb 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -17,7 +17,7 @@ failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } [dev-dependencies] -tempdir = "0.3.5" +wabt = "0.3" [features] default = ["std"] diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs index 1871b4d781..3492c7bd89 100644 --- a/lib/wasm/tests/wasm_testsuite.rs +++ b/lib/wasm/tests/wasm_testsuite.rs @@ -1,20 +1,17 @@ extern crate cretonne_codegen; extern crate cretonne_wasm; -extern crate tempdir; +extern crate wabt; use cretonne_codegen::print_errors::pretty_verifier_error; use cretonne_codegen::settings::{self, Configurable, Flags}; use cretonne_codegen::verifier; use cretonne_wasm::{translate_module, DummyEnvironment}; -use std::error::Error; use std::fs; use std::fs::File; use std::io; use std::io::prelude::*; -use std::path::PathBuf; -use std::process::Command; -use std::str; -use tempdir::TempDir; +use std::path::Path; +use wabt::wat2wasm; #[test] fn testsuite() { @@ -44,56 +41,32 @@ fn return_at_end() { let mut flag_builder = settings::builder(); flag_builder.enable("return_at_end").unwrap(); let flags = Flags::new(flag_builder); - handle_module(&PathBuf::from("../../wasmtests/return_at_end.wat"), &flags); + handle_module(Path::new("../../wasmtests/return_at_end.wat"), &flags); } -fn read_wasm_file(path: PathBuf) -> Result, io::Error> { +fn read_file(path: &Path) -> Result, io::Error> { let mut buf: Vec = Vec::new(); let mut file = File::open(path)?; file.read_to_end(&mut buf)?; Ok(buf) } -fn handle_module(path: &PathBuf, flags: &Flags) { +fn handle_module(path: &Path, flags: &Flags) { let data = match path.extension() { None => { panic!("the file extension is not wasm or wat"); } Some(ext) => { match ext.to_str() { - Some("wasm") => read_wasm_file(path.clone()).expect("error reading wasm file"), + Some("wasm") => read_file(path).expect("error reading wasm file"), Some("wat") => { - let tmp_dir = TempDir::new("cretonne-wasm").unwrap(); - let file_path = tmp_dir.path().join("module.wasm"); - File::create(file_path.clone()).unwrap(); - let result_output = Command::new("wat2wasm") - .arg(path.clone()) - .arg("-o") - .arg(file_path.to_str().unwrap()) - .output(); - match result_output { + let wat = read_file(path).expect("error reading wat file"); + match wat2wasm(&wat) { + Ok(wasm) => wasm, Err(e) => { - if e.kind() == io::ErrorKind::NotFound { - println!( - "wat2wasm not found; disabled test {}", - path.to_str().unwrap() - ); - return; - } - panic!("error convering wat file: {}", e.description()); - } - Ok(output) => { - if !output.status.success() { - panic!( - "error running wat2wasm: {}", - str::from_utf8(&output.stderr).expect( - "wat2wasm's error message should be valid UTF-8", - ) - ); - } + panic!("error converting wat to wasm: {:?}", e); } } - read_wasm_file(file_path).expect("error reading converted wasm file") } None | Some(&_) => panic!("the file extension for {:?} is not wasm or wat", path), } From 191bab162b93ab5dbeb5c7abb58f76aaac258806 Mon Sep 17 00:00:00 2001 From: data-pup Date: Thu, 24 May 2018 17:16:25 -0400 Subject: [PATCH 1790/3084] Optimize 0.0 floating point constants. (#344) * Optimize 0.0 floating point constants. Rather than using the existing process of emitting bit patterns and moving them into floating point registers, use the `xorps` instruction to zero out the register. * is_zero predicate function will not accept negative zero. Fixed formatting for encoding recipe and filetests. --- .../x86/optimized-zero-constants-32bit.cton | 19 +++++++++++ .../isa/x86/optimized-zero-constants.cton | 33 +++++++++++++++++++ lib/codegen/meta/cdsl/predicates.py | 32 ++++++++++++++++++ lib/codegen/meta/isa/x86/encodings.py | 16 ++++++++- lib/codegen/meta/isa/x86/recipes.py | 22 ++++++++++++- lib/codegen/src/predicates.rs | 14 ++++++++ 6 files changed, 134 insertions(+), 2 deletions(-) create mode 100644 cranelift/filetests/isa/x86/optimized-zero-constants-32bit.cton create mode 100644 cranelift/filetests/isa/x86/optimized-zero-constants.cton diff --git a/cranelift/filetests/isa/x86/optimized-zero-constants-32bit.cton b/cranelift/filetests/isa/x86/optimized-zero-constants-32bit.cton new file mode 100644 index 0000000000..338569d055 --- /dev/null +++ b/cranelift/filetests/isa/x86/optimized-zero-constants-32bit.cton @@ -0,0 +1,19 @@ +; Check that floating-point constants equal to zero are optimized correctly. +test binemit +set is_64bit=0 +isa x86 + +function %foo() -> f32 fast { +ebb0: + ; asm: xorps %xmm0, %xmm0 + [-,%xmm0] v0 = f32const 0.0 ; bin: 0f 57 c0 + return v0 +} + +function %bar() -> f64 fast { +ebb0: + ; asm: xorpd %xmm0, %xmm0 + [-,%xmm0] v1 = f64const 0.0 ; bin: 66 0f 57 c0 + return v1 +} + diff --git a/cranelift/filetests/isa/x86/optimized-zero-constants.cton b/cranelift/filetests/isa/x86/optimized-zero-constants.cton new file mode 100644 index 0000000000..d83bfd3ade --- /dev/null +++ b/cranelift/filetests/isa/x86/optimized-zero-constants.cton @@ -0,0 +1,33 @@ +; Check that floating-point constants equal to zero are optimized correctly. +test binemit +set is_64bit=1 +isa x86 + +function %zero_const_32bit_no_rex() -> f32 fast { +ebb0: + ; asm: xorps %xmm0, %xmm0 + [-,%xmm0] v0 = f32const 0.0 ; bin: 40 0f 57 c0 + return v0 +} + +function %zero_const_32bit_rex() -> f32 fast { +ebb0: + ; asm: xorps %xmm8, %xmm8 + [-,%xmm8] v1 = f32const 0.0 ; bin: 45 0f 57 c0 + return v1 +} + +function %zero_const_64bit_no_rex() -> f64 fast { +ebb0: + ; asm: xorpd %xmm0, %xmm0 + [-,%xmm0] v0 = f64const 0.0 ; bin: 66 40 0f 57 c0 + return v0 +} + +function %zero_const_64bit_rex() -> f64 fast { +ebb0: + ; asm: xorpd %xmm8, %xmm8 + [-,%xmm8] v1 = f64const 0.0 ; bin: 66 45 0f 57 c0 + return v1 +} + diff --git a/lib/codegen/meta/cdsl/predicates.py b/lib/codegen/meta/cdsl/predicates.py index 13957c5d02..db1223e09e 100644 --- a/lib/codegen/meta/cdsl/predicates.py +++ b/lib/codegen/meta/cdsl/predicates.py @@ -262,6 +262,38 @@ class IsEqual(FieldPredicate): self.value = value +class IsZero32BitFloat(FieldPredicate): + """ + Instruction predicate that checks if an immediate instruction format field + is equal to zero. + + :param field: `FormatField` to be checked. + :param value: The constant value to check. + """ + + def __init__(self, field): + # type: (FormatField) -> None + super(IsZero32BitFloat, self).__init__(field, + 'is_zero_32_bit_float', + ()) + + +class IsZero64BitFloat(FieldPredicate): + """ + Instruction predicate that checks if an immediate instruction format field + is equal to zero. + + :param field: `FormatField` to be checked. + :param value: The constant value to check. + """ + + def __init__(self, field): + # type: (FormatField) -> None + super(IsZero64BitFloat, self).__init__(field, + 'is_zero_64_bit_float', + ()) + + class IsSignedInt(FieldPredicate): """ Instruction predicate that checks if an immediate instruction format field diff --git a/lib/codegen/meta/isa/x86/encodings.py b/lib/codegen/meta/isa/x86/encodings.py index 1a9260a28d..2909aac315 100644 --- a/lib/codegen/meta/isa/x86/encodings.py +++ b/lib/codegen/meta/isa/x86/encodings.py @@ -2,10 +2,12 @@ x86 Encodings. """ from __future__ import absolute_import +from cdsl.predicates import IsZero32BitFloat, IsZero64BitFloat from cdsl.predicates import IsUnsignedInt, Not, And from base.predicates import IsColocatedFunc, IsColocatedData, LengthEquals from base import instructions as base -from base.formats import UnaryImm, FuncAddr, Call, LoadComplex, StoreComplex +from base.formats import UnaryIeee32, UnaryIeee64, UnaryImm +from base.formats import FuncAddr, Call, LoadComplex, StoreComplex from .defs import X86_64, X86_32 from . import recipes as r from . import settings as cfg @@ -604,6 +606,18 @@ X86_64.enc(base.uextend.i64.i32, *r.umr(0x89)) # Floating point # +# floating-point constants equal to 0.0 can be encoded using either +# `xorps` or `xorpd`, for 32-bit and 64-bit floats respectively. +X86_32.enc(base.f32const, *r.f32imm_z(0x0f, 0x57), + instp=IsZero32BitFloat(UnaryIeee32.imm)) +X86_32.enc(base.f64const, *r.f64imm_z(0x66, 0x0f, 0x57), + instp=IsZero64BitFloat(UnaryIeee64.imm)) + +enc_x86_64_instp(base.f32const, r.f32imm_z, + IsZero32BitFloat(UnaryIeee32.imm), 0x0f, 0x57) +enc_x86_64_instp(base.f64const, r.f64imm_z, + IsZero64BitFloat(UnaryIeee64.imm), 0x66, 0x0f, 0x57) + # movd enc_both(base.bitcast.f32.i32, r.frurm, 0x66, 0x0f, 0x6e) enc_both(base.bitcast.i32.f32, r.rfumr, 0x66, 0x0f, 0x7e) diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py index d1f4bac0c2..047408db9c 100644 --- a/lib/codegen/meta/isa/x86/recipes.py +++ b/lib/codegen/meta/isa/x86/recipes.py @@ -4,8 +4,10 @@ x86 Encoding recipes. from __future__ import absolute_import from cdsl.isa import EncRecipe from cdsl.predicates import IsSignedInt, IsEqual, Or +from cdsl.predicates import IsZero32BitFloat, IsZero64BitFloat from cdsl.registers import RegClass -from base.formats import Unary, UnaryImm, UnaryBool, Binary, BinaryImm +from base.formats import Unary, UnaryIeee32, UnaryIeee64, UnaryImm, UnaryBool +from base.formats import Binary, BinaryImm from base.formats import MultiAry, NullAry from base.formats import Trap, Call, CallIndirect, Store, Load from base.formats import IntCompare, IntCompareImm, FloatCompare @@ -544,6 +546,24 @@ pu_iq = TailRecipe( sink.put8(imm as u64); ''') +# XX /n Unary with floating point 32-bit immediate equal to zero. +f32imm_z = TailRecipe( + 'f32imm_z', UnaryIeee32, size=1, ins=(), outs=FPR, + instp=IsZero32BitFloat(UnaryIeee32.imm), + emit=''' + PUT_OP(bits, rex2(out_reg0, out_reg0), sink); + modrm_rr(out_reg0, out_reg0, sink); + ''') + +# XX /n Unary with floating point 64-bit immediate equal to zero. +f64imm_z = TailRecipe( + 'f64imm_z', UnaryIeee64, size=1, ins=(), outs=FPR, + instp=IsZero64BitFloat(UnaryIeee64.imm), + emit=''' + PUT_OP(bits, rex2(out_reg0, out_reg0), sink); + modrm_rr(out_reg0, out_reg0, sink); + ''') + pushq = TailRecipe( 'pushq', Unary, size=0, ins=GPR, outs=(), emit=''' diff --git a/lib/codegen/src/predicates.rs b/lib/codegen/src/predicates.rs index 1fb700024e..74b6e1d8cd 100644 --- a/lib/codegen/src/predicates.rs +++ b/lib/codegen/src/predicates.rs @@ -11,6 +11,20 @@ use ir; +/// Check that a 64-bit floating point value is zero. +#[allow(dead_code)] +pub fn is_zero_64_bit_float>(x: T) -> bool { + let x64 = x.into(); + x64.bits() == 0 +} + +/// Check that a 32-bit floating point value is zero. +#[allow(dead_code)] +pub fn is_zero_32_bit_float>(x: T) -> bool { + let x32 = x.into(); + x32.bits() == 0 +} + /// Check that `x` is the same as `y`. #[allow(dead_code)] pub fn is_equal + Copy>(x: T, y: O) -> bool { From d46ceedc981fd51f92e0a783a1423665c60660f6 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 24 May 2018 13:14:10 -0700 Subject: [PATCH 1791/3084] Tidy up unneeded commmas. --- lib/filetests/src/test_binemit.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index a0711414d4..332ef3e574 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -79,11 +79,11 @@ impl binemit::CodeSink for TextSink { name: &ir::ExternalName, addend: binemit::Addend, ) { - write!(self.text, "{}({}", reloc, name,).unwrap(); + write!(self.text, "{}({}", reloc, name).unwrap(); if addend != 0 { - write!(self.text, "{:+}", addend,).unwrap(); + write!(self.text, "{:+}", addend).unwrap(); } - write!(self.text, ") ",).unwrap(); + write!(self.text, ") ").unwrap(); } fn reloc_jt(&mut self, reloc: binemit::Reloc, jt: ir::JumpTable) { From a6e6b79a2ef9b216f4be861e05520b1c40f8dbb5 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 24 May 2018 13:59:54 -0700 Subject: [PATCH 1792/3084] Remove the install of the wasm-toolchain binary package. With 8f4a3586e5a787fd32a9afe35df4a9811d92c66b, we now have proper wabt integration, so we no longer need to download and install a separate binary package. --- .travis.yml | 3 --- 1 file changed, 3 deletions(-) diff --git a/.travis.yml b/.travis.yml index 9329904a88..15152f6823 100644 --- a/.travis.yml +++ b/.travis.yml @@ -13,9 +13,6 @@ addons: apt: packages: - python3-pip -before_install: - - wget https://storage.googleapis.com/wasm-llvm/builds/linux/26619/wasm-toolchain_0.1.26619_amd64.deb - - sudo dpkg -i wasm-toolchain_0.1.26619_amd64.deb install: - pip3 install --user --upgrade mypy flake8 - mypy --version From 99f6055c55c8c62078ca7960ce514a0404be38d0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 24 May 2018 16:53:34 -0700 Subject: [PATCH 1793/3084] Fix warnings reported by bashate. See https://pypi.org/project/bashate/ for more info. --- cranelift/test-all.sh | 2 +- cranelift/test-no_std.sh | 5 ++--- lib/codegen/meta/check.sh | 2 +- 3 files changed, 4 insertions(+), 5 deletions(-) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 96e81f1864..2f25e26f33 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -18,7 +18,7 @@ export PYTHONDONTWRITEBYTECODE=1 cd $(dirname "$0") topdir=$(pwd) -function banner() { +function banner { echo "====== $@ ======" } diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh index 8c4d6ecfc1..8ca93f5d04 100755 --- a/cranelift/test-no_std.sh +++ b/cranelift/test-no_std.sh @@ -8,15 +8,14 @@ set -euo pipefail cd $(dirname "$0") topdir=$(pwd) -function banner() { +function banner { echo "====== $@ ======" } # Test those packages which have no_std support. LIBS="codegen frontend wasm native module simplejit umbrella" cd "$topdir" -for LIB in $LIBS -do +for LIB in $LIBS; do banner "Rust unit tests in $LIB" cd "lib/$LIB" diff --git a/lib/codegen/meta/check.sh b/lib/codegen/meta/check.sh index fa86bf0804..d8f2178f43 100755 --- a/lib/codegen/meta/check.sh +++ b/lib/codegen/meta/check.sh @@ -2,7 +2,7 @@ set -euo pipefail cd $(dirname "$0") -function runif() { +function runif { if command -v "$1" > /dev/null; then echo " === $1 ===" "$@" From 6b88cd44a887cd278c53368f47e7094abcfd3cd8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 25 May 2018 11:38:38 -0700 Subject: [PATCH 1794/3084] Update to rustfmt-preview (#348) * Update to rustfmt-preview. * Run "cargo fmt --all" with rustfmt 0.4.1. rustfmt 0.4.1 is the latest release of rustfmt-preview available on the stable channel. * Fix a long line that rustfmt 0.4.1 can't handle. * Remove unneeded commas left behind by rustfmt. --- .travis.yml | 11 +- check-rustfmt.sh | 40 -- cranelift/format-all.sh | 2 +- cranelift/src/cat.rs | 8 +- cranelift/src/compile.rs | 16 +- cranelift/src/cton-util.rs | 16 +- cranelift/src/print_cfg.rs | 8 +- cranelift/src/rsfilecheck.rs | 34 +- cranelift/test-all.sh | 16 +- lib/codegen/build.rs | 22 +- lib/codegen/src/abi.rs | 6 +- lib/codegen/src/bforest/map.rs | 36 +- lib/codegen/src/bforest/node.rs | 62 +-- lib/codegen/src/bforest/path.rs | 20 +- lib/codegen/src/bforest/pool.rs | 6 +- lib/codegen/src/bforest/set.rs | 18 +- lib/codegen/src/binemit/mod.rs | 2 +- lib/codegen/src/binemit/relaxation.rs | 16 +- lib/codegen/src/binemit/shrink.rs | 5 +- lib/codegen/src/bitset.rs | 5 +- lib/codegen/src/context.rs | 17 +- lib/codegen/src/cursor.rs | 32 +- lib/codegen/src/dbg.rs | 2 +- lib/codegen/src/dce.rs | 10 +- lib/codegen/src/divconst_magic_numbers.rs | 16 +- lib/codegen/src/dominator_tree.rs | 54 +- lib/codegen/src/flowgraph.rs | 24 +- lib/codegen/src/ir/dfg.rs | 91 ++-- lib/codegen/src/ir/entities.rs | 24 +- lib/codegen/src/ir/extfunc.rs | 3 +- lib/codegen/src/ir/function.rs | 8 +- lib/codegen/src/ir/immediates.rs | 48 +- lib/codegen/src/ir/instructions.rs | 71 ++- lib/codegen/src/ir/jumptable.rs | 6 +- lib/codegen/src/ir/layout.rs | 21 +- lib/codegen/src/ir/libcall.rs | 43 +- lib/codegen/src/ir/mod.rs | 2 +- lib/codegen/src/ir/valueloc.rs | 20 +- lib/codegen/src/isa/arm64/abi.rs | 6 +- lib/codegen/src/isa/constraints.rs | 6 +- lib/codegen/src/isa/encoding.rs | 7 +- lib/codegen/src/isa/mod.rs | 7 +- lib/codegen/src/isa/registers.rs | 10 +- lib/codegen/src/isa/riscv/abi.rs | 10 +- lib/codegen/src/isa/stack.rs | 6 +- lib/codegen/src/isa/x86/abi.rs | 25 +- lib/codegen/src/isa/x86/enc_tables.rs | 57 +-- lib/codegen/src/iterators.rs | 3 +- lib/codegen/src/legalizer/boundary.rs | 52 +- lib/codegen/src/legalizer/call.rs | 9 +- lib/codegen/src/legalizer/globalvar.rs | 5 +- lib/codegen/src/legalizer/heap.rs | 35 +- lib/codegen/src/legalizer/libcall.rs | 14 +- lib/codegen/src/legalizer/mod.rs | 2 +- lib/codegen/src/legalizer/split.rs | 15 +- lib/codegen/src/lib.rs | 22 +- lib/codegen/src/licm.rs | 6 +- lib/codegen/src/nan_canonicalization.rs | 20 +- lib/codegen/src/postopt.rs | 41 +- lib/codegen/src/preopt.rs | 101 ++-- lib/codegen/src/regalloc/affinity.rs | 13 +- lib/codegen/src/regalloc/coalescing.rs | 73 +-- lib/codegen/src/regalloc/coloring.rs | 102 ++-- lib/codegen/src/regalloc/context.rs | 9 +- lib/codegen/src/regalloc/diversion.rs | 13 +- .../src/regalloc/live_value_tracker.rs | 22 +- lib/codegen/src/regalloc/liveness.rs | 13 +- lib/codegen/src/regalloc/liverange.rs | 19 +- lib/codegen/src/regalloc/mod.rs | 4 +- lib/codegen/src/regalloc/pressure.rs | 25 +- lib/codegen/src/regalloc/register_set.rs | 14 +- lib/codegen/src/regalloc/reload.rs | 45 +- lib/codegen/src/regalloc/solver.rs | 92 ++-- lib/codegen/src/regalloc/spilling.rs | 54 +- lib/codegen/src/regalloc/virtregs.rs | 18 +- lib/codegen/src/result.rs | 5 +- lib/codegen/src/settings.rs | 2 +- lib/codegen/src/simple_gvn.rs | 6 +- lib/codegen/src/stack_layout.rs | 27 +- lib/codegen/src/timing.rs | 11 +- lib/codegen/src/verifier/cssa.rs | 8 +- lib/codegen/src/verifier/liveness.rs | 4 +- lib/codegen/src/verifier/locations.rs | 20 +- lib/codegen/src/verifier/mod.rs | 158 +++--- lib/codegen/src/write.rs | 27 +- lib/entity/src/lib.rs | 15 +- lib/entity/src/map.rs | 2 +- lib/entity/src/packed_option.rs | 6 +- lib/entity/src/primary.rs | 2 +- lib/entity/src/set.rs | 2 +- lib/entity/src/sparse.rs | 2 +- lib/faerie/src/backend.rs | 46 +- lib/faerie/src/lib.rs | 16 +- lib/faerie/src/target.rs | 7 +- lib/faerie/src/traps.rs | 3 +- lib/filetests/src/concurrent.rs | 10 +- lib/filetests/src/lib.rs | 11 +- lib/filetests/src/runner.rs | 35 +- lib/filetests/src/runone.rs | 12 +- lib/filetests/src/subtest.rs | 24 +- lib/filetests/src/test_binemit.rs | 37 +- lib/filetests/src/test_compile.rs | 14 +- lib/filetests/src/test_dce.rs | 10 +- lib/filetests/src/test_domtree.rs | 20 +- lib/filetests/src/test_legalizer.rs | 9 +- lib/filetests/src/test_licm.rs | 10 +- lib/filetests/src/test_postopt.rs | 10 +- lib/filetests/src/test_preopt.rs | 10 +- lib/filetests/src/test_regalloc.rs | 15 +- lib/filetests/src/test_simple_gvn.rs | 10 +- lib/filetests/src/test_verifier.rs | 37 +- lib/frontend/src/frontend.rs | 87 ++-- lib/frontend/src/lib.rs | 13 +- lib/frontend/src/ssa.rs | 49 +- lib/module/src/backend.rs | 2 +- lib/module/src/data_context.rs | 8 +- lib/module/src/lib.rs | 18 +- lib/module/src/module.rs | 127 +++-- lib/native/src/lib.rs | 14 +- lib/reader/src/isaspec.rs | 26 +- lib/reader/src/lexer.rs | 65 +-- lib/reader/src/lib.rs | 14 +- lib/reader/src/parser.rs | 462 +++++------------- lib/reader/src/sourcemap.rs | 68 +-- lib/simplejit/src/backend.rs | 61 +-- lib/simplejit/src/lib.rs | 18 +- lib/simplejit/src/memory.rs | 15 +- lib/umbrella/src/lib.rs | 27 +- lib/wasm/src/code_translator.rs | 370 +++++++------- lib/wasm/src/environ/dummy.rs | 56 +-- lib/wasm/src/func_translator.rs | 70 +-- lib/wasm/src/lib.rs | 21 +- lib/wasm/src/module_translator.rs | 73 ++- lib/wasm/src/sections_translator.rs | 69 ++- lib/wasm/src/state.rs | 72 ++- lib/wasm/src/translation_utils.rs | 8 +- lib/wasm/tests/wasm_testsuite.rs | 22 +- 137 files changed, 1914 insertions(+), 2380 deletions(-) delete mode 100755 check-rustfmt.sh diff --git a/.travis.yml b/.travis.yml index 15152f6823..b213bd1010 100644 --- a/.travis.yml +++ b/.travis.yml @@ -16,7 +16,16 @@ addons: install: - pip3 install --user --upgrade mypy flake8 - mypy --version - - travis_wait ./check-rustfmt.sh --install +before_script: + - cargo uninstall rustfmt || true + - cargo install --list + - rustup toolchain install stable + - rustup component add --toolchain=stable rustfmt-preview + - rustup component list --toolchain=stable + - rustup show + - rustfmt +stable --version || echo fail + - rustup update + - rustfmt +stable --version script: ./test-all.sh cache: cargo: true diff --git a/check-rustfmt.sh b/check-rustfmt.sh deleted file mode 100755 index 9a49e8bac5..0000000000 --- a/check-rustfmt.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/bash -set -euo pipefail - -# Usage: check-rustfmt.sh [--install] -# -# Check that the desired version of rustfmt is installed. -# -# Rustfmt is still immature enough that its formatting decisions can change -# between versions. This makes it difficult to enforce a certain style in a -# test script since not all developers will upgrade rustfmt at the same time. -# To work around this, we only verify formatting when a specific version of -# rustfmt is installed. -# -# Exits 0 if the right version of rustfmt is installed, 1 otherwise. -# -# With the --install option, also tries to install the right version. - -# This version should always be bumped to the newest version available that -# works with stable Rust. -# ... but not 0.10.0, since it's the same as 0.9.0 except for a deprecation -# error (and it requires --force to disable the error and enable normal -# operation, however that doesn't appear to be possible through "cargo fmt"). -VERS="0.9.0" - -if cargo install --list | tee /dev/null | grep -q "^rustfmt v$VERS"; then - exit 0 -fi - -if [[ ${1:-""} != "--install" ]]; then - echo "********************************************************************" - echo "* Please install rustfmt v$VERS to verify formatting. *" - echo "* If a newer version of rustfmt is available, update this script. *" - echo "********************************************************************" - echo "$0 --install" - sleep 1 - exit 1 -fi - -echo "Installing rustfmt v$VERS." -cargo install --force --vers="$VERS" rustfmt diff --git a/cranelift/format-all.sh b/cranelift/format-all.sh index a99da7db09..1e27718801 100755 --- a/cranelift/format-all.sh +++ b/cranelift/format-all.sh @@ -8,4 +8,4 @@ cd $(dirname "$0") # Make sure we can find rustfmt. export PATH="$PATH:$HOME/.cargo/bin" -exec cargo fmt --all -- "$@" +exec cargo +stable fmt --all -- "$@" diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs index c4890ffb24..a906214730 100644 --- a/cranelift/src/cat.rs +++ b/cranelift/src/cat.rs @@ -18,12 +18,8 @@ pub fn run(files: &[String]) -> CommandResult { } fn cat_one(filename: &str) -> CommandResult { - let buffer = read_to_string(&filename).map_err( - |e| format!("{}: {}", filename, e), - )?; - let items = parse_functions(&buffer).map_err( - |e| format!("{}: {}", filename, e), - )?; + let buffer = read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))?; + let items = parse_functions(&buffer).map_err(|e| format!("{}: {}", filename, e))?; for (idx, func) in items.into_iter().enumerate() { if idx != 0 { diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index def23bf7de..560c9f3497 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -1,10 +1,10 @@ //! CLI tool to read Cretonne IR files and compile them into native code. use capstone::prelude::*; +use cretonne_codegen::Context; use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::print_errors::pretty_error; use cretonne_codegen::settings::FlagsOrIsa; -use cretonne_codegen::Context; use cretonne_codegen::{binemit, ir}; use cretonne_reader::parse_test; use std::path::Path; @@ -80,9 +80,7 @@ fn handle_module( name: &str, fisa: FlagsOrIsa, ) -> Result<(), String> { - let buffer = read_to_string(&path).map_err( - |e| format!("{}: {}", name, e), - )?; + let buffer = read_to_string(&path).map_err(|e| format!("{}: {}", name, e))?; let test_file = parse_test(&buffer).map_err(|e| format!("{}: {}", name, e))?; // If we have an isa from the command-line, use that. Otherwise if the @@ -154,12 +152,10 @@ fn get_disassembler(isa: &TargetIsa) -> Result { } } "arm32" => Capstone::new().arm().mode(arch::arm::ArchMode::Arm).build(), - "arm64" => { - Capstone::new() - .arm64() - .mode(arch::arm64::ArchMode::Arm) - .build() - } + "arm64" => Capstone::new() + .arm64() + .mode(arch::arm64::ArchMode::Arm) + .build(), _ => return Err(String::from("Unknown ISA")), }; diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 1ae6991dfb..3723cbecda 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -1,14 +1,8 @@ #![deny(trivial_numeric_casts)] #![warn(unused_import_braces, unstable_features, unused_extern_crates)] -#![cfg_attr(feature="cargo-clippy", warn( - float_arithmetic, - mut_mut, - nonminimal_bool, - option_map_unwrap_or, - option_map_unwrap_or_else, - unicode_not_nfc, - use_self, - ))] +#![cfg_attr(feature = "cargo-clippy", + warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, + option_map_unwrap_or_else, unicode_not_nfc, use_self))] #[macro_use] extern crate cfg_if; @@ -134,9 +128,7 @@ fn cton_util() -> CommandResult { ); #[cfg(not(feature = "wasm"))] - let result = Err( - "Error: cton-util was compiled without wasm support.".to_owned(), - ); + let result = Err("Error: cton-util was compiled without wasm support.".to_owned()); result } else { diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index f078c7ee54..28c8a81100 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -19,12 +19,8 @@ pub fn run(files: &[String]) -> CommandResult { } fn print_cfg(filename: &str) -> CommandResult { - let buffer = read_to_string(filename).map_err( - |e| format!("{}: {}", filename, e), - )?; - let items = parse_functions(&buffer).map_err( - |e| format!("{}: {}", filename, e), - )?; + let buffer = read_to_string(filename).map_err(|e| format!("{}: {}", filename, e))?; + let items = parse_functions(&buffer).map_err(|e| format!("{}: {}", filename, e))?; for (idx, func) in items.into_iter().enumerate() { if idx != 0 { diff --git a/cranelift/src/rsfilecheck.rs b/cranelift/src/rsfilecheck.rs index 6269ed226b..b2b1c66ba8 100644 --- a/cranelift/src/rsfilecheck.rs +++ b/cranelift/src/rsfilecheck.rs @@ -22,14 +22,14 @@ pub fn run(files: &[String], verbose: bool) -> CommandResult { } let mut buffer = String::new(); - io::stdin().read_to_string(&mut buffer).map_err(|e| { - format!("stdin: {}", e) - })?; + io::stdin() + .read_to_string(&mut buffer) + .map_err(|e| format!("stdin: {}", e))?; if verbose { - let (success, explain) = checker.explain(&buffer, NO_VARIABLES).map_err( - |e| e.to_string(), - )?; + let (success, explain) = checker + .explain(&buffer, NO_VARIABLES) + .map_err(|e| e.to_string())?; print!("{}", explain); if success { println!("OK"); @@ -37,27 +37,25 @@ pub fn run(files: &[String], verbose: bool) -> CommandResult { } else { Err("Check failed".to_string()) } - } else if checker.check(&buffer, NO_VARIABLES).map_err( - |e| e.to_string(), - )? + } else if checker + .check(&buffer, NO_VARIABLES) + .map_err(|e| e.to_string())? { Ok(()) } else { - let (_, explain) = checker.explain(&buffer, NO_VARIABLES).map_err( - |e| e.to_string(), - )?; + let (_, explain) = checker + .explain(&buffer, NO_VARIABLES) + .map_err(|e| e.to_string())?; print!("{}", explain); Err("Check failed".to_string()) } } fn read_checkfile(filename: &str) -> Result { - let buffer = read_to_string(&filename).map_err( - |e| format!("{}: {}", filename, e), - )?; + let buffer = read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))?; let mut builder = CheckerBuilder::new(); - builder.text(&buffer).map_err( - |e| format!("{}: {}", filename, e), - )?; + builder + .text(&buffer) + .map_err(|e| format!("{}: {}", filename, e))?; Ok(builder.finish()) } diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 2f25e26f33..fcdf7e57c4 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -23,9 +23,19 @@ function banner { } # Run rustfmt if we have it. -if $topdir/check-rustfmt.sh; then - banner "Rust formatting" - $topdir/format-all.sh --write-mode=diff +banner "Rust formatting" +if command -v rustfmt > /dev/null; then + # In newer versions of rustfmt, replace --write-mode=diff with --check. + if ! $topdir/format-all.sh --write-mode=diff ; then + echo "Formatting diffs detected! Run \"cargo fmt --all\" to correct." + exit 1 + fi +else + echo "rustfmt not available; formatting not checked!" + echo + echo "If you are using rustup, rustfmt can be installed via" + echo "\"rustup component add --toolchain=stable rustfmt-preview\", or see" + echo "https://github.com/rust-lang-nursery/rustfmt for more information." fi # Check if any Python files have changed since we last checked them. diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs index 87d4b754ae..7e58b0d012 100644 --- a/lib/codegen/build.rs +++ b/lib/codegen/build.rs @@ -67,9 +67,7 @@ fn main() { .arg("--out-dir") .arg(out_dir) .status() - .expect( - "Failed to launch second-level build script; is python installed?", - ); + .expect("Failed to launch second-level build script; is python installed?"); if !status.success() { process::exit(status.code().unwrap()); } @@ -132,16 +130,14 @@ impl Isa { /// Returns isa targets to configure conditional compilation. fn isa_targets(cretonne_targets: Option<&str>, target_triple: &str) -> Result, String> { match cretonne_targets { - Some("native") => { - Isa::from_arch(target_triple.split('-').next().unwrap()) - .map(|isa| vec![isa]) - .ok_or_else(|| { - format!( - "no supported isa found for target triple `{}`", - target_triple - ) - }) - } + Some("native") => Isa::from_arch(target_triple.split('-').next().unwrap()) + .map(|isa| vec![isa]) + .ok_or_else(|| { + format!( + "no supported isa found for target triple `{}`", + target_triple + ) + }), Some(targets) => { let unknown_isa_targets = targets .split(',') diff --git a/lib/codegen/src/abi.rs b/lib/codegen/src/abi.rs index 5f1452159b..66dcf7f8f1 100644 --- a/lib/codegen/src/abi.rs +++ b/lib/codegen/src/abi.rs @@ -62,16 +62,14 @@ impl ValueConversion { ValueConversion::IntSplit => ty.half_width().expect("Integer type too small to split"), ValueConversion::VectorSplit => ty.half_vector().expect("Not a vector"), ValueConversion::IntBits => Type::int(ty.bits()).expect("Bad integer size"), - ValueConversion::Sext(nty) | - ValueConversion::Uext(nty) => nty, + ValueConversion::Sext(nty) | ValueConversion::Uext(nty) => nty, } } /// Is this a split conversion that results in two arguments? pub fn is_split(self) -> bool { match self { - ValueConversion::IntSplit | - ValueConversion::VectorSplit => true, + ValueConversion::IntSplit | ValueConversion::VectorSplit => true, _ => false, } } diff --git a/lib/codegen/src/bforest/map.rs b/lib/codegen/src/bforest/map.rs index 7cfa824a43..270d186230 100644 --- a/lib/codegen/src/bforest/map.rs +++ b/lib/codegen/src/bforest/map.rs @@ -2,9 +2,9 @@ use super::{Comparator, Forest, Node, NodeData, NodePool, Path, INNER_SIZE}; use packed_option::PackedOption; -use std::marker::PhantomData; #[cfg(test)] use std::fmt; +use std::marker::PhantomData; #[cfg(test)] use std::string::String; @@ -50,7 +50,9 @@ where { /// Create a new empty forest. pub fn new() -> Self { - Self { nodes: NodePool::new() } + Self { + nodes: NodePool::new(), + } } /// Clear all maps in the forest. @@ -101,9 +103,9 @@ where /// Get the value stored for `key`. pub fn get(&self, key: K, forest: &MapForest, comp: &C) -> Option { - self.root.expand().and_then(|root| { - Path::default().find(key, root, &forest.nodes, comp) - }) + self.root + .expand() + .and_then(|root| Path::default().find(key, root, &forest.nodes, comp)) } /// Look up the value stored for `key`. @@ -292,30 +294,30 @@ where /// /// If the cursor is already pointing at the first entry, leave it there and return `None`. pub fn prev(&mut self) -> Option<(K, V)> { - self.root.expand().and_then( - |root| self.path.prev(root, self.pool), - ) + self.root + .expand() + .and_then(|root| self.path.prev(root, self.pool)) } /// Get the current key, or `None` if the cursor is at the end. pub fn key(&self) -> Option { - self.path.leaf_pos().and_then(|(node, entry)| { - self.pool[node].unwrap_leaf().0.get(entry).cloned() - }) + self.path + .leaf_pos() + .and_then(|(node, entry)| self.pool[node].unwrap_leaf().0.get(entry).cloned()) } /// Get the current value, or `None` if the cursor is at the end. pub fn value(&self) -> Option { - self.path.leaf_pos().and_then(|(node, entry)| { - self.pool[node].unwrap_leaf().1.get(entry).cloned() - }) + self.path + .leaf_pos() + .and_then(|(node, entry)| self.pool[node].unwrap_leaf().1.get(entry).cloned()) } /// Get a mutable reference to the current value, or `None` if the cursor is at the end. pub fn value_mut(&mut self) -> Option<&mut V> { - self.path.leaf_pos().and_then(move |(node, entry)| { - self.pool[node].unwrap_leaf_mut().1.get_mut(entry) - }) + self.path + .leaf_pos() + .and_then(move |(node, entry)| self.pool[node].unwrap_leaf_mut().1.get_mut(entry)) } /// Move this cursor to `key`. diff --git a/lib/codegen/src/bforest/node.rs b/lib/codegen/src/bforest/node.rs index d47a16283b..b529816afb 100644 --- a/lib/codegen/src/bforest/node.rs +++ b/lib/codegen/src/bforest/node.rs @@ -362,16 +362,18 @@ impl NodeData { /// right sibling node is returned. pub fn balance(&mut self, crit_key: F::Key, rhs: &mut Self) -> Option { match (self, rhs) { - (&mut NodeData::Inner { - size: ref mut l_size, - keys: ref mut l_keys, - tree: ref mut l_tree, - }, - &mut NodeData::Inner { - size: ref mut r_size, - keys: ref mut r_keys, - tree: ref mut r_tree, - }) => { + ( + &mut NodeData::Inner { + size: ref mut l_size, + keys: ref mut l_keys, + tree: ref mut l_tree, + }, + &mut NodeData::Inner { + size: ref mut r_size, + keys: ref mut r_keys, + tree: ref mut r_tree, + }, + ) => { let l_ents = usize::from(*l_size) + 1; let r_ents = usize::from(*r_size) + 1; let ents = l_ents + r_ents; @@ -408,16 +410,18 @@ impl NodeData { Some(new_crit) } } - (&mut NodeData::Leaf { - size: ref mut l_size, - keys: ref mut l_keys, - vals: ref mut l_vals, - }, - &mut NodeData::Leaf { - size: ref mut r_size, - keys: ref mut r_keys, - vals: ref mut r_vals, - }) => { + ( + &mut NodeData::Leaf { + size: ref mut l_size, + keys: ref mut l_keys, + vals: ref mut l_vals, + }, + &mut NodeData::Leaf { + size: ref mut r_size, + keys: ref mut r_keys, + vals: ref mut r_vals, + }, + ) => { let l_ents = usize::from(*l_size); let l_keys = l_keys.borrow_mut(); let l_vals = l_vals.borrow_mut(); @@ -677,11 +681,7 @@ mod test { assert!(leaf.try_leaf_insert(2, 'c', SetValue())); assert_eq!(leaf.to_string(), "[ a b c d ]"); for i in 4..15 { - assert!(leaf.try_leaf_insert( - usize::from(i), - ('a' as u8 + i) as char, - SetValue(), - )); + assert!(leaf.try_leaf_insert(usize::from(i), ('a' as u8 + i) as char, SetValue())); } assert_eq!(leaf.to_string(), "[ a b c d e f g h i j k l m n o ]"); @@ -779,21 +779,13 @@ mod test { fn leaf_balance() { let mut lhs = NodeData::::leaf('a', SetValue()); for i in 1..6 { - assert!(lhs.try_leaf_insert( - usize::from(i), - ('a' as u8 + i) as char, - SetValue(), - )); + assert!(lhs.try_leaf_insert(usize::from(i), ('a' as u8 + i) as char, SetValue())); } assert_eq!(lhs.to_string(), "[ a b c d e f ]"); let mut rhs = NodeData::::leaf('0', SetValue()); for i in 1..8 { - assert!(rhs.try_leaf_insert( - usize::from(i), - ('0' as u8 + i) as char, - SetValue(), - )); + assert!(rhs.try_leaf_insert(usize::from(i), ('0' as u8 + i) as char, SetValue())); } assert_eq!(rhs.to_string(), "[ 0 1 2 3 4 5 6 7 ]"); diff --git a/lib/codegen/src/bforest/path.rs b/lib/codegen/src/bforest/path.rs index 3ed91ee09a..e845c6a9a2 100644 --- a/lib/codegen/src/bforest/path.rs +++ b/lib/codegen/src/bforest/path.rs @@ -303,9 +303,9 @@ impl Path { // When inserting into an inner node (`ins_node.is_some()`), we must point to a valid // entry in the current node since the new entry is inserted *after* the insert // location. - if entry > split.lhs_entries || - (entry == split.lhs_entries && - (split.lhs_entries > split.rhs_entries || ins_node.is_some())) + if entry > split.lhs_entries + || (entry == split.lhs_entries + && (split.lhs_entries > split.rhs_entries || ins_node.is_some())) { node = rhs_node; entry -= split.lhs_entries; @@ -406,7 +406,9 @@ impl Path { let crit_node = self.node[crit_level]; match pool[crit_node] { - NodeData::Inner { size, ref mut keys, .. } => { + NodeData::Inner { + size, ref mut keys, .. + } => { debug_assert!(crit_kidx < size); keys[usize::from(crit_kidx)] = crit_key; } @@ -436,7 +438,10 @@ impl Path { // Discard the root node if it has shrunk to a single sub-tree. let mut ns = 0; - while let NodeData::Inner { size: 0, ref tree, .. } = pool[self.node[ns]] { + while let NodeData::Inner { + size: 0, ref tree, .. + } = pool[self.node[ns]] + { ns += 1; self.node[ns] = tree[0]; } @@ -616,9 +621,8 @@ impl Path { /// Update the critical key for the right sibling node at `level`. fn update_right_crit_key(&self, level: usize, crit_key: F::Key, pool: &mut NodePool) { - let bl = self.right_sibling_branch_level(level, pool).expect( - "No right sibling exists", - ); + let bl = self.right_sibling_branch_level(level, pool) + .expect("No right sibling exists"); match pool[self.node[bl]] { NodeData::Inner { ref mut keys, .. } => { keys[usize::from(self.entry[bl])] = crit_key; diff --git a/lib/codegen/src/bforest/pool.rs b/lib/codegen/src/bforest/pool.rs index d543991312..e6b08e4f1c 100644 --- a/lib/codegen/src/bforest/pool.rs +++ b/lib/codegen/src/bforest/pool.rs @@ -2,9 +2,9 @@ use super::{Forest, Node, NodeData}; use entity::PrimaryMap; -use std::ops::{Index, IndexMut}; #[cfg(test)] use std::fmt; +use std::ops::{Index, IndexMut}; /// A pool of nodes, including a free list. pub(super) struct NodePool { @@ -51,7 +51,9 @@ impl NodePool { pub fn free_node(&mut self, node: Node) { // Quick check for a double free. debug_assert!(!self.nodes[node].is_free(), "{} is already free", node); - self.nodes[node] = NodeData::Free { next: self.freelist }; + self.nodes[node] = NodeData::Free { + next: self.freelist, + }; self.freelist = Some(node); } diff --git a/lib/codegen/src/bforest/set.rs b/lib/codegen/src/bforest/set.rs index 667cd133e2..8371fd3bc6 100644 --- a/lib/codegen/src/bforest/set.rs +++ b/lib/codegen/src/bforest/set.rs @@ -2,9 +2,9 @@ use super::{Comparator, Forest, Node, NodeData, NodePool, Path, SetValue, INNER_SIZE}; use packed_option::PackedOption; -use std::marker::PhantomData; #[cfg(test)] use std::fmt; +use std::marker::PhantomData; #[cfg(test)] use std::string::String; @@ -47,7 +47,9 @@ where { /// Create a new empty forest. pub fn new() -> Self { - Self { nodes: NodePool::new() } + Self { + nodes: NodePool::new(), + } } /// Clear all sets in the forest. @@ -232,16 +234,16 @@ where /// /// If the cursor is already pointing at the first element, leave it there and return `None`. pub fn prev(&mut self) -> Option { - self.root.expand().and_then(|root| { - self.path.prev(root, self.pool).map(|(k, _)| k) - }) + self.root + .expand() + .and_then(|root| self.path.prev(root, self.pool).map(|(k, _)| k)) } /// Get the current element, or `None` if the cursor is at the end. pub fn elem(&self) -> Option { - self.path.leaf_pos().and_then(|(node, entry)| { - self.pool[node].unwrap_leaf().0.get(entry).cloned() - }) + self.path + .leaf_pos() + .and_then(|(node, entry)| self.pool[node].unwrap_leaf().0.get(entry).cloned()) } /// Move this cursor to `elem`. diff --git a/lib/codegen/src/binemit/mod.rs b/lib/codegen/src/binemit/mod.rs index 653e38c9ed..4a95ed4eb7 100644 --- a/lib/codegen/src/binemit/mod.rs +++ b/lib/codegen/src/binemit/mod.rs @@ -7,7 +7,7 @@ mod memorysink; mod relaxation; mod shrink; -pub use self::memorysink::{MemoryCodeSink, RelocSink, TrapSink, NullTrapSink}; +pub use self::memorysink::{MemoryCodeSink, NullTrapSink, RelocSink, TrapSink}; pub use self::relaxation::relax_branches; pub use self::shrink::shrink_instructions; pub use regalloc::RegDiversions; diff --git a/lib/codegen/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs index 93c6afedec..7003df1eb0 100644 --- a/lib/codegen/src/binemit/relaxation.rs +++ b/lib/codegen/src/binemit/relaxation.rs @@ -78,8 +78,8 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> Result(4 | 8 | 256 | 1024); assert!( - !s4.contains(0) && !s4.contains(1) && !s4.contains(4) && !s4.contains(5) && - !s4.contains(6) && !s4.contains(7) && !s4.contains(9) && !s4.contains(11) + !s4.contains(0) && !s4.contains(1) && !s4.contains(4) && !s4.contains(5) + && !s4.contains(6) && !s4.contains(7) && !s4.contains(9) + && !s4.contains(11) ); assert!(s4.contains(2) && s4.contains(3) && s4.contains(8) && s4.contains(10)); } diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index b383160703..b9f7511865 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -24,8 +24,8 @@ use preopt::do_preopt; use regalloc; use result::{CtonError, CtonResult}; use settings::{FlagsOrIsa, OptLevel}; -use std::vec::Vec; use simple_gvn::do_simple_gvn; +use std::vec::Vec; use timing; use unreachable_code::eliminate_unreachable_code; use verifier; @@ -251,11 +251,8 @@ impl Context { /// Compute the loop analysis. pub fn compute_loop_analysis(&mut self) { - self.loop_analysis.compute( - &self.func, - &self.cfg, - &self.domtree, - ) + self.loop_analysis + .compute(&self.func, &self.cfg, &self.domtree) } /// Compute the control flow graph and dominator tree. @@ -292,12 +289,8 @@ impl Context { /// Run the register allocator. pub fn regalloc(&mut self, isa: &TargetIsa) -> CtonResult { - self.regalloc.run( - isa, - &mut self.func, - &self.cfg, - &mut self.domtree, - ) + self.regalloc + .run(isa, &mut self.func, &self.cfg, &mut self.domtree) } /// Insert prologue and epilogues after computing the stack frame layout. diff --git a/lib/codegen/src/cursor.rs b/lib/codegen/src/cursor.rs index 6842d8b348..e905545675 100644 --- a/lib/codegen/src/cursor.rs +++ b/lib/codegen/src/cursor.rs @@ -246,9 +246,11 @@ pub trait Cursor { let new_pos = if let Some(next) = self.layout().next_inst(inst) { CursorPosition::At(next) } else { - CursorPosition::After(self.layout().inst_ebb(inst).expect( - "current instruction removed?", - )) + CursorPosition::After( + self.layout() + .inst_ebb(inst) + .expect("current instruction removed?"), + ) }; self.set_position(new_pos); } @@ -413,9 +415,11 @@ pub trait Cursor { self.set_position(At(next)); Some(next) } else { - let pos = After(self.layout().inst_ebb(inst).expect( - "current instruction removed?", - )); + let pos = After( + self.layout() + .inst_ebb(inst) + .expect("current instruction removed?"), + ); self.set_position(pos); None } @@ -465,9 +469,11 @@ pub trait Cursor { self.set_position(At(prev)); Some(prev) } else { - let pos = Before(self.layout().inst_ebb(inst).expect( - "current instruction removed?", - )); + let pos = Before( + self.layout() + .inst_ebb(inst) + .expect("current instruction removed?"), + ); self.set_position(pos); None } @@ -746,11 +752,9 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut EncCursor<'f> { // Assign an encoding. // XXX Is there a way to describe this error to the user? #[cfg_attr(feature = "cargo-clippy", allow(match_wild_err_arm))] - match self.isa.encode( - &self.func, - &self.func.dfg[inst], - ctrl_typevar, - ) { + match self.isa + .encode(&self.func, &self.func.dfg[inst], ctrl_typevar) + { Ok(e) => self.func.encodings[inst] = e, Err(_) => panic!("can't encode {}", self.display_inst(inst)), } diff --git a/lib/codegen/src/dbg.rs b/lib/codegen/src/dbg.rs index 8dd47f625d..e1f62198f8 100644 --- a/lib/codegen/src/dbg.rs +++ b/lib/codegen/src/dbg.rs @@ -126,7 +126,7 @@ macro_rules! dbg { #[cfg(not(feature = "std"))] #[macro_export] macro_rules! dbg { - ($($arg:tt)+) => {} + ($($arg:tt)+) => {}; } /// Helper for printing lists. diff --git a/lib/codegen/src/dce.rs b/lib/codegen/src/dce.rs index 895b922dd7..797e905c6d 100644 --- a/lib/codegen/src/dce.rs +++ b/lib/codegen/src/dce.rs @@ -13,9 +13,8 @@ use timing; /// Test whether the given opcode is unsafe to even consider for DCE. fn trivially_unsafe_for_dce(opcode: Opcode) -> bool { - opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || - opcode.is_return() || opcode.can_trap() || opcode.other_side_effects() || - opcode.can_store() + opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || opcode.is_return() + || opcode.can_trap() || opcode.other_side_effects() || opcode.can_store() } /// Preserve instructions with used result values. @@ -51,9 +50,8 @@ pub fn do_dce(func: &mut Function, domtree: &mut DominatorTree) { { let data = &pos.func.dfg[inst]; let opcode = data.opcode(); - if trivially_unsafe_for_dce(opcode) || - is_load_with_defined_trapping(opcode, &data) || - any_inst_results_used(inst, &live, &pos.func.dfg) + if trivially_unsafe_for_dce(opcode) || is_load_with_defined_trapping(opcode, &data) + || any_inst_results_used(inst, &live, &pos.func.dfg) { for arg in pos.func.dfg.inst_args(inst) { let v = pos.func.dfg.resolve_aliases(*arg); diff --git a/lib/codegen/src/divconst_magic_numbers.rs b/lib/codegen/src/divconst_magic_numbers.rs index 8a848711a4..a56ed730d4 100644 --- a/lib/codegen/src/divconst_magic_numbers.rs +++ b/lib/codegen/src/divconst_magic_numbers.rs @@ -167,10 +167,10 @@ pub fn magicS32(d: i32) -> MS32 { MS32 { mulBy: (if d < 0 { - u32::wrapping_neg(q2 + 1) - } else { - q2 + 1 - }) as i32, + u32::wrapping_neg(q2 + 1) + } else { + q2 + 1 + }) as i32, shiftBy: p - 32, } } @@ -210,10 +210,10 @@ pub fn magicS64(d: i64) -> MS64 { MS64 { mulBy: (if d < 0 { - u64::wrapping_neg(q2 + 1) - } else { - q2 + 1 - }) as i64, + u64::wrapping_neg(q2 + 1) + } else { + q2 + 1 + }) as i64, shiftBy: p - 64, } } diff --git a/lib/codegen/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs index 0e48ce8d81..c8dec582aa 100644 --- a/lib/codegen/src/dominator_tree.rs +++ b/lib/codegen/src/dominator_tree.rs @@ -101,9 +101,8 @@ impl DominatorTree { { let a = a.into(); let b = b.into(); - self.rpo_cmp_ebb(layout.pp_ebb(a), layout.pp_ebb(b)).then( - layout.cmp(a, b), - ) + self.rpo_cmp_ebb(layout.pp_ebb(a), layout.pp_ebb(b)) + .then(layout.cmp(a, b)) } /// Returns `true` if `a` dominates `b`. @@ -145,9 +144,7 @@ impl DominatorTree { let (mut ebb_b, mut inst_b) = match b.into() { ExpandedProgramPoint::Ebb(ebb) => (ebb, None), ExpandedProgramPoint::Inst(inst) => ( - layout.inst_ebb(inst).expect( - "Instruction not in layout.", - ), + layout.inst_ebb(inst).expect("Instruction not in layout."), Some(inst), ), }; @@ -163,7 +160,11 @@ impl DominatorTree { ebb_b = layout.inst_ebb(idom).expect("Dominator got removed."); inst_b = Some(idom); } - if a == ebb_b { inst_b } else { None } + if a == ebb_b { + inst_b + } else { + None + } } /// Compute the common dominator of two basic blocks. @@ -418,14 +419,13 @@ impl DominatorTree { // Get an iterator with just the reachable, already visited predecessors to `ebb`. // Note that during the first pass, `rpo_number` is 1 for reachable blocks that haven't // been visited yet, 0 for unreachable blocks. - let mut reachable_preds = cfg.pred_iter(ebb).filter(|&(pred, _)| { - self.nodes[pred].rpo_number > 1 - }); + let mut reachable_preds = cfg.pred_iter(ebb) + .filter(|&(pred, _)| self.nodes[pred].rpo_number > 1); // The RPO must visit at least one predecessor before this node. - let mut idom = reachable_preds.next().expect( - "EBB node must have one reachable predecessor", - ); + let mut idom = reachable_preds + .next() + .expect("EBB node must have one reachable predecessor"); for pred in reachable_preds { idom = self.common_dominator(idom, pred, layout); @@ -450,11 +450,10 @@ impl DominatorTree { } // We use the RPO comparison on the postorder list so we invert the operands of the // comparison - let old_ebb_postorder_index = - self.postorder - .as_slice() - .binary_search_by(|probe| self.rpo_cmp_ebb(old_ebb, *probe)) - .expect("the old ebb is not declared to the dominator tree"); + let old_ebb_postorder_index = self.postorder + .as_slice() + .binary_search_by(|probe| self.rpo_cmp_ebb(old_ebb, *probe)) + .expect("the old ebb is not declared to the dominator tree"); let new_ebb_rpo = self.insert_after_rpo(old_ebb, old_ebb_postorder_index, new_ebb); self.nodes[new_ebb] = DomNode { rpo_number: new_ebb_rpo, @@ -471,11 +470,10 @@ impl DominatorTree { // If there is no gaps in RPo numbers to insert this new number, we iterate // forward in RPO numbers and backwards in the postorder list of EBBs, renumbering the Ebbs // until we find a gap - for (¤t_ebb, current_rpo) in - self.postorder[0..ebb_postorder_index].iter().rev().zip( - inserted_rpo_number + - 1.., - ) + for (¤t_ebb, current_rpo) in self.postorder[0..ebb_postorder_index] + .iter() + .rev() + .zip(inserted_rpo_number + 1..) { if self.nodes[current_ebb].rpo_number < current_rpo { // There is no gap, we renumber @@ -644,9 +642,8 @@ impl DominatorTreePreorder { { let a = a.into(); let b = b.into(); - self.pre_cmp_ebb(layout.pp_ebb(a), layout.pp_ebb(b)).then( - layout.cmp(a, b), - ) + self.pre_cmp_ebb(layout.pp_ebb(a), layout.pp_ebb(b)) + .then(layout.cmp(a, b)) } /// Compare two value defs according to the dominator tree pre-order. @@ -658,9 +655,8 @@ impl DominatorTreePreorder { pub fn pre_cmp_def(&self, a: Value, b: Value, func: &Function) -> Ordering { let da = func.dfg.value_def(a); let db = func.dfg.value_def(b); - self.pre_cmp(da, db, &func.layout).then_with( - || da.num().cmp(&db.num()), - ) + self.pre_cmp(da, db, &func.layout) + .then_with(|| da.num().cmp(&db.num())) } } diff --git a/lib/codegen/src/flowgraph.rs b/lib/codegen/src/flowgraph.rs index 66f377ce35..a954b73ac1 100644 --- a/lib/codegen/src/flowgraph.rs +++ b/lib/codegen/src/flowgraph.rs @@ -128,10 +128,9 @@ impl ControlFlowGraph { // our iteration over successors. let mut successors = mem::replace(&mut self.data[ebb].successors, Default::default()); for succ in successors.iter(&self.succ_forest) { - self.data[succ].predecessors.retain( - &mut self.pred_forest, - |_, &mut e| e != ebb, - ); + self.data[succ] + .predecessors + .retain(&mut self.pred_forest, |_, &mut e| e != ebb); } successors.clear(&mut self.succ_forest); } @@ -149,17 +148,12 @@ impl ControlFlowGraph { } fn add_edge(&mut self, from: BasicBlock, to: Ebb) { - self.data[from.0].successors.insert( - to, - &mut self.succ_forest, - &(), - ); - self.data[to].predecessors.insert( - from.1, - from.0, - &mut self.pred_forest, - &(), - ); + self.data[from.0] + .successors + .insert(to, &mut self.succ_forest, &()); + self.data[to] + .predecessors + .insert(from.1, from.0, &mut self.pred_forest, &()); } /// Get an iterator over the CFG predecessors to `ebb`. diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs index c2f07beb25..4fde255e82 100644 --- a/lib/codegen/src/ir/dfg.rs +++ b/lib/codegen/src/ir/dfg.rs @@ -166,9 +166,9 @@ impl DataFlowGraph { /// Get the type of a value. pub fn value_type(&self, v: Value) -> Type { match self.values[v] { - ValueData::Inst { ty, .. } | - ValueData::Param { ty, .. } | - ValueData::Alias { ty, .. } => ty, + ValueData::Inst { ty, .. } + | ValueData::Param { ty, .. } + | ValueData::Alias { ty, .. } => ty, } } @@ -235,11 +235,9 @@ impl DataFlowGraph { // This also avoids the creation of loops. let original = self.resolve_aliases(src); debug_assert_ne!( - dest, - original, + dest, original, "Aliasing {} to {} would create a loop", - dest, - src + dest, src ); let ty = self.value_type(original); debug_assert_eq!( @@ -267,8 +265,7 @@ impl DataFlowGraph { /// pub fn replace_with_aliases(&mut self, dest_inst: Inst, src_inst: Inst) { debug_assert_ne!( - dest_inst, - src_inst, + dest_inst, src_inst, "Replacing {} with itself would create a loop", dest_inst ); @@ -342,8 +339,7 @@ impl ValueDef { /// this value. pub fn num(self) -> usize { match self { - ValueDef::Result(_, n) | - ValueDef::Param(_, n) => n, + ValueDef::Result(_, n) | ValueDef::Param(_, n) => n, } } } @@ -574,9 +570,9 @@ impl DataFlowGraph { /// /// Panics if the instruction doesn't support arguments. pub fn append_inst_arg(&mut self, inst: Inst, new_arg: Value) { - let mut branch_values = self.insts[inst].take_value_list().expect( - "the instruction doesn't have value arguments", - ); + let mut branch_values = self.insts[inst] + .take_value_list() + .expect("the instruction doesn't have value arguments"); branch_values.push(new_arg, &mut self.value_lists); self.insts[inst].put_value_list(branch_values) } @@ -585,9 +581,9 @@ impl DataFlowGraph { /// /// This function panics if the instruction doesn't have any result. pub fn first_result(&self, inst: Inst) -> Value { - self.results[inst].first(&self.value_lists).expect( - "Instruction has no results", - ) + self.results[inst] + .first(&self.value_lists) + .expect("Instruction has no results") } /// Test if `inst` has any result values currently. @@ -653,9 +649,11 @@ impl DataFlowGraph { } else if constraints.requires_typevar_operand() { // Not all instruction formats have a designated operand, but in that case // `requires_typevar_operand()` should never be true. - self.value_type(self[inst].typevar_operand(&self.value_lists).expect( - "Instruction format doesn't have a designated operand, bad opcode.", - )) + self.value_type( + self[inst] + .typevar_operand(&self.value_lists) + .expect("Instruction format doesn't have a designated operand, bad opcode."), + ) } else { self.value_type(self.first_result(inst)) } @@ -721,13 +719,16 @@ impl DataFlowGraph { } else { panic!("{} must be an EBB parameter", val); }; - self.ebbs[ebb].params.swap_remove( - num as usize, - &mut self.value_lists, - ); + self.ebbs[ebb] + .params + .swap_remove(num as usize, &mut self.value_lists); if let Some(last_arg_val) = self.ebbs[ebb].params.get(num as usize, &self.value_lists) { // We update the position of the old last arg. - if let ValueData::Param { num: ref mut old_num, .. } = self.values[last_arg_val] { + if let ValueData::Param { + num: ref mut old_num, + .. + } = self.values[last_arg_val] + { *old_num = num; } else { panic!("{} should be an Ebb parameter", last_arg_val); @@ -744,27 +745,25 @@ impl DataFlowGraph { } else { panic!("{} must be an EBB parameter", val); }; - self.ebbs[ebb].params.remove( - num as usize, - &mut self.value_lists, - ); + self.ebbs[ebb] + .params + .remove(num as usize, &mut self.value_lists); for index in num..(self.num_ebb_params(ebb) as u16) { match self.values[self.ebbs[ebb] - .params - .get(index as usize, &self.value_lists) - .unwrap()] { + .params + .get(index as usize, &self.value_lists) + .unwrap()] + { ValueData::Param { ref mut num, .. } => { *num -= 1; } - _ => { - panic!( - "{} must be an EBB parameter", - self.ebbs[ebb] - .params - .get(index as usize, &self.value_lists) - .unwrap() - ) - } + _ => panic!( + "{} must be an EBB parameter", + self.ebbs[ebb] + .params + .get(index as usize, &self.value_lists) + .unwrap() + ), } } } @@ -835,7 +834,9 @@ struct EbbData { impl EbbData { fn new() -> Self { - Self { params: ValueList::new() } + Self { + params: ValueList::new(), + } } } @@ -878,9 +879,9 @@ impl DataFlowGraph { "this function is only for assigning types to previously invalid values" ); match self.values[v] { - ValueData::Inst { ref mut ty, .. } | - ValueData::Param { ref mut ty, .. } | - ValueData::Alias { ref mut ty, .. } => *ty = t, + ValueData::Inst { ref mut ty, .. } + | ValueData::Param { ref mut ty, .. } + | ValueData::Alias { ref mut ty, .. } => *ty = t, } } diff --git a/lib/codegen/src/ir/entities.rs b/lib/codegen/src/ir/entities.rs index 30c7cd3fb4..73e0984245 100644 --- a/lib/codegen/src/ir/entities.rs +++ b/lib/codegen/src/ir/entities.rs @@ -32,7 +32,11 @@ impl Ebb { /// /// This method is for use by the parser. pub fn with_number(n: u32) -> Option { - if n < u32::MAX { Some(Ebb(n)) } else { None } + if n < u32::MAX { + Some(Ebb(n)) + } else { + None + } } } @@ -124,7 +128,11 @@ impl FuncRef { /// /// This method is for use by the parser. pub fn with_number(n: u32) -> Option { - if n < u32::MAX { Some(FuncRef(n)) } else { None } + if n < u32::MAX { + Some(FuncRef(n)) + } else { + None + } } } @@ -138,7 +146,11 @@ impl SigRef { /// /// This method is for use by the parser. pub fn with_number(n: u32) -> Option { - if n < u32::MAX { Some(SigRef(n)) } else { None } + if n < u32::MAX { + Some(SigRef(n)) + } else { + None + } } } @@ -152,7 +164,11 @@ impl Heap { /// /// This method is for use by the parser. pub fn with_number(n: u32) -> Option { - if n < u32::MAX { Some(Heap(n)) } else { None } + if n < u32::MAX { + Some(Heap(n)) + } else { + None + } } } diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs index 95f86be8fd..a4abc8bfc4 100644 --- a/lib/codegen/src/ir/extfunc.rs +++ b/lib/codegen/src/ir/extfunc.rs @@ -384,8 +384,7 @@ mod tests { CallConv::SystemV, CallConv::WindowsFastcall, CallConv::Baldrdash, - ] - { + ] { assert_eq!(Ok(cc), cc.to_string().parse()) } } diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs index 10350c506c..d15e539214 100644 --- a/lib/codegen/src/ir/function.rs +++ b/lib/codegen/src/ir/function.rs @@ -10,7 +10,7 @@ use ir::{DataFlowGraph, ExternalName, Layout, Signature}; use ir::{Ebb, ExtFuncData, FuncRef, GlobalVar, GlobalVarData, Heap, HeapData, JumpTable, JumpTableData, SigRef, StackSlot, StackSlotData}; use ir::{EbbOffsets, InstEncodings, JumpTables, SourceLocs, StackSlots, ValueLocations}; -use isa::{EncInfo, Legalize, TargetIsa, Encoding}; +use isa::{EncInfo, Encoding, Legalize, TargetIsa}; use settings::CallConv; use std::fmt; use write::write_function; @@ -151,9 +151,9 @@ impl Function { /// Returns the value of the last `purpose` parameter, or `None` if no such parameter exists. pub fn special_param(&self, purpose: ir::ArgumentPurpose) -> Option { let entry = self.layout.entry_block().expect("Function is empty"); - self.signature.special_param_index(purpose).map(|i| { - self.dfg.ebb_params(entry)[i] - }) + self.signature + .special_param_index(purpose) + .map(|i| self.dfg.ebb_params(entry)[i]) } /// Get an iterator over the instructions in `ebb`, including offsets and encoded instruction diff --git a/lib/codegen/src/ir/immediates.rs b/lib/codegen/src/ir/immediates.rs index 32b64dabf8..fbc2637d0a 100644 --- a/lib/codegen/src/ir/immediates.rs +++ b/lib/codegen/src/ir/immediates.rs @@ -192,10 +192,12 @@ impl FromStr for Uimm32 { // Parse a decimal or hexadecimal `Uimm32`, formatted as above. fn from_str(s: &str) -> Result { - parse_i64(s).and_then(|x| if 0 <= x && x <= i64::from(u32::MAX) { - Ok(Uimm32(x as u32)) - } else { - Err("Uimm32 out of range") + parse_i64(s).and_then(|x| { + if 0 <= x && x <= i64::from(u32::MAX) { + Ok(Uimm32(x as u32)) + } else { + Err("Uimm32 out of range") + } }) } } @@ -259,12 +261,12 @@ impl FromStr for Offset32 { if !(s.starts_with('-') || s.starts_with('+')) { return Err("Offset must begin with sign"); } - parse_i64(s).and_then(|x| if i64::from(i32::MIN) <= x && - x <= i64::from(i32::MAX) - { - Ok(Self::new(x as i32)) - } else { - Err("Offset out of range") + parse_i64(s).and_then(|x| { + if i64::from(i32::MIN) <= x && x <= i64::from(i32::MAX) { + Ok(Self::new(x as i32)) + } else { + Err("Offset out of range") + } }) } } @@ -447,18 +449,16 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result { Err(_) => return Err("Bad exponent"), } } - _ => { - match ch.to_digit(16) { - Some(digit) => { - digits += 1; - if digits > 16 { - return Err("Too many digits"); - } - significand = (significand << 4) | u64::from(digit); + _ => match ch.to_digit(16) { + Some(digit) => { + digits += 1; + if digits > 16 { + return Err("Too many digits"); } - None => return Err("Invalid character"), + significand = (significand << 4) | u64::from(digit); } - } + None => return Err("Invalid character"), + }, } } @@ -546,9 +546,7 @@ impl Ieee32 { let n = n.into(); debug_assert!(n < 32); debug_assert!(23 + 1 - n < 32); - Self::with_bits( - (1u32 << (32 - 1)) | Self::pow2(n - 1).0 | (1u32 << (23 + 1 - n)), - ) + Self::with_bits((1u32 << (32 - 1)) | Self::pow2(n - 1).0 | (1u32 << (23 + 1 - n))) } /// Return self negated. @@ -609,9 +607,7 @@ impl Ieee64 { let n = n.into(); debug_assert!(n < 64); debug_assert!(52 + 1 - n < 64); - Self::with_bits( - (1u64 << (64 - 1)) | Self::pow2(n - 1).0 | (1u64 << (52 + 1 - n)), - ) + Self::with_bits((1u64 << (64 - 1)) | Self::pow2(n - 1).0 | (1u64 << (52 + 1 - n))) } /// Return self negated. diff --git a/lib/codegen/src/ir/instructions.rs b/lib/codegen/src/ir/instructions.rs index e1db66099b..32b4681b8b 100644 --- a/lib/codegen/src/ir/instructions.rs +++ b/lib/codegen/src/ir/instructions.rs @@ -178,13 +178,13 @@ impl InstructionData { destination, ref args, .. - } | - InstructionData::BranchFloat { + } + | InstructionData::BranchFloat { destination, ref args, .. - } | - InstructionData::Branch { + } + | InstructionData::Branch { destination, ref args, .. @@ -208,11 +208,11 @@ impl InstructionData { /// Multi-destination branches like `br_table` return `None`. pub fn branch_destination(&self) -> Option { match *self { - InstructionData::Jump { destination, .. } | - InstructionData::Branch { destination, .. } | - InstructionData::BranchInt { destination, .. } | - InstructionData::BranchFloat { destination, .. } | - InstructionData::BranchIcmp { destination, .. } => Some(destination), + InstructionData::Jump { destination, .. } + | InstructionData::Branch { destination, .. } + | InstructionData::BranchInt { destination, .. } + | InstructionData::BranchFloat { destination, .. } + | InstructionData::BranchIcmp { destination, .. } => Some(destination), InstructionData::BranchTable { .. } => None, _ => { debug_assert!(!self.opcode().is_branch()); @@ -227,11 +227,26 @@ impl InstructionData { /// Multi-destination branches like `br_table` return `None`. pub fn branch_destination_mut(&mut self) -> Option<&mut Ebb> { match *self { - InstructionData::Jump { ref mut destination, .. } | - InstructionData::Branch { ref mut destination, .. } | - InstructionData::BranchInt { ref mut destination, .. } | - InstructionData::BranchFloat { ref mut destination, .. } | - InstructionData::BranchIcmp { ref mut destination, .. } => Some(destination), + InstructionData::Jump { + ref mut destination, + .. + } + | InstructionData::Branch { + ref mut destination, + .. + } + | InstructionData::BranchInt { + ref mut destination, + .. + } + | InstructionData::BranchFloat { + ref mut destination, + .. + } + | InstructionData::BranchIcmp { + ref mut destination, + .. + } => Some(destination), InstructionData::BranchTable { .. } => None, _ => { debug_assert!(!self.opcode().is_branch()); @@ -245,12 +260,12 @@ impl InstructionData { /// Any instruction that can call another function reveals its call signature here. pub fn analyze_call<'a>(&'a self, pool: &'a ValueListPool) -> CallInfo<'a> { match *self { - InstructionData::Call { func_ref, ref args, .. } => { - CallInfo::Direct(func_ref, args.as_slice(pool)) - } - InstructionData::CallIndirect { sig_ref, ref args, .. } => { - CallInfo::Indirect(sig_ref, &args.as_slice(pool)[1..]) - } + InstructionData::Call { + func_ref, ref args, .. + } => CallInfo::Direct(func_ref, args.as_slice(pool)), + InstructionData::CallIndirect { + sig_ref, ref args, .. + } => CallInfo::Indirect(sig_ref, &args.as_slice(pool)[1..]), _ => { debug_assert!(!self.opcode().is_call()); CallInfo::NotACall @@ -512,12 +527,16 @@ impl OperandConstraint { LaneOf => Bound(ctrl_type.lane_type()), AsBool => Bound(ctrl_type.as_bool()), HalfWidth => Bound(ctrl_type.half_width().expect("invalid type for half_width")), - DoubleWidth => Bound(ctrl_type.double_width().expect( - "invalid type for double_width", - )), - HalfVector => Bound(ctrl_type.half_vector().expect( - "invalid type for half_vector", - )), + DoubleWidth => Bound( + ctrl_type + .double_width() + .expect("invalid type for double_width"), + ), + HalfVector => Bound( + ctrl_type + .half_vector() + .expect("invalid type for half_vector"), + ), DoubleVector => Bound(ctrl_type.by(2).expect("invalid type for double_vector")), } } diff --git a/lib/codegen/src/ir/jumptable.rs b/lib/codegen/src/ir/jumptable.rs index 90b18b1bfe..17706d5959 100644 --- a/lib/codegen/src/ir/jumptable.rs +++ b/lib/codegen/src/ir/jumptable.rs @@ -90,9 +90,9 @@ impl JumpTableData { /// Checks if any of the entries branch to `ebb`. pub fn branches_to(&self, ebb: Ebb) -> bool { - self.table.iter().any(|target_ebb| { - target_ebb.expand() == Some(ebb) - }) + self.table + .iter() + .any(|target_ebb| target_ebb.expand() == Some(ebb)) } /// Access the whole table as a mutable slice. diff --git a/lib/codegen/src/ir/layout.rs b/lib/codegen/src/ir/layout.rs index 2e3ac89903..1e1110a3d8 100644 --- a/lib/codegen/src/ir/layout.rs +++ b/lib/codegen/src/ir/layout.rs @@ -90,7 +90,11 @@ fn midpoint(a: SequenceNumber, b: SequenceNumber) -> Option { debug_assert!(a < b); // Avoid integer overflow. let m = a + (b - a) / 2; - if m > a { Some(m) } else { None } + if m > a { + Some(m) + } else { + None + } } #[test] @@ -178,9 +182,8 @@ impl Layout { /// Assign a valid sequence number to `inst` such that the numbers are still monotonic. This may /// require renumbering. fn assign_inst_seq(&mut self, inst: Inst) { - let ebb = self.inst_ebb(inst).expect( - "inst must be inserted before assigning an seq", - ); + let ebb = self.inst_ebb(inst) + .expect("inst must be inserted before assigning an seq"); // Get the sequence number immediately before `inst`. let prev_seq = match self.insts[inst].prev.expand() { @@ -566,9 +569,8 @@ impl Layout { /// Insert `inst` before the instruction `before` in the same EBB. pub fn insert_inst(&mut self, inst: Inst, before: Inst) { debug_assert_eq!(self.inst_ebb(inst), None); - let ebb = self.inst_ebb(before).expect( - "Instruction before insertion point not in the layout", - ); + let ebb = self.inst_ebb(before) + .expect("Instruction before insertion point not in the layout"); let after = self.insts[before].prev; { let inst_node = &mut self.insts[inst]; @@ -641,9 +643,8 @@ impl Layout { /// i4 /// ``` pub fn split_ebb(&mut self, new_ebb: Ebb, before: Inst) { - let old_ebb = self.inst_ebb(before).expect( - "The `before` instruction must be in the layout", - ); + let old_ebb = self.inst_ebb(before) + .expect("The `before` instruction must be in the layout"); debug_assert!(!self.is_ebb_inserted(new_ebb)); // Insert new_ebb after old_ebb. diff --git a/lib/codegen/src/ir/libcall.rs b/lib/codegen/src/ir/libcall.rs index 8b8e2a0599..26440d4235 100644 --- a/lib/codegen/src/ir/libcall.rs +++ b/lib/codegen/src/ir/libcall.rs @@ -1,9 +1,9 @@ //! Naming well-known routines in the runtime library. -use ir::{types, Opcode, Type, Inst, Function, FuncRef, ExternalName, Signature, AbiParam, - ExtFuncData, ArgumentPurpose}; +use ir::{types, AbiParam, ArgumentPurpose, ExtFuncData, ExternalName, FuncRef, Function, Inst, + Opcode, Signature, Type}; +use isa::{RegUnit, TargetIsa}; use settings::CallConv; -use isa::{TargetIsa, RegUnit}; use std::fmt; use std::str::FromStr; @@ -82,24 +82,20 @@ impl LibCall { /// Returns `None` if no well-known library routine name exists for that instruction. pub fn for_inst(opcode: Opcode, ctrl_type: Type) -> Option { Some(match ctrl_type { - types::F32 => { - match opcode { - Opcode::Ceil => LibCall::CeilF32, - Opcode::Floor => LibCall::FloorF32, - Opcode::Trunc => LibCall::TruncF32, - Opcode::Nearest => LibCall::NearestF32, - _ => return None, - } - } - types::F64 => { - match opcode { - Opcode::Ceil => LibCall::CeilF64, - Opcode::Floor => LibCall::FloorF64, - Opcode::Trunc => LibCall::TruncF64, - Opcode::Nearest => LibCall::NearestF64, - _ => return None, - } - } + types::F32 => match opcode { + Opcode::Ceil => LibCall::CeilF32, + Opcode::Floor => LibCall::FloorF32, + Opcode::Trunc => LibCall::TruncF32, + Opcode::Nearest => LibCall::NearestF32, + _ => return None, + }, + types::F64 => match opcode { + Opcode::Ceil => LibCall::CeilF64, + Opcode::Floor => LibCall::FloorF64, + Opcode::Trunc => LibCall::TruncF64, + Opcode::Nearest => LibCall::NearestF64, + _ => return None, + }, _ => return None, }) } @@ -127,9 +123,8 @@ pub fn get_probestack_funcref( arg_reg: RegUnit, isa: &TargetIsa, ) -> FuncRef { - find_funcref(LibCall::Probestack, func).unwrap_or_else(|| { - make_funcref_for_probestack(func, reg_type, arg_reg, isa) - }) + find_funcref(LibCall::Probestack, func) + .unwrap_or_else(|| make_funcref_for_probestack(func, reg_type, arg_reg, isa)) } /// Get the existing function reference for `libcall` in `func` if it exists. diff --git a/lib/codegen/src/ir/mod.rs b/lib/codegen/src/ir/mod.rs index 3ee7dc9176..daca83cc3a 100644 --- a/lib/codegen/src/ir/mod.rs +++ b/lib/codegen/src/ir/mod.rs @@ -33,7 +33,7 @@ pub use ir::heap::{HeapBase, HeapData, HeapStyle}; pub use ir::instructions::{InstructionData, Opcode, ValueList, ValueListPool, VariableArgs}; pub use ir::jumptable::JumpTableData; pub use ir::layout::Layout; -pub use ir::libcall::{LibCall, get_libcall_funcref, get_probestack_funcref}; +pub use ir::libcall::{get_libcall_funcref, get_probestack_funcref, LibCall}; pub use ir::memflags::MemFlags; pub use ir::progpoint::{ExpandedProgramPoint, ProgramOrder, ProgramPoint}; pub use ir::sourceloc::SourceLoc; diff --git a/lib/codegen/src/ir/valueloc.rs b/lib/codegen/src/ir/valueloc.rs index 7f68e09792..e4762c778f 100644 --- a/lib/codegen/src/ir/valueloc.rs +++ b/lib/codegen/src/ir/valueloc.rs @@ -66,12 +66,10 @@ impl<'a> fmt::Display for DisplayValueLoc<'a> { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match self.0 { ValueLoc::Unassigned => write!(f, "-"), - ValueLoc::Reg(ru) => { - match self.1 { - Some(regs) => write!(f, "{}", regs.display_regunit(ru)), - None => write!(f, "%{}", ru), - } - } + ValueLoc::Reg(ru) => match self.1 { + Some(regs) => write!(f, "{}", regs.display_regunit(ru)), + None => write!(f, "%{}", ru), + }, ValueLoc::Stack(ss) => write!(f, "{}", ss), } } @@ -153,12 +151,10 @@ impl<'a> fmt::Display for DisplayArgumentLoc<'a> { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match self.0 { ArgumentLoc::Unassigned => write!(f, "-"), - ArgumentLoc::Reg(ru) => { - match self.1 { - Some(regs) => write!(f, "{}", regs.display_regunit(ru)), - None => write!(f, "%{}", ru), - } - } + ArgumentLoc::Reg(ru) => match self.1 { + Some(regs) => write!(f, "{}", regs.display_regunit(ru)), + None => write!(f, "%{}", ru), + }, ArgumentLoc::Stack(offset) => write!(f, "{}", offset), } } diff --git a/lib/codegen/src/isa/arm64/abi.rs b/lib/codegen/src/isa/arm64/abi.rs index 4bd9aad7be..79b7a97cd7 100644 --- a/lib/codegen/src/isa/arm64/abi.rs +++ b/lib/codegen/src/isa/arm64/abi.rs @@ -17,7 +17,11 @@ pub fn legalize_signature( /// Get register class for a type appearing in a legalized signature. pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { - if ty.is_int() { GPR } else { FPR } + if ty.is_int() { + GPR + } else { + FPR + } } /// Get the set of allocatable registers for `func`. diff --git a/lib/codegen/src/isa/constraints.rs b/lib/codegen/src/isa/constraints.rs index b9a4838a9d..7c5da6032d 100644 --- a/lib/codegen/src/isa/constraints.rs +++ b/lib/codegen/src/isa/constraints.rs @@ -30,16 +30,14 @@ impl OperandConstraint { /// counterpart operand has the same value location. pub fn satisfied(&self, loc: ValueLoc) -> bool { match self.kind { - ConstraintKind::Reg | - ConstraintKind::Tied(_) => { + ConstraintKind::Reg | ConstraintKind::Tied(_) => { if let ValueLoc::Reg(reg) = loc { self.regclass.contains(reg) } else { false } } - ConstraintKind::FixedReg(reg) | - ConstraintKind::FixedTied(reg) => { + ConstraintKind::FixedReg(reg) | ConstraintKind::FixedTied(reg) => { loc == ValueLoc::Reg(reg) && self.regclass.contains(reg) } ConstraintKind::Stack => { diff --git a/lib/codegen/src/isa/encoding.rs b/lib/codegen/src/isa/encoding.rs index c13c4de40d..6bb7e30aec 100644 --- a/lib/codegen/src/isa/encoding.rs +++ b/lib/codegen/src/isa/encoding.rs @@ -122,10 +122,9 @@ impl EncInfo { /// /// Returns 0 for illegal encodings. pub fn bytes(&self, enc: Encoding) -> CodeOffset { - self.sizing.get(enc.recipe()).map_or( - 0, - |s| CodeOffset::from(s.bytes), - ) + self.sizing + .get(enc.recipe()) + .map_or(0, |s| CodeOffset::from(s.bytes)) } /// Get the branch range that is supported by `enc`, if any. diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs index 7b21e68d28..d9f4851e68 100644 --- a/lib/codegen/src/isa/mod.rs +++ b/lib/codegen/src/isa/mod.rs @@ -142,11 +142,8 @@ impl settings::Configurable for Builder { /// legalize it? /// /// The `Encodings` iterator returns a legalization function to call. -pub type Legalize = fn(ir::Inst, - &mut ir::Function, - &mut flowgraph::ControlFlowGraph, - &TargetIsa) - -> bool; +pub type Legalize = + fn(ir::Inst, &mut ir::Function, &mut flowgraph::ControlFlowGraph, &TargetIsa) -> bool; /// Methods that are specialized to a target ISA. Implies a Display trait that shows the /// shared flags, as well as any isa-specific flags. diff --git a/lib/codegen/src/isa/registers.rs b/lib/codegen/src/isa/registers.rs index 7defead608..a78dfac60c 100644 --- a/lib/codegen/src/isa/registers.rs +++ b/lib/codegen/src/isa/registers.rs @@ -89,10 +89,12 @@ impl RegBank { None } } - }.and_then(|offset| if offset < self.units { - Some(offset + self.first_unit) - } else { - None + }.and_then(|offset| { + if offset < self.units { + Some(offset + self.first_unit) + } else { + None + } }) } diff --git a/lib/codegen/src/isa/riscv/abi.rs b/lib/codegen/src/isa/riscv/abi.rs index ebc4a67fab..f45dd31426 100644 --- a/lib/codegen/src/isa/riscv/abi.rs +++ b/lib/codegen/src/isa/riscv/abi.rs @@ -117,17 +117,21 @@ pub fn legalize_signature( /// Get register class for a type appearing in a legalized signature. pub fn regclass_for_abi_type(ty: Type) -> RegClass { - if ty.is_float() { FPR } else { GPR } + if ty.is_float() { + FPR + } else { + GPR + } } pub fn allocatable_registers(_func: &ir::Function, isa_flags: &settings::Flags) -> RegisterSet { let mut regs = RegisterSet::new(); regs.take(GPR, GPR.unit(0)); // Hard-wired 0. - // %x1 is the link register which is available for allocation. + // %x1 is the link register which is available for allocation. regs.take(GPR, GPR.unit(2)); // Stack pointer. regs.take(GPR, GPR.unit(3)); // Global pointer. regs.take(GPR, GPR.unit(4)); // Thread pointer. - // TODO: %x8 is the frame pointer. Reserve it? + // TODO: %x8 is the frame pointer. Reserve it? // Remove %x16 and up for RV32E. if isa_flags.enable_e() { diff --git a/lib/codegen/src/isa/stack.rs b/lib/codegen/src/isa/stack.rs index 91df090cb4..eefa523ae3 100644 --- a/lib/codegen/src/isa/stack.rs +++ b/lib/codegen/src/isa/stack.rs @@ -35,9 +35,9 @@ impl StackRef { /// Get a reference to `ss` using the stack pointer as a base. pub fn sp(ss: StackSlot, frame: &StackSlots) -> Self { - let size = frame.frame_size.expect( - "Stack layout must be computed before referencing stack slots", - ); + let size = frame + .frame_size + .expect("Stack layout must be computed before referencing stack slots"); let slot = &frame[ss]; let offset = if slot.kind == StackSlotKind::OutgoingArg { // Outgoing argument slots have offsets relative to our stack pointer. diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs index 15b7d94654..e7511482ca 100644 --- a/lib/codegen/src/isa/x86/abi.rs +++ b/lib/codegen/src/isa/x86/abi.rs @@ -6,8 +6,8 @@ use cursor::{Cursor, CursorPosition, EncCursor}; use ir; use ir::immediates::Imm64; use ir::stackslot::{StackOffset, StackSize}; -use ir::{AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, InstBuilder, ValueLoc, - get_probestack_funcref}; +use ir::{get_probestack_funcref, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, + InstBuilder, ValueLoc}; use isa::{RegClass, RegUnit, TargetIsa}; use regalloc::RegisterSet; use result; @@ -97,7 +97,8 @@ impl ArgAssigner for Args { RU::r14 } else { RU::rsi - } as RegUnit).into() + } as RegUnit) + .into() } // This is SpiderMonkey's `WasmTableCallSigReg`. ArgumentPurpose::SignatureId => return ArgumentLoc::Reg(RU::rbx as RegUnit).into(), @@ -235,8 +236,8 @@ fn callee_saved_gprs_used(flags: &shared_settings::Flags, func: &ir::Function) - for ebb in &func.layout { for inst in func.layout.ebb_insts(ebb) { match func.dfg[inst] { - ir::instructions::InstructionData::RegMove { dst, .. } | - ir::instructions::InstructionData::RegFill { dst, .. } => { + ir::instructions::InstructionData::RegMove { dst, .. } + | ir::instructions::InstructionData::RegFill { dst, .. } => { if !used.is_avail(GPR, dst) { used.free(GPR, dst); } @@ -431,10 +432,8 @@ fn insert_common_prologue( pos.func.locations[fp] = ir::ValueLoc::Reg(RU::rbp as RegUnit); pos.ins().x86_push(fp); - pos.ins().copy_special( - RU::rsp as RegUnit, - RU::rbp as RegUnit, - ); + pos.ins() + .copy_special(RU::rsp as RegUnit, RU::rbp as RegUnit); for reg in csrs.iter(GPR) { // Append param to entry EBB @@ -449,8 +448,8 @@ fn insert_common_prologue( // Allocate stack frame storage. if stack_size > 0 { - if isa.flags().probestack_enabled() && - stack_size > (1 << isa.flags().probestack_size_log2()) + if isa.flags().probestack_enabled() + && stack_size > (1 << isa.flags().probestack_size_log2()) { // Emit a stack probe. let rax = RU::rax as RegUnit; @@ -464,8 +463,8 @@ fn insert_common_prologue( let callee = get_probestack_funcref(pos.func, reg_type, rax, isa); // Make the call. - let call = if !isa.flags().is_pic() && isa.flags().is_64bit() && - !pos.func.dfg.ext_funcs[callee].colocated + let call = if !isa.flags().is_pic() && isa.flags().is_64bit() + && !pos.func.dfg.ext_funcs[callee].colocated { // 64-bit non-PIC non-colocated calls need to be legalized to call_indirect. // Use r11 as it may be clobbered under all supported calling conventions. diff --git a/lib/codegen/src/isa/x86/enc_tables.rs b/lib/codegen/src/isa/x86/enc_tables.rs index 01c9ede2f0..4c964ad802 100644 --- a/lib/codegen/src/isa/x86/enc_tables.rs +++ b/lib/codegen/src/isa/x86/enc_tables.rs @@ -84,11 +84,8 @@ fn expand_sdivrem( // Explicitly check for overflow: Trap when x == INT_MIN. debug_assert!(avoid_div_traps, "Native trapping divide handled above"); let f = pos.ins().ifcmp_imm(x, -1 << (ty.lane_bits() - 1)); - pos.ins().trapif( - IntCC::Equal, - f, - ir::TrapCode::IntegerOverflow, - ); + pos.ins() + .trapif(IntCC::Equal, f, ir::TrapCode::IntegerOverflow); // x / -1 = -x. pos.ins().irsub_imm(x, 0) }; @@ -348,11 +345,8 @@ fn expand_fcvt_to_sint( let mut pos = FuncCursor::new(func).after_inst(inst); pos.use_srcloc(inst); - let is_done = pos.ins().icmp_imm( - IntCC::NotEqual, - result, - 1 << (ty.lane_bits() - 1), - ); + let is_done = pos.ins() + .icmp_imm(IntCC::NotEqual, result, 1 << (ty.lane_bits() - 1)); pos.ins().brnz(is_done, done, &[]); // We now have the following possibilities: @@ -364,10 +358,8 @@ fn expand_fcvt_to_sint( // Check for NaN. let is_nan = pos.ins().fcmp(FloatCC::Unordered, x, x); - pos.ins().trapnz( - is_nan, - ir::TrapCode::BadConversionToInteger, - ); + pos.ins() + .trapnz(is_nan, ir::TrapCode::BadConversionToInteger); // Check for case 1: INT_MIN is the correct result. // Determine the smallest floating point number that would convert to INT_MIN. @@ -376,14 +368,12 @@ fn expand_fcvt_to_sint( let flimit = match xty { // An f32 can represent `i16::min_value() - 1` exactly with precision to spare, so // there are values less than -2^(N-1) that convert correctly to INT_MIN. - ir::types::F32 => { - pos.ins().f32const(if output_bits < 32 { - overflow_cc = FloatCC::LessThanOrEqual; - Ieee32::fcvt_to_sint_negative_overflow(output_bits) - } else { - Ieee32::pow2(output_bits - 1).neg() - }) - } + ir::types::F32 => pos.ins().f32const(if output_bits < 32 { + overflow_cc = FloatCC::LessThanOrEqual; + Ieee32::fcvt_to_sint_negative_overflow(output_bits) + } else { + Ieee32::pow2(output_bits - 1).neg() + }), ir::types::F64 => { // An f64 can represent `i32::min_value() - 1` exactly with precision to spare, so // there are values less than -2^(N-1) that convert correctly to INT_MIN. @@ -458,12 +448,8 @@ fn expand_fcvt_to_uint( _ => panic!("Can't convert {}", xty), }; let is_large = pos.ins().ffcmp(x, pow2nm1); - pos.ins().brff( - FloatCC::GreaterThanOrEqual, - is_large, - large, - &[], - ); + pos.ins() + .brff(FloatCC::GreaterThanOrEqual, is_large, large, &[]); // We need to generate a specific trap code when `x` is NaN, so reuse the flags from the // previous comparison. @@ -476,12 +462,8 @@ fn expand_fcvt_to_uint( // Now we know that x < 2^(N-1) and not NaN. let sres = pos.ins().x86_cvtt2si(ty, x); let is_neg = pos.ins().ifcmp_imm(sres, 0); - pos.ins().brif( - IntCC::SignedGreaterThanOrEqual, - is_neg, - done, - &[sres], - ); + pos.ins() + .brif(IntCC::SignedGreaterThanOrEqual, is_neg, done, &[sres]); pos.ins().trap(ir::TrapCode::IntegerOverflow); // Handle the case where x >= 2^(N-1) and not NaN. @@ -489,11 +471,8 @@ fn expand_fcvt_to_uint( let adjx = pos.ins().fsub(x, pow2nm1); let lres = pos.ins().x86_cvtt2si(ty, adjx); let is_neg = pos.ins().ifcmp_imm(lres, 0); - pos.ins().trapif( - IntCC::SignedLessThan, - is_neg, - ir::TrapCode::IntegerOverflow, - ); + pos.ins() + .trapif(IntCC::SignedLessThan, is_neg, ir::TrapCode::IntegerOverflow); let lfinal = pos.ins().iadd_imm(lres, 1 << (ty.lane_bits() - 1)); // Recycle the original instruction as a jump. diff --git a/lib/codegen/src/iterators.rs b/lib/codegen/src/iterators.rs index 0524343028..220b90da0c 100644 --- a/lib/codegen/src/iterators.rs +++ b/lib/codegen/src/iterators.rs @@ -87,8 +87,7 @@ mod tests { vec![] ); assert_eq!( - [] - .iter() + [].iter() .cloned() .adjacent_pairs() .collect::>(), diff --git a/lib/codegen/src/legalizer/boundary.rs b/lib/codegen/src/legalizer/boundary.rs index 61f07884ca..3c0a198f7f 100644 --- a/lib/codegen/src/legalizer/boundary.rs +++ b/lib/codegen/src/legalizer/boundary.rs @@ -133,8 +133,7 @@ fn legalize_entry_params(func: &mut Function, entry: Ebb) { } // The callee-save parameters should not appear until after register allocation is // done. - ArgumentPurpose::FramePointer | - ArgumentPurpose::CalleeSaved => { + ArgumentPurpose::FramePointer | ArgumentPurpose::CalleeSaved => { panic!("Premature callee-saved arg {}", arg); } // These can be meaningfully added by `legalize_signature()`. @@ -174,9 +173,8 @@ fn legalize_inst_results(pos: &mut FuncCursor, mut get_abi_type: ResTyp where ResType: FnMut(&Function, usize) -> AbiParam, { - let call = pos.current_inst().expect( - "Cursor must point to a call instruction", - ); + let call = pos.current_inst() + .expect("Cursor must point to a call instruction"); // We theoretically allow for call instructions that return a number of fixed results before // the call return values. In practice, it doesn't happen. @@ -377,8 +375,8 @@ fn check_call_signature(dfg: &DataFlowGraph, inst: Inst) -> Result<(), SigRef> { }; let sig = &dfg.signatures[sig_ref]; - if check_arg_types(dfg, args, &sig.params[..]) && - check_arg_types(dfg, dfg.inst_results(inst), &sig.returns[..]) + if check_arg_types(dfg, args, &sig.params[..]) + && check_arg_types(dfg, dfg.inst_results(inst), &sig.returns[..]) { // All types check out. Ok(()) @@ -407,14 +405,13 @@ fn legalize_inst_arguments( ) where ArgType: FnMut(&Function, usize) -> AbiParam, { - let inst = pos.current_inst().expect( - "Cursor must point to a call instruction", - ); + let inst = pos.current_inst() + .expect("Cursor must point to a call instruction"); // Lift the value list out of the call instruction so we modify it. - let mut vlist = pos.func.dfg[inst].take_value_list().expect( - "Call must have a value list", - ); + let mut vlist = pos.func.dfg[inst] + .take_value_list() + .expect("Call must have a value list"); // The value list contains all arguments to the instruction, including the callee on an // indirect call which isn't part of the call arguments that must match the ABI signature. @@ -544,8 +541,8 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph .iter() .rev() .take_while(|&rt| { - rt.purpose == ArgumentPurpose::Link || rt.purpose == ArgumentPurpose::StructReturn || - rt.purpose == ArgumentPurpose::VMContext + rt.purpose == ArgumentPurpose::Link || rt.purpose == ArgumentPurpose::StructReturn + || rt.purpose == ArgumentPurpose::VMContext }) .count(); let abi_args = func.signature.returns.len() - special_args; @@ -570,9 +567,9 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph let mut vlist = pos.func.dfg[inst].take_value_list().unwrap(); for arg in &pos.func.signature.returns[abi_args..] { match arg.purpose { - ArgumentPurpose::Link | - ArgumentPurpose::StructReturn | - ArgumentPurpose::VMContext => {} + ArgumentPurpose::Link + | ArgumentPurpose::StructReturn + | ArgumentPurpose::VMContext => {} ArgumentPurpose::Normal => panic!("unexpected return value {}", arg), _ => panic!("Unsupported special purpose return value {}", arg), } @@ -587,10 +584,9 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph .expect("No matching special purpose argument."); // Get the corresponding entry block value and add it to the return instruction's // arguments. - let val = pos.func.dfg.ebb_params( - pos.func.layout.entry_block().unwrap(), - ) - [idx]; + let val = pos.func + .dfg + .ebb_params(pos.func.layout.entry_block().unwrap())[idx]; debug_assert_eq!(pos.func.dfg.value_type(val), arg.value_type); vlist.push(val, &mut pos.func.dfg.value_lists); } @@ -630,12 +626,12 @@ fn spill_entry_params(func: &mut Function, entry: Ebb) { /// or calls between writing the stack slots and the call instruction. Writing the slots earlier /// could help reduce register pressure before the call. fn spill_call_arguments(pos: &mut FuncCursor) -> bool { - let inst = pos.current_inst().expect( - "Cursor must point to a call instruction", - ); - let sig_ref = pos.func.dfg.call_signature(inst).expect( - "Call instruction expected.", - ); + let inst = pos.current_inst() + .expect("Cursor must point to a call instruction"); + let sig_ref = pos.func + .dfg + .call_signature(inst) + .expect("Call instruction expected."); // Start by building a list of stack slots and arguments to be replaced. // This requires borrowing `pos.func.dfg`, so we can't change anything. diff --git a/lib/codegen/src/legalizer/call.rs b/lib/codegen/src/legalizer/call.rs index ed499f53f3..d2e0ca542e 100644 --- a/lib/codegen/src/legalizer/call.rs +++ b/lib/codegen/src/legalizer/call.rs @@ -51,10 +51,7 @@ pub fn expand_call( ); } - func.dfg.replace(inst).CallIndirect( - ir::Opcode::CallIndirect, - ptr_ty, - sig, - new_args, - ); + func.dfg + .replace(inst) + .CallIndirect(ir::Opcode::CallIndirect, ptr_ty, sig, new_args); } diff --git a/lib/codegen/src/legalizer/globalvar.rs b/lib/codegen/src/legalizer/globalvar.rs index 9c6e701cf2..d8fd0b94b7 100644 --- a/lib/codegen/src/legalizer/globalvar.rs +++ b/lib/codegen/src/legalizer/globalvar.rs @@ -34,9 +34,8 @@ pub fn expand_global_addr( /// Expand a `global_addr` instruction for a vmctx global. fn vmctx_addr(inst: ir::Inst, func: &mut ir::Function, offset: i64) { // Get the value representing the `vmctx` argument. - let vmctx = func.special_param(ir::ArgumentPurpose::VMContext).expect( - "Missing vmctx parameter", - ); + let vmctx = func.special_param(ir::ArgumentPurpose::VMContext) + .expect("Missing vmctx parameter"); // Simply replace the `global_addr` instruction with an `iadd_imm`, reusing the result value. func.dfg.replace(inst).iadd_imm(vmctx, offset); diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs index b3a595833f..3b0b5dfd1c 100644 --- a/lib/codegen/src/legalizer/heap.rs +++ b/lib/codegen/src/legalizer/heap.rs @@ -67,30 +67,21 @@ fn dynamic_addr( let oob; if size == 1 { // `offset > bound - 1` is the same as `offset >= bound`. - oob = pos.ins().icmp( - IntCC::UnsignedGreaterThanOrEqual, - offset, - bound, - ); + oob = pos.ins() + .icmp(IntCC::UnsignedGreaterThanOrEqual, offset, bound); } else if size <= min_size { // We know that bound >= min_size, so here we can compare `offset > bound - size` without // wrapping. let adj_bound = pos.ins().iadd_imm(bound, -size); - oob = pos.ins().icmp( - IntCC::UnsignedGreaterThan, - offset, - adj_bound, - ); + oob = pos.ins() + .icmp(IntCC::UnsignedGreaterThan, offset, adj_bound); } else { // We need an overflow check for the adjusted offset. let size_val = pos.ins().iconst(offset_ty, size); let (adj_offset, overflow) = pos.ins().iadd_cout(offset, size_val); pos.ins().trapnz(overflow, ir::TrapCode::HeapOutOfBounds); - oob = pos.ins().icmp( - IntCC::UnsignedGreaterThan, - adj_offset, - bound, - ); + oob = pos.ins() + .icmp(IntCC::UnsignedGreaterThan, adj_offset, bound); } pos.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds); @@ -137,17 +128,11 @@ fn static_addr( let oob = if limit & 1 == 1 { // Prefer testing `offset >= limit - 1` when limit is odd because an even number is // likely to be a convenient constant on ARM and other RISC architectures. - pos.ins().icmp_imm( - IntCC::UnsignedGreaterThanOrEqual, - offset, - limit - 1, - ) + pos.ins() + .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, offset, limit - 1) } else { - pos.ins().icmp_imm( - IntCC::UnsignedGreaterThan, - offset, - limit, - ) + pos.ins() + .icmp_imm(IntCC::UnsignedGreaterThan, offset, limit) }; pos.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds); } diff --git a/lib/codegen/src/legalizer/libcall.rs b/lib/codegen/src/legalizer/libcall.rs index fd7dd17e7e..70982da3cb 100644 --- a/lib/codegen/src/legalizer/libcall.rs +++ b/lib/codegen/src/legalizer/libcall.rs @@ -1,18 +1,18 @@ //! Expanding instructions as runtime library calls. use ir; -use ir::{InstBuilder, get_libcall_funcref}; -use std::vec::Vec; +use ir::{get_libcall_funcref, InstBuilder}; use isa::TargetIsa; +use std::vec::Vec; /// Try to expand `inst` as a library call, returning true is successful. pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function, isa: &TargetIsa) -> bool { // Does the opcode/ctrl_type combo even have a well-known runtime library name. - let libcall = - match ir::LibCall::for_inst(func.dfg[inst].opcode(), func.dfg.ctrl_typevar(inst)) { - Some(lc) => lc, - None => return false, - }; + let libcall = match ir::LibCall::for_inst(func.dfg[inst].opcode(), func.dfg.ctrl_typevar(inst)) + { + Some(lc) => lc, + None => return false, + }; // Now we convert `inst` to a call. First save the arguments. let mut args = Vec::new(); diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs index 9be4975a00..95a6666b2d 100644 --- a/lib/codegen/src/legalizer/mod.rs +++ b/lib/codegen/src/legalizer/mod.rs @@ -27,9 +27,9 @@ mod heap; mod libcall; mod split; +use self::call::expand_call; use self::globalvar::expand_global_addr; use self::heap::expand_heap_addr; -use self::call::expand_call; use self::libcall::expand_as_libcall; /// Legalize `inst` for `isa`. Return true if any changes to the code were diff --git a/lib/codegen/src/legalizer/split.rs b/lib/codegen/src/legalizer/split.rs index 8f82166db8..99babd4a27 100644 --- a/lib/codegen/src/legalizer/split.rs +++ b/lib/codegen/src/legalizer/split.rs @@ -134,9 +134,9 @@ fn split_any( pos.func.dfg.display_inst(inst, None) ); let fixed_args = branch_opc.constraints().fixed_value_arguments(); - let mut args = pos.func.dfg[inst].take_value_list().expect( - "Branches must have value lists.", - ); + let mut args = pos.func.dfg[inst] + .take_value_list() + .expect("Branches must have value lists."); let num_args = args.len(&pos.func.dfg.value_lists); // Get the old value passed to the EBB argument we're repairing. let old_arg = args.get(fixed_args + repair.num, &pos.func.dfg.value_lists) @@ -236,12 +236,9 @@ fn split_value( // Note that it is safe to move `pos` here since `reuse` was set above, so we don't // need to insert a split instruction before returning. pos.goto_first_inst(ebb); - pos.ins().with_result(value).Binary( - concat, - split_type, - lo, - hi, - ); + pos.ins() + .with_result(value) + .Binary(concat, split_type, lo, hi); // Finally, splitting the EBB parameter is not enough. We also have to repair all // of the predecessor instructions that branch here. diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs index 05fe7c03df..9636d317b0 100644 --- a/lib/codegen/src/lib.rs +++ b/lib/codegen/src/lib.rs @@ -31,17 +31,9 @@ redundant_field_names, useless_let_if_seq, len_without_is_empty))] -#![cfg_attr(feature="cargo-clippy", warn( - float_arithmetic, - mut_mut, - nonminimal_bool, - option_map_unwrap_or, - option_map_unwrap_or_else, - print_stdout, - unicode_not_nfc, - use_self, - ))] - +#![cfg_attr(feature = "cargo-clippy", + warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, + option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] // Turns on no_std and alloc features if std is not available. #![cfg_attr(not(feature = "std"), no_std)] #![cfg_attr(not(feature = "std"), feature(alloc))] @@ -90,7 +82,6 @@ pub use entity::packed_option; mod abi; mod bitset; -mod nan_canonicalization; mod constant_hash; mod context; mod dce; @@ -99,6 +90,7 @@ mod fx; mod iterators; mod legalizer; mod licm; +mod nan_canonicalization; mod partition_slice; mod postopt; mod predicates; @@ -115,11 +107,11 @@ mod write; /// This replaces `std` in builds with `core`. #[cfg(not(feature = "std"))] mod std { + pub use alloc::{boxed, string, vec}; pub use core::*; - pub use alloc::{boxed, vec, string}; pub mod collections { - pub use hashmap_core::{HashMap, HashSet}; - pub use hashmap_core::map as hash_map; pub use alloc::BTreeSet; + pub use hashmap_core::map as hash_map; + pub use hashmap_core::{HashMap, HashSet}; } } diff --git a/lib/codegen/src/licm.rs b/lib/codegen/src/licm.rs index f52d192000..5771e89fb1 100644 --- a/lib/codegen/src/licm.rs +++ b/lib/codegen/src/licm.rs @@ -132,9 +132,9 @@ fn change_branch_jump_destination(inst: Inst, new_ebb: Ebb, func: &mut Function) /// Test whether the given opcode is unsafe to even consider for LICM. fn trivially_unsafe_for_licm(opcode: Opcode) -> bool { - opcode.can_load() || opcode.can_store() || opcode.is_call() || opcode.is_branch() || - opcode.is_terminator() || opcode.is_return() || - opcode.can_trap() || opcode.other_side_effects() || opcode.writes_cpu_flags() + opcode.can_load() || opcode.can_store() || opcode.is_call() || opcode.is_branch() + || opcode.is_terminator() || opcode.is_return() || opcode.can_trap() + || opcode.other_side_effects() || opcode.writes_cpu_flags() } /// Test whether the given instruction is loop-invariant. diff --git a/lib/codegen/src/nan_canonicalization.rs b/lib/codegen/src/nan_canonicalization.rs index 20877c3e18..0db8c12056 100644 --- a/lib/codegen/src/nan_canonicalization.rs +++ b/lib/codegen/src/nan_canonicalization.rs @@ -3,11 +3,11 @@ //! that will replace nondeterministic NaN's with a single canonical NaN value. use cursor::{Cursor, FuncCursor}; -use ir::{Function, Inst, InstBuilder, InstructionData, Opcode, Value}; use ir::condcodes::FloatCC; use ir::immediates::{Ieee32, Ieee64}; use ir::types; use ir::types::Type; +use ir::{Function, Inst, InstBuilder, InstructionData, Opcode, Value}; use timing; // Canonical 32-bit and 64-bit NaN values. @@ -33,13 +33,13 @@ pub fn do_nan_canonicalization(func: &mut Function) { fn is_fp_arith(pos: &mut FuncCursor, inst: Inst) -> bool { match pos.func.dfg[inst] { InstructionData::Unary { opcode, .. } => { - opcode == Opcode::Ceil || opcode == Opcode::Floor || opcode == Opcode::Nearest || - opcode == Opcode::Sqrt || opcode == Opcode::Trunc + opcode == Opcode::Ceil || opcode == Opcode::Floor || opcode == Opcode::Nearest + || opcode == Opcode::Sqrt || opcode == Opcode::Trunc } InstructionData::Binary { opcode, .. } => { - opcode == Opcode::Fadd || opcode == Opcode::Fdiv || opcode == Opcode::Fmax || - opcode == Opcode::Fmin || opcode == Opcode::Fmul || - opcode == Opcode::Fsub + opcode == Opcode::Fadd || opcode == Opcode::Fdiv || opcode == Opcode::Fmax + || opcode == Opcode::Fmin || opcode == Opcode::Fmul + || opcode == Opcode::Fsub } InstructionData::Ternary { opcode, .. } => opcode == Opcode::Fma, _ => false, @@ -59,11 +59,9 @@ fn add_nan_canon_seq(pos: &mut FuncCursor, inst: Inst) { // the canonical NaN value if `val` is NaN, assign the result to `inst`. let is_nan = pos.ins().fcmp(FloatCC::NotEqual, new_res, new_res); let canon_nan = insert_nan_const(pos, val_type); - pos.ins().with_result(val).select( - is_nan, - canon_nan, - new_res, - ); + pos.ins() + .with_result(val) + .select(is_nan, canon_nan, new_res); pos.prev_inst(); // Step backwards so the pass does not skip instructions. } diff --git a/lib/codegen/src/postopt.rs b/lib/codegen/src/postopt.rs index 50661f4cc1..517621bef5 100644 --- a/lib/codegen/src/postopt.rs +++ b/lib/codegen/src/postopt.rs @@ -7,7 +7,7 @@ use ir::condcodes::{CondCode, FloatCC, IntCC}; use ir::dfg::ValueDef; use ir::immediates::{Imm64, Offset32}; use ir::instructions::{Opcode, ValueList}; -use ir::{Ebb, Function, Inst, InstBuilder, InstructionData, Value, Type, MemFlags}; +use ir::{Ebb, Function, Inst, InstBuilder, InstructionData, MemFlags, Type, Value}; use isa::TargetIsa; use timing; @@ -135,12 +135,10 @@ fn optimize_cpu_flags( if info.invert_branch_cond { cond = cond.inverse(); } - pos.func.dfg.replace(info.br_inst).brif( - cond, - flags, - info.destination, - &args, - ); + pos.func + .dfg + .replace(info.br_inst) + .brif(cond, flags, info.destination, &args); } CmpBrKind::IcmpImm { mut cond, imm } => { let flags = pos.ins().ifcmp_imm(info.cmp_arg, imm); @@ -148,12 +146,10 @@ fn optimize_cpu_flags( if info.invert_branch_cond { cond = cond.inverse(); } - pos.func.dfg.replace(info.br_inst).brif( - cond, - flags, - info.destination, - &args, - ); + pos.func + .dfg + .replace(info.br_inst) + .brif(cond, flags, info.destination, &args); } CmpBrKind::Fcmp { mut cond, arg } => { let flags = pos.ins().ffcmp(info.cmp_arg, arg); @@ -161,12 +157,10 @@ fn optimize_cpu_flags( if info.invert_branch_cond { cond = cond.inverse(); } - pos.func.dfg.replace(info.br_inst).brff( - cond, - flags, - info.destination, - &args, - ); + pos.func + .dfg + .replace(info.br_inst) + .brff(cond, flags, info.destination, &args); } } let ok = pos.func.update_encoding(info.cmp_inst, isa).is_ok(); @@ -175,7 +169,6 @@ fn optimize_cpu_flags( debug_assert!(ok); } - struct MemOpInfo { opcode: Opcode, inst: Inst, @@ -326,8 +319,6 @@ fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa) debug_assert!(ok); } - - //---------------------------------------------------------------------- // // The main post-opt pass. @@ -343,10 +334,8 @@ pub fn do_postopt(func: &mut Function, isa: &TargetIsa) { optimize_cpu_flags(&mut pos, inst, last_flags_clobber, isa); // Track the most recent seen instruction that clobbers the flags. - if let Some(constraints) = - isa.encoding_info().operand_constraints( - pos.func.encodings[inst], - ) + if let Some(constraints) = isa.encoding_info() + .operand_constraints(pos.func.encodings[inst]) { if constraints.clobbers_flags { last_flags_clobber = Some(inst) diff --git a/lib/codegen/src/preopt.rs b/lib/codegen/src/preopt.rs index cbe7484fd0..ead49b9486 100644 --- a/lib/codegen/src/preopt.rs +++ b/lib/codegen/src/preopt.rs @@ -137,27 +137,25 @@ fn get_div_info(inst: Inst, dfg: &DataFlowGraph) -> Option { /// cannot do any transformation, in which case `inst` is left unchanged. fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCursor, inst: Inst) { let isRem = match *divrem_info { - DivRemByConstInfo::DivU32(_, _) | - DivRemByConstInfo::DivU64(_, _) | - DivRemByConstInfo::DivS32(_, _) | - DivRemByConstInfo::DivS64(_, _) => false, - DivRemByConstInfo::RemU32(_, _) | - DivRemByConstInfo::RemU64(_, _) | - DivRemByConstInfo::RemS32(_, _) | - DivRemByConstInfo::RemS64(_, _) => true, + DivRemByConstInfo::DivU32(_, _) + | DivRemByConstInfo::DivU64(_, _) + | DivRemByConstInfo::DivS32(_, _) + | DivRemByConstInfo::DivS64(_, _) => false, + DivRemByConstInfo::RemU32(_, _) + | DivRemByConstInfo::RemU64(_, _) + | DivRemByConstInfo::RemS32(_, _) + | DivRemByConstInfo::RemS64(_, _) => true, }; match *divrem_info { // -------------------- U32 -------------------- // U32 div, rem by zero: ignore - DivRemByConstInfo::DivU32(_n1, 0) | - DivRemByConstInfo::RemU32(_n1, 0) => {} + DivRemByConstInfo::DivU32(_n1, 0) | DivRemByConstInfo::RemU32(_n1, 0) => {} // U32 div by 1: identity // U32 rem by 1: zero - DivRemByConstInfo::DivU32(n1, 1) | - DivRemByConstInfo::RemU32(n1, 1) => { + DivRemByConstInfo::DivU32(n1, 1) | DivRemByConstInfo::RemU32(n1, 1) => { if isRem { pos.func.dfg.replace(inst).iconst(I32, 0); } else { @@ -166,8 +164,9 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } // U32 div, rem by a power-of-2 - DivRemByConstInfo::DivU32(n1, d) | - DivRemByConstInfo::RemU32(n1, d) if d.is_power_of_two() => { + DivRemByConstInfo::DivU32(n1, d) | DivRemByConstInfo::RemU32(n1, d) + if d.is_power_of_two() => + { debug_assert!(d >= 2); // compute k where d == 2^k let k = d.trailing_zeros(); @@ -181,8 +180,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } // U32 div, rem by non-power-of-2 - DivRemByConstInfo::DivU32(n1, d) | - DivRemByConstInfo::RemU32(n1, d) => { + DivRemByConstInfo::DivU32(n1, d) | DivRemByConstInfo::RemU32(n1, d) => { debug_assert!(d >= 3); let MU32 { mulBy, @@ -223,13 +221,11 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso // -------------------- U64 -------------------- // U64 div, rem by zero: ignore - DivRemByConstInfo::DivU64(_n1, 0) | - DivRemByConstInfo::RemU64(_n1, 0) => {} + DivRemByConstInfo::DivU64(_n1, 0) | DivRemByConstInfo::RemU64(_n1, 0) => {} // U64 div by 1: identity // U64 rem by 1: zero - DivRemByConstInfo::DivU64(n1, 1) | - DivRemByConstInfo::RemU64(n1, 1) => { + DivRemByConstInfo::DivU64(n1, 1) | DivRemByConstInfo::RemU64(n1, 1) => { if isRem { pos.func.dfg.replace(inst).iconst(I64, 0); } else { @@ -238,8 +234,9 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } // U64 div, rem by a power-of-2 - DivRemByConstInfo::DivU64(n1, d) | - DivRemByConstInfo::RemU64(n1, d) if d.is_power_of_two() => { + DivRemByConstInfo::DivU64(n1, d) | DivRemByConstInfo::RemU64(n1, d) + if d.is_power_of_two() => + { debug_assert!(d >= 2); // compute k where d == 2^k let k = d.trailing_zeros(); @@ -253,8 +250,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } // U64 div, rem by non-power-of-2 - DivRemByConstInfo::DivU64(n1, d) | - DivRemByConstInfo::RemU64(n1, d) => { + DivRemByConstInfo::DivU64(n1, d) | DivRemByConstInfo::RemU64(n1, d) => { debug_assert!(d >= 3); let MU64 { mulBy, @@ -295,15 +291,14 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso // -------------------- S32 -------------------- // S32 div, rem by zero or -1: ignore - DivRemByConstInfo::DivS32(_n1, -1) | - DivRemByConstInfo::RemS32(_n1, -1) | - DivRemByConstInfo::DivS32(_n1, 0) | - DivRemByConstInfo::RemS32(_n1, 0) => {} + DivRemByConstInfo::DivS32(_n1, -1) + | DivRemByConstInfo::RemS32(_n1, -1) + | DivRemByConstInfo::DivS32(_n1, 0) + | DivRemByConstInfo::RemS32(_n1, 0) => {} // S32 div by 1: identity // S32 rem by 1: zero - DivRemByConstInfo::DivS32(n1, 1) | - DivRemByConstInfo::RemS32(n1, 1) => { + DivRemByConstInfo::DivS32(n1, 1) | DivRemByConstInfo::RemS32(n1, 1) => { if isRem { pos.func.dfg.replace(inst).iconst(I32, 0); } else { @@ -311,8 +306,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } } - DivRemByConstInfo::DivS32(n1, d) | - DivRemByConstInfo::RemS32(n1, d) => { + DivRemByConstInfo::DivS32(n1, d) | DivRemByConstInfo::RemS32(n1, d) => { if let Some((isNeg, k)) = isPowerOf2_S32(d) { // k can be 31 only in the case that d is -2^31. debug_assert!(k >= 1 && k <= 31); @@ -372,15 +366,14 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso // -------------------- S64 -------------------- // S64 div, rem by zero or -1: ignore - DivRemByConstInfo::DivS64(_n1, -1) | - DivRemByConstInfo::RemS64(_n1, -1) | - DivRemByConstInfo::DivS64(_n1, 0) | - DivRemByConstInfo::RemS64(_n1, 0) => {} + DivRemByConstInfo::DivS64(_n1, -1) + | DivRemByConstInfo::RemS64(_n1, -1) + | DivRemByConstInfo::DivS64(_n1, 0) + | DivRemByConstInfo::RemS64(_n1, 0) => {} // S64 div by 1: identity // S64 rem by 1: zero - DivRemByConstInfo::DivS64(n1, 1) | - DivRemByConstInfo::RemS64(n1, 1) => { + DivRemByConstInfo::DivS64(n1, 1) | DivRemByConstInfo::RemS64(n1, 1) => { if isRem { pos.func.dfg.replace(inst).iconst(I64, 0); } else { @@ -388,8 +381,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso } } - DivRemByConstInfo::DivS64(n1, d) | - DivRemByConstInfo::RemS64(n1, d) => { + DivRemByConstInfo::DivS64(n1, d) | DivRemByConstInfo::RemS64(n1, d) => { if let Some((isNeg, k)) = isPowerOf2_S64(d) { // k can be 63 only in the case that d is -2^63. debug_assert!(k >= 1 && k <= 63); @@ -483,12 +475,10 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) { _ => return, }; let ty = pos.func.dfg.ctrl_typevar(inst); - pos.func.dfg.replace(inst).BinaryImm( - new_opcode, - ty, - imm, - args[0], - ); + pos.func + .dfg + .replace(inst) + .BinaryImm(new_opcode, ty, imm, args[0]); } } else if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(args[0]) { if let InstructionData::UnaryImm { @@ -501,12 +491,10 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) { _ => return, }; let ty = pos.func.dfg.ctrl_typevar(inst); - pos.func.dfg.replace(inst).BinaryImm( - new_opcode, - ty, - imm, - args[1], - ); + pos.func + .dfg + .replace(inst) + .BinaryImm(new_opcode, ty, imm, args[1]); } } } @@ -522,9 +510,12 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) { } } } - InstructionData::CondTrap { .. } | - InstructionData::Branch { .. } | - InstructionData::Ternary { opcode: Opcode::Select, .. } => { + InstructionData::CondTrap { .. } + | InstructionData::Branch { .. } + | InstructionData::Ternary { + opcode: Opcode::Select, + .. + } => { // Fold away a redundant `bint`. let maybe = { let args = pos.func.dfg.inst_args(inst); diff --git a/lib/codegen/src/regalloc/affinity.rs b/lib/codegen/src/regalloc/affinity.rs index a8fa423049..44d3dbb11b 100644 --- a/lib/codegen/src/regalloc/affinity.rs +++ b/lib/codegen/src/regalloc/affinity.rs @@ -90,8 +90,7 @@ impl Affinity { Affinity::Reg(rc) => { // If the preferred register class is a subclass of the constraint, there's no need // to change anything. - if constraint.kind != ConstraintKind::Stack && - !constraint.regclass.has_subclass(rc) + if constraint.kind != ConstraintKind::Stack && !constraint.regclass.has_subclass(rc) { // If the register classes don't overlap, `intersect` returns `Unassigned`, and // we just keep our previous affinity. @@ -120,12 +119,10 @@ impl<'a> fmt::Display for DisplayAffinity<'a> { match self.0 { Affinity::Unassigned => write!(f, "unassigned"), Affinity::Stack => write!(f, "stack"), - Affinity::Reg(rci) => { - match self.1 { - Some(regs) => write!(f, "{}", regs.rc(rci)), - None => write!(f, "{}", rci), - } - } + Affinity::Reg(rci) => match self.1 { + Some(regs) => write!(f, "{}", regs.rc(rci)), + None => write!(f, "{}", rci), + }, } } } diff --git a/lib/codegen/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs index 5cd0815e5b..dd77b41937 100644 --- a/lib/codegen/src/regalloc/coalescing.rs +++ b/lib/codegen/src/regalloc/coalescing.rs @@ -196,8 +196,7 @@ impl<'a> Context<'a> { pred_inst, pred_ebb, self.liveness.context(&self.func.layout), - ) - { + ) { self.isolate_param(ebb, param); } } @@ -219,8 +218,8 @@ impl<'a> Context<'a> { // pre-spilled, and the rest of the virtual register would be forced to spill to the // `incoming_arg` stack slot too. if let ir::ValueDef::Param(def_ebb, def_num) = self.func.dfg.value_def(arg) { - if Some(def_ebb) == self.func.layout.entry_block() && - self.func.signature.params[def_num].location.is_stack() + if Some(def_ebb) == self.func.layout.entry_block() + && self.func.signature.params[def_num].location.is_stack() { dbg!("-> isolating function stack parameter {}", arg); let new_arg = self.isolate_arg(pred_ebb, pred_inst, argnum, arg); @@ -303,16 +302,11 @@ impl<'a> Context<'a> { &self.encinfo .operand_constraints(pos.func.encodings[inst]) .expect("Bad copy encoding") - .outs - [0], + .outs[0], ); self.liveness.create_dead(new_val, ebb, affinity); - self.liveness.extend_locally( - new_val, - ebb, - inst, - &pos.func.layout, - ); + self.liveness + .extend_locally(new_val, ebb, inst, &pos.func.layout); new_val } @@ -353,16 +347,11 @@ impl<'a> Context<'a> { &self.encinfo .operand_constraints(pos.func.encodings[inst]) .expect("Bad copy encoding") - .outs - [0], + .outs[0], ); self.liveness.create_dead(copy, inst, affinity); - self.liveness.extend_locally( - copy, - pred_ebb, - pred_inst, - &pos.func.layout, - ); + self.liveness + .extend_locally(copy, pred_ebb, pred_inst, &pos.func.layout); pos.func.dfg.inst_variable_args_mut(pred_inst)[argnum] = copy; @@ -422,12 +411,9 @@ impl<'a> Context<'a> { let node = Node::value(value, 0, self.func); // Push this value and get the nearest dominating def back. - let parent = match self.forest.push_node( - node, - self.func, - self.domtree, - self.preorder, - ) { + let parent = match self.forest + .push_node(node, self.func, self.domtree, self.preorder) + { None => continue, Some(n) => n, }; @@ -525,12 +511,8 @@ impl<'a> Context<'a> { // Can't merge because of interference. Insert a copy instead. let pred_ebb = self.func.layout.pp_ebb(pred_inst); let new_arg = self.isolate_arg(pred_ebb, pred_inst, argnum, arg); - self.virtregs.insert_single( - param, - new_arg, - self.func, - self.preorder, - ); + self.virtregs + .insert_single(param, new_arg, self.func, self.preorder); } } @@ -564,12 +546,8 @@ impl<'a> Context<'a> { // Restrict the virtual copy nodes we look at and key the `set_id` and `value` properties // of the nodes. Set_id 0 will be `param` and set_id 1 will be `arg`. - self.vcopies.set_filter( - [param, arg], - func, - self.virtregs, - preorder, - ); + self.vcopies + .set_filter([param, arg], func, self.virtregs, preorder); // Now create an ordered sequence of dom-forest nodes from three sources: The two virtual // registers and the filtered virtual copies. @@ -625,8 +603,8 @@ impl<'a> Context<'a> { // Check if the parent value interferes with the virtual copy. let inst = node.def.unwrap_inst(); - if node.set_id != parent.set_id && - self.liveness[parent.value].reaches_use(inst, node.ebb, ctx) + if node.set_id != parent.set_id + && self.liveness[parent.value].reaches_use(inst, node.ebb, ctx) { dbg!( " - interference: {} overlaps vcopy at {}:{}", @@ -649,8 +627,8 @@ impl<'a> Context<'a> { // Both node and parent are values, so check for interference. debug_assert!(node.is_value() && parent.is_value()); - if node.set_id != parent.set_id && - self.liveness[parent.value].overlaps_def(node.def, node.ebb, ctx) + if node.set_id != parent.set_id + && self.liveness[parent.value].overlaps_def(node.def, node.ebb, ctx) { // The two values are interfering. dbg!(" - interference: {} overlaps def of {}", parent, node.value); @@ -945,9 +923,8 @@ impl VirtualCopies { } // Reorder the predecessor branches as required by the dominator forest. - self.branches.sort_unstable_by(|&(a, _), &(b, _)| { - preorder.pre_cmp(a, b, &func.layout) - }); + self.branches + .sort_unstable_by(|&(a, _), &(b, _)| preorder.pre_cmp(a, b, &func.layout)); } /// Get the next unmerged parameter value. @@ -1097,9 +1074,9 @@ where let ord = match (self.a.peek(), self.b.peek()) { (Some(a), Some(b)) => { let layout = self.layout; - self.preorder.pre_cmp_ebb(a.ebb, b.ebb).then_with(|| { - layout.cmp(a.def, b.def) - }) + self.preorder + .pre_cmp_ebb(a.ebb, b.ebb) + .then_with(|| layout.cmp(a.def, b.def)) } (Some(_), None) => cmp::Ordering::Less, (None, Some(_)) => cmp::Ordering::Greater, diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs index d080213ade..aa7cbbeb47 100644 --- a/lib/codegen/src/regalloc/coloring.rs +++ b/lib/codegen/src/regalloc/coloring.rs @@ -51,10 +51,10 @@ use isa::{regs_overlap, RegClass, RegInfo, RegUnit}; use packed_option::PackedOption; use regalloc::RegDiversions; use regalloc::affinity::Affinity; -use regalloc::register_set::RegisterSet; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; use regalloc::liverange::{LiveRange, LiveRangeContext}; +use regalloc::register_set::RegisterSet; use regalloc::solver::{Solver, SolverError}; use std::mem; use timing; @@ -142,9 +142,10 @@ impl Coloring { impl<'a> Context<'a> { /// Run the coloring algorithm. fn run(&mut self, tracker: &mut LiveValueTracker) { - self.cur.func.locations.resize( - self.cur.func.dfg.num_values(), - ); + self.cur + .func + .locations + .resize(self.cur.func.dfg.num_values()); // Visit blocks in reverse post-order. We need to ensure that at least one predecessor has // been visited before each EBB. That guarantees that the EBB arguments have been colored. @@ -372,10 +373,8 @@ impl<'a> Context<'a> { // Update the global register set which has no diversions. if !lv.is_local { - regs.global.free( - rc, - self.cur.func.locations[lv.value].unwrap_reg(), - ); + regs.global + .free(rc, self.cur.func.locations[lv.value].unwrap_reg()); } } } @@ -500,20 +499,14 @@ impl<'a> Context<'a> { // already in a register. let cur_reg = self.divert.reg(value, &self.cur.func.locations); match op.kind { - ConstraintKind::FixedReg(regunit) | - ConstraintKind::FixedTied(regunit) => { + ConstraintKind::FixedReg(regunit) | ConstraintKind::FixedTied(regunit) => { // Add the fixed constraint even if `cur_reg == regunit`. // It is possible that we will want to convert the value to a variable later, // and this identity assignment prevents that from happening. - self.solver.reassign_in( - value, - op.regclass, - cur_reg, - regunit, - ); + self.solver + .reassign_in(value, op.regclass, cur_reg, regunit); } - ConstraintKind::Reg | - ConstraintKind::Tied(_) => { + ConstraintKind::Reg | ConstraintKind::Tied(_) => { if !op.regclass.contains(cur_reg) { self.solver.add_var(value, op.regclass, cur_reg); } @@ -541,8 +534,7 @@ impl<'a> Context<'a> { for (op, &value) in constraints.iter().zip(self.cur.func.dfg.inst_args(inst)) { match op.kind { - ConstraintKind::Reg | - ConstraintKind::Tied(_) => { + ConstraintKind::Reg | ConstraintKind::Tied(_) => { let cur_reg = self.divert.reg(value, &self.cur.func.locations); // This is the opposite condition of `program_input_constraints()`. if op.regclass.contains(cur_reg) { @@ -556,9 +548,9 @@ impl<'a> Context<'a> { } } } - ConstraintKind::FixedReg(_) | - ConstraintKind::FixedTied(_) | - ConstraintKind::Stack => {} + ConstraintKind::FixedReg(_) + | ConstraintKind::FixedTied(_) + | ConstraintKind::Stack => {} } } } @@ -651,9 +643,9 @@ impl<'a> Context<'a> { Pred: FnMut(&LiveRange, LiveRangeContext) -> bool, { for rdiv in self.divert.all() { - let lr = self.liveness.get(rdiv.value).expect( - "Missing live range for diverted register", - ); + let lr = self.liveness + .get(rdiv.value) + .expect("Missing live range for diverted register"); if pred(lr, self.liveness.context(&self.cur.func.layout)) { if let Affinity::Reg(rci) = lr.affinity { let rc = self.reginfo.rc(rci); @@ -703,8 +695,7 @@ impl<'a> Context<'a> { ) { for (op, lv) in constraints.iter().zip(defs) { match op.kind { - ConstraintKind::FixedReg(reg) | - ConstraintKind::FixedTied(reg) => { + ConstraintKind::FixedReg(reg) | ConstraintKind::FixedTied(reg) => { self.add_fixed_output(lv.value, op.regclass, reg, throughs); if !lv.is_local && !global_regs.is_avail(op.regclass, reg) { dbg!( @@ -716,9 +707,7 @@ impl<'a> Context<'a> { *replace_global_defines = true; } } - ConstraintKind::Reg | - ConstraintKind::Tied(_) | - ConstraintKind::Stack => {} + ConstraintKind::Reg | ConstraintKind::Tied(_) | ConstraintKind::Stack => {} } } } @@ -801,9 +790,9 @@ impl<'a> Context<'a> { ) { for (op, lv) in constraints.iter().zip(defs) { match op.kind { - ConstraintKind::FixedReg(_) | - ConstraintKind::FixedTied(_) | - ConstraintKind::Stack => continue, + ConstraintKind::FixedReg(_) + | ConstraintKind::FixedTied(_) + | ConstraintKind::Stack => continue, ConstraintKind::Reg => { self.solver.add_def(lv.value, op.regclass, !lv.is_local); } @@ -816,8 +805,7 @@ impl<'a> Context<'a> { op.regclass, self.divert.reg(arg, &self.cur.func.locations), !lv.is_local, - ) - { + ) { // The value we're tied to has been assigned to a fixed register. // We need to make sure that fixed output register is compatible with the // global register set. @@ -881,8 +869,8 @@ impl<'a> Context<'a> { // not actually constrained by the instruction. We just want it out of the way. let toprc2 = self.reginfo.toprc(rci); let reg2 = self.divert.reg(lv.value, &self.cur.func.locations); - if rc.contains(reg2) && self.solver.can_add_var(lv.value, toprc2, reg2) && - !self.is_live_on_outgoing_edge(lv.value) + if rc.contains(reg2) && self.solver.can_add_var(lv.value, toprc2, reg2) + && !self.is_live_on_outgoing_edge(lv.value) { self.solver.add_through_var(lv.value, toprc2, reg2); return true; @@ -911,10 +899,10 @@ impl<'a> Context<'a> { } Table(jt) => { let lr = &self.liveness[value]; - !lr.is_local() && - self.cur.func.jump_tables[jt].entries().any(|(_, ebb)| { - lr.is_livein(ebb, ctx) - }) + !lr.is_local() + && self.cur.func.jump_tables[jt] + .entries() + .any(|(_, ebb)| lr.is_livein(ebb, ctx)) } } } @@ -940,7 +928,9 @@ impl<'a> Context<'a> { for m in self.solver.moves() { match *m { - Reg { value, from, to, .. } => { + Reg { + value, from, to, .. + } => { self.divert.regmove(value, from, to); self.cur.ins().regmove(value, from, to); } @@ -951,10 +941,10 @@ impl<'a> Context<'a> { .. } => { debug_assert_eq!(slot[to_slot].expand(), None, "Overwriting slot in use"); - let ss = self.cur.func.stack_slots.get_emergency_slot( - self.cur.func.dfg.value_type(value), - &slot[0..spills], - ); + let ss = self.cur + .func + .stack_slots + .get_emergency_slot(self.cur.func.dfg.value_type(value), &slot[0..spills]); slot[to_slot] = ss.into(); self.divert.regspill(value, from, ss); self.cur.ins().regspill(value, from, ss); @@ -1013,8 +1003,7 @@ impl<'a> Context<'a> { if match self.cur.func.dfg.value_def(lv.value) { ValueDef::Result(i, _) => i != inst, _ => true, - } - { + } { break; } if lv.is_local || !lv.affinity.is_reg() { @@ -1072,10 +1061,8 @@ impl<'a> Context<'a> { }; regs.input.free(rc, loc.unwrap_reg()); if !lv.is_local { - regs.global.free( - rc, - self.cur.func.locations[lv.value].unwrap_reg(), - ); + regs.global + .free(rc, self.cur.func.locations[lv.value].unwrap_reg()); } } } @@ -1096,11 +1083,10 @@ fn program_input_abi( ) { for (abi, &value) in abi_types.iter().zip(func.dfg.inst_variable_args(inst)) { if let ArgumentLoc::Reg(reg) = abi.location { - if let Affinity::Reg(rci) = - liveness - .get(value) - .expect("ABI register must have live range") - .affinity + if let Affinity::Reg(rci) = liveness + .get(value) + .expect("ABI register must have live range") + .affinity { let rc = reginfo.rc(rci); let cur_reg = divert.reg(value, &func.locations); diff --git a/lib/codegen/src/regalloc/context.rs b/lib/codegen/src/regalloc/context.rs index cb29963e89..ce1d68a096 100644 --- a/lib/codegen/src/regalloc/context.rs +++ b/lib/codegen/src/regalloc/context.rs @@ -140,13 +140,8 @@ impl Context { } // Pass: Coloring. - self.coloring.run( - isa, - func, - domtree, - &mut self.liveness, - &mut self.tracker, - ); + self.coloring + .run(isa, func, domtree, &mut self.liveness, &mut self.tracker); if isa.flags().enable_verifier() { verify_context(func, cfg, domtree, isa)?; diff --git a/lib/codegen/src/regalloc/diversion.rs b/lib/codegen/src/regalloc/diversion.rs index 1a747fc7e6..3572acd543 100644 --- a/lib/codegen/src/regalloc/diversion.rs +++ b/lib/codegen/src/regalloc/diversion.rs @@ -46,7 +46,9 @@ pub struct RegDiversions { impl RegDiversions { /// Create a new empty diversion tracker. pub fn new() -> Self { - Self { current: Vec::new() } + Self { + current: Vec::new(), + } } /// Clear the tracker, preparing for a new EBB. @@ -152,11 +154,10 @@ impl RegDiversions { /// /// Returns the `to` location of the removed diversion. pub fn remove(&mut self, value: Value) -> Option { - self.current.iter().position(|d| d.value == value).map( - |i| { - self.current.swap_remove(i).to - }, - ) + self.current + .iter() + .position(|d| d.value == value) + .map(|i| self.current.swap_remove(i).to) } /// Return an object that can display the diversions. diff --git a/lib/codegen/src/regalloc/live_value_tracker.rs b/lib/codegen/src/regalloc/live_value_tracker.rs index 12bda20ff4..2665c245c1 100644 --- a/lib/codegen/src/regalloc/live_value_tracker.rs +++ b/lib/codegen/src/regalloc/live_value_tracker.rs @@ -187,15 +187,15 @@ impl LiveValueTracker { // If the immediate dominator exits, we must have a stored list for it. This is a // requirement to the order EBBs are visited: All dominators must have been processed // before the current EBB. - let idom_live_list = self.idom_sets.get(&idom).expect( - "No stored live set for dominator", - ); + let idom_live_list = self.idom_sets + .get(&idom) + .expect("No stored live set for dominator"); let ctx = liveness.context(layout); // Get just the values that are live-in to `ebb`. for &value in idom_live_list.as_slice(&self.idom_pool) { - let lr = liveness.get(value).expect( - "Immediate dominator value has no live range", - ); + let lr = liveness + .get(value) + .expect("Immediate dominator value has no live range"); // Check if this value is live-in here. if let Some(endpoint) = lr.livein_local_end(ebb, ctx) { @@ -217,17 +217,13 @@ impl LiveValueTracker { // This is a dead EBB parameter which is not even live into the first // instruction in the EBB. debug_assert_eq!( - local_ebb, - ebb, + local_ebb, ebb, "EBB parameter live range ends at wrong EBB header" ); // Give this value a fake endpoint that is the first instruction in the EBB. // We expect it to be removed by calling `drop_dead_args()`. - self.live.push( - value, - layout.first_inst(ebb).expect("Empty EBB"), - lr, - ); + self.live + .push(value, layout.first_inst(ebb).expect("Empty EBB"), lr); } } } diff --git a/lib/codegen/src/regalloc/liveness.rs b/lib/codegen/src/regalloc/liveness.rs index faf35f6a1f..959e48183a 100644 --- a/lib/codegen/src/regalloc/liveness.rs +++ b/lib/codegen/src/regalloc/liveness.rs @@ -179,7 +179,7 @@ use entity::SparseMap; use flowgraph::ControlFlowGraph; use ir::dfg::ValueDef; use ir::{Ebb, Function, Inst, Layout, ProgramPoint, Value}; -use isa::{EncInfo, TargetIsa, OperandConstraint}; +use isa::{EncInfo, OperandConstraint, TargetIsa}; use regalloc::affinity::Affinity; use regalloc::liverange::{LiveRange, LiveRangeContext, LiveRangeForest}; use std::mem; @@ -217,9 +217,9 @@ fn get_or_create<'a>( .map(Affinity::new) .or_else(|| { // If this is a call, get the return value affinity. - func.dfg.call_signature(inst).map(|sig| { - Affinity::abi(&func.dfg.signatures[sig].returns[rnum], isa) - }) + func.dfg + .call_signature(inst) + .map(|sig| Affinity::abi(&func.dfg.signatures[sig].returns[rnum], isa)) }) .unwrap_or_default(); } @@ -336,9 +336,8 @@ impl Liveness { where PP: Into, { - let old = self.ranges.insert( - LiveRange::new(value, def.into(), affinity), - ); + let old = self.ranges + .insert(LiveRange::new(value, def.into(), affinity)); debug_assert!(old.is_none(), "{} already has a live range", value); } diff --git a/lib/codegen/src/regalloc/liverange.rs b/lib/codegen/src/regalloc/liverange.rs index d9ccc4fac5..a1e73b6206 100644 --- a/lib/codegen/src/regalloc/liverange.rs +++ b/lib/codegen/src/regalloc/liverange.rs @@ -249,13 +249,12 @@ impl GenLiveRange { // // We're assuming here that `to` never precedes `def_begin` in the same EBB, but we can't // check it without a method for getting `to`'s EBB. - if order.cmp(ebb, self.def_end) != Ordering::Greater && - order.cmp(to, self.def_begin) != Ordering::Less + if order.cmp(ebb, self.def_end) != Ordering::Greater + && order.cmp(to, self.def_begin) != Ordering::Less { let to_pp = to.into(); debug_assert_ne!( - to_pp, - self.def_begin, + to_pp, self.def_begin, "Can't use value in the defining instruction." ); if order.cmp(to, self.def_end) == Ordering::Greater { @@ -411,8 +410,8 @@ impl GenLiveRange { } // Check for an overlap with the local range. - if ctx.order.cmp(def, self.def_begin) != Ordering::Less && - ctx.order.cmp(def, self.def_end) == Ordering::Less + if ctx.order.cmp(def, self.def_begin) != Ordering::Less + && ctx.order.cmp(def, self.def_end) == Ordering::Less { return true; } @@ -427,8 +426,8 @@ impl GenLiveRange { /// Check if this live range reaches a use at `user` in `ebb`. pub fn reaches_use(&self, user: Inst, ebb: Ebb, ctx: LiveRangeContext) -> bool { // Check for an overlap with the local range. - if ctx.order.cmp(user, self.def_begin) == Ordering::Greater && - ctx.order.cmp(user, self.def_end) != Ordering::Greater + if ctx.order.cmp(user, self.def_begin) == Ordering::Greater + && ctx.order.cmp(user, self.def_end) != Ordering::Greater { return true; } @@ -535,8 +534,8 @@ mod tests { } assert!( - self.cmp(lr.def_end, begin) == Ordering::Less || - self.cmp(lr.def_begin, end) == Ordering::Greater, + self.cmp(lr.def_end, begin) == Ordering::Less + || self.cmp(lr.def_begin, end) == Ordering::Greater, "Interval can't overlap the def EBB" ); diff --git a/lib/codegen/src/regalloc/mod.rs b/lib/codegen/src/regalloc/mod.rs index 1444874b83..dbec691602 100644 --- a/lib/codegen/src/regalloc/mod.rs +++ b/lib/codegen/src/regalloc/mod.rs @@ -2,11 +2,11 @@ //! //! This module contains data structures and algorithms used for register allocation. -pub mod register_set; pub mod coloring; pub mod live_value_tracker; pub mod liveness; pub mod liverange; +pub mod register_set; pub mod virtregs; mod affinity; @@ -18,6 +18,6 @@ mod reload; mod solver; mod spilling; -pub use self::register_set::RegisterSet; pub use self::context::Context; pub use self::diversion::RegDiversions; +pub use self::register_set::RegisterSet; diff --git a/lib/codegen/src/regalloc/pressure.rs b/lib/codegen/src/regalloc/pressure.rs index 49eb224e6d..355fce582e 100644 --- a/lib/codegen/src/regalloc/pressure.rs +++ b/lib/codegen/src/regalloc/pressure.rs @@ -114,9 +114,10 @@ impl Pressure { } // Compute per-class limits from `usable`. - for (toprc, rc) in p.toprc.iter_mut().take_while(|t| t.num_toprcs > 0).zip( - reginfo.classes, - ) + for (toprc, rc) in p.toprc + .iter_mut() + .take_while(|t| t.num_toprcs > 0) + .zip(reginfo.classes) { toprc.limit = usable.iter(rc).len() as u32; toprc.width = rc.width; @@ -203,16 +204,16 @@ impl Pressure { /// /// This does not check if there are enough registers available. pub fn take(&mut self, rc: RegClass) { - self.toprc.get_mut(rc.toprc as usize).map( - |t| t.base_count += 1, - ); + self.toprc + .get_mut(rc.toprc as usize) + .map(|t| t.base_count += 1); } /// Free a register in `rc`. pub fn free(&mut self, rc: RegClass) { - self.toprc.get_mut(rc.toprc as usize).map( - |t| t.base_count -= 1, - ); + self.toprc + .get_mut(rc.toprc as usize) + .map(|t| t.base_count -= 1); } /// Reset all counts to 0, both base and transient. @@ -229,9 +230,9 @@ impl Pressure { pub fn take_transient(&mut self, rc: RegClass) -> Result<(), RegClassMask> { let mask = self.check_avail(rc); if mask == 0 { - self.toprc.get_mut(rc.toprc as usize).map(|t| { - t.transient_count += 1 - }); + self.toprc + .get_mut(rc.toprc as usize) + .map(|t| t.transient_count += 1); Ok(()) } else { Err(mask) diff --git a/lib/codegen/src/regalloc/register_set.rs b/lib/codegen/src/regalloc/register_set.rs index 59f312150e..b297578cae 100644 --- a/lib/codegen/src/regalloc/register_set.rs +++ b/lib/codegen/src/regalloc/register_set.rs @@ -104,9 +104,10 @@ impl RegisterSet { /// /// This assumes that unused bits are 1. pub fn interferes_with(&self, other: &Self) -> bool { - self.avail.iter().zip(&other.avail).any( - |(&x, &y)| (x | y) != !0, - ) + self.avail + .iter() + .zip(&other.avail) + .any(|(&x, &y)| (x | y) != !0) } /// Intersect this set of registers with `other`. This has the effect of removing any register @@ -203,9 +204,10 @@ impl<'a> fmt::Display for DisplayRegisterSet<'a> { bank.names .get(offset as usize) .and_then(|name| name.chars().nth(1)) - .unwrap_or_else( - || char::from_digit(u32::from(offset % 10), 10).unwrap(), - ) + .unwrap_or_else(|| char::from_digit( + u32::from(offset % 10), + 10 + ).unwrap()) )?; } } diff --git a/lib/codegen/src/regalloc/reload.rs b/lib/codegen/src/regalloc/reload.rs index 9397d1a606..ff204a4fd9 100644 --- a/lib/codegen/src/regalloc/reload.rs +++ b/lib/codegen/src/regalloc/reload.rs @@ -166,10 +166,10 @@ impl<'a> Context<'a> { if arg.affinity.is_stack() { // An incoming register parameter was spilled. Replace the parameter value // with a temporary register value that is immediately spilled. - let reg = self.cur.func.dfg.replace_ebb_param( - arg.value, - abi.value_type, - ); + let reg = self.cur + .func + .dfg + .replace_ebb_param(arg.value, abi.value_type); let affinity = Affinity::abi(&abi, self.cur.isa); self.liveness.create_dead(reg, ebb, affinity); self.insert_spill(ebb, arg.value, reg); @@ -199,9 +199,9 @@ impl<'a> Context<'a> { self.cur.use_srcloc(inst); // Get the operand constraints for `inst` that we are trying to satisfy. - let constraints = self.encinfo.operand_constraints(encoding).expect( - "Missing instruction encoding", - ); + let constraints = self.encinfo + .operand_constraints(encoding) + .expect("Missing instruction encoding"); // Identify reload candidates. debug_assert!(self.candidates.is_empty()); @@ -226,12 +226,8 @@ impl<'a> Context<'a> { // Create a live range for the new reload. let affinity = Affinity::Reg(cand.regclass.into()); self.liveness.create_dead(reg, fill, affinity); - self.liveness.extend_locally( - reg, - ebb, - inst, - &self.cur.func.layout, - ); + self.liveness + .extend_locally(reg, ebb, inst, &self.cur.func.layout); } // Rewrite instruction arguments. @@ -280,19 +276,18 @@ impl<'a> Context<'a> { // Same thing for spilled call return values. let retvals = &defs[constraints.outs.len()..]; if !retvals.is_empty() { - let sig = self.cur.func.dfg.call_signature(inst).expect( - "Extra results on non-call instruction", - ); + let sig = self.cur + .func + .dfg + .call_signature(inst) + .expect("Extra results on non-call instruction"); for (i, lv) in retvals.iter().enumerate() { let abi = self.cur.func.dfg.signatures[sig].returns[i]; debug_assert!(abi.location.is_reg()); if lv.affinity.is_stack() { let reg = self.cur.func.dfg.replace_result(lv.value, abi.value_type); - self.liveness.create_dead( - reg, - inst, - Affinity::abi(&abi, self.cur.isa), - ); + self.liveness + .create_dead(reg, inst, Affinity::abi(&abi, self.cur.isa)); self.insert_spill(ebb, lv.value, reg); } } @@ -355,12 +350,8 @@ impl<'a> Context<'a> { // Update live ranges. self.liveness.move_def_locally(stack, inst); - self.liveness.extend_locally( - reg, - ebb, - inst, - &self.cur.func.layout, - ); + self.liveness + .extend_locally(reg, ebb, inst, &self.cur.func.layout); } } diff --git a/lib/codegen/src/regalloc/solver.rs b/lib/codegen/src/regalloc/solver.rs index 80c7f862e2..33217eab86 100644 --- a/lib/codegen/src/regalloc/solver.rs +++ b/lib/codegen/src/regalloc/solver.rs @@ -297,8 +297,7 @@ impl Move { #[cfg_attr(feature = "cargo-clippy", allow(wrong_self_convention))] fn from_reg(&self) -> Option<(RegClass, RegUnit)> { match *self { - Move::Reg { rc, from, .. } | - Move::Spill { rc, from, .. } => Some((rc, from)), + Move::Reg { rc, from, .. } | Move::Spill { rc, from, .. } => Some((rc, from)), Move::Fill { .. } => None, } } @@ -306,8 +305,7 @@ impl Move { /// Get the "to" register and register class, if possible. fn to_reg(&self) -> Option<(RegClass, RegUnit)> { match *self { - Move::Reg { rc, to, .. } | - Move::Fill { rc, to, .. } => Some((rc, to)), + Move::Reg { rc, to, .. } | Move::Fill { rc, to, .. } => Some((rc, to)), Move::Spill { .. } => None, } } @@ -316,8 +314,7 @@ impl Move { fn replace_to_reg(&mut self, new: RegUnit) -> RegUnit { mem::replace( match *self { - Move::Reg { ref mut to, .. } | - Move::Fill { ref mut to, .. } => to, + Move::Reg { ref mut to, .. } | Move::Fill { ref mut to, .. } => to, Move::Spill { .. } => panic!("No to register in a spill {}", self), }, new, @@ -348,18 +345,14 @@ impl Move { /// Get the value being moved. fn value(&self) -> Value { match *self { - Move::Reg { value, .. } | - Move::Fill { value, .. } | - Move::Spill { value, .. } => value, + Move::Reg { value, .. } | Move::Fill { value, .. } | Move::Spill { value, .. } => value, } } /// Get the associated register class. fn rc(&self) -> RegClass { match *self { - Move::Reg { rc, .. } | - Move::Fill { rc, .. } | - Move::Spill { rc, .. } => rc, + Move::Reg { rc, .. } | Move::Fill { rc, .. } | Move::Spill { rc, .. } => rc, } } } @@ -372,46 +365,40 @@ impl fmt::Display for Move { from, to, rc, - } => { - write!( - f, - "{}:{}({} -> {})", - value, - rc, - rc.info.display_regunit(from), - rc.info.display_regunit(to) - ) - } + } => write!( + f, + "{}:{}({} -> {})", + value, + rc, + rc.info.display_regunit(from), + rc.info.display_regunit(to) + ), Move::Spill { value, from, to_slot, rc, - } => { - write!( - f, - "{}:{}({} -> slot {})", - value, - rc, - rc.info.display_regunit(from), - to_slot - ) - } + } => write!( + f, + "{}:{}({} -> slot {})", + value, + rc, + rc.info.display_regunit(from), + to_slot + ), Move::Fill { value, from_slot, to, rc, - } => { - write!( - f, - "{}:{}(slot {} -> {})", - value, - rc, - from_slot, - rc.info.display_regunit(to) - ) - } + } => write!( + f, + "{}:{}(slot {} -> {})", + value, + rc, + from_slot, + rc.info.display_regunit(to) + ), } } } @@ -824,9 +811,8 @@ impl Solver { /// This is similar to `add_var`, except the value doesn't have a prior register assignment. pub fn add_def(&mut self, value: Value, constraint: RegClass, is_global: bool) { debug_assert!(self.inputs_done); - self.vars.push( - Variable::new_def(value, constraint, is_global), - ); + self.vars + .push(Variable::new_def(value, constraint, is_global)); } /// Clear the `is_global` flag on all solver variables. @@ -992,9 +978,8 @@ impl Solver { // Convert all of the fixed register assignments into moves, but omit the ones that are // already in the right register. - self.moves.extend(self.assignments.values().filter_map( - Move::with_assignment, - )); + self.moves + .extend(self.assignments.values().filter_map(Move::with_assignment)); if !(self.moves.is_empty()) { dbg!("collect_moves: {}", DisplayList(&self.moves)); @@ -1029,8 +1014,7 @@ impl Solver { if let Some(j) = self.moves[i..].iter().position(|m| match m.to_reg() { Some((rc, reg)) => avail.is_avail(rc, reg), None => true, - }) - { + }) { // This move can be executed now. self.moves.swap(i, i + j); let m = &self.moves[i]; @@ -1164,9 +1148,11 @@ mod tests { // Get a register class by name. fn rc_by_name(reginfo: &RegInfo, name: &str) -> RegClass { - reginfo.classes.iter().find(|rc| rc.name == name).expect( - "Can't find named register class.", - ) + reginfo + .classes + .iter() + .find(|rc| rc.name == name) + .expect("Can't find named register class.") } // Construct a register move. diff --git a/lib/codegen/src/regalloc/spilling.rs b/lib/codegen/src/regalloc/spilling.rs index 34eb48a031..65d43e7cb3 100644 --- a/lib/codegen/src/regalloc/spilling.rs +++ b/lib/codegen/src/regalloc/spilling.rs @@ -125,10 +125,8 @@ impl<'a> Context<'a> { self.process_spills(tracker); while let Some(inst) = self.cur.next_inst() { - if let Some(constraints) = - self.encinfo.operand_constraints( - self.cur.func.encodings[inst], - ) + if let Some(constraints) = self.encinfo + .operand_constraints(self.cur.func.encodings[inst]) { self.visit_inst(inst, ebb, constraints, tracker); } else { @@ -283,13 +281,11 @@ impl<'a> Context<'a> { dbg!("Need {} reg from {} throughs", op.regclass, throughs.len()); match self.spill_candidate(mask, throughs) { Some(cand) => self.spill_reg(cand), - None => { - panic!( - "Ran out of {} registers for {}", - op.regclass, - self.cur.display_inst(inst) - ) - } + None => panic!( + "Ran out of {} registers for {}", + op.regclass, + self.cur.display_inst(inst) + ), } } } @@ -349,12 +345,11 @@ impl<'a> Context<'a> { .constraints() .fixed_value_arguments(); let args = self.cur.func.dfg.inst_variable_args(inst); - for (idx, (abi, &arg)) in - self.cur.func.dfg.signatures[sig] - .params - .iter() - .zip(args) - .enumerate() + for (idx, (abi, &arg)) in self.cur.func.dfg.signatures[sig] + .params + .iter() + .zip(args) + .enumerate() { if abi.location.is_reg() { let (rci, spilled) = match self.liveness[arg].affinity { @@ -393,9 +388,9 @@ impl<'a> Context<'a> { } else if ru.fixed { // This is a fixed register use which doesn't necessarily require a copy. // Make a copy only if this is not the first use of the value. - self.reg_uses.get(i.wrapping_sub(1)).map_or(false, |ru2| { - ru2.value == ru.value - }) + self.reg_uses + .get(i.wrapping_sub(1)) + .map_or(false, |ru2| ru2.value == ru.value) } else { false }; @@ -430,13 +425,11 @@ impl<'a> Context<'a> { ) } { Some(cand) => self.spill_reg(cand), - None => { - panic!( - "Ran out of {} registers when inserting copy before {}", - rc, - self.cur.display_inst(inst) - ) - } + None => panic!( + "Ran out of {} registers when inserting copy before {}", + rc, + self.cur.display_inst(inst) + ), } } } @@ -501,9 +494,10 @@ impl<'a> Context<'a> { } // Assign a spill slot for the whole virtual register. - let ss = self.cur.func.stack_slots.make_spill_slot( - self.cur.func.dfg.value_type(value), - ); + let ss = self.cur + .func + .stack_slots + .make_spill_slot(self.cur.func.dfg.value_type(value)); for &v in self.virtregs.congruence_class(&value) { self.liveness.spill(v); self.cur.func.locations[v] = ValueLoc::Stack(ss); diff --git a/lib/codegen/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs index 58c7c380de..dbd4dceae7 100644 --- a/lib/codegen/src/regalloc/virtregs.rs +++ b/lib/codegen/src/regalloc/virtregs.rs @@ -101,10 +101,8 @@ impl VirtRegs { where 'a: 'b, { - self.get(*value).map_or_else( - || ref_slice(value), - |vr| self.values(vr), - ) + self.get(*value) + .map_or_else(|| ref_slice(value), |vr| self.values(vr)) } /// Check if `a` and `b` belong to the same congruence class. @@ -153,9 +151,9 @@ impl VirtRegs { }); // Determine the insertion position for `single`. - let index = match self.values(vreg).binary_search_by( - |&v| preorder.pre_cmp_def(v, single, func), - ) { + let index = match self.values(vreg) + .binary_search_by(|&v| preorder.pre_cmp_def(v, single, func)) + { Ok(_) => panic!("{} already in {}", single, vreg), Err(i) => i, }; @@ -181,9 +179,9 @@ impl VirtRegs { /// Allocate a new empty virtual register. fn alloc(&mut self) -> VirtReg { - self.unused_vregs.pop().unwrap_or_else(|| { - self.vregs.push(Default::default()) - }) + self.unused_vregs + .pop() + .unwrap_or_else(|| self.vregs.push(Default::default())) } /// Unify `values` into a single virtual register. diff --git a/lib/codegen/src/result.rs b/lib/codegen/src/result.rs index 9d347e351a..483b975ff4 100644 --- a/lib/codegen/src/result.rs +++ b/lib/codegen/src/result.rs @@ -12,10 +12,7 @@ pub enum CtonError { /// This always represents a bug, either in the code that generated IR for Cretonne, or a bug /// in Cretonne itself. #[fail(display = "Verifier error: {}", _0)] - Verifier( - #[cause] - verifier::Error - ), + Verifier(#[cause] verifier::Error), /// An implementation limit was exceeded. /// diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs index 422eedc342..85a3a46b2f 100644 --- a/lib/codegen/src/settings.rs +++ b/lib/codegen/src/settings.rs @@ -22,9 +22,9 @@ use constant_hash::{probe, simple_hash}; use isa::TargetIsa; +use std::boxed::Box; use std::fmt; use std::result; -use std::boxed::Box; use std::str; /// A string-based configurator for settings groups. diff --git a/lib/codegen/src/simple_gvn.rs b/lib/codegen/src/simple_gvn.rs index ae8dd387d6..141199b3d9 100644 --- a/lib/codegen/src/simple_gvn.rs +++ b/lib/codegen/src/simple_gvn.rs @@ -9,9 +9,9 @@ use timing; /// Test whether the given opcode is unsafe to even consider for GVN. fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { - opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || - opcode.is_return() || opcode.can_trap() || opcode.other_side_effects() || - opcode.can_store() || opcode.can_load() || opcode.writes_cpu_flags() + opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || opcode.is_return() + || opcode.can_trap() || opcode.other_side_effects() || opcode.can_store() + || opcode.can_load() || opcode.writes_cpu_flags() } /// Perform simple GVN on `func`. diff --git a/lib/codegen/src/stack_layout.rs b/lib/codegen/src/stack_layout.rs index 3b07ee154a..e40499f976 100644 --- a/lib/codegen/src/stack_layout.rs +++ b/lib/codegen/src/stack_layout.rs @@ -55,9 +55,9 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result { + StackSlotKind::SpillSlot + | StackSlotKind::ExplicitSlot + | StackSlotKind::EmergencySlot => { // Determine the smallest alignment of any explicit or spill slot. min_align = slot.alignment(min_align); } @@ -73,20 +73,19 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result { + StackSlotKind::SpillSlot + | StackSlotKind::ExplicitSlot + | StackSlotKind::EmergencySlot => { if slot.alignment(alignment) != min_align { continue; } } - StackSlotKind::IncomingArg | - StackSlotKind::OutgoingArg => continue, + StackSlotKind::IncomingArg | StackSlotKind::OutgoingArg => continue, } - offset = offset.checked_sub(slot.size as StackOffset).ok_or( - CtonError::ImplLimitExceeded, - )?; + offset = offset + .checked_sub(slot.size as StackOffset) + .ok_or(CtonError::ImplLimitExceeded)?; // Aligning the negative offset can never cause overflow. We're only clearing bits. offset &= -(min_align as StackOffset); @@ -98,9 +97,9 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result CssaVerifier<'a> { } // Enforce topological ordering of defs in the virtual register. - if self.preorder.dominates(def_ebb, prev_ebb) && - self.domtree.dominates(def, prev_def, &self.func.layout) + if self.preorder.dominates(def_ebb, prev_ebb) + && self.domtree.dominates(def, prev_def, &self.func.layout) { return err!( val, @@ -112,8 +112,8 @@ impl<'a> CssaVerifier<'a> { let prev_def: ExpandedProgramPoint = self.func.dfg.value_def(prev_val).into(); let prev_ebb = self.func.layout.pp_ebb(prev_def); - if self.preorder.dominates(prev_ebb, def_ebb) && - self.domtree.dominates(prev_def, def, &self.func.layout) + if self.preorder.dominates(prev_ebb, def_ebb) + && self.domtree.dominates(prev_def, def, &self.func.layout) { let ctx = self.liveness.context(&self.func.layout); if self.liveness[prev_val].overlaps_def(def, def_ebb, ctx) { diff --git a/lib/codegen/src/verifier/liveness.rs b/lib/codegen/src/verifier/liveness.rs index eeb041c88a..167225f24f 100644 --- a/lib/codegen/src/verifier/liveness.rs +++ b/lib/codegen/src/verifier/liveness.rs @@ -127,8 +127,8 @@ impl<'a> LivenessVerifier<'a> { let ctx = self.liveness.context(&self.func.layout); // Check if `inst` is in the def range, not including the def itself. - if ctx.order.cmp(lr.def(), inst) == Ordering::Less && - ctx.order.cmp(inst, lr.def_local_end()) != Ordering::Greater + if ctx.order.cmp(lr.def(), inst) == Ordering::Less + && ctx.order.cmp(inst, lr.def_local_end()) != Ordering::Greater { return true; } diff --git a/lib/codegen/src/verifier/locations.rs b/lib/codegen/src/verifier/locations.rs index 7fdfcf96ce..89e2b09e9a 100644 --- a/lib/codegen/src/verifier/locations.rs +++ b/lib/codegen/src/verifier/locations.rs @@ -89,9 +89,9 @@ impl<'a> LocationVerifier<'a> { enc: isa::Encoding, divert: &RegDiversions, ) -> Result { - let constraints = self.encinfo.operand_constraints(enc).expect( - "check_enc_constraints requires a legal encoding", - ); + let constraints = self.encinfo + .operand_constraints(enc) + .expect("check_enc_constraints requires a legal encoding"); if constraints.satisfied(inst, divert, self.func) { return Ok(()); @@ -235,8 +235,8 @@ impl<'a> LocationVerifier<'a> { /// Update diversions to reflect the current instruction and check their consistency. fn update_diversions(&self, inst: ir::Inst, divert: &mut RegDiversions) -> Result { let (arg, src) = match self.func.dfg[inst] { - ir::InstructionData::RegMove { arg, src, .. } | - ir::InstructionData::RegSpill { arg, src, .. } => (arg, ir::ValueLoc::Reg(src)), + ir::InstructionData::RegMove { arg, src, .. } + | ir::InstructionData::RegSpill { arg, src, .. } => (arg, ir::ValueLoc::Reg(src)), ir::InstructionData::RegFill { arg, src, .. } => (arg, ir::ValueLoc::Stack(src)), _ => return Ok(()), }; @@ -275,12 +275,10 @@ impl<'a> LocationVerifier<'a> { let dfg = &self.func.dfg; match dfg.analyze_branch(inst) { - NotABranch => { - panic!( - "No branch information for {}", - dfg.display_inst(inst, self.isa) - ) - } + NotABranch => panic!( + "No branch information for {}", + dfg.display_inst(inst, self.isa) + ), SingleDest(ebb, _) => { for d in divert.all() { let lr = &liveness[d.value]; diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs index 490222272e..f8644034b2 100644 --- a/lib/codegen/src/verifier/mod.rs +++ b/lib/codegen/src/verifier/mod.rs @@ -237,9 +237,8 @@ impl<'a> Verifier<'a> { let fixed_results = inst_data.opcode().constraints().fixed_results(); // var_results is 0 if we aren't a call instruction - let var_results = dfg.call_signature(inst).map_or(0, |sig| { - dfg.signatures[sig].returns.len() - }); + let var_results = dfg.call_signature(inst) + .map_or(0, |sig| dfg.signatures[sig].returns.len()); let total_results = fixed_results + var_results; // All result values for multi-valued instructions are created @@ -281,23 +280,23 @@ impl<'a> Verifier<'a> { destination, ref args, .. - } | - Branch { + } + | Branch { destination, ref args, .. - } | - BranchInt { + } + | BranchInt { destination, ref args, .. - } | - BranchFloat { + } + | BranchFloat { destination, ref args, .. - } | - BranchIcmp { + } + | BranchIcmp { destination, ref args, .. @@ -308,19 +307,22 @@ impl<'a> Verifier<'a> { BranchTable { table, .. } => { self.verify_jump_table(inst, table)?; } - Call { func_ref, ref args, .. } => { + Call { + func_ref, ref args, .. + } => { self.verify_func_ref(inst, func_ref)?; self.verify_value_list(inst, args)?; } - CallIndirect { sig_ref, ref args, .. } => { + CallIndirect { + sig_ref, ref args, .. + } => { self.verify_sig_ref(inst, sig_ref)?; self.verify_value_list(inst, args)?; } FuncAddr { func_ref, .. } => { self.verify_func_ref(inst, func_ref)?; } - StackLoad { stack_slot, .. } | - StackStore { stack_slot, .. } => { + StackLoad { stack_slot, .. } | StackStore { stack_slot, .. } => { self.verify_stack_slot(inst, stack_slot)?; } UnaryGlobalVar { global_var, .. } => { @@ -343,31 +345,31 @@ impl<'a> Verifier<'a> { } // Exhaustive list so we can't forget to add new formats - Unary { .. } | - UnaryImm { .. } | - UnaryIeee32 { .. } | - UnaryIeee64 { .. } | - UnaryBool { .. } | - Binary { .. } | - BinaryImm { .. } | - Ternary { .. } | - InsertLane { .. } | - ExtractLane { .. } | - IntCompare { .. } | - IntCompareImm { .. } | - IntCond { .. } | - FloatCompare { .. } | - FloatCond { .. } | - IntSelect { .. } | - Load { .. } | - Store { .. } | - RegMove { .. } | - CopySpecial { .. } | - Trap { .. } | - CondTrap { .. } | - IntCondTrap { .. } | - FloatCondTrap { .. } | - NullAry { .. } => {} + Unary { .. } + | UnaryImm { .. } + | UnaryIeee32 { .. } + | UnaryIeee64 { .. } + | UnaryBool { .. } + | Binary { .. } + | BinaryImm { .. } + | Ternary { .. } + | InsertLane { .. } + | ExtractLane { .. } + | IntCompare { .. } + | IntCompareImm { .. } + | IntCond { .. } + | FloatCompare { .. } + | FloatCond { .. } + | IntSelect { .. } + | Load { .. } + | Store { .. } + | RegMove { .. } + | CopySpecial { .. } + | Trap { .. } + | CondTrap { .. } + | IntCondTrap { .. } + | FloatCondTrap { .. } + | NullAry { .. } => {} } Ok(()) @@ -480,11 +482,8 @@ impl<'a> Verifier<'a> { } // Defining instruction dominates the instruction that uses the value. if is_reachable { - if !self.expected_domtree.dominates( - def_inst, - loc_inst, - &self.func.layout, - ) + if !self.expected_domtree + .dominates(def_inst, loc_inst, &self.func.layout) { return err!(loc_inst, "uses value from non-dominating {}", def_inst); } @@ -513,12 +512,9 @@ impl<'a> Verifier<'a> { ); } // The defining EBB dominates the instruction using this value. - if is_reachable && - !self.expected_domtree.dominates( - ebb, - loc_inst, - &self.func.layout, - ) + if is_reachable + && !self.expected_domtree + .dominates(ebb, loc_inst, &self.func.layout) { return err!(loc_inst, "uses value arg from non-dominating {}", ebb); } @@ -542,13 +538,11 @@ impl<'a> Verifier<'a> { Ok(()) } } - ValueDef::Param(_, _) => { - err!( - loc_inst, - "instruction result {} is not defined by the instruction", - v - ) - } + ValueDef::Param(_, _) => err!( + loc_inst, + "instruction result {} is not defined by the instruction", + v + ), } } @@ -576,12 +570,11 @@ impl<'a> Verifier<'a> { "incorrect number of Ebbs in postorder traversal" ); } - for (index, (&test_ebb, &true_ebb)) in - domtree - .cfg_postorder() - .iter() - .zip(self.expected_domtree.cfg_postorder().iter()) - .enumerate() + for (index, (&test_ebb, &true_ebb)) in domtree + .cfg_postorder() + .iter() + .zip(self.expected_domtree.cfg_postorder().iter()) + .enumerate() { if test_ebb != true_ebb { return err!( @@ -595,11 +588,8 @@ impl<'a> Verifier<'a> { } // We verify rpo_cmp on pairs of adjacent ebbs in the postorder for (&prev_ebb, &next_ebb) in domtree.cfg_postorder().iter().adjacent_pairs() { - if self.expected_domtree.rpo_cmp( - prev_ebb, - next_ebb, - &self.func.layout, - ) != Ordering::Greater + if self.expected_domtree + .rpo_cmp(prev_ebb, next_ebb, &self.func.layout) != Ordering::Greater { return err!( next_ebb, @@ -737,9 +727,11 @@ impl<'a> Verifier<'a> { fn typecheck_variable_args(&self, inst: Inst) -> Result { match self.func.dfg.analyze_branch(inst) { BranchInfo::SingleDest(ebb, _) => { - let iter = self.func.dfg.ebb_params(ebb).iter().map(|&v| { - self.func.dfg.value_type(v) - }); + let iter = self.func + .dfg + .ebb_params(ebb) + .iter() + .map(|&v| self.func.dfg.value_type(v)); self.typecheck_variable_args_iterator(inst, iter)?; } BranchInfo::Table(table) => { @@ -761,16 +753,18 @@ impl<'a> Verifier<'a> { match self.func.dfg[inst].analyze_call(&self.func.dfg.value_lists) { CallInfo::Direct(func_ref, _) => { let sig_ref = self.func.dfg.ext_funcs[func_ref].signature; - let arg_types = self.func.dfg.signatures[sig_ref].params.iter().map(|a| { - a.value_type - }); + let arg_types = self.func.dfg.signatures[sig_ref] + .params + .iter() + .map(|a| a.value_type); self.typecheck_variable_args_iterator(inst, arg_types)?; self.check_outgoing_args(inst, sig_ref)?; } CallInfo::Indirect(sig_ref, _) => { - let arg_types = self.func.dfg.signatures[sig_ref].params.iter().map(|a| { - a.value_type - }); + let arg_types = self.func.dfg.signatures[sig_ref] + .params + .iter() + .map(|a| a.value_type); self.typecheck_variable_args_iterator(inst, arg_types)?; self.check_outgoing_args(inst, sig_ref)?; } @@ -1047,8 +1041,7 @@ impl<'a> Verifier<'a> { &self.func, &self.func.dfg[inst], self.func.dfg.ctrl_typevar(inst), - ) - { + ) { if !possible_encodings.is_empty() { possible_encodings.push_str(", "); multiple_encodings = true; @@ -1119,8 +1112,7 @@ impl<'a> Verifier<'a> { fn verify_return_at_end(&self) -> Result { for ebb in self.func.layout.ebbs() { let inst = self.func.layout.last_inst(ebb).unwrap(); - if self.func.dfg[inst].opcode().is_return() && - Some(ebb) != self.func.layout.last_ebb() + if self.func.dfg[inst].opcode().is_return() && Some(ebb) != self.func.layout.last_ebb() { return err!(inst, "Internal return not allowed with return_at_end=1"); } @@ -1155,8 +1147,8 @@ impl<'a> Verifier<'a> { mod tests { use super::{Error, Verifier}; use entity::EntityList; - use ir::instructions::{InstructionData, Opcode}; use ir::Function; + use ir::instructions::{InstructionData, Opcode}; use settings; macro_rules! assert_err_with_msg { diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs index 9a5be48b6f..1bd7bc3b3c 100644 --- a/lib/codegen/src/write.rs +++ b/lib/codegen/src/write.rs @@ -346,10 +346,12 @@ pub fn write_operands( write_ebb_args(w, &args[2..]) } BranchTable { arg, table, .. } => write!(w, " {}, {}", arg, table), - Call { func_ref, ref args, .. } => { - write!(w, " {}({})", func_ref, DisplayValues(args.as_slice(pool))) - } - CallIndirect { sig_ref, ref args, .. } => { + Call { + func_ref, ref args, .. + } => write!(w, " {}({})", func_ref, DisplayValues(args.as_slice(pool))), + CallIndirect { + sig_ref, ref args, .. + } => { let args = args.as_slice(pool); write!( w, @@ -360,7 +362,9 @@ pub fn write_operands( ) } FuncAddr { func_ref, .. } => write!(w, " {}", func_ref), - StackLoad { stack_slot, offset, .. } => write!(w, " {}{}", stack_slot, offset), + StackLoad { + stack_slot, offset, .. + } => write!(w, " {}{}", stack_slot, offset), StackStore { arg, stack_slot, @@ -368,7 +372,9 @@ pub fn write_operands( .. } => write!(w, " {}, {}{}", arg, stack_slot, offset), HeapAddr { heap, arg, imm, .. } => write!(w, " {}, {}, {}", heap, arg, imm), - Load { flags, arg, offset, .. } => write!(w, "{} {}{}", flags, arg, offset), + Load { + flags, arg, offset, .. + } => write!(w, "{} {}{}", flags, arg, offset), LoadComplex { flags, ref args, @@ -383,7 +389,6 @@ pub fn write_operands( DisplayValuesWithDelimiter(&args, '+'), offset ) - } Store { flags, @@ -452,8 +457,12 @@ pub fn write_operands( } Trap { code, .. } => write!(w, " {}", code), CondTrap { arg, code, .. } => write!(w, " {}, {}", arg, code), - IntCondTrap { cond, arg, code, .. } => write!(w, " {} {}, {}", cond, arg, code), - FloatCondTrap { cond, arg, code, .. } => write!(w, " {} {}, {}", cond, arg, code), + IntCondTrap { + cond, arg, code, .. + } => write!(w, " {} {}, {}", cond, arg, code), + FloatCondTrap { + cond, arg, code, .. + } => write!(w, " {} {}, {}", cond, arg, code), } } diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs index 693e5d4f8c..6b27064c7d 100644 --- a/lib/entity/src/lib.rs +++ b/lib/entity/src/lib.rs @@ -33,19 +33,10 @@ #![warn(unused_import_braces)] #![cfg_attr(feature = "std", warn(unstable_features))] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] #![cfg_attr(feature = "cargo-clippy", - allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature="cargo-clippy", warn( - float_arithmetic, - mut_mut, - nonminimal_bool, - option_map_unwrap_or, - option_map_unwrap_or_else, - print_stdout, - unicode_not_nfc, - use_self, - ))] - + warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, + option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] // Turns on no_std and alloc features if std is not available. #![cfg_attr(not(feature = "std"), no_std)] #![cfg_attr(not(feature = "std"), feature(alloc))] diff --git a/lib/entity/src/map.rs b/lib/entity/src/map.rs index 8c09cd6486..ad8dca7e01 100644 --- a/lib/entity/src/map.rs +++ b/lib/entity/src/map.rs @@ -1,10 +1,10 @@ //! Densely numbered entity references as mapping keys. -use {EntityRef, Iter, IterMut, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; use std::slice; use std::vec::Vec; +use {EntityRef, Iter, IterMut, Keys}; /// A mapping `K -> V` for densely indexed entity references. /// diff --git a/lib/entity/src/packed_option.rs b/lib/entity/src/packed_option.rs index 869c57bb0d..a33d9ed58e 100644 --- a/lib/entity/src/packed_option.rs +++ b/lib/entity/src/packed_option.rs @@ -33,7 +33,11 @@ impl PackedOption { /// Expand the packed option into a normal `Option`. pub fn expand(self) -> Option { - if self.is_none() { None } else { Some(self.0) } + if self.is_none() { + None + } else { + Some(self.0) + } } /// Maps a `PackedOption` to `Option` by applying a function to a contained value. diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs index 805ccc6866..36b76f09d6 100644 --- a/lib/entity/src/primary.rs +++ b/lib/entity/src/primary.rs @@ -1,9 +1,9 @@ //! Densely numbered entity references as mapping keys. -use {EntityRef, Iter, IterMut, Keys}; use std::marker::PhantomData; use std::ops::{Index, IndexMut}; use std::slice; use std::vec::Vec; +use {EntityRef, Iter, IterMut, Keys}; /// A primary mapping `K -> V` allocating dense entity references. /// diff --git a/lib/entity/src/set.rs b/lib/entity/src/set.rs index 392813401c..21f518eb66 100644 --- a/lib/entity/src/set.rs +++ b/lib/entity/src/set.rs @@ -1,8 +1,8 @@ //! Densely numbered entity references as set keys. -use {EntityRef, Keys}; use std::marker::PhantomData; use std::vec::Vec; +use {EntityRef, Keys}; /// A set of `K` for densely indexed entity references. /// diff --git a/lib/entity/src/sparse.rs b/lib/entity/src/sparse.rs index 95d0bba41b..28cbd80ad8 100644 --- a/lib/entity/src/sparse.rs +++ b/lib/entity/src/sparse.rs @@ -7,11 +7,11 @@ //! > Briggs, Torczon, *An efficient representation for sparse sets*, //! ACM Letters on Programming Languages and Systems, Volume 2, Issue 1-4, March-Dec. 1993. -use {EntityMap, EntityRef}; use std::mem; use std::slice; use std::u32; use std::vec::Vec; +use {EntityMap, EntityRef}; /// Trait for extracting keys from values stored in a `SparseMap`. /// diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index d338daf89b..571ed1db77 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -1,13 +1,13 @@ //! Defines `FaerieBackend`. use container; -use cretonne_codegen::binemit::{Addend, CodeOffset, Reloc, RelocSink, NullTrapSink}; +use cretonne_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSink}; use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::{self, binemit, ir}; -use cretonne_module::{Backend, DataContext, Linkage, ModuleNamespace, Init, DataDescription, - ModuleError}; -use failure::Error; +use cretonne_module::{Backend, DataContext, DataDescription, Init, Linkage, ModuleError, + ModuleNamespace}; use faerie; +use failure::Error; use std::fs::File; use target; use traps::{FaerieTrapManifest, FaerieTrapSink}; @@ -33,7 +33,6 @@ pub struct FaerieBuilder { libcall_names: Box String>, } - impl FaerieBuilder { /// Create a new `FaerieBuilder` using the given Cretonne target, that /// can be passed to @@ -89,7 +88,6 @@ impl FaerieBuilder { } } - /// A `FaerieBackend` implements `Backend` and emits ".o" files using the `faerie` library. pub struct FaerieBackend { isa: Box, @@ -192,9 +190,9 @@ impl Backend for FaerieBackend { } } - self.artifact.define(name, code).expect( - "inconsistent declaration", - ); + self.artifact + .define(name, code) + .expect("inconsistent declaration"); Ok(FaerieCompiledFunction {}) } @@ -239,8 +237,7 @@ impl Backend for FaerieBackend { } for &(offset, id, addend) in data_relocs { debug_assert_eq!( - addend, - 0, + addend, 0, "faerie doesn't support addends in data section relocations yet" ); let to = &namespace.get_data_decl(&data_decls[id]).name; @@ -253,9 +250,9 @@ impl Backend for FaerieBackend { .map_err(|e| ModuleError::Backend(format!("{}", e)))?; } - self.artifact.define(name, bytes).expect( - "inconsistent declaration", - ); + self.artifact + .define(name, bytes) + .expect("inconsistent declaration"); Ok(FaerieCompiledData {}) } @@ -346,25 +343,20 @@ fn translate_function_linkage(linkage: Linkage) -> faerie::Decl { fn translate_data_linkage(linkage: Linkage, writable: bool) -> faerie::Decl { match linkage { Linkage::Import => faerie::Decl::DataImport, - Linkage::Local => { - faerie::Decl::Data { - global: false, - writeable: writable, - } - } - Linkage::Export => { - faerie::Decl::Data { - global: true, - writeable: writable, - } - } + Linkage::Local => faerie::Decl::Data { + global: false, + writeable: writable, + }, + Linkage::Export => faerie::Decl::Data { + global: true, + writeable: writable, + }, Linkage::Preemptible => { unimplemented!("faerie doesn't support preemptible globals yet"); } } } - struct FaerieRelocSink<'a> { format: container::Format, artifact: &'a mut faerie::Artifact, diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs index b56a3836f3..6cec1c9e52 100644 --- a/lib/faerie/src/lib.rs +++ b/lib/faerie/src/lib.rs @@ -5,18 +5,10 @@ #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] #![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] #![cfg_attr(feature = "cargo-clippy", - allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature="cargo-clippy", warn( - float_arithmetic, - mut_mut, - nonminimal_bool, - option_map_unwrap_or, - option_map_unwrap_or_else, - print_stdout, - unicode_not_nfc, - use_self, - ))] + warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, + option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] extern crate cretonne_codegen; extern crate cretonne_module; @@ -29,5 +21,5 @@ mod container; mod target; pub mod traps; -pub use backend::{FaerieBuilder, FaerieBackend, FaerieProduct, FaerieTrapCollection}; +pub use backend::{FaerieBackend, FaerieBuilder, FaerieProduct, FaerieTrapCollection}; pub use container::Format; diff --git a/lib/faerie/src/target.rs b/lib/faerie/src/target.rs index ac597fe1a9..d6e3749430 100644 --- a/lib/faerie/src/target.rs +++ b/lib/faerie/src/target.rs @@ -13,8 +13,9 @@ pub fn translate(isa: &isa::TargetIsa) -> Result { }), "arm32" => Ok(Target::ARMv7), "arm64" => Ok(Target::ARM64), - _ => Err(ModuleError::Backend( - format!("unsupported faerie isa: {}", name), - )), + _ => Err(ModuleError::Backend(format!( + "unsupported faerie isa: {}", + name + ))), } } diff --git a/lib/faerie/src/traps.rs b/lib/faerie/src/traps.rs index 3874583ee7..e9526b757e 100644 --- a/lib/faerie/src/traps.rs +++ b/lib/faerie/src/traps.rs @@ -1,7 +1,7 @@ //! Faerie trap manifests record every `TrapCode` that cretonne outputs during code generation, //! for every function in the module. This data may be useful at runtime. -use cretonne_codegen::{ir, binemit}; +use cretonne_codegen::{binemit, ir}; /// Record of the arguments cretonne passes to `TrapSink::trap` pub struct FaerieTrapSite { @@ -23,7 +23,6 @@ pub struct FaerieTrapSink { pub sites: Vec, } - impl FaerieTrapSink { /// Create an empty `FaerieTrapSink` pub fn new(name: &str, code_size: u32) -> Self { diff --git a/lib/filetests/src/concurrent.rs b/lib/filetests/src/concurrent.rs index ac45624114..d9ee4f3c07 100644 --- a/lib/filetests/src/concurrent.rs +++ b/lib/filetests/src/concurrent.rs @@ -47,9 +47,7 @@ impl ConcurrentRunner { heartbeat_thread(reply_tx.clone()); let handles = (0..num_cpus::get()) - .map(|num| { - worker_thread(num, request_mutex.clone(), reply_tx.clone()) - }) + .map(|num| worker_thread(num, request_mutex.clone(), reply_tx.clone())) .collect(); Self { @@ -101,8 +99,10 @@ impl ConcurrentRunner { fn heartbeat_thread(replies: Sender) -> thread::JoinHandle<()> { thread::Builder::new() .name("heartbeat".to_string()) - .spawn(move || while replies.send(Reply::Tick).is_ok() { - thread::sleep(Duration::from_secs(1)); + .spawn(move || { + while replies.send(Reply::Tick).is_ok() { + thread::sleep(Duration::from_secs(1)); + } }) .unwrap() } diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs index 30a2edc68c..e0dca34cc2 100644 --- a/lib/filetests/src/lib.rs +++ b/lib/filetests/src/lib.rs @@ -9,14 +9,9 @@ type_complexity, // Rustfmt 0.9.0 is at odds with this lint: block_in_if_condition_stmt))] -#![cfg_attr(feature="cargo-clippy", warn( - mut_mut, - nonminimal_bool, - option_map_unwrap_or, - option_map_unwrap_or_else, - unicode_not_nfc, - use_self, - ))] +#![cfg_attr(feature = "cargo-clippy", + warn(mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, + unicode_not_nfc, use_self))] #[macro_use(dbg)] extern crate cretonne_codegen; diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs index d4865b92e9..1e544ba0aa 100644 --- a/lib/filetests/src/runner.rs +++ b/lib/filetests/src/runner.rs @@ -40,15 +40,13 @@ impl Display for QueueEntry { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { let p = self.path.to_string_lossy(); match self.state { - State::Done(Ok(dur)) => { - write!( - f, - "{}.{:03} {}", - dur.as_secs(), - dur.subsec_nanos() / 1_000_000, - p - ) - } + State::Done(Ok(dur)) => write!( + f, + "{}.{:03} {}", + dur.as_secs(), + dur.subsec_nanos() / 1_000_000, + p + ), State::Done(Err(ref e)) => write!(f, "FAIL {}: {}", p, e), _ => write!(f, "{}", p), } @@ -180,7 +178,11 @@ impl TestRunner { /// Report on the next in-order job, if it's done. fn report_job(&self) -> bool { let jobid = self.reported_tests; - if let Some(&QueueEntry { state: State::Done(ref result), .. }) = self.tests.get(jobid) { + if let Some(&QueueEntry { + state: State::Done(ref result), + .. + }) = self.tests.get(jobid) + { if self.verbose || result.is_err() { println!("{}", self.tests[jobid]); } @@ -283,7 +285,10 @@ impl TestRunner { let mut times = self.tests .iter() .filter_map(|entry| match *entry { - QueueEntry { state: State::Done(Ok(dur)), .. } => Some(dur), + QueueEntry { + state: State::Done(Ok(dur)), + .. + } => Some(dur), _ => None, }) .collect::>(); @@ -312,10 +317,12 @@ impl TestRunner { } for t in self.tests.iter().filter(|entry| match **entry { - QueueEntry { state: State::Done(Ok(dur)), .. } => dur > cut, + QueueEntry { + state: State::Done(Ok(dur)), + .. + } => dur > cut, _ => false, - }) - { + }) { println!("slow: {}", t) } } diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs index 5389cfdb95..7688785a57 100644 --- a/lib/filetests/src/runone.rs +++ b/lib/filetests/src/runone.rs @@ -129,15 +129,11 @@ fn run_one_test<'a>( // Should we run the verifier before this test? if !context.verified && test.needs_verifier() { - verify_function(&func, context.flags_or_isa()).map_err( - |e| { - pretty_verifier_error(&func, isa, &e) - }, - )?; + verify_function(&func, context.flags_or_isa()) + .map_err(|e| pretty_verifier_error(&func, isa, &e))?; context.verified = true; } - test.run(func, context).map_err( - |e| format!("{}: {}", name, e), - ) + test.run(func, context) + .map_err(|e| format!("{}: {}", name, e)) } diff --git a/lib/filetests/src/subtest.rs b/lib/filetests/src/subtest.rs index 2ed560cb3f..18904d99b9 100644 --- a/lib/filetests/src/subtest.rs +++ b/lib/filetests/src/subtest.rs @@ -70,16 +70,16 @@ pub trait SubTest { /// Run filecheck on `text`, using directives extracted from `context`. pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { let checker = build_filechecker(context)?; - if checker.check(text, NO_VARIABLES).map_err(|e| { - format!("filecheck: {}", e) - })? + if checker + .check(text, NO_VARIABLES) + .map_err(|e| format!("filecheck: {}", e))? { Ok(()) } else { // Filecheck mismatch. Emit an explanation as output. - let (_, explain) = checker.explain(text, NO_VARIABLES).map_err(|e| { - format!("explain: {}", e) - })?; + let (_, explain) = checker + .explain(text, NO_VARIABLES) + .map_err(|e| format!("explain: {}", e))?; Err(format!("filecheck failed:\n{}{}", checker, explain)) } } @@ -89,14 +89,14 @@ pub fn build_filechecker(context: &Context) -> Result { let mut builder = CheckerBuilder::new(); // Preamble comments apply to all functions. for comment in context.preamble_comments { - builder.directive(comment.text).map_err(|e| { - format!("filecheck: {}", e) - })?; + builder + .directive(comment.text) + .map_err(|e| format!("filecheck: {}", e))?; } for comment in &context.details.comments { - builder.directive(comment.text).map_err(|e| { - format!("filecheck: {}", e) - })?; + builder + .directive(comment.text) + .map_err(|e| format!("filecheck: {}", e))?; } Ok(builder.finish()) } diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index 332ef3e574..f6a91f61a1 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -138,9 +138,9 @@ impl SubTest for TestBinEmit { &func.dfg[inst], func.dfg.ctrl_typevar(inst), ).filter(|e| { - let recipe_constraints = &encinfo.constraints[e.recipe()]; - recipe_constraints.satisfied(inst, &divert, &func) - }); + let recipe_constraints = &encinfo.constraints[e.recipe()]; + recipe_constraints.satisfied(inst, &divert, &func) + }); if opt_level == OptLevel::Best { // Get the smallest legal encoding @@ -149,8 +149,7 @@ impl SubTest for TestBinEmit { // If not optimizing, just use the first encoding. legal_encodings.next() } - } - { + } { func.encodings[inst] = enc; } } @@ -159,9 +158,8 @@ impl SubTest for TestBinEmit { } // Relax branches and compute EBB offsets based on the encodings. - let code_size = binemit::relax_branches(&mut func, isa).map_err(|e| { - pretty_error(&func, context.isa, e) - })?; + let code_size = binemit::relax_branches(&mut func, isa) + .map_err(|e| pretty_error(&func, context.isa, e))?; // Collect all of the 'bin:' directives on instructions. let mut bins = HashMap::new(); @@ -181,8 +179,7 @@ impl SubTest for TestBinEmit { _ => { return Err(format!( "'bin:' directive on non-inst {}: {}", - comment.entity, - comment.text + comment.entity, comment.text )) } } @@ -198,8 +195,7 @@ impl SubTest for TestBinEmit { divert.clear(); // Correct header offsets should have been computed by `relax_branches()`. assert_eq!( - sink.offset, - func.offsets[ebb], + sink.offset, func.offsets[ebb], "Inconsistent {} header offset", ebb ); @@ -211,9 +207,10 @@ impl SubTest for TestBinEmit { // Send legal encodings into the emitter. if enc.is_legal() { // Generate a better error message if output locations are not specified. - if let Some(&v) = func.dfg.inst_results(inst).iter().find(|&&v| { - !func.locations[v].is_assigned() - }) + if let Some(&v) = func.dfg + .inst_results(inst) + .iter() + .find(|&&v| !func.locations[v].is_assigned()) { return Err(format!( "Missing register/stack slot for {} in {}", @@ -239,9 +236,10 @@ impl SubTest for TestBinEmit { if !enc.is_legal() { // A possible cause of an unencoded instruction is a missing location for // one of the input operands. - if let Some(&v) = func.dfg.inst_args(inst).iter().find(|&&v| { - !func.locations[v].is_assigned() - }) + if let Some(&v) = func.dfg + .inst_args(inst) + .iter() + .find(|&&v| !func.locations[v].is_assigned()) { return Err(format!( "Missing register/stack slot for {} in {}", @@ -287,8 +285,7 @@ impl SubTest for TestBinEmit { if sink.offset != code_size { return Err(format!( "Expected code size {}, got {}", - code_size, - sink.offset + code_size, sink.offset )); } diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs index c4b737bfc6..f9b5deebd8 100644 --- a/lib/filetests/src/test_compile.rs +++ b/lib/filetests/src/test_compile.rs @@ -3,8 +3,8 @@ //! The `compile` test command runs each function through the full code generator pipeline use cretonne_codegen; -use cretonne_codegen::{binemit, ir}; use cretonne_codegen::print_errors::pretty_error; +use cretonne_codegen::{binemit, ir}; use cretonne_reader::TestCommand; use std::borrow::Cow; use std::fmt::Write; @@ -41,9 +41,9 @@ impl SubTest for TestCompile { let mut comp_ctx = cretonne_codegen::Context::new(); comp_ctx.func = func.into_owned(); - let code_size = comp_ctx.compile(isa).map_err(|e| { - pretty_error(&comp_ctx.func, context.isa, e) - })?; + let code_size = comp_ctx + .compile(isa) + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; dbg!( "Generated {} bytes of code:\n{}", @@ -62,15 +62,13 @@ impl SubTest for TestCompile { if sink.offset != code_size { return Err(format!( "Expected code size {}, got {}", - code_size, - sink.offset + code_size, sink.offset )); } // Run final code through filecheck. let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))) - .map_err(|e| e.to_string())?; + write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))).map_err(|e| e.to_string())?; run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_dce.rs b/lib/filetests/src/test_dce.rs index 14d535aca4..13bad80a7f 100644 --- a/lib/filetests/src/test_dce.rs +++ b/lib/filetests/src/test_dce.rs @@ -40,14 +40,12 @@ impl SubTest for TestDCE { comp_ctx.flowgraph(); comp_ctx.compute_loop_analysis(); - comp_ctx.dce(context.flags_or_isa()).map_err(|e| { - pretty_error(&comp_ctx.func, context.isa, Into::into(e)) - })?; + comp_ctx + .dce(context.flags_or_isa()) + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?; let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func).map_err( - |e| e.to_string(), - )?; + write!(&mut text, "{}", &comp_ctx.func).map_err(|e| e.to_string())?; run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_domtree.rs b/lib/filetests/src/test_domtree.rs index 8bf4a36cfe..cdc2a37ae4 100644 --- a/lib/filetests/src/test_domtree.rs +++ b/lib/filetests/src/test_domtree.rs @@ -55,8 +55,7 @@ impl SubTest for TestDomtree { _ => { return Err(format!( "annotation on non-inst {}: {}", - comment.entity, - comment.text + comment.entity, comment.text )) } }; @@ -77,17 +76,14 @@ impl SubTest for TestDomtree { return Err(format!( "mismatching idoms for {}:\n\ want: {}, got: {}", - src_ebb, - inst, - got_inst + src_ebb, inst, got_inst )); } None => { return Err(format!( "mismatching idoms for {}:\n\ want: {}, got: unreachable", - src_ebb, - inst + src_ebb, inst )); } _ => {} @@ -98,16 +94,16 @@ impl SubTest for TestDomtree { // Now we know that everything in `expected` is consistent with `domtree`. // All other EBB's should be either unreachable or the entry block. - for ebb in func.layout.ebbs().skip(1).filter( - |ebb| !expected.contains_key(ebb), - ) + for ebb in func.layout + .ebbs() + .skip(1) + .filter(|ebb| !expected.contains_key(ebb)) { if let Some(got_inst) = domtree.idom(ebb) { return Err(format!( "mismatching idoms for renumbered {}:\n\ want: unrechable, got: {}", - ebb, - got_inst + ebb, got_inst )); } } diff --git a/lib/filetests/src/test_legalizer.rs b/lib/filetests/src/test_legalizer.rs index cb24ba44a5..d86720612e 100644 --- a/lib/filetests/src/test_legalizer.rs +++ b/lib/filetests/src/test_legalizer.rs @@ -41,13 +41,12 @@ impl SubTest for TestLegalizer { let isa = context.isa.expect("legalizer needs an ISA"); comp_ctx.compute_cfg(); - comp_ctx.legalize(isa).map_err(|e| { - pretty_error(&comp_ctx.func, context.isa, e) - })?; + comp_ctx + .legalize(isa) + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))) - .map_err(|e| e.to_string())?; + write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))).map_err(|e| e.to_string())?; run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_licm.rs b/lib/filetests/src/test_licm.rs index 3add99b1ab..a125da484f 100644 --- a/lib/filetests/src/test_licm.rs +++ b/lib/filetests/src/test_licm.rs @@ -40,14 +40,12 @@ impl SubTest for TestLICM { comp_ctx.flowgraph(); comp_ctx.compute_loop_analysis(); - comp_ctx.licm(context.flags_or_isa()).map_err(|e| { - pretty_error(&comp_ctx.func, context.isa, Into::into(e)) - })?; + comp_ctx + .licm(context.flags_or_isa()) + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?; let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func).map_err( - |e| e.to_string(), - )?; + write!(&mut text, "{}", &comp_ctx.func).map_err(|e| e.to_string())?; run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_postopt.rs b/lib/filetests/src/test_postopt.rs index 14c0060e6d..2813a1562c 100644 --- a/lib/filetests/src/test_postopt.rs +++ b/lib/filetests/src/test_postopt.rs @@ -37,14 +37,12 @@ impl SubTest for TestPostopt { let isa = context.isa.expect("postopt needs an ISA"); comp_ctx.flowgraph(); - comp_ctx.postopt(isa).map_err(|e| { - pretty_error(&comp_ctx.func, context.isa, Into::into(e)) - })?; + comp_ctx + .postopt(isa) + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?; let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func).map_err( - |e| e.to_string(), - )?; + write!(&mut text, "{}", &comp_ctx.func).map_err(|e| e.to_string())?; run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_preopt.rs b/lib/filetests/src/test_preopt.rs index f4c53ce267..7d5609b6a0 100644 --- a/lib/filetests/src/test_preopt.rs +++ b/lib/filetests/src/test_preopt.rs @@ -37,14 +37,12 @@ impl SubTest for TestPreopt { let isa = context.isa.expect("preopt needs an ISA"); comp_ctx.flowgraph(); - comp_ctx.preopt(isa).map_err(|e| { - pretty_error(&comp_ctx.func, context.isa, Into::into(e)) - })?; + comp_ctx + .preopt(isa) + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?; let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func).map_err( - |e| e.to_string(), - )?; + write!(&mut text, "{}", &comp_ctx.func).map_err(|e| e.to_string())?; run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_regalloc.rs b/lib/filetests/src/test_regalloc.rs index 5010c84ad4..3ff5134ca4 100644 --- a/lib/filetests/src/test_regalloc.rs +++ b/lib/filetests/src/test_regalloc.rs @@ -46,17 +46,16 @@ impl SubTest for TestRegalloc { comp_ctx.compute_cfg(); // TODO: Should we have an option to skip legalization? - comp_ctx.legalize(isa).map_err(|e| { - pretty_error(&comp_ctx.func, context.isa, e) - })?; + comp_ctx + .legalize(isa) + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; comp_ctx.compute_domtree(); - comp_ctx.regalloc(isa).map_err(|e| { - pretty_error(&comp_ctx.func, context.isa, e) - })?; + comp_ctx + .regalloc(isa) + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))) - .map_err(|e| e.to_string())?; + write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))).map_err(|e| e.to_string())?; run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_simple_gvn.rs b/lib/filetests/src/test_simple_gvn.rs index 467559eecb..54ccab6d5b 100644 --- a/lib/filetests/src/test_simple_gvn.rs +++ b/lib/filetests/src/test_simple_gvn.rs @@ -39,14 +39,12 @@ impl SubTest for TestSimpleGVN { comp_ctx.func = func.into_owned(); comp_ctx.flowgraph(); - comp_ctx.simple_gvn(context.flags_or_isa()).map_err(|e| { - pretty_error(&comp_ctx.func, context.isa, Into::into(e)) - })?; + comp_ctx + .simple_gvn(context.flags_or_isa()) + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?; let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func).map_err( - |e| e.to_string(), - )?; + write!(&mut text, "{}", &comp_ctx.func).map_err(|e| e.to_string())?; run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs index 514e988936..7b8c606ced 100644 --- a/lib/filetests/src/test_verifier.rs +++ b/lib/filetests/src/test_verifier.rs @@ -54,29 +54,24 @@ impl SubTest for TestVerifier { } match verify_function(func, context.flags_or_isa()) { - Ok(_) => { - match expected { - None => Ok(()), - Some((_, msg)) => Err(format!("passed, expected error: {}", msg)), - } - } - Err(got) => { - match expected { - None => Err(format!("verifier pass, got {}", got)), - Some((want_loc, want_msg)) if got.message.contains(want_msg) => { - if want_loc == got.location { - Ok(()) - } else { - Err(format!( - "correct error reported on {}, but wanted {}", - got.location, - want_loc - )) - } + Ok(_) => match expected { + None => Ok(()), + Some((_, msg)) => Err(format!("passed, expected error: {}", msg)), + }, + Err(got) => match expected { + None => Err(format!("verifier pass, got {}", got)), + Some((want_loc, want_msg)) if got.message.contains(want_msg) => { + if want_loc == got.location { + Ok(()) + } else { + Err(format!( + "correct error reported on {}, but wanted {}", + got.location, want_loc + )) } - Some(_) => Err(format!("mismatching error: {}", got)), } - } + Some(_) => Err(format!("mismatching error: {}", got)), + }, } } } diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 20aac932ce..2688a1055a 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -141,7 +141,7 @@ where // instruction being inserted to add related info to the DFG and the SSA building system, // and perform debug sanity checks. fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'short mut DataFlowGraph) { -// We only insert the Ebb in the layout when an instruction is added to it + // We only insert the Ebb in the layout when an instruction is added to it self.builder.ensure_inserted_ebb(); let inst = self.builder.func.dfg.make_inst(data.clone()); @@ -154,17 +154,17 @@ where if data.opcode().is_branch() { match data.branch_destination() { Some(dest_ebb) => { -// If the user has supplied jump arguments we must adapt the arguments of -// the destination ebb + // If the user has supplied jump arguments we must adapt the arguments of + // the destination ebb self.builder.declare_successor(dest_ebb, inst); } None => { -// branch_destination() doesn't detect jump_tables -// If jump table we declare all entries successor + // branch_destination() doesn't detect jump_tables + // If jump table we declare all entries successor if let InstructionData::BranchTable { table, .. } = data { -// Unlike all other jumps/branches, jump tables are -// capable of having the same successor appear -// multiple times, so we must deduplicate. + // Unlike all other jumps/branches, jump tables are + // capable of having the same successor appear + // multiple times, so we must deduplicate. let mut unique = EntitySet::::new(); for dest_ebb in self.builder .func @@ -273,8 +273,8 @@ where pub fn switch_to_block(&mut self, ebb: Ebb) { // First we check that the previous block has been filled. debug_assert!( - self.position.is_default() || self.is_unreachable() || self.is_pristine() || - self.is_filled(), + self.position.is_default() || self.is_unreachable() || self.is_pristine() + || self.is_filled(), "you have to fill your block before switching" ); // We cannot switch to a filled block @@ -324,12 +324,9 @@ where var ) }); - self.func_ctx.ssa.use_var( - self.func, - var, - ty, - self.position.basic_block.unwrap(), - ) + self.func_ctx + .ssa + .use_var(self.func, var, ty, self.position.basic_block.unwrap()) }; self.handle_ssa_side_effects(side_effects); val @@ -339,23 +336,19 @@ where /// the same as the type registered for the variable. pub fn def_var(&mut self, var: Variable, val: Value) { debug_assert_eq!( - *self.func_ctx.types.get(var).unwrap_or_else(|| { - panic!( - "variable {:?} is used but its type has not been declared", - var - ) - }), + *self.func_ctx.types.get(var).unwrap_or_else(|| panic!( + "variable {:?} is used but its type has not been declared", + var + )), self.func.dfg.value_type(val), "declared type of variable {:?} doesn't match type of value {}", var, val ); - self.func_ctx.ssa.def_var( - var, - val, - self.position.basic_block.unwrap(), - ); + self.func_ctx + .ssa + .def_var(var, val, self.position.basic_block.unwrap()); } /// Creates a jump table in the function, to be used by `br_table` instructions. @@ -460,15 +453,17 @@ where pub fn finalize(&mut self) { // Check that all the `Ebb`s are filled and sealed. debug_assert!( - self.func_ctx.ebbs.iter().all(|(ebb, ebb_data)| { - ebb_data.pristine || self.func_ctx.ssa.is_sealed(ebb) - }), + self.func_ctx + .ebbs + .iter() + .all(|(ebb, ebb_data)| ebb_data.pristine || self.func_ctx.ssa.is_sealed(ebb)), "all blocks should be sealed before dropping a FunctionBuilder" ); debug_assert!( - self.func_ctx.ebbs.values().all(|ebb_data| { - ebb_data.pristine || ebb_data.filled - }), + self.func_ctx + .ebbs + .values() + .all(|ebb_data| ebb_data.pristine || ebb_data.filled), "all blocks should be filled before dropping a FunctionBuilder" ); @@ -527,16 +522,14 @@ where /// **Note:** You are responsible for maintaining the coherence with the arguments of /// other jump instructions. pub fn change_jump_destination(&mut self, inst: Inst, new_dest: Ebb) { - let old_dest = self.func.dfg[inst].branch_destination_mut().expect( - "you want to change the jump destination of a non-jump instruction", - ); + let old_dest = self.func.dfg[inst] + .branch_destination_mut() + .expect("you want to change the jump destination of a non-jump instruction"); let pred = self.func_ctx.ssa.remove_ebb_predecessor(*old_dest, inst); *old_dest = new_dest; - self.func_ctx.ssa.declare_ebb_predecessor( - new_dest, - pred, - inst, - ); + self.func_ctx + .ssa + .declare_ebb_predecessor(new_dest, pred, inst); } /// Returns `true` if and only if the current `Ebb` is sealed and has no predecessors declared. @@ -547,8 +540,8 @@ where None => false, Some(entry) => self.position.ebb.unwrap() == entry, }; - !is_entry && self.func_ctx.ssa.is_sealed(self.position.ebb.unwrap()) && - self.func_ctx + !is_entry && self.func_ctx.ssa.is_sealed(self.position.ebb.unwrap()) + && self.func_ctx .ssa .predecessors(self.position.ebb.unwrap()) .is_empty() @@ -582,9 +575,11 @@ where Variable: EntityRef + Debug, { fn move_to_next_basic_block(&mut self) { - self.position.basic_block = PackedOption::from(self.func_ctx.ssa.declare_ebb_body_block( - self.position.basic_block.unwrap(), - )); + self.position.basic_block = PackedOption::from( + self.func_ctx + .ssa + .declare_ebb_body_block(self.position.basic_block.unwrap()), + ); } fn fill_current_block(&mut self) { diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index b9880c9696..ad0760dc25 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -131,16 +131,9 @@ #![warn(unused_import_braces)] #![cfg_attr(feature = "std", warn(unstable_features))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default))] -#![cfg_attr(feature="cargo-clippy", warn( - float_arithmetic, - mut_mut, - nonminimal_bool, - option_map_unwrap_or, - option_map_unwrap_or_else, - print_stdout, - unicode_not_nfc, - use_self, - ))] +#![cfg_attr(feature = "cargo-clippy", + warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, + option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] #![cfg_attr(not(feature = "std"), no_std)] #![cfg_attr(not(feature = "std"), feature(alloc))] diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index d13b9f3d75..578264ef3a 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -102,11 +102,10 @@ impl BlockData { BlockData::EbbHeader(ref mut data) => { // This a linear complexity operation but the number of predecessors is low // in all non-pathological cases - let pred: usize = - data.predecessors - .iter() - .position(|pair| pair.1 == inst) - .expect("the predecessor you are trying to remove is not declared"); + let pred: usize = data.predecessors + .iter() + .position(|pair| pair.1 == inst) + .expect("the predecessor you are trying to remove is not declared"); data.predecessors.swap_remove(pred).0 } } @@ -173,9 +172,9 @@ where /// Tests whether an `SSABuilder` is in a cleared state. pub fn is_empty(&self) -> bool { - self.variables.is_empty() && self.blocks.is_empty() && self.ebb_headers.is_empty() && - self.calls.is_empty() && - self.results.is_empty() && self.side_effects.is_empty() + self.variables.is_empty() && self.blocks.is_empty() && self.ebb_headers.is_empty() + && self.calls.is_empty() && self.results.is_empty() + && self.side_effects.is_empty() } } @@ -493,15 +492,16 @@ where /// Initiate use lookups in all predecessors of `dest_ebb`, and arrange for a call /// to `finish_predecessors_lookup` once they complete. fn begin_predecessors_lookup(&mut self, temp_arg_val: Value, dest_ebb: Ebb) { - self.calls.push(Call::FinishPredecessorsLookup( - temp_arg_val, - dest_ebb, - )); + self.calls + .push(Call::FinishPredecessorsLookup(temp_arg_val, dest_ebb)); // Iterate over the predecessors. let mut calls = mem::replace(&mut self.calls, Vec::new()); - calls.extend(self.predecessors(dest_ebb).iter().rev().map(|&(pred, _)| { - Call::UseVar(pred) - })); + calls.extend( + self.predecessors(dest_ebb) + .iter() + .rev() + .map(|&(pred, _)| Call::UseVar(pred)), + ); self.calls = calls; } @@ -584,16 +584,15 @@ where .get(*pred_block) .unwrap() .unwrap(); - if let Some((middle_ebb, middle_block, middle_jump_inst)) = - self.append_jump_argument( - func, - *last_inst, - *pred_block, - dest_ebb, - pred_val, - temp_arg_var, - ) - { + let jump_arg = self.append_jump_argument( + func, + *last_inst, + *pred_block, + dest_ebb, + pred_val, + temp_arg_var, + ); + if let Some((middle_ebb, middle_block, middle_jump_inst)) = jump_arg { *pred_block = middle_block; *last_inst = middle_jump_inst; self.side_effects.split_ebbs_created.push(middle_ebb); diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs index aa6439690d..82b7e3295a 100644 --- a/lib/module/src/backend.rs +++ b/lib/module/src/backend.rs @@ -2,8 +2,8 @@ use DataContext; use Linkage; -use ModuleNamespace; use ModuleError; +use ModuleNamespace; use cretonne_codegen::Context; use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::{binemit, ir}; diff --git a/lib/module/src/data_context.rs b/lib/module/src/data_context.rs index 3f9c9f8061..ef327fce88 100644 --- a/lib/module/src/data_context.rs +++ b/lib/module/src/data_context.rs @@ -1,7 +1,7 @@ //! Defines `DataContext`. +use cretonne_codegen::binemit::{Addend, CodeOffset}; use cretonne_codegen::entity::PrimaryMap; -use cretonne_codegen::binemit::{CodeOffset, Addend}; use cretonne_codegen::ir; /// This specifies how data is to be initialized. @@ -145,8 +145,8 @@ impl DataContext { #[cfg(test)] mod tests { - use {DataContext, Writability, Init}; use cretonne_codegen::ir; + use {DataContext, Init, Writability}; #[test] fn basic_data_context() { @@ -202,7 +202,9 @@ mod tests { assert_eq!(description.writable, Writability::Readonly); assert_eq!( description.init, - Init::Bytes { contents: contents_clone.into_boxed_slice() } + Init::Bytes { + contents: contents_clone.into_boxed_slice() + } ); assert_eq!(description.function_decls.len(), 0); assert_eq!(description.data_decls.len(), 0); diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs index 5ada469555..d7f9118288 100644 --- a/lib/module/src/lib.rs +++ b/lib/module/src/lib.rs @@ -3,18 +3,10 @@ #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] #![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] #![cfg_attr(feature = "cargo-clippy", - allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature="cargo-clippy", warn( - float_arithmetic, - mut_mut, - nonminimal_bool, - option_map_unwrap_or, - option_map_unwrap_or_else, - print_stdout, - unicode_not_nfc, - use_self, - ))] + warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, + option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] #[macro_use] extern crate cretonne_codegen; @@ -28,5 +20,5 @@ mod data_context; mod module; pub use backend::Backend; -pub use data_context::{DataContext, Writability, DataDescription, Init}; -pub use module::{DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleNamespace, ModuleError}; +pub use data_context::{DataContext, DataDescription, Init, Writability}; +pub use module::{DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleError, ModuleNamespace}; diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index 409adf97c9..d9a7c7fda9 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -42,7 +42,6 @@ impl From for ir::ExternalName { } } - /// Linkage refers to where an entity is defined and who can see it. #[derive(Copy, Clone, PartialEq, Eq)] pub enum Linkage { @@ -60,19 +59,15 @@ impl Linkage { fn merge(a: Self, b: Self) -> Self { match a { Linkage::Export => Linkage::Export, - Linkage::Preemptible => { - match b { - Linkage::Export => Linkage::Export, - _ => Linkage::Preemptible, - } - } - Linkage::Local => { - match b { - Linkage::Export => Linkage::Export, - Linkage::Preemptible => Linkage::Preemptible, - _ => Linkage::Local, - } - } + Linkage::Preemptible => match b { + Linkage::Export => Linkage::Export, + _ => Linkage::Preemptible, + }, + Linkage::Local => match b { + Linkage::Export => Linkage::Export, + Linkage::Preemptible => Linkage::Preemptible, + _ => Linkage::Local, + }, Linkage::Import => b, } } @@ -94,7 +89,6 @@ impl Linkage { } } - /// A declared name may refer to either a function or data declaration #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)] pub enum FuncOrDataId { @@ -360,19 +354,17 @@ where // TODO: Can we avoid allocating names so often? use std::collections::hash_map::Entry::*; match self.names.entry(name.to_owned()) { - Occupied(entry) => { - match *entry.get() { - FuncOrDataId::Func(id) => { - let existing = &mut self.contents.functions[id]; - existing.merge(linkage); - self.backend.declare_function(name, existing.decl.linkage); - Ok(id) - } - FuncOrDataId::Data(..) => Err( - ModuleError::IncompatibleDeclaration(name.to_owned()), - ), + Occupied(entry) => match *entry.get() { + FuncOrDataId::Func(id) => { + let existing = &mut self.contents.functions[id]; + existing.merge(linkage); + self.backend.declare_function(name, existing.decl.linkage); + Ok(id) } - } + FuncOrDataId::Data(..) => { + Err(ModuleError::IncompatibleDeclaration(name.to_owned())) + } + }, Vacant(entry) => { let id = self.contents.functions.push(ModuleFunction { decl: FunctionDeclaration { @@ -400,24 +392,19 @@ where // TODO: Can we avoid allocating names so often? use std::collections::hash_map::Entry::*; match self.names.entry(name.to_owned()) { - Occupied(entry) => { - match *entry.get() { - FuncOrDataId::Data(id) => { - let existing = &mut self.contents.data_objects[id]; - existing.merge(linkage, writable); - self.backend.declare_data( - name, - existing.decl.linkage, - existing.decl.writable, - ); - Ok(id) - } - - FuncOrDataId::Func(..) => Err( - ModuleError::IncompatibleDeclaration(name.to_owned()), - ), + Occupied(entry) => match *entry.get() { + FuncOrDataId::Data(id) => { + let existing = &mut self.contents.data_objects[id]; + existing.merge(linkage, writable); + self.backend + .declare_data(name, existing.decl.linkage, existing.decl.writable); + Ok(id) } - } + + FuncOrDataId::Func(..) => { + Err(ModuleError::IncompatibleDeclaration(name.to_owned())) + } + }, Vacant(entry) => { let id = self.contents.data_objects.push(ModuleData { decl: DataDeclaration { @@ -535,9 +522,9 @@ where "imported data cannot contain references" ); self.backend.write_data_funcaddr( - &mut info.compiled.as_mut().expect( - "`data` must refer to a defined data object", - ), + &mut info.compiled + .as_mut() + .expect("`data` must refer to a defined data object"), offset, what, ); @@ -558,9 +545,9 @@ where "imported data cannot contain references" ); self.backend.write_data_dataaddr( - &mut info.compiled.as_mut().expect( - "`data` must refer to a defined data object", - ), + &mut info.compiled + .as_mut() + .expect("`data` must refer to a defined data object"), offset, what, addend, @@ -577,10 +564,12 @@ where "imported function cannot be finalized" ); self.backend.finalize_function( - info.compiled.as_ref().expect( - "function must be compiled before it can be finalized", - ), - &ModuleNamespace:: { contents: &self.contents }, + info.compiled + .as_ref() + .expect("function must be compiled before it can be finalized"), + &ModuleNamespace:: { + contents: &self.contents, + }, ) }; self.contents.functions[func].finalized = true; @@ -597,10 +586,12 @@ where "imported data cannot be finalized" ); self.backend.finalize_data( - info.compiled.as_ref().expect( - "data object must be compiled before it can be finalized", - ), - &ModuleNamespace:: { contents: &self.contents }, + info.compiled + .as_ref() + .expect("data object must be compiled before it can be finalized"), + &ModuleNamespace:: { + contents: &self.contents, + }, ) }; self.contents.data_objects[data].finalized = true; @@ -614,20 +605,24 @@ where for info in self.contents.functions.values() { if info.decl.linkage.is_definable() && !info.finalized { self.backend.finalize_function( - info.compiled.as_ref().expect( - "function must be compiled before it can be finalized", - ), - &ModuleNamespace:: { contents: &self.contents }, + info.compiled + .as_ref() + .expect("function must be compiled before it can be finalized"), + &ModuleNamespace:: { + contents: &self.contents, + }, ); } } for info in self.contents.data_objects.values() { if info.decl.linkage.is_definable() && !info.finalized { self.backend.finalize_data( - info.compiled.as_ref().expect( - "data object must be compiled before it can be finalized", - ), - &ModuleNamespace:: { contents: &self.contents }, + info.compiled + .as_ref() + .expect("data object must be compiled before it can be finalized"), + &ModuleNamespace:: { + contents: &self.contents, + }, ); } } diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index 734c33d1f6..54fa134503 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -4,18 +4,10 @@ #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] #![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] #![cfg_attr(feature = "cargo-clippy", - allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature="cargo-clippy", warn( - float_arithmetic, - mut_mut, - nonminimal_bool, - option_map_unwrap_or, - option_map_unwrap_or_else, - print_stdout, - unicode_not_nfc, - use_self, - ))] + warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, + option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] #![cfg_attr(not(feature = "std"), no_std)] extern crate cretonne_codegen; diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs index 85b08df368..852a05b646 100644 --- a/lib/reader/src/isaspec.rs +++ b/lib/reader/src/isaspec.rs @@ -41,21 +41,17 @@ where { for opt in iter.map(TestOption::new) { match opt { - TestOption::Flag(name) => { - match config.enable(name) { - Ok(_) => {} - Err(SetError::BadName) => return err!(loc, "unknown flag '{}'", opt), - Err(_) => return err!(loc, "not a boolean flag: '{}'", opt), - } - } - TestOption::Value(name, value) => { - match config.set(name, value) { - Ok(_) => {} - Err(SetError::BadName) => return err!(loc, "unknown setting '{}'", opt), - Err(SetError::BadType) => return err!(loc, "invalid setting type: '{}'", opt), - Err(SetError::BadValue) => return err!(loc, "invalid setting value: '{}'", opt), - } - } + TestOption::Flag(name) => match config.enable(name) { + Ok(_) => {} + Err(SetError::BadName) => return err!(loc, "unknown flag '{}'", opt), + Err(_) => return err!(loc, "not a boolean flag: '{}'", opt), + }, + TestOption::Value(name, value) => match config.set(name, value) { + Ok(_) => {} + Err(SetError::BadName) => return err!(loc, "unknown setting '{}'", opt), + Err(SetError::BadType) => return err!(loc, "invalid setting type: '{}'", opt), + Err(SetError::BadValue) => return err!(loc, "invalid setting value: '{}'", opt), + }, } } Ok(()) diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 17b99249fb..53a0ef8496 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -15,35 +15,35 @@ use std::u16; #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub enum Token<'a> { Comment(&'a str), - LPar, // '(' - RPar, // ')' - LBrace, // '{' - RBrace, // '}' - LBracket, // '[' - RBracket, // ']' - Minus, // '-' - Plus, // '+' - Comma, // ',' - Dot, // '.' - Colon, // ':' - Equal, // '=' - Arrow, // '->' - Float(&'a str), // Floating point immediate - Integer(&'a str), // Integer immediate - Type(types::Type), // i32, f32, b32x4, ... - Value(Value), // v12, v7 - Ebb(Ebb), // ebb3 - StackSlot(u32), // ss3 - GlobalVar(u32), // gv3 - Heap(u32), // heap2 - JumpTable(u32), // jt2 - FuncRef(u32), // fn2 - SigRef(u32), // sig2 - UserRef(u32), // u345 - Name(&'a str), // %9arbitrary_alphanum, %x3, %0, %function ... + LPar, // '(' + RPar, // ')' + LBrace, // '{' + RBrace, // '}' + LBracket, // '[' + RBracket, // ']' + Minus, // '-' + Plus, // '+' + Comma, // ',' + Dot, // '.' + Colon, // ':' + Equal, // '=' + Arrow, // '->' + Float(&'a str), // Floating point immediate + Integer(&'a str), // Integer immediate + Type(types::Type), // i32, f32, b32x4, ... + Value(Value), // v12, v7 + Ebb(Ebb), // ebb3 + StackSlot(u32), // ss3 + GlobalVar(u32), // gv3 + Heap(u32), // heap2 + JumpTable(u32), // jt2 + FuncRef(u32), // fn2 + SigRef(u32), // sig2 + UserRef(u32), // u345 + Name(&'a str), // %9arbitrary_alphanum, %x3, %0, %function ... HexSequence(&'a str), // #89AF - Identifier(&'a str), // Unrecognized identifier (opcode, enumerator, ...) - SourceLoc(&'a str), // @00c7 + Identifier(&'a str), // Unrecognized identifier (opcode, enumerator, ...) + SourceLoc(&'a str), // @00c7 } /// A `Token` with an associated location. @@ -162,7 +162,9 @@ impl<'a> Lexer<'a> { // Get the location corresponding to `lookahead`. fn loc(&self) -> Location { - Location { line_number: self.line_number } + Location { + line_number: self.line_number, + } } // Starting from `lookahead`, are we looking at `prefix`? @@ -317,9 +319,8 @@ impl<'a> Lexer<'a> { token( split_entity_name(text) .and_then(|(prefix, number)| { - Self::numbered_entity(prefix, number).or_else(|| { - Self::value_type(text, prefix, number) - }) + Self::numbered_entity(prefix, number) + .or_else(|| Self::value_type(text, prefix, number)) }) .unwrap_or_else(|| match text { "iflags" => Token::Type(types::IFLAGS), diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index 3527bbf587..b623035249 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -6,18 +6,10 @@ #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] #![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] #![cfg_attr(feature = "cargo-clippy", - allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature="cargo-clippy", warn( - float_arithmetic, - mut_mut, - nonminimal_bool, - option_map_unwrap_or, - option_map_unwrap_or_else, - print_stdout, - unicode_not_nfc, - use_self, - ))] + warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, + option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] extern crate cretonne_codegen; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 40af898efc..8fadf88566 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -30,9 +30,7 @@ use testfile::{Comment, Details, TestFile}; /// Any test commands or ISA declarations are ignored. pub fn parse_functions(text: &str) -> Result> { let _tt = timing::parse_text(); - parse_test(text).map(|file| { - file.functions.into_iter().map(|(func, _)| func).collect() - }) + parse_test(text).map(|file| file.functions.into_iter().map(|(func, _)| func).collect()) } /// Parse the entire `text` as a test case file. @@ -124,9 +122,8 @@ impl<'a> Context<'a> { // Allocate a new stack slot. fn add_ss(&mut self, ss: StackSlot, data: StackSlotData, loc: &Location) -> Result<()> { while self.function.stack_slots.next_key().index() <= ss.index() { - self.function.create_stack_slot( - StackSlotData::new(StackSlotKind::SpillSlot, 0), - ); + self.function + .create_stack_slot(StackSlotData::new(StackSlotKind::SpillSlot, 0)); } self.function.stack_slots[ss] = data; self.map.def_ss(ss, loc) @@ -169,7 +166,9 @@ impl<'a> Context<'a> { base: HeapBase::ReservedReg, min_size: Imm64::new(0), guard_size: Imm64::new(0), - style: HeapStyle::Static { bound: Imm64::new(0) }, + style: HeapStyle::Static { + bound: Imm64::new(0), + }, }); } self.function.heaps[heap] = data; @@ -188,9 +187,8 @@ impl<'a> Context<'a> { // Allocate a new signature. fn add_sig(&mut self, sig: SigRef, data: Signature, loc: &Location) -> Result<()> { while self.function.dfg.signatures.next_key().index() <= sig.index() { - self.function.import_signature( - Signature::new(CallConv::Fast), - ); + self.function + .import_signature(Signature::new(CallConv::Fast)); } self.function.dfg.signatures[sig] = data; self.map.def_sig(sig, loc) @@ -325,9 +323,9 @@ impl<'a> Parser<'a> { debug_assert!(self.gathering_comments); let entity = entity.into(); self.comments.extend( - self.gathered_comments.drain(..).map(|text| { - Comment { entity, text } - }), + self.gathered_comments + .drain(..) + .map(|text| Comment { entity, text }), ); self.gathering_comments = false; } @@ -501,9 +499,8 @@ impl<'a> Parser<'a> { self.consume(); // Lexer just gives us raw text that looks like an integer. // Parse it as a u8 to check for overflow and other issues. - text.parse().map_err( - |_| self.error("expected u8 decimal immediate"), - ) + text.parse() + .map_err(|_| self.error("expected u8 decimal immediate")) } else { err!(self.loc, err_msg) } @@ -516,9 +513,8 @@ impl<'a> Parser<'a> { self.consume(); // Lexer just gives us raw text that looks like an integer. // Parse it as a i32 to check for overflow and other issues. - text.parse().map_err( - |_| self.error("expected i32 decimal immediate"), - ) + text.parse() + .map_err(|_| self.error("expected i32 decimal immediate")) } else { err!(self.loc, err_msg) } @@ -619,9 +615,8 @@ impl<'a> Parser<'a> { // The only error we anticipate from this parse is overflow, the lexer should // already have ensured that the string doesn't contain invalid characters, and // isn't empty or negative. - u16::from_str_radix(bits_str, 16).map_err(|_| { - self.error("the hex sequence given overflows the u16 type") - }) + u16::from_str_radix(bits_str, 16) + .map_err(|_| self.error("the hex sequence given overflows the u16 type")) } else { err!(self.loc, err_msg) } @@ -632,16 +627,11 @@ impl<'a> Parser<'a> { if let Some(Token::Name(name)) = self.token() { self.consume(); match isa { - Some(isa) => { - isa.register_info().parse_regunit(name).ok_or_else(|| { - self.error("invalid register name") - }) - } - None => { - name.parse().map_err( - |_| self.error("invalid register number"), - ) - } + Some(isa) => isa.register_info() + .parse_regunit(name) + .ok_or_else(|| self.error("invalid register name")), + None => name.parse() + .map_err(|_| self.error("invalid register number")), } } else { match isa { @@ -725,9 +715,7 @@ impl<'a> Parser<'a> { isaspec::parse_options(words, &mut isa_builder, &self.loc)?; // Construct a trait object with the aggregate settings. - isas.push(isa_builder.finish( - settings::Flags::new(flag_builder.clone()), - )); + isas.push(isa_builder.finish(settings::Flags::new(flag_builder.clone()))); } _ => break, } @@ -791,10 +779,7 @@ impl<'a> Parser<'a> { let mut ctx = Context::new(Function::with_name_signature(name, sig), unique_isa); // function ::= "function" name signature * "{" preamble function-body "}" - self.match_token( - Token::LBrace, - "expected '{' before function body", - )?; + self.match_token(Token::LBrace, "expected '{' before function body")?; self.token(); self.claim_gathered_comments(AnyEntity::Function); @@ -804,10 +789,7 @@ impl<'a> Parser<'a> { // function ::= "function" name signature "{" preamble * function-body "}" self.parse_function_body(&mut ctx)?; // function ::= "function" name signature "{" preamble function-body * "}" - self.match_token( - Token::RBrace, - "expected '}' after function body", - )?; + self.match_token(Token::RBrace, "expected '}' after function body")?; // Collect any comments following the end of the function, then stop gathering comments. self.start_gathering_comments(); @@ -833,9 +815,8 @@ impl<'a> Parser<'a> { match self.token() { Some(Token::Name(s)) => { self.consume(); - s.parse().map_err( - |_| self.error("invalid test case or libcall name"), - ) + s.parse() + .map_err(|_| self.error("invalid test case or libcall name")) } Some(Token::UserRef(namespace)) => { self.consume(); @@ -844,9 +825,9 @@ impl<'a> Parser<'a> { self.consume(); match self.token() { Some(Token::Integer(index_str)) => { - let index: u32 = u32::from_str_radix(index_str, 10).map_err(|_| { - self.error("the integer given overflows the u32 type") - })?; + let index: u32 = u32::from_str_radix(index_str, 10).map_err( + |_| self.error("the integer given overflows the u32 type"), + )?; self.consume(); Ok(ExternalName::user(namespace, index)) } @@ -868,18 +849,12 @@ impl<'a> Parser<'a> { // Calling convention defaults to `fast`, but can be changed. let mut sig = Signature::new(CallConv::Fast); - self.match_token( - Token::LPar, - "expected function signature: ( args... )", - )?; + self.match_token(Token::LPar, "expected function signature: ( args... )")?; // signature ::= "(" * [abi-param-list] ")" ["->" retlist] [callconv] if self.token() != Some(Token::RPar) { sig.params = self.parse_abi_param_list(unique_isa)?; } - self.match_token( - Token::RPar, - "expected ')' after function arguments", - )?; + self.match_token(Token::RPar, "expected ')' after function arguments")?; if self.optional(Token::Arrow) { sig.returns = self.parse_abi_param_list(unique_isa)?; } @@ -1001,41 +976,33 @@ impl<'a> Parser<'a> { Some(Token::StackSlot(..)) => { self.start_gathering_comments(); let loc = self.loc; - self.parse_stack_slot_decl().and_then(|(ss, dat)| { - ctx.add_ss(ss, dat, &loc) - }) + self.parse_stack_slot_decl() + .and_then(|(ss, dat)| ctx.add_ss(ss, dat, &loc)) } Some(Token::GlobalVar(..)) => { self.start_gathering_comments(); - self.parse_global_var_decl().and_then(|(gv, dat)| { - ctx.add_gv(gv, dat, &self.loc) - }) + self.parse_global_var_decl() + .and_then(|(gv, dat)| ctx.add_gv(gv, dat, &self.loc)) } Some(Token::Heap(..)) => { self.start_gathering_comments(); - self.parse_heap_decl().and_then(|(heap, dat)| { - ctx.add_heap(heap, dat, &self.loc) - }) + self.parse_heap_decl() + .and_then(|(heap, dat)| ctx.add_heap(heap, dat, &self.loc)) } Some(Token::SigRef(..)) => { self.start_gathering_comments(); - self.parse_signature_decl(ctx.unique_isa).and_then( - |(sig, dat)| { - ctx.add_sig(sig, dat, &self.loc) - }, - ) + self.parse_signature_decl(ctx.unique_isa) + .and_then(|(sig, dat)| ctx.add_sig(sig, dat, &self.loc)) } Some(Token::FuncRef(..)) => { self.start_gathering_comments(); - self.parse_function_decl(ctx).and_then(|(fn_, dat)| { - ctx.add_fn(fn_, dat, &self.loc) - }) + self.parse_function_decl(ctx) + .and_then(|(fn_, dat)| ctx.add_fn(fn_, dat, &self.loc)) } Some(Token::JumpTable(..)) => { self.start_gathering_comments(); - self.parse_jump_table_decl().and_then(|(jt, dat)| { - ctx.add_jt(jt, dat, &self.loc) - }) + self.parse_jump_table_decl() + .and_then(|(jt, dat)| ctx.add_jt(jt, dat, &self.loc)) } // More to come.. _ => return Ok(()), @@ -1052,10 +1019,7 @@ impl<'a> Parser<'a> { // | "outgoing_arg" fn parse_stack_slot_decl(&mut self) -> Result<(StackSlot, StackSlotData)> { let ss = self.match_ss("expected stack slot number: ss«n»")?; - self.match_token( - Token::Equal, - "expected '=' in stack slot declaration", - )?; + self.match_token(Token::Equal, "expected '=' in stack slot declaration")?; let kind = self.match_enum("expected stack slot kind")?; // stack-slot-decl ::= StackSlot(ss) "=" stack-slot-kind * Bytes {"," stack-slot-flag} @@ -1095,10 +1059,7 @@ impl<'a> Parser<'a> { fn parse_global_var_decl(&mut self) -> Result<(GlobalVar, GlobalVarData)> { let gv = self.match_gv("expected global variable number: gv«n»")?; - self.match_token( - Token::Equal, - "expected '=' in global variable declaration", - )?; + self.match_token(Token::Equal, "expected '=' in global variable declaration")?; let data = match self.match_any_identifier("expected global variable kind")? { "vmctx" => { @@ -1106,15 +1067,9 @@ impl<'a> Parser<'a> { GlobalVarData::VMContext { offset } } "deref" => { - self.match_token( - Token::LPar, - "expected '(' in 'deref' global variable decl", - )?; + self.match_token(Token::LPar, "expected '(' in 'deref' global variable decl")?; let base = self.match_gv("expected global variable: gv«n»")?; - self.match_token( - Token::RPar, - "expected ')' in 'deref' global variable decl", - )?; + self.match_token(Token::RPar, "expected ')' in 'deref' global variable decl")?; let offset = self.optional_offset32()?; GlobalVarData::Deref { base, offset } } @@ -1146,10 +1101,7 @@ impl<'a> Parser<'a> { // fn parse_heap_decl(&mut self) -> Result<(Heap, HeapData)> { let heap = self.match_heap("expected heap number: heap«n»")?; - self.match_token( - Token::Equal, - "expected '=' in heap declaration", - )?; + self.match_token(Token::Equal, "expected '=' in heap declaration")?; let style_name = self.match_any_identifier("expected 'static' or 'dynamic'")?; @@ -1216,10 +1168,7 @@ impl<'a> Parser<'a> { unique_isa: Option<&TargetIsa>, ) -> Result<(SigRef, Signature)> { let sig = self.match_sig("expected signature number: sig«n»")?; - self.match_token( - Token::Equal, - "expected '=' in signature decl", - )?; + self.match_token(Token::Equal, "expected '=' in signature decl")?; let data = self.parse_signature(unique_isa)?; // Collect any trailing comments. @@ -1241,10 +1190,7 @@ impl<'a> Parser<'a> { // fn parse_function_decl(&mut self, ctx: &mut Context) -> Result<(FuncRef, ExtFuncData)> { let fn_ = self.match_fn("expected function number: fn«n»")?; - self.match_token( - Token::Equal, - "expected '=' in function decl", - )?; + self.match_token(Token::Equal, "expected '=' in function decl")?; let loc = self.loc; @@ -1260,9 +1206,9 @@ impl<'a> Parser<'a> { // function-decl ::= FuncRef(fnref) "=" ["colocated"] name * signature let sig = self.parse_signature(ctx.unique_isa)?; let sigref = ctx.function.import_signature(sig); - ctx.map.def_entity(sigref.into(), &loc).expect( - "duplicate SigRef entities created", - ); + ctx.map + .def_entity(sigref.into(), &loc) + .expect("duplicate SigRef entities created"); ExtFuncData { name, signature: sigref, @@ -1299,10 +1245,7 @@ impl<'a> Parser<'a> { // jump-table-decl ::= * JumpTable(jt) "=" "jump_table" jt-entry {"," jt-entry} fn parse_jump_table_decl(&mut self) -> Result<(JumpTable, JumpTableData)> { let jt = self.match_jt()?; - self.match_token( - Token::Equal, - "expected '=' in jump_table decl", - )?; + self.match_token(Token::Equal, "expected '=' in jump_table decl")?; self.match_identifier("jump_table", "expected 'jump_table'")?; let mut data = JumpTableData::new(); @@ -1393,10 +1336,7 @@ impl<'a> Parser<'a> { if !self.optional(Token::Colon) { // ebb-header ::= Ebb(ebb) [ * ebb-params ] ":" self.parse_ebb_params(ctx, ebb)?; - self.match_token( - Token::Colon, - "expected ':' after EBB parameters", - )?; + self.match_token(Token::Colon, "expected ':' after EBB parameters")?; } // Collect any trailing comments. @@ -1405,13 +1345,12 @@ impl<'a> Parser<'a> { // extended-basic-block ::= ebb-header * { instruction } while match self.token() { - Some(Token::Value(_)) | - Some(Token::Identifier(_)) | - Some(Token::LBracket) | - Some(Token::SourceLoc(_)) => true, + Some(Token::Value(_)) + | Some(Token::Identifier(_)) + | Some(Token::LBracket) + | Some(Token::SourceLoc(_)) => true, _ => false, - } - { + } { let srcloc = self.optional_srcloc()?; let (encoding, result_locations) = self.parse_instruction_encoding(ctx)?; @@ -1434,25 +1373,11 @@ impl<'a> Parser<'a> { } Some(Token::Equal) => { self.consume(); - self.parse_instruction( - &results, - srcloc, - encoding, - result_locations, - ctx, - ebb, - )?; + self.parse_instruction(&results, srcloc, encoding, result_locations, ctx, ebb)?; } _ if !results.is_empty() => return err!(self.loc, "expected -> or ="), _ => { - self.parse_instruction( - &results, - srcloc, - encoding, - result_locations, - ctx, - ebb, - )? + self.parse_instruction(&results, srcloc, encoding, result_locations, ctx, ebb)? } } } @@ -1466,10 +1391,7 @@ impl<'a> Parser<'a> { // ebb-params ::= * "(" ebb-param { "," ebb-param } ")" fn parse_ebb_params(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { // ebb-params ::= * "(" ebb-param { "," ebb-param } ")" - self.match_token( - Token::LPar, - "expected '(' before EBB parameters", - )?; + self.match_token(Token::LPar, "expected '(' before EBB parameters")?; // ebb-params ::= "(" * ebb-param { "," ebb-param } ")" self.parse_ebb_param(ctx, ebb)?; @@ -1481,10 +1403,7 @@ impl<'a> Parser<'a> { } // ebb-params ::= "(" ebb-param { "," ebb-param } * ")" - self.match_token( - Token::RPar, - "expected ')' after EBB parameters", - )?; + self.match_token(Token::RPar, "expected ')' after EBB parameters")?; Ok(()) } @@ -1499,10 +1418,7 @@ impl<'a> Parser<'a> { let v = self.match_value("EBB argument must be a value")?; let v_location = self.loc; // ebb-param ::= Value(v) * ":" Type(t) arg-loc? - self.match_token( - Token::Colon, - "expected ':' after EBB argument", - )?; + self.match_token(Token::Colon, "expected ':' after EBB argument")?; // ebb-param ::= Value(v) ":" * Type(t) arg-loc? while ctx.function.dfg.num_values() <= v.index() { @@ -1518,10 +1434,7 @@ impl<'a> Parser<'a> { if self.optional(Token::LBracket) { let loc = self.parse_value_location(ctx)?; ctx.function.locations[v] = loc; - self.match_token( - Token::RBracket, - "expected ']' after value location", - )?; + self.match_token(Token::RBracket, "expected ']' after value location")?; } Ok(()) @@ -1573,9 +1486,7 @@ impl<'a> Parser<'a> { if self.optional(Token::LBracket) { // encoding_literal ::= "-" | Identifier HexSequence if !self.optional(Token::Minus) { - let recipe = self.match_any_identifier( - "expected instruction encoding or '-'", - )?; + let recipe = self.match_any_identifier("expected instruction encoding or '-'")?; let bits = self.match_hex16("expected a hex sequence")?; if let Some(recipe_index) = ctx.find_recipe_index(recipe) { @@ -1645,10 +1556,9 @@ impl<'a> Parser<'a> { } let dest = self.match_value("expected value alias")?; - ctx.function.dfg.make_value_alias_for_parser( - dest, - results[0], - ); + ctx.function + .dfg + .make_value_alias_for_parser(dest, results[0]); ctx.map.def_value(results[0], &self.loc)?; ctx.aliases.push(results[0]); Ok(()) @@ -1703,22 +1613,16 @@ impl<'a> Parser<'a> { // We still need to check that the number of result values in the source matches the opcode // or function call signature. We also need to create values with the right type for all // the instruction results. - let ctrl_typevar = self.infer_typevar( - ctx, - opcode, - explicit_ctrl_type, - &inst_data, - )?; + let ctrl_typevar = self.infer_typevar(ctx, opcode, explicit_ctrl_type, &inst_data)?; let inst = ctx.function.dfg.make_inst(inst_data); - let num_results = ctx.function.dfg.make_inst_results_for_parser( - inst, - ctrl_typevar, - results, - ); + let num_results = + ctx.function + .dfg + .make_inst_results_for_parser(inst, ctrl_typevar, results); ctx.function.layout.append_inst(inst, ebb); - ctx.map.def_entity(inst.into(), &opcode_loc).expect( - "duplicate inst references created", - ); + ctx.map + .def_entity(inst.into(), &opcode_loc) + .expect("duplicate inst references created"); if !srcloc.is_default() { ctx.function.srclocs[inst] = srcloc; @@ -1750,9 +1654,11 @@ impl<'a> Parser<'a> { } if let Some(result_locations) = result_locations { - for (&value, loc) in ctx.function.dfg.inst_results(inst).iter().zip( - result_locations, - ) + for (&value, loc) in ctx.function + .dfg + .inst_results(inst) + .iter() + .zip(result_locations) { ctx.function.locations[value] = loc; } @@ -1887,7 +1793,6 @@ impl<'a> Parser<'a> { } Ok(args) - } // Parse an optional value list enclosed in parantheses. @@ -1898,10 +1803,7 @@ impl<'a> Parser<'a> { let args = self.parse_value_list()?; - self.match_token( - Token::RPar, - "expected ')' after arguments", - )?; + self.match_token(Token::RPar, "expected ')' after arguments")?; Ok(args) } @@ -1944,10 +1846,7 @@ impl<'a> Parser<'a> { } InstructionFormat::Binary => { let lhs = self.match_value("expected SSA value first operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_value("expected SSA value second operand")?; InstructionData::Binary { opcode, @@ -1956,13 +1855,8 @@ impl<'a> Parser<'a> { } InstructionFormat::BinaryImm => { let lhs = self.match_value("expected SSA value first operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; - let rhs = self.match_imm64( - "expected immediate integer second operand", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; + let rhs = self.match_imm64("expected immediate integer second operand")?; InstructionData::BinaryImm { opcode, arg: lhs, @@ -1973,15 +1867,9 @@ impl<'a> Parser<'a> { // Names here refer to the `select` instruction. // This format is also use by `fma`. let ctrl_arg = self.match_value("expected SSA value control operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let true_arg = self.match_value("expected SSA value true operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let false_arg = self.match_value("expected SSA value false operand")?; InstructionData::Ternary { opcode, @@ -2008,10 +1896,7 @@ impl<'a> Parser<'a> { } InstructionFormat::Branch => { let ctrl_arg = self.match_value("expected SSA value control operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let ebb_num = self.match_ebb("expected branch destination EBB")?; let args = self.parse_opt_value_list()?; InstructionData::Branch { @@ -2023,10 +1908,7 @@ impl<'a> Parser<'a> { InstructionFormat::BranchInt => { let cond = self.match_enum("expected intcc condition code")?; let arg = self.match_value("expected SSA value first operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let ebb_num = self.match_ebb("expected branch destination EBB")?; let args = self.parse_opt_value_list()?; InstructionData::BranchInt { @@ -2039,10 +1921,7 @@ impl<'a> Parser<'a> { InstructionFormat::BranchFloat => { let cond = self.match_enum("expected floatcc condition code")?; let arg = self.match_value("expected SSA value first operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let ebb_num = self.match_ebb("expected branch destination EBB")?; let args = self.parse_opt_value_list()?; InstructionData::BranchFloat { @@ -2055,15 +1934,9 @@ impl<'a> Parser<'a> { InstructionFormat::BranchIcmp => { let cond = self.match_enum("expected intcc condition code")?; let lhs = self.match_value("expected SSA value first operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_value("expected SSA value second operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let ebb_num = self.match_ebb("expected branch destination EBB")?; let args = self.parse_opt_value_list()?; InstructionData::BranchIcmp { @@ -2075,25 +1948,16 @@ impl<'a> Parser<'a> { } InstructionFormat::BranchTable => { let arg = self.match_value("expected SSA value operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let table = self.match_jt()?; ctx.check_jt(table, &self.loc)?; InstructionData::BranchTable { opcode, arg, table } } InstructionFormat::InsertLane => { let lhs = self.match_value("expected SSA value first operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let lane = self.match_uimm8("expected lane number")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_value("expected SSA value last operand")?; InstructionData::InsertLane { opcode, @@ -2103,20 +1967,14 @@ impl<'a> Parser<'a> { } InstructionFormat::ExtractLane => { let arg = self.match_value("expected SSA value last operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let lane = self.match_uimm8("expected lane number")?; InstructionData::ExtractLane { opcode, lane, arg } } InstructionFormat::IntCompare => { let cond = self.match_enum("expected intcc condition code")?; let lhs = self.match_value("expected SSA value first operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_value("expected SSA value second operand")?; InstructionData::IntCompare { opcode, @@ -2127,10 +1985,7 @@ impl<'a> Parser<'a> { InstructionFormat::IntCompareImm => { let cond = self.match_enum("expected intcc condition code")?; let lhs = self.match_value("expected SSA value first operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_imm64("expected immediate second operand")?; InstructionData::IntCompareImm { opcode, @@ -2147,10 +2002,7 @@ impl<'a> Parser<'a> { InstructionFormat::FloatCompare => { let cond = self.match_enum("expected floatcc condition code")?; let lhs = self.match_value("expected SSA value first operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let rhs = self.match_value("expected SSA value second operand")?; InstructionData::FloatCompare { opcode, @@ -2166,15 +2018,9 @@ impl<'a> Parser<'a> { InstructionFormat::IntSelect => { let cond = self.match_enum("expected intcc condition code")?; let guard = self.match_value("expected SSA value first operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let v_true = self.match_value("expected SSA value second operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let v_false = self.match_value("expected SSA value third operand")?; InstructionData::IntSelect { opcode, @@ -2185,15 +2031,9 @@ impl<'a> Parser<'a> { InstructionFormat::Call => { let func_ref = self.match_fn("expected function reference")?; ctx.check_fn(func_ref, &self.loc)?; - self.match_token( - Token::LPar, - "expected '(' before arguments", - )?; + self.match_token(Token::LPar, "expected '(' before arguments")?; let args = self.parse_value_list()?; - self.match_token( - Token::RPar, - "expected ')' after arguments", - )?; + self.match_token(Token::RPar, "expected ')' after arguments")?; InstructionData::Call { opcode, func_ref, @@ -2203,20 +2043,11 @@ impl<'a> Parser<'a> { InstructionFormat::CallIndirect => { let sig_ref = self.match_sig("expected signature reference")?; ctx.check_sig(sig_ref, &self.loc)?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let callee = self.match_value("expected SSA value callee operand")?; - self.match_token( - Token::LPar, - "expected '(' before arguments", - )?; + self.match_token(Token::LPar, "expected '(' before arguments")?; let args = self.parse_value_list()?; - self.match_token( - Token::RPar, - "expected ')' after arguments", - )?; + self.match_token(Token::RPar, "expected ')' after arguments")?; InstructionData::CallIndirect { opcode, sig_ref, @@ -2240,10 +2071,7 @@ impl<'a> Parser<'a> { } InstructionFormat::StackStore => { let arg = self.match_value("expected SSA value operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let ss = self.match_ss("expected stack slot number: ss«n»")?; ctx.check_ss(ss, &self.loc)?; let offset = self.optional_offset32()?; @@ -2257,15 +2085,9 @@ impl<'a> Parser<'a> { InstructionFormat::HeapAddr => { let heap = self.match_heap("expected heap identifier")?; ctx.check_heap(heap, &self.loc)?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let arg = self.match_value("expected SSA value heap address")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let imm = self.match_uimm32("expected 32-bit integer size")?; InstructionData::HeapAddr { opcode, @@ -2299,10 +2121,7 @@ impl<'a> Parser<'a> { InstructionFormat::Store => { let flags = self.optional_memflags(); let arg = self.match_value("expected SSA value operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let addr = self.match_value("expected SSA value address")?; let offset = self.optional_offset32()?; InstructionData::Store { @@ -2316,10 +2135,7 @@ impl<'a> Parser<'a> { InstructionFormat::StoreComplex => { let flags = self.optional_memflags(); let src = self.match_value("expected SSA value operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let args = self.parse_value_sequence()?; let offset = self.optional_offset32()?; InstructionData::StoreComplex { @@ -2331,15 +2147,9 @@ impl<'a> Parser<'a> { } InstructionFormat::RegMove => { let arg = self.match_value("expected SSA value operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let src = self.match_regunit(ctx.unique_isa)?; - self.match_token( - Token::Arrow, - "expected '->' between register units", - )?; + self.match_token(Token::Arrow, "expected '->' between register units")?; let dst = self.match_regunit(ctx.unique_isa)?; InstructionData::RegMove { opcode, @@ -2350,24 +2160,15 @@ impl<'a> Parser<'a> { } InstructionFormat::CopySpecial => { let src = self.match_regunit(ctx.unique_isa)?; - self.match_token( - Token::Arrow, - "expected '->' between register units", - )?; + self.match_token(Token::Arrow, "expected '->' between register units")?; let dst = self.match_regunit(ctx.unique_isa)?; InstructionData::CopySpecial { opcode, src, dst } } InstructionFormat::RegSpill => { let arg = self.match_value("expected SSA value operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let src = self.match_regunit(ctx.unique_isa)?; - self.match_token( - Token::Arrow, - "expected '->' before destination stack slot", - )?; + self.match_token(Token::Arrow, "expected '->' before destination stack slot")?; let dst = self.match_ss("expected stack slot number: ss«n»")?; ctx.check_ss(dst, &self.loc)?; InstructionData::RegSpill { @@ -2379,10 +2180,7 @@ impl<'a> Parser<'a> { } InstructionFormat::RegFill => { let arg = self.match_value("expected SSA value operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let src = self.match_ss("expected stack slot number: ss«n»")?; ctx.check_ss(src, &self.loc)?; self.match_token( @@ -2403,20 +2201,14 @@ impl<'a> Parser<'a> { } InstructionFormat::CondTrap => { let arg = self.match_value("expected SSA value operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let code = self.match_enum("expected trap code")?; InstructionData::CondTrap { opcode, arg, code } } InstructionFormat::IntCondTrap => { let cond = self.match_enum("expected intcc condition code")?; let arg = self.match_value("expected SSA value operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let code = self.match_enum("expected trap code")?; InstructionData::IntCondTrap { opcode, @@ -2428,10 +2220,7 @@ impl<'a> Parser<'a> { InstructionFormat::FloatCondTrap => { let cond = self.match_enum("expected floatcc condition code")?; let arg = self.match_value("expected SSA value operand")?; - self.match_token( - Token::Comma, - "expected ',' between operands", - )?; + self.match_token(Token::Comma, "expected ',' between operands")?; let code = self.match_enum("expected trap code")?; InstructionData::FloatCondTrap { opcode, @@ -2448,9 +2237,9 @@ impl<'a> Parser<'a> { #[cfg(test)] mod tests { use super::*; + use cretonne_codegen::ir::StackSlotKind; use cretonne_codegen::ir::entities::AnyEntity; use cretonne_codegen::ir::types; - use cretonne_codegen::ir::StackSlotKind; use cretonne_codegen::ir::{ArgumentExtension, ArgumentPurpose}; use cretonne_codegen::settings::CallConv; use error::Error; @@ -2695,7 +2484,8 @@ mod tests { isa riscv function %foo() system_v {}", ).unwrap() - .isa_spec { + .isa_spec + { IsaSpec::None(_) => panic!("Expected some ISA"), IsaSpec::Some(v) => { assert_eq!(v.len(), 1); diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index a4bc1954c1..8bd44fb6d6 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -65,62 +65,62 @@ impl SourceMap { /// Returns the entity reference corresponding to `name`, if it exists. pub fn lookup_str(&self, name: &str) -> Option { split_entity_name(name).and_then(|(ent, num)| match ent { - "v" => { - Value::with_number(num).and_then(|v| if !self.contains_value(v) { + "v" => Value::with_number(num).and_then(|v| { + if !self.contains_value(v) { None } else { Some(v.into()) - }) - } - "ebb" => { - Ebb::with_number(num).and_then(|ebb| if !self.contains_ebb(ebb) { + } + }), + "ebb" => Ebb::with_number(num).and_then(|ebb| { + if !self.contains_ebb(ebb) { None } else { Some(ebb.into()) - }) - } - "ss" => { - StackSlot::with_number(num).and_then(|ss| if !self.contains_ss(ss) { + } + }), + "ss" => StackSlot::with_number(num).and_then(|ss| { + if !self.contains_ss(ss) { None } else { Some(ss.into()) - }) - } - "gv" => { - GlobalVar::with_number(num).and_then(|gv| if !self.contains_gv(gv) { + } + }), + "gv" => GlobalVar::with_number(num).and_then(|gv| { + if !self.contains_gv(gv) { None } else { Some(gv.into()) - }) - } - "heap" => { - Heap::with_number(num).and_then(|heap| if !self.contains_heap(heap) { + } + }), + "heap" => Heap::with_number(num).and_then(|heap| { + if !self.contains_heap(heap) { None } else { Some(heap.into()) - }) - } - "sig" => { - SigRef::with_number(num).and_then(|sig| if !self.contains_sig(sig) { + } + }), + "sig" => SigRef::with_number(num).and_then(|sig| { + if !self.contains_sig(sig) { None } else { Some(sig.into()) - }) - } - "fn" => { - FuncRef::with_number(num).and_then(|fn_| if !self.contains_fn(fn_) { + } + }), + "fn" => FuncRef::with_number(num).and_then(|fn_| { + if !self.contains_fn(fn_) { None } else { Some(fn_.into()) - }) - } - "jt" => { - JumpTable::with_number(num).and_then(|jt| if !self.contains_jt(jt) { + } + }), + "jt" => JumpTable::with_number(num).and_then(|jt| { + if !self.contains_jt(jt) { None } else { Some(jt.into()) - }) - } + } + }), _ => None, }) } @@ -134,7 +134,9 @@ impl SourceMap { impl SourceMap { /// Create a new empty `SourceMap`. pub fn new() -> Self { - Self { locations: HashMap::new() } + Self { + locations: HashMap::new(), + } } /// Define the value `entity`. diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs index 1fcfd8f93e..873578e708 100644 --- a/lib/simplejit/src/backend.rs +++ b/lib/simplejit/src/backend.rs @@ -1,17 +1,17 @@ //! Defines `SimpleJITBackend`. -use cretonne_codegen::binemit::{Addend, CodeOffset, Reloc, RelocSink, NullTrapSink}; +use cretonne_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSink}; use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::{self, ir, settings}; -use cretonne_module::{Backend, DataContext, Linkage, ModuleNamespace, Writability, - DataDescription, Init, ModuleError}; +use cretonne_module::{Backend, DataContext, DataDescription, Init, Linkage, ModuleError, + ModuleNamespace, Writability}; use cretonne_native; +use libc; +use memory::Memory; use std::ffi::CString; use std::ptr; -use libc; #[cfg(windows)] use winapi; -use memory::Memory; /// A builder for `SimpleJITBackend`. pub struct SimpleJITBuilder { @@ -119,9 +119,9 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { code_size: u32, ) -> Result { let size = code_size as usize; - let ptr = self.code_memory.allocate(size).expect( - "TODO: handle OOM etc.", - ); + let ptr = self.code_memory + .allocate(size) + .expect("TODO: handle OOM etc."); let mut reloc_sink = SimpleJITRelocSink::new(); // Ignore traps for now. For now, frontends should just avoid generating code // that traps. @@ -152,16 +152,12 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { let size = init.size(); let storage = match writable { - Writability::Readonly => { - self.writable_memory.allocate(size).expect( - "TODO: handle OOM etc.", - ) - } - Writability::Writable => { - self.writable_memory.allocate(size).expect( - "TODO: handle OOM etc.", - ) - } + Writability::Readonly => self.writable_memory + .allocate(size) + .expect("TODO: handle OOM etc."), + Writability::Writable => self.writable_memory + .allocate(size) + .expect("TODO: handle OOM etc."), }; match *init { @@ -262,20 +258,25 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { Reloc::Abs4 => { // TODO: Handle overflow. #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] - unsafe { write_unaligned(at as *mut u32, what as u32) }; + unsafe { + write_unaligned(at as *mut u32, what as u32) + }; } Reloc::Abs8 => { #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] - unsafe { write_unaligned(at as *mut u64, what as u64) }; + unsafe { + write_unaligned(at as *mut u64, what as u64) + }; } Reloc::X86PCRel4 => { // TODO: Handle overflow. let pcrel = ((what as isize) - (at as isize)) as i32; #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] - unsafe { write_unaligned(at as *mut i32, pcrel) }; + unsafe { + write_unaligned(at as *mut i32, pcrel) + }; } - Reloc::X86GOTPCRel4 | - Reloc::X86PLTRel4 => panic!("unexpected PIC relocation"), + Reloc::X86GOTPCRel4 | Reloc::X86PLTRel4 => panic!("unexpected PIC relocation"), _ => unimplemented!(), } } @@ -322,15 +323,19 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { Reloc::Abs4 => { // TODO: Handle overflow. #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] - unsafe { write_unaligned(at as *mut u32, what as u32) }; + unsafe { + write_unaligned(at as *mut u32, what as u32) + }; } Reloc::Abs8 => { #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] - unsafe { write_unaligned(at as *mut u64, what as u64) }; + unsafe { + write_unaligned(at as *mut u64, what as u64) + }; + } + Reloc::X86PCRel4 | Reloc::X86GOTPCRel4 | Reloc::X86PLTRel4 => { + panic!("unexpected text relocation in data") } - Reloc::X86PCRel4 | - Reloc::X86GOTPCRel4 | - Reloc::X86PLTRel4 => panic!("unexpected text relocation in data"), _ => unimplemented!(), } } diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs index 92c088cd71..90ee03d9d5 100644 --- a/lib/simplejit/src/lib.rs +++ b/lib/simplejit/src/lib.rs @@ -3,25 +3,17 @@ #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] #![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] #![cfg_attr(feature = "cargo-clippy", - allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature="cargo-clippy", warn( - float_arithmetic, - mut_mut, - nonminimal_bool, - option_map_unwrap_or, - option_map_unwrap_or_else, - print_stdout, - unicode_not_nfc, - use_self, - ))] + warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, + option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] extern crate cretonne_codegen; extern crate cretonne_module; extern crate cretonne_native; extern crate errno; -extern crate region; extern crate libc; +extern crate region; #[cfg(target_os = "windows")] extern crate winapi; @@ -29,4 +21,4 @@ extern crate winapi; mod backend; mod memory; -pub use backend::{SimpleJITBuilder, SimpleJITBackend}; +pub use backend::{SimpleJITBackend, SimpleJITBuilder}; diff --git a/lib/simplejit/src/memory.rs b/lib/simplejit/src/memory.rs index 3ec2995372..b856c19009 100644 --- a/lib/simplejit/src/memory.rs +++ b/lib/simplejit/src/memory.rs @@ -1,8 +1,8 @@ -use std::mem; -use std::ptr; use errno; use libc; use region; +use std::mem; +use std::ptr; /// Round `size` up to the nearest multiple of `page_size`. fn round_up_to_page_size(size: usize, page_size: usize) -> usize { @@ -91,10 +91,8 @@ impl Memory { } fn finish_current(&mut self) { - self.allocations.push(mem::replace( - &mut self.current, - PtrLen::new(), - )); + self.allocations + .push(mem::replace(&mut self.current, PtrLen::new())); self.position = 0; } @@ -136,9 +134,8 @@ impl Memory { for &PtrLen { ptr, len } in &self.allocations[self.executable..] { if len != 0 { unsafe { - region::protect(ptr, len, region::Protection::Read).expect( - "unable to make memory readonly", - ); + region::protect(ptr, len, region::Protection::Read) + .expect("unable to make memory readonly"); } } } diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs index cc1a3e1697..ebfb0c731e 100644 --- a/lib/umbrella/src/lib.rs +++ b/lib/umbrella/src/lib.rs @@ -3,18 +3,10 @@ #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] #![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] #![cfg_attr(feature = "cargo-clippy", - allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature="cargo-clippy", warn( - float_arithmetic, - mut_mut, - nonminimal_bool, - option_map_unwrap_or, - option_map_unwrap_or_else, - print_stdout, - unicode_not_nfc, - use_self, - ))] + warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, + option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] /// Provide these crates, renamed to reduce stutter. pub extern crate cretonne_codegen as codegen; @@ -25,14 +17,13 @@ pub extern crate cretonne_frontend as frontend; pub mod prelude { pub use codegen; pub use codegen::entity::EntityRef; - pub use codegen::ir::{AbiParam, InstBuilder, Value, Ebb, Signature, Type, JumpTableData, - MemFlags, ExtFuncData, GlobalVarData, StackSlotData, StackSlotKind, - TrapCode}; - pub use codegen::ir::types; - pub use codegen::ir::condcodes::{IntCC, FloatCC}; + pub use codegen::ir::condcodes::{FloatCC, IntCC}; pub use codegen::ir::immediates::{Ieee32, Ieee64, Imm64}; - pub use codegen::settings::{self, Configurable, CallConv}; + pub use codegen::ir::types; + pub use codegen::ir::{AbiParam, Ebb, ExtFuncData, GlobalVarData, InstBuilder, JumpTableData, + MemFlags, Signature, StackSlotData, StackSlotKind, TrapCode, Type, Value}; pub use codegen::isa; + pub use codegen::settings::{self, CallConv, Configurable}; - pub use frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; + pub use frontend::{FunctionBuilder, FunctionBuilderContext, Variable}; } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index dbe23c4d15..60eaf8f853 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -194,10 +194,9 @@ pub fn translate_operator( let frame = state.control_stack.pop().unwrap(); if !builder.is_unreachable() || !builder.is_pristine() { let return_count = frame.num_return_values(); - builder.ins().jump( - frame.following_code(), - state.peekn(return_count), - ); + builder + .ins() + .jump(frame.following_code(), state.peekn(return_count)); } builder.switch_to_block(frame.following_code()); builder.seal_block(frame.following_code()); @@ -206,9 +205,9 @@ pub fn translate_operator( builder.seal_block(header) } state.stack.truncate(frame.original_stack_size()); - state.stack.extend_from_slice( - builder.ebb_params(frame.following_code()), - ); + state + .stack + .extend_from_slice(builder.ebb_params(frame.following_code())); } /**************************** Branch instructions ********************************* * The branch instructions all have as arguments a target nesting level, which @@ -244,10 +243,9 @@ pub fn translate_operator( }; (return_count, frame.br_destination()) }; - builder.ins().jump( - br_destination, - state.peekn(return_count), - ); + builder + .ins() + .jump(br_destination, state.peekn(return_count)); state.popn(return_count); state.reachable = false; } @@ -392,67 +390,86 @@ pub fn translate_operator( let heap_index = reserved as MemoryIndex; let heap = state.get_heap(builder.func, reserved, environ); let val = state.pop1(); - state.push1(environ.translate_grow_memory( - builder.cursor(), - heap_index, - heap, - val, - )?) + state.push1(environ.translate_grow_memory(builder.cursor(), heap_index, heap, val)?) } Operator::CurrentMemory { reserved } => { let heap_index = reserved as MemoryIndex; let heap = state.get_heap(builder.func, reserved, environ); - state.push1(environ.translate_current_memory( - builder.cursor(), - heap_index, - heap, - )?); + state.push1(environ.translate_current_memory(builder.cursor(), heap_index, heap)?); } /******************************* Load instructions *********************************** * Wasm specifies an integer alignment flag but we drop it in Cretonne. * The memory base address is provided by the environment. * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not ************************************************************************************/ - Operator::I32Load8U { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I32Load8U { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Uload8, I32, builder, state, environ); } - Operator::I32Load16U { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I32Load16U { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Uload16, I32, builder, state, environ); } - Operator::I32Load8S { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I32Load8S { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Sload8, I32, builder, state, environ); } - Operator::I32Load16S { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I32Load16S { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Sload16, I32, builder, state, environ); } - Operator::I64Load8U { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load8U { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Uload8, I64, builder, state, environ); } - Operator::I64Load16U { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load16U { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Uload16, I64, builder, state, environ); } - Operator::I64Load8S { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load8S { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Sload8, I64, builder, state, environ); } - Operator::I64Load16S { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load16S { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Sload16, I64, builder, state, environ); } - Operator::I64Load32S { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load32S { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Sload32, I64, builder, state, environ); } - Operator::I64Load32U { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load32U { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Uload32, I64, builder, state, environ); } - Operator::I32Load { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I32Load { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Load, I32, builder, state, environ); } - Operator::F32Load { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::F32Load { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Load, F32, builder, state, environ); } - Operator::I64Load { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I64Load { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Load, I64, builder, state, environ); } - Operator::F64Load { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::F64Load { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_load(offset, ir::Opcode::Load, F64, builder, state, environ); } /****************************** Store instructions *********************************** @@ -460,21 +477,39 @@ pub fn translate_operator( * The memory base address is provided by the environment. * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not ************************************************************************************/ - Operator::I32Store { memarg: MemoryImmediate { flags: _, offset } } | - Operator::I64Store { memarg: MemoryImmediate { flags: _, offset } } | - Operator::F32Store { memarg: MemoryImmediate { flags: _, offset } } | - Operator::F64Store { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I32Store { + memarg: MemoryImmediate { flags: _, offset }, + } + | Operator::I64Store { + memarg: MemoryImmediate { flags: _, offset }, + } + | Operator::F32Store { + memarg: MemoryImmediate { flags: _, offset }, + } + | Operator::F64Store { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_store(offset, ir::Opcode::Store, builder, state, environ); } - Operator::I32Store8 { memarg: MemoryImmediate { flags: _, offset } } | - Operator::I64Store8 { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I32Store8 { + memarg: MemoryImmediate { flags: _, offset }, + } + | Operator::I64Store8 { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_store(offset, ir::Opcode::Istore8, builder, state, environ); } - Operator::I32Store16 { memarg: MemoryImmediate { flags: _, offset } } | - Operator::I64Store16 { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I32Store16 { + memarg: MemoryImmediate { flags: _, offset }, + } + | Operator::I64Store16 { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_store(offset, ir::Opcode::Istore16, builder, state, environ); } - Operator::I64Store32 { memarg: MemoryImmediate { flags: _, offset } } => { + Operator::I64Store32 { + memarg: MemoryImmediate { flags: _, offset }, + } => { translate_store(offset, ir::Opcode::Istore32, builder, state, environ); } /****************************** Nullary Operators ************************************/ @@ -495,8 +530,7 @@ pub fn translate_operator( let arg = state.pop1(); state.push1(builder.ins().ctz(arg)); } - Operator::I32Popcnt | - Operator::I64Popcnt => { + Operator::I32Popcnt | Operator::I64Popcnt => { let arg = state.pop1(); state.push1(builder.ins().popcnt(arg)); } @@ -512,28 +546,23 @@ pub fn translate_operator( let val = state.pop1(); state.push1(builder.ins().ireduce(I32, val)); } - Operator::F32Sqrt | - Operator::F64Sqrt => { + Operator::F32Sqrt | Operator::F64Sqrt => { let arg = state.pop1(); state.push1(builder.ins().sqrt(arg)); } - Operator::F32Ceil | - Operator::F64Ceil => { + Operator::F32Ceil | Operator::F64Ceil => { let arg = state.pop1(); state.push1(builder.ins().ceil(arg)); } - Operator::F32Floor | - Operator::F64Floor => { + Operator::F32Floor | Operator::F64Floor => { let arg = state.pop1(); state.push1(builder.ins().floor(arg)); } - Operator::F32Trunc | - Operator::F64Trunc => { + Operator::F32Trunc | Operator::F64Trunc => { let arg = state.pop1(); state.push1(builder.ins().trunc(arg)); } - Operator::F32Nearest | - Operator::F64Nearest => { + Operator::F32Nearest | Operator::F64Nearest => { let arg = state.pop1(); state.push1(builder.ins().nearest(arg)); } @@ -545,23 +574,19 @@ pub fn translate_operator( let arg = state.pop1(); state.push1(builder.ins().fneg(arg)); } - Operator::F64ConvertUI64 | - Operator::F64ConvertUI32 => { + Operator::F64ConvertUI64 | Operator::F64ConvertUI32 => { let val = state.pop1(); state.push1(builder.ins().fcvt_from_uint(F64, val)); } - Operator::F64ConvertSI64 | - Operator::F64ConvertSI32 => { + Operator::F64ConvertSI64 | Operator::F64ConvertSI32 => { let val = state.pop1(); state.push1(builder.ins().fcvt_from_sint(F64, val)); } - Operator::F32ConvertSI64 | - Operator::F32ConvertSI32 => { + Operator::F32ConvertSI64 | Operator::F32ConvertSI32 => { let val = state.pop1(); state.push1(builder.ins().fcvt_from_sint(F32, val)); } - Operator::F32ConvertUI64 | - Operator::F32ConvertUI32 => { + Operator::F32ConvertUI64 | Operator::F32ConvertUI32 => { let val = state.pop1(); state.push1(builder.ins().fcvt_from_uint(F32, val)); } @@ -573,34 +598,30 @@ pub fn translate_operator( let val = state.pop1(); state.push1(builder.ins().fdemote(F32, val)); } - Operator::I64TruncSF64 | - Operator::I64TruncSF32 => { + Operator::I64TruncSF64 | Operator::I64TruncSF32 => { let val = state.pop1(); state.push1(builder.ins().fcvt_to_sint(I64, val)); } - Operator::I32TruncSF64 | - Operator::I32TruncSF32 => { + Operator::I32TruncSF64 | Operator::I32TruncSF32 => { let val = state.pop1(); state.push1(builder.ins().fcvt_to_sint(I32, val)); } - Operator::I64TruncUF64 | - Operator::I64TruncUF32 => { + Operator::I64TruncUF64 | Operator::I64TruncUF32 => { let val = state.pop1(); state.push1(builder.ins().fcvt_to_uint(I64, val)); } - Operator::I32TruncUF64 | - Operator::I32TruncUF32 => { + Operator::I32TruncUF64 | Operator::I32TruncUF32 => { let val = state.pop1(); state.push1(builder.ins().fcvt_to_uint(I32, val)); } - Operator::I64TruncSSatF64 | - Operator::I64TruncSSatF32 | - Operator::I32TruncSSatF64 | - Operator::I32TruncSSatF32 | - Operator::I64TruncUSatF64 | - Operator::I64TruncUSatF32 | - Operator::I32TruncUSatF64 | - Operator::I32TruncUSatF32 => { + Operator::I64TruncSSatF64 + | Operator::I64TruncSSatF32 + | Operator::I32TruncSSatF64 + | Operator::I32TruncSSatF32 + | Operator::I64TruncUSatF64 + | Operator::I64TruncUSatF32 + | Operator::I32TruncUSatF64 + | Operator::I32TruncUSatF32 => { panic!("proposed saturating conversion operators not yet supported"); } Operator::F32ReinterpretI32 => { @@ -670,23 +691,19 @@ pub fn translate_operator( let (arg1, arg2) = state.pop2(); state.push1(builder.ins().ishl(arg1, arg2)); } - Operator::I32ShrS | - Operator::I64ShrS => { + Operator::I32ShrS | Operator::I64ShrS => { let (arg1, arg2) = state.pop2(); state.push1(builder.ins().sshr(arg1, arg2)); } - Operator::I32ShrU | - Operator::I64ShrU => { + Operator::I32ShrU | Operator::I64ShrU => { let (arg1, arg2) = state.pop2(); state.push1(builder.ins().ushr(arg1, arg2)); } - Operator::I32Rotl | - Operator::I64Rotl => { + Operator::I32Rotl | Operator::I64Rotl => { let (arg1, arg2) = state.pop2(); state.push1(builder.ins().rotl(arg1, arg2)); } - Operator::I32Rotr | - Operator::I64Rotr => { + Operator::I32Rotr | Operator::I64Rotr => { let (arg1, arg2) = state.pop2(); state.push1(builder.ins().rotr(arg1, arg2)); } @@ -714,23 +731,19 @@ pub fn translate_operator( let (arg1, arg2) = state.pop2(); state.push1(builder.ins().fdiv(arg1, arg2)); } - Operator::I32DivS | - Operator::I64DivS => { + Operator::I32DivS | Operator::I64DivS => { let (arg1, arg2) = state.pop2(); state.push1(builder.ins().sdiv(arg1, arg2)); } - Operator::I32DivU | - Operator::I64DivU => { + Operator::I32DivU | Operator::I64DivU => { let (arg1, arg2) = state.pop2(); state.push1(builder.ins().udiv(arg1, arg2)); } - Operator::I32RemS | - Operator::I64RemS => { + Operator::I32RemS | Operator::I64RemS => { let (arg1, arg2) = state.pop2(); state.push1(builder.ins().srem(arg1, arg2)); } - Operator::I32RemU | - Operator::I64RemU => { + Operator::I32RemU | Operator::I64RemU => { let (arg1, arg2) = state.pop2(); state.push1(builder.ins().urem(arg1, arg2)); } @@ -742,8 +755,7 @@ pub fn translate_operator( let (arg1, arg2) = state.pop2(); state.push1(builder.ins().fmax(arg1, arg2)); } - Operator::F32Copysign | - Operator::F64Copysign => { + Operator::F32Copysign | Operator::F64Copysign => { let (arg1, arg2) = state.pop2(); state.push1(builder.ins().fcopysign(arg1, arg2)); } @@ -789,72 +801,72 @@ pub fn translate_operator( Operator::F32Le | Operator::F64Le => { translate_fcmp(FloatCC::LessThanOrEqual, builder, state) } - Operator::Wake { .. } | - Operator::I32Wait { .. } | - Operator::I64Wait { .. } | - Operator::I32AtomicLoad { .. } | - Operator::I64AtomicLoad { .. } | - Operator::I32AtomicLoad8U { .. } | - Operator::I32AtomicLoad16U { .. } | - Operator::I64AtomicLoad8U { .. } | - Operator::I64AtomicLoad16U { .. } | - Operator::I64AtomicLoad32U { .. } | - Operator::I32AtomicStore { .. } | - Operator::I64AtomicStore { .. } | - Operator::I32AtomicStore8 { .. } | - Operator::I32AtomicStore16 { .. } | - Operator::I64AtomicStore8 { .. } | - Operator::I64AtomicStore16 { .. } | - Operator::I64AtomicStore32 { .. } | - Operator::I32AtomicRmwAdd { .. } | - Operator::I64AtomicRmwAdd { .. } | - Operator::I32AtomicRmw8UAdd { .. } | - Operator::I32AtomicRmw16UAdd { .. } | - Operator::I64AtomicRmw8UAdd { .. } | - Operator::I64AtomicRmw16UAdd { .. } | - Operator::I64AtomicRmw32UAdd { .. } | - Operator::I32AtomicRmwSub { .. } | - Operator::I64AtomicRmwSub { .. } | - Operator::I32AtomicRmw8USub { .. } | - Operator::I32AtomicRmw16USub { .. } | - Operator::I64AtomicRmw8USub { .. } | - Operator::I64AtomicRmw16USub { .. } | - Operator::I64AtomicRmw32USub { .. } | - Operator::I32AtomicRmwAnd { .. } | - Operator::I64AtomicRmwAnd { .. } | - Operator::I32AtomicRmw8UAnd { .. } | - Operator::I32AtomicRmw16UAnd { .. } | - Operator::I64AtomicRmw8UAnd { .. } | - Operator::I64AtomicRmw16UAnd { .. } | - Operator::I64AtomicRmw32UAnd { .. } | - Operator::I32AtomicRmwOr { .. } | - Operator::I64AtomicRmwOr { .. } | - Operator::I32AtomicRmw8UOr { .. } | - Operator::I32AtomicRmw16UOr { .. } | - Operator::I64AtomicRmw8UOr { .. } | - Operator::I64AtomicRmw16UOr { .. } | - Operator::I64AtomicRmw32UOr { .. } | - Operator::I32AtomicRmwXor { .. } | - Operator::I64AtomicRmwXor { .. } | - Operator::I32AtomicRmw8UXor { .. } | - Operator::I32AtomicRmw16UXor { .. } | - Operator::I64AtomicRmw8UXor { .. } | - Operator::I64AtomicRmw16UXor { .. } | - Operator::I64AtomicRmw32UXor { .. } | - Operator::I32AtomicRmwXchg { .. } | - Operator::I64AtomicRmwXchg { .. } | - Operator::I32AtomicRmw8UXchg { .. } | - Operator::I32AtomicRmw16UXchg { .. } | - Operator::I64AtomicRmw8UXchg { .. } | - Operator::I64AtomicRmw16UXchg { .. } | - Operator::I64AtomicRmw32UXchg { .. } | - Operator::I32AtomicRmwCmpxchg { .. } | - Operator::I64AtomicRmwCmpxchg { .. } | - Operator::I32AtomicRmw8UCmpxchg { .. } | - Operator::I32AtomicRmw16UCmpxchg { .. } | - Operator::I64AtomicRmw8UCmpxchg { .. } | - Operator::I64AtomicRmw16UCmpxchg { .. } | - Operator::I64AtomicRmw32UCmpxchg { .. } => { + Operator::Wake { .. } + | Operator::I32Wait { .. } + | Operator::I64Wait { .. } + | Operator::I32AtomicLoad { .. } + | Operator::I64AtomicLoad { .. } + | Operator::I32AtomicLoad8U { .. } + | Operator::I32AtomicLoad16U { .. } + | Operator::I64AtomicLoad8U { .. } + | Operator::I64AtomicLoad16U { .. } + | Operator::I64AtomicLoad32U { .. } + | Operator::I32AtomicStore { .. } + | Operator::I64AtomicStore { .. } + | Operator::I32AtomicStore8 { .. } + | Operator::I32AtomicStore16 { .. } + | Operator::I64AtomicStore8 { .. } + | Operator::I64AtomicStore16 { .. } + | Operator::I64AtomicStore32 { .. } + | Operator::I32AtomicRmwAdd { .. } + | Operator::I64AtomicRmwAdd { .. } + | Operator::I32AtomicRmw8UAdd { .. } + | Operator::I32AtomicRmw16UAdd { .. } + | Operator::I64AtomicRmw8UAdd { .. } + | Operator::I64AtomicRmw16UAdd { .. } + | Operator::I64AtomicRmw32UAdd { .. } + | Operator::I32AtomicRmwSub { .. } + | Operator::I64AtomicRmwSub { .. } + | Operator::I32AtomicRmw8USub { .. } + | Operator::I32AtomicRmw16USub { .. } + | Operator::I64AtomicRmw8USub { .. } + | Operator::I64AtomicRmw16USub { .. } + | Operator::I64AtomicRmw32USub { .. } + | Operator::I32AtomicRmwAnd { .. } + | Operator::I64AtomicRmwAnd { .. } + | Operator::I32AtomicRmw8UAnd { .. } + | Operator::I32AtomicRmw16UAnd { .. } + | Operator::I64AtomicRmw8UAnd { .. } + | Operator::I64AtomicRmw16UAnd { .. } + | Operator::I64AtomicRmw32UAnd { .. } + | Operator::I32AtomicRmwOr { .. } + | Operator::I64AtomicRmwOr { .. } + | Operator::I32AtomicRmw8UOr { .. } + | Operator::I32AtomicRmw16UOr { .. } + | Operator::I64AtomicRmw8UOr { .. } + | Operator::I64AtomicRmw16UOr { .. } + | Operator::I64AtomicRmw32UOr { .. } + | Operator::I32AtomicRmwXor { .. } + | Operator::I64AtomicRmwXor { .. } + | Operator::I32AtomicRmw8UXor { .. } + | Operator::I32AtomicRmw16UXor { .. } + | Operator::I64AtomicRmw8UXor { .. } + | Operator::I64AtomicRmw16UXor { .. } + | Operator::I64AtomicRmw32UXor { .. } + | Operator::I32AtomicRmwXchg { .. } + | Operator::I64AtomicRmwXchg { .. } + | Operator::I32AtomicRmw8UXchg { .. } + | Operator::I32AtomicRmw16UXchg { .. } + | Operator::I64AtomicRmw8UXchg { .. } + | Operator::I64AtomicRmw16UXchg { .. } + | Operator::I64AtomicRmw32UXchg { .. } + | Operator::I32AtomicRmwCmpxchg { .. } + | Operator::I64AtomicRmwCmpxchg { .. } + | Operator::I32AtomicRmw8UCmpxchg { .. } + | Operator::I32AtomicRmw16UCmpxchg { .. } + | Operator::I64AtomicRmw8UCmpxchg { .. } + | Operator::I64AtomicRmw16UCmpxchg { .. } + | Operator::I64AtomicRmw32UCmpxchg { .. } => { panic!("proposed thread operators not yet supported"); } }) @@ -876,8 +888,7 @@ fn translate_unreachable_operator( // so we don't have any branches anywhere. state.push_if(ir::Inst::reserved_value(), ir::Ebb::reserved_value(), 0); } - Operator::Loop { ty: _ } | - Operator::Block { ty: _ } => { + Operator::Loop { ty: _ } | Operator::Block { ty: _ } => { state.push_block(ir::Ebb::reserved_value(), 0); } Operator::Else => { @@ -919,7 +930,9 @@ fn translate_unreachable_operator( // And loops can't have branches to the end. false } - ControlStackFrame::If { reachable_from_top, .. } => { + ControlStackFrame::If { + reachable_from_top, .. + } => { // A reachable if without an else has a branch from the top // directly to the bottom. reachable_from_top @@ -998,13 +1011,9 @@ fn translate_load( // alignment immediate says it's aligned, because WebAssembly's immediate // field is just a hint, while Cretonne's aligned flag needs a guarantee. let flags = MemFlags::new(); - let (load, dfg) = builder.ins().Load( - opcode, - result_ty, - flags, - offset.into(), - base, - ); + let (load, dfg) = builder + .ins() + .Load(opcode, result_ty, flags, offset.into(), base); state.push1(dfg.first_result(load)); } @@ -1024,14 +1033,9 @@ fn translate_store( let (base, offset) = get_heap_addr(heap, addr32, offset, environ.native_pointer(), builder); // See the comments in `translate_load` about the flags. let flags = MemFlags::new(); - builder.ins().Store( - opcode, - val_ty, - flags, - offset.into(), - val, - base, - ); + builder + .ins() + .Store(opcode, val_ty, flags, offset.into(), val, base); } fn translate_icmp( diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 90f1af3b5a..8eae69fc2b 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -165,7 +165,9 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ base: ir::HeapBase::GlobalVar(gv), min_size: 0.into(), guard_size: 0x8000_0000.into(), - style: ir::HeapStyle::Static { bound: 0x1_0000_0000.into() }, + style: ir::HeapStyle::Static { + bound: 0x1_0000_0000.into(), + }, }) } @@ -224,11 +226,9 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ args.extend(call_args.iter().cloned(), &mut pos.func.dfg.value_lists); args.push(vmctx, &mut pos.func.dfg.value_lists); - Ok( - pos.ins() - .CallIndirect(ir::Opcode::CallIndirect, VOID, sig_ref, args) - .0, - ) + Ok(pos.ins() + .CallIndirect(ir::Opcode::CallIndirect, VOID, sig_ref, args) + .0) } fn translate_call( @@ -301,10 +301,9 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment { "Imported functions must be declared first" ); self.info.functions.push(Exportable::new(sig_index)); - self.info.imported_funcs.push(( - String::from(module), - String::from(field), - )); + self.info + .imported_funcs + .push((String::from(module), String::from(field))); } fn get_num_func_imports(&self) -> usize { @@ -353,33 +352,27 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment { } fn declare_func_export(&mut self, func_index: FunctionIndex, name: &'data str) { - self.info.functions[func_index].export_names.push( - String::from( - name, - ), - ); + self.info.functions[func_index] + .export_names + .push(String::from(name)); } fn declare_table_export(&mut self, table_index: TableIndex, name: &'data str) { - self.info.tables[table_index].export_names.push( - String::from(name), - ); + self.info.tables[table_index] + .export_names + .push(String::from(name)); } fn declare_memory_export(&mut self, memory_index: MemoryIndex, name: &'data str) { - self.info.memories[memory_index].export_names.push( - String::from( - name, - ), - ); + self.info.memories[memory_index] + .export_names + .push(String::from(name)); } fn declare_global_export(&mut self, global_index: GlobalIndex, name: &'data str) { - self.info.globals[global_index].export_names.push( - String::from( - name, - ), - ); + self.info.globals[global_index] + .export_names + .push(String::from(name)); } fn declare_start_func(&mut self, func_index: FunctionIndex) { @@ -395,11 +388,8 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment { let sig = func_environ.vmctx_sig(self.get_func_type(function_index)); let mut func = ir::Function::with_name_signature(name, sig); let reader = wasmparser::BinaryReader::new(body_bytes); - self.trans.translate_from_reader( - reader, - &mut func, - &mut func_environ, - )?; + self.trans + .translate_from_reader(reader, &mut func, &mut func_environ)?; func }; self.func_bytecode_sizes.push(body_bytes.len()); diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index ed693af4bf..fbac424fef 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -135,16 +135,16 @@ fn parse_local_decls( num_params: usize, ) -> WasmResult<()> { let mut next_local = num_params; - let local_count = reader.read_local_count().map_err(|e| { - WasmError::from_binary_reader_error(e) - })?; + let local_count = reader + .read_local_count() + .map_err(|e| WasmError::from_binary_reader_error(e))?; let mut locals_total = 0; for _ in 0..local_count { builder.set_srcloc(cur_srcloc(reader)); - let (count, ty) = reader.read_local_decl(&mut locals_total).map_err(|e| { - WasmError::from_binary_reader_error(e) - })?; + let (count, ty) = reader + .read_local_decl(&mut locals_total) + .map_err(|e| WasmError::from_binary_reader_error(e))?; declare_locals(builder, count, ty, &mut next_local); } @@ -195,9 +195,9 @@ fn parse_function_body( // Keep going until the final `End` operator which pops the outermost block. while !state.control_stack.is_empty() { builder.set_srcloc(cur_srcloc(&reader)); - let op = reader.read_operator().map_err(|e| { - WasmError::from_binary_reader_error(e) - })?; + let op = reader + .read_operator() + .map_err(|e| WasmError::from_binary_reader_error(e))?; translate_operator(op, builder, state, environ)?; } @@ -246,11 +246,13 @@ mod tests { // (i32.add (get_local 0) (i32.const 1)) // ) const BODY: [u8; 7] = [ - 0x00, // local decl count - 0x20, 0x00, // get_local 0 - 0x41, 0x01, // i32.const 1 - 0x6a, // i32.add - 0x0b, // end + 0x00, // local decl count + 0x20, + 0x00, // get_local 0 + 0x41, + 0x01, // i32.const 1 + 0x6a, // i32.add + 0x0b, // end ]; let mut trans = FuncTranslator::new(); @@ -276,12 +278,14 @@ mod tests { // (return (i32.add (get_local 0) (i32.const 1))) // ) const BODY: [u8; 8] = [ - 0x00, // local decl count - 0x20, 0x00, // get_local 0 - 0x41, 0x01, // i32.const 1 - 0x6a, // i32.add - 0x0f, // return - 0x0b, // end + 0x00, // local decl count + 0x20, + 0x00, // get_local 0 + 0x41, + 0x01, // i32.const 1 + 0x6a, // i32.add + 0x0f, // return + 0x0b, // end ]; let mut trans = FuncTranslator::new(); @@ -312,16 +316,22 @@ mod tests { // ) // ) const BODY: [u8; 16] = [ - 0x01, // 1 local decl. - 0x01, 0x7f, // 1 i32 local. - 0x03, 0x7f, // loop i32 - 0x20, 0x00, // get_local 0 - 0x41, 0x01, // i32.const 0 - 0x6a, // i32.add - 0x21, 0x00, // set_local 0 - 0x0c, 0x00, // br 0 - 0x0b, // end - 0x0b, // end + 0x01, // 1 local decl. + 0x01, + 0x7f, // 1 i32 local. + 0x03, + 0x7f, // loop i32 + 0x20, + 0x00, // get_local 0 + 0x41, + 0x01, // i32.const 0 + 0x6a, // i32.add + 0x21, + 0x00, // set_local 0 + 0x0c, + 0x00, // br 0 + 0x0b, // end + 0x0b, // end ]; let mut trans = FuncTranslator::new(); diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 87d1d87ee2..9842f36852 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -13,19 +13,10 @@ #![warn(unused_import_braces)] #![cfg_attr(feature = "std", warn(unstable_features))] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] +#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] #![cfg_attr(feature = "cargo-clippy", - allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature="cargo-clippy", warn( - float_arithmetic, - mut_mut, - nonminimal_bool, - option_map_unwrap_or, - option_map_unwrap_or_else, - print_stdout, - unicode_not_nfc, - use_self, - ))] - + warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, + option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] #![cfg_attr(not(feature = "std"), no_std)] #![cfg_attr(not(feature = "std"), feature(alloc))] @@ -62,12 +53,12 @@ pub use translation_utils::{FunctionIndex, Global, GlobalIndex, GlobalInit, Memo #[cfg(not(feature = "std"))] mod std { - pub use alloc::vec; pub use alloc::string; - pub use core::{u32, i32, str, cmp}; + pub use alloc::vec; pub use core::fmt; pub use core::option; + pub use core::{cmp, str, i32, u32}; pub mod collections { - pub use hashmap_core::{HashMap, map as hash_map}; + pub use hashmap_core::{map as hash_map, HashMap}; } } diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 683517e7f5..835a26fb3f 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -28,42 +28,72 @@ pub fn translate_module<'data>( let mut next_input = ParserInput::Default; loop { match *parser.read_with_input(next_input) { - ParserState::BeginSection { code: SectionCode::Type, .. } => { + ParserState::BeginSection { + code: SectionCode::Type, + .. + } => { parse_function_signatures(&mut parser, environ)?; next_input = ParserInput::Default; } - ParserState::BeginSection { code: SectionCode::Import, .. } => { + ParserState::BeginSection { + code: SectionCode::Import, + .. + } => { parse_import_section(&mut parser, environ)?; next_input = ParserInput::Default; } - ParserState::BeginSection { code: SectionCode::Function, .. } => { + ParserState::BeginSection { + code: SectionCode::Function, + .. + } => { parse_function_section(&mut parser, environ)?; next_input = ParserInput::Default; } - ParserState::BeginSection { code: SectionCode::Table, .. } => { + ParserState::BeginSection { + code: SectionCode::Table, + .. + } => { parse_table_section(&mut parser, environ)?; } - ParserState::BeginSection { code: SectionCode::Memory, .. } => { + ParserState::BeginSection { + code: SectionCode::Memory, + .. + } => { parse_memory_section(&mut parser, environ)?; next_input = ParserInput::Default; } - ParserState::BeginSection { code: SectionCode::Global, .. } => { + ParserState::BeginSection { + code: SectionCode::Global, + .. + } => { parse_global_section(&mut parser, environ)?; next_input = ParserInput::Default; } - ParserState::BeginSection { code: SectionCode::Export, .. } => { + ParserState::BeginSection { + code: SectionCode::Export, + .. + } => { parse_export_section(&mut parser, environ)?; next_input = ParserInput::Default; } - ParserState::BeginSection { code: SectionCode::Start, .. } => { + ParserState::BeginSection { + code: SectionCode::Start, + .. + } => { parse_start_section(&mut parser, environ)?; next_input = ParserInput::Default; } - ParserState::BeginSection { code: SectionCode::Element, .. } => { + ParserState::BeginSection { + code: SectionCode::Element, + .. + } => { parse_elements_section(&mut parser, environ)?; next_input = ParserInput::Default; } - ParserState::BeginSection { code: SectionCode::Code, .. } => { + ParserState::BeginSection { + code: SectionCode::Code, + .. + } => { // The code section begins break; } @@ -71,10 +101,16 @@ pub fn translate_module<'data>( next_input = ParserInput::Default; } ParserState::EndWasm => return Ok(()), - ParserState::BeginSection { code: SectionCode::Data, .. } => { + ParserState::BeginSection { + code: SectionCode::Data, + .. + } => { parse_data_section(&mut parser, environ)?; } - ParserState::BeginSection { code: SectionCode::Custom { .. }, .. } => { + ParserState::BeginSection { + code: SectionCode::Custom { .. }, + .. + } => { // Ignore unknown custom sections. next_input = ParserInput::SkipSection; } @@ -92,15 +128,16 @@ pub fn translate_module<'data>( } let mut reader = parser.create_binary_reader(); let size = reader.bytes_remaining(); - environ.define_function_body( - reader.read_bytes(size).map_err(|e| { - WasmError::from_binary_reader_error(e) - })?, - )?; + environ.define_function_body(reader + .read_bytes(size) + .map_err(|e| WasmError::from_binary_reader_error(e))?)?; } loop { match *parser.read() { - ParserState::BeginSection { code: SectionCode::Data, .. } => { + ParserState::BeginSection { + code: SectionCode::Data, + .. + } => { parse_data_section(&mut parser, environ)?; } ParserState::EndWasm => break, diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 222433fafb..99a0051121 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -26,21 +26,19 @@ pub fn parse_function_signatures( match *parser.read() { ParserState::EndSection => break, ParserState::TypeSectionEntry(FuncType { - form: wasmparser::Type::Func, - ref params, - ref returns, - }) => { + form: wasmparser::Type::Func, + ref params, + ref returns, + }) => { let mut sig = Signature::new(environ.flags().call_conv()); sig.params.extend(params.iter().map(|ty| { - let cret_arg: ir::Type = type_to_type(ty).expect( - "only numeric types are supported in function signatures", - ); + let cret_arg: ir::Type = type_to_type(ty) + .expect("only numeric types are supported in function signatures"); AbiParam::new(cret_arg) })); sig.returns.extend(returns.iter().map(|ty| { - let cret_arg: ir::Type = type_to_type(ty).expect( - "only numeric types are supported in function signatures", - ); + let cret_arg: ir::Type = type_to_type(ty) + .expect("only numeric types are supported in function signatures"); AbiParam::new(cret_arg) })); environ.declare_signature(&sig); @@ -72,10 +70,11 @@ pub fn parse_import_section<'data>( environ.declare_func_import(sig as SignatureIndex, module_name, field_name); } ParserState::ImportSectionEntry { - ty: ImportSectionEntryType::Memory(MemoryType { - limits: ref memlimits, - shared, - }), + ty: + ImportSectionEntryType::Memory(MemoryType { + limits: ref memlimits, + shared, + }), .. } => { environ.declare_memory(Memory { @@ -85,7 +84,8 @@ pub fn parse_import_section<'data>( }); } ParserState::ImportSectionEntry { - ty: ImportSectionEntryType::Global(ref ty), .. + ty: ImportSectionEntryType::Global(ref ty), + .. } => { environ.declare_global(Global { ty: type_to_type(&ty.content_type).unwrap(), @@ -94,17 +94,16 @@ pub fn parse_import_section<'data>( }); } ParserState::ImportSectionEntry { - ty: ImportSectionEntryType::Table(ref tab), .. - } => { - environ.declare_table(Table { - ty: match type_to_type(&tab.element_type) { - Ok(t) => TableElementType::Val(t), - Err(()) => TableElementType::Func(), - }, - size: tab.limits.initial as usize, - maximum: tab.limits.maximum.map(|x| x as usize), - }) - } + ty: ImportSectionEntryType::Table(ref tab), + .. + } => environ.declare_table(Table { + ty: match type_to_type(&tab.element_type) { + Ok(t) => TableElementType::Val(t), + Err(()) => TableElementType::Func(), + }, + size: tab.limits.initial as usize, + maximum: tab.limits.maximum.map(|x| x as usize), + }), ParserState::EndSection => break, ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), ref s => panic!("unexpected section content: {:?}", s), @@ -325,16 +324,14 @@ pub fn parse_data_section<'data>( pub fn parse_table_section(parser: &mut Parser, environ: &mut ModuleEnvironment) -> WasmResult<()> { loop { match *parser.read() { - ParserState::TableSectionEntry(ref table) => { - environ.declare_table(Table { - ty: match type_to_type(&table.element_type) { - Ok(t) => TableElementType::Val(t), - Err(()) => TableElementType::Func(), - }, - size: table.limits.initial as usize, - maximum: table.limits.maximum.map(|x| x as usize), - }) - } + ParserState::TableSectionEntry(ref table) => environ.declare_table(Table { + ty: match type_to_type(&table.element_type) { + Ok(t) => TableElementType::Val(t), + Err(()) => TableElementType::Func(), + }, + size: table.limits.initial as usize, + maximum: table.limits.maximum.map(|x| x as usize), + }), ParserState::EndSection => break, ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)), ref s => panic!("unexpected section content: {:?}", s), diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index c322906c95..b0bb4f5c7a 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -47,54 +47,78 @@ pub enum ControlStackFrame { impl ControlStackFrame { pub fn num_return_values(&self) -> usize { match *self { - ControlStackFrame::If { num_return_values, .. } | - ControlStackFrame::Block { num_return_values, .. } | - ControlStackFrame::Loop { num_return_values, .. } => num_return_values, + ControlStackFrame::If { + num_return_values, .. + } + | ControlStackFrame::Block { + num_return_values, .. + } + | ControlStackFrame::Loop { + num_return_values, .. + } => num_return_values, } } pub fn following_code(&self) -> Ebb { match *self { - ControlStackFrame::If { destination, .. } | - ControlStackFrame::Block { destination, .. } | - ControlStackFrame::Loop { destination, .. } => destination, + ControlStackFrame::If { destination, .. } + | ControlStackFrame::Block { destination, .. } + | ControlStackFrame::Loop { destination, .. } => destination, } } pub fn br_destination(&self) -> Ebb { match *self { - ControlStackFrame::If { destination, .. } | - ControlStackFrame::Block { destination, .. } => destination, + ControlStackFrame::If { destination, .. } + | ControlStackFrame::Block { destination, .. } => destination, ControlStackFrame::Loop { header, .. } => header, } } pub fn original_stack_size(&self) -> usize { match *self { - ControlStackFrame::If { original_stack_size, .. } | - ControlStackFrame::Block { original_stack_size, .. } | - ControlStackFrame::Loop { original_stack_size, .. } => original_stack_size, + ControlStackFrame::If { + original_stack_size, + .. + } + | ControlStackFrame::Block { + original_stack_size, + .. + } + | ControlStackFrame::Loop { + original_stack_size, + .. + } => original_stack_size, } } pub fn is_loop(&self) -> bool { match *self { - ControlStackFrame::If { .. } | - ControlStackFrame::Block { .. } => false, + ControlStackFrame::If { .. } | ControlStackFrame::Block { .. } => false, ControlStackFrame::Loop { .. } => true, } } pub fn exit_is_branched_to(&self) -> bool { match *self { - ControlStackFrame::If { exit_is_branched_to, .. } | - ControlStackFrame::Block { exit_is_branched_to, .. } => exit_is_branched_to, + ControlStackFrame::If { + exit_is_branched_to, + .. + } + | ControlStackFrame::Block { + exit_is_branched_to, + .. + } => exit_is_branched_to, ControlStackFrame::Loop { .. } => false, } } pub fn set_branched_to_exit(&mut self) { match *self { - ControlStackFrame::If { ref mut exit_is_branched_to, .. } | - ControlStackFrame::Block { ref mut exit_is_branched_to, .. } => { - *exit_is_branched_to = true + ControlStackFrame::If { + ref mut exit_is_branched_to, + .. } + | ControlStackFrame::Block { + ref mut exit_is_branched_to, + .. + } => *exit_is_branched_to = true, ControlStackFrame::Loop { .. } => {} } } @@ -258,9 +282,9 @@ impl TranslationState { environ: &mut FE, ) -> GlobalValue { let index = index as GlobalIndex; - *self.globals.entry(index).or_insert_with( - || environ.make_global(func, index), - ) + *self.globals + .entry(index) + .or_insert_with(|| environ.make_global(func, index)) } /// Get the `Heap` reference that should be used to access linear memory `index`. @@ -272,9 +296,9 @@ impl TranslationState { environ: &mut FE, ) -> ir::Heap { let index = index as MemoryIndex; - *self.heaps.entry(index).or_insert_with( - || environ.make_heap(func, index), - ) + *self.heaps + .entry(index) + .or_insert_with(|| environ.make_heap(func, index)) } /// Get the `SigRef` reference that should be used to make an indirect call with signature diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 0089c5a75d..3ed13899e6 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -96,10 +96,10 @@ pub fn f64_translation(x: wasmparser::Ieee64) -> ir::immediates::Ieee64 { pub fn num_return_values(ty: wasmparser::Type) -> usize { match ty { wasmparser::Type::EmptyBlockType => 0, - wasmparser::Type::I32 | - wasmparser::Type::F32 | - wasmparser::Type::I64 | - wasmparser::Type::F64 => 1, + wasmparser::Type::I32 + | wasmparser::Type::F32 + | wasmparser::Type::I64 + | wasmparser::Type::F64 => 1, _ => panic!("unsupported return value type"), } } diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs index 3492c7bd89..a5d5badaed 100644 --- a/lib/wasm/tests/wasm_testsuite.rs +++ b/lib/wasm/tests/wasm_testsuite.rs @@ -56,21 +56,19 @@ fn handle_module(path: &Path, flags: &Flags) { None => { panic!("the file extension is not wasm or wat"); } - Some(ext) => { - match ext.to_str() { - Some("wasm") => read_file(path).expect("error reading wasm file"), - Some("wat") => { - let wat = read_file(path).expect("error reading wat file"); - match wat2wasm(&wat) { - Ok(wasm) => wasm, - Err(e) => { - panic!("error converting wat to wasm: {:?}", e); - } + Some(ext) => match ext.to_str() { + Some("wasm") => read_file(path).expect("error reading wasm file"), + Some("wat") => { + let wat = read_file(path).expect("error reading wat file"); + match wat2wasm(&wat) { + Ok(wasm) => wasm, + Err(e) => { + panic!("error converting wat to wasm: {:?}", e); } } - None | Some(&_) => panic!("the file extension for {:?} is not wasm or wat", path), } - } + None | Some(&_) => panic!("the file extension for {:?} is not wasm or wat", path), + }, }; let mut dummy_environ = DummyEnvironment::with_flags(flags.clone()); translate_module(&data, &mut dummy_environ).unwrap(); From 09fb9148236b256ec5bde3efa4d174f5631bdab2 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 25 May 2018 08:53:52 -0700 Subject: [PATCH 1795/3084] Make an assert message more verbose. --- lib/codegen/src/abi.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/codegen/src/abi.rs b/lib/codegen/src/abi.rs index 66dcf7f8f1..e4b4b223f4 100644 --- a/lib/codegen/src/abi.rs +++ b/lib/codegen/src/abi.rs @@ -163,7 +163,7 @@ pub fn legalize_abi_value(have: Type, arg: &AbiParam) -> ValueConversion { Ordering::Equal => { // This must be an integer vector that is split and then extended. debug_assert!(arg.value_type.is_int()); - debug_assert!(have.is_vector()); + debug_assert!(have.is_vector(), "expected vector type, got {}", have); ValueConversion::VectorSplit } // We have more bits than the argument. From 7720a571fccd8722ed2882a9828eaf5c506bb38c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 May 2018 16:53:34 -0700 Subject: [PATCH 1796/3084] Fix reverse iteration in cretonne-entity's Iter. A `DoubleEndedIterator` needs to track both a forward position and a reverse position, so that `next_back()` can pop from the back of the sequence. --- lib/entity/src/iter.rs | 20 +++++++---- lib/entity/src/keys.rs | 6 ++-- lib/entity/src/map.rs | 6 ++-- lib/entity/src/primary.rs | 74 +++++++++++++++++++++++++++++++++++++-- lib/entity/src/set.rs | 2 +- 5 files changed, 92 insertions(+), 16 deletions(-) diff --git a/lib/entity/src/iter.rs b/lib/entity/src/iter.rs index 9eeba6c7a6..0455e40519 100644 --- a/lib/entity/src/iter.rs +++ b/lib/entity/src/iter.rs @@ -10,6 +10,7 @@ where V: 'a, { pos: usize, + rev_pos: usize, iter: slice::Iter<'a, V>, unused: PhantomData, } @@ -17,9 +18,10 @@ where impl<'a, K: EntityRef, V> Iter<'a, K, V> { /// Create an `Iter` iterator that visits the `PrimaryMap` keys and values /// of `iter`. - pub fn new(key: K, iter: slice::Iter<'a, V>) -> Self { + pub fn new(iter: slice::Iter<'a, V>) -> Self { Self { - pos: key.index(), + pos: 0, + rev_pos: iter.len(), iter, unused: PhantomData, } @@ -47,7 +49,9 @@ impl<'a, K: EntityRef, V> Iterator for Iter<'a, K, V> { impl<'a, K: EntityRef, V> DoubleEndedIterator for Iter<'a, K, V> { fn next_back(&mut self) -> Option { if let Some(next_back) = self.iter.next_back() { - Some((K::new(self.pos), next_back)) + let rev_pos = self.rev_pos - 1; + self.rev_pos -= 1; + Some((K::new(rev_pos), next_back)) } else { None } @@ -62,6 +66,7 @@ where V: 'a, { pos: usize, + rev_pos: usize, iter: slice::IterMut<'a, V>, unused: PhantomData, } @@ -69,9 +74,10 @@ where impl<'a, K: EntityRef, V> IterMut<'a, K, V> { /// Create an `IterMut` iterator that visits the `PrimaryMap` keys and values /// of `iter`. - pub fn new(key: K, iter: slice::IterMut<'a, V>) -> Self { + pub fn new(iter: slice::IterMut<'a, V>) -> Self { Self { - pos: key.index(), + pos: 0, + rev_pos: iter.len(), iter, unused: PhantomData, } @@ -99,7 +105,9 @@ impl<'a, K: EntityRef, V> Iterator for IterMut<'a, K, V> { impl<'a, K: EntityRef, V> DoubleEndedIterator for IterMut<'a, K, V> { fn next_back(&mut self) -> Option { if let Some(next_back) = self.iter.next_back() { - Some((K::new(self.pos), next_back)) + let rev_pos = self.rev_pos - 1; + self.rev_pos -= 1; + Some((K::new(rev_pos), next_back)) } else { None } diff --git a/lib/entity/src/keys.rs b/lib/entity/src/keys.rs index 52481097cb..e9d1e91505 100644 --- a/lib/entity/src/keys.rs +++ b/lib/entity/src/keys.rs @@ -11,11 +11,11 @@ pub struct Keys { } impl Keys { - /// Create a `Keys` iterator that visits `count` entities starting from 0. - pub fn new(count: usize) -> Self { + /// Create a `Keys` iterator that visits `len` entities starting from 0. + pub fn with_len(len: usize) -> Self { Self { pos: 0, - rev_pos: count, + rev_pos: len, unused: PhantomData, } } diff --git a/lib/entity/src/map.rs b/lib/entity/src/map.rs index ad8dca7e01..285e61e4c6 100644 --- a/lib/entity/src/map.rs +++ b/lib/entity/src/map.rs @@ -71,17 +71,17 @@ where /// Iterate over all the keys and values in this map. pub fn iter(&self) -> Iter { - Iter::new(K::new(0), self.elems.iter()) + Iter::new(self.elems.iter()) } /// Iterate over all the keys and values in this map, mutable edition. pub fn iter_mut(&mut self) -> IterMut { - IterMut::new(K::new(0), self.elems.iter_mut()) + IterMut::new(self.elems.iter_mut()) } /// Iterate over all the keys in this map. pub fn keys(&self) -> Keys { - Keys::new(self.elems.len()) + Keys::with_len(self.elems.len()) } /// Iterate over all the keys in this map. diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs index 36b76f09d6..4f7307fb88 100644 --- a/lib/entity/src/primary.rs +++ b/lib/entity/src/primary.rs @@ -57,7 +57,7 @@ where /// Iterate over all the keys in this map. pub fn keys(&self) -> Keys { - Keys::new(self.elems.len()) + Keys::with_len(self.elems.len()) } /// Iterate over all the values in this map. @@ -72,12 +72,12 @@ where /// Iterate over all the keys and values in this map. pub fn iter(&self) -> Iter { - Iter::new(K::new(0), self.elems.iter()) + Iter::new(self.elems.iter()) } /// Iterate over all the keys and values in this map, mutable edition. pub fn iter_mut(&mut self) -> IterMut { - IterMut::new(K::new(0), self.elems.iter_mut()) + IterMut::new(self.elems.iter_mut()) } /// Remove all entries from this map. @@ -192,6 +192,34 @@ mod tests { } } + #[test] + fn iter_rev() { + let mut m: PrimaryMap = PrimaryMap::new(); + m.push(12); + m.push(33); + + let mut i = 2; + for (key, value) in m.iter().rev() { + i -= 1; + assert_eq!(key.index(), i); + match i { + 0 => assert_eq!(*value, 12), + 1 => assert_eq!(*value, 33), + _ => panic!(), + } + } + + i = 2; + for (key, value) in m.iter_mut().rev() { + i -= 1; + assert_eq!(key.index(), i); + match i { + 0 => assert_eq!(*value, 12), + 1 => assert_eq!(*value, 33), + _ => panic!(), + } + } + } #[test] fn keys() { let mut m: PrimaryMap = PrimaryMap::new(); @@ -205,6 +233,19 @@ mod tests { } } + #[test] + fn keys_rev() { + let mut m: PrimaryMap = PrimaryMap::new(); + m.push(12); + m.push(33); + + let mut i = 2; + for key in m.keys().rev() { + i -= 1; + assert_eq!(key.index(), i); + } + } + #[test] fn values() { let mut m: PrimaryMap = PrimaryMap::new(); @@ -230,4 +271,31 @@ mod tests { i += 1; } } + + #[test] + fn values_rev() { + let mut m: PrimaryMap = PrimaryMap::new(); + m.push(12); + m.push(33); + + let mut i = 2; + for value in m.values().rev() { + i -= 1; + match i { + 0 => assert_eq!(*value, 12), + 1 => assert_eq!(*value, 33), + _ => panic!(), + } + } + i = 2; + for value_mut in m.values_mut().rev() { + i -= 1; + match i { + 0 => assert_eq!(*value_mut, 12), + 1 => assert_eq!(*value_mut, 33), + _ => panic!(), + } + } + } + } diff --git a/lib/entity/src/set.rs b/lib/entity/src/set.rs index 21f518eb66..7644db49ad 100644 --- a/lib/entity/src/set.rs +++ b/lib/entity/src/set.rs @@ -55,7 +55,7 @@ where /// Iterate over all the keys in this set. pub fn keys(&self) -> Keys { - Keys::new(self.len) + Keys::with_len(self.len) } /// Resize the set to have `n` entries by adding default entries as needed. From d8eaa19ddafbe8bfbbbe583b49809a67e6cbc4fb Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 May 2018 14:11:07 -0700 Subject: [PATCH 1797/3084] Make an assert message more verbose. --- lib/codegen/src/regalloc/reload.rs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/codegen/src/regalloc/reload.rs b/lib/codegen/src/regalloc/reload.rs index ff204a4fd9..b8ba2166ca 100644 --- a/lib/codegen/src/regalloc/reload.rs +++ b/lib/codegen/src/regalloc/reload.rs @@ -283,7 +283,11 @@ impl<'a> Context<'a> { .expect("Extra results on non-call instruction"); for (i, lv) in retvals.iter().enumerate() { let abi = self.cur.func.dfg.signatures[sig].returns[i]; - debug_assert!(abi.location.is_reg()); + debug_assert!( + abi.location.is_reg(), + "expected reg; got {:?}", + abi.location + ); if lv.affinity.is_stack() { let reg = self.cur.func.dfg.replace_result(lv.value, abi.value_type); self.liveness From c21af29c79d3c3f4b9d350757726d0d8961c964e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 May 2018 14:11:20 -0700 Subject: [PATCH 1798/3084] Legalize libcall signatures. Explicitly legalize signatures created for libcalls. --- .../filetests/isa/x86/legalize-libcall.cton | 3 ++- cranelift/filetests/wasm/f32-arith.cton | 6 ++++++ cranelift/filetests/wasm/f64-arith.cton | 3 +++ cranelift/filetests/wasm/i32-arith.cton | 6 ++++++ cranelift/filetests/wasm/i64-arith.cton | 3 +++ lib/codegen/src/legalizer/boundary.rs | 20 +++++++++++++++---- lib/codegen/src/legalizer/libcall.rs | 6 +++++- 7 files changed, 41 insertions(+), 6 deletions(-) diff --git a/cranelift/filetests/isa/x86/legalize-libcall.cton b/cranelift/filetests/isa/x86/legalize-libcall.cton index d633bd4e46..fdad2a3cf2 100644 --- a/cranelift/filetests/isa/x86/legalize-libcall.cton +++ b/cranelift/filetests/isa/x86/legalize-libcall.cton @@ -3,6 +3,7 @@ test legalizer ; Pre-SSE 4.1, we need to use runtime library calls for floating point rounding operations. set is_64bit set is_pic +set call_conv=system_v isa x86 function %floor(f32) -> f32 { @@ -11,6 +12,6 @@ ebb0(v0: f32): return v1 } ; check: function %floor(f32 [%xmm0]) -> f32 [%xmm0] fast { -; check: sig0 = (f32) -> f32 fast +; check: sig0 = (f32 [%xmm0]) -> f32 [%xmm0] system_v ; check: fn0 = %FloorF32 sig0 ; check: v1 = call fn0(v0) diff --git a/cranelift/filetests/wasm/f32-arith.cton b/cranelift/filetests/wasm/f32-arith.cton index d21316d293..7ce8eb4142 100644 --- a/cranelift/filetests/wasm/f32-arith.cton +++ b/cranelift/filetests/wasm/f32-arith.cton @@ -4,9 +4,15 @@ test compile set is_64bit=0 isa x86 haswell +set is_64bit=0 +isa x86 baseline + set is_64bit=1 isa x86 haswell +set is_64bit=1 +isa x86 baseline + ; Constants. function %f32_const() -> f32 { diff --git a/cranelift/filetests/wasm/f64-arith.cton b/cranelift/filetests/wasm/f64-arith.cton index 7b870ab2cb..150d9a0d8e 100644 --- a/cranelift/filetests/wasm/f64-arith.cton +++ b/cranelift/filetests/wasm/f64-arith.cton @@ -4,6 +4,9 @@ test compile set is_64bit=1 isa x86 haswell +set is_64bit=1 +isa x86 baseline + ; Constants. function %f64_const() -> f64 { diff --git a/cranelift/filetests/wasm/i32-arith.cton b/cranelift/filetests/wasm/i32-arith.cton index 7fb388054f..f6ee971c69 100644 --- a/cranelift/filetests/wasm/i32-arith.cton +++ b/cranelift/filetests/wasm/i32-arith.cton @@ -4,9 +4,15 @@ test compile set is_64bit=0 isa x86 haswell +set is_64bit=0 +isa x86 baseline + set is_64bit=1 isa x86 haswell +set is_64bit=1 +isa x86 baseline + ; Constants. function %i32_const() -> i32 { diff --git a/cranelift/filetests/wasm/i64-arith.cton b/cranelift/filetests/wasm/i64-arith.cton index 6cf6f2e4a2..52016039e0 100644 --- a/cranelift/filetests/wasm/i64-arith.cton +++ b/cranelift/filetests/wasm/i64-arith.cton @@ -4,6 +4,9 @@ test compile set is_64bit=1 isa x86 haswell +set is_64bit=1 +isa x86 baseline + ; Constants. function %i64_const() -> i64 { diff --git a/lib/codegen/src/legalizer/boundary.rs b/lib/codegen/src/legalizer/boundary.rs index 3c0a198f7f..4a865f2965 100644 --- a/lib/codegen/src/legalizer/boundary.rs +++ b/lib/codegen/src/legalizer/boundary.rs @@ -33,11 +33,9 @@ use std::vec::Vec; /// change the entry block arguments, calls, or return instructions, so this can leave the function /// in a state with type discrepancies. pub fn legalize_signatures(func: &mut Function, isa: &TargetIsa) { - isa.legalize_signature(&mut func.signature, true); - func.signature.compute_argument_bytes(); + legalize_signature(&mut func.signature, true, isa); for sig_data in func.dfg.signatures.values_mut() { - isa.legalize_signature(sig_data, false); - sig_data.compute_argument_bytes(); + legalize_signature(sig_data, false, isa); } if let Some(entry) = func.layout.entry_block() { @@ -46,6 +44,20 @@ pub fn legalize_signatures(func: &mut Function, isa: &TargetIsa) { } } +/// Legalize the libcall signature, which we may generate on the fly after +/// `legalize_signatures` has been called. +pub fn legalize_libcall_signature(signature: &mut Signature, isa: &TargetIsa) { + legalize_signature(signature, false, isa); +} + +/// Legalize the given signature. +/// +/// `current` is true if this is the signature for the current function. +fn legalize_signature(signature: &mut Signature, current: bool, isa: &TargetIsa) { + isa.legalize_signature(signature, current); + signature.compute_argument_bytes(); +} + /// Legalize the entry block parameters after `func`'s signature has been legalized. /// /// The legalized signature may contain more parameters than the original signature, and the diff --git a/lib/codegen/src/legalizer/libcall.rs b/lib/codegen/src/legalizer/libcall.rs index 70982da3cb..fb1ba6e313 100644 --- a/lib/codegen/src/legalizer/libcall.rs +++ b/lib/codegen/src/legalizer/libcall.rs @@ -3,6 +3,7 @@ use ir; use ir::{get_libcall_funcref, InstBuilder}; use isa::TargetIsa; +use legalizer::boundary::legalize_libcall_signature; use std::vec::Vec; /// Try to expand `inst` as a library call, returning true is successful. @@ -21,7 +22,10 @@ pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function, isa: &TargetIs let funcref = get_libcall_funcref(libcall, func, inst, isa); func.dfg.replace(inst).call(funcref, &args); - // TODO: ask the ISA to legalize the signature. + // Ask the ISA to legalize the signature. + let fn_data = &func.dfg.ext_funcs[funcref]; + let sig_data = &mut func.dfg.signatures[fn_data.signature]; + legalize_libcall_signature(sig_data, isa); true } From 7733ca1173bcacfa4aa6dfc8eb987bb08c3f98f8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 May 2018 19:24:23 -0700 Subject: [PATCH 1799/3084] Implement entity's `Iter` in terms of `std::iter::Enumerate`. Entity's `Iter` is essentially an `Enumerate`, so just use an `Enumerate` under the covers. This makes the code simpler. --- lib/entity/src/iter.rs | 53 +++++++++--------------------------------- 1 file changed, 11 insertions(+), 42 deletions(-) diff --git a/lib/entity/src/iter.rs b/lib/entity/src/iter.rs index 0455e40519..ea7bfb7199 100644 --- a/lib/entity/src/iter.rs +++ b/lib/entity/src/iter.rs @@ -1,6 +1,7 @@ //! A double-ended iterator over entity references and entities. use EntityRef; +use std::iter::Enumerate; use std::marker::PhantomData; use std::slice; @@ -9,9 +10,7 @@ pub struct Iter<'a, K: EntityRef, V> where V: 'a, { - pos: usize, - rev_pos: usize, - iter: slice::Iter<'a, V>, + enumerate: Enumerate>, unused: PhantomData, } @@ -20,9 +19,7 @@ impl<'a, K: EntityRef, V> Iter<'a, K, V> { /// of `iter`. pub fn new(iter: slice::Iter<'a, V>) -> Self { Self { - pos: 0, - rev_pos: iter.len(), - iter, + enumerate: iter.enumerate(), unused: PhantomData, } } @@ -32,29 +29,17 @@ impl<'a, K: EntityRef, V> Iterator for Iter<'a, K, V> { type Item = (K, &'a V); fn next(&mut self) -> Option { - if let Some(next) = self.iter.next() { - let pos = self.pos; - self.pos += 1; - Some((K::new(pos), next)) - } else { - None - } + self.enumerate.next().map(|(i, v)| (K::new(i), v)) } fn size_hint(&self) -> (usize, Option) { - self.iter.size_hint() + self.enumerate.size_hint() } } impl<'a, K: EntityRef, V> DoubleEndedIterator for Iter<'a, K, V> { fn next_back(&mut self) -> Option { - if let Some(next_back) = self.iter.next_back() { - let rev_pos = self.rev_pos - 1; - self.rev_pos -= 1; - Some((K::new(rev_pos), next_back)) - } else { - None - } + self.enumerate.next_back().map(|(i, v)| (K::new(i), v)) } } @@ -65,9 +50,7 @@ pub struct IterMut<'a, K: EntityRef, V> where V: 'a, { - pos: usize, - rev_pos: usize, - iter: slice::IterMut<'a, V>, + enumerate: Enumerate>, unused: PhantomData, } @@ -76,9 +59,7 @@ impl<'a, K: EntityRef, V> IterMut<'a, K, V> { /// of `iter`. pub fn new(iter: slice::IterMut<'a, V>) -> Self { Self { - pos: 0, - rev_pos: iter.len(), - iter, + enumerate: iter.enumerate(), unused: PhantomData, } } @@ -88,29 +69,17 @@ impl<'a, K: EntityRef, V> Iterator for IterMut<'a, K, V> { type Item = (K, &'a mut V); fn next(&mut self) -> Option { - if let Some(next) = self.iter.next() { - let pos = self.pos; - self.pos += 1; - Some((K::new(pos), next)) - } else { - None - } + self.enumerate.next().map(|(i, v)| (K::new(i), v)) } fn size_hint(&self) -> (usize, Option) { - self.iter.size_hint() + self.enumerate.size_hint() } } impl<'a, K: EntityRef, V> DoubleEndedIterator for IterMut<'a, K, V> { fn next_back(&mut self) -> Option { - if let Some(next_back) = self.iter.next_back() { - let rev_pos = self.rev_pos - 1; - self.rev_pos -= 1; - Some((K::new(rev_pos), next_back)) - } else { - None - } + self.enumerate.next_back().map(|(i, v)| (K::new(i), v)) } } From 7045c41418a5497ea616baa2c71b91d9bf2a945b Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 May 2018 19:26:13 -0700 Subject: [PATCH 1800/3084] Add a comment about implementing `Keys` in terms of `Range` in the future. --- lib/entity/src/keys.rs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/entity/src/keys.rs b/lib/entity/src/keys.rs index e9d1e91505..4301bbe689 100644 --- a/lib/entity/src/keys.rs +++ b/lib/entity/src/keys.rs @@ -1,4 +1,7 @@ //! A double-ended iterator over entity references. +//! +//! When `std::iter::Step` is stablized, `Keys` could be implemented as a wrapper around +//! `std::ops::Range`, but for now, we implment it manually. use EntityRef; use std::marker::PhantomData; From a1fe0f82e16982f38b24950bed8c29dd9a74204a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 29 May 2018 10:37:38 -0700 Subject: [PATCH 1801/3084] Update to the latest stable rustfmt, 0.4.2-stable (febbb36 2018-04-12). --- cranelift/src/cat.rs | 2 +- cranelift/src/compile.rs | 2 +- cranelift/src/cton-util.rs | 10 +++++++--- cranelift/src/print_cfg.rs | 2 +- cranelift/src/rsfilecheck.rs | 2 +- lib/codegen/src/abi.rs | 2 +- lib/codegen/src/cfg_printer.rs | 2 +- lib/codegen/src/divconst_magic_numbers.rs | 2 +- lib/codegen/src/ir/dfg.rs | 2 +- lib/codegen/src/ir/heap.rs | 2 +- lib/codegen/src/ir/stackslot.rs | 2 +- lib/codegen/src/isa/arm32/mod.rs | 2 +- lib/codegen/src/isa/arm64/mod.rs | 2 +- lib/codegen/src/isa/riscv/mod.rs | 4 ++-- lib/codegen/src/isa/stack.rs | 2 +- lib/codegen/src/isa/x86/mod.rs | 2 +- lib/codegen/src/lib.rs | 10 +++++++--- lib/codegen/src/preopt.rs | 4 ++-- lib/codegen/src/regalloc/coloring.rs | 4 ++-- lib/codegen/src/regalloc/solver.rs | 2 +- lib/codegen/src/stack_layout.rs | 2 +- lib/codegen/src/verifier/locations.rs | 2 +- lib/codegen/src/verifier/mod.rs | 2 +- lib/entity/src/iter.rs | 2 +- lib/entity/src/keys.rs | 2 +- lib/entity/src/lib.rs | 10 +++++++--- lib/entity/src/list.rs | 2 +- lib/faerie/src/lib.rs | 10 +++++++--- lib/filetests/src/lib.rs | 10 +++++++--- lib/filetests/src/runone.rs | 2 +- lib/filetests/src/test_domtree.rs | 2 +- lib/frontend/src/frontend.rs | 2 +- lib/frontend/src/lib.rs | 10 +++++++--- lib/frontend/src/ssa.rs | 2 +- lib/module/src/backend.rs | 8 ++++---- lib/module/src/lib.rs | 10 +++++++--- lib/module/src/module.rs | 2 +- lib/native/src/lib.rs | 10 +++++++--- lib/reader/src/lib.rs | 10 +++++++--- lib/reader/src/parser.rs | 2 +- lib/reader/src/testfile.rs | 2 +- lib/simplejit/src/lib.rs | 10 +++++++--- lib/umbrella/src/lib.rs | 10 +++++++--- lib/wasm/src/code_translator.rs | 2 +- lib/wasm/src/lib.rs | 12 ++++++++---- 45 files changed, 124 insertions(+), 76 deletions(-) diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs index a906214730..b668e0b51f 100644 --- a/cranelift/src/cat.rs +++ b/cranelift/src/cat.rs @@ -3,9 +3,9 @@ //! Read a sequence of Cretonne IR files and print them again to stdout. This has the effect of //! normalizing formatting and removing comments. -use CommandResult; use cretonne_reader::parse_functions; use utils::read_to_string; +use CommandResult; pub fn run(files: &[String]) -> CommandResult { for (i, f) in files.into_iter().enumerate() { diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 560c9f3497..37f70df364 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -1,10 +1,10 @@ //! CLI tool to read Cretonne IR files and compile them into native code. use capstone::prelude::*; -use cretonne_codegen::Context; use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::print_errors::pretty_error; use cretonne_codegen::settings::FlagsOrIsa; +use cretonne_codegen::Context; use cretonne_codegen::{binemit, ir}; use cretonne_reader::parse_test; use std::path::Path; diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 3723cbecda..d1805d6dc8 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -1,8 +1,12 @@ #![deny(trivial_numeric_casts)] #![warn(unused_import_braces, unstable_features, unused_extern_crates)] -#![cfg_attr(feature = "cargo-clippy", - warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, - option_map_unwrap_or_else, unicode_not_nfc, use_self))] +#![cfg_attr( + feature = "cargo-clippy", + warn( + float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, + unicode_not_nfc, use_self + ) +)] #[macro_use] extern crate cfg_if; diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index 28c8a81100..bf945da958 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -3,10 +3,10 @@ //! Read a series of Cretonne IR files and print their control flow graphs //! in graphviz format. -use CommandResult; use cretonne_codegen::cfg_printer::CFGPrinter; use cretonne_reader::parse_functions; use utils::read_to_string; +use CommandResult; pub fn run(files: &[String]) -> CommandResult { for (i, f) in files.into_iter().enumerate() { diff --git a/cranelift/src/rsfilecheck.rs b/cranelift/src/rsfilecheck.rs index b2b1c66ba8..48e25b3f12 100644 --- a/cranelift/src/rsfilecheck.rs +++ b/cranelift/src/rsfilecheck.rs @@ -2,10 +2,10 @@ //! //! This file is named to avoid a name collision with the filecheck crate. -use CommandResult; use filecheck::{Checker, CheckerBuilder, NO_VARIABLES}; use std::io::{self, Read}; use utils::read_to_string; +use CommandResult; pub fn run(files: &[String], verbose: bool) -> CommandResult { if files.is_empty() { diff --git a/lib/codegen/src/abi.rs b/lib/codegen/src/abi.rs index e4b4b223f4..fe69c67d35 100644 --- a/lib/codegen/src/abi.rs +++ b/lib/codegen/src/abi.rs @@ -184,8 +184,8 @@ pub fn legalize_abi_value(have: Type, arg: &AbiParam) -> ValueConversion { #[cfg(test)] mod tests { use super::*; - use ir::AbiParam; use ir::types; + use ir::AbiParam; #[test] fn legalize() { diff --git a/lib/codegen/src/cfg_printer.rs b/lib/codegen/src/cfg_printer.rs index 46b8754e3e..932345ef5b 100644 --- a/lib/codegen/src/cfg_printer.rs +++ b/lib/codegen/src/cfg_printer.rs @@ -3,8 +3,8 @@ use std::fmt::{Display, Formatter, Result, Write}; use flowgraph::ControlFlowGraph; -use ir::Function; use ir::instructions::BranchInfo; +use ir::Function; /// A utility for pretty-printing the CFG of a `Function`. pub struct CFGPrinter<'a> { diff --git a/lib/codegen/src/divconst_magic_numbers.rs b/lib/codegen/src/divconst_magic_numbers.rs index a56ed730d4..32a342703c 100644 --- a/lib/codegen/src/divconst_magic_numbers.rs +++ b/lib/codegen/src/divconst_magic_numbers.rs @@ -220,8 +220,8 @@ pub fn magicS64(d: i64) -> MS64 { #[cfg(test)] mod tests { - use super::{MS32, MS64, MU32, MU64}; use super::{magicS32, magicS64, magicU32, magicU64}; + use super::{MS32, MS64, MU32, MU64}; fn mkMU32(mulBy: u32, doAdd: bool, shiftBy: i32) -> MU32 { MU32 { diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs index 4fde255e82..21c3cc9b66 100644 --- a/lib/codegen/src/ir/dfg.rs +++ b/lib/codegen/src/ir/dfg.rs @@ -1151,8 +1151,8 @@ mod tests { #[test] fn aliases() { - use ir::InstBuilder; use ir::condcodes::IntCC; + use ir::InstBuilder; let mut func = Function::new(); let ebb0 = func.dfg.make_ebb(); diff --git a/lib/codegen/src/ir/heap.rs b/lib/codegen/src/ir/heap.rs index 01beb15b35..b41565b25b 100644 --- a/lib/codegen/src/ir/heap.rs +++ b/lib/codegen/src/ir/heap.rs @@ -1,7 +1,7 @@ //! Heaps. -use ir::GlobalVar; use ir::immediates::Imm64; +use ir::GlobalVar; use std::fmt; /// Information about a heap declaration. diff --git a/lib/codegen/src/ir/stackslot.rs b/lib/codegen/src/ir/stackslot.rs index aecf8abc8f..16794628df 100644 --- a/lib/codegen/src/ir/stackslot.rs +++ b/lib/codegen/src/ir/stackslot.rs @@ -338,8 +338,8 @@ impl StackSlots { #[cfg(test)] mod tests { use super::*; - use ir::Function; use ir::types; + use ir::Function; use std::string::ToString; #[test] diff --git a/lib/codegen/src/isa/arm32/mod.rs b/lib/codegen/src/isa/arm32/mod.rs index 227ba39fd9..560c71248c 100644 --- a/lib/codegen/src/isa/arm32/mod.rs +++ b/lib/codegen/src/isa/arm32/mod.rs @@ -9,8 +9,8 @@ pub mod settings; use super::super::settings as shared_settings; use binemit::{emit_function, CodeSink, MemoryCodeSink}; use ir; -use isa::Builder as IsaBuilder; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; +use isa::Builder as IsaBuilder; use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use regalloc; use std::boxed::Box; diff --git a/lib/codegen/src/isa/arm64/mod.rs b/lib/codegen/src/isa/arm64/mod.rs index 0d4f4a5397..240e92883d 100644 --- a/lib/codegen/src/isa/arm64/mod.rs +++ b/lib/codegen/src/isa/arm64/mod.rs @@ -9,8 +9,8 @@ pub mod settings; use super::super::settings as shared_settings; use binemit::{emit_function, CodeSink, MemoryCodeSink}; use ir; -use isa::Builder as IsaBuilder; use isa::enc_tables::{lookup_enclist, Encodings}; +use isa::Builder as IsaBuilder; use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use regalloc; use std::boxed::Box; diff --git a/lib/codegen/src/isa/riscv/mod.rs b/lib/codegen/src/isa/riscv/mod.rs index bcddc496d0..f704a5e7ff 100644 --- a/lib/codegen/src/isa/riscv/mod.rs +++ b/lib/codegen/src/isa/riscv/mod.rs @@ -9,8 +9,8 @@ pub mod settings; use super::super::settings as shared_settings; use binemit::{emit_function, CodeSink, MemoryCodeSink}; use ir; -use isa::Builder as IsaBuilder; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; +use isa::Builder as IsaBuilder; use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use regalloc; use std::boxed::Box; @@ -113,8 +113,8 @@ impl TargetIsa for Isa { #[cfg(test)] mod tests { - use ir::{Function, InstructionData, Opcode}; use ir::{immediates, types}; + use ir::{Function, InstructionData, Opcode}; use isa; use settings::{self, Configurable}; use std::string::{String, ToString}; diff --git a/lib/codegen/src/isa/stack.rs b/lib/codegen/src/isa/stack.rs index eefa523ae3..df61f77edc 100644 --- a/lib/codegen/src/isa/stack.rs +++ b/lib/codegen/src/isa/stack.rs @@ -4,8 +4,8 @@ //! defined in this module expresses the low-level details of accessing a stack slot from an //! encoded instruction. -use ir::StackSlot; use ir::stackslot::{StackOffset, StackSlotKind, StackSlots}; +use ir::StackSlot; /// A method for referencing a stack slot in the current stack frame. /// diff --git a/lib/codegen/src/isa/x86/mod.rs b/lib/codegen/src/isa/x86/mod.rs index 335d95cd22..e77b7d5c78 100644 --- a/lib/codegen/src/isa/x86/mod.rs +++ b/lib/codegen/src/isa/x86/mod.rs @@ -9,8 +9,8 @@ pub mod settings; use super::super::settings as shared_settings; use binemit::{emit_function, CodeSink, MemoryCodeSink}; use ir; -use isa::Builder as IsaBuilder; use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; +use isa::Builder as IsaBuilder; use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use regalloc; use result; diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs index 9636d317b0..05d781dcd7 100644 --- a/lib/codegen/src/lib.rs +++ b/lib/codegen/src/lib.rs @@ -31,9 +31,13 @@ redundant_field_names, useless_let_if_seq, len_without_is_empty))] -#![cfg_attr(feature = "cargo-clippy", - warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, - option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] +#![cfg_attr( + feature = "cargo-clippy", + warn( + float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, + print_stdout, unicode_not_nfc, use_self + ) +)] // Turns on no_std and alloc features if std is not available. #![cfg_attr(not(feature = "std"), no_std)] #![cfg_attr(not(feature = "std"), feature(alloc))] diff --git a/lib/codegen/src/preopt.rs b/lib/codegen/src/preopt.rs index ead49b9486..b96c3d9531 100644 --- a/lib/codegen/src/preopt.rs +++ b/lib/codegen/src/preopt.rs @@ -3,12 +3,12 @@ #![allow(non_snake_case)] use cursor::{Cursor, FuncCursor}; -use divconst_magic_numbers::{MS32, MS64, MU32, MU64}; use divconst_magic_numbers::{magicS32, magicS64, magicU32, magicU64}; -use ir::Inst; +use divconst_magic_numbers::{MS32, MS64, MU32, MU64}; use ir::dfg::ValueDef; use ir::instructions::Opcode; use ir::types::{I32, I64}; +use ir::Inst; use ir::{DataFlowGraph, Function, InstBuilder, InstructionData, Type, Value}; use timing; diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs index aa7cbbeb47..33ce0ff3b4 100644 --- a/lib/codegen/src/regalloc/coloring.rs +++ b/lib/codegen/src/regalloc/coloring.rs @@ -46,16 +46,16 @@ use cursor::{Cursor, EncCursor}; use dominator_tree::DominatorTree; use ir::{AbiParam, ArgumentLoc, InstBuilder, ValueDef}; use ir::{Ebb, Function, Inst, Layout, SigRef, Value, ValueLoc}; -use isa::{ConstraintKind, EncInfo, OperandConstraint, RecipeConstraints, TargetIsa}; use isa::{regs_overlap, RegClass, RegInfo, RegUnit}; +use isa::{ConstraintKind, EncInfo, OperandConstraint, RecipeConstraints, TargetIsa}; use packed_option::PackedOption; -use regalloc::RegDiversions; use regalloc::affinity::Affinity; use regalloc::live_value_tracker::{LiveValue, LiveValueTracker}; use regalloc::liveness::Liveness; use regalloc::liverange::{LiveRange, LiveRangeContext}; use regalloc::register_set::RegisterSet; use regalloc::solver::{Solver, SolverError}; +use regalloc::RegDiversions; use std::mem; use timing; diff --git a/lib/codegen/src/regalloc/solver.rs b/lib/codegen/src/regalloc/solver.rs index 33217eab86..af935b2104 100644 --- a/lib/codegen/src/regalloc/solver.rs +++ b/lib/codegen/src/regalloc/solver.rs @@ -1367,7 +1367,7 @@ mod tests { mov(v15, gpr, r5, r3), mov(v14, gpr, r4, r5), mov(v13, gpr, r1, r4), - fill(v10, gpr, 0, r1) // Finally complete cycle 1. + fill(v10, gpr, 0, r1), // Finally complete cycle 1. ] ); } diff --git a/lib/codegen/src/stack_layout.rs b/lib/codegen/src/stack_layout.rs index e40499f976..cda2251900 100644 --- a/lib/codegen/src/stack_layout.rs +++ b/lib/codegen/src/stack_layout.rs @@ -1,7 +1,7 @@ //! Computing stack layout. -use ir::StackSlots; use ir::stackslot::{StackOffset, StackSize, StackSlotKind}; +use ir::StackSlots; use result::CtonError; use std::cmp::{max, min}; diff --git a/lib/codegen/src/verifier/locations.rs b/lib/codegen/src/verifier/locations.rs index 89e2b09e9a..5460b938dd 100644 --- a/lib/codegen/src/verifier/locations.rs +++ b/lib/codegen/src/verifier/locations.rs @@ -2,8 +2,8 @@ use ir; use isa; -use regalloc::RegDiversions; use regalloc::liveness::Liveness; +use regalloc::RegDiversions; use timing; use verifier::Result; diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs index f8644034b2..20507594e3 100644 --- a/lib/codegen/src/verifier/mod.rs +++ b/lib/codegen/src/verifier/mod.rs @@ -1147,8 +1147,8 @@ impl<'a> Verifier<'a> { mod tests { use super::{Error, Verifier}; use entity::EntityList; - use ir::Function; use ir::instructions::{InstructionData, Opcode}; + use ir::Function; use settings; macro_rules! assert_err_with_msg { diff --git a/lib/entity/src/iter.rs b/lib/entity/src/iter.rs index ea7bfb7199..f818bb0cf5 100644 --- a/lib/entity/src/iter.rs +++ b/lib/entity/src/iter.rs @@ -1,9 +1,9 @@ //! A double-ended iterator over entity references and entities. -use EntityRef; use std::iter::Enumerate; use std::marker::PhantomData; use std::slice; +use EntityRef; /// Iterate over all keys in order. pub struct Iter<'a, K: EntityRef, V> diff --git a/lib/entity/src/keys.rs b/lib/entity/src/keys.rs index 4301bbe689..5014dc3be5 100644 --- a/lib/entity/src/keys.rs +++ b/lib/entity/src/keys.rs @@ -3,8 +3,8 @@ //! When `std::iter::Step` is stablized, `Keys` could be implemented as a wrapper around //! `std::ops::Range`, but for now, we implment it manually. -use EntityRef; use std::marker::PhantomData; +use EntityRef; /// Iterate over all keys in order. pub struct Keys { diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs index 6b27064c7d..cb1c248db1 100644 --- a/lib/entity/src/lib.rs +++ b/lib/entity/src/lib.rs @@ -34,9 +34,13 @@ #![cfg_attr(feature = "std", warn(unstable_features))] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature = "cargo-clippy", - warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, - option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] +#![cfg_attr( + feature = "cargo-clippy", + warn( + float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, + print_stdout, unicode_not_nfc, use_self + ) +)] // Turns on no_std and alloc features if std is not available. #![cfg_attr(not(feature = "std"), no_std)] #![cfg_attr(not(feature = "std"), feature(alloc))] diff --git a/lib/entity/src/list.rs b/lib/entity/src/list.rs index 803504bc76..8a0aecbdfd 100644 --- a/lib/entity/src/list.rs +++ b/lib/entity/src/list.rs @@ -1,9 +1,9 @@ //! Small lists of entity references. -use EntityRef; use std::hash::{Hash, Hasher}; use std::marker::PhantomData; use std::mem; use std::vec::Vec; +use EntityRef; /// A small list of entity references allocated from a pool. /// diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs index 6cec1c9e52..7d7bb27a0a 100644 --- a/lib/faerie/src/lib.rs +++ b/lib/faerie/src/lib.rs @@ -6,9 +6,13 @@ #![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature = "cargo-clippy", - warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, - option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] +#![cfg_attr( + feature = "cargo-clippy", + warn( + float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, + print_stdout, unicode_not_nfc, use_self + ) +)] extern crate cretonne_codegen; extern crate cretonne_module; diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs index e0dca34cc2..7dd87302a5 100644 --- a/lib/filetests/src/lib.rs +++ b/lib/filetests/src/lib.rs @@ -9,9 +9,13 @@ type_complexity, // Rustfmt 0.9.0 is at odds with this lint: block_in_if_condition_stmt))] -#![cfg_attr(feature = "cargo-clippy", - warn(mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, - unicode_not_nfc, use_self))] +#![cfg_attr( + feature = "cargo-clippy", + warn( + mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, unicode_not_nfc, + use_self + ) +)] #[macro_use(dbg)] extern crate cretonne_codegen; diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs index 7688785a57..4d3731619b 100644 --- a/lib/filetests/src/runone.rs +++ b/lib/filetests/src/runone.rs @@ -6,8 +6,8 @@ use cretonne_codegen::print_errors::pretty_verifier_error; use cretonne_codegen::settings::Flags; use cretonne_codegen::timing; use cretonne_codegen::verify_function; -use cretonne_reader::IsaSpec; use cretonne_reader::parse_test; +use cretonne_reader::IsaSpec; use std::borrow::Cow; use std::fs; use std::io::{self, Read}; diff --git a/lib/filetests/src/test_domtree.rs b/lib/filetests/src/test_domtree.rs index cdc2a37ae4..9dbd5fe849 100644 --- a/lib/filetests/src/test_domtree.rs +++ b/lib/filetests/src/test_domtree.rs @@ -14,8 +14,8 @@ use cretonne_codegen::dominator_tree::{DominatorTree, DominatorTreePreorder}; use cretonne_codegen::flowgraph::ControlFlowGraph; -use cretonne_codegen::ir::Function; use cretonne_codegen::ir::entities::AnyEntity; +use cretonne_codegen::ir::Function; use cretonne_reader::TestCommand; use match_directive::match_directive; use std::borrow::{Borrow, Cow}; diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 2688a1055a..453fc09c0e 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -607,7 +607,6 @@ where #[cfg(test)] mod tests { - use Variable; use cretonne_codegen::entity::EntityRef; use cretonne_codegen::ir::types::*; use cretonne_codegen::ir::{AbiParam, ExternalName, Function, InstBuilder, Signature}; @@ -615,6 +614,7 @@ mod tests { use cretonne_codegen::settings::CallConv; use cretonne_codegen::verifier::verify_function; use frontend::{FunctionBuilder, FunctionBuilderContext}; + use Variable; fn sample_function(lazy_seal: bool) { let mut sig = Signature::new(CallConv::SystemV); diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index ad0760dc25..8b25becb33 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -131,9 +131,13 @@ #![warn(unused_import_braces)] #![cfg_attr(feature = "std", warn(unstable_features))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default))] -#![cfg_attr(feature = "cargo-clippy", - warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, - option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] +#![cfg_attr( + feature = "cargo-clippy", + warn( + float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, + print_stdout, unicode_not_nfc, use_self + ) +)] #![cfg_attr(not(feature = "std"), no_std)] #![cfg_attr(not(feature = "std"), feature(alloc))] diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 578264ef3a..59a031f6d1 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -712,7 +712,6 @@ where #[cfg(test)] mod tests { - use Variable; use cretonne_codegen::cursor::{Cursor, FuncCursor}; use cretonne_codegen::entity::EntityRef; use cretonne_codegen::ir::instructions::BranchInfo; @@ -721,6 +720,7 @@ mod tests { use cretonne_codegen::settings; use cretonne_codegen::verify_function; use ssa::SSABuilder; + use Variable; #[test] fn simple_block() { diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs index 82b7e3295a..15258544d8 100644 --- a/lib/module/src/backend.rs +++ b/lib/module/src/backend.rs @@ -1,13 +1,13 @@ //! Defines the `Backend` trait. +use cretonne_codegen::isa::TargetIsa; +use cretonne_codegen::Context; +use cretonne_codegen::{binemit, ir}; +use std::marker; use DataContext; use Linkage; use ModuleError; use ModuleNamespace; -use cretonne_codegen::Context; -use cretonne_codegen::isa::TargetIsa; -use cretonne_codegen::{binemit, ir}; -use std::marker; /// A `Backend` implements the functionality needed to support a `Module`. pub trait Backend diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs index d7f9118288..5c1091dd3f 100644 --- a/lib/module/src/lib.rs +++ b/lib/module/src/lib.rs @@ -4,9 +4,13 @@ #![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature = "cargo-clippy", - warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, - option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] +#![cfg_attr( + feature = "cargo-clippy", + warn( + float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, + print_stdout, unicode_not_nfc, use_self + ) +)] #[macro_use] extern crate cretonne_codegen; diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index d9a7c7fda9..6b696d2b33 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -5,12 +5,12 @@ // TODO: Factor out `ir::Function`'s `ext_funcs` and `global_vars` into a struct // shared with `DataContext`? -use Backend; use cretonne_codegen::entity::{EntityRef, PrimaryMap}; use cretonne_codegen::result::CtonError; use cretonne_codegen::{binemit, ir, Context}; use data_context::DataContext; use std::collections::HashMap; +use Backend; /// A function identifier for use in the `Module` interface. #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index 54fa134503..f9f6d4cc7a 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -5,9 +5,13 @@ #![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature = "cargo-clippy", - warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, - option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] +#![cfg_attr( + feature = "cargo-clippy", + warn( + float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, + print_stdout, unicode_not_nfc, use_self + ) +)] #![cfg_attr(not(feature = "std"), no_std)] extern crate cretonne_codegen; diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index b623035249..687abd1b85 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -7,9 +7,13 @@ #![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature = "cargo-clippy", - warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, - option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] +#![cfg_attr( + feature = "cargo-clippy", + warn( + float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, + print_stdout, unicode_not_nfc, use_self + ) +)] extern crate cretonne_codegen; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 8fadf88566..7c013cd857 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -2237,9 +2237,9 @@ impl<'a> Parser<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne_codegen::ir::StackSlotKind; use cretonne_codegen::ir::entities::AnyEntity; use cretonne_codegen::ir::types; + use cretonne_codegen::ir::StackSlotKind; use cretonne_codegen::ir::{ArgumentExtension, ArgumentPurpose}; use cretonne_codegen::settings::CallConv; use error::Error; diff --git a/lib/reader/src/testfile.rs b/lib/reader/src/testfile.rs index 2fae3c2d3d..0c591cea26 100644 --- a/lib/reader/src/testfile.rs +++ b/lib/reader/src/testfile.rs @@ -4,8 +4,8 @@ //! file-based test case. //! -use cretonne_codegen::ir::Function; use cretonne_codegen::ir::entities::AnyEntity; +use cretonne_codegen::ir::Function; use error::Location; use isaspec::IsaSpec; use sourcemap::SourceMap; diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs index 90ee03d9d5..3212447065 100644 --- a/lib/simplejit/src/lib.rs +++ b/lib/simplejit/src/lib.rs @@ -4,9 +4,13 @@ #![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature = "cargo-clippy", - warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, - option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] +#![cfg_attr( + feature = "cargo-clippy", + warn( + float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, + print_stdout, unicode_not_nfc, use_self + ) +)] extern crate cretonne_codegen; extern crate cretonne_module; diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs index ebfb0c731e..11b773b7f5 100644 --- a/lib/umbrella/src/lib.rs +++ b/lib/umbrella/src/lib.rs @@ -4,9 +4,13 @@ #![warn(unused_import_braces, unstable_features)] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature = "cargo-clippy", - warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, - option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] +#![cfg_attr( + feature = "cargo-clippy", + warn( + float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, + print_stdout, unicode_not_nfc, use_self + ) +)] /// Provide these crates, renamed to reduce stutter. pub extern crate cretonne_codegen as codegen; diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 60eaf8f853..7aae3d6678 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -32,8 +32,8 @@ use state::{ControlStackFrame, TranslationState}; use std::collections::{hash_map, HashMap}; use std::vec::Vec; use std::{i32, u32}; +use translation_utils::{f32_translation, f64_translation, num_return_values, type_to_type}; use translation_utils::{FunctionIndex, MemoryIndex, SignatureIndex, TableIndex}; -use translation_utils::{num_return_values, type_to_type, f32_translation, f64_translation}; use wasmparser::{MemoryImmediate, Operator}; // Clippy warns about "flags: _" but its important to document that the flags field is ignored diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 9842f36852..2c73f73953 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -14,9 +14,13 @@ #![cfg_attr(feature = "std", warn(unstable_features))] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] -#![cfg_attr(feature = "cargo-clippy", - warn(float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, - option_map_unwrap_or_else, print_stdout, unicode_not_nfc, use_self))] +#![cfg_attr( + feature = "cargo-clippy", + warn( + float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, + print_stdout, unicode_not_nfc, use_self + ) +)] #![cfg_attr(not(feature = "std"), no_std)] #![cfg_attr(not(feature = "std"), feature(alloc))] @@ -57,7 +61,7 @@ mod std { pub use alloc::vec; pub use core::fmt; pub use core::option; - pub use core::{cmp, str, i32, u32}; + pub use core::{cmp, i32, str, u32}; pub mod collections { pub use hashmap_core::{map as hash_map, HashMap}; } From 2f3008aa40316f48ba75ab138d4467959557a07c Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Tue, 29 May 2018 18:12:18 -0400 Subject: [PATCH 1802/3084] Use "nightly" feature of raw-cpuid when possible. --- lib/native/Cargo.toml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index c5845b3444..17e2578339 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -16,7 +16,10 @@ raw-cpuid = "3.1.0" [features] default = ["std"] std = ["cretonne-codegen/std"] -core = ["cretonne-codegen/core"] +# when compiling with the "core" feature, nightly must be enabled +# enabling the "nightly" feature for raw-cpuid allows avoiding +# linking in a c-library. +core = ["cretonne-codegen/core", "raw-cpuid/nightly"] [badges] maintenance = { status = "experimental" } From 4e67e08efdf276e9a77a49d18de0353aee17ecd3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 25 May 2018 11:41:14 -0700 Subject: [PATCH 1803/3084] Use the target-lexicon crate. This switches from a custom list of architectures to use the target-lexicon crate. - "set is_64bit=1; isa x86" is replaced with "target x86_64", and similar for other architectures, and the `is_64bit` flag is removed entirely. - The `is_compressed` flag is removed too; it's no longer being used to control REX prefixes on x86-64, ARM and Thumb are separate architectures in target-lexicon, and we can figure out how to select RISC-V compressed encodings when we're ready. --- cranelift/Cargo.toml | 1 + cranelift/docs/testing.rst | 10 +- cranelift/filetests/isa/riscv/abi-e.cton | 2 +- cranelift/filetests/isa/riscv/abi.cton | 2 +- cranelift/filetests/isa/riscv/binary32.cton | 2 +- cranelift/filetests/isa/riscv/encoding.cton | 2 +- cranelift/filetests/isa/riscv/expand-i32.cton | 6 +- .../filetests/isa/riscv/legalize-abi.cton | 2 +- .../filetests/isa/riscv/legalize-i64.cton | 2 +- .../filetests/isa/riscv/parse-encoding.cton | 2 +- cranelift/filetests/isa/riscv/regmove.cton | 2 +- cranelift/filetests/isa/riscv/split-args.cton | 2 +- .../filetests/isa/riscv/verify-encoding.cton | 2 +- cranelift/filetests/isa/x86/abcd.cton | 2 +- cranelift/filetests/isa/x86/abi-bool.cton | 3 +- cranelift/filetests/isa/x86/abi32.cton | 2 +- cranelift/filetests/isa/x86/abi64.cton | 3 +- .../isa/x86/allones_funcaddrs32.cton | 2 +- .../isa/x86/allones_funcaddrs64.cton | 3 +- .../isa/x86/baseline_clz_ctz_popcount.cton | 4 +- .../baseline_clz_ctz_popcount_encoding.cton | 3 +- .../filetests/isa/x86/binary32-float.cton | 2 +- cranelift/filetests/isa/x86/binary32.cton | 2 +- .../filetests/isa/x86/binary64-float.cton | 3 +- cranelift/filetests/isa/x86/binary64-pic.cton | 3 +- cranelift/filetests/isa/x86/binary64.cton | 3 +- .../filetests/isa/x86/legalize-call.cton | 3 +- .../filetests/isa/x86/legalize-custom.cton | 5 +- .../filetests/isa/x86/legalize-div-traps.cton | 3 +- cranelift/filetests/isa/x86/legalize-div.cton | 3 +- .../filetests/isa/x86/legalize-libcall.cton | 3 +- .../filetests/isa/x86/legalize-memory.cton | 3 +- .../filetests/isa/x86/legalize-mulhi.cton | 4 +- .../x86/optimized-zero-constants-32bit.cton | 3 +- .../isa/x86/optimized-zero-constants.cton | 3 +- .../isa/x86/probestack-adjusts-sp.cton | 3 +- .../isa/x86/probestack-disabled.cton | 3 +- .../isa/x86/probestack-noncolocated.cton | 3 +- .../filetests/isa/x86/probestack-size.cton | 3 +- cranelift/filetests/isa/x86/probestack.cton | 3 +- .../filetests/isa/x86/prologue-epilogue.cton | 3 +- cranelift/filetests/isa/x86/shrink.cton | 3 +- .../isa/x86/windows_fastcall_x64.cton | 3 +- .../parser/instruction_encoding.cton | 2 +- cranelift/filetests/postopt/basic.cton | 2 +- .../filetests/postopt/complex_memory_ops.cton | 3 +- .../preopt/div_by_const_indirect.cton | 3 +- .../preopt/div_by_const_non_power_of_2.cton | 3 +- .../preopt/div_by_const_power_of_2.cton | 3 +- .../preopt/rem_by_const_non_power_of_2.cton | 3 +- .../preopt/rem_by_const_power_of_2.cton | 3 +- cranelift/filetests/preopt/simplify.cton | 2 +- cranelift/filetests/regalloc/aliases.cton | 3 +- cranelift/filetests/regalloc/basic.cton | 2 +- cranelift/filetests/regalloc/coalesce.cton | 2 +- .../filetests/regalloc/coalescing-207.cton | 3 +- .../filetests/regalloc/coalescing-216.cton | 3 +- .../filetests/regalloc/coloring-227.cton | 3 +- cranelift/filetests/regalloc/constraints.cton | 2 +- cranelift/filetests/regalloc/ghost-param.cton | 3 +- .../regalloc/global-constraints.cton | 2 +- .../filetests/regalloc/global-fixed.cton | 3 +- .../regalloc/infinite-interference.cton | 2 +- cranelift/filetests/regalloc/iterate.cton | 3 +- .../filetests/regalloc/multi-constraints.cton | 3 +- .../regalloc/output-interference.cton | 3 +- cranelift/filetests/regalloc/reload-208.cton | 3 +- cranelift/filetests/regalloc/reload.cton | 2 +- .../filetests/regalloc/schedule-moves.cton | 2 +- .../filetests/regalloc/spill-noregs.cton | 3 +- cranelift/filetests/regalloc/spill.cton | 2 +- .../filetests/regalloc/unreachable_code.cton | 3 +- cranelift/filetests/regalloc/x86-regres.cton | 2 +- cranelift/filetests/verifier/flags.cton | 2 +- cranelift/filetests/wasm/control.cton | 6 +- cranelift/filetests/wasm/conversions.cton | 3 +- cranelift/filetests/wasm/f32-arith.cton | 15 +-- cranelift/filetests/wasm/f32-compares.cton | 6 +- cranelift/filetests/wasm/f32-memory64.cton | 3 +- cranelift/filetests/wasm/f64-arith.cton | 7 +- cranelift/filetests/wasm/f64-compares.cton | 6 +- cranelift/filetests/wasm/f64-memory64.cton | 3 +- cranelift/filetests/wasm/i32-arith.cton | 15 +-- cranelift/filetests/wasm/i32-compares.cton | 6 +- cranelift/filetests/wasm/i32-memory64.cton | 3 +- cranelift/filetests/wasm/i64-arith.cton | 7 +- cranelift/filetests/wasm/i64-compares.cton | 3 +- cranelift/filetests/wasm/i64-memory64.cton | 3 +- cranelift/filetests/wasm/select.cton | 6 +- cranelift/fuzz/Cargo.toml | 3 + cranelift/fuzz/fuzz_translate_module.rs | 5 +- cranelift/src/compile.rs | 42 +++--- cranelift/src/cton-util.rs | 14 +- cranelift/src/utils.rs | 30 +++-- cranelift/src/wasm.rs | 35 +++-- lib/codegen/Cargo.toml | 1 + lib/codegen/meta/base/settings.py | 4 - lib/codegen/src/isa/arm32/mod.rs | 25 +++- lib/codegen/src/isa/arm64/mod.rs | 11 +- lib/codegen/src/isa/mod.rs | 61 ++++++--- lib/codegen/src/isa/riscv/abi.rs | 26 ++-- lib/codegen/src/isa/riscv/mod.rs | 42 +++--- lib/codegen/src/isa/x86/abi.rs | 124 +++++++++--------- lib/codegen/src/isa/x86/mod.rs | 23 +++- lib/codegen/src/legalizer/call.rs | 6 +- lib/codegen/src/legalizer/mod.rs | 6 +- lib/codegen/src/lib.rs | 5 + lib/codegen/src/regalloc/pressure.rs | 6 +- lib/codegen/src/regalloc/solver.rs | 6 +- lib/codegen/src/settings.rs | 2 - lib/faerie/Cargo.toml | 3 +- lib/faerie/src/backend.rs | 28 ++-- lib/faerie/src/container.rs | 8 +- lib/faerie/src/lib.rs | 2 +- lib/faerie/src/target.rs | 21 --- lib/module/src/module.rs | 6 +- lib/native/Cargo.toml | 1 + lib/native/src/lib.rs | 23 +--- lib/reader/Cargo.toml | 1 + lib/reader/src/error.rs | 2 +- lib/reader/src/lib.rs | 1 + lib/reader/src/parser.rs | 61 +++++---- lib/simplejit/Cargo.toml | 1 + lib/simplejit/src/backend.rs | 9 +- lib/simplejit/src/lib.rs | 1 + lib/wasm/Cargo.toml | 1 + lib/wasm/src/environ/dummy.rs | 19 ++- lib/wasm/src/environ/spec.rs | 10 +- lib/wasm/src/func_translator.rs | 7 +- lib/wasm/src/lib.rs | 1 + lib/wasm/tests/wasm_testsuite.rs | 5 +- 131 files changed, 487 insertions(+), 499 deletions(-) delete mode 100644 lib/faerie/src/target.rs diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 993c96166d..df16b8cccb 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -31,6 +31,7 @@ serde_derive = "1.0.8" term = "0.5.1" capstone = "0.3.1" wabt = { version = "0.3", optional = true } +target-lexicon = "0.0.0" [features] default = ["wasm"] diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index 0265419305..02071f872d 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -125,15 +125,15 @@ The ``set`` lines apply settings cumulatively:: test legalizer set opt_level=best - set is_64bit=1 - isa riscv - set is_64bit=0 - isa riscv supports_m=false + set is_pic=1 + isa riscv64 + set is_pic=0 + isa riscv32 supports_m=false function %foo() {} This example will run the legalizer test twice. Both runs will have -``opt_level=best``, but they will have different ``is_64bit`` settings. The 32-bit +``opt_level=best``, but they will have different ``is_pic`` settings. The 32-bit run will also have the RISC-V specific flag ``supports_m`` disabled. The filetests are run automatically as part of `cargo test`, and they can diff --git a/cranelift/filetests/isa/riscv/abi-e.cton b/cranelift/filetests/isa/riscv/abi-e.cton index 50100ebf4e..80b275506c 100644 --- a/cranelift/filetests/isa/riscv/abi-e.cton +++ b/cranelift/filetests/isa/riscv/abi-e.cton @@ -1,6 +1,6 @@ ; Test the legalization of function signatures for RV32E. test legalizer -isa riscv enable_e +target riscv32 enable_e ; regex: V=v\d+ diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.cton index 12371ac537..b3c9513aef 100644 --- a/cranelift/filetests/isa/riscv/abi.cton +++ b/cranelift/filetests/isa/riscv/abi.cton @@ -1,6 +1,6 @@ ; Test the legalization of function signatures. test legalizer -isa riscv +target riscv32 ; regex: V=v\d+ diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.cton index 7837672bcc..4d63084f9a 100644 --- a/cranelift/filetests/isa/riscv/binary32.cton +++ b/cranelift/filetests/isa/riscv/binary32.cton @@ -1,6 +1,6 @@ ; Binary emission of 32-bit code. test binemit -isa riscv +target riscv32 function %RV32I(i32 link [%x1]) -> i32 link [%x1] { sig0 = () diff --git a/cranelift/filetests/isa/riscv/encoding.cton b/cranelift/filetests/isa/riscv/encoding.cton index 9dc6f388be..98b5f66db6 100644 --- a/cranelift/filetests/isa/riscv/encoding.cton +++ b/cranelift/filetests/isa/riscv/encoding.cton @@ -1,5 +1,5 @@ test legalizer -isa riscv supports_m=1 +target riscv32 supports_m=1 function %int32(i32, i32) { ebb0(v1: i32, v2: i32): diff --git a/cranelift/filetests/isa/riscv/expand-i32.cton b/cranelift/filetests/isa/riscv/expand-i32.cton index 30048622ca..eb63d7cdcd 100644 --- a/cranelift/filetests/isa/riscv/expand-i32.cton +++ b/cranelift/filetests/isa/riscv/expand-i32.cton @@ -1,11 +1,9 @@ ; Test the legalization of i32 instructions that don't have RISC-V versions. test legalizer -set is_64bit=0 -isa riscv supports_m=1 +target riscv32 supports_m=1 -set is_64bit=1 -isa riscv supports_m=1 +target riscv64 supports_m=1 ; regex: V=v\d+ diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.cton index df60b91f81..88c5989db6 100644 --- a/cranelift/filetests/isa/riscv/legalize-abi.cton +++ b/cranelift/filetests/isa/riscv/legalize-abi.cton @@ -1,6 +1,6 @@ ; Test legalizer's handling of ABI boundaries. test legalizer -isa riscv +target riscv32 ; regex: V=v\d+ ; regex: SS=ss\d+ diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.cton index f06cf0803a..d043337a21 100644 --- a/cranelift/filetests/isa/riscv/legalize-i64.cton +++ b/cranelift/filetests/isa/riscv/legalize-i64.cton @@ -1,6 +1,6 @@ ; Test the legalization of i64 arithmetic instructions. test legalizer -isa riscv supports_m=1 +target riscv32 supports_m=1 ; regex: V=v\d+ diff --git a/cranelift/filetests/isa/riscv/parse-encoding.cton b/cranelift/filetests/isa/riscv/parse-encoding.cton index e61f43ef84..f79e552e6d 100644 --- a/cranelift/filetests/isa/riscv/parse-encoding.cton +++ b/cranelift/filetests/isa/riscv/parse-encoding.cton @@ -1,6 +1,6 @@ ; Test the parser's support for encoding annotations. test legalizer -isa riscv +target riscv32 function %parse_encoding(i32 [%x5]) -> i32 [%x10] { ; check: function %parse_encoding(i32 [%x5], i32 link [%x1]) -> i32 [%x10], i32 link [%x1] fast { diff --git a/cranelift/filetests/isa/riscv/regmove.cton b/cranelift/filetests/isa/riscv/regmove.cton index c316f74f21..6ec17ef813 100644 --- a/cranelift/filetests/isa/riscv/regmove.cton +++ b/cranelift/filetests/isa/riscv/regmove.cton @@ -1,6 +1,6 @@ ; Test tracking of register moves. test binemit -isa riscv +target riscv32 function %regmoves(i32 link [%x1]) -> i32 link [%x1] { ebb0(v9999: i32): diff --git a/cranelift/filetests/isa/riscv/split-args.cton b/cranelift/filetests/isa/riscv/split-args.cton index 80de18fd7a..dd605de81a 100644 --- a/cranelift/filetests/isa/riscv/split-args.cton +++ b/cranelift/filetests/isa/riscv/split-args.cton @@ -1,6 +1,6 @@ ; Test the legalization of EBB arguments that are split. test legalizer -isa riscv +target riscv32 ; regex: V=v\d+ diff --git a/cranelift/filetests/isa/riscv/verify-encoding.cton b/cranelift/filetests/isa/riscv/verify-encoding.cton index 1aaedf1c89..99ed4697bb 100644 --- a/cranelift/filetests/isa/riscv/verify-encoding.cton +++ b/cranelift/filetests/isa/riscv/verify-encoding.cton @@ -1,5 +1,5 @@ test verifier -isa riscv +target riscv32 function %RV32I(i32 link [%x1]) -> i32 link [%x1] { fn0 = %foo() diff --git a/cranelift/filetests/isa/x86/abcd.cton b/cranelift/filetests/isa/x86/abcd.cton index f0007d795d..67acac970b 100644 --- a/cranelift/filetests/isa/x86/abcd.cton +++ b/cranelift/filetests/isa/x86/abcd.cton @@ -1,5 +1,5 @@ test regalloc -isa x86 +target i686 ; %rdi can't be used in a movsbl instruction, so test that the register ; allocator can move it to a register that can be. diff --git a/cranelift/filetests/isa/x86/abi-bool.cton b/cranelift/filetests/isa/x86/abi-bool.cton index 48ac976a84..fdf21ba055 100644 --- a/cranelift/filetests/isa/x86/abi-bool.cton +++ b/cranelift/filetests/isa/x86/abi-bool.cton @@ -1,6 +1,5 @@ test compile -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %foo(i64, i64, i64, i32) -> b1 system_v { ebb3(v0: i64, v1: i64, v2: i64, v3: i32): diff --git a/cranelift/filetests/isa/x86/abi32.cton b/cranelift/filetests/isa/x86/abi32.cton index 04b8d2ec6f..4b9f5fbcd1 100644 --- a/cranelift/filetests/isa/x86/abi32.cton +++ b/cranelift/filetests/isa/x86/abi32.cton @@ -1,6 +1,6 @@ ; Test the legalization of function signatures. test legalizer -isa x86 +target i686 ; regex: V=v\d+ diff --git a/cranelift/filetests/isa/x86/abi64.cton b/cranelift/filetests/isa/x86/abi64.cton index 121e5cc1a8..209b036ee2 100644 --- a/cranelift/filetests/isa/x86/abi64.cton +++ b/cranelift/filetests/isa/x86/abi64.cton @@ -1,7 +1,6 @@ ; Test the legalization of function signatures. test legalizer -set is_64bit -isa x86 +target x86_64 ; regex: V=v\d+ diff --git a/cranelift/filetests/isa/x86/allones_funcaddrs32.cton b/cranelift/filetests/isa/x86/allones_funcaddrs32.cton index cf0036f4d0..2e3dbf50f3 100644 --- a/cranelift/filetests/isa/x86/allones_funcaddrs32.cton +++ b/cranelift/filetests/isa/x86/allones_funcaddrs32.cton @@ -2,7 +2,7 @@ test binemit set opt_level=best set allones_funcaddrs -isa x86 haswell +target i686 haswell ; The binary encodings can be verified with the command: ; diff --git a/cranelift/filetests/isa/x86/allones_funcaddrs64.cton b/cranelift/filetests/isa/x86/allones_funcaddrs64.cton index 3f47c78052..6edbbcf4d2 100644 --- a/cranelift/filetests/isa/x86/allones_funcaddrs64.cton +++ b/cranelift/filetests/isa/x86/allones_funcaddrs64.cton @@ -1,9 +1,8 @@ ; binary emission of 64-bit code. test binemit -set is_64bit set opt_level=best set allones_funcaddrs -isa x86 haswell +target x86_64 haswell ; The binary encodings can be verified with the command: ; diff --git a/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount.cton b/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount.cton index c79de42317..595e5e99bb 100644 --- a/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount.cton +++ b/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount.cton @@ -1,7 +1,5 @@ - test compile -set is_64bit -isa x86 baseline +target x86_64 baseline ; clz/ctz on 64 bit operands diff --git a/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton b/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton index 72104272e1..16acad9cb6 100644 --- a/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton +++ b/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton @@ -1,7 +1,6 @@ test binemit -set is_64bit set opt_level=best -isa x86 baseline +target x86_64 baseline ; The binary encodings can be verified with the command: ; diff --git a/cranelift/filetests/isa/x86/binary32-float.cton b/cranelift/filetests/isa/x86/binary32-float.cton index 7b1a07853a..05847e9f58 100644 --- a/cranelift/filetests/isa/x86/binary32-float.cton +++ b/cranelift/filetests/isa/x86/binary32-float.cton @@ -1,6 +1,6 @@ ; Binary emission of 32-bit floating point code. test binemit -isa x86 haswell +target i686 haswell ; The binary encodings can be verified with the command: ; diff --git a/cranelift/filetests/isa/x86/binary32.cton b/cranelift/filetests/isa/x86/binary32.cton index 5ce1b228c9..b953d663f6 100644 --- a/cranelift/filetests/isa/x86/binary32.cton +++ b/cranelift/filetests/isa/x86/binary32.cton @@ -1,7 +1,7 @@ ; binary emission of x86-32 code. test binemit set opt_level=best -isa x86 haswell +target i686 haswell ; The binary encodings can be verified with the command: ; diff --git a/cranelift/filetests/isa/x86/binary64-float.cton b/cranelift/filetests/isa/x86/binary64-float.cton index 2f17c75ea6..9bc92e686b 100644 --- a/cranelift/filetests/isa/x86/binary64-float.cton +++ b/cranelift/filetests/isa/x86/binary64-float.cton @@ -1,8 +1,7 @@ ; Binary emission of 64-bit floating point code. test binemit -set is_64bit set opt_level=best -isa x86 haswell +target x86_64 haswell ; The binary encodings can be verified with the command: ; diff --git a/cranelift/filetests/isa/x86/binary64-pic.cton b/cranelift/filetests/isa/x86/binary64-pic.cton index 8d3b2e54ab..3678906135 100644 --- a/cranelift/filetests/isa/x86/binary64-pic.cton +++ b/cranelift/filetests/isa/x86/binary64-pic.cton @@ -1,9 +1,8 @@ ; binary emission of 64-bit code. test binemit -set is_64bit set opt_level=best set is_pic -isa x86 haswell +target x86_64 haswell ; The binary encodings can be verified with the command: ; diff --git a/cranelift/filetests/isa/x86/binary64.cton b/cranelift/filetests/isa/x86/binary64.cton index 2432736e64..96788c629b 100644 --- a/cranelift/filetests/isa/x86/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.cton @@ -1,8 +1,7 @@ ; binary emission of x86-64 code. test binemit -set is_64bit set opt_level=best -isa x86 haswell +target x86_64 haswell ; The binary encodings can be verified with the command: ; diff --git a/cranelift/filetests/isa/x86/legalize-call.cton b/cranelift/filetests/isa/x86/legalize-call.cton index 0eaaba0dfa..b66e7e3a53 100644 --- a/cranelift/filetests/isa/x86/legalize-call.cton +++ b/cranelift/filetests/isa/x86/legalize-call.cton @@ -1,8 +1,7 @@ ; Test legalization of a non-colocated call in 64-bit non-PIC mode. test legalizer -set is_64bit set opt_level=best -isa x86 haswell +target x86_64 haswell function %call() { fn0 = %foo() diff --git a/cranelift/filetests/isa/x86/legalize-custom.cton b/cranelift/filetests/isa/x86/legalize-custom.cton index 8f719056cd..667c08804c 100644 --- a/cranelift/filetests/isa/x86/legalize-custom.cton +++ b/cranelift/filetests/isa/x86/legalize-custom.cton @@ -1,8 +1,7 @@ ; Test the custom legalizations. test legalizer -isa x86 -set is_64bit -isa x86 +target i686 +target x86_64 ; regex: V=v\d+ ; regex: EBB=ebb\d+ diff --git a/cranelift/filetests/isa/x86/legalize-div-traps.cton b/cranelift/filetests/isa/x86/legalize-div-traps.cton index 4d7beb59b8..b849f07e1b 100644 --- a/cranelift/filetests/isa/x86/legalize-div-traps.cton +++ b/cranelift/filetests/isa/x86/legalize-div-traps.cton @@ -1,9 +1,8 @@ ; Test the division legalizations. test legalizer -set is_64bit ; See also legalize-div.cton. set avoid_div_traps=1 -isa x86 +target x86_64 ; regex: V=v\d+ ; regex: EBB=ebb\d+ diff --git a/cranelift/filetests/isa/x86/legalize-div.cton b/cranelift/filetests/isa/x86/legalize-div.cton index 3777be659e..d3a4d359da 100644 --- a/cranelift/filetests/isa/x86/legalize-div.cton +++ b/cranelift/filetests/isa/x86/legalize-div.cton @@ -1,9 +1,8 @@ ; Test the division legalizations. test legalizer -set is_64bit ; See also legalize-div-traps.cton. set avoid_div_traps=0 -isa x86 +target x86_64 ; regex: V=v\d+ ; regex: EBB=ebb\d+ diff --git a/cranelift/filetests/isa/x86/legalize-libcall.cton b/cranelift/filetests/isa/x86/legalize-libcall.cton index fdad2a3cf2..3605f2ff7b 100644 --- a/cranelift/filetests/isa/x86/legalize-libcall.cton +++ b/cranelift/filetests/isa/x86/legalize-libcall.cton @@ -1,10 +1,9 @@ test legalizer ; Pre-SSE 4.1, we need to use runtime library calls for floating point rounding operations. -set is_64bit set is_pic set call_conv=system_v -isa x86 +target x86_64 function %floor(f32) -> f32 { ebb0(v0: f32): diff --git a/cranelift/filetests/isa/x86/legalize-memory.cton b/cranelift/filetests/isa/x86/legalize-memory.cton index fc77c757e3..13375e124d 100644 --- a/cranelift/filetests/isa/x86/legalize-memory.cton +++ b/cranelift/filetests/isa/x86/legalize-memory.cton @@ -1,7 +1,6 @@ ; Test the legalization of memory objects. test legalizer -set is_64bit -isa x86 +target x86_64 ; regex: V=v\d+ ; regex: EBB=ebb\d+ diff --git a/cranelift/filetests/isa/x86/legalize-mulhi.cton b/cranelift/filetests/isa/x86/legalize-mulhi.cton index a588f25d4e..946bcd8428 100644 --- a/cranelift/filetests/isa/x86/legalize-mulhi.cton +++ b/cranelift/filetests/isa/x86/legalize-mulhi.cton @@ -1,7 +1,5 @@ - test compile -set is_64bit -isa x86 baseline +target x86_64 baseline ; umulhi/smulhi on 64 bit operands diff --git a/cranelift/filetests/isa/x86/optimized-zero-constants-32bit.cton b/cranelift/filetests/isa/x86/optimized-zero-constants-32bit.cton index 338569d055..c90e455527 100644 --- a/cranelift/filetests/isa/x86/optimized-zero-constants-32bit.cton +++ b/cranelift/filetests/isa/x86/optimized-zero-constants-32bit.cton @@ -1,7 +1,6 @@ ; Check that floating-point constants equal to zero are optimized correctly. test binemit -set is_64bit=0 -isa x86 +target i686 function %foo() -> f32 fast { ebb0: diff --git a/cranelift/filetests/isa/x86/optimized-zero-constants.cton b/cranelift/filetests/isa/x86/optimized-zero-constants.cton index d83bfd3ade..44060e9b97 100644 --- a/cranelift/filetests/isa/x86/optimized-zero-constants.cton +++ b/cranelift/filetests/isa/x86/optimized-zero-constants.cton @@ -1,7 +1,6 @@ ; Check that floating-point constants equal to zero are optimized correctly. test binemit -set is_64bit=1 -isa x86 +target x86_64 function %zero_const_32bit_no_rex() -> f32 fast { ebb0: diff --git a/cranelift/filetests/isa/x86/probestack-adjusts-sp.cton b/cranelift/filetests/isa/x86/probestack-adjusts-sp.cton index 2204886046..f0e351d84f 100644 --- a/cranelift/filetests/isa/x86/probestack-adjusts-sp.cton +++ b/cranelift/filetests/isa/x86/probestack-adjusts-sp.cton @@ -1,8 +1,7 @@ test compile -set is_64bit=1 set colocated_libcalls=1 set probestack_func_adjusts_sp=1 -isa x86 +target x86_64 ; Like %big in probestack.cton, but with the probestack function adjusting ; the stack pointer itself. diff --git a/cranelift/filetests/isa/x86/probestack-disabled.cton b/cranelift/filetests/isa/x86/probestack-disabled.cton index 282e088a3a..cbf1ed0b0d 100644 --- a/cranelift/filetests/isa/x86/probestack-disabled.cton +++ b/cranelift/filetests/isa/x86/probestack-disabled.cton @@ -1,8 +1,7 @@ test compile -set is_64bit=1 set colocated_libcalls=1 set probestack_enabled=0 -isa x86 +target x86_64 ; Like %big in probestack.cton, but with probes disabled. diff --git a/cranelift/filetests/isa/x86/probestack-noncolocated.cton b/cranelift/filetests/isa/x86/probestack-noncolocated.cton index 3248f4a142..8ec23719cc 100644 --- a/cranelift/filetests/isa/x86/probestack-noncolocated.cton +++ b/cranelift/filetests/isa/x86/probestack-noncolocated.cton @@ -1,6 +1,5 @@ test compile -set is_64bit=1 -isa x86 +target x86_64 ; Like %big in probestack.cton, but without a colocated libcall. diff --git a/cranelift/filetests/isa/x86/probestack-size.cton b/cranelift/filetests/isa/x86/probestack-size.cton index d7c92a5aa5..e413db3b1b 100644 --- a/cranelift/filetests/isa/x86/probestack-size.cton +++ b/cranelift/filetests/isa/x86/probestack-size.cton @@ -1,8 +1,7 @@ test compile -set is_64bit=1 set colocated_libcalls=1 set probestack_size_log2=13 -isa x86 +target x86_64 ; Like %big in probestack.cton, but now the probestack size is bigger ; and it no longer needs a probe. diff --git a/cranelift/filetests/isa/x86/probestack.cton b/cranelift/filetests/isa/x86/probestack.cton index 8b961da3bd..fbfd656309 100644 --- a/cranelift/filetests/isa/x86/probestack.cton +++ b/cranelift/filetests/isa/x86/probestack.cton @@ -1,7 +1,6 @@ test compile -set is_64bit=1 set colocated_libcalls=1 -isa x86 +target x86_64 ; A function with a big stack frame. This should have a stack probe. diff --git a/cranelift/filetests/isa/x86/prologue-epilogue.cton b/cranelift/filetests/isa/x86/prologue-epilogue.cton index 249f2a9ff9..a19af2c09b 100644 --- a/cranelift/filetests/isa/x86/prologue-epilogue.cton +++ b/cranelift/filetests/isa/x86/prologue-epilogue.cton @@ -1,8 +1,7 @@ test compile -set is_64bit set opt_level=best set is_pic -isa x86 haswell +target x86_64 haswell ; An empty function. diff --git a/cranelift/filetests/isa/x86/shrink.cton b/cranelift/filetests/isa/x86/shrink.cton index 1dc5fbcd42..7648a6dd22 100644 --- a/cranelift/filetests/isa/x86/shrink.cton +++ b/cranelift/filetests/isa/x86/shrink.cton @@ -1,7 +1,6 @@ test binemit -set is_64bit=1 set opt_level=best -isa x86 +target x86_64 ; Test that instruction shrinking eliminates REX prefixes when possible. diff --git a/cranelift/filetests/isa/x86/windows_fastcall_x64.cton b/cranelift/filetests/isa/x86/windows_fastcall_x64.cton index 204bd5bb70..7af721ed8e 100644 --- a/cranelift/filetests/isa/x86/windows_fastcall_x64.cton +++ b/cranelift/filetests/isa/x86/windows_fastcall_x64.cton @@ -1,8 +1,7 @@ test compile -set is_64bit set opt_level=best set is_pic -isa x86 haswell +target x86_64 haswell ; check if for one arg we use the right register function %one_arg(i64) windows_fastcall { diff --git a/cranelift/filetests/parser/instruction_encoding.cton b/cranelift/filetests/parser/instruction_encoding.cton index 7d75e4362a..9a405030a6 100644 --- a/cranelift/filetests/parser/instruction_encoding.cton +++ b/cranelift/filetests/parser/instruction_encoding.cton @@ -1,6 +1,6 @@ test cat -isa riscv +target riscv32 ; regex: WS=[ \t]* diff --git a/cranelift/filetests/postopt/basic.cton b/cranelift/filetests/postopt/basic.cton index 678cecc27c..5b65407004 100644 --- a/cranelift/filetests/postopt/basic.cton +++ b/cranelift/filetests/postopt/basic.cton @@ -1,5 +1,5 @@ test postopt -isa x86 +target i686 ; Test that compare+branch sequences are folded effectively on x86. diff --git a/cranelift/filetests/postopt/complex_memory_ops.cton b/cranelift/filetests/postopt/complex_memory_ops.cton index 8f12e61577..0977fa0d7d 100644 --- a/cranelift/filetests/postopt/complex_memory_ops.cton +++ b/cranelift/filetests/postopt/complex_memory_ops.cton @@ -1,6 +1,5 @@ test postopt -set is_64bit -isa x86 +target x86_64 function %dual_loads(i64, i64) -> i64 { ebb0(v0: i64, v1: i64): diff --git a/cranelift/filetests/preopt/div_by_const_indirect.cton b/cranelift/filetests/preopt/div_by_const_indirect.cton index 5d3f9475fc..5d3266d26a 100644 --- a/cranelift/filetests/preopt/div_by_const_indirect.cton +++ b/cranelift/filetests/preopt/div_by_const_indirect.cton @@ -1,6 +1,5 @@ - test preopt -isa x86 baseline +target i686 baseline ; Cases where the denominator is created by an iconst diff --git a/cranelift/filetests/preopt/div_by_const_non_power_of_2.cton b/cranelift/filetests/preopt/div_by_const_non_power_of_2.cton index fa3f7fa96a..1d05d4a53c 100644 --- a/cranelift/filetests/preopt/div_by_const_non_power_of_2.cton +++ b/cranelift/filetests/preopt/div_by_const_non_power_of_2.cton @@ -1,6 +1,5 @@ - test preopt -isa x86 baseline +target i686 baseline ; -------- U32 -------- diff --git a/cranelift/filetests/preopt/div_by_const_power_of_2.cton b/cranelift/filetests/preopt/div_by_const_power_of_2.cton index 2318aa1f81..a047107c26 100644 --- a/cranelift/filetests/preopt/div_by_const_power_of_2.cton +++ b/cranelift/filetests/preopt/div_by_const_power_of_2.cton @@ -1,6 +1,5 @@ - test preopt -isa x86 baseline +target i686 baseline ; -------- U32 -------- diff --git a/cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton b/cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton index 3d4b986213..f440aa4a6e 100644 --- a/cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton +++ b/cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton @@ -1,6 +1,5 @@ - test preopt -isa x86 baseline +target i686 baseline ; -------- U32 -------- diff --git a/cranelift/filetests/preopt/rem_by_const_power_of_2.cton b/cranelift/filetests/preopt/rem_by_const_power_of_2.cton index 7b537fdbb4..70bd1bbd4e 100644 --- a/cranelift/filetests/preopt/rem_by_const_power_of_2.cton +++ b/cranelift/filetests/preopt/rem_by_const_power_of_2.cton @@ -1,6 +1,5 @@ - test preopt -isa x86 baseline +target i686 baseline ; -------- U32 -------- diff --git a/cranelift/filetests/preopt/simplify.cton b/cranelift/filetests/preopt/simplify.cton index 32f19f4f03..4426b98186 100644 --- a/cranelift/filetests/preopt/simplify.cton +++ b/cranelift/filetests/preopt/simplify.cton @@ -1,5 +1,5 @@ test preopt -isa x86 +target i686 function %iadd_imm(i32) -> i32 { ebb0(v0: i32): diff --git a/cranelift/filetests/regalloc/aliases.cton b/cranelift/filetests/regalloc/aliases.cton index eb84c77f98..b9b76c3f1d 100644 --- a/cranelift/filetests/regalloc/aliases.cton +++ b/cranelift/filetests/regalloc/aliases.cton @@ -1,6 +1,5 @@ test regalloc -set is_64bit -isa x86 haswell +target x86_64 haswell function %value_aliases(i32, f32, i64 vmctx) baldrdash { gv0 = vmctx diff --git a/cranelift/filetests/regalloc/basic.cton b/cranelift/filetests/regalloc/basic.cton index c24f87c1f9..d10ea726ce 100644 --- a/cranelift/filetests/regalloc/basic.cton +++ b/cranelift/filetests/regalloc/basic.cton @@ -1,7 +1,7 @@ test regalloc ; We can add more ISAs once they have defined encodings. -isa riscv +target riscv32 ; regex: RX=%x\d+ diff --git a/cranelift/filetests/regalloc/coalesce.cton b/cranelift/filetests/regalloc/coalesce.cton index 04782fc1fc..0085ce0fd8 100644 --- a/cranelift/filetests/regalloc/coalesce.cton +++ b/cranelift/filetests/regalloc/coalesce.cton @@ -1,5 +1,5 @@ test regalloc -isa riscv +target riscv32 ; Test the coalescer. ; regex: V=v\d+ diff --git a/cranelift/filetests/regalloc/coalescing-207.cton b/cranelift/filetests/regalloc/coalescing-207.cton index d8af887da3..2fc3cfc429 100644 --- a/cranelift/filetests/regalloc/coalescing-207.cton +++ b/cranelift/filetests/regalloc/coalescing-207.cton @@ -1,6 +1,5 @@ test regalloc -set is_64bit -isa x86 haswell +target x86_64 haswell ; Reported as https://github.com/cretonne/cretonne/issues/207 ; diff --git a/cranelift/filetests/regalloc/coalescing-216.cton b/cranelift/filetests/regalloc/coalescing-216.cton index fc986f7fec..fc4e877f68 100644 --- a/cranelift/filetests/regalloc/coalescing-216.cton +++ b/cranelift/filetests/regalloc/coalescing-216.cton @@ -1,6 +1,5 @@ test regalloc -set is_64bit -isa x86 haswell +target x86_64 haswell ; Reported as https://github.com/cretonne/cretonne/issues/216 from the Binaryen fuzzer. ; diff --git a/cranelift/filetests/regalloc/coloring-227.cton b/cranelift/filetests/regalloc/coloring-227.cton index 2b327633c7..248f75dbef 100644 --- a/cranelift/filetests/regalloc/coloring-227.cton +++ b/cranelift/filetests/regalloc/coloring-227.cton @@ -1,6 +1,5 @@ test regalloc -set is_64bit -isa x86 haswell +target x86_64 haswell function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) system_v { gv0 = vmctx diff --git a/cranelift/filetests/regalloc/constraints.cton b/cranelift/filetests/regalloc/constraints.cton index e4db3bf9da..a0478baae9 100644 --- a/cranelift/filetests/regalloc/constraints.cton +++ b/cranelift/filetests/regalloc/constraints.cton @@ -1,5 +1,5 @@ test regalloc -isa x86 +target i686 ; regex: V=v\d+ ; regex: REG=%r([abcd]x|[sd]i) diff --git a/cranelift/filetests/regalloc/ghost-param.cton b/cranelift/filetests/regalloc/ghost-param.cton index 2064231318..8a022b4af5 100644 --- a/cranelift/filetests/regalloc/ghost-param.cton +++ b/cranelift/filetests/regalloc/ghost-param.cton @@ -1,6 +1,5 @@ test regalloc -set is_64bit -isa x86 haswell +target x86_64 haswell ; This test case would create an EBB parameter that was a ghost value. ; The coalescer would insert a copy of the ghost value, leading to verifier errors. diff --git a/cranelift/filetests/regalloc/global-constraints.cton b/cranelift/filetests/regalloc/global-constraints.cton index 4975adeac5..e3e59dd470 100644 --- a/cranelift/filetests/regalloc/global-constraints.cton +++ b/cranelift/filetests/regalloc/global-constraints.cton @@ -1,5 +1,5 @@ test regalloc -isa x86 +target i686 ; This test covers the troubles when values with global live ranges are defined ; by instructions with constrained register classes. diff --git a/cranelift/filetests/regalloc/global-fixed.cton b/cranelift/filetests/regalloc/global-fixed.cton index 61d4928806..eb8e23d7af 100644 --- a/cranelift/filetests/regalloc/global-fixed.cton +++ b/cranelift/filetests/regalloc/global-fixed.cton @@ -1,6 +1,5 @@ test regalloc -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %foo() system_v { ebb4: diff --git a/cranelift/filetests/regalloc/infinite-interference.cton b/cranelift/filetests/regalloc/infinite-interference.cton index 8f07c4cf77..bce607b7f6 100644 --- a/cranelift/filetests/regalloc/infinite-interference.cton +++ b/cranelift/filetests/regalloc/infinite-interference.cton @@ -1,5 +1,5 @@ test regalloc -isa riscv +target riscv32 ; Here, the coalescer initially builds vreg0 = [v1, v2, v3] ; diff --git a/cranelift/filetests/regalloc/iterate.cton b/cranelift/filetests/regalloc/iterate.cton index 44e1265a96..c7ad214432 100644 --- a/cranelift/filetests/regalloc/iterate.cton +++ b/cranelift/filetests/regalloc/iterate.cton @@ -1,6 +1,5 @@ test regalloc -set is_64bit -isa x86 haswell +target x86_64 haswell function u0:9(i64 [%rdi], f32 [%xmm0], f64 [%xmm1], i32 [%rsi], i32 [%rdx], i64 vmctx [%r14]) -> i64 [%rax] baldrdash { ebb0(v0: i64, v1: f32, v2: f64, v3: i32, v4: i32, v5: i64): diff --git a/cranelift/filetests/regalloc/multi-constraints.cton b/cranelift/filetests/regalloc/multi-constraints.cton index 590d93eb38..d0b6f7faf0 100644 --- a/cranelift/filetests/regalloc/multi-constraints.cton +++ b/cranelift/filetests/regalloc/multi-constraints.cton @@ -1,6 +1,5 @@ test regalloc -set is_64bit -isa x86 haswell +target x86_64 haswell ; Test combinations of constraints. ; diff --git a/cranelift/filetests/regalloc/output-interference.cton b/cranelift/filetests/regalloc/output-interference.cton index ac2bb537c7..ab027a72fb 100644 --- a/cranelift/filetests/regalloc/output-interference.cton +++ b/cranelift/filetests/regalloc/output-interference.cton @@ -1,6 +1,5 @@ test regalloc -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %test(i64) -> i64 system_v { ebb0(v0: i64): diff --git a/cranelift/filetests/regalloc/reload-208.cton b/cranelift/filetests/regalloc/reload-208.cton index 259c4a169b..6b4dc74292 100644 --- a/cranelift/filetests/regalloc/reload-208.cton +++ b/cranelift/filetests/regalloc/reload-208.cton @@ -1,6 +1,5 @@ test regalloc -set is_64bit -isa x86 haswell +target x86_64 haswell ; regex: V=v\d+ diff --git a/cranelift/filetests/regalloc/reload.cton b/cranelift/filetests/regalloc/reload.cton index d0a9646808..517b0990d1 100644 --- a/cranelift/filetests/regalloc/reload.cton +++ b/cranelift/filetests/regalloc/reload.cton @@ -1,5 +1,5 @@ test regalloc -isa riscv enable_e +target riscv32 enable_e ; regex: V=v\d+ diff --git a/cranelift/filetests/regalloc/schedule-moves.cton b/cranelift/filetests/regalloc/schedule-moves.cton index 946bcc9cf3..afd652ece9 100644 --- a/cranelift/filetests/regalloc/schedule-moves.cton +++ b/cranelift/filetests/regalloc/schedule-moves.cton @@ -1,5 +1,5 @@ test regalloc -isa x86 haswell +target i686 haswell function %pr165() system_v { ebb0: diff --git a/cranelift/filetests/regalloc/spill-noregs.cton b/cranelift/filetests/regalloc/spill-noregs.cton index ffb9f041c1..6e0b2baabb 100644 --- a/cranelift/filetests/regalloc/spill-noregs.cton +++ b/cranelift/filetests/regalloc/spill-noregs.cton @@ -1,6 +1,5 @@ test regalloc -set is_64bit -isa x86 +target x86_64 ; Test case found by the Binaryen fuzzer. ; diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.cton index 7205d6a851..21731f321a 100644 --- a/cranelift/filetests/regalloc/spill.cton +++ b/cranelift/filetests/regalloc/spill.cton @@ -12,7 +12,7 @@ test regalloc ; regex: V=v\d+ ; regex: WS=\s+ -isa riscv enable_e +target riscv32 enable_e ; In straight-line code, the first value defined is spilled. ; That is in order: diff --git a/cranelift/filetests/regalloc/unreachable_code.cton b/cranelift/filetests/regalloc/unreachable_code.cton index 8bfffc22a2..9d11ded01a 100644 --- a/cranelift/filetests/regalloc/unreachable_code.cton +++ b/cranelift/filetests/regalloc/unreachable_code.cton @@ -1,9 +1,8 @@ ; Use "test compile" here otherwise the dead blocks won't be eliminated. test compile -set is_64bit set probestack_enabled=0 -isa x86 haswell +target x86_64 haswell ; This function contains unreachable blocks which trip up the register ; allocator if they don't get cleared out. diff --git a/cranelift/filetests/regalloc/x86-regres.cton b/cranelift/filetests/regalloc/x86-regres.cton index e396fb1022..66c3d45c2e 100644 --- a/cranelift/filetests/regalloc/x86-regres.cton +++ b/cranelift/filetests/regalloc/x86-regres.cton @@ -1,6 +1,6 @@ test regalloc -isa x86 +target i686 ; regex: V=v\d+ diff --git a/cranelift/filetests/verifier/flags.cton b/cranelift/filetests/verifier/flags.cton index 55d9bdf4e1..8a6abe8b4d 100644 --- a/cranelift/filetests/verifier/flags.cton +++ b/cranelift/filetests/verifier/flags.cton @@ -1,5 +1,5 @@ test verifier -isa x86 +target i686 ; Simple, correct use of CPU flags. function %simple(i32) -> i32 { diff --git a/cranelift/filetests/wasm/control.cton b/cranelift/filetests/wasm/control.cton index 30d4240490..a34bbeb935 100644 --- a/cranelift/filetests/wasm/control.cton +++ b/cranelift/filetests/wasm/control.cton @@ -1,11 +1,9 @@ ; Test basic code generation for control flow WebAssembly instructions. test compile -set is_64bit=0 -isa x86 haswell +target i686 haswell -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %br_if(i32) -> i32 { ebb0(v0: i32): diff --git a/cranelift/filetests/wasm/conversions.cton b/cranelift/filetests/wasm/conversions.cton index 5ca8072067..0b7630da0d 100644 --- a/cranelift/filetests/wasm/conversions.cton +++ b/cranelift/filetests/wasm/conversions.cton @@ -1,8 +1,7 @@ ; Test code generation for WebAssembly type conversion operators. test compile -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %i32_wrap_i64(i64) -> i32 { ebb0(v0: i64): diff --git a/cranelift/filetests/wasm/f32-arith.cton b/cranelift/filetests/wasm/f32-arith.cton index 7ce8eb4142..c439d2e798 100644 --- a/cranelift/filetests/wasm/f32-arith.cton +++ b/cranelift/filetests/wasm/f32-arith.cton @@ -1,17 +1,10 @@ ; Test basic code generation for f32 arithmetic WebAssembly instructions. test compile -set is_64bit=0 -isa x86 haswell - -set is_64bit=0 -isa x86 baseline - -set is_64bit=1 -isa x86 haswell - -set is_64bit=1 -isa x86 baseline +target i686 haswell +target i686 baseline +target x86_64 haswell +target x86_64 baseline ; Constants. diff --git a/cranelift/filetests/wasm/f32-compares.cton b/cranelift/filetests/wasm/f32-compares.cton index 0cdee58ef3..e314646b1c 100644 --- a/cranelift/filetests/wasm/f32-compares.cton +++ b/cranelift/filetests/wasm/f32-compares.cton @@ -1,11 +1,9 @@ ; Test code generation for WebAssembly f32 comparison operators. test compile -set is_64bit=0 -isa x86 haswell +target i686 haswell -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %f32_eq(f32, f32) -> i32 { ebb0(v0: f32, v1: f32): diff --git a/cranelift/filetests/wasm/f32-memory64.cton b/cranelift/filetests/wasm/f32-memory64.cton index a6e0d54af7..42b183311b 100644 --- a/cranelift/filetests/wasm/f32-memory64.cton +++ b/cranelift/filetests/wasm/f32-memory64.cton @@ -3,8 +3,7 @@ test compile ; We only test on 64-bit since the heap_addr instructions and vmctx parameters ; explicitly mention the pointer width. -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %f32_load(i32, i64 vmctx) -> f32 { gv0 = vmctx diff --git a/cranelift/filetests/wasm/f64-arith.cton b/cranelift/filetests/wasm/f64-arith.cton index 150d9a0d8e..7209ed6024 100644 --- a/cranelift/filetests/wasm/f64-arith.cton +++ b/cranelift/filetests/wasm/f64-arith.cton @@ -1,11 +1,8 @@ ; Test basic code generation for f64 arithmetic WebAssembly instructions. test compile -set is_64bit=1 -isa x86 haswell - -set is_64bit=1 -isa x86 baseline +target x86_64 haswell +target x86_64 baseline ; Constants. diff --git a/cranelift/filetests/wasm/f64-compares.cton b/cranelift/filetests/wasm/f64-compares.cton index ff1b9526d0..5d51ebb443 100644 --- a/cranelift/filetests/wasm/f64-compares.cton +++ b/cranelift/filetests/wasm/f64-compares.cton @@ -1,11 +1,9 @@ ; Test code generation for WebAssembly f64 comparison operators. test compile -set is_64bit=0 -isa x86 haswell +target i686 haswell -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %f64_eq(f64, f64) -> i32 { ebb0(v0: f64, v1: f64): diff --git a/cranelift/filetests/wasm/f64-memory64.cton b/cranelift/filetests/wasm/f64-memory64.cton index 0a15e69a37..b495dfd344 100644 --- a/cranelift/filetests/wasm/f64-memory64.cton +++ b/cranelift/filetests/wasm/f64-memory64.cton @@ -3,8 +3,7 @@ test compile ; We only test on 64-bit since the heap_addr instructions and vmctx parameters ; explicitly mention the pointer width. -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %f64_load(i32, i64 vmctx) -> f64 { gv0 = vmctx diff --git a/cranelift/filetests/wasm/i32-arith.cton b/cranelift/filetests/wasm/i32-arith.cton index f6ee971c69..8c92613150 100644 --- a/cranelift/filetests/wasm/i32-arith.cton +++ b/cranelift/filetests/wasm/i32-arith.cton @@ -1,17 +1,10 @@ ; Test basic code generation for i32 arithmetic WebAssembly instructions. test compile -set is_64bit=0 -isa x86 haswell - -set is_64bit=0 -isa x86 baseline - -set is_64bit=1 -isa x86 haswell - -set is_64bit=1 -isa x86 baseline +target i686 haswell +target i686 baseline +target x86_64 haswell +target x86_64 baseline ; Constants. diff --git a/cranelift/filetests/wasm/i32-compares.cton b/cranelift/filetests/wasm/i32-compares.cton index a36df1b28a..1b43c5938a 100644 --- a/cranelift/filetests/wasm/i32-compares.cton +++ b/cranelift/filetests/wasm/i32-compares.cton @@ -1,11 +1,9 @@ ; Test code generation for WebAssembly i32 comparison operators. test compile -set is_64bit=0 -isa x86 haswell +target i686 haswell -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %i32_eqz(i32) -> i32 { ebb0(v0: i32): diff --git a/cranelift/filetests/wasm/i32-memory64.cton b/cranelift/filetests/wasm/i32-memory64.cton index 319609e9d3..b32325561a 100644 --- a/cranelift/filetests/wasm/i32-memory64.cton +++ b/cranelift/filetests/wasm/i32-memory64.cton @@ -3,8 +3,7 @@ test compile ; We only test on 64-bit since the heap_addr instructions and vmctx parameters ; explicitly mention the pointer width. -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %i32_load(i32, i64 vmctx) -> i32 { gv0 = vmctx diff --git a/cranelift/filetests/wasm/i64-arith.cton b/cranelift/filetests/wasm/i64-arith.cton index 52016039e0..c4cd4a4507 100644 --- a/cranelift/filetests/wasm/i64-arith.cton +++ b/cranelift/filetests/wasm/i64-arith.cton @@ -1,11 +1,8 @@ ; Test basic code generation for i64 arithmetic WebAssembly instructions. test compile -set is_64bit=1 -isa x86 haswell - -set is_64bit=1 -isa x86 baseline +target x86_64 haswell +target x86_64 baseline ; Constants. diff --git a/cranelift/filetests/wasm/i64-compares.cton b/cranelift/filetests/wasm/i64-compares.cton index 4679268018..e137b7e19c 100644 --- a/cranelift/filetests/wasm/i64-compares.cton +++ b/cranelift/filetests/wasm/i64-compares.cton @@ -1,8 +1,7 @@ ; Test code generation for WebAssembly i64 comparison operators. test compile -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %i64_eqz(i64) -> i32 { ebb0(v0: i64): diff --git a/cranelift/filetests/wasm/i64-memory64.cton b/cranelift/filetests/wasm/i64-memory64.cton index 7c9836dbf3..76a1a02e4b 100644 --- a/cranelift/filetests/wasm/i64-memory64.cton +++ b/cranelift/filetests/wasm/i64-memory64.cton @@ -3,8 +3,7 @@ test compile ; We only test on 64-bit since the heap_addr instructions and vmctx parameters ; explicitly mention the pointer width. -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %i64_load(i32, i64 vmctx) -> i64 { gv0 = vmctx diff --git a/cranelift/filetests/wasm/select.cton b/cranelift/filetests/wasm/select.cton index 7f1c26e724..f5f55cda24 100644 --- a/cranelift/filetests/wasm/select.cton +++ b/cranelift/filetests/wasm/select.cton @@ -1,11 +1,9 @@ ; Test basic code generation for the select WebAssembly instruction. test compile -set is_64bit=0 -isa x86 haswell +target i686 haswell -set is_64bit=1 -isa x86 haswell +target x86_64 haswell function %select_i32(i32, i32, i32) -> i32 { ebb0(v0: i32, v1: i32, v2: i32): diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml index d48922c1f4..f1169e2763 100644 --- a/cranelift/fuzz/Cargo.toml +++ b/cranelift/fuzz/Cargo.toml @@ -20,6 +20,9 @@ git = "https://github.com/rust-fuzz/libfuzzer-sys.git" [dependencies.cretonne-wasm] path = "../lib/wasm" +[dependencies.target-lexicon] +version = "0.0.0" + # Prevent this from interfering with workspaces [workspace] members = ["."] diff --git a/cranelift/fuzz/fuzz_translate_module.rs b/cranelift/fuzz/fuzz_translate_module.rs index 45180c14af..d398483961 100644 --- a/cranelift/fuzz/fuzz_translate_module.rs +++ b/cranelift/fuzz/fuzz_translate_module.rs @@ -3,13 +3,16 @@ extern crate libfuzzer_sys; extern crate binaryen; extern crate cretonne_wasm; +#[macro_use] +extern crate target_lexicon; use cretonne_wasm::{translate_module, DummyEnvironment}; +use std::str::FromStr; fuzz_target!(|data: &[u8]| { let binaryen_module = binaryen::tools::translate_to_fuzz_mvp(data); let wasm = binaryen_module.write(); - let mut dummy_environ = DummyEnvironment::default(); + let mut dummy_environ = DummyEnvironment::with_triple(triple!("x86_64")); translate_module(&wasm, &mut dummy_environ).unwrap(); }); diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 37f70df364..7eea32262e 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -9,7 +9,8 @@ use cretonne_codegen::{binemit, ir}; use cretonne_reader::parse_test; use std::path::Path; use std::path::PathBuf; -use utils::{parse_sets_and_isa, read_to_string}; +use target_lexicon::Architecture; +use utils::{parse_sets_and_triple, read_to_string}; struct PrintRelocs { flag_print: bool, @@ -64,7 +65,7 @@ pub fn run( flag_set: &[String], flag_isa: &str, ) -> Result<(), String> { - let parsed = parse_sets_and_isa(flag_set, flag_isa)?; + let parsed = parse_sets_and_triple(flag_set, flag_isa)?; for filename in files { let path = Path::new(&filename); @@ -136,23 +137,28 @@ fn handle_module( } fn get_disassembler(isa: &TargetIsa) -> Result { - let cs = match isa.name() { - "riscv" => return Err(String::from("No disassembler for RiscV")), - "x86" => { - if isa.flags().is_64bit() { - Capstone::new() - .x86() - .mode(arch::x86::ArchMode::Mode64) - .build() - } else { - Capstone::new() - .x86() - .mode(arch::x86::ArchMode::Mode32) - .build() - } + let cs = match isa.triple().architecture { + Architecture::Riscv32 | Architecture::Riscv64 => { + return Err(String::from("No disassembler for RiscV")) } - "arm32" => Capstone::new().arm().mode(arch::arm::ArchMode::Arm).build(), - "arm64" => Capstone::new() + Architecture::I386 | Architecture::I586 | Architecture::I686 => Capstone::new() + .x86() + .mode(arch::x86::ArchMode::Mode32) + .build(), + Architecture::X86_64 => Capstone::new() + .x86() + .mode(arch::x86::ArchMode::Mode64) + .build(), + Architecture::Arm + | Architecture::Armv4t + | Architecture::Armv5te + | Architecture::Armv7 + | Architecture::Armv7s => Capstone::new().arm().mode(arch::arm::ArchMode::Arm).build(), + Architecture::Thumbv6m | Architecture::Thumbv7em | Architecture::Thumbv7m => Capstone::new( + ).arm() + .mode(arch::arm::ArchMode::Thumb) + .build(), + Architecture::Aarch64 => Capstone::new() .arm64() .mode(arch::arm64::ArchMode::Arm) .build(), diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index d1805d6dc8..370e6d5c65 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -27,6 +27,7 @@ cfg_if! { mod wasm; } } +extern crate target_lexicon; use cretonne_codegen::{timing, VERSION}; use docopt::Docopt; @@ -47,8 +48,8 @@ Usage: cton-util cat ... cton-util filecheck [-v] cton-util print-cfg ... - cton-util compile [-vpT] [--set ]... [--isa ] ... - cton-util wasm [-ctvpTs] [--set ]... [--isa ] ... + cton-util compile [-vpT] [--set ]... [--target ] ... + cton-util wasm [-ctvpTs] [--set ]... [--target ] ... cton-util --help | --version Options: @@ -64,7 +65,8 @@ Options: -p, --print print the resulting Cretonne IR -h, --help print this help message --set= configure Cretonne settings - --isa= specify the Cretonne ISA + --target= + specify the Cretonne target --version print the Cretonne version "; @@ -83,7 +85,7 @@ struct Args { flag_print: bool, flag_verbose: bool, flag_set: Vec, - flag_isa: String, + flag_target: String, flag_time_passes: bool, flag_print_size: bool, } @@ -116,7 +118,7 @@ fn cton_util() -> CommandResult { args.arg_file, args.flag_print, &args.flag_set, - &args.flag_isa, + &args.flag_target, ) } else if args.cmd_wasm { #[cfg(feature = "wasm")] @@ -127,7 +129,7 @@ fn cton_util() -> CommandResult { args.flag_check_translation, args.flag_print, &args.flag_set, - &args.flag_isa, + &args.flag_target, args.flag_print_size, ); diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index ee879d0843..138d4f5dcc 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -7,6 +7,8 @@ use cretonne_reader::{parse_options, Location}; use std::fs::File; use std::io::{self, Read}; use std::path::Path; +use std::str::FromStr; +use target_lexicon::Triple; /// Read an entire file into a string. pub fn read_to_string>(path: P) -> io::Result { @@ -40,8 +42,11 @@ impl OwnedFlagsOrIsa { } } -/// Parse "set" and "isa" commands. -pub fn parse_sets_and_isa(flag_set: &[String], flag_isa: &str) -> Result { +/// Parse "set" and "triple" commands. +pub fn parse_sets_and_triple( + flag_set: &[String], + flag_triple: &str, +) -> Result { let mut flag_builder = settings::builder(); parse_options( flag_set.iter().map(|x| x.as_str()), @@ -49,12 +54,21 @@ pub fn parse_sets_and_isa(flag_set: &[String], flag_isa: &str) -> Result format!("unknown ISA '{}'", isa_name), - isa::LookupError::Unsupported => format!("support for ISA '{}' not enabled", isa_name), + let mut words = flag_triple.trim().split_whitespace(); + // Look for `target foo`. + if let Some(triple_name) = words.next() { + let triple = match Triple::from_str(triple_name) { + Ok(triple) => triple, + Err(parse_error) => return Err(format!("{}", parse_error)), + }; + let mut isa_builder = isa::lookup(triple).map_err(|err| match err { + isa::LookupError::SupportDisabled => { + format!("support for triple '{}' is disabled", triple_name) + } + isa::LookupError::Unsupported => format!( + "support for triple '{}' is not implemented yet", + triple_name + ), })?; // Apply the ISA-specific settings to `isa_builder`. parse_options(words, &mut isa_builder, &Location { line_number: 0 }) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 4c1907b6ce..13356a4282 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -12,7 +12,7 @@ use std::error::Error; use std::path::Path; use std::path::PathBuf; use term; -use utils::{parse_sets_and_isa, read_to_end}; +use utils::{parse_sets_and_triple, read_to_end}; use wabt::wat2wasm; macro_rules! vprintln { @@ -38,10 +38,10 @@ pub fn run( flag_check_translation: bool, flag_print: bool, flag_set: &[String], - flag_isa: &str, + flag_triple: &str, flag_print_size: bool, ) -> Result<(), String> { - let parsed = parse_sets_and_isa(flag_set, flag_isa)?; + let parsed = parse_sets_and_triple(flag_set, flag_triple)?; for filename in files { let path = Path::new(&filename); @@ -79,9 +79,7 @@ fn handle_module( vprint!(flag_verbose, "Translating... "); terminal.reset().unwrap(); - let mut data = read_to_end(path.clone()).map_err(|err| { - String::from(err.description()) - })?; + let mut data = read_to_end(path.clone()).map_err(|err| String::from(err.description()))?; if !data.starts_with(&[b'\0', b'a', b's', b'm']) { data = match wat2wasm(&data) { @@ -90,10 +88,9 @@ fn handle_module( }; } - let mut dummy_environ = DummyEnvironment::with_flags(fisa.flags.clone()); - translate_module(&data, &mut dummy_environ).map_err( - |e| e.to_string(), - )?; + let mut dummy_environ = + DummyEnvironment::with_triple_flags(fisa.isa.unwrap().triple().clone(), fisa.flags.clone()); + translate_module(&data, &mut dummy_environ).map_err(|e| e.to_string())?; terminal.fg(term::color::GREEN).unwrap(); vprintln!(flag_verbose, "ok"); @@ -142,24 +139,22 @@ fn handle_module( let mut context = Context::new(); context.func = func.clone(); if flag_check_translation { - context.verify(fisa).map_err(|err| { - pretty_verifier_error(&context.func, fisa.isa, &err) - })?; + context + .verify(fisa) + .map_err(|err| pretty_verifier_error(&context.func, fisa.isa, &err))?; } else if let Some(isa) = fisa.isa { - let compiled_size = context.compile(isa).map_err(|err| { - pretty_error(&context.func, fisa.isa, err) - })?; + let compiled_size = context + .compile(isa) + .map_err(|err| pretty_error(&context.func, fisa.isa, err))?; if flag_print_size { println!( "Function #{} code size: {} bytes", - func_index, - compiled_size + func_index, compiled_size ); total_module_code_size += compiled_size; println!( "Function #{} bytecode size: {} bytes", - func_index, - dummy_environ.func_bytecode_sizes[def_index] + func_index, dummy_environ.func_bytecode_sizes[def_index] ); } } else { diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 26bf497347..0e7b5f32ea 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -15,6 +15,7 @@ cretonne-entity = { path = "../entity", version = "0.8.0", default-features = fa failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } hashmap_core = { version = "0.1.4", optional = true } +target-lexicon = { version = "0.0.0", default-features = false } # It is a goal of the cretonne-codegen crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be diff --git a/lib/codegen/meta/base/settings.py b/lib/codegen/meta/base/settings.py index 5753c35273..8e88a45abc 100644 --- a/lib/codegen/meta/base/settings.py +++ b/lib/codegen/meta/base/settings.py @@ -27,8 +27,6 @@ enable_verifier = BoolSetting( """, default=True) -is_64bit = BoolSetting("Enable 64-bit code generation") - call_conv = EnumSetting( """ Default calling convention: @@ -89,8 +87,6 @@ avoid_div_traps = BoolSetting( this setting has no effect - explicit checks are always inserted. """) -is_compressed = BoolSetting("Enable compressed instructions") - enable_float = BoolSetting( """ Enable the use of floating-point instructions diff --git a/lib/codegen/src/isa/arm32/mod.rs b/lib/codegen/src/isa/arm32/mod.rs index 560c71248c..bb0214b3ec 100644 --- a/lib/codegen/src/isa/arm32/mod.rs +++ b/lib/codegen/src/isa/arm32/mod.rs @@ -15,32 +15,43 @@ use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use regalloc; use std::boxed::Box; use std::fmt; +use target_lexicon::{Architecture, Triple}; #[allow(dead_code)] struct Isa { + triple: Triple, shared_flags: shared_settings::Flags, isa_flags: settings::Flags, cpumode: &'static [shared_enc_tables::Level1Entry], } /// Get an ISA builder for creating ARM32 targets. -pub fn isa_builder() -> IsaBuilder { +pub fn isa_builder(triple: Triple) -> IsaBuilder { IsaBuilder { + triple, setup: settings::builder(), constructor: isa_constructor, } } fn isa_constructor( + triple: Triple, shared_flags: shared_settings::Flags, builder: shared_settings::Builder, ) -> Box { - let level1 = if shared_flags.is_compressed() { - &enc_tables::LEVEL1_T32[..] - } else { - &enc_tables::LEVEL1_A32[..] + let level1 = match triple.architecture { + Architecture::Thumbv6m | Architecture::Thumbv7em | Architecture::Thumbv7m => { + &enc_tables::LEVEL1_T32[..] + } + Architecture::Arm + | Architecture::Armv4t + | Architecture::Armv5te + | Architecture::Armv7 + | Architecture::Armv7s => &enc_tables::LEVEL1_A32[..], + _ => panic!(), }; Box::new(Isa { + triple, isa_flags: settings::Flags::new(&shared_flags, builder), shared_flags, cpumode: level1, @@ -52,6 +63,10 @@ impl TargetIsa for Isa { "arm32" } + fn triple(&self) -> &Triple { + &self.triple + } + fn flags(&self) -> &shared_settings::Flags { &self.shared_flags } diff --git a/lib/codegen/src/isa/arm64/mod.rs b/lib/codegen/src/isa/arm64/mod.rs index 240e92883d..50a43b13fc 100644 --- a/lib/codegen/src/isa/arm64/mod.rs +++ b/lib/codegen/src/isa/arm64/mod.rs @@ -15,26 +15,31 @@ use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use regalloc; use std::boxed::Box; use std::fmt; +use target_lexicon::Triple; #[allow(dead_code)] struct Isa { + triple: Triple, shared_flags: shared_settings::Flags, isa_flags: settings::Flags, } /// Get an ISA builder for creating ARM64 targets. -pub fn isa_builder() -> IsaBuilder { +pub fn isa_builder(triple: Triple) -> IsaBuilder { IsaBuilder { + triple, setup: settings::builder(), constructor: isa_constructor, } } fn isa_constructor( + triple: Triple, shared_flags: shared_settings::Flags, builder: shared_settings::Builder, ) -> Box { Box::new(Isa { + triple, isa_flags: settings::Flags::new(&shared_flags, builder), shared_flags, }) @@ -45,6 +50,10 @@ impl TargetIsa for Isa { "arm64" } + fn triple(&self) -> &Triple { + &self.triple + } + fn flags(&self) -> &shared_settings::Flags { &self.shared_flags } diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs index d9f4851e68..fc7c166bac 100644 --- a/lib/codegen/src/isa/mod.rs +++ b/lib/codegen/src/isa/mod.rs @@ -20,13 +20,18 @@ //! appropriate for the requested ISA: //! //! ``` +//! # extern crate cretonne_codegen; +//! # #[macro_use] extern crate target_lexicon; +//! # fn main() { //! use cretonne_codegen::settings::{self, Configurable}; //! use cretonne_codegen::isa; +//! use std::str::FromStr; +//! use target_lexicon::Triple; //! //! let shared_builder = settings::builder(); //! let shared_flags = settings::Flags::new(shared_builder); //! -//! match isa::lookup("riscv") { +//! match isa::lookup(triple!("riscv32")) { //! Err(_) => { //! // The RISC-V target ISA is not available. //! } @@ -35,6 +40,7 @@ //! let isa = isa_builder.finish(shared_flags); //! } //! } +//! # } //! ``` //! //! The configured target ISA trait object is a `Box` which can be used for multiple @@ -55,6 +61,7 @@ use settings; use settings::CallConv; use std::boxed::Box; use std::fmt; +use target_lexicon::{Architecture, Triple}; use timing; #[cfg(build_riscv)] @@ -80,51 +87,61 @@ mod stack; macro_rules! isa_builder { ($module:ident, $name:ident) => {{ #[cfg($name)] - fn $name() -> Result { - Ok($module::isa_builder()) + fn $name(triple: Triple) -> Result { + Ok($module::isa_builder(triple)) }; #[cfg(not($name))] - fn $name() -> Result { + fn $name(_triple: Triple) -> Result { Err(LookupError::Unsupported) } - $name() + $name }}; } /// Look for a supported ISA with the given `name`. /// Return a builder that can create a corresponding `TargetIsa`. -pub fn lookup(name: &str) -> Result { - match name { - "riscv" => isa_builder!(riscv, build_riscv), - "x86" => isa_builder!(x86, build_x86), - "arm32" => isa_builder!(arm32, build_arm32), - "arm64" => isa_builder!(arm64, build_arm64), - _ => Err(LookupError::Unknown), +pub fn lookup(triple: Triple) -> Result { + match triple.architecture { + Architecture::Riscv32 | Architecture::Riscv64 => isa_builder!(riscv, build_riscv)(triple), + Architecture::I386 | Architecture::I586 | Architecture::I686 | Architecture::X86_64 => { + isa_builder!(x86, build_x86)(triple) + } + Architecture::Thumbv6m + | Architecture::Thumbv7em + | Architecture::Thumbv7m + | Architecture::Arm + | Architecture::Armv4t + | Architecture::Armv5te + | Architecture::Armv7 + | Architecture::Armv7s => isa_builder!(arm32, build_arm32)(triple), + Architecture::Aarch64 => isa_builder!(arm64, build_arm64)(triple), + _ => Err(LookupError::Unsupported), } } /// Describes reason for target lookup failure #[derive(PartialEq, Eq, Copy, Clone, Debug)] pub enum LookupError { - /// Unknown Target - Unknown, + /// Support for this target was disabled in the current build. + SupportDisabled, - /// Target known but not built and thus not supported + /// Support for this target has not yet been implemented. Unsupported, } /// Builder for a `TargetIsa`. /// Modify the ISA-specific settings before creating the `TargetIsa` trait object with `finish`. pub struct Builder { + triple: Triple, setup: settings::Builder, - constructor: fn(settings::Flags, settings::Builder) -> Box, + constructor: fn(Triple, settings::Flags, settings::Builder) -> Box, } impl Builder { /// Combine the ISA-specific settings with the provided ISA-independent settings and allocate a /// fully configured `TargetIsa` trait object. pub fn finish(self, shared_flags: settings::Flags) -> Box { - (self.constructor)(shared_flags, self.setup) + (self.constructor)(self.triple, shared_flags, self.setup) } } @@ -151,9 +168,17 @@ pub trait TargetIsa: fmt::Display { /// Get the name of this ISA. fn name(&self) -> &'static str; + /// Get the target triple that was used to make this trait object. + fn triple(&self) -> &Triple; + /// Get the ISA-independent flags that were used to make this trait object. fn flags(&self) -> &settings::Flags; + /// Get the pointer type of this ISA. + fn pointer_type(&self) -> ir::Type { + ir::Type::int(u16::from(self.triple().pointer_width().unwrap().bits())).unwrap() + } + /// Does the CPU implement scalar comparisons using a CPU flags register? fn uses_cpu_flags(&self) -> bool { false @@ -252,7 +277,7 @@ pub trait TargetIsa: fmt::Display { use ir::stackslot::{StackOffset, StackSize}; use stack_layout::layout_stack; - let word_size = if self.flags().is_64bit() { 8 } else { 4 }; + let word_size = StackSize::from(self.triple().pointer_width().unwrap().bytes()); // Account for the SpiderMonkey standard prologue pushes. if func.signature.call_conv == CallConv::Baldrdash { diff --git a/lib/codegen/src/isa/riscv/abi.rs b/lib/codegen/src/isa/riscv/abi.rs index f45dd31426..ed7230d2cb 100644 --- a/lib/codegen/src/isa/riscv/abi.rs +++ b/lib/codegen/src/isa/riscv/abi.rs @@ -11,12 +11,12 @@ use abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion}; use ir::{self, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, Type}; use isa::RegClass; use regalloc::RegisterSet; -use settings as shared_settings; use std::i32; +use target_lexicon::Triple; struct Args { - pointer_bits: u16, - pointer_bytes: u32, + pointer_bits: u8, + pointer_bytes: u8, pointer_type: Type, regs: u32, reg_limit: u32, @@ -24,11 +24,11 @@ struct Args { } impl Args { - fn new(bits: u16, enable_e: bool) -> Self { + fn new(bits: u8, enable_e: bool) -> Self { Self { pointer_bits: bits, - pointer_bytes: u32::from(bits) / 8, - pointer_type: Type::int(bits).unwrap(), + pointer_bytes: bits / 8, + pointer_type: Type::int(u16::from(bits)).unwrap(), regs: 0, reg_limit: if enable_e { 6 } else { 8 }, offset: 0, @@ -51,15 +51,15 @@ impl ArgAssigner for Args { } // Large integers and booleans are broken down to fit in a register. - if !ty.is_float() && ty.bits() > self.pointer_bits { + if !ty.is_float() && ty.bits() > u16::from(self.pointer_bits) { // Align registers and stack to a multiple of two pointers. self.regs = align(self.regs, 2); - self.offset = align(self.offset, 2 * self.pointer_bytes); + self.offset = align(self.offset, 2 * u32::from(self.pointer_bytes)); return ValueConversion::IntSplit.into(); } // Small integers are extended to the size of a pointer register. - if ty.is_int() && ty.bits() < self.pointer_bits { + if ty.is_int() && ty.bits() < u16::from(self.pointer_bits) { match arg.extension { ArgumentExtension::None => {} ArgumentExtension::Uext => return ValueConversion::Uext(self.pointer_type).into(), @@ -79,7 +79,7 @@ impl ArgAssigner for Args { } else { // Assign a stack location. let loc = ArgumentLoc::Stack(self.offset as i32); - self.offset += self.pointer_bytes; + self.offset += u32::from(self.pointer_bytes); debug_assert!(self.offset <= i32::MAX as u32); loc.into() } @@ -89,11 +89,11 @@ impl ArgAssigner for Args { /// Legalize `sig` for RISC-V. pub fn legalize_signature( sig: &mut ir::Signature, - flags: &shared_settings::Flags, + triple: &Triple, isa_flags: &settings::Flags, current: bool, ) { - let bits = if flags.is_64bit() { 64 } else { 32 }; + let bits = triple.pointer_width().unwrap().bits(); let mut args = Args::new(bits, isa_flags.enable_e()); legalize_args(&mut sig.params, &mut args); @@ -102,7 +102,7 @@ pub fn legalize_signature( legalize_args(&mut sig.returns, &mut rets); if current { - let ptr = Type::int(bits).unwrap(); + let ptr = Type::int(u16::from(bits)).unwrap(); // Add the link register as an argument and return value. // diff --git a/lib/codegen/src/isa/riscv/mod.rs b/lib/codegen/src/isa/riscv/mod.rs index f704a5e7ff..aff23390e5 100644 --- a/lib/codegen/src/isa/riscv/mod.rs +++ b/lib/codegen/src/isa/riscv/mod.rs @@ -15,32 +15,37 @@ use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use regalloc; use std::boxed::Box; use std::fmt; +use target_lexicon::{PointerWidth, Triple}; #[allow(dead_code)] struct Isa { + triple: Triple, shared_flags: shared_settings::Flags, isa_flags: settings::Flags, cpumode: &'static [shared_enc_tables::Level1Entry], } /// Get an ISA builder for creating RISC-V targets. -pub fn isa_builder() -> IsaBuilder { +pub fn isa_builder(triple: Triple) -> IsaBuilder { IsaBuilder { + triple, setup: settings::builder(), constructor: isa_constructor, } } fn isa_constructor( + triple: Triple, shared_flags: shared_settings::Flags, builder: shared_settings::Builder, ) -> Box { - let level1 = if shared_flags.is_64bit() { - &enc_tables::LEVEL1_RV64[..] - } else { - &enc_tables::LEVEL1_RV32[..] + let level1 = match triple.pointer_width().unwrap() { + PointerWidth::U16 => panic!("16-bit RISC-V unrecognized"), + PointerWidth::U32 => &enc_tables::LEVEL1_RV32[..], + PointerWidth::U64 => &enc_tables::LEVEL1_RV64[..], }; Box::new(Isa { + triple, isa_flags: settings::Flags::new(&shared_flags, builder), shared_flags, cpumode: level1, @@ -52,6 +57,10 @@ impl TargetIsa for Isa { "riscv" } + fn triple(&self) -> &Triple { + &self.triple + } + fn flags(&self) -> &shared_settings::Flags { &self.shared_flags } @@ -85,7 +94,7 @@ impl TargetIsa for Isa { } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { - abi::legalize_signature(sig, &self.shared_flags, &self.isa_flags, current) + abi::legalize_signature(sig, &self.triple, &self.isa_flags, current) } fn regclass_for_abi_type(&self, ty: ir::Type) -> RegClass { @@ -117,7 +126,9 @@ mod tests { use ir::{Function, InstructionData, Opcode}; use isa; use settings::{self, Configurable}; + use std::str::FromStr; use std::string::{String, ToString}; + use target_lexicon; fn encstr(isa: &isa::TargetIsa, enc: Result) -> String { match enc { @@ -128,10 +139,11 @@ mod tests { #[test] fn test_64bitenc() { - let mut shared_builder = settings::builder(); - shared_builder.enable("is_64bit").unwrap(); + let shared_builder = settings::builder(); let shared_flags = settings::Flags::new(shared_builder); - let isa = isa::lookup("riscv").unwrap().finish(shared_flags); + let isa = isa::lookup(triple!("riscv64")) + .unwrap() + .finish(shared_flags); let mut func = Function::new(); let ebb = func.dfg.make_ebb(); @@ -178,10 +190,11 @@ mod tests { // Same as above, but for RV32. #[test] fn test_32bitenc() { - let mut shared_builder = settings::builder(); - shared_builder.set("is_64bit", "false").unwrap(); + let shared_builder = settings::builder(); let shared_flags = settings::Flags::new(shared_builder); - let isa = isa::lookup("riscv").unwrap().finish(shared_flags); + let isa = isa::lookup(triple!("riscv32")) + .unwrap() + .finish(shared_flags); let mut func = Function::new(); let ebb = func.dfg.make_ebb(); @@ -232,13 +245,12 @@ mod tests { #[test] fn test_rv32m() { - let mut shared_builder = settings::builder(); - shared_builder.set("is_64bit", "false").unwrap(); + let shared_builder = settings::builder(); let shared_flags = settings::Flags::new(shared_builder); // Set the supports_m stting which in turn enables the use_m predicate that unlocks // encodings for imul. - let mut isa_builder = isa::lookup("riscv").unwrap(); + let mut isa_builder = isa::lookup(triple!("riscv32")).unwrap(); isa_builder.enable("supports_m").unwrap(); let isa = isa_builder.finish(shared_flags); diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs index e7511482ca..89c5c30549 100644 --- a/lib/codegen/src/isa/x86/abi.rs +++ b/lib/codegen/src/isa/x86/abi.rs @@ -11,10 +11,10 @@ use ir::{get_probestack_funcref, AbiParam, ArgumentExtension, ArgumentLoc, Argum use isa::{RegClass, RegUnit, TargetIsa}; use regalloc::RegisterSet; use result; -use settings as shared_settings; use settings::CallConv; use stack_layout::layout_stack; use std::i32; +use target_lexicon::{PointerWidth, Triple}; /// Argument registers for x86-64 static ARG_GPRS: [RU; 6] = [RU::rdi, RU::rsi, RU::rdx, RU::rcx, RU::r8, RU::r9]; @@ -29,8 +29,8 @@ static ARG_GPRS_WIN_FASTCALL_X64: [RU; 4] = [RU::rcx, RU::rdx, RU::r8, RU::r9]; static RET_GPRS_WIN_FASTCALL_X64: [RU; 1] = [RU::rax]; struct Args { - pointer_bytes: u32, - pointer_bits: u16, + pointer_bytes: u8, + pointer_bits: u8, pointer_type: ir::Type, gpr: &'static [RU], gpr_used: usize, @@ -41,7 +41,7 @@ struct Args { } impl Args { - fn new(bits: u16, gpr: &'static [RU], fpr_limit: usize, call_conv: CallConv) -> Self { + fn new(bits: u8, gpr: &'static [RU], fpr_limit: usize, call_conv: CallConv) -> Self { let offset = if let CallConv::WindowsFastcall = call_conv { // [1] "The caller is responsible for allocating space for parameters to the callee, // and must always allocate sufficient space to store four register parameters" @@ -51,9 +51,9 @@ impl Args { }; Self { - pointer_bytes: u32::from(bits) / 8, + pointer_bytes: bits / 8, pointer_bits: bits, - pointer_type: ir::Type::int(bits).unwrap(), + pointer_type: ir::Type::int(u16::from(bits)).unwrap(), gpr, gpr_used: 0, fpr_limit, @@ -75,12 +75,12 @@ impl ArgAssigner for Args { } // Large integers and booleans are broken down to fit in a register. - if !ty.is_float() && ty.bits() > self.pointer_bits { + if !ty.is_float() && ty.bits() > u16::from(self.pointer_bits) { return ValueConversion::IntSplit.into(); } // Small integers are extended to the size of a pointer register. - if ty.is_int() && ty.bits() < self.pointer_bits { + if ty.is_int() && ty.bits() < u16::from(self.pointer_bits) { match arg.extension { ArgumentExtension::None => {} ArgumentExtension::Uext => return ValueConversion::Uext(self.pointer_type).into(), @@ -122,27 +122,31 @@ impl ArgAssigner for Args { // Assign a stack location. let loc = ArgumentLoc::Stack(self.offset as i32); - self.offset += self.pointer_bytes; + self.offset += u32::from(self.pointer_bytes); debug_assert!(self.offset <= i32::MAX as u32); loc.into() } } /// Legalize `sig`. -pub fn legalize_signature(sig: &mut ir::Signature, flags: &shared_settings::Flags, _current: bool) { +pub fn legalize_signature(sig: &mut ir::Signature, triple: &Triple, _current: bool) { let bits; let mut args; - if flags.is_64bit() { - bits = 64; - args = if sig.call_conv == CallConv::WindowsFastcall { - Args::new(bits, &ARG_GPRS_WIN_FASTCALL_X64[..], 4, sig.call_conv) - } else { - Args::new(bits, &ARG_GPRS[..], 8, sig.call_conv) - }; - } else { - bits = 32; - args = Args::new(bits, &[], 0, sig.call_conv); + match triple.pointer_width().unwrap() { + PointerWidth::U16 => panic!(), + PointerWidth::U32 => { + bits = 32; + args = Args::new(bits, &[], 0, sig.call_conv); + } + PointerWidth::U64 => { + bits = 64; + args = if sig.call_conv == CallConv::WindowsFastcall { + Args::new(bits, &ARG_GPRS_WIN_FASTCALL_X64[..], 4, sig.call_conv) + } else { + Args::new(bits, &ARG_GPRS[..], 8, sig.call_conv) + }; + } } legalize_args(&mut sig.params, &mut args); @@ -167,13 +171,13 @@ pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass { } /// Get the set of allocatable registers for `func`. -pub fn allocatable_registers(_func: &ir::Function, flags: &shared_settings::Flags) -> RegisterSet { +pub fn allocatable_registers(_func: &ir::Function, triple: &Triple) -> RegisterSet { let mut regs = RegisterSet::new(); regs.take(GPR, RU::rsp as RegUnit); regs.take(GPR, RU::rbp as RegUnit); // 32-bit arch only has 8 registers. - if !flags.is_64bit() { + if triple.pointer_width().unwrap() != PointerWidth::U64 { for i in 8..16 { regs.take(GPR, GPR.unit(i)); regs.take(FPR, FPR.unit(i)); @@ -184,34 +188,36 @@ pub fn allocatable_registers(_func: &ir::Function, flags: &shared_settings::Flag } /// Get the set of callee-saved registers. -fn callee_saved_gprs(flags: &shared_settings::Flags) -> &'static [RU] { - if flags.is_64bit() { - if flags.call_conv() == CallConv::WindowsFastcall { - // "registers RBX, RBP, RDI, RSI, RSP, R12, R13, R14, R15 are considered nonvolatile - // and must be saved and restored by a function that uses them." - // as per https://msdn.microsoft.com/en-us/library/6t169e9c.aspx - // RSP & RSB are not listed below, since they are restored automatically during - // a function call. If that wasn't the case, function calls (RET) would not work. - &[ - RU::rbx, - RU::rdi, - RU::rsi, - RU::r12, - RU::r13, - RU::r14, - RU::r15, - ] - } else { - &[RU::rbx, RU::r12, RU::r13, RU::r14, RU::r15] +fn callee_saved_gprs(isa: &TargetIsa) -> &'static [RU] { + match isa.triple().pointer_width().unwrap() { + PointerWidth::U16 => panic!(), + PointerWidth::U32 => &[RU::rbx, RU::rsi, RU::rdi], + PointerWidth::U64 => { + if isa.flags().call_conv() == CallConv::WindowsFastcall { + // "registers RBX, RBP, RDI, RSI, RSP, R12, R13, R14, R15 are considered nonvolatile + // and must be saved and restored by a function that uses them." + // as per https://msdn.microsoft.com/en-us/library/6t169e9c.aspx + // RSP & RSB are not listed below, since they are restored automatically during + // a function call. If that wasn't the case, function calls (RET) would not work. + &[ + RU::rbx, + RU::rdi, + RU::rsi, + RU::r12, + RU::r13, + RU::r14, + RU::r15, + ] + } else { + &[RU::rbx, RU::r12, RU::r13, RU::r14, RU::r15] + } } - } else { - &[RU::rbx, RU::rsi, RU::rdi] } } -fn callee_saved_gprs_used(flags: &shared_settings::Flags, func: &ir::Function) -> RegisterSet { +fn callee_saved_gprs_used(isa: &TargetIsa, func: &ir::Function) -> RegisterSet { let mut all_callee_saved = RegisterSet::empty(); - for reg in callee_saved_gprs(flags) { + for reg in callee_saved_gprs(isa) { all_callee_saved.free(GPR, *reg as RegUnit); } @@ -271,7 +277,7 @@ pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> // Baldrdash on 32-bit x86 always aligns its stack pointer to 16 bytes. let stack_align = 16; - let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; + let word_size = StackSize::from(isa.triple().pointer_width().unwrap().bytes()); let bytes = StackSize::from(isa.flags().baldrdash_prologue_words()) * word_size; let mut ss = ir::StackSlotData::new(ir::StackSlotKind::IncomingArg, bytes); @@ -285,7 +291,7 @@ pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> /// Implementation of the fastcall-based Win64 calling convention described at [1] /// [1] https://msdn.microsoft.com/en-us/library/ms235286.aspx pub fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { - if !isa.flags().is_64bit() { + if isa.triple().pointer_width().unwrap() != PointerWidth::U64 { panic!("TODO: windows-fastcall: x86-32 not implemented yet"); } @@ -293,14 +299,10 @@ pub fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r // which are aligned to 16 bytes in order to aid performance" let stack_align = 16; - let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; - let reg_type = if isa.flags().is_64bit() { - ir::types::I64 - } else { - ir::types::I32 - }; + let word_size = isa.triple().pointer_width().unwrap().bytes() as usize; + let reg_type = isa.pointer_type(); - let csrs = callee_saved_gprs_used(isa.flags(), func); + let csrs = callee_saved_gprs_used(isa, func); // [1] "Space is allocated on the call stack as a shadow store for callees to save" // This shadow store contains the parameters which are passed through registers (ARG_GPRS) @@ -364,14 +366,11 @@ pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r // The original 32-bit x86 ELF ABI had a 4-byte aligned stack pointer, but // newer versions use a 16-byte aligned stack pointer. let stack_align = 16; - let word_size = if isa.flags().is_64bit() { 8 } else { 4 }; - let reg_type = if isa.flags().is_64bit() { - ir::types::I64 - } else { - ir::types::I32 - }; + let pointer_width = isa.triple().pointer_width().unwrap(); + let word_size = pointer_width.bytes() as usize; + let reg_type = ir::Type::int(u16::from(pointer_width.bits())).unwrap(); - let csrs = callee_saved_gprs_used(isa.flags(), func); + let csrs = callee_saved_gprs_used(isa, func); // The reserved stack area is composed of: // return address + frame pointer + all callee-saved registers @@ -463,7 +462,8 @@ fn insert_common_prologue( let callee = get_probestack_funcref(pos.func, reg_type, rax, isa); // Make the call. - let call = if !isa.flags().is_pic() && isa.flags().is_64bit() + let call = if !isa.flags().is_pic() + && isa.triple().pointer_width().unwrap() == PointerWidth::U64 && !pos.func.dfg.ext_funcs[callee].colocated { // 64-bit non-PIC non-colocated calls need to be legalized to call_indirect. diff --git a/lib/codegen/src/isa/x86/mod.rs b/lib/codegen/src/isa/x86/mod.rs index e77b7d5c78..6e2c035d38 100644 --- a/lib/codegen/src/isa/x86/mod.rs +++ b/lib/codegen/src/isa/x86/mod.rs @@ -16,33 +16,38 @@ use regalloc; use result; use std::boxed::Box; use std::fmt; +use target_lexicon::{PointerWidth, Triple}; use timing; #[allow(dead_code)] struct Isa { + triple: Triple, shared_flags: shared_settings::Flags, isa_flags: settings::Flags, cpumode: &'static [shared_enc_tables::Level1Entry], } /// Get an ISA builder for creating x86 targets. -pub fn isa_builder() -> IsaBuilder { +pub fn isa_builder(triple: Triple) -> IsaBuilder { IsaBuilder { + triple, setup: settings::builder(), constructor: isa_constructor, } } fn isa_constructor( + triple: Triple, shared_flags: shared_settings::Flags, builder: shared_settings::Builder, ) -> Box { - let level1 = if shared_flags.is_64bit() { - &enc_tables::LEVEL1_I64[..] - } else { - &enc_tables::LEVEL1_I32[..] + let level1 = match triple.pointer_width().unwrap() { + PointerWidth::U16 => unimplemented!("x86-16"), + PointerWidth::U32 => &enc_tables::LEVEL1_I32[..], + PointerWidth::U64 => &enc_tables::LEVEL1_I64[..], }; Box::new(Isa { + triple, isa_flags: settings::Flags::new(&shared_flags, builder), shared_flags, cpumode: level1, @@ -54,6 +59,10 @@ impl TargetIsa for Isa { "x86" } + fn triple(&self) -> &Triple { + &self.triple + } + fn flags(&self) -> &shared_settings::Flags { &self.shared_flags } @@ -95,7 +104,7 @@ impl TargetIsa for Isa { } fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) { - abi::legalize_signature(sig, &self.shared_flags, current) + abi::legalize_signature(sig, &self.triple, current) } fn regclass_for_abi_type(&self, ty: ir::Type) -> RegClass { @@ -103,7 +112,7 @@ impl TargetIsa for Isa { } fn allocatable_registers(&self, func: &ir::Function) -> regalloc::RegisterSet { - abi::allocatable_registers(func, &self.shared_flags) + abi::allocatable_registers(func, &self.triple) } fn emit_inst( diff --git a/lib/codegen/src/legalizer/call.rs b/lib/codegen/src/legalizer/call.rs index d2e0ca542e..365ba3988d 100644 --- a/lib/codegen/src/legalizer/call.rs +++ b/lib/codegen/src/legalizer/call.rs @@ -28,11 +28,7 @@ pub fn expand_call( _ => panic!("Wanted call: {}", func.dfg.display_inst(inst, None)), }; - let ptr_ty = if isa.flags().is_64bit() { - ir::types::I64 - } else { - ir::types::I32 - }; + let ptr_ty = isa.pointer_type(); let sig = func.dfg.ext_funcs[func_ref].signature; diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs index 95a6666b2d..b3d6d6c0ce 100644 --- a/lib/codegen/src/legalizer/mod.rs +++ b/lib/codegen/src/legalizer/mod.rs @@ -283,11 +283,7 @@ pub fn expand_stack_check( ir::InstructionData::UnaryGlobalVar { global_var, .. } => global_var, _ => panic!("Want stack_check: {}", func.dfg.display_inst(inst, isa)), }; - let ptr_ty = if isa.flags().is_64bit() { - ir::types::I64 - } else { - ir::types::I32 - }; + let ptr_ty = isa.pointer_type(); let mut pos = FuncCursor::new(func).at_inst(inst); pos.use_srcloc(inst); diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs index 05d781dcd7..cfcae5d344 100644 --- a/lib/codegen/src/lib.rs +++ b/lib/codegen/src/lib.rs @@ -52,6 +52,11 @@ extern crate alloc; extern crate failure; #[macro_use] extern crate failure_derive; +#[cfg(test)] +#[macro_use] +extern crate target_lexicon; +#[cfg(not(test))] +extern crate target_lexicon; pub use context::Context; pub use legalizer::legalize_function; diff --git a/lib/codegen/src/regalloc/pressure.rs b/lib/codegen/src/regalloc/pressure.rs index 355fce582e..a85bef28ca 100644 --- a/lib/codegen/src/regalloc/pressure.rs +++ b/lib/codegen/src/regalloc/pressure.rs @@ -275,6 +275,8 @@ mod tests { use regalloc::RegisterSet; use std::borrow::Borrow; use std::boxed::Box; + use std::str::FromStr; + use target_lexicon; // Make an arm32 `TargetIsa`, if possible. fn arm32() -> Option> { @@ -284,7 +286,9 @@ mod tests { let shared_builder = settings::builder(); let shared_flags = settings::Flags::new(shared_builder); - isa::lookup("arm32").ok().map(|b| b.finish(shared_flags)) + isa::lookup(triple!("arm")) + .ok() + .map(|b| b.finish(shared_flags)) } // Get a register class by name. diff --git a/lib/codegen/src/regalloc/solver.rs b/lib/codegen/src/regalloc/solver.rs index af935b2104..d5d000fa95 100644 --- a/lib/codegen/src/regalloc/solver.rs +++ b/lib/codegen/src/regalloc/solver.rs @@ -1134,6 +1134,8 @@ mod tests { use isa::{RegClass, RegInfo, RegUnit, TargetIsa}; use regalloc::RegisterSet; use std::boxed::Box; + use std::str::FromStr; + use target_lexicon; // Make an arm32 `TargetIsa`, if possible. fn arm32() -> Option> { @@ -1143,7 +1145,9 @@ mod tests { let shared_builder = settings::builder(); let shared_flags = settings::Flags::new(shared_builder); - isa::lookup("arm32").ok().map(|b| b.finish(shared_flags)) + isa::lookup(triple!("arm")) + .ok() + .map(|b| b.finish(shared_flags)) } // Get a register class by name. diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs index 85a3a46b2f..d66cbae7c1 100644 --- a/lib/codegen/src/settings.rs +++ b/lib/codegen/src/settings.rs @@ -361,13 +361,11 @@ mod tests { "[shared]\n\ opt_level = \"default\"\n\ enable_verifier = true\n\ - is_64bit = false\n\ call_conv = \"fast\"\n\ is_pic = false\n\ colocated_libcalls = false\n\ return_at_end = false\n\ avoid_div_traps = false\n\ - is_compressed = false\n\ enable_float = true\n\ enable_nan_canonicalization = false\n\ enable_simd = true\n\ diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index f1923ef09e..b09b16c688 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -11,9 +11,10 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.8.0" } cretonne-module = { path = "../module", version = "0.8.0" } -faerie = "0.3.0" +faerie = "0.4.0" goblin = "0.0.14" failure = "0.1.1" +target-lexicon = { version = "0.0.0" } [badges] maintenance = { status = "experimental" } diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index 571ed1db77..14b4729e26 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -9,7 +9,7 @@ use cretonne_module::{Backend, DataContext, DataDescription, Init, Linkage, Modu use faerie; use failure::Error; use std::fs::File; -use target; +use target_lexicon::BinaryFormat; use traps::{FaerieTrapManifest, FaerieTrapSink}; #[derive(Debug)] @@ -27,8 +27,7 @@ pub enum FaerieTrapCollection { pub struct FaerieBuilder { isa: Box, name: String, - format: container::Format, - faerie_target: faerie::Target, + format: BinaryFormat, collect_traps: FaerieTrapCollection, libcall_names: Box String>, } @@ -50,7 +49,7 @@ impl FaerieBuilder { pub fn new( isa: Box, name: String, - format: container::Format, + format: BinaryFormat, collect_traps: FaerieTrapCollection, libcall_names: Box String>, ) -> Result { @@ -59,12 +58,10 @@ impl FaerieBuilder { "faerie requires TargetIsa be PIC".to_owned(), )); } - let faerie_target = target::translate(&*isa)?; Ok(Self { isa, name, format, - faerie_target, collect_traps, libcall_names, }) @@ -92,7 +89,7 @@ impl FaerieBuilder { pub struct FaerieBackend { isa: Box, artifact: faerie::Artifact, - format: container::Format, + format: BinaryFormat, trap_manifest: Option, libcall_names: Box String>, } @@ -119,8 +116,8 @@ impl Backend for FaerieBackend { /// Create a new `FaerieBackend` using the given Cretonne target. fn new(builder: FaerieBuilder) -> Self { Self { + artifact: faerie::Artifact::new(builder.isa.triple().clone(), builder.name), isa: builder.isa, - artifact: faerie::Artifact::new(builder.faerie_target, builder.name), format: builder.format, trap_manifest: match builder.collect_traps { FaerieTrapCollection::Enabled => Some(FaerieTrapManifest::new()), @@ -290,7 +287,6 @@ impl Backend for FaerieBackend { fn finish(self) -> FaerieProduct { FaerieProduct { artifact: self.artifact, - format: self.format, trap_manifest: self.trap_manifest, } } @@ -305,8 +301,6 @@ pub struct FaerieProduct { /// Optional trap manifest. Contains `FaerieTrapManifest` when `FaerieBuilder.collect_traps` is /// set to `FaerieTrapCollection::Enabled`. pub trap_manifest: Option, - /// The format that the builder specified for output. - format: container::Format, } impl FaerieProduct { @@ -317,18 +311,12 @@ impl FaerieProduct { /// Call `emit` on the faerie `Artifact`, producing bytes in memory. pub fn emit(&self) -> Result, Error> { - match self.format { - container::Format::ELF => self.artifact.emit::(), - container::Format::MachO => self.artifact.emit::(), - } + self.artifact.emit() } /// Call `write` on the faerie `Artifact`, writing to a file. pub fn write(&self, sink: File) -> Result<(), Error> { - match self.format { - container::Format::ELF => self.artifact.write::(sink), - container::Format::MachO => self.artifact.write::(sink), - } + self.artifact.write(sink) } } @@ -358,7 +346,7 @@ fn translate_data_linkage(linkage: Linkage, writable: bool) -> faerie::Decl { } struct FaerieRelocSink<'a> { - format: container::Format, + format: BinaryFormat, artifact: &'a mut faerie::Artifact, name: &'a str, namespace: &'a ModuleNamespace<'a, FaerieBackend>, diff --git a/lib/faerie/src/container.rs b/lib/faerie/src/container.rs index c78f05878c..117482020c 100644 --- a/lib/faerie/src/container.rs +++ b/lib/faerie/src/container.rs @@ -1,6 +1,7 @@ //! Utilities for working with Faerie container formats. use cretonne_codegen::binemit::Reloc; +use target_lexicon::BinaryFormat; /// An object file format. #[derive(Copy, Clone, Debug, Eq, PartialEq)] @@ -13,9 +14,9 @@ pub enum Format { /// Translate from a Cretonne `Reloc` to a raw object-file-format-specific /// relocation code. -pub fn raw_relocation(reloc: Reloc, format: Format) -> u32 { +pub fn raw_relocation(reloc: Reloc, format: BinaryFormat) -> u32 { match format { - Format::ELF => { + BinaryFormat::Elf => { use goblin::elf; match reloc { Reloc::Abs4 => elf::reloc::R_X86_64_32, @@ -28,6 +29,7 @@ pub fn raw_relocation(reloc: Reloc, format: Format) -> u32 { _ => unimplemented!(), } } - Format::MachO => unimplemented!(), + BinaryFormat::Macho => unimplemented!("macho relocations"), + _ => unimplemented!("unsupported format"), } } diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs index 7d7bb27a0a..c3f196004c 100644 --- a/lib/faerie/src/lib.rs +++ b/lib/faerie/src/lib.rs @@ -19,10 +19,10 @@ extern crate cretonne_module; extern crate faerie; extern crate failure; extern crate goblin; +extern crate target_lexicon; mod backend; mod container; -mod target; pub mod traps; pub use backend::{FaerieBackend, FaerieBuilder, FaerieProduct, FaerieTrapCollection}; diff --git a/lib/faerie/src/target.rs b/lib/faerie/src/target.rs deleted file mode 100644 index d6e3749430..0000000000 --- a/lib/faerie/src/target.rs +++ /dev/null @@ -1,21 +0,0 @@ -use cretonne_codegen::isa; -use cretonne_module::ModuleError; -use faerie::Target; - -/// Translate from a Cretonne `TargetIsa` to a Faerie `Target`. -pub fn translate(isa: &isa::TargetIsa) -> Result { - let name = isa.name(); - match name { - "x86" => Ok(if isa.flags().is_64bit() { - Target::X86_64 - } else { - Target::X86 - }), - "arm32" => Ok(Target::ARMv7), - "arm64" => Ok(Target::ARM64), - _ => Err(ModuleError::Backend(format!( - "unsupported faerie isa: {}", - name - ))), - } -} diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index 6b696d2b33..ee6943432c 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -318,11 +318,7 @@ where /// Return then pointer type for the current target. pub fn pointer_type(&self) -> ir::types::Type { - if self.backend.isa().flags().is_64bit() { - ir::types::I64 - } else { - ir::types::I32 - } + self.backend.isa().pointer_type() } /// Create a new `Context` initialized for use with this `Module`. diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 17e2578339..4a6a4c1ccf 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -9,6 +9,7 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } +target-lexicon = { version = "0.0.0", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.1.0" diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index f9f6d4cc7a..9ab9df71c7 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -17,9 +17,11 @@ extern crate cretonne_codegen; #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] extern crate raw_cpuid; +extern crate target_lexicon; use cretonne_codegen::isa; use cretonne_codegen::settings::{self, Configurable}; +use target_lexicon::Triple; #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] use raw_cpuid::CpuId; @@ -38,25 +40,8 @@ pub fn builders() -> Result<(settings::Builder, isa::Builder), &'static str> { return Err("unrecognized environment"); } - if cfg!(target_pointer_width = "64") { - flag_builder.enable("is_64bit").unwrap(); - } else if !cfg!(target_pointer_width = "32") { - return Err("unrecognized pointer size"); - } - - // TODO: Add RISC-V support once Rust supports it. - let name = if cfg!(any(target_arch = "x86", target_arch = "x86_64")) { - "x86" - } else if cfg!(target_arch = "arm") { - "arm32" - } else if cfg!(target_arch = "aarch64") { - "arm64" - } else { - return Err("unrecognized architecture"); - }; - - let mut isa_builder = isa::lookup(name).map_err(|err| match err { - isa::LookupError::Unknown => panic!(), + let mut isa_builder = isa::lookup(Triple::host()).map_err(|err| match err { + isa::LookupError::SupportDisabled => "support for architecture disabled at compile time", isa::LookupError::Unsupported => "unsupported architecture", })?; diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index d37ed2bbcf..92734dc9dc 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -10,6 +10,7 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.8.0" } +target-lexicon = { version = "0.0.0" } [badges] maintenance = { status = "experimental" } diff --git a/lib/reader/src/error.rs b/lib/reader/src/error.rs index 965eba3cb8..ed9eeb3cd9 100644 --- a/lib/reader/src/error.rs +++ b/lib/reader/src/error.rs @@ -40,7 +40,7 @@ macro_rules! err { ( $loc:expr, $msg:expr ) => { Err($crate::Error { location: $loc.clone(), - message: String::from($msg), + message: $msg.to_string(), }) }; diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index 687abd1b85..1fbd10a607 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -16,6 +16,7 @@ )] extern crate cretonne_codegen; +extern crate target_lexicon; pub use error::{Error, Location, Result}; pub use isaspec::{parse_options, IsaSpec}; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 7c013cd857..0f64885147 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -22,12 +22,13 @@ use sourcemap::SourceMap; use std::mem; use std::str::FromStr; use std::{u16, u32}; +use target_lexicon::Triple; use testcommand::TestCommand; use testfile::{Comment, Details, TestFile}; /// Parse the entire `text` into a list of functions. /// -/// Any test commands or ISA declarations are ignored. +/// Any test commands or target declarations are ignored. pub fn parse_functions(text: &str) -> Result> { let _tt = timing::parse_text(); parse_test(text).map(|file| file.functions.into_iter().map(|(func, _)| func).collect()) @@ -43,7 +44,7 @@ pub fn parse_test(text: &str) -> Result { parser.start_gathering_comments(); let commands = parser.parse_test_commands(); - let isa_spec = parser.parse_isa_specs()?; + let isa_spec = parser.parse_target_specs()?; parser.token(); parser.claim_gathered_comments(AnyEntity::Function); @@ -88,10 +89,10 @@ struct Context<'a> { /// Aliases to resolve once value definitions are known. aliases: Vec, - /// Reference to the unique_isa for things like parsing ISA-specific instruction encoding + /// Reference to the unique_isa for things like parsing target-specific instruction encoding /// information. This is only `Some` if exactly one set of `isa` directives were found in the - /// prologue (it is valid to have directives for multiple different ISAs, but in that case we - /// couldn't know which ISA the provided encodings are intended for) + /// prologue (it is valid to have directives for multiple different targets, but in that case we + /// couldn't know which target the provided encodings are intended for) unique_isa: Option<&'a TargetIsa>, } @@ -667,17 +668,17 @@ impl<'a> Parser<'a> { list } - /// Parse a list of ISA specs. + /// Parse a list of target specs. /// - /// Accept a mix of `isa` and `set` command lines. The `set` commands are cumulative. + /// Accept a mix of `target` and `set` command lines. The `set` commands are cumulative. /// - pub fn parse_isa_specs(&mut self) -> Result { - // Was there any `isa` commands? - let mut seen_isa = false; - // Location of last `set` command since the last `isa`. + pub fn parse_target_specs(&mut self) -> Result { + // Was there any `target` commands? + let mut seen_target = false; + // Location of last `set` command since the last `target`. let mut last_set_loc = None; - let mut isas = Vec::new(); + let mut targets = Vec::new(); let mut flag_builder = settings::builder(); while let Some(Token::Identifier(command)) = self.token() { @@ -690,38 +691,42 @@ impl<'a> Parser<'a> { &self.loc, )?; } - "isa" => { + "target" => { let loc = self.loc; // Grab the whole line so the lexer won't go looking for tokens on the // following lines. let mut words = self.consume_line().trim().split_whitespace(); - // Look for `isa foo`. - let isa_name = match words.next() { - None => return err!(loc, "expected ISA name"), + // Look for `target foo`. + let target_name = match words.next() { Some(w) => w, + None => return err!(loc, "expected target triple"), }; - let mut isa_builder = match isa::lookup(isa_name) { - Err(isa::LookupError::Unknown) => { - return err!(loc, "unknown ISA '{}'", isa_name) + let triple = match Triple::from_str(target_name) { + Ok(triple) => triple, + Err(err) => return err!(loc, err), + }; + let mut isa_builder = match isa::lookup(triple) { + Err(isa::LookupError::SupportDisabled) => { + continue; } Err(isa::LookupError::Unsupported) => { - continue; + return err!(loc, "unsupported target '{}'", target_name) } Ok(b) => b, }; last_set_loc = None; - seen_isa = true; - // Apply the ISA-specific settings to `isa_builder`. + seen_target = true; + // Apply the target-specific settings to `isa_builder`. isaspec::parse_options(words, &mut isa_builder, &self.loc)?; // Construct a trait object with the aggregate settings. - isas.push(isa_builder.finish(settings::Flags::new(flag_builder.clone()))); + targets.push(isa_builder.finish(settings::Flags::new(flag_builder.clone()))); } _ => break, } } - if !seen_isa { - // No `isa` commands, but we allow for `set` commands. + if !seen_target { + // No `target` commands, but we allow for `set` commands. Ok(isaspec::IsaSpec::None(settings::Flags::new(flag_builder))) } else if let Some(loc) = last_set_loc { err!( @@ -729,7 +734,7 @@ impl<'a> Parser<'a> { "dangling 'set' command after ISA specification has no effect." ) } else { - Ok(isaspec::IsaSpec::Some(isas)) + Ok(isaspec::IsaSpec::Some(targets)) } } @@ -2466,14 +2471,14 @@ mod tests { fn isa_spec() { assert!( parse_test( - "isa + "target function %foo() system_v {}", ).is_err() ); assert!( parse_test( - "isa riscv + "target riscv32 set enable_float=false function %foo() system_v {}", ).is_err() diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index d672c1dc63..fb4c1d8e1a 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -15,6 +15,7 @@ cretonne-native = { path = "../native", version = "0.8.0", default-features = fa region = "0.2.0" libc = { version = "0.2.40", default-features = false } errno = "0.2.3" +target-lexicon = { version = "0.0.0", default-features = false } [target.'cfg(target_os = "windows")'.dependencies] winapi = { version = "0.3", features = ["winbase", "memoryapi"] } diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs index 873578e708..889601a819 100644 --- a/lib/simplejit/src/backend.rs +++ b/lib/simplejit/src/backend.rs @@ -10,6 +10,7 @@ use libc; use memory::Memory; use std::ffi::CString; use std::ptr; +use target_lexicon::PointerWidth; #[cfg(windows)] use winapi; @@ -173,10 +174,10 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { } } - let reloc = if self.isa.flags().is_64bit() { - Reloc::Abs8 - } else { - Reloc::Abs4 + let reloc = match self.isa.triple().pointer_width().unwrap() { + PointerWidth::U16 => panic!(), + PointerWidth::U32 => Reloc::Abs4, + PointerWidth::U64 => Reloc::Abs8, }; let mut relocs = Vec::new(); for &(offset, id) in function_relocs { diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs index 3212447065..0e908dd414 100644 --- a/lib/simplejit/src/lib.rs +++ b/lib/simplejit/src/lib.rs @@ -18,6 +18,7 @@ extern crate cretonne_native; extern crate errno; extern crate libc; extern crate region; +extern crate target_lexicon; #[cfg(target_os = "windows")] extern crate winapi; diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 3882a850cb..a0e580924e 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -15,6 +15,7 @@ cretonne-frontend = { path = "../frontend", version = "0.8.0", default-features hashmap_core = { version = "0.1.4", optional = true } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } +target-lexicon = { version = "0.0.0", default-features = false } [dev-dependencies] wabt = "0.3" diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 8eae69fc2b..4794524b5a 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -8,6 +8,7 @@ use environ::{FuncEnvironment, GlobalValue, ModuleEnvironment, WasmResult}; use func_translator::FuncTranslator; use std::string::String; use std::vec::Vec; +use target_lexicon::Triple; use translation_utils::{FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex}; use wasmparser; @@ -39,6 +40,9 @@ impl Exportable { /// `DummyEnvironment` to allow it to be borrowed separately from the /// `FuncTranslator` field. pub struct DummyModuleInfo { + /// Target description. + pub triple: Triple, + /// Compilation setting flags. pub flags: settings::Flags, @@ -69,8 +73,9 @@ pub struct DummyModuleInfo { impl DummyModuleInfo { /// Allocates the data structures with the given flags. - pub fn with_flags(flags: settings::Flags) -> Self { + pub fn with_triple_flags(triple: Triple, flags: settings::Flags) -> Self { Self { + triple, flags, signatures: Vec::new(), imported_funcs: Vec::new(), @@ -100,14 +105,14 @@ pub struct DummyEnvironment { impl DummyEnvironment { /// Allocates the data structures with default flags. - pub fn default() -> Self { - Self::with_flags(settings::Flags::new(settings::builder())) + pub fn with_triple(triple: Triple) -> Self { + Self::with_triple_flags(triple, settings::Flags::new(settings::builder())) } /// Allocates the data structures with the given flags. - pub fn with_flags(flags: settings::Flags) -> Self { + pub fn with_triple_flags(triple: Triple, flags: settings::Flags) -> Self { Self { - info: DummyModuleInfo::with_flags(flags), + info: DummyModuleInfo::with_triple_flags(triple, flags), trans: FuncTranslator::new(), func_bytecode_sizes: Vec::new(), } @@ -143,6 +148,10 @@ impl<'dummy_environment> DummyFuncEnvironment<'dummy_environment> { } impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environment> { + fn triple(&self) -> &Triple { + &self.mod_info.triple + } + fn flags(&self) -> &settings::Flags { &self.mod_info.flags } diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 5b26ca785f..5b4ca58301 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -4,6 +4,7 @@ use cretonne_codegen::cursor::FuncCursor; use cretonne_codegen::ir::{self, InstBuilder}; use cretonne_codegen::settings::Flags; use std::vec::Vec; +use target_lexicon::Triple; use translation_utils::{FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex}; use wasmparser::BinaryReaderError; @@ -74,6 +75,9 @@ pub type WasmResult = Result; /// IR. The function environment provides information about the WebAssembly module as well as the /// runtime environment. pub trait FuncEnvironment { + /// Get the triple for the current compilation. + fn triple(&self) -> &Triple; + /// Get the flags for the current compilation. fn flags(&self) -> &Flags; @@ -81,11 +85,7 @@ pub trait FuncEnvironment { /// /// This returns `I64` for 64-bit architectures and `I32` for 32-bit architectures. fn native_pointer(&self) -> ir::Type { - if self.flags().is_64bit() { - ir::types::I64 - } else { - ir::types::I32 - } + ir::Type::int(u16::from(self.triple().pointer_width().unwrap().bits())).unwrap() } /// Set up the necessary preamble definitions in `func` to access the global variable diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index fbac424fef..9f93904381 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -237,6 +237,7 @@ mod tests { use cretonne_codegen::ir::types::I32; use cretonne_codegen::{ir, Context}; use environ::{DummyEnvironment, FuncEnvironment}; + use target_lexicon::Triple; #[test] fn small1() { @@ -256,7 +257,7 @@ mod tests { ]; let mut trans = FuncTranslator::new(); - let runtime = DummyEnvironment::default(); + let runtime = DummyEnvironment::with_triple(Triple::default()); let mut ctx = Context::new(); ctx.func.name = ir::ExternalName::testcase("small1"); @@ -289,7 +290,7 @@ mod tests { ]; let mut trans = FuncTranslator::new(); - let runtime = DummyEnvironment::default(); + let runtime = DummyEnvironment::with_triple(Triple::default()); let mut ctx = Context::new(); ctx.func.name = ir::ExternalName::testcase("small2"); @@ -335,7 +336,7 @@ mod tests { ]; let mut trans = FuncTranslator::new(); - let runtime = DummyEnvironment::default(); + let runtime = DummyEnvironment::with_triple(Triple::default()); let mut ctx = Context::new(); ctx.func.name = ir::ExternalName::testcase("infloop"); diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 2c73f73953..a7ceeee389 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -34,6 +34,7 @@ extern crate hashmap_core; #[macro_use(dbg)] extern crate cretonne_codegen; extern crate cretonne_frontend; +extern crate target_lexicon; extern crate wasmparser; extern crate failure; diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs index a5d5badaed..2418193f34 100644 --- a/lib/wasm/tests/wasm_testsuite.rs +++ b/lib/wasm/tests/wasm_testsuite.rs @@ -1,5 +1,7 @@ extern crate cretonne_codegen; extern crate cretonne_wasm; +#[macro_use] +extern crate target_lexicon; extern crate wabt; use cretonne_codegen::print_errors::pretty_verifier_error; @@ -11,6 +13,7 @@ use std::fs::File; use std::io; use std::io::prelude::*; use std::path::Path; +use std::str::FromStr; use wabt::wat2wasm; #[test] @@ -70,7 +73,7 @@ fn handle_module(path: &Path, flags: &Flags) { None | Some(&_) => panic!("the file extension for {:?} is not wasm or wat", path), }, }; - let mut dummy_environ = DummyEnvironment::with_flags(flags.clone()); + let mut dummy_environ = DummyEnvironment::with_triple_flags(triple!("riscv64"), flags.clone()); translate_module(&data, &mut dummy_environ).unwrap(); for func in &dummy_environ.info.function_bodies { verifier::verify_function(func, flags) From f48c1d4f89ecd41203f90ae422611b9b9af65d84 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 30 May 2018 17:25:48 -0700 Subject: [PATCH 1804/3084] Update to target-lexicon 0.0.1 and faerie 0.4.1. This fixes handling of custom targets. --- cranelift/Cargo.toml | 2 +- cranelift/fuzz/Cargo.toml | 2 +- lib/codegen/Cargo.toml | 2 +- lib/faerie/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 2 +- lib/reader/Cargo.toml | 2 +- lib/simplejit/Cargo.toml | 2 +- lib/wasm/Cargo.toml | 2 +- 8 files changed, 9 insertions(+), 9 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index df16b8cccb..57193f7af9 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -31,7 +31,7 @@ serde_derive = "1.0.8" term = "0.5.1" capstone = "0.3.1" wabt = { version = "0.3", optional = true } -target-lexicon = "0.0.0" +target-lexicon = "0.0.1" [features] default = ["wasm"] diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml index f1169e2763..a5bbdbd912 100644 --- a/cranelift/fuzz/Cargo.toml +++ b/cranelift/fuzz/Cargo.toml @@ -21,7 +21,7 @@ git = "https://github.com/rust-fuzz/libfuzzer-sys.git" path = "../lib/wasm" [dependencies.target-lexicon] -version = "0.0.0" +version = "0.0.1" # Prevent this from interfering with workspaces [workspace] diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 0e7b5f32ea..e593e7a448 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -15,7 +15,7 @@ cretonne-entity = { path = "../entity", version = "0.8.0", default-features = fa failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } hashmap_core = { version = "0.1.4", optional = true } -target-lexicon = { version = "0.0.0", default-features = false } +target-lexicon = { version = "0.0.1", default-features = false } # It is a goal of the cretonne-codegen crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index b09b16c688..bec2b0959c 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -11,10 +11,10 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.8.0" } cretonne-module = { path = "../module", version = "0.8.0" } -faerie = "0.4.0" +faerie = "0.4.1" goblin = "0.0.14" failure = "0.1.1" -target-lexicon = { version = "0.0.0" } +target-lexicon = "0.0.1" [badges] maintenance = { status = "experimental" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 4a6a4c1ccf..87201854f0 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -9,7 +9,7 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } -target-lexicon = { version = "0.0.0", default-features = false } +target-lexicon = { version = "0.0.1", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.1.0" diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 92734dc9dc..c755c13855 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -10,7 +10,7 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.8.0" } -target-lexicon = { version = "0.0.0" } +target-lexicon = "0.0.1" [badges] maintenance = { status = "experimental" } diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index fb4c1d8e1a..ef551a3e79 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -15,7 +15,7 @@ cretonne-native = { path = "../native", version = "0.8.0", default-features = fa region = "0.2.0" libc = { version = "0.2.40", default-features = false } errno = "0.2.3" -target-lexicon = { version = "0.0.0", default-features = false } +target-lexicon = { version = "0.0.1", default-features = false } [target.'cfg(target_os = "windows")'.dependencies] winapi = { version = "0.3", features = ["winbase", "memoryapi"] } diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index a0e580924e..0aff1caa40 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -15,7 +15,7 @@ cretonne-frontend = { path = "../frontend", version = "0.8.0", default-features hashmap_core = { version = "0.1.4", optional = true } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } -target-lexicon = { version = "0.0.0", default-features = false } +target-lexicon = { version = "0.0.1", default-features = false } [dev-dependencies] wabt = "0.3" From 35f2cae00c23d264a41d6f9a954d38c7c3a45cbc Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 May 2018 09:32:02 -0700 Subject: [PATCH 1805/3084] Add comments explaining the rustfmt installation dance. --- .travis.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.travis.yml b/.travis.yml index b213bd1010..26e34b45d2 100644 --- a/.travis.yml +++ b/.travis.yml @@ -17,13 +17,20 @@ install: - pip3 install --user --upgrade mypy flake8 - mypy --version before_script: + # If an old version of rustfmt from cargo is already installed, uninstall + # it, since it can prevent the installation of the new version from rustup. - cargo uninstall rustfmt || true - cargo install --list + # If we're testing beta or nightly, we still need to install the stable + # toolchain so that we can run the stable version of rustfmt. - rustup toolchain install stable + # Install the stable version of rustfmt. - rustup component add --toolchain=stable rustfmt-preview - rustup component list --toolchain=stable - rustup show - rustfmt +stable --version || echo fail + # Sometimes the component isn't actually ready after being installed, and + # rustup update makes it ready. - rustup update - rustfmt +stable --version script: ./test-all.sh From 6c003d68cd70f1885faf2312fbd0aa36266e2b3b Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 31 May 2018 11:21:26 -0700 Subject: [PATCH 1806/3084] Factor out pointer_bits() and pointer_bytes() helper functions. --- lib/codegen/src/isa/mod.rs | 14 ++++++++++++-- lib/codegen/src/isa/x86/abi.rs | 4 ++-- 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs index fc7c166bac..b68f9fecb7 100644 --- a/lib/codegen/src/isa/mod.rs +++ b/lib/codegen/src/isa/mod.rs @@ -176,7 +176,17 @@ pub trait TargetIsa: fmt::Display { /// Get the pointer type of this ISA. fn pointer_type(&self) -> ir::Type { - ir::Type::int(u16::from(self.triple().pointer_width().unwrap().bits())).unwrap() + ir::Type::int(u16::from(self.pointer_bits())).unwrap() + } + + /// Get the width of pointers on this ISA, in units of bits. + fn pointer_bits(&self) -> u8 { + self.triple().pointer_width().unwrap().bits() + } + + /// Get the width of pointers on this ISA, in units of bytes. + fn pointer_bytes(&self) -> u8 { + self.triple().pointer_width().unwrap().bytes() } /// Does the CPU implement scalar comparisons using a CPU flags register? @@ -277,7 +287,7 @@ pub trait TargetIsa: fmt::Display { use ir::stackslot::{StackOffset, StackSize}; use stack_layout::layout_stack; - let word_size = StackSize::from(self.triple().pointer_width().unwrap().bytes()); + let word_size = StackSize::from(self.pointer_bytes()); // Account for the SpiderMonkey standard prologue pushes. if func.signature.call_conv == CallConv::Baldrdash { diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs index 89c5c30549..d10f406ab3 100644 --- a/lib/codegen/src/isa/x86/abi.rs +++ b/lib/codegen/src/isa/x86/abi.rs @@ -277,7 +277,7 @@ pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> // Baldrdash on 32-bit x86 always aligns its stack pointer to 16 bytes. let stack_align = 16; - let word_size = StackSize::from(isa.triple().pointer_width().unwrap().bytes()); + let word_size = StackSize::from(isa.pointer_bytes()); let bytes = StackSize::from(isa.flags().baldrdash_prologue_words()) * word_size; let mut ss = ir::StackSlotData::new(ir::StackSlotKind::IncomingArg, bytes); @@ -299,7 +299,7 @@ pub fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r // which are aligned to 16 bytes in order to aid performance" let stack_align = 16; - let word_size = isa.triple().pointer_width().unwrap().bytes() as usize; + let word_size = isa.pointer_bytes() as usize; let reg_type = isa.pointer_type(); let csrs = callee_saved_gprs_used(isa, func); From ae8fc69406dccef8443986b5a965d21ea59cb164 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 4 Jun 2018 16:17:02 -0700 Subject: [PATCH 1807/3084] Use `&'static str` rather than Cow for pass names. --- lib/filetests/src/subtest.rs | 2 +- lib/filetests/src/test_binemit.rs | 4 ++-- lib/filetests/src/test_cat.rs | 4 ++-- lib/filetests/src/test_compile.rs | 4 ++-- lib/filetests/src/test_dce.rs | 4 ++-- lib/filetests/src/test_domtree.rs | 4 ++-- lib/filetests/src/test_legalizer.rs | 4 ++-- lib/filetests/src/test_licm.rs | 4 ++-- lib/filetests/src/test_postopt.rs | 4 ++-- lib/filetests/src/test_preopt.rs | 4 ++-- lib/filetests/src/test_print_cfg.rs | 4 ++-- lib/filetests/src/test_regalloc.rs | 4 ++-- lib/filetests/src/test_simple_gvn.rs | 4 ++-- lib/filetests/src/test_verifier.rs | 4 ++-- 14 files changed, 27 insertions(+), 27 deletions(-) diff --git a/lib/filetests/src/subtest.rs b/lib/filetests/src/subtest.rs index 18904d99b9..951aa76e56 100644 --- a/lib/filetests/src/subtest.rs +++ b/lib/filetests/src/subtest.rs @@ -45,7 +45,7 @@ impl<'a> Context<'a> { /// trait object. pub trait SubTest { /// Name identifying this subtest. Typically the same as the test command. - fn name(&self) -> Cow; + fn name(&self) -> &'static str; /// Should the verifier be run on the function before running the test? fn needs_verifier(&self) -> bool { diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index f6a91f61a1..d736faceee 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -96,8 +96,8 @@ impl binemit::CodeSink for TextSink { } impl SubTest for TestBinEmit { - fn name(&self) -> Cow { - Cow::from("binemit") + fn name(&self) -> &'static str { + "binemit" } fn is_mutating(&self) -> bool { diff --git a/lib/filetests/src/test_cat.rs b/lib/filetests/src/test_cat.rs index 4fdedf115f..363567695f 100644 --- a/lib/filetests/src/test_cat.rs +++ b/lib/filetests/src/test_cat.rs @@ -23,8 +23,8 @@ pub fn subtest(parsed: &TestCommand) -> STResult> { } impl SubTest for TestCat { - fn name(&self) -> Cow { - Cow::from("cat") + fn name(&self) -> &'static str { + "cat" } fn needs_verifier(&self) -> bool { diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs index f9b5deebd8..ac326cb762 100644 --- a/lib/filetests/src/test_compile.rs +++ b/lib/filetests/src/test_compile.rs @@ -22,8 +22,8 @@ pub fn subtest(parsed: &TestCommand) -> Result> { } impl SubTest for TestCompile { - fn name(&self) -> Cow { - Cow::from("compile") + fn name(&self) -> &'static str { + "compile" } fn is_mutating(&self) -> bool { diff --git a/lib/filetests/src/test_dce.rs b/lib/filetests/src/test_dce.rs index 13bad80a7f..741ca1aac5 100644 --- a/lib/filetests/src/test_dce.rs +++ b/lib/filetests/src/test_dce.rs @@ -25,8 +25,8 @@ pub fn subtest(parsed: &TestCommand) -> Result> { } impl SubTest for TestDCE { - fn name(&self) -> Cow { - Cow::from("dce") + fn name(&self) -> &'static str { + "dce" } fn is_mutating(&self) -> bool { diff --git a/lib/filetests/src/test_domtree.rs b/lib/filetests/src/test_domtree.rs index 9dbd5fe849..a262293567 100644 --- a/lib/filetests/src/test_domtree.rs +++ b/lib/filetests/src/test_domtree.rs @@ -36,8 +36,8 @@ pub fn subtest(parsed: &TestCommand) -> Result> { } impl SubTest for TestDomtree { - fn name(&self) -> Cow { - Cow::from("domtree") + fn name(&self) -> &'static str { + "domtree" } // Extract our own dominator tree from diff --git a/lib/filetests/src/test_legalizer.rs b/lib/filetests/src/test_legalizer.rs index d86720612e..c3d84861ab 100644 --- a/lib/filetests/src/test_legalizer.rs +++ b/lib/filetests/src/test_legalizer.rs @@ -23,8 +23,8 @@ pub fn subtest(parsed: &TestCommand) -> Result> { } impl SubTest for TestLegalizer { - fn name(&self) -> Cow { - Cow::from("legalizer") + fn name(&self) -> &'static str { + "legalizer" } fn is_mutating(&self) -> bool { diff --git a/lib/filetests/src/test_licm.rs b/lib/filetests/src/test_licm.rs index a125da484f..e3d6812f91 100644 --- a/lib/filetests/src/test_licm.rs +++ b/lib/filetests/src/test_licm.rs @@ -25,8 +25,8 @@ pub fn subtest(parsed: &TestCommand) -> Result> { } impl SubTest for TestLICM { - fn name(&self) -> Cow { - Cow::from("licm") + fn name(&self) -> &'static str { + "licm" } fn is_mutating(&self) -> bool { diff --git a/lib/filetests/src/test_postopt.rs b/lib/filetests/src/test_postopt.rs index 2813a1562c..192a89c8b3 100644 --- a/lib/filetests/src/test_postopt.rs +++ b/lib/filetests/src/test_postopt.rs @@ -22,8 +22,8 @@ pub fn subtest(parsed: &TestCommand) -> Result> { } impl SubTest for TestPostopt { - fn name(&self) -> Cow { - Cow::from("postopt") + fn name(&self) -> &'static str { + "postopt" } fn is_mutating(&self) -> bool { diff --git a/lib/filetests/src/test_preopt.rs b/lib/filetests/src/test_preopt.rs index 7d5609b6a0..dbd68e8bac 100644 --- a/lib/filetests/src/test_preopt.rs +++ b/lib/filetests/src/test_preopt.rs @@ -22,8 +22,8 @@ pub fn subtest(parsed: &TestCommand) -> Result> { } impl SubTest for TestPreopt { - fn name(&self) -> Cow { - Cow::from("preopt") + fn name(&self) -> &'static str { + "preopt" } fn is_mutating(&self) -> bool { diff --git a/lib/filetests/src/test_print_cfg.rs b/lib/filetests/src/test_print_cfg.rs index 3f48b1d4e9..741dae762b 100644 --- a/lib/filetests/src/test_print_cfg.rs +++ b/lib/filetests/src/test_print_cfg.rs @@ -23,8 +23,8 @@ pub fn subtest(parsed: &TestCommand) -> STResult> { } impl SubTest for TestPrintCfg { - fn name(&self) -> Cow { - Cow::from("print-cfg") + fn name(&self) -> &'static str { + "print-cfg" } fn needs_verifier(&self) -> bool { diff --git a/lib/filetests/src/test_regalloc.rs b/lib/filetests/src/test_regalloc.rs index 3ff5134ca4..91217899e1 100644 --- a/lib/filetests/src/test_regalloc.rs +++ b/lib/filetests/src/test_regalloc.rs @@ -25,8 +25,8 @@ pub fn subtest(parsed: &TestCommand) -> Result> { } impl SubTest for TestRegalloc { - fn name(&self) -> Cow { - Cow::from("regalloc") + fn name(&self) -> &'static str { + "regalloc" } fn is_mutating(&self) -> bool { diff --git a/lib/filetests/src/test_simple_gvn.rs b/lib/filetests/src/test_simple_gvn.rs index 54ccab6d5b..2036301bcc 100644 --- a/lib/filetests/src/test_simple_gvn.rs +++ b/lib/filetests/src/test_simple_gvn.rs @@ -25,8 +25,8 @@ pub fn subtest(parsed: &TestCommand) -> Result> { } impl SubTest for TestSimpleGVN { - fn name(&self) -> Cow { - Cow::from("simple-gvn") + fn name(&self) -> &'static str { + "simple-gvn" } fn is_mutating(&self) -> bool { diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs index 7b8c606ced..69e56b1cd5 100644 --- a/lib/filetests/src/test_verifier.rs +++ b/lib/filetests/src/test_verifier.rs @@ -28,8 +28,8 @@ pub fn subtest(parsed: &TestCommand) -> Result> { } impl SubTest for TestVerifier { - fn name(&self) -> Cow { - Cow::from("verifier") + fn name(&self) -> &'static str { + "verifier" } fn needs_verifier(&self) -> bool { From 22a7c561082bfea7edd4e070b6c8142fd31405dc Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 4 Jun 2018 16:21:00 -0700 Subject: [PATCH 1808/3084] Use `Context::for_function` to simplify the code. --- lib/filetests/src/test_compile.rs | 5 +---- lib/filetests/src/test_dce.rs | 4 +--- lib/filetests/src/test_legalizer.rs | 3 +-- lib/filetests/src/test_licm.rs | 4 +--- lib/filetests/src/test_postopt.rs | 4 +--- lib/filetests/src/test_preopt.rs | 4 +--- lib/filetests/src/test_regalloc.rs | 5 +---- lib/filetests/src/test_simple_gvn.rs | 4 +--- 8 files changed, 8 insertions(+), 25 deletions(-) diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs index ac326cb762..8cb26b4f1f 100644 --- a/lib/filetests/src/test_compile.rs +++ b/lib/filetests/src/test_compile.rs @@ -36,10 +36,7 @@ impl SubTest for TestCompile { fn run(&self, func: Cow, context: &Context) -> Result<()> { let isa = context.isa.expect("compile needs an ISA"); - - // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne_codegen::Context::new(); - comp_ctx.func = func.into_owned(); + let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); let code_size = comp_ctx .compile(isa) diff --git a/lib/filetests/src/test_dce.rs b/lib/filetests/src/test_dce.rs index 741ca1aac5..38e2b907a9 100644 --- a/lib/filetests/src/test_dce.rs +++ b/lib/filetests/src/test_dce.rs @@ -34,9 +34,7 @@ impl SubTest for TestDCE { } fn run(&self, func: Cow, context: &Context) -> Result<()> { - // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne_codegen::Context::new(); - comp_ctx.func = func.into_owned(); + let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); comp_ctx.flowgraph(); comp_ctx.compute_loop_analysis(); diff --git a/lib/filetests/src/test_legalizer.rs b/lib/filetests/src/test_legalizer.rs index c3d84861ab..8ce1cffb1b 100644 --- a/lib/filetests/src/test_legalizer.rs +++ b/lib/filetests/src/test_legalizer.rs @@ -36,8 +36,7 @@ impl SubTest for TestLegalizer { } fn run(&self, func: Cow, context: &Context) -> Result<()> { - let mut comp_ctx = cretonne_codegen::Context::new(); - comp_ctx.func = func.into_owned(); + let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); let isa = context.isa.expect("legalizer needs an ISA"); comp_ctx.compute_cfg(); diff --git a/lib/filetests/src/test_licm.rs b/lib/filetests/src/test_licm.rs index e3d6812f91..2dda7b976b 100644 --- a/lib/filetests/src/test_licm.rs +++ b/lib/filetests/src/test_licm.rs @@ -34,9 +34,7 @@ impl SubTest for TestLICM { } fn run(&self, func: Cow, context: &Context) -> Result<()> { - // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne_codegen::Context::new(); - comp_ctx.func = func.into_owned(); + let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); comp_ctx.flowgraph(); comp_ctx.compute_loop_analysis(); diff --git a/lib/filetests/src/test_postopt.rs b/lib/filetests/src/test_postopt.rs index 192a89c8b3..72bc3dfc2f 100644 --- a/lib/filetests/src/test_postopt.rs +++ b/lib/filetests/src/test_postopt.rs @@ -31,9 +31,7 @@ impl SubTest for TestPostopt { } fn run(&self, func: Cow, context: &Context) -> Result<()> { - // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne_codegen::Context::new(); - comp_ctx.func = func.into_owned(); + let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); let isa = context.isa.expect("postopt needs an ISA"); comp_ctx.flowgraph(); diff --git a/lib/filetests/src/test_preopt.rs b/lib/filetests/src/test_preopt.rs index dbd68e8bac..1a3131ba83 100644 --- a/lib/filetests/src/test_preopt.rs +++ b/lib/filetests/src/test_preopt.rs @@ -31,9 +31,7 @@ impl SubTest for TestPreopt { } fn run(&self, func: Cow, context: &Context) -> Result<()> { - // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne_codegen::Context::new(); - comp_ctx.func = func.into_owned(); + let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); let isa = context.isa.expect("preopt needs an ISA"); comp_ctx.flowgraph(); diff --git a/lib/filetests/src/test_regalloc.rs b/lib/filetests/src/test_regalloc.rs index 91217899e1..c74662c19a 100644 --- a/lib/filetests/src/test_regalloc.rs +++ b/lib/filetests/src/test_regalloc.rs @@ -39,10 +39,7 @@ impl SubTest for TestRegalloc { fn run(&self, func: Cow, context: &Context) -> Result<()> { let isa = context.isa.expect("register allocator needs an ISA"); - - // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne_codegen::Context::new(); - comp_ctx.func = func.into_owned(); + let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); comp_ctx.compute_cfg(); // TODO: Should we have an option to skip legalization? diff --git a/lib/filetests/src/test_simple_gvn.rs b/lib/filetests/src/test_simple_gvn.rs index 2036301bcc..7def78e95a 100644 --- a/lib/filetests/src/test_simple_gvn.rs +++ b/lib/filetests/src/test_simple_gvn.rs @@ -34,9 +34,7 @@ impl SubTest for TestSimpleGVN { } fn run(&self, func: Cow, context: &Context) -> Result<()> { - // Create a compilation context, and drop in the function. - let mut comp_ctx = cretonne_codegen::Context::new(); - comp_ctx.func = func.into_owned(); + let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); comp_ctx.flowgraph(); comp_ctx From 4a4453dc479fb9c964217014006d4d2f3ddcafa7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 4 Jun 2018 16:27:09 -0700 Subject: [PATCH 1809/3084] Use `to_string()` instead of `format!("{}", ...)`. --- cranelift/src/utils.rs | 2 +- lib/faerie/src/backend.rs | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index 138d4f5dcc..00f5bab347 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -59,7 +59,7 @@ pub fn parse_sets_and_triple( if let Some(triple_name) = words.next() { let triple = match Triple::from_str(triple_name) { Ok(triple) => triple, - Err(parse_error) => return Err(format!("{}", parse_error)), + Err(parse_error) => return Err(parse_error.to_string()), }; let mut isa_builder = isa::lookup(triple).map_err(|err| match err { isa::LookupError::SupportDisabled => { diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index 14b4729e26..890a934fd2 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -230,7 +230,7 @@ impl Backend for FaerieBackend { to, at: offset as usize, }) - .map_err(|e| ModuleError::Backend(format!("{}", e)))?; + .map_err(|e| ModuleError::Backend(e.to_string()))?; } for &(offset, id, addend) in data_relocs { debug_assert_eq!( @@ -244,7 +244,7 @@ impl Backend for FaerieBackend { to, at: offset as usize, }) - .map_err(|e| ModuleError::Backend(format!("{}", e)))?; + .map_err(|e| ModuleError::Backend(e.to_string()))?; } self.artifact From 1087ff3a0152a04ebb64e2caff417427b35d0b49 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 4 Jun 2018 16:29:13 -0700 Subject: [PATCH 1810/3084] Use `to_string()` instead of `write!(&mut text, "{}", ...`. --- lib/filetests/src/test_compile.rs | 4 +--- lib/filetests/src/test_dce.rs | 4 +--- lib/filetests/src/test_legalizer.rs | 4 +--- lib/filetests/src/test_licm.rs | 4 +--- lib/filetests/src/test_postopt.rs | 4 +--- lib/filetests/src/test_preopt.rs | 4 +--- lib/filetests/src/test_regalloc.rs | 4 +--- lib/filetests/src/test_simple_gvn.rs | 4 +--- 8 files changed, 8 insertions(+), 24 deletions(-) diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs index 8cb26b4f1f..73f178b9b7 100644 --- a/lib/filetests/src/test_compile.rs +++ b/lib/filetests/src/test_compile.rs @@ -7,7 +7,6 @@ use cretonne_codegen::print_errors::pretty_error; use cretonne_codegen::{binemit, ir}; use cretonne_reader::TestCommand; use std::borrow::Cow; -use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; struct TestCompile; @@ -64,8 +63,7 @@ impl SubTest for TestCompile { } // Run final code through filecheck. - let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))).map_err(|e| e.to_string())?; + let text = comp_ctx.func.display(Some(isa)).to_string(); run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_dce.rs b/lib/filetests/src/test_dce.rs index 38e2b907a9..4c795b5a2e 100644 --- a/lib/filetests/src/test_dce.rs +++ b/lib/filetests/src/test_dce.rs @@ -10,7 +10,6 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; struct TestDCE; @@ -42,8 +41,7 @@ impl SubTest for TestDCE { .dce(context.flags_or_isa()) .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?; - let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func).map_err(|e| e.to_string())?; + let text = comp_ctx.func.to_string(); run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_legalizer.rs b/lib/filetests/src/test_legalizer.rs index 8ce1cffb1b..afaaf3fdcb 100644 --- a/lib/filetests/src/test_legalizer.rs +++ b/lib/filetests/src/test_legalizer.rs @@ -8,7 +8,6 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; struct TestLegalizer; @@ -44,8 +43,7 @@ impl SubTest for TestLegalizer { .legalize(isa) .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; - let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))).map_err(|e| e.to_string())?; + let text = comp_ctx.func.display(Some(isa)).to_string(); run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_licm.rs b/lib/filetests/src/test_licm.rs index 2dda7b976b..9f42254a18 100644 --- a/lib/filetests/src/test_licm.rs +++ b/lib/filetests/src/test_licm.rs @@ -10,7 +10,6 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; struct TestLICM; @@ -42,8 +41,7 @@ impl SubTest for TestLICM { .licm(context.flags_or_isa()) .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?; - let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func).map_err(|e| e.to_string())?; + let text = comp_ctx.func.to_string(); run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_postopt.rs b/lib/filetests/src/test_postopt.rs index 72bc3dfc2f..6e3712922f 100644 --- a/lib/filetests/src/test_postopt.rs +++ b/lib/filetests/src/test_postopt.rs @@ -7,7 +7,6 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; struct TestPostopt; @@ -39,8 +38,7 @@ impl SubTest for TestPostopt { .postopt(isa) .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?; - let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func).map_err(|e| e.to_string())?; + let text = comp_ctx.func.to_string(); run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_preopt.rs b/lib/filetests/src/test_preopt.rs index 1a3131ba83..ad68e59c09 100644 --- a/lib/filetests/src/test_preopt.rs +++ b/lib/filetests/src/test_preopt.rs @@ -7,7 +7,6 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; struct TestPreopt; @@ -39,8 +38,7 @@ impl SubTest for TestPreopt { .preopt(isa) .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?; - let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func).map_err(|e| e.to_string())?; + let text = &comp_ctx.func.to_string(); run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_regalloc.rs b/lib/filetests/src/test_regalloc.rs index c74662c19a..b592cd12dc 100644 --- a/lib/filetests/src/test_regalloc.rs +++ b/lib/filetests/src/test_regalloc.rs @@ -10,7 +10,6 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; struct TestRegalloc; @@ -51,8 +50,7 @@ impl SubTest for TestRegalloc { .regalloc(isa) .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?; - let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func.display(Some(isa))).map_err(|e| e.to_string())?; + let text = comp_ctx.func.display(Some(isa)).to_string(); run_filecheck(&text, context) } } diff --git a/lib/filetests/src/test_simple_gvn.rs b/lib/filetests/src/test_simple_gvn.rs index 7def78e95a..a78d30753c 100644 --- a/lib/filetests/src/test_simple_gvn.rs +++ b/lib/filetests/src/test_simple_gvn.rs @@ -10,7 +10,6 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use std::fmt::Write; use subtest::{run_filecheck, Context, Result, SubTest}; struct TestSimpleGVN; @@ -41,8 +40,7 @@ impl SubTest for TestSimpleGVN { .simple_gvn(context.flags_or_isa()) .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?; - let mut text = String::new(); - write!(&mut text, "{}", &comp_ctx.func).map_err(|e| e.to_string())?; + let text = comp_ctx.func.to_string(); run_filecheck(&text, context) } } From 4e64fc11c99bf4d6c76f6596d844b80ded9e6ad3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 4 Jun 2018 16:32:55 -0700 Subject: [PATCH 1811/3084] Rename subtest's `Result` to `SubtestResult`. This avoids naming confusion with the standard `Result`, which is included in the prelude. --- lib/filetests/src/lib.rs | 2 +- lib/filetests/src/runone.rs | 8 ++++---- lib/filetests/src/subtest.rs | 9 ++++----- lib/filetests/src/test_binemit.rs | 6 +++--- lib/filetests/src/test_cat.rs | 2 +- lib/filetests/src/test_compile.rs | 6 +++--- lib/filetests/src/test_dce.rs | 6 +++--- lib/filetests/src/test_domtree.rs | 9 ++++----- lib/filetests/src/test_legalizer.rs | 6 +++--- lib/filetests/src/test_licm.rs | 6 +++--- lib/filetests/src/test_postopt.rs | 6 +++--- lib/filetests/src/test_preopt.rs | 6 +++--- lib/filetests/src/test_print_cfg.rs | 2 +- lib/filetests/src/test_regalloc.rs | 6 +++--- lib/filetests/src/test_simple_gvn.rs | 6 +++--- lib/filetests/src/test_verifier.rs | 6 +++--- 16 files changed, 45 insertions(+), 47 deletions(-) diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs index 7dd87302a5..811b0e9e10 100644 --- a/lib/filetests/src/lib.rs +++ b/lib/filetests/src/lib.rs @@ -79,7 +79,7 @@ pub fn run(verbose: bool, files: &[String]) -> TestResult { /// /// This function knows how to create all of the possible `test ` commands that can appear in /// a `.cton` test file. -fn new_subtest(parsed: &TestCommand) -> subtest::Result> { +fn new_subtest(parsed: &TestCommand) -> subtest::SubtestResult> { match parsed.command { "binemit" => test_binemit::subtest(parsed), "cat" => test_cat::subtest(parsed), diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs index 4d3731619b..26cc31a839 100644 --- a/lib/filetests/src/runone.rs +++ b/lib/filetests/src/runone.rs @@ -13,7 +13,7 @@ use std::fs; use std::io::{self, Read}; use std::path::Path; use std::time; -use subtest::{Context, Result, SubTest}; +use subtest::{Context, SubTest, SubtestResult}; use {new_subtest, TestResult}; /// Read an entire file into a string. @@ -42,7 +42,7 @@ pub fn run(path: &Path) -> TestResult { .commands .iter() .map(new_subtest) - .collect::>>()?; + .collect::>>()?; // Flags to use for those tests that don't need an ISA. // This is the cumulative effect of all the `set` commands in the file. @@ -90,7 +90,7 @@ fn test_tuples<'a>( tests: &'a [Box], isa_spec: &'a IsaSpec, no_isa_flags: &'a Flags, -) -> Result)>> { +) -> SubtestResult)>> { let mut out = Vec::new(); for test in tests { if test.needs_isa() { @@ -119,7 +119,7 @@ fn run_one_test<'a>( tuple: (&'a SubTest, &'a Flags, Option<&'a TargetIsa>), func: Cow, context: &mut Context<'a>, -) -> Result<()> { +) -> SubtestResult<()> { let (test, flags, isa) = tuple; let name = format!("{}({})", test.name(), func.name); dbg!("Test: {} {}", name, isa.map_or("-", TargetIsa::name)); diff --git a/lib/filetests/src/subtest.rs b/lib/filetests/src/subtest.rs index 951aa76e56..91c92d251a 100644 --- a/lib/filetests/src/subtest.rs +++ b/lib/filetests/src/subtest.rs @@ -6,9 +6,8 @@ use cretonne_codegen::settings::{Flags, FlagsOrIsa}; use cretonne_reader::{Comment, Details}; use filecheck::{Checker, CheckerBuilder, NO_VARIABLES}; use std::borrow::Cow; -use std::result; -pub type Result = result::Result; +pub type SubtestResult = Result; /// Context for running a test on a single function. pub struct Context<'a> { @@ -64,11 +63,11 @@ pub trait SubTest { } /// Run this test on `func`. - fn run(&self, func: Cow, context: &Context) -> Result<()>; + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()>; } /// Run filecheck on `text`, using directives extracted from `context`. -pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { +pub fn run_filecheck(text: &str, context: &Context) -> SubtestResult<()> { let checker = build_filechecker(context)?; if checker .check(text, NO_VARIABLES) @@ -85,7 +84,7 @@ pub fn run_filecheck(text: &str, context: &Context) -> Result<()> { } /// Build a filechecker using the directives in the file preamble and the function's comments. -pub fn build_filechecker(context: &Context) -> Result { +pub fn build_filechecker(context: &Context) -> SubtestResult { let mut builder = CheckerBuilder::new(); // Preamble comments apply to all functions. for comment in context.preamble_comments { diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index d736faceee..1eb24cf31a 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -15,11 +15,11 @@ use match_directive::match_directive; use std::borrow::Cow; use std::collections::HashMap; use std::fmt::Write; -use subtest::{Context, Result, SubTest}; +use subtest::{Context, SubTest, SubtestResult}; struct TestBinEmit; -pub fn subtest(parsed: &TestCommand) -> Result> { +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { assert_eq!(parsed.command, "binemit"); if !parsed.options.is_empty() { Err(format!("No options allowed on {}", parsed)) @@ -108,7 +108,7 @@ impl SubTest for TestBinEmit { true } - fn run(&self, func: Cow, context: &Context) -> Result<()> { + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let isa = context.isa.expect("binemit needs an ISA"); let encinfo = isa.encoding_info(); // TODO: Run a verifier pass over the code first to detect any bad encodings or missing/bad diff --git a/lib/filetests/src/test_cat.rs b/lib/filetests/src/test_cat.rs index 363567695f..aaecb2d735 100644 --- a/lib/filetests/src/test_cat.rs +++ b/lib/filetests/src/test_cat.rs @@ -3,7 +3,7 @@ use cretonne_codegen::ir::Function; use cretonne_reader::TestCommand; use std::borrow::Cow; -use subtest::{self, Context, Result as STResult, SubTest}; +use subtest::{self, Context, SubTest, SubtestResult as STResult}; /// Object implementing the `test cat` sub-test. /// diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs index 73f178b9b7..7fce3b56ad 100644 --- a/lib/filetests/src/test_compile.rs +++ b/lib/filetests/src/test_compile.rs @@ -7,11 +7,11 @@ use cretonne_codegen::print_errors::pretty_error; use cretonne_codegen::{binemit, ir}; use cretonne_reader::TestCommand; use std::borrow::Cow; -use subtest::{run_filecheck, Context, Result, SubTest}; +use subtest::{run_filecheck, Context, SubTest, SubtestResult}; struct TestCompile; -pub fn subtest(parsed: &TestCommand) -> Result> { +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { assert_eq!(parsed.command, "compile"); if !parsed.options.is_empty() { Err(format!("No options allowed on {}", parsed)) @@ -33,7 +33,7 @@ impl SubTest for TestCompile { true } - fn run(&self, func: Cow, context: &Context) -> Result<()> { + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let isa = context.isa.expect("compile needs an ISA"); let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); diff --git a/lib/filetests/src/test_dce.rs b/lib/filetests/src/test_dce.rs index 4c795b5a2e..748c7ecf79 100644 --- a/lib/filetests/src/test_dce.rs +++ b/lib/filetests/src/test_dce.rs @@ -10,11 +10,11 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use subtest::{run_filecheck, Context, Result, SubTest}; +use subtest::{run_filecheck, Context, SubTest, SubtestResult}; struct TestDCE; -pub fn subtest(parsed: &TestCommand) -> Result> { +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { assert_eq!(parsed.command, "dce"); if !parsed.options.is_empty() { Err(format!("No options allowed on {}", parsed)) @@ -32,7 +32,7 @@ impl SubTest for TestDCE { true } - fn run(&self, func: Cow, context: &Context) -> Result<()> { + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); comp_ctx.flowgraph(); diff --git a/lib/filetests/src/test_domtree.rs b/lib/filetests/src/test_domtree.rs index a262293567..b6d6922c7f 100644 --- a/lib/filetests/src/test_domtree.rs +++ b/lib/filetests/src/test_domtree.rs @@ -21,12 +21,11 @@ use match_directive::match_directive; use std::borrow::{Borrow, Cow}; use std::collections::HashMap; use std::fmt::{self, Write}; -use std::result; -use subtest::{run_filecheck, Context, Result, SubTest}; +use subtest::{run_filecheck, Context, SubTest, SubtestResult}; struct TestDomtree; -pub fn subtest(parsed: &TestCommand) -> Result> { +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { assert_eq!(parsed.command, "domtree"); if !parsed.options.is_empty() { Err(format!("No options allowed on {}", parsed)) @@ -41,7 +40,7 @@ impl SubTest for TestDomtree { } // Extract our own dominator tree from - fn run(&self, func: Cow, context: &Context) -> Result<()> { + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let func = func.borrow(); let cfg = ControlFlowGraph::with_function(func); let domtree = DominatorTree::with_function(func, &cfg); @@ -114,7 +113,7 @@ impl SubTest for TestDomtree { } // Generate some output for filecheck testing -fn filecheck_text(func: &Function, domtree: &DominatorTree) -> result::Result { +fn filecheck_text(func: &Function, domtree: &DominatorTree) -> Result { let mut s = String::new(); write!(s, "cfg_postorder:")?; diff --git a/lib/filetests/src/test_legalizer.rs b/lib/filetests/src/test_legalizer.rs index afaaf3fdcb..eda564e34f 100644 --- a/lib/filetests/src/test_legalizer.rs +++ b/lib/filetests/src/test_legalizer.rs @@ -8,11 +8,11 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use subtest::{run_filecheck, Context, Result, SubTest}; +use subtest::{run_filecheck, Context, SubTest, SubtestResult}; struct TestLegalizer; -pub fn subtest(parsed: &TestCommand) -> Result> { +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { assert_eq!(parsed.command, "legalizer"); if !parsed.options.is_empty() { Err(format!("No options allowed on {}", parsed)) @@ -34,7 +34,7 @@ impl SubTest for TestLegalizer { true } - fn run(&self, func: Cow, context: &Context) -> Result<()> { + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); let isa = context.isa.expect("legalizer needs an ISA"); diff --git a/lib/filetests/src/test_licm.rs b/lib/filetests/src/test_licm.rs index 9f42254a18..6004b1529e 100644 --- a/lib/filetests/src/test_licm.rs +++ b/lib/filetests/src/test_licm.rs @@ -10,11 +10,11 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use subtest::{run_filecheck, Context, Result, SubTest}; +use subtest::{run_filecheck, Context, SubTest, SubtestResult}; struct TestLICM; -pub fn subtest(parsed: &TestCommand) -> Result> { +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { assert_eq!(parsed.command, "licm"); if !parsed.options.is_empty() { Err(format!("No options allowed on {}", parsed)) @@ -32,7 +32,7 @@ impl SubTest for TestLICM { true } - fn run(&self, func: Cow, context: &Context) -> Result<()> { + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); comp_ctx.flowgraph(); diff --git a/lib/filetests/src/test_postopt.rs b/lib/filetests/src/test_postopt.rs index 6e3712922f..16367350c4 100644 --- a/lib/filetests/src/test_postopt.rs +++ b/lib/filetests/src/test_postopt.rs @@ -7,11 +7,11 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use subtest::{run_filecheck, Context, Result, SubTest}; +use subtest::{run_filecheck, Context, SubTest, SubtestResult}; struct TestPostopt; -pub fn subtest(parsed: &TestCommand) -> Result> { +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { assert_eq!(parsed.command, "postopt"); if !parsed.options.is_empty() { Err(format!("No options allowed on {}", parsed)) @@ -29,7 +29,7 @@ impl SubTest for TestPostopt { true } - fn run(&self, func: Cow, context: &Context) -> Result<()> { + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); let isa = context.isa.expect("postopt needs an ISA"); diff --git a/lib/filetests/src/test_preopt.rs b/lib/filetests/src/test_preopt.rs index ad68e59c09..169db9a583 100644 --- a/lib/filetests/src/test_preopt.rs +++ b/lib/filetests/src/test_preopt.rs @@ -7,11 +7,11 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use subtest::{run_filecheck, Context, Result, SubTest}; +use subtest::{run_filecheck, Context, SubTest, SubtestResult}; struct TestPreopt; -pub fn subtest(parsed: &TestCommand) -> Result> { +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { assert_eq!(parsed.command, "preopt"); if !parsed.options.is_empty() { Err(format!("No options allowed on {}", parsed)) @@ -29,7 +29,7 @@ impl SubTest for TestPreopt { true } - fn run(&self, func: Cow, context: &Context) -> Result<()> { + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); let isa = context.isa.expect("preopt needs an ISA"); diff --git a/lib/filetests/src/test_print_cfg.rs b/lib/filetests/src/test_print_cfg.rs index 741dae762b..d211381886 100644 --- a/lib/filetests/src/test_print_cfg.rs +++ b/lib/filetests/src/test_print_cfg.rs @@ -8,7 +8,7 @@ use std::borrow::Cow; use cretonne_codegen::cfg_printer::CFGPrinter; use cretonne_codegen::ir::Function; use cretonne_reader::TestCommand; -use subtest::{self, Context, Result as STResult, SubTest}; +use subtest::{self, Context, SubTest, SubtestResult as STResult}; /// Object implementing the `test print-cfg` sub-test. struct TestPrintCfg; diff --git a/lib/filetests/src/test_regalloc.rs b/lib/filetests/src/test_regalloc.rs index b592cd12dc..eb13ee402d 100644 --- a/lib/filetests/src/test_regalloc.rs +++ b/lib/filetests/src/test_regalloc.rs @@ -10,11 +10,11 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use subtest::{run_filecheck, Context, Result, SubTest}; +use subtest::{run_filecheck, Context, SubTest, SubtestResult}; struct TestRegalloc; -pub fn subtest(parsed: &TestCommand) -> Result> { +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { assert_eq!(parsed.command, "regalloc"); if !parsed.options.is_empty() { Err(format!("No options allowed on {}", parsed)) @@ -36,7 +36,7 @@ impl SubTest for TestRegalloc { true } - fn run(&self, func: Cow, context: &Context) -> Result<()> { + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let isa = context.isa.expect("register allocator needs an ISA"); let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); diff --git a/lib/filetests/src/test_simple_gvn.rs b/lib/filetests/src/test_simple_gvn.rs index a78d30753c..faf995cff3 100644 --- a/lib/filetests/src/test_simple_gvn.rs +++ b/lib/filetests/src/test_simple_gvn.rs @@ -10,11 +10,11 @@ use cretonne_codegen::ir::Function; use cretonne_codegen::print_errors::pretty_error; use cretonne_reader::TestCommand; use std::borrow::Cow; -use subtest::{run_filecheck, Context, Result, SubTest}; +use subtest::{run_filecheck, Context, SubTest, SubtestResult}; struct TestSimpleGVN; -pub fn subtest(parsed: &TestCommand) -> Result> { +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { assert_eq!(parsed.command, "simple-gvn"); if !parsed.options.is_empty() { Err(format!("No options allowed on {}", parsed)) @@ -32,7 +32,7 @@ impl SubTest for TestSimpleGVN { true } - fn run(&self, func: Cow, context: &Context) -> Result<()> { + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); comp_ctx.flowgraph(); diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs index 69e56b1cd5..9be54e7482 100644 --- a/lib/filetests/src/test_verifier.rs +++ b/lib/filetests/src/test_verifier.rs @@ -14,11 +14,11 @@ use cretonne_codegen::verify_function; use cretonne_reader::TestCommand; use match_directive::match_directive; use std::borrow::{Borrow, Cow}; -use subtest::{Context, Result, SubTest}; +use subtest::{Context, SubTest, SubtestResult}; struct TestVerifier; -pub fn subtest(parsed: &TestCommand) -> Result> { +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { assert_eq!(parsed.command, "verifier"); if !parsed.options.is_empty() { Err(format!("No options allowed on {}", parsed)) @@ -37,7 +37,7 @@ impl SubTest for TestVerifier { false } - fn run(&self, func: Cow, context: &Context) -> Result<()> { + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let func = func.borrow(); // Scan source annotations for "error:" directives. From eb946642856e8a5ba7fcf1f3e41e090de116db78 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 5 Jun 2018 09:19:08 -0700 Subject: [PATCH 1812/3084] Add assertions to check translate_call and translate_call_indirect. Assert that the results produced by translate_call and translate_call_indirect match the results of the call signatures. --- lib/wasm/src/code_translator.rs | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 7aae3d6678..1d5386184e 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -361,8 +361,16 @@ pub fn translate_operator( fref, state.peekn(num_args), )?; + let inst_results = builder.inst_results(call); + debug_assert_eq!( + inst_results.len(), + builder.func.dfg.signatures[builder.func.dfg.ext_funcs[fref].signature] + .returns + .len(), + "translate_call results should match the call signature" + ); state.popn(num_args); - state.pushn(builder.inst_results(call)); + state.pushn(inst_results); } Operator::CallIndirect { index, table_index } => { // `index` is the index of the function's signature and `table_index` is the index of @@ -377,8 +385,14 @@ pub fn translate_operator( callee, state.peekn(num_args), )?; + let inst_results = builder.inst_results(call); + debug_assert_eq!( + inst_results.len(), + builder.func.dfg.signatures[sigref].returns.len(), + "translate_call_indirect results should match the call signature" + ); state.popn(num_args); - state.pushn(builder.inst_results(call)); + state.pushn(inst_results); } /******************************* Memory management *********************************** * Memory management is handled by environment. It is usually translated into calls to From 970045c3097c57d22e1d704745f7217582a8cf48 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 5 Jun 2018 11:23:57 -0700 Subject: [PATCH 1813/3084] Set clobbers_flags to False for urm_noflags_abcd. `urm_noflags` is a variant of `urm` which doesn't clobber the flags, and `urm_noflags_abcd` is a further variant for ABCD registers, so it also doesn't clobber the flags. --- cranelift/docs/testing.rst | 8 +++++ lib/codegen/meta/isa/x86/recipes.py | 1 + lib/filetests/src/lib.rs | 2 ++ lib/filetests/src/test_shrink.rs | 46 +++++++++++++++++++++++++++++ 4 files changed, 57 insertions(+) create mode 100644 lib/filetests/src/test_shrink.rs diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index 02071f872d..2b02bb6d79 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -342,6 +342,14 @@ Test the DCE pass. The DCE pass is run on each function, and then results are run through filecheck. +`test shrink` +----------------- + +Test the instruction shrinking pass. + +The shrink pass is run on each function, and then results are run +through filecheck. + `test preopt` ----------------- diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py index 047408db9c..8828f8a06e 100644 --- a/lib/codegen/meta/isa/x86/recipes.py +++ b/lib/codegen/meta/isa/x86/recipes.py @@ -389,6 +389,7 @@ urm_noflags = TailRecipe( urm_noflags_abcd = TailRecipe( 'urm_noflags_abcd', Unary, size=1, ins=ABCD, outs=GPR, when_prefixed=urm_noflags, + clobbers_flags=False, emit=''' PUT_OP(bits, rex2(in_reg0, out_reg0), sink); modrm_rr(in_reg0, out_reg0, sink); diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs index 811b0e9e10..a86c0a0aae 100644 --- a/lib/filetests/src/lib.rs +++ b/lib/filetests/src/lib.rs @@ -45,6 +45,7 @@ mod test_postopt; mod test_preopt; mod test_print_cfg; mod test_regalloc; +mod test_shrink; mod test_simple_gvn; mod test_verifier; @@ -92,6 +93,7 @@ fn new_subtest(parsed: &TestCommand) -> subtest::SubtestResult test_preopt::subtest(parsed), "print-cfg" => test_print_cfg::subtest(parsed), "regalloc" => test_regalloc::subtest(parsed), + "shrink" => test_shrink::subtest(parsed), "simple-gvn" => test_simple_gvn::subtest(parsed), "verifier" => test_verifier::subtest(parsed), _ => Err(format!("unknown test command '{}'", parsed.command)), diff --git a/lib/filetests/src/test_shrink.rs b/lib/filetests/src/test_shrink.rs new file mode 100644 index 0000000000..4388919f36 --- /dev/null +++ b/lib/filetests/src/test_shrink.rs @@ -0,0 +1,46 @@ +//! Test command for testing the Shrink pass. +//! +//! The `shrink` test command runs each function through the Shrink pass after ensuring +//! that all instructions are legal for the target. +//! +//! The resulting function is sent to `filecheck`. + +use cretonne_codegen; +use cretonne_codegen::ir::Function; +use cretonne_codegen::print_errors::pretty_error; +use cretonne_reader::TestCommand; +use std::borrow::Cow; +use subtest::{run_filecheck, Context, SubTest, SubtestResult}; + +struct TestShrink; + +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { + assert_eq!(parsed.command, "shrink"); + if !parsed.options.is_empty() { + Err(format!("No options allowed on {}", parsed)) + } else { + Ok(Box::new(TestShrink)) + } +} + +impl SubTest for TestShrink { + fn name(&self) -> &'static str { + "shrink" + } + + fn is_mutating(&self) -> bool { + true + } + + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { + let isa = context.isa.expect("shrink needs an ISA"); + let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); + + comp_ctx + .shrink_instructions(isa) + .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?; + + let text = comp_ctx.func.to_string(); + run_filecheck(&text, context) + } +} From 5c5e66cebd0ec393b51fd672083003b886c469bc Mon Sep 17 00:00:00 2001 From: Bruce Mitchener Date: Wed, 6 Jun 2018 23:18:45 +0700 Subject: [PATCH 1814/3084] Update to capstone 0.4. --- cranelift/Cargo.toml | 2 +- cranelift/src/compile.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 57193f7af9..1db64e84f5 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -29,7 +29,7 @@ docopt = "1" serde = "1.0.8" serde_derive = "1.0.8" term = "0.5.1" -capstone = "0.3.1" +capstone = "0.4" wabt = { version = "0.3", optional = true } target-lexicon = "0.0.1" diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 7eea32262e..516a1be84a 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -123,7 +123,7 @@ fn handle_module( } println!(); - let cs = get_disassembler(isa)?; + let mut cs = get_disassembler(isa)?; println!("\nDisassembly:"); let insns = cs.disasm_all(&mem, 0x0).unwrap(); From 36d4e0a13c120965a23d32cbcd03e2ccf887fecb Mon Sep 17 00:00:00 2001 From: Bruce Mitchener Date: Wed, 6 Jun 2018 23:19:10 +0700 Subject: [PATCH 1815/3084] Update to goblin 0.0.15. --- lib/faerie/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index bec2b0959c..efdc4aa648 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -12,7 +12,7 @@ readme = "README.md" cretonne-codegen = { path = "../codegen", version = "0.8.0" } cretonne-module = { path = "../module", version = "0.8.0" } faerie = "0.4.1" -goblin = "0.0.14" +goblin = "0.0.15" failure = "0.1.1" target-lexicon = "0.0.1" From e49e1e20c682d06856d9c6d67861ad82ffbf32ff Mon Sep 17 00:00:00 2001 From: Bruce Mitchener Date: Wed, 6 Jun 2018 23:22:29 +0700 Subject: [PATCH 1816/3084] Update to region 0.3. --- lib/simplejit/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index ef551a3e79..286f6fc2d0 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -12,7 +12,7 @@ readme = "README.md" cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } cretonne-module = { path = "../module", version = "0.8.0", default-features = false } cretonne-native = { path = "../native", version = "0.8.0", default-features = false } -region = "0.2.0" +region = "0.3.0" libc = { version = "0.2.40", default-features = false } errno = "0.2.3" target-lexicon = { version = "0.0.1", default-features = false } From 6ec3a3f7a1cbb5debfd648816cf4c270cf2b237e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Jun 2018 09:56:45 -0700 Subject: [PATCH 1817/3084] Pass the "std" feature through to target-lexicon. --- lib/codegen/Cargo.toml | 2 +- lib/wasm/Cargo.toml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index e593e7a448..417fc80bf4 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -26,7 +26,7 @@ target-lexicon = { version = "0.0.1", default-features = false } # of some minimal std-like replacement libraries. At least one of these two # features need to be enabled. default = ["std"] -std = ["cretonne-entity/std"] +std = ["cretonne-entity/std", "target-lexicon/std"] core = ["hashmap_core"] [badges] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 0aff1caa40..650c8a574b 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -22,7 +22,7 @@ wabt = "0.3" [features] default = ["std"] -std = ["cretonne-codegen/std", "cretonne-frontend/std", "wasmparser/std"] +std = ["cretonne-codegen/std", "cretonne-frontend/std", "wasmparser/std", "target-lexicon/std"] core = ["hashmap_core", "cretonne-codegen/core", "cretonne-frontend/core", "wasmparser/core"] [badges] From 750483b9acc50b3105882e27f6b6eb3226c2a558 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Jun 2018 10:45:30 -0700 Subject: [PATCH 1818/3084] Update to hashmap_core 0.1.6. --- lib/codegen/Cargo.toml | 2 +- lib/module/Cargo.toml | 2 +- lib/wasm/Cargo.toml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 417fc80bf4..340b080578 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -14,7 +14,7 @@ build = "build.rs" cretonne-entity = { path = "../entity", version = "0.8.0", default-features = false } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } -hashmap_core = { version = "0.1.4", optional = true } +hashmap_core = { version = "0.1.6", optional = true } target-lexicon = { version = "0.0.1", default-features = false } # It is a goal of the cretonne-codegen crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 8708478ad6..60bb1834d6 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -11,7 +11,7 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } cretonne-entity = { path = "../entity", version = "0.8.0", default-features = false } -hashmap_core = { version = "0.1.4", optional = true } +hashmap_core = { version = "0.1.6", optional = true } failure = "0.1.1" [features] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 650c8a574b..3d6c0a93d6 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -12,7 +12,7 @@ keywords = ["webassembly", "wasm"] wasmparser = { version = "0.16.1", default-features = false } cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } cretonne-frontend = { path = "../frontend", version = "0.8.0", default-features = false } -hashmap_core = { version = "0.1.4", optional = true } +hashmap_core = { version = "0.1.6", optional = true } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } target-lexicon = { version = "0.0.1", default-features = false } From b2b20a95a10e414be9c66fc931801d15ee295c69 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Jun 2018 11:26:09 -0700 Subject: [PATCH 1819/3084] Fix missing no_std support in cretonne-module. And, tidy up the extern crate declarations in the std replacement modules. --- lib/codegen/src/lib.rs | 11 +++++------ lib/frontend/src/lib.rs | 7 +++---- lib/module/src/data_context.rs | 2 ++ lib/module/src/lib.rs | 24 +++++++++++++++++++++++- lib/module/src/module.rs | 2 ++ lib/wasm/src/lib.rs | 18 ++++++++---------- 6 files changed, 43 insertions(+), 21 deletions(-) diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs index cfcae5d344..dce082e02a 100644 --- a/lib/codegen/src/lib.rs +++ b/lib/codegen/src/lib.rs @@ -42,10 +42,6 @@ #![cfg_attr(not(feature = "std"), no_std)] #![cfg_attr(not(feature = "std"), feature(alloc))] -// Include the `hashmap_core` crate if std is not available. -#[allow(unused_extern_crates)] -#[cfg(not(feature = "std"))] -extern crate hashmap_core; #[cfg(not(feature = "std"))] #[macro_use] extern crate alloc; @@ -119,8 +115,11 @@ mod std { pub use alloc::{boxed, string, vec}; pub use core::*; pub mod collections { + #[allow(unused_extern_crates)] + extern crate hashmap_core; + + pub use self::hashmap_core::map as hash_map; + pub use self::hashmap_core::{HashMap, HashSet}; pub use alloc::BTreeSet; - pub use hashmap_core::map as hash_map; - pub use hashmap_core::{HashMap, HashSet}; } } diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 8b25becb33..8a414dd849 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -143,9 +143,6 @@ extern crate cretonne_codegen; -#[cfg(not(feature = "std"))] -extern crate alloc; - pub use frontend::{FunctionBuilder, FunctionBuilderContext}; pub use variable::Variable; @@ -155,6 +152,8 @@ mod variable; #[cfg(not(feature = "std"))] mod std { - pub use alloc::vec; + extern crate alloc; + + pub use self::alloc::vec; pub use core::*; } diff --git a/lib/module/src/data_context.rs b/lib/module/src/data_context.rs index ef327fce88..0fef3f6400 100644 --- a/lib/module/src/data_context.rs +++ b/lib/module/src/data_context.rs @@ -3,6 +3,8 @@ use cretonne_codegen::binemit::{Addend, CodeOffset}; use cretonne_codegen::entity::PrimaryMap; use cretonne_codegen::ir; +use std::boxed::Box; +use std::vec::Vec; /// This specifies how data is to be initialized. #[derive(PartialEq, Eq, Debug)] diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs index 5c1091dd3f..445b588c40 100644 --- a/lib/module/src/lib.rs +++ b/lib/module/src/lib.rs @@ -1,7 +1,8 @@ //! Top-level lib.rs for `cretonne_module`. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] -#![warn(unused_import_braces, unstable_features)] +#![warn(unused_import_braces)] +#![cfg_attr(feature = "std", warn(unstable_features))] #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))] #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))] #![cfg_attr( @@ -11,6 +12,13 @@ print_stdout, unicode_not_nfc, use_self ) )] +// Turns on no_std and alloc features if std is not available. +#![cfg_attr(not(feature = "std"), no_std)] +#![cfg_attr(not(feature = "std"), feature(alloc))] + +#[cfg(not(feature = "std"))] +#[cfg_attr(test, macro_use)] +extern crate alloc; #[macro_use] extern crate cretonne_codegen; @@ -26,3 +34,17 @@ mod module; pub use backend::Backend; pub use data_context::{DataContext, DataDescription, Init, Writability}; pub use module::{DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleError, ModuleNamespace}; + +/// This replaces `std` in builds with `core`. +#[cfg(not(feature = "std"))] +mod std { + pub use alloc::{borrow, boxed, string, vec}; + pub use core::*; + pub mod collections { + #[allow(unused_extern_crates)] + extern crate hashmap_core; + + pub use self::hashmap_core::map as hash_map; + pub use self::hashmap_core::{HashMap, HashSet}; + } +} diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index ee6943432c..23b8328730 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -9,7 +9,9 @@ use cretonne_codegen::entity::{EntityRef, PrimaryMap}; use cretonne_codegen::result::CtonError; use cretonne_codegen::{binemit, ir, Context}; use data_context::DataContext; +use std::borrow::ToOwned; use std::collections::HashMap; +use std::string::String; use Backend; /// A function identifier for use in the `Module` interface. diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index a7ceeee389..35256c924c 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -24,13 +24,6 @@ #![cfg_attr(not(feature = "std"), no_std)] #![cfg_attr(not(feature = "std"), feature(alloc))] -#[cfg(not(feature = "std"))] -extern crate alloc; - -#[allow(unused_extern_crates)] -#[cfg(not(feature = "std"))] -extern crate hashmap_core; - #[macro_use(dbg)] extern crate cretonne_codegen; extern crate cretonne_frontend; @@ -58,12 +51,17 @@ pub use translation_utils::{FunctionIndex, Global, GlobalIndex, GlobalInit, Memo #[cfg(not(feature = "std"))] mod std { - pub use alloc::string; - pub use alloc::vec; + extern crate alloc; + + pub use self::alloc::string; + pub use self::alloc::vec; pub use core::fmt; pub use core::option; pub use core::{cmp, i32, str, u32}; pub mod collections { - pub use hashmap_core::{map as hash_map, HashMap}; + #[allow(unused_extern_crates)] + extern crate hashmap_core; + + pub use self::hashmap_core::{map as hash_map, HashMap}; } } From f5c1273fcf9eb61638f337303b1f9899e4fc5d79 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Jun 2018 11:29:02 -0700 Subject: [PATCH 1820/3084] Use cfg_attr to tidy up an extern crate declaration. --- lib/codegen/src/lib.rs | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs index dce082e02a..0c7eb52989 100644 --- a/lib/codegen/src/lib.rs +++ b/lib/codegen/src/lib.rs @@ -48,10 +48,7 @@ extern crate alloc; extern crate failure; #[macro_use] extern crate failure_derive; -#[cfg(test)] -#[macro_use] -extern crate target_lexicon; -#[cfg(not(test))] +#[cfg_attr(test, macro_use)] extern crate target_lexicon; pub use context::Context; From f37ebaa4ba637ad1e51779be5587ca109a2a7132 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Jun 2018 11:42:52 -0700 Subject: [PATCH 1821/3084] Bump version to 0.9.0 --- cranelift/publish-all.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 8adbff780e..aa99a28872 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ cd $(dirname "$0") topdir="$(pwd)" # All the cretonne-* crates have the same version number -version="0.8.0" +version="0.9.0" # Update all of the Cargo.toml files. # From 944251260b3dbe3a39bb2441863d6930d1c561b2 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Jun 2018 11:46:36 -0700 Subject: [PATCH 1822/3084] Bump version to 0.9.0 --- cranelift/Cargo.toml | 22 +++++++++++----------- lib/codegen/Cargo.toml | 4 ++-- lib/entity/Cargo.toml | 2 +- lib/faerie/Cargo.toml | 6 +++--- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/module/Cargo.toml | 6 +++--- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/simplejit/Cargo.toml | 8 ++++---- lib/umbrella/Cargo.toml | 6 +++--- lib/wasm/Cargo.toml | 6 +++--- 12 files changed, 39 insertions(+), 39 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 1db64e84f5..6be8f607bb 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.8.0" +version = "0.9.0" description = "Binaries for testing the Cretonne libraries" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -14,16 +14,16 @@ path = "src/cton-util.rs" [dependencies] cfg-if = "0.1" -cretonne-codegen = { path = "lib/codegen", version = "0.8.0" } -cretonne-reader = { path = "lib/reader", version = "0.8.0" } -cretonne-frontend = { path = "lib/frontend", version = "0.8.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.8.0", optional = true } -cretonne-native = { path = "lib/native", version = "0.8.0" } -cretonne-filetests = { path = "lib/filetests", version = "0.8.0" } -cretonne-module = { path = "lib/module", version = "0.8.0" } -cretonne-faerie = { path = "lib/faerie", version = "0.8.0" } -cretonne-simplejit = { path = "lib/simplejit", version = "0.8.0" } -cretonne = { path = "lib/umbrella", version = "0.8.0" } +cretonne-codegen = { path = "lib/codegen", version = "0.9.0" } +cretonne-reader = { path = "lib/reader", version = "0.9.0" } +cretonne-frontend = { path = "lib/frontend", version = "0.9.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.9.0", optional = true } +cretonne-native = { path = "lib/native", version = "0.9.0" } +cretonne-filetests = { path = "lib/filetests", version = "0.9.0" } +cretonne-module = { path = "lib/module", version = "0.9.0" } +cretonne-faerie = { path = "lib/faerie", version = "0.9.0" } +cretonne-simplejit = { path = "lib/simplejit", version = "0.9.0" } +cretonne = { path = "lib/umbrella", version = "0.9.0" } filecheck = "0.3.0" docopt = "1" serde = "1.0.8" diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 340b080578..7a6f76a85a 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-codegen" -version = "0.8.0" +version = "0.9.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"] build = "build.rs" [dependencies] -cretonne-entity = { path = "../entity", version = "0.8.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.9.0", default-features = false } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } hashmap_core = { version = "0.1.6", optional = true } diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index 2ba2db2687..669842223d 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-entity" -version = "0.8.0" +version = "0.9.0" description = "Data structures using entity references as mapping keys" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index efdc4aa648..886a5222a6 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-faerie" -version = "0.8.0" +version = "0.9.0" authors = ["The Cretonne Project Developers"] description = "Emit Cretonne output to native object files with Faerie" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.8.0" } -cretonne-module = { path = "../module", version = "0.8.0" } +cretonne-codegen = { path = "../codegen", version = "0.9.0" } +cretonne-module = { path = "../module", version = "0.9.0" } faerie = "0.4.1" goblin = "0.0.15" failure = "0.1.1" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 171ca4beae..9dd00a4cb6 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.8.0" +version = "0.9.0" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" publish = false [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.8.0" } -cretonne-reader = { path = "../reader", version = "0.8.0" } +cretonne-codegen = { path = "../codegen", version = "0.9.0" } +cretonne-reader = { path = "../reader", version = "0.9.0" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 5dc4d38ca9..97d935a589 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.8.0" +version = "0.9.0" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } [features] default = ["std"] diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 60bb1834d6..39ef56eb19 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-module" -version = "0.8.0" +version = "0.9.0" authors = ["The Cretonne Project Developers"] description = "Support for linking functions and data with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } -cretonne-entity = { path = "../entity", version = "0.8.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.9.0", default-features = false } hashmap_core = { version = "0.1.6", optional = true } failure = "0.1.1" diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 87201854f0..733016df77 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.8.0" +version = "0.9.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -8,7 +8,7 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } target-lexicon = { version = "0.0.1", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index c755c13855..a57971e963 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.8.0" +version = "0.9.0" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.8.0" } +cretonne-codegen = { path = "../codegen", version = "0.9.0" } target-lexicon = "0.0.1" [badges] diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index 286f6fc2d0..bf23dfb8f0 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-simplejit" -version = "0.8.0" +version = "0.9.0" authors = ["The Cretonne Project Developers"] description = "A simple JIT library backed by Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,9 +9,9 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } -cretonne-module = { path = "../module", version = "0.8.0", default-features = false } -cretonne-native = { path = "../native", version = "0.8.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } +cretonne-module = { path = "../module", version = "0.9.0", default-features = false } +cretonne-native = { path = "../native", version = "0.9.0", default-features = false } region = "0.3.0" libc = { version = "0.2.40", default-features = false } errno = "0.2.3" diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml index cd7e25dfc1..244c4661ba 100644 --- a/lib/umbrella/Cargo.toml +++ b/lib/umbrella/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.8.0" +version = "0.9.0" description = "Umbrella for commonly-used cretonne crates" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -10,8 +10,8 @@ readme = "README.md" keywords = ["compile", "compiler", "jit"] [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.8.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.9.0", default-features = false } [features] default = ["std"] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 3d6c0a93d6..ebecc70999 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.8.0" +version = "0.9.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/cretonne/cretonne" @@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"] [dependencies] wasmparser = { version = "0.16.1", default-features = false } -cretonne-codegen = { path = "../codegen", version = "0.8.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.8.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.9.0", default-features = false } hashmap_core = { version = "0.1.6", optional = true } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } From e9111d1de264713f0ad8d04b55a42379c99bbda5 Mon Sep 17 00:00:00 2001 From: Sergey Pepyakin Date: Wed, 30 May 2018 23:31:56 +0200 Subject: [PATCH 1823/3084] Ditch stack_check instruction --- cranelift/docs/langref.rst | 2 -- .../filetests/isa/x86/legalize-memory.cton | 14 -------- lib/codegen/meta/base/instructions.py | 21 ------------ lib/codegen/meta/base/legalize.py | 1 - lib/codegen/src/legalizer/mod.rs | 32 ------------------- 5 files changed, 70 deletions(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 2758a1c7b8..0eb2c0896c 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -546,8 +546,6 @@ instructions before instruction selection:: When Cretonne code is running in a sandbox, it can also be necessary to include stack overflow checks in the prologue. -.. autoinst:: stack_check - Global variables ---------------- diff --git a/cranelift/filetests/isa/x86/legalize-memory.cton b/cranelift/filetests/isa/x86/legalize-memory.cton index 13375e124d..9762aa1f6c 100644 --- a/cranelift/filetests/isa/x86/legalize-memory.cton +++ b/cranelift/filetests/isa/x86/legalize-memory.cton @@ -108,17 +108,3 @@ ebb0(v0: i32, v999: i64): ; nextln: v2 = load.f32 v1+0x7fff_ffff return v2 } - -; Stack overflow check. -; The stack limit is stored in a pointer-sized global variable. -function %stkchk(i64 vmctx) baldrdash { - gv0 = vmctx+64 - -ebb0(v0: i64): - ; check: ebb0( - stack_check gv0 - ; check: $(limit=$V) = load.i64 notrap aligned - ; check: $(flags=$V) = ifcmp_sp $limit - ; check: trapif uge $flags, stk_ovf - return -} diff --git a/lib/codegen/meta/base/instructions.py b/lib/codegen/meta/base/instructions.py index 3107c9ca41..485de44c5e 100644 --- a/lib/codegen/meta/base/instructions.py +++ b/lib/codegen/meta/base/instructions.py @@ -668,27 +668,6 @@ copy_special = Instruction( ins=(src, dst), other_side_effects=True) -GV = Operand( - 'GV', entities.global_var, doc=r""" - Global variable containing the stack limit. - """) - -stack_check = Instruction( - 'stack_check', r""" - Check for stack overflow. - - Read the stack limit from ``GV`` and compare it to the stack pointer. If - the stack pointer has reached or exceeded the limit, generate a trap with a - ``stk_ovf`` code. - - The global variable must be accessible and naturally aligned for a - pointer-sized value. - - `stack_check` is an alternative way to detect stack overflow, when using - a calling convention that doesn't perform stack probes. - """, - ins=GV, can_trap=True) - delta = Operand('delta', Int) adjust_sp_down = Instruction( 'adjust_sp_down', r""" diff --git a/lib/codegen/meta/base/legalize.py b/lib/codegen/meta/base/legalize.py index 0a7e8ea7b3..ad396af747 100644 --- a/lib/codegen/meta/base/legalize.py +++ b/lib/codegen/meta/base/legalize.py @@ -287,7 +287,6 @@ for ty, minus_zero in [ # Expansions using CPU flags. -expand_flags.custom_legalize(insts.stack_check, 'expand_stack_check') expand_flags.legalize( insts.trapnz(x, c), diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs index b3d6d6c0ce..9dcc2dff2f 100644 --- a/lib/codegen/src/legalizer/mod.rs +++ b/lib/codegen/src/legalizer/mod.rs @@ -269,35 +269,3 @@ fn expand_fconst( }; pos.func.dfg.replace(inst).bitcast(ty, ival); } - -/// Expand the stack check instruction. -pub fn expand_stack_check( - inst: ir::Inst, - func: &mut ir::Function, - _cfg: &mut ControlFlowGraph, - isa: &TargetIsa, -) { - use ir::condcodes::IntCC; - - let gv = match func.dfg[inst] { - ir::InstructionData::UnaryGlobalVar { global_var, .. } => global_var, - _ => panic!("Want stack_check: {}", func.dfg.display_inst(inst, isa)), - }; - let ptr_ty = isa.pointer_type(); - - let mut pos = FuncCursor::new(func).at_inst(inst); - pos.use_srcloc(inst); - - let limit_addr = pos.ins().global_addr(ptr_ty, gv); - - let mut mflags = ir::MemFlags::new(); - mflags.set_aligned(); - mflags.set_notrap(); - let limit = pos.ins().load(ptr_ty, mflags, limit_addr, 0); - let cflags = pos.ins().ifcmp_sp(limit); - pos.func.dfg.replace(inst).trapif( - IntCC::UnsignedGreaterThanOrEqual, - cflags, - ir::TrapCode::StackOverflow, - ); -} From effe770c5f785d8ab86d0592f01363d617cab9c0 Mon Sep 17 00:00:00 2001 From: Sergey Pepyakin Date: Wed, 30 May 2018 23:48:04 +0200 Subject: [PATCH 1824/3084] Add stack_limit global_var in Function --- lib/codegen/src/ir/function.rs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs index d15e539214..c59c3ffdec 100644 --- a/lib/codegen/src/ir/function.rs +++ b/lib/codegen/src/ir/function.rs @@ -30,6 +30,10 @@ pub struct Function { /// Stack slots allocated in this function. pub stack_slots: StackSlots, + /// If not `None`, represents the address that the stack pointer should + /// be checked against. + pub stack_limit: Option, + /// Global variables referenced. pub global_vars: PrimaryMap, @@ -73,6 +77,7 @@ impl Function { name, signature: sig, stack_slots: StackSlots::new(), + stack_limit: None, global_vars: PrimaryMap::new(), heaps: PrimaryMap::new(), jump_tables: PrimaryMap::new(), From 73b6468d2553cc0df0bea3af8bf4d789ec55a19c Mon Sep 17 00:00:00 2001 From: Sergey Pepyakin Date: Thu, 31 May 2018 00:12:13 +0200 Subject: [PATCH 1825/3084] Parsing stack_limit --- cranelift/filetests/parser/preamble.cton | 18 ++++++++++++++++++ lib/codegen/src/ir/function.rs | 9 +++++++++ lib/reader/src/parser.rs | 21 +++++++++++++++++++++ 3 files changed, 48 insertions(+) create mode 100644 cranelift/filetests/parser/preamble.cton diff --git a/cranelift/filetests/parser/preamble.cton b/cranelift/filetests/parser/preamble.cton new file mode 100644 index 0000000000..00841c0faf --- /dev/null +++ b/cranelift/filetests/parser/preamble.cton @@ -0,0 +1,18 @@ +test cat + +; Verify parsing of stack_limit. +function %minimal(i64 vmctx) { +gv0 = vmctx +; Stack limit +stack_limit = gv0 + +ebb0: + trap user0 +} +; sameln: function %minimal(i64 vmctx) fast { +; nextln: gv0 = vmctx +; nextln: stack_limit = gv0 +; nextln: +; nextln: ebb0: +; nextln: trap user0 +; nextln: } diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs index c59c3ffdec..f7773f2258 100644 --- a/lib/codegen/src/ir/function.rs +++ b/lib/codegen/src/ir/function.rs @@ -126,6 +126,15 @@ impl Function { self.stack_slots.push(data) } + /// Sets the stack limit for the function. + /// + /// Returns previous one if any. + pub fn set_stack_limit(&mut self, stack_limit: Option) -> Option { + let prev = self.stack_limit.take(); + self.stack_limit = stack_limit; + prev + } + /// Adds a signature which can later be used to declare an external function import. pub fn import_signature(&mut self, signature: Signature) -> SigRef { self.dfg.signatures.push(signature) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 0f64885147..a6586fea74 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -244,6 +244,15 @@ impl<'a> Context<'a> { } } + // Assign the global for the stack limit. + fn set_stack_limit(&mut self, gv: GlobalVar, loc: &Location) -> Result<()> { + if let Some(_) = self.function.set_stack_limit(Some(gv)) { + err!(loc, "multiple stack_limit declarations") + } else { + Ok(()) + } + } + // Allocate a new EBB. fn add_ebb(&mut self, ebb: Ebb, loc: &Location) -> Result { while self.function.dfg.num_ebbs() <= ebb.index() { @@ -973,6 +982,7 @@ impl<'a> Parser<'a> { // * function-decl // * signature-decl // * jump-table-decl + // * stack-limit-decl // // The parsed decls are added to `ctx` rather than returned. fn parse_preamble(&mut self, ctx: &mut Context) -> Result<()> { @@ -1009,6 +1019,8 @@ impl<'a> Parser<'a> { self.parse_jump_table_decl() .and_then(|(jt, dat)| ctx.add_jt(jt, dat, &self.loc)) } + Some(Token::Identifier("stack_limit")) => self.parse_stack_limit_decl() + .and_then(|gv| ctx.set_stack_limit(gv, &self.loc)), // More to come.. _ => return Ok(()), }?; @@ -1291,6 +1303,15 @@ impl<'a> Parser<'a> { } } + /// stack-limit-decl ::= "stack_limit" "=" GlobalVar(gv) + fn parse_stack_limit_decl(&mut self) -> Result { + self.consume(); + self.match_token(Token::Equal, "expected '=' in stack limit declaration")?; + let gv = self.match_gv("expected global variable")?; + + Ok(gv) + } + // Parse a function body, add contents to `ctx`. // // function-body ::= * { extended-basic-block } From ddc8c914193095c73ff2f7674980bf5d09ac6e10 Mon Sep 17 00:00:00 2001 From: Sergey Pepyakin Date: Thu, 31 May 2018 00:12:22 +0200 Subject: [PATCH 1826/3084] Displaying stack_limit --- lib/codegen/src/write.rs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs index 1bd7bc3b3c..3f666fda24 100644 --- a/lib/codegen/src/write.rs +++ b/lib/codegen/src/write.rs @@ -79,6 +79,11 @@ fn write_preamble( writeln!(w, " {} = {}", jt, jt_data)?; } + if let Some(stack_limit) = func.stack_limit { + any = true; + writeln!(w, " stack_limit = {}", stack_limit)?; + } + Ok(any) } From f10682c2466da279dd4c03261a9d73ed49292eb7 Mon Sep 17 00:00:00 2001 From: Sergey Pepyakin Date: Wed, 6 Jun 2018 23:11:40 +0300 Subject: [PATCH 1827/3084] Add documentation. --- cranelift/docs/langref.rst | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 0eb2c0896c..e6d944dcd8 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -546,6 +546,20 @@ instructions before instruction selection:: When Cretonne code is running in a sandbox, it can also be necessary to include stack overflow checks in the prologue. +.. inst:: stack_limit = GV + + Set the stack limit of the current function. + + If set, in the preamble read the stack limit from ``GV`` and compare it to the stack pointer. If + the stack pointer has reached or exceeded the limit, generate a trap with a + ``stk_ovf`` code. + + The global variable must be accessible and naturally aligned for a + pointer-sized value. + + Setting `stack_limit` is an alternative way to detect stack overflow, when using + a calling convention that doesn't perform stack probes. + Global variables ---------------- From e4fe1db3b8dbb2aed52de71d9de1a2c7d27d69ba Mon Sep 17 00:00:00 2001 From: Sergey Pepyakin Date: Thu, 7 Jun 2018 00:06:27 +0300 Subject: [PATCH 1828/3084] Assert that stack_limit is none. --- lib/codegen/src/context.rs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index b9f7511865..cc8352774e 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -295,6 +295,10 @@ impl Context { /// Insert prologue and epilogues after computing the stack frame layout. pub fn prologue_epilogue(&mut self, isa: &TargetIsa) -> CtonResult { + assert!( + self.func.stack_limit.is_none(), + "stack_limit isn't implemented yet" + ); isa.prologue_epilogue(&mut self.func)?; self.verify_if(isa)?; self.verify_locations_if(isa)?; From fdf89edbfed8c803c76c540278d3d443f05b476a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Jun 2018 14:12:12 -0700 Subject: [PATCH 1829/3084] Fail loudly if a build.rs dependency path can't be printed. --- lib/codegen/build.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs index 7e58b0d012..d127f357b4 100644 --- a/lib/codegen/build.rs +++ b/lib/codegen/build.rs @@ -50,7 +50,7 @@ fn main() { // The `build.py` script prints out its own dependencies. println!( "cargo:rerun-if-changed={}", - crate_dir.join("build.rs").to_string_lossy() + crate_dir.join("build.rs").to_str().unwrap() ); // Scripts are in `$crate_dir/meta`. From 00d7d3a7749e7f405523f29e1ea3e636727fd3c7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Jun 2018 15:32:11 -0700 Subject: [PATCH 1830/3084] Implement `IntoIterator` for `PrimaryMap`. This makes iterating through a `PrimaryMap` more consistent with iterating through a `Vec`. --- lib/codegen/src/dce.rs | 2 +- lib/codegen/src/regalloc/pressure.rs | 2 +- lib/codegen/src/write.rs | 10 +++++----- lib/entity/src/primary.rs | 26 +++++++++++++++++++++++++- 4 files changed, 32 insertions(+), 8 deletions(-) diff --git a/lib/codegen/src/dce.rs b/lib/codegen/src/dce.rs index 797e905c6d..5373948e88 100644 --- a/lib/codegen/src/dce.rs +++ b/lib/codegen/src/dce.rs @@ -44,7 +44,7 @@ pub fn do_dce(func: &mut Function, domtree: &mut DominatorTree) { let mut live = Vec::with_capacity(func.dfg.num_values()); live.resize(func.dfg.num_values(), false); - for &ebb in domtree.cfg_postorder().iter() { + for &ebb in domtree.cfg_postorder() { let mut pos = FuncCursor::new(func).at_bottom(ebb); while let Some(inst) = pos.prev_inst() { { diff --git a/lib/codegen/src/regalloc/pressure.rs b/lib/codegen/src/regalloc/pressure.rs index a85bef28ca..65651987a7 100644 --- a/lib/codegen/src/regalloc/pressure.rs +++ b/lib/codegen/src/regalloc/pressure.rs @@ -88,7 +88,7 @@ impl Pressure { }; // Get the layout of aliasing top-level register classes from the register banks. - for bank in reginfo.banks.iter() { + for bank in reginfo.banks { let first = bank.first_toprc; let num = bank.num_toprcs; diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs index 3f666fda24..d7f6ce8747 100644 --- a/lib/codegen/src/write.rs +++ b/lib/codegen/src/write.rs @@ -50,31 +50,31 @@ fn write_preamble( writeln!(w, " {} = {}", ss, slot)?; } - for (gv, gv_data) in func.global_vars.iter() { + for (gv, gv_data) in &func.global_vars { any = true; writeln!(w, " {} = {}", gv, gv_data)?; } - for (heap, heap_data) in func.heaps.iter() { + for (heap, heap_data) in &func.heaps { any = true; writeln!(w, " {} = {}", heap, heap_data)?; } // Write out all signatures before functions since function declarations can refer to // signatures. - for (sig, sig_data) in func.dfg.signatures.iter() { + for (sig, sig_data) in &func.dfg.signatures { any = true; writeln!(w, " {} = {}", sig, sig_data.display(regs))?; } - for (fnref, ext_func) in func.dfg.ext_funcs.iter() { + for (fnref, ext_func) in &func.dfg.ext_funcs { any = true; if ext_func.signature != SigRef::reserved_value() { writeln!(w, " {} = {}", fnref, ext_func)?; } } - for (jt, jt_data) in func.jump_tables.iter() { + for (jt, jt_data) in &func.jump_tables { any = true; writeln!(w, " {} = {}", jt, jt_data)?; } diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs index 4f7307fb88..07e00cb884 100644 --- a/lib/entity/src/primary.rs +++ b/lib/entity/src/primary.rs @@ -121,6 +121,30 @@ where } } +impl<'a, K, V> IntoIterator for &'a PrimaryMap +where + K: EntityRef, +{ + type Item = (K, &'a V); + type IntoIter = Iter<'a, K, V>; + + fn into_iter(self) -> Self::IntoIter { + Iter::new(self.elems.iter()) + } +} + +impl<'a, K, V> IntoIterator for &'a mut PrimaryMap +where + K: EntityRef, +{ + type Item = (K, &'a mut V); + type IntoIter = IterMut<'a, K, V>; + + fn into_iter(self) -> Self::IntoIter { + IterMut::new(self.elems.iter_mut()) + } +} + #[cfg(test)] mod tests { use super::*; @@ -171,7 +195,7 @@ mod tests { m.push(33); let mut i = 0; - for (key, value) in m.iter() { + for (key, value) in &m { assert_eq!(key.index(), i); match i { 0 => assert_eq!(*value, 12), From 0daa560368d168ea6b992354771de976b22e72cf Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Jun 2018 15:56:23 -0700 Subject: [PATCH 1831/3084] Accept redundant alias definitions. In the text format, allow aliases to be defined multiple times, as long as they're always aliasing the same value. write.rs is already emitting redundant aliases, because it emits them at their uses, so this change allows the parser to be able to parse such code. --- cranelift/filetests/parser/alias.cton | 32 +++++++++++++++++++++++++++ lib/codegen/src/ir/dfg.rs | 12 ++++++++++ lib/reader/src/parser.rs | 31 +++++++++++++++++++++----- 3 files changed, 70 insertions(+), 5 deletions(-) create mode 100644 cranelift/filetests/parser/alias.cton diff --git a/cranelift/filetests/parser/alias.cton b/cranelift/filetests/parser/alias.cton new file mode 100644 index 0000000000..4e253bdb03 --- /dev/null +++ b/cranelift/filetests/parser/alias.cton @@ -0,0 +1,32 @@ +test cat +test verifier + +function %basic(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 -> v0 + v3 -> v1 + v4 = iadd.i32 v2, v3 + return v4 +} + +function %transitive() -> i32 { +ebb0: + v0 = iconst.i32 0 + v1 -> v0 + v2 -> v1 + v3 -> v2 + v4 -> v3 + return v4 +} + +function %duplicate(i32, i32) -> i32 { +ebb0(v0: i32, v1: i32): + v2 -> v0 + v2 -> v0 + v2 -> v0 + v3 -> v1 + v3 -> v1 + v3 -> v1 + v4 = iadd.i32 v2, v3 + return v4 +} diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs index 21c3cc9b66..9db7364002 100644 --- a/lib/codegen/src/ir/dfg.rs +++ b/lib/codegen/src/ir/dfg.rs @@ -949,6 +949,18 @@ impl DataFlowGraph { self.values[dest] = data; } + /// If `v` is already defined as an alias, return its destination value. + /// Otherwise return None. This allows the parser to coalesce identical + /// alias definitions. + #[cold] + pub fn value_alias_dest_for_parser(&self, v: Value) -> Option { + if let ValueData::Alias { original, .. } = self.values[v] { + Some(original) + } else { + None + } + } + /// Compute the type of an alias. This is only for use in the parser. /// Returns false if an alias cycle was encountered. #[cold] diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index a6586fea74..256ba50847 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1580,13 +1580,34 @@ impl<'a> Parser<'a> { if results.len() != 1 { return err!(self.loc, "wrong number of aliases"); } + let result = results[0]; let dest = self.match_value("expected value alias")?; - ctx.function - .dfg - .make_value_alias_for_parser(dest, results[0]); - ctx.map.def_value(results[0], &self.loc)?; - ctx.aliases.push(results[0]); + // Allow duplicate definitions of aliases, as long as they are identical. + if ctx.map.contains_value(result) { + if let Some(old) = ctx.function.dfg.value_alias_dest_for_parser(result) { + if old != dest { + return err!( + self.loc, + "value {} is already defined as an alias with destination {}", + result, + old + ); + } + } else { + return err!(self.loc, "value {} is already defined"); + } + } else { + ctx.map.def_value(result, &self.loc)?; + } + + if !ctx.map.contains_value(dest) { + return err!(self.loc, "value {} is not yet defined", dest); + } + + ctx.function.dfg.make_value_alias_for_parser(dest, result); + + ctx.aliases.push(result); Ok(()) } From c03d67a4ff74a018069be37854d22beb72c0aa39 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Jun 2018 17:01:41 -0700 Subject: [PATCH 1832/3084] Update for wasm memory instruction renaming. The wasm spec renamed `grow_memory` and `current_memory` to `memory.grow` and `memory.size`, respectively. Update cretonne's names to follow. Also, update to wasmparser 0.17.0, which also renames its corresponding operators. --- lib/wasm/Cargo.toml | 2 +- lib/wasm/src/code_translator.rs | 10 +++++----- lib/wasm/src/environ/dummy.rs | 4 ++-- lib/wasm/src/environ/spec.rs | 8 ++++---- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index ebecc70999..60fa48a776 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -9,7 +9,7 @@ readme = "README.md" keywords = ["webassembly", "wasm"] [dependencies] -wasmparser = { version = "0.16.1", default-features = false } +wasmparser = { version = "0.17.0", default-features = false } cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } cretonne-frontend = { path = "../frontend", version = "0.9.0", default-features = false } hashmap_core = { version = "0.1.6", optional = true } diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 1d5386184e..4b92c2ccc3 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -16,7 +16,7 @@ //! //! - the loads and stores need the memory base address; //! - the `get_global` et `set_global` instructions depends on how the globals are implemented; -//! - `current_memory` and `grow_memory` are runtime functions; +//! - `memory.size` and `memory.grow` are runtime functions; //! - `call_indirect` has to translate the function index into the address of where this //! is; //! @@ -398,18 +398,18 @@ pub fn translate_operator( * Memory management is handled by environment. It is usually translated into calls to * special functions. ************************************************************************************/ - Operator::GrowMemory { reserved } => { + Operator::MemoryGrow { reserved } => { // The WebAssembly MVP only supports one linear memory, but we expect the reserved // argument to be a memory index. let heap_index = reserved as MemoryIndex; let heap = state.get_heap(builder.func, reserved, environ); let val = state.pop1(); - state.push1(environ.translate_grow_memory(builder.cursor(), heap_index, heap, val)?) + state.push1(environ.translate_memory_grow(builder.cursor(), heap_index, heap, val)?) } - Operator::CurrentMemory { reserved } => { + Operator::MemorySize { reserved } => { let heap_index = reserved as MemoryIndex; let heap = state.get_heap(builder.func, reserved, environ); - state.push1(environ.translate_current_memory(builder.cursor(), heap_index, heap)?); + state.push1(environ.translate_memory_size(builder.cursor(), heap_index, heap)?); } /******************************* Load instructions *********************************** * Wasm specifies an integer alignment flag but we drop it in Cretonne. diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 4794524b5a..7b71f8b69e 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -261,7 +261,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ Ok(pos.ins().Call(ir::Opcode::Call, VOID, callee, args).0) } - fn translate_grow_memory( + fn translate_memory_grow( &mut self, mut pos: FuncCursor, _index: MemoryIndex, @@ -271,7 +271,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ Ok(pos.ins().iconst(I32, -1)) } - fn translate_current_memory( + fn translate_memory_size( &mut self, mut pos: FuncCursor, _index: MemoryIndex, diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 5b4ca58301..00882d035f 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -163,7 +163,7 @@ pub trait FuncEnvironment { Ok(pos.ins().call(callee, call_args)) } - /// Translate a `grow_memory` WebAssembly instruction. + /// Translate a `memory.grow` WebAssembly instruction. /// /// The `index` provided identifies the linear memory to grow, and `heap` is the heap reference /// returned by `make_heap` for the same index. @@ -171,7 +171,7 @@ pub trait FuncEnvironment { /// The `val` value is the requested memory size in pages. /// /// Returns the old size (in pages) of the memory. - fn translate_grow_memory( + fn translate_memory_grow( &mut self, pos: FuncCursor, index: MemoryIndex, @@ -179,13 +179,13 @@ pub trait FuncEnvironment { val: ir::Value, ) -> WasmResult; - /// Translates a `current_memory` WebAssembly instruction. + /// Translates a `memory.size` WebAssembly instruction. /// /// The `index` provided identifies the linear memory to query, and `heap` is the heap reference /// returned by `make_heap` for the same index. /// /// Returns the size in pages of the memory. - fn translate_current_memory( + fn translate_memory_size( &mut self, pos: FuncCursor, index: MemoryIndex, From 967da591813ed71a16fad1e7a183ac48b10c841e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 11 Jun 2018 13:48:35 -0700 Subject: [PATCH 1833/3084] Fix shellcheck warnings in shell scripts. --- clippy-all.sh | 2 +- cranelift/format-all.sh | 3 ++- cranelift/publish-all.sh | 10 ++-------- cranelift/test-all.sh | 26 +++++++++++++------------- cranelift/test-no_std.sh | 6 +++--- lib/codegen/meta/check.sh | 5 +++-- 6 files changed, 24 insertions(+), 28 deletions(-) diff --git a/clippy-all.sh b/clippy-all.sh index 6ec2fd1411..f82f6a07db 100755 --- a/clippy-all.sh +++ b/clippy-all.sh @@ -4,4 +4,4 @@ set -euo pipefail # Check all sources with clippy. # In the cton-util crate (root dir) clippy will only work with nightly cargo - # there is a bug where it will reject the commands passed to it by cargo 0.25.0 -cargo +nightly clippy --all +exec cargo +nightly clippy --all diff --git a/cranelift/format-all.sh b/cranelift/format-all.sh index 1e27718801..64c967ad24 100755 --- a/cranelift/format-all.sh +++ b/cranelift/format-all.sh @@ -3,7 +3,8 @@ set -euo pipefail # Format all sources using rustfmt. -cd $(dirname "$0") +topdir=$(dirname "$0") +cd "$topdir" # Make sure we can find rustfmt. export PATH="$PATH:$HOME/.cargo/bin" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index aa99a28872..543f9bbeca 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -1,7 +1,7 @@ #!/bin/bash set -euo pipefail -cd $(dirname "$0") -topdir="$(pwd)" +topdir=$(dirname "$0") +cd "$topdir" # All the cretonne-* crates have the same version number version="0.9.0" @@ -28,12 +28,6 @@ cargo update echo git commit -a -m "\"Bump version to $version"\" echo git push for crate in entity codegen frontend native reader wasm module simplejit faerie umbrella ; do - if [ "$crate" == "umbrella" ]; then - dir="cretonne" - else - dir="$crate" - fi - echo cargo publish --manifest-path "lib/$crate/Cargo.toml" done echo diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index fcdf7e57c4..5c280523d3 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -15,18 +15,18 @@ set -euo pipefail export PYTHONDONTWRITEBYTECODE=1 # Repository top-level directory. -cd $(dirname "$0") -topdir=$(pwd) +topdir=$(dirname "$0") +cd "$topdir" function banner { - echo "====== $@ ======" + echo "====== $* ======" } # Run rustfmt if we have it. banner "Rust formatting" -if command -v rustfmt > /dev/null; then +if type rustfmt > /dev/null; then # In newer versions of rustfmt, replace --write-mode=diff with --check. - if ! $topdir/format-all.sh --write-mode=diff ; then + if ! "$topdir/format-all.sh" --write-mode=diff ; then echo "Formatting diffs detected! Run \"cargo fmt --all\" to correct." exit 1 fi @@ -39,16 +39,16 @@ else fi # Check if any Python files have changed since we last checked them. -tsfile=$topdir/target/meta-checked -if [ -f $tsfile ]; then - needcheck=$(find $topdir/lib/codegen/meta -name '*.py' -newer $tsfile) +tsfile="$topdir/target/meta-checked" +if [ -f "$tsfile" ]; then + needcheck=$(find "$topdir/lib/codegen/meta" -name '*.py' -newer "$tsfile") else needcheck=yes fi if [ -n "$needcheck" ]; then banner "$(python --version 2>&1), $(python3 --version 2>&1)" - $topdir/lib/codegen/meta/check.sh - touch $tsfile || echo no target directory + "$topdir/lib/codegen/meta/check.sh" + touch "$tsfile" || echo no target directory fi # Make sure the code builds in release mode. @@ -69,8 +69,8 @@ cargo doc # Run clippy if we have it. banner "Rust linter" -if $topdir/check-clippy.sh; then - $topdir/clippy-all.sh --write-mode=diff +if "$topdir/check-clippy.sh"; then + "$topdir/clippy-all.sh" --write-mode=diff else echo "\`cargo +nightly install clippy\` for optional rust linting" fi @@ -85,7 +85,7 @@ if rustup toolchain list | grep -q nightly; then echo "installing cargo-fuzz" cargo +nightly install cargo-fuzz fi - ASAN_OPTIONS=detect_leaks=0 cargo +nightly fuzz run fuzz_translate_module $topdir/fuzz/corpus/fuzz_translate_module/ffaefab69523eb11935a9b420d58826c8ea65c4c + ASAN_OPTIONS=detect_leaks=0 cargo +nightly fuzz run fuzz_translate_module "$topdir/fuzz/corpus/fuzz_translate_module/ffaefab69523eb11935a9b420d58826c8ea65c4c" else echo "nightly toolchain not found, skipping fuzz target integration test" fi diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh index 8ca93f5d04..26678c856e 100755 --- a/cranelift/test-no_std.sh +++ b/cranelift/test-no_std.sh @@ -5,11 +5,11 @@ set -euo pipefail # packages which support it. # Repository top-level directory. -cd $(dirname "$0") -topdir=$(pwd) +topdir=$(dirname "$0") +cd "$topdir" function banner { - echo "====== $@ ======" + echo "====== $* ======" } # Test those packages which have no_std support. diff --git a/lib/codegen/meta/check.sh b/lib/codegen/meta/check.sh index d8f2178f43..ce34f78f4b 100755 --- a/lib/codegen/meta/check.sh +++ b/lib/codegen/meta/check.sh @@ -1,9 +1,10 @@ #!/bin/bash set -euo pipefail -cd $(dirname "$0") +topdir=$(dirname "$0") +cd "$topdir" function runif { - if command -v "$1" > /dev/null; then + if type "$1" > /dev/null; then echo " === $1 ===" "$@" else From de2a2fe4184410e231b906a6953b23af0e97f740 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 11 Jun 2018 16:03:18 -0700 Subject: [PATCH 1834/3084] Minor code cleanup. --- lib/codegen/src/abi.rs | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/lib/codegen/src/abi.rs b/lib/codegen/src/abi.rs index fe69c67d35..1a86295b8b 100644 --- a/lib/codegen/src/abi.rs +++ b/lib/codegen/src/abi.rs @@ -107,11 +107,9 @@ pub fn legalize_args(args: &mut Vec, aa: &mut AA) { } // Split this argument into two smaller ones. Then revisit both. ArgAction::Convert(conv) => { - let new_arg = AbiParam { - value_type: conv.apply(arg.value_type), - ..arg - }; - args[argno].value_type = new_arg.value_type; + let value_type = conv.apply(arg.value_type); + let new_arg = AbiParam { value_type, ..arg }; + args[argno].value_type = value_type; if conv.is_split() { args.insert(argno + 1, new_arg); } From 3027579cbe667f009d6772670f46983ba306bc87 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 11 Jun 2018 16:04:46 -0700 Subject: [PATCH 1835/3084] Add a test for instruction shrinking. --- .../filetests/isa/x86/shrink-multiple-uses.cton | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 cranelift/filetests/isa/x86/shrink-multiple-uses.cton diff --git a/cranelift/filetests/isa/x86/shrink-multiple-uses.cton b/cranelift/filetests/isa/x86/shrink-multiple-uses.cton new file mode 100644 index 0000000000..d20cc78066 --- /dev/null +++ b/cranelift/filetests/isa/x86/shrink-multiple-uses.cton @@ -0,0 +1,15 @@ +test shrink +set opt_level=best +target x86_64 + +function %test_multiple_uses(i32 [%rdi]) -> i32 { +ebb0(v0: i32 [%rdi]): +[Op1rcmp_ib#7083,%rflags] v3 = ifcmp_imm v0, 0 +[Op2seti_abcd#490,%rax] v1 = trueif eq v3 +[RexOp2urm_noflags#4b6,%rax] v2 = bint.i32 v1 +[Op1brib#70] brif eq v3, ebb1 +[Op1ret#c3] return v2 + +ebb1: +[Op2trap#40b] trap user0 +} From ffe57d9051d50e3380cb3aee6ad281f8a9efe4d9 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 11 Jun 2018 20:24:18 -0700 Subject: [PATCH 1836/3084] Fix a typo in a comment. --- lib/codegen/src/isa/mod.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs index b68f9fecb7..34733b92f7 100644 --- a/lib/codegen/src/isa/mod.rs +++ b/lib/codegen/src/isa/mod.rs @@ -246,8 +246,8 @@ pub trait TargetIsa: fmt::Display { /// /// The legalizer will adapt argument and return values as necessary at all ABI boundaries. /// - /// When this function is called to legalize the signature of the function currently begin - /// compiler, `current` is true. The legalized signature can then also contain special purpose + /// When this function is called to legalize the signature of the function currently being + /// compiled, `current` is true. The legalized signature can then also contain special purpose /// arguments and return values such as: /// /// - A `link` argument representing the link registers on RISC architectures that don't push From 683880bd02a284cd917b1eb18c0cc1a4d442594e Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Jun 2018 04:44:07 -0700 Subject: [PATCH 1837/3084] Avoid uninteresting renaming of imports. --- lib/filetests/src/test_cat.rs | 6 +++--- lib/filetests/src/test_print_cfg.rs | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/filetests/src/test_cat.rs b/lib/filetests/src/test_cat.rs index aaecb2d735..c3b1a1aa4a 100644 --- a/lib/filetests/src/test_cat.rs +++ b/lib/filetests/src/test_cat.rs @@ -3,7 +3,7 @@ use cretonne_codegen::ir::Function; use cretonne_reader::TestCommand; use std::borrow::Cow; -use subtest::{self, Context, SubTest, SubtestResult as STResult}; +use subtest::{self, Context, SubTest, SubtestResult}; /// Object implementing the `test cat` sub-test. /// @@ -13,7 +13,7 @@ use subtest::{self, Context, SubTest, SubtestResult as STResult}; /// The result is verified by filecheck. struct TestCat; -pub fn subtest(parsed: &TestCommand) -> STResult> { +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { assert_eq!(parsed.command, "cat"); if !parsed.options.is_empty() { Err(format!("No options allowed on {}", parsed)) @@ -31,7 +31,7 @@ impl SubTest for TestCat { false } - fn run(&self, func: Cow, context: &Context) -> STResult<()> { + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { subtest::run_filecheck(&func.display(context.isa).to_string(), context) } } diff --git a/lib/filetests/src/test_print_cfg.rs b/lib/filetests/src/test_print_cfg.rs index d211381886..5f196d6228 100644 --- a/lib/filetests/src/test_print_cfg.rs +++ b/lib/filetests/src/test_print_cfg.rs @@ -8,12 +8,12 @@ use std::borrow::Cow; use cretonne_codegen::cfg_printer::CFGPrinter; use cretonne_codegen::ir::Function; use cretonne_reader::TestCommand; -use subtest::{self, Context, SubTest, SubtestResult as STResult}; +use subtest::{self, Context, SubTest, SubtestResult}; /// Object implementing the `test print-cfg` sub-test. struct TestPrintCfg; -pub fn subtest(parsed: &TestCommand) -> STResult> { +pub fn subtest(parsed: &TestCommand) -> SubtestResult> { assert_eq!(parsed.command, "print-cfg"); if !parsed.options.is_empty() { Err(format!("No options allowed on {}", parsed)) @@ -31,7 +31,7 @@ impl SubTest for TestPrintCfg { false } - fn run(&self, func: Cow, context: &Context) -> STResult<()> { + fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { subtest::run_filecheck(&CFGPrinter::new(&func).to_string(), context) } } From 13f22065a261c25872bf1d2009ca34d8d9b3b075 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Jun 2018 14:57:52 -0700 Subject: [PATCH 1838/3084] Rename verifier's `Result` and `Error`. This provides consistency with similar types in other parts of Cretonne, and avoids shadowing `Result` from the standard prelude. --- lib/codegen/src/context.rs | 10 ++-- lib/codegen/src/print_errors.rs | 4 +- lib/codegen/src/result.rs | 8 +-- lib/codegen/src/verifier/cssa.rs | 8 +-- lib/codegen/src/verifier/flags.rs | 11 ++-- lib/codegen/src/verifier/liveness.rs | 10 ++-- lib/codegen/src/verifier/locations.rs | 25 ++++---- lib/codegen/src/verifier/mod.rs | 82 ++++++++++++++------------- 8 files changed, 82 insertions(+), 76 deletions(-) diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index cc8352774e..83dd3d0368 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -28,7 +28,7 @@ use simple_gvn::do_simple_gvn; use std::vec::Vec; use timing; use unreachable_code::eliminate_unreachable_code; -use verifier; +use verifier::{verify_context, verify_locations, VerifierResult}; /// Persistent data structures and compilation pipeline. pub struct Context { @@ -174,8 +174,8 @@ impl Context { /// Run the verifier on the function. /// /// Also check that the dominator tree and control flow graph are consistent with the function. - pub fn verify<'a, FOI: Into>>(&self, fisa: FOI) -> verifier::Result { - verifier::verify_context(&self.func, &self.cfg, &self.domtree, fisa) + pub fn verify<'a, FOI: Into>>(&self, fisa: FOI) -> VerifierResult<()> { + verify_context(&self.func, &self.cfg, &self.domtree, fisa) } /// Run the verifier only if the `enable_verifier` setting is true. @@ -189,8 +189,8 @@ impl Context { } /// Run the locations verifier on the function. - pub fn verify_locations(&self, isa: &TargetIsa) -> verifier::Result { - verifier::verify_locations(isa, &self.func, None) + pub fn verify_locations(&self, isa: &TargetIsa) -> VerifierResult<()> { + verify_locations(isa, &self.func, None) } /// Run the locations verifier only if the `enable_verifier` setting is true. diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs index ffc8bab4ec..283b3d6d08 100644 --- a/lib/codegen/src/print_errors.rs +++ b/lib/codegen/src/print_errors.rs @@ -5,13 +5,13 @@ use isa::TargetIsa; use result::CtonError; use std::fmt::Write; use std::string::{String, ToString}; -use verifier; +use verifier::VerifierError; /// Pretty-print a verifier error. pub fn pretty_verifier_error( func: &ir::Function, isa: Option<&TargetIsa>, - err: &verifier::Error, + err: &VerifierError, ) -> String { let mut msg = err.to_string(); match err.location { diff --git a/lib/codegen/src/result.rs b/lib/codegen/src/result.rs index 483b975ff4..1821c2e3b5 100644 --- a/lib/codegen/src/result.rs +++ b/lib/codegen/src/result.rs @@ -1,6 +1,6 @@ //! Result and error types representing the outcome of compiling a function. -use verifier; +use verifier::VerifierError; /// A compilation error. /// @@ -12,7 +12,7 @@ pub enum CtonError { /// This always represents a bug, either in the code that generated IR for Cretonne, or a bug /// in Cretonne itself. #[fail(display = "Verifier error: {}", _0)] - Verifier(#[cause] verifier::Error), + Verifier(#[cause] VerifierError), /// An implementation limit was exceeded. /// @@ -34,8 +34,8 @@ pub enum CtonError { /// A Cretonne compilation result. pub type CtonResult = Result<(), CtonError>; -impl From for CtonError { - fn from(e: verifier::Error) -> Self { +impl From for CtonError { + fn from(e: VerifierError) -> Self { CtonError::Verifier(e) } } diff --git a/lib/codegen/src/verifier/cssa.rs b/lib/codegen/src/verifier/cssa.rs index 5d0e2be87f..b9ef1bad4c 100644 --- a/lib/codegen/src/verifier/cssa.rs +++ b/lib/codegen/src/verifier/cssa.rs @@ -7,7 +7,7 @@ use ir::{ExpandedProgramPoint, Function}; use regalloc::liveness::Liveness; use regalloc::virtregs::VirtRegs; use timing; -use verifier::Result; +use verifier::VerifierResult; /// Verify conventional SSA form for `func`. /// @@ -29,7 +29,7 @@ pub fn verify_cssa( domtree: &DominatorTree, liveness: &Liveness, virtregs: &VirtRegs, -) -> Result { +) -> VerifierResult<()> { let _tt = timing::verify_cssa(); let mut preorder = DominatorTreePreorder::new(); @@ -58,7 +58,7 @@ struct CssaVerifier<'a> { } impl<'a> CssaVerifier<'a> { - fn check_virtregs(&self) -> Result { + fn check_virtregs(&self) -> VerifierResult<()> { for vreg in self.virtregs.all_virtregs() { let values = self.virtregs.values(vreg); @@ -135,7 +135,7 @@ impl<'a> CssaVerifier<'a> { Ok(()) } - fn check_cssa(&self) -> Result { + fn check_cssa(&self) -> VerifierResult<()> { for ebb in self.func.layout.ebbs() { let ebb_params = self.func.dfg.ebb_params(ebb); for (_, pred) in self.cfg.pred_iter(ebb) { diff --git a/lib/codegen/src/verifier/flags.rs b/lib/codegen/src/verifier/flags.rs index 582ae525b8..05ce764310 100644 --- a/lib/codegen/src/verifier/flags.rs +++ b/lib/codegen/src/verifier/flags.rs @@ -6,9 +6,8 @@ use ir; use ir::instructions::BranchInfo; use isa; use packed_option::PackedOption; -use std::result; use timing; -use verifier::{Error, Result}; +use verifier::VerifierResult; /// Verify that CPU flags are used correctly. /// @@ -26,7 +25,7 @@ pub fn verify_flags( func: &ir::Function, cfg: &ControlFlowGraph, isa: Option<&isa::TargetIsa>, -) -> Result { +) -> VerifierResult<()> { let _tt = timing::verify_flags(); let mut verifier = FlagsVerifier { func, @@ -47,7 +46,7 @@ struct FlagsVerifier<'a> { } impl<'a> FlagsVerifier<'a> { - fn check(&mut self) -> Result { + fn check(&mut self) -> VerifierResult<()> { // List of EBBs that need to be processed. EBBs may be re-added to this list when we detect // that one of their successor blocks needs a live-in flags value. let mut worklist = SparseSet::new(); @@ -81,7 +80,7 @@ impl<'a> FlagsVerifier<'a> { } /// Check flags usage in `ebb` and return the live-in flags value, if any. - fn visit_ebb(&self, ebb: ir::Ebb) -> result::Result, Error> { + fn visit_ebb(&self, ebb: ir::Ebb) -> VerifierResult> { // The single currently live flags value. let mut live_val = None; @@ -139,7 +138,7 @@ impl<'a> FlagsVerifier<'a> { } // Merge live flags values, or return an error on conflicting values. -fn merge(a: &mut Option, b: ir::Value, inst: ir::Inst) -> Result { +fn merge(a: &mut Option, b: ir::Value, inst: ir::Inst) -> VerifierResult<()> { if let Some(va) = *a { if b != va { return err!(inst, "conflicting live CPU flags: {} and {}", va, b); diff --git a/lib/codegen/src/verifier/liveness.rs b/lib/codegen/src/verifier/liveness.rs index 167225f24f..e2bf6a21ef 100644 --- a/lib/codegen/src/verifier/liveness.rs +++ b/lib/codegen/src/verifier/liveness.rs @@ -8,7 +8,7 @@ use regalloc::liveness::Liveness; use regalloc::liverange::LiveRange; use std::cmp::Ordering; use timing; -use verifier::Result; +use verifier::VerifierResult; /// Verify liveness information for `func`. /// @@ -27,7 +27,7 @@ pub fn verify_liveness( func: &Function, cfg: &ControlFlowGraph, liveness: &Liveness, -) -> Result { +) -> VerifierResult<()> { let _tt = timing::verify_liveness(); let verifier = LivenessVerifier { isa, @@ -49,7 +49,7 @@ struct LivenessVerifier<'a> { impl<'a> LivenessVerifier<'a> { /// Check all EBB arguments. - fn check_ebbs(&self) -> Result { + fn check_ebbs(&self) -> VerifierResult<()> { for ebb in self.func.layout.ebbs() { for &val in self.func.dfg.ebb_params(ebb) { let lr = match self.liveness.get(val) { @@ -63,7 +63,7 @@ impl<'a> LivenessVerifier<'a> { } /// Check all instructions. - fn check_insts(&self) -> Result { + fn check_insts(&self) -> VerifierResult<()> { for ebb in self.func.layout.ebbs() { for inst in self.func.layout.ebb_insts(ebb) { let encoding = self.func.encodings[inst]; @@ -141,7 +141,7 @@ impl<'a> LivenessVerifier<'a> { } /// Check the integrity of the live range `lr`. - fn check_lr(&self, def: ProgramPoint, val: Value, lr: &LiveRange) -> Result { + fn check_lr(&self, def: ProgramPoint, val: Value, lr: &LiveRange) -> VerifierResult<()> { let l = &self.func.layout; let loc: AnyEntity = match def.into() { diff --git a/lib/codegen/src/verifier/locations.rs b/lib/codegen/src/verifier/locations.rs index 5460b938dd..d7378f50b7 100644 --- a/lib/codegen/src/verifier/locations.rs +++ b/lib/codegen/src/verifier/locations.rs @@ -5,7 +5,7 @@ use isa; use regalloc::liveness::Liveness; use regalloc::RegDiversions; use timing; -use verifier::Result; +use verifier::VerifierResult; /// Verify value locations for `func`. /// @@ -22,7 +22,7 @@ pub fn verify_locations( isa: &isa::TargetIsa, func: &ir::Function, liveness: Option<&Liveness>, -) -> Result { +) -> VerifierResult<()> { let _tt = timing::verify_locations(); let verifier = LocationVerifier { isa, @@ -45,7 +45,7 @@ struct LocationVerifier<'a> { impl<'a> LocationVerifier<'a> { /// Check that the assigned value locations match the operand constraints of their uses. - fn check_constraints(&self) -> Result { + fn check_constraints(&self) -> VerifierResult<()> { let dfg = &self.func.dfg; let mut divert = RegDiversions::new(); @@ -88,7 +88,7 @@ impl<'a> LocationVerifier<'a> { inst: ir::Inst, enc: isa::Encoding, divert: &RegDiversions, - ) -> Result { + ) -> VerifierResult<()> { let constraints = self.encinfo .operand_constraints(enc) .expect("check_enc_constraints requires a legal encoding"); @@ -107,7 +107,7 @@ impl<'a> LocationVerifier<'a> { /// Check that the result values produced by a ghost instruction are not assigned a value /// location. - fn check_ghost_results(&self, inst: ir::Inst) -> Result { + fn check_ghost_results(&self, inst: ir::Inst) -> VerifierResult<()> { let results = self.func.dfg.inst_results(inst); for &res in results { @@ -126,7 +126,12 @@ impl<'a> LocationVerifier<'a> { } /// Check the ABI argument and result locations for a call. - fn check_call_abi(&self, inst: ir::Inst, sig: ir::SigRef, divert: &RegDiversions) -> Result { + fn check_call_abi( + &self, + inst: ir::Inst, + sig: ir::SigRef, + divert: &RegDiversions, + ) -> VerifierResult<()> { let sig = &self.func.dfg.signatures[sig]; let varargs = self.func.dfg.inst_variable_args(inst); let results = self.func.dfg.inst_results(inst); @@ -155,7 +160,7 @@ impl<'a> LocationVerifier<'a> { } /// Check the ABI argument locations for a return. - fn check_return_abi(&self, inst: ir::Inst, divert: &RegDiversions) -> Result { + fn check_return_abi(&self, inst: ir::Inst, divert: &RegDiversions) -> VerifierResult<()> { let sig = &self.func.signature; let varargs = self.func.dfg.inst_variable_args(inst); @@ -180,7 +185,7 @@ impl<'a> LocationVerifier<'a> { abi: &ir::AbiParam, loc: ir::ValueLoc, want_kind: ir::StackSlotKind, - ) -> Result { + ) -> VerifierResult<()> { match abi.location { ir::ArgumentLoc::Unassigned => {} ir::ArgumentLoc::Reg(reg) => { @@ -233,7 +238,7 @@ impl<'a> LocationVerifier<'a> { } /// Update diversions to reflect the current instruction and check their consistency. - fn update_diversions(&self, inst: ir::Inst, divert: &mut RegDiversions) -> Result { + fn update_diversions(&self, inst: ir::Inst, divert: &mut RegDiversions) -> VerifierResult<()> { let (arg, src) = match self.func.dfg[inst] { ir::InstructionData::RegMove { arg, src, .. } | ir::InstructionData::RegSpill { arg, src, .. } => (arg, ir::ValueLoc::Reg(src)), @@ -264,7 +269,7 @@ impl<'a> LocationVerifier<'a> { /// We have active diversions before a branch. Make sure none of the diverted values are live /// on the outgoing CFG edges. - fn check_cfg_edges(&self, inst: ir::Inst, divert: &RegDiversions) -> Result { + fn check_cfg_edges(&self, inst: ir::Inst, divert: &RegDiversions) -> VerifierResult<()> { use ir::instructions::BranchInfo::*; // We can only check CFG edges if we have a liveness analysis. diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs index 20507594e3..12643fc61b 100644 --- a/lib/codegen/src/verifier/mod.rs +++ b/lib/codegen/src/verifier/mod.rs @@ -71,7 +71,6 @@ use settings::{Flags, FlagsOrIsa}; use std::cmp::Ordering; use std::collections::BTreeSet; use std::fmt::{self, Display, Formatter, Write}; -use std::result; use std::string::String; use std::vec::Vec; use timing; @@ -80,17 +79,17 @@ pub use self::cssa::verify_cssa; pub use self::liveness::verify_liveness; pub use self::locations::verify_locations; -// Create an `Err` variant of `Result` from a location and `format!` arguments. +// Create an `Err` variant of `VerifierResult` from a location and `format!` arguments. macro_rules! err { ( $loc:expr, $msg:expr ) => { - Err(::verifier::Error { + Err(::verifier::VerifierError { location: $loc.into(), message: String::from($msg), }) }; ( $loc:expr, $fmt:expr, $( $arg:expr ),+ ) => { - Err(::verifier::Error { + Err(::verifier::VerifierError { location: $loc.into(), message: format!( $fmt, $( $arg ),+ ), }) @@ -104,24 +103,27 @@ mod locations; /// A verifier error. #[derive(Fail, Debug, PartialEq, Eq)] -pub struct Error { +pub struct VerifierError { /// The entity causing the verifier error. pub location: AnyEntity, /// Error message. pub message: String, } -impl Display for Error { +impl Display for VerifierError { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "{}: {}", self.location, self.message) } } /// Verifier result. -pub type Result = result::Result<(), Error>; +pub type VerifierResult = Result; /// Verify `func`. -pub fn verify_function<'a, FOI: Into>>(func: &Function, fisa: FOI) -> Result { +pub fn verify_function<'a, FOI: Into>>( + func: &Function, + fisa: FOI, +) -> VerifierResult<()> { let _tt = timing::verifier(); Verifier::new(func, fisa.into()).run() } @@ -133,7 +135,7 @@ pub fn verify_context<'a, FOI: Into>>( cfg: &ControlFlowGraph, domtree: &DominatorTree, fisa: FOI, -) -> Result { +) -> VerifierResult<()> { let _tt = timing::verifier(); let verifier = Verifier::new(func, fisa.into()); if cfg.is_valid() { @@ -167,7 +169,7 @@ impl<'a> Verifier<'a> { } // Check for cycles in the global variable declarations. - fn verify_global_vars(&self) -> Result { + fn verify_global_vars(&self) -> VerifierResult<()> { let mut seen = SparseSet::new(); for gv in self.func.global_vars.keys() { @@ -187,7 +189,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn ebb_integrity(&self, ebb: Ebb, inst: Inst) -> Result { + fn ebb_integrity(&self, ebb: Ebb, inst: Inst) -> VerifierResult<()> { let is_terminator = self.func.dfg[inst].opcode().is_terminator(); let is_last_inst = self.func.layout.last_inst(ebb) == Some(inst); @@ -226,7 +228,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn instruction_integrity(&self, inst: Inst) -> Result { + fn instruction_integrity(&self, inst: Inst) -> VerifierResult<()> { let inst_data = &self.func.dfg[inst]; let dfg = &self.func.dfg; @@ -255,7 +257,7 @@ impl<'a> Verifier<'a> { self.verify_entity_references(inst) } - fn verify_entity_references(&self, inst: Inst) -> Result { + fn verify_entity_references(&self, inst: Inst) -> VerifierResult<()> { use ir::instructions::InstructionData::*; for &arg in self.func.dfg.inst_args(inst) { @@ -375,7 +377,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn verify_ebb(&self, inst: Inst, e: Ebb) -> Result { + fn verify_ebb(&self, inst: Inst, e: Ebb) -> VerifierResult<()> { if !self.func.dfg.ebb_is_valid(e) || !self.func.layout.is_ebb_inserted(e) { return err!(inst, "invalid ebb reference {}", e); } @@ -387,7 +389,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn verify_sig_ref(&self, inst: Inst, s: SigRef) -> Result { + fn verify_sig_ref(&self, inst: Inst, s: SigRef) -> VerifierResult<()> { if !self.func.dfg.signatures.is_valid(s) { err!(inst, "invalid signature reference {}", s) } else { @@ -395,7 +397,7 @@ impl<'a> Verifier<'a> { } } - fn verify_func_ref(&self, inst: Inst, f: FuncRef) -> Result { + fn verify_func_ref(&self, inst: Inst, f: FuncRef) -> VerifierResult<()> { if !self.func.dfg.ext_funcs.is_valid(f) { err!(inst, "invalid function reference {}", f) } else { @@ -403,7 +405,7 @@ impl<'a> Verifier<'a> { } } - fn verify_stack_slot(&self, inst: Inst, ss: StackSlot) -> Result { + fn verify_stack_slot(&self, inst: Inst, ss: StackSlot) -> VerifierResult<()> { if !self.func.stack_slots.is_valid(ss) { err!(inst, "invalid stack slot {}", ss) } else { @@ -411,7 +413,7 @@ impl<'a> Verifier<'a> { } } - fn verify_global_var(&self, inst: Inst, gv: GlobalVar) -> Result { + fn verify_global_var(&self, inst: Inst, gv: GlobalVar) -> VerifierResult<()> { if !self.func.global_vars.is_valid(gv) { err!(inst, "invalid global variable {}", gv) } else { @@ -419,7 +421,7 @@ impl<'a> Verifier<'a> { } } - fn verify_heap(&self, inst: Inst, heap: ir::Heap) -> Result { + fn verify_heap(&self, inst: Inst, heap: ir::Heap) -> VerifierResult<()> { if !self.func.heaps.is_valid(heap) { err!(inst, "invalid heap {}", heap) } else { @@ -427,7 +429,7 @@ impl<'a> Verifier<'a> { } } - fn verify_value_list(&self, inst: Inst, l: &ValueList) -> Result { + fn verify_value_list(&self, inst: Inst, l: &ValueList) -> VerifierResult<()> { if !l.is_valid(&self.func.dfg.value_lists) { err!(inst, "invalid value list reference {:?}", l) } else { @@ -435,7 +437,7 @@ impl<'a> Verifier<'a> { } } - fn verify_jump_table(&self, inst: Inst, j: JumpTable) -> Result { + fn verify_jump_table(&self, inst: Inst, j: JumpTable) -> VerifierResult<()> { if !self.func.jump_tables.is_valid(j) { err!(inst, "invalid jump table reference {}", j) } else { @@ -443,7 +445,7 @@ impl<'a> Verifier<'a> { } } - fn verify_value(&self, loc_inst: Inst, v: Value) -> Result { + fn verify_value(&self, loc_inst: Inst, v: Value) -> VerifierResult<()> { let dfg = &self.func.dfg; if !dfg.value_is_valid(v) { err!(loc_inst, "invalid value reference {}", v) @@ -452,7 +454,7 @@ impl<'a> Verifier<'a> { } } - fn verify_inst_arg(&self, loc_inst: Inst, v: Value) -> Result { + fn verify_inst_arg(&self, loc_inst: Inst, v: Value) -> VerifierResult<()> { self.verify_value(loc_inst, v)?; let dfg = &self.func.dfg; @@ -523,7 +525,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn verify_inst_result(&self, loc_inst: Inst, v: Value) -> Result { + fn verify_inst_result(&self, loc_inst: Inst, v: Value) -> VerifierResult<()> { self.verify_value(loc_inst, v)?; match self.func.dfg.value_def(v) { @@ -546,7 +548,7 @@ impl<'a> Verifier<'a> { } } - fn domtree_integrity(&self, domtree: &DominatorTree) -> Result { + fn domtree_integrity(&self, domtree: &DominatorTree) -> VerifierResult<()> { // We consider two `DominatorTree`s to be equal if they return the same immediate // dominator for each EBB. Therefore the current domtree is valid if it matches the freshly // computed one. @@ -602,7 +604,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck_entry_block_params(&self) -> Result { + fn typecheck_entry_block_params(&self) -> VerifierResult<()> { if let Some(ebb) = self.func.layout.entry_block() { let expected_types = &self.func.signature.params; let ebb_param_count = self.func.dfg.num_ebb_params(ebb); @@ -632,7 +634,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck(&self, inst: Inst) -> Result { + fn typecheck(&self, inst: Inst) -> VerifierResult<()> { let inst_data = &self.func.dfg[inst]; let constraints = inst_data.opcode().constraints(); @@ -660,7 +662,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck_results(&self, inst: Inst, ctrl_type: Type) -> Result { + fn typecheck_results(&self, inst: Inst, ctrl_type: Type) -> VerifierResult<()> { let mut i = 0; for &result in self.func.dfg.inst_results(inst) { let result_type = self.func.dfg.value_type(result); @@ -689,7 +691,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck_fixed_args(&self, inst: Inst, ctrl_type: Type) -> Result { + fn typecheck_fixed_args(&self, inst: Inst, ctrl_type: Type) -> VerifierResult<()> { let constraints = self.func.dfg[inst].opcode().constraints(); for (i, &arg) in self.func.dfg.inst_fixed_args(inst).iter().enumerate() { @@ -724,7 +726,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck_variable_args(&self, inst: Inst) -> Result { + fn typecheck_variable_args(&self, inst: Inst) -> VerifierResult<()> { match self.func.dfg.analyze_branch(inst) { BranchInfo::SingleDest(ebb, _) => { let iter = self.func @@ -777,7 +779,7 @@ impl<'a> Verifier<'a> { &self, inst: Inst, iter: I, - ) -> Result { + ) -> VerifierResult<()> { let variable_args = self.func.dfg.inst_variable_args(inst); let mut i = 0; @@ -816,7 +818,7 @@ impl<'a> Verifier<'a> { /// /// When a signature has been legalized, all values passed as outgoing arguments on the stack /// must be assigned to a matching `OutgoingArg` stack slot. - fn check_outgoing_args(&self, inst: Inst, sig_ref: SigRef) -> Result { + fn check_outgoing_args(&self, inst: Inst, sig_ref: SigRef) -> VerifierResult<()> { let sig = &self.func.dfg.signatures[sig_ref]; // Before legalization, there's nothing to check. @@ -878,7 +880,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn typecheck_return(&self, inst: Inst) -> Result { + fn typecheck_return(&self, inst: Inst) -> VerifierResult<()> { if self.func.dfg[inst].opcode().is_return() { let args = self.func.dfg.inst_variable_args(inst); let expected_types = &self.func.signature.returns; @@ -904,7 +906,7 @@ impl<'a> Verifier<'a> { // Check special-purpose type constraints that can't be expressed in the normal opcode // constraints. - fn typecheck_special(&self, inst: Inst, ctrl_type: Type) -> Result { + fn typecheck_special(&self, inst: Inst, ctrl_type: Type) -> VerifierResult<()> { if let ir::InstructionData::Unary { opcode, arg } = self.func.dfg[inst] { let arg_type = self.func.dfg.value_type(arg); match opcode { @@ -950,7 +952,7 @@ impl<'a> Verifier<'a> { Ok(()) } - fn cfg_integrity(&self, cfg: &ControlFlowGraph) -> Result { + fn cfg_integrity(&self, cfg: &ControlFlowGraph) -> VerifierResult<()> { let mut expected_succs = BTreeSet::::new(); let mut got_succs = BTreeSet::::new(); let mut expected_preds = BTreeSet::::new(); @@ -1001,7 +1003,7 @@ impl<'a> Verifier<'a> { /// If the verifier has been set up with an ISA, make sure that the recorded encoding for the /// instruction (if any) matches how the ISA would encode it. - fn verify_encoding(&self, inst: Inst) -> Result { + fn verify_encoding(&self, inst: Inst) -> VerifierResult<()> { // When the encodings table is empty, we don't require any instructions to be encoded. // // Once some instructions are encoded, we require all side-effecting instructions to have a @@ -1109,7 +1111,7 @@ impl<'a> Verifier<'a> { /// Verify the `return_at_end` property which requires that there are no internal return /// instructions. - fn verify_return_at_end(&self) -> Result { + fn verify_return_at_end(&self) -> VerifierResult<()> { for ebb in self.func.layout.ebbs() { let inst = self.func.layout.last_inst(ebb).unwrap(); if self.func.dfg[inst].opcode().is_return() && Some(ebb) != self.func.layout.last_ebb() @@ -1121,7 +1123,7 @@ impl<'a> Verifier<'a> { Ok(()) } - pub fn run(&self) -> Result { + pub fn run(&self) -> VerifierResult<()> { self.verify_global_vars()?; self.typecheck_entry_block_params()?; for ebb in self.func.layout.ebbs() { @@ -1145,7 +1147,7 @@ impl<'a> Verifier<'a> { #[cfg(test)] mod tests { - use super::{Error, Verifier}; + use super::{Verifier, VerifierError}; use entity::EntityList; use ir::instructions::{InstructionData, Opcode}; use ir::Function; @@ -1155,7 +1157,7 @@ mod tests { ($e:expr, $msg:expr) => { match $e { Ok(_) => panic!("Expected an error"), - Err(Error { message, .. }) => { + Err(VerifierError { message, .. }) => { if !message.contains($msg) { #[cfg(feature = "std")] panic!(format!( From 02e34d1bf79722d2a219bec9977e8168069ec7d3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 7 Jun 2018 15:06:31 -0700 Subject: [PATCH 1839/3084] Make `CtonError` parameterized on the result type. This makes it more consistent with other custom `Result` types. --- lib/codegen/src/binemit/relaxation.rs | 4 ++-- lib/codegen/src/context.rs | 34 +++++++++++++-------------- lib/codegen/src/isa/mod.rs | 4 ++-- lib/codegen/src/isa/x86/abi.rs | 10 ++++---- lib/codegen/src/isa/x86/mod.rs | 4 ++-- lib/codegen/src/regalloc/context.rs | 2 +- lib/codegen/src/result.rs | 2 +- lib/codegen/src/stack_layout.rs | 4 ++-- 8 files changed, 32 insertions(+), 32 deletions(-) diff --git a/lib/codegen/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs index 7003df1eb0..299d0a1fb9 100644 --- a/lib/codegen/src/binemit/relaxation.rs +++ b/lib/codegen/src/binemit/relaxation.rs @@ -32,12 +32,12 @@ use cursor::{Cursor, FuncCursor}; use ir::{Function, InstructionData, Opcode}; use isa::{EncInfo, TargetIsa}; use iterators::IteratorExtras; -use result::CtonError; +use result::CtonResult; /// Relax branches and compute the final layout of EBB headers in `func`. /// /// Fill in the `func.offsets` table so the function is ready for binary emission. -pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> Result { +pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> CtonResult { let encinfo = isa.encoding_info(); // Clear all offsets so we can recognize EBBs that haven't been visited yet. diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index 83dd3d0368..5c33d56314 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -22,7 +22,7 @@ use nan_canonicalization::do_nan_canonicalization; use postopt::do_postopt; use preopt::do_preopt; use regalloc; -use result::{CtonError, CtonResult}; +use result::CtonResult; use settings::{FlagsOrIsa, OptLevel}; use simple_gvn::do_simple_gvn; use std::vec::Vec; @@ -95,7 +95,7 @@ impl Context { mem: &mut Vec, relocs: &mut RelocSink, traps: &mut TrapSink, - ) -> CtonResult { + ) -> CtonResult<()> { let code_size = self.compile(isa)?; let old_len = mem.len(); mem.resize(old_len + code_size as usize, 0); @@ -117,7 +117,7 @@ impl Context { /// code sink. /// /// Returns the size of the function's code. - pub fn compile(&mut self, isa: &TargetIsa) -> Result { + pub fn compile(&mut self, isa: &TargetIsa) -> CtonResult { let _tt = timing::compile(); self.verify_if(isa)?; @@ -179,7 +179,7 @@ impl Context { } /// Run the verifier only if the `enable_verifier` setting is true. - pub fn verify_if<'a, FOI: Into>>(&self, fisa: FOI) -> CtonResult { + pub fn verify_if<'a, FOI: Into>>(&self, fisa: FOI) -> CtonResult<()> { let fisa = fisa.into(); if fisa.flags.enable_verifier() { self.verify(fisa).map_err(Into::into) @@ -194,7 +194,7 @@ impl Context { } /// Run the locations verifier only if the `enable_verifier` setting is true. - pub fn verify_locations_if(&self, isa: &TargetIsa) -> CtonResult { + pub fn verify_locations_if(&self, isa: &TargetIsa) -> CtonResult<()> { if isa.flags().enable_verifier() { self.verify_locations(isa).map_err(Into::into) } else { @@ -203,27 +203,27 @@ impl Context { } /// Perform dead-code elimination on the function. - pub fn dce<'a, FOI: Into>>(&mut self, fisa: FOI) -> CtonResult { + pub fn dce<'a, FOI: Into>>(&mut self, fisa: FOI) -> CtonResult<()> { do_dce(&mut self.func, &mut self.domtree); self.verify_if(fisa)?; Ok(()) } /// Perform pre-legalization rewrites on the function. - pub fn preopt(&mut self, isa: &TargetIsa) -> CtonResult { + pub fn preopt(&mut self, isa: &TargetIsa) -> CtonResult<()> { do_preopt(&mut self.func); self.verify_if(isa)?; Ok(()) } /// Perform NaN canonicalizing rewrites on the function. - pub fn canonicalize_nans(&mut self, isa: &TargetIsa) -> CtonResult { + pub fn canonicalize_nans(&mut self, isa: &TargetIsa) -> CtonResult<()> { do_nan_canonicalization(&mut self.func); self.verify_if(isa) } /// Run the legalizer for `isa` on the function. - pub fn legalize(&mut self, isa: &TargetIsa) -> CtonResult { + pub fn legalize(&mut self, isa: &TargetIsa) -> CtonResult<()> { // Legalization invalidates the domtree and loop_analysis by mutating the CFG. // TODO: Avoid doing this when legalization doesn't actually mutate the CFG. self.domtree.clear(); @@ -233,7 +233,7 @@ impl Context { } /// Perform post-legalization rewrites on the function. - pub fn postopt(&mut self, isa: &TargetIsa) -> CtonResult { + pub fn postopt(&mut self, isa: &TargetIsa) -> CtonResult<()> { do_postopt(&mut self.func, isa); self.verify_if(isa)?; Ok(()) @@ -262,13 +262,13 @@ impl Context { } /// Perform simple GVN on the function. - pub fn simple_gvn<'a, FOI: Into>>(&mut self, fisa: FOI) -> CtonResult { + pub fn simple_gvn<'a, FOI: Into>>(&mut self, fisa: FOI) -> CtonResult<()> { do_simple_gvn(&mut self.func, &mut self.domtree); self.verify_if(fisa) } /// Perform LICM on the function. - pub fn licm<'a, FOI: Into>>(&mut self, fisa: FOI) -> CtonResult { + pub fn licm<'a, FOI: Into>>(&mut self, fisa: FOI) -> CtonResult<()> { do_licm( &mut self.func, &mut self.cfg, @@ -279,7 +279,7 @@ impl Context { } /// Perform unreachable code elimination. - pub fn eliminate_unreachable_code<'a, FOI>(&mut self, fisa: FOI) -> CtonResult + pub fn eliminate_unreachable_code<'a, FOI>(&mut self, fisa: FOI) -> CtonResult<()> where FOI: Into>, { @@ -288,13 +288,13 @@ impl Context { } /// Run the register allocator. - pub fn regalloc(&mut self, isa: &TargetIsa) -> CtonResult { + pub fn regalloc(&mut self, isa: &TargetIsa) -> CtonResult<()> { self.regalloc .run(isa, &mut self.func, &self.cfg, &mut self.domtree) } /// Insert prologue and epilogues after computing the stack frame layout. - pub fn prologue_epilogue(&mut self, isa: &TargetIsa) -> CtonResult { + pub fn prologue_epilogue(&mut self, isa: &TargetIsa) -> CtonResult<()> { assert!( self.func.stack_limit.is_none(), "stack_limit isn't implemented yet" @@ -306,7 +306,7 @@ impl Context { } /// Run the instruction shrinking pass. - pub fn shrink_instructions(&mut self, isa: &TargetIsa) -> CtonResult { + pub fn shrink_instructions(&mut self, isa: &TargetIsa) -> CtonResult<()> { shrink_instructions(&mut self.func, isa); self.verify_if(isa)?; self.verify_locations_if(isa)?; @@ -314,7 +314,7 @@ impl Context { } /// Run the branch relaxation pass and return the final code size. - pub fn relax_branches(&mut self, isa: &TargetIsa) -> Result { + pub fn relax_branches(&mut self, isa: &TargetIsa) -> CtonResult { let code_size = relax_branches(&mut self.func, isa)?; self.verify_if(isa)?; self.verify_locations_if(isa)?; diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs index 34733b92f7..c042a42600 100644 --- a/lib/codegen/src/isa/mod.rs +++ b/lib/codegen/src/isa/mod.rs @@ -56,7 +56,7 @@ use flowgraph; use ir; use isa::enc_tables::Encodings; use regalloc; -use result; +use result::CtonResult; use settings; use settings::CallConv; use std::boxed::Box; @@ -281,7 +281,7 @@ pub trait TargetIsa: fmt::Display { /// Compute the stack layout and insert prologue and epilogue code into `func`. /// /// Return an error if the stack frame is too large. - fn prologue_epilogue(&self, func: &mut ir::Function) -> result::CtonResult { + fn prologue_epilogue(&self, func: &mut ir::Function) -> CtonResult<()> { let _tt = timing::prologue_epilogue(); // This default implementation is unlikely to be good enough. use ir::stackslot::{StackOffset, StackSize}; diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs index d10f406ab3..264a45a137 100644 --- a/lib/codegen/src/isa/x86/abi.rs +++ b/lib/codegen/src/isa/x86/abi.rs @@ -10,7 +10,7 @@ use ir::{get_probestack_funcref, AbiParam, ArgumentExtension, ArgumentLoc, Argum InstBuilder, ValueLoc}; use isa::{RegClass, RegUnit, TargetIsa}; use regalloc::RegisterSet; -use result; +use result::CtonResult; use settings::CallConv; use stack_layout::layout_stack; use std::i32; @@ -257,7 +257,7 @@ fn callee_saved_gprs_used(isa: &TargetIsa, func: &ir::Function) -> RegisterSet { used } -pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { +pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CtonResult<()> { match func.signature.call_conv { // For now, just translate fast and cold as system_v. CallConv::Fast | CallConv::Cold | CallConv::SystemV => { @@ -269,7 +269,7 @@ pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::Ct } } -pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { +pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CtonResult<()> { debug_assert!( !isa.flags().probestack_enabled(), "baldrdash does not expect cretonne to emit stack probes" @@ -290,7 +290,7 @@ pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> /// Implementation of the fastcall-based Win64 calling convention described at [1] /// [1] https://msdn.microsoft.com/en-us/library/ms235286.aspx -pub fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { +pub fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CtonResult<()> { if isa.triple().pointer_width().unwrap() != PointerWidth::U64 { panic!("TODO: windows-fastcall: x86-32 not implemented yet"); } @@ -362,7 +362,7 @@ pub fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> r } /// Insert a System V-compatible prologue and epilogue. -pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> result::CtonResult { +pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CtonResult<()> { // The original 32-bit x86 ELF ABI had a 4-byte aligned stack pointer, but // newer versions use a 16-byte aligned stack pointer. let stack_align = 16; diff --git a/lib/codegen/src/isa/x86/mod.rs b/lib/codegen/src/isa/x86/mod.rs index 6e2c035d38..cba7a640f9 100644 --- a/lib/codegen/src/isa/x86/mod.rs +++ b/lib/codegen/src/isa/x86/mod.rs @@ -13,7 +13,7 @@ use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use regalloc; -use result; +use result::CtonResult; use std::boxed::Box; use std::fmt; use target_lexicon::{PointerWidth, Triple}; @@ -129,7 +129,7 @@ impl TargetIsa for Isa { emit_function(func, binemit::emit_inst, sink) } - fn prologue_epilogue(&self, func: &mut ir::Function) -> result::CtonResult { + fn prologue_epilogue(&self, func: &mut ir::Function) -> CtonResult<()> { let _tt = timing::prologue_epilogue(); abi::prologue_epilogue(func, self) } diff --git a/lib/codegen/src/regalloc/context.rs b/lib/codegen/src/regalloc/context.rs index ce1d68a096..10f83b979d 100644 --- a/lib/codegen/src/regalloc/context.rs +++ b/lib/codegen/src/regalloc/context.rs @@ -72,7 +72,7 @@ impl Context { func: &mut Function, cfg: &ControlFlowGraph, domtree: &mut DominatorTree, - ) -> CtonResult { + ) -> CtonResult<()> { let _tt = timing::regalloc(); debug_assert!(domtree.is_valid()); diff --git a/lib/codegen/src/result.rs b/lib/codegen/src/result.rs index 1821c2e3b5..9d684dc172 100644 --- a/lib/codegen/src/result.rs +++ b/lib/codegen/src/result.rs @@ -32,7 +32,7 @@ pub enum CtonError { } /// A Cretonne compilation result. -pub type CtonResult = Result<(), CtonError>; +pub type CtonResult = Result; impl From for CtonError { fn from(e: VerifierError) -> Self { diff --git a/lib/codegen/src/stack_layout.rs b/lib/codegen/src/stack_layout.rs index cda2251900..6334f4d5ed 100644 --- a/lib/codegen/src/stack_layout.rs +++ b/lib/codegen/src/stack_layout.rs @@ -2,7 +2,7 @@ use ir::stackslot::{StackOffset, StackSize, StackSlotKind}; use ir::StackSlots; -use result::CtonError; +use result::{CtonError, CtonResult}; use std::cmp::{max, min}; /// Compute the stack frame layout. @@ -15,7 +15,7 @@ use std::cmp::{max, min}; /// Returns the total stack frame size which is also saved in `frame.frame_size`. /// /// If the stack frame is too big, returns an `ImplLimitExceeded` error. -pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> Result { +pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CtonResult { // Each object and the whole stack frame must fit in 2 GB such that any relative offset within // the frame fits in a `StackOffset`. let max_size = StackOffset::max_value() as StackSize; From 8a26a504752649b6538b74805977be5054b8aea5 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sat, 19 May 2018 21:33:55 -0700 Subject: [PATCH 1840/3084] Rename `CtonError` and `CtonResult` to `CodegenError` and `CodegenResult`. --- lib/codegen/src/binemit/relaxation.rs | 4 ++-- lib/codegen/src/context.rs | 34 +++++++++++++-------------- lib/codegen/src/isa/mod.rs | 4 ++-- lib/codegen/src/isa/x86/abi.rs | 10 ++++---- lib/codegen/src/isa/x86/mod.rs | 4 ++-- lib/codegen/src/print_errors.rs | 6 ++--- lib/codegen/src/regalloc/context.rs | 4 ++-- lib/codegen/src/result.rs | 10 ++++---- lib/codegen/src/stack_layout.rs | 16 ++++++------- lib/module/src/module.rs | 4 ++-- 10 files changed, 48 insertions(+), 48 deletions(-) diff --git a/lib/codegen/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs index 299d0a1fb9..3c4e631bfc 100644 --- a/lib/codegen/src/binemit/relaxation.rs +++ b/lib/codegen/src/binemit/relaxation.rs @@ -32,12 +32,12 @@ use cursor::{Cursor, FuncCursor}; use ir::{Function, InstructionData, Opcode}; use isa::{EncInfo, TargetIsa}; use iterators::IteratorExtras; -use result::CtonResult; +use result::CodegenResult; /// Relax branches and compute the final layout of EBB headers in `func`. /// /// Fill in the `func.offsets` table so the function is ready for binary emission. -pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> CtonResult { +pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> CodegenResult { let encinfo = isa.encoding_info(); // Clear all offsets so we can recognize EBBs that haven't been visited yet. diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index 5c33d56314..6e8c54051b 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -22,7 +22,7 @@ use nan_canonicalization::do_nan_canonicalization; use postopt::do_postopt; use preopt::do_preopt; use regalloc; -use result::CtonResult; +use result::CodegenResult; use settings::{FlagsOrIsa, OptLevel}; use simple_gvn::do_simple_gvn; use std::vec::Vec; @@ -95,7 +95,7 @@ impl Context { mem: &mut Vec, relocs: &mut RelocSink, traps: &mut TrapSink, - ) -> CtonResult<()> { + ) -> CodegenResult<()> { let code_size = self.compile(isa)?; let old_len = mem.len(); mem.resize(old_len + code_size as usize, 0); @@ -117,7 +117,7 @@ impl Context { /// code sink. /// /// Returns the size of the function's code. - pub fn compile(&mut self, isa: &TargetIsa) -> CtonResult { + pub fn compile(&mut self, isa: &TargetIsa) -> CodegenResult { let _tt = timing::compile(); self.verify_if(isa)?; @@ -179,7 +179,7 @@ impl Context { } /// Run the verifier only if the `enable_verifier` setting is true. - pub fn verify_if<'a, FOI: Into>>(&self, fisa: FOI) -> CtonResult<()> { + pub fn verify_if<'a, FOI: Into>>(&self, fisa: FOI) -> CodegenResult<()> { let fisa = fisa.into(); if fisa.flags.enable_verifier() { self.verify(fisa).map_err(Into::into) @@ -194,7 +194,7 @@ impl Context { } /// Run the locations verifier only if the `enable_verifier` setting is true. - pub fn verify_locations_if(&self, isa: &TargetIsa) -> CtonResult<()> { + pub fn verify_locations_if(&self, isa: &TargetIsa) -> CodegenResult<()> { if isa.flags().enable_verifier() { self.verify_locations(isa).map_err(Into::into) } else { @@ -203,27 +203,27 @@ impl Context { } /// Perform dead-code elimination on the function. - pub fn dce<'a, FOI: Into>>(&mut self, fisa: FOI) -> CtonResult<()> { + pub fn dce<'a, FOI: Into>>(&mut self, fisa: FOI) -> CodegenResult<()> { do_dce(&mut self.func, &mut self.domtree); self.verify_if(fisa)?; Ok(()) } /// Perform pre-legalization rewrites on the function. - pub fn preopt(&mut self, isa: &TargetIsa) -> CtonResult<()> { + pub fn preopt(&mut self, isa: &TargetIsa) -> CodegenResult<()> { do_preopt(&mut self.func); self.verify_if(isa)?; Ok(()) } /// Perform NaN canonicalizing rewrites on the function. - pub fn canonicalize_nans(&mut self, isa: &TargetIsa) -> CtonResult<()> { + pub fn canonicalize_nans(&mut self, isa: &TargetIsa) -> CodegenResult<()> { do_nan_canonicalization(&mut self.func); self.verify_if(isa) } /// Run the legalizer for `isa` on the function. - pub fn legalize(&mut self, isa: &TargetIsa) -> CtonResult<()> { + pub fn legalize(&mut self, isa: &TargetIsa) -> CodegenResult<()> { // Legalization invalidates the domtree and loop_analysis by mutating the CFG. // TODO: Avoid doing this when legalization doesn't actually mutate the CFG. self.domtree.clear(); @@ -233,7 +233,7 @@ impl Context { } /// Perform post-legalization rewrites on the function. - pub fn postopt(&mut self, isa: &TargetIsa) -> CtonResult<()> { + pub fn postopt(&mut self, isa: &TargetIsa) -> CodegenResult<()> { do_postopt(&mut self.func, isa); self.verify_if(isa)?; Ok(()) @@ -262,13 +262,13 @@ impl Context { } /// Perform simple GVN on the function. - pub fn simple_gvn<'a, FOI: Into>>(&mut self, fisa: FOI) -> CtonResult<()> { + pub fn simple_gvn<'a, FOI: Into>>(&mut self, fisa: FOI) -> CodegenResult<()> { do_simple_gvn(&mut self.func, &mut self.domtree); self.verify_if(fisa) } /// Perform LICM on the function. - pub fn licm<'a, FOI: Into>>(&mut self, fisa: FOI) -> CtonResult<()> { + pub fn licm<'a, FOI: Into>>(&mut self, fisa: FOI) -> CodegenResult<()> { do_licm( &mut self.func, &mut self.cfg, @@ -279,7 +279,7 @@ impl Context { } /// Perform unreachable code elimination. - pub fn eliminate_unreachable_code<'a, FOI>(&mut self, fisa: FOI) -> CtonResult<()> + pub fn eliminate_unreachable_code<'a, FOI>(&mut self, fisa: FOI) -> CodegenResult<()> where FOI: Into>, { @@ -288,13 +288,13 @@ impl Context { } /// Run the register allocator. - pub fn regalloc(&mut self, isa: &TargetIsa) -> CtonResult<()> { + pub fn regalloc(&mut self, isa: &TargetIsa) -> CodegenResult<()> { self.regalloc .run(isa, &mut self.func, &self.cfg, &mut self.domtree) } /// Insert prologue and epilogues after computing the stack frame layout. - pub fn prologue_epilogue(&mut self, isa: &TargetIsa) -> CtonResult<()> { + pub fn prologue_epilogue(&mut self, isa: &TargetIsa) -> CodegenResult<()> { assert!( self.func.stack_limit.is_none(), "stack_limit isn't implemented yet" @@ -306,7 +306,7 @@ impl Context { } /// Run the instruction shrinking pass. - pub fn shrink_instructions(&mut self, isa: &TargetIsa) -> CtonResult<()> { + pub fn shrink_instructions(&mut self, isa: &TargetIsa) -> CodegenResult<()> { shrink_instructions(&mut self.func, isa); self.verify_if(isa)?; self.verify_locations_if(isa)?; @@ -314,7 +314,7 @@ impl Context { } /// Run the branch relaxation pass and return the final code size. - pub fn relax_branches(&mut self, isa: &TargetIsa) -> CtonResult { + pub fn relax_branches(&mut self, isa: &TargetIsa) -> CodegenResult { let code_size = relax_branches(&mut self.func, isa)?; self.verify_if(isa)?; self.verify_locations_if(isa)?; diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs index c042a42600..d785c32f57 100644 --- a/lib/codegen/src/isa/mod.rs +++ b/lib/codegen/src/isa/mod.rs @@ -56,7 +56,7 @@ use flowgraph; use ir; use isa::enc_tables::Encodings; use regalloc; -use result::CtonResult; +use result::CodegenResult; use settings; use settings::CallConv; use std::boxed::Box; @@ -281,7 +281,7 @@ pub trait TargetIsa: fmt::Display { /// Compute the stack layout and insert prologue and epilogue code into `func`. /// /// Return an error if the stack frame is too large. - fn prologue_epilogue(&self, func: &mut ir::Function) -> CtonResult<()> { + fn prologue_epilogue(&self, func: &mut ir::Function) -> CodegenResult<()> { let _tt = timing::prologue_epilogue(); // This default implementation is unlikely to be good enough. use ir::stackslot::{StackOffset, StackSize}; diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs index 264a45a137..c74572ecc7 100644 --- a/lib/codegen/src/isa/x86/abi.rs +++ b/lib/codegen/src/isa/x86/abi.rs @@ -10,7 +10,7 @@ use ir::{get_probestack_funcref, AbiParam, ArgumentExtension, ArgumentLoc, Argum InstBuilder, ValueLoc}; use isa::{RegClass, RegUnit, TargetIsa}; use regalloc::RegisterSet; -use result::CtonResult; +use result::CodegenResult; use settings::CallConv; use stack_layout::layout_stack; use std::i32; @@ -257,7 +257,7 @@ fn callee_saved_gprs_used(isa: &TargetIsa, func: &ir::Function) -> RegisterSet { used } -pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CtonResult<()> { +pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> { match func.signature.call_conv { // For now, just translate fast and cold as system_v. CallConv::Fast | CallConv::Cold | CallConv::SystemV => { @@ -269,7 +269,7 @@ pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CtonResult } } -pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CtonResult<()> { +pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> { debug_assert!( !isa.flags().probestack_enabled(), "baldrdash does not expect cretonne to emit stack probes" @@ -290,7 +290,7 @@ pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> /// Implementation of the fastcall-based Win64 calling convention described at [1] /// [1] https://msdn.microsoft.com/en-us/library/ms235286.aspx -pub fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CtonResult<()> { +pub fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> { if isa.triple().pointer_width().unwrap() != PointerWidth::U64 { panic!("TODO: windows-fastcall: x86-32 not implemented yet"); } @@ -362,7 +362,7 @@ pub fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> C } /// Insert a System V-compatible prologue and epilogue. -pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CtonResult<()> { +pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> { // The original 32-bit x86 ELF ABI had a 4-byte aligned stack pointer, but // newer versions use a 16-byte aligned stack pointer. let stack_align = 16; diff --git a/lib/codegen/src/isa/x86/mod.rs b/lib/codegen/src/isa/x86/mod.rs index cba7a640f9..2681e044d6 100644 --- a/lib/codegen/src/isa/x86/mod.rs +++ b/lib/codegen/src/isa/x86/mod.rs @@ -13,7 +13,7 @@ use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings}; use isa::Builder as IsaBuilder; use isa::{EncInfo, RegClass, RegInfo, TargetIsa}; use regalloc; -use result::CtonResult; +use result::CodegenResult; use std::boxed::Box; use std::fmt; use target_lexicon::{PointerWidth, Triple}; @@ -129,7 +129,7 @@ impl TargetIsa for Isa { emit_function(func, binemit::emit_inst, sink) } - fn prologue_epilogue(&self, func: &mut ir::Function) -> CtonResult<()> { + fn prologue_epilogue(&self, func: &mut ir::Function) -> CodegenResult<()> { let _tt = timing::prologue_epilogue(); abi::prologue_epilogue(func, self) } diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs index 283b3d6d08..777fde05b8 100644 --- a/lib/codegen/src/print_errors.rs +++ b/lib/codegen/src/print_errors.rs @@ -2,7 +2,7 @@ use ir; use isa::TargetIsa; -use result::CtonError; +use result::CodegenError; use std::fmt::Write; use std::string::{String, ToString}; use verifier::VerifierError; @@ -25,8 +25,8 @@ pub fn pretty_verifier_error( } /// Pretty-print a Cretonne error. -pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CtonError) -> String { - if let CtonError::Verifier(e) = err { +pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CodegenError) -> String { + if let CodegenError::Verifier(e) = err { pretty_verifier_error(func, isa, &e) } else { err.to_string() diff --git a/lib/codegen/src/regalloc/context.rs b/lib/codegen/src/regalloc/context.rs index 10f83b979d..349fa212cb 100644 --- a/lib/codegen/src/regalloc/context.rs +++ b/lib/codegen/src/regalloc/context.rs @@ -15,7 +15,7 @@ use regalloc::liveness::Liveness; use regalloc::reload::Reload; use regalloc::spilling::Spilling; use regalloc::virtregs::VirtRegs; -use result::CtonResult; +use result::CodegenResult; use timing; use topo_order::TopoOrder; use verifier::{verify_context, verify_cssa, verify_liveness, verify_locations}; @@ -72,7 +72,7 @@ impl Context { func: &mut Function, cfg: &ControlFlowGraph, domtree: &mut DominatorTree, - ) -> CtonResult<()> { + ) -> CodegenResult<()> { let _tt = timing::regalloc(); debug_assert!(domtree.is_valid()); diff --git a/lib/codegen/src/result.rs b/lib/codegen/src/result.rs index 9d684dc172..ffdcbb152b 100644 --- a/lib/codegen/src/result.rs +++ b/lib/codegen/src/result.rs @@ -6,7 +6,7 @@ use verifier::VerifierError; /// /// When Cretonne fails to compile a function, it will return one of these error codes. #[derive(Fail, Debug, PartialEq, Eq)] -pub enum CtonError { +pub enum CodegenError { /// An IR verifier error. /// /// This always represents a bug, either in the code that generated IR for Cretonne, or a bug @@ -31,11 +31,11 @@ pub enum CtonError { CodeTooLarge, } -/// A Cretonne compilation result. -pub type CtonResult = Result; +/// A convenient alias for a `Result` that uses `CodegenError` as the error type. +pub type CodegenResult = Result; -impl From for CtonError { +impl From for CodegenError { fn from(e: VerifierError) -> Self { - CtonError::Verifier(e) + CodegenError::Verifier(e) } } diff --git a/lib/codegen/src/stack_layout.rs b/lib/codegen/src/stack_layout.rs index 6334f4d5ed..eced3ada0a 100644 --- a/lib/codegen/src/stack_layout.rs +++ b/lib/codegen/src/stack_layout.rs @@ -2,7 +2,7 @@ use ir::stackslot::{StackOffset, StackSize, StackSlotKind}; use ir::StackSlots; -use result::{CtonError, CtonResult}; +use result::{CodegenError, CodegenResult}; use std::cmp::{max, min}; /// Compute the stack frame layout. @@ -15,7 +15,7 @@ use std::cmp::{max, min}; /// Returns the total stack frame size which is also saved in `frame.frame_size`. /// /// If the stack frame is too big, returns an `ImplLimitExceeded` error. -pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CtonResult { +pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CodegenResult { // Each object and the whole stack frame must fit in 2 GB such that any relative offset within // the frame fits in a `StackOffset`. let max_size = StackOffset::max_value() as StackSize; @@ -41,7 +41,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CtonResult< for slot in frame.values() { if slot.size > max_size { - return Err(CtonError::ImplLimitExceeded); + return Err(CodegenError::ImplLimitExceeded); } match slot.kind { @@ -52,7 +52,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CtonResult< let offset = slot.offset .unwrap() .checked_add(slot.size as StackOffset) - .ok_or(CtonError::ImplLimitExceeded)?; + .ok_or(CodegenError::ImplLimitExceeded)?; outgoing_max = max(outgoing_max, offset); } StackSlotKind::SpillSlot @@ -85,7 +85,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CtonResult< offset = offset .checked_sub(slot.size as StackOffset) - .ok_or(CtonError::ImplLimitExceeded)?; + .ok_or(CodegenError::ImplLimitExceeded)?; // Aligning the negative offset can never cause overflow. We're only clearing bits. offset &= -(min_align as StackOffset); @@ -99,7 +99,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CtonResult< // Finally, make room for the outgoing arguments. offset = offset .checked_sub(outgoing_max) - .ok_or(CtonError::ImplLimitExceeded)?; + .ok_or(CodegenError::ImplLimitExceeded)?; offset &= -(alignment as StackOffset); let frame_size = (offset as StackSize).wrapping_neg(); @@ -113,7 +113,7 @@ mod tests { use ir::stackslot::StackOffset; use ir::types; use ir::{StackSlotData, StackSlotKind, StackSlots}; - use result::CtonError; + use result::CodegenError; #[test] fn layout() { @@ -187,7 +187,7 @@ mod tests { // Also test that an unsupported offset is rejected. sss.get_outgoing_arg(types::I8, StackOffset::max_value() - 1); - assert_eq!(layout_stack(sss, 1), Err(CtonError::ImplLimitExceeded)); + assert_eq!(layout_stack(sss, 1), Err(CodegenError::ImplLimitExceeded)); } #[test] diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index 23b8328730..74eb1aaf42 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -6,7 +6,7 @@ // shared with `DataContext`? use cretonne_codegen::entity::{EntityRef, PrimaryMap}; -use cretonne_codegen::result::CtonError; +use cretonne_codegen::result::CodegenError; use cretonne_codegen::{binemit, ir, Context}; use data_context::DataContext; use std::borrow::ToOwned; @@ -134,7 +134,7 @@ pub enum ModuleError { InvalidImportDefinition(String), /// Wraps a `cretonne-codegen` error #[fail(display = "Compilation error: {}", _0)] - Compilation(CtonError), + Compilation(CodegenError), /// Wraps a generic error from a backend #[fail(display = "Backend error: {}", _0)] Backend(String), From 6971ae1c268017b1721206b4dfb10d7add6756e0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sat, 19 May 2018 21:49:10 -0700 Subject: [PATCH 1841/3084] Export `CodegenError` and `CodegenResult` at the top level. Make cretonne-codegen's `result` module private, and instead just export `CodegenError` and `CodegenResult` at the top level of the cretonne-codegen crate. This makes them more consistent with Result and Error types in other cretonne crates. --- lib/codegen/src/binemit/relaxation.rs | 2 +- lib/codegen/src/lib.rs | 4 +++- lib/module/src/module.rs | 3 +-- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/lib/codegen/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs index 3c4e631bfc..7cfb6491ca 100644 --- a/lib/codegen/src/binemit/relaxation.rs +++ b/lib/codegen/src/binemit/relaxation.rs @@ -32,7 +32,7 @@ use cursor::{Cursor, FuncCursor}; use ir::{Function, InstructionData, Opcode}; use isa::{EncInfo, TargetIsa}; use iterators::IteratorExtras; -use result::CodegenResult; +use CodegenResult; /// Relax branches and compute the final layout of EBB headers in `func`. /// diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs index 0c7eb52989..d892ecffe9 100644 --- a/lib/codegen/src/lib.rs +++ b/lib/codegen/src/lib.rs @@ -75,7 +75,6 @@ pub mod ir; pub mod isa; pub mod loop_analysis; pub mod print_errors; -pub mod result; pub mod settings; pub mod timing; pub mod verifier; @@ -99,6 +98,7 @@ mod predicates; mod preopt; mod ref_slice; mod regalloc; +mod result; mod scoped_hash_map; mod simple_gvn; mod stack_layout; @@ -106,6 +106,8 @@ mod topo_order; mod unreachable_code; mod write; +pub use result::{CodegenError, CodegenResult}; + /// This replaces `std` in builds with `core`. #[cfg(not(feature = "std"))] mod std { diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index 74eb1aaf42..a01d58ffdf 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -6,8 +6,7 @@ // shared with `DataContext`? use cretonne_codegen::entity::{EntityRef, PrimaryMap}; -use cretonne_codegen::result::CodegenError; -use cretonne_codegen::{binemit, ir, Context}; +use cretonne_codegen::{binemit, ir, CodegenError, Context}; use data_context::DataContext; use std::borrow::ToOwned; use std::collections::HashMap; From 43bd3cb2a3f3d66afef972c3b62316185c05dff7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sat, 19 May 2018 21:41:05 -0700 Subject: [PATCH 1842/3084] Introduce a `ModuleResult` alias for `Result`. This follows the pattern used by cretonne-codegen, cretonne-wasm, and others. --- lib/faerie/src/backend.rs | 8 ++++---- lib/module/src/backend.rs | 5 +++-- lib/module/src/lib.rs | 3 ++- lib/module/src/module.rs | 11 +++++++---- lib/simplejit/src/backend.rs | 8 ++++---- 5 files changed, 20 insertions(+), 15 deletions(-) diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index 890a934fd2..4c900c843a 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -5,7 +5,7 @@ use cretonne_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSi use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::{self, binemit, ir}; use cretonne_module::{Backend, DataContext, DataDescription, Init, Linkage, ModuleError, - ModuleNamespace}; + ModuleNamespace, ModuleResult}; use faerie; use failure::Error; use std::fs::File; @@ -52,7 +52,7 @@ impl FaerieBuilder { format: BinaryFormat, collect_traps: FaerieTrapCollection, libcall_names: Box String>, - ) -> Result { + ) -> ModuleResult { if !isa.flags().is_pic() { return Err(ModuleError::Backend( "faerie requires TargetIsa be PIC".to_owned(), @@ -149,7 +149,7 @@ impl Backend for FaerieBackend { ctx: &cretonne_codegen::Context, namespace: &ModuleNamespace, code_size: u32, - ) -> Result { + ) -> ModuleResult { let mut code: Vec = Vec::with_capacity(code_size as usize); code.resize(code_size as usize, 0); @@ -198,7 +198,7 @@ impl Backend for FaerieBackend { name: &str, data_ctx: &DataContext, namespace: &ModuleNamespace, - ) -> Result { + ) -> ModuleResult { let &DataDescription { writable: _writable, ref init, diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs index 15258544d8..298e526d6c 100644 --- a/lib/module/src/backend.rs +++ b/lib/module/src/backend.rs @@ -8,6 +8,7 @@ use DataContext; use Linkage; use ModuleError; use ModuleNamespace; +use ModuleResult; /// A `Backend` implements the functionality needed to support a `Module`. pub trait Backend @@ -57,7 +58,7 @@ where ctx: &Context, namespace: &ModuleNamespace, code_size: u32, - ) -> Result; + ) -> ModuleResult; /// Define a zero-initialized data object of the given size. /// @@ -67,7 +68,7 @@ where name: &str, data_ctx: &DataContext, namespace: &ModuleNamespace, - ) -> Result; + ) -> ModuleResult; /// Write the address of `what` into the data for `data` at `offset`. `data` must refer to a /// defined data object. diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs index 445b588c40..85e04caaff 100644 --- a/lib/module/src/lib.rs +++ b/lib/module/src/lib.rs @@ -33,7 +33,8 @@ mod module; pub use backend::Backend; pub use data_context::{DataContext, DataDescription, Init, Writability}; -pub use module::{DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleError, ModuleNamespace}; +pub use module::{DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleError, ModuleNamespace, + ModuleResult}; /// This replaces `std` in builds with `core`. #[cfg(not(feature = "std"))] diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index a01d58ffdf..84188ccd75 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -139,6 +139,9 @@ pub enum ModuleError { Backend(String), } +/// A convenient alias for a `Result` that uses `ModuleError` as the error type. +pub type ModuleResult = Result; + /// A function belonging to a `Module`. struct ModuleFunction where @@ -347,7 +350,7 @@ where name: &str, linkage: Linkage, signature: &ir::Signature, - ) -> Result { + ) -> ModuleResult { // TODO: Can we avoid allocating names so often? use std::collections::hash_map::Entry::*; match self.names.entry(name.to_owned()) { @@ -385,7 +388,7 @@ where name: &str, linkage: Linkage, writable: bool, - ) -> Result { + ) -> ModuleResult { // TODO: Can we avoid allocating names so often? use std::collections::hash_map::Entry::*; match self.names.entry(name.to_owned()) { @@ -457,7 +460,7 @@ where } /// Define a function, producing the function body from the given `Context`. - pub fn define_function(&mut self, func: FuncId, ctx: &mut Context) -> Result<(), ModuleError> { + pub fn define_function(&mut self, func: FuncId, ctx: &mut Context) -> ModuleResult<()> { let compiled = { let code_size = ctx.compile(self.backend.isa()).map_err(|e| { dbg!( @@ -489,7 +492,7 @@ where } /// Define a function, producing the data contents from the given `DataContext`. - pub fn define_data(&mut self, data: DataId, data_ctx: &DataContext) -> Result<(), ModuleError> { + pub fn define_data(&mut self, data: DataId, data_ctx: &DataContext) -> ModuleResult<()> { let compiled = { let info = &self.contents.data_objects[data]; if !info.compiled.is_none() { diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs index 889601a819..4df071efc9 100644 --- a/lib/simplejit/src/backend.rs +++ b/lib/simplejit/src/backend.rs @@ -3,8 +3,8 @@ use cretonne_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSink}; use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::{self, ir, settings}; -use cretonne_module::{Backend, DataContext, DataDescription, Init, Linkage, ModuleError, - ModuleNamespace, Writability}; +use cretonne_module::{Backend, DataContext, DataDescription, Init, Linkage, ModuleNamespace, + ModuleResult, Writability}; use cretonne_native; use libc; use memory::Memory; @@ -118,7 +118,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { ctx: &cretonne_codegen::Context, _namespace: &ModuleNamespace, code_size: u32, - ) -> Result { + ) -> ModuleResult { let size = code_size as usize; let ptr = self.code_memory .allocate(size) @@ -141,7 +141,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { _name: &str, data: &DataContext, _namespace: &ModuleNamespace, - ) -> Result { + ) -> ModuleResult { let &DataDescription { writable, ref init, From 1b55a2d005e0da3be0a71d4d56d97c386057250b Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Jun 2018 04:43:02 -0700 Subject: [PATCH 1843/3084] Rename more `Error` and `Result` types. --- lib/codegen/meta/gen_settings.py | 2 +- lib/codegen/src/isa/mod.rs | 6 +- lib/codegen/src/settings.rs | 35 ++++--- lib/codegen/src/write.rs | 32 +++--- lib/module/src/backend.rs | 1 - lib/reader/src/error.rs | 17 ++-- lib/reader/src/isaspec.rs | 6 +- lib/reader/src/lexer.rs | 16 +-- lib/reader/src/lib.rs | 2 +- lib/reader/src/parser.rs | 168 ++++++++++++++++--------------- lib/reader/src/sourcemap.rs | 20 ++-- lib/wasm/tests/wasm_testsuite.rs | 2 +- 12 files changed, 157 insertions(+), 150 deletions(-) diff --git a/lib/codegen/meta/gen_settings.py b/lib/codegen/meta/gen_settings.py index 3fa0cf7b2e..01335803d4 100644 --- a/lib/codegen/meta/gen_settings.py +++ b/lib/codegen/meta/gen_settings.py @@ -36,7 +36,7 @@ def gen_to_and_from_str(ty, values, fmt): with fmt.indented('impl str::FromStr for {} {{'.format(ty), '}'): fmt.line('type Err = ();') with fmt.indented( - 'fn from_str(s: &str) -> result::Result {', + 'fn from_str(s: &str) -> Result {', '}'): with fmt.indented('match s {', '}'): for v in values: diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs index d785c32f57..d85f0c3e10 100644 --- a/lib/codegen/src/isa/mod.rs +++ b/lib/codegen/src/isa/mod.rs @@ -58,7 +58,7 @@ use isa::enc_tables::Encodings; use regalloc; use result::CodegenResult; use settings; -use settings::CallConv; +use settings::{CallConv, SetResult}; use std::boxed::Box; use std::fmt; use target_lexicon::{Architecture, Triple}; @@ -146,11 +146,11 @@ impl Builder { } impl settings::Configurable for Builder { - fn set(&mut self, name: &str, value: &str) -> settings::Result<()> { + fn set(&mut self, name: &str, value: &str) -> SetResult<()> { self.setup.set(name, value) } - fn enable(&mut self, name: &str) -> settings::Result<()> { + fn enable(&mut self, name: &str) -> SetResult<()> { self.setup.enable(name) } } diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs index d66cbae7c1..2e6ad5f684 100644 --- a/lib/codegen/src/settings.rs +++ b/lib/codegen/src/settings.rs @@ -24,7 +24,6 @@ use constant_hash::{probe, simple_hash}; use isa::TargetIsa; use std::boxed::Box; use std::fmt; -use std::result; use std::str; /// A string-based configurator for settings groups. @@ -35,12 +34,12 @@ pub trait Configurable { /// Set the string value of any setting by name. /// /// This can set any type of setting whether it is numeric, boolean, or enumerated. - fn set(&mut self, name: &str, value: &str) -> Result<()>; + fn set(&mut self, name: &str, value: &str) -> SetResult<()>; /// Enable a boolean setting or apply a preset. /// /// If the identified setting isn't a boolean or a preset, a `BadType` error is returned. - fn enable(&mut self, name: &str) -> Result<()>; + fn enable(&mut self, name: &str) -> SetResult<()>; } /// Collect settings values based on a template. @@ -84,9 +83,9 @@ impl Builder { } /// Look up a descriptor by name. - fn lookup(&self, name: &str) -> Result<(usize, detail::Detail)> { + fn lookup(&self, name: &str) -> SetResult<(usize, detail::Detail)> { match probe(self.template, name, simple_hash(name)) { - Err(_) => Err(Error::BadName), + Err(_) => Err(SetError::BadName), Ok(entry) => { let d = &self.template.descriptors[self.template.hash_table[entry] as usize]; Ok((d.offset as usize, d.detail)) @@ -95,23 +94,23 @@ impl Builder { } } -fn parse_bool_value(value: &str) -> Result { +fn parse_bool_value(value: &str) -> SetResult { match value { "true" | "on" | "yes" | "1" => Ok(true), "false" | "off" | "no" | "0" => Ok(false), - _ => Err(Error::BadValue), + _ => Err(SetError::BadValue), } } -fn parse_enum_value(value: &str, choices: &[&str]) -> Result { +fn parse_enum_value(value: &str, choices: &[&str]) -> SetResult { match choices.iter().position(|&tag| tag == value) { Some(idx) => Ok(idx as u8), - None => Err(Error::BadValue), + None => Err(SetError::BadValue), } } impl Configurable for Builder { - fn enable(&mut self, name: &str) -> Result<()> { + fn enable(&mut self, name: &str) -> SetResult<()> { use self::detail::Detail; let (offset, detail) = self.lookup(name)?; match detail { @@ -123,27 +122,27 @@ impl Configurable for Builder { self.apply_preset(&self.template.presets[offset..]); Ok(()) } - _ => Err(Error::BadType), + _ => Err(SetError::BadType), } } - fn set(&mut self, name: &str, value: &str) -> Result<()> { + fn set(&mut self, name: &str, value: &str) -> SetResult<()> { use self::detail::Detail; let (offset, detail) = self.lookup(name)?; match detail { Detail::Bool { bit } => { - // Cannot currently propagate Result<()> up on functions returning () + // Cannot currently propagate SetResult<()> up on functions returning () // with the `?` operator self.set_bit(offset, bit, parse_bool_value(value)?); } Detail::Num => { - self.bytes[offset] = value.parse().map_err(|_| Error::BadValue)?; + self.bytes[offset] = value.parse().map_err(|_| SetError::BadValue)?; } Detail::Enum { last, enumerators } => { self.bytes[offset] = parse_enum_value(value, self.template.enums(last, enumerators))?; } - Detail::Preset => return Err(Error::BadName), + Detail::Preset => return Err(SetError::BadName), } Ok(()) } @@ -151,7 +150,7 @@ impl Configurable for Builder { /// An error produced when changing a setting. #[derive(Debug, PartialEq, Eq)] -pub enum Error { +pub enum SetError { /// No setting by this name exists. BadName, @@ -163,7 +162,7 @@ pub enum Error { } /// A result returned when changing a setting. -pub type Result = result::Result; +pub type SetResult = Result; /// A reference to just the boolean predicates of a settings object. /// @@ -348,7 +347,7 @@ impl<'a> From<&'a TargetIsa> for FlagsOrIsa<'a> { #[cfg(test)] mod tests { use super::Configurable; - use super::Error::*; + use super::SetError::*; use super::{builder, Flags}; use std::string::ToString; diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs index d7f6ce8747..b99066ea53 100644 --- a/lib/codegen/src/write.rs +++ b/lib/codegen/src/write.rs @@ -6,13 +6,12 @@ use ir::{DataFlowGraph, Ebb, Function, Inst, SigRef, Type, Value, ValueDef}; use isa::{RegInfo, TargetIsa}; use packed_option::ReservedValue; -use std::fmt::{self, Error, Result, Write}; -use std::result; +use std::fmt::{self, Write}; use std::string::String; /// Write `func` to `w` as equivalent text. /// Use `isa` to emit ISA-dependent annotations. -pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) -> Result { +pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) -> fmt::Result { let regs = isa.map(TargetIsa::register_info); let regs = regs.as_ref(); @@ -34,7 +33,7 @@ pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) - // // Function spec. -fn write_spec(w: &mut Write, func: &Function, regs: Option<&RegInfo>) -> Result { +fn write_spec(w: &mut Write, func: &Function, regs: Option<&RegInfo>) -> fmt::Result { write!(w, "{}{}", func.name, func.signature.display(regs)) } @@ -42,7 +41,7 @@ fn write_preamble( w: &mut Write, func: &Function, regs: Option<&RegInfo>, -) -> result::Result { +) -> Result { let mut any = false; for (ss, slot) in func.stack_slots.iter() { @@ -91,7 +90,12 @@ fn write_preamble( // // Basic blocks -pub fn write_arg(w: &mut Write, func: &Function, regs: Option<&RegInfo>, arg: Value) -> Result { +pub fn write_arg( + w: &mut Write, + func: &Function, + regs: Option<&RegInfo>, + arg: Value, +) -> fmt::Result { write!(w, "{}: {}", arg, func.dfg.value_type(arg))?; let loc = func.locations[arg]; if loc.is_assigned() { @@ -107,7 +111,7 @@ pub fn write_ebb_header( isa: Option<&TargetIsa>, ebb: Ebb, indent: usize, -) -> Result { +) -> fmt::Result { // Write out the basic block header, outdented: // // ebb1: @@ -137,7 +141,7 @@ pub fn write_ebb_header( writeln!(w, "):") } -pub fn write_ebb(w: &mut Write, func: &Function, isa: Option<&TargetIsa>, ebb: Ebb) -> Result { +pub fn write_ebb(w: &mut Write, func: &Function, isa: Option<&TargetIsa>, ebb: Ebb) -> fmt::Result { // Indent all instructions if any encodings are present. let indent = if func.encodings.is_empty() && func.srclocs.is_empty() { 4 @@ -191,7 +195,7 @@ fn type_suffix(func: &Function, inst: Inst) -> Option { } // Write out any value aliases appearing in `inst`. -fn write_value_aliases(w: &mut Write, func: &Function, inst: Inst, indent: usize) -> Result { +fn write_value_aliases(w: &mut Write, func: &Function, inst: Inst, indent: usize) -> fmt::Result { for &arg in func.dfg.inst_args(inst) { let resolved = func.dfg.resolve_aliases(arg); if resolved != arg { @@ -207,7 +211,7 @@ fn write_instruction( isa: Option<&TargetIsa>, inst: Inst, indent: usize, -) -> Result { +) -> fmt::Result { // Value aliases come out on lines before the instruction using them. write_value_aliases(w, func, inst, indent)?; @@ -272,7 +276,7 @@ pub fn write_operands( dfg: &DataFlowGraph, isa: Option<&TargetIsa>, inst: Inst, -) -> Result { +) -> fmt::Result { let pool = &dfg.value_lists; use ir::instructions::InstructionData::*; match dfg[inst] { @@ -472,7 +476,7 @@ pub fn write_operands( } /// Write EBB args using optional parantheses. -fn write_ebb_args(w: &mut Write, args: &[Value]) -> Result { +fn write_ebb_args(w: &mut Write, args: &[Value]) -> fmt::Result { if args.is_empty() { Ok(()) } else { @@ -484,7 +488,7 @@ fn write_ebb_args(w: &mut Write, args: &[Value]) -> Result { struct DisplayValues<'a>(&'a [Value]); impl<'a> fmt::Display for DisplayValues<'a> { - fn fmt(&self, f: &mut fmt::Formatter) -> Result { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { for (i, val) in self.0.iter().enumerate() { if i == 0 { write!(f, "{}", val)?; @@ -499,7 +503,7 @@ impl<'a> fmt::Display for DisplayValues<'a> { struct DisplayValuesWithDelimiter<'a>(&'a [Value], char); impl<'a> fmt::Display for DisplayValuesWithDelimiter<'a> { - fn fmt(&self, f: &mut fmt::Formatter) -> Result { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { for (i, val) in self.0.iter().enumerate() { if i == 0 { write!(f, "{}", val)?; diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs index 298e526d6c..85ff12a849 100644 --- a/lib/module/src/backend.rs +++ b/lib/module/src/backend.rs @@ -6,7 +6,6 @@ use cretonne_codegen::{binemit, ir}; use std::marker; use DataContext; use Linkage; -use ModuleError; use ModuleNamespace; use ModuleResult; diff --git a/lib/reader/src/error.rs b/lib/reader/src/error.rs index ed9eeb3cd9..e006ad5c3b 100644 --- a/lib/reader/src/error.rs +++ b/lib/reader/src/error.rs @@ -1,9 +1,8 @@ -//! Define the `Location`, `Error`, and `Result` types. +//! Define the `Location`, `ParseError`, and `ParseResult` types. #![macro_use] use std::fmt; -use std::result; /// The location of a `Token` or `Error`. #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] @@ -15,14 +14,14 @@ pub struct Location { /// A parse error is returned when the parse failed. #[derive(Debug)] -pub struct Error { +pub struct ParseError { /// Location of the error. pub location: Location, /// Error message. pub message: String, } -impl fmt::Display for Error { +impl fmt::Display for ParseError { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { if self.location.line_number == 0 { write!(f, "command-line arguments: {}", self.message) @@ -32,20 +31,20 @@ impl fmt::Display for Error { } } -/// Result of a parser operation. The `Error` variant includes a location. -pub type Result = result::Result; +/// Result of a parser operation. The `ParseError` variant includes a location. +pub type ParseResult = Result; -// Create an `Err` variant of `Result` from a location and `format!` args. +// Create an `Err` variant of `ParseResult` from a location and `format!` args. macro_rules! err { ( $loc:expr, $msg:expr ) => { - Err($crate::Error { + Err($crate::ParseError { location: $loc.clone(), message: $msg.to_string(), }) }; ( $loc:expr, $fmt:expr, $( $arg:expr ),+ ) => { - Err($crate::Error { + Err($crate::ParseError { location: $loc.clone(), message: format!( $fmt, $( $arg ),+ ), }) diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs index 852a05b646..18a11ced2d 100644 --- a/lib/reader/src/isaspec.rs +++ b/lib/reader/src/isaspec.rs @@ -7,8 +7,8 @@ //! ISAs. If the file contains no `isa` commands, the tests will be run against all supported ISAs. use cretonne_codegen::isa::TargetIsa; -use cretonne_codegen::settings::{Configurable, Error as SetError, Flags}; -use error::{Location, Result}; +use cretonne_codegen::settings::{Configurable, Flags, SetError}; +use error::{Location, ParseResult}; use testcommand::TestOption; /// The ISA specifications in a `.cton` file. @@ -35,7 +35,7 @@ impl IsaSpec { } /// Parse an iterator of command line options and apply them to `config`. -pub fn parse_options<'a, I>(iter: I, config: &mut Configurable, loc: &Location) -> Result<()> +pub fn parse_options<'a, I>(iter: I, config: &mut Configurable, loc: &Location) -> ParseResult<()> where I: Iterator, { diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 53a0ef8496..6c622ff64f 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -63,19 +63,19 @@ fn token(token: Token, loc: Location) -> Result { /// An error from the lexical analysis. #[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum Error { +pub enum LexError { InvalidChar, } -/// An `Error` with an associated Location. +/// A `LexError` with an associated Location. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct LocatedError { - pub error: Error, + pub error: LexError, pub location: Location, } -/// Wrap up an `Error` with the given location. -fn error<'a>(error: Error, loc: Location) -> Result, LocatedError> { +/// Wrap up a `LexError` with the given location. +fn error<'a>(error: LexError, loc: Location) -> Result, LocatedError> { Err(LocatedError { error, location: loc, @@ -468,7 +468,7 @@ impl<'a> Lexer<'a> { _ => { // Skip invalid char, return error. self.next_ch(); - Some(error(Error::InvalidChar, loc)) + Some(error(LexError::InvalidChar, loc)) } }; } @@ -511,7 +511,7 @@ mod tests { Some(super::token(token, Location { line_number: line })) } - fn error<'a>(error: Error, line: usize) -> Option, LocatedError>> { + fn error<'a>(error: LexError, line: usize) -> Option, LocatedError>> { Some(super::error(error, Location { line_number: line })) } @@ -539,7 +539,7 @@ mod tests { // Scan a comment after an invalid char. let mut lex = Lexer::new("$; hello"); - assert_eq!(lex.next(), error(Error::InvalidChar, 1)); + assert_eq!(lex.next(), error(LexError::InvalidChar, 1)); assert_eq!(lex.next(), token(Token::Comment("; hello"), 1)); assert_eq!(lex.next(), None); } diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index 1fbd10a607..edff983292 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -18,7 +18,7 @@ extern crate cretonne_codegen; extern crate target_lexicon; -pub use error::{Error, Location, Result}; +pub use error::{Location, ParseError, ParseResult}; pub use isaspec::{parse_options, IsaSpec}; pub use parser::{parse_functions, parse_test}; pub use sourcemap::SourceMap; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 256ba50847..4fba2a19c8 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -15,9 +15,9 @@ use cretonne_codegen::isa::{self, Encoding, RegUnit, TargetIsa}; use cretonne_codegen::packed_option::ReservedValue; use cretonne_codegen::settings::CallConv; use cretonne_codegen::{settings, timing}; -use error::{Error, Location, Result}; +use error::{Location, ParseError, ParseResult}; use isaspec; -use lexer::{self, Lexer, Token}; +use lexer::{LexError, Lexer, LocatedError, LocatedToken, Token}; use sourcemap::SourceMap; use std::mem; use std::str::FromStr; @@ -29,7 +29,7 @@ use testfile::{Comment, Details, TestFile}; /// Parse the entire `text` into a list of functions. /// /// Any test commands or target declarations are ignored. -pub fn parse_functions(text: &str) -> Result> { +pub fn parse_functions(text: &str) -> ParseResult> { let _tt = timing::parse_text(); parse_test(text).map(|file| file.functions.into_iter().map(|(func, _)| func).collect()) } @@ -37,7 +37,7 @@ pub fn parse_functions(text: &str) -> Result> { /// Parse the entire `text` as a test case file. /// /// The returned `TestFile` contains direct references to substrings of `text`. -pub fn parse_test(text: &str) -> Result { +pub fn parse_test(text: &str) -> ParseResult { let _tt = timing::parse_text(); let mut parser = Parser::new(text); // Gather the preamble comments. @@ -63,7 +63,7 @@ pub fn parse_test(text: &str) -> Result { pub struct Parser<'a> { lex: Lexer<'a>, - lex_error: Option, + lex_error: Option, /// Current lookahead token. lookahead: Option>, @@ -121,7 +121,7 @@ impl<'a> Context<'a> { } // Allocate a new stack slot. - fn add_ss(&mut self, ss: StackSlot, data: StackSlotData, loc: &Location) -> Result<()> { + fn add_ss(&mut self, ss: StackSlot, data: StackSlotData, loc: &Location) -> ParseResult<()> { while self.function.stack_slots.next_key().index() <= ss.index() { self.function .create_stack_slot(StackSlotData::new(StackSlotKind::SpillSlot, 0)); @@ -131,7 +131,7 @@ impl<'a> Context<'a> { } // Resolve a reference to a stack slot. - fn check_ss(&self, ss: StackSlot, loc: &Location) -> Result<()> { + fn check_ss(&self, ss: StackSlot, loc: &Location) -> ParseResult<()> { if !self.map.contains_ss(ss) { err!(loc, "undefined stack slot {}", ss) } else { @@ -140,7 +140,7 @@ impl<'a> Context<'a> { } // Allocate a global variable slot. - fn add_gv(&mut self, gv: GlobalVar, data: GlobalVarData, loc: &Location) -> Result<()> { + fn add_gv(&mut self, gv: GlobalVar, data: GlobalVarData, loc: &Location) -> ParseResult<()> { while self.function.global_vars.next_key().index() <= gv.index() { self.function.create_global_var(GlobalVarData::Sym { name: ExternalName::testcase(""), @@ -152,7 +152,7 @@ impl<'a> Context<'a> { } // Resolve a reference to a global variable. - fn check_gv(&self, gv: GlobalVar, loc: &Location) -> Result<()> { + fn check_gv(&self, gv: GlobalVar, loc: &Location) -> ParseResult<()> { if !self.map.contains_gv(gv) { err!(loc, "undefined global variable {}", gv) } else { @@ -161,7 +161,7 @@ impl<'a> Context<'a> { } // Allocate a heap slot. - fn add_heap(&mut self, heap: Heap, data: HeapData, loc: &Location) -> Result<()> { + fn add_heap(&mut self, heap: Heap, data: HeapData, loc: &Location) -> ParseResult<()> { while self.function.heaps.next_key().index() <= heap.index() { self.function.create_heap(HeapData { base: HeapBase::ReservedReg, @@ -177,7 +177,7 @@ impl<'a> Context<'a> { } // Resolve a reference to a heap. - fn check_heap(&self, heap: Heap, loc: &Location) -> Result<()> { + fn check_heap(&self, heap: Heap, loc: &Location) -> ParseResult<()> { if !self.map.contains_heap(heap) { err!(loc, "undefined heap {}", heap) } else { @@ -186,7 +186,7 @@ impl<'a> Context<'a> { } // Allocate a new signature. - fn add_sig(&mut self, sig: SigRef, data: Signature, loc: &Location) -> Result<()> { + fn add_sig(&mut self, sig: SigRef, data: Signature, loc: &Location) -> ParseResult<()> { while self.function.dfg.signatures.next_key().index() <= sig.index() { self.function .import_signature(Signature::new(CallConv::Fast)); @@ -196,7 +196,7 @@ impl<'a> Context<'a> { } // Resolve a reference to a signature. - fn check_sig(&self, sig: SigRef, loc: &Location) -> Result<()> { + fn check_sig(&self, sig: SigRef, loc: &Location) -> ParseResult<()> { if !self.map.contains_sig(sig) { err!(loc, "undefined signature {}", sig) } else { @@ -205,7 +205,7 @@ impl<'a> Context<'a> { } // Allocate a new external function. - fn add_fn(&mut self, fn_: FuncRef, data: ExtFuncData, loc: &Location) -> Result<()> { + fn add_fn(&mut self, fn_: FuncRef, data: ExtFuncData, loc: &Location) -> ParseResult<()> { while self.function.dfg.ext_funcs.next_key().index() <= fn_.index() { self.function.import_function(ExtFuncData { name: ExternalName::testcase(""), @@ -218,7 +218,7 @@ impl<'a> Context<'a> { } // Resolve a reference to a function. - fn check_fn(&self, fn_: FuncRef, loc: &Location) -> Result<()> { + fn check_fn(&self, fn_: FuncRef, loc: &Location) -> ParseResult<()> { if !self.map.contains_fn(fn_) { err!(loc, "undefined function {}", fn_) } else { @@ -227,7 +227,7 @@ impl<'a> Context<'a> { } // Allocate a new jump table. - fn add_jt(&mut self, jt: JumpTable, data: JumpTableData, loc: &Location) -> Result<()> { + fn add_jt(&mut self, jt: JumpTable, data: JumpTableData, loc: &Location) -> ParseResult<()> { while self.function.jump_tables.next_key().index() <= jt.index() { self.function.create_jump_table(JumpTableData::new()); } @@ -236,7 +236,7 @@ impl<'a> Context<'a> { } // Resolve a reference to a jump table. - fn check_jt(&self, jt: JumpTable, loc: &Location) -> Result<()> { + fn check_jt(&self, jt: JumpTable, loc: &Location) -> ParseResult<()> { if !self.map.contains_jt(jt) { err!(loc, "undefined jump table {}", jt) } else { @@ -245,7 +245,7 @@ impl<'a> Context<'a> { } // Assign the global for the stack limit. - fn set_stack_limit(&mut self, gv: GlobalVar, loc: &Location) -> Result<()> { + fn set_stack_limit(&mut self, gv: GlobalVar, loc: &Location) -> ParseResult<()> { if let Some(_) = self.function.set_stack_limit(Some(gv)) { err!(loc, "multiple stack_limit declarations") } else { @@ -254,7 +254,7 @@ impl<'a> Context<'a> { } // Allocate a new EBB. - fn add_ebb(&mut self, ebb: Ebb, loc: &Location) -> Result { + fn add_ebb(&mut self, ebb: Ebb, loc: &Location) -> ParseResult { while self.function.dfg.num_ebbs() <= ebb.index() { self.function.dfg.make_ebb(); } @@ -298,7 +298,7 @@ impl<'a> Parser<'a> { #[cfg_attr(feature = "cargo-clippy", allow(while_immutable_condition))] while self.lookahead == None { match self.lex.next() { - Some(Ok(lexer::LocatedToken { token, location })) => { + Some(Ok(LocatedToken { token, location })) => { match token { Token::Comment(text) => { if self.gathering_comments { @@ -309,7 +309,7 @@ impl<'a> Parser<'a> { } self.loc = location; } - Some(Err(lexer::LocatedError { error, location })) => { + Some(Err(LocatedError { error, location })) => { self.lex_error = Some(error); self.loc = location; break; @@ -347,7 +347,7 @@ impl<'a> Parser<'a> { } // Match and consume a token without payload. - fn match_token(&mut self, want: Token<'a>, err_msg: &str) -> Result> { + fn match_token(&mut self, want: Token<'a>, err_msg: &str) -> ParseResult> { if self.token() == Some(want) { Ok(self.consume()) } else { @@ -367,7 +367,7 @@ impl<'a> Parser<'a> { // Match and consume a specific identifier string. // Used for pseudo-keywords like "stack_slot" that only appear in certain contexts. - fn match_identifier(&mut self, want: &'static str, err_msg: &str) -> Result> { + fn match_identifier(&mut self, want: &'static str, err_msg: &str) -> ParseResult> { if self.token() == Some(Token::Identifier(want)) { Ok(self.consume()) } else { @@ -376,7 +376,7 @@ impl<'a> Parser<'a> { } // Match and consume a type. - fn match_type(&mut self, err_msg: &str) -> Result { + fn match_type(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::Type(t)) = self.token() { self.consume(); Ok(t) @@ -386,7 +386,7 @@ impl<'a> Parser<'a> { } // Match and consume a stack slot reference. - fn match_ss(&mut self, err_msg: &str) -> Result { + fn match_ss(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::StackSlot(ss)) = self.token() { self.consume(); if let Some(ss) = StackSlot::with_number(ss) { @@ -397,7 +397,7 @@ impl<'a> Parser<'a> { } // Match and consume a global variable reference. - fn match_gv(&mut self, err_msg: &str) -> Result { + fn match_gv(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::GlobalVar(gv)) = self.token() { self.consume(); if let Some(gv) = GlobalVar::with_number(gv) { @@ -408,7 +408,7 @@ impl<'a> Parser<'a> { } // Match and consume a function reference. - fn match_fn(&mut self, err_msg: &str) -> Result { + fn match_fn(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::FuncRef(fnref)) = self.token() { self.consume(); if let Some(fnref) = FuncRef::with_number(fnref) { @@ -419,7 +419,7 @@ impl<'a> Parser<'a> { } // Match and consume a signature reference. - fn match_sig(&mut self, err_msg: &str) -> Result { + fn match_sig(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::SigRef(sigref)) = self.token() { self.consume(); if let Some(sigref) = SigRef::with_number(sigref) { @@ -430,7 +430,7 @@ impl<'a> Parser<'a> { } // Match and consume a heap reference. - fn match_heap(&mut self, err_msg: &str) -> Result { + fn match_heap(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::Heap(heap)) = self.token() { self.consume(); if let Some(heap) = Heap::with_number(heap) { @@ -441,7 +441,7 @@ impl<'a> Parser<'a> { } // Match and consume a jump table reference. - fn match_jt(&mut self) -> Result { + fn match_jt(&mut self) -> ParseResult { if let Some(Token::JumpTable(jt)) = self.token() { self.consume(); if let Some(jt) = JumpTable::with_number(jt) { @@ -452,7 +452,7 @@ impl<'a> Parser<'a> { } // Match and consume an ebb reference. - fn match_ebb(&mut self, err_msg: &str) -> Result { + fn match_ebb(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::Ebb(ebb)) = self.token() { self.consume(); Ok(ebb) @@ -462,7 +462,7 @@ impl<'a> Parser<'a> { } // Match and consume a value reference, direct or vtable. - fn match_value(&mut self, err_msg: &str) -> Result { + fn match_value(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::Value(v)) = self.token() { self.consume(); Ok(v) @@ -471,15 +471,15 @@ impl<'a> Parser<'a> { } } - fn error(&self, message: &str) -> Error { - Error { + fn error(&self, message: &str) -> ParseError { + ParseError { location: self.loc, message: message.to_string(), } } // Match and consume an Imm64 immediate. - fn match_imm64(&mut self, err_msg: &str) -> Result { + fn match_imm64(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::Integer(text)) = self.token() { self.consume(); // Lexer just gives us raw text that looks like an integer. @@ -491,7 +491,7 @@ impl<'a> Parser<'a> { } // Match and consume a Uimm32 immediate. - fn match_uimm32(&mut self, err_msg: &str) -> Result { + fn match_uimm32(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::Integer(text)) = self.token() { self.consume(); // Lexer just gives us raw text that looks like an integer. @@ -504,7 +504,7 @@ impl<'a> Parser<'a> { // Match and consume a u8 immediate. // This is used for lane numbers in SIMD vectors. - fn match_uimm8(&mut self, err_msg: &str) -> Result { + fn match_uimm8(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::Integer(text)) = self.token() { self.consume(); // Lexer just gives us raw text that looks like an integer. @@ -518,7 +518,7 @@ impl<'a> Parser<'a> { // Match and consume an i32 immediate. // This is used for stack argument byte offsets. - fn match_imm32(&mut self, err_msg: &str) -> Result { + fn match_imm32(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::Integer(text)) = self.token() { self.consume(); // Lexer just gives us raw text that looks like an integer. @@ -534,7 +534,7 @@ impl<'a> Parser<'a> { // // Note that this will match an empty string as an empty offset, and that if an offset is // present, it must contain a sign. - fn optional_offset32(&mut self) -> Result { + fn optional_offset32(&mut self) -> ParseResult { if let Some(Token::Integer(text)) = self.token() { self.consume(); // Lexer just gives us raw text that looks like an integer. @@ -547,7 +547,7 @@ impl<'a> Parser<'a> { } // Match and consume an Ieee32 immediate. - fn match_ieee32(&mut self, err_msg: &str) -> Result { + fn match_ieee32(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::Float(text)) = self.token() { self.consume(); // Lexer just gives us raw text that looks like a float. @@ -559,7 +559,7 @@ impl<'a> Parser<'a> { } // Match and consume an Ieee64 immediate. - fn match_ieee64(&mut self, err_msg: &str) -> Result { + fn match_ieee64(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::Float(text)) = self.token() { self.consume(); // Lexer just gives us raw text that looks like a float. @@ -571,7 +571,7 @@ impl<'a> Parser<'a> { } // Match and consume a boolean immediate. - fn match_bool(&mut self, err_msg: &str) -> Result { + fn match_bool(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::Identifier(text)) = self.token() { self.consume(); match text { @@ -585,7 +585,7 @@ impl<'a> Parser<'a> { } // Match and consume an enumerated immediate, like one of the condition codes. - fn match_enum(&mut self, err_msg: &str) -> Result { + fn match_enum(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::Identifier(text)) = self.token() { self.consume(); text.parse().map_err(|_| self.error(err_msg)) @@ -608,7 +608,7 @@ impl<'a> Parser<'a> { } // Match and consume an identifier. - fn match_any_identifier(&mut self, err_msg: &str) -> Result<&'a str> { + fn match_any_identifier(&mut self, err_msg: &str) -> ParseResult<&'a str> { if let Some(Token::Identifier(text)) = self.token() { self.consume(); Ok(text) @@ -619,7 +619,7 @@ impl<'a> Parser<'a> { // Match and consume a HexSequence that fits into a u16. // This is used for instruction encodings. - fn match_hex16(&mut self, err_msg: &str) -> Result { + fn match_hex16(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::HexSequence(bits_str)) = self.token() { self.consume(); // The only error we anticipate from this parse is overflow, the lexer should @@ -633,7 +633,7 @@ impl<'a> Parser<'a> { } // Match and consume a register unit either by number `%15` or by name `%rax`. - fn match_regunit(&mut self, isa: Option<&TargetIsa>) -> Result { + fn match_regunit(&mut self, isa: Option<&TargetIsa>) -> ParseResult { if let Some(Token::Name(name)) = self.token() { self.consume(); match isa { @@ -654,7 +654,7 @@ impl<'a> Parser<'a> { /// Parse an optional source location. /// /// Return an optional source location if no real location is present. - fn optional_srcloc(&mut self) -> Result { + fn optional_srcloc(&mut self) -> ParseResult { if let Some(Token::SourceLoc(text)) = self.token() { match u32::from_str_radix(text, 16) { Ok(num) => { @@ -681,7 +681,7 @@ impl<'a> Parser<'a> { /// /// Accept a mix of `target` and `set` command lines. The `set` commands are cumulative. /// - pub fn parse_target_specs(&mut self) -> Result { + pub fn parse_target_specs(&mut self) -> ParseResult { // Was there any `target` commands? let mut seen_target = false; // Location of last `set` command since the last `target`. @@ -753,14 +753,14 @@ impl<'a> Parser<'a> { pub fn parse_function_list( &mut self, unique_isa: Option<&TargetIsa>, - ) -> Result)>> { + ) -> ParseResult)>> { let mut list = Vec::new(); while self.token().is_some() { list.push(self.parse_function(unique_isa)?); } if let Some(err) = self.lex_error { return match err { - lexer::Error::InvalidChar => err!(self.loc, "invalid character"), + LexError::InvalidChar => err!(self.loc, "invalid character"), }; } Ok(list) @@ -773,7 +773,7 @@ impl<'a> Parser<'a> { fn parse_function( &mut self, unique_isa: Option<&TargetIsa>, - ) -> Result<(Function, Details<'a>)> { + ) -> ParseResult<(Function, Details<'a>)> { // Begin gathering comments. // Make sure we don't include any comments before the `function` keyword. self.token(); @@ -825,7 +825,7 @@ impl<'a> Parser<'a> { // // function ::= "function" * name signature { ... } // - fn parse_external_name(&mut self) -> Result { + fn parse_external_name(&mut self) -> ParseResult { match self.token() { Some(Token::Name(s)) => { self.consume(); @@ -859,7 +859,7 @@ impl<'a> Parser<'a> { // // signature ::= * "(" [paramlist] ")" ["->" retlist] [callconv] // - fn parse_signature(&mut self, unique_isa: Option<&TargetIsa>) -> Result { + fn parse_signature(&mut self, unique_isa: Option<&TargetIsa>) -> ParseResult { // Calling convention defaults to `fast`, but can be changed. let mut sig = Signature::new(CallConv::Fast); @@ -895,7 +895,10 @@ impl<'a> Parser<'a> { // // paramlist ::= * param { "," param } // - fn parse_abi_param_list(&mut self, unique_isa: Option<&TargetIsa>) -> Result> { + fn parse_abi_param_list( + &mut self, + unique_isa: Option<&TargetIsa>, + ) -> ParseResult> { let mut list = Vec::new(); // abi-param-list ::= * abi-param { "," abi-param } @@ -911,7 +914,7 @@ impl<'a> Parser<'a> { } // Parse a single argument type with flags. - fn parse_abi_param(&mut self, unique_isa: Option<&TargetIsa>) -> Result { + fn parse_abi_param(&mut self, unique_isa: Option<&TargetIsa>) -> ParseResult { // abi-param ::= * type { flag } [ argumentloc ] let mut arg = AbiParam::new(self.match_type("expected parameter type")?); @@ -938,7 +941,10 @@ impl<'a> Parser<'a> { } // Parse an argument location specifier; either a register or a byte offset into the stack. - fn parse_argument_location(&mut self, unique_isa: Option<&TargetIsa>) -> Result { + fn parse_argument_location( + &mut self, + unique_isa: Option<&TargetIsa>, + ) -> ParseResult { // argumentloc ::= '[' regname | uimm32 ']' if self.optional(Token::LBracket) { let result = match self.token() { @@ -985,7 +991,7 @@ impl<'a> Parser<'a> { // * stack-limit-decl // // The parsed decls are added to `ctx` rather than returned. - fn parse_preamble(&mut self, ctx: &mut Context) -> Result<()> { + fn parse_preamble(&mut self, ctx: &mut Context) -> ParseResult<()> { loop { match self.token() { Some(Token::StackSlot(..)) => { @@ -1034,7 +1040,7 @@ impl<'a> Parser<'a> { // | "spill_slot" // | "incoming_arg" // | "outgoing_arg" - fn parse_stack_slot_decl(&mut self) -> Result<(StackSlot, StackSlotData)> { + fn parse_stack_slot_decl(&mut self) -> ParseResult<(StackSlot, StackSlotData)> { let ss = self.match_ss("expected stack slot number: ss«n»")?; self.match_token(Token::Equal, "expected '=' in stack slot declaration")?; let kind = self.match_enum("expected stack slot kind")?; @@ -1073,7 +1079,7 @@ impl<'a> Parser<'a> { // | "deref" "(" GlobalVar(base) ")" offset32 // | globalsym ["colocated"] name // - fn parse_global_var_decl(&mut self) -> Result<(GlobalVar, GlobalVarData)> { + fn parse_global_var_decl(&mut self) -> ParseResult<(GlobalVar, GlobalVarData)> { let gv = self.match_gv("expected global variable number: gv«n»")?; self.match_token(Token::Equal, "expected '=' in global variable declaration")?; @@ -1116,7 +1122,7 @@ impl<'a> Parser<'a> { // | "max" Imm64(bytes) // | "guard" Imm64(bytes) // - fn parse_heap_decl(&mut self) -> Result<(Heap, HeapData)> { + fn parse_heap_decl(&mut self) -> ParseResult<(Heap, HeapData)> { let heap = self.match_heap("expected heap number: heap«n»")?; self.match_token(Token::Equal, "expected '=' in heap declaration")?; @@ -1183,7 +1189,7 @@ impl<'a> Parser<'a> { fn parse_signature_decl( &mut self, unique_isa: Option<&TargetIsa>, - ) -> Result<(SigRef, Signature)> { + ) -> ParseResult<(SigRef, Signature)> { let sig = self.match_sig("expected signature number: sig«n»")?; self.match_token(Token::Equal, "expected '=' in signature decl")?; let data = self.parse_signature(unique_isa)?; @@ -1205,7 +1211,7 @@ impl<'a> Parser<'a> { // The first variant allocates a new signature reference. The second references an existing // signature which must be declared first. // - fn parse_function_decl(&mut self, ctx: &mut Context) -> Result<(FuncRef, ExtFuncData)> { + fn parse_function_decl(&mut self, ctx: &mut Context) -> ParseResult<(FuncRef, ExtFuncData)> { let fn_ = self.match_fn("expected function number: fn«n»")?; self.match_token(Token::Equal, "expected '=' in function decl")?; @@ -1260,7 +1266,7 @@ impl<'a> Parser<'a> { // Parse a jump table decl. // // jump-table-decl ::= * JumpTable(jt) "=" "jump_table" jt-entry {"," jt-entry} - fn parse_jump_table_decl(&mut self) -> Result<(JumpTable, JumpTableData)> { + fn parse_jump_table_decl(&mut self) -> ParseResult<(JumpTable, JumpTableData)> { let jt = self.match_jt()?; self.match_token(Token::Equal, "expected '=' in jump_table decl")?; self.match_identifier("jump_table", "expected 'jump_table'")?; @@ -1285,7 +1291,7 @@ impl<'a> Parser<'a> { } // jt-entry ::= * Ebb(dest) | "0" - fn parse_jump_table_entry(&mut self) -> Result> { + fn parse_jump_table_entry(&mut self) -> ParseResult> { match self.token() { Some(Token::Integer(s)) => { if s == "0" { @@ -1304,7 +1310,7 @@ impl<'a> Parser<'a> { } /// stack-limit-decl ::= "stack_limit" "=" GlobalVar(gv) - fn parse_stack_limit_decl(&mut self) -> Result { + fn parse_stack_limit_decl(&mut self) -> ParseResult { self.consume(); self.match_token(Token::Equal, "expected '=' in stack limit declaration")?; let gv = self.match_gv("expected global variable")?; @@ -1316,7 +1322,7 @@ impl<'a> Parser<'a> { // // function-body ::= * { extended-basic-block } // - fn parse_function_body(&mut self, ctx: &mut Context) -> Result<()> { + fn parse_function_body(&mut self, ctx: &mut Context) -> ParseResult<()> { while self.token() != Some(Token::RBrace) { self.parse_extended_basic_block(ctx)?; } @@ -1352,7 +1358,7 @@ impl<'a> Parser<'a> { // extended-basic-block ::= * ebb-header { instruction } // ebb-header ::= Ebb(ebb) [ebb-params] ":" // - fn parse_extended_basic_block(&mut self, ctx: &mut Context) -> Result<()> { + fn parse_extended_basic_block(&mut self, ctx: &mut Context) -> ParseResult<()> { // Collect comments for the next ebb. self.start_gathering_comments(); @@ -1415,7 +1421,7 @@ impl<'a> Parser<'a> { // value numbers of the defined values and the defined types. // // ebb-params ::= * "(" ebb-param { "," ebb-param } ")" - fn parse_ebb_params(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { + fn parse_ebb_params(&mut self, ctx: &mut Context, ebb: Ebb) -> ParseResult<()> { // ebb-params ::= * "(" ebb-param { "," ebb-param } ")" self.match_token(Token::LPar, "expected '(' before EBB parameters")?; @@ -1439,7 +1445,7 @@ impl<'a> Parser<'a> { // ebb-param ::= * Value(v) ":" Type(t) arg-loc? // arg-loc ::= "[" value-location "]" // - fn parse_ebb_param(&mut self, ctx: &mut Context, ebb: Ebb) -> Result<()> { + fn parse_ebb_param(&mut self, ctx: &mut Context, ebb: Ebb) -> ParseResult<()> { // ebb-param ::= * Value(v) ":" Type(t) arg-loc? let v = self.match_value("EBB argument must be a value")?; let v_location = self.loc; @@ -1466,7 +1472,7 @@ impl<'a> Parser<'a> { Ok(()) } - fn parse_value_location(&mut self, ctx: &Context) -> Result { + fn parse_value_location(&mut self, ctx: &Context) -> ParseResult { match self.token() { Some(Token::StackSlot(src_num)) => { self.consume(); @@ -1505,7 +1511,7 @@ impl<'a> Parser<'a> { fn parse_instruction_encoding( &mut self, ctx: &Context, - ) -> Result<(Option, Option>)> { + ) -> ParseResult<(Option, Option>)> { let (mut encoding, mut result_locations) = (None, None); // encoding ::= "[" encoding_literal result_locations "]" @@ -1551,7 +1557,7 @@ impl<'a> Parser<'a> { // // inst-results ::= Value(v) { "," Value(v) } // - fn parse_inst_results(&mut self) -> Result> { + fn parse_inst_results(&mut self) -> ParseResult> { // Result value numbers. let mut results = Vec::new(); @@ -1576,7 +1582,7 @@ impl<'a> Parser<'a> { // // value_alias ::= [inst-results] "->" Value(v) // - fn parse_value_alias(&mut self, results: &[Value], ctx: &mut Context) -> Result<()> { + fn parse_value_alias(&mut self, results: &[Value], ctx: &mut Context) -> ParseResult<()> { if results.len() != 1 { return err!(self.loc, "wrong number of aliases"); } @@ -1623,7 +1629,7 @@ impl<'a> Parser<'a> { result_locations: Option>, ctx: &mut Context, ebb: Ebb, - ) -> Result<()> { + ) -> ParseResult<()> { // Define the result values. for val in results { ctx.map.def_value(*val, &self.loc)?; @@ -1731,7 +1737,7 @@ impl<'a> Parser<'a> { opcode: Opcode, explicit_ctrl_type: Option, inst_data: &InstructionData, - ) -> Result { + ) -> ParseResult { let constraints = opcode.constraints(); let ctrl_type = match explicit_ctrl_type { Some(t) => t, @@ -1808,7 +1814,7 @@ impl<'a> Parser<'a> { // // value_list ::= [ value { "," value } ] // - fn parse_value_list(&mut self) -> Result { + fn parse_value_list(&mut self) -> ParseResult { let mut args = VariableArgs::new(); if let Some(Token::Value(v)) = self.token() { @@ -1825,7 +1831,7 @@ impl<'a> Parser<'a> { Ok(args) } - fn parse_value_sequence(&mut self) -> Result { + fn parse_value_sequence(&mut self) -> ParseResult { let mut args = VariableArgs::new(); if let Some(Token::Value(v)) = self.token() { @@ -1843,7 +1849,7 @@ impl<'a> Parser<'a> { } // Parse an optional value list enclosed in parantheses. - fn parse_opt_value_list(&mut self) -> Result { + fn parse_opt_value_list(&mut self) -> ParseResult { if !self.optional(Token::LPar) { return Ok(VariableArgs::new()); } @@ -1861,7 +1867,7 @@ impl<'a> Parser<'a> { &mut self, ctx: &mut Context, opcode: Opcode, - ) -> Result { + ) -> ParseResult { let idata = match opcode.format() { InstructionFormat::Unary => InstructionData::Unary { opcode, @@ -2289,7 +2295,7 @@ mod tests { use cretonne_codegen::ir::StackSlotKind; use cretonne_codegen::ir::{ArgumentExtension, ArgumentPurpose}; use cretonne_codegen::settings::CallConv; - use error::Error; + use error::ParseError; use isaspec::IsaSpec; use testfile::{Comment, Details}; @@ -2300,7 +2306,7 @@ mod tests { assert_eq!(arg.value_type, types::I32); assert_eq!(arg.extension, ArgumentExtension::Sext); assert_eq!(arg.purpose, ArgumentPurpose::Normal); - let Error { location, message } = p.parse_abi_param(None).unwrap_err(); + let ParseError { location, message } = p.parse_abi_param(None).unwrap_err(); assert_eq!(location.line_number, 1); assert_eq!(message, "expected parameter type"); } diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 8bd44fb6d6..6b9dc51d02 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -8,7 +8,7 @@ use cretonne_codegen::ir::entities::AnyEntity; use cretonne_codegen::ir::{Ebb, FuncRef, GlobalVar, Heap, JumpTable, SigRef, StackSlot, Value}; -use error::{Location, Result}; +use error::{Location, ParseResult}; use lexer::split_entity_name; use std::collections::HashMap; @@ -140,48 +140,48 @@ impl SourceMap { } /// Define the value `entity`. - pub fn def_value(&mut self, entity: Value, loc: &Location) -> Result<()> { + pub fn def_value(&mut self, entity: Value, loc: &Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the ebb `entity`. - pub fn def_ebb(&mut self, entity: Ebb, loc: &Location) -> Result<()> { + pub fn def_ebb(&mut self, entity: Ebb, loc: &Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the stack slot `entity`. - pub fn def_ss(&mut self, entity: StackSlot, loc: &Location) -> Result<()> { + pub fn def_ss(&mut self, entity: StackSlot, loc: &Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the global variable `entity`. - pub fn def_gv(&mut self, entity: GlobalVar, loc: &Location) -> Result<()> { + pub fn def_gv(&mut self, entity: GlobalVar, loc: &Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the heap `entity`. - pub fn def_heap(&mut self, entity: Heap, loc: &Location) -> Result<()> { + pub fn def_heap(&mut self, entity: Heap, loc: &Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the signature `entity`. - pub fn def_sig(&mut self, entity: SigRef, loc: &Location) -> Result<()> { + pub fn def_sig(&mut self, entity: SigRef, loc: &Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the external function `entity`. - pub fn def_fn(&mut self, entity: FuncRef, loc: &Location) -> Result<()> { + pub fn def_fn(&mut self, entity: FuncRef, loc: &Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the jump table `entity`. - pub fn def_jt(&mut self, entity: JumpTable, loc: &Location) -> Result<()> { + pub fn def_jt(&mut self, entity: JumpTable, loc: &Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define an entity. This can be used for instructions whose numbers never /// appear in source, or implicitly defined signatures. - pub fn def_entity(&mut self, entity: AnyEntity, loc: &Location) -> Result<()> { + pub fn def_entity(&mut self, entity: AnyEntity, loc: &Location) -> ParseResult<()> { if self.locations.insert(entity, *loc).is_some() { err!(loc, "duplicate entity: {}", entity) } else { diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs index 2418193f34..3f0ef726cd 100644 --- a/lib/wasm/tests/wasm_testsuite.rs +++ b/lib/wasm/tests/wasm_testsuite.rs @@ -47,7 +47,7 @@ fn return_at_end() { handle_module(Path::new("../../wasmtests/return_at_end.wat"), &flags); } -fn read_file(path: &Path) -> Result, io::Error> { +fn read_file(path: &Path) -> io::Result> { let mut buf: Vec = Vec::new(); let mut file = File::open(path)?; file.read_to_end(&mut buf)?; From a7813c4448853cbf627463536cdbc7ec98b23ed8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Jun 2018 06:36:07 -0700 Subject: [PATCH 1844/3084] Update to target-lexicon 0.0.2. This fixes compilation on rust 1.22.1. --- cranelift/Cargo.toml | 2 +- cranelift/fuzz/Cargo.toml | 21 ++++++--------------- lib/codegen/Cargo.toml | 2 +- lib/faerie/Cargo.toml | 4 ++-- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 2 +- lib/simplejit/Cargo.toml | 4 ++-- lib/wasm/Cargo.toml | 2 +- 8 files changed, 16 insertions(+), 25 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 6be8f607bb..c0d22b0b4d 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -31,7 +31,7 @@ serde_derive = "1.0.8" term = "0.5.1" capstone = "0.4" wabt = { version = "0.3", optional = true } -target-lexicon = "0.0.1" +target-lexicon = "0.0.2" [features] default = ["wasm"] diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml index a5bbdbd912..dcda236717 100644 --- a/cranelift/fuzz/Cargo.toml +++ b/cranelift/fuzz/Cargo.toml @@ -7,21 +7,12 @@ publish = false [package.metadata] cargo-fuzz = true -[dependencies.cargo-fuzz] -version = "*" - -[dependencies.binaryen] -git = "https://github.com/pepyakin/binaryen-rs.git" -version = "*" - -[dependencies.libfuzzer-sys] -git = "https://github.com/rust-fuzz/libfuzzer-sys.git" - -[dependencies.cretonne-wasm] -path = "../lib/wasm" - -[dependencies.target-lexicon] -version = "0.0.1" +[dependencies] +cargo-fuzz = "*" +binaryen = { git = "https://github.com/pepyakin/binaryen-rs.git" } +libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" } +cretonne-wasm = { path = "../lib/wasm" } +target-lexicon = "0.0.2" # Prevent this from interfering with workspaces [workspace] diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 7a6f76a85a..052e61c46c 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -15,7 +15,7 @@ cretonne-entity = { path = "../entity", version = "0.9.0", default-features = fa failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } hashmap_core = { version = "0.1.6", optional = true } -target-lexicon = { version = "0.0.1", default-features = false } +target-lexicon = { version = "0.0.2", default-features = false } # It is a goal of the cretonne-codegen crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index 886a5222a6..d37dd8e2f7 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -11,10 +11,10 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.9.0" } cretonne-module = { path = "../module", version = "0.9.0" } -faerie = "0.4.1" +faerie = "0.4.2" goblin = "0.0.15" failure = "0.1.1" -target-lexicon = "0.0.1" +target-lexicon = "0.0.2" [badges] maintenance = { status = "experimental" } diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 733016df77..502b4b757e 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -9,14 +9,14 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } -target-lexicon = { version = "0.0.1", default-features = false } +target-lexicon = { version = "0.0.2", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] raw-cpuid = "3.1.0" [features] default = ["std"] -std = ["cretonne-codegen/std"] +std = ["cretonne-codegen/std", "target-lexicon/std"] # when compiling with the "core" feature, nightly must be enabled # enabling the "nightly" feature for raw-cpuid allows avoiding # linking in a c-library. diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index a57971e963..3c3064f0a3 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -10,7 +10,7 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.9.0" } -target-lexicon = "0.0.1" +target-lexicon = "0.0.2" [badges] maintenance = { status = "experimental" } diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index bf23dfb8f0..34953956c2 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -15,14 +15,14 @@ cretonne-native = { path = "../native", version = "0.9.0", default-features = fa region = "0.3.0" libc = { version = "0.2.40", default-features = false } errno = "0.2.3" -target-lexicon = { version = "0.0.1", default-features = false } +target-lexicon = { version = "0.0.2", default-features = false } [target.'cfg(target_os = "windows")'.dependencies] winapi = { version = "0.3", features = ["winbase", "memoryapi"] } [features] default = ["std"] -std = ["libc/use_std", "cretonne-codegen/std", "cretonne-module/std", "cretonne-native/std"] +std = ["libc/use_std", "cretonne-codegen/std", "cretonne-module/std", "cretonne-native/std", "target-lexicon/std"] core = ["cretonne-codegen/core", "cretonne-module/core", "cretonne-native/core"] [badges] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 60fa48a776..8d5a67c013 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -15,7 +15,7 @@ cretonne-frontend = { path = "../frontend", version = "0.9.0", default-features hashmap_core = { version = "0.1.6", optional = true } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } -target-lexicon = { version = "0.0.1", default-features = false } +target-lexicon = { version = "0.0.2", default-features = false } [dev-dependencies] wabt = "0.3" From 399860e2aacdc74ba5befd19b2fafe7ef67abdc3 Mon Sep 17 00:00:00 2001 From: Sergey Pepyakin Date: Sat, 9 Jun 2018 03:20:12 +0300 Subject: [PATCH 1845/3084] Encode bnot on x86. --- cranelift/filetests/isa/x86/binary64.cton | 9 +++++++++ lib/codegen/meta/isa/x86/encodings.py | 3 +++ lib/codegen/meta/isa/x86/recipes.py | 8 ++++++++ 3 files changed, 20 insertions(+) diff --git a/cranelift/filetests/isa/x86/binary64.cton b/cranelift/filetests/isa/x86/binary64.cton index 96788c629b..b9e56c1ccb 100644 --- a/cranelift/filetests/isa/x86/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.cton @@ -913,6 +913,15 @@ ebb0: ; asm: movsbl -50000(%rsi), %edx [-,%rdx] v39 = sload8.i32 v2-50000 ; bin: heap_oob 0f be 96 ffff3cb0 + ; Integer Register Operations. + + ; asm: notl %ecx + [-,%rcx] v4000 = bnot v1 ; bin: f7 d1 + ; asm: notl %esi + [-,%rsi] v4001 = bnot v2 ; bin: f7 d6 + ; asm: notl %r10d + [-,%r10] v4002 = bnot v3 ; bin: 41 f7 d2 + ; Integer Register-Register Operations. ; asm: addl %esi, %ecx diff --git a/lib/codegen/meta/isa/x86/encodings.py b/lib/codegen/meta/isa/x86/encodings.py index 2909aac315..62dbf1dcf7 100644 --- a/lib/codegen/meta/isa/x86/encodings.py +++ b/lib/codegen/meta/isa/x86/encodings.py @@ -150,6 +150,9 @@ for inst, opc in [ (base.bxor, 0x31)]: enc_i32_i64(inst, r.rr, opc) +# x86 has a bitwise not instruction NOT. +enc_i32_i64(base.bnot, r.ur, 0xf7, rrr=2) + # Also add a `b1` encodings for the logic instructions. # TODO: Should this be done with 8-bit instructions? It would improve # partial register dependencies. diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py index 8828f8a06e..1caf6136d1 100644 --- a/lib/codegen/meta/isa/x86/recipes.py +++ b/lib/codegen/meta/isa/x86/recipes.py @@ -348,6 +348,14 @@ fax = TailRecipe( modrm_rr(in_reg0, in_reg1, sink); ''') +# XX /n for a unary operation with extension bits. +ur = TailRecipe( + 'ur', Unary, size=1, ins=GPR, outs=0, + emit=''' + PUT_OP(bits, rex1(in_reg0), sink); + modrm_r_bits(in_reg0, bits, sink); + ''') + # XX /r, but for a unary operator with separate input/output register, like # copies. MR form, preserving flags. umr = TailRecipe( From a8923e218515a0bfdb9d29b031afd4a8349899f9 Mon Sep 17 00:00:00 2001 From: Sergey Pepyakin Date: Sun, 10 Jun 2018 15:41:13 +0300 Subject: [PATCH 1846/3084] Add 64-bit versions. --- cranelift/filetests/isa/x86/binary64.cton | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/cranelift/filetests/isa/x86/binary64.cton b/cranelift/filetests/isa/x86/binary64.cton index b9e56c1ccb..0b0deb4143 100644 --- a/cranelift/filetests/isa/x86/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.cton @@ -43,6 +43,15 @@ ebb0: ; asm: movb $1, %sil [-,%r10] v9008 = bconst.b1 true ; bin: 41 ba 00000001 + ; Integer Register Operations. + + ; asm: notq %rcx + [-,%rcx] v4000 = bnot v1 ; bin: 48 f7 d1 + ; asm: notq %rsi + [-,%rsi] v4001 = bnot v2 ; bin: 48 f7 d6 + ; asm: notq %r10 + [-,%r10] v4002 = bnot v3 ; bin: 49 f7 d2 + ; Integer Register-Register Operations. ; asm: addq %rsi, %rcx From 9542cab5edeed6a2924081df5df9a79332a1f8c5 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 12 Jun 2018 12:29:38 -0700 Subject: [PATCH 1847/3084] Bump version to 0.10.0 --- cranelift/Cargo.toml | 22 +++++++++++----------- cranelift/publish-all.sh | 2 +- lib/codegen/Cargo.toml | 4 ++-- lib/entity/Cargo.toml | 2 +- lib/faerie/Cargo.toml | 6 +++--- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/module/Cargo.toml | 6 +++--- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/simplejit/Cargo.toml | 8 ++++---- lib/umbrella/Cargo.toml | 6 +++--- lib/wasm/Cargo.toml | 6 +++--- 13 files changed, 40 insertions(+), 40 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index c0d22b0b4d..faf0ef661e 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.9.0" +version = "0.10.0" description = "Binaries for testing the Cretonne libraries" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -14,16 +14,16 @@ path = "src/cton-util.rs" [dependencies] cfg-if = "0.1" -cretonne-codegen = { path = "lib/codegen", version = "0.9.0" } -cretonne-reader = { path = "lib/reader", version = "0.9.0" } -cretonne-frontend = { path = "lib/frontend", version = "0.9.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.9.0", optional = true } -cretonne-native = { path = "lib/native", version = "0.9.0" } -cretonne-filetests = { path = "lib/filetests", version = "0.9.0" } -cretonne-module = { path = "lib/module", version = "0.9.0" } -cretonne-faerie = { path = "lib/faerie", version = "0.9.0" } -cretonne-simplejit = { path = "lib/simplejit", version = "0.9.0" } -cretonne = { path = "lib/umbrella", version = "0.9.0" } +cretonne-codegen = { path = "lib/codegen", version = "0.10.0" } +cretonne-reader = { path = "lib/reader", version = "0.10.0" } +cretonne-frontend = { path = "lib/frontend", version = "0.10.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.10.0", optional = true } +cretonne-native = { path = "lib/native", version = "0.10.0" } +cretonne-filetests = { path = "lib/filetests", version = "0.10.0" } +cretonne-module = { path = "lib/module", version = "0.10.0" } +cretonne-faerie = { path = "lib/faerie", version = "0.10.0" } +cretonne-simplejit = { path = "lib/simplejit", version = "0.10.0" } +cretonne = { path = "lib/umbrella", version = "0.10.0" } filecheck = "0.3.0" docopt = "1" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 543f9bbeca..2ca20cc972 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ topdir=$(dirname "$0") cd "$topdir" # All the cretonne-* crates have the same version number -version="0.9.0" +version="0.10.0" # Update all of the Cargo.toml files. # diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 052e61c46c..208d806cb8 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-codegen" -version = "0.9.0" +version = "0.10.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"] build = "build.rs" [dependencies] -cretonne-entity = { path = "../entity", version = "0.9.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.10.0", default-features = false } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } hashmap_core = { version = "0.1.6", optional = true } diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index 669842223d..bfd868692a 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-entity" -version = "0.9.0" +version = "0.10.0" description = "Data structures using entity references as mapping keys" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index d37dd8e2f7..793640383b 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-faerie" -version = "0.9.0" +version = "0.10.0" authors = ["The Cretonne Project Developers"] description = "Emit Cretonne output to native object files with Faerie" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.9.0" } -cretonne-module = { path = "../module", version = "0.9.0" } +cretonne-codegen = { path = "../codegen", version = "0.10.0" } +cretonne-module = { path = "../module", version = "0.10.0" } faerie = "0.4.2" goblin = "0.0.15" failure = "0.1.1" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 9dd00a4cb6..90c11b1129 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.9.0" +version = "0.10.0" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" publish = false [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.9.0" } -cretonne-reader = { path = "../reader", version = "0.9.0" } +cretonne-codegen = { path = "../codegen", version = "0.10.0" } +cretonne-reader = { path = "../reader", version = "0.10.0" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 97d935a589..389bb9064c 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.9.0" +version = "0.10.0" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.10.0", default-features = false } [features] default = ["std"] diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 39ef56eb19..0afe1fd540 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-module" -version = "0.9.0" +version = "0.10.0" authors = ["The Cretonne Project Developers"] description = "Support for linking functions and data with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } -cretonne-entity = { path = "../entity", version = "0.9.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.10.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.10.0", default-features = false } hashmap_core = { version = "0.1.6", optional = true } failure = "0.1.1" diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 502b4b757e..35eff0c99f 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.9.0" +version = "0.10.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -8,7 +8,7 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.10.0", default-features = false } target-lexicon = { version = "0.0.2", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 3c3064f0a3..3132c063ff 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.9.0" +version = "0.10.0" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.9.0" } +cretonne-codegen = { path = "../codegen", version = "0.10.0" } target-lexicon = "0.0.2" [badges] diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index 34953956c2..c66a66f5ec 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-simplejit" -version = "0.9.0" +version = "0.10.0" authors = ["The Cretonne Project Developers"] description = "A simple JIT library backed by Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,9 +9,9 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } -cretonne-module = { path = "../module", version = "0.9.0", default-features = false } -cretonne-native = { path = "../native", version = "0.9.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.10.0", default-features = false } +cretonne-module = { path = "../module", version = "0.10.0", default-features = false } +cretonne-native = { path = "../native", version = "0.10.0", default-features = false } region = "0.3.0" libc = { version = "0.2.40", default-features = false } errno = "0.2.3" diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml index 244c4661ba..edb702ff9a 100644 --- a/lib/umbrella/Cargo.toml +++ b/lib/umbrella/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.9.0" +version = "0.10.0" description = "Umbrella for commonly-used cretonne crates" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -10,8 +10,8 @@ readme = "README.md" keywords = ["compile", "compiler", "jit"] [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.9.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.10.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.10.0", default-features = false } [features] default = ["std"] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 8d5a67c013..c39f3460eb 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.9.0" +version = "0.10.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/cretonne/cretonne" @@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"] [dependencies] wasmparser = { version = "0.17.0", default-features = false } -cretonne-codegen = { path = "../codegen", version = "0.9.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.9.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.10.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.10.0", default-features = false } hashmap_core = { version = "0.1.6", optional = true } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } From 1cab2f2d732afdcd236769462c59a72be702983a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 13 Jun 2018 07:41:39 -0700 Subject: [PATCH 1848/3084] Remove an unneeded comment. --- lib/codegen/src/settings.rs | 2 -- 1 file changed, 2 deletions(-) diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs index 2e6ad5f684..c11ca1a775 100644 --- a/lib/codegen/src/settings.rs +++ b/lib/codegen/src/settings.rs @@ -131,8 +131,6 @@ impl Configurable for Builder { let (offset, detail) = self.lookup(name)?; match detail { Detail::Bool { bit } => { - // Cannot currently propagate SetResult<()> up on functions returning () - // with the `?` operator self.set_bit(offset, bit, parse_bool_value(value)?); } Detail::Num => { From c506f4cf6b34c5e5afdf210c4c4082e23946643a Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 15 Jun 2018 08:38:40 -0700 Subject: [PATCH 1849/3084] Use WasmError::Unsupported rather than panic for unsupported extensions. --- lib/wasm/src/code_translator.rs | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 4b92c2ccc3..07e912da1d 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -27,7 +27,7 @@ use cretonne_codegen::ir::types::*; use cretonne_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags}; use cretonne_codegen::packed_option::ReservedValue; use cretonne_frontend::{FunctionBuilder, Variable}; -use environ::{FuncEnvironment, GlobalValue, WasmResult}; +use environ::{FuncEnvironment, GlobalValue, WasmError, WasmResult}; use state::{ControlStackFrame, TranslationState}; use std::collections::{hash_map, HashMap}; use std::vec::Vec; @@ -636,7 +636,9 @@ pub fn translate_operator( | Operator::I64TruncUSatF32 | Operator::I32TruncUSatF64 | Operator::I32TruncUSatF32 => { - panic!("proposed saturating conversion operators not yet supported"); + return Err(WasmError::Unsupported( + "proposed saturating conversion operators", + )); } Operator::F32ReinterpretI32 => { let val = state.pop1(); @@ -881,7 +883,7 @@ pub fn translate_operator( | Operator::I64AtomicRmw8UCmpxchg { .. } | Operator::I64AtomicRmw16UCmpxchg { .. } | Operator::I64AtomicRmw32UCmpxchg { .. } => { - panic!("proposed thread operators not yet supported"); + return Err(WasmError::Unsupported("proposed thread operators")); } }) } From dfc7065252da53e325d8fccb1987579091bc120c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 15 Jun 2018 08:46:25 -0700 Subject: [PATCH 1850/3084] Bump version to 0.11.0 --- cranelift/Cargo.toml | 22 +++++++++++----------- cranelift/publish-all.sh | 2 +- lib/codegen/Cargo.toml | 4 ++-- lib/entity/Cargo.toml | 2 +- lib/faerie/Cargo.toml | 6 +++--- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/module/Cargo.toml | 6 +++--- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/simplejit/Cargo.toml | 8 ++++---- lib/umbrella/Cargo.toml | 6 +++--- lib/wasm/Cargo.toml | 6 +++--- 13 files changed, 40 insertions(+), 40 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index faf0ef661e..ee41dbac43 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.10.0" +version = "0.11.0" description = "Binaries for testing the Cretonne libraries" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -14,16 +14,16 @@ path = "src/cton-util.rs" [dependencies] cfg-if = "0.1" -cretonne-codegen = { path = "lib/codegen", version = "0.10.0" } -cretonne-reader = { path = "lib/reader", version = "0.10.0" } -cretonne-frontend = { path = "lib/frontend", version = "0.10.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.10.0", optional = true } -cretonne-native = { path = "lib/native", version = "0.10.0" } -cretonne-filetests = { path = "lib/filetests", version = "0.10.0" } -cretonne-module = { path = "lib/module", version = "0.10.0" } -cretonne-faerie = { path = "lib/faerie", version = "0.10.0" } -cretonne-simplejit = { path = "lib/simplejit", version = "0.10.0" } -cretonne = { path = "lib/umbrella", version = "0.10.0" } +cretonne-codegen = { path = "lib/codegen", version = "0.11.0" } +cretonne-reader = { path = "lib/reader", version = "0.11.0" } +cretonne-frontend = { path = "lib/frontend", version = "0.11.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.11.0", optional = true } +cretonne-native = { path = "lib/native", version = "0.11.0" } +cretonne-filetests = { path = "lib/filetests", version = "0.11.0" } +cretonne-module = { path = "lib/module", version = "0.11.0" } +cretonne-faerie = { path = "lib/faerie", version = "0.11.0" } +cretonne-simplejit = { path = "lib/simplejit", version = "0.11.0" } +cretonne = { path = "lib/umbrella", version = "0.11.0" } filecheck = "0.3.0" docopt = "1" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 2ca20cc972..e536b5d710 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ topdir=$(dirname "$0") cd "$topdir" # All the cretonne-* crates have the same version number -version="0.10.0" +version="0.11.0" # Update all of the Cargo.toml files. # diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 208d806cb8..7abab7e938 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-codegen" -version = "0.10.0" +version = "0.11.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"] build = "build.rs" [dependencies] -cretonne-entity = { path = "../entity", version = "0.10.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.11.0", default-features = false } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } hashmap_core = { version = "0.1.6", optional = true } diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index bfd868692a..aee7848a08 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-entity" -version = "0.10.0" +version = "0.11.0" description = "Data structures using entity references as mapping keys" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index 793640383b..0df347f635 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-faerie" -version = "0.10.0" +version = "0.11.0" authors = ["The Cretonne Project Developers"] description = "Emit Cretonne output to native object files with Faerie" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.10.0" } -cretonne-module = { path = "../module", version = "0.10.0" } +cretonne-codegen = { path = "../codegen", version = "0.11.0" } +cretonne-module = { path = "../module", version = "0.11.0" } faerie = "0.4.2" goblin = "0.0.15" failure = "0.1.1" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 90c11b1129..ae1fc74359 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.10.0" +version = "0.11.0" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" publish = false [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.10.0" } -cretonne-reader = { path = "../reader", version = "0.10.0" } +cretonne-codegen = { path = "../codegen", version = "0.11.0" } +cretonne-reader = { path = "../reader", version = "0.11.0" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 389bb9064c..f369d6da93 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.10.0" +version = "0.11.0" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.10.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } [features] default = ["std"] diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 0afe1fd540..094bdf075c 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-module" -version = "0.10.0" +version = "0.11.0" authors = ["The Cretonne Project Developers"] description = "Support for linking functions and data with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.10.0", default-features = false } -cretonne-entity = { path = "../entity", version = "0.10.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.11.0", default-features = false } hashmap_core = { version = "0.1.6", optional = true } failure = "0.1.1" diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 35eff0c99f..69ec2f9b67 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.10.0" +version = "0.11.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -8,7 +8,7 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.10.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } target-lexicon = { version = "0.0.2", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 3132c063ff..1ad05ee299 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.10.0" +version = "0.11.0" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.10.0" } +cretonne-codegen = { path = "../codegen", version = "0.11.0" } target-lexicon = "0.0.2" [badges] diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index c66a66f5ec..2452ccae55 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-simplejit" -version = "0.10.0" +version = "0.11.0" authors = ["The Cretonne Project Developers"] description = "A simple JIT library backed by Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,9 +9,9 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.10.0", default-features = false } -cretonne-module = { path = "../module", version = "0.10.0", default-features = false } -cretonne-native = { path = "../native", version = "0.10.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } +cretonne-module = { path = "../module", version = "0.11.0", default-features = false } +cretonne-native = { path = "../native", version = "0.11.0", default-features = false } region = "0.3.0" libc = { version = "0.2.40", default-features = false } errno = "0.2.3" diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml index edb702ff9a..274ba20d07 100644 --- a/lib/umbrella/Cargo.toml +++ b/lib/umbrella/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.10.0" +version = "0.11.0" description = "Umbrella for commonly-used cretonne crates" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -10,8 +10,8 @@ readme = "README.md" keywords = ["compile", "compiler", "jit"] [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.10.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.10.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.11.0", default-features = false } [features] default = ["std"] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index c39f3460eb..1fa516edb0 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.10.0" +version = "0.11.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/cretonne/cretonne" @@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"] [dependencies] wasmparser = { version = "0.17.0", default-features = false } -cretonne-codegen = { path = "../codegen", version = "0.10.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.10.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.11.0", default-features = false } hashmap_core = { version = "0.1.6", optional = true } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } From 49cc693d6442c6efbe1c7390550ae6992e60a869 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 15 Jun 2018 10:35:00 -0700 Subject: [PATCH 1851/3084] Update to hashmap_core 0.1.7. --- lib/codegen/Cargo.toml | 2 +- lib/module/Cargo.toml | 2 +- lib/wasm/Cargo.toml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 7abab7e938..ca08a6c7e3 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -14,7 +14,7 @@ build = "build.rs" cretonne-entity = { path = "../entity", version = "0.11.0", default-features = false } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } -hashmap_core = { version = "0.1.6", optional = true } +hashmap_core = { version = "0.1.7", optional = true } target-lexicon = { version = "0.0.2", default-features = false } # It is a goal of the cretonne-codegen crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 094bdf075c..1e527bd3b9 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -11,7 +11,7 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } cretonne-entity = { path = "../entity", version = "0.11.0", default-features = false } -hashmap_core = { version = "0.1.6", optional = true } +hashmap_core = { version = "0.1.7", optional = true } failure = "0.1.1" [features] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 1fa516edb0..ccfc7ac0ba 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -12,7 +12,7 @@ keywords = ["webassembly", "wasm"] wasmparser = { version = "0.17.0", default-features = false } cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } cretonne-frontend = { path = "../frontend", version = "0.11.0", default-features = false } -hashmap_core = { version = "0.1.6", optional = true } +hashmap_core = { version = "0.1.7", optional = true } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } target-lexicon = { version = "0.0.2", default-features = false } From 5c320a0d30760da61e32ddeef2bb2222fb89d9af Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Thu, 14 Jun 2018 01:07:27 -0400 Subject: [PATCH 1852/3084] Change GlobalVar to GlobalValue --- cranelift/docs/compare-llvm.rst | 2 +- cranelift/docs/langref.rst | 32 +++--- cranelift/docs/metaref.rst | 4 +- .../filetests/isa/x86/legalize-memory.cton | 8 +- cranelift/filetests/parser/memory.cton | 20 ++-- lib/codegen/meta/base/entities.py | 4 +- lib/codegen/meta/base/formats.py | 4 +- lib/codegen/meta/base/instructions.py | 12 +-- lib/codegen/meta/base/legalize.py | 2 +- lib/codegen/meta/base/predicates.py | 4 +- lib/codegen/meta/base/settings.py | 2 +- lib/codegen/meta/cdsl/formats.py | 2 +- lib/codegen/meta/cdsl/registers.py | 2 +- lib/codegen/meta/isa/x86/recipes.py | 18 ++-- lib/codegen/src/ir/entities.rs | 22 ++-- lib/codegen/src/ir/extfunc.rs | 2 +- lib/codegen/src/ir/function.rs | 18 ++-- lib/codegen/src/ir/globalvalue.rs | 70 ++++++++++++ lib/codegen/src/ir/globalvar.rs | 28 ++--- lib/codegen/src/ir/heap.rs | 10 +- lib/codegen/src/ir/mod.rs | 6 +- .../{globalvar.rs => globalvalue.rs} | 45 ++++---- lib/codegen/src/legalizer/heap.rs | 8 +- lib/codegen/src/legalizer/mod.rs | 4 +- lib/codegen/src/predicates.rs | 6 +- lib/codegen/src/regalloc/solver.rs | 4 +- lib/codegen/src/verifier/mod.rs | 22 ++-- lib/codegen/src/write.rs | 4 +- lib/faerie/src/backend.rs | 2 +- lib/frontend/src/frontend.rs | 14 +-- lib/module/src/backend.rs | 2 +- lib/module/src/data_context.rs | 14 +-- lib/module/src/module.rs | 12 +-- lib/reader/src/lexer.rs | 4 +- lib/reader/src/parser.rs | 100 ++++++++++-------- lib/reader/src/sourcemap.rs | 12 +-- lib/simplejit/src/backend.rs | 2 +- lib/umbrella/src/lib.rs | 2 +- lib/wasm/src/code_translator.rs | 4 +- lib/wasm/src/environ/dummy.rs | 6 +- lib/wasm/src/environ/spec.rs | 14 +-- lib/wasm/src/sections_translator.rs | 2 +- lib/wasm/src/state.rs | 4 +- lib/wasm/src/translation_utils.rs | 2 +- 44 files changed, 324 insertions(+), 237 deletions(-) create mode 100644 lib/codegen/src/ir/globalvalue.rs rename lib/codegen/src/legalizer/{globalvar.rs => globalvalue.rs} (50%) diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst index abc47caaa3..4765cdc18b 100644 --- a/cranelift/docs/compare-llvm.rst +++ b/cranelift/docs/compare-llvm.rst @@ -110,7 +110,7 @@ Program structure In LLVM IR, the largest representable unit is the *module* which corresponds more or less to a C translation unit. It is a collection of functions and -global variables that may contain references to external symbols too. +global valueiables that may contain references to external symbols too. In Cretonne IR, the largest representable unit is the *function*. This is so that functions can easily be compiled in parallel without worrying about diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index e6d944dcd8..707927f454 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -554,7 +554,7 @@ stack overflow checks in the prologue. the stack pointer has reached or exceeded the limit, generate a trap with a ``stk_ovf`` code. - The global variable must be accessible and naturally aligned for a + The global valueiable must be accessible and naturally aligned for a pointer-sized value. Setting `stack_limit` is an alternative way to detect stack overflow, when using @@ -563,12 +563,12 @@ stack overflow checks in the prologue. Global variables ---------------- -A *global variable* is an :term:`accessible` object in memory whose address is +A *global valueiable* is an :term:`accessible` object in memory whose address is not known at compile time. The address is computed at runtime by -:inst:`global_addr`, possibly using information provided by the linker via -relocations. There are multiple kinds of global variables using different +:inst:`global_value`, possibly using information provided by the linker via +relocations. There are multiple kinds of global valueiables using different methods for determining their address. Cretonne does not track the type or even -the size of global variables, they are just pointers to non-stack memory. +the size of global valueiables, they are just pointers to non-stack memory. When Cretonne is generating code for a virtual machine environment, globals can be used to access data structures in the VM's runtime. This requires functions @@ -578,26 +578,26 @@ Cretonne functions. .. inst:: GV = vmctx+Offset - Declare a global variable in the VM context struct. + Declare a global valueiable in the VM context struct. - This declares a global variable whose address is a constant offset from the + This declares a global valueiable whose address is a constant offset from the VM context pointer which is passed as a hidden argument to all functions JIT-compiled for the VM. - Typically, the VM context is a C struct, and the declared global variable + Typically, the VM context is a C struct, and the declared global valueiable is a member of the struct. :arg Offset: Byte offset from the VM context pointer to the global variable. :result GV: Global variable. -The address of a global variable can also be derived by treating another global +The address of a global valueiable can also be derived by treating another global variable as a struct pointer. This makes it possible to chase pointers into VM runtime data structures. .. inst:: GV = deref(BaseGV)+Offset - Declare a global variable in a struct pointed to by BaseGV. + Declare a global valueiable in a struct pointed to by BaseGV. The address of GV can be computed by first loading a pointer from BaseGV and adding Offset to it. @@ -605,7 +605,7 @@ runtime data structures. It is assumed the BaseGV resides in readable memory with the appropriate alignment for storing a pointer. - Chains of ``deref`` global variables are possible, but cycles are not + Chains of ``deref`` global valueiables are possible, but cycles are not allowed. They will be caught by the IR verifier. :arg BaseGV: Global variable containing the base pointer. @@ -615,7 +615,7 @@ runtime data structures. .. inst:: GV = [colocated] globalsym name - Declare a global variable at a symbolic address. + Declare a global valueiable at a symbolic address. The address of GV is symbolic and will be assigned a relocation, so that it can be resolved by a later linking phase. @@ -627,7 +627,7 @@ runtime data structures. :arg name: External name. :result GV: Global variable. -.. autoinst:: global_addr +.. autoinst:: global_value .. autoinst:: globalsym_addr @@ -701,7 +701,7 @@ trap when accessed. address space reserved for the heap, not including the guard pages. :arg GuardBytes: Size of the guard pages in bytes. -When the base is a global variable, it must be :term:`accessible` and naturally +When the base is a global valueiable, it must be :term:`accessible` and naturally aligned for a pointer value. The ``reserved_reg`` option is not yet implemented. @@ -711,7 +711,7 @@ Dynamic heaps A *dynamic heap* can be relocated to a different base address when it is resized, and its bound can move dynamically. The guard pages move when the heap -is resized. The bound of a dynamic heap is stored in a global variable. +is resized. The bound of a dynamic heap is stored in a global valueiable. .. inst:: H = dynamic Base, min MinBytes, bound BoundGV, guard GuardBytes @@ -724,7 +724,7 @@ is resized. The bound of a dynamic heap is stored in a global variable. :arg BoundGV: Global variable containing the current heap bound in bytes. :arg GuardBytes: Size of the guard pages in bytes. -When the base is a global variable, it must be :term:`accessible` and naturally +When the base is a global valueiable, it must be :term:`accessible` and naturally aligned for a pointer value. The ``reserved_reg`` option is not yet implemented. diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 134628a8b1..70cc5ecc98 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -15,7 +15,7 @@ The meta language descriptions are Python modules under the steps: 1. The Python modules are imported. This has the effect of building static data - structures in global variables in the modules. These static data structures + structures in global valueiables in the modules. These static data structures in the :mod:`base` and :mod:`isa` packages use the classes in the :mod:`cdsl` package to describe instruction sets and other properties. @@ -81,7 +81,7 @@ open :class:`InstructionGroup`. :members: The basic Cretonne instruction set described in :doc:`langref` is defined by the -Python module :mod:`base.instructions`. This module has a global variable +Python module :mod:`base.instructions`. This module has a global valueiable :data:`base.instructions.GROUP` which is an :class:`InstructionGroup` instance containing all the base instructions. diff --git a/cranelift/filetests/isa/x86/legalize-memory.cton b/cranelift/filetests/isa/x86/legalize-memory.cton index 9762aa1f6c..0a35c841ba 100644 --- a/cranelift/filetests/isa/x86/legalize-memory.cton +++ b/cranelift/filetests/isa/x86/legalize-memory.cton @@ -9,7 +9,7 @@ function %vmctx(i64 vmctx) -> i64 { gv1 = vmctx-16 ebb1(v1: i64): - v2 = global_addr.i64 gv1 + v2 = global_value.i64 gv1 ; check: v2 = iadd_imm v1, -16 return v2 ; check: return v2 @@ -20,7 +20,7 @@ function %deref(i64 vmctx) -> i64 { gv2 = deref(gv1)+32 ebb1(v1: i64): - v2 = global_addr.i64 gv2 + v2 = global_value.i64 gv2 ; check: $(a1=$V) = iadd_imm v1, -16 ; check: $(p1=$V) = load.i64 notrap aligned $a1 ; check: v2 = iadd_imm $p1, 32 @@ -33,9 +33,9 @@ function %sym() -> i64 { gv1 = globalsym u123:456 ebb1: - v0 = global_addr.i64 gv0 + v0 = global_value.i64 gv0 ; check: v0 = globalsym_addr.i64 gv0 - v1 = global_addr.i64 gv1 + v1 = global_value.i64 gv1 ; check: v1 = globalsym_addr.i64 gv1 v2 = bxor v0, v1 return v2 diff --git a/cranelift/filetests/parser/memory.cton b/cranelift/filetests/parser/memory.cton index 1b36071243..8e01eb8cae 100644 --- a/cranelift/filetests/parser/memory.cton +++ b/cranelift/filetests/parser/memory.cton @@ -10,8 +10,8 @@ function %vmglobal() -> i32 { gv5 = vmctx -256 ; check: gv5 = vmctx-256 ebb0: - v1 = global_addr.i32 gv3 - ; check: v1 = global_addr.i32 gv3 + v1 = global_value.i32 gv3 + ; check: v1 = global_value.i32 gv3 return v1 } @@ -20,19 +20,19 @@ function %deref() -> i32 { gv4 = deref(gv3)-32 ; check: gv4 = deref(gv3)-32 ebb0: - v1 = global_addr.i32 gv4 - ; check: v1 = global_addr.i32 gv4 + v1 = global_value.i32 gv4 + ; check: v1 = global_value.i32 gv4 return v1 } -; Refer to a global variable before it's been declared. +; Refer to a global valueiable before it's been declared. function %backref() -> i32 { gv1 = deref(gv2)-32 ; check: gv1 = deref(gv2)-32 gv2 = vmctx+16 ; check: gv2 = vmctx+16 ebb0: - v1 = global_addr.i32 gv1 + v1 = global_value.i32 gv1 return v1 } @@ -42,10 +42,10 @@ function %sym() -> i32 { gv1 = globalsym u8:9 ; check: gv1 = globalsym u8:9 ebb0: - v0 = global_addr.i32 gv0 - ; check: v0 = global_addr.i32 gv0 - v1 = global_addr.i32 gv1 - ; check: v1 = global_addr.i32 gv1 + v0 = global_value.i32 gv0 + ; check: v0 = global_value.i32 gv0 + v1 = global_value.i32 gv1 + ; check: v1 = global_value.i32 gv1 v2 = bxor v0, v1 return v2 } diff --git a/lib/codegen/meta/base/entities.py b/lib/codegen/meta/base/entities.py index 996913471b..e8915a0f79 100644 --- a/lib/codegen/meta/base/entities.py +++ b/lib/codegen/meta/base/entities.py @@ -16,8 +16,8 @@ ebb = EntityRefKind( #: A reference to a stack slot declared in the function preamble. stack_slot = EntityRefKind('stack_slot', 'A stack slot.') -#: A reference to a global variable. -global_var = EntityRefKind('global_var', 'A global variable.') +#: A reference to a global valueiable. +global_value = EntityRefKind('global_value', 'A global value.') #: A reference to a function sugnature declared in the function preamble. #: This is used to provide the call signature in a call_indirect instruction. diff --git a/lib/codegen/meta/base/formats.py b/lib/codegen/meta/base/formats.py index 817ac0a87d..1c0d7284f2 100644 --- a/lib/codegen/meta/base/formats.py +++ b/lib/codegen/meta/base/formats.py @@ -18,7 +18,7 @@ UnaryImm = InstructionFormat(imm64) UnaryIeee32 = InstructionFormat(ieee32) UnaryIeee64 = InstructionFormat(ieee64) UnaryBool = InstructionFormat(boolean) -UnaryGlobalVar = InstructionFormat(entities.global_var) +UnaryGlobalValue = InstructionFormat(entities.global_value) Binary = InstructionFormat(VALUE, VALUE) BinaryImm = InstructionFormat(VALUE, imm64) @@ -79,5 +79,5 @@ CondTrap = InstructionFormat(VALUE, trapcode) IntCondTrap = InstructionFormat(intcc, VALUE, trapcode) FloatCondTrap = InstructionFormat(floatcc, VALUE, trapcode) -# Finally extract the names of global variables in this module. +# Finally extract the names of global valueiables in this module. InstructionFormat.extract_names(globals()) diff --git a/lib/codegen/meta/base/instructions.py b/lib/codegen/meta/base/instructions.py index 485de44c5e..5efd344c6f 100644 --- a/lib/codegen/meta/base/instructions.py +++ b/lib/codegen/meta/base/instructions.py @@ -491,19 +491,19 @@ stack_addr = Instruction( # Global variables. # -GV = Operand('GV', entities.global_var) +GV = Operand('GV', entities.global_value) -global_addr = Instruction( - 'global_addr', r""" - Compute the address of global variable GV. +global_value = Instruction( + 'global_value', r""" + Compute the value of global GV. """, ins=GV, outs=addr) -# A specialized form of global_addr instructions that only handles +# A specialized form of global_value instructions that only handles # symbolic names. globalsym_addr = Instruction( 'globalsym_addr', r""" - Compute the address of global variable GV, which is a symbolic name. + Compute the address of global GV, which is a symbolic name. """, ins=GV, outs=addr) diff --git a/lib/codegen/meta/base/legalize.py b/lib/codegen/meta/base/legalize.py index ad396af747..de196cc645 100644 --- a/lib/codegen/meta/base/legalize.py +++ b/lib/codegen/meta/base/legalize.py @@ -62,7 +62,7 @@ expand_flags = XFormGroup('expand_flags', """ # Custom expansions for memory objects. -expand.custom_legalize(insts.global_addr, 'expand_global_addr') +expand.custom_legalize(insts.global_value, 'expand_global_value') expand.custom_legalize(insts.heap_addr, 'expand_heap_addr') # Custom expansions for calls. diff --git a/lib/codegen/meta/base/predicates.py b/lib/codegen/meta/base/predicates.py index 44f135e0d3..16ea856138 100644 --- a/lib/codegen/meta/base/predicates.py +++ b/lib/codegen/meta/base/predicates.py @@ -2,7 +2,7 @@ Cretonne predicates that consider `Function` fields. """ from cdsl.predicates import FieldPredicate -from .formats import UnaryGlobalVar, InstructionFormat +from .formats import UnaryGlobalValue, InstructionFormat try: from typing import TYPE_CHECKING @@ -32,7 +32,7 @@ class IsColocatedData(FieldPredicate): def __init__(self): # type: () -> None super(IsColocatedData, self).__init__( - UnaryGlobalVar.global_var, 'is_colocated_data', ('func',)) + UnaryGlobalValue.global_value, 'is_colocated_data', ('func',)) class LengthEquals(FieldPredicate): diff --git a/lib/codegen/meta/base/settings.py b/lib/codegen/meta/base/settings.py index 8e88a45abc..c3f9ae182b 100644 --- a/lib/codegen/meta/base/settings.py +++ b/lib/codegen/meta/base/settings.py @@ -53,7 +53,7 @@ call_conv = EnumSetting( # Note that Cretonne doesn't currently need an is_pie flag, because PIE is just # PIC where symbols can't be pre-empted, which can be expressed with the -# `colocated` flag on external functions and global variables. +# `colocated` flag on external functions and global valueiables. is_pic = BoolSetting("Enable Position-Independent Code generation") colocated_libcalls = BoolSetting( diff --git a/lib/codegen/meta/cdsl/formats.py b/lib/codegen/meta/cdsl/formats.py index c8dd58fc7f..ebee96281e 100644 --- a/lib/codegen/meta/cdsl/formats.py +++ b/lib/codegen/meta/cdsl/formats.py @@ -208,7 +208,7 @@ class InstructionFormat(object): """ Given a dict mapping name -> object as returned by `globals()`, find all the InstructionFormat objects and set their name from the dict key. - This is used to name a bunch of global variables in a module. + This is used to name a bunch of global valueiables in a module. """ for name, obj in globs.items(): if isinstance(obj, InstructionFormat): diff --git a/lib/codegen/meta/cdsl/registers.py b/lib/codegen/meta/cdsl/registers.py index 6f57e380d9..4dccbc808a 100644 --- a/lib/codegen/meta/cdsl/registers.py +++ b/lib/codegen/meta/cdsl/registers.py @@ -365,7 +365,7 @@ class RegClass(object): """ Given a dict mapping name -> object as returned by `globals()`, find all the RegClass objects and set their name from the dict key. - This is used to name a bunch of global variables in a module. + This is used to name a bunch of global valueiables in a module. """ for name, obj in globs.items(): if isinstance(obj, RegClass): diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py index 1caf6136d1..2d6ac898a4 100644 --- a/lib/codegen/meta/isa/x86/recipes.py +++ b/lib/codegen/meta/isa/x86/recipes.py @@ -14,7 +14,7 @@ from base.formats import IntCompare, IntCompareImm, FloatCompare from base.formats import IntCond, FloatCond from base.formats import IntSelect, IntCondTrap, FloatCondTrap from base.formats import Jump, Branch, BranchInt, BranchFloat -from base.formats import Ternary, FuncAddr, UnaryGlobalVar +from base.formats import Ternary, FuncAddr, UnaryGlobalValue from base.formats import RegMove, RegSpill, RegFill, CopySpecial from base.formats import LoadComplex, StoreComplex from .registers import GPR, ABCD, FPR, GPR_DEREF_SAFE, GPR_ZERO_DEREF_SAFE @@ -703,50 +703,50 @@ got_fnaddr8 = TailRecipe( # XX+rd id with Abs4 globalsym relocation. gvaddr4 = TailRecipe( - 'gvaddr4', UnaryGlobalVar, size=4, ins=(), outs=GPR, + 'gvaddr4', UnaryGlobalValue, size=4, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); sink.reloc_external(Reloc::Abs4, - &func.global_vars[global_var].symbol_name(), + &func.global_values[global_value].symbol_name(), 0); sink.put4(0); ''') # XX+rd iq with Abs8 globalsym relocation. gvaddr8 = TailRecipe( - 'gvaddr8', UnaryGlobalVar, size=8, ins=(), outs=GPR, + 'gvaddr8', UnaryGlobalValue, size=8, ins=(), outs=GPR, emit=''' PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); sink.reloc_external(Reloc::Abs8, - &func.global_vars[global_var].symbol_name(), + &func.global_values[global_value].symbol_name(), 0); sink.put8(0); ''') # XX+rd iq with PCRel4 globalsym relocation. pcrel_gvaddr8 = TailRecipe( - 'pcrel_gvaddr8', UnaryGlobalVar, size=5, ins=(), outs=GPR, + 'pcrel_gvaddr8', UnaryGlobalValue, size=5, ins=(), outs=GPR, emit=''' PUT_OP(bits, rex2(0, out_reg0), sink); modrm_rm(5, out_reg0, sink); // The addend adjusts for the difference between the end of the // instruction and the beginning of the immediate field. sink.reloc_external(Reloc::X86PCRel4, - &func.global_vars[global_var].symbol_name(), + &func.global_values[global_value].symbol_name(), -4); sink.put4(0); ''') # XX+rd iq with Abs8 globalsym relocation. got_gvaddr8 = TailRecipe( - 'got_gvaddr8', UnaryGlobalVar, size=5, ins=(), outs=GPR, + 'got_gvaddr8', UnaryGlobalValue, size=5, ins=(), outs=GPR, emit=''' PUT_OP(bits, rex2(0, out_reg0), sink); modrm_rm(5, out_reg0, sink); // The addend adjusts for the difference between the end of the // instruction and the beginning of the immediate field. sink.reloc_external(Reloc::X86GOTPCRel4, - &func.global_vars[global_var].symbol_name(), + &func.global_values[global_value].symbol_name(), -4); sink.put4(0); ''') diff --git a/lib/codegen/src/ir/entities.rs b/lib/codegen/src/ir/entities.rs index 73e0984245..d6461d47df 100644 --- a/lib/codegen/src/ir/entities.rs +++ b/lib/codegen/src/ir/entities.rs @@ -82,18 +82,18 @@ impl StackSlot { } } -/// An opaque reference to a global variable. +/// An opaque reference to a global valueiable. #[derive(Copy, Clone, PartialEq, Eq, Hash)] -pub struct GlobalVar(u32); -entity_impl!(GlobalVar, "gv"); +pub struct GlobalValue(u32); +entity_impl!(GlobalValue, "gv"); -impl GlobalVar { - /// Create a new global variable reference from its number. +impl GlobalValue { + /// Create a new global valueiable reference from its number. /// /// This method is for use by the parser. pub fn with_number(n: u32) -> Option { if n < u32::MAX { - Some(GlobalVar(n)) + Some(GlobalValue(n)) } else { None } @@ -186,7 +186,7 @@ pub enum AnyEntity { /// A stack slot. StackSlot(StackSlot), /// A Global variable. - GlobalVar(GlobalVar), + GlobalValue(GlobalValue), /// A jump table. JumpTable(JumpTable), /// An external function. @@ -205,7 +205,7 @@ impl fmt::Display for AnyEntity { AnyEntity::Inst(r) => r.fmt(f), AnyEntity::Value(r) => r.fmt(f), AnyEntity::StackSlot(r) => r.fmt(f), - AnyEntity::GlobalVar(r) => r.fmt(f), + AnyEntity::GlobalValue(r) => r.fmt(f), AnyEntity::JumpTable(r) => r.fmt(f), AnyEntity::FuncRef(r) => r.fmt(f), AnyEntity::SigRef(r) => r.fmt(f), @@ -244,9 +244,9 @@ impl From for AnyEntity { } } -impl From for AnyEntity { - fn from(r: GlobalVar) -> Self { - AnyEntity::GlobalVar(r) +impl From for AnyEntity { + fn from(r: GlobalValue) -> Self { + AnyEntity::GlobalValue(r) } } diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs index a4abc8bfc4..98b6e4fb40 100644 --- a/lib/codegen/src/ir/extfunc.rs +++ b/lib/codegen/src/ir/extfunc.rs @@ -284,7 +284,7 @@ pub enum ArgumentPurpose { /// A VM context pointer. /// /// This is a pointer to a context struct containing details about the current sandbox. It is - /// used as a base pointer for `vmctx` global variables. + /// used as a base pointer for `vmctx` global valueiables. VMContext, /// A signature identifier. diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs index f7773f2258..5d59e6f3ce 100644 --- a/lib/codegen/src/ir/function.rs +++ b/lib/codegen/src/ir/function.rs @@ -7,7 +7,7 @@ use binemit::CodeOffset; use entity::{EntityMap, PrimaryMap}; use ir; use ir::{DataFlowGraph, ExternalName, Layout, Signature}; -use ir::{Ebb, ExtFuncData, FuncRef, GlobalVar, GlobalVarData, Heap, HeapData, JumpTable, +use ir::{Ebb, ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, JumpTable, JumpTableData, SigRef, StackSlot, StackSlotData}; use ir::{EbbOffsets, InstEncodings, JumpTables, SourceLocs, StackSlots, ValueLocations}; use isa::{EncInfo, Encoding, Legalize, TargetIsa}; @@ -32,10 +32,10 @@ pub struct Function { /// If not `None`, represents the address that the stack pointer should /// be checked against. - pub stack_limit: Option, + pub stack_limit: Option, /// Global variables referenced. - pub global_vars: PrimaryMap, + pub global_values: PrimaryMap, /// Heaps referenced. pub heaps: PrimaryMap, @@ -78,7 +78,7 @@ impl Function { signature: sig, stack_slots: StackSlots::new(), stack_limit: None, - global_vars: PrimaryMap::new(), + global_values: PrimaryMap::new(), heaps: PrimaryMap::new(), jump_tables: PrimaryMap::new(), dfg: DataFlowGraph::new(), @@ -94,7 +94,7 @@ impl Function { pub fn clear(&mut self) { self.signature.clear(CallConv::Fast); self.stack_slots.clear(); - self.global_vars.clear(); + self.global_values.clear(); self.heaps.clear(); self.jump_tables.clear(); self.dfg.clear(); @@ -129,7 +129,7 @@ impl Function { /// Sets the stack limit for the function. /// /// Returns previous one if any. - pub fn set_stack_limit(&mut self, stack_limit: Option) -> Option { + pub fn set_stack_limit(&mut self, stack_limit: Option) -> Option { let prev = self.stack_limit.take(); self.stack_limit = stack_limit; prev @@ -145,9 +145,9 @@ impl Function { self.dfg.ext_funcs.push(data) } - /// Declares a global variable accessible to the function. - pub fn create_global_var(&mut self, data: GlobalVarData) -> GlobalVar { - self.global_vars.push(data) + /// Declares a global valueiable accessible to the function. + pub fn create_global_value(&mut self, data: GlobalValueData) -> GlobalValue { + self.global_values.push(data) } /// Declares a heap accessible to the function. diff --git a/lib/codegen/src/ir/globalvalue.rs b/lib/codegen/src/ir/globalvalue.rs new file mode 100644 index 0000000000..5d068a7583 --- /dev/null +++ b/lib/codegen/src/ir/globalvalue.rs @@ -0,0 +1,70 @@ +//! Global variables. + +use ir::immediates::Offset32; +use ir::{ExternalName, GlobalValue}; +use std::fmt; + +/// Information about a global valueiable declaration. +#[derive(Clone)] +pub enum GlobalValueData { + /// Variable is part of the VM context struct, it's address is a constant offset from the VM + /// context pointer. + VMContext { + /// Offset from the `vmctx` pointer to this global. + offset: Offset32, + }, + + /// Variable is part of a struct pointed to by another global valueiable. + /// + /// The `base` global valueiable is assumed to contain a pointer to a struct. This global + /// variable lives at an offset into the struct. The memory must be accessible, and + /// naturally aligned to hold a pointer value. + Deref { + /// The base pointer global valueiable. + base: GlobalValue, + + /// Byte offset to be added to the pointer loaded from `base`. + offset: Offset32, + }, + + /// Variable is at an address identified by a symbolic name. Cretonne itself + /// does not interpret this name; it's used by embedders to link with other + /// data structures. + Sym { + /// The symbolic name. + name: ExternalName, + + /// Will this variable be defined nearby, such that it will always be a certain distance + /// away, after linking? If so, references to it can avoid going through a GOT. Note that + /// symbols meant to be preemptible cannot be colocated. + colocated: bool, + }, +} + +impl GlobalValueData { + /// Assume that `self` is an `GlobalValueData::Sym` and return its name. + pub fn symbol_name(&self) -> &ExternalName { + match *self { + GlobalValueData::Sym { ref name, .. } => name, + _ => panic!("only symbols have names"), + } + } +} + +impl fmt::Display for GlobalValueData { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match *self { + GlobalValueData::VMContext { offset } => write!(f, "vmctx{}", offset), + GlobalValueData::Deref { base, offset } => write!(f, "deref({}){}", base, offset), + GlobalValueData::Sym { + ref name, + colocated, + } => { + if colocated { + write!(f, "colocated ")?; + } + write!(f, "globalsym {}", name) + } + } + } +} diff --git a/lib/codegen/src/ir/globalvar.rs b/lib/codegen/src/ir/globalvar.rs index 399f3b7ae4..5d068a7583 100644 --- a/lib/codegen/src/ir/globalvar.rs +++ b/lib/codegen/src/ir/globalvar.rs @@ -1,12 +1,12 @@ //! Global variables. use ir::immediates::Offset32; -use ir::{ExternalName, GlobalVar}; +use ir::{ExternalName, GlobalValue}; use std::fmt; -/// Information about a global variable declaration. +/// Information about a global valueiable declaration. #[derive(Clone)] -pub enum GlobalVarData { +pub enum GlobalValueData { /// Variable is part of the VM context struct, it's address is a constant offset from the VM /// context pointer. VMContext { @@ -14,14 +14,14 @@ pub enum GlobalVarData { offset: Offset32, }, - /// Variable is part of a struct pointed to by another global variable. + /// Variable is part of a struct pointed to by another global valueiable. /// - /// The `base` global variable is assumed to contain a pointer to a struct. This global + /// The `base` global valueiable is assumed to contain a pointer to a struct. This global /// variable lives at an offset into the struct. The memory must be accessible, and /// naturally aligned to hold a pointer value. Deref { - /// The base pointer global variable. - base: GlobalVar, + /// The base pointer global valueiable. + base: GlobalValue, /// Byte offset to be added to the pointer loaded from `base`. offset: Offset32, @@ -41,22 +41,22 @@ pub enum GlobalVarData { }, } -impl GlobalVarData { - /// Assume that `self` is an `GlobalVarData::Sym` and return its name. +impl GlobalValueData { + /// Assume that `self` is an `GlobalValueData::Sym` and return its name. pub fn symbol_name(&self) -> &ExternalName { match *self { - GlobalVarData::Sym { ref name, .. } => name, + GlobalValueData::Sym { ref name, .. } => name, _ => panic!("only symbols have names"), } } } -impl fmt::Display for GlobalVarData { +impl fmt::Display for GlobalValueData { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match *self { - GlobalVarData::VMContext { offset } => write!(f, "vmctx{}", offset), - GlobalVarData::Deref { base, offset } => write!(f, "deref({}){}", base, offset), - GlobalVarData::Sym { + GlobalValueData::VMContext { offset } => write!(f, "vmctx{}", offset), + GlobalValueData::Deref { base, offset } => write!(f, "deref({}){}", base, offset), + GlobalValueData::Sym { ref name, colocated, } => { diff --git a/lib/codegen/src/ir/heap.rs b/lib/codegen/src/ir/heap.rs index b41565b25b..812ffa01d3 100644 --- a/lib/codegen/src/ir/heap.rs +++ b/lib/codegen/src/ir/heap.rs @@ -1,7 +1,7 @@ //! Heaps. use ir::immediates::Imm64; -use ir::GlobalVar; +use ir::GlobalValue; use std::fmt; /// Information about a heap declaration. @@ -29,9 +29,9 @@ pub enum HeapBase { /// This feature is not yet implemented. ReservedReg, - /// The heap base is in a global variable. The variable must be accessible and naturally + /// The heap base is in a global valueiable. The variable must be accessible and naturally /// aligned for a pointer. - GlobalVar(GlobalVar), + GlobalValue(GlobalValue), } /// Style of heap including style-specific information. @@ -41,7 +41,7 @@ pub enum HeapStyle { Dynamic { /// Global variable holding the current bound of the heap in bytes. It is /// required to be accessible and naturally aligned for a pointer-sized integer. - bound_gv: GlobalVar, + bound_gv: GlobalValue, }, /// A static heap has a fixed base address and a number of not-yet-allocated pages before the @@ -61,7 +61,7 @@ impl fmt::Display for HeapData { match self.base { HeapBase::ReservedReg => write!(f, " reserved_reg")?, - HeapBase::GlobalVar(gv) => write!(f, " {}", gv)?, + HeapBase::GlobalValue(gv) => write!(f, " {}", gv)?, } write!(f, ", min {}", self.min_size)?; diff --git a/lib/codegen/src/ir/mod.rs b/lib/codegen/src/ir/mod.rs index daca83cc3a..71e56ec5a0 100644 --- a/lib/codegen/src/ir/mod.rs +++ b/lib/codegen/src/ir/mod.rs @@ -7,7 +7,7 @@ pub mod entities; mod extfunc; mod extname; pub mod function; -mod globalvar; +mod globalvalue; mod heap; pub mod immediates; pub mod instructions; @@ -24,11 +24,11 @@ mod valueloc; pub use ir::builder::{InsertBuilder, InstBuilder, InstBuilderBase, InstInserterBase}; pub use ir::dfg::{DataFlowGraph, ValueDef}; -pub use ir::entities::{Ebb, FuncRef, GlobalVar, Heap, Inst, JumpTable, SigRef, StackSlot, Value}; +pub use ir::entities::{Ebb, FuncRef, GlobalValue, Heap, Inst, JumpTable, SigRef, StackSlot, Value}; pub use ir::extfunc::{AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData, Signature}; pub use ir::extname::ExternalName; pub use ir::function::Function; -pub use ir::globalvar::GlobalVarData; +pub use ir::globalvalue::GlobalValueData; pub use ir::heap::{HeapBase, HeapData, HeapStyle}; pub use ir::instructions::{InstructionData, Opcode, ValueList, ValueListPool, VariableArgs}; pub use ir::jumptable::JumpTableData; diff --git a/lib/codegen/src/legalizer/globalvar.rs b/lib/codegen/src/legalizer/globalvalue.rs similarity index 50% rename from lib/codegen/src/legalizer/globalvar.rs rename to lib/codegen/src/legalizer/globalvalue.rs index d8fd0b94b7..fbea36b9e6 100644 --- a/lib/codegen/src/legalizer/globalvar.rs +++ b/lib/codegen/src/legalizer/globalvalue.rs @@ -1,15 +1,15 @@ -//! Legalization of global variables. +//! Legalization of global valueiables. //! -//! This module exports the `expand_global_addr` function which transforms a `global_addr` -//! instruction into code that depends on the kind of global variable referenced. +//! This module exports the `expand_global_value` function which transforms a `global_value` +//! instruction into code that depends on the kind of global valueiable referenced. use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::{self, InstBuilder}; use isa::TargetIsa; -/// Expand a `global_addr` instruction according to the definition of the global variable. -pub fn expand_global_addr( +/// Expand a `global_value` instruction according to the definition of the global valueiable. +pub fn expand_global_value( inst: ir::Inst, func: &mut ir::Function, _cfg: &mut ControlFlowGraph, @@ -17,40 +17,43 @@ pub fn expand_global_addr( ) { // Unpack the instruction. let gv = match func.dfg[inst] { - ir::InstructionData::UnaryGlobalVar { opcode, global_var } => { - debug_assert_eq!(opcode, ir::Opcode::GlobalAddr); - global_var + ir::InstructionData::UnaryGlobalValue { + opcode, + global_value, + } => { + debug_assert_eq!(opcode, ir::Opcode::GlobalValue); + global_value } - _ => panic!("Wanted global_addr: {}", func.dfg.display_inst(inst, None)), + _ => panic!("Wanted global_value: {}", func.dfg.display_inst(inst, None)), }; - match func.global_vars[gv] { - ir::GlobalVarData::VMContext { offset } => vmctx_addr(inst, func, offset.into()), - ir::GlobalVarData::Deref { base, offset } => deref_addr(inst, func, base, offset.into()), - ir::GlobalVarData::Sym { .. } => globalsym(inst, func, gv), + match func.global_values[gv] { + ir::GlobalValueData::VMContext { offset } => vmctx_addr(inst, func, offset.into()), + ir::GlobalValueData::Deref { base, offset } => deref_addr(inst, func, base, offset.into()), + ir::GlobalValueData::Sym { .. } => globalsym(inst, func, gv), } } -/// Expand a `global_addr` instruction for a vmctx global. +/// Expand a `global_value` instruction for a vmctx global. fn vmctx_addr(inst: ir::Inst, func: &mut ir::Function, offset: i64) { // Get the value representing the `vmctx` argument. let vmctx = func.special_param(ir::ArgumentPurpose::VMContext) .expect("Missing vmctx parameter"); - // Simply replace the `global_addr` instruction with an `iadd_imm`, reusing the result value. + // Simply replace the `global_value` instruction with an `iadd_imm`, reusing the result value. func.dfg.replace(inst).iadd_imm(vmctx, offset); } -/// Expand a `global_addr` instruction for a deref global. -fn deref_addr(inst: ir::Inst, func: &mut ir::Function, base: ir::GlobalVar, offset: i64) { - // We need to load a pointer from the `base` global variable, so insert a new `global_addr` +/// Expand a `global_value` instruction for a deref global. +fn deref_addr(inst: ir::Inst, func: &mut ir::Function, base: ir::GlobalValue, offset: i64) { + // We need to load a pointer from the `base` global valueiable, so insert a new `global_value` // instruction. This depends on the iterative legalization loop. Note that the IR verifier // detects any cycles in the `deref` globals. let ptr_ty = func.dfg.value_type(func.dfg.first_result(inst)); let mut pos = FuncCursor::new(func).at_inst(inst); pos.use_srcloc(inst); - let base_addr = pos.ins().global_addr(ptr_ty, base); + let base_addr = pos.ins().global_value(ptr_ty, base); let mut mflags = ir::MemFlags::new(); // Deref globals are required to be accessible and aligned. mflags.set_notrap(); @@ -59,8 +62,8 @@ fn deref_addr(inst: ir::Inst, func: &mut ir::Function, base: ir::GlobalVar, offs pos.func.dfg.replace(inst).iadd_imm(base_ptr, offset); } -/// Expand a `global_addr` instruction for a symbolic name global. -fn globalsym(inst: ir::Inst, func: &mut ir::Function, gv: ir::GlobalVar) { +/// Expand a `global_value` instruction for a symbolic name global. +fn globalsym(inst: ir::Inst, func: &mut ir::Function, gv: ir::GlobalValue) { let ptr_ty = func.dfg.value_type(func.dfg.first_result(inst)); func.dfg.replace(inst).globalsym_addr(ptr_ty, gv); } diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs index 3b0b5dfd1c..ab68976d57 100644 --- a/lib/codegen/src/legalizer/heap.rs +++ b/lib/codegen/src/legalizer/heap.rs @@ -46,7 +46,7 @@ fn dynamic_addr( heap: ir::Heap, offset: ir::Value, size: u32, - bound_gv: ir::GlobalVar, + bound_gv: ir::GlobalValue, func: &mut ir::Function, ) { let size = i64::from(size); @@ -57,7 +57,7 @@ fn dynamic_addr( pos.use_srcloc(inst); // Start with the bounds check. Trap if `offset + size > bound`. - let bound_addr = pos.ins().global_addr(addr_ty, bound_gv); + let bound_addr = pos.ins().global_value(addr_ty, bound_gv); let mut mflags = MemFlags::new(); // The bound variable is requied to be accessible and aligned. mflags.set_notrap(); @@ -162,8 +162,8 @@ fn offset_addr( // Add the heap base address base match pos.func.heaps[heap].base { ir::HeapBase::ReservedReg => unimplemented!(), - ir::HeapBase::GlobalVar(base_gv) => { - let base_addr = pos.ins().global_addr(addr_ty, base_gv); + ir::HeapBase::GlobalValue(base_gv) => { + let base_addr = pos.ins().global_value(addr_ty, base_gv); let mut mflags = MemFlags::new(); // The base address variable is requied to be accessible and aligned. mflags.set_notrap(); diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs index 9dcc2dff2f..f099ca7447 100644 --- a/lib/codegen/src/legalizer/mod.rs +++ b/lib/codegen/src/legalizer/mod.rs @@ -22,13 +22,13 @@ use timing; mod boundary; mod call; -mod globalvar; +mod globalvalue; mod heap; mod libcall; mod split; use self::call::expand_call; -use self::globalvar::expand_global_addr; +use self::globalvalue::expand_global_value; use self::heap::expand_heap_addr; use self::libcall::expand_as_libcall; diff --git a/lib/codegen/src/predicates.rs b/lib/codegen/src/predicates.rs index 74b6e1d8cd..96cf0f38c6 100644 --- a/lib/codegen/src/predicates.rs +++ b/lib/codegen/src/predicates.rs @@ -53,9 +53,9 @@ pub fn is_colocated_func(func_ref: ir::FuncRef, func: &ir::Function) -> bool { } #[allow(dead_code)] -pub fn is_colocated_data(global_var: ir::GlobalVar, func: &ir::Function) -> bool { - match func.global_vars[global_var] { - ir::GlobalVarData::Sym { colocated, .. } => colocated, +pub fn is_colocated_data(global_value: ir::GlobalValue, func: &ir::Function) -> bool { + match func.global_values[global_value] { + ir::GlobalValueData::Sym { colocated, .. } => colocated, _ => panic!("is_colocated_data only makes sense for data with symbolic addresses"), } } diff --git a/lib/codegen/src/regalloc/solver.rs b/lib/codegen/src/regalloc/solver.rs index d5d000fa95..779396cc57 100644 --- a/lib/codegen/src/regalloc/solver.rs +++ b/lib/codegen/src/regalloc/solver.rs @@ -777,7 +777,7 @@ impl Solver { if is_global { let mut new_var = Variable::new_live(value, rc, reg, true); new_var.is_global = true; - dbg!("add_tied_input: new tied-global var: {}", new_var); + dbg!("add_tied_input: new tied-global value: {}", new_var); self.vars.push(new_var); self.regs_in.free(rc, reg); } else { @@ -920,7 +920,7 @@ impl Solver { Some(reg) => reg, None => { // If `v` must avoid global interference, there is not point in requesting - // live registers be diverted. We need to make it a non-global variable. + // live registers be diverted. We need to make it a non-global valueiable. if v.is_global && gregs.iter(rc).next().is_none() { return Err(SolverError::Global(v.value)); } else { diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs index 12643fc61b..98766e233d 100644 --- a/lib/codegen/src/verifier/mod.rs +++ b/lib/codegen/src/verifier/mod.rs @@ -63,7 +63,7 @@ use flowgraph::ControlFlowGraph; use ir; use ir::entities::AnyEntity; use ir::instructions::{BranchInfo, CallInfo, InstructionFormat, ResolvedConstraint}; -use ir::{types, ArgumentLoc, Ebb, FuncRef, Function, GlobalVar, Inst, JumpTable, Opcode, SigRef, +use ir::{types, ArgumentLoc, Ebb, FuncRef, Function, GlobalValue, Inst, JumpTable, Opcode, SigRef, StackSlot, StackSlotKind, Type, Value, ValueDef, ValueList, ValueLoc}; use isa::TargetIsa; use iterators::IteratorExtras; @@ -168,16 +168,16 @@ impl<'a> Verifier<'a> { } } - // Check for cycles in the global variable declarations. - fn verify_global_vars(&self) -> VerifierResult<()> { + // Check for cycles in the global valueiable declarations. + fn verify_global_values(&self) -> VerifierResult<()> { let mut seen = SparseSet::new(); - for gv in self.func.global_vars.keys() { + for gv in self.func.global_values.keys() { seen.clear(); seen.insert(gv); let mut cur = gv; - while let ir::GlobalVarData::Deref { base, .. } = self.func.global_vars[cur] { + while let ir::GlobalValueData::Deref { base, .. } = self.func.global_values[cur] { if seen.insert(base).is_some() { return err!(gv, "deref cycle: {}", DisplayList(seen.as_slice())); } @@ -327,8 +327,8 @@ impl<'a> Verifier<'a> { StackLoad { stack_slot, .. } | StackStore { stack_slot, .. } => { self.verify_stack_slot(inst, stack_slot)?; } - UnaryGlobalVar { global_var, .. } => { - self.verify_global_var(inst, global_var)?; + UnaryGlobalValue { global_value, .. } => { + self.verify_global_value(inst, global_value)?; } HeapAddr { heap, .. } => { self.verify_heap(inst, heap)?; @@ -413,9 +413,9 @@ impl<'a> Verifier<'a> { } } - fn verify_global_var(&self, inst: Inst, gv: GlobalVar) -> VerifierResult<()> { - if !self.func.global_vars.is_valid(gv) { - err!(inst, "invalid global variable {}", gv) + fn verify_global_value(&self, inst: Inst, gv: GlobalValue) -> VerifierResult<()> { + if !self.func.global_values.is_valid(gv) { + err!(inst, "invalid global valueiable {}", gv) } else { Ok(()) } @@ -1124,7 +1124,7 @@ impl<'a> Verifier<'a> { } pub fn run(&self) -> VerifierResult<()> { - self.verify_global_vars()?; + self.verify_global_values()?; self.typecheck_entry_block_params()?; for ebb in self.func.layout.ebbs() { for inst in self.func.layout.ebb_insts(ebb) { diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs index b99066ea53..124034b39a 100644 --- a/lib/codegen/src/write.rs +++ b/lib/codegen/src/write.rs @@ -49,7 +49,7 @@ fn write_preamble( writeln!(w, " {} = {}", ss, slot)?; } - for (gv, gv_data) in &func.global_vars { + for (gv, gv_data) in &func.global_values { any = true; writeln!(w, " {} = {}", gv, gv_data)?; } @@ -285,7 +285,7 @@ pub fn write_operands( UnaryIeee32 { imm, .. } => write!(w, " {}", imm), UnaryIeee64 { imm, .. } => write!(w, " {}", imm), UnaryBool { imm, .. } => write!(w, " {}", imm), - UnaryGlobalVar { global_var, .. } => write!(w, " {}", global_var), + UnaryGlobalValue { global_value, .. } => write!(w, " {}", global_value), Binary { args, .. } => write!(w, " {}, {}", args[0], args[1]), BinaryImm { arg, imm, .. } => write!(w, " {}, {}", arg, imm), Ternary { args, .. } => write!(w, " {}, {}, {}", args[0], args[1], args[2]), diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index 4c900c843a..0bb82efa9e 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -266,7 +266,7 @@ impl Backend for FaerieBackend { &mut self, _data: &mut FaerieCompiledData, _offset: usize, - _what: ir::GlobalVar, + _what: ir::GlobalValue, _usize: binemit::Addend, ) { unimplemented!() diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 453fc09c0e..fc2280623a 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -3,10 +3,10 @@ use cretonne_codegen::cursor::{Cursor, FuncCursor}; use cretonne_codegen::entity::{EntityMap, EntityRef, EntitySet}; use cretonne_codegen::ir; use cretonne_codegen::ir::function::DisplayFunction; -use cretonne_codegen::ir::{DataFlowGraph, Ebb, ExtFuncData, FuncRef, Function, GlobalVar, - GlobalVarData, Heap, HeapData, Inst, InstBuilderBase, InstructionData, - JumpTable, JumpTableData, SigRef, Signature, StackSlot, StackSlotData, - Type, Value}; +use cretonne_codegen::ir::{DataFlowGraph, Ebb, ExtFuncData, FuncRef, Function, GlobalValue, + GlobalValueData, Heap, HeapData, Inst, InstBuilderBase, + InstructionData, JumpTable, JumpTableData, SigRef, Signature, + StackSlot, StackSlotData, Type, Value}; use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::packed_option::PackedOption; use ssa::{Block, SSABuilder, SideEffects}; @@ -377,9 +377,9 @@ where self.func.import_function(data) } - /// Declares a global variable accessible to the function. - pub fn create_global_var(&mut self, data: GlobalVarData) -> GlobalVar { - self.func.create_global_var(data) + /// Declares a global valueiable accessible to the function. + pub fn create_global_value(&mut self, data: GlobalValueData) -> GlobalValue { + self.func.create_global_value(data) } /// Declares a heap accessible to the function. diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs index 85ff12a849..ef8eef9713 100644 --- a/lib/module/src/backend.rs +++ b/lib/module/src/backend.rs @@ -84,7 +84,7 @@ where &mut self, data: &mut Self::CompiledData, offset: usize, - what: ir::GlobalVar, + what: ir::GlobalValue, addend: binemit::Addend, ); diff --git a/lib/module/src/data_context.rs b/lib/module/src/data_context.rs index 0fef3f6400..69cca4b5fe 100644 --- a/lib/module/src/data_context.rs +++ b/lib/module/src/data_context.rs @@ -52,11 +52,11 @@ pub struct DataDescription { /// External function declarations. pub function_decls: PrimaryMap, /// External data object declarations. - pub data_decls: PrimaryMap, + pub data_decls: PrimaryMap, /// Function addresses to write at specified offsets. pub function_relocs: Vec<(CodeOffset, ir::FuncRef)>, /// Data addresses to write at specified offsets. - pub data_relocs: Vec<(CodeOffset, ir::GlobalVar, Addend)>, + pub data_relocs: Vec<(CodeOffset, ir::GlobalValue, Addend)>, } /// This is to data objects what cretonne_codegen::Context is to functions. @@ -114,14 +114,14 @@ impl DataContext { self.description.function_decls.push(name) } - /// Declares a global variable import. + /// Declares a global valueiable import. /// /// TODO: Rename to import_data? /// /// Users of the `Module` API generally should call /// `Module::declare_data_in_data` instead, as it takes care of generating /// the appropriate `ExternalName`. - pub fn import_global_var(&mut self, name: ir::ExternalName) -> ir::GlobalVar { + pub fn import_global_value(&mut self, name: ir::ExternalName) -> ir::GlobalValue { self.description.data_decls.push(name) } @@ -131,7 +131,7 @@ impl DataContext { } /// Write the address of `data` into the data at offset `offset`. - pub fn write_data_addr(&mut self, offset: CodeOffset, data: ir::GlobalVar, addend: Addend) { + pub fn write_data_addr(&mut self, offset: CodeOffset, data: ir::GlobalValue, addend: Addend) { self.description.data_relocs.push((offset, data, addend)) } @@ -168,8 +168,8 @@ mod tests { let _func_a = data_ctx.import_function(ir::ExternalName::user(0, 0)); let func_b = data_ctx.import_function(ir::ExternalName::user(0, 1)); let func_c = data_ctx.import_function(ir::ExternalName::user(1, 0)); - let _data_a = data_ctx.import_global_var(ir::ExternalName::user(2, 2)); - let data_b = data_ctx.import_global_var(ir::ExternalName::user(2, 3)); + let _data_a = data_ctx.import_global_value(ir::ExternalName::user(2, 2)); + let data_b = data_ctx.import_global_value(ir::ExternalName::user(2, 3)); data_ctx.write_function_addr(8, func_b); data_ctx.write_function_addr(16, func_c); diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index 84188ccd75..76ac8926b9 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -2,7 +2,7 @@ // TODO: Should `ir::Function` really have a `name`? -// TODO: Factor out `ir::Function`'s `ext_funcs` and `global_vars` into a struct +// TODO: Factor out `ir::Function`'s `ext_funcs` and `global_values` into a struct // shared with `DataContext`? use cretonne_codegen::entity::{EntityRef, PrimaryMap}; @@ -440,10 +440,10 @@ where /// Use this when you're building the IR of a function to reference a data object. /// /// TODO: Same as above. - pub fn declare_data_in_func(&self, data: DataId, func: &mut ir::Function) -> ir::GlobalVar { + pub fn declare_data_in_func(&self, data: DataId, func: &mut ir::Function) -> ir::GlobalValue { let decl = &self.contents.data_objects[data].decl; let colocated = decl.linkage.is_final(); - func.create_global_var(ir::GlobalVarData::Sym { + func.create_global_value(ir::GlobalValueData::Sym { name: ir::ExternalName::user(1, data.index() as u32), colocated, }) @@ -455,8 +455,8 @@ where } /// TODO: Same as above. - pub fn declare_data_in_data(&self, data: DataId, ctx: &mut DataContext) -> ir::GlobalVar { - ctx.import_global_var(ir::ExternalName::user(1, data.index() as u32)) + pub fn declare_data_in_data(&self, data: DataId, ctx: &mut DataContext) -> ir::GlobalValue { + ctx.import_global_value(ir::ExternalName::user(1, data.index() as u32)) } /// Define a function, producing the function body from the given `Context`. @@ -536,7 +536,7 @@ where &mut self, data: DataId, offset: usize, - what: ir::GlobalVar, + what: ir::GlobalValue, addend: binemit::Addend, ) { let info = &mut self.contents.data_objects[data]; diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 6c622ff64f..43bee2c399 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -34,7 +34,7 @@ pub enum Token<'a> { Value(Value), // v12, v7 Ebb(Ebb), // ebb3 StackSlot(u32), // ss3 - GlobalVar(u32), // gv3 + GlobalValue(u32), // gv3 Heap(u32), // heap2 JumpTable(u32), // jt2 FuncRef(u32), // fn2 @@ -338,7 +338,7 @@ impl<'a> Lexer<'a> { "v" => Value::with_number(number).map(Token::Value), "ebb" => Ebb::with_number(number).map(Token::Ebb), "ss" => Some(Token::StackSlot(number)), - "gv" => Some(Token::GlobalVar(number)), + "gv" => Some(Token::GlobalValue(number)), "heap" => Some(Token::Heap(number)), "jt" => Some(Token::JumpTable(number)), "fn" => Some(Token::FuncRef(number)), diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 4fba2a19c8..996c7af6ca 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -7,7 +7,7 @@ use cretonne_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32}; use cretonne_codegen::ir::instructions::{InstructionData, InstructionFormat, VariableArgs}; use cretonne_codegen::ir::types::VOID; use cretonne_codegen::ir::{AbiParam, ArgumentExtension, ArgumentLoc, Ebb, ExtFuncData, - ExternalName, FuncRef, Function, GlobalVar, GlobalVarData, Heap, + ExternalName, FuncRef, Function, GlobalValue, GlobalValueData, Heap, HeapBase, HeapData, HeapStyle, JumpTable, JumpTableData, MemFlags, Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind, Type, Value, ValueLoc}; @@ -139,22 +139,27 @@ impl<'a> Context<'a> { } } - // Allocate a global variable slot. - fn add_gv(&mut self, gv: GlobalVar, data: GlobalVarData, loc: &Location) -> ParseResult<()> { - while self.function.global_vars.next_key().index() <= gv.index() { - self.function.create_global_var(GlobalVarData::Sym { + // Allocate a global valueiable slot. + fn add_gv( + &mut self, + gv: GlobalValue, + data: GlobalValueData, + loc: &Location, + ) -> ParseResult<()> { + while self.function.global_values.next_key().index() <= gv.index() { + self.function.create_global_value(GlobalValueData::Sym { name: ExternalName::testcase(""), colocated: false, }); } - self.function.global_vars[gv] = data; + self.function.global_values[gv] = data; self.map.def_gv(gv, loc) } - // Resolve a reference to a global variable. - fn check_gv(&self, gv: GlobalVar, loc: &Location) -> ParseResult<()> { + // Resolve a reference to a global valueiable. + fn check_gv(&self, gv: GlobalValue, loc: &Location) -> ParseResult<()> { if !self.map.contains_gv(gv) { - err!(loc, "undefined global variable {}", gv) + err!(loc, "undefined global valueiable {}", gv) } else { Ok(()) } @@ -245,7 +250,7 @@ impl<'a> Context<'a> { } // Assign the global for the stack limit. - fn set_stack_limit(&mut self, gv: GlobalVar, loc: &Location) -> ParseResult<()> { + fn set_stack_limit(&mut self, gv: GlobalValue, loc: &Location) -> ParseResult<()> { if let Some(_) = self.function.set_stack_limit(Some(gv)) { err!(loc, "multiple stack_limit declarations") } else { @@ -396,11 +401,11 @@ impl<'a> Parser<'a> { err!(self.loc, err_msg) } - // Match and consume a global variable reference. - fn match_gv(&mut self, err_msg: &str) -> ParseResult { - if let Some(Token::GlobalVar(gv)) = self.token() { + // Match and consume a global valueiable reference. + fn match_gv(&mut self, err_msg: &str) -> ParseResult { + if let Some(Token::GlobalValue(gv)) = self.token() { self.consume(); - if let Some(gv) = GlobalVar::with_number(gv) { + if let Some(gv) = GlobalValue::with_number(gv) { return Ok(gv); } } @@ -1000,9 +1005,9 @@ impl<'a> Parser<'a> { self.parse_stack_slot_decl() .and_then(|(ss, dat)| ctx.add_ss(ss, dat, &loc)) } - Some(Token::GlobalVar(..)) => { + Some(Token::GlobalValue(..)) => { self.start_gathering_comments(); - self.parse_global_var_decl() + self.parse_global_value_decl() .and_then(|(gv, dat)| ctx.add_gv(gv, dat, &self.loc)) } Some(Token::Heap(..)) => { @@ -1072,36 +1077,45 @@ impl<'a> Parser<'a> { Ok((ss, data)) } - // Parse a global variable decl. + // Parse a global valueiable decl. // - // global-var-decl ::= * GlobalVar(gv) "=" global-var-desc + // global-var-decl ::= * GlobalValue(gv) "=" global-var-desc // global-var-desc ::= "vmctx" offset32 - // | "deref" "(" GlobalVar(base) ")" offset32 + // | "deref" "(" GlobalValue(base) ")" offset32 // | globalsym ["colocated"] name // - fn parse_global_var_decl(&mut self) -> ParseResult<(GlobalVar, GlobalVarData)> { - let gv = self.match_gv("expected global variable number: gv«n»")?; + fn parse_global_value_decl(&mut self) -> ParseResult<(GlobalValue, GlobalValueData)> { + let gv = self.match_gv("expected global valueiable number: gv«n»")?; - self.match_token(Token::Equal, "expected '=' in global variable declaration")?; + self.match_token( + Token::Equal, + "expected '=' in global valueiable declaration", + )?; - let data = match self.match_any_identifier("expected global variable kind")? { + let data = match self.match_any_identifier("expected global valueiable kind")? { "vmctx" => { let offset = self.optional_offset32()?; - GlobalVarData::VMContext { offset } + GlobalValueData::VMContext { offset } } "deref" => { - self.match_token(Token::LPar, "expected '(' in 'deref' global variable decl")?; - let base = self.match_gv("expected global variable: gv«n»")?; - self.match_token(Token::RPar, "expected ')' in 'deref' global variable decl")?; + self.match_token( + Token::LPar, + "expected '(' in 'deref' global valueiable decl", + )?; + let base = self.match_gv("expected global valueiable: gv«n»")?; + self.match_token( + Token::RPar, + "expected ')' in 'deref' global valueiable decl", + )?; let offset = self.optional_offset32()?; - GlobalVarData::Deref { base, offset } + GlobalValueData::Deref { base, offset } } "globalsym" => { let colocated = self.optional(Token::Identifier("colocated")); let name = self.parse_external_name()?; - GlobalVarData::Sym { name, colocated } + GlobalValueData::Sym { name, colocated } } - other => return err!(self.loc, "Unknown global variable kind '{}'", other), + other => return err!(self.loc, "Unknown global valueiable kind '{}'", other), }; // Collect any trailing comments. @@ -1117,7 +1131,7 @@ impl<'a> Parser<'a> { // heap-desc ::= heap-style heap-base { "," heap-attr } // heap-style ::= "static" | "dynamic" // heap-base ::= "reserved_reg" - // | GlobalVar(base) + // | GlobalValue(base) // heap-attr ::= "min" Imm64(bytes) // | "max" Imm64(bytes) // | "guard" Imm64(bytes) @@ -1130,15 +1144,15 @@ impl<'a> Parser<'a> { // heap-desc ::= heap-style * heap-base { "," heap-attr } // heap-base ::= * "reserved_reg" - // | * GlobalVar(base) + // | * GlobalValue(base) let base = match self.token() { Some(Token::Identifier("reserved_reg")) => HeapBase::ReservedReg, - Some(Token::GlobalVar(base_num)) => { - let base_gv = match GlobalVar::with_number(base_num) { + Some(Token::GlobalValue(base_num)) => { + let base_gv = match GlobalValue::with_number(base_num) { Some(gv) => gv, - None => return err!(self.loc, "invalid global variable number for heap base"), + None => return err!(self.loc, "invalid global valueiable number for heap base"), }; - HeapBase::GlobalVar(base_gv) + HeapBase::GlobalValue(base_gv) } _ => return err!(self.loc, "expected heap base"), }; @@ -1309,11 +1323,11 @@ impl<'a> Parser<'a> { } } - /// stack-limit-decl ::= "stack_limit" "=" GlobalVar(gv) - fn parse_stack_limit_decl(&mut self) -> ParseResult { + /// stack-limit-decl ::= "stack_limit" "=" GlobalValue(gv) + fn parse_stack_limit_decl(&mut self) -> ParseResult { self.consume(); self.match_token(Token::Equal, "expected '=' in stack limit declaration")?; - let gv = self.match_gv("expected global variable")?; + let gv = self.match_gv("expected global valueiable")?; Ok(gv) } @@ -1889,12 +1903,12 @@ impl<'a> Parser<'a> { opcode, imm: self.match_bool("expected immediate boolean operand")?, }, - InstructionFormat::UnaryGlobalVar => { - let gv = self.match_gv("expected global variable")?; + InstructionFormat::UnaryGlobalValue => { + let gv = self.match_gv("expected global valueiable")?; ctx.check_gv(gv, &self.loc)?; - InstructionData::UnaryGlobalVar { + InstructionData::UnaryGlobalValue { opcode, - global_var: gv, + global_value: gv, } } InstructionFormat::Binary => { diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 6b9dc51d02..1bb6231596 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -7,7 +7,7 @@ //! to parser clients. use cretonne_codegen::ir::entities::AnyEntity; -use cretonne_codegen::ir::{Ebb, FuncRef, GlobalVar, Heap, JumpTable, SigRef, StackSlot, Value}; +use cretonne_codegen::ir::{Ebb, FuncRef, GlobalValue, Heap, JumpTable, SigRef, StackSlot, Value}; use error::{Location, ParseResult}; use lexer::split_entity_name; use std::collections::HashMap; @@ -36,8 +36,8 @@ impl SourceMap { self.locations.contains_key(&ss.into()) } - /// Look up a global variable entity. - pub fn contains_gv(&self, gv: GlobalVar) -> bool { + /// Look up a global valueiable entity. + pub fn contains_gv(&self, gv: GlobalValue) -> bool { self.locations.contains_key(&gv.into()) } @@ -86,7 +86,7 @@ impl SourceMap { Some(ss.into()) } }), - "gv" => GlobalVar::with_number(num).and_then(|gv| { + "gv" => GlobalValue::with_number(num).and_then(|gv| { if !self.contains_gv(gv) { None } else { @@ -154,8 +154,8 @@ impl SourceMap { self.def_entity(entity.into(), loc) } - /// Define the global variable `entity`. - pub fn def_gv(&mut self, entity: GlobalVar, loc: &Location) -> ParseResult<()> { + /// Define the global valueiable `entity`. + pub fn def_gv(&mut self, entity: GlobalValue, loc: &Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs index 4df071efc9..5886b7ad76 100644 --- a/lib/simplejit/src/backend.rs +++ b/lib/simplejit/src/backend.rs @@ -217,7 +217,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { &mut self, _data: &mut Self::CompiledData, _offset: usize, - _what: ir::GlobalVar, + _what: ir::GlobalValue, _usize: Addend, ) { unimplemented!(); diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs index 11b773b7f5..1048b4d780 100644 --- a/lib/umbrella/src/lib.rs +++ b/lib/umbrella/src/lib.rs @@ -24,7 +24,7 @@ pub mod prelude { pub use codegen::ir::condcodes::{FloatCC, IntCC}; pub use codegen::ir::immediates::{Ieee32, Ieee64, Imm64}; pub use codegen::ir::types; - pub use codegen::ir::{AbiParam, Ebb, ExtFuncData, GlobalVarData, InstBuilder, JumpTableData, + pub use codegen::ir::{AbiParam, Ebb, ExtFuncData, GlobalValueData, InstBuilder, JumpTableData, MemFlags, Signature, StackSlotData, StackSlotKind, TrapCode, Type, Value}; pub use codegen::isa; pub use codegen::settings::{self, CallConv, Configurable}; diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 07e912da1d..7f9d390399 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -74,7 +74,7 @@ pub fn translate_operator( let val = match state.get_global(builder.func, global_index, environ) { GlobalValue::Const(val) => val, GlobalValue::Memory { gv, ty } => { - let addr = builder.ins().global_addr(environ.native_pointer(), gv); + let addr = builder.ins().global_value(environ.native_pointer(), gv); let mut flags = ir::MemFlags::new(); flags.set_notrap(); flags.set_aligned(); @@ -87,7 +87,7 @@ pub fn translate_operator( match state.get_global(builder.func, global_index, environ) { GlobalValue::Const(_) => panic!("global #{} is a constant", global_index), GlobalValue::Memory { gv, .. } => { - let addr = builder.ins().global_addr(environ.native_pointer(), gv); + let addr = builder.ins().global_value(environ.native_pointer(), gv); let mut flags = ir::MemFlags::new(); flags.set_notrap(); flags.set_aligned(); diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 7b71f8b69e..0bd7fc1a94 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -159,7 +159,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue { // Just create a dummy `vmctx` global. let offset = ((index * 8) as i32 + 8).into(); - let gv = func.create_global_var(ir::GlobalVarData::VMContext { offset }); + let gv = func.create_global_value(ir::GlobalValueData::VMContext { offset }); GlobalValue::Memory { gv, ty: self.mod_info.globals[index].entity.ty, @@ -168,10 +168,10 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ fn make_heap(&mut self, func: &mut ir::Function, _index: MemoryIndex) -> ir::Heap { // Create a static heap whose base address is stored at `vmctx+0`. - let gv = func.create_global_var(ir::GlobalVarData::VMContext { offset: 0.into() }); + let gv = func.create_global_value(ir::GlobalValueData::VMContext { offset: 0.into() }); func.create_heap(ir::HeapData { - base: ir::HeapBase::GlobalVar(gv), + base: ir::HeapBase::GlobalValue(gv), min_size: 0.into(), guard_size: 0x8000_0000.into(), style: ir::HeapStyle::Static { diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 00882d035f..848f29747c 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -9,17 +9,17 @@ use translation_utils::{FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, Table, TableIndex}; use wasmparser::BinaryReaderError; -/// The value of a WebAssembly global variable. +/// The value of a WebAssembly global valueiable. #[derive(Clone, Copy)] pub enum GlobalValue { /// This is a constant global with a value known at compile time. Const(ir::Value), - /// This is a variable in memory that should be referenced as a `GlobalVar`. + /// This is a variable in memory that should be referenced as a `GlobalValue`. Memory { - /// Which global variable should be referenced. - gv: ir::GlobalVar, - /// The global variable's type. + /// Which global valueiable should be referenced. + gv: ir::GlobalValue, + /// The global valueiable's type. ty: ir::Type, }, } @@ -88,12 +88,12 @@ pub trait FuncEnvironment { ir::Type::int(u16::from(self.triple().pointer_width().unwrap().bits())).unwrap() } - /// Set up the necessary preamble definitions in `func` to access the global variable + /// Set up the necessary preamble definitions in `func` to access the global valueiable /// identified by `index`. /// /// The index space covers both imported globals and globals defined by the module. /// - /// Return the global variable reference that should be used to access the global and the + /// Return the global valueiable reference that should be used to access the global and the /// WebAssembly type of the global. fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue; diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 99a0051121..d3dc64496a 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -4,7 +4,7 @@ //! The code of theses helper function is straightforward since it is only about reading metadata //! about linear memories, tables, globals, etc. and storing them for later use. //! -//! The special case of the initialize expressions for table elements offsets or global variables +//! The special case of the initialize expressions for table elements offsets or global valueiables //! is handled, according to the semantics of WebAssembly, to only specific expressions that are //! interpreted on the fly. use cretonne_codegen::ir::{self, AbiParam, Signature}; diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index b0bb4f5c7a..372cd297b7 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -134,7 +134,7 @@ pub struct TranslationState { pub control_stack: Vec, pub reachable: bool, - // Map of global variables that have already been created by `FuncEnvironment::make_global`. + // Map of global valueiables that have already been created by `FuncEnvironment::make_global`. globals: HashMap, // Map of heaps that have been created by `FuncEnvironment::make_heap`. @@ -272,7 +272,7 @@ impl TranslationState { /// Methods for handling entity references. impl TranslationState { - /// Get the `GlobalVar` reference that should be used to access the global variable `index`. + /// Get the `GlobalValue` reference that should be used to access the global valueiable `index`. /// Create the reference if necessary. /// Also return the WebAssembly type of the global. pub fn get_global( diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 3ed13899e6..9257870264 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -7,7 +7,7 @@ use wasmparser; pub type FunctionIndex = usize; /// Index of a table (imported or defined) inside the WebAssembly module. pub type TableIndex = usize; -/// Index of a global variable (imported or defined) inside the WebAssembly module. +/// Index of a global valueiable (imported or defined) inside the WebAssembly module. pub type GlobalIndex = usize; /// Index of a linear memory (imported or defined) inside the WebAssembly module. pub type MemoryIndex = usize; From 3686fc2fc770048f365a718d05b78951ab3f6705 Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Thu, 14 Jun 2018 01:14:59 -0400 Subject: [PATCH 1853/3084] Fix typos caused by find-and-replace --- cranelift/docs/compare-llvm.rst | 2 +- cranelift/docs/langref.rst | 28 ++++++++--------- cranelift/docs/metaref.rst | 4 +-- cranelift/filetests/parser/memory.cton | 2 +- lib/codegen/meta/base/entities.py | 2 +- lib/codegen/meta/base/formats.py | 2 +- lib/codegen/meta/base/settings.py | 2 +- lib/codegen/meta/cdsl/formats.py | 2 +- lib/codegen/meta/cdsl/registers.py | 2 +- lib/codegen/src/ir/entities.rs | 4 +-- lib/codegen/src/ir/extfunc.rs | 2 +- lib/codegen/src/ir/function.rs | 2 +- lib/codegen/src/ir/globalvalue.rs | 8 ++--- lib/codegen/src/ir/globalvar.rs | 8 ++--- lib/codegen/src/ir/heap.rs | 2 +- lib/codegen/src/legalizer/globalvalue.rs | 8 ++--- lib/codegen/src/regalloc/solver.rs | 2 +- lib/codegen/src/verifier/mod.rs | 4 +-- lib/frontend/src/frontend.rs | 2 +- lib/module/src/data_context.rs | 2 +- lib/reader/src/parser.rs | 39 +++++++++--------------- lib/reader/src/sourcemap.rs | 4 +-- lib/wasm/src/environ/spec.rs | 10 +++--- lib/wasm/src/sections_translator.rs | 2 +- lib/wasm/src/state.rs | 4 +-- lib/wasm/src/translation_utils.rs | 2 +- 26 files changed, 71 insertions(+), 80 deletions(-) diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst index 4765cdc18b..6afc5ed419 100644 --- a/cranelift/docs/compare-llvm.rst +++ b/cranelift/docs/compare-llvm.rst @@ -110,7 +110,7 @@ Program structure In LLVM IR, the largest representable unit is the *module* which corresponds more or less to a C translation unit. It is a collection of functions and -global valueiables that may contain references to external symbols too. +global values that may contain references to external symbols too. In Cretonne IR, the largest representable unit is the *function*. This is so that functions can easily be compiled in parallel without worrying about diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 707927f454..3736565f94 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -554,7 +554,7 @@ stack overflow checks in the prologue. the stack pointer has reached or exceeded the limit, generate a trap with a ``stk_ovf`` code. - The global valueiable must be accessible and naturally aligned for a + The global value must be accessible and naturally aligned for a pointer-sized value. Setting `stack_limit` is an alternative way to detect stack overflow, when using @@ -563,12 +563,12 @@ stack overflow checks in the prologue. Global variables ---------------- -A *global valueiable* is an :term:`accessible` object in memory whose address is +A *global value* is an :term:`accessible` object in memory whose address is not known at compile time. The address is computed at runtime by :inst:`global_value`, possibly using information provided by the linker via -relocations. There are multiple kinds of global valueiables using different +relocations. There are multiple kinds of global values using different methods for determining their address. Cretonne does not track the type or even -the size of global valueiables, they are just pointers to non-stack memory. +the size of global values, they are just pointers to non-stack memory. When Cretonne is generating code for a virtual machine environment, globals can be used to access data structures in the VM's runtime. This requires functions @@ -578,26 +578,26 @@ Cretonne functions. .. inst:: GV = vmctx+Offset - Declare a global valueiable in the VM context struct. + Declare a global value in the VM context struct. - This declares a global valueiable whose address is a constant offset from the + This declares a global value whose address is a constant offset from the VM context pointer which is passed as a hidden argument to all functions JIT-compiled for the VM. - Typically, the VM context is a C struct, and the declared global valueiable + Typically, the VM context is a C struct, and the declared global value is a member of the struct. :arg Offset: Byte offset from the VM context pointer to the global variable. :result GV: Global variable. -The address of a global valueiable can also be derived by treating another global +The address of a global value can also be derived by treating another global variable as a struct pointer. This makes it possible to chase pointers into VM runtime data structures. .. inst:: GV = deref(BaseGV)+Offset - Declare a global valueiable in a struct pointed to by BaseGV. + Declare a global value in a struct pointed to by BaseGV. The address of GV can be computed by first loading a pointer from BaseGV and adding Offset to it. @@ -605,7 +605,7 @@ runtime data structures. It is assumed the BaseGV resides in readable memory with the appropriate alignment for storing a pointer. - Chains of ``deref`` global valueiables are possible, but cycles are not + Chains of ``deref`` global values are possible, but cycles are not allowed. They will be caught by the IR verifier. :arg BaseGV: Global variable containing the base pointer. @@ -615,7 +615,7 @@ runtime data structures. .. inst:: GV = [colocated] globalsym name - Declare a global valueiable at a symbolic address. + Declare a global value at a symbolic address. The address of GV is symbolic and will be assigned a relocation, so that it can be resolved by a later linking phase. @@ -701,7 +701,7 @@ trap when accessed. address space reserved for the heap, not including the guard pages. :arg GuardBytes: Size of the guard pages in bytes. -When the base is a global valueiable, it must be :term:`accessible` and naturally +When the base is a global value, it must be :term:`accessible` and naturally aligned for a pointer value. The ``reserved_reg`` option is not yet implemented. @@ -711,7 +711,7 @@ Dynamic heaps A *dynamic heap* can be relocated to a different base address when it is resized, and its bound can move dynamically. The guard pages move when the heap -is resized. The bound of a dynamic heap is stored in a global valueiable. +is resized. The bound of a dynamic heap is stored in a global value. .. inst:: H = dynamic Base, min MinBytes, bound BoundGV, guard GuardBytes @@ -724,7 +724,7 @@ is resized. The bound of a dynamic heap is stored in a global valueiable. :arg BoundGV: Global variable containing the current heap bound in bytes. :arg GuardBytes: Size of the guard pages in bytes. -When the base is a global valueiable, it must be :term:`accessible` and naturally +When the base is a global value, it must be :term:`accessible` and naturally aligned for a pointer value. The ``reserved_reg`` option is not yet implemented. diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 70cc5ecc98..357f1f1fde 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -15,7 +15,7 @@ The meta language descriptions are Python modules under the steps: 1. The Python modules are imported. This has the effect of building static data - structures in global valueiables in the modules. These static data structures + structures in global values in the modules. These static data structures in the :mod:`base` and :mod:`isa` packages use the classes in the :mod:`cdsl` package to describe instruction sets and other properties. @@ -81,7 +81,7 @@ open :class:`InstructionGroup`. :members: The basic Cretonne instruction set described in :doc:`langref` is defined by the -Python module :mod:`base.instructions`. This module has a global valueiable +Python module :mod:`base.instructions`. This module has a global value :data:`base.instructions.GROUP` which is an :class:`InstructionGroup` instance containing all the base instructions. diff --git a/cranelift/filetests/parser/memory.cton b/cranelift/filetests/parser/memory.cton index 8e01eb8cae..6df76ce561 100644 --- a/cranelift/filetests/parser/memory.cton +++ b/cranelift/filetests/parser/memory.cton @@ -25,7 +25,7 @@ ebb0: return v1 } -; Refer to a global valueiable before it's been declared. +; Refer to a global value before it's been declared. function %backref() -> i32 { gv1 = deref(gv2)-32 ; check: gv1 = deref(gv2)-32 diff --git a/lib/codegen/meta/base/entities.py b/lib/codegen/meta/base/entities.py index e8915a0f79..a91d894e04 100644 --- a/lib/codegen/meta/base/entities.py +++ b/lib/codegen/meta/base/entities.py @@ -16,7 +16,7 @@ ebb = EntityRefKind( #: A reference to a stack slot declared in the function preamble. stack_slot = EntityRefKind('stack_slot', 'A stack slot.') -#: A reference to a global valueiable. +#: A reference to a global value. global_value = EntityRefKind('global_value', 'A global value.') #: A reference to a function sugnature declared in the function preamble. diff --git a/lib/codegen/meta/base/formats.py b/lib/codegen/meta/base/formats.py index 1c0d7284f2..90adc83176 100644 --- a/lib/codegen/meta/base/formats.py +++ b/lib/codegen/meta/base/formats.py @@ -79,5 +79,5 @@ CondTrap = InstructionFormat(VALUE, trapcode) IntCondTrap = InstructionFormat(intcc, VALUE, trapcode) FloatCondTrap = InstructionFormat(floatcc, VALUE, trapcode) -# Finally extract the names of global valueiables in this module. +# Finally extract the names of global values in this module. InstructionFormat.extract_names(globals()) diff --git a/lib/codegen/meta/base/settings.py b/lib/codegen/meta/base/settings.py index c3f9ae182b..0abeab59e4 100644 --- a/lib/codegen/meta/base/settings.py +++ b/lib/codegen/meta/base/settings.py @@ -53,7 +53,7 @@ call_conv = EnumSetting( # Note that Cretonne doesn't currently need an is_pie flag, because PIE is just # PIC where symbols can't be pre-empted, which can be expressed with the -# `colocated` flag on external functions and global valueiables. +# `colocated` flag on external functions and global values. is_pic = BoolSetting("Enable Position-Independent Code generation") colocated_libcalls = BoolSetting( diff --git a/lib/codegen/meta/cdsl/formats.py b/lib/codegen/meta/cdsl/formats.py index ebee96281e..a1e84e4737 100644 --- a/lib/codegen/meta/cdsl/formats.py +++ b/lib/codegen/meta/cdsl/formats.py @@ -208,7 +208,7 @@ class InstructionFormat(object): """ Given a dict mapping name -> object as returned by `globals()`, find all the InstructionFormat objects and set their name from the dict key. - This is used to name a bunch of global valueiables in a module. + This is used to name a bunch of global values in a module. """ for name, obj in globs.items(): if isinstance(obj, InstructionFormat): diff --git a/lib/codegen/meta/cdsl/registers.py b/lib/codegen/meta/cdsl/registers.py index 4dccbc808a..0c96583db0 100644 --- a/lib/codegen/meta/cdsl/registers.py +++ b/lib/codegen/meta/cdsl/registers.py @@ -365,7 +365,7 @@ class RegClass(object): """ Given a dict mapping name -> object as returned by `globals()`, find all the RegClass objects and set their name from the dict key. - This is used to name a bunch of global valueiables in a module. + This is used to name a bunch of global values in a module. """ for name, obj in globs.items(): if isinstance(obj, RegClass): diff --git a/lib/codegen/src/ir/entities.rs b/lib/codegen/src/ir/entities.rs index d6461d47df..847a9ba925 100644 --- a/lib/codegen/src/ir/entities.rs +++ b/lib/codegen/src/ir/entities.rs @@ -82,13 +82,13 @@ impl StackSlot { } } -/// An opaque reference to a global valueiable. +/// An opaque reference to a global value. #[derive(Copy, Clone, PartialEq, Eq, Hash)] pub struct GlobalValue(u32); entity_impl!(GlobalValue, "gv"); impl GlobalValue { - /// Create a new global valueiable reference from its number. + /// Create a new global value reference from its number. /// /// This method is for use by the parser. pub fn with_number(n: u32) -> Option { diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs index 98b6e4fb40..00ac6da86f 100644 --- a/lib/codegen/src/ir/extfunc.rs +++ b/lib/codegen/src/ir/extfunc.rs @@ -284,7 +284,7 @@ pub enum ArgumentPurpose { /// A VM context pointer. /// /// This is a pointer to a context struct containing details about the current sandbox. It is - /// used as a base pointer for `vmctx` global valueiables. + /// used as a base pointer for `vmctx` global values. VMContext, /// A signature identifier. diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs index 5d59e6f3ce..e060c94029 100644 --- a/lib/codegen/src/ir/function.rs +++ b/lib/codegen/src/ir/function.rs @@ -145,7 +145,7 @@ impl Function { self.dfg.ext_funcs.push(data) } - /// Declares a global valueiable accessible to the function. + /// Declares a global value accessible to the function. pub fn create_global_value(&mut self, data: GlobalValueData) -> GlobalValue { self.global_values.push(data) } diff --git a/lib/codegen/src/ir/globalvalue.rs b/lib/codegen/src/ir/globalvalue.rs index 5d068a7583..26c0e48434 100644 --- a/lib/codegen/src/ir/globalvalue.rs +++ b/lib/codegen/src/ir/globalvalue.rs @@ -4,7 +4,7 @@ use ir::immediates::Offset32; use ir::{ExternalName, GlobalValue}; use std::fmt; -/// Information about a global valueiable declaration. +/// Information about a global value declaration. #[derive(Clone)] pub enum GlobalValueData { /// Variable is part of the VM context struct, it's address is a constant offset from the VM @@ -14,13 +14,13 @@ pub enum GlobalValueData { offset: Offset32, }, - /// Variable is part of a struct pointed to by another global valueiable. + /// Variable is part of a struct pointed to by another global value. /// - /// The `base` global valueiable is assumed to contain a pointer to a struct. This global + /// The `base` global value is assumed to contain a pointer to a struct. This global /// variable lives at an offset into the struct. The memory must be accessible, and /// naturally aligned to hold a pointer value. Deref { - /// The base pointer global valueiable. + /// The base pointer global value. base: GlobalValue, /// Byte offset to be added to the pointer loaded from `base`. diff --git a/lib/codegen/src/ir/globalvar.rs b/lib/codegen/src/ir/globalvar.rs index 5d068a7583..26c0e48434 100644 --- a/lib/codegen/src/ir/globalvar.rs +++ b/lib/codegen/src/ir/globalvar.rs @@ -4,7 +4,7 @@ use ir::immediates::Offset32; use ir::{ExternalName, GlobalValue}; use std::fmt; -/// Information about a global valueiable declaration. +/// Information about a global value declaration. #[derive(Clone)] pub enum GlobalValueData { /// Variable is part of the VM context struct, it's address is a constant offset from the VM @@ -14,13 +14,13 @@ pub enum GlobalValueData { offset: Offset32, }, - /// Variable is part of a struct pointed to by another global valueiable. + /// Variable is part of a struct pointed to by another global value. /// - /// The `base` global valueiable is assumed to contain a pointer to a struct. This global + /// The `base` global value is assumed to contain a pointer to a struct. This global /// variable lives at an offset into the struct. The memory must be accessible, and /// naturally aligned to hold a pointer value. Deref { - /// The base pointer global valueiable. + /// The base pointer global value. base: GlobalValue, /// Byte offset to be added to the pointer loaded from `base`. diff --git a/lib/codegen/src/ir/heap.rs b/lib/codegen/src/ir/heap.rs index 812ffa01d3..4bb0b69166 100644 --- a/lib/codegen/src/ir/heap.rs +++ b/lib/codegen/src/ir/heap.rs @@ -29,7 +29,7 @@ pub enum HeapBase { /// This feature is not yet implemented. ReservedReg, - /// The heap base is in a global valueiable. The variable must be accessible and naturally + /// The heap base is in a global value. The variable must be accessible and naturally /// aligned for a pointer. GlobalValue(GlobalValue), } diff --git a/lib/codegen/src/legalizer/globalvalue.rs b/lib/codegen/src/legalizer/globalvalue.rs index fbea36b9e6..c6f1a1d930 100644 --- a/lib/codegen/src/legalizer/globalvalue.rs +++ b/lib/codegen/src/legalizer/globalvalue.rs @@ -1,14 +1,14 @@ -//! Legalization of global valueiables. +//! Legalization of global values. //! //! This module exports the `expand_global_value` function which transforms a `global_value` -//! instruction into code that depends on the kind of global valueiable referenced. +//! instruction into code that depends on the kind of global value referenced. use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::{self, InstBuilder}; use isa::TargetIsa; -/// Expand a `global_value` instruction according to the definition of the global valueiable. +/// Expand a `global_value` instruction according to the definition of the global value. pub fn expand_global_value( inst: ir::Inst, func: &mut ir::Function, @@ -46,7 +46,7 @@ fn vmctx_addr(inst: ir::Inst, func: &mut ir::Function, offset: i64) { /// Expand a `global_value` instruction for a deref global. fn deref_addr(inst: ir::Inst, func: &mut ir::Function, base: ir::GlobalValue, offset: i64) { - // We need to load a pointer from the `base` global valueiable, so insert a new `global_value` + // We need to load a pointer from the `base` global value, so insert a new `global_value` // instruction. This depends on the iterative legalization loop. Note that the IR verifier // detects any cycles in the `deref` globals. let ptr_ty = func.dfg.value_type(func.dfg.first_result(inst)); diff --git a/lib/codegen/src/regalloc/solver.rs b/lib/codegen/src/regalloc/solver.rs index 779396cc57..1140aa7536 100644 --- a/lib/codegen/src/regalloc/solver.rs +++ b/lib/codegen/src/regalloc/solver.rs @@ -920,7 +920,7 @@ impl Solver { Some(reg) => reg, None => { // If `v` must avoid global interference, there is not point in requesting - // live registers be diverted. We need to make it a non-global valueiable. + // live registers be diverted. We need to make it a non-global value. if v.is_global && gregs.iter(rc).next().is_none() { return Err(SolverError::Global(v.value)); } else { diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs index 98766e233d..b25d770c3f 100644 --- a/lib/codegen/src/verifier/mod.rs +++ b/lib/codegen/src/verifier/mod.rs @@ -168,7 +168,7 @@ impl<'a> Verifier<'a> { } } - // Check for cycles in the global valueiable declarations. + // Check for cycles in the global value declarations. fn verify_global_values(&self) -> VerifierResult<()> { let mut seen = SparseSet::new(); @@ -415,7 +415,7 @@ impl<'a> Verifier<'a> { fn verify_global_value(&self, inst: Inst, gv: GlobalValue) -> VerifierResult<()> { if !self.func.global_values.is_valid(gv) { - err!(inst, "invalid global valueiable {}", gv) + err!(inst, "invalid global value {}", gv) } else { Ok(()) } diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index fc2280623a..89d933d2ae 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -377,7 +377,7 @@ where self.func.import_function(data) } - /// Declares a global valueiable accessible to the function. + /// Declares a global value accessible to the function. pub fn create_global_value(&mut self, data: GlobalValueData) -> GlobalValue { self.func.create_global_value(data) } diff --git a/lib/module/src/data_context.rs b/lib/module/src/data_context.rs index 69cca4b5fe..9ecd29399f 100644 --- a/lib/module/src/data_context.rs +++ b/lib/module/src/data_context.rs @@ -114,7 +114,7 @@ impl DataContext { self.description.function_decls.push(name) } - /// Declares a global valueiable import. + /// Declares a global value import. /// /// TODO: Rename to import_data? /// diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 996c7af6ca..7e85a8cc00 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -139,7 +139,7 @@ impl<'a> Context<'a> { } } - // Allocate a global valueiable slot. + // Allocate a global value slot. fn add_gv( &mut self, gv: GlobalValue, @@ -156,10 +156,10 @@ impl<'a> Context<'a> { self.map.def_gv(gv, loc) } - // Resolve a reference to a global valueiable. + // Resolve a reference to a global value. fn check_gv(&self, gv: GlobalValue, loc: &Location) -> ParseResult<()> { if !self.map.contains_gv(gv) { - err!(loc, "undefined global valueiable {}", gv) + err!(loc, "undefined global value {}", gv) } else { Ok(()) } @@ -401,7 +401,7 @@ impl<'a> Parser<'a> { err!(self.loc, err_msg) } - // Match and consume a global valueiable reference. + // Match and consume a global value reference. fn match_gv(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::GlobalValue(gv)) = self.token() { self.consume(); @@ -1077,7 +1077,7 @@ impl<'a> Parser<'a> { Ok((ss, data)) } - // Parse a global valueiable decl. + // Parse a global value decl. // // global-var-decl ::= * GlobalValue(gv) "=" global-var-desc // global-var-desc ::= "vmctx" offset32 @@ -1085,28 +1085,19 @@ impl<'a> Parser<'a> { // | globalsym ["colocated"] name // fn parse_global_value_decl(&mut self) -> ParseResult<(GlobalValue, GlobalValueData)> { - let gv = self.match_gv("expected global valueiable number: gv«n»")?; + let gv = self.match_gv("expected global value number: gv«n»")?; - self.match_token( - Token::Equal, - "expected '=' in global valueiable declaration", - )?; + self.match_token(Token::Equal, "expected '=' in global value declaration")?; - let data = match self.match_any_identifier("expected global valueiable kind")? { + let data = match self.match_any_identifier("expected global value kind")? { "vmctx" => { let offset = self.optional_offset32()?; GlobalValueData::VMContext { offset } } "deref" => { - self.match_token( - Token::LPar, - "expected '(' in 'deref' global valueiable decl", - )?; - let base = self.match_gv("expected global valueiable: gv«n»")?; - self.match_token( - Token::RPar, - "expected ')' in 'deref' global valueiable decl", - )?; + self.match_token(Token::LPar, "expected '(' in 'deref' global value decl")?; + let base = self.match_gv("expected global value: gv«n»")?; + self.match_token(Token::RPar, "expected ')' in 'deref' global value decl")?; let offset = self.optional_offset32()?; GlobalValueData::Deref { base, offset } } @@ -1115,7 +1106,7 @@ impl<'a> Parser<'a> { let name = self.parse_external_name()?; GlobalValueData::Sym { name, colocated } } - other => return err!(self.loc, "Unknown global valueiable kind '{}'", other), + other => return err!(self.loc, "Unknown global value kind '{}'", other), }; // Collect any trailing comments. @@ -1150,7 +1141,7 @@ impl<'a> Parser<'a> { Some(Token::GlobalValue(base_num)) => { let base_gv = match GlobalValue::with_number(base_num) { Some(gv) => gv, - None => return err!(self.loc, "invalid global valueiable number for heap base"), + None => return err!(self.loc, "invalid global value number for heap base"), }; HeapBase::GlobalValue(base_gv) } @@ -1327,7 +1318,7 @@ impl<'a> Parser<'a> { fn parse_stack_limit_decl(&mut self) -> ParseResult { self.consume(); self.match_token(Token::Equal, "expected '=' in stack limit declaration")?; - let gv = self.match_gv("expected global valueiable")?; + let gv = self.match_gv("expected global value")?; Ok(gv) } @@ -1904,7 +1895,7 @@ impl<'a> Parser<'a> { imm: self.match_bool("expected immediate boolean operand")?, }, InstructionFormat::UnaryGlobalValue => { - let gv = self.match_gv("expected global valueiable")?; + let gv = self.match_gv("expected global value")?; ctx.check_gv(gv, &self.loc)?; InstructionData::UnaryGlobalValue { opcode, diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 1bb6231596..814baf8e1b 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -36,7 +36,7 @@ impl SourceMap { self.locations.contains_key(&ss.into()) } - /// Look up a global valueiable entity. + /// Look up a global value entity. pub fn contains_gv(&self, gv: GlobalValue) -> bool { self.locations.contains_key(&gv.into()) } @@ -154,7 +154,7 @@ impl SourceMap { self.def_entity(entity.into(), loc) } - /// Define the global valueiable `entity`. + /// Define the global value `entity`. pub fn def_gv(&mut self, entity: GlobalValue, loc: &Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 848f29747c..4d97a34cc5 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -9,7 +9,7 @@ use translation_utils::{FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, Table, TableIndex}; use wasmparser::BinaryReaderError; -/// The value of a WebAssembly global valueiable. +/// The value of a WebAssembly global value. #[derive(Clone, Copy)] pub enum GlobalValue { /// This is a constant global with a value known at compile time. @@ -17,9 +17,9 @@ pub enum GlobalValue { /// This is a variable in memory that should be referenced as a `GlobalValue`. Memory { - /// Which global valueiable should be referenced. + /// Which global value should be referenced. gv: ir::GlobalValue, - /// The global valueiable's type. + /// The global value's type. ty: ir::Type, }, } @@ -88,12 +88,12 @@ pub trait FuncEnvironment { ir::Type::int(u16::from(self.triple().pointer_width().unwrap().bits())).unwrap() } - /// Set up the necessary preamble definitions in `func` to access the global valueiable + /// Set up the necessary preamble definitions in `func` to access the global value /// identified by `index`. /// /// The index space covers both imported globals and globals defined by the module. /// - /// Return the global valueiable reference that should be used to access the global and the + /// Return the global value reference that should be used to access the global and the /// WebAssembly type of the global. fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue; diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index d3dc64496a..5fec99afb3 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -4,7 +4,7 @@ //! The code of theses helper function is straightforward since it is only about reading metadata //! about linear memories, tables, globals, etc. and storing them for later use. //! -//! The special case of the initialize expressions for table elements offsets or global valueiables +//! The special case of the initialize expressions for table elements offsets or global values //! is handled, according to the semantics of WebAssembly, to only specific expressions that are //! interpreted on the fly. use cretonne_codegen::ir::{self, AbiParam, Signature}; diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index 372cd297b7..beba6e666f 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -134,7 +134,7 @@ pub struct TranslationState { pub control_stack: Vec, pub reachable: bool, - // Map of global valueiables that have already been created by `FuncEnvironment::make_global`. + // Map of global values that have already been created by `FuncEnvironment::make_global`. globals: HashMap, // Map of heaps that have been created by `FuncEnvironment::make_heap`. @@ -272,7 +272,7 @@ impl TranslationState { /// Methods for handling entity references. impl TranslationState { - /// Get the `GlobalValue` reference that should be used to access the global valueiable `index`. + /// Get the `GlobalValue` reference that should be used to access the global value `index`. /// Create the reference if necessary. /// Also return the WebAssembly type of the global. pub fn get_global( diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 9257870264..3c16d0897e 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -7,7 +7,7 @@ use wasmparser; pub type FunctionIndex = usize; /// Index of a table (imported or defined) inside the WebAssembly module. pub type TableIndex = usize; -/// Index of a global valueiable (imported or defined) inside the WebAssembly module. +/// Index of a global value (imported or defined) inside the WebAssembly module. pub type GlobalIndex = usize; /// Index of a linear memory (imported or defined) inside the WebAssembly module. pub type MemoryIndex = usize; From 38ab82bcc0af0c16f2b164ac872c85f599b91944 Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Fri, 15 Jun 2018 15:35:53 -0400 Subject: [PATCH 1854/3084] Made changes for review --- cranelift/docs/compare-llvm.rst | 2 +- cranelift/docs/langref.rst | 19 ++++++++----------- lib/codegen/src/legalizer/heap.rs | 16 +++------------- lib/wasm/src/environ/spec.rs | 2 +- 4 files changed, 13 insertions(+), 26 deletions(-) diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst index 6afc5ed419..abc47caaa3 100644 --- a/cranelift/docs/compare-llvm.rst +++ b/cranelift/docs/compare-llvm.rst @@ -110,7 +110,7 @@ Program structure In LLVM IR, the largest representable unit is the *module* which corresponds more or less to a C translation unit. It is a collection of functions and -global values that may contain references to external symbols too. +global variables that may contain references to external symbols too. In Cretonne IR, the largest representable unit is the *function*. This is so that functions can easily be compiled in parallel without worrying about diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 3736565f94..7630066550 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -554,21 +554,18 @@ stack overflow checks in the prologue. the stack pointer has reached or exceeded the limit, generate a trap with a ``stk_ovf`` code. - The global value must be accessible and naturally aligned for a - pointer-sized value. - Setting `stack_limit` is an alternative way to detect stack overflow, when using a calling convention that doesn't perform stack probes. -Global variables ----------------- +Global values +------------- -A *global value* is an :term:`accessible` object in memory whose address is -not known at compile time. The address is computed at runtime by -:inst:`global_value`, possibly using information provided by the linker via -relocations. There are multiple kinds of global values using different -methods for determining their address. Cretonne does not track the type or even -the size of global values, they are just pointers to non-stack memory. +A *global value* is an object whose value is not known at compile time. The +value is computed at runtime by :inst:`global_value`, possibly using +information provided by the linker via relocations. There are multiple +kinds of global values using different methods for determining their value. +Cretonne does not track the type of a global value, for they are just +values stored in non-stack memory. When Cretonne is generating code for a virtual machine environment, globals can be used to access data structures in the VM's runtime. This requires functions diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs index ab68976d57..3655d7fa5a 100644 --- a/lib/codegen/src/legalizer/heap.rs +++ b/lib/codegen/src/legalizer/heap.rs @@ -6,7 +6,7 @@ use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::condcodes::IntCC; -use ir::{self, InstBuilder, MemFlags}; +use ir::{self, InstBuilder}; use isa::TargetIsa; /// Expand a `heap_addr` instruction according to the definition of the heap. @@ -57,12 +57,7 @@ fn dynamic_addr( pos.use_srcloc(inst); // Start with the bounds check. Trap if `offset + size > bound`. - let bound_addr = pos.ins().global_value(addr_ty, bound_gv); - let mut mflags = MemFlags::new(); - // The bound variable is requied to be accessible and aligned. - mflags.set_notrap(); - mflags.set_aligned(); - let bound = pos.ins().load(offset_ty, mflags, bound_addr, 0); + let bound = pos.ins().global_value(addr_ty, bound_gv); let oob; if size == 1 { @@ -163,12 +158,7 @@ fn offset_addr( match pos.func.heaps[heap].base { ir::HeapBase::ReservedReg => unimplemented!(), ir::HeapBase::GlobalValue(base_gv) => { - let base_addr = pos.ins().global_value(addr_ty, base_gv); - let mut mflags = MemFlags::new(); - // The base address variable is requied to be accessible and aligned. - mflags.set_notrap(); - mflags.set_aligned(); - let base = pos.ins().load(addr_ty, mflags, base_addr, 0); + let base = pos.ins().global_value(addr_ty, base_gv); pos.func.dfg.replace(inst).iadd(base, offset); } } diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 4d97a34cc5..38ed9e6f1b 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -93,7 +93,7 @@ pub trait FuncEnvironment { /// /// The index space covers both imported globals and globals defined by the module. /// - /// Return the global value reference that should be used to access the global and the + /// Return the global variable reference that should be used to access the global and the /// WebAssembly type of the global. fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue; From f97ad59991e313262f45c7055a01f4d3e21bc871 Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Fri, 15 Jun 2018 18:42:33 -0400 Subject: [PATCH 1855/3084] Fix breaking change --- lib/codegen/src/legalizer/heap.rs | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs index 3655d7fa5a..ab68976d57 100644 --- a/lib/codegen/src/legalizer/heap.rs +++ b/lib/codegen/src/legalizer/heap.rs @@ -6,7 +6,7 @@ use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::condcodes::IntCC; -use ir::{self, InstBuilder}; +use ir::{self, InstBuilder, MemFlags}; use isa::TargetIsa; /// Expand a `heap_addr` instruction according to the definition of the heap. @@ -57,7 +57,12 @@ fn dynamic_addr( pos.use_srcloc(inst); // Start with the bounds check. Trap if `offset + size > bound`. - let bound = pos.ins().global_value(addr_ty, bound_gv); + let bound_addr = pos.ins().global_value(addr_ty, bound_gv); + let mut mflags = MemFlags::new(); + // The bound variable is requied to be accessible and aligned. + mflags.set_notrap(); + mflags.set_aligned(); + let bound = pos.ins().load(offset_ty, mflags, bound_addr, 0); let oob; if size == 1 { @@ -158,7 +163,12 @@ fn offset_addr( match pos.func.heaps[heap].base { ir::HeapBase::ReservedReg => unimplemented!(), ir::HeapBase::GlobalValue(base_gv) => { - let base = pos.ins().global_value(addr_ty, base_gv); + let base_addr = pos.ins().global_value(addr_ty, base_gv); + let mut mflags = MemFlags::new(); + // The base address variable is requied to be accessible and aligned. + mflags.set_notrap(); + mflags.set_aligned(); + let base = pos.ins().load(addr_ty, mflags, base_addr, 0); pos.func.dfg.replace(inst).iadd(base, offset); } } From b3c3ca331ba0766098d56fa2876b39b2ced5e41b Mon Sep 17 00:00:00 2001 From: Lachlan Sneff Date: Sat, 16 Jun 2018 10:31:52 -0400 Subject: [PATCH 1856/3084] Removed implicit indirection when computing heap base. (#363) Fix expected legalized heap_addr --- .../filetests/isa/x86/legalize-memory.cton | 6 ++---- lib/codegen/src/legalizer/heap.rs | 17 +++-------------- 2 files changed, 5 insertions(+), 18 deletions(-) diff --git a/cranelift/filetests/isa/x86/legalize-memory.cton b/cranelift/filetests/isa/x86/legalize-memory.cton index 0a35c841ba..ce6d194507 100644 --- a/cranelift/filetests/isa/x86/legalize-memory.cton +++ b/cranelift/filetests/isa/x86/legalize-memory.cton @@ -53,8 +53,7 @@ ebb0(v0: i32, v999: i64): ; Boundscheck should be eliminated. ; Checks here are assuming that no pipehole opts fold the load offsets. ; nextln: $(xoff=$V) = uextend.i64 v0 - ; nextln: $(haddr=$V) = iadd_imm v999, 64 - ; nextln: $(hbase=$V) = load.i64 notrap aligned $haddr + ; nextln: $(hbase=$V) = iadd_imm v999, 64 ; nextln: v1 = iadd $hbase, $xoff v2 = load.f32 v1+16 ; nextln: v2 = load.f32 v1+16 @@ -101,8 +100,7 @@ ebb0(v0: i32, v999: i64): ; check: $ok: ; Checks here are assuming that no pipehole opts fold the load offsets. ; nextln: $(xoff=$V) = uextend.i64 v0 - ; nextln: $(haddr=$V) = iadd_imm.i64 v999, 64 - ; nextln: $(hbase=$V) = load.i64 notrap aligned $haddr + ; nextln: $(hbase=$V) = iadd_imm.i64 v999, 64 ; nextln: v1 = iadd $hbase, $xoff v2 = load.f32 v1+0x7fff_ffff ; nextln: v2 = load.f32 v1+0x7fff_ffff diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs index ab68976d57..fdf3fdb094 100644 --- a/lib/codegen/src/legalizer/heap.rs +++ b/lib/codegen/src/legalizer/heap.rs @@ -6,7 +6,7 @@ use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::condcodes::IntCC; -use ir::{self, InstBuilder, MemFlags}; +use ir::{self, InstBuilder}; use isa::TargetIsa; /// Expand a `heap_addr` instruction according to the definition of the heap. @@ -57,13 +57,7 @@ fn dynamic_addr( pos.use_srcloc(inst); // Start with the bounds check. Trap if `offset + size > bound`. - let bound_addr = pos.ins().global_value(addr_ty, bound_gv); - let mut mflags = MemFlags::new(); - // The bound variable is requied to be accessible and aligned. - mflags.set_notrap(); - mflags.set_aligned(); - let bound = pos.ins().load(offset_ty, mflags, bound_addr, 0); - + let bound = pos.ins().global_value(addr_ty, bound_gv); let oob; if size == 1 { // `offset > bound - 1` is the same as `offset >= bound`. @@ -163,12 +157,7 @@ fn offset_addr( match pos.func.heaps[heap].base { ir::HeapBase::ReservedReg => unimplemented!(), ir::HeapBase::GlobalValue(base_gv) => { - let base_addr = pos.ins().global_value(addr_ty, base_gv); - let mut mflags = MemFlags::new(); - // The base address variable is requied to be accessible and aligned. - mflags.set_notrap(); - mflags.set_aligned(); - let base = pos.ins().load(addr_ty, mflags, base_addr, 0); + let base = pos.ins().global_value(addr_ty, base_gv); pos.func.dfg.replace(inst).iadd(base, offset); } } From 0349270b58b6868a211c82832b313e1851daeaf6 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sat, 16 Jun 2018 07:50:27 -0700 Subject: [PATCH 1857/3084] Bump version to 0.12.0 --- cranelift/Cargo.toml | 22 +++++++++++----------- cranelift/publish-all.sh | 2 +- lib/codegen/Cargo.toml | 4 ++-- lib/entity/Cargo.toml | 2 +- lib/faerie/Cargo.toml | 6 +++--- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/module/Cargo.toml | 6 +++--- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/simplejit/Cargo.toml | 8 ++++---- lib/umbrella/Cargo.toml | 6 +++--- lib/wasm/Cargo.toml | 6 +++--- 13 files changed, 40 insertions(+), 40 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index ee41dbac43..d74d5cd2c2 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.11.0" +version = "0.12.0" description = "Binaries for testing the Cretonne libraries" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -14,16 +14,16 @@ path = "src/cton-util.rs" [dependencies] cfg-if = "0.1" -cretonne-codegen = { path = "lib/codegen", version = "0.11.0" } -cretonne-reader = { path = "lib/reader", version = "0.11.0" } -cretonne-frontend = { path = "lib/frontend", version = "0.11.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.11.0", optional = true } -cretonne-native = { path = "lib/native", version = "0.11.0" } -cretonne-filetests = { path = "lib/filetests", version = "0.11.0" } -cretonne-module = { path = "lib/module", version = "0.11.0" } -cretonne-faerie = { path = "lib/faerie", version = "0.11.0" } -cretonne-simplejit = { path = "lib/simplejit", version = "0.11.0" } -cretonne = { path = "lib/umbrella", version = "0.11.0" } +cretonne-codegen = { path = "lib/codegen", version = "0.12.0" } +cretonne-reader = { path = "lib/reader", version = "0.12.0" } +cretonne-frontend = { path = "lib/frontend", version = "0.12.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.12.0", optional = true } +cretonne-native = { path = "lib/native", version = "0.12.0" } +cretonne-filetests = { path = "lib/filetests", version = "0.12.0" } +cretonne-module = { path = "lib/module", version = "0.12.0" } +cretonne-faerie = { path = "lib/faerie", version = "0.12.0" } +cretonne-simplejit = { path = "lib/simplejit", version = "0.12.0" } +cretonne = { path = "lib/umbrella", version = "0.12.0" } filecheck = "0.3.0" docopt = "1" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index e536b5d710..f9fc306059 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ topdir=$(dirname "$0") cd "$topdir" # All the cretonne-* crates have the same version number -version="0.11.0" +version="0.12.0" # Update all of the Cargo.toml files. # diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index ca08a6c7e3..c68710f358 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-codegen" -version = "0.11.0" +version = "0.12.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"] build = "build.rs" [dependencies] -cretonne-entity = { path = "../entity", version = "0.11.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.12.0", default-features = false } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } hashmap_core = { version = "0.1.7", optional = true } diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index aee7848a08..83516f86ec 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-entity" -version = "0.11.0" +version = "0.12.0" description = "Data structures using entity references as mapping keys" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index 0df347f635..b5cd90f295 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-faerie" -version = "0.11.0" +version = "0.12.0" authors = ["The Cretonne Project Developers"] description = "Emit Cretonne output to native object files with Faerie" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.11.0" } -cretonne-module = { path = "../module", version = "0.11.0" } +cretonne-codegen = { path = "../codegen", version = "0.12.0" } +cretonne-module = { path = "../module", version = "0.12.0" } faerie = "0.4.2" goblin = "0.0.15" failure = "0.1.1" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index ae1fc74359..756e81f13a 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.11.0" +version = "0.12.0" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" publish = false [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.11.0" } -cretonne-reader = { path = "../reader", version = "0.11.0" } +cretonne-codegen = { path = "../codegen", version = "0.12.0" } +cretonne-reader = { path = "../reader", version = "0.12.0" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index f369d6da93..a46c9f2f05 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.11.0" +version = "0.12.0" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } [features] default = ["std"] diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 1e527bd3b9..41e2621a91 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-module" -version = "0.11.0" +version = "0.12.0" authors = ["The Cretonne Project Developers"] description = "Support for linking functions and data with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } -cretonne-entity = { path = "../entity", version = "0.11.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.12.0", default-features = false } hashmap_core = { version = "0.1.7", optional = true } failure = "0.1.1" diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 69ec2f9b67..765c83f62d 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.11.0" +version = "0.12.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -8,7 +8,7 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } target-lexicon = { version = "0.0.2", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 1ad05ee299..c0e6f40402 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.11.0" +version = "0.12.0" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.11.0" } +cretonne-codegen = { path = "../codegen", version = "0.12.0" } target-lexicon = "0.0.2" [badges] diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index 2452ccae55..ed04e90a07 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-simplejit" -version = "0.11.0" +version = "0.12.0" authors = ["The Cretonne Project Developers"] description = "A simple JIT library backed by Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,9 +9,9 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } -cretonne-module = { path = "../module", version = "0.11.0", default-features = false } -cretonne-native = { path = "../native", version = "0.11.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } +cretonne-module = { path = "../module", version = "0.12.0", default-features = false } +cretonne-native = { path = "../native", version = "0.12.0", default-features = false } region = "0.3.0" libc = { version = "0.2.40", default-features = false } errno = "0.2.3" diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml index 274ba20d07..8220c75861 100644 --- a/lib/umbrella/Cargo.toml +++ b/lib/umbrella/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.11.0" +version = "0.12.0" description = "Umbrella for commonly-used cretonne crates" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -10,8 +10,8 @@ readme = "README.md" keywords = ["compile", "compiler", "jit"] [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.11.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.12.0", default-features = false } [features] default = ["std"] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index ccfc7ac0ba..b57b944f68 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.11.0" +version = "0.12.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/cretonne/cretonne" @@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"] [dependencies] wasmparser = { version = "0.17.0", default-features = false } -cretonne-codegen = { path = "../codegen", version = "0.11.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.11.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.12.0", default-features = false } hashmap_core = { version = "0.1.7", optional = true } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } From 47a96641f92e681b759f866140a81fdab976c9b8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 19 Jun 2018 14:12:52 -0700 Subject: [PATCH 1858/3084] Rename the `memflags` operand name from `Flags` to `MemFlags`. This reduces confusion with other kinds of flags. --- lib/codegen/meta/base/instructions.py | 46 +++++++++++++-------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/lib/codegen/meta/base/instructions.py b/lib/codegen/meta/base/instructions.py index 5efd344c6f..4572c4914b 100644 --- a/lib/codegen/meta/base/instructions.py +++ b/lib/codegen/meta/base/instructions.py @@ -245,7 +245,7 @@ Offset = Operand('Offset', offset32, 'Byte offset from base address') x = Operand('x', Mem, doc='Value to be stored') a = Operand('a', Mem, doc='Value loaded') p = Operand('p', iAddr) -Flags = Operand('Flags', memflags) +MemFlags = Operand('MemFlags', memflags) args = Operand('args', VARIABLE_ARGS, doc='Address arguments') load = Instruction( @@ -255,7 +255,7 @@ load = Instruction( This is a polymorphic instruction that can load any value type which has a memory representation. """, - ins=(Flags, p, Offset), outs=a, can_load=True) + ins=(MemFlags, p, Offset), outs=a, can_load=True) load_complex = Instruction( 'load_complex', r""" @@ -264,7 +264,7 @@ load_complex = Instruction( This is a polymorphic instruction that can load any value type which has a memory representation. """, - ins=(Flags, args, Offset), outs=a, can_load=True) + ins=(MemFlags, args, Offset), outs=a, can_load=True) store = Instruction( 'store', r""" @@ -273,7 +273,7 @@ store = Instruction( This is a polymorphic instruction that can store any value type with a memory representation. """, - ins=(Flags, x, p, Offset), can_store=True) + ins=(MemFlags, x, p, Offset), can_store=True) store_complex = Instruction( 'store_complex', r""" @@ -282,7 +282,7 @@ store_complex = Instruction( This is a polymorphic instruction that can store any value type with a memory representation. """, - ins=(Flags, x, args, Offset), can_store=True) + ins=(MemFlags, x, args, Offset), can_store=True) iExt8 = TypeVar( @@ -297,7 +297,7 @@ uload8 = Instruction( This is equivalent to ``load.i8`` followed by ``uextend``. """, - ins=(Flags, p, Offset), outs=a, can_load=True) + ins=(MemFlags, p, Offset), outs=a, can_load=True) uload8_complex = Instruction( 'uload8_complex', r""" @@ -305,7 +305,7 @@ uload8_complex = Instruction( This is equivalent to ``load.i8`` followed by ``uextend``. """, - ins=(Flags, args, Offset), outs=a, can_load=True) + ins=(MemFlags, args, Offset), outs=a, can_load=True) sload8 = Instruction( 'sload8', r""" @@ -313,7 +313,7 @@ sload8 = Instruction( This is equivalent to ``load.i8`` followed by ``uextend``. """, - ins=(Flags, p, Offset), outs=a, can_load=True) + ins=(MemFlags, p, Offset), outs=a, can_load=True) sload8_complex = Instruction( 'sload8_complex', r""" @@ -321,7 +321,7 @@ sload8_complex = Instruction( This is equivalent to ``load.i8`` followed by ``uextend``. """, - ins=(Flags, args, Offset), outs=a, can_load=True) + ins=(MemFlags, args, Offset), outs=a, can_load=True) istore8 = Instruction( 'istore8', r""" @@ -329,7 +329,7 @@ istore8 = Instruction( This is equivalent to ``ireduce.i8`` followed by ``store.i8``. """, - ins=(Flags, x, p, Offset), can_store=True) + ins=(MemFlags, x, p, Offset), can_store=True) istore8_complex = Instruction( 'istore8_complex', r""" @@ -337,7 +337,7 @@ istore8_complex = Instruction( This is equivalent to ``ireduce.i8`` followed by ``store.i8``. """, - ins=(Flags, x, args, Offset), can_store=True) + ins=(MemFlags, x, args, Offset), can_store=True) iExt16 = TypeVar( 'iExt16', 'An integer type with more than 16 bits', @@ -351,7 +351,7 @@ uload16 = Instruction( This is equivalent to ``load.i16`` followed by ``uextend``. """, - ins=(Flags, p, Offset), outs=a, can_load=True) + ins=(MemFlags, p, Offset), outs=a, can_load=True) uload16_complex = Instruction( 'uload16_complex', r""" @@ -359,7 +359,7 @@ uload16_complex = Instruction( This is equivalent to ``load.i16`` followed by ``uextend``. """, - ins=(Flags, args, Offset), outs=a, can_load=True) + ins=(MemFlags, args, Offset), outs=a, can_load=True) sload16 = Instruction( 'sload16', r""" @@ -367,7 +367,7 @@ sload16 = Instruction( This is equivalent to ``load.i16`` followed by ``uextend``. """, - ins=(Flags, p, Offset), outs=a, can_load=True) + ins=(MemFlags, p, Offset), outs=a, can_load=True) sload16_complex = Instruction( 'sload16_complex', r""" @@ -375,7 +375,7 @@ sload16_complex = Instruction( This is equivalent to ``load.i16`` followed by ``uextend``. """, - ins=(Flags, args, Offset), outs=a, can_load=True) + ins=(MemFlags, args, Offset), outs=a, can_load=True) istore16 = Instruction( 'istore16', r""" @@ -383,7 +383,7 @@ istore16 = Instruction( This is equivalent to ``ireduce.i16`` followed by ``store.i16``. """, - ins=(Flags, x, p, Offset), can_store=True) + ins=(MemFlags, x, p, Offset), can_store=True) istore16_complex = Instruction( 'istore16_complex', r""" @@ -391,7 +391,7 @@ istore16_complex = Instruction( This is equivalent to ``ireduce.i16`` followed by ``store.i16``. """, - ins=(Flags, x, args, Offset), can_store=True) + ins=(MemFlags, x, args, Offset), can_store=True) iExt32 = TypeVar( 'iExt32', 'An integer type with more than 32 bits', @@ -405,7 +405,7 @@ uload32 = Instruction( This is equivalent to ``load.i32`` followed by ``uextend``. """, - ins=(Flags, p, Offset), outs=a, can_load=True) + ins=(MemFlags, p, Offset), outs=a, can_load=True) uload32_complex = Instruction( 'uload32_complex', r""" @@ -413,7 +413,7 @@ uload32_complex = Instruction( This is equivalent to ``load.i32`` followed by ``uextend``. """, - ins=(Flags, args, Offset), outs=a, can_load=True) + ins=(MemFlags, args, Offset), outs=a, can_load=True) sload32 = Instruction( 'sload32', r""" @@ -421,7 +421,7 @@ sload32 = Instruction( This is equivalent to ``load.i32`` followed by ``uextend``. """, - ins=(Flags, p, Offset), outs=a, can_load=True) + ins=(MemFlags, p, Offset), outs=a, can_load=True) sload32_complex = Instruction( 'sload32_complex', r""" @@ -429,7 +429,7 @@ sload32_complex = Instruction( This is equivalent to ``load.i32`` followed by ``uextend``. """, - ins=(Flags, args, Offset), outs=a, can_load=True) + ins=(MemFlags, args, Offset), outs=a, can_load=True) istore32 = Instruction( 'istore32', r""" @@ -437,7 +437,7 @@ istore32 = Instruction( This is equivalent to ``ireduce.i32`` followed by ``store.i32``. """, - ins=(Flags, x, p, Offset), can_store=True) + ins=(MemFlags, x, p, Offset), can_store=True) istore32_complex = Instruction( 'istore32_complex', r""" @@ -445,7 +445,7 @@ istore32_complex = Instruction( This is equivalent to ``ireduce.i32`` followed by ``store.i32``. """, - ins=(Flags, x, args, Offset), can_store=True) + ins=(MemFlags, x, args, Offset), can_store=True) x = Operand('x', Mem, doc='Value to be stored') a = Operand('a', Mem, doc='Value loaded') From 2b04099604dcc5c8240f1d8e9bc8a62644eeff91 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 19 Jun 2018 14:16:42 -0700 Subject: [PATCH 1859/3084] Fix a typo in a comment. --- lib/entity/src/list.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/entity/src/list.rs b/lib/entity/src/list.rs index 8a0aecbdfd..e48a839759 100644 --- a/lib/entity/src/list.rs +++ b/lib/entity/src/list.rs @@ -34,7 +34,7 @@ use EntityRef; /// function they belong to. *Cloning an entity list does not allocate new memory for the clone*. /// It creates an alias of the same memory. /// -/// Entity lists can also be hashed and compared for equality, but those operations just panic if, +/// Entity lists can also be hashed and compared for equality, but those operations just panic if /// they're ever actually called, because it's not possible to compare the contents of the list /// without the pool reference. /// From 603bb4ac0225089c871c873996eb1cb68d897622 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 21 Jun 2018 15:45:24 -0700 Subject: [PATCH 1860/3084] Remove an obsolete comment. --- lib/reader/src/parser.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 7e85a8cc00..72dc466335 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -466,7 +466,7 @@ impl<'a> Parser<'a> { } } - // Match and consume a value reference, direct or vtable. + // Match and consume a value reference. fn match_value(&mut self, err_msg: &str) -> ParseResult { if let Some(Token::Value(v)) = self.token() { self.consume(); From d209137149cc422ca2faca2f6081d55e687212c8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 21 Jun 2018 15:50:29 -0700 Subject: [PATCH 1861/3084] Rename `TargetIsa`'s `emit_function` to `emit_function_to_memory`. This reflects the fact that it takes a concrete `MemoryCodeSink` rather than a `CodeSink` trait object. --- lib/codegen/src/context.rs | 2 +- lib/codegen/src/isa/arm32/mod.rs | 2 +- lib/codegen/src/isa/arm64/mod.rs | 2 +- lib/codegen/src/isa/mod.rs | 6 +++--- lib/codegen/src/isa/riscv/mod.rs | 2 +- lib/codegen/src/isa/x86/mod.rs | 2 +- 6 files changed, 8 insertions(+), 8 deletions(-) diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index 6e8c54051b..fa8b761ad4 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -168,7 +168,7 @@ impl Context { traps: &mut TrapSink, ) { let _tt = timing::binemit(); - isa.emit_function(&self.func, &mut MemoryCodeSink::new(mem, relocs, traps)); + isa.emit_function_to_memory(&self.func, &mut MemoryCodeSink::new(mem, relocs, traps)); } /// Run the verifier on the function. diff --git a/lib/codegen/src/isa/arm32/mod.rs b/lib/codegen/src/isa/arm32/mod.rs index bb0214b3ec..3606c18676 100644 --- a/lib/codegen/src/isa/arm32/mod.rs +++ b/lib/codegen/src/isa/arm32/mod.rs @@ -121,7 +121,7 @@ impl TargetIsa for Isa { binemit::emit_inst(func, inst, divert, sink) } - fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { + fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { emit_function(func, binemit::emit_inst, sink) } } diff --git a/lib/codegen/src/isa/arm64/mod.rs b/lib/codegen/src/isa/arm64/mod.rs index 50a43b13fc..96e62a9e37 100644 --- a/lib/codegen/src/isa/arm64/mod.rs +++ b/lib/codegen/src/isa/arm64/mod.rs @@ -108,7 +108,7 @@ impl TargetIsa for Isa { binemit::emit_inst(func, inst, divert, sink) } - fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { + fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { emit_function(func, binemit::emit_inst, sink) } } diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs index d85f0c3e10..56ff2b5f4a 100644 --- a/lib/codegen/src/isa/mod.rs +++ b/lib/codegen/src/isa/mod.rs @@ -303,8 +303,8 @@ pub trait TargetIsa: fmt::Display { /// Emit binary machine code for a single instruction into the `sink` trait object. /// - /// Note that this will call `put*` methods on the trait object via its vtable which is not the - /// fastest way of emitting code. + /// Note that this will call `put*` methods on the `sink` trait object via its vtable which + /// is not the fastest way of emitting code. fn emit_inst( &self, func: &ir::Function, @@ -316,5 +316,5 @@ pub trait TargetIsa: fmt::Display { /// Emit a whole function into memory. /// /// This is more performant than calling `emit_inst` for each instruction. - fn emit_function(&self, func: &ir::Function, sink: &mut binemit::MemoryCodeSink); + fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut binemit::MemoryCodeSink); } diff --git a/lib/codegen/src/isa/riscv/mod.rs b/lib/codegen/src/isa/riscv/mod.rs index aff23390e5..1a09f89bc1 100644 --- a/lib/codegen/src/isa/riscv/mod.rs +++ b/lib/codegen/src/isa/riscv/mod.rs @@ -115,7 +115,7 @@ impl TargetIsa for Isa { binemit::emit_inst(func, inst, divert, sink) } - fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { + fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { emit_function(func, binemit::emit_inst, sink) } } diff --git a/lib/codegen/src/isa/x86/mod.rs b/lib/codegen/src/isa/x86/mod.rs index 2681e044d6..0546c648cd 100644 --- a/lib/codegen/src/isa/x86/mod.rs +++ b/lib/codegen/src/isa/x86/mod.rs @@ -125,7 +125,7 @@ impl TargetIsa for Isa { binemit::emit_inst(func, inst, divert, sink) } - fn emit_function(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { + fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut MemoryCodeSink) { emit_function(func, binemit::emit_inst, sink) } From 593e2bae6c8e086415e05c7b1305fbd33f588626 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 21 Jun 2018 16:18:34 -0700 Subject: [PATCH 1862/3084] Add timers for branch relaxation and instruction shrinking. --- lib/codegen/src/binemit/relaxation.rs | 3 +++ lib/codegen/src/binemit/shrink.rs | 3 +++ lib/codegen/src/timing.rs | 2 ++ 3 files changed, 8 insertions(+) diff --git a/lib/codegen/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs index 7cfb6491ca..dfebc5cb9d 100644 --- a/lib/codegen/src/binemit/relaxation.rs +++ b/lib/codegen/src/binemit/relaxation.rs @@ -32,12 +32,15 @@ use cursor::{Cursor, FuncCursor}; use ir::{Function, InstructionData, Opcode}; use isa::{EncInfo, TargetIsa}; use iterators::IteratorExtras; +use timing; use CodegenResult; /// Relax branches and compute the final layout of EBB headers in `func`. /// /// Fill in the `func.offsets` table so the function is ready for binary emission. pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> CodegenResult { + let _tt = timing::relax_branches(); + let encinfo = isa.encoding_info(); // Clear all offsets so we can recognize EBBs that haven't been visited yet. diff --git a/lib/codegen/src/binemit/shrink.rs b/lib/codegen/src/binemit/shrink.rs index 813110ac97..3c7d3390ef 100644 --- a/lib/codegen/src/binemit/shrink.rs +++ b/lib/codegen/src/binemit/shrink.rs @@ -8,9 +8,12 @@ use ir::Function; use isa::TargetIsa; use regalloc::RegDiversions; +use timing; /// Pick the smallest valid encodings for instructions. pub fn shrink_instructions(func: &mut Function, isa: &TargetIsa) { + let _tt = timing::shrink_instructions(); + let encinfo = isa.encoding_info(); let mut divert = RegDiversions::new(); diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs index 34195c8e32..961c6fea30 100644 --- a/lib/codegen/src/timing.rs +++ b/lib/codegen/src/timing.rs @@ -71,6 +71,8 @@ define_passes!{ ra_coloring: "RA coloring", prologue_epilogue: "Prologue/epilogue insertion", + shrink_instructions: "Instruction encoding shrinking", + relax_branches: "Branch relaxation", binemit: "Binary machine code emission", layout_renumber: "Layout full renumbering", From 979162522eef0bab3b3bd2d840b3c796bd579986 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 21 Jun 2018 16:24:33 -0700 Subject: [PATCH 1863/3084] Keep a comment in sync with the code. --- lib/codegen/src/binemit/relaxation.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/codegen/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs index dfebc5cb9d..a0d1ad3591 100644 --- a/lib/codegen/src/binemit/relaxation.rs +++ b/lib/codegen/src/binemit/relaxation.rs @@ -128,9 +128,9 @@ fn fallthroughs(func: &mut Function) { } } -/// Relax the branch instruction at `pos` so it can cover the range `offset - dest_offset`. +/// Relax the branch instruction at `cur` so it can cover the range `offset - dest_offset`. /// -/// Return the size of the replacement instructions up to and including the location where `pos` is +/// Return the size of the replacement instructions up to and including the location where `cur` is /// left. fn relax_branch( cur: &mut FuncCursor, From fbd637e14283b0569799b07b4e86e54038deec17 Mon Sep 17 00:00:00 2001 From: Bruce Mitchener Date: Sat, 23 Jun 2018 02:13:00 +0700 Subject: [PATCH 1864/3084] Update to raw-cpuid 4.0. (#369) --- lib/native/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 765c83f62d..18d92b43d4 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -12,7 +12,7 @@ cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = target-lexicon = { version = "0.0.2", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] -raw-cpuid = "3.1.0" +raw-cpuid = "4" [features] default = ["std"] From 7d2b44289c30fac3db394524b4753342795f979f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 26 Jun 2018 06:15:21 -0700 Subject: [PATCH 1865/3084] Implement stack_addr, stack_load, stack_store for x86-64. (#370) --- cranelift/filetests/isa/x86/stack-addr64.cton | 45 ++++++++++++ .../filetests/isa/x86/stack-load-store64.cton | 21 ++++++ lib/codegen/meta/base/legalize.py | 4 ++ lib/codegen/meta/isa/x86/encodings.py | 9 +++ lib/codegen/meta/isa/x86/recipes.py | 31 ++++++++ lib/codegen/src/legalizer/mod.rs | 70 ++++++++++++++++++- 6 files changed, 179 insertions(+), 1 deletion(-) create mode 100644 cranelift/filetests/isa/x86/stack-addr64.cton create mode 100644 cranelift/filetests/isa/x86/stack-load-store64.cton diff --git a/cranelift/filetests/isa/x86/stack-addr64.cton b/cranelift/filetests/isa/x86/stack-addr64.cton new file mode 100644 index 0000000000..ded05221db --- /dev/null +++ b/cranelift/filetests/isa/x86/stack-addr64.cton @@ -0,0 +1,45 @@ +; binary emission of stack address instructions on x86-64. +test binemit +set opt_level=fastest +target x86_64 haswell + +; The binary encodings can be verified with the command: +; +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/stack-addr64.cton | llvm-mc -show-encoding -triple=x86_64 +; + +function %stack_addr() { + ss0 = incoming_arg 8, offset 0 + ss1 = incoming_arg 1024, offset -1024 + ss2 = incoming_arg 1024, offset -2048 + ss3 = incoming_arg 8, offset -2056 + ss4 = explicit_slot 8, offset 0 + ss5 = explicit_slot 8, offset 1024 + +ebb0: +[-,%rcx] v0 = stack_addr.i64 ss0 ; bin: 48 8d 8c 24 00000808 +[-,%rcx] v1 = stack_addr.i64 ss1 ; bin: 48 8d 8c 24 00000408 +[-,%rcx] v2 = stack_addr.i64 ss2 ; bin: 48 8d 8c 24 00000008 +[-,%rcx] v3 = stack_addr.i64 ss3 ; bin: 48 8d 8c 24 00000000 +[-,%rcx] v4 = stack_addr.i64 ss4 ; bin: 48 8d 8c 24 00000808 +[-,%rcx] v5 = stack_addr.i64 ss5 ; bin: 48 8d 8c 24 00000c08 + +[-,%rcx] v20 = stack_addr.i64 ss4+1 ; bin: 48 8d 8c 24 00000809 +[-,%rcx] v21 = stack_addr.i64 ss4+2 ; bin: 48 8d 8c 24 0000080a +[-,%rcx] v22 = stack_addr.i64 ss4+2048 ; bin: 48 8d 8c 24 00001008 +[-,%rcx] v23 = stack_addr.i64 ss4-4096 ; bin: 48 8d 8c 24 fffff808 + +[-,%r8] v50 = stack_addr.i64 ss0 ; bin: 4c 8d 84 24 00000808 +[-,%r8] v51 = stack_addr.i64 ss1 ; bin: 4c 8d 84 24 00000408 +[-,%r8] v52 = stack_addr.i64 ss2 ; bin: 4c 8d 84 24 00000008 +[-,%r8] v53 = stack_addr.i64 ss3 ; bin: 4c 8d 84 24 00000000 +[-,%r8] v54 = stack_addr.i64 ss4 ; bin: 4c 8d 84 24 00000808 +[-,%r8] v55 = stack_addr.i64 ss5 ; bin: 4c 8d 84 24 00000c08 + +[-,%r8] v70 = stack_addr.i64 ss4+1 ; bin: 4c 8d 84 24 00000809 +[-,%r8] v71 = stack_addr.i64 ss4+2 ; bin: 4c 8d 84 24 0000080a +[-,%r8] v72 = stack_addr.i64 ss4+2048 ; bin: 4c 8d 84 24 00001008 +[-,%r8] v73 = stack_addr.i64 ss4-4096 ; bin: 4c 8d 84 24 fffff808 + + return +} diff --git a/cranelift/filetests/isa/x86/stack-load-store64.cton b/cranelift/filetests/isa/x86/stack-load-store64.cton new file mode 100644 index 0000000000..c1854e623a --- /dev/null +++ b/cranelift/filetests/isa/x86/stack-load-store64.cton @@ -0,0 +1,21 @@ +; legalization of stack load and store instructions on x86-64. +test legalizer +set opt_level=fastest +target x86_64 haswell + +function %stack_load_and_store() { + ss0 = explicit_slot 8, offset 0 + +ebb0: + v0 = stack_load.i64 ss0 + +; check: v1 = stack_addr.i64 ss0 +; check: v0 = load.i64 notrap aligned v1 + + stack_store.i64 v0, ss0 + +; check: v2 = stack_addr.i64 ss0 +; check: store notrap aligned v0, v2 + + return +} diff --git a/lib/codegen/meta/base/legalize.py b/lib/codegen/meta/base/legalize.py index de196cc645..f575418f8d 100644 --- a/lib/codegen/meta/base/legalize.py +++ b/lib/codegen/meta/base/legalize.py @@ -80,6 +80,10 @@ expand.custom_legalize(insts.select, 'expand_select') expand.custom_legalize(insts.f32const, 'expand_fconst') expand.custom_legalize(insts.f64const, 'expand_fconst') +# Custom expansions for stack memory accesses. +expand.custom_legalize(insts.stack_load, 'expand_stack_load') +expand.custom_legalize(insts.stack_store, 'expand_stack_store') + x = Var('x') y = Var('y') a = Var('a') diff --git a/lib/codegen/meta/isa/x86/encodings.py b/lib/codegen/meta/isa/x86/encodings.py index 62dbf1dcf7..28e2f73740 100644 --- a/lib/codegen/meta/isa/x86/encodings.py +++ b/lib/codegen/meta/isa/x86/encodings.py @@ -436,6 +436,15 @@ X86_64.enc(base.globalsym_addr.i64, *r.pcrel_gvaddr8.rex(0x8d, w=1), X86_64.enc(base.globalsym_addr.i64, *r.got_gvaddr8.rex(0x8b, w=1), isap=is_pic) +# +# Stack addresses. +# +# TODO: Add encoding rules for stack_load and stack_store, so that they +# don't get legalized to stack_addr + load/store. +# +X86_32.enc(base.stack_addr.i32, *r.spaddr4_id(0x8d)) +X86_64.enc(base.stack_addr.i64, *r.spaddr8_id.rex(0x8d, w=1)) + # # Call/return # diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py index 2d6ac898a4..75c666a8af 100644 --- a/lib/codegen/meta/isa/x86/recipes.py +++ b/lib/codegen/meta/isa/x86/recipes.py @@ -17,6 +17,7 @@ from base.formats import Jump, Branch, BranchInt, BranchFloat from base.formats import Ternary, FuncAddr, UnaryGlobalValue from base.formats import RegMove, RegSpill, RegFill, CopySpecial from base.formats import LoadComplex, StoreComplex +from base.formats import StackLoad from .registers import GPR, ABCD, FPR, GPR_DEREF_SAFE, GPR_ZERO_DEREF_SAFE from .registers import GPR8, FPR8, GPR8_DEREF_SAFE, GPR8_ZERO_DEREF_SAFE, FLAG from .registers import StackGPR32, StackFPR32 @@ -751,6 +752,36 @@ got_gvaddr8 = TailRecipe( sink.put4(0); ''') +# +# Stack addresses. +# +# TODO: Alternative forms for 8-bit immediates, when applicable. +# + +spaddr4_id = TailRecipe( + 'spaddr4_id', StackLoad, size=6, ins=(), outs=GPR, + emit=''' + let sp = StackRef::sp(stack_slot, &func.stack_slots); + let base = stk_base(sp.base); + PUT_OP(bits, rex2(out_reg0, base), sink); + modrm_sib_disp8(out_reg0, sink); + sib_noindex(base, sink); + let imm : i32 = offset.into(); + sink.put4(sp.offset.checked_add(imm).unwrap() as u32); + ''') + +spaddr8_id = TailRecipe( + 'spaddr8_id', StackLoad, size=6, ins=(), outs=GPR, + emit=''' + let sp = StackRef::sp(stack_slot, &func.stack_slots); + let base = stk_base(sp.base); + PUT_OP(bits, rex2(base, out_reg0), sink); + modrm_sib_disp32(out_reg0, sink); + sib_noindex(base, sink); + let imm : i32 = offset.into(); + sink.put4(sp.offset.checked_add(imm).unwrap() as u32); + ''') + # # Store recipes. diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs index f099ca7447..f6ca7efeec 100644 --- a/lib/codegen/src/legalizer/mod.rs +++ b/lib/codegen/src/legalizer/mod.rs @@ -16,7 +16,7 @@ use bitset::BitSet; use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; -use ir::{self, InstBuilder}; +use ir::{self, InstBuilder, MemFlags}; use isa::TargetIsa; use timing; @@ -269,3 +269,71 @@ fn expand_fconst( }; pos.func.dfg.replace(inst).bitcast(ty, ival); } + +/// Expand illegal `stack_load` instructions. +fn expand_stack_load( + inst: ir::Inst, + func: &mut ir::Function, + _cfg: &mut ControlFlowGraph, + isa: &TargetIsa, +) { + let ty = func.dfg.value_type(func.dfg.first_result(inst)); + let addr_ty = isa.pointer_type(); + + let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); + + let (stack_slot, offset) = match pos.func.dfg[inst] { + ir::InstructionData::StackLoad { + opcode: _opcode, + stack_slot, + offset, + } => (stack_slot, offset), + _ => panic!( + "Expected stack_load: {}", + pos.func.dfg.display_inst(inst, None) + ), + }; + + let addr = pos.ins().stack_addr(addr_ty, stack_slot, offset); + + let mut mflags = MemFlags::new(); + // Stack slots are required to be accessible and aligned. + mflags.set_notrap(); + mflags.set_aligned(); + pos.func.dfg.replace(inst).load(ty, mflags, addr, 0); +} + +/// Expand illegal `stack_store` instructions. +fn expand_stack_store( + inst: ir::Inst, + func: &mut ir::Function, + _cfg: &mut ControlFlowGraph, + isa: &TargetIsa, +) { + let addr_ty = isa.pointer_type(); + + let mut pos = FuncCursor::new(func).at_inst(inst); + pos.use_srcloc(inst); + + let (val, stack_slot, offset) = match pos.func.dfg[inst] { + ir::InstructionData::StackStore { + opcode: _opcode, + arg, + stack_slot, + offset, + } => (arg, stack_slot, offset), + _ => panic!( + "Expected stack_store: {}", + pos.func.dfg.display_inst(inst, None) + ), + }; + + let addr = pos.ins().stack_addr(addr_ty, stack_slot, offset); + + let mut mflags = MemFlags::new(); + // Stack slots are required to be accessible and aligned. + mflags.set_notrap(); + mflags.set_aligned(); + pos.func.dfg.replace(inst).store(mflags, val, addr, 0); +} From cc94adca3bc9c8f11a64af28429f09ae19b9a0cd Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 26 Jun 2018 13:06:16 -0700 Subject: [PATCH 1866/3084] Update to the rustfmt in rust 1.27, which is now stable. (#377) --- lib/codegen/src/bitset.rs | 9 +++++++-- lib/codegen/src/context.rs | 4 +++- lib/codegen/src/dce.rs | 12 +++++++++--- lib/codegen/src/ir/function.rs | 6 ++++-- lib/codegen/src/ir/libcall.rs | 6 ++++-- lib/codegen/src/ir/mod.rs | 4 +++- lib/codegen/src/isa/x86/abi.rs | 6 ++++-- lib/codegen/src/legalizer/boundary.rs | 9 ++++++--- lib/codegen/src/licm.rs | 12 +++++++++--- lib/codegen/src/nan_canonicalization.rs | 14 ++++++++++---- lib/codegen/src/regalloc/coloring.rs | 3 ++- lib/codegen/src/simple_gvn.rs | 12 +++++++++--- lib/codegen/src/verifier/mod.rs | 6 ++++-- lib/faerie/src/backend.rs | 6 ++++-- lib/frontend/src/frontend.rs | 13 ++++++++----- lib/frontend/src/ssa.rs | 7 +++++-- lib/module/src/lib.rs | 5 +++-- lib/reader/src/parser.rs | 11 ++++++----- lib/simplejit/src/backend.rs | 6 ++++-- lib/umbrella/src/lib.rs | 6 ++++-- lib/wasm/src/environ/dummy.rs | 5 +++-- lib/wasm/src/environ/spec.rs | 5 +++-- lib/wasm/src/lib.rs | 11 +++++++---- lib/wasm/src/module_translator.rs | 9 +++++---- lib/wasm/src/sections_translator.rs | 12 ++++++++---- 25 files changed, 134 insertions(+), 65 deletions(-) diff --git a/lib/codegen/src/bitset.rs b/lib/codegen/src/bitset.rs index 0029b0c42c..10d624d8a2 100644 --- a/lib/codegen/src/bitset.rs +++ b/lib/codegen/src/bitset.rs @@ -107,8 +107,13 @@ mod tests { let s4 = BitSet::(4 | 8 | 256 | 1024); assert!( - !s4.contains(0) && !s4.contains(1) && !s4.contains(4) && !s4.contains(5) - && !s4.contains(6) && !s4.contains(7) && !s4.contains(9) + !s4.contains(0) + && !s4.contains(1) + && !s4.contains(4) + && !s4.contains(5) + && !s4.contains(6) + && !s4.contains(7) + && !s4.contains(9) && !s4.contains(11) ); assert!(s4.contains(2) && s4.contains(3) && s4.contains(8) && s4.contains(10)); diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index fa8b761ad4..955fe6111d 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -9,7 +9,9 @@ //! contexts concurrently. Typically, you would have one context per compilation thread and only a //! single ISA instance. -use binemit::{relax_branches, shrink_instructions, CodeOffset, MemoryCodeSink, RelocSink, TrapSink}; +use binemit::{ + relax_branches, shrink_instructions, CodeOffset, MemoryCodeSink, RelocSink, TrapSink, +}; use dce::do_dce; use dominator_tree::DominatorTree; use flowgraph::ControlFlowGraph; diff --git a/lib/codegen/src/dce.rs b/lib/codegen/src/dce.rs index 5373948e88..2c5d5e2a32 100644 --- a/lib/codegen/src/dce.rs +++ b/lib/codegen/src/dce.rs @@ -13,8 +13,13 @@ use timing; /// Test whether the given opcode is unsafe to even consider for DCE. fn trivially_unsafe_for_dce(opcode: Opcode) -> bool { - opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || opcode.is_return() - || opcode.can_trap() || opcode.other_side_effects() || opcode.can_store() + opcode.is_call() + || opcode.is_branch() + || opcode.is_terminator() + || opcode.is_return() + || opcode.can_trap() + || opcode.other_side_effects() + || opcode.can_store() } /// Preserve instructions with used result values. @@ -50,7 +55,8 @@ pub fn do_dce(func: &mut Function, domtree: &mut DominatorTree) { { let data = &pos.func.dfg[inst]; let opcode = data.opcode(); - if trivially_unsafe_for_dce(opcode) || is_load_with_defined_trapping(opcode, &data) + if trivially_unsafe_for_dce(opcode) + || is_load_with_defined_trapping(opcode, &data) || any_inst_results_used(inst, &live, &pos.func.dfg) { for arg in pos.func.dfg.inst_args(inst) { diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs index e060c94029..9c6647ba0c 100644 --- a/lib/codegen/src/ir/function.rs +++ b/lib/codegen/src/ir/function.rs @@ -7,8 +7,10 @@ use binemit::CodeOffset; use entity::{EntityMap, PrimaryMap}; use ir; use ir::{DataFlowGraph, ExternalName, Layout, Signature}; -use ir::{Ebb, ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, JumpTable, - JumpTableData, SigRef, StackSlot, StackSlotData}; +use ir::{ + Ebb, ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, JumpTable, + JumpTableData, SigRef, StackSlot, StackSlotData, +}; use ir::{EbbOffsets, InstEncodings, JumpTables, SourceLocs, StackSlots, ValueLocations}; use isa::{EncInfo, Encoding, Legalize, TargetIsa}; use settings::CallConv; diff --git a/lib/codegen/src/ir/libcall.rs b/lib/codegen/src/ir/libcall.rs index 26440d4235..43ac22e568 100644 --- a/lib/codegen/src/ir/libcall.rs +++ b/lib/codegen/src/ir/libcall.rs @@ -1,7 +1,9 @@ //! Naming well-known routines in the runtime library. -use ir::{types, AbiParam, ArgumentPurpose, ExtFuncData, ExternalName, FuncRef, Function, Inst, - Opcode, Signature, Type}; +use ir::{ + types, AbiParam, ArgumentPurpose, ExtFuncData, ExternalName, FuncRef, Function, Inst, Opcode, + Signature, Type, +}; use isa::{RegUnit, TargetIsa}; use settings::CallConv; use std::fmt; diff --git a/lib/codegen/src/ir/mod.rs b/lib/codegen/src/ir/mod.rs index 71e56ec5a0..1133f90af9 100644 --- a/lib/codegen/src/ir/mod.rs +++ b/lib/codegen/src/ir/mod.rs @@ -24,7 +24,9 @@ mod valueloc; pub use ir::builder::{InsertBuilder, InstBuilder, InstBuilderBase, InstInserterBase}; pub use ir::dfg::{DataFlowGraph, ValueDef}; -pub use ir::entities::{Ebb, FuncRef, GlobalValue, Heap, Inst, JumpTable, SigRef, StackSlot, Value}; +pub use ir::entities::{ + Ebb, FuncRef, GlobalValue, Heap, Inst, JumpTable, SigRef, StackSlot, Value, +}; pub use ir::extfunc::{AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData, Signature}; pub use ir::extname::ExternalName; pub use ir::function::Function; diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs index c74572ecc7..5b9639e622 100644 --- a/lib/codegen/src/isa/x86/abi.rs +++ b/lib/codegen/src/isa/x86/abi.rs @@ -6,8 +6,10 @@ use cursor::{Cursor, CursorPosition, EncCursor}; use ir; use ir::immediates::Imm64; use ir::stackslot::{StackOffset, StackSize}; -use ir::{get_probestack_funcref, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, - InstBuilder, ValueLoc}; +use ir::{ + get_probestack_funcref, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, InstBuilder, + ValueLoc, +}; use isa::{RegClass, RegUnit, TargetIsa}; use regalloc::RegisterSet; use result::CodegenResult; diff --git a/lib/codegen/src/legalizer/boundary.rs b/lib/codegen/src/legalizer/boundary.rs index 4a865f2965..dfc4629566 100644 --- a/lib/codegen/src/legalizer/boundary.rs +++ b/lib/codegen/src/legalizer/boundary.rs @@ -21,8 +21,10 @@ use abi::{legalize_abi_value, ValueConversion}; use cursor::{Cursor, FuncCursor}; use flowgraph::ControlFlowGraph; use ir::instructions::CallInfo; -use ir::{AbiParam, ArgumentLoc, ArgumentPurpose, DataFlowGraph, Ebb, Function, Inst, InstBuilder, - SigRef, Signature, Type, Value, ValueLoc}; +use ir::{ + AbiParam, ArgumentLoc, ArgumentPurpose, DataFlowGraph, Ebb, Function, Inst, InstBuilder, + SigRef, Signature, Type, Value, ValueLoc, +}; use isa::TargetIsa; use legalizer::split::{isplit, vsplit}; use std::vec::Vec; @@ -553,7 +555,8 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph .iter() .rev() .take_while(|&rt| { - rt.purpose == ArgumentPurpose::Link || rt.purpose == ArgumentPurpose::StructReturn + rt.purpose == ArgumentPurpose::Link + || rt.purpose == ArgumentPurpose::StructReturn || rt.purpose == ArgumentPurpose::VMContext }) .count(); diff --git a/lib/codegen/src/licm.rs b/lib/codegen/src/licm.rs index 5771e89fb1..8f32b640ac 100644 --- a/lib/codegen/src/licm.rs +++ b/lib/codegen/src/licm.rs @@ -132,9 +132,15 @@ fn change_branch_jump_destination(inst: Inst, new_ebb: Ebb, func: &mut Function) /// Test whether the given opcode is unsafe to even consider for LICM. fn trivially_unsafe_for_licm(opcode: Opcode) -> bool { - opcode.can_load() || opcode.can_store() || opcode.is_call() || opcode.is_branch() - || opcode.is_terminator() || opcode.is_return() || opcode.can_trap() - || opcode.other_side_effects() || opcode.writes_cpu_flags() + opcode.can_load() + || opcode.can_store() + || opcode.is_call() + || opcode.is_branch() + || opcode.is_terminator() + || opcode.is_return() + || opcode.can_trap() + || opcode.other_side_effects() + || opcode.writes_cpu_flags() } /// Test whether the given instruction is loop-invariant. diff --git a/lib/codegen/src/nan_canonicalization.rs b/lib/codegen/src/nan_canonicalization.rs index 0db8c12056..0f11c39fe8 100644 --- a/lib/codegen/src/nan_canonicalization.rs +++ b/lib/codegen/src/nan_canonicalization.rs @@ -33,12 +33,18 @@ pub fn do_nan_canonicalization(func: &mut Function) { fn is_fp_arith(pos: &mut FuncCursor, inst: Inst) -> bool { match pos.func.dfg[inst] { InstructionData::Unary { opcode, .. } => { - opcode == Opcode::Ceil || opcode == Opcode::Floor || opcode == Opcode::Nearest - || opcode == Opcode::Sqrt || opcode == Opcode::Trunc + opcode == Opcode::Ceil + || opcode == Opcode::Floor + || opcode == Opcode::Nearest + || opcode == Opcode::Sqrt + || opcode == Opcode::Trunc } InstructionData::Binary { opcode, .. } => { - opcode == Opcode::Fadd || opcode == Opcode::Fdiv || opcode == Opcode::Fmax - || opcode == Opcode::Fmin || opcode == Opcode::Fmul + opcode == Opcode::Fadd + || opcode == Opcode::Fdiv + || opcode == Opcode::Fmax + || opcode == Opcode::Fmin + || opcode == Opcode::Fmul || opcode == Opcode::Fsub } InstructionData::Ternary { opcode, .. } => opcode == Opcode::Fma, diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs index 33ce0ff3b4..8bb29c62cc 100644 --- a/lib/codegen/src/regalloc/coloring.rs +++ b/lib/codegen/src/regalloc/coloring.rs @@ -869,7 +869,8 @@ impl<'a> Context<'a> { // not actually constrained by the instruction. We just want it out of the way. let toprc2 = self.reginfo.toprc(rci); let reg2 = self.divert.reg(lv.value, &self.cur.func.locations); - if rc.contains(reg2) && self.solver.can_add_var(lv.value, toprc2, reg2) + if rc.contains(reg2) + && self.solver.can_add_var(lv.value, toprc2, reg2) && !self.is_live_on_outgoing_edge(lv.value) { self.solver.add_through_var(lv.value, toprc2, reg2); diff --git a/lib/codegen/src/simple_gvn.rs b/lib/codegen/src/simple_gvn.rs index 141199b3d9..cc8e7c37c5 100644 --- a/lib/codegen/src/simple_gvn.rs +++ b/lib/codegen/src/simple_gvn.rs @@ -9,9 +9,15 @@ use timing; /// Test whether the given opcode is unsafe to even consider for GVN. fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { - opcode.is_call() || opcode.is_branch() || opcode.is_terminator() || opcode.is_return() - || opcode.can_trap() || opcode.other_side_effects() || opcode.can_store() - || opcode.can_load() || opcode.writes_cpu_flags() + opcode.is_call() + || opcode.is_branch() + || opcode.is_terminator() + || opcode.is_return() + || opcode.can_trap() + || opcode.other_side_effects() + || opcode.can_store() + || opcode.can_load() + || opcode.writes_cpu_flags() } /// Perform simple GVN on `func`. diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs index b25d770c3f..4bd027a58f 100644 --- a/lib/codegen/src/verifier/mod.rs +++ b/lib/codegen/src/verifier/mod.rs @@ -63,8 +63,10 @@ use flowgraph::ControlFlowGraph; use ir; use ir::entities::AnyEntity; use ir::instructions::{BranchInfo, CallInfo, InstructionFormat, ResolvedConstraint}; -use ir::{types, ArgumentLoc, Ebb, FuncRef, Function, GlobalValue, Inst, JumpTable, Opcode, SigRef, - StackSlot, StackSlotKind, Type, Value, ValueDef, ValueList, ValueLoc}; +use ir::{ + types, ArgumentLoc, Ebb, FuncRef, Function, GlobalValue, Inst, JumpTable, Opcode, SigRef, + StackSlot, StackSlotKind, Type, Value, ValueDef, ValueList, ValueLoc, +}; use isa::TargetIsa; use iterators::IteratorExtras; use settings::{Flags, FlagsOrIsa}; diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index 0bb82efa9e..4126ed7582 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -4,8 +4,10 @@ use container; use cretonne_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSink}; use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::{self, binemit, ir}; -use cretonne_module::{Backend, DataContext, DataDescription, Init, Linkage, ModuleError, - ModuleNamespace, ModuleResult}; +use cretonne_module::{ + Backend, DataContext, DataDescription, Init, Linkage, ModuleError, ModuleNamespace, + ModuleResult, +}; use faerie; use failure::Error; use std::fs::File; diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index 89d933d2ae..eabadad15f 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -3,10 +3,11 @@ use cretonne_codegen::cursor::{Cursor, FuncCursor}; use cretonne_codegen::entity::{EntityMap, EntityRef, EntitySet}; use cretonne_codegen::ir; use cretonne_codegen::ir::function::DisplayFunction; -use cretonne_codegen::ir::{DataFlowGraph, Ebb, ExtFuncData, FuncRef, Function, GlobalValue, - GlobalValueData, Heap, HeapData, Inst, InstBuilderBase, - InstructionData, JumpTable, JumpTableData, SigRef, Signature, - StackSlot, StackSlotData, Type, Value}; +use cretonne_codegen::ir::{ + DataFlowGraph, Ebb, ExtFuncData, FuncRef, Function, GlobalValue, GlobalValueData, Heap, + HeapData, Inst, InstBuilderBase, InstructionData, JumpTable, JumpTableData, SigRef, Signature, + StackSlot, StackSlotData, Type, Value, +}; use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::packed_option::PackedOption; use ssa::{Block, SSABuilder, SideEffects}; @@ -273,7 +274,9 @@ where pub fn switch_to_block(&mut self, ebb: Ebb) { // First we check that the previous block has been filled. debug_assert!( - self.position.is_default() || self.is_unreachable() || self.is_pristine() + self.position.is_default() + || self.is_unreachable() + || self.is_pristine() || self.is_filled(), "you have to fill your block before switching" ); diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index 59a031f6d1..a7f4833fc3 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -172,8 +172,11 @@ where /// Tests whether an `SSABuilder` is in a cleared state. pub fn is_empty(&self) -> bool { - self.variables.is_empty() && self.blocks.is_empty() && self.ebb_headers.is_empty() - && self.calls.is_empty() && self.results.is_empty() + self.variables.is_empty() + && self.blocks.is_empty() + && self.ebb_headers.is_empty() + && self.calls.is_empty() + && self.results.is_empty() && self.side_effects.is_empty() } } diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs index 85e04caaff..4d251413d3 100644 --- a/lib/module/src/lib.rs +++ b/lib/module/src/lib.rs @@ -33,8 +33,9 @@ mod module; pub use backend::Backend; pub use data_context::{DataContext, DataDescription, Init, Writability}; -pub use module::{DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleError, ModuleNamespace, - ModuleResult}; +pub use module::{ + DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleError, ModuleNamespace, ModuleResult, +}; /// This replaces `std` in builds with `core`. #[cfg(not(feature = "std"))] diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 72dc466335..a8bea8b2b0 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -6,11 +6,12 @@ use cretonne_codegen::ir::entities::AnyEntity; use cretonne_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32}; use cretonne_codegen::ir::instructions::{InstructionData, InstructionFormat, VariableArgs}; use cretonne_codegen::ir::types::VOID; -use cretonne_codegen::ir::{AbiParam, ArgumentExtension, ArgumentLoc, Ebb, ExtFuncData, - ExternalName, FuncRef, Function, GlobalValue, GlobalValueData, Heap, - HeapBase, HeapData, HeapStyle, JumpTable, JumpTableData, MemFlags, - Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind, - Type, Value, ValueLoc}; +use cretonne_codegen::ir::{ + AbiParam, ArgumentExtension, ArgumentLoc, Ebb, ExtFuncData, ExternalName, FuncRef, Function, + GlobalValue, GlobalValueData, Heap, HeapBase, HeapData, HeapStyle, JumpTable, JumpTableData, + MemFlags, Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind, Type, Value, + ValueLoc, +}; use cretonne_codegen::isa::{self, Encoding, RegUnit, TargetIsa}; use cretonne_codegen::packed_option::ReservedValue; use cretonne_codegen::settings::CallConv; diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs index 5886b7ad76..23bed09bdc 100644 --- a/lib/simplejit/src/backend.rs +++ b/lib/simplejit/src/backend.rs @@ -3,8 +3,10 @@ use cretonne_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSink}; use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::{self, ir, settings}; -use cretonne_module::{Backend, DataContext, DataDescription, Init, Linkage, ModuleNamespace, - ModuleResult, Writability}; +use cretonne_module::{ + Backend, DataContext, DataDescription, Init, Linkage, ModuleNamespace, ModuleResult, + Writability, +}; use cretonne_native; use libc; use memory::Memory; diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs index 1048b4d780..c586227df2 100644 --- a/lib/umbrella/src/lib.rs +++ b/lib/umbrella/src/lib.rs @@ -24,8 +24,10 @@ pub mod prelude { pub use codegen::ir::condcodes::{FloatCC, IntCC}; pub use codegen::ir::immediates::{Ieee32, Ieee64, Imm64}; pub use codegen::ir::types; - pub use codegen::ir::{AbiParam, Ebb, ExtFuncData, GlobalValueData, InstBuilder, JumpTableData, - MemFlags, Signature, StackSlotData, StackSlotKind, TrapCode, Type, Value}; + pub use codegen::ir::{ + AbiParam, Ebb, ExtFuncData, GlobalValueData, InstBuilder, JumpTableData, MemFlags, + Signature, StackSlotData, StackSlotKind, TrapCode, Type, Value, + }; pub use codegen::isa; pub use codegen::settings::{self, CallConv, Configurable}; diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 0bd7fc1a94..6ccefe3213 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -9,8 +9,9 @@ use func_translator::FuncTranslator; use std::string::String; use std::vec::Vec; use target_lexicon::Triple; -use translation_utils::{FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, - Table, TableIndex}; +use translation_utils::{ + FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex, +}; use wasmparser; /// Compute a `ir::ExternalName` for a given wasm function index. diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 38ed9e6f1b..09a8dd5371 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -5,8 +5,9 @@ use cretonne_codegen::ir::{self, InstBuilder}; use cretonne_codegen::settings::Flags; use std::vec::Vec; use target_lexicon::Triple; -use translation_utils::{FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, - Table, TableIndex}; +use translation_utils::{ + FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex, +}; use wasmparser::BinaryReaderError; /// The value of a WebAssembly global value. diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 35256c924c..5d4fd24a37 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -42,12 +42,15 @@ mod sections_translator; mod state; mod translation_utils; -pub use environ::{DummyEnvironment, FuncEnvironment, GlobalValue, ModuleEnvironment, WasmError, - WasmResult}; +pub use environ::{ + DummyEnvironment, FuncEnvironment, GlobalValue, ModuleEnvironment, WasmError, WasmResult, +}; pub use func_translator::FuncTranslator; pub use module_translator::translate_module; -pub use translation_utils::{FunctionIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, - SignatureIndex, Table, TableIndex}; +pub use translation_utils::{ + FunctionIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex, Table, + TableIndex, +}; #[cfg(not(feature = "std"))] mod std { diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index 835a26fb3f..ffeafc2378 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -2,10 +2,11 @@ //! to deal with each part of it. use cretonne_codegen::timing; use environ::{ModuleEnvironment, WasmError, WasmResult}; -use sections_translator::{parse_data_section, parse_elements_section, parse_export_section, - parse_function_section, parse_function_signatures, parse_global_section, - parse_import_section, parse_memory_section, parse_start_section, - parse_table_section}; +use sections_translator::{ + parse_data_section, parse_elements_section, parse_export_section, parse_function_section, + parse_function_signatures, parse_global_section, parse_import_section, parse_memory_section, + parse_start_section, parse_table_section, +}; use wasmparser::{Parser, ParserInput, ParserState, SectionCode, WasmDecoder}; /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IR diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 5fec99afb3..112bd3784e 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -11,11 +11,15 @@ use cretonne_codegen::ir::{self, AbiParam, Signature}; use environ::{ModuleEnvironment, WasmError, WasmResult}; use std::str::from_utf8; use std::vec::Vec; -use translation_utils::{type_to_type, FunctionIndex, Global, GlobalIndex, GlobalInit, Memory, - MemoryIndex, SignatureIndex, Table, TableElementType, TableIndex}; +use translation_utils::{ + type_to_type, FunctionIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, + SignatureIndex, Table, TableElementType, TableIndex, +}; use wasmparser; -use wasmparser::{ExternalKind, FuncType, ImportSectionEntryType, MemoryType, Operator, Parser, - ParserState, WasmDecoder}; +use wasmparser::{ + ExternalKind, FuncType, ImportSectionEntryType, MemoryType, Operator, Parser, ParserState, + WasmDecoder, +}; /// Reads the Type Section of the wasm module and returns the corresponding function signatures. pub fn parse_function_signatures( From c5aad1eb5f13934b1bf2075941faf4ff998b9985 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 28 Jun 2018 10:15:10 -0700 Subject: [PATCH 1867/3084] Add support for macho relocations. (#378) This requires splitting X86PCRel4 into two separate relocations, to distinguish the case where the instruction is a call, as Mach-O uses a different relocation in that case. This also makes it explicit that only x86-64 relocations are supported currently. --- cranelift/filetests/isa/x86/binary32.cton | 2 +- cranelift/filetests/isa/x86/binary64-pic.cton | 4 +- cranelift/filetests/isa/x86/binary64.cton | 2 +- lib/codegen/meta/isa/x86/recipes.py | 4 +- lib/codegen/src/binemit/mod.rs | 9 ++- lib/faerie/src/backend.rs | 19 +++--- lib/faerie/src/container.rs | 58 ++++++++++++++----- lib/simplejit/src/backend.rs | 11 ++-- 8 files changed, 70 insertions(+), 39 deletions(-) diff --git a/cranelift/filetests/isa/x86/binary32.cton b/cranelift/filetests/isa/x86/binary32.cton index b953d663f6..a6476e3e81 100644 --- a/cranelift/filetests/isa/x86/binary32.cton +++ b/cranelift/filetests/isa/x86/binary32.cton @@ -352,7 +352,7 @@ ebb0: [-,%rsi] v351 = bint.i32 v301 ; bin: 0f b6 f2 ; asm: call foo - call fn0() ; bin: stk_ovf e8 PCRel4(%foo-4) 00000000 + call fn0() ; bin: stk_ovf e8 CallPCRel4(%foo-4) 00000000 ; asm: movl $0, %ecx [-,%rcx] v400 = func_addr.i32 fn0 ; bin: b9 Abs4(%foo) 00000000 diff --git a/cranelift/filetests/isa/x86/binary64-pic.cton b/cranelift/filetests/isa/x86/binary64-pic.cton index 3678906135..2fa66a820f 100644 --- a/cranelift/filetests/isa/x86/binary64-pic.cton +++ b/cranelift/filetests/isa/x86/binary64-pic.cton @@ -30,7 +30,7 @@ ebb0: ; Colocated functions. ; asm: call foo - call fn1() ; bin: stk_ovf e8 PCRel4(%bar-4) 00000000 + call fn1() ; bin: stk_ovf e8 CallPCRel4(%bar-4) 00000000 ; asm: lea 0x0(%rip), %rax [-,%rax] v0 = func_addr.i64 fn1 ; bin: 48 8d 05 PCRel4(%bar-4) 00000000 @@ -49,7 +49,7 @@ ebb0: ; Non-colocated functions. ; asm: call foo@PLT - call fn0() ; bin: stk_ovf e8 PLTRel4(%foo-4) 00000000 + call fn0() ; bin: stk_ovf e8 CallPLTRel4(%foo-4) 00000000 ; asm: mov 0x0(%rip), %rax [-,%rax] v100 = func_addr.i64 fn0 ; bin: 48 8b 05 GOTPCRel4(%foo-4) 00000000 diff --git a/cranelift/filetests/isa/x86/binary64.cton b/cranelift/filetests/isa/x86/binary64.cton index 0b0deb4143..c6ba888579 100644 --- a/cranelift/filetests/isa/x86/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.cton @@ -485,7 +485,7 @@ ebb0: ; Colocated functions. ; asm: call bar - call fn1() ; bin: stk_ovf e8 PCRel4(%bar-4) 00000000 + call fn1() ; bin: stk_ovf e8 CallPCRel4(%bar-4) 00000000 ; asm: lea 0x0(%rip), %rcx [-,%rcx] v400 = func_addr.i64 fn1 ; bin: 48 8d 0d PCRel4(%bar-4) 00000000 diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py index 75c666a8af..66c6326a3a 100644 --- a/lib/codegen/meta/isa/x86/recipes.py +++ b/lib/codegen/meta/isa/x86/recipes.py @@ -1388,7 +1388,7 @@ call_id = TailRecipe( PUT_OP(bits, BASE_REX, sink); // The addend adjusts for the difference between the end of the // instruction and the beginning of the immediate field. - sink.reloc_external(Reloc::X86PCRel4, + sink.reloc_external(Reloc::X86CallPCRel4, &func.dfg.ext_funcs[func_ref].name, -4); sink.put4(0); @@ -1399,7 +1399,7 @@ call_plt_id = TailRecipe( emit=''' sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); PUT_OP(bits, BASE_REX, sink); - sink.reloc_external(Reloc::X86PLTRel4, + sink.reloc_external(Reloc::X86CallPLTRel4, &func.dfg.ext_funcs[func_ref].name, -4); sink.put4(0); diff --git a/lib/codegen/src/binemit/mod.rs b/lib/codegen/src/binemit/mod.rs index 4a95ed4eb7..9e4036b2ed 100644 --- a/lib/codegen/src/binemit/mod.rs +++ b/lib/codegen/src/binemit/mod.rs @@ -33,10 +33,12 @@ pub enum Reloc { Abs8, /// x86 PC-relative 4-byte X86PCRel4, + /// x86 call to PC-relative 4-byte + X86CallPCRel4, + /// x86 call to PLT-relative 4-byte + X86CallPLTRel4, /// x86 GOT PC-relative 4-byte X86GOTPCRel4, - /// x86 PLT-relative 4-byte - X86PLTRel4, /// Arm32 call target Arm32Call, /// Arm64 call target @@ -53,8 +55,9 @@ impl fmt::Display for Reloc { Reloc::Abs4 => write!(f, "Abs4"), Reloc::Abs8 => write!(f, "Abs8"), Reloc::X86PCRel4 => write!(f, "PCRel4"), + Reloc::X86CallPCRel4 => write!(f, "CallPCRel4"), + Reloc::X86CallPLTRel4 => write!(f, "CallPLTRel4"), Reloc::X86GOTPCRel4 => write!(f, "GOTPCRel4"), - Reloc::X86PLTRel4 => write!(f, "PLTRel4"), Reloc::Arm32Call | Reloc::Arm64Call | Reloc::RiscvCall => write!(f, "Call"), } } diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index 4126ed7582..73d4b3330d 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -11,7 +11,7 @@ use cretonne_module::{ use faerie; use failure::Error; use std::fs::File; -use target_lexicon::BinaryFormat; +use target_lexicon::Triple; use traps::{FaerieTrapManifest, FaerieTrapSink}; #[derive(Debug)] @@ -29,7 +29,6 @@ pub enum FaerieTrapCollection { pub struct FaerieBuilder { isa: Box, name: String, - format: BinaryFormat, collect_traps: FaerieTrapCollection, libcall_names: Box String>, } @@ -51,7 +50,6 @@ impl FaerieBuilder { pub fn new( isa: Box, name: String, - format: BinaryFormat, collect_traps: FaerieTrapCollection, libcall_names: Box String>, ) -> ModuleResult { @@ -63,7 +61,6 @@ impl FaerieBuilder { Ok(Self { isa, name, - format, collect_traps, libcall_names, }) @@ -91,7 +88,6 @@ impl FaerieBuilder { pub struct FaerieBackend { isa: Box, artifact: faerie::Artifact, - format: BinaryFormat, trap_manifest: Option, libcall_names: Box String>, } @@ -120,7 +116,6 @@ impl Backend for FaerieBackend { Self { artifact: faerie::Artifact::new(builder.isa.triple().clone(), builder.name), isa: builder.isa, - format: builder.format, trap_manifest: match builder.collect_traps { FaerieTrapCollection::Enabled => Some(FaerieTrapManifest::new()), FaerieTrapCollection::Disabled => None, @@ -158,7 +153,7 @@ impl Backend for FaerieBackend { // Non-lexical lifetimes would obviate the braces here. { let mut reloc_sink = FaerieRelocSink { - format: self.format, + triple: self.isa.triple().clone(), artifact: &mut self.artifact, name, namespace, @@ -348,7 +343,7 @@ fn translate_data_linkage(linkage: Linkage, writable: bool) -> faerie::Decl { } struct FaerieRelocSink<'a> { - format: BinaryFormat, + triple: Triple, artifact: &'a mut faerie::Artifact, name: &'a str, namespace: &'a ModuleNamespace<'a, FaerieBackend>, @@ -384,9 +379,11 @@ impl<'a> RelocSink for FaerieRelocSink<'a> { } _ => panic!("invalid ExternalName {}", name), }; - let addend_i32 = addend as i32; - debug_assert!(i64::from(addend_i32) == addend); - let raw_reloc = container::raw_relocation(reloc, self.format); + let (raw_reloc, raw_addend) = container::raw_relocation(reloc, &self.triple); + // TODO: Handle overflow. + let final_addend = addend + raw_addend; + let addend_i32 = final_addend as i32; + debug_assert!(i64::from(addend_i32) == final_addend); self.artifact .link_with( faerie::Link { diff --git a/lib/faerie/src/container.rs b/lib/faerie/src/container.rs index 117482020c..72766c1f0f 100644 --- a/lib/faerie/src/container.rs +++ b/lib/faerie/src/container.rs @@ -1,7 +1,7 @@ //! Utilities for working with Faerie container formats. use cretonne_codegen::binemit::Reloc; -use target_lexicon::BinaryFormat; +use target_lexicon::{Architecture, BinaryFormat, Triple}; /// An object file format. #[derive(Copy, Clone, Debug, Eq, PartialEq)] @@ -13,23 +13,53 @@ pub enum Format { } /// Translate from a Cretonne `Reloc` to a raw object-file-format-specific -/// relocation code. -pub fn raw_relocation(reloc: Reloc, format: BinaryFormat) -> u32 { - match format { +/// relocation code and relocation-implied addend. +pub fn raw_relocation(reloc: Reloc, triple: &Triple) -> (u32, i64) { + match triple.binary_format { BinaryFormat::Elf => { use goblin::elf; - match reloc { - Reloc::Abs4 => elf::reloc::R_X86_64_32, - Reloc::Abs8 => elf::reloc::R_X86_64_64, - Reloc::X86PCRel4 => elf::reloc::R_X86_64_PC32, - // TODO: Get Cretonne to tell us when we can use - // R_X86_64_GOTPCRELX/R_X86_64_REX_GOTPCRELX. - Reloc::X86GOTPCRel4 => elf::reloc::R_X86_64_GOTPCREL, - Reloc::X86PLTRel4 => elf::reloc::R_X86_64_PLT32, - _ => unimplemented!(), + ( + match triple.architecture { + Architecture::X86_64 => { + match reloc { + Reloc::Abs4 => elf::reloc::R_X86_64_32, + Reloc::Abs8 => elf::reloc::R_X86_64_64, + Reloc::X86PCRel4 | Reloc::X86CallPCRel4 => elf::reloc::R_X86_64_PC32, + // TODO: Get Cretonne to tell us when we can use + // R_X86_64_GOTPCRELX/R_X86_64_REX_GOTPCRELX. + Reloc::X86CallPLTRel4 => elf::reloc::R_X86_64_PLT32, + Reloc::X86GOTPCRel4 => elf::reloc::R_X86_64_GOTPCREL, + _ => unimplemented!(), + } + } + _ => unimplemented!("unsupported architecture: {}", triple), + }, + // Most ELF relocations do not include an implicit addend. + 0, + ) + } + BinaryFormat::Macho => { + use goblin::mach; + match triple.architecture { + Architecture::X86_64 => { + match reloc { + Reloc::Abs8 => (u32::from(mach::relocation::R_ABS), 0), + // Mach-O doesn't need us to distinguish between PC-relative calls + // and PLT calls, but it does need us to distinguish between calls + // and non-calls. And, it includes the 4-byte addend implicitly. + Reloc::X86PCRel4 => (u32::from(mach::relocation::X86_64_RELOC_SIGNED), 4), + Reloc::X86CallPCRel4 | Reloc::X86CallPLTRel4 => { + (u32::from(mach::relocation::X86_64_RELOC_BRANCH), 4) + } + Reloc::X86GOTPCRel4 => { + (u32::from(mach::relocation::X86_64_RELOC_GOT_LOAD), 4) + } + _ => unimplemented!("unsupported mach-o reloc: {}", reloc), + } + } + _ => unimplemented!("unsupported architecture: {}", triple), } } - BinaryFormat::Macho => unimplemented!("macho relocations"), _ => unimplemented!("unsupported format"), } } diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs index 23bed09bdc..29c9c26a32 100644 --- a/lib/simplejit/src/backend.rs +++ b/lib/simplejit/src/backend.rs @@ -271,7 +271,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { write_unaligned(at as *mut u64, what as u64) }; } - Reloc::X86PCRel4 => { + Reloc::X86PCRel4 | Reloc::X86CallPCRel4 => { // TODO: Handle overflow. let pcrel = ((what as isize) - (at as isize)) as i32; #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))] @@ -279,7 +279,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { write_unaligned(at as *mut i32, pcrel) }; } - Reloc::X86GOTPCRel4 | Reloc::X86PLTRel4 => panic!("unexpected PIC relocation"), + Reloc::X86GOTPCRel4 | Reloc::X86CallPLTRel4 => panic!("unexpected PIC relocation"), _ => unimplemented!(), } } @@ -336,9 +336,10 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend { write_unaligned(at as *mut u64, what as u64) }; } - Reloc::X86PCRel4 | Reloc::X86GOTPCRel4 | Reloc::X86PLTRel4 => { - panic!("unexpected text relocation in data") - } + Reloc::X86PCRel4 + | Reloc::X86CallPCRel4 + | Reloc::X86GOTPCRel4 + | Reloc::X86CallPLTRel4 => panic!("unexpected text relocation in data"), _ => unimplemented!(), } } From 7bed3426a76c6f31deff1872299284f20db5493d Mon Sep 17 00:00:00 2001 From: Caroline Cullen Date: Thu, 28 Jun 2018 10:17:27 -0700 Subject: [PATCH 1868/3084] Adds decoration to the verifier errors. (#375) * Adds decoration to the verifier errors. example: function %bad(i32) fast { ebb0(v0: i32): brnz.i32 v0, ebb1 return ^~~~~~ verifier inst1: Internal return not allowed with return_at_end=1 ebb1: trapz.i32 v0, user6 return } Fixes #68 --- lib/codegen/src/print_errors.rs | 51 +++++++++++++++++++++++++++++---- lib/codegen/src/write.rs | 45 +++++++++++++++++++++++++++-- 2 files changed, 88 insertions(+), 8 deletions(-) diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs index 777fde05b8..61e10d7ed1 100644 --- a/lib/codegen/src/print_errors.rs +++ b/lib/codegen/src/print_errors.rs @@ -1,11 +1,15 @@ //! Utility routines for pretty-printing error messages. use ir; +use ir::entities::Inst; +use ir::function::Function; use isa::TargetIsa; use result::CodegenError; +use std::fmt; use std::fmt::Write; use std::string::{String, ToString}; use verifier::VerifierError; +use write::decorate_function; /// Pretty-print a verifier error. pub fn pretty_verifier_error( @@ -13,15 +17,52 @@ pub fn pretty_verifier_error( isa: Option<&TargetIsa>, err: &VerifierError, ) -> String { - let mut msg = err.to_string(); + let mut w = String::new(); + decorate_function( + &mut |w, func, isa, inst, indent| pretty_function_error(w, func, isa, inst, indent, err), + &mut w, + func, + isa, + ).unwrap(); + w +} + +/// Pretty-print a function verifier error. +fn pretty_function_error( + w: &mut Write, + func: &Function, + isa: Option<&TargetIsa>, + cur_inst: Inst, + indent: usize, + err: &VerifierError, +) -> fmt::Result { match err.location { ir::entities::AnyEntity::Inst(inst) => { - write!(msg, "\n{}: {}\n\n", inst, func.dfg.display_inst(inst, isa)).unwrap() + if inst == cur_inst { + write!( + w, + "{1:0$}{2}\n", + indent, + "", + func.dfg.display_inst(cur_inst, isa) + )?; + write!(w, "{1:0$}{2}", indent, "", "^")?; + for _c in cur_inst.to_string().chars() { + write!(w, "~")?; + } + write!(w, "\n\nverifier {}\n\n", err.to_string()) + } else { + write!( + w, + "{1:0$}{2}\n", + indent, + "", + func.dfg.display_inst(cur_inst, isa) + ) + } } - _ => msg.push('\n'), + _ => write!(w, "{}", "\n"), } - write!(msg, "{}", func.display(isa)).unwrap(); - msg } /// Pretty-print a Cretonne error. diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs index 124034b39a..9914c45517 100644 --- a/lib/codegen/src/write.rs +++ b/lib/codegen/src/write.rs @@ -9,9 +9,39 @@ use packed_option::ReservedValue; use std::fmt::{self, Write}; use std::string::String; +fn write_function_plain( + w: &mut Write, + func: &Function, + isa: Option<&TargetIsa>, + inst: Inst, + indent: usize, +) -> fmt::Result { + write_instruction(w, func, isa, inst, indent)?; + Ok(()) +} + /// Write `func` to `w` as equivalent text. /// Use `isa` to emit ISA-dependent annotations. pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) -> fmt::Result { + decorate_function( + &mut |w, func, isa, inst, indent| write_function_plain(w, func, isa, inst, indent), + w, + func, + isa, + ) +} + +/// Writes 'func' to 'w' as text. +/// write_function_plain is passed as 'closure' to print instructions as text. +/// pretty_function_error is passed as 'closure' to add error decoration. +pub fn decorate_function< + WL: FnMut(&mut Write, &Function, Option<&TargetIsa>, Inst, usize) -> fmt::Result, +>( + closure: &mut WL, + w: &mut Write, + func: &Function, + isa: Option<&TargetIsa>, +) -> fmt::Result { let regs = isa.map(TargetIsa::register_info); let regs = regs.as_ref(); @@ -23,7 +53,7 @@ pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) - if any { writeln!(w)?; } - write_ebb(w, func, isa, ebb)?; + decorate_ebb(closure, w, func, isa, ebb)?; any = true; } writeln!(w, "}}") @@ -141,7 +171,15 @@ pub fn write_ebb_header( writeln!(w, "):") } -pub fn write_ebb(w: &mut Write, func: &Function, isa: Option<&TargetIsa>, ebb: Ebb) -> fmt::Result { +pub fn decorate_ebb< + WL: FnMut(&mut Write, &Function, Option<&TargetIsa>, Inst, usize) -> fmt::Result, +>( + closure: &mut WL, + w: &mut Write, + func: &Function, + isa: Option<&TargetIsa>, + ebb: Ebb, +) -> fmt::Result { // Indent all instructions if any encodings are present. let indent = if func.encodings.is_empty() && func.srclocs.is_empty() { 4 @@ -151,8 +189,9 @@ pub fn write_ebb(w: &mut Write, func: &Function, isa: Option<&TargetIsa>, ebb: E write_ebb_header(w, func, isa, ebb, indent)?; for inst in func.layout.ebb_insts(ebb) { - write_instruction(w, func, isa, inst, indent)?; + closure(w, func, isa, inst, indent)?; } + Ok(()) } From 99b9b96eee1cd6ccfffe2e014eab4d16f02779bb Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Thu, 28 Jun 2018 12:47:13 -0700 Subject: [PATCH 1869/3084] Pop and fill don't trigger StackOverflow. (#365) --- .../filetests/isa/x86/binary32-float.cton | 12 +++++------ cranelift/filetests/isa/x86/binary32.cton | 8 ++++---- .../filetests/isa/x86/binary64-float.cton | 12 +++++------ cranelift/filetests/isa/x86/binary64.cton | 20 +++++++++---------- lib/codegen/meta/isa/x86/recipes.py | 5 ----- 5 files changed, 26 insertions(+), 31 deletions(-) diff --git a/cranelift/filetests/isa/x86/binary32-float.cton b/cranelift/filetests/isa/x86/binary32-float.cton index 05847e9f58..9982b44606 100644 --- a/cranelift/filetests/isa/x86/binary32-float.cton +++ b/cranelift/filetests/isa/x86/binary32-float.cton @@ -181,14 +181,14 @@ ebb0: [-,ss1] v201 = spill v101 ; bin: stk_ovf f3 0f 11 94 24 00000408 ; asm: movss 1032(%esp), %xmm5 - [-,%xmm5] v210 = fill v200 ; bin: stk_ovf f3 0f 10 ac 24 00000408 + [-,%xmm5] v210 = fill v200 ; bin: f3 0f 10 ac 24 00000408 ; asm: movss 1032(%esp), %xmm2 - [-,%xmm2] v211 = fill v201 ; bin: stk_ovf f3 0f 10 94 24 00000408 + [-,%xmm2] v211 = fill v201 ; bin: f3 0f 10 94 24 00000408 ; asm: movss %xmm5, 1032(%esp) regspill v100, %xmm5 -> ss1 ; bin: stk_ovf f3 0f 11 ac 24 00000408 ; asm: movss 1032(%esp), %xmm5 - regfill v100, ss1 -> %xmm5 ; bin: stk_ovf f3 0f 10 ac 24 00000408 + regfill v100, ss1 -> %xmm5 ; bin: f3 0f 10 ac 24 00000408 ; Comparisons. ; @@ -422,14 +422,14 @@ ebb0: [-,ss1] v201 = spill v101 ; bin: stk_ovf f2 0f 11 94 24 00000408 ; asm: movsd 1032(%esp), %xmm5 - [-,%xmm5] v210 = fill v200 ; bin: stk_ovf f2 0f 10 ac 24 00000408 + [-,%xmm5] v210 = fill v200 ; bin: f2 0f 10 ac 24 00000408 ; asm: movsd 1032(%esp), %xmm2 - [-,%xmm2] v211 = fill v201 ; bin: stk_ovf f2 0f 10 94 24 00000408 + [-,%xmm2] v211 = fill v201 ; bin: f2 0f 10 94 24 00000408 ; asm: movsd %xmm5, 1032(%esp) regspill v100, %xmm5 -> ss1 ; bin: stk_ovf f2 0f 11 ac 24 00000408 ; asm: movsd 1032(%esp), %xmm5 - regfill v100, ss1 -> %xmm5 ; bin: stk_ovf f2 0f 10 ac 24 00000408 + regfill v100, ss1 -> %xmm5 ; bin: f2 0f 10 ac 24 00000408 ; Comparisons. ; diff --git a/cranelift/filetests/isa/x86/binary32.cton b/cranelift/filetests/isa/x86/binary32.cton index a6476e3e81..6764388aff 100644 --- a/cranelift/filetests/isa/x86/binary32.cton +++ b/cranelift/filetests/isa/x86/binary32.cton @@ -377,20 +377,20 @@ ebb0: [-,ss1] v501 = spill v2 ; bin: stk_ovf 89 b4 24 00000408 ; asm: movl 1032(%esp), %ecx - [-,%rcx] v510 = fill v500 ; bin: stk_ovf 8b 8c 24 00000408 + [-,%rcx] v510 = fill v500 ; bin: 8b 8c 24 00000408 ; asm: movl 1032(%esp), %esi - [-,%rsi] v511 = fill v501 ; bin: stk_ovf 8b b4 24 00000408 + [-,%rsi] v511 = fill v501 ; bin: 8b b4 24 00000408 ; asm: movl %ecx, 1032(%esp) regspill v1, %rcx -> ss1 ; bin: stk_ovf 89 8c 24 00000408 ; asm: movl 1032(%esp), %ecx - regfill v1, ss1 -> %rcx ; bin: stk_ovf 8b 8c 24 00000408 + regfill v1, ss1 -> %rcx ; bin: 8b 8c 24 00000408 ; Push and Pop ; asm: pushl %ecx x86_push v1 ; bin: stk_ovf 51 ; asm: popl %ecx - [-,%rcx] v512 = x86_pop.i32 ; bin: stk_ovf 59 + [-,%rcx] v512 = x86_pop.i32 ; bin: 59 ; Adjust Stack Pointer Up ; asm: addl $64, %esp diff --git a/cranelift/filetests/isa/x86/binary64-float.cton b/cranelift/filetests/isa/x86/binary64-float.cton index 9bc92e686b..6d85b52976 100644 --- a/cranelift/filetests/isa/x86/binary64-float.cton +++ b/cranelift/filetests/isa/x86/binary64-float.cton @@ -194,14 +194,14 @@ ebb0: [-,ss1] v201 = spill v101 ; bin: stk_ovf f3 44 0f 11 94 24 00000408 ; asm: movss 1032(%rsp), %xmm5 - [-,%xmm5] v210 = fill v200 ; bin: stk_ovf f3 0f 10 ac 24 00000408 + [-,%xmm5] v210 = fill v200 ; bin: f3 0f 10 ac 24 00000408 ; asm: movss 1032(%rsp), %xmm10 - [-,%xmm10] v211 = fill v201 ; bin: stk_ovf f3 44 0f 10 94 24 00000408 + [-,%xmm10] v211 = fill v201 ; bin: f3 44 0f 10 94 24 00000408 ; asm: movss %xmm5, 1032(%rsp) regspill v100, %xmm5 -> ss1 ; bin: stk_ovf f3 0f 11 ac 24 00000408 ; asm: movss 1032(%rsp), %xmm5 - regfill v100, ss1 -> %xmm5 ; bin: stk_ovf f3 0f 10 ac 24 00000408 + regfill v100, ss1 -> %xmm5 ; bin: f3 0f 10 ac 24 00000408 ; Comparisons. ; @@ -457,14 +457,14 @@ ebb0: [-,ss1] v201 = spill v101 ; bin: stk_ovf f2 44 0f 11 94 24 00000408 ; asm: movsd 1032(%rsp), %xmm5 - [-,%xmm5] v210 = fill v200 ; bin: stk_ovf f2 0f 10 ac 24 00000408 + [-,%xmm5] v210 = fill v200 ; bin: f2 0f 10 ac 24 00000408 ; asm: movsd 1032(%rsp), %xmm10 - [-,%xmm10] v211 = fill v201 ; bin: stk_ovf f2 44 0f 10 94 24 00000408 + [-,%xmm10] v211 = fill v201 ; bin: f2 44 0f 10 94 24 00000408 ; asm: movsd %xmm5, 1032(%rsp) regspill v100, %xmm5 -> ss1 ; bin: stk_ovf f2 0f 11 ac 24 00000408 ; asm: movsd 1032(%rsp), %xmm5 - regfill v100, ss1 -> %xmm5 ; bin: stk_ovf f2 0f 10 ac 24 00000408 + regfill v100, ss1 -> %xmm5 ; bin: f2 0f 10 ac 24 00000408 ; Comparisons. ; diff --git a/cranelift/filetests/isa/x86/binary64.cton b/cranelift/filetests/isa/x86/binary64.cton index c6ba888579..1e3da24fa2 100644 --- a/cranelift/filetests/isa/x86/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.cton @@ -534,16 +534,16 @@ ebb0: [-,ss1] v502 = spill v3 ; bin: stk_ovf 4c 89 94 24 00000408 ; asm: movq 1032(%rsp), %rcx - [-,%rcx] v510 = fill v500 ; bin: stk_ovf 48 8b 8c 24 00000408 + [-,%rcx] v510 = fill v500 ; bin: 48 8b 8c 24 00000408 ; asm: movq 1032(%rsp), %rsi - [-,%rsi] v511 = fill v501 ; bin: stk_ovf 48 8b b4 24 00000408 + [-,%rsi] v511 = fill v501 ; bin: 48 8b b4 24 00000408 ; asm: movq 1032(%rsp), %r10 - [-,%r10] v512 = fill v502 ; bin: stk_ovf 4c 8b 94 24 00000408 + [-,%r10] v512 = fill v502 ; bin: 4c 8b 94 24 00000408 ; asm: movq %rcx, 1032(%rsp) regspill v1, %rcx -> ss1 ; bin: stk_ovf 48 89 8c 24 00000408 ; asm: movq 1032(%rsp), %rcx - regfill v1, ss1 -> %rcx ; bin: stk_ovf 48 8b 8c 24 00000408 + regfill v1, ss1 -> %rcx ; bin: 48 8b 8c 24 00000408 ; Push and Pop ; asm: pushq %rcx @@ -551,9 +551,9 @@ ebb0: ; asm: pushq %r10 x86_push v3 ; bin: stk_ovf 41 52 ; asm: popq %rcx - [-,%rcx] v513 = x86_pop.i64 ; bin: stk_ovf 59 + [-,%rcx] v513 = x86_pop.i64 ; bin: 59 ; asm: popq %r10 - [-,%r10] v514 = x86_pop.i64 ; bin: stk_ovf 41 5a + [-,%r10] v514 = x86_pop.i64 ; bin: 41 5a ; Adjust Stack Pointer Up ; asm: addq $64, %rsp @@ -1192,16 +1192,16 @@ ebb0: [-,ss1] v502 = spill v3 ; bin: stk_ovf 44 89 94 24 00000408 ; asm: movl 1032(%rsp), %ecx - [-,%rcx] v510 = fill v500 ; bin: stk_ovf 8b 8c 24 00000408 + [-,%rcx] v510 = fill v500 ; bin: 8b 8c 24 00000408 ; asm: movl 1032(%rsp), %esi - [-,%rsi] v511 = fill v501 ; bin: stk_ovf 8b b4 24 00000408 + [-,%rsi] v511 = fill v501 ; bin: 8b b4 24 00000408 ; asm: movl 1032(%rsp), %r10d - [-,%r10] v512 = fill v502 ; bin: stk_ovf 44 8b 94 24 00000408 + [-,%r10] v512 = fill v502 ; bin: 44 8b 94 24 00000408 ; asm: movl %ecx, 1032(%rsp) regspill v1, %rcx -> ss1 ; bin: stk_ovf 89 8c 24 00000408 ; asm: movl 1032(%rsp), %ecx - regfill v1, ss1 -> %rcx ; bin: stk_ovf 8b 8c 24 00000408 + regfill v1, ss1 -> %rcx ; bin: 8b 8c 24 00000408 ; asm: cmpl %esi, %ecx [-,%rflags] v520 = ifcmp v1, v2 ; bin: 39 f1 diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py index 66c6326a3a..b65eac5113 100644 --- a/lib/codegen/meta/isa/x86/recipes.py +++ b/lib/codegen/meta/isa/x86/recipes.py @@ -584,7 +584,6 @@ pushq = TailRecipe( popq = TailRecipe( 'popq', NullAry, size=0, ins=(), outs=GPR, emit=''' - sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink); ''') @@ -1329,7 +1328,6 @@ fillSib32 = TailRecipe( 'fillSib32', Unary, size=6, ins=StackGPR32, outs=GPR, clobbers_flags=False, emit=''' - sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); let base = stk_base(in_stk0.base); PUT_OP(bits, rex2(base, out_reg0), sink); modrm_sib_disp32(out_reg0, sink); @@ -1342,7 +1340,6 @@ ffillSib32 = TailRecipe( 'ffillSib32', Unary, size=6, ins=StackFPR32, outs=FPR, clobbers_flags=False, emit=''' - sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); let base = stk_base(in_stk0.base); PUT_OP(bits, rex2(base, out_reg0), sink); modrm_sib_disp32(out_reg0, sink); @@ -1355,7 +1352,6 @@ regfill32 = TailRecipe( 'regfill32', RegFill, size=6, ins=StackGPR32, outs=(), clobbers_flags=False, emit=''' - sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); let src = StackRef::sp(src, &func.stack_slots); let base = stk_base(src.base); PUT_OP(bits, rex2(base, dst), sink); @@ -1369,7 +1365,6 @@ fregfill32 = TailRecipe( 'fregfill32', RegFill, size=6, ins=StackFPR32, outs=(), clobbers_flags=False, emit=''' - sink.trap(TrapCode::StackOverflow, func.srclocs[inst]); let src = StackRef::sp(src, &func.stack_slots); let base = stk_base(src.base); PUT_OP(bits, rex2(base, dst), sink); From 8f3c49bc6ce30241b27642474b878caa40ee36f3 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 26 Jun 2018 13:54:46 -0700 Subject: [PATCH 1870/3084] Update more references to "global variables". This continues the transition to "global values", which aren't implicitly dereferenced. --- cranelift/docs/langref.rst | 33 +++++-------- lib/codegen/meta/base/instructions.py | 2 +- lib/codegen/src/ir/entities.rs | 2 +- lib/codegen/src/ir/function.rs | 2 +- lib/codegen/src/ir/globalvalue.rs | 23 +++++---- lib/codegen/src/ir/globalvar.rs | 70 --------------------------- lib/codegen/src/ir/heap.rs | 6 +-- lib/codegen/src/verifier/mod.rs | 2 +- lib/reader/src/parser.rs | 4 +- 9 files changed, 32 insertions(+), 112 deletions(-) delete mode 100644 lib/codegen/src/ir/globalvar.rs diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 7630066550..5516812ceb 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -575,18 +575,18 @@ Cretonne functions. .. inst:: GV = vmctx+Offset - Declare a global value in the VM context struct. + Declare a global value of the address of a field in the VM context struct. - This declares a global value whose address is a constant offset from the + This declares a global value which is a constant offset from the VM context pointer which is passed as a hidden argument to all functions JIT-compiled for the VM. Typically, the VM context is a C struct, and the declared global value - is a member of the struct. + is the address of a member of the struct. :arg Offset: Byte offset from the VM context pointer to the global - variable. - :result GV: Global variable. + value. + :result GV: Global value. The address of a global value can also be derived by treating another global variable as a struct pointer. This makes it possible to chase pointers into VM @@ -599,16 +599,15 @@ runtime data structures. The address of GV can be computed by first loading a pointer from BaseGV and adding Offset to it. - It is assumed the BaseGV resides in readable memory with the appropriate + It is assumed the BaseGV resides in accessible memory with the appropriate alignment for storing a pointer. Chains of ``deref`` global values are possible, but cycles are not allowed. They will be caught by the IR verifier. - :arg BaseGV: Global variable containing the base pointer. - :arg Offset: Byte offset from the loaded base pointer to the global - variable. - :result GV: Global variable. + :arg BaseGV: Global value providing the base pointer. + :arg Offset: Byte offset added to the loaded value. + :result GV: Global value. .. inst:: GV = [colocated] globalsym name @@ -622,7 +621,7 @@ runtime data structures. efficient addressing. :arg name: External name. - :result GV: Global variable. + :result GV: Global value. .. autoinst:: global_value .. autoinst:: globalsym_addr @@ -690,7 +689,7 @@ trap when accessed. Declare a static heap in the preamble. - :arg Base: Global variable holding the heap's base address or + :arg Base: Global value holding the heap's base address or ``reserved_reg``. :arg MinBytes: Guaranteed minimum heap size in bytes. Accesses below this size will never trap. @@ -698,9 +697,6 @@ trap when accessed. address space reserved for the heap, not including the guard pages. :arg GuardBytes: Size of the guard pages in bytes. -When the base is a global value, it must be :term:`accessible` and naturally -aligned for a pointer value. - The ``reserved_reg`` option is not yet implemented. Dynamic heaps @@ -714,16 +710,13 @@ is resized. The bound of a dynamic heap is stored in a global value. Declare a dynamic heap in the preamble. - :arg Base: Global variable holding the heap's base address or + :arg Base: Global value holding the heap's base address or ``reserved_reg``. :arg MinBytes: Guaranteed minimum heap size in bytes. Accesses below this size will never trap. - :arg BoundGV: Global variable containing the current heap bound in bytes. + :arg BoundGV: Global value containing the current heap bound in bytes. :arg GuardBytes: Size of the guard pages in bytes. -When the base is a global value, it must be :term:`accessible` and naturally -aligned for a pointer value. - The ``reserved_reg`` option is not yet implemented. Heap examples diff --git a/lib/codegen/meta/base/instructions.py b/lib/codegen/meta/base/instructions.py index 4572c4914b..1a490418d7 100644 --- a/lib/codegen/meta/base/instructions.py +++ b/lib/codegen/meta/base/instructions.py @@ -488,7 +488,7 @@ stack_addr = Instruction( ins=(SS, Offset), outs=addr) # -# Global variables. +# Global values. # GV = Operand('GV', entities.global_value) diff --git a/lib/codegen/src/ir/entities.rs b/lib/codegen/src/ir/entities.rs index 847a9ba925..8c38926754 100644 --- a/lib/codegen/src/ir/entities.rs +++ b/lib/codegen/src/ir/entities.rs @@ -185,7 +185,7 @@ pub enum AnyEntity { Value(Value), /// A stack slot. StackSlot(StackSlot), - /// A Global variable. + /// A Global value. GlobalValue(GlobalValue), /// A jump table. JumpTable(JumpTable), diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs index 9c6647ba0c..cff7bb2db0 100644 --- a/lib/codegen/src/ir/function.rs +++ b/lib/codegen/src/ir/function.rs @@ -36,7 +36,7 @@ pub struct Function { /// be checked against. pub stack_limit: Option, - /// Global variables referenced. + /// Global values referenced. pub global_values: PrimaryMap, /// Heaps referenced. diff --git a/lib/codegen/src/ir/globalvalue.rs b/lib/codegen/src/ir/globalvalue.rs index 26c0e48434..d522e26c1b 100644 --- a/lib/codegen/src/ir/globalvalue.rs +++ b/lib/codegen/src/ir/globalvalue.rs @@ -1,4 +1,4 @@ -//! Global variables. +//! Global values. use ir::immediates::Offset32; use ir::{ExternalName, GlobalValue}; @@ -7,34 +7,33 @@ use std::fmt; /// Information about a global value declaration. #[derive(Clone)] pub enum GlobalValueData { - /// Variable is part of the VM context struct, it's address is a constant offset from the VM + /// Value is the address of a field in the VM context struct, a constant offset from the VM /// context pointer. VMContext { - /// Offset from the `vmctx` pointer to this global. + /// Offset from the `vmctx` pointer. offset: Offset32, }, - /// Variable is part of a struct pointed to by another global value. + /// Value is pointed to by another global value. /// - /// The `base` global value is assumed to contain a pointer to a struct. This global - /// variable lives at an offset into the struct. The memory must be accessible, and - /// naturally aligned to hold a pointer value. + /// The `base` global value is assumed to contain a pointer. This global value is computed + /// by loading from memory at that pointer value, and then adding an offset. The memory must + /// be accessible, and naturally aligned to hold a pointer value. Deref { /// The base pointer global value. base: GlobalValue, - /// Byte offset to be added to the pointer loaded from `base`. + /// Byte offset to be added to the loaded value. offset: Offset32, }, - /// Variable is at an address identified by a symbolic name. Cretonne itself - /// does not interpret this name; it's used by embedders to link with other - /// data structures. + /// Value is identified by a symbolic name. Cretonne itself does not interpret this name; + /// it's used by embedders to link with other data structures. Sym { /// The symbolic name. name: ExternalName, - /// Will this variable be defined nearby, such that it will always be a certain distance + /// Will this symbol be defined nearby, such that it will always be a certain distance /// away, after linking? If so, references to it can avoid going through a GOT. Note that /// symbols meant to be preemptible cannot be colocated. colocated: bool, diff --git a/lib/codegen/src/ir/globalvar.rs b/lib/codegen/src/ir/globalvar.rs deleted file mode 100644 index 26c0e48434..0000000000 --- a/lib/codegen/src/ir/globalvar.rs +++ /dev/null @@ -1,70 +0,0 @@ -//! Global variables. - -use ir::immediates::Offset32; -use ir::{ExternalName, GlobalValue}; -use std::fmt; - -/// Information about a global value declaration. -#[derive(Clone)] -pub enum GlobalValueData { - /// Variable is part of the VM context struct, it's address is a constant offset from the VM - /// context pointer. - VMContext { - /// Offset from the `vmctx` pointer to this global. - offset: Offset32, - }, - - /// Variable is part of a struct pointed to by another global value. - /// - /// The `base` global value is assumed to contain a pointer to a struct. This global - /// variable lives at an offset into the struct. The memory must be accessible, and - /// naturally aligned to hold a pointer value. - Deref { - /// The base pointer global value. - base: GlobalValue, - - /// Byte offset to be added to the pointer loaded from `base`. - offset: Offset32, - }, - - /// Variable is at an address identified by a symbolic name. Cretonne itself - /// does not interpret this name; it's used by embedders to link with other - /// data structures. - Sym { - /// The symbolic name. - name: ExternalName, - - /// Will this variable be defined nearby, such that it will always be a certain distance - /// away, after linking? If so, references to it can avoid going through a GOT. Note that - /// symbols meant to be preemptible cannot be colocated. - colocated: bool, - }, -} - -impl GlobalValueData { - /// Assume that `self` is an `GlobalValueData::Sym` and return its name. - pub fn symbol_name(&self) -> &ExternalName { - match *self { - GlobalValueData::Sym { ref name, .. } => name, - _ => panic!("only symbols have names"), - } - } -} - -impl fmt::Display for GlobalValueData { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match *self { - GlobalValueData::VMContext { offset } => write!(f, "vmctx{}", offset), - GlobalValueData::Deref { base, offset } => write!(f, "deref({}){}", base, offset), - GlobalValueData::Sym { - ref name, - colocated, - } => { - if colocated { - write!(f, "colocated ")?; - } - write!(f, "globalsym {}", name) - } - } - } -} diff --git a/lib/codegen/src/ir/heap.rs b/lib/codegen/src/ir/heap.rs index 4bb0b69166..439b8aac49 100644 --- a/lib/codegen/src/ir/heap.rs +++ b/lib/codegen/src/ir/heap.rs @@ -29,8 +29,7 @@ pub enum HeapBase { /// This feature is not yet implemented. ReservedReg, - /// The heap base is in a global value. The variable must be accessible and naturally - /// aligned for a pointer. + /// The heap base is a global value. GlobalValue(GlobalValue), } @@ -39,8 +38,7 @@ pub enum HeapBase { pub enum HeapStyle { /// A dynamic heap can be relocated to a different base address when it is grown. Dynamic { - /// Global variable holding the current bound of the heap in bytes. It is - /// required to be accessible and naturally aligned for a pointer-sized integer. + /// Global value providing the current bound of the heap in bytes. bound_gv: GlobalValue, }, diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs index 4bd027a58f..4585cc8176 100644 --- a/lib/codegen/src/verifier/mod.rs +++ b/lib/codegen/src/verifier/mod.rs @@ -41,7 +41,7 @@ //! - All return instructions must have return value operands matching the current //! function signature. //! -//! Global variables +//! Global values //! //! - Detect cycles in deref(base) declarations. //! diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index a8bea8b2b0..79adb9f3d4 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1080,8 +1080,8 @@ impl<'a> Parser<'a> { // Parse a global value decl. // - // global-var-decl ::= * GlobalValue(gv) "=" global-var-desc - // global-var-desc ::= "vmctx" offset32 + // global-val-decl ::= * GlobalValue(gv) "=" global-val-desc + // global-val-desc ::= "vmctx" offset32 // | "deref" "(" GlobalValue(base) ")" offset32 // | globalsym ["colocated"] name // From 7a55a107aee338c0ec6b2f6f507085298dd233b8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 26 Jun 2018 14:03:22 -0700 Subject: [PATCH 1871/3084] Say "Global Variable" when referring to the WebAssembly concept. This helps avoid confusion between wasm global variables and cretonne global values. --- lib/wasm/src/code_translator.rs | 10 +++++----- lib/wasm/src/environ/dummy.rs | 6 +++--- lib/wasm/src/environ/mod.rs | 4 +++- lib/wasm/src/environ/spec.rs | 14 +++++++------- lib/wasm/src/lib.rs | 2 +- lib/wasm/src/sections_translator.rs | 2 +- lib/wasm/src/state.rs | 10 +++++----- lib/wasm/src/translation_utils.rs | 2 +- 8 files changed, 26 insertions(+), 24 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 7f9d390399..76290364f2 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -27,7 +27,7 @@ use cretonne_codegen::ir::types::*; use cretonne_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags}; use cretonne_codegen::packed_option::ReservedValue; use cretonne_frontend::{FunctionBuilder, Variable}; -use environ::{FuncEnvironment, GlobalValue, WasmError, WasmResult}; +use environ::{FuncEnvironment, GlobalVariable, WasmError, WasmResult}; use state::{ControlStackFrame, TranslationState}; use std::collections::{hash_map, HashMap}; use std::vec::Vec; @@ -72,8 +72,8 @@ pub fn translate_operator( ***********************************************************************************/ Operator::GetGlobal { global_index } => { let val = match state.get_global(builder.func, global_index, environ) { - GlobalValue::Const(val) => val, - GlobalValue::Memory { gv, ty } => { + GlobalVariable::Const(val) => val, + GlobalVariable::Memory { gv, ty } => { let addr = builder.ins().global_value(environ.native_pointer(), gv); let mut flags = ir::MemFlags::new(); flags.set_notrap(); @@ -85,8 +85,8 @@ pub fn translate_operator( } Operator::SetGlobal { global_index } => { match state.get_global(builder.func, global_index, environ) { - GlobalValue::Const(_) => panic!("global #{} is a constant", global_index), - GlobalValue::Memory { gv, .. } => { + GlobalVariable::Const(_) => panic!("global #{} is a constant", global_index), + GlobalVariable::Memory { gv, .. } => { let addr = builder.ins().global_value(environ.native_pointer(), gv); let mut flags = ir::MemFlags::new(); flags.set_notrap(); diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs index 6ccefe3213..912c39b107 100644 --- a/lib/wasm/src/environ/dummy.rs +++ b/lib/wasm/src/environ/dummy.rs @@ -4,7 +4,7 @@ use cretonne_codegen::cursor::FuncCursor; use cretonne_codegen::ir::types::*; use cretonne_codegen::ir::{self, InstBuilder}; use cretonne_codegen::settings; -use environ::{FuncEnvironment, GlobalValue, ModuleEnvironment, WasmResult}; +use environ::{FuncEnvironment, GlobalVariable, ModuleEnvironment, WasmResult}; use func_translator::FuncTranslator; use std::string::String; use std::vec::Vec; @@ -157,11 +157,11 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ &self.mod_info.flags } - fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue { + fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalVariable { // Just create a dummy `vmctx` global. let offset = ((index * 8) as i32 + 8).into(); let gv = func.create_global_value(ir::GlobalValueData::VMContext { offset }); - GlobalValue::Memory { + GlobalVariable::Memory { gv, ty: self.mod_info.globals[index].entity.ty, } diff --git a/lib/wasm/src/environ/mod.rs b/lib/wasm/src/environ/mod.rs index 923d65be11..0ca34575f7 100644 --- a/lib/wasm/src/environ/mod.rs +++ b/lib/wasm/src/environ/mod.rs @@ -4,4 +4,6 @@ mod dummy; mod spec; pub use environ::dummy::DummyEnvironment; -pub use environ::spec::{FuncEnvironment, GlobalValue, ModuleEnvironment, WasmError, WasmResult}; +pub use environ::spec::{ + FuncEnvironment, GlobalVariable, ModuleEnvironment, WasmError, WasmResult, +}; diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs index 09a8dd5371..959a411899 100644 --- a/lib/wasm/src/environ/spec.rs +++ b/lib/wasm/src/environ/spec.rs @@ -10,17 +10,17 @@ use translation_utils::{ }; use wasmparser::BinaryReaderError; -/// The value of a WebAssembly global value. +/// The value of a WebAssembly global variable. #[derive(Clone, Copy)] -pub enum GlobalValue { +pub enum GlobalVariable { /// This is a constant global with a value known at compile time. Const(ir::Value), - /// This is a variable in memory that should be referenced as a `GlobalValue`. + /// This is a variable in memory that should be referenced through a `GlobalValue`. Memory { - /// Which global value should be referenced. + /// The address of the global variable storage. gv: ir::GlobalValue, - /// The global value's type. + /// The global variable's type. ty: ir::Type, }, } @@ -89,14 +89,14 @@ pub trait FuncEnvironment { ir::Type::int(u16::from(self.triple().pointer_width().unwrap().bits())).unwrap() } - /// Set up the necessary preamble definitions in `func` to access the global value + /// Set up the necessary preamble definitions in `func` to access the global variable /// identified by `index`. /// /// The index space covers both imported globals and globals defined by the module. /// /// Return the global variable reference that should be used to access the global and the /// WebAssembly type of the global. - fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalValue; + fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalVariable; /// Set up the necessary preamble definitions in `func` to access the linear memory identified /// by `index`. diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs index 5d4fd24a37..70826f72fb 100644 --- a/lib/wasm/src/lib.rs +++ b/lib/wasm/src/lib.rs @@ -43,7 +43,7 @@ mod state; mod translation_utils; pub use environ::{ - DummyEnvironment, FuncEnvironment, GlobalValue, ModuleEnvironment, WasmError, WasmResult, + DummyEnvironment, FuncEnvironment, GlobalVariable, ModuleEnvironment, WasmError, WasmResult, }; pub use func_translator::FuncTranslator; pub use module_translator::translate_module; diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 112bd3784e..0c47a3e940 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -4,7 +4,7 @@ //! The code of theses helper function is straightforward since it is only about reading metadata //! about linear memories, tables, globals, etc. and storing them for later use. //! -//! The special case of the initialize expressions for table elements offsets or global values +//! The special case of the initialize expressions for table elements offsets or global variables //! is handled, according to the semantics of WebAssembly, to only specific expressions that are //! interpreted on the fly. use cretonne_codegen::ir::{self, AbiParam, Signature}; diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs index beba6e666f..d301d87348 100644 --- a/lib/wasm/src/state.rs +++ b/lib/wasm/src/state.rs @@ -4,7 +4,7 @@ //! value and control stacks during the translation of a single function. use cretonne_codegen::ir::{self, Ebb, Inst, Value}; -use environ::{FuncEnvironment, GlobalValue}; +use environ::{FuncEnvironment, GlobalVariable}; use std::collections::HashMap; use std::vec::Vec; use translation_utils::{FunctionIndex, GlobalIndex, MemoryIndex, SignatureIndex}; @@ -134,8 +134,8 @@ pub struct TranslationState { pub control_stack: Vec, pub reachable: bool, - // Map of global values that have already been created by `FuncEnvironment::make_global`. - globals: HashMap, + // Map of global variables that have already been created by `FuncEnvironment::make_global`. + globals: HashMap, // Map of heaps that have been created by `FuncEnvironment::make_heap`. heaps: HashMap, @@ -272,7 +272,7 @@ impl TranslationState { /// Methods for handling entity references. impl TranslationState { - /// Get the `GlobalValue` reference that should be used to access the global value `index`. + /// Get the `GlobalVariable` reference that should be used to access the global variable `index`. /// Create the reference if necessary. /// Also return the WebAssembly type of the global. pub fn get_global( @@ -280,7 +280,7 @@ impl TranslationState { func: &mut ir::Function, index: u32, environ: &mut FE, - ) -> GlobalValue { + ) -> GlobalVariable { let index = index as GlobalIndex; *self.globals .entry(index) diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 3c16d0897e..3ed13899e6 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -7,7 +7,7 @@ use wasmparser; pub type FunctionIndex = usize; /// Index of a table (imported or defined) inside the WebAssembly module. pub type TableIndex = usize; -/// Index of a global value (imported or defined) inside the WebAssembly module. +/// Index of a global variable (imported or defined) inside the WebAssembly module. pub type GlobalIndex = usize; /// Index of a linear memory (imported or defined) inside the WebAssembly module. pub type MemoryIndex = usize; From 1074e2c7551b65c714d3a48cbfec19d99ddf1199 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 2 Jul 2018 07:59:54 -0700 Subject: [PATCH 1872/3084] Tidy up trailing whitespace. --- cranelift/docs/langref.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 5516812ceb..4197bd5a3d 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -564,7 +564,7 @@ A *global value* is an object whose value is not known at compile time. The value is computed at runtime by :inst:`global_value`, possibly using information provided by the linker via relocations. There are multiple kinds of global values using different methods for determining their value. -Cretonne does not track the type of a global value, for they are just +Cretonne does not track the type of a global value, for they are just values stored in non-stack memory. When Cretonne is generating code for a virtual machine environment, globals can From 276ba8b97d8127fb1efbf84a22bef010d784b9ee Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 2 Jul 2018 13:02:00 -0700 Subject: [PATCH 1873/3084] Support systems which don't have a "python" command. (#386) Add support for finding an appropriate python command on systems which don't have "python". Try "python3" and "python2.7". Fixed #381. --- cranelift/test-all.sh | 2 +- lib/codegen/build.rs | 16 +++++++++++++++- lib/codegen/meta/check.sh | 5 +++-- 3 files changed, 19 insertions(+), 4 deletions(-) diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 5c280523d3..4bb7330f09 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -46,7 +46,7 @@ else needcheck=yes fi if [ -n "$needcheck" ]; then - banner "$(python --version 2>&1), $(python3 --version 2>&1)" + banner "Checking python source files" "$topdir/lib/codegen/meta/check.sh" touch "$tsfile" || echo no target directory fi diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs index d127f357b4..c52c70873e 100644 --- a/lib/codegen/build.rs +++ b/lib/codegen/build.rs @@ -26,6 +26,7 @@ fn main() { let target_triple = env::var("TARGET").expect("The TARGET environment variable must be set"); let cretonne_targets = env::var("CRETONNE_TARGETS").ok(); let cretonne_targets = cretonne_targets.as_ref().map(|s| s.as_ref()); + let python = identify_python(); // Configure isa targets cfg. match isa_targets(cretonne_targets, &target_triple) { @@ -60,7 +61,7 @@ fn main() { // Launch build script with Python. We'll just find python in the path. // Use -B to disable .pyc files, because they cause trouble for vendoring // scripts, and this is a build step that isn't run very often anyway. - let status = process::Command::new("python") + let status = process::Command::new(python) .current_dir(crate_dir) .arg("-B") .arg(build_script) @@ -73,6 +74,19 @@ fn main() { } } +fn identify_python() -> &'static str { + for python in &["python", "python3", "python2.7"] { + if process::Command::new(python) + .arg("--version") + .status() + .is_ok() + { + return python; + } + } + panic!("The Cretonne build requires Python (version 2.7 or version 3)"); +} + /// Represents known ISA target. #[derive(Copy, Clone)] enum Isa { diff --git a/lib/codegen/meta/check.sh b/lib/codegen/meta/check.sh index ce34f78f4b..14e51da065 100755 --- a/lib/codegen/meta/check.sh +++ b/lib/codegen/meta/check.sh @@ -5,7 +5,8 @@ cd "$topdir" function runif { if type "$1" > /dev/null; then - echo " === $1 ===" + version=$("$1" --version 2>&1) + echo " === $1: $version ===" "$@" else echo "$1 not found" @@ -19,7 +20,7 @@ runif flake8 . runif mypy --py2 build.py # Python unit tests. -runif python -m unittest discover +runif python2.7 -m unittest discover # Then run the unit tests again with Python 3. # We get deprecation warnings about assertRaisesRegexp which was renamed in From b17d1617cacb0c88c7b9362ed5e13ab738802689 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 2 Jul 2018 15:09:30 -0700 Subject: [PATCH 1874/3084] Update to hashmap_core 0.1.8. --- lib/codegen/Cargo.toml | 2 +- lib/module/Cargo.toml | 2 +- lib/wasm/Cargo.toml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index c68710f358..43c073c721 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -14,7 +14,7 @@ build = "build.rs" cretonne-entity = { path = "../entity", version = "0.12.0", default-features = false } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } -hashmap_core = { version = "0.1.7", optional = true } +hashmap_core = { version = "0.1.8", optional = true } target-lexicon = { version = "0.0.2", default-features = false } # It is a goal of the cretonne-codegen crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 41e2621a91..60a7a5d2b8 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -11,7 +11,7 @@ readme = "README.md" [dependencies] cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } cretonne-entity = { path = "../entity", version = "0.12.0", default-features = false } -hashmap_core = { version = "0.1.7", optional = true } +hashmap_core = { version = "0.1.8", optional = true } failure = "0.1.1" [features] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index b57b944f68..00f0a772a5 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -12,7 +12,7 @@ keywords = ["webassembly", "wasm"] wasmparser = { version = "0.17.0", default-features = false } cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } cretonne-frontend = { path = "../frontend", version = "0.12.0", default-features = false } -hashmap_core = { version = "0.1.7", optional = true } +hashmap_core = { version = "0.1.8", optional = true } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } target-lexicon = { version = "0.0.2", default-features = false } From d2d8b56ddb1ce5190b7233686b83a42cf6ce1435 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 2 Jul 2018 15:17:14 -0700 Subject: [PATCH 1875/3084] Fix test-no_std.sh to handle the case where topdir is ".". Instead of using "cd $topdir" to get back to the top, use popd. --- cranelift/test-no_std.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh index 26678c856e..568dffb638 100755 --- a/cranelift/test-no_std.sh +++ b/cranelift/test-no_std.sh @@ -14,10 +14,9 @@ function banner { # Test those packages which have no_std support. LIBS="codegen frontend wasm native module simplejit umbrella" -cd "$topdir" for LIB in $LIBS; do banner "Rust unit tests in $LIB" - cd "lib/$LIB" + pushd "lib/$LIB" >/dev/null # Test with just "core" enabled. cargo +nightly test --no-default-features --features core @@ -25,7 +24,7 @@ for LIB in $LIBS; do # Test with "core" and "std" enabled at the same time. cargo +nightly test --features core - cd "$topdir" + popd >/dev/null done banner "OK" From b0cce6daecf04d71c50952197c2a7d69dcd324da Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 2 Jul 2018 15:43:15 -0700 Subject: [PATCH 1876/3084] Update the documentation for the new `adjust_sp_*` instruction names. `adjust_sp_imm` has been split into `adjust_sp_up_imm` and `adjust_sp_down_imm`, and `adjust_sp_down` has been added. --- cranelift/docs/langref.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 4197bd5a3d..776c61d75b 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -999,7 +999,9 @@ Special register operations The prologue and epilogue of a function needs to manipulate special registers like the stack pointer and the frame pointer. These instructions should not be used in regular code. -.. autoinst:: adjust_sp_imm +.. autoinst:: adjust_sp_down +.. autoinst:: adjust_sp_up_imm +.. autoinst:: adjust_sp_down_imm .. autoinst:: ifcmp_sp .. autoinst:: copy_special From 112ae6df56dcbe7740044cbdb24edc665c1a81ee Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 2 Jul 2018 15:48:18 -0700 Subject: [PATCH 1877/3084] Bump version to 0.13.0 --- cranelift/Cargo.toml | 22 +++++++++++----------- cranelift/publish-all.sh | 2 +- lib/codegen/Cargo.toml | 4 ++-- lib/entity/Cargo.toml | 2 +- lib/faerie/Cargo.toml | 6 +++--- lib/filetests/Cargo.toml | 6 +++--- lib/frontend/Cargo.toml | 4 ++-- lib/module/Cargo.toml | 6 +++--- lib/native/Cargo.toml | 4 ++-- lib/reader/Cargo.toml | 4 ++-- lib/simplejit/Cargo.toml | 8 ++++---- lib/umbrella/Cargo.toml | 6 +++--- lib/wasm/Cargo.toml | 6 +++--- 13 files changed, 40 insertions(+), 40 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index d74d5cd2c2..8c1730e6a8 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-tools" authors = ["The Cretonne Project Developers"] -version = "0.12.0" +version = "0.13.0" description = "Binaries for testing the Cretonne libraries" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -14,16 +14,16 @@ path = "src/cton-util.rs" [dependencies] cfg-if = "0.1" -cretonne-codegen = { path = "lib/codegen", version = "0.12.0" } -cretonne-reader = { path = "lib/reader", version = "0.12.0" } -cretonne-frontend = { path = "lib/frontend", version = "0.12.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.12.0", optional = true } -cretonne-native = { path = "lib/native", version = "0.12.0" } -cretonne-filetests = { path = "lib/filetests", version = "0.12.0" } -cretonne-module = { path = "lib/module", version = "0.12.0" } -cretonne-faerie = { path = "lib/faerie", version = "0.12.0" } -cretonne-simplejit = { path = "lib/simplejit", version = "0.12.0" } -cretonne = { path = "lib/umbrella", version = "0.12.0" } +cretonne-codegen = { path = "lib/codegen", version = "0.13.0" } +cretonne-reader = { path = "lib/reader", version = "0.13.0" } +cretonne-frontend = { path = "lib/frontend", version = "0.13.0" } +cretonne-wasm = { path = "lib/wasm", version = "0.13.0", optional = true } +cretonne-native = { path = "lib/native", version = "0.13.0" } +cretonne-filetests = { path = "lib/filetests", version = "0.13.0" } +cretonne-module = { path = "lib/module", version = "0.13.0" } +cretonne-faerie = { path = "lib/faerie", version = "0.13.0" } +cretonne-simplejit = { path = "lib/simplejit", version = "0.13.0" } +cretonne = { path = "lib/umbrella", version = "0.13.0" } filecheck = "0.3.0" docopt = "1" serde = "1.0.8" diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index f9fc306059..8f5f26fc09 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -4,7 +4,7 @@ topdir=$(dirname "$0") cd "$topdir" # All the cretonne-* crates have the same version number -version="0.12.0" +version="0.13.0" # Update all of the Cargo.toml files. # diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 43c073c721..415aa6534a 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-codegen" -version = "0.12.0" +version = "0.13.0" description = "Low-level code generator library" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"] build = "build.rs" [dependencies] -cretonne-entity = { path = "../entity", version = "0.12.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.13.0", default-features = false } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } hashmap_core = { version = "0.1.8", optional = true } diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index 83516f86ec..d438f4e065 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-entity" -version = "0.12.0" +version = "0.13.0" description = "Data structures using entity references as mapping keys" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index b5cd90f295..cc248678d3 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-faerie" -version = "0.12.0" +version = "0.13.0" authors = ["The Cretonne Project Developers"] description = "Emit Cretonne output to native object files with Faerie" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.12.0" } -cretonne-module = { path = "../module", version = "0.12.0" } +cretonne-codegen = { path = "../codegen", version = "0.13.0" } +cretonne-module = { path = "../module", version = "0.13.0" } faerie = "0.4.2" goblin = "0.0.15" failure = "0.1.1" diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 756e81f13a..2ff5589d16 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] -version = "0.12.0" +version = "0.13.0" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/en/latest/testing.html#file-tests" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" publish = false [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.12.0" } -cretonne-reader = { path = "../reader", version = "0.12.0" } +cretonne-codegen = { path = "../codegen", version = "0.13.0" } +cretonne-reader = { path = "../reader", version = "0.13.0" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index a46c9f2f05..aef504dfe6 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" -version = "0.12.0" +version = "0.13.0" description = "Cretonne IR builder helper" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.13.0", default-features = false } [features] default = ["std"] diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 60a7a5d2b8..de97231b33 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-module" -version = "0.12.0" +version = "0.13.0" authors = ["The Cretonne Project Developers"] description = "Support for linking functions and data with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,8 +9,8 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } -cretonne-entity = { path = "../entity", version = "0.12.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.13.0", default-features = false } +cretonne-entity = { path = "../entity", version = "0.13.0", default-features = false } hashmap_core = { version = "0.1.8", optional = true } failure = "0.1.1" diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 18d92b43d4..0efa247104 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-native" -version = "0.12.0" +version = "0.13.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -8,7 +8,7 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.13.0", default-features = false } target-lexicon = { version = "0.0.2", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index c0e6f40402..b6b3a19b50 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne-reader" -version = "0.12.0" +version = "0.13.0" description = "Cretonne textual IR reader" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -9,7 +9,7 @@ repository = "https://github.com/cretonne/cretonne" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.12.0" } +cretonne-codegen = { path = "../codegen", version = "0.13.0" } target-lexicon = "0.0.2" [badges] diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index ed04e90a07..8d359ac48d 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-simplejit" -version = "0.12.0" +version = "0.13.0" authors = ["The Cretonne Project Developers"] description = "A simple JIT library backed by Cretonne" repository = "https://github.com/cretonne/cretonne" @@ -9,9 +9,9 @@ license = "Apache-2.0" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } -cretonne-module = { path = "../module", version = "0.12.0", default-features = false } -cretonne-native = { path = "../native", version = "0.12.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.13.0", default-features = false } +cretonne-module = { path = "../module", version = "0.13.0", default-features = false } +cretonne-native = { path = "../native", version = "0.13.0", default-features = false } region = "0.3.0" libc = { version = "0.2.40", default-features = false } errno = "0.2.3" diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml index 8220c75861..a23be961cb 100644 --- a/lib/umbrella/Cargo.toml +++ b/lib/umbrella/Cargo.toml @@ -1,7 +1,7 @@ [package] authors = ["The Cretonne Project Developers"] name = "cretonne" -version = "0.12.0" +version = "0.13.0" description = "Umbrella for commonly-used cretonne crates" license = "Apache-2.0" documentation = "https://cretonne.readthedocs.io/" @@ -10,8 +10,8 @@ readme = "README.md" keywords = ["compile", "compiler", "jit"] [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.12.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.13.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.13.0", default-features = false } [features] default = ["std"] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 00f0a772a5..aa9c2d772c 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cretonne-wasm" -version = "0.12.0" +version = "0.13.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/cretonne/cretonne" @@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"] [dependencies] wasmparser = { version = "0.17.0", default-features = false } -cretonne-codegen = { path = "../codegen", version = "0.12.0", default-features = false } -cretonne-frontend = { path = "../frontend", version = "0.12.0", default-features = false } +cretonne-codegen = { path = "../codegen", version = "0.13.0", default-features = false } +cretonne-frontend = { path = "../frontend", version = "0.13.0", default-features = false } hashmap_core = { version = "0.1.8", optional = true } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } From 632bbf2008d96b15bc20c5b7990b9544bc58086d Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 2 Jul 2018 22:38:29 -0700 Subject: [PATCH 1878/3084] Permit {s,u}{div,rem}_imm instructions to be potentially trapping. The documentation for these instructions suggests that immediate values which could lead to trapping should be invalid. While it seems nice to have these instructions be always non-trapping, it's also nice to say that the `_imm` forms of instructions are interchangeable with the corresponding non-`_imm` forms accompanied by `iconst` instructions. --- lib/codegen/meta/base/instructions.py | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/lib/codegen/meta/base/instructions.py b/lib/codegen/meta/base/instructions.py index 1a490418d7..1d02b60cdc 100644 --- a/lib/codegen/meta/base/instructions.py +++ b/lib/codegen/meta/base/instructions.py @@ -1016,7 +1016,7 @@ udiv_imm = Instruction( 'udiv_imm', """ Unsigned integer division by an immediate constant. - This instruction never traps because a divisor of zero is not allowed. + This operation traps if the divisor is zero. """, ins=(x, Y), outs=a) @@ -1024,15 +1024,17 @@ sdiv_imm = Instruction( 'sdiv_imm', """ Signed integer division by an immediate constant. - This instruction never traps because a divisor of -1 or 0 is not - allowed. """, + This operation traps if the divisor is zero, or if the result is not + representable in :math:`B` bits two's complement. This only happens + when :math:`x = -2^{B-1}, Y = -1`. + """, ins=(x, Y), outs=a) urem_imm = Instruction( 'urem_imm', """ Unsigned integer remainder with immediate divisor. - This instruction never traps because a divisor of zero is not allowed. + This operation traps if the divisor is zero. """, ins=(x, Y), outs=a) @@ -1040,8 +1042,8 @@ srem_imm = Instruction( 'srem_imm', """ Signed integer remainder with immediate divisor. - This instruction never traps because a divisor of 0 or -1 is not - allowed. """, + This operation traps if the divisor is zero. + """, ins=(x, Y), outs=a) irsub_imm = Instruction( From 4930ef8ba80071db0a619fa93832bf4eee6e53ab Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Sun, 1 Jul 2018 13:36:04 -0700 Subject: [PATCH 1879/3084] Remove srclocs from a regalloc testcase. These aren't relevant to the testcase. --- .../filetests/regalloc/coloring-227.cton | 154 +++++++++--------- 1 file changed, 77 insertions(+), 77 deletions(-) diff --git a/cranelift/filetests/regalloc/coloring-227.cton b/cranelift/filetests/regalloc/coloring-227.cton index 248f75dbef..e5aeb3aa82 100644 --- a/cranelift/filetests/regalloc/coloring-227.cton +++ b/cranelift/filetests/regalloc/coloring-227.cton @@ -5,96 +5,96 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) gv0 = vmctx heap0 = static gv0, min 0, bound 0x0001_0000_0000, guard 0x8000_0000 - ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i64): -@0001 [RexOp1pu_id#b8] v5 = iconst.i32 0 -@0003 [RexOp1pu_id#b8] v6 = iconst.i32 0 -@0005 [RexOp1tjccb#74] brz v6, ebb10 -@0007 [RexOp1jmpb#eb] jump ebb3(v5, v5, v5, v5, v5, v5, v0, v1, v2, v3) + ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i64): +[RexOp1pu_id#b8] v5 = iconst.i32 0 +[RexOp1pu_id#b8] v6 = iconst.i32 0 +[RexOp1tjccb#74] brz v6, ebb10 +[RexOp1jmpb#eb] jump ebb3(v5, v5, v5, v5, v5, v5, v0, v1, v2, v3) - ebb3(v15: i32, v17: i32, v25: i32, v31: i32, v40: i32, v47: i32, v54: i32, v61: i32, v68: i32, v75: i32): -@000b [RexOp1jmpb#eb] jump ebb6 + ebb3(v15: i32, v17: i32, v25: i32, v31: i32, v40: i32, v47: i32, v54: i32, v61: i32, v68: i32, v75: i32): +[RexOp1jmpb#eb] jump ebb6 - ebb6: -@000d [RexOp1pu_id#b8] v8 = iconst.i32 0 -@000f [RexOp1tjccb#75] brnz v8, ebb5 -@0011 [RexOp1pu_id#b8] v9 = iconst.i32 0 -@0015 [RexOp1pu_id#b8] v11 = iconst.i32 0 -@0017 [RexOp1icscc#39] v12 = icmp.i32 eq v15, v11 -@0017 [RexOp2urm_noflags#4b6] v13 = bint.i32 v12 -@001a [RexOp1rr#21] v14 = band v9, v13 -@001b [RexOp1tjccb#75] brnz v14, ebb6 -@001d [RexOp1jmpb#eb] jump ebb7 + ebb6: +[RexOp1pu_id#b8] v8 = iconst.i32 0 +[RexOp1tjccb#75] brnz v8, ebb5 +[RexOp1pu_id#b8] v9 = iconst.i32 0 +[RexOp1pu_id#b8] v11 = iconst.i32 0 +[RexOp1icscc#39] v12 = icmp.i32 eq v15, v11 +[RexOp2urm_noflags#4b6] v13 = bint.i32 v12 +[RexOp1rr#21] v14 = band v9, v13 +[RexOp1tjccb#75] brnz v14, ebb6 +[RexOp1jmpb#eb] jump ebb7 - ebb7: -@0020 [RexOp1tjccb#74] brz.i32 v17, ebb8 -@0022 [RexOp1pu_id#b8] v18 = iconst.i32 0 -@0024 [RexOp1tjccb#74] brz v18, ebb9 -@0028 [RexOp1pu_id#b8] v21 = iconst.i32 0 -@002a [RexOp1umr#89] v79 = uextend.i64 v5 -@002a [RexOp1r_ib#8083] v80 = iadd_imm.i64 v4, 0 -@002a [RexOp1ld#808b] v81 = load.i64 v80 -@002a [RexOp1rr#8001] v22 = iadd v81, v79 -@002a [RexMp1st#189] istore16 v21, v22 -@002d [RexOp1jmpb#eb] jump ebb9 + ebb7: +[RexOp1tjccb#74] brz.i32 v17, ebb8 +[RexOp1pu_id#b8] v18 = iconst.i32 0 +[RexOp1tjccb#74] brz v18, ebb9 +[RexOp1pu_id#b8] v21 = iconst.i32 0 +[RexOp1umr#89] v79 = uextend.i64 v5 +[RexOp1r_ib#8083] v80 = iadd_imm.i64 v4, 0 +[RexOp1ld#808b] v81 = load.i64 v80 +[RexOp1rr#8001] v22 = iadd v81, v79 +[RexMp1st#189] istore16 v21, v22 +[RexOp1jmpb#eb] jump ebb9 - ebb9: -@002e [RexOp1jmpb#eb] jump ebb8 + ebb9: +[RexOp1jmpb#eb] jump ebb8 - ebb8: -@0033 [RexOp1pu_id#b8] v27 = iconst.i32 3 -@0035 [RexOp1pu_id#b8] v28 = iconst.i32 4 -@003b [RexOp1rr#09] v35 = bor.i32 v31, v13 -@003c [RexOp1tjccb#75] brnz v35, ebb15(v27) -@003c [RexOp1jmpb#eb] jump ebb15(v28) + ebb8: +[RexOp1pu_id#b8] v27 = iconst.i32 3 +[RexOp1pu_id#b8] v28 = iconst.i32 4 +[RexOp1rr#09] v35 = bor.i32 v31, v13 +[RexOp1tjccb#75] brnz v35, ebb15(v27) +[RexOp1jmpb#eb] jump ebb15(v28) - ebb15(v36: i32): -@003f [RexOp1jmpb#eb] jump ebb3(v25, v36, v25, v31, v40, v47, v54, v61, v68, v75) + ebb15(v36: i32): +[RexOp1jmpb#eb] jump ebb3(v25, v36, v25, v31, v40, v47, v54, v61, v68, v75) - ebb5: -@0042 [RexOp1jmpb#eb] jump ebb4 + ebb5: +[RexOp1jmpb#eb] jump ebb4 - ebb4: -@0045 [RexOp1jmpb#eb] jump ebb2(v40, v47, v54, v61, v68, v75) + ebb4: +[RexOp1jmpb#eb] jump ebb2(v40, v47, v54, v61, v68, v75) - ebb10: -@0046 [RexOp1pu_id#b8] v43 = iconst.i32 0 -@0048 [RexOp1jmpb#eb] jump ebb2(v43, v5, v0, v1, v2, v3) + ebb10: +[RexOp1pu_id#b8] v43 = iconst.i32 0 +[RexOp1jmpb#eb] jump ebb2(v43, v5, v0, v1, v2, v3) - ebb2(v7: i32, v45: i32, v52: i32, v59: i32, v66: i32, v73: i32): -@004c [RexOp1pu_id#b8] v44 = iconst.i32 0 -@004e [RexOp1tjccb#74] brz v44, ebb12 -@0052 [RexOp1pu_id#b8] v50 = iconst.i32 11 -@0054 [RexOp1tjccb#74] brz v50, ebb14 -@0058 [RexOp1umr#89] v82 = uextend.i64 v52 -@0058 [RexOp1r_ib#8083] v83 = iadd_imm.i64 v4, 0 -@0058 [RexOp1ld#808b] v84 = load.i64 v83 -@0058 [RexOp1rr#8001] v57 = iadd v84, v82 -@0058 [RexOp1ld#8b] v58 = load.i32 v57 -@005d [RexOp1umr#89] v85 = uextend.i64 v58 -@005d [RexOp1r_ib#8083] v86 = iadd_imm.i64 v4, 0 -@005d [RexOp1ld#808b] v87 = load.i64 v86 -@005d [RexOp1rr#8001] v64 = iadd v87, v85 -@005d [RexOp1st#88] istore8 v59, v64 -@0060 [RexOp1pu_id#b8] v65 = iconst.i32 0 -@0062 [RexOp1jmpb#eb] jump ebb13(v65) + ebb2(v7: i32, v45: i32, v52: i32, v59: i32, v66: i32, v73: i32): +[RexOp1pu_id#b8] v44 = iconst.i32 0 +[RexOp1tjccb#74] brz v44, ebb12 +[RexOp1pu_id#b8] v50 = iconst.i32 11 +[RexOp1tjccb#74] brz v50, ebb14 +[RexOp1umr#89] v82 = uextend.i64 v52 +[RexOp1r_ib#8083] v83 = iadd_imm.i64 v4, 0 +[RexOp1ld#808b] v84 = load.i64 v83 +[RexOp1rr#8001] v57 = iadd v84, v82 +[RexOp1ld#8b] v58 = load.i32 v57 +[RexOp1umr#89] v85 = uextend.i64 v58 +[RexOp1r_ib#8083] v86 = iadd_imm.i64 v4, 0 +[RexOp1ld#808b] v87 = load.i64 v86 +[RexOp1rr#8001] v64 = iadd v87, v85 +[RexOp1st#88] istore8 v59, v64 +[RexOp1pu_id#b8] v65 = iconst.i32 0 +[RexOp1jmpb#eb] jump ebb13(v65) - ebb14: -@0065 [RexOp1jmpb#eb] jump ebb13(v66) + ebb14: +[RexOp1jmpb#eb] jump ebb13(v66) - ebb13(v51: i32): -@0066 [RexOp1umr#89] v88 = uextend.i64 v45 -@0066 [RexOp1r_ib#8083] v89 = iadd_imm.i64 v4, 0 -@0066 [RexOp1ld#808b] v90 = load.i64 v89 -@0066 [RexOp1rr#8001] v71 = iadd v90, v88 -@0066 [RexOp1st#89] store v51, v71 -@0069 [RexOp1jmpb#eb] jump ebb12 + ebb13(v51: i32): +[RexOp1umr#89] v88 = uextend.i64 v45 +[RexOp1r_ib#8083] v89 = iadd_imm.i64 v4, 0 +[RexOp1ld#808b] v90 = load.i64 v89 +[RexOp1rr#8001] v71 = iadd v90, v88 +[RexOp1st#89] store v51, v71 +[RexOp1jmpb#eb] jump ebb12 - ebb12: -@006a [RexOp1jmpb#eb] jump ebb11 + ebb12: +[RexOp1jmpb#eb] jump ebb11 - ebb11: -@006e [RexOp1jmpb#eb] jump ebb1 + ebb11: +[RexOp1jmpb#eb] jump ebb1 - ebb1: -@006e [Op1ret#c3] return + ebb1: +[Op1ret#c3] return } From ff5660624d116b18603c04d7a4af5fe446b733d8 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Mon, 2 Jul 2018 22:37:40 -0700 Subject: [PATCH 1880/3084] Clarify wording in a comment. --- lib/codegen/src/regalloc/liveness.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/codegen/src/regalloc/liveness.rs b/lib/codegen/src/regalloc/liveness.rs index 959e48183a..609b71c156 100644 --- a/lib/codegen/src/regalloc/liveness.rs +++ b/lib/codegen/src/regalloc/liveness.rs @@ -42,9 +42,9 @@ //! This algorithm has some disadvantages that makes us look elsewhere: //! //! - Quadratic memory use. We need a bit per variable per basic block in the function. -//! - Sparse representation. In practice, the majority of SSA values never leave their basic block, -//! and those that do span basic blocks rarely span a large number of basic blocks. This makes -//! the bit-vectors quite sparse. +//! - Dense representation of sparse data. In practice, the majority of SSA values never leave +//! their basic block, and those that do span basic blocks rarely span a large number of basic +//! blocks. This makes the data stored in the bitvectors quite sparse. //! - Traditionally, the data-flow equations were solved for real program *variables* which does //! not include temporaries used in evaluating expressions. We have an SSA form program which //! blurs the distinction between temporaries and variables. This makes the quadratic memory From e5014e0fff6b2e96463ae763bfa2f0b61805ec4e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gr=C3=A9goire=20Geis?= Date: Wed, 4 Jul 2018 05:54:54 +0200 Subject: [PATCH 1881/3084] Made Capstone an optional dependency (fixes #382) (#383) * Made Capstone an optional dependency (fixes #382). * Introduced feature 'disas' for disassembly (related to #382). * Made 'disas' a default feature in cretonne-tools. * Fixed errors in src/compile.rs introduced by get_disassembler changes. - Moves `use` statements before the function declaration. - Returns an error if the disassembler cannot be found created. --- cranelift/Cargo.toml | 5 ++- cranelift/src/compile.rs | 89 ++++++++++++++++++++++---------------- cranelift/src/cton-util.rs | 1 + 3 files changed, 56 insertions(+), 39 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 8c1730e6a8..ea4d05fcae 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -29,12 +29,13 @@ docopt = "1" serde = "1.0.8" serde_derive = "1.0.8" term = "0.5.1" -capstone = "0.4" +capstone = { version = "0.4", optional = true } wabt = { version = "0.3", optional = true } target-lexicon = "0.0.2" [features] -default = ["wasm"] +default = ["disas", "wasm"] +disas = ["capstone"] wasm = ["wabt", "cretonne-wasm"] [workspace] diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 516a1be84a..98ad672f34 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -1,6 +1,5 @@ //! CLI tool to read Cretonne IR files and compile them into native code. -use capstone::prelude::*; use cretonne_codegen::isa::TargetIsa; use cretonne_codegen::print_errors::pretty_error; use cretonne_codegen::settings::FlagsOrIsa; @@ -9,7 +8,6 @@ use cretonne_codegen::{binemit, ir}; use cretonne_reader::parse_test; use std::path::Path; use std::path::PathBuf; -use target_lexicon::Architecture; use utils::{parse_sets_and_triple, read_to_string}; struct PrintRelocs { @@ -121,8 +119,53 @@ fn handle_module( } print!("{}", byte); } - println!(); + println!(); + print_disassembly(isa, &mem)?; + } + } + + Ok(()) +} + +cfg_if! { + if #[cfg(feature = "disas")] { + use capstone::prelude::*; + use target_lexicon::Architecture; + + fn get_disassembler(isa: &TargetIsa) -> Result { + let cs = match isa.triple().architecture { + Architecture::Riscv32 | Architecture::Riscv64 => { + return Err(String::from("No disassembler for RiscV")) + } + Architecture::I386 | Architecture::I586 | Architecture::I686 => Capstone::new() + .x86() + .mode(arch::x86::ArchMode::Mode32) + .build(), + Architecture::X86_64 => Capstone::new() + .x86() + .mode(arch::x86::ArchMode::Mode64) + .build(), + Architecture::Arm + | Architecture::Armv4t + | Architecture::Armv5te + | Architecture::Armv7 + | Architecture::Armv7s => Capstone::new().arm().mode(arch::arm::ArchMode::Arm).build(), + Architecture::Thumbv6m | Architecture::Thumbv7em | Architecture::Thumbv7m => Capstone::new( + ).arm() + .mode(arch::arm::ArchMode::Thumb) + .build(), + Architecture::Aarch64 => Capstone::new() + .arm64() + .mode(arch::arm64::ArchMode::Arm) + .build(), + _ => return Err(String::from("Unknown ISA")), + }; + + cs.map_err(|err| err.to_string()) + } + + fn print_disassembly(isa: &TargetIsa, mem: &[u8]) -> Result<(), String> { let mut cs = get_disassembler(isa)?; println!("\nDisassembly:"); @@ -130,40 +173,12 @@ fn handle_module( for i in insns.iter() { println!("{}", i); } + Ok(()) + } + } else { + fn print_disassembly(_: &TargetIsa, _: &[u8]) -> Result<(), String> { + println!("\nNo disassembly available."); + Ok(()) } } - - Ok(()) -} - -fn get_disassembler(isa: &TargetIsa) -> Result { - let cs = match isa.triple().architecture { - Architecture::Riscv32 | Architecture::Riscv64 => { - return Err(String::from("No disassembler for RiscV")) - } - Architecture::I386 | Architecture::I586 | Architecture::I686 => Capstone::new() - .x86() - .mode(arch::x86::ArchMode::Mode32) - .build(), - Architecture::X86_64 => Capstone::new() - .x86() - .mode(arch::x86::ArchMode::Mode64) - .build(), - Architecture::Arm - | Architecture::Armv4t - | Architecture::Armv5te - | Architecture::Armv7 - | Architecture::Armv7s => Capstone::new().arm().mode(arch::arm::ArchMode::Arm).build(), - Architecture::Thumbv6m | Architecture::Thumbv7em | Architecture::Thumbv7m => Capstone::new( - ).arm() - .mode(arch::arm::ArchMode::Thumb) - .build(), - Architecture::Aarch64 => Capstone::new() - .arm64() - .mode(arch::arm64::ArchMode::Arm) - .build(), - _ => return Err(String::from("Unknown ISA")), - }; - - cs.map_err(|err| err.to_string()) } diff --git a/cranelift/src/cton-util.rs b/cranelift/src/cton-util.rs index 370e6d5c65..ecb461ea99 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/cton-util.rs @@ -17,6 +17,7 @@ extern crate docopt; extern crate filecheck; #[macro_use] extern crate serde_derive; +#[cfg(feature = "disas")] extern crate capstone; extern crate term; From dd72b54eef5db20d136b5a1a3b4ed8792bb97843 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gr=C3=A9goire=20Geis?= Date: Wed, 4 Jul 2018 05:59:32 +0200 Subject: [PATCH 1882/3084] Now diagnosing missing vmctx arguments (fixes #376) (#384) * Now diagnosing missing vmctx arguments (fixes #376). * Added filetest for fix of #376. * Respect formatting rules in verifier/mod.rs. * Added parameters for each use of vmctx in test files. * Added comments on additions on vmctx verifications. --- cranelift/docs/heapex-dyn.cton | 4 +-- cranelift/docs/heapex-sm32.cton | 4 +-- cranelift/docs/heapex-sm64.cton | 4 +-- cranelift/filetests/parser/memory.cton | 32 +++++++++---------- .../filetests/verifier/undeclared_vmctx.cton | 17 ++++++++++ lib/codegen/src/verifier/mod.rs | 14 +++++++- 6 files changed, 52 insertions(+), 23 deletions(-) create mode 100644 cranelift/filetests/verifier/undeclared_vmctx.cton diff --git a/cranelift/docs/heapex-dyn.cton b/cranelift/docs/heapex-dyn.cton index 9f1f2ac078..7ad0cd933f 100644 --- a/cranelift/docs/heapex-dyn.cton +++ b/cranelift/docs/heapex-dyn.cton @@ -1,11 +1,11 @@ test verifier -function %add_members(i32) -> f32 baldrdash { +function %add_members(i32, i64 vmctx) -> f32 baldrdash { gv0 = vmctx+64 gv1 = vmctx+72 heap0 = dynamic gv0, min 0x1000, bound gv1, guard 0 -ebb0(v0: i32): +ebb0(v0: i32, v6: i64): v1 = heap_addr.i64 heap0, v0, 20 v2 = load.f32 v1+16 v3 = heap_addr.i64 heap0, v0, 24 diff --git a/cranelift/docs/heapex-sm32.cton b/cranelift/docs/heapex-sm32.cton index 61014d05f5..9ac1c8bfbd 100644 --- a/cranelift/docs/heapex-sm32.cton +++ b/cranelift/docs/heapex-sm32.cton @@ -1,10 +1,10 @@ test verifier -function %add_members(i32) -> f32 baldrdash { +function %add_members(i32, i32 vmctx) -> f32 baldrdash { gv0 = vmctx+64 heap0 = static gv0, min 0x1000, bound 0x10_0000, guard 0x1000 -ebb0(v0: i32): +ebb0(v0: i32, v5: i32): v1 = heap_addr.i32 heap0, v0, 1 v2 = load.f32 v1+16 v3 = load.f32 v1+20 diff --git a/cranelift/docs/heapex-sm64.cton b/cranelift/docs/heapex-sm64.cton index 4755e8790a..c9057b6bb0 100644 --- a/cranelift/docs/heapex-sm64.cton +++ b/cranelift/docs/heapex-sm64.cton @@ -1,10 +1,10 @@ test verifier -function %add_members(i32) -> f32 baldrdash { +function %add_members(i32, i64 vmctx) -> f32 baldrdash { gv0 = vmctx+64 heap0 = static gv0, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000 -ebb0(v0: i32): +ebb0(v0: i32, v5: i64): v1 = heap_addr.i64 heap0, v0, 1 v2 = load.f32 v1+16 v3 = load.f32 v1+20 diff --git a/cranelift/filetests/parser/memory.cton b/cranelift/filetests/parser/memory.cton index 6df76ce561..6c9dd6440f 100644 --- a/cranelift/filetests/parser/memory.cton +++ b/cranelift/filetests/parser/memory.cton @@ -1,7 +1,7 @@ test cat test verifier -function %vmglobal() -> i32 { +function %vmglobal(i64 vmctx) -> i32 { gv3 = vmctx+16 ; check: gv3 = vmctx+16 gv4 = vmctx+0 @@ -9,29 +9,29 @@ function %vmglobal() -> i32 { ; not: +0 gv5 = vmctx -256 ; check: gv5 = vmctx-256 -ebb0: +ebb0(v0: i64): v1 = global_value.i32 gv3 ; check: v1 = global_value.i32 gv3 return v1 } -function %deref() -> i32 { +function %deref(i64 vmctx) -> i32 { gv3 = vmctx+16 gv4 = deref(gv3)-32 ; check: gv4 = deref(gv3)-32 -ebb0: +ebb0(v0: i64): v1 = global_value.i32 gv4 ; check: v1 = global_value.i32 gv4 return v1 } ; Refer to a global value before it's been declared. -function %backref() -> i32 { +function %backref(i64 vmctx) -> i32 { gv1 = deref(gv2)-32 ; check: gv1 = deref(gv2)-32 gv2 = vmctx+16 ; check: gv2 = vmctx+16 -ebb0: +ebb0(v0: i64): v1 = global_value.i32 gv1 return v1 } @@ -51,21 +51,21 @@ ebb0: } ; Declare static heaps. -function %sheap(i32) -> i64 { +function %sheap(i32, i64 vmctx) -> i64 { heap1 = static reserved_reg, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000 heap2 = static gv5, guard 0x1000, bound 0x1_0000 gv5 = vmctx+64 ; check: heap1 = static reserved_reg, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000 ; check: heap2 = static gv5, min 0, bound 0x0001_0000, guard 4096 -ebb0(v1: i32): - v2 = heap_addr.i64 heap1, v1, 0 - ; check: v2 = heap_addr.i64 heap1, v1, 0 - return v2 +ebb0(v1: i32, v2: i64): + v3 = heap_addr.i64 heap1, v1, 0 + ; check: v3 = heap_addr.i64 heap1, v1, 0 + return v3 } ; Declare dynamic heaps. -function %dheap(i32) -> i64 { +function %dheap(i32, i64 vmctx) -> i64 { heap1 = dynamic reserved_reg, min 0x1_0000, bound gv6, guard 0x8000_0000 heap2 = dynamic gv5, bound gv6, guard 0x1000 gv5 = vmctx+64 @@ -73,8 +73,8 @@ function %dheap(i32) -> i64 { ; check: heap1 = dynamic reserved_reg, min 0x0001_0000, bound gv6, guard 0x8000_0000 ; check: heap2 = dynamic gv5, min 0, bound gv6, guard 4096 -ebb0(v1: i32): - v2 = heap_addr.i64 heap2, v1, 0 - ; check: v2 = heap_addr.i64 heap2, v1, 0 - return v2 +ebb0(v1: i32, v2: i64): + v3 = heap_addr.i64 heap2, v1, 0 + ; check: v3 = heap_addr.i64 heap2, v1, 0 + return v3 } diff --git a/cranelift/filetests/verifier/undeclared_vmctx.cton b/cranelift/filetests/verifier/undeclared_vmctx.cton new file mode 100644 index 0000000000..47f1192569 --- /dev/null +++ b/cranelift/filetests/verifier/undeclared_vmctx.cton @@ -0,0 +1,17 @@ +test verifier + +; Using a vmctx global value without declaring it first leads to an error. +function %vmglobal_err(i64) -> i64 { + gv4 = vmctx+0 ; error: undeclared vmctx reference +ebb0(v0: i64): + v1 = global_value.i64 gv4 + return v1 +} + +; If it is declared, all is fine. +function %vmglobal_ok(i64 vmctx) -> i64 { + gv4 = vmctx+0 +ebb0(v0: i64): + v1 = global_value.i64 gv4 + return v1 +} diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs index 4585cc8176..aa3923456c 100644 --- a/lib/codegen/src/verifier/mod.rs +++ b/lib/codegen/src/verifier/mod.rs @@ -44,6 +44,7 @@ //! Global values //! //! - Detect cycles in deref(base) declarations. +//! - Detect use of 'vmctx' global value when no corresponding parameter is defined. //! //! TODO: //! Ad hoc checking @@ -170,7 +171,9 @@ impl<'a> Verifier<'a> { } } - // Check for cycles in the global value declarations. + // Check for: + // - cycles in the global value declarations. + // - use of 'vmctx' when no special parameter declares it. fn verify_global_values(&self) -> VerifierResult<()> { let mut seen = SparseSet::new(); @@ -186,6 +189,15 @@ impl<'a> Verifier<'a> { cur = base; } + + if let ir::GlobalValueData::VMContext { .. } = self.func.global_values[cur] { + if self.func + .special_param(ir::ArgumentPurpose::VMContext) + .is_none() + { + return err!(cur, "undeclared vmctx reference {}", cur); + } + } } Ok(()) From 5db45d26ccf48d90f92ec79a539854ff0d8fd434 Mon Sep 17 00:00:00 2001 From: bjorn3 Date: Wed, 4 Jul 2018 15:31:00 +0200 Subject: [PATCH 1883/3084] Legalize several i8 insts (#380) * Legalize several i8 insts * X86: implement regmove.{i8,i16} * Legalize bnot * Remove comments * Nicer type param binding in legalize.py * Legalize sdiv_imm.i8 * Hopefully fix mypy error * Add missing trailing newlines * Fix tests --- .../filetests/isa/x86/legalize-bnot.cton | 28 +++++ .../filetests/isa/x86/legalize-br-table.cton | 17 +++ .../filetests/isa/x86/legalize-iconst-i8.cton | 18 +++ .../filetests/isa/x86/legalize-imul-i8.cton | 11 ++ .../isa/x86/legalize-imul-imm-i8.cton | 15 +++ .../isa/x86/legalize-load-store-i8.cton | 31 +++++ .../isa/x86/legalize-regmove-i8.cton | 36 ++++++ lib/codegen/meta/base/legalize.py | 112 +++++++++++++++++- lib/codegen/meta/isa/x86/encodings.py | 12 +- 9 files changed, 275 insertions(+), 5 deletions(-) create mode 100644 cranelift/filetests/isa/x86/legalize-bnot.cton create mode 100644 cranelift/filetests/isa/x86/legalize-br-table.cton create mode 100644 cranelift/filetests/isa/x86/legalize-iconst-i8.cton create mode 100644 cranelift/filetests/isa/x86/legalize-imul-i8.cton create mode 100644 cranelift/filetests/isa/x86/legalize-imul-imm-i8.cton create mode 100644 cranelift/filetests/isa/x86/legalize-load-store-i8.cton create mode 100644 cranelift/filetests/isa/x86/legalize-regmove-i8.cton diff --git a/cranelift/filetests/isa/x86/legalize-bnot.cton b/cranelift/filetests/isa/x86/legalize-bnot.cton new file mode 100644 index 0000000000..07a41dfdc4 --- /dev/null +++ b/cranelift/filetests/isa/x86/legalize-bnot.cton @@ -0,0 +1,28 @@ +test compile + +target x86_64 + +function u0:51(i64, i64) system_v { + ss0 = explicit_slot 0 + ss1 = explicit_slot 1 + ss2 = explicit_slot 1 + ss3 = explicit_slot 1 + +ebb0(v0: i64, v1: i64): + v2 = stack_addr.i64 ss1 + v3 = load.i8 v1 + store v3, v2 + v4 = stack_addr.i64 ss2 + v5 = stack_addr.i64 ss3 + jump ebb1 + +ebb1: + v6 = load.i8 v2 + store v6, v5 + v7 = load.i8 v5 + v8 = bnot v7 + store v8, v4 + v9 = load.i8 v4 + store v9, v0 + return +} diff --git a/cranelift/filetests/isa/x86/legalize-br-table.cton b/cranelift/filetests/isa/x86/legalize-br-table.cton new file mode 100644 index 0000000000..3047acd9d1 --- /dev/null +++ b/cranelift/filetests/isa/x86/legalize-br-table.cton @@ -0,0 +1,17 @@ +test compile + +target x86_64 + +function u0:0(i64) system_v { + ss0 = explicit_slot 1 + jt0 = jump_table ebb1 + +ebb0(v0: i64): + v1 = stack_addr.i64 ss0 + v2 = load.i8 v1 + br_table v2, jt0 + jump ebb1 + +ebb1: + return +} diff --git a/cranelift/filetests/isa/x86/legalize-iconst-i8.cton b/cranelift/filetests/isa/x86/legalize-iconst-i8.cton new file mode 100644 index 0000000000..8245ee73ce --- /dev/null +++ b/cranelift/filetests/isa/x86/legalize-iconst-i8.cton @@ -0,0 +1,18 @@ +test compile + +target x86_64 + +function u0:0(i64) system_v { + ss0 = explicit_slot 0 + +ebb0(v0: i64): + jump ebb1 + +ebb1: +; _0 = const 42u8 + v1 = iconst.i8 42 + store v1, v0 +; +; return + return +} diff --git a/cranelift/filetests/isa/x86/legalize-imul-i8.cton b/cranelift/filetests/isa/x86/legalize-imul-i8.cton new file mode 100644 index 0000000000..d56ff787eb --- /dev/null +++ b/cranelift/filetests/isa/x86/legalize-imul-i8.cton @@ -0,0 +1,11 @@ +test compile + +target x86_64 + +function u0:0(i64, i8, i8) system_v { + +ebb0(v0: i64, v1: i8, v2: i8): + v11 = imul v1, v2 + store v11, v0 + return +} diff --git a/cranelift/filetests/isa/x86/legalize-imul-imm-i8.cton b/cranelift/filetests/isa/x86/legalize-imul-imm-i8.cton new file mode 100644 index 0000000000..6655c562e7 --- /dev/null +++ b/cranelift/filetests/isa/x86/legalize-imul-imm-i8.cton @@ -0,0 +1,15 @@ +test compile + +target x86_64 + +function u0:0(i64, i8) system_v { + ss0 = explicit_slot 1 + +ebb0(v0: i64, v1: i8): + v3 = stack_addr.i64 ss0 + v5 = load.i8 v3 + v6 = iconst.i8 2 + v7 = imul_imm v5, 42 + store v7, v0 + return +} diff --git a/cranelift/filetests/isa/x86/legalize-load-store-i8.cton b/cranelift/filetests/isa/x86/legalize-load-store-i8.cton new file mode 100644 index 0000000000..cecf0e145f --- /dev/null +++ b/cranelift/filetests/isa/x86/legalize-load-store-i8.cton @@ -0,0 +1,31 @@ +test compile + +target x86_64 + +function u0:0(i64, i8, i8) system_v { + ss0 = explicit_slot 0 + ss1 = explicit_slot 1 + ss2 = explicit_slot 1 + ss3 = explicit_slot 1 + ss4 = explicit_slot 1 + +ebb0(v0: i64, v1: i8, v2: i8): + v3 = stack_addr.i64 ss1 + store v1, v3 + v4 = stack_addr.i64 ss2 + store v2, v4 + v5 = stack_addr.i64 ss3 + v6 = stack_addr.i64 ss4 + jump ebb1 + +ebb1: + v7 = load.i8 v3 + store v7, v5 + v8 = load.i8 v4 + store v8, v6 + v9 = load.i8 v5 + v10 = load.i8 v6 + v11 = imul v9, v10 + store v11, v0 + return +} diff --git a/cranelift/filetests/isa/x86/legalize-regmove-i8.cton b/cranelift/filetests/isa/x86/legalize-regmove-i8.cton new file mode 100644 index 0000000000..8dc746d701 --- /dev/null +++ b/cranelift/filetests/isa/x86/legalize-regmove-i8.cton @@ -0,0 +1,36 @@ +test compile + +target x86_64 + +function u0:0(i64, i64, i64) system_v { + ss0 = explicit_slot 0 + ss1 = explicit_slot 8 + ss2 = explicit_slot 8 + ss3 = explicit_slot 2 + ss4 = explicit_slot 8 + sig0 = (i64, i16, i64) system_v + fn0 = colocated u0:11 sig0 + +ebb0(v0: i64, v1: i64, v2: i64): + v3 = stack_addr.i64 ss1 + store v1, v3 + v4 = stack_addr.i64 ss2 + store v2, v4 + v5 = stack_addr.i64 ss3 + v6 = stack_addr.i64 ss4 + jump ebb1 + +ebb1: + v7 = load.i64 v3 + v8 = load.i16 v7 + store v8, v5 + v9 = load.i64 v4 + store v9, v6 + v10 = load.i16 v5 + v11 = load.i64 v6 + call fn0(v0, v10, v11) + jump ebb2 + +ebb2: + return +} diff --git a/lib/codegen/meta/base/legalize.py b/lib/codegen/meta/base/legalize.py index f575418f8d..1bcb492922 100644 --- a/lib/codegen/meta/base/legalize.py +++ b/lib/codegen/meta/base/legalize.py @@ -10,6 +10,7 @@ from __future__ import absolute_import from .immediates import intcc, imm64, ieee32, ieee64 from . import instructions as insts from . import types +from .instructions import uextend, sextend, ireduce from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm from .instructions import isub, isub_bin, isub_bout, isub_borrow, irsub_imm from .instructions import imul, imul_imm @@ -23,6 +24,8 @@ from .instructions import iconst, bint, select from .instructions import ishl, ishl_imm, sshr, sshr_imm, ushr, ushr_imm from .instructions import rotl, rotl_imm, rotr, rotr_imm from .instructions import f32const, f64const +from .instructions import store, load +from .instructions import br_table from cdsl.ast import Var from cdsl.xform import Rtl, XFormGroup @@ -41,8 +44,6 @@ widen = XFormGroup('widen', """ The transformations in the 'widen' group work by expressing instructions in terms of larger types. - - This group is not yet implemented. """) expand = XFormGroup('expand', """ @@ -99,6 +100,7 @@ c1 = Var('c1') c2 = Var('c2') c_in = Var('c_in') c_int = Var('c_int') +d = Var('d') xl = Var('xl') xh = Var('xh') yl = Var('yl') @@ -106,6 +108,10 @@ yh = Var('yh') al = Var('al') ah = Var('ah') cc = Var('cc') +ptr = Var('ptr') +flags = Var('flags') +offset = Var('off') +ss = Var('ss') narrow.legalize( a << iadd(x, y), @@ -148,6 +154,108 @@ narrow.legalize( a << iconcat(al, ah) )) +for int_ty in [types.i8, types.i16]: + widen.legalize( + a << iconst.bind(int_ty)(b), + Rtl( + c << iconst.i32(b), + a << ireduce.bind(int_ty)(c) + )) + +widen.legalize( + store.i8(flags, a, ptr, offset), + Rtl( + b << uextend.i32(a), + insts.istore8(flags, b, ptr, offset) + )) + +widen.legalize( + store.i16(flags, a, ptr, offset), + Rtl( + b << uextend.i32(a), + insts.istore16(flags, b, ptr, offset) + )) + +widen.legalize( + a << load.i8(flags, ptr, offset), + Rtl( + b << insts.uload8.i32(flags, ptr, offset), + a << ireduce(b) + )) + +widen.legalize( + a << load.i16(flags, ptr, offset), + Rtl( + b << insts.uload16.i32(flags, ptr, offset), + a << ireduce(b) + )) + +for binop in [iadd, isub, imul, udiv, band, bor, bxor]: + for int_ty in [types.i8, types.i16]: + widen.legalize( + a << binop.bind(int_ty)(x, y), + Rtl( + b << uextend.i32(x), + c << uextend.i32(y), + d << binop(b, c), + a << ireduce(d) + ) + ) + +for binop in [sdiv]: + for int_ty in [types.i8, types.i16]: + widen.legalize( + a << binop.bind(int_ty)(x, y), + Rtl( + b << sextend.i32(x), + c << sextend.i32(y), + d << binop(b, c), + a << ireduce(d) + ) + ) + +for unop in [bnot]: + for int_ty in [types.i8, types.i16]: + widen.legalize( + a << unop.bind(int_ty)(x), + Rtl( + b << sextend.i32(x), + d << unop(b), + a << ireduce(d) + ) + ) + +for binop in [iadd_imm, imul_imm, udiv_imm]: + for int_ty in [types.i8, types.i16]: + widen.legalize( + a << binop.bind(int_ty)(x, y), + Rtl( + b << uextend.i32(x), + c << binop(b, y), + a << ireduce(c) + ) + ) + +for binop in [sdiv_imm]: + for int_ty in [types.i8, types.i16]: + widen.legalize( + a << binop.bind(int_ty)(x, y), + Rtl( + b << sextend.i32(x), + c << binop(b, y), + a << ireduce(c) + ) + ) + +for int_ty in [types.i8, types.i16]: + widen.legalize( + br_table.bind(int_ty)(x, y), + Rtl( + b << uextend.i32(x), + br_table(b, y), + ) + ) + # Expand integer operations with carry for RISC architectures that don't have # the flags. expand.legalize( diff --git a/lib/codegen/meta/isa/x86/encodings.py b/lib/codegen/meta/isa/x86/encodings.py index 28e2f73740..1e406fb8fa 100644 --- a/lib/codegen/meta/isa/x86/encodings.py +++ b/lib/codegen/meta/isa/x86/encodings.py @@ -6,6 +6,7 @@ from cdsl.predicates import IsZero32BitFloat, IsZero64BitFloat from cdsl.predicates import IsUnsignedInt, Not, And from base.predicates import IsColocatedFunc, IsColocatedData, LengthEquals from base import instructions as base +from base import types from base.formats import UnaryIeee32, UnaryIeee64, UnaryImm from base.formats import FuncAddr, Call, LoadComplex, StoreComplex from .defs import X86_64, X86_32 @@ -13,7 +14,7 @@ from . import recipes as r from . import settings as cfg from . import instructions as x86 from .legalize import x86_expand -from base.legalize import narrow, expand_flags +from base.legalize import narrow, widen, expand_flags from base.settings import allones_funcaddrs, is_pic from .settings import use_sse41 @@ -30,6 +31,8 @@ X86_32.legalize_monomorphic(expand_flags) X86_32.legalize_type( default=narrow, b1=expand_flags, + i8=widen, + i16=widen, i32=x86_expand, f32=x86_expand, f64=x86_expand) @@ -38,6 +41,8 @@ X86_64.legalize_monomorphic(expand_flags) X86_64.legalize_type( default=narrow, b1=expand_flags, + i8=widen, + i16=widen, i32=x86_expand, i64=x86_expand, f32=x86_expand, @@ -172,8 +177,9 @@ enc_both(base.copy.b1, r.umr, 0x89) # For x86-64, only define REX forms for now, since we can't describe the # special regunit immediate operands with the current constraint language. -X86_32.enc(base.regmove.i32, *r.rmov(0x89)) -X86_64.enc(base.regmove.i32, *r.rmov.rex(0x89)) +for ty in [types.i8, types.i16, types.i32]: + X86_32.enc(base.regmove.bind(ty), *r.rmov(0x89)) + X86_64.enc(base.regmove.bind(ty), *r.rmov.rex(0x89)) X86_64.enc(base.regmove.i64, *r.rmov.rex(0x89, w=1)) enc_both(base.regmove.b1, r.rmov, 0x89) From c6badde836c708ff3bda6472ee239787529a6b8d Mon Sep 17 00:00:00 2001 From: Amanieu d'Antras Date: Fri, 6 Jul 2018 03:17:53 +0100 Subject: [PATCH 1884/3084] Update wabt to 0.4 (#389) This fixes compilation on system with GCC 8. --- cranelift/Cargo.toml | 2 +- lib/wasm/Cargo.toml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index ea4d05fcae..3de5d020bb 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -30,7 +30,7 @@ serde = "1.0.8" serde_derive = "1.0.8" term = "0.5.1" capstone = { version = "0.4", optional = true } -wabt = { version = "0.3", optional = true } +wabt = { version = "0.4", optional = true } target-lexicon = "0.0.2" [features] diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index aa9c2d772c..4fece39a1c 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -18,7 +18,7 @@ failure_derive = { version = "0.1.1", default-features = false } target-lexicon = { version = "0.0.2", default-features = false } [dev-dependencies] -wabt = "0.3" +wabt = "0.4" [features] default = ["std"] From 1987d4dba93046d214742f50ebd729a3561ed4f2 Mon Sep 17 00:00:00 2001 From: Benjamin Bouvier Date: Fri, 6 Jul 2018 21:04:00 +0200 Subject: [PATCH 1885/3084] Reject with a plain text error when no ISA is defined with the wasm command; (#391) And restructure code a bit to make it easier to understand. --- cranelift/src/wasm.rs | 63 ++++++++++++++++++++++++------------------- 1 file changed, 35 insertions(+), 28 deletions(-) diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 13356a4282..101d31be3a 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -79,44 +79,51 @@ fn handle_module( vprint!(flag_verbose, "Translating... "); terminal.reset().unwrap(); - let mut data = read_to_end(path.clone()).map_err(|err| String::from(err.description()))?; + let mut module_binary = read_to_end(path.clone()).map_err(|err| String::from(err.description()))?; - if !data.starts_with(&[b'\0', b'a', b's', b'm']) { - data = match wat2wasm(&data) { + if !module_binary.starts_with(&[b'\0', b'a', b's', b'm']) { + module_binary = match wat2wasm(&module_binary) { Ok(data) => data, Err(e) => return Err(String::from(e.description())), }; } + let isa = match fisa.isa { + Some(isa) => isa, + None => return Err(String::from("Error: the wasm command requires an explicit isa.")) + }; + let mut dummy_environ = - DummyEnvironment::with_triple_flags(fisa.isa.unwrap().triple().clone(), fisa.flags.clone()); - translate_module(&data, &mut dummy_environ).map_err(|e| e.to_string())?; + DummyEnvironment::with_triple_flags(isa.triple().clone(), fisa.flags.clone()); + translate_module(&module_binary, &mut dummy_environ).map_err(|e| e.to_string())?; terminal.fg(term::color::GREEN).unwrap(); vprintln!(flag_verbose, "ok"); terminal.reset().unwrap(); if flag_just_decode { - if flag_print { - let num_func_imports = dummy_environ.get_num_func_imports(); - for (def_index, func) in dummy_environ.info.function_bodies.iter().enumerate() { - let func_index = num_func_imports + def_index; - let mut context = Context::new(); - context.func = func.clone(); - if let Some(start_func) = dummy_environ.info.start_func { - if func_index == start_func { - println!("; Selected as wasm start function"); - } - } - vprintln!(flag_verbose, ""); - for export_name in &dummy_environ.info.functions[func_index].export_names { - println!("; Exported as \"{}\"", export_name); - } - println!("{}", context.func.display(None)); - vprintln!(flag_verbose, ""); - } - terminal.reset().unwrap(); + if !flag_print { + return Ok(()); } + + let num_func_imports = dummy_environ.get_num_func_imports(); + for (def_index, func) in dummy_environ.info.function_bodies.iter().enumerate() { + let func_index = num_func_imports + def_index; + let mut context = Context::new(); + context.func = func.clone(); + if let Some(start_func) = dummy_environ.info.start_func { + if func_index == start_func { + println!("; Selected as wasm start function"); + } + } + vprintln!(flag_verbose, ""); + for export_name in &dummy_environ.info.functions[func_index].export_names { + println!("; Exported as \"{}\"", export_name); + } + println!("{}", context.func.display(None)); + vprintln!(flag_verbose, ""); + } + terminal.reset().unwrap(); return Ok(()); } @@ -135,14 +142,15 @@ fn handle_module( let num_func_imports = dummy_environ.get_num_func_imports(); let mut total_module_code_size = 0; for (def_index, func) in dummy_environ.info.function_bodies.iter().enumerate() { - let func_index = num_func_imports + def_index; let mut context = Context::new(); context.func = func.clone(); + + let func_index = num_func_imports + def_index; if flag_check_translation { context .verify(fisa) .map_err(|err| pretty_verifier_error(&context.func, fisa.isa, &err))?; - } else if let Some(isa) = fisa.isa { + } else { let compiled_size = context .compile(isa) .map_err(|err| pretty_error(&context.func, fisa.isa, err))?; @@ -157,9 +165,8 @@ fn handle_module( func_index, dummy_environ.func_bytecode_sizes[def_index] ); } - } else { - return Err(String::from("compilation requires a target isa")); } + if flag_print { vprintln!(flag_verbose, ""); if let Some(start_func) = dummy_environ.info.start_func { From 4f352fa6f192f8c4b1b1589ddc90ca5cf1c32a0d Mon Sep 17 00:00:00 2001 From: bjorn3 Date: Fri, 6 Jul 2018 21:05:01 +0200 Subject: [PATCH 1886/3084] s/uextend/sextend/ for sload* (#390) --- lib/codegen/meta/base/instructions.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/codegen/meta/base/instructions.py b/lib/codegen/meta/base/instructions.py index 1d02b60cdc..7a63914dbc 100644 --- a/lib/codegen/meta/base/instructions.py +++ b/lib/codegen/meta/base/instructions.py @@ -311,7 +311,7 @@ sload8 = Instruction( 'sload8', r""" Load 8 bits from memory at ``p + Offset`` and sign-extend. - This is equivalent to ``load.i8`` followed by ``uextend``. + This is equivalent to ``load.i8`` followed by ``sextend``. """, ins=(MemFlags, p, Offset), outs=a, can_load=True) @@ -319,7 +319,7 @@ sload8_complex = Instruction( 'sload8_complex', r""" Load 8 bits from memory at ``sum(args) + Offset`` and sign-extend. - This is equivalent to ``load.i8`` followed by ``uextend``. + This is equivalent to ``load.i8`` followed by ``sextend``. """, ins=(MemFlags, args, Offset), outs=a, can_load=True) @@ -365,7 +365,7 @@ sload16 = Instruction( 'sload16', r""" Load 16 bits from memory at ``p + Offset`` and sign-extend. - This is equivalent to ``load.i16`` followed by ``uextend``. + This is equivalent to ``load.i16`` followed by ``sextend``. """, ins=(MemFlags, p, Offset), outs=a, can_load=True) @@ -373,7 +373,7 @@ sload16_complex = Instruction( 'sload16_complex', r""" Load 16 bits from memory at ``sum(args) + Offset`` and sign-extend. - This is equivalent to ``load.i16`` followed by ``uextend``. + This is equivalent to ``load.i16`` followed by ``sextend``. """, ins=(MemFlags, args, Offset), outs=a, can_load=True) @@ -419,7 +419,7 @@ sload32 = Instruction( 'sload32', r""" Load 32 bits from memory at ``p + Offset`` and sign-extend. - This is equivalent to ``load.i32`` followed by ``uextend``. + This is equivalent to ``load.i32`` followed by ``sextend``. """, ins=(MemFlags, p, Offset), outs=a, can_load=True) @@ -427,7 +427,7 @@ sload32_complex = Instruction( 'sload32_complex', r""" Load 32 bits from memory at ``sum(args) + Offset`` and sign-extend. - This is equivalent to ``load.i32`` followed by ``uextend``. + This is equivalent to ``load.i32`` followed by ``sextend``. """, ins=(MemFlags, args, Offset), outs=a, can_load=True) From 5c7aeb46dd264ea38870ddf727daf4e88ee5bba7 Mon Sep 17 00:00:00 2001 From: Amanieu d'Antras Date: Fri, 6 Jul 2018 20:09:20 +0100 Subject: [PATCH 1887/3084] Make InstructionData comparable and hashable (#388) * Don't implement Eq and Hash for EntityList * Generate eq and hash methods for InstructionData * Use the eq() and hash() methods of InstructionData in simple_gvn --- lib/codegen/meta/gen_instr.py | 91 ++++++++++++++++++++++++++++++++++- lib/codegen/src/simple_gvn.rs | 91 +++++++++++++++++++++++++---------- lib/entity/src/list.rs | 19 +------- 3 files changed, 158 insertions(+), 43 deletions(-) diff --git a/lib/codegen/meta/gen_instr.py b/lib/codegen/meta/gen_instr.py index 0b25cc1835..da9699ee97 100644 --- a/lib/codegen/meta/gen_instr.py +++ b/lib/codegen/meta/gen_instr.py @@ -114,7 +114,7 @@ def gen_instruction_data(fmt): store the additional information out of line. """ - fmt.line('#[derive(Clone, Debug, Hash, PartialEq, Eq)]') + fmt.line('#[derive(Clone, Debug)]') fmt.line('#[allow(missing_docs)]') with fmt.indented('pub enum InstructionData {', '}'): for f in InstructionFormat.all_formats: @@ -147,6 +147,8 @@ def gen_instruction_data_impl(fmt): - `pub fn arguments_mut(&mut self, &pool) -> &mut [Value]` - `pub fn take_value_list(&mut self) -> Option` - `pub fn put_value_list(&mut self, args: ir::ValueList>` + - `pub fn eq(&self, &other: Self, &pool) -> bool` + - `pub fn hash(&self, state: &mut H, &pool)` """ # The `opcode` method simply reads the `opcode` members. This is really a @@ -243,6 +245,93 @@ def gen_instruction_data_impl(fmt): fmt.line( 'debug_assert!(args.is_empty(), "Value list already in use");') fmt.line('*args = vlist;') + fmt.line() + + fmt.doc_comment( + """ + Compare two `InstructionData` for equality. + + This operation requires a reference to a `ValueListPool` to + determine if the contents of any `ValueLists` are equal. + """) + with fmt.indented( + 'pub fn eq(&self, other: &Self, pool: &ir::ValueListPool)' + ' -> bool {', + '}'): + with fmt.indented('if ::std::mem::discriminant(self) != ' + '::std::mem::discriminant(other) {', '}'): + fmt.line('return false;') + with fmt.indented('match (self, other) {', '}'): + for f in InstructionFormat.all_formats: + n = '&InstructionData::' + f.name + members = ['opcode'] + if f.typevar_operand is None: + args_eq = 'true' + elif f.has_value_list: + members.append('args') + args_eq = 'args1.as_slice(pool) == ' \ + 'args2.as_slice(pool)' + elif f.num_value_operands == 1: + members.append('arg') + args_eq = 'arg1 == arg2' + else: + members.append('args') + args_eq = 'args1 == args2' + for field in f.imm_fields: + members.append(field.member) + pat1 = ', '.join('{}: ref {}1'.format(x, x) + for x in members) + pat2 = ', '.join('{}: ref {}2'.format(x, x) + for x in members) + with fmt.indented('({} {{ {} }}, {} {{ {} }}) => {{' + .format(n, pat1, n, pat2), '}'): + fmt.line('opcode1 == opcode2 &&') + for field in f.imm_fields: + fmt.line('{}1 == {}2 &&' + .format(field.member, field.member)) + fmt.line(args_eq) + fmt.line('_ => unsafe { ' + '::std::hint::unreachable_unchecked() }') + fmt.line() + + fmt.doc_comment( + """ + Hash an `InstructionData`. + + This operation requires a reference to a `ValueListPool` to + hash the contents of any `ValueLists`. + """) + with fmt.indented( + 'pub fn hash' + '(&self, state: &mut H, pool: &ir::ValueListPool) {', + '}'): + with fmt.indented('match self {', '}'): + for f in InstructionFormat.all_formats: + n = 'InstructionData::' + f.name + members = ['opcode'] + if f.typevar_operand is None: + args = '&()' + elif f.has_value_list: + members.append('args') + args = 'args.as_slice(pool)' + elif f.num_value_operands == 1: + members.append('arg') + args = 'arg' + else: + members.append('args') + args = 'args' + for field in f.imm_fields: + members.append(field.member) + pat = n + ' { ' + ', '.join(members) + ' }' + with fmt.indented(pat + ' => {', '}'): + fmt.line('::std::hash::Hash::hash( ' + '&::std::mem::discriminant(self), state);') + fmt.line('::std::hash::Hash::hash(opcode, state);') + for field in f.imm_fields: + fmt.line('::std::hash::Hash::hash({}, state);' + .format(field.member)) + fmt.line('::std::hash::Hash::hash({}, state);' + .format(args)) def collect_instr_groups(isas): diff --git a/lib/codegen/src/simple_gvn.rs b/lib/codegen/src/simple_gvn.rs index cc8e7c37c5..79560af674 100644 --- a/lib/codegen/src/simple_gvn.rs +++ b/lib/codegen/src/simple_gvn.rs @@ -4,6 +4,8 @@ use cursor::{Cursor, FuncCursor}; use dominator_tree::DominatorTree; use ir::{Function, Inst, InstructionData, Opcode, Type}; use scoped_hash_map::ScopedHashMap; +use std::cell::{Ref, RefCell}; +use std::hash::{Hash, Hasher}; use std::vec::Vec; use timing; @@ -20,64 +22,103 @@ fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool { || opcode.writes_cpu_flags() } +/// Wrapper around `InstructionData` which implements `Eq` and `Hash` +#[derive(Clone)] +struct HashKey<'a, 'f: 'a> { + inst: InstructionData, + ty: Type, + pos: &'a RefCell>, +} +impl<'a, 'f: 'a> Hash for HashKey<'a, 'f> { + fn hash(&self, state: &mut H) { + let pool = &self.pos.borrow().func.dfg.value_lists; + self.inst.hash(state, pool); + self.ty.hash(state); + } +} +impl<'a, 'f: 'a> PartialEq for HashKey<'a, 'f> { + fn eq(&self, other: &Self) -> bool { + let pool = &self.pos.borrow().func.dfg.value_lists; + self.inst.eq(&other.inst, pool) && self.ty == other.ty + } +} +impl<'a, 'f: 'a> Eq for HashKey<'a, 'f> {} + /// Perform simple GVN on `func`. /// pub fn do_simple_gvn(func: &mut Function, domtree: &mut DominatorTree) { let _tt = timing::gvn(); debug_assert!(domtree.is_valid()); - let mut visible_values: ScopedHashMap<(InstructionData, Type), Inst> = ScopedHashMap::new(); + // Visit EBBs in a reverse post-order. + // + // The RefCell here is a bit ugly since the HashKeys in the ScopedHashMap + // need a reference to the function. + let pos = RefCell::new(FuncCursor::new(func)); + + let mut visible_values: ScopedHashMap = ScopedHashMap::new(); let mut scope_stack: Vec = Vec::new(); - // Visit EBBs in a reverse post-order. - let mut pos = FuncCursor::new(func); - for &ebb in domtree.cfg_postorder().iter().rev() { - // Pop any scopes that we just exited. - loop { - if let Some(current) = scope_stack.last() { - if domtree.dominates(*current, ebb, &pos.func.layout) { + { + // Pop any scopes that we just exited. + let layout = &pos.borrow().func.layout; + loop { + if let Some(current) = scope_stack.last() { + if domtree.dominates(*current, ebb, layout) { + break; + } + } else { break; } - } else { - break; + scope_stack.pop(); + visible_values.decrement_depth(); } - scope_stack.pop(); - visible_values.decrement_depth(); + + // Push a scope for the current block. + scope_stack.push(layout.first_inst(ebb).unwrap()); + visible_values.increment_depth(); } - // Push a scope for the current block. - scope_stack.push(pos.func.layout.first_inst(ebb).unwrap()); - visible_values.increment_depth(); - - pos.goto_top(ebb); - while let Some(inst) = pos.next_inst() { + pos.borrow_mut().goto_top(ebb); + while let Some(inst) = { + let mut pos = pos.borrow_mut(); + pos.next_inst() + } { // Resolve aliases, particularly aliases we created earlier. - pos.func.dfg.resolve_aliases_in_arguments(inst); + pos.borrow_mut().func.dfg.resolve_aliases_in_arguments(inst); - let opcode = pos.func.dfg[inst].opcode(); + let func = Ref::map(pos.borrow(), |pos| &pos.func); + + let opcode = func.dfg[inst].opcode(); if opcode.is_branch() && !opcode.is_terminator() { - scope_stack.push(pos.func.layout.next_inst(inst).unwrap()); + scope_stack.push(func.layout.next_inst(inst).unwrap()); visible_values.increment_depth(); } if trivially_unsafe_for_gvn(opcode) { continue; } - let ctrl_typevar = pos.func.dfg.ctrl_typevar(inst); - let key = (pos.func.dfg[inst].clone(), ctrl_typevar); + let ctrl_typevar = func.dfg.ctrl_typevar(inst); + let key = HashKey { + inst: func.dfg[inst].clone(), + ty: ctrl_typevar, + pos: &pos, + }; let entry = visible_values.entry(key); use scoped_hash_map::Entry::*; match entry { Occupied(entry) => { - debug_assert!(domtree.dominates(*entry.get(), inst, &pos.func.layout)); + debug_assert!(domtree.dominates(*entry.get(), inst, &func.layout)); // If the redundant instruction is representing the current // scope, pick a new representative. let old = scope_stack.last_mut().unwrap(); if *old == inst { - *old = pos.func.layout.next_inst(inst).unwrap(); + *old = func.layout.next_inst(inst).unwrap(); } // Replace the redundant instruction and remove it. + drop(func); + let mut pos = pos.borrow_mut(); pos.func.dfg.replace_with_aliases(inst, *entry.get()); pos.remove_inst_and_step_back(); } diff --git a/lib/entity/src/list.rs b/lib/entity/src/list.rs index e48a839759..3c5bd25a13 100644 --- a/lib/entity/src/list.rs +++ b/lib/entity/src/list.rs @@ -1,5 +1,4 @@ //! Small lists of entity references. -use std::hash::{Hash, Hasher}; use std::marker::PhantomData; use std::mem; use std::vec::Vec; @@ -34,9 +33,8 @@ use EntityRef; /// function they belong to. *Cloning an entity list does not allocate new memory for the clone*. /// It creates an alias of the same memory. /// -/// Entity lists can also be hashed and compared for equality, but those operations just panic if -/// they're ever actually called, because it's not possible to compare the contents of the list -/// without the pool reference. +/// Entity lists cannot be hashed and compared for equality because it's not possible to compare the +/// contents of the list without the pool reference. /// /// # Implementation /// @@ -76,19 +74,6 @@ impl Default for EntityList { } } -impl Hash for EntityList { - fn hash(&self, _: &mut H) { - panic!("hash called on EntityList"); - } -} - -impl PartialEq for EntityList { - fn eq(&self, _: &Self) -> bool { - panic!("eq called on EntityList"); - } -} -impl Eq for EntityList {} - /// A memory pool for storing lists of `T`. #[derive(Clone, Debug)] pub struct ListPool { From b6166444951a3451f92ffc50925eca03f8eb9300 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 3 Jul 2018 08:45:52 -0700 Subject: [PATCH 1888/3084] Document that cretonne-module is an optional part of Cretonne. --- lib/module/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/module/README.md b/lib/module/README.md index 370bde4333..4ff52c3d2d 100644 --- a/lib/module/README.md +++ b/lib/module/README.md @@ -1,3 +1,7 @@ This crate provides the `Module` trait, which provides an interface for multiple functions and data to be emitted with [Cretonne](https://crates.io/crates/cretonne) and then linked together. + +This crate is structured as an optional layer on top of cretonne-codegen. +It provides additional functionality, such as linking, however users that +require greater flexibility don't need to use it. From 0ad7dbf689cb1ded36084b5dbe9c023db0a98ddd Mon Sep 17 00:00:00 2001 From: Benjamin Bouvier Date: Mon, 9 Jul 2018 16:03:25 +0200 Subject: [PATCH 1889/3084] [clippy] Remove unnecessary closures; --- lib/wasm/src/func_translator.rs | 6 +++--- lib/wasm/src/module_translator.rs | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs index 9f93904381..105cb190a8 100644 --- a/lib/wasm/src/func_translator.rs +++ b/lib/wasm/src/func_translator.rs @@ -137,14 +137,14 @@ fn parse_local_decls( let mut next_local = num_params; let local_count = reader .read_local_count() - .map_err(|e| WasmError::from_binary_reader_error(e))?; + .map_err(WasmError::from_binary_reader_error)?; let mut locals_total = 0; for _ in 0..local_count { builder.set_srcloc(cur_srcloc(reader)); let (count, ty) = reader .read_local_decl(&mut locals_total) - .map_err(|e| WasmError::from_binary_reader_error(e))?; + .map_err(WasmError::from_binary_reader_error)?; declare_locals(builder, count, ty, &mut next_local); } @@ -197,7 +197,7 @@ fn parse_function_body( builder.set_srcloc(cur_srcloc(&reader)); let op = reader .read_operator() - .map_err(|e| WasmError::from_binary_reader_error(e))?; + .map_err(WasmError::from_binary_reader_error)?; translate_operator(op, builder, state, environ)?; } diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs index ffeafc2378..fcbcd51708 100644 --- a/lib/wasm/src/module_translator.rs +++ b/lib/wasm/src/module_translator.rs @@ -131,7 +131,7 @@ pub fn translate_module<'data>( let size = reader.bytes_remaining(); environ.define_function_body(reader .read_bytes(size) - .map_err(|e| WasmError::from_binary_reader_error(e))?)?; + .map_err(WasmError::from_binary_reader_error)?)?; } loop { match *parser.read() { From b263a8344c663bc12c9cab9da692371f7230e02d Mon Sep 17 00:00:00 2001 From: Benjamin Bouvier Date: Mon, 9 Jul 2018 16:03:42 +0200 Subject: [PATCH 1890/3084] [clippy] Use subsec_millis() instead of nanos and division; --- lib/filetests/src/runner.rs | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs index 1e544ba0aa..3b261cd44b 100644 --- a/lib/filetests/src/runner.rs +++ b/lib/filetests/src/runner.rs @@ -40,13 +40,7 @@ impl Display for QueueEntry { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { let p = self.path.to_string_lossy(); match self.state { - State::Done(Ok(dur)) => write!( - f, - "{}.{:03} {}", - dur.as_secs(), - dur.subsec_nanos() / 1_000_000, - p - ), + State::Done(Ok(dur)) => write!(f, "{}.{:03} {}", dur.as_secs(), dur.subsec_millis(), p), State::Done(Err(ref e)) => write!(f, "FAIL {}: {}", p, e), _ => write!(f, "{}", p), } From 26523fdf5c60815c4160a98c9d2ee35c8dd8082e Mon Sep 17 00:00:00 2001 From: Benjamin Bouvier Date: Mon, 9 Jul 2018 16:15:17 +0200 Subject: [PATCH 1891/3084] [clippy] Pass a few argument types by value, not by reference; Since Location is basically just a usize, and wasmparser::Type is an enum, and both are copiable, this passes them down by value instead of by reference, as suggested by Clippy. --- cranelift/src/utils.rs | 4 +- lib/reader/src/isaspec.rs | 2 +- lib/reader/src/parser.rs | 83 +++++++++++++++++-------------------- lib/reader/src/sourcemap.rs | 20 ++++----- 4 files changed, 52 insertions(+), 57 deletions(-) diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index 00f5bab347..8940778f86 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -51,7 +51,7 @@ pub fn parse_sets_and_triple( parse_options( flag_set.iter().map(|x| x.as_str()), &mut flag_builder, - &Location { line_number: 0 }, + Location { line_number: 0 }, ).map_err(|err| err.to_string())?; let mut words = flag_triple.trim().split_whitespace(); @@ -71,7 +71,7 @@ pub fn parse_sets_and_triple( ), })?; // Apply the ISA-specific settings to `isa_builder`. - parse_options(words, &mut isa_builder, &Location { line_number: 0 }) + parse_options(words, &mut isa_builder, Location { line_number: 0 }) .map_err(|err| err.to_string())?; Ok(OwnedFlagsOrIsa::Isa( diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs index 18a11ced2d..2998f17766 100644 --- a/lib/reader/src/isaspec.rs +++ b/lib/reader/src/isaspec.rs @@ -35,7 +35,7 @@ impl IsaSpec { } /// Parse an iterator of command line options and apply them to `config`. -pub fn parse_options<'a, I>(iter: I, config: &mut Configurable, loc: &Location) -> ParseResult<()> +pub fn parse_options<'a, I>(iter: I, config: &mut Configurable, loc: Location) -> ParseResult<()> where I: Iterator, { diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 79adb9f3d4..641fe2a89d 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -122,7 +122,7 @@ impl<'a> Context<'a> { } // Allocate a new stack slot. - fn add_ss(&mut self, ss: StackSlot, data: StackSlotData, loc: &Location) -> ParseResult<()> { + fn add_ss(&mut self, ss: StackSlot, data: StackSlotData, loc: Location) -> ParseResult<()> { while self.function.stack_slots.next_key().index() <= ss.index() { self.function .create_stack_slot(StackSlotData::new(StackSlotKind::SpillSlot, 0)); @@ -132,7 +132,7 @@ impl<'a> Context<'a> { } // Resolve a reference to a stack slot. - fn check_ss(&self, ss: StackSlot, loc: &Location) -> ParseResult<()> { + fn check_ss(&self, ss: StackSlot, loc: Location) -> ParseResult<()> { if !self.map.contains_ss(ss) { err!(loc, "undefined stack slot {}", ss) } else { @@ -141,12 +141,7 @@ impl<'a> Context<'a> { } // Allocate a global value slot. - fn add_gv( - &mut self, - gv: GlobalValue, - data: GlobalValueData, - loc: &Location, - ) -> ParseResult<()> { + fn add_gv(&mut self, gv: GlobalValue, data: GlobalValueData, loc: Location) -> ParseResult<()> { while self.function.global_values.next_key().index() <= gv.index() { self.function.create_global_value(GlobalValueData::Sym { name: ExternalName::testcase(""), @@ -158,7 +153,7 @@ impl<'a> Context<'a> { } // Resolve a reference to a global value. - fn check_gv(&self, gv: GlobalValue, loc: &Location) -> ParseResult<()> { + fn check_gv(&self, gv: GlobalValue, loc: Location) -> ParseResult<()> { if !self.map.contains_gv(gv) { err!(loc, "undefined global value {}", gv) } else { @@ -167,7 +162,7 @@ impl<'a> Context<'a> { } // Allocate a heap slot. - fn add_heap(&mut self, heap: Heap, data: HeapData, loc: &Location) -> ParseResult<()> { + fn add_heap(&mut self, heap: Heap, data: HeapData, loc: Location) -> ParseResult<()> { while self.function.heaps.next_key().index() <= heap.index() { self.function.create_heap(HeapData { base: HeapBase::ReservedReg, @@ -183,7 +178,7 @@ impl<'a> Context<'a> { } // Resolve a reference to a heap. - fn check_heap(&self, heap: Heap, loc: &Location) -> ParseResult<()> { + fn check_heap(&self, heap: Heap, loc: Location) -> ParseResult<()> { if !self.map.contains_heap(heap) { err!(loc, "undefined heap {}", heap) } else { @@ -192,7 +187,7 @@ impl<'a> Context<'a> { } // Allocate a new signature. - fn add_sig(&mut self, sig: SigRef, data: Signature, loc: &Location) -> ParseResult<()> { + fn add_sig(&mut self, sig: SigRef, data: Signature, loc: Location) -> ParseResult<()> { while self.function.dfg.signatures.next_key().index() <= sig.index() { self.function .import_signature(Signature::new(CallConv::Fast)); @@ -211,7 +206,7 @@ impl<'a> Context<'a> { } // Allocate a new external function. - fn add_fn(&mut self, fn_: FuncRef, data: ExtFuncData, loc: &Location) -> ParseResult<()> { + fn add_fn(&mut self, fn_: FuncRef, data: ExtFuncData, loc: Location) -> ParseResult<()> { while self.function.dfg.ext_funcs.next_key().index() <= fn_.index() { self.function.import_function(ExtFuncData { name: ExternalName::testcase(""), @@ -224,7 +219,7 @@ impl<'a> Context<'a> { } // Resolve a reference to a function. - fn check_fn(&self, fn_: FuncRef, loc: &Location) -> ParseResult<()> { + fn check_fn(&self, fn_: FuncRef, loc: Location) -> ParseResult<()> { if !self.map.contains_fn(fn_) { err!(loc, "undefined function {}", fn_) } else { @@ -233,7 +228,7 @@ impl<'a> Context<'a> { } // Allocate a new jump table. - fn add_jt(&mut self, jt: JumpTable, data: JumpTableData, loc: &Location) -> ParseResult<()> { + fn add_jt(&mut self, jt: JumpTable, data: JumpTableData, loc: Location) -> ParseResult<()> { while self.function.jump_tables.next_key().index() <= jt.index() { self.function.create_jump_table(JumpTableData::new()); } @@ -242,7 +237,7 @@ impl<'a> Context<'a> { } // Resolve a reference to a jump table. - fn check_jt(&self, jt: JumpTable, loc: &Location) -> ParseResult<()> { + fn check_jt(&self, jt: JumpTable, loc: Location) -> ParseResult<()> { if !self.map.contains_jt(jt) { err!(loc, "undefined jump table {}", jt) } else { @@ -251,8 +246,8 @@ impl<'a> Context<'a> { } // Assign the global for the stack limit. - fn set_stack_limit(&mut self, gv: GlobalValue, loc: &Location) -> ParseResult<()> { - if let Some(_) = self.function.set_stack_limit(Some(gv)) { + fn set_stack_limit(&mut self, gv: GlobalValue, loc: Location) -> ParseResult<()> { + if self.function.set_stack_limit(Some(gv)).is_some() { err!(loc, "multiple stack_limit declarations") } else { Ok(()) @@ -260,7 +255,7 @@ impl<'a> Context<'a> { } // Allocate a new EBB. - fn add_ebb(&mut self, ebb: Ebb, loc: &Location) -> ParseResult { + fn add_ebb(&mut self, ebb: Ebb, loc: Location) -> ParseResult { while self.function.dfg.num_ebbs() <= ebb.index() { self.function.dfg.make_ebb(); } @@ -703,7 +698,7 @@ impl<'a> Parser<'a> { isaspec::parse_options( self.consume_line().trim().split_whitespace(), &mut flag_builder, - &self.loc, + self.loc, )?; } "target" => { @@ -732,7 +727,7 @@ impl<'a> Parser<'a> { last_set_loc = None; seen_target = true; // Apply the target-specific settings to `isa_builder`. - isaspec::parse_options(words, &mut isa_builder, &self.loc)?; + isaspec::parse_options(words, &mut isa_builder, self.loc)?; // Construct a trait object with the aggregate settings. targets.push(isa_builder.finish(settings::Flags::new(flag_builder.clone()))); @@ -1004,35 +999,35 @@ impl<'a> Parser<'a> { self.start_gathering_comments(); let loc = self.loc; self.parse_stack_slot_decl() - .and_then(|(ss, dat)| ctx.add_ss(ss, dat, &loc)) + .and_then(|(ss, dat)| ctx.add_ss(ss, dat, loc)) } Some(Token::GlobalValue(..)) => { self.start_gathering_comments(); self.parse_global_value_decl() - .and_then(|(gv, dat)| ctx.add_gv(gv, dat, &self.loc)) + .and_then(|(gv, dat)| ctx.add_gv(gv, dat, self.loc)) } Some(Token::Heap(..)) => { self.start_gathering_comments(); self.parse_heap_decl() - .and_then(|(heap, dat)| ctx.add_heap(heap, dat, &self.loc)) + .and_then(|(heap, dat)| ctx.add_heap(heap, dat, self.loc)) } Some(Token::SigRef(..)) => { self.start_gathering_comments(); self.parse_signature_decl(ctx.unique_isa) - .and_then(|(sig, dat)| ctx.add_sig(sig, dat, &self.loc)) + .and_then(|(sig, dat)| ctx.add_sig(sig, dat, self.loc)) } Some(Token::FuncRef(..)) => { self.start_gathering_comments(); self.parse_function_decl(ctx) - .and_then(|(fn_, dat)| ctx.add_fn(fn_, dat, &self.loc)) + .and_then(|(fn_, dat)| ctx.add_fn(fn_, dat, self.loc)) } Some(Token::JumpTable(..)) => { self.start_gathering_comments(); self.parse_jump_table_decl() - .and_then(|(jt, dat)| ctx.add_jt(jt, dat, &self.loc)) + .and_then(|(jt, dat)| ctx.add_jt(jt, dat, self.loc)) } Some(Token::Identifier("stack_limit")) => self.parse_stack_limit_decl() - .and_then(|gv| ctx.set_stack_limit(gv, &self.loc)), + .and_then(|gv| ctx.set_stack_limit(gv, self.loc)), // More to come.. _ => return Ok(()), }?; @@ -1236,7 +1231,7 @@ impl<'a> Parser<'a> { let sig = self.parse_signature(ctx.unique_isa)?; let sigref = ctx.function.import_signature(sig); ctx.map - .def_entity(sigref.into(), &loc) + .def_entity(sigref.into(), loc) .expect("duplicate SigRef entities created"); ExtFuncData { name, @@ -1369,7 +1364,7 @@ impl<'a> Parser<'a> { self.start_gathering_comments(); let ebb_num = self.match_ebb("expected EBB header")?; - let ebb = ctx.add_ebb(ebb_num, &self.loc)?; + let ebb = ctx.add_ebb(ebb_num, self.loc)?; if !self.optional(Token::Colon) { // ebb-header ::= Ebb(ebb) [ * ebb-params ] ":" @@ -1466,7 +1461,7 @@ impl<'a> Parser<'a> { let t = self.match_type("expected EBB argument type")?; // Allocate the EBB argument. ctx.function.dfg.append_ebb_param_for_parser(ebb, t, v); - ctx.map.def_value(v, &v_location)?; + ctx.map.def_value(v, v_location)?; // ebb-param ::= Value(v) ":" Type(t) * arg-loc? if self.optional(Token::LBracket) { @@ -1492,7 +1487,7 @@ impl<'a> Parser<'a> { } Some(ss) => ss, }; - ctx.check_ss(ss, &self.loc)?; + ctx.check_ss(ss, self.loc)?; Ok(ValueLoc::Stack(ss)) } Some(Token::Name(name)) => { @@ -1610,7 +1605,7 @@ impl<'a> Parser<'a> { return err!(self.loc, "value {} is already defined"); } } else { - ctx.map.def_value(result, &self.loc)?; + ctx.map.def_value(result, self.loc)?; } if !ctx.map.contains_value(dest) { @@ -1638,7 +1633,7 @@ impl<'a> Parser<'a> { ) -> ParseResult<()> { // Define the result values. for val in results { - ctx.map.def_value(*val, &self.loc)?; + ctx.map.def_value(*val, self.loc)?; } // Collect comments for the next instruction. @@ -1680,7 +1675,7 @@ impl<'a> Parser<'a> { .make_inst_results_for_parser(inst, ctrl_typevar, results); ctx.function.layout.append_inst(inst, ebb); ctx.map - .def_entity(inst.into(), &opcode_loc) + .def_entity(inst.into(), opcode_loc) .expect("duplicate inst references created"); if !srcloc.is_default() { @@ -1897,7 +1892,7 @@ impl<'a> Parser<'a> { }, InstructionFormat::UnaryGlobalValue => { let gv = self.match_gv("expected global value")?; - ctx.check_gv(gv, &self.loc)?; + ctx.check_gv(gv, self.loc)?; InstructionData::UnaryGlobalValue { opcode, global_value: gv, @@ -2009,7 +2004,7 @@ impl<'a> Parser<'a> { let arg = self.match_value("expected SSA value operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; let table = self.match_jt()?; - ctx.check_jt(table, &self.loc)?; + ctx.check_jt(table, self.loc)?; InstructionData::BranchTable { opcode, arg, table } } InstructionFormat::InsertLane => { @@ -2089,7 +2084,7 @@ impl<'a> Parser<'a> { } InstructionFormat::Call => { let func_ref = self.match_fn("expected function reference")?; - ctx.check_fn(func_ref, &self.loc)?; + ctx.check_fn(func_ref, self.loc)?; self.match_token(Token::LPar, "expected '(' before arguments")?; let args = self.parse_value_list()?; self.match_token(Token::RPar, "expected ')' after arguments")?; @@ -2115,12 +2110,12 @@ impl<'a> Parser<'a> { } InstructionFormat::FuncAddr => { let func_ref = self.match_fn("expected function reference")?; - ctx.check_fn(func_ref, &self.loc)?; + ctx.check_fn(func_ref, self.loc)?; InstructionData::FuncAddr { opcode, func_ref } } InstructionFormat::StackLoad => { let ss = self.match_ss("expected stack slot number: ss«n»")?; - ctx.check_ss(ss, &self.loc)?; + ctx.check_ss(ss, self.loc)?; let offset = self.optional_offset32()?; InstructionData::StackLoad { opcode, @@ -2132,7 +2127,7 @@ impl<'a> Parser<'a> { let arg = self.match_value("expected SSA value operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; let ss = self.match_ss("expected stack slot number: ss«n»")?; - ctx.check_ss(ss, &self.loc)?; + ctx.check_ss(ss, self.loc)?; let offset = self.optional_offset32()?; InstructionData::StackStore { opcode, @@ -2143,7 +2138,7 @@ impl<'a> Parser<'a> { } InstructionFormat::HeapAddr => { let heap = self.match_heap("expected heap identifier")?; - ctx.check_heap(heap, &self.loc)?; + ctx.check_heap(heap, self.loc)?; self.match_token(Token::Comma, "expected ',' between operands")?; let arg = self.match_value("expected SSA value heap address")?; self.match_token(Token::Comma, "expected ',' between operands")?; @@ -2229,7 +2224,7 @@ impl<'a> Parser<'a> { let src = self.match_regunit(ctx.unique_isa)?; self.match_token(Token::Arrow, "expected '->' before destination stack slot")?; let dst = self.match_ss("expected stack slot number: ss«n»")?; - ctx.check_ss(dst, &self.loc)?; + ctx.check_ss(dst, self.loc)?; InstructionData::RegSpill { opcode, arg, @@ -2241,7 +2236,7 @@ impl<'a> Parser<'a> { let arg = self.match_value("expected SSA value operand")?; self.match_token(Token::Comma, "expected ',' between operands")?; let src = self.match_ss("expected stack slot number: ss«n»")?; - ctx.check_ss(src, &self.loc)?; + ctx.check_ss(src, self.loc)?; self.match_token( Token::Arrow, "expected '->' before destination register units", diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index 814baf8e1b..b9fddde64f 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -140,49 +140,49 @@ impl SourceMap { } /// Define the value `entity`. - pub fn def_value(&mut self, entity: Value, loc: &Location) -> ParseResult<()> { + pub fn def_value(&mut self, entity: Value, loc: Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the ebb `entity`. - pub fn def_ebb(&mut self, entity: Ebb, loc: &Location) -> ParseResult<()> { + pub fn def_ebb(&mut self, entity: Ebb, loc: Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the stack slot `entity`. - pub fn def_ss(&mut self, entity: StackSlot, loc: &Location) -> ParseResult<()> { + pub fn def_ss(&mut self, entity: StackSlot, loc: Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the global value `entity`. - pub fn def_gv(&mut self, entity: GlobalValue, loc: &Location) -> ParseResult<()> { + pub fn def_gv(&mut self, entity: GlobalValue, loc: Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the heap `entity`. - pub fn def_heap(&mut self, entity: Heap, loc: &Location) -> ParseResult<()> { + pub fn def_heap(&mut self, entity: Heap, loc: Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the signature `entity`. - pub fn def_sig(&mut self, entity: SigRef, loc: &Location) -> ParseResult<()> { + pub fn def_sig(&mut self, entity: SigRef, loc: Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the external function `entity`. - pub fn def_fn(&mut self, entity: FuncRef, loc: &Location) -> ParseResult<()> { + pub fn def_fn(&mut self, entity: FuncRef, loc: Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define the jump table `entity`. - pub fn def_jt(&mut self, entity: JumpTable, loc: &Location) -> ParseResult<()> { + pub fn def_jt(&mut self, entity: JumpTable, loc: Location) -> ParseResult<()> { self.def_entity(entity.into(), loc) } /// Define an entity. This can be used for instructions whose numbers never /// appear in source, or implicitly defined signatures. - pub fn def_entity(&mut self, entity: AnyEntity, loc: &Location) -> ParseResult<()> { - if self.locations.insert(entity, *loc).is_some() { + pub fn def_entity(&mut self, entity: AnyEntity, loc: Location) -> ParseResult<()> { + if self.locations.insert(entity, loc).is_some() { err!(loc, "duplicate entity: {}", entity) } else { Ok(()) From bea843519cf647000cd1276614a5fc06917eb145 Mon Sep 17 00:00:00 2001 From: Benjamin Bouvier Date: Tue, 10 Jul 2018 15:39:36 +0200 Subject: [PATCH 1892/3084] [clippy] Return explicit unit values; --- lib/wasm/src/code_translator.rs | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index 76290364f2..e6018a11bb 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -47,11 +47,12 @@ pub fn translate_operator( environ: &mut FE, ) -> WasmResult<()> { if !state.reachable { - return Ok(translate_unreachable_operator(&op, builder, state)); + translate_unreachable_operator(&op, builder, state); + return Ok(()); } // This big match treats all Wasm code operators. - Ok(match op { + match op { /********************************** Locals **************************************** * `get_local` and `set_local` are treated as non-SSA variables and will completely * disappear in the Cretonne Code @@ -885,7 +886,8 @@ pub fn translate_operator( | Operator::I64AtomicRmw32UCmpxchg { .. } => { return Err(WasmError::Unsupported("proposed thread operators")); } - }) + }; + Ok(()) } // Clippy warns us of some fields we are deliberately ignoring From 25508ac66ee15333558875606059088ea75b132a Mon Sep 17 00:00:00 2001 From: Benjamin Bouvier Date: Tue, 10 Jul 2018 15:41:59 +0200 Subject: [PATCH 1893/3084] [clippy] Pass more types by value; wasmparser::Type is an enum, and there was one Location I missed. --- lib/reader/src/parser.rs | 6 +++--- lib/wasm/src/code_translator.rs | 6 +++--- lib/wasm/src/sections_translator.rs | 12 ++++++------ lib/wasm/src/translation_utils.rs | 4 ++-- 4 files changed, 14 insertions(+), 14 deletions(-) diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index 641fe2a89d..e1cc270c60 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -197,7 +197,7 @@ impl<'a> Context<'a> { } // Resolve a reference to a signature. - fn check_sig(&self, sig: SigRef, loc: &Location) -> ParseResult<()> { + fn check_sig(&self, sig: SigRef, loc: Location) -> ParseResult<()> { if !self.map.contains_sig(sig) { err!(loc, "undefined signature {}", sig) } else { @@ -1246,7 +1246,7 @@ impl<'a> Parser<'a> { } Some(sig) => sig, }; - ctx.check_sig(sig, &self.loc)?; + ctx.check_sig(sig, self.loc)?; self.consume(); ExtFuncData { name, @@ -2096,7 +2096,7 @@ impl<'a> Parser<'a> { } InstructionFormat::CallIndirect => { let sig_ref = self.match_sig("expected signature reference")?; - ctx.check_sig(sig_ref, &self.loc)?; + ctx.check_sig(sig_ref, self.loc)?; self.match_token(Token::Comma, "expected ',' between operands")?; let callee = self.match_value("expected SSA value callee operand")?; self.match_token(Token::LPar, "expected '(' before arguments")?; diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs index e6018a11bb..dc9be39fa5 100644 --- a/lib/wasm/src/code_translator.rs +++ b/lib/wasm/src/code_translator.rs @@ -129,7 +129,7 @@ pub fn translate_operator( ***********************************************************************************/ Operator::Block { ty } => { let next = builder.create_ebb(); - if let Ok(ty_cre) = type_to_type(&ty) { + if let Ok(ty_cre) = type_to_type(ty) { builder.append_ebb_param(next, ty_cre); } state.push_block(next, num_return_values(ty)); @@ -137,7 +137,7 @@ pub fn translate_operator( Operator::Loop { ty } => { let loop_body = builder.create_ebb(); let next = builder.create_ebb(); - if let Ok(ty_cre) = type_to_type(&ty) { + if let Ok(ty_cre) = type_to_type(ty) { builder.append_ebb_param(next, ty_cre); } builder.ins().jump(loop_body, &[]); @@ -155,7 +155,7 @@ pub fn translate_operator( // and we add nothing; // - either the If have an Else clause, in that case the destination of this jump // instruction will be changed later when we translate the Else operator. - if let Ok(ty_cre) = type_to_type(&ty) { + if let Ok(ty_cre) = type_to_type(ty) { builder.append_ebb_param(if_not, ty_cre); } state.push_if(jump_inst, if_not, num_return_values(ty)); diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs index 0c47a3e940..c21dbb0c25 100644 --- a/lib/wasm/src/sections_translator.rs +++ b/lib/wasm/src/sections_translator.rs @@ -36,12 +36,12 @@ pub fn parse_function_signatures( }) => { let mut sig = Signature::new(environ.flags().call_conv()); sig.params.extend(params.iter().map(|ty| { - let cret_arg: ir::Type = type_to_type(ty) + let cret_arg: ir::Type = type_to_type(*ty) .expect("only numeric types are supported in function signatures"); AbiParam::new(cret_arg) })); sig.returns.extend(returns.iter().map(|ty| { - let cret_arg: ir::Type = type_to_type(ty) + let cret_arg: ir::Type = type_to_type(*ty) .expect("only numeric types are supported in function signatures"); AbiParam::new(cret_arg) })); @@ -92,7 +92,7 @@ pub fn parse_import_section<'data>( .. } => { environ.declare_global(Global { - ty: type_to_type(&ty.content_type).unwrap(), + ty: type_to_type(ty.content_type).unwrap(), mutability: ty.mutable, initializer: GlobalInit::Import(), }); @@ -101,7 +101,7 @@ pub fn parse_import_section<'data>( ty: ImportSectionEntryType::Table(ref tab), .. } => environ.declare_table(Table { - ty: match type_to_type(&tab.element_type) { + ty: match type_to_type(tab.element_type) { Ok(t) => TableElementType::Val(t), Err(()) => TableElementType::Func(), }, @@ -245,7 +245,7 @@ pub fn parse_global_section( ref s => panic!("unexpected section content: {:?}", s), } let global = Global { - ty: type_to_type(&content_type).unwrap(), + ty: type_to_type(content_type).unwrap(), mutability, initializer, }; @@ -329,7 +329,7 @@ pub fn parse_table_section(parser: &mut Parser, environ: &mut ModuleEnvironment) loop { match *parser.read() { ParserState::TableSectionEntry(ref table) => environ.declare_table(Table { - ty: match type_to_type(&table.element_type) { + ty: match type_to_type(table.element_type) { Ok(t) => TableElementType::Val(t), Err(()) => TableElementType::Func(), }, diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs index 3ed13899e6..16b5737b80 100644 --- a/lib/wasm/src/translation_utils.rs +++ b/lib/wasm/src/translation_utils.rs @@ -72,8 +72,8 @@ pub struct Memory { } /// Helper function translating wasmparser types to Cretonne types when possible. -pub fn type_to_type(ty: &wasmparser::Type) -> Result { - match *ty { +pub fn type_to_type(ty: wasmparser::Type) -> Result { + match ty { wasmparser::Type::I32 => Ok(ir::types::I32), wasmparser::Type::I64 => Ok(ir::types::I64), wasmparser::Type::F32 => Ok(ir::types::F32), From 0616a960d6470d3afc4c466ebe72e3e612df1e95 Mon Sep 17 00:00:00 2001 From: Benjamin Bouvier Date: Tue, 10 Jul 2018 15:55:53 +0200 Subject: [PATCH 1894/3084] [clippy] A few fixes in module/src/module.rs; - use Self more to indicate the current type; - explicitly clone one Option; - invert !is_none to is_some; --- lib/module/src/module.rs | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index 76ac8926b9..216e3ed6e1 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -20,7 +20,7 @@ entity_impl!(FuncId, "funcid"); /// Function identifiers are namespace 0 in `ir::ExternalName` impl From for ir::ExternalName { - fn from(id: FuncId) -> ir::ExternalName { + fn from(id: FuncId) -> Self { ir::ExternalName::User { namespace: 0, index: id.0, @@ -35,7 +35,7 @@ entity_impl!(DataId, "dataid"); /// Data identifiers are namespace 1 in `ir::ExternalName` impl From for ir::ExternalName { - fn from(id: DataId) -> ir::ExternalName { + fn from(id: DataId) -> Self { ir::ExternalName::User { namespace: 1, index: id.0, @@ -74,16 +74,16 @@ impl Linkage { } /// Test whether this linkage can have a definition. - pub fn is_definable(&self) -> bool { - match *self { + pub fn is_definable(self) -> bool { + match self { Linkage::Import => false, Linkage::Local | Linkage::Preemptible | Linkage::Export => true, } } /// Test whether this linkage will have a definition that cannot be preempted. - pub fn is_final(&self) -> bool { - match *self { + pub fn is_final(self) -> bool { + match self { Linkage::Import | Linkage::Preemptible => false, Linkage::Local | Linkage::Export => true, } @@ -101,10 +101,10 @@ pub enum FuncOrDataId { /// Mapping to `ir::ExternalName` is trivial based on the `FuncId` and `DataId` mapping. impl From for ir::ExternalName { - fn from(id: FuncOrDataId) -> ir::ExternalName { + fn from(id: FuncOrDataId) -> Self { match id { - FuncOrDataId::Func(funcid) => ir::ExternalName::from(funcid), - FuncOrDataId::Data(dataid) => ir::ExternalName::from(dataid), + FuncOrDataId::Func(funcid) => Self::from(funcid), + FuncOrDataId::Data(dataid) => Self::from(dataid), } } } @@ -317,7 +317,7 @@ where /// Get the module identifier for a given name, if that name /// has been declared. pub fn get_name(&self, name: &str) -> Option { - self.names.get(name).map(|e| *e) + self.names.get(name).cloned() } /// Return then pointer type for the current target. @@ -472,7 +472,7 @@ where })?; let info = &self.contents.functions[func]; - if !info.compiled.is_none() { + if info.compiled.is_some() { return Err(ModuleError::DuplicateDefinition(info.decl.name.clone())); } if !info.decl.linkage.is_definable() { @@ -495,7 +495,7 @@ where pub fn define_data(&mut self, data: DataId, data_ctx: &DataContext) -> ModuleResult<()> { let compiled = { let info = &self.contents.data_objects[data]; - if !info.compiled.is_none() { + if info.compiled.is_some() { return Err(ModuleError::DuplicateDefinition(info.decl.name.clone())); } if !info.decl.linkage.is_definable() { From bcc268a3cd5379a828852b7abbe039f1c79439c5 Mon Sep 17 00:00:00 2001 From: Benjamin Bouvier Date: Tue, 10 Jul 2018 16:55:19 +0200 Subject: [PATCH 1895/3084] [clippy] Fix a few clippy issues in lib/codegen/; - don't generate "&& true" when generating instruction eq() fn; - use more Self; - use subsec_millis instead of subsec_nanos and divide; - coalesce two ifs; --- lib/codegen/meta/gen_instr.py | 9 +++++---- lib/codegen/src/fx.rs | 4 ++-- lib/codegen/src/ir/extname.rs | 2 +- lib/codegen/src/timing.rs | 3 +-- lib/codegen/src/verifier/locations.rs | 6 ++---- 5 files changed, 11 insertions(+), 13 deletions(-) diff --git a/lib/codegen/meta/gen_instr.py b/lib/codegen/meta/gen_instr.py index da9699ee97..3a33110a00 100644 --- a/lib/codegen/meta/gen_instr.py +++ b/lib/codegen/meta/gen_instr.py @@ -266,7 +266,7 @@ def gen_instruction_data_impl(fmt): n = '&InstructionData::' + f.name members = ['opcode'] if f.typevar_operand is None: - args_eq = 'true' + args_eq = None elif f.has_value_list: members.append('args') args_eq = 'args1.as_slice(pool) == ' \ @@ -285,11 +285,12 @@ def gen_instruction_data_impl(fmt): for x in members) with fmt.indented('({} {{ {} }}, {} {{ {} }}) => {{' .format(n, pat1, n, pat2), '}'): - fmt.line('opcode1 == opcode2 &&') + fmt.line('opcode1 == opcode2') for field in f.imm_fields: - fmt.line('{}1 == {}2 &&' + fmt.line('&& {}1 == {}2' .format(field.member, field.member)) - fmt.line(args_eq) + if args_eq is not None: + fmt.line('&& {}'.format(args_eq)) fmt.line('_ => unsafe { ' '::std::hint::unreachable_unchecked() }') fmt.line() diff --git a/lib/codegen/src/fx.rs b/lib/codegen/src/fx.rs index b59bac27be..eb3b7f2b36 100644 --- a/lib/codegen/src/fx.rs +++ b/lib/codegen/src/fx.rs @@ -50,8 +50,8 @@ const K: usize = 0x517cc1b727220a95; impl Default for FxHasher { #[inline] - fn default() -> FxHasher { - FxHasher { hash: 0 } + fn default() -> Self { + Self { hash: 0 } } } diff --git a/lib/codegen/src/ir/extname.rs b/lib/codegen/src/ir/extname.rs index ed723be6fe..371b70664f 100644 --- a/lib/codegen/src/ir/extname.rs +++ b/lib/codegen/src/ir/extname.rs @@ -111,7 +111,7 @@ impl FromStr for ExternalName { // Try to parse as a libcall name, otherwise it's a test case. match s.parse() { Ok(lc) => Ok(ExternalName::LibCall(lc)), - Err(_) => Ok(ExternalName::testcase(s.as_bytes())), + Err(_) => Ok(Self::testcase(s.as_bytes())), } } } diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs index 961c6fea30..0dd0eb7991 100644 --- a/lib/codegen/src/timing.rs +++ b/lib/codegen/src/timing.rs @@ -154,8 +154,7 @@ mod details { fn fmtdur(mut dur: Duration, f: &mut fmt::Formatter) -> fmt::Result { // Round to nearest ms by adding 500us. dur += Duration::new(0, 500_000); - let ms = dur.subsec_nanos() / 1_000_000; - write!(f, "{:4}.{:03} ", dur.as_secs(), ms) + write!(f, "{:4}.{:03} ", dur.as_secs(), dur.subsec_millis()) } fmtdur(time.total, f)?; diff --git a/lib/codegen/src/verifier/locations.rs b/lib/codegen/src/verifier/locations.rs index d7378f50b7..97dfbdc6fb 100644 --- a/lib/codegen/src/verifier/locations.rs +++ b/lib/codegen/src/verifier/locations.rs @@ -69,10 +69,8 @@ impl<'a> LocationVerifier<'a> { let opcode = dfg[inst].opcode(); if opcode.is_return() { self.check_return_abi(inst, &divert)?; - } else if opcode.is_branch() { - if !divert.is_empty() { - self.check_cfg_edges(inst, &divert)?; - } + } else if opcode.is_branch() && !divert.is_empty() { + self.check_cfg_edges(inst, &divert)?; } self.update_diversions(inst, &mut divert)?; From 7204026bc8ce9b9e7354464ccb3d07d69159ee4f Mon Sep 17 00:00:00 2001 From: Benjamin Bouvier Date: Tue, 10 Jul 2018 17:06:03 +0200 Subject: [PATCH 1896/3084] Don't display stderr for the testing command in check.sh's runif; And remove an obsolete comment in check-clippy.sh. --- check-clippy.sh | 2 +- lib/codegen/meta/check.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/check-clippy.sh b/check-clippy.sh index 20ac595851..072069cb9b 100755 --- a/check-clippy.sh +++ b/check-clippy.sh @@ -1,7 +1,7 @@ #!/bin/bash set -euo pipefail -# Usage: check-clippy.sh [--install] +# Usage: check-clippy.sh if cargo install --list | tee /dev/null | grep -q "^clippy v0"; then exit 0 diff --git a/lib/codegen/meta/check.sh b/lib/codegen/meta/check.sh index 14e51da065..7b9a9e7760 100755 --- a/lib/codegen/meta/check.sh +++ b/lib/codegen/meta/check.sh @@ -4,7 +4,7 @@ topdir=$(dirname "$0") cd "$topdir" function runif { - if type "$1" > /dev/null; then + if type "$1" > /dev/null 2>&1; then version=$("$1" --version 2>&1) echo " === $1: $version ===" "$@" From 65ef4a7583bbe485d81c90b268b928df6f745d5f Mon Sep 17 00:00:00 2001 From: Amanieu d'Antras Date: Wed, 11 Jul 2018 21:02:22 +0100 Subject: [PATCH 1897/3084] Add EntityList::from_slice to build a list from an existing slice --- lib/entity/src/list.rs | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/lib/entity/src/list.rs b/lib/entity/src/list.rs index 3c5bd25a13..f94f8ef56e 100644 --- a/lib/entity/src/list.rs +++ b/lib/entity/src/list.rs @@ -225,6 +225,23 @@ impl EntityList { Default::default() } + /// Create a new list with the contents initialized from a slice. + pub fn from_slice(slice: &[T], pool: &mut ListPool) -> Self { + let len = slice.len(); + if len == 0 { + return EntityList::new(); + } + + let block = pool.alloc(sclass_for_length(len)); + pool.data[block] = T::new(len); + pool.data[block + 1..block + len + 1].copy_from_slice(slice); + + EntityList { + index: (block + 1) as u32, + unused: PhantomData, + } + } + /// Returns `true` if the list has a length of 0. pub fn is_empty(&self) -> bool { // 0 is a magic value for the empty list. Any list in the pool array must have a positive @@ -538,6 +555,25 @@ mod tests { assert_eq!(list.first(pool), None); } + #[test] + fn from_slice() { + let pool = &mut ListPool::::new(); + + let list = EntityList::::from_slice(&[Inst(0), Inst(1)], pool); + assert!(!list.is_empty()); + assert_eq!(list.len(pool), 2); + assert_eq!(list.as_slice(pool), &[Inst(0), Inst(1)]); + assert_eq!(list.get(0, pool), Some(Inst(0))); + assert_eq!(list.get(100, pool), None); + + let list = EntityList::::from_slice(&[], pool); + assert!(list.is_empty()); + assert_eq!(list.len(pool), 0); + assert_eq!(list.as_slice(pool), &[]); + assert_eq!(list.get(0, pool), None); + assert_eq!(list.get(100, pool), None); + } + #[test] fn push() { let pool = &mut ListPool::::new(); From 19a636af96b7ba9ad60ffd726d26c5773f24dce0 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 13 Jun 2018 07:53:36 -0700 Subject: [PATCH 1898/3084] Change Cretonne's license to "Apache-2.0 WITH LLVM-exception". This adds the "LLVM-exception" to Cretonne's existing Apache-2.0 license. https://spdx.org/licenses/LLVM-exception.html --- cranelift/Cargo.toml | 2 +- cranelift/LICENSE | 17 +++++++++++++++++ lib/codegen/Cargo.toml | 2 +- lib/entity/Cargo.toml | 2 +- lib/faerie/Cargo.toml | 2 +- lib/filetests/Cargo.toml | 2 +- lib/frontend/Cargo.toml | 2 +- lib/module/Cargo.toml | 2 +- lib/native/Cargo.toml | 2 +- lib/reader/Cargo.toml | 2 +- lib/simplejit/Cargo.toml | 2 +- lib/umbrella/Cargo.toml | 2 +- lib/wasm/Cargo.toml | 2 +- 13 files changed, 29 insertions(+), 12 deletions(-) diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 3de5d020bb..4011542acb 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -3,7 +3,7 @@ name = "cretonne-tools" authors = ["The Cretonne Project Developers"] version = "0.13.0" description = "Binaries for testing the Cretonne libraries" -license = "Apache-2.0" +license = "Apache-2.0 WITH LLVM-exception" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/cretonne/cretonne" publish = false diff --git a/cranelift/LICENSE b/cranelift/LICENSE index d645695673..be1d7c438a 100644 --- a/cranelift/LICENSE +++ b/cranelift/LICENSE @@ -200,3 +200,20 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. + + +--- LLVM Exceptions to the Apache 2.0 License ---- + +As an exception, if, as a result of your compiling your source code, portions +of this Software are embedded into an Object form of such source code, you +may redistribute such embedded portions in such Object form without complying +with the conditions of Sections 4(a), 4(b) and 4(d) of the License. + +In addition, if you combine or link compiled forms of this Software with +software that is licensed under the GPLv2 ("Combined Software") and if a +court of competent jurisdiction determines that the patent provision (Section +3), the indemnity provision (Section 9) or other Section of the License +conflicts with the conditions of the GPLv2, you may retroactively and +prospectively choose to deem waived or otherwise exclude such Section(s) of +the License, but only in their entirety and only with respect to the Combined +Software. diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index 415aa6534a..e5a50d258b 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -3,7 +3,7 @@ authors = ["The Cretonne Project Developers"] name = "cretonne-codegen" version = "0.13.0" description = "Low-level code generator library" -license = "Apache-2.0" +license = "Apache-2.0 WITH LLVM-exception" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/cretonne/cretonne" readme = "README.md" diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index d438f4e065..a836db8293 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -3,7 +3,7 @@ authors = ["The Cretonne Project Developers"] name = "cretonne-entity" version = "0.13.0" description = "Data structures using entity references as mapping keys" -license = "Apache-2.0" +license = "Apache-2.0 WITH LLVM-exception" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/cretonne/cretonne" readme = "README.md" diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index cc248678d3..08760e8cd3 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -5,7 +5,7 @@ authors = ["The Cretonne Project Developers"] description = "Emit Cretonne output to native object files with Faerie" repository = "https://github.com/cretonne/cretonne" documentation = "https://cretonne.readthedocs.io/" -license = "Apache-2.0" +license = "Apache-2.0 WITH LLVM-exception" readme = "README.md" [dependencies] diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 2ff5589d16..2ed1e7db8d 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -3,7 +3,7 @@ name = "cretonne-filetests" authors = ["The Cretonne Project Developers"] version = "0.13.0" description = "Test driver and implementations of the filetest commands" -license = "Apache-2.0" +license = "Apache-2.0 WITH LLVM-exception" documentation = "https://cretonne.readthedocs.io/en/latest/testing.html#file-tests" repository = "https://github.com/cretonne/cretonne" publish = false diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index aef504dfe6..9740c124bc 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -3,7 +3,7 @@ authors = ["The Cretonne Project Developers"] name = "cretonne-frontend" version = "0.13.0" description = "Cretonne IR builder helper" -license = "Apache-2.0" +license = "Apache-2.0 WITH LLVM-exception" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/cretonne/cretonne" readme = "README.md" diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index de97231b33..85bc72d9eb 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -5,7 +5,7 @@ authors = ["The Cretonne Project Developers"] description = "Support for linking functions and data with Cretonne" repository = "https://github.com/cretonne/cretonne" documentation = "https://cretonne.readthedocs.io/" -license = "Apache-2.0" +license = "Apache-2.0 WITH LLVM-exception" readme = "README.md" [dependencies] diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 0efa247104..1c566349f3 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -4,7 +4,7 @@ version = "0.13.0" authors = ["The Cretonne Project Developers"] description = "Support for targeting the host with Cretonne" repository = "https://github.com/cretonne/cretonne" -license = "Apache-2.0" +license = "Apache-2.0 WITH LLVM-exception" readme = "README.md" [dependencies] diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index b6b3a19b50..252587e5cf 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -3,7 +3,7 @@ authors = ["The Cretonne Project Developers"] name = "cretonne-reader" version = "0.13.0" description = "Cretonne textual IR reader" -license = "Apache-2.0" +license = "Apache-2.0 WITH LLVM-exception" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/cretonne/cretonne" readme = "README.md" diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml index 8d359ac48d..06f9f1b33c 100644 --- a/lib/simplejit/Cargo.toml +++ b/lib/simplejit/Cargo.toml @@ -5,7 +5,7 @@ authors = ["The Cretonne Project Developers"] description = "A simple JIT library backed by Cretonne" repository = "https://github.com/cretonne/cretonne" documentation = "https://cretonne.readthedocs.io/" -license = "Apache-2.0" +license = "Apache-2.0 WITH LLVM-exception" readme = "README.md" [dependencies] diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml index a23be961cb..ef3b6a9db5 100644 --- a/lib/umbrella/Cargo.toml +++ b/lib/umbrella/Cargo.toml @@ -3,7 +3,7 @@ authors = ["The Cretonne Project Developers"] name = "cretonne" version = "0.13.0" description = "Umbrella for commonly-used cretonne crates" -license = "Apache-2.0" +license = "Apache-2.0 WITH LLVM-exception" documentation = "https://cretonne.readthedocs.io/" repository = "https://github.com/cretonne/cretonne" readme = "README.md" diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml index 4fece39a1c..8db1c1665e 100644 --- a/lib/wasm/Cargo.toml +++ b/lib/wasm/Cargo.toml @@ -4,7 +4,7 @@ version = "0.13.0" authors = ["The Cretonne Project Developers"] description = "Translator from WebAssembly to Cretonne IR" repository = "https://github.com/cretonne/cretonne" -license = "Apache-2.0" +license = "Apache-2.0 WITH LLVM-exception" readme = "README.md" keywords = ["webassembly", "wasm"] From f4dbd38a4c4e46963468e64a4521080847ed697c Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 13 Jul 2018 09:01:28 -0700 Subject: [PATCH 1899/3084] Rename Cretonne to Cranelift! --- README.rst | 76 ++++++------- clippy-all.sh | 2 +- cranelift/.gitignore | 2 +- cranelift/Cargo.toml | 36 +++---- cranelift/FUZZING.md | 2 +- cranelift/docs/Makefile | 2 +- cranelift/docs/{callex.cton => callex.clif} | 0 .../docs/{cton_domain.py => clif_domain.py} | 68 ++++++------ .../docs/{cton_lexer.py => clif_lexer.py} | 12 +-- cranelift/docs/compare-llvm.rst | 72 ++++++------- cranelift/docs/conf.py | 24 ++--- cranelift/docs/{example.cton => example.clif} | 0 .../docs/{heapex-dyn.cton => heapex-dyn.clif} | 0 .../{heapex-sm32.cton => heapex-sm32.clif} | 0 .../{heapex-sm64.cton => heapex-sm64.clif} | 0 cranelift/docs/index.rst | 32 +++--- cranelift/docs/langref.rst | 102 +++++++++--------- cranelift/docs/make.bat | 2 +- cranelift/docs/metaref.rst | 28 ++--- cranelift/docs/regalloc.rst | 22 ++-- cranelift/docs/testing.rst | 30 +++--- .../filetests/cfg/{loop.cton => loop.clif} | 0 .../{traps_early.cton => traps_early.clif} | 0 .../{unused_node.cton => unused_node.clif} | 0 .../filetests/dce/{basic.cton => basic.clif} | 0 .../domtree/{basic.cton => basic.clif} | 0 .../domtree/{loops.cton => loops.clif} | 0 .../domtree/{loops2.cton => loops2.clif} | 0 .../{tall-tree.cton => tall-tree.clif} | 0 .../{wide-tree.cton => wide-tree.clif} | 0 .../isa/riscv/{abi-e.cton => abi-e.clif} | 0 .../isa/riscv/{abi.cton => abi.clif} | 0 .../riscv/{binary32.cton => binary32.clif} | 0 .../riscv/{encoding.cton => encoding.clif} | 0 .../{expand-i32.cton => expand-i32.clif} | 0 .../{legalize-abi.cton => legalize-abi.clif} | 0 .../{legalize-i64.cton => legalize-i64.clif} | 0 ...arse-encoding.cton => parse-encoding.clif} | 0 .../isa/riscv/{regmove.cton => regmove.clif} | 0 .../{split-args.cton => split-args.clif} | 0 ...ify-encoding.cton => verify-encoding.clif} | 0 .../isa/x86/{abcd.cton => abcd.clif} | 0 .../isa/x86/{abi-bool.cton => abi-bool.clif} | 0 .../isa/x86/{abi32.cton => abi32.clif} | 0 .../isa/x86/{abi64.cton => abi64.clif} | 0 ...caddrs32.cton => allones_funcaddrs32.clif} | 4 +- ...caddrs64.cton => allones_funcaddrs64.clif} | 4 +- ...nt.cton => baseline_clz_ctz_popcount.clif} | 0 ...> baseline_clz_ctz_popcount_encoding.clif} | 2 +- ...inary32-float.cton => binary32-float.clif} | 2 +- .../isa/x86/{binary32.cton => binary32.clif} | 2 +- ...inary64-float.cton => binary64-float.clif} | 2 +- .../{binary64-pic.cton => binary64-pic.clif} | 2 +- .../isa/x86/{binary64.cton => binary64.clif} | 2 +- ...{legalize-bnot.cton => legalize-bnot.clif} | 0 ...e-br-table.cton => legalize-br-table.clif} | 0 ...{legalize-call.cton => legalize-call.clif} | 0 ...alize-custom.cton => legalize-custom.clif} | 0 ...div-traps.cton => legalize-div-traps.clif} | 2 +- .../{legalize-div.cton => legalize-div.clif} | 2 +- ...iconst-i8.cton => legalize-iconst-i8.clif} | 0 ...ize-imul-i8.cton => legalize-imul-i8.clif} | 0 ...-imm-i8.cton => legalize-imul-imm-i8.clif} | 0 ...ize-libcall.cton => legalize-libcall.clif} | 0 ...re-i8.cton => legalize-load-store-i8.clif} | 0 ...alize-memory.cton => legalize-memory.clif} | 0 ...egalize-mulhi.cton => legalize-mulhi.clif} | 0 ...gmove-i8.cton => legalize-regmove-i8.clif} | 0 ...on => optimized-zero-constants-32bit.clif} | 0 ...nts.cton => optimized-zero-constants.clif} | 0 ...sts-sp.cton => probestack-adjusts-sp.clif} | 2 +- ...disabled.cton => probestack-disabled.clif} | 2 +- ...ated.cton => probestack-noncolocated.clif} | 2 +- ...bestack-size.cton => probestack-size.clif} | 2 +- .../x86/{probestack.cton => probestack.clif} | 0 ...e-epilogue.cton => prologue-epilogue.clif} | 0 ...le-uses.cton => shrink-multiple-uses.clif} | 0 .../isa/x86/{shrink.cton => shrink.clif} | 2 +- .../{stack-addr64.cton => stack-addr64.clif} | 2 +- ...d-store64.cton => stack-load-store64.clif} | 0 ...all_x64.cton => windows_fastcall_x64.clif} | 0 .../filetests/licm/{basic.cton => basic.clif} | 0 .../licm/{complex.cton => complex.clif} | 0 ...tiple-blocks.cton => multiple-blocks.clif} | 0 .../{nested_loops.cton => nested_loops.clif} | 0 .../licm/{reject.cton => reject.clif} | 0 .../parser/{alias.cton => alias.clif} | 0 .../parser/{branch.cton => branch.clif} | 0 .../filetests/parser/{call.cton => call.clif} | 0 .../parser/{flags.cton => flags.clif} | 0 ...ncoding.cton => instruction_encoding.clif} | 0 .../parser/{keywords.cton => keywords.clif} | 0 .../parser/{memory.cton => memory.clif} | 0 .../parser/{preamble.cton => preamble.clif} | 0 .../parser/{rewrite.cton => rewrite.clif} | 0 .../parser/{ternary.cton => ternary.clif} | 0 .../filetests/parser/{tiny.cton => tiny.clif} | 0 .../postopt/{basic.cton => basic.clif} | 0 ...emory_ops.cton => complex_memory_ops.clif} | 0 ...direct.cton => div_by_const_indirect.clif} | 0 ....cton => div_by_const_non_power_of_2.clif} | 0 ...of_2.cton => div_by_const_power_of_2.clif} | 0 ....cton => rem_by_const_non_power_of_2.clif} | 0 ...of_2.cton => rem_by_const_power_of_2.clif} | 0 .../preopt/{simplify.cton => simplify.clif} | 0 .../regalloc/{aliases.cton => aliases.clif} | 0 .../regalloc/{basic.cton => basic.clif} | 0 .../regalloc/{coalesce.cton => coalesce.clif} | 0 ...oalescing-207.cton => coalescing-207.clif} | 2 +- ...oalescing-216.cton => coalescing-216.clif} | 2 +- .../{coloring-227.cton => coloring-227.clif} | 0 .../{constraints.cton => constraints.clif} | 0 .../{ghost-param.cton => ghost-param.clif} | 0 ...nstraints.cton => global-constraints.clif} | 0 .../{global-fixed.cton => global-fixed.clif} | 0 ...erence.cton => infinite-interference.clif} | 0 .../regalloc/{iterate.cton => iterate.clif} | 0 ...onstraints.cton => multi-constraints.clif} | 0 ...rference.cton => output-interference.clif} | 0 .../{reload-208.cton => reload-208.clif} | 2 +- .../regalloc/{reload.cton => reload.clif} | 0 ...chedule-moves.cton => schedule-moves.clif} | 0 .../{spill-noregs.cton => spill-noregs.clif} | 0 .../regalloc/{spill.cton => spill.clif} | 0 ...chable_code.cton => unreachable_code.clif} | 0 .../{x86-regres.cton => x86-regres.clif} | 0 .../simple_gvn/{basic.cton => basic.clif} | 0 .../simple_gvn/{reject.cton => reject.clif} | 0 .../simple_gvn/{scopes.cton => scopes.clif} | 0 .../{bad_layout.cton => bad_layout.clif} | 0 ...tes_uses.cton => defs_dominates_uses.clif} | 0 .../verifier/{flags.cton => flags.clif} | 0 .../verifier/{memory.cton => memory.clif} | 0 ...{return_at_end.cton => return_at_end.clif} | 0 .../{type_check.cton => type_check.clif} | 0 ...lared_vmctx.cton => undeclared_vmctx.clif} | 0 ...chable_code.cton => unreachable_code.clif} | 0 .../wasm/{control.cton => control.clif} | 0 .../{conversions.cton => conversions.clif} | 0 .../wasm/{f32-arith.cton => f32-arith.clif} | 0 .../{f32-compares.cton => f32-compares.clif} | 0 .../{f32-memory64.cton => f32-memory64.clif} | 0 .../wasm/{f64-arith.cton => f64-arith.clif} | 0 .../{f64-compares.cton => f64-compares.clif} | 0 .../{f64-memory64.cton => f64-memory64.clif} | 0 .../wasm/{i32-arith.cton => i32-arith.clif} | 0 .../{i32-compares.cton => i32-compares.clif} | 0 .../{i32-memory64.cton => i32-memory64.clif} | 0 .../wasm/{i64-arith.cton => i64-arith.clif} | 0 .../{i64-compares.cton => i64-compares.clif} | 0 .../{i64-memory64.cton => i64-memory64.clif} | 0 .../wasm/{select.cton => select.clif} | 0 cranelift/fuzz/Cargo.toml | 4 +- cranelift/fuzz/fuzz_translate_module.rs | 4 +- cranelift/publish-all.sh | 10 +- cranelift/src/cat.rs | 4 +- cranelift/src/{cton-util.rs => clif-util.rs} | 48 ++++----- cranelift/src/compile.rs | 14 +-- cranelift/src/print_cfg.rs | 6 +- cranelift/src/utils.rs | 8 +- cranelift/src/wasm.rs | 12 +-- cranelift/test-all.sh | 2 +- cranelift/tests/filetests.rs | 4 +- lib/codegen/Cargo.toml | 16 +-- lib/codegen/README.md | 2 +- lib/codegen/build.rs | 14 +-- lib/codegen/meta/base/__init__.py | 2 +- lib/codegen/meta/base/entities.py | 4 +- lib/codegen/meta/base/formats.py | 4 +- lib/codegen/meta/base/immediates.py | 14 +-- lib/codegen/meta/base/instructions.py | 12 +-- lib/codegen/meta/base/legalize.py | 2 +- lib/codegen/meta/base/predicates.py | 2 +- lib/codegen/meta/base/settings.py | 12 +-- lib/codegen/meta/base/types.py | 2 +- lib/codegen/meta/build.py | 2 +- lib/codegen/meta/cdsl/__init__.py | 4 +- lib/codegen/meta/cdsl/ast.py | 2 +- lib/codegen/meta/cdsl/formats.py | 2 +- lib/codegen/meta/cdsl/operands.py | 4 +- lib/codegen/meta/cdsl/predicates.py | 2 +- lib/codegen/meta/cdsl/settings.py | 2 +- lib/codegen/meta/cdsl/types.py | 2 +- lib/codegen/meta/cdsl/typevar.py | 2 +- lib/codegen/meta/gen_legalizer.py | 2 +- lib/codegen/meta/isa/__init__.py | 8 +- lib/codegen/meta/isa/riscv/encodings.py | 2 +- lib/codegen/meta/isa/x86/__init__.py | 4 +- lib/codegen/meta/isa/x86/recipes.py | 4 +- lib/codegen/meta/semantics/__init__.py | 2 +- lib/codegen/meta/semantics/elaborate.py | 2 +- lib/codegen/meta/semantics/primitives.py | 6 +- lib/codegen/meta/semantics/smtlib.py | 6 +- lib/codegen/src/binemit/memorysink.rs | 4 +- lib/codegen/src/binemit/mod.rs | 6 +- lib/codegen/src/binemit/relaxation.rs | 4 +- lib/codegen/src/binemit/shrink.rs | 2 +- lib/codegen/src/context.rs | 2 +- lib/codegen/src/cursor.rs | 52 ++++----- lib/codegen/src/dbg.rs | 12 +-- lib/codegen/src/ir/builder.rs | 4 +- lib/codegen/src/ir/condcodes.rs | 2 +- lib/codegen/src/ir/entities.rs | 4 +- lib/codegen/src/ir/extfunc.rs | 2 +- lib/codegen/src/ir/extname.rs | 14 +-- lib/codegen/src/ir/function.rs | 4 +- lib/codegen/src/ir/globalvalue.rs | 2 +- lib/codegen/src/ir/immediates.rs | 6 +- lib/codegen/src/ir/instructions.rs | 2 +- lib/codegen/src/ir/libcall.rs | 4 +- lib/codegen/src/ir/memflags.rs | 6 +- lib/codegen/src/ir/mod.rs | 2 +- lib/codegen/src/ir/progpoint.rs | 2 +- lib/codegen/src/ir/sourceloc.rs | 4 +- lib/codegen/src/ir/stackslot.rs | 2 +- lib/codegen/src/ir/types.rs | 2 +- lib/codegen/src/isa/mod.rs | 12 +-- lib/codegen/src/isa/x86/abi.rs | 4 +- lib/codegen/src/legalizer/split.rs | 10 +- lib/codegen/src/lib.rs | 6 +- lib/codegen/src/print_errors.rs | 2 +- lib/codegen/src/regalloc/liveness.rs | 26 ++--- lib/codegen/src/regalloc/liverange.rs | 2 +- lib/codegen/src/result.rs | 10 +- lib/codegen/src/settings.rs | 2 +- lib/codegen/src/timing.rs | 4 +- lib/codegen/src/verifier/cssa.rs | 2 +- lib/codegen/src/write.rs | 4 +- lib/entity/Cargo.toml | 10 +- lib/entity/README.md | 2 +- lib/entity/src/packed_option.rs | 2 +- lib/faerie/Cargo.toml | 16 +-- lib/faerie/README.md | 2 +- lib/faerie/src/backend.rs | 24 ++--- lib/faerie/src/container.rs | 6 +- lib/faerie/src/lib.rs | 6 +- lib/faerie/src/traps.rs | 10 +- lib/filetests/Cargo.toml | 12 +-- lib/filetests/src/concurrent.rs | 2 +- lib/filetests/src/lib.rs | 14 +-- lib/filetests/src/runner.rs | 4 +- lib/filetests/src/runone.rs | 16 +-- lib/filetests/src/subtest.rs | 10 +- lib/filetests/src/test_binemit.rs | 16 +-- lib/filetests/src/test_cat.rs | 4 +- lib/filetests/src/test_compile.rs | 10 +- lib/filetests/src/test_dce.rs | 10 +- lib/filetests/src/test_domtree.rs | 12 +-- lib/filetests/src/test_legalizer.rs | 10 +- lib/filetests/src/test_licm.rs | 10 +- lib/filetests/src/test_postopt.rs | 10 +- lib/filetests/src/test_preopt.rs | 10 +- lib/filetests/src/test_print_cfg.rs | 8 +- lib/filetests/src/test_regalloc.rs | 10 +- lib/filetests/src/test_shrink.rs | 10 +- lib/filetests/src/test_simple_gvn.rs | 10 +- lib/filetests/src/test_verifier.rs | 8 +- lib/frontend/Cargo.toml | 18 ++-- lib/frontend/README.md | 4 +- lib/frontend/src/frontend.rs | 50 ++++----- lib/frontend/src/lib.rs | 30 +++--- lib/frontend/src/ssa.rs | 32 +++--- lib/frontend/src/variable.rs | 2 +- lib/module/Cargo.toml | 20 ++-- lib/module/README.md | 4 +- lib/module/src/backend.rs | 6 +- lib/module/src/data_context.rs | 10 +- lib/module/src/lib.rs | 6 +- lib/module/src/module.rs | 6 +- lib/native/Cargo.toml | 16 +-- lib/native/README.md | 2 +- lib/native/src/lib.rs | 8 +- lib/reader/Cargo.toml | 14 +-- lib/reader/README.md | 4 +- lib/reader/src/isaspec.rs | 6 +- lib/reader/src/lexer.rs | 10 +- lib/reader/src/lib.rs | 8 +- lib/reader/src/parser.rs | 34 +++--- lib/reader/src/sourcemap.rs | 4 +- lib/reader/src/testcommand.rs | 2 +- lib/reader/src/testfile.rs | 8 +- lib/simplejit/Cargo.toml | 22 ++-- lib/simplejit/README.md | 2 +- lib/simplejit/src/backend.rs | 14 +-- lib/simplejit/src/lib.rs | 8 +- lib/umbrella/Cargo.toml | 20 ++-- lib/umbrella/README.md | 2 +- lib/umbrella/src/lib.rs | 10 +- lib/wasm/Cargo.toml | 18 ++-- lib/wasm/README.md | 2 +- lib/wasm/src/code_translator.rs | 28 ++--- lib/wasm/src/environ/dummy.rs | 8 +- lib/wasm/src/environ/spec.rs | 18 ++-- lib/wasm/src/func_translator.rs | 24 ++--- lib/wasm/src/lib.rs | 6 +- lib/wasm/src/module_translator.rs | 4 +- lib/wasm/src/sections_translator.rs | 2 +- lib/wasm/src/state.rs | 2 +- lib/wasm/src/translation_utils.rs | 10 +- lib/wasm/tests/wasm_testsuite.rs | 12 +-- misc/vim/ftdetect/clif.vim | 1 + misc/vim/ftdetect/cton.vim | 1 - misc/vim/syntax/clif.vim | 44 ++++++++ misc/vim/syntax/cton.vim | 44 -------- rustc.rst | 14 +-- spidermonkey.rst | 16 +-- 306 files changed, 977 insertions(+), 975 deletions(-) rename cranelift/docs/{callex.cton => callex.clif} (100%) rename cranelift/docs/{cton_domain.py => clif_domain.py} (89%) rename cranelift/docs/{cton_lexer.py => clif_lexer.py} (92%) rename cranelift/docs/{example.cton => example.clif} (100%) rename cranelift/docs/{heapex-dyn.cton => heapex-dyn.clif} (100%) rename cranelift/docs/{heapex-sm32.cton => heapex-sm32.clif} (100%) rename cranelift/docs/{heapex-sm64.cton => heapex-sm64.clif} (100%) rename cranelift/filetests/cfg/{loop.cton => loop.clif} (100%) rename cranelift/filetests/cfg/{traps_early.cton => traps_early.clif} (100%) rename cranelift/filetests/cfg/{unused_node.cton => unused_node.clif} (100%) rename cranelift/filetests/dce/{basic.cton => basic.clif} (100%) rename cranelift/filetests/domtree/{basic.cton => basic.clif} (100%) rename cranelift/filetests/domtree/{loops.cton => loops.clif} (100%) rename cranelift/filetests/domtree/{loops2.cton => loops2.clif} (100%) rename cranelift/filetests/domtree/{tall-tree.cton => tall-tree.clif} (100%) rename cranelift/filetests/domtree/{wide-tree.cton => wide-tree.clif} (100%) rename cranelift/filetests/isa/riscv/{abi-e.cton => abi-e.clif} (100%) rename cranelift/filetests/isa/riscv/{abi.cton => abi.clif} (100%) rename cranelift/filetests/isa/riscv/{binary32.cton => binary32.clif} (100%) rename cranelift/filetests/isa/riscv/{encoding.cton => encoding.clif} (100%) rename cranelift/filetests/isa/riscv/{expand-i32.cton => expand-i32.clif} (100%) rename cranelift/filetests/isa/riscv/{legalize-abi.cton => legalize-abi.clif} (100%) rename cranelift/filetests/isa/riscv/{legalize-i64.cton => legalize-i64.clif} (100%) rename cranelift/filetests/isa/riscv/{parse-encoding.cton => parse-encoding.clif} (100%) rename cranelift/filetests/isa/riscv/{regmove.cton => regmove.clif} (100%) rename cranelift/filetests/isa/riscv/{split-args.cton => split-args.clif} (100%) rename cranelift/filetests/isa/riscv/{verify-encoding.cton => verify-encoding.clif} (100%) rename cranelift/filetests/isa/x86/{abcd.cton => abcd.clif} (100%) rename cranelift/filetests/isa/x86/{abi-bool.cton => abi-bool.clif} (100%) rename cranelift/filetests/isa/x86/{abi32.cton => abi32.clif} (100%) rename cranelift/filetests/isa/x86/{abi64.cton => abi64.clif} (100%) rename cranelift/filetests/isa/x86/{allones_funcaddrs32.cton => allones_funcaddrs32.clif} (84%) rename cranelift/filetests/isa/x86/{allones_funcaddrs64.cton => allones_funcaddrs64.clif} (87%) rename cranelift/filetests/isa/x86/{baseline_clz_ctz_popcount.cton => baseline_clz_ctz_popcount.clif} (100%) rename cranelift/filetests/isa/x86/{baseline_clz_ctz_popcount_encoding.cton => baseline_clz_ctz_popcount_encoding.clif} (97%) rename cranelift/filetests/isa/x86/{binary32-float.cton => binary32-float.clif} (99%) rename cranelift/filetests/isa/x86/{binary32.cton => binary32.clif} (99%) rename cranelift/filetests/isa/x86/{binary64-float.cton => binary64-float.clif} (99%) rename cranelift/filetests/isa/x86/{binary64-pic.cton => binary64-pic.clif} (96%) rename cranelift/filetests/isa/x86/{binary64.cton => binary64.clif} (99%) rename cranelift/filetests/isa/x86/{legalize-bnot.cton => legalize-bnot.clif} (100%) rename cranelift/filetests/isa/x86/{legalize-br-table.cton => legalize-br-table.clif} (100%) rename cranelift/filetests/isa/x86/{legalize-call.cton => legalize-call.clif} (100%) rename cranelift/filetests/isa/x86/{legalize-custom.cton => legalize-custom.clif} (100%) rename cranelift/filetests/isa/x86/{legalize-div-traps.cton => legalize-div-traps.clif} (98%) rename cranelift/filetests/isa/x86/{legalize-div.cton => legalize-div.clif} (97%) rename cranelift/filetests/isa/x86/{legalize-iconst-i8.cton => legalize-iconst-i8.clif} (100%) rename cranelift/filetests/isa/x86/{legalize-imul-i8.cton => legalize-imul-i8.clif} (100%) rename cranelift/filetests/isa/x86/{legalize-imul-imm-i8.cton => legalize-imul-imm-i8.clif} (100%) rename cranelift/filetests/isa/x86/{legalize-libcall.cton => legalize-libcall.clif} (100%) rename cranelift/filetests/isa/x86/{legalize-load-store-i8.cton => legalize-load-store-i8.clif} (100%) rename cranelift/filetests/isa/x86/{legalize-memory.cton => legalize-memory.clif} (100%) rename cranelift/filetests/isa/x86/{legalize-mulhi.cton => legalize-mulhi.clif} (100%) rename cranelift/filetests/isa/x86/{legalize-regmove-i8.cton => legalize-regmove-i8.clif} (100%) rename cranelift/filetests/isa/x86/{optimized-zero-constants-32bit.cton => optimized-zero-constants-32bit.clif} (100%) rename cranelift/filetests/isa/x86/{optimized-zero-constants.cton => optimized-zero-constants.clif} (100%) rename cranelift/filetests/isa/x86/{probestack-adjusts-sp.cton => probestack-adjusts-sp.clif} (94%) rename cranelift/filetests/isa/x86/{probestack-disabled.cton => probestack-disabled.clif} (93%) rename cranelift/filetests/isa/x86/{probestack-noncolocated.cton => probestack-noncolocated.clif} (94%) rename cranelift/filetests/isa/x86/{probestack-size.cton => probestack-size.clif} (97%) rename cranelift/filetests/isa/x86/{probestack.cton => probestack.clif} (100%) rename cranelift/filetests/isa/x86/{prologue-epilogue.cton => prologue-epilogue.clif} (100%) rename cranelift/filetests/isa/x86/{shrink-multiple-uses.cton => shrink-multiple-uses.clif} (100%) rename cranelift/filetests/isa/x86/{shrink.cton => shrink.clif} (91%) rename cranelift/filetests/isa/x86/{stack-addr64.cton => stack-addr64.clif} (95%) rename cranelift/filetests/isa/x86/{stack-load-store64.cton => stack-load-store64.clif} (100%) rename cranelift/filetests/isa/x86/{windows_fastcall_x64.cton => windows_fastcall_x64.clif} (100%) rename cranelift/filetests/licm/{basic.cton => basic.clif} (100%) rename cranelift/filetests/licm/{complex.cton => complex.clif} (100%) rename cranelift/filetests/licm/{multiple-blocks.cton => multiple-blocks.clif} (100%) rename cranelift/filetests/licm/{nested_loops.cton => nested_loops.clif} (100%) rename cranelift/filetests/licm/{reject.cton => reject.clif} (100%) rename cranelift/filetests/parser/{alias.cton => alias.clif} (100%) rename cranelift/filetests/parser/{branch.cton => branch.clif} (100%) rename cranelift/filetests/parser/{call.cton => call.clif} (100%) rename cranelift/filetests/parser/{flags.cton => flags.clif} (100%) rename cranelift/filetests/parser/{instruction_encoding.cton => instruction_encoding.clif} (100%) rename cranelift/filetests/parser/{keywords.cton => keywords.clif} (100%) rename cranelift/filetests/parser/{memory.cton => memory.clif} (100%) rename cranelift/filetests/parser/{preamble.cton => preamble.clif} (100%) rename cranelift/filetests/parser/{rewrite.cton => rewrite.clif} (100%) rename cranelift/filetests/parser/{ternary.cton => ternary.clif} (100%) rename cranelift/filetests/parser/{tiny.cton => tiny.clif} (100%) rename cranelift/filetests/postopt/{basic.cton => basic.clif} (100%) rename cranelift/filetests/postopt/{complex_memory_ops.cton => complex_memory_ops.clif} (100%) rename cranelift/filetests/preopt/{div_by_const_indirect.cton => div_by_const_indirect.clif} (100%) rename cranelift/filetests/preopt/{div_by_const_non_power_of_2.cton => div_by_const_non_power_of_2.clif} (100%) rename cranelift/filetests/preopt/{div_by_const_power_of_2.cton => div_by_const_power_of_2.clif} (100%) rename cranelift/filetests/preopt/{rem_by_const_non_power_of_2.cton => rem_by_const_non_power_of_2.clif} (100%) rename cranelift/filetests/preopt/{rem_by_const_power_of_2.cton => rem_by_const_power_of_2.clif} (100%) rename cranelift/filetests/preopt/{simplify.cton => simplify.clif} (100%) rename cranelift/filetests/regalloc/{aliases.cton => aliases.clif} (100%) rename cranelift/filetests/regalloc/{basic.cton => basic.clif} (100%) rename cranelift/filetests/regalloc/{coalesce.cton => coalesce.clif} (100%) rename cranelift/filetests/regalloc/{coalescing-207.cton => coalescing-207.clif} (99%) rename cranelift/filetests/regalloc/{coalescing-216.cton => coalescing-216.clif} (93%) rename cranelift/filetests/regalloc/{coloring-227.cton => coloring-227.clif} (100%) rename cranelift/filetests/regalloc/{constraints.cton => constraints.clif} (100%) rename cranelift/filetests/regalloc/{ghost-param.cton => ghost-param.clif} (100%) rename cranelift/filetests/regalloc/{global-constraints.cton => global-constraints.clif} (100%) rename cranelift/filetests/regalloc/{global-fixed.cton => global-fixed.clif} (100%) rename cranelift/filetests/regalloc/{infinite-interference.cton => infinite-interference.clif} (100%) rename cranelift/filetests/regalloc/{iterate.cton => iterate.clif} (100%) rename cranelift/filetests/regalloc/{multi-constraints.cton => multi-constraints.clif} (100%) rename cranelift/filetests/regalloc/{output-interference.cton => output-interference.clif} (100%) rename cranelift/filetests/regalloc/{reload-208.cton => reload-208.clif} (96%) rename cranelift/filetests/regalloc/{reload.cton => reload.clif} (100%) rename cranelift/filetests/regalloc/{schedule-moves.cton => schedule-moves.clif} (100%) rename cranelift/filetests/regalloc/{spill-noregs.cton => spill-noregs.clif} (100%) rename cranelift/filetests/regalloc/{spill.cton => spill.clif} (100%) rename cranelift/filetests/regalloc/{unreachable_code.cton => unreachable_code.clif} (100%) rename cranelift/filetests/regalloc/{x86-regres.cton => x86-regres.clif} (100%) rename cranelift/filetests/simple_gvn/{basic.cton => basic.clif} (100%) rename cranelift/filetests/simple_gvn/{reject.cton => reject.clif} (100%) rename cranelift/filetests/simple_gvn/{scopes.cton => scopes.clif} (100%) rename cranelift/filetests/verifier/{bad_layout.cton => bad_layout.clif} (100%) rename cranelift/filetests/verifier/{defs_dominates_uses.cton => defs_dominates_uses.clif} (100%) rename cranelift/filetests/verifier/{flags.cton => flags.clif} (100%) rename cranelift/filetests/verifier/{memory.cton => memory.clif} (100%) rename cranelift/filetests/verifier/{return_at_end.cton => return_at_end.clif} (100%) rename cranelift/filetests/verifier/{type_check.cton => type_check.clif} (100%) rename cranelift/filetests/verifier/{undeclared_vmctx.cton => undeclared_vmctx.clif} (100%) rename cranelift/filetests/verifier/{unreachable_code.cton => unreachable_code.clif} (100%) rename cranelift/filetests/wasm/{control.cton => control.clif} (100%) rename cranelift/filetests/wasm/{conversions.cton => conversions.clif} (100%) rename cranelift/filetests/wasm/{f32-arith.cton => f32-arith.clif} (100%) rename cranelift/filetests/wasm/{f32-compares.cton => f32-compares.clif} (100%) rename cranelift/filetests/wasm/{f32-memory64.cton => f32-memory64.clif} (100%) rename cranelift/filetests/wasm/{f64-arith.cton => f64-arith.clif} (100%) rename cranelift/filetests/wasm/{f64-compares.cton => f64-compares.clif} (100%) rename cranelift/filetests/wasm/{f64-memory64.cton => f64-memory64.clif} (100%) rename cranelift/filetests/wasm/{i32-arith.cton => i32-arith.clif} (100%) rename cranelift/filetests/wasm/{i32-compares.cton => i32-compares.clif} (100%) rename cranelift/filetests/wasm/{i32-memory64.cton => i32-memory64.clif} (100%) rename cranelift/filetests/wasm/{i64-arith.cton => i64-arith.clif} (100%) rename cranelift/filetests/wasm/{i64-compares.cton => i64-compares.clif} (100%) rename cranelift/filetests/wasm/{i64-memory64.cton => i64-memory64.clif} (100%) rename cranelift/filetests/wasm/{select.cton => select.clif} (100%) rename cranelift/src/{cton-util.rs => clif-util.rs} (74%) create mode 100644 misc/vim/ftdetect/clif.vim delete mode 100644 misc/vim/ftdetect/cton.vim create mode 100644 misc/vim/syntax/clif.vim delete mode 100644 misc/vim/syntax/cton.vim diff --git a/README.rst b/README.rst index 24a601cba4..aeeb3f4c4c 100644 --- a/README.rst +++ b/README.rst @@ -1,61 +1,61 @@ -======================= -Cretonne Code Generator -======================= +======================== +Cranelift Code Generator +======================== -Cretonne is a low-level retargetable code generator. It translates a `target-independent -intermediate representation `_ into executable +Cranelift is a low-level retargetable code generator. It translates a `target-independent +intermediate representation `_ into executable machine code. -.. image:: https://readthedocs.org/projects/cretonne/badge/?version=latest - :target: https://cretonne.readthedocs.io/en/latest/?badge=latest +.. image:: https://readthedocs.org/projects/cranelift/badge/?version=latest + :target: https://cranelift.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status -.. image:: https://travis-ci.org/cretonne/cretonne.svg?branch=master - :target: https://travis-ci.org/cretonne/cretonne +.. image:: https://travis-ci.org/CraneStation/cranelift.svg?branch=master + :target: https://travis-ci.org/CraneStation/cranelift :alt: Build Status -.. image:: https://badges.gitter.im/cretonne/cretonne.svg - :target: https://gitter.im/cretonne/Lobby/~chat +.. image:: https://badges.gitter.im/CraneStation/CraneStation.svg + :target: https://gitter.im/CraneStation/Lobby/~chat :alt: Gitter chat For more information, see `the documentation -`_. +`_. Status ------ -Cretonne currently supports enough functionality to run a wide variety of +Cranelift currently supports enough functionality to run a wide variety of programs, including all the functionality needed to execute WebAssembly MVP functions, although it needs to be used within an external WebAssembly embedding to be part of a complete WebAssembly implementation. The x86-64 backend is currently the most complete and stable; other -architectures are in various stages of development. Cretonne currently supports +architectures are in various stages of development. Cranelift currently supports the System V AMD64 ABI calling convention used on many platforms, but does not yet support the Windows x64 calling convention. The performance of code -produced by Cretonne is not yet impressive, though we have plans to fix that. +produced by Cranelift is not yet impressive, though we have plans to fix that. The core codegen crates have minimal dependencies, support `no_std <#building-with-no-std>`_ mode, and do not require any host floating-point support. -Cretonne does not yet perform mitigations for Spectre or related security +Cranelift does not yet perform mitigations for Spectre or related security issues, though it may do so in the future. It does not currently make any security-relevant instruction timing guarantees. It has seen a fair amount of testing and fuzzing, although more work is needed before it would be ready for a production use case. -Cretonne's APIs are not yet stable. +Cranelift's APIs are not yet stable. -Cretonne currently supports Rust 1.22.1 and later. We intend to always support +Cranelift currently supports Rust 1.22.1 and later. We intend to always support the latest *stable* Rust. And, we currently support the version of Rust in the latest Ubuntu LTS, although whether we will always do so is not yet determined. -Cretonne requires Python 2.7 or Python 3 to build. +Cranelift requires Python 2.7 or Python 3 to build. Planned uses ------------ -Cretonne is designed to be a code generator for WebAssembly, but it is general enough to be useful +Cranelift is designed to be a code generator for WebAssembly, but it is general enough to be useful elsewhere too. The initial planned uses that affected its design are: 1. `WebAssembly compiler for the SpiderMonkey engine in Firefox @@ -64,13 +64,13 @@ elsewhere too. The initial planned uses that affected its design are: `_. 3. `Debug build backend for the Rust compiler `_. -Building Cretonne ------------------ +Building Cranelift +------------------ -Cretonne is using the Cargo package manager format. First, ensure you have +Cranelift is using the Cargo package manager format. First, ensure you have installed a current stable rust (stable, beta, and nightly should all work, but only stable and beta are tested consistently). Then, change the working -directory to your clone of cretonne and run:: +directory to your clone of cranelift and run:: cargo build @@ -89,30 +89,30 @@ Building with `no_std` ---------------------- The following crates support `no_std`: - - `cretonne-entity` - - `cretonne-codegen` - - `cretonne-frontend` - - `cretonne-native` - - `cretonne-wasm` - - `cretonne-module` - - `cretonne-simplejit` - - `cretonne` + - `cranelift-entity` + - `cranelift-codegen` + - `cranelift-frontend` + - `cranelift-native` + - `cranelift-wasm` + - `cranelift-module` + - `cranelift-simplejit` + - `cranelift` To use `no_std` mode, disable the `std` feature and enable the `core` feature. This currently requires nightly rust. -For example, to build `cretonne-codegen`: +For example, to build `cranelift-codegen`: .. code-block:: sh cd lib/codegen cargo build --no-default-features --features core -Or, when using `cretonne-codegen` as a dependency (in Cargo.toml): +Or, when using `cranelift-codegen` as a dependency (in Cargo.toml): .. code-block:: - [dependency.cretonne-codegen] + [dependency.cranelift-codegen] ... default-features = false features = ["core"] @@ -125,7 +125,7 @@ build and test `no_std` when submitting patches. Accordingly, the There is a separate `./test-no_std.sh` script that tests the `no_std` support in packages which support it. -It's important to note that cretonne still needs liballoc to compile. +It's important to note that cranelift still needs liballoc to compile. Thus, whatever environment is used must implement an allocator. Also, to allow the use of HashMaps with `no_std`, an external crate called @@ -136,11 +136,11 @@ Just something to think about. Building the documentation -------------------------- -To build the Cretonne documentation, you need the `Sphinx documentation +To build the Cranelift documentation, you need the `Sphinx documentation generator `_:: $ pip install sphinx sphinx-autobuild sphinx_rtd_theme - $ cd cretonne/docs + $ cd cranelift/docs $ make html $ open _build/html/index.html diff --git a/clippy-all.sh b/clippy-all.sh index f82f6a07db..ba840b1d00 100755 --- a/clippy-all.sh +++ b/clippy-all.sh @@ -2,6 +2,6 @@ set -euo pipefail # Check all sources with clippy. -# In the cton-util crate (root dir) clippy will only work with nightly cargo - +# In the clif-util crate (root dir) clippy will only work with nightly cargo - # there is a bug where it will reject the commands passed to it by cargo 0.25.0 exec cargo +nightly clippy --all diff --git a/cranelift/.gitignore b/cranelift/.gitignore index 041b7ad3a8..1714fd39af 100644 --- a/cranelift/.gitignore +++ b/cranelift/.gitignore @@ -7,7 +7,7 @@ tags target Cargo.lock .*.rustfmt -cretonne.dbg* +cranelift.dbg* .mypy_cache rusty-tags.* docs/_build diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml index 4011542acb..381949a8da 100644 --- a/cranelift/Cargo.toml +++ b/cranelift/Cargo.toml @@ -1,29 +1,29 @@ [package] -name = "cretonne-tools" -authors = ["The Cretonne Project Developers"] +name = "cranelift-tools" +authors = ["The Cranelift Project Developers"] version = "0.13.0" -description = "Binaries for testing the Cretonne libraries" +description = "Binaries for testing the Cranelift libraries" license = "Apache-2.0 WITH LLVM-exception" -documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/cretonne/cretonne" +documentation = "https://cranelift.readthedocs.io/" +repository = "https://github.com/cranelift/cranelift" publish = false [[bin]] -name = "cton-util" -path = "src/cton-util.rs" +name = "clif-util" +path = "src/clif-util.rs" [dependencies] cfg-if = "0.1" -cretonne-codegen = { path = "lib/codegen", version = "0.13.0" } -cretonne-reader = { path = "lib/reader", version = "0.13.0" } -cretonne-frontend = { path = "lib/frontend", version = "0.13.0" } -cretonne-wasm = { path = "lib/wasm", version = "0.13.0", optional = true } -cretonne-native = { path = "lib/native", version = "0.13.0" } -cretonne-filetests = { path = "lib/filetests", version = "0.13.0" } -cretonne-module = { path = "lib/module", version = "0.13.0" } -cretonne-faerie = { path = "lib/faerie", version = "0.13.0" } -cretonne-simplejit = { path = "lib/simplejit", version = "0.13.0" } -cretonne = { path = "lib/umbrella", version = "0.13.0" } +cranelift-codegen = { path = "lib/codegen", version = "0.13.0" } +cranelift-reader = { path = "lib/reader", version = "0.13.0" } +cranelift-frontend = { path = "lib/frontend", version = "0.13.0" } +cranelift-wasm = { path = "lib/wasm", version = "0.13.0", optional = true } +cranelift-native = { path = "lib/native", version = "0.13.0" } +cranelift-filetests = { path = "lib/filetests", version = "0.13.0" } +cranelift-module = { path = "lib/module", version = "0.13.0" } +cranelift-faerie = { path = "lib/faerie", version = "0.13.0" } +cranelift-simplejit = { path = "lib/simplejit", version = "0.13.0" } +cranelift = { path = "lib/umbrella", version = "0.13.0" } filecheck = "0.3.0" docopt = "1" serde = "1.0.8" @@ -36,7 +36,7 @@ target-lexicon = "0.0.2" [features] default = ["disas", "wasm"] disas = ["capstone"] -wasm = ["wabt", "cretonne-wasm"] +wasm = ["wabt", "cranelift-wasm"] [workspace] diff --git a/cranelift/FUZZING.md b/cranelift/FUZZING.md index a542e139d6..55643a73b9 100644 --- a/cranelift/FUZZING.md +++ b/cranelift/FUZZING.md @@ -1,6 +1,6 @@ # Fuzzing -This document describes how to fuzz cretonne with [`cargo-fuzz`](https://github.com/rust-fuzz/cargo-fuzz). The fuzz targets use `wasm-opt` from [`binaryen-rs`](https://github.com/pepyakin/binaryen-rs) to generate valid WebAssembly modules from the fuzzed input supplied by `cargo-fuzz` (via [libfuzzer](http://llvm.org/docs/LibFuzzer.html)). In this scheme coverage feedback from both cretonne and the `wasm-opt` input generation code is used to inform the fuzzer. +This document describes how to fuzz cranelift with [`cargo-fuzz`](https://github.com/rust-fuzz/cargo-fuzz). The fuzz targets use `wasm-opt` from [`binaryen-rs`](https://github.com/pepyakin/binaryen-rs) to generate valid WebAssembly modules from the fuzzed input supplied by `cargo-fuzz` (via [libfuzzer](http://llvm.org/docs/LibFuzzer.html)). In this scheme coverage feedback from both cranelift and the `wasm-opt` input generation code is used to inform the fuzzer. # Usage diff --git a/cranelift/docs/Makefile b/cranelift/docs/Makefile index 6ac763bdd3..c66a99211d 100644 --- a/cranelift/docs/Makefile +++ b/cranelift/docs/Makefile @@ -5,7 +5,7 @@ SPHINXOPTS = SPHINXBUILD = sphinx-build SPHINXABUILD = sphinx-autobuild -SPHINXPROJ = cretonne +SPHINXPROJ = cranelift SOURCEDIR = . BUILDDIR = _build diff --git a/cranelift/docs/callex.cton b/cranelift/docs/callex.clif similarity index 100% rename from cranelift/docs/callex.cton rename to cranelift/docs/callex.clif diff --git a/cranelift/docs/cton_domain.py b/cranelift/docs/clif_domain.py similarity index 89% rename from cranelift/docs/cton_domain.py rename to cranelift/docs/clif_domain.py index 4d28fbdadc..80db764bd5 100644 --- a/cranelift/docs/cton_domain.py +++ b/cranelift/docs/clif_domain.py @@ -2,11 +2,11 @@ # # Sphinx domain for documenting compiler intermediate representations. # -# This defines a 'cton' Sphinx domain with the following directives and roles: +# This defines a 'clif' Sphinx domain with the following directives and roles: # -# .. cton::type:: type +# .. clif::type:: type # Document an IR type. -# .. cton:inst:: v0, v1 = inst op0, op1 +# .. clif:inst:: v0, v1 = inst op0, op1 # Document an IR instruction. # from __future__ import absolute_import @@ -27,12 +27,12 @@ from sphinx.util.nodes import make_refnode import sphinx.ext.autodoc -class CtonObject(ObjectDescription): +class ClifObject(ObjectDescription): """ - Any kind of Cretonne IR object. + Any kind of Cranelift IR object. This is a shared base class for the different kinds of indexable objects - in the Cretonne IR reference. + in the Cranelift IR reference. """ option_spec = { 'noindex': directives.flag, @@ -54,10 +54,10 @@ class CtonObject(ObjectDescription): signode['ids'].append(targetname) signode['first'] = (not self.names) self.state.document.note_explicit_target(signode) - inv = self.env.domaindata['cton']['objects'] + inv = self.env.domaindata['clif']['objects'] if name in inv: self.state_machine.reporter.warning( - 'duplicate Cretonne object description of %s, ' % name + + 'duplicate Cranelift object description of %s, ' % name + 'other instance in ' + self.env.doc2path(inv[name][0]), line=self.lineno) inv[name] = (self.env.docname, self.objtype) @@ -97,8 +97,8 @@ def parse_type(name, signode): return re_str -class CtonType(CtonObject): - """A Cretonne IR type description.""" +class ClifType(ClifObject): + """A Cranelift IR type description.""" def handle_signature(self, sig, signode): """ @@ -126,8 +126,8 @@ def parse_params(s, signode): signode += nodes.emphasis(p, p) -class CtonInst(CtonObject): - """A Cretonne IR instruction.""" +class ClifInst(ClifObject): + """A Cranelift IR instruction.""" doc_field_types = [ TypedField('argument', label=l_('Arguments'), @@ -175,14 +175,14 @@ class CtonInst(CtonObject): return name -class CtonInstGroup(CtonObject): - """A Cretonne IR instruction group.""" +class ClifInstGroup(ClifObject): + """A Cranelift IR instruction group.""" -class CretonneDomain(Domain): - """Cretonne domain for IR objects.""" - name = 'cton' - label = 'Cretonne' +class CraneliftDomain(Domain): + """Cranelift domain for IR objects.""" + name = 'clif' + label = 'Cranelift' object_types = { 'type': ObjType(l_('type'), 'type'), @@ -190,9 +190,9 @@ class CretonneDomain(Domain): } directives = { - 'type': CtonType, - 'inst': CtonInst, - 'instgroup': CtonInstGroup, + 'type': ClifType, + 'inst': ClifInst, + 'instgroup': ClifInstGroup, } roles = { @@ -230,16 +230,16 @@ class CretonneDomain(Domain): if target not in objects: return [] obj = objects[target] - return [('cton:' + self.role_for_objtype(obj[1]), + return [('clif:' + self.role_for_objtype(obj[1]), make_refnode(builder, fromdocname, obj[0], obj[1] + '-' + target, contnode, target))] class TypeDocumenter(sphinx.ext.autodoc.Documenter): - # Invoke with .. autoctontype:: - objtype = 'ctontype' - # Convert into cton:type directives - domain = 'cton' + # Invoke with .. autocliftype:: + objtype = 'cliftype' + # Convert into clif:type directives + domain = 'clif' directivetype = 'type' @classmethod @@ -262,8 +262,8 @@ class TypeDocumenter(sphinx.ext.autodoc.Documenter): class InstDocumenter(sphinx.ext.autodoc.Documenter): # Invoke with .. autoinst:: objtype = 'inst' - # Convert into cton:inst directives - domain = 'cton' + # Convert into clif:inst directives + domain = 'clif' directivetype = 'inst' @classmethod @@ -297,7 +297,7 @@ class InstDocumenter(sphinx.ext.autodoc.Documenter): def add_directive_header(self, sig): """Add the directive header and options to the generated content.""" - domain = getattr(self, 'domain', 'cton') + domain = getattr(self, 'domain', 'clif') directive = getattr(self, 'directivetype', self.objtype) sourcename = self.get_sourcename() self.add_line(u'.. %s:%s:: %s' % (domain, directive, sig), sourcename) @@ -350,8 +350,8 @@ class InstDocumenter(sphinx.ext.autodoc.Documenter): class InstGroupDocumenter(sphinx.ext.autodoc.ModuleLevelDocumenter): # Invoke with .. autoinstgroup:: objtype = 'instgroup' - # Convert into cton:instgroup directives - domain = 'cton' + # Convert into clif:instgroup directives + domain = 'clif' directivetype = 'instgroup' @classmethod @@ -365,19 +365,19 @@ class InstGroupDocumenter(sphinx.ext.autodoc.ModuleLevelDocumenter): super(InstGroupDocumenter, self).add_content( more_content, no_docstring) sourcename = self.get_sourcename() - indexed = self.env.domaindata['cton']['objects'] + indexed = self.env.domaindata['clif']['objects'] names = [inst.name for inst in self.object.instructions] names.sort() for name in names: if name in indexed: - self.add_line(u':cton:inst:`{}`'.format(name), sourcename) + self.add_line(u':clif:inst:`{}`'.format(name), sourcename) else: self.add_line(u'``{}``'.format(name), sourcename) def setup(app): - app.add_domain(CretonneDomain) + app.add_domain(CraneliftDomain) app.add_autodocumenter(TypeDocumenter) app.add_autodocumenter(InstDocumenter) app.add_autodocumenter(InstGroupDocumenter) diff --git a/cranelift/docs/cton_lexer.py b/cranelift/docs/clif_lexer.py similarity index 92% rename from cranelift/docs/cton_lexer.py rename to cranelift/docs/clif_lexer.py index bd1a47758f..2bcde35348 100644 --- a/cranelift/docs/cton_lexer.py +++ b/cranelift/docs/clif_lexer.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- # -# Pygments lexer for Cretonne. +# Pygments lexer for Cranelift. from __future__ import absolute_import from pygments.lexer import RegexLexer, bygroups, words @@ -12,10 +12,10 @@ def keywords(*args): return words(args, prefix=r'\b', suffix=r'\b') -class CretonneLexer(RegexLexer): - name = 'Cretonne' - aliases = ['cton'] - filenames = ['*.cton'] +class CraneliftLexer(RegexLexer): + name = 'Cranelift' + aliases = ['clif'] + filenames = ['*.clif'] tokens = { 'root': [ @@ -64,6 +64,6 @@ class CretonneLexer(RegexLexer): def setup(app): """Setup Sphinx extension.""" - app.add_lexer('cton', CretonneLexer()) + app.add_lexer('clif', CraneliftLexer()) return {'version': '0.1'} diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst index abc47caaa3..78129e500e 100644 --- a/cranelift/docs/compare-llvm.rst +++ b/cranelift/docs/compare-llvm.rst @@ -1,5 +1,5 @@ ************************* -Cretonne compared to LLVM +Cranelift compared to LLVM ************************* `LLVM `_ is a collection of compiler components implemented as @@ -10,18 +10,18 @@ popular. `Chris Lattner's chapter about LLVM Applications `_ book gives an excellent overview of the architecture and design of LLVM. -Cretonne and LLVM are superficially similar projects, so it is worth +Cranelift and LLVM are superficially similar projects, so it is worth highlighting some of the differences and similarities. Both projects: - Use an ISA-agnostic input language in order to mostly abstract away the differences between target instruction set architectures. - Depend extensively on SSA form. - Have both textual and in-memory forms of their primary intermediate - representation. (LLVM also has a binary bitcode format; Cretonne doesn't.) + representation. (LLVM also has a binary bitcode format; Cranelift doesn't.) - Can target multiple ISAs. - Can cross-compile by default without rebuilding the code generator. -Cretonne's scope is much smaller than that of LLVM. The classical three main +Cranelift's scope is much smaller than that of LLVM. The classical three main parts of a compiler are: 1. The language-dependent front end parses and type-checks the input program. @@ -29,9 +29,9 @@ parts of a compiler are: target ISA. 3. The code generator which depends strongly on the target ISA. -LLVM provides both common optimizations *and* a code generator. Cretonne only +LLVM provides both common optimizations *and* a code generator. Cranelift only provides the last part, the code generator. LLVM additionally provides -infrastructure for building assemblers and disassemblers. Cretonne does not +infrastructure for building assemblers and disassemblers. Cranelift does not handle assembly at all---it only generates binary machine code. Intermediate representations @@ -89,20 +89,20 @@ representation. Some target ISAs have a fast instruction selector that can translate simple code directly to MachineInstrs, bypassing SelectionDAG when possible. -:doc:`Cretonne ` uses a single intermediate representation to cover -these levels of abstraction. This is possible in part because of Cretonne's +:doc:`Cranelift ` uses a single intermediate representation to cover +these levels of abstraction. This is possible in part because of Cranelift's smaller scope. -- Cretonne does not provide assemblers and disassemblers, so it is not +- Cranelift does not provide assemblers and disassemblers, so it is not necessary to be able to represent every weird instruction in an ISA. Only those instructions that the code generator emits have a representation. -- Cretonne's opcodes are ISA-agnostic, but after legalization / instruction +- Cranelift's opcodes are ISA-agnostic, but after legalization / instruction selection, each instruction is annotated with an ISA-specific encoding which represents a native instruction. - SSA form is preserved throughout. After register allocation, each SSA value is annotated with an assigned ISA register or stack slot. -The Cretonne intermediate representation is similar to LLVM IR, but at a slightly +The Cranelift intermediate representation is similar to LLVM IR, but at a slightly lower level of abstraction. Program structure @@ -112,22 +112,22 @@ In LLVM IR, the largest representable unit is the *module* which corresponds more or less to a C translation unit. It is a collection of functions and global variables that may contain references to external symbols too. -In Cretonne IR, the largest representable unit is the *function*. This is so +In Cranelift IR, the largest representable unit is the *function*. This is so that functions can easily be compiled in parallel without worrying about -references to shared data structures. Cretonne does not have any +references to shared data structures. Cranelift does not have any inter-procedural optimizations like inlining. -An LLVM IR function is a graph of *basic blocks*. A Cretonne IR function is a +An LLVM IR function is a graph of *basic blocks*. A Cranelift IR function is a graph of *extended basic blocks* that may contain internal branch instructions. The main difference is that an LLVM conditional branch instruction has two -target basic blocks---a true and a false edge. A Cretonne branch instruction +target basic blocks---a true and a false edge. A Cranelift branch instruction only has a single target and falls through to the next instruction when its -condition is false. The Cretonne representation is closer to how machine code +condition is false. The Cranelift representation is closer to how machine code works; LLVM's representation is more abstract. LLVM uses `phi instructions `_ in its SSA -representation. Cretonne passes arguments to EBBs instead. The two +representation. Cranelift passes arguments to EBBs instead. The two representations are equivalent, but the EBB arguments are better suited to handle EBBs that may contain multiple branches to the same destination block with different arguments. Passing arguments to an EBB looks a lot like passing @@ -137,42 +137,42 @@ similarly. Arguments are assigned to registers or stack locations. Value types ----------- -:ref:`Cretonne's type system ` is mostly a subset of LLVM's type +:ref:`Cranelift's type system ` is mostly a subset of LLVM's type system. It is less abstract and closer to the types that common ISA registers can hold. -- Integer types are limited to powers of two from :cton:type:`i8` to - :cton:type:`i64`. LLVM can represent integer types of arbitrary bit width. -- Floating point types are limited to :cton:type:`f32` and :cton:type:`f64` +- Integer types are limited to powers of two from :clif:type:`i8` to + :clif:type:`i64`. LLVM can represent integer types of arbitrary bit width. +- Floating point types are limited to :clif:type:`f32` and :clif:type:`f64` which is what WebAssembly provides. It is possible that 16-bit and 128-bit types will be added in the future. -- Addresses are represented as integers---There are no Cretonne pointer types. +- Addresses are represented as integers---There are no Cranelift pointer types. LLVM currently has rich pointer types that include the pointee type. It may - move to a simpler 'address' type in the future. Cretonne may add a single + move to a simpler 'address' type in the future. Cranelift may add a single address type too. - SIMD vector types are limited to a power-of-two number of vector lanes up to 256. LLVM allows an arbitrary number of SIMD lanes. -- Cretonne has no aggregate types. LLVM has named and anonymous struct types as +- Cranelift has no aggregate types. LLVM has named and anonymous struct types as well as array types. -Cretonne has multiple boolean types, whereas LLVM simply uses `i1`. The sized -Cretonne boolean types are used to represent SIMD vector masks like ``b32x4`` +Cranelift has multiple boolean types, whereas LLVM simply uses `i1`. The sized +Cranelift boolean types are used to represent SIMD vector masks like ``b32x4`` where each lane is either all 0 or all 1 bits. -Cretonne instructions and function calls can return multiple result values. LLVM +Cranelift instructions and function calls can return multiple result values. LLVM instead models this by returning a single value of an aggregate type. Instruction set --------------- LLVM has a small well-defined basic instruction set and a large number of -intrinsics, some of which are ISA-specific. Cretonne has a larger instruction -set and no intrinsics. Some Cretonne instructions are ISA-specific. +intrinsics, some of which are ISA-specific. Cranelift has a larger instruction +set and no intrinsics. Some Cranelift instructions are ISA-specific. -Since Cretonne instructions are used all the way until the binary machine code +Since Cranelift instructions are used all the way until the binary machine code is emitted, there are opcodes for every native instruction that can be generated. There is a lot of overlap between different ISAs, so for example the -:cton:inst:`iadd_imm` instruction is used by every ISA that can add an +:clif:inst:`iadd_imm` instruction is used by every ISA that can add an immediate integer to a register. A simple RISC ISA like RISC-V can be defined with only shared instructions, while x86 needs a number of specific instructions to model addressing modes. @@ -180,20 +180,20 @@ instructions to model addressing modes. Undefined behavior ================== -Cretonne does not generally exploit undefined behavior in its optimizations. +Cranelift does not generally exploit undefined behavior in its optimizations. LLVM's mid-level optimizations do, but it should be noted that LLVM's low-level code generator rarely needs to make use of undefined behavior either. LLVM provides ``nsw`` and ``nuw`` flags for its arithmetic that invoke -undefined behavior on overflow. Cretonne does not provide this functionality. +undefined behavior on overflow. Cranelift does not provide this functionality. Its arithmetic instructions either produce a value or a trap. LLVM has an ``unreachable`` instruction which is used to indicate impossible -code paths. Cretonne only has an explicit :cton:inst:`trap` instruction. +code paths. Cranelift only has an explicit :clif:inst:`trap` instruction. -Cretonne does make assumptions about aliasing. For example, it assumes that it +Cranelift does make assumptions about aliasing. For example, it assumes that it has full control of the stack objects in a function, and that they can only be modified by function calls if their address have escaped. It is quite likely -that Cretonne will admit more detailed aliasing annotations on load/store +that Cranelift will admit more detailed aliasing annotations on load/store instructions in the future. When these annotations are incorrect, undefined behavior ensues. diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py index d9a6dcb6eb..2220d72141 100644 --- a/cranelift/docs/conf.py +++ b/cranelift/docs/conf.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- # -# cretonne documentation build configuration file, created by +# cranelift documentation build configuration file, created by # sphinx-quickstart on Fri Mar 2 12:49:24 2018. # # This file is execfile()d with the current directory set to its @@ -21,7 +21,7 @@ import os import sys sys.path.insert(0, os.path.abspath('.')) -# Also add the meta directory to sys.path so autodoc can find the Cretonne meta +# Also add the meta directory to sys.path so autodoc can find the Cranelift meta # language definitions. sys.path.insert(0, os.path.abspath('../lib/codegen/meta')) @@ -41,8 +41,8 @@ extensions = [ 'sphinx.ext.ifconfig', 'sphinx.ext.graphviz', 'sphinx.ext.inheritance_diagram', - 'cton_domain', - 'cton_lexer', + 'clif_domain', + 'clif_lexer', ] # Add any paths that contain templates here, relative to this directory. @@ -58,9 +58,9 @@ source_suffix = '.rst' master_doc = 'index' # General information about the project. -project = u'cretonne' -copyright = u'2018, Cretonne Developers' -author = u'Cretonne Developers' +project = u'cranelift' +copyright = u'2018, Cranelift Developers' +author = u'Cranelift Developers' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the @@ -113,7 +113,7 @@ html_theme = 'sphinx_rtd_theme' # -- Options for HTMLHelp output ------------------------------------------ # Output file base name for HTML help builder. -htmlhelp_basename = 'cretonnedoc' +htmlhelp_basename = 'craneliftdoc' # -- Options for LaTeX output --------------------------------------------- @@ -140,7 +140,7 @@ latex_elements = { # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ - (master_doc, 'cretonne.tex', u'cretonne Documentation', + (master_doc, 'cranelift.tex', u'cranelift Documentation', author, 'manual'), ] @@ -150,7 +150,7 @@ latex_documents = [ # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ - (master_doc, 'cretonne', u'cretonne Documentation', + (master_doc, 'cranelift', u'cranelift Documentation', [author], 1) ] @@ -161,8 +161,8 @@ man_pages = [ # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ - (master_doc, 'cretonne', u'cretonne Documentation', - author, 'cretonne', 'One line description of project.', + (master_doc, 'cranelift', u'cranelift Documentation', + author, 'cranelift', 'One line description of project.', 'Miscellaneous'), ] diff --git a/cranelift/docs/example.cton b/cranelift/docs/example.clif similarity index 100% rename from cranelift/docs/example.cton rename to cranelift/docs/example.clif diff --git a/cranelift/docs/heapex-dyn.cton b/cranelift/docs/heapex-dyn.clif similarity index 100% rename from cranelift/docs/heapex-dyn.cton rename to cranelift/docs/heapex-dyn.clif diff --git a/cranelift/docs/heapex-sm32.cton b/cranelift/docs/heapex-sm32.clif similarity index 100% rename from cranelift/docs/heapex-sm32.cton rename to cranelift/docs/heapex-sm32.clif diff --git a/cranelift/docs/heapex-sm64.cton b/cranelift/docs/heapex-sm64.clif similarity index 100% rename from cranelift/docs/heapex-sm64.cton rename to cranelift/docs/heapex-sm64.clif diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst index 60b153e2a1..2a566ea286 100644 --- a/cranelift/docs/index.rst +++ b/cranelift/docs/index.rst @@ -1,4 +1,4 @@ -Cretonne Code Generator +Cranelift Code Generator ======================= Contents: @@ -15,36 +15,36 @@ Contents: Rust Crate Documentation ======================== -`cretonne `_ - This is the core code generator crate. It takes Cretonne IR as input +`cranelift `_ + This is the core code generator crate. It takes Cranelift IR as input and emits encoded machine instructions, along with symbolic relocations, as output. -`cretonne-wasm `_ - This crate translates WebAssembly code into Cretonne IR. +`cranelift-wasm `_ + This crate translates WebAssembly code into Cranelift IR. -`cretonne-frontend `_ - This crate provides utilities for translating code into Cretonne IR. +`cranelift-frontend `_ + This crate provides utilities for translating code into Cranelift IR. -`cretonne-native `_ - This crate performs auto-detection of the host, allowing Cretonne to +`cranelift-native `_ + This crate performs auto-detection of the host, allowing Cranelift to generate code optimized for the machine it's running on. -`cretonne-reader `_ - This crate translates from Cretonne IR's text format into Cretonne IR +`cranelift-reader `_ + This crate translates from Cranelift IR's text format into Cranelift IR in in-memory data structures. -`cretonne-module `_ +`cranelift-module `_ This crate manages compiling multiple functions and data objects together. -`cretonne-faerie `_ - This crate provides a faerie-based backend for `cretonne-module`, which +`cranelift-faerie `_ + This crate provides a faerie-based backend for `cranelift-module`, which emits native object files using the `faerie `_ library. -`cretonne-simplejit `_ - This crate provides a simple JIT backend for `cretonne-module`, which +`cranelift-simplejit `_ + This crate provides a simple JIT backend for `cranelift-module`, which emits code and data into memory. Indices and tables diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst index 776c61d75b..cd585627ec 100644 --- a/cranelift/docs/langref.rst +++ b/cranelift/docs/langref.rst @@ -1,14 +1,14 @@ *************************** -Cretonne Language Reference +Cranelift Language Reference *************************** -.. default-domain:: cton -.. highlight:: cton +.. default-domain:: clif +.. highlight:: clif -The Cretonne intermediate representation (:term:`IR`) has two primary forms: +The Cranelift intermediate representation (:term:`IR`) has two primary forms: an *in-memory data structure* that the code generator library is using, and a *text format* which is used for test cases and debug output. -Files containing Cretonne textual IR have the ``.cton`` filename extension. +Files containing Cranelift textual IR have the ``.clif`` filename extension. This reference uses the text format to describe IR semantics but glosses over the finer details of the lexical and syntactic structure of the format. @@ -17,7 +17,7 @@ the finer details of the lexical and syntactic structure of the format. Overall structure ================= -Cretonne compiles functions independently. A ``.cton`` IR file may contain +Cranelift compiles functions independently. A ``.clif`` IR file may contain multiple functions, and the programmatic API can create multiple function handles at the same time, but the functions don't share any data or reference each other directly. @@ -27,10 +27,10 @@ This is a simple C function that computes the average of an array of floats: .. literalinclude:: example.c :language: c -Here is the same function compiled into Cretonne IR: +Here is the same function compiled into Cranelift IR: -.. literalinclude:: example.cton - :language: cton +.. literalinclude:: example.clif + :language: clif :lines: 2- The first line of a function definition provides the function *name* and @@ -44,7 +44,7 @@ After the preamble follows the :term:`function body` which consists of :term:`entry block`. Every EBB ends with a :term:`terminator instruction`, so execution can never fall through to the next EBB without an explicit branch. -A ``.cton`` file consists of a sequence of independent function definitions: +A ``.clif`` file consists of a sequence of independent function definitions: .. productionlist:: function_list : { function } @@ -59,7 +59,7 @@ The instructions in the function body use and produce *values* in SSA form. This means that every value is defined exactly once, and every use of a value must be dominated by the definition. -Cretonne does not have phi instructions but uses :term:`EBB parameter`\s +Cranelift does not have phi instructions but uses :term:`EBB parameter`\s instead. An EBB can be defined with a list of typed parameters. Whenever control is transferred to the EBB, argument values for the parameters must be provided. When entering a function, the incoming function parameters are passed as @@ -74,11 +74,11 @@ SSA values: In the entry block, ``v4`` is the initial value. In the loop block variable during each iteration. Finally, ``v12`` is computed as the induction variable value for the next iteration. -The `cretonne_frontend` crate contains utilities for translating from programs +The `cranelift_frontend` crate contains utilities for translating from programs containing multiple assignments to the same variables into SSA form for -Cretonne :term:`IR`. +Cranelift :term:`IR`. -Such variables can also be presented to Cretonne as :term:`stack slot`\s. +Such variables can also be presented to Cranelift as :term:`stack slot`\s. Stack slots are accessed with the :inst:`stack_store` and :inst:`stack_load` instructions, and can have their address taken with :inst:`stack_addr`, which supports C-like programming languages where local variables can have their @@ -103,11 +103,11 @@ value. It can only exist as an SSA value, it can't be stored in memory or converted to another type. The larger boolean types can be stored in memory. They are represented as either all zero bits or all one bits. -.. autoctontype:: b1 -.. autoctontype:: b8 -.. autoctontype:: b16 -.. autoctontype:: b32 -.. autoctontype:: b64 +.. autocliftype:: b1 +.. autocliftype:: b8 +.. autocliftype:: b16 +.. autocliftype:: b32 +.. autocliftype:: b64 Integer types ------------- @@ -118,10 +118,10 @@ number, others don't care. The support for i8 and i16 arithmetic is incomplete and use could lead to bugs. -.. autoctontype:: i8 -.. autoctontype:: i16 -.. autoctontype:: i32 -.. autoctontype:: i64 +.. autocliftype:: i8 +.. autocliftype:: i16 +.. autocliftype:: i32 +.. autocliftype:: i64 Floating point types -------------------- @@ -149,8 +149,8 @@ instructions are encoded as follows: and all bits of the trailing significand other than the MSB set to nondeterministic values. -.. autoctontype:: f32 -.. autoctontype:: f64 +.. autocliftype:: f32 +.. autocliftype:: f64 CPU flags types --------------- @@ -168,8 +168,8 @@ live at the same time. After legalization, some instruction encodings will clobber the flags, and flags values are not allowed to be live across such instructions either. The verifier enforces these rules. -.. autoctontype:: iflags -.. autoctontype:: fflags +.. autocliftype:: iflags +.. autocliftype:: fflags SIMD vector types ----------------- @@ -420,7 +420,7 @@ baldrdash SpiderMonkey WebAssembly convention ========== =========================================== The "not-ABI-stable" conventions do not follow an external specification and -may change between versions of Cretonne. +may change between versions of Cranelift. The "fastcall" convention is not yet implemented. @@ -448,8 +448,8 @@ preamble`: This simple example illustrates direct function calls and signatures: -.. literalinclude:: callex.cton - :language: cton +.. literalinclude:: callex.clif + :language: clif :lines: 3- Indirect function calls use a signature declared in the preamble. @@ -462,7 +462,7 @@ Indirect function calls use a signature declared in the preamble. Memory ====== -Cretonne provides fully general :inst:`load` and :inst:`store` instructions for +Cranelift provides fully general :inst:`load` and :inst:`store` instructions for accessing memory, as well as :ref:`extending loads and truncating stores `. @@ -515,7 +515,7 @@ frame. Allocate a stack slot in the preamble. - If no alignment is specified, Cretonne will pick an appropriate alignment + If no alignment is specified, Cranelift will pick an appropriate alignment for the stack slot based on its size and access patterns. :arg Bytes: Stack slot size on bytes. @@ -543,7 +543,7 @@ instructions before instruction selection:: v1 = stack_addr ss3, 16 v0 = load.f64 v1 -When Cretonne code is running in a sandbox, it can also be necessary to include +When Cranelift code is running in a sandbox, it can also be necessary to include stack overflow checks in the prologue. .. inst:: stack_limit = GV @@ -564,14 +564,14 @@ A *global value* is an object whose value is not known at compile time. The value is computed at runtime by :inst:`global_value`, possibly using information provided by the linker via relocations. There are multiple kinds of global values using different methods for determining their value. -Cretonne does not track the type of a global value, for they are just +Cranelift does not track the type of a global value, for they are just values stored in non-stack memory. -When Cretonne is generating code for a virtual machine environment, globals can +When Cranelift is generating code for a virtual machine environment, globals can be used to access data structures in the VM's runtime. This requires functions to have access to a *VM context pointer* which is used as the base address. Typically, the VM context pointer is passed as a hidden function argument to -Cretonne functions. +Cranelift functions. .. inst:: GV = vmctx+Offset @@ -632,7 +632,7 @@ Heaps Code compiled from WebAssembly or asm.js runs in a sandbox where it can't access all process memory. Instead, it is given a small set of memory areas to work -in, and all accesses are bounds checked. Cretonne models this through the +in, and all accesses are bounds checked. Cranelift models this through the concept of *heaps*. A heap is declared in the function preamble and can be accessed with the @@ -727,16 +727,16 @@ guard pages when running WebAssembly code on 64-bit CPUs. The combination of a 4 GB fixed bound and 1-byte bounds checks means that no code needs to be generated for bounds checks at all: -.. literalinclude:: heapex-sm64.cton - :language: cton +.. literalinclude:: heapex-sm64.clif + :language: clif :lines: 2- A static heap can also be used for 32-bit code when the WebAssembly module declares a small upper bound on its memory. A 1 MB static bound with a single 4 KB guard page still has opportunities for sharing bounds checking code: -.. literalinclude:: heapex-sm32.cton - :language: cton +.. literalinclude:: heapex-sm32.clif + :language: clif :lines: 2- If the upper bound on the heap size is too large, a dynamic heap is required @@ -746,8 +746,8 @@ Finally, a runtime environment that simply allocates a heap with :c:func:`malloc()` may not have any guard pages at all. In that case, full bounds checking is required for each access: -.. literalinclude:: heapex-dyn.cton - :language: cton +.. literalinclude:: heapex-dyn.clif + :language: clif :lines: 2- @@ -772,7 +772,7 @@ load a constant into an SSA value. Live range splitting -------------------- -Cretonne's register allocator assigns each SSA value to a register or a spill +Cranelift's register allocator assigns each SSA value to a register or a spill slot on the stack for its entire live range. Since the live range of an SSA value can be quite large, it is sometimes beneficial to split the live range into smaller parts. @@ -1014,7 +1014,7 @@ Most ISAs provide instructions that load an integer value smaller than a registe and extends it to the width of the register. Similarly, store instructions that only write the low bits of an integer register are common. -In addition to the normal :inst:`load` and :inst:`store` instructions, Cretonne +In addition to the normal :inst:`load` and :inst:`store` instructions, Cranelift provides extending loads and truncation stores for 8, 16, and 32-bit memory accesses. @@ -1067,7 +1067,7 @@ Target ISAs may define further instructions in their own instruction groups: Implementation limits ===================== -Cretonne's intermediate representation imposes some limits on the size of +Cranelift's intermediate representation imposes some limits on the size of functions and the number of entities allowed. If these limits are exceeded, the implementation will panic. @@ -1098,7 +1098,7 @@ Number of arguments to a function At most :math:`2^{16}`. This follows from the limit on arguments to the entry EBB. Note that - Cretonne may add a handful of ABI register arguments as function signatures + Cranelift may add a handful of ABI register arguments as function signatures are lowered. This is for representing things like the link register, the incoming frame pointer, and callee-saved registers that are saved in the prologue. @@ -1134,7 +1134,7 @@ Glossary entry block The :term:`EBB` that is executed first in a function. Currently, a - Cretonne function must have exactly one entry block which must be the + Cranelift function must have exactly one entry block which must be the first block in the function. The types of the entry block arguments must match the types of arguments in the function signature. @@ -1152,7 +1152,7 @@ Glossary Note that some textbooks define an EBB as a maximal *subtree* in the control flow graph where only the root can be a join node. This - definition is not equivalent to Cretonne EBBs. + definition is not equivalent to Cranelift EBBs. EBB parameter A formal parameter for an EBB is an SSA value that dominates everything @@ -1195,8 +1195,8 @@ Glossary intermediate representation IR - The language used to describe functions to Cretonne. This reference - describes the syntax and semantics of Cretonne IR. The IR has two + The language used to describe functions to Cranelift. This reference + describes the syntax and semantics of Cranelift IR. The IR has two forms: Textual, and an in-memory data structure. stack slot diff --git a/cranelift/docs/make.bat b/cranelift/docs/make.bat index 2958a2ba6e..1b9759a7b5 100644 --- a/cranelift/docs/make.bat +++ b/cranelift/docs/make.bat @@ -9,7 +9,7 @@ if "%SPHINXBUILD%" == "" ( ) set SOURCEDIR=. set BUILDDIR=_build -set SPHINXPROJ=cretonne +set SPHINXPROJ=cranelift if "%1" == "" goto help diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst index 357f1f1fde..50c15e4a99 100644 --- a/cranelift/docs/metaref.rst +++ b/cranelift/docs/metaref.rst @@ -1,12 +1,12 @@ ******************************** -Cretonne Meta Language Reference +Cranelift Meta Language Reference ******************************** .. default-domain:: py .. highlight:: python .. module:: cdsl -The Cretonne meta language is used to define instructions for Cretonne. It is a +The Cranelift meta language is used to define instructions for Cranelift. It is a domain specific language embedded in Python. This document describes the Python modules that form the embedded DSL. @@ -33,7 +33,7 @@ since the last build. Settings ======== -Settings are used by the environment embedding Cretonne to control the details +Settings are used by the environment embedding Cranelift to control the details of code generation. Each setting is defined in the meta language so a compact and consistent Rust representation can be generated. Shared settings are defined in the :mod:`base.settings` module. Some settings are specific to a target ISA, @@ -80,7 +80,7 @@ open :class:`InstructionGroup`. .. autoclass:: InstructionGroup :members: -The basic Cretonne instruction set described in :doc:`langref` is defined by the +The basic Cranelift instruction set described in :doc:`langref` is defined by the Python module :mod:`base.instructions`. This module has a global value :data:`base.instructions.GROUP` which is an :class:`InstructionGroup` instance containing all the base instructions. @@ -94,7 +94,7 @@ must be instances of the :class:`Operand` class. .. autoclass:: Operand -Cretonne uses two separate type systems for operand kinds and SSA values. +Cranelift uses two separate type systems for operand kinds and SSA values. .. module:: cdsl.typevar @@ -191,7 +191,7 @@ Instruction representation The Rust in-memory representation of instructions is derived from the instruction descriptions. Part of the representation is generated, and part is -written as Rust code in the ``cretonne.instructions`` module. The instruction +written as Rust code in the ``cranelift.instructions`` module. The instruction representation depends on the input operand kinds and whether the instruction can produce multiple results. @@ -259,9 +259,9 @@ Encodings .. currentmodule:: cdsl.isa -Encodings describe how Cretonne instructions are mapped to binary machine code +Encodings describe how Cranelift instructions are mapped to binary machine code for the target architecture. After the legalization pass, all remaining -instructions are expected to map 1-1 to native instruction encodings. Cretonne +instructions are expected to map 1-1 to native instruction encodings. Cranelift instructions that can't be encoded for the current architecture are called :term:`illegal instruction`\s. @@ -270,7 +270,7 @@ incompatible encodings. For example, a modern ARMv8 CPU might support three different CPU modes: *A64* where instructions are encoded in 32 bits, *A32* where all instructions are 32 bits, and *T32* which has a mix of 16-bit and 32-bit instruction encodings. These are incompatible encoding spaces, and while -an :cton:inst:`iadd` instruction can be encoded in 32 bits in each of them, it's +an :clif:inst:`iadd` instruction can be encoded in 32 bits in each of them, it's not the same 32 bits. It's a judgement call if CPU modes should be modelled as separate targets, or as sub-modes of the same target. In the ARMv8 case, the different register banks means that it makes sense to model A64 as a separate @@ -293,7 +293,7 @@ is false, the SSE 4.1 instructions are not available. Encodings also have a :term:`instruction predicate` which depends on the specific values of the instruction's immediate fields. This is used to ensure that immediate address offsets are within range, for example. The instructions -in the base Cretonne instruction set can often represent a wider range of +in the base Cranelift instruction set can often represent a wider range of immediates than any specific encoding. The fixed-size RISC-style encodings tend to have more range limitations than CISC-style variable length encodings like x86. @@ -320,7 +320,7 @@ An :py:class:`Encoding` instance specifies the encoding of a concrete instruction. The following properties are used to select instructions to be encoded: -- An opcode, i.e. :cton:inst:`iadd_imm`, that must match the instruction's +- An opcode, i.e. :clif:inst:`iadd_imm`, that must match the instruction's opcode. - Values for any type variables if the opcode represents a polymorphic instruction. @@ -412,8 +412,8 @@ class which consists of all the XMM registers. Stack operands -------------- -Cretonne's register allocator can assign an SSA value to a stack slot if there -isn't enough registers. It will insert :cton:inst:`spill` and :cton:inst:`fill` +Cranelift's register allocator can assign an SSA value to a stack slot if there +isn't enough registers. It will insert :clif:inst:`spill` and :clif:inst:`fill` instructions as needed to satisfy instruction operand constraints, but it is also possible to have instructions that can access stack slots directly:: @@ -427,7 +427,7 @@ load. Targets ======= -Cretonne can be compiled with support for multiple target instruction set +Cranelift can be compiled with support for multiple target instruction set architectures. Each ISA is represented by a :py:class:`cdsl.isa.TargetISA` instance. .. autoclass:: TargetISA diff --git a/cranelift/docs/regalloc.rst b/cranelift/docs/regalloc.rst index a87753a34f..66ead056f8 100644 --- a/cranelift/docs/regalloc.rst +++ b/cranelift/docs/regalloc.rst @@ -1,11 +1,11 @@ ******************************* -Register Allocation in Cretonne +Register Allocation in Cranelift ******************************* -.. default-domain:: cton -.. highlight:: cton +.. default-domain:: clif +.. highlight:: clif -Cretonne uses a *decoupled, SSA-based* register allocator. Decoupled means that +Cranelift uses a *decoupled, SSA-based* register allocator. Decoupled means that register allocation is split into two primary phases: *spilling* and *coloring*. SSA-based means that the code stays in SSA form throughout the register allocator, and in fact is still in SSA form after register allocation. @@ -162,13 +162,13 @@ linearly with the number of EBBs covered by a live range. This representation is very similar to LLVM's ``LiveInterval`` data structure with a few important differences: -- The Cretonne ``LiveRange`` only covers a single SSA value, while LLVM's +- The Cranelift ``LiveRange`` only covers a single SSA value, while LLVM's ``LiveInterval`` represents the union of multiple related SSA values in a - virtual register. This makes Cretonne's representation smaller because + virtual register. This makes Cranelift's representation smaller because individual segments don't have to annotated with a value number. -- Cretonne stores the def-interval separately from a list of coalesced live-in +- Cranelift stores the def-interval separately from a list of coalesced live-in intervals, while LLVM stores an array of segments. The two representations - are equivalent, but Cretonne optimizes for the common case of a value that is + are equivalent, but Cranelift optimizes for the common case of a value that is only used locally. - It is simpler to check if two live ranges are overlapping. The dominance properties of SSA form means that it is only necessary to check the @@ -182,12 +182,12 @@ with a few important differences: allows 'tombstone' program points corresponding to instructions that have been deleted. - Cretonne uses a 32-bit program point representation that encodes an + Cranelift uses a 32-bit program point representation that encodes an instruction or EBB number directly. There are no 'tombstones' for deleted instructions, and no mirrored linked list of instructions. Live ranges must be updated when instructions are deleted. -A consequence of Cretonne's more compact representation is that two program +A consequence of Cranelift's more compact representation is that two program points can't be compared without the context of a function layout. Coalescing algorithm @@ -273,7 +273,7 @@ extra registers to solve, raising the register pressure: the tied input value doesn't interfere with the output value by inserting a copy if needed. -The spilling heuristic used by Cretonne is very simple. Whenever the spiller +The spilling heuristic used by Cranelift is very simple. Whenever the spiller determines that the register pressure is too high at some instruction, it picks the live SSA value whose definition is farthest away as the spill candidate. Then it spills all values in the corresponding virtual register to the same diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst index 2b02bb6d79..d1107b6f67 100644 --- a/cranelift/docs/testing.rst +++ b/cranelift/docs/testing.rst @@ -1,21 +1,21 @@ **************** -Testing Cretonne +Testing Cranelift **************** -Cretonne is tested at multiple levels of abstraction and integration. When +Cranelift is tested at multiple levels of abstraction and integration. When possible, Rust unit tests are used to verify single functions and types. When testing the interaction between compiler passes, file-level tests are appropriate. The top-level shell script :file:`test-all.sh` runs all of the tests in the -Cretonne repository. +Cranelift repository. Rust tests ========== .. highlight:: rust -Rust and Cargo have good support for testing. Cretonne uses unit tests, doc +Rust and Cargo have good support for testing. Cranelift uses unit tests, doc tests, and integration tests where appropriate. Unit tests @@ -51,7 +51,7 @@ tested:: //! //! # Example //! ``` - //! use cretonne_codegen::settings::{self, Configurable}; + //! use cranelift_codegen::settings::{self, Configurable}; //! //! let mut b = settings::builder(); //! b.set("opt_level", "fastest"); @@ -72,7 +72,7 @@ individually. They are used to exercise the external API of the crates under test. These tests are usually found in the :file:`tests` top-level directory where -they have access to all the crates in the Cretonne repository. The +they have access to all the crates in the Cranelift repository. The :file:`lib/codegen` and :file:`lib/reader` crates have no external dependencies, which can make testing tedious. Integration tests that don't need to depend on other crates can be placed in :file:`lib/codegen/tests` and @@ -81,15 +81,15 @@ to depend on other crates can be placed in :file:`lib/codegen/tests` and File tests ========== -.. highlight:: cton +.. highlight:: clif Compilers work with large data structures representing programs, and it quickly gets unwieldy to generate test data programmatically. File-level tests make it easier to provide substantial input functions for the compiler tests. -File tests are :file:`*.cton` files in the :file:`filetests/` directory +File tests are :file:`*.clif` files in the :file:`filetests/` directory hierarchy. Each file has a header describing what to test followed by a number -of input functions in the :doc:`Cretonne textual intermediate representation +of input functions in the :doc:`Cranelift textual intermediate representation `: .. productionlist:: @@ -111,7 +111,7 @@ header: The options given on the ``isa`` line modify the ISA-specific settings defined in :file:`lib/codegen/meta/isa/*/settings.py`. -All types of tests allow shared Cretonne settings to be modified: +All types of tests allow shared Cranelift settings to be modified: .. productionlist:: settings : { setting } @@ -137,7 +137,7 @@ This example will run the legalizer test twice. Both runs will have run will also have the RISC-V specific flag ``supports_m`` disabled. The filetests are run automatically as part of `cargo test`, and they can -also be run manually with the `cton-util test` command. +also be run manually with the `clif-util test` command. Filecheck --------- @@ -146,7 +146,7 @@ Many of the test commands described below use *filecheck* to verify their output. Filecheck is a Rust implementation of the LLVM tool of the same name. See the `documentation `_ for details of its syntax. -Comments in :file:`.cton` files are associated with the entity they follow. +Comments in :file:`.clif` files are associated with the entity they follow. This typically means an instruction or the whole function. Those tests that use filecheck will extract comments associated with each function (or its entities) and scan them for filecheck directives. The test output for each @@ -160,7 +160,7 @@ Note that LLVM's file tests don't separate filecheck directives by their associated function. It verifies the concatenated output against all filecheck directives in the test file. LLVM's :command:`FileCheck` command has a ``CHECK-LABEL:`` directive to help separate the output from different functions. -Cretonne's tests don't need this. +Cranelift's tests don't need this. `test cat` ---------- @@ -214,7 +214,7 @@ function verifies correctly. ---------------- Print the control flow graph of each function as a Graphviz graph, and run -filecheck over the result. See also the :command:`cton-util print-cfg` +filecheck over the result. See also the :command:`clif-util print-cfg` command:: ; For testing cfg generation. This code is nonsense. @@ -375,4 +375,4 @@ Each function is passed through the full ``Context::compile()`` function which is normally used to compile code. This type of test often depends on assertions or verifier errors, but it is also possible to use filecheck directives which will be matched against the final form of the -Cretonne IR right before binary machine code emission. +Cranelift IR right before binary machine code emission. diff --git a/cranelift/filetests/cfg/loop.cton b/cranelift/filetests/cfg/loop.clif similarity index 100% rename from cranelift/filetests/cfg/loop.cton rename to cranelift/filetests/cfg/loop.clif diff --git a/cranelift/filetests/cfg/traps_early.cton b/cranelift/filetests/cfg/traps_early.clif similarity index 100% rename from cranelift/filetests/cfg/traps_early.cton rename to cranelift/filetests/cfg/traps_early.clif diff --git a/cranelift/filetests/cfg/unused_node.cton b/cranelift/filetests/cfg/unused_node.clif similarity index 100% rename from cranelift/filetests/cfg/unused_node.cton rename to cranelift/filetests/cfg/unused_node.clif diff --git a/cranelift/filetests/dce/basic.cton b/cranelift/filetests/dce/basic.clif similarity index 100% rename from cranelift/filetests/dce/basic.cton rename to cranelift/filetests/dce/basic.clif diff --git a/cranelift/filetests/domtree/basic.cton b/cranelift/filetests/domtree/basic.clif similarity index 100% rename from cranelift/filetests/domtree/basic.cton rename to cranelift/filetests/domtree/basic.clif diff --git a/cranelift/filetests/domtree/loops.cton b/cranelift/filetests/domtree/loops.clif similarity index 100% rename from cranelift/filetests/domtree/loops.cton rename to cranelift/filetests/domtree/loops.clif diff --git a/cranelift/filetests/domtree/loops2.cton b/cranelift/filetests/domtree/loops2.clif similarity index 100% rename from cranelift/filetests/domtree/loops2.cton rename to cranelift/filetests/domtree/loops2.clif diff --git a/cranelift/filetests/domtree/tall-tree.cton b/cranelift/filetests/domtree/tall-tree.clif similarity index 100% rename from cranelift/filetests/domtree/tall-tree.cton rename to cranelift/filetests/domtree/tall-tree.clif diff --git a/cranelift/filetests/domtree/wide-tree.cton b/cranelift/filetests/domtree/wide-tree.clif similarity index 100% rename from cranelift/filetests/domtree/wide-tree.cton rename to cranelift/filetests/domtree/wide-tree.clif diff --git a/cranelift/filetests/isa/riscv/abi-e.cton b/cranelift/filetests/isa/riscv/abi-e.clif similarity index 100% rename from cranelift/filetests/isa/riscv/abi-e.cton rename to cranelift/filetests/isa/riscv/abi-e.clif diff --git a/cranelift/filetests/isa/riscv/abi.cton b/cranelift/filetests/isa/riscv/abi.clif similarity index 100% rename from cranelift/filetests/isa/riscv/abi.cton rename to cranelift/filetests/isa/riscv/abi.clif diff --git a/cranelift/filetests/isa/riscv/binary32.cton b/cranelift/filetests/isa/riscv/binary32.clif similarity index 100% rename from cranelift/filetests/isa/riscv/binary32.cton rename to cranelift/filetests/isa/riscv/binary32.clif diff --git a/cranelift/filetests/isa/riscv/encoding.cton b/cranelift/filetests/isa/riscv/encoding.clif similarity index 100% rename from cranelift/filetests/isa/riscv/encoding.cton rename to cranelift/filetests/isa/riscv/encoding.clif diff --git a/cranelift/filetests/isa/riscv/expand-i32.cton b/cranelift/filetests/isa/riscv/expand-i32.clif similarity index 100% rename from cranelift/filetests/isa/riscv/expand-i32.cton rename to cranelift/filetests/isa/riscv/expand-i32.clif diff --git a/cranelift/filetests/isa/riscv/legalize-abi.cton b/cranelift/filetests/isa/riscv/legalize-abi.clif similarity index 100% rename from cranelift/filetests/isa/riscv/legalize-abi.cton rename to cranelift/filetests/isa/riscv/legalize-abi.clif diff --git a/cranelift/filetests/isa/riscv/legalize-i64.cton b/cranelift/filetests/isa/riscv/legalize-i64.clif similarity index 100% rename from cranelift/filetests/isa/riscv/legalize-i64.cton rename to cranelift/filetests/isa/riscv/legalize-i64.clif diff --git a/cranelift/filetests/isa/riscv/parse-encoding.cton b/cranelift/filetests/isa/riscv/parse-encoding.clif similarity index 100% rename from cranelift/filetests/isa/riscv/parse-encoding.cton rename to cranelift/filetests/isa/riscv/parse-encoding.clif diff --git a/cranelift/filetests/isa/riscv/regmove.cton b/cranelift/filetests/isa/riscv/regmove.clif similarity index 100% rename from cranelift/filetests/isa/riscv/regmove.cton rename to cranelift/filetests/isa/riscv/regmove.clif diff --git a/cranelift/filetests/isa/riscv/split-args.cton b/cranelift/filetests/isa/riscv/split-args.clif similarity index 100% rename from cranelift/filetests/isa/riscv/split-args.cton rename to cranelift/filetests/isa/riscv/split-args.clif diff --git a/cranelift/filetests/isa/riscv/verify-encoding.cton b/cranelift/filetests/isa/riscv/verify-encoding.clif similarity index 100% rename from cranelift/filetests/isa/riscv/verify-encoding.cton rename to cranelift/filetests/isa/riscv/verify-encoding.clif diff --git a/cranelift/filetests/isa/x86/abcd.cton b/cranelift/filetests/isa/x86/abcd.clif similarity index 100% rename from cranelift/filetests/isa/x86/abcd.cton rename to cranelift/filetests/isa/x86/abcd.clif diff --git a/cranelift/filetests/isa/x86/abi-bool.cton b/cranelift/filetests/isa/x86/abi-bool.clif similarity index 100% rename from cranelift/filetests/isa/x86/abi-bool.cton rename to cranelift/filetests/isa/x86/abi-bool.clif diff --git a/cranelift/filetests/isa/x86/abi32.cton b/cranelift/filetests/isa/x86/abi32.clif similarity index 100% rename from cranelift/filetests/isa/x86/abi32.cton rename to cranelift/filetests/isa/x86/abi32.clif diff --git a/cranelift/filetests/isa/x86/abi64.cton b/cranelift/filetests/isa/x86/abi64.clif similarity index 100% rename from cranelift/filetests/isa/x86/abi64.cton rename to cranelift/filetests/isa/x86/abi64.clif diff --git a/cranelift/filetests/isa/x86/allones_funcaddrs32.cton b/cranelift/filetests/isa/x86/allones_funcaddrs32.clif similarity index 84% rename from cranelift/filetests/isa/x86/allones_funcaddrs32.cton rename to cranelift/filetests/isa/x86/allones_funcaddrs32.clif index 2e3dbf50f3..b6d7542fc3 100644 --- a/cranelift/filetests/isa/x86/allones_funcaddrs32.cton +++ b/cranelift/filetests/isa/x86/allones_funcaddrs32.clif @@ -6,10 +6,10 @@ target i686 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/allones_funcaddrs32.cton | llvm-mc -show-encoding -triple=i386 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/allones_funcaddrs32.clif | llvm-mc -show-encoding -triple=i386 ; -; Tests from binary32.cton affected by allones_funcaddrs. +; Tests from binary32.clif affected by allones_funcaddrs. function %I32() { sig0 = () fn0 = %foo() diff --git a/cranelift/filetests/isa/x86/allones_funcaddrs64.cton b/cranelift/filetests/isa/x86/allones_funcaddrs64.clif similarity index 87% rename from cranelift/filetests/isa/x86/allones_funcaddrs64.cton rename to cranelift/filetests/isa/x86/allones_funcaddrs64.clif index 6edbbcf4d2..06ab24ed80 100644 --- a/cranelift/filetests/isa/x86/allones_funcaddrs64.cton +++ b/cranelift/filetests/isa/x86/allones_funcaddrs64.clif @@ -6,10 +6,10 @@ target x86_64 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/allones_funcaddrs64.cton | llvm-mc -show-encoding -triple=x86_64 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/allones_funcaddrs64.clif | llvm-mc -show-encoding -triple=x86_64 ; -; Tests from binary64.cton affected by allones_funcaddrs. +; Tests from binary64.clif affected by allones_funcaddrs. function %I64() { sig0 = () fn0 = %foo() diff --git a/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount.cton b/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount.clif similarity index 100% rename from cranelift/filetests/isa/x86/baseline_clz_ctz_popcount.cton rename to cranelift/filetests/isa/x86/baseline_clz_ctz_popcount.clif diff --git a/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton b/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif similarity index 97% rename from cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton rename to cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif index 16acad9cb6..b2a0c9617f 100644 --- a/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton +++ b/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif @@ -4,7 +4,7 @@ target x86_64 baseline ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/baseline_clz_ctz_popcount_encoding.cton | llvm-mc -show-encoding -triple=x86_64 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif | llvm-mc -show-encoding -triple=x86_64 ; function %Foo() { diff --git a/cranelift/filetests/isa/x86/binary32-float.cton b/cranelift/filetests/isa/x86/binary32-float.clif similarity index 99% rename from cranelift/filetests/isa/x86/binary32-float.cton rename to cranelift/filetests/isa/x86/binary32-float.clif index 9982b44606..53bbcbff8e 100644 --- a/cranelift/filetests/isa/x86/binary32-float.cton +++ b/cranelift/filetests/isa/x86/binary32-float.clif @@ -4,7 +4,7 @@ target i686 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary32-float.cton | llvm-mc -show-encoding -triple=i386 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary32-float.clif | llvm-mc -show-encoding -triple=i386 ; function %F32() { diff --git a/cranelift/filetests/isa/x86/binary32.cton b/cranelift/filetests/isa/x86/binary32.clif similarity index 99% rename from cranelift/filetests/isa/x86/binary32.cton rename to cranelift/filetests/isa/x86/binary32.clif index 6764388aff..3f14b74c80 100644 --- a/cranelift/filetests/isa/x86/binary32.cton +++ b/cranelift/filetests/isa/x86/binary32.clif @@ -5,7 +5,7 @@ target i686 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary32.cton | llvm-mc -show-encoding -triple=i386 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary32.clif | llvm-mc -show-encoding -triple=i386 ; function %I32() { diff --git a/cranelift/filetests/isa/x86/binary64-float.cton b/cranelift/filetests/isa/x86/binary64-float.clif similarity index 99% rename from cranelift/filetests/isa/x86/binary64-float.cton rename to cranelift/filetests/isa/x86/binary64-float.clif index 6d85b52976..12beb1fc78 100644 --- a/cranelift/filetests/isa/x86/binary64-float.cton +++ b/cranelift/filetests/isa/x86/binary64-float.clif @@ -5,7 +5,7 @@ target x86_64 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary64-float.cton | llvm-mc -show-encoding -triple=x86_64 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary64-float.clif | llvm-mc -show-encoding -triple=x86_64 ; function %F32() { diff --git a/cranelift/filetests/isa/x86/binary64-pic.cton b/cranelift/filetests/isa/x86/binary64-pic.clif similarity index 96% rename from cranelift/filetests/isa/x86/binary64-pic.cton rename to cranelift/filetests/isa/x86/binary64-pic.clif index 2fa66a820f..3dec2af62b 100644 --- a/cranelift/filetests/isa/x86/binary64-pic.cton +++ b/cranelift/filetests/isa/x86/binary64-pic.clif @@ -6,7 +6,7 @@ target x86_64 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary64-pic.cton | llvm-mc -show-encoding -triple=x86_64 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary64-pic.clif | llvm-mc -show-encoding -triple=x86_64 ; ; Tests for i64 instructions. diff --git a/cranelift/filetests/isa/x86/binary64.cton b/cranelift/filetests/isa/x86/binary64.clif similarity index 99% rename from cranelift/filetests/isa/x86/binary64.cton rename to cranelift/filetests/isa/x86/binary64.clif index 1e3da24fa2..0aac9bbafc 100644 --- a/cranelift/filetests/isa/x86/binary64.cton +++ b/cranelift/filetests/isa/x86/binary64.clif @@ -5,7 +5,7 @@ target x86_64 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary64.cton | llvm-mc -show-encoding -triple=x86_64 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/binary64.clif | llvm-mc -show-encoding -triple=x86_64 ; ; Tests for i64 instructions. diff --git a/cranelift/filetests/isa/x86/legalize-bnot.cton b/cranelift/filetests/isa/x86/legalize-bnot.clif similarity index 100% rename from cranelift/filetests/isa/x86/legalize-bnot.cton rename to cranelift/filetests/isa/x86/legalize-bnot.clif diff --git a/cranelift/filetests/isa/x86/legalize-br-table.cton b/cranelift/filetests/isa/x86/legalize-br-table.clif similarity index 100% rename from cranelift/filetests/isa/x86/legalize-br-table.cton rename to cranelift/filetests/isa/x86/legalize-br-table.clif diff --git a/cranelift/filetests/isa/x86/legalize-call.cton b/cranelift/filetests/isa/x86/legalize-call.clif similarity index 100% rename from cranelift/filetests/isa/x86/legalize-call.cton rename to cranelift/filetests/isa/x86/legalize-call.clif diff --git a/cranelift/filetests/isa/x86/legalize-custom.cton b/cranelift/filetests/isa/x86/legalize-custom.clif similarity index 100% rename from cranelift/filetests/isa/x86/legalize-custom.cton rename to cranelift/filetests/isa/x86/legalize-custom.clif diff --git a/cranelift/filetests/isa/x86/legalize-div-traps.cton b/cranelift/filetests/isa/x86/legalize-div-traps.clif similarity index 98% rename from cranelift/filetests/isa/x86/legalize-div-traps.cton rename to cranelift/filetests/isa/x86/legalize-div-traps.clif index b849f07e1b..40c09aee0e 100644 --- a/cranelift/filetests/isa/x86/legalize-div-traps.cton +++ b/cranelift/filetests/isa/x86/legalize-div-traps.clif @@ -1,6 +1,6 @@ ; Test the division legalizations. test legalizer -; See also legalize-div.cton. +; See also legalize-div.clif. set avoid_div_traps=1 target x86_64 diff --git a/cranelift/filetests/isa/x86/legalize-div.cton b/cranelift/filetests/isa/x86/legalize-div.clif similarity index 97% rename from cranelift/filetests/isa/x86/legalize-div.cton rename to cranelift/filetests/isa/x86/legalize-div.clif index d3a4d359da..be30accbc6 100644 --- a/cranelift/filetests/isa/x86/legalize-div.cton +++ b/cranelift/filetests/isa/x86/legalize-div.clif @@ -1,6 +1,6 @@ ; Test the division legalizations. test legalizer -; See also legalize-div-traps.cton. +; See also legalize-div-traps.clif. set avoid_div_traps=0 target x86_64 diff --git a/cranelift/filetests/isa/x86/legalize-iconst-i8.cton b/cranelift/filetests/isa/x86/legalize-iconst-i8.clif similarity index 100% rename from cranelift/filetests/isa/x86/legalize-iconst-i8.cton rename to cranelift/filetests/isa/x86/legalize-iconst-i8.clif diff --git a/cranelift/filetests/isa/x86/legalize-imul-i8.cton b/cranelift/filetests/isa/x86/legalize-imul-i8.clif similarity index 100% rename from cranelift/filetests/isa/x86/legalize-imul-i8.cton rename to cranelift/filetests/isa/x86/legalize-imul-i8.clif diff --git a/cranelift/filetests/isa/x86/legalize-imul-imm-i8.cton b/cranelift/filetests/isa/x86/legalize-imul-imm-i8.clif similarity index 100% rename from cranelift/filetests/isa/x86/legalize-imul-imm-i8.cton rename to cranelift/filetests/isa/x86/legalize-imul-imm-i8.clif diff --git a/cranelift/filetests/isa/x86/legalize-libcall.cton b/cranelift/filetests/isa/x86/legalize-libcall.clif similarity index 100% rename from cranelift/filetests/isa/x86/legalize-libcall.cton rename to cranelift/filetests/isa/x86/legalize-libcall.clif diff --git a/cranelift/filetests/isa/x86/legalize-load-store-i8.cton b/cranelift/filetests/isa/x86/legalize-load-store-i8.clif similarity index 100% rename from cranelift/filetests/isa/x86/legalize-load-store-i8.cton rename to cranelift/filetests/isa/x86/legalize-load-store-i8.clif diff --git a/cranelift/filetests/isa/x86/legalize-memory.cton b/cranelift/filetests/isa/x86/legalize-memory.clif similarity index 100% rename from cranelift/filetests/isa/x86/legalize-memory.cton rename to cranelift/filetests/isa/x86/legalize-memory.clif diff --git a/cranelift/filetests/isa/x86/legalize-mulhi.cton b/cranelift/filetests/isa/x86/legalize-mulhi.clif similarity index 100% rename from cranelift/filetests/isa/x86/legalize-mulhi.cton rename to cranelift/filetests/isa/x86/legalize-mulhi.clif diff --git a/cranelift/filetests/isa/x86/legalize-regmove-i8.cton b/cranelift/filetests/isa/x86/legalize-regmove-i8.clif similarity index 100% rename from cranelift/filetests/isa/x86/legalize-regmove-i8.cton rename to cranelift/filetests/isa/x86/legalize-regmove-i8.clif diff --git a/cranelift/filetests/isa/x86/optimized-zero-constants-32bit.cton b/cranelift/filetests/isa/x86/optimized-zero-constants-32bit.clif similarity index 100% rename from cranelift/filetests/isa/x86/optimized-zero-constants-32bit.cton rename to cranelift/filetests/isa/x86/optimized-zero-constants-32bit.clif diff --git a/cranelift/filetests/isa/x86/optimized-zero-constants.cton b/cranelift/filetests/isa/x86/optimized-zero-constants.clif similarity index 100% rename from cranelift/filetests/isa/x86/optimized-zero-constants.cton rename to cranelift/filetests/isa/x86/optimized-zero-constants.clif diff --git a/cranelift/filetests/isa/x86/probestack-adjusts-sp.cton b/cranelift/filetests/isa/x86/probestack-adjusts-sp.clif similarity index 94% rename from cranelift/filetests/isa/x86/probestack-adjusts-sp.cton rename to cranelift/filetests/isa/x86/probestack-adjusts-sp.clif index f0e351d84f..656bd776ec 100644 --- a/cranelift/filetests/isa/x86/probestack-adjusts-sp.cton +++ b/cranelift/filetests/isa/x86/probestack-adjusts-sp.clif @@ -3,7 +3,7 @@ set colocated_libcalls=1 set probestack_func_adjusts_sp=1 target x86_64 -; Like %big in probestack.cton, but with the probestack function adjusting +; Like %big in probestack.clif, but with the probestack function adjusting ; the stack pointer itself. function %big() system_v { diff --git a/cranelift/filetests/isa/x86/probestack-disabled.cton b/cranelift/filetests/isa/x86/probestack-disabled.clif similarity index 93% rename from cranelift/filetests/isa/x86/probestack-disabled.cton rename to cranelift/filetests/isa/x86/probestack-disabled.clif index cbf1ed0b0d..267085b0c5 100644 --- a/cranelift/filetests/isa/x86/probestack-disabled.cton +++ b/cranelift/filetests/isa/x86/probestack-disabled.clif @@ -3,7 +3,7 @@ set colocated_libcalls=1 set probestack_enabled=0 target x86_64 -; Like %big in probestack.cton, but with probes disabled. +; Like %big in probestack.clif, but with probes disabled. function %big() system_v { ss0 = explicit_slot 300000 diff --git a/cranelift/filetests/isa/x86/probestack-noncolocated.cton b/cranelift/filetests/isa/x86/probestack-noncolocated.clif similarity index 94% rename from cranelift/filetests/isa/x86/probestack-noncolocated.cton rename to cranelift/filetests/isa/x86/probestack-noncolocated.clif index 8ec23719cc..c304945231 100644 --- a/cranelift/filetests/isa/x86/probestack-noncolocated.cton +++ b/cranelift/filetests/isa/x86/probestack-noncolocated.clif @@ -1,7 +1,7 @@ test compile target x86_64 -; Like %big in probestack.cton, but without a colocated libcall. +; Like %big in probestack.clif, but without a colocated libcall. function %big() system_v { ss0 = explicit_slot 300000 diff --git a/cranelift/filetests/isa/x86/probestack-size.cton b/cranelift/filetests/isa/x86/probestack-size.clif similarity index 97% rename from cranelift/filetests/isa/x86/probestack-size.cton rename to cranelift/filetests/isa/x86/probestack-size.clif index e413db3b1b..4c147b9e24 100644 --- a/cranelift/filetests/isa/x86/probestack-size.cton +++ b/cranelift/filetests/isa/x86/probestack-size.clif @@ -3,7 +3,7 @@ set colocated_libcalls=1 set probestack_size_log2=13 target x86_64 -; Like %big in probestack.cton, but now the probestack size is bigger +; Like %big in probestack.clif, but now the probestack size is bigger ; and it no longer needs a probe. function %big() system_v { diff --git a/cranelift/filetests/isa/x86/probestack.cton b/cranelift/filetests/isa/x86/probestack.clif similarity index 100% rename from cranelift/filetests/isa/x86/probestack.cton rename to cranelift/filetests/isa/x86/probestack.clif diff --git a/cranelift/filetests/isa/x86/prologue-epilogue.cton b/cranelift/filetests/isa/x86/prologue-epilogue.clif similarity index 100% rename from cranelift/filetests/isa/x86/prologue-epilogue.cton rename to cranelift/filetests/isa/x86/prologue-epilogue.clif diff --git a/cranelift/filetests/isa/x86/shrink-multiple-uses.cton b/cranelift/filetests/isa/x86/shrink-multiple-uses.clif similarity index 100% rename from cranelift/filetests/isa/x86/shrink-multiple-uses.cton rename to cranelift/filetests/isa/x86/shrink-multiple-uses.clif diff --git a/cranelift/filetests/isa/x86/shrink.cton b/cranelift/filetests/isa/x86/shrink.clif similarity index 91% rename from cranelift/filetests/isa/x86/shrink.cton rename to cranelift/filetests/isa/x86/shrink.clif index 7648a6dd22..1d1dbfde31 100644 --- a/cranelift/filetests/isa/x86/shrink.cton +++ b/cranelift/filetests/isa/x86/shrink.clif @@ -6,7 +6,7 @@ target x86_64 ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/shrink.cton | llvm-mc -show-encoding -triple=x86_64 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/shrink.clif | llvm-mc -show-encoding -triple=x86_64 ; function %test_shrinking(i32) -> i32 { diff --git a/cranelift/filetests/isa/x86/stack-addr64.cton b/cranelift/filetests/isa/x86/stack-addr64.clif similarity index 95% rename from cranelift/filetests/isa/x86/stack-addr64.cton rename to cranelift/filetests/isa/x86/stack-addr64.clif index ded05221db..a333f2cd5d 100644 --- a/cranelift/filetests/isa/x86/stack-addr64.cton +++ b/cranelift/filetests/isa/x86/stack-addr64.clif @@ -5,7 +5,7 @@ target x86_64 haswell ; The binary encodings can be verified with the command: ; -; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/stack-addr64.cton | llvm-mc -show-encoding -triple=x86_64 +; sed -ne 's/^ *; asm: *//p' filetests/isa/x86/stack-addr64.clif | llvm-mc -show-encoding -triple=x86_64 ; function %stack_addr() { diff --git a/cranelift/filetests/isa/x86/stack-load-store64.cton b/cranelift/filetests/isa/x86/stack-load-store64.clif similarity index 100% rename from cranelift/filetests/isa/x86/stack-load-store64.cton rename to cranelift/filetests/isa/x86/stack-load-store64.clif diff --git a/cranelift/filetests/isa/x86/windows_fastcall_x64.cton b/cranelift/filetests/isa/x86/windows_fastcall_x64.clif similarity index 100% rename from cranelift/filetests/isa/x86/windows_fastcall_x64.cton rename to cranelift/filetests/isa/x86/windows_fastcall_x64.clif diff --git a/cranelift/filetests/licm/basic.cton b/cranelift/filetests/licm/basic.clif similarity index 100% rename from cranelift/filetests/licm/basic.cton rename to cranelift/filetests/licm/basic.clif diff --git a/cranelift/filetests/licm/complex.cton b/cranelift/filetests/licm/complex.clif similarity index 100% rename from cranelift/filetests/licm/complex.cton rename to cranelift/filetests/licm/complex.clif diff --git a/cranelift/filetests/licm/multiple-blocks.cton b/cranelift/filetests/licm/multiple-blocks.clif similarity index 100% rename from cranelift/filetests/licm/multiple-blocks.cton rename to cranelift/filetests/licm/multiple-blocks.clif diff --git a/cranelift/filetests/licm/nested_loops.cton b/cranelift/filetests/licm/nested_loops.clif similarity index 100% rename from cranelift/filetests/licm/nested_loops.cton rename to cranelift/filetests/licm/nested_loops.clif diff --git a/cranelift/filetests/licm/reject.cton b/cranelift/filetests/licm/reject.clif similarity index 100% rename from cranelift/filetests/licm/reject.cton rename to cranelift/filetests/licm/reject.clif diff --git a/cranelift/filetests/parser/alias.cton b/cranelift/filetests/parser/alias.clif similarity index 100% rename from cranelift/filetests/parser/alias.cton rename to cranelift/filetests/parser/alias.clif diff --git a/cranelift/filetests/parser/branch.cton b/cranelift/filetests/parser/branch.clif similarity index 100% rename from cranelift/filetests/parser/branch.cton rename to cranelift/filetests/parser/branch.clif diff --git a/cranelift/filetests/parser/call.cton b/cranelift/filetests/parser/call.clif similarity index 100% rename from cranelift/filetests/parser/call.cton rename to cranelift/filetests/parser/call.clif diff --git a/cranelift/filetests/parser/flags.cton b/cranelift/filetests/parser/flags.clif similarity index 100% rename from cranelift/filetests/parser/flags.cton rename to cranelift/filetests/parser/flags.clif diff --git a/cranelift/filetests/parser/instruction_encoding.cton b/cranelift/filetests/parser/instruction_encoding.clif similarity index 100% rename from cranelift/filetests/parser/instruction_encoding.cton rename to cranelift/filetests/parser/instruction_encoding.clif diff --git a/cranelift/filetests/parser/keywords.cton b/cranelift/filetests/parser/keywords.clif similarity index 100% rename from cranelift/filetests/parser/keywords.cton rename to cranelift/filetests/parser/keywords.clif diff --git a/cranelift/filetests/parser/memory.cton b/cranelift/filetests/parser/memory.clif similarity index 100% rename from cranelift/filetests/parser/memory.cton rename to cranelift/filetests/parser/memory.clif diff --git a/cranelift/filetests/parser/preamble.cton b/cranelift/filetests/parser/preamble.clif similarity index 100% rename from cranelift/filetests/parser/preamble.cton rename to cranelift/filetests/parser/preamble.clif diff --git a/cranelift/filetests/parser/rewrite.cton b/cranelift/filetests/parser/rewrite.clif similarity index 100% rename from cranelift/filetests/parser/rewrite.cton rename to cranelift/filetests/parser/rewrite.clif diff --git a/cranelift/filetests/parser/ternary.cton b/cranelift/filetests/parser/ternary.clif similarity index 100% rename from cranelift/filetests/parser/ternary.cton rename to cranelift/filetests/parser/ternary.clif diff --git a/cranelift/filetests/parser/tiny.cton b/cranelift/filetests/parser/tiny.clif similarity index 100% rename from cranelift/filetests/parser/tiny.cton rename to cranelift/filetests/parser/tiny.clif diff --git a/cranelift/filetests/postopt/basic.cton b/cranelift/filetests/postopt/basic.clif similarity index 100% rename from cranelift/filetests/postopt/basic.cton rename to cranelift/filetests/postopt/basic.clif diff --git a/cranelift/filetests/postopt/complex_memory_ops.cton b/cranelift/filetests/postopt/complex_memory_ops.clif similarity index 100% rename from cranelift/filetests/postopt/complex_memory_ops.cton rename to cranelift/filetests/postopt/complex_memory_ops.clif diff --git a/cranelift/filetests/preopt/div_by_const_indirect.cton b/cranelift/filetests/preopt/div_by_const_indirect.clif similarity index 100% rename from cranelift/filetests/preopt/div_by_const_indirect.cton rename to cranelift/filetests/preopt/div_by_const_indirect.clif diff --git a/cranelift/filetests/preopt/div_by_const_non_power_of_2.cton b/cranelift/filetests/preopt/div_by_const_non_power_of_2.clif similarity index 100% rename from cranelift/filetests/preopt/div_by_const_non_power_of_2.cton rename to cranelift/filetests/preopt/div_by_const_non_power_of_2.clif diff --git a/cranelift/filetests/preopt/div_by_const_power_of_2.cton b/cranelift/filetests/preopt/div_by_const_power_of_2.clif similarity index 100% rename from cranelift/filetests/preopt/div_by_const_power_of_2.cton rename to cranelift/filetests/preopt/div_by_const_power_of_2.clif diff --git a/cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton b/cranelift/filetests/preopt/rem_by_const_non_power_of_2.clif similarity index 100% rename from cranelift/filetests/preopt/rem_by_const_non_power_of_2.cton rename to cranelift/filetests/preopt/rem_by_const_non_power_of_2.clif diff --git a/cranelift/filetests/preopt/rem_by_const_power_of_2.cton b/cranelift/filetests/preopt/rem_by_const_power_of_2.clif similarity index 100% rename from cranelift/filetests/preopt/rem_by_const_power_of_2.cton rename to cranelift/filetests/preopt/rem_by_const_power_of_2.clif diff --git a/cranelift/filetests/preopt/simplify.cton b/cranelift/filetests/preopt/simplify.clif similarity index 100% rename from cranelift/filetests/preopt/simplify.cton rename to cranelift/filetests/preopt/simplify.clif diff --git a/cranelift/filetests/regalloc/aliases.cton b/cranelift/filetests/regalloc/aliases.clif similarity index 100% rename from cranelift/filetests/regalloc/aliases.cton rename to cranelift/filetests/regalloc/aliases.clif diff --git a/cranelift/filetests/regalloc/basic.cton b/cranelift/filetests/regalloc/basic.clif similarity index 100% rename from cranelift/filetests/regalloc/basic.cton rename to cranelift/filetests/regalloc/basic.clif diff --git a/cranelift/filetests/regalloc/coalesce.cton b/cranelift/filetests/regalloc/coalesce.clif similarity index 100% rename from cranelift/filetests/regalloc/coalesce.cton rename to cranelift/filetests/regalloc/coalesce.clif diff --git a/cranelift/filetests/regalloc/coalescing-207.cton b/cranelift/filetests/regalloc/coalescing-207.clif similarity index 99% rename from cranelift/filetests/regalloc/coalescing-207.cton rename to cranelift/filetests/regalloc/coalescing-207.clif index 2fc3cfc429..7de96e69ec 100644 --- a/cranelift/filetests/regalloc/coalescing-207.cton +++ b/cranelift/filetests/regalloc/coalescing-207.clif @@ -1,7 +1,7 @@ test regalloc target x86_64 haswell -; Reported as https://github.com/cretonne/cretonne/issues/207 +; Reported as https://github.com/cranelift/cranelift/issues/207 ; ; The coalescer creates a virtual register with two interfering values. function %pr207(i64 vmctx, i32, i32) -> i32 system_v { diff --git a/cranelift/filetests/regalloc/coalescing-216.cton b/cranelift/filetests/regalloc/coalescing-216.clif similarity index 93% rename from cranelift/filetests/regalloc/coalescing-216.cton rename to cranelift/filetests/regalloc/coalescing-216.clif index fc4e877f68..212b03dc36 100644 --- a/cranelift/filetests/regalloc/coalescing-216.cton +++ b/cranelift/filetests/regalloc/coalescing-216.clif @@ -1,7 +1,7 @@ test regalloc target x86_64 haswell -; Reported as https://github.com/cretonne/cretonne/issues/216 from the Binaryen fuzzer. +; Reported as https://github.com/cranelift/cranelift/issues/216 from the Binaryen fuzzer. ; ; The (old) coalescer creates a virtual register with two identical values. function %pr216(i32 [%rdi], i64 vmctx [%rsi]) -> i64 [%rax] system_v { diff --git a/cranelift/filetests/regalloc/coloring-227.cton b/cranelift/filetests/regalloc/coloring-227.clif similarity index 100% rename from cranelift/filetests/regalloc/coloring-227.cton rename to cranelift/filetests/regalloc/coloring-227.clif diff --git a/cranelift/filetests/regalloc/constraints.cton b/cranelift/filetests/regalloc/constraints.clif similarity index 100% rename from cranelift/filetests/regalloc/constraints.cton rename to cranelift/filetests/regalloc/constraints.clif diff --git a/cranelift/filetests/regalloc/ghost-param.cton b/cranelift/filetests/regalloc/ghost-param.clif similarity index 100% rename from cranelift/filetests/regalloc/ghost-param.cton rename to cranelift/filetests/regalloc/ghost-param.clif diff --git a/cranelift/filetests/regalloc/global-constraints.cton b/cranelift/filetests/regalloc/global-constraints.clif similarity index 100% rename from cranelift/filetests/regalloc/global-constraints.cton rename to cranelift/filetests/regalloc/global-constraints.clif diff --git a/cranelift/filetests/regalloc/global-fixed.cton b/cranelift/filetests/regalloc/global-fixed.clif similarity index 100% rename from cranelift/filetests/regalloc/global-fixed.cton rename to cranelift/filetests/regalloc/global-fixed.clif diff --git a/cranelift/filetests/regalloc/infinite-interference.cton b/cranelift/filetests/regalloc/infinite-interference.clif similarity index 100% rename from cranelift/filetests/regalloc/infinite-interference.cton rename to cranelift/filetests/regalloc/infinite-interference.clif diff --git a/cranelift/filetests/regalloc/iterate.cton b/cranelift/filetests/regalloc/iterate.clif similarity index 100% rename from cranelift/filetests/regalloc/iterate.cton rename to cranelift/filetests/regalloc/iterate.clif diff --git a/cranelift/filetests/regalloc/multi-constraints.cton b/cranelift/filetests/regalloc/multi-constraints.clif similarity index 100% rename from cranelift/filetests/regalloc/multi-constraints.cton rename to cranelift/filetests/regalloc/multi-constraints.clif diff --git a/cranelift/filetests/regalloc/output-interference.cton b/cranelift/filetests/regalloc/output-interference.clif similarity index 100% rename from cranelift/filetests/regalloc/output-interference.cton rename to cranelift/filetests/regalloc/output-interference.clif diff --git a/cranelift/filetests/regalloc/reload-208.cton b/cranelift/filetests/regalloc/reload-208.clif similarity index 96% rename from cranelift/filetests/regalloc/reload-208.cton rename to cranelift/filetests/regalloc/reload-208.clif index 6b4dc74292..5da6bfa9ef 100644 --- a/cranelift/filetests/regalloc/reload-208.cton +++ b/cranelift/filetests/regalloc/reload-208.clif @@ -3,7 +3,7 @@ target x86_64 haswell ; regex: V=v\d+ -; Filed as https://github.com/cretonne/cretonne/issues/208 +; Filed as https://github.com/cranelift/cranelift/issues/208 ; ; The verifier complains about a branch argument that is not in the same virtual register as the ; corresponding EBB argument. diff --git a/cranelift/filetests/regalloc/reload.cton b/cranelift/filetests/regalloc/reload.clif similarity index 100% rename from cranelift/filetests/regalloc/reload.cton rename to cranelift/filetests/regalloc/reload.clif diff --git a/cranelift/filetests/regalloc/schedule-moves.cton b/cranelift/filetests/regalloc/schedule-moves.clif similarity index 100% rename from cranelift/filetests/regalloc/schedule-moves.cton rename to cranelift/filetests/regalloc/schedule-moves.clif diff --git a/cranelift/filetests/regalloc/spill-noregs.cton b/cranelift/filetests/regalloc/spill-noregs.clif similarity index 100% rename from cranelift/filetests/regalloc/spill-noregs.cton rename to cranelift/filetests/regalloc/spill-noregs.clif diff --git a/cranelift/filetests/regalloc/spill.cton b/cranelift/filetests/regalloc/spill.clif similarity index 100% rename from cranelift/filetests/regalloc/spill.cton rename to cranelift/filetests/regalloc/spill.clif diff --git a/cranelift/filetests/regalloc/unreachable_code.cton b/cranelift/filetests/regalloc/unreachable_code.clif similarity index 100% rename from cranelift/filetests/regalloc/unreachable_code.cton rename to cranelift/filetests/regalloc/unreachable_code.clif diff --git a/cranelift/filetests/regalloc/x86-regres.cton b/cranelift/filetests/regalloc/x86-regres.clif similarity index 100% rename from cranelift/filetests/regalloc/x86-regres.cton rename to cranelift/filetests/regalloc/x86-regres.clif diff --git a/cranelift/filetests/simple_gvn/basic.cton b/cranelift/filetests/simple_gvn/basic.clif similarity index 100% rename from cranelift/filetests/simple_gvn/basic.cton rename to cranelift/filetests/simple_gvn/basic.clif diff --git a/cranelift/filetests/simple_gvn/reject.cton b/cranelift/filetests/simple_gvn/reject.clif similarity index 100% rename from cranelift/filetests/simple_gvn/reject.cton rename to cranelift/filetests/simple_gvn/reject.clif diff --git a/cranelift/filetests/simple_gvn/scopes.cton b/cranelift/filetests/simple_gvn/scopes.clif similarity index 100% rename from cranelift/filetests/simple_gvn/scopes.cton rename to cranelift/filetests/simple_gvn/scopes.clif diff --git a/cranelift/filetests/verifier/bad_layout.cton b/cranelift/filetests/verifier/bad_layout.clif similarity index 100% rename from cranelift/filetests/verifier/bad_layout.cton rename to cranelift/filetests/verifier/bad_layout.clif diff --git a/cranelift/filetests/verifier/defs_dominates_uses.cton b/cranelift/filetests/verifier/defs_dominates_uses.clif similarity index 100% rename from cranelift/filetests/verifier/defs_dominates_uses.cton rename to cranelift/filetests/verifier/defs_dominates_uses.clif diff --git a/cranelift/filetests/verifier/flags.cton b/cranelift/filetests/verifier/flags.clif similarity index 100% rename from cranelift/filetests/verifier/flags.cton rename to cranelift/filetests/verifier/flags.clif diff --git a/cranelift/filetests/verifier/memory.cton b/cranelift/filetests/verifier/memory.clif similarity index 100% rename from cranelift/filetests/verifier/memory.cton rename to cranelift/filetests/verifier/memory.clif diff --git a/cranelift/filetests/verifier/return_at_end.cton b/cranelift/filetests/verifier/return_at_end.clif similarity index 100% rename from cranelift/filetests/verifier/return_at_end.cton rename to cranelift/filetests/verifier/return_at_end.clif diff --git a/cranelift/filetests/verifier/type_check.cton b/cranelift/filetests/verifier/type_check.clif similarity index 100% rename from cranelift/filetests/verifier/type_check.cton rename to cranelift/filetests/verifier/type_check.clif diff --git a/cranelift/filetests/verifier/undeclared_vmctx.cton b/cranelift/filetests/verifier/undeclared_vmctx.clif similarity index 100% rename from cranelift/filetests/verifier/undeclared_vmctx.cton rename to cranelift/filetests/verifier/undeclared_vmctx.clif diff --git a/cranelift/filetests/verifier/unreachable_code.cton b/cranelift/filetests/verifier/unreachable_code.clif similarity index 100% rename from cranelift/filetests/verifier/unreachable_code.cton rename to cranelift/filetests/verifier/unreachable_code.clif diff --git a/cranelift/filetests/wasm/control.cton b/cranelift/filetests/wasm/control.clif similarity index 100% rename from cranelift/filetests/wasm/control.cton rename to cranelift/filetests/wasm/control.clif diff --git a/cranelift/filetests/wasm/conversions.cton b/cranelift/filetests/wasm/conversions.clif similarity index 100% rename from cranelift/filetests/wasm/conversions.cton rename to cranelift/filetests/wasm/conversions.clif diff --git a/cranelift/filetests/wasm/f32-arith.cton b/cranelift/filetests/wasm/f32-arith.clif similarity index 100% rename from cranelift/filetests/wasm/f32-arith.cton rename to cranelift/filetests/wasm/f32-arith.clif diff --git a/cranelift/filetests/wasm/f32-compares.cton b/cranelift/filetests/wasm/f32-compares.clif similarity index 100% rename from cranelift/filetests/wasm/f32-compares.cton rename to cranelift/filetests/wasm/f32-compares.clif diff --git a/cranelift/filetests/wasm/f32-memory64.cton b/cranelift/filetests/wasm/f32-memory64.clif similarity index 100% rename from cranelift/filetests/wasm/f32-memory64.cton rename to cranelift/filetests/wasm/f32-memory64.clif diff --git a/cranelift/filetests/wasm/f64-arith.cton b/cranelift/filetests/wasm/f64-arith.clif similarity index 100% rename from cranelift/filetests/wasm/f64-arith.cton rename to cranelift/filetests/wasm/f64-arith.clif diff --git a/cranelift/filetests/wasm/f64-compares.cton b/cranelift/filetests/wasm/f64-compares.clif similarity index 100% rename from cranelift/filetests/wasm/f64-compares.cton rename to cranelift/filetests/wasm/f64-compares.clif diff --git a/cranelift/filetests/wasm/f64-memory64.cton b/cranelift/filetests/wasm/f64-memory64.clif similarity index 100% rename from cranelift/filetests/wasm/f64-memory64.cton rename to cranelift/filetests/wasm/f64-memory64.clif diff --git a/cranelift/filetests/wasm/i32-arith.cton b/cranelift/filetests/wasm/i32-arith.clif similarity index 100% rename from cranelift/filetests/wasm/i32-arith.cton rename to cranelift/filetests/wasm/i32-arith.clif diff --git a/cranelift/filetests/wasm/i32-compares.cton b/cranelift/filetests/wasm/i32-compares.clif similarity index 100% rename from cranelift/filetests/wasm/i32-compares.cton rename to cranelift/filetests/wasm/i32-compares.clif diff --git a/cranelift/filetests/wasm/i32-memory64.cton b/cranelift/filetests/wasm/i32-memory64.clif similarity index 100% rename from cranelift/filetests/wasm/i32-memory64.cton rename to cranelift/filetests/wasm/i32-memory64.clif diff --git a/cranelift/filetests/wasm/i64-arith.cton b/cranelift/filetests/wasm/i64-arith.clif similarity index 100% rename from cranelift/filetests/wasm/i64-arith.cton rename to cranelift/filetests/wasm/i64-arith.clif diff --git a/cranelift/filetests/wasm/i64-compares.cton b/cranelift/filetests/wasm/i64-compares.clif similarity index 100% rename from cranelift/filetests/wasm/i64-compares.cton rename to cranelift/filetests/wasm/i64-compares.clif diff --git a/cranelift/filetests/wasm/i64-memory64.cton b/cranelift/filetests/wasm/i64-memory64.clif similarity index 100% rename from cranelift/filetests/wasm/i64-memory64.cton rename to cranelift/filetests/wasm/i64-memory64.clif diff --git a/cranelift/filetests/wasm/select.cton b/cranelift/filetests/wasm/select.clif similarity index 100% rename from cranelift/filetests/wasm/select.cton rename to cranelift/filetests/wasm/select.clif diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml index dcda236717..e51dcb8672 100644 --- a/cranelift/fuzz/Cargo.toml +++ b/cranelift/fuzz/Cargo.toml @@ -1,5 +1,5 @@ [package] -name = "cton-wasm-fuzz" +name = "clif-wasm-fuzz" version = "0.0.1" authors = ["foote@fastly.com"] publish = false @@ -11,7 +11,7 @@ cargo-fuzz = true cargo-fuzz = "*" binaryen = { git = "https://github.com/pepyakin/binaryen-rs.git" } libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" } -cretonne-wasm = { path = "../lib/wasm" } +cranelift-wasm = { path = "../lib/wasm" } target-lexicon = "0.0.2" # Prevent this from interfering with workspaces diff --git a/cranelift/fuzz/fuzz_translate_module.rs b/cranelift/fuzz/fuzz_translate_module.rs index d398483961..df339be545 100644 --- a/cranelift/fuzz/fuzz_translate_module.rs +++ b/cranelift/fuzz/fuzz_translate_module.rs @@ -2,10 +2,10 @@ #[macro_use] extern crate libfuzzer_sys; extern crate binaryen; -extern crate cretonne_wasm; +extern crate cranelift_wasm; #[macro_use] extern crate target_lexicon; -use cretonne_wasm::{translate_module, DummyEnvironment}; +use cranelift_wasm::{translate_module, DummyEnvironment}; use std::str::FromStr; fuzz_target!(|data: &[u8]| { diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh index 8f5f26fc09..9dd37e093b 100755 --- a/cranelift/publish-all.sh +++ b/cranelift/publish-all.sh @@ -3,18 +3,18 @@ set -euo pipefail topdir=$(dirname "$0") cd "$topdir" -# All the cretonne-* crates have the same version number +# All the cranelift-* crates have the same version number version="0.13.0" # Update all of the Cargo.toml files. # -# The main Cargo.toml in the top-level directory is the cretonne-tools crate which we don't publish. +# The main Cargo.toml in the top-level directory is the cranelift-tools crate which we don't publish. echo "Updating crate versions to $version" for crate in . lib/*; do # Update the version number of this crate to $version. sed -i.bk -e "s/^version = .*/version = \"$version\"/" "$crate/Cargo.toml" - # Update the required version number of any cretonne* dependencies. - sed -i.bk -e "/^cretonne/s/version = \"[^\"]*\"/version = \"$version\"/" "$crate/Cargo.toml" + # Update the required version number of any cranelift* dependencies. + sed -i.bk -e "/^cranelift/s/version = \"[^\"]*\"/version = \"$version\"/" "$crate/Cargo.toml" done # Update our local Cargo.lock (not checked in). @@ -31,4 +31,4 @@ for crate in entity codegen frontend native reader wasm module simplejit faerie echo cargo publish --manifest-path "lib/$crate/Cargo.toml" done echo -echo Then, go to https://github.com/cretonne/cretonne/releases/ and define a new release. +echo Then, go to https://github.com/cranelift/cranelift/releases/ and define a new release. diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs index b668e0b51f..cb824a6eb7 100644 --- a/cranelift/src/cat.rs +++ b/cranelift/src/cat.rs @@ -1,9 +1,9 @@ //! The `cat` sub-command. //! -//! Read a sequence of Cretonne IR files and print them again to stdout. This has the effect of +//! Read a sequence of Cranelift IR files and print them again to stdout. This has the effect of //! normalizing formatting and removing comments. -use cretonne_reader::parse_functions; +use cranelift_reader::parse_functions; use utils::read_to_string; use CommandResult; diff --git a/cranelift/src/cton-util.rs b/cranelift/src/clif-util.rs similarity index 74% rename from cranelift/src/cton-util.rs rename to cranelift/src/clif-util.rs index ecb461ea99..8df6a603d7 100644 --- a/cranelift/src/cton-util.rs +++ b/cranelift/src/clif-util.rs @@ -10,9 +10,9 @@ #[macro_use] extern crate cfg_if; -extern crate cretonne_codegen; -extern crate cretonne_filetests; -extern crate cretonne_reader; +extern crate cranelift_codegen; +extern crate cranelift_filetests; +extern crate cranelift_reader; extern crate docopt; extern crate filecheck; #[macro_use] @@ -23,14 +23,14 @@ extern crate term; cfg_if! { if #[cfg(feature = "wasm")] { - extern crate cretonne_wasm; + extern crate cranelift_wasm; extern crate wabt; mod wasm; } } extern crate target_lexicon; -use cretonne_codegen::{timing, VERSION}; +use cranelift_codegen::{timing, VERSION}; use docopt::Docopt; use std::io::{self, Write}; use std::process; @@ -42,33 +42,33 @@ mod rsfilecheck; mod utils; const USAGE: &str = " -Cretonne code generator utility +Cranelift code generator utility Usage: - cton-util test [-vT] ... - cton-util cat ... - cton-util filecheck [-v] - cton-util print-cfg ... - cton-util compile [-vpT] [--set ]... [--target ] ... - cton-util wasm [-ctvpTs] [--set ]... [--target ] ... - cton-util --help | --version + clif-util test [-vT] ... + clif-util cat ... + clif-util filecheck [-v] + clif-util print-cfg ... + clif-util compile [-vpT] [--set ]... [--target ] ... + clif-util wasm [-ctvpTs] [--set ]... [--target ] ... + clif-util --help | --version Options: -v, --verbose be more verbose -T, --time-passes print pass timing report -t, --just-decode - just decode WebAssembly to Cretonne IR + just decode WebAssembly to Cranelift IR -s, --print-size prints generated code size -c, --check-translation - just checks the correctness of Cretonne IR translated from WebAssembly - -p, --print print the resulting Cretonne IR + just checks the correctness of Cranelift IR translated from WebAssembly + -p, --print print the resulting Cranelift IR -h, --help print this help message - --set= configure Cretonne settings + --set= configure Cranelift settings --target= - specify the Cretonne target - --version print the Cretonne version + specify the Cranelift target + --version print the Cranelift version "; @@ -95,19 +95,19 @@ struct Args { pub type CommandResult = Result<(), String>; /// Parse the command line arguments and run the requested command. -fn cton_util() -> CommandResult { +fn clif_util() -> CommandResult { // Parse command line arguments. let args: Args = Docopt::new(USAGE) .and_then(|d| { d.help(true) - .version(Some(format!("Cretonne {}", VERSION))) + .version(Some(format!("Cranelift {}", VERSION))) .deserialize() }) .unwrap_or_else(|e| e.exit()); // Find the sub-command to execute. let result = if args.cmd_test { - cretonne_filetests::run(args.flag_verbose, &args.arg_file).map(|_time| ()) + cranelift_filetests::run(args.flag_verbose, &args.arg_file).map(|_time| ()) } else if args.cmd_cat { cat::run(&args.arg_file) } else if args.cmd_filecheck { @@ -135,7 +135,7 @@ fn cton_util() -> CommandResult { ); #[cfg(not(feature = "wasm"))] - let result = Err("Error: cton-util was compiled without wasm support.".to_owned()); + let result = Err("Error: clif-util was compiled without wasm support.".to_owned()); result } else { @@ -151,7 +151,7 @@ fn cton_util() -> CommandResult { } fn main() { - if let Err(mut msg) = cton_util() { + if let Err(mut msg) = clif_util() { if !msg.ends_with('\n') { msg.push('\n'); } diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs index 98ad672f34..5a21d5789d 100644 --- a/cranelift/src/compile.rs +++ b/cranelift/src/compile.rs @@ -1,11 +1,11 @@ -//! CLI tool to read Cretonne IR files and compile them into native code. +//! CLI tool to read Cranelift IR files and compile them into native code. -use cretonne_codegen::isa::TargetIsa; -use cretonne_codegen::print_errors::pretty_error; -use cretonne_codegen::settings::FlagsOrIsa; -use cretonne_codegen::Context; -use cretonne_codegen::{binemit, ir}; -use cretonne_reader::parse_test; +use cranelift_codegen::isa::TargetIsa; +use cranelift_codegen::print_errors::pretty_error; +use cranelift_codegen::settings::FlagsOrIsa; +use cranelift_codegen::Context; +use cranelift_codegen::{binemit, ir}; +use cranelift_reader::parse_test; use std::path::Path; use std::path::PathBuf; use utils::{parse_sets_and_triple, read_to_string}; diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs index bf945da958..6d16c77262 100644 --- a/cranelift/src/print_cfg.rs +++ b/cranelift/src/print_cfg.rs @@ -1,10 +1,10 @@ //! The `print-cfg` sub-command. //! -//! Read a series of Cretonne IR files and print their control flow graphs +//! Read a series of Cranelift IR files and print their control flow graphs //! in graphviz format. -use cretonne_codegen::cfg_printer::CFGPrinter; -use cretonne_reader::parse_functions; +use cranelift_codegen::cfg_printer::CFGPrinter; +use cranelift_reader::parse_functions; use utils::read_to_string; use CommandResult; diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs index 8940778f86..871483c891 100644 --- a/cranelift/src/utils.rs +++ b/cranelift/src/utils.rs @@ -1,9 +1,9 @@ //! Utility functions. -use cretonne_codegen::isa; -use cretonne_codegen::isa::TargetIsa; -use cretonne_codegen::settings::{self, FlagsOrIsa}; -use cretonne_reader::{parse_options, Location}; +use cranelift_codegen::isa; +use cranelift_codegen::isa::TargetIsa; +use cranelift_codegen::settings::{self, FlagsOrIsa}; +use cranelift_reader::{parse_options, Location}; use std::fs::File; use std::io::{self, Read}; use std::path::Path; diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs index 101d31be3a..1e900537ff 100644 --- a/cranelift/src/wasm.rs +++ b/cranelift/src/wasm.rs @@ -1,13 +1,13 @@ -//! CLI tool to use the functions provided by the [cretonne-wasm](../cretonne_wasm/index.html) +//! CLI tool to use the functions provided by the [cranelift-wasm](../cranelift_wasm/index.html) //! crate. //! -//! Reads Wasm binary files, translates the functions' code to Cretonne IR. +//! Reads Wasm binary files, translates the functions' code to Cranelift IR. #![cfg_attr(feature = "cargo-clippy", allow(too_many_arguments, cyclomatic_complexity))] -use cretonne_codegen::Context; -use cretonne_codegen::print_errors::{pretty_error, pretty_verifier_error}; -use cretonne_codegen::settings::FlagsOrIsa; -use cretonne_wasm::{translate_module, DummyEnvironment, ModuleEnvironment}; +use cranelift_codegen::Context; +use cranelift_codegen::print_errors::{pretty_error, pretty_verifier_error}; +use cranelift_codegen::settings::FlagsOrIsa; +use cranelift_wasm::{translate_module, DummyEnvironment, ModuleEnvironment}; use std::error::Error; use std::path::Path; use std::path::PathBuf; diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh index 4bb7330f09..9b13390783 100755 --- a/cranelift/test-all.sh +++ b/cranelift/test-all.sh @@ -64,7 +64,7 @@ banner "Rust unit tests" cargo test --all # Make sure the documentation builds. -banner "Rust documentation: $topdir/target/doc/cretonne/index.html" +banner "Rust documentation: $topdir/target/doc/cranelift/index.html" cargo doc # Run clippy if we have it. diff --git a/cranelift/tests/filetests.rs b/cranelift/tests/filetests.rs index d99f168b61..5eb4ca3638 100644 --- a/cranelift/tests/filetests.rs +++ b/cranelift/tests/filetests.rs @@ -1,7 +1,7 @@ -extern crate cretonne_filetests; +extern crate cranelift_filetests; #[test] fn filetests() { // Run all the filetests in the following directories. - cretonne_filetests::run(false, &["filetests".into(), "docs".into()]).expect("test harness"); + cranelift_filetests::run(false, &["filetests".into(), "docs".into()]).expect("test harness"); } diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml index e5a50d258b..b7f601ab50 100644 --- a/lib/codegen/Cargo.toml +++ b/lib/codegen/Cargo.toml @@ -1,22 +1,22 @@ [package] -authors = ["The Cretonne Project Developers"] -name = "cretonne-codegen" +authors = ["The Cranelift Project Developers"] +name = "cranelift-codegen" version = "0.13.0" description = "Low-level code generator library" license = "Apache-2.0 WITH LLVM-exception" -documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/cretonne/cretonne" +documentation = "https://cranelift.readthedocs.io/" +repository = "https://github.com/cranelift/cranelift" readme = "README.md" keywords = ["compile", "compiler", "jit"] build = "build.rs" [dependencies] -cretonne-entity = { path = "../entity", version = "0.13.0", default-features = false } +cranelift-entity = { path = "../entity", version = "0.13.0", default-features = false } failure = { version = "0.1.1", default-features = false, features = ["derive"] } failure_derive = { version = "0.1.1", default-features = false } hashmap_core = { version = "0.1.8", optional = true } target-lexicon = { version = "0.0.2", default-features = false } -# It is a goal of the cretonne-codegen crate to have minimal external dependencies. +# It is a goal of the cranelift-codegen crate to have minimal external dependencies. # Please don't add any unless they are essential to the task of creating binary # machine code. Integration tests that need external dependencies can be # accomodated in `tests`. @@ -26,9 +26,9 @@ target-lexicon = { version = "0.0.2", default-features = false } # of some minimal std-like replacement libraries. At least one of these two # features need to be enabled. default = ["std"] -std = ["cretonne-entity/std", "target-lexicon/std"] +std = ["cranelift-entity/std", "target-lexicon/std"] core = ["hashmap_core"] [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "cretonne/cretonne" } +travis-ci = { repository = "cranelift/cranelift" } diff --git a/lib/codegen/README.md b/lib/codegen/README.md index c12dfbf5a9..18b9756aad 100644 --- a/lib/codegen/README.md +++ b/lib/codegen/README.md @@ -1,2 +1,2 @@ -This crate contains the core Cretonne code generator. It translates code from an +This crate contains the core Cranelift code generator. It translates code from an intermediate representation into executable machine code. diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs index c52c70873e..b5a2f5b435 100644 --- a/lib/codegen/build.rs +++ b/lib/codegen/build.rs @@ -11,7 +11,7 @@ // TARGET // Target triple provided by Cargo. // -// CRETONNE_TARGETS (Optional) +// CRANELIFT_TARGETS (Optional) // A setting for conditional compilation of isa targets. Possible values can be "native" or // known isa targets separated by ','. // @@ -24,12 +24,12 @@ use std::process; fn main() { let out_dir = env::var("OUT_DIR").expect("The OUT_DIR environment variable must be set"); let target_triple = env::var("TARGET").expect("The TARGET environment variable must be set"); - let cretonne_targets = env::var("CRETONNE_TARGETS").ok(); - let cretonne_targets = cretonne_targets.as_ref().map(|s| s.as_ref()); + let cranelift_targets = env::var("CRANELIFT_TARGETS").ok(); + let cranelift_targets = cranelift_targets.as_ref().map(|s| s.as_ref()); let python = identify_python(); // Configure isa targets cfg. - match isa_targets(cretonne_targets, &target_triple) { + match isa_targets(cranelift_targets, &target_triple) { Ok(isa_targets) => { for isa in &isa_targets { println!("cargo:rustc-cfg=build_{}", isa.name()); @@ -84,7 +84,7 @@ fn identify_python() -> &'static str { return python; } } - panic!("The Cretonne build requires Python (version 2.7 or version 3)"); + panic!("The Cranelift build requires Python (version 2.7 or version 3)"); } /// Represents known ISA target. @@ -142,8 +142,8 @@ impl Isa { } /// Returns isa targets to configure conditional compilation. -fn isa_targets(cretonne_targets: Option<&str>, target_triple: &str) -> Result, String> { - match cretonne_targets { +fn isa_targets(cranelift_targets: Option<&str>, target_triple: &str) -> Result, String> { + match cranelift_targets { Some("native") => Isa::from_arch(target_triple.split('-').next().unwrap()) .map(|isa| vec![isa]) .ok_or_else(|| { diff --git a/lib/codegen/meta/base/__init__.py b/lib/codegen/meta/base/__init__.py index 132b469ee6..79f6ccbf46 100644 --- a/lib/codegen/meta/base/__init__.py +++ b/lib/codegen/meta/base/__init__.py @@ -1 +1 @@ -"""Definitions for the base Cretonne language.""" +"""Definitions for the base Cranelift language.""" diff --git a/lib/codegen/meta/base/entities.py b/lib/codegen/meta/base/entities.py index a91d894e04..d0a0bc470f 100644 --- a/lib/codegen/meta/base/entities.py +++ b/lib/codegen/meta/base/entities.py @@ -1,6 +1,6 @@ """ -The `cretonne.entities` module predefines all the Cretonne entity reference -operand types. There are corresponding definitions in the `cretonne.entities` +The `cranelift.entities` module predefines all the Cranelift entity reference +operand types. There are corresponding definitions in the `cranelift.entities` Rust module. """ from __future__ import absolute_import diff --git a/lib/codegen/meta/base/formats.py b/lib/codegen/meta/base/formats.py index 90adc83176..2642db6085 100644 --- a/lib/codegen/meta/base/formats.py +++ b/lib/codegen/meta/base/formats.py @@ -1,8 +1,8 @@ """ -The cretonne.formats defines all instruction formats. +The cranelift.formats defines all instruction formats. Every instruction format has a corresponding `InstructionData` variant in the -Rust representation of Cretonne IR, so all instruction formats must be defined +Rust representation of Cranelift IR, so all instruction formats must be defined in this module. """ from __future__ import absolute_import diff --git a/lib/codegen/meta/base/immediates.py b/lib/codegen/meta/base/immediates.py index 53293acc77..e8a6fccc78 100644 --- a/lib/codegen/meta/base/immediates.py +++ b/lib/codegen/meta/base/immediates.py @@ -1,6 +1,6 @@ """ -The `cretonne.immediates` module predefines all the Cretonne immediate operand -types. +The `cranelift.immediates` module predefines all the Cranelift immediate +operand types. """ from __future__ import absolute_import from cdsl.operands import ImmediateKind @@ -8,7 +8,7 @@ from cdsl.operands import ImmediateKind #: A 64-bit immediate integer operand. #: #: This type of immediate integer can interact with SSA values with any -#: :py:class:`cretonne.IntType` type. +#: :py:class:`cranelift.IntType` type. imm64 = ImmediateKind('imm64', 'A 64-bit immediate integer.') #: An unsigned 8-bit immediate integer operand. @@ -42,13 +42,13 @@ ieee64 = ImmediateKind('ieee64', 'A 64-bit immediate floating point number.') #: An immediate boolean operand. #: #: This type of immediate boolean can interact with SSA values with any -#: :py:class:`cretonne.BoolType` type. +#: :py:class:`cranelift.BoolType` type. boolean = ImmediateKind('bool', 'An immediate boolean.', rust_type='bool') #: A condition code for comparing integer values. #: -#: This enumerated operand kind is used for the :cton:inst:`icmp` instruction +#: This enumerated operand kind is used for the :clif:inst:`icmp` instruction #: and corresponds to the `condcodes::IntCC` Rust type. intcc = ImmediateKind( 'intcc', @@ -70,7 +70,7 @@ intcc = ImmediateKind( #: A condition code for comparing floating point values. #: -#: This enumerated operand kind is used for the :cton:inst:`fcmp` instruction +#: This enumerated operand kind is used for the :clif:inst:`fcmp` instruction #: and corresponds to the `condcodes::FloatCC` Rust type. floatcc = ImmediateKind( 'floatcc', @@ -94,7 +94,7 @@ floatcc = ImmediateKind( 'uge': 'UnorderedOrGreaterThanOrEqual', }) -#: Flags for memory operations like :cton:inst:`load` and :cton:inst:`store`. +#: Flags for memory operations like :clif:inst:`load` and :clif:inst:`store`. memflags = ImmediateKind( 'memflags', 'Memory operation flags', diff --git a/lib/codegen/meta/base/instructions.py b/lib/codegen/meta/base/instructions.py index 7a63914dbc..147c93631b 100644 --- a/lib/codegen/meta/base/instructions.py +++ b/lib/codegen/meta/base/instructions.py @@ -1,7 +1,7 @@ """ -Cretonne base instruction set. +Cranelift base instruction set. -This module defines the basic Cretonne instruction set that all targets +This module defines the basic Cranelift instruction set that all targets support. """ from __future__ import absolute_import @@ -1524,8 +1524,8 @@ fdiv = Instruction( 'fdiv', r""" Floating point division. - Unlike the integer division instructions :cton:inst:`sdiv` and - :cton:inst:`udiv`, this can't trap. Division by zero is infinity or + Unlike the integer division instructions :clif:inst:`sdiv` and + :clif:inst:`udiv`, this can't trap. Division by zero is infinity or NaN, depending on the dividend. """, ins=(x, y), outs=a) @@ -1800,7 +1800,7 @@ fpromote = Instruction( Each lane in `x` is converted to the destination floating point format. This is an exact operation. - Cretonne currently only supports two floating point formats + Cranelift currently only supports two floating point formats - :type:`f32` and :type:`f64`. This may change in the future. The result type must have the same number of vector lanes as the input, @@ -1816,7 +1816,7 @@ fdemote = Instruction( Each lane in `x` is converted to the destination floating point format by rounding to nearest, ties to even. - Cretonne currently only supports two floating point formats + Cranelift currently only supports two floating point formats - :type:`f32` and :type:`f64`. This may change in the future. The result type must have the same number of vector lanes as the input, diff --git a/lib/codegen/meta/base/legalize.py b/lib/codegen/meta/base/legalize.py index 1bcb492922..ca4c5b9a6e 100644 --- a/lib/codegen/meta/base/legalize.py +++ b/lib/codegen/meta/base/legalize.py @@ -1,7 +1,7 @@ """ Patterns for legalizing the `base` instruction set. -The base Cretonne instruction set is 'fat', and many instructions don't have +The base Cranelift instruction set is 'fat', and many instructions don't have legal representations in a given target ISA. This module defines legalization patterns that describe how base instructions can be transformed to other base instructions that are legal. diff --git a/lib/codegen/meta/base/predicates.py b/lib/codegen/meta/base/predicates.py index 16ea856138..2a521f7ccb 100644 --- a/lib/codegen/meta/base/predicates.py +++ b/lib/codegen/meta/base/predicates.py @@ -1,5 +1,5 @@ """ -Cretonne predicates that consider `Function` fields. +Cranelift predicates that consider `Function` fields. """ from cdsl.predicates import FieldPredicate from .formats import UnaryGlobalValue, InstructionFormat diff --git a/lib/codegen/meta/base/settings.py b/lib/codegen/meta/base/settings.py index 0abeab59e4..7e116b1668 100644 --- a/lib/codegen/meta/base/settings.py +++ b/lib/codegen/meta/base/settings.py @@ -1,5 +1,5 @@ """ -Cretonne shared settings. +Cranelift shared settings. This module defines settings relevant for all code generators. """ @@ -20,10 +20,10 @@ opt_level = EnumSetting( enable_verifier = BoolSetting( """ - Run the Cretonne IR verifier at strategic times during compilation. + Run the Cranelift IR verifier at strategic times during compilation. This makes compilation slower but catches many bugs. The verifier is - disabled by default, except when reading Cretonne IR from a text file. + disabled by default, except when reading Cranelift IR from a text file. """, default=True) @@ -51,8 +51,8 @@ call_conv = EnumSetting( 'probestack' ) -# Note that Cretonne doesn't currently need an is_pie flag, because PIE is just -# PIC where symbols can't be pre-empted, which can be expressed with the +# Note that Cranelift doesn't currently need an is_pie flag, because PIE is +# just PIC where symbols can't be pre-empted, which can be expressed with the # `colocated` flag on external functions and global values. is_pic = BoolSetting("Enable Position-Independent Code generation") @@ -126,7 +126,7 @@ baldrdash_prologue_words = NumSetting( in the epilogue. This setting configures the number of pointer-sized words pushed on the - stack when the Cretonne-generated code is entered. This includes the + stack when the Cranelift-generated code is entered. This includes the pushed return address on x86. """) diff --git a/lib/codegen/meta/base/types.py b/lib/codegen/meta/base/types.py index 648220a41b..bcef9c1d51 100644 --- a/lib/codegen/meta/base/types.py +++ b/lib/codegen/meta/base/types.py @@ -1,5 +1,5 @@ """ -The base.types module predefines all the Cretonne scalar types. +The base.types module predefines all the Cranelift scalar types. """ from __future__ import absolute_import from cdsl.types import IntType, FloatType, BoolType, FlagsType diff --git a/lib/codegen/meta/build.py b/lib/codegen/meta/build.py index d7ad4cdba2..9d3d43f95d 100644 --- a/lib/codegen/meta/build.py +++ b/lib/codegen/meta/build.py @@ -18,7 +18,7 @@ import gen_binemit def main(): # type: () -> None parser = argparse.ArgumentParser( - description='Generate sources for Cretonne.') + description='Generate sources for Cranelift.') parser.add_argument('--out-dir', help='set output directory') args = parser.parse_args() diff --git a/lib/codegen/meta/cdsl/__init__.py b/lib/codegen/meta/cdsl/__init__.py index 44445268ea..43314a6f18 100644 --- a/lib/codegen/meta/cdsl/__init__.py +++ b/lib/codegen/meta/cdsl/__init__.py @@ -1,7 +1,7 @@ """ -Cretonne DSL classes. +Cranelift DSL classes. -This module defines the classes that are used to define Cretonne instructions +This module defines the classes that are used to define Cranelift instructions and other entitties. """ from __future__ import absolute_import diff --git a/lib/codegen/meta/cdsl/ast.py b/lib/codegen/meta/cdsl/ast.py index 5eefe222f8..09d4ad2514 100644 --- a/lib/codegen/meta/cdsl/ast.py +++ b/lib/codegen/meta/cdsl/ast.py @@ -2,7 +2,7 @@ Abstract syntax trees. This module defines classes that can be used to create abstract syntax trees -for patern matching an rewriting of cretonne instructions. +for patern matching an rewriting of cranelift instructions. """ from __future__ import absolute_import from . import instructions diff --git a/lib/codegen/meta/cdsl/formats.py b/lib/codegen/meta/cdsl/formats.py index a1e84e4737..3eee94d248 100644 --- a/lib/codegen/meta/cdsl/formats.py +++ b/lib/codegen/meta/cdsl/formats.py @@ -50,7 +50,7 @@ class InstructionFormat(object): of values. All instruction formats must be predefined in the - :py:mod:`cretonne.formats` module. + :py:mod:`cranelift.formats` module. :param kinds: List of `OperandKind` objects describing the operands. :param name: Instruction format name in CamelCase. This is used as a Rust diff --git a/lib/codegen/meta/cdsl/operands.py b/lib/codegen/meta/cdsl/operands.py index b6f81e4c26..cf99645df9 100644 --- a/lib/codegen/meta/cdsl/operands.py +++ b/lib/codegen/meta/cdsl/operands.py @@ -67,7 +67,7 @@ VARIABLE_ARGS = OperandKind( # Instances of immediate operand types are provided in the -# `cretonne.immediates` module. +# `cranelift.immediates` module. class ImmediateKind(OperandKind): """ The kind of an immediate instruction operand. @@ -152,7 +152,7 @@ class ImmediateKind(OperandKind): # Instances of entity reference operand types are provided in the -# `cretonne.entities` module. +# `cranelift.entities` module. class EntityRefKind(OperandKind): """ The kind of an entity reference instruction operand. diff --git a/lib/codegen/meta/cdsl/predicates.py b/lib/codegen/meta/cdsl/predicates.py index db1223e09e..ee7b81f3da 100644 --- a/lib/codegen/meta/cdsl/predicates.py +++ b/lib/codegen/meta/cdsl/predicates.py @@ -1,5 +1,5 @@ """ -Cretonne predicates. +Cranelift predicates. A *predicate* is a function that computes a boolean result. The inputs to the function determine the kind of predicate: diff --git a/lib/codegen/meta/cdsl/settings.py b/lib/codegen/meta/cdsl/settings.py index 3a9f803c05..cbcac2d98d 100644 --- a/lib/codegen/meta/cdsl/settings.py +++ b/lib/codegen/meta/cdsl/settings.py @@ -14,7 +14,7 @@ except ImportError: class Setting(object): """ - A named setting variable that can be configured externally to Cretonne. + A named setting variable that can be configured externally to Cranelift. Settings are normally not named when they are created. They get their name from the `extract_names` method. diff --git a/lib/codegen/meta/cdsl/types.py b/lib/codegen/meta/cdsl/types.py index c0e989e710..13bc5deab1 100644 --- a/lib/codegen/meta/cdsl/types.py +++ b/lib/codegen/meta/cdsl/types.py @@ -1,4 +1,4 @@ -"""Cretonne ValueType hierarchy""" +"""Cranelift ValueType hierarchy""" from __future__ import absolute_import import math diff --git a/lib/codegen/meta/cdsl/typevar.py b/lib/codegen/meta/cdsl/typevar.py index d58381529f..d1d7c7137d 100644 --- a/lib/codegen/meta/cdsl/typevar.py +++ b/lib/codegen/meta/cdsl/typevar.py @@ -1,7 +1,7 @@ """ Type variables for Parametric polymorphism. -Cretonne instructions and instruction transformations can be specified to be +Cranelift instructions and instruction transformations can be specified to be polymorphic by using type variables. """ from __future__ import absolute_import diff --git a/lib/codegen/meta/gen_legalizer.py b/lib/codegen/meta/gen_legalizer.py index 2312071173..c0b664d214 100644 --- a/lib/codegen/meta/gen_legalizer.py +++ b/lib/codegen/meta/gen_legalizer.py @@ -1,7 +1,7 @@ """ Generate legalizer transformations. -The transformations defined in the `cretonne.legalize` module are all of the +The transformations defined in the `cranelift.legalize` module are all of the macro-expansion form where the input pattern is a single instruction. We generate a Rust function for each `XFormGroup` which takes a `Cursor` pointing at the instruction to be legalized. The expanded destination pattern replaces diff --git a/lib/codegen/meta/isa/__init__.py b/lib/codegen/meta/isa/__init__.py index 1add6e033f..d3cff62c91 100644 --- a/lib/codegen/meta/isa/__init__.py +++ b/lib/codegen/meta/isa/__init__.py @@ -1,9 +1,9 @@ """ -Cretonne target ISA definitions -------------------------------- +Cranelift target ISA definitions +-------------------------------- The :py:mod:`isa` package contains sub-packages for each target instruction set -architecture supported by Cretonne. +architecture supported by Cranelift. """ from __future__ import absolute_import from cdsl.isa import TargetISA # noqa @@ -19,6 +19,6 @@ def all_isas(): # type: () -> List[TargetISA] """ Get a list of all the supported target ISAs. Each target ISA is represented - as a :py:class:`cretonne.TargetISA` instance. + as a :py:class:`cranelift.TargetISA` instance. """ return [riscv.ISA, x86.ISA, arm32.ISA, arm64.ISA] diff --git a/lib/codegen/meta/isa/riscv/encodings.py b/lib/codegen/meta/isa/riscv/encodings.py index 374aef17d1..801f5c6932 100644 --- a/lib/codegen/meta/isa/riscv/encodings.py +++ b/lib/codegen/meta/isa/riscv/encodings.py @@ -61,7 +61,7 @@ RV32.enc(base.iconst.i32, Iz, OPIMM(0b000)) RV64.enc(base.iconst.i32, Iz, OPIMM(0b000)) RV64.enc(base.iconst.i64, Iz, OPIMM(0b000)) -# Dynamic shifts have the same masking semantics as the cton base instructions. +# Dynamic shifts have the same masking semantics as the clif base instructions. for inst, inst_imm, f3, f7 in [ (base.ishl, base.ishl_imm, 0b001, 0b0000000), (base.ushr, base.ushr_imm, 0b101, 0b0000000), diff --git a/lib/codegen/meta/isa/x86/__init__.py b/lib/codegen/meta/isa/x86/__init__.py index 93e71b316c..9691ce6470 100644 --- a/lib/codegen/meta/isa/x86/__init__.py +++ b/lib/codegen/meta/isa/x86/__init__.py @@ -1,12 +1,12 @@ """ x86 Target Architecture -------------------------- +----------------------- This target ISA generates code for x86 CPUs with two separate CPU modes: `I32` 32-bit x86 architecture, also known as 'IA-32', also sometimes referred - to as 'i386', however note that Cretonne depends on instructions not + to as 'i386', however note that Cranelift depends on instructions not in the original `i386`, such as SSE2, CMOVcc, and UD2. `I64` diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta/isa/x86/recipes.py index b65eac5113..c2baa515e7 100644 --- a/lib/codegen/meta/isa/x86/recipes.py +++ b/lib/codegen/meta/isa/x86/recipes.py @@ -34,7 +34,7 @@ except ImportError: # Opcode representation. # -# Cretonne requires each recipe to have a single encoding size in bytes, and +# Cranelift requires each recipe to have a single encoding size in bytes, and # x86 opcodes are variable length, so we use separate recipes for different # styles of opcodes and prefixes. The opcode format is indicated by the recipe # name prefix: @@ -1600,7 +1600,7 @@ rcmp_sp = TailRecipe( # # 1. Guarantee that the test and branch get scheduled next to each other so # macro fusion is guaranteed to be possible. -# 2. Hide the status flags from Cretonne which doesn't currently model flags. +# 2. Hide the status flags from Cranelift which doesn't currently model flags. # # The encoding bits affect both the test and the branch instruction: # diff --git a/lib/codegen/meta/semantics/__init__.py b/lib/codegen/meta/semantics/__init__.py index 8a55b1d595..260ecae331 100644 --- a/lib/codegen/meta/semantics/__init__.py +++ b/lib/codegen/meta/semantics/__init__.py @@ -1,4 +1,4 @@ -"""Definitions for the semantics segment of the Cretonne language.""" +"""Definitions for the semantics segment of the Cranelift language.""" from cdsl.ti import TypeEnv, ti_rtl, get_type_env from cdsl.operands import ImmediateKind from cdsl.ast import Var diff --git a/lib/codegen/meta/semantics/elaborate.py b/lib/codegen/meta/semantics/elaborate.py index 8d2ecd7fd6..1525b1deef 100644 --- a/lib/codegen/meta/semantics/elaborate.py +++ b/lib/codegen/meta/semantics/elaborate.py @@ -1,7 +1,7 @@ """ Tools to elaborate a given Rtl with concrete types into its semantically equivalent primitive version. Its elaborated primitive version contains only -primitive cretonne instructions, which map well to SMTLIB functions. +primitive cranelift instructions, which map well to SMTLIB functions. """ from .primitives import GROUP as PRIMITIVES, prim_to_bv, prim_from_bv from cdsl.xform import Rtl diff --git a/lib/codegen/meta/semantics/primitives.py b/lib/codegen/meta/semantics/primitives.py index 656db41538..a3e498f4e6 100644 --- a/lib/codegen/meta/semantics/primitives.py +++ b/lib/codegen/meta/semantics/primitives.py @@ -1,5 +1,5 @@ """ -Cretonne primitive instruction set. +Cranelift primitive instruction set. This module defines a primitive instruction set, in terms of which the base set is described. Most instructions in this set correspond 1-1 with an SMTLIB @@ -26,9 +26,9 @@ y = Operand('x', BV, doc="A semantic value Y (same width as X)") a = Operand('a', BV, doc="A semantic value A (same width as X)") cond = Operand('b', TypeVar.singleton(b1), doc='A b1 value') -real = Operand('real', Real, doc="A real cretonne value") +real = Operand('real', Real, doc="A real cranelift value") fromReal = Operand('fromReal', Real.to_bitvec(), - doc="A real cretonne value converted to a BV") + doc="A real cranelift value converted to a BV") # # BV Conversion/Materialization diff --git a/lib/codegen/meta/semantics/smtlib.py b/lib/codegen/meta/semantics/smtlib.py index 3a2c819153..40b0694d3b 100644 --- a/lib/codegen/meta/semantics/smtlib.py +++ b/lib/codegen/meta/semantics/smtlib.py @@ -186,14 +186,16 @@ def equivalent(r1, r2, inp_m, out_m): (q1, m1) = to_smt(r1) (q2, m2) = to_smt(r2) - # Build an expression for the equality of real Cretone inputs of r1 and r2 + # Build an expression for the equality of real Cranelift inputs of + # r1 and r2 args_eq_exp = [] # type: List[ExprRef] for (v1, v2) in inp_m.items(): assert isinstance(v2, Var) args_eq_exp.append(mk_eq(m1[v1], m2[v2])) - # Build an expression for the equality of real Cretone outputs of r1 and r2 + # Build an expression for the equality of real Cranelift outputs of + # r1 and r2 results_eq_exp = [] # type: List[ExprRef] for (v1, v2) in out_m.items(): assert isinstance(v2, Var) diff --git a/lib/codegen/src/binemit/memorysink.rs b/lib/codegen/src/binemit/memorysink.rs index fa4aacc233..d3b98b63cb 100644 --- a/lib/codegen/src/binemit/memorysink.rs +++ b/lib/codegen/src/binemit/memorysink.rs @@ -1,6 +1,6 @@ //! Code sink that writes binary machine code into contiguous memory. //! -//! The `CodeSink` trait is the most general way of extracting binary machine code from Cretonne, +//! The `CodeSink` trait is the most general way of extracting binary machine code from Cranelift, //! and it is implemented by things like the `test binemit` file test driver to generate //! hexadecimal machine code. The `CodeSink` has some undesirable performance properties because of //! the dual abstraction: `TargetIsa` is a trait object implemented by each supported ISA, so it @@ -20,7 +20,7 @@ use std::ptr::write_unaligned; /// A `CodeSink` that writes binary machine code directly into memory. /// -/// A `MemoryCodeSink` object should be used when emitting a Cretonne IR function into executable +/// A `MemoryCodeSink` object should be used when emitting a Cranelift IR function into executable /// memory. It writes machine code directly to a raw pointer without any bounds checking, so make /// sure to allocate enough memory for the whole function. The number of bytes required is returned /// by the `Context::compile()` function. diff --git a/lib/codegen/src/binemit/mod.rs b/lib/codegen/src/binemit/mod.rs index 9e4036b2ed..fd8cff031a 100644 --- a/lib/codegen/src/binemit/mod.rs +++ b/lib/codegen/src/binemit/mod.rs @@ -1,6 +1,6 @@ //! Binary machine code emission. //! -//! The `binemit` module contains code for translating Cretonne's intermediate representation into +//! The `binemit` module contains code for translating Cranelift's intermediate representation into //! binary machine code. mod memorysink; @@ -17,7 +17,7 @@ use std::fmt; /// Offset in bytes from the beginning of the function. /// -/// Cretonne can be used as a cross compiler, so we don't want to use a type like `usize` which +/// Cranelift can be used as a cross compiler, so we don't want to use a type like `usize` which /// depends on the *host* platform, not the *target* platform. pub type CodeOffset = u32; @@ -49,7 +49,7 @@ pub enum Reloc { impl fmt::Display for Reloc { /// Display trait implementation drops the arch, since its used in contexts where the arch is - /// already unambigious, e.g. cton syntax with isa specified. In other contexts, use Debug. + /// already unambigious, e.g. clif syntax with isa specified. In other contexts, use Debug. fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match *self { Reloc::Abs4 => write!(f, "Abs4"), diff --git a/lib/codegen/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs index a0d1ad3591..0466e8a615 100644 --- a/lib/codegen/src/binemit/relaxation.rs +++ b/lib/codegen/src/binemit/relaxation.rs @@ -15,13 +15,13 @@ //! On RISC architectures, it can happen that conditional branches have a shorter range than //! unconditional branches: //! -//! ```cton +//! ```clif //! brz v1, ebb17 //! ``` //! //! can be transformed into: //! -//! ```cton +//! ```clif //! brnz v1, ebb23 //! jump ebb17 //! ebb23: diff --git a/lib/codegen/src/binemit/shrink.rs b/lib/codegen/src/binemit/shrink.rs index 3c7d3390ef..d18ee7fdb1 100644 --- a/lib/codegen/src/binemit/shrink.rs +++ b/lib/codegen/src/binemit/shrink.rs @@ -1,6 +1,6 @@ //! Instruction shrinking. //! -//! Sometimes there are multiple valid encodings for a given instruction. Cretonne often initially +//! Sometimes there are multiple valid encodings for a given instruction. Cranelift often initially //! chooses the largest one, because this typically provides the register allocator the most //! flexibility. However, once register allocation is done, this is no longer important, and we //! can switch to smaller encodings when possible. diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs index 955fe6111d..5573d48e1f 100644 --- a/lib/codegen/src/context.rs +++ b/lib/codegen/src/context.rs @@ -1,4 +1,4 @@ -//! Cretonne compilation context and main entry point. +//! Cranelift compilation context and main entry point. //! //! When compiling many small functions, it is important to avoid repeatedly allocating and //! deallocating the data structures needed for compilation. The `Context` struct is used to hold diff --git a/lib/codegen/src/cursor.rs b/lib/codegen/src/cursor.rs index e905545675..9fb559a79b 100644 --- a/lib/codegen/src/cursor.rs +++ b/lib/codegen/src/cursor.rs @@ -46,8 +46,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne_codegen::ir::{Function, Ebb, SourceLoc}; - /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; + /// # use cranelift_codegen::ir::{Function, Ebb, SourceLoc}; + /// # use cranelift_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, srcloc: SourceLoc) { /// let mut pos = FuncCursor::new(func).with_srcloc(srcloc); /// @@ -76,8 +76,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; - /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; + /// # use cranelift_codegen::ir::{Function, Ebb, Inst}; + /// # use cranelift_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, inst: Inst) { /// let mut pos = FuncCursor::new(func).at_inst(inst); /// @@ -99,8 +99,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; - /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; + /// # use cranelift_codegen::ir::{Function, Ebb, Inst}; + /// # use cranelift_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { /// let mut pos = FuncCursor::new(func).at_first_insertion_point(ebb); /// @@ -120,8 +120,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; - /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; + /// # use cranelift_codegen::ir::{Function, Ebb, Inst}; + /// # use cranelift_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { /// let mut pos = FuncCursor::new(func).at_first_inst(ebb); /// @@ -141,8 +141,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; - /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; + /// # use cranelift_codegen::ir::{Function, Ebb, Inst}; + /// # use cranelift_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { /// let mut pos = FuncCursor::new(func).at_last_inst(ebb); /// @@ -162,8 +162,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; - /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; + /// # use cranelift_codegen::ir::{Function, Ebb, Inst}; + /// # use cranelift_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, inst: Inst) { /// let mut pos = FuncCursor::new(func).after_inst(inst); /// @@ -183,8 +183,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; - /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; + /// # use cranelift_codegen::ir::{Function, Ebb, Inst}; + /// # use cranelift_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { /// let mut pos = FuncCursor::new(func).at_top(ebb); /// @@ -204,8 +204,8 @@ pub trait Cursor { /// This is intended to be used as a builder method: /// /// ``` - /// # use cretonne_codegen::ir::{Function, Ebb, Inst}; - /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; + /// # use cranelift_codegen::ir::{Function, Ebb, Inst}; + /// # use cranelift_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function, ebb: Ebb) { /// let mut pos = FuncCursor::new(func).at_bottom(ebb); /// @@ -311,8 +311,8 @@ pub trait Cursor { /// The `next_ebb()` method is intended for iterating over the EBBs in layout order: /// /// ``` - /// # use cretonne_codegen::ir::{Function, Ebb}; - /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; + /// # use cranelift_codegen::ir::{Function, Ebb}; + /// # use cranelift_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function) { /// let mut cursor = FuncCursor::new(func); /// while let Some(ebb) = cursor.next_ebb() { @@ -344,8 +344,8 @@ pub trait Cursor { /// The `prev_ebb()` method is intended for iterating over the EBBs in backwards layout order: /// /// ``` - /// # use cretonne_codegen::ir::{Function, Ebb}; - /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; + /// # use cranelift_codegen::ir::{Function, Ebb}; + /// # use cranelift_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function) { /// let mut cursor = FuncCursor::new(func); /// while let Some(ebb) = cursor.prev_ebb() { @@ -381,8 +381,8 @@ pub trait Cursor { /// this: /// /// ``` - /// # use cretonne_codegen::ir::{Function, Ebb}; - /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; + /// # use cranelift_codegen::ir::{Function, Ebb}; + /// # use cranelift_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_ebb(func: &mut Function, ebb: Ebb) { /// let mut cursor = FuncCursor::new(func).at_top(ebb); /// while let Some(inst) = cursor.next_inst() { @@ -395,8 +395,8 @@ pub trait Cursor { /// Iterating over all the instructions in a function looks like this: /// /// ``` - /// # use cretonne_codegen::ir::{Function, Ebb}; - /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; + /// # use cranelift_codegen::ir::{Function, Ebb}; + /// # use cranelift_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_func(func: &mut Function) { /// let mut cursor = FuncCursor::new(func); /// while let Some(ebb) = cursor.next_ebb() { @@ -451,8 +451,8 @@ pub trait Cursor { /// EBB like this: /// /// ``` - /// # use cretonne_codegen::ir::{Function, Ebb}; - /// # use cretonne_codegen::cursor::{Cursor, FuncCursor}; + /// # use cranelift_codegen::ir::{Function, Ebb}; + /// # use cranelift_codegen::cursor::{Cursor, FuncCursor}; /// fn edit_ebb(func: &mut Function, ebb: Ebb) { /// let mut cursor = FuncCursor::new(func).at_bottom(ebb); /// while let Some(inst) = cursor.prev_inst() { diff --git a/lib/codegen/src/dbg.rs b/lib/codegen/src/dbg.rs index e1f62198f8..0b32bf2bf5 100644 --- a/lib/codegen/src/dbg.rs +++ b/lib/codegen/src/dbg.rs @@ -1,12 +1,12 @@ //! Debug tracing macros. //! //! This module defines the `dbg!` macro which works like `println!` except it writes to the -//! Cretonne tracing output file if enabled. +//! Cranelift tracing output file if enabled. //! -//! Tracing can be enabled by setting the `CRETONNE_DBG` environment variable to something +//! Tracing can be enabled by setting the `CRANELIFT_DBG` environment variable to something /// other than `0`. /// -/// The output will appear in files named `cretonne.dbg.*`, where the suffix is named after the +/// The output will appear in files named `cranelift.dbg.*`, where the suffix is named after the /// thread doing the logging. #[cfg(feature = "std")] use std::cell::RefCell; @@ -29,7 +29,7 @@ static STATE: atomic::AtomicIsize = atomic::ATOMIC_ISIZE_INIT; /// Is debug tracing enabled? /// -/// Debug tracing can be enabled by setting the `CRETONNE_DBG` environment variable to something +/// Debug tracing can be enabled by setting the `CRANELIFT_DBG` environment variable to something /// other than `0`. /// /// This inline function turns into a constant `false` when debug assertions are disabled. @@ -56,7 +56,7 @@ pub fn enabled() -> bool { /// Initialize `STATE` from the environment variable. #[cfg(feature = "std")] fn initialize() -> bool { - let enable = match env::var_os("CRETONNE_DBG") { + let enable = match env::var_os("CRANELIFT_DBG") { Some(s) => s != OsStr::new("0"), None => false, }; @@ -92,7 +92,7 @@ pub fn writeln_with_format_args(args: fmt::Arguments) -> io::Result<()> { fn open_file() -> io::BufWriter { let curthread = thread::current(); let tmpstr; - let mut path = "cretonne.dbg.".to_owned(); + let mut path = "cranelift.dbg.".to_owned(); path.extend( match curthread.name() { Some(name) => name.chars(), diff --git a/lib/codegen/src/ir/builder.rs b/lib/codegen/src/ir/builder.rs index e27cda0bcd..48cbf03238 100644 --- a/lib/codegen/src/ir/builder.rs +++ b/lib/codegen/src/ir/builder.rs @@ -1,6 +1,6 @@ -//! Cretonne instruction builder. +//! Cranelift instruction builder. //! -//! A `Builder` provides a convenient interface for inserting instructions into a Cretonne +//! A `Builder` provides a convenient interface for inserting instructions into a Cranelift //! function. Many of its methods are generated from the meta language instruction definitions. use ir; diff --git a/lib/codegen/src/ir/condcodes.rs b/lib/codegen/src/ir/condcodes.rs index 61967a5871..d7208278c7 100644 --- a/lib/codegen/src/ir/condcodes.rs +++ b/lib/codegen/src/ir/condcodes.rs @@ -1,4 +1,4 @@ -//! Condition codes for the Cretonne code generator. +//! Condition codes for the Cranelift code generator. //! //! A condition code here is an enumerated type that determined how to compare two numbers. There //! are different rules for comparing integers and floating point numbers, so they use different diff --git a/lib/codegen/src/ir/entities.rs b/lib/codegen/src/ir/entities.rs index 8c38926754..7196c78f31 100644 --- a/lib/codegen/src/ir/entities.rs +++ b/lib/codegen/src/ir/entities.rs @@ -1,6 +1,6 @@ -//! Cretonne IR entity references. +//! Cranelift IR entity references. //! -//! Instructions in Cretonne IR need to reference other entities in the function. This can be other +//! Instructions in Cranelift IR need to reference other entities in the function. This can be other //! parts of the function like extended basic blocks or stack slots, or it can be external entities //! that are declared in the function preamble in the text format. //! diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs index 00ac6da86f..5b7ac8fc5e 100644 --- a/lib/codegen/src/ir/extfunc.rs +++ b/lib/codegen/src/ir/extfunc.rs @@ -1,6 +1,6 @@ //! External function calls. //! -//! To a Cretonne function, all functions are "external". Directly called functions must be +//! To a Cranelift function, all functions are "external". Directly called functions must be //! declared in the preamble, and all function calls must have a signature. //! //! This module declares the data types used to represent external functions and call signatures. diff --git a/lib/codegen/src/ir/extname.rs b/lib/codegen/src/ir/extname.rs index 371b70664f..05f1adc54a 100644 --- a/lib/codegen/src/ir/extname.rs +++ b/lib/codegen/src/ir/extname.rs @@ -2,7 +2,7 @@ //! //! These are identifiers for declaring entities defined outside the current //! function. The name of an external declaration doesn't have any meaning to -//! Cretonne, which compiles functions independently. +//! Cranelift, which compiles functions independently. use ir::LibCall; use std::cmp; @@ -15,16 +15,16 @@ const TESTCASE_NAME_LENGTH: usize = 16; /// table, or a short sequence of ascii bytes so that test cases do not have /// to keep track of a sy mbol table. /// -/// External names are primarily used as keys by code using Cretonne to map -/// from a `cretonne_codegen::ir::FuncRef` or similar to additional associated +/// External names are primarily used as keys by code using Cranelift to map +/// from a `cranelift_codegen::ir::FuncRef` or similar to additional associated /// data. /// /// External names can also serve as a primitive testing and debugging tool. -/// In particular, many `.cton` test files use function names to identify +/// In particular, many `.clif` test files use function names to identify /// functions. #[derive(Debug, Clone, PartialEq, Eq)] pub enum ExternalName { - /// A name in a user-defined symbol table. Cretonne does not interpret + /// A name in a user-defined symbol table. Cranelift does not interpret /// these numbers in any way. User { /// Arbitrary. @@ -51,7 +51,7 @@ impl ExternalName { /// # Examples /// /// ```rust - /// # use cretonne_codegen::ir::ExternalName; + /// # use cranelift_codegen::ir::ExternalName; /// // Create `ExternalName` from a string. /// let name = ExternalName::testcase("hello"); /// assert_eq!(name.to_string(), "%hello"); @@ -72,7 +72,7 @@ impl ExternalName { /// /// # Examples /// ```rust - /// # use cretonne_codegen::ir::ExternalName; + /// # use cranelift_codegen::ir::ExternalName; /// // Create `ExternalName` from integer indicies /// let name = ExternalName::user(123, 456); /// assert_eq!(name.to_string(), "u123:456"); diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs index cff7bb2db0..3898709317 100644 --- a/lib/codegen/src/ir/function.rs +++ b/lib/codegen/src/ir/function.rs @@ -23,7 +23,7 @@ use write::write_function; /// The clone will have all the same entity numbers as the original. #[derive(Clone)] pub struct Function { - /// Name of this function. Mostly used by `.cton` files. + /// Name of this function. Mostly used by `.clif` files. pub name: ExternalName, /// Signature of this function. @@ -68,7 +68,7 @@ pub struct Function { /// Source locations. /// /// Track the original source location for each instruction. The source locations are not - /// interpreted by Cretonne, only preserved. + /// interpreted by Cranelift, only preserved. pub srclocs: SourceLocs, } diff --git a/lib/codegen/src/ir/globalvalue.rs b/lib/codegen/src/ir/globalvalue.rs index d522e26c1b..b29a1b5fd5 100644 --- a/lib/codegen/src/ir/globalvalue.rs +++ b/lib/codegen/src/ir/globalvalue.rs @@ -27,7 +27,7 @@ pub enum GlobalValueData { offset: Offset32, }, - /// Value is identified by a symbolic name. Cretonne itself does not interpret this name; + /// Value is identified by a symbolic name. Cranelift itself does not interpret this name; /// it's used by embedders to link with other data structures. Sym { /// The symbolic name. diff --git a/lib/codegen/src/ir/immediates.rs b/lib/codegen/src/ir/immediates.rs index fbc2637d0a..f61fce2196 100644 --- a/lib/codegen/src/ir/immediates.rs +++ b/lib/codegen/src/ir/immediates.rs @@ -1,7 +1,7 @@ -//! Immediate operands for Cretonne instructions +//! Immediate operands for Cranelift instructions //! -//! This module defines the types of immediate operands that can appear on Cretonne instructions. -//! Each type here should have a corresponding definition in the `cretonne.immediates` Python +//! This module defines the types of immediate operands that can appear on Cranelift instructions. +//! Each type here should have a corresponding definition in the `cranelift.immediates` Python //! module in the meta language. use std::fmt::{self, Display, Formatter}; diff --git a/lib/codegen/src/ir/instructions.rs b/lib/codegen/src/ir/instructions.rs index 32b4681b8b..d242622dac 100644 --- a/lib/codegen/src/ir/instructions.rs +++ b/lib/codegen/src/ir/instructions.rs @@ -64,7 +64,7 @@ impl Opcode { } } -// This trait really belongs in lib/reader where it is used by the `.cton` file parser, but since +// This trait really belongs in lib/reader where it is used by the `.clif` file parser, but since // it critically depends on the `opcode_name()` function which is needed here anyway, it lives in // this module. This also saves us from running the build script twice to generate code for the two // separate crates. diff --git a/lib/codegen/src/ir/libcall.rs b/lib/codegen/src/ir/libcall.rs index 43ac22e568..8bf8cc08c1 100644 --- a/lib/codegen/src/ir/libcall.rs +++ b/lib/codegen/src/ir/libcall.rs @@ -11,9 +11,9 @@ use std::str::FromStr; /// The name of a runtime library routine. /// -/// Runtime library calls are generated for Cretonne IR instructions that don't have an equivalent +/// Runtime library calls are generated for Cranelift IR instructions that don't have an equivalent /// ISA instruction or an easy macro expansion. A `LibCall` is used as a well-known name to refer to -/// the runtime library routine. This way, Cretonne doesn't have to know about the naming +/// the runtime library routine. This way, Cranelift doesn't have to know about the naming /// convention in the embedding VM's runtime library. /// /// This list is likely to grow over time. diff --git a/lib/codegen/src/ir/memflags.rs b/lib/codegen/src/ir/memflags.rs index e87fb3c4ec..b05cea2913 100644 --- a/lib/codegen/src/ir/memflags.rs +++ b/lib/codegen/src/ir/memflags.rs @@ -51,10 +51,10 @@ impl MemFlags { /// Test if the `notrap` flag is set. /// /// Normally, trapping is part of the semantics of a load/store operation. If the platform - /// would cause a trap when accessing the effective address, the Cretonne memory operation is + /// would cause a trap when accessing the effective address, the Cranelift memory operation is /// also required to trap. /// - /// The `notrap` flag tells Cretonne that the memory is *accessible*, which means that + /// The `notrap` flag tells Cranelift that the memory is *accessible*, which means that /// accesses will not trap. This makes it possible to delete an unused load or a dead store /// instruction. pub fn notrap(self) -> bool { @@ -68,7 +68,7 @@ impl MemFlags { /// Test if the `aligned` flag is set. /// - /// By default, Cretonne memory instructions work with any unaligned effective address. If the + /// By default, Cranelift memory instructions work with any unaligned effective address. If the /// `aligned` flag is set, the instruction is permitted to trap or return a wrong result if the /// effective address is misaligned. pub fn aligned(self) -> bool { diff --git a/lib/codegen/src/ir/mod.rs b/lib/codegen/src/ir/mod.rs index 1133f90af9..1580896902 100644 --- a/lib/codegen/src/ir/mod.rs +++ b/lib/codegen/src/ir/mod.rs @@ -1,4 +1,4 @@ -//! Representation of Cretonne IR functions. +//! Representation of Cranelift IR functions. mod builder; pub mod condcodes; diff --git a/lib/codegen/src/ir/progpoint.rs b/lib/codegen/src/ir/progpoint.rs index 86c0e591e4..b505af05a0 100644 --- a/lib/codegen/src/ir/progpoint.rs +++ b/lib/codegen/src/ir/progpoint.rs @@ -12,7 +12,7 @@ use std::u32; /// 1. An instruction or /// 2. An EBB header. /// -/// This corresponds more or less to the lines in the textual form of Cretonne IR. +/// This corresponds more or less to the lines in the textual form of Cranelift IR. #[derive(PartialEq, Eq, Clone, Copy)] pub struct ProgramPoint(u32); diff --git a/lib/codegen/src/ir/sourceloc.rs b/lib/codegen/src/ir/sourceloc.rs index 6de2206c8a..b6bea0e7d9 100644 --- a/lib/codegen/src/ir/sourceloc.rs +++ b/lib/codegen/src/ir/sourceloc.rs @@ -1,13 +1,13 @@ //! Source locations. //! -//! Cretonne tracks the original source location of each instruction, and preserves the source +//! Cranelift tracks the original source location of each instruction, and preserves the source //! location when instructions are transformed. use std::fmt; /// A source location. /// -/// This is an opaque 32-bit number attached to each Cretonne IR instruction. Cretonne does not +/// This is an opaque 32-bit number attached to each Cranelift IR instruction. Cranelift does not /// interpret source locations in any way, they are simply preserved from the input to the output. /// /// The default source location uses the all-ones bit pattern `!0`. It is used for instructions diff --git a/lib/codegen/src/ir/stackslot.rs b/lib/codegen/src/ir/stackslot.rs index 16794628df..fd5c02314b 100644 --- a/lib/codegen/src/ir/stackslot.rs +++ b/lib/codegen/src/ir/stackslot.rs @@ -15,7 +15,7 @@ use std::vec::Vec; /// The size of an object on the stack, or the size of a stack frame. /// -/// We don't use `usize` to represent object sizes on the target platform because Cretonne supports +/// We don't use `usize` to represent object sizes on the target platform because Cranelift supports /// cross-compilation, and `usize` is a type that depends on the host platform, not the target /// platform. pub type StackSize = u32; diff --git a/lib/codegen/src/ir/types.rs b/lib/codegen/src/ir/types.rs index c4dd5ad724..edaee867da 100644 --- a/lib/codegen/src/ir/types.rs +++ b/lib/codegen/src/ir/types.rs @@ -1,4 +1,4 @@ -//! Common types for the Cretonne code generator. +//! Common types for the Cranelift code generator. use std::default::Default; use std::fmt::{self, Debug, Display, Formatter}; diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs index 56ff2b5f4a..f5dd62f294 100644 --- a/lib/codegen/src/isa/mod.rs +++ b/lib/codegen/src/isa/mod.rs @@ -2,15 +2,15 @@ //! //! The `isa` module provides a `TargetIsa` trait which provides the behavior specialization needed //! by the ISA-independent code generator. The sub-modules of this module provide definitions for -//! the instruction sets that Cretonne can target. Each sub-module has it's own implementation of +//! the instruction sets that Cranelift can target. Each sub-module has it's own implementation of //! `TargetIsa`. //! //! # Constructing a `TargetIsa` instance //! //! The target ISA is built from the following information: //! -//! - The name of the target ISA as a string. Cretonne is a cross-compiler, so the ISA to target -//! can be selected dynamically. Individual ISAs can be left out when Cretonne is compiled, so a +//! - The name of the target ISA as a string. Cranelift is a cross-compiler, so the ISA to target +//! can be selected dynamically. Individual ISAs can be left out when Cranelift is compiled, so a //! string is used to identify the proper sub-module. //! - Values for settings that apply to all ISAs. This is represented by a `settings::Flags` //! instance. @@ -20,11 +20,11 @@ //! appropriate for the requested ISA: //! //! ``` -//! # extern crate cretonne_codegen; +//! # extern crate cranelift_codegen; //! # #[macro_use] extern crate target_lexicon; //! # fn main() { -//! use cretonne_codegen::settings::{self, Configurable}; -//! use cretonne_codegen::isa; +//! use cranelift_codegen::settings::{self, Configurable}; +//! use cranelift_codegen::isa; //! use std::str::FromStr; //! use target_lexicon::Triple; //! diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs index 5b9639e622..8bb1decd85 100644 --- a/lib/codegen/src/isa/x86/abi.rs +++ b/lib/codegen/src/isa/x86/abi.rs @@ -274,7 +274,7 @@ pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenRes pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> { debug_assert!( !isa.flags().probestack_enabled(), - "baldrdash does not expect cretonne to emit stack probes" + "baldrdash does not expect cranelift to emit stack probes" ); // Baldrdash on 32-bit x86 always aligns its stack pointer to 16 bytes. @@ -325,7 +325,7 @@ pub fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> C let csr_stack_size = ((csrs.iter(GPR).len() + 2) * word_size) as i32; // TODO: eventually use the 32 bytes (shadow store) as spill slot. This currently doesn't work - // since cretonne does not support spill slots before incoming args + // since cranelift does not support spill slots before incoming args func.create_stack_slot(ir::StackSlotData { kind: ir::StackSlotKind::IncomingArg, diff --git a/lib/codegen/src/legalizer/split.rs b/lib/codegen/src/legalizer/split.rs index 99babd4a27..bbb63397a5 100644 --- a/lib/codegen/src/legalizer/split.rs +++ b/lib/codegen/src/legalizer/split.rs @@ -14,13 +14,13 @@ //! //! When legalizing a single instruction, it is wrapped in splits and concatenations: //! -//!```cton +//!```clif //! v1 = bxor.i64 v2, v3 //! ``` //! //! becomes: //! -//!```cton +//!```clif //! v20, v21 = isplit v2 //! v30, v31 = isplit v3 //! v10 = bxor.i32 v20, v30 @@ -38,13 +38,13 @@ //! first check if the value is defined by the corresponding concatenation. If so, then just use //! the two concatenation inputs directly: //! -//! ```cton +//! ```clif //! v4 = iadd_imm.i64 v1, 1 //! ``` //! //! becomes, using the expanded code from above: //! -//! ```cton +//! ```clif //! v40, v5 = iadd_imm_cout.i32 v10, 1 //! v6 = bint.i32 //! v41 = iadd.i32 v11, v6 @@ -283,7 +283,7 @@ fn add_repair( /// /// Given this input: /// -/// ```cton +/// ```clif /// v10 = iconcat v1, v2 /// v11, v12 = isplit v10 /// ``` diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs index d892ecffe9..d4806f9665 100644 --- a/lib/codegen/src/lib.rs +++ b/lib/codegen/src/lib.rs @@ -1,4 +1,4 @@ -//! Cretonne code generation library. +//! Cranelift code generation library. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] #![warn(unused_import_braces)] @@ -56,11 +56,11 @@ pub use legalizer::legalize_function; pub use verifier::verify_function; pub use write::write_function; -/// Version number of the cretonne-codegen crate. +/// Version number of the cranelift-codegen crate. pub const VERSION: &str = env!("CARGO_PKG_VERSION"); #[macro_use] -pub extern crate cretonne_entity as entity; +pub extern crate cranelift_entity as entity; #[macro_use] pub mod dbg; diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs index 61e10d7ed1..068997e5c4 100644 --- a/lib/codegen/src/print_errors.rs +++ b/lib/codegen/src/print_errors.rs @@ -65,7 +65,7 @@ fn pretty_function_error( } } -/// Pretty-print a Cretonne error. +/// Pretty-print a Cranelift error. pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CodegenError) -> String { if let CodegenError::Verifier(e) = err { pretty_verifier_error(func, isa, &e) diff --git a/lib/codegen/src/regalloc/liveness.rs b/lib/codegen/src/regalloc/liveness.rs index 609b71c156..7f5ee4ae5d 100644 --- a/lib/codegen/src/regalloc/liveness.rs +++ b/lib/codegen/src/regalloc/liveness.rs @@ -64,7 +64,7 @@ //! LLVM's register allocator computes liveness per *virtual register*, where a virtual register is //! a disjoint union of related SSA values that should be assigned to the same physical register. //! It uses a compact data structure very similar to our `LiveRange`. The important difference is -//! that Cretonne's `LiveRange` only describes a single SSA value, while LLVM's `LiveInterval` +//! that Cranelift's `LiveRange` only describes a single SSA value, while LLVM's `LiveInterval` //! describes the live range of a virtual register *and* which one of the related SSA values is //! live at any given program point. //! @@ -93,11 +93,11 @@ //! functions. //! - A single live range can be recomputed after making modifications to the IR. No global //! algorithm is necessary. This feature depends on having use-def chains for virtual registers -//! which Cretonne doesn't. +//! which Cranelift doesn't. //! -//! Cretonne uses a very similar data structures and algorithms to LLVM, with the important +//! Cranelift uses a very similar data structures and algorithms to LLVM, with the important //! difference that live ranges are computed per SSA value instead of per virtual register, and the -//! uses in Cretonne IR refers to SSA values instead of virtual registers. This means that Cretonne +//! uses in Cranelift IR refers to SSA values instead of virtual registers. This means that Cranelift //! can skip the last step of reconstructing SSA form for the virtual register uses. //! //! ## Fast Liveness Checking for SSA-Form Programs @@ -112,27 +112,27 @@ //! then allows liveness queries for any (value, program point) pair. Each query traverses the use //! chain of the value and performs lookups in the precomputed bit-vectors. //! -//! I did not seriously consider this analysis for Cretonne because: +//! I did not seriously consider this analysis for Cranelift because: //! -//! - It depends critically on use chains which Cretonne doesn't have. +//! - It depends critically on use chains which Cranelift doesn't have. //! - Popular variables like the `this` pointer in a C++ method can have very large use chains. //! Traversing such a long use chain on every liveness lookup has the potential for some nasty //! quadratic behavior in unfortunate cases. //! - It says "fast" in the title, but the paper only claims to be 16% faster than a data-flow //! based approach, which isn't that impressive. //! -//! Nevertheless, the property of only depending in the CFG structure is very useful. If Cretonne +//! Nevertheless, the property of only depending in the CFG structure is very useful. If Cranelift //! gains use chains, this approach would be worth a proper evaluation. //! //! -//! # Cretonne's liveness analysis +//! # Cranelift's liveness analysis //! //! The algorithm implemented in this module is similar to LLVM's with these differences: //! //! - The `LiveRange` data structure describes the liveness of a single SSA value, not a virtual //! register. -//! - Instructions in Cretonne IR contains references to SSA values, not virtual registers. -//! - All live ranges are computed in one traversal of the program. Cretonne doesn't have use +//! - Instructions in Cranelift IR contains references to SSA values, not virtual registers. +//! - All live ranges are computed in one traversal of the program. Cranelift doesn't have use //! chains, so it is not possible to compute the live range for a single SSA value independently. //! //! The liveness computation visits all instructions in the program. The order is not important for @@ -150,18 +150,18 @@ //! visited. No data about each value beyond the live range is needed between visiting uses, so //! nothing is lost by computing the live range of all values simultaneously. //! -//! ## Cache efficiency of Cretonne vs LLVM +//! ## Cache efficiency of Cranelift vs LLVM //! //! Since LLVM computes the complete live range of a virtual register in one go, it can keep the //! whole `LiveInterval` for the register in L1 cache. Since it is visiting the instructions in use //! chain order, some cache thrashing can occur as a result of pulling instructions into cache //! somewhat chaotically. //! -//! Cretonne uses a transposed algorithm, visiting instructions in order. This means that each +//! Cranelift uses a transposed algorithm, visiting instructions in order. This means that each //! instruction is brought into cache only once, and it is likely that the other instructions on //! the same cache line will be visited before the line is evicted. //! -//! Cretonne's problem is that the `LiveRange` structs are visited many times and not always +//! Cranelift's problem is that the `LiveRange` structs are visited many times and not always //! regularly. We should strive to make the `LiveRange` struct as small as possible such that //! multiple related values can live on the same cache line. //! diff --git a/lib/codegen/src/regalloc/liverange.rs b/lib/codegen/src/regalloc/liverange.rs index a1e73b6206..f330fb12b5 100644 --- a/lib/codegen/src/regalloc/liverange.rs +++ b/lib/codegen/src/regalloc/liverange.rs @@ -41,7 +41,7 @@ //! //! If one live range ends at an instruction that defines another live range, those two live ranges //! are not considered to interfere. This is because most ISAs allow instructions to reuse an input -//! register for an output value. If Cretonne gets support for inline assembly, we will need to +//! register for an output value. If Cranelift gets support for inline assembly, we will need to //! handle *early clobbers* which are output registers that are not allowed to alias any input //! registers. //! diff --git a/lib/codegen/src/result.rs b/lib/codegen/src/result.rs index ffdcbb152b..883ccebda2 100644 --- a/lib/codegen/src/result.rs +++ b/lib/codegen/src/result.rs @@ -4,22 +4,22 @@ use verifier::VerifierError; /// A compilation error. /// -/// When Cretonne fails to compile a function, it will return one of these error codes. +/// When Cranelift fails to compile a function, it will return one of these error codes. #[derive(Fail, Debug, PartialEq, Eq)] pub enum CodegenError { /// An IR verifier error. /// - /// This always represents a bug, either in the code that generated IR for Cretonne, or a bug - /// in Cretonne itself. + /// This always represents a bug, either in the code that generated IR for Cranelift, or a bug + /// in Cranelift itself. #[fail(display = "Verifier error: {}", _0)] Verifier(#[cause] VerifierError), /// An implementation limit was exceeded. /// - /// Cretonne can compile very large and complicated functions, but the [implementation has + /// Cranelift can compile very large and complicated functions, but the [implementation has /// limits][limits] that cause compilation to fail when they are exceeded. /// - /// [limits]: https://cretonne.readthedocs.io/en/latest/langref.html#implementation-limits + /// [limits]: https://cranelift.readthedocs.io/en/latest/langref.html#implementation-limits #[fail(display = "Implementation limit exceeded")] ImplLimitExceeded, diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs index c11ca1a775..82f078a6a4 100644 --- a/lib/codegen/src/settings.rs +++ b/lib/codegen/src/settings.rs @@ -11,7 +11,7 @@ //! //! # Example //! ``` -//! use cretonne_codegen::settings::{self, Configurable}; +//! use cranelift_codegen::settings::{self, Configurable}; //! //! let mut b = settings::builder(); //! b.set("opt_level", "fastest"); diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs index 0dd0eb7991..c0bf1b8fdb 100644 --- a/lib/codegen/src/timing.rs +++ b/lib/codegen/src/timing.rs @@ -41,11 +41,11 @@ define_passes!{ Pass, NUM_PASSES, DESCRIPTIONS; process_file: "Processing test file", - parse_text: "Parsing textual Cretonne IR", + parse_text: "Parsing textual Cranelift IR", wasm_translate_module: "Translate WASM module", wasm_translate_function: "Translate WASM function", - verifier: "Verify Cretonne IR", + verifier: "Verify Cranelift IR", verify_cssa: "Verify CSSA", verify_liveness: "Verify live ranges", verify_locations: "Verify value locations", diff --git a/lib/codegen/src/verifier/cssa.rs b/lib/codegen/src/verifier/cssa.rs index b9ef1bad4c..8e3402208b 100644 --- a/lib/codegen/src/verifier/cssa.rs +++ b/lib/codegen/src/verifier/cssa.rs @@ -11,7 +11,7 @@ use verifier::VerifierResult; /// Verify conventional SSA form for `func`. /// -/// Conventional SSA form is represented in Cretonne with the help of virtual registers: +/// Conventional SSA form is represented in Cranelift with the help of virtual registers: /// /// - Two values are said to be *PHI-related* if one is an EBB argument and the other is passed as /// a branch argument in a location that matches the first value. diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs index 9914c45517..6423664f95 100644 --- a/lib/codegen/src/write.rs +++ b/lib/codegen/src/write.rs @@ -1,7 +1,7 @@ -//! Converting Cretonne IR to text. +//! Converting Cranelift IR to text. //! //! The `write` module provides the `write_function` function which converts an IR `Function` to an -//! equivalent textual form. This textual form can be read back by the `cretonne-reader` crate. +//! equivalent textual form. This textual form can be read back by the `cranelift-reader` crate. use ir::{DataFlowGraph, Ebb, Function, Inst, SigRef, Type, Value, ValueDef}; use isa::{RegInfo, TargetIsa}; diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml index a836db8293..2bb6e6f62f 100644 --- a/lib/entity/Cargo.toml +++ b/lib/entity/Cargo.toml @@ -1,11 +1,11 @@ [package] -authors = ["The Cretonne Project Developers"] -name = "cretonne-entity" +authors = ["The Cranelift Project Developers"] +name = "cranelift-entity" version = "0.13.0" description = "Data structures using entity references as mapping keys" license = "Apache-2.0 WITH LLVM-exception" -documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/cretonne/cretonne" +documentation = "https://cranelift.readthedocs.io/" +repository = "https://github.com/cranelift/cranelift" readme = "README.md" keywords = ["entity", "set", "map"] @@ -15,4 +15,4 @@ std = [] [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "cretonne/cretonne" } +travis-ci = { repository = "cranelift/cranelift" } diff --git a/lib/entity/README.md b/lib/entity/README.md index abe3b6210c..4679e74b40 100644 --- a/lib/entity/README.md +++ b/lib/entity/README.md @@ -1,2 +1,2 @@ -This crate contains array-based data structures used by the core Cretonne code +This crate contains array-based data structures used by the core Cranelift code generator which use densely numbered entity references as mapping keys. diff --git a/lib/entity/src/packed_option.rs b/lib/entity/src/packed_option.rs index a33d9ed58e..cf246a2498 100644 --- a/lib/entity/src/packed_option.rs +++ b/lib/entity/src/packed_option.rs @@ -1,6 +1,6 @@ //! Compact representation of `Option` for types with a reserved value. //! -//! Small Cretonne types like the 32-bit entity references are often used in tables and linked +//! Small Cranelift types like the 32-bit entity references are often used in tables and linked //! lists where an `Option` is needed. Unfortunately, that would double the size of the tables //! because `Option` is twice as big as `T`. //! diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml index 08760e8cd3..a9fbcbd778 100644 --- a/lib/faerie/Cargo.toml +++ b/lib/faerie/Cargo.toml @@ -1,16 +1,16 @@ [package] -name = "cretonne-faerie" +name = "cranelift-faerie" version = "0.13.0" -authors = ["The Cretonne Project Developers"] -description = "Emit Cretonne output to native object files with Faerie" -repository = "https://github.com/cretonne/cretonne" -documentation = "https://cretonne.readthedocs.io/" +authors = ["The Cranelift Project Developers"] +description = "Emit Cranelift output to native object files with Faerie" +repository = "https://github.com/cranelift/cranelift" +documentation = "https://cranelift.readthedocs.io/" license = "Apache-2.0 WITH LLVM-exception" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.13.0" } -cretonne-module = { path = "../module", version = "0.13.0" } +cranelift-codegen = { path = "../codegen", version = "0.13.0" } +cranelift-module = { path = "../module", version = "0.13.0" } faerie = "0.4.2" goblin = "0.0.15" failure = "0.1.1" @@ -18,4 +18,4 @@ target-lexicon = "0.0.2" [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "cretonne/cretonne" } +travis-ci = { repository = "cranelift/cranelift" } diff --git a/lib/faerie/README.md b/lib/faerie/README.md index 7817ccb21e..666d2db59d 100644 --- a/lib/faerie/README.md +++ b/lib/faerie/README.md @@ -1,4 +1,4 @@ This crate contains a library that enables -[Cretonne](https://crates.io/crates/cretonne) +[Cranelift](https://crates.io/crates/cranelift) to emit native object (".o") files, using the [Faerie](https://crates.io/crates/faerie) library. diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs index 73d4b3330d..09349cba8d 100644 --- a/lib/faerie/src/backend.rs +++ b/lib/faerie/src/backend.rs @@ -1,10 +1,10 @@ //! Defines `FaerieBackend`. use container; -use cretonne_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSink}; -use cretonne_codegen::isa::TargetIsa; -use cretonne_codegen::{self, binemit, ir}; -use cretonne_module::{ +use cranelift_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSink}; +use cranelift_codegen::isa::TargetIsa; +use cranelift_codegen::{self, binemit, ir}; +use cranelift_module::{ Backend, DataContext, DataDescription, Init, Linkage, ModuleError, ModuleNamespace, ModuleResult, }; @@ -34,16 +34,16 @@ pub struct FaerieBuilder { } impl FaerieBuilder { - /// Create a new `FaerieBuilder` using the given Cretonne target, that + /// Create a new `FaerieBuilder` using the given Cranelift target, that /// can be passed to - /// [`Module::new`](cretonne_module/struct.Module.html#method.new]. + /// [`Module::new`](cranelift_module/struct.Module.html#method.new]. /// /// Faerie output requires that TargetIsa have PIC (Position Independent Code) enabled. /// /// `collect_traps` setting determines whether trap information is collected in a /// `FaerieTrapManifest` available in the `FaerieProduct`. /// - /// The `libcall_names` function provides a way to translate `cretonne_codegen`'s `ir::LibCall` + /// The `libcall_names` function provides a way to translate `cranelift_codegen`'s `ir::LibCall` /// enum to symbols. LibCalls are inserted in the IR as part of the legalization for certain /// floating point instructions, and for stack probes. If you don't know what to use for this /// argument, use `FaerieBuilder::default_libcall_names()`. @@ -68,10 +68,10 @@ impl FaerieBuilder { /// Default names for `ir::LibCall`s. A function by this name is imported into the object as /// part of the translation of a `ir::ExternalName::LibCall` variant. Calls to a LibCall should - /// only be inserted into the IR by the `cretonne_codegen` legalizer pass. + /// only be inserted into the IR by the `cranelift_codegen` legalizer pass. pub fn default_libcall_names() -> Box String> { Box::new(move |libcall| match libcall { - ir::LibCall::Probestack => "__cretonne_probestack".to_owned(), + ir::LibCall::Probestack => "__cranelift_probestack".to_owned(), ir::LibCall::CeilF32 => "ceilf".to_owned(), ir::LibCall::CeilF64 => "ceil".to_owned(), ir::LibCall::FloorF32 => "floorf".to_owned(), @@ -111,7 +111,7 @@ impl Backend for FaerieBackend { /// to memory and files. type Product = FaerieProduct; - /// Create a new `FaerieBackend` using the given Cretonne target. + /// Create a new `FaerieBackend` using the given Cranelift target. fn new(builder: FaerieBuilder) -> Self { Self { artifact: faerie::Artifact::new(builder.isa.triple().clone(), builder.name), @@ -143,7 +143,7 @@ impl Backend for FaerieBackend { fn define_function( &mut self, name: &str, - ctx: &cretonne_codegen::Context, + ctx: &cranelift_codegen::Context, namespace: &ModuleNamespace, code_size: u32, ) -> ModuleResult { @@ -290,7 +290,7 @@ impl Backend for FaerieBackend { } /// This is the output of `Module`'s -/// [`finish`](../cretonne_module/struct.Module.html#method.finish) function. +/// [`finish`](../cranelift_module/struct.Module.html#method.finish) function. /// It provides functions for writing out the object file to memory or a file. pub struct FaerieProduct { /// Faerie artifact with all functions, data, and links from the module defined diff --git a/lib/faerie/src/container.rs b/lib/faerie/src/container.rs index 72766c1f0f..51355f4435 100644 --- a/lib/faerie/src/container.rs +++ b/lib/faerie/src/container.rs @@ -1,6 +1,6 @@ //! Utilities for working with Faerie container formats. -use cretonne_codegen::binemit::Reloc; +use cranelift_codegen::binemit::Reloc; use target_lexicon::{Architecture, BinaryFormat, Triple}; /// An object file format. @@ -12,7 +12,7 @@ pub enum Format { MachO, } -/// Translate from a Cretonne `Reloc` to a raw object-file-format-specific +/// Translate from a Cranelift `Reloc` to a raw object-file-format-specific /// relocation code and relocation-implied addend. pub fn raw_relocation(reloc: Reloc, triple: &Triple) -> (u32, i64) { match triple.binary_format { @@ -25,7 +25,7 @@ pub fn raw_relocation(reloc: Reloc, triple: &Triple) -> (u32, i64) { Reloc::Abs4 => elf::reloc::R_X86_64_32, Reloc::Abs8 => elf::reloc::R_X86_64_64, Reloc::X86PCRel4 | Reloc::X86CallPCRel4 => elf::reloc::R_X86_64_PC32, - // TODO: Get Cretonne to tell us when we can use + // TODO: Get Cranelift to tell us when we can use // R_X86_64_GOTPCRELX/R_X86_64_REX_GOTPCRELX. Reloc::X86CallPLTRel4 => elf::reloc::R_X86_64_PLT32, Reloc::X86GOTPCRel4 => elf::reloc::R_X86_64_GOTPCREL, diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs index c3f196004c..14397afbd0 100644 --- a/lib/faerie/src/lib.rs +++ b/lib/faerie/src/lib.rs @@ -1,4 +1,4 @@ -//! Top-level lib.rs for `cretonne_faerie`. +//! Top-level lib.rs for `cranelift_faerie`. //! //! Users of this module should not have to depend on faerie directly. @@ -14,8 +14,8 @@ ) )] -extern crate cretonne_codegen; -extern crate cretonne_module; +extern crate cranelift_codegen; +extern crate cranelift_module; extern crate faerie; extern crate failure; extern crate goblin; diff --git a/lib/faerie/src/traps.rs b/lib/faerie/src/traps.rs index e9526b757e..0a414d7981 100644 --- a/lib/faerie/src/traps.rs +++ b/lib/faerie/src/traps.rs @@ -1,15 +1,15 @@ -//! Faerie trap manifests record every `TrapCode` that cretonne outputs during code generation, +//! Faerie trap manifests record every `TrapCode` that cranelift outputs during code generation, //! for every function in the module. This data may be useful at runtime. -use cretonne_codegen::{binemit, ir}; +use cranelift_codegen::{binemit, ir}; -/// Record of the arguments cretonne passes to `TrapSink::trap` +/// Record of the arguments cranelift passes to `TrapSink::trap` pub struct FaerieTrapSite { /// Offset into function pub offset: binemit::CodeOffset, - /// Source location given to cretonne + /// Source location given to cranelift pub srcloc: ir::SourceLoc, - /// Trap code, as determined by cretonne + /// Trap code, as determined by cranelift pub code: ir::TrapCode, } diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml index 2ed1e7db8d..dea40496c9 100644 --- a/lib/filetests/Cargo.toml +++ b/lib/filetests/Cargo.toml @@ -1,15 +1,15 @@ [package] -name = "cretonne-filetests" -authors = ["The Cretonne Project Developers"] +name = "cranelift-filetests" +authors = ["The Cranelift Project Developers"] version = "0.13.0" description = "Test driver and implementations of the filetest commands" license = "Apache-2.0 WITH LLVM-exception" -documentation = "https://cretonne.readthedocs.io/en/latest/testing.html#file-tests" -repository = "https://github.com/cretonne/cretonne" +documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests" +repository = "https://github.com/cranelift/cranelift" publish = false [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.13.0" } -cretonne-reader = { path = "../reader", version = "0.13.0" } +cranelift-codegen = { path = "../codegen", version = "0.13.0" } +cranelift-reader = { path = "../reader", version = "0.13.0" } filecheck = "0.3.0" num_cpus = "1.8.0" diff --git a/lib/filetests/src/concurrent.rs b/lib/filetests/src/concurrent.rs index d9ee4f3c07..f874e952f2 100644 --- a/lib/filetests/src/concurrent.rs +++ b/lib/filetests/src/concurrent.rs @@ -3,7 +3,7 @@ //! This module provides the `ConcurrentRunner` struct which uses a pool of threads to run tests //! concurrently. -use cretonne_codegen::timing; +use cranelift_codegen::timing; use num_cpus; use std::panic::catch_unwind; use std::path::{Path, PathBuf}; diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs index a86c0a0aae..6d98c991f2 100644 --- a/lib/filetests/src/lib.rs +++ b/lib/filetests/src/lib.rs @@ -18,12 +18,12 @@ )] #[macro_use(dbg)] -extern crate cretonne_codegen; -extern crate cretonne_reader; +extern crate cranelift_codegen; +extern crate cranelift_reader; extern crate filecheck; extern crate num_cpus; -use cretonne_reader::TestCommand; +use cranelift_reader::TestCommand; use runner::TestRunner; use std::path::Path; use std::time; @@ -52,13 +52,13 @@ mod test_verifier; /// The result of running the test in a file. type TestResult = Result; -/// Main entry point for `cton-util test`. +/// Main entry point for `clif-util test`. /// -/// Take a list of filenames which can be either `.cton` files or directories. +/// Take a list of filenames which can be either `.clif` files or directories. /// /// Files are interpreted as test cases and executed immediately. /// -/// Directories are scanned recursively for test cases ending in `.cton`. These test cases are +/// Directories are scanned recursively for test cases ending in `.clif`. These test cases are /// executed on background threads. /// pub fn run(verbose: bool, files: &[String]) -> TestResult { @@ -79,7 +79,7 @@ pub fn run(verbose: bool, files: &[String]) -> TestResult { /// Create a new subcommand trait object to match `parsed.command`. /// /// This function knows how to create all of the possible `test ` commands that can appear in -/// a `.cton` test file. +/// a `.clif` test file. fn new_subtest(parsed: &TestCommand) -> subtest::SubtestResult> { match parsed.command { "binemit" => test_binemit::subtest(parsed), diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs index 3b261cd44b..3d630d83a4 100644 --- a/lib/filetests/src/runner.rs +++ b/lib/filetests/src/runner.rs @@ -116,7 +116,7 @@ impl TestRunner { // This recursive search tries to minimize statting in a directory hierarchy containing // mostly test cases. // - // - Directory entries with a "cton" extension are presumed to be test case files. + // - Directory entries with a "clif" extension are presumed to be test case files. // - Directory entries with no extension are presumed to be subdirectories. // - Anything else is ignored. // @@ -149,7 +149,7 @@ impl TestRunner { // Recognize directories and tests by extension. // Yes, this means we ignore directories with '.' in their name. match path.extension().and_then(OsStr::to_str) { - Some("cton") => self.push_test(path), + Some("clif") => self.push_test(path), Some(_) => {} None => self.push_dir(path), } diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs index 26cc31a839..4036e482ad 100644 --- a/lib/filetests/src/runone.rs +++ b/lib/filetests/src/runone.rs @@ -1,13 +1,13 @@ //! Run the tests in a single test file. -use cretonne_codegen::ir::Function; -use cretonne_codegen::isa::TargetIsa; -use cretonne_codegen::print_errors::pretty_verifier_error; -use cretonne_codegen::settings::Flags; -use cretonne_codegen::timing; -use cretonne_codegen::verify_function; -use cretonne_reader::parse_test; -use cretonne_reader::IsaSpec; +use cranelift_codegen::ir::Function; +use cranelift_codegen::isa::TargetIsa; +use cranelift_codegen::print_errors::pretty_verifier_error; +use cranelift_codegen::settings::Flags; +use cranelift_codegen::timing; +use cranelift_codegen::verify_function; +use cranelift_reader::parse_test; +use cranelift_reader::IsaSpec; use std::borrow::Cow; use std::fs; use std::io::{self, Read}; diff --git a/lib/filetests/src/subtest.rs b/lib/filetests/src/subtest.rs index 91c92d251a..4400561cc8 100644 --- a/lib/filetests/src/subtest.rs +++ b/lib/filetests/src/subtest.rs @@ -1,9 +1,9 @@ //! `SubTest` trait. -use cretonne_codegen::ir::Function; -use cretonne_codegen::isa::TargetIsa; -use cretonne_codegen::settings::{Flags, FlagsOrIsa}; -use cretonne_reader::{Comment, Details}; +use cranelift_codegen::ir::Function; +use cranelift_codegen::isa::TargetIsa; +use cranelift_codegen::settings::{Flags, FlagsOrIsa}; +use cranelift_reader::{Comment, Details}; use filecheck::{Checker, CheckerBuilder, NO_VARIABLES}; use std::borrow::Cow; @@ -40,7 +40,7 @@ impl<'a> Context<'a> { /// Common interface for implementations of test commands. /// -/// Each `.cton` test file may contain multiple test commands, each represented by a `SubTest` +/// Each `.clif` test file may contain multiple test commands, each represented by a `SubTest` /// trait object. pub trait SubTest { /// Name identifying this subtest. Typically the same as the test command. diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs index 1eb24cf31a..0dd2ea6fcf 100644 --- a/lib/filetests/src/test_binemit.rs +++ b/lib/filetests/src/test_binemit.rs @@ -3,14 +3,14 @@ //! The `binemit` test command generates binary machine code for every instruction in the input //! functions and compares the results to the expected output. -use cretonne_codegen::binemit; -use cretonne_codegen::binemit::RegDiversions; -use cretonne_codegen::dbg::DisplayList; -use cretonne_codegen::ir; -use cretonne_codegen::ir::entities::AnyEntity; -use cretonne_codegen::print_errors::pretty_error; -use cretonne_codegen::settings::OptLevel; -use cretonne_reader::TestCommand; +use cranelift_codegen::binemit; +use cranelift_codegen::binemit::RegDiversions; +use cranelift_codegen::dbg::DisplayList; +use cranelift_codegen::ir; +use cranelift_codegen::ir::entities::AnyEntity; +use cranelift_codegen::print_errors::pretty_error; +use cranelift_codegen::settings::OptLevel; +use cranelift_reader::TestCommand; use match_directive::match_directive; use std::borrow::Cow; use std::collections::HashMap; diff --git a/lib/filetests/src/test_cat.rs b/lib/filetests/src/test_cat.rs index c3b1a1aa4a..ad37b444e6 100644 --- a/lib/filetests/src/test_cat.rs +++ b/lib/filetests/src/test_cat.rs @@ -1,7 +1,7 @@ //! The `cat` subtest. -use cretonne_codegen::ir::Function; -use cretonne_reader::TestCommand; +use cranelift_codegen::ir::Function; +use cranelift_reader::TestCommand; use std::borrow::Cow; use subtest::{self, Context, SubTest, SubtestResult}; diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs index 7fce3b56ad..0e79bab429 100644 --- a/lib/filetests/src/test_compile.rs +++ b/lib/filetests/src/test_compile.rs @@ -2,10 +2,10 @@ //! //! The `compile` test command runs each function through the full code generator pipeline -use cretonne_codegen; -use cretonne_codegen::print_errors::pretty_error; -use cretonne_codegen::{binemit, ir}; -use cretonne_reader::TestCommand; +use cranelift_codegen; +use cranelift_codegen::print_errors::pretty_error; +use cranelift_codegen::{binemit, ir}; +use cranelift_reader::TestCommand; use std::borrow::Cow; use subtest::{run_filecheck, Context, SubTest, SubtestResult}; @@ -35,7 +35,7 @@ impl SubTest for TestCompile { fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let isa = context.isa.expect("compile needs an ISA"); - let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); + let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned()); let code_size = comp_ctx .compile(isa) diff --git a/lib/filetests/src/test_dce.rs b/lib/filetests/src/test_dce.rs index 748c7ecf79..e5a4e79f98 100644 --- a/lib/filetests/src/test_dce.rs +++ b/lib/filetests/src/test_dce.rs @@ -5,10 +5,10 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne_codegen; -use cretonne_codegen::ir::Function; -use cretonne_codegen::print_errors::pretty_error; -use cretonne_reader::TestCommand; +use cranelift_codegen; +use cranelift_codegen::ir::Function; +use cranelift_codegen::print_errors::pretty_error; +use cranelift_reader::TestCommand; use std::borrow::Cow; use subtest::{run_filecheck, Context, SubTest, SubtestResult}; @@ -33,7 +33,7 @@ impl SubTest for TestDCE { } fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { - let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); + let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned()); comp_ctx.flowgraph(); comp_ctx.compute_loop_analysis(); diff --git a/lib/filetests/src/test_domtree.rs b/lib/filetests/src/test_domtree.rs index b6d6922c7f..87df48a7a2 100644 --- a/lib/filetests/src/test_domtree.rs +++ b/lib/filetests/src/test_domtree.rs @@ -2,7 +2,7 @@ //! //! The `test domtree` test command looks for annotations on instructions like this: //! -//! ```cton +//! ```clif //! jump ebb3 ; dominates: ebb3 //! ``` //! @@ -12,11 +12,11 @@ //! We verify that the dominator tree annotations are complete and correct. //! -use cretonne_codegen::dominator_tree::{DominatorTree, DominatorTreePreorder}; -use cretonne_codegen::flowgraph::ControlFlowGraph; -use cretonne_codegen::ir::entities::AnyEntity; -use cretonne_codegen::ir::Function; -use cretonne_reader::TestCommand; +use cranelift_codegen::dominator_tree::{DominatorTree, DominatorTreePreorder}; +use cranelift_codegen::flowgraph::ControlFlowGraph; +use cranelift_codegen::ir::entities::AnyEntity; +use cranelift_codegen::ir::Function; +use cranelift_reader::TestCommand; use match_directive::match_directive; use std::borrow::{Borrow, Cow}; use std::collections::HashMap; diff --git a/lib/filetests/src/test_legalizer.rs b/lib/filetests/src/test_legalizer.rs index eda564e34f..10821ee6ab 100644 --- a/lib/filetests/src/test_legalizer.rs +++ b/lib/filetests/src/test_legalizer.rs @@ -3,10 +3,10 @@ //! The `test legalizer` test command runs each function through `legalize_function()` and sends //! the result to filecheck. -use cretonne_codegen; -use cretonne_codegen::ir::Function; -use cretonne_codegen::print_errors::pretty_error; -use cretonne_reader::TestCommand; +use cranelift_codegen; +use cranelift_codegen::ir::Function; +use cranelift_codegen::print_errors::pretty_error; +use cranelift_reader::TestCommand; use std::borrow::Cow; use subtest::{run_filecheck, Context, SubTest, SubtestResult}; @@ -35,7 +35,7 @@ impl SubTest for TestLegalizer { } fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { - let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); + let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned()); let isa = context.isa.expect("legalizer needs an ISA"); comp_ctx.compute_cfg(); diff --git a/lib/filetests/src/test_licm.rs b/lib/filetests/src/test_licm.rs index 6004b1529e..149cf788cf 100644 --- a/lib/filetests/src/test_licm.rs +++ b/lib/filetests/src/test_licm.rs @@ -5,10 +5,10 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne_codegen; -use cretonne_codegen::ir::Function; -use cretonne_codegen::print_errors::pretty_error; -use cretonne_reader::TestCommand; +use cranelift_codegen; +use cranelift_codegen::ir::Function; +use cranelift_codegen::print_errors::pretty_error; +use cranelift_reader::TestCommand; use std::borrow::Cow; use subtest::{run_filecheck, Context, SubTest, SubtestResult}; @@ -33,7 +33,7 @@ impl SubTest for TestLICM { } fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { - let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); + let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned()); comp_ctx.flowgraph(); comp_ctx.compute_loop_analysis(); diff --git a/lib/filetests/src/test_postopt.rs b/lib/filetests/src/test_postopt.rs index 16367350c4..248528c2b6 100644 --- a/lib/filetests/src/test_postopt.rs +++ b/lib/filetests/src/test_postopt.rs @@ -2,10 +2,10 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne_codegen; -use cretonne_codegen::ir::Function; -use cretonne_codegen::print_errors::pretty_error; -use cretonne_reader::TestCommand; +use cranelift_codegen; +use cranelift_codegen::ir::Function; +use cranelift_codegen::print_errors::pretty_error; +use cranelift_reader::TestCommand; use std::borrow::Cow; use subtest::{run_filecheck, Context, SubTest, SubtestResult}; @@ -30,7 +30,7 @@ impl SubTest for TestPostopt { } fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { - let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); + let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned()); let isa = context.isa.expect("postopt needs an ISA"); comp_ctx.flowgraph(); diff --git a/lib/filetests/src/test_preopt.rs b/lib/filetests/src/test_preopt.rs index 169db9a583..7e4ae15db7 100644 --- a/lib/filetests/src/test_preopt.rs +++ b/lib/filetests/src/test_preopt.rs @@ -2,10 +2,10 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne_codegen; -use cretonne_codegen::ir::Function; -use cretonne_codegen::print_errors::pretty_error; -use cretonne_reader::TestCommand; +use cranelift_codegen; +use cranelift_codegen::ir::Function; +use cranelift_codegen::print_errors::pretty_error; +use cranelift_reader::TestCommand; use std::borrow::Cow; use subtest::{run_filecheck, Context, SubTest, SubtestResult}; @@ -30,7 +30,7 @@ impl SubTest for TestPreopt { } fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { - let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); + let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned()); let isa = context.isa.expect("preopt needs an ISA"); comp_ctx.flowgraph(); diff --git a/lib/filetests/src/test_print_cfg.rs b/lib/filetests/src/test_print_cfg.rs index 5f196d6228..4de4a3e7f4 100644 --- a/lib/filetests/src/test_print_cfg.rs +++ b/lib/filetests/src/test_print_cfg.rs @@ -1,13 +1,13 @@ //! The `print-cfg` sub-command. //! -//! Read a series of Cretonne IR files and print their control flow graphs +//! Read a series of Cranelift IR files and print their control flow graphs //! in graphviz format. use std::borrow::Cow; -use cretonne_codegen::cfg_printer::CFGPrinter; -use cretonne_codegen::ir::Function; -use cretonne_reader::TestCommand; +use cranelift_codegen::cfg_printer::CFGPrinter; +use cranelift_codegen::ir::Function; +use cranelift_reader::TestCommand; use subtest::{self, Context, SubTest, SubtestResult}; /// Object implementing the `test print-cfg` sub-test. diff --git a/lib/filetests/src/test_regalloc.rs b/lib/filetests/src/test_regalloc.rs index eb13ee402d..e09623a691 100644 --- a/lib/filetests/src/test_regalloc.rs +++ b/lib/filetests/src/test_regalloc.rs @@ -5,10 +5,10 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne_codegen; -use cretonne_codegen::ir::Function; -use cretonne_codegen::print_errors::pretty_error; -use cretonne_reader::TestCommand; +use cranelift_codegen; +use cranelift_codegen::ir::Function; +use cranelift_codegen::print_errors::pretty_error; +use cranelift_reader::TestCommand; use std::borrow::Cow; use subtest::{run_filecheck, Context, SubTest, SubtestResult}; @@ -38,7 +38,7 @@ impl SubTest for TestRegalloc { fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let isa = context.isa.expect("register allocator needs an ISA"); - let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); + let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned()); comp_ctx.compute_cfg(); // TODO: Should we have an option to skip legalization? diff --git a/lib/filetests/src/test_shrink.rs b/lib/filetests/src/test_shrink.rs index 4388919f36..69eaabc007 100644 --- a/lib/filetests/src/test_shrink.rs +++ b/lib/filetests/src/test_shrink.rs @@ -5,10 +5,10 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne_codegen; -use cretonne_codegen::ir::Function; -use cretonne_codegen::print_errors::pretty_error; -use cretonne_reader::TestCommand; +use cranelift_codegen; +use cranelift_codegen::ir::Function; +use cranelift_codegen::print_errors::pretty_error; +use cranelift_reader::TestCommand; use std::borrow::Cow; use subtest::{run_filecheck, Context, SubTest, SubtestResult}; @@ -34,7 +34,7 @@ impl SubTest for TestShrink { fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { let isa = context.isa.expect("shrink needs an ISA"); - let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); + let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned()); comp_ctx .shrink_instructions(isa) diff --git a/lib/filetests/src/test_simple_gvn.rs b/lib/filetests/src/test_simple_gvn.rs index faf995cff3..d6d06f49ce 100644 --- a/lib/filetests/src/test_simple_gvn.rs +++ b/lib/filetests/src/test_simple_gvn.rs @@ -5,10 +5,10 @@ //! //! The resulting function is sent to `filecheck`. -use cretonne_codegen; -use cretonne_codegen::ir::Function; -use cretonne_codegen::print_errors::pretty_error; -use cretonne_reader::TestCommand; +use cranelift_codegen; +use cranelift_codegen::ir::Function; +use cranelift_codegen::print_errors::pretty_error; +use cranelift_reader::TestCommand; use std::borrow::Cow; use subtest::{run_filecheck, Context, SubTest, SubtestResult}; @@ -33,7 +33,7 @@ impl SubTest for TestSimpleGVN { } fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> { - let mut comp_ctx = cretonne_codegen::Context::for_function(func.into_owned()); + let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned()); comp_ctx.flowgraph(); comp_ctx diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs index 9be54e7482..422deaf25b 100644 --- a/lib/filetests/src/test_verifier.rs +++ b/lib/filetests/src/test_verifier.rs @@ -2,16 +2,16 @@ //! //! The `test verifier` test command looks for annotations on instructions like this: //! -//! ```cton +//! ```clif //! jump ebb3 ; error: jump to non-existent EBB //! ``` //! //! This annotation means that the verifier is expected to given an error for the jump instruction //! containing the substring "jump to non-existent EBB". -use cretonne_codegen::ir::Function; -use cretonne_codegen::verify_function; -use cretonne_reader::TestCommand; +use cranelift_codegen::ir::Function; +use cranelift_codegen::verify_function; +use cranelift_reader::TestCommand; use match_directive::match_directive; use std::borrow::{Borrow, Cow}; use subtest::{Context, SubTest, SubtestResult}; diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml index 9740c124bc..b9833b4582 100644 --- a/lib/frontend/Cargo.toml +++ b/lib/frontend/Cargo.toml @@ -1,21 +1,21 @@ [package] -authors = ["The Cretonne Project Developers"] -name = "cretonne-frontend" +authors = ["The Cranelift Project Developers"] +name = "cranelift-frontend" version = "0.13.0" -description = "Cretonne IR builder helper" +description = "Cranelift IR builder helper" license = "Apache-2.0 WITH LLVM-exception" -documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/cretonne/cretonne" +documentation = "https://cranelift.readthedocs.io/" +repository = "https://github.com/cranelift/cranelift" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.13.0", default-features = false } +cranelift-codegen = { path = "../codegen", version = "0.13.0", default-features = false } [features] default = ["std"] -std = ["cretonne-codegen/std"] -core = ["cretonne-codegen/core"] +std = ["cranelift-codegen/std"] +core = ["cranelift-codegen/core"] [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "cretonne/cretonne" } +travis-ci = { repository = "cranelift/cranelift" } diff --git a/lib/frontend/README.md b/lib/frontend/README.md index cca2b12ab5..e43ad48f45 100644 --- a/lib/frontend/README.md +++ b/lib/frontend/README.md @@ -1,5 +1,5 @@ This crate provides a straightforward way to create a -[Cretonne](https://crates.io/crates/cretonne) IR function and fill it with +[Cranelift](https://crates.io/crates/cranelift) IR function and fill it with instructions translated from another language. It contains an SSA construction module that provides convenient methods for translating non-SSA variables into -SSA Cretonne IR values via `use_var` and `def_var` calls. +SSA Cranelift IR values via `use_var` and `def_var` calls. diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs index eabadad15f..429b72c49e 100644 --- a/lib/frontend/src/frontend.rs +++ b/lib/frontend/src/frontend.rs @@ -1,19 +1,19 @@ -//! A frontend for building Cretonne IR from other languages. -use cretonne_codegen::cursor::{Cursor, FuncCursor}; -use cretonne_codegen::entity::{EntityMap, EntityRef, EntitySet}; -use cretonne_codegen::ir; -use cretonne_codegen::ir::function::DisplayFunction; -use cretonne_codegen::ir::{ +//! A frontend for building Cranelift IR from other languages. +use cranelift_codegen::cursor::{Cursor, FuncCursor}; +use cranelift_codegen::entity::{EntityMap, EntityRef, EntitySet}; +use cranelift_codegen::ir; +use cranelift_codegen::ir::function::DisplayFunction; +use cranelift_codegen::ir::{ DataFlowGraph, Ebb, ExtFuncData, FuncRef, Function, GlobalValue, GlobalValueData, Heap, HeapData, Inst, InstBuilderBase, InstructionData, JumpTable, JumpTableData, SigRef, Signature, StackSlot, StackSlotData, Type, Value, }; -use cretonne_codegen::isa::TargetIsa; -use cretonne_codegen::packed_option::PackedOption; +use cranelift_codegen::isa::TargetIsa; +use cranelift_codegen::packed_option::PackedOption; use ssa::{Block, SSABuilder, SideEffects}; use std::fmt::Debug; -/// Structure used for translating a series of functions into Cretonne IR. +/// Structure used for translating a series of functions into Cranelift IR. /// /// In order to reduce memory reallocations when compiling multiple functions, /// `FunctionBuilderContext` holds various data structures which are cleared between @@ -31,7 +31,7 @@ where types: EntityMap, } -/// Temporary object used to build a single Cretonne IR `Function`. +/// Temporary object used to build a single Cranelift IR `Function`. pub struct FunctionBuilder<'a, Variable: 'a> where Variable: EntityRef + Debug, @@ -105,7 +105,7 @@ where } /// Implementation of the [`InstBuilder`](../codegen/ir/builder/trait.InstBuilder.html) that has -/// one convenience method per Cretonne IR instruction. +/// one convenience method per Cranelift IR instruction. pub struct FuncInstBuilder<'short, 'long: 'short, Variable: 'long> where Variable: EntityRef + Debug, @@ -195,7 +195,7 @@ where } } -/// This module allows you to create a function in Cretonne IR in a straightforward way, hiding +/// This module allows you to create a function in Cranelift IR in a straightforward way, hiding /// all the complexity of its internal representation. /// /// The module is parametrized by one type which is the representation of variables in your @@ -207,10 +207,10 @@ where /// - the last instruction of each block is a terminator instruction which has no natural successor, /// and those instructions can only appear at the end of extended blocks. /// -/// The parameters of Cretonne IR instructions are Cretonne IR values, which can only be created -/// as results of other Cretonne IR instructions. To be able to create variables redefined multiple +/// The parameters of Cranelift IR instructions are Cranelift IR values, which can only be created +/// as results of other Cranelift IR instructions. To be able to create variables redefined multiple /// times in your program, use the `def_var` and `use_var` command, that will maintain the -/// correspondence between your variables and Cretonne IR SSA values. +/// correspondence between your variables and Cranelift IR SSA values. /// /// The first block for which you call `switch_to_block` will be assumed to be the beginning of /// the function. @@ -224,7 +224,7 @@ where /// /// # Errors /// -/// The functions below will panic in debug mode whenever you try to modify the Cretonne IR +/// The functions below will panic in debug mode whenever you try to modify the Cranelift IR /// function in a way that violate the coherence of the code. For instance: switching to a new /// `Ebb` when you haven't filled the current one with a terminator instruction, inserting a /// return instruction with arguments that don't match the function's signature. @@ -317,7 +317,7 @@ where self.func_ctx.types[var] = ty; } - /// Returns the Cretonne IR value corresponding to the utilization at the current program + /// Returns the Cranelift IR value corresponding to the utilization at the current program /// position of a previously defined user variable. pub fn use_var(&mut self, var: Variable) -> Value { let (val, side_effects) = { @@ -481,8 +481,8 @@ where } /// All the functions documented in the previous block are write-only and help you build a valid -/// Cretonne IR functions via multiple debug asserts. However, you might need to improve the -/// performance of your translation perform more complex transformations to your Cretonne IR +/// Cranelift IR functions via multiple debug asserts. However, you might need to improve the +/// performance of your translation perform more complex transformations to your Cranelift IR /// function. The functions below help you inspect the function you're creating and modify it /// in ways that can be unsafe if used incorrectly. impl<'a, Variable> FunctionBuilder<'a, Variable> @@ -610,12 +610,12 @@ where #[cfg(test)] mod tests { - use cretonne_codegen::entity::EntityRef; - use cretonne_codegen::ir::types::*; - use cretonne_codegen::ir::{AbiParam, ExternalName, Function, InstBuilder, Signature}; - use cretonne_codegen::settings; - use cretonne_codegen::settings::CallConv; - use cretonne_codegen::verifier::verify_function; + use cranelift_codegen::entity::EntityRef; + use cranelift_codegen::ir::types::*; + use cranelift_codegen::ir::{AbiParam, ExternalName, Function, InstBuilder, Signature}; + use cranelift_codegen::settings; + use cranelift_codegen::settings::CallConv; + use cranelift_codegen::verifier::verify_function; use frontend::{FunctionBuilder, FunctionBuilderContext}; use Variable; diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs index 8a414dd849..be19ebc2be 100644 --- a/lib/frontend/src/lib.rs +++ b/lib/frontend/src/lib.rs @@ -1,17 +1,17 @@ -//! Cretonne IR builder library. +//! Cranelift IR builder library. //! -//! Provides a straightforward way to create a Cretonne IR function and fill it with instructions +//! Provides a straightforward way to create a Cranelift IR function and fill it with instructions //! translated from another language. Contains an SSA construction module that lets you translate -//! your non-SSA variables into SSA Cretonne IR values via `use_var` and `def_var` calls. +//! your non-SSA variables into SSA Cranelift IR values via `use_var` and `def_var` calls. //! //! To get started, create an [`FunctionBuilderContext`](struct.FunctionBuilderContext.html) and //! pass it as an argument to a [`FunctionBuilder`](struct.FunctionBuilder.html). //! //! # Example //! -//! Here is a pseudo-program we want to transform into Cretonne IR: +//! Here is a pseudo-program we want to transform into Cranelift IR: //! -//! ```cton +//! ```clif //! function(x) { //! x, y, z : i32 //! block0: @@ -29,18 +29,18 @@ //! } //! ``` //! -//! Here is how you build the corresponding Cretonne IR function using `FunctionBuilderContext`: +//! Here is how you build the corresponding Cranelift IR function using `FunctionBuilderContext`: //! //! ```rust -//! extern crate cretonne_codegen; -//! extern crate cretonne_frontend; +//! extern crate cranelift_codegen; +//! extern crate cranelift_frontend; //! -//! use cretonne_codegen::entity::EntityRef; -//! use cretonne_codegen::ir::{ExternalName, Function, Signature, AbiParam, InstBuilder}; -//! use cretonne_codegen::ir::types::*; -//! use cretonne_codegen::settings::{self, CallConv}; -//! use cretonne_frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; -//! use cretonne_codegen::verifier::verify_function; +//! use cranelift_codegen::entity::EntityRef; +//! use cranelift_codegen::ir::{ExternalName, Function, Signature, AbiParam, InstBuilder}; +//! use cranelift_codegen::ir::types::*; +//! use cranelift_codegen::settings::{self, CallConv}; +//! use cranelift_frontend::{FunctionBuilderContext, FunctionBuilder, Variable}; +//! use cranelift_codegen::verifier::verify_function; //! //! fn main() { //! let mut sig = Signature::new(CallConv::SystemV); @@ -141,7 +141,7 @@ #![cfg_attr(not(feature = "std"), no_std)] #![cfg_attr(not(feature = "std"), feature(alloc))] -extern crate cretonne_codegen; +extern crate cranelift_codegen; pub use frontend::{FunctionBuilder, FunctionBuilderContext}; pub use variable::Variable; diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs index a7f4833fc3..9c676ceb0b 100644 --- a/lib/frontend/src/ssa.rs +++ b/lib/frontend/src/ssa.rs @@ -5,14 +5,14 @@ //! In: Jhala R., De Bosschere K. (eds) Compiler Construction. CC 2013. //! Lecture Notes in Computer Science, vol 7791. Springer, Berlin, Heidelberg -use cretonne_codegen::cursor::{Cursor, FuncCursor}; -use cretonne_codegen::entity::{EntityMap, EntityRef, PrimaryMap}; -use cretonne_codegen::ir::immediates::{Ieee32, Ieee64}; -use cretonne_codegen::ir::instructions::BranchInfo; -use cretonne_codegen::ir::types::{F32, F64}; -use cretonne_codegen::ir::{Ebb, Function, Inst, InstBuilder, Type, Value}; -use cretonne_codegen::packed_option::PackedOption; -use cretonne_codegen::packed_option::ReservedValue; +use cranelift_codegen::cursor::{Cursor, FuncCursor}; +use cranelift_codegen::entity::{EntityMap, EntityRef, PrimaryMap}; +use cranelift_codegen::ir::immediates::{Ieee32, Ieee64}; +use cranelift_codegen::ir::instructions::BranchInfo; +use cranelift_codegen::ir::types::{F32, F64}; +use cranelift_codegen::ir::{Ebb, Function, Inst, InstBuilder, Type, Value}; +use cranelift_codegen::packed_option::PackedOption; +use cranelift_codegen::packed_option::ReservedValue; use std::mem; use std::u32; use std::vec::Vec; @@ -233,7 +233,7 @@ fn emit_zero(ty: Type, mut cur: FuncCursor) -> Value { } } /// The following methods are the API of the SSA builder. Here is how it should be used when -/// translating to Cretonne IR: +/// translating to Cranelift IR: /// /// - for each sequence of contiguous instructions (with no branches), create a corresponding /// basic block with `declare_ebb_body_block` or `declare_ebb_header_block` depending on the @@ -715,13 +715,13 @@ where #[cfg(test)] mod tests { - use cretonne_codegen::cursor::{Cursor, FuncCursor}; - use cretonne_codegen::entity::EntityRef; - use cretonne_codegen::ir::instructions::BranchInfo; - use cretonne_codegen::ir::types::*; - use cretonne_codegen::ir::{Function, Inst, InstBuilder, JumpTableData, Opcode}; - use cretonne_codegen::settings; - use cretonne_codegen::verify_function; + use cranelift_codegen::cursor::{Cursor, FuncCursor}; + use cranelift_codegen::entity::EntityRef; + use cranelift_codegen::ir::instructions::BranchInfo; + use cranelift_codegen::ir::types::*; + use cranelift_codegen::ir::{Function, Inst, InstBuilder, JumpTableData, Opcode}; + use cranelift_codegen::settings; + use cranelift_codegen::verify_function; use ssa::SSABuilder; use Variable; diff --git a/lib/frontend/src/variable.rs b/lib/frontend/src/variable.rs index 368d4bfb18..ad60006dd7 100644 --- a/lib/frontend/src/variable.rs +++ b/lib/frontend/src/variable.rs @@ -5,7 +5,7 @@ //! their own index types to use them directly. Frontends which don't //! can use the `Variable` defined here. -use cretonne_codegen::entity::EntityRef; +use cranelift_codegen::entity::EntityRef; use std::u32; ///! An opaque reference to a variable. diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml index 85bc72d9eb..c971224b46 100644 --- a/lib/module/Cargo.toml +++ b/lib/module/Cargo.toml @@ -1,24 +1,24 @@ [package] -name = "cretonne-module" +name = "cranelift-module" version = "0.13.0" -authors = ["The Cretonne Project Developers"] -description = "Support for linking functions and data with Cretonne" -repository = "https://github.com/cretonne/cretonne" -documentation = "https://cretonne.readthedocs.io/" +authors = ["The Cranelift Project Developers"] +description = "Support for linking functions and data with Cranelift" +repository = "https://github.com/cranelift/cranelift" +documentation = "https://cranelift.readthedocs.io/" license = "Apache-2.0 WITH LLVM-exception" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.13.0", default-features = false } -cretonne-entity = { path = "../entity", version = "0.13.0", default-features = false } +cranelift-codegen = { path = "../codegen", version = "0.13.0", default-features = false } +cranelift-entity = { path = "../entity", version = "0.13.0", default-features = false } hashmap_core = { version = "0.1.8", optional = true } failure = "0.1.1" [features] default = ["std"] -std = ["cretonne-codegen/std", "cretonne-entity/std"] -core = ["hashmap_core", "cretonne-codegen/core"] +std = ["cranelift-codegen/std", "cranelift-entity/std"] +core = ["hashmap_core", "cranelift-codegen/core"] [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "cretonne/cretonne" } +travis-ci = { repository = "cranelift/cranelift" } diff --git a/lib/module/README.md b/lib/module/README.md index 4ff52c3d2d..6bda716324 100644 --- a/lib/module/README.md +++ b/lib/module/README.md @@ -1,7 +1,7 @@ This crate provides the `Module` trait, which provides an interface for multiple functions and data to be emitted with -[Cretonne](https://crates.io/crates/cretonne) and then linked together. +[Cranelift](https://crates.io/crates/cranelift) and then linked together. -This crate is structured as an optional layer on top of cretonne-codegen. +This crate is structured as an optional layer on top of cranelift-codegen. It provides additional functionality, such as linking, however users that require greater flexibility don't need to use it. diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs index ef8eef9713..bc151efce9 100644 --- a/lib/module/src/backend.rs +++ b/lib/module/src/backend.rs @@ -1,8 +1,8 @@ //! Defines the `Backend` trait. -use cretonne_codegen::isa::TargetIsa; -use cretonne_codegen::Context; -use cretonne_codegen::{binemit, ir}; +use cranelift_codegen::isa::TargetIsa; +use cranelift_codegen::Context; +use cranelift_codegen::{binemit, ir}; use std::marker; use DataContext; use Linkage; diff --git a/lib/module/src/data_context.rs b/lib/module/src/data_context.rs index 9ecd29399f..e29389517b 100644 --- a/lib/module/src/data_context.rs +++ b/lib/module/src/data_context.rs @@ -1,8 +1,8 @@ //! Defines `DataContext`. -use cretonne_codegen::binemit::{Addend, CodeOffset}; -use cretonne_codegen::entity::PrimaryMap; -use cretonne_codegen::ir; +use cranelift_codegen::binemit::{Addend, CodeOffset}; +use cranelift_codegen::entity::PrimaryMap; +use cranelift_codegen::ir; use std::boxed::Box; use std::vec::Vec; @@ -59,7 +59,7 @@ pub struct DataDescription { pub data_relocs: Vec<(CodeOffset, ir::GlobalValue, Addend)>, } -/// This is to data objects what cretonne_codegen::Context is to functions. +/// This is to data objects what cranelift_codegen::Context is to functions. pub struct DataContext { description: DataDescription, } @@ -147,7 +147,7 @@ impl DataContext { #[cfg(test)] mod tests { - use cretonne_codegen::ir; + use cranelift_codegen::ir; use {DataContext, Init, Writability}; #[test] diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs index 4d251413d3..32b774824a 100644 --- a/lib/module/src/lib.rs +++ b/lib/module/src/lib.rs @@ -1,4 +1,4 @@ -//! Top-level lib.rs for `cretonne_module`. +//! Top-level lib.rs for `cranelift_module`. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] #![warn(unused_import_braces)] @@ -21,9 +21,9 @@ extern crate alloc; #[macro_use] -extern crate cretonne_codegen; +extern crate cranelift_codegen; #[macro_use] -extern crate cretonne_entity; +extern crate cranelift_entity; #[macro_use] extern crate failure; diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs index 216e3ed6e1..e5b7c81883 100644 --- a/lib/module/src/module.rs +++ b/lib/module/src/module.rs @@ -5,8 +5,8 @@ // TODO: Factor out `ir::Function`'s `ext_funcs` and `global_values` into a struct // shared with `DataContext`? -use cretonne_codegen::entity::{EntityRef, PrimaryMap}; -use cretonne_codegen::{binemit, ir, CodegenError, Context}; +use cranelift_codegen::entity::{EntityRef, PrimaryMap}; +use cranelift_codegen::{binemit, ir, CodegenError, Context}; use data_context::DataContext; use std::borrow::ToOwned; use std::collections::HashMap; @@ -131,7 +131,7 @@ pub enum ModuleError { /// Indicates an identifier was defined, but was declared as an import #[fail(display = "Invalid to define identifier declared as an import: {}", _0)] InvalidImportDefinition(String), - /// Wraps a `cretonne-codegen` error + /// Wraps a `cranelift-codegen` error #[fail(display = "Compilation error: {}", _0)] Compilation(CodegenError), /// Wraps a generic error from a backend diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml index 1c566349f3..ec7abc28b1 100644 --- a/lib/native/Cargo.toml +++ b/lib/native/Cargo.toml @@ -1,14 +1,14 @@ [package] -name = "cretonne-native" +name = "cranelift-native" version = "0.13.0" -authors = ["The Cretonne Project Developers"] -description = "Support for targeting the host with Cretonne" -repository = "https://github.com/cretonne/cretonne" +authors = ["The Cranelift Project Developers"] +description = "Support for targeting the host with Cranelift" +repository = "https://github.com/cranelift/cranelift" license = "Apache-2.0 WITH LLVM-exception" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.13.0", default-features = false } +cranelift-codegen = { path = "../codegen", version = "0.13.0", default-features = false } target-lexicon = { version = "0.0.2", default-features = false } [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies] @@ -16,12 +16,12 @@ raw-cpuid = "4" [features] default = ["std"] -std = ["cretonne-codegen/std", "target-lexicon/std"] +std = ["cranelift-codegen/std", "target-lexicon/std"] # when compiling with the "core" feature, nightly must be enabled # enabling the "nightly" feature for raw-cpuid allows avoiding # linking in a c-library. -core = ["cretonne-codegen/core", "raw-cpuid/nightly"] +core = ["cranelift-codegen/core", "raw-cpuid/nightly"] [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "cretonne/cretonne" } +travis-ci = { repository = "cranelift/cranelift" } diff --git a/lib/native/README.md b/lib/native/README.md index 2e91c82ab1..10f01bc906 100644 --- a/lib/native/README.md +++ b/lib/native/README.md @@ -1,3 +1,3 @@ This crate performs autodetection of the host architecture, which can be used to -configure [Cretonne](https://crates.io/crates/cretonne) to generate code +configure [Cranelift](https://crates.io/crates/cranelift) to generate code specialized for the machine it's running on. diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs index 9ab9df71c7..5439d743cc 100644 --- a/lib/native/src/lib.rs +++ b/lib/native/src/lib.rs @@ -1,5 +1,5 @@ //! Performs autodetection of the host for the purposes of running -//! Cretonne to generate code to run on the same machine. +//! Cranelift to generate code to run on the same machine. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] #![warn(unused_import_braces, unstable_features)] @@ -14,13 +14,13 @@ )] #![cfg_attr(not(feature = "std"), no_std)] -extern crate cretonne_codegen; +extern crate cranelift_codegen; #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] extern crate raw_cpuid; extern crate target_lexicon; -use cretonne_codegen::isa; -use cretonne_codegen::settings::{self, Configurable}; +use cranelift_codegen::isa; +use cranelift_codegen::settings::{self, Configurable}; use target_lexicon::Triple; #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml index 252587e5cf..837fe4abed 100644 --- a/lib/reader/Cargo.toml +++ b/lib/reader/Cargo.toml @@ -1,17 +1,17 @@ [package] -authors = ["The Cretonne Project Developers"] -name = "cretonne-reader" +authors = ["The Cranelift Project Developers"] +name = "cranelift-reader" version = "0.13.0" -description = "Cretonne textual IR reader" +description = "Cranelift textual IR reader" license = "Apache-2.0 WITH LLVM-exception" -documentation = "https://cretonne.readthedocs.io/" -repository = "https://github.com/cretonne/cretonne" +documentation = "https://cranelift.readthedocs.io/" +repository = "https://github.com/cranelift/cranelift" readme = "README.md" [dependencies] -cretonne-codegen = { path = "../codegen", version = "0.13.0" } +cranelift-codegen = { path = "../codegen", version = "0.13.0" } target-lexicon = "0.0.2" [badges] maintenance = { status = "experimental" } -travis-ci = { repository = "cretonne/cretonne" } +travis-ci = { repository = "cranelift/cranelift" } diff --git a/lib/reader/README.md b/lib/reader/README.md index 8430d11919..094cd92314 100644 --- a/lib/reader/README.md +++ b/lib/reader/README.md @@ -1,3 +1,3 @@ -This crate library supports reading .cton files. This functionality is needed -for testing [Cretonne](https://crates.io/crates/cretonne), but is not essential +This crate library supports reading .clif files. This functionality is needed +for testing [Cranelift](https://crates.io/crates/cranelift), but is not essential for a JIT compiler. diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs index 2998f17766..8a0e1f07b8 100644 --- a/lib/reader/src/isaspec.rs +++ b/lib/reader/src/isaspec.rs @@ -6,12 +6,12 @@ //! If a test case file contains `isa` commands, the tests will only be run against the specified //! ISAs. If the file contains no `isa` commands, the tests will be run against all supported ISAs. -use cretonne_codegen::isa::TargetIsa; -use cretonne_codegen::settings::{Configurable, Flags, SetError}; +use cranelift_codegen::isa::TargetIsa; +use cranelift_codegen::settings::{Configurable, Flags, SetError}; use error::{Location, ParseResult}; use testcommand::TestOption; -/// The ISA specifications in a `.cton` file. +/// The ISA specifications in a `.clif` file. pub enum IsaSpec { /// The parsed file does not contain any `isa` commands, but it may contain `set` commands /// which are reflected in the finished `Flags` object. diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs index 43bee2c399..41e1a37b8b 100644 --- a/lib/reader/src/lexer.rs +++ b/lib/reader/src/lexer.rs @@ -1,7 +1,7 @@ -//! Lexical analysis for .cton files. +//! Lexical analysis for .clif files. -use cretonne_codegen::ir::types; -use cretonne_codegen::ir::{Ebb, Value}; +use cranelift_codegen::ir::types; +use cranelift_codegen::ir::{Ebb, Value}; use error::Location; #[allow(unused_imports, deprecated)] use std::ascii::AsciiExt; @@ -479,8 +479,8 @@ impl<'a> Lexer<'a> { mod tests { use super::trailing_digits; use super::*; - use cretonne_codegen::ir::types; - use cretonne_codegen::ir::{Ebb, Value}; + use cranelift_codegen::ir::types; + use cranelift_codegen::ir::{Ebb, Value}; use error::Location; #[test] diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs index edff983292..edac07622f 100644 --- a/lib/reader/src/lib.rs +++ b/lib/reader/src/lib.rs @@ -1,7 +1,7 @@ -//! Cretonne file reader library. +//! Cranelift file reader library. //! -//! The `cretonne_reader` library supports reading .cton files. This functionality is needed for -//! testing Cretonne, but is not essential for a JIT compiler. +//! The `cranelift_reader` library supports reading .clif files. This functionality is needed for +//! testing Cranelift, but is not essential for a JIT compiler. #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)] #![warn(unused_import_braces, unstable_features)] @@ -15,7 +15,7 @@ ) )] -extern crate cretonne_codegen; +extern crate cranelift_codegen; extern crate target_lexicon; pub use error::{Location, ParseError, ParseResult}; diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs index e1cc270c60..91b98da1ae 100644 --- a/lib/reader/src/parser.rs +++ b/lib/reader/src/parser.rs @@ -1,21 +1,21 @@ -//! Parser for .cton files. +//! Parser for .clif files. -use cretonne_codegen::entity::EntityRef; -use cretonne_codegen::ir; -use cretonne_codegen::ir::entities::AnyEntity; -use cretonne_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32}; -use cretonne_codegen::ir::instructions::{InstructionData, InstructionFormat, VariableArgs}; -use cretonne_codegen::ir::types::VOID; -use cretonne_codegen::ir::{ +use cranelift_codegen::entity::EntityRef; +use cranelift_codegen::ir; +use cranelift_codegen::ir::entities::AnyEntity; +use cranelift_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32}; +use cranelift_codegen::ir::instructions::{InstructionData, InstructionFormat, VariableArgs}; +use cranelift_codegen::ir::types::VOID; +use cranelift_codegen::ir::{ AbiParam, ArgumentExtension, ArgumentLoc, Ebb, ExtFuncData, ExternalName, FuncRef, Function, GlobalValue, GlobalValueData, Heap, HeapBase, HeapData, HeapStyle, JumpTable, JumpTableData, MemFlags, Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind, Type, Value, ValueLoc, }; -use cretonne_codegen::isa::{self, Encoding, RegUnit, TargetIsa}; -use cretonne_codegen::packed_option::ReservedValue; -use cretonne_codegen::settings::CallConv; -use cretonne_codegen::{settings, timing}; +use cranelift_codegen::isa::{self, Encoding, RegUnit, TargetIsa}; +use cranelift_codegen::packed_option::ReservedValue; +use cranelift_codegen::settings::CallConv; +use cranelift_codegen::{settings, timing}; use error::{Location, ParseError, ParseResult}; use isaspec; use lexer::{LexError, Lexer, LocatedError, LocatedToken, Token}; @@ -2291,11 +2291,11 @@ impl<'a> Parser<'a> { #[cfg(test)] mod tests { use super::*; - use cretonne_codegen::ir::entities::AnyEntity; - use cretonne_codegen::ir::types; - use cretonne_codegen::ir::StackSlotKind; - use cretonne_codegen::ir::{ArgumentExtension, ArgumentPurpose}; - use cretonne_codegen::settings::CallConv; + use cranelift_codegen::ir::entities::AnyEntity; + use cranelift_codegen::ir::types; + use cranelift_codegen::ir::StackSlotKind; + use cranelift_codegen::ir::{ArgumentExtension, ArgumentPurpose}; + use cranelift_codegen::settings::CallConv; use error::ParseError; use isaspec::IsaSpec; use testfile::{Comment, Details}; diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs index b9fddde64f..17276d325d 100644 --- a/lib/reader/src/sourcemap.rs +++ b/lib/reader/src/sourcemap.rs @@ -6,8 +6,8 @@ //! The `SourceMap` struct defined in this module makes this mapping available //! to parser clients. -use cretonne_codegen::ir::entities::AnyEntity; -use cretonne_codegen::ir::{Ebb, FuncRef, GlobalValue, Heap, JumpTable, SigRef, StackSlot, Value}; +use cranelift_codegen::ir::entities::AnyEntity; +use cranelift_codegen::ir::{Ebb, FuncRef, GlobalValue, Heap, JumpTable, SigRef, StackSlot, Value}; use error::{Location, ParseResult}; use lexer::split_entity_name; use std::collections::HashMap; diff --git a/lib/reader/src/testcommand.rs b/lib/reader/src/testcommand.rs index 32220e006e..229957b17e 100644 --- a/lib/reader/src/testcommand.rs +++ b/lib/reader/src/testcommand.rs @@ -1,6 +1,6 @@ //! Test commands. //! -//! A `.cton` file can begin with one or more *test commands* which specify what is to be tested. +//! A `.clif` file can begin with one or more *test commands* which specify what is to be tested. //! The general syntax is: //! //! 
diff --git a/lib/reader/src/testfile.rs b/lib/reader/src/testfile.rs
index 0c591cea26..685b7d99e6 100644
--- a/lib/reader/src/testfile.rs
+++ b/lib/reader/src/testfile.rs
@@ -1,11 +1,11 @@
 //! Data structures representing a parsed test file.
 //!
-//! A test file is a `.cton` file which contains test commands and settings for running a
+//! A test file is a `.clif` file which contains test commands and settings for running a
 //! file-based test case.
 //!
 
-use cretonne_codegen::ir::entities::AnyEntity;
-use cretonne_codegen::ir::Function;
+use cranelift_codegen::ir::entities::AnyEntity;
+use cranelift_codegen::ir::Function;
 use error::Location;
 use isaspec::IsaSpec;
 use sourcemap::SourceMap;
@@ -13,7 +13,7 @@ use testcommand::TestCommand;
 
 /// A parsed test case.
 ///
-/// This is the result of parsing a `.cton` file which contains a number of test commands and ISA
+/// This is the result of parsing a `.clif` file which contains a number of test commands and ISA
 /// specs followed by the functions that should be tested.
 pub struct TestFile<'a> {
     /// `test foo ...` lines.
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 06f9f1b33c..1bbad6a2a5 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,17 +1,17 @@
 [package]
-name = "cretonne-simplejit"
+name = "cranelift-simplejit"
 version = "0.13.0"
-authors = ["The Cretonne Project Developers"]
-description = "A simple JIT library backed by Cretonne"
-repository = "https://github.com/cretonne/cretonne"
-documentation = "https://cretonne.readthedocs.io/"
+authors = ["The Cranelift Project Developers"]
+description = "A simple JIT library backed by Cranelift"
+repository = "https://github.com/cranelift/cranelift"
+documentation = "https://cranelift.readthedocs.io/"
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cretonne-codegen = { path = "../codegen", version = "0.13.0", default-features = false }
-cretonne-module = { path = "../module", version = "0.13.0", default-features = false }
-cretonne-native = { path = "../native", version = "0.13.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.13.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.13.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.13.0", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.40", default-features = false }
 errno = "0.2.3"
@@ -22,9 +22,9 @@ winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [features]
 default = ["std"]
-std = ["libc/use_std", "cretonne-codegen/std", "cretonne-module/std", "cretonne-native/std", "target-lexicon/std"]
-core = ["cretonne-codegen/core", "cretonne-module/core", "cretonne-native/core"]
+std = ["libc/use_std", "cranelift-codegen/std", "cranelift-module/std", "cranelift-native/std", "target-lexicon/std"]
+core = ["cranelift-codegen/core", "cranelift-module/core", "cranelift-native/core"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "cretonne/cretonne" }
+travis-ci = { repository = "cranelift/cranelift" }
diff --git a/lib/simplejit/README.md b/lib/simplejit/README.md
index 8b0aea6d28..d5f2cecc37 100644
--- a/lib/simplejit/README.md
+++ b/lib/simplejit/README.md
@@ -1,4 +1,4 @@
 This crate provides a simple JIT library that uses
-[Cretonne](https://crates.io/crates/cretonne).
+[Cranelift](https://crates.io/crates/cranelift).
 
 This crate is extremely experimental.
diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index 29c9c26a32..e78c9d1124 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -1,13 +1,13 @@
 //! Defines `SimpleJITBackend`.
 
-use cretonne_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSink};
-use cretonne_codegen::isa::TargetIsa;
-use cretonne_codegen::{self, ir, settings};
-use cretonne_module::{
+use cranelift_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSink};
+use cranelift_codegen::isa::TargetIsa;
+use cranelift_codegen::{self, ir, settings};
+use cranelift_module::{
     Backend, DataContext, DataDescription, Init, Linkage, ModuleNamespace, ModuleResult,
     Writability,
 };
-use cretonne_native;
+use cranelift_native;
 use libc;
 use memory::Memory;
 use std::ffi::CString;
@@ -24,7 +24,7 @@ pub struct SimpleJITBuilder {
 impl SimpleJITBuilder {
     /// Create a new `SimpleJITBuilder`.
     pub fn new() -> Self {
-        let (flag_builder, isa_builder) = cretonne_native::builders().unwrap_or_else(|_| {
+        let (flag_builder, isa_builder) = cranelift_native::builders().unwrap_or_else(|_| {
             panic!("host machine is not a supported target");
         });
         let isa = isa_builder.finish(settings::Flags::new(flag_builder));
@@ -117,7 +117,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
     fn define_function(
         &mut self,
         _name: &str,
-        ctx: &cretonne_codegen::Context,
+        ctx: &cranelift_codegen::Context,
         _namespace: &ModuleNamespace,
         code_size: u32,
     ) -> ModuleResult {
diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs
index 0e908dd414..01dd0938f0 100644
--- a/lib/simplejit/src/lib.rs
+++ b/lib/simplejit/src/lib.rs
@@ -1,4 +1,4 @@
-//! Top-level lib.rs for `cretonne_simplejit`.
+//! Top-level lib.rs for `cranelift_simplejit`.
 
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces, unstable_features)]
@@ -12,9 +12,9 @@
     )
 )]
 
-extern crate cretonne_codegen;
-extern crate cretonne_module;
-extern crate cretonne_native;
+extern crate cranelift_codegen;
+extern crate cranelift_module;
+extern crate cranelift_native;
 extern crate errno;
 extern crate libc;
 extern crate region;
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index ef3b6a9db5..0650d3da63 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,23 +1,23 @@
 [package]
-authors = ["The Cretonne Project Developers"]
-name = "cretonne"
+authors = ["The Cranelift Project Developers"]
+name = "cranelift"
 version = "0.13.0"
-description = "Umbrella for commonly-used cretonne crates"
+description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
-documentation = "https://cretonne.readthedocs.io/"
-repository = "https://github.com/cretonne/cretonne"
+documentation = "https://cranelift.readthedocs.io/"
+repository = "https://github.com/cranelift/cranelift"
 readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cretonne-codegen = { path = "../codegen", version = "0.13.0", default-features = false }
-cretonne-frontend = { path = "../frontend", version = "0.13.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.13.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.13.0", default-features = false }
 
 [features]
 default = ["std"]
-std = ["cretonne-codegen/std", "cretonne-frontend/std"]
-core = ["cretonne-codegen/core", "cretonne-frontend/core"]
+std = ["cranelift-codegen/std", "cranelift-frontend/std"]
+core = ["cranelift-codegen/core", "cranelift-frontend/core"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "cretonne/cretonne" }
+travis-ci = { repository = "cranelift/cranelift" }
diff --git a/lib/umbrella/README.md b/lib/umbrella/README.md
index 96320ade2e..134b3140bf 100644
--- a/lib/umbrella/README.md
+++ b/lib/umbrella/README.md
@@ -1,3 +1,3 @@
 This is an umbrella crate which contains no code of its own, but pulls in
-other cretonne library crates to provide a convenient one-line dependency,
+other cranelift library crates to provide a convenient one-line dependency,
 and a prelude, for common use cases.
diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs
index c586227df2..ad241483e7 100644
--- a/lib/umbrella/src/lib.rs
+++ b/lib/umbrella/src/lib.rs
@@ -1,4 +1,4 @@
-//! Cretonne umbrella crate, providing a convenient one-line dependency.
+//! Cranelift umbrella crate, providing a convenient one-line dependency.
 
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces, unstable_features)]
@@ -13,11 +13,11 @@
 )]
 
 /// Provide these crates, renamed to reduce stutter.
-pub extern crate cretonne_codegen as codegen;
-pub extern crate cretonne_frontend as frontend;
+pub extern crate cranelift_codegen as codegen;
+pub extern crate cranelift_frontend as frontend;
 
-/// A prelude providing convenient access to commonly-used cretonne features. Use
-/// as `use cretonne::prelude::*`.
+/// A prelude providing convenient access to commonly-used cranelift features. Use
+/// as `use cranelift::prelude::*`.
 pub mod prelude {
     pub use codegen;
     pub use codegen::entity::EntityRef;
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 8db1c1665e..058b2e44df 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,17 +1,17 @@
 [package]
-name = "cretonne-wasm"
+name = "cranelift-wasm"
 version = "0.13.0"
-authors = ["The Cretonne Project Developers"]
-description = "Translator from WebAssembly to Cretonne IR"
-repository = "https://github.com/cretonne/cretonne"
+authors = ["The Cranelift Project Developers"]
+description = "Translator from WebAssembly to Cranelift IR"
+repository = "https://github.com/cranelift/cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.0", default-features = false }
-cretonne-codegen = { path = "../codegen", version = "0.13.0", default-features = false }
-cretonne-frontend = { path = "../frontend", version = "0.13.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.13.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.13.0", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
@@ -22,9 +22,9 @@ wabt = "0.4"
 
 [features]
 default = ["std"]
-std = ["cretonne-codegen/std", "cretonne-frontend/std", "wasmparser/std", "target-lexicon/std"]
-core = ["hashmap_core", "cretonne-codegen/core", "cretonne-frontend/core", "wasmparser/core"]
+std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std", "target-lexicon/std"]
+core = ["hashmap_core", "cranelift-codegen/core", "cranelift-frontend/core", "wasmparser/core"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "cretonne/cretonne" }
+travis-ci = { repository = "cranelift/cranelift" }
diff --git a/lib/wasm/README.md b/lib/wasm/README.md
index 946a41f26d..981ab41423 100644
--- a/lib/wasm/README.md
+++ b/lib/wasm/README.md
@@ -1,2 +1,2 @@
 This crate performs the translation from a wasm module in binary format to the
-in-memory form of the [Cretonne](https://crates.io/crates/cretonne) IR.
+in-memory form of the [Cranelift](https://crates.io/crates/cranelift) IR.
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index dc9be39fa5..578ba9dfcb 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -1,5 +1,5 @@
 //! This module contains the bulk of the interesting code performing the translation between
-//! WebAssembly and Cretonne IR.
+//! WebAssembly and Cranelift IR.
 //!
 //! The translation is done in one pass, opcode by opcode. Two main data structures are used during
 //! code translations: the value stack and the control stack. The value stack mimics the execution
@@ -22,11 +22,11 @@
 //!
 //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as
 //! argument.
-use cretonne_codegen::ir::condcodes::{FloatCC, IntCC};
-use cretonne_codegen::ir::types::*;
-use cretonne_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags};
-use cretonne_codegen::packed_option::ReservedValue;
-use cretonne_frontend::{FunctionBuilder, Variable};
+use cranelift_codegen::ir::condcodes::{FloatCC, IntCC};
+use cranelift_codegen::ir::types::*;
+use cranelift_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags};
+use cranelift_codegen::packed_option::ReservedValue;
+use cranelift_frontend::{FunctionBuilder, Variable};
 use environ::{FuncEnvironment, GlobalVariable, WasmError, WasmResult};
 use state::{ControlStackFrame, TranslationState};
 use std::collections::{hash_map, HashMap};
@@ -38,7 +38,7 @@ use wasmparser::{MemoryImmediate, Operator};
 
 // Clippy warns about "flags: _" but its important to document that the flags field is ignored
 #[cfg_attr(feature = "cargo-clippy", allow(unneeded_field_pattern))]
-/// Translates wasm operators into Cretonne IR instructions. Returns `true` if it inserted
+/// Translates wasm operators into Cranelift IR instructions. Returns `true` if it inserted
 /// a return.
 pub fn translate_operator(
     op: Operator,
@@ -55,7 +55,7 @@ pub fn translate_operator(
     match op {
         /********************************** Locals ****************************************
          *  `get_local` and `set_local` are treated as non-SSA variables and will completely
-         *  disappear in the Cretonne Code
+         *  disappear in the Cranelift Code
          ***********************************************************************************/
         Operator::GetLocal { local_index } => {
             state.push1(builder.use_var(Variable::with_u32(local_index)))
@@ -218,7 +218,7 @@ pub fn translate_operator(
          * Once the destination `Ebb` is found, we sometimes have to declare a certain depth
          * of the stack unreachable, because some branch instructions are terminator.
          *
-         * The `br_table` case is much more complicated because Cretonne's `br_table` instruction
+         * The `br_table` case is much more complicated because Cranelift's `br_table` instruction
          * does not support jump arguments like all the other branch instructions. That is why, in
          * the case where we would use jump arguments for every other branch instructions, we
          * need to split the critical edges leaving the `br_tables` by creating one `Ebb` per
@@ -226,7 +226,7 @@ pub fn translate_operator(
          * `Ebb`s contain only a jump instruction pointing to the final destination, this time with
          * jump arguments.
          *
-         * This system is also implemented in Cretonne's SSA construction algorithm, because
+         * This system is also implemented in Cranelift's SSA construction algorithm, because
          * `use_var` located in a destination `Ebb` of a `br_table` might trigger the addition
          * of jump arguments in each predecessor branch instruction, one of which might be a
          * `br_table`.
@@ -291,7 +291,7 @@ pub fn translate_operator(
                 };
                 builder.ins().jump(ebb, &[]);
             } else {
-                // Here we have jump arguments, but Cretonne's br_table doesn't support them
+                // Here we have jump arguments, but Cranelift's br_table doesn't support them
                 // We then proceed to split the edges going out of the br_table
                 let return_count = jump_args_count;
                 let mut dest_ebb_sequence = Vec::new();
@@ -413,7 +413,7 @@ pub fn translate_operator(
             state.push1(environ.translate_memory_size(builder.cursor(), heap_index, heap)?);
         }
         /******************************* Load instructions ***********************************
-         * Wasm specifies an integer alignment flag but we drop it in Cretonne.
+         * Wasm specifies an integer alignment flag but we drop it in Cranelift.
          * The memory base address is provided by the environment.
          * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not
          ************************************************************************************/
@@ -488,7 +488,7 @@ pub fn translate_operator(
             translate_load(offset, ir::Opcode::Load, F64, builder, state, environ);
         }
         /****************************** Store instructions ***********************************
-         * Wasm specifies an integer alignment flag but we drop it in Cretonne.
+         * Wasm specifies an integer alignment flag but we drop it in Cranelift.
          * The memory base address is provided by the environment.
          * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not
          ************************************************************************************/
@@ -1027,7 +1027,7 @@ fn translate_load(
     let (base, offset) = get_heap_addr(heap, addr32, offset, environ.native_pointer(), builder);
     // Note that we don't set `is_aligned` here, even if the load instruction's
     // alignment immediate says it's aligned, because WebAssembly's immediate
-    // field is just a hint, while Cretonne's aligned flag needs a guarantee.
+    // field is just a hint, while Cranelift's aligned flag needs a guarantee.
     let flags = MemFlags::new();
     let (load, dfg) = builder
         .ins()
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 912c39b107..37013cad0a 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -1,9 +1,9 @@
 //! "Dummy" environment for testing wasm translation.
 
-use cretonne_codegen::cursor::FuncCursor;
-use cretonne_codegen::ir::types::*;
-use cretonne_codegen::ir::{self, InstBuilder};
-use cretonne_codegen::settings;
+use cranelift_codegen::cursor::FuncCursor;
+use cranelift_codegen::ir::types::*;
+use cranelift_codegen::ir::{self, InstBuilder};
+use cranelift_codegen::settings;
 use environ::{FuncEnvironment, GlobalVariable, ModuleEnvironment, WasmResult};
 use func_translator::FuncTranslator;
 use std::string::String;
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 959a411899..7368b60e58 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -1,8 +1,8 @@
-//! All the runtime support necessary for the wasm to cretonne translation is formalized by the
+//! All the runtime support necessary for the wasm to cranelift translation is formalized by the
 //! traits `FunctionEnvironment` and `ModuleEnvironment`.
-use cretonne_codegen::cursor::FuncCursor;
-use cretonne_codegen::ir::{self, InstBuilder};
-use cretonne_codegen::settings::Flags;
+use cranelift_codegen::cursor::FuncCursor;
+use cranelift_codegen::ir::{self, InstBuilder};
+use cranelift_codegen::settings::Flags;
 use std::vec::Vec;
 use target_lexicon::Triple;
 use translation_utils::{
@@ -51,10 +51,10 @@ pub enum WasmError {
 
     /// An implementation limit was exceeded.
     ///
-    /// Cretonne can compile very large and complicated functions, but the [implementation has
+    /// Cranelift can compile very large and complicated functions, but the [implementation has
     /// limits][limits] that cause compilation to fail when they are exceeded.
     ///
-    /// [limits]: https://cretonne.readthedocs.io/en/latest/langref.html#implementation-limits
+    /// [limits]: https://cranelift.readthedocs.io/en/latest/langref.html#implementation-limits
     #[fail(display = "Implementation limit exceeded")]
     ImplLimitExceeded,
 }
@@ -72,7 +72,7 @@ pub type WasmResult = Result;
 
 /// Environment affecting the translation of a single WebAssembly function.
 ///
-/// A `FuncEnvironment` trait object is required to translate a WebAssembly function to Cretonne
+/// A `FuncEnvironment` trait object is required to translate a WebAssembly function to Cranelift
 /// IR. The function environment provides information about the WebAssembly module as well as the
 /// runtime environment.
 pub trait FuncEnvironment {
@@ -82,7 +82,7 @@ pub trait FuncEnvironment {
     /// Get the flags for the current compilation.
     fn flags(&self) -> &Flags;
 
-    /// Get the Cretonne integer type to use for native pointers.
+    /// Get the Cranelift integer type to use for native pointers.
     ///
     /// This returns `I64` for 64-bit architectures and `I32` for 32-bit architectures.
     fn native_pointer(&self) -> ir::Type {
@@ -204,7 +204,7 @@ pub trait FuncEnvironment {
 
 /// An object satisfying the `ModuleEnvironment` trait can be passed as argument to the
 /// [`translate_module`](fn.translate_module.html) function. These methods should not be called
-/// by the user, they are only for `cretonne-wasm` internal use.
+/// by the user, they are only for `cranelift-wasm` internal use.
 pub trait ModuleEnvironment<'data> {
     /// Get the flags for the current compilation.
     fn flags(&self) -> &Flags;
diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs
index 105cb190a8..01481e9b8f 100644
--- a/lib/wasm/src/func_translator.rs
+++ b/lib/wasm/src/func_translator.rs
@@ -1,21 +1,21 @@
-//! Stand-alone WebAssembly to Cretonne IR translator.
+//! Stand-alone WebAssembly to Cranelift IR translator.
 //!
 //! This module defines the `FuncTranslator` type which can translate a single WebAssembly
-//! function to Cretonne IR guided by a `FuncEnvironment` which provides information about the
+//! function to Cranelift IR guided by a `FuncEnvironment` which provides information about the
 //! WebAssembly module and the runtime environment.
 
 use code_translator::translate_operator;
-use cretonne_codegen::entity::EntityRef;
-use cretonne_codegen::ir::{self, Ebb, InstBuilder};
-use cretonne_codegen::timing;
-use cretonne_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
+use cranelift_codegen::entity::EntityRef;
+use cranelift_codegen::ir::{self, Ebb, InstBuilder};
+use cranelift_codegen::timing;
+use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
 use environ::{FuncEnvironment, WasmError, WasmResult};
 use state::TranslationState;
 use wasmparser::{self, BinaryReader};
 
-/// WebAssembly to Cretonne IR function translator.
+/// WebAssembly to Cranelift IR function translator.
 ///
-/// A `FuncTranslator` is used to translate a binary WebAssembly function into Cretonne IR guided
+/// A `FuncTranslator` is used to translate a binary WebAssembly function into Cranelift IR guided
 /// by a `FuncEnvironment` object. A single translator instance can be reused to translate multiple
 /// functions which will reduce heap allocation traffic.
 pub struct FuncTranslator {
@@ -45,7 +45,7 @@ impl FuncTranslator {
     ///
     /// [wasm]: https://webassembly.github.io/spec/binary/modules.html#code-section
     ///
-    /// The Cretonne IR function `func` should be completely empty except for the `func.signature`
+    /// The Cranelift IR function `func` should be completely empty except for the `func.signature`
     /// and `func.name` fields. The signature may contain special-purpose arguments which are not
     /// regarded as WebAssembly local variables. Any signature arguments marked as
     /// `ArgumentPurpose::Normal` are made accessible as WebAssembly local variables.
@@ -226,7 +226,7 @@ fn parse_function_body(
 fn cur_srcloc(reader: &BinaryReader) -> ir::SourceLoc {
     // We record source locations as byte code offsets relative to the beginning of the function.
     // This will wrap around of a single function's byte code is larger than 4 GB, but a) the
-    // WebAssembly format doesn't allow for that, and b) that would hit other Cretonne
+    // WebAssembly format doesn't allow for that, and b) that would hit other Cranelift
     // implementation limits anyway.
     ir::SourceLoc::new(reader.current_position() as u32)
 }
@@ -234,8 +234,8 @@ fn cur_srcloc(reader: &BinaryReader) -> ir::SourceLoc {
 #[cfg(test)]
 mod tests {
     use super::FuncTranslator;
-    use cretonne_codegen::ir::types::I32;
-    use cretonne_codegen::{ir, Context};
+    use cranelift_codegen::ir::types::I32;
+    use cranelift_codegen::{ir, Context};
     use environ::{DummyEnvironment, FuncEnvironment};
     use target_lexicon::Triple;
 
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index 70826f72fb..995adf10de 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -1,5 +1,5 @@
 //! Performs translation from a wasm module in binary format to the in-memory form
-//! of Cretonne IR. More particularly, it translates the code of all the functions bodies and
+//! of Cranelift IR. More particularly, it translates the code of all the functions bodies and
 //! interacts with an environment implementing the
 //! [`ModuleEnvironment`](trait.ModuleEnvironment.html)
 //! trait to deal with tables, globals and linear memory.
@@ -25,8 +25,8 @@
 #![cfg_attr(not(feature = "std"), feature(alloc))]
 
 #[macro_use(dbg)]
-extern crate cretonne_codegen;
-extern crate cretonne_frontend;
+extern crate cranelift_codegen;
+extern crate cranelift_frontend;
 extern crate target_lexicon;
 extern crate wasmparser;
 
diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs
index fcbcd51708..5010aba87f 100644
--- a/lib/wasm/src/module_translator.rs
+++ b/lib/wasm/src/module_translator.rs
@@ -1,6 +1,6 @@
 //! Translation skeleton that traverses the whole WebAssembly module and call helper functions
 //! to deal with each part of it.
-use cretonne_codegen::timing;
+use cranelift_codegen::timing;
 use environ::{ModuleEnvironment, WasmError, WasmResult};
 use sections_translator::{
     parse_data_section, parse_elements_section, parse_export_section, parse_function_section,
@@ -9,7 +9,7 @@ use sections_translator::{
 };
 use wasmparser::{Parser, ParserInput, ParserState, SectionCode, WasmDecoder};
 
-/// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cretonne IR
+/// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cranelift IR
 /// [`Function`](../codegen/ir/function/struct.Function.html).
 /// Returns the functions and also the mappings for imported functions and signature between the
 /// indexes in the wasm module and the indexes inside each functions.
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index c21dbb0c25..c17b0e680b 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -7,7 +7,7 @@
 //! The special case of the initialize expressions for table elements offsets or global variables
 //! is handled, according to the semantics of WebAssembly, to only specific expressions that are
 //! interpreted on the fly.
-use cretonne_codegen::ir::{self, AbiParam, Signature};
+use cranelift_codegen::ir::{self, AbiParam, Signature};
 use environ::{ModuleEnvironment, WasmError, WasmResult};
 use std::str::from_utf8;
 use std::vec::Vec;
diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs
index d301d87348..c08c514d3a 100644
--- a/lib/wasm/src/state.rs
+++ b/lib/wasm/src/state.rs
@@ -3,7 +3,7 @@
 //! The `TranslationState` struct defined in this module is used to keep track of the WebAssembly
 //! value and control stacks during the translation of a single function.
 
-use cretonne_codegen::ir::{self, Ebb, Inst, Value};
+use cranelift_codegen::ir::{self, Ebb, Inst, Value};
 use environ::{FuncEnvironment, GlobalVariable};
 use std::collections::HashMap;
 use std::vec::Vec;
diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index 16b5737b80..e6822af761 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -1,5 +1,5 @@
 //! Helper functions and structures for the translation.
-use cretonne_codegen::ir;
+use cranelift_codegen::ir;
 use std::u32;
 use wasmparser;
 
@@ -71,7 +71,7 @@ pub struct Memory {
     pub shared: bool,
 }
 
-/// Helper function translating wasmparser types to Cretonne types when possible.
+/// Helper function translating wasmparser types to Cranelift types when possible.
 pub fn type_to_type(ty: wasmparser::Type) -> Result {
     match ty {
         wasmparser::Type::I32 => Ok(ir::types::I32),
@@ -82,17 +82,17 @@ pub fn type_to_type(ty: wasmparser::Type) -> Result {
     }
 }
 
-/// Turns a `wasmparser` `f32` into a `Cretonne` one.
+/// Turns a `wasmparser` `f32` into a `Cranelift` one.
 pub fn f32_translation(x: wasmparser::Ieee32) -> ir::immediates::Ieee32 {
     ir::immediates::Ieee32::with_bits(x.bits())
 }
 
-/// Turns a `wasmparser` `f64` into a `Cretonne` one.
+/// Turns a `wasmparser` `f64` into a `Cranelift` one.
 pub fn f64_translation(x: wasmparser::Ieee64) -> ir::immediates::Ieee64 {
     ir::immediates::Ieee64::with_bits(x.bits())
 }
 
-/// Translate a `wasmparser` type into its `Cretonne` equivalent, when possible
+/// Translate a `wasmparser` type into its `Cranelift` equivalent, when possible
 pub fn num_return_values(ty: wasmparser::Type) -> usize {
     match ty {
         wasmparser::Type::EmptyBlockType => 0,
diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs
index 3f0ef726cd..d72ba4afa3 100644
--- a/lib/wasm/tests/wasm_testsuite.rs
+++ b/lib/wasm/tests/wasm_testsuite.rs
@@ -1,13 +1,13 @@
-extern crate cretonne_codegen;
-extern crate cretonne_wasm;
+extern crate cranelift_codegen;
+extern crate cranelift_wasm;
 #[macro_use]
 extern crate target_lexicon;
 extern crate wabt;
 
-use cretonne_codegen::print_errors::pretty_verifier_error;
-use cretonne_codegen::settings::{self, Configurable, Flags};
-use cretonne_codegen::verifier;
-use cretonne_wasm::{translate_module, DummyEnvironment};
+use cranelift_codegen::print_errors::pretty_verifier_error;
+use cranelift_codegen::settings::{self, Configurable, Flags};
+use cranelift_codegen::verifier;
+use cranelift_wasm::{translate_module, DummyEnvironment};
 use std::fs;
 use std::fs::File;
 use std::io;
diff --git a/misc/vim/ftdetect/clif.vim b/misc/vim/ftdetect/clif.vim
new file mode 100644
index 0000000000..58312284a1
--- /dev/null
+++ b/misc/vim/ftdetect/clif.vim
@@ -0,0 +1 @@
+au BufRead,BufNewFile *.clif set filetype=clif
diff --git a/misc/vim/ftdetect/cton.vim b/misc/vim/ftdetect/cton.vim
deleted file mode 100644
index 9d7754c472..0000000000
--- a/misc/vim/ftdetect/cton.vim
+++ /dev/null
@@ -1 +0,0 @@
-au BufRead,BufNewFile *.cton set filetype=cton
diff --git a/misc/vim/syntax/clif.vim b/misc/vim/syntax/clif.vim
new file mode 100644
index 0000000000..6636abc220
--- /dev/null
+++ b/misc/vim/syntax/clif.vim
@@ -0,0 +1,44 @@
+" Vim syntax file
+" Language:     Cranelift
+" Maintainer:   Jakob Stoklund Olesen /
+syn match clifEntity /\<\(v\|ss\|jt\|fn\|sig\)\d\+\>/
+syn match clifLabel /\/
+syn match clifName /%\w\+\>/
+
+syn match clifNumber /-\?\<[0-9_]\+\>/
+syn match clifNumber /-\?\<0x[0-9a-fA-F_]\+\(\.[0-9a-fA-F_]*\)\?\(p[+-]\?\d\+\)\?\>/
+syn match clifHexSeq /#\x\+\>/
+syn match clifSourceLoc /@[0-9a-f]\+\>/
+
+syn region clifCommentLine start=";" end="$" contains=clifFilecheck
+
+hi def link clifHeader        Keyword
+hi def link clifDecl          Keyword
+hi def link clifType          Type
+hi def link clifEntity        Identifier
+hi def link clifLabel         Label
+hi def link clifName          String
+hi def link clifNumber        Number
+hi def link clifHexSeq        Number
+hi def link clifCommentLine   Comment
+hi def link clifFilecheck     SpecialComment
+hi def link clifSourceLoc     LineNr
+
+let b:current_syntax = "clif"
diff --git a/misc/vim/syntax/cton.vim b/misc/vim/syntax/cton.vim
deleted file mode 100644
index 4862c38ec8..0000000000
--- a/misc/vim/syntax/cton.vim
+++ /dev/null
@@ -1,44 +0,0 @@
-" Vim syntax file
-" Language:     Cretonne
-" Maintainer:   Jakob Stoklund Olesen /
-syn match ctonEntity /\<\(v\|ss\|jt\|fn\|sig\)\d\+\>/
-syn match ctonLabel /\/
-syn match ctonName /%\w\+\>/
-
-syn match ctonNumber /-\?\<[0-9_]\+\>/
-syn match ctonNumber /-\?\<0x[0-9a-fA-F_]\+\(\.[0-9a-fA-F_]*\)\?\(p[+-]\?\d\+\)\?\>/
-syn match ctonHexSeq /#\x\+\>/
-syn match ctonSourceLoc /@[0-9a-f]\+\>/
-
-syn region ctonCommentLine start=";" end="$" contains=ctonFilecheck
-
-hi def link ctonHeader        Keyword
-hi def link ctonDecl          Keyword
-hi def link ctonType          Type
-hi def link ctonEntity        Identifier
-hi def link ctonLabel         Label
-hi def link ctonName          String
-hi def link ctonNumber        Number
-hi def link ctonHexSeq        Number
-hi def link ctonCommentLine   Comment
-hi def link ctonFilecheck     SpecialComment
-hi def link ctonSourceLoc     LineNr
-
-let b:current_syntax = "cton"
diff --git a/rustc.rst b/rustc.rst
index 20cebeb5d6..b7c01f8669 100644
--- a/rustc.rst
+++ b/rustc.rst
@@ -1,16 +1,16 @@
 =================
-Cretonne in Rustc
+Cranelift in Rustc
 =================
 
-One goal for Cretonne is to be usable as a backend suitable for compiling Rust
+One goal for Cranelift is to be usable as a backend suitable for compiling Rust
 in debug mode. This mode doesn't require a lot of mid-level optimization, and it
 does want very fast compile times, and this matches up fairly well with what we
-expect Cretonne's initial strengths and weaknesses will be. Cretonne is being
+expect Cranelift's initial strengths and weaknesses will be. Cranelift is being
 designed to take aggressive advantage of multiple cores, and to be very efficient
 with its use of memory.
 
 Another goal is a "pretty good" backend. The idea here is to do the work to get
-MIR-level inlining enabled, do some basic optimizations in Cretonne to capture the
+MIR-level inlining enabled, do some basic optimizations in Cranelift to capture the
 low-hanging fruit, and then use that along with good low-level optimizations to
 produce code which has a chance of being decently fast, with quite fast compile
 times. It obviously wouldn't compete with LLVM-based release builds in terms of
@@ -22,7 +22,7 @@ enabled a Rust compiler written entirely in Rust, and enabled faster Rust compil
 times for important use cases.
 
 With all that said, there is a potential goal beyond that, which is to build a
-full optimizing release-capable backend. We can't predict how far Cretonne will go
+full optimizing release-capable backend. We can't predict how far Cranelift will go
 yet, but we do have some crazy ideas about what such a thing might look like,
 including:
 
@@ -44,8 +44,8 @@ including:
   less manual effort.
 
 - Build an optimizer IR without the constraints of fast-debug-build compilation.
-  Cretonne's base IR is focused on Codegen, so a full-strength optimizer would either
-  use an IR layer on top of it (possibly using Cretonne's flexible EntityMap system),
+  Cranelift's base IR is focused on Codegen, so a full-strength optimizer would either
+  use an IR layer on top of it (possibly using Cranelift's flexible EntityMap system),
   or possibly an independent IR that could be translated to/from the base IR. Either
   way, this overall architecture would keep the optimizer out of the way of the
   non-optimizing build path, which keeps that path fast and simple, and gives the
diff --git a/spidermonkey.rst b/spidermonkey.rst
index 57cbccc03e..874634ea56 100644
--- a/spidermonkey.rst
+++ b/spidermonkey.rst
@@ -1,9 +1,9 @@
 ========================
-Cretonne in SpiderMonkey
+Cranelift in SpiderMonkey
 ========================
 
 `SpiderMonkey `_ is the
-JavaScript and WebAssembly engine in Firefox. Cretonne is designed to be used in SpiderMonkey with
+JavaScript and WebAssembly engine in Firefox. Cranelift is designed to be used in SpiderMonkey with
 the goal of enabling better code generation for ARM's 32-bit and 64-bit architectures, and building
 a framework for improved low-level code optimizations in the future.
 
@@ -17,11 +17,11 @@ allocation.
 .. image:: media/spidermonkey1.png
     :align: center
     :width: 80%
-    :alt: Cretonne in SpiderMonkey phase 1
+    :alt: Cranelift in SpiderMonkey phase 1
 
-In phase 1, Cretonne aims to replace the IonMonkey-based tier 2 compiler for WebAssembly only. It
+In phase 1, Cranelift aims to replace the IonMonkey-based tier 2 compiler for WebAssembly only. It
 will still be orchestrated by the BaldrMonkey engine and compile WebAssembly modules on multiple
-threads. Cretonne translates binary wasm functions directly into its own intermediate
+threads. Cranelift translates binary wasm functions directly into its own intermediate
 representation, and it generates binary machine code without depending on SpiderMonkey's macro
 assembler.
 
@@ -38,7 +38,7 @@ intermediate representations to do that:
 .. image:: media/spidermonkey2.png
     :align: center
     :width: 80%
-    :alt: Cretonne in SpiderMonkey phase 2
+    :alt: Cranelift in SpiderMonkey phase 2
 
-Cretonne has its own register allocator, so the LIR representation can be skipped when using
-Cretonne as a backend for IonMonkey.
+Cranelift has its own register allocator, so the LIR representation can be skipped when using
+Cranelift as a backend for IonMonkey.

From 401c872c52dc6a93597d559932e09f4621e02dd4 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 13 Jul 2018 09:22:15 -0700
Subject: [PATCH 1900/3084] Update more paths for the CraneStation/cranelift
 rename.

---
 cranelift/Cargo.toml                             | 2 +-
 cranelift/filetests/regalloc/coalescing-207.clif | 2 +-
 cranelift/filetests/regalloc/coalescing-216.clif | 2 +-
 cranelift/filetests/regalloc/reload-208.clif     | 2 +-
 cranelift/publish-all.sh                         | 2 +-
 lib/codegen/Cargo.toml                           | 4 ++--
 lib/entity/Cargo.toml                            | 4 ++--
 lib/faerie/Cargo.toml                            | 4 ++--
 lib/filetests/Cargo.toml                         | 2 +-
 lib/frontend/Cargo.toml                          | 4 ++--
 lib/module/Cargo.toml                            | 4 ++--
 lib/native/Cargo.toml                            | 4 ++--
 lib/reader/Cargo.toml                            | 4 ++--
 lib/simplejit/Cargo.toml                         | 4 ++--
 lib/umbrella/Cargo.toml                          | 4 ++--
 lib/wasm/Cargo.toml                              | 4 ++--
 16 files changed, 26 insertions(+), 26 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 381949a8da..e796776c46 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.13.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
-repository = "https://github.com/cranelift/cranelift"
+repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [[bin]]
diff --git a/cranelift/filetests/regalloc/coalescing-207.clif b/cranelift/filetests/regalloc/coalescing-207.clif
index 7de96e69ec..08519965b6 100644
--- a/cranelift/filetests/regalloc/coalescing-207.clif
+++ b/cranelift/filetests/regalloc/coalescing-207.clif
@@ -1,7 +1,7 @@
 test regalloc
 target x86_64 haswell
 
-; Reported as https://github.com/cranelift/cranelift/issues/207
+; Reported as https://github.com/CraneStation/cranelift/issues/207
 ;
 ; The coalescer creates a virtual register with two interfering values.
 function %pr207(i64 vmctx, i32, i32) -> i32 system_v {
diff --git a/cranelift/filetests/regalloc/coalescing-216.clif b/cranelift/filetests/regalloc/coalescing-216.clif
index 212b03dc36..b4d6e6393b 100644
--- a/cranelift/filetests/regalloc/coalescing-216.clif
+++ b/cranelift/filetests/regalloc/coalescing-216.clif
@@ -1,7 +1,7 @@
 test regalloc
 target x86_64 haswell
 
-; Reported as https://github.com/cranelift/cranelift/issues/216 from the Binaryen fuzzer.
+; Reported as https://github.com/CraneStation/cranelift/issues/216 from the Binaryen fuzzer.
 ;
 ; The (old) coalescer creates a virtual register with two identical values.
 function %pr216(i32 [%rdi], i64 vmctx [%rsi]) -> i64 [%rax] system_v {
diff --git a/cranelift/filetests/regalloc/reload-208.clif b/cranelift/filetests/regalloc/reload-208.clif
index 5da6bfa9ef..0f4a2e4bb4 100644
--- a/cranelift/filetests/regalloc/reload-208.clif
+++ b/cranelift/filetests/regalloc/reload-208.clif
@@ -3,7 +3,7 @@ target x86_64 haswell
 
 ; regex: V=v\d+
 
-; Filed as https://github.com/cranelift/cranelift/issues/208
+; Filed as https://github.com/CraneStation/cranelift/issues/208
 ;
 ; The verifier complains about a branch argument that is not in the same virtual register as the
 ; corresponding EBB argument.
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 9dd37e093b..e4a5ffc7a6 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -31,4 +31,4 @@ for crate in entity codegen frontend native reader wasm module simplejit faerie
     echo cargo publish --manifest-path "lib/$crate/Cargo.toml"
 done
 echo
-echo Then, go to https://github.com/cranelift/cranelift/releases/ and define a new release.
+echo Then, go to https://github.com/CraneStation/cranelift/releases/ and define a new release.
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index b7f601ab50..cb4149bbd4 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.13.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
-repository = "https://github.com/cranelift/cranelift"
+repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
@@ -31,4 +31,4 @@ core = ["hashmap_core"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "cranelift/cranelift" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index 2bb6e6f62f..338dbc499b 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.13.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
-repository = "https://github.com/cranelift/cranelift"
+repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 keywords = ["entity", "set", "map"]
 
@@ -15,4 +15,4 @@ std = []
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "cranelift/cranelift" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index a9fbcbd778..28233eb35d 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -3,7 +3,7 @@ name = "cranelift-faerie"
 version = "0.13.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
-repository = "https://github.com/cranelift/cranelift"
+repository = "https://github.com/CraneStation/cranelift"
 documentation = "https://cranelift.readthedocs.io/"
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
@@ -18,4 +18,4 @@ target-lexicon = "0.0.2"
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "cranelift/cranelift" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index dea40496c9..67ff90074c 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.13.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
-repository = "https://github.com/cranelift/cranelift"
+repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index b9833b4582..a5241e6b95 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.13.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
-repository = "https://github.com/cranelift/cranelift"
+repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
@@ -18,4 +18,4 @@ core = ["cranelift-codegen/core"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "cranelift/cranelift" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index c971224b46..1591fd8c9a 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -3,7 +3,7 @@ name = "cranelift-module"
 version = "0.13.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
-repository = "https://github.com/cranelift/cranelift"
+repository = "https://github.com/CraneStation/cranelift"
 documentation = "https://cranelift.readthedocs.io/"
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
@@ -21,4 +21,4 @@ core = ["hashmap_core", "cranelift-codegen/core"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "cranelift/cranelift" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index ec7abc28b1..cea09d3233 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -3,7 +3,7 @@ name = "cranelift-native"
 version = "0.13.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
-repository = "https://github.com/cranelift/cranelift"
+repository = "https://github.com/CraneStation/cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
@@ -24,4 +24,4 @@ core = ["cranelift-codegen/core", "raw-cpuid/nightly"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "cranelift/cranelift" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 837fe4abed..49d629cde8 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.13.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
-repository = "https://github.com/cranelift/cranelift"
+repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
@@ -14,4 +14,4 @@ target-lexicon = "0.0.2"
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "cranelift/cranelift" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 1bbad6a2a5..52341ac735 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -3,7 +3,7 @@ name = "cranelift-simplejit"
 version = "0.13.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
-repository = "https://github.com/cranelift/cranelift"
+repository = "https://github.com/CraneStation/cranelift"
 documentation = "https://cranelift.readthedocs.io/"
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
@@ -27,4 +27,4 @@ core = ["cranelift-codegen/core", "cranelift-module/core", "cranelift-native/cor
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "cranelift/cranelift" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index 0650d3da63..8fb271f789 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.13.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
-repository = "https://github.com/cranelift/cranelift"
+repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
@@ -20,4 +20,4 @@ core = ["cranelift-codegen/core", "cranelift-frontend/core"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "cranelift/cranelift" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 058b2e44df..0530b953c7 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -3,7 +3,7 @@ name = "cranelift-wasm"
 version = "0.13.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
-repository = "https://github.com/cranelift/cranelift"
+repository = "https://github.com/CraneStation/cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 keywords = ["webassembly", "wasm"]
@@ -27,4 +27,4 @@ core = ["hashmap_core", "cranelift-codegen/core", "cranelift-frontend/core", "wa
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "cranelift/cranelift" }
+travis-ci = { repository = "CraneStation/cranelift" }

From 262689908ed513538e4977def8df0e2fef55a2fa Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 13 Jul 2018 09:26:46 -0700
Subject: [PATCH 1901/3084] Bump version to 0.14.0

---
 cranelift/Cargo.toml     | 22 +++++++++++-----------
 cranelift/publish-all.sh |  2 +-
 lib/codegen/Cargo.toml   |  4 ++--
 lib/entity/Cargo.toml    |  2 +-
 lib/faerie/Cargo.toml    |  6 +++---
 lib/filetests/Cargo.toml |  6 +++---
 lib/frontend/Cargo.toml  |  4 ++--
 lib/module/Cargo.toml    |  6 +++---
 lib/native/Cargo.toml    |  4 ++--
 lib/reader/Cargo.toml    |  4 ++--
 lib/simplejit/Cargo.toml |  8 ++++----
 lib/umbrella/Cargo.toml  |  6 +++---
 lib/wasm/Cargo.toml      |  6 +++---
 13 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index e796776c46..9716d48212 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.13.0"
+version = "0.14.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,16 +14,16 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.13.0" }
-cranelift-reader = { path = "lib/reader", version = "0.13.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.13.0" }
-cranelift-wasm = { path = "lib/wasm", version = "0.13.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.13.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.13.0" }
-cranelift-module = { path = "lib/module", version = "0.13.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.13.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.13.0" }
-cranelift = { path = "lib/umbrella", version = "0.13.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.14.0" }
+cranelift-reader = { path = "lib/reader", version = "0.14.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.14.0" }
+cranelift-wasm = { path = "lib/wasm", version = "0.14.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.14.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.14.0" }
+cranelift-module = { path = "lib/module", version = "0.14.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.14.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.14.0" }
+cranelift = { path = "lib/umbrella", version = "0.14.0" }
 filecheck = "0.3.0"
 docopt = "1"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index e4a5ffc7a6..41c90e7224 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -4,7 +4,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.13.0"
+version="0.14.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index cb4149bbd4..0a7488f8d8 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.13.0"
+version = "0.14.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.13.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.14.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index 338dbc499b..a474e3dad7 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.13.0"
+version = "0.14.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 28233eb35d..f5e4914535 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.13.0"
+version = "0.14.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.13.0" }
-cranelift-module = { path = "../module", version = "0.13.0" }
+cranelift-codegen = { path = "../codegen", version = "0.14.0" }
+cranelift-module = { path = "../module", version = "0.14.0" }
 faerie = "0.4.2"
 goblin = "0.0.15"
 failure = "0.1.1"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 67ff90074c..6349582c94 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.13.0"
+version = "0.14.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.13.0" }
-cranelift-reader = { path = "../reader", version = "0.13.0" }
+cranelift-codegen = { path = "../codegen", version = "0.14.0" }
+cranelift-reader = { path = "../reader", version = "0.14.0" }
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index a5241e6b95..0679983e28 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.13.0"
+version = "0.14.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.13.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 1591fd8c9a..d70ecaa8b5 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.13.0"
+version = "0.14.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.13.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.13.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.14.0", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
 failure = "0.1.1"
 
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index cea09d3233..1771c916e8 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.13.0"
+version = "0.14.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -8,7 +8,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.13.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features = false }
 target-lexicon = { version = "0.0.2", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 49d629cde8..32ee8cb3b1 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.13.0"
+version = "0.14.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.13.0" }
+cranelift-codegen = { path = "../codegen", version = "0.14.0" }
 target-lexicon = "0.0.2"
 
 [badges]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 52341ac735..576d7b98cf 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.13.0"
+version = "0.14.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,9 +9,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.13.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.13.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.13.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.14.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.14.0", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.40", default-features = false }
 errno = "0.2.3"
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index 8fb271f789..f33a970be1 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.13.0"
+version = "0.14.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,8 +10,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.13.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.13.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.14.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 0530b953c7..fb0b8166cc 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.13.0"
+version = "0.14.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.0", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.13.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.13.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.14.0", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 202e45c2130c2bf6e1bdeff56706b3bc39bd9118 Mon Sep 17 00:00:00 2001
From: Lachlan Sneff 
Date: Thu, 12 Jul 2018 22:17:25 -0400
Subject: [PATCH 1902/3084] Fix broken build on no_std

---
 lib/codegen/src/lib.rs | 2 +-
 lib/native/Cargo.toml  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index d4806f9665..f9ce64bcf8 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -119,6 +119,6 @@ mod std {
 
         pub use self::hashmap_core::map as hash_map;
         pub use self::hashmap_core::{HashMap, HashSet};
-        pub use alloc::BTreeSet;
+        pub use alloc::collections::BTreeSet;
     }
 }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 1771c916e8..d0eec66679 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -12,7 +12,7 @@ cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features
 target-lexicon = { version = "0.0.2", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
-raw-cpuid = "4"
+raw-cpuid = "3.1.0"
 
 [features]
 default = ["std"]

From 2db2d946b87a09eec5fd3d0cb57798db8e2f086f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 13 Jul 2018 16:19:48 -0700
Subject: [PATCH 1903/3084] Support the rustc in the latest Ubuntu LTS.

At this time, this is Bionic, with Rust 1.25.0.
---
 .travis.yml                   |  2 ++
 lib/codegen/meta/gen_instr.py | 15 +++++++--------
 lib/codegen/src/timing.rs     |  3 ++-
 lib/entity/src/list.rs        |  4 ++--
 lib/filetests/src/runner.rs   |  8 +++++++-
 5 files changed, 20 insertions(+), 12 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 26e34b45d2..e52f99be01 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,6 +1,8 @@
 language: rust
 rust:
     - stable
+    # The version of rust in the latest Ubuntu LTS, currently Bionic.
+    - 1.25.0
     - beta
     - nightly
 matrix:
diff --git a/lib/codegen/meta/gen_instr.py b/lib/codegen/meta/gen_instr.py
index 3a33110a00..fa48dcb945 100644
--- a/lib/codegen/meta/gen_instr.py
+++ b/lib/codegen/meta/gen_instr.py
@@ -291,8 +291,7 @@ def gen_instruction_data_impl(fmt):
                                      .format(field.member, field.member))
                         if args_eq is not None:
                             fmt.line('&& {}'.format(args_eq))
-                fmt.line('_ => unsafe { '
-                         '::std::hint::unreachable_unchecked() }')
+                fmt.line('_ => unreachable!()')
         fmt.line()
 
         fmt.doc_comment(
@@ -306,20 +305,20 @@ def gen_instruction_data_impl(fmt):
                 'pub fn hash'
                 '(&self, state: &mut H, pool: &ir::ValueListPool) {',
                 '}'):
-            with fmt.indented('match self {', '}'):
+            with fmt.indented('match *self {', '}'):
                 for f in InstructionFormat.all_formats:
                     n = 'InstructionData::' + f.name
                     members = ['opcode']
                     if f.typevar_operand is None:
                         args = '&()'
                     elif f.has_value_list:
-                        members.append('args')
+                        members.append('ref args')
                         args = 'args.as_slice(pool)'
                     elif f.num_value_operands == 1:
-                        members.append('arg')
+                        members.append('ref arg')
                         args = 'arg'
                     else:
-                        members.append('args')
+                        members.append('ref args')
                         args = 'args'
                     for field in f.imm_fields:
                         members.append(field.member)
@@ -327,9 +326,9 @@ def gen_instruction_data_impl(fmt):
                     with fmt.indented(pat + ' => {', '}'):
                         fmt.line('::std::hash::Hash::hash( '
                                  '&::std::mem::discriminant(self), state);')
-                        fmt.line('::std::hash::Hash::hash(opcode, state);')
+                        fmt.line('::std::hash::Hash::hash(&opcode, state);')
                         for field in f.imm_fields:
-                            fmt.line('::std::hash::Hash::hash({}, state);'
+                            fmt.line('::std::hash::Hash::hash(&{}, state);'
                                      .format(field.member))
                         fmt.line('::std::hash::Hash::hash({}, state);'
                                  .format(args))
diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs
index c0bf1b8fdb..707efbf3f8 100644
--- a/lib/codegen/src/timing.rs
+++ b/lib/codegen/src/timing.rs
@@ -154,7 +154,8 @@ mod details {
                 fn fmtdur(mut dur: Duration, f: &mut fmt::Formatter) -> fmt::Result {
                     // Round to nearest ms by adding 500us.
                     dur += Duration::new(0, 500_000);
-                    write!(f, "{:4}.{:03} ", dur.as_secs(), dur.subsec_millis())
+                    let ms = dur.subsec_nanos() / 1_000_000;
+                    write!(f, "{:4}.{:03} ", dur.as_secs(), ms)
                 }
 
                 fmtdur(time.total, f)?;
diff --git a/lib/entity/src/list.rs b/lib/entity/src/list.rs
index f94f8ef56e..c061419eab 100644
--- a/lib/entity/src/list.rs
+++ b/lib/entity/src/list.rs
@@ -229,14 +229,14 @@ impl EntityList {
     pub fn from_slice(slice: &[T], pool: &mut ListPool) -> Self {
         let len = slice.len();
         if len == 0 {
-            return EntityList::new();
+            return Self::new();
         }
 
         let block = pool.alloc(sclass_for_length(len));
         pool.data[block] = T::new(len);
         pool.data[block + 1..block + len + 1].copy_from_slice(slice);
 
-        EntityList {
+        Self {
             index: (block + 1) as u32,
             unused: PhantomData,
         }
diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs
index 3d630d83a4..80b2e47d15 100644
--- a/lib/filetests/src/runner.rs
+++ b/lib/filetests/src/runner.rs
@@ -40,7 +40,13 @@ impl Display for QueueEntry {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         let p = self.path.to_string_lossy();
         match self.state {
-            State::Done(Ok(dur)) => write!(f, "{}.{:03} {}", dur.as_secs(), dur.subsec_millis(), p),
+            State::Done(Ok(dur)) => write!(
+                f,
+                "{}.{:03} {}",
+                dur.as_secs(),
+                dur.subsec_nanos() / 1_000_000,
+                p
+            ),
             State::Done(Err(ref e)) => write!(f, "FAIL {}: {}", p, e),
             _ => write!(f, "{}", p),
         }

From f833d25ad137ca1784b84442622dbedd8b15d10b Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 13 Jul 2018 16:27:19 -0700
Subject: [PATCH 1904/3084] Add a comment mentioning why we're remaining at
 cpuid 3.x for now.

---
 lib/native/Cargo.toml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index d0eec66679..a1c89a9f60 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -12,6 +12,7 @@ cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features
 target-lexicon = { version = "0.0.2", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
+# Due to https://github.com/gz/rust-cpuid/issues/27, stay at 3.x for now.
 raw-cpuid = "3.1.0"
 
 [features]

From 8930cb1b5da00cb5f0dcb0dfb3da344e2c2079ce Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 13 Jul 2018 16:31:20 -0700
Subject: [PATCH 1905/3084] Update dependency versions.

---
 cranelift/Cargo.toml      | 2 +-
 cranelift/fuzz/Cargo.toml | 2 +-
 lib/codegen/Cargo.toml    | 2 +-
 lib/faerie/Cargo.toml     | 2 +-
 lib/native/Cargo.toml     | 2 +-
 lib/reader/Cargo.toml     | 2 +-
 lib/simplejit/Cargo.toml  | 6 +++---
 lib/wasm/Cargo.toml       | 2 +-
 8 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 9716d48212..0ab09e1580 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -31,7 +31,7 @@ serde_derive = "1.0.8"
 term = "0.5.1"
 capstone = { version = "0.4", optional = true }
 wabt = { version = "0.4", optional = true }
-target-lexicon = "0.0.2"
+target-lexicon = "0.0.3"
 
 [features]
 default = ["disas", "wasm"]
diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
index e51dcb8672..3805d5f871 100644
--- a/cranelift/fuzz/Cargo.toml
+++ b/cranelift/fuzz/Cargo.toml
@@ -12,7 +12,7 @@ cargo-fuzz = "*"
 binaryen = { git = "https://github.com/pepyakin/binaryen-rs.git" }
 libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" }
 cranelift-wasm = { path = "../lib/wasm" }
-target-lexicon = "0.0.2"
+target-lexicon = "0.0.3"
 
 # Prevent this from interfering with workspaces
 [workspace]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 0a7488f8d8..c60de2b5b5 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -15,7 +15,7 @@ cranelift-entity = { path = "../entity", version = "0.14.0", default-features =
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
-target-lexicon = { version = "0.0.2", default-features = false }
+target-lexicon = { version = "0.0.3", default-features = false }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
 # machine code. Integration tests that need external dependencies can be
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index f5e4914535..618163fe2d 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -14,7 +14,7 @@ cranelift-module = { path = "../module", version = "0.14.0" }
 faerie = "0.4.2"
 goblin = "0.0.15"
 failure = "0.1.1"
-target-lexicon = "0.0.2"
+target-lexicon = "0.0.3"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index a1c89a9f60..7b4e3d0fef 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -9,7 +9,7 @@ readme = "README.md"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features = false }
-target-lexicon = { version = "0.0.2", default-features = false }
+target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
 # Due to https://github.com/gz/rust-cpuid/issues/27, stay at 3.x for now.
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 32ee8cb3b1..038347ca25 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -10,7 +10,7 @@ readme = "README.md"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.14.0" }
-target-lexicon = "0.0.2"
+target-lexicon = "0.0.3"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 576d7b98cf..571c063ed7 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -13,9 +13,9 @@ cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features
 cranelift-module = { path = "../module", version = "0.14.0", default-features = false }
 cranelift-native = { path = "../native", version = "0.14.0", default-features = false }
 region = "0.3.0"
-libc = { version = "0.2.40", default-features = false }
-errno = "0.2.3"
-target-lexicon = { version = "0.0.2", default-features = false }
+libc = { version = "0.2.42", default-features = false }
+errno = "0.2.4"
+target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(target_os = "windows")'.dependencies]
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index fb0b8166cc..d6c47d2645 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -15,7 +15,7 @@ cranelift-frontend = { path = "../frontend", version = "0.14.0", default-feature
 hashmap_core = { version = "0.1.8", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
-target-lexicon = { version = "0.0.2", default-features = false }
+target-lexicon = { version = "0.0.3", default-features = false }
 
 [dev-dependencies]
 wabt = "0.4"

From 96e43b366dae1a9916ee42a703ba9bac17be0683 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 13 Jul 2018 17:03:57 -0700
Subject: [PATCH 1906/3084] Add more comments explaining .travis.yml contents.

---
 .travis.yml | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index e52f99be01..95a4e0e711 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,3 +1,5 @@
+# Travis CI script. See https://travis-ci.org/ for more info.
+
 language: rust
 rust:
     - stable
@@ -7,6 +9,8 @@ rust:
     - nightly
 matrix:
     allow_failures:
+        # We try to be compatible with beta and nightly, but they occasionally
+        # fail, so we don't allow them to hold up people using stable.
         - rust: beta
         - rust: nightly
 dist: trusty

From fe1a69ac1901fdf0f98af0a89cce0fcf389b895f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 13 Jul 2018 17:08:51 -0700
Subject: [PATCH 1907/3084] Simplify the README.rst entry on building Cretonne.

Include a mention of cargo workspaces, and briefly describe the
test-all.sh script.
---
 README.rst | 29 ++++++++++++-----------------
 1 file changed, 12 insertions(+), 17 deletions(-)

diff --git a/README.rst b/README.rst
index aeeb3f4c4c..ecfded78fb 100644
--- a/README.rst
+++ b/README.rst
@@ -55,8 +55,9 @@ Cranelift requires Python 2.7 or Python 3 to build.
 Planned uses
 ------------
 
-Cranelift is designed to be a code generator for WebAssembly, but it is general enough to be useful
-elsewhere too. The initial planned uses that affected its design are:
+Cranelift is designed to be a code generator for WebAssembly, but it is general
+enough to be useful elsewhere too. The initial planned uses that affected its
+design are:
 
 1. `WebAssembly compiler for the SpiderMonkey engine in Firefox
    `_.
@@ -67,23 +68,17 @@ elsewhere too. The initial planned uses that affected its design are:
 Building Cranelift
 ------------------
 
-Cranelift is using the Cargo package manager format. First, ensure you have
-installed a current stable rust (stable, beta, and nightly should all work, but
-only stable and beta are tested consistently). Then, change the working
-directory to your clone of cranelift and run::
+Cranelift uses a `conventional Cargo build process
+`_.
 
-    cargo build
+Cranelift consists of a collection of crates, and uses a `Cargo Workspace
+`_,
+so for some cargo commands, such as
+``cargo test``, the ``--all`` is needed to tell cargo to visit all
+of the crates.
 
-This will create a *target/debug* directory where you can find the generated
-binary.
-
-To build the optimized binary for release::
-
-    cargo build --release
-
-You can then run tests with::
-
-    ./test-all.sh
+``test-all.sh`` at the top level is a script which runs all the cargo
+tests and also performs code format, lint, and documentation checks.
 
 Building with `no_std`
 ----------------------

From ff188042bb3a0bb4694a85c301601c15d7f6929d Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 13 Jul 2018 17:21:03 -0700
Subject: [PATCH 1908/3084] Update comments and formatting in test-all.sh and
 publish-all.sh.

In particular, we no longer use the Github Releases page, so remove the
comment about that.
---
 cranelift/publish-all.sh | 20 +++++++++++++++-----
 cranelift/test-all.sh    |  8 +++++++-
 2 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 41c90e7224..d0e2e01ab3 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -1,5 +1,10 @@
 #!/bin/bash
 set -euo pipefail
+
+# This is a convenience script for maintainers publishing a new version of
+# Cranelift to crates.io. To use, bump the version number below, run the
+# script, and then run the commands that the script prints.
+
 topdir=$(dirname "$0")
 cd "$topdir"
 
@@ -12,9 +17,12 @@ version="0.14.0"
 echo "Updating crate versions to $version"
 for crate in . lib/*; do
     # Update the version number of this crate to $version.
-    sed -i.bk -e "s/^version = .*/version = \"$version\"/" "$crate/Cargo.toml"
+    sed -i.bk -e "s/^version = .*/version = \"$version\"/" \
+        "$crate/Cargo.toml"
+
     # Update the required version number of any cranelift* dependencies.
-    sed -i.bk -e "/^cranelift/s/version = \"[^\"]*\"/version = \"$version\"/" "$crate/Cargo.toml"
+    sed -i.bk -e "/^cranelift/s/version = \"[^\"]*\"/version = \"$version\"/" \
+        "$crate/Cargo.toml"
 done
 
 # Update our local Cargo.lock (not checked in).
@@ -27,8 +35,10 @@ cargo update
 
 echo git commit -a -m "\"Bump version to $version"\"
 echo git push
-for crate in entity codegen frontend native reader wasm module simplejit faerie umbrella ; do
+for crate in \
+    entity codegen frontend native \
+    reader wasm module simplejit \
+    faerie umbrella
+do
     echo cargo publish --manifest-path "lib/$crate/Cargo.toml"
 done
-echo
-echo Then, go to https://github.com/CraneStation/cranelift/releases/ and define a new release.
diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index 9b13390783..580c41d47b 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -3,10 +3,12 @@ set -euo pipefail
 
 # This is the top-level test script:
 #
+# - Check code formatting.
 # - Make a debug build.
 # - Make a release build.
 # - Run unit tests for all Rust crates (including the filetests)
 # - Build API documentation.
+# - Optionally, run clippy and fuzzing.
 #
 # All tests run by this script should be passing at all times.
 
@@ -85,7 +87,11 @@ if rustup toolchain list | grep -q nightly; then
         echo "installing cargo-fuzz"
         cargo +nightly install cargo-fuzz
     fi
-    ASAN_OPTIONS=detect_leaks=0 cargo +nightly fuzz run fuzz_translate_module "$topdir/fuzz/corpus/fuzz_translate_module/ffaefab69523eb11935a9b420d58826c8ea65c4c"
+
+    fuzz_module="ffaefab69523eb11935a9b420d58826c8ea65c4c"
+    ASAN_OPTIONS=detect_leaks=0 \
+    cargo +nightly fuzz run fuzz_translate_module \
+        "$topdir/fuzz/corpus/fuzz_translate_module/$fuzz_module"
 else
     echo "nightly toolchain not found, skipping fuzz target integration test"
 fi

From 753bb049f8cd688be70fef48b7a817cac4301521 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 13 Jul 2018 17:23:23 -0700
Subject: [PATCH 1909/3084] Fix "Title overline too short." warnings in *.rst
 files.

---
 cranelift/docs/compare-llvm.rst | 4 ++--
 cranelift/docs/index.rst        | 2 +-
 cranelift/docs/langref.rst      | 4 ++--
 cranelift/docs/metaref.rst      | 4 ++--
 4 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst
index 78129e500e..7f3c320ace 100644
--- a/cranelift/docs/compare-llvm.rst
+++ b/cranelift/docs/compare-llvm.rst
@@ -1,6 +1,6 @@
-*************************
+**************************
 Cranelift compared to LLVM
-*************************
+**************************
 
 `LLVM `_ is a collection of compiler components implemented as
 a set of C++ libraries. It can be used to build both JIT compilers and static
diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst
index 2a566ea286..552c3089d5 100644
--- a/cranelift/docs/index.rst
+++ b/cranelift/docs/index.rst
@@ -1,5 +1,5 @@
 Cranelift Code Generator
-=======================
+========================
 
 Contents:
 
diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst
index cd585627ec..5265d9b7b6 100644
--- a/cranelift/docs/langref.rst
+++ b/cranelift/docs/langref.rst
@@ -1,6 +1,6 @@
-***************************
+****************************
 Cranelift Language Reference
-***************************
+****************************
 
 .. default-domain:: clif
 .. highlight:: clif
diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst
index 50c15e4a99..a1b0dd1f32 100644
--- a/cranelift/docs/metaref.rst
+++ b/cranelift/docs/metaref.rst
@@ -1,6 +1,6 @@
-********************************
+*********************************
 Cranelift Meta Language Reference
-********************************
+*********************************
 
 .. default-domain:: py
 .. highlight:: python

From eb980821555bf36a7cd72ce9ef27a9cd912847e0 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Sat, 14 Jul 2018 06:46:22 -0700
Subject: [PATCH 1910/3084] Update to faerie 0.4.3.

This resolves conflicts when multiple versions of target-lexicon are
used at the same time.
---
 lib/faerie/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 618163fe2d..9700bf3a6a 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -11,7 +11,7 @@ readme = "README.md"
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.14.0" }
 cranelift-module = { path = "../module", version = "0.14.0" }
-faerie = "0.4.2"
+faerie = "0.4.3"
 goblin = "0.0.15"
 failure = "0.1.1"
 target-lexicon = "0.0.3"

From a28a3c3ea9cb0265f4639a72a95105f538b64088 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Sat, 14 Jul 2018 07:27:35 -0700
Subject: [PATCH 1911/3084] Bump version to 0.15.0

---
 cranelift/Cargo.toml     | 22 +++++++++++-----------
 cranelift/publish-all.sh |  2 +-
 lib/codegen/Cargo.toml   |  4 ++--
 lib/entity/Cargo.toml    |  2 +-
 lib/faerie/Cargo.toml    |  6 +++---
 lib/filetests/Cargo.toml |  6 +++---
 lib/frontend/Cargo.toml  |  4 ++--
 lib/module/Cargo.toml    |  6 +++---
 lib/native/Cargo.toml    |  4 ++--
 lib/reader/Cargo.toml    |  4 ++--
 lib/simplejit/Cargo.toml |  8 ++++----
 lib/umbrella/Cargo.toml  |  6 +++---
 lib/wasm/Cargo.toml      |  6 +++---
 13 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 0ab09e1580..d02e818de9 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.14.0"
+version = "0.15.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,16 +14,16 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.14.0" }
-cranelift-reader = { path = "lib/reader", version = "0.14.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.14.0" }
-cranelift-wasm = { path = "lib/wasm", version = "0.14.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.14.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.14.0" }
-cranelift-module = { path = "lib/module", version = "0.14.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.14.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.14.0" }
-cranelift = { path = "lib/umbrella", version = "0.14.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.15.0" }
+cranelift-reader = { path = "lib/reader", version = "0.15.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.15.0" }
+cranelift-wasm = { path = "lib/wasm", version = "0.15.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.15.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.15.0" }
+cranelift-module = { path = "lib/module", version = "0.15.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.15.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.15.0" }
+cranelift = { path = "lib/umbrella", version = "0.15.0" }
 filecheck = "0.3.0"
 docopt = "1"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index d0e2e01ab3..75c6778201 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.14.0"
+version="0.15.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index c60de2b5b5..aa445322fe 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.14.0"
+version = "0.15.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.14.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.15.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index a474e3dad7..30bffdeade 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.14.0"
+version = "0.15.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 9700bf3a6a..538c59b5fc 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.14.0"
+version = "0.15.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.14.0" }
-cranelift-module = { path = "../module", version = "0.14.0" }
+cranelift-codegen = { path = "../codegen", version = "0.15.0" }
+cranelift-module = { path = "../module", version = "0.15.0" }
 faerie = "0.4.3"
 goblin = "0.0.15"
 failure = "0.1.1"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 6349582c94..66c3a42863 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.14.0"
+version = "0.15.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.14.0" }
-cranelift-reader = { path = "../reader", version = "0.14.0" }
+cranelift-codegen = { path = "../codegen", version = "0.15.0" }
+cranelift-reader = { path = "../reader", version = "0.15.0" }
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 0679983e28..d57443db96 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.14.0"
+version = "0.15.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index d70ecaa8b5..e4a4e6833b 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.14.0"
+version = "0.15.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.14.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.15.0", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
 failure = "0.1.1"
 
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 7b4e3d0fef..c42b23fb81 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.14.0"
+version = "0.15.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -8,7 +8,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 038347ca25..8b33740741 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.14.0"
+version = "0.15.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.14.0" }
+cranelift-codegen = { path = "../codegen", version = "0.15.0" }
 target-lexicon = "0.0.3"
 
 [badges]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 571c063ed7..968c411e4c 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.14.0"
+version = "0.15.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,9 +9,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.14.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.14.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.15.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.15.0", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index f33a970be1..c7391d29ce 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.14.0"
+version = "0.15.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,8 +10,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.14.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.15.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index d6c47d2645..dc4faf3004 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.14.0"
+version = "0.15.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.0", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.14.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.14.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.15.0", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 17db4e6be87f3d53d82d09c722d76aa65c70dc59 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 17 Jul 2018 12:28:46 -0700
Subject: [PATCH 1912/3084] Add a LICENSE file to each crate.

The individual crates are published separately from the main repository
on crates.io. To ensure that a copy of the LICENSE file accompanies all
published copies of the code, give each crate its own LICENSE file.
---
 lib/codegen/LICENSE   | 219 ++++++++++++++++++++++++++++++++++++++++++
 lib/entity/LICENSE    | 219 ++++++++++++++++++++++++++++++++++++++++++
 lib/faerie/LICENSE    | 219 ++++++++++++++++++++++++++++++++++++++++++
 lib/filetests/LICENSE | 219 ++++++++++++++++++++++++++++++++++++++++++
 lib/frontend/LICENSE  | 219 ++++++++++++++++++++++++++++++++++++++++++
 lib/module/LICENSE    | 219 ++++++++++++++++++++++++++++++++++++++++++
 lib/native/LICENSE    | 219 ++++++++++++++++++++++++++++++++++++++++++
 lib/reader/LICENSE    | 219 ++++++++++++++++++++++++++++++++++++++++++
 lib/simplejit/LICENSE | 219 ++++++++++++++++++++++++++++++++++++++++++
 lib/umbrella/LICENSE  | 219 ++++++++++++++++++++++++++++++++++++++++++
 lib/wasm/LICENSE      | 219 ++++++++++++++++++++++++++++++++++++++++++
 11 files changed, 2409 insertions(+)
 create mode 100644 lib/codegen/LICENSE
 create mode 100644 lib/entity/LICENSE
 create mode 100644 lib/faerie/LICENSE
 create mode 100644 lib/filetests/LICENSE
 create mode 100644 lib/frontend/LICENSE
 create mode 100644 lib/module/LICENSE
 create mode 100644 lib/native/LICENSE
 create mode 100644 lib/reader/LICENSE
 create mode 100644 lib/simplejit/LICENSE
 create mode 100644 lib/umbrella/LICENSE
 create mode 100644 lib/wasm/LICENSE

diff --git a/lib/codegen/LICENSE b/lib/codegen/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/codegen/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
diff --git a/lib/entity/LICENSE b/lib/entity/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/entity/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
diff --git a/lib/faerie/LICENSE b/lib/faerie/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/faerie/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
diff --git a/lib/filetests/LICENSE b/lib/filetests/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/filetests/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
diff --git a/lib/frontend/LICENSE b/lib/frontend/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/frontend/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
diff --git a/lib/module/LICENSE b/lib/module/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/module/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
diff --git a/lib/native/LICENSE b/lib/native/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/native/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
diff --git a/lib/reader/LICENSE b/lib/reader/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/reader/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
diff --git a/lib/simplejit/LICENSE b/lib/simplejit/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/simplejit/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
diff --git a/lib/umbrella/LICENSE b/lib/umbrella/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/umbrella/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
diff --git a/lib/wasm/LICENSE b/lib/wasm/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/wasm/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.

From 4d5451ad117ec5859681fb7a2151d74a49c60d99 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 17 Jul 2018 14:45:01 -0700
Subject: [PATCH 1913/3084] Fix "Title overline too short." warnings in more
 *.rst files.

---
 rustc.rst        | 4 ++--
 spidermonkey.rst | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/rustc.rst b/rustc.rst
index b7c01f8669..03dfa82499 100644
--- a/rustc.rst
+++ b/rustc.rst
@@ -1,6 +1,6 @@
-=================
+==================
 Cranelift in Rustc
-=================
+==================
 
 One goal for Cranelift is to be usable as a backend suitable for compiling Rust
 in debug mode. This mode doesn't require a lot of mid-level optimization, and it
diff --git a/spidermonkey.rst b/spidermonkey.rst
index 874634ea56..15662b0fd5 100644
--- a/spidermonkey.rst
+++ b/spidermonkey.rst
@@ -1,6 +1,6 @@
-========================
+=========================
 Cranelift in SpiderMonkey
-========================
+=========================
 
 `SpiderMonkey `_ is the
 JavaScript and WebAssembly engine in Firefox. Cranelift is designed to be used in SpiderMonkey with

From 8d0f34310f59c4978ac7ee50f0410c639e9a136e Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 17 Jul 2018 14:38:19 -0700
Subject: [PATCH 1914/3084] Convert top-level *.rst files to markdown.

These files don't use any Sphinx has that Markdown lacks, and Markdown
is more widely used and easier to edit.
---
 README.rst                | 143 --------------------------------------
 cranelift/README.md       | 137 ++++++++++++++++++++++++++++++++++++
 cranelift/rustc.md        |  69 ++++++++++++++++++
 cranelift/spidermonkey.md |  40 +++++++++++
 rustc.rst                 |  64 -----------------
 spidermonkey.rst          |  44 ------------
 6 files changed, 246 insertions(+), 251 deletions(-)
 delete mode 100644 README.rst
 create mode 100644 cranelift/README.md
 create mode 100644 cranelift/rustc.md
 create mode 100644 cranelift/spidermonkey.md
 delete mode 100644 rustc.rst
 delete mode 100644 spidermonkey.rst

diff --git a/README.rst b/README.rst
deleted file mode 100644
index ecfded78fb..0000000000
--- a/README.rst
+++ /dev/null
@@ -1,143 +0,0 @@
-========================
-Cranelift Code Generator
-========================
-
-Cranelift is a low-level retargetable code generator. It translates a `target-independent
-intermediate representation `_ into executable
-machine code.
-
-.. image:: https://readthedocs.org/projects/cranelift/badge/?version=latest
-    :target: https://cranelift.readthedocs.io/en/latest/?badge=latest
-    :alt: Documentation Status
-
-.. image:: https://travis-ci.org/CraneStation/cranelift.svg?branch=master
-    :target: https://travis-ci.org/CraneStation/cranelift
-    :alt: Build Status
-
-.. image:: https://badges.gitter.im/CraneStation/CraneStation.svg
-    :target: https://gitter.im/CraneStation/Lobby/~chat
-    :alt: Gitter chat
-
-For more information, see `the documentation
-`_.
-
-Status
-------
-
-Cranelift currently supports enough functionality to run a wide variety of
-programs, including all the functionality needed to execute WebAssembly MVP
-functions, although it needs to be used within an external WebAssembly
-embedding to be part of a complete WebAssembly implementation.
-
-The x86-64 backend is currently the most complete and stable; other
-architectures are in various stages of development. Cranelift currently supports
-the System V AMD64 ABI calling convention used on many platforms, but does not
-yet support the Windows x64 calling convention. The performance of code
-produced by Cranelift is not yet impressive, though we have plans to fix that.
-
-The core codegen crates have minimal dependencies, support
-`no_std <#building-with-no-std>`_ mode, and do not require any host
-floating-point support.
-
-Cranelift does not yet perform mitigations for Spectre or related security
-issues, though it may do so in the future. It does not currently make any
-security-relevant instruction timing guarantees. It has seen a fair amount
-of testing and fuzzing, although more work is needed before it would be
-ready for a production use case.
-
-Cranelift's APIs are not yet stable.
-
-Cranelift currently supports Rust 1.22.1 and later. We intend to always support
-the latest *stable* Rust. And, we currently support the version of Rust in the
-latest Ubuntu LTS, although whether we will always do so is not yet determined.
-Cranelift requires Python 2.7 or Python 3 to build.
-
-Planned uses
-------------
-
-Cranelift is designed to be a code generator for WebAssembly, but it is general
-enough to be useful elsewhere too. The initial planned uses that affected its
-design are:
-
-1. `WebAssembly compiler for the SpiderMonkey engine in Firefox
-   `_.
-2. `Backend for the IonMonkey JavaScript JIT compiler in Firefox
-   `_.
-3. `Debug build backend for the Rust compiler `_.
-
-Building Cranelift
-------------------
-
-Cranelift uses a `conventional Cargo build process
-`_.
-
-Cranelift consists of a collection of crates, and uses a `Cargo Workspace
-`_,
-so for some cargo commands, such as
-``cargo test``, the ``--all`` is needed to tell cargo to visit all
-of the crates.
-
-``test-all.sh`` at the top level is a script which runs all the cargo
-tests and also performs code format, lint, and documentation checks.
-
-Building with `no_std`
-----------------------
-
-The following crates support `no_std`:
- - `cranelift-entity`
- - `cranelift-codegen`
- - `cranelift-frontend`
- - `cranelift-native`
- - `cranelift-wasm`
- - `cranelift-module`
- - `cranelift-simplejit`
- - `cranelift`
-
-To use `no_std` mode, disable the `std` feature and enable the `core` feature.
-This currently requires nightly rust.
-
-For example, to build `cranelift-codegen`:
-
-.. code-block:: sh
-
-    cd lib/codegen
-    cargo build --no-default-features --features core
-
-Or, when using `cranelift-codegen` as a dependency (in Cargo.toml):
-
-.. code-block::
-
-    [dependency.cranelift-codegen]
-    ...
-    default-features = false
-    features = ["core"]
-
-`no_std` support is currently "best effort". We won't try to break it, and
-we'll accept patches fixing problems, however we don't expect all developers to
-build and test `no_std` when submitting patches. Accordingly, the
-`./test-all.sh` script does not test `no_std`.
-
-There is a separate `./test-no_std.sh` script that tests the `no_std`
-support in packages which support it.
-
-It's important to note that cranelift still needs liballoc to compile.
-Thus, whatever environment is used must implement an allocator.
-
-Also, to allow the use of HashMaps with `no_std`, an external crate called
-`hashmap_core` is pulled in (via the `core` feature). This is mostly the same
-as `std::collections::HashMap`, except that it doesn't have DOS protection.
-Just something to think about.
-
-Building the documentation
---------------------------
-
-To build the Cranelift documentation, you need the `Sphinx documentation
-generator `_::
-
-    $ pip install sphinx sphinx-autobuild sphinx_rtd_theme
-    $ cd cranelift/docs
-    $ make html
-    $ open _build/html/index.html
-
-We don't support Sphinx versions before 1.4 since the format of index tuples
-has changed.
diff --git a/cranelift/README.md b/cranelift/README.md
new file mode 100644
index 0000000000..566cd7a78a
--- /dev/null
+++ b/cranelift/README.md
@@ -0,0 +1,137 @@
+Cranelift Code Generator
+========================
+
+Cranelift is a low-level retargetable code generator. It translates a
+[target-independent intermediate
+representation](https://cranelift.readthedocs.io/en/latest/langref.html)
+into executable machine code.
+
+[![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
+[![Build Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
+[![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby/~chat)
+
+For more information, see [the
+documentation](https://cranelift.readthedocs.io/en/latest/?badge=latest).
+
+Status
+------
+
+Cranelift currently supports enough functionality to run a wide variety
+of programs, including all the functionality needed to execute
+WebAssembly MVP functions, although it needs to be used within an
+external WebAssembly embedding to be part of a complete WebAssembly
+implementation.
+
+The x86-64 backend is currently the most complete and stable; other
+architectures are in various stages of development. Cranelift currently
+supports the System V AMD64 ABI calling convention used on many
+platforms, but does not yet support the Windows x64 calling convention.
+The performance of code produced by Cranelift is not yet impressive,
+though we have plans to fix that.
+
+The core codegen crates have minimal dependencies, support
+[no\_std](#building-with-no-std) mode, and do not require any host
+floating-point support.
+
+Cranelift does not yet perform mitigations for Spectre or related
+security issues, though it may do so in the future. It does not
+currently make any security-relevant instruction timing guarantees. It
+has seen a fair amount of testing and fuzzing, although more work is
+needed before it would be ready for a production use case.
+
+Cranelift's APIs are not yet stable.
+
+Cranelift currently supports Rust 1.22.1 and later. We intend to always
+support the latest *stable* Rust. And, we currently support the version
+of Rust in the latest Ubuntu LTS, although whether we will always do so
+is not yet determined. Cranelift requires Python 2.7 or Python 3 to
+build.
+
+Planned uses
+------------
+
+Cranelift is designed to be a code generator for WebAssembly, but it is
+general enough to be useful elsewhere too. The initial planned uses that
+affected its design are:
+
+1.  [WebAssembly compiler for the SpiderMonkey engine in
+    Firefox](spidermonkey.md#phase-1-webassembly).
+2.  [Backend for the IonMonkey JavaScript JIT compiler in
+    Firefox](spidermonkey.md#phase-2-ionmonkey).
+3.  [Debug build backend for the Rust compiler](rustc.md).
+
+Building Cranelift
+------------------
+
+Cranelift uses a [conventional Cargo build
+process](https://doc.rust-lang.org/cargo/guide/working-on-an-existing-project.html).
+
+Cranelift consists of a collection of crates, and uses a [Cargo
+Workspace](https://doc.rust-lang.org/book/second-edition/ch14-03-cargo-workspaces.html),
+so for some cargo commands, such as `cargo test`, the `--all` is needed
+to tell cargo to visit all of the crates.
+
+`test-all.sh` at the top level is a script which runs all the cargo
+tests and also performs code format, lint, and documentation checks.
+
+Building with no\_std
+---------------------
+
+The following crates support \`no\_std\`:
+ - cranelift-entity
+ - cranelift-codegen
+ - cranelift-frontend
+ - cranelift-native
+ - cranelift-wasm
+ - cranelift-module
+ - cranelift-simplejit
+ - cranelift
+
+To use no\_std mode, disable the std feature and enable the core
+feature. This currently requires nightly rust.
+
+For example, to build \`cranelift-codegen\`:
+
+``` {.sourceCode .sh}
+cd lib/codegen
+cargo build --no-default-features --features core
+```
+
+Or, when using cranelift-codegen as a dependency (in Cargo.toml):
+
+``` {.sourceCode .}
+[dependency.cranelift-codegen]
+...
+default-features = false
+features = ["core"]
+```
+
+no\_std support is currently "best effort". We won't try to break it,
+and we'll accept patches fixing problems, however we don't expect all
+developers to build and test no\_std when submitting patches.
+Accordingly, the ./test-all.sh script does not test no\_std.
+
+There is a separate ./test-no\_std.sh script that tests the no\_std
+support in packages which support it.
+
+It's important to note that cranelift still needs liballoc to compile.
+Thus, whatever environment is used must implement an allocator.
+
+Also, to allow the use of HashMaps with no\_std, an external crate
+called hashmap\_core is pulled in (via the core feature). This is mostly
+the same as std::collections::HashMap, except that it doesn't have DOS
+protection. Just something to think about.
+
+Building the documentation
+--------------------------
+
+To build the Cranelift documentation, you need the [Sphinx documentation
+generator](https://www.sphinx-doc.org/):
+
+    $ pip install sphinx sphinx-autobuild sphinx_rtd_theme
+    $ cd cranelift/docs
+    $ make html
+    $ open _build/html/index.html
+
+We don't support Sphinx versions before 1.4 since the format of index
+tuples has changed.
diff --git a/cranelift/rustc.md b/cranelift/rustc.md
new file mode 100644
index 0000000000..0a460da9d8
--- /dev/null
+++ b/cranelift/rustc.md
@@ -0,0 +1,69 @@
+Cranelift in Rustc
+==================
+
+One goal for Cranelift is to be usable as a backend suitable for
+compiling Rust in debug mode. This mode doesn't require a lot of
+mid-level optimization, and it does want very fast compile times, and
+this matches up fairly well with what we expect Cranelift's initial
+strengths and weaknesses will be. Cranelift is being designed to take
+aggressive advantage of multiple cores, and to be very efficient with
+its use of memory.
+
+Another goal is a "pretty good" backend. The idea here is to do the work
+to get MIR-level inlining enabled, do some basic optimizations in
+Cranelift to capture the low-hanging fruit, and then use that along with
+good low-level optimizations to produce code which has a chance of being
+decently fast, with quite fast compile times. It obviously wouldn't
+compete with LLVM-based release builds in terms of optimization, but for
+some users, completely unoptimized code is too slow to test with, so a
+"pretty good" mode might be good enough.
+
+There's plenty of work to do to achieve these goals, and if achieve
+them, we'll have enabled a Rust compiler written entirely in Rust, and
+enabled faster Rust compile times for important use cases.
+
+With all that said, there is a potential goal beyond that, which is to
+build a full optimizing release-capable backend. We can't predict how
+far Cranelift will go yet, but we do have some crazy ideas about what
+such a thing might look like, including:
+
+ - Take advantage of Rust language properties in the optimizer. With
+   LLVM, Rust is able to use annotations to describe some of its
+   aliasing guarantees, however the annotations are awkward and
+   limited. An optimizer that can represent the core aliasing
+   relationships that Rust provides directly has the potential to be
+   very powerful without the need for complex alias analysis logic.
+   Unsafe blocks are an interesting challenge, however in many simple
+   cases, like Vec, it may be possible to recover what the optimizer
+   needs to know.
+ - Design for superoptimization. Traditionally, compiler development
+   teams have spent many years of manual effort to identify patterns of
+   code that can be matched and replaced. Superoptimizers have been
+   contributing some to this effort, but in the future, we may be able
+   to reverse roles. Superoptimizers will do the bulk of the work, and
+   humans will contribute specialized optimizations that
+   superoptimizers miss. This has the potential to take a new optimizer
+   from scratch to diminishing-returns territory with much less manual
+   effort.
+ - Build an optimizer IR without the constraints of fast-debug-build
+   compilation. Cranelift's base IR is focused on Codegen, so a
+   full-strength optimizer would either use an IR layer on top of it
+   (possibly using Cranelift's flexible EntityMap system), or possibly
+   an independent IR that could be translated to/from the base IR.
+   Either way, this overall architecture would keep the optimizer out
+   of the way of the non-optimizing build path, which keeps that path
+   fast and simple, and gives the optimizer more flexibility. If we
+   then want to base the IR on a powerful data structure like the Value
+   State Dependence Graph (VSDG), we can do so with fewer compromises.
+
+And, these ideas build on each other. For example, one of the challenges
+for dependence-graph-oriented IRs like the VSDG is getting good enough
+memory dependence information. But if we can get high-quality aliasing
+information directly from the Rust front-end, we should be in great
+shape. As another example, it's often harder for superoptimizers to
+reason about control flow than expression graphs. But, graph-oriented
+IRs like the VSDG represent control flow as control dependencies. It's
+difficult to say how powerful this combination will be until we try it,
+but if nothing else, it should be very convenient to express
+pattern-matching over a single graph that includes both data and control
+dependencies.
diff --git a/cranelift/spidermonkey.md b/cranelift/spidermonkey.md
new file mode 100644
index 0000000000..516aa84a81
--- /dev/null
+++ b/cranelift/spidermonkey.md
@@ -0,0 +1,40 @@
+Cranelift in SpiderMonkey
+=========================
+
+[SpiderMonkey](https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey)
+is the JavaScript and WebAssembly engine in Firefox. Cranelift is
+designed to be used in SpiderMonkey with the goal of enabling better
+code generation for ARM's 32-bit and 64-bit architectures, and building
+a framework for improved low-level code optimizations in the future.
+
+Phase 1: WebAssembly
+--------------------
+
+SpiderMonkey currently has two WebAssembly compilers: The tier 1
+baseline compiler (not shown below) and the tier 2 compiler using the
+IonMonkey JavaScript compiler's optimizations and register allocation.
+
+![Cranelift in SpiderMonkey phase 1](media/spidermonkey1.png)
+
+In phase 1, Cranelift aims to replace the IonMonkey-based tier 2
+compiler for WebAssembly only. It will still be orchestrated by the
+BaldrMonkey engine and compile WebAssembly modules on multiple threads.
+Cranelift translates binary wasm functions directly into its own
+intermediate representation, and it generates binary machine code
+without depending on SpiderMonkey's macro assembler.
+
+Phase 2: IonMonkey
+------------------
+
+The IonMonkey JIT compiler is designed to compile JavaScript code. It
+uses two separate intermediate representations to do that:
+
+ - MIR is used for optimizations that are specific to JavaScript JIT
+   compilation. It has good support for JS types and the special tricks
+   needed to make JS fast.
+ - LIR is used for register allocation.
+
+![Cranelift in SpiderMonkey phase 2](media/spidermonkey2.png)
+
+Cranelift has its own register allocator, so the LIR representation can
+be skipped when using Cranelift as a backend for IonMonkey.
diff --git a/rustc.rst b/rustc.rst
deleted file mode 100644
index 03dfa82499..0000000000
--- a/rustc.rst
+++ /dev/null
@@ -1,64 +0,0 @@
-==================
-Cranelift in Rustc
-==================
-
-One goal for Cranelift is to be usable as a backend suitable for compiling Rust
-in debug mode. This mode doesn't require a lot of mid-level optimization, and it
-does want very fast compile times, and this matches up fairly well with what we
-expect Cranelift's initial strengths and weaknesses will be. Cranelift is being
-designed to take aggressive advantage of multiple cores, and to be very efficient
-with its use of memory.
-
-Another goal is a "pretty good" backend. The idea here is to do the work to get
-MIR-level inlining enabled, do some basic optimizations in Cranelift to capture the
-low-hanging fruit, and then use that along with good low-level optimizations to
-produce code which has a chance of being decently fast, with quite fast compile
-times. It obviously wouldn't compete with LLVM-based release builds in terms of
-optimization, but for some users, completely unoptimized code is too slow to test
-with, so a "pretty good" mode might be good enough.
-
-There's plenty of work to do to achieve these goals, and if achieve them, we'll have
-enabled a Rust compiler written entirely in Rust, and enabled faster Rust compile
-times for important use cases.
-
-With all that said, there is a potential goal beyond that, which is to build a
-full optimizing release-capable backend. We can't predict how far Cranelift will go
-yet, but we do have some crazy ideas about what such a thing might look like,
-including:
-
-- Take advantage of Rust language properties in the optimizer. With LLVM, Rust is
-  able to use annotations to describe some of its aliasing guarantees, however the
-  annotations are awkward and limited. An optimizer that can represent the core
-  aliasing relationships that Rust provides directly has the potential to be very
-  powerful without the need for complex alias analysis logic. Unsafe blocks are an
-  interesting challenge, however in many simple cases, like Vec, it may be possible
-  to recover what the optimizer needs to know.
-
-- Design for superoptimization. Traditionally, compiler development teams have
-  spent many years of manual effort to identify patterns of code that can be
-  matched and replaced. Superoptimizers have been contributing some to this
-  effort, but in the future, we may be able to reverse roles.
-  Superoptimizers will do the bulk of the work, and humans will contribute
-  specialized optimizations that superoptimizers miss. This has the potential to
-  take a new optimizer from scratch to diminishing-returns territory with much
-  less manual effort.
-
-- Build an optimizer IR without the constraints of fast-debug-build compilation.
-  Cranelift's base IR is focused on Codegen, so a full-strength optimizer would either
-  use an IR layer on top of it (possibly using Cranelift's flexible EntityMap system),
-  or possibly an independent IR that could be translated to/from the base IR. Either
-  way, this overall architecture would keep the optimizer out of the way of the
-  non-optimizing build path, which keeps that path fast and simple, and gives the
-  optimizer more flexibility. If we then want to base the IR on a powerful data
-  structure like the Value State Dependence Graph (VSDG), we can do so with fewer
-  compromises.
-
-And, these ideas build on each other. For example, one of the challenges for
-dependence-graph-oriented IRs like the VSDG is getting good enough memory dependence
-information. But if we can get high-quality aliasing information directly from the
-Rust front-end, we should be in great shape. As another example, it's often harder
-for superoptimizers to reason about control flow than expression graphs. But,
-graph-oriented IRs like the VSDG represent control flow as control dependencies.
-It's difficult to say how powerful this combination will be until we try it, but
-if nothing else, it should be very convenient to express pattern-matching over a
-single graph that includes both data and control dependencies.
diff --git a/spidermonkey.rst b/spidermonkey.rst
deleted file mode 100644
index 15662b0fd5..0000000000
--- a/spidermonkey.rst
+++ /dev/null
@@ -1,44 +0,0 @@
-=========================
-Cranelift in SpiderMonkey
-=========================
-
-`SpiderMonkey `_ is the
-JavaScript and WebAssembly engine in Firefox. Cranelift is designed to be used in SpiderMonkey with
-the goal of enabling better code generation for ARM's 32-bit and 64-bit architectures, and building
-a framework for improved low-level code optimizations in the future.
-
-Phase 1: WebAssembly
---------------------
-
-SpiderMonkey currently has two WebAssembly compilers: The tier 1 baseline compiler (not shown
-below) and the tier 2 compiler using the IonMonkey JavaScript compiler's optimizations and register
-allocation.
-
-.. image:: media/spidermonkey1.png
-    :align: center
-    :width: 80%
-    :alt: Cranelift in SpiderMonkey phase 1
-
-In phase 1, Cranelift aims to replace the IonMonkey-based tier 2 compiler for WebAssembly only. It
-will still be orchestrated by the BaldrMonkey engine and compile WebAssembly modules on multiple
-threads. Cranelift translates binary wasm functions directly into its own intermediate
-representation, and it generates binary machine code without depending on SpiderMonkey's macro
-assembler.
-
-Phase 2: IonMonkey
-------------------
-
-The IonMonkey JIT compiler is designed to compile JavaScript code. It uses two separate
-intermediate representations to do that:
-
-- MIR is used for optimizations that are specific to JavaScript JIT compilation. It has good
-  support for JS types and the special tricks needed to make JS fast.
-- LIR is used for register allocation.
-
-.. image:: media/spidermonkey2.png
-    :align: center
-    :width: 80%
-    :alt: Cranelift in SpiderMonkey phase 2
-
-Cranelift has its own register allocator, so the LIR representation can be skipped when using
-Cranelift as a backend for IonMonkey.

From 82ea38e4af32fa5ad7439656ef300ca9c80cb688 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 17 Jul 2018 14:53:59 -0700
Subject: [PATCH 1915/3084] Convert expected uses to an unnumbered list, as
 they aren't ordered.

---
 cranelift/README.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index 566cd7a78a..2cecd8ee69 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -54,11 +54,11 @@ Cranelift is designed to be a code generator for WebAssembly, but it is
 general enough to be useful elsewhere too. The initial planned uses that
 affected its design are:
 
-1.  [WebAssembly compiler for the SpiderMonkey engine in
+ - [WebAssembly compiler for the SpiderMonkey engine in
     Firefox](spidermonkey.md#phase-1-webassembly).
-2.  [Backend for the IonMonkey JavaScript JIT compiler in
+ - [Backend for the IonMonkey JavaScript JIT compiler in
     Firefox](spidermonkey.md#phase-2-ionmonkey).
-3.  [Debug build backend for the Rust compiler](rustc.md).
+ - [Debug build backend for the Rust compiler](rustc.md).
 
 Building Cranelift
 ------------------

From c0d5ffc380cfd148286e35cc3918be49ba448249 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 17 Jul 2018 14:54:39 -0700
Subject: [PATCH 1916/3084] The latest Ubuntu LTS now has Rust 1.25.

---
 cranelift/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index 2cecd8ee69..c943d760be 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -41,7 +41,7 @@ needed before it would be ready for a production use case.
 
 Cranelift's APIs are not yet stable.
 
-Cranelift currently supports Rust 1.22.1 and later. We intend to always
+Cranelift currently supports Rust 1.25.1 and later. We intend to always
 support the latest *stable* Rust. And, we currently support the version
 of Rust in the latest Ubuntu LTS, although whether we will always do so
 is not yet determined. Cranelift requires Python 2.7 or Python 3 to

From 289145e7cf2ba5db25f79f82a62e72c8d583e870 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 17 Jul 2018 15:43:39 -0700
Subject: [PATCH 1917/3084] Update to wasmparser.rs 0.17.1.

---
 lib/wasm/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index dc4faf3004..1f2fdf4464 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -9,7 +9,7 @@ readme = "README.md"
 keywords = ["webassembly", "wasm"]
 
 [dependencies]
-wasmparser = { version = "0.17.0", default-features = false }
+wasmparser = { version = "0.17.1", default-features = false }
 cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
 cranelift-frontend = { path = "../frontend", version = "0.15.0", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }

From 76a537e3d4c5344d1bf2c6cf347317caf6482284 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 17 Jul 2018 15:46:35 -0700
Subject: [PATCH 1918/3084] Update to wasmparser.rs 0.17.2.

---
 lib/wasm/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 1f2fdf4464..4c287cc05d 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -9,7 +9,7 @@ readme = "README.md"
 keywords = ["webassembly", "wasm"]
 
 [dependencies]
-wasmparser = { version = "0.17.1", default-features = false }
+wasmparser = { version = "0.17.2", default-features = false }
 cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
 cranelift-frontend = { path = "../frontend", version = "0.15.0", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }

From ce27b2a74f0b999126782fe9c7c9a6de429b560a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 17 Jul 2018 15:49:04 -0700
Subject: [PATCH 1919/3084] Bump version to 0.16.0

---
 cranelift/Cargo.toml     | 22 +++++++++++-----------
 cranelift/publish-all.sh |  2 +-
 lib/codegen/Cargo.toml   |  4 ++--
 lib/entity/Cargo.toml    |  2 +-
 lib/faerie/Cargo.toml    |  6 +++---
 lib/filetests/Cargo.toml |  6 +++---
 lib/frontend/Cargo.toml  |  4 ++--
 lib/module/Cargo.toml    |  6 +++---
 lib/native/Cargo.toml    |  4 ++--
 lib/reader/Cargo.toml    |  4 ++--
 lib/simplejit/Cargo.toml |  8 ++++----
 lib/umbrella/Cargo.toml  |  6 +++---
 lib/wasm/Cargo.toml      |  6 +++---
 13 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index d02e818de9..59dcbf858b 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.15.0"
+version = "0.16.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,16 +14,16 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.15.0" }
-cranelift-reader = { path = "lib/reader", version = "0.15.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.15.0" }
-cranelift-wasm = { path = "lib/wasm", version = "0.15.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.15.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.15.0" }
-cranelift-module = { path = "lib/module", version = "0.15.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.15.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.15.0" }
-cranelift = { path = "lib/umbrella", version = "0.15.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.16.0" }
+cranelift-reader = { path = "lib/reader", version = "0.16.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.16.0" }
+cranelift-wasm = { path = "lib/wasm", version = "0.16.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.16.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.16.0" }
+cranelift-module = { path = "lib/module", version = "0.16.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.16.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.16.0" }
+cranelift = { path = "lib/umbrella", version = "0.16.0" }
 filecheck = "0.3.0"
 docopt = "1"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 75c6778201..f845210cd6 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.15.0"
+version="0.16.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index aa445322fe..2760307674 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.15.0"
+version = "0.16.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.15.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.16.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index 30bffdeade..f6af67f99b 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.15.0"
+version = "0.16.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 538c59b5fc..30fac496bb 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.15.0"
+version = "0.16.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.15.0" }
-cranelift-module = { path = "../module", version = "0.15.0" }
+cranelift-codegen = { path = "../codegen", version = "0.16.0" }
+cranelift-module = { path = "../module", version = "0.16.0" }
 faerie = "0.4.3"
 goblin = "0.0.15"
 failure = "0.1.1"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 66c3a42863..17d8f70445 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.15.0"
+version = "0.16.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.15.0" }
-cranelift-reader = { path = "../reader", version = "0.15.0" }
+cranelift-codegen = { path = "../codegen", version = "0.16.0" }
+cranelift-reader = { path = "../reader", version = "0.16.0" }
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index d57443db96..d013086856 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.15.0"
+version = "0.16.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.16.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index e4a4e6833b..3b596e9fce 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.15.0"
+version = "0.16.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.15.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.16.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.16.0", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
 failure = "0.1.1"
 
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index c42b23fb81..6ac83e8f7f 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.15.0"
+version = "0.16.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -8,7 +8,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.16.0", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 8b33740741..1c20b4c95e 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.15.0"
+version = "0.16.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.15.0" }
+cranelift-codegen = { path = "../codegen", version = "0.16.0" }
 target-lexicon = "0.0.3"
 
 [badges]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 968c411e4c..906fee1a23 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.15.0"
+version = "0.16.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,9 +9,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.15.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.15.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.16.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.16.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.16.0", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index c7391d29ce..4bad79ed37 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.15.0"
+version = "0.16.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,8 +10,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.15.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.16.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.16.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 4c287cc05d..9f7823186b 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.15.0"
+version = "0.16.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.2", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.15.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.15.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.16.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.16.0", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 1081d061734c91d464f24e8c1884128bbc21840a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 17 Jul 2018 22:44:00 -0700
Subject: [PATCH 1920/3084] Use the new realm-switching ABI in the baldrdash
 calling convention.

---
 lib/codegen/src/isa/x86/abi.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs
index 8bb1decd85..42cacf27a8 100644
--- a/lib/codegen/src/isa/x86/abi.rs
+++ b/lib/codegen/src/isa/x86/abi.rs
@@ -103,7 +103,7 @@ impl ArgAssigner for Args {
                         .into()
                 }
                 // This is SpiderMonkey's `WasmTableCallSigReg`.
-                ArgumentPurpose::SignatureId => return ArgumentLoc::Reg(RU::rbx as RegUnit).into(),
+                ArgumentPurpose::SignatureId => return ArgumentLoc::Reg(RU::r10 as RegUnit).into(),
                 _ => {}
             }
         }

From c068721964dcfcee79f47149903ab5bd5bec0901 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 18 Jul 2018 13:32:56 -0700
Subject: [PATCH 1921/3084] Bump version to 0.16.1

---
 cranelift/Cargo.toml     | 22 +++++++++++-----------
 cranelift/publish-all.sh |  2 +-
 lib/codegen/Cargo.toml   |  4 ++--
 lib/entity/Cargo.toml    |  2 +-
 lib/faerie/Cargo.toml    |  6 +++---
 lib/filetests/Cargo.toml |  6 +++---
 lib/frontend/Cargo.toml  |  4 ++--
 lib/module/Cargo.toml    |  6 +++---
 lib/native/Cargo.toml    |  4 ++--
 lib/reader/Cargo.toml    |  4 ++--
 lib/simplejit/Cargo.toml |  8 ++++----
 lib/umbrella/Cargo.toml  |  6 +++---
 lib/wasm/Cargo.toml      |  6 +++---
 13 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 59dcbf858b..982687738e 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.16.0"
+version = "0.16.1"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,16 +14,16 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.16.0" }
-cranelift-reader = { path = "lib/reader", version = "0.16.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.16.0" }
-cranelift-wasm = { path = "lib/wasm", version = "0.16.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.16.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.16.0" }
-cranelift-module = { path = "lib/module", version = "0.16.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.16.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.16.0" }
-cranelift = { path = "lib/umbrella", version = "0.16.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.16.1" }
+cranelift-reader = { path = "lib/reader", version = "0.16.1" }
+cranelift-frontend = { path = "lib/frontend", version = "0.16.1" }
+cranelift-wasm = { path = "lib/wasm", version = "0.16.1", optional = true }
+cranelift-native = { path = "lib/native", version = "0.16.1" }
+cranelift-filetests = { path = "lib/filetests", version = "0.16.1" }
+cranelift-module = { path = "lib/module", version = "0.16.1" }
+cranelift-faerie = { path = "lib/faerie", version = "0.16.1" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.16.1" }
+cranelift = { path = "lib/umbrella", version = "0.16.1" }
 filecheck = "0.3.0"
 docopt = "1"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index f845210cd6..52a2ecd496 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.16.0"
+version="0.16.1"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 2760307674..fbd442b5b1 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.16.0"
+version = "0.16.1"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.16.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.16.1", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index f6af67f99b..cb52df1f89 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.16.0"
+version = "0.16.1"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 30fac496bb..6be4989023 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.16.0"
+version = "0.16.1"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.0" }
-cranelift-module = { path = "../module", version = "0.16.0" }
+cranelift-codegen = { path = "../codegen", version = "0.16.1" }
+cranelift-module = { path = "../module", version = "0.16.1" }
 faerie = "0.4.3"
 goblin = "0.0.15"
 failure = "0.1.1"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 17d8f70445..de65853d4c 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.16.0"
+version = "0.16.1"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.0" }
-cranelift-reader = { path = "../reader", version = "0.16.0" }
+cranelift-codegen = { path = "../codegen", version = "0.16.1" }
+cranelift-reader = { path = "../reader", version = "0.16.1" }
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index d013086856..2bba26a18d 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.16.0"
+version = "0.16.1"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 3b596e9fce..0b40d449eb 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.16.0"
+version = "0.16.1"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.16.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.16.1", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
 failure = "0.1.1"
 
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 6ac83e8f7f..40e635ecf4 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.16.0"
+version = "0.16.1"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -8,7 +8,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 1c20b4c95e..19a80c77cb 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.16.0"
+version = "0.16.1"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.0" }
+cranelift-codegen = { path = "../codegen", version = "0.16.1" }
 target-lexicon = "0.0.3"
 
 [badges]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 906fee1a23..8ee0532dbf 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.16.0"
+version = "0.16.1"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,9 +9,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.16.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.16.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
+cranelift-module = { path = "../module", version = "0.16.1", default-features = false }
+cranelift-native = { path = "../native", version = "0.16.1", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index 4bad79ed37..ea50e5b730 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.16.0"
+version = "0.16.1"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,8 +10,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.16.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.16.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 9f7823186b..3a0556ae8a 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.16.0"
+version = "0.16.1"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.2", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.16.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.16.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.16.1", default-features = false }
 hashmap_core = { version = "0.1.8", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 03159a920050f740740c62035bc1e66a76459577 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 12 Jul 2018 19:59:06 +0200
Subject: [PATCH 1922/3084] Misc refactorings when looking at the wasm code;

---
 cranelift/src/wasm.rs             |  4 ++-
 lib/codegen/src/dominator_tree.rs |  2 +-
 lib/codegen/src/preopt.rs         | 28 ++++++++------------
 lib/wasm/src/module_translator.rs | 44 +++++++++----------------------
 4 files changed, 28 insertions(+), 50 deletions(-)

diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 1e900537ff..057c167fbc 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -141,8 +141,8 @@ fn handle_module(
 
     let num_func_imports = dummy_environ.get_num_func_imports();
     let mut total_module_code_size = 0;
+    let mut context = Context::new();
     for (def_index, func) in dummy_environ.info.function_bodies.iter().enumerate() {
-        let mut context = Context::new();
         context.func = func.clone();
 
         let func_index = num_func_imports + def_index;
@@ -180,6 +180,8 @@ fn handle_module(
             println!("{}", context.func.display(fisa.isa));
             vprintln!(flag_verbose, "");
         }
+
+        context.clear();
     }
 
     if !flag_check_translation && flag_print_size {
diff --git a/lib/codegen/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs
index c8dec582aa..ba066fe496 100644
--- a/lib/codegen/src/dominator_tree.rs
+++ b/lib/codegen/src/dominator_tree.rs
@@ -271,7 +271,7 @@ impl DominatorTree {
         //
         // There are two ways of viewing the CFG as a graph:
         //
-        // 1. Each EBB is a node, with outgoing edges for all the branches in the EBB>
+        // 1. Each EBB is a node, with outgoing edges for all the branches in the EBB.
         // 2. Each basic block is a node, with outgoing edges for the single branch at the end of
         //    the BB. (An EBB is a linear sequence of basic blocks).
         //
diff --git a/lib/codegen/src/preopt.rs b/lib/codegen/src/preopt.rs
index b96c3d9531..e252659d35 100644
--- a/lib/codegen/src/preopt.rs
+++ b/lib/codegen/src/preopt.rs
@@ -517,25 +517,19 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
             ..
         } => {
             // Fold away a redundant `bint`.
-            let maybe = {
+            let condition_def = {
                 let args = pos.func.dfg.inst_args(inst);
-                if let ValueDef::Result(def_inst, _) = pos.func.dfg.value_def(args[0]) {
-                    if let InstructionData::Unary {
-                        opcode: Opcode::Bint,
-                        arg: bool_val,
-                    } = pos.func.dfg[def_inst]
-                    {
-                        Some(bool_val)
-                    } else {
-                        None
-                    }
-                } else {
-                    None
-                }
+                pos.func.dfg.value_def(args[0])
             };
-            if let Some(bool_val) = maybe {
-                let args = pos.func.dfg.inst_args_mut(inst);
-                args[0] = bool_val;
+            if let ValueDef::Result(def_inst, _) = condition_def {
+                if let InstructionData::Unary {
+                    opcode: Opcode::Bint,
+                    arg: bool_val,
+                } = pos.func.dfg[def_inst]
+                {
+                    let args = pos.func.dfg.inst_args_mut(inst);
+                    args[0] = bool_val;
+                }
             }
         }
         _ => {}
diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs
index 5010aba87f..907a92660b 100644
--- a/lib/wasm/src/module_translator.rs
+++ b/lib/wasm/src/module_translator.rs
@@ -94,10 +94,19 @@ pub fn translate_module<'data>(
             ParserState::BeginSection {
                 code: SectionCode::Code,
                 ..
-            } => {
-                // The code section begins
-                break;
-            }
+            } => loop {
+                match *parser.read() {
+                    ParserState::BeginFunctionBody { .. } => {}
+                    ParserState::EndSection => break,
+                    ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
+                    ref s => panic!("wrong content in code section: {:?}", s),
+                }
+                let mut reader = parser.create_binary_reader();
+                let size = reader.bytes_remaining();
+                environ.define_function_body(reader
+                    .read_bytes(size)
+                    .map_err(WasmError::from_binary_reader_error)?)?;
+            },
             ParserState::EndSection => {
                 next_input = ParserInput::Default;
             }
@@ -119,31 +128,4 @@ pub fn translate_module<'data>(
             _ => panic!("wrong content in the preamble"),
         };
     }
-    // At this point we've entered the code section
-    loop {
-        match *parser.read() {
-            ParserState::BeginFunctionBody { .. } => {}
-            ParserState::EndSection => break,
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("wrong content in code section: {:?}", s),
-        }
-        let mut reader = parser.create_binary_reader();
-        let size = reader.bytes_remaining();
-        environ.define_function_body(reader
-            .read_bytes(size)
-            .map_err(WasmError::from_binary_reader_error)?)?;
-    }
-    loop {
-        match *parser.read() {
-            ParserState::BeginSection {
-                code: SectionCode::Data,
-                ..
-            } => {
-                parse_data_section(&mut parser, environ)?;
-            }
-            ParserState::EndWasm => break,
-            _ => (),
-        }
-    }
-    Ok(())
 }

From 7b290cd9000073383d67dcb3f5f499d269c92cd5 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 19 Jul 2018 14:56:58 +0200
Subject: [PATCH 1923/3084] Move the code section parsing into its own
 function;

---
 lib/wasm/src/module_translator.rs   | 20 ++++----------------
 lib/wasm/src/sections_translator.rs | 21 +++++++++++++++++++++
 2 files changed, 25 insertions(+), 16 deletions(-)

diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs
index 907a92660b..023d14dba1 100644
--- a/lib/wasm/src/module_translator.rs
+++ b/lib/wasm/src/module_translator.rs
@@ -3,9 +3,9 @@
 use cranelift_codegen::timing;
 use environ::{ModuleEnvironment, WasmError, WasmResult};
 use sections_translator::{
-    parse_data_section, parse_elements_section, parse_export_section, parse_function_section,
-    parse_function_signatures, parse_global_section, parse_import_section, parse_memory_section,
-    parse_start_section, parse_table_section,
+    parse_code_section, parse_data_section, parse_elements_section, parse_export_section,
+    parse_function_section, parse_function_signatures, parse_global_section, parse_import_section,
+    parse_memory_section, parse_start_section, parse_table_section,
 };
 use wasmparser::{Parser, ParserInput, ParserState, SectionCode, WasmDecoder};
 
@@ -94,19 +94,7 @@ pub fn translate_module<'data>(
             ParserState::BeginSection {
                 code: SectionCode::Code,
                 ..
-            } => loop {
-                match *parser.read() {
-                    ParserState::BeginFunctionBody { .. } => {}
-                    ParserState::EndSection => break,
-                    ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-                    ref s => panic!("wrong content in code section: {:?}", s),
-                }
-                let mut reader = parser.create_binary_reader();
-                let size = reader.bytes_remaining();
-                environ.define_function_body(reader
-                    .read_bytes(size)
-                    .map_err(WasmError::from_binary_reader_error)?)?;
-            },
+            } => parse_code_section(&mut parser, environ)?,
             ParserState::EndSection => {
                 next_input = ParserInput::Default;
             }
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index c17b0e680b..6bfc55372b 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -397,3 +397,24 @@ pub fn parse_elements_section(
     }
     Ok(())
 }
+
+/// Parses every function body in the code section and defines the corresponding function.
+pub fn parse_code_section<'data>(
+    parser: &mut Parser<'data>,
+    environ: &mut ModuleEnvironment<'data>,
+) -> WasmResult<()> {
+    loop {
+        match *parser.read() {
+            ParserState::BeginFunctionBody { .. } => {}
+            ParserState::EndSection => break,
+            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
+            ref s => panic!("wrong content in code section: {:?}", s),
+        }
+        let mut reader = parser.create_binary_reader();
+        let size = reader.bytes_remaining();
+        environ.define_function_body(reader
+            .read_bytes(size)
+            .map_err(WasmError::from_binary_reader_error)?)?;
+    }
+    Ok(())
+}

From 06319b415a1a836e9d854b305314fd86df2b990a Mon Sep 17 00:00:00 2001
From: data-pup 
Date: Thu, 19 Jul 2018 12:56:23 -0400
Subject: [PATCH 1924/3084] Added initial Rust codegen-meta implementation.
 (#403)

* Added initial Rust codegen-meta implementation.

* Replace 'Cretonne' in comments.

* Prevent iterator overflow.

* 1.25.0 compatibility changes.

* Implemented debug traits for type variants.

* Added consistent comments.

* Cleaned up a loop via clippy fix.

* Added new license to codegen-meta Cargo.toml

* Edited lane type iterator `next` method.

* Removed functions that are not needed in Rust, and edited desc.

* Debug trait derived for valuetype.

* Added comments for iterator types in the base types submodule.

* Numbering is now handled in the cdsl/types.rs file.

* Moved type number logic into cdsl/types.

* Repeating the lane change cleanup.

* Removed codegen-meta crate from codegen deps.

* Typo fix.

* Addressing a patch note.

* Addressing patch note.

* Lowercase in vector names.

* Fixing a comment bug.

* Added a copy of the license file.

* Formatting changes.

* Cleaned up the vector type numbering.

* 1.25 compatibility.

* Fixed pattern match arms.
---
 lib/codegen-meta/Cargo.toml            |  18 +
 lib/codegen-meta/LICENSE               | 219 ++++++++++++
 lib/codegen-meta/src/base/mod.rs       |   3 +
 lib/codegen-meta/src/base/types.rs     | 188 ++++++++++
 lib/codegen-meta/src/cdsl/mod.rs       |  38 ++
 lib/codegen-meta/src/cdsl/types.rs     | 473 +++++++++++++++++++++++++
 lib/codegen-meta/src/error.rs          |  47 +++
 lib/codegen-meta/src/gen_build_deps.rs |  48 +++
 lib/codegen-meta/src/gen_types.rs      |  74 ++++
 lib/codegen-meta/src/lib.rs            |   7 +
 lib/codegen-meta/src/srcgen.rs         | 316 +++++++++++++++++
 lib/codegen/Cargo.toml                 |   3 +
 lib/codegen/build.rs                   |  27 +-
 13 files changed, 1460 insertions(+), 1 deletion(-)
 create mode 100644 lib/codegen-meta/Cargo.toml
 create mode 100644 lib/codegen-meta/LICENSE
 create mode 100644 lib/codegen-meta/src/base/mod.rs
 create mode 100644 lib/codegen-meta/src/base/types.rs
 create mode 100644 lib/codegen-meta/src/cdsl/mod.rs
 create mode 100644 lib/codegen-meta/src/cdsl/types.rs
 create mode 100644 lib/codegen-meta/src/error.rs
 create mode 100644 lib/codegen-meta/src/gen_build_deps.rs
 create mode 100644 lib/codegen-meta/src/gen_types.rs
 create mode 100644 lib/codegen-meta/src/lib.rs
 create mode 100644 lib/codegen-meta/src/srcgen.rs

diff --git a/lib/codegen-meta/Cargo.toml b/lib/codegen-meta/Cargo.toml
new file mode 100644
index 0000000000..9d75e91ce8
--- /dev/null
+++ b/lib/codegen-meta/Cargo.toml
@@ -0,0 +1,18 @@
+[package]
+name = "cranelift-codegen-meta"
+authors = ["The Cranelift Project Developers"]
+version = "0.15.0"
+description = "Metaprogram for cranelift-codegen code generator library"
+license = "Apache-2.0 WITH LLVM-exception"
+documentation = "https://cranelift.readthedocs.io/"
+repository = "https://github.com/CraneStation/cranelift"
+keywords = ["compile", "compiler", "jit"]
+
+[dependencies]
+# It is a goal of the cranelift-codegen crate to have minimal external dependencies.
+# Please don't add any unless they are essential to the task of creating binary
+# machine code.
+
+[badges]
+maintenance = { status = "experimental" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/codegen-meta/LICENSE b/lib/codegen-meta/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/codegen-meta/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
diff --git a/lib/codegen-meta/src/base/mod.rs b/lib/codegen-meta/src/base/mod.rs
new file mode 100644
index 0000000000..a75614d9c8
--- /dev/null
+++ b/lib/codegen-meta/src/base/mod.rs
@@ -0,0 +1,3 @@
+//! Definitions for the base Cranelift language.
+
+pub mod types;
diff --git a/lib/codegen-meta/src/base/types.rs b/lib/codegen-meta/src/base/types.rs
new file mode 100644
index 0000000000..7ce42b1978
--- /dev/null
+++ b/lib/codegen-meta/src/base/types.rs
@@ -0,0 +1,188 @@
+//! This module predefines all the Cranelift scalar types.
+
+#[derive(Debug, Clone, Copy, Eq, PartialEq)]
+pub enum Bool {
+    /// 1-bit bool.
+    B1 = 1,
+    /// 8-bit bool.
+    B8 = 8,
+    /// 16-bit bool.
+    B16 = 16,
+    /// 32-bit bool.
+    B32 = 32,
+    /// 64-bit bool.
+    B64 = 64,
+}
+
+/// This provides an iterator through all of the supported bool variants.
+pub struct BoolIterator {
+    index: u8,
+}
+
+impl BoolIterator {
+    pub fn new() -> Self {
+        Self { index: 0 }
+    }
+}
+
+impl Iterator for BoolIterator {
+    type Item = Bool;
+    fn next(&mut self) -> Option {
+        let res = match self.index {
+            0 => Some(Bool::B1),
+            1 => Some(Bool::B8),
+            2 => Some(Bool::B16),
+            3 => Some(Bool::B32),
+            4 => Some(Bool::B64),
+            _ => return None,
+        };
+        self.index += 1;
+        res
+    }
+}
+
+#[derive(Debug, Clone, Copy, Eq, PartialEq)]
+pub enum Int {
+    /// 8-bit int.
+    I8 = 8,
+    /// 16-bit int.
+    I16 = 16,
+    /// 32-bit int.
+    I32 = 32,
+    /// 64-bit int.
+    I64 = 64,
+}
+
+/// This provides an iterator through all of the supported int variants.
+pub struct IntIterator {
+    index: u8,
+}
+
+impl IntIterator {
+    pub fn new() -> Self {
+        Self { index: 0 }
+    }
+}
+
+impl Iterator for IntIterator {
+    type Item = Int;
+    fn next(&mut self) -> Option {
+        let res = match self.index {
+            0 => Some(Int::I8),
+            1 => Some(Int::I16),
+            2 => Some(Int::I32),
+            3 => Some(Int::I64),
+            _ => return None,
+        };
+        self.index += 1;
+        res
+    }
+}
+
+#[derive(Debug, Clone, Copy, Eq, PartialEq)]
+pub enum Float {
+    F32 = 32,
+    F64 = 64,
+}
+
+/// Iterator through the variants of the Float enum.
+pub struct FloatIterator {
+    index: u8,
+}
+
+impl FloatIterator {
+    pub fn new() -> Self {
+        Self { index: 0 }
+    }
+}
+
+/// This provides an iterator through all of the supported float variants.
+impl Iterator for FloatIterator {
+    type Item = Float;
+    fn next(&mut self) -> Option {
+        let res = match self.index {
+            0 => Some(Float::F32),
+            1 => Some(Float::F64),
+            _ => return None,
+        };
+        self.index += 1;
+        res
+    }
+}
+
+/// A type representing CPU flags.
+///
+/// Flags can't be stored in memory.
+#[derive(Debug, Clone, Copy, Eq, PartialEq)]
+pub enum Flag {
+    /// CPU flags from an integer comparison.
+    IFlags,
+    /// CPU flags from a floating point comparison.
+    FFlags,
+}
+
+/// Iterator through the variants of the Flag enum.
+pub struct FlagIterator {
+    index: u8,
+}
+
+impl FlagIterator {
+    pub fn new() -> Self {
+        Self { index: 0 }
+    }
+}
+
+impl Iterator for FlagIterator {
+    type Item = Flag;
+    fn next(&mut self) -> Option {
+        let res = match self.index {
+            0 => Some(Flag::IFlags),
+            1 => Some(Flag::FFlags),
+            _ => return None,
+        };
+        self.index += 1;
+        res
+    }
+}
+
+#[cfg(test)]
+mod iter_tests {
+    use super::*;
+
+    #[test]
+    fn bool_iter_works() {
+        let mut bool_iter = BoolIterator::new();
+        assert_eq!(bool_iter.next(), Some(Bool::B1));
+        assert_eq!(bool_iter.next(), Some(Bool::B8));
+        assert_eq!(bool_iter.next(), Some(Bool::B16));
+        assert_eq!(bool_iter.next(), Some(Bool::B32));
+        assert_eq!(bool_iter.next(), Some(Bool::B64));
+        assert_eq!(bool_iter.next(), None);
+    }
+
+    #[test]
+    fn int_iter_works() {
+        let mut int_iter = IntIterator::new();
+        assert_eq!(int_iter.next(), Some(Int::I8));
+        assert_eq!(int_iter.next(), Some(Int::I16));
+        assert_eq!(int_iter.next(), Some(Int::I32));
+        assert_eq!(int_iter.next(), Some(Int::I64));
+        assert_eq!(int_iter.next(), None);
+    }
+
+    #[test]
+    fn float_iter_works() {
+        let mut float_iter = FloatIterator::new();
+        assert_eq!(float_iter.next(), Some(Float::F32));
+        assert_eq!(float_iter.next(), Some(Float::F64));
+        assert_eq!(float_iter.next(), None);
+    }
+
+    #[test]
+    fn flag_iter_works() {
+        let mut flag_iter = FlagIterator::new();
+        assert_eq!(flag_iter.next(), Some(Flag::IFlags));
+        assert_eq!(flag_iter.next(), Some(Flag::FFlags));
+        assert_eq!(flag_iter.next(), None);
+    }
+}
diff --git a/lib/codegen-meta/src/cdsl/mod.rs b/lib/codegen-meta/src/cdsl/mod.rs
new file mode 100644
index 0000000000..8f087a7c3d
--- /dev/null
+++ b/lib/codegen-meta/src/cdsl/mod.rs
@@ -0,0 +1,38 @@
+//! Cranelift DSL classes.
+//!
+//! This module defines the classes that are used to define Cranelift
+//! instructions and other entitties.
+
+pub mod types;
+
+/// Convert the string `s` to CamelCase.
+fn _camel_case(s: &str) -> String {
+    let mut output_chars = String::with_capacity(s.len());
+
+    let mut capitalize = true;
+    for curr_char in s.chars() {
+        if curr_char == '_' {
+            capitalize = true;
+        } else {
+            if capitalize {
+                output_chars.extend(curr_char.to_uppercase());
+            } else {
+                output_chars.push(curr_char);
+            }
+            capitalize = false;
+        }
+    }
+
+    output_chars
+}
+
+#[cfg(test)]
+mod tests {
+    use super::_camel_case as camel_case;
+
+    #[test]
+    fn camel_case_works() {
+        assert_eq!(camel_case("x"), "X");
+        assert_eq!(camel_case("camel_case"), "CamelCase");
+    }
+}
diff --git a/lib/codegen-meta/src/cdsl/types.rs b/lib/codegen-meta/src/cdsl/types.rs
new file mode 100644
index 0000000000..90438e9179
--- /dev/null
+++ b/lib/codegen-meta/src/cdsl/types.rs
@@ -0,0 +1,473 @@
+//! Cranelift ValueType hierarchy
+
+// Temporary disabled: Unused at the moment.
+// use std::collections::HashMap;
+
+use std::fmt;
+
+use base::types as base_types;
+
+// Numbering scheme for value types:
+//
+// 0: Void
+// 0x01-0x6f: Special types
+// 0x70-0x7f: Lane types
+// 0x80-0xff: Vector types
+//
+// Vector types are encoded with the lane type in the low 4 bits and log2(lanes)
+// in the high 4 bits, giving a range of 2-256 lanes.
+static LANE_BASE: u8 = 0x70;
+
+// Rust name prefix used for the `rust_name` method.
+static _RUST_NAME_PREFIX: &'static str = "ir::types::";
+
+// ValueType variants (i8, i32, ...) are provided in `base::types.rs`.
+
+/// A concrete SSA value type.
+///
+/// All SSA values have a type that is described by an instance of `ValueType`
+/// or one of its subclasses.
+#[derive(Debug)]
+pub enum ValueType {
+    BV(BVType),
+    Lane(LaneType),
+    Special(SpecialType),
+    Vector(VectorType),
+}
+
+impl ValueType {
+    /// Iterate through all of the lane types.
+    pub fn all_lane_types() -> LaneTypeIterator {
+        LaneTypeIterator::new()
+    }
+
+    /// Iterate through all of the special types (neither lanes nor vectors).
+    pub fn all_special_types() -> SpecialTypeIterator {
+        SpecialTypeIterator::new()
+    }
+
+    /// Return a string containing the documentation comment for this type.
+    pub fn doc(&self) -> String {
+        match *self {
+            ValueType::BV(ref b) => b.doc(),
+            ValueType::Lane(l) => l.doc(),
+            ValueType::Special(s) => s.doc(),
+            ValueType::Vector(ref v) => v.doc(),
+        }
+    }
+
+    /// Return the number of bits in a lane.
+    pub fn lane_bits(&self) -> u64 {
+        match *self {
+            ValueType::BV(ref b) => b.lane_bits(),
+            ValueType::Lane(l) => l.lane_bits(),
+            ValueType::Special(s) => s.lane_bits(),
+            ValueType::Vector(ref v) => v.lane_bits(),
+        }
+    }
+
+    /// Return the number of lanes.
+    pub fn lane_count(&self) -> u64 {
+        match *self {
+            ValueType::Vector(ref v) => v.lane_count(),
+            _ => 1,
+        }
+    }
+
+    /// Find the number of bytes that this type occupies in memory.
+    pub fn membytes(&self) -> u64 {
+        self.width() / 8
+    }
+
+    /// Get the name of this type.
+    pub fn name(&self) -> String {
+        match *self {
+            ValueType::BV(ref b) => b.name(),
+            ValueType::Lane(l) => l.name(),
+            ValueType::Special(s) => s.name(),
+            ValueType::Vector(ref v) => v.name(),
+        }
+    }
+
+    /// Find the unique number associated with this type.
+    pub fn number(&self) -> Option {
+        match *self {
+            ValueType::BV(_) => None,
+            ValueType::Lane(l) => Some(l.number()),
+            ValueType::Special(s) => Some(s.number()),
+            ValueType::Vector(ref v) => Some(v.number()),
+        }
+    }
+
+    /// Return the name of this type for generated Rust source files.
+    pub fn _rust_name(&self) -> String {
+        format!("{}{}", _RUST_NAME_PREFIX, self.name().to_uppercase())
+    }
+
+    /// Return true iff:
+    ///     1. self and other have equal number of lanes
+    ///     2. each lane in self has at least as many bits as a lane in other
+    pub fn _wider_or_equal(&self, rhs: &ValueType) -> bool {
+        (self.lane_count() == rhs.lane_count()) && (self.lane_bits() >= rhs.lane_bits())
+    }
+
+    /// Return the total number of bits of an instance of this type.
+    pub fn width(&self) -> u64 {
+        self.lane_count() * self.lane_bits()
+    }
+}
+
+impl fmt::Display for ValueType {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "{}", self.name())
+    }
+}
+
+/// Create a ValueType from a given bitvector type.
+impl From for ValueType {
+    fn from(bv: BVType) -> Self {
+        ValueType::BV(bv)
+    }
+}
+
+/// Create a ValueType from a given lane type.
+impl From for ValueType {
+    fn from(lane: LaneType) -> Self {
+        ValueType::Lane(lane)
+    }
+}
+
+/// Create a ValueType from a given special type.
+impl From for ValueType {
+    fn from(spec: SpecialType) -> Self {
+        ValueType::Special(spec)
+    }
+}
+
+/// Create a ValueType from a given vector type.
+impl From for ValueType {
+    fn from(vector: VectorType) -> Self {
+        ValueType::Vector(vector)
+    }
+}
+
+/// A concrete scalar type that can appear as a vector lane too.
+#[derive(Clone, Copy)]
+pub enum LaneType {
+    BoolType(base_types::Bool),
+    FloatType(base_types::Float),
+    IntType(base_types::Int),
+}
+
+impl LaneType {
+    /// Return a string containing the documentation comment for this lane type.
+    pub fn doc(&self) -> String {
+        match *self {
+            LaneType::BoolType(_) => format!("A boolean type with {} bits.", self.lane_bits()),
+            LaneType::FloatType(base_types::Float::F32) => String::from(
+                "A 32-bit floating point type represented in the IEEE 754-2008
+                *binary32* interchange format. This corresponds to the :c:type:`float`
+                type in most C implementations.",
+            ),
+            LaneType::FloatType(base_types::Float::F64) => String::from(
+                "A 64-bit floating point type represented in the IEEE 754-2008
+                *binary64* interchange format. This corresponds to the :c:type:`double`
+                type in most C implementations.",
+            ),
+            LaneType::IntType(_) if self.lane_bits() < 32 => format!(
+                "An integer type with {} bits.
+                WARNING: arithmetic on {}bit integers is incomplete",
+                self.lane_bits(),
+                self.lane_bits()
+            ),
+            LaneType::IntType(_) => format!("An integer type with {} bits.", self.lane_bits()),
+        }
+    }
+
+    /// Return the number of bits in a lane.
+    pub fn lane_bits(&self) -> u64 {
+        match *self {
+            LaneType::BoolType(ref b) => *b as u64,
+            LaneType::FloatType(ref f) => *f as u64,
+            LaneType::IntType(ref i) => *i as u64,
+        }
+    }
+
+    /// Get the name of this lane type.
+    pub fn name(&self) -> String {
+        match *self {
+            LaneType::BoolType(_) => format!("b{}", self.lane_bits()),
+            LaneType::FloatType(_) => format!("f{}", self.lane_bits()),
+            LaneType::IntType(_) => format!("i{}", self.lane_bits()),
+        }
+    }
+
+    /// Find the unique number associated with this lane type.
+    pub fn number(&self) -> u8 {
+        LANE_BASE + match *self {
+            LaneType::BoolType(base_types::Bool::B1) => 0,
+            LaneType::BoolType(base_types::Bool::B8) => 1,
+            LaneType::BoolType(base_types::Bool::B16) => 2,
+            LaneType::BoolType(base_types::Bool::B32) => 3,
+            LaneType::BoolType(base_types::Bool::B64) => 4,
+            LaneType::IntType(base_types::Int::I8) => 5,
+            LaneType::IntType(base_types::Int::I16) => 6,
+            LaneType::IntType(base_types::Int::I32) => 7,
+            LaneType::IntType(base_types::Int::I64) => 8,
+            LaneType::FloatType(base_types::Float::F32) => 9,
+            LaneType::FloatType(base_types::Float::F64) => 10,
+        }
+    }
+}
+
+impl fmt::Debug for LaneType {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        let inner_msg = format!("bits={}", self.lane_bits());
+        write!(
+            f,
+            "{}",
+            match *self {
+                LaneType::BoolType(_) => format!("BoolType({})", inner_msg),
+                LaneType::FloatType(_) => format!("FloatType({})", inner_msg),
+                LaneType::IntType(_) => format!("IntType({})", inner_msg),
+            }
+        )
+    }
+}
+
+/// Create a LaneType from a given bool variant.
+impl From for LaneType {
+    fn from(b: base_types::Bool) -> Self {
+        LaneType::BoolType(b)
+    }
+}
+
+/// Create a LaneType from a given float variant.
+impl From for LaneType {
+    fn from(f: base_types::Float) -> Self {
+        LaneType::FloatType(f)
+    }
+}
+
+/// Create a LaneType from a given int variant.
+impl From for LaneType {
+    fn from(i: base_types::Int) -> Self {
+        LaneType::IntType(i)
+    }
+}
+
+/// An iterator for different lane types.
+pub struct LaneTypeIterator {
+    bool_iter: base_types::BoolIterator,
+    int_iter: base_types::IntIterator,
+    float_iter: base_types::FloatIterator,
+}
+
+impl LaneTypeIterator {
+    /// Create a new lane type iterator.
+    fn new() -> Self {
+        Self {
+            bool_iter: base_types::BoolIterator::new(),
+            int_iter: base_types::IntIterator::new(),
+            float_iter: base_types::FloatIterator::new(),
+        }
+    }
+}
+
+impl Iterator for LaneTypeIterator {
+    type Item = LaneType;
+    fn next(&mut self) -> Option {
+        if let Some(b) = self.bool_iter.next() {
+            Some(LaneType::from(b))
+        } else if let Some(i) = self.int_iter.next() {
+            Some(LaneType::from(i))
+        } else if let Some(f) = self.float_iter.next() {
+            Some(LaneType::from(f))
+        } else {
+            None
+        }
+    }
+}
+
+/// A concrete SIMD vector type.
+///
+/// A vector type has a lane type which is an instance of `LaneType`,
+/// and a positive number of lanes.
+pub struct VectorType {
+    base: LaneType,
+    lanes: u64,
+}
+
+impl VectorType {
+    /// Initialize a new integer type with `n` bits.
+    pub fn new(base: LaneType, lanes: u64) -> VectorType {
+        VectorType { base, lanes }
+    }
+
+    /// Return a string containing the documentation comment for this vector type.
+    pub fn doc(&self) -> String {
+        format!(
+            "A SIMD vector with {} lanes containing a `{}` each.",
+            self.lane_count(),
+            self.base.name()
+        )
+    }
+
+    /// Return the number of bits in a lane.
+    pub fn lane_bits(&self) -> u64 {
+        self.base.lane_bits()
+    }
+
+    /// Return the number of lanes.
+    pub fn lane_count(&self) -> u64 {
+        self.lanes
+    }
+
+    /// Get the name of this vector type.
+    pub fn name(&self) -> String {
+        format!("{}x{}", self.base.name(), self.lane_count())
+    }
+
+    /// Find the unique number associated with this vector type.
+    ///
+    /// Vector types are encoded with the lane type in the low 4 bits and
+    /// log2(lanes) in the high 4 bits, giving a range of 2-256 lanes.
+    pub fn number(&self) -> u8 {
+        let lanes_log_2: u32 = 63 - self.lane_count().leading_zeros();
+        let base_num = u32::from(self.base.number());
+        let num = (lanes_log_2 << 4) + base_num;
+        num as u8
+    }
+}
+
+impl fmt::Debug for VectorType {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(
+            f,
+            "VectorType(base={}, lanes={})",
+            self.base.name(),
+            self.lane_count()
+        )
+    }
+}
+
+/// A flat bitvector type. Used for semantics description only.
+pub struct BVType {
+    bits: u64,
+}
+
+impl BVType {
+    /// Initialize a new bitvector type with `n` bits.
+    pub fn _new(bits: u64) -> Self {
+        Self { bits }
+    }
+
+    /// Return a string containing the documentation comment for this bitvector type.
+    pub fn doc(&self) -> String {
+        format!("A bitvector type with {} bits.", self.bits)
+    }
+
+    /// Return the number of bits in a lane.
+    pub fn lane_bits(&self) -> u64 {
+        self.bits
+    }
+
+    /// Get the name of this bitvector type.
+    pub fn name(&self) -> String {
+        format!("bv{}", self.bits)
+    }
+}
+
+impl fmt::Debug for BVType {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "BVType(bits={})", self.lane_bits())
+    }
+}
+
+/// A concrete scalar type that is neither a vector nor a lane type.
+///
+/// Special types cannot be used to form vectors.
+#[derive(Clone, Copy)]
+pub enum SpecialType {
+    Flag(base_types::Flag),
+}
+
+impl SpecialType {
+    /// Return a string containing the documentation comment for this special type.
+    pub fn doc(&self) -> String {
+        match *self {
+            SpecialType::Flag(base_types::Flag::IFlags) => String::from(
+                "CPU flags representing the result of an integer comparison. These flags
+                can be tested with an :type:`intcc` condition code.",
+            ),
+            SpecialType::Flag(base_types::Flag::FFlags) => String::from(
+                "CPU flags representing the result of a floating point comparison. These
+                flags can be tested with a :type:`floatcc` condition code.",
+            ),
+        }
+    }
+
+    /// Return the number of bits in a lane.
+    pub fn lane_bits(&self) -> u64 {
+        match *self {
+            SpecialType::Flag(_) => 0,
+        }
+    }
+
+    /// Get the name of this special type.
+    pub fn name(&self) -> String {
+        match *self {
+            SpecialType::Flag(base_types::Flag::IFlags) => "iflags".to_string(),
+            SpecialType::Flag(base_types::Flag::FFlags) => "fflags".to_string(),
+        }
+    }
+
+    /// Find the unique number associated with this special type.
+    pub fn number(&self) -> u8 {
+        match *self {
+            SpecialType::Flag(base_types::Flag::IFlags) => 1,
+            SpecialType::Flag(base_types::Flag::FFlags) => 2,
+        }
+    }
+}
+
+impl fmt::Debug for SpecialType {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(
+            f,
+            "{}",
+            match *self {
+                SpecialType::Flag(_) => format!("FlagsType({})", self.name()),
+            }
+        )
+    }
+}
+
+impl From for SpecialType {
+    fn from(f: base_types::Flag) -> Self {
+        SpecialType::Flag(f)
+    }
+}
+
+pub struct SpecialTypeIterator {
+    flag_iter: base_types::FlagIterator,
+}
+
+impl SpecialTypeIterator {
+    fn new() -> Self {
+        Self {
+            flag_iter: base_types::FlagIterator::new(),
+        }
+    }
+}
+
+impl Iterator for SpecialTypeIterator {
+    type Item = SpecialType;
+    fn next(&mut self) -> Option {
+        if let Some(f) = self.flag_iter.next() {
+            Some(SpecialType::from(f))
+        } else {
+            None
+        }
+    }
+}
diff --git a/lib/codegen-meta/src/error.rs b/lib/codegen-meta/src/error.rs
new file mode 100644
index 0000000000..316e69bbc5
--- /dev/null
+++ b/lib/codegen-meta/src/error.rs
@@ -0,0 +1,47 @@
+use std::fmt;
+use std::io;
+
+/// An error that occurred when the cranelift_codegen_meta crate was generating
+/// source files for the cranelift_codegen crate.
+#[derive(Debug)]
+pub struct Error {
+    inner: Box,
+}
+
+impl Error {
+    /// Create a new error object with the given message.
+    pub fn with_msg>(msg: S) -> Error {
+        Error {
+            inner: Box::new(ErrorInner::Msg(msg.into())),
+        }
+    }
+}
+
+impl fmt::Display for Error {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "{}", self.inner)
+    }
+}
+
+impl From for Error {
+    fn from(e: io::Error) -> Self {
+        Error {
+            inner: Box::new(ErrorInner::IoError(e)),
+        }
+    }
+}
+
+#[derive(Debug)]
+enum ErrorInner {
+    Msg(String),
+    IoError(io::Error),
+}
+
+impl fmt::Display for ErrorInner {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        match *self {
+            ErrorInner::Msg(ref s) => write!(f, "{}", s),
+            ErrorInner::IoError(ref e) => write!(f, "{}", e),
+        }
+    }
+}
diff --git a/lib/codegen-meta/src/gen_build_deps.rs b/lib/codegen-meta/src/gen_build_deps.rs
new file mode 100644
index 0000000000..a267bc664e
--- /dev/null
+++ b/lib/codegen-meta/src/gen_build_deps.rs
@@ -0,0 +1,48 @@
+//! Generate build dependencies for Cargo.
+//!
+//! The `build.rs` script is invoked by cargo when building lib/codegen to
+//! generate Rust code from the instruction descriptions. Cargo needs to know when
+//! it is necessary to rerun the build script.
+//!
+//! If the build script outputs lines of the form:
+//!     cargo:rerun-if-changed=/path/to/file
+//!
+//! cargo will rerun the build script when those files have changed since the last
+//! build.
+
+use error;
+
+use std::fs;
+use std::path;
+
+/// Recursively find all interesting source files and directories in the
+/// directory tree starting at `dir`. Yield a path to each file.
+fn source_files(dir: &path::PathBuf) -> Result, error::Error> {
+    let mut files = Vec::new();
+    if dir.is_dir() {
+        for entry in fs::read_dir(&dir)? {
+            let entry = entry?;
+            let path = entry.path();
+            if path.is_dir() {
+                let mut child_dir_files = source_files(&path)?;
+                files.append(&mut child_dir_files);
+            } else if let Some(ext) = path.extension() {
+                if ext == "rs" {
+                    files.push(path.to_str().unwrap().to_string());
+                }
+            }
+        }
+    }
+    Ok(files)
+}
+
+/// Generate the lines of `cargo:rerun-if-changed` output, for each Rust source
+/// file inside of the cranelift-codegen-meta crate.
+pub fn generate(meta_dir: &path::PathBuf) -> Result<(), error::Error> {
+    println!("Dependencies from Rust meta language directory:");
+    source_files(&meta_dir)?
+        .into_iter()
+        .for_each(|p| println!("cargo:rerun-if-changed={}", p));
+
+    Ok(())
+}
diff --git a/lib/codegen-meta/src/gen_types.rs b/lib/codegen-meta/src/gen_types.rs
new file mode 100644
index 0000000000..1055f0de0e
--- /dev/null
+++ b/lib/codegen-meta/src/gen_types.rs
@@ -0,0 +1,74 @@
+//! Generate sources with type info.
+//!
+//! This generates a `types.rs` file which is included in
+//! `lib/codegen/ir/types.rs`. The file provides constant definitions for the
+//! most commonly used types, including all of the scalar types.
+//!
+//! This ensures that the metaprogram and the generated program see the same
+//! type numbering.
+
+use cdsl::types as cdsl_types;
+use error;
+use srcgen;
+
+/// Emit a constant definition of a single value type.
+fn emit_type(ty: &cdsl_types::ValueType, fmt: &mut srcgen::Formatter) -> Result<(), error::Error> {
+    let name = ty.name().to_uppercase();
+    let number = ty.number().ok_or_else(|| {
+        error::Error::with_msg(format!(
+            "Could not emit type `{}` which has no number.",
+            name
+        ))
+    })?;
+
+    let definition = format!("pub const {}: Type = Type({:#x});\n", name, number);
+
+    fmt.doc_comment(&ty.doc());
+    fmt.line(&definition);
+
+    Ok(())
+}
+
+/// Emit definition for all vector types with `bits` total size.
+fn emit_vectors(bits: u64, fmt: &mut srcgen::Formatter) -> Result<(), error::Error> {
+    let vec_size: u64 = bits / 8;
+    for vec in cdsl_types::ValueType::all_lane_types()
+        .map(|ty| (ty, cdsl_types::ValueType::from(ty).membytes()))
+        .filter(|&(_, lane_size)| lane_size != 0 && lane_size < vec_size)
+        .map(|(ty, lane_size)| (ty, vec_size / lane_size))
+        .map(|(ty, lanes)| cdsl_types::VectorType::new(ty, lanes))
+    {
+        emit_type(&cdsl_types::ValueType::from(vec), fmt)?;
+    }
+
+    Ok(())
+}
+
+/// Emit types using the given formatter object.
+fn emit_types(fmt: &mut srcgen::Formatter) -> Result<(), error::Error> {
+    // Emit all of the special types, such as types for CPU flags.
+    for spec in cdsl_types::ValueType::all_special_types().map(|ty| cdsl_types::ValueType::from(ty))
+    {
+        emit_type(&spec, fmt)?;
+    }
+
+    // Emit all of the lane types, such integers, floats, and booleans.
+    for ty in cdsl_types::ValueType::all_lane_types().map(cdsl_types::ValueType::from) {
+        emit_type(&ty, fmt)?;
+    }
+
+    // Emit vector definitions for common SIMD sizes.
+    for vec_size in &[64_u64, 128, 256, 512] {
+        emit_vectors(*vec_size, fmt)?;
+    }
+
+    Ok(())
+}
+
+/// Generate the types file.
+pub fn generate(filename: &str, out_dir: &str) -> Result<(), error::Error> {
+    let mut fmt = srcgen::Formatter::new();
+    emit_types(&mut fmt)?;
+    fmt.update_file(filename, out_dir)?;
+    Ok(())
+}
diff --git a/lib/codegen-meta/src/lib.rs b/lib/codegen-meta/src/lib.rs
new file mode 100644
index 0000000000..7a199be9f5
--- /dev/null
+++ b/lib/codegen-meta/src/lib.rs
@@ -0,0 +1,7 @@
+pub mod error;
+pub mod gen_build_deps;
+pub mod gen_types;
+
+mod base;
+mod cdsl;
+mod srcgen;
diff --git a/lib/codegen-meta/src/srcgen.rs b/lib/codegen-meta/src/srcgen.rs
new file mode 100644
index 0000000000..53a4dc5f75
--- /dev/null
+++ b/lib/codegen-meta/src/srcgen.rs
@@ -0,0 +1,316 @@
+//! Source code generator.
+//!
+//! The `srcgen` module contains generic helper routines and classes for
+//! generating source code.
+
+use std::collections::{BTreeMap, HashSet};
+use std::fs;
+use std::io::Write;
+use std::path;
+
+use error;
+
+static SHIFTWIDTH: usize = 4;
+
+struct _IndentedScope {
+    fmt: Formatter,
+    after: Option,
+}
+
+impl _IndentedScope {
+    fn _enter(&mut self) {
+        self.fmt._indent_push();
+    }
+
+    fn _exit(&mut self) {
+        self.fmt._indent_pop();
+        if let Some(ref s) = self.after {
+            self.fmt.line(&s);
+        }
+    }
+}
+
+pub struct Formatter {
+    indent: usize,
+    lines: Vec,
+}
+
+impl Formatter {
+    /// Source code formatter class. Used to collect source code to be written
+    /// to a file, and keep track of indentation.
+    pub fn new() -> Formatter {
+        Formatter {
+            indent: 0,
+            lines: Vec::new(),
+        }
+    }
+
+    /// Increase current indentation level by one.
+    pub fn _indent_push(&mut self) {
+        self.indent += 1;
+    }
+
+    /// Decrease indentation by one level.
+    pub fn _indent_pop(&mut self) {
+        assert!(self.indent > 0, "Already at top level indentation");
+        self.indent -= 1;
+    }
+
+    /// Get the current whitespace indentation in the form of a String.
+    fn get_indent(&self) -> String {
+        if self.indent == 0 {
+            String::new()
+        } else {
+            format!("{:-1$}", " ", self.indent * SHIFTWIDTH)
+        }
+    }
+
+    /// Get a string containing whitespace outdented one level. Used for
+    /// lines of code that are inside a single indented block.
+    fn _get_outdent(&mut self) -> String {
+        self._indent_push();
+        let s = self.get_indent();
+        self._indent_pop();
+        s
+    }
+
+    /// Add an indented line.
+    pub fn line(&mut self, contents: &str) {
+        let indented_line = format!("{}{}\n", self.get_indent(), contents);
+        self.lines.push(indented_line);
+    }
+
+    /// Emit a line outdented one level.
+    pub fn _outdented_line(&mut self, s: &str) {
+        let new_line = format!("{}{}", self._get_outdent(), s);
+        self.lines.push(new_line);
+    }
+
+    /// Write `self.lines` to a file.
+    pub fn update_file(&self, filename: &str, directory: &str) -> Result<(), error::Error> {
+        #[cfg(target_family = "windows")]
+        let path_str = format!("{}\\{}", directory, filename);
+        #[cfg(not(target_family = "windows"))]
+        let path_str = format!("{}/{}", directory, filename);
+
+        let path = path::Path::new(&path_str);
+        let mut f = fs::File::create(path)?;
+
+        for l in self.lines.iter().map(|l| l.as_bytes()) {
+            f.write_all(l)?;
+        }
+
+        Ok(())
+    }
+
+    /// Return a scope object for use with a `with` statement.
+    /// The optional `before` and `after` parameters are surrounding lines
+    /// which are *not* indented.
+    fn _indented(&self, _before: Option<&str>, _after: Option<&str>) -> _IndentedScope {
+        unimplemented!();
+    }
+
+    /// Add one or more lines after stripping common indentation.
+    pub fn _multi_line(&mut self, s: &str) {
+        parse_multiline(s).into_iter().for_each(|l| self.line(&l));
+    }
+
+    /// Add a comment line.
+    pub fn _comment(&mut self, s: &str) {
+        let commented_line = format!("// {}", s);
+        self.line(&commented_line);
+    }
+
+    /// Add a (multi-line) documentation comment.
+    pub fn doc_comment(&mut self, contents: &str) {
+        parse_multiline(contents)
+            .iter()
+            .map(|l| format!("/// {}", l))
+            .for_each(|s| self.line(s.as_str()));
+    }
+
+    /// Add a match expression.
+    fn _add_match(&mut self, _m: &_Match) {
+        unimplemented!();
+    }
+}
+
+/// Compute the indentation of s, or None of an empty line.
+fn _indent(s: &str) -> Option {
+    if s.is_empty() {
+        None
+    } else {
+        let t = s.trim_left();
+        Some(s.len() - t.len())
+    }
+}
+
+/// Given a multi-line string, split it into a sequence of lines after
+/// stripping a common indentation. This is useful for strings defined with
+/// doc strings.
+fn parse_multiline(s: &str) -> Vec {
+    // Convert tabs into spaces.
+    let expanded_tab = format!("{:-1$}", " ", SHIFTWIDTH);
+    let lines: Vec = s.lines().map(|l| l.replace("\t", &expanded_tab)).collect();
+
+    // Determine minimum indentation, ignoring the first line.
+    let indent = lines
+        .iter()
+        .skip(1)
+        .map(|l| l.len() - l.trim_left().len())
+        .filter(|&i| i > 0)
+        .min();
+
+    // Strip off leading blank lines.
+    let mut lines_iter = lines.iter().skip_while(|l| l.is_empty());
+    let mut trimmed = Vec::with_capacity(lines.len());
+
+    // Remove indentation (first line is special)
+    if let Some(s) = lines_iter.next().map(|l| l.trim()).map(|l| l.to_string()) {
+        trimmed.push(s);
+    }
+
+    // Remove trailing whitespace from other lines.
+    let mut other_lines = if let Some(indent) = indent {
+        lines_iter
+            .map(|l| &l[indent..])
+            .map(|l| l.trim_right())
+            .map(|l| l.to_string())
+            .collect::>()
+    } else {
+        lines_iter
+            .map(|l| l.trim_right())
+            .map(|l| l.to_string())
+            .collect::>()
+    };
+
+    trimmed.append(&mut other_lines);
+
+    // Strip off trailing blank lines.
+    while let Some(s) = trimmed.pop() {
+        if s.is_empty() {
+            continue;
+        } else {
+            trimmed.push(s);
+            break;
+        }
+    }
+
+    trimmed
+}
+
+/// Match formatting class.
+///
+/// Match objects collect all the information needed to emit a Rust `match`
+/// expression, automatically deduplicating overlapping identical arms.
+///
+/// Note that this class is ignorant of Rust types, and considers two fields
+/// with the same name to be equivalent. A BTreeMap is used to represent the
+/// arms in order to make the order deterministic.
+struct _Match<'a> {
+    _expr: &'a str,
+    arms: BTreeMap<(Vec<&'a str>, &'a str), HashSet<&'a str>>,
+}
+
+impl<'a> _Match<'a> {
+    /// Create a new match statement on `expr`.
+    fn _new(expr: &'a str) -> Self {
+        Self {
+            _expr: expr,
+            arms: BTreeMap::new(),
+        }
+    }
+
+    /// Add an arm to the Match statement.
+    fn _arm(&mut self, name: &'a str, fields: Vec<&'a str>, body: &'a str) {
+        // let key = (fields, body);
+        let match_arm = self.arms.entry((fields, body)).or_insert_with(HashSet::new);
+        match_arm.insert(name);
+    }
+}
+
+#[cfg(test)]
+mod srcgen_tests {
+    use super::_Match;
+    use super::parse_multiline;
+    use super::Formatter;
+
+    #[test]
+    fn adding_arms_works() {
+        let mut m = _Match::_new("x");
+        m._arm("Orange", vec!["a", "b"], "some body");
+        m._arm("Yellow", vec!["a", "b"], "some body");
+        m._arm("Green", vec!["a", "b"], "different body");
+        m._arm("Blue", vec!["x", "y"], "some body");
+        assert_eq!(m.arms.len(), 3);
+    }
+
+    #[test]
+    fn parse_multiline_works() {
+        let input = "\n    hello\n    world\n";
+        let expected = vec!["hello", "world"];
+        let output = parse_multiline(input);
+        assert_eq!(output, expected);
+    }
+
+    #[test]
+    fn formatter_basic_example_works() {
+        let mut fmt = Formatter::new();
+        fmt.line("Hello line 1");
+        fmt._indent_push();
+        fmt._comment("Nested comment");
+        fmt._indent_pop();
+        fmt.line("Back home again");
+        let expected_lines = vec![
+            "Hello line 1\n",
+            "    // Nested comment\n",
+            "Back home again\n",
+        ];
+        assert_eq!(fmt.lines, expected_lines);
+    }
+
+    #[test]
+    fn get_indent_works() {
+        let mut fmt = Formatter::new();
+        let expected_results = vec!["", "    ", "        ", ""];
+
+        let actual_results = Vec::with_capacity(4);
+        (0..3).for_each(|_| {
+            fmt.get_indent();
+            fmt._indent_push();
+        });
+        (0..3).for_each(|_| fmt._indent_pop());
+        fmt.get_indent();
+
+        actual_results
+            .into_iter()
+            .zip(expected_results.into_iter())
+            .for_each(|(actual, expected): (String, &str)| assert_eq!(&actual, expected));
+    }
+
+    #[test]
+    fn fmt_can_add_type_to_lines() {
+        let mut fmt = Formatter::new();
+        fmt.line(&format!("pub const {}: Type = Type({:#x});", "example", 0,));
+        let expected_lines = vec!["pub const example: Type = Type(0x0);\n"];
+        assert_eq!(fmt.lines, expected_lines);
+    }
+
+    #[test]
+    fn fmt_can_add_indented_line() {
+        let mut fmt = Formatter::new();
+        fmt.line("hello");
+        fmt._indent_push();
+        fmt.line("world");
+        let expected_lines = vec!["hello\n", "    world\n"];
+        assert_eq!(fmt.lines, expected_lines);
+    }
+
+    #[test]
+    fn fmt_can_add_doc_comments() {
+        let mut fmt = Formatter::new();
+        fmt.doc_comment("documentation\nis\ngood");
+        let expected_lines = vec!["/// documentation\n", "/// is\n", "/// good\n"];
+        assert_eq!(fmt.lines, expected_lines);
+    }
+}
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index fbd442b5b1..18cf15822a 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -21,6 +21,9 @@ target-lexicon = { version = "0.0.3", default-features = false }
 # machine code. Integration tests that need external dependencies can be
 # accomodated in `tests`.
 
+[build-dependencies]
+cranelift-codegen-meta = { path = "../codegen-meta", version = "0.15.0" }
+
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
 # of some minimal std-like replacement libraries. At least one of these two
diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index b5a2f5b435..f9ba9b8e4a 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -18,6 +18,8 @@
 // The build script expects to be run from the directory where this build.rs file lives. The
 // current directory is used to find the sources.
 
+extern crate cranelift_codegen_meta as meta;
+
 use std::env;
 use std::process;
 
@@ -66,12 +68,35 @@ fn main() {
         .arg("-B")
         .arg(build_script)
         .arg("--out-dir")
-        .arg(out_dir)
+        .arg(out_dir.clone())
         .status()
         .expect("Failed to launch second-level build script; is python installed?");
     if !status.success() {
         process::exit(status.code().unwrap());
     }
+
+    // DEVELOPMENT:
+    // ------------------------------------------------------------------------
+    // Now that the Python build process is complete, generate files that are
+    // emitted by the `cretonne_codegen_meta` crate.
+    // ------------------------------------------------------------------------
+
+    // Identify the directory of the Rust codegen-meta external crate.
+    let rust_meta_dir = crate_dir
+        .parent()
+        .map(|d| d.join("codegen-meta"))
+        .unwrap_or_else(|| {
+            eprintln!("Error: Could not find path to lib/codegen-meta crate.");
+            process::exit(1);
+        });
+
+    if let Err(err) = meta::gen_types::generate("new_types.rs", &out_dir) {
+        eprintln!("Error: {}", err);
+        process::exit(1);
+    } else if let Err(err) = meta::gen_build_deps::generate(&rust_meta_dir) {
+        eprintln!("Error: {}", err);
+        process::exit(1);
+    }
 }
 
 fn identify_python() -> &'static str {

From c77df6f6d9be8ee6d1d0782917fa7c0b55300683 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 19 Jul 2018 06:33:36 -0700
Subject: [PATCH 1925/3084] Add a TODO about a potential optimization
 opportunity.

---
 lib/frontend/src/ssa.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs
index 9c676ceb0b..0f896fa1c9 100644
--- a/lib/frontend/src/ssa.rs
+++ b/lib/frontend/src/ssa.rs
@@ -38,6 +38,7 @@ where
 {
     // Records for every variable and for every relevant block, the last definition of
     // the variable in the block.
+    // TODO: Consider a sparse representation rather than EntityMap-of-EntityMap.
     variables: EntityMap>>,
     // Records the position of the basic blocks and the list of values used but not defined in the
     // block.

From 45ef3149f16e8144bfd63348097fbcd3cadbfbed Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 19 Jul 2018 06:40:34 -0700
Subject: [PATCH 1926/3084] Update hashmap_core to 0.1.9.

---
 lib/codegen/Cargo.toml | 2 +-
 lib/module/Cargo.toml  | 2 +-
 lib/wasm/Cargo.toml    | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 18cf15822a..025096be5e 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -14,7 +14,7 @@ build = "build.rs"
 cranelift-entity = { path = "../entity", version = "0.16.1", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
-hashmap_core = { version = "0.1.8", optional = true }
+hashmap_core = { version = "0.1.9", optional = true }
 target-lexicon = { version = "0.0.3", default-features = false }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 0b40d449eb..226d798fbd 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -11,7 +11,7 @@ readme = "README.md"
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
 cranelift-entity = { path = "../entity", version = "0.16.1", default-features = false }
-hashmap_core = { version = "0.1.8", optional = true }
+hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 
 [features]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 3a0556ae8a..bc745d4fff 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -12,7 +12,7 @@ keywords = ["webassembly", "wasm"]
 wasmparser = { version = "0.17.2", default-features = false }
 cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
 cranelift-frontend = { path = "../frontend", version = "0.16.1", default-features = false }
-hashmap_core = { version = "0.1.8", optional = true }
+hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }

From 20899d04a1cfb0e83ffa7afeb48cd30809f9c998 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 19 Jul 2018 06:52:34 -0700
Subject: [PATCH 1927/3084] Fix "Title overline too short." warnings in more
 *.rst files.

---
 cranelift/docs/regalloc.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/docs/regalloc.rst b/cranelift/docs/regalloc.rst
index 66ead056f8..da227586a1 100644
--- a/cranelift/docs/regalloc.rst
+++ b/cranelift/docs/regalloc.rst
@@ -1,6 +1,6 @@
-*******************************
+********************************
 Register Allocation in Cranelift
-*******************************
+********************************
 
 .. default-domain:: clif
 .. highlight:: clif

From 7f98f436f098def253d8d3d350ac78cfc2bcd7c4 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 19 Jul 2018 06:57:11 -0700
Subject: [PATCH 1928/3084] Add a link to the faerie github page.

---
 cranelift/docs/index.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst
index 552c3089d5..08b27162c3 100644
--- a/cranelift/docs/index.rst
+++ b/cranelift/docs/index.rst
@@ -41,7 +41,7 @@ Rust Crate Documentation
 `cranelift-faerie `_
     This crate provides a faerie-based backend for `cranelift-module`, which
     emits native object files using the
-    `faerie `_ library.
+    `faerie `_ library.
 
 `cranelift-simplejit `_
     This crate provides a simple JIT backend for `cranelift-module`, which

From 32d657d62a5c55283b745f8b6e16b9458c17d9ee Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 19 Jul 2018 11:38:46 -0700
Subject: [PATCH 1929/3084] Update to faerie 0.4.4 and goblin 0.0.17.

---
 lib/faerie/Cargo.toml     | 4 ++--
 lib/faerie/src/backend.rs | 6 +++---
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 6be4989023..1f3f9d538c 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -11,8 +11,8 @@ readme = "README.md"
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.16.1" }
 cranelift-module = { path = "../module", version = "0.16.1" }
-faerie = "0.4.3"
-goblin = "0.0.15"
+faerie = "0.4.4"
+goblin = "0.0.17"
 failure = "0.1.1"
 target-lexicon = "0.0.3"
 
diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index 09349cba8d..3b18c21d0e 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -225,7 +225,7 @@ impl Backend for FaerieBackend {
                 .link(faerie::Link {
                     from: name,
                     to,
-                    at: offset as usize,
+                    at: u64::from(offset),
                 })
                 .map_err(|e| ModuleError::Backend(e.to_string()))?;
         }
@@ -239,7 +239,7 @@ impl Backend for FaerieBackend {
                 .link(faerie::Link {
                     from: name,
                     to,
-                    at: offset as usize,
+                    at: u64::from(offset),
                 })
                 .map_err(|e| ModuleError::Backend(e.to_string()))?;
         }
@@ -389,7 +389,7 @@ impl<'a> RelocSink for FaerieRelocSink<'a> {
                 faerie::Link {
                     from: self.name,
                     to: &ref_name,
-                    at: offset as usize,
+                    at: u64::from(offset),
                 },
                 faerie::RelocOverride {
                     reloc: raw_reloc,

From 5f3cd868cd780e580a27a7eb6d8d6b009c278be5 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 20 Jul 2018 17:45:34 +0200
Subject: [PATCH 1930/3084] [wasm] Rename native_pointer() to pointer_type() to
 make it more apparent it's a type;

---
 lib/wasm/src/code_translator.rs | 8 ++++----
 lib/wasm/src/environ/dummy.rs   | 4 ++--
 lib/wasm/src/environ/spec.rs    | 2 +-
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index 578ba9dfcb..dec6dda896 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -75,7 +75,7 @@ pub fn translate_operator(
             let val = match state.get_global(builder.func, global_index, environ) {
                 GlobalVariable::Const(val) => val,
                 GlobalVariable::Memory { gv, ty } => {
-                    let addr = builder.ins().global_value(environ.native_pointer(), gv);
+                    let addr = builder.ins().global_value(environ.pointer_type(), gv);
                     let mut flags = ir::MemFlags::new();
                     flags.set_notrap();
                     flags.set_aligned();
@@ -88,7 +88,7 @@ pub fn translate_operator(
             match state.get_global(builder.func, global_index, environ) {
                 GlobalVariable::Const(_) => panic!("global #{} is a constant", global_index),
                 GlobalVariable::Memory { gv, .. } => {
-                    let addr = builder.ins().global_value(environ.native_pointer(), gv);
+                    let addr = builder.ins().global_value(environ.pointer_type(), gv);
                     let mut flags = ir::MemFlags::new();
                     flags.set_notrap();
                     flags.set_aligned();
@@ -1024,7 +1024,7 @@ fn translate_load(
     let addr32 = state.pop1();
     // We don't yet support multiple linear memories.
     let heap = state.get_heap(builder.func, 0, environ);
-    let (base, offset) = get_heap_addr(heap, addr32, offset, environ.native_pointer(), builder);
+    let (base, offset) = get_heap_addr(heap, addr32, offset, environ.pointer_type(), builder);
     // Note that we don't set `is_aligned` here, even if the load instruction's
     // alignment immediate says it's aligned, because WebAssembly's immediate
     // field is just a hint, while Cranelift's aligned flag needs a guarantee.
@@ -1048,7 +1048,7 @@ fn translate_store(
 
     // We don't yet support multiple linear memories.
     let heap = state.get_heap(builder.func, 0, environ);
-    let (base, offset) = get_heap_addr(heap, addr32, offset, environ.native_pointer(), builder);
+    let (base, offset) = get_heap_addr(heap, addr32, offset, environ.pointer_type(), builder);
     // See the comments in `translate_load` about the flags.
     let flags = MemFlags::new();
     builder
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 37013cad0a..b2f58de97b 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -141,7 +141,7 @@ impl<'dummy_environment> DummyFuncEnvironment<'dummy_environment> {
     fn vmctx_sig(&self, sigidx: SignatureIndex) -> ir::Signature {
         let mut sig = self.mod_info.signatures[sigidx].clone();
         sig.params.push(ir::AbiParam::special(
-            self.native_pointer(),
+            self.pointer_type(),
             ir::ArgumentPurpose::VMContext,
         ));
         sig
@@ -217,7 +217,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         // The `callee` value is an index into a table of function pointers.
         // Apparently, that table is stored at absolute address 0 in this dummy environment.
         // TODO: Generate bounds checking code.
-        let ptr = self.native_pointer();
+        let ptr = self.pointer_type();
         let callee_offset = if ptr == I32 {
             pos.ins().imul_imm(callee, 4)
         } else {
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 7368b60e58..3c52ba32fe 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -85,7 +85,7 @@ pub trait FuncEnvironment {
     /// Get the Cranelift integer type to use for native pointers.
     ///
     /// This returns `I64` for 64-bit architectures and `I32` for 32-bit architectures.
-    fn native_pointer(&self) -> ir::Type {
+    fn pointer_type(&self) -> ir::Type {
         ir::Type::int(u16::from(self.triple().pointer_width().unwrap().bits())).unwrap()
     }
 

From ce177d643e3372f4123d8c151d5ff77eec37da44 Mon Sep 17 00:00:00 2001
From: data-pup 
Date: Sun, 22 Jul 2018 13:05:38 -0400
Subject: [PATCH 1931/3084] Remove gen_build_deps module.

---
 lib/codegen-meta/src/gen_build_deps.rs | 48 --------------------------
 lib/codegen-meta/src/lib.rs            |  1 -
 lib/codegen/build.rs                   | 13 -------
 3 files changed, 62 deletions(-)
 delete mode 100644 lib/codegen-meta/src/gen_build_deps.rs

diff --git a/lib/codegen-meta/src/gen_build_deps.rs b/lib/codegen-meta/src/gen_build_deps.rs
deleted file mode 100644
index a267bc664e..0000000000
--- a/lib/codegen-meta/src/gen_build_deps.rs
+++ /dev/null
@@ -1,48 +0,0 @@
-//! Generate build dependencies for Cargo.
-//!
-//! The `build.rs` script is invoked by cargo when building lib/codegen to
-//! generate Rust code from the instruction descriptions. Cargo needs to know when
-//! it is necessary to rerun the build script.
-//!
-//! If the build script outputs lines of the form:
-//!     cargo:rerun-if-changed=/path/to/file
-//!
-//! cargo will rerun the build script when those files have changed since the last
-//! build.
-
-use error;
-
-use std::fs;
-use std::path;
-
-/// Recursively find all interesting source files and directories in the
-/// directory tree starting at `dir`. Yield a path to each file.
-fn source_files(dir: &path::PathBuf) -> Result, error::Error> {
-    let mut files = Vec::new();
-    if dir.is_dir() {
-        for entry in fs::read_dir(&dir)? {
-            let entry = entry?;
-            let path = entry.path();
-            if path.is_dir() {
-                let mut child_dir_files = source_files(&path)?;
-                files.append(&mut child_dir_files);
-            } else if let Some(ext) = path.extension() {
-                if ext == "rs" {
-                    files.push(path.to_str().unwrap().to_string());
-                }
-            }
-        }
-    }
-    Ok(files)
-}
-
-/// Generate the lines of `cargo:rerun-if-changed` output, for each Rust source
-/// file inside of the cranelift-codegen-meta crate.
-pub fn generate(meta_dir: &path::PathBuf) -> Result<(), error::Error> {
-    println!("Dependencies from Rust meta language directory:");
-    source_files(&meta_dir)?
-        .into_iter()
-        .for_each(|p| println!("cargo:rerun-if-changed={}", p));
-
-    Ok(())
-}
diff --git a/lib/codegen-meta/src/lib.rs b/lib/codegen-meta/src/lib.rs
index 7a199be9f5..f7533e9709 100644
--- a/lib/codegen-meta/src/lib.rs
+++ b/lib/codegen-meta/src/lib.rs
@@ -1,5 +1,4 @@
 pub mod error;
-pub mod gen_build_deps;
 pub mod gen_types;
 
 mod base;
diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index f9ba9b8e4a..5e9795d8d5 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -80,22 +80,9 @@ fn main() {
     // Now that the Python build process is complete, generate files that are
     // emitted by the `cretonne_codegen_meta` crate.
     // ------------------------------------------------------------------------
-
-    // Identify the directory of the Rust codegen-meta external crate.
-    let rust_meta_dir = crate_dir
-        .parent()
-        .map(|d| d.join("codegen-meta"))
-        .unwrap_or_else(|| {
-            eprintln!("Error: Could not find path to lib/codegen-meta crate.");
-            process::exit(1);
-        });
-
     if let Err(err) = meta::gen_types::generate("new_types.rs", &out_dir) {
         eprintln!("Error: {}", err);
         process::exit(1);
-    } else if let Err(err) = meta::gen_build_deps::generate(&rust_meta_dir) {
-        eprintln!("Error: {}", err);
-        process::exit(1);
     }
 }
 

From f72ff791b454deb2b727d43550f3db12932bd25c Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 19 Jul 2018 15:29:29 +0200
Subject: [PATCH 1932/3084] Promote the BasicBlock tuple to a real struct;

It makes reading code that uses it easier to understand.
---
 lib/codegen/src/cfg_printer.rs         |  4 +-
 lib/codegen/src/dominator_tree.rs      | 21 +++++----
 lib/codegen/src/flowgraph.rs           | 65 ++++++++++++++++++++------
 lib/codegen/src/legalizer/split.rs     |  4 +-
 lib/codegen/src/licm.rs                | 13 ++++--
 lib/codegen/src/loop_analysis.rs       | 15 ++++--
 lib/codegen/src/regalloc/coalescing.rs | 25 ++++++++--
 lib/codegen/src/regalloc/liveness.rs   |  8 +++-
 lib/codegen/src/verifier/cssa.rs       |  4 +-
 lib/codegen/src/verifier/flags.rs      |  4 +-
 lib/codegen/src/verifier/liveness.rs   |  4 +-
 lib/codegen/src/verifier/mod.rs        | 10 ++--
 12 files changed, 127 insertions(+), 50 deletions(-)

diff --git a/lib/codegen/src/cfg_printer.rs b/lib/codegen/src/cfg_printer.rs
index 932345ef5b..bdd57bc1cd 100644
--- a/lib/codegen/src/cfg_printer.rs
+++ b/lib/codegen/src/cfg_printer.rs
@@ -2,7 +2,7 @@
 
 use std::fmt::{Display, Formatter, Result, Write};
 
-use flowgraph::ControlFlowGraph;
+use flowgraph::{BasicBlock, ControlFlowGraph};
 use ir::instructions::BranchInfo;
 use ir::Function;
 
@@ -61,7 +61,7 @@ impl<'a> CFGPrinter<'a> {
 
     fn cfg_connections(&self, w: &mut Write) -> Result {
         for ebb in &self.func.layout {
-            for (parent, inst) in self.cfg.pred_iter(ebb) {
+            for BasicBlock { ebb: parent, inst } in self.cfg.pred_iter(ebb) {
                 writeln!(w, "    {}:{} -> {}", parent, inst, ebb)?;
             }
         }
diff --git a/lib/codegen/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs
index ba066fe496..d8789fecb9 100644
--- a/lib/codegen/src/dominator_tree.rs
+++ b/lib/codegen/src/dominator_tree.rs
@@ -177,19 +177,19 @@ impl DominatorTree {
         layout: &Layout,
     ) -> BasicBlock {
         loop {
-            match self.rpo_cmp_ebb(a.0, b.0) {
+            match self.rpo_cmp_ebb(a.ebb, b.ebb) {
                 Ordering::Less => {
                     // `a` comes before `b` in the RPO. Move `b` up.
-                    let idom = self.nodes[b.0].idom.expect("Unreachable basic block?");
-                    b = (
+                    let idom = self.nodes[b.ebb].idom.expect("Unreachable basic block?");
+                    b = BasicBlock::new(
                         layout.inst_ebb(idom).expect("Dangling idom instruction"),
                         idom,
                     );
                 }
                 Ordering::Greater => {
                     // `b` comes before `a` in the RPO. Move `a` up.
-                    let idom = self.nodes[a.0].idom.expect("Unreachable basic block?");
-                    a = (
+                    let idom = self.nodes[a.ebb].idom.expect("Unreachable basic block?");
+                    a = BasicBlock::new(
                         layout.inst_ebb(idom).expect("Dangling idom instruction"),
                         idom,
                     );
@@ -198,10 +198,13 @@ impl DominatorTree {
             }
         }
 
-        debug_assert_eq!(a.0, b.0, "Unreachable block passed to common_dominator?");
+        debug_assert_eq!(
+            a.ebb, b.ebb,
+            "Unreachable block passed to common_dominator?"
+        );
 
         // We're in the same EBB. The common dominator is the earlier instruction.
-        if layout.cmp(a.1, b.1) == Ordering::Less {
+        if layout.cmp(a.inst, b.inst) == Ordering::Less {
             a
         } else {
             b
@@ -420,7 +423,7 @@ impl DominatorTree {
         // Note that during the first pass, `rpo_number` is 1 for reachable blocks that haven't
         // been visited yet, 0 for unreachable blocks.
         let mut reachable_preds = cfg.pred_iter(ebb)
-            .filter(|&(pred, _)| self.nodes[pred].rpo_number > 1);
+            .filter(|&BasicBlock { ebb: pred, .. }| self.nodes[pred].rpo_number > 1);
 
         // The RPO must visit at least one predecessor before this node.
         let mut idom = reachable_preds
@@ -431,7 +434,7 @@ impl DominatorTree {
             idom = self.common_dominator(idom, pred, layout);
         }
 
-        idom.1
+        idom.inst
     }
 }
 
diff --git a/lib/codegen/src/flowgraph.rs b/lib/codegen/src/flowgraph.rs
index a954b73ac1..09a9708214 100644
--- a/lib/codegen/src/flowgraph.rs
+++ b/lib/codegen/src/flowgraph.rs
@@ -31,7 +31,20 @@ use std::mem;
 use timing;
 
 /// A basic block denoted by its enclosing Ebb and last instruction.
-pub type BasicBlock = (Ebb, Inst);
+#[derive(PartialEq, Eq)]
+pub struct BasicBlock {
+    /// Enclosing Ebb key.
+    pub ebb: Ebb,
+    /// Last instruction in the basic block.
+    pub inst: Inst,
+}
+
+impl BasicBlock {
+    /// Convenient method to construct new BasicBlock.
+    pub fn new(ebb: Ebb, inst: Inst) -> Self {
+        Self { ebb, inst }
+    }
+}
 
 /// A container for the successors and predecessors of some Ebb.
 #[derive(Clone, Default)]
@@ -110,11 +123,11 @@ impl ControlFlowGraph {
         for inst in func.layout.ebb_insts(ebb) {
             match func.dfg.analyze_branch(inst) {
                 BranchInfo::SingleDest(dest, _) => {
-                    self.add_edge((ebb, inst), dest);
+                    self.add_edge(BasicBlock::new(ebb, inst), dest);
                 }
                 BranchInfo::Table(jt) => {
                     for (_, dest) in func.jump_tables[jt].entries() {
-                        self.add_edge((ebb, inst), dest);
+                        self.add_edge(BasicBlock::new(ebb, inst), dest);
                     }
                 }
                 BranchInfo::NotABranch => {}
@@ -148,12 +161,12 @@ impl ControlFlowGraph {
     }
 
     fn add_edge(&mut self, from: BasicBlock, to: Ebb) {
-        self.data[from.0]
+        self.data[from.ebb]
             .successors
             .insert(to, &mut self.succ_forest, &());
         self.data[to]
             .predecessors
-            .insert(from.1, from.0, &mut self.pred_forest, &());
+            .insert(from.inst, from.ebb, &mut self.pred_forest, &());
     }
 
     /// Get an iterator over the CFG predecessors to `ebb`.
@@ -186,7 +199,7 @@ impl<'a> Iterator for PredIter<'a> {
     type Item = BasicBlock;
 
     fn next(&mut self) -> Option {
-        self.0.next().map(|(i, e)| (e, i))
+        self.0.next().map(|(i, e)| BasicBlock::new(e, i))
     }
 }
 
@@ -268,10 +281,22 @@ mod tests {
             assert_eq!(ebb1_predecessors.len(), 2);
             assert_eq!(ebb2_predecessors.len(), 2);
 
-            assert_eq!(ebb1_predecessors.contains(&(ebb0, jmp_ebb0_ebb1)), true);
-            assert_eq!(ebb1_predecessors.contains(&(ebb1, br_ebb1_ebb1)), true);
-            assert_eq!(ebb2_predecessors.contains(&(ebb0, br_ebb0_ebb2)), true);
-            assert_eq!(ebb2_predecessors.contains(&(ebb1, jmp_ebb1_ebb2)), true);
+            assert_eq!(
+                ebb1_predecessors.contains(&BasicBlock::new(ebb0, jmp_ebb0_ebb1)),
+                true
+            );
+            assert_eq!(
+                ebb1_predecessors.contains(&BasicBlock::new(ebb1, br_ebb1_ebb1)),
+                true
+            );
+            assert_eq!(
+                ebb2_predecessors.contains(&BasicBlock::new(ebb0, br_ebb0_ebb2)),
+                true
+            );
+            assert_eq!(
+                ebb2_predecessors.contains(&BasicBlock::new(ebb1, jmp_ebb1_ebb2)),
+                true
+            );
 
             assert_eq!(ebb0_successors, [ebb1, ebb2]);
             assert_eq!(ebb1_successors, [ebb1, ebb2]);
@@ -297,10 +322,22 @@ mod tests {
             assert_eq!(ebb1_predecessors.len(), 2);
             assert_eq!(ebb2_predecessors.len(), 1);
 
-            assert_eq!(ebb1_predecessors.contains(&(ebb0, br_ebb0_ebb1)), true);
-            assert_eq!(ebb1_predecessors.contains(&(ebb1, br_ebb1_ebb1)), true);
-            assert_eq!(ebb2_predecessors.contains(&(ebb0, br_ebb0_ebb2)), false);
-            assert_eq!(ebb2_predecessors.contains(&(ebb1, jmp_ebb1_ebb2)), true);
+            assert_eq!(
+                ebb1_predecessors.contains(&BasicBlock::new(ebb0, br_ebb0_ebb1)),
+                true
+            );
+            assert_eq!(
+                ebb1_predecessors.contains(&BasicBlock::new(ebb1, br_ebb1_ebb1)),
+                true
+            );
+            assert_eq!(
+                ebb2_predecessors.contains(&BasicBlock::new(ebb0, br_ebb0_ebb2)),
+                false
+            );
+            assert_eq!(
+                ebb2_predecessors.contains(&BasicBlock::new(ebb1, jmp_ebb1_ebb2)),
+                true
+            );
 
             assert_eq!(ebb0_successors.collect::>(), [ebb1]);
             assert_eq!(ebb1_successors.collect::>(), [ebb1, ebb2]);
diff --git a/lib/codegen/src/legalizer/split.rs b/lib/codegen/src/legalizer/split.rs
index bbb63397a5..a7cd37dc60 100644
--- a/lib/codegen/src/legalizer/split.rs
+++ b/lib/codegen/src/legalizer/split.rs
@@ -65,7 +65,7 @@
 //! instructions. These loops will remain in the program.
 
 use cursor::{Cursor, CursorPosition, FuncCursor};
-use flowgraph::ControlFlowGraph;
+use flowgraph::{BasicBlock, ControlFlowGraph};
 use ir::{self, Ebb, Inst, InstBuilder, InstructionData, Opcode, Type, Value, ValueDef};
 use std::iter;
 use std::vec::Vec;
@@ -126,7 +126,7 @@ fn split_any(
 
     // We have split the value requested, and now we may need to fix some EBB predecessors.
     while let Some(repair) = repairs.pop() {
-        for (_, inst) in cfg.pred_iter(repair.ebb) {
+        for BasicBlock { inst, .. } in cfg.pred_iter(repair.ebb) {
             let branch_opc = pos.func.dfg[inst].opcode();
             debug_assert!(
                 branch_opc.is_branch(),
diff --git a/lib/codegen/src/licm.rs b/lib/codegen/src/licm.rs
index 8f32b640ac..f9954087b4 100644
--- a/lib/codegen/src/licm.rs
+++ b/lib/codegen/src/licm.rs
@@ -3,7 +3,7 @@
 use cursor::{Cursor, FuncCursor};
 use dominator_tree::DominatorTree;
 use entity::{EntityList, ListPool};
-use flowgraph::ControlFlowGraph;
+use flowgraph::{BasicBlock, ControlFlowGraph};
 use fx::FxHashSet;
 use ir::{DataFlowGraph, Ebb, Function, Inst, InstBuilder, Layout, Opcode, Type, Value};
 use loop_analysis::{Loop, LoopAnalysis};
@@ -77,7 +77,10 @@ fn create_pre_header(
     for typ in header_args_types {
         pre_header_args_value.push(func.dfg.append_ebb_param(pre_header, typ), pool);
     }
-    for (_, last_inst) in cfg.pred_iter(header) {
+    for BasicBlock {
+        inst: last_inst, ..
+    } in cfg.pred_iter(header)
+    {
         // We only follow normal edges (not the back edges)
         if !domtree.dominates(header, last_inst, &func.layout) {
             change_branch_jump_destination(last_inst, pre_header, func);
@@ -106,7 +109,11 @@ fn has_pre_header(
 ) -> Option<(Ebb, Inst)> {
     let mut result = None;
     let mut found = false;
-    for (pred_ebb, last_inst) in cfg.pred_iter(header) {
+    for BasicBlock {
+        ebb: pred_ebb,
+        inst: last_inst,
+    } in cfg.pred_iter(header)
+    {
         // We only count normal edges (not the back edges)
         if !domtree.dominates(header, last_inst, layout) {
             if found {
diff --git a/lib/codegen/src/loop_analysis.rs b/lib/codegen/src/loop_analysis.rs
index c2c73bba3f..fc1591b0cf 100644
--- a/lib/codegen/src/loop_analysis.rs
+++ b/lib/codegen/src/loop_analysis.rs
@@ -4,7 +4,7 @@
 use dominator_tree::DominatorTree;
 use entity::EntityMap;
 use entity::{Keys, PrimaryMap};
-use flowgraph::ControlFlowGraph;
+use flowgraph::{BasicBlock, ControlFlowGraph};
 use ir::{Ebb, Function, Layout};
 use packed_option::PackedOption;
 use std::vec::Vec;
@@ -137,7 +137,10 @@ impl LoopAnalysis {
     ) {
         // We traverse the CFG in reverse postorder
         for &ebb in domtree.cfg_postorder().iter().rev() {
-            for (_, pred_inst) in cfg.pred_iter(ebb) {
+            for BasicBlock {
+                inst: pred_inst, ..
+            } in cfg.pred_iter(ebb)
+            {
                 // If the ebb dominates one of its predecessors it is a back edge
                 if domtree.dominates(ebb, pred_inst, layout) {
                     // This ebb is a loop header, so we create its associated loop
@@ -163,7 +166,11 @@ impl LoopAnalysis {
         // We handle each loop header in reverse order, corresponding to a pseudo postorder
         // traversal of the graph.
         for lp in self.loops().rev() {
-            for (pred, pred_inst) in cfg.pred_iter(self.loops[lp].header) {
+            for BasicBlock {
+                ebb: pred,
+                inst: pred_inst,
+            } in cfg.pred_iter(self.loops[lp].header)
+            {
                 // We follow the back edges
                 if domtree.dominates(self.loops[lp].header, pred_inst, layout) {
                     stack.push(pred);
@@ -213,7 +220,7 @@ impl LoopAnalysis {
                 // Now we have handled the popped node and need to continue the DFS by adding the
                 // predecessors of that node
                 if let Some(continue_dfs) = continue_dfs {
-                    for (pred, _) in cfg.pred_iter(continue_dfs) {
+                    for BasicBlock { ebb: pred, .. } in cfg.pred_iter(continue_dfs) {
                         stack.push(pred)
                     }
                 }
diff --git a/lib/codegen/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs
index dd77b41937..0f00730fb8 100644
--- a/lib/codegen/src/regalloc/coalescing.rs
+++ b/lib/codegen/src/regalloc/coalescing.rs
@@ -9,7 +9,7 @@ use cursor::{Cursor, EncCursor};
 #[cfg(feature = "std")]
 use dbg::DisplayList;
 use dominator_tree::{DominatorTree, DominatorTreePreorder};
-use flowgraph::ControlFlowGraph;
+use flowgraph::{BasicBlock, ControlFlowGraph};
 use fx::FxHashMap;
 use ir::{self, InstBuilder, ProgramOrder};
 use ir::{Ebb, ExpandedProgramPoint, Function, Inst, Value};
@@ -174,7 +174,11 @@ impl<'a> Context<'a> {
         debug_assert_eq!(num_params, self.func.dfg.num_ebb_params(ebb));
         // The only way a parameter value can interfere with a predecessor branch is if the EBB is
         // dominating the predecessor branch. That is, we are looking for loop back-edges.
-        for (pred_ebb, pred_inst) in self.cfg.pred_iter(ebb) {
+        for BasicBlock {
+            ebb: pred_ebb,
+            inst: pred_inst,
+        } in self.cfg.pred_iter(ebb)
+        {
             // The quick pre-order dominance check is accurate because the EBB parameter is defined
             // at the top of the EBB before any branches.
             if !self.preorder.dominates(ebb, pred_ebb) {
@@ -211,7 +215,11 @@ impl<'a> Context<'a> {
     fn union_pred_args(&mut self, ebb: Ebb, argnum: usize) {
         let param = self.func.dfg.ebb_params(ebb)[argnum];
 
-        for (pred_ebb, pred_inst) in self.cfg.pred_iter(ebb) {
+        for BasicBlock {
+            ebb: pred_ebb,
+            inst: pred_inst,
+        } in self.cfg.pred_iter(ebb)
+        {
             let arg = self.func.dfg.inst_variable_args(pred_inst)[argnum];
 
             // Never coalesce incoming function parameters on the stack. These parameters are
@@ -484,7 +492,11 @@ impl<'a> Context<'a> {
         // not loop backedges.
         debug_assert!(self.predecessors.is_empty());
         debug_assert!(self.backedges.is_empty());
-        for (pred_ebb, pred_inst) in self.cfg.pred_iter(ebb) {
+        for BasicBlock {
+            ebb: pred_ebb,
+            inst: pred_inst,
+        } in self.cfg.pred_iter(ebb)
+        {
             if self.preorder.dominates(ebb, pred_ebb) {
                 self.backedges.push(pred_inst);
             } else {
@@ -915,7 +927,10 @@ impl VirtualCopies {
                 }
 
                 // This EBB hasn't been seen before.
-                for (_, pred_inst) in cfg.pred_iter(ebb) {
+                for BasicBlock {
+                    inst: pred_inst, ..
+                } in cfg.pred_iter(ebb)
+                {
                     self.branches.push((pred_inst, ebb));
                 }
                 last_ebb = Some(ebb);
diff --git a/lib/codegen/src/regalloc/liveness.rs b/lib/codegen/src/regalloc/liveness.rs
index 7f5ee4ae5d..d169acc459 100644
--- a/lib/codegen/src/regalloc/liveness.rs
+++ b/lib/codegen/src/regalloc/liveness.rs
@@ -176,7 +176,7 @@
 //! There is some room for improvement.
 
 use entity::SparseMap;
-use flowgraph::ControlFlowGraph;
+use flowgraph::{BasicBlock, ControlFlowGraph};
 use ir::dfg::ValueDef;
 use ir::{Ebb, Function, Inst, Layout, ProgramPoint, Value};
 use isa::{EncInfo, OperandConstraint, TargetIsa};
@@ -272,7 +272,11 @@ fn extend_to_use(
     while let Some(livein) = worklist.pop() {
         // We've learned that the value needs to be live-in to the `livein` EBB.
         // Make sure it is also live at all predecessor branches to `livein`.
-        for (pred, branch) in cfg.pred_iter(livein) {
+        for BasicBlock {
+            ebb: pred,
+            inst: branch,
+        } in cfg.pred_iter(livein)
+        {
             if lr.extend_in_ebb(pred, branch, &func.layout, forest) {
                 // This predecessor EBB also became live-in. We need to process it later.
                 worklist.push(pred);
diff --git a/lib/codegen/src/verifier/cssa.rs b/lib/codegen/src/verifier/cssa.rs
index 8e3402208b..c2d745c4fb 100644
--- a/lib/codegen/src/verifier/cssa.rs
+++ b/lib/codegen/src/verifier/cssa.rs
@@ -2,7 +2,7 @@
 
 use dbg::DisplayList;
 use dominator_tree::{DominatorTree, DominatorTreePreorder};
-use flowgraph::ControlFlowGraph;
+use flowgraph::{BasicBlock, ControlFlowGraph};
 use ir::{ExpandedProgramPoint, Function};
 use regalloc::liveness::Liveness;
 use regalloc::virtregs::VirtRegs;
@@ -138,7 +138,7 @@ impl<'a> CssaVerifier<'a> {
     fn check_cssa(&self) -> VerifierResult<()> {
         for ebb in self.func.layout.ebbs() {
             let ebb_params = self.func.dfg.ebb_params(ebb);
-            for (_, pred) in self.cfg.pred_iter(ebb) {
+            for BasicBlock { inst: pred, .. } in self.cfg.pred_iter(ebb) {
                 let pred_args = self.func.dfg.inst_variable_args(pred);
                 // This should have been caught by an earlier verifier pass.
                 assert_eq!(
diff --git a/lib/codegen/src/verifier/flags.rs b/lib/codegen/src/verifier/flags.rs
index 05ce764310..04615bacd7 100644
--- a/lib/codegen/src/verifier/flags.rs
+++ b/lib/codegen/src/verifier/flags.rs
@@ -1,7 +1,7 @@
 //! Verify CPU flags values.
 
 use entity::{EntityMap, SparseSet};
-use flowgraph::ControlFlowGraph;
+use flowgraph::{BasicBlock, ControlFlowGraph};
 use ir;
 use ir::instructions::BranchInfo;
 use isa;
@@ -61,7 +61,7 @@ impl<'a> FlagsVerifier<'a> {
                     // Revisit any predecessor blocks the first time we see a live-in for `ebb`.
                     None => {
                         self.livein[ebb] = value.into();
-                        for (pred, _) in self.cfg.pred_iter(ebb) {
+                        for BasicBlock { ebb: pred, .. } in self.cfg.pred_iter(ebb) {
                             worklist.insert(pred);
                         }
                     }
diff --git a/lib/codegen/src/verifier/liveness.rs b/lib/codegen/src/verifier/liveness.rs
index e2bf6a21ef..f884feea79 100644
--- a/lib/codegen/src/verifier/liveness.rs
+++ b/lib/codegen/src/verifier/liveness.rs
@@ -1,6 +1,6 @@
 //! Liveness verifier.
 
-use flowgraph::ControlFlowGraph;
+use flowgraph::{BasicBlock, ControlFlowGraph};
 use ir::entities::AnyEntity;
 use ir::{ExpandedProgramPoint, Function, Inst, ProgramOrder, ProgramPoint, Value};
 use isa::TargetIsa;
@@ -194,7 +194,7 @@ impl<'a> LivenessVerifier<'a> {
             // Check all the EBBs in the interval independently.
             loop {
                 // If `val` is live-in at `ebb`, it must be live at all the predecessors.
-                for (_, pred) in self.cfg.pred_iter(ebb) {
+                for BasicBlock { inst: pred, .. } in self.cfg.pred_iter(ebb) {
                     if !self.live_at_use(lr, pred) {
                         return err!(
                             pred,
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index aa3923456c..f92bdda1d8 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -60,7 +60,7 @@ use self::flags::verify_flags;
 use dbg::DisplayList;
 use dominator_tree::DominatorTree;
 use entity::SparseSet;
-use flowgraph::ControlFlowGraph;
+use flowgraph::{BasicBlock, ControlFlowGraph};
 use ir;
 use ir::entities::AnyEntity;
 use ir::instructions::{BranchInfo, CallInfo, InstructionFormat, ResolvedConstraint};
@@ -990,8 +990,12 @@ impl<'a> Verifier<'a> {
                 return err!(ebb, "cfg had unexpected successor(s) {:?}", excess_succs);
             }
 
-            expected_preds.extend(self.expected_cfg.pred_iter(ebb).map(|(_, inst)| inst));
-            got_preds.extend(cfg.pred_iter(ebb).map(|(_, inst)| inst));
+            expected_preds.extend(
+                self.expected_cfg
+                    .pred_iter(ebb)
+                    .map(|BasicBlock { inst, .. }| inst),
+            );
+            got_preds.extend(cfg.pred_iter(ebb).map(|BasicBlock { inst, .. }| inst));
 
             let missing_preds: Vec = expected_preds.difference(&got_preds).cloned().collect();
             if !missing_preds.is_empty() {

From 78b04fc8abab139e9745cd3d07706a93722d41f4 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 19 Jul 2018 16:06:47 +0200
Subject: [PATCH 1933/3084] Promote the (Block, Inst) tuple into a PredBlock
 struct;

---
 lib/frontend/src/frontend.rs |  5 ++---
 lib/frontend/src/ssa.rs      | 38 +++++++++++++++++++++++++++---------
 2 files changed, 31 insertions(+), 12 deletions(-)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 429b72c49e..bccf6cc533 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -544,10 +544,9 @@ where
             Some(entry) => self.position.ebb.unwrap() == entry,
         };
         !is_entry && self.func_ctx.ssa.is_sealed(self.position.ebb.unwrap())
-            && self.func_ctx
+            && !self.func_ctx
                 .ssa
-                .predecessors(self.position.ebb.unwrap())
-                .is_empty()
+                .has_any_predecessors(self.position.ebb.unwrap())
     }
 
     /// Returns `true` if and only if no instructions have been added since the last call to
diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs
index 0f896fa1c9..09040bf6fb 100644
--- a/lib/frontend/src/ssa.rs
+++ b/lib/frontend/src/ssa.rs
@@ -93,7 +93,7 @@ impl BlockData {
             BlockData::EbbBody { .. } => panic!("you can't add a predecessor to a body block"),
             BlockData::EbbHeader(ref mut data) => {
                 debug_assert!(!data.sealed, "sealed blocks cannot accept new predecessors");
-                data.predecessors.push((pred, inst));
+                data.predecessors.push(PredBlock::new(pred, inst));
             }
         }
     }
@@ -105,17 +105,28 @@ impl BlockData {
                 // in all non-pathological cases
                 let pred: usize = data.predecessors
                     .iter()
-                    .position(|pair| pair.1 == inst)
+                    .position(|&PredBlock { branch, .. }| branch == inst)
                     .expect("the predecessor you are trying to remove is not declared");
-                data.predecessors.swap_remove(pred).0
+                data.predecessors.swap_remove(pred).block
             }
         }
     }
 }
 
+struct PredBlock {
+    block: Block,
+    branch: Inst,
+}
+
+impl PredBlock {
+    fn new(block: Block, branch: Inst) -> Self {
+        Self { block, branch }
+    }
+}
+
 struct EbbHeaderBlockData {
     // The predecessors of the Ebb header block, with the block and branch instruction.
-    predecessors: Vec<(Block, Inst)>,
+    predecessors: Vec,
     // A ebb header block is sealed if all of its predecessors have been declared.
     sealed: bool,
     // The ebb which this block is part of.
@@ -307,7 +318,7 @@ where
                 if data.sealed {
                     if data.predecessors.len() == 1 {
                         // Only one predecessor, straightforward case
-                        UseVarCases::SealedOnePredecessor(data.predecessors[0].0)
+                        UseVarCases::SealedOnePredecessor(data.predecessors[0].block)
                     } else {
                         let val = func.dfg.append_ebb_param(data.ebb, ty);
                         UseVarCases::SealedMultiplePredecessors(val, data.ebb)
@@ -504,7 +515,7 @@ where
             self.predecessors(dest_ebb)
                 .iter()
                 .rev()
-                .map(|&(pred, _)| Call::UseVar(pred)),
+                .map(|&PredBlock { block: pred, .. }| Call::UseVar(pred)),
         );
         self.calls = calls;
     }
@@ -580,7 +591,11 @@ where
                 // to keep the ebb argument. To avoid borrowing `self` for the whole loop,
                 // temporarily detach the predecessors list and replace it with an empty list.
                 let mut preds = mem::replace(self.predecessors_mut(dest_ebb), Vec::new());
-                for &mut (ref mut pred_block, ref mut last_inst) in &mut preds {
+                for &mut PredBlock {
+                    block: ref mut pred_block,
+                    branch: ref mut last_inst,
+                } in &mut preds
+                {
                     // We already did a full `use_var` above, so we can do just the fast path.
                     let pred_val = self.variables
                         .get(temp_arg_var)
@@ -657,7 +672,7 @@ where
     }
 
     /// Returns the list of `Ebb`s that have been declared as predecessors of the argument.
-    pub fn predecessors(&self, ebb: Ebb) -> &[(Block, Inst)] {
+    fn predecessors(&self, ebb: Ebb) -> &[PredBlock] {
         let block = self.header_block(ebb);
         match self.blocks[block] {
             BlockData::EbbBody { .. } => panic!("should not happen"),
@@ -665,8 +680,13 @@ where
         }
     }
 
+    /// Returns whether the given Ebb has any predecessor or not.
+    pub fn has_any_predecessors(&self, ebb: Ebb) -> bool {
+        !self.predecessors(ebb).is_empty()
+    }
+
     /// Same as predecessors, but for &mut.
-    pub fn predecessors_mut(&mut self, ebb: Ebb) -> &mut Vec<(Block, Inst)> {
+    fn predecessors_mut(&mut self, ebb: Ebb) -> &mut Vec {
         let block = self.header_block(ebb);
         match self.blocks[block] {
             BlockData::EbbBody { .. } => panic!("should not happen"),

From 6cb03a873d145d37fa8bb4dbf09c1153aedb1068 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 23 Jul 2018 16:15:53 -0700
Subject: [PATCH 1934/3084] Bump version to 0.17.0-alpha

---
 cranelift/Cargo.toml        | 22 +++++++++++-----------
 cranelift/publish-all.sh    |  2 +-
 lib/codegen-meta/Cargo.toml |  2 +-
 lib/codegen/Cargo.toml      |  6 +++---
 lib/entity/Cargo.toml       |  2 +-
 lib/faerie/Cargo.toml       |  6 +++---
 lib/filetests/Cargo.toml    |  6 +++---
 lib/frontend/Cargo.toml     |  4 ++--
 lib/module/Cargo.toml       |  6 +++---
 lib/native/Cargo.toml       |  4 ++--
 lib/reader/Cargo.toml       |  4 ++--
 lib/simplejit/Cargo.toml    |  8 ++++----
 lib/umbrella/Cargo.toml     |  6 +++---
 lib/wasm/Cargo.toml         |  6 +++---
 14 files changed, 42 insertions(+), 42 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 982687738e..5eab8a1876 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.16.1"
+version = "0.17.0-alpha"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,16 +14,16 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.16.1" }
-cranelift-reader = { path = "lib/reader", version = "0.16.1" }
-cranelift-frontend = { path = "lib/frontend", version = "0.16.1" }
-cranelift-wasm = { path = "lib/wasm", version = "0.16.1", optional = true }
-cranelift-native = { path = "lib/native", version = "0.16.1" }
-cranelift-filetests = { path = "lib/filetests", version = "0.16.1" }
-cranelift-module = { path = "lib/module", version = "0.16.1" }
-cranelift-faerie = { path = "lib/faerie", version = "0.16.1" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.16.1" }
-cranelift = { path = "lib/umbrella", version = "0.16.1" }
+cranelift-codegen = { path = "lib/codegen", version = "0.17.0-alpha" }
+cranelift-reader = { path = "lib/reader", version = "0.17.0-alpha" }
+cranelift-frontend = { path = "lib/frontend", version = "0.17.0-alpha" }
+cranelift-wasm = { path = "lib/wasm", version = "0.17.0-alpha", optional = true }
+cranelift-native = { path = "lib/native", version = "0.17.0-alpha" }
+cranelift-filetests = { path = "lib/filetests", version = "0.17.0-alpha" }
+cranelift-module = { path = "lib/module", version = "0.17.0-alpha" }
+cranelift-faerie = { path = "lib/faerie", version = "0.17.0-alpha" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.17.0-alpha" }
+cranelift = { path = "lib/umbrella", version = "0.17.0-alpha" }
 filecheck = "0.3.0"
 docopt = "1"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 52a2ecd496..4da855b96c 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.16.1"
+version="0.17.0-alpha"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/codegen-meta/Cargo.toml b/lib/codegen-meta/Cargo.toml
index 9d75e91ce8..623f11d005 100644
--- a/lib/codegen-meta/Cargo.toml
+++ b/lib/codegen-meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.15.0"
+version = "0.17.0-alpha"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 025096be5e..4f3cda4cad 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.16.1"
+version = "0.17.0-alpha"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.16.1", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.17.0-alpha", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -22,7 +22,7 @@ target-lexicon = { version = "0.0.3", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "../codegen-meta", version = "0.15.0" }
+cranelift-codegen-meta = { path = "../codegen-meta", version = "0.17.0-alpha" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index cb52df1f89..fa071e7d64 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.16.1"
+version = "0.17.0-alpha"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 1f3f9d538c..c2c118f688 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.16.1"
+version = "0.17.0-alpha"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.1" }
-cranelift-module = { path = "../module", version = "0.16.1" }
+cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha" }
+cranelift-module = { path = "../module", version = "0.17.0-alpha" }
 faerie = "0.4.4"
 goblin = "0.0.17"
 failure = "0.1.1"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index de65853d4c..bb26ad720e 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.16.1"
+version = "0.17.0-alpha"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.1" }
-cranelift-reader = { path = "../reader", version = "0.16.1" }
+cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha" }
+cranelift-reader = { path = "../reader", version = "0.17.0-alpha" }
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 2bba26a18d..36a9818ce2 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.16.1"
+version = "0.17.0-alpha"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 226d798fbd..d7760eefde 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.16.1"
+version = "0.17.0-alpha"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.16.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.17.0-alpha", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 40e635ecf4..abb216cafb 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.16.1"
+version = "0.17.0-alpha"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -8,7 +8,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 19a80c77cb..aea061cf09 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.16.1"
+version = "0.17.0-alpha"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.1" }
+cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha" }
 target-lexicon = "0.0.3"
 
 [badges]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 8ee0532dbf..c6174ba89f 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.16.1"
+version = "0.17.0-alpha"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,9 +9,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
-cranelift-module = { path = "../module", version = "0.16.1", default-features = false }
-cranelift-native = { path = "../native", version = "0.16.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha", default-features = false }
+cranelift-module = { path = "../module", version = "0.17.0-alpha", default-features = false }
+cranelift-native = { path = "../native", version = "0.17.0-alpha", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index ea50e5b730..4a974f5fdb 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.16.1"
+version = "0.17.0-alpha"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,8 +10,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.16.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.17.0-alpha", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index bc745d4fff..bc5a8ddd8c 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.16.1"
+version = "0.17.0-alpha"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.2", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.16.1", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.16.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.17.0-alpha", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 8904ec77c965053e97f130472e25a35f3ad5fbc8 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 23 Jul 2018 16:53:14 -0700
Subject: [PATCH 1935/3084] Temporarily disable use of cretonne-codegen-meta.

Once we figure out how to publish this code, we can re-enable it.
---
 lib/codegen/Cargo.toml |  5 +++--
 lib/codegen/build.rs   | 12 +++++++-----
 2 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 4f3cda4cad..cd22362c63 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -21,8 +21,9 @@ target-lexicon = { version = "0.0.3", default-features = false }
 # machine code. Integration tests that need external dependencies can be
 # accomodated in `tests`.
 
-[build-dependencies]
-cranelift-codegen-meta = { path = "../codegen-meta", version = "0.17.0-alpha" }
+# Temporarily disable this while we work out how to publish this crate.
+#[build-dependencies]
+#cranelift-codegen-meta = { path = "../codegen-meta", version = "0.17.0-alpha" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index 5e9795d8d5..78a87f1ee7 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -18,7 +18,8 @@
 // The build script expects to be run from the directory where this build.rs file lives. The
 // current directory is used to find the sources.
 
-extern crate cranelift_codegen_meta as meta;
+// Temporarily disable this while we work out how to publish this crate.
+//extern crate cranelift_codegen_meta as meta;
 
 use std::env;
 use std::process;
@@ -80,10 +81,11 @@ fn main() {
     // Now that the Python build process is complete, generate files that are
     // emitted by the `cretonne_codegen_meta` crate.
     // ------------------------------------------------------------------------
-    if let Err(err) = meta::gen_types::generate("new_types.rs", &out_dir) {
-        eprintln!("Error: {}", err);
-        process::exit(1);
-    }
+    // Temporarily disable this while we work out how to publish this crate.
+    //if let Err(err) = meta::gen_types::generate("new_types.rs", &out_dir) {
+    //    eprintln!("Error: {}", err);
+    //    process::exit(1);
+    //}
 }
 
 fn identify_python() -> &'static str {

From d6d1e7253dbe4a35e1ac3c89b39eeef103fefbbd Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 23 Jul 2018 16:56:25 -0700
Subject: [PATCH 1936/3084] Bump version to 0.17.0

---
 cranelift/Cargo.toml        | 22 +++++++++++-----------
 cranelift/publish-all.sh    |  2 +-
 lib/codegen-meta/Cargo.toml |  2 +-
 lib/codegen/Cargo.toml      |  4 ++--
 lib/entity/Cargo.toml       |  2 +-
 lib/faerie/Cargo.toml       |  6 +++---
 lib/filetests/Cargo.toml    |  6 +++---
 lib/frontend/Cargo.toml     |  4 ++--
 lib/module/Cargo.toml       |  6 +++---
 lib/native/Cargo.toml       |  4 ++--
 lib/reader/Cargo.toml       |  4 ++--
 lib/simplejit/Cargo.toml    |  8 ++++----
 lib/umbrella/Cargo.toml     |  6 +++---
 lib/wasm/Cargo.toml         |  6 +++---
 14 files changed, 41 insertions(+), 41 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 5eab8a1876..7f3d378517 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.17.0-alpha"
+version = "0.17.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,16 +14,16 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.17.0-alpha" }
-cranelift-reader = { path = "lib/reader", version = "0.17.0-alpha" }
-cranelift-frontend = { path = "lib/frontend", version = "0.17.0-alpha" }
-cranelift-wasm = { path = "lib/wasm", version = "0.17.0-alpha", optional = true }
-cranelift-native = { path = "lib/native", version = "0.17.0-alpha" }
-cranelift-filetests = { path = "lib/filetests", version = "0.17.0-alpha" }
-cranelift-module = { path = "lib/module", version = "0.17.0-alpha" }
-cranelift-faerie = { path = "lib/faerie", version = "0.17.0-alpha" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.17.0-alpha" }
-cranelift = { path = "lib/umbrella", version = "0.17.0-alpha" }
+cranelift-codegen = { path = "lib/codegen", version = "0.17.0" }
+cranelift-reader = { path = "lib/reader", version = "0.17.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.17.0" }
+cranelift-wasm = { path = "lib/wasm", version = "0.17.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.17.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.17.0" }
+cranelift-module = { path = "lib/module", version = "0.17.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.17.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.17.0" }
+cranelift = { path = "lib/umbrella", version = "0.17.0" }
 filecheck = "0.3.0"
 docopt = "1"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 4da855b96c..cedc725aa2 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.17.0-alpha"
+version="0.17.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/codegen-meta/Cargo.toml b/lib/codegen-meta/Cargo.toml
index 623f11d005..cebb2545b0 100644
--- a/lib/codegen-meta/Cargo.toml
+++ b/lib/codegen-meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.17.0-alpha"
+version = "0.17.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index cd22362c63..f5bee967fa 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.17.0-alpha"
+version = "0.17.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.17.0-alpha", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.17.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index fa071e7d64..8582e5b82b 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.17.0-alpha"
+version = "0.17.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index c2c118f688..4ceab48af3 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.17.0-alpha"
+version = "0.17.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha" }
-cranelift-module = { path = "../module", version = "0.17.0-alpha" }
+cranelift-codegen = { path = "../codegen", version = "0.17.0" }
+cranelift-module = { path = "../module", version = "0.17.0" }
 faerie = "0.4.4"
 goblin = "0.0.17"
 failure = "0.1.1"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index bb26ad720e..5f3f70c130 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.17.0-alpha"
+version = "0.17.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha" }
-cranelift-reader = { path = "../reader", version = "0.17.0-alpha" }
+cranelift-codegen = { path = "../codegen", version = "0.17.0" }
+cranelift-reader = { path = "../reader", version = "0.17.0" }
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 36a9818ce2..d4c4d3d268 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.17.0-alpha"
+version = "0.17.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index d7760eefde..d46b01bc00 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.17.0-alpha"
+version = "0.17.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.17.0-alpha", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.17.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index abb216cafb..e9ce77ed4d 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.17.0-alpha"
+version = "0.17.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -8,7 +8,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index aea061cf09..055e041d19 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.17.0-alpha"
+version = "0.17.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha" }
+cranelift-codegen = { path = "../codegen", version = "0.17.0" }
 target-lexicon = "0.0.3"
 
 [badges]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index c6174ba89f..4577488d2d 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.17.0-alpha"
+version = "0.17.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,9 +9,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha", default-features = false }
-cranelift-module = { path = "../module", version = "0.17.0-alpha", default-features = false }
-cranelift-native = { path = "../native", version = "0.17.0-alpha", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.17.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.17.0", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index 4a974f5fdb..5f31eba10d 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.17.0-alpha"
+version = "0.17.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,8 +10,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.17.0-alpha", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.17.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index bc5a8ddd8c..b72cb584ab 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.17.0-alpha"
+version = "0.17.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.2", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.17.0-alpha", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.17.0-alpha", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.17.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From eed861c6e16d7cd15ac42e2aa2f5584b01e4e96b Mon Sep 17 00:00:00 2001
From: Aaron Power 
Date: Tue, 24 Jul 2018 14:17:21 +0100
Subject: [PATCH 1937/3084] Implemented clippy improvements

---
 lib/codegen/src/postopt.rs             |  2 +-
 lib/codegen/src/print_errors.rs        |  2 +-
 lib/codegen/src/regalloc/coalescing.rs |  2 +-
 lib/codegen/src/regalloc/pressure.rs   | 19 ++++++++++---------
 lib/faerie/src/backend.rs              |  4 ++--
 5 files changed, 15 insertions(+), 14 deletions(-)

diff --git a/lib/codegen/src/postopt.rs b/lib/codegen/src/postopt.rs
index 517621bef5..29ef49f758 100644
--- a/lib/codegen/src/postopt.rs
+++ b/lib/codegen/src/postopt.rs
@@ -218,7 +218,7 @@ fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa)
     if let ValueDef::Result(result_inst, _) = pos.func.dfg.value_def(info.arg) {
         match pos.func.dfg[result_inst] {
             InstructionData::Binary { opcode, args } if opcode == Opcode::Iadd => {
-                info.add_args = Some(args.clone());
+                info.add_args = Some(args);
             }
             _ => return,
         }
diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 068997e5c4..3cf6af6daa 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -61,7 +61,7 @@ fn pretty_function_error(
                 )
             }
         }
-        _ => write!(w, "{}", "\n"),
+        _ => writeln!(w),
     }
 }
 
diff --git a/lib/codegen/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs
index 0f00730fb8..687cb84c71 100644
--- a/lib/codegen/src/regalloc/coalescing.rs
+++ b/lib/codegen/src/regalloc/coalescing.rs
@@ -1012,7 +1012,7 @@ impl VirtualCopies {
     /// Returns `None` if none of the currently active parameters are defined at `ebb`. Otherwise
     /// returns `(set_id, argnum)` for an active parameter defined at `ebb`.
     fn lookup(&self, ebb: Ebb) -> Option<(u8, usize)> {
-        self.filter.get(&ebb).map(|t| *t)
+        self.filter.get(&ebb).cloned()
     }
 
     /// Get an iterator of dom-forest nodes corresponding to the current filter.
diff --git a/lib/codegen/src/regalloc/pressure.rs b/lib/codegen/src/regalloc/pressure.rs
index 65651987a7..ef8ec425aa 100644
--- a/lib/codegen/src/regalloc/pressure.rs
+++ b/lib/codegen/src/regalloc/pressure.rs
@@ -204,16 +204,16 @@ impl Pressure {
     ///
     /// This does not check if there are enough registers available.
     pub fn take(&mut self, rc: RegClass) {
-        self.toprc
-            .get_mut(rc.toprc as usize)
-            .map(|t| t.base_count += 1);
+        if let Some(t) = self.toprc.get_mut(rc.toprc as usize) {
+            t.base_count += 1;
+        }
     }
 
     /// Free a register in `rc`.
     pub fn free(&mut self, rc: RegClass) {
-        self.toprc
-            .get_mut(rc.toprc as usize)
-            .map(|t| t.base_count -= 1);
+        if let Some(t) = self.toprc.get_mut(rc.toprc as usize) {
+            t.base_count -= 1;
+        }
     }
 
     /// Reset all counts to 0, both base and transient.
@@ -230,9 +230,10 @@ impl Pressure {
     pub fn take_transient(&mut self, rc: RegClass) -> Result<(), RegClassMask> {
         let mask = self.check_avail(rc);
         if mask == 0 {
-            self.toprc
-                .get_mut(rc.toprc as usize)
-                .map(|t| t.transient_count += 1);
+            if let Some(t) = self.toprc.get_mut(rc.toprc as usize) {
+                t.transient_count += 1;
+            }
+
             Ok(())
         } else {
             Err(mask)
diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index 3b18c21d0e..a2ce1e2b4b 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -157,7 +157,7 @@ impl Backend for FaerieBackend {
                 artifact: &mut self.artifact,
                 name,
                 namespace,
-                libcall_names: &self.libcall_names,
+                libcall_names: &*self.libcall_names,
             };
 
             if let Some(ref mut trap_manifest) = self.trap_manifest {
@@ -347,7 +347,7 @@ struct FaerieRelocSink<'a> {
     artifact: &'a mut faerie::Artifact,
     name: &'a str,
     namespace: &'a ModuleNamespace<'a, FaerieBackend>,
-    libcall_names: &'a Box String>,
+    libcall_names: &'a Fn(ir::LibCall) -> String,
 }
 
 impl<'a> RelocSink for FaerieRelocSink<'a> {

From 299898d494b786f65c49968223b542a991611566 Mon Sep 17 00:00:00 2001
From: Corey Farwell 
Date: Tue, 24 Jul 2018 09:59:21 -0400
Subject: [PATCH 1938/3084] Add fuzz target for cranelift_reader::parse_test.

---
 cranelift/fuzz/Cargo.toml                |  5 +++++
 cranelift/fuzz/fuzz_reader_parse_test.rs | 11 +++++++++++
 2 files changed, 16 insertions(+)
 create mode 100644 cranelift/fuzz/fuzz_reader_parse_test.rs

diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
index 3805d5f871..af5f216836 100644
--- a/cranelift/fuzz/Cargo.toml
+++ b/cranelift/fuzz/Cargo.toml
@@ -12,6 +12,7 @@ cargo-fuzz = "*"
 binaryen = { git = "https://github.com/pepyakin/binaryen-rs.git" }
 libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" }
 cranelift-wasm = { path = "../lib/wasm" }
+cranelift-reader = { path = "../lib/reader" }
 target-lexicon = "0.0.3"
 
 # Prevent this from interfering with workspaces
@@ -21,3 +22,7 @@ members = ["."]
 [[bin]]
 name = "fuzz_translate_module"
 path = "fuzz_translate_module.rs"
+
+[[bin]]
+name = "fuzz_reader_parse_test"
+path = "fuzz_reader_parse_test.rs"
diff --git a/cranelift/fuzz/fuzz_reader_parse_test.rs b/cranelift/fuzz/fuzz_reader_parse_test.rs
new file mode 100644
index 0000000000..fc5709e4b2
--- /dev/null
+++ b/cranelift/fuzz/fuzz_reader_parse_test.rs
@@ -0,0 +1,11 @@
+#![no_main]
+#[macro_use]
+extern crate libfuzzer_sys;
+extern crate cranelift_reader;
+use std::str;
+
+fuzz_target!(|data: &[u8]| {
+    if let Ok(s) = str::from_utf8(data) {
+        let _ = cranelift_reader::parse_test(s);
+    }
+});

From 952a086f32d29a176c40f7283b71f30e3cfff3bd Mon Sep 17 00:00:00 2001
From: Aaron Power 
Date: Wed, 25 Jul 2018 16:12:39 +0100
Subject: [PATCH 1939/3084] Fixed trivially_copy_pass_by_ref warnings

---
 lib/codegen/src/flowgraph.rs       | 10 +++++-----
 lib/codegen/src/ir/instructions.rs |  6 +++---
 lib/codegen/src/ir/valueloc.rs     | 16 ++++++++--------
 lib/codegen/src/settings.rs        |  8 ++++----
 4 files changed, 20 insertions(+), 20 deletions(-)

diff --git a/lib/codegen/src/flowgraph.rs b/lib/codegen/src/flowgraph.rs
index 09a9708214..9eda2741af 100644
--- a/lib/codegen/src/flowgraph.rs
+++ b/lib/codegen/src/flowgraph.rs
@@ -123,11 +123,11 @@ impl ControlFlowGraph {
         for inst in func.layout.ebb_insts(ebb) {
             match func.dfg.analyze_branch(inst) {
                 BranchInfo::SingleDest(dest, _) => {
-                    self.add_edge(BasicBlock::new(ebb, inst), dest);
+                    self.add_edge(ebb, inst, dest);
                 }
                 BranchInfo::Table(jt) => {
                     for (_, dest) in func.jump_tables[jt].entries() {
-                        self.add_edge(BasicBlock::new(ebb, inst), dest);
+                        self.add_edge(ebb, inst, dest);
                     }
                 }
                 BranchInfo::NotABranch => {}
@@ -160,13 +160,13 @@ impl ControlFlowGraph {
         self.compute_ebb(func, ebb);
     }
 
-    fn add_edge(&mut self, from: BasicBlock, to: Ebb) {
-        self.data[from.ebb]
+    fn add_edge(&mut self, from: Ebb, from_inst: Inst, to: Ebb) {
+        self.data[from]
             .successors
             .insert(to, &mut self.succ_forest, &());
         self.data[to]
             .predecessors
-            .insert(from.inst, from.ebb, &mut self.pred_forest, &());
+            .insert(from_inst, from, &mut self.pred_forest, &());
     }
 
     /// Get an iterator over the CFG predecessors to `ebb`.
diff --git a/lib/codegen/src/ir/instructions.rs b/lib/codegen/src/ir/instructions.rs
index d242622dac..c9abb67f3d 100644
--- a/lib/codegen/src/ir/instructions.rs
+++ b/lib/codegen/src/ir/instructions.rs
@@ -447,7 +447,7 @@ impl ValueTypeSet {
     /// Is `scalar` part of the base type set?
     ///
     /// Note that the base type set does not have to be included in the type set proper.
-    fn is_base_type(&self, scalar: Type) -> bool {
+    fn is_base_type(self, scalar: Type) -> bool {
         let l2b = scalar.log2_lane_bits();
         if scalar.is_int() {
             self.ints.contains(l2b)
@@ -461,7 +461,7 @@ impl ValueTypeSet {
     }
 
     /// Does `typ` belong to this set?
-    pub fn contains(&self, typ: Type) -> bool {
+    pub fn contains(self, typ: Type) -> bool {
         let l2l = typ.log2_lane_count();
         self.lanes.contains(l2l) && self.is_base_type(typ.lane_type())
     }
@@ -469,7 +469,7 @@ impl ValueTypeSet {
     /// Get an example member of this type set.
     ///
     /// This is used for error messages to avoid suggesting invalid types.
-    pub fn example(&self) -> Type {
+    pub fn example(self) -> Type {
         let t = if self.ints.max().unwrap_or(0) > 5 {
             types::I32
         } else if self.floats.max().unwrap_or(0) > 5 {
diff --git a/lib/codegen/src/ir/valueloc.rs b/lib/codegen/src/ir/valueloc.rs
index e4762c778f..a88a3af4de 100644
--- a/lib/codegen/src/ir/valueloc.rs
+++ b/lib/codegen/src/ir/valueloc.rs
@@ -26,8 +26,8 @@ impl Default for ValueLoc {
 
 impl ValueLoc {
     /// Is this an assigned location? (That is, not `Unassigned`).
-    pub fn is_assigned(&self) -> bool {
-        match *self {
+    pub fn is_assigned(self) -> bool {
+        match self {
             ValueLoc::Unassigned => false,
             _ => true,
         }
@@ -111,24 +111,24 @@ impl Default for ArgumentLoc {
 
 impl ArgumentLoc {
     /// Is this an assigned location? (That is, not `Unassigned`).
-    pub fn is_assigned(&self) -> bool {
-        match *self {
+    pub fn is_assigned(self) -> bool {
+        match self {
             ArgumentLoc::Unassigned => false,
             _ => true,
         }
     }
 
     /// Is this a register location?
-    pub fn is_reg(&self) -> bool {
-        match *self {
+    pub fn is_reg(self) -> bool {
+        match self {
             ArgumentLoc::Reg(_) => true,
             _ => false,
         }
     }
 
     /// Is this a stack location?
-    pub fn is_stack(&self) -> bool {
-        match *self {
+    pub fn is_stack(self) -> bool {
+        match self {
             ArgumentLoc::Stack(_) => true,
             _ => false,
         }
diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs
index 82f078a6a4..aaa11e3d8c 100644
--- a/lib/codegen/src/settings.rs
+++ b/lib/codegen/src/settings.rs
@@ -59,9 +59,9 @@ impl Builder {
     }
 
     /// Extract contents of builder once everything is configured.
-    pub fn state_for(&self, name: &str) -> &[u8] {
+    pub fn state_for(self, name: &str) -> Box<[u8]> {
         assert_eq!(name, self.template.name);
-        &self.bytes[..]
+        self.bytes
     }
 
     /// Set the value of a single bit.
@@ -301,8 +301,8 @@ pub mod detail {
     impl Detail {
         /// Check if a detail is a Detail::Preset. Useful because the Descriptor
         /// offset field has a different meaning when the detail is a preset.
-        pub fn is_preset(&self) -> bool {
-            match *self {
+        pub fn is_preset(self) -> bool {
+            match self {
                 Detail::Preset => true,
                 _ => false,
             }

From cc7ba7e69a41cc7ff102f0ca4fc48c28feb01e40 Mon Sep 17 00:00:00 2001
From: Aaron Power 
Date: Wed, 25 Jul 2018 20:18:15 +0100
Subject: [PATCH 1940/3084] Fixed remaning clippy warnings

---
 lib/codegen/src/print_errors.rs      | 10 +++++-----
 lib/codegen/src/regalloc/virtregs.rs |  1 +
 2 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 3cf6af6daa..5783056c03 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -39,22 +39,22 @@ fn pretty_function_error(
     match err.location {
         ir::entities::AnyEntity::Inst(inst) => {
             if inst == cur_inst {
-                write!(
+                writeln!(
                     w,
-                    "{1:0$}{2}\n",
+                    "{1:0$}{2}",
                     indent,
                     "",
                     func.dfg.display_inst(cur_inst, isa)
                 )?;
-                write!(w, "{1:0$}{2}", indent, "", "^")?;
+                write!(w, "{1:0$}^", indent, "")?;
                 for _c in cur_inst.to_string().chars() {
                     write!(w, "~")?;
                 }
-                write!(w, "\n\nverifier {}\n\n", err.to_string())
+                writeln!(w, "\n\nverifier {}\n", err.to_string())
             } else {
                 write!(
                     w,
-                    "{1:0$}{2}\n",
+                    "{1:0$}{2}",
                     indent,
                     "",
                     func.dfg.display_inst(cur_inst, isa)
diff --git a/lib/codegen/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs
index dbd4dceae7..ab2b749f7c 100644
--- a/lib/codegen/src/regalloc/virtregs.rs
+++ b/lib/codegen/src/regalloc/virtregs.rs
@@ -97,6 +97,7 @@ impl VirtRegs {
     ///
     /// If `value` belongs to a virtual register, the congruence class is the values of the virtual
     /// register. Otherwise it is just the value itself.
+    #[cfg_attr(feature = "cargo-clippy", allow(trivially_copy_pass_by_ref))]
     pub fn congruence_class<'a, 'b>(&'a self, value: &'b Value) -> &'b [Value]
     where
         'a: 'b,

From 15520fa961ebb577126dcd2cb06c25171a025aa2 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 26 Jul 2018 10:10:36 -0700
Subject: [PATCH 1941/3084] Enable a few more clippy lints.

---
 lib/codegen/meta/gen_instr.py |  2 +-
 lib/codegen/src/lib.rs        |  4 ----
 lib/codegen/src/postopt.rs    | 18 +++++++++---------
 lib/filetests/src/lib.rs      |  4 +---
 4 files changed, 11 insertions(+), 17 deletions(-)

diff --git a/lib/codegen/meta/gen_instr.py b/lib/codegen/meta/gen_instr.py
index fa48dcb945..a59f5d71da 100644
--- a/lib/codegen/meta/gen_instr.py
+++ b/lib/codegen/meta/gen_instr.py
@@ -631,7 +631,7 @@ def gen_member_inits(iform, fmt):
     # Immediate operands.
     # We have local variables with the same names as the members.
     for f in iform.imm_fields:
-        fmt.line('{}: {},'.format(f.member, f.member))
+        fmt.line('{},'.format(f.member))
 
     # Value operands.
     if iform.has_value_list:
diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index f9ce64bcf8..b58a6b0e17 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -5,8 +5,6 @@
 #![cfg_attr(feature = "std", warn(unstable_features))]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature="cargo-clippy", allow(
-// Rustfmt 0.9.0 is at odds with this lint:
-                block_in_if_condition_stmt,
 // Produces only a false positive:
                 while_let_loop,
 // Produces many false positives, but did produce some valid lints, now fixed:
@@ -28,8 +26,6 @@
                 new_without_default,
                 new_without_default_derive,
                 should_implement_trait,
-                redundant_field_names,
-                useless_let_if_seq,
                 len_without_is_empty))]
 #![cfg_attr(
     feature = "cargo-clippy",
diff --git a/lib/codegen/src/postopt.rs b/lib/codegen/src/postopt.rs
index 29ef49f758..b850b5ac08 100644
--- a/lib/codegen/src/postopt.rs
+++ b/lib/codegen/src/postopt.rs
@@ -188,13 +188,13 @@ fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa)
             flags,
             offset,
         } => MemOpInfo {
-            opcode: opcode,
-            inst: inst,
+            opcode,
+            inst,
             itype: pos.func.dfg.ctrl_typevar(inst),
-            arg: arg,
+            arg,
             st_arg: None,
-            flags: flags,
-            offset: offset,
+            flags,
+            offset,
             add_args: None,
         },
         InstructionData::Store {
@@ -203,13 +203,13 @@ fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa)
             flags,
             offset,
         } => MemOpInfo {
-            opcode: opcode,
-            inst: inst,
+            opcode,
+            inst,
             itype: pos.func.dfg.ctrl_typevar(inst),
             arg: args[1],
             st_arg: Some(args[0]),
-            flags: flags,
-            offset: offset,
+            flags,
+            offset,
             add_args: None,
         },
         _ => return,
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index 6d98c991f2..db1898f25f 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -6,9 +6,7 @@
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces, unstable_features)]
 #![cfg_attr(feature="cargo-clippy", allow(
-        type_complexity,
-// Rustfmt 0.9.0 is at odds with this lint:
-        block_in_if_condition_stmt))]
+        type_complexity))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(

From e0124fa82fba000412856476f97d078fba37c5e2 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 26 Jul 2018 11:09:11 -0700
Subject: [PATCH 1942/3084] Rename local variables to clarify their purpose.

---
 lib/codegen/src/legalizer/heap.rs | 34 +++++++++++++++----------------
 1 file changed, 17 insertions(+), 17 deletions(-)

diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs
index fdf3fdb094..122474c63e 100644
--- a/lib/codegen/src/legalizer/heap.rs
+++ b/lib/codegen/src/legalizer/heap.rs
@@ -17,7 +17,7 @@ pub fn expand_heap_addr(
     _isa: &TargetIsa,
 ) {
     // Unpack the instruction.
-    let (heap, offset, size) = match func.dfg[inst] {
+    let (heap, offset, access_size) = match func.dfg[inst] {
         ir::InstructionData::HeapAddr {
             opcode,
             heap,
@@ -32,10 +32,10 @@ pub fn expand_heap_addr(
 
     match func.heaps[heap].style {
         ir::HeapStyle::Dynamic { bound_gv } => {
-            dynamic_addr(inst, heap, offset, size, bound_gv, func)
+            dynamic_addr(inst, heap, offset, access_size, bound_gv, func)
         }
         ir::HeapStyle::Static { bound } => {
-            static_addr(inst, heap, offset, size, bound.into(), func, cfg)
+            static_addr(inst, heap, offset, access_size, bound.into(), func, cfg)
         }
     }
 }
@@ -45,34 +45,34 @@ fn dynamic_addr(
     inst: ir::Inst,
     heap: ir::Heap,
     offset: ir::Value,
-    size: u32,
+    access_size: u32,
     bound_gv: ir::GlobalValue,
     func: &mut ir::Function,
 ) {
-    let size = i64::from(size);
+    let access_size = i64::from(access_size);
     let offset_ty = func.dfg.value_type(offset);
     let addr_ty = func.dfg.value_type(func.dfg.first_result(inst));
     let min_size = func.heaps[heap].min_size.into();
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
 
-    // Start with the bounds check. Trap if `offset + size > bound`.
+    // Start with the bounds check. Trap if `offset + access_size > bound`.
     let bound = pos.ins().global_value(addr_ty, bound_gv);
     let oob;
-    if size == 1 {
+    if access_size == 1 {
         // `offset > bound - 1` is the same as `offset >= bound`.
         oob = pos.ins()
             .icmp(IntCC::UnsignedGreaterThanOrEqual, offset, bound);
-    } else if size <= min_size {
-        // We know that bound >= min_size, so here we can compare `offset > bound - size` without
+    } else if access_size <= min_size {
+        // We know that bound >= min_size, so here we can compare `offset > bound - access_size` without
         // wrapping.
-        let adj_bound = pos.ins().iadd_imm(bound, -size);
+        let adj_bound = pos.ins().iadd_imm(bound, -access_size);
         oob = pos.ins()
             .icmp(IntCC::UnsignedGreaterThan, offset, adj_bound);
     } else {
         // We need an overflow check for the adjusted offset.
-        let size_val = pos.ins().iconst(offset_ty, size);
-        let (adj_offset, overflow) = pos.ins().iadd_cout(offset, size_val);
+        let access_size_val = pos.ins().iconst(offset_ty, access_size);
+        let (adj_offset, overflow) = pos.ins().iadd_cout(offset, access_size_val);
         pos.ins().trapnz(overflow, ir::TrapCode::HeapOutOfBounds);
         oob = pos.ins()
             .icmp(IntCC::UnsignedGreaterThan, adj_offset, bound);
@@ -87,19 +87,19 @@ fn static_addr(
     inst: ir::Inst,
     heap: ir::Heap,
     offset: ir::Value,
-    size: u32,
+    access_size: u32,
     bound: i64,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
 ) {
-    let size = i64::from(size);
+    let access_size = i64::from(access_size);
     let offset_ty = func.dfg.value_type(offset);
     let addr_ty = func.dfg.value_type(func.dfg.first_result(inst));
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
 
-    // Start with the bounds check. Trap if `offset + size > bound`.
-    if size > bound {
+    // Start with the bounds check. Trap if `offset + access_size > bound`.
+    if access_size > bound {
         // This will simply always trap since `offset >= 0`.
         pos.ins().trap(ir::TrapCode::HeapOutOfBounds);
         pos.func.dfg.replace(inst).iconst(addr_ty, 0);
@@ -114,7 +114,7 @@ fn static_addr(
     }
 
     // Check `offset > limit` which is now known non-negative.
-    let limit = bound - size;
+    let limit = bound - access_size;
 
     // We may be able to omit the check entirely for 32-bit offsets if the heap bound is 4 GB or
     // more.

From d0fbf47987d52b6fb738dfe7354ec21791d489fa Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 26 Jul 2018 11:51:12 -0700
Subject: [PATCH 1943/3084] Fix rustfmt errors.

---
 lib/filetests/src/lib.rs | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index db1898f25f..2a587f9ea0 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -5,8 +5,7 @@
 
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces, unstable_features)]
-#![cfg_attr(feature="cargo-clippy", allow(
-        type_complexity))]
+#![cfg_attr(feature = "cargo-clippy", allow(type_complexity))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(

From 13fea26c95f13279aa77a5e3c7a200fa4e1eb9a8 Mon Sep 17 00:00:00 2001
From: Aaron Power 
Date: Thu, 26 Jul 2018 19:17:45 +0100
Subject: [PATCH 1944/3084] Correctly handle duplicate definitions

---
 lib/reader/src/parser.rs | 111 ++++++++++++++++++++++++++++++++++++---
 1 file changed, 105 insertions(+), 6 deletions(-)

diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 91b98da1ae..11cb57cef8 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -123,12 +123,13 @@ impl<'a> Context<'a> {
 
     // Allocate a new stack slot.
     fn add_ss(&mut self, ss: StackSlot, data: StackSlotData, loc: Location) -> ParseResult<()> {
+        self.map.def_ss(ss, loc)?;
         while self.function.stack_slots.next_key().index() <= ss.index() {
             self.function
                 .create_stack_slot(StackSlotData::new(StackSlotKind::SpillSlot, 0));
         }
         self.function.stack_slots[ss] = data;
-        self.map.def_ss(ss, loc)
+        Ok(())
     }
 
     // Resolve a reference to a stack slot.
@@ -142,6 +143,7 @@ impl<'a> Context<'a> {
 
     // Allocate a global value slot.
     fn add_gv(&mut self, gv: GlobalValue, data: GlobalValueData, loc: Location) -> ParseResult<()> {
+        self.map.def_gv(gv, loc)?;
         while self.function.global_values.next_key().index() <= gv.index() {
             self.function.create_global_value(GlobalValueData::Sym {
                 name: ExternalName::testcase(""),
@@ -149,7 +151,7 @@ impl<'a> Context<'a> {
             });
         }
         self.function.global_values[gv] = data;
-        self.map.def_gv(gv, loc)
+        Ok(())
     }
 
     // Resolve a reference to a global value.
@@ -163,6 +165,7 @@ impl<'a> Context<'a> {
 
     // Allocate a heap slot.
     fn add_heap(&mut self, heap: Heap, data: HeapData, loc: Location) -> ParseResult<()> {
+        self.map.def_heap(heap, loc)?;
         while self.function.heaps.next_key().index() <= heap.index() {
             self.function.create_heap(HeapData {
                 base: HeapBase::ReservedReg,
@@ -174,7 +177,7 @@ impl<'a> Context<'a> {
             });
         }
         self.function.heaps[heap] = data;
-        self.map.def_heap(heap, loc)
+        Ok(())
     }
 
     // Resolve a reference to a heap.
@@ -188,12 +191,13 @@ impl<'a> Context<'a> {
 
     // Allocate a new signature.
     fn add_sig(&mut self, sig: SigRef, data: Signature, loc: Location) -> ParseResult<()> {
+        self.map.def_sig(sig, loc)?;
         while self.function.dfg.signatures.next_key().index() <= sig.index() {
             self.function
                 .import_signature(Signature::new(CallConv::Fast));
         }
         self.function.dfg.signatures[sig] = data;
-        self.map.def_sig(sig, loc)
+        Ok(())
     }
 
     // Resolve a reference to a signature.
@@ -207,6 +211,7 @@ impl<'a> Context<'a> {
 
     // Allocate a new external function.
     fn add_fn(&mut self, fn_: FuncRef, data: ExtFuncData, loc: Location) -> ParseResult<()> {
+        self.map.def_fn(fn_, loc)?;
         while self.function.dfg.ext_funcs.next_key().index() <= fn_.index() {
             self.function.import_function(ExtFuncData {
                 name: ExternalName::testcase(""),
@@ -215,7 +220,7 @@ impl<'a> Context<'a> {
             });
         }
         self.function.dfg.ext_funcs[fn_] = data;
-        self.map.def_fn(fn_, loc)
+        Ok(())
     }
 
     // Resolve a reference to a function.
@@ -256,11 +261,12 @@ impl<'a> Context<'a> {
 
     // Allocate a new EBB.
     fn add_ebb(&mut self, ebb: Ebb, loc: Location) -> ParseResult {
+        self.map.def_ebb(ebb, loc)?;
         while self.function.dfg.num_ebbs() <= ebb.index() {
             self.function.dfg.make_ebb();
         }
         self.function.layout.append_ebb(ebb);
-        self.map.def_ebb(ebb, loc).and(Ok(ebb))
+        Ok(ebb)
     }
 }
 
@@ -2449,6 +2455,99 @@ mod tests {
         assert_eq!(func.dfg.value_type(ebb4_args[0]), types::I32);
     }
 
+    #[test]
+    fn duplicate_ebb() {
+        let ParseError { location, message } = Parser::new(
+            "function %ebbs() system_v {
+                ebb0:
+                ebb0:
+                    return 2",
+        ).parse_function(None)
+            .unwrap_err();
+
+        assert_eq!(location.line_number, 3);
+        assert_eq!(message, "duplicate entity: ebb0");
+    }
+
+    #[test]
+    fn duplicate_jt() {
+        let ParseError { location, message } = Parser::new(
+            "function %ebbs() system_v {
+                jt0 = jump_table 0, 0
+                jt0 = jump_table 0, 0",
+        ).parse_function(None)
+            .unwrap_err();
+
+        assert_eq!(location.line_number, 3);
+        assert_eq!(message, "duplicate entity: jt0");
+    }
+
+    #[test]
+    fn duplicate_ss() {
+        let ParseError { location, message } = Parser::new(
+            "function %ebbs() system_v {
+                ss0 = explicit_slot 8
+                ss0 = explicit_slot 8",
+        ).parse_function(None)
+            .unwrap_err();
+
+        assert_eq!(location.line_number, 3);
+        assert_eq!(message, "duplicate entity: ss0");
+    }
+
+    #[test]
+    fn duplicate_gv() {
+        let ParseError { location, message } = Parser::new(
+            "function %ebbs() system_v {
+                gv0 = vmctx+64
+                gv0 = vmctx+64",
+        ).parse_function(None)
+            .unwrap_err();
+
+        assert_eq!(location.line_number, 3);
+        assert_eq!(message, "duplicate entity: gv0");
+    }
+
+    #[test]
+    fn duplicate_heap() {
+        let ParseError { location, message } = Parser::new(
+            "function %ebbs() system_v {
+                heap0 = static gv0, min 0x1000, bound 0x10_0000, guard 0x1000
+                heap0 = static gv0, min 0x1000, bound 0x10_0000, guard 0x1000",
+        ).parse_function(None)
+            .unwrap_err();
+
+        assert_eq!(location.line_number, 3);
+        assert_eq!(message, "duplicate entity: heap0");
+    }
+
+    #[test]
+    fn duplicate_sig() {
+        let ParseError { location, message } = Parser::new(
+            "function %ebbs() system_v {
+                sig0 = ()
+                sig0 = ()",
+        ).parse_function(None)
+            .unwrap_err();
+
+        assert_eq!(location.line_number, 3);
+        assert_eq!(message, "duplicate entity: sig0");
+    }
+
+    #[test]
+    fn duplicate_fn() {
+        let ParseError { location, message } = Parser::new(
+            "function %ebbs() system_v {
+                sig0 = ()
+                fn0 = %foo sig0
+                fn0 = %foo sig0",
+        ).parse_function(None)
+            .unwrap_err();
+
+        assert_eq!(location.line_number, 4);
+        assert_eq!(message, "duplicate entity: fn0");
+    }
+
     #[test]
     fn comments() {
         let (func, Details { comments, .. }) = Parser::new(

From 65a1a6bb28f98b68465597921b6c2d4a1a07e22d Mon Sep 17 00:00:00 2001
From: Caroline Cullen 
Date: Tue, 31 Jul 2018 07:48:12 -0700
Subject: [PATCH 1945/3084] Add serde functionality into lib. (#422)

* Adds decoration to the verifier errors.

example:

function %bad(i32) fast {
ebb0(v0: i32):
    brnz.i32 v0, ebb1
    return
    ^~~~~~

verifier inst1: Internal return not allowed with return_at_end=1

ebb1:
    trapz.i32 v0, user6
    return
}

Fixes #68

* Making it so that pretty_function_error and write_function_plain are both private.

* Changes write_ebb to decorate_ebb.
Adds documentation line to decorate_function.

* Removing Cargo.toml lib/serde addition

* Add serde functionality into lib.

* Fix so code is compatible with Rust version 1.25.0.

* Move ser/de functions to utility file, update description, remove borrow from arms.

* Remove commented out code.
---
 cranelift/Cargo.toml             |   1 +
 lib/serde/Cargo.toml             |  38 ++
 lib/serde/README.md              |  30 ++
 lib/serde/src/clif-json.rs       | 121 +++++
 lib/serde/src/serde_clif_json.rs | 801 +++++++++++++++++++++++++++++++
 5 files changed, 991 insertions(+)
 create mode 100644 lib/serde/Cargo.toml
 create mode 100644 lib/serde/README.md
 create mode 100644 lib/serde/src/clif-json.rs
 create mode 100644 lib/serde/src/serde_clif_json.rs

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 7f3d378517..89dc89233c 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -17,6 +17,7 @@ cfg-if = "0.1"
 cranelift-codegen = { path = "lib/codegen", version = "0.17.0" }
 cranelift-reader = { path = "lib/reader", version = "0.17.0" }
 cranelift-frontend = { path = "lib/frontend", version = "0.17.0" }
+cranelift-serde = { path = "lib/serde", version = "0.17.0", optional = true }
 cranelift-wasm = { path = "lib/wasm", version = "0.17.0", optional = true }
 cranelift-native = { path = "lib/native", version = "0.17.0" }
 cranelift-filetests = { path = "lib/filetests", version = "0.17.0" }
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
new file mode 100644
index 0000000000..248276c93c
--- /dev/null
+++ b/lib/serde/Cargo.toml
@@ -0,0 +1,38 @@
+[package]
+name = "cranelift-serde"
+version = "0.17.0"
+authors = ["The Cranelift Project Developers"]
+description = "Serializer/Deserializer for Cranelift IR"
+repository = "https://github.com/CraneStation/cranelift"
+license = "Apache-2.0"
+readme = "README.md"
+keywords = ["webassembly", "serde"]
+
+[[bin]]
+name = "clif-json"
+path = "src/clif-json.rs"
+
+[dependencies]
+cfg-if = "0.1"
+filecheck = "0.3.0"
+docopt = "1"
+serde = "1.0.8"
+serde_derive = "1.0.8"
+serde_json = "1.0"
+term = "0.5.1"
+target-lexicon = "0.0.2"
+wasmparser = { version = "0.17.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
+cranelift-reader = { path = "../reader", version = "0.17.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.17.0", default-features = false }
+failure = { version = "0.1.1", default-features = false, features = ["derive"] }
+failure_derive = { version = "0.1.1", default-features = false }
+
+[features]
+default = ["std"]
+std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std", "target-lexicon/std"]
+core = ["cranelift-codegen/core", "cranelift-frontend/core", "wasmparser/core"]
+
+[badges]
+maintenance = { status = "experimental" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/serde/README.md b/lib/serde/README.md
new file mode 100644
index 0000000000..89b54cca72
--- /dev/null
+++ b/lib/serde/README.md
@@ -0,0 +1,30 @@
+This crate performs serialization of the [Cranelift](https://crates.io/crates/cranelift) IR.
+
+This crate is structured as an optional ability to serialize and deserialize cranelift IR into JSON format.
+
+Status
+------
+
+Cranelift IR can be serialized into JSON.
+
+Deserialize is a work in progress, as it currently deserializes into the serializable data structure that can be utilized by serde instead of the actual Cranelift IR data structure. 
+
+
+Building and Using Cranelift Serde
+----------------------------------
+
+clif-json usage: 
+
+    clif-json serialize [-p] 
+    clif-json deserialize 
+
+Where the -p flag outputs Cranelift IR as pretty JSON.
+
+For example to build and use clif-json:
+
+``` {.sourceCode .sh}
+cd lib/serde
+cargo build
+clif-json serialize -p test.clif
+```
+
diff --git a/lib/serde/src/clif-json.rs b/lib/serde/src/clif-json.rs
new file mode 100644
index 0000000000..eb991343ce
--- /dev/null
+++ b/lib/serde/src/clif-json.rs
@@ -0,0 +1,121 @@
+#![deny(trivial_numeric_casts)]
+#![warn(unused_import_braces, unstable_features, unused_extern_crates)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    warn(
+        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
+        unicode_not_nfc, use_self
+    )
+)]
+
+extern crate cranelift_reader;
+extern crate docopt;
+#[macro_use]
+extern crate serde_derive;
+extern crate cranelift_codegen;
+
+extern crate serde_json;
+
+use cranelift_reader::parse_functions;
+use docopt::Docopt;
+use std::fs::File;
+use std::io::prelude::*;
+use std::io::{self, Write};
+use std::process;
+use std::vec::Vec;
+
+mod serde_clif_json;
+
+const USAGE: &str = "
+Cranelift JSON serializer/deserializer utility
+
+Usage:
+    clif-json serialize [-p] 
+    clif-json deserialize 
+
+Options:
+    -p, --pretty     print pretty json
+
+";
+
+#[derive(Deserialize, Debug)]
+struct Args {
+    cmd_serialize: bool,
+    cmd_deserialize: bool,
+    flag_pretty: bool,
+    arg_file: Vec,
+}
+
+/// A command either succeeds or fails with an error message.
+pub type CommandResult = Result<(), String>;
+
+fn call_ser(file: &str, pretty: bool) -> CommandResult {
+    let ret_of_parse = parse_functions(file);
+    match ret_of_parse {
+        Ok(funcs) => {
+            let ser_funcs = serde_clif_json::SerObj::new(&funcs);
+            let ser_str: String;
+            if pretty {
+                ser_str = serde_json::to_string_pretty(&ser_funcs).unwrap();
+            } else {
+                ser_str = serde_json::to_string(&ser_funcs).unwrap();
+            }
+            println!("{}", ser_str);
+            Ok(())
+        }
+        Err(_pe) => Err(format!("this was a parsing error")),
+    }
+}
+
+fn call_de(file: &File) -> CommandResult {
+    let de: serde_clif_json::SerObj = match serde_json::from_reader(file) {
+        Result::Ok(val) => val,
+        Result::Err(err) => panic!("{}", err),
+    };
+    println!("{:?}", de);
+    Ok(())
+}
+
+/// Parse the command line arguments and run the requested command.
+fn clif_json() -> CommandResult {
+    // Parse command line arguments.
+    let args: Args = Docopt::new(USAGE)
+        .and_then(|d| d.help(true).deserialize())
+        .unwrap_or_else(|e| e.exit());
+
+    // Find the sub-command to execute.
+    let result = if args.cmd_serialize {
+        if let Some(first_file) = args.arg_file.first() {
+            let mut file = File::open(first_file).expect("Unable to open the file");
+            let mut contents = String::new();
+            file.read_to_string(&mut contents)
+                .expect("Unable to read the file");
+            call_ser(&contents, args.flag_pretty)
+        } else {
+            Err(format!("No file was passed"))
+        }
+    } else if args.cmd_deserialize {
+        if let Some(first_file) = args.arg_file.first() {
+            let mut file = File::open(first_file).expect("Unable to open the file");
+            call_de(&file)
+        } else {
+            Err(format!("No file was passed"))
+        }
+    } else {
+        // Debugging / shouldn't happen with proper command line handling above.
+        Err(format!("Unhandled args: {:?}", args))
+    };
+
+    result
+}
+
+fn main() {
+    if let Err(mut msg) = clif_json() {
+        if !msg.ends_with('\n') {
+            msg.push('\n');
+        }
+        io::stdout().flush().expect("flushing stdout");
+        io::stderr().write_all(msg.as_bytes()).unwrap();
+        process::exit(1);
+    }
+}
diff --git a/lib/serde/src/serde_clif_json.rs b/lib/serde/src/serde_clif_json.rs
new file mode 100644
index 0000000000..d466e4d0d0
--- /dev/null
+++ b/lib/serde/src/serde_clif_json.rs
@@ -0,0 +1,801 @@
+use cranelift_codegen::ir::{Ebb, Function, Inst, InstructionData, Signature};
+
+/// Serializable version of the original Cranelift IR
+#[derive(Clone, Debug, Hash, PartialEq, Eq, Serialize, Deserialize)]
+pub enum SerInstData {
+    Unary {
+        opcode: String,
+        arg: String,
+    },
+    UnaryImm {
+        opcode: String,
+        imm: String,
+    },
+    UnaryIeee32 {
+        opcode: String,
+        imm: String,
+    },
+    UnaryIeee64 {
+        opcode: String,
+        imm: String,
+    },
+    UnaryBool {
+        opcode: String,
+        imm: bool,
+    },
+    UnaryGlobalValue {
+        opcode: String,
+        global_value: String,
+    },
+    Binary {
+        opcode: String,
+        args: [String; 2],
+    },
+    BinaryImm {
+        opcode: String,
+        arg: String,
+        imm: String,
+    },
+    Ternary {
+        opcode: String,
+        args: [String; 3],
+    },
+    MultiAry {
+        opcode: String,
+        args: Vec,
+    },
+    NullAry {
+        opcode: String,
+    },
+    InsertLane {
+        opcode: String,
+        args: [String; 2],
+        lane: String,
+    },
+    ExtractLane {
+        opcode: String,
+        arg: String,
+        lane: String,
+    },
+    IntCompare {
+        opcode: String,
+        args: [String; 2],
+        cond: String,
+    },
+    IntCompareImm {
+        opcode: String,
+        arg: String,
+        cond: String,
+        imm: String,
+    },
+    IntCond {
+        opcode: String,
+        arg: String,
+        cond: String,
+    },
+    FloatCompare {
+        opcode: String,
+        args: [String; 2],
+        cond: String,
+    },
+    FloatCond {
+        opcode: String,
+        arg: String,
+        cond: String,
+    },
+    IntSelect {
+        opcode: String,
+        args: [String; 3],
+        cond: String,
+    },
+    Jump {
+        opcode: String,
+        args: Vec,
+        destination: String,
+    },
+    Branch {
+        opcode: String,
+        args: Vec,
+        destination: String,
+    },
+    BranchInt {
+        opcode: String,
+        args: Vec,
+        cond: String,
+        destination: String,
+    },
+    BranchFloat {
+        opcode: String,
+        args: Vec,
+        cond: String,
+        destination: String,
+    },
+    BranchIcmp {
+        opcode: String,
+        args: Vec,
+        cond: String,
+        destination: String,
+    },
+    BranchTable {
+        opcode: String,
+        arg: String,
+        table: String,
+    },
+    Call {
+        opcode: String,
+        args: Vec,
+        func_ref: String,
+    },
+    CallIndirect {
+        opcode: String,
+        args: Vec,
+        sig_ref: String,
+    },
+    FuncAddr {
+        opcode: String,
+        func_ref: String,
+    },
+    Load {
+        opcode: String,
+        arg: String,
+        flags: String,
+        offset: String,
+    },
+    LoadComplex {
+        opcode: String,
+        args: Vec,
+        flags: String,
+        offset: String,
+    },
+    Store {
+        opcode: String,
+        args: [String; 2],
+        flags: String,
+        offset: String,
+    },
+    StoreComplex {
+        opcode: String,
+        args: Vec,
+        flags: String,
+        offset: String,
+    },
+    StackLoad {
+        opcode: String,
+        stack_slot: String,
+        offset: String,
+    },
+    StackStore {
+        opcode: String,
+        arg: String,
+        stack_slot: String,
+        offset: String,
+    },
+    HeapAddr {
+        opcode: String,
+        arg: String,
+        heap: String,
+        imm: String,
+    },
+    RegMove {
+        opcode: String,
+        arg: String,
+        src: String,
+        dst: String,
+    },
+    CopySpecial {
+        opcode: String,
+        src: String,
+        dst: String,
+    },
+    RegSpill {
+        opcode: String,
+        arg: String,
+        src: String,
+        dst: String,
+    },
+    RegFill {
+        opcode: String,
+        arg: String,
+        src: String,
+        dst: String,
+    },
+    Trap {
+        opcode: String,
+        code: String,
+    },
+    CondTrap {
+        opcode: String,
+        arg: String,
+        code: String,
+    },
+    IntCondTrap {
+        opcode: String,
+        arg: String,
+        cond: String,
+        code: String,
+    },
+    FloatCondTrap {
+        opcode: String,
+        arg: String,
+        cond: String,
+        code: String,
+    },
+}
+
+pub fn get_inst_data(inst_index: Inst, func: &Function) -> SerInstData {
+    let inst = &func.dfg[inst_index];
+    match *inst {
+        InstructionData::Unary { opcode, arg } => SerInstData::Unary {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+        },
+        InstructionData::UnaryImm { opcode, imm } => SerInstData::UnaryImm {
+            opcode: opcode.to_string(),
+            imm: imm.to_string(),
+        },
+        InstructionData::UnaryIeee32 { opcode, imm } => SerInstData::UnaryIeee32 {
+            opcode: opcode.to_string(),
+            imm: imm.to_string(),
+        },
+        InstructionData::UnaryIeee64 { opcode, imm } => SerInstData::UnaryIeee64 {
+            opcode: opcode.to_string(),
+            imm: imm.to_string(),
+        },
+        InstructionData::UnaryBool { opcode, imm } => SerInstData::UnaryBool {
+            opcode: opcode.to_string(),
+            imm: imm,
+        },
+        InstructionData::UnaryGlobalValue {
+            opcode,
+            global_value,
+        } => SerInstData::UnaryGlobalValue {
+            opcode: opcode.to_string(),
+            global_value: global_value.to_string(),
+        },
+        InstructionData::Binary { opcode, args } => {
+            let hold_args = [args[0].to_string(), args[1].to_string()];
+            SerInstData::Binary {
+                opcode: opcode.to_string(),
+                args: hold_args,
+            }
+        }
+        InstructionData::BinaryImm { opcode, arg, imm } => SerInstData::BinaryImm {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            imm: imm.to_string(),
+        },
+        InstructionData::Ternary { opcode, args } => {
+            let hold_args = [
+                args[0].to_string(),
+                args[1].to_string(),
+                args[2].to_string(),
+            ];
+            SerInstData::Ternary {
+                opcode: opcode.to_string(),
+                args: hold_args,
+            }
+        }
+        InstructionData::MultiAry { opcode, ref args } => {
+            let mut hold_args = Vec::new();
+            let args_iter = args.as_slice(&func.dfg.value_lists);
+            for arg in args_iter {
+                hold_args.push(arg.to_string());
+            }
+
+            SerInstData::MultiAry {
+                opcode: opcode.to_string(),
+                args: hold_args,
+            }
+        }
+        InstructionData::NullAry { opcode } => SerInstData::NullAry {
+            opcode: opcode.to_string(),
+        },
+        InstructionData::InsertLane { opcode, args, lane } => {
+            let hold_args = [args[0].to_string(), args[1].to_string()];
+            SerInstData::InsertLane {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                lane: lane.to_string(),
+            }
+        }
+        InstructionData::ExtractLane { opcode, arg, lane } => SerInstData::ExtractLane {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            lane: lane.to_string(),
+        },
+        InstructionData::IntCompare { opcode, args, cond } => {
+            let hold_args = [args[0].to_string(), args[1].to_string()];
+            SerInstData::IntCompare {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                cond: cond.to_string(),
+            }
+        }
+        InstructionData::IntCompareImm {
+            opcode,
+            arg,
+            cond,
+            imm,
+        } => SerInstData::IntCompareImm {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            cond: cond.to_string(),
+            imm: imm.to_string(),
+        },
+        InstructionData::IntCond { opcode, arg, cond } => SerInstData::IntCond {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            cond: cond.to_string(),
+        },
+        InstructionData::FloatCompare { opcode, args, cond } => {
+            let hold_args = [args[0].to_string(), args[1].to_string()];
+            SerInstData::FloatCompare {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                cond: cond.to_string(),
+            }
+        }
+        InstructionData::FloatCond { opcode, arg, cond } => SerInstData::FloatCond {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            cond: cond.to_string(),
+        },
+        InstructionData::IntSelect { opcode, args, cond } => {
+            let hold_args = [
+                args[0].to_string(),
+                args[1].to_string(),
+                args[2].to_string(),
+            ];
+            SerInstData::IntSelect {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                cond: cond.to_string(),
+            }
+        }
+        InstructionData::Jump {
+            opcode,
+            ref args,
+            destination,
+        } => {
+            let mut hold_args = Vec::new();
+            let args_iter = args.as_slice(&func.dfg.value_lists);
+            for arg in args_iter {
+                hold_args.push(arg.to_string());
+            }
+            SerInstData::Jump {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                destination: destination.to_string(),
+            }
+        }
+        InstructionData::Branch {
+            opcode,
+            ref args,
+            destination,
+        } => {
+            let mut hold_args = Vec::new();
+            let args_iter = args.as_slice(&func.dfg.value_lists);
+            for arg in args_iter {
+                hold_args.push(arg.to_string());
+            }
+            SerInstData::Branch {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                destination: destination.to_string(),
+            }
+        }
+        InstructionData::BranchInt {
+            opcode,
+            ref args,
+            cond,
+            destination,
+        } => {
+            let mut hold_args = Vec::new();
+            let args_iter = args.as_slice(&func.dfg.value_lists);
+            for arg in args_iter {
+                hold_args.push(arg.to_string());
+            }
+            SerInstData::BranchInt {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                cond: cond.to_string(),
+                destination: destination.to_string(),
+            }
+        }
+        InstructionData::BranchFloat {
+            opcode,
+            ref args,
+            cond,
+            destination,
+        } => {
+            let mut hold_args = Vec::new();
+            let args_iter = args.as_slice(&func.dfg.value_lists);
+            for arg in args_iter {
+                hold_args.push(arg.to_string());
+            }
+            SerInstData::BranchFloat {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                cond: cond.to_string(),
+                destination: destination.to_string(),
+            }
+        }
+        InstructionData::BranchIcmp {
+            opcode,
+            ref args,
+            cond,
+            destination,
+        } => {
+            let mut hold_args = Vec::new();
+            let args_iter = args.as_slice(&func.dfg.value_lists);
+            for arg in args_iter {
+                hold_args.push(arg.to_string());
+            }
+            SerInstData::BranchIcmp {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                cond: cond.to_string(),
+                destination: destination.to_string(),
+            }
+        }
+        // to add: jump table serialization
+        InstructionData::BranchTable { opcode, arg, table } => SerInstData::BranchTable {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            table: table.to_string(),
+        },
+        InstructionData::Call {
+            opcode,
+            ref args,
+            func_ref,
+        } => {
+            let mut hold_args = Vec::new();
+            let args_iter = args.as_slice(&func.dfg.value_lists);
+            for arg in args_iter {
+                hold_args.push(arg.to_string());
+            }
+            SerInstData::Call {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                func_ref: func_ref.to_string(),
+            }
+        }
+        InstructionData::CallIndirect {
+            opcode,
+            ref args,
+            sig_ref,
+        } => {
+            let mut hold_args = Vec::new();
+            let args_iter = args.as_slice(&func.dfg.value_lists);
+            for arg in args_iter {
+                hold_args.push(arg.to_string());
+            }
+            SerInstData::CallIndirect {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                sig_ref: sig_ref.to_string(),
+            }
+        }
+        InstructionData::FuncAddr { opcode, func_ref } => SerInstData::FuncAddr {
+            opcode: opcode.to_string(),
+            func_ref: func_ref.to_string(),
+        },
+        InstructionData::Load {
+            opcode,
+            arg,
+            flags,
+            offset,
+        } => SerInstData::Load {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            flags: flags.to_string(),
+            offset: offset.to_string(),
+        },
+        InstructionData::LoadComplex {
+            opcode,
+            ref args,
+            flags,
+            offset,
+        } => {
+            let mut hold_args = Vec::new();
+            let args_iter = args.as_slice(&func.dfg.value_lists);
+            for arg in args_iter {
+                hold_args.push(arg.to_string());
+            }
+            SerInstData::LoadComplex {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                flags: flags.to_string(),
+                offset: offset.to_string(),
+            }
+        }
+        InstructionData::Store {
+            opcode,
+            args,
+            flags,
+            offset,
+        } => {
+            let hold_args = [args[0].to_string(), args[1].to_string()];
+            SerInstData::Store {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                flags: flags.to_string(),
+                offset: offset.to_string(),
+            }
+        }
+        InstructionData::StoreComplex {
+            opcode,
+            ref args,
+            flags,
+            offset,
+        } => {
+            let mut hold_args = Vec::new();
+            let args_iter = args.as_slice(&func.dfg.value_lists);
+            for arg in args_iter {
+                hold_args.push(arg.to_string());
+            }
+            SerInstData::StoreComplex {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                flags: flags.to_string(),
+                offset: offset.to_string(),
+            }
+        }
+        InstructionData::StackLoad {
+            opcode,
+            stack_slot,
+            offset,
+        } => SerInstData::StackLoad {
+            opcode: opcode.to_string(),
+            stack_slot: stack_slot.to_string(),
+            offset: offset.to_string(),
+        },
+        InstructionData::StackStore {
+            opcode,
+            arg,
+            stack_slot,
+            offset,
+        } => SerInstData::StackStore {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            stack_slot: stack_slot.to_string(),
+            offset: offset.to_string(),
+        },
+        InstructionData::HeapAddr {
+            opcode,
+            arg,
+            heap,
+            imm,
+        } => SerInstData::HeapAddr {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            heap: heap.to_string(),
+            imm: imm.to_string(),
+        },
+        InstructionData::RegMove {
+            opcode,
+            arg,
+            src,
+            dst,
+        } => SerInstData::RegMove {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            src: src.to_string(),
+            dst: dst.to_string(),
+        },
+        InstructionData::CopySpecial { opcode, src, dst } => SerInstData::CopySpecial {
+            opcode: opcode.to_string(),
+            src: src.to_string(),
+            dst: dst.to_string(),
+        },
+        InstructionData::RegSpill {
+            opcode,
+            arg,
+            src,
+            dst,
+        } => SerInstData::RegSpill {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            src: src.to_string(),
+            dst: dst.to_string(),
+        },
+        InstructionData::RegFill {
+            opcode,
+            arg,
+            src,
+            dst,
+        } => SerInstData::RegFill {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            src: src.to_string(),
+            dst: dst.to_string(),
+        },
+        InstructionData::Trap { opcode, code } => SerInstData::Trap {
+            opcode: opcode.to_string(),
+            code: code.to_string(),
+        },
+        InstructionData::CondTrap { opcode, arg, code } => SerInstData::CondTrap {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            code: code.to_string(),
+        },
+        InstructionData::IntCondTrap {
+            opcode,
+            arg,
+            cond,
+            code,
+        } => SerInstData::IntCondTrap {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            cond: cond.to_string(),
+            code: code.to_string(),
+        },
+        InstructionData::FloatCondTrap {
+            opcode,
+            arg,
+            cond,
+            code,
+        } => SerInstData::FloatCondTrap {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            cond: cond.to_string(),
+            code: code.to_string(),
+        },
+    }
+}
+
+#[derive(Clone, Deserialize, Serialize, Debug)]
+pub struct SerInst {
+    pub inst_name: String,
+    pub inst_data: SerInstData,
+}
+
+impl SerInst {
+    pub fn new(inst: Inst, func: &Function) -> Self {
+        Self {
+            inst_name: inst.to_string(),
+            inst_data: get_inst_data(inst, func),
+        }
+    }
+}
+
+#[derive(Clone, Deserialize, Serialize, Debug)]
+pub struct SerEbb {
+    pub ebb: String,
+    pub params: Vec,
+    pub insts: Vec,
+}
+
+impl SerEbb {
+    pub fn new(name: String) -> Self {
+        Self {
+            ebb: name,
+            params: Vec::new(),
+            insts: Vec::new(),
+        }
+    }
+}
+
+pub fn populate_inst(func: &Function, ebb: Ebb) -> Vec {
+    let mut ser_vec: Vec = Vec::new();
+    let ret_iter = func.layout.ebb_insts(ebb);
+    for inst in ret_iter {
+        let mut ser_inst: SerInst = SerInst::new(inst, &func);
+        ser_vec.push(ser_inst);
+    }
+    ser_vec
+}
+
+pub fn populate_params(func: &Function, ebb: &Ebb) -> Vec {
+    let mut ser_vec: Vec = Vec::new();
+    let parameters = func.dfg.ebb_params(*ebb);
+    for param in parameters {
+        ser_vec.push(param.to_string());
+    }
+    ser_vec
+}
+
+/// Serializable Data Flow Graph
+#[derive(Deserialize, Serialize, Debug)]
+pub struct SerDataFlowGraph {
+    ebbs: Vec,
+}
+
+pub fn populate_ebbs(func: &Function) -> Vec {
+    let mut ebb_vec: Vec = Vec::new();
+    for ebb in func.layout.ebbs() {
+        let mut ser_ebb: SerEbb = SerEbb::new(ebb.to_string());
+        ser_ebb.params = populate_params(&func, &ebb);
+        ser_ebb.insts = populate_inst(&func, ebb);
+        ebb_vec.push(ser_ebb);
+    }
+    ebb_vec
+}
+
+impl SerDataFlowGraph {
+    pub fn create_new(func: &Function) -> Self {
+        Self {
+            ebbs: populate_ebbs(func),
+        }
+    }
+
+    pub fn new(func: &Function) -> Self {
+        Self::create_new(func)
+    }
+}
+
+#[derive(Serialize, Deserialize, Debug)]
+pub struct SerSignature {
+    pub func_params: Vec,
+    pub func_returns: Vec,
+}
+
+impl SerSignature {
+    fn create_new(sig: &Signature) -> Self {
+        let mut params_vec: Vec = Vec::new();
+        let mut returns_vec: Vec = Vec::new();
+        for param in sig.params.iter() {
+            params_vec.push(param.to_string());
+        }
+        for ret in sig.returns.iter() {
+            returns_vec.push(ret.to_string());
+        }
+        Self {
+            func_params: params_vec,
+            func_returns: returns_vec,
+        }
+    }
+
+    pub fn new(func: &Function) -> Self {
+        Self::create_new(&func.signature)
+    }
+}
+
+/// Serializable Function type
+#[derive(Serialize, Deserialize, Debug)]
+pub struct SerFunction {
+    pub name: String,
+    pub signature: SerSignature,
+    pub globals: Vec,
+    pub dfg: SerDataFlowGraph,
+}
+
+impl SerFunction {
+    fn create_new(func: &Function) -> Self {
+        let mut global_vec: Vec = Vec::new();
+        for (glob_name, _) in func.global_values.iter() {
+            global_vec.push(glob_name.to_string());
+        }
+        Self {
+            name: func.name.to_string(),
+            signature: SerSignature::new(&func),
+            globals: global_vec,
+            dfg: SerDataFlowGraph::new(&func),
+        }
+    }
+
+    pub fn new(func: &Function) -> Self {
+        Self::create_new(func)
+    }
+}
+
+/// Must have SerObj for deserialization
+#[derive(Serialize, Deserialize, Debug)]
+pub struct SerObj {
+    pub functions: Vec,
+}
+
+impl SerObj {
+    fn create_new(funcs: Vec) -> Self {
+        Self { functions: funcs }
+    }
+
+    pub fn new(funcs: &Vec) -> Self {
+        let mut func_vec: Vec = Vec::new();
+        for func in funcs {
+            let mut ser_func: SerFunction = SerFunction::new(&func);
+            func_vec.push(ser_func);
+        }
+        Self::create_new(func_vec)
+    }
+}

From d9d40e1cdfd90f90d5cf76ee3f975a02a89519a3 Mon Sep 17 00:00:00 2001
From: data-pup 
Date: Tue, 31 Jul 2018 10:56:26 -0400
Subject: [PATCH 1946/3084] lib/codegen-meta moved into lib/codegen. (#423)

* lib/codegen-meta moved into lib/codegen.

* Renamed codegen-meta and existing meta.
---
 lib/codegen/Cargo.toml                                   | 5 ++---
 lib/codegen/build.rs                                     | 9 ++++-----
 lib/codegen/{meta => meta-python}/base/__init__.py       | 0
 lib/codegen/{meta => meta-python}/base/entities.py       | 0
 lib/codegen/{meta => meta-python}/base/formats.py        | 0
 lib/codegen/{meta => meta-python}/base/immediates.py     | 0
 lib/codegen/{meta => meta-python}/base/instructions.py   | 0
 lib/codegen/{meta => meta-python}/base/legalize.py       | 0
 lib/codegen/{meta => meta-python}/base/predicates.py     | 0
 lib/codegen/{meta => meta-python}/base/semantics.py      | 0
 lib/codegen/{meta => meta-python}/base/settings.py       | 0
 lib/codegen/{meta => meta-python}/base/types.py          | 0
 lib/codegen/{meta => meta-python}/build.py               | 0
 lib/codegen/{meta => meta-python}/cdsl/__init__.py       | 0
 lib/codegen/{meta => meta-python}/cdsl/ast.py            | 0
 lib/codegen/{meta => meta-python}/cdsl/formats.py        | 0
 lib/codegen/{meta => meta-python}/cdsl/instructions.py   | 0
 lib/codegen/{meta => meta-python}/cdsl/isa.py            | 0
 lib/codegen/{meta => meta-python}/cdsl/operands.py       | 0
 lib/codegen/{meta => meta-python}/cdsl/predicates.py     | 0
 lib/codegen/{meta => meta-python}/cdsl/registers.py      | 0
 lib/codegen/{meta => meta-python}/cdsl/settings.py       | 0
 lib/codegen/{meta => meta-python}/cdsl/test_ast.py       | 0
 lib/codegen/{meta => meta-python}/cdsl/test_package.py   | 0
 lib/codegen/{meta => meta-python}/cdsl/test_ti.py        | 0
 lib/codegen/{meta => meta-python}/cdsl/test_typevar.py   | 0
 lib/codegen/{meta => meta-python}/cdsl/test_xform.py     | 0
 lib/codegen/{meta => meta-python}/cdsl/ti.py             | 0
 lib/codegen/{meta => meta-python}/cdsl/types.py          | 0
 lib/codegen/{meta => meta-python}/cdsl/typevar.py        | 0
 lib/codegen/{meta => meta-python}/cdsl/xform.py          | 0
 lib/codegen/{meta => meta-python}/check.sh               | 0
 lib/codegen/{meta => meta-python}/constant_hash.py       | 0
 lib/codegen/{meta => meta-python}/gen_binemit.py         | 0
 lib/codegen/{meta => meta-python}/gen_build_deps.py      | 0
 lib/codegen/{meta => meta-python}/gen_encoding.py        | 0
 lib/codegen/{meta => meta-python}/gen_instr.py           | 0
 lib/codegen/{meta => meta-python}/gen_legalizer.py       | 0
 lib/codegen/{meta => meta-python}/gen_registers.py       | 0
 lib/codegen/{meta => meta-python}/gen_settings.py        | 0
 lib/codegen/{meta => meta-python}/gen_types.py           | 0
 lib/codegen/{meta => meta-python}/isa/__init__.py        | 0
 lib/codegen/{meta => meta-python}/isa/arm32/__init__.py  | 0
 lib/codegen/{meta => meta-python}/isa/arm32/defs.py      | 0
 lib/codegen/{meta => meta-python}/isa/arm32/registers.py | 0
 lib/codegen/{meta => meta-python}/isa/arm32/settings.py  | 0
 lib/codegen/{meta => meta-python}/isa/arm64/__init__.py  | 0
 lib/codegen/{meta => meta-python}/isa/arm64/defs.py      | 0
 lib/codegen/{meta => meta-python}/isa/arm64/registers.py | 0
 lib/codegen/{meta => meta-python}/isa/arm64/settings.py  | 0
 lib/codegen/{meta => meta-python}/isa/riscv/__init__.py  | 0
 lib/codegen/{meta => meta-python}/isa/riscv/defs.py      | 0
 lib/codegen/{meta => meta-python}/isa/riscv/encodings.py | 0
 lib/codegen/{meta => meta-python}/isa/riscv/recipes.py   | 0
 lib/codegen/{meta => meta-python}/isa/riscv/registers.py | 0
 lib/codegen/{meta => meta-python}/isa/riscv/settings.py  | 0
 lib/codegen/{meta => meta-python}/isa/x86/__init__.py    | 0
 lib/codegen/{meta => meta-python}/isa/x86/defs.py        | 0
 lib/codegen/{meta => meta-python}/isa/x86/encodings.py   | 0
 .../{meta => meta-python}/isa/x86/instructions.py        | 0
 lib/codegen/{meta => meta-python}/isa/x86/legalize.py    | 0
 lib/codegen/{meta => meta-python}/isa/x86/recipes.py     | 0
 lib/codegen/{meta => meta-python}/isa/x86/registers.py   | 0
 lib/codegen/{meta => meta-python}/isa/x86/settings.py    | 0
 lib/codegen/{meta => meta-python}/mypy.ini               | 0
 lib/codegen/{meta => meta-python}/semantics/__init__.py  | 0
 lib/codegen/{meta => meta-python}/semantics/elaborate.py | 0
 lib/codegen/{meta => meta-python}/semantics/macros.py    | 0
 .../{meta => meta-python}/semantics/primitives.py        | 0
 lib/codegen/{meta => meta-python}/semantics/smtlib.py    | 0
 .../{meta => meta-python}/semantics/test_elaborate.py    | 0
 lib/codegen/{meta => meta-python}/srcgen.py              | 0
 lib/codegen/{meta => meta-python}/stubs/z3/__init__.pyi  | 0
 lib/codegen/{meta => meta-python}/stubs/z3/z3core.pyi    | 0
 lib/codegen/{meta => meta-python}/stubs/z3/z3types.pyi   | 0
 lib/codegen/{meta => meta-python}/test_constant_hash.py  | 0
 lib/codegen/{meta => meta-python}/test_gen_legalizer.py  | 0
 lib/codegen/{meta => meta-python}/test_srcgen.py         | 0
 lib/codegen/{meta => meta-python}/unique_table.py        | 0
 lib/{codegen-meta => codegen/meta}/Cargo.toml            | 2 +-
 lib/{codegen-meta => codegen/meta}/LICENSE               | 0
 lib/{codegen-meta => codegen/meta}/src/base/mod.rs       | 0
 lib/{codegen-meta => codegen/meta}/src/base/types.rs     | 0
 lib/{codegen-meta => codegen/meta}/src/cdsl/mod.rs       | 0
 lib/{codegen-meta => codegen/meta}/src/cdsl/types.rs     | 0
 lib/{codegen-meta => codegen/meta}/src/error.rs          | 0
 lib/{codegen-meta => codegen/meta}/src/gen_types.rs      | 0
 lib/{codegen-meta => codegen/meta}/src/lib.rs            | 0
 lib/{codegen-meta => codegen/meta}/src/srcgen.rs         | 0
 89 files changed, 7 insertions(+), 9 deletions(-)
 rename lib/codegen/{meta => meta-python}/base/__init__.py (100%)
 rename lib/codegen/{meta => meta-python}/base/entities.py (100%)
 rename lib/codegen/{meta => meta-python}/base/formats.py (100%)
 rename lib/codegen/{meta => meta-python}/base/immediates.py (100%)
 rename lib/codegen/{meta => meta-python}/base/instructions.py (100%)
 rename lib/codegen/{meta => meta-python}/base/legalize.py (100%)
 rename lib/codegen/{meta => meta-python}/base/predicates.py (100%)
 rename lib/codegen/{meta => meta-python}/base/semantics.py (100%)
 rename lib/codegen/{meta => meta-python}/base/settings.py (100%)
 rename lib/codegen/{meta => meta-python}/base/types.py (100%)
 rename lib/codegen/{meta => meta-python}/build.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/__init__.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/ast.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/formats.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/instructions.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/isa.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/operands.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/predicates.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/registers.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/settings.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/test_ast.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/test_package.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/test_ti.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/test_typevar.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/test_xform.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/ti.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/types.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/typevar.py (100%)
 rename lib/codegen/{meta => meta-python}/cdsl/xform.py (100%)
 rename lib/codegen/{meta => meta-python}/check.sh (100%)
 rename lib/codegen/{meta => meta-python}/constant_hash.py (100%)
 rename lib/codegen/{meta => meta-python}/gen_binemit.py (100%)
 rename lib/codegen/{meta => meta-python}/gen_build_deps.py (100%)
 rename lib/codegen/{meta => meta-python}/gen_encoding.py (100%)
 rename lib/codegen/{meta => meta-python}/gen_instr.py (100%)
 rename lib/codegen/{meta => meta-python}/gen_legalizer.py (100%)
 rename lib/codegen/{meta => meta-python}/gen_registers.py (100%)
 rename lib/codegen/{meta => meta-python}/gen_settings.py (100%)
 rename lib/codegen/{meta => meta-python}/gen_types.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/__init__.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/arm32/__init__.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/arm32/defs.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/arm32/registers.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/arm32/settings.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/arm64/__init__.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/arm64/defs.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/arm64/registers.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/arm64/settings.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/riscv/__init__.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/riscv/defs.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/riscv/encodings.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/riscv/recipes.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/riscv/registers.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/riscv/settings.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/x86/__init__.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/x86/defs.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/x86/encodings.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/x86/instructions.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/x86/legalize.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/x86/recipes.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/x86/registers.py (100%)
 rename lib/codegen/{meta => meta-python}/isa/x86/settings.py (100%)
 rename lib/codegen/{meta => meta-python}/mypy.ini (100%)
 rename lib/codegen/{meta => meta-python}/semantics/__init__.py (100%)
 rename lib/codegen/{meta => meta-python}/semantics/elaborate.py (100%)
 rename lib/codegen/{meta => meta-python}/semantics/macros.py (100%)
 rename lib/codegen/{meta => meta-python}/semantics/primitives.py (100%)
 rename lib/codegen/{meta => meta-python}/semantics/smtlib.py (100%)
 rename lib/codegen/{meta => meta-python}/semantics/test_elaborate.py (100%)
 rename lib/codegen/{meta => meta-python}/srcgen.py (100%)
 rename lib/codegen/{meta => meta-python}/stubs/z3/__init__.pyi (100%)
 rename lib/codegen/{meta => meta-python}/stubs/z3/z3core.pyi (100%)
 rename lib/codegen/{meta => meta-python}/stubs/z3/z3types.pyi (100%)
 rename lib/codegen/{meta => meta-python}/test_constant_hash.py (100%)
 rename lib/codegen/{meta => meta-python}/test_gen_legalizer.py (100%)
 rename lib/codegen/{meta => meta-python}/test_srcgen.py (100%)
 rename lib/codegen/{meta => meta-python}/unique_table.py (100%)
 rename lib/{codegen-meta => codegen/meta}/Cargo.toml (95%)
 rename lib/{codegen-meta => codegen/meta}/LICENSE (100%)
 rename lib/{codegen-meta => codegen/meta}/src/base/mod.rs (100%)
 rename lib/{codegen-meta => codegen/meta}/src/base/types.rs (100%)
 rename lib/{codegen-meta => codegen/meta}/src/cdsl/mod.rs (100%)
 rename lib/{codegen-meta => codegen/meta}/src/cdsl/types.rs (100%)
 rename lib/{codegen-meta => codegen/meta}/src/error.rs (100%)
 rename lib/{codegen-meta => codegen/meta}/src/gen_types.rs (100%)
 rename lib/{codegen-meta => codegen/meta}/src/lib.rs (100%)
 rename lib/{codegen-meta => codegen/meta}/src/srcgen.rs (100%)

diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index f5bee967fa..b25fb6f16c 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -21,9 +21,8 @@ target-lexicon = { version = "0.0.3", default-features = false }
 # machine code. Integration tests that need external dependencies can be
 # accomodated in `tests`.
 
-# Temporarily disable this while we work out how to publish this crate.
-#[build-dependencies]
-#cranelift-codegen-meta = { path = "../codegen-meta", version = "0.17.0-alpha" }
+[build-dependencies]
+cranelift-meta = { path = "meta", version = "0.17.0-alpha" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index 78a87f1ee7..dedaa35f8a 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -18,8 +18,7 @@
 // The build script expects to be run from the directory where this build.rs file lives. The
 // current directory is used to find the sources.
 
-// Temporarily disable this while we work out how to publish this crate.
-//extern crate cranelift_codegen_meta as meta;
+extern crate cranelift_meta as meta;
 
 use std::env;
 use std::process;
@@ -57,8 +56,8 @@ fn main() {
         crate_dir.join("build.rs").to_str().unwrap()
     );
 
-    // Scripts are in `$crate_dir/meta`.
-    let meta_dir = crate_dir.join("meta");
+    // Scripts are in `$crate_dir/meta-python`.
+    let meta_dir = crate_dir.join("meta-python");
     let build_script = meta_dir.join("build.py");
 
     // Launch build script with Python. We'll just find python in the path.
@@ -79,7 +78,7 @@ fn main() {
     // DEVELOPMENT:
     // ------------------------------------------------------------------------
     // Now that the Python build process is complete, generate files that are
-    // emitted by the `cretonne_codegen_meta` crate.
+    // emitted by the `meta` crate.
     // ------------------------------------------------------------------------
     // Temporarily disable this while we work out how to publish this crate.
     //if let Err(err) = meta::gen_types::generate("new_types.rs", &out_dir) {
diff --git a/lib/codegen/meta/base/__init__.py b/lib/codegen/meta-python/base/__init__.py
similarity index 100%
rename from lib/codegen/meta/base/__init__.py
rename to lib/codegen/meta-python/base/__init__.py
diff --git a/lib/codegen/meta/base/entities.py b/lib/codegen/meta-python/base/entities.py
similarity index 100%
rename from lib/codegen/meta/base/entities.py
rename to lib/codegen/meta-python/base/entities.py
diff --git a/lib/codegen/meta/base/formats.py b/lib/codegen/meta-python/base/formats.py
similarity index 100%
rename from lib/codegen/meta/base/formats.py
rename to lib/codegen/meta-python/base/formats.py
diff --git a/lib/codegen/meta/base/immediates.py b/lib/codegen/meta-python/base/immediates.py
similarity index 100%
rename from lib/codegen/meta/base/immediates.py
rename to lib/codegen/meta-python/base/immediates.py
diff --git a/lib/codegen/meta/base/instructions.py b/lib/codegen/meta-python/base/instructions.py
similarity index 100%
rename from lib/codegen/meta/base/instructions.py
rename to lib/codegen/meta-python/base/instructions.py
diff --git a/lib/codegen/meta/base/legalize.py b/lib/codegen/meta-python/base/legalize.py
similarity index 100%
rename from lib/codegen/meta/base/legalize.py
rename to lib/codegen/meta-python/base/legalize.py
diff --git a/lib/codegen/meta/base/predicates.py b/lib/codegen/meta-python/base/predicates.py
similarity index 100%
rename from lib/codegen/meta/base/predicates.py
rename to lib/codegen/meta-python/base/predicates.py
diff --git a/lib/codegen/meta/base/semantics.py b/lib/codegen/meta-python/base/semantics.py
similarity index 100%
rename from lib/codegen/meta/base/semantics.py
rename to lib/codegen/meta-python/base/semantics.py
diff --git a/lib/codegen/meta/base/settings.py b/lib/codegen/meta-python/base/settings.py
similarity index 100%
rename from lib/codegen/meta/base/settings.py
rename to lib/codegen/meta-python/base/settings.py
diff --git a/lib/codegen/meta/base/types.py b/lib/codegen/meta-python/base/types.py
similarity index 100%
rename from lib/codegen/meta/base/types.py
rename to lib/codegen/meta-python/base/types.py
diff --git a/lib/codegen/meta/build.py b/lib/codegen/meta-python/build.py
similarity index 100%
rename from lib/codegen/meta/build.py
rename to lib/codegen/meta-python/build.py
diff --git a/lib/codegen/meta/cdsl/__init__.py b/lib/codegen/meta-python/cdsl/__init__.py
similarity index 100%
rename from lib/codegen/meta/cdsl/__init__.py
rename to lib/codegen/meta-python/cdsl/__init__.py
diff --git a/lib/codegen/meta/cdsl/ast.py b/lib/codegen/meta-python/cdsl/ast.py
similarity index 100%
rename from lib/codegen/meta/cdsl/ast.py
rename to lib/codegen/meta-python/cdsl/ast.py
diff --git a/lib/codegen/meta/cdsl/formats.py b/lib/codegen/meta-python/cdsl/formats.py
similarity index 100%
rename from lib/codegen/meta/cdsl/formats.py
rename to lib/codegen/meta-python/cdsl/formats.py
diff --git a/lib/codegen/meta/cdsl/instructions.py b/lib/codegen/meta-python/cdsl/instructions.py
similarity index 100%
rename from lib/codegen/meta/cdsl/instructions.py
rename to lib/codegen/meta-python/cdsl/instructions.py
diff --git a/lib/codegen/meta/cdsl/isa.py b/lib/codegen/meta-python/cdsl/isa.py
similarity index 100%
rename from lib/codegen/meta/cdsl/isa.py
rename to lib/codegen/meta-python/cdsl/isa.py
diff --git a/lib/codegen/meta/cdsl/operands.py b/lib/codegen/meta-python/cdsl/operands.py
similarity index 100%
rename from lib/codegen/meta/cdsl/operands.py
rename to lib/codegen/meta-python/cdsl/operands.py
diff --git a/lib/codegen/meta/cdsl/predicates.py b/lib/codegen/meta-python/cdsl/predicates.py
similarity index 100%
rename from lib/codegen/meta/cdsl/predicates.py
rename to lib/codegen/meta-python/cdsl/predicates.py
diff --git a/lib/codegen/meta/cdsl/registers.py b/lib/codegen/meta-python/cdsl/registers.py
similarity index 100%
rename from lib/codegen/meta/cdsl/registers.py
rename to lib/codegen/meta-python/cdsl/registers.py
diff --git a/lib/codegen/meta/cdsl/settings.py b/lib/codegen/meta-python/cdsl/settings.py
similarity index 100%
rename from lib/codegen/meta/cdsl/settings.py
rename to lib/codegen/meta-python/cdsl/settings.py
diff --git a/lib/codegen/meta/cdsl/test_ast.py b/lib/codegen/meta-python/cdsl/test_ast.py
similarity index 100%
rename from lib/codegen/meta/cdsl/test_ast.py
rename to lib/codegen/meta-python/cdsl/test_ast.py
diff --git a/lib/codegen/meta/cdsl/test_package.py b/lib/codegen/meta-python/cdsl/test_package.py
similarity index 100%
rename from lib/codegen/meta/cdsl/test_package.py
rename to lib/codegen/meta-python/cdsl/test_package.py
diff --git a/lib/codegen/meta/cdsl/test_ti.py b/lib/codegen/meta-python/cdsl/test_ti.py
similarity index 100%
rename from lib/codegen/meta/cdsl/test_ti.py
rename to lib/codegen/meta-python/cdsl/test_ti.py
diff --git a/lib/codegen/meta/cdsl/test_typevar.py b/lib/codegen/meta-python/cdsl/test_typevar.py
similarity index 100%
rename from lib/codegen/meta/cdsl/test_typevar.py
rename to lib/codegen/meta-python/cdsl/test_typevar.py
diff --git a/lib/codegen/meta/cdsl/test_xform.py b/lib/codegen/meta-python/cdsl/test_xform.py
similarity index 100%
rename from lib/codegen/meta/cdsl/test_xform.py
rename to lib/codegen/meta-python/cdsl/test_xform.py
diff --git a/lib/codegen/meta/cdsl/ti.py b/lib/codegen/meta-python/cdsl/ti.py
similarity index 100%
rename from lib/codegen/meta/cdsl/ti.py
rename to lib/codegen/meta-python/cdsl/ti.py
diff --git a/lib/codegen/meta/cdsl/types.py b/lib/codegen/meta-python/cdsl/types.py
similarity index 100%
rename from lib/codegen/meta/cdsl/types.py
rename to lib/codegen/meta-python/cdsl/types.py
diff --git a/lib/codegen/meta/cdsl/typevar.py b/lib/codegen/meta-python/cdsl/typevar.py
similarity index 100%
rename from lib/codegen/meta/cdsl/typevar.py
rename to lib/codegen/meta-python/cdsl/typevar.py
diff --git a/lib/codegen/meta/cdsl/xform.py b/lib/codegen/meta-python/cdsl/xform.py
similarity index 100%
rename from lib/codegen/meta/cdsl/xform.py
rename to lib/codegen/meta-python/cdsl/xform.py
diff --git a/lib/codegen/meta/check.sh b/lib/codegen/meta-python/check.sh
similarity index 100%
rename from lib/codegen/meta/check.sh
rename to lib/codegen/meta-python/check.sh
diff --git a/lib/codegen/meta/constant_hash.py b/lib/codegen/meta-python/constant_hash.py
similarity index 100%
rename from lib/codegen/meta/constant_hash.py
rename to lib/codegen/meta-python/constant_hash.py
diff --git a/lib/codegen/meta/gen_binemit.py b/lib/codegen/meta-python/gen_binemit.py
similarity index 100%
rename from lib/codegen/meta/gen_binemit.py
rename to lib/codegen/meta-python/gen_binemit.py
diff --git a/lib/codegen/meta/gen_build_deps.py b/lib/codegen/meta-python/gen_build_deps.py
similarity index 100%
rename from lib/codegen/meta/gen_build_deps.py
rename to lib/codegen/meta-python/gen_build_deps.py
diff --git a/lib/codegen/meta/gen_encoding.py b/lib/codegen/meta-python/gen_encoding.py
similarity index 100%
rename from lib/codegen/meta/gen_encoding.py
rename to lib/codegen/meta-python/gen_encoding.py
diff --git a/lib/codegen/meta/gen_instr.py b/lib/codegen/meta-python/gen_instr.py
similarity index 100%
rename from lib/codegen/meta/gen_instr.py
rename to lib/codegen/meta-python/gen_instr.py
diff --git a/lib/codegen/meta/gen_legalizer.py b/lib/codegen/meta-python/gen_legalizer.py
similarity index 100%
rename from lib/codegen/meta/gen_legalizer.py
rename to lib/codegen/meta-python/gen_legalizer.py
diff --git a/lib/codegen/meta/gen_registers.py b/lib/codegen/meta-python/gen_registers.py
similarity index 100%
rename from lib/codegen/meta/gen_registers.py
rename to lib/codegen/meta-python/gen_registers.py
diff --git a/lib/codegen/meta/gen_settings.py b/lib/codegen/meta-python/gen_settings.py
similarity index 100%
rename from lib/codegen/meta/gen_settings.py
rename to lib/codegen/meta-python/gen_settings.py
diff --git a/lib/codegen/meta/gen_types.py b/lib/codegen/meta-python/gen_types.py
similarity index 100%
rename from lib/codegen/meta/gen_types.py
rename to lib/codegen/meta-python/gen_types.py
diff --git a/lib/codegen/meta/isa/__init__.py b/lib/codegen/meta-python/isa/__init__.py
similarity index 100%
rename from lib/codegen/meta/isa/__init__.py
rename to lib/codegen/meta-python/isa/__init__.py
diff --git a/lib/codegen/meta/isa/arm32/__init__.py b/lib/codegen/meta-python/isa/arm32/__init__.py
similarity index 100%
rename from lib/codegen/meta/isa/arm32/__init__.py
rename to lib/codegen/meta-python/isa/arm32/__init__.py
diff --git a/lib/codegen/meta/isa/arm32/defs.py b/lib/codegen/meta-python/isa/arm32/defs.py
similarity index 100%
rename from lib/codegen/meta/isa/arm32/defs.py
rename to lib/codegen/meta-python/isa/arm32/defs.py
diff --git a/lib/codegen/meta/isa/arm32/registers.py b/lib/codegen/meta-python/isa/arm32/registers.py
similarity index 100%
rename from lib/codegen/meta/isa/arm32/registers.py
rename to lib/codegen/meta-python/isa/arm32/registers.py
diff --git a/lib/codegen/meta/isa/arm32/settings.py b/lib/codegen/meta-python/isa/arm32/settings.py
similarity index 100%
rename from lib/codegen/meta/isa/arm32/settings.py
rename to lib/codegen/meta-python/isa/arm32/settings.py
diff --git a/lib/codegen/meta/isa/arm64/__init__.py b/lib/codegen/meta-python/isa/arm64/__init__.py
similarity index 100%
rename from lib/codegen/meta/isa/arm64/__init__.py
rename to lib/codegen/meta-python/isa/arm64/__init__.py
diff --git a/lib/codegen/meta/isa/arm64/defs.py b/lib/codegen/meta-python/isa/arm64/defs.py
similarity index 100%
rename from lib/codegen/meta/isa/arm64/defs.py
rename to lib/codegen/meta-python/isa/arm64/defs.py
diff --git a/lib/codegen/meta/isa/arm64/registers.py b/lib/codegen/meta-python/isa/arm64/registers.py
similarity index 100%
rename from lib/codegen/meta/isa/arm64/registers.py
rename to lib/codegen/meta-python/isa/arm64/registers.py
diff --git a/lib/codegen/meta/isa/arm64/settings.py b/lib/codegen/meta-python/isa/arm64/settings.py
similarity index 100%
rename from lib/codegen/meta/isa/arm64/settings.py
rename to lib/codegen/meta-python/isa/arm64/settings.py
diff --git a/lib/codegen/meta/isa/riscv/__init__.py b/lib/codegen/meta-python/isa/riscv/__init__.py
similarity index 100%
rename from lib/codegen/meta/isa/riscv/__init__.py
rename to lib/codegen/meta-python/isa/riscv/__init__.py
diff --git a/lib/codegen/meta/isa/riscv/defs.py b/lib/codegen/meta-python/isa/riscv/defs.py
similarity index 100%
rename from lib/codegen/meta/isa/riscv/defs.py
rename to lib/codegen/meta-python/isa/riscv/defs.py
diff --git a/lib/codegen/meta/isa/riscv/encodings.py b/lib/codegen/meta-python/isa/riscv/encodings.py
similarity index 100%
rename from lib/codegen/meta/isa/riscv/encodings.py
rename to lib/codegen/meta-python/isa/riscv/encodings.py
diff --git a/lib/codegen/meta/isa/riscv/recipes.py b/lib/codegen/meta-python/isa/riscv/recipes.py
similarity index 100%
rename from lib/codegen/meta/isa/riscv/recipes.py
rename to lib/codegen/meta-python/isa/riscv/recipes.py
diff --git a/lib/codegen/meta/isa/riscv/registers.py b/lib/codegen/meta-python/isa/riscv/registers.py
similarity index 100%
rename from lib/codegen/meta/isa/riscv/registers.py
rename to lib/codegen/meta-python/isa/riscv/registers.py
diff --git a/lib/codegen/meta/isa/riscv/settings.py b/lib/codegen/meta-python/isa/riscv/settings.py
similarity index 100%
rename from lib/codegen/meta/isa/riscv/settings.py
rename to lib/codegen/meta-python/isa/riscv/settings.py
diff --git a/lib/codegen/meta/isa/x86/__init__.py b/lib/codegen/meta-python/isa/x86/__init__.py
similarity index 100%
rename from lib/codegen/meta/isa/x86/__init__.py
rename to lib/codegen/meta-python/isa/x86/__init__.py
diff --git a/lib/codegen/meta/isa/x86/defs.py b/lib/codegen/meta-python/isa/x86/defs.py
similarity index 100%
rename from lib/codegen/meta/isa/x86/defs.py
rename to lib/codegen/meta-python/isa/x86/defs.py
diff --git a/lib/codegen/meta/isa/x86/encodings.py b/lib/codegen/meta-python/isa/x86/encodings.py
similarity index 100%
rename from lib/codegen/meta/isa/x86/encodings.py
rename to lib/codegen/meta-python/isa/x86/encodings.py
diff --git a/lib/codegen/meta/isa/x86/instructions.py b/lib/codegen/meta-python/isa/x86/instructions.py
similarity index 100%
rename from lib/codegen/meta/isa/x86/instructions.py
rename to lib/codegen/meta-python/isa/x86/instructions.py
diff --git a/lib/codegen/meta/isa/x86/legalize.py b/lib/codegen/meta-python/isa/x86/legalize.py
similarity index 100%
rename from lib/codegen/meta/isa/x86/legalize.py
rename to lib/codegen/meta-python/isa/x86/legalize.py
diff --git a/lib/codegen/meta/isa/x86/recipes.py b/lib/codegen/meta-python/isa/x86/recipes.py
similarity index 100%
rename from lib/codegen/meta/isa/x86/recipes.py
rename to lib/codegen/meta-python/isa/x86/recipes.py
diff --git a/lib/codegen/meta/isa/x86/registers.py b/lib/codegen/meta-python/isa/x86/registers.py
similarity index 100%
rename from lib/codegen/meta/isa/x86/registers.py
rename to lib/codegen/meta-python/isa/x86/registers.py
diff --git a/lib/codegen/meta/isa/x86/settings.py b/lib/codegen/meta-python/isa/x86/settings.py
similarity index 100%
rename from lib/codegen/meta/isa/x86/settings.py
rename to lib/codegen/meta-python/isa/x86/settings.py
diff --git a/lib/codegen/meta/mypy.ini b/lib/codegen/meta-python/mypy.ini
similarity index 100%
rename from lib/codegen/meta/mypy.ini
rename to lib/codegen/meta-python/mypy.ini
diff --git a/lib/codegen/meta/semantics/__init__.py b/lib/codegen/meta-python/semantics/__init__.py
similarity index 100%
rename from lib/codegen/meta/semantics/__init__.py
rename to lib/codegen/meta-python/semantics/__init__.py
diff --git a/lib/codegen/meta/semantics/elaborate.py b/lib/codegen/meta-python/semantics/elaborate.py
similarity index 100%
rename from lib/codegen/meta/semantics/elaborate.py
rename to lib/codegen/meta-python/semantics/elaborate.py
diff --git a/lib/codegen/meta/semantics/macros.py b/lib/codegen/meta-python/semantics/macros.py
similarity index 100%
rename from lib/codegen/meta/semantics/macros.py
rename to lib/codegen/meta-python/semantics/macros.py
diff --git a/lib/codegen/meta/semantics/primitives.py b/lib/codegen/meta-python/semantics/primitives.py
similarity index 100%
rename from lib/codegen/meta/semantics/primitives.py
rename to lib/codegen/meta-python/semantics/primitives.py
diff --git a/lib/codegen/meta/semantics/smtlib.py b/lib/codegen/meta-python/semantics/smtlib.py
similarity index 100%
rename from lib/codegen/meta/semantics/smtlib.py
rename to lib/codegen/meta-python/semantics/smtlib.py
diff --git a/lib/codegen/meta/semantics/test_elaborate.py b/lib/codegen/meta-python/semantics/test_elaborate.py
similarity index 100%
rename from lib/codegen/meta/semantics/test_elaborate.py
rename to lib/codegen/meta-python/semantics/test_elaborate.py
diff --git a/lib/codegen/meta/srcgen.py b/lib/codegen/meta-python/srcgen.py
similarity index 100%
rename from lib/codegen/meta/srcgen.py
rename to lib/codegen/meta-python/srcgen.py
diff --git a/lib/codegen/meta/stubs/z3/__init__.pyi b/lib/codegen/meta-python/stubs/z3/__init__.pyi
similarity index 100%
rename from lib/codegen/meta/stubs/z3/__init__.pyi
rename to lib/codegen/meta-python/stubs/z3/__init__.pyi
diff --git a/lib/codegen/meta/stubs/z3/z3core.pyi b/lib/codegen/meta-python/stubs/z3/z3core.pyi
similarity index 100%
rename from lib/codegen/meta/stubs/z3/z3core.pyi
rename to lib/codegen/meta-python/stubs/z3/z3core.pyi
diff --git a/lib/codegen/meta/stubs/z3/z3types.pyi b/lib/codegen/meta-python/stubs/z3/z3types.pyi
similarity index 100%
rename from lib/codegen/meta/stubs/z3/z3types.pyi
rename to lib/codegen/meta-python/stubs/z3/z3types.pyi
diff --git a/lib/codegen/meta/test_constant_hash.py b/lib/codegen/meta-python/test_constant_hash.py
similarity index 100%
rename from lib/codegen/meta/test_constant_hash.py
rename to lib/codegen/meta-python/test_constant_hash.py
diff --git a/lib/codegen/meta/test_gen_legalizer.py b/lib/codegen/meta-python/test_gen_legalizer.py
similarity index 100%
rename from lib/codegen/meta/test_gen_legalizer.py
rename to lib/codegen/meta-python/test_gen_legalizer.py
diff --git a/lib/codegen/meta/test_srcgen.py b/lib/codegen/meta-python/test_srcgen.py
similarity index 100%
rename from lib/codegen/meta/test_srcgen.py
rename to lib/codegen/meta-python/test_srcgen.py
diff --git a/lib/codegen/meta/unique_table.py b/lib/codegen/meta-python/unique_table.py
similarity index 100%
rename from lib/codegen/meta/unique_table.py
rename to lib/codegen/meta-python/unique_table.py
diff --git a/lib/codegen-meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
similarity index 95%
rename from lib/codegen-meta/Cargo.toml
rename to lib/codegen/meta/Cargo.toml
index cebb2545b0..40ac0f416a 100644
--- a/lib/codegen-meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,5 +1,5 @@
 [package]
-name = "cranelift-codegen-meta"
+name = "cranelift-meta"
 authors = ["The Cranelift Project Developers"]
 version = "0.17.0"
 description = "Metaprogram for cranelift-codegen code generator library"
diff --git a/lib/codegen-meta/LICENSE b/lib/codegen/meta/LICENSE
similarity index 100%
rename from lib/codegen-meta/LICENSE
rename to lib/codegen/meta/LICENSE
diff --git a/lib/codegen-meta/src/base/mod.rs b/lib/codegen/meta/src/base/mod.rs
similarity index 100%
rename from lib/codegen-meta/src/base/mod.rs
rename to lib/codegen/meta/src/base/mod.rs
diff --git a/lib/codegen-meta/src/base/types.rs b/lib/codegen/meta/src/base/types.rs
similarity index 100%
rename from lib/codegen-meta/src/base/types.rs
rename to lib/codegen/meta/src/base/types.rs
diff --git a/lib/codegen-meta/src/cdsl/mod.rs b/lib/codegen/meta/src/cdsl/mod.rs
similarity index 100%
rename from lib/codegen-meta/src/cdsl/mod.rs
rename to lib/codegen/meta/src/cdsl/mod.rs
diff --git a/lib/codegen-meta/src/cdsl/types.rs b/lib/codegen/meta/src/cdsl/types.rs
similarity index 100%
rename from lib/codegen-meta/src/cdsl/types.rs
rename to lib/codegen/meta/src/cdsl/types.rs
diff --git a/lib/codegen-meta/src/error.rs b/lib/codegen/meta/src/error.rs
similarity index 100%
rename from lib/codegen-meta/src/error.rs
rename to lib/codegen/meta/src/error.rs
diff --git a/lib/codegen-meta/src/gen_types.rs b/lib/codegen/meta/src/gen_types.rs
similarity index 100%
rename from lib/codegen-meta/src/gen_types.rs
rename to lib/codegen/meta/src/gen_types.rs
diff --git a/lib/codegen-meta/src/lib.rs b/lib/codegen/meta/src/lib.rs
similarity index 100%
rename from lib/codegen-meta/src/lib.rs
rename to lib/codegen/meta/src/lib.rs
diff --git a/lib/codegen-meta/src/srcgen.rs b/lib/codegen/meta/src/srcgen.rs
similarity index 100%
rename from lib/codegen-meta/src/srcgen.rs
rename to lib/codegen/meta/src/srcgen.rs

From 1b42105faa52df2d768c266a2a20fa9c9d95abb3 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 31 Jul 2018 07:57:37 -0700
Subject: [PATCH 1947/3084] Remove reserved_reg functionality. (#424)

* Remove reserved_reg functionality.

This wasn't implemented, and if we need it in the future, it seems like
it would be better to extend the concept of global values to cover this.

* Use GlobalValue::reserved_value() for sentinal values.
---
 cranelift/docs/langref.rst             | 10 ++--------
 cranelift/filetests/parser/memory.clif |  8 ++++----
 lib/codegen/src/ir/heap.rs             | 23 +++--------------------
 lib/codegen/src/ir/mod.rs              |  2 +-
 lib/codegen/src/legalizer/heap.rs      | 10 +++-------
 lib/reader/src/parser.rs               | 25 +++++++++----------------
 lib/wasm/src/environ/dummy.rs          |  2 +-
 7 files changed, 23 insertions(+), 57 deletions(-)

diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst
index 5265d9b7b6..52be70c7aa 100644
--- a/cranelift/docs/langref.rst
+++ b/cranelift/docs/langref.rst
@@ -689,16 +689,13 @@ trap when accessed.
 
     Declare a static heap in the preamble.
 
-    :arg Base: Global value holding the heap's base address or
-            ``reserved_reg``.
+    :arg Base: Global value holding the heap's base address.
     :arg MinBytes: Guaranteed minimum heap size in bytes. Accesses below this
             size will never trap.
     :arg BoundBytes: Fixed heap bound in bytes. This defines the amount of
             address space reserved for the heap, not including the guard pages.
     :arg GuardBytes: Size of the guard pages in bytes.
 
-The ``reserved_reg`` option is not yet implemented.
-
 Dynamic heaps
 ~~~~~~~~~~~~~
 
@@ -710,15 +707,12 @@ is resized. The bound of a dynamic heap is stored in a global value.
 
     Declare a dynamic heap in the preamble.
 
-    :arg Base: Global value holding the heap's base address or
-            ``reserved_reg``.
+    :arg Base: Global value holding the heap's base address.
     :arg MinBytes: Guaranteed minimum heap size in bytes. Accesses below this
             size will never trap.
     :arg BoundGV: Global value containing the current heap bound in bytes.
     :arg GuardBytes: Size of the guard pages in bytes.
 
-The ``reserved_reg`` option is not yet implemented.
-
 Heap examples
 ~~~~~~~~~~~~~
 
diff --git a/cranelift/filetests/parser/memory.clif b/cranelift/filetests/parser/memory.clif
index 6c9dd6440f..cd59b892c6 100644
--- a/cranelift/filetests/parser/memory.clif
+++ b/cranelift/filetests/parser/memory.clif
@@ -52,11 +52,11 @@ ebb0:
 
 ; Declare static heaps.
 function %sheap(i32, i64 vmctx) -> i64 {
-    heap1 = static reserved_reg, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000
+    heap1 = static gv5, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000
     heap2 = static gv5, guard 0x1000, bound 0x1_0000
     gv5 = vmctx+64
 
-    ; check: heap1 = static reserved_reg, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    ; check: heap1 = static gv5, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
     ; check: heap2 = static gv5, min 0, bound 0x0001_0000, guard 4096
 ebb0(v1: i32, v2: i64):
     v3 = heap_addr.i64 heap1, v1, 0
@@ -66,12 +66,12 @@ ebb0(v1: i32, v2: i64):
 
 ; Declare dynamic heaps.
 function %dheap(i32, i64 vmctx) -> i64 {
-    heap1 = dynamic reserved_reg, min 0x1_0000, bound gv6, guard 0x8000_0000
+    heap1 = dynamic gv5, min 0x1_0000, bound gv6, guard 0x8000_0000
     heap2 = dynamic gv5, bound gv6, guard 0x1000
     gv5 = vmctx+64
     gv6 = vmctx+72
 
-    ; check: heap1 = dynamic reserved_reg, min 0x0001_0000, bound gv6, guard 0x8000_0000
+    ; check: heap1 = dynamic gv5, min 0x0001_0000, bound gv6, guard 0x8000_0000
     ; check: heap2 = dynamic gv5, min 0, bound gv6, guard 4096
 ebb0(v1: i32, v2: i64):
     v3 = heap_addr.i64 heap2, v1, 0
diff --git a/lib/codegen/src/ir/heap.rs b/lib/codegen/src/ir/heap.rs
index 439b8aac49..e0c942d366 100644
--- a/lib/codegen/src/ir/heap.rs
+++ b/lib/codegen/src/ir/heap.rs
@@ -7,8 +7,8 @@ use std::fmt;
 /// Information about a heap declaration.
 #[derive(Clone)]
 pub struct HeapData {
-    /// Method for determining the heap base address.
-    pub base: HeapBase,
+    /// The address of the start of the heap's storage.
+    pub base: GlobalValue,
 
     /// Guaranteed minimum heap size in bytes. Heap accesses before `min_size` don't need bounds
     /// checking.
@@ -21,18 +21,6 @@ pub struct HeapData {
     pub style: HeapStyle,
 }
 
-/// Method for determining the base address of a heap.
-#[derive(Clone)]
-pub enum HeapBase {
-    /// The heap base lives in a reserved register.
-    ///
-    /// This feature is not yet implemented.
-    ReservedReg,
-
-    /// The heap base is a global value.
-    GlobalValue(GlobalValue),
-}
-
 /// Style of heap including style-specific information.
 #[derive(Clone)]
 pub enum HeapStyle {
@@ -57,12 +45,7 @@ impl fmt::Display for HeapData {
             HeapStyle::Static { .. } => "static",
         })?;
 
-        match self.base {
-            HeapBase::ReservedReg => write!(f, " reserved_reg")?,
-            HeapBase::GlobalValue(gv) => write!(f, " {}", gv)?,
-        }
-
-        write!(f, ", min {}", self.min_size)?;
+        write!(f, " {}, min {}", self.base, self.min_size)?;
         match self.style {
             HeapStyle::Dynamic { bound_gv } => write!(f, ", bound {}", bound_gv)?,
             HeapStyle::Static { bound } => write!(f, ", bound {}", bound)?,
diff --git a/lib/codegen/src/ir/mod.rs b/lib/codegen/src/ir/mod.rs
index 1580896902..f93ed993b4 100644
--- a/lib/codegen/src/ir/mod.rs
+++ b/lib/codegen/src/ir/mod.rs
@@ -31,7 +31,7 @@ pub use ir::extfunc::{AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData,
 pub use ir::extname::ExternalName;
 pub use ir::function::Function;
 pub use ir::globalvalue::GlobalValueData;
-pub use ir::heap::{HeapBase, HeapData, HeapStyle};
+pub use ir::heap::{HeapData, HeapStyle};
 pub use ir::instructions::{InstructionData, Opcode, ValueList, ValueListPool, VariableArgs};
 pub use ir::jumptable::JumpTableData;
 pub use ir::layout::Layout;
diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs
index 122474c63e..e4a04a0bce 100644
--- a/lib/codegen/src/legalizer/heap.rs
+++ b/lib/codegen/src/legalizer/heap.rs
@@ -154,11 +154,7 @@ fn offset_addr(
     }
 
     // Add the heap base address base
-    match pos.func.heaps[heap].base {
-        ir::HeapBase::ReservedReg => unimplemented!(),
-        ir::HeapBase::GlobalValue(base_gv) => {
-            let base = pos.ins().global_value(addr_ty, base_gv);
-            pos.func.dfg.replace(inst).iadd(base, offset);
-        }
-    }
+    let base_gv = pos.func.heaps[heap].base;
+    let base = pos.ins().global_value(addr_ty, base_gv);
+    pos.func.dfg.replace(inst).iadd(base, offset);
 }
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 11cb57cef8..3c9cb4c56f 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -8,9 +8,8 @@ use cranelift_codegen::ir::instructions::{InstructionData, InstructionFormat, Va
 use cranelift_codegen::ir::types::VOID;
 use cranelift_codegen::ir::{
     AbiParam, ArgumentExtension, ArgumentLoc, Ebb, ExtFuncData, ExternalName, FuncRef, Function,
-    GlobalValue, GlobalValueData, Heap, HeapBase, HeapData, HeapStyle, JumpTable, JumpTableData,
-    MemFlags, Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind, Type, Value,
-    ValueLoc,
+    GlobalValue, GlobalValueData, Heap, HeapData, HeapStyle, JumpTable, JumpTableData, MemFlags,
+    Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind, Type, Value, ValueLoc,
 };
 use cranelift_codegen::isa::{self, Encoding, RegUnit, TargetIsa};
 use cranelift_codegen::packed_option::ReservedValue;
@@ -168,7 +167,7 @@ impl<'a> Context<'a> {
         self.map.def_heap(heap, loc)?;
         while self.function.heaps.next_key().index() <= heap.index() {
             self.function.create_heap(HeapData {
-                base: HeapBase::ReservedReg,
+                base: GlobalValue::reserved_value(),
                 min_size: Imm64::new(0),
                 guard_size: Imm64::new(0),
                 style: HeapStyle::Static {
@@ -1123,8 +1122,7 @@ impl<'a> Parser<'a> {
     // heap-decl ::= * Heap(heap) "=" heap-desc
     // heap-desc ::= heap-style heap-base { "," heap-attr }
     // heap-style ::= "static" | "dynamic"
-    // heap-base ::= "reserved_reg"
-    //             | GlobalValue(base)
+    // heap-base ::= GlobalValue(base)
     // heap-attr ::= "min" Imm64(bytes)
     //             | "max" Imm64(bytes)
     //             | "guard" Imm64(bytes)
@@ -1136,17 +1134,12 @@ impl<'a> Parser<'a> {
         let style_name = self.match_any_identifier("expected 'static' or 'dynamic'")?;
 
         // heap-desc ::= heap-style * heap-base { "," heap-attr }
-        // heap-base ::= * "reserved_reg"
-        //             | * GlobalValue(base)
+        // heap-base ::= * GlobalValue(base)
         let base = match self.token() {
-            Some(Token::Identifier("reserved_reg")) => HeapBase::ReservedReg,
-            Some(Token::GlobalValue(base_num)) => {
-                let base_gv = match GlobalValue::with_number(base_num) {
-                    Some(gv) => gv,
-                    None => return err!(self.loc, "invalid global value number for heap base"),
-                };
-                HeapBase::GlobalValue(base_gv)
-            }
+            Some(Token::GlobalValue(base_num)) => match GlobalValue::with_number(base_num) {
+                Some(gv) => gv,
+                None => return err!(self.loc, "invalid global value number for heap base"),
+            },
             _ => return err!(self.loc, "expected heap base"),
         };
         self.consume();
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index b2f58de97b..d13fd4367d 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -172,7 +172,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         let gv = func.create_global_value(ir::GlobalValueData::VMContext { offset: 0.into() });
 
         func.create_heap(ir::HeapData {
-            base: ir::HeapBase::GlobalValue(gv),
+            base: gv,
             min_size: 0.into(),
             guard_size: 0x8000_0000.into(),
             style: ir::HeapStyle::Static {

From 3a550d185ff35cb4971b87f0d50b5cc5898a83bb Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 27 Jul 2018 17:55:26 +0200
Subject: [PATCH 1948/3084] Enhance Verifier error reporting;

---
 cranelift/filetests/verifier/type_check.clif |  4 +-
 lib/codegen/src/print_errors.rs              | 52 +++++++++++---------
 lib/codegen/src/verifier/mod.rs              |  3 +-
 lib/filetests/src/runone.rs                  |  2 +-
 4 files changed, 34 insertions(+), 27 deletions(-)

diff --git a/cranelift/filetests/verifier/type_check.clif b/cranelift/filetests/verifier/type_check.clif
index 8783c4aced..1e7278e973 100644
--- a/cranelift/filetests/verifier/type_check.clif
+++ b/cranelift/filetests/verifier/type_check.clif
@@ -44,7 +44,7 @@ function %type_mismatch_controlling_variable() {
 function %fn_call_too_few_args() {
     fn2 = %great_fn(i32, f32)
     ebb0:
-        call fn2() ; error: mismatched argument count, got 0, expected 2
+        call fn2() ; error: mismatched argument count for `call fn2()`: got 0, expected 2
         return
 }
 
@@ -53,7 +53,7 @@ function %fn_call_too_many_args() {
     ebb0:
         v0 = iconst.i64 56
         v1 = f32const 0.0
-        call fn5(v0, v1) ; error: mismatched argument count, got 2, expected 0
+        call fn5(v0, v1) ; error: mismatched argument count for `call fn5(v0, v1)`: got 2, expected 0
         return
 }
 
diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 5783056c03..6fbe3c9af1 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -18,6 +18,16 @@ pub fn pretty_verifier_error(
     err: &VerifierError,
 ) -> String {
     let mut w = String::new();
+
+    match err.location {
+        ir::entities::AnyEntity::Inst(_) => {}
+        _ => {
+            // Print the error, because the pretty_function_error below won't do it since it isn't
+            // tied to an instruction.
+            writeln!(w, "verifier error summary: {}\n", err.to_string()).unwrap();
+        }
+    }
+
     decorate_function(
         &mut |w, func, isa, inst, indent| pretty_function_error(w, func, isa, inst, indent, err),
         &mut w,
@@ -37,31 +47,27 @@ fn pretty_function_error(
     err: &VerifierError,
 ) -> fmt::Result {
     match err.location {
-        ir::entities::AnyEntity::Inst(inst) => {
-            if inst == cur_inst {
-                writeln!(
-                    w,
-                    "{1:0$}{2}",
-                    indent,
-                    "",
-                    func.dfg.display_inst(cur_inst, isa)
-                )?;
-                write!(w, "{1:0$}^", indent, "")?;
-                for _c in cur_inst.to_string().chars() {
-                    write!(w, "~")?;
-                }
-                writeln!(w, "\n\nverifier {}\n", err.to_string())
-            } else {
-                write!(
-                    w,
-                    "{1:0$}{2}",
-                    indent,
-                    "",
-                    func.dfg.display_inst(cur_inst, isa)
-                )
+        ir::entities::AnyEntity::Inst(inst) if inst == cur_inst => {
+            writeln!(
+                w,
+                "{1:0$}{2}",
+                indent,
+                "",
+                func.dfg.display_inst(cur_inst, isa)
+            )?;
+            write!(w, "{1:0$}^", indent, "")?;
+            for _c in cur_inst.to_string().chars() {
+                write!(w, "~")?;
             }
+            writeln!(w, " verifier {}\n", err.to_string())
         }
-        _ => writeln!(w),
+        _ => writeln!(
+            w,
+            "{1:0$}{2}",
+            indent,
+            "",
+            func.dfg.display_inst(cur_inst, isa)
+        ),
     }
 }
 
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index f92bdda1d8..a25be6bcec 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -820,7 +820,8 @@ impl<'a> Verifier<'a> {
         if i != variable_args.len() {
             return err!(
                 inst,
-                "mismatched argument count, got {}, expected {}",
+                "mismatched argument count for `{}`: got {}, expected {}",
+                self.func.dfg.display_inst(inst, None),
                 variable_args.len(),
                 i
             );
diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs
index 4036e482ad..60cb057e60 100644
--- a/lib/filetests/src/runone.rs
+++ b/lib/filetests/src/runone.rs
@@ -135,5 +135,5 @@ fn run_one_test<'a>(
     }
 
     test.run(func, context)
-        .map_err(|e| format!("{}: {}", name, e))
+        .map_err(|e| format!("{}:\n{}", name, e))
 }

From c42bed7452cb48f12b244fa3d1cc95613ec44cbb Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 1 Aug 2018 05:05:33 -0700
Subject: [PATCH 1949/3084] Update paths for the meta => meta-python rename.

---
 cranelift/docs/Makefile               |  2 +-
 cranelift/docs/conf.py                |  4 ++--
 cranelift/docs/metaref.rst            | 10 +++++-----
 cranelift/docs/testing.rst            |  8 ++++----
 cranelift/test-all.sh                 |  4 ++--
 lib/codegen/src/constant_hash.rs      |  6 +++---
 lib/codegen/src/ir/builder.rs         |  2 +-
 lib/codegen/src/ir/instructions.rs    |  2 +-
 lib/codegen/src/ir/types.rs           |  4 ++--
 lib/codegen/src/isa/arm32/settings.rs |  4 ++--
 lib/codegen/src/isa/arm64/settings.rs |  4 ++--
 lib/codegen/src/isa/enc_tables.rs     |  6 +++---
 lib/codegen/src/isa/registers.rs      |  6 +++---
 lib/codegen/src/isa/riscv/settings.rs |  4 ++--
 lib/codegen/src/isa/x86/settings.rs   |  4 ++--
 lib/codegen/src/legalizer/mod.rs      |  2 +-
 lib/codegen/src/predicates.rs         |  2 +-
 lib/codegen/src/settings.rs           |  5 +++--
 18 files changed, 40 insertions(+), 39 deletions(-)

diff --git a/cranelift/docs/Makefile b/cranelift/docs/Makefile
index c66a99211d..082189bfa3 100644
--- a/cranelift/docs/Makefile
+++ b/cranelift/docs/Makefile
@@ -14,7 +14,7 @@ help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
 autohtml: html
-	$(SPHINXABUILD) -z ../lib/codegen/meta --ignore '.*' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	$(SPHINXABUILD) -z ../lib/codegen/meta-python --ignore '.*' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html
 
 .PHONY: help Makefile
 
diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py
index 2220d72141..2ab7d8a8c5 100644
--- a/cranelift/docs/conf.py
+++ b/cranelift/docs/conf.py
@@ -21,9 +21,9 @@ import os
 import sys
 sys.path.insert(0, os.path.abspath('.'))
 
-# Also add the meta directory to sys.path so autodoc can find the Cranelift meta
+# Also add the meta-python directory to sys.path so autodoc can find the Cranelift meta
 # language definitions.
-sys.path.insert(0, os.path.abspath('../lib/codegen/meta'))
+sys.path.insert(0, os.path.abspath('../lib/codegen/meta-python'))
 
 # -- General configuration ------------------------------------------------
 
diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/metaref.rst
index a1b0dd1f32..361dca9172 100644
--- a/cranelift/docs/metaref.rst
+++ b/cranelift/docs/metaref.rst
@@ -11,7 +11,7 @@ domain specific language embedded in Python. This document describes the Python
 modules that form the embedded DSL.
 
 The meta language descriptions are Python modules under the
-:file:`lib/codegen/meta` directory. The descriptions are processed in two
+:file:`lib/codegen/meta-python` directory. The descriptions are processed in two
 steps:
 
 1. The Python modules are imported. This has the effect of building static data
@@ -23,8 +23,8 @@ steps:
    constant tables.
 
 The main driver for this source code generation process is the
-:file:`lib/codegen/meta/build.py` script which is invoked as part of the build
-process if anything in the :file:`lib/codegen/meta` directory has changed
+:file:`lib/codegen/meta-python/build.py` script which is invoked as part of the build
+process if anything in the :file:`lib/codegen/meta-python` directory has changed
 since the last build.
 
 
@@ -38,7 +38,7 @@ of code generation. Each setting is defined in the meta language so a compact
 and consistent Rust representation can be generated. Shared settings are defined
 in the :mod:`base.settings` module. Some settings are specific to a target ISA,
 and defined in a :file:`settings.py` module under the appropriate
-:file:`lib/codegen/meta/isa/*` directory.
+:file:`lib/codegen/meta-python/isa/*` directory.
 
 Settings can take boolean on/off values, small numbers, or explicitly enumerated
 symbolic values. Each type is represented by a sub-class of :class:`Setting`:
@@ -433,7 +433,7 @@ architectures. Each ISA is represented by a :py:class:`cdsl.isa.TargetISA` insta
 .. autoclass:: TargetISA
 
 The definitions for each supported target live in a package under
-:file:`lib/codegen/meta/isa`.
+:file:`lib/codegen/meta-python/isa`.
 
 .. automodule:: isa
     :members:
diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst
index d1107b6f67..345940c772 100644
--- a/cranelift/docs/testing.rst
+++ b/cranelift/docs/testing.rst
@@ -1,6 +1,6 @@
-****************
+*****************
 Testing Cranelift
-****************
+*****************
 
 Cranelift is tested at multiple levels of abstraction and integration. When
 possible, Rust unit tests are used to verify single functions and types. When
@@ -109,7 +109,7 @@ header:
     isa_spec      : "isa" isa_name { `option` } "\n"
 
 The options given on the ``isa`` line modify the ISA-specific settings defined in
-:file:`lib/codegen/meta/isa/*/settings.py`.
+:file:`lib/codegen/meta-python/isa/*/settings.py`.
 
 All types of tests allow shared Cranelift settings to be modified:
 
@@ -119,7 +119,7 @@ All types of tests allow shared Cranelift settings to be modified:
     option        : flag | setting "=" value
 
 The shared settings available for all target ISAs are defined in
-:file:`lib/codegen/meta/base/settings.py`.
+:file:`lib/codegen/meta-python/base/settings.py`.
 
 The ``set`` lines apply settings cumulatively::
 
diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index 580c41d47b..fb17e35d82 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -43,13 +43,13 @@ fi
 # Check if any Python files have changed since we last checked them.
 tsfile="$topdir/target/meta-checked"
 if [ -f "$tsfile" ]; then
-    needcheck=$(find "$topdir/lib/codegen/meta" -name '*.py' -newer "$tsfile")
+    needcheck=$(find "$topdir/lib/codegen/meta-python" -name '*.py' -newer "$tsfile")
 else
     needcheck=yes
 fi
 if [ -n "$needcheck" ]; then
     banner "Checking python source files"
-    "$topdir/lib/codegen/meta/check.sh"
+    "$topdir/lib/codegen/meta-python/check.sh"
     touch "$tsfile" || echo no target directory
 fi
 
diff --git a/lib/codegen/src/constant_hash.rs b/lib/codegen/src/constant_hash.rs
index 2bfeaf58c3..553543ec60 100644
--- a/lib/codegen/src/constant_hash.rs
+++ b/lib/codegen/src/constant_hash.rs
@@ -1,6 +1,6 @@
 //! Runtime support for precomputed constant hash tables.
 //!
-//! The `lib/codegen/meta/constant_hash.py` Python module can generate constant hash tables using
+//! The `lib/codegen/meta-python/constant_hash.py` Python module can generate constant hash tables using
 //! open addressing and quadratic probing. The hash tables are arrays that are guaranteed to:
 //!
 //! - Have a power-of-two size.
@@ -56,7 +56,7 @@ pub fn probe + ?Sized>(
 }
 
 /// A primitive hash function for matching opcodes.
-/// Must match `lib/codegen/meta/constant_hash.py`.
+/// Must match `lib/codegen/meta-python/constant_hash.py`.
 pub fn simple_hash(s: &str) -> usize {
     let mut h: u32 = 5381;
     for c in s.chars() {
@@ -71,7 +71,7 @@ mod tests {
 
     #[test]
     fn basic() {
-        // c.f. `meta/constant_hash.py` tests.
+        // c.f. `meta-python/constant_hash.py` tests.
         assert_eq!(simple_hash("Hello"), 0x2fa70c01);
         assert_eq!(simple_hash("world"), 0x5b0c31d5);
     }
diff --git a/lib/codegen/src/ir/builder.rs b/lib/codegen/src/ir/builder.rs
index 48cbf03238..5cd2bfe36d 100644
--- a/lib/codegen/src/ir/builder.rs
+++ b/lib/codegen/src/ir/builder.rs
@@ -32,7 +32,7 @@ pub trait InstBuilderBase<'f>: Sized {
     fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'f mut DataFlowGraph);
 }
 
-// Include trait code generated by `lib/codegen/meta/gen_instr.py`.
+// Include trait code generated by `lib/codegen/meta-python/gen_instr.py`.
 //
 // This file defines the `InstBuilder` trait as an extension of `InstBuilderBase` with methods per
 // instruction format and per opcode.
diff --git a/lib/codegen/src/ir/instructions.rs b/lib/codegen/src/ir/instructions.rs
index c9abb67f3d..e709aa01c1 100644
--- a/lib/codegen/src/ir/instructions.rs
+++ b/lib/codegen/src/ir/instructions.rs
@@ -28,7 +28,7 @@ pub type ValueList = entity::EntityList;
 /// Memory pool for holding value lists. See `ValueList`.
 pub type ValueListPool = entity::ListPool;
 
-// Include code generated by `lib/codegen/meta/gen_instr.py`. This file contains:
+// Include code generated by `lib/codegen/meta-python/gen_instr.py`. This file contains:
 //
 // - The `pub enum InstructionFormat` enum with all the instruction formats.
 // - The `pub enum InstructionData` enum with all the instruction data fields.
diff --git a/lib/codegen/src/ir/types.rs b/lib/codegen/src/ir/types.rs
index edaee867da..195d90968a 100644
--- a/lib/codegen/src/ir/types.rs
+++ b/lib/codegen/src/ir/types.rs
@@ -24,13 +24,13 @@ pub struct Type(u8);
 /// a SIMD vector.
 pub const VOID: Type = Type(0);
 
-/// Start of the lane types. See also `meta/cdsl.types.py`.
+/// Start of the lane types. See also `meta-python/cdsl.types.py`.
 const LANE_BASE: u8 = 0x70;
 
 /// Start of the 2-lane vector types.
 const VECTOR_BASE: u8 = LANE_BASE + 16;
 
-// Include code generated by `lib/codegen/meta/gen_types.py`. This file contains constant
+// Include code generated by `lib/codegen/meta-python/gen_types.py`. This file contains constant
 // definitions for all the scalar types as well as common vector types for 64, 128, 256, and
 // 512-bit SIMD vectors.
 include!(concat!(env!("OUT_DIR"), "/types.rs"));
diff --git a/lib/codegen/src/isa/arm32/settings.rs b/lib/codegen/src/isa/arm32/settings.rs
index ed7dd01281..a5075597d9 100644
--- a/lib/codegen/src/isa/arm32/settings.rs
+++ b/lib/codegen/src/isa/arm32/settings.rs
@@ -3,7 +3,7 @@
 use settings::{self, detail, Builder};
 use std::fmt;
 
-// Include code generated by `lib/codegen/meta/gen_settings.py`. This file contains a public
+// Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
 // `Flags` struct with an impl for all of the settings defined in
-// `lib/codegen/meta/isa/arm32/settings.py`.
+// `lib/codegen/meta-python/isa/arm32/settings.py`.
 include!(concat!(env!("OUT_DIR"), "/settings-arm32.rs"));
diff --git a/lib/codegen/src/isa/arm64/settings.rs b/lib/codegen/src/isa/arm64/settings.rs
index cdaa129b2d..6d51aed6ad 100644
--- a/lib/codegen/src/isa/arm64/settings.rs
+++ b/lib/codegen/src/isa/arm64/settings.rs
@@ -3,7 +3,7 @@
 use settings::{self, detail, Builder};
 use std::fmt;
 
-// Include code generated by `lib/codegen/meta/gen_settings.py`. This file contains a public
+// Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
 // `Flags` struct with an impl for all of the settings defined in
-// `lib/codegen/meta/isa/arm64/settings.py`.
+// `lib/codegen/meta-python/isa/arm64/settings.py`.
 include!(concat!(env!("OUT_DIR"), "/settings-arm64.rs"));
diff --git a/lib/codegen/src/isa/enc_tables.rs b/lib/codegen/src/isa/enc_tables.rs
index b3deb85de1..0a98ace71a 100644
--- a/lib/codegen/src/isa/enc_tables.rs
+++ b/lib/codegen/src/isa/enc_tables.rs
@@ -1,7 +1,7 @@
 //! Support types for generated encoding tables.
 //!
 //! This module contains types and functions for working with the encoding tables generated by
-//! `lib/codegen/meta/gen_encoding.py`.
+//! `lib/codegen/meta-python/gen_encoding.py`.
 
 use constant_hash::{probe, Table};
 use ir::{Function, InstructionData, Opcode, Type};
@@ -164,10 +164,10 @@ where
 /// Encoding lists are represented as sequences of u16 words.
 pub type EncListEntry = u16;
 
-/// Number of bits used to represent a predicate. c.f. `meta/gen_encoding.py`.
+/// Number of bits used to represent a predicate. c.f. `meta-python/gen_encoding.py`.
 const PRED_BITS: u8 = 12;
 const PRED_MASK: usize = (1 << PRED_BITS) - 1;
-/// First code word representing a predicate check. c.f. `meta/gen_encoding.py`.
+/// First code word representing a predicate check. c.f. `meta-python/gen_encoding.py`.
 const PRED_START: usize = 0x1000;
 
 /// An iterator over legal encodings for the instruction.
diff --git a/lib/codegen/src/isa/registers.rs b/lib/codegen/src/isa/registers.rs
index a78dfac60c..9296b70af5 100644
--- a/lib/codegen/src/isa/registers.rs
+++ b/lib/codegen/src/isa/registers.rs
@@ -17,19 +17,19 @@ pub type RegUnit = u16;
 /// The size of this type is determined by the target ISA that has the most register units defined.
 /// Currently that is arm32 which has 64+16 units.
 ///
-/// This type should be coordinated with meta/cdsl/registers.py.
+/// This type should be coordinated with meta-python/cdsl/registers.py.
 pub type RegUnitMask = [u32; 3];
 
 /// A bit mask indexed by register classes.
 ///
 /// The size of this type is determined by the ISA with the most register classes.
 ///
-/// This type should be coordinated with meta/cdsl/isa.py.
+/// This type should be coordinated with meta-python/cdsl/isa.py.
 pub type RegClassMask = u32;
 
 /// Guaranteed maximum number of top-level register classes with pressure tracking in any ISA.
 ///
-/// This can be increased, but should be coordinated with meta/cdsl/isa.py.
+/// This can be increased, but should be coordinated with meta-python/cdsl/isa.py.
 pub const MAX_TRACKED_TOPRCS: usize = 4;
 
 /// The register units in a target ISA are divided into disjoint register banks. Each bank covers a
diff --git a/lib/codegen/src/isa/riscv/settings.rs b/lib/codegen/src/isa/riscv/settings.rs
index a5bdd6b850..972f5c6b5f 100644
--- a/lib/codegen/src/isa/riscv/settings.rs
+++ b/lib/codegen/src/isa/riscv/settings.rs
@@ -3,9 +3,9 @@
 use settings::{self, detail, Builder};
 use std::fmt;
 
-// Include code generated by `lib/codegen/meta/gen_settings.py`. This file contains a public
+// Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
 // `Flags` struct with an impl for all of the settings defined in
-// `lib/codegen/meta/isa/riscv/settings.py`.
+// `lib/codegen/meta-python/isa/riscv/settings.py`.
 include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs"));
 
 #[cfg(test)]
diff --git a/lib/codegen/src/isa/x86/settings.rs b/lib/codegen/src/isa/x86/settings.rs
index 84d0e96f06..66699b1289 100644
--- a/lib/codegen/src/isa/x86/settings.rs
+++ b/lib/codegen/src/isa/x86/settings.rs
@@ -3,9 +3,9 @@
 use settings::{self, detail, Builder};
 use std::fmt;
 
-// Include code generated by `lib/codegen/meta/gen_settings.py`. This file contains a public
+// Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
 // `Flags` struct with an impl for all of the settings defined in
-// `lib/codegen/meta/isa/x86/settings.py`.
+// `lib/codegen/meta-python/isa/x86/settings.py`.
 include!(concat!(env!("OUT_DIR"), "/settings-x86.rs"));
 
 #[cfg(test)]
diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs
index f6ca7efeec..f11a652828 100644
--- a/lib/codegen/src/legalizer/mod.rs
+++ b/lib/codegen/src/legalizer/mod.rs
@@ -110,7 +110,7 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
 }
 
 // Include legalization patterns that were generated by `gen_legalizer.py` from the `XForms` in
-// `lib/codegen/meta/base/legalize.py`.
+// `lib/codegen/meta-python/base/legalize.py`.
 //
 // Concretely, this defines private functions `narrow()`, and `expand()`.
 include!(concat!(env!("OUT_DIR"), "/legalizer.rs"));
diff --git a/lib/codegen/src/predicates.rs b/lib/codegen/src/predicates.rs
index 96cf0f38c6..58ae0a40fb 100644
--- a/lib/codegen/src/predicates.rs
+++ b/lib/codegen/src/predicates.rs
@@ -1,7 +1,7 @@
 //! Predicate functions for testing instruction fields.
 //!
 //! This module defines functions that are used by the instruction predicates defined by
-//! `lib/codegen/meta/cdsl/predicates.py` classes.
+//! `lib/codegen/meta-python/cdsl/predicates.py` classes.
 //!
 //! The predicates the operate on integer fields use `Into` as a shared trait bound. This
 //! bound is implemented by all the native integer types as well as `Imm64`.
diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs
index aaa11e3d8c..da0ac133aa 100644
--- a/lib/codegen/src/settings.rs
+++ b/lib/codegen/src/settings.rs
@@ -310,8 +310,9 @@ pub mod detail {
     }
 }
 
-// Include code generated by `meta/gen_settings.py`. This file contains a public `Flags` struct
-// with an impl for all of the settings defined in `lib/codegen/meta/base/settings.py`.
+// Include code generated by `meta-python/gen_settings.py`. This file contains a public `Flags`
+// struct with an impl for all of the settings defined in
+// `lib/codegen/meta-python/base/settings.py`.
 include!(concat!(env!("OUT_DIR"), "/settings.rs"));
 
 /// Wrapper containing flags and optionally a `TargetIsa` trait object.

From 01729be8d737cc2c8f4db430d188d250574fed4e Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 1 Aug 2018 20:21:05 +0200
Subject: [PATCH 1950/3084] Add comment support (#379)

* Add comment support

* Don't print empty comments

* Add nop instruction

* Add test and note

* Add FuncWriter trait

* Remove comment support

* Add write_preamble to FuncWriter

* Fix test

* Some changes
---
 cranelift/filetests/isa/x86/nop.cton         |  10 ++
 cranelift/src/wasm.rs                        |   2 +-
 lib/codegen/meta-python/base/instructions.py |   7 ++
 lib/codegen/src/lib.rs                       |   2 +-
 lib/codegen/src/print_errors.rs              |  44 +++++---
 lib/codegen/src/write.rs                     | 106 +++++++++++++------
 lib/filetests/src/runone.rs                  |   2 +-
 lib/wasm/tests/wasm_testsuite.rs             |   2 +-
 8 files changed, 124 insertions(+), 51 deletions(-)
 create mode 100644 cranelift/filetests/isa/x86/nop.cton

diff --git a/cranelift/filetests/isa/x86/nop.cton b/cranelift/filetests/isa/x86/nop.cton
new file mode 100644
index 0000000000..2863185e41
--- /dev/null
+++ b/cranelift/filetests/isa/x86/nop.cton
@@ -0,0 +1,10 @@
+test compile
+
+target x86_64
+
+function %test(i32) -> i32 system_v {
+ebb0(v0: i32):
+    nop
+    v1 = iconst.i32 42
+    return v1
+}
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 057c167fbc..72fcb0d729 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -149,7 +149,7 @@ fn handle_module(
         if flag_check_translation {
             context
                 .verify(fisa)
-                .map_err(|err| pretty_verifier_error(&context.func, fisa.isa, &err))?;
+                .map_err(|err| pretty_verifier_error(&context.func, fisa.isa, None, &err))?;
         } else {
             let compiled_size = context
                 .compile(isa)
diff --git a/lib/codegen/meta-python/base/instructions.py b/lib/codegen/meta-python/base/instructions.py
index 147c93631b..f2b9782ec3 100644
--- a/lib/codegen/meta-python/base/instructions.py
+++ b/lib/codegen/meta-python/base/instructions.py
@@ -581,6 +581,13 @@ bconst = Instruction(
 # Generics.
 #
 
+nop = Instruction(
+        'nop', r"""
+        Just a dummy instruction
+
+        Note: this doesn't compile to a machine code nop
+        """)
+
 c = Operand('c', Testable, doc='Controlling value to test')
 x = Operand('x', Any, doc='Value to use when `c` is true')
 y = Operand('y', Any, doc='Value to use when `c` is false')
diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index b58a6b0e17..56320c645f 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -74,6 +74,7 @@ pub mod print_errors;
 pub mod settings;
 pub mod timing;
 pub mod verifier;
+pub mod write;
 
 pub use entity::packed_option;
 
@@ -100,7 +101,6 @@ mod simple_gvn;
 mod stack_layout;
 mod topo_order;
 mod unreachable_code;
-mod write;
 
 pub use result::{CodegenError, CodegenResult};
 
diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 6fbe3c9af1..15c6fe7dda 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -3,18 +3,19 @@
 use ir;
 use ir::entities::Inst;
 use ir::function::Function;
-use isa::TargetIsa;
+use isa::{RegInfo, TargetIsa};
 use result::CodegenError;
 use std::fmt;
 use std::fmt::Write;
 use std::string::{String, ToString};
 use verifier::VerifierError;
-use write::decorate_function;
+use write::{decorate_function, FuncWriter, PlainWriter};
 
 /// Pretty-print a verifier error.
-pub fn pretty_verifier_error(
+pub fn pretty_verifier_error<'a>(
     func: &ir::Function,
     isa: Option<&TargetIsa>,
+    func_w: Option>,
     err: &VerifierError,
 ) -> String {
     let mut w = String::new();
@@ -29,7 +30,7 @@ pub fn pretty_verifier_error(
     }
 
     decorate_function(
-        &mut |w, func, isa, inst, indent| pretty_function_error(w, func, isa, inst, indent, err),
+        &mut PrettyVerifierError(func_w.unwrap_or(Box::new(PlainWriter)), err),
         &mut w,
         func,
         isa,
@@ -37,6 +38,30 @@ pub fn pretty_verifier_error(
     w
 }
 
+struct PrettyVerifierError<'a>(Box, &'a VerifierError);
+
+impl<'a> FuncWriter for PrettyVerifierError<'a> {
+    fn write_instruction(
+        &mut self,
+        w: &mut Write,
+        func: &Function,
+        isa: Option<&TargetIsa>,
+        inst: Inst,
+        indent: usize,
+    ) -> fmt::Result {
+        pretty_function_error(w, func, isa, inst, indent, &mut *self.0, self.1)
+    }
+
+    fn write_preamble(
+        &mut self,
+        w: &mut Write,
+        func: &Function,
+        regs: Option<&RegInfo>,
+    ) -> Result {
+        self.0.write_preamble(w, func, regs)
+    }
+}
+
 /// Pretty-print a function verifier error.
 fn pretty_function_error(
     w: &mut Write,
@@ -44,17 +69,12 @@ fn pretty_function_error(
     isa: Option<&TargetIsa>,
     cur_inst: Inst,
     indent: usize,
+    func_w: &mut FuncWriter,
     err: &VerifierError,
 ) -> fmt::Result {
     match err.location {
         ir::entities::AnyEntity::Inst(inst) if inst == cur_inst => {
-            writeln!(
-                w,
-                "{1:0$}{2}",
-                indent,
-                "",
-                func.dfg.display_inst(cur_inst, isa)
-            )?;
+            func_w.write_instruction(w, func, isa, cur_inst, indent)?;
             write!(w, "{1:0$}^", indent, "")?;
             for _c in cur_inst.to_string().chars() {
                 write!(w, "~")?;
@@ -74,7 +94,7 @@ fn pretty_function_error(
 /// Pretty-print a Cranelift error.
 pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CodegenError) -> String {
     if let CodegenError::Verifier(e) = err {
-        pretty_verifier_error(func, isa, &e)
+        pretty_verifier_error(func, isa, None, &e)
     } else {
         err.to_string()
     }
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index 6423664f95..38367adb1b 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -9,35 +9,63 @@ use packed_option::ReservedValue;
 use std::fmt::{self, Write};
 use std::string::String;
 
-fn write_function_plain(
-    w: &mut Write,
-    func: &Function,
-    isa: Option<&TargetIsa>,
-    inst: Inst,
-    indent: usize,
-) -> fmt::Result {
-    write_instruction(w, func, isa, inst, indent)?;
-    Ok(())
+/// A `FuncWriter` is used to decorate functions during printing
+pub trait FuncWriter {
+    /// Write the given inst to w
+    fn write_instruction(
+        &mut self,
+        w: &mut Write,
+        func: &Function,
+        isa: Option<&TargetIsa>,
+        inst: Inst,
+        ident: usize,
+    ) -> fmt::Result;
+
+    /// Write the preamble to w
+    fn write_preamble(
+        &mut self,
+        w: &mut Write,
+        func: &Function,
+        regs: Option<&RegInfo>,
+    ) -> Result;
+}
+
+/// A `PlainWriter` doesn't decorate the function
+pub struct PlainWriter;
+
+impl FuncWriter for PlainWriter {
+    fn write_instruction(
+        &mut self,
+        w: &mut Write,
+        func: &Function,
+        isa: Option<&TargetIsa>,
+        inst: Inst,
+        indent: usize,
+    ) -> fmt::Result {
+        write_instruction(w, func, isa, inst, indent)
+    }
+
+    fn write_preamble(
+        &mut self,
+        w: &mut Write,
+        func: &Function,
+        regs: Option<&RegInfo>,
+    ) -> Result {
+        write_preamble(w, func, regs)
+    }
 }
 
 /// Write `func` to `w` as equivalent text.
 /// Use `isa` to emit ISA-dependent annotations.
 pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) -> fmt::Result {
-    decorate_function(
-        &mut |w, func, isa, inst, indent| write_function_plain(w, func, isa, inst, indent),
-        w,
-        func,
-        isa,
-    )
+    decorate_function(&mut PlainWriter, w, func, isa)
 }
 
 /// Writes 'func' to 'w' as text.
 /// write_function_plain is passed as 'closure' to print instructions as text.
 /// pretty_function_error is passed as 'closure' to add error decoration.
-pub fn decorate_function<
-    WL: FnMut(&mut Write, &Function, Option<&TargetIsa>, Inst, usize) -> fmt::Result,
->(
-    closure: &mut WL,
+pub fn decorate_function(
+    func_w: &mut FW,
     w: &mut Write,
     func: &Function,
     isa: Option<&TargetIsa>,
@@ -48,12 +76,12 @@ pub fn decorate_function<
     write!(w, "function ")?;
     write_spec(w, func, regs)?;
     writeln!(w, " {{")?;
-    let mut any = write_preamble(w, func, regs)?;
+    let mut any = func_w.write_preamble(w, func, regs)?;
     for ebb in &func.layout {
         if any {
             writeln!(w)?;
         }
-        decorate_ebb(closure, w, func, isa, ebb)?;
+        decorate_ebb(func_w, w, func, isa, ebb)?;
         any = true;
     }
     writeln!(w, "}}")
@@ -120,7 +148,7 @@ fn write_preamble(
 //
 // Basic blocks
 
-pub fn write_arg(
+fn write_arg(
     w: &mut Write,
     func: &Function,
     regs: Option<&RegInfo>,
@@ -135,6 +163,12 @@ pub fn write_arg(
     Ok(())
 }
 
+/// Write out the basic block header, outdented:
+///
+///    ebb1:
+///    ebb1(v1: i32):
+///    ebb10(v4: f64, v5: b1):
+///
 pub fn write_ebb_header(
     w: &mut Write,
     func: &Function,
@@ -142,13 +176,6 @@ pub fn write_ebb_header(
     ebb: Ebb,
     indent: usize,
 ) -> fmt::Result {
-    // Write out the basic block header, outdented:
-    //
-    //    ebb1:
-    //    ebb1(v1: i32):
-    //    ebb10(v4: f64, v5: b1):
-    //
-
     // The `indent` is the instruction indentation. EBB headers are 4 spaces out from that.
     write!(w, "{1:0$}{2}", indent - 4, "", ebb)?;
 
@@ -171,10 +198,8 @@ pub fn write_ebb_header(
     writeln!(w, "):")
 }
 
-pub fn decorate_ebb<
-    WL: FnMut(&mut Write, &Function, Option<&TargetIsa>, Inst, usize) -> fmt::Result,
->(
-    closure: &mut WL,
+fn decorate_ebb(
+    func_w: &mut FW,
     w: &mut Write,
     func: &Function,
     isa: Option<&TargetIsa>,
@@ -189,7 +214,7 @@ pub fn decorate_ebb<
 
     write_ebb_header(w, func, isa, ebb, indent)?;
     for inst in func.layout.ebb_insts(ebb) {
-        closure(w, func, isa, inst, indent)?;
+        func_w.write_instruction(w, func, isa, inst, indent)?;
     }
 
     Ok(())
@@ -556,8 +581,9 @@ impl<'a> fmt::Display for DisplayValuesWithDelimiter<'a> {
 
 #[cfg(test)]
 mod tests {
+    use cursor::{Cursor, CursorPosition, FuncCursor};
     use ir::types;
-    use ir::{ExternalName, Function, StackSlotData, StackSlotKind};
+    use ir::{ExternalName, Function, InstBuilder, StackSlotData, StackSlotKind};
     use std::string::ToString;
 
     #[test]
@@ -592,5 +618,15 @@ mod tests {
             f.to_string(),
             "function %foo() fast {\n    ss0 = explicit_slot 4\n\nebb0(v0: i8, v1: f32x4):\n}\n"
         );
+
+        {
+            let mut cursor = FuncCursor::new(&mut f);
+            cursor.set_position(CursorPosition::After(ebb));
+            cursor.ins().return_(&[])
+        };
+        assert_eq!(
+            f.to_string(),
+            "function %foo() fast {\n    ss0 = explicit_slot 4\n\nebb0(v0: i8, v1: f32x4):\n    return\n}\n"
+        );
     }
 }
diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs
index 60cb057e60..8a28755003 100644
--- a/lib/filetests/src/runone.rs
+++ b/lib/filetests/src/runone.rs
@@ -130,7 +130,7 @@ fn run_one_test<'a>(
     // Should we run the verifier before this test?
     if !context.verified && test.needs_verifier() {
         verify_function(&func, context.flags_or_isa())
-            .map_err(|e| pretty_verifier_error(&func, isa, &e))?;
+            .map_err(|e| pretty_verifier_error(&func, isa, None, &e))?;
         context.verified = true;
     }
 
diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs
index d72ba4afa3..1924a17d07 100644
--- a/lib/wasm/tests/wasm_testsuite.rs
+++ b/lib/wasm/tests/wasm_testsuite.rs
@@ -77,7 +77,7 @@ fn handle_module(path: &Path, flags: &Flags) {
     translate_module(&data, &mut dummy_environ).unwrap();
     for func in &dummy_environ.info.function_bodies {
         verifier::verify_function(func, flags)
-            .map_err(|err| panic!(pretty_verifier_error(func, None, &err)))
+            .map_err(|err| panic!(pretty_verifier_error(func, None, None, &err)))
             .unwrap();
     }
 }

From f89cb1df64bb9921e495e530aae4a5d48077af53 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 1 Aug 2018 12:13:45 -0700
Subject: [PATCH 1951/3084] Fix the test script to return an error if
 formatting diffs are found.

---
 cranelift/test-all.sh    | 3 +--
 lib/codegen/src/write.rs | 7 +------
 2 files changed, 2 insertions(+), 8 deletions(-)

diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index fb17e35d82..f741c54eb4 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -27,8 +27,7 @@ function banner {
 # Run rustfmt if we have it.
 banner "Rust formatting"
 if type rustfmt > /dev/null; then
-    # In newer versions of rustfmt, replace --write-mode=diff with --check.
-    if ! "$topdir/format-all.sh" --write-mode=diff ; then
+    if ! "$topdir/format-all.sh" --write-mode=check ; then
         echo "Formatting diffs detected! Run \"cargo fmt --all\" to correct."
         exit 1
     fi
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index 38367adb1b..2f0f6ba25f 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -148,12 +148,7 @@ fn write_preamble(
 //
 // Basic blocks
 
-fn write_arg(
-    w: &mut Write,
-    func: &Function,
-    regs: Option<&RegInfo>,
-    arg: Value,
-) -> fmt::Result {
+fn write_arg(w: &mut Write, func: &Function, regs: Option<&RegInfo>, arg: Value) -> fmt::Result {
     write!(w, "{}: {}", arg, func.dfg.value_type(arg))?;
     let loc = func.locations[arg];
     if loc.is_assigned() {

From 63e87895eb8cbd2b3b925b718d51ca1c10bd8f67 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 1 Aug 2018 12:16:26 -0700
Subject: [PATCH 1952/3084] Remove an unneeded command-line option.

---
 cranelift/test-all.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index f741c54eb4..35d68920d8 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -71,7 +71,7 @@ cargo doc
 # Run clippy if we have it.
 banner "Rust linter"
 if "$topdir/check-clippy.sh"; then
-    "$topdir/clippy-all.sh" --write-mode=diff
+    "$topdir/clippy-all.sh"
 else
     echo "\`cargo +nightly install clippy\` for optional rust linting"
 fi

From c49c20713c422c7936626195c1d9ff3a725c3704 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 1 Aug 2018 13:33:54 -0700
Subject: [PATCH 1953/3084] Check for duplicate jump tables before performing
 any IR mutations.

This isn't fixing an actual bug, but it does make `add_jt` more
consistent with the other `add_*` functions.
---
 lib/reader/src/parser.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 3c9cb4c56f..bef37a86fd 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -233,11 +233,12 @@ impl<'a> Context<'a> {
 
     // Allocate a new jump table.
     fn add_jt(&mut self, jt: JumpTable, data: JumpTableData, loc: Location) -> ParseResult<()> {
+        self.map.def_jt(jt, loc)?;
         while self.function.jump_tables.next_key().index() <= jt.index() {
             self.function.create_jump_table(JumpTableData::new());
         }
         self.function.jump_tables[jt] = data;
-        self.map.def_jt(jt, loc)
+        Ok(())
     }
 
     // Resolve a reference to a jump table.

From b8dbbce99d13bdc108e7bbd4de1f9eff59517ee5 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 1 Aug 2018 13:36:58 -0700
Subject: [PATCH 1954/3084] Update cranelift-serde's license field.

This updates it to "Apache-2.0 WITH LLVM-exception".
---
 lib/serde/Cargo.toml |   2 +-
 lib/serde/LICENSE    | 219 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 220 insertions(+), 1 deletion(-)
 create mode 100644 lib/serde/LICENSE

diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index 248276c93c..270ab0be4d 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -4,7 +4,7 @@ version = "0.17.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
-license = "Apache-2.0"
+license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 keywords = ["webassembly", "serde"]
 
diff --git a/lib/serde/LICENSE b/lib/serde/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/serde/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.

From c8350f4e8ff167cee121a9eea6ec854245221b93 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 1 Aug 2018 14:06:16 -0700
Subject: [PATCH 1955/3084] Update to raw-cpuid 5.0.0.

---
 lib/native/Cargo.toml | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index e9ce77ed4d..20c3909d7a 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -12,8 +12,7 @@ cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
-# Due to https://github.com/gz/rust-cpuid/issues/27, stay at 3.x for now.
-raw-cpuid = "3.1.0"
+raw-cpuid = "5.0.0"
 
 [features]
 default = ["std"]

From bc8e5f82c890349131b27b4b3f1a24911e78d4b9 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 1 Aug 2018 14:10:54 -0700
Subject: [PATCH 1956/3084] Add use declarations needed for no_std mode.

---
 lib/codegen/src/print_errors.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 15c6fe7dda..f85f5ccfc4 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -5,6 +5,7 @@ use ir::entities::Inst;
 use ir::function::Function;
 use isa::{RegInfo, TargetIsa};
 use result::CodegenError;
+use std::boxed::Box;
 use std::fmt;
 use std::fmt::Write;
 use std::string::{String, ToString};

From a8ded3a6f18036b2db198f9e17d9724d5a3ca97d Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 1 Aug 2018 14:12:46 -0700
Subject: [PATCH 1957/3084] Remove unneeded dependencies.

---
 lib/serde/Cargo.toml | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)

diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index 270ab0be4d..91bf4b0a65 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -13,25 +13,17 @@ name = "clif-json"
 path = "src/clif-json.rs"
 
 [dependencies]
-cfg-if = "0.1"
-filecheck = "0.3.0"
 docopt = "1"
 serde = "1.0.8"
 serde_derive = "1.0.8"
 serde_json = "1.0"
-term = "0.5.1"
-target-lexicon = "0.0.2"
-wasmparser = { version = "0.17.0", default-features = false }
 cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
 cranelift-reader = { path = "../reader", version = "0.17.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.17.0", default-features = false }
-failure = { version = "0.1.1", default-features = false, features = ["derive"] }
-failure_derive = { version = "0.1.1", default-features = false }
 
 [features]
 default = ["std"]
-std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std", "target-lexicon/std"]
-core = ["cranelift-codegen/core", "cranelift-frontend/core", "wasmparser/core"]
+std = ["cranelift-codegen/std"]
+core = ["cranelift-codegen/core"]
 
 [badges]
 maintenance = { status = "experimental" }

From 5389b7784e090b1c4a0969d4922ed753d3b6444c Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sun, 29 Jul 2018 17:58:53 +0200
Subject: [PATCH 1958/3084] Check signature compatibility in declare_function
 (fixes #427)

---
 cranelift/tests/moduletests.rs | 22 ++++++++++++++++++++++
 lib/module/src/module.rs       |  8 ++++++--
 2 files changed, 28 insertions(+), 2 deletions(-)
 create mode 100644 cranelift/tests/moduletests.rs

diff --git a/cranelift/tests/moduletests.rs b/cranelift/tests/moduletests.rs
new file mode 100644
index 0000000000..d829aa8e9e
--- /dev/null
+++ b/cranelift/tests/moduletests.rs
@@ -0,0 +1,22 @@
+extern crate cranelift_codegen;
+extern crate cranelift_module;
+extern crate cranelift_simplejit;
+
+use cranelift_codegen::settings::*;
+use cranelift_codegen::ir::*;
+use cranelift_module::*;
+use cranelift_simplejit::*;
+
+#[test]
+fn error_on_incompatible_sig_in_declare_function() {
+    let mut module: Module = Module::new(SimpleJITBuilder::new());
+    let mut sig = Signature {
+        params: vec![AbiParam::new(types::I64)],
+        returns: vec![],
+        call_conv: CallConv::SystemV,
+        argument_bytes: None,
+    };
+    module.declare_function("abc", Linkage::Local, &sig).unwrap();
+    sig.params[0] = AbiParam::new(types::I32);
+    module.declare_function("abc", Linkage::Local, &sig).err().unwrap(); // Make sure this is an error
+}
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index e5b7c81883..10008caa51 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -159,8 +159,12 @@ impl ModuleFunction
 where
     B: Backend,
 {
-    fn merge(&mut self, linkage: Linkage) {
+    fn merge(&mut self, linkage: Linkage, sig: &ir::Signature) -> Result<(), ModuleError> {
         self.decl.linkage = Linkage::merge(self.decl.linkage, linkage);
+        if &self.decl.signature != sig {
+            return Err(ModuleError::IncompatibleDeclaration(self.decl.name.clone()));
+        }
+        Ok(())
     }
 }
 
@@ -357,7 +361,7 @@ where
             Occupied(entry) => match *entry.get() {
                 FuncOrDataId::Func(id) => {
                     let existing = &mut self.contents.functions[id];
-                    existing.merge(linkage);
+                    existing.merge(linkage, signature)?;
                     self.backend.declare_function(name, existing.decl.linkage);
                     Ok(id)
                 }

From 987bbfa922b5fb58314b9e93d96e60c560617d2c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 1 Aug 2018 15:14:49 -0700
Subject: [PATCH 1959/3084] Fix rustfmt errors.

---
 cranelift/tests/moduletests.rs | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/cranelift/tests/moduletests.rs b/cranelift/tests/moduletests.rs
index d829aa8e9e..89a04e2cd5 100644
--- a/cranelift/tests/moduletests.rs
+++ b/cranelift/tests/moduletests.rs
@@ -2,8 +2,8 @@ extern crate cranelift_codegen;
 extern crate cranelift_module;
 extern crate cranelift_simplejit;
 
-use cranelift_codegen::settings::*;
 use cranelift_codegen::ir::*;
+use cranelift_codegen::settings::*;
 use cranelift_module::*;
 use cranelift_simplejit::*;
 
@@ -16,7 +16,12 @@ fn error_on_incompatible_sig_in_declare_function() {
         call_conv: CallConv::SystemV,
         argument_bytes: None,
     };
-    module.declare_function("abc", Linkage::Local, &sig).unwrap();
+    module
+        .declare_function("abc", Linkage::Local, &sig)
+        .unwrap();
     sig.params[0] = AbiParam::new(types::I32);
-    module.declare_function("abc", Linkage::Local, &sig).err().unwrap(); // Make sure this is an error
+    module
+        .declare_function("abc", Linkage::Local, &sig)
+        .err()
+        .unwrap(); // Make sure this is an error
 }

From c61f8a5bafa1c914d381e54586accb763abb3ed3 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 2 Aug 2018 09:18:59 -0700
Subject: [PATCH 1960/3084] Rename langref.rst to ir.rst.

This aligns with our use of the term "IR" rather than "IL" or
other terms involving "language".
---
 cranelift/README.md                      | 2 +-
 cranelift/docs/compare-llvm.rst          | 2 +-
 cranelift/docs/index.rst                 | 4 ++--
 cranelift/docs/{langref.rst => ir.rst}   | 6 +++---
 cranelift/docs/{metaref.rst => meta.rst} | 2 +-
 cranelift/docs/testing.rst               | 2 +-
 lib/codegen/src/result.rs                | 2 +-
 lib/wasm/src/environ/spec.rs             | 2 +-
 8 files changed, 11 insertions(+), 11 deletions(-)
 rename cranelift/docs/{langref.rst => ir.rst} (99%)
 rename cranelift/docs/{metaref.rst => meta.rst} (99%)

diff --git a/cranelift/README.md b/cranelift/README.md
index c943d760be..b4c42e849b 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -3,7 +3,7 @@ Cranelift Code Generator
 
 Cranelift is a low-level retargetable code generator. It translates a
 [target-independent intermediate
-representation](https://cranelift.readthedocs.io/en/latest/langref.html)
+representation](https://cranelift.readthedocs.io/en/latest/ir.html)
 into executable machine code.
 
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst
index 7f3c320ace..6ca562bcd3 100644
--- a/cranelift/docs/compare-llvm.rst
+++ b/cranelift/docs/compare-llvm.rst
@@ -89,7 +89,7 @@ representation. Some target ISAs have a fast instruction selector that can
 translate simple code directly to MachineInstrs, bypassing SelectionDAG when
 possible.
 
-:doc:`Cranelift ` uses a single intermediate representation to cover
+:doc:`Cranelift ` uses a single intermediate representation to cover
 these levels of abstraction. This is possible in part because of Cranelift's
 smaller scope.
 
diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst
index 08b27162c3..f37c83e34a 100644
--- a/cranelift/docs/index.rst
+++ b/cranelift/docs/index.rst
@@ -6,8 +6,8 @@ Contents:
 .. toctree::
    :maxdepth: 1
 
-   langref
-   metaref
+   ir
+   meta
    testing
    regalloc
    compare-llvm
diff --git a/cranelift/docs/langref.rst b/cranelift/docs/ir.rst
similarity index 99%
rename from cranelift/docs/langref.rst
rename to cranelift/docs/ir.rst
index 52be70c7aa..777693ea13 100644
--- a/cranelift/docs/langref.rst
+++ b/cranelift/docs/ir.rst
@@ -1,6 +1,6 @@
-****************************
-Cranelift Language Reference
-****************************
+**********************
+Cranelift IR Reference
+**********************
 
 .. default-domain:: clif
 .. highlight:: clif
diff --git a/cranelift/docs/metaref.rst b/cranelift/docs/meta.rst
similarity index 99%
rename from cranelift/docs/metaref.rst
rename to cranelift/docs/meta.rst
index 361dca9172..f805e08875 100644
--- a/cranelift/docs/metaref.rst
+++ b/cranelift/docs/meta.rst
@@ -80,7 +80,7 @@ open :class:`InstructionGroup`.
 .. autoclass:: InstructionGroup
     :members:
 
-The basic Cranelift instruction set described in :doc:`langref` is defined by the
+The basic Cranelift instruction set described in :doc:`ir` is defined by the
 Python module :mod:`base.instructions`. This module has a global value
 :data:`base.instructions.GROUP` which is an :class:`InstructionGroup` instance
 containing all the base instructions.
diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst
index 345940c772..747ba53d54 100644
--- a/cranelift/docs/testing.rst
+++ b/cranelift/docs/testing.rst
@@ -90,7 +90,7 @@ easier to provide substantial input functions for the compiler tests.
 File tests are :file:`*.clif` files in the :file:`filetests/` directory
 hierarchy. Each file has a header describing what to test followed by a number
 of input functions in the :doc:`Cranelift textual intermediate representation
-`:
+`:
 
 .. productionlist::
     test_file     : test_header `function_list`
diff --git a/lib/codegen/src/result.rs b/lib/codegen/src/result.rs
index 883ccebda2..0956b4f4d2 100644
--- a/lib/codegen/src/result.rs
+++ b/lib/codegen/src/result.rs
@@ -19,7 +19,7 @@ pub enum CodegenError {
     /// Cranelift can compile very large and complicated functions, but the [implementation has
     /// limits][limits] that cause compilation to fail when they are exceeded.
     ///
-    /// [limits]: https://cranelift.readthedocs.io/en/latest/langref.html#implementation-limits
+    /// [limits]: https://cranelift.readthedocs.io/en/latest/ir.html#implementation-limits
     #[fail(display = "Implementation limit exceeded")]
     ImplLimitExceeded,
 
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 3c52ba32fe..4852790834 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -54,7 +54,7 @@ pub enum WasmError {
     /// Cranelift can compile very large and complicated functions, but the [implementation has
     /// limits][limits] that cause compilation to fail when they are exceeded.
     ///
-    /// [limits]: https://cranelift.readthedocs.io/en/latest/langref.html#implementation-limits
+    /// [limits]: https://cranelift.readthedocs.io/en/latest/ir.html#implementation-limits
     #[fail(display = "Implementation limit exceeded")]
     ImplLimitExceeded,
 }

From 26d122306d951810199879dc41f80c62de753024 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 2 Aug 2018 09:30:37 -0700
Subject: [PATCH 1961/3084] Don't attempt to shrink regfill/regspill/regmove
 instructions.

This is a temporary workaround for bugs such as #420, where normal
operand constraint checking doesn't correctly handle such instructions.
---
 lib/codegen/src/binemit/shrink.rs | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/lib/codegen/src/binemit/shrink.rs b/lib/codegen/src/binemit/shrink.rs
index d18ee7fdb1..dc718e97ca 100644
--- a/lib/codegen/src/binemit/shrink.rs
+++ b/lib/codegen/src/binemit/shrink.rs
@@ -5,6 +5,7 @@
 //! flexibility. However, once register allocation is done, this is no longer important, and we
 //! can switch to smaller encodings when possible.
 
+use ir::instructions::InstructionData;
 use ir::Function;
 use isa::TargetIsa;
 use regalloc::RegDiversions;
@@ -22,6 +23,25 @@ pub fn shrink_instructions(func: &mut Function, isa: &TargetIsa) {
         for inst in func.layout.ebb_insts(ebb) {
             let enc = func.encodings[inst];
             if enc.is_legal() {
+                // regmove/regfill/regspill are special instructions with register immediates
+                // that represented as normal operands, so the normal predicates below don't
+                // handle them correctly.
+                //
+                // Also, they need to be presented to the `RegDiversions` to update the
+                // location tracking.
+                //
+                // TODO: Eventually, we want the register allocator to avoid leaving these special
+                // instructions behind, but for now, just temporarily avoid trying to shrink them.
+                match func.dfg[inst] {
+                    InstructionData::RegMove { .. }
+                    | InstructionData::RegFill { .. }
+                    | InstructionData::RegSpill { .. } => {
+                        divert.apply(&func.dfg[inst]);
+                        continue;
+                   }
+                   _ => ()
+                }
+
                 let ctrl_type = func.dfg.ctrl_typevar(inst);
 
                 // Pick the last encoding with constraints that are satisfied.
@@ -43,7 +63,6 @@ pub fn shrink_instructions(func: &mut Function, isa: &TargetIsa) {
                     );
                 }
             }
-            divert.apply(&func.dfg[inst]);
         }
     }
 }

From cd75176f10522be2db3ebc54a8a8dee59506f1ae Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 2 Aug 2018 09:05:47 -0700
Subject: [PATCH 1962/3084] Update to the rustfmt in rust 1.28, which is now
 stable.

Also, rustfmt's --write-mode=check is now named --check.
---
 cranelift/test-all.sh                         |  2 +-
 lib/codegen/src/bforest/path.rs               |  3 +-
 lib/codegen/src/binemit/relaxation.rs         |  3 +-
 lib/codegen/src/binemit/shrink.rs             |  7 +++--
 lib/codegen/src/cursor.rs                     |  3 +-
 lib/codegen/src/dominator_tree.rs             |  6 ++--
 lib/codegen/src/ir/extfunc.rs                 |  3 +-
 lib/codegen/src/ir/layout.rs                  |  9 ++++--
 lib/codegen/src/ir/stackslot.rs               |  6 ++--
 lib/codegen/src/isa/x86/enc_tables.rs         |  3 +-
 lib/codegen/src/iterators.rs                  |  6 +---
 lib/codegen/src/legalizer/boundary.rs         | 21 ++++++++-----
 lib/codegen/src/legalizer/globalvalue.rs      |  3 +-
 lib/codegen/src/legalizer/heap.rs             |  9 ++++--
 lib/codegen/src/legalizer/split.rs            |  9 ++++--
 lib/codegen/src/postopt.rs                    |  3 +-
 lib/codegen/src/regalloc/coalescing.rs        |  9 ++++--
 lib/codegen/src/regalloc/coloring.rs          |  9 ++++--
 .../src/regalloc/live_value_tracker.rs        |  3 +-
 lib/codegen/src/regalloc/liveness.rs          |  3 +-
 lib/codegen/src/regalloc/pressure.rs          |  3 +-
 lib/codegen/src/regalloc/reload.rs            |  9 ++++--
 lib/codegen/src/regalloc/spilling.rs          |  6 ++--
 lib/codegen/src/regalloc/virtregs.rs          |  3 +-
 lib/codegen/src/stack_layout.rs               |  3 +-
 lib/codegen/src/verifier/flags.rs             |  3 +-
 lib/codegen/src/verifier/locations.rs         |  3 +-
 lib/codegen/src/verifier/mod.rs               | 29 ++++++++++-------
 lib/filetests/src/runner.rs                   |  3 +-
 lib/filetests/src/test_binemit.rs             | 31 +++++++++----------
 lib/filetests/src/test_domtree.rs             |  3 +-
 lib/frontend/src/frontend.rs                  |  6 ++--
 lib/frontend/src/ssa.rs                       |  6 ++--
 lib/module/src/module.rs                      |  6 ++--
 lib/reader/src/parser.rs                      | 15 ++++++---
 lib/simplejit/src/backend.rs                  |  9 ++++--
 lib/wasm/src/environ/dummy.rs                 |  9 ++++--
 lib/wasm/src/func_translator.rs               | 30 ++++++------------
 lib/wasm/src/sections_translator.rs           |  8 +++--
 lib/wasm/src/state.rs                         |  6 ++--
 40 files changed, 186 insertions(+), 125 deletions(-)

diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index 35d68920d8..4941e04610 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -27,7 +27,7 @@ function banner {
 # Run rustfmt if we have it.
 banner "Rust formatting"
 if type rustfmt > /dev/null; then
-    if ! "$topdir/format-all.sh" --write-mode=check ; then
+    if ! "$topdir/format-all.sh" --check ; then
         echo "Formatting diffs detected! Run \"cargo fmt --all\" to correct."
         exit 1
     fi
diff --git a/lib/codegen/src/bforest/path.rs b/lib/codegen/src/bforest/path.rs
index e845c6a9a2..c0e4a089af 100644
--- a/lib/codegen/src/bforest/path.rs
+++ b/lib/codegen/src/bforest/path.rs
@@ -621,7 +621,8 @@ impl Path {
 
     /// Update the critical key for the right sibling node at `level`.
     fn update_right_crit_key(&self, level: usize, crit_key: F::Key, pool: &mut NodePool) {
-        let bl = self.right_sibling_branch_level(level, pool)
+        let bl = self
+            .right_sibling_branch_level(level, pool)
             .expect("No right sibling exists");
         match pool[self.node[bl]] {
             NodeData::Inner { ref mut keys, .. } => {
diff --git a/lib/codegen/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs
index 0466e8a615..f23d3218f8 100644
--- a/lib/codegen/src/binemit/relaxation.rs
+++ b/lib/codegen/src/binemit/relaxation.rs
@@ -151,7 +151,8 @@ fn relax_branch(
     // Pick the first encoding that can handle the branch range.
     let dfg = &cur.func.dfg;
     let ctrl_type = dfg.ctrl_typevar(inst);
-    if let Some(enc) = isa.legal_encodings(cur.func, &dfg[inst], ctrl_type)
+    if let Some(enc) = isa
+        .legal_encodings(cur.func, &dfg[inst], ctrl_type)
         .find(|&enc| {
             let range = encinfo.branch_range(enc).expect("Branch with no range");
             if !range.contains(offset, dest_offset) {
diff --git a/lib/codegen/src/binemit/shrink.rs b/lib/codegen/src/binemit/shrink.rs
index dc718e97ca..63b0329278 100644
--- a/lib/codegen/src/binemit/shrink.rs
+++ b/lib/codegen/src/binemit/shrink.rs
@@ -38,14 +38,15 @@ pub fn shrink_instructions(func: &mut Function, isa: &TargetIsa) {
                     | InstructionData::RegSpill { .. } => {
                         divert.apply(&func.dfg[inst]);
                         continue;
-                   }
-                   _ => ()
+                    }
+                    _ => (),
                 }
 
                 let ctrl_type = func.dfg.ctrl_typevar(inst);
 
                 // Pick the last encoding with constraints that are satisfied.
-                let best_enc = isa.legal_encodings(func, &func.dfg[inst], ctrl_type)
+                let best_enc = isa
+                    .legal_encodings(func, &func.dfg[inst], ctrl_type)
                     .filter(|e| encinfo.constraints[e.recipe()].satisfied(inst, &divert, &func))
                     .min_by_key(|e| encinfo.bytes(*e))
                     .unwrap();
diff --git a/lib/codegen/src/cursor.rs b/lib/codegen/src/cursor.rs
index 9fb559a79b..6d475331ad 100644
--- a/lib/codegen/src/cursor.rs
+++ b/lib/codegen/src/cursor.rs
@@ -752,7 +752,8 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut EncCursor<'f> {
         // Assign an encoding.
         // XXX Is there a way to describe this error to the user?
         #[cfg_attr(feature = "cargo-clippy", allow(match_wild_err_arm))]
-        match self.isa
+        match self
+            .isa
             .encode(&self.func, &self.func.dfg[inst], ctrl_typevar)
         {
             Ok(e) => self.func.encodings[inst] = e,
diff --git a/lib/codegen/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs
index d8789fecb9..bd75a16a9f 100644
--- a/lib/codegen/src/dominator_tree.rs
+++ b/lib/codegen/src/dominator_tree.rs
@@ -422,7 +422,8 @@ impl DominatorTree {
         // Get an iterator with just the reachable, already visited predecessors to `ebb`.
         // Note that during the first pass, `rpo_number` is 1 for reachable blocks that haven't
         // been visited yet, 0 for unreachable blocks.
-        let mut reachable_preds = cfg.pred_iter(ebb)
+        let mut reachable_preds = cfg
+            .pred_iter(ebb)
             .filter(|&BasicBlock { ebb: pred, .. }| self.nodes[pred].rpo_number > 1);
 
         // The RPO must visit at least one predecessor before this node.
@@ -453,7 +454,8 @@ impl DominatorTree {
         }
         // We use the RPO comparison on the postorder list so we invert the operands of the
         // comparison
-        let old_ebb_postorder_index = self.postorder
+        let old_ebb_postorder_index = self
+            .postorder
             .as_slice()
             .binary_search_by(|probe| self.rpo_cmp_ebb(old_ebb, *probe))
             .expect("the old ebb is not declared to the dominator tree");
diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs
index 5b7ac8fc5e..5e002b6a0e 100644
--- a/lib/codegen/src/ir/extfunc.rs
+++ b/lib/codegen/src/ir/extfunc.rs
@@ -62,7 +62,8 @@ impl Signature {
     /// Even if there are no stack arguments, this will set `params` to `Some(0)` instead
     /// of `None`. This indicates that the signature has been legalized.
     pub fn compute_argument_bytes(&mut self) {
-        let bytes = self.params
+        let bytes = self
+            .params
             .iter()
             .filter_map(|arg| match arg.location {
                 ArgumentLoc::Stack(offset) if offset >= 0 => {
diff --git a/lib/codegen/src/ir/layout.rs b/lib/codegen/src/ir/layout.rs
index 1e1110a3d8..9f03969061 100644
--- a/lib/codegen/src/ir/layout.rs
+++ b/lib/codegen/src/ir/layout.rs
@@ -182,7 +182,8 @@ impl Layout {
     /// Assign a valid sequence number to `inst` such that the numbers are still monotonic. This may
     /// require renumbering.
     fn assign_inst_seq(&mut self, inst: Inst) {
-        let ebb = self.inst_ebb(inst)
+        let ebb = self
+            .inst_ebb(inst)
             .expect("inst must be inserted before assigning an seq");
 
         // Get the sequence number immediately before `inst`.
@@ -569,7 +570,8 @@ impl Layout {
     /// Insert `inst` before the instruction `before` in the same EBB.
     pub fn insert_inst(&mut self, inst: Inst, before: Inst) {
         debug_assert_eq!(self.inst_ebb(inst), None);
-        let ebb = self.inst_ebb(before)
+        let ebb = self
+            .inst_ebb(before)
             .expect("Instruction before insertion point not in the layout");
         let after = self.insts[before].prev;
         {
@@ -643,7 +645,8 @@ impl Layout {
     ///     i4
     /// ```
     pub fn split_ebb(&mut self, new_ebb: Ebb, before: Inst) {
-        let old_ebb = self.inst_ebb(before)
+        let old_ebb = self
+            .inst_ebb(before)
             .expect("The `before` instruction must be in the layout");
         debug_assert!(!self.is_ebb_inserted(new_ebb));
 
diff --git a/lib/codegen/src/ir/stackslot.rs b/lib/codegen/src/ir/stackslot.rs
index fd5c02314b..41913ac0f0 100644
--- a/lib/codegen/src/ir/stackslot.rs
+++ b/lib/codegen/src/ir/stackslot.rs
@@ -309,7 +309,8 @@ impl StackSlots {
         let size = spill_size(ty);
 
         // Find the smallest existing slot that can fit the type.
-        if let Some(&ss) = self.emergency
+        if let Some(&ss) = self
+            .emergency
             .iter()
             .filter(|&&ss| self[ss].size >= size && !in_use.contains(&ss.into()))
             .min_by_key(|&&ss| self[ss].size)
@@ -318,7 +319,8 @@ impl StackSlots {
         }
 
         // Alternatively, use the largest available slot and make it larger.
-        if let Some(&ss) = self.emergency
+        if let Some(&ss) = self
+            .emergency
             .iter()
             .filter(|&&ss| !in_use.contains(&ss.into()))
             .max_by_key(|&&ss| self[ss].size)
diff --git a/lib/codegen/src/isa/x86/enc_tables.rs b/lib/codegen/src/isa/x86/enc_tables.rs
index 4c964ad802..03c33f52c8 100644
--- a/lib/codegen/src/isa/x86/enc_tables.rs
+++ b/lib/codegen/src/isa/x86/enc_tables.rs
@@ -345,7 +345,8 @@ fn expand_fcvt_to_sint(
     let mut pos = FuncCursor::new(func).after_inst(inst);
     pos.use_srcloc(inst);
 
-    let is_done = pos.ins()
+    let is_done = pos
+        .ins()
         .icmp_imm(IntCC::NotEqual, result, 1 << (ty.lane_bits() - 1));
     pos.ins().brnz(is_done, done, &[]);
 
diff --git a/lib/codegen/src/iterators.rs b/lib/codegen/src/iterators.rs
index 220b90da0c..71cb8cbdf6 100644
--- a/lib/codegen/src/iterators.rs
+++ b/lib/codegen/src/iterators.rs
@@ -13,11 +13,7 @@ pub trait IteratorExtras: Iterator {
     }
 }
 
-impl IteratorExtras for T
-where
-    T: Iterator,
-{
-}
+impl IteratorExtras for T where T: Iterator {}
 
 /// Adjacent pairs iterator returned by `adjacent_pairs()`.
 ///
diff --git a/lib/codegen/src/legalizer/boundary.rs b/lib/codegen/src/legalizer/boundary.rs
index dfc4629566..d89a455454 100644
--- a/lib/codegen/src/legalizer/boundary.rs
+++ b/lib/codegen/src/legalizer/boundary.rs
@@ -187,7 +187,8 @@ fn legalize_inst_results(pos: &mut FuncCursor, mut get_abi_type: ResTyp
 where
     ResType: FnMut(&Function, usize) -> AbiParam,
 {
-    let call = pos.current_inst()
+    let call = pos
+        .current_inst()
         .expect("Cursor must point to a call instruction");
 
     // We theoretically allow for call instructions that return a number of fixed results before
@@ -419,7 +420,8 @@ fn legalize_inst_arguments(
 ) where
     ArgType: FnMut(&Function, usize) -> AbiParam,
 {
-    let inst = pos.current_inst()
+    let inst = pos
+        .current_inst()
         .expect("Cursor must point to a call instruction");
 
     // Lift the value list out of the call instruction so we modify it.
@@ -550,7 +552,8 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph
 
     // Count the special-purpose return values (`link`, `sret`, and `vmctx`) that were appended to
     // the legalized signature.
-    let special_args = func.signature
+    let special_args = func
+        .signature
         .returns
         .iter()
         .rev()
@@ -591,7 +594,8 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph
             // A `link`/`sret`/`vmctx` return value can only appear in a signature that has a
             // unique matching argument. They are appended at the end, so search the signature from
             // the end.
-            let idx = pos.func
+            let idx = pos
+                .func
                 .signature
                 .params
                 .iter()
@@ -599,7 +603,8 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph
                 .expect("No matching special purpose argument.");
             // Get the corresponding entry block value and add it to the return instruction's
             // arguments.
-            let val = pos.func
+            let val = pos
+                .func
                 .dfg
                 .ebb_params(pos.func.layout.entry_block().unwrap())[idx];
             debug_assert_eq!(pos.func.dfg.value_type(val), arg.value_type);
@@ -641,9 +646,11 @@ fn spill_entry_params(func: &mut Function, entry: Ebb) {
 /// or calls between writing the stack slots and the call instruction. Writing the slots earlier
 /// could help reduce register pressure before the call.
 fn spill_call_arguments(pos: &mut FuncCursor) -> bool {
-    let inst = pos.current_inst()
+    let inst = pos
+        .current_inst()
         .expect("Cursor must point to a call instruction");
-    let sig_ref = pos.func
+    let sig_ref = pos
+        .func
         .dfg
         .call_signature(inst)
         .expect("Call instruction expected.");
diff --git a/lib/codegen/src/legalizer/globalvalue.rs b/lib/codegen/src/legalizer/globalvalue.rs
index c6f1a1d930..733d10fa92 100644
--- a/lib/codegen/src/legalizer/globalvalue.rs
+++ b/lib/codegen/src/legalizer/globalvalue.rs
@@ -37,7 +37,8 @@ pub fn expand_global_value(
 /// Expand a `global_value` instruction for a vmctx global.
 fn vmctx_addr(inst: ir::Inst, func: &mut ir::Function, offset: i64) {
     // Get the value representing the `vmctx` argument.
-    let vmctx = func.special_param(ir::ArgumentPurpose::VMContext)
+    let vmctx = func
+        .special_param(ir::ArgumentPurpose::VMContext)
         .expect("Missing vmctx parameter");
 
     // Simply replace the `global_value` instruction with an `iadd_imm`, reusing the result value.
diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs
index e4a04a0bce..c23b5d73b9 100644
--- a/lib/codegen/src/legalizer/heap.rs
+++ b/lib/codegen/src/legalizer/heap.rs
@@ -61,20 +61,23 @@ fn dynamic_addr(
     let oob;
     if access_size == 1 {
         // `offset > bound - 1` is the same as `offset >= bound`.
-        oob = pos.ins()
+        oob = pos
+            .ins()
             .icmp(IntCC::UnsignedGreaterThanOrEqual, offset, bound);
     } else if access_size <= min_size {
         // We know that bound >= min_size, so here we can compare `offset > bound - access_size` without
         // wrapping.
         let adj_bound = pos.ins().iadd_imm(bound, -access_size);
-        oob = pos.ins()
+        oob = pos
+            .ins()
             .icmp(IntCC::UnsignedGreaterThan, offset, adj_bound);
     } else {
         // We need an overflow check for the adjusted offset.
         let access_size_val = pos.ins().iconst(offset_ty, access_size);
         let (adj_offset, overflow) = pos.ins().iadd_cout(offset, access_size_val);
         pos.ins().trapnz(overflow, ir::TrapCode::HeapOutOfBounds);
-        oob = pos.ins()
+        oob = pos
+            .ins()
             .icmp(IntCC::UnsignedGreaterThan, adj_offset, bound);
     }
     pos.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds);
diff --git a/lib/codegen/src/legalizer/split.rs b/lib/codegen/src/legalizer/split.rs
index a7cd37dc60..1cce097f26 100644
--- a/lib/codegen/src/legalizer/split.rs
+++ b/lib/codegen/src/legalizer/split.rs
@@ -139,7 +139,8 @@ fn split_any(
                 .expect("Branches must have value lists.");
             let num_args = args.len(&pos.func.dfg.value_lists);
             // Get the old value passed to the EBB argument we're repairing.
-            let old_arg = args.get(fixed_args + repair.num, &pos.func.dfg.value_lists)
+            let old_arg = args
+                .get(fixed_args + repair.num, &pos.func.dfg.value_lists)
                 .expect("Too few branch arguments");
 
             // It's possible that the CFG's predecessor list has duplicates. Detect them here.
@@ -153,13 +154,15 @@ fn split_any(
             let (lo, hi) = split_value(pos, old_arg, repair.concat, &mut repairs);
 
             // The `lo` part replaces the original argument.
-            *args.get_mut(fixed_args + repair.num, &mut pos.func.dfg.value_lists)
+            *args
+                .get_mut(fixed_args + repair.num, &mut pos.func.dfg.value_lists)
                 .unwrap() = lo;
 
             // The `hi` part goes at the end. Since multiple repairs may have been scheduled to the
             // same EBB, there could be multiple arguments missing.
             if num_args > fixed_args + repair.hi_num {
-                *args.get_mut(fixed_args + repair.hi_num, &mut pos.func.dfg.value_lists)
+                *args
+                    .get_mut(fixed_args + repair.hi_num, &mut pos.func.dfg.value_lists)
                     .unwrap() = hi;
             } else {
                 // We need to append one or more arguments. If we're adding more than one argument,
diff --git a/lib/codegen/src/postopt.rs b/lib/codegen/src/postopt.rs
index b850b5ac08..178f143c6f 100644
--- a/lib/codegen/src/postopt.rs
+++ b/lib/codegen/src/postopt.rs
@@ -334,7 +334,8 @@ pub fn do_postopt(func: &mut Function, isa: &TargetIsa) {
                 optimize_cpu_flags(&mut pos, inst, last_flags_clobber, isa);
 
                 // Track the most recent seen instruction that clobbers the flags.
-                if let Some(constraints) = isa.encoding_info()
+                if let Some(constraints) = isa
+                    .encoding_info()
                     .operand_constraints(pos.func.encodings[inst])
                 {
                     if constraints.clobbers_flags {
diff --git a/lib/codegen/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs
index 687cb84c71..b9415524a0 100644
--- a/lib/codegen/src/regalloc/coalescing.rs
+++ b/lib/codegen/src/regalloc/coalescing.rs
@@ -307,7 +307,8 @@ impl<'a> Context<'a> {
         // Create a live range for the new value.
         // TODO: Should we handle ghost values?
         let affinity = Affinity::new(
-            &self.encinfo
+            &self
+                .encinfo
                 .operand_constraints(pos.func.encodings[inst])
                 .expect("Bad copy encoding")
                 .outs[0],
@@ -352,7 +353,8 @@ impl<'a> Context<'a> {
         // Create a live range for the new value.
         // TODO: Handle affinity for ghost values.
         let affinity = Affinity::new(
-            &self.encinfo
+            &self
+                .encinfo
                 .operand_constraints(pos.func.encodings[inst])
                 .expect("Bad copy encoding")
                 .outs[0],
@@ -419,7 +421,8 @@ impl<'a> Context<'a> {
             let node = Node::value(value, 0, self.func);
 
             // Push this value and get the nearest dominating def back.
-            let parent = match self.forest
+            let parent = match self
+                .forest
                 .push_node(node, self.func, self.domtree, self.preorder)
             {
                 None => continue,
diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs
index 8bb29c62cc..b824246d5b 100644
--- a/lib/codegen/src/regalloc/coloring.rs
+++ b/lib/codegen/src/regalloc/coloring.rs
@@ -527,7 +527,8 @@ impl<'a> Context<'a> {
     /// all values used by the instruction.
     fn program_complete_input_constraints(&mut self) {
         let inst = self.cur.current_inst().expect("Not on an instruction");
-        let constraints = self.encinfo
+        let constraints = self
+            .encinfo
             .operand_constraints(self.cur.func.encodings[inst])
             .expect("Current instruction not encoded")
             .ins;
@@ -643,7 +644,8 @@ impl<'a> Context<'a> {
         Pred: FnMut(&LiveRange, LiveRangeContext) -> bool,
     {
         for rdiv in self.divert.all() {
-            let lr = self.liveness
+            let lr = self
+                .liveness
                 .get(rdiv.value)
                 .expect("Missing live range for diverted register");
             if pred(lr, self.liveness.context(&self.cur.func.layout)) {
@@ -942,7 +944,8 @@ impl<'a> Context<'a> {
                     ..
                 } => {
                     debug_assert_eq!(slot[to_slot].expand(), None, "Overwriting slot in use");
-                    let ss = self.cur
+                    let ss = self
+                        .cur
                         .func
                         .stack_slots
                         .get_emergency_slot(self.cur.func.dfg.value_type(value), &slot[0..spills]);
diff --git a/lib/codegen/src/regalloc/live_value_tracker.rs b/lib/codegen/src/regalloc/live_value_tracker.rs
index 2665c245c1..9fe6e2615a 100644
--- a/lib/codegen/src/regalloc/live_value_tracker.rs
+++ b/lib/codegen/src/regalloc/live_value_tracker.rs
@@ -187,7 +187,8 @@ impl LiveValueTracker {
             // If the immediate dominator exits, we must have a stored list for it. This is a
             // requirement to the order EBBs are visited: All dominators must have been processed
             // before the current EBB.
-            let idom_live_list = self.idom_sets
+            let idom_live_list = self
+                .idom_sets
                 .get(&idom)
                 .expect("No stored live set for dominator");
             let ctx = liveness.context(layout);
diff --git a/lib/codegen/src/regalloc/liveness.rs b/lib/codegen/src/regalloc/liveness.rs
index d169acc459..6e71345b13 100644
--- a/lib/codegen/src/regalloc/liveness.rs
+++ b/lib/codegen/src/regalloc/liveness.rs
@@ -340,7 +340,8 @@ impl Liveness {
     where
         PP: Into,
     {
-        let old = self.ranges
+        let old = self
+            .ranges
             .insert(LiveRange::new(value, def.into(), affinity));
         debug_assert!(old.is_none(), "{} already has a live range", value);
     }
diff --git a/lib/codegen/src/regalloc/pressure.rs b/lib/codegen/src/regalloc/pressure.rs
index ef8ec425aa..a70407bad1 100644
--- a/lib/codegen/src/regalloc/pressure.rs
+++ b/lib/codegen/src/regalloc/pressure.rs
@@ -114,7 +114,8 @@ impl Pressure {
         }
 
         // Compute per-class limits from `usable`.
-        for (toprc, rc) in p.toprc
+        for (toprc, rc) in p
+            .toprc
             .iter_mut()
             .take_while(|t| t.num_toprcs > 0)
             .zip(reginfo.classes)
diff --git a/lib/codegen/src/regalloc/reload.rs b/lib/codegen/src/regalloc/reload.rs
index b8ba2166ca..a712616af8 100644
--- a/lib/codegen/src/regalloc/reload.rs
+++ b/lib/codegen/src/regalloc/reload.rs
@@ -166,7 +166,8 @@ impl<'a> Context<'a> {
                     if arg.affinity.is_stack() {
                         // An incoming register parameter was spilled. Replace the parameter value
                         // with a temporary register value that is immediately spilled.
-                        let reg = self.cur
+                        let reg = self
+                            .cur
                             .func
                             .dfg
                             .replace_ebb_param(arg.value, abi.value_type);
@@ -199,7 +200,8 @@ impl<'a> Context<'a> {
         self.cur.use_srcloc(inst);
 
         // Get the operand constraints for `inst` that we are trying to satisfy.
-        let constraints = self.encinfo
+        let constraints = self
+            .encinfo
             .operand_constraints(encoding)
             .expect("Missing instruction encoding");
 
@@ -276,7 +278,8 @@ impl<'a> Context<'a> {
         // Same thing for spilled call return values.
         let retvals = &defs[constraints.outs.len()..];
         if !retvals.is_empty() {
-            let sig = self.cur
+            let sig = self
+                .cur
                 .func
                 .dfg
                 .call_signature(inst)
diff --git a/lib/codegen/src/regalloc/spilling.rs b/lib/codegen/src/regalloc/spilling.rs
index 65d43e7cb3..457b07c663 100644
--- a/lib/codegen/src/regalloc/spilling.rs
+++ b/lib/codegen/src/regalloc/spilling.rs
@@ -125,7 +125,8 @@ impl<'a> Context<'a> {
         self.process_spills(tracker);
 
         while let Some(inst) = self.cur.next_inst() {
-            if let Some(constraints) = self.encinfo
+            if let Some(constraints) = self
+                .encinfo
                 .operand_constraints(self.cur.func.encodings[inst])
             {
                 self.visit_inst(inst, ebb, constraints, tracker);
@@ -494,7 +495,8 @@ impl<'a> Context<'a> {
         }
 
         // Assign a spill slot for the whole virtual register.
-        let ss = self.cur
+        let ss = self
+            .cur
             .func
             .stack_slots
             .make_spill_slot(self.cur.func.dfg.value_type(value));
diff --git a/lib/codegen/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs
index ab2b749f7c..0e4bcdf5d0 100644
--- a/lib/codegen/src/regalloc/virtregs.rs
+++ b/lib/codegen/src/regalloc/virtregs.rs
@@ -152,7 +152,8 @@ impl VirtRegs {
         });
 
         // Determine the insertion position for `single`.
-        let index = match self.values(vreg)
+        let index = match self
+            .values(vreg)
             .binary_search_by(|&v| preorder.pre_cmp_def(v, single, func))
         {
             Ok(_) => panic!("{} already in {}", single, vreg),
diff --git a/lib/codegen/src/stack_layout.rs b/lib/codegen/src/stack_layout.rs
index eced3ada0a..9a34e06374 100644
--- a/lib/codegen/src/stack_layout.rs
+++ b/lib/codegen/src/stack_layout.rs
@@ -49,7 +49,8 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CodegenResu
                 incoming_min = min(incoming_min, slot.offset.unwrap());
             }
             StackSlotKind::OutgoingArg => {
-                let offset = slot.offset
+                let offset = slot
+                    .offset
                     .unwrap()
                     .checked_add(slot.size as StackOffset)
                     .ok_or(CodegenError::ImplLimitExceeded)?;
diff --git a/lib/codegen/src/verifier/flags.rs b/lib/codegen/src/verifier/flags.rs
index 04615bacd7..3895cd63a0 100644
--- a/lib/codegen/src/verifier/flags.rs
+++ b/lib/codegen/src/verifier/flags.rs
@@ -98,7 +98,8 @@ impl<'a> FlagsVerifier<'a> {
                 }
 
                 // Does the instruction have an encoding that clobbers the CPU flags?
-                if self.encinfo
+                if self
+                    .encinfo
                     .as_ref()
                     .and_then(|ei| ei.operand_constraints(self.func.encodings[inst]))
                     .map_or(false, |c| c.clobbers_flags) && live_val.is_some()
diff --git a/lib/codegen/src/verifier/locations.rs b/lib/codegen/src/verifier/locations.rs
index 97dfbdc6fb..cc393bb46f 100644
--- a/lib/codegen/src/verifier/locations.rs
+++ b/lib/codegen/src/verifier/locations.rs
@@ -87,7 +87,8 @@ impl<'a> LocationVerifier<'a> {
         enc: isa::Encoding,
         divert: &RegDiversions,
     ) -> VerifierResult<()> {
-        let constraints = self.encinfo
+        let constraints = self
+            .encinfo
             .operand_constraints(enc)
             .expect("check_enc_constraints requires a legal encoding");
 
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index a25be6bcec..e0ff926bbc 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -191,7 +191,8 @@ impl<'a> Verifier<'a> {
             }
 
             if let ir::GlobalValueData::VMContext { .. } = self.func.global_values[cur] {
-                if self.func
+                if self
+                    .func
                     .special_param(ir::ArgumentPurpose::VMContext)
                     .is_none()
                 {
@@ -253,7 +254,8 @@ impl<'a> Verifier<'a> {
 
         let fixed_results = inst_data.opcode().constraints().fixed_results();
         // var_results is 0 if we aren't a call instruction
-        let var_results = dfg.call_signature(inst)
+        let var_results = dfg
+            .call_signature(inst)
             .map_or(0, |sig| dfg.signatures[sig].returns.len());
         let total_results = fixed_results + var_results;
 
@@ -498,7 +500,8 @@ impl<'a> Verifier<'a> {
                 }
                 // Defining instruction dominates the instruction that uses the value.
                 if is_reachable {
-                    if !self.expected_domtree
+                    if !self
+                        .expected_domtree
                         .dominates(def_inst, loc_inst, &self.func.layout)
                     {
                         return err!(loc_inst, "uses value from non-dominating {}", def_inst);
@@ -529,7 +532,8 @@ impl<'a> Verifier<'a> {
                 }
                 // The defining EBB dominates the instruction using this value.
                 if is_reachable
-                    && !self.expected_domtree
+                    && !self
+                        .expected_domtree
                         .dominates(ebb, loc_inst, &self.func.layout)
                 {
                     return err!(loc_inst, "uses value arg from non-dominating {}", ebb);
@@ -604,7 +608,8 @@ impl<'a> Verifier<'a> {
         }
         // We verify rpo_cmp on pairs of adjacent ebbs in the postorder
         for (&prev_ebb, &next_ebb) in domtree.cfg_postorder().iter().adjacent_pairs() {
-            if self.expected_domtree
+            if self
+                .expected_domtree
                 .rpo_cmp(prev_ebb, next_ebb, &self.func.layout) != Ordering::Greater
             {
                 return err!(
@@ -743,7 +748,8 @@ impl<'a> Verifier<'a> {
     fn typecheck_variable_args(&self, inst: Inst) -> VerifierResult<()> {
         match self.func.dfg.analyze_branch(inst) {
             BranchInfo::SingleDest(ebb, _) => {
-                let iter = self.func
+                let iter = self
+                    .func
                     .dfg
                     .ebb_params(ebb)
                     .iter()
@@ -1038,11 +1044,12 @@ impl<'a> Verifier<'a> {
 
         let encoding = self.func.encodings[inst];
         if encoding.is_legal() {
-            let mut encodings = isa.legal_encodings(
-                &self.func,
-                &self.func.dfg[inst],
-                self.func.dfg.ctrl_typevar(inst),
-            ).peekable();
+            let mut encodings =
+                isa.legal_encodings(
+                    &self.func,
+                    &self.func.dfg[inst],
+                    self.func.dfg.ctrl_typevar(inst),
+                ).peekable();
 
             if encodings.peek().is_none() {
                 return err!(
diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs
index 80b2e47d15..08ee75430b 100644
--- a/lib/filetests/src/runner.rs
+++ b/lib/filetests/src/runner.rs
@@ -282,7 +282,8 @@ impl TestRunner {
     /// Print out a report of slow tests.
     fn report_slow_tests(&self) {
         // Collect runtimes of succeeded tests.
-        let mut times = self.tests
+        let mut times = self
+            .tests
             .iter()
             .filter_map(|entry| match *entry {
                 QueueEntry {
diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs
index 0dd2ea6fcf..ef8e93936a 100644
--- a/lib/filetests/src/test_binemit.rs
+++ b/lib/filetests/src/test_binemit.rs
@@ -116,7 +116,8 @@ impl SubTest for TestBinEmit {
         let mut func = func.into_owned();
 
         // Fix the stack frame layout so we can test spill/fill encodings.
-        let min_offset = func.stack_slots
+        let min_offset = func
+            .stack_slots
             .values()
             .map(|slot| slot.offset.unwrap())
             .min();
@@ -133,14 +134,12 @@ impl SubTest for TestBinEmit {
                     // Find an encoding that satisfies both immediate field and register
                     // constraints.
                     if let Some(enc) = {
-                        let mut legal_encodings = isa.legal_encodings(
-                            &func,
-                            &func.dfg[inst],
-                            func.dfg.ctrl_typevar(inst),
-                        ).filter(|e| {
-                            let recipe_constraints = &encinfo.constraints[e.recipe()];
-                            recipe_constraints.satisfied(inst, &divert, &func)
-                        });
+                        let mut legal_encodings = isa
+                            .legal_encodings(&func, &func.dfg[inst], func.dfg.ctrl_typevar(inst))
+                            .filter(|e| {
+                                let recipe_constraints = &encinfo.constraints[e.recipe()];
+                                recipe_constraints.satisfied(inst, &divert, &func)
+                            });
 
                         if opt_level == OptLevel::Best {
                             // Get the smallest legal encoding
@@ -207,7 +206,8 @@ impl SubTest for TestBinEmit {
                 // Send legal encodings into the emitter.
                 if enc.is_legal() {
                     // Generate a better error message if output locations are not specified.
-                    if let Some(&v) = func.dfg
+                    if let Some(&v) = func
+                        .dfg
                         .inst_results(inst)
                         .iter()
                         .find(|&&v| !func.locations[v].is_assigned())
@@ -236,7 +236,8 @@ impl SubTest for TestBinEmit {
                     if !enc.is_legal() {
                         // A possible cause of an unencoded instruction is a missing location for
                         // one of the input operands.
-                        if let Some(&v) = func.dfg
+                        if let Some(&v) = func
+                            .dfg
                             .inst_args(inst)
                             .iter()
                             .find(|&&v| !func.locations[v].is_assigned())
@@ -249,11 +250,9 @@ impl SubTest for TestBinEmit {
                         }
 
                         // Do any encodings exist?
-                        let encodings = isa.legal_encodings(
-                            &func,
-                            &func.dfg[inst],
-                            func.dfg.ctrl_typevar(inst),
-                        ).map(|e| encinfo.display(e))
+                        let encodings = isa
+                            .legal_encodings(&func, &func.dfg[inst], func.dfg.ctrl_typevar(inst))
+                            .map(|e| encinfo.display(e))
                             .collect::>();
 
                         if encodings.is_empty() {
diff --git a/lib/filetests/src/test_domtree.rs b/lib/filetests/src/test_domtree.rs
index 87df48a7a2..93a1b74a9d 100644
--- a/lib/filetests/src/test_domtree.rs
+++ b/lib/filetests/src/test_domtree.rs
@@ -93,7 +93,8 @@ impl SubTest for TestDomtree {
 
         // Now we know that everything in `expected` is consistent with `domtree`.
         // All other EBB's should be either unreachable or the entry block.
-        for ebb in func.layout
+        for ebb in func
+            .layout
             .ebbs()
             .skip(1)
             .filter(|ebb| !expected.contains_key(ebb))
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index bccf6cc533..16934282a8 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -167,7 +167,8 @@ where
                         // capable of having the same successor appear
                         // multiple times, so we must deduplicate.
                         let mut unique = EntitySet::::new();
-                        for dest_ebb in self.builder
+                        for dest_ebb in self
+                            .builder
                             .func
                             .jump_tables
                             .get(table)
@@ -544,7 +545,8 @@ where
             Some(entry) => self.position.ebb.unwrap() == entry,
         };
         !is_entry && self.func_ctx.ssa.is_sealed(self.position.ebb.unwrap())
-            && !self.func_ctx
+            && !self
+                .func_ctx
                 .ssa
                 .has_any_predecessors(self.position.ebb.unwrap())
     }
diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs
index 09040bf6fb..f5a3c4bd8a 100644
--- a/lib/frontend/src/ssa.rs
+++ b/lib/frontend/src/ssa.rs
@@ -103,7 +103,8 @@ impl BlockData {
             BlockData::EbbHeader(ref mut data) => {
                 // This a linear complexity operation but the number of predecessors is low
                 // in all non-pathological cases
-                let pred: usize = data.predecessors
+                let pred: usize = data
+                    .predecessors
                     .iter()
                     .position(|&PredBlock { branch, .. }| branch == inst)
                     .expect("the predecessor you are trying to remove is not declared");
@@ -597,7 +598,8 @@ where
                 } in &mut preds
                 {
                     // We already did a full `use_var` above, so we can do just the fast path.
-                    let pred_val = self.variables
+                    let pred_val = self
+                        .variables
                         .get(temp_arg_var)
                         .unwrap()
                         .get(*pred_block)
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 10008caa51..d5c616db66 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -526,7 +526,8 @@ where
             "imported data cannot contain references"
         );
         self.backend.write_data_funcaddr(
-            &mut info.compiled
+            &mut info
+                .compiled
                 .as_mut()
                 .expect("`data` must refer to a defined data object"),
             offset,
@@ -549,7 +550,8 @@ where
             "imported data cannot contain references"
         );
         self.backend.write_data_dataaddr(
-            &mut info.compiled
+            &mut info
+                .compiled
                 .as_mut()
                 .expect("`data` must refer to a defined data object"),
             offset,
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index bef37a86fd..2c73d099d9 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -644,10 +644,12 @@ impl<'a> Parser<'a> {
         if let Some(Token::Name(name)) = self.token() {
             self.consume();
             match isa {
-                Some(isa) => isa.register_info()
+                Some(isa) => isa
+                    .register_info()
                     .parse_regunit(name)
                     .ok_or_else(|| self.error("invalid register name")),
-                None => name.parse()
+                None => name
+                    .parse()
                     .map_err(|_| self.error("invalid register number")),
             }
         } else {
@@ -1032,7 +1034,8 @@ impl<'a> Parser<'a> {
                     self.parse_jump_table_decl()
                         .and_then(|(jt, dat)| ctx.add_jt(jt, dat, self.loc))
                 }
-                Some(Token::Identifier("stack_limit")) => self.parse_stack_limit_decl()
+                Some(Token::Identifier("stack_limit")) => self
+                    .parse_stack_limit_decl()
                     .and_then(|gv| ctx.set_stack_limit(gv, self.loc)),
                 // More to come..
                 _ => return Ok(()),
@@ -1053,7 +1056,8 @@ impl<'a> Parser<'a> {
         let kind = self.match_enum("expected stack slot kind")?;
 
         // stack-slot-decl ::= StackSlot(ss) "=" stack-slot-kind * Bytes {"," stack-slot-flag}
-        let bytes: i64 = self.match_imm64("expected byte-size in stack_slot decl")?
+        let bytes: i64 = self
+            .match_imm64("expected byte-size in stack_slot decl")?
             .into();
         if bytes < 0 {
             return err!(self.loc, "negative stack slot size");
@@ -1708,7 +1712,8 @@ impl<'a> Parser<'a> {
         }
 
         if let Some(result_locations) = result_locations {
-            for (&value, loc) in ctx.function
+            for (&value, loc) in ctx
+                .function
                 .dfg
                 .inst_results(inst)
                 .iter()
diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index e78c9d1124..0ae65a8a99 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -122,7 +122,8 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         code_size: u32,
     ) -> ModuleResult {
         let size = code_size as usize;
-        let ptr = self.code_memory
+        let ptr = self
+            .code_memory
             .allocate(size)
             .expect("TODO: handle OOM etc.");
         let mut reloc_sink = SimpleJITRelocSink::new();
@@ -155,10 +156,12 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
 
         let size = init.size();
         let storage = match writable {
-            Writability::Readonly => self.writable_memory
+            Writability::Readonly => self
+                .writable_memory
                 .allocate(size)
                 .expect("TODO: handle OOM etc."),
-            Writability::Writable => self.writable_memory
+            Writability::Writable => self
+                .writable_memory
                 .allocate(size)
                 .expect("TODO: handle OOM etc."),
         };
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index d13fd4367d..68e7e2f556 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -210,7 +210,8 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         call_args: &[ir::Value],
     ) -> WasmResult {
         // Pass the current function's vmctx parameter on to the callee.
-        let vmctx = pos.func
+        let vmctx = pos
+            .func
             .special_param(ir::ArgumentPurpose::VMContext)
             .expect("Missing vmctx parameter");
 
@@ -236,7 +237,8 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         args.extend(call_args.iter().cloned(), &mut pos.func.dfg.value_lists);
         args.push(vmctx, &mut pos.func.dfg.value_lists);
 
-        Ok(pos.ins()
+        Ok(pos
+            .ins()
             .CallIndirect(ir::Opcode::CallIndirect, VOID, sig_ref, args)
             .0)
     }
@@ -249,7 +251,8 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         call_args: &[ir::Value],
     ) -> WasmResult {
         // Pass the current function's vmctx parameter on to the callee.
-        let vmctx = pos.func
+        let vmctx = pos
+            .func
             .special_param(ir::ArgumentPurpose::VMContext)
             .expect("Missing vmctx parameter");
 
diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs
index 01481e9b8f..8f764c102a 100644
--- a/lib/wasm/src/func_translator.rs
+++ b/lib/wasm/src/func_translator.rs
@@ -248,10 +248,8 @@ mod tests {
         // )
         const BODY: [u8; 7] = [
             0x00, // local decl count
-            0x20,
-            0x00, // get_local 0
-            0x41,
-            0x01, // i32.const 1
+            0x20, 0x00, // get_local 0
+            0x41, 0x01, // i32.const 1
             0x6a, // i32.add
             0x0b, // end
         ];
@@ -280,10 +278,8 @@ mod tests {
         // )
         const BODY: [u8; 8] = [
             0x00, // local decl count
-            0x20,
-            0x00, // get_local 0
-            0x41,
-            0x01, // i32.const 1
+            0x20, 0x00, // get_local 0
+            0x41, 0x01, // i32.const 1
             0x6a, // i32.add
             0x0f, // return
             0x0b, // end
@@ -318,19 +314,13 @@ mod tests {
         // )
         const BODY: [u8; 16] = [
             0x01, // 1 local decl.
-            0x01,
-            0x7f, // 1 i32 local.
-            0x03,
-            0x7f, // loop i32
-            0x20,
-            0x00, // get_local 0
-            0x41,
-            0x01, // i32.const 0
+            0x01, 0x7f, // 1 i32 local.
+            0x03, 0x7f, // loop i32
+            0x20, 0x00, // get_local 0
+            0x41, 0x01, // i32.const 0
             0x6a, // i32.add
-            0x21,
-            0x00, // set_local 0
-            0x0c,
-            0x00, // br 0
+            0x21, 0x00, // set_local 0
+            0x0c, 0x00, // br 0
             0x0b, // end
             0x0b, // end
         ];
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 6bfc55372b..6fdeb38719 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -412,9 +412,11 @@ pub fn parse_code_section<'data>(
         }
         let mut reader = parser.create_binary_reader();
         let size = reader.bytes_remaining();
-        environ.define_function_body(reader
-            .read_bytes(size)
-            .map_err(WasmError::from_binary_reader_error)?)?;
+        environ.define_function_body(
+            reader
+                .read_bytes(size)
+                .map_err(WasmError::from_binary_reader_error)?,
+        )?;
     }
     Ok(())
 }
diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs
index c08c514d3a..ff7ced6b78 100644
--- a/lib/wasm/src/state.rs
+++ b/lib/wasm/src/state.rs
@@ -282,7 +282,8 @@ impl TranslationState {
         environ: &mut FE,
     ) -> GlobalVariable {
         let index = index as GlobalIndex;
-        *self.globals
+        *self
+            .globals
             .entry(index)
             .or_insert_with(|| environ.make_global(func, index))
     }
@@ -296,7 +297,8 @@ impl TranslationState {
         environ: &mut FE,
     ) -> ir::Heap {
         let index = index as MemoryIndex;
-        *self.heaps
+        *self
+            .heaps
             .entry(index)
             .or_insert_with(|| environ.make_heap(func, index))
     }

From 1b30265c5c98e9b184bce197fa6dd9b97700a5df Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Sun, 20 May 2018 07:48:46 -0700
Subject: [PATCH 1963/3084] Define a "table" concept.

"Table" is to WebAssembly tables as "Heap" is to WebAssembly linear
memories.
---
 .../filetests/isa/x86/legalize-table.clif     |  27 ++++
 lib/codegen/meta-python/base/entities.py      |   3 +
 lib/codegen/meta-python/base/formats.py       |   5 +-
 lib/codegen/meta-python/base/instructions.py  |  32 +++++
 lib/codegen/meta-python/base/legalize.py      |   1 +
 lib/codegen/src/ir/entities.rs                |  27 ++++
 lib/codegen/src/ir/function.rs                |  12 +-
 lib/codegen/src/ir/mod.rs                     |   4 +-
 lib/codegen/src/ir/table.rs                   |  32 +++++
 lib/codegen/src/ir/trapcode.rs                |  12 +-
 lib/codegen/src/legalizer/heap.rs             |   8 +-
 lib/codegen/src/legalizer/mod.rs              |   2 +
 lib/codegen/src/legalizer/table.rs            | 115 +++++++++++++++++
 lib/codegen/src/verifier/mod.rs               |  11 ++
 lib/codegen/src/write.rs                      |   1 +
 lib/reader/src/lexer.rs                       |   2 +
 lib/reader/src/parser.rs                      | 118 +++++++++++++++++-
 lib/reader/src/sourcemap.rs                   |  21 +++-
 lib/serde/src/serde_clif_json.rs              |  17 +++
 lib/wasm/src/code_translator.rs               |   2 +
 lib/wasm/src/environ/dummy.rs                 |  31 ++++-
 lib/wasm/src/environ/spec.rs                  |  12 ++
 lib/wasm/src/state.rs                         |  21 +++-
 23 files changed, 500 insertions(+), 16 deletions(-)
 create mode 100644 cranelift/filetests/isa/x86/legalize-table.clif
 create mode 100644 lib/codegen/src/ir/table.rs
 create mode 100644 lib/codegen/src/legalizer/table.rs

diff --git a/cranelift/filetests/isa/x86/legalize-table.clif b/cranelift/filetests/isa/x86/legalize-table.clif
new file mode 100644
index 0000000000..afad7d8e21
--- /dev/null
+++ b/cranelift/filetests/isa/x86/legalize-table.clif
@@ -0,0 +1,27 @@
+; Test legalization of tables
+test legalizer
+target x86_64
+
+; regex: V=v\d+
+; regex: EBB=ebb\d+
+
+function %test0(i64 vmctx, i64) -> i64 {
+    gv0 = vmctx+12
+    gv1 = vmctx+14
+    table0 = dynamic gv0, min 20, bound gv1, element_size 4
+
+ebb0(v0: i64, v1: i64):
+    v2 = table_addr.i64 table0, v1, +3
+    return v2
+}
+
+; check:    $(bound=$V) = iadd_imm $(input=$V), 14
+; nextln:   $(cond=$V) = icmp uge $(limit=$V), $bound
+; nextln:   brz $cond, ebb1
+; nextln:   trap table_oob
+; nextln: 
+; nextln: ebb1:
+; nextln:   $(base=$V) = iadd_imm.i64 $(vmctx=$V), 12
+; nextln:   $(scaled=$V) = ishl_imm.i64 $(index=$V), 2
+; nextln:   $(elem_addr=$V) = iadd $base, $scaled
+; nextln:   $(field_addr=$V) = iadd_imm $elem_addr, 3
diff --git a/lib/codegen/meta-python/base/entities.py b/lib/codegen/meta-python/base/entities.py
index d0a0bc470f..7b11810f48 100644
--- a/lib/codegen/meta-python/base/entities.py
+++ b/lib/codegen/meta-python/base/entities.py
@@ -33,3 +33,6 @@ jump_table = EntityRefKind(
 
 #: A reference to a heap declared in the function preamble.
 heap = EntityRefKind('heap', 'A heap.')
+
+#: A reference to a table declared in the function preamble.
+table = EntityRefKind('table', 'A table.')
diff --git a/lib/codegen/meta-python/base/formats.py b/lib/codegen/meta-python/base/formats.py
index 2642db6085..c008306409 100644
--- a/lib/codegen/meta-python/base/formats.py
+++ b/lib/codegen/meta-python/base/formats.py
@@ -11,7 +11,7 @@ from cdsl.operands import VALUE, VARIABLE_ARGS
 from .immediates import imm64, uimm8, uimm32, ieee32, ieee64, offset32
 from .immediates import boolean, intcc, floatcc, memflags, regunit, trapcode
 from . import entities
-from .entities import ebb, sig_ref, func_ref, stack_slot, heap
+from .entities import ebb, sig_ref, func_ref, stack_slot, heap, table
 
 Unary = InstructionFormat(VALUE)
 UnaryImm = InstructionFormat(imm64)
@@ -67,6 +67,9 @@ StackStore = InstructionFormat(VALUE, stack_slot, offset32)
 # Accessing a WebAssembly heap.
 HeapAddr = InstructionFormat(heap, VALUE, uimm32)
 
+# Accessing a WebAssembly table.
+TableAddr = InstructionFormat(table, VALUE, offset32)
+
 RegMove = InstructionFormat(VALUE, ('src', regunit), ('dst', regunit))
 CopySpecial = InstructionFormat(('src', regunit), ('dst', regunit))
 RegSpill = InstructionFormat(
diff --git a/lib/codegen/meta-python/base/instructions.py b/lib/codegen/meta-python/base/instructions.py
index f2b9782ec3..84c704f888 100644
--- a/lib/codegen/meta-python/base/instructions.py
+++ b/lib/codegen/meta-python/base/instructions.py
@@ -221,6 +221,11 @@ call_indirect = Instruction(
 
         Call the function pointed to by `callee` with the given arguments. The
         called function must match the specified signature.
+
+        Note that this is different from WebAssembly's ``call_indirect``; the
+        callee is a native address, rather than a table index. For WebAssembly,
+        :inst:`table_addr` and :inst:`load` are used to obtain a native address
+        from a table.
         """,
         ins=(SIG, callee, args), outs=rvals, is_call=True)
 
@@ -531,6 +536,33 @@ heap_addr = Instruction(
         """,
         ins=(H, p, Size), outs=addr)
 
+#
+# WebAssembly bounds-checked table accesses.
+#
+
+TableOffset = TypeVar('TableOffset', 'An unsigned table offset', ints=(32, 64))
+
+T = Operand('T', entities.table)
+p = Operand('p', TableOffset)
+Offset = Operand('Offset', offset32, 'Byte offset from element address')
+
+table_addr = Instruction(
+        'table_addr', r"""
+        Bounds check and compute absolute address of a table entry.
+
+        Verify that the offset ``p`` is in bounds for the table T, and generate
+        an absolute address that is safe to dereference.
+
+        ``Offset`` must be less than the size of a table element.
+
+        1. If ``p`` is not greater than the table bound, return an absolute
+           address corresponding to a byte offset of ``p`` from the table's
+           base address.
+        2. If ``p`` is greater than the table bound, generate a trap.
+        """,
+        ins=(T, p, Offset), outs=addr)
+
+
 #
 # Materializing constants.
 #
diff --git a/lib/codegen/meta-python/base/legalize.py b/lib/codegen/meta-python/base/legalize.py
index ca4c5b9a6e..c63dde9289 100644
--- a/lib/codegen/meta-python/base/legalize.py
+++ b/lib/codegen/meta-python/base/legalize.py
@@ -65,6 +65,7 @@ expand_flags = XFormGroup('expand_flags', """
 # Custom expansions for memory objects.
 expand.custom_legalize(insts.global_value, 'expand_global_value')
 expand.custom_legalize(insts.heap_addr, 'expand_heap_addr')
+expand.custom_legalize(insts.table_addr, 'expand_table_addr')
 
 # Custom expansions for calls.
 expand.custom_legalize(insts.call, 'expand_call')
diff --git a/lib/codegen/src/ir/entities.rs b/lib/codegen/src/ir/entities.rs
index 7196c78f31..185af8b9e7 100644
--- a/lib/codegen/src/ir/entities.rs
+++ b/lib/codegen/src/ir/entities.rs
@@ -172,6 +172,24 @@ impl Heap {
     }
 }
 
+/// A reference to a table.
+#[derive(Copy, Clone, PartialEq, Eq, Hash)]
+pub struct Table(u32);
+entity_impl!(Table, "table");
+
+impl Table {
+    /// Create a new table reference from its number.
+    ///
+    /// This method is for use by the parser.
+    pub fn with_number(n: u32) -> Option {
+        if n < u32::MAX {
+            Some(Table(n))
+        } else {
+            None
+        }
+    }
+}
+
 /// A reference to any of the entities defined in this module.
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
 pub enum AnyEntity {
@@ -195,6 +213,8 @@ pub enum AnyEntity {
     SigRef(SigRef),
     /// A heap.
     Heap(Heap),
+    /// A table.
+    Table(Table),
 }
 
 impl fmt::Display for AnyEntity {
@@ -210,6 +230,7 @@ impl fmt::Display for AnyEntity {
             AnyEntity::FuncRef(r) => r.fmt(f),
             AnyEntity::SigRef(r) => r.fmt(f),
             AnyEntity::Heap(r) => r.fmt(f),
+            AnyEntity::Table(r) => r.fmt(f),
         }
     }
 }
@@ -274,6 +295,12 @@ impl From for AnyEntity {
     }
 }
 
+impl From for AnyEntity {
+    fn from(r: Table) -> Self {
+        AnyEntity::Table(r)
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs
index 3898709317..b6d0c80b43 100644
--- a/lib/codegen/src/ir/function.rs
+++ b/lib/codegen/src/ir/function.rs
@@ -9,7 +9,7 @@ use ir;
 use ir::{DataFlowGraph, ExternalName, Layout, Signature};
 use ir::{
     Ebb, ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, JumpTable,
-    JumpTableData, SigRef, StackSlot, StackSlotData,
+    JumpTableData, SigRef, StackSlot, StackSlotData, Table, TableData,
 };
 use ir::{EbbOffsets, InstEncodings, JumpTables, SourceLocs, StackSlots, ValueLocations};
 use isa::{EncInfo, Encoding, Legalize, TargetIsa};
@@ -42,6 +42,9 @@ pub struct Function {
     /// Heaps referenced.
     pub heaps: PrimaryMap,
 
+    /// Tables referenced.
+    pub tables: PrimaryMap,
+
     /// Jump tables used in this function.
     pub jump_tables: JumpTables,
 
@@ -82,6 +85,7 @@ impl Function {
             stack_limit: None,
             global_values: PrimaryMap::new(),
             heaps: PrimaryMap::new(),
+            tables: PrimaryMap::new(),
             jump_tables: PrimaryMap::new(),
             dfg: DataFlowGraph::new(),
             layout: Layout::new(),
@@ -98,6 +102,7 @@ impl Function {
         self.stack_slots.clear();
         self.global_values.clear();
         self.heaps.clear();
+        self.tables.clear();
         self.jump_tables.clear();
         self.dfg.clear();
         self.layout.clear();
@@ -157,6 +162,11 @@ impl Function {
         self.heaps.push(data)
     }
 
+    /// Declares a table accessible to the function.
+    pub fn create_table(&mut self, data: TableData) -> Table {
+        self.tables.push(data)
+    }
+
     /// Return an object that can display this function with correct ISA-specific annotations.
     pub fn display<'a, I: Into>>(&'a self, isa: I) -> DisplayFunction<'a> {
         DisplayFunction(self, isa.into())
diff --git a/lib/codegen/src/ir/mod.rs b/lib/codegen/src/ir/mod.rs
index f93ed993b4..5421911886 100644
--- a/lib/codegen/src/ir/mod.rs
+++ b/lib/codegen/src/ir/mod.rs
@@ -18,6 +18,7 @@ mod memflags;
 mod progpoint;
 mod sourceloc;
 pub mod stackslot;
+mod table;
 mod trapcode;
 pub mod types;
 mod valueloc;
@@ -25,7 +26,7 @@ mod valueloc;
 pub use ir::builder::{InsertBuilder, InstBuilder, InstBuilderBase, InstInserterBase};
 pub use ir::dfg::{DataFlowGraph, ValueDef};
 pub use ir::entities::{
-    Ebb, FuncRef, GlobalValue, Heap, Inst, JumpTable, SigRef, StackSlot, Value,
+    Ebb, FuncRef, GlobalValue, Heap, Inst, JumpTable, SigRef, StackSlot, Table, Value,
 };
 pub use ir::extfunc::{AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData, Signature};
 pub use ir::extname::ExternalName;
@@ -40,6 +41,7 @@ pub use ir::memflags::MemFlags;
 pub use ir::progpoint::{ExpandedProgramPoint, ProgramOrder, ProgramPoint};
 pub use ir::sourceloc::SourceLoc;
 pub use ir::stackslot::{StackSlotData, StackSlotKind, StackSlots};
+pub use ir::table::TableData;
 pub use ir::trapcode::TrapCode;
 pub use ir::types::Type;
 pub use ir::valueloc::{ArgumentLoc, ValueLoc};
diff --git a/lib/codegen/src/ir/table.rs b/lib/codegen/src/ir/table.rs
new file mode 100644
index 0000000000..edfe18aec7
--- /dev/null
+++ b/lib/codegen/src/ir/table.rs
@@ -0,0 +1,32 @@
+//! Tables.
+
+use ir::immediates::Imm64;
+use ir::GlobalValue;
+use std::fmt;
+
+/// Information about a table declaration.
+#[derive(Clone)]
+pub struct TableData {
+    /// Global value giving the address of the start of the table.
+    pub base_gv: GlobalValue,
+
+    /// Guaranteed minimum table size in elements. Table accesses before `min_size` don't need
+    /// bounds checking.
+    pub min_size: Imm64,
+
+    /// Global value giving the current bound of the table, in elements.
+    pub bound_gv: GlobalValue,
+
+    /// The size of a table element, in bytes.
+    pub element_size: Imm64,
+}
+
+impl fmt::Display for TableData {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(
+            f,
+            "{}, min {}, bound {}, element_size {}",
+            self.base_gv, self.min_size, self.bound_gv, self.element_size
+        )
+    }
+}
diff --git a/lib/codegen/src/ir/trapcode.rs b/lib/codegen/src/ir/trapcode.rs
index fe712b7ffa..5bdb5a541d 100644
--- a/lib/codegen/src/ir/trapcode.rs
+++ b/lib/codegen/src/ir/trapcode.rs
@@ -16,10 +16,13 @@ pub enum TrapCode {
 
     /// A `heap_addr` instruction detected an out-of-bounds error.
     ///
-    /// Some out-of-bounds heap accesses are detected by a segmentation fault on the heap guard
-    /// pages.
+    /// Note that not all out-of-bounds heap accesses are reported this way;
+    /// some are detected by a segmentation fault on the heap guard pages.
     HeapOutOfBounds,
 
+    /// A `table_addr` instruction detected an out-of-bounds error.
+    TableOutOfBounds,
+
     /// Other bounds checking error.
     OutOfBounds,
 
@@ -52,6 +55,7 @@ impl Display for TrapCode {
         let identifier = match *self {
             StackOverflow => "stk_ovf",
             HeapOutOfBounds => "heap_oob",
+            TableOutOfBounds => "table_oob",
             OutOfBounds => "oob",
             IndirectCallToNull => "icall_null",
             BadSignature => "bad_sig",
@@ -73,6 +77,7 @@ impl FromStr for TrapCode {
         match s {
             "stk_ovf" => Ok(StackOverflow),
             "heap_oob" => Ok(HeapOutOfBounds),
+            "table_oob" => Ok(TableOutOfBounds),
             "oob" => Ok(OutOfBounds),
             "icall_null" => Ok(IndirectCallToNull),
             "bad_sig" => Ok(BadSignature),
@@ -92,9 +97,10 @@ mod tests {
     use std::string::ToString;
 
     // Everything but user-defined codes.
-    const CODES: [TrapCode; 8] = [
+    const CODES: [TrapCode; 9] = [
         TrapCode::StackOverflow,
         TrapCode::HeapOutOfBounds,
+        TrapCode::TableOutOfBounds,
         TrapCode::OutOfBounds,
         TrapCode::IndirectCallToNull,
         TrapCode::BadSignature,
diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs
index c23b5d73b9..44f434ff8a 100644
--- a/lib/codegen/src/legalizer/heap.rs
+++ b/lib/codegen/src/legalizer/heap.rs
@@ -82,7 +82,7 @@ fn dynamic_addr(
     }
     pos.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds);
 
-    offset_addr(inst, heap, addr_ty, offset, offset_ty, pos.func);
+    compute_addr(inst, heap, addr_ty, offset, offset_ty, pos.func);
 }
 
 /// Expand a `heap_addr` for a static heap.
@@ -134,13 +134,11 @@ fn static_addr(
         pos.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds);
     }
 
-    offset_addr(inst, heap, addr_ty, offset, offset_ty, pos.func);
+    compute_addr(inst, heap, addr_ty, offset, offset_ty, pos.func);
 }
 
 /// Emit code for the base address computation of a `heap_addr` instruction.
-///
-///
-fn offset_addr(
+fn compute_addr(
     inst: ir::Inst,
     heap: ir::Heap,
     addr_ty: ir::Type,
diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs
index f11a652828..7c24a86723 100644
--- a/lib/codegen/src/legalizer/mod.rs
+++ b/lib/codegen/src/legalizer/mod.rs
@@ -26,11 +26,13 @@ mod globalvalue;
 mod heap;
 mod libcall;
 mod split;
+mod table;
 
 use self::call::expand_call;
 use self::globalvalue::expand_global_value;
 use self::heap::expand_heap_addr;
 use self::libcall::expand_as_libcall;
+use self::table::expand_table_addr;
 
 /// Legalize `inst` for `isa`. Return true if any changes to the code were
 /// made; return false if the instruction was successfully encoded as is.
diff --git a/lib/codegen/src/legalizer/table.rs b/lib/codegen/src/legalizer/table.rs
new file mode 100644
index 0000000000..fe017eb096
--- /dev/null
+++ b/lib/codegen/src/legalizer/table.rs
@@ -0,0 +1,115 @@
+//! Legalization of tables.
+//!
+//! This module exports the `expand_table_addr` function which transforms a `table_addr`
+//! instruction into code that depends on the kind of table referenced.
+
+use cursor::{Cursor, FuncCursor};
+use flowgraph::ControlFlowGraph;
+use ir::condcodes::IntCC;
+use ir::immediates::Offset32;
+use ir::{self, InstBuilder};
+use isa::TargetIsa;
+
+/// Expand a `table_addr` instruction according to the definition of the table.
+pub fn expand_table_addr(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    _cfg: &mut ControlFlowGraph,
+    _isa: &TargetIsa,
+) {
+    // Unpack the instruction.
+    let (table, index, element_offset) = match func.dfg[inst] {
+        ir::InstructionData::TableAddr {
+            opcode,
+            table,
+            arg,
+            offset,
+        } => {
+            debug_assert_eq!(opcode, ir::Opcode::TableAddr);
+            (table, arg, offset)
+        }
+        _ => panic!("Wanted table_addr: {}", func.dfg.display_inst(inst, None)),
+    };
+
+    dynamic_addr(inst, table, index, element_offset, func);
+}
+
+/// Expand a `table_addr` for a dynamic table.
+fn dynamic_addr(
+    inst: ir::Inst,
+    table: ir::Table,
+    index: ir::Value,
+    element_offset: Offset32,
+    func: &mut ir::Function,
+) {
+    let bound_gv = func.tables[table].bound_gv;
+    let index_ty = func.dfg.value_type(index);
+    let addr_ty = func.dfg.value_type(func.dfg.first_result(inst));
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+    pos.use_srcloc(inst);
+
+    // Start with the bounds check. Trap if `index + 1 > bound`.
+    let bound = pos.ins().global_value(addr_ty, bound_gv);
+
+    // `index > bound - 1` is the same as `index >= bound`.
+    let oob = pos
+        .ins()
+        .icmp(IntCC::UnsignedGreaterThanOrEqual, index, bound);
+    pos.ins().trapnz(oob, ir::TrapCode::TableOutOfBounds);
+
+    compute_addr(
+        inst,
+        table,
+        addr_ty,
+        index,
+        index_ty,
+        element_offset,
+        pos.func,
+    );
+}
+
+/// Emit code for the base address computation of a `table_addr` instruction.
+fn compute_addr(
+    inst: ir::Inst,
+    table: ir::Table,
+    addr_ty: ir::Type,
+    mut index: ir::Value,
+    index_ty: ir::Type,
+    element_offset: Offset32,
+    func: &mut ir::Function,
+) {
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+    pos.use_srcloc(inst);
+
+    // Convert `index` to `addr_ty`.
+    if index_ty != addr_ty {
+        index = pos.ins().uextend(addr_ty, index);
+    }
+
+    // Add the table base address base
+    let base_gv = pos.func.tables[table].base_gv;
+    let base = pos.ins().global_value(addr_ty, base_gv);
+
+    let element_size = pos.func.tables[table].element_size;
+    let mut offset;
+    let element_size_i64: i64 = element_size.into();
+    debug_assert!(element_size_i64 >= 0);
+    let element_size_u64 = element_size_i64 as u64;
+    if element_size_u64 == 1 {
+        offset = index;
+    } else if element_size_u64.is_power_of_two() {
+        offset = pos
+            .ins()
+            .ishl_imm(index, i64::from(element_size_u64.trailing_zeros()));
+    } else {
+        offset = pos.ins().imul_imm(index, element_size);
+    }
+
+    if element_offset == Offset32::new(0) {
+        pos.func.dfg.replace(inst).iadd(base, offset);
+    } else {
+        let imm: i64 = element_offset.into();
+        offset = pos.ins().iadd(base, offset);
+        pos.func.dfg.replace(inst).iadd_imm(offset, imm);
+    }
+}
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index e0ff926bbc..2888d5336d 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -349,6 +349,9 @@ impl<'a> Verifier<'a> {
             HeapAddr { heap, .. } => {
                 self.verify_heap(inst, heap)?;
             }
+            TableAddr { table, .. } => {
+                self.verify_table(inst, table)?;
+            }
             RegSpill { dst, .. } => {
                 self.verify_stack_slot(inst, dst)?;
             }
@@ -445,6 +448,14 @@ impl<'a> Verifier<'a> {
         }
     }
 
+    fn verify_table(&self, inst: Inst, table: ir::Table) -> VerifierResult<()> {
+        if !self.func.tables.is_valid(table) {
+            err!(inst, "invalid table {}", table)
+        } else {
+            Ok(())
+        }
+    }
+
     fn verify_value_list(&self, inst: Inst, l: &ValueList) -> VerifierResult<()> {
         if !l.is_valid(&self.func.dfg.value_lists) {
             err!(inst, "invalid value list reference {:?}", l)
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index 2f0f6ba25f..dc9282a07e 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -440,6 +440,7 @@ pub fn write_operands(
             ..
         } => write!(w, " {}, {}{}", arg, stack_slot, offset),
         HeapAddr { heap, arg, imm, .. } => write!(w, " {}, {}, {}", heap, arg, imm),
+        TableAddr { table, arg, .. } => write!(w, " {}, {}", table, arg),
         Load {
             flags, arg, offset, ..
         } => write!(w, "{} {}{}", flags, arg, offset),
diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs
index 41e1a37b8b..5a2a0124a2 100644
--- a/lib/reader/src/lexer.rs
+++ b/lib/reader/src/lexer.rs
@@ -36,6 +36,7 @@ pub enum Token<'a> {
     StackSlot(u32),       // ss3
     GlobalValue(u32),     // gv3
     Heap(u32),            // heap2
+    Table(u32),           // table2
     JumpTable(u32),       // jt2
     FuncRef(u32),         // fn2
     SigRef(u32),          // sig2
@@ -340,6 +341,7 @@ impl<'a> Lexer<'a> {
             "ss" => Some(Token::StackSlot(number)),
             "gv" => Some(Token::GlobalValue(number)),
             "heap" => Some(Token::Heap(number)),
+            "table" => Some(Token::Table(number)),
             "jt" => Some(Token::JumpTable(number)),
             "fn" => Some(Token::FuncRef(number)),
             "sig" => Some(Token::SigRef(number)),
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 2c73d099d9..62154a6944 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -9,7 +9,8 @@ use cranelift_codegen::ir::types::VOID;
 use cranelift_codegen::ir::{
     AbiParam, ArgumentExtension, ArgumentLoc, Ebb, ExtFuncData, ExternalName, FuncRef, Function,
     GlobalValue, GlobalValueData, Heap, HeapData, HeapStyle, JumpTable, JumpTableData, MemFlags,
-    Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind, Type, Value, ValueLoc,
+    Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind, Table, TableData, Type,
+    Value, ValueLoc,
 };
 use cranelift_codegen::isa::{self, Encoding, RegUnit, TargetIsa};
 use cranelift_codegen::packed_option::ReservedValue;
@@ -188,6 +189,29 @@ impl<'a> Context<'a> {
         }
     }
 
+    // Allocate a table slot.
+    fn add_table(&mut self, table: Table, data: TableData, loc: Location) -> ParseResult<()> {
+        while self.function.tables.next_key().index() <= table.index() {
+            self.function.create_table(TableData {
+                base_gv: GlobalValue::reserved_value(),
+                min_size: Imm64::new(0),
+                bound_gv: GlobalValue::reserved_value(),
+                element_size: Imm64::new(0),
+            });
+        }
+        self.function.tables[table] = data;
+        self.map.def_table(table, loc)
+    }
+
+    // Resolve a reference to a table.
+    fn check_table(&self, table: Table, loc: &Location) -> ParseResult<()> {
+        if !self.map.contains_table(table) {
+            err!(loc, "undefined table {}", table)
+        } else {
+            Ok(())
+        }
+    }
+
     // Allocate a new signature.
     fn add_sig(&mut self, sig: SigRef, data: Signature, loc: Location) -> ParseResult<()> {
         self.map.def_sig(sig, loc)?;
@@ -447,6 +471,17 @@ impl<'a> Parser<'a> {
         err!(self.loc, err_msg)
     }
 
+    // Match and consume a table reference.
+    fn match_table(&mut self, err_msg: &str) -> ParseResult
 {
+        if let Some(Token::Table(table)) = self.token() {
+            self.consume();
+            if let Some(table) = Table::with_number(table) {
+                return Ok(table);
+            }
+        }
+        err!(self.loc, err_msg)
+    }
+
     // Match and consume a jump table reference.
     fn match_jt(&mut self) -> ParseResult {
         if let Some(Token::JumpTable(jt)) = self.token() {
@@ -1019,6 +1054,11 @@ impl<'a> Parser<'a> {
                     self.parse_heap_decl()
                         .and_then(|(heap, dat)| ctx.add_heap(heap, dat, self.loc))
                 }
+                Some(Token::Table(..)) => {
+                    self.start_gathering_comments();
+                    self.parse_table_decl()
+                        .and_then(|(table, dat)| ctx.add_table(table, dat, self.loc))
+                }
                 Some(Token::SigRef(..)) => {
                     self.start_gathering_comments();
                     self.parse_signature_decl(ctx.unique_isa)
@@ -1129,7 +1169,7 @@ impl<'a> Parser<'a> {
     // heap-style ::= "static" | "dynamic"
     // heap-base ::= GlobalValue(base)
     // heap-attr ::= "min" Imm64(bytes)
-    //             | "max" Imm64(bytes)
+    //             | "bound" Imm64(bytes)
     //             | "guard" Imm64(bytes)
     //
     fn parse_heap_decl(&mut self) -> ParseResult<(Heap, HeapData)> {
@@ -1187,6 +1227,66 @@ impl<'a> Parser<'a> {
         Ok((heap, data))
     }
 
+    // Parse a table decl.
+    //
+    // table-decl ::= * Table(table) "=" table-desc
+    // table-desc ::= table-style table-base { "," table-attr }
+    // table-style ::= "dynamic"
+    // table-base ::= GlobalValue(base)
+    // table-attr ::= "min" Imm64(bytes)
+    //              | "bound" Imm64(bytes)
+    //              | "element_size" Imm64(bytes)
+    //
+    fn parse_table_decl(&mut self) -> ParseResult<(Table, TableData)> {
+        let table = self.match_table("expected table number: table«n»")?;
+        self.match_token(Token::Equal, "expected '=' in table declaration")?;
+
+        let style_name = self.match_any_identifier("expected 'static' or 'dynamic'")?;
+
+        // table-desc ::= table-style * table-base { "," table-attr }
+        // table-base ::= * GlobalValue(base)
+        let base = match self.token() {
+            Some(Token::GlobalValue(base_num)) => match GlobalValue::with_number(base_num) {
+                Some(gv) => gv,
+                None => return err!(self.loc, "invalid global value number for table base"),
+            },
+            _ => return err!(self.loc, "expected table base"),
+        };
+        self.consume();
+
+        let mut data = TableData {
+            base_gv: base,
+            min_size: 0.into(),
+            bound_gv: GlobalValue::reserved_value(),
+            element_size: 0.into(),
+        };
+
+        // table-desc ::= * { "," table-attr }
+        while self.optional(Token::Comma) {
+            match self.match_any_identifier("expected table attribute name")? {
+                "min" => {
+                    data.min_size = self.match_imm64("expected integer min size")?;
+                }
+                "bound" => {
+                    data.bound_gv = match style_name {
+                        "dynamic" => self.match_gv("expected gv bound")?,
+                        t => return err!(self.loc, "unknown table style '{}'", t),
+                    };
+                }
+                "element_size" => {
+                    data.element_size = self.match_imm64("expected integer element size")?;
+                }
+                t => return err!(self.loc, "unknown table attribute '{}'", t),
+            }
+        }
+
+        // Collect any trailing comments.
+        self.token();
+        self.claim_gathered_comments(table);
+
+        Ok((table, data))
+    }
+
     // Parse a signature decl.
     //
     // signature-decl ::= SigRef(sigref) "=" signature
@@ -2155,6 +2255,20 @@ impl<'a> Parser<'a> {
                     imm,
                 }
             }
+            InstructionFormat::TableAddr => {
+                let table = self.match_table("expected table identifier")?;
+                ctx.check_table(table, &self.loc)?;
+                self.match_token(Token::Comma, "expected ',' between operands")?;
+                let arg = self.match_value("expected SSA value table address")?;
+                self.match_token(Token::Comma, "expected ',' between operands")?;
+                let offset = self.optional_offset32()?;
+                InstructionData::TableAddr {
+                    opcode,
+                    table,
+                    arg,
+                    offset,
+                }
+            }
             InstructionFormat::Load => {
                 let flags = self.optional_memflags();
                 let addr = self.match_value("expected SSA value address")?;
diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs
index 17276d325d..4103d08450 100644
--- a/lib/reader/src/sourcemap.rs
+++ b/lib/reader/src/sourcemap.rs
@@ -7,7 +7,9 @@
 //! to parser clients.
 
 use cranelift_codegen::ir::entities::AnyEntity;
-use cranelift_codegen::ir::{Ebb, FuncRef, GlobalValue, Heap, JumpTable, SigRef, StackSlot, Value};
+use cranelift_codegen::ir::{
+    Ebb, FuncRef, GlobalValue, Heap, JumpTable, SigRef, StackSlot, Table, Value,
+};
 use error::{Location, ParseResult};
 use lexer::split_entity_name;
 use std::collections::HashMap;
@@ -46,6 +48,11 @@ impl SourceMap {
         self.locations.contains_key(&heap.into())
     }
 
+    /// Look up a table entity.
+    pub fn contains_table(&self, table: Table) -> bool {
+        self.locations.contains_key(&table.into())
+    }
+
     /// Look up a signature entity.
     pub fn contains_sig(&self, sig: SigRef) -> bool {
         self.locations.contains_key(&sig.into())
@@ -100,6 +107,13 @@ impl SourceMap {
                     Some(heap.into())
                 }
             }),
+            "table" => Table::with_number(num).and_then(|table| {
+                if !self.contains_table(table) {
+                    None
+                } else {
+                    Some(table.into())
+                }
+            }),
             "sig" => SigRef::with_number(num).and_then(|sig| {
                 if !self.contains_sig(sig) {
                     None
@@ -164,6 +178,11 @@ impl SourceMap {
         self.def_entity(entity.into(), loc)
     }
 
+    /// Define the table `entity`.
+    pub fn def_table(&mut self, entity: Table, loc: Location) -> ParseResult<()> {
+        self.def_entity(entity.into(), loc)
+    }
+
     /// Define the signature `entity`.
     pub fn def_sig(&mut self, entity: SigRef, loc: Location) -> ParseResult<()> {
         self.def_entity(entity.into(), loc)
diff --git a/lib/serde/src/serde_clif_json.rs b/lib/serde/src/serde_clif_json.rs
index d466e4d0d0..9bd44b5992 100644
--- a/lib/serde/src/serde_clif_json.rs
+++ b/lib/serde/src/serde_clif_json.rs
@@ -176,6 +176,12 @@ pub enum SerInstData {
         heap: String,
         imm: String,
     },
+    TableAddr {
+        opcode: String,
+        arg: String,
+        table: String,
+        offset: String,
+    },
     RegMove {
         opcode: String,
         arg: String,
@@ -572,6 +578,17 @@ pub fn get_inst_data(inst_index: Inst, func: &Function) -> SerInstData {
             heap: heap.to_string(),
             imm: imm.to_string(),
         },
+        InstructionData::TableAddr {
+            opcode,
+            arg,
+            table,
+            offset,
+        } => SerInstData::TableAddr {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            table: table.to_string(),
+            offset: offset.to_string(),
+        },
         InstructionData::RegMove {
             opcode,
             arg,
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index dec6dda896..dacae0a2c9 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -377,10 +377,12 @@ pub fn translate_operator(
             // `index` is the index of the function's signature and `table_index` is the index of
             // the table to search the function in.
             let (sigref, num_args) = state.get_indirect_sig(builder.func, index, environ);
+            let table = state.get_table(builder.func, table_index, environ);
             let callee = state.pop1();
             let call = environ.translate_call_indirect(
                 builder.cursor(),
                 table_index as TableIndex,
+                table,
                 index as SignatureIndex,
                 sigref,
                 callee,
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 68e7e2f556..1fdbbe7df3 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -1,6 +1,7 @@
 //! "Dummy" environment for testing wasm translation.
 
 use cranelift_codegen::cursor::FuncCursor;
+use cranelift_codegen::ir::immediates::Imm64;
 use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::settings;
@@ -169,7 +170,11 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
 
     fn make_heap(&mut self, func: &mut ir::Function, _index: MemoryIndex) -> ir::Heap {
         // Create a static heap whose base address is stored at `vmctx+0`.
-        let gv = func.create_global_value(ir::GlobalValueData::VMContext { offset: 0.into() });
+        let addr = func.create_global_value(ir::GlobalValueData::VMContext { offset: 0.into() });
+        let gv = func.create_global_value(ir::GlobalValueData::Deref {
+            base: addr,
+            offset: 0.into(),
+        });
 
         func.create_heap(ir::HeapData {
             base: gv,
@@ -181,6 +186,29 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         })
     }
 
+    fn make_table(&mut self, func: &mut ir::Function, _index: TableIndex) -> ir::Table {
+        // Create a table whose base address is stored at `vmctx+0`.
+        let base_gv_addr =
+            func.create_global_value(ir::GlobalValueData::VMContext { offset: 0.into() });
+        let base_gv = func.create_global_value(ir::GlobalValueData::Deref {
+            base: base_gv_addr,
+            offset: 0.into(),
+        });
+        let bound_gv_addr =
+            func.create_global_value(ir::GlobalValueData::VMContext { offset: 0.into() });
+        let bound_gv = func.create_global_value(ir::GlobalValueData::Deref {
+            base: bound_gv_addr,
+            offset: 0.into(),
+        });
+
+        func.create_table(ir::TableData {
+            base_gv,
+            min_size: Imm64::new(0),
+            bound_gv,
+            element_size: Imm64::new(i64::from(self.pointer_bytes()) * 2),
+        })
+    }
+
     fn make_indirect_sig(&mut self, func: &mut ir::Function, index: SignatureIndex) -> ir::SigRef {
         // A real implementation would probably change the calling convention and add `vmctx` and
         // signature index arguments.
@@ -204,6 +232,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         &mut self,
         mut pos: FuncCursor,
         _table_index: TableIndex,
+        _table: ir::Table,
         _sig_index: SignatureIndex,
         sig_ref: ir::SigRef,
         callee: ir::Value,
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 4852790834..1a002d5da1 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -89,6 +89,11 @@ pub trait FuncEnvironment {
         ir::Type::int(u16::from(self.triple().pointer_width().unwrap().bits())).unwrap()
     }
 
+    /// Get the size of a native pointer, in bytes.
+    fn pointer_bytes(&self) -> u8 {
+        self.triple().pointer_width().unwrap().bytes()
+    }
+
     /// Set up the necessary preamble definitions in `func` to access the global variable
     /// identified by `index`.
     ///
@@ -104,6 +109,12 @@ pub trait FuncEnvironment {
     /// The index space covers both imported and locally declared memories.
     fn make_heap(&mut self, func: &mut ir::Function, index: MemoryIndex) -> ir::Heap;
 
+    /// Set up the necessary preamble definitions in `func` to access the table identified
+    /// by `index`.
+    ///
+    /// The index space covers both imported and locally declared tables.
+    fn make_table(&mut self, func: &mut ir::Function, index: TableIndex) -> ir::Table;
+
     /// Set up a signature definition in the preamble of `func` that can be used for an indirect
     /// call with signature `index`.
     ///
@@ -141,6 +152,7 @@ pub trait FuncEnvironment {
         &mut self,
         pos: FuncCursor,
         table_index: TableIndex,
+        table: ir::Table,
         sig_index: SignatureIndex,
         sig_ref: ir::SigRef,
         callee: ir::Value,
diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs
index ff7ced6b78..d81234fbde 100644
--- a/lib/wasm/src/state.rs
+++ b/lib/wasm/src/state.rs
@@ -7,7 +7,7 @@ use cranelift_codegen::ir::{self, Ebb, Inst, Value};
 use environ::{FuncEnvironment, GlobalVariable};
 use std::collections::HashMap;
 use std::vec::Vec;
-use translation_utils::{FunctionIndex, GlobalIndex, MemoryIndex, SignatureIndex};
+use translation_utils::{FunctionIndex, GlobalIndex, MemoryIndex, SignatureIndex, TableIndex};
 
 /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following
 /// fields:
@@ -140,6 +140,9 @@ pub struct TranslationState {
     // Map of heaps that have been created by `FuncEnvironment::make_heap`.
     heaps: HashMap,
 
+    // Map of tables that have been created by `FuncEnvironment::make_table`.
+    tables: HashMap,
+
     // Map of indirect call signatures that have been created by
     // `FuncEnvironment::make_indirect_sig()`.
     // Stores both the signature reference and the number of WebAssembly arguments
@@ -159,6 +162,7 @@ impl TranslationState {
             reachable: true,
             globals: HashMap::new(),
             heaps: HashMap::new(),
+            tables: HashMap::new(),
             signatures: HashMap::new(),
             functions: HashMap::new(),
         }
@@ -303,6 +307,21 @@ impl TranslationState {
             .or_insert_with(|| environ.make_heap(func, index))
     }
 
+    /// Get the `Table` reference that should be used to access table `index`.
+    /// Create the reference if necessary.
+    pub fn get_table(
+        &mut self,
+        func: &mut ir::Function,
+        index: u32,
+        environ: &mut FE,
+    ) -> ir::Table {
+        let index = index as TableIndex;
+        *self
+            .tables
+            .entry(index)
+            .or_insert_with(|| environ.make_table(func, index))
+    }
+
     /// Get the `SigRef` reference that should be used to make an indirect call with signature
     /// `index`. Also return the number of WebAssembly arguments in the signature.
     ///

From 7f530cd7517eb2338a8d4a3412d356bc554d5c7a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 2 Aug 2018 11:12:12 -0700
Subject: [PATCH 1964/3084] Add a langref.rst page that redirects to the new
 ir.rst page.

---
 cranelift/docs/langref.rst | 8 ++++++++
 1 file changed, 8 insertions(+)
 create mode 100644 cranelift/docs/langref.rst

diff --git a/cranelift/docs/langref.rst b/cranelift/docs/langref.rst
new file mode 100644
index 0000000000..904584ab13
--- /dev/null
+++ b/cranelift/docs/langref.rst
@@ -0,0 +1,8 @@
+:orphan:
+
+****************
+Redirection Page
+****************
+
+Cranelift's IR is documented in :doc:`ir`. Please update links to point to
+this new page.

From 5cf2c6f0f75b89a5f9844f8e47b94ac5e557d8b5 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 2 Aug 2018 11:13:25 -0700
Subject: [PATCH 1965/3084] Update cranelift-meta's Cargo.toml to reflect that
 it's not published.

---
 lib/codegen/Cargo.toml      |  2 +-
 lib/codegen/meta/Cargo.toml | 12 ++----------
 2 files changed, 3 insertions(+), 11 deletions(-)

diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index b25fb6f16c..faf6df011c 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -22,7 +22,7 @@ target-lexicon = { version = "0.0.3", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-meta = { path = "meta", version = "0.17.0-alpha" }
+cranelift-meta = { path = "meta", version = "0.0.0" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 40ac0f416a..5803d6e742 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,18 +1,10 @@
 [package]
 name = "cranelift-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.17.0"
+version = "0.0.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
 repository = "https://github.com/CraneStation/cranelift"
 keywords = ["compile", "compiler", "jit"]
-
-[dependencies]
-# It is a goal of the cranelift-codegen crate to have minimal external dependencies.
-# Please don't add any unless they are essential to the task of creating binary
-# machine code.
-
-[badges]
-maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+publish = false

From c5a554db83f6a69ef62cf2707c029f74a7bb0511 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 2 Aug 2018 13:12:07 -0700
Subject: [PATCH 1966/3084] Move the comment about Sphinx 1.4 out of the
 top-level README.

---
 cranelift/README.md    | 3 ---
 cranelift/docs/conf.py | 4 ++--
 2 files changed, 2 insertions(+), 5 deletions(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index b4c42e849b..b84e89bad1 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -132,6 +132,3 @@ generator](https://www.sphinx-doc.org/):
     $ cd cranelift/docs
     $ make html
     $ open _build/html/index.html
-
-We don't support Sphinx versions before 1.4 since the format of index
-tuples has changed.
diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py
index 2ab7d8a8c5..b2987be684 100644
--- a/cranelift/docs/conf.py
+++ b/cranelift/docs/conf.py
@@ -27,8 +27,8 @@ sys.path.insert(0, os.path.abspath('../lib/codegen/meta-python'))
 
 # -- General configuration ------------------------------------------------
 
-# If your documentation needs a minimal Sphinx version, state it here.
-#
+# We don't support Sphinx versions before 1.4 since the format of index
+# tuples has changed.
 needs_sphinx = '1.4'
 
 # Add any Sphinx extension module names here, as strings. They can be

From 570f7bc20b5be5c780a1fbd89dc4063154d75a6f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 2 Aug 2018 15:36:54 -0700
Subject: [PATCH 1967/3084] Rename "cranelift-meta" to just "meta".

This makes it easier for the publish-all.sh script to know to skip this
crate, and it avoids the need to use `extern cranelift_meta as meta`.
---
 lib/codegen/Cargo.toml      | 2 +-
 lib/codegen/build.rs        | 2 +-
 lib/codegen/meta/Cargo.toml | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index faf6df011c..d9094daab1 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -22,7 +22,7 @@ target-lexicon = { version = "0.0.3", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-meta = { path = "meta", version = "0.0.0" }
+meta = { path = "meta", version = "0.0.0" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index dedaa35f8a..5c88f63fa3 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -18,7 +18,7 @@
 // The build script expects to be run from the directory where this build.rs file lives. The
 // current directory is used to find the sources.
 
-extern crate cranelift_meta as meta;
+extern crate meta;
 
 use std::env;
 use std::process;
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 5803d6e742..17dfd1ce7c 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,5 +1,5 @@
 [package]
-name = "cranelift-meta"
+name = "meta"
 authors = ["The Cranelift Project Developers"]
 version = "0.0.0"
 description = "Metaprogram for cranelift-codegen code generator library"

From c4a056a7a087deabe7d22568b2ef7840cb8246db Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 2 Aug 2018 18:34:22 -0700
Subject: [PATCH 1968/3084] Bump version to 0.18.0

---
 cranelift/Cargo.toml     | 24 ++++++++++++------------
 cranelift/publish-all.sh |  2 +-
 lib/codegen/Cargo.toml   |  4 ++--
 lib/entity/Cargo.toml    |  2 +-
 lib/faerie/Cargo.toml    |  6 +++---
 lib/filetests/Cargo.toml |  6 +++---
 lib/frontend/Cargo.toml  |  4 ++--
 lib/module/Cargo.toml    |  6 +++---
 lib/native/Cargo.toml    |  4 ++--
 lib/reader/Cargo.toml    |  4 ++--
 lib/serde/Cargo.toml     |  6 +++---
 lib/simplejit/Cargo.toml |  8 ++++----
 lib/umbrella/Cargo.toml  |  6 +++---
 lib/wasm/Cargo.toml      |  6 +++---
 14 files changed, 44 insertions(+), 44 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 89dc89233c..a56ffbd622 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.17.0"
+version = "0.18.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,17 +14,17 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.17.0" }
-cranelift-reader = { path = "lib/reader", version = "0.17.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.17.0" }
-cranelift-serde = { path = "lib/serde", version = "0.17.0", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.17.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.17.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.17.0" }
-cranelift-module = { path = "lib/module", version = "0.17.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.17.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.17.0" }
-cranelift = { path = "lib/umbrella", version = "0.17.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.18.0" }
+cranelift-reader = { path = "lib/reader", version = "0.18.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.18.0" }
+cranelift-serde = { path = "lib/serde", version = "0.18.0", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.18.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.18.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.18.0" }
+cranelift-module = { path = "lib/module", version = "0.18.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.18.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.18.0" }
+cranelift = { path = "lib/umbrella", version = "0.18.0" }
 filecheck = "0.3.0"
 docopt = "1"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index cedc725aa2..43cc8763a3 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.17.0"
+version="0.18.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index d9094daab1..3cc06a1f62 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.17.0"
+version = "0.18.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.17.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.18.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index 8582e5b82b..d555ff97f6 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.17.0"
+version = "0.18.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 4ceab48af3..de76302466 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.17.0"
+version = "0.18.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0" }
-cranelift-module = { path = "../module", version = "0.17.0" }
+cranelift-codegen = { path = "../codegen", version = "0.18.0" }
+cranelift-module = { path = "../module", version = "0.18.0" }
 faerie = "0.4.4"
 goblin = "0.0.17"
 failure = "0.1.1"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 5f3f70c130..0ef0f4f5b5 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.17.0"
+version = "0.18.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0" }
-cranelift-reader = { path = "../reader", version = "0.17.0" }
+cranelift-codegen = { path = "../codegen", version = "0.18.0" }
+cranelift-reader = { path = "../reader", version = "0.18.0" }
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index d4c4d3d268..6a5336b8c9 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.17.0"
+version = "0.18.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index d46b01bc00..5de44857da 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.17.0"
+version = "0.18.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.17.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.18.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 20c3909d7a..b7959594d8 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.17.0"
+version = "0.18.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -8,7 +8,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 055e041d19..23cbda69fc 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.17.0"
+version = "0.18.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0" }
+cranelift-codegen = { path = "../codegen", version = "0.18.0" }
 target-lexicon = "0.0.3"
 
 [badges]
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index 91bf4b0a65..26fc5cae7f 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.17.0"
+version = "0.18.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -17,8 +17,8 @@ docopt = "1"
 serde = "1.0.8"
 serde_derive = "1.0.8"
 serde_json = "1.0"
-cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
-cranelift-reader = { path = "../reader", version = "0.17.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
+cranelift-reader = { path = "../reader", version = "0.18.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 4577488d2d..750a0d9197 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.17.0"
+version = "0.18.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,9 +9,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.17.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.17.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.18.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.18.0", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index 5f31eba10d..91d58c5897 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.17.0"
+version = "0.18.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,8 +10,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.17.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.18.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index b72cb584ab..1f6e06b0ce 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.17.0"
+version = "0.18.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.2", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.17.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.17.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.18.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From a52c547d0e16121622b7379c383f3209eed67269 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 2 Aug 2018 18:38:30 -0700
Subject: [PATCH 1969/3084] Rename "meta" back to "cranelift-codegen-meta" and
 publish it.

It appears that having the meta directory crate be inside the codegen
directory is not enough to allow codegen to depend on it without it
being published. So, let's just publish it.
---
 lib/codegen/Cargo.toml      |  2 +-
 lib/codegen/build.rs        | 11 +++++------
 lib/codegen/meta/Cargo.toml | 12 +++++++-----
 lib/codegen/meta/README.md  |  2 ++
 4 files changed, 15 insertions(+), 12 deletions(-)
 create mode 100644 lib/codegen/meta/README.md

diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 3cc06a1f62..f49fe9caab 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -22,7 +22,7 @@ target-lexicon = { version = "0.0.3", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-meta = { path = "meta", version = "0.0.0" }
+cranelift-codegen-meta = { path = "meta" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index 5c88f63fa3..92770e9b16 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -18,7 +18,7 @@
 // The build script expects to be run from the directory where this build.rs file lives. The
 // current directory is used to find the sources.
 
-extern crate meta;
+extern crate cranelift_codegen_meta as meta;
 
 use std::env;
 use std::process;
@@ -80,11 +80,10 @@ fn main() {
     // Now that the Python build process is complete, generate files that are
     // emitted by the `meta` crate.
     // ------------------------------------------------------------------------
-    // Temporarily disable this while we work out how to publish this crate.
-    //if let Err(err) = meta::gen_types::generate("new_types.rs", &out_dir) {
-    //    eprintln!("Error: {}", err);
-    //    process::exit(1);
-    //}
+    if let Err(err) = meta::gen_types::generate("new_types.rs", &out_dir) {
+        eprintln!("Error: {}", err);
+        process::exit(1);
+    }
 }
 
 fn identify_python() -> &'static str {
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 17dfd1ce7c..0a9f294642 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,10 +1,12 @@
 [package]
-name = "meta"
+name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.0.0"
+version = "0.18.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
-documentation = "https://cranelift.readthedocs.io/"
 repository = "https://github.com/CraneStation/cranelift"
-keywords = ["compile", "compiler", "jit"]
-publish = false
+readme = "README.md"
+
+[badges]
+maintenance = { status = "experimental" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/codegen/meta/README.md b/lib/codegen/meta/README.md
new file mode 100644
index 0000000000..c0c8648b29
--- /dev/null
+++ b/lib/codegen/meta/README.md
@@ -0,0 +1,2 @@
+This crate contains the metaprogram used by cranelift-codegen. It's not
+useful on its own.

From cd02010a78b1c5b4e8809d70a974d5e0886863c6 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 2 Aug 2018 20:10:23 -0700
Subject: [PATCH 1970/3084] Bump version to 0.18.1

---
 cranelift/Cargo.toml     | 24 ++++++++++++------------
 cranelift/publish-all.sh |  2 +-
 lib/codegen/Cargo.toml   |  4 ++--
 lib/entity/Cargo.toml    |  2 +-
 lib/faerie/Cargo.toml    |  6 +++---
 lib/filetests/Cargo.toml |  6 +++---
 lib/frontend/Cargo.toml  |  4 ++--
 lib/module/Cargo.toml    |  6 +++---
 lib/native/Cargo.toml    |  4 ++--
 lib/reader/Cargo.toml    |  4 ++--
 lib/serde/Cargo.toml     |  6 +++---
 lib/simplejit/Cargo.toml |  8 ++++----
 lib/umbrella/Cargo.toml  |  6 +++---
 lib/wasm/Cargo.toml      |  6 +++---
 14 files changed, 44 insertions(+), 44 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index a56ffbd622..3db93b57c9 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.18.0"
+version = "0.18.1"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,17 +14,17 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.18.0" }
-cranelift-reader = { path = "lib/reader", version = "0.18.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.18.0" }
-cranelift-serde = { path = "lib/serde", version = "0.18.0", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.18.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.18.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.18.0" }
-cranelift-module = { path = "lib/module", version = "0.18.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.18.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.18.0" }
-cranelift = { path = "lib/umbrella", version = "0.18.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.18.1" }
+cranelift-reader = { path = "lib/reader", version = "0.18.1" }
+cranelift-frontend = { path = "lib/frontend", version = "0.18.1" }
+cranelift-serde = { path = "lib/serde", version = "0.18.1", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.18.1", optional = true }
+cranelift-native = { path = "lib/native", version = "0.18.1" }
+cranelift-filetests = { path = "lib/filetests", version = "0.18.1" }
+cranelift-module = { path = "lib/module", version = "0.18.1" }
+cranelift-faerie = { path = "lib/faerie", version = "0.18.1" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.18.1" }
+cranelift = { path = "lib/umbrella", version = "0.18.1" }
 filecheck = "0.3.0"
 docopt = "1"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 43cc8763a3..b41024bec2 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.18.0"
+version="0.18.1"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index f49fe9caab..c81f204ee0 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.18.0"
+version = "0.18.1"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.18.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.18.1", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index d555ff97f6..48e3a9cf57 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.18.0"
+version = "0.18.1"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index de76302466..bb8ffb4c29 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.18.0"
+version = "0.18.1"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.0" }
-cranelift-module = { path = "../module", version = "0.18.0" }
+cranelift-codegen = { path = "../codegen", version = "0.18.1" }
+cranelift-module = { path = "../module", version = "0.18.1" }
 faerie = "0.4.4"
 goblin = "0.0.17"
 failure = "0.1.1"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 0ef0f4f5b5..ac66ca660c 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.18.0"
+version = "0.18.1"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.0" }
-cranelift-reader = { path = "../reader", version = "0.18.0" }
+cranelift-codegen = { path = "../codegen", version = "0.18.1" }
+cranelift-reader = { path = "../reader", version = "0.18.1" }
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 6a5336b8c9..4ef7c4382b 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.18.0"
+version = "0.18.1"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 5de44857da..d8a2bf9ccb 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.18.0"
+version = "0.18.1"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.18.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.18.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index b7959594d8..112899533b 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.18.0"
+version = "0.18.1"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -8,7 +8,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 23cbda69fc..f1e148708a 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.18.0"
+version = "0.18.1"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.0" }
+cranelift-codegen = { path = "../codegen", version = "0.18.1" }
 target-lexicon = "0.0.3"
 
 [badges]
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index 26fc5cae7f..27196ae151 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.18.0"
+version = "0.18.1"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -17,8 +17,8 @@ docopt = "1"
 serde = "1.0.8"
 serde_derive = "1.0.8"
 serde_json = "1.0"
-cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
-cranelift-reader = { path = "../reader", version = "0.18.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
+cranelift-reader = { path = "../reader", version = "0.18.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 750a0d9197..7007b3bbd7 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.18.0"
+version = "0.18.1"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,9 +9,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.18.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.18.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
+cranelift-module = { path = "../module", version = "0.18.1", default-features = false }
+cranelift-native = { path = "../native", version = "0.18.1", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index 91d58c5897..c67c40b594 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.18.0"
+version = "0.18.1"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,8 +10,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.18.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.18.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 1f6e06b0ce..ff1a8fc48b 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.18.0"
+version = "0.18.1"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.2", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.18.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.18.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.18.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From cf2bac139d9238180e2c9b022ca37f0d90671265 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 2 Aug 2018 20:27:31 -0700
Subject: [PATCH 1971/3084] Add codegen/meta to publish-all.sh.

---
 cranelift/publish-all.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index b41024bec2..67311f5f6f 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -36,7 +36,7 @@ cargo update
 echo git commit -a -m "\"Bump version to $version"\"
 echo git push
 for crate in \
-    entity codegen frontend native \
+    entity codegen/meta codegen frontend native \
     reader wasm module simplejit \
     faerie umbrella
 do

From c0af810ec0e384bbf1939f322c4835f68184c556 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 2 Aug 2018 20:37:09 -0700
Subject: [PATCH 1972/3084] Add an explicit version to the
 cranelift-codegen-meta dependency.

This allows it to work in crates.io.
---
 lib/codegen/Cargo.toml      | 2 +-
 lib/codegen/meta/Cargo.toml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index c81f204ee0..bb767dd67f 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -22,7 +22,7 @@ target-lexicon = { version = "0.0.3", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta" }
+cranelift-codegen-meta = { path = "meta", version = "0.18.1" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 0a9f294642..1b25d932ec 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.18.0"
+version = "0.18.1"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"

From 76a7efc8db8458009372ad63370a3eff0f47c978 Mon Sep 17 00:00:00 2001
From: Bruce Mitchener 
Date: Fri, 3 Aug 2018 12:10:51 +0700
Subject: [PATCH 1973/3084] Clippy improvements (#408)

* clippy: Allow subsec_nanos usage for now.

The recommendation from clippy requires Rust 1.27, but we currently
support Rust 1.25 and later.

* Simplify ref pattern matches.

This was recommended by clippy.
---
 lib/codegen/src/lib.rs    | 2 ++
 lib/faerie/src/backend.rs | 6 +++---
 lib/filetests/src/lib.rs  | 6 +++++-
 3 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index 56320c645f..5d24ddf709 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -5,6 +5,8 @@
 #![cfg_attr(feature = "std", warn(unstable_features))]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature="cargo-clippy", allow(
+// This requires Rust 1.27 or later.
+                duration_subsec,
 // Produces only a false positive:
                 while_let_loop,
 // Produces many false positives, but did produce some valid lints, now fixed:
diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index a2ce1e2b4b..4f30406a09 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -362,15 +362,15 @@ impl<'a> RelocSink for FaerieRelocSink<'a> {
         name: &ir::ExternalName,
         addend: Addend,
     ) {
-        let ref_name: String = match name {
-            &ir::ExternalName::User { .. } => {
+        let ref_name: String = match *name {
+            ir::ExternalName::User { .. } => {
                 if self.namespace.is_function(name) {
                     self.namespace.get_function_decl(name).name.clone()
                 } else {
                     self.namespace.get_data_decl(name).name.clone()
                 }
             }
-            &ir::ExternalName::LibCall(ref libcall) => {
+            ir::ExternalName::LibCall(ref libcall) => {
                 let sym = (self.libcall_names)(*libcall);
                 self.artifact
                     .declare(sym.clone(), faerie::Decl::FunctionImport)
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index 2a587f9ea0..cdeac02df9 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -5,7 +5,11 @@
 
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces, unstable_features)]
-#![cfg_attr(feature = "cargo-clippy", allow(type_complexity))]
+#![cfg_attr(feature = "cargo-clippy",
+    allow(
+          type_complexity,
+          // This requires Rust 1.27 or later.
+          duration_subsec))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(

From 217786e9695706f8bece8929db07047a707a9136 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 3 Aug 2018 08:18:29 -0700
Subject: [PATCH 1974/3084] Add a note about the "rustc" tag in the issue
 tracker.

And other minor documentation fixes.
---
 cranelift/rustc.md | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/cranelift/rustc.md b/cranelift/rustc.md
index 0a460da9d8..22d1ce8238 100644
--- a/cranelift/rustc.md
+++ b/cranelift/rustc.md
@@ -18,10 +18,13 @@ compete with LLVM-based release builds in terms of optimization, but for
 some users, completely unoptimized code is too slow to test with, so a
 "pretty good" mode might be good enough.
 
-There's plenty of work to do to achieve these goals, and if achieve
+There's plenty of work to do to achieve these goals, and if we achieve
 them, we'll have enabled a Rust compiler written entirely in Rust, and
 enabled faster Rust compile times for important use cases.
 
+See [issues tagged "rustc"](https://github.com/CraneStation/cranelift/labels/rustc)
+for a list of some of the things that will be needed.
+
 With all that said, there is a potential goal beyond that, which is to
 build a full optimizing release-capable backend. We can't predict how
 far Cranelift will go yet, but we do have some crazy ideas about what
@@ -34,7 +37,7 @@ such a thing might look like, including:
    relationships that Rust provides directly has the potential to be
    very powerful without the need for complex alias analysis logic.
    Unsafe blocks are an interesting challenge, however in many simple
-   cases, like Vec, it may be possible to recover what the optimizer
+   cases, like `Vec`, it may be possible to recover what the optimizer
    needs to know.
  - Design for superoptimization. Traditionally, compiler development
    teams have spent many years of manual effort to identify patterns of

From 9dbfbbde103c455dac03b328587c0834c790515f Mon Sep 17 00:00:00 2001
From: Sergey Pepyakin 
Date: Sat, 4 Aug 2018 16:16:21 +0300
Subject: [PATCH 1975/3084] Stack Limit as an Argument Purpose (#372)

* Initial approach.

* Move stack_limit check before opening the frame

* Account for GPRs and frame pointer in stack check

* Check stack_limit example.

* Remove stack_limit attribute code.

Amends #359

* fmt
---
 cranelift/docs/ir.rst                         | 11 ------
 .../filetests/isa/x86/prologue-epilogue.clif  | 25 ++++++++++++
 cranelift/filetests/parser/preamble.clif      | 18 ---------
 lib/codegen/src/context.rs                    |  4 --
 lib/codegen/src/ir/extfunc.rs                 | 20 +++++++++-
 lib/codegen/src/ir/function.rs                | 14 -------
 lib/codegen/src/isa/x86/abi.rs                | 38 +++++++++++++++++++
 lib/codegen/src/legalizer/boundary.rs         |  9 +++++
 lib/codegen/src/write.rs                      |  5 ---
 lib/reader/src/parser.rs                      | 22 -----------
 10 files changed, 91 insertions(+), 75 deletions(-)
 delete mode 100644 cranelift/filetests/parser/preamble.clif

diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index 777693ea13..65d538d66b 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -546,17 +546,6 @@ instructions before instruction selection::
 When Cranelift code is running in a sandbox, it can also be necessary to include
 stack overflow checks in the prologue.
 
-.. inst:: stack_limit = GV
-
-    Set the stack limit of the current function.
-
-    If set, in the preamble read the stack limit from ``GV`` and compare it to the stack pointer. If
-    the stack pointer has reached or exceeded the limit, generate a trap with a
-    ``stk_ovf`` code.
-
-    Setting `stack_limit` is an alternative way to detect stack overflow, when using
-    a calling convention that doesn't perform stack probes.
-
 Global values
 -------------
 
diff --git a/cranelift/filetests/isa/x86/prologue-epilogue.clif b/cranelift/filetests/isa/x86/prologue-epilogue.clif
index a19af2c09b..f408d1a193 100644
--- a/cranelift/filetests/isa/x86/prologue-epilogue.clif
+++ b/cranelift/filetests/isa/x86/prologue-epilogue.clif
@@ -228,3 +228,28 @@ ebb4:
 ; check: function %divert
 ; check: regmove v5, %rcx -> %rbx
 ; check: [Op1popq#58,%rbx]                   v15 = x86_pop.i64
+
+; Stack limit checking
+
+function %stack_limit(i64 stack_limit) {
+    ss0 = explicit_slot 168
+ebb0(v0: i64):
+    return
+}
+
+; check: function %stack_limit(i64 stack_limit [%rdi], i64 fp [%rbp]) -> i64 fp [%rbp] fast {
+; nextln:     ss0 = explicit_slot 168, offset -184
+; nextln:     ss1 = incoming_arg 16, offset -16
+; nextln: 
+; nextln: ebb0(v0: i64 [%rdi], v4: i64 [%rbp]):
+; nextln:     v1 = copy v0
+; nextln:     v2 = iadd_imm v1, 16
+; nextln:     v3 = ifcmp_sp v2
+; nextln:     trapif uge v3, stk_ovf
+; nextln:     x86_push v4
+; nextln:     copy_special %rsp -> %rbp
+; nextln:     adjust_sp_down_imm 176
+; nextln:     adjust_sp_up_imm 176
+; nextln:     v5 = x86_pop.i64
+; nextln:     return v5
+; nextln: }
diff --git a/cranelift/filetests/parser/preamble.clif b/cranelift/filetests/parser/preamble.clif
deleted file mode 100644
index 00841c0faf..0000000000
--- a/cranelift/filetests/parser/preamble.clif
+++ /dev/null
@@ -1,18 +0,0 @@
-test cat
-
-; Verify parsing of stack_limit.
-function %minimal(i64 vmctx) {
-gv0 = vmctx
-; Stack limit
-stack_limit = gv0
-
-ebb0:
-    trap user0
-}
-; sameln: function %minimal(i64 vmctx) fast {
-; nextln: gv0 = vmctx
-; nextln: stack_limit = gv0
-; nextln: 
-; nextln: ebb0:
-; nextln:     trap user0
-; nextln: }
diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs
index 5573d48e1f..afb2b3ee55 100644
--- a/lib/codegen/src/context.rs
+++ b/lib/codegen/src/context.rs
@@ -297,10 +297,6 @@ impl Context {
 
     /// Insert prologue and epilogues after computing the stack frame layout.
     pub fn prologue_epilogue(&mut self, isa: &TargetIsa) -> CodegenResult<()> {
-        assert!(
-            self.func.stack_limit.is_none(),
-            "stack_limit isn't implemented yet"
-        );
         isa.prologue_epilogue(&mut self.func)?;
         self.verify_if(isa)?;
         self.verify_locations_if(isa)?;
diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs
index 5e002b6a0e..30f0eaab74 100644
--- a/lib/codegen/src/ir/extfunc.rs
+++ b/lib/codegen/src/ir/extfunc.rs
@@ -293,10 +293,25 @@ pub enum ArgumentPurpose {
     /// This is a special-purpose argument used to identify the calling convention expected by the
     /// caller in an indirect call. The callee can verify that the expected signature ID matches.
     SignatureId,
+
+    /// A stack limit pointer.
+    ///
+    /// This is a pointer to a stack limit. It is used to check the current stack pointer
+    /// against. Can only appear once in a signature.
+    StackLimit,
 }
 
 /// Text format names of the `ArgumentPurpose` variants.
-static PURPOSE_NAMES: [&str; 7] = ["normal", "sret", "link", "fp", "csr", "vmctx", "sigid"];
+static PURPOSE_NAMES: [&str; 8] = [
+    "normal",
+    "sret",
+    "link",
+    "fp",
+    "csr",
+    "vmctx",
+    "sigid",
+    "stack_limit",
+];
 
 impl fmt::Display for ArgumentPurpose {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
@@ -315,6 +330,7 @@ impl FromStr for ArgumentPurpose {
             "csr" => Ok(ArgumentPurpose::CalleeSaved),
             "vmctx" => Ok(ArgumentPurpose::VMContext),
             "sigid" => Ok(ArgumentPurpose::SignatureId),
+            "stack_limit" => Ok(ArgumentPurpose::StackLimit),
             _ => Err(()),
         }
     }
@@ -370,6 +386,8 @@ mod tests {
             ArgumentPurpose::FramePointer,
             ArgumentPurpose::CalleeSaved,
             ArgumentPurpose::VMContext,
+            ArgumentPurpose::SignatureId,
+            ArgumentPurpose::StackLimit,
         ];
         for (&e, &n) in all_purpose.iter().zip(PURPOSE_NAMES.iter()) {
             assert_eq!(e.to_string(), n);
diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs
index b6d0c80b43..cf9cbdbe29 100644
--- a/lib/codegen/src/ir/function.rs
+++ b/lib/codegen/src/ir/function.rs
@@ -32,10 +32,6 @@ pub struct Function {
     /// Stack slots allocated in this function.
     pub stack_slots: StackSlots,
 
-    /// If not `None`, represents the address that the stack pointer should
-    /// be checked against.
-    pub stack_limit: Option,
-
     /// Global values referenced.
     pub global_values: PrimaryMap,
 
@@ -82,7 +78,6 @@ impl Function {
             name,
             signature: sig,
             stack_slots: StackSlots::new(),
-            stack_limit: None,
             global_values: PrimaryMap::new(),
             heaps: PrimaryMap::new(),
             tables: PrimaryMap::new(),
@@ -133,15 +128,6 @@ impl Function {
         self.stack_slots.push(data)
     }
 
-    /// Sets the stack limit for the function.
-    ///
-    /// Returns previous one if any.
-    pub fn set_stack_limit(&mut self, stack_limit: Option) -> Option {
-        let prev = self.stack_limit.take();
-        self.stack_limit = stack_limit;
-        prev
-    }
-
     /// Adds a signature which can later be used to declare an external function import.
     pub fn import_signature(&mut self, signature: Signature) -> SigRef {
         self.dfg.signatures.push(signature)
diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs
index 42cacf27a8..d0305d918b 100644
--- a/lib/codegen/src/isa/x86/abi.rs
+++ b/lib/codegen/src/isa/x86/abi.rs
@@ -427,6 +427,21 @@ fn insert_common_prologue(
     csrs: &RegisterSet,
     isa: &TargetIsa,
 ) {
+    if stack_size > 0 {
+        // Check if there is a special stack limit parameter. If so insert stack check.
+        if let Some(stack_limit_arg) = pos.func.special_param(ArgumentPurpose::StackLimit) {
+            // Total stack size is the size of all stack area used by the function, including
+            // pushed CSRs, frame pointer.
+            // Also, the size of a return address, implicitly pushed by a x86 `call` instruction, also
+            // should be accounted for.
+            // TODO: Check if the function body actually contains a `call` instruction.
+            let word_size = isa.pointer_bytes();
+            let total_stack_size = (csrs.iter(GPR).len() + 1 + 1) as i64 * word_size as i64;
+
+            insert_stack_check(pos, total_stack_size, stack_limit_arg);
+        }
+    }
+
     // Append param to entry EBB
     let ebb = pos.current_ebb().expect("missing ebb under cursor");
     let fp = pos.func.dfg.append_ebb_param(ebb, reg_type);
@@ -493,6 +508,29 @@ fn insert_common_prologue(
     }
 }
 
+/// Insert a check that generates a trap if the stack pointer goes
+/// below a value in `stack_limit_arg`.
+fn insert_stack_check(pos: &mut EncCursor, stack_size: i64, stack_limit_arg: ir::Value) {
+    use ir::condcodes::IntCC;
+
+    // Copy `stack_limit_arg` into a %rax and use it for calculating
+    // a SP threshold.
+    let stack_limit_copy = pos.ins().copy(stack_limit_arg);
+    pos.func.locations[stack_limit_copy] = ir::ValueLoc::Reg(RU::rax as RegUnit);
+    let sp_threshold = pos.ins().iadd_imm(stack_limit_copy, stack_size);
+    pos.func.locations[sp_threshold] = ir::ValueLoc::Reg(RU::rax as RegUnit);
+
+    // If the stack pointer currently reaches the SP threshold or below it then after opening
+    // the current stack frame, the current stack pointer will reach the limit.
+    let cflags = pos.ins().ifcmp_sp(sp_threshold);
+    pos.func.locations[cflags] = ir::ValueLoc::Reg(RU::rflags as RegUnit);
+    pos.ins().trapif(
+        IntCC::UnsignedGreaterThanOrEqual,
+        cflags,
+        ir::TrapCode::StackOverflow,
+    );
+}
+
 /// Find all `return` instructions and insert epilogues before them.
 fn insert_common_epilogues(
     pos: &mut EncCursor,
diff --git a/lib/codegen/src/legalizer/boundary.rs b/lib/codegen/src/legalizer/boundary.rs
index d89a455454..640c3a82b0 100644
--- a/lib/codegen/src/legalizer/boundary.rs
+++ b/lib/codegen/src/legalizer/boundary.rs
@@ -73,6 +73,7 @@ fn legalize_entry_params(func: &mut Function, entry: Ebb) {
     let mut has_link = false;
     let mut has_vmctx = false;
     let mut has_sigid = false;
+    let mut has_stack_limit = false;
 
     // Insert position for argument conversion code.
     // We want to insert instructions before the first instruction in the entry block.
@@ -111,6 +112,10 @@ fn legalize_entry_params(func: &mut Function, entry: Ebb) {
                     debug_assert!(!has_sigid, "Multiple sigid arguments found");
                     has_sigid = true;
                 }
+                ArgumentPurpose::StackLimit => {
+                    debug_assert!(!has_stack_limit, "Multiple stack_limit arguments found");
+                    has_stack_limit = true;
+                }
                 _ => panic!("Unexpected special-purpose arg {}", abi_type),
             }
             abi_arg += 1;
@@ -167,6 +172,10 @@ fn legalize_entry_params(func: &mut Function, entry: Ebb) {
                 debug_assert!(!has_sigid, "Multiple sigid parameters found");
                 has_sigid = true;
             }
+            ArgumentPurpose::StackLimit => {
+                debug_assert!(!has_stack_limit, "Multiple stack_limit parameters found");
+                has_stack_limit = true;
+            }
         }
 
         // Just create entry block values to match here. We will use them in `handle_return_abi()`
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index dc9282a07e..d4d98b9ced 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -136,11 +136,6 @@ fn write_preamble(
         writeln!(w, "    {} = {}", jt, jt_data)?;
     }
 
-    if let Some(stack_limit) = func.stack_limit {
-        any = true;
-        writeln!(w, "    stack_limit = {}", stack_limit)?;
-    }
-
     Ok(any)
 }
 
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 62154a6944..7959d0688a 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -274,15 +274,6 @@ impl<'a> Context<'a> {
         }
     }
 
-    // Assign the global for the stack limit.
-    fn set_stack_limit(&mut self, gv: GlobalValue, loc: Location) -> ParseResult<()> {
-        if self.function.set_stack_limit(Some(gv)).is_some() {
-            err!(loc, "multiple stack_limit declarations")
-        } else {
-            Ok(())
-        }
-    }
-
     // Allocate a new EBB.
     fn add_ebb(&mut self, ebb: Ebb, loc: Location) -> ParseResult {
         self.map.def_ebb(ebb, loc)?;
@@ -1032,7 +1023,6 @@ impl<'a> Parser<'a> {
     //                   * function-decl
     //                   * signature-decl
     //                   * jump-table-decl
-    //                   * stack-limit-decl
     //
     // The parsed decls are added to `ctx` rather than returned.
     fn parse_preamble(&mut self, ctx: &mut Context) -> ParseResult<()> {
@@ -1074,9 +1064,6 @@ impl<'a> Parser<'a> {
                     self.parse_jump_table_decl()
                         .and_then(|(jt, dat)| ctx.add_jt(jt, dat, self.loc))
                 }
-                Some(Token::Identifier("stack_limit")) => self
-                    .parse_stack_limit_decl()
-                    .and_then(|gv| ctx.set_stack_limit(gv, self.loc)),
                 // More to come..
                 _ => return Ok(()),
             }?;
@@ -1414,15 +1401,6 @@ impl<'a> Parser<'a> {
         }
     }
 
-    /// stack-limit-decl ::= "stack_limit" "=" GlobalValue(gv)
-    fn parse_stack_limit_decl(&mut self) -> ParseResult {
-        self.consume();
-        self.match_token(Token::Equal, "expected '=' in stack limit declaration")?;
-        let gv = self.match_gv("expected global value")?;
-
-        Ok(gv)
-    }
-
     // Parse a function body, add contents to `ctx`.
     //
     // function-body ::= * { extended-basic-block }

From f7e481d9aca4c7d7f164858f61ed1470e6f67351 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 27 Jul 2018 19:11:39 +0200
Subject: [PATCH 1976/3084] Implement wasm saturating conversions;

---
 cranelift/filetests/wasm/conversions.clif    |  48 ++++
 lib/codegen/meta-python/base/instructions.py |  15 ++
 lib/codegen/meta-python/isa/x86/legalize.py  |   4 +-
 lib/codegen/src/isa/x86/enc_tables.rs        | 243 +++++++++++++++++--
 lib/wasm/src/code_translator.rs              |  26 +-
 5 files changed, 305 insertions(+), 31 deletions(-)

diff --git a/cranelift/filetests/wasm/conversions.clif b/cranelift/filetests/wasm/conversions.clif
index 0b7630da0d..6784637136 100644
--- a/cranelift/filetests/wasm/conversions.clif
+++ b/cranelift/filetests/wasm/conversions.clif
@@ -69,6 +69,54 @@ ebb0(v0: f64):
     return v1
 }
 
+function %i32_trunc_s_sat_f32(f32) -> i32 {
+ebb0(v0: f32):
+    v1 = fcvt_to_sint_sat.i32 v0
+    return v1
+}
+
+function %i32_trunc_u_sat_f32(f32) -> i32 {
+ebb0(v0: f32):
+    v1 = fcvt_to_uint_sat.i32 v0
+    return v1
+}
+
+function %i32_trunc_s_sat_f64(f64) -> i32 {
+ebb0(v0: f64):
+    v1 = fcvt_to_sint_sat.i32 v0
+    return v1
+}
+
+function %i32_trunc_u_sat_f64(f64) -> i32 {
+ebb0(v0: f64):
+    v1 = fcvt_to_uint_sat.i32 v0
+    return v1
+}
+
+function %i64_trunc_s_sat_f32(f32) -> i64 {
+ebb0(v0: f32):
+    v1 = fcvt_to_sint_sat.i64 v0
+    return v1
+}
+
+function %i64_trunc_u_sat_f32(f32) -> i64 {
+ebb0(v0: f32):
+    v1 = fcvt_to_uint_sat.i64 v0
+    return v1
+}
+
+function %i64_trunc_s_sat_f64(f64) -> i64 {
+ebb0(v0: f64):
+    v1 = fcvt_to_sint_sat.i64 v0
+    return v1
+}
+
+function %i64_trunc_u_sat_f64(f64) -> i64 {
+ebb0(v0: f64):
+    v1 = fcvt_to_uint_sat.i64 v0
+    return v1
+}
+
 function %f32_trunc_f64(f64) -> f32 {
 ebb0(v0: f64):
     v1 = fdemote.f32 v0
diff --git a/lib/codegen/meta-python/base/instructions.py b/lib/codegen/meta-python/base/instructions.py
index 84c704f888..25b8ba414b 100644
--- a/lib/codegen/meta-python/base/instructions.py
+++ b/lib/codegen/meta-python/base/instructions.py
@@ -1879,6 +1879,14 @@ fcvt_to_uint = Instruction(
         """,
         ins=x, outs=a, can_trap=True)
 
+fcvt_to_uint_sat = Instruction(
+        'fcvt_to_uint_sat', r"""
+        Convert floating point to unsigned integer as fcvt_to_uint does, but
+        saturates the input instead of trapping. NaN and negative values are
+        converted to 0.
+        """,
+        ins=x, outs=a)
+
 fcvt_to_sint = Instruction(
         'fcvt_to_sint', r"""
         Convert floating point to signed integer.
@@ -1891,6 +1899,13 @@ fcvt_to_sint = Instruction(
         """,
         ins=x, outs=a, can_trap=True)
 
+fcvt_to_sint_sat = Instruction(
+        'fcvt_to_sint_sat', r"""
+        Convert floating point to signed integer as fcvt_to_sint does, but
+        saturates the input instead of trapping. NaN values are converted to 0.
+        """,
+        ins=x, outs=a)
+
 x = Operand('x', Int)
 a = Operand('a', FloatTo)
 
diff --git a/lib/codegen/meta-python/isa/x86/legalize.py b/lib/codegen/meta-python/isa/x86/legalize.py
index 6da7d53340..15f08a07b9 100644
--- a/lib/codegen/meta-python/isa/x86/legalize.py
+++ b/lib/codegen/meta-python/isa/x86/legalize.py
@@ -94,9 +94,11 @@ x86_expand.custom_legalize(insts.fmax, 'expand_minmax')
 
 # Conversions from unsigned need special handling.
 x86_expand.custom_legalize(insts.fcvt_from_uint, 'expand_fcvt_from_uint')
-# Conversions from float to int can trap.
+# Conversions from float to int can trap and modify the control flow graph.
 x86_expand.custom_legalize(insts.fcvt_to_sint, 'expand_fcvt_to_sint')
 x86_expand.custom_legalize(insts.fcvt_to_uint, 'expand_fcvt_to_uint')
+x86_expand.custom_legalize(insts.fcvt_to_sint_sat, 'expand_fcvt_to_sint_sat')
+x86_expand.custom_legalize(insts.fcvt_to_uint_sat, 'expand_fcvt_to_uint_sat')
 
 # Count leading and trailing zeroes, for baseline x86_64
 c_minus_one = Var('c_minus_one')
diff --git a/lib/codegen/src/isa/x86/enc_tables.rs b/lib/codegen/src/isa/x86/enc_tables.rs
index 03c33f52c8..52af977ad6 100644
--- a/lib/codegen/src/isa/x86/enc_tables.rs
+++ b/lib/codegen/src/isa/x86/enc_tables.rs
@@ -322,14 +322,13 @@ fn expand_fcvt_to_sint(
     use ir::condcodes::{FloatCC, IntCC};
     use ir::immediates::{Ieee32, Ieee64};
 
-    let x;
-    match func.dfg[inst] {
+    let x = match func.dfg[inst] {
         ir::InstructionData::Unary {
             opcode: ir::Opcode::FcvtToSint,
             arg,
-        } => x = arg,
+        } => arg,
         _ => panic!("Need fcvt_to_sint: {}", func.dfg.display_inst(inst, None)),
-    }
+    };
     let old_ebb = func.layout.pp_ebb(inst);
     let xty = func.dfg.value_type(x);
     let result = func.dfg.first_result(inst);
@@ -367,15 +366,16 @@ fn expand_fcvt_to_sint(
     let mut overflow_cc = FloatCC::LessThan;
     let output_bits = ty.lane_bits();
     let flimit = match xty {
-        // An f32 can represent `i16::min_value() - 1` exactly with precision to spare, so
-        // there are values less than -2^(N-1) that convert correctly to INT_MIN.
-        ir::types::F32 => pos.ins().f32const(if output_bits < 32 {
-            overflow_cc = FloatCC::LessThanOrEqual;
-            Ieee32::fcvt_to_sint_negative_overflow(output_bits)
-        } else {
-            Ieee32::pow2(output_bits - 1).neg()
-        }),
-        ir::types::F64 => {
+        ir::types::F32 =>
+            // An f32 can represent `i16::min_value() - 1` exactly with precision to spare, so
+            // there are values less than -2^(N-1) that convert correctly to INT_MIN.
+            pos.ins().f32const(if output_bits < 32 {
+                overflow_cc = FloatCC::LessThanOrEqual;
+                Ieee32::fcvt_to_sint_negative_overflow(output_bits)
+            } else {
+                Ieee32::pow2(output_bits - 1).neg()
+            }),
+        ir::types::F64 =>
             // An f64 can represent `i32::min_value() - 1` exactly with precision to spare, so
             // there are values less than -2^(N-1) that convert correctly to INT_MIN.
             pos.ins().f64const(if output_bits < 64 {
@@ -383,8 +383,7 @@ fn expand_fcvt_to_sint(
                 Ieee64::fcvt_to_sint_negative_overflow(output_bits)
             } else {
                 Ieee64::pow2(output_bits - 1).neg()
-            })
-        }
+            }),
         _ => panic!("Can't convert {}", xty),
     };
     let overflow = pos.ins().fcmp(overflow_cc, x, flimit);
@@ -406,6 +405,122 @@ fn expand_fcvt_to_sint(
     cfg.recompute_ebb(pos.func, done);
 }
 
+fn expand_fcvt_to_sint_sat(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    cfg: &mut ControlFlowGraph,
+    _isa: &isa::TargetIsa,
+) {
+    use ir::condcodes::{FloatCC, IntCC};
+    use ir::immediates::{Ieee32, Ieee64};
+
+    let x = match func.dfg[inst] {
+        ir::InstructionData::Unary {
+            opcode: ir::Opcode::FcvtToSintSat,
+            arg,
+        } => arg,
+        _ => panic!(
+            "Need fcvt_to_sint_sat: {}",
+            func.dfg.display_inst(inst, None)
+        ),
+    };
+
+    let old_ebb = func.layout.pp_ebb(inst);
+    let xty = func.dfg.value_type(x);
+    let result = func.dfg.first_result(inst);
+    let ty = func.dfg.value_type(result);
+
+    // Final EBB after the bad value checks.
+    let done_ebb = func.dfg.make_ebb();
+    func.dfg.clear_results(inst);
+    func.dfg.attach_ebb_param(done_ebb, result);
+
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+    pos.use_srcloc(inst);
+
+    // The `x86_cvtt2si` performs the desired conversion, but it doesn't trap on NaN or
+    // overflow. It produces an INT_MIN result instead.
+    let cvtt2si = pos.ins().x86_cvtt2si(ty, x);
+
+    let is_done = pos
+        .ins()
+        .icmp_imm(IntCC::NotEqual, cvtt2si, 1 << (ty.lane_bits() - 1));
+    pos.ins().brnz(is_done, done_ebb, &[cvtt2si]);
+
+    // We now have the following possibilities:
+    //
+    // 1. INT_MIN was actually the correct conversion result.
+    // 2. The input was NaN -> replace the result value with 0.
+    // 3. The input was out of range -> saturate the result to the min/max value.
+
+    // Check for NaN, which is truncated to 0.
+    let zero = pos.ins().iconst(ty, 0);
+    let is_nan = pos.ins().fcmp(FloatCC::Unordered, x, x);
+    pos.ins().brnz(is_nan, done_ebb, &[zero]);
+
+    // Check for case 1: INT_MIN is the correct result.
+    // Determine the smallest floating point number that would convert to INT_MIN.
+    let mut overflow_cc = FloatCC::LessThan;
+    let output_bits = ty.lane_bits();
+    let flimit = match xty {
+        ir::types::F32 =>
+            // An f32 can represent `i16::min_value() - 1` exactly with precision to spare, so
+            // there are values less than -2^(N-1) that convert correctly to INT_MIN.
+            pos.ins().f32const(if output_bits < 32 {
+                overflow_cc = FloatCC::LessThanOrEqual;
+                Ieee32::fcvt_to_sint_negative_overflow(output_bits)
+            } else {
+                Ieee32::pow2(output_bits - 1).neg()
+            }),
+        ir::types::F64 =>
+            // An f64 can represent `i32::min_value() - 1` exactly with precision to spare, so
+            // there are values less than -2^(N-1) that convert correctly to INT_MIN.
+            pos.ins().f64const(if output_bits < 64 {
+                overflow_cc = FloatCC::LessThanOrEqual;
+                Ieee64::fcvt_to_sint_negative_overflow(output_bits)
+            } else {
+                Ieee64::pow2(output_bits - 1).neg()
+            }),
+        _ => panic!("Can't convert {}", xty),
+    };
+
+    let overflow = pos.ins().fcmp(overflow_cc, x, flimit);
+    let min_imm = match ty {
+        ir::types::I32 => i32::min_value() as i64,
+        ir::types::I64 => i64::min_value(),
+        _ => panic!("Don't know the min value for {}", ty),
+    };
+    let min_value = pos.ins().iconst(ty, min_imm);
+    pos.ins().brnz(overflow, done_ebb, &[min_value]);
+
+    // Finally, we could have a positive value that is too large.
+    let fzero = match xty {
+        ir::types::F32 => pos.ins().f32const(Ieee32::with_bits(0)),
+        ir::types::F64 => pos.ins().f64const(Ieee64::with_bits(0)),
+        _ => panic!("Can't convert {}", xty),
+    };
+
+    let max_imm = match ty {
+        ir::types::I32 => i32::max_value() as i64,
+        ir::types::I64 => i64::max_value(),
+        _ => panic!("Don't know the max value for {}", ty),
+    };
+    let max_value = pos.ins().iconst(ty, max_imm);
+
+    let overflow = pos.ins().fcmp(FloatCC::GreaterThanOrEqual, x, fzero);
+    pos.ins().brnz(overflow, done_ebb, &[max_value]);
+
+    // Recycle the original instruction.
+    pos.func.dfg.replace(inst).jump(done_ebb, &[cvtt2si]);
+
+    // Finally insert a label for the completion.
+    pos.next_inst();
+    pos.insert_ebb(done_ebb);
+
+    cfg.recompute_ebb(pos.func, old_ebb);
+    cfg.recompute_ebb(pos.func, done_ebb);
+}
+
 fn expand_fcvt_to_uint(
     inst: ir::Inst,
     func: &mut ir::Function,
@@ -415,14 +530,14 @@ fn expand_fcvt_to_uint(
     use ir::condcodes::{FloatCC, IntCC};
     use ir::immediates::{Ieee32, Ieee64};
 
-    let x;
-    match func.dfg[inst] {
+    let x = match func.dfg[inst] {
         ir::InstructionData::Unary {
             opcode: ir::Opcode::FcvtToUint,
             arg,
-        } => x = arg,
+        } => arg,
         _ => panic!("Need fcvt_to_uint: {}", func.dfg.display_inst(inst, None)),
-    }
+    };
+
     let old_ebb = func.layout.pp_ebb(inst);
     let xty = func.dfg.value_type(x);
     let result = func.dfg.first_result(inst);
@@ -487,3 +602,93 @@ fn expand_fcvt_to_uint(
     cfg.recompute_ebb(pos.func, large);
     cfg.recompute_ebb(pos.func, done);
 }
+
+fn expand_fcvt_to_uint_sat(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    cfg: &mut ControlFlowGraph,
+    _isa: &isa::TargetIsa,
+) {
+    use ir::condcodes::{FloatCC, IntCC};
+    use ir::immediates::{Ieee32, Ieee64};
+
+    let x = match func.dfg[inst] {
+        ir::InstructionData::Unary {
+            opcode: ir::Opcode::FcvtToUintSat,
+            arg,
+        } => arg,
+        _ => panic!(
+            "Need fcvt_to_uint_sat: {}",
+            func.dfg.display_inst(inst, None)
+        ),
+    };
+
+    let old_ebb = func.layout.pp_ebb(inst);
+    let xty = func.dfg.value_type(x);
+    let result = func.dfg.first_result(inst);
+    let ty = func.dfg.value_type(result);
+
+    // EBB handling numbers >= 2^(N-1).
+    let large = func.dfg.make_ebb();
+
+    // Final EBB after the bad value checks.
+    let done = func.dfg.make_ebb();
+
+    // Move the `inst` result value onto the `done` EBB.
+    func.dfg.clear_results(inst);
+    func.dfg.attach_ebb_param(done, result);
+
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+    pos.use_srcloc(inst);
+
+    // Start by materializing the floating point constant 2^(N-1) where N is the number of bits in
+    // the destination integer type.
+    let pow2nm1 = match xty {
+        ir::types::F32 => pos.ins().f32const(Ieee32::pow2(ty.lane_bits() - 1)),
+        ir::types::F64 => pos.ins().f64const(Ieee64::pow2(ty.lane_bits() - 1)),
+        _ => panic!("Can't convert {}", xty),
+    };
+    let zero = pos.ins().iconst(ty, 0);
+    let is_large = pos.ins().ffcmp(x, pow2nm1);
+    pos.ins()
+        .brff(FloatCC::GreaterThanOrEqual, is_large, large, &[]);
+
+    // We need to generate zero when `x` is NaN, so reuse the flags from the previous comparison.
+    pos.ins().brff(FloatCC::Unordered, is_large, done, &[zero]);
+
+    // Now we know that x < 2^(N-1) and not NaN. If the result of the cvtt2si is positive, we're
+    // done; otherwise saturate to the minimum unsigned value, that is 0.
+    let sres = pos.ins().x86_cvtt2si(ty, x);
+    let is_neg = pos.ins().ifcmp_imm(sres, 0);
+    pos.ins()
+        .brif(IntCC::SignedGreaterThanOrEqual, is_neg, done, &[sres]);
+    pos.ins().jump(done, &[zero]);
+
+    // Handle the case where x >= 2^(N-1) and not NaN.
+    pos.insert_ebb(large);
+    let adjx = pos.ins().fsub(x, pow2nm1);
+    let lres = pos.ins().x86_cvtt2si(ty, adjx);
+    let max_value = pos.ins().iconst(
+        ty,
+        match ty {
+            ir::types::I32 => u32::max_value() as i64,
+            ir::types::I64 => u64::max_value() as i64,
+            _ => panic!("Can't convert {}", ty),
+        },
+    );
+    let is_neg = pos.ins().ifcmp_imm(lres, 0);
+    pos.ins()
+        .brif(IntCC::SignedLessThan, is_neg, done, &[max_value]);
+    let lfinal = pos.ins().iadd_imm(lres, 1 << (ty.lane_bits() - 1));
+
+    // Recycle the original instruction as a jump.
+    pos.func.dfg.replace(inst).jump(done, &[lfinal]);
+
+    // Finally insert a label for the completion.
+    pos.next_inst();
+    pos.insert_ebb(done);
+
+    cfg.recompute_ebb(pos.func, old_ebb);
+    cfg.recompute_ebb(pos.func, large);
+    cfg.recompute_ebb(pos.func, done);
+}
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index dacae0a2c9..e158b76c08 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -631,17 +631,21 @@ pub fn translate_operator(
             let val = state.pop1();
             state.push1(builder.ins().fcvt_to_uint(I32, val));
         }
-        Operator::I64TruncSSatF64
-        | Operator::I64TruncSSatF32
-        | Operator::I32TruncSSatF64
-        | Operator::I32TruncSSatF32
-        | Operator::I64TruncUSatF64
-        | Operator::I64TruncUSatF32
-        | Operator::I32TruncUSatF64
-        | Operator::I32TruncUSatF32 => {
-            return Err(WasmError::Unsupported(
-                "proposed saturating conversion operators",
-            ));
+        Operator::I64TruncSSatF64 | Operator::I64TruncSSatF32 => {
+            let val = state.pop1();
+            state.push1(builder.ins().fcvt_to_sint_sat(I64, val));
+        }
+        Operator::I32TruncSSatF64 | Operator::I32TruncSSatF32 => {
+            let val = state.pop1();
+            state.push1(builder.ins().fcvt_to_sint_sat(I32, val));
+        }
+        Operator::I64TruncUSatF64 | Operator::I64TruncUSatF32 => {
+            let val = state.pop1();
+            state.push1(builder.ins().fcvt_to_uint_sat(I64, val));
+        }
+        Operator::I32TruncUSatF64 | Operator::I32TruncUSatF32 => {
+            let val = state.pop1();
+            state.push1(builder.ins().fcvt_to_uint_sat(I32, val));
         }
         Operator::F32ReinterpretI32 => {
             let val = state.pop1();

From f5d46cabe73097ed0bdd19b779e95e539ebca89e Mon Sep 17 00:00:00 2001
From: Maddy 
Date: Thu, 2 Aug 2018 01:29:47 +0000
Subject: [PATCH 1977/3084] Use types to distinguish between wasm function body
 indices and wasm function indices.

---
 cranelift/Cargo.toml                |  1 +
 cranelift/src/clif-util.rs          |  1 +
 cranelift/src/wasm.rs               | 21 ++++++++--------
 lib/wasm/Cargo.toml                 |  1 +
 lib/wasm/src/code_translator.rs     |  5 ++--
 lib/wasm/src/environ/dummy.rs       | 37 +++++++++++++++--------------
 lib/wasm/src/environ/spec.rs        | 16 ++++++-------
 lib/wasm/src/lib.rs                 |  4 +++-
 lib/wasm/src/sections_translator.rs | 17 ++++++-------
 lib/wasm/src/state.rs               |  7 +++---
 lib/wasm/src/translation_utils.rs   | 12 ++++++++--
 lib/wasm/tests/wasm_testsuite.rs    |  2 +-
 12 files changed, 71 insertions(+), 53 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 3db93b57c9..b674fbf7f9 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -15,6 +15,7 @@ path = "src/clif-util.rs"
 [dependencies]
 cfg-if = "0.1"
 cranelift-codegen = { path = "lib/codegen", version = "0.18.1" }
+cranelift-entity = { path = "lib/entity", version = "0.18.1" }
 cranelift-reader = { path = "lib/reader", version = "0.18.1" }
 cranelift-frontend = { path = "lib/frontend", version = "0.18.1" }
 cranelift-serde = { path = "lib/serde", version = "0.18.1", optional = true }
diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 8df6a603d7..0f7edd4ce3 100644
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -23,6 +23,7 @@ extern crate term;
 
 cfg_if! {
     if #[cfg(feature = "wasm")] {
+        extern crate cranelift_entity;
         extern crate cranelift_wasm;
         extern crate wabt;
         mod wasm;
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 72fcb0d729..d448f75389 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -7,7 +7,8 @@
 use cranelift_codegen::Context;
 use cranelift_codegen::print_errors::{pretty_error, pretty_verifier_error};
 use cranelift_codegen::settings::FlagsOrIsa;
-use cranelift_wasm::{translate_module, DummyEnvironment, ModuleEnvironment};
+use cranelift_entity::EntityRef;
+use cranelift_wasm::{translate_module, DummyEnvironment, FuncIndex, ModuleEnvironment};
 use std::error::Error;
 use std::path::Path;
 use std::path::PathBuf;
@@ -107,17 +108,17 @@ fn handle_module(
         }
 
         let num_func_imports = dummy_environ.get_num_func_imports();
-        for (def_index, func) in dummy_environ.info.function_bodies.iter().enumerate() {
-            let func_index = num_func_imports + def_index;
+        for (def_index, func) in dummy_environ.info.function_bodies.iter() {
+            let func_index = num_func_imports + def_index.index();
             let mut context = Context::new();
             context.func = func.clone();
             if let Some(start_func) = dummy_environ.info.start_func {
-                if func_index == start_func {
+                if func_index == start_func.index() {
                     println!("; Selected as wasm start function");
                 }
             }
             vprintln!(flag_verbose, "");
-            for export_name in &dummy_environ.info.functions[func_index].export_names {
+            for export_name in &dummy_environ.info.functions[FuncIndex::new(func_index)].export_names {
                 println!("; Exported as \"{}\"", export_name);
             }
             println!("{}", context.func.display(None));
@@ -142,10 +143,10 @@ fn handle_module(
     let num_func_imports = dummy_environ.get_num_func_imports();
     let mut total_module_code_size = 0;
     let mut context = Context::new();
-    for (def_index, func) in dummy_environ.info.function_bodies.iter().enumerate() {
+    for (def_index, func) in dummy_environ.info.function_bodies.iter() {
         context.func = func.clone();
 
-        let func_index = num_func_imports + def_index;
+        let func_index = num_func_imports + def_index.index();
         if flag_check_translation {
             context
                 .verify(fisa)
@@ -162,7 +163,7 @@ fn handle_module(
                 total_module_code_size += compiled_size;
                 println!(
                     "Function #{} bytecode size: {} bytes",
-                    func_index, dummy_environ.func_bytecode_sizes[def_index]
+                    func_index, dummy_environ.func_bytecode_sizes[def_index.index()]
                 );
             }
         }
@@ -170,11 +171,11 @@ fn handle_module(
         if flag_print {
             vprintln!(flag_verbose, "");
             if let Some(start_func) = dummy_environ.info.start_func {
-                if func_index == start_func {
+                if func_index == start_func.index() {
                     println!("; Selected as wasm start function");
                 }
             }
-            for export_name in &dummy_environ.info.functions[func_index].export_names {
+            for export_name in &dummy_environ.info.functions[FuncIndex::new(func_index)].export_names {
                 println!("; Exported as \"{}\"", export_name);
             }
             println!("{}", context.func.display(fisa.isa));
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index ff1a8fc48b..abc33510a1 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -11,6 +11,7 @@ keywords = ["webassembly", "wasm"]
 [dependencies]
 wasmparser = { version = "0.17.2", default-features = false }
 cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.18.1", default-features = false }
 cranelift-frontend = { path = "../frontend", version = "0.18.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index e158b76c08..7ab5f00cb6 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -26,6 +26,7 @@ use cranelift_codegen::ir::condcodes::{FloatCC, IntCC};
 use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags};
 use cranelift_codegen::packed_option::ReservedValue;
+use cranelift_entity::EntityRef;
 use cranelift_frontend::{FunctionBuilder, Variable};
 use environ::{FuncEnvironment, GlobalVariable, WasmError, WasmResult};
 use state::{ControlStackFrame, TranslationState};
@@ -33,7 +34,7 @@ use std::collections::{hash_map, HashMap};
 use std::vec::Vec;
 use std::{i32, u32};
 use translation_utils::{f32_translation, f64_translation, num_return_values, type_to_type};
-use translation_utils::{FunctionIndex, MemoryIndex, SignatureIndex, TableIndex};
+use translation_utils::{FuncIndex, MemoryIndex, SignatureIndex, TableIndex};
 use wasmparser::{MemoryImmediate, Operator};
 
 // Clippy warns about "flags: _" but its important to document that the flags field is ignored
@@ -358,7 +359,7 @@ pub fn translate_operator(
             let (fref, num_args) = state.get_direct_func(builder.func, function_index, environ);
             let call = environ.translate_call(
                 builder.cursor(),
-                function_index as FunctionIndex,
+                FuncIndex::new(function_index as usize),
                 fref,
                 state.peekn(num_args),
             )?;
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 1fdbbe7df3..b2db269e24 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -5,19 +5,20 @@ use cranelift_codegen::ir::immediates::Imm64;
 use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::settings;
+use cranelift_entity::{EntityRef, PrimaryMap};
 use environ::{FuncEnvironment, GlobalVariable, ModuleEnvironment, WasmResult};
 use func_translator::FuncTranslator;
 use std::string::String;
 use std::vec::Vec;
 use target_lexicon::Triple;
 use translation_utils::{
-    FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
+    DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
 };
 use wasmparser;
 
 /// Compute a `ir::ExternalName` for a given wasm function index.
-fn get_func_name(func_index: FunctionIndex) -> ir::ExternalName {
-    ir::ExternalName::user(0, func_index as u32)
+fn get_func_name(func_index: FuncIndex) -> ir::ExternalName {
+    ir::ExternalName::user(0, func_index.index() as u32)
 }
 
 /// A collection of names under which a given entity is exported.
@@ -55,10 +56,10 @@ pub struct DummyModuleInfo {
     pub imported_funcs: Vec<(String, String)>,
 
     /// Functions, imported and local.
-    pub functions: Vec>,
+    pub functions: PrimaryMap>,
 
     /// Function bodies.
-    pub function_bodies: Vec,
+    pub function_bodies: PrimaryMap,
 
     /// Tables as provided by `declare_table`.
     pub tables: Vec>,
@@ -70,7 +71,7 @@ pub struct DummyModuleInfo {
     pub globals: Vec>,
 
     /// The start function.
-    pub start_func: Option,
+    pub start_func: Option,
 }
 
 impl DummyModuleInfo {
@@ -81,8 +82,8 @@ impl DummyModuleInfo {
             flags,
             signatures: Vec::new(),
             imported_funcs: Vec::new(),
-            functions: Vec::new(),
-            function_bodies: Vec::new(),
+            functions: PrimaryMap::new(),
+            function_bodies: PrimaryMap::new(),
             tables: Vec::new(),
             memories: Vec::new(),
             globals: Vec::new(),
@@ -215,7 +216,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         func.import_signature(self.vmctx_sig(index))
     }
 
-    fn make_direct_func(&mut self, func: &mut ir::Function, index: FunctionIndex) -> ir::FuncRef {
+    fn make_direct_func(&mut self, func: &mut ir::Function, index: FuncIndex) -> ir::FuncRef {
         let sigidx = self.mod_info.functions[index].entity;
         // A real implementation would probably add a `vmctx` argument.
         // And maybe attempt some signature de-duplication.
@@ -275,7 +276,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
     fn translate_call(
         &mut self,
         mut pos: FuncCursor,
-        _callee_index: FunctionIndex,
+        _callee_index: FuncIndex,
         callee: ir::FuncRef,
         call_args: &[ir::Value],
     ) -> WasmResult {
@@ -319,7 +320,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         &self.info.flags
     }
 
-    fn get_func_name(&self, func_index: FunctionIndex) -> ir::ExternalName {
+    fn get_func_name(&self, func_index: FuncIndex) -> ir::ExternalName {
         get_func_name(func_index)
     }
 
@@ -356,7 +357,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         self.info.functions.push(Exportable::new(sig_index));
     }
 
-    fn get_func_type(&self, func_index: FunctionIndex) -> SignatureIndex {
+    fn get_func_type(&self, func_index: FuncIndex) -> SignatureIndex {
         self.info.functions[func_index].entity
     }
 
@@ -376,7 +377,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         _table_index: TableIndex,
         _base: Option,
         _offset: usize,
-        _elements: Vec,
+        _elements: Vec,
     ) {
         // We do nothing
     }
@@ -393,7 +394,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         // We do nothing
     }
 
-    fn declare_func_export(&mut self, func_index: FunctionIndex, name: &'data str) {
+    fn declare_func_export(&mut self, func_index: FuncIndex, name: &'data str) {
         self.info.functions[func_index]
             .export_names
             .push(String::from(name));
@@ -417,7 +418,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
             .push(String::from(name));
     }
 
-    fn declare_start_func(&mut self, func_index: FunctionIndex) {
+    fn declare_start_func(&mut self, func_index: DefinedFuncIndex) {
         debug_assert!(self.info.start_func.is_none());
         self.info.start_func = Some(func_index);
     }
@@ -425,9 +426,9 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
     fn define_function_body(&mut self, body_bytes: &'data [u8]) -> WasmResult<()> {
         let func = {
             let mut func_environ = DummyFuncEnvironment::new(&self.info);
-            let function_index = self.get_num_func_imports() + self.info.function_bodies.len();
-            let name = get_func_name(function_index);
-            let sig = func_environ.vmctx_sig(self.get_func_type(function_index));
+            let func_index = FuncIndex::new(self.get_num_func_imports() + self.info.function_bodies.len());
+            let name = get_func_name(func_index);
+            let sig = func_environ.vmctx_sig(self.get_func_type(func_index));
             let mut func = ir::Function::with_name_signature(name, sig);
             let reader = wasmparser::BinaryReader::new(body_bytes);
             self.trans
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 1a002d5da1..157f531270 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -6,7 +6,7 @@ use cranelift_codegen::settings::Flags;
 use std::vec::Vec;
 use target_lexicon::Triple;
 use translation_utils::{
-    FunctionIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
+    DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
 };
 use wasmparser::BinaryReaderError;
 
@@ -137,7 +137,7 @@ pub trait FuncEnvironment {
     ///
     /// The function's signature will only be used for direct calls, even if the module has
     /// indirect calls with the same WebAssembly type.
-    fn make_direct_func(&mut self, func: &mut ir::Function, index: FunctionIndex) -> ir::FuncRef;
+    fn make_direct_func(&mut self, func: &mut ir::Function, index: FuncIndex) -> ir::FuncRef;
 
     /// Translate a `call_indirect` WebAssembly instruction at `pos`.
     ///
@@ -169,7 +169,7 @@ pub trait FuncEnvironment {
     fn translate_call(
         &mut self,
         mut pos: FuncCursor,
-        _callee_index: FunctionIndex,
+        _callee_index: FuncIndex,
         callee: ir::FuncRef,
         call_args: &[ir::Value],
     ) -> WasmResult {
@@ -222,7 +222,7 @@ pub trait ModuleEnvironment<'data> {
     fn flags(&self) -> &Flags;
 
     /// Return the name for the given function index.
-    fn get_func_name(&self, func_index: FunctionIndex) -> ir::ExternalName;
+    fn get_func_name(&self, func_index: FuncIndex) -> ir::ExternalName;
 
     /// Declares a function signature to the environment.
     fn declare_signature(&mut self, sig: &ir::Signature);
@@ -245,7 +245,7 @@ pub trait ModuleEnvironment<'data> {
     fn declare_func_type(&mut self, sig_index: SignatureIndex);
 
     /// Return the signature index for the given function index.
-    fn get_func_type(&self, func_index: FunctionIndex) -> SignatureIndex;
+    fn get_func_type(&self, func_index: FuncIndex) -> SignatureIndex;
 
     /// Declares a global to the environment.
     fn declare_global(&mut self, global: Global);
@@ -261,7 +261,7 @@ pub trait ModuleEnvironment<'data> {
         table_index: TableIndex,
         base: Option,
         offset: usize,
-        elements: Vec,
+        elements: Vec,
     );
     /// Declares a memory to the environment
     fn declare_memory(&mut self, memory: Memory);
@@ -275,7 +275,7 @@ pub trait ModuleEnvironment<'data> {
     );
 
     /// Declares a function export to the environment.
-    fn declare_func_export(&mut self, func_index: FunctionIndex, name: &'data str);
+    fn declare_func_export(&mut self, func_index: FuncIndex, name: &'data str);
     /// Declares a table export to the environment.
     fn declare_table_export(&mut self, table_index: TableIndex, name: &'data str);
     /// Declares a memory export to the environment.
@@ -284,7 +284,7 @@ pub trait ModuleEnvironment<'data> {
     fn declare_global_export(&mut self, global_index: GlobalIndex, name: &'data str);
 
     /// Declares a start function.
-    fn declare_start_func(&mut self, index: FunctionIndex);
+    fn declare_start_func(&mut self, index: DefinedFuncIndex);
 
     /// Provides the contents of a function body.
     fn define_function_body(&mut self, body_bytes: &'data [u8]) -> WasmResult<()>;
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index 995adf10de..b537534bd5 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -26,6 +26,8 @@
 
 #[macro_use(dbg)]
 extern crate cranelift_codegen;
+#[macro_use]
+extern crate cranelift_entity;
 extern crate cranelift_frontend;
 extern crate target_lexicon;
 extern crate wasmparser;
@@ -48,7 +50,7 @@ pub use environ::{
 pub use func_translator::FuncTranslator;
 pub use module_translator::translate_module;
 pub use translation_utils::{
-    FunctionIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex, Table,
+    DefinedFuncIndex, FuncIndex, Global, GlobalIndex, GlobalInit,  Memory, MemoryIndex, SignatureIndex, Table,
     TableIndex,
 };
 
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 6fdeb38719..e1450ee24b 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -8,11 +8,12 @@
 //! is handled, according to the semantics of WebAssembly, to only specific expressions that are
 //! interpreted on the fly.
 use cranelift_codegen::ir::{self, AbiParam, Signature};
+use cranelift_entity::EntityRef;
 use environ::{ModuleEnvironment, WasmError, WasmResult};
 use std::str::from_utf8;
 use std::vec::Vec;
 use translation_utils::{
-    type_to_type, FunctionIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex,
+    type_to_type, DefinedFuncIndex, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex,
     SignatureIndex, Table, TableElementType, TableIndex,
 };
 use wasmparser;
@@ -150,12 +151,12 @@ pub fn parse_export_section<'data>(
                 // assume valid UTF-8 and use `from_utf8_unchecked` if performance
                 // becomes a concern here.
                 let name = from_utf8(field).unwrap();
-                let func_index = index as FunctionIndex;
+                let func_index = FuncIndex::new(index as usize);
                 match *kind {
                     ExternalKind::Function => environ.declare_func_export(func_index, name),
-                    ExternalKind::Table => environ.declare_table_export(func_index, name),
-                    ExternalKind::Memory => environ.declare_memory_export(func_index, name),
-                    ExternalKind::Global => environ.declare_global_export(func_index, name),
+                    ExternalKind::Table => environ.declare_table_export(func_index.index(), name),
+                    ExternalKind::Memory => environ.declare_memory_export(func_index.index(), name),
+                    ExternalKind::Global => environ.declare_global_export(func_index.index(), name),
                 }
             }
             ParserState::EndSection => break,
@@ -171,7 +172,7 @@ pub fn parse_start_section(parser: &mut Parser, environ: &mut ModuleEnvironment)
     loop {
         match *parser.read() {
             ParserState::StartSectionEntry(index) => {
-                environ.declare_start_func(index as FunctionIndex);
+                environ.declare_start_func(DefinedFuncIndex::new(index as usize));
             }
             ParserState::EndSection => break,
             ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
@@ -382,8 +383,8 @@ pub fn parse_elements_section(
         };
         match *parser.read() {
             ParserState::ElementSectionEntryBody(ref elements) => {
-                let elems: Vec =
-                    elements.iter().map(|&x| x as FunctionIndex).collect();
+                let elems: Vec =
+                    elements.iter().map(|&x| FuncIndex::new(x as usize)).collect();
                 environ.declare_table_elements(table_index, base, offset, elems)
             }
             ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs
index d81234fbde..9b32fe3bbd 100644
--- a/lib/wasm/src/state.rs
+++ b/lib/wasm/src/state.rs
@@ -4,10 +4,11 @@
 //! value and control stacks during the translation of a single function.
 
 use cranelift_codegen::ir::{self, Ebb, Inst, Value};
+use cranelift_entity::EntityRef;
 use environ::{FuncEnvironment, GlobalVariable};
 use std::collections::HashMap;
 use std::vec::Vec;
-use translation_utils::{FunctionIndex, GlobalIndex, MemoryIndex, SignatureIndex, TableIndex};
+use translation_utils::{FuncIndex, GlobalIndex, MemoryIndex, SignatureIndex, TableIndex};
 
 /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following
 /// fields:
@@ -151,7 +152,7 @@ pub struct TranslationState {
     // Imported and local functions that have been created by
     // `FuncEnvironment::make_direct_func()`.
     // Stores both the function reference and the number of WebAssembly arguments
-    functions: HashMap,
+    functions: HashMap,
 }
 
 impl TranslationState {
@@ -349,7 +350,7 @@ impl TranslationState {
         index: u32,
         environ: &mut FE,
     ) -> (ir::FuncRef, usize) {
-        let index = index as FunctionIndex;
+        let index = FuncIndex::new(index as usize);
         *self.functions.entry(index).or_insert_with(|| {
             let fref = environ.make_direct_func(func, index);
             let sig = func.dfg.ext_funcs[fref].signature;
diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index e6822af761..242e3feef0 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -3,8 +3,16 @@ use cranelift_codegen::ir;
 use std::u32;
 use wasmparser;
 
-/// Index of a function (imported or defined) inside the WebAssembly module.
-pub type FunctionIndex = usize;
+/// Index type of a function (imported or defined) inside the WebAssembly module.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct FuncIndex(u32);
+entity_impl!(FuncIndex);
+
+/// Index type of a defined function inside the WebAssembly module. 
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct DefinedFuncIndex(u32);
+entity_impl!(DefinedFuncIndex);
+
 /// Index of a table (imported or defined) inside the WebAssembly module.
 pub type TableIndex = usize;
 /// Index of a global variable (imported or defined) inside the WebAssembly module.
diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs
index 1924a17d07..166b3af076 100644
--- a/lib/wasm/tests/wasm_testsuite.rs
+++ b/lib/wasm/tests/wasm_testsuite.rs
@@ -75,7 +75,7 @@ fn handle_module(path: &Path, flags: &Flags) {
     };
     let mut dummy_environ = DummyEnvironment::with_triple_flags(triple!("riscv64"), flags.clone());
     translate_module(&data, &mut dummy_environ).unwrap();
-    for func in &dummy_environ.info.function_bodies {
+    for func in dummy_environ.info.function_bodies.values() {
         verifier::verify_function(func, flags)
             .map_err(|err| panic!(pretty_verifier_error(func, None, None, &err)))
             .unwrap();

From ee9e5cba495ab5b6613e705aaccc686b0bca2a44 Mon Sep 17 00:00:00 2001
From: Maddy 
Date: Fri, 3 Aug 2018 13:00:31 +0000
Subject: [PATCH 1978/3084] Convert the start function index from a
 `DefinedFuncIndex` to a `FuncIndex`.

---
 lib/wasm/src/environ/dummy.rs       | 4 ++--
 lib/wasm/src/environ/spec.rs        | 4 ++--
 lib/wasm/src/sections_translator.rs | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index b2db269e24..4eaab00fb3 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -71,7 +71,7 @@ pub struct DummyModuleInfo {
     pub globals: Vec>,
 
     /// The start function.
-    pub start_func: Option,
+    pub start_func: Option,
 }
 
 impl DummyModuleInfo {
@@ -418,7 +418,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
             .push(String::from(name));
     }
 
-    fn declare_start_func(&mut self, func_index: DefinedFuncIndex) {
+    fn declare_start_func(&mut self, func_index: FuncIndex) {
         debug_assert!(self.info.start_func.is_none());
         self.info.start_func = Some(func_index);
     }
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 157f531270..bc8e3fa9fa 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -6,7 +6,7 @@ use cranelift_codegen::settings::Flags;
 use std::vec::Vec;
 use target_lexicon::Triple;
 use translation_utils::{
-    DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
+    FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
 };
 use wasmparser::BinaryReaderError;
 
@@ -284,7 +284,7 @@ pub trait ModuleEnvironment<'data> {
     fn declare_global_export(&mut self, global_index: GlobalIndex, name: &'data str);
 
     /// Declares a start function.
-    fn declare_start_func(&mut self, index: DefinedFuncIndex);
+    fn declare_start_func(&mut self, index: FuncIndex);
 
     /// Provides the contents of a function body.
     fn define_function_body(&mut self, body_bytes: &'data [u8]) -> WasmResult<()>;
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index e1450ee24b..949a74f9d1 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -13,7 +13,7 @@ use environ::{ModuleEnvironment, WasmError, WasmResult};
 use std::str::from_utf8;
 use std::vec::Vec;
 use translation_utils::{
-    type_to_type, DefinedFuncIndex, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex,
+    type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex,
     SignatureIndex, Table, TableElementType, TableIndex,
 };
 use wasmparser;
@@ -172,7 +172,7 @@ pub fn parse_start_section(parser: &mut Parser, environ: &mut ModuleEnvironment)
     loop {
         match *parser.read() {
             ParserState::StartSectionEntry(index) => {
-                environ.declare_start_func(DefinedFuncIndex::new(index as usize));
+                environ.declare_start_func(FuncIndex::new(index as usize));
             }
             ParserState::EndSection => break,
             ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),

From 32b332b733a9b78e6669c9ab284498c7bb861c81 Mon Sep 17 00:00:00 2001
From: Maddy 
Date: Fri, 3 Aug 2018 15:17:52 +0000
Subject: [PATCH 1979/3084] Format files.

---
 lib/wasm/src/environ/dummy.rs       |  6 ++++--
 lib/wasm/src/lib.rs                 |  4 ++--
 lib/wasm/src/sections_translator.rs | 10 ++++++----
 lib/wasm/src/translation_utils.rs   |  2 +-
 4 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 4eaab00fb3..21ee1d08a3 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -12,7 +12,8 @@ use std::string::String;
 use std::vec::Vec;
 use target_lexicon::Triple;
 use translation_utils::{
-    DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
+    DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table,
+    TableIndex,
 };
 use wasmparser;
 
@@ -426,7 +427,8 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
     fn define_function_body(&mut self, body_bytes: &'data [u8]) -> WasmResult<()> {
         let func = {
             let mut func_environ = DummyFuncEnvironment::new(&self.info);
-            let func_index = FuncIndex::new(self.get_num_func_imports() + self.info.function_bodies.len());
+            let func_index =
+                FuncIndex::new(self.get_num_func_imports() + self.info.function_bodies.len());
             let name = get_func_name(func_index);
             let sig = func_environ.vmctx_sig(self.get_func_type(func_index));
             let mut func = ir::Function::with_name_signature(name, sig);
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index b537534bd5..7606ebc0b4 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -50,8 +50,8 @@ pub use environ::{
 pub use func_translator::FuncTranslator;
 pub use module_translator::translate_module;
 pub use translation_utils::{
-    DefinedFuncIndex, FuncIndex, Global, GlobalIndex, GlobalInit,  Memory, MemoryIndex, SignatureIndex, Table,
-    TableIndex,
+    DefinedFuncIndex, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex,
+    SignatureIndex, Table, TableIndex,
 };
 
 #[cfg(not(feature = "std"))]
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 949a74f9d1..1ea3ce26ec 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -13,8 +13,8 @@ use environ::{ModuleEnvironment, WasmError, WasmResult};
 use std::str::from_utf8;
 use std::vec::Vec;
 use translation_utils::{
-    type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex,
-    SignatureIndex, Table, TableElementType, TableIndex,
+    type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex,
+    Table, TableElementType, TableIndex,
 };
 use wasmparser;
 use wasmparser::{
@@ -383,8 +383,10 @@ pub fn parse_elements_section(
         };
         match *parser.read() {
             ParserState::ElementSectionEntryBody(ref elements) => {
-                let elems: Vec =
-                    elements.iter().map(|&x| FuncIndex::new(x as usize)).collect();
+                let elems: Vec = elements
+                    .iter()
+                    .map(|&x| FuncIndex::new(x as usize))
+                    .collect();
                 environ.declare_table_elements(table_index, base, offset, elems)
             }
             ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index 242e3feef0..752256f3f1 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -8,7 +8,7 @@ use wasmparser;
 pub struct FuncIndex(u32);
 entity_impl!(FuncIndex);
 
-/// Index type of a defined function inside the WebAssembly module. 
+/// Index type of a defined function inside the WebAssembly module.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct DefinedFuncIndex(u32);
 entity_impl!(DefinedFuncIndex);

From 9683adec644cd0a7c5f779142e54625c4564312e Mon Sep 17 00:00:00 2001
From: Caroline Cullen 
Date: Mon, 6 Aug 2018 16:06:39 -0700
Subject: [PATCH 1980/3084] Updating documentation for serde

---
 lib/serde/src/clif-json.rs       |  4 ++++
 lib/serde/src/serde_clif_json.rs | 17 +++++++++++++----
 2 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/lib/serde/src/clif-json.rs b/lib/serde/src/clif-json.rs
index eb991343ce..81fe47b020 100644
--- a/lib/serde/src/clif-json.rs
+++ b/lib/serde/src/clif-json.rs
@@ -1,3 +1,5 @@
+//! Utility for `cranelift_serde`.
+
 #![deny(trivial_numeric_casts)]
 #![warn(unused_import_braces, unstable_features, unused_extern_crates)]
 #![cfg_attr(
@@ -49,6 +51,7 @@ struct Args {
 /// A command either succeeds or fails with an error message.
 pub type CommandResult = Result<(), String>;
 
+/// Serialize Cranelift IR to JSON
 fn call_ser(file: &str, pretty: bool) -> CommandResult {
     let ret_of_parse = parse_functions(file);
     match ret_of_parse {
@@ -67,6 +70,7 @@ fn call_ser(file: &str, pretty: bool) -> CommandResult {
     }
 }
 
+/// Deserialize JSON to Cranelift IR 
 fn call_de(file: &File) -> CommandResult {
     let de: serde_clif_json::SerObj = match serde_json::from_reader(file) {
         Result::Ok(val) => val,
diff --git a/lib/serde/src/serde_clif_json.rs b/lib/serde/src/serde_clif_json.rs
index 9bd44b5992..2cc6c006c8 100644
--- a/lib/serde/src/serde_clif_json.rs
+++ b/lib/serde/src/serde_clif_json.rs
@@ -228,6 +228,7 @@ pub enum SerInstData {
     },
 }
 
+/// Convert Cranelift IR instructions to JSON format.
 pub fn get_inst_data(inst_index: Inst, func: &Function) -> SerInstData {
     let inst = &func.dfg[inst_index];
     match *inst {
@@ -444,7 +445,6 @@ pub fn get_inst_data(inst_index: Inst, func: &Function) -> SerInstData {
                 destination: destination.to_string(),
             }
         }
-        // to add: jump table serialization
         InstructionData::BranchTable { opcode, arg, table } => SerInstData::BranchTable {
             opcode: opcode.to_string(),
             arg: arg.to_string(),
@@ -661,6 +661,7 @@ pub fn get_inst_data(inst_index: Inst, func: &Function) -> SerInstData {
     }
 }
 
+/// Serializable version of Cranelift IR instructions.
 #[derive(Clone, Deserialize, Serialize, Debug)]
 pub struct SerInst {
     pub inst_name: String,
@@ -676,6 +677,7 @@ impl SerInst {
     }
 }
 
+/// Serializable version of Cranelift IR Ebbs.
 #[derive(Clone, Deserialize, Serialize, Debug)]
 pub struct SerEbb {
     pub ebb: String,
@@ -703,6 +705,7 @@ pub fn populate_inst(func: &Function, ebb: Ebb) -> Vec {
     ser_vec
 }
 
+/// Translating Ebb parameters into serializable parameters.
 pub fn populate_params(func: &Function, ebb: &Ebb) -> Vec {
     let mut ser_vec: Vec = Vec::new();
     let parameters = func.dfg.ebb_params(*ebb);
@@ -712,12 +715,13 @@ pub fn populate_params(func: &Function, ebb: &Ebb) -> Vec {
     ser_vec
 }
 
-/// Serializable Data Flow Graph
+/// Serializable Data Flow Graph.
 #[derive(Deserialize, Serialize, Debug)]
 pub struct SerDataFlowGraph {
     ebbs: Vec,
 }
 
+/// Serialize all parts of the Cranelift Ebb data structure, this includes name, parameters, and instructions.
 pub fn populate_ebbs(func: &Function) -> Vec {
     let mut ebb_vec: Vec = Vec::new();
     for ebb in func.layout.ebbs() {
@@ -729,6 +733,7 @@ pub fn populate_ebbs(func: &Function) -> Vec {
     ebb_vec
 }
 
+/// Serializable Cranelift IR data flow graph, including all ebbs.
 impl SerDataFlowGraph {
     pub fn create_new(func: &Function) -> Self {
         Self {
@@ -741,6 +746,7 @@ impl SerDataFlowGraph {
     }
 }
 
+/// Serializable signature including function parameters and returns.
 #[derive(Serialize, Deserialize, Debug)]
 pub struct SerSignature {
     pub func_params: Vec,
@@ -748,6 +754,7 @@ pub struct SerSignature {
 }
 
 impl SerSignature {
+    /// Creating serializable signature data structure from all Cranelift IR functions.
     fn create_new(sig: &Signature) -> Self {
         let mut params_vec: Vec = Vec::new();
         let mut returns_vec: Vec = Vec::new();
@@ -768,7 +775,7 @@ impl SerSignature {
     }
 }
 
-/// Serializable Function type
+/// Serializable Function type, including name, signature, global values, and data flow graph.
 #[derive(Serialize, Deserialize, Debug)]
 pub struct SerFunction {
     pub name: String,
@@ -778,6 +785,7 @@ pub struct SerFunction {
 }
 
 impl SerFunction {
+    /// Creates serializable global values, as well as the functions signature, name, and data flow graph.
     fn create_new(func: &Function) -> Self {
         let mut global_vec: Vec = Vec::new();
         for (glob_name, _) in func.global_values.iter() {
@@ -796,7 +804,8 @@ impl SerFunction {
     }
 }
 
-/// Must have SerObj for deserialization
+/// Must have SerObj for deserialization, contains all of the functions from inside the file to be serialized.
+/// Files have one SerObj each, with all SerFunctions contained inside that SerObj.
 #[derive(Serialize, Deserialize, Debug)]
 pub struct SerObj {
     pub functions: Vec,

From ad184ff9aaba4a3e68330ebf8b7735f1519af51b Mon Sep 17 00:00:00 2001
From: Caroline Cullen 
Date: Tue, 7 Aug 2018 12:36:49 -0700
Subject: [PATCH 1981/3084] Fixing serde formatting.

---
 lib/serde/src/clif-json.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/serde/src/clif-json.rs b/lib/serde/src/clif-json.rs
index 81fe47b020..659bc818a8 100644
--- a/lib/serde/src/clif-json.rs
+++ b/lib/serde/src/clif-json.rs
@@ -70,7 +70,7 @@ fn call_ser(file: &str, pretty: bool) -> CommandResult {
     }
 }
 
-/// Deserialize JSON to Cranelift IR 
+/// Deserialize JSON to Cranelift IR
 fn call_de(file: &File) -> CommandResult {
     let de: serde_clif_json::SerObj = match serde_json::from_reader(file) {
         Result::Ok(val) => val,

From ae3a3c368b1e80a6d4fdf2b1d273b8df2a36a9f4 Mon Sep 17 00:00:00 2001
From: Amanieu d'Antras 
Date: Tue, 10 Jul 2018 23:24:33 +0100
Subject: [PATCH 1982/3084] Move bforest into a separate crate

---
 cranelift/README.md                           |  1 +
 cranelift/publish-all.sh                      |  2 +-
 lib/bforest/Cargo.toml                        | 21 +++++++++++
 lib/bforest/README.md                         |  2 ++
 .../src/bforest/mod.rs => bforest/src/lib.rs} | 36 +++++++++++++++++--
 .../src/bforest => bforest/src}/map.rs        |  0
 .../src/bforest => bforest/src}/node.rs       |  0
 .../src/bforest => bforest/src}/path.rs       |  0
 .../src/bforest => bforest/src}/pool.rs       |  0
 .../src/bforest => bforest/src}/set.rs        |  0
 lib/codegen/Cargo.toml                        |  3 +-
 lib/codegen/src/lib.rs                        |  3 +-
 12 files changed, 63 insertions(+), 5 deletions(-)
 create mode 100644 lib/bforest/Cargo.toml
 create mode 100644 lib/bforest/README.md
 rename lib/{codegen/src/bforest/mod.rs => bforest/src/lib.rs} (79%)
 rename lib/{codegen/src/bforest => bforest/src}/map.rs (100%)
 rename lib/{codegen/src/bforest => bforest/src}/node.rs (100%)
 rename lib/{codegen/src/bforest => bforest/src}/path.rs (100%)
 rename lib/{codegen/src/bforest => bforest/src}/pool.rs (100%)
 rename lib/{codegen/src/bforest => bforest/src}/set.rs (100%)

diff --git a/cranelift/README.md b/cranelift/README.md
index b84e89bad1..b304a97891 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -79,6 +79,7 @@ Building with no\_std
 
 The following crates support \`no\_std\`:
  - cranelift-entity
+ - cranelift-bforest
  - cranelift-codegen
  - cranelift-frontend
  - cranelift-native
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 67311f5f6f..089e7f27cb 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -36,7 +36,7 @@ cargo update
 echo git commit -a -m "\"Bump version to $version"\"
 echo git push
 for crate in \
-    entity codegen/meta codegen frontend native \
+    entity bforest codegen/meta codegen frontend native \
     reader wasm module simplejit \
     faerie umbrella
 do
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
new file mode 100644
index 0000000000..4b1d1d076c
--- /dev/null
+++ b/lib/bforest/Cargo.toml
@@ -0,0 +1,21 @@
+[package]
+authors = ["The Cranelift Project Developers"]
+name = "cranelift-bforest"
+version = "0.18.1"
+description = "A forest of B+-trees"
+license = "Apache-2.0"
+documentation = "https://cranelift.readthedocs.io/"
+repository = "https://github.com/CraneStation/cranelift"
+readme = "README.md"
+keywords = ["btree", "forest", "set", "map"]
+
+[dependencies]
+cranelift-entity = { path = "../entity", version = "0.18.1", default-features = false }
+
+[features]
+default = ["std"]
+std = ["cranelift-entity/std"]
+
+[badges]
+maintenance = { status = "experimental" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/bforest/README.md b/lib/bforest/README.md
new file mode 100644
index 0000000000..718a4ed8fe
--- /dev/null
+++ b/lib/bforest/README.md
@@ -0,0 +1,2 @@
+This crate contains array-based data structures used by the core Cranelift code
+generator which represent a set of small ordered sets or maps.
diff --git a/lib/codegen/src/bforest/mod.rs b/lib/bforest/src/lib.rs
similarity index 79%
rename from lib/codegen/src/bforest/mod.rs
rename to lib/bforest/src/lib.rs
index bc22cf88c0..1bddcd8a73 100644
--- a/lib/codegen/src/bforest/mod.rs
+++ b/lib/bforest/src/lib.rs
@@ -1,6 +1,6 @@
 //! A forest of B+-trees.
 //!
-//! This module provides a data structures representing a set of small ordered sets or maps.
+//! This crate provides a data structures representing a set of small ordered sets or maps.
 //! It is implemented as a forest of B+-trees all allocating nodes out of the same pool.
 //!
 //! **These are not general purpose data structures that are somehow magically faster that the
@@ -13,6 +13,34 @@
 //! - Empty trees have a very small 32-bit footprint.
 //! - All the trees in a forest can be cleared in constant time.
 
+#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
+#![warn(unused_import_braces)]
+#![cfg_attr(feature = "std", warn(unstable_features))]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
+#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    warn(
+        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
+        print_stdout, unicode_not_nfc, use_self
+    )
+)]
+// Turns on no_std and alloc features if std is not available.
+#![cfg_attr(not(feature = "std"), no_std)]
+#![cfg_attr(not(feature = "std"), feature(alloc))]
+
+/// This replaces `std` in builds with `core`.
+#[cfg(not(feature = "std"))]
+mod std {
+    extern crate alloc;
+    pub use self::alloc::{boxed, string, vec};
+    pub use core::*;
+}
+
+#[macro_use]
+extern crate cranelift_entity as entity;
+use entity::packed_option;
+
 use std::borrow::BorrowMut;
 use std::cmp::Ordering;
 
@@ -124,7 +152,11 @@ fn slice_shift(s: &mut [T], n: usize) {
 mod test {
     use super::*;
     use entity::EntityRef;
-    use ir::Ebb;
+
+    /// An opaque reference to an extended basic block in a function.
+    #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+    pub struct Ebb(u32);
+    entity_impl!(Ebb, "ebb");
 
     #[test]
     fn comparator() {
diff --git a/lib/codegen/src/bforest/map.rs b/lib/bforest/src/map.rs
similarity index 100%
rename from lib/codegen/src/bforest/map.rs
rename to lib/bforest/src/map.rs
diff --git a/lib/codegen/src/bforest/node.rs b/lib/bforest/src/node.rs
similarity index 100%
rename from lib/codegen/src/bforest/node.rs
rename to lib/bforest/src/node.rs
diff --git a/lib/codegen/src/bforest/path.rs b/lib/bforest/src/path.rs
similarity index 100%
rename from lib/codegen/src/bforest/path.rs
rename to lib/bforest/src/path.rs
diff --git a/lib/codegen/src/bforest/pool.rs b/lib/bforest/src/pool.rs
similarity index 100%
rename from lib/codegen/src/bforest/pool.rs
rename to lib/bforest/src/pool.rs
diff --git a/lib/codegen/src/bforest/set.rs b/lib/bforest/src/set.rs
similarity index 100%
rename from lib/codegen/src/bforest/set.rs
rename to lib/bforest/src/set.rs
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index bb767dd67f..5bddad6418 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -12,6 +12,7 @@ build = "build.rs"
 
 [dependencies]
 cranelift-entity = { path = "../entity", version = "0.18.1", default-features = false }
+cranelift-bforest = { path = "../bforest", version = "0.18.1", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -29,7 +30,7 @@ cranelift-codegen-meta = { path = "meta", version = "0.18.1" }
 # of some minimal std-like replacement libraries. At least one of these two
 # features need to be enabled.
 default = ["std"]
-std = ["cranelift-entity/std", "target-lexicon/std"]
+std = ["cranelift-entity/std", "cranelift-bforest/std", "target-lexicon/std"]
 core = ["hashmap_core"]
 
 [badges]
diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index 5d24ddf709..23959e67ac 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -60,10 +60,11 @@ pub const VERSION: &str = env!("CARGO_PKG_VERSION");
 #[macro_use]
 pub extern crate cranelift_entity as entity;
 
+pub extern crate cranelift_bforest as bforest;
+
 #[macro_use]
 pub mod dbg;
 
-pub mod bforest;
 pub mod binemit;
 pub mod cfg_printer;
 pub mod cursor;

From 0b548c720cd488c2f31d8ad08588b7a28a143b9c Mon Sep 17 00:00:00 2001
From: Amanieu d'Antras 
Date: Sun, 12 Aug 2018 15:31:38 +0200
Subject: [PATCH 1983/3084] Don't make the Comparator a type argument for
 bforest

---
 lib/bforest/src/lib.rs                |   3 -
 lib/bforest/src/map.rs                | 116 +++++++++++++-------------
 lib/bforest/src/node.rs               |   1 -
 lib/bforest/src/path.rs               |   3 +-
 lib/bforest/src/pool.rs               |   5 +-
 lib/bforest/src/set.rs                |  93 +++++++++++----------
 lib/codegen/src/flowgraph.rs          |  12 +--
 lib/codegen/src/regalloc/liverange.rs |  37 ++++----
 8 files changed, 134 insertions(+), 136 deletions(-)

diff --git a/lib/bforest/src/lib.rs b/lib/bforest/src/lib.rs
index 1bddcd8a73..2a1c5f8436 100644
--- a/lib/bforest/src/lib.rs
+++ b/lib/bforest/src/lib.rs
@@ -114,9 +114,6 @@ trait Forest {
     /// An array of values for the leaf nodes.
     type LeafValues: Copy + BorrowMut<[Self::Value]>;
 
-    /// Type used for key comparisons.
-    type Comparator: Comparator;
-
     /// Splat a single key into a whole array.
     fn splat_key(key: Self::Key) -> Self::LeafKeys;
 
diff --git a/lib/bforest/src/map.rs b/lib/bforest/src/map.rs
index 270d186230..497ca2c112 100644
--- a/lib/bforest/src/map.rs
+++ b/lib/bforest/src/map.rs
@@ -9,19 +9,17 @@ use std::marker::PhantomData;
 use std::string::String;
 
 /// Tag type defining forest types for a map.
-struct MapTypes(PhantomData<(K, V, C)>);
+struct MapTypes(PhantomData<(K, V)>);
 
-impl Forest for MapTypes
+impl Forest for MapTypes
 where
     K: Copy,
     V: Copy,
-    C: Comparator,
 {
     type Key = K;
     type Value = V;
     type LeafKeys = [K; INNER_SIZE - 1];
     type LeafValues = [V; INNER_SIZE - 1];
-    type Comparator = C;
 
     fn splat_key(key: Self::Key) -> Self::LeafKeys {
         [key; INNER_SIZE - 1]
@@ -33,20 +31,18 @@ where
 }
 
 /// Memory pool for a forest of `Map` instances.
-pub struct MapForest
+pub struct MapForest
 where
     K: Copy,
     V: Copy,
-    C: Comparator,
 {
-    nodes: NodePool>,
+    nodes: NodePool>,
 }
 
-impl MapForest
+impl MapForest
 where
     K: Copy,
     V: Copy,
-    C: Comparator,
 {
     /// Create a new empty forest.
     pub fn new() -> Self {
@@ -63,7 +59,7 @@ where
     }
 }
 
-/// B-tree mapping from `K` to `V` using `C` for comparing keys.
+/// B-tree mapping from `K` to `V`.
 ///
 /// This is not a general-purpose replacement for `BTreeMap`. See the [module
 /// documentation](index.html) for more information about design tradeoffs.
@@ -72,21 +68,19 @@ where
 /// they belong to. *Cloning a map does not allocate new memory for the clone*. It creates an alias
 /// of the same memory.
 #[derive(Clone)]
-pub struct Map
+pub struct Map
 where
     K: Copy,
     V: Copy,
-    C: Comparator,
 {
     root: PackedOption,
-    unused: PhantomData<(K, V, C)>,
+    unused: PhantomData<(K, V)>,
 }
 
-impl Map
+impl Map
 where
     K: Copy,
     V: Copy,
-    C: Comparator,
 {
     /// Make an empty map.
     pub fn new() -> Self {
@@ -102,7 +96,7 @@ where
     }
 
     /// Get the value stored for `key`.
-    pub fn get(&self, key: K, forest: &MapForest, comp: &C) -> Option {
+    pub fn get>(&self, key: K, forest: &MapForest, comp: &C) -> Option {
         self.root
             .expand()
             .and_then(|root| Path::default().find(key, root, &forest.nodes, comp))
@@ -115,7 +109,12 @@ where
     /// Otherwise, return the last key-value pair with a key that is less than or equal to `key`.
     ///
     /// If no stored keys are less than or equal to `key`, return `None`.
-    pub fn get_or_less(&self, key: K, forest: &MapForest, comp: &C) -> Option<(K, V)> {
+    pub fn get_or_less>(
+        &self,
+        key: K,
+        forest: &MapForest,
+        comp: &C,
+    ) -> Option<(K, V)> {
         self.root.expand().and_then(|root| {
             let mut path = Path::default();
             match path.find(key, root, &forest.nodes, comp) {
@@ -126,18 +125,23 @@ where
     }
 
     /// Insert `key, value` into the map and return the old value stored for `key`, if any.
-    pub fn insert(
+    pub fn insert>(
         &mut self,
         key: K,
         value: V,
-        forest: &mut MapForest,
+        forest: &mut MapForest,
         comp: &C,
     ) -> Option {
         self.cursor(forest, comp).insert(key, value)
     }
 
     /// Remove `key` from the map and return the removed value for `key`, if any.
-    pub fn remove(&mut self, key: K, forest: &mut MapForest, comp: &C) -> Option {
+    pub fn remove>(
+        &mut self,
+        key: K,
+        forest: &mut MapForest,
+        comp: &C,
+    ) -> Option {
         let mut c = self.cursor(forest, comp);
         if c.goto(key).is_some() {
             c.remove()
@@ -147,7 +151,7 @@ where
     }
 
     /// Remove all entries.
-    pub fn clear(&mut self, forest: &mut MapForest) {
+    pub fn clear(&mut self, forest: &mut MapForest) {
         if let Some(root) = self.root.take() {
             forest.nodes.free_tree(root);
         }
@@ -158,7 +162,7 @@ where
     /// Remove all key-value pairs where the predicate returns false.
     ///
     /// The predicate is allowed to update the values stored in the map.
-    pub fn retain(&mut self, forest: &mut MapForest, mut predicate: F)
+    pub fn retain(&mut self, forest: &mut MapForest, mut predicate: F)
     where
         F: FnMut(K, &mut V) -> bool,
     {
@@ -181,16 +185,16 @@ where
 
     /// Create a cursor for navigating this map. The cursor is initially positioned off the end of
     /// the map.
-    pub fn cursor<'a>(
+    pub fn cursor<'a, C: Comparator>(
         &'a mut self,
-        forest: &'a mut MapForest,
+        forest: &'a mut MapForest,
         comp: &'a C,
     ) -> MapCursor<'a, K, V, C> {
         MapCursor::new(self, forest, comp)
     }
 
     /// Create an iterator traversing this map. The iterator type is `(K, V)`.
-    pub fn iter<'a>(&'a self, forest: &'a MapForest) -> MapIter<'a, K, V, C> {
+    pub fn iter<'a>(&'a self, forest: &'a MapForest) -> MapIter<'a, K, V> {
         MapIter {
             root: self.root,
             pool: &forest.nodes,
@@ -199,11 +203,10 @@ where
     }
 }
 
-impl Default for Map
+impl Default for Map
 where
     K: Copy,
     V: Copy,
-    C: Comparator,
 {
     fn default() -> Self {
         Self::new()
@@ -211,16 +214,15 @@ where
 }
 
 #[cfg(test)]
-impl Map
+impl Map
 where
     K: Copy + fmt::Display,
     V: Copy,
-    C: Comparator,
 {
     /// Verify consistency.
-    fn verify(&self, forest: &MapForest, comp: &C)
+    fn verify>(&self, forest: &MapForest, comp: &C)
     where
-        NodeData>: fmt::Display,
+        NodeData>: fmt::Display,
     {
         if let Some(root) = self.root.expand() {
             forest.nodes.verify_tree(root, comp);
@@ -228,7 +230,7 @@ where
     }
 
     /// Get a text version of the path to `key`.
-    fn tpath(&self, key: K, forest: &MapForest, comp: &C) -> String {
+    fn tpath>(&self, key: K, forest: &MapForest, comp: &C) -> String {
         use std::string::ToString;
         match self.root.expand() {
             None => "map(empty)".to_string(),
@@ -252,9 +254,9 @@ where
     C: 'a + Comparator,
 {
     root: &'a mut PackedOption,
-    pool: &'a mut NodePool>,
+    pool: &'a mut NodePool>,
     comp: &'a C,
-    path: Path>,
+    path: Path>,
 }
 
 impl<'a, K, V, C> MapCursor<'a, K, V, C>
@@ -265,8 +267,8 @@ where
 {
     /// Create a cursor with a default (off-the-end) location.
     fn new(
-        container: &'a mut Map,
-        forest: &'a mut MapForest,
+        container: &'a mut Map,
+        forest: &'a mut MapForest,
         comp: &'a C,
     ) -> MapCursor<'a, K, V, C> {
         MapCursor {
@@ -379,22 +381,20 @@ where
 }
 
 /// An iterator visiting the key-value pairs of a `Map`.
-pub struct MapIter<'a, K, V, C>
+pub struct MapIter<'a, K, V>
 where
     K: 'a + Copy,
     V: 'a + Copy,
-    C: 'a + Comparator,
 {
     root: PackedOption,
-    pool: &'a NodePool>,
-    path: Path>,
+    pool: &'a NodePool>,
+    path: Path>,
 }
 
-impl<'a, K, V, C> Iterator for MapIter<'a, K, V, C>
+impl<'a, K, V> Iterator for MapIter<'a, K, V>
 where
     K: 'a + Copy,
     V: 'a + Copy,
-    C: 'a + Comparator,
 {
     type Item = (K, V);
 
@@ -438,16 +438,16 @@ mod test {
     #[test]
     fn node_size() {
         // check that nodes are cache line sized when keys and values are 32 bits.
-        type F = MapTypes;
+        type F = MapTypes;
         assert_eq!(mem::size_of::>(), 64);
     }
 
     #[test]
     fn empty() {
-        let mut f = MapForest::::new();
+        let mut f = MapForest::::new();
         f.clear();
 
-        let mut m = Map::::new();
+        let mut m = Map::::new();
         assert!(m.is_empty());
         m.clear(&mut f);
 
@@ -470,8 +470,8 @@ mod test {
 
     #[test]
     fn inserting() {
-        let f = &mut MapForest::::new();
-        let mut m = Map::::new();
+        let f = &mut MapForest::::new();
+        let mut m = Map::::new();
 
         // The first seven values stay in a single leaf node.
         assert_eq!(m.insert(50, 5.0, f, &()), None);
@@ -577,9 +577,9 @@ mod test {
     #[test]
     fn split_level0_leaf() {
         // Various ways of splitting a full leaf node at level 0.
-        let f = &mut MapForest::::new();
+        let f = &mut MapForest::::new();
 
-        fn full_leaf(f: &mut MapForest) -> Map {
+        fn full_leaf(f: &mut MapForest) -> Map {
             let mut m = Map::new();
             for n in 1..8 {
                 m.insert(n * 10, n as f32 * 1.1, f, &());
@@ -628,7 +628,7 @@ mod test {
     #[test]
     fn split_level1_leaf() {
         // Various ways of splitting a full leaf node at level 1.
-        let f = &mut MapForest::::new();
+        let f = &mut MapForest::::new();
 
         // Return a map whose root node is a full inner node, and the leaf nodes are all full
         // containing:
@@ -637,7 +637,7 @@ mod test {
         // 210, 220, ..., 270
         // ...
         // 810, 820, ..., 870
-        fn full(f: &mut MapForest) -> Map {
+        fn full(f: &mut MapForest) -> Map {
             let mut m = Map::new();
 
             // Start by inserting elements in order.
@@ -756,7 +756,7 @@ mod test {
 
     // Make a tree with two barely healthy leaf nodes:
     // [ 10 20 30 40 ] [ 50 60 70 80 ]
-    fn two_leaf(f: &mut MapForest) -> Map {
+    fn two_leaf(f: &mut MapForest) -> Map {
         f.clear();
         let mut m = Map::new();
         for n in 1..9 {
@@ -767,7 +767,7 @@ mod test {
 
     #[test]
     fn remove_level1() {
-        let f = &mut MapForest::::new();
+        let f = &mut MapForest::::new();
         let mut m = two_leaf(f);
 
         // Verify geometry.
@@ -830,7 +830,7 @@ mod test {
 
     #[test]
     fn remove_level1_rightmost() {
-        let f = &mut MapForest::::new();
+        let f = &mut MapForest::::new();
         let mut m = two_leaf(f);
 
         // [ 10 20 30 40 ] [ 50 60 70 80 ]
@@ -852,7 +852,7 @@ mod test {
 
     // Make a 3-level tree with barely healthy nodes.
     // 1 root, 8 inner nodes, 7*4+5=33 leaf nodes, 4 entries each.
-    fn level3_sparse(f: &mut MapForest) -> Map {
+    fn level3_sparse(f: &mut MapForest) -> Map {
         f.clear();
         let mut m = Map::new();
         for n in 1..133 {
@@ -863,7 +863,7 @@ mod test {
 
     #[test]
     fn level3_removes() {
-        let f = &mut MapForest::::new();
+        let f = &mut MapForest::::new();
         let mut m = level3_sparse(f);
         m.verify(f, &());
 
@@ -894,8 +894,8 @@ mod test {
 
     #[test]
     fn insert_many() {
-        let f = &mut MapForest::::new();
-        let mut m = Map::::new();
+        let f = &mut MapForest::::new();
+        let mut m = Map::::new();
 
         let mm = 4096;
         let mut x = 0;
diff --git a/lib/bforest/src/node.rs b/lib/bforest/src/node.rs
index b529816afb..de5337053d 100644
--- a/lib/bforest/src/node.rs
+++ b/lib/bforest/src/node.rs
@@ -595,7 +595,6 @@ mod test {
         type Value = SetValue;
         type LeafKeys = [char; 15];
         type LeafValues = [SetValue; 15];
-        type Comparator = ();
 
         fn splat_key(key: Self::Key) -> Self::LeafKeys {
             [key; 15]
diff --git a/lib/bforest/src/path.rs b/lib/bforest/src/path.rs
index c0e4a089af..153c487547 100644
--- a/lib/bforest/src/path.rs
+++ b/lib/bforest/src/path.rs
@@ -49,7 +49,7 @@ impl Path {
         key: F::Key,
         root: Node,
         pool: &NodePool,
-        comp: &F::Comparator,
+        comp: &Comparator,
     ) -> Option {
         let mut node = root;
         for level in 0.. {
@@ -723,7 +723,6 @@ mod test {
         type Value = char;
         type LeafKeys = [i32; 7];
         type LeafValues = [char; 7];
-        type Comparator = TC;
 
         fn splat_key(key: Self::Key) -> Self::LeafKeys {
             [key; 7]
diff --git a/lib/bforest/src/pool.rs b/lib/bforest/src/pool.rs
index e6b08e4f1c..ceab96d38c 100644
--- a/lib/bforest/src/pool.rs
+++ b/lib/bforest/src/pool.rs
@@ -1,5 +1,7 @@
 //! B+-tree node pool.
 
+#[cfg(test)]
+use super::Comparator;
 use super::{Forest, Node, NodeData};
 use entity::PrimaryMap;
 #[cfg(test)]
@@ -76,12 +78,11 @@ impl NodePool {
 #[cfg(test)]
 impl NodePool {
     /// Verify the consistency of the tree rooted at `node`.
-    pub fn verify_tree(&self, node: Node, comp: &F::Comparator)
+    pub fn verify_tree>(&self, node: Node, comp: &C)
     where
         NodeData: fmt::Display,
         F::Key: fmt::Display,
     {
-        use super::Comparator;
         use entity::SparseSet;
         use std::borrow::Borrow;
         use std::cmp::Ordering;
diff --git a/lib/bforest/src/set.rs b/lib/bforest/src/set.rs
index 8371fd3bc6..911907ee61 100644
--- a/lib/bforest/src/set.rs
+++ b/lib/bforest/src/set.rs
@@ -9,18 +9,16 @@ use std::marker::PhantomData;
 use std::string::String;
 
 /// Tag type defining forest types for a set.
-struct SetTypes(PhantomData<(K, C)>);
+struct SetTypes(PhantomData);
 
-impl Forest for SetTypes
+impl Forest for SetTypes
 where
     K: Copy,
-    C: Comparator,
 {
     type Key = K;
     type Value = SetValue;
     type LeafKeys = [K; 2 * INNER_SIZE - 1];
     type LeafValues = [SetValue; 2 * INNER_SIZE - 1];
-    type Comparator = C;
 
     fn splat_key(key: Self::Key) -> Self::LeafKeys {
         [key; 2 * INNER_SIZE - 1]
@@ -32,18 +30,16 @@ where
 }
 
 /// Memory pool for a forest of `Set` instances.
-pub struct SetForest
+pub struct SetForest
 where
     K: Copy,
-    C: Comparator,
 {
-    nodes: NodePool>,
+    nodes: NodePool>,
 }
 
-impl SetForest
+impl SetForest
 where
     K: Copy,
-    C: Comparator,
 {
     /// Create a new empty forest.
     pub fn new() -> Self {
@@ -69,19 +65,17 @@ where
 /// they belong to. *Cloning a set does not allocate new memory for the clone*. It creates an alias
 /// of the same memory.
 #[derive(Clone)]
-pub struct Set
+pub struct Set
 where
     K: Copy,
-    C: Comparator,
 {
     root: PackedOption,
-    unused: PhantomData<(K, C)>,
+    unused: PhantomData,
 }
 
-impl Set
+impl Set
 where
     K: Copy,
-    C: Comparator,
 {
     /// Make an empty set.
     pub fn new() -> Self {
@@ -97,7 +91,7 @@ where
     }
 
     /// Does the set contain `key`?.
-    pub fn contains(&self, key: K, forest: &SetForest, comp: &C) -> bool {
+    pub fn contains>(&self, key: K, forest: &SetForest, comp: &C) -> bool {
         self.root
             .expand()
             .and_then(|root| Path::default().find(key, root, &forest.nodes, comp))
@@ -109,14 +103,24 @@ where
     /// If the set did not contain `key`, insert it and return true.
     ///
     /// If `key` is already present, don't change the set and return false.
-    pub fn insert(&mut self, key: K, forest: &mut SetForest, comp: &C) -> bool {
+    pub fn insert>(
+        &mut self,
+        key: K,
+        forest: &mut SetForest,
+        comp: &C,
+    ) -> bool {
         self.cursor(forest, comp).insert(key)
     }
 
     /// Remove `key` from the set and return true.
     ///
     /// If `key` was not present in the set, return false.
-    pub fn remove(&mut self, key: K, forest: &mut SetForest, comp: &C) -> bool {
+    pub fn remove>(
+        &mut self,
+        key: K,
+        forest: &mut SetForest,
+        comp: &C,
+    ) -> bool {
         let mut c = self.cursor(forest, comp);
         if c.goto(key) {
             c.remove();
@@ -127,7 +131,7 @@ where
     }
 
     /// Remove all entries.
-    pub fn clear(&mut self, forest: &mut SetForest) {
+    pub fn clear(&mut self, forest: &mut SetForest) {
         if let Some(root) = self.root.take() {
             forest.nodes.free_tree(root);
         }
@@ -136,7 +140,7 @@ where
     /// Retains only the elements specified by the predicate.
     ///
     /// Remove all elements where the predicate returns false.
-    pub fn retain(&mut self, forest: &mut SetForest, mut predicate: F)
+    pub fn retain(&mut self, forest: &mut SetForest, mut predicate: F)
     where
         F: FnMut(K) -> bool,
     {
@@ -155,16 +159,16 @@ where
 
     /// Create a cursor for navigating this set. The cursor is initially positioned off the end of
     /// the set.
-    pub fn cursor<'a>(
+    pub fn cursor<'a, C: Comparator>(
         &'a mut self,
-        forest: &'a mut SetForest,
+        forest: &'a mut SetForest,
         comp: &'a C,
     ) -> SetCursor<'a, K, C> {
         SetCursor::new(self, forest, comp)
     }
 
     /// Create an iterator traversing this set. The iterator type is `K`.
-    pub fn iter<'a>(&'a self, forest: &'a SetForest) -> SetIter<'a, K, C> {
+    pub fn iter<'a>(&'a self, forest: &'a SetForest) -> SetIter<'a, K> {
         SetIter {
             root: self.root,
             pool: &forest.nodes,
@@ -173,10 +177,9 @@ where
     }
 }
 
-impl Default for Set
+impl Default for Set
 where
     K: Copy,
-    C: Comparator,
 {
     fn default() -> Self {
         Self::new()
@@ -193,9 +196,9 @@ where
     C: 'a + Comparator,
 {
     root: &'a mut PackedOption,
-    pool: &'a mut NodePool>,
+    pool: &'a mut NodePool>,
     comp: &'a C,
-    path: Path>,
+    path: Path>,
 }
 
 impl<'a, K, C> SetCursor<'a, K, C>
@@ -205,8 +208,8 @@ where
 {
     /// Create a cursor with a default (invalid) location.
     fn new(
-        container: &'a mut Set,
-        forest: &'a mut SetForest,
+        container: &'a mut Set,
+        forest: &'a mut SetForest,
         comp: &'a C,
     ) -> SetCursor<'a, K, C> {
         SetCursor {
@@ -327,20 +330,18 @@ where
 }
 
 /// An iterator visiting the elements of a `Set`.
-pub struct SetIter<'a, K, C>
+pub struct SetIter<'a, K>
 where
     K: 'a + Copy,
-    C: 'a + Comparator,
 {
     root: PackedOption,
-    pool: &'a NodePool>,
-    path: Path>,
+    pool: &'a NodePool>,
+    path: Path>,
 }
 
-impl<'a, K, C> Iterator for SetIter<'a, K, C>
+impl<'a, K> Iterator for SetIter<'a, K>
 where
     K: 'a + Copy,
-    C: 'a + Comparator,
 {
     type Item = K;
 
@@ -365,16 +366,16 @@ mod test {
     #[test]
     fn node_size() {
         // check that nodes are cache line sized when keys are 32 bits.
-        type F = SetTypes;
+        type F = SetTypes;
         assert_eq!(mem::size_of::>(), 64);
     }
 
     #[test]
     fn empty() {
-        let mut f = SetForest::::new();
+        let mut f = SetForest::::new();
         f.clear();
 
-        let mut s = Set::::new();
+        let mut s = Set::::new();
         assert!(s.is_empty());
         s.clear(&mut f);
         assert!(!s.contains(7, &f, &()));
@@ -394,8 +395,8 @@ mod test {
 
     #[test]
     fn simple_cursor() {
-        let mut f = SetForest::::new();
-        let mut s = Set::::new();
+        let mut f = SetForest::::new();
+        let mut s = Set::::new();
         let mut c = SetCursor::new(&mut s, &mut f, &());
 
         assert!(c.insert(50));
@@ -436,8 +437,8 @@ mod test {
 
     #[test]
     fn two_level_sparse_tree() {
-        let mut f = SetForest::::new();
-        let mut s = Set::::new();
+        let mut f = SetForest::::new();
+        let mut s = Set::::new();
         let mut c = SetCursor::new(&mut s, &mut f, &());
 
         // Insert enough elements that we get a two-level tree.
@@ -482,8 +483,8 @@ mod test {
 
     #[test]
     fn three_level_sparse_tree() {
-        let mut f = SetForest::::new();
-        let mut s = Set::::new();
+        let mut f = SetForest::::new();
+        let mut s = Set::::new();
         let mut c = SetCursor::new(&mut s, &mut f, &());
 
         // Insert enough elements that we get a 3-level tree.
@@ -535,7 +536,7 @@ mod test {
     // Level 4: 512 leafs, up to 7680 elements
     //
     // A 3-level tree can hold at most 960 elements.
-    fn dense4l(f: &mut SetForest) -> Set {
+    fn dense4l(f: &mut SetForest) -> Set {
         f.clear();
         let mut s = Set::new();
 
@@ -549,7 +550,7 @@ mod test {
 
     #[test]
     fn four_level() {
-        let mut f = SetForest::::new();
+        let mut f = SetForest::::new();
         let mut s = dense4l(&mut f);
 
         assert_eq!(
@@ -593,7 +594,7 @@ mod test {
 
     #[test]
     fn four_level_clear() {
-        let mut f = SetForest::::new();
+        let mut f = SetForest::::new();
         let mut s = dense4l(&mut f);
         s.clear(&mut f);
     }
diff --git a/lib/codegen/src/flowgraph.rs b/lib/codegen/src/flowgraph.rs
index 9eda2741af..b84da7cd0b 100644
--- a/lib/codegen/src/flowgraph.rs
+++ b/lib/codegen/src/flowgraph.rs
@@ -61,11 +61,11 @@ struct CFGNode {
     ///
     /// The redundant EBB stored here is always consistent with the CFG successor lists, even after
     /// the IR has been edited.
-    pub predecessors: bforest::Map,
+    pub predecessors: bforest::Map,
 
     /// Set of EBBs that are the targets of branches and jumps in this EBB.
     /// The set is ordered by EBB number, indicated by the `()` comparator type.
-    pub successors: bforest::Set,
+    pub successors: bforest::Set,
 }
 
 /// The Control Flow Graph maintains a mapping of ebbs to their predecessors
@@ -73,8 +73,8 @@ struct CFGNode {
 /// extended basic blocks.
 pub struct ControlFlowGraph {
     data: EntityMap,
-    pred_forest: bforest::MapForest,
-    succ_forest: bforest::SetForest,
+    pred_forest: bforest::MapForest,
+    succ_forest: bforest::SetForest,
     valid: bool,
 }
 
@@ -193,7 +193,7 @@ impl ControlFlowGraph {
 /// An iterator over EBB predecessors. The iterator type is `BasicBlock`.
 ///
 /// Each predecessor is an instruction that branches to the EBB.
-pub struct PredIter<'a>(bforest::MapIter<'a, Inst, Ebb, ()>);
+pub struct PredIter<'a>(bforest::MapIter<'a, Inst, Ebb>);
 
 impl<'a> Iterator for PredIter<'a> {
     type Item = BasicBlock;
@@ -204,7 +204,7 @@ impl<'a> Iterator for PredIter<'a> {
 }
 
 /// An iterator over EBB successors. The iterator type is `Ebb`.
-pub type SuccIter<'a> = bforest::SetIter<'a, Ebb, ()>;
+pub type SuccIter<'a> = bforest::SetIter<'a, Ebb>;
 
 #[cfg(test)]
 mod tests {
diff --git a/lib/codegen/src/regalloc/liverange.rs b/lib/codegen/src/regalloc/liverange.rs
index f330fb12b5..cc6a985641 100644
--- a/lib/codegen/src/regalloc/liverange.rs
+++ b/lib/codegen/src/regalloc/liverange.rs
@@ -112,6 +112,7 @@ use entity::SparseMapValue;
 use ir::{Ebb, ExpandedProgramPoint, Inst, Layout, ProgramOrder, ProgramPoint, Value};
 use regalloc::affinity::Affinity;
 use std::cmp::Ordering;
+use std::marker::PhantomData;
 
 /// Global live range of a single SSA value.
 ///
@@ -172,7 +173,9 @@ pub struct GenLiveRange {
     ///
     /// The entries are non-overlapping, and none of them overlap the EBB where the value is
     /// defined.
-    liveins: bforest::Map,
+    liveins: bforest::Map,
+
+    po: PhantomData<*const PO>,
 }
 
 /// Context information needed to query a `LiveRange`.
@@ -180,14 +183,14 @@ pub struct LiveRangeContext<'a, PO: 'a + ProgramOrder> {
     /// Ordering of EBBs.
     pub order: &'a PO,
     /// Memory pool.
-    pub forest: &'a bforest::MapForest,
+    pub forest: &'a bforest::MapForest,
 }
 
 impl<'a, PO: ProgramOrder> LiveRangeContext<'a, PO> {
     /// Make a new context.
     pub fn new(
         order: &'a PO,
-        forest: &'a bforest::MapForest,
+        forest: &'a bforest::MapForest,
     ) -> LiveRangeContext<'a, PO> {
         LiveRangeContext { order, forest }
     }
@@ -205,11 +208,13 @@ impl<'a, PO: ProgramOrder> Clone for LiveRangeContext<'a, PO> {
 impl<'a, PO: ProgramOrder> Copy for LiveRangeContext<'a, PO> {}
 
 /// Forest of B-trees used for storing live ranges.
-pub type LiveRangeForest = bforest::MapForest;
+pub type LiveRangeForest = bforest::MapForest;
 
-impl bforest::Comparator for PO {
+struct Cmp<'a, PO: ProgramOrder + 'a>(&'a PO);
+
+impl<'a, PO: ProgramOrder> bforest::Comparator for Cmp<'a, PO> {
     fn cmp(&self, a: Ebb, b: Ebb) -> Ordering {
-        self.cmp(a, b)
+        self.0.cmp(a, b)
     }
 }
 
@@ -224,6 +229,7 @@ impl GenLiveRange {
             def_begin: def,
             def_end: def,
             liveins: bforest::Map::new(),
+            po: PhantomData,
         }
     }
 
@@ -243,7 +249,7 @@ impl GenLiveRange {
         ebb: Ebb,
         to: Inst,
         order: &PO,
-        forest: &mut bforest::MapForest,
+        forest: &mut bforest::MapForest,
     ) -> bool {
         // First check if we're extending the def interval.
         //
@@ -264,7 +270,8 @@ impl GenLiveRange {
         }
 
         // Now check if we're extending any of the existing live-in intervals.
-        let mut c = self.liveins.cursor(forest, order);
+        let cmp = Cmp(order);
+        let mut c = self.liveins.cursor(forest, &cmp);
         let first_time_livein;
 
         if let Some(end) = c.goto(ebb) {
@@ -367,8 +374,9 @@ impl GenLiveRange {
     /// answer, but it is also possible that an even later program point is returned. So don't
     /// depend on the returned `Inst` to belong to `ebb`.
     pub fn livein_local_end(&self, ebb: Ebb, ctx: LiveRangeContext) -> Option {
+        let cmp = Cmp(ctx.order);
         self.liveins
-            .get_or_less(ebb, ctx.forest, ctx.order)
+            .get_or_less(ebb, ctx.forest, &cmp)
             .and_then(|(_, inst)| {
                 // We have an entry that ends at `inst`.
                 if ctx.order.cmp(inst, ebb) == Ordering::Greater {
@@ -390,10 +398,7 @@ impl GenLiveRange {
     ///
     /// Note that the intervals are stored in a compressed form so each entry may span multiple
     /// EBBs where the value is live in.
-    pub fn liveins<'a>(
-        &'a self,
-        ctx: LiveRangeContext<'a, PO>,
-    ) -> bforest::MapIter<'a, Ebb, Inst, PO> {
+    pub fn liveins<'a>(&'a self, ctx: LiveRangeContext<'a, PO>) -> bforest::MapIter<'a, Ebb, Inst> {
         self.liveins.iter(ctx.forest)
     }
 
@@ -507,11 +512,7 @@ mod tests {
         }
 
         // Validate the live range invariants.
-        fn validate(
-            &self,
-            lr: &GenLiveRange,
-            forest: &bforest::MapForest,
-        ) {
+        fn validate(&self, lr: &GenLiveRange, forest: &bforest::MapForest) {
             // The def interval must cover a single EBB.
             let def_ebb = self.pp_ebb(lr.def_begin);
             assert_eq!(def_ebb, self.pp_ebb(lr.def_end));

From eb01ae530bc0566a5611b95ca2364235fbf254e0 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Mon, 13 Aug 2018 20:03:53 +0200
Subject: [PATCH 1984/3084] Fix gitter link (fixes #458)

---
 cranelift/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index b84e89bad1..9011d4b752 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -8,7 +8,7 @@ into executable machine code.
 
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
 [![Build Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
-[![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby/~chat)
+[![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
 
 For more information, see [the
 documentation](https://cranelift.readthedocs.io/en/latest/?badge=latest).

From 60c2cad06ea160c545085884b31a2abbe6936117 Mon Sep 17 00:00:00 2001
From: Daniel Keep 
Date: Tue, 7 Aug 2018 17:47:30 +1000
Subject: [PATCH 1985/3084] Add SimpleJIT internal symbol table.

Allows for host programs to directly expose symbols to jitted code without
needing to deal with platform-specific linker arguments, or dynamic
dispatch.
---
 lib/simplejit/src/backend.rs | 60 +++++++++++++++++++++++++++++++++---
 1 file changed, 55 insertions(+), 5 deletions(-)

diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index 0ae65a8a99..c867c0d825 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -10,6 +10,7 @@ use cranelift_module::{
 use cranelift_native;
 use libc;
 use memory::Memory;
+use std::collections::HashMap;
 use std::ffi::CString;
 use std::ptr;
 use target_lexicon::PointerWidth;
@@ -19,6 +20,7 @@ use winapi;
 /// A builder for `SimpleJITBackend`.
 pub struct SimpleJITBuilder {
     isa: Box,
+    symbols: HashMap,
 }
 
 impl SimpleJITBuilder {
@@ -40,7 +42,44 @@ impl SimpleJITBuilder {
     /// instead.
     pub fn with_isa(isa: Box) -> Self {
         debug_assert!(!isa.flags().is_pic(), "SimpleJIT requires non-PIC code");
-        Self { isa }
+        let symbols = HashMap::new();
+        Self { isa, symbols }
+    }
+
+    /// Define a symbol in the internal symbol table.
+    ///
+    /// The JIT will use the symbol table to resolve names that are declared,
+    /// but not defined, in the module being compiled.  A common example is
+    /// external functions.  With this method, functions and data can be exposed
+    /// to the code being compiled which are defined by the host.
+    ///
+    /// If a symbol is defined more than once, the most recent definition will
+    /// be retained.
+    ///
+    /// If the JIT fails to find a symbol in its internal table, it will fall
+    /// back to a platform-specific search (this typically involves searching
+    /// the current process for public symbols, followed by searching the
+    /// platform's C runtime).
+    pub fn symbol<'a, K>(&'a mut self, name: K, ptr: *const u8) -> &'a mut Self
+    where
+        K: Into,
+    {
+        self.symbols.insert(name.into(), ptr);
+        self
+    }
+
+    /// Define multiple symbols in the internal symbol table.
+    ///
+    /// Using this is equivalent to calling `symbol` on each element.
+    pub fn symbols<'a, It, K>(&'a mut self, symbols: It) -> &'a mut Self
+    where
+        It: IntoIterator,
+        K: Into,
+    {
+        for (name, ptr) in symbols {
+            self.symbols.insert(name.into(), ptr);
+        }
+        self
     }
 }
 
@@ -48,6 +87,7 @@ impl SimpleJITBuilder {
 /// directly called and accessed.
 pub struct SimpleJITBackend {
     isa: Box,
+    symbols: HashMap,
     code_memory: Memory,
     readonly_memory: Memory,
     writable_memory: Memory,
@@ -73,6 +113,15 @@ pub struct SimpleJITCompiledData {
     relocs: Vec,
 }
 
+impl SimpleJITBackend {
+    fn lookup_symbol(&self, name: &str) -> *const u8 {
+        match self.symbols.get(name) {
+            Some(&ptr) => ptr,
+            None => lookup_with_dlsym(name),
+        }
+    }
+}
+
 impl<'simple_jit_backend> Backend for SimpleJITBackend {
     type Builder = SimpleJITBuilder;
 
@@ -96,6 +145,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
     fn new(builder: SimpleJITBuilder) -> Self {
         Self {
             isa: builder.isa,
+            symbols: builder.symbols,
             code_memory: Memory::new(),
             readonly_memory: Memory::new(),
             writable_memory: Memory::new(),
@@ -249,13 +299,13 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
                 let (def, name_str, _signature) = namespace.get_function_definition(&name);
                 match def {
                     Some(compiled) => compiled.code,
-                    None => lookup_with_dlsym(name_str),
+                    None => self.lookup_symbol(name_str),
                 }
             } else {
                 let (def, name_str, _writable) = namespace.get_data_definition(&name);
                 match def {
                     Some(compiled) => compiled.storage,
-                    None => lookup_with_dlsym(name_str),
+                    None => self.lookup_symbol(name_str),
                 }
             };
             // TODO: Handle overflow.
@@ -314,13 +364,13 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
                         let (def, name_str, _signature) = namespace.get_function_definition(&name);
                         match def {
                             Some(compiled) => compiled.code,
-                            None => lookup_with_dlsym(name_str),
+                            None => self.lookup_symbol(name_str),
                         }
                     } else {
                         let (def, name_str, _writable) = namespace.get_data_definition(&name);
                         match def {
                             Some(compiled) => compiled.storage,
-                            None => lookup_with_dlsym(name_str),
+                            None => self.lookup_symbol(name_str),
                         }
                     };
                     // TODO: Handle overflow.

From fa65ee7a6808dfc164ec58c09aa0a2def1d3dcd2 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Mon, 13 Aug 2018 21:23:25 +0200
Subject: [PATCH 1986/3084] Legalize bint.i8

---
 cranelift/filetests/isa/x86/legalize-bint-i8.clif | 10 ++++++++++
 lib/codegen/meta-python/base/legalize.py          |  9 +++++++++
 2 files changed, 19 insertions(+)
 create mode 100644 cranelift/filetests/isa/x86/legalize-bint-i8.clif

diff --git a/cranelift/filetests/isa/x86/legalize-bint-i8.clif b/cranelift/filetests/isa/x86/legalize-bint-i8.clif
new file mode 100644
index 0000000000..b2684ae105
--- /dev/null
+++ b/cranelift/filetests/isa/x86/legalize-bint-i8.clif
@@ -0,0 +1,10 @@
+test compile
+
+target x86_64
+
+function u0:0() -> i8 fast {
+ebb0:
+    v14 = bconst.b1 false
+    v15 = bint.i8 v14
+    return v15
+}
diff --git a/lib/codegen/meta-python/base/legalize.py b/lib/codegen/meta-python/base/legalize.py
index c63dde9289..cf66074809 100644
--- a/lib/codegen/meta-python/base/legalize.py
+++ b/lib/codegen/meta-python/base/legalize.py
@@ -257,6 +257,15 @@ for int_ty in [types.i8, types.i16]:
         )
     )
 
+for int_ty in [types.i8, types.i16]:
+    widen.legalize(
+        a << insts.bint.bind(int_ty)(b),
+        Rtl(
+            x << insts.bint.i32(b),
+            a << ireduce.bind(int_ty)(x)
+        )
+    )
+
 # Expand integer operations with carry for RISC architectures that don't have
 # the flags.
 expand.legalize(

From a044f58cea93c33b6db0cc9f49650206e5ea61f1 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 20 Jul 2018 19:14:04 +0200
Subject: [PATCH 1987/3084] Fixes #404: Use log.rs and a file-per-thread logger
 instead of the dbg! macro;

---
 cranelift/Cargo.toml                   |   2 +
 cranelift/src/clif-util.rs             |  23 +++--
 lib/codegen/Cargo.toml                 |   1 +
 lib/codegen/src/binemit/relaxation.rs  |   8 +-
 lib/codegen/src/binemit/shrink.rs      |   2 +-
 lib/codegen/src/dbg.rs                 | 131 +------------------------
 lib/codegen/src/ir/layout.rs           |   2 +-
 lib/codegen/src/legalizer/boundary.rs  |   6 +-
 lib/codegen/src/lib.rs                 |   8 +-
 lib/codegen/src/regalloc/coalescing.rs |  26 ++---
 lib/codegen/src/regalloc/coloring.rs   |  32 +++---
 lib/codegen/src/regalloc/reload.rs     |   4 +-
 lib/codegen/src/regalloc/solver.rs     |  30 +++---
 lib/codegen/src/regalloc/spilling.rs   |  24 +++--
 lib/codegen/src/timing.rs              |   4 +-
 lib/codegen/src/unreachable_code.rs    |   4 +-
 lib/filetests/Cargo.toml               |   2 +
 lib/filetests/src/concurrent.rs        |   6 +-
 lib/filetests/src/lib.rs               |   4 +-
 lib/filetests/src/runone.rs            |   4 +-
 lib/filetests/src/test_compile.rs      |   2 +-
 lib/module/Cargo.toml                  |   1 +
 lib/module/src/lib.rs                  |   3 +-
 lib/module/src/module.rs               |   2 +-
 lib/wasm/Cargo.toml                    |   1 +
 lib/wasm/src/func_translator.rs        |   8 +-
 lib/wasm/src/lib.rs                    |   4 +-
 27 files changed, 123 insertions(+), 221 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index b674fbf7f9..b3c7672c0c 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -34,6 +34,8 @@ term = "0.5.1"
 capstone = { version = "0.4", optional = true }
 wabt = { version = "0.4", optional = true }
 target-lexicon = "0.0.3"
+pretty_env_logger = "0.2.4"
+file-per-thread-logger = "0.1.1"
 
 [features]
 default = ["disas", "wasm"]
diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 0f7edd4ce3..86e892420a 100644
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -12,6 +12,7 @@
 extern crate cfg_if;
 extern crate cranelift_codegen;
 extern crate cranelift_filetests;
+extern crate file_per_thread_logger;
 extern crate cranelift_reader;
 extern crate docopt;
 extern crate filecheck;
@@ -19,6 +20,7 @@ extern crate filecheck;
 extern crate serde_derive;
 #[cfg(feature = "disas")]
 extern crate capstone;
+extern crate pretty_env_logger;
 extern crate term;
 
 cfg_if! {
@@ -31,6 +33,7 @@ cfg_if! {
 }
 extern crate target_lexicon;
 
+use cranelift_codegen::dbg::LOG_FILENAME_PREFIX;
 use cranelift_codegen::{timing, VERSION};
 use docopt::Docopt;
 use std::io::{self, Write};
@@ -46,16 +49,17 @@ const USAGE: &str = "
 Cranelift code generator utility
 
 Usage:
-    clif-util test [-vT] ...
-    clif-util cat ...
-    clif-util filecheck [-v] 
-    clif-util print-cfg ...
-    clif-util compile [-vpT] [--set ]... [--target ] ...
-    clif-util wasm [-ctvpTs] [--set ]... [--target ] ...
+    clif-util test [-vTd] ...
+    clif-util cat [-d] ...
+    clif-util filecheck [-vd] 
+    clif-util print-cfg [-d] ...
+    clif-util compile [-vpTd] [--set ]... [--target ] ...
+    clif-util wasm [-ctvpTsd] [--set ]... [--target ] ...
     clif-util --help | --version
 
 Options:
     -v, --verbose   be more verbose
+    -d, --debug     enable debug output on stderr/stdout
     -T, --time-passes
                     print pass timing report
     -t, --just-decode
@@ -90,6 +94,7 @@ struct Args {
     flag_target: String,
     flag_time_passes: bool,
     flag_print_size: bool,
+    flag_debug: bool,
 }
 
 /// A command either succeeds or fails with an error message.
@@ -106,6 +111,12 @@ fn clif_util() -> CommandResult {
         })
         .unwrap_or_else(|e| e.exit());
 
+    if args.flag_debug {
+        pretty_env_logger::init();
+    } else {
+        file_per_thread_logger::initialize(LOG_FILENAME_PREFIX);
+    }
+
     // Find the sub-command to execute.
     let result = if args.cmd_test {
         cranelift_filetests::run(args.flag_verbose, &args.arg_file).map(|_time| ())
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index bb767dd67f..70ab856cf3 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -16,6 +16,7 @@ failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 target-lexicon = { version = "0.0.3", default-features = false }
+log = { version = "0.4.3", default-features = false, features = ["release_max_level_warn"] }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
 # machine code. Integration tests that need external dependencies can be
diff --git a/lib/codegen/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs
index f23d3218f8..ad13dc21c1 100644
--- a/lib/codegen/src/binemit/relaxation.rs
+++ b/lib/codegen/src/binemit/relaxation.rs
@@ -140,7 +140,7 @@ fn relax_branch(
     isa: &TargetIsa,
 ) -> CodeOffset {
     let inst = cur.current_inst().unwrap();
-    dbg!(
+    debug!(
         "Relaxing [{}] {} for {:#x}-{:#x} range",
         encinfo.display(cur.func.encodings[inst]),
         cur.func.dfg.display_inst(inst, isa),
@@ -156,7 +156,7 @@ fn relax_branch(
         .find(|&enc| {
             let range = encinfo.branch_range(enc).expect("Branch with no range");
             if !range.contains(offset, dest_offset) {
-                dbg!("  trying [{}]: out of range", encinfo.display(enc));
+                debug!("  trying [{}]: out of range", encinfo.display(enc));
                 false
             } else if encinfo.operand_constraints(enc)
                 != encinfo.operand_constraints(cur.func.encodings[inst])
@@ -166,10 +166,10 @@ fn relax_branch(
                 // which the existing operands don't satisfy. We can't check for
                 // validity directly because we don't have a RegDiversions active so
                 // we don't know which registers are actually in use.
-                dbg!("  trying [{}]: constraints differ", encinfo.display(enc));
+                debug!("  trying [{}]: constraints differ", encinfo.display(enc));
                 false
             } else {
-                dbg!("  trying [{}]: OK", encinfo.display(enc));
+                debug!("  trying [{}]: OK", encinfo.display(enc));
                 true
             }
         }) {
diff --git a/lib/codegen/src/binemit/shrink.rs b/lib/codegen/src/binemit/shrink.rs
index 63b0329278..5027aeeeb7 100644
--- a/lib/codegen/src/binemit/shrink.rs
+++ b/lib/codegen/src/binemit/shrink.rs
@@ -54,7 +54,7 @@ pub fn shrink_instructions(func: &mut Function, isa: &TargetIsa) {
                 if best_enc != enc {
                     func.encodings[inst] = best_enc;
 
-                    dbg!(
+                    debug!(
                         "Shrunk [{}] to [{}] in {}, reducing the size from {} to {}",
                         encinfo.display(enc),
                         encinfo.display(best_enc),
diff --git a/lib/codegen/src/dbg.rs b/lib/codegen/src/dbg.rs
index 0b32bf2bf5..f19c222caf 100644
--- a/lib/codegen/src/dbg.rs
+++ b/lib/codegen/src/dbg.rs
@@ -1,133 +1,8 @@
-//! Debug tracing macros.
-//!
-//! This module defines the `dbg!` macro which works like `println!` except it writes to the
-//! Cranelift tracing output file if enabled.
-//!
-//! Tracing can be enabled by setting the `CRANELIFT_DBG` environment variable to something
-/// other than `0`.
-///
-/// The output will appear in files named `cranelift.dbg.*`, where the suffix is named after the
-/// thread doing the logging.
-#[cfg(feature = "std")]
-use std::cell::RefCell;
-#[cfg(feature = "std")]
-use std::env;
-#[cfg(feature = "std")]
-use std::ffi::OsStr;
+//! Debug tracing helpers.
 use std::fmt;
-#[cfg(feature = "std")]
-use std::fs::File;
-#[cfg(feature = "std")]
-use std::io::{self, Write};
-#[cfg(feature = "std")]
-use std::sync::atomic;
-#[cfg(feature = "std")]
-use std::thread;
 
-#[cfg(feature = "std")]
-static STATE: atomic::AtomicIsize = atomic::ATOMIC_ISIZE_INIT;
-
-/// Is debug tracing enabled?
-///
-/// Debug tracing can be enabled by setting the `CRANELIFT_DBG` environment variable to something
-/// other than `0`.
-///
-/// This inline function turns into a constant `false` when debug assertions are disabled.
-#[cfg(feature = "std")]
-#[inline]
-pub fn enabled() -> bool {
-    if cfg!(debug_assertions) {
-        match STATE.load(atomic::Ordering::Relaxed) {
-            0 => initialize(),
-            s => s > 0,
-        }
-    } else {
-        false
-    }
-}
-
-/// Does nothing
-#[cfg(not(feature = "std"))]
-#[inline]
-pub fn enabled() -> bool {
-    false
-}
-
-/// Initialize `STATE` from the environment variable.
-#[cfg(feature = "std")]
-fn initialize() -> bool {
-    let enable = match env::var_os("CRANELIFT_DBG") {
-        Some(s) => s != OsStr::new("0"),
-        None => false,
-    };
-
-    if enable {
-        STATE.store(1, atomic::Ordering::Relaxed);
-    } else {
-        STATE.store(-1, atomic::Ordering::Relaxed);
-    }
-
-    enable
-}
-
-#[cfg(feature = "std")]
-thread_local! {
-    static WRITER : RefCell> = RefCell::new(open_file());
-}
-
-/// Write a line with the given format arguments.
-///
-/// This is for use by the `dbg!` macro.
-#[cfg(feature = "std")]
-pub fn writeln_with_format_args(args: fmt::Arguments) -> io::Result<()> {
-    WRITER.with(|rc| {
-        let mut w = rc.borrow_mut();
-        writeln!(*w, "{}", args)?;
-        w.flush()
-    })
-}
-
-/// Open the tracing file for the current thread.
-#[cfg(feature = "std")]
-fn open_file() -> io::BufWriter {
-    let curthread = thread::current();
-    let tmpstr;
-    let mut path = "cranelift.dbg.".to_owned();
-    path.extend(
-        match curthread.name() {
-            Some(name) => name.chars(),
-            // The thread is unnamed, so use the thread ID instead.
-            None => {
-                tmpstr = format!("{:?}", curthread.id());
-                tmpstr.chars()
-            }
-        }.filter(|ch| ch.is_alphanumeric() || *ch == '-' || *ch == '_'),
-    );
-    let file = File::create(path).expect("Can't open tracing file");
-    io::BufWriter::new(file)
-}
-
-/// Write a line to the debug trace file if tracing is enabled.
-///
-/// Arguments are the same as for `printf!`.
-#[cfg(feature = "std")]
-#[macro_export]
-macro_rules! dbg {
-    ($($arg:tt)+) => {
-        if $crate::dbg::enabled() {
-            // Drop the error result so we don't get compiler errors for ignoring it.
-            // What are you going to do, log the error?
-            $crate::dbg::writeln_with_format_args(format_args!($($arg)+)).ok();
-        }
-    }
-}
-
-/// `dbg!` isn't supported in `no_std` mode, so expand it into nothing.
-#[cfg(not(feature = "std"))]
-#[macro_export]
-macro_rules! dbg {
-    ($($arg:tt)+) => {};
-}
+/// Prefix added to the log file names, just before the thread name or id.
+pub static LOG_FILENAME_PREFIX: &str = "cranelift.dbg.";
 
 /// Helper for printing lists.
 pub struct DisplayList<'a, T>(pub &'a [T])
diff --git a/lib/codegen/src/ir/layout.rs b/lib/codegen/src/ir/layout.rs
index 9f03969061..a9dc08d786 100644
--- a/lib/codegen/src/ir/layout.rs
+++ b/lib/codegen/src/ir/layout.rs
@@ -316,7 +316,7 @@ impl Layout {
                 next_inst = self.insts[inst].next.expand();
             }
         }
-        dbg!("Renumbered {} program points", seq / MAJOR_STRIDE);
+        debug!("Renumbered {} program points", seq / MAJOR_STRIDE);
     }
 }
 
diff --git a/lib/codegen/src/legalizer/boundary.rs b/lib/codegen/src/legalizer/boundary.rs
index 640c3a82b0..6f3143028b 100644
--- a/lib/codegen/src/legalizer/boundary.rs
+++ b/lib/codegen/src/legalizer/boundary.rs
@@ -270,7 +270,7 @@ where
     // Reconstruct how `ty` was legalized into the `arg_type` argument.
     let conversion = legalize_abi_value(ty, &arg_type);
 
-    dbg!("convert_from_abi({}): {:?}", ty, conversion);
+    debug!("convert_from_abi({}): {:?}", ty, conversion);
 
     // The conversion describes value to ABI argument. We implement the reverse conversion here.
     match conversion {
@@ -279,7 +279,7 @@ where
             let abi_ty = ty.half_width().expect("Invalid type for conversion");
             let lo = convert_from_abi(pos, abi_ty, None, get_arg);
             let hi = convert_from_abi(pos, abi_ty, None, get_arg);
-            dbg!(
+            debug!(
                 "intsplit {}: {}, {}: {}",
                 lo,
                 pos.func.dfg.value_type(lo),
@@ -586,7 +586,7 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph
     // the legalized signature. These values should simply be propagated from the entry block
     // arguments.
     if special_args > 0 {
-        dbg!(
+        debug!(
             "Adding {} special-purpose arguments to {}",
             special_args,
             pos.func.dfg.display_inst(inst, None)
diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index 5d24ddf709..5ae6812a52 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -49,6 +49,10 @@ extern crate failure_derive;
 #[cfg_attr(test, macro_use)]
 extern crate target_lexicon;
 
+#[cfg(feature = "std")]
+#[macro_use]
+extern crate log;
+
 pub use context::Context;
 pub use legalizer::legalize_function;
 pub use verifier::verify_function;
@@ -60,13 +64,11 @@ pub const VERSION: &str = env!("CARGO_PKG_VERSION");
 #[macro_use]
 pub extern crate cranelift_entity as entity;
 
-#[macro_use]
-pub mod dbg;
-
 pub mod bforest;
 pub mod binemit;
 pub mod cfg_printer;
 pub mod cursor;
+pub mod dbg;
 pub mod dominator_tree;
 pub mod flowgraph;
 pub mod ir;
diff --git a/lib/codegen/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs
index b9415524a0..885e0b82fe 100644
--- a/lib/codegen/src/regalloc/coalescing.rs
+++ b/lib/codegen/src/regalloc/coalescing.rs
@@ -116,7 +116,7 @@ impl Coalescing {
         virtregs: &mut VirtRegs,
     ) {
         let _tt = timing::ra_cssa();
-        dbg!("Coalescing for:\n{}", func.display(isa));
+        debug!("Coalescing for:\n{}", func.display(isa));
         self.preorder.compute(domtree, &func.layout);
         let mut context = Context {
             isa,
@@ -185,7 +185,7 @@ impl<'a> Context<'a> {
                 continue;
             }
 
-            dbg!(
+            debug!(
                 " - checking {} params at back-edge {}: {}",
                 num_params,
                 pred_ebb,
@@ -229,7 +229,7 @@ impl<'a> Context<'a> {
                 if Some(def_ebb) == self.func.layout.entry_block()
                     && self.func.signature.params[def_num].location.is_stack()
                 {
-                    dbg!("-> isolating function stack parameter {}", arg);
+                    debug!("-> isolating function stack parameter {}", arg);
                     let new_arg = self.isolate_arg(pred_ebb, pred_inst, argnum, arg);
                     self.virtregs.union(param, new_arg);
                     continue;
@@ -296,7 +296,7 @@ impl<'a> Context<'a> {
         let inst = pos.built_inst();
         self.liveness.move_def_locally(param, inst);
 
-        dbg!(
+        debug!(
             "-> inserted {}, following {}({}: {})",
             pos.display_inst(inst),
             ebb,
@@ -365,7 +365,7 @@ impl<'a> Context<'a> {
 
         pos.func.dfg.inst_variable_args_mut(pred_inst)[argnum] = copy;
 
-        dbg!(
+        debug!(
             "-> inserted {}, before {}: {}",
             pos.display_inst(inst),
             pred_ebb,
@@ -381,7 +381,7 @@ impl<'a> Context<'a> {
     /// closure of the relation formed by EBB parameter-argument pairs found by `union_find_ebb()`.
     fn finish_union_find(&mut self) {
         self.virtregs.finish_union_find(None);
-        dbg!("After union-find phase:{}", self.virtregs);
+        debug!("After union-find phase:{}", self.virtregs);
     }
 }
 
@@ -412,7 +412,7 @@ impl<'a> Context<'a> {
     fn check_vreg(&mut self, vreg: VirtReg) -> bool {
         // Order the values according to the dominator pre-order of their definition.
         let values = self.virtregs.sort_values(vreg, self.func, self.preorder);
-        dbg!("Checking {} = {}", vreg, DisplayList(values));
+        debug!("Checking {} = {}", vreg, DisplayList(values));
 
         // Now push the values in order to the dominator forest.
         // This gives us the closest dominating value def for each of the values.
@@ -434,7 +434,7 @@ impl<'a> Context<'a> {
             let ctx = self.liveness.context(&self.func.layout);
             if self.liveness[parent.value].overlaps_def(node.def, node.ebb, ctx) {
                 // The two values are interfering, so they can't be in the same virtual register.
-                dbg!("-> interference: {} overlaps def of {}", parent, value);
+                debug!("-> interference: {} overlaps def of {}", parent, value);
                 return false;
             }
         }
@@ -458,7 +458,7 @@ impl<'a> Context<'a> {
             self.cfg,
             self.preorder,
         );
-        dbg!(
+        debug!(
             "Synthesizing {} from {} branches and params {}",
             vreg,
             self.vcopies.branches.len(),
@@ -546,7 +546,7 @@ impl<'a> Context<'a> {
         }
 
         let _vreg = self.virtregs.unify(self.values);
-        dbg!("-> merged into {} = {}", _vreg, DisplayList(self.values));
+        debug!("-> merged into {} = {}", _vreg, DisplayList(self.values));
         true
     }
 
@@ -568,7 +568,7 @@ impl<'a> Context<'a> {
         // registers and the filtered virtual copies.
         let v0 = self.virtregs.congruence_class(¶m);
         let v1 = self.virtregs.congruence_class(&arg);
-        dbg!(
+        debug!(
             " - set 0: {}\n - set 1: {}",
             DisplayList(v0),
             DisplayList(v1)
@@ -621,7 +621,7 @@ impl<'a> Context<'a> {
                 if node.set_id != parent.set_id
                     && self.liveness[parent.value].reaches_use(inst, node.ebb, ctx)
                 {
-                    dbg!(
+                    debug!(
                         " - interference: {} overlaps vcopy at {}:{}",
                         parent,
                         node.ebb,
@@ -646,7 +646,7 @@ impl<'a> Context<'a> {
                 && self.liveness[parent.value].overlaps_def(node.def, node.ebb, ctx)
             {
                 // The two values are interfering.
-                dbg!(" - interference: {} overlaps def of {}", parent, node.value);
+                debug!(" - interference: {} overlaps def of {}", parent, node.value);
                 return false;
             }
         }
diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs
index b824246d5b..56691eecf3 100644
--- a/lib/codegen/src/regalloc/coloring.rs
+++ b/lib/codegen/src/regalloc/coloring.rs
@@ -124,7 +124,7 @@ impl Coloring {
         tracker: &mut LiveValueTracker,
     ) {
         let _tt = timing::ra_coloring();
-        dbg!("Coloring for:\n{}", func.display(isa));
+        debug!("Coloring for:\n{}", func.display(isa));
         let mut ctx = Context {
             usable_regs: isa.allocatable_registers(func),
             cur: EncCursor::new(func, isa),
@@ -156,7 +156,7 @@ impl<'a> Context<'a> {
 
     /// Visit `ebb`, assuming that the immediate dominator has already been visited.
     fn visit_ebb(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) {
-        dbg!("Coloring {}:", ebb);
+        debug!("Coloring {}:", ebb);
         let mut regs = self.visit_ebb_header(ebb, tracker);
         tracker.drop_dead_params();
         self.divert.clear();
@@ -216,7 +216,7 @@ impl<'a> Context<'a> {
         let mut regs = AvailableRegs::new(&self.usable_regs);
 
         for lv in live.iter().filter(|lv| !lv.is_dead) {
-            dbg!(
+            debug!(
                 "Live-in: {}:{} in {}",
                 lv.value,
                 lv.affinity.display(&self.reginfo),
@@ -294,7 +294,7 @@ impl<'a> Context<'a> {
         tracker: &mut LiveValueTracker,
         regs: &mut AvailableRegs,
     ) -> bool {
-        dbg!(
+        debug!(
             "Coloring {}\n    from {}",
             self.cur.display_inst(inst),
             regs.input.display(&self.reginfo),
@@ -362,7 +362,7 @@ impl<'a> Context<'a> {
             if let Affinity::Reg(rci) = lv.affinity {
                 let rc = self.reginfo.rc(rci);
                 let reg = self.divert.reg(lv.value, &self.cur.func.locations);
-                dbg!(
+                debug!(
                     "    kill {} in {} ({} {})",
                     lv.value,
                     self.reginfo.display_regunit(reg),
@@ -380,7 +380,7 @@ impl<'a> Context<'a> {
         }
 
         // This aligns with the "    from" line at the top of the function.
-        dbg!("    glob {}", regs.global.display(&self.reginfo));
+        debug!("    glob {}", regs.global.display(&self.reginfo));
 
         // This flag is set when the solver failed to find a solution for the global defines that
         // doesn't interfere with `regs.global`. We need to rewrite all of `inst`s global defines
@@ -419,7 +419,7 @@ impl<'a> Context<'a> {
         // Finally, we've fully programmed the constraint solver.
         // We expect a quick solution in most cases.
         let output_regs = self.solver.quick_solve(®s.global).unwrap_or_else(|_| {
-            dbg!("quick_solve failed for {}", self.solver);
+            debug!("quick_solve failed for {}", self.solver);
             self.iterate_solution(throughs, ®s.global, &mut replace_global_defines)
         });
 
@@ -454,7 +454,7 @@ impl<'a> Context<'a> {
         regs.input = output_regs;
         for lv in defs {
             let loc = self.cur.func.locations[lv.value];
-            dbg!(
+            debug!(
                 "    color {} -> {}{}",
                 lv.value,
                 loc.display(&self.reginfo),
@@ -700,7 +700,7 @@ impl<'a> Context<'a> {
                 ConstraintKind::FixedReg(reg) | ConstraintKind::FixedTied(reg) => {
                     self.add_fixed_output(lv.value, op.regclass, reg, throughs);
                     if !lv.is_local && !global_regs.is_avail(op.regclass, reg) {
-                        dbg!(
+                        debug!(
                             "Fixed output {} in {}:{} is not available in global regs",
                             lv.value,
                             op.regclass,
@@ -736,7 +736,7 @@ impl<'a> Context<'a> {
                     let rc = self.reginfo.rc(rci);
                     self.add_fixed_output(lv.value, rc, reg, throughs);
                     if !lv.is_local && !global_regs.is_avail(rc, reg) {
-                        dbg!(
+                        debug!(
                             "ABI output {} in {}:{} is not available in global regs",
                             lv.value,
                             rc,
@@ -812,7 +812,7 @@ impl<'a> Context<'a> {
                         // We need to make sure that fixed output register is compatible with the
                         // global register set.
                         if !lv.is_local && !global_regs.is_avail(op.regclass, reg) {
-                            dbg!(
+                            debug!(
                                 "Tied output {} in {}:{} is not available in global regs",
                                 lv.value,
                                 op.regclass,
@@ -848,7 +848,7 @@ impl<'a> Context<'a> {
                     debug_assert!(added, "Ran out of registers in {}", rc);
                 }
                 Err(SolverError::Global(_value)) => {
-                    dbg!(
+                    debug!(
                         "Not enough global registers for {}, trying as local",
                         _value
                     );
@@ -863,7 +863,7 @@ impl<'a> Context<'a> {
 
     /// Try to add an `rc` variable to the solver from the `throughs` set.
     fn try_add_var(&mut self, rc: RegClass, throughs: &[LiveValue]) -> bool {
-        dbg!("Trying to add a {} reg from {} values", rc, throughs.len());
+        debug!("Trying to add a {} reg from {} values", rc, throughs.len());
 
         for lv in throughs {
             if let Affinity::Reg(rci) = lv.affinity {
@@ -995,7 +995,7 @@ impl<'a> Context<'a> {
     /// the constraints on the instruction operands.
     ///
     fn replace_global_defines(&mut self, inst: Inst, tracker: &mut LiveValueTracker) {
-        dbg!("Replacing global defs on {}", self.cur.display_inst(inst));
+        debug!("Replacing global defs on {}", self.cur.display_inst(inst));
 
         // We'll insert copies *after `inst`. Our caller will move the cursor back.
         self.cur.next_inst();
@@ -1042,14 +1042,14 @@ impl<'a> Context<'a> {
             lv.endpoint = copy;
             lv.is_local = true;
 
-            dbg!(
+            debug!(
                 "  + {} with {} in {}",
                 self.cur.display_inst(copy),
                 local,
                 loc.display(&self.reginfo)
             );
         }
-        dbg!("Done: {}", self.cur.display_inst(inst));
+        debug!("Done: {}", self.cur.display_inst(inst));
     }
 
     /// Process kills on a ghost instruction.
diff --git a/lib/codegen/src/regalloc/reload.rs b/lib/codegen/src/regalloc/reload.rs
index a712616af8..f07f35cea0 100644
--- a/lib/codegen/src/regalloc/reload.rs
+++ b/lib/codegen/src/regalloc/reload.rs
@@ -72,7 +72,7 @@ impl Reload {
         tracker: &mut LiveValueTracker,
     ) {
         let _tt = timing::ra_reload();
-        dbg!("Reload for:\n{}", func.display(isa));
+        debug!("Reload for:\n{}", func.display(isa));
         let mut ctx = Context {
             cur: EncCursor::new(func, isa),
             encinfo: isa.encoding_info(),
@@ -119,7 +119,7 @@ impl<'a> Context<'a> {
     }
 
     fn visit_ebb(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) {
-        dbg!("Reloading {}:", ebb);
+        debug!("Reloading {}:", ebb);
         self.visit_ebb_header(ebb, tracker);
         tracker.drop_dead_params();
 
diff --git a/lib/codegen/src/regalloc/solver.rs b/lib/codegen/src/regalloc/solver.rs
index 1140aa7536..283a461378 100644
--- a/lib/codegen/src/regalloc/solver.rs
+++ b/lib/codegen/src/regalloc/solver.rs
@@ -532,7 +532,7 @@ impl Solver {
     /// In either case, `to` will not be available for variables on the input side of the
     /// instruction.
     pub fn reassign_in(&mut self, value: Value, rc: RegClass, from: RegUnit, to: RegUnit) {
-        dbg!(
+        debug!(
             "reassign_in({}:{}, {} -> {})",
             value,
             rc,
@@ -545,7 +545,7 @@ impl Solver {
             // added as a variable previously. A fixed constraint beats a variable, so convert it.
             if let Some(idx) = self.vars.iter().position(|v| v.value == value) {
                 let v = self.vars.remove(idx);
-                dbg!("-> converting variable {} to a fixed constraint", v);
+                debug!("-> converting variable {} to a fixed constraint", v);
                 // The spiller is responsible for ensuring that all constraints on the uses of a
                 // value are compatible.
                 debug_assert!(
@@ -578,7 +578,7 @@ impl Solver {
     /// This function can only be used before calling `inputs_done()`. Afterwards, more input-side
     /// variables can be added by calling `add_killed_var()` and `add_through_var()`
     pub fn add_var(&mut self, value: Value, constraint: RegClass, from: RegUnit) {
-        dbg!(
+        debug!(
             "add_var({}:{}, from={})",
             value,
             constraint,
@@ -593,7 +593,7 @@ impl Solver {
     ///
     /// This function should be called after `inputs_done()` only. Use `add_var()` before.
     pub fn add_killed_var(&mut self, value: Value, constraint: RegClass, from: RegUnit) {
-        dbg!(
+        debug!(
             "add_killed_var({}:{}, from={})",
             value,
             constraint,
@@ -608,7 +608,7 @@ impl Solver {
     ///
     /// This function should be called after `inputs_done()` only. Use `add_var()` before.
     pub fn add_through_var(&mut self, value: Value, constraint: RegClass, from: RegUnit) {
-        dbg!(
+        debug!(
             "add_through_var({}:{}, from={})",
             value,
             constraint,
@@ -635,7 +635,7 @@ impl Solver {
             if let Some(v) = self.vars.iter_mut().find(|v| v.value == value) {
                 // We have an existing variable entry for `value`. Combine the constraints.
                 if let Some(rc) = v.constraint.intersect(constraint) {
-                    dbg!("-> combining constraint with {} yields {}", v, rc);
+                    debug!("-> combining constraint with {} yields {}", v, rc);
                     v.constraint = rc;
                     return;
                 } else {
@@ -647,7 +647,7 @@ impl Solver {
 
             // No variable, then it must be a fixed reassignment.
             if let Some(a) = self.assignments.get(value) {
-                dbg!("-> already fixed assignment {}", a);
+                debug!("-> already fixed assignment {}", a);
                 debug_assert!(
                     constraint.contains(a.to),
                     "Incompatible constraints for {}",
@@ -656,12 +656,12 @@ impl Solver {
                 return;
             }
 
-            dbg!("{}", self);
+            debug!("{}", self);
             panic!("Wrong from register for {}", value);
         }
 
         let new_var = Variable::new_live(value, constraint, from, live_through);
-        dbg!("-> new var: {}", new_var);
+        debug!("-> new var: {}", new_var);
 
         self.regs_in.free(constraint, from);
         if self.inputs_done && live_through {
@@ -777,7 +777,7 @@ impl Solver {
         if is_global {
             let mut new_var = Variable::new_live(value, rc, reg, true);
             new_var.is_global = true;
-            dbg!("add_tied_input: new tied-global value: {}", new_var);
+            debug!("add_tied_input: new tied-global value: {}", new_var);
             self.vars.push(new_var);
             self.regs_in.free(rc, reg);
         } else {
@@ -899,7 +899,7 @@ impl Solver {
             )
         });
 
-        dbg!("real_solve for {}", self);
+        debug!("real_solve for {}", self);
         self.find_solution(global_regs)
     }
 
@@ -982,7 +982,7 @@ impl Solver {
             .extend(self.assignments.values().filter_map(Move::with_assignment));
 
         if !(self.moves.is_empty()) {
-            dbg!("collect_moves: {}", DisplayList(&self.moves));
+            debug!("collect_moves: {}", DisplayList(&self.moves));
         }
     }
 
@@ -1024,7 +1024,7 @@ impl Solver {
                 if let Some((rc, reg)) = m.from_reg() {
                     avail.free(rc, reg);
                 }
-                dbg!("move #{}: {}", i, m);
+                debug!("move #{}: {}", i, m);
                 i += 1;
                 continue;
             }
@@ -1058,7 +1058,7 @@ impl Solver {
             let m = self.moves[i].clone();
             let toprc = m.rc().toprc();
             if let Some(reg) = avail.iter(toprc).next() {
-                dbg!(
+                debug!(
                     "breaking cycle at {} with available {} register {}",
                     m,
                     toprc,
@@ -1089,7 +1089,7 @@ impl Solver {
             // a last resort.
             let slot = num_spill_slots;
             num_spill_slots += 1;
-            dbg!("breaking cycle at {} with slot {}", m, slot);
+            debug!("breaking cycle at {} with slot {}", m, slot);
             let old_to_reg = self.moves[i].change_to_spill(slot);
             self.fills.push(Move::Fill {
                 value: m.value(),
diff --git a/lib/codegen/src/regalloc/spilling.rs b/lib/codegen/src/regalloc/spilling.rs
index 457b07c663..d1244c0774 100644
--- a/lib/codegen/src/regalloc/spilling.rs
+++ b/lib/codegen/src/regalloc/spilling.rs
@@ -90,7 +90,7 @@ impl Spilling {
         tracker: &mut LiveValueTracker,
     ) {
         let _tt = timing::ra_spilling();
-        dbg!("Spilling for:\n{}", func.display(isa));
+        debug!("Spilling for:\n{}", func.display(isa));
         let reginfo = isa.register_info();
         let usable_regs = isa.allocatable_registers(func);
         let mut ctx = Context {
@@ -118,7 +118,7 @@ impl<'a> Context<'a> {
     }
 
     fn visit_ebb(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) {
-        dbg!("Spilling {}:", ebb);
+        debug!("Spilling {}:", ebb);
         self.cur.goto_top(ebb);
         self.visit_ebb_header(ebb, tracker);
         tracker.drop_dead_params();
@@ -198,14 +198,12 @@ impl<'a> Context<'a> {
             if let Affinity::Reg(rci) = lv.affinity {
                 let rc = self.reginfo.rc(rci);
                 'try_take: while let Err(mask) = self.pressure.take_transient(rc) {
-                    dbg!("Need {} reg for EBB param {}", rc, lv.value);
+                    debug!("Need {} reg for EBB param {}", rc, lv.value);
                     match self.spill_candidate(mask, liveins) {
                         Some(cand) => {
-                            dbg!(
+                            debug!(
                                 "Spilling live-in {} to make room for {} EBB param {}",
-                                cand,
-                                rc,
-                                lv.value
+                                cand, rc, lv.value
                             );
                             self.spill_reg(cand);
                         }
@@ -213,7 +211,7 @@ impl<'a> Context<'a> {
                             // We can't spill any of the live-in registers, so we have to spill an
                             // EBB argument. Since the current spill metric would consider all the
                             // EBB arguments equal, just spill the present register.
-                            dbg!("Spilling {} EBB argument {}", rc, lv.value);
+                            debug!("Spilling {} EBB argument {}", rc, lv.value);
 
                             // Since `spill_reg` will free a register, add the current one here.
                             self.pressure.take(rc);
@@ -237,7 +235,7 @@ impl<'a> Context<'a> {
         constraints: &RecipeConstraints,
         tracker: &mut LiveValueTracker,
     ) {
-        dbg!("Inst {}, {}", self.cur.display_inst(inst), self.pressure);
+        debug!("Inst {}, {}", self.cur.display_inst(inst), self.pressure);
         debug_assert_eq!(self.cur.current_inst(), Some(inst));
         debug_assert_eq!(self.cur.current_ebb(), Some(ebb));
 
@@ -279,7 +277,7 @@ impl<'a> Context<'a> {
             if op.kind != ConstraintKind::Stack {
                 // Add register def to pressure, spill if needed.
                 while let Err(mask) = self.pressure.take_transient(op.regclass) {
-                    dbg!("Need {} reg from {} throughs", op.regclass, throughs.len());
+                    debug!("Need {} reg from {} throughs", op.regclass, throughs.len());
                     match self.spill_candidate(mask, throughs) {
                         Some(cand) => self.spill_reg(cand),
                         None => panic!(
@@ -333,7 +331,7 @@ impl<'a> Context<'a> {
 
             // Only collect the interesting register uses.
             if reguse.fixed || reguse.tied || reguse.spilled {
-                dbg!("  reguse: {}", reguse);
+                debug!("  reguse: {}", reguse);
                 self.reg_uses.push(reguse);
             }
         }
@@ -406,7 +404,7 @@ impl<'a> Context<'a> {
             if need_copy || ru.spilled {
                 let rc = self.reginfo.rc(ru.rci);
                 while let Err(mask) = self.pressure.take_transient(rc) {
-                    dbg!("Copy of {} reg causes spill", rc);
+                    debug!("Copy of {} reg causes spill", rc);
                     // Spill a live register that is *not* used by the current instruction.
                     // Spilling a use wouldn't help.
                     //
@@ -489,7 +487,7 @@ impl<'a> Context<'a> {
             let rc = self.reginfo.rc(rci);
             self.pressure.free(rc);
             self.spills.push(value);
-            dbg!("Spilled {}:{} -> {}", value, rc, self.pressure);
+            debug!("Spilled {}:{} -> {}", value, rc, self.pressure);
         } else {
             panic!("Cannot spill {} that was already on the stack", value);
         }
diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs
index 707efbf3f8..b340796578 100644
--- a/lib/codegen/src/timing.rs
+++ b/lib/codegen/src/timing.rs
@@ -179,7 +179,7 @@ mod details {
     /// This function is called by the publicly exposed pass functions.
     pub(super) fn start_pass(pass: Pass) -> TimingToken {
         let prev = CURRENT_PASS.with(|p| p.replace(pass));
-        dbg!("timing: Starting {}, (during {})", pass, prev);
+        debug!("timing: Starting {}, (during {})", pass, prev);
         TimingToken {
             start: Instant::now(),
             pass,
@@ -191,7 +191,7 @@ mod details {
     impl Drop for TimingToken {
         fn drop(&mut self) {
             let duration = self.start.elapsed();
-            dbg!("timing: Ending {}", self.pass);
+            debug!("timing: Ending {}", self.pass);
             let old_cur = CURRENT_PASS.with(|p| p.replace(self.prev));
             debug_assert_eq!(self.pass, old_cur, "Timing tokens dropped out of order");
             PASS_TIME.with(|rc| {
diff --git a/lib/codegen/src/unreachable_code.rs b/lib/codegen/src/unreachable_code.rs
index 36ec7e673c..74cc6f4ea4 100644
--- a/lib/codegen/src/unreachable_code.rs
+++ b/lib/codegen/src/unreachable_code.rs
@@ -24,14 +24,14 @@ pub fn eliminate_unreachable_code(
             continue;
         }
 
-        dbg!("Eliminating unreachable {}", ebb);
+        debug!("Eliminating unreachable {}", ebb);
         // Move the cursor out of the way and make sure the next lop iteration goes to the right
         // EBB.
         pos.prev_ebb();
 
         // Remove all instructions from `ebb`.
         while let Some(inst) = pos.func.layout.first_inst(ebb) {
-            dbg!(" - {}", pos.func.dfg.display_inst(inst, None));
+            debug!(" - {}", pos.func.dfg.display_inst(inst, None));
             pos.func.layout.remove_inst(inst);
         }
 
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index ac66ca660c..487675971f 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -11,5 +11,7 @@ publish = false
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.18.1" }
 cranelift-reader = { path = "../reader", version = "0.18.1" }
+file-per-thread-logger = "0.1.1"
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
+log = "0.4.3"
diff --git a/lib/filetests/src/concurrent.rs b/lib/filetests/src/concurrent.rs
index f874e952f2..104967d757 100644
--- a/lib/filetests/src/concurrent.rs
+++ b/lib/filetests/src/concurrent.rs
@@ -3,7 +3,9 @@
 //! This module provides the `ConcurrentRunner` struct which uses a pool of threads to run tests
 //! concurrently.
 
+use cranelift_codegen::dbg::LOG_FILENAME_PREFIX;
 use cranelift_codegen::timing;
+use file_per_thread_logger;
 use num_cpus;
 use std::panic::catch_unwind;
 use std::path::{Path, PathBuf};
@@ -100,6 +102,7 @@ fn heartbeat_thread(replies: Sender) -> thread::JoinHandle<()> {
     thread::Builder::new()
         .name("heartbeat".to_string())
         .spawn(move || {
+            file_per_thread_logger::initialize(LOG_FILENAME_PREFIX);
             while replies.send(Reply::Tick).is_ok() {
                 thread::sleep(Duration::from_secs(1));
             }
@@ -116,6 +119,7 @@ fn worker_thread(
     thread::Builder::new()
         .name(format!("worker #{}", thread_num))
         .spawn(move || {
+            file_per_thread_logger::initialize(LOG_FILENAME_PREFIX);
             loop {
                 // Lock the mutex only long enough to extract a request.
                 let Request(jobid, path) = match requests.lock().unwrap().recv() {
@@ -140,7 +144,7 @@ fn worker_thread(
                 });
 
                 if let Err(ref msg) = result {
-                    dbg!("FAIL: {}", msg);
+                    error!("FAIL: {}", msg);
                 }
 
                 replies.send(Reply::Done { jobid, result }).unwrap();
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index cdeac02df9..b052253e67 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -18,11 +18,13 @@
     )
 )]
 
-#[macro_use(dbg)]
 extern crate cranelift_codegen;
+extern crate file_per_thread_logger;
 extern crate cranelift_reader;
 extern crate filecheck;
 extern crate num_cpus;
+#[macro_use]
+extern crate log;
 
 use cranelift_reader::TestCommand;
 use runner::TestRunner;
diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs
index 8a28755003..b98f5f8cfb 100644
--- a/lib/filetests/src/runone.rs
+++ b/lib/filetests/src/runone.rs
@@ -29,7 +29,7 @@ fn read_to_string>(path: P) -> io::Result {
 /// If running this test causes a panic, it will propagate as normal.
 pub fn run(path: &Path) -> TestResult {
     let _tt = timing::process_file();
-    dbg!("---\nFile: {}", path.to_string_lossy());
+    info!("---\nFile: {}", path.to_string_lossy());
     let started = time::Instant::now();
     let buffer = read_to_string(path).map_err(|e| e.to_string())?;
     let testfile = parse_test(&buffer).map_err(|e| e.to_string())?;
@@ -122,7 +122,7 @@ fn run_one_test<'a>(
 ) -> SubtestResult<()> {
     let (test, flags, isa) = tuple;
     let name = format!("{}({})", test.name(), func.name);
-    dbg!("Test: {} {}", name, isa.map_or("-", TargetIsa::name));
+    info!("Test: {} {}", name, isa.map_or("-", TargetIsa::name));
 
     context.flags = flags;
     context.isa = isa;
diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs
index 0e79bab429..1fb6d0e4c0 100644
--- a/lib/filetests/src/test_compile.rs
+++ b/lib/filetests/src/test_compile.rs
@@ -41,7 +41,7 @@ impl SubTest for TestCompile {
             .compile(isa)
             .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?;
 
-        dbg!(
+        info!(
             "Generated {} bytes of code:\n{}",
             code_size,
             comp_ctx.func.display(isa)
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index d8a2bf9ccb..a137bcebd0 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -13,6 +13,7 @@ cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features
 cranelift-entity = { path = "../entity", version = "0.18.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
+log = { version = "0.4.3", default-features = false, features = ["release_max_level_warn"] }
 
 [features]
 default = ["std"]
diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs
index 32b774824a..a7676c4a8c 100644
--- a/lib/module/src/lib.rs
+++ b/lib/module/src/lib.rs
@@ -20,12 +20,13 @@
 #[cfg_attr(test, macro_use)]
 extern crate alloc;
 
-#[macro_use]
 extern crate cranelift_codegen;
 #[macro_use]
 extern crate cranelift_entity;
 #[macro_use]
 extern crate failure;
+#[macro_use]
+extern crate log;
 
 mod backend;
 mod data_context;
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index d5c616db66..133ad5af9f 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -467,7 +467,7 @@ where
     pub fn define_function(&mut self, func: FuncId, ctx: &mut Context) -> ModuleResult<()> {
         let compiled = {
             let code_size = ctx.compile(self.backend.isa()).map_err(|e| {
-                dbg!(
+                info!(
                     "defining function {}: {}",
                     func,
                     ctx.func.display(self.backend.isa())
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index abc33510a1..4e40b5022d 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -17,6 +17,7 @@ hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
+log = { version = "0.4.3", default-features = false, features = ["release_max_level_warn"] }
 
 [dev-dependencies]
 wabt = "0.4"
diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs
index 8f764c102a..f9ab937ad4 100644
--- a/lib/wasm/src/func_translator.rs
+++ b/lib/wasm/src/func_translator.rs
@@ -67,7 +67,7 @@ impl FuncTranslator {
         environ: &mut FE,
     ) -> WasmResult<()> {
         let _tt = timing::wasm_translate_function();
-        dbg!(
+        info!(
             "translate({} bytes, {}{})",
             reader.bytes_remaining(),
             func.name,
@@ -265,7 +265,7 @@ mod tests {
         trans
             .translate(&BODY, &mut ctx.func, &mut runtime.func_env())
             .unwrap();
-        dbg!("{}", ctx.func.display(None));
+        debug!("{}", ctx.func.display(None));
         ctx.verify(runtime.func_env().flags()).unwrap();
     }
 
@@ -296,7 +296,7 @@ mod tests {
         trans
             .translate(&BODY, &mut ctx.func, &mut runtime.func_env())
             .unwrap();
-        dbg!("{}", ctx.func.display(None));
+        debug!("{}", ctx.func.display(None));
         ctx.verify(runtime.func_env().flags()).unwrap();
     }
 
@@ -335,7 +335,7 @@ mod tests {
         trans
             .translate(&BODY, &mut ctx.func, &mut runtime.func_env())
             .unwrap();
-        dbg!("{}", ctx.func.display(None));
+        debug!("{}", ctx.func.display(None));
         ctx.verify(runtime.func_env().flags()).unwrap();
     }
 }
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index 7606ebc0b4..1711a351ab 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -24,7 +24,6 @@
 #![cfg_attr(not(feature = "std"), no_std)]
 #![cfg_attr(not(feature = "std"), feature(alloc))]
 
-#[macro_use(dbg)]
 extern crate cranelift_codegen;
 #[macro_use]
 extern crate cranelift_entity;
@@ -36,6 +35,9 @@ extern crate failure;
 #[macro_use]
 extern crate failure_derive;
 
+#[macro_use]
+extern crate log;
+
 mod code_translator;
 mod environ;
 mod func_translator;

From 8bd1b877efabc6303006edf4b64736e0b25f59cb Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 3 Aug 2018 14:55:01 -0700
Subject: [PATCH 1988/3084] Elaborate on some comments.

---
 lib/wasm/src/environ/dummy.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 21ee1d08a3..5077612529 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -1,4 +1,5 @@
-//! "Dummy" environment for testing wasm translation.
+//! "Dummy" implementations of `ModuleEnvironment` and `FuncEnvironment` for testing
+//! wasm translation.
 
 use cranelift_codegen::cursor::FuncCursor;
 use cranelift_codegen::ir::immediates::Imm64;

From cc4bf1c7fbb18ccb8841b913361fb89d7180afc4 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 3 Aug 2018 15:03:05 -0700
Subject: [PATCH 1989/3084] Deny unstable_features in "std" builds.

---
 cranelift/src/clif-util.rs | 4 ++--
 lib/codegen/src/lib.rs     | 2 +-
 lib/entity/src/lib.rs      | 2 +-
 lib/faerie/src/lib.rs      | 4 ++--
 lib/filetests/src/lib.rs   | 4 ++--
 lib/frontend/src/lib.rs    | 2 +-
 lib/module/src/lib.rs      | 2 +-
 lib/native/src/lib.rs      | 4 ++--
 lib/reader/src/lib.rs      | 4 ++--
 lib/serde/src/clif-json.rs | 4 ++--
 lib/simplejit/src/lib.rs   | 4 ++--
 lib/umbrella/src/lib.rs    | 4 ++--
 lib/wasm/src/lib.rs        | 2 +-
 13 files changed, 21 insertions(+), 21 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 86e892420a..7db013a023 100644
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -1,5 +1,5 @@
-#![deny(trivial_numeric_casts)]
-#![warn(unused_import_braces, unstable_features, unused_extern_crates)]
+#![deny(trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![warn(unused_import_braces)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index 5ae6812a52..2d740668f6 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -2,7 +2,7 @@
 
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "std", warn(unstable_features))]
+#![cfg_attr(feature = "std", deny(unstable_features))]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature="cargo-clippy", allow(
 // This requires Rust 1.27 or later.
diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs
index cb1c248db1..dd666c6ac7 100644
--- a/lib/entity/src/lib.rs
+++ b/lib/entity/src/lib.rs
@@ -31,7 +31,7 @@
 
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "std", warn(unstable_features))]
+#![cfg_attr(feature = "std", deny(unstable_features))]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
 #![cfg_attr(
diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs
index 14397afbd0..078488a953 100644
--- a/lib/faerie/src/lib.rs
+++ b/lib/faerie/src/lib.rs
@@ -2,8 +2,8 @@
 //!
 //! Users of this module should not have to depend on faerie directly.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
-#![warn(unused_import_braces, unstable_features)]
+#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![warn(unused_import_braces)]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
 #![cfg_attr(
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index b052253e67..da665a6f2b 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -3,8 +3,8 @@
 //! This crate contains the main test driver as well as implementations of the
 //! available filetest commands.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
-#![warn(unused_import_braces, unstable_features)]
+#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![warn(unused_import_braces)]
 #![cfg_attr(feature = "cargo-clippy",
     allow(
           type_complexity,
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index be19ebc2be..7fca5ae122 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -129,7 +129,7 @@
 
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "std", warn(unstable_features))]
+#![cfg_attr(feature = "std", deny(unstable_features))]
 #![cfg_attr(feature = "cargo-clippy", allow(new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs
index a7676c4a8c..45e3bd683e 100644
--- a/lib/module/src/lib.rs
+++ b/lib/module/src/lib.rs
@@ -2,7 +2,7 @@
 
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "std", warn(unstable_features))]
+#![cfg_attr(feature = "std", deny(unstable_features))]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
 #![cfg_attr(
diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs
index 5439d743cc..3d4e4bac2c 100644
--- a/lib/native/src/lib.rs
+++ b/lib/native/src/lib.rs
@@ -1,8 +1,8 @@
 //! Performs autodetection of the host for the purposes of running
 //! Cranelift to generate code to run on the same machine.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
-#![warn(unused_import_braces, unstable_features)]
+#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![warn(unused_import_braces)]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
 #![cfg_attr(
diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs
index edac07622f..2873a644ac 100644
--- a/lib/reader/src/lib.rs
+++ b/lib/reader/src/lib.rs
@@ -3,8 +3,8 @@
 //! The `cranelift_reader` library supports reading .clif files. This functionality is needed for
 //! testing Cranelift, but is not essential for a JIT compiler.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
-#![warn(unused_import_braces, unstable_features)]
+#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![warn(unused_import_braces)]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
 #![cfg_attr(
diff --git a/lib/serde/src/clif-json.rs b/lib/serde/src/clif-json.rs
index 659bc818a8..9c3338e600 100644
--- a/lib/serde/src/clif-json.rs
+++ b/lib/serde/src/clif-json.rs
@@ -1,7 +1,7 @@
 //! Utility for `cranelift_serde`.
 
-#![deny(trivial_numeric_casts)]
-#![warn(unused_import_braces, unstable_features, unused_extern_crates)]
+#![deny(trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![warn(unused_import_braces)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs
index 01dd0938f0..99297f33b0 100644
--- a/lib/simplejit/src/lib.rs
+++ b/lib/simplejit/src/lib.rs
@@ -1,7 +1,7 @@
 //! Top-level lib.rs for `cranelift_simplejit`.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
-#![warn(unused_import_braces, unstable_features)]
+#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![warn(unused_import_braces)]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
 #![cfg_attr(
diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs
index ad241483e7..4faeabdd8a 100644
--- a/lib/umbrella/src/lib.rs
+++ b/lib/umbrella/src/lib.rs
@@ -1,7 +1,7 @@
 //! Cranelift umbrella crate, providing a convenient one-line dependency.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
-#![warn(unused_import_braces, unstable_features)]
+#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![warn(unused_import_braces)]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
 #![cfg_attr(
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index 1711a351ab..a559ae941d 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -11,7 +11,7 @@
 
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "std", warn(unstable_features))]
+#![cfg_attr(feature = "std", deny(unstable_features))]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
 #![cfg_attr(

From f34531ab365431111c1c502cbc1d077da81ebce2 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 3 Aug 2018 15:07:23 -0700
Subject: [PATCH 1990/3084] Synchronize cranelift-serde's lint settings with
 the other crates.

---
 lib/serde/src/clif-json.rs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/serde/src/clif-json.rs b/lib/serde/src/clif-json.rs
index 9c3338e600..5fc8a1d469 100644
--- a/lib/serde/src/clif-json.rs
+++ b/lib/serde/src/clif-json.rs
@@ -1,7 +1,9 @@
 //! Utility for `cranelift_serde`.
 
-#![deny(trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
 #![warn(unused_import_braces)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
+#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(

From d4a83576e4a187b9468ce72f10703ce640d00b23 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Sat, 11 Aug 2018 07:22:38 -0700
Subject: [PATCH 1991/3084] Add a few miscellaneous comments.

---
 lib/codegen/src/isa/x86/abi.rs    | 1 +
 lib/codegen/src/legalizer/call.rs | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs
index d0305d918b..21796e2a10 100644
--- a/lib/codegen/src/isa/x86/abi.rs
+++ b/lib/codegen/src/isa/x86/abi.rs
@@ -217,6 +217,7 @@ fn callee_saved_gprs(isa: &TargetIsa) -> &'static [RU] {
     }
 }
 
+/// Get the set of callee-saved registers that are used.
 fn callee_saved_gprs_used(isa: &TargetIsa, func: &ir::Function) -> RegisterSet {
     let mut all_callee_saved = RegisterSet::empty();
     for reg in callee_saved_gprs(isa) {
diff --git a/lib/codegen/src/legalizer/call.rs b/lib/codegen/src/legalizer/call.rs
index 365ba3988d..21753d19db 100644
--- a/lib/codegen/src/legalizer/call.rs
+++ b/lib/codegen/src/legalizer/call.rs
@@ -8,7 +8,8 @@ use flowgraph::ControlFlowGraph;
 use ir::{self, InstBuilder};
 use isa::TargetIsa;
 
-/// Expand a `call` instruction.
+/// Expand a `call` instruction. This lowers it to a `call_indirect`, which
+/// is only done if the ABI doesn't support direct calls.
 pub fn expand_call(
     inst: ir::Inst,
     func: &mut ir::Function,

From 3b56b2f4fb5e8f682e06d2108228daa650f3968f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 13 Aug 2018 12:52:09 -0700
Subject: [PATCH 1992/3084] Fix rustfmt errors.

---
 cranelift/src/clif-util.rs | 2 +-
 lib/filetests/src/lib.rs   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 7db013a023..890bf95be7 100644
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -12,9 +12,9 @@
 extern crate cfg_if;
 extern crate cranelift_codegen;
 extern crate cranelift_filetests;
-extern crate file_per_thread_logger;
 extern crate cranelift_reader;
 extern crate docopt;
+extern crate file_per_thread_logger;
 extern crate filecheck;
 #[macro_use]
 extern crate serde_derive;
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index da665a6f2b..90565b5cb4 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -19,8 +19,8 @@
 )]
 
 extern crate cranelift_codegen;
-extern crate file_per_thread_logger;
 extern crate cranelift_reader;
+extern crate file_per_thread_logger;
 extern crate filecheck;
 extern crate num_cpus;
 #[macro_use]

From 4769e674688662237249c5e32852c36f48bcff96 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 13 Aug 2018 12:52:30 -0700
Subject: [PATCH 1993/3084] Fix a few declarations for the `no_std` build.

---
 lib/codegen/src/lib.rs                 | 1 -
 lib/codegen/src/regalloc/coalescing.rs | 1 -
 2 files changed, 2 deletions(-)

diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index 2d740668f6..5ba44b8a81 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -49,7 +49,6 @@ extern crate failure_derive;
 #[cfg_attr(test, macro_use)]
 extern crate target_lexicon;
 
-#[cfg(feature = "std")]
 #[macro_use]
 extern crate log;
 
diff --git a/lib/codegen/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs
index 885e0b82fe..c7db253767 100644
--- a/lib/codegen/src/regalloc/coalescing.rs
+++ b/lib/codegen/src/regalloc/coalescing.rs
@@ -6,7 +6,6 @@
 //! parameter will belong to the same virtual register as the EBB parameter value itself.
 
 use cursor::{Cursor, EncCursor};
-#[cfg(feature = "std")]
 use dbg::DisplayList;
 use dominator_tree::{DominatorTree, DominatorTreePreorder};
 use flowgraph::{BasicBlock, ControlFlowGraph};

From df7fa1980795cb31c6d1f51205a6785d4df52000 Mon Sep 17 00:00:00 2001
From: Amanieu d'Antras 
Date: Mon, 13 Aug 2018 23:30:09 +0200
Subject: [PATCH 1994/3084] Fix README and license

---
 lib/bforest/Cargo.toml |   2 +-
 lib/bforest/LICENSE    | 219 +++++++++++++++++++++++++++++++++++++++++
 lib/bforest/README.md  |  10 ++
 3 files changed, 230 insertions(+), 1 deletion(-)
 create mode 100644 lib/bforest/LICENSE

diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index 4b1d1d076c..8598d7439b 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -3,7 +3,7 @@ authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
 version = "0.18.1"
 description = "A forest of B+-trees"
-license = "Apache-2.0"
+license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
 repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
diff --git a/lib/bforest/LICENSE b/lib/bforest/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/bforest/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
diff --git a/lib/bforest/README.md b/lib/bforest/README.md
index 718a4ed8fe..391d6287d2 100644
--- a/lib/bforest/README.md
+++ b/lib/bforest/README.md
@@ -1,2 +1,12 @@
 This crate contains array-based data structures used by the core Cranelift code
 generator which represent a set of small ordered sets or maps.
+
+**These are not general purpose data structures that are somehow magically faster that the
+standard library's `BTreeSet` and `BTreeMap` types.**
+
+The tradeoffs are different:
+
+- Keys and values are expected to be small and copyable. We optimize for 32-bit types.
+- A comparator object is used to compare keys, allowing smaller "context free" keys.
+- Empty trees have a very small 32-bit footprint.
+- All the trees in a forest can be cleared in constant time.

From 6f9982fdf5d4398176e2939efb2d821e630678d2 Mon Sep 17 00:00:00 2001
From: Caroline Cullen 
Date: Fri, 3 Aug 2018 13:07:12 -0700
Subject: [PATCH 1995/3084] Updating Sphinx link and install instructions

---
 cranelift/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index 9011d4b752..2ce83df9f6 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -126,7 +126,7 @@ Building the documentation
 --------------------------
 
 To build the Cranelift documentation, you need the [Sphinx documentation
-generator](https://www.sphinx-doc.org/):
+generator](http://www.sphinx-doc.org/) as well as Python 3::
 
     $ pip install sphinx sphinx-autobuild sphinx_rtd_theme
     $ cd cranelift/docs

From 3f582f7cbde7e0ba8d3929427fac524605852ea3 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Tue, 14 Aug 2018 03:31:39 +0200
Subject: [PATCH 1996/3084] Legalize br_icmp (#449)

* Legalize br_icmp
---
 .../filetests/isa/x86/legalize-br-icmp.clif   | 46 +++++++++++++++++++
 lib/codegen/meta-python/base/legalize.py      |  1 +
 lib/codegen/src/legalizer/mod.rs              | 34 ++++++++++++++
 3 files changed, 81 insertions(+)
 create mode 100644 cranelift/filetests/isa/x86/legalize-br-icmp.clif

diff --git a/cranelift/filetests/isa/x86/legalize-br-icmp.clif b/cranelift/filetests/isa/x86/legalize-br-icmp.clif
new file mode 100644
index 0000000000..e4694c36b1
--- /dev/null
+++ b/cranelift/filetests/isa/x86/legalize-br-icmp.clif
@@ -0,0 +1,46 @@
+test legalizer
+
+target x86_64
+
+function %br_icmp(i64) fast {
+ebb0(v0: i64):
+    v1 = iconst.i64 0
+    br_icmp eq v0, v1, ebb1
+    jump ebb1
+
+ebb1:
+    return
+}
+
+; sameln: function %br_icmp(i64 [%rdi]) fast {
+; nextln:                                 ebb0(v0: i64):
+; nextln: [RexOp1pu_id#b8]                    v1 = iconst.i64 0
+; nextln: [RexOp1icscc#8039]                  v2 = icmp eq v0, v1
+; nextln: [RexOp1t8jccb#75]                   brnz v2, ebb1
+; nextln: [RexOp1jmpb#eb]                     jump ebb1
+; nextln: 
+; nextln:                                 ebb1:
+; nextln: [Op1ret#c3]                         return
+; nextln: }
+
+
+function %br_icmp_ebb_args(i64) fast {
+ebb0(v0: i64):
+    v1 = iconst.i64 0
+    br_icmp eq v0, v1, ebb1(v0)
+    jump ebb1(v0)
+
+ebb1(v2: i64):
+    return
+}
+
+; sameln: function %br_icmp_ebb_args(i64 [%rdi]) fast {
+; nextln:                                 ebb0(v0: i64):
+; nextln: [RexOp1pu_id#b8]                    v1 = iconst.i64 0
+; nextln: [RexOp1icscc#8039]                  v3 = icmp eq v0, v1
+; nextln: [RexOp1t8jccb#75]                   brnz v3, ebb1(v0)
+; nextln: [RexOp1jmpb#eb]                     jump ebb1(v0)
+; nextln: 
+; nextln:                                 ebb1(v2: i64):
+; nextln: [Op1ret#c3]                         return
+; nextln: }
diff --git a/lib/codegen/meta-python/base/legalize.py b/lib/codegen/meta-python/base/legalize.py
index cf66074809..3427d8ef44 100644
--- a/lib/codegen/meta-python/base/legalize.py
+++ b/lib/codegen/meta-python/base/legalize.py
@@ -407,6 +407,7 @@ for ty,             minus_zero in [
                 a << bor(a1, a2)
             ))
 
+expand.custom_legalize(insts.br_icmp, 'expand_br_icmp')
 
 # Expansions using CPU flags.
 
diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs
index 7c24a86723..a7645fb643 100644
--- a/lib/codegen/src/legalizer/mod.rs
+++ b/lib/codegen/src/legalizer/mod.rs
@@ -244,6 +244,40 @@ fn expand_select(
     cfg.recompute_ebb(pos.func, old_ebb);
 }
 
+fn expand_br_icmp(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    cfg: &mut ControlFlowGraph,
+    _isa: &TargetIsa,
+) {
+    let (cond, a, b, destination, ebb_args) = match func.dfg[inst] {
+        ir::InstructionData::BranchIcmp {
+            cond,
+            destination,
+            ref args,
+            ..
+        } => (
+            cond,
+            args.get(0, &func.dfg.value_lists).unwrap(),
+            args.get(1, &func.dfg.value_lists).unwrap(),
+            destination,
+            args.as_slice(&func.dfg.value_lists)[2..].to_vec(),
+        ),
+        _ => panic!("Expected br_icmp {}", func.dfg.display_inst(inst, None)),
+    };
+
+    let old_ebb = func.layout.pp_ebb(inst);
+    func.dfg.clear_results(inst);
+
+    let icmp_res = func.dfg.replace(inst).icmp(cond, a, b);
+    let mut pos = FuncCursor::new(func).after_inst(inst);
+    pos.use_srcloc(inst);
+    pos.ins().brnz(icmp_res, destination, &ebb_args);
+
+    cfg.recompute_ebb(pos.func, destination);
+    cfg.recompute_ebb(pos.func, old_ebb);
+}
+
 /// Expand illegal `f32const` and `f64const` instructions.
 fn expand_fconst(
     inst: ir::Inst,

From dbc547091f93f84c2cbb49e4d6b756a493562900 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gr=C3=A9goire=20Geis?= 
Date: Tue, 14 Aug 2018 19:55:10 +0200
Subject: [PATCH 1997/3084] Verifier now accepts multiple errors (fixes #387).
 (#452)

* Verifier now accepts multiple errors (fixes #387).
---
 cranelift/src/wasm.rs                 |   6 +-
 lib/codegen/src/context.rs            |  30 +-
 lib/codegen/src/dominator_tree.rs     |   8 +-
 lib/codegen/src/print_errors.rs       |  61 +--
 lib/codegen/src/regalloc/context.rs   |  89 +++-
 lib/codegen/src/result.rs             |  12 +-
 lib/codegen/src/verifier/cssa.rs      |  31 +-
 lib/codegen/src/verifier/flags.rs     |  44 +-
 lib/codegen/src/verifier/liveness.rs  |  86 +++-
 lib/codegen/src/verifier/locations.rs |  90 ++--
 lib/codegen/src/verifier/mod.rs       | 636 +++++++++++++++++++-------
 lib/filetests/src/runone.rs           |   2 +-
 lib/filetests/src/test_verifier.rs    |  58 ++-
 lib/frontend/src/frontend.rs          |   6 +-
 lib/frontend/src/lib.rs               |   5 +-
 lib/frontend/src/ssa.rs               |  12 +-
 lib/wasm/tests/wasm_testsuite.rs      |   2 +-
 17 files changed, 845 insertions(+), 333 deletions(-)

diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index d448f75389..34fcb26ad0 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -148,9 +148,9 @@ fn handle_module(
 
         let func_index = num_func_imports + def_index.index();
         if flag_check_translation {
-            context
-                .verify(fisa)
-                .map_err(|err| pretty_verifier_error(&context.func, fisa.isa, None, &err))?;
+            if let Err(errors) = context.verify(fisa) {
+                return Err(pretty_verifier_error(&context.func, fisa.isa, None, errors));
+            }
         } else {
             let compiled_size = context
                 .compile(isa)
diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs
index afb2b3ee55..e5930cd575 100644
--- a/lib/codegen/src/context.rs
+++ b/lib/codegen/src/context.rs
@@ -30,7 +30,7 @@ use simple_gvn::do_simple_gvn;
 use std::vec::Vec;
 use timing;
 use unreachable_code::eliminate_unreachable_code;
-use verifier::{verify_context, verify_locations, VerifierResult};
+use verifier::{verify_context, verify_locations, VerifierErrors, VerifierResult};
 
 /// Persistent data structures and compilation pipeline.
 pub struct Context {
@@ -177,31 +177,43 @@ impl Context {
     ///
     /// Also check that the dominator tree and control flow graph are consistent with the function.
     pub fn verify<'a, FOI: Into>>(&self, fisa: FOI) -> VerifierResult<()> {
-        verify_context(&self.func, &self.cfg, &self.domtree, fisa)
+        let mut errors = VerifierErrors::default();
+        let _ = verify_context(&self.func, &self.cfg, &self.domtree, fisa, &mut errors);
+
+        if errors.is_empty() {
+            Ok(())
+        } else {
+            Err(errors)
+        }
     }
 
     /// Run the verifier only if the `enable_verifier` setting is true.
     pub fn verify_if<'a, FOI: Into>>(&self, fisa: FOI) -> CodegenResult<()> {
         let fisa = fisa.into();
         if fisa.flags.enable_verifier() {
-            self.verify(fisa).map_err(Into::into)
-        } else {
-            Ok(())
+            self.verify(fisa)?;
         }
+        Ok(())
     }
 
     /// Run the locations verifier on the function.
     pub fn verify_locations(&self, isa: &TargetIsa) -> VerifierResult<()> {
-        verify_locations(isa, &self.func, None)
+        let mut errors = VerifierErrors::default();
+        let _ = verify_locations(isa, &self.func, None, &mut errors);
+
+        if errors.is_empty() {
+            Ok(())
+        } else {
+            Err(errors)
+        }
     }
 
     /// Run the locations verifier only if the `enable_verifier` setting is true.
     pub fn verify_locations_if(&self, isa: &TargetIsa) -> CodegenResult<()> {
         if isa.flags().enable_verifier() {
-            self.verify_locations(isa).map_err(Into::into)
-        } else {
-            Ok(())
+            self.verify_locations(isa)?;
         }
+        Ok(())
     }
 
     /// Perform dead-code elimination on the function.
diff --git a/lib/codegen/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs
index bd75a16a9f..a246f671bc 100644
--- a/lib/codegen/src/dominator_tree.rs
+++ b/lib/codegen/src/dominator_tree.rs
@@ -673,7 +673,7 @@ mod test {
     use ir::types::*;
     use ir::{Function, InstBuilder, TrapCode};
     use settings;
-    use verifier::verify_context;
+    use verifier::{verify_context, VerifierErrors};
 
     #[test]
     fn empty() {
@@ -931,6 +931,10 @@ mod test {
         cfg.compute(cur.func);
 
         let flags = settings::Flags::new(settings::builder());
-        verify_context(cur.func, &cfg, &dt, &flags).unwrap();
+        let mut errors = VerifierErrors::default();
+
+        verify_context(cur.func, &cfg, &dt, &flags, &mut errors).unwrap();
+
+        assert!(errors.0.is_empty());
     }
 }
diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index f85f5ccfc4..03ba2457a1 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -9,7 +9,7 @@ use std::boxed::Box;
 use std::fmt;
 use std::fmt::Write;
 use std::string::{String, ToString};
-use verifier::VerifierError;
+use verifier::{VerifierError, VerifierErrors};
 use write::{decorate_function, FuncWriter, PlainWriter};
 
 /// Pretty-print a verifier error.
@@ -17,21 +17,26 @@ pub fn pretty_verifier_error<'a>(
     func: &ir::Function,
     isa: Option<&TargetIsa>,
     func_w: Option>,
-    err: &VerifierError,
+    errors: VerifierErrors,
 ) -> String {
+    let mut errors = errors.0;
     let mut w = String::new();
 
-    match err.location {
-        ir::entities::AnyEntity::Inst(_) => {}
-        _ => {
-            // Print the error, because the pretty_function_error below won't do it since it isn't
-            // tied to an instruction.
-            writeln!(w, "verifier error summary: {}\n", err.to_string()).unwrap();
+    // TODO: Use drain_filter here when it gets stabilized
+    let mut i = 0;
+
+    while i != errors.len() {
+        if let ir::entities::AnyEntity::Inst(_) = errors[i].location {
+            let err = errors.remove(i);
+
+            writeln!(w, "Miscellaneous error: {}\n", err).unwrap()
+        } else {
+            i += 1;
         }
     }
 
     decorate_function(
-        &mut PrettyVerifierError(func_w.unwrap_or(Box::new(PlainWriter)), err),
+        &mut PrettyVerifierError(func_w.unwrap_or(Box::new(PlainWriter)), &mut errors),
         &mut w,
         func,
         isa,
@@ -39,7 +44,7 @@ pub fn pretty_verifier_error<'a>(
     w
 }
 
-struct PrettyVerifierError<'a>(Box, &'a VerifierError);
+struct PrettyVerifierError<'a>(Box, &'a mut Vec);
 
 impl<'a> FuncWriter for PrettyVerifierError<'a> {
     fn write_instruction(
@@ -71,31 +76,35 @@ fn pretty_function_error(
     cur_inst: Inst,
     indent: usize,
     func_w: &mut FuncWriter,
-    err: &VerifierError,
+    errors: &mut Vec,
 ) -> fmt::Result {
-    match err.location {
-        ir::entities::AnyEntity::Inst(inst) if inst == cur_inst => {
-            func_w.write_instruction(w, func, isa, cur_inst, indent)?;
-            write!(w, "{1:0$}^", indent, "")?;
-            for _c in cur_inst.to_string().chars() {
-                write!(w, "~")?;
+    // TODO: Use drain_filter here when it gets stabilized
+    let mut i = 0;
+
+    while i != errors.len() {
+        match errors[i].location {
+            ir::entities::AnyEntity::Inst(inst) if inst == cur_inst => {
+                let err = errors.remove(i);
+
+                func_w.write_instruction(w, func, isa, cur_inst, indent)?;
+                write!(w, "{1:0$}^", indent, "")?;
+                for _c in cur_inst.to_string().chars() {
+                    write!(w, "~")?;
+                }
+                writeln!(w, " verifier {}\n", err.to_string())?;
             }
-            writeln!(w, " verifier {}\n", err.to_string())
+            ir::entities::AnyEntity::Inst(_) => i += 1,
+            _ => unreachable!(),
         }
-        _ => writeln!(
-            w,
-            "{1:0$}{2}",
-            indent,
-            "",
-            func.dfg.display_inst(cur_inst, isa)
-        ),
     }
+
+    Ok(())
 }
 
 /// Pretty-print a Cranelift error.
 pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CodegenError) -> String {
     if let CodegenError::Verifier(e) = err {
-        pretty_verifier_error(func, isa, None, &e)
+        pretty_verifier_error(func, isa, None, e)
     } else {
         err.to_string()
     }
diff --git a/lib/codegen/src/regalloc/context.rs b/lib/codegen/src/regalloc/context.rs
index 349fa212cb..08e658bd4c 100644
--- a/lib/codegen/src/regalloc/context.rs
+++ b/lib/codegen/src/regalloc/context.rs
@@ -18,7 +18,7 @@ use regalloc::virtregs::VirtRegs;
 use result::CodegenResult;
 use timing;
 use topo_order::TopoOrder;
-use verifier::{verify_context, verify_cssa, verify_liveness, verify_locations};
+use verifier::{verify_context, verify_cssa, verify_liveness, verify_locations, VerifierErrors};
 
 /// Persistent memory allocations for register allocation.
 pub struct Context {
@@ -76,6 +76,8 @@ impl Context {
         let _tt = timing::regalloc();
         debug_assert!(domtree.is_valid());
 
+        let mut errors = VerifierErrors::default();
+
         // `Liveness` and `Coloring` are self-clearing.
         self.virtregs.clear();
 
@@ -87,7 +89,11 @@ impl Context {
         self.liveness.compute(isa, func, cfg);
 
         if isa.flags().enable_verifier() {
-            verify_liveness(isa, func, cfg, &self.liveness)?;
+            let ok = verify_liveness(isa, func, cfg, &self.liveness, &mut errors).is_ok();
+
+            if !ok {
+                return Err(errors.into());
+            }
         }
 
         // Pass: Coalesce and create Conventional SSA form.
@@ -101,9 +107,20 @@ impl Context {
         );
 
         if isa.flags().enable_verifier() {
-            verify_context(func, cfg, domtree, isa)?;
-            verify_liveness(isa, func, cfg, &self.liveness)?;
-            verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?;
+            let ok = verify_context(func, cfg, domtree, isa, &mut errors).is_ok()
+                && verify_liveness(isa, func, cfg, &self.liveness, &mut errors).is_ok()
+                && verify_cssa(
+                    func,
+                    cfg,
+                    domtree,
+                    &self.liveness,
+                    &self.virtregs,
+                    &mut errors,
+                ).is_ok();
+
+            if !ok {
+                return Err(errors.into());
+            }
         }
 
         // Pass: Spilling.
@@ -118,9 +135,20 @@ impl Context {
         );
 
         if isa.flags().enable_verifier() {
-            verify_context(func, cfg, domtree, isa)?;
-            verify_liveness(isa, func, cfg, &self.liveness)?;
-            verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?;
+            let ok = verify_context(func, cfg, domtree, isa, &mut errors).is_ok()
+                && verify_liveness(isa, func, cfg, &self.liveness, &mut errors).is_ok()
+                && verify_cssa(
+                    func,
+                    cfg,
+                    domtree,
+                    &self.liveness,
+                    &self.virtregs,
+                    &mut errors,
+                ).is_ok();
+
+            if !ok {
+                return Err(errors.into());
+            }
         }
 
         // Pass: Reload.
@@ -134,9 +162,20 @@ impl Context {
         );
 
         if isa.flags().enable_verifier() {
-            verify_context(func, cfg, domtree, isa)?;
-            verify_liveness(isa, func, cfg, &self.liveness)?;
-            verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?;
+            let ok = verify_context(func, cfg, domtree, isa, &mut errors).is_ok()
+                && verify_liveness(isa, func, cfg, &self.liveness, &mut errors).is_ok()
+                && verify_cssa(
+                    func,
+                    cfg,
+                    domtree,
+                    &self.liveness,
+                    &self.virtregs,
+                    &mut errors,
+                ).is_ok();
+
+            if !ok {
+                return Err(errors.into());
+            }
         }
 
         // Pass: Coloring.
@@ -144,11 +183,29 @@ impl Context {
             .run(isa, func, domtree, &mut self.liveness, &mut self.tracker);
 
         if isa.flags().enable_verifier() {
-            verify_context(func, cfg, domtree, isa)?;
-            verify_liveness(isa, func, cfg, &self.liveness)?;
-            verify_locations(isa, func, Some(&self.liveness))?;
-            verify_cssa(func, cfg, domtree, &self.liveness, &self.virtregs)?;
+            let ok = verify_context(func, cfg, domtree, isa, &mut errors).is_ok()
+                && verify_liveness(isa, func, cfg, &self.liveness, &mut errors).is_ok()
+                && verify_locations(isa, func, Some(&self.liveness), &mut errors).is_ok()
+                && verify_cssa(
+                    func,
+                    cfg,
+                    domtree,
+                    &self.liveness,
+                    &self.virtregs,
+                    &mut errors,
+                ).is_ok();
+
+            if !ok {
+                return Err(errors.into());
+            }
+        }
+
+        // Even if we arrive here, (non-fatal) errors might have been reported, so we
+        // must make sure absolutely nothing is wrong
+        if errors.is_empty() {
+            Ok(())
+        } else {
+            Err(errors.into())
         }
-        Ok(())
     }
 }
diff --git a/lib/codegen/src/result.rs b/lib/codegen/src/result.rs
index 0956b4f4d2..dace896049 100644
--- a/lib/codegen/src/result.rs
+++ b/lib/codegen/src/result.rs
@@ -1,18 +1,18 @@
 //! Result and error types representing the outcome of compiling a function.
 
-use verifier::VerifierError;
+use verifier::VerifierErrors;
 
 /// A compilation error.
 ///
 /// When Cranelift fails to compile a function, it will return one of these error codes.
 #[derive(Fail, Debug, PartialEq, Eq)]
 pub enum CodegenError {
-    /// An IR verifier error.
+    /// A list of IR verifier errors.
     ///
     /// This always represents a bug, either in the code that generated IR for Cranelift, or a bug
     /// in Cranelift itself.
-    #[fail(display = "Verifier error: {}", _0)]
-    Verifier(#[cause] VerifierError),
+    #[fail(display = "Verifier errors:\n{}", _0)]
+    Verifier(#[cause] VerifierErrors),
 
     /// An implementation limit was exceeded.
     ///
@@ -34,8 +34,8 @@ pub enum CodegenError {
 /// A convenient alias for a `Result` that uses `CodegenError` as the error type.
 pub type CodegenResult = Result;
 
-impl From for CodegenError {
-    fn from(e: VerifierError) -> Self {
+impl From for CodegenError {
+    fn from(e: VerifierErrors) -> Self {
         CodegenError::Verifier(e)
     }
 }
diff --git a/lib/codegen/src/verifier/cssa.rs b/lib/codegen/src/verifier/cssa.rs
index c2d745c4fb..3162bfb5e9 100644
--- a/lib/codegen/src/verifier/cssa.rs
+++ b/lib/codegen/src/verifier/cssa.rs
@@ -7,7 +7,7 @@ use ir::{ExpandedProgramPoint, Function};
 use regalloc::liveness::Liveness;
 use regalloc::virtregs::VirtRegs;
 use timing;
-use verifier::VerifierResult;
+use verifier::{VerifierErrors, VerifierStepResult};
 
 /// Verify conventional SSA form for `func`.
 ///
@@ -29,7 +29,8 @@ pub fn verify_cssa(
     domtree: &DominatorTree,
     liveness: &Liveness,
     virtregs: &VirtRegs,
-) -> VerifierResult<()> {
+    errors: &mut VerifierErrors,
+) -> VerifierStepResult<()> {
     let _tt = timing::verify_cssa();
 
     let mut preorder = DominatorTreePreorder::new();
@@ -43,8 +44,8 @@ pub fn verify_cssa(
         liveness,
         preorder,
     };
-    verifier.check_virtregs()?;
-    verifier.check_cssa()?;
+    verifier.check_virtregs(errors)?;
+    verifier.check_cssa(errors)?;
     Ok(())
 }
 
@@ -58,19 +59,19 @@ struct CssaVerifier<'a> {
 }
 
 impl<'a> CssaVerifier<'a> {
-    fn check_virtregs(&self) -> VerifierResult<()> {
+    fn check_virtregs(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         for vreg in self.virtregs.all_virtregs() {
             let values = self.virtregs.values(vreg);
 
             for (idx, &val) in values.iter().enumerate() {
                 if !self.func.dfg.value_is_valid(val) {
-                    return err!(val, "Invalid value in {}", vreg);
+                    return fatal!(errors, val, "Invalid value in {}", vreg);
                 }
                 if !self.func.dfg.value_is_attached(val) {
-                    return err!(val, "Detached value in {}", vreg);
+                    return fatal!(errors, val, "Detached value in {}", vreg);
                 }
                 if self.liveness.get(val).is_none() {
-                    return err!(val, "Value in {} has no live range", vreg);
+                    return fatal!(errors, val, "Value in {} has no live range", vreg);
                 };
 
                 // Check topological ordering with the previous values in the virtual register.
@@ -81,7 +82,8 @@ impl<'a> CssaVerifier<'a> {
                     let prev_ebb = self.func.layout.pp_ebb(prev_def);
 
                     if prev_def == def {
-                        return err!(
+                        return fatal!(
+                            errors,
                             val,
                             "Values {} and {} in {} = {} defined at the same program point",
                             prev_val,
@@ -95,7 +97,8 @@ impl<'a> CssaVerifier<'a> {
                     if self.preorder.dominates(def_ebb, prev_ebb)
                         && self.domtree.dominates(def, prev_def, &self.func.layout)
                     {
-                        return err!(
+                        return fatal!(
+                            errors,
                             val,
                             "Value in {} = {} def dominates previous {}",
                             vreg,
@@ -117,7 +120,8 @@ impl<'a> CssaVerifier<'a> {
                     {
                         let ctx = self.liveness.context(&self.func.layout);
                         if self.liveness[prev_val].overlaps_def(def, def_ebb, ctx) {
-                            return err!(
+                            return fatal!(
+                                errors,
                                 val,
                                 "Value def in {} = {} interferes with {}",
                                 vreg,
@@ -135,7 +139,7 @@ impl<'a> CssaVerifier<'a> {
         Ok(())
     }
 
-    fn check_cssa(&self) -> VerifierResult<()> {
+    fn check_cssa(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         for ebb in self.func.layout.ebbs() {
             let ebb_params = self.func.dfg.ebb_params(ebb);
             for BasicBlock { inst: pred, .. } in self.cfg.pred_iter(ebb) {
@@ -149,7 +153,8 @@ impl<'a> CssaVerifier<'a> {
 
                 for (&ebb_param, &pred_arg) in ebb_params.iter().zip(pred_args) {
                     if !self.virtregs.same_class(ebb_param, pred_arg) {
-                        return err!(
+                        return fatal!(
+                            errors,
                             pred,
                             "{} and {} must be in the same virtual register",
                             ebb_param,
diff --git a/lib/codegen/src/verifier/flags.rs b/lib/codegen/src/verifier/flags.rs
index 3895cd63a0..0d16d17ef5 100644
--- a/lib/codegen/src/verifier/flags.rs
+++ b/lib/codegen/src/verifier/flags.rs
@@ -7,7 +7,7 @@ use ir::instructions::BranchInfo;
 use isa;
 use packed_option::PackedOption;
 use timing;
-use verifier::VerifierResult;
+use verifier::{VerifierErrors, VerifierStepResult};
 
 /// Verify that CPU flags are used correctly.
 ///
@@ -25,7 +25,8 @@ pub fn verify_flags(
     func: &ir::Function,
     cfg: &ControlFlowGraph,
     isa: Option<&isa::TargetIsa>,
-) -> VerifierResult<()> {
+    errors: &mut VerifierErrors,
+) -> VerifierStepResult<()> {
     let _tt = timing::verify_flags();
     let mut verifier = FlagsVerifier {
         func,
@@ -33,7 +34,7 @@ pub fn verify_flags(
         encinfo: isa.map(|isa| isa.encoding_info()),
         livein: EntityMap::new(),
     };
-    verifier.check()
+    verifier.check(errors)
 }
 
 struct FlagsVerifier<'a> {
@@ -46,7 +47,7 @@ struct FlagsVerifier<'a> {
 }
 
 impl<'a> FlagsVerifier<'a> {
-    fn check(&mut self) -> VerifierResult<()> {
+    fn check(&mut self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         // List of EBBs that need to be processed. EBBs may be re-added to this list when we detect
         // that one of their successor blocks needs a live-in flags value.
         let mut worklist = SparseSet::new();
@@ -55,7 +56,7 @@ impl<'a> FlagsVerifier<'a> {
         }
 
         while let Some(ebb) = worklist.pop() {
-            if let Some(value) = self.visit_ebb(ebb)? {
+            if let Some(value) = self.visit_ebb(ebb, errors)? {
                 // The EBB has live-in flags. Check if the value changed.
                 match self.livein[ebb].expand() {
                     // Revisit any predecessor blocks the first time we see a live-in for `ebb`.
@@ -66,7 +67,13 @@ impl<'a> FlagsVerifier<'a> {
                         }
                     }
                     Some(old) if old != value => {
-                        return err!(ebb, "conflicting live-in CPU flags: {} and {}", old, value);
+                        return fatal!(
+                            errors,
+                            ebb,
+                            "conflicting live-in CPU flags: {} and {}",
+                            old,
+                            value
+                        );
                     }
                     x => assert_eq!(x, Some(value)),
                 }
@@ -80,7 +87,11 @@ impl<'a> FlagsVerifier<'a> {
     }
 
     /// Check flags usage in `ebb` and return the live-in flags value, if any.
-    fn visit_ebb(&self, ebb: ir::Ebb) -> VerifierResult> {
+    fn visit_ebb(
+        &self,
+        ebb: ir::Ebb,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult> {
         // The single currently live flags value.
         let mut live_val = None;
 
@@ -93,7 +104,7 @@ impl<'a> FlagsVerifier<'a> {
                         // We've reached the def of `live_flags`, so it is no longer live above.
                         live_val = None;
                     } else if self.func.dfg.value_type(res).is_flags() {
-                        return err!(inst, "{} clobbers live CPU flags in {}", res, live);
+                        return fatal!(errors, inst, "{} clobbers live CPU flags in {}", res, live);
                     }
                 }
 
@@ -104,14 +115,14 @@ impl<'a> FlagsVerifier<'a> {
                     .and_then(|ei| ei.operand_constraints(self.func.encodings[inst]))
                     .map_or(false, |c| c.clobbers_flags) && live_val.is_some()
                 {
-                    return err!(inst, "encoding clobbers live CPU flags in {}", live);
+                    return fatal!(errors, inst, "encoding clobbers live CPU flags in {}", live);
                 }
             }
 
             // Now look for live ranges of CPU flags that end here.
             for &arg in self.func.dfg.inst_args(inst) {
                 if self.func.dfg.value_type(arg).is_flags() {
-                    merge(&mut live_val, arg, inst)?;
+                    merge(&mut live_val, arg, inst, errors)?;
                 }
             }
 
@@ -120,13 +131,13 @@ impl<'a> FlagsVerifier<'a> {
                 BranchInfo::NotABranch => {}
                 BranchInfo::SingleDest(dest, _) => {
                     if let Some(val) = self.livein[dest].expand() {
-                        merge(&mut live_val, val, inst)?;
+                        merge(&mut live_val, val, inst, errors)?;
                     }
                 }
                 BranchInfo::Table(jt) => {
                     for (_, dest) in self.func.jump_tables[jt].entries() {
                         if let Some(val) = self.livein[dest].expand() {
-                            merge(&mut live_val, val, inst)?;
+                            merge(&mut live_val, val, inst, errors)?;
                         }
                     }
                 }
@@ -139,10 +150,15 @@ impl<'a> FlagsVerifier<'a> {
 }
 
 // Merge live flags values, or return an error on conflicting values.
-fn merge(a: &mut Option, b: ir::Value, inst: ir::Inst) -> VerifierResult<()> {
+fn merge(
+    a: &mut Option,
+    b: ir::Value,
+    inst: ir::Inst,
+    errors: &mut VerifierErrors,
+) -> VerifierStepResult<()> {
     if let Some(va) = *a {
         if b != va {
-            return err!(inst, "conflicting live CPU flags: {} and {}", va, b);
+            return fatal!(errors, inst, "conflicting live CPU flags: {} and {}", va, b);
         }
     } else {
         *a = Some(b);
diff --git a/lib/codegen/src/verifier/liveness.rs b/lib/codegen/src/verifier/liveness.rs
index f884feea79..e5fae0585f 100644
--- a/lib/codegen/src/verifier/liveness.rs
+++ b/lib/codegen/src/verifier/liveness.rs
@@ -8,7 +8,7 @@ use regalloc::liveness::Liveness;
 use regalloc::liverange::LiveRange;
 use std::cmp::Ordering;
 use timing;
-use verifier::VerifierResult;
+use verifier::{VerifierErrors, VerifierStepResult};
 
 /// Verify liveness information for `func`.
 ///
@@ -27,7 +27,8 @@ pub fn verify_liveness(
     func: &Function,
     cfg: &ControlFlowGraph,
     liveness: &Liveness,
-) -> VerifierResult<()> {
+    errors: &mut VerifierErrors,
+) -> VerifierStepResult<()> {
     let _tt = timing::verify_liveness();
     let verifier = LivenessVerifier {
         isa,
@@ -35,8 +36,8 @@ pub fn verify_liveness(
         cfg,
         liveness,
     };
-    verifier.check_ebbs()?;
-    verifier.check_insts()?;
+    verifier.check_ebbs(errors)?;
+    verifier.check_insts(errors)?;
     Ok(())
 }
 
@@ -49,21 +50,21 @@ struct LivenessVerifier<'a> {
 
 impl<'a> LivenessVerifier<'a> {
     /// Check all EBB arguments.
-    fn check_ebbs(&self) -> VerifierResult<()> {
+    fn check_ebbs(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         for ebb in self.func.layout.ebbs() {
             for &val in self.func.dfg.ebb_params(ebb) {
                 let lr = match self.liveness.get(val) {
                     Some(lr) => lr,
-                    None => return err!(ebb, "EBB arg {} has no live range", val),
+                    None => return fatal!(errors, ebb, "EBB arg {} has no live range", val),
                 };
-                self.check_lr(ebb.into(), val, lr)?;
+                self.check_lr(ebb.into(), val, lr, errors)?;
             }
         }
         Ok(())
     }
 
     /// Check all instructions.
-    fn check_insts(&self) -> VerifierResult<()> {
+    fn check_insts(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         for ebb in self.func.layout.ebbs() {
             for inst in self.func.layout.ebb_insts(ebb) {
                 let encoding = self.func.encodings[inst];
@@ -72,14 +73,15 @@ impl<'a> LivenessVerifier<'a> {
                 for &val in self.func.dfg.inst_results(inst) {
                     let lr = match self.liveness.get(val) {
                         Some(lr) => lr,
-                        None => return err!(inst, "{} has no live range", val),
+                        None => return fatal!(errors, inst, "{} has no live range", val),
                     };
-                    self.check_lr(inst.into(), val, lr)?;
+                    self.check_lr(inst.into(), val, lr, errors)?;
 
                     if encoding.is_legal() {
                         // A legal instruction is not allowed to define ghost values.
                         if lr.affinity.is_unassigned() {
-                            return err!(
+                            return fatal!(
+                                errors,
                                 inst,
                                 "{} is a ghost value defined by a real [{}] instruction",
                                 val,
@@ -88,7 +90,8 @@ impl<'a> LivenessVerifier<'a> {
                         }
                     } else if !lr.affinity.is_unassigned() {
                         // A non-encoded instruction can only define ghost values.
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "{} is a real {} value defined by a ghost instruction",
                             val,
@@ -101,15 +104,16 @@ impl<'a> LivenessVerifier<'a> {
                 for &val in self.func.dfg.inst_args(inst) {
                     let lr = match self.liveness.get(val) {
                         Some(lr) => lr,
-                        None => return err!(inst, "{} has no live range", val),
+                        None => return fatal!(errors, inst, "{} has no live range", val),
                     };
                     if !self.live_at_use(lr, inst) {
-                        return err!(inst, "{} is not live at this use", val);
+                        return fatal!(errors, inst, "{} is not live at this use", val);
                     }
 
                     // A legal instruction is not allowed to depend on ghost values.
                     if encoding.is_legal() && lr.affinity.is_unassigned() {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "{} is a ghost value used by a real [{}] instruction",
                             val,
@@ -141,7 +145,13 @@ impl<'a> LivenessVerifier<'a> {
     }
 
     /// Check the integrity of the live range `lr`.
-    fn check_lr(&self, def: ProgramPoint, val: Value, lr: &LiveRange) -> VerifierResult<()> {
+    fn check_lr(
+        &self,
+        def: ProgramPoint,
+        val: Value,
+        lr: &LiveRange,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let l = &self.func.layout;
 
         let loc: AnyEntity = match def.into() {
@@ -149,11 +159,17 @@ impl<'a> LivenessVerifier<'a> {
             ExpandedProgramPoint::Inst(i) => i.into(),
         };
         if lr.def() != def {
-            return err!(loc, "Wrong live range def ({}) for {}", lr.def(), val);
+            return fatal!(
+                errors,
+                loc,
+                "Wrong live range def ({}) for {}",
+                lr.def(),
+                val
+            );
         }
         if lr.is_dead() {
             if !lr.is_local() {
-                return err!(loc, "Dead live range {} should be local", val);
+                return fatal!(errors, loc, "Dead live range {} should be local", val);
             } else {
                 return Ok(());
             }
@@ -164,11 +180,17 @@ impl<'a> LivenessVerifier<'a> {
         };
         match lr.def_local_end().into() {
             ExpandedProgramPoint::Ebb(e) => {
-                return err!(loc, "Def local range for {} can't end at {}", val, e)
+                return fatal!(
+                    errors,
+                    loc,
+                    "Def local range for {} can't end at {}",
+                    val,
+                    e
+                )
             }
             ExpandedProgramPoint::Inst(i) => {
                 if self.func.layout.inst_ebb(i) != Some(def_ebb) {
-                    return err!(loc, "Def local end for {} in wrong ebb", val);
+                    return fatal!(errors, loc, "Def local end for {} in wrong ebb", val);
                 }
             }
         }
@@ -176,12 +198,19 @@ impl<'a> LivenessVerifier<'a> {
         // Now check the live-in intervals against the CFG.
         for (mut ebb, end) in lr.liveins(self.liveness.context(l)) {
             if !l.is_ebb_inserted(ebb) {
-                return err!(loc, "{} livein at {} which is not in the layout", val, ebb);
+                return fatal!(
+                    errors,
+                    loc,
+                    "{} livein at {} which is not in the layout",
+                    val,
+                    ebb
+                );
             }
             let end_ebb = match l.inst_ebb(end) {
                 Some(e) => e,
                 None => {
-                    return err!(
+                    return fatal!(
+                        errors,
                         loc,
                         "{} livein for {} ends at {} which is not in the layout",
                         val,
@@ -196,7 +225,8 @@ impl<'a> LivenessVerifier<'a> {
                 // If `val` is live-in at `ebb`, it must be live at all the predecessors.
                 for BasicBlock { inst: pred, .. } in self.cfg.pred_iter(ebb) {
                     if !self.live_at_use(lr, pred) {
-                        return err!(
+                        return fatal!(
+                            errors,
                             pred,
                             "{} is live in to {} but not live at predecessor",
                             val,
@@ -210,7 +240,15 @@ impl<'a> LivenessVerifier<'a> {
                 }
                 ebb = match l.next_ebb(ebb) {
                     Some(e) => e,
-                    None => return err!(loc, "end of {} livein ({}) never reached", val, end_ebb),
+                    None => {
+                        return fatal!(
+                            errors,
+                            loc,
+                            "end of {} livein ({}) never reached",
+                            val,
+                            end_ebb
+                        )
+                    }
                 };
             }
         }
diff --git a/lib/codegen/src/verifier/locations.rs b/lib/codegen/src/verifier/locations.rs
index cc393bb46f..73997d0a92 100644
--- a/lib/codegen/src/verifier/locations.rs
+++ b/lib/codegen/src/verifier/locations.rs
@@ -5,7 +5,7 @@ use isa;
 use regalloc::liveness::Liveness;
 use regalloc::RegDiversions;
 use timing;
-use verifier::VerifierResult;
+use verifier::{VerifierErrors, VerifierStepResult};
 
 /// Verify value locations for `func`.
 ///
@@ -22,7 +22,8 @@ pub fn verify_locations(
     isa: &isa::TargetIsa,
     func: &ir::Function,
     liveness: Option<&Liveness>,
-) -> VerifierResult<()> {
+    errors: &mut VerifierErrors,
+) -> VerifierStepResult<()> {
     let _tt = timing::verify_locations();
     let verifier = LocationVerifier {
         isa,
@@ -31,7 +32,7 @@ pub fn verify_locations(
         encinfo: isa.encoding_info(),
         liveness,
     };
-    verifier.check_constraints()?;
+    verifier.check_constraints(errors)?;
     Ok(())
 }
 
@@ -45,7 +46,7 @@ struct LocationVerifier<'a> {
 
 impl<'a> LocationVerifier<'a> {
     /// Check that the assigned value locations match the operand constraints of their uses.
-    fn check_constraints(&self) -> VerifierResult<()> {
+    fn check_constraints(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         let dfg = &self.func.dfg;
         let mut divert = RegDiversions::new();
 
@@ -57,23 +58,23 @@ impl<'a> LocationVerifier<'a> {
                 let enc = self.func.encodings[inst];
 
                 if enc.is_legal() {
-                    self.check_enc_constraints(inst, enc, &divert)?
+                    self.check_enc_constraints(inst, enc, &divert, errors)?
                 } else {
-                    self.check_ghost_results(inst)?;
+                    self.check_ghost_results(inst, errors)?;
                 }
 
                 if let Some(sig) = dfg.call_signature(inst) {
-                    self.check_call_abi(inst, sig, &divert)?;
+                    self.check_call_abi(inst, sig, &divert, errors)?;
                 }
 
                 let opcode = dfg[inst].opcode();
                 if opcode.is_return() {
-                    self.check_return_abi(inst, &divert)?;
+                    self.check_return_abi(inst, &divert, errors)?;
                 } else if opcode.is_branch() && !divert.is_empty() {
-                    self.check_cfg_edges(inst, &divert)?;
+                    self.check_cfg_edges(inst, &divert, errors)?;
                 }
 
-                self.update_diversions(inst, &mut divert)?;
+                self.update_diversions(inst, &mut divert, errors)?;
             }
         }
 
@@ -86,7 +87,8 @@ impl<'a> LocationVerifier<'a> {
         inst: ir::Inst,
         enc: isa::Encoding,
         divert: &RegDiversions,
-    ) -> VerifierResult<()> {
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let constraints = self
             .encinfo
             .operand_constraints(enc)
@@ -97,7 +99,8 @@ impl<'a> LocationVerifier<'a> {
         }
 
         // TODO: We could give a better error message here.
-        err!(
+        fatal!(
+            errors,
             inst,
             "{} constraints not satisfied",
             self.encinfo.display(enc)
@@ -106,13 +109,18 @@ impl<'a> LocationVerifier<'a> {
 
     /// Check that the result values produced by a ghost instruction are not assigned a value
     /// location.
-    fn check_ghost_results(&self, inst: ir::Inst) -> VerifierResult<()> {
+    fn check_ghost_results(
+        &self,
+        inst: ir::Inst,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let results = self.func.dfg.inst_results(inst);
 
         for &res in results {
             let loc = self.func.locations[res];
             if loc.is_assigned() {
-                return err!(
+                return fatal!(
+                    errors,
                     inst,
                     "ghost result {} value must not have a location ({}).",
                     res,
@@ -130,7 +138,8 @@ impl<'a> LocationVerifier<'a> {
         inst: ir::Inst,
         sig: ir::SigRef,
         divert: &RegDiversions,
-    ) -> VerifierResult<()> {
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let sig = &self.func.dfg.signatures[sig];
         let varargs = self.func.dfg.inst_variable_args(inst);
         let results = self.func.dfg.inst_results(inst);
@@ -142,6 +151,7 @@ impl<'a> LocationVerifier<'a> {
                 abi,
                 divert.get(value, &self.func.locations),
                 ir::StackSlotKind::OutgoingArg,
+                errors,
             )?;
         }
 
@@ -152,6 +162,7 @@ impl<'a> LocationVerifier<'a> {
                 abi,
                 self.func.locations[value],
                 ir::StackSlotKind::OutgoingArg,
+                errors,
             )?;
         }
 
@@ -159,7 +170,12 @@ impl<'a> LocationVerifier<'a> {
     }
 
     /// Check the ABI argument locations for a return.
-    fn check_return_abi(&self, inst: ir::Inst, divert: &RegDiversions) -> VerifierResult<()> {
+    fn check_return_abi(
+        &self,
+        inst: ir::Inst,
+        divert: &RegDiversions,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let sig = &self.func.signature;
         let varargs = self.func.dfg.inst_variable_args(inst);
 
@@ -170,6 +186,7 @@ impl<'a> LocationVerifier<'a> {
                 abi,
                 divert.get(value, &self.func.locations),
                 ir::StackSlotKind::IncomingArg,
+                errors,
             )?;
         }
 
@@ -184,12 +201,14 @@ impl<'a> LocationVerifier<'a> {
         abi: &ir::AbiParam,
         loc: ir::ValueLoc,
         want_kind: ir::StackSlotKind,
-    ) -> VerifierResult<()> {
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         match abi.location {
             ir::ArgumentLoc::Unassigned => {}
             ir::ArgumentLoc::Reg(reg) => {
                 if loc != ir::ValueLoc::Reg(reg) {
-                    return err!(
+                    return fatal!(
+                        errors,
                         inst,
                         "ABI expects {} in {}, got {}",
                         value,
@@ -202,7 +221,8 @@ impl<'a> LocationVerifier<'a> {
                 if let ir::ValueLoc::Stack(ss) = loc {
                     let slot = &self.func.stack_slots[ss];
                     if slot.kind != want_kind {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "call argument {} should be in a {} slot, but {} is {}",
                             value,
@@ -212,7 +232,8 @@ impl<'a> LocationVerifier<'a> {
                         );
                     }
                     if slot.offset.unwrap() != offset {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "ABI expects {} at stack offset {}, but {} is at {}",
                             value,
@@ -222,7 +243,8 @@ impl<'a> LocationVerifier<'a> {
                         );
                     }
                 } else {
-                    return err!(
+                    return fatal!(
+                        errors,
                         inst,
                         "ABI expects {} at stack offset {}, got {}",
                         value,
@@ -237,7 +259,12 @@ impl<'a> LocationVerifier<'a> {
     }
 
     /// Update diversions to reflect the current instruction and check their consistency.
-    fn update_diversions(&self, inst: ir::Inst, divert: &mut RegDiversions) -> VerifierResult<()> {
+    fn update_diversions(
+        &self,
+        inst: ir::Inst,
+        divert: &mut RegDiversions,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let (arg, src) = match self.func.dfg[inst] {
             ir::InstructionData::RegMove { arg, src, .. }
             | ir::InstructionData::RegSpill { arg, src, .. } => (arg, ir::ValueLoc::Reg(src)),
@@ -247,14 +274,16 @@ impl<'a> LocationVerifier<'a> {
 
         if let Some(d) = divert.diversion(arg) {
             if d.to != src {
-                return err!(
+                return fatal!(
+                    errors,
                     inst,
                     "inconsistent with current diversion to {}",
                     d.to.display(&self.reginfo)
                 );
             }
         } else if self.func.locations[arg] != src {
-            return err!(
+            return fatal!(
+                errors,
                 inst,
                 "inconsistent with global location {}",
                 self.func.locations[arg].display(&self.reginfo)
@@ -268,7 +297,12 @@ impl<'a> LocationVerifier<'a> {
 
     /// We have active diversions before a branch. Make sure none of the diverted values are live
     /// on the outgoing CFG edges.
-    fn check_cfg_edges(&self, inst: ir::Inst, divert: &RegDiversions) -> VerifierResult<()> {
+    fn check_cfg_edges(
+        &self,
+        inst: ir::Inst,
+        divert: &RegDiversions,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         use ir::instructions::BranchInfo::*;
 
         // We can only check CFG edges if we have a liveness analysis.
@@ -287,7 +321,8 @@ impl<'a> LocationVerifier<'a> {
                 for d in divert.all() {
                     let lr = &liveness[d.value];
                     if lr.is_livein(ebb, liveness.context(&self.func.layout)) {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "{} is diverted to {} and live in to {}",
                             d.value,
@@ -302,7 +337,8 @@ impl<'a> LocationVerifier<'a> {
                     let lr = &liveness[d.value];
                     for (_, ebb) in self.func.jump_tables[jt].entries() {
                         if lr.is_livein(ebb, liveness.context(&self.func.layout)) {
-                            return err!(
+                            return fatal!(
+                                errors,
                                 inst,
                                 "{} is diverted to {} and live in to {}",
                                 d.value,
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 2888d5336d..66ad44fa11 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -82,23 +82,91 @@ pub use self::cssa::verify_cssa;
 pub use self::liveness::verify_liveness;
 pub use self::locations::verify_locations;
 
-// Create an `Err` variant of `VerifierResult` from a location and `format!` arguments.
-macro_rules! err {
-    ( $loc:expr, $msg:expr ) => {
-        Err(::verifier::VerifierError {
+/// Report an error.
+///
+/// The first argument must be a `&mut VerifierErrors` reference, and the following
+/// argument defines the location of the error and must implement `Into`.
+/// Finally, subsequent arguments will be formatted using `format!()` and set
+/// as the error message.
+macro_rules! report {
+    ( $errors: expr, $loc: expr, $msg: tt ) => {
+        $errors.0.push(::verifier::VerifierError {
             location: $loc.into(),
             message: String::from($msg),
         })
     };
 
-    ( $loc:expr, $fmt:expr, $( $arg:expr ),+ ) => {
-        Err(::verifier::VerifierError {
+    ( $errors: expr, $loc: expr, $fmt: tt, $( $arg: expr ),+ ) => {
+        $errors.0.push(::verifier::VerifierError {
             location: $loc.into(),
             message: format!( $fmt, $( $arg ),+ ),
         })
     };
 }
 
+/// Diagnose a fatal error, and return `Err`.
+macro_rules! fatal {
+    ( $( $arg: expr ),+ ) => ({
+        report!( $( $arg ),+ );
+        Err(())
+    });
+}
+
+/// Diagnose a non-fatal error, and return `Ok`.
+macro_rules! nonfatal {
+    ( $( $arg: expr ),+ ) => ({
+        report!( $( $arg ),+ );
+        Ok(())
+    });
+}
+
+/// Shorthand syntax for calling functions of the form
+/// `verify_foo(a, b, &mut VerifierErrors) -> VerifierStepResult`
+/// as if they had the form `verify_foo(a, b) -> VerifierResult`.
+///
+/// This syntax also ensures that no errors whatsoever were reported,
+/// even if they were not fatal.
+///
+/// # Example
+/// ```rust,ignore
+/// verify!(verify_context, func, cfg, domtree, fisa)
+///
+/// // ... is equivalent to...
+///
+/// let mut errors = VerifierErrors::new();
+/// let result = verify_context(func, cfg, domtree, fisa, &mut errors);
+///
+/// if errors.is_empty() {
+///     Ok(result.unwrap())
+/// } else {
+///     Err(errors)
+/// }
+/// ```
+#[macro_export]
+macro_rules! verify {
+    ( $verifier: expr; $fun: ident $(, $arg: expr )* ) => ({
+        let mut errors = $crate::verifier::VerifierErrors::default();
+        let result = $verifier.$fun( $( $arg, )* &mut errors);
+
+        if errors.is_empty() {
+            Ok(result.unwrap())
+        } else {
+            Err(errors)
+        }
+    });
+
+    ( $fun: path, $(, $arg: expr )* ) => ({
+        let mut errors = $crate::verifier::VerifierErrors::default();
+        let result = $fun( $( $arg, )* &mut errors);
+
+        if errors.is_empty() {
+            Ok(result.unwrap())
+        } else {
+            Err(errors)
+        }
+    });
+}
+
 mod cssa;
 mod flags;
 mod liveness;
@@ -109,7 +177,7 @@ mod locations;
 pub struct VerifierError {
     /// The entity causing the verifier error.
     pub location: AnyEntity,
-    /// Error message.
+    /// The error message.
     pub message: String,
 }
 
@@ -119,8 +187,71 @@ impl Display for VerifierError {
     }
 }
 
-/// Verifier result.
-pub type VerifierResult = Result;
+/// Result of a step in the verification process.
+///
+/// Functions that return `VerifierStepResult<()>` should also take a
+/// mutable reference to `VerifierErrors` as argument in order to report
+/// errors.
+///
+/// Here, `Ok` represents a step that **did not lead to a fatal error**,
+/// meaning that the verification process may continue. However, other (non-fatal)
+/// errors might have been reported through the previously mentioned `VerifierErrors`
+/// argument.
+pub type VerifierStepResult = Result;
+
+/// Result of a verification operation.
+///
+/// Unlike `VerifierStepResult<()>` which may be `Ok` while still having reported
+/// errors, this type always returns `Err` if an error (fatal or not) was reported.
+///
+/// Typically, this error will be constructed by using `verify!` on a function
+/// that returns `VerifierStepResult`.
+pub type VerifierResult = Result;
+
+/// List of verifier errors.
+#[derive(Fail, Debug, Default, PartialEq, Eq)]
+pub struct VerifierErrors(pub Vec);
+
+impl VerifierErrors {
+    /// Return a new `VerifierErrors` struct.
+    #[inline]
+    pub fn new() -> Self {
+        VerifierErrors(Vec::new())
+    }
+
+    /// Return whether no errors were reported.
+    #[inline]
+    pub fn is_empty(&self) -> bool {
+        self.0.is_empty()
+    }
+
+    /// Return whether one or more errors were reported.
+    #[inline]
+    pub fn has_error(&self) -> bool {
+        !self.0.is_empty()
+    }
+}
+
+impl From> for VerifierErrors {
+    fn from(v: Vec) -> Self {
+        VerifierErrors(v)
+    }
+}
+
+impl Into> for VerifierErrors {
+    fn into(self) -> Vec {
+        self.0
+    }
+}
+
+impl Display for VerifierErrors {
+    fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+        for err in &self.0 {
+            writeln!(f, "- {}", err)?;
+        }
+        Ok(())
+    }
+}
 
 /// Verify `func`.
 pub fn verify_function<'a, FOI: Into>>(
@@ -128,7 +259,7 @@ pub fn verify_function<'a, FOI: Into>>(
     fisa: FOI,
 ) -> VerifierResult<()> {
     let _tt = timing::verifier();
-    Verifier::new(func, fisa.into()).run()
+    verify!(Verifier::new(func, fisa.into()); run)
 }
 
 /// Verify `func` after checking the integrity of associated context data structures `cfg` and
@@ -138,16 +269,17 @@ pub fn verify_context<'a, FOI: Into>>(
     cfg: &ControlFlowGraph,
     domtree: &DominatorTree,
     fisa: FOI,
-) -> VerifierResult<()> {
+    errors: &mut VerifierErrors,
+) -> VerifierStepResult<()> {
     let _tt = timing::verifier();
     let verifier = Verifier::new(func, fisa.into());
     if cfg.is_valid() {
-        verifier.cfg_integrity(cfg)?;
+        verifier.cfg_integrity(cfg, errors)?;
     }
     if domtree.is_valid() {
-        verifier.domtree_integrity(domtree)?;
+        verifier.domtree_integrity(domtree, errors)?;
     }
-    verifier.run()
+    verifier.run(errors)
 }
 
 struct Verifier<'a> {
@@ -174,7 +306,7 @@ impl<'a> Verifier<'a> {
     // Check for:
     //  - cycles in the global value declarations.
     //  - use of 'vmctx' when no special parameter declares it.
-    fn verify_global_values(&self) -> VerifierResult<()> {
+    fn verify_global_values(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         let mut seen = SparseSet::new();
 
         for gv in self.func.global_values.keys() {
@@ -184,7 +316,7 @@ impl<'a> Verifier<'a> {
             let mut cur = gv;
             while let ir::GlobalValueData::Deref { base, .. } = self.func.global_values[cur] {
                 if seen.insert(base).is_some() {
-                    return err!(gv, "deref cycle: {}", DisplayList(seen.as_slice()));
+                    return fatal!(errors, gv, "deref cycle: {}", DisplayList(seen.as_slice()));
                 }
 
                 cur = base;
@@ -196,7 +328,7 @@ impl<'a> Verifier<'a> {
                     .special_param(ir::ArgumentPurpose::VMContext)
                     .is_none()
                 {
-                    return err!(cur, "undeclared vmctx reference {}", cur);
+                    return fatal!(errors, cur, "undeclared vmctx reference {}", cur);
                 }
             }
         }
@@ -204,26 +336,36 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
-    fn ebb_integrity(&self, ebb: Ebb, inst: Inst) -> VerifierResult<()> {
+    fn ebb_integrity(
+        &self,
+        ebb: Ebb,
+        inst: Inst,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let is_terminator = self.func.dfg[inst].opcode().is_terminator();
         let is_last_inst = self.func.layout.last_inst(ebb) == Some(inst);
 
         if is_terminator && !is_last_inst {
             // Terminating instructions only occur at the end of blocks.
-            return err!(
+            return fatal!(
+                errors,
                 inst,
                 "a terminator instruction was encountered before the end of {}",
                 ebb
             );
         }
         if is_last_inst && !is_terminator {
-            return err!(ebb, "block does not end in a terminator instruction");
+            return fatal!(
+                errors,
+                ebb,
+                "block does not end in a terminator instruction"
+            );
         }
 
         // Instructions belong to the correct ebb.
         let inst_ebb = self.func.layout.inst_ebb(inst);
         if inst_ebb != Some(ebb) {
-            return err!(inst, "should belong to {} not {:?}", ebb, inst_ebb);
+            return fatal!(errors, inst, "should belong to {} not {:?}", ebb, inst_ebb);
         }
 
         // Parameters belong to the correct ebb.
@@ -231,11 +373,11 @@ impl<'a> Verifier<'a> {
             match self.func.dfg.value_def(arg) {
                 ValueDef::Param(arg_ebb, _) => {
                     if ebb != arg_ebb {
-                        return err!(arg, "does not belong to {}", ebb);
+                        return fatal!(errors, arg, "does not belong to {}", ebb);
                     }
                 }
                 _ => {
-                    return err!(arg, "expected an argument, found a result");
+                    return fatal!(errors, arg, "expected an argument, found a result");
                 }
             }
         }
@@ -243,13 +385,21 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
-    fn instruction_integrity(&self, inst: Inst) -> VerifierResult<()> {
+    fn instruction_integrity(
+        &self,
+        inst: Inst,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let inst_data = &self.func.dfg[inst];
         let dfg = &self.func.dfg;
 
         // The instruction format matches the opcode
         if inst_data.opcode().format() != InstructionFormat::from(inst_data) {
-            return err!(inst, "instruction opcode doesn't match instruction format");
+            return fatal!(
+                errors,
+                inst,
+                "instruction opcode doesn't match instruction format"
+            );
         }
 
         let fixed_results = inst_data.opcode().constraints().fixed_results();
@@ -262,7 +412,8 @@ impl<'a> Verifier<'a> {
         // All result values for multi-valued instructions are created
         let got_results = dfg.inst_results(inst).len();
         if got_results != total_results {
-            return err!(
+            return fatal!(
+                errors,
                 inst,
                 "expected {} result values, found {}",
                 total_results,
@@ -270,29 +421,39 @@ impl<'a> Verifier<'a> {
             );
         }
 
-        self.verify_entity_references(inst)
+        self.verify_entity_references(inst, errors)
     }
 
-    fn verify_entity_references(&self, inst: Inst) -> VerifierResult<()> {
+    fn verify_entity_references(
+        &self,
+        inst: Inst,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         use ir::instructions::InstructionData::*;
 
         for &arg in self.func.dfg.inst_args(inst) {
-            self.verify_inst_arg(inst, arg)?;
+            self.verify_inst_arg(inst, arg, errors)?;
 
             // All used values must be attached to something.
             let original = self.func.dfg.resolve_aliases(arg);
             if !self.func.dfg.value_is_attached(original) {
-                return err!(inst, "argument {} -> {} is not attached", arg, original);
+                return fatal!(
+                    errors,
+                    inst,
+                    "argument {} -> {} is not attached",
+                    arg,
+                    original
+                );
             }
         }
 
         for &res in self.func.dfg.inst_results(inst) {
-            self.verify_inst_result(inst, res)?;
+            self.verify_inst_result(inst, res, errors)?;
         }
 
         match self.func.dfg[inst] {
             MultiAry { ref args, .. } => {
-                self.verify_value_list(inst, args)?;
+                self.verify_value_list(inst, args, errors)?;
             }
             Jump {
                 destination,
@@ -319,50 +480,50 @@ impl<'a> Verifier<'a> {
                 ref args,
                 ..
             } => {
-                self.verify_ebb(inst, destination)?;
-                self.verify_value_list(inst, args)?;
+                self.verify_ebb(inst, destination, errors)?;
+                self.verify_value_list(inst, args, errors)?;
             }
             BranchTable { table, .. } => {
-                self.verify_jump_table(inst, table)?;
+                self.verify_jump_table(inst, table, errors)?;
             }
             Call {
                 func_ref, ref args, ..
             } => {
-                self.verify_func_ref(inst, func_ref)?;
-                self.verify_value_list(inst, args)?;
+                self.verify_func_ref(inst, func_ref, errors)?;
+                self.verify_value_list(inst, args, errors)?;
             }
             CallIndirect {
                 sig_ref, ref args, ..
             } => {
-                self.verify_sig_ref(inst, sig_ref)?;
-                self.verify_value_list(inst, args)?;
+                self.verify_sig_ref(inst, sig_ref, errors)?;
+                self.verify_value_list(inst, args, errors)?;
             }
             FuncAddr { func_ref, .. } => {
-                self.verify_func_ref(inst, func_ref)?;
+                self.verify_func_ref(inst, func_ref, errors)?;
             }
             StackLoad { stack_slot, .. } | StackStore { stack_slot, .. } => {
-                self.verify_stack_slot(inst, stack_slot)?;
+                self.verify_stack_slot(inst, stack_slot, errors)?;
             }
             UnaryGlobalValue { global_value, .. } => {
-                self.verify_global_value(inst, global_value)?;
+                self.verify_global_value(inst, global_value, errors)?;
             }
             HeapAddr { heap, .. } => {
-                self.verify_heap(inst, heap)?;
+                self.verify_heap(inst, heap, errors)?;
             }
             TableAddr { table, .. } => {
-                self.verify_table(inst, table)?;
+                self.verify_table(inst, table, errors)?;
             }
             RegSpill { dst, .. } => {
-                self.verify_stack_slot(inst, dst)?;
+                self.verify_stack_slot(inst, dst, errors)?;
             }
             RegFill { src, .. } => {
-                self.verify_stack_slot(inst, src)?;
+                self.verify_stack_slot(inst, src, errors)?;
             }
             LoadComplex { ref args, .. } => {
-                self.verify_value_list(inst, args)?;
+                self.verify_value_list(inst, args, errors)?;
             }
             StoreComplex { ref args, .. } => {
-                self.verify_value_list(inst, args)?;
+                self.verify_value_list(inst, args, errors)?;
             }
 
             // Exhaustive list so we can't forget to add new formats
@@ -396,93 +557,148 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
-    fn verify_ebb(&self, inst: Inst, e: Ebb) -> VerifierResult<()> {
+    fn verify_ebb(
+        &self,
+        inst: Inst,
+        e: Ebb,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         if !self.func.dfg.ebb_is_valid(e) || !self.func.layout.is_ebb_inserted(e) {
-            return err!(inst, "invalid ebb reference {}", e);
+            return fatal!(errors, inst, "invalid ebb reference {}", e);
         }
         if let Some(entry_block) = self.func.layout.entry_block() {
             if e == entry_block {
-                return err!(inst, "invalid reference to entry ebb {}", e);
+                return fatal!(errors, inst, "invalid reference to entry ebb {}", e);
             }
         }
         Ok(())
     }
 
-    fn verify_sig_ref(&self, inst: Inst, s: SigRef) -> VerifierResult<()> {
+    fn verify_sig_ref(
+        &self,
+        inst: Inst,
+        s: SigRef,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         if !self.func.dfg.signatures.is_valid(s) {
-            err!(inst, "invalid signature reference {}", s)
+            fatal!(errors, inst, "invalid signature reference {}", s)
         } else {
             Ok(())
         }
     }
 
-    fn verify_func_ref(&self, inst: Inst, f: FuncRef) -> VerifierResult<()> {
+    fn verify_func_ref(
+        &self,
+        inst: Inst,
+        f: FuncRef,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         if !self.func.dfg.ext_funcs.is_valid(f) {
-            err!(inst, "invalid function reference {}", f)
+            fatal!(errors, inst, "invalid function reference {}", f)
         } else {
             Ok(())
         }
     }
 
-    fn verify_stack_slot(&self, inst: Inst, ss: StackSlot) -> VerifierResult<()> {
+    fn verify_stack_slot(
+        &self,
+        inst: Inst,
+        ss: StackSlot,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         if !self.func.stack_slots.is_valid(ss) {
-            err!(inst, "invalid stack slot {}", ss)
+            fatal!(errors, inst, "invalid stack slot {}", ss)
         } else {
             Ok(())
         }
     }
 
-    fn verify_global_value(&self, inst: Inst, gv: GlobalValue) -> VerifierResult<()> {
+    fn verify_global_value(
+        &self,
+        inst: Inst,
+        gv: GlobalValue,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         if !self.func.global_values.is_valid(gv) {
-            err!(inst, "invalid global value {}", gv)
+            fatal!(errors, inst, "invalid global value {}", gv)
         } else {
             Ok(())
         }
     }
 
-    fn verify_heap(&self, inst: Inst, heap: ir::Heap) -> VerifierResult<()> {
+    fn verify_heap(
+        &self,
+        inst: Inst,
+        heap: ir::Heap,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         if !self.func.heaps.is_valid(heap) {
-            err!(inst, "invalid heap {}", heap)
+            fatal!(errors, inst, "invalid heap {}", heap)
         } else {
             Ok(())
         }
     }
 
-    fn verify_table(&self, inst: Inst, table: ir::Table) -> VerifierResult<()> {
+    fn verify_table(
+        &self,
+        inst: Inst,
+        table: ir::Table,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         if !self.func.tables.is_valid(table) {
-            err!(inst, "invalid table {}", table)
+            fatal!(errors, inst, "invalid table {}", table)
         } else {
             Ok(())
         }
     }
 
-    fn verify_value_list(&self, inst: Inst, l: &ValueList) -> VerifierResult<()> {
+    fn verify_value_list(
+        &self,
+        inst: Inst,
+        l: &ValueList,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         if !l.is_valid(&self.func.dfg.value_lists) {
-            err!(inst, "invalid value list reference {:?}", l)
+            fatal!(errors, inst, "invalid value list reference {:?}", l)
         } else {
             Ok(())
         }
     }
 
-    fn verify_jump_table(&self, inst: Inst, j: JumpTable) -> VerifierResult<()> {
+    fn verify_jump_table(
+        &self,
+        inst: Inst,
+        j: JumpTable,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         if !self.func.jump_tables.is_valid(j) {
-            err!(inst, "invalid jump table reference {}", j)
+            fatal!(errors, inst, "invalid jump table reference {}", j)
         } else {
             Ok(())
         }
     }
 
-    fn verify_value(&self, loc_inst: Inst, v: Value) -> VerifierResult<()> {
+    fn verify_value(
+        &self,
+        loc_inst: Inst,
+        v: Value,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let dfg = &self.func.dfg;
         if !dfg.value_is_valid(v) {
-            err!(loc_inst, "invalid value reference {}", v)
+            fatal!(errors, loc_inst, "invalid value reference {}", v)
         } else {
             Ok(())
         }
     }
 
-    fn verify_inst_arg(&self, loc_inst: Inst, v: Value) -> VerifierResult<()> {
-        self.verify_value(loc_inst, v)?;
+    fn verify_inst_arg(
+        &self,
+        loc_inst: Inst,
+        v: Value,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
+        self.verify_value(loc_inst, v, errors)?;
 
         let dfg = &self.func.dfg;
         let loc_ebb = self.func.layout.pp_ebb(loc_inst);
@@ -493,7 +709,8 @@ impl<'a> Verifier<'a> {
             ValueDef::Result(def_inst, _) => {
                 // Value is defined by an instruction that exists.
                 if !dfg.inst_is_valid(def_inst) {
-                    return err!(
+                    return fatal!(
+                        errors,
                         loc_inst,
                         "{} is defined by invalid instruction {}",
                         v,
@@ -502,7 +719,8 @@ impl<'a> Verifier<'a> {
                 }
                 // Defining instruction is inserted in an EBB.
                 if self.func.layout.inst_ebb(def_inst) == None {
-                    return err!(
+                    return fatal!(
+                        errors,
                         loc_inst,
                         "{} is defined by {} which has no EBB",
                         v,
@@ -515,10 +733,16 @@ impl<'a> Verifier<'a> {
                         .expected_domtree
                         .dominates(def_inst, loc_inst, &self.func.layout)
                     {
-                        return err!(loc_inst, "uses value from non-dominating {}", def_inst);
+                        return fatal!(
+                            errors,
+                            loc_inst,
+                            "uses value from non-dominating {}",
+                            def_inst
+                        );
                     }
                     if def_inst == loc_inst {
-                        return err!(
+                        return fatal!(
+                            errors,
                             loc_inst,
                             "uses value from itself {},  {}",
                             def_inst,
@@ -530,11 +754,12 @@ impl<'a> Verifier<'a> {
             ValueDef::Param(ebb, _) => {
                 // Value is defined by an existing EBB.
                 if !dfg.ebb_is_valid(ebb) {
-                    return err!(loc_inst, "{} is defined by invalid EBB {}", v, ebb);
+                    return fatal!(errors, loc_inst, "{} is defined by invalid EBB {}", v, ebb);
                 }
                 // Defining EBB is inserted in the layout
                 if !self.func.layout.is_ebb_inserted(ebb) {
-                    return err!(
+                    return fatal!(
+                        errors,
                         loc_inst,
                         "{} is defined by {} which is not in the layout",
                         v,
@@ -547,20 +772,31 @@ impl<'a> Verifier<'a> {
                         .expected_domtree
                         .dominates(ebb, loc_inst, &self.func.layout)
                 {
-                    return err!(loc_inst, "uses value arg from non-dominating {}", ebb);
+                    return fatal!(
+                        errors,
+                        loc_inst,
+                        "uses value arg from non-dominating {}",
+                        ebb
+                    );
                 }
             }
         }
         Ok(())
     }
 
-    fn verify_inst_result(&self, loc_inst: Inst, v: Value) -> VerifierResult<()> {
-        self.verify_value(loc_inst, v)?;
+    fn verify_inst_result(
+        &self,
+        loc_inst: Inst,
+        v: Value,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
+        self.verify_value(loc_inst, v, errors)?;
 
         match self.func.dfg.value_def(v) {
             ValueDef::Result(def_inst, _) => {
                 if def_inst != loc_inst {
-                    err!(
+                    fatal!(
+                        errors,
                         loc_inst,
                         "instruction result {} is not defined by the instruction",
                         v
@@ -569,7 +805,8 @@ impl<'a> Verifier<'a> {
                     Ok(())
                 }
             }
-            ValueDef::Param(_, _) => err!(
+            ValueDef::Param(_, _) => fatal!(
+                errors,
                 loc_inst,
                 "instruction result {} is not defined by the instruction",
                 v
@@ -577,7 +814,11 @@ impl<'a> Verifier<'a> {
         }
     }
 
-    fn domtree_integrity(&self, domtree: &DominatorTree) -> VerifierResult<()> {
+    fn domtree_integrity(
+        &self,
+        domtree: &DominatorTree,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         // We consider two `DominatorTree`s to be equal if they return the same immediate
         // dominator for each EBB. Therefore the current domtree is valid if it matches the freshly
         // computed one.
@@ -585,7 +826,8 @@ impl<'a> Verifier<'a> {
             let expected = self.expected_domtree.idom(ebb);
             let got = domtree.idom(ebb);
             if got != expected {
-                return err!(
+                return fatal!(
+                    errors,
                     ebb,
                     "invalid domtree, expected idom({}) = {:?}, got {:?}",
                     ebb,
@@ -596,7 +838,8 @@ impl<'a> Verifier<'a> {
         }
         // We also verify if the postorder defined by `DominatorTree` is sane
         if domtree.cfg_postorder().len() != self.expected_domtree.cfg_postorder().len() {
-            return err!(
+            return fatal!(
+                errors,
                 AnyEntity::Function,
                 "incorrect number of Ebbs in postorder traversal"
             );
@@ -608,7 +851,8 @@ impl<'a> Verifier<'a> {
             .enumerate()
         {
             if test_ebb != true_ebb {
-                return err!(
+                return fatal!(
+                    errors,
                     test_ebb,
                     "invalid domtree, postorder ebb number {} should be {}, got {}",
                     index,
@@ -623,7 +867,8 @@ impl<'a> Verifier<'a> {
                 .expected_domtree
                 .rpo_cmp(prev_ebb, next_ebb, &self.func.layout) != Ordering::Greater
             {
-                return err!(
+                return fatal!(
+                    errors,
                     next_ebb,
                     "invalid domtree, rpo_cmp does not says {} is greater than {}",
                     prev_ebb,
@@ -634,13 +879,14 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
-    fn typecheck_entry_block_params(&self) -> VerifierResult<()> {
+    fn typecheck_entry_block_params(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         if let Some(ebb) = self.func.layout.entry_block() {
             let expected_types = &self.func.signature.params;
             let ebb_param_count = self.func.dfg.num_ebb_params(ebb);
 
             if ebb_param_count != expected_types.len() {
-                return err!(
+                return fatal!(
+                    errors,
                     ebb,
                     "entry block parameters ({}) must match function signature ({})",
                     ebb_param_count,
@@ -651,7 +897,8 @@ impl<'a> Verifier<'a> {
             for (i, &arg) in self.func.dfg.ebb_params(ebb).iter().enumerate() {
                 let arg_type = self.func.dfg.value_type(arg);
                 if arg_type != expected_types[i].value_type {
-                    return err!(
+                    return fatal!(
+                        errors,
                         ebb,
                         "entry block parameter {} expected to have type {}, got {}",
                         i,
@@ -664,7 +911,7 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
-    fn typecheck(&self, inst: Inst) -> VerifierResult<()> {
+    fn typecheck(&self, inst: Inst, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         let inst_data = &self.func.dfg[inst];
         let constraints = inst_data.opcode().constraints();
 
@@ -673,7 +920,12 @@ impl<'a> Verifier<'a> {
             let ctrl_type = self.func.dfg.ctrl_typevar(inst);
 
             if !value_typeset.contains(ctrl_type) {
-                return err!(inst, "has an invalid controlling type {}", ctrl_type);
+                return fatal!(
+                    errors,
+                    inst,
+                    "has an invalid controlling type {}",
+                    ctrl_type
+                );
             }
 
             ctrl_type
@@ -683,23 +935,29 @@ impl<'a> Verifier<'a> {
             types::VOID
         };
 
-        self.typecheck_results(inst, ctrl_type)?;
-        self.typecheck_fixed_args(inst, ctrl_type)?;
-        self.typecheck_variable_args(inst)?;
-        self.typecheck_return(inst)?;
-        self.typecheck_special(inst, ctrl_type)?;
+        self.typecheck_results(inst, ctrl_type, errors)?;
+        self.typecheck_fixed_args(inst, ctrl_type, errors)?;
+        self.typecheck_variable_args(inst, errors)?;
+        self.typecheck_return(inst, errors)?;
+        self.typecheck_special(inst, ctrl_type, errors)?;
 
         Ok(())
     }
 
-    fn typecheck_results(&self, inst: Inst, ctrl_type: Type) -> VerifierResult<()> {
+    fn typecheck_results(
+        &self,
+        inst: Inst,
+        ctrl_type: Type,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let mut i = 0;
         for &result in self.func.dfg.inst_results(inst) {
             let result_type = self.func.dfg.value_type(result);
             let expected_type = self.func.dfg.compute_result_type(inst, i, ctrl_type);
             if let Some(expected_type) = expected_type {
                 if result_type != expected_type {
-                    return err!(
+                    return fatal!(
+                        errors,
                         inst,
                         "expected result {} ({}) to have type {}, found {}",
                         i,
@@ -709,19 +967,24 @@ impl<'a> Verifier<'a> {
                     );
                 }
             } else {
-                return err!(inst, "has more result values than expected");
+                return fatal!(errors, inst, "has more result values than expected");
             }
             i += 1;
         }
 
         // There aren't any more result types left.
         if self.func.dfg.compute_result_type(inst, i, ctrl_type) != None {
-            return err!(inst, "has fewer result values than expected");
+            return fatal!(errors, inst, "has fewer result values than expected");
         }
         Ok(())
     }
 
-    fn typecheck_fixed_args(&self, inst: Inst, ctrl_type: Type) -> VerifierResult<()> {
+    fn typecheck_fixed_args(
+        &self,
+        inst: Inst,
+        ctrl_type: Type,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let constraints = self.func.dfg[inst].opcode().constraints();
 
         for (i, &arg) in self.func.dfg.inst_fixed_args(inst).iter().enumerate() {
@@ -729,7 +992,8 @@ impl<'a> Verifier<'a> {
             match constraints.value_argument_constraint(i, ctrl_type) {
                 ResolvedConstraint::Bound(expected_type) => {
                     if arg_type != expected_type {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "arg {} ({}) has type {}, expected {}",
                             i,
@@ -741,7 +1005,8 @@ impl<'a> Verifier<'a> {
                 }
                 ResolvedConstraint::Free(type_set) => {
                     if !type_set.contains(arg_type) {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "arg {} ({}) with type {} failed to satisfy type set {:?}",
                             i,
@@ -756,7 +1021,11 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
-    fn typecheck_variable_args(&self, inst: Inst) -> VerifierResult<()> {
+    fn typecheck_variable_args(
+        &self,
+        inst: Inst,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         match self.func.dfg.analyze_branch(inst) {
             BranchInfo::SingleDest(ebb, _) => {
                 let iter = self
@@ -765,13 +1034,14 @@ impl<'a> Verifier<'a> {
                     .ebb_params(ebb)
                     .iter()
                     .map(|&v| self.func.dfg.value_type(v));
-                self.typecheck_variable_args_iterator(inst, iter)?;
+                self.typecheck_variable_args_iterator(inst, iter, errors)?;
             }
             BranchInfo::Table(table) => {
                 for (_, ebb) in self.func.jump_tables[table].entries() {
                     let arg_count = self.func.dfg.num_ebb_params(ebb);
                     if arg_count != 0 {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "takes no arguments, but had target {} with {} arguments",
                             ebb,
@@ -790,16 +1060,16 @@ impl<'a> Verifier<'a> {
                     .params
                     .iter()
                     .map(|a| a.value_type);
-                self.typecheck_variable_args_iterator(inst, arg_types)?;
-                self.check_outgoing_args(inst, sig_ref)?;
+                self.typecheck_variable_args_iterator(inst, arg_types, errors)?;
+                self.check_outgoing_args(inst, sig_ref, errors)?;
             }
             CallInfo::Indirect(sig_ref, _) => {
                 let arg_types = self.func.dfg.signatures[sig_ref]
                     .params
                     .iter()
                     .map(|a| a.value_type);
-                self.typecheck_variable_args_iterator(inst, arg_types)?;
-                self.check_outgoing_args(inst, sig_ref)?;
+                self.typecheck_variable_args_iterator(inst, arg_types, errors)?;
+                self.check_outgoing_args(inst, sig_ref, errors)?;
             }
             CallInfo::NotACall => {}
         }
@@ -810,7 +1080,8 @@ impl<'a> Verifier<'a> {
         &self,
         inst: Inst,
         iter: I,
-    ) -> VerifierResult<()> {
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let variable_args = self.func.dfg.inst_variable_args(inst);
         let mut i = 0;
 
@@ -823,7 +1094,8 @@ impl<'a> Verifier<'a> {
             let arg = variable_args[i];
             let arg_type = self.func.dfg.value_type(arg);
             if expected_type != arg_type {
-                return err!(
+                return fatal!(
+                    errors,
                     inst,
                     "arg {} ({}) has type {}, expected {}",
                     i,
@@ -835,7 +1107,8 @@ impl<'a> Verifier<'a> {
             i += 1;
         }
         if i != variable_args.len() {
-            return err!(
+            return fatal!(
+                errors,
                 inst,
                 "mismatched argument count for `{}`: got {}, expected {}",
                 self.func.dfg.display_inst(inst, None),
@@ -850,7 +1123,12 @@ impl<'a> Verifier<'a> {
     ///
     /// When a signature has been legalized, all values passed as outgoing arguments on the stack
     /// must be assigned to a matching `OutgoingArg` stack slot.
-    fn check_outgoing_args(&self, inst: Inst, sig_ref: SigRef) -> VerifierResult<()> {
+    fn check_outgoing_args(
+        &self,
+        inst: Inst,
+        sig_ref: SigRef,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let sig = &self.func.dfg.signatures[sig_ref];
 
         // Before legalization, there's nothing to check.
@@ -867,10 +1145,11 @@ impl<'a> Verifier<'a> {
                 let arg_loc = self.func.locations[arg];
                 if let ValueLoc::Stack(ss) = arg_loc {
                     // Argument value is assigned to a stack slot as expected.
-                    self.verify_stack_slot(inst, ss)?;
+                    self.verify_stack_slot(inst, ss, errors)?;
                     let slot = &self.func.stack_slots[ss];
                     if slot.kind != StackSlotKind::OutgoingArg {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "Outgoing stack argument {} in wrong stack slot: {} = {}",
                             arg,
@@ -879,7 +1158,8 @@ impl<'a> Verifier<'a> {
                         );
                     }
                     if slot.offset != Some(offset) {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "Outgoing stack argument {} should have offset {}: {} = {}",
                             arg,
@@ -889,7 +1169,8 @@ impl<'a> Verifier<'a> {
                         );
                     }
                     if slot.size != abi.value_type.bytes() {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "Outgoing stack argument {} wrong size for {}: {} = {}",
                             arg,
@@ -900,7 +1181,8 @@ impl<'a> Verifier<'a> {
                     }
                 } else {
                     let reginfo = self.isa.map(|i| i.register_info());
-                    return err!(
+                    return fatal!(
+                        errors,
                         inst,
                         "Outgoing stack argument {} in wrong location: {}",
                         arg,
@@ -912,17 +1194,22 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
-    fn typecheck_return(&self, inst: Inst) -> VerifierResult<()> {
+    fn typecheck_return(&self, inst: Inst, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         if self.func.dfg[inst].opcode().is_return() {
             let args = self.func.dfg.inst_variable_args(inst);
             let expected_types = &self.func.signature.returns;
             if args.len() != expected_types.len() {
-                return err!(inst, "arguments of return must match function signature");
+                return fatal!(
+                    errors,
+                    inst,
+                    "arguments of return must match function signature"
+                );
             }
             for (i, (&arg, &expected_type)) in args.iter().zip(expected_types).enumerate() {
                 let arg_type = self.func.dfg.value_type(arg);
                 if arg_type != expected_type.value_type {
-                    return err!(
+                    return fatal!(
+                        errors,
                         inst,
                         "arg {} ({}) has type {}, must match function signature of {}",
                         i,
@@ -938,13 +1225,19 @@ impl<'a> Verifier<'a> {
 
     // Check special-purpose type constraints that can't be expressed in the normal opcode
     // constraints.
-    fn typecheck_special(&self, inst: Inst, ctrl_type: Type) -> VerifierResult<()> {
+    fn typecheck_special(
+        &self,
+        inst: Inst,
+        ctrl_type: Type,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         if let ir::InstructionData::Unary { opcode, arg } = self.func.dfg[inst] {
             let arg_type = self.func.dfg.value_type(arg);
             match opcode {
                 Opcode::Bextend | Opcode::Uextend | Opcode::Sextend | Opcode::Fpromote => {
                     if arg_type.lane_count() != ctrl_type.lane_count() {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "input {} and output {} must have same number of lanes",
                             arg_type,
@@ -952,7 +1245,8 @@ impl<'a> Verifier<'a> {
                         );
                     }
                     if arg_type.lane_bits() >= ctrl_type.lane_bits() {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "input {} must be smaller than output {}",
                             arg_type,
@@ -962,7 +1256,8 @@ impl<'a> Verifier<'a> {
                 }
                 Opcode::Breduce | Opcode::Ireduce | Opcode::Fdemote => {
                     if arg_type.lane_count() != ctrl_type.lane_count() {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "input {} and output {} must have same number of lanes",
                             arg_type,
@@ -970,7 +1265,8 @@ impl<'a> Verifier<'a> {
                         );
                     }
                     if arg_type.lane_bits() <= ctrl_type.lane_bits() {
-                        return err!(
+                        return fatal!(
+                            errors,
                             inst,
                             "input {} must be larger than output {}",
                             arg_type,
@@ -984,7 +1280,11 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
-    fn cfg_integrity(&self, cfg: &ControlFlowGraph) -> VerifierResult<()> {
+    fn cfg_integrity(
+        &self,
+        cfg: &ControlFlowGraph,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
         let mut expected_succs = BTreeSet::::new();
         let mut got_succs = BTreeSet::::new();
         let mut expected_preds = BTreeSet::::new();
@@ -996,7 +1296,8 @@ impl<'a> Verifier<'a> {
 
             let missing_succs: Vec = expected_succs.difference(&got_succs).cloned().collect();
             if !missing_succs.is_empty() {
-                return err!(
+                return fatal!(
+                    errors,
                     ebb,
                     "cfg lacked the following successor(s) {:?}",
                     missing_succs
@@ -1005,7 +1306,12 @@ impl<'a> Verifier<'a> {
 
             let excess_succs: Vec = got_succs.difference(&expected_succs).cloned().collect();
             if !excess_succs.is_empty() {
-                return err!(ebb, "cfg had unexpected successor(s) {:?}", excess_succs);
+                return fatal!(
+                    errors,
+                    ebb,
+                    "cfg had unexpected successor(s) {:?}",
+                    excess_succs
+                );
             }
 
             expected_preds.extend(
@@ -1017,7 +1323,8 @@ impl<'a> Verifier<'a> {
 
             let missing_preds: Vec = expected_preds.difference(&got_preds).cloned().collect();
             if !missing_preds.is_empty() {
-                return err!(
+                return fatal!(
+                    errors,
                     ebb,
                     "cfg lacked the following predecessor(s) {:?}",
                     missing_preds
@@ -1026,7 +1333,12 @@ impl<'a> Verifier<'a> {
 
             let excess_preds: Vec = got_preds.difference(&expected_preds).cloned().collect();
             if !excess_preds.is_empty() {
-                return err!(ebb, "cfg had unexpected predecessor(s) {:?}", excess_preds);
+                return fatal!(
+                    errors,
+                    ebb,
+                    "cfg had unexpected predecessor(s) {:?}",
+                    excess_preds
+                );
             }
 
             expected_succs.clear();
@@ -1039,7 +1351,7 @@ impl<'a> Verifier<'a> {
 
     /// If the verifier has been set up with an ISA, make sure that the recorded encoding for the
     /// instruction (if any) matches how the ISA would encode it.
-    fn verify_encoding(&self, inst: Inst) -> VerifierResult<()> {
+    fn verify_encoding(&self, inst: Inst, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         // When the encodings table is empty, we don't require any instructions to be encoded.
         //
         // Once some instructions are encoded, we require all side-effecting instructions to have a
@@ -1063,7 +1375,8 @@ impl<'a> Verifier<'a> {
                 ).peekable();
 
             if encodings.peek().is_none() {
-                return err!(
+                return fatal!(
+                    errors,
                     inst,
                     "Instruction failed to re-encode {}",
                     isa.encoding_info().display(encoding)
@@ -1090,7 +1403,8 @@ impl<'a> Verifier<'a> {
                         .unwrap();
                 }
 
-                return err!(
+                return fatal!(
+                    errors,
                     inst,
                     "encoding {} should be {}{}",
                     isa.encoding_info().display(encoding),
@@ -1132,14 +1446,15 @@ impl<'a> Verifier<'a> {
             // Provide the ISA default encoding as a hint.
             match self.func.encode(inst, isa) {
                 Ok(enc) => {
-                    return err!(
+                    return fatal!(
+                        errors,
                         inst,
                         "{} must have an encoding (e.g., {})",
                         text,
                         isa.encoding_info().display(enc)
                     )
                 }
-                Err(_) => return err!(inst, "{} must have an encoding", text),
+                Err(_) => return fatal!(errors, inst, "{} must have an encoding", text),
             }
         }
 
@@ -1148,35 +1463,39 @@ impl<'a> Verifier<'a> {
 
     /// Verify the `return_at_end` property which requires that there are no internal return
     /// instructions.
-    fn verify_return_at_end(&self) -> VerifierResult<()> {
+    fn verify_return_at_end(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         for ebb in self.func.layout.ebbs() {
             let inst = self.func.layout.last_inst(ebb).unwrap();
             if self.func.dfg[inst].opcode().is_return() && Some(ebb) != self.func.layout.last_ebb()
             {
-                return err!(inst, "Internal return not allowed with return_at_end=1");
+                return fatal!(
+                    errors,
+                    inst,
+                    "Internal return not allowed with return_at_end=1"
+                );
             }
         }
 
         Ok(())
     }
 
-    pub fn run(&self) -> VerifierResult<()> {
-        self.verify_global_values()?;
-        self.typecheck_entry_block_params()?;
+    pub fn run(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
+        self.verify_global_values(errors)?;
+        self.typecheck_entry_block_params(errors)?;
         for ebb in self.func.layout.ebbs() {
             for inst in self.func.layout.ebb_insts(ebb) {
-                self.ebb_integrity(ebb, inst)?;
-                self.instruction_integrity(inst)?;
-                self.typecheck(inst)?;
-                self.verify_encoding(inst)?;
+                self.ebb_integrity(ebb, inst, errors)?;
+                self.instruction_integrity(inst, errors)?;
+                self.typecheck(inst, errors)?;
+                self.verify_encoding(inst, errors)?;
             }
         }
 
         if self.flags.return_at_end() {
-            self.verify_return_at_end()?;
+            self.verify_return_at_end(errors)?;
         }
 
-        verify_flags(self.func, &self.expected_cfg, self.isa)?;
+        verify_flags(self.func, &self.expected_cfg, self.isa, errors)?;
 
         Ok(())
     }
@@ -1184,7 +1503,7 @@ impl<'a> Verifier<'a> {
 
 #[cfg(test)]
 mod tests {
-    use super::{Verifier, VerifierError};
+    use super::{Verifier, VerifierError, VerifierErrors};
     use entity::EntityList;
     use ir::instructions::{InstructionData, Opcode};
     use ir::Function;
@@ -1192,9 +1511,9 @@ mod tests {
 
     macro_rules! assert_err_with_msg {
         ($e:expr, $msg:expr) => {
-            match $e {
-                Ok(_) => panic!("Expected an error"),
-                Err(VerifierError { message, .. }) => {
+            match $e.0.get(0) {
+                None => panic!("Expected an error"),
+                Some(&VerifierError { ref message, .. }) => {
                     if !message.contains($msg) {
                         #[cfg(feature = "std")]
                         panic!(format!(
@@ -1214,7 +1533,10 @@ mod tests {
         let func = Function::new();
         let flags = &settings::Flags::new(settings::builder());
         let verifier = Verifier::new(&func, flags.into());
-        assert_eq!(verifier.run(), Ok(()));
+        let mut errors = VerifierErrors::default();
+
+        assert_eq!(verifier.run(&mut errors), Ok(()));
+        assert!(errors.0.is_empty());
     }
 
     #[test]
@@ -1237,6 +1559,10 @@ mod tests {
         );
         let flags = &settings::Flags::new(settings::builder());
         let verifier = Verifier::new(&func, flags.into());
-        assert_err_with_msg!(verifier.run(), "instruction format");
+        let mut errors = VerifierErrors::default();
+
+        let _ = verifier.run(&mut errors);
+
+        assert_err_with_msg!(errors, "instruction format");
     }
 }
diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs
index b98f5f8cfb..b365b4a96b 100644
--- a/lib/filetests/src/runone.rs
+++ b/lib/filetests/src/runone.rs
@@ -130,7 +130,7 @@ fn run_one_test<'a>(
     // Should we run the verifier before this test?
     if !context.verified && test.needs_verifier() {
         verify_function(&func, context.flags_or_isa())
-            .map_err(|e| pretty_verifier_error(&func, isa, None, &e))?;
+            .map_err(|errors| pretty_verifier_error(&func, isa, None, errors))?;
         context.verified = true;
     }
 
diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs
index 422deaf25b..9c356d6733 100644
--- a/lib/filetests/src/test_verifier.rs
+++ b/lib/filetests/src/test_verifier.rs
@@ -14,6 +14,7 @@ use cranelift_codegen::verify_function;
 use cranelift_reader::TestCommand;
 use match_directive::match_directive;
 use std::borrow::{Borrow, Cow};
+use std::fmt::Write;
 use subtest::{Context, SubTest, SubtestResult};
 
 struct TestVerifier;
@@ -41,37 +42,48 @@ impl SubTest for TestVerifier {
         let func = func.borrow();
 
         // Scan source annotations for "error:" directives.
-        let mut expected = None;
+        let mut expected = Vec::new();
+
         for comment in &context.details.comments {
             if let Some(tail) = match_directive(comment.text, "error:") {
-                // Currently, the verifier can only report one problem at a time.
-                // Reject more than one `error:` directives.
-                if expected.is_some() {
-                    return Err("cannot handle multiple error: directives".to_string());
-                }
-                expected = Some((comment.entity, tail));
+                expected.push((comment.entity, tail));
             }
         }
 
         match verify_function(func, context.flags_or_isa()) {
-            Ok(_) => match expected {
-                None => Ok(()),
-                Some((_, msg)) => Err(format!("passed, expected error: {}", msg)),
-            },
-            Err(got) => match expected {
-                None => Err(format!("verifier pass, got {}", got)),
-                Some((want_loc, want_msg)) if got.message.contains(want_msg) => {
-                    if want_loc == got.location {
-                        Ok(())
-                    } else {
-                        Err(format!(
-                            "correct error reported on {}, but wanted {}",
-                            got.location, want_loc
-                        ))
+            Ok(()) if expected.len() == 0 => Ok(()),
+            Ok(()) => Err(format!("passed, but expected errors: {:?}", expected)),
+
+            Err(ref errors) if expected.len() == 0 => {
+                Err(format!("expected no error, but got:\n{}", errors))
+            }
+
+            Err(errors) => {
+                let mut errors = errors.0;
+                let mut msg = String::new();
+
+                // for each expected error, find a suitable match
+                for expect in expected {
+                    let pos = errors
+                        .iter()
+                        .position(|err| err.message.contains(expect.1) && err.location == expect.0);
+
+                    match pos {
+                        None => {
+                            write!(msg, "expected error {}", expect.0).unwrap();
+                        }
+                        Some(pos) => {
+                            errors.swap_remove(pos);
+                        }
                     }
                 }
-                Some(_) => Err(format!("mismatching error: {}", got)),
-            },
+
+                if msg.len() == 0 {
+                    Ok(())
+                } else {
+                    Err(msg)
+                }
+            }
         }
     }
 }
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 16934282a8..5888828ba8 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -707,11 +707,9 @@ mod tests {
         }
 
         let flags = settings::Flags::new(settings::builder());
-        let res = verify_function(&func, &flags);
         // println!("{}", func.display(None));
-        match res {
-            Ok(_) => {}
-            Err(err) => panic!("{}{}", func.display(None), err),
+        if let Err(errors) = verify_function(&func, &flags) {
+            panic!("{}\n{}", func.display(None), errors)
         }
     }
 
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index 7fca5ae122..6830d23857 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -120,9 +120,8 @@
 //!     let flags = settings::Flags::new(settings::builder());
 //!     let res = verify_function(&func, &flags);
 //!     println!("{}", func.display(None));
-//!     match res {
-//!         Ok(_) => {}
-//!         Err(err) => panic!("{}", err),
+//!     if let Err(errors) = res {
+//!         panic!("{}", errors);
 //!     }
 //! }
 //! ```
diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs
index f5a3c4bd8a..5e6c4fbf9b 100644
--- a/lib/frontend/src/ssa.rs
+++ b/lib/frontend/src/ssa.rs
@@ -1049,9 +1049,9 @@ mod tests {
         let flags = settings::Flags::new(settings::builder());
         match verify_function(&func, &flags) {
             Ok(()) => {}
-            Err(_err) => {
+            Err(_errors) => {
                 #[cfg(feature = "std")]
-                panic!(_err.message);
+                panic!(_errors);
                 #[cfg(not(feature = "std"))]
                 panic!("function failed to verify");
             }
@@ -1228,9 +1228,9 @@ mod tests {
         let flags = settings::Flags::new(settings::builder());
         match verify_function(&func, &flags) {
             Ok(()) => {}
-            Err(_err) => {
+            Err(_errors) => {
                 #[cfg(feature = "std")]
-                panic!(_err.message);
+                panic!(_errors);
                 #[cfg(not(feature = "std"))]
                 panic!("function failed to verify");
             }
@@ -1279,9 +1279,9 @@ mod tests {
         let flags = settings::Flags::new(settings::builder());
         match verify_function(&func, &flags) {
             Ok(()) => {}
-            Err(_err) => {
+            Err(_errors) => {
                 #[cfg(feature = "std")]
-                panic!(_err.message);
+                panic!(_errors);
                 #[cfg(not(feature = "std"))]
                 panic!("function failed to verify");
             }
diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs
index 166b3af076..06d7da4e7b 100644
--- a/lib/wasm/tests/wasm_testsuite.rs
+++ b/lib/wasm/tests/wasm_testsuite.rs
@@ -77,7 +77,7 @@ fn handle_module(path: &Path, flags: &Flags) {
     translate_module(&data, &mut dummy_environ).unwrap();
     for func in dummy_environ.info.function_bodies.values() {
         verifier::verify_function(func, flags)
-            .map_err(|err| panic!(pretty_verifier_error(func, None, None, &err)))
+            .map_err(|errors| panic!(pretty_verifier_error(func, None, None, errors)))
             .unwrap();
     }
 }

From 3f0103f936a2d349b8f5f62df2a310e7079f2476 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Tue, 14 Aug 2018 17:10:49 +0200
Subject: [PATCH 1998/3084] Prevent finalize being called more than once per
 func/data (fixes #407)

---
 cranelift/tests/moduletests.rs | 52 ++++++++++++++++++++++++++++++++++
 lib/module/src/module.rs       | 10 +++++++
 2 files changed, 62 insertions(+)

diff --git a/cranelift/tests/moduletests.rs b/cranelift/tests/moduletests.rs
index 89a04e2cd5..1f4c9d204a 100644
--- a/cranelift/tests/moduletests.rs
+++ b/cranelift/tests/moduletests.rs
@@ -1,9 +1,14 @@
 extern crate cranelift_codegen;
+extern crate cranelift_entity;
+extern crate cranelift_frontend;
 extern crate cranelift_module;
 extern crate cranelift_simplejit;
 
 use cranelift_codegen::ir::*;
 use cranelift_codegen::settings::*;
+use cranelift_codegen::Context;
+use cranelift_entity::EntityRef;
+use cranelift_frontend::*;
 use cranelift_module::*;
 use cranelift_simplejit::*;
 
@@ -25,3 +30,50 @@ fn error_on_incompatible_sig_in_declare_function() {
         .err()
         .unwrap(); // Make sure this is an error
 }
+
+fn define_simple_function(module: &mut Module) -> FuncId {
+    let sig = Signature {
+        params: vec![],
+        returns: vec![],
+        call_conv: CallConv::SystemV,
+        argument_bytes: None,
+    };
+
+    let func_id = module
+        .declare_function("abc", Linkage::Local, &sig)
+        .unwrap();
+
+    let mut ctx = Context::new();
+    ctx.func = Function::with_name_signature(ExternalName::user(0, func_id.index() as u32), sig);
+    let mut func_ctx = FunctionBuilderContext::new();
+    {
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
+        let ebb = bcx.create_ebb();
+        bcx.switch_to_block(ebb);
+        bcx.ins().return_(&[]);
+    }
+
+    module.define_function(func_id, &mut ctx).unwrap();
+
+    func_id
+}
+
+#[test]
+#[should_panic(expected = "function can't be finalized twice")]
+fn panic_on_double_finalize() {
+    let mut module: Module = Module::new(SimpleJITBuilder::new());
+
+    let func_id = define_simple_function(&mut module);
+    module.finalize_function(func_id);
+    module.finalize_function(func_id);
+}
+
+#[test]
+#[should_panic(expected = "Result::unwrap()` on an `Err` value: DuplicateDefinition(\"abc\")")]
+fn panic_on_define_after_finalize() {
+    let mut module: Module = Module::new(SimpleJITBuilder::new());
+
+    let func_id = define_simple_function(&mut module);
+    module.finalize_function(func_id);
+    define_simple_function(&mut module);
+}
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 133ad5af9f..223f748107 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -562,6 +562,10 @@ where
 
     /// Perform all outstanding relocations on the given function. This requires all `Local`
     /// and `Export` entities referenced to be defined.
+    ///
+    /// # Panics
+    ///
+    /// When the function has already been finalized this panics
     pub fn finalize_function(&mut self, func: FuncId) -> B::FinalizedFunction {
         let output = {
             let info = &self.contents.functions[func];
@@ -569,6 +573,7 @@ where
                 info.decl.linkage.is_definable(),
                 "imported function cannot be finalized"
             );
+            assert!(!info.finalized, "function can't be finalized twice");
             self.backend.finalize_function(
                 info.compiled
                     .as_ref()
@@ -584,6 +589,10 @@ where
 
     /// Perform all outstanding relocations on the given data object. This requires all
     /// `Local` and `Export` entities referenced to be defined.
+    ///
+    /// # Panics
+    ///
+    /// When the data object has already been finalized this panics
     pub fn finalize_data(&mut self, data: DataId) -> B::FinalizedData {
         let output = {
             let info = &self.contents.data_objects[data];
@@ -591,6 +600,7 @@ where
                 info.decl.linkage.is_definable(),
                 "imported data cannot be finalized"
             );
+            assert!(!info.finalized, "data object can't be finalized twice");
             self.backend.finalize_data(
                 info.compiled
                     .as_ref()

From 6cf7a975a11b04ddfb0ad98137576acdeb107a97 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 13 Aug 2018 16:12:03 -0700
Subject: [PATCH 1999/3084] Factor out a repeated string, and reduce the length
 of a long line.

---
 cranelift/test-all.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index 4941e04610..3547f16000 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -41,14 +41,15 @@ fi
 
 # Check if any Python files have changed since we last checked them.
 tsfile="$topdir/target/meta-checked"
+meta_python="$topdir/lib/codegen/meta-python"
 if [ -f "$tsfile" ]; then
-    needcheck=$(find "$topdir/lib/codegen/meta-python" -name '*.py' -newer "$tsfile")
+    needcheck=$(find "$meta_python" -name '*.py' -newer "$tsfile")
 else
     needcheck=yes
 fi
 if [ -n "$needcheck" ]; then
     banner "Checking python source files"
-    "$topdir/lib/codegen/meta-python/check.sh"
+    "$meta_python/check.sh"
     touch "$tsfile" || echo no target directory
 fi
 

From 932b4ef9f37bb31dc7617104a5e358da9e42210c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 14 Aug 2018 12:48:47 -0700
Subject: [PATCH 2000/3084] Fix a few declarations for the `no_std` build.

---
 lib/codegen/src/print_errors.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 03ba2457a1..3d012ad8fd 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -9,6 +9,7 @@ use std::boxed::Box;
 use std::fmt;
 use std::fmt::Write;
 use std::string::{String, ToString};
+use std::vec::Vec;
 use verifier::{VerifierError, VerifierErrors};
 use write::{decorate_function, FuncWriter, PlainWriter};
 

From 6a07c72867948bf301574471311ff0c767a9d17e Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 14 Aug 2018 12:55:34 -0700
Subject: [PATCH 2001/3084] Bump version to 0.19.0

---
 cranelift/Cargo.toml        | 26 +++++++++++++-------------
 cranelift/publish-all.sh    |  4 ++--
 lib/bforest/Cargo.toml      |  4 ++--
 lib/codegen/Cargo.toml      |  8 ++++----
 lib/codegen/meta/Cargo.toml |  2 +-
 lib/entity/Cargo.toml       |  2 +-
 lib/faerie/Cargo.toml       |  6 +++---
 lib/filetests/Cargo.toml    |  6 +++---
 lib/frontend/Cargo.toml     |  4 ++--
 lib/module/Cargo.toml       |  6 +++---
 lib/native/Cargo.toml       |  4 ++--
 lib/reader/Cargo.toml       |  4 ++--
 lib/serde/Cargo.toml        |  6 +++---
 lib/simplejit/Cargo.toml    |  8 ++++----
 lib/umbrella/Cargo.toml     |  6 +++---
 lib/wasm/Cargo.toml         |  8 ++++----
 16 files changed, 52 insertions(+), 52 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index b3c7672c0c..f27fe9c2eb 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.18.1"
+version = "0.19.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,18 +14,18 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.18.1" }
-cranelift-entity = { path = "lib/entity", version = "0.18.1" }
-cranelift-reader = { path = "lib/reader", version = "0.18.1" }
-cranelift-frontend = { path = "lib/frontend", version = "0.18.1" }
-cranelift-serde = { path = "lib/serde", version = "0.18.1", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.18.1", optional = true }
-cranelift-native = { path = "lib/native", version = "0.18.1" }
-cranelift-filetests = { path = "lib/filetests", version = "0.18.1" }
-cranelift-module = { path = "lib/module", version = "0.18.1" }
-cranelift-faerie = { path = "lib/faerie", version = "0.18.1" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.18.1" }
-cranelift = { path = "lib/umbrella", version = "0.18.1" }
+cranelift-codegen = { path = "lib/codegen", version = "0.19.0" }
+cranelift-entity = { path = "lib/entity", version = "0.19.0" }
+cranelift-reader = { path = "lib/reader", version = "0.19.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.19.0" }
+cranelift-serde = { path = "lib/serde", version = "0.19.0", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.19.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.19.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.19.0" }
+cranelift-module = { path = "lib/module", version = "0.19.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.19.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.19.0" }
+cranelift = { path = "lib/umbrella", version = "0.19.0" }
 filecheck = "0.3.0"
 docopt = "1"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 089e7f27cb..7e0b6bb078 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,13 +9,13 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.18.1"
+version="0.19.0"
 
 # Update all of the Cargo.toml files.
 #
 # The main Cargo.toml in the top-level directory is the cranelift-tools crate which we don't publish.
 echo "Updating crate versions to $version"
-for crate in . lib/*; do
+for crate in . lib/* lib/codegen/meta; do
     # Update the version number of this crate to $version.
     sed -i.bk -e "s/^version = .*/version = \"$version\"/" \
         "$crate/Cargo.toml"
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index 8598d7439b..ab244abc80 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.18.1"
+version = "0.19.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 keywords = ["btree", "forest", "set", "map"]
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.18.1", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.19.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 4d8f48f012..59f9cf1737 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.18.1"
+version = "0.19.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,8 +11,8 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.18.1", default-features = false }
-cranelift-bforest = { path = "../bforest", version = "0.18.1", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.19.0", default-features = false }
+cranelift-bforest = { path = "../bforest", version = "0.19.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -24,7 +24,7 @@ log = { version = "0.4.3", default-features = false, features = ["release_max_le
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.18.1" }
+cranelift-codegen-meta = { path = "meta", version = "0.19.0" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 1b25d932ec..8f9837c2d4 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.18.1"
+version = "0.19.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index 48e3a9cf57..e96837669b 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.18.1"
+version = "0.19.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index bb8ffb4c29..3df9d2d84b 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.18.1"
+version = "0.19.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.1" }
-cranelift-module = { path = "../module", version = "0.18.1" }
+cranelift-codegen = { path = "../codegen", version = "0.19.0" }
+cranelift-module = { path = "../module", version = "0.19.0" }
 faerie = "0.4.4"
 goblin = "0.0.17"
 failure = "0.1.1"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 487675971f..667ed21418 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.18.1"
+version = "0.19.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,8 +9,8 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.1" }
-cranelift-reader = { path = "../reader", version = "0.18.1" }
+cranelift-codegen = { path = "../codegen", version = "0.19.0" }
+cranelift-reader = { path = "../reader", version = "0.19.0" }
 file-per-thread-logger = "0.1.1"
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 4ef7c4382b..d231b017f8 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.18.1"
+version = "0.19.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index a137bcebd0..4d57a6b2af 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.18.1"
+version = "0.19.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.18.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.19.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 log = { version = "0.4.3", default-features = false, features = ["release_max_level_warn"] }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 112899533b..f05126a27b 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.18.1"
+version = "0.19.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -8,7 +8,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index f1e148708a..70edf52930 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.18.1"
+version = "0.19.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.1" }
+cranelift-codegen = { path = "../codegen", version = "0.19.0" }
 target-lexicon = "0.0.3"
 
 [badges]
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index 27196ae151..ecbdd781e8 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.18.1"
+version = "0.19.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -17,8 +17,8 @@ docopt = "1"
 serde = "1.0.8"
 serde_derive = "1.0.8"
 serde_json = "1.0"
-cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
-cranelift-reader = { path = "../reader", version = "0.18.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
+cranelift-reader = { path = "../reader", version = "0.19.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 7007b3bbd7..fc459471f8 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.18.1"
+version = "0.19.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,9 +9,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
-cranelift-module = { path = "../module", version = "0.18.1", default-features = false }
-cranelift-native = { path = "../native", version = "0.18.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.19.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.19.0", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index c67c40b594..1f2d8bb605 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.18.1"
+version = "0.19.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,8 +10,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.18.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.19.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 4e40b5022d..369220c4a7 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.18.1"
+version = "0.19.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.2", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.18.1", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.18.1", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.18.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.19.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.19.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 73511435d018a40e464b55e000a77297018b5dcc Mon Sep 17 00:00:00 2001
From: Denis Merigoux 
Date: Thu, 2 Aug 2018 10:51:20 +0200
Subject: [PATCH 2002/3084] Better explanation for how to use FunctionBuilder
 to deal with variable translation

---
 lib/frontend/src/lib.rs | 35 +++++++++++++++++++++++++++++++++--
 1 file changed, 33 insertions(+), 2 deletions(-)

diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index 6830d23857..ed0b21a30d 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -1,12 +1,43 @@
 //! Cranelift IR builder library.
 //!
 //! Provides a straightforward way to create a Cranelift IR function and fill it with instructions
-//! translated from another language. Contains an SSA construction module that lets you translate
-//! your non-SSA variables into SSA Cranelift IR values via `use_var` and `def_var` calls.
+//! corresponding to your source program written in another language.
 //!
 //! To get started, create an [`FunctionBuilderContext`](struct.FunctionBuilderContext.html) and
 //! pass it as an argument to a [`FunctionBuilder`](struct.FunctionBuilder.html).
 //!
+//! # Source Language Variables and Cranelift IR values
+//!
+//! The most interesting feature of this API is that it provides a single way to deal with all your
+//! variable problems. Indeed, the [`FunctionBuilder`](struct.FunctionBuilder.html) struct has a
+//! type parameter `Variable` that should be instantiated with the type of your Source Language
+//! Variables (SLV). Then, through calling the functions
+//! [`declare_var`](struct.FunctionBuilder.html#method.declare_var),
+//! [`def_var`](struct.FunctionBuilder.html#method.def_var) and
+//! [`use_var`](struct.FunctionBuilder.html#method.use_var), the
+//! [`FunctionBuilder`](struct.FunctionBuilder.html) will create for you all the Cranelift IR
+//! values corresponding to your SLVs.
+//!
+//! If a SLV is immutable (defined only once), then it will get mapped to one and only Cranelift IR
+//! value. If a SLV is mutable ([`def_var`](struct.FunctionBuilder.html#method.def_var) multiple
+//! times), then internally a [`SSA`](https://en.wikipedia.org/wiki/Static_single_assignment_form)
+//! construction algorithm will automatically create multiple Cranelift IR values that will be
+//! returned to you by [`use_var`](struct.FunctionBuilder.html#method.use_var) depending on where
+//! you want to use your SLV.
+//!
+//! The morality is that you should use these three functions to handle all your SLVs, even those
+//! that are not present in the source code but artefacts of the translation. For instance, if your
+//! source language is expression-based, then you will need to introduce artifical SLVs to store
+//! intermediate results of the computation of your expressions. Hence The `Variable` type that you
+//! would pass to [`FunctionBuilder`](struct.FunctionBuilder.html) could look like this
+//!
+//! ```
+//! enum Variable {
+//!     OriginalSourceVariable(String),
+//!     IntermediateExpressionVariable(u32)
+//! }
+//! ```
+//!
 //! # Example
 //!
 //! Here is a pseudo-program we want to transform into Cranelift IR:

From b7d2df9307bb404b5548e8d72ae549a0e867ba10 Mon Sep 17 00:00:00 2001
From: Denis Merigoux 
Date: Mon, 6 Aug 2018 11:06:42 +0200
Subject: [PATCH 2003/3084] Rewrote doc with @sunfishcode's comments in mind

---
 lib/frontend/src/lib.rs | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)

diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index ed0b21a30d..969ed227c4 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -6,35 +6,38 @@
 //! To get started, create an [`FunctionBuilderContext`](struct.FunctionBuilderContext.html) and
 //! pass it as an argument to a [`FunctionBuilder`](struct.FunctionBuilder.html).
 //!
-//! # Source Language Variables and Cranelift IR values
+//! # Mutable variables and Cranelift IR values
 //!
 //! The most interesting feature of this API is that it provides a single way to deal with all your
 //! variable problems. Indeed, the [`FunctionBuilder`](struct.FunctionBuilder.html) struct has a
-//! type parameter `Variable` that should be instantiated with the type of your Source Language
-//! Variables (SLV). Then, through calling the functions
+//! type parameter `Variable` that should be instantiated with the type of your source language
+//! variables. Then, through calling the functions
 //! [`declare_var`](struct.FunctionBuilder.html#method.declare_var),
 //! [`def_var`](struct.FunctionBuilder.html#method.def_var) and
 //! [`use_var`](struct.FunctionBuilder.html#method.use_var), the
 //! [`FunctionBuilder`](struct.FunctionBuilder.html) will create for you all the Cranelift IR
-//! values corresponding to your SLVs.
+//! values corresponding to your variables.
 //!
-//! If a SLV is immutable (defined only once), then it will get mapped to one and only Cranelift IR
-//! value. If a SLV is mutable ([`def_var`](struct.FunctionBuilder.html#method.def_var) multiple
-//! times), then internally a [`SSA`](https://en.wikipedia.org/wiki/Static_single_assignment_form)
-//! construction algorithm will automatically create multiple Cranelift IR values that will be
-//! returned to you by [`use_var`](struct.FunctionBuilder.html#method.use_var) depending on where
-//! you want to use your SLV.
+//! This API has been designed to help you translate your mutable variables into
+//! [`SSA`](https://en.wikipedia.org/wiki/Static_single_assignment_form) form.
+//! [`use_var`](struct.FunctionBuilder.html#method.use_var) will returns the Cranelift IR value
+//! that corresponds to your mutable variable at a precise point in the program. However, you know
+//! beforehand that one of your variables is defined only once, for instance if it is the result
+//! of an intermediate expression in an expression-based language, then you can translate it
+//! directly by the Cranelift IR value returned by the instruction builder. Using the
+//! [`use_var`](struct.FunctionBuilder.html#method.use_var) API for such an immutable variable
+//! would also work but with a slight additional overhead (the SSA algorithm does not know
+//! beforehand if a variable is immutable or not).
 //!
-//! The morality is that you should use these three functions to handle all your SLVs, even those
-//! that are not present in the source code but artefacts of the translation. For instance, if your
-//! source language is expression-based, then you will need to introduce artifical SLVs to store
-//! intermediate results of the computation of your expressions. Hence The `Variable` type that you
+//!
+//! The moral is that you should use these three functions to handle all your mutable variables, even those
+//! that are not present in the source code but artefacts of the translation. Hence The `Variable` type that you
 //! would pass to [`FunctionBuilder`](struct.FunctionBuilder.html) could look like this
 //!
 //! ```
 //! enum Variable {
 //!     OriginalSourceVariable(String),
-//!     IntermediateExpressionVariable(u32)
+//!     TranslationArtefact(u32)
 //! }
 //! ```
 //!

From bed8e33c9d435d33e4258f42de493e360685e48a Mon Sep 17 00:00:00 2001
From: Denis Merigoux 
Date: Tue, 14 Aug 2018 14:37:19 +0200
Subject: [PATCH 2004/3084] Removed "Variable" parametricity for
 FunctionBuilder as discussed in PR
 https://github.com/CraneStation/cranelift/pull/437

---
 lib/frontend/src/frontend.rs    | 68 +++++++++------------------------
 lib/frontend/src/lib.rs         |  4 +-
 lib/wasm/src/code_translator.rs | 24 ++++--------
 lib/wasm/src/func_translator.rs | 10 ++---
 4 files changed, 34 insertions(+), 72 deletions(-)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 5888828ba8..31b27b4f8b 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -1,6 +1,6 @@
 //! A frontend for building Cranelift IR from other languages.
 use cranelift_codegen::cursor::{Cursor, FuncCursor};
-use cranelift_codegen::entity::{EntityMap, EntityRef, EntitySet};
+use cranelift_codegen::entity::{EntityMap, EntitySet};
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::function::DisplayFunction;
 use cranelift_codegen::ir::{
@@ -11,7 +11,7 @@ use cranelift_codegen::ir::{
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::packed_option::PackedOption;
 use ssa::{Block, SSABuilder, SideEffects};
-use std::fmt::Debug;
+use variable::Variable;
 
 /// Structure used for translating a series of functions into Cranelift IR.
 ///
@@ -22,20 +22,14 @@ use std::fmt::Debug;
 /// The `Variable` parameter can be any index-like type that can be made to
 /// implement `EntityRef`. For frontends that don't have an obvious type to
 /// use here, `variable::Variable` can be used.
-pub struct FunctionBuilderContext
-where
-    Variable: EntityRef + Debug,
-{
+pub struct FunctionBuilderContext {
     ssa: SSABuilder,
     ebbs: EntityMap,
     types: EntityMap,
 }
 
 /// Temporary object used to build a single Cranelift IR `Function`.
-pub struct FunctionBuilder<'a, Variable: 'a>
-where
-    Variable: EntityRef + Debug,
-{
+pub struct FunctionBuilder<'a> {
     /// The function currently being built.
     /// This field is public so the function can be re-borrowed.
     pub func: &'a mut Function,
@@ -43,7 +37,7 @@ where
     /// Source location to assign to all new instructions.
     srcloc: ir::SourceLoc,
 
-    func_ctx: &'a mut FunctionBuilderContext,
+    func_ctx: &'a mut FunctionBuilderContext,
     position: Position,
 }
 
@@ -79,10 +73,7 @@ impl Position {
     }
 }
 
-impl FunctionBuilderContext
-where
-    Variable: EntityRef + Debug,
-{
+impl FunctionBuilderContext {
     /// Creates a FunctionBuilderContext structure. The structure is automatically cleared after
     /// each [`FunctionBuilder`](struct.FunctionBuilder.html) completes translating a function.
     pub fn new() -> Self {
@@ -106,30 +97,18 @@ where
 
 /// Implementation of the [`InstBuilder`](../codegen/ir/builder/trait.InstBuilder.html) that has
 /// one convenience method per Cranelift IR instruction.
-pub struct FuncInstBuilder<'short, 'long: 'short, Variable: 'long>
-where
-    Variable: EntityRef + Debug,
-{
-    builder: &'short mut FunctionBuilder<'long, Variable>,
+pub struct FuncInstBuilder<'short, 'long: 'short> {
+    builder: &'short mut FunctionBuilder<'long>,
     ebb: Ebb,
 }
 
-impl<'short, 'long, Variable> FuncInstBuilder<'short, 'long, Variable>
-where
-    Variable: EntityRef + Debug,
-{
-    fn new<'s, 'l>(
-        builder: &'s mut FunctionBuilder<'l, Variable>,
-        ebb: Ebb,
-    ) -> FuncInstBuilder<'s, 'l, Variable> {
+impl<'short, 'long> FuncInstBuilder<'short, 'long> {
+    fn new<'s, 'l>(builder: &'s mut FunctionBuilder<'l>, ebb: Ebb) -> FuncInstBuilder<'s, 'l> {
         FuncInstBuilder { builder, ebb }
     }
 }
 
-impl<'short, 'long, Variable> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long, Variable>
-where
-    Variable: EntityRef + Debug,
-{
+impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
     fn data_flow_graph(&self) -> &DataFlowGraph {
         &self.builder.func.dfg
     }
@@ -229,16 +208,13 @@ where
 /// function in a way that violate the coherence of the code. For instance: switching to a new
 /// `Ebb` when you haven't filled the current one with a terminator instruction, inserting a
 /// return instruction with arguments that don't match the function's signature.
-impl<'a, Variable> FunctionBuilder<'a, Variable>
-where
-    Variable: EntityRef + Debug,
-{
+impl<'a> FunctionBuilder<'a> {
     /// Creates a new FunctionBuilder structure that will operate on a `Function` using a
     /// `FunctionBuilderContext`.
     pub fn new(
         func: &'a mut Function,
-        func_ctx: &'a mut FunctionBuilderContext,
-    ) -> FunctionBuilder<'a, Variable> {
+        func_ctx: &'a mut FunctionBuilderContext,
+    ) -> FunctionBuilder<'a> {
         debug_assert!(func_ctx.is_empty());
         FunctionBuilder {
             func,
@@ -393,7 +369,7 @@ where
 
     /// Returns an object with the [`InstBuilder`](../codegen/ir/builder/trait.InstBuilder.html)
     /// trait that allows to conveniently append an instruction to the current `Ebb` being built.
-    pub fn ins<'short>(&'short mut self) -> FuncInstBuilder<'short, 'a, Variable> {
+    pub fn ins<'short>(&'short mut self) -> FuncInstBuilder<'short, 'a> {
         let ebb = self.position.ebb.unwrap();
         FuncInstBuilder::new(self, ebb)
     }
@@ -486,10 +462,7 @@ where
 /// performance of your translation perform more complex transformations to your Cranelift IR
 /// function. The functions below help you inspect the function you're creating and modify it
 /// in ways that can be unsafe if used incorrectly.
-impl<'a, Variable> FunctionBuilder<'a, Variable>
-where
-    Variable: EntityRef + Debug,
-{
+impl<'a> FunctionBuilder<'a> {
     /// Retrieves all the parameters for an `Ebb` currently inferred from the jump instructions
     /// inserted that target it and the SSA construction.
     pub fn ebb_params(&self, ebb: Ebb) -> &[Value] {
@@ -574,10 +547,7 @@ where
 }
 
 // Helper functions
-impl<'a, Variable> FunctionBuilder<'a, Variable>
-where
-    Variable: EntityRef + Debug,
-{
+impl<'a> FunctionBuilder<'a> {
     fn move_to_next_basic_block(&mut self) {
         self.position.basic_block = PackedOption::from(
             self.func_ctx
@@ -625,10 +595,10 @@ mod tests {
         sig.returns.push(AbiParam::new(I32));
         sig.params.push(AbiParam::new(I32));
 
-        let mut fn_ctx = FunctionBuilderContext::::new();
+        let mut fn_ctx = FunctionBuilderContext::new();
         let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
         {
-            let mut builder = FunctionBuilder::::new(&mut func, &mut fn_ctx);
+            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
 
             let block0 = builder.create_ebb();
             let block1 = builder.create_ebb();
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index 969ed227c4..f6fbf6fece 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -80,10 +80,10 @@
 //!     let mut sig = Signature::new(CallConv::SystemV);
 //!     sig.returns.push(AbiParam::new(I32));
 //!     sig.params.push(AbiParam::new(I32));
-//!     let mut fn_builder_ctx = FunctionBuilderContext::::new();
+//!     let mut fn_builder_ctx = FunctionBuilderContext::new();
 //!     let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
 //!     {
-//!         let mut builder = FunctionBuilder::::new(&mut func, &mut fn_builder_ctx);
+//!         let mut builder = FunctionBuilder::new(&mut func, &mut fn_builder_ctx);
 //!
 //!         let block0 = builder.create_ebb();
 //!         let block1 = builder.create_ebb();
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index 7ab5f00cb6..92619f1bf9 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -43,7 +43,7 @@ use wasmparser::{MemoryImmediate, Operator};
 /// a return.
 pub fn translate_operator(
     op: Operator,
-    builder: &mut FunctionBuilder,
+    builder: &mut FunctionBuilder,
     state: &mut TranslationState,
     environ: &mut FE,
 ) -> WasmResult<()> {
@@ -904,7 +904,7 @@ pub fn translate_operator(
 /// portion so the translation state muts be updated accordingly.
 fn translate_unreachable_operator(
     op: &Operator,
-    builder: &mut FunctionBuilder,
+    builder: &mut FunctionBuilder,
     state: &mut TranslationState,
 ) {
     match *op {
@@ -988,7 +988,7 @@ fn get_heap_addr(
     addr32: ir::Value,
     offset: u32,
     addr_ty: Type,
-    builder: &mut FunctionBuilder,
+    builder: &mut FunctionBuilder,
 ) -> (ir::Value, i32) {
     use std::cmp::min;
 
@@ -1024,7 +1024,7 @@ fn translate_load(
     offset: u32,
     opcode: ir::Opcode,
     result_ty: Type,
-    builder: &mut FunctionBuilder,
+    builder: &mut FunctionBuilder,
     state: &mut TranslationState,
     environ: &mut FE,
 ) {
@@ -1046,7 +1046,7 @@ fn translate_load(
 fn translate_store(
     offset: u32,
     opcode: ir::Opcode,
-    builder: &mut FunctionBuilder,
+    builder: &mut FunctionBuilder,
     state: &mut TranslationState,
     environ: &mut FE,
 ) {
@@ -1063,21 +1063,13 @@ fn translate_store(
         .Store(opcode, val_ty, flags, offset.into(), val, base);
 }
 
-fn translate_icmp(
-    cc: IntCC,
-    builder: &mut FunctionBuilder,
-    state: &mut TranslationState,
-) {
+fn translate_icmp(cc: IntCC, builder: &mut FunctionBuilder, state: &mut TranslationState) {
     let (arg0, arg1) = state.pop2();
     let val = builder.ins().icmp(cc, arg0, arg1);
     state.push1(builder.ins().bint(I32, val));
 }
 
-fn translate_fcmp(
-    cc: FloatCC,
-    builder: &mut FunctionBuilder,
-    state: &mut TranslationState,
-) {
+fn translate_fcmp(cc: FloatCC, builder: &mut FunctionBuilder, state: &mut TranslationState) {
     let (arg0, arg1) = state.pop2();
     let val = builder.ins().fcmp(cc, arg0, arg1);
     state.push1(builder.ins().bint(I32, val));
@@ -1085,7 +1077,7 @@ fn translate_fcmp(
 
 fn translate_br_if(
     relative_depth: u32,
-    builder: &mut FunctionBuilder,
+    builder: &mut FunctionBuilder,
     state: &mut TranslationState,
 ) {
     let val = state.pop1();
diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs
index f9ab937ad4..0f946751db 100644
--- a/lib/wasm/src/func_translator.rs
+++ b/lib/wasm/src/func_translator.rs
@@ -19,7 +19,7 @@ use wasmparser::{self, BinaryReader};
 /// by a `FuncEnvironment` object. A single translator instance can be reused to translate multiple
 /// functions which will reduce heap allocation traffic.
 pub struct FuncTranslator {
-    func_ctx: FunctionBuilderContext,
+    func_ctx: FunctionBuilderContext,
     state: TranslationState,
 }
 
@@ -105,7 +105,7 @@ impl FuncTranslator {
 /// Declare local variables for the signature parameters that correspond to WebAssembly locals.
 ///
 /// Return the number of local variables declared.
-fn declare_wasm_parameters(builder: &mut FunctionBuilder, entry_block: Ebb) -> usize {
+fn declare_wasm_parameters(builder: &mut FunctionBuilder, entry_block: Ebb) -> usize {
     let sig_len = builder.func.signature.params.len();
     let mut next_local = 0;
     for i in 0..sig_len {
@@ -131,7 +131,7 @@ fn declare_wasm_parameters(builder: &mut FunctionBuilder, entry_block:
 /// Declare local variables, starting from `num_params`.
 fn parse_local_decls(
     reader: &mut BinaryReader,
-    builder: &mut FunctionBuilder,
+    builder: &mut FunctionBuilder,
     num_params: usize,
 ) -> WasmResult<()> {
     let mut next_local = num_params;
@@ -155,7 +155,7 @@ fn parse_local_decls(
 ///
 /// Fail of too many locals are declared in the function, or if the type is not valid for a local.
 fn declare_locals(
-    builder: &mut FunctionBuilder,
+    builder: &mut FunctionBuilder,
     count: u32,
     wasm_type: wasmparser::Type,
     next_local: &mut usize,
@@ -185,7 +185,7 @@ fn declare_locals(
 /// arguments and locals are declared in the builder.
 fn parse_function_body(
     mut reader: BinaryReader,
-    builder: &mut FunctionBuilder,
+    builder: &mut FunctionBuilder,
     state: &mut TranslationState,
     environ: &mut FE,
 ) -> WasmResult<()> {

From ce7b72743c3d275d73e783fa2af5cff88c4d201d Mon Sep 17 00:00:00 2001
From: Denis Merigoux 
Date: Tue, 14 Aug 2018 14:51:06 +0200
Subject: [PATCH 2005/3084] Updated doc now that Variable is now longer a type
 parameter

---
 lib/frontend/src/lib.rs | 22 +++++++++-------------
 1 file changed, 9 insertions(+), 13 deletions(-)

diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index f6fbf6fece..07a79f4357 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -10,8 +10,8 @@
 //!
 //! The most interesting feature of this API is that it provides a single way to deal with all your
 //! variable problems. Indeed, the [`FunctionBuilder`](struct.FunctionBuilder.html) struct has a
-//! type parameter `Variable` that should be instantiated with the type of your source language
-//! variables. Then, through calling the functions
+//! type `Variable` that should be an index of your source language variables. Then, through
+//! calling the functions
 //! [`declare_var`](struct.FunctionBuilder.html#method.declare_var),
 //! [`def_var`](struct.FunctionBuilder.html#method.def_var) and
 //! [`use_var`](struct.FunctionBuilder.html#method.use_var), the
@@ -29,17 +29,13 @@
 //! would also work but with a slight additional overhead (the SSA algorithm does not know
 //! beforehand if a variable is immutable or not).
 //!
-//!
-//! The moral is that you should use these three functions to handle all your mutable variables, even those
-//! that are not present in the source code but artefacts of the translation. Hence The `Variable` type that you
-//! would pass to [`FunctionBuilder`](struct.FunctionBuilder.html) could look like this
-//!
-//! ```
-//! enum Variable {
-//!     OriginalSourceVariable(String),
-//!     TranslationArtefact(u32)
-//! }
-//! ```
+//! The moral is that you should use these three functions to handle all your mutable variables,
+//! even those that are not present in the source code but artefacts of the translation. It is up
+//! to you to keep a mapping between the mutable variables of your language and their `Variable`
+//! index that is used by Cranelift. Caution: as the `Variable` is used by Cranelift to index an
+//! array containing information about your mutable variables, when you create a new `Variable`
+//! with [`Variable::new(var_index)`] you should make sure that `var_index` is provided by a
+//! counter incremented by 1 each time you encounter a new mutable variable.
 //!
 //! # Example
 //!

From 5f679a73108ccfa89e349ba81ce19847348d05ac Mon Sep 17 00:00:00 2001
From: Caroline Cullen 
Date: Tue, 14 Aug 2018 15:18:10 -0700
Subject: [PATCH 2006/3084] Change command line parsing to clap in serde util
 #434 (#435)

* Change command line parsing to clap in serde util #434
---
 lib/serde/Cargo.toml       |   2 +-
 lib/serde/src/clif-json.rs | 104 ++++++++++++++++---------------------
 2 files changed, 46 insertions(+), 60 deletions(-)

diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index ecbdd781e8..b1224c9a12 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -13,7 +13,7 @@ name = "clif-json"
 path = "src/clif-json.rs"
 
 [dependencies]
-docopt = "1"
+clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.8"
 serde_json = "1.0"
diff --git a/lib/serde/src/clif-json.rs b/lib/serde/src/clif-json.rs
index 5fc8a1d469..7cc674c2b6 100644
--- a/lib/serde/src/clif-json.rs
+++ b/lib/serde/src/clif-json.rs
@@ -12,49 +12,23 @@
     )
 )]
 
+extern crate clap;
 extern crate cranelift_reader;
-extern crate docopt;
+extern crate serde_json;
 #[macro_use]
 extern crate serde_derive;
 extern crate cranelift_codegen;
 
-extern crate serde_json;
-
+use clap::{App, Arg, SubCommand};
 use cranelift_reader::parse_functions;
-use docopt::Docopt;
 use std::fs::File;
 use std::io::prelude::*;
 use std::io::{self, Write};
 use std::process;
-use std::vec::Vec;
 
 mod serde_clif_json;
 
-const USAGE: &str = "
-Cranelift JSON serializer/deserializer utility
-
-Usage:
-    clif-json serialize [-p] 
-    clif-json deserialize 
-
-Options:
-    -p, --pretty     print pretty json
-
-";
-
-#[derive(Deserialize, Debug)]
-struct Args {
-    cmd_serialize: bool,
-    cmd_deserialize: bool,
-    flag_pretty: bool,
-    arg_file: Vec,
-}
-
-/// A command either succeeds or fails with an error message.
-pub type CommandResult = Result<(), String>;
-
-/// Serialize Cranelift IR to JSON
-fn call_ser(file: &str, pretty: bool) -> CommandResult {
+fn call_ser(file: &str, pretty: bool) -> Result<(), String> {
     let ret_of_parse = parse_functions(file);
     match ret_of_parse {
         Ok(funcs) => {
@@ -68,12 +42,11 @@ fn call_ser(file: &str, pretty: bool) -> CommandResult {
             println!("{}", ser_str);
             Ok(())
         }
-        Err(_pe) => Err(format!("this was a parsing error")),
+        Err(_pe) => Err(format!("There was a parsing error")),
     }
 }
 
-/// Deserialize JSON to Cranelift IR
-fn call_de(file: &File) -> CommandResult {
+fn call_de(file: &File) -> Result<(), String> {
     let de: serde_clif_json::SerObj = match serde_json::from_reader(file) {
         Result::Ok(val) => val,
         Result::Err(err) => panic!("{}", err),
@@ -82,41 +55,54 @@ fn call_de(file: &File) -> CommandResult {
     Ok(())
 }
 
-/// Parse the command line arguments and run the requested command.
-fn clif_json() -> CommandResult {
-    // Parse command line arguments.
-    let args: Args = Docopt::new(USAGE)
-        .and_then(|d| d.help(true).deserialize())
-        .unwrap_or_else(|e| e.exit());
+fn main() {
+    let matches = App::new("Cranelift JSON serializer/deserializer utility")
+        .subcommand(
+            SubCommand::with_name("serialize")
+                .display_order(1)
+                .about("Serializes Cranelift IR into JSON.")
+                .arg(Arg::with_name("pretty").short("p").help("pretty json"))
+                .arg(
+                    Arg::with_name("FILE")
+                        .required(true)
+                        .value_name("FILE")
+                        .help("Input file for serialization"),
+                ),
+        )
+        .subcommand(
+            SubCommand::with_name("deserialize")
+                .about("Deserializes Cranelift IR into JSON.")
+                .arg(
+                    Arg::with_name("FILE")
+                        .required(true)
+                        .value_name("FILE")
+                        .help("Input file for deserialization"),
+                ),
+        )
+        .get_matches();
 
-    // Find the sub-command to execute.
-    let result = if args.cmd_serialize {
-        if let Some(first_file) = args.arg_file.first() {
-            let mut file = File::open(first_file).expect("Unable to open the file");
+    let res_serde = match matches.subcommand() {
+        ("serialize", Some(m)) => {
+            let mut file =
+                File::open(m.value_of("FILE").unwrap()).expect("Unable to open the file");
             let mut contents = String::new();
             file.read_to_string(&mut contents)
                 .expect("Unable to read the file");
-            call_ser(&contents, args.flag_pretty)
-        } else {
-            Err(format!("No file was passed"))
+
+            match m.occurrences_of("pretty") {
+                0 => call_ser(&contents, false),
+                _ => call_ser(&contents, true),
+            }
         }
-    } else if args.cmd_deserialize {
-        if let Some(first_file) = args.arg_file.first() {
-            let mut file = File::open(first_file).expect("Unable to open the file");
+        ("deserialize", Some(m)) => {
+            let mut file =
+                File::open(m.value_of("FILE").unwrap()).expect("Unable to open the file");
             call_de(&file)
-        } else {
-            Err(format!("No file was passed"))
         }
-    } else {
-        // Debugging / shouldn't happen with proper command line handling above.
-        Err(format!("Unhandled args: {:?}", args))
+        _ => Err(format!("Invalid subcommand.")),
     };
 
-    result
-}
-
-fn main() {
-    if let Err(mut msg) = clif_json() {
+    if let Err(mut msg) = res_serde {
         if !msg.ends_with('\n') {
             msg.push('\n');
         }

From 304134d351d6f539ec2a73b982840436cea993ee Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 14 Aug 2018 16:47:45 -0700
Subject: [PATCH 2007/3084] Update a testcase for API changes.

---
 cranelift/tests/moduletests.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/tests/moduletests.rs b/cranelift/tests/moduletests.rs
index 1f4c9d204a..c2312fe4f2 100644
--- a/cranelift/tests/moduletests.rs
+++ b/cranelift/tests/moduletests.rs
@@ -47,7 +47,7 @@ fn define_simple_function(module: &mut Module) -> FuncId {
     ctx.func = Function::with_name_signature(ExternalName::user(0, func_id.index() as u32), sig);
     let mut func_ctx = FunctionBuilderContext::new();
     {
-        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
         let ebb = bcx.create_ebb();
         bcx.switch_to_block(ebb);
         bcx.ins().return_(&[]);

From e2badb0ad63ad7a358f2dc35c3218b69b5118732 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gr=C3=A9goire=20Geis?= 
Date: Thu, 16 Aug 2018 20:34:52 +0200
Subject: [PATCH 2008/3084] Improvements to error reporting (#470)

* Fixed error reporting.

* Fixed compile time error when wasm feature is disabled.

* Fixed valid instructions not being printed in print_function_error.

* Fixed errors print_function_error not writing valid instructions after end.

* Made multiple checks non-fatal.

* verify_global_values is no longer fatal.

* Slightly better formatting of errors in pretty_verifier_error.
---
 cranelift/src/clif-util.rs      |   2 +-
 lib/codegen/src/print_errors.rs |  33 +++++++--
 lib/codegen/src/verifier/mod.rs | 121 ++++++++++++++++++++------------
 3 files changed, 104 insertions(+), 52 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 890bf95be7..5b8eb2f3c7 100644
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -21,12 +21,12 @@ extern crate serde_derive;
 #[cfg(feature = "disas")]
 extern crate capstone;
 extern crate pretty_env_logger;
-extern crate term;
 
 cfg_if! {
     if #[cfg(feature = "wasm")] {
         extern crate cranelift_entity;
         extern crate cranelift_wasm;
+        extern crate term;
         extern crate wabt;
         mod wasm;
     }
diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 3d012ad8fd..472f8ab005 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -25,17 +25,23 @@ pub fn pretty_verifier_error<'a>(
 
     // TODO: Use drain_filter here when it gets stabilized
     let mut i = 0;
+    let mut wrote_error = false;
 
     while i != errors.len() {
         if let ir::entities::AnyEntity::Inst(_) = errors[i].location {
+            i += 1;
+        } else {
             let err = errors.remove(i);
 
-            writeln!(w, "Miscellaneous error: {}\n", err).unwrap()
-        } else {
-            i += 1;
+            writeln!(w, "verifier at {}", err).unwrap();
+            wrote_error = true;
         }
     }
 
+    if wrote_error {
+        w.push('\n');
+    }
+
     decorate_function(
         &mut PrettyVerifierError(func_w.unwrap_or(Box::new(PlainWriter)), &mut errors),
         &mut w,
@@ -81,24 +87,41 @@ fn pretty_function_error(
 ) -> fmt::Result {
     // TODO: Use drain_filter here when it gets stabilized
     let mut i = 0;
+    let mut printed_instr = false;
 
     while i != errors.len() {
         match errors[i].location {
             ir::entities::AnyEntity::Inst(inst) if inst == cur_inst => {
                 let err = errors.remove(i);
 
-                func_w.write_instruction(w, func, isa, cur_inst, indent)?;
+                if !printed_instr {
+                    func_w.write_instruction(w, func, isa, cur_inst, indent)?;
+                    printed_instr = true;
+                }
+
                 write!(w, "{1:0$}^", indent, "")?;
                 for _c in cur_inst.to_string().chars() {
                     write!(w, "~")?;
                 }
-                writeln!(w, " verifier {}\n", err.to_string())?;
+                writeln!(w, " verifier {}", err.to_string())?;
             }
             ir::entities::AnyEntity::Inst(_) => i += 1,
             _ => unreachable!(),
         }
     }
 
+    if printed_instr {
+        w.write_char('\n')?;
+    } else {
+        writeln!(
+            w,
+            "{1:0$}{2}",
+            indent,
+            "",
+            func.dfg.display_inst(cur_inst, isa)
+        )?;
+    }
+
     Ok(())
 }
 
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 66ad44fa11..bb116b00b3 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -230,6 +230,17 @@ impl VerifierErrors {
     pub fn has_error(&self) -> bool {
         !self.0.is_empty()
     }
+
+    /// Return a `VerifierStepResult` that is fatal if at least one error was reported,
+    /// and non-fatal otherwise.
+    #[inline]
+    pub fn as_result(&self) -> VerifierStepResult<()> {
+        if self.is_empty() {
+            Ok(())
+        } else {
+            Err(())
+        }
+    }
 }
 
 impl From> for VerifierErrors {
@@ -244,6 +255,16 @@ impl Into> for VerifierErrors {
     }
 }
 
+impl Into> for VerifierErrors {
+    fn into(self) -> VerifierResult<()> {
+        if self.is_empty() {
+            Ok(())
+        } else {
+            Err(self)
+        }
+    }
+}
+
 impl Display for VerifierErrors {
     fn fmt(&self, f: &mut Formatter) -> fmt::Result {
         for err in &self.0 {
@@ -309,14 +330,15 @@ impl<'a> Verifier<'a> {
     fn verify_global_values(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         let mut seen = SparseSet::new();
 
-        for gv in self.func.global_values.keys() {
+        'gvs: for gv in self.func.global_values.keys() {
             seen.clear();
             seen.insert(gv);
 
             let mut cur = gv;
             while let ir::GlobalValueData::Deref { base, .. } = self.func.global_values[cur] {
                 if seen.insert(base).is_some() {
-                    return fatal!(errors, gv, "deref cycle: {}", DisplayList(seen.as_slice()));
+                    report!(errors, gv, "deref cycle: {}", DisplayList(seen.as_slice()));
+                    continue 'gvs;
                 }
 
                 cur = base;
@@ -328,11 +350,12 @@ impl<'a> Verifier<'a> {
                     .special_param(ir::ArgumentPurpose::VMContext)
                     .is_none()
                 {
-                    return fatal!(errors, cur, "undeclared vmctx reference {}", cur);
+                    report!(errors, cur, "undeclared vmctx reference {}", cur);
                 }
             }
         }
 
+        // Invalid global values shouldn't stop us from verifying the rest of the function
         Ok(())
     }
 
@@ -437,7 +460,7 @@ impl<'a> Verifier<'a> {
             // All used values must be attached to something.
             let original = self.func.dfg.resolve_aliases(arg);
             if !self.func.dfg.value_is_attached(original) {
-                return fatal!(
+                report!(
                     errors,
                     inst,
                     "argument {} -> {} is not attached",
@@ -448,7 +471,7 @@ impl<'a> Verifier<'a> {
         }
 
         for &res in self.func.dfg.inst_results(inst) {
-            self.verify_inst_result(inst, res, errors)?;
+            self.verify_inst_result(inst, res, errors).is_ok();
         }
 
         match self.func.dfg[inst] {
@@ -594,7 +617,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.dfg.ext_funcs.is_valid(f) {
-            fatal!(errors, inst, "invalid function reference {}", f)
+            nonfatal!(errors, inst, "invalid function reference {}", f)
         } else {
             Ok(())
         }
@@ -607,7 +630,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.stack_slots.is_valid(ss) {
-            fatal!(errors, inst, "invalid stack slot {}", ss)
+            nonfatal!(errors, inst, "invalid stack slot {}", ss)
         } else {
             Ok(())
         }
@@ -620,7 +643,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.global_values.is_valid(gv) {
-            fatal!(errors, inst, "invalid global value {}", gv)
+            nonfatal!(errors, inst, "invalid global value {}", gv)
         } else {
             Ok(())
         }
@@ -633,7 +656,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.heaps.is_valid(heap) {
-            fatal!(errors, inst, "invalid heap {}", heap)
+            nonfatal!(errors, inst, "invalid heap {}", heap)
         } else {
             Ok(())
         }
@@ -646,7 +669,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.tables.is_valid(table) {
-            fatal!(errors, inst, "invalid table {}", table)
+            nonfatal!(errors, inst, "invalid table {}", table)
         } else {
             Ok(())
         }
@@ -659,7 +682,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !l.is_valid(&self.func.dfg.value_lists) {
-            fatal!(errors, inst, "invalid value list reference {:?}", l)
+            nonfatal!(errors, inst, "invalid value list reference {:?}", l)
         } else {
             Ok(())
         }
@@ -672,7 +695,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.jump_tables.is_valid(j) {
-            fatal!(errors, inst, "invalid jump table reference {}", j)
+            nonfatal!(errors, inst, "invalid jump table reference {}", j)
         } else {
             Ok(())
         }
@@ -686,7 +709,7 @@ impl<'a> Verifier<'a> {
     ) -> VerifierStepResult<()> {
         let dfg = &self.func.dfg;
         if !dfg.value_is_valid(v) {
-            fatal!(errors, loc_inst, "invalid value reference {}", v)
+            nonfatal!(errors, loc_inst, "invalid value reference {}", v)
         } else {
             Ok(())
         }
@@ -897,7 +920,7 @@ impl<'a> Verifier<'a> {
             for (i, &arg) in self.func.dfg.ebb_params(ebb).iter().enumerate() {
                 let arg_type = self.func.dfg.value_type(arg);
                 if arg_type != expected_types[i].value_type {
-                    return fatal!(
+                    report!(
                         errors,
                         ebb,
                         "entry block parameter {} expected to have type {}, got {}",
@@ -908,7 +931,8 @@ impl<'a> Verifier<'a> {
                 }
             }
         }
-        Ok(())
+
+        errors.as_result()
     }
 
     fn typecheck(&self, inst: Inst, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
@@ -920,7 +944,7 @@ impl<'a> Verifier<'a> {
             let ctrl_type = self.func.dfg.ctrl_typevar(inst);
 
             if !value_typeset.contains(ctrl_type) {
-                return fatal!(
+                report!(
                     errors,
                     inst,
                     "has an invalid controlling type {}",
@@ -935,11 +959,12 @@ impl<'a> Verifier<'a> {
             types::VOID
         };
 
-        self.typecheck_results(inst, ctrl_type, errors)?;
-        self.typecheck_fixed_args(inst, ctrl_type, errors)?;
-        self.typecheck_variable_args(inst, errors)?;
-        self.typecheck_return(inst, errors)?;
-        self.typecheck_special(inst, ctrl_type, errors)?;
+        // Typechecking instructions is never fatal
+        self.typecheck_results(inst, ctrl_type, errors).is_ok();
+        self.typecheck_fixed_args(inst, ctrl_type, errors).is_ok();
+        self.typecheck_variable_args(inst, errors).is_ok();
+        self.typecheck_return(inst, errors).is_ok();
+        self.typecheck_special(inst, ctrl_type, errors).is_ok();
 
         Ok(())
     }
@@ -956,7 +981,7 @@ impl<'a> Verifier<'a> {
             let expected_type = self.func.dfg.compute_result_type(inst, i, ctrl_type);
             if let Some(expected_type) = expected_type {
                 if result_type != expected_type {
-                    return fatal!(
+                    report!(
                         errors,
                         inst,
                         "expected result {} ({}) to have type {}, found {}",
@@ -967,14 +992,14 @@ impl<'a> Verifier<'a> {
                     );
                 }
             } else {
-                return fatal!(errors, inst, "has more result values than expected");
+                return nonfatal!(errors, inst, "has more result values than expected");
             }
             i += 1;
         }
 
         // There aren't any more result types left.
         if self.func.dfg.compute_result_type(inst, i, ctrl_type) != None {
-            return fatal!(errors, inst, "has fewer result values than expected");
+            return nonfatal!(errors, inst, "has fewer result values than expected");
         }
         Ok(())
     }
@@ -992,7 +1017,7 @@ impl<'a> Verifier<'a> {
             match constraints.value_argument_constraint(i, ctrl_type) {
                 ResolvedConstraint::Bound(expected_type) => {
                     if arg_type != expected_type {
-                        return fatal!(
+                        report!(
                             errors,
                             inst,
                             "arg {} ({}) has type {}, expected {}",
@@ -1005,7 +1030,7 @@ impl<'a> Verifier<'a> {
                 }
                 ResolvedConstraint::Free(type_set) => {
                     if !type_set.contains(arg_type) {
-                        return fatal!(
+                        report!(
                             errors,
                             inst,
                             "arg {} ({}) with type {} failed to satisfy type set {:?}",
@@ -1040,7 +1065,7 @@ impl<'a> Verifier<'a> {
                 for (_, ebb) in self.func.jump_tables[table].entries() {
                     let arg_count = self.func.dfg.num_ebb_params(ebb);
                     if arg_count != 0 {
-                        return fatal!(
+                        return nonfatal!(
                             errors,
                             inst,
                             "takes no arguments, but had target {} with {} arguments",
@@ -1094,7 +1119,7 @@ impl<'a> Verifier<'a> {
             let arg = variable_args[i];
             let arg_type = self.func.dfg.value_type(arg);
             if expected_type != arg_type {
-                return fatal!(
+                report!(
                     errors,
                     inst,
                     "arg {} ({}) has type {}, expected {}",
@@ -1107,7 +1132,7 @@ impl<'a> Verifier<'a> {
             i += 1;
         }
         if i != variable_args.len() {
-            return fatal!(
+            return nonfatal!(
                 errors,
                 inst,
                 "mismatched argument count for `{}`: got {}, expected {}",
@@ -1199,7 +1224,7 @@ impl<'a> Verifier<'a> {
             let args = self.func.dfg.inst_variable_args(inst);
             let expected_types = &self.func.signature.returns;
             if args.len() != expected_types.len() {
-                return fatal!(
+                return nonfatal!(
                     errors,
                     inst,
                     "arguments of return must match function signature"
@@ -1208,7 +1233,7 @@ impl<'a> Verifier<'a> {
             for (i, (&arg, &expected_type)) in args.iter().zip(expected_types).enumerate() {
                 let arg_type = self.func.dfg.value_type(arg);
                 if arg_type != expected_type.value_type {
-                    return fatal!(
+                    report!(
                         errors,
                         inst,
                         "arg {} ({}) has type {}, must match function signature of {}",
@@ -1236,7 +1261,7 @@ impl<'a> Verifier<'a> {
             match opcode {
                 Opcode::Bextend | Opcode::Uextend | Opcode::Sextend | Opcode::Fpromote => {
                     if arg_type.lane_count() != ctrl_type.lane_count() {
-                        return fatal!(
+                        return nonfatal!(
                             errors,
                             inst,
                             "input {} and output {} must have same number of lanes",
@@ -1245,7 +1270,7 @@ impl<'a> Verifier<'a> {
                         );
                     }
                     if arg_type.lane_bits() >= ctrl_type.lane_bits() {
-                        return fatal!(
+                        return nonfatal!(
                             errors,
                             inst,
                             "input {} must be smaller than output {}",
@@ -1256,7 +1281,7 @@ impl<'a> Verifier<'a> {
                 }
                 Opcode::Breduce | Opcode::Ireduce | Opcode::Fdemote => {
                     if arg_type.lane_count() != ctrl_type.lane_count() {
-                        return fatal!(
+                        return nonfatal!(
                             errors,
                             inst,
                             "input {} and output {} must have same number of lanes",
@@ -1265,7 +1290,7 @@ impl<'a> Verifier<'a> {
                         );
                     }
                     if arg_type.lane_bits() <= ctrl_type.lane_bits() {
-                        return fatal!(
+                        return nonfatal!(
                             errors,
                             inst,
                             "input {} must be larger than output {}",
@@ -1296,22 +1321,24 @@ impl<'a> Verifier<'a> {
 
             let missing_succs: Vec = expected_succs.difference(&got_succs).cloned().collect();
             if !missing_succs.is_empty() {
-                return fatal!(
+                report!(
                     errors,
                     ebb,
                     "cfg lacked the following successor(s) {:?}",
                     missing_succs
                 );
+                continue;
             }
 
             let excess_succs: Vec = got_succs.difference(&expected_succs).cloned().collect();
             if !excess_succs.is_empty() {
-                return fatal!(
+                report!(
                     errors,
                     ebb,
                     "cfg had unexpected successor(s) {:?}",
                     excess_succs
                 );
+                continue;
             }
 
             expected_preds.extend(
@@ -1323,22 +1350,24 @@ impl<'a> Verifier<'a> {
 
             let missing_preds: Vec = expected_preds.difference(&got_preds).cloned().collect();
             if !missing_preds.is_empty() {
-                return fatal!(
+                report!(
                     errors,
                     ebb,
                     "cfg lacked the following predecessor(s) {:?}",
                     missing_preds
                 );
+                continue;
             }
 
             let excess_preds: Vec = got_preds.difference(&expected_preds).cloned().collect();
             if !excess_preds.is_empty() {
-                return fatal!(
+                report!(
                     errors,
                     ebb,
                     "cfg had unexpected predecessor(s) {:?}",
                     excess_preds
                 );
+                continue;
             }
 
             expected_succs.clear();
@@ -1346,7 +1375,7 @@ impl<'a> Verifier<'a> {
             expected_preds.clear();
             got_preds.clear();
         }
-        Ok(())
+        errors.as_result()
     }
 
     /// If the verifier has been set up with an ISA, make sure that the recorded encoding for the
@@ -1375,7 +1404,7 @@ impl<'a> Verifier<'a> {
                 ).peekable();
 
             if encodings.peek().is_none() {
-                return fatal!(
+                return nonfatal!(
                     errors,
                     inst,
                     "Instruction failed to re-encode {}",
@@ -1403,7 +1432,7 @@ impl<'a> Verifier<'a> {
                         .unwrap();
                 }
 
-                return fatal!(
+                return nonfatal!(
                     errors,
                     inst,
                     "encoding {} should be {}{}",
@@ -1446,7 +1475,7 @@ impl<'a> Verifier<'a> {
             // Provide the ISA default encoding as a hint.
             match self.func.encode(inst, isa) {
                 Ok(enc) => {
-                    return fatal!(
+                    return nonfatal!(
                         errors,
                         inst,
                         "{} must have an encoding (e.g., {})",
@@ -1454,7 +1483,7 @@ impl<'a> Verifier<'a> {
                         isa.encoding_info().display(enc)
                     )
                 }
-                Err(_) => return fatal!(errors, inst, "{} must have an encoding", text),
+                Err(_) => return nonfatal!(errors, inst, "{} must have an encoding", text),
             }
         }
 
@@ -1468,7 +1497,7 @@ impl<'a> Verifier<'a> {
             let inst = self.func.layout.last_inst(ebb).unwrap();
             if self.func.dfg[inst].opcode().is_return() && Some(ebb) != self.func.layout.last_ebb()
             {
-                return fatal!(
+                report!(
                     errors,
                     inst,
                     "Internal return not allowed with return_at_end=1"
@@ -1476,7 +1505,7 @@ impl<'a> Verifier<'a> {
             }
         }
 
-        Ok(())
+        errors.as_result()
     }
 
     pub fn run(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {

From 3d89a8645b86d9e2670af25b327f00ada05060e5 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 14 Aug 2018 22:03:07 -0700
Subject: [PATCH 2009/3084] Fix rustfmt errors.

---
 cranelift/fuzz/fuzz_reader_parse_test.rs |  6 ++---
 cranelift/src/wasm.rs                    | 22 +++++++++++-----
 lib/codegen/src/constant_hash.rs         |  4 +--
 lib/codegen/src/isa/mod.rs               |  2 +-
 lib/codegen/src/isa/x86/abi.rs           |  4 +--
 lib/codegen/src/isa/x86/enc_tables.rs    | 32 +++++++++++++++---------
 lib/codegen/src/legalizer/heap.rs        |  4 +--
 lib/codegen/src/regalloc/coalescing.rs   |  4 +--
 lib/codegen/src/regalloc/liveness.rs     |  4 +--
 lib/frontend/src/lib.rs                  |  4 +--
 lib/reader/src/parser.rs                 |  4 +--
 lib/serde/README.md                      |  8 +++---
 lib/serde/src/serde_clif_json.rs         | 10 +++++---
 lib/wasm/src/state.rs                    |  4 +--
 14 files changed, 66 insertions(+), 46 deletions(-)

diff --git a/cranelift/fuzz/fuzz_reader_parse_test.rs b/cranelift/fuzz/fuzz_reader_parse_test.rs
index fc5709e4b2..8ef582af05 100644
--- a/cranelift/fuzz/fuzz_reader_parse_test.rs
+++ b/cranelift/fuzz/fuzz_reader_parse_test.rs
@@ -4,8 +4,6 @@ extern crate libfuzzer_sys;
 extern crate cranelift_reader;
 use std::str;
 
-fuzz_target!(|data: &[u8]| {
-    if let Ok(s) = str::from_utf8(data) {
-        let _ = cranelift_reader::parse_test(s);
-    }
+fuzz_target!(|data: &[u8]| if let Ok(s) = str::from_utf8(data) {
+    let _ = cranelift_reader::parse_test(s);
 });
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 34fcb26ad0..f446bb6ccc 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -4,9 +4,9 @@
 //! Reads Wasm binary files, translates the functions' code to Cranelift IR.
 #![cfg_attr(feature = "cargo-clippy", allow(too_many_arguments, cyclomatic_complexity))]
 
-use cranelift_codegen::Context;
 use cranelift_codegen::print_errors::{pretty_error, pretty_verifier_error};
 use cranelift_codegen::settings::FlagsOrIsa;
+use cranelift_codegen::Context;
 use cranelift_entity::EntityRef;
 use cranelift_wasm::{translate_module, DummyEnvironment, FuncIndex, ModuleEnvironment};
 use std::error::Error;
@@ -80,7 +80,8 @@ fn handle_module(
     vprint!(flag_verbose, "Translating... ");
     terminal.reset().unwrap();
 
-    let mut module_binary = read_to_end(path.clone()).map_err(|err| String::from(err.description()))?;
+    let mut module_binary =
+        read_to_end(path.clone()).map_err(|err| String::from(err.description()))?;
 
     if !module_binary.starts_with(&[b'\0', b'a', b's', b'm']) {
         module_binary = match wat2wasm(&module_binary) {
@@ -91,7 +92,11 @@ fn handle_module(
 
     let isa = match fisa.isa {
         Some(isa) => isa,
-        None => return Err(String::from("Error: the wasm command requires an explicit isa."))
+        None => {
+            return Err(String::from(
+                "Error: the wasm command requires an explicit isa.",
+            ))
+        }
     };
 
     let mut dummy_environ =
@@ -118,7 +123,9 @@ fn handle_module(
                 }
             }
             vprintln!(flag_verbose, "");
-            for export_name in &dummy_environ.info.functions[FuncIndex::new(func_index)].export_names {
+            for export_name in
+                &dummy_environ.info.functions[FuncIndex::new(func_index)].export_names
+            {
                 println!("; Exported as \"{}\"", export_name);
             }
             println!("{}", context.func.display(None));
@@ -163,7 +170,8 @@ fn handle_module(
                 total_module_code_size += compiled_size;
                 println!(
                     "Function #{} bytecode size: {} bytes",
-                    func_index, dummy_environ.func_bytecode_sizes[def_index.index()]
+                    func_index,
+                    dummy_environ.func_bytecode_sizes[def_index.index()]
                 );
             }
         }
@@ -175,7 +183,9 @@ fn handle_module(
                     println!("; Selected as wasm start function");
                 }
             }
-            for export_name in &dummy_environ.info.functions[FuncIndex::new(func_index)].export_names {
+            for export_name in
+                &dummy_environ.info.functions[FuncIndex::new(func_index)].export_names
+            {
                 println!("; Exported as \"{}\"", export_name);
             }
             println!("{}", context.func.display(fisa.isa));
diff --git a/lib/codegen/src/constant_hash.rs b/lib/codegen/src/constant_hash.rs
index 553543ec60..0aaa3c51c2 100644
--- a/lib/codegen/src/constant_hash.rs
+++ b/lib/codegen/src/constant_hash.rs
@@ -1,7 +1,7 @@
 //! Runtime support for precomputed constant hash tables.
 //!
-//! The `lib/codegen/meta-python/constant_hash.py` Python module can generate constant hash tables using
-//! open addressing and quadratic probing. The hash tables are arrays that are guaranteed to:
+//! The `lib/codegen/meta-python/constant_hash.py` Python module can generate constant hash tables
+//! using open addressing and quadratic probing. The hash tables are arrays that are guaranteed to:
 //!
 //! - Have a power-of-two size.
 //! - Contain at least one empty slot.
diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs
index f5dd62f294..b95c32f967 100644
--- a/lib/codegen/src/isa/mod.rs
+++ b/lib/codegen/src/isa/mod.rs
@@ -23,8 +23,8 @@
 //! # extern crate cranelift_codegen;
 //! # #[macro_use] extern crate target_lexicon;
 //! # fn main() {
-//! use cranelift_codegen::settings::{self, Configurable};
 //! use cranelift_codegen::isa;
+//! use cranelift_codegen::settings::{self, Configurable};
 //! use std::str::FromStr;
 //! use target_lexicon::Triple;
 //!
diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs
index 21796e2a10..6028fa507d 100644
--- a/lib/codegen/src/isa/x86/abi.rs
+++ b/lib/codegen/src/isa/x86/abi.rs
@@ -433,8 +433,8 @@ fn insert_common_prologue(
         if let Some(stack_limit_arg) = pos.func.special_param(ArgumentPurpose::StackLimit) {
             // Total stack size is the size of all stack area used by the function, including
             // pushed CSRs, frame pointer.
-            // Also, the size of a return address, implicitly pushed by a x86 `call` instruction, also
-            // should be accounted for.
+            // Also, the size of a return address, implicitly pushed by a x86 `call` instruction,
+            // also should be accounted for.
             // TODO: Check if the function body actually contains a `call` instruction.
             let word_size = isa.pointer_bytes();
             let total_stack_size = (csrs.iter(GPR).len() + 1 + 1) as i64 * word_size as i64;
diff --git a/lib/codegen/src/isa/x86/enc_tables.rs b/lib/codegen/src/isa/x86/enc_tables.rs
index 52af977ad6..ebaaa526f7 100644
--- a/lib/codegen/src/isa/x86/enc_tables.rs
+++ b/lib/codegen/src/isa/x86/enc_tables.rs
@@ -367,23 +367,27 @@ fn expand_fcvt_to_sint(
     let output_bits = ty.lane_bits();
     let flimit = match xty {
         ir::types::F32 =>
-            // An f32 can represent `i16::min_value() - 1` exactly with precision to spare, so
-            // there are values less than -2^(N-1) that convert correctly to INT_MIN.
+        // An f32 can represent `i16::min_value() - 1` exactly with precision to spare, so
+        // there are values less than -2^(N-1) that convert correctly to INT_MIN.
+        {
             pos.ins().f32const(if output_bits < 32 {
                 overflow_cc = FloatCC::LessThanOrEqual;
                 Ieee32::fcvt_to_sint_negative_overflow(output_bits)
             } else {
                 Ieee32::pow2(output_bits - 1).neg()
-            }),
+            })
+        }
         ir::types::F64 =>
-            // An f64 can represent `i32::min_value() - 1` exactly with precision to spare, so
-            // there are values less than -2^(N-1) that convert correctly to INT_MIN.
+        // An f64 can represent `i32::min_value() - 1` exactly with precision to spare, so
+        // there are values less than -2^(N-1) that convert correctly to INT_MIN.
+        {
             pos.ins().f64const(if output_bits < 64 {
                 overflow_cc = FloatCC::LessThanOrEqual;
                 Ieee64::fcvt_to_sint_negative_overflow(output_bits)
             } else {
                 Ieee64::pow2(output_bits - 1).neg()
-            }),
+            })
+        }
         _ => panic!("Can't convert {}", xty),
     };
     let overflow = pos.ins().fcmp(overflow_cc, x, flimit);
@@ -464,23 +468,27 @@ fn expand_fcvt_to_sint_sat(
     let output_bits = ty.lane_bits();
     let flimit = match xty {
         ir::types::F32 =>
-            // An f32 can represent `i16::min_value() - 1` exactly with precision to spare, so
-            // there are values less than -2^(N-1) that convert correctly to INT_MIN.
+        // An f32 can represent `i16::min_value() - 1` exactly with precision to spare, so
+        // there are values less than -2^(N-1) that convert correctly to INT_MIN.
+        {
             pos.ins().f32const(if output_bits < 32 {
                 overflow_cc = FloatCC::LessThanOrEqual;
                 Ieee32::fcvt_to_sint_negative_overflow(output_bits)
             } else {
                 Ieee32::pow2(output_bits - 1).neg()
-            }),
+            })
+        }
         ir::types::F64 =>
-            // An f64 can represent `i32::min_value() - 1` exactly with precision to spare, so
-            // there are values less than -2^(N-1) that convert correctly to INT_MIN.
+        // An f64 can represent `i32::min_value() - 1` exactly with precision to spare, so
+        // there are values less than -2^(N-1) that convert correctly to INT_MIN.
+        {
             pos.ins().f64const(if output_bits < 64 {
                 overflow_cc = FloatCC::LessThanOrEqual;
                 Ieee64::fcvt_to_sint_negative_overflow(output_bits)
             } else {
                 Ieee64::pow2(output_bits - 1).neg()
-            }),
+            })
+        }
         _ => panic!("Can't convert {}", xty),
     };
 
diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs
index 44f434ff8a..143a01e234 100644
--- a/lib/codegen/src/legalizer/heap.rs
+++ b/lib/codegen/src/legalizer/heap.rs
@@ -65,8 +65,8 @@ fn dynamic_addr(
             .ins()
             .icmp(IntCC::UnsignedGreaterThanOrEqual, offset, bound);
     } else if access_size <= min_size {
-        // We know that bound >= min_size, so here we can compare `offset > bound - access_size` without
-        // wrapping.
+        // We know that bound >= min_size, so here we can compare `offset > bound - access_size`
+        // without wrapping.
         let adj_bound = pos.ins().iadd_imm(bound, -access_size);
         oob = pos
             .ins()
diff --git a/lib/codegen/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs
index c7db253767..053d7f7dee 100644
--- a/lib/codegen/src/regalloc/coalescing.rs
+++ b/lib/codegen/src/regalloc/coalescing.rs
@@ -876,8 +876,8 @@ struct VirtualCopies {
 
     // Filter for the currently active node iterator.
     //
-    // An ebb => (set_id, num) entry means that branches to `ebb` are active in `set_id` with branch
-    // argument number `num`.
+    // An ebb => (set_id, num) entry means that branches to `ebb` are active in `set_id` with
+    // branch argument number `num`.
     filter: FxHashMap,
 }
 
diff --git a/lib/codegen/src/regalloc/liveness.rs b/lib/codegen/src/regalloc/liveness.rs
index 6e71345b13..01f72d9adb 100644
--- a/lib/codegen/src/regalloc/liveness.rs
+++ b/lib/codegen/src/regalloc/liveness.rs
@@ -97,8 +97,8 @@
 //!
 //! Cranelift uses a very similar data structures and algorithms to LLVM, with the important
 //! difference that live ranges are computed per SSA value instead of per virtual register, and the
-//! uses in Cranelift IR refers to SSA values instead of virtual registers. This means that Cranelift
-//! can skip the last step of reconstructing SSA form for the virtual register uses.
+//! uses in Cranelift IR refers to SSA values instead of virtual registers. This means that
+//! Cranelift can skip the last step of reconstructing SSA form for the virtual register uses.
 //!
 //! ## Fast Liveness Checking for SSA-Form Programs
 //!
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index 07a79f4357..aec18899cb 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -66,11 +66,11 @@
 //! extern crate cranelift_frontend;
 //!
 //! use cranelift_codegen::entity::EntityRef;
-//! use cranelift_codegen::ir::{ExternalName, Function, Signature, AbiParam, InstBuilder};
 //! use cranelift_codegen::ir::types::*;
+//! use cranelift_codegen::ir::{AbiParam, ExternalName, Function, InstBuilder, Signature};
 //! use cranelift_codegen::settings::{self, CallConv};
-//! use cranelift_frontend::{FunctionBuilderContext, FunctionBuilder, Variable};
 //! use cranelift_codegen::verifier::verify_function;
+//! use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
 //!
 //! fn main() {
 //!     let mut sig = Signature::new(CallConv::SystemV);
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 7959d0688a..ff155fc195 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -92,8 +92,8 @@ struct Context<'a> {
 
     /// Reference to the unique_isa for things like parsing target-specific instruction encoding
     /// information. This is only `Some` if exactly one set of `isa` directives were found in the
-    /// prologue (it is valid to have directives for multiple different targets, but in that case we
-    /// couldn't know which target the provided encodings are intended for)
+    /// prologue (it is valid to have directives for multiple different targets, but in that case
+    /// we couldn't know which target the provided encodings are intended for)
     unique_isa: Option<&'a TargetIsa>,
 }
 
diff --git a/lib/serde/README.md b/lib/serde/README.md
index 89b54cca72..081573e413 100644
--- a/lib/serde/README.md
+++ b/lib/serde/README.md
@@ -1,19 +1,21 @@
 This crate performs serialization of the [Cranelift](https://crates.io/crates/cranelift) IR.
 
-This crate is structured as an optional ability to serialize and deserialize cranelift IR into JSON format.
+This crate is structured as an optional ability to serialize and deserialize cranelift IR into JSON
+format.
 
 Status
 ------
 
 Cranelift IR can be serialized into JSON.
 
-Deserialize is a work in progress, as it currently deserializes into the serializable data structure that can be utilized by serde instead of the actual Cranelift IR data structure. 
+Deserialize is a work in progress, as it currently deserializes into the serializable data structure
+that can be utilized by serde instead of the actual Cranelift IR data structure.
 
 
 Building and Using Cranelift Serde
 ----------------------------------
 
-clif-json usage: 
+clif-json usage:
 
     clif-json serialize [-p] 
     clif-json deserialize 
diff --git a/lib/serde/src/serde_clif_json.rs b/lib/serde/src/serde_clif_json.rs
index 2cc6c006c8..c113f8622b 100644
--- a/lib/serde/src/serde_clif_json.rs
+++ b/lib/serde/src/serde_clif_json.rs
@@ -721,7 +721,8 @@ pub struct SerDataFlowGraph {
     ebbs: Vec,
 }
 
-/// Serialize all parts of the Cranelift Ebb data structure, this includes name, parameters, and instructions.
+/// Serialize all parts of the Cranelift Ebb data structure, this includes name, parameters, and
+/// instructions.
 pub fn populate_ebbs(func: &Function) -> Vec {
     let mut ebb_vec: Vec = Vec::new();
     for ebb in func.layout.ebbs() {
@@ -785,7 +786,8 @@ pub struct SerFunction {
 }
 
 impl SerFunction {
-    /// Creates serializable global values, as well as the functions signature, name, and data flow graph.
+    /// Creates serializable global values, as well as the functions signature, name, and data flow
+    /// graph.
     fn create_new(func: &Function) -> Self {
         let mut global_vec: Vec = Vec::new();
         for (glob_name, _) in func.global_values.iter() {
@@ -804,8 +806,8 @@ impl SerFunction {
     }
 }
 
-/// Must have SerObj for deserialization, contains all of the functions from inside the file to be serialized.
-/// Files have one SerObj each, with all SerFunctions contained inside that SerObj.
+/// Must have SerObj for deserialization, contains all of the functions from inside the file to be
+/// serialized. Files have one SerObj each, with all SerFunctions contained inside that SerObj.
 #[derive(Serialize, Deserialize, Debug)]
 pub struct SerObj {
     pub functions: Vec,
diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs
index 9b32fe3bbd..88374be17f 100644
--- a/lib/wasm/src/state.rs
+++ b/lib/wasm/src/state.rs
@@ -277,8 +277,8 @@ impl TranslationState {
 
 /// Methods for handling entity references.
 impl TranslationState {
-    /// Get the `GlobalVariable` reference that should be used to access the global variable `index`.
-    /// Create the reference if necessary.
+    /// Get the `GlobalVariable` reference that should be used to access the global variable
+    /// `index`. Create the reference if necessary.
     /// Also return the WebAssembly type of the global.
     pub fn get_global(
         &mut self,

From fb1ac22c2105b331ca202d15fb79702375328af7 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 16 Aug 2018 10:36:06 -0700
Subject: [PATCH 2010/3084] Add `fast_finish = true` to .travis.yml.

This allows us to get the results for stable builds as soon as they're
ready.
---
 .travis.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index 95a4e0e711..a62423fb7a 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -13,6 +13,9 @@ matrix:
         # fail, so we don't allow them to hold up people using stable.
         - rust: beta
         - rust: nightly
+    # Similarly, we don't need to hold up people using stable while we wait
+    # for the results which may fail.
+    fast_finish: true
 dist: trusty
 sudo: false
 addons:

From ae1a17f6b3775c6fdbc8c68be91638976193fc6f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 16 Aug 2018 18:14:58 -0700
Subject: [PATCH 2011/3084] [SimpleJIT] Fix allocation of readonly memory.

Fixes #457.
---
 lib/simplejit/src/backend.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index c867c0d825..043e765be4 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -207,7 +207,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         let size = init.size();
         let storage = match writable {
             Writability::Readonly => self
-                .writable_memory
+                .readonly_memory
                 .allocate(size)
                 .expect("TODO: handle OOM etc."),
             Writability::Writable => self

From ad170c7412759a6e2c5a9e7cd473887539ad5586 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 17 Aug 2018 12:04:01 -0700
Subject: [PATCH 2012/3084] Update to log 0.4.4.

---
 lib/codegen/Cargo.toml   | 2 +-
 lib/filetests/Cargo.toml | 2 +-
 lib/module/Cargo.toml    | 2 +-
 lib/umbrella/src/lib.rs  | 4 ++--
 lib/wasm/Cargo.toml      | 2 +-
 5 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 59f9cf1737..b8333e0bf8 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -17,7 +17,7 @@ failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 target-lexicon = { version = "0.0.3", default-features = false }
-log = { version = "0.4.3", default-features = false, features = ["release_max_level_warn"] }
+log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
 # machine code. Integration tests that need external dependencies can be
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 667ed21418..d74e5efa0a 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -14,4 +14,4 @@ cranelift-reader = { path = "../reader", version = "0.19.0" }
 file-per-thread-logger = "0.1.1"
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
-log = "0.4.3"
+log = "0.4.4"
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 4d57a6b2af..33e04dbcd8 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -13,7 +13,7 @@ cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features
 cranelift-entity = { path = "../entity", version = "0.19.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
-log = { version = "0.4.3", default-features = false, features = ["release_max_level_warn"] }
+log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
 
 [features]
 default = ["std"]
diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs
index 4faeabdd8a..d56bdd6f71 100644
--- a/lib/umbrella/src/lib.rs
+++ b/lib/umbrella/src/lib.rs
@@ -25,8 +25,8 @@ pub mod prelude {
     pub use codegen::ir::immediates::{Ieee32, Ieee64, Imm64};
     pub use codegen::ir::types;
     pub use codegen::ir::{
-        AbiParam, Ebb, ExtFuncData, GlobalValueData, InstBuilder, JumpTableData, MemFlags,
-        Signature, StackSlotData, StackSlotKind, TrapCode, Type, Value,
+        AbiParam, Ebb, ExtFuncData, ExternalName, GlobalValueData, InstBuilder, JumpTableData,
+        MemFlags, Signature, StackSlotData, StackSlotKind, TrapCode, Type, Value,
     };
     pub use codegen::isa;
     pub use codegen::settings::{self, CallConv, Configurable};
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 369220c4a7..df35edb25f 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -17,7 +17,7 @@ hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
-log = { version = "0.4.3", default-features = false, features = ["release_max_level_warn"] }
+log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
 
 [dev-dependencies]
 wabt = "0.4"

From 37272f5ceba5305a1070485a829ab4b560516741 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 17 Aug 2018 12:13:34 -0700
Subject: [PATCH 2013/3084] Removed "Variable" parametricity for SSABuilder and
 related code too.

---
 lib/frontend/src/frontend.rs |  2 +-
 lib/frontend/src/ssa.rs      | 46 +++++++++++++++---------------------
 2 files changed, 20 insertions(+), 28 deletions(-)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 31b27b4f8b..0a6732a89b 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -23,7 +23,7 @@ use variable::Variable;
 /// implement `EntityRef`. For frontends that don't have an obvious type to
 /// use here, `variable::Variable` can be used.
 pub struct FunctionBuilderContext {
-    ssa: SSABuilder,
+    ssa: SSABuilder,
     ebbs: EntityMap,
     types: EntityMap,
 }
diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs
index 5e6c4fbf9b..b15eea172c 100644
--- a/lib/frontend/src/ssa.rs
+++ b/lib/frontend/src/ssa.rs
@@ -16,6 +16,7 @@ use cranelift_codegen::packed_option::ReservedValue;
 use std::mem;
 use std::u32;
 use std::vec::Vec;
+use Variable;
 
 /// Structure containing the data relevant the construction of SSA for a given function.
 ///
@@ -32,17 +33,14 @@ use std::vec::Vec;
 /// A basic block is said _filled_ if all the instruction that it contains have been translated,
 /// and it is said _sealed_ if all of its predecessors have been declared. Only filled predecessors
 /// can be declared.
-pub struct SSABuilder
-where
-    Variable: EntityRef,
-{
+pub struct SSABuilder {
     // Records for every variable and for every relevant block, the last definition of
     // the variable in the block.
     // TODO: Consider a sparse representation rather than EntityMap-of-EntityMap.
     variables: EntityMap>>,
     // Records the position of the basic blocks and the list of values used but not defined in the
     // block.
-    blocks: PrimaryMap>,
+    blocks: PrimaryMap,
     // Records the basic blocks at the beginning of the `Ebb`s.
     ebb_headers: EntityMap>,
 
@@ -79,15 +77,15 @@ impl SideEffects {
 }
 
 /// Describes the current position of a basic block in the control flow graph.
-enum BlockData {
+enum BlockData {
     /// A block at the top of an `Ebb`.
-    EbbHeader(EbbHeaderBlockData),
+    EbbHeader(EbbHeaderBlockData),
     /// A block inside an `Ebb` with an unique other block as its predecessor.
     /// The block is implicitly sealed at creation.
     EbbBody { predecessor: Block },
 }
 
-impl BlockData {
+impl BlockData {
     fn add_predecessor(&mut self, pred: Block, inst: Inst) {
         match *self {
             BlockData::EbbBody { .. } => panic!("you can't add a predecessor to a body block"),
@@ -125,7 +123,7 @@ impl PredBlock {
     }
 }
 
-struct EbbHeaderBlockData {
+struct EbbHeaderBlockData {
     // The predecessors of the Ebb header block, with the block and branch instruction.
     predecessors: Vec,
     // A ebb header block is sealed if all of its predecessors have been declared.
@@ -156,10 +154,7 @@ impl ReservedValue for Block {
     }
 }
 
-impl SSABuilder
-where
-    Variable: EntityRef,
-{
+impl SSABuilder {
     /// Allocate a new blank SSA builder struct. Use the API function to interact with the struct.
     pub fn new() -> Self {
         Self {
@@ -266,10 +261,7 @@ fn emit_zero(ty: Type, mut cur: FuncCursor) -> Value {
 /// as well as modify the jump instruction and `Ebb` headers parameters to account for the SSA
 /// Phi functions.
 ///
-impl SSABuilder
-where
-    Variable: EntityRef,
-{
+impl SSABuilder {
     /// Declares a new definition of a variable in a given basic block.
     /// The SSA value is passed as an argument because it should be created with
     /// `ir::DataFlowGraph::append_result`.
@@ -751,7 +743,7 @@ mod tests {
     #[test]
     fn simple_block() {
         let mut func = Function::new();
-        let mut ssa: SSABuilder = SSABuilder::new();
+        let mut ssa = SSABuilder::new();
         let ebb0 = func.dfg.make_ebb();
         // Here is the pseudo-program we want to translate:
         // x = 1;
@@ -798,7 +790,7 @@ mod tests {
     #[test]
     fn sequence_of_blocks() {
         let mut func = Function::new();
-        let mut ssa: SSABuilder = SSABuilder::new();
+        let mut ssa = SSABuilder::new();
         let ebb0 = func.dfg.make_ebb();
         let ebb1 = func.dfg.make_ebb();
         // Here is the pseudo-program we want to translate:
@@ -879,7 +871,7 @@ mod tests {
     #[test]
     fn program_with_loop() {
         let mut func = Function::new();
-        let mut ssa: SSABuilder = SSABuilder::new();
+        let mut ssa = SSABuilder::new();
         let ebb0 = func.dfg.make_ebb();
         let ebb1 = func.dfg.make_ebb();
         let ebb2 = func.dfg.make_ebb();
@@ -991,7 +983,7 @@ mod tests {
     fn br_table_with_args() {
         // This tests the on-demand splitting of critical edges for br_table with jump arguments
         let mut func = Function::new();
-        let mut ssa: SSABuilder = SSABuilder::new();
+        let mut ssa = SSABuilder::new();
         let ebb0 = func.dfg.make_ebb();
         let ebb1 = func.dfg.make_ebb();
         // Here is the pseudo-program we want to translate:
@@ -1061,7 +1053,7 @@ mod tests {
     #[test]
     fn undef_values_reordering() {
         let mut func = Function::new();
-        let mut ssa: SSABuilder = SSABuilder::new();
+        let mut ssa = SSABuilder::new();
         let ebb0 = func.dfg.make_ebb();
         let ebb1 = func.dfg.make_ebb();
         // Here is the pseudo-program we want to translate:
@@ -1137,7 +1129,7 @@ mod tests {
     fn undef() {
         // Use vars of various types which have not been defined.
         let mut func = Function::new();
-        let mut ssa: SSABuilder = SSABuilder::new();
+        let mut ssa = SSABuilder::new();
         let ebb0 = func.dfg.make_ebb();
         let block = ssa.declare_ebb_header_block(ebb0);
         ssa.seal_ebb_header_block(ebb0, &mut func);
@@ -1159,7 +1151,7 @@ mod tests {
         // Use a var which has not been defined. The search should hit the
         // top of the entry block, and then fall back to inserting an iconst.
         let mut func = Function::new();
-        let mut ssa: SSABuilder = SSABuilder::new();
+        let mut ssa = SSABuilder::new();
         let ebb0 = func.dfg.make_ebb();
         let block = ssa.declare_ebb_header_block(ebb0);
         ssa.seal_ebb_header_block(ebb0, &mut func);
@@ -1179,7 +1171,7 @@ mod tests {
         // until afterward. Before sealing, the SSA builder should insert an
         // ebb param; after sealing, it should be removed.
         let mut func = Function::new();
-        let mut ssa: SSABuilder = SSABuilder::new();
+        let mut ssa = SSABuilder::new();
         let ebb0 = func.dfg.make_ebb();
         let block = ssa.declare_ebb_header_block(ebb0);
         let x_var = Variable::new(0);
@@ -1197,7 +1189,7 @@ mod tests {
     #[test]
     fn unreachable_use() {
         let mut func = Function::new();
-        let mut ssa: SSABuilder = SSABuilder::new();
+        let mut ssa = SSABuilder::new();
         let ebb0 = func.dfg.make_ebb();
         let ebb1 = func.dfg.make_ebb();
         // Here is the pseudo-program we want to translate:
@@ -1240,7 +1232,7 @@ mod tests {
     #[test]
     fn unreachable_use_with_multiple_preds() {
         let mut func = Function::new();
-        let mut ssa: SSABuilder = SSABuilder::new();
+        let mut ssa = SSABuilder::new();
         let ebb0 = func.dfg.make_ebb();
         let ebb1 = func.dfg.make_ebb();
         let ebb2 = func.dfg.make_ebb();

From 30d09cf6b06cc8de9e4e37b4cd07699a78187902 Mon Sep 17 00:00:00 2001
From: ms2300 
Date: Fri, 17 Aug 2018 11:13:06 -0600
Subject: [PATCH 2014/3084] Fixing a couple clippy warnings : #392

---
 lib/filetests/src/test_verifier.rs |  6 +++---
 lib/serde/src/clif-json.rs         | 13 ++++++-------
 lib/serde/src/serde_clif_json.rs   |  8 ++++----
 lib/simplejit/src/backend.rs       |  4 ++--
 4 files changed, 15 insertions(+), 16 deletions(-)

diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs
index 9c356d6733..0f5291c70c 100644
--- a/lib/filetests/src/test_verifier.rs
+++ b/lib/filetests/src/test_verifier.rs
@@ -51,10 +51,10 @@ impl SubTest for TestVerifier {
         }
 
         match verify_function(func, context.flags_or_isa()) {
-            Ok(()) if expected.len() == 0 => Ok(()),
+            Ok(()) if expected.is_empty() => Ok(()),
             Ok(()) => Err(format!("passed, but expected errors: {:?}", expected)),
 
-            Err(ref errors) if expected.len() == 0 => {
+            Err(ref errors) if expected.is_empty() => {
                 Err(format!("expected no error, but got:\n{}", errors))
             }
 
@@ -78,7 +78,7 @@ impl SubTest for TestVerifier {
                     }
                 }
 
-                if msg.len() == 0 {
+                if msg.is_empty() {
                     Ok(())
                 } else {
                     Err(msg)
diff --git a/lib/serde/src/clif-json.rs b/lib/serde/src/clif-json.rs
index 7cc674c2b6..228b13703d 100644
--- a/lib/serde/src/clif-json.rs
+++ b/lib/serde/src/clif-json.rs
@@ -33,16 +33,15 @@ fn call_ser(file: &str, pretty: bool) -> Result<(), String> {
     match ret_of_parse {
         Ok(funcs) => {
             let ser_funcs = serde_clif_json::SerObj::new(&funcs);
-            let ser_str: String;
-            if pretty {
-                ser_str = serde_json::to_string_pretty(&ser_funcs).unwrap();
+            let ser_str = if pretty {
+                serde_json::to_string_pretty(&ser_funcs).unwrap()
             } else {
-                ser_str = serde_json::to_string(&ser_funcs).unwrap();
-            }
+                serde_json::to_string(&ser_funcs).unwrap()
+            };
             println!("{}", ser_str);
             Ok(())
         }
-        Err(_pe) => Err(format!("There was a parsing error")),
+        Err(_pe) => Err("There was a parsing error".to_string()),
     }
 }
 
@@ -99,7 +98,7 @@ fn main() {
                 File::open(m.value_of("FILE").unwrap()).expect("Unable to open the file");
             call_de(&file)
         }
-        _ => Err(format!("Invalid subcommand.")),
+        _ => Err("Invalid subcommand.".to_string()),
     };
 
     if let Err(mut msg) = res_serde {
diff --git a/lib/serde/src/serde_clif_json.rs b/lib/serde/src/serde_clif_json.rs
index c113f8622b..52fdc6fca1 100644
--- a/lib/serde/src/serde_clif_json.rs
+++ b/lib/serde/src/serde_clif_json.rs
@@ -250,7 +250,7 @@ pub fn get_inst_data(inst_index: Inst, func: &Function) -> SerInstData {
         },
         InstructionData::UnaryBool { opcode, imm } => SerInstData::UnaryBool {
             opcode: opcode.to_string(),
-            imm: imm,
+            imm,
         },
         InstructionData::UnaryGlobalValue {
             opcode,
@@ -759,10 +759,10 @@ impl SerSignature {
     fn create_new(sig: &Signature) -> Self {
         let mut params_vec: Vec = Vec::new();
         let mut returns_vec: Vec = Vec::new();
-        for param in sig.params.iter() {
+        for param in &sig.params {
             params_vec.push(param.to_string());
         }
-        for ret in sig.returns.iter() {
+        for ret in &sig.returns {
             returns_vec.push(ret.to_string());
         }
         Self {
@@ -818,7 +818,7 @@ impl SerObj {
         Self { functions: funcs }
     }
 
-    pub fn new(funcs: &Vec) -> Self {
+    pub fn new(funcs: &[Function]) -> Self {
         let mut func_vec: Vec = Vec::new();
         for func in funcs {
             let mut ser_func: SerFunction = SerFunction::new(&func);
diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index 043e765be4..15c3f2b2c1 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -60,7 +60,7 @@ impl SimpleJITBuilder {
     /// back to a platform-specific search (this typically involves searching
     /// the current process for public symbols, followed by searching the
     /// platform's C runtime).
-    pub fn symbol<'a, K>(&'a mut self, name: K, ptr: *const u8) -> &'a mut Self
+    pub fn symbol(&mut self, name: K, ptr: *const u8) -> &Self
     where
         K: Into,
     {
@@ -71,7 +71,7 @@ impl SimpleJITBuilder {
     /// Define multiple symbols in the internal symbol table.
     ///
     /// Using this is equivalent to calling `symbol` on each element.
-    pub fn symbols<'a, It, K>(&'a mut self, symbols: It) -> &'a mut Self
+    pub fn symbols(&mut self, symbols: It) -> &Self
     where
         It: IntoIterator,
         K: Into,

From 77eb38c41fe5f3fa14f85ed5d682187c08066be8 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 16 Aug 2018 18:31:05 -0700
Subject: [PATCH 2015/3084] [Module] Remove DataDescription's writable field.

It was redundant, as data object declarations also have a writable
field, so just use that, avoiding the need for users to declare the
same thing twice.

Fixes #456.
---
 lib/faerie/src/backend.rs      |  2 +-
 lib/module/src/backend.rs      |  1 +
 lib/module/src/data_context.rs | 29 +++++------------------------
 lib/module/src/lib.rs          |  2 +-
 lib/module/src/module.rs       |  1 +
 lib/simplejit/src/backend.rs   | 16 +++++++---------
 6 files changed, 16 insertions(+), 35 deletions(-)

diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index 4f30406a09..b1d5368bad 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -193,11 +193,11 @@ impl Backend for FaerieBackend {
     fn define_data(
         &mut self,
         name: &str,
+        _writable: bool,
         data_ctx: &DataContext,
         namespace: &ModuleNamespace,
     ) -> ModuleResult {
         let &DataDescription {
-            writable: _writable,
             ref init,
             ref function_decls,
             ref data_decls,
diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs
index bc151efce9..44a2aa95af 100644
--- a/lib/module/src/backend.rs
+++ b/lib/module/src/backend.rs
@@ -65,6 +65,7 @@ where
     fn define_data(
         &mut self,
         name: &str,
+        writable: bool,
         data_ctx: &DataContext,
         namespace: &ModuleNamespace,
     ) -> ModuleResult;
diff --git a/lib/module/src/data_context.rs b/lib/module/src/data_context.rs
index e29389517b..4a7fdd7a6e 100644
--- a/lib/module/src/data_context.rs
+++ b/lib/module/src/data_context.rs
@@ -34,19 +34,8 @@ impl Init {
     }
 }
 
-/// A flag specifying whether data is readonly or may be written to.
-#[derive(Copy, Clone, PartialEq, Eq, Debug)]
-pub enum Writability {
-    /// Data is readonly, meaning writes to it will trap.
-    Readonly,
-    /// Data is writable.
-    Writable,
-}
-
 /// A description of a data object.
 pub struct DataDescription {
-    /// Whether the data readonly or writable.
-    pub writable: Writability,
     /// How the data should be initialized.
     pub init: Init,
     /// External function declarations.
@@ -69,7 +58,6 @@ impl DataContext {
     pub fn new() -> Self {
         Self {
             description: DataDescription {
-                writable: Writability::Readonly,
                 init: Init::Uninitialized,
                 function_decls: PrimaryMap::new(),
                 data_decls: PrimaryMap::new(),
@@ -81,7 +69,6 @@ impl DataContext {
 
     /// Clear all data structures in this context.
     pub fn clear(&mut self) {
-        self.description.writable = Writability::Readonly;
         self.description.init = Init::Uninitialized;
         self.description.function_decls.clear();
         self.description.data_decls.clear();
@@ -90,18 +77,16 @@ impl DataContext {
     }
 
     /// Define a zero-initialized object with the given size.
-    pub fn define_zeroinit(&mut self, size: usize, writable: Writability) {
+    pub fn define_zeroinit(&mut self, size: usize) {
         debug_assert_eq!(self.description.init, Init::Uninitialized);
-        self.description.writable = writable;
         self.description.init = Init::Zeros { size };
     }
 
     /// Define an object initialized with the given contents.
     ///
     /// TODO: Can we avoid a Box here?
-    pub fn define(&mut self, contents: Box<[u8]>, writable: Writability) {
+    pub fn define(&mut self, contents: Box<[u8]>) {
         debug_assert_eq!(self.description.init, Init::Uninitialized);
-        self.description.writable = writable;
         self.description.init = Init::Bytes { contents };
     }
 
@@ -148,14 +133,13 @@ impl DataContext {
 #[cfg(test)]
 mod tests {
     use cranelift_codegen::ir;
-    use {DataContext, Init, Writability};
+    use {DataContext, Init};
 
     #[test]
     fn basic_data_context() {
         let mut data_ctx = DataContext::new();
         {
             let description = &data_ctx.description;
-            assert_eq!(description.writable, Writability::Readonly);
             assert_eq!(description.init, Init::Uninitialized);
             assert!(description.function_decls.is_empty());
             assert!(description.data_decls.is_empty());
@@ -163,7 +147,7 @@ mod tests {
             assert!(description.data_relocs.is_empty());
         }
 
-        data_ctx.define_zeroinit(256, Writability::Writable);
+        data_ctx.define_zeroinit(256);
 
         let _func_a = data_ctx.import_function(ir::ExternalName::user(0, 0));
         let func_b = data_ctx.import_function(ir::ExternalName::user(0, 1));
@@ -177,7 +161,6 @@ mod tests {
 
         {
             let description = data_ctx.description();
-            assert_eq!(description.writable, Writability::Writable);
             assert_eq!(description.init, Init::Zeros { size: 256 });
             assert_eq!(description.function_decls.len(), 3);
             assert_eq!(description.data_decls.len(), 2);
@@ -188,7 +171,6 @@ mod tests {
         data_ctx.clear();
         {
             let description = &data_ctx.description;
-            assert_eq!(description.writable, Writability::Readonly);
             assert_eq!(description.init, Init::Uninitialized);
             assert!(description.function_decls.is_empty());
             assert!(description.data_decls.is_empty());
@@ -198,10 +180,9 @@ mod tests {
 
         let contents = vec![33, 34, 35, 36];
         let contents_clone = contents.clone();
-        data_ctx.define(contents.into_boxed_slice(), Writability::Readonly);
+        data_ctx.define(contents.into_boxed_slice());
         {
             let description = data_ctx.description();
-            assert_eq!(description.writable, Writability::Readonly);
             assert_eq!(
                 description.init,
                 Init::Bytes {
diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs
index 45e3bd683e..479288df57 100644
--- a/lib/module/src/lib.rs
+++ b/lib/module/src/lib.rs
@@ -33,7 +33,7 @@ mod data_context;
 mod module;
 
 pub use backend::Backend;
-pub use data_context::{DataContext, DataDescription, Init, Writability};
+pub use data_context::{DataContext, DataDescription, Init};
 pub use module::{
     DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleError, ModuleNamespace, ModuleResult,
 };
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 223f748107..185057b946 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -507,6 +507,7 @@ where
             }
             Some(self.backend.define_data(
                 &info.decl.name,
+                info.decl.writable,
                 data_ctx,
                 &ModuleNamespace:: {
                     contents: &self.contents,
diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index 15c3f2b2c1..2a23b93f0c 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -5,7 +5,6 @@ use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::{self, ir, settings};
 use cranelift_module::{
     Backend, DataContext, DataDescription, Init, Linkage, ModuleNamespace, ModuleResult,
-    Writability,
 };
 use cranelift_native;
 use libc;
@@ -192,11 +191,11 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
     fn define_data(
         &mut self,
         _name: &str,
+        writable: bool,
         data: &DataContext,
         _namespace: &ModuleNamespace,
     ) -> ModuleResult {
         let &DataDescription {
-            writable,
             ref init,
             ref function_decls,
             ref data_decls,
@@ -205,15 +204,14 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         } = data.description();
 
         let size = init.size();
-        let storage = match writable {
-            Writability::Readonly => self
-                .readonly_memory
+        let storage = if writable {
+            self.writable_memory
                 .allocate(size)
-                .expect("TODO: handle OOM etc."),
-            Writability::Writable => self
-                .writable_memory
+                .expect("TODO: handle OOM etc.")
+        } else {
+            self.readonly_memory
                 .allocate(size)
-                .expect("TODO: handle OOM etc."),
+                .expect("TODO: handle OOM etc.")
         };
 
         match *init {

From 0f93ef5cee35d3fb20550fe554aa24fd04fb861a Mon Sep 17 00:00:00 2001
From: Caroline Cullen 
Date: Wed, 22 Aug 2018 11:25:55 -0700
Subject: [PATCH 2016/3084]  Changing from docopt to clap for the clif-util
 #434 (#463)

* Changing from docopt to clap for the clif-util

* Updates to cargo file.

* Remove filecheck subcommand.
---
 cranelift/Cargo.toml         |   3 +-
 cranelift/src/clif-util.rs   | 300 +++++++++++++++++++++--------------
 cranelift/src/rsfilecheck.rs |  61 -------
 3 files changed, 181 insertions(+), 183 deletions(-)
 delete mode 100644 cranelift/src/rsfilecheck.rs

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index f27fe9c2eb..93f35f0c69 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -27,9 +27,8 @@ cranelift-faerie = { path = "lib/faerie", version = "0.19.0" }
 cranelift-simplejit = { path = "lib/simplejit", version = "0.19.0" }
 cranelift = { path = "lib/umbrella", version = "0.19.0" }
 filecheck = "0.3.0"
-docopt = "1"
+clap = "2.32.0"
 serde = "1.0.8"
-serde_derive = "1.0.8"
 term = "0.5.1"
 capstone = { version = "0.4", optional = true }
 wabt = { version = "0.4", optional = true }
diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 5b8eb2f3c7..2a7ae79d30 100644
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -8,23 +8,20 @@
     )
 )]
 
+extern crate file_per_thread_logger;
 #[macro_use]
 extern crate cfg_if;
-extern crate cranelift_codegen;
-extern crate cranelift_filetests;
-extern crate cranelift_reader;
-extern crate docopt;
-extern crate file_per_thread_logger;
-extern crate filecheck;
-#[macro_use]
-extern crate serde_derive;
 #[cfg(feature = "disas")]
 extern crate capstone;
+extern crate clap;
+extern crate cranelift_codegen;
+extern crate cranelift_entity;
+extern crate cranelift_filetests;
+extern crate cranelift_reader;
 extern crate pretty_env_logger;
 
 cfg_if! {
     if #[cfg(feature = "wasm")] {
-        extern crate cranelift_entity;
         extern crate cranelift_wasm;
         extern crate term;
         extern crate wabt;
@@ -33,137 +30,200 @@ cfg_if! {
 }
 extern crate target_lexicon;
 
+use clap::{App, Arg, SubCommand};
 use cranelift_codegen::dbg::LOG_FILENAME_PREFIX;
-use cranelift_codegen::{timing, VERSION};
-use docopt::Docopt;
+use cranelift_codegen::VERSION;
 use std::io::{self, Write};
+use std::option::Option;
 use std::process;
 
 mod cat;
 mod compile;
 mod print_cfg;
-mod rsfilecheck;
 mod utils;
 
-const USAGE: &str = "
-Cranelift code generator utility
-
-Usage:
-    clif-util test [-vTd] ...
-    clif-util cat [-d] ...
-    clif-util filecheck [-vd] 
-    clif-util print-cfg [-d] ...
-    clif-util compile [-vpTd] [--set ]... [--target ] ...
-    clif-util wasm [-ctvpTsd] [--set ]... [--target ] ...
-    clif-util --help | --version
-
-Options:
-    -v, --verbose   be more verbose
-    -d, --debug     enable debug output on stderr/stdout
-    -T, --time-passes
-                    print pass timing report
-    -t, --just-decode
-                    just decode WebAssembly to Cranelift IR
-    -s, --print-size
-                    prints generated code size
-    -c, --check-translation
-                    just checks the correctness of Cranelift IR translated from WebAssembly
-    -p, --print     print the resulting Cranelift IR
-    -h, --help      print this help message
-    --set=     configure Cranelift settings
-    --target=
-                    specify the Cranelift target
-    --version       print the Cranelift version
-
-";
-
-#[derive(Deserialize, Debug)]
-struct Args {
-    cmd_test: bool,
-    cmd_cat: bool,
-    cmd_filecheck: bool,
-    cmd_print_cfg: bool,
-    cmd_compile: bool,
-    cmd_wasm: bool,
-    arg_file: Vec,
-    flag_just_decode: bool,
-    flag_check_translation: bool,
-    flag_print: bool,
-    flag_verbose: bool,
-    flag_set: Vec,
-    flag_target: String,
-    flag_time_passes: bool,
-    flag_print_size: bool,
-    flag_debug: bool,
-}
-
 /// A command either succeeds or fails with an error message.
 pub type CommandResult = Result<(), String>;
 
-/// Parse the command line arguments and run the requested command.
-fn clif_util() -> CommandResult {
-    // Parse command line arguments.
-    let args: Args = Docopt::new(USAGE)
-        .and_then(|d| {
-            d.help(true)
-                .version(Some(format!("Cranelift {}", VERSION)))
-                .deserialize()
-        })
-        .unwrap_or_else(|e| e.exit());
+fn add_input_file_arg<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("file")
+        .required(true)
+        .multiple(true)
+        .value_name("file")
+        .help("Specify file(s) to be used for test")
+}
 
-    if args.flag_debug {
+fn add_verbose_flag<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("verbose").short("v").help("Be more verbose")
+}
+
+fn add_time_flag<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("time-passes")
+        .short("T")
+        .help("Print pass timing report for test")
+}
+
+fn add_set_flag<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("set")
+        .long("set")
+        .takes_value(true)
+        .multiple(true)
+        .help("Configure Cranelift settings")
+}
+
+fn add_target_flag<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("target")
+        .takes_value(true)
+        .long("target")
+        .help("Specify the Cranelift target")
+}
+
+fn add_print_flag<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("print")
+        .short("p")
+        .help("Print the resulting Cranelift IR")
+}
+
+fn add_debug_flag<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("debug")
+        .short("d")
+        .help("enable debug output on stderr/stdout")
+}
+
+/// Returns a vector of clap value options and changes these options into a vector of strings
+fn get_vec<'a>(argument_vec: Option>) -> Vec {
+    let mut ret_vec: Vec = Vec::new();
+    if let Some(clap_vec) = argument_vec {
+        for val in clap_vec {
+            ret_vec.push(val.to_string());
+        }
+    }
+
+    ret_vec
+}
+
+fn add_wasm_or_compile<'a>(cmd: &str) -> clap::App<'a, 'a> {
+    let about_str = match cmd {
+        "wasm" => "Compiles Cranelift IR into target language",
+        "compile" => "Compiles Cranelift IR into target language",
+        _ => panic!("Invalid command"),
+    };
+
+    SubCommand::with_name(cmd)
+        .about(about_str)
+        .arg(add_verbose_flag())
+        .arg(add_print_flag())
+        .arg(add_time_flag())
+        .arg(add_set_flag())
+        .arg(add_target_flag())
+        .arg(add_input_file_arg())
+        .arg(add_debug_flag())
+}
+
+fn handle_debug_flag(debug: bool) {
+    if debug {
         pretty_env_logger::init();
     } else {
         file_per_thread_logger::initialize(LOG_FILENAME_PREFIX);
     }
-
-    // Find the sub-command to execute.
-    let result = if args.cmd_test {
-        cranelift_filetests::run(args.flag_verbose, &args.arg_file).map(|_time| ())
-    } else if args.cmd_cat {
-        cat::run(&args.arg_file)
-    } else if args.cmd_filecheck {
-        rsfilecheck::run(&args.arg_file, args.flag_verbose)
-    } else if args.cmd_print_cfg {
-        print_cfg::run(&args.arg_file)
-    } else if args.cmd_compile {
-        compile::run(
-            args.arg_file,
-            args.flag_print,
-            &args.flag_set,
-            &args.flag_target,
-        )
-    } else if args.cmd_wasm {
-        #[cfg(feature = "wasm")]
-        let result = wasm::run(
-            args.arg_file,
-            args.flag_verbose,
-            args.flag_just_decode,
-            args.flag_check_translation,
-            args.flag_print,
-            &args.flag_set,
-            &args.flag_target,
-            args.flag_print_size,
-        );
-
-        #[cfg(not(feature = "wasm"))]
-        let result = Err("Error: clif-util was compiled without wasm support.".to_owned());
-
-        result
-    } else {
-        // Debugging / shouldn't happen with proper command line handling above.
-        Err(format!("Unhandled args: {:?}", args))
-    };
-
-    if args.flag_time_passes {
-        print!("{}", timing::take_current());
-    }
-
-    result
 }
 
 fn main() {
-    if let Err(mut msg) = clif_util() {
+    let app_cmds = App::new("Cranelift code generator utility")
+        .version(VERSION)
+        .subcommand(
+            SubCommand::with_name("test")
+                .about("Run Cranelift tests")
+                .arg(add_verbose_flag())
+                .arg(add_time_flag())
+                .arg(add_input_file_arg())
+                .arg(add_debug_flag()),
+        )
+        .subcommand(
+            SubCommand::with_name("cat")
+                .about("Outputs .clif file")
+                .arg(add_input_file_arg())
+                .arg(add_debug_flag()),
+        )
+        .subcommand(
+            SubCommand::with_name("print-cfg")
+                .about("Prints out cfg in dot format")
+                .arg(add_input_file_arg())
+                .arg(add_debug_flag()),
+        )
+        .subcommand(
+            add_wasm_or_compile("compile")
+                .arg(
+                    Arg::with_name("just-decode")
+                        .short("t")
+                        .help("Just decode WebAssembly to Cranelift IR"),
+                )
+                .arg(Arg::with_name("check-translation").short("c").help(
+                    "Just checks the correctness of Cranelift IR translated from WebAssembly",
+                )),
+        )
+        .subcommand(add_wasm_or_compile("wasm"));
+
+    let res_util = match app_cmds.get_matches().subcommand() {
+        ("cat", Some(rest_cmd)) => {
+            handle_debug_flag(rest_cmd.is_present("debug"));
+            cat::run(&get_vec(rest_cmd.values_of("file")))
+        }
+        ("test", Some(rest_cmd)) => {
+            handle_debug_flag(rest_cmd.is_present("debug"));
+            cranelift_filetests::run(
+                rest_cmd.is_present("time-passes"),
+                &get_vec(rest_cmd.values_of("file")),
+            ).map(|_time| ())
+        }
+        ("print-cfg", Some(rest_cmd)) => {
+            handle_debug_flag(rest_cmd.is_present("debug"));
+            print_cfg::run(&get_vec(rest_cmd.values_of("file")))
+        }
+        ("compile", Some(rest_cmd)) => {
+            handle_debug_flag(rest_cmd.is_present("debug"));
+
+            let mut target_val: &str = "";
+            if let Some(clap_target) = rest_cmd.value_of("target") {
+                target_val = clap_target;
+            }
+
+            compile::run(
+                get_vec(rest_cmd.values_of("file")),
+                rest_cmd.is_present("print"),
+                &get_vec(rest_cmd.values_of("set")),
+                target_val,
+            )
+        }
+        ("wasm", Some(rest_cmd)) => {
+            handle_debug_flag(rest_cmd.is_present("debug"));
+
+            let mut target_val: &str = "";
+            if let Some(clap_target) = rest_cmd.value_of("target") {
+                target_val = clap_target;
+            }
+
+            #[cfg(feature = "wasm")]
+            let result = wasm::run(
+                get_vec(rest_cmd.values_of("file")),
+                rest_cmd.is_present("verbose"),
+                rest_cmd.is_present("just-decode"),
+                rest_cmd.is_present("check-translation"),
+                rest_cmd.is_present("print"),
+                &get_vec(rest_cmd.values_of("set")),
+                target_val,
+                rest_cmd.is_present("print-size"),
+            );
+
+            #[cfg(not(feature = "wasm"))]
+            let result = Err("Error: clif-util was compiled without wasm support.".to_owned());
+
+            result
+        }
+        _ => Err(format!("Invalid subcommand.")),
+    };
+
+    if let Err(mut msg) = res_util {
         if !msg.ends_with('\n') {
             msg.push('\n');
         }
diff --git a/cranelift/src/rsfilecheck.rs b/cranelift/src/rsfilecheck.rs
deleted file mode 100644
index 48e25b3f12..0000000000
--- a/cranelift/src/rsfilecheck.rs
+++ /dev/null
@@ -1,61 +0,0 @@
-//! The `filecheck` sub-command.
-//!
-//! This file is named to avoid a name collision with the filecheck crate.
-
-use filecheck::{Checker, CheckerBuilder, NO_VARIABLES};
-use std::io::{self, Read};
-use utils::read_to_string;
-use CommandResult;
-
-pub fn run(files: &[String], verbose: bool) -> CommandResult {
-    if files.is_empty() {
-        return Err("No check files".to_string());
-    }
-    let checker = read_checkfile(&files[0])?;
-    if checker.is_empty() {
-        return Err(format!("{}: no filecheck directives found", files[0]));
-    }
-
-    // Print out the directives under --verbose.
-    if verbose {
-        println!("{}", checker);
-    }
-
-    let mut buffer = String::new();
-    io::stdin()
-        .read_to_string(&mut buffer)
-        .map_err(|e| format!("stdin: {}", e))?;
-
-    if verbose {
-        let (success, explain) = checker
-            .explain(&buffer, NO_VARIABLES)
-            .map_err(|e| e.to_string())?;
-        print!("{}", explain);
-        if success {
-            println!("OK");
-            Ok(())
-        } else {
-            Err("Check failed".to_string())
-        }
-    } else if checker
-        .check(&buffer, NO_VARIABLES)
-        .map_err(|e| e.to_string())?
-    {
-        Ok(())
-    } else {
-        let (_, explain) = checker
-            .explain(&buffer, NO_VARIABLES)
-            .map_err(|e| e.to_string())?;
-        print!("{}", explain);
-        Err("Check failed".to_string())
-    }
-}
-
-fn read_checkfile(filename: &str) -> Result {
-    let buffer = read_to_string(&filename).map_err(|e| format!("{}: {}", filename, e))?;
-    let mut builder = CheckerBuilder::new();
-    builder
-        .text(&buffer)
-        .map_err(|e| format!("{}: {}", filename, e))?;
-    Ok(builder.finish())
-}

From 18f781e2ab53cc80183f5ce70964f35a70d9ee36 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 20 Aug 2018 16:27:45 -0700
Subject: [PATCH 2017/3084] Update to faerie 0.5.0.

---
 lib/faerie/Cargo.toml     | 4 ++--
 lib/faerie/src/backend.rs | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 3df9d2d84b..320a8fae6e 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -11,9 +11,9 @@ readme = "README.md"
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.19.0" }
 cranelift-module = { path = "../module", version = "0.19.0" }
-faerie = "0.4.4"
+faerie = "0.5.0"
 goblin = "0.0.17"
-failure = "0.1.1"
+failure = "0.1.2"
 target-lexicon = "0.0.3"
 
 [badges]
diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index b1d5368bad..0d27ed490c 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -330,11 +330,11 @@ fn translate_data_linkage(linkage: Linkage, writable: bool) -> faerie::Decl {
         Linkage::Import => faerie::Decl::DataImport,
         Linkage::Local => faerie::Decl::Data {
             global: false,
-            writeable: writable,
+            writable,
         },
         Linkage::Export => faerie::Decl::Data {
             global: true,
-            writeable: writable,
+            writable,
         },
         Linkage::Preemptible => {
             unimplemented!("faerie doesn't support preemptible globals yet");

From 2c9b7fd73a80500d5875b33ca5848699f2121f5a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 20 Aug 2018 16:30:30 -0700
Subject: [PATCH 2018/3084] Add "no-std" category to crates supporting no-std.

And add "wasm" category to cranelift-wasm.
---
 lib/bforest/Cargo.toml   | 1 +
 lib/codegen/Cargo.toml   | 1 +
 lib/entity/Cargo.toml    | 1 +
 lib/frontend/Cargo.toml  | 1 +
 lib/module/Cargo.toml    | 1 +
 lib/native/Cargo.toml    | 1 +
 lib/serde/Cargo.toml     | 1 +
 lib/simplejit/Cargo.toml | 1 +
 lib/umbrella/Cargo.toml  | 1 +
 lib/wasm/Cargo.toml      | 1 +
 10 files changed, 10 insertions(+)

diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index ab244abc80..72f146adb9 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -6,6 +6,7 @@ description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
 repository = "https://github.com/CraneStation/cranelift"
+categories = ["no-std"]
 readme = "README.md"
 keywords = ["btree", "forest", "set", "map"]
 
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index b8333e0bf8..1e55075751 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -6,6 +6,7 @@ description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
 repository = "https://github.com/CraneStation/cranelift"
+categories = ["no-std"]
 readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index e96837669b..6bc4a928bf 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -6,6 +6,7 @@ description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
 repository = "https://github.com/CraneStation/cranelift"
+categories = ["no-std"]
 readme = "README.md"
 keywords = ["entity", "set", "map"]
 
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index d231b017f8..d0b84b6e3a 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -5,6 +5,7 @@ version = "0.19.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
+categories = ["no-std"]
 repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 33e04dbcd8..ef7427f8b9 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -5,6 +5,7 @@ authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
 documentation = "https://cranelift.readthedocs.io/"
+categories = ["no-std"]
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index f05126a27b..94e4c789f4 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -4,6 +4,7 @@ version = "0.19.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
+categories = ["no-std"]
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index b1224c9a12..0ed69ebad8 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -5,6 +5,7 @@ authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
+categories = ["no-std"]
 readme = "README.md"
 keywords = ["webassembly", "serde"]
 
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index fc459471f8..4ed87adf58 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -5,6 +5,7 @@ authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
 documentation = "https://cranelift.readthedocs.io/"
+categories = ["no-std"]
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index 1f2d8bb605..e7530aa24e 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -6,6 +6,7 @@ description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
 repository = "https://github.com/CraneStation/cranelift"
+categories = ["no-std"]
 readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index df35edb25f..61068ddeca 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -5,6 +5,7 @@ authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
+categories = ["no-std", "wasm"]
 readme = "README.md"
 keywords = ["webassembly", "wasm"]
 

From 0a65089a36dbf15568598572af99acb51190dc0d Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 23 Aug 2018 11:03:52 -0700
Subject: [PATCH 2019/3084] Add a CONTRIBUTING.md file (#479)

* Add a CONTRIBUTING.md file.

* Document the basic PR process.

This also introduces the Core Team.
---
 cranelift/CONTRIBUTING.md | 117 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)
 create mode 100644 cranelift/CONTRIBUTING.md

diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
new file mode 100644
index 0000000000..e753059d64
--- /dev/null
+++ b/cranelift/CONTRIBUTING.md
@@ -0,0 +1,117 @@
+# Contributing to Cranelift
+
+## Welcome!
+
+Cranelift is a very ambitious project with many goals, and while we're
+confident we can achieve some of them, we see many opportunities for people
+to get involved and help us achieve even more.
+
+### Ask questions! Give feedback!
+
+This is a relatively young project, and not everything we hope to do with it
+is reflected in the code or documentation yet. If you see things that seem
+missing or that don't make sense, or even that just don't work the way you
+expect them to, we're interested to hear about it!
+
+We have a [Cranelift chat on Gitter], and questions are also welcome as issues
+in the [Cranelift issue tracker]. Some folks also hang out in the #cranelift
+IRC channel on [irc.mozilla.org].
+
+[Cranelift chat on Gitter]: https://gitter.im/CraneStation/Lobby
+[Cranelift issue tracker]: https://github.com/CraneStation/cranelift/issues/new
+[irc.mozilla.org]: https://wiki.mozilla.org/IRC
+
+### Mentoring
+
+We're happy to mentor people, whether you're learning Rust, learning about
+compiler backends, learning about machine code, learning about how Cranelift
+does things, or all together at once.
+
+We tag issues in the issue tracker marked [good first issue] when we can, so
+that's sometimes a good place to get started. Also, we encourage people to just
+look around and find things they're interested in. This a good time to get
+involved, as there aren't a lot of things set in stone yet.
+
+[good first issue]: https://github.com/CraneStation/cranelift/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22
+
+## Coding Guidelines
+
+For the most part, Cranelift follows common Rust conventions and
+[pull request] (PR) workflows, though we do have a few additional things to
+be aware of.
+
+[pull request]: https://help.github.com/articles/about-pull-requests/
+
+### rustfmt
+
+All PRs must be formatted according to the current stable [rustfmt], and this
+is checked in the continuous integration tests. See the [rustfmt quickstart]
+for setup.
+
+[format-all.sh] is a script for running the appropriate version of rustfmt,
+which may be convenient when there are multiple versions installed.
+
+[rustfmt]: https://github.com/rust-lang-nursery/rustfmt
+[rustfmt quickstart]: https://github.com/rust-lang-nursery/rustfmt#quick-start
+[format-all.sh]: https://github.com/CraneStation/cranelift/blob/master/format-all.sh
+
+### Rustc version support
+
+Our current policy is to support the version of Rustc that ships with the
+latest Ubuntu LTS release, as well as the current stable version. This means
+we don't use some of the very latest released Rust features.
+
+Some of the developer scripts depend on nightly Rust, for example to run
+clippy and other tools, however we avoid depending on these for the main
+build.
+
+That said, if there are any new Rust features that would be particularly
+valuable to use, please bring them up, as we may be able to find ways to
+accommodate them.
+
+### Python
+
+Our Python code is checked with [mypy](http://mypy-lang.org/) and
+[flake8](http://flake8.pycqa.org/en/latest/); see the
+[check.sh](https://github.com/CraneStation/cranelift/blob/master/lib/codegen/meta-python/check.sh)
+file for details. The versions available in common package repositories such
+as Ubuntu or Homebrew typically work fine.
+
+## Development Process
+
+We use [issues] for asking questions and tracking bugs and unimplemented
+features, and [pull requests] (PRs) for tracking and reviewing code
+submissions.
+
+When submitting PRs:
+
+ - Write clear commit messages that start with a one-line summary of the
+   change (and if it's difficult to summarize in one line, consider
+   splitting the change into multiple PRs), optionally followed by
+   additional context. Good things to mention include which areas of the
+   code are affected, which features are affected, and anything that
+   reviewers might want to pay special attention to.
+
+ - If there is code which needs explanation, prefer to put the explanation in
+   a comment in the code, or in documentation, rather than in the commit
+   message.
+
+ - For pull requests that fix existing issues, use [issue keywords]. Note that
+   not all pull requests need to have accompanying issues.
+
+Anyone may submit a pull request, and anyone may comment on or review others'
+pull requests. Pull requests are merged by members of the [Core Team].
+
+We generally squash sequences of incremental-development commits together into
+logical commits (though keeping logical commits focused). Developers may do
+this themselves before submitting a PR or during the PR process, or Core Team
+members may do it when merging a PR. Ideally, the continuous-integration tests
+should pass at each logical commit.
+
+Core Team members may push minor changes directly, though should create PRs
+for significant changes.
+
+[issues]: https://guides.github.com/features/issues/
+[pull requests]: https://help.github.com/articles/about-pull-requests/
+[issue keywords]: https://help.github.com/articles/closing-issues-using-keywords/
+[Core Team]: https://github.com/orgs/CraneStation/people/

From 8e74a4f8fcbf9a3c31d842f5056ee6a3232bb386 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gr=C3=A9goire=20Geis?= 
Date: Mon, 27 Aug 2018 18:38:44 +0200
Subject: [PATCH 2020/3084] Pretty printing preamble errors. (#472)

* Pretty printing preamble errors.
---
 lib/codegen/src/print_errors.rs |  78 +++++++++++++--------
 lib/codegen/src/write.rs        | 116 ++++++++++++++++----------------
 2 files changed, 108 insertions(+), 86 deletions(-)

diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 472f8ab005..6e14182b1d 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -1,9 +1,9 @@
 //! Utility routines for pretty-printing error messages.
 
 use ir;
-use ir::entities::Inst;
+use ir::entities::{AnyEntity, Inst};
 use ir::function::Function;
-use isa::{RegInfo, TargetIsa};
+use isa::TargetIsa;
 use result::CodegenError;
 use std::boxed::Box;
 use std::fmt;
@@ -23,25 +23,6 @@ pub fn pretty_verifier_error<'a>(
     let mut errors = errors.0;
     let mut w = String::new();
 
-    // TODO: Use drain_filter here when it gets stabilized
-    let mut i = 0;
-    let mut wrote_error = false;
-
-    while i != errors.len() {
-        if let ir::entities::AnyEntity::Inst(_) = errors[i].location {
-            i += 1;
-        } else {
-            let err = errors.remove(i);
-
-            writeln!(w, "verifier at {}", err).unwrap();
-            wrote_error = true;
-        }
-    }
-
-    if wrote_error {
-        w.push('\n');
-    }
-
     decorate_function(
         &mut PrettyVerifierError(func_w.unwrap_or(Box::new(PlainWriter)), &mut errors),
         &mut w,
@@ -62,21 +43,22 @@ impl<'a> FuncWriter for PrettyVerifierError<'a> {
         inst: Inst,
         indent: usize,
     ) -> fmt::Result {
-        pretty_function_error(w, func, isa, inst, indent, &mut *self.0, self.1)
+        pretty_instruction_error(w, func, isa, inst, indent, &mut *self.0, self.1)
     }
 
-    fn write_preamble(
+    fn write_entity_definition(
         &mut self,
         w: &mut Write,
         func: &Function,
-        regs: Option<&RegInfo>,
-    ) -> Result {
-        self.0.write_preamble(w, func, regs)
+        entity: AnyEntity,
+        value: &fmt::Display,
+    ) -> fmt::Result {
+        pretty_preamble_error(w, func, entity, value, &mut *self.0, self.1)
     }
 }
 
 /// Pretty-print a function verifier error.
-fn pretty_function_error(
+fn pretty_instruction_error(
     w: &mut Write,
     func: &Function,
     isa: Option<&TargetIsa>,
@@ -125,6 +107,48 @@ fn pretty_function_error(
     Ok(())
 }
 
+fn pretty_preamble_error(
+    w: &mut Write,
+    func: &Function,
+    entity: AnyEntity,
+    value: &fmt::Display,
+    func_w: &mut FuncWriter,
+    errors: &mut Vec,
+) -> fmt::Result {
+    // TODO: Use drain_filter here when it gets stabilized
+    let indent = 4;
+
+    let mut i = 0;
+    let mut printed_entity = false;
+
+    while i != errors.len() {
+        if entity == errors[i].location {
+            let err = errors.remove(i);
+
+            if !printed_entity {
+                func_w.write_entity_definition(w, func, entity, value)?;
+                printed_entity = true;
+            }
+
+            write!(w, "{1:0$}^", indent, "")?;
+            for _c in entity.to_string().chars() {
+                write!(w, "~")?;
+            }
+            writeln!(w, " verifier {}", err.to_string())?;
+        } else {
+            i += 1
+        }
+    }
+
+    if printed_entity {
+        w.write_char('\n')?;
+    } else {
+        func_w.write_entity_definition(w, func, entity, value)?;
+    }
+
+    Ok(())
+}
+
 /// Pretty-print a Cranelift error.
 pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CodegenError) -> String {
     if let CodegenError::Verifier(e) = err {
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index d4d98b9ced..2d2fcf5621 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -3,15 +3,16 @@
 //! The `write` module provides the `write_function` function which converts an IR `Function` to an
 //! equivalent textual form. This textual form can be read back by the `cranelift-reader` crate.
 
+use ir::entities::AnyEntity;
 use ir::{DataFlowGraph, Ebb, Function, Inst, SigRef, Type, Value, ValueDef};
 use isa::{RegInfo, TargetIsa};
 use packed_option::ReservedValue;
 use std::fmt::{self, Write};
 use std::string::String;
 
-/// A `FuncWriter` is used to decorate functions during printing
+/// A `FuncWriter` used to decorate functions during printing.
 pub trait FuncWriter {
-    /// Write the given inst to w
+    /// Write the given `inst` to `w`.
     fn write_instruction(
         &mut self,
         w: &mut Write,
@@ -21,16 +22,66 @@ pub trait FuncWriter {
         ident: usize,
     ) -> fmt::Result;
 
-    /// Write the preamble to w
+    /// Write the preamble to `w`. By default, this uses `write_entity_definition`.
     fn write_preamble(
         &mut self,
         w: &mut Write,
         func: &Function,
         regs: Option<&RegInfo>,
-    ) -> Result;
+    ) -> Result {
+        let mut any = false;
+
+        for (ss, slot) in func.stack_slots.iter() {
+            any = true;
+            self.write_entity_definition(w, func, ss.into(), slot)?;
+        }
+
+        for (gv, gv_data) in &func.global_values {
+            any = true;
+            self.write_entity_definition(w, func, gv.into(), gv_data)?;
+        }
+
+        for (heap, heap_data) in &func.heaps {
+            any = true;
+            self.write_entity_definition(w, func, heap.into(), heap_data)?;
+        }
+
+        // Write out all signatures before functions since function declarations can refer to
+        // signatures.
+        for (sig, sig_data) in &func.dfg.signatures {
+            any = true;
+            self.write_entity_definition(w, func, sig.into(), &sig_data.display(regs))?;
+        }
+
+        for (fnref, ext_func) in &func.dfg.ext_funcs {
+            any = true;
+            if ext_func.signature != SigRef::reserved_value() {
+                self.write_entity_definition(w, func, fnref.into(), ext_func)?;
+            }
+        }
+
+        for (jt, jt_data) in &func.jump_tables {
+            any = true;
+            self.write_entity_definition(w, func, jt.into(), jt_data)?;
+        }
+
+        Ok(any)
+    }
+
+    /// Write an entity definition defined in the preamble to `w`.
+    #[allow(unused_variables)]
+    fn write_entity_definition(
+        &mut self,
+        w: &mut Write,
+        func: &Function,
+        entity: AnyEntity,
+        value: &fmt::Display,
+    ) -> fmt::Result {
+        writeln!(w, "    {} = {}", entity, value)
+    }
 }
 
-/// A `PlainWriter` doesn't decorate the function
+/// A `PlainWriter` that doesn't decorate the function.
 pub struct PlainWriter;
 
 impl FuncWriter for PlainWriter {
@@ -44,15 +95,6 @@ impl FuncWriter for PlainWriter {
     ) -> fmt::Result {
         write_instruction(w, func, isa, inst, indent)
     }
-
-    fn write_preamble(
-        &mut self,
-        w: &mut Write,
-        func: &Function,
-        regs: Option<&RegInfo>,
-    ) -> Result {
-        write_preamble(w, func, regs)
-    }
 }
 
 /// Write `func` to `w` as equivalent text.
@@ -61,7 +103,7 @@ pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) -
     decorate_function(&mut PlainWriter, w, func, isa)
 }
 
-/// Writes 'func' to 'w' as text.
+/// Writes `func` to `w` as text.
 /// write_function_plain is passed as 'closure' to print instructions as text.
 /// pretty_function_error is passed as 'closure' to add error decoration.
 pub fn decorate_function(
@@ -95,50 +137,6 @@ fn write_spec(w: &mut Write, func: &Function, regs: Option<&RegInfo>) -> fmt::Re
     write!(w, "{}{}", func.name, func.signature.display(regs))
 }
 
-fn write_preamble(
-    w: &mut Write,
-    func: &Function,
-    regs: Option<&RegInfo>,
-) -> Result {
-    let mut any = false;
-
-    for (ss, slot) in func.stack_slots.iter() {
-        any = true;
-        writeln!(w, "    {} = {}", ss, slot)?;
-    }
-
-    for (gv, gv_data) in &func.global_values {
-        any = true;
-        writeln!(w, "    {} = {}", gv, gv_data)?;
-    }
-
-    for (heap, heap_data) in &func.heaps {
-        any = true;
-        writeln!(w, "    {} = {}", heap, heap_data)?;
-    }
-
-    // Write out all signatures before functions since function declarations can refer to
-    // signatures.
-    for (sig, sig_data) in &func.dfg.signatures {
-        any = true;
-        writeln!(w, "    {} = {}", sig, sig_data.display(regs))?;
-    }
-
-    for (fnref, ext_func) in &func.dfg.ext_funcs {
-        any = true;
-        if ext_func.signature != SigRef::reserved_value() {
-            writeln!(w, "    {} = {}", fnref, ext_func)?;
-        }
-    }
-
-    for (jt, jt_data) in &func.jump_tables {
-        any = true;
-        writeln!(w, "    {} = {}", jt, jt_data)?;
-    }
-
-    Ok(any)
-}
-
 //----------------------------------------------------------------------
 //
 // Basic blocks

From eab8f784fae5e8b7345044d4f834e0e56ccd5c73 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 27 Aug 2018 08:03:06 -0700
Subject: [PATCH 2021/3084] Rename nop.cton to nop.clif.

---
 cranelift/filetests/isa/x86/{nop.cton => nop.clif} | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename cranelift/filetests/isa/x86/{nop.cton => nop.clif} (100%)

diff --git a/cranelift/filetests/isa/x86/nop.cton b/cranelift/filetests/isa/x86/nop.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/nop.cton
rename to cranelift/filetests/isa/x86/nop.clif

From 1b51314381b7e6e4b627dbe60cccdff2fbb8c445 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 27 Aug 2018 08:21:29 -0700
Subject: [PATCH 2022/3084] Update test keywords in VIM syntax highlighting.

---
 misc/vim/syntax/clif.vim | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/misc/vim/syntax/clif.vim b/misc/vim/syntax/clif.vim
index 6636abc220..631d682e27 100644
--- a/misc/vim/syntax/clif.vim
+++ b/misc/vim/syntax/clif.vim
@@ -13,7 +13,7 @@ endif
 " They tend to refer to weird stuff like assembler mnemonics anyway.
 syn spell notoplevel
 
-syn keyword clifHeader test isa set
+syn keyword clifHeader test target set
 syn keyword clifDecl function jump_table incoming_arg outgoing_arg spill_slot explicit_slot emergency_slot
 syn keyword clifFilecheck check sameln nextln unordered not regex contained
 

From f39428a5cd303eed95cb13eda6c4832e97b991f1 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 27 Aug 2018 08:21:44 -0700
Subject: [PATCH 2023/3084] Update maintainers in VIM syntax highlighting.

---
 misc/vim/syntax/clif.vim | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/misc/vim/syntax/clif.vim b/misc/vim/syntax/clif.vim
index 631d682e27..337e3a78a7 100644
--- a/misc/vim/syntax/clif.vim
+++ b/misc/vim/syntax/clif.vim
@@ -1,7 +1,6 @@
 " Vim syntax file
 " Language:     Cranelift
-" Maintainer:   Jakob Stoklund Olesen 
Date: Mon, 27 Aug 2018 09:51:02 -0700
Subject: [PATCH 2024/3084] Clarify that we use rustfmt-preview.

This may help avoid confusion with older rustfmt versions.
---
 cranelift/CONTRIBUTING.md | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
index e753059d64..351681e290 100644
--- a/cranelift/CONTRIBUTING.md
+++ b/cranelift/CONTRIBUTING.md
@@ -44,14 +44,14 @@ be aware of.
 
 ### rustfmt
 
-All PRs must be formatted according to the current stable [rustfmt], and this
-is checked in the continuous integration tests. See the [rustfmt quickstart]
-for setup.
+All PRs must be formatted according to rustfmt, and this is checked in the
+continuous integration tests. We use the current stable [rustfmt-preview]
+version. See the [rustfmt quickstart] for setup.
 
 [format-all.sh] is a script for running the appropriate version of rustfmt,
 which may be convenient when there are multiple versions installed.
 
-[rustfmt]: https://github.com/rust-lang-nursery/rustfmt
+[rustfmt-preview]: https://github.com/rust-lang-nursery/rustfmt
 [rustfmt quickstart]: https://github.com/rust-lang-nursery/rustfmt#quick-start
 [format-all.sh]: https://github.com/CraneStation/cranelift/blob/master/format-all.sh
 

From 0e67255f52aab8eede6f9b7368872b965b9dc6fd Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gr=C3=A9goire=20Geis?= 
Date: Tue, 28 Aug 2018 19:33:46 +0200
Subject: [PATCH 2025/3084] Fix error not reported if at least one other error
 expected. (#485)

* fix error not reported if at least one other error expected.

* Fixed unused extern crate error if wasm feature is not enabled.

* No longer reporting deref cycles multiple times.

* Fix filetest type_check.clif.

* Switched comparison order for perf.

* Fixed isa/riscv/verify-encoding.clif filetest.
---
 cranelift/filetests/isa/riscv/verify-encoding.clif | 4 ++--
 cranelift/filetests/verifier/type_check.clif       | 2 ++
 cranelift/src/clif-util.rs                         | 2 +-
 lib/codegen/src/verifier/mod.rs                    | 7 ++++++-
 lib/filetests/src/test_verifier.rs                 | 7 ++++++-
 5 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/cranelift/filetests/isa/riscv/verify-encoding.clif b/cranelift/filetests/isa/riscv/verify-encoding.clif
index 99ed4697bb..0ee7eb7a83 100644
--- a/cranelift/filetests/isa/riscv/verify-encoding.clif
+++ b/cranelift/filetests/isa/riscv/verify-encoding.clif
@@ -7,7 +7,7 @@ function %RV32I(i32 link [%x1]) -> i32 link [%x1] {
 ebb0(v9999: i32):
     ; iconst.i32 needs legalizing, so it should throw a
     [R#0,-]         v1 = iconst.i32 0xf0f0f0f0f0 ; error: Instruction failed to re-encode
-    return v9999
+    [Iret#19]       return v9999
 }
 
 function %RV32I(i32 link [%x1]) -> i32 link [%x1] {
@@ -17,5 +17,5 @@ ebb0(v9999: i32):
     v1 = iconst.i32 1
     v2 = iconst.i32 2
     [R#0,-]         v3 = iadd v1, v2 ; error: encoding R#00 should be R#0c
-    return v9999
+    [Iret#19]       return v9999
 }
diff --git a/cranelift/filetests/verifier/type_check.clif b/cranelift/filetests/verifier/type_check.clif
index 1e7278e973..9cd9c9f8b0 100644
--- a/cranelift/filetests/verifier/type_check.clif
+++ b/cranelift/filetests/verifier/type_check.clif
@@ -82,6 +82,7 @@ function %jump_args() {
         v0 = iconst.i16 10
         v3 = iconst.i64 20
         jump ebb1(v0, v3) ; error: arg 0 (v0) has type i16, expected i64
+                          ; error: arg 1 (v3) has type i64, expected i16
     ebb1(v10: i64, v11: i16):
         return
 }
@@ -91,6 +92,7 @@ function %jump_args2() {
         v0 = iconst.i16 10
         v3 = iconst.i64 20
         brz v0, ebb1(v0, v3) ; error: arg 0 (v0) has type i16, expected i64
+                             ; error: arg 1 (v3) has type i64, expected i16
         jump ebb1(v3, v0)
     ebb1(v10: i64, v11: i16):
         return
diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 2a7ae79d30..10875b9276 100644
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -15,13 +15,13 @@ extern crate cfg_if;
 extern crate capstone;
 extern crate clap;
 extern crate cranelift_codegen;
-extern crate cranelift_entity;
 extern crate cranelift_filetests;
 extern crate cranelift_reader;
 extern crate pretty_env_logger;
 
 cfg_if! {
     if #[cfg(feature = "wasm")] {
+        extern crate cranelift_entity;
         extern crate cranelift_wasm;
         extern crate term;
         extern crate wabt;
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index bb116b00b3..bce49501e3 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -328,6 +328,7 @@ impl<'a> Verifier<'a> {
     //  - cycles in the global value declarations.
     //  - use of 'vmctx' when no special parameter declares it.
     fn verify_global_values(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
+        let mut cycle_seen = false;
         let mut seen = SparseSet::new();
 
         'gvs: for gv in self.func.global_values.keys() {
@@ -337,7 +338,10 @@ impl<'a> Verifier<'a> {
             let mut cur = gv;
             while let ir::GlobalValueData::Deref { base, .. } = self.func.global_values[cur] {
                 if seen.insert(base).is_some() {
-                    report!(errors, gv, "deref cycle: {}", DisplayList(seen.as_slice()));
+                    if !cycle_seen {
+                        report!(errors, gv, "deref cycle: {}", DisplayList(seen.as_slice()));
+                        cycle_seen = true; // ensures we don't report the cycle multiple times
+                    }
                     continue 'gvs;
                 }
 
@@ -1511,6 +1515,7 @@ impl<'a> Verifier<'a> {
     pub fn run(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         self.verify_global_values(errors)?;
         self.typecheck_entry_block_params(errors)?;
+
         for ebb in self.func.layout.ebbs() {
             for inst in self.func.layout.ebb_insts(ebb) {
                 self.ebb_integrity(ebb, inst, errors)?;
diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs
index 0f5291c70c..842e720cc4 100644
--- a/lib/filetests/src/test_verifier.rs
+++ b/lib/filetests/src/test_verifier.rs
@@ -66,7 +66,7 @@ impl SubTest for TestVerifier {
                 for expect in expected {
                     let pos = errors
                         .iter()
-                        .position(|err| err.message.contains(expect.1) && err.location == expect.0);
+                        .position(|err| err.location == expect.0 && err.message.contains(expect.1));
 
                     match pos {
                         None => {
@@ -78,6 +78,11 @@ impl SubTest for TestVerifier {
                     }
                 }
 
+                // report remaining errors
+                for err in errors {
+                    write!(msg, "unexpected error {}", err).unwrap();
+                }
+
                 if msg.is_empty() {
                     Ok(())
                 } else {

From e60477092a26a2f5986fd26cf2754363417479ca Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 12:31:38 -0700
Subject: [PATCH 2026/3084] Add a `make_signature` function for making callable
 signatures.

The `Module` can create signatures with the appropriate calling
convention.
---
 lib/module/src/module.rs | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 185057b946..cc89ad68c0 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -339,7 +339,7 @@ where
         ctx
     }
 
-    /// Create a new `Context` initialized for use with this `Module`.
+    /// Clear the given `Context` and reset it for use with a new function.
     ///
     /// This ensures that the `Context` is initialized with the default calling
     /// convention for the `TargetIsa`.
@@ -348,6 +348,21 @@ where
         ctx.func.signature.call_conv = self.backend.isa().flags().call_conv();
     }
 
+    /// Create a new empty `Signature` with the default calling convention for
+    /// the `TargetIsa`, to which parameter and return types can be added for
+    /// declaring a function to be called by this `Module`.
+    pub fn make_signature(&self) -> ir::Signature {
+        ir::Signature::new(self.backend.isa().flags().call_conv())
+    }
+
+    /// Clear the given `Signature` and reset for use with a new function.
+    ///
+    /// This ensures that the `Signature` is initialized with the default
+    /// calling convention for the `TargetIsa`.
+    pub fn clear_signature(&self, sig: &mut ir::Signature) {
+        sig.clear(self.backend.isa().flags().call_conv());
+    }
+
     /// Declare a function in this module.
     pub fn declare_function(
         &mut self,

From 6af407144c6441720b2953d080455e565ee81b0e Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 12:36:21 -0700
Subject: [PATCH 2027/3084] Remove `Signature`'s `argument_bytes` field.

It's not currently used. If we do need such information, it would be
better to compute it on demand.
---
 cranelift/tests/moduletests.rs        |  2 --
 lib/codegen/src/ir/extfunc.rs         | 37 +--------------------------
 lib/codegen/src/legalizer/boundary.rs |  1 -
 lib/codegen/src/verifier/mod.rs       |  5 ----
 lib/reader/src/parser.rs              |  4 ---
 5 files changed, 1 insertion(+), 48 deletions(-)

diff --git a/cranelift/tests/moduletests.rs b/cranelift/tests/moduletests.rs
index c2312fe4f2..a4aea4b389 100644
--- a/cranelift/tests/moduletests.rs
+++ b/cranelift/tests/moduletests.rs
@@ -19,7 +19,6 @@ fn error_on_incompatible_sig_in_declare_function() {
         params: vec![AbiParam::new(types::I64)],
         returns: vec![],
         call_conv: CallConv::SystemV,
-        argument_bytes: None,
     };
     module
         .declare_function("abc", Linkage::Local, &sig)
@@ -36,7 +35,6 @@ fn define_simple_function(module: &mut Module) -> FuncId {
         params: vec![],
         returns: vec![],
         call_conv: CallConv::SystemV,
-        argument_bytes: None,
     };
 
     let func_id = module
diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs
index 30f0eaab74..451a90a7f9 100644
--- a/lib/codegen/src/ir/extfunc.rs
+++ b/lib/codegen/src/ir/extfunc.rs
@@ -8,7 +8,6 @@
 use ir::{ArgumentLoc, ExternalName, SigRef, Type};
 use isa::{RegInfo, RegUnit};
 use settings::CallConv;
-use std::cmp;
 use std::fmt;
 use std::str::FromStr;
 use std::vec::Vec;
@@ -29,13 +28,6 @@ pub struct Signature {
 
     /// Calling convention.
     pub call_conv: CallConv,
-
-    /// When the signature has been legalized to a specific ISA, this holds the size of the
-    /// argument array on the stack. Before legalization, this is `None`.
-    ///
-    /// This can be computed from the legalized `params` array as the maximum (offset plus
-    /// byte size) of the `ArgumentLoc::Stack(offset)` argument.
-    pub argument_bytes: Option,
 }
 
 impl Signature {
@@ -45,7 +37,6 @@ impl Signature {
             params: Vec::new(),
             returns: Vec::new(),
             call_conv,
-            argument_bytes: None,
         }
     }
 
@@ -54,25 +45,6 @@ impl Signature {
         self.params.clear();
         self.returns.clear();
         self.call_conv = call_conv;
-        self.argument_bytes = None;
-    }
-
-    /// Compute the size of the stack arguments and mark signature as legalized.
-    ///
-    /// Even if there are no stack arguments, this will set `params` to `Some(0)` instead
-    /// of `None`. This indicates that the signature has been legalized.
-    pub fn compute_argument_bytes(&mut self) {
-        let bytes = self
-            .params
-            .iter()
-            .filter_map(|arg| match arg.location {
-                ArgumentLoc::Stack(offset) if offset >= 0 => {
-                    Some(offset as u32 + arg.value_type.bytes())
-                }
-                _ => None,
-            })
-            .fold(0, cmp::max);
-        self.argument_bytes = Some(bytes);
     }
 
     /// Return an object that can display `self` with correct register names.
@@ -421,16 +393,9 @@ mod tests {
         sig.returns.push(AbiParam::new(B8));
         assert_eq!(sig.to_string(), "(i32, i32x4) -> f32, b8 baldrdash");
 
-        // Test the offset computation algorithm.
-        assert_eq!(sig.argument_bytes, None);
-        sig.params[1].location = ArgumentLoc::Stack(8);
-        sig.compute_argument_bytes();
-        // An `i32x4` at offset 8 requires a 24-byte argument array.
-        assert_eq!(sig.argument_bytes, Some(24));
         // Order does not matter.
         sig.params[0].location = ArgumentLoc::Stack(24);
-        sig.compute_argument_bytes();
-        assert_eq!(sig.argument_bytes, Some(28));
+        sig.params[1].location = ArgumentLoc::Stack(8);
 
         // Writing ABI-annotated signatures.
         assert_eq!(
diff --git a/lib/codegen/src/legalizer/boundary.rs b/lib/codegen/src/legalizer/boundary.rs
index 6f3143028b..e052e63be6 100644
--- a/lib/codegen/src/legalizer/boundary.rs
+++ b/lib/codegen/src/legalizer/boundary.rs
@@ -57,7 +57,6 @@ pub fn legalize_libcall_signature(signature: &mut Signature, isa: &TargetIsa) {
 /// `current` is true if this is the signature for the current function.
 fn legalize_signature(signature: &mut Signature, current: bool, isa: &TargetIsa) {
     isa.legalize_signature(signature, current);
-    signature.compute_argument_bytes();
 }
 
 /// Legalize the entry block parameters after `func`'s signature has been legalized.
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index bce49501e3..2f9c1dbcb4 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -1160,11 +1160,6 @@ impl<'a> Verifier<'a> {
     ) -> VerifierStepResult<()> {
         let sig = &self.func.dfg.signatures[sig_ref];
 
-        // Before legalization, there's nothing to check.
-        if sig.argument_bytes.is_none() {
-            return Ok(());
-        }
-
         let args = self.func.dfg.inst_variable_args(inst);
         let expected_args = &sig.params[..];
 
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index ff155fc195..c5bbe47f8f 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -919,10 +919,6 @@ impl<'a> Parser<'a> {
             }
         }
 
-        if sig.params.iter().all(|a| a.location.is_assigned()) {
-            sig.compute_argument_bytes();
-        }
-
         Ok(sig)
     }
 

From ba8dd836fffd7101edac54b4c63d0229a19a39d3 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 13:23:37 -0700
Subject: [PATCH 2028/3084] Update to wabt 0.5.0.

---
 cranelift/Cargo.toml | 2 +-
 lib/wasm/Cargo.toml  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 93f35f0c69..e132bf3e0e 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -31,7 +31,7 @@ clap = "2.32.0"
 serde = "1.0.8"
 term = "0.5.1"
 capstone = { version = "0.4", optional = true }
-wabt = { version = "0.4", optional = true }
+wabt = { version = "0.5", optional = true }
 target-lexicon = "0.0.3"
 pretty_env_logger = "0.2.4"
 file-per-thread-logger = "0.1.1"
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 61068ddeca..3681537c51 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -21,7 +21,7 @@ target-lexicon = { version = "0.0.3", default-features = false }
 log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
 
 [dev-dependencies]
-wabt = "0.4"
+wabt = "0.5.0"
 
 [features]
 default = ["std"]

From 00ddf3a7a660ebf1905f5995e7832515f1f6223d Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 13:23:55 -0700
Subject: [PATCH 2029/3084] Update to serde_derive 1.0.75.

---
 lib/serde/Cargo.toml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index 0ed69ebad8..d4e66b2868 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -16,8 +16,8 @@ path = "src/clif-json.rs"
 [dependencies]
 clap = "2.32.0"
 serde = "1.0.8"
-serde_derive = "1.0.8"
-serde_json = "1.0"
+serde_derive = "1.0.75"
+serde_json = "1.0.26"
 cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
 cranelift-reader = { path = "../reader", version = "0.19.0", default-features = false }
 

From eb439c9a68570885c420cd3e6d6edfa1c646b5f1 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 13:37:33 -0700
Subject: [PATCH 2030/3084] Fix legalization of heap_addrs with 32-bit indices.
 (#480)

This makes several changes:
 - It adds an index_type to heap declarations, allowing heaps to specify the
   type for indexing. This also anticipates 64-bit heap support.

 - It adds a memory_type to deref global values, allowing deref globals to
   have types other than pointers. This is used to allow the bound variable
   in dynamic heaps to have type i32, to match the index type in heaps
   with i32 index type.

 - And, it fixes heap legalization to do the bounds check in the heap's
   index type.
---
 .../filetests/isa/x86/legalize-heaps.clif     | 111 ++++++++++
 .../filetests/isa/x86/legalize-memory.clif    |   2 +-
 cranelift/filetests/parser/memory.clif        |   4 +-
 cranelift/filetests/verifier/globals.clif     |  19 ++
 cranelift/filetests/verifier/heap.clif        |  45 ++++
 cranelift/filetests/verifier/memory.clif      |   6 +-
 lib/codegen/src/ir/globalvalue.rs             |  22 +-
 lib/codegen/src/ir/heap.rs                    |  11 +-
 lib/codegen/src/legalizer/globalvalue.rs      |  29 ++-
 lib/codegen/src/legalizer/heap.rs             |   2 +-
 lib/codegen/src/verifier/mod.rs               | 198 +++++++++++++-----
 lib/reader/src/parser.rs                      |  13 +-
 lib/wasm/src/environ/dummy.rs                 |   4 +
 13 files changed, 396 insertions(+), 70 deletions(-)
 create mode 100644 cranelift/filetests/isa/x86/legalize-heaps.clif
 create mode 100644 cranelift/filetests/verifier/globals.clif
 create mode 100644 cranelift/filetests/verifier/heap.clif

diff --git a/cranelift/filetests/isa/x86/legalize-heaps.clif b/cranelift/filetests/isa/x86/legalize-heaps.clif
new file mode 100644
index 0000000000..8fe2b8737c
--- /dev/null
+++ b/cranelift/filetests/isa/x86/legalize-heaps.clif
@@ -0,0 +1,111 @@
+test legalizer
+target x86_64
+
+; Test legalization for various forms of heap addresses.
+
+function %heap_addrs(i32, i64, i64 vmctx) {
+    gv0 = vmctx+64
+    gv1 = vmctx+72
+    gv2 = vmctx+80
+    gv3 = vmctx+88
+    gv4 = deref(gv3): i32
+
+    heap0 = static gv0, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000, index_type i32
+    heap1 = static gv0, guard 0x1000, bound 0x1_0000, index_type i32
+    heap2 = static gv0, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000, index_type i64
+    heap3 = static gv0, guard 0x1000, bound 0x1_0000, index_type i64
+    heap4 = dynamic gv1, min 0x1_0000, bound gv4, guard 0x8000_0000, index_type i32
+    heap5 = dynamic gv1, bound gv4, guard 0x1000, index_type i32
+    heap6 = dynamic gv1, min 0x1_0000, bound gv2, guard 0x8000_0000, index_type i64
+    heap7 = dynamic gv1, bound gv2, guard 0x1000, index_type i64
+
+    ; check: heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000, index_type i32
+    ; check: heap1 = static gv0, min 0, bound 0x0001_0000, guard 4096, index_type i32
+    ; check: heap2 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000, index_type i64
+    ; check: heap3 = static gv0, min 0, bound 0x0001_0000, guard 4096, index_type i64
+    ; check: heap4 = dynamic gv1, min 0x0001_0000, bound gv4, guard 0x8000_0000, index_type i32
+    ; check: heap5 = dynamic gv1, min 0, bound gv4, guard 4096, index_type i32
+    ; check: heap6 = dynamic gv1, min 0x0001_0000, bound gv2, guard 0x8000_0000, index_type i64
+    ; check: heap7 = dynamic gv1, min 0, bound gv2, guard 4096, index_type i64
+
+ebb0(v0: i32, v1: i64, v3: i64):
+    ; The fast-path; 32-bit index, static heap with a sufficient bound, no bounds check needed!
+    v4 = heap_addr.i64 heap0, v0, 0
+    ; check:         v12 = uextend.i64 v0
+    ; check:         v13 = iadd_imm v3, 64
+    ; check:         v4 = iadd v13, v12
+
+    v5 = heap_addr.i64 heap1, v0, 0
+    ; check:         v14 = icmp_imm ugt v0, 0x0001_0000
+    ; check:         brz v14, ebb1
+    ; check:         trap heap_oob
+    ; check:     ebb1:
+    ; check:         v15 = uextend.i64 v0
+    ; check:         v16 = iadd_imm.i64 v3, 64
+    ; check:         v5 = iadd v16, v15
+
+    v6 = heap_addr.i64 heap2, v1, 0
+    ; check:         v19 = iconst.i64 0x0001_0000_0000
+    ; check:         v17 = icmp.i64 ugt v1, v19
+    ; check:         brz v17, ebb2
+    ; check:         trap heap_oob
+    ; check:     ebb2:
+    ; check:         v18 = iadd_imm.i64 v3, 64
+    ; check:         v6 = iadd v18, v1
+
+    v7 = heap_addr.i64 heap3, v1, 0
+    ; check:         v20 = icmp_imm.i64 ugt v1, 0x0001_0000
+    ; check:         brz v20, ebb3
+    ; check:         trap heap_oob
+    ; check:     ebb3:
+    ; check:         v21 = iadd_imm.i64 v3, 64
+    ; check:         v7 = iadd v21, v1
+
+    v8 = heap_addr.i64 heap4, v0, 0
+    ; check:         v27 = iadd_imm.i64 v3, 88
+    ; check:         v28 = load.i32 notrap aligned v27
+    ; check:         v22 = iadd_imm v28, 0
+    ; check:         v23 = iadd_imm v22, 0
+    ; check:         v24 = icmp.i32 ugt v0, v23
+    ; check:         brz v24, ebb4
+    ; check:         trap heap_oob
+    ; check:     ebb4:
+    ; check:         v25 = uextend.i64 v0
+    ; check:         v26 = iadd_imm.i64 v3, 72
+    ; check:         v8 = iadd v26, v25
+
+    v9 = heap_addr.i64 heap5, v0, 0
+    ; check:         v34 = iadd_imm.i64 v3, 88
+    ; check:         v35 = load.i32 notrap aligned v34
+    ; check:         v29 = iadd_imm v35, 0
+    ; check:         v30 = iadd_imm v29, 0
+    ; check:         v31 = icmp.i32 ugt v0, v30
+    ; check:         brz v31, ebb5
+    ; check:         trap heap_oob
+    ; check:     ebb5:
+    ; check:         v32 = uextend.i64 v0
+    ; check:         v33 = iadd_imm.i64 v3, 72
+    ; check:         v9 = iadd v33, v32
+
+    v10 = heap_addr.i64 heap6, v1, 0
+    ; check:         v36 = iadd_imm.i64 v3, 80
+    ; check:         v37 = iadd_imm v36, 0
+    ; check:         v38 = icmp.i64 ugt v1, v37
+    ; check:         brz v38, ebb6
+    ; check:         trap heap_oob
+    ; check:     ebb6:
+    ; check:         v39 = iadd_imm.i64 v3, 72
+    ; check:         v10 = iadd v39, v1
+
+    v11 = heap_addr.i64 heap7, v1, 0
+    ; check:         v40 = iadd_imm.i64 v3, 80
+    ; check:         v41 = iadd_imm v40, 0
+    ; check:         v42 = icmp.i64 ugt v1, v41
+    ; check:         brz v42, ebb7
+    ; check:         trap heap_oob
+    ; check:     ebb7:
+    ; check:         v43 = iadd_imm.i64 v3, 72
+    ; check:         v11 = iadd v43, v1
+
+    return
+}
diff --git a/cranelift/filetests/isa/x86/legalize-memory.clif b/cranelift/filetests/isa/x86/legalize-memory.clif
index ce6d194507..3728e84b58 100644
--- a/cranelift/filetests/isa/x86/legalize-memory.clif
+++ b/cranelift/filetests/isa/x86/legalize-memory.clif
@@ -17,7 +17,7 @@ ebb1(v1: i64):
 
 function %deref(i64 vmctx) -> i64 {
     gv1 = vmctx-16
-    gv2 = deref(gv1)+32
+    gv2 = deref(gv1)+32: i64
 
 ebb1(v1: i64):
     v2 = global_value.i64 gv2
diff --git a/cranelift/filetests/parser/memory.clif b/cranelift/filetests/parser/memory.clif
index cd59b892c6..1c2fe4cc4c 100644
--- a/cranelift/filetests/parser/memory.clif
+++ b/cranelift/filetests/parser/memory.clif
@@ -17,7 +17,7 @@ ebb0(v0: i64):
 
 function %deref(i64 vmctx) -> i32 {
     gv3 = vmctx+16
-    gv4 = deref(gv3)-32
+    gv4 = deref(gv3)-32: i32
     ; check: gv4 = deref(gv3)-32
 ebb0(v0: i64):
     v1 = global_value.i32 gv4
@@ -27,7 +27,7 @@ ebb0(v0: i64):
 
 ; Refer to a global value before it's been declared.
 function %backref(i64 vmctx) -> i32 {
-    gv1 = deref(gv2)-32
+    gv1 = deref(gv2)-32: i32
     ; check: gv1 = deref(gv2)-32
     gv2 = vmctx+16
     ; check: gv2 = vmctx+16
diff --git a/cranelift/filetests/verifier/globals.clif b/cranelift/filetests/verifier/globals.clif
new file mode 100644
index 0000000000..f5f99a95ae
--- /dev/null
+++ b/cranelift/filetests/verifier/globals.clif
@@ -0,0 +1,19 @@
+test verifier
+target x86_64
+
+function %deref_base_type(i64 vmctx) {
+    gv0 = vmctx+0
+    gv1 = deref(gv0): i32
+    gv2 = deref(gv1): i32 ; error: deref base gv1 has type i32, which is not the pointer type i64
+
+ebb0(v0: i64):
+    return
+}
+
+function %global_value_wrong_type(i64 vmctx) {
+    gv0 = vmctx+0
+
+ebb0(v0: i64):
+    v1 = global_value.i32 gv0 ; error: global_value instruction with type i32 references global value with type i64
+    return
+}
diff --git a/cranelift/filetests/verifier/heap.clif b/cranelift/filetests/verifier/heap.clif
new file mode 100644
index 0000000000..f3d1fee2f8
--- /dev/null
+++ b/cranelift/filetests/verifier/heap.clif
@@ -0,0 +1,45 @@
+test verifier
+target x86_64
+
+function %heap_base_type(i64 vmctx) {
+    gv0 = vmctx+0
+    gv1 = deref(gv0): i32
+    heap0 = static gv1, guard 0x1000, bound 0x1_0000, index_type i32 ; error: heap base has type i32, which is not the pointer type i64
+
+ebb0(v0: i64):
+    return
+}
+
+function %invalid_base(i64 vmctx) {
+    gv0 = vmctx+0
+    heap0 = dynamic gv1, bound gv0, guard 0x1000, index_type i64 ; error: invalid base global value gv1
+
+ebb0(v0: i64):
+    return
+}
+
+function %invalid_bound(i64 vmctx) {
+    gv0 = vmctx+0
+    heap0 = dynamic gv0, bound gv1, guard 0x1000, index_type i64 ; error: invalid bound global value gv1
+
+ebb0(v0: i64):
+    return
+}
+
+function %heap_bound_type(i64 vmctx) {
+    gv0 = vmctx+0
+    gv1 = deref(gv0): i16
+    heap0 = dynamic gv1, bound gv1, guard 0x1000, index_type i32 ; error: heap index type i32 differs from the type of its bound, i16
+
+ebb0(v0: i64):
+    return
+}
+
+function %heap_addr_index_type(i64 vmctx, i64) {
+    gv0 = vmctx+0
+    heap0 = static gv0, guard 0x1000, bound 0x1_0000, index_type i32
+
+ebb0(v0: i64, v1: i64):
+    v2 = heap_addr.i64 heap0, v1, 0; error: index type i64 differs from heap index type i32
+    return
+}
diff --git a/cranelift/filetests/verifier/memory.clif b/cranelift/filetests/verifier/memory.clif
index 4818d4c686..a0bcaa11a1 100644
--- a/cranelift/filetests/verifier/memory.clif
+++ b/cranelift/filetests/verifier/memory.clif
@@ -1,15 +1,15 @@
 test verifier
 
 function %deref_cycle() {
-    gv1 = deref(gv2)-32 ; error: deref cycle: [gv1, gv2]
-    gv2 = deref(gv1)
+    gv1 = deref(gv2)-32: i32 ; error: deref cycle: [gv1, gv2]
+    gv2 = deref(gv1): i32
 
 ebb1:
     return
 }
 
 function %self_cycle() {
-    gv0 = deref(gv0)-32 ; error: deref cycle: [gv0]
+    gv0 = deref(gv0)-32: i32 ; error: deref cycle: [gv0]
 
 ebb1:
     return
diff --git a/lib/codegen/src/ir/globalvalue.rs b/lib/codegen/src/ir/globalvalue.rs
index b29a1b5fd5..8a119bce51 100644
--- a/lib/codegen/src/ir/globalvalue.rs
+++ b/lib/codegen/src/ir/globalvalue.rs
@@ -1,7 +1,8 @@
 //! Global values.
 
 use ir::immediates::Offset32;
-use ir::{ExternalName, GlobalValue};
+use ir::{ExternalName, GlobalValue, Type};
+use isa::TargetIsa;
 use std::fmt;
 
 /// Information about a global value declaration.
@@ -18,13 +19,16 @@ pub enum GlobalValueData {
     ///
     /// The `base` global value is assumed to contain a pointer. This global value is computed
     /// by loading from memory at that pointer value, and then adding an offset. The memory must
-    /// be accessible, and naturally aligned to hold a pointer value.
+    /// be accessible, and naturally aligned to hold a value of the type.
     Deref {
         /// The base pointer global value.
         base: GlobalValue,
 
         /// Byte offset to be added to the loaded value.
         offset: Offset32,
+
+        /// Type of the loaded value.
+        memory_type: Type,
     },
 
     /// Value is identified by a symbolic name. Cranelift itself does not interpret this name;
@@ -48,13 +52,25 @@ impl GlobalValueData {
             _ => panic!("only symbols have names"),
         }
     }
+
+    /// Return the type of this global.
+    pub fn global_type(&self, isa: &TargetIsa) -> Type {
+        match *self {
+            GlobalValueData::VMContext { .. } | GlobalValueData::Sym { .. } => isa.pointer_type(),
+            GlobalValueData::Deref { memory_type, .. } => memory_type,
+        }
+    }
 }
 
 impl fmt::Display for GlobalValueData {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
             GlobalValueData::VMContext { offset } => write!(f, "vmctx{}", offset),
-            GlobalValueData::Deref { base, offset } => write!(f, "deref({}){}", base, offset),
+            GlobalValueData::Deref {
+                base,
+                offset,
+                memory_type,
+            } => write!(f, "deref({}){}: {}", base, offset, memory_type),
             GlobalValueData::Sym {
                 ref name,
                 colocated,
diff --git a/lib/codegen/src/ir/heap.rs b/lib/codegen/src/ir/heap.rs
index e0c942d366..d31c4c1f15 100644
--- a/lib/codegen/src/ir/heap.rs
+++ b/lib/codegen/src/ir/heap.rs
@@ -1,7 +1,7 @@
 //! Heaps.
 
 use ir::immediates::Imm64;
-use ir::GlobalValue;
+use ir::{GlobalValue, Type};
 use std::fmt;
 
 /// Information about a heap declaration.
@@ -19,6 +19,9 @@ pub struct HeapData {
 
     /// Heap style, with additional style-specific info.
     pub style: HeapStyle,
+
+    /// The index type for the heap.
+    pub index_type: Type,
 }
 
 /// Style of heap including style-specific information.
@@ -50,6 +53,10 @@ impl fmt::Display for HeapData {
             HeapStyle::Dynamic { bound_gv } => write!(f, ", bound {}", bound_gv)?,
             HeapStyle::Static { bound } => write!(f, ", bound {}", bound)?,
         }
-        write!(f, ", guard {}", self.guard_size)
+        write!(
+            f,
+            ", guard {}, index_type {}",
+            self.guard_size, self.index_type
+        )
     }
 }
diff --git a/lib/codegen/src/legalizer/globalvalue.rs b/lib/codegen/src/legalizer/globalvalue.rs
index 733d10fa92..61723af0d7 100644
--- a/lib/codegen/src/legalizer/globalvalue.rs
+++ b/lib/codegen/src/legalizer/globalvalue.rs
@@ -13,7 +13,7 @@ pub fn expand_global_value(
     inst: ir::Inst,
     func: &mut ir::Function,
     _cfg: &mut ControlFlowGraph,
-    _isa: &TargetIsa,
+    isa: &TargetIsa,
 ) {
     // Unpack the instruction.
     let gv = match func.dfg[inst] {
@@ -29,8 +29,12 @@ pub fn expand_global_value(
 
     match func.global_values[gv] {
         ir::GlobalValueData::VMContext { offset } => vmctx_addr(inst, func, offset.into()),
-        ir::GlobalValueData::Deref { base, offset } => deref_addr(inst, func, base, offset.into()),
-        ir::GlobalValueData::Sym { .. } => globalsym(inst, func, gv),
+        ir::GlobalValueData::Deref {
+            base,
+            offset,
+            memory_type,
+        } => deref_addr(inst, func, base, offset.into(), memory_type, isa),
+        ir::GlobalValueData::Sym { .. } => globalsym(inst, func, gv, isa),
     }
 }
 
@@ -46,11 +50,18 @@ fn vmctx_addr(inst: ir::Inst, func: &mut ir::Function, offset: i64) {
 }
 
 /// Expand a `global_value` instruction for a deref global.
-fn deref_addr(inst: ir::Inst, func: &mut ir::Function, base: ir::GlobalValue, offset: i64) {
+fn deref_addr(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    base: ir::GlobalValue,
+    offset: i64,
+    memory_type: ir::Type,
+    isa: &TargetIsa,
+) {
     // We need to load a pointer from the `base` global value, so insert a new `global_value`
     // instruction. This depends on the iterative legalization loop. Note that the IR verifier
     // detects any cycles in the `deref` globals.
-    let ptr_ty = func.dfg.value_type(func.dfg.first_result(inst));
+    let ptr_ty = isa.pointer_type();
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
 
@@ -59,12 +70,12 @@ fn deref_addr(inst: ir::Inst, func: &mut ir::Function, base: ir::GlobalValue, of
     // Deref globals are required to be accessible and aligned.
     mflags.set_notrap();
     mflags.set_aligned();
-    let base_ptr = pos.ins().load(ptr_ty, mflags, base_addr, 0);
-    pos.func.dfg.replace(inst).iadd_imm(base_ptr, offset);
+    let loaded = pos.ins().load(memory_type, mflags, base_addr, 0);
+    pos.func.dfg.replace(inst).iadd_imm(loaded, offset);
 }
 
 /// Expand a `global_value` instruction for a symbolic name global.
-fn globalsym(inst: ir::Inst, func: &mut ir::Function, gv: ir::GlobalValue) {
-    let ptr_ty = func.dfg.value_type(func.dfg.first_result(inst));
+fn globalsym(inst: ir::Inst, func: &mut ir::Function, gv: ir::GlobalValue, isa: &TargetIsa) {
+    let ptr_ty = isa.pointer_type();
     func.dfg.replace(inst).globalsym_addr(ptr_ty, gv);
 }
diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs
index 143a01e234..70e926920a 100644
--- a/lib/codegen/src/legalizer/heap.rs
+++ b/lib/codegen/src/legalizer/heap.rs
@@ -57,7 +57,7 @@ fn dynamic_addr(
     pos.use_srcloc(inst);
 
     // Start with the bounds check. Trap if `offset + access_size > bound`.
-    let bound = pos.ins().global_value(addr_ty, bound_gv);
+    let bound = pos.ins().global_value(offset_ty, bound_gv);
     let oob;
     if access_size == 1 {
         // `offset > bound - 1` is the same as `offset >= bound`.
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 2f9c1dbcb4..eed8bc2d5b 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -348,14 +348,33 @@ impl<'a> Verifier<'a> {
                 cur = base;
             }
 
-            if let ir::GlobalValueData::VMContext { .. } = self.func.global_values[cur] {
-                if self
-                    .func
-                    .special_param(ir::ArgumentPurpose::VMContext)
-                    .is_none()
-                {
-                    report!(errors, cur, "undeclared vmctx reference {}", cur);
+            match self.func.global_values[gv] {
+                ir::GlobalValueData::VMContext { .. } => {
+                    if self
+                        .func
+                        .special_param(ir::ArgumentPurpose::VMContext)
+                        .is_none()
+                    {
+                        report!(errors, gv, "undeclared vmctx reference {}", gv);
+                    }
                 }
+                ir::GlobalValueData::Deref { base, .. } => {
+                    if let Some(isa) = self.isa {
+                        let base_type = self.func.global_values[base].global_type(isa);
+                        let pointer_type = isa.pointer_type();
+                        if base_type != pointer_type {
+                            report!(
+                                errors,
+                                gv,
+                                "deref base {} has type {}, which is not the pointer type {}",
+                                base,
+                                base_type,
+                                pointer_type
+                            );
+                        }
+                    }
+                }
+                _ => {}
             }
         }
 
@@ -363,6 +382,57 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
+    fn verify_heaps(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
+        if let Some(isa) = self.isa {
+            for (heap, heap_data) in &self.func.heaps {
+                let base = heap_data.base;
+                if !self.func.global_values.is_valid(base) {
+                    return nonfatal!(errors, heap, "invalid base global value {}", base);
+                }
+
+                let pointer_type = isa.pointer_type();
+                let base_type = self.func.global_values[base].global_type(isa);
+                if base_type != pointer_type {
+                    report!(
+                        errors,
+                        heap,
+                        "heap base has type {}, which is not the pointer type {}",
+                        base_type,
+                        pointer_type
+                    );
+                }
+
+                match heap_data.style {
+                    ir::HeapStyle::Dynamic { bound_gv, .. } => {
+                        if !self.func.global_values.is_valid(bound_gv) {
+                            return nonfatal!(
+                                errors,
+                                heap,
+                                "invalid bound global value {}",
+                                bound_gv
+                            );
+                        }
+
+                        let index_type = heap_data.index_type;
+                        let bound_type = self.func.global_values[bound_gv].global_type(isa);
+                        if index_type != bound_type {
+                            report!(
+                                errors,
+                                heap,
+                                "heap index type {} differs from the type of its bound, {}",
+                                index_type,
+                                bound_type
+                            );
+                        }
+                    }
+                    _ => {}
+                }
+            }
+        }
+
+        Ok(())
+    }
+
     fn ebb_integrity(
         &self,
         ebb: Ebb,
@@ -1255,51 +1325,82 @@ impl<'a> Verifier<'a> {
         ctrl_type: Type,
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
-        if let ir::InstructionData::Unary { opcode, arg } = self.func.dfg[inst] {
-            let arg_type = self.func.dfg.value_type(arg);
-            match opcode {
-                Opcode::Bextend | Opcode::Uextend | Opcode::Sextend | Opcode::Fpromote => {
-                    if arg_type.lane_count() != ctrl_type.lane_count() {
-                        return nonfatal!(
-                            errors,
-                            inst,
-                            "input {} and output {} must have same number of lanes",
-                            arg_type,
-                            ctrl_type
-                        );
+        match self.func.dfg[inst] {
+            ir::InstructionData::Unary { opcode, arg } => {
+                let arg_type = self.func.dfg.value_type(arg);
+                match opcode {
+                    Opcode::Bextend | Opcode::Uextend | Opcode::Sextend | Opcode::Fpromote => {
+                        if arg_type.lane_count() != ctrl_type.lane_count() {
+                            return nonfatal!(
+                                errors,
+                                inst,
+                                "input {} and output {} must have same number of lanes",
+                                arg_type,
+                                ctrl_type
+                            );
+                        }
+                        if arg_type.lane_bits() >= ctrl_type.lane_bits() {
+                            return nonfatal!(
+                                errors,
+                                inst,
+                                "input {} must be smaller than output {}",
+                                arg_type,
+                                ctrl_type
+                            );
+                        }
                     }
-                    if arg_type.lane_bits() >= ctrl_type.lane_bits() {
-                        return nonfatal!(
-                            errors,
-                            inst,
-                            "input {} must be smaller than output {}",
-                            arg_type,
-                            ctrl_type
-                        );
+                    Opcode::Breduce | Opcode::Ireduce | Opcode::Fdemote => {
+                        if arg_type.lane_count() != ctrl_type.lane_count() {
+                            return nonfatal!(
+                                errors,
+                                inst,
+                                "input {} and output {} must have same number of lanes",
+                                arg_type,
+                                ctrl_type
+                            );
+                        }
+                        if arg_type.lane_bits() <= ctrl_type.lane_bits() {
+                            return nonfatal!(
+                                errors,
+                                inst,
+                                "input {} must be larger than output {}",
+                                arg_type,
+                                ctrl_type
+                            );
+                        }
                     }
+                    _ => {}
                 }
-                Opcode::Breduce | Opcode::Ireduce | Opcode::Fdemote => {
-                    if arg_type.lane_count() != ctrl_type.lane_count() {
-                        return nonfatal!(
-                            errors,
-                            inst,
-                            "input {} and output {} must have same number of lanes",
-                            arg_type,
-                            ctrl_type
-                        );
-                    }
-                    if arg_type.lane_bits() <= ctrl_type.lane_bits() {
-                        return nonfatal!(
-                            errors,
-                            inst,
-                            "input {} must be larger than output {}",
-                            arg_type,
-                            ctrl_type
-                        );
-                    }
-                }
-                _ => {}
             }
+            ir::InstructionData::HeapAddr { heap, arg, .. } => {
+                let index_type = self.func.dfg.value_type(arg);
+                let heap_index_type = self.func.heaps[heap].index_type;
+                if index_type != heap_index_type {
+                    return nonfatal!(
+                        errors,
+                        inst,
+                        "index type {} differs from heap index type {}",
+                        index_type,
+                        heap_index_type
+                    );
+                }
+            }
+            ir::InstructionData::UnaryGlobalValue { global_value, .. } => {
+                if let Some(isa) = self.isa {
+                    let inst_type = self.func.dfg.value_type(self.func.dfg.first_result(inst));
+                    let global_type = self.func.global_values[global_value].global_type(isa);
+                    if inst_type != global_type {
+                        return nonfatal!(
+                        errors,
+                        inst,
+                        "global_value instruction with type {} references global value with type {}",
+                        inst_type,
+                        global_type
+                    );
+                    }
+                }
+            }
+            _ => {}
         }
         Ok(())
     }
@@ -1509,6 +1610,7 @@ impl<'a> Verifier<'a> {
 
     pub fn run(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         self.verify_global_values(errors)?;
+        self.verify_heaps(errors)?;
         self.typecheck_entry_block_params(errors)?;
 
         for ebb in self.func.layout.ebbs() {
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index c5bbe47f8f..b7980c34be 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -174,6 +174,7 @@ impl<'a> Context<'a> {
                 style: HeapStyle::Static {
                     bound: Imm64::new(0),
                 },
+                index_type: VOID,
             });
         }
         self.function.heaps[heap] = data;
@@ -1128,7 +1129,13 @@ impl<'a> Parser<'a> {
                 let base = self.match_gv("expected global value: gv«n»")?;
                 self.match_token(Token::RPar, "expected ')' in 'deref' global value decl")?;
                 let offset = self.optional_offset32()?;
-                GlobalValueData::Deref { base, offset }
+                self.match_token(Token::Colon, "expected ':' in 'deref' global value decl")?;
+                let memory_type = self.match_type("expected deref type")?;
+                GlobalValueData::Deref {
+                    base,
+                    offset,
+                    memory_type,
+                }
             }
             "globalsym" => {
                 let colocated = self.optional(Token::Identifier("colocated"));
@@ -1177,6 +1184,7 @@ impl<'a> Parser<'a> {
             min_size: 0.into(),
             guard_size: 0.into(),
             style: HeapStyle::Static { bound: 0.into() },
+            index_type: ir::types::I32,
         };
 
         // heap-desc ::= heap-style heap-base * { "," heap-attr }
@@ -1199,6 +1207,9 @@ impl<'a> Parser<'a> {
                 "guard" => {
                     data.guard_size = self.match_imm64("expected integer guard size")?;
                 }
+                "index_type" => {
+                    data.index_type = self.match_type("expected index type")?;
+                }
                 t => return err!(self.loc, "unknown heap attribute '{}'", t),
             }
         }
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 5077612529..0010a41beb 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -177,6 +177,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         let gv = func.create_global_value(ir::GlobalValueData::Deref {
             base: addr,
             offset: 0.into(),
+            memory_type: self.pointer_type(),
         });
 
         func.create_heap(ir::HeapData {
@@ -186,6 +187,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
             style: ir::HeapStyle::Static {
                 bound: 0x1_0000_0000.into(),
             },
+            index_type: I32,
         })
     }
 
@@ -196,12 +198,14 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         let base_gv = func.create_global_value(ir::GlobalValueData::Deref {
             base: base_gv_addr,
             offset: 0.into(),
+            memory_type: self.pointer_type(),
         });
         let bound_gv_addr =
             func.create_global_value(ir::GlobalValueData::VMContext { offset: 0.into() });
         let bound_gv = func.create_global_value(ir::GlobalValueData::Deref {
             base: bound_gv_addr,
             offset: 0.into(),
+            memory_type: self.pointer_type(),
         });
 
         func.create_table(ir::TableData {

From 0d24641f2188bb1f7f514ffd1d663e5ced1ff563 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 13:48:54 -0700
Subject: [PATCH 2031/3084] Fix a verifier test failure.

This test was accidentally relying on the bug that #485 fixed.
---
 cranelift/filetests/verifier/heap.clif | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/filetests/verifier/heap.clif b/cranelift/filetests/verifier/heap.clif
index f3d1fee2f8..68d13e66f2 100644
--- a/cranelift/filetests/verifier/heap.clif
+++ b/cranelift/filetests/verifier/heap.clif
@@ -29,7 +29,7 @@ ebb0(v0: i64):
 function %heap_bound_type(i64 vmctx) {
     gv0 = vmctx+0
     gv1 = deref(gv0): i16
-    heap0 = dynamic gv1, bound gv1, guard 0x1000, index_type i32 ; error: heap index type i32 differs from the type of its bound, i16
+    heap0 = dynamic gv0, bound gv1, guard 0x1000, index_type i32 ; error: heap index type i32 differs from the type of its bound, i16
 
 ebb0(v0: i64):
     return

From 8e2d01a675f6bda77ed3876eaa4ecd1f4a309e3e Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 13:57:34 -0700
Subject: [PATCH 2032/3084] Add an `index_type` field to `Table`.

This parallels the `index_type` field in `Heap`.
---
 .../filetests/isa/x86/legalize-table.clif     |  2 +-
 .../filetests/isa/x86/legalize-tables.clif    | 68 +++++++++++++++++++
 cranelift/filetests/verifier/table.clif       | 46 +++++++++++++
 lib/codegen/src/ir/table.rs                   | 10 ++-
 lib/codegen/src/legalizer/table.rs            |  2 +-
 lib/codegen/src/verifier/mod.rs               | 56 +++++++++++++++
 lib/codegen/src/write.rs                      |  5 ++
 lib/reader/src/parser.rs                      |  7 ++
 lib/wasm/src/environ/dummy.rs                 |  1 +
 9 files changed, 192 insertions(+), 5 deletions(-)
 create mode 100644 cranelift/filetests/isa/x86/legalize-tables.clif
 create mode 100644 cranelift/filetests/verifier/table.clif

diff --git a/cranelift/filetests/isa/x86/legalize-table.clif b/cranelift/filetests/isa/x86/legalize-table.clif
index afad7d8e21..f86913115d 100644
--- a/cranelift/filetests/isa/x86/legalize-table.clif
+++ b/cranelift/filetests/isa/x86/legalize-table.clif
@@ -8,7 +8,7 @@ target x86_64
 function %test0(i64 vmctx, i64) -> i64 {
     gv0 = vmctx+12
     gv1 = vmctx+14
-    table0 = dynamic gv0, min 20, bound gv1, element_size 4
+    table0 = dynamic gv0, min 20, bound gv1, element_size 4, index_type i64
 
 ebb0(v0: i64, v1: i64):
     v2 = table_addr.i64 table0, v1, +3
diff --git a/cranelift/filetests/isa/x86/legalize-tables.clif b/cranelift/filetests/isa/x86/legalize-tables.clif
new file mode 100644
index 0000000000..a9882f3cf9
--- /dev/null
+++ b/cranelift/filetests/isa/x86/legalize-tables.clif
@@ -0,0 +1,68 @@
+test legalizer
+target x86_64
+
+; Test legalization for various forms of table addresses.
+
+function %table_addrs(i32, i64, i64 vmctx) {
+    gv0 = vmctx+72
+    gv1 = vmctx+80
+    gv2 = vmctx+88
+    gv3 = deref(gv2): i32
+
+    table0 = dynamic gv0, min 0x1_0000, bound gv3, element_size 1, index_type i32
+    table1 = dynamic gv0, bound gv3, element_size 16, index_type i32
+    table2 = dynamic gv0, min 0x1_0000, bound gv1, element_size 1, index_type i64
+    table3 = dynamic gv0, bound gv1, element_size 16, index_type i64
+
+    ; check: table0 = dynamic gv0, min 0x0001_0000, bound gv3, element_size 1, index_type i32
+    ; check: table1 = dynamic gv0, min 0, bound gv3, element_size 16, index_type i32
+    ; check: table2 = dynamic gv0, min 0x0001_0000, bound gv1, element_size 1, index_type i64
+    ; check: table3 = dynamic gv0, min 0, bound gv1, element_size 16, index_type i64
+
+ebb0(v0: i32, v1: i64, v3: i64):
+    v4 = table_addr.i64 table0, v0, +0
+    ; check:         v12 = iadd_imm v3, 88
+    ; check:         v13 = load.i32 notrap aligned v12
+    ; check:         v8 = iadd_imm v13, 0
+    ; check:         v9 = icmp uge v0, v8
+    ; check:         brz v9, ebb1
+    ; check:         trap table_oob
+    ; check:     ebb1:
+    ; check:         v10 = uextend.i64 v0
+    ; check:         v11 = iadd_imm.i64 v3, 72
+    ; check:         v4 = iadd v11, v10
+
+    v5 = table_addr.i64 table1, v0, +0
+    ; check:         v19 = iadd_imm.i64 v3, 88
+    ; check:         v20 = load.i32 notrap aligned v19
+    ; check:         v14 = iadd_imm v20, 0
+    ; check:         v15 = icmp.i32 uge v0, v14
+    ; check:         brz v15, ebb2
+    ; check:         trap table_oob
+    ; check:     ebb2:
+    ; check:         v16 = uextend.i64 v0
+    ; check:         v17 = iadd_imm.i64 v3, 72
+    ; check:         v18 = ishl_imm v16, 4
+    ; check:         v5 = iadd v17, v18
+
+    v6 = table_addr.i64 table2, v1, +0
+    ; check:         v21 = iadd_imm.i64 v3, 80
+    ; check:         v22 = icmp.i64 uge v1, v21
+    ; check:         brz v22, ebb3
+    ; check:         trap table_oob
+    ; check:     ebb3:
+    ; check:         v23 = iadd_imm.i64 v3, 72
+    ; check:         v6 = iadd v23, v1
+
+    v7 = table_addr.i64 table3, v1, +0
+    ; check:         v24 = iadd_imm.i64 v3, 80
+    ; check:         v25 = icmp.i64 uge v1, v24
+    ; check:         brz v25, ebb4
+    ; check:         trap table_oob
+    ; check:     ebb4:
+    ; check:         v26 = iadd_imm.i64 v3, 72
+    ; check:         v27 = ishl_imm.i64 v1, 4
+    ; check:         v7 = iadd v26, v27
+
+    return
+}
diff --git a/cranelift/filetests/verifier/table.clif b/cranelift/filetests/verifier/table.clif
new file mode 100644
index 0000000000..10fdaaf87f
--- /dev/null
+++ b/cranelift/filetests/verifier/table.clif
@@ -0,0 +1,46 @@
+test verifier
+target x86_64
+
+function %table_base_type(i64 vmctx) {
+    gv0 = vmctx+0
+    gv1 = deref(gv0): i32
+    table0 = dynamic gv1, element_size 1, bound gv1, index_type i32 ; error: table base has type i32, which is not the pointer type i64
+
+ebb0(v0: i64):
+    return
+}
+
+function %invalid_base(i64 vmctx) {
+    gv0 = vmctx+0
+    table0 = dynamic gv1, bound gv0, element_size 1, index_type i64 ; error: invalid base global value gv1
+
+ebb0(v0: i64):
+    return
+}
+
+function %invalid_bound(i64 vmctx) {
+    gv0 = vmctx+0
+    table0 = dynamic gv0, bound gv1, element_size 1, index_type i64 ; error: invalid bound global value gv1
+
+ebb0(v0: i64):
+    return
+}
+
+function %table_bound_type(i64 vmctx) {
+    gv0 = vmctx+0
+    gv1 = deref(gv0): i16
+    table0 = dynamic gv0, bound gv1, element_size 1, index_type i32 ; error: table index type i32 differs from the type of its bound, i16
+
+ebb0(v0: i64):
+    return
+}
+
+function %table_addr_index_type(i64 vmctx, i64) {
+    gv0 = vmctx+0
+    gv1 = deref(gv0): i32
+    table0 = dynamic gv0, element_size 1, bound gv1, index_type i32
+
+ebb0(v0: i64, v1: i64):
+    v2 = table_addr.i64 table0, v1, +0; error: index type i64 differs from table index type i32
+    return
+}
diff --git a/lib/codegen/src/ir/table.rs b/lib/codegen/src/ir/table.rs
index edfe18aec7..5db2402793 100644
--- a/lib/codegen/src/ir/table.rs
+++ b/lib/codegen/src/ir/table.rs
@@ -1,7 +1,7 @@
 //! Tables.
 
 use ir::immediates::Imm64;
-use ir::GlobalValue;
+use ir::{GlobalValue, Type};
 use std::fmt;
 
 /// Information about a table declaration.
@@ -19,14 +19,18 @@ pub struct TableData {
 
     /// The size of a table element, in bytes.
     pub element_size: Imm64,
+
+    /// The index type for the table.
+    pub index_type: Type,
 }
 
 impl fmt::Display for TableData {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        f.write_str("dynamic")?;
         write!(
             f,
-            "{}, min {}, bound {}, element_size {}",
-            self.base_gv, self.min_size, self.bound_gv, self.element_size
+            " {}, min {}, bound {}, element_size {}, index_type {}",
+            self.base_gv, self.min_size, self.bound_gv, self.element_size, self.index_type
         )
     }
 }
diff --git a/lib/codegen/src/legalizer/table.rs b/lib/codegen/src/legalizer/table.rs
index fe017eb096..d7633780f2 100644
--- a/lib/codegen/src/legalizer/table.rs
+++ b/lib/codegen/src/legalizer/table.rs
@@ -49,7 +49,7 @@ fn dynamic_addr(
     pos.use_srcloc(inst);
 
     // Start with the bounds check. Trap if `index + 1 > bound`.
-    let bound = pos.ins().global_value(addr_ty, bound_gv);
+    let bound = pos.ins().global_value(index_ty, bound_gv);
 
     // `index > bound - 1` is the same as `index >= bound`.
     let oob = pos
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index eed8bc2d5b..33ec402449 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -433,6 +433,48 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
+    fn verify_tables(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
+        if let Some(isa) = self.isa {
+            for (table, table_data) in &self.func.tables {
+                let base = table_data.base_gv;
+                if !self.func.global_values.is_valid(base) {
+                    return nonfatal!(errors, table, "invalid base global value {}", base);
+                }
+
+                let pointer_type = isa.pointer_type();
+                let base_type = self.func.global_values[base].global_type(isa);
+                if base_type != pointer_type {
+                    report!(
+                        errors,
+                        table,
+                        "table base has type {}, which is not the pointer type {}",
+                        base_type,
+                        pointer_type
+                    );
+                }
+
+                let bound_gv = table_data.bound_gv;
+                if !self.func.global_values.is_valid(bound_gv) {
+                    return nonfatal!(errors, table, "invalid bound global value {}", bound_gv);
+                }
+
+                let index_type = table_data.index_type;
+                let bound_type = self.func.global_values[bound_gv].global_type(isa);
+                if index_type != bound_type {
+                    report!(
+                        errors,
+                        table,
+                        "table index type {} differs from the type of its bound, {}",
+                        index_type,
+                        bound_type
+                    );
+                }
+            }
+        }
+
+        Ok(())
+    }
+
     fn ebb_integrity(
         &self,
         ebb: Ebb,
@@ -1385,6 +1427,19 @@ impl<'a> Verifier<'a> {
                     );
                 }
             }
+            ir::InstructionData::TableAddr { table, arg, .. } => {
+                let index_type = self.func.dfg.value_type(arg);
+                let table_index_type = self.func.tables[table].index_type;
+                if index_type != table_index_type {
+                    return nonfatal!(
+                        errors,
+                        inst,
+                        "index type {} differs from table index type {}",
+                        index_type,
+                        table_index_type
+                    );
+                }
+            }
             ir::InstructionData::UnaryGlobalValue { global_value, .. } => {
                 if let Some(isa) = self.isa {
                     let inst_type = self.func.dfg.value_type(self.func.dfg.first_result(inst));
@@ -1611,6 +1666,7 @@ impl<'a> Verifier<'a> {
     pub fn run(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         self.verify_global_values(errors)?;
         self.verify_heaps(errors)?;
+        self.verify_tables(errors)?;
         self.typecheck_entry_block_params(errors)?;
 
         for ebb in self.func.layout.ebbs() {
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index 2d2fcf5621..71714727ae 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -46,6 +46,11 @@ pub trait FuncWriter {
             self.write_entity_definition(w, func, heap.into(), heap_data)?;
         }
 
+        for (table, table_data) in &func.tables {
+            any = true;
+            self.write_entity_definition(w, func, table.into(), table_data)?;
+        }
+
         // Write out all signatures before functions since function declarations can refer to
         // signatures.
         for (sig, sig_data) in &func.dfg.signatures {
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index b7980c34be..3f607f1d15 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -198,6 +198,7 @@ impl<'a> Context<'a> {
                 min_size: Imm64::new(0),
                 bound_gv: GlobalValue::reserved_value(),
                 element_size: Imm64::new(0),
+                index_type: VOID,
             });
         }
         self.function.tables[table] = data;
@@ -1161,6 +1162,7 @@ impl<'a> Parser<'a> {
     // heap-attr ::= "min" Imm64(bytes)
     //             | "bound" Imm64(bytes)
     //             | "guard" Imm64(bytes)
+    //             | "index_type" type
     //
     fn parse_heap_decl(&mut self) -> ParseResult<(Heap, HeapData)> {
         let heap = self.match_heap("expected heap number: heap«n»")?;
@@ -1230,6 +1232,7 @@ impl<'a> Parser<'a> {
     // table-attr ::= "min" Imm64(bytes)
     //              | "bound" Imm64(bytes)
     //              | "element_size" Imm64(bytes)
+    //              | "index_type" type
     //
     fn parse_table_decl(&mut self) -> ParseResult<(Table, TableData)> {
         let table = self.match_table("expected table number: table«n»")?;
@@ -1253,6 +1256,7 @@ impl<'a> Parser<'a> {
             min_size: 0.into(),
             bound_gv: GlobalValue::reserved_value(),
             element_size: 0.into(),
+            index_type: ir::types::I32,
         };
 
         // table-desc ::= * { "," table-attr }
@@ -1270,6 +1274,9 @@ impl<'a> Parser<'a> {
                 "element_size" => {
                     data.element_size = self.match_imm64("expected integer element size")?;
                 }
+                "index_type" => {
+                    data.index_type = self.match_type("expected index type")?;
+                }
                 t => return err!(self.loc, "unknown table attribute '{}'", t),
             }
         }
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 0010a41beb..767ae2c9d9 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -213,6 +213,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
             min_size: Imm64::new(0),
             bound_gv,
             element_size: Imm64::new(i64::from(self.pointer_bytes()) * 2),
+            index_type: I32,
         })
     }
 

From 9ada394d1184dcc2837451d56d3691f5129dc8e7 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 15:27:52 -0700
Subject: [PATCH 2033/3084] [SimpleJIT] When finalizing multiple functions,
 make them all executable at the end. (#474)

Add `publish()` function to cranelift-module's `Backend` trait, which
allows `finalize_all()` to defer making memory executable until it
has finished all of the patching it needs to do.
---
 lib/faerie/src/backend.rs    |  4 ++++
 lib/module/src/backend.rs    |  3 +++
 lib/module/src/module.rs     | 11 ++++++++++-
 lib/simplejit/src/backend.rs |  9 ++++++---
 4 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index 0d27ed490c..395a5c376c 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -281,6 +281,10 @@ impl Backend for FaerieBackend {
         // Nothing to do.
     }
 
+    fn publish(&mut self) {
+        // Nothing to do.
+    }
+
     fn finish(self) -> FaerieProduct {
         FaerieProduct {
             artifact: self.artifact,
diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs
index 44a2aa95af..f167126191 100644
--- a/lib/module/src/backend.rs
+++ b/lib/module/src/backend.rs
@@ -105,6 +105,9 @@ where
         namespace: &ModuleNamespace,
     ) -> Self::FinalizedData;
 
+    /// "Publish" all finalized functions and data objects to their ultimate destinations.
+    fn publish(&mut self);
+
     /// Consume this `Backend` and return a result. Some implementations may
     /// provide additional functionality through this result.
     fn finish(self) -> Self::Product;
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index cc89ad68c0..c15aa1ff58 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -600,6 +600,7 @@ where
             )
         };
         self.contents.functions[func].finalized = true;
+        self.backend.publish();
         output
     }
 
@@ -627,6 +628,7 @@ where
             )
         };
         self.contents.data_objects[data].finalized = true;
+        self.backend.publish();
         output
     }
 
@@ -646,6 +648,9 @@ where
                 );
             }
         }
+        for info in self.contents.functions.values_mut() {
+            info.finalized = true;
+        }
         for info in self.contents.data_objects.values() {
             if info.decl.linkage.is_definable() && !info.finalized {
                 self.backend.finalize_data(
@@ -658,12 +663,16 @@ where
                 );
             }
         }
+        for info in self.contents.data_objects.values_mut() {
+            info.finalized = true;
+        }
     }
 
     /// Consume the module and return the resulting `Product`. Some `Backend`
     /// implementations may provide additional functionality available after
     /// a `Module` is complete.
-    pub fn finish(self) -> B::Product {
+    pub fn finish(mut self) -> B::Product {
+        self.backend.publish();
         self.backend.finish()
     }
 }
diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index 2a23b93f0c..15a8985578 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -335,8 +335,6 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
             }
         }
 
-        // Now that we're done patching, make the memory executable.
-        self.code_memory.set_executable();
         func.code
     }
 
@@ -397,10 +395,15 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
             }
         }
 
-        self.readonly_memory.set_readonly();
         (data.storage, data.size)
     }
 
+    fn publish(&mut self) {
+        // Now that we're done patching, prepare the memory for execution!
+        self.readonly_memory.set_readonly();
+        self.code_memory.set_executable();
+    }
+
     /// SimpleJIT emits code and data into memory as it processes them, so it
     /// doesn't need to provide anything after the `Module` is complete.
     fn finish(self) -> () {}

From bdd1949b34b853ca15d86087b260e7ad25a2f493 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 14:34:11 -0700
Subject: [PATCH 2034/3084] Don't pass Copy objects by reference.

---
 lib/reader/src/parser.rs         | 4 ++--
 lib/serde/src/serde_clif_json.rs | 6 +++---
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 3f607f1d15..ee04728213 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -206,7 +206,7 @@ impl<'a> Context<'a> {
     }
 
     // Resolve a reference to a table.
-    fn check_table(&self, table: Table, loc: &Location) -> ParseResult<()> {
+    fn check_table(&self, table: Table, loc: Location) -> ParseResult<()> {
         if !self.map.contains_table(table) {
             err!(loc, "undefined table {}", table)
         } else {
@@ -2249,7 +2249,7 @@ impl<'a> Parser<'a> {
             }
             InstructionFormat::TableAddr => {
                 let table = self.match_table("expected table identifier")?;
-                ctx.check_table(table, &self.loc)?;
+                ctx.check_table(table, self.loc)?;
                 self.match_token(Token::Comma, "expected ',' between operands")?;
                 let arg = self.match_value("expected SSA value table address")?;
                 self.match_token(Token::Comma, "expected ',' between operands")?;
diff --git a/lib/serde/src/serde_clif_json.rs b/lib/serde/src/serde_clif_json.rs
index 52fdc6fca1..7ad03df646 100644
--- a/lib/serde/src/serde_clif_json.rs
+++ b/lib/serde/src/serde_clif_json.rs
@@ -706,9 +706,9 @@ pub fn populate_inst(func: &Function, ebb: Ebb) -> Vec {
 }
 
 /// Translating Ebb parameters into serializable parameters.
-pub fn populate_params(func: &Function, ebb: &Ebb) -> Vec {
+pub fn populate_params(func: &Function, ebb: Ebb) -> Vec {
     let mut ser_vec: Vec = Vec::new();
-    let parameters = func.dfg.ebb_params(*ebb);
+    let parameters = func.dfg.ebb_params(ebb);
     for param in parameters {
         ser_vec.push(param.to_string());
     }
@@ -727,7 +727,7 @@ pub fn populate_ebbs(func: &Function) -> Vec {
     let mut ebb_vec: Vec = Vec::new();
     for ebb in func.layout.ebbs() {
         let mut ser_ebb: SerEbb = SerEbb::new(ebb.to_string());
-        ser_ebb.params = populate_params(&func, &ebb);
+        ser_ebb.params = populate_params(&func, ebb);
         ser_ebb.insts = populate_inst(&func, ebb);
         ebb_vec.push(ser_ebb);
     }

From d2943ec32dd952f809191bc30bdf4dfe67aff6c8 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 16:10:39 -0700
Subject: [PATCH 2035/3084] Add a minimal SimpleJIT example program.

This minimally demonstrates usage of the API, and serves as a very small
testcase to test that the basic JIT mechanisms are working.
---
 lib/faerie/src/backend.rs                   |  8 +++
 lib/module/src/backend.rs                   |  6 +++
 lib/module/src/module.rs                    | 34 ++++++++++--
 lib/simplejit/Cargo.toml                    |  3 ++
 lib/simplejit/README.md                     |  4 ++
 lib/simplejit/examples/simplejit-minimal.rs | 58 +++++++++++++++++++++
 lib/simplejit/src/backend.rs                |  6 +++
 7 files changed, 114 insertions(+), 5 deletions(-)
 create mode 100644 lib/simplejit/examples/simplejit-minimal.rs

diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index 395a5c376c..7100bf1917 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -277,10 +277,18 @@ impl Backend for FaerieBackend {
         // Nothing to do.
     }
 
+    fn get_finalized_function(&self, _func: &FaerieCompiledFunction) {
+        // Nothing to do.
+    }
+
     fn finalize_data(&mut self, _data: &FaerieCompiledData, _namespace: &ModuleNamespace) {
         // Nothing to do.
     }
 
+    fn get_finalized_data(&self, _data: &FaerieCompiledData) {
+        // Nothing to do.
+    }
+
     fn publish(&mut self) {
         // Nothing to do.
     }
diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs
index f167126191..42e77acb2f 100644
--- a/lib/module/src/backend.rs
+++ b/lib/module/src/backend.rs
@@ -97,6 +97,9 @@ where
         namespace: &ModuleNamespace,
     ) -> Self::FinalizedFunction;
 
+    /// Return the finalized artifact from the backend, if relevant.
+    fn get_finalized_function(&self, func: &Self::CompiledFunction) -> Self::FinalizedFunction;
+
     /// Perform all outstanding relocations on the given data object. This requires all
     /// `Local` and `Export` entities referenced to be defined.
     fn finalize_data(
@@ -105,6 +108,9 @@ where
         namespace: &ModuleNamespace,
     ) -> Self::FinalizedData;
 
+    /// Return the finalized artifact from the backend, if relevant.
+    fn get_finalized_data(&self, data: &Self::CompiledData) -> Self::FinalizedData;
+
     /// "Publish" all finalized functions and data objects to their ultimate destinations.
     fn publish(&mut self);
 
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index c15aa1ff58..91faef517e 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -581,7 +581,7 @@ where
     ///
     /// # Panics
     ///
-    /// When the function has already been finalized this panics
+    /// When the function has already been finalized this panics.
     pub fn finalize_function(&mut self, func: FuncId) -> B::FinalizedFunction {
         let output = {
             let info = &self.contents.functions[func];
@@ -604,12 +604,23 @@ where
         output
     }
 
+    /// Return the finalized artifact from the backend, if it provides one.
+    pub fn get_finalized_function(&mut self, func: FuncId) -> B::FinalizedFunction {
+        let info = &self.contents.functions[func];
+        debug_assert!(info.finalized, "data object not yet finalized");
+        self.backend.get_finalized_function(
+            info.compiled
+                .as_ref()
+                .expect("function must be compiled before it can be finalized"),
+        )
+    }
+
     /// Perform all outstanding relocations on the given data object. This requires all
     /// `Local` and `Export` entities referenced to be defined.
     ///
     /// # Panics
     ///
-    /// When the data object has already been finalized this panics
+    /// When the data object has already been finalized this panics.
     pub fn finalize_data(&mut self, data: DataId) -> B::FinalizedData {
         let output = {
             let info = &self.contents.data_objects[data];
@@ -632,8 +643,21 @@ where
         output
     }
 
+    /// Return the finalized artifact from the backend, if it provides one.
+    pub fn get_finalized_data(&mut self, data: DataId) -> B::FinalizedData {
+        let info = &self.contents.data_objects[data];
+        debug_assert!(info.finalized, "data object not yet finalized");
+        self.backend.get_finalized_data(
+            info.compiled
+                .as_ref()
+                .expect("data object must be compiled before it can be finalized"),
+        )
+    }
+
     /// Finalize all functions and data objects. Note that this doesn't return the
-    /// final artifacts returned from `finalize_function` or `finalize_data`.
+    /// final artifacts returned from `finalize_function` or `finalize_data`. Use
+    /// `get_finalized_function` and `get_finalized_data` to obtain the final
+    /// artifacts.
     pub fn finalize_all(&mut self) {
         // TODO: Could we use something like `into_iter()` here?
         for info in self.contents.functions.values() {
@@ -666,13 +690,13 @@ where
         for info in self.contents.data_objects.values_mut() {
             info.finalized = true;
         }
+        self.backend.publish();
     }
 
     /// Consume the module and return the resulting `Product`. Some `Backend`
     /// implementations may provide additional functionality available after
     /// a `Module` is complete.
-    pub fn finish(mut self) -> B::Product {
-        self.backend.publish();
+    pub fn finish(self) -> B::Product {
         self.backend.finish()
     }
 }
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 4ed87adf58..799fe4ae03 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -21,6 +21,9 @@ target-lexicon = { version = "0.0.3", default-features = false }
 [target.'cfg(target_os = "windows")'.dependencies]
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
+[dev-dependencies]
+cranelift = { path = "../umbrella", version = "0.19.0", default-features = false }
+
 [features]
 default = ["std"]
 std = ["libc/use_std", "cranelift-codegen/std", "cranelift-module/std", "cranelift-native/std", "target-lexicon/std"]
diff --git a/lib/simplejit/README.md b/lib/simplejit/README.md
index d5f2cecc37..3d31bb98e9 100644
--- a/lib/simplejit/README.md
+++ b/lib/simplejit/README.md
@@ -2,3 +2,7 @@ This crate provides a simple JIT library that uses
 [Cranelift](https://crates.io/crates/cranelift).
 
 This crate is extremely experimental.
+
+See the [example program] for a brief overview of how to use this.
+
+[example program]: https://github.com/CraneStation/cranelift/tree/master/lib/simplejit/examples/simplejit-minimal.rs
diff --git a/lib/simplejit/examples/simplejit-minimal.rs b/lib/simplejit/examples/simplejit-minimal.rs
new file mode 100644
index 0000000000..054204882e
--- /dev/null
+++ b/lib/simplejit/examples/simplejit-minimal.rs
@@ -0,0 +1,58 @@
+extern crate cranelift;
+extern crate cranelift_module;
+extern crate cranelift_simplejit;
+
+use cranelift::prelude::*;
+use cranelift_module::{Linkage, Module};
+use cranelift_simplejit::{SimpleJITBackend, SimpleJITBuilder};
+use std::mem;
+
+fn main() {
+    let mut module: Module = Module::new(SimpleJITBuilder::new());
+    let mut ctx = module.make_context();
+    let mut func_ctx = FunctionBuilderContext::new();
+    let sig = module.make_signature();
+
+    let func_a = module.declare_function("a", Linkage::Local, &sig).unwrap();
+    let func_b = module.declare_function("b", Linkage::Local, &sig).unwrap();
+
+    ctx.func.name = ExternalName::user(0, func_a.index() as u32);
+    {
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
+        let ebb = bcx.create_ebb();
+
+        bcx.switch_to_block(ebb);
+        bcx.ins().return_(&[]);
+        bcx.seal_all_blocks();
+        bcx.finalize();
+    }
+    module.define_function(func_a, &mut ctx).unwrap();
+    module.clear_context(&mut ctx);
+
+    ctx.func.name = ExternalName::user(0, func_b.index() as u32);
+    {
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
+        let ebb = bcx.create_ebb();
+
+        bcx.switch_to_block(ebb);
+        let local_func = module.declare_func_in_func(func_a, &mut bcx.func);
+        bcx.ins().call(local_func, &[]);
+        bcx.ins().return_(&[]);
+        bcx.seal_all_blocks();
+        bcx.finalize();
+    }
+    module.define_function(func_b, &mut ctx).unwrap();
+    module.clear_context(&mut ctx);
+
+    // Perform linking.
+    module.finalize_all();
+
+    // Get a raw pointer to the generated code.
+    let code_b = module.get_finalized_function(func_b);
+
+    // Cast it to a rust function pointer type.
+    let ptr_b = unsafe { mem::transmute::<_, fn()>(code_b) };
+
+    // Call it!
+    ptr_b();
+}
diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index 15a8985578..fa3dbdc0fc 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -334,7 +334,10 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
                 _ => unimplemented!(),
             }
         }
+        func.code
+    }
 
+    fn get_finalized_function(&self, func: &Self::CompiledFunction) -> Self::FinalizedFunction {
         func.code
     }
 
@@ -394,7 +397,10 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
                 }
             }
         }
+        (data.storage, data.size)
+    }
 
+    fn get_finalized_data(&self, data: &Self::CompiledData) -> Self::FinalizedData {
         (data.storage, data.size)
     }
 

From c23bfdaa91cb56d19e038cf068413297f69d3126 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 16:24:10 -0700
Subject: [PATCH 2036/3084] Minor code simplification.

---
 lib/codegen/meta-python/gen_settings.py | 22 ++++++++--------------
 1 file changed, 8 insertions(+), 14 deletions(-)

diff --git a/lib/codegen/meta-python/gen_settings.py b/lib/codegen/meta-python/gen_settings.py
index 01335803d4..287f1a2a39 100644
--- a/lib/codegen/meta-python/gen_settings.py
+++ b/lib/codegen/meta-python/gen_settings.py
@@ -271,23 +271,17 @@ def gen_constructor(sgrp, parent, fmt):
         with fmt.indented(
                 'pub fn new({}) -> Self {{'.format(args), '}'):
             fmt.line('let bvec = builder.state_for("{}");'.format(sgrp.name))
-            fmt.line('let mut bytes = [0; {}];'.format(sgrp.byte_size()))
             fmt.line(
-                'debug_assert_eq!(bvec.len(), {});'.format(sgrp.settings_size))
-            with fmt.indented(
-                    'for (i, b) in bvec.iter().enumerate() {', '}'):
-                fmt.line('bytes[i] = *b;')
-
-            # Stop here without predicates.
-            if len(sgrp.predicate_number) == sgrp.boolean_settings:
-                fmt.line('Self { bytes }')
-                return
+                'let mut {} = Self {{ bytes: [0; {}] }};'
+                .format(sgrp.name, sgrp.byte_size()))
+            fmt.line(
+                'debug_assert_eq!(bvec.len(), {});'
+                .format(sgrp.settings_size))
+            fmt.line(
+                '{}.bytes[0..{}].copy_from_slice(&bvec);'
+                .format(sgrp.name, sgrp.settings_size))
 
             # Now compute the predicates.
-            fmt.line(
-                    'let mut {} = Self {{ bytes }};'
-                    .format(sgrp.name))
-
             for pred, number in sgrp.predicate_number.items():
                 # Don't compute our own settings.
                 if number < sgrp.boolean_settings:

From 0842825c385a4cf27bb4e1a3aa3f59ab86678d1a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 16:30:51 -0700
Subject: [PATCH 2037/3084] Minor code simplification.

---
 lib/wasm/src/environ/dummy.rs | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 767ae2c9d9..e0c972c27d 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -16,7 +16,6 @@ use translation_utils::{
     DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table,
     TableIndex,
 };
-use wasmparser;
 
 /// Compute a `ir::ExternalName` for a given wasm function index.
 fn get_func_name(func_index: FuncIndex) -> ir::ExternalName {
@@ -438,9 +437,8 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
             let name = get_func_name(func_index);
             let sig = func_environ.vmctx_sig(self.get_func_type(func_index));
             let mut func = ir::Function::with_name_signature(name, sig);
-            let reader = wasmparser::BinaryReader::new(body_bytes);
             self.trans
-                .translate_from_reader(reader, &mut func, &mut func_environ)?;
+                .translate(body_bytes, &mut func, &mut func_environ)?;
             func
         };
         self.func_bytecode_sizes.push(body_bytes.len());

From 7fa0a387933c9d5f7adf35d5c43523365c147d5c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 16:37:52 -0700
Subject: [PATCH 2038/3084] Bump version to 0.20.0

---
 cranelift/Cargo.toml        | 26 +++++++++++++-------------
 cranelift/publish-all.sh    |  2 +-
 lib/bforest/Cargo.toml      |  4 ++--
 lib/codegen/Cargo.toml      |  8 ++++----
 lib/codegen/meta/Cargo.toml |  2 +-
 lib/entity/Cargo.toml       |  2 +-
 lib/faerie/Cargo.toml       |  6 +++---
 lib/filetests/Cargo.toml    |  6 +++---
 lib/frontend/Cargo.toml     |  4 ++--
 lib/module/Cargo.toml       |  6 +++---
 lib/native/Cargo.toml       |  4 ++--
 lib/reader/Cargo.toml       |  4 ++--
 lib/serde/Cargo.toml        |  6 +++---
 lib/simplejit/Cargo.toml    | 10 +++++-----
 lib/umbrella/Cargo.toml     |  6 +++---
 lib/wasm/Cargo.toml         |  8 ++++----
 16 files changed, 52 insertions(+), 52 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index e132bf3e0e..86111cf8f5 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.19.0"
+version = "0.20.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,18 +14,18 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.19.0" }
-cranelift-entity = { path = "lib/entity", version = "0.19.0" }
-cranelift-reader = { path = "lib/reader", version = "0.19.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.19.0" }
-cranelift-serde = { path = "lib/serde", version = "0.19.0", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.19.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.19.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.19.0" }
-cranelift-module = { path = "lib/module", version = "0.19.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.19.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.19.0" }
-cranelift = { path = "lib/umbrella", version = "0.19.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.20.0" }
+cranelift-entity = { path = "lib/entity", version = "0.20.0" }
+cranelift-reader = { path = "lib/reader", version = "0.20.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.20.0" }
+cranelift-serde = { path = "lib/serde", version = "0.20.0", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.20.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.20.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.20.0" }
+cranelift-module = { path = "lib/module", version = "0.20.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.20.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.20.0" }
+cranelift = { path = "lib/umbrella", version = "0.20.0" }
 filecheck = "0.3.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 7e0b6bb078..0fa8c0ab76 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.19.0"
+version="0.20.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index 72f146adb9..60384aa950 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.19.0"
+version = "0.20.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 keywords = ["btree", "forest", "set", "map"]
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.19.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.20.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 1e55075751..6d01017994 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.19.0"
+version = "0.20.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.19.0", default-features = false }
-cranelift-bforest = { path = "../bforest", version = "0.19.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.20.0", default-features = false }
+cranelift-bforest = { path = "../bforest", version = "0.20.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -25,7 +25,7 @@ log = { version = "0.4.4", default-features = false, features = ["release_max_le
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.19.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.20.0" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 8f9837c2d4..4a72b3e19e 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.19.0"
+version = "0.20.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index 6bc4a928bf..55706b934c 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.19.0"
+version = "0.20.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 320a8fae6e..6a4bb2343e 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.19.0"
+version = "0.20.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.19.0" }
-cranelift-module = { path = "../module", version = "0.19.0" }
+cranelift-codegen = { path = "../codegen", version = "0.20.0" }
+cranelift-module = { path = "../module", version = "0.20.0" }
 faerie = "0.5.0"
 goblin = "0.0.17"
 failure = "0.1.2"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index d74e5efa0a..ca03057505 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.19.0"
+version = "0.20.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,8 +9,8 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.19.0" }
-cranelift-reader = { path = "../reader", version = "0.19.0" }
+cranelift-codegen = { path = "../codegen", version = "0.20.0" }
+cranelift-reader = { path = "../reader", version = "0.20.0" }
 file-per-thread-logger = "0.1.1"
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index d0b84b6e3a..833baa8399 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.19.0"
+version = "0.20.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index ef7427f8b9..b11955802a 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.19.0"
+version = "0.20.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.19.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.20.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 94e4c789f4..362d9f5286 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.19.0"
+version = "0.20.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 70edf52930..c5c308aed9 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.19.0"
+version = "0.20.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.19.0" }
+cranelift-codegen = { path = "../codegen", version = "0.20.0" }
 target-lexicon = "0.0.3"
 
 [badges]
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index d4e66b2868..756d403b19 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.19.0"
+version = "0.20.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
-cranelift-reader = { path = "../reader", version = "0.19.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
+cranelift-reader = { path = "../reader", version = "0.20.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 799fe4ae03..0f84030f21 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.19.0"
+version = "0.20.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.19.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.19.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.20.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.20.0", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
@@ -22,7 +22,7 @@ target-lexicon = { version = "0.0.3", default-features = false }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../umbrella", version = "0.19.0", default-features = false }
+cranelift = { path = "../umbrella", version = "0.20.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index e7530aa24e..14589c0992 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.19.0"
+version = "0.20.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,8 +11,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.19.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.20.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 3681537c51..e30f51f4ed 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.19.0"
+version = "0.20.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,9 +11,9 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.2", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.19.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.19.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.19.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.20.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.20.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From fcd859a45e6c7bf782db0f7416f6d5c29eeff96d Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 17:05:26 -0700
Subject: [PATCH 2039/3084] Make FuncIndex and DefinedFuncIndex implement the
 Debug trait.

---
 lib/wasm/src/translation_utils.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index 752256f3f1..5d40539720 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -4,12 +4,12 @@ use std::u32;
 use wasmparser;
 
 /// Index type of a function (imported or defined) inside the WebAssembly module.
-#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
 pub struct FuncIndex(u32);
 entity_impl!(FuncIndex);
 
 /// Index type of a defined function inside the WebAssembly module.
-#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
 pub struct DefinedFuncIndex(u32);
 entity_impl!(DefinedFuncIndex);
 

From f834afb5f6c4d59a97482cf424bfabf7eee4db9b Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 17:18:36 -0700
Subject: [PATCH 2040/3084] Bump cranelift-wasm version to 0.20.1.

---
 cranelift/Cargo.toml | 2 +-
 lib/wasm/Cargo.toml  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 86111cf8f5..1bd2dd33ee 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -19,7 +19,7 @@ cranelift-entity = { path = "lib/entity", version = "0.20.0" }
 cranelift-reader = { path = "lib/reader", version = "0.20.0" }
 cranelift-frontend = { path = "lib/frontend", version = "0.20.0" }
 cranelift-serde = { path = "lib/serde", version = "0.20.0", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.20.0", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.20.1", optional = true }
 cranelift-native = { path = "lib/native", version = "0.20.0" }
 cranelift-filetests = { path = "lib/filetests", version = "0.20.0" }
 cranelift-module = { path = "lib/module", version = "0.20.0" }
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index e30f51f4ed..2f765068a3 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.20.0"
+version = "0.20.1"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"

From 1affd5eae710589273c2b51d033a5cb025230b72 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 20:19:01 -0700
Subject: [PATCH 2041/3084] Add Deref and DerefMut implementations for
 PrimaryMap.

---
 lib/entity/src/primary.rs | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)

diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs
index 07e00cb884..d071373180 100644
--- a/lib/entity/src/primary.rs
+++ b/lib/entity/src/primary.rs
@@ -1,6 +1,6 @@
 //! Densely numbered entity references as mapping keys.
 use std::marker::PhantomData;
-use std::ops::{Index, IndexMut};
+use std::ops::{Deref, DerefMut, Index, IndexMut};
 use std::slice;
 use std::vec::Vec;
 use {EntityRef, Iter, IterMut, Keys};
@@ -145,6 +145,26 @@ where
     }
 }
 
+impl Deref for PrimaryMap
+where
+    K: EntityRef,
+{
+    type Target = [V];
+
+    fn deref(&self) -> &Self::Target {
+        &self.elems
+    }
+}
+
+impl DerefMut for PrimaryMap
+where
+    K: EntityRef,
+{
+    fn deref_mut(&mut self) -> &mut [V] {
+        &mut self.elems
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -322,4 +342,12 @@ mod tests {
         }
     }
 
+    #[test]
+    fn deref() {
+        let mut m = PrimaryMap::::new();
+        let _: &[isize] = m.as_ref();
+        let _: &mut [isize] = m.as_mut();
+        let _: &[isize] = &m;
+        let _: &mut [isize] = &mut m;
+    }
 }

From c836a96e30442a9a19f314ddcdd1353a4ede5e49 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 20:19:09 -0700
Subject: [PATCH 2042/3084] Bump cranelift-entity version to 0.20.1.

---
 cranelift/Cargo.toml   | 2 +-
 lib/bforest/Cargo.toml | 2 +-
 lib/codegen/Cargo.toml | 2 +-
 lib/entity/Cargo.toml  | 2 +-
 lib/module/Cargo.toml  | 2 +-
 lib/wasm/Cargo.toml    | 2 +-
 6 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 1bd2dd33ee..f154579a9f 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -15,7 +15,7 @@ path = "src/clif-util.rs"
 [dependencies]
 cfg-if = "0.1"
 cranelift-codegen = { path = "lib/codegen", version = "0.20.0" }
-cranelift-entity = { path = "lib/entity", version = "0.20.0" }
+cranelift-entity = { path = "lib/entity", version = "0.20.1" }
 cranelift-reader = { path = "lib/reader", version = "0.20.0" }
 cranelift-frontend = { path = "lib/frontend", version = "0.20.0" }
 cranelift-serde = { path = "lib/serde", version = "0.20.0", optional = true }
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index 60384aa950..b68d9e9ea2 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -11,7 +11,7 @@ readme = "README.md"
 keywords = ["btree", "forest", "set", "map"]
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.20.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.20.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 6d01017994..9ec8ea6260 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -12,7 +12,7 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.20.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.20.1", default-features = false }
 cranelift-bforest = { path = "../bforest", version = "0.20.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index 55706b934c..c74e605f76 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.20.0"
+version = "0.20.1"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index b11955802a..26f7a621af 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -11,7 +11,7 @@ readme = "README.md"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.20.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.20.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 2f765068a3..dfe3f18388 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -12,7 +12,7 @@ keywords = ["webassembly", "wasm"]
 [dependencies]
 wasmparser = { version = "0.17.2", default-features = false }
 cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.20.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.20.1", default-features = false }
 cranelift-frontend = { path = "../frontend", version = "0.20.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }

From 67b7a8594a0605178d2c2f82242f3255b61862b9 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 28 Aug 2018 20:51:15 -0700
Subject: [PATCH 2043/3084] Implementing Deref in PrimaryMap turns out to be
 error-prone, so remove it.

One of the big advantages of PrimaryMap is that it protects against
using the wrong indices via a distinct index type. A Deref trait that
returns a plain slice would accept other indices. Add a comment
explaining this.
---
 lib/entity/src/primary.rs | 36 ++++++------------------------------
 1 file changed, 6 insertions(+), 30 deletions(-)

diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs
index d071373180..0902621c22 100644
--- a/lib/entity/src/primary.rs
+++ b/lib/entity/src/primary.rs
@@ -1,6 +1,6 @@
 //! Densely numbered entity references as mapping keys.
 use std::marker::PhantomData;
-use std::ops::{Deref, DerefMut, Index, IndexMut};
+use std::ops::{Index, IndexMut};
 use std::slice;
 use std::vec::Vec;
 use {EntityRef, Iter, IterMut, Keys};
@@ -14,6 +14,11 @@ use {EntityRef, Iter, IterMut, Keys};
 ///
 /// There should only be a single `PrimaryMap` instance for a given `EntityRef` type, otherwise
 /// conflicting references will be created. Using unknown keys for indexing will cause a panic.
+///
+/// Note that `PrimaryMap` doesn't implement `Deref` or `DerefMut`, which would allow
+/// `&PrimaryMap` to convert to `&[V]`. One of the main advantages of `PrimaryMap` is
+/// that it only allows indexing with the distinct `EntityRef` key type, so converting to a
+/// plain slice would make it easier to use incorrectly.
 #[derive(Debug, Clone)]
 pub struct PrimaryMap
 where
@@ -145,26 +150,6 @@ where
     }
 }
 
-impl Deref for PrimaryMap
-where
-    K: EntityRef,
-{
-    type Target = [V];
-
-    fn deref(&self) -> &Self::Target {
-        &self.elems
-    }
-}
-
-impl DerefMut for PrimaryMap
-where
-    K: EntityRef,
-{
-    fn deref_mut(&mut self) -> &mut [V] {
-        &mut self.elems
-    }
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -341,13 +326,4 @@ mod tests {
             }
         }
     }
-
-    #[test]
-    fn deref() {
-        let mut m = PrimaryMap::::new();
-        let _: &[isize] = m.as_ref();
-        let _: &mut [isize] = m.as_mut();
-        let _: &[isize] = &m;
-        let _: &mut [isize] = &mut m;
-    }
 }

From 300a76469f0b0c832813357eabbe45821c65a94b Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 30 Aug 2018 18:01:07 +0200
Subject: [PATCH 2044/3084] wasm: Clear tables too in
 TranslationState::clear(); (#491)

---
 lib/wasm/src/state.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs
index 88374be17f..2d6d3610af 100644
--- a/lib/wasm/src/state.rs
+++ b/lib/wasm/src/state.rs
@@ -175,6 +175,7 @@ impl TranslationState {
         self.reachable = true;
         self.globals.clear();
         self.heaps.clear();
+        self.tables.clear();
         self.signatures.clear();
         self.functions.clear();
     }

From 4045d50b7b606528f10f743fb7faf59cf619c2b5 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 31 Aug 2018 04:16:36 -0700
Subject: [PATCH 2045/3084] Add a Code of Conduct (#481)

* Create CODE_OF_CONDUCT.md

* Mention the Code of Conduct in CONTRIBUTING.md.

* Add @tyler's email address as an additional contact.
---
 cranelift/CODE_OF_CONDUCT.md | 46 ++++++++++++++++++++++++++++++++++++
 cranelift/CONTRIBUTING.md    |  6 +++++
 2 files changed, 52 insertions(+)
 create mode 100644 cranelift/CODE_OF_CONDUCT.md

diff --git a/cranelift/CODE_OF_CONDUCT.md b/cranelift/CODE_OF_CONDUCT.md
new file mode 100644
index 0000000000..38ef2efa50
--- /dev/null
+++ b/cranelift/CODE_OF_CONDUCT.md
@@ -0,0 +1,46 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at either sunfish@mozilla.com or tyler@fastly.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/
diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
index 351681e290..b5223d006b 100644
--- a/cranelift/CONTRIBUTING.md
+++ b/cranelift/CONTRIBUTING.md
@@ -34,6 +34,12 @@ involved, as there aren't a lot of things set in stone yet.
 
 [good first issue]: https://github.com/CraneStation/cranelift/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22
 
+### Code of Conduct
+
+We abide by our [Code of Conduct] and ask that you do as well.
+
+[Code of Conduct](CODE_OF_CONDUCT.md)
+
 ## Coding Guidelines
 
 For the most part, Cranelift follows common Rust conventions and

From 7e571f4a4949fd2def8fed92008a83d6bf9953db Mon Sep 17 00:00:00 2001
From: Kaz Wesley 
Date: Tue, 4 Sep 2018 15:02:46 -0700
Subject: [PATCH 2046/3084] Print value aliases at referrent definition (#492)

* Print value aliases at referrent definition

Closes #488.
---
 lib/codegen/src/ir/dfg.rs       | 49 ++++++++++++++++--
 lib/codegen/src/print_errors.rs |  9 ++--
 lib/codegen/src/write.rs        | 91 ++++++++++++++++++++++++++++-----
 lib/reader/src/parser.rs        |  6 ++-
 4 files changed, 131 insertions(+), 24 deletions(-)

diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs
index 9db7364002..6d799a43c9 100644
--- a/lib/codegen/src/ir/dfg.rs
+++ b/lib/codegen/src/ir/dfg.rs
@@ -1,6 +1,6 @@
 //! Data flow graph tracking Instructions, Values, and EBBs.
 
-use entity::{EntityMap, PrimaryMap};
+use entity::{self, EntityMap, PrimaryMap};
 use ir;
 use ir::builder::ReplaceBuilder;
 use ir::extfunc::ExtFuncData;
@@ -149,6 +149,38 @@ fn resolve_aliases(values: &PrimaryMap, value: Value) -> Value
     }
 }
 
+/// Iterator over all Values in a DFG
+pub struct Values<'a> {
+    inner: entity::Iter<'a, Value, ValueData>,
+}
+
+/// Check for non-values
+fn valid_valuedata(data: &ValueData) -> bool {
+    if let &ValueData::Alias {
+        ty: types::VOID,
+        original,
+    } = data
+    {
+        if original == Value::reserved_value() {
+            return false;
+        }
+    }
+    return true;
+}
+
+impl<'a> Iterator for Values<'a> {
+    type Item = Value;
+
+    fn next(&mut self) -> Option {
+        return self
+            .inner
+            .by_ref()
+            .filter(|kv| valid_valuedata(kv.1))
+            .next()
+            .map(|kv| kv.0);
+    }
+}
+
 /// Handling values.
 ///
 /// Values are either EBB parameters or instruction results.
@@ -158,6 +190,13 @@ impl DataFlowGraph {
         self.values.push(data)
     }
 
+    /// Get an iterator over all values.
+    pub fn values<'a>(&'a self) -> Values {
+        Values {
+            inner: self.values.iter(),
+        }
+    }
+
     /// Check if a value reference is valid.
     pub fn value_is_valid(&self, v: Value) -> bool {
         self.values.is_valid(v)
@@ -932,9 +971,9 @@ impl DataFlowGraph {
     }
 
     /// Create a new value alias. This is only for use by the parser to create
-    /// aliases with specific values.
+    /// aliases with specific values, and the printer for testing.
     #[cold]
-    pub fn make_value_alias_for_parser(&mut self, src: Value, dest: Value) {
+    pub fn make_value_alias_for_serialization(&mut self, src: Value, dest: Value) {
         assert_ne!(src, Value::reserved_value());
         assert_ne!(dest, Value::reserved_value());
 
@@ -951,9 +990,9 @@ impl DataFlowGraph {
 
     /// If `v` is already defined as an alias, return its destination value.
     /// Otherwise return None. This allows the parser to coalesce identical
-    /// alias definitions.
+    /// alias definitions, and the printer to identify an alias's immediate target.
     #[cold]
-    pub fn value_alias_dest_for_parser(&self, v: Value) -> Option {
+    pub fn value_alias_dest_for_serialization(&self, v: Value) -> Option {
         if let ValueData::Alias { original, .. } = self.values[v] {
             Some(original)
         } else {
diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 6e14182b1d..db557c865e 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -1,7 +1,8 @@
 //! Utility routines for pretty-printing error messages.
 
+use entity::EntityMap;
 use ir;
-use ir::entities::{AnyEntity, Inst};
+use ir::entities::{AnyEntity, Inst, Value};
 use ir::function::Function;
 use isa::TargetIsa;
 use result::CodegenError;
@@ -39,11 +40,12 @@ impl<'a> FuncWriter for PrettyVerifierError<'a> {
         &mut self,
         w: &mut Write,
         func: &Function,
+        aliases: &EntityMap>,
         isa: Option<&TargetIsa>,
         inst: Inst,
         indent: usize,
     ) -> fmt::Result {
-        pretty_instruction_error(w, func, isa, inst, indent, &mut *self.0, self.1)
+        pretty_instruction_error(w, func, aliases, isa, inst, indent, &mut *self.0, self.1)
     }
 
     fn write_entity_definition(
@@ -61,6 +63,7 @@ impl<'a> FuncWriter for PrettyVerifierError<'a> {
 fn pretty_instruction_error(
     w: &mut Write,
     func: &Function,
+    aliases: &EntityMap>,
     isa: Option<&TargetIsa>,
     cur_inst: Inst,
     indent: usize,
@@ -77,7 +80,7 @@ fn pretty_instruction_error(
                 let err = errors.remove(i);
 
                 if !printed_instr {
-                    func_w.write_instruction(w, func, isa, cur_inst, indent)?;
+                    func_w.write_instruction(w, func, aliases, isa, cur_inst, indent)?;
                     printed_instr = true;
                 }
 
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index 71714727ae..24cfdbe284 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -3,6 +3,7 @@
 //! The `write` module provides the `write_function` function which converts an IR `Function` to an
 //! equivalent textual form. This textual form can be read back by the `cranelift-reader` crate.
 
+use entity::EntityMap;
 use ir::entities::AnyEntity;
 use ir::{DataFlowGraph, Ebb, Function, Inst, SigRef, Type, Value, ValueDef};
 use isa::{RegInfo, TargetIsa};
@@ -17,6 +18,7 @@ pub trait FuncWriter {
         &mut self,
         w: &mut Write,
         func: &Function,
+        aliases: &EntityMap>,
         isa: Option<&TargetIsa>,
         inst: Inst,
         ident: usize,
@@ -94,11 +96,12 @@ impl FuncWriter for PlainWriter {
         &mut self,
         w: &mut Write,
         func: &Function,
+        aliases: &EntityMap>,
         isa: Option<&TargetIsa>,
         inst: Inst,
         indent: usize,
     ) -> fmt::Result {
-        write_instruction(w, func, isa, inst, indent)
+        write_instruction(w, func, aliases, isa, inst, indent)
     }
 }
 
@@ -108,6 +111,18 @@ pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) -
     decorate_function(&mut PlainWriter, w, func, isa)
 }
 
+/// Create a reverse-alias map from a value to all aliases having that value as a direct target
+fn alias_map(func: &Function) -> EntityMap> {
+    let mut aliases = EntityMap::<_, Vec<_>>::new();
+    for v in func.dfg.values() {
+        // VADFS returns the immediate target of an alias
+        if let Some(k) = func.dfg.value_alias_dest_for_serialization(v) {
+            aliases[k].push(v);
+        }
+    }
+    aliases
+}
+
 /// Writes `func` to `w` as text.
 /// write_function_plain is passed as 'closure' to print instructions as text.
 /// pretty_function_error is passed as 'closure' to add error decoration.
@@ -123,12 +138,13 @@ pub fn decorate_function(
     write!(w, "function ")?;
     write_spec(w, func, regs)?;
     writeln!(w, " {{")?;
+    let aliases = alias_map(func);
     let mut any = func_w.write_preamble(w, func, regs)?;
     for ebb in &func.layout {
         if any {
             writeln!(w)?;
         }
-        decorate_ebb(func_w, w, func, isa, ebb)?;
+        decorate_ebb(func_w, w, func, &aliases, isa, ebb)?;
         any = true;
     }
     writeln!(w, "}}")
@@ -195,6 +211,7 @@ fn decorate_ebb(
     func_w: &mut FW,
     w: &mut Write,
     func: &Function,
+    aliases: &EntityMap>,
     isa: Option<&TargetIsa>,
     ebb: Ebb,
 ) -> fmt::Result {
@@ -206,8 +223,11 @@ fn decorate_ebb(
     };
 
     write_ebb_header(w, func, isa, ebb, indent)?;
+    for a in func.dfg.ebb_params(ebb).iter().cloned() {
+        write_value_aliases(w, aliases, a, indent)?;
+    }
     for inst in func.layout.ebb_insts(ebb) {
-        func_w.write_instruction(w, func, isa, inst, indent)?;
+        func_w.write_instruction(w, func, aliases, isa, inst, indent)?;
     }
 
     Ok(())
@@ -251,13 +271,16 @@ fn type_suffix(func: &Function, inst: Inst) -> Option {
     Some(rtype)
 }
 
-// Write out any value aliases appearing in `inst`.
-fn write_value_aliases(w: &mut Write, func: &Function, inst: Inst, indent: usize) -> fmt::Result {
-    for &arg in func.dfg.inst_args(inst) {
-        let resolved = func.dfg.resolve_aliases(arg);
-        if resolved != arg {
-            writeln!(w, "{1:0$}{2} -> {3}", indent, "", arg, resolved)?;
-        }
+/// Write out any aliases to the given target, including indirect aliases
+fn write_value_aliases(
+    w: &mut Write,
+    aliases: &EntityMap>,
+    target: Value,
+    indent: usize,
+) -> fmt::Result {
+    for &a in &aliases[target] {
+        writeln!(w, "{1:0$}{2} -> {3}", indent, "", a, target)?;
+        write_value_aliases(w, aliases, a, indent)?;
     }
     Ok(())
 }
@@ -265,13 +288,11 @@ fn write_value_aliases(w: &mut Write, func: &Function, inst: Inst, indent: usize
 fn write_instruction(
     w: &mut Write,
     func: &Function,
+    aliases: &EntityMap>,
     isa: Option<&TargetIsa>,
     inst: Inst,
     indent: usize,
 ) -> fmt::Result {
-    // Value aliases come out on lines before the instruction using them.
-    write_value_aliases(w, func, inst, indent)?;
-
     // Prefix containing source location, encoding, and value locations.
     let mut s = String::with_capacity(16);
 
@@ -324,7 +345,13 @@ fn write_instruction(
     }
 
     write_operands(w, &func.dfg, isa, inst)?;
-    writeln!(w)
+    writeln!(w)?;
+
+    // Value aliases come out on lines after the instruction defining the referrent.
+    for r in func.dfg.inst_results(inst) {
+        write_value_aliases(w, aliases, *r, indent)?;
+    }
+    Ok(())
 }
 
 /// Write the operands of `inst` to `w` with a prepended space.
@@ -623,4 +650,40 @@ mod tests {
             "function %foo() fast {\n    ss0 = explicit_slot 4\n\nebb0(v0: i8, v1: f32x4):\n    return\n}\n"
         );
     }
+
+    #[test]
+    fn aliases() {
+        use ir::InstBuilder;
+
+        let mut func = Function::new();
+        {
+            let ebb0 = func.dfg.make_ebb();
+            let mut pos = FuncCursor::new(&mut func);
+            pos.insert_ebb(ebb0);
+
+            // make some detached values for change_to_alias
+            let v0 = pos.func.dfg.append_ebb_param(ebb0, types::I32);
+            let v1 = pos.func.dfg.append_ebb_param(ebb0, types::I32);
+            let v2 = pos.func.dfg.append_ebb_param(ebb0, types::I32);
+            pos.func.dfg.detach_ebb_params(ebb0);
+
+            // alias to a param--will be printed at beginning of ebb defining param
+            let v3 = pos.func.dfg.append_ebb_param(ebb0, types::I32);
+            pos.func.dfg.change_to_alias(v0, v3);
+
+            // alias to an alias--should print attached to alias, not ultimate target
+            pos.func.dfg.make_value_alias_for_serialization(v0, v2); // v0 <- v2
+
+            // alias to a result--will be printed after instruction producing result
+            let _dummy0 = pos.ins().iconst(types::I32, 42);
+            let v4 = pos.ins().iadd(v0, v0);
+            pos.func.dfg.change_to_alias(v1, v4);
+            let _dummy1 = pos.ins().iconst(types::I32, 23);
+            let _v7 = pos.ins().iadd(v1, v1);
+        }
+        assert_eq!(
+            func.to_string(),
+            "function u0:0() fast {\nebb0(v3: i32):\n    v0 -> v3\n    v2 -> v0\n    v4 = iconst.i32 42\n    v5 = iadd v0, v0\n    v1 -> v5\n    v6 = iconst.i32 23\n    v7 = iadd v1, v1\n}\n"
+        );
+    }
 }
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index ee04728213..d1e14da50c 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -1688,7 +1688,7 @@ impl<'a> Parser<'a> {
 
         // Allow duplicate definitions of aliases, as long as they are identical.
         if ctx.map.contains_value(result) {
-            if let Some(old) = ctx.function.dfg.value_alias_dest_for_parser(result) {
+            if let Some(old) = ctx.function.dfg.value_alias_dest_for_serialization(result) {
                 if old != dest {
                     return err!(
                         self.loc,
@@ -1708,7 +1708,9 @@ impl<'a> Parser<'a> {
             return err!(self.loc, "value {} is not yet defined", dest);
         }
 
-        ctx.function.dfg.make_value_alias_for_parser(dest, result);
+        ctx.function
+            .dfg
+            .make_value_alias_for_serialization(dest, result);
 
         ctx.aliases.push(result);
         Ok(())

From 17bb62c16c86cbfb6aef75ccce164953b31e187b Mon Sep 17 00:00:00 2001
From: Aaron Power 
Date: Wed, 5 Sep 2018 00:23:50 +0100
Subject: [PATCH 2047/3084] Added bitrev instruction for 32 and 64 bit integers
 (#486)

---
 cranelift/filetests/legalizer/bitrev.clif    | 149 +++++++++++++++++++
 lib/codegen/meta-python/base/instructions.py |   8 +
 lib/codegen/meta-python/base/legalize.py     | 128 ++++++++++++++++
 lib/codegen/meta-python/cdsl/ast.py          |  12 +-
 4 files changed, 293 insertions(+), 4 deletions(-)
 create mode 100644 cranelift/filetests/legalizer/bitrev.clif

diff --git a/cranelift/filetests/legalizer/bitrev.clif b/cranelift/filetests/legalizer/bitrev.clif
new file mode 100644
index 0000000000..0f75c373a9
--- /dev/null
+++ b/cranelift/filetests/legalizer/bitrev.clif
@@ -0,0 +1,149 @@
+test legalizer
+target x86_64
+
+function %reverse_bits_8(i8) -> i8 {
+ebb0(v0: i8):
+    v1 = bitrev.i8 v0
+    return v1
+}
+; check: v2 = band_imm v0, 170
+; check: v3 = ushr_imm v2, 1
+; check: v4 = band_imm v0, 85
+; check: v5 = ishl_imm v4, 1
+; check: v16 = uextend.i32 v3
+; check: v17 = uextend.i32 v5
+; check: v18 = bor v16, v17
+; check: v6 = ireduce.i8 v18
+; check: v7 = band_imm v6, 204
+; check: v8 = ushr_imm v7, 2
+; check: v9 = band_imm v6, 51
+; check: v10 = ushr_imm v9, 2
+; check: v19 = uextend.i32 v8
+; check: v20 = uextend.i32 v10
+; check: v21 = bor v19, v20
+; check: v11 = ireduce.i8 v21
+; check: v12 = band_imm v11, 240
+; check: v13 = ushr_imm v12, 4
+; check: v14 = band_imm v11, 15
+; check: v15 = ishl_imm v14, 4
+; check: v22 = uextend.i32 v13
+; check: v23 = uextend.i32 v15
+; check: v24 = bor v22, v23
+; check: v1 = ireduce.i8 v24
+
+function %reverse_bits_16(i16) -> i16 {
+ebb0(v0: i16):
+    v1 = bitrev.i16 v0
+    return v1
+}
+; check: v2 = band_imm v0, 0xaaaa
+; check: v3 = ushr_imm v2, 1
+; check: v4 = band_imm v0, 0x5555
+; check: v5 = ishl_imm v4, 1
+; check: v21 = uextend.i32 v3
+; check: v22 = uextend.i32 v5
+; check: v23 = bor v21, v22
+; check: v6 = ireduce.i16 v23
+; check: v7 = band_imm v6, 0xcccc
+; check: v8 = ushr_imm v7, 2
+; check: v9 = band_imm v6, 0x3333
+; check: v10 = ushr_imm v9, 2
+; check: v24 = uextend.i32 v8
+; check: v25 = uextend.i32 v10
+; check: v26 = bor v24, v25
+; check: v11 = ireduce.i16 v26
+; check: v12 = band_imm v11, 0xf0f0
+; check: v13 = ushr_imm v12, 4
+; check: v14 = band_imm v11, 3855
+; check: v15 = ishl_imm v14, 4
+; check: v27 = uextend.i32 v13
+; check: v28 = uextend.i32 v15
+; check: v29 = bor v27, v28
+; check: v16 = ireduce.i16 v29
+; check: v17 = band_imm v16, 0xff00
+; check: v18 = ushr_imm v17, 8
+; check: v19 = band_imm v16, 255
+; check: v20 = ishl_imm v19, 8
+; check: v30 = uextend.i32 v18
+; check: v31 = uextend.i32 v20
+; check: v32 = bor v30, v31
+; check: v1 = ireduce.i16 v32
+; check: return v1
+
+function %reverse_bits_32(i32) -> i32 {
+ebb0(v0: i32):
+    v1 = bitrev.i32 v0
+    return v1
+}
+; check: v24 = iconst.i32 0xaaaa_aaaa
+; check: v2 = band v0, v24
+; check: v3 = ushr_imm v2, 1
+; check: v4 = band_imm v0, 0x5555_5555
+; check: v5 = ishl_imm v4, 1
+; check: v6 = bor v3, v5
+; check: v25 = iconst.i32 0xcccc_cccc
+; check: v7 = band v6, v25
+; check: v8 = ushr_imm v7, 2
+; check: v9 = band_imm v6, 0x3333_3333
+; check: v10 = ushr_imm v9, 2
+; check: v11 = bor v8, v10
+; check: v26 = iconst.i32 0xf0f0_f0f0
+; check: v12 = band v11, v26
+; check: v13 = ushr_imm v12, 4
+; check: v14 = band_imm v11, 0x0f0f_0f0f
+; check: v15 = ishl_imm v14, 4
+; check: v16 = bor v13, v15
+; check: v27 = iconst.i32 0xff00_ff00
+; check: v17 = band v16, v27
+; check: v18 = ushr_imm v17, 8
+; check: v19 = band_imm v16, 0x00ff_00ff
+; check: v20 = ishl_imm v19, 8
+; check: v21 = bor v18, v20
+; check: v22 = ushr_imm v21, 16
+; check: v23 = ishl_imm v21, 16
+; check: v1 = bor v22, v23
+
+
+function %reverse_bits_64(i64) -> i64 {
+ebb0(v0: i64):
+    v1 = bitrev.i64 v0
+    return v1
+}
+; check: v29 = iconst.i64 0xaaaa_aaaa_aaaa_aaaa
+; check: v2 = band v0, v29
+; check: v3 = ushr_imm v2, 1
+; check: v30 = iconst.i64 0x5555_5555_5555_5555
+; check: v4 = band v0, v30
+; check: v5 = ishl_imm v4, 1
+; check: v6 = bor v3, v5
+; check: v31 = iconst.i64 0xcccc_cccc_cccc_cccc
+; check: v7 = band v6, v31
+; check: v8 = ushr_imm v7, 2
+; check: v32 = iconst.i64 0x3333_3333_3333_3333
+; check: v9 = band v6, v32
+; check: v10 = ushr_imm v9, 2
+; check: v11 = bor v8, v10
+; check: v33 = iconst.i64 0xf0f0_f0f0_f0f0_f0f0
+; check: v12 = band v11, v33
+; check: v13 = ushr_imm v12, 4
+; check: v34 = iconst.i64 0x0f0f_0f0f_0f0f_0f0f
+; check: v14 = band v11, v34
+; check: v15 = ishl_imm v14, 4
+; check: v16 = bor v13, v15
+; check: v35 = iconst.i64 0xff00_ff00_ff00_ff00
+; check: v17 = band v16, v35
+; check: v18 = ushr_imm v17, 8
+; check: v36 = iconst.i64 0x00ff_00ff_00ff_00ff
+; check: v19 = band v16, v36
+; check: v20 = ishl_imm v19, 8
+; check: v21 = bor v18, v20
+; check: v37 = iconst.i64 0xffff_0000_ffff_0000
+; check: v22 = band v21, v37
+; check: v23 = ushr_imm v22, 16
+; check: v38 = iconst.i64 0xffff_0000_ffff
+; check: v24 = band v21, v38
+; check: v25 = ishl_imm v24, 16
+; check: v26 = bor v23, v25
+; check: v27 = ushr_imm v26, 32
+; check: v28 = ishl_imm v26, 32
+; check: v1 = bor v27, v28
diff --git a/lib/codegen/meta-python/base/instructions.py b/lib/codegen/meta-python/base/instructions.py
index 25b8ba414b..9f7caf5c48 100644
--- a/lib/codegen/meta-python/base/instructions.py
+++ b/lib/codegen/meta-python/base/instructions.py
@@ -1409,6 +1409,14 @@ sshr_imm = Instruction(
 x = Operand('x', iB)
 a = Operand('a', iB)
 
+bitrev = Instruction(
+        'bitrev', r"""
+        Reverse the bits of a integer.
+
+        Reverses the bits in ``x``.
+        """,
+        ins=x, outs=a)
+
 clz = Instruction(
         'clz', r"""
         Count leading zero bits.
diff --git a/lib/codegen/meta-python/base/legalize.py b/lib/codegen/meta-python/base/legalize.py
index 3427d8ef44..9f6175f138 100644
--- a/lib/codegen/meta-python/base/legalize.py
+++ b/lib/codegen/meta-python/base/legalize.py
@@ -26,6 +26,7 @@ from .instructions import rotl, rotl_imm, rotr, rotr_imm
 from .instructions import f32const, f64const
 from .instructions import store, load
 from .instructions import br_table
+from .instructions import bitrev
 from cdsl.ast import Var
 from cdsl.xform import Rtl, XFormGroup
 
@@ -91,17 +92,35 @@ y = Var('y')
 a = Var('a')
 a1 = Var('a1')
 a2 = Var('a2')
+a3 = Var('a3')
+a4 = Var('a4')
 b = Var('b')
 b1 = Var('b1')
 b2 = Var('b2')
+b3 = Var('b3')
+b4 = Var('b4')
 b_in = Var('b_in')
 b_int = Var('b_int')
 c = Var('c')
 c1 = Var('c1')
 c2 = Var('c2')
+c3 = Var('c3')
+c4 = Var('c4')
 c_in = Var('c_in')
 c_int = Var('c_int')
 d = Var('d')
+d1 = Var('d1')
+d2 = Var('d2')
+d3 = Var('d3')
+d4 = Var('d4')
+e = Var('e')
+e1 = Var('e1')
+e2 = Var('e2')
+e3 = Var('e3')
+e4 = Var('e4')
+f = Var('f')
+f1 = Var('f1')
+f2 = Var('f2')
 xl = Var('xl')
 xh = Var('xh')
 yl = Var('yl')
@@ -382,6 +401,115 @@ expand.legalize(
             a << bxor(x, y)
         ))
 
+# Expand bitrev
+# Adapted from Stack Overflow.
+# https://stackoverflow.com/questions/746171/most-efficient-algorithm-for-bit-reversal-from-msb-lsb-to-lsb-msb-in-c
+widen.legalize(
+        a << bitrev.i8(x),
+        Rtl(
+            a1 << band_imm(x, imm64(0xaa)),
+            a2 << ushr_imm(a1, imm64(1)),
+            a3 << band_imm(x, imm64(0x55)),
+            a4 << ishl_imm(a3, imm64(1)),
+            b << bor(a2, a4),
+            b1 << band_imm(b, imm64(0xcc)),
+            b2 << ushr_imm(b1, imm64(2)),
+            b3 << band_imm(b, imm64(0x33)),
+            b4 << ushr_imm(b3, imm64(2)),
+            c << bor(b2, b4),
+            c1 << band_imm(c, imm64(0xf0)),
+            c2 << ushr_imm(c1, imm64(4)),
+            c3 << band_imm(c, imm64(0x0f)),
+            c4 << ishl_imm(c3, imm64(4)),
+            a << bor(c2, c4),
+        ))
+
+widen.legalize(
+        a << bitrev.i16(x),
+        Rtl(
+            a1 << band_imm(x, imm64(0xaaaa)),
+            a2 << ushr_imm(a1, imm64(1)),
+            a3 << band_imm(x, imm64(0x5555)),
+            a4 << ishl_imm(a3, imm64(1)),
+            b << bor(a2, a4),
+            b1 << band_imm(b, imm64(0xcccc)),
+            b2 << ushr_imm(b1, imm64(2)),
+            b3 << band_imm(b, imm64(0x3333)),
+            b4 << ushr_imm(b3, imm64(2)),
+            c << bor(b2, b4),
+            c1 << band_imm(c, imm64(0xf0f0)),
+            c2 << ushr_imm(c1, imm64(4)),
+            c3 << band_imm(c, imm64(0x0f0f)),
+            c4 << ishl_imm(c3, imm64(4)),
+            d << bor(c2, c4),
+            d1 << band_imm(d, imm64(0xff00)),
+            d2 << ushr_imm(d1, imm64(8)),
+            d3 << band_imm(d, imm64(0x00ff)),
+            d4 << ishl_imm(d3, imm64(8)),
+            a << bor(d2, d4),
+        ))
+
+expand.legalize(
+        a << bitrev.i32(x),
+        Rtl(
+            a1 << band_imm(x, imm64(0xaaaaaaaa)),
+            a2 << ushr_imm(a1, imm64(1)),
+            a3 << band_imm(x, imm64(0x55555555)),
+            a4 << ishl_imm(a3, imm64(1)),
+            b << bor(a2, a4),
+            b1 << band_imm(b, imm64(0xcccccccc)),
+            b2 << ushr_imm(b1, imm64(2)),
+            b3 << band_imm(b, imm64(0x33333333)),
+            b4 << ushr_imm(b3, imm64(2)),
+            c << bor(b2, b4),
+            c1 << band_imm(c, imm64(0xf0f0f0f0)),
+            c2 << ushr_imm(c1, imm64(4)),
+            c3 << band_imm(c, imm64(0x0f0f0f0f)),
+            c4 << ishl_imm(c3, imm64(4)),
+            d << bor(c2, c4),
+            d1 << band_imm(d, imm64(0xff00ff00)),
+            d2 << ushr_imm(d1, imm64(8)),
+            d3 << band_imm(d, imm64(0x00ff00ff)),
+            d4 << ishl_imm(d3, imm64(8)),
+            e << bor(d2, d4),
+            e1 << ushr_imm(e, imm64(16)),
+            e2 << ishl_imm(e, imm64(16)),
+            a << bor(e1, e2),
+        ))
+
+expand.legalize(
+        a << bitrev.i64(x),
+        Rtl(
+            a1 << band_imm(x, imm64(0xaaaaaaaaaaaaaaaa)),
+            a2 << ushr_imm(a1, imm64(1)),
+            a3 << band_imm(x, imm64(0x5555555555555555)),
+            a4 << ishl_imm(a3, imm64(1)),
+            b << bor(a2, a4),
+            b1 << band_imm(b, imm64(0xcccccccccccccccc)),
+            b2 << ushr_imm(b1, imm64(2)),
+            b3 << band_imm(b, imm64(0x3333333333333333)),
+            b4 << ushr_imm(b3, imm64(2)),
+            c << bor(b2, b4),
+            c1 << band_imm(c, imm64(0xf0f0f0f0f0f0f0f0)),
+            c2 << ushr_imm(c1, imm64(4)),
+            c3 << band_imm(c, imm64(0x0f0f0f0f0f0f0f0f)),
+            c4 << ishl_imm(c3, imm64(4)),
+            d << bor(c2, c4),
+            d1 << band_imm(d, imm64(0xff00ff00ff00ff00)),
+            d2 << ushr_imm(d1, imm64(8)),
+            d3 << band_imm(d, imm64(0x00ff00ff00ff00ff)),
+            d4 << ishl_imm(d3, imm64(8)),
+            e << bor(d2, d4),
+            e1 << band_imm(e, imm64(0xffff0000ffff0000)),
+            e2 << ushr_imm(e1, imm64(16)),
+            e3 << band_imm(e, imm64(0x0000ffff0000ffff)),
+            e4 << ishl_imm(e3, imm64(16)),
+            f << bor(e2, e4),
+            f1 << ushr_imm(f, imm64(32)),
+            f2 << ishl_imm(f, imm64(32)),
+            a << bor(f1, f2),
+        ))
+
 # Floating-point sign manipulations.
 for ty,             minus_zero in [
         (types.f32, f32const(ieee32.bits(0x80000000))),
diff --git a/lib/codegen/meta-python/cdsl/ast.py b/lib/codegen/meta-python/cdsl/ast.py
index 09d4ad2514..75716d36a9 100644
--- a/lib/codegen/meta-python/cdsl/ast.py
+++ b/lib/codegen/meta-python/cdsl/ast.py
@@ -523,10 +523,14 @@ class ConstantInt(Literal):
 
     def __str__(self):
         # type: () -> str
-        """
-        Get the Rust expression form of this constant.
-        """
-        return str(self.value)
+        # If the value is in the signed imm64 range, print it as-is.
+        if self.value >= -(2**63) and self.value < (2**63):
+            return str(self.value)
+        # Otherwise if the value is in the unsigned imm64 range, print its
+        # bitwise counterpart in the signed imm64 range.
+        if self.value >= (2**63) and self.value < (2**64):
+            return str(self.value - (2**64))
+        assert False, "immediate value not in signed or unsigned imm64 range"
 
 
 class ConstantBits(Literal):

From 59b83912ba66575057e2a6bdbf2199410010f1bb Mon Sep 17 00:00:00 2001
From: Caroline Cullen 
Date: Tue, 4 Sep 2018 16:31:24 -0700
Subject: [PATCH 2048/3084] Adds pass command to clif-util. (#487)

* Adds pass command to clif-util.
---
 cranelift/src/clif-util.rs      | 50 +++++++++++++++++--
 cranelift/src/compile.rs        |  2 +-
 lib/filetests/src/concurrent.rs | 23 ++++-----
 lib/filetests/src/lib.rs        | 18 +++++++
 lib/filetests/src/runner.rs     | 52 +++++++++++++++++---
 lib/filetests/src/runone.rs     |  4 +-
 lib/reader/src/parser.rs        | 86 +++++++++++++++++++++++++++++++--
 lib/reader/src/sourcemap.rs     |  2 +
 8 files changed, 207 insertions(+), 30 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 10875b9276..c54c5b2d2c 100644
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -1,5 +1,5 @@
-#![deny(trivial_numeric_casts, unused_extern_crates, unstable_features)]
-#![warn(unused_import_braces)]
+#![deny(trivial_numeric_casts)]
+#![warn(unused_import_braces, unstable_features, unused_extern_crates)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
@@ -15,13 +15,13 @@ extern crate cfg_if;
 extern crate capstone;
 extern crate clap;
 extern crate cranelift_codegen;
+extern crate cranelift_entity;
 extern crate cranelift_filetests;
 extern crate cranelift_reader;
 extern crate pretty_env_logger;
 
 cfg_if! {
     if #[cfg(feature = "wasm")] {
-        extern crate cranelift_entity;
         extern crate cranelift_wasm;
         extern crate term;
         extern crate wabt;
@@ -53,6 +53,21 @@ fn add_input_file_arg<'a>() -> clap::Arg<'a, 'a> {
         .help("Specify file(s) to be used for test")
 }
 
+fn add_single_input_file_arg<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("single-file")
+        .required(true)
+        .value_name("single-file")
+        .help("Specify a file to be used")
+}
+
+fn add_pass_arg<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("pass")
+        .required(true)
+        .multiple(true)
+        .value_name("pass")
+        .help("Specify pass(s) to be run on test file")
+}
+
 fn add_verbose_flag<'a>() -> clap::Arg<'a, 'a> {
     Arg::with_name("verbose").short("v").help("Be more verbose")
 }
@@ -162,7 +177,16 @@ fn main() {
                     "Just checks the correctness of Cranelift IR translated from WebAssembly",
                 )),
         )
-        .subcommand(add_wasm_or_compile("wasm"));
+        .subcommand(add_wasm_or_compile("wasm"))
+        .subcommand(
+            SubCommand::with_name("pass")
+                .about("Run specified pass(s) on an input file.")
+                .arg(add_single_input_file_arg())
+                .arg(add_target_flag())
+                .arg(add_pass_arg())
+                .arg(add_debug_flag())
+                .arg(add_time_flag()),
+        );
 
     let res_util = match app_cmds.get_matches().subcommand() {
         ("cat", Some(rest_cmd)) => {
@@ -172,10 +196,26 @@ fn main() {
         ("test", Some(rest_cmd)) => {
             handle_debug_flag(rest_cmd.is_present("debug"));
             cranelift_filetests::run(
-                rest_cmd.is_present("time-passes"),
+                rest_cmd.is_present("verbose"),
                 &get_vec(rest_cmd.values_of("file")),
             ).map(|_time| ())
         }
+        ("pass", Some(rest_cmd)) => {
+            handle_debug_flag(rest_cmd.is_present("debug"));
+
+            let mut target_val: &str = "";
+            if let Some(clap_target) = rest_cmd.value_of("target") {
+                target_val = clap_target;
+            }
+
+            // Can be unwrapped because 'single-file' is required
+            cranelift_filetests::run_passes(
+                rest_cmd.is_present("verbose"),
+                &get_vec(rest_cmd.values_of("pass")),
+                target_val,
+                &rest_cmd.value_of("single-file").unwrap().to_string(),
+            ).map(|_time| ())
+        }
         ("print-cfg", Some(rest_cmd)) => {
             handle_debug_flag(rest_cmd.is_present("debug"));
             print_cfg::run(&get_vec(rest_cmd.values_of("file")))
diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs
index 5a21d5789d..c49b5d3aef 100644
--- a/cranelift/src/compile.rs
+++ b/cranelift/src/compile.rs
@@ -80,7 +80,7 @@ fn handle_module(
     fisa: FlagsOrIsa,
 ) -> Result<(), String> {
     let buffer = read_to_string(&path).map_err(|e| format!("{}: {}", name, e))?;
-    let test_file = parse_test(&buffer).map_err(|e| format!("{}: {}", name, e))?;
+    let test_file = parse_test(&buffer, None, None).map_err(|e| format!("{}: {}", name, e))?;
 
     // If we have an isa from the command-line, use that. Otherwise if the
     // file contains a unique isa, use that.
diff --git a/lib/filetests/src/concurrent.rs b/lib/filetests/src/concurrent.rs
index 104967d757..df9034618b 100644
--- a/lib/filetests/src/concurrent.rs
+++ b/lib/filetests/src/concurrent.rs
@@ -131,17 +131,18 @@ fn worker_thread(
                 // The receiver should always be present for this as long as we have jobs.
                 replies.send(Reply::Starting { jobid, thread_num }).unwrap();
 
-                let result = catch_unwind(|| runone::run(path.as_path())).unwrap_or_else(|e| {
-                    // The test panicked, leaving us a `Box`.
-                    // Panics are usually strings.
-                    if let Some(msg) = e.downcast_ref::() {
-                        Err(format!("panicked in worker #{}: {}", thread_num, msg))
-                    } else if let Some(msg) = e.downcast_ref::<&'static str>() {
-                        Err(format!("panicked in worker #{}: {}", thread_num, msg))
-                    } else {
-                        Err(format!("panicked in worker #{}", thread_num))
-                    }
-                });
+                let result = catch_unwind(|| runone::run(path.as_path(), None, None))
+                    .unwrap_or_else(|e| {
+                        // The test panicked, leaving us a `Box`.
+                        // Panics are usually strings.
+                        if let Some(msg) = e.downcast_ref::() {
+                            Err(format!("panicked in worker #{}: {}", thread_num, msg))
+                        } else if let Some(msg) = e.downcast_ref::<&'static str>() {
+                            Err(format!("panicked in worker #{}: {}", thread_num, msg))
+                        } else {
+                            Err(format!("panicked in worker #{}", thread_num))
+                        }
+                    });
 
                 if let Err(ref msg) = result {
                     error!("FAIL: {}", msg);
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index 90565b5cb4..d7238bb561 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -79,6 +79,24 @@ pub fn run(verbose: bool, files: &[String]) -> TestResult {
     runner.run()
 }
 
+/// Used for 'pass' subcommand.
+/// Commands are interpreted as test and executed.
+///
+/// Directories are scanned recursively for test cases ending in `.clif`.
+///
+pub fn run_passes(verbose: bool, passes: &[String], target: &str, file: &String) -> TestResult {
+    let mut runner = TestRunner::new(verbose);
+
+    let path = Path::new(file);
+    if path.is_file() {
+        runner.push_test(path);
+    } else {
+        runner.push_dir(path);
+    }
+
+    runner.run_passes(passes, target)
+}
+
 /// Create a new subcommand trait object to match `parsed.command`.
 ///
 /// This function knows how to create all of the possible `test ` commands that can appear in
diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs
index 08ee75430b..a05f54c390 100644
--- a/lib/filetests/src/runner.rs
+++ b/lib/filetests/src/runner.rs
@@ -30,6 +30,12 @@ enum State {
     Done(TestResult),
 }
 
+#[derive(PartialEq, Eq, Debug)]
+pub enum IsPass {
+    Pass,
+    NotPass,
+}
+
 impl QueueEntry {
     pub fn path(&self) -> &Path {
         self.path.as_path()
@@ -118,7 +124,7 @@ impl TestRunner {
 
     /// Scan any directories pushed so far.
     /// Push any potential test cases found.
-    pub fn scan_dirs(&mut self) {
+    pub fn scan_dirs(&mut self, pass_status: IsPass) {
         // This recursive search tries to minimize statting in a directory hierarchy containing
         // mostly test cases.
         //
@@ -164,8 +170,12 @@ impl TestRunner {
                     }
                 }
             }
-            // Get the new jobs running before moving on to the next directory.
-            self.schedule_jobs();
+            if pass_status == IsPass::Pass {
+                continue;
+            } else {
+                // Get the new jobs running before moving on to the next directory.
+                self.schedule_jobs();
+            }
         }
     }
 
@@ -203,7 +213,7 @@ impl TestRunner {
             } else {
                 // Run test synchronously.
                 self.tests[jobid].state = State::Running;
-                let result = runone::run(self.tests[jobid].path());
+                let result = runone::run(self.tests[jobid].path(), None, None);
                 self.finish_job(jobid, result);
             }
             self.new_tests = jobid + 1;
@@ -215,6 +225,20 @@ impl TestRunner {
         }
     }
 
+    /// Schedule any new job to run for the pass command.
+    fn schedule_pass_job(&mut self, passes: &[String], target: &str) {
+        self.tests[0].state = State::Running;
+        let result: Result;
+
+        let specified_target = match target {
+            "" => None,
+            targ => Some(targ),
+        };
+
+        result = runone::run(self.tests[0].path(), Some(passes), specified_target);
+        self.finish_job(0, result);
+    }
+
     /// Report the end of a job.
     fn finish_job(&mut self, jobid: usize, result: TestResult) {
         assert_eq!(self.tests[jobid].state, State::Running);
@@ -331,10 +355,26 @@ impl TestRunner {
     /// Scan pushed directories for tests and run them.
     pub fn run(&mut self) -> TestResult {
         let started = time::Instant::now();
-        self.scan_dirs();
+        self.scan_dirs(IsPass::NotPass);
         self.schedule_jobs();
-        self.drain_threads();
         self.report_slow_tests();
+        self.drain_threads();
+
+        println!("{} tests", self.tests.len());
+        match self.errors {
+            0 => Ok(started.elapsed()),
+            1 => Err("1 failure".to_string()),
+            n => Err(format!("{} failures", n)),
+        }
+    }
+
+    /// Scan pushed directories for tests and run specified passes from commandline on them.
+    pub fn run_passes(&mut self, passes: &[String], target: &str) -> TestResult {
+        let started = time::Instant::now();
+        self.scan_dirs(IsPass::Pass);
+        self.schedule_pass_job(passes, target);
+        self.report_slow_tests();
+
         println!("{} tests", self.tests.len());
         match self.errors {
             0 => Ok(started.elapsed()),
diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs
index b365b4a96b..04c1e22f3b 100644
--- a/lib/filetests/src/runone.rs
+++ b/lib/filetests/src/runone.rs
@@ -27,12 +27,12 @@ fn read_to_string>(path: P) -> io::Result {
 /// Load `path` and run the test in it.
 ///
 /// If running this test causes a panic, it will propagate as normal.
-pub fn run(path: &Path) -> TestResult {
+pub fn run(path: &Path, passes: Option<&[String]>, target: Option<&str>) -> TestResult {
     let _tt = timing::process_file();
     info!("---\nFile: {}", path.to_string_lossy());
     let started = time::Instant::now();
     let buffer = read_to_string(path).map_err(|e| e.to_string())?;
-    let testfile = parse_test(&buffer).map_err(|e| e.to_string())?;
+    let testfile = parse_test(&buffer, passes, target).map_err(|e| e.to_string())?;
     if testfile.functions.is_empty() {
         return Err("no functions found".to_string());
     }
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index d1e14da50c..3003f037d9 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -32,20 +32,39 @@ use testfile::{Comment, Details, TestFile};
 /// Any test commands or target declarations are ignored.
 pub fn parse_functions(text: &str) -> ParseResult> {
     let _tt = timing::parse_text();
-    parse_test(text).map(|file| file.functions.into_iter().map(|(func, _)| func).collect())
+    parse_test(text, None, None)
+        .map(|file| file.functions.into_iter().map(|(func, _)| func).collect())
 }
 
 /// Parse the entire `text` as a test case file.
 ///
 /// The returned `TestFile` contains direct references to substrings of `text`.
-pub fn parse_test(text: &str) -> ParseResult {
+pub fn parse_test<'a>(
+    text: &'a str,
+    passes: Option<&'a [String]>,
+    target: Option<&str>,
+) -> ParseResult> {
     let _tt = timing::parse_text();
     let mut parser = Parser::new(text);
     // Gather the preamble comments.
     parser.start_gathering_comments();
 
-    let commands = parser.parse_test_commands();
-    let isa_spec = parser.parse_target_specs()?;
+    let isa_spec: isaspec::IsaSpec;
+    let commands: Vec>;
+
+    // Check for specified passes and target, if present throw out test commands/targets specified in file.
+    match passes {
+        Some(pass_vec) => {
+            parser.parse_test_commands();
+            commands = parser.parse_cmdline_passes(pass_vec);
+            parser.parse_target_specs()?;
+            isa_spec = parser.parse_cmdline_target(target)?;
+        }
+        None => {
+            commands = parser.parse_test_commands();
+            isa_spec = parser.parse_target_specs()?;
+        }
+    };
 
     parser.token();
     parser.claim_gathered_comments(AnyEntity::Function);
@@ -705,6 +724,15 @@ impl<'a> Parser<'a> {
         }
     }
 
+    /// Parse a list of test command passes specified in command line.
+    pub fn parse_cmdline_passes(&mut self, passes: &'a [String]) -> Vec> {
+        let mut list = Vec::new();
+        for pass in passes {
+            list.push(TestCommand::new(pass));
+        }
+        list
+    }
+
     /// Parse a list of test commands.
     pub fn parse_test_commands(&mut self) -> Vec> {
         let mut list = Vec::new();
@@ -714,12 +742,58 @@ impl<'a> Parser<'a> {
         list
     }
 
+    /// Parse a target spec.
+    ///
+    /// Accept the target from the command line for pass command.
+    ///
+    pub fn parse_cmdline_target(
+        &mut self,
+        target_pass: Option<&str>,
+    ) -> ParseResult {
+        // Were there any `target` commands specified?
+        let mut specified_target = false;
+
+        let mut targets = Vec::new();
+        let flag_builder = settings::builder();
+
+        match target_pass {
+            Some(targ) => {
+                let loc = self.loc;
+                let triple = match Triple::from_str(targ) {
+                    Ok(triple) => triple,
+                    Err(err) => return err!(loc, err),
+                };
+                let mut isa_builder = match isa::lookup(triple) {
+                    Err(isa::LookupError::SupportDisabled) => {
+                        return err!(loc, "support disabled target '{}'", targ)
+                    }
+                    Err(isa::LookupError::Unsupported) => {
+                        return err!(loc, "unsupported target '{}'", targ)
+                    }
+                    Ok(b) => b,
+                };
+                specified_target = true;
+
+                // Construct a trait object with the aggregate settings.
+                targets.push(isa_builder.finish(settings::Flags::new(flag_builder.clone())));
+            }
+            None => (),
+        };
+
+        if !specified_target {
+            // No `target` commands.
+            Ok(isaspec::IsaSpec::None(settings::Flags::new(flag_builder)))
+        } else {
+            Ok(isaspec::IsaSpec::Some(targets))
+        }
+    }
+
     /// Parse a list of target specs.
     ///
     /// Accept a mix of `target` and `set` command lines. The `set` commands are cumulative.
     ///
     pub fn parse_target_specs(&mut self) -> ParseResult {
-        // Was there any `target` commands?
+        // Were there any `target` commands?
         let mut seen_target = false;
         // Location of last `set` command since the last `target`.
         let mut last_set_loc = None;
@@ -2703,6 +2777,8 @@ mod tests {
                              set enable_float=false
                              ; still preamble
                              function %comment() system_v {}",
+            None,
+            None,
         ).unwrap();
         assert_eq!(tf.commands.len(), 2);
         assert_eq!(tf.commands[0].command, "cfg");
diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs
index 4103d08450..ca653d21b5 100644
--- a/lib/reader/src/sourcemap.rs
+++ b/lib/reader/src/sourcemap.rs
@@ -222,6 +222,8 @@ mod tests {
                              ebb0(v4: i32, v7: i32):
                                v10 = iadd v4, v7
                              }",
+            None,
+            None,
         ).unwrap();
         let map = &tf.functions[0].1.map;
 

From ca9da7702edced4dba677906e9ae7f0b86f9bf90 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 4 Sep 2018 21:09:04 -0700
Subject: [PATCH 2049/3084] Reorganize the global value kinds. (#490)

* Reorganize the global value kinds.

This:
 - renames "deref" global values to "load" and gives it a offset that works
   like the "load" instructions' does
 - adds an explicit "iadd_imm" global value kind, which replaces the
   builtin iadd in "vmctx" and "deref" global values.
 - also renames "globalsym" to "symbol"
---
 cranelift/docs/heapex-dyn.clif                |   7 +-
 cranelift/docs/heapex-sm32.clif               |   5 +-
 cranelift/docs/heapex-sm64.clif               |   5 +-
 cranelift/docs/ir.rst                         |  56 +++++-----
 cranelift/filetests/isa/x86/binary32.clif     |   6 +-
 cranelift/filetests/isa/x86/binary64-pic.clif |  16 +--
 cranelift/filetests/isa/x86/binary64.clif     |   8 +-
 .../filetests/isa/x86/legalize-heaps.clif     |  64 ++++++-----
 .../filetests/isa/x86/legalize-memory.clif    |  38 ++++---
 .../filetests/isa/x86/legalize-table.clif     |  27 -----
 .../filetests/isa/x86/legalize-tables.clif    |  58 +++++-----
 cranelift/filetests/parser/memory.clif        |  48 +++++----
 .../filetests/regalloc/coalescing-207.clif    |   3 +-
 cranelift/filetests/regalloc/iterate.clif     |   3 +-
 cranelift/filetests/regalloc/reload-208.clif  |   3 +-
 cranelift/filetests/verifier/globals.clif     |  10 +-
 cranelift/filetests/verifier/heap.clif        |  14 +--
 cranelift/filetests/verifier/memory.clif      |   8 +-
 cranelift/filetests/verifier/table.clif       |  16 +--
 .../filetests/verifier/undeclared_vmctx.clif  |   4 +-
 lib/codegen/meta-python/base/instructions.py  |   6 +-
 lib/codegen/meta-python/isa/x86/encodings.py  |   8 +-
 lib/codegen/src/ir/globalvalue.rs             |  77 +++++++++-----
 lib/codegen/src/ir/immediates.rs              |  10 ++
 lib/codegen/src/legalizer/globalvalue.rs      |  81 ++++++++++----
 lib/codegen/src/predicates.rs                 |   2 +-
 lib/codegen/src/verifier/mod.rs               |  59 ++++++++---
 lib/module/src/module.rs                      |   3 +-
 lib/reader/src/parser.rs                      | 100 +++++++++++++-----
 lib/wasm/src/environ/dummy.rs                 |  42 ++++----
 30 files changed, 467 insertions(+), 320 deletions(-)
 delete mode 100644 cranelift/filetests/isa/x86/legalize-table.clif

diff --git a/cranelift/docs/heapex-dyn.clif b/cranelift/docs/heapex-dyn.clif
index 7ad0cd933f..276cbdd491 100644
--- a/cranelift/docs/heapex-dyn.clif
+++ b/cranelift/docs/heapex-dyn.clif
@@ -1,9 +1,10 @@
 test verifier
 
 function %add_members(i32, i64 vmctx) -> f32 baldrdash {
-    gv0 = vmctx+64
-    gv1 = vmctx+72
-    heap0 = dynamic gv0, min 0x1000, bound gv1, guard 0
+    gv0 = vmctx
+    gv1 = load.i64 notrap aligned gv0+64
+    gv2 = load.i32 notrap aligned gv0+72
+    heap0 = dynamic gv1, min 0x1000, bound gv2, guard 0
 
 ebb0(v0: i32, v6: i64):
     v1 = heap_addr.i64 heap0, v0, 20
diff --git a/cranelift/docs/heapex-sm32.clif b/cranelift/docs/heapex-sm32.clif
index 9ac1c8bfbd..88b78e9ea7 100644
--- a/cranelift/docs/heapex-sm32.clif
+++ b/cranelift/docs/heapex-sm32.clif
@@ -1,8 +1,9 @@
 test verifier
 
 function %add_members(i32, i32 vmctx) -> f32 baldrdash {
-    gv0 = vmctx+64
-    heap0 = static gv0, min 0x1000, bound 0x10_0000, guard 0x1000
+    gv0 = vmctx
+    gv1 = load.i32 notrap aligned gv0+64
+    heap0 = static gv1, min 0x1000, bound 0x10_0000, guard 0x1000
 
 ebb0(v0: i32, v5: i32):
     v1 = heap_addr.i32 heap0, v0, 1
diff --git a/cranelift/docs/heapex-sm64.clif b/cranelift/docs/heapex-sm64.clif
index c9057b6bb0..1c14ea6392 100644
--- a/cranelift/docs/heapex-sm64.clif
+++ b/cranelift/docs/heapex-sm64.clif
@@ -1,8 +1,9 @@
 test verifier
 
 function %add_members(i32, i64 vmctx) -> f32 baldrdash {
-    gv0 = vmctx+64
-    heap0 = static gv0, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000
+    gv0 = vmctx
+    gv1 = load.i64 notrap aligned gv0+64
+    heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000
 
 ebb0(v0: i32, v5: i64):
     v1 = heap_addr.i64 heap0, v0, 1
diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index 65d538d66b..775362ad69 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -562,58 +562,58 @@ to have access to a *VM context pointer* which is used as the base address.
 Typically, the VM context pointer is passed as a hidden function argument to
 Cranelift functions.
 
-.. inst:: GV = vmctx+Offset
+Chains of global value expressions are possible, but cycles are not allowed.
+They will be caught by the IR verifier.
 
-    Declare a global value of the address of a field in the VM context struct.
+.. inst:: GV = vmctx
 
-    This declares a global value which is a constant offset from the
-    VM context pointer which is passed as a hidden argument to all functions
-    JIT-compiled for the VM.
+    Declare a global value of the address of the VM context struct.
 
-    Typically, the VM context is a C struct, and the declared global value
-    is the address of a member of the struct.
+    This declares a global value which is the VM context pointer which may
+    be passed as a hidden argument to functions JIT-compiled for a VM.
+
+    Typically, the VM context is a `#[repr(C, packed)]` struct.
 
-    :arg Offset: Byte offset from the VM context pointer to the global
-                 value.
     :result GV: Global value.
 
-The address of a global value can also be derived by treating another global
-variable as a struct pointer. This makes it possible to chase pointers into VM
-runtime data structures.
+A global value can also be derived by treating another global variable as a
+struct pointer and loading from one of its fields. This makes it possible to
+chase pointers into VM runtime data structures.
 
-.. inst:: GV = deref(BaseGV)+Offset
+.. inst:: GV = load.Type BaseGV [Offset]
 
-    Declare a global value in a struct pointed to by BaseGV.
+    Declare a global value pointed to by BaseGV plus Offset, with type Type.
 
-    The address of GV can be computed by first loading a pointer from BaseGV
-    and adding Offset to it.
-
-    It is assumed the BaseGV resides in accessible memory with the appropriate
-    alignment for storing a pointer.
-
-    Chains of ``deref`` global values are possible, but cycles are not
-    allowed. They will be caught by the IR verifier.
+    It is assumed the BaseGV plus Offset resides in accessible memory with the
+    appropriate alignment for storing a value with type Type.
 
     :arg BaseGV: Global value providing the base pointer.
-    :arg Offset: Byte offset added to the loaded value.
+    :arg Offset: Offset added to the base before loading.
     :result GV: Global value.
 
-.. inst:: GV = [colocated] globalsym name
+.. inst:: GV = iadd_imm BaseGV, Offset
 
-    Declare a global value at a symbolic address.
+    Declare a global value which has the value of BaseGV offset by Offset.
 
-    The address of GV is symbolic and will be assigned a relocation, so that
+    :arg BaseGV: Global value providing the base value.
+    :arg Offset: Offset added to the base value.
+
+.. inst:: GV = [colocated] symbol Name
+
+    Declare a symbolic address global value.
+
+    The value of GV is symbolic and will be assigned a relocation, so that
     it can be resolved by a later linking phase.
 
     If the colocated keyword is present, the symbol's definition will be
     defined along with the current function, such that it can use more
     efficient addressing.
 
-    :arg name: External name.
+    :arg Name: External name.
     :result GV: Global value.
 
 .. autoinst:: global_value
-.. autoinst:: globalsym_addr
+.. autoinst:: symbol_value
 
 
 Heaps
diff --git a/cranelift/filetests/isa/x86/binary32.clif b/cranelift/filetests/isa/x86/binary32.clif
index 3f14b74c80..fec3479e41 100644
--- a/cranelift/filetests/isa/x86/binary32.clif
+++ b/cranelift/filetests/isa/x86/binary32.clif
@@ -12,7 +12,7 @@ function %I32() {
     sig0 = ()
     fn0 = %foo()
 
-    gv0 = globalsym %some_gv
+    gv0 = symbol %some_gv
 
     ss0 = incoming_arg 8, offset 0
     ss1 = incoming_arg 1024, offset -1024
@@ -365,9 +365,9 @@ ebb0:
     call_indirect sig0, v401()                  ; bin: stk_ovf ff d6
 
     ; asm: movl $0, %ecx
-    [-,%rcx]            v450 = globalsym_addr.i32 gv0    ; bin: b9 Abs4(%some_gv) 00000000
+    [-,%rcx]            v450 = symbol_value.i32 gv0    ; bin: b9 Abs4(%some_gv) 00000000
     ; asm: movl $0, %esi
-    [-,%rsi]            v451 = globalsym_addr.i32 gv0    ; bin: be Abs4(%some_gv) 00000000
+    [-,%rsi]            v451 = symbol_value.i32 gv0    ; bin: be Abs4(%some_gv) 00000000
 
     ; Spill / Fill.
 
diff --git a/cranelift/filetests/isa/x86/binary64-pic.clif b/cranelift/filetests/isa/x86/binary64-pic.clif
index 3dec2af62b..adda09da70 100644
--- a/cranelift/filetests/isa/x86/binary64-pic.clif
+++ b/cranelift/filetests/isa/x86/binary64-pic.clif
@@ -15,8 +15,8 @@ function %I64() {
     fn0 = %foo()
     fn1 = colocated %bar()
 
-    gv0 = globalsym %some_gv
-    gv1 = globalsym colocated %some_gv
+    gv0 = symbol %some_gv
+    gv1 = symbol colocated %some_gv
 
     ; Use incoming_arg stack slots because they won't be relocated by the frame
     ; layout.
@@ -66,18 +66,18 @@ ebb0:
     call_indirect sig0, v102()                  ; bin: stk_ovf 41 ff d2
 
     ; asm: mov 0x0(%rip), %rcx
-    [-,%rcx]            v3 = globalsym_addr.i64 gv0    ; bin: 48 8b 0d GOTPCRel4(%some_gv-4) 00000000
+    [-,%rcx]            v3 = symbol_value.i64 gv0    ; bin: 48 8b 0d GOTPCRel4(%some_gv-4) 00000000
     ; asm: mov 0x0(%rip), %rsi
-    [-,%rsi]            v4 = globalsym_addr.i64 gv0    ; bin: 48 8b 35 GOTPCRel4(%some_gv-4) 00000000
+    [-,%rsi]            v4 = symbol_value.i64 gv0    ; bin: 48 8b 35 GOTPCRel4(%some_gv-4) 00000000
     ; asm: mov 0x0(%rip), %r10
-    [-,%r10]            v5 = globalsym_addr.i64 gv0    ; bin: 4c 8b 15 GOTPCRel4(%some_gv-4) 00000000
+    [-,%r10]            v5 = symbol_value.i64 gv0    ; bin: 4c 8b 15 GOTPCRel4(%some_gv-4) 00000000
 
     ; asm: lea 0x0(%rip), %rcx
-    [-,%rcx]            v6 = globalsym_addr.i64 gv1    ; bin: 48 8d 0d PCRel4(%some_gv-4) 00000000
+    [-,%rcx]            v6 = symbol_value.i64 gv1    ; bin: 48 8d 0d PCRel4(%some_gv-4) 00000000
     ; asm: lea 0x0(%rip), %rsi
-    [-,%rsi]            v7 = globalsym_addr.i64 gv1    ; bin: 48 8d 35 PCRel4(%some_gv-4) 00000000
+    [-,%rsi]            v7 = symbol_value.i64 gv1    ; bin: 48 8d 35 PCRel4(%some_gv-4) 00000000
     ; asm: lea 0x0(%rip), %r10
-    [-,%r10]            v8 = globalsym_addr.i64 gv1    ; bin: 4c 8d 15 PCRel4(%some_gv-4) 00000000
+    [-,%r10]            v8 = symbol_value.i64 gv1    ; bin: 4c 8d 15 PCRel4(%some_gv-4) 00000000
 
     return
 }
diff --git a/cranelift/filetests/isa/x86/binary64.clif b/cranelift/filetests/isa/x86/binary64.clif
index 0aac9bbafc..b50e0d33db 100644
--- a/cranelift/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/isa/x86/binary64.clif
@@ -14,7 +14,7 @@ function %I64() {
     fn0 = %foo()
     fn1 = colocated %bar()
 
-    gv0 = globalsym %some_gv
+    gv0 = symbol %some_gv
 
     ; Use incoming_arg stack slots because they won't be relocated by the frame
     ; layout.
@@ -518,11 +518,11 @@ ebb0:
     call_indirect sig0, v412()                  ; bin: stk_ovf 41 ff d2
 
     ; asm: movabsq $-1, %rcx
-    [-,%rcx]            v450 = globalsym_addr.i64 gv0    ; bin: 48 b9 Abs8(%some_gv) 0000000000000000
+    [-,%rcx]            v450 = symbol_value.i64 gv0    ; bin: 48 b9 Abs8(%some_gv) 0000000000000000
     ; asm: movabsq $-1, %rsi
-    [-,%rsi]            v451 = globalsym_addr.i64 gv0    ; bin: 48 be Abs8(%some_gv) 0000000000000000
+    [-,%rsi]            v451 = symbol_value.i64 gv0    ; bin: 48 be Abs8(%some_gv) 0000000000000000
     ; asm: movabsq $-1, %r10
-    [-,%r10]            v452 = globalsym_addr.i64 gv0    ; bin: 49 ba Abs8(%some_gv) 0000000000000000
+    [-,%r10]            v452 = symbol_value.i64 gv0    ; bin: 49 ba Abs8(%some_gv) 0000000000000000
 
     ; Spill / Fill.
 
diff --git a/cranelift/filetests/isa/x86/legalize-heaps.clif b/cranelift/filetests/isa/x86/legalize-heaps.clif
index 8fe2b8737c..df2be3f287 100644
--- a/cranelift/filetests/isa/x86/legalize-heaps.clif
+++ b/cranelift/filetests/isa/x86/legalize-heaps.clif
@@ -4,18 +4,18 @@ target x86_64
 ; Test legalization for various forms of heap addresses.
 
 function %heap_addrs(i32, i64, i64 vmctx) {
-    gv0 = vmctx+64
-    gv1 = vmctx+72
-    gv2 = vmctx+80
-    gv3 = vmctx+88
-    gv4 = deref(gv3): i32
+    gv4 = vmctx
+    gv0 = iadd_imm.i64 gv4, 64
+    gv1 = iadd_imm.i64 gv4, 72
+    gv2 = iadd_imm.i64 gv4, 80
+    gv3 = load.i32 notrap aligned gv4+88
 
     heap0 = static gv0, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000, index_type i32
     heap1 = static gv0, guard 0x1000, bound 0x1_0000, index_type i32
     heap2 = static gv0, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000, index_type i64
     heap3 = static gv0, guard 0x1000, bound 0x1_0000, index_type i64
-    heap4 = dynamic gv1, min 0x1_0000, bound gv4, guard 0x8000_0000, index_type i32
-    heap5 = dynamic gv1, bound gv4, guard 0x1000, index_type i32
+    heap4 = dynamic gv1, min 0x1_0000, bound gv3, guard 0x8000_0000, index_type i32
+    heap5 = dynamic gv1, bound gv3, guard 0x1000, index_type i32
     heap6 = dynamic gv1, min 0x1_0000, bound gv2, guard 0x8000_0000, index_type i64
     heap7 = dynamic gv1, bound gv2, guard 0x1000, index_type i64
 
@@ -23,8 +23,8 @@ function %heap_addrs(i32, i64, i64 vmctx) {
     ; check: heap1 = static gv0, min 0, bound 0x0001_0000, guard 4096, index_type i32
     ; check: heap2 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000, index_type i64
     ; check: heap3 = static gv0, min 0, bound 0x0001_0000, guard 4096, index_type i64
-    ; check: heap4 = dynamic gv1, min 0x0001_0000, bound gv4, guard 0x8000_0000, index_type i32
-    ; check: heap5 = dynamic gv1, min 0, bound gv4, guard 4096, index_type i32
+    ; check: heap4 = dynamic gv1, min 0x0001_0000, bound gv3, guard 0x8000_0000, index_type i32
+    ; check: heap5 = dynamic gv1, min 0, bound gv3, guard 4096, index_type i32
     ; check: heap6 = dynamic gv1, min 0x0001_0000, bound gv2, guard 0x8000_0000, index_type i64
     ; check: heap7 = dynamic gv1, min 0, bound gv2, guard 4096, index_type i64
 
@@ -62,9 +62,7 @@ ebb0(v0: i32, v1: i64, v3: i64):
     ; check:         v7 = iadd v21, v1
 
     v8 = heap_addr.i64 heap4, v0, 0
-    ; check:         v27 = iadd_imm.i64 v3, 88
-    ; check:         v28 = load.i32 notrap aligned v27
-    ; check:         v22 = iadd_imm v28, 0
+    ; check:         v22 = load.i32 notrap aligned v3+88
     ; check:         v23 = iadd_imm v22, 0
     ; check:         v24 = icmp.i32 ugt v0, v23
     ; check:         brz v24, ebb4
@@ -75,37 +73,35 @@ ebb0(v0: i32, v1: i64, v3: i64):
     ; check:         v8 = iadd v26, v25
 
     v9 = heap_addr.i64 heap5, v0, 0
-    ; check:         v34 = iadd_imm.i64 v3, 88
-    ; check:         v35 = load.i32 notrap aligned v34
-    ; check:         v29 = iadd_imm v35, 0
-    ; check:         v30 = iadd_imm v29, 0
-    ; check:         v31 = icmp.i32 ugt v0, v30
-    ; check:         brz v31, ebb5
+    ; check:         v27 = load.i32 notrap aligned v3+88
+    ; check:         v28 = iadd_imm v27, 0
+    ; check:         v29 = icmp.i32 ugt v0, v28
+    ; check:         brz v29, ebb5
     ; check:         trap heap_oob
     ; check:     ebb5:
-    ; check:         v32 = uextend.i64 v0
-    ; check:         v33 = iadd_imm.i64 v3, 72
-    ; check:         v9 = iadd v33, v32
+    ; check:         v30 = uextend.i64 v0
+    ; check:         v31 = iadd_imm.i64 v3, 72
+    ; check:         v9 = iadd v31, v30
 
     v10 = heap_addr.i64 heap6, v1, 0
+    ; check:         v32 = iadd_imm.i64 v3, 80
+    ; check:         v33 = iadd_imm v32, 0
+    ; check:         v34 = icmp.i64 ugt v1, v33
+    ; check:         brz v34, ebb6
+    ; check:         trap heap_oob
+    ; check:     ebb6:
+    ; check:         v35 = iadd_imm.i64 v3, 72
+    ; check:         v10 = iadd v35, v1
+
+    v11 = heap_addr.i64 heap7, v1, 0
     ; check:         v36 = iadd_imm.i64 v3, 80
     ; check:         v37 = iadd_imm v36, 0
     ; check:         v38 = icmp.i64 ugt v1, v37
-    ; check:         brz v38, ebb6
-    ; check:         trap heap_oob
-    ; check:     ebb6:
-    ; check:         v39 = iadd_imm.i64 v3, 72
-    ; check:         v10 = iadd v39, v1
-
-    v11 = heap_addr.i64 heap7, v1, 0
-    ; check:         v40 = iadd_imm.i64 v3, 80
-    ; check:         v41 = iadd_imm v40, 0
-    ; check:         v42 = icmp.i64 ugt v1, v41
-    ; check:         brz v42, ebb7
+    ; check:         brz v38, ebb7
     ; check:         trap heap_oob
     ; check:     ebb7:
-    ; check:         v43 = iadd_imm.i64 v3, 72
-    ; check:         v11 = iadd v43, v1
+    ; check:         v39 = iadd_imm.i64 v3, 72
+    ; check:         v11 = iadd v39, v1
 
     return
 }
diff --git a/cranelift/filetests/isa/x86/legalize-memory.clif b/cranelift/filetests/isa/x86/legalize-memory.clif
index 3728e84b58..fbb9ca1f37 100644
--- a/cranelift/filetests/isa/x86/legalize-memory.clif
+++ b/cranelift/filetests/isa/x86/legalize-memory.clif
@@ -6,7 +6,8 @@ target x86_64
 ; regex: EBB=ebb\d+
 
 function %vmctx(i64 vmctx) -> i64 {
-    gv1 = vmctx-16
+    gv0 = vmctx
+    gv1 = iadd_imm.i64 gv0, -16
 
 ebb1(v1: i64):
     v2 = global_value.i64 gv1
@@ -15,28 +16,28 @@ ebb1(v1: i64):
     ; check: return v2
 }
 
-function %deref(i64 vmctx) -> i64 {
-    gv1 = vmctx-16
-    gv2 = deref(gv1)+32: i64
+function %load(i64 vmctx) -> i64 {
+    gv0 = vmctx
+    gv1 = load.i64 notrap aligned gv0-16
+    gv2 = iadd_imm.i64 gv1, 32
 
 ebb1(v1: i64):
     v2 = global_value.i64 gv2
-    ; check: $(a1=$V) = iadd_imm v1, -16
-    ; check: $(p1=$V) = load.i64 notrap aligned $a1
+    ; check: $(p1=$V) = load.i64 notrap aligned v1-16
     ; check: v2 = iadd_imm $p1, 32
     return v2
     ; check: return v2
 }
 
-function %sym() -> i64 {
-    gv0 = globalsym %something
-    gv1 = globalsym u123:456
+function %symbol() -> i64 {
+    gv0 = symbol %something
+    gv1 = symbol u123:456
 
 ebb1:
     v0 = global_value.i64 gv0
-    ; check: v0 = globalsym_addr.i64 gv0
+    ; check: v0 = symbol_value.i64 gv0
     v1 = global_value.i64 gv1
-    ; check: v1 = globalsym_addr.i64 gv1
+    ; check: v1 = symbol_value.i64 gv1
     v2 = bxor v0, v1
     return v2
 }
@@ -44,8 +45,9 @@ ebb1:
 ; SpiderMonkey VM-style static 4+2 GB heap.
 ; This eliminates bounds checks completely for offsets < 2GB.
 function %staticheap_sm64(i32, i64 vmctx) -> f32 baldrdash {
-    gv0 = vmctx+64
-    heap0 = static gv0, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000
+    gv0 = vmctx
+    gv1 = iadd_imm.i64 gv0, 64
+    heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000
 
 ebb0(v0: i32, v999: i64):
     ; check: ebb0(
@@ -64,8 +66,9 @@ ebb0(v0: i32, v999: i64):
 }
 
 function %staticheap_static_oob_sm64(i32, i64 vmctx) -> f32 baldrdash {
-    gv0 = vmctx+64
-    heap0 = static gv0, min 0x1000, bound 0x1000_0000, guard 0x8000_0000
+    gv0 = vmctx
+    gv1 = iadd_imm.i64 gv0, 64
+    heap0 = static gv1, min 0x1000, bound 0x1000_0000, guard 0x8000_0000
 
 ebb0(v0: i32, v999: i64):
     ; Everything after the obviously OOB access should be eliminated, leaving
@@ -87,8 +90,9 @@ ebb0(v0: i32, v999: i64):
 ; SpiderMonkey VM-style static 4+2 GB heap.
 ; Offsets >= 2 GB do require a boundscheck.
 function %staticheap_sm64(i32, i64 vmctx) -> f32 baldrdash {
-    gv0 = vmctx+64
-    heap0 = static gv0, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000
+    gv0 = vmctx
+    gv1 = iadd_imm.i64 gv0, 64
+    heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000
 
 ebb0(v0: i32, v999: i64):
     ; check: ebb0(
diff --git a/cranelift/filetests/isa/x86/legalize-table.clif b/cranelift/filetests/isa/x86/legalize-table.clif
deleted file mode 100644
index f86913115d..0000000000
--- a/cranelift/filetests/isa/x86/legalize-table.clif
+++ /dev/null
@@ -1,27 +0,0 @@
-; Test legalization of tables
-test legalizer
-target x86_64
-
-; regex: V=v\d+
-; regex: EBB=ebb\d+
-
-function %test0(i64 vmctx, i64) -> i64 {
-    gv0 = vmctx+12
-    gv1 = vmctx+14
-    table0 = dynamic gv0, min 20, bound gv1, element_size 4, index_type i64
-
-ebb0(v0: i64, v1: i64):
-    v2 = table_addr.i64 table0, v1, +3
-    return v2
-}
-
-; check:    $(bound=$V) = iadd_imm $(input=$V), 14
-; nextln:   $(cond=$V) = icmp uge $(limit=$V), $bound
-; nextln:   brz $cond, ebb1
-; nextln:   trap table_oob
-; nextln: 
-; nextln: ebb1:
-; nextln:   $(base=$V) = iadd_imm.i64 $(vmctx=$V), 12
-; nextln:   $(scaled=$V) = ishl_imm.i64 $(index=$V), 2
-; nextln:   $(elem_addr=$V) = iadd $base, $scaled
-; nextln:   $(field_addr=$V) = iadd_imm $elem_addr, 3
diff --git a/cranelift/filetests/isa/x86/legalize-tables.clif b/cranelift/filetests/isa/x86/legalize-tables.clif
index a9882f3cf9..5995c230e0 100644
--- a/cranelift/filetests/isa/x86/legalize-tables.clif
+++ b/cranelift/filetests/isa/x86/legalize-tables.clif
@@ -4,26 +4,24 @@ target x86_64
 ; Test legalization for various forms of table addresses.
 
 function %table_addrs(i32, i64, i64 vmctx) {
-    gv0 = vmctx+72
-    gv1 = vmctx+80
-    gv2 = vmctx+88
-    gv3 = deref(gv2): i32
+    gv4 = vmctx
+    gv0 = iadd_imm.i64 gv4, 72
+    gv1 = iadd_imm.i64 gv4, 80
+    gv2 = load.i32 notrap aligned gv4+88
 
-    table0 = dynamic gv0, min 0x1_0000, bound gv3, element_size 1, index_type i32
-    table1 = dynamic gv0, bound gv3, element_size 16, index_type i32
+    table0 = dynamic gv0, min 0x1_0000, bound gv2, element_size 1, index_type i32
+    table1 = dynamic gv0, bound gv2, element_size 16, index_type i32
     table2 = dynamic gv0, min 0x1_0000, bound gv1, element_size 1, index_type i64
     table3 = dynamic gv0, bound gv1, element_size 16, index_type i64
 
-    ; check: table0 = dynamic gv0, min 0x0001_0000, bound gv3, element_size 1, index_type i32
-    ; check: table1 = dynamic gv0, min 0, bound gv3, element_size 16, index_type i32
+    ; check: table0 = dynamic gv0, min 0x0001_0000, bound gv2, element_size 1, index_type i32
+    ; check: table1 = dynamic gv0, min 0, bound gv2, element_size 16, index_type i32
     ; check: table2 = dynamic gv0, min 0x0001_0000, bound gv1, element_size 1, index_type i64
     ; check: table3 = dynamic gv0, min 0, bound gv1, element_size 16, index_type i64
 
 ebb0(v0: i32, v1: i64, v3: i64):
     v4 = table_addr.i64 table0, v0, +0
-    ; check:         v12 = iadd_imm v3, 88
-    ; check:         v13 = load.i32 notrap aligned v12
-    ; check:         v8 = iadd_imm v13, 0
+    ; check:         v8 = load.i32 notrap aligned v3+88
     ; check:         v9 = icmp uge v0, v8
     ; check:         brz v9, ebb1
     ; check:         trap table_oob
@@ -33,36 +31,34 @@ ebb0(v0: i32, v1: i64, v3: i64):
     ; check:         v4 = iadd v11, v10
 
     v5 = table_addr.i64 table1, v0, +0
-    ; check:         v19 = iadd_imm.i64 v3, 88
-    ; check:         v20 = load.i32 notrap aligned v19
-    ; check:         v14 = iadd_imm v20, 0
-    ; check:         v15 = icmp.i32 uge v0, v14
-    ; check:         brz v15, ebb2
+    ; check:         v12 = load.i32 notrap aligned v3+88
+    ; check:         v13 = icmp.i32 uge v0, v12
+    ; check:         brz v13, ebb2
     ; check:         trap table_oob
     ; check:     ebb2:
-    ; check:         v16 = uextend.i64 v0
-    ; check:         v17 = iadd_imm.i64 v3, 72
-    ; check:         v18 = ishl_imm v16, 4
-    ; check:         v5 = iadd v17, v18
+    ; check:         v14 = uextend.i64 v0
+    ; check:         v15 = iadd_imm.i64 v3, 72
+    ; check:         v16 = ishl_imm v14, 4
+    ; check:         v5 = iadd v15, v16
 
     v6 = table_addr.i64 table2, v1, +0
-    ; check:         v21 = iadd_imm.i64 v3, 80
-    ; check:         v22 = icmp.i64 uge v1, v21
-    ; check:         brz v22, ebb3
+    ; check:         v17 = iadd_imm.i64 v3, 80
+    ; check:         v18 = icmp.i64 uge v1, v17
+    ; check:         brz v18, ebb3
     ; check:         trap table_oob
     ; check:     ebb3:
-    ; check:         v23 = iadd_imm.i64 v3, 72
-    ; check:         v6 = iadd v23, v1
+    ; check:         v19 = iadd_imm.i64 v3, 72
+    ; check:         v6 = iadd v19, v1
 
     v7 = table_addr.i64 table3, v1, +0
-    ; check:         v24 = iadd_imm.i64 v3, 80
-    ; check:         v25 = icmp.i64 uge v1, v24
-    ; check:         brz v25, ebb4
+    ; check:         v20 = iadd_imm.i64 v3, 80
+    ; check:         v21 = icmp.i64 uge v1, v20
+    ; check:         brz v21, ebb4
     ; check:         trap table_oob
     ; check:     ebb4:
-    ; check:         v26 = iadd_imm.i64 v3, 72
-    ; check:         v27 = ishl_imm.i64 v1, 4
-    ; check:         v7 = iadd v26, v27
+    ; check:         v22 = iadd_imm.i64 v3, 72
+    ; check:         v23 = ishl_imm.i64 v1, 4
+    ; check:         v7 = iadd v22, v23
 
     return
 }
diff --git a/cranelift/filetests/parser/memory.clif b/cranelift/filetests/parser/memory.clif
index 1c2fe4cc4c..d406007dfe 100644
--- a/cranelift/filetests/parser/memory.clif
+++ b/cranelift/filetests/parser/memory.clif
@@ -2,23 +2,21 @@ test cat
 test verifier
 
 function %vmglobal(i64 vmctx) -> i32 {
-    gv3 = vmctx+16
-    ; check: gv3 = vmctx+16
-    gv4 = vmctx+0
-    ; check: gv4 = vmctx
-    ; not: +0
-    gv5 = vmctx -256
-    ; check: gv5 = vmctx-256
+    gv3 = vmctx
+    ; check: gv3 = vmctx
 ebb0(v0: i64):
     v1 = global_value.i32 gv3
     ; check: v1 = global_value.i32 gv3
     return v1
 }
 
-function %deref(i64 vmctx) -> i32 {
-    gv3 = vmctx+16
-    gv4 = deref(gv3)-32: i32
-    ; check: gv4 = deref(gv3)-32
+function %load_and_add_imm(i64 vmctx) -> i32 {
+    gv2 = vmctx
+    gv3 = load.i32 notrap aligned gv2-72
+    gv4 = iadd_imm.i32 gv3, -32
+    ; check: gv2 = vmctx
+    ; check: gv3 = load.i32 notrap aligned gv2-72
+    ; check: gv4 = iadd_imm.i32 gv3, -32
 ebb0(v0: i64):
     v1 = global_value.i32 gv4
     ; check: v1 = global_value.i32 gv4
@@ -27,20 +25,22 @@ ebb0(v0: i64):
 
 ; Refer to a global value before it's been declared.
 function %backref(i64 vmctx) -> i32 {
-    gv1 = deref(gv2)-32: i32
-    ; check: gv1 = deref(gv2)-32
-    gv2 = vmctx+16
-    ; check: gv2 = vmctx+16
+    gv0 = iadd_imm.i32 gv1, -32
+    ; check: gv0 = iadd_imm.i32 gv1, -32
+    gv1 = load.i32 notrap aligned gv2
+    ; check: gv1 = load.i32 notrap aligned gv2
+    gv2 = vmctx
+    ; check: gv2 = vmctx
 ebb0(v0: i64):
     v1 = global_value.i32 gv1
     return v1
 }
 
-function %sym() -> i32 {
-    gv0 = globalsym %something
-    ; check: gv0 = globalsym %something
-    gv1 = globalsym u8:9
-    ; check: gv1 = globalsym u8:9
+function %symbol() -> i32 {
+    gv0 = symbol %something
+    ; check: gv0 = symbol %something
+    gv1 = symbol u8:9
+    ; check: gv1 = symbol u8:9
 ebb0:
     v0 = global_value.i32 gv0
     ; check: v0 = global_value.i32 gv0
@@ -54,7 +54,8 @@ ebb0:
 function %sheap(i32, i64 vmctx) -> i64 {
     heap1 = static gv5, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000
     heap2 = static gv5, guard 0x1000, bound 0x1_0000
-    gv5 = vmctx+64
+    gv4 = vmctx
+    gv5 = iadd_imm.i64 gv4, 64
 
     ; check: heap1 = static gv5, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
     ; check: heap2 = static gv5, min 0, bound 0x0001_0000, guard 4096
@@ -68,8 +69,9 @@ ebb0(v1: i32, v2: i64):
 function %dheap(i32, i64 vmctx) -> i64 {
     heap1 = dynamic gv5, min 0x1_0000, bound gv6, guard 0x8000_0000
     heap2 = dynamic gv5, bound gv6, guard 0x1000
-    gv5 = vmctx+64
-    gv6 = vmctx+72
+    gv4 = vmctx
+    gv5 = iadd_imm.i64 gv4, 64
+    gv6 = iadd_imm.i64 gv4, 72
 
     ; check: heap1 = dynamic gv5, min 0x0001_0000, bound gv6, guard 0x8000_0000
     ; check: heap2 = dynamic gv5, min 0, bound gv6, guard 4096
diff --git a/cranelift/filetests/regalloc/coalescing-207.clif b/cranelift/filetests/regalloc/coalescing-207.clif
index 08519965b6..37b028af03 100644
--- a/cranelift/filetests/regalloc/coalescing-207.clif
+++ b/cranelift/filetests/regalloc/coalescing-207.clif
@@ -5,7 +5,8 @@ target x86_64 haswell
 ;
 ; The coalescer creates a virtual register with two interfering values.
 function %pr207(i64 vmctx, i32, i32) -> i32 system_v {
-    gv0 = vmctx-8
+    gv1 = vmctx
+    gv0 = iadd_imm.i64 gv1, -8
     heap0 = static gv0, min 0, bound 0x5000, guard 0x0040_0000
     sig0 = (i64 vmctx, i32, i32) -> i32 system_v
     sig1 = (i64 vmctx, i32, i32, i32) -> i32 system_v
diff --git a/cranelift/filetests/regalloc/iterate.clif b/cranelift/filetests/regalloc/iterate.clif
index c7ad214432..424ecfa740 100644
--- a/cranelift/filetests/regalloc/iterate.clif
+++ b/cranelift/filetests/regalloc/iterate.clif
@@ -104,7 +104,8 @@ ebb1(v31: i64):
 }
 
 function u0:26(i64 vmctx [%r14]) -> i64 [%rax] baldrdash {
-    gv0 = vmctx+48
+    gv1 = vmctx
+    gv0 = iadd_imm.i64 gv1, 48
     sig0 = (i32 [%rdi], i64 [%rsi], i64 vmctx [%r14], i64 sigid [%rbx]) -> i64 [%rax] baldrdash
 
 ebb0(v0: i64):
diff --git a/cranelift/filetests/regalloc/reload-208.clif b/cranelift/filetests/regalloc/reload-208.clif
index 0f4a2e4bb4..629d984f62 100644
--- a/cranelift/filetests/regalloc/reload-208.clif
+++ b/cranelift/filetests/regalloc/reload-208.clif
@@ -11,7 +11,8 @@ target x86_64 haswell
 ; The problem was the reload pass rewriting EBB arguments on "brnz v9, ebb3(v9)"
 
 function %pr208(i64 vmctx [%rdi]) system_v {
-    gv0 = vmctx-8
+    gv1 = vmctx
+    gv0 = iadd_imm.i64 gv1, -8
     heap0 = static gv0, min 0, bound 0x5000, guard 0x0040_0000
     sig0 = (i64 vmctx [%rdi]) -> i32 [%rax] system_v
     sig1 = (i64 vmctx [%rdi], i32 [%rsi]) system_v
diff --git a/cranelift/filetests/verifier/globals.clif b/cranelift/filetests/verifier/globals.clif
index f5f99a95ae..4882cae2ee 100644
--- a/cranelift/filetests/verifier/globals.clif
+++ b/cranelift/filetests/verifier/globals.clif
@@ -1,17 +1,17 @@
 test verifier
 target x86_64
 
-function %deref_base_type(i64 vmctx) {
-    gv0 = vmctx+0
-    gv1 = deref(gv0): i32
-    gv2 = deref(gv1): i32 ; error: deref base gv1 has type i32, which is not the pointer type i64
+function %load_base_type(i64 vmctx) {
+    gv0 = vmctx
+    gv1 = load.i32 notrap aligned gv0
+    gv2 = load.i32 notrap aligned gv1 ; error: base gv1 has type i32, which is not the pointer type i64
 
 ebb0(v0: i64):
     return
 }
 
 function %global_value_wrong_type(i64 vmctx) {
-    gv0 = vmctx+0
+    gv0 = vmctx
 
 ebb0(v0: i64):
     v1 = global_value.i32 gv0 ; error: global_value instruction with type i32 references global value with type i64
diff --git a/cranelift/filetests/verifier/heap.clif b/cranelift/filetests/verifier/heap.clif
index 68d13e66f2..a436c623d8 100644
--- a/cranelift/filetests/verifier/heap.clif
+++ b/cranelift/filetests/verifier/heap.clif
@@ -2,8 +2,8 @@ test verifier
 target x86_64
 
 function %heap_base_type(i64 vmctx) {
-    gv0 = vmctx+0
-    gv1 = deref(gv0): i32
+    gv0 = vmctx
+    gv1 = load.i32 notrap aligned gv0
     heap0 = static gv1, guard 0x1000, bound 0x1_0000, index_type i32 ; error: heap base has type i32, which is not the pointer type i64
 
 ebb0(v0: i64):
@@ -11,7 +11,7 @@ ebb0(v0: i64):
 }
 
 function %invalid_base(i64 vmctx) {
-    gv0 = vmctx+0
+    gv0 = vmctx
     heap0 = dynamic gv1, bound gv0, guard 0x1000, index_type i64 ; error: invalid base global value gv1
 
 ebb0(v0: i64):
@@ -19,7 +19,7 @@ ebb0(v0: i64):
 }
 
 function %invalid_bound(i64 vmctx) {
-    gv0 = vmctx+0
+    gv0 = vmctx
     heap0 = dynamic gv0, bound gv1, guard 0x1000, index_type i64 ; error: invalid bound global value gv1
 
 ebb0(v0: i64):
@@ -27,8 +27,8 @@ ebb0(v0: i64):
 }
 
 function %heap_bound_type(i64 vmctx) {
-    gv0 = vmctx+0
-    gv1 = deref(gv0): i16
+    gv0 = vmctx
+    gv1 = load.i16 notrap aligned gv0
     heap0 = dynamic gv0, bound gv1, guard 0x1000, index_type i32 ; error: heap index type i32 differs from the type of its bound, i16
 
 ebb0(v0: i64):
@@ -36,7 +36,7 @@ ebb0(v0: i64):
 }
 
 function %heap_addr_index_type(i64 vmctx, i64) {
-    gv0 = vmctx+0
+    gv0 = vmctx
     heap0 = static gv0, guard 0x1000, bound 0x1_0000, index_type i32
 
 ebb0(v0: i64, v1: i64):
diff --git a/cranelift/filetests/verifier/memory.clif b/cranelift/filetests/verifier/memory.clif
index a0bcaa11a1..cbaddcb13b 100644
--- a/cranelift/filetests/verifier/memory.clif
+++ b/cranelift/filetests/verifier/memory.clif
@@ -1,15 +1,15 @@
 test verifier
 
-function %deref_cycle() {
-    gv1 = deref(gv2)-32: i32 ; error: deref cycle: [gv1, gv2]
-    gv2 = deref(gv1): i32
+function %cycle() {
+    gv0 = load.i32 notrap aligned gv1 ; error: global value cycle: [gv0, gv1]
+    gv1 = load.i32 notrap aligned gv0-32
 
 ebb1:
     return
 }
 
 function %self_cycle() {
-    gv0 = deref(gv0)-32: i32 ; error: deref cycle: [gv0]
+    gv0 = load.i32 notrap aligned gv0 ; error: global value cycle: [gv0]
 
 ebb1:
     return
diff --git a/cranelift/filetests/verifier/table.clif b/cranelift/filetests/verifier/table.clif
index 10fdaaf87f..7502e00044 100644
--- a/cranelift/filetests/verifier/table.clif
+++ b/cranelift/filetests/verifier/table.clif
@@ -2,8 +2,8 @@ test verifier
 target x86_64
 
 function %table_base_type(i64 vmctx) {
-    gv0 = vmctx+0
-    gv1 = deref(gv0): i32
+    gv0 = vmctx
+    gv1 = load.i32 notrap aligned gv0
     table0 = dynamic gv1, element_size 1, bound gv1, index_type i32 ; error: table base has type i32, which is not the pointer type i64
 
 ebb0(v0: i64):
@@ -11,7 +11,7 @@ ebb0(v0: i64):
 }
 
 function %invalid_base(i64 vmctx) {
-    gv0 = vmctx+0
+    gv0 = vmctx
     table0 = dynamic gv1, bound gv0, element_size 1, index_type i64 ; error: invalid base global value gv1
 
 ebb0(v0: i64):
@@ -19,7 +19,7 @@ ebb0(v0: i64):
 }
 
 function %invalid_bound(i64 vmctx) {
-    gv0 = vmctx+0
+    gv0 = vmctx
     table0 = dynamic gv0, bound gv1, element_size 1, index_type i64 ; error: invalid bound global value gv1
 
 ebb0(v0: i64):
@@ -27,8 +27,8 @@ ebb0(v0: i64):
 }
 
 function %table_bound_type(i64 vmctx) {
-    gv0 = vmctx+0
-    gv1 = deref(gv0): i16
+    gv0 = vmctx
+    gv1 = load.i16 notrap aligned gv0
     table0 = dynamic gv0, bound gv1, element_size 1, index_type i32 ; error: table index type i32 differs from the type of its bound, i16
 
 ebb0(v0: i64):
@@ -36,8 +36,8 @@ ebb0(v0: i64):
 }
 
 function %table_addr_index_type(i64 vmctx, i64) {
-    gv0 = vmctx+0
-    gv1 = deref(gv0): i32
+    gv0 = vmctx
+    gv1 = load.i32 notrap aligned gv0
     table0 = dynamic gv0, element_size 1, bound gv1, index_type i32
 
 ebb0(v0: i64, v1: i64):
diff --git a/cranelift/filetests/verifier/undeclared_vmctx.clif b/cranelift/filetests/verifier/undeclared_vmctx.clif
index 47f1192569..1b760b3ff3 100644
--- a/cranelift/filetests/verifier/undeclared_vmctx.clif
+++ b/cranelift/filetests/verifier/undeclared_vmctx.clif
@@ -2,7 +2,7 @@ test verifier
 
 ; Using a vmctx global value without declaring it first leads to an error.
 function %vmglobal_err(i64) -> i64 {
-    gv4 = vmctx+0 ; error: undeclared vmctx reference
+    gv4 = vmctx ; error: undeclared vmctx reference
 ebb0(v0: i64):
     v1 = global_value.i64 gv4
     return v1
@@ -10,7 +10,7 @@ ebb0(v0: i64):
 
 ; If it is declared, all is fine.
 function %vmglobal_ok(i64 vmctx) -> i64 {
-    gv4 = vmctx+0
+    gv4 = vmctx
 ebb0(v0: i64):
     v1 = global_value.i64 gv4
     return v1
diff --git a/lib/codegen/meta-python/base/instructions.py b/lib/codegen/meta-python/base/instructions.py
index 9f7caf5c48..0b48fb84f2 100644
--- a/lib/codegen/meta-python/base/instructions.py
+++ b/lib/codegen/meta-python/base/instructions.py
@@ -506,9 +506,9 @@ global_value = Instruction(
 
 # A specialized form of global_value instructions that only handles
 # symbolic names.
-globalsym_addr = Instruction(
-        'globalsym_addr', r"""
-        Compute the address of global GV, which is a symbolic name.
+symbol_value = Instruction(
+        'symbol_value', r"""
+        Compute the value of global GV, which is a symbolic address.
         """,
         ins=GV, outs=addr)
 
diff --git a/lib/codegen/meta-python/isa/x86/encodings.py b/lib/codegen/meta-python/isa/x86/encodings.py
index 1e406fb8fa..a5101261c6 100644
--- a/lib/codegen/meta-python/isa/x86/encodings.py
+++ b/lib/codegen/meta-python/isa/x86/encodings.py
@@ -428,18 +428,18 @@ X86_64.enc(base.func_addr.i64, *r.got_fnaddr8.rex(0x8b, w=1),
 #
 
 # Non-PIC
-X86_32.enc(base.globalsym_addr.i32, *r.gvaddr4(0xb8),
+X86_32.enc(base.symbol_value.i32, *r.gvaddr4(0xb8),
            isap=Not(is_pic))
-X86_64.enc(base.globalsym_addr.i64, *r.gvaddr8.rex(0xb8, w=1),
+X86_64.enc(base.symbol_value.i64, *r.gvaddr8.rex(0xb8, w=1),
            isap=Not(is_pic))
 
 # PIC, colocated
-X86_64.enc(base.globalsym_addr.i64, *r.pcrel_gvaddr8.rex(0x8d, w=1),
+X86_64.enc(base.symbol_value.i64, *r.pcrel_gvaddr8.rex(0x8d, w=1),
            isap=is_pic,
            instp=IsColocatedData())
 
 # PIC, non-colocated
-X86_64.enc(base.globalsym_addr.i64, *r.got_gvaddr8.rex(0x8b, w=1),
+X86_64.enc(base.symbol_value.i64, *r.got_gvaddr8.rex(0x8b, w=1),
            isap=is_pic)
 
 #
diff --git a/lib/codegen/src/ir/globalvalue.rs b/lib/codegen/src/ir/globalvalue.rs
index 8a119bce51..97ebdae262 100644
--- a/lib/codegen/src/ir/globalvalue.rs
+++ b/lib/codegen/src/ir/globalvalue.rs
@@ -1,6 +1,6 @@
 //! Global values.
 
-use ir::immediates::Offset32;
+use ir::immediates::{Imm64, Offset32};
 use ir::{ExternalName, GlobalValue, Type};
 use isa::TargetIsa;
 use std::fmt;
@@ -8,35 +8,47 @@ use std::fmt;
 /// Information about a global value declaration.
 #[derive(Clone)]
 pub enum GlobalValueData {
-    /// Value is the address of a field in the VM context struct, a constant offset from the VM
-    /// context pointer.
-    VMContext {
-        /// Offset from the `vmctx` pointer.
-        offset: Offset32,
-    },
+    /// Value is the address of the VM context struct.
+    VMContext,
 
     /// Value is pointed to by another global value.
     ///
     /// The `base` global value is assumed to contain a pointer. This global value is computed
-    /// by loading from memory at that pointer value, and then adding an offset. The memory must
-    /// be accessible, and naturally aligned to hold a value of the type.
-    Deref {
+    /// by loading from memory at that pointer value. The memory must be accessible, and
+    /// naturally aligned to hold a value of the type.
+    Load {
         /// The base pointer global value.
         base: GlobalValue,
 
-        /// Byte offset to be added to the loaded value.
+        /// Offset added to the base pointer before doing the load.
         offset: Offset32,
 
         /// Type of the loaded value.
-        memory_type: Type,
+        global_type: Type,
     },
 
-    /// Value is identified by a symbolic name. Cranelift itself does not interpret this name;
+    /// Value is an offset from another global value.
+    IAddImm {
+        /// The base pointer global value.
+        base: GlobalValue,
+
+        /// Byte offset to be added to the value.
+        offset: Imm64,
+
+        /// Type of the iadd.
+        global_type: Type,
+    },
+
+    /// Value is a symbolic address. Cranelift itself does not interpret this name;
     /// it's used by embedders to link with other data structures.
-    Sym {
+    Symbol {
         /// The symbolic name.
         name: ExternalName,
 
+        /// Offset from the symbol. This can be used instead of IAddImm to represent folding an
+        /// offset into a symbol.
+        offset: Imm64,
+
         /// Will this symbol be defined nearby, such that it will always be a certain distance
         /// away, after linking? If so, references to it can avoid going through a GOT. Note that
         /// symbols meant to be preemptible cannot be colocated.
@@ -45,10 +57,10 @@ pub enum GlobalValueData {
 }
 
 impl GlobalValueData {
-    /// Assume that `self` is an `GlobalValueData::Sym` and return its name.
+    /// Assume that `self` is an `GlobalValueData::Symbol` and return its name.
     pub fn symbol_name(&self) -> &ExternalName {
         match *self {
-            GlobalValueData::Sym { ref name, .. } => name,
+            GlobalValueData::Symbol { ref name, .. } => name,
             _ => panic!("only symbols have names"),
         }
     }
@@ -56,8 +68,11 @@ impl GlobalValueData {
     /// Return the type of this global.
     pub fn global_type(&self, isa: &TargetIsa) -> Type {
         match *self {
-            GlobalValueData::VMContext { .. } | GlobalValueData::Sym { .. } => isa.pointer_type(),
-            GlobalValueData::Deref { memory_type, .. } => memory_type,
+            GlobalValueData::VMContext { .. } | GlobalValueData::Symbol { .. } => {
+                isa.pointer_type()
+            }
+            GlobalValueData::IAddImm { global_type, .. }
+            | GlobalValueData::Load { global_type, .. } => global_type,
         }
     }
 }
@@ -65,20 +80,34 @@ impl GlobalValueData {
 impl fmt::Display for GlobalValueData {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
-            GlobalValueData::VMContext { offset } => write!(f, "vmctx{}", offset),
-            GlobalValueData::Deref {
+            GlobalValueData::VMContext => write!(f, "vmctx"),
+            GlobalValueData::Load {
                 base,
                 offset,
-                memory_type,
-            } => write!(f, "deref({}){}: {}", base, offset, memory_type),
-            GlobalValueData::Sym {
+                global_type,
+            } => write!(f, "load.{} notrap aligned {}{}", global_type, base, offset),
+            GlobalValueData::IAddImm {
+                global_type,
+                base,
+                offset,
+            } => write!(f, "iadd_imm.{} {}, {}", global_type, base, offset),
+            GlobalValueData::Symbol {
                 ref name,
+                offset,
                 colocated,
             } => {
                 if colocated {
                     write!(f, "colocated ")?;
                 }
-                write!(f, "globalsym {}", name)
+                write!(f, "symbol {}", name)?;
+                let offset_val: i64 = offset.into();
+                if offset_val > 0 {
+                    write!(f, "+")?;
+                }
+                if offset_val != 0 {
+                    write!(f, "{}", offset)?;
+                }
+                Ok(())
             }
         }
     }
diff --git a/lib/codegen/src/ir/immediates.rs b/lib/codegen/src/ir/immediates.rs
index f61fce2196..3297769ac4 100644
--- a/lib/codegen/src/ir/immediates.rs
+++ b/lib/codegen/src/ir/immediates.rs
@@ -214,6 +214,16 @@ impl Offset32 {
     pub fn new(x: i32) -> Self {
         Offset32(x)
     }
+
+    /// Create a new `Offset32` representing the signed numver `x` if possible.
+    pub fn try_from_i64(x: i64) -> Option {
+        let casted = x as i32;
+        if casted as i64 == x {
+            Some(Self::new(casted))
+        } else {
+            None
+        }
+    }
 }
 
 impl Into for Offset32 {
diff --git a/lib/codegen/src/legalizer/globalvalue.rs b/lib/codegen/src/legalizer/globalvalue.rs
index 61723af0d7..352c03e13b 100644
--- a/lib/codegen/src/legalizer/globalvalue.rs
+++ b/lib/codegen/src/legalizer/globalvalue.rs
@@ -28,54 +28,99 @@ pub fn expand_global_value(
     };
 
     match func.global_values[gv] {
-        ir::GlobalValueData::VMContext { offset } => vmctx_addr(inst, func, offset.into()),
-        ir::GlobalValueData::Deref {
+        ir::GlobalValueData::VMContext => vmctx_addr(inst, func),
+        ir::GlobalValueData::IAddImm {
             base,
             offset,
-            memory_type,
-        } => deref_addr(inst, func, base, offset.into(), memory_type, isa),
-        ir::GlobalValueData::Sym { .. } => globalsym(inst, func, gv, isa),
+            global_type,
+        } => iadd_imm_addr(inst, func, base, offset.into(), global_type),
+        ir::GlobalValueData::Load {
+            base,
+            offset,
+            global_type,
+        } => load_addr(inst, func, base, offset, global_type, isa),
+        ir::GlobalValueData::Symbol { .. } => symbol(inst, func, gv, isa),
     }
 }
 
 /// Expand a `global_value` instruction for a vmctx global.
-fn vmctx_addr(inst: ir::Inst, func: &mut ir::Function, offset: i64) {
+fn vmctx_addr(inst: ir::Inst, func: &mut ir::Function) {
     // Get the value representing the `vmctx` argument.
     let vmctx = func
         .special_param(ir::ArgumentPurpose::VMContext)
         .expect("Missing vmctx parameter");
 
-    // Simply replace the `global_value` instruction with an `iadd_imm`, reusing the result value.
-    func.dfg.replace(inst).iadd_imm(vmctx, offset);
+    // Replace the `global_value` instruction's value with an alias to the vmctx arg.
+    let result = func.dfg.first_result(inst);
+    func.dfg.clear_results(inst);
+    func.dfg.change_to_alias(result, vmctx);
+    func.layout.remove_inst(inst);
 }
 
-/// Expand a `global_value` instruction for a deref global.
-fn deref_addr(
+/// Expand a `global_value` instruction for an iadd_imm global.
+fn iadd_imm_addr(
     inst: ir::Inst,
     func: &mut ir::Function,
     base: ir::GlobalValue,
     offset: i64,
-    memory_type: ir::Type,
+    global_type: ir::Type,
+) {
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+
+    // Get the value for the lhs. For tidiness, expand VMContext here so that we avoid
+    // `vmctx_addr` which creates an otherwise unneeded value alias.
+    let lhs = if let ir::GlobalValueData::VMContext = pos.func.global_values[base] {
+        pos.func
+            .special_param(ir::ArgumentPurpose::VMContext)
+            .expect("Missing vmctx parameter")
+    } else {
+        pos.ins().global_value(global_type, base)
+    };
+
+    // Simply replace the `global_value` instruction with an `iadd_imm`, reusing the result value.
+    pos.func.dfg.replace(inst).iadd_imm(lhs, offset);
+}
+
+/// Expand a `global_value` instruction for a load global.
+fn load_addr(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    base: ir::GlobalValue,
+    offset: ir::immediates::Offset32,
+    global_type: ir::Type,
     isa: &TargetIsa,
 ) {
     // We need to load a pointer from the `base` global value, so insert a new `global_value`
     // instruction. This depends on the iterative legalization loop. Note that the IR verifier
-    // detects any cycles in the `deref` globals.
+    // detects any cycles in the `load` globals.
     let ptr_ty = isa.pointer_type();
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
 
-    let base_addr = pos.ins().global_value(ptr_ty, base);
+    // Get the value for the base. For tidiness, expand VMContext here so that we avoid
+    // `vmctx_addr` which creates an otherwise unneeded value alias.
+    let base_addr = if let ir::GlobalValueData::VMContext = pos.func.global_values[base] {
+        pos.func
+            .special_param(ir::ArgumentPurpose::VMContext)
+            .expect("Missing vmctx parameter")
+    } else {
+        pos.ins().global_value(ptr_ty, base)
+    };
+
+    // Global-value loads are always notrap and aligned.
     let mut mflags = ir::MemFlags::new();
-    // Deref globals are required to be accessible and aligned.
     mflags.set_notrap();
     mflags.set_aligned();
-    let loaded = pos.ins().load(memory_type, mflags, base_addr, 0);
-    pos.func.dfg.replace(inst).iadd_imm(loaded, offset);
+
+    // Perform the load.
+    pos.func
+        .dfg
+        .replace(inst)
+        .load(global_type, mflags, base_addr, offset);
 }
 
 /// Expand a `global_value` instruction for a symbolic name global.
-fn globalsym(inst: ir::Inst, func: &mut ir::Function, gv: ir::GlobalValue, isa: &TargetIsa) {
+fn symbol(inst: ir::Inst, func: &mut ir::Function, gv: ir::GlobalValue, isa: &TargetIsa) {
     let ptr_ty = isa.pointer_type();
-    func.dfg.replace(inst).globalsym_addr(ptr_ty, gv);
+    func.dfg.replace(inst).symbol_value(ptr_ty, gv);
 }
diff --git a/lib/codegen/src/predicates.rs b/lib/codegen/src/predicates.rs
index 58ae0a40fb..eca8f2318c 100644
--- a/lib/codegen/src/predicates.rs
+++ b/lib/codegen/src/predicates.rs
@@ -55,7 +55,7 @@ pub fn is_colocated_func(func_ref: ir::FuncRef, func: &ir::Function) -> bool {
 #[allow(dead_code)]
 pub fn is_colocated_data(global_value: ir::GlobalValue, func: &ir::Function) -> bool {
     match func.global_values[global_value] {
-        ir::GlobalValueData::Sym { colocated, .. } => colocated,
+        ir::GlobalValueData::Symbol { colocated, .. } => colocated,
         _ => panic!("is_colocated_data only makes sense for data with symbolic addresses"),
     }
 }
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 33ec402449..16e21565ac 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -43,7 +43,7 @@
 //!
 //! Global values
 //!
-//! - Detect cycles in deref(base) declarations.
+//! - Detect cycles in global values.
 //! - Detect use of 'vmctx' global value when no corresponding parameter is defined.
 //!
 //! TODO:
@@ -336,16 +336,28 @@ impl<'a> Verifier<'a> {
             seen.insert(gv);
 
             let mut cur = gv;
-            while let ir::GlobalValueData::Deref { base, .. } = self.func.global_values[cur] {
-                if seen.insert(base).is_some() {
-                    if !cycle_seen {
-                        report!(errors, gv, "deref cycle: {}", DisplayList(seen.as_slice()));
-                        cycle_seen = true; // ensures we don't report the cycle multiple times
-                    }
-                    continue 'gvs;
-                }
+            loop {
+                match self.func.global_values[cur] {
+                    ir::GlobalValueData::Load { base, .. }
+                    | ir::GlobalValueData::IAddImm { base, .. } => {
+                        if seen.insert(base).is_some() {
+                            if !cycle_seen {
+                                report!(
+                                    errors,
+                                    gv,
+                                    "global value cycle: {}",
+                                    DisplayList(seen.as_slice())
+                                );
+                                // ensures we don't report the cycle multiple times
+                                cycle_seen = true;
+                            }
+                            continue 'gvs;
+                        }
 
-                cur = base;
+                        cur = base;
+                    }
+                    _ => break,
+                }
             }
 
             match self.func.global_values[gv] {
@@ -358,7 +370,30 @@ impl<'a> Verifier<'a> {
                         report!(errors, gv, "undeclared vmctx reference {}", gv);
                     }
                 }
-                ir::GlobalValueData::Deref { base, .. } => {
+                ir::GlobalValueData::IAddImm {
+                    base, global_type, ..
+                } => {
+                    if !global_type.is_int() {
+                        report!(
+                            errors,
+                            gv,
+                            "iadd_imm global value with non-int type {}",
+                            global_type
+                        );
+                    } else if let Some(isa) = self.isa {
+                        let base_type = self.func.global_values[base].global_type(isa);
+                        if global_type != base_type {
+                            report!(
+                                errors,
+                                gv,
+                                "iadd_imm type {} differs from operand type {}",
+                                global_type,
+                                base_type
+                            );
+                        }
+                    }
+                }
+                ir::GlobalValueData::Load { base, .. } => {
                     if let Some(isa) = self.isa {
                         let base_type = self.func.global_values[base].global_type(isa);
                         let pointer_type = isa.pointer_type();
@@ -366,7 +401,7 @@ impl<'a> Verifier<'a> {
                             report!(
                                 errors,
                                 gv,
-                                "deref base {} has type {}, which is not the pointer type {}",
+                                "base {} has type {}, which is not the pointer type {}",
                                 base,
                                 base_type,
                                 pointer_type
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 91faef517e..909fa36c6d 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -462,8 +462,9 @@ where
     pub fn declare_data_in_func(&self, data: DataId, func: &mut ir::Function) -> ir::GlobalValue {
         let decl = &self.contents.data_objects[data].decl;
         let colocated = decl.linkage.is_final();
-        func.create_global_value(ir::GlobalValueData::Sym {
+        func.create_global_value(ir::GlobalValueData::Symbol {
             name: ir::ExternalName::user(1, data.index() as u32),
+            offset: ir::immediates::Imm64::new(0),
             colocated,
         })
     }
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 3003f037d9..5358109a2e 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -164,8 +164,9 @@ impl<'a> Context<'a> {
     fn add_gv(&mut self, gv: GlobalValue, data: GlobalValueData, loc: Location) -> ParseResult<()> {
         self.map.def_gv(gv, loc)?;
         while self.function.global_values.next_key().index() <= gv.index() {
-            self.function.create_global_value(GlobalValueData::Sym {
+            self.function.create_global_value(GlobalValueData::Symbol {
                 name: ExternalName::testcase(""),
+                offset: Imm64::new(0),
                 colocated: false,
             });
         }
@@ -590,14 +591,32 @@ impl<'a> Parser<'a> {
     // present, it must contain a sign.
     fn optional_offset32(&mut self) -> ParseResult {
         if let Some(Token::Integer(text)) = self.token() {
-            self.consume();
-            // Lexer just gives us raw text that looks like an integer.
-            // Parse it as an `Offset32` to check for overflow and other issues.
-            text.parse().map_err(|e| self.error(e))
-        } else {
-            // An offset32 operand can be absent.
-            Ok(Offset32::new(0))
+            if text.starts_with('+') || text.starts_with('-') {
+                self.consume();
+                // Lexer just gives us raw text that looks like an integer.
+                // Parse it as an `Offset32` to check for overflow and other issues.
+                return text.parse().map_err(|e| self.error(e));
+            }
         }
+        // An offset32 operand can be absent.
+        Ok(Offset32::new(0))
+    }
+
+    // Match and consume an optional offset32 immediate.
+    //
+    // Note that this will match an empty string as an empty offset, and that if an offset is
+    // present, it must contain a sign.
+    fn optional_offset_imm64(&mut self) -> ParseResult {
+        if let Some(Token::Integer(text)) = self.token() {
+            if text.starts_with('+') || text.starts_with('-') {
+                self.consume();
+                // Lexer just gives us raw text that looks like an integer.
+                // Parse it as an `Offset32` to check for overflow and other issues.
+                return text.parse().map_err(|e| self.error(e));
+            }
+        }
+        // If no explicit offset is present, the offset is 0.
+        Ok(Imm64::new(0))
     }
 
     // Match and consume an Ieee32 immediate.
@@ -1185,9 +1204,10 @@ impl<'a> Parser<'a> {
     // Parse a global value decl.
     //
     // global-val-decl ::= * GlobalValue(gv) "=" global-val-desc
-    // global-val-desc ::= "vmctx" offset32
-    //                   | "deref" "(" GlobalValue(base) ")" offset32
-    //                   | globalsym ["colocated"] name
+    // global-val-desc ::= "vmctx"
+    //                   | "load" "." type "notrap" "aligned" GlobalValue(base) [offset]
+    //                   | "iadd_imm" "(" GlobalValue(base) ")" imm64
+    //                   | "symbol" ["colocated"] name + imm64
     //
     fn parse_global_value_decl(&mut self) -> ParseResult<(GlobalValue, GlobalValueData)> {
         let gv = self.match_gv("expected global value number: gv«n»")?;
@@ -1195,27 +1215,55 @@ impl<'a> Parser<'a> {
         self.match_token(Token::Equal, "expected '=' in global value declaration")?;
 
         let data = match self.match_any_identifier("expected global value kind")? {
-            "vmctx" => {
-                let offset = self.optional_offset32()?;
-                GlobalValueData::VMContext { offset }
-            }
-            "deref" => {
-                self.match_token(Token::LPar, "expected '(' in 'deref' global value decl")?;
+            "vmctx" => GlobalValueData::VMContext,
+            "load" => {
+                self.match_token(
+                    Token::Dot,
+                    "expected '.' followed by type in load global value decl",
+                )?;
+                let global_type = self.match_type("expected load type")?;
+                let flags = self.optional_memflags();
                 let base = self.match_gv("expected global value: gv«n»")?;
-                self.match_token(Token::RPar, "expected ')' in 'deref' global value decl")?;
                 let offset = self.optional_offset32()?;
-                self.match_token(Token::Colon, "expected ':' in 'deref' global value decl")?;
-                let memory_type = self.match_type("expected deref type")?;
-                GlobalValueData::Deref {
+                let mut expected_flags = MemFlags::new();
+                expected_flags.set_notrap();
+                expected_flags.set_aligned();
+                if flags != expected_flags {
+                    return err!(self.loc, "global-value load must be notrap and aligned");
+                }
+                GlobalValueData::Load {
                     base,
                     offset,
-                    memory_type,
+                    global_type,
                 }
             }
-            "globalsym" => {
+            "iadd_imm" => {
+                self.match_token(
+                    Token::Dot,
+                    "expected '.' followed by type in iadd_imm global value decl",
+                )?;
+                let global_type = self.match_type("expected iadd type")?;
+                let base = self.match_gv("expected global value: gv«n»")?;
+                self.match_token(
+                    Token::Comma,
+                    "expected ',' followed by rhs in iadd_imm global value decl",
+                )?;
+                let offset = self.match_imm64("expected iadd_imm immediate")?;
+                GlobalValueData::IAddImm {
+                    base,
+                    offset,
+                    global_type,
+                }
+            }
+            "symbol" => {
                 let colocated = self.optional(Token::Identifier("colocated"));
                 let name = self.parse_external_name()?;
-                GlobalValueData::Sym { name, colocated }
+                let offset = self.optional_offset_imm64()?;
+                GlobalValueData::Symbol {
+                    name,
+                    offset,
+                    colocated,
+                }
             }
             other => return err!(self.loc, "Unknown global value kind '{}'", other),
         };
@@ -2680,8 +2728,8 @@ mod tests {
     fn duplicate_gv() {
         let ParseError { location, message } = Parser::new(
             "function %ebbs() system_v {
-                gv0 = vmctx+64
-                gv0 = vmctx+64",
+                gv0 = vmctx
+                gv0 = vmctx",
         ).parse_function(None)
             .unwrap_err();
 
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index e0c972c27d..d7c9c47c10 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -2,7 +2,7 @@
 //! wasm translation.
 
 use cranelift_codegen::cursor::FuncCursor;
-use cranelift_codegen::ir::immediates::Imm64;
+use cranelift_codegen::ir::immediates::{Imm64, Offset32};
 use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::settings;
@@ -162,21 +162,26 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
 
     fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalVariable {
         // Just create a dummy `vmctx` global.
-        let offset = ((index * 8) as i32 + 8).into();
-        let gv = func.create_global_value(ir::GlobalValueData::VMContext { offset });
+        let offset = ((index * 8) as i64 + 8).into();
+        let vmctx = func.create_global_value(ir::GlobalValueData::VMContext {});
+        let iadd = func.create_global_value(ir::GlobalValueData::IAddImm {
+            base: vmctx,
+            offset,
+            global_type: self.pointer_type(),
+        });
         GlobalVariable::Memory {
-            gv,
+            gv: iadd,
             ty: self.mod_info.globals[index].entity.ty,
         }
     }
 
     fn make_heap(&mut self, func: &mut ir::Function, _index: MemoryIndex) -> ir::Heap {
         // Create a static heap whose base address is stored at `vmctx+0`.
-        let addr = func.create_global_value(ir::GlobalValueData::VMContext { offset: 0.into() });
-        let gv = func.create_global_value(ir::GlobalValueData::Deref {
+        let addr = func.create_global_value(ir::GlobalValueData::VMContext);
+        let gv = func.create_global_value(ir::GlobalValueData::Load {
             base: addr,
-            offset: 0.into(),
-            memory_type: self.pointer_type(),
+            offset: Offset32::new(0),
+            global_type: self.pointer_type(),
         });
 
         func.create_heap(ir::HeapData {
@@ -192,19 +197,16 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
 
     fn make_table(&mut self, func: &mut ir::Function, _index: TableIndex) -> ir::Table {
         // Create a table whose base address is stored at `vmctx+0`.
-        let base_gv_addr =
-            func.create_global_value(ir::GlobalValueData::VMContext { offset: 0.into() });
-        let base_gv = func.create_global_value(ir::GlobalValueData::Deref {
-            base: base_gv_addr,
-            offset: 0.into(),
-            memory_type: self.pointer_type(),
+        let vmctx = func.create_global_value(ir::GlobalValueData::VMContext);
+        let base_gv = func.create_global_value(ir::GlobalValueData::Load {
+            base: vmctx,
+            offset: Offset32::new(0),
+            global_type: self.pointer_type(),
         });
-        let bound_gv_addr =
-            func.create_global_value(ir::GlobalValueData::VMContext { offset: 0.into() });
-        let bound_gv = func.create_global_value(ir::GlobalValueData::Deref {
-            base: bound_gv_addr,
-            offset: 0.into(),
-            memory_type: self.pointer_type(),
+        let bound_gv = func.create_global_value(ir::GlobalValueData::Load {
+            base: vmctx,
+            offset: Offset32::new(0),
+            global_type: self.pointer_type(),
         });
 
         func.create_table(ir::TableData {

From 18900df4d5564c2792805bec1572b2834493b2c8 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 4 Sep 2018 21:22:50 -0700
Subject: [PATCH 2050/3084] Clean up obsolete comments.

---
 lib/codegen/meta-python/gen_instr.py | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/lib/codegen/meta-python/gen_instr.py b/lib/codegen/meta-python/gen_instr.py
index a59f5d71da..4b73b95bc6 100644
--- a/lib/codegen/meta-python/gen_instr.py
+++ b/lib/codegen/meta-python/gen_instr.py
@@ -107,11 +107,10 @@ def gen_instruction_data(fmt):
     """
     Generate the InstructionData enum.
 
-    Every variant must contain `opcode` and `ty` fields. An instruction that
-    doesn't produce a value should have its `ty` field set to `VOID`. The size
-    of `InstructionData` should be kept at 16 bytes on 64-bit architectures. If
-    more space is needed to represent an instruction, use a `Box` to
-    store the additional information out of line.
+    Every variant must contain an `opcode` field. The size of `InstructionData`
+    should be kept at 16 bytes on 64-bit architectures. If more space is needed
+    to represent an instruction, use a `Box` to store the additional
+    information out of line.
     """
 
     fmt.line('#[derive(Clone, Debug)]')

From d4b862239327e593134412c53b7ea13785adfa74 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 4 Sep 2018 21:46:22 -0700
Subject: [PATCH 2051/3084] Rename the VOID type to INVALID and clean up
 obsolete comments.

The VOID type isn't used for anything resembling what "void" means in C,
so rename it to INVALID to avoid confusion.
---
 lib/codegen/meta-python/cdsl/types.py   |  2 +-
 lib/codegen/meta-python/gen_encoding.py |  8 ++---
 lib/codegen/meta-python/gen_instr.py    |  2 +-
 lib/codegen/src/ir/builder.rs           |  2 +-
 lib/codegen/src/ir/dfg.rs               | 26 ++++++++--------
 lib/codegen/src/ir/types.rs             | 41 ++++++++++++-------------
 lib/codegen/src/isa/enc_tables.rs       |  2 +-
 lib/codegen/src/verifier/mod.rs         |  4 +--
 lib/codegen/src/write.rs                | 16 ++++++----
 lib/reader/src/parser.rs                | 12 ++++----
 lib/wasm/src/environ/dummy.rs           |  4 +--
 11 files changed, 61 insertions(+), 58 deletions(-)

diff --git a/lib/codegen/meta-python/cdsl/types.py b/lib/codegen/meta-python/cdsl/types.py
index 13bc5deab1..26777152da 100644
--- a/lib/codegen/meta-python/cdsl/types.py
+++ b/lib/codegen/meta-python/cdsl/types.py
@@ -179,7 +179,7 @@ class SpecialType(ValueType):
     def __init__(self, name, membytes, doc):
         # type: (str, int, str) -> None
         super(SpecialType, self).__init__(name, membytes, doc)
-        # Assign numbers starting from 1. (0 is VOID)
+        # Assign numbers starting from 1. (0 is INVALID)
         ValueType.all_special_types.append(self)
         self.number = len(ValueType.all_special_types)
         assert self.number < LANE_BASE, 'Too many special types'
diff --git a/lib/codegen/meta-python/gen_encoding.py b/lib/codegen/meta-python/gen_encoding.py
index a81b66dcca..59875bf489 100644
--- a/lib/codegen/meta-python/gen_encoding.py
+++ b/lib/codegen/meta-python/gen_encoding.py
@@ -22,8 +22,8 @@ This is the information available to us:
 ## Level 1 table lookup
 
 The CPU mode provides the first table. The key is the instruction's controlling
-type variable. If the instruction is not polymorphic, use `VOID` for the type
-variable. The table values are level 2 tables.
+type variable. If the instruction is not polymorphic, use `INVALID` for the
+type variable. The table values are level 2 tables.
 
 ## Level 2 table lookup
 
@@ -682,7 +682,7 @@ def emit_level1_hashtable(cpumode, level1, offt, fmt):
             # Empty hash table entry. Include the default legalization action.
             if not level2:
                 fmt.format(
-                        'Level1Entry {{ ty: ir::types::VOID, log2len: !0, '
+                        'Level1Entry {{ ty: ir::types::INVALID, log2len: !0, '
                         'offset: 0, legalize: {} }},',
                         level1.legalize_code)
                 continue
@@ -690,7 +690,7 @@ def emit_level1_hashtable(cpumode, level1, offt, fmt):
             if level2.ty is not None:
                 tyname = level2.ty.rust_name()
             else:
-                tyname = 'ir::types::VOID'
+                tyname = 'ir::types::INVALID'
 
             lcode = cpumode.isa.legalize_code(level2.legalize)
 
diff --git a/lib/codegen/meta-python/gen_instr.py b/lib/codegen/meta-python/gen_instr.py
index 4b73b95bc6..2f5eeae93d 100644
--- a/lib/codegen/meta-python/gen_instr.py
+++ b/lib/codegen/meta-python/gen_instr.py
@@ -705,7 +705,7 @@ def gen_inst_builder(inst, fmt):
             args.append(inst.ctrl_typevar.name)
         elif not inst.is_polymorphic:
             # No controlling type variable needed.
-            args.append('types::VOID')
+            args.append('types::INVALID')
         else:
             assert inst.is_polymorphic and inst.use_typevar_operand
             # Infer the controlling type variable from the input operands.
diff --git a/lib/codegen/src/ir/builder.rs b/lib/codegen/src/ir/builder.rs
index 5cd2bfe36d..9128092f67 100644
--- a/lib/codegen/src/ir/builder.rs
+++ b/lib/codegen/src/ir/builder.rs
@@ -28,7 +28,7 @@ pub trait InstBuilderBase<'f>: Sized {
     /// Insert an instruction and return a reference to it, consuming the builder.
     ///
     /// The result types may depend on a controlling type variable. For non-polymorphic
-    /// instructions with multiple results, pass `VOID` for the `ctrl_typevar` argument.
+    /// instructions with multiple results, pass `INVALID` for the `ctrl_typevar` argument.
     fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'f mut DataFlowGraph);
 }
 
diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs
index 6d799a43c9..08bec5d624 100644
--- a/lib/codegen/src/ir/dfg.rs
+++ b/lib/codegen/src/ir/dfg.rs
@@ -157,7 +157,7 @@ pub struct Values<'a> {
 /// Check for non-values
 fn valid_valuedata(data: &ValueData) -> bool {
     if let &ValueData::Alias {
-        ty: types::VOID,
+        ty: types::INVALID,
         original,
     } = data
     {
@@ -288,7 +288,7 @@ impl DataFlowGraph {
             self.value_type(dest),
             ty
         );
-        debug_assert_ne!(ty, types::VOID);
+        debug_assert_ne!(ty, types::INVALID);
 
         self.values[dest] = ValueData::Alias { ty, original };
     }
@@ -332,7 +332,7 @@ impl DataFlowGraph {
                 self.value_type(dest),
                 ty
             );
-            debug_assert_ne!(ty, types::VOID);
+            debug_assert_ne!(ty, types::INVALID);
 
             self.values[dest] = ValueData::Alias { ty, original };
         }
@@ -461,7 +461,7 @@ impl DataFlowGraph {
     ///
     /// The result value types are determined from the instruction's value type constraints and the
     /// provided `ctrl_typevar` type for polymorphic instructions. For non-polymorphic
-    /// instructions, `ctrl_typevar` is ignored, and `VOID` can be used.
+    /// instructions, `ctrl_typevar` is ignored, and `INVALID` can be used.
     ///
     /// The type of the first result value is also set, even if it was already set in the
     /// `InstructionData` passed to `make_inst`. If this function is called with a single-result
@@ -679,12 +679,12 @@ impl DataFlowGraph {
         })
     }
 
-    /// Get the controlling type variable, or `VOID` if `inst` isn't polymorphic.
+    /// Get the controlling type variable, or `INVALID` if `inst` isn't polymorphic.
     pub fn ctrl_typevar(&self, inst: Inst) -> Type {
         let constraints = self[inst].opcode().constraints();
 
         if !constraints.is_polymorphic() {
-            types::VOID
+            types::INVALID
         } else if constraints.requires_typevar_operand() {
             // Not all instruction formats have a designated operand, but in that case
             // `requires_typevar_operand()` should never be true.
@@ -897,7 +897,7 @@ impl<'a> fmt::Display for DisplayInst<'a> {
         }
 
         let typevar = dfg.ctrl_typevar(inst);
-        if typevar.is_void() {
+        if typevar.is_invalid() {
             write!(f, "{}", dfg[inst].opcode())?;
         } else {
             write!(f, "{}.{}", dfg[inst].opcode(), typevar)?;
@@ -914,7 +914,7 @@ impl DataFlowGraph {
     fn set_value_type_for_parser(&mut self, v: Value, t: Type) {
         assert_eq!(
             self.value_type(v),
-            types::VOID,
+            types::INVALID,
             "this function is only for assigning types to previously invalid values"
         );
         match self.values[v] {
@@ -980,9 +980,9 @@ impl DataFlowGraph {
         let ty = if self.values.is_valid(src) {
             self.value_type(src)
         } else {
-            // As a special case, if we can't resolve the aliasee yet, use VOID
+            // As a special case, if we can't resolve the aliasee yet, use INVALID
             // temporarily. It will be resolved later in parsing.
-            types::VOID
+            types::INVALID
         };
         let data = ValueData::Alias { ty, original: src };
         self.values[dest] = data;
@@ -1007,7 +1007,7 @@ impl DataFlowGraph {
         if let Some(resolved) = maybe_resolve_aliases(&self.values, v) {
             let old_ty = self.value_type(v);
             let new_ty = self.value_type(resolved);
-            if old_ty == types::VOID {
+            if old_ty == types::INVALID {
                 self.set_value_type_for_parser(v, new_ty);
             } else {
                 assert_eq!(old_ty, new_ty);
@@ -1023,7 +1023,7 @@ impl DataFlowGraph {
     #[cold]
     pub fn make_invalid_value_for_parser(&mut self) {
         let data = ValueData::Alias {
-            ty: types::VOID,
+            ty: types::INVALID,
             original: Value::reserved_value(),
         };
         self.make_value(data);
@@ -1037,7 +1037,7 @@ impl DataFlowGraph {
             return false;
         }
         if let ValueData::Alias { ty, .. } = self.values[v] {
-            ty != types::VOID
+            ty != types::INVALID
         } else {
             true
         }
diff --git a/lib/codegen/src/ir/types.rs b/lib/codegen/src/ir/types.rs
index 195d90968a..1d19f5c6d7 100644
--- a/lib/codegen/src/ir/types.rs
+++ b/lib/codegen/src/ir/types.rs
@@ -5,8 +5,9 @@ use std::fmt::{self, Debug, Display, Formatter};
 
 /// The type of an SSA value.
 ///
-/// The `VOID` type is only used for instructions that produce no value. It can't be part of a SIMD
-/// vector.
+/// The `INVALID` type isn't a real type, and is used as a placeholder in the IR where a type
+/// field is present put no type is needed, such as the controlling type variable for a
+/// non-polymorphic instruction.
 ///
 /// Basic integer types: `I8`, `I16`, `I32`, and `I64`. These types are sign-agnostic.
 ///
@@ -20,9 +21,8 @@ use std::fmt::{self, Debug, Display, Formatter};
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
 pub struct Type(u8);
 
-/// No type. Used for functions without a return value. Can't be loaded or stored. Can't be part of
-/// a SIMD vector.
-pub const VOID: Type = Type(0);
+/// Not a valid type. Can't be loaded or stored. Can't be part of a SIMD vector.
+pub const INVALID: Type = Type(0);
 
 /// Start of the lane types. See also `meta-python/cdsl.types.py`.
 const LANE_BASE: u8 = 0x70;
@@ -146,9 +146,9 @@ impl Type {
         }))
     }
 
-    /// Is this the VOID type?
-    pub fn is_void(self) -> bool {
-        self == VOID
+    /// Is this the INVALID type?
+    pub fn is_invalid(self) -> bool {
+        self == INVALID
     }
 
     /// Is this a special type?
@@ -285,10 +285,10 @@ impl Display for Type {
             write!(f, "{}x{}", self.lane_type(), self.lane_count())
         } else {
             f.write_str(match *self {
-                VOID => "void",
                 IFLAGS => "iflags",
                 FFLAGS => "fflags",
-                _ => panic!("Invalid Type(0x{:x})", self.0),
+                INVALID => panic!("INVALID encountered"),
+                _ => panic!("Unknown Type(0x{:x})", self.0),
             })
         }
     }
@@ -306,7 +306,7 @@ impl Debug for Type {
             write!(f, "{:?}X{}", self.lane_type(), self.lane_count())
         } else {
             match *self {
-                VOID => write!(f, "types::VOID"),
+                INVALID => write!(f, "types::INVALID"),
                 IFLAGS => write!(f, "types::IFLAGS"),
                 FFLAGS => write!(f, "types::FFLAGS"),
                 _ => write!(f, "Type(0x{:x})", self.0),
@@ -317,7 +317,7 @@ impl Debug for Type {
 
 impl Default for Type {
     fn default() -> Self {
-        VOID
+        INVALID
     }
 }
 
@@ -328,8 +328,8 @@ mod tests {
 
     #[test]
     fn basic_scalars() {
-        assert_eq!(VOID, VOID.lane_type());
-        assert_eq!(0, VOID.bits());
+        assert_eq!(INVALID, INVALID.lane_type());
+        assert_eq!(0, INVALID.bits());
         assert_eq!(IFLAGS, IFLAGS.lane_type());
         assert_eq!(0, IFLAGS.bits());
         assert_eq!(FFLAGS, FFLAGS.lane_type());
@@ -346,7 +346,7 @@ mod tests {
         assert_eq!(F32, F32.lane_type());
         assert_eq!(F64, F64.lane_type());
 
-        assert_eq!(VOID.lane_bits(), 0);
+        assert_eq!(INVALID.lane_bits(), 0);
         assert_eq!(IFLAGS.lane_bits(), 0);
         assert_eq!(FFLAGS.lane_bits(), 0);
         assert_eq!(B1.lane_bits(), 1);
@@ -364,8 +364,8 @@ mod tests {
 
     #[test]
     fn typevar_functions() {
-        assert_eq!(VOID.half_width(), None);
-        assert_eq!(IFLAGS.half_width(), None);
+        assert_eq!(INVALID.half_width(), None);
+        assert_eq!(INVALID.half_width(), None);
         assert_eq!(FFLAGS.half_width(), None);
         assert_eq!(B1.half_width(), None);
         assert_eq!(B8.half_width(), None);
@@ -380,7 +380,7 @@ mod tests {
         assert_eq!(F32.half_width(), None);
         assert_eq!(F64.half_width(), Some(F32));
 
-        assert_eq!(VOID.double_width(), None);
+        assert_eq!(INVALID.double_width(), None);
         assert_eq!(IFLAGS.double_width(), None);
         assert_eq!(FFLAGS.double_width(), None);
         assert_eq!(B1.double_width(), None);
@@ -407,7 +407,7 @@ mod tests {
         assert_eq!(big.half_vector().unwrap().to_string(), "f64x128");
         assert_eq!(B1.by(2).unwrap().half_vector().unwrap().to_string(), "b1");
         assert_eq!(I32.half_vector(), None);
-        assert_eq!(VOID.half_vector(), None);
+        assert_eq!(INVALID.half_vector(), None);
 
         // Check that the generated constants match the computed vector types.
         assert_eq!(I32.by(4), Some(I32X4));
@@ -416,7 +416,6 @@ mod tests {
 
     #[test]
     fn format_scalars() {
-        assert_eq!(VOID.to_string(), "void");
         assert_eq!(IFLAGS.to_string(), "iflags");
         assert_eq!(FFLAGS.to_string(), "fflags");
         assert_eq!(B1.to_string(), "b1");
@@ -443,7 +442,7 @@ mod tests {
         assert_eq!(F64.by(2).unwrap().to_string(), "f64x2");
         assert_eq!(I8.by(3), None);
         assert_eq!(I8.by(512), None);
-        assert_eq!(VOID.by(4), None);
+        assert_eq!(INVALID.by(4), None);
     }
 
     #[test]
diff --git a/lib/codegen/src/isa/enc_tables.rs b/lib/codegen/src/isa/enc_tables.rs
index 0a98ace71a..1d8ed94780 100644
--- a/lib/codegen/src/isa/enc_tables.rs
+++ b/lib/codegen/src/isa/enc_tables.rs
@@ -30,7 +30,7 @@ pub type LegalizeCode = u8;
 /// Level 1 hash table entry.
 ///
 /// One level 1 hash table is generated per CPU mode. This table is keyed by the controlling type
-/// variable, using `VOID` for non-polymorphic instructions.
+/// variable, using `INVALID` for non-polymorphic instructions.
 ///
 /// The hash table values are references to level 2 hash tables, encoded as an offset in `LEVEL2`
 /// where the table begins, and the binary logarithm of its length. All the level 2 hash tables
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 16e21565ac..51aa484525 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -1106,8 +1106,8 @@ impl<'a> Verifier<'a> {
             ctrl_type
         } else {
             // Non-polymorphic instructions don't check the controlling type variable, so `Option`
-            // is unnecessary and we can just make it `VOID`.
-            types::VOID
+            // is unnecessary and we can just make it `INVALID`.
+            types::INVALID
         };
 
         // Typechecking instructions is never fatal
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index 24cfdbe284..2e9faf6b40 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -44,13 +44,17 @@ pub trait FuncWriter {
         }
 
         for (heap, heap_data) in &func.heaps {
-            any = true;
-            self.write_entity_definition(w, func, heap.into(), heap_data)?;
+            if !heap_data.index_type.is_invalid() {
+                any = true;
+                self.write_entity_definition(w, func, heap.into(), heap_data)?;
+            }
         }
 
         for (table, table_data) in &func.tables {
-            any = true;
-            self.write_entity_definition(w, func, table.into(), table_data)?;
+            if !table_data.index_type.is_invalid() {
+                any = true;
+                self.write_entity_definition(w, func, table.into(), table_data)?;
+            }
         }
 
         // Write out all signatures before functions since function declarations can refer to
@@ -61,8 +65,8 @@ pub trait FuncWriter {
         }
 
         for (fnref, ext_func) in &func.dfg.ext_funcs {
-            any = true;
             if ext_func.signature != SigRef::reserved_value() {
+                any = true;
                 self.write_entity_definition(w, func, fnref.into(), ext_func)?;
             }
         }
@@ -265,7 +269,7 @@ fn type_suffix(func: &Function, inst: Inst) -> Option {
 
     let rtype = func.dfg.ctrl_typevar(inst);
     assert!(
-        !rtype.is_void(),
+        !rtype.is_invalid(),
         "Polymorphic instruction must produce a result"
     );
     Some(rtype)
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 5358109a2e..4a116699b8 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -5,7 +5,7 @@ use cranelift_codegen::ir;
 use cranelift_codegen::ir::entities::AnyEntity;
 use cranelift_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32};
 use cranelift_codegen::ir::instructions::{InstructionData, InstructionFormat, VariableArgs};
-use cranelift_codegen::ir::types::VOID;
+use cranelift_codegen::ir::types::INVALID;
 use cranelift_codegen::ir::{
     AbiParam, ArgumentExtension, ArgumentLoc, Ebb, ExtFuncData, ExternalName, FuncRef, Function,
     GlobalValue, GlobalValueData, Heap, HeapData, HeapStyle, JumpTable, JumpTableData, MemFlags,
@@ -194,7 +194,7 @@ impl<'a> Context<'a> {
                 style: HeapStyle::Static {
                     bound: Imm64::new(0),
                 },
-                index_type: VOID,
+                index_type: INVALID,
             });
         }
         self.function.heaps[heap] = data;
@@ -218,7 +218,7 @@ impl<'a> Context<'a> {
                 min_size: Imm64::new(0),
                 bound_gv: GlobalValue::reserved_value(),
                 element_size: Imm64::new(0),
-                index_type: VOID,
+                index_type: INVALID,
             });
         }
         self.function.tables[table] = data;
@@ -1951,7 +1951,7 @@ impl<'a> Parser<'a> {
     // The controlling type variable can be specified explicitly as 'splat.i32x4 v5', or it can be
     // inferred from `inst_data.typevar_operand` for some opcodes.
     //
-    // Returns the controlling typevar for a polymorphic opcode, or `VOID` for a non-polymorphic
+    // Returns the controlling typevar for a polymorphic opcode, or `INVALID` for a non-polymorphic
     // opcode.
     fn infer_typevar(
         &self,
@@ -2006,7 +2006,7 @@ impl<'a> Parser<'a> {
                     );
                 } else {
                     // This is a non-polymorphic opcode. No typevar needed.
-                    VOID
+                    INVALID
                 }
             }
         };
@@ -2025,7 +2025,7 @@ impl<'a> Parser<'a> {
                 );
             }
         // Treat it as a syntax error to speficy a typevar on a non-polymorphic opcode.
-        } else if ctrl_type != VOID {
+        } else if ctrl_type != INVALID {
             return err!(self.loc, "{} does not take a typevar", opcode);
         }
 
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index d7c9c47c10..229e2a7274 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -277,7 +277,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
 
         Ok(pos
             .ins()
-            .CallIndirect(ir::Opcode::CallIndirect, VOID, sig_ref, args)
+            .CallIndirect(ir::Opcode::CallIndirect, INVALID, sig_ref, args)
             .0)
     }
 
@@ -300,7 +300,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         args.extend(call_args.iter().cloned(), &mut pos.func.dfg.value_lists);
         args.push(vmctx, &mut pos.func.dfg.value_lists);
 
-        Ok(pos.ins().Call(ir::Opcode::Call, VOID, callee, args).0)
+        Ok(pos.ins().Call(ir::Opcode::Call, INVALID, callee, args).0)
     }
 
     fn translate_memory_grow(

From 112e4a60836772e75be097adfcb2a569f8780aa4 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 4 Sep 2018 21:57:32 -0700
Subject: [PATCH 2052/3084] Add a `use std::vec::Vec;` to fix the no_std build.

---
 lib/codegen/src/write.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index 2e9faf6b40..0c3b7cbae9 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -10,6 +10,7 @@ use isa::{RegInfo, TargetIsa};
 use packed_option::ReservedValue;
 use std::fmt::{self, Write};
 use std::string::String;
+use std::vec::Vec;
 
 /// A `FuncWriter` used to decorate functions during printing.
 pub trait FuncWriter {

From e8878ba5047f1b76f0976e1ee7cb8d8437bfc561 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 4 Sep 2018 22:04:22 -0700
Subject: [PATCH 2053/3084] Bump version to 0.21.0

---
 cranelift/Cargo.toml        | 26 +++++++++++++-------------
 cranelift/publish-all.sh    |  2 +-
 lib/bforest/Cargo.toml      |  4 ++--
 lib/codegen/Cargo.toml      |  8 ++++----
 lib/codegen/meta/Cargo.toml |  2 +-
 lib/entity/Cargo.toml       |  2 +-
 lib/faerie/Cargo.toml       |  6 +++---
 lib/filetests/Cargo.toml    |  6 +++---
 lib/frontend/Cargo.toml     |  4 ++--
 lib/module/Cargo.toml       |  6 +++---
 lib/native/Cargo.toml       |  4 ++--
 lib/reader/Cargo.toml       |  4 ++--
 lib/serde/Cargo.toml        |  6 +++---
 lib/simplejit/Cargo.toml    | 10 +++++-----
 lib/umbrella/Cargo.toml     |  6 +++---
 lib/wasm/Cargo.toml         |  8 ++++----
 16 files changed, 52 insertions(+), 52 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index f154579a9f..e4c28dad9f 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.20.0"
+version = "0.21.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,18 +14,18 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.20.0" }
-cranelift-entity = { path = "lib/entity", version = "0.20.1" }
-cranelift-reader = { path = "lib/reader", version = "0.20.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.20.0" }
-cranelift-serde = { path = "lib/serde", version = "0.20.0", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.20.1", optional = true }
-cranelift-native = { path = "lib/native", version = "0.20.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.20.0" }
-cranelift-module = { path = "lib/module", version = "0.20.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.20.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.20.0" }
-cranelift = { path = "lib/umbrella", version = "0.20.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.21.0" }
+cranelift-entity = { path = "lib/entity", version = "0.21.0" }
+cranelift-reader = { path = "lib/reader", version = "0.21.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.21.0" }
+cranelift-serde = { path = "lib/serde", version = "0.21.0", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.21.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.21.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.21.0" }
+cranelift-module = { path = "lib/module", version = "0.21.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.21.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.21.0" }
+cranelift = { path = "lib/umbrella", version = "0.21.0" }
 filecheck = "0.3.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 0fa8c0ab76..7eb2ff2734 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.20.0"
+version="0.21.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index b68d9e9ea2..b2a5f6b25f 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.20.0"
+version = "0.21.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 keywords = ["btree", "forest", "set", "map"]
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.20.1", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.21.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 9ec8ea6260..2c18b92820 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.20.0"
+version = "0.21.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.20.1", default-features = false }
-cranelift-bforest = { path = "../bforest", version = "0.20.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.21.0", default-features = false }
+cranelift-bforest = { path = "../bforest", version = "0.21.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -25,7 +25,7 @@ log = { version = "0.4.4", default-features = false, features = ["release_max_le
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.20.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.21.0" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 4a72b3e19e..9a16c0fad2 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.20.0"
+version = "0.21.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index c74e605f76..cf4378618f 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.20.1"
+version = "0.21.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 6a4bb2343e..a3c39abdad 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.20.0"
+version = "0.21.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.20.0" }
-cranelift-module = { path = "../module", version = "0.20.0" }
+cranelift-codegen = { path = "../codegen", version = "0.21.0" }
+cranelift-module = { path = "../module", version = "0.21.0" }
 faerie = "0.5.0"
 goblin = "0.0.17"
 failure = "0.1.2"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index ca03057505..78e58fea29 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.20.0"
+version = "0.21.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,8 +9,8 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.20.0" }
-cranelift-reader = { path = "../reader", version = "0.20.0" }
+cranelift-codegen = { path = "../codegen", version = "0.21.0" }
+cranelift-reader = { path = "../reader", version = "0.21.0" }
 file-per-thread-logger = "0.1.1"
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 833baa8399..b569fb3dcf 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.20.0"
+version = "0.21.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 26f7a621af..f9007e5da9 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.20.0"
+version = "0.21.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.20.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.21.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 362d9f5286..c188595171 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.20.0"
+version = "0.21.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index c5c308aed9..c9092bd324 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.20.0"
+version = "0.21.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.20.0" }
+cranelift-codegen = { path = "../codegen", version = "0.21.0" }
 target-lexicon = "0.0.3"
 
 [badges]
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index 756d403b19..275ede6352 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.20.0"
+version = "0.21.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
-cranelift-reader = { path = "../reader", version = "0.20.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
+cranelift-reader = { path = "../reader", version = "0.21.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 0f84030f21..ebbe58558c 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.20.0"
+version = "0.21.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.20.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.20.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.21.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.21.0", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
@@ -22,7 +22,7 @@ target-lexicon = { version = "0.0.3", default-features = false }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../umbrella", version = "0.20.0", default-features = false }
+cranelift = { path = "../umbrella", version = "0.21.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index 14589c0992..d97a6c7697 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.20.0"
+version = "0.21.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,8 +11,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.20.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.21.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index dfe3f18388..0f9d5f7c76 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.20.1"
+version = "0.21.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,9 +11,9 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.2", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.20.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.20.1", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.20.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.21.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.21.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 1e0c9b546bcf1f14f63906578b3f5730766b2c8d Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 5 Sep 2018 13:49:10 -0700
Subject: [PATCH 2054/3084] Move simplejit after the umbrella crate in
 publish-all.sh.

Simplejit's example program uses the umbrella, so publish it after the
umbrella crate to preserve the topological ordering.
---
 cranelift/publish-all.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 7eb2ff2734..ee477f17e8 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -37,8 +37,8 @@ echo git commit -a -m "\"Bump version to $version"\"
 echo git push
 for crate in \
     entity bforest codegen/meta codegen frontend native \
-    reader wasm module simplejit \
-    faerie umbrella
+    reader wasm module \
+    faerie umbrella simplejit
 do
     echo cargo publish --manifest-path "lib/$crate/Cargo.toml"
 done

From 608e74d8cb8fb9a1c63e4ce9fe52b25144a19a12 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 5 Sep 2018 14:23:03 -0700
Subject: [PATCH 2055/3084] Document that `b8` etc. are intended for use as
 SIMD elements.

---
 cranelift/docs/ir.rst                 | 16 ++++++++++------
 lib/codegen/meta-python/base/types.py |  7 +++++--
 2 files changed, 15 insertions(+), 8 deletions(-)

diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index 775362ad69..d0cfa5ff16 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -96,12 +96,16 @@ different types.
 Boolean types
 -------------
 
-Boolean values are either true or false. While this only requires a single bit
-to represent, more bits are often used when holding a boolean value in a
-register or in memory. The :type:`b1` type represents an abstract boolean
-value. It can only exist as an SSA value, it can't be stored in memory or
-converted to another type. The larger boolean types can be stored in memory.
-They are represented as either all zero bits or all one bits.
+Boolean values are either true or false.
+
+The :type:`b1` type represents an abstract boolean value. It can only exist as
+an SSA value, and can't be directly stored in memory. It can, however, be
+converted into an integer with value 0 or 1 by the :inst:`bint` instruction (and
+converted back with :inst:`icmp_imm` with 0).
+
+Several larger boolean types are also defined, primarily to be used as SIMD
+element types. They can be stored in memory, and are represented as either all
+zero bits or all one bits.
 
 .. autocliftype:: b1
 .. autocliftype:: b8
diff --git a/lib/codegen/meta-python/base/types.py b/lib/codegen/meta-python/base/types.py
index bcef9c1d51..9141f4fbce 100644
--- a/lib/codegen/meta-python/base/types.py
+++ b/lib/codegen/meta-python/base/types.py
@@ -4,13 +4,16 @@ The base.types module predefines all the Cranelift scalar types.
 from __future__ import absolute_import
 from cdsl.types import IntType, FloatType, BoolType, FlagsType
 
-#: Boolean.
-b1 = BoolType(1)    #: 1-bit bool. Type is abstract (can't be stored in mem)
+#: Abstract boolean (can't be stored in memory, use bint to convert to 0 or 1).
+b1 = BoolType(1)    #: 1-bit bool.
+
+#: Booleans used as SIMD elements (can be stored in memory, true is all-ones).
 b8 = BoolType(8)    #: 8-bit bool.
 b16 = BoolType(16)  #: 16-bit bool.
 b32 = BoolType(32)  #: 32-bit bool.
 b64 = BoolType(64)  #: 64-bit bool.
 
+# Integers.
 i8 = IntType(8)     #: 8-bit int.
 i16 = IntType(16)   #: 16-bit int.
 i32 = IntType(32)   #: 32-bit int.

From 8d41d2cc4362ba10eb5b0fcd1f9d87063e1686b9 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 16 Aug 2018 15:11:45 -0700
Subject: [PATCH 2056/3084] Add more documentation for special parameters.

---
 cranelift/docs/ir.rst | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index d0cfa5ff16..26a81ba54d 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -404,7 +404,7 @@ convention:
     retlist      : paramlist
     param        : type [paramext] [paramspecial]
     paramext     : "uext" | "sext"
-    paramspecial : "sret" | "link" | "fp" | "csr" | "vmctx"
+    paramspecial : "sret" | "link" | "fp" | "csr" | "vmctx" | "sigid" | "stack_limit"
     callconv     : "fast" | "cold" | "system_v" | "fastcall" | "baldrdash"
 
 A function's calling convention determines exactly how arguments and return
@@ -413,6 +413,18 @@ depend on both the instruction set /// architecture and possibly the operating
 system, a function's calling convention is only fully determined by a
 `(TargetIsa, CallConv)` tuple.
 
+=========== ===========================================
+Name        Description
+=========== ===========================================
+sret        pointer to a return value in memory
+link        return address
+fp          the initial value of the frame pointer
+csr         callee-saved register
+vmctx       VM context pointer, which may contain pointers to heaps etc.
+sigid       signature id, for checking caller/callee signature compatibility
+stack_limit limit value for the size of the stack
+=========== ===========================================
+
 ========== ===========================================
 Name       Description
 ========== ===========================================

From 437a657899108808114ac87726e78e5e55d0b439 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 16 Aug 2018 15:24:34 -0700
Subject: [PATCH 2057/3084] Document tables.

---
 cranelift/docs/ir.rst | 42 ++++++++++++++++++++++++++++++++++++++----
 1 file changed, 38 insertions(+), 4 deletions(-)

diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index 26a81ba54d..5f1982e1b4 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -750,6 +750,40 @@ bounds checking is required for each access:
     :lines: 2-
 
 
+Tables
+------
+
+Code compiled from WebAssembly often needs access to objects outside of its
+linear memory. WebAssembly uses *tables* to allow programs to refer to opaque
+values through integer indices.
+
+A table is declared in the function preamble and can be accessed with the
+:inst:`table_addr` instruction that :term:`traps` on out-of-bounds accesses.
+Table addresses can be smaller than the native pointer size, for example
+unsigned :type:`i32` offsets on a 64-bit architecture.
+
+A table appears as a consecutive range of address space, conceptually
+divided into elements of fixed sizes, which are identified by their index.
+The memory is :term:`accessible`.
+
+The *table bound* is the number of elements currently in the table. This is
+the bound that :inst:`table_addr` checks against.
+
+.. autoinst:: table_addr
+
+A table can be relocated to a different base address when it is resized, and
+its bound can move dynamically. The bound of a table is stored in a global
+value.
+
+.. inst:: T = dynamic Base, min MinElements, bound BoundGV, element_size ElementSize
+
+    Declare a table in the preamble.
+
+    :arg Base: Global value holding the table's base address.
+    :arg MinElements: Guaranteed minimum table size in elements.
+    :arg BoundGV: Global value containing the current heap bound in elements.
+    :arg ElementSize: Size of each element.
+
 Operations
 ==========
 
@@ -1121,10 +1155,10 @@ Glossary
     accessible
         :term:`Addressable` memory in which loads and stores always succeed
         without :term:`trapping`, except where specified otherwise (eg. with the
-        `aligned` flag). Heaps, globals, and the stack may contain accessible,
-        merely addressable, and outright unaddressable regions. There may also
-        be additional regions of addressable and/or accessible memory not
-        explicitly declared.
+        `aligned` flag). Heaps, globals, tables, and the stack may contain
+        accessible, merely addressable, and outright unaddressable regions.
+        There may also be additional regions of addressable and/or accessible
+        memory not explicitly declared.
 
     basic block
         A maximal sequence of instructions that can only be entered from the

From f3c46ad2a2c651d6457532fbda62f4db8cee951e Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 5 Sep 2018 15:19:14 -0700
Subject: [PATCH 2058/3084] Add more documentation about `Module` and
 `Backend`.

---
 lib/faerie/src/backend.rs    |  2 ++
 lib/module/README.md         | 17 +++++++++++++++--
 lib/module/src/backend.rs    |  9 +++++++++
 lib/simplejit/src/backend.rs |  2 ++
 4 files changed, 28 insertions(+), 2 deletions(-)

diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index 7100bf1917..6aa948703e 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -85,6 +85,8 @@ impl FaerieBuilder {
 }
 
 /// A `FaerieBackend` implements `Backend` and emits ".o" files using the `faerie` library.
+///
+/// See the `FaerieBuilder` for a convenient way to construct `FaerieBackend` instances.
 pub struct FaerieBackend {
     isa: Box,
     artifact: faerie::Artifact,
diff --git a/lib/module/README.md b/lib/module/README.md
index 6bda716324..35f973e916 100644
--- a/lib/module/README.md
+++ b/lib/module/README.md
@@ -1,7 +1,20 @@
-This crate provides the `Module` trait, which provides an interface for
-multiple functions and data to be emitted with
+This crate provides module-level functionality, which allow multiple
+functions and data to be emitted with
 [Cranelift](https://crates.io/crates/cranelift) and then linked together.
 
 This crate is structured as an optional layer on top of cranelift-codegen.
 It provides additional functionality, such as linking, however users that
 require greater flexibility don't need to use it.
+
+A `Module` is a collection of functions and data objects that are linked
+together. `Backend` is a trait that defines an interface for backends
+that compile modules into various forms. Most users will use one of the
+following `Backend` implementations:
+
+ - `SimpleJITBackend`, provided by [cranelift-simplejit], which JITs
+   code to memory for direct execution.
+ - `FaerieBackend`, provided by [cranelift-faerie], which emits native
+   object files.
+
+[cranelift-simplejit]: https://crates.io/crates/cranelift-simplejit
+[cranelift-faerie]: https://crates.io/crates/cranelift-faerie
diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs
index 42e77acb2f..8138009341 100644
--- a/lib/module/src/backend.rs
+++ b/lib/module/src/backend.rs
@@ -10,6 +10,15 @@ use ModuleNamespace;
 use ModuleResult;
 
 /// A `Backend` implements the functionality needed to support a `Module`.
+///
+/// Two notable implementations of this trait are:
+///  - `SimpleJITBackend`, defined in [cranelift-simplejit], which JITs
+///    the contents of a `Module` to memory which can be directly executed.
+///  - `FaerieBackend`, defined in [cranelift-faerie], which writes the
+///    contents of a `Module` out as a native object file.
+///
+/// [cranelift-simplejit]: https://docs.rs/cranelift-simplejit/
+/// [cranelift-faerie]: https://docs.rs/cranelift-faerie/
 pub trait Backend
 where
     Self: marker::Sized,
diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index fa3dbdc0fc..ea9dd36115 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -84,6 +84,8 @@ impl SimpleJITBuilder {
 
 /// A `SimpleJITBackend` implements `Backend` and emits code and data into memory where it can be
 /// directly called and accessed.
+///
+/// See the `SimpleJITBuilder` for a convenient way to construct `SimpleJITBackend` instances.
 pub struct SimpleJITBackend {
     isa: Box,
     symbols: HashMap,

From da0243b0ab460ee344e8efad21effb4aa34ef27d Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 5 Sep 2018 16:35:41 -0700
Subject: [PATCH 2059/3084] Update docs to reflect that saturating fp-to-int
 conversion is now implemented.

---
 cranelift/docs/ir.rst | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index 5f1982e1b4..8bd17cacbb 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -1008,15 +1008,11 @@ Conversion operations
 .. autoinst:: fdemote
 .. autoinst:: fcvt_to_uint
 .. autoinst:: fcvt_to_sint
+.. autoinst:: fcvt_to_uint_sat
+.. autoinst:: fcvt_to_sint_sat
 .. autoinst:: fcvt_from_uint
 .. autoinst:: fcvt_from_sint
 
-.. todo:: Saturating fcvt_to_sint and fcvt_to_uint.
-
-    For example, these appear in
-    `Rust `_ and
-    `WebAssembly `_.
-
 Legalization operations
 -----------------------
 

From 8fb681b86df5a17a3f4717c11b25a8195d26d06d Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Thu, 6 Sep 2018 16:44:58 -0700
Subject: [PATCH 2060/3084] clif-util wasm: don't panic when terminal colors
 are unsupported

This breaks inside of Emacs' `M-x shell` and `compilation-mode`.
---
 cranelift/src/wasm.rs | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index f446bb6ccc..f125096525 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -72,13 +72,13 @@ fn handle_module(
     fisa: FlagsOrIsa,
 ) -> Result<(), String> {
     let mut terminal = term::stdout().unwrap();
-    terminal.fg(term::color::YELLOW).unwrap();
+    let _ = terminal.fg(term::color::YELLOW);
     vprint!(flag_verbose, "Handling: ");
-    terminal.reset().unwrap();
+    let _ = terminal.reset();
     vprintln!(flag_verbose, "\"{}\"", name);
-    terminal.fg(term::color::MAGENTA).unwrap();
+    let _ = terminal.fg(term::color::MAGENTA);
     vprint!(flag_verbose, "Translating... ");
-    terminal.reset().unwrap();
+    let _ = terminal.reset();
 
     let mut module_binary =
         read_to_end(path.clone()).map_err(|err| String::from(err.description()))?;
@@ -103,9 +103,9 @@ fn handle_module(
         DummyEnvironment::with_triple_flags(isa.triple().clone(), fisa.flags.clone());
     translate_module(&module_binary, &mut dummy_environ).map_err(|e| e.to_string())?;
 
-    terminal.fg(term::color::GREEN).unwrap();
+    let _ = terminal.fg(term::color::GREEN);
     vprintln!(flag_verbose, "ok");
-    terminal.reset().unwrap();
+    let _ = terminal.reset();
 
     if flag_just_decode {
         if !flag_print {
@@ -131,17 +131,17 @@ fn handle_module(
             println!("{}", context.func.display(None));
             vprintln!(flag_verbose, "");
         }
-        terminal.reset().unwrap();
+        let _ = terminal.reset();
         return Ok(());
     }
 
-    terminal.fg(term::color::MAGENTA).unwrap();
+    let _ = terminal.fg(term::color::MAGENTA);
     if flag_check_translation {
         vprint!(flag_verbose, "Checking... ");
     } else {
         vprint!(flag_verbose, "Compiling... ");
     }
-    terminal.reset().unwrap();
+    let _ = terminal.reset();
 
     if flag_print_size {
         vprintln!(flag_verbose, "");
@@ -201,8 +201,8 @@ fn handle_module(
         println!("Total module bytecode size: {} bytes", total_bytecode_size);
     }
 
-    terminal.fg(term::color::GREEN).unwrap();
+    let _ = terminal.fg(term::color::GREEN);
     vprintln!(flag_verbose, "ok");
-    terminal.reset().unwrap();
+    let _ = terminal.reset();
     Ok(())
 }

From 90756a8a01523afcea0e67b2efe417388fdc0386 Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Thu, 6 Sep 2018 16:46:49 -0700
Subject: [PATCH 2061/3084] clif-util: Default to reading input files from
 stdin

Fixes #495
---
 cranelift/src/clif-util.rs |  8 ++++----
 cranelift/src/utils.rs     | 20 ++++++++++++++++----
 lib/filetests/src/lib.rs   |  2 +-
 3 files changed, 21 insertions(+), 9 deletions(-)
 mode change 100644 => 100755 cranelift/src/clif-util.rs

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
old mode 100644
new mode 100755
index c54c5b2d2c..925f42f3fd
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -47,17 +47,17 @@ pub type CommandResult = Result<(), String>;
 
 fn add_input_file_arg<'a>() -> clap::Arg<'a, 'a> {
     Arg::with_name("file")
-        .required(true)
+        .default_value("-")
         .multiple(true)
         .value_name("file")
-        .help("Specify file(s) to be used for test")
+        .help("Specify file(s) to be used for test. Defaults to reading from stdin.")
 }
 
 fn add_single_input_file_arg<'a>() -> clap::Arg<'a, 'a> {
     Arg::with_name("single-file")
-        .required(true)
+        .default_value("-")
         .value_name("single-file")
-        .help("Specify a file to be used")
+        .help("Specify a file to be used. Defaults to reading from stdin.")
 }
 
 fn add_pass_arg<'a>() -> clap::Arg<'a, 'a> {
diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs
index 871483c891..9315ff33d3 100644
--- a/cranelift/src/utils.rs
+++ b/cranelift/src/utils.rs
@@ -12,17 +12,29 @@ use target_lexicon::Triple;
 
 /// Read an entire file into a string.
 pub fn read_to_string>(path: P) -> io::Result {
-    let mut file = File::open(path)?;
     let mut buffer = String::new();
-    file.read_to_string(&mut buffer)?;
+    if path.as_ref() == Path::new("-") {
+        let stdin = io::stdin();
+        let mut stdin = stdin.lock();
+        stdin.read_to_string(&mut buffer)?;
+    } else {
+        let mut file = File::open(path)?;
+        file.read_to_string(&mut buffer)?;
+    }
     Ok(buffer)
 }
 
 /// Read an entire file into a vector of bytes.
 pub fn read_to_end>(path: P) -> io::Result> {
-    let mut file = File::open(path)?;
     let mut buffer = Vec::new();
-    file.read_to_end(&mut buffer)?;
+    if path.as_ref() == Path::new("-") {
+        let stdin = io::stdin();
+        let mut stdin = stdin.lock();
+        stdin.read_to_end(&mut buffer)?;
+    } else {
+        let mut file = File::open(path)?;
+        file.read_to_end(&mut buffer)?;
+    }
     Ok(buffer)
 }
 
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index d7238bb561..f8e796182e 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -88,7 +88,7 @@ pub fn run_passes(verbose: bool, passes: &[String], target: &str, file: &String)
     let mut runner = TestRunner::new(verbose);
 
     let path = Path::new(file);
-    if path.is_file() {
+    if path == Path::new("-") || path.is_file() {
         runner.push_test(path);
     } else {
         runner.push_dir(path);

From e051e9f4c874ea84a74ddf42046ea11b2f53bf1c Mon Sep 17 00:00:00 2001
From: Joshua Warner 
Date: Thu, 6 Sep 2018 21:30:23 -0700
Subject: [PATCH 2062/3084] Fix std feature propagation

---
 lib/simplejit/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index ebbe58558c..bfbbfe0f7d 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -22,7 +22,7 @@ target-lexicon = { version = "0.0.3", default-features = false }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../umbrella", version = "0.21.0", default-features = false }
+cranelift = { path = "../umbrella", version = "0.21.0" }
 
 [features]
 default = ["std"]

From f012bd85007a112cab361857650467be34e56d26 Mon Sep 17 00:00:00 2001
From: Joshua Warner 
Date: Thu, 6 Sep 2018 21:30:40 -0700
Subject: [PATCH 2063/3084] Demonstrate arguments, returns, and adds in example

---
 lib/simplejit/examples/simplejit-minimal.rs | 40 ++++++++++++++++-----
 1 file changed, 32 insertions(+), 8 deletions(-)

diff --git a/lib/simplejit/examples/simplejit-minimal.rs b/lib/simplejit/examples/simplejit-minimal.rs
index 054204882e..e393363768 100644
--- a/lib/simplejit/examples/simplejit-minimal.rs
+++ b/lib/simplejit/examples/simplejit-minimal.rs
@@ -11,24 +11,40 @@ fn main() {
     let mut module: Module = Module::new(SimpleJITBuilder::new());
     let mut ctx = module.make_context();
     let mut func_ctx = FunctionBuilderContext::new();
-    let sig = module.make_signature();
 
-    let func_a = module.declare_function("a", Linkage::Local, &sig).unwrap();
-    let func_b = module.declare_function("b", Linkage::Local, &sig).unwrap();
+    let mut sig_a = module.make_signature();
+    sig_a.params.push(AbiParam::new(types::I32));
+    sig_a.returns.push(AbiParam::new(types::I32));
 
+    let mut sig_b = module.make_signature();
+    sig_b.returns.push(AbiParam::new(types::I32));
+
+    let func_a = module
+        .declare_function("a", Linkage::Local, &sig_a)
+        .unwrap();
+    let func_b = module
+        .declare_function("b", Linkage::Local, &sig_b)
+        .unwrap();
+
+    ctx.func.signature = sig_a;
     ctx.func.name = ExternalName::user(0, func_a.index() as u32);
     {
         let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
         let ebb = bcx.create_ebb();
 
         bcx.switch_to_block(ebb);
-        bcx.ins().return_(&[]);
+        bcx.append_ebb_params_for_function_params(ebb);
+        let param = bcx.ebb_params(ebb)[0];
+        let cst = bcx.ins().iconst(types::I32, 37);
+        let add = bcx.ins().iadd(cst, param);
+        bcx.ins().return_(&[add]);
         bcx.seal_all_blocks();
         bcx.finalize();
     }
     module.define_function(func_a, &mut ctx).unwrap();
     module.clear_context(&mut ctx);
 
+    ctx.func.signature = sig_b;
     ctx.func.name = ExternalName::user(0, func_b.index() as u32);
     {
         let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
@@ -36,8 +52,14 @@ fn main() {
 
         bcx.switch_to_block(ebb);
         let local_func = module.declare_func_in_func(func_a, &mut bcx.func);
-        bcx.ins().call(local_func, &[]);
-        bcx.ins().return_(&[]);
+        let arg = bcx.ins().iconst(types::I32, 5);
+        let call = bcx.ins().call(local_func, &[arg]);
+        let value = {
+            let results = bcx.inst_results(call);
+            assert_eq!(results.len(), 1);
+            results[0].clone()
+        };
+        bcx.ins().return_(&[value]);
         bcx.seal_all_blocks();
         bcx.finalize();
     }
@@ -51,8 +73,10 @@ fn main() {
     let code_b = module.get_finalized_function(func_b);
 
     // Cast it to a rust function pointer type.
-    let ptr_b = unsafe { mem::transmute::<_, fn()>(code_b) };
+    let ptr_b = unsafe { mem::transmute::<_, fn() -> u32>(code_b) };
 
     // Call it!
-    ptr_b();
+    let res = ptr_b();
+
+    assert_eq!(res, 42);
 }

From b4c7451ae5f8bc2fd98c2ecdf3655548c1ff7d25 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 5 Sep 2018 16:39:57 -0700
Subject: [PATCH 2064/3084] Bump version to 0.21.1

---
 cranelift/Cargo.toml        | 26 +++++++++++++-------------
 cranelift/publish-all.sh    |  2 +-
 lib/bforest/Cargo.toml      |  4 ++--
 lib/codegen/Cargo.toml      |  8 ++++----
 lib/codegen/meta/Cargo.toml |  2 +-
 lib/entity/Cargo.toml       |  2 +-
 lib/faerie/Cargo.toml       |  6 +++---
 lib/filetests/Cargo.toml    |  6 +++---
 lib/frontend/Cargo.toml     |  4 ++--
 lib/module/Cargo.toml       |  6 +++---
 lib/native/Cargo.toml       |  4 ++--
 lib/reader/Cargo.toml       |  4 ++--
 lib/serde/Cargo.toml        |  6 +++---
 lib/simplejit/Cargo.toml    | 10 +++++-----
 lib/umbrella/Cargo.toml     |  6 +++---
 lib/wasm/Cargo.toml         |  8 ++++----
 16 files changed, 52 insertions(+), 52 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index e4c28dad9f..4ae89e3e35 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.21.0"
+version = "0.21.1"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,18 +14,18 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.21.0" }
-cranelift-entity = { path = "lib/entity", version = "0.21.0" }
-cranelift-reader = { path = "lib/reader", version = "0.21.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.21.0" }
-cranelift-serde = { path = "lib/serde", version = "0.21.0", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.21.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.21.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.21.0" }
-cranelift-module = { path = "lib/module", version = "0.21.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.21.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.21.0" }
-cranelift = { path = "lib/umbrella", version = "0.21.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.21.1" }
+cranelift-entity = { path = "lib/entity", version = "0.21.1" }
+cranelift-reader = { path = "lib/reader", version = "0.21.1" }
+cranelift-frontend = { path = "lib/frontend", version = "0.21.1" }
+cranelift-serde = { path = "lib/serde", version = "0.21.1", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.21.1", optional = true }
+cranelift-native = { path = "lib/native", version = "0.21.1" }
+cranelift-filetests = { path = "lib/filetests", version = "0.21.1" }
+cranelift-module = { path = "lib/module", version = "0.21.1" }
+cranelift-faerie = { path = "lib/faerie", version = "0.21.1" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.21.1" }
+cranelift = { path = "lib/umbrella", version = "0.21.1" }
 filecheck = "0.3.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index ee477f17e8..70ad24c09f 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.21.0"
+version="0.21.1"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index b2a5f6b25f..a4a71c48f7 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.21.0"
+version = "0.21.1"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 keywords = ["btree", "forest", "set", "map"]
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.21.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.21.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 2c18b92820..4dc06b7c44 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.21.0"
+version = "0.21.1"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.21.0", default-features = false }
-cranelift-bforest = { path = "../bforest", version = "0.21.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.21.1", default-features = false }
+cranelift-bforest = { path = "../bforest", version = "0.21.1", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -25,7 +25,7 @@ log = { version = "0.4.4", default-features = false, features = ["release_max_le
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.21.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.21.1" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 9a16c0fad2..01f19298a5 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.21.0"
+version = "0.21.1"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index cf4378618f..35c0c69df5 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.21.0"
+version = "0.21.1"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index a3c39abdad..1576dce164 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.21.0"
+version = "0.21.1"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.0" }
-cranelift-module = { path = "../module", version = "0.21.0" }
+cranelift-codegen = { path = "../codegen", version = "0.21.1" }
+cranelift-module = { path = "../module", version = "0.21.1" }
 faerie = "0.5.0"
 goblin = "0.0.17"
 failure = "0.1.2"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 78e58fea29..6ef936a4b9 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.21.0"
+version = "0.21.1"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,8 +9,8 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.0" }
-cranelift-reader = { path = "../reader", version = "0.21.0" }
+cranelift-codegen = { path = "../codegen", version = "0.21.1" }
+cranelift-reader = { path = "../reader", version = "0.21.1" }
 file-per-thread-logger = "0.1.1"
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index b569fb3dcf..22b4e7076c 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.21.0"
+version = "0.21.1"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index f9007e5da9..03885eaaf9 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.21.0"
+version = "0.21.1"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.21.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.21.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index c188595171..0bd870ccdc 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.21.0"
+version = "0.21.1"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index c9092bd324..000f4a6660 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.21.0"
+version = "0.21.1"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.0" }
+cranelift-codegen = { path = "../codegen", version = "0.21.1" }
 target-lexicon = "0.0.3"
 
 [badges]
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index 275ede6352..10777476b0 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.21.0"
+version = "0.21.1"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
-cranelift-reader = { path = "../reader", version = "0.21.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
+cranelift-reader = { path = "../reader", version = "0.21.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index ebbe58558c..d9bc47b375 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.21.0"
+version = "0.21.1"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.21.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.21.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
+cranelift-module = { path = "../module", version = "0.21.1", default-features = false }
+cranelift-native = { path = "../native", version = "0.21.1", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
@@ -22,7 +22,7 @@ target-lexicon = { version = "0.0.3", default-features = false }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../umbrella", version = "0.21.0", default-features = false }
+cranelift = { path = "../umbrella", version = "0.21.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index d97a6c7697..a15adc2390 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.21.0"
+version = "0.21.1"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,8 +11,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.21.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.21.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 0f9d5f7c76..e17b7d73b4 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.21.0"
+version = "0.21.1"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,9 +11,9 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.2", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.21.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.21.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.21.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.21.1", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.21.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From cadb76ef71f288708f312cacd9ed3427766d00b6 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 10 Sep 2018 09:59:55 -0700
Subject: [PATCH 2065/3084] Rename "Cretonne" to "Cranelift" in image files.

---
 cranelift/media/spidermonkey1.png | Bin 127604 -> 101882 bytes
 cranelift/media/spidermonkey2.png | Bin 123766 -> 97736 bytes
 2 files changed, 0 insertions(+), 0 deletions(-)

diff --git a/cranelift/media/spidermonkey1.png b/cranelift/media/spidermonkey1.png
index 0304a0a4f3422f8df6b69ecab7a673e21aa5b827..0a8d0670cc9f44b3035998c568b8210b36eec25e 100644
GIT binary patch
literal 101882
zcma&ObzGI})-}99F+fG7q!B4eL0U=_a8Uw+bc=u}-MK_T5ox5PL>
zZg}U-E}!Rn{`fq16Qy3HqyZ+26
z_=zqxS3LaVl!1)YUDOfspZJR85ctVi%X{iJC=@Y0@?R{p65Vh3;b~j6oW$v|v!_WI
z&oz5tX`oOvDD>Ui%8vaDLk_Nlc87HhL9`d&(A1p4{_rwTjH^mC`25YhJhsH#
ztm$#ifJbvnEl*c>np~&$D9^%E{Qvkv=;hlAFN~3i{>o01bqaNnz{dxLQSkL85>}|W
zS1PTf^|eh}=^2_w_72^MAzbiYLy&C)H^sY922-i8*(&YB7l-&Jx=>=WbPSgXurS4r
zJ~a}mI4)V|={CYT)AURl90=f3H8ZEtGI-wf!S0=(98|A3@t`mf_he6^UW#eis$jNM
zwcBr?uhDG~Hk)CgPzr+Q(2aC*A>5knrYeJYx^Q`v0WIex)I|fUpDO+%4_MG1LALJ`
z8@J`*x8u*n1aUuT&udu56=OaksjN@KLVdq$K#QVzId0&@j4BzCkwW9kl&UL5WD%u4
z%t$rMERlTHpRcH;6+JdK)|rSV=t@!iX!-hWqUJ;71qB2I1nuUON3%VS+!uRu?%c9p
zXe)d+kZ(RDXk=uxGawtAs2&^}w*JgXX6eU2H^UU0;J=(4748gKk4_w|B$k(z?ZjZ6
z=uA`P&~_NIJE;vM!oi>
z!pdRQy`|ND1$`7qIGL{}EBn?PpE~o>HOUl}w1=+aa4m(g_V#w}XZ_rAaU$9-TT2^-
zy=Tz@(#~>9=A`sn|E`#r);igh#xWl4U<0}XRi%Uu&y|&xD1NJi;NW0FV&YmpJh;{;
zb`FmFHq$LC*4DYa=7TfOateZ%0`q%9JHu_Q@9UgL(I8uk`Yut0m7A~IN^hEy$VV?R
zKr+|)IV3nFgvauiSfvD9hly{hH7?7CfRUMpr_0pdJF$$u`MTHlKi^EmE4+Q$A=3K{
z`Y3qaGm&}9DB<&fF2aG&l#qg|b7gi_~^Wjd#ZE<$mQ_+6r(y@@_(!n-Re`J_D&z1vfD_-U>y
zZEX>*Bj%-0;cPc#KkzJ7yR0ybRXt&FP-|_WJGfMOS52vq{jip0^$7~a85D5_MZtqT18k?lq}GUp&qKL
z*$$3DswYsdeZJpCp{#Q5NhhEl;n%CsePXIgRY`H%+tkR`drrptw7v)~f+8CJaOYKL
zm50Z(;qpF_c^32u+Yde-@jk+TERHX0dW5L2FwH9&=Mk#FGW*5U$;NPkKsY&;KTLYF
zKZJ4UW@c)a^&8u<)%$#tbIKKPVE88ypvDYntT>r!)KR&oAB87Q8@Zg(?a9m_KQ^2sqnH^jkL8q-7&+zDUu2Q>@6FMT_4Nux
z=0nj@+anC7{_Nb#MSLraf8068_Oxm=4$WGdczaJU?njHf$Ibaqx|pNJVEt(5C6zu2bLf^?puxc;g!rCK$
zd-L`!_kDl%`ghkTR!#ocy8dy!ATQks7dzaBEt%5B^0q{Dh;4@)pVcFH;fvNjZnn1h
zN=iyiA>?R5W>d;P7TfK6jqez+L^`hTU86~R0T^m?l_(PH>5B2TXEhe!4;EmrczKW!{qVRoDb<#
zY?qnxigj5M*jO5R|M3ju{lJG1(vVb|hs&!f5}krC+?wd(2>vGm#p-Ko?$aOG<_tJ!
ze;1qz6V-3<5c_#!OFTxEI3@g$s#`o`wwt}a#GjLuHwwTZr9$$nzDfPc3`
zT!j5XkJm#ni8);M^w-NjhOtxLCDN^7|M>CacthgF936`xlPrdW(ACUV%c)rX`x;_e
z%I-roI=*M6vvFv`!$(`Ep4HUVnf?0moI(Ir^UEXne{TW;il)cM?M4lH9RJuw=BlN!
z3-+c%K4B&H^6R)%&CTBeLnf}L?nroi${KsKJBNh}Cv{@PR$LD4UNd_89=K10>o?|a!w^WYaK&wWPm<*Nj
zNFku@jOT+3a(@)C_biirIME|i!`RnRPG(r04dn_@bEI(ouU~z*I*eV7c?
zOl;S~fQc&9#44xqHKr5485tR+vZ@YCjWk?c%S-JS8(Oa8L`FtBuhftf0A^VnEO}7o
zu-x63XUx`kS3=^m)7sPDXc{-XW4*FIurP1_TFE>Q66MTUiiC55qRo+^kjtn>eDK2+M!*qA5lu|4`~qw~Ri
zoP$az7fAv(Q))>Shr9EoTf;5_CSN{Lli1Ml1lh{(WMG=^#IE9rIc+XFx}1U&wY>E8
z^7FIxC061-bT6ma
zYpaG2|HO8~!{y-$;iL?$60Vs=R-N+RkK5PKcqHts?R(bF_-?!FqZq6Ymq@Nn?wq69
z+13u;!VYx9*?ZxILQS^C3cGYB-=_{>7&X6i>Czh?AI0gGn67ds8!s;>{1bay%c>)l
zZafylLiSJ3@;j|2Msb@Okn%q3|L%{RYz)_wGD;)IK%#A6<>5?UvDL>5)FKp}h4zd6
zpW@tie{f}}=RJlOn1lzQPl=LKyTnWN`-C^ZNJc(B)jQtjR_@W7b*6k^jCB;>+g$8G
z;^km##B8$RU4Nc&0N2f%1*RPgZtFj#=7uZE>UdjYh0{j%r}zXNf79GTvMcK1Wk@rV
zoEc1P7zn!w{-GP^vewG2oZXVei7+x6Z5xrVY0X2iFQP(kT2$^N6+Z1Vi0lRsRI#^I
zRyy{Eg`(%P(x*2F*w|bgFq>)n;C#3W03j^VU+vd{&iCUtgcs+EhzS8U!IudB!Tk_3J(yP@+?^9gF!;
zp_RiGE}FPv-ufo6kYqxRmSXb*WgcmoMVbB-0!sYwhH;HI&Xs6A*M4j_=jo4q@uGEk
zm+@f7xjkI{+vXG~g$vI{D&ch&b7ftYN~Yf^XQS`kYm4ACWbJ5)6U{n0SoT;jK~`Vv
zo+WeU$&T~U6)0}=L4Lq_B>qS19rvZ>lzos`oI$JET@Jc
zx8AvO8WoyTVAkU-I}Zmxs0dz`Kfsin_oW
z_TYeC;=xBrzc=Nr?|ICUYW>JHcUcOnBx%krCrJ3_>wU#u9I4V3Jy;J(`1p~CoIDNU
zFGcP>FB!jOeD$|)2j=tBaUO+2;VFwkVB0yGCc^0P<2isM!HFS^_nn(WttVbz
zpcI_gu)#(tDB9ex@TU?^1zctda2u9093u3i^gHIy@-c!>y3-?f*JnRMyz>6iT1<{}
zS28y@A9Vyw26fidrzXyAGZMPPeB)NCu5|SWIXOAfu+?@w+S7Jr#;vhkM~7RY&@iZ<
zzCZzq^+{X&^ZtC(k6&M$PJ=u(xq5|o>ktQ{pp^EI4WO2$xwlr8dqr+uo=S|g!PWFC
z!O9X!Jm_$lRQg$5v|s2*oYp7`B@e*h+}@Gz%6Q%B@};#_QH$^2zdJttCZe3G{NoDm
z(})~sGp=!Pw9fNFNtnr5+FP!oUZ0NQH1w{s4`bKXNR6vm$f=`R>urH0WuNO6K3MOl
z&DQ^h`wDUoZTtp2v!8LSQ!RA&-XpF3xkfUE@S>9D=9#U}7#J8dx;672`{`ft4hj!9
zE?;e6k%S0jThn#lF>2g}1J4CJj`yl)#C>4J?qRx`zIvunW0=<(bjT)yj^lpbjgzap
zPGo%MPi5mohUTZYr$vuu9pWNh!5LV+fAS30L>E)QjwX7F&0GJ9aZ5BO09-$a#=&K?
z&J4{S$H{l9Z=vbl88ScE-C&z%*C{Xly%FoWiv9?DtfQNJKTf2UZ^U&xsAqp8%Yz5{
z5R<)|5(&wqZp*sn4Uv0~%dh9BS?33#
z)WFSa@kRIbBB>xYq6O_5mN!dnXARUjXlbkYt;X&~)>@AVDC}D0#IxH-+>V)A^(>Wp
z=P4*(?CCfW@BoV`Rqxvd6AJ6h6t=h`uGo9tXIyYJUX-dWQ6a&?W7=2gR#vx)Mx$?g
zWF{ukHS!u57?f8&M!ON`_2{@X8xBF8E9eZ^LY_g(u1n>%N6X8XC&$-kJ9LWXl~IBo
zhviml&3w1aTXj&mi~*3G?ObDsSRSf{nf*-gg5AlnomC2;4aw5454>6%dDB>^@IKZV
z6iJ!F@H@%jTD|@-m*WpeW2}e
zupw!0Z|_4vvNyd8P@wVA5%%}HFQdHhNSs|c;MAyB%@;fy=+132H*K)^w
z(T$Rfg@vcc@e8wa-)YnhGd+%O0usDAt_|EdZpG(>nCdsZp|3N4p+X~GoW_3wO-Vz2
zy=Am+mX2^&hNdce_*zwh!-$(|735jU>6)4vC+^ca(8)tJJmK!act5ste`k%Y16k5-
zcj?u<2;}C52m*pSU9NP?{FcL#ayYwoK2)2!oytk
zX?+(QoPdBZSYE8GwI7UeDIQ0Mu*O0jZ77uWjgV_n{xe~k=0D-E3r`_QNiK?erC)df
zQVqA>7nDr+jl*buD{&}ucCI=JApq)zoaZwHAvvA8K#^imb-??pN2A306P43M5SOs`
zrwR|OUrIfnWJqLs4400!Di=m__p1Lh`xGxEAQ3r|CKs{rv9xj-$DN
zUv3tf4?U5MxbZbbJuj(FaEYPX9unuKn*CH$B$h`KEN)M3X5%Dtd0t|^sIMj*@}JrI
zcj}S`p`7w`!yY8m?EnEI2^COCr^;@=ySDT51HXg4E%K6*l3F0G-onpF?~5MfE5acQ
zm>h8&KVP>vu)LbXmmK~XuMiF;>2l>>W^36i>}OcY@H^MIxNUk|LRt3_QElxAS~zVHv7fu`A0SM+2AFT4HV)FZ35QZvT3L|CT{6%JJzD
zEYZ1bfBiMbM~@OYA2(e&PHs=0q^`iWUm;+Ww{1cL&@duh?#=nQG7)m0Y;CIfCX&d?
z0VhcYPzh7maI94_vp>&AG5a0hT_m{l25PDKj!yUI2X)3QkwwGKjk)^;JvwHQRZtlN
zGJm>R&7lqYa*d4a&wDyv2-VhNO=;vWK)q4E1Ox*K=E(yT_Qkh)cw&Ig*VGP0k9Gn^
zdeotK)ZH{bJlv|f#qP2`qu40nhOT8R8od15nCylg#%Kt-gU~)>&b?NT!|^;)oAFvd
zzmdbuqT18cu6h(uc+~>dUf_$&tT^)x*@jqcE~sbCCw}@w3T1g?hM@O)P$3D1mX;RR
zBOoAvkC;PeAx}-I6FbnU7gXdq0qv&P8ua}
zuZBUo)(C?hJ1QtDYR+aaSw4o#`y8o(dKRP=owCO{b*@`&pC2%e4_H*$X&J-YOBOuo
zqUcG}D#=}eNA8$k!-c1<>gN9#u6+Ed#erTLk+FSb6<_4Q4{R#+6n6VuO+!d8e)
z*DPw`I!x0&klq@!2^iV9eEBlB{eoI!K95;9Kf5MW=^>~_VWw--t*Q?nhK=+HIIpL`
zIy{ZEpN??|gOx3tj&-rEqjYeL@%Z%xOD>X2qUvxXtEb;BY^5
za&nRod^|fl>pc4@oV8(Oh5`>_iJk>zys3=Gb;C0kI}=OG{`@X1kP7n1hbb(;~u>rZ)W
zchDW`?FQsw%_K@8M+HFDXSVZ@Y^m$I;stHvFa9ckRHciB0~FK3`@g^KRvqn$0<`0A
zIT{!_OalI;F`r&0Ec6E_YuW)GDqJvmsk_|nMELvn^H$AYh9@-v?7`*u8o70czVzuf
ze%35)FGms&Gcm3hs^J~vBh6Z9Af_-q6;n}Wtm6Yk7F(qY&=<3fL;oy|Mz8FV*-SP4
zgzk(L{kCXJO#=W2v@$&a+uTGf>J|Zz^gyPyJ381(gTj>xnduZhRXQ9T8CXNCsvf>!
z$8Wj!>0mK+R-@D+&r|R#^yqq281;wix|6oe+s}8)WtFeKX{CK&bp{{52TsiWxEg93
zTKD;kB8{Yq{nbW6*XI;nn`sS-^t1vUm&u%<@JBNQ!o7=IKd+t}4uwjzf55{$lJuu}i&
zJ)U)^tCI=ZrnEk5iQxRkg*-g)K2MBx)tj`~!x2aWRK6Q@nZd)ZYjW^t%<*>jV}%eR
z!gRRY$+|=f!Pl|QbJw#RdWzv(H**bVYTvS(ZKwa$%4^a_(?}_Y#Kcn7k$csT$cipS
zbjfVe`%WlYm~d)Q_i_=4j_MVt(a(lTRaCXCa6Ep&inV-|kLK4W)h&0-fihb$rL<|(
zEFc%p=3!`HV7pwgwX2<}Q{ha76kVb=KZea)-pxBkrp&$&OCREuK1t7YXS@eNLzll<
z6OYE_DgwxG@$t9RGNGh8|NeR!uMP@wD!>k>!KT@D!C_pnPj*ONRAwc_fP`UK`}X>3
zsjl0Gio)4JN{2!H>$I+$ea0YVY(`y4OirHn<*3Y-UF$y3G<7FtM4AIUj`Daw^BS;4
zZ9u3aEIa5_=pF6XyN|DULjpazPERiZ*Zm0}or%b&5^+^KZs?$iQczg7Tc7#)kNf)w5)z9a1zB4UAxi)VO1JC>q}-WjeE;wZ7V2uu3n+PV)NbpFquT+Z
zhbfhAdvoIfq7NZB+%f@B$U*mTb7rgi!=2N06t(Pvg8hb30-`H5!{-AIs?|)dAFot`
z!}4$#b{o_uyY1il0pth7tZI6s%Oh1)P?i9rfg
za&>i$)OkS@dx8j0`wLb|8xndMIu#lohih>gW4;`x3NkCmi{WEC3l0ttVqs-^ndPg}
zc8mQ5TCT*pXJxZ=D%KbBT0w16d4*5i%^2sJP_;kFf$SUTd~-Gj<>f*ZyK`yBDP$S7
zyE~sHoCIg^M#`2N05b&Hc)h5&ULKN`S(@5eX$bikpFP{#%gL1k*xqJp78K2d&5h*u
zS0TB5nrePO4+*{taN4`bNcHsPJfK0a*rq$}G7jW_iQ!o1$)TB!SA;fj8bvAUJ~%EQ
zRM}#&xVBJp?MGf(39vHEtm%&I)5IHOCs!y^)S@*|B~L}M;$>q*+>ZAgP)dQrvQDEV
zba%8Cd{?jDR@Jt92e?Or|M-XrIcATqZZ3^Lz^a&w$ixHDg>-Es?zIqvrjYF*WhqKa
zzX2*|qIej<&c<9?)*Qaa_6wt3*dJfE5?o0jF=QsormaiftZP7zllyi+6ODzs_(q^t
zMOFLL#Ka@HX#SsE!Z&X|uqa>g9tIGklB$e8d-g0s*6eOB&TriAdHw|2sGuI`d9Azz
zb&PU6uxONRXWDMt&2{Do1sDTAAuTPGDpD@ujxrT;heB2l
zXhyVEJz}L%*Da)7jj7(i;vNAYZaph-bhv-2inAW
zGVXhTcF*q?xUan*QDgsY{OFO%k6?PL#^z{&j`sZ`)hrz>dYVr=>Tm8q-dI1}TQ)=f
zHGZ6s-CQULTC5$=lCzF0Dk@UVt2~le#2bli3QJC)Mxl&iii9A^Y5V?iNw1hDyD
zNE%_Iu+mr=SNUjoFSmF67|+_5NHU&g(PQYw7?!
z9t-dSvZ$Cb1mxlX9ncTM-CUza*740lyO`a9jQTfGkEnzk+qiTfZ5{@JfLzet@)f~d
zAmSl6>PV97$S4|~`{~)e^n3IhqNh{2{=R+T${qJbICZ?g!5FbadO=d=aOQ
zAdP4NUA5}irV}UPW(%FwXU!sW`u2)m1f@9ix$dvj?E(mWpw`q2BoUl2U4Bt!=7#oG
z3w4h$tMOV>IH_LaRogWj`M}ut4lgYhjbx4h0TgB`dbs`3s2nJbu$7Ne0n9+i0o_zV
zSGgy_mUQs@8-AD-pd`#{db;%uhqn|WuEk(MVE=X_->8uo)OlH75{_=z6iS;P9!Gm4
zzz1*CKllQ?Q7|FUW{;brOo3ze>Ufx`g%IjawnyD?6HF=la!*zq(5oxeE3KkOTHg9h
zw{8uLmk+CArLZEa!8Brmbu#|0fUxonC^uGpNI3+WDqEJE&pcY>Xy4{MId79iu7Ss6
zxp)XfWmuf*6^MCtjnZDtqd}17-SH?Oz@PMFJ|i6gJR{YvN(cyl(@}2{dOX%~`R3Ne
zxH4J*n`%L&w>d|4=zlCzDge*;DYxP40>Bl7HWocx3D^b7({7=c(+z6&+(1!QT`p9j
z$uE%X;ydBQHF7O2B{c(OQM9naz{&ZueZ$e&xwP_NQ!_)eNFFRCVMcC5mm=T
zh3}XX^4WwK>qcw5aAZLtZUnC3Jmc2*VdHF)xn@a!3RBX!TMp)swVOHgk?7vn-`vvAfpgh5UyHe?ORu9F*r)ULmuZ!VAM^7odJ
zvzmc<<~G|AM6oJ#bnXF{K=Y6=onV8cp&~0QJ8B8o?o_!Ou*7v#E=^pZf7y_{+tYs1
zTLsRd1vZR7wP+@waBX#Ub*~PMe3Q|i+NB)wtys95JEF%3?u4=sAh_+x_5ENmO{d-v
zVl>sz8+1GHoOToSmm2c{3Y^}jy2-`0KKhE9c;_a71#Q65EEEWd3wDa7p9cdm4#Tq!
znx78TGb?wW1epIyl8sDQUtjkXT!KEUP%6a6WuW-baB<0lY_2A80j@YAg59sf!F(;20wlR+hNoX
z4Q>Xk;f@vuRDXm+HqrpBZoAQ?>cXn|Nt&qU`**b@2(#<)Tj8L8j~pyH_~R3u3vHZi
ze|_m1YX=;9quE0?A)!G-Qbu_5>gsA}8V-JNEkcikwEA^7U9by)z!F}qzZPZ+RkJ7l
z_6wmfIeGbqpQPWh{+Nbdk-BaEE#Y0z+04Hf^b1)
zRrB_(fk1CJY=8AuV5nto-(E?DCp6Lk+9tE=(Af<%Tdd_5a0uohfE1y>
z5DM-Exkf5n+di^R5c<1PC@QQSu})K@2)Q7lJ6}VhyKbV-7zyp|HZjVFP
zxJKC+!LE-iUuR}a{HR5Tt#+VybJ=Heg~i*?gKSXRUatuZx81=;k7kk|<*kQMspmBxiIpyphnBkdS!i3($=`
ze(VG4Bn=ISqgi!cJ8<%3f!2NiXteU^&;{a18tG~P6Cui^8>m%bliKXzz(Or<@R*mc
zNH%W&2x06hve0dY7IGZxrDoyNpD)i6bSBA>!_krnHdeJejZc*?R8bwQn?9
zT!OEAJ$q@SVJ=TO?V;RF;}&JWo@^apDoO>koeG@^(sTkGP8_cb0CfzgQeZYs5%>uU
z%>$g7w$tQ0GGXWZx|>fyWl0A7o0WA0N?|HsY^srfpleb{N5t+L4=N#J0g%n-uiOv=
z_>spN7Z~_!ji;dNqFY#Lf%N~g!cY3r(x|(QPn|)R9dkVmD6{#1S%@3
zrJs~zz-6U?ilhT*rNa{k&vt*?>Tm~zlGZYvWNQ`Pik_T$1*C)|=-h3%k}
zGMm>b@&Njs{HJ}n)w3X!@9vEpwLCWjTp2k7$Q(A3a@uZbP#sPczVrJ^T(QI0qX+#x
zG_U!@YNnOk^_j$#5))H{Q)S)+D{CqVqy6Yfr?9?!vgk}v^ggfXR$L~y<@r@=y}H>djo?6
zqn&Ua=j$nkidF7zjk{23?X=86$e`N$G0P7aqR#{%W^1bptk1TaE-+TVOO~q+cg+r0
zP**YKWRtgH?L2^)FP~xnvh~vnuw@O~%@*#4w#&mqDZrwR=fX#|zHlDFZeEyKOHN91
zE^h(t4qIa@NQx>ay(d8eFRqaf{CG1}dt#!h=w)3@#2Nm$kEO3bK}xze&{0tB{NJotdhcvn|fnb&bX2D75xtsmlmHKOtr
zm2>^d(G2wNzU6Ys*z)g~Ro()Sw{|$@edV;)l~Ac;X(&Py63bKPKcWz`BoH|WWa@%N
z#dMp3E-SJ-a}81Caq-rUIFdmH=r6TPtM#Xxci1T|=0{janQ(R$I0f{R4ieqKvC(?e
zLh5$rjtTReKg>uMZRBuY!9#>qfU8wqpwGtlSd3Iw0Kb@yXe{8)L|CRT^B|il1LNEQ
zeE32Ex_p30RqF#BKy`$L-Vtleg?z!ork;Bj;&!c!BeP_REJH0v{6UiJ*_plJ978EY
zxd%iyPJIp)FZ=G{uo7_lh)0)zLH10lZ!~bDK%Zo^gIv%!xrW_{hg8@L$>x1flH;M7
zAa;5$KlBCQy##d1*By?JT*+vW`S>D7v-Wjdj%{JB9U{6d1$4Xlq-q!)8gzc}9X<*_
z@nLnO=Egd3qhMFEla`U`KsXf02HI9NM5z$7U{2p`AiuEy-ves@1NaFrfJul%5B(){AjnU7;JgL;`#>wK;Oy)?x?M-@
zk$&}#_qNun13Uv-F;))0<6_l0X5gQwhMuO2;X7Xvyo&2ihEqjS8iC!$faV!u+23Dl
z1&c2)c*qp=cUVwZVpg{RoI%HJ7%8c9{!1nk4h^IS)E9apk^q^xcpBj9Nmi)1ny&#h
zJC=gD9d29x=jB0xIXpx?bWuwdYlizaDg0Ze?nH1!V_ro^Q`lp__5n+(y_S&n@ozAF
z6UqhSxj!<6aPed^MHKv=Iw(SbTFoO79uF}u!3Q2{H8C~VvnQd}E1Vv|i&GH62WZa7
z3vh$cFAuT<)Vs!f@Mz{Cwk;G*{@B=-M4$ZKz{dA^Ey91A*-GJ%LT`}-F!VM!$qSkg
zQwgZJy=WQ+Y}7>!tDjl{oY+J1-zCaV%V3$n4*q#B3=SUg4D{;$Yf#KHK=8^5`R9JI
z2~X>ENi4ksX2ULbmfFob-k?WbDjwb%LZQV_5T4ZHaJx>FSwJ9?5(_N*s1GD04)qQd
zwTd@dQCv?&@udYPVk6(Z`vh8tr7e6IgGyJ=lLX{Wg}4Wc!GgxZ&Mpt?^DgLO>p&VZ
z)6#lUo;bb~5efeFDynNv-KMUqVFF~qT3G2-8tpf>>JcshbOHa1sCe;GaCE&353ln_
zDKLYlV9tihiRWUe{X*{tfDCESx*})w)!)Mk_F7t$zf!3rK4jDA4rCvgJB@~h68c|&
z%h;4?g<+wj-((F&mXn=%hQKv4o+o!#P9IMqV2YV0d-OpS!4$j7;@>pzpHH1dLfn><7AVE^X`{w`sowUDsS{;P%P+%)}Ul>1cVDH*F2CRn3AbbHU#
zSs*0FnGo}q2>(RW2=Tc;JcG7-8asAZ$}k#OJIpB33*?+nU(mT3=;#5*$aU_;HN^VJ
zU*Fi+0T_KTgoNaOeAK2#H~PuV+~&25_wZ
zX0f&p+gCkDS^uz^f}K{K?udshtJ>2Q-}{@TM)2h4yWshyV1t`}d44CeTvqAv*X*}{
zm|z>S%A&(;v>9j)!Rj=dJZcnmb(U>XRPSDJzSX#7bDU`0HN@8$(G(}D3zPu&{f3Zr
zU2R9bX^(%5b8y&F_Dg*$X2#PVkAKy~s{g%G*73Vy^*t?76!0`aND5fP6)=gA$q=qn
zWCstuC+oCP>1q1!s$DF#vI<(AXCHpTWj^d4K2e6FK;uQd_B+XG5|Xf7tWe~~({PP@
zQ9@3+N)H|^hKzK0-HI9d=JMYa+O1aPa2|$R**wLWSY0X2^t_s`cZLt#juQO)t`FO3
zwoZYTgLDwcl)_(5?02!TnNS&Hgq+%ioSr8#QeL(TPn4qD_}3GU3H0@iL+|<~t_8OU
z>^Je8UwY3#5qF-D_`3_I#g~fPd#%&-G1gg@W8cpr(+(68@Qg3>L8KaqZHo(A&t=nh
zz%RjBlMngV-hHp8=0QVN9)FkaU~8=n>n8u%EzZ^%f{ch|>QMQ8dV-Qh{(tGwWrdvmh?dTp-~kT9Y(KH~97_IgBV
z?iYJFPp+;}HPyseYR}-ZS9AB4J@*aemQ%X5B}sej8pr|6Kzdm!LkwT!;(gj14`qP`
zm|Y>Kw8Rl&y5H?yWyOPo8SCHZ<$`Sg8~xB)wX*U}ycBFbF?jIakLtm#>yn
zQu!eKQ(HfnjOA#G`CqedasK$&Vz7Igmw2ndz;a10F~(KoND6nKRBUsqXo^8{b7T$I
zb8DoTytl|A4zWdncf$V$EEc;duoQEkhX{WNPh_N9jaE1(z6jKvNWL-Ulh$N|Yhm|xvz!dy^I3HN`ugJCyLUA~a0U6WEc@QCy$@HH{&z3JZMC#$r)zoT+1#S%c>O(1vFHX)Ra9KE
zS^G#r=cYg01{R%9P?4=1kh8@ufIKhIbd|evEPLr3bO()LH>Xc=9&-G*7Ifodc-EO&
zjx2}8JI}BtaDP>lOdP3;Z5pE_BBoT{t*0jHM)37R`@p7_jd9n2UA08>+s*{Sx-8rB
z_3#DPN(-_~K8>}qEmc<
zY^LRj+-s727Q^WfA7;>%xFXg6s2(>(Qs`9JZ7F!P{#Wd@7vk!TS66
zRGSPJyWmFQR9Y{=ZyQYY3D>wKVzFWOExzcG{Ci4L=<|c*!d7E1hn=P=!Sp!aZ@9?rp;rm-Y$;!XX(CWx
z5eVCb5H~EWtZyw4B+(E9=?>AtV00(~QdSIc$dv^
z7tDE^jWtDQ;C}z#v9)D7VcOaF
za?#%YeFd}R%4i}n|3B_!uC-2bO(q{Tp;M=Wvc|0)_OyO!x|8j>@*#Lh#ob05pOcf5
z40vwIw^u$u&#=-ASwhT527r0KX$QC4t}Qa|F<_8Pe<0N!?MXM}n%wOqimuqnG5S`V0Z!?AMAfPLV-$1O0N1r
z6n~Lu1T+i?JRkgSpAr*IK;}GrsLnKNgqC`Sc8X=Q+OlU1FG(YJsiZV-%<>IZ-TE)G
zl>&V6*de-!JQiB(@$M-Ol$K(
zG|w{NsA})yb?Linaza2BcHo2gI3aD54a2xtmR9x=_R?)b`Fp-MzuMtXO>g(cZH&8L
zP>5g`!k@uKv9MnrS*}FU0?Y!>7W#3u8)p5J5)E$(SD1=PkaAaAJs5>C?VBcNtQF(jo
zhwi0%Vc4U0bVJu?Z=-plc_|18K8}8SW%m8`CB!JND}~(r%q1{@BQqdL_v11_)^OHb
z$cHJ^=AT}}Ow#iS;T69L7)(=ET?$iL1lVEp*MPPmysL0UH&+Ii6su*eXSHs4x9)TH
zcv{W!paCbs8;8JH-U-GD^LWItrIZ0x%a$k$40YFBh+j_<4pjWaSi
z3jW;!0HGGXZj7^+(61=u`(->UwmTHsLxTIQ{qay|aw1;CZ62`(X^*`dH5ydKs{cH|
zXTgSvej~Dpt&l_F0fsy@{g+Mbz>+iEciUxx6Bfv4{+OY4MsA1l-C3JHeh4`^yC^s5
zc2PREHLJ_I(bE;?!t}z-uVUHwVSTl#IWJB2oEmV%F8h8bTf3^S%ndIIHKDwI{pVPK
zusWOk)T7?v^IG)i)_Y`zy4RVRrH`P|vIVCSi;z%s7D&2~aulJjMCcSLYBy!DSX*jL
zR0SWZShi;v2z@9d@u;Tv3%5NzU5vt9^z~sHSy1^Zzi_!~WJgTnkRLLQ7mY`T+iLmK
zqTVBRZTj*%Lm=NQtI*4Z*nV{T3bZ)jWOeXZO`9Rf8g=F#=wpb3r#a?UkLe~Cnt;w|
zKz_m3*4E?9mFXhbs?H<`!YE@?5cAnqzBiG$Sl)
ze~LwzvHvQ-j*|l{eV-97)noe`_4e-Z`Fk^gRO`5-edEcAiC*K3aPEgYL%;ZA+D^Op
z&mUHJC#uqGH~$?X%>I;xEBy98*S?Kz>1n_Km(yxF>jUNt_vql>;NGMiQn8=G#4>Xx
zI@!Q#B6R-c%QNnvIRKooLO?1=s)!7;sXjn=ASqGyYcgYdtum1}dK?=i&cu*l&}T?)
z@s)k0LGx05Paw&!`+KXOb{(dfKP*Q2imK3QuP~_m;{|9A@9Zkq10-4iPD`Gry@6RI
z$Zg`)kTF*tFf9nC$uezy@
zM?ds&|Mc4Kt+Nh-b?3v^y%XL4eH-Z_zcoGaW$eXMEh1DVU+aG7E?%Byo?46`7{7t$
zxVKv&%&7xY0Kk$q^dOi6r9cZ~ex2a@NXFB#kalhweKh3CwX{gFwH&-Oh3vFA=Ov<^
z*1{FRh09Xj;VWWsrzb2b8&;Q7`SHXyzi%@@L|pGgP^4~UbXfv&NfbT%wQF}E2inZ-
zKF&`(`my6LXE9w|6}-;-cbuktc{RiY-ThQ9^*SJgz7geL+pmc^890t#-1t
zaIon;2I7$D;Og;QigYtFLk-%@-kt3G_vWsf#zikX7sieAhUCjC5guRljSt~-=E4Vs
zD`X47;*yxPav}_NWYX$yz6oaUjm0}J6FKO$$90&vxL{@?LrGj5TNmkYQDUF_!KE$e
zu#^srN8=;38+}apngsirz!EL`?BC0(bTW0jrMH&DF#P1Y8}I%;)xDhL!Tq6^y5CEu
zM1sD_iq_qO0pwqQ4044#kx)kPLm<0tj08fVfXjmrqs90Tg3}YYMS>QS-*KB8M
zu>yM+A;A!SA>Wbt4dvod#eq=|Pq#5UrUnNs=41)#n>`YlX6Bcv89rAgS{OYcjtdP<
zgbBCbBM6IxQb+=vl?s$>6k|-_Iv!f|M+siBb8zMH@igH4skQ%sNTJ5Ag;&Q%d+h5O
z>&it8%xQ*MUREUbaDGJ_QU7nFP7VK{W{V$)&zMJ)-fy5Utu*>+p>
zogqB%Dxv3lvQ!=Uy^uB;CAJgz{(V|nR;9-0&!4A$*gSG+2}pD}K3*oW6zt0+HHucf
z0pY&dmz}|+KC#O69i1zLbrXoI@5eK13Yj1pvIq$DiR>hKu4!zY_dk9ZVFu^23+Ljo
z&$5g;gVSfcc>pxc9-zzm%8>n3BWy7&-Pe+r=Q?e{H@z}@&
z^FfK|Xt3lx=jK00m^ayxPseM)l;eY^Pt%XdoBd?ZIF92Ywi3ZF+0KnFv394r*dQ8<
z^%A3rDseqm^qvIdQhp!K`!#!n
zNCPaLV|<*OCp9!QqJY+J3OTn|!gYMK{CzU$DG4($%*-07tMSBQmCO~w&AQRwzu*0P
zr@h)zRan4FEEscyJRc2%PVuG;yq_m-5@p#=z`zi0%6q5hSSLH3|a2ib>Gdc+4mJrVl$snMXfDn
zxK|_N%awKSB|WttfE?Hq1gzJns|^Kq>Uwzn)AQjQR(Wr;+7)?L(3uWK3d7co~CBi;Q8fcL=G)
zzU+f>{!NDCC9bv-jGKfcDJB~l>T$mJ^Y)!fXHrlg-YVsr|#3YH>18%=KwQu#|9;gM0pcWAMgE*`W_Uvo7PKkZ~
zfVz9As01$JJD-x1H4U_YPG_-Y-LTt7+}sfn5e7~%z~)qdw(Y+LHZWU|%wIK)G|6yO
zJR+?EK^_@z7bOUND4WhOZT8Ka@Hxi137E&paR&_)785bh)w>;WhmRt8!E%i&Mqhv9
z99SCUU?M6T8Tqs611lmEh#HW25RiemX8+f0Ga?{ci;FbcA`oa(OkvN!pQH#R
zNXH3$p?HK(IQj4qlziuaC02FueRMevpL>5d!y=;H$14#>cAKWh^H!;yv3zLVYKHMj%
zP)sAdK*G#tmdPXbG|hdhRj_!^g8xAoG73T**_)iTcKBgT1w4)F(#RID0%=tTix%+p
z1s&y1YrWwRXWx)G8+`LYXo~v7!p8s+=E7&-Qw*WR4GoBxiTyP%$ceFLX04x$Nv(iH
zX%2J3b1NQW4bul!oM`HR
z8W7Z-R~OhwMuOf{a$upvY8pkw`?ii+hu!L=6sJ5;M+@ecF9%Fr(ph)Z3!v8AOJ^qio;Kn!?cFN1Y0
z@Y(yA7(bA-aKRcrNNSGnv9~N)QDFO
zu5$^#_UA4=Ow5FdGW!4>FbYHIEI&m+MGlIKYyD1VP3;_T<|-}Uw9#}m?+oiW4)TnZ
zAht&fIVlxB1?obr)b=`H|=C6g&#wZY`I;0IF$T5J#$#BV%LDX&DX03?>Gc=2X_>;Nv3!Ul$h4LtWL3h4%rO@gz83j1w-y)OsWJ
zyYE1T{D!gSZ+cAb9hY8tT(iFA{H!|6$n_9E=slO9(+7e}L9c>D&Jm-)X_O7B=FqPI
z=8reXIDx=e7%C$HIX0+*n4H`fM4AA(_Yo0Sgmz~>b|Zwxel_8XfB<|%qGr{tq|nh>
zb&-KV5b{Tl9^D9f4SyyBaGANe34un-ZPBfCB?Ync`sQJ#Zk0bUY?{+G
z&(4TUp~&%zsDE*VvOMKH@08E$dO2t#$6@tzjbL8>I=EwJ*DJx>iGeX3`9fCM31lc*
zoB;J}rH)z(=6j7AVN%s~00=#+7zT)jjS%!1eB3gy%bi$5^e}<5dP;L6-$B(1S
z%dIaNn0230Gps7q8fY!tg}u2zC6uCp*eeZ|EmkLfq#8;2Q|OC_sK9|yX$-x654IZF
z>kD~(Z^3cU7zWzgg~xd%YI_e(1t6MTzbe{0{93~0(Hf}<{c=Z6f*?i~CZ;cNa*=1!
zDz?NTVb{WZ)e{3l&m}UlH*IYS>vLUrU%!5}--3BVS{JHwD41wfQ^hH3P#5#JHmKRAdX)71V9A#w%}?;Cspklo0)OOStSVA}2_WWrGp
z2SPCcHVD4-i3lbR&xHJVaG56qvyKy}4P~qFdoi1R{71yo}X+H{1Tm(Sfdy
zr=BxRE3Lq^<|xS2Uut|wE?v3u8cfkbFfljt^W*8ak^@^>kBp42vay{<1p?*jd;s=_
zV&{$2C96_jP_9$L(5?8WM~@$;`sV;-Nxgb%@keuW4e0Aejqqd_uHKDIG;((r0Z8*4
z^2t>J0a7p;iCbFU0P_tf=~A}ftygZ(=FY;e7>NRk!T?h9HjE1eMMuAdFW^Xk=Rb4)
zd<`UP43O;~;IH(87p|TJUt_@bwxbrj6%Y*|Ut@uC_!5TVzSPwvD_6N>3W~xmyHi~E
zD9+VU)M#MlU|U?Z(b1uTIV!(jzx2$8Xs%yB3-h6O!Gj*N`xFGW-V&Q+JLFpiu5g=N
zM22H7_mDRx?At8%^Qc>V3JkocUNZ8O*AUvx-YO4Kn1Xo&DP!JcbG`>3@%MEsE5Lg~
zuxDy4&8=FfhaW*((?DIlS~rclV_*SE9~cO>5@866!h-NV0dff#X#*8e}I-UA-X
z_J1G0C8LrkBH2ntMhlfyX{%&!GNLlFGUG1U6*5XPA|;AQ_R0viM9RucMrCD_^*=6s
zp5OoL_4*92=ecj!bzbN79>@DQjgA)A9-~`qoe8dzb(@}apG`u_YP(Y~X2(1Xs
zx`P#he|dg>z87!IyM=6)}c1=);b
zF`3+CYnkNa*R46pBdu!(Pe$>xb)JoXAFzcPm%fAWHf
z7?V;uFgO?lwwdMV1>IX3n$-|CzS>TDkOq0h7Tf`(yE+*J1Fy4PvCy#V}t$u
zJHZ)mSKUa$yd(%z&88bjgU>p*`%@BD7v26eNl1AYsjr;GM6KAje2bf(KiICH!pguY
z!$j1J>2!2-gkeCw=}`%=Z$PRq9JmJO36+gJ8L(MFz&p4&52LUQAeU{Kgd)H*HdA(M
z5F*^mNJ3yZha_a+*7Ye?E@-r~;0IVvkJH;ZJbNH8aDh(@7B$t$3IlxxbIRuFwf5*eue>C&A(!
z2(ibT#D3r^3-N-^%82Xdk|9{@1u>KH1++RZ8UQqbOIL1PzloVyG^64U>?Fe{5v
zkHAVIV-Rj6)iDL7-5cw6DZg0$FABpCzG8yfQ`huoXMPzCH({oNXO
zHV$vX@8CuxO~^sH_aTJu7!w=YECA&QL!vs~Y37}aMXHJKYS)|j2
zq<-4OoI%Hw=e&v6qLwF3Jm~Nj>govBE2k;-x`Xw#wY7gA+e0-zWW*&OhZZBkdZAh%
zSldXA(a$0kpQb42POrPtS!*omj}#GvN+SplIsgGLi1<)%ltt|4-Szjl2!J$
zRxY(`7vut$b-keCLa%ehPz$@&E@<)?7A16nhI~eM`MQ%%$7jUY$$v36?z3f9aW)8Y
zO_R4c>#h|z4GFeua(G;fy{1RSs}4%)rPw|6C^0a#w;_l_k
z&wq9n&TR8f)41dyMVdnQcX(bi&G@bGA9<2}a0=rixU%p5|TuUWUbo@>R7-$R3nS
zzed6<;d<_om#@-;af$1VjEve2foARaGwR4u`V7o*o7@WlWTZ@^8c~m!hjp74{Czq{
z-QkPS_3MPFRtyini78t|%*1Z5-2C06r%2UJ|9dwsaF8}1^#?h;#;_Z}cIQpQLYIr|
z4755u{YV|nEnFK(c`1vDT5%*^7&Gyqfv}h@l3$)PZOV45|D3@_f+q?SaV;sIE^_ya
zYK5=7w3<+&C9{)|Qs-(|w-Z#UX;uxTdF
zYdNoT1j^=kj;p=s68tw*T*=1J@$2u7H@H3h5~3r%eWh
z6u*P%U>!}%gRSN`Czs_b{P|jzJ!(0AsT&vDgzG2#QfqtfuO;h8J&ep+eC4zpP8E-2
zMltm24cIk2ItitBh&4Py*xQEg(~|WymOrOnkomUZ96cY(c99KkR4N;DUd%4+*hoU*
zmP+v9r1T6;GGEj7_3KT+`FMp?gi4K+1amzSFxyaUEhqI%`V
ze6I>3p6ehH`(+zsTSZwFqoE@d3FTvQQ0})af;})|W4bp=5cU_sJy`l|~m`aLU801t%w@k@_o;CFn|
z(k+LK3T{{zLMU~9m=ZT;wAF%=H-tb`}%SVpu;DJA_+B$RNR9Hw^3D4
zA^|d~P0OQ7VJz|6_iKoPXrsjXcJU&&xt%*6Lw>^IA_9K(0BXWu3~r*^U*Q2tDL_4H
zQj~-6?{@Uqv2pCuLs>ROZqp5LnQ6uo18&e7ijrHxlyoDm6-6aH}3axQd1`!`lVf9F5SoY?B#7ERrnCRM*;R~61t%(H1)uhp64w?
ziuML|c$55@TKoyKNwBhfh&nHbKS{en328dK$&(#vMC$R5!1-E5G(Y
zMKd=ySN)0bx*_^$QYli)bx1JVq#zD_tTy%;YF$VTFreRJUKst5yY&oC=0$0117We3pKcV~@L|uO$)zrbI;xv(!*1f3AUQh%9
zs=kKUM`T<5r=rv>H`eIfMU?Lz$O5_qc1o0yIXAxI7
zoJ`|MijU{TsH-*-Jze`EBijqAm*{kDoc?5k0BA-psq2dvHt+M61l
zH>_4tve~(B-xlB~9}pgq_GsGFAMo8IU^SrowZ1=Lr!EHHo`k$&`?(x!t-QJnnFL_o
z?69Pno`b~Y4QR_1m6d$pi#u26pe?-g{x*#Zka6{NozXeb*MTn+p=i*VNz(Ivsy&wY
zibdF+?85Km<>mPME8(Y7hSDq$&DmO_HMfd^@P8}e+r_yVN%b4i2N5w*M}XYbu#G}{
z8w8faH~kV~F~|ZEcrtq!C7IM1Z*$`CMQ8Y$JT&B?EV;Y7^6l91OnYHHqyJ)V^?zId
z#>ksmNky$$9Ipw{=ajq9b{)#XA`|n)N^!4THJy%#)wM+^`vBgCSwZTzk+tLg*s>M!z}qXM~@iA+&D%pNR(P-y~WTPwYcx9N_*@g53U$=^G^f_RaWEnf==#}w~f+d
z0XC|S*Yb?2Zgn;{{Czc8cEUfU>)Es6PUbCJ)Q0zlA^~Q*xaQ4VY^U`;!pc-bp_J{b
zm;G_0-^h9Q$`)n~T0`YE{2~FcUePbOaV~qej7;m(qXk`WO{c8dvo~T3XgG;I)QrTU
z`T~DVL^84LE|w$SV?b1w032(hPG8eSMUJz;D7&qwoa%kOhY<)wc%7@
z);YhlxJX!IZ;ly*Quq?|Y%WNW=Cxk){cKI09u!J&s^~HE)7z@;zCuxFHVFmEtB{fn
z1TqcS*36kvcXat{@8F;x-c(b#U}Ro9*H)TI#O_6)cDk=(wcWBtnXJ*(mLIEXO(=}c
zKt&C`QHp13(AoJSYk)^Kk1-o&8IlX1KlN{8wf1k)Y7u=>(@nwNdIx;^2c}~0KG}BU
zG6TEFEYv%=7(LN;1h1YQTUS>XdT(KFxbI-Nv~%aq
zm(5tMD63_J*8Cry?vH!Bgq_EGkGFXo-$EG
zHBF+Nb#H5lWkY$Lh`J9Q6g#O0r2uT0dHz6>yGF?4ay@*muNssP(E0Dq??@$w|IE%-
zp*d&(0!u0dQt$?liVoKT+TbJT*y;v(BP`}AT>n@K!J`|D@Iae6@u$
zhln0w9j)_SO_e+61t`&W`E=2mmHP9_@qtPQV{>cFG7r+a16DQf7DbAOVcC!e#-qap
z-5B#{Az`a4yw#+c!?l<<^QuO+Ksk5EY8hj|N{#_yOavr9efs6wSVq6#qZggZyP&+vpfI3*_>t&1X1D;@-p8U
zwIfHU@R<6iq!7kK#D;(?`Yc(?K%kg%)aIJysl?BgMcdVfbk(Go!YezyaPdlrZD&PF
z&2Qcm+8%a?Nyd5f$(J|>nD(UjrXeYOlV#K^U;1@XekH!}|LmG4!dt>Mh-K)X1)mwV
zTGT?F-@Em7um}YFCKe5e&Q-_zNC7psMMl+&?fq7vx>6DobHphiQR=mvBUjU%ataJd
zKT%g^x?1?rUeC@j{7t+o7kz%fYqUQC#Lzvvo*DP%cBxUa%8kE#(`&L_HrWrZ}M7m7^`?7>@+GBz(a
z+hOW_*gVMwvzS~ImJpon9CEeT&r$nqjpcq0JEC=sMEtA$)ny3d97{}Ri0Ke|)~9&!
zQwmPi{{4%gn%aohcyHzT7d6OYwrw89#=I~+*nN%RqWnTnv~987Z#fJJ{Q*%G5cqXM
zpx&}jy_?Tgt%0O|;(IcqV?k?q1Lqmk<2{(wzZJ&};sgeWXQvEND(8wyNJuP#ZKZNE
z4&IZ`Qv7ny&ZiU-h4KI5ldi!BuA-ou_(Y01JC(zRoZPL&p
zk+g~K;Eyc|sb`%x^Cq+p-55^hptXT-*-o3E`k7|yP-mFeX00)z^`;^i5>{=fYPdW-
zKcN{=KN1OuJZFA1T|1-fdtfNj8n~W55JOoXJHQneJBnyQ4A0eTN949|5GoAIKyq)M
z$Bd}r!v}sTsjZlIH1~rnaztBOJIcfJmkNjCcdih^qgidiWnm=8Zh6slhoB(;o;}Ry
zpS12386s0+@CmO>^_CLN`m=f^T9m^lfCjkgln2Je@gVXD>RABc&-Gl&FCW_n$TR0K
zkwN#=^7I>|H}nF2mX(FwMdm!wx;EN@(=6q;;l>?g(a~3k0PJwDv(iyj-AvR9IlGPW
zj?KsUCnuPa{Qb{7LoZ06s#a|ecAgD^N3lZz3Zq5Xz(t6qLY+#ioNh}?3mdla12Kmw
zMrj0@m4%X(Fgb>|GOPg*_fjH(`)W!pM%S3BzD}%+s505lG2VDqq9zjV)BGS1PDaVy
zm%$Dv6dP@sWHN~X9#RfVu%D^+?Zi;(1IpZi26uQQtwW{WJdb%Jju6PJb6$ZpEn{~~
z9)w#>oe7k{D1pK5PP7DxXo*_kyJO|5uFk^kKD@_o=4s(uTWsOXr@SOW?7s8wlkCQc
zmoH*f<${JG$G_p?@({wBODK6V&E2Z7;+{?H(hQP?x6ysN`PIOX8o41KK!49f2B)RpMCAp~&~JnNKMDqfHio*eoB
zp>*2}Mj4s&*uj3Y{S?!hj~`k1`1n$+!FLk#MAhP_fT?b*HjuAm!=n7FM#LPo&f3W3
zJI#w?9?St@CUNdvooIStLY@s^dh;Q)oAuv+s6MZQ3(SF*c1$)ee}mPOhE?mE_XF+(
zRPW;MUYnhYZv(2^)G4T-VC3TJT9Z8wQFd+LU%38c|IUS>H_o?0-`ERnnF+#(3C@Dx
z=jB!07lV4F_s?LS%Xk82q|MeqkIk-NtS*E~o3qQvu8|=9XmFLo&IE0_Z`}wQp`8#I
z3AKv@)g8za9~v@+k3ou)!dwq^!LJ#3`6OpYA96|={fh^o`g#Q*%6dRh1$Aitl(0n(
zl!I;&6&w4%^UQS2GgFhr
z+Db1weeWUXns6haNS0I{#AbelOwH9$Wh#2`}OrP9*fSrLDp5~8b
zxQv@LNkMzgL|7JRL?jT{Ib5*M!K$)0pO^${j5$M{`QrXty!-hPk;KrakVlKB7Px>C
zOcwSlI0KlgU^DXI^IRB_4h6g7i}xMdWNC>4*q{JjMhX>6_RR#FGqY
zQ0sbxNaKLe+Lo*+`_na2Bj@mjRU-QGB5x@{4H0#q6eDy$!J_@4=TXu*{@sTIg!G)f
zh`#R2f`gQpgM-6y45Ic^zW~n3g2E-!DVRq=ybFGx8jrCHgHWJG3{Bgih}0>OUno95{929omBiwr!Ab
zM<~CiNi-_}?eoM*eJTHN)JP2j;HW1_mT;rUA0)hSvI>g!tquk{4w81vp@
z*+>)SNMXa+2+ZEJ8$~pa6D`S`_Wo|wD=J$TN6w*s%9gs1x=P0(2HM9%j(1URKX&dX
z%=msyi3FP-gq0TY6az$?0Z^HM*VK#bex7R*DfdoENr~E6Q`Qx=f@4h)CrOa6p9OCx
z5JQN=PO`+FJl5^RcZZn4T&_QKfu2OP#1&OlYzzB<@Li?l9|Hz!GLoKqCF;qO-6-(-
zrp6ck{tW~Mms>e~X*}3#@mS8S^5(v=F$oyf{P<&s620oO?zIEYV4|uFy2+Q9q>$8q
zOEtNJ7W$qPQ>F{Mw^}-c{7zZjcnh^zB#y)P8}$S#m^H_n-dO
z)percwf8!)t9ZAPQa(1%ALPVdoJ~@Sr#93e_4Mge?YEaV!kb`y;^*c1i>FS}qptP=
zN2<-B1W0%@odHfHU?=?n`7)lMMah3ag=qYz=Y2{)QDl8tz-mIJB8N~GNKh~>(Ybz!
zJGgi6W*jF|f!W8NgCESHg0W$Z1ofo$X>JlgrD?
zz$HW>8nTiW!^+*kIMNig8{^Fus(2LKt%a`p02_DCFKM+S4w82>lIvU<%BMs}XB~59c@H?0ZwUKCgQa*p16%hvB=1TLf+c;zyR4h
zhT_)(zC8|HmVPSv_*zsUmHq>R>Dw)G%uDm`M9GYyJ3Yp%??J63X8d5nv4I+A`8k0G
zA`M^o?H=l2Pjdu}ZPl623h2EIK|=x2biZW-2`S}-vN8i95y>;SaYKsWmI2Y_1vL^`
zL=m~*RYcx|3U`%>R5X!uhP#Tk1AVImI{>-iHoCe*fM~FeJ_*>0nVb7Q`Y)i!iRz+x
z7_66~3bC8$JH2D)&ZqJ5Bw(uqK29JnNC&0e-QCv3UqEa;0$lC&BnDo1kKTSlZ@Qt|
zb3u&AKj0=2zy*rXouGnnqw;$SG`cdabzoqi7gG)l&=fCU7DRAtKu3$yD6>9_)Q%d1
z|CD)R4N!0WJbM~od*UoC4D_eB!pk6bw|y@J-E+B;$gDZB;>L3~p-h&eR_MsB}$3a-bm
z!f5>cwU>8wE=q*GJldCcW;5wb#3%iH$AFiom6VirPGJ809iFDm^LROXaJq_V8Ib3w
z#+3TZ`)8FZCKM)>NP$wu{1dwk&!Q_ae4VWmEu!ZUA1T-|9d6O>gs
zG#stDb^?fQ$@6ZQc89;WM~caN2Z
z)^RK9hU!0kltwMXYj6pWqp}9^eEIq9-8rUPOpk9NHoAJ-WCgX5)q+Mg-Pw#?Org;Z
zm+57mzO^cH?8j^|l{@_0zboxK|MBuZ5H-{E3zIBOh-V?<9-GTO)9w>cUAHYp-^fxq
znyTa7D`G-KOu3YG{`a@^V4j$J{rhdli&A8+<7!f?e>7mSqCm)MvgzsuLHd{01pN$ZBdY+fqr)>f^fO+IvCBYUo`H
zx7GWeO`+Qz%;;s@!}5NPDI{&@#;G?*@n-9{{CiyOAg1u4(Ri-En4XYtedM#5xbBBT
z;)rC~vg2_fIrE1&==u4j&E8Btrt&$$Uu(9eM`Nm`;HTHG?luyw&J89&#~>=-xpPM{
z>EEp->+jsPD-r4Z)P)NV#kz{n7`r<0Yi$YpSV3(&zM4rb_xxtgEe1
z3|UZ(Wup#zHnHc>-0_R@eP=mnBA1s~6n)LhVKAutL?-m5^1nszx51#7;+v*Sn*3u?
z&(ps@-I8l{^%DjIwh39aH(j07n3=oO;8%U@5vO()?|kYotC`ATo%mxnZK?ag!6$*y
zUWa4yo_GIUEHT}}c@7uFV>k4ud|si@ic^yhlSbF%#;vYpk83X*Y34^g-^C=aq!qo}
zM@iI2Mn`W}E;FHvua~>eK=l@TJ6c|llIsDjnMbA
z`So^SSV=(_s9e_jl(&(5?&(Ax$}Ei&xa>luy(a#^>dredSrJC!rVH6H-(btRT=&+F
zUdG;%UPf{0y2O<%t$8o`^;Lmcj5IjRB8VuTfPpF7{ckr!0q-(__6}>qK(dX0bFo%6
z_}yZi*6vzS-G{4plI+^6e!BQY%V;Wu0{okdB}mU^7kPdKlkGQ?#E*Qg+zo&WPLGOL
zuZpOFxJ~>}^zjQV13MW5%l@-pf_}e_Cu(ehVNRv-h{ml7)~;5+q)Jmr%Le@
z7x{Gvo!3XkkGAHkuYqxFF3s=3rJpojyV>XXCYVm_YHnJ;NPk#?DUNFHczzg00YuSJ
zcsB5V7pjB{g^x@y{(-R3Uz*ZE6*`w^^)f=GarN>lw0GoJ1GI=r?A$i1?w)ijy36T2
z{F_X#o5)eGr#=x`vm>f7P|&t!C9-By=~2>1(&4*z?^Xa`CIqFiH2;=ZgcbA5N*s!s
zHEO@iTB<~H9=9;AYAn?txh6&?FK`F>-S%jeF1lo$Qkcg@>jb$bbr3r?m3=gm{$UsMtK-(=_GqdnA!@lARoEbF}`AC6m;M`Z0e
zo0@UdhoSkU`qTd8NKcj6cv+xuu$~nWH1W3!!s*wo%v3
zS>~*tms<19bAkEd|88tj2{Y>~zumYuA1YpvnbSE3Se{&bA^EE5xS{T;w2eK-Nz_(F
zcEA*wxw!7h-~1a78uFj<@$WjQPVtk%i+Gj+&LuIy1p^!ellY=@QdE!VdbV0AwD1}Y
zrcaDs{`u7@=Jk)})tpQbzEG%hd-ZNPakl~I^Se~8yUE>}T3=1Ej;40mTuOa>GJkP#
zu>rb!c-n3TV_h+5s~BeF)DU>+?%EprZ+SaT+SrK0q2SdQj8a0NfCC1EPW}g=%3Sx^
zw4%;O7w2TXy{KfUr(-BH9Qjsq-J~~59qqke?c!F`EJfzRVm8a9662EQ9!ZkYPDauZ
zRiWmgb!Lr1?Cw`s?w<=>+P_wEtD+I9SXSfTyF~mqiEKsEBkiUR@mHXhq8Q3jN`t#5
zJ;SAsU;imxUqxbyn0vOpr}g@E-CI2UT&j)6A&v^mLWOEqN#vQuRsGuP7QeW22WfFV
zFQZS9XD08a=2PDCGrnGLSr@-I*s=~T@K@fLMl}~+q7<^&+WPIp!6+6WNkjj#HSu{3
z!5Y$imUDho9i?$2Nw@^^NKbHch%HUSro}hsBCE??Bg*8%71my6p}qc5s+lctMeIG#
zOJ5<6tttxF>1F=KGUEAZPmnr7m9LP*sdD|xE_HrNlq5Z4dvcMMoaMIiXxt*5r+;Ei
zvY&_PAHU=^<06$ePkXUU-BzS11~*en>XBS%OTFJrl9fRvB|1P%?6D6Oal|HgaLi0h
z#xZkb<&=-uBB4=HS7Ori^KPTRBTjC>v~M?23;}p_FW5ufdcTZrcsVV%t20T`_*WUn
zE&Mv;OTDRe1Vlab?Ui)Ho0&BdJT8v$6+qYc0R#)qo7HJMjbsO(&p++ARXaI(T!Kg7
z_q!Bd-Fu4{_jT~RD@)~nlP2Mj;`a26#uog~`#$SFl!VE*g{t=n%$}FGA7$ai|Lje(
z_|aT>_*C8_A3e6r5XVDY^LYb-Dq1^Q+__5=AI#5_H23N_uG?N4F5RHF{E+tQM~*PV
z_>+fhCFamN-g(L}ljOl2GUY1YXt5*|N2ktKx}UVE-7O3q*wjcn8!(WQsQ23x&#JFM
z-v{orBtg^z$GQi-_>5GWzPd}nh@g*q?b!Lcgrfr_JyGY;j%^_3fbqTwr!kgwZ{p`y
z?GGZDbST`RFz|ym_glN!7P17@v)t33mh*blr5i|p+rK3nre3vdOp@ZbksQ#_e1zc-
z6LYA>k=Sio=acxxPp$tnRub^+%dc2Xt!ERHvgE3rlao{)Rc0BR8H=AEaC`8&OF{P*
zm!6#G$2VMy-{0x&DY5mSRoQ7Kl(@7lo^h=(qEczFH2DRokT(8t^Ld;rLApO;TIo`1cEik;SY4Vm$W_F#N@3IF}P4kB5tny=C=)NqDcE`{~RFC=pKfnXGOTM+8?-_zqjn;3Q$TT
z(3MN&w757r6e$fv40j(fu)Fl$@b7{uPiYlN_u6ZAR#qPn-S`9reM@-Iuk?EC!9)>x
z=1r5{3=27jq%Y5<>$BJ3@4fwup?dW=3}DC;dHZ}3(lZIf}^w%^aUtv*ai5zei$b-n_Xi49P@1
z?EltMVnPg``r;iq3K#-CKGT&Bl9m(nGhFKr{hU1`y7p_zzfXj^iqa|E^yIL((REz|
zSKJ~>iVgkf6LfRs7_zvHlr=7(+JeB+Vr^Cot+O^qHIVUEWWA5z_nwY`5|pv+YjTU(#g(4D!x
zSeDwMSg02JI;VvsyDKVgV-Jt=Rs**qyn116?@K7tt|Jh3CNLj*1_A|K*VsN8ZTZe-#?fb_6%=|zs17JC=
zn(<$Ap5JUwl|&kRb2>i3kWTGlxL09A=^DWxaR^9CXC$xdmY20{;2w8NE$o)KvUFR7
z`PjCR3%aMLY2Gjg#wwtxEJHJP&L7H
zC!9mm;0qA|3c^!N4`tUu#4ANeZDCc?gEoosaOPTH(5HcbMvrm%Ux7}K*BXCiyGZ|P
zq&5t>)rRlKWhu=P4;JZ97Ll3;p|VxcB*OzS!;U{pEH4!vvfvRxhw1F^1
z`P7hfpJ_D@X8^9!7&zeK60pvkdNyRsbk?Rq8TQ<iJ>S)u%8K3X}5!
z)FS}jg#;Y2ty}*79`zsUsqF<>lqT=+d6O7HDi@CoXepgs5qmHPm&W*ZfUeA5>?>j4t-+_HmT{M4u%{&j0
zFFe}Kc=Lr%%nH}KnLpq7^!|l;({(x0mdb*pTHbZb81WJZq?)6GyuD%Fy7%A^ZLqa%
z>#ROP+RCh#~s())NxmiB8A&t&fYYs@b<|=ktIW=Z@i?LvGf>
zZuLui{Fx#(H_x`}SG)g|!7Zy`uy&%!CMZ
zX??t40cgN?vyCWMY#BZ8+EUw)jwEPY6?YrthYO!muerNjni1pkaPc$LOfx;!guPgLyUahCPuKrj;YD|Z&T;8Ut0zF&s
z%#vQ!^~-}Bi}Dk){HWdcyF8y&kzsgJxj>RP%WT~lIL{HfuUnLnN^2dfGrf%Wr~PWW
zr#B)=BgaKilp-34TnC}jGEim
zW2&NFEgv}$q_J4kFczVy48w^DmnFk1J|fG8*}ntN42$);ZmEB2Dn|}4&p-G$R!NCb
z^gO9Ley+Q5;e{CaWRfVx+R@;SUKZJ+aTK#N7n}zk5yV4vJ-yGle8PS6%f%(444Chb
zL@y%=_57djeksvLJYp2;i2mR1K@-ZU+&`lE<26|YcYpy@NY>S8F!fNw%}@E#_y5s)
zBD-&{e6lK@kvSr!MmzRxE>B!j3*FuDRP7^tv4Px<(lp+;6>Yzg#Z
zFBLHDqfs(;{NLWGZTvoyze8dF)ri!g6KmsdDpL#cqDNy#sS$6ZL^d?IcreuBdx#pM
zPDL9jq^rSJj8q-lxC|M|d$YZJ@R#<4)!ZS@hRt)a{A}ynFK*E^Rt2HS+yX!x*1bkg
zaIpQCd6RSt0ptA{BX@voGbYxM4eL+*><@6WzR?6vSDC_X%%R)u1WC^dC(9me)1sEt&+kRG{3uh&Y)G1q
z)@RdS29dwr#0ikf#N%k#KXItjBLl_m>aTD1Pvm_Uva)!PeK>eBHq(Mlo#u}unXj=s7`#TJz+Q(lVpw%C7{w&}
z*+(`%mi~Lq9(#l;*u>nL&2JOe-IQQqu-0cz1j=9)It_Uy|Dytp>}!o&1e|t#eB(GdT|5*u4eV
zqzKJD*$WX8*%zh#iPl|)W8UBbOSW_d8_=$c>ke)vWjeJ~Qa38gh=^bg$BY?>Dj*IO
z`6*pZ|FMp|eJ}|4bO)ocGfy$F`w4ZeO|f>m#(}R4vld5*JJv$KJFTmsHW09hTtt>-
zb;mq6kJ-YFB<#Mz{`BY47s%%^fBTg=y&Wh{-CE+SDmA-Wo}IN8#3g8OaLgNCSZtfA
zC=Ir_1a3z4J()=c-d`(`r?--RcNHC03-@MJUf(=03{?bsZ9ybnkd!Xn7a28D%r-*)hmyW
zmEbbatCnh#PT9^$*LJdHYU*PXR5Rl_U6O~sW9?oW^VgLea+`B@!cBo9P>tMl7D^#tiI6#pBTi&>&)(v
zuAVVs*tpUB>gG@b{3#Fs{@Rm(%dH;ch*I*AXFB$XO
z8&{Sz0iUhji)tf<`eusTp8xJ^jd6iNV8@n!*QfXHtaqFyw2cpf%F+StB3j{5SkDf7BYvxkxm>(kY{b{!Vjv`c#D
zmEx_92^?jkuMJZMr9ImZm~~NvJri#QGkYHC`ja|Jz4BN-d7CCHtw6iO%c(1UUdwN|
z=JqIZS;bEPMZ?qnyn(dMYO$UhA_uGgmZ^=}|4i=yk521e=_A?0?}f#-k?c!-1cY0Q
z|GXNwwoN*7ytHB7fCzp;SpLzFNl^@n_X2Uc`3F~TYIS}Nv
z*L8c8q>_!`^22LM+38>O!sHH#jXoRruB5-YxoCWQvHN#PRdMt24M&rX3aq>}dyxI@
zdG*q`K8Di*7S|qfUc=az_z{W7dN!>3>$tYa@Sy(L6rgB`of@HlFvOgo3FBuAqcm{d~Nmk6nUo8msJI2!bC_nYquI{+WAU}$Pu72Tqs&}cSIgfLk2VFu&lFa9Uih8ESQe1Gqb+l4`@uqTb<-Rt?`%2VACw)nHOdhR9Rz-j
z{23y-VKMvr`WrWw4ri*`JGC5M?oO3_#E|Q3e6wT4Y>WDa87Iq?ptuf+_OZ#8UjgKA
z8ph&GGFc+bbXtPnJh~5^TO(Q1Ep&$U$Pq>6^Gd%A)|@&mwC+yG(W4wMceGi}FNK=D-S;-y>L}&%?4)PW
zOa#tu{y6OE9AMc
zME(QOFH%qbxwXE#@H%5jTA|^k&ClQMKuNRCEtlc573*fSh=G^N$4Ka}#T22dExCXZ$;H`;|#Y
zQHV}bOO(rjvn5u8&lFX}X_hgRlxDsqcl+2EVYrLxW`u6YnYaUH)
zHkplnxU=YBVS8>*#*4-8-JiL-to5I&P@XpZ46zrWdpNuE`$X}M_dAOdjVj{pCQn!%
zE^Qw+9dldSJF%BCc=yVebm4D|e|!t#wmj(=3E>C}su0!jUp%nws>Zzbm8%h3cS%>6
z47&$6*;t(k+qXB0VT5j;*W_fOXXkXeu;sMNmbWtwlm0GqInUoXsL=#~{phaY3X>nErL*|nay;*-
z@8zeXmRo<{54j~j_iIE{h@v*sGUl$Z*}>WOeNEk`SEdhldU3dZGwr2_TQzqyDCg_>7on|Gj%2U5$gWsNdF(t>
z+P8A!;SR3~UAeBc<=sDdh1BG}_AG81*KlLJpLRxe!TiGMi7J)rUbYO6nohX9TC;I;
znpu<1tH(b1<()Ymp3!Cc2YUJUigbz(dh$Kse>+_}bC1QcGxfChVqvm|uKs*Kx0Jl|
zR_9OajNDb?p1W8*Q&>s%r#Ie&KowL*y!*tD)X#~o9hMBfn$+a?67Rf=!pdHa9E<&P
zB-QZT^}*l?vr?a!@t@L(2m01erp->re%l)vWQ
zFROx7qhimW^0)uGXyn%2OPr;teZN!`@2K*1Xkl6MtWR85v5Dsrok(AYgp!Re-~P%*
z?z44AJNFE9p1)i&sAM+%2oXJ#mw2qiPg}g`O2IDenT>Ydwt6`tBLY(r#kbe0TuS4Z
zHMQ?y>oxq9Eq;!xVJZFAWv;5Em((3Mxk6)9W2c0x#nZmHyyNK=Jw>W~a>`pb$)~Yv
zs+5Cu4YqEgQ!&l!?RV0#@$t)XVI!FxbcsY4fgdwdZg~~H
zeczwPzZz;Ki;TTE=Os}cQ5IdWMLbG+zh1Q>PpigL>APWmMaFq)TNI0}JA1aNbm_em
zGP>8Q(0H!7{=Bf_#Jh?42u}TQVpFHiGqfG7e(I>AnE3sjA%6-JQBpMtxYDdVn&0$z
zb#qj&-+BCW*#4%d=ntPO8Exe6-_E>SFn`&4=+~Hfl4-!;PSw=?PTV;&-Z$6Qwrq+T
zJuNV5{zdBJhV_Ss^61p2RlM;&_C&W0hQ#SqD2SwND)cmul3B~i97?kX?fUXtL1HWN
zXKwE8QrPRm@aY~`$~%d{4>Q$gL>eU61|GHSt9sv*cBU#+r!=P#f5gshl51(=wb~US
z@kaKM8b^b0()w+E*Onr@r7JZB6MO_^<~Z&Ta9KRRk1
zZ_=^yp^ZP9zR{f1JtQEIwqu>De*F89FTd!w2@P)xkC9J*_3qZcZUo=$Ff+*_VwoFD}TkbswI2tRI+&Z
zN9r6Gln<@E`QJNKd&tX_{(rA^P)wra=|qj8l%n*G{ofZQxa&A{oQ5kVNuBaxj2gGD
z%kz)_u^kaJo6C6&^8TMz|H}_1Ub)jOmhKH7M2shX%uHpK*2Vj6QdddM(&!4-e0|*Wx%EymDZa>sUB;VRq4#_DHy`(7
zX<2FMT{rHpcX@Y%Is7H})j7Z8N+e>h>aYl+r6qo>?&i`lkz`MKX!kEw#MIw8A1*!d
zC#Ptw%5a+_OMfaANxA~M(wTP@c9~ou8eBF$NKe{=#oy{f+$+Dx{PX8uAY!5d^`R0B
zl8~sVns_E=9bqk7I-y}7Bf9iOiI+7mT{9zi`(bcGop$7~Z~ai(=(*-Q`QfI_j{mCWh(LNvhuoXUA}-k#DS*
zE?r_cFIHP$Uj;B8|I#gJb~}o1ZujviJILG7+4&jtN+49Ld=1!MnC%_P53q&Cy1Kr;
zMPd`kNH|`%hwhI5tH;WM$0k2<`v(bKm@#T|-gL=p(*ylh>Ei~^j!b2!AJNGPsKcw?
zs>E=ZVf;Ad9_P};^s(8{nB7(8O`w>g5rcxhIW=EGmZ&gbNKM#
zRR1bM^Y=j2lWqYlaw42Y|EbvUzLjO%&od8^X;PsgacrvjvbC4vh*fSWRx;OMnj2^tdLy@QO5hZ{GaD}-{a7q_izllRIhD#
z5xiEJpT#3SATd6ae?ji~XiTBa=KP=RLouiB$t2BNZZY`xcl`dtKam^*c~4d%+4hyA
zqTL@9X@V5Pka_P5vkW~_qz`ozivt|6>iM_n3uygO1G|0)>=B9F!mW@y0UVW69*~3A
z73*|^*Cuw-X?`+Q-tndTO__DLM#+Nm$EwFe$=r^tTrwqp1+}xw#V)FD>K~eMIuajv
z`d{TEkGW?&Lrihs-0eM!UG6f^#6+7`C<*hOAqVT)*DFW;MQ^p#N}zDU?g=MkRkpOS
zLyLwRkBJ$|AVO>doL3muu7xzhO(+0;Cj!VoGSPN$aB!S~rV?18vJg_@vf@jjt9xI|
zdh8w3o5~jHRRiJwB3Yj+afMMGHIY>njM*k8b-=INJ8{N;Whe4XK&68p4iq}Gn4O)$
zm;_q6QZIs2Q~3!^1C$X9n2nP*>LePHDrv`uN8v~O9bPH0d;Uc6ONys|#CGkIeClVNerx`^PK*q?lS;tD&bE6
zS1V=f@!T{7pi>KQT!vbo^4_^~XL^)~uYdz|@0IwGz0l6SuIyj;!+A50HH4FG
zC90yrR1`N#mWv^0x0Ag(zWSf$84ih$t0m%`yQjNe7VNOPto&F^{B5I7FEVuVzlJa4
zeDK_+BW=vMy>lqC3a=H7mG7}-`g2!k`99F#TU$}B{!LxK*z2~1{hXCxgOyMmuLiTS
z#e~jX#WBiudgEKqB-JAQ-+hO~r@U?Jq-W<{lqOYGsxm55m<ttP{&;gaJd&Uyj9A=;S1;+3j&z=uTQasw*RDGk8FNHgG=a=tF
zG%Q>k85CM7A+^KDlg5&%lEfA>L>BMcG`^)QnfTRqtTtz>(%#<7v~`VhYqBa+1+TKO
zYIiPa&9R0@qx+WdP*hYj`|Cdw$aHL{astd6hk}j&NHVwf&sj#=7M?_N!#UbgiFF6{
z2FDXM6jRTzmaWQNFCcxh_4ZN5{o%S9l-bmu>mUBB><`s9C$E@WK3MH0xalC>aRXiE
z9~gKOK^e?$UH?
z)`Drm{rk|!K0WSQeEt(PgyJ-`v?Sev8h|z+wDu~|41PFWlG>-41{}LTt_n
z<%!{TO4Bu{sIC1W0J$Ka`={pre8wa11vy|232r3ms=n*Q3D!ZgRv-qLkfDljdhJslAU!WBb^&9&~w~y*h>(K1@kc{D^RNGR(
zvCxg|fnjt#UX(XqD)S_$#;ofCw7%YYafD)brsN3?+2GCQST~lK-OY^^Gy)oAXmJWB
zM9>xmZis!nA)lGrH?<5_cul_bTUu*E0Ykq&bNVGb6n1i%jf2bq$yW3GFPZD52Nyi+
zr=VcY%EqQQ-qsi|%|dvV;&8&O%pTVhW``>5ZNyxts~P4mVdD9>`{vD?lc3{Rf)mUx
zs0|zQokIgovtuVgS6GXr5i%i_JoYbdK<;JnFSPX-)+>{z4tGHDr;r&3JSJ?@AhaJq
zQEC;3?g}?RPZV#=e0wqFe*-E{5VP&`i|rD$w6x~n&cuQo{HZk$;3BY>>-tAwIZ|Ri
z4=5SoQY^C?ZR2-QMrq%E=3h*0x=~PYdOYn%7a^{{q5eS~tGCi9M;a??s}1MXF*rB?
zEo~D<=IuaA5>dq`k{75c@Vf);l#>v%AOOPsO2hsleF4Nv4CbHPb(fvW{RWK(`N`@H
z&a<-Bpl=FTgL>xraIPGd6GT&ciNsF*7e{dnfg9%wVaTfUQ|h9Ef^?0F|3tTVEjS)t
zpC!Q^Ob2zx*T7G#s^>l7%*4xHqi4!VAs7WrZ=pit>XBac7#f<6D#U@9k*2$e&LS?`{e
z#4|9n8mi8!Hf(kbC|UpR2n&!^qQ<1$5xtM2vZ1+T=3;
zxbVU+ujRe}<`*%#6A~6~5`SwYp_Qok>ceTjWIS3)+b>!=26}qn=?j`KEzJ8Qf3s25
z)n$DX%1ad1RouXxB=j>x05%Q|J~=raWR6BdhG0P)~(9~lfD0puU{7Z}%GNku)}6B50N~vM*+yo
z0A&dUBHs^bO|#1{v9X3*!Ce0nDtU|dWYIfUZ8vQ1i_4U131JTF6Joy-UwQezyTV%0
zV*G8nrO7fZ`aj_@ili0*!}#QbOGlWY%Ag%UC_Gu*P&k^<41hrzf?(YIYb$S#qw;)JI{pR(FZt!18
z655uiN|-(?c(w7T|2#yU{zR_!8KunHKj2+a#*LrxBUCG4H0j_$di7Ns_g~%)0qIl2
zSAjICgi4_7Ejdp?^q;-rXw@;OD1-QXyLuctELUW`1oLiBDqL#hVf}
zK_CQ3L?o<493Ybj%;2n+RREBk++ve8a~=MjJ(0&0SyyTmrG*KaO`A4pUEBbw6jNU3^a`OHon{BwKg1YIZQ-4qF_7MZC16B-Zn~^Q7$d
zbN(j9L=S@fL-7gdO3KaQ93Z-v1hO+RJzWiKYaRqzA3l5-yJ%Vt&leRf3c2eW$1}2>
zIuI$6pv2~r4~$`)^EZNeLVBFFyIyYt8^fxlU&~8RU-uu213l0Z#$lOdzyJ!O>f^Hsn3@2?*kWDKjxOwLU6tFJl6FkswZ2wzog;GV=omw+i}bp(r{;
zO-pnnqD&7F6&%O}$}cbcCREu6=o5p(n$|R4bdmJ~dj~`*r+;;KkM$|eG@tMpNL3Ft
zcoBRcIc!=sQw+Cx2z0#nIpqLHR9;5dK`5K^JG$OQ+)9ETRr}lVE-v~=Ijw0U-_?}a
z<_X0sl!fY#>-F0o(d8e@Jg4c4_rx6m-oAlH2YAS_`i&nxFcUKS)_`2-hqJM>$BpSi
zLY?ohb}aYOlN>X@oj7Xepp2F4Ie$4p-fcbMNdxWT0D1U=A8nt8o#r4oM4WLf5z7h4
zijCp+Lo>u#qB113VTY6O-YY+}p23u@eC+2>vtcP{^BzR)HQN6h5dR+GmR~q^AtV&^
z?%h=2CY;`{@$2Vnk5cEI`^-A~XDgi#S_qFiH|#%gb!5#2KL7fpzlE$754pc}GgDSE
zis(j<(}Yna9cCIAPjyPPvL;n5U)J3R4?<9^16j5fJSHiRQh;jLtXC$uX@?#P{7CGc
zEn%^(QgB4jznPnx>+ZFgJ+HX97BaKDp(vYe`h5NH@Gv9|CBgO##?HaHlJfd_=rdz6
z#pQ)7SdQiZV~4%Hy*Y4su8*Z&ieD`#z3IyIzwIJbr0ug0@+xjJZd2Rw?ukykD|DarZ4(JV{QUWJbJrZ?
z8uzp)0e)Wbiq~EodKr>=9*R!HgvBcuxe2)`a?0YB#c!IjgL*O1eSjVTeWRS^nKRp+
zZ~EAjtE+-`ec$sZtbqzc27ZYzgL_?!u`164D&W3@@1sjwRCoBFv8cz!S)rFcJ5l
zgs;(E9l{_Y$&jhGo5X1$?Z&@2ntQ#W9NErQBuU_?W#-fBlN_mnfeAaWZMRv94Z2LVuo&Du%
zNJxl}^%2d#;nwxqJFDMYW{_ZY@=1K{33RT{8|hJcC=XwG#ZKpme%h_5by|Oja?wI4
zU`xB;+V<5ag-A#q3TNl!3ZcMaAivGs^P2hawQ0!@(FbyGkw^F^}FgVcEhvSR`4)(x}FAWX3P8{Gu{fY
zjN1ob3dAZZ65YTM!z1g+S8EFWYM%jej3&~@Ii3BznGVp7iy8MjShljfei+(&%#o`l
zF*vkjaSz3jk^d9_KaPsjZ~EzUxzQHS%r;SnVBFP?VtfLm_caY4AO{HX4Z|09*RPkg
z*SbygP?2nC$c~1h>dRQUR{SNnbac&g>ntH*Q-#9m(di|uwp^$!UX$p9$e@hZ(#z|w
zN=xN%LFe{7wS+QjTYlQYn-_>w=%B3)ec4TafG>zZo=bupI9y8KUz`)6(cosROS7BByV
zx(dlymY?YSN@c#!Tdn2WgQ{}YmLR>T?TzaCSN<(he(nwW5~x6qPP2i5fx`{*^727n
zP9ax>56CL(wH%@qNUK7SXzJaQWNkFD41tUqxYi`)&JTD9B
zWj>=ax8IMpPwH)k!pRNjmn4bG$(iJSE3z}GzxutkGYooM2G3XX=t!K4+BTV>e2@~t
zvul%y$68~ir}n74XFl3E$rR1rEt!+1xR}dQ>FM(MkC-ugceZ?!;;qp34NHG>G4ruq
z2GvT+xFrsUQWQH*jKU{*v|$;v{w0H&a6<@YK6Lg@iQk?50b$5>EWoezk*KohAt`F5
z>=j?HthcSbFo#%p6rC74G&IER*3GoRk^<7cn32?7%AMfgCrx`$RR%y9XOReQaQuGW
zdS251&YjFDDOB!0`N$-v66pih_?m0HiuMas65K6}lW*UyrC9G~lZanbr<1p5q#hbL
zyj9RkRMpwgWw;iEU#nPXVi_DdphkeX~IC+O86SIzdNZXN-Zh_
zp4#(^r}~Ej7Dgv7o=RT}pV_7M)jf0#kxITwRD#z;TUaSBS$UgbPL_F6n9`A%*ICqL
zSAS;@Y1N0mYuhCFV|E()YEcih`^(VJDql3ltre!jf?IQde0pFz8IuTEh}<
zx5v)}q}+#nNWQ}{(+(MrtoEvkVJF#aqr3w;aM3yU>@i!r{ATY)3x{(Tb<4QRuQ}ac
zxwBuzp*gBEI=de_?($x)aGg^B*5?3m5I=WfN!
z%nbDn)+jsh+Hq>JUy4ag488X6)~jG7@ppjI))6wc{BpjU3n0M`#Ie;*V!tPM#$M^3(
zj~}>e9(RF$KyE?7XK_vrj_(kYksOsluT#~+BKdlIZS88b(hJiYm!N8si`2AW1PZCA
znmk5P!b<-$YKgP==N8@Utwr%+F`S;1Oz7rn99oCd&-wZ}`IOMd0f9<1Va%WFc?Eq$
zJAMe4ahl9^v9{d7S!0mA)Y4>E*Rb+l_Z^TAYC!>=hqNnlMqij6LCX)dAbmq^Fwg3=
z;}@KGCTo^~`#)U7c?8ID)M7-trCv5z&2n~h!wwfHTKFRcTAV^>iP*>$%<4lD9C2=9
zW`;eAC@kY`S3wWueaH>tCvULQiM%PG$fQ*k?N=2s##&^WffqjcD
zFv}5J3>xKsGg?W+)m)o}468M<t0U!DFv!w!l6c5atTp9`(+m5tadeMP`Y`ZU9)C6afOOXYm3o6IZB9rvLadV
z3h_1>yY`u{BoM)|aFFc(z5pk;Jr~<{k9C*xg5stE@+LH%h|8cwUD&p{MVCr8&aiqb
zuznuZTL!@gCN}{F02qD{p<+Mn9Sd9F=pBIGZZh=3h@1~)SjU;1RQo-Jn-D9}z)&wY
z)zR5%WdC8per84%{XQaKCL<$5-;kP;LPSUyx5#gVF7Rut0hClULr${z@7pU6PLdvD
z%t0i?o?ZA%hqE&3oyBM5^r9w3@A@aZ-ABbrUcAs$BPpFVSEElAB?Dq0s;SD^uV4|-2}M>5FcxhbGsZ4ErE`2MQ3N{kRm(7Z7ZQ_
zLo>Hkyw~Tp!EHAXJ|65WTjYC`wv6gutvtfiV;993d?NqAld7LJG2!uMb&yffy@@aU
zI|Id4vp5n)e^49geeDX^pcE}|qb~oVWI@*j${t2e-6!)!(_S{#Xtzk17O9*~(ejPx
z58DLpOS0&-HY$zX-!$YCsaV5zLfps4dUBP$O$niay^qiGQf-A#g#LxbFw|)pKW!LTM>F6Qjp&6|ovkkloHJ-Jpd-5>do3EANI@oHu&U=ehA#Db+-!7|hr2u9
z{{1H-%=4bLl2uMTStVRTV-F;I-@kv~d~V-~7gxvc@Omxi?pBY;uv{Lu_qu6pw_w@~
zuLP?DRi8g^fIjT*{f5Vh?*j}A7N>S3dir=lZ1ik7Ct##L`@_+)F^4|+Nb^n1y4dN
z-JUa7C{uXpzczi@75mqXMdroVJ>EE~F)6{O%v5
zwbA7^cwckWt}Pc88dby;6u(SiqXwh87t!@6V02}9pjbeZn7}Xr&9`js`$m?&EqO4~6Dkm1iLbK!NOjUKL
zb;IfjRRL#)n!&vX^+cpvbT;R&|4$4cRT|P$BKOk=<)FQ`Jm}uTeUOFdKJ`yE#0&g~
zLTmwDx9BhSASJ@1ClsN6bOXKps~HZ+eVaUK9ULCs$yLUb{uLX1QI_;sm6jJa;C1eS
zO4O-=aXE~$Am&X>53$XCA|oT)J3?SZf~2_?ysBg!A>r)N@$q2NP{8+M-iccuMwoCM
z`nU~h14&Ir(_`%$&_k+*I@bHY*@TIU$lwh-(BUQdI}_MrWz&(#@}_UD%w@
zN^<0pO@7`5OlXr6l`Rma!PX(8LiCj?Iy!hjgdr;Nq&o@DeXR4IZ?ULe$hZL+)oTRj
zBpsoT@I8KJ)53s5e{b+hwnzVs1=ufdU|~VTTzt{0RZcrjN*I_u>b#nf$ava{CX^#&
zCdn!z2C)|}io&pM62VZ?rkNgcdPJOznug{-4w<<&I~uE-nwp3;8ybGt&0OeUgAkzt
zYfEUk5!a2#SGPw3UFl`gj_-_Z-GQ{5bn8%*u=4VTE-ZLL5+(@)>>;%N&{{Wm0@2K9
z(o-gi#u%ma{QMaL(@@Og^ZddJ6T*&k*Zx%d@ZOrTiQ-9}l$CVKZrtba@;F25qYy?x
zlu@_7O&=IDntPo0K#7y;C4CZQpz3AZ-syN5u7h`Xax3QsIP+7+&H?OKy@R3^9JsX8
z+e_l0)bzE_2f5Jc)3ZgRA;mX6pLU&Y;jy~3HAy+_35lf?qqWMr@1|lR{YHG0v3s2~
zT@-$UJ`TJwGf%Gle*{z+x@mXw2)t5mj+PhJT$)(>aKL>-g#`aVJ>_0Z+2ezH|6Wj)
zn4dZ261BarUhTj6^>5F}j0N`|5@+(>6lE&^@3X_rgnzf^e%!tyQ@;1XCw{&Z_bW&)
z%G`Pru$2x)?eNliKW{1e$#P8P?r40|ADuCi-Set?F}tWsY#uHM$1ddUUm1ntDR2Y_
zsJGwFGo}2d+O+SRM%I_!o?2-j*^bn`6*|KPUdo4lYc>lY{2V6w;y-2z?7x?qHg3Rt
z>&l{D^OwBZV^vY~JxSqH?-`kN?{`?8u(d*UfMXi$|_rW-N~}x
zl!3LBl#LuIvVX)heVav7cZu=yq*~Gc=OZ-RYs8bRe)2Lk3{tH>P5+NxpsG@gs>itK
zW{T%N=Ifpt3}*$ld1~{}&P<-s!@Z~hUE^zMYrk#A;6Yx?bIKA^553l(**#hdv6|F-
zhCfd%&~9Dptv);@=zA;O>>+g#oWjZS2<`6(1;du&=X{P=j1nW~=fA=Mp!0_H$C*m6)HNXi3qzYFzYgjtNhQmZW=ZbR|L3z5V+6dPQP{%F=#Nk)Tqql;UQ#M*
z90u^)7M}Gp0K	`;KL^-+Z!T(q4W*rOwu)t<-fmM#wX2%R88&W|GTPh?0;)`0
zJX1%5Gk18U_qN}Pp#7*w;bL5^`)bbu&$n8KHo^b<(heA%jM>c9FgPLka`z)4u@f|H
zHzKyEEYnUrbggaQd0XbyJhQfa8BO`@rSqMoJB^*}8JBfjgvMkPkWNICZZTVQa4Dw*
ztB2I)`qnGL9sP!sM2d!b&6P#}4NC0qsc3IXg>v7$Ldl_Od#AImMFevvy7EY-Gb>Oe
z|GNCBqn=`A6uPMG9$GWh!bj-SsEioQMZCEvbSd(V4`YDT!mo9(u@sG(y*%gapdQH)
z{88p3*HLPo%9ak++2AOL^AEkR@e6D~ZDeo|KupYx53%giQPAU;E?mbo$r8?$7#dW{
z67DI+V-sDJ-+5kFqP9^mit&)J_ILjciZ&V^R41oC+UIj;T4pWXjM%U4O0A+--mpGz
z!9;SgGfQy9NHi`WKfiSCK-%eDZ?`lw`EAzpjr)ikys!2)nZyeu_!~d&V>)%PS@{Fw
zq2;GCF$^trYo1SEX=WJUtO;VG*&*#J#Pp$ssj+*2&Kb%V92M&*Awe=INK+0_R#RQ1
za%fF@)pQ`Xd5_dsJmTS`lq|iycY2_B;CJ5=0nH!HwuXP{=|0kWXH`}`;H5#=A%!~3
zc7oW;^_(5%XHZUOGvsve`RT
zXqc%KvC>ymf~%TvHwFp%$-l0r-{dPz`l?$B-qlyAxMoLo%Bf4s+jA~=F;r+#R4WOf
zX6O?;^&C@$8pUO+)?a@w!+0TSt)-1es9TKo+D@FHOA_-wL=mJqUtJu53#w41$Waz}
zHGfoZ=&m*_@0LCqMM|*3l|q#7W~BWEYN2(nEiN;H%Vkz45k0-r!-;|*9+1eeYth>c80wuYj`1cMzVXl+9y>eJ(iZbqhbfu
z7F4a?4D$2u;>CJSGPoep5zHYFko1&ddUM!J(f4h%fS32Z=SG6s3r=(sblz*C3j;zl
z1>EQMsh0|No$qK{YZvdjt)hh{XO`K)(ZM_G70np?CLKTjx_jx8HvO4*!VL9pnd-By
zzf!R7ez@<@JIQbFG-~GiY1uMcemn~-rDRhaxHux{$-W!eL`GK!>2`F#0U~al*=~_i
zjkB&FE{%DfqN-S35l?wz;AHy0H@u*$zrV(@6WNE3>q-5DZxWzOkVj7+3>QhF4H
zxBNUB`1R-#w!EUybAAt-Lp(DRwjmZzI<~z$#4|m`V&wzEEv+8+z9o%2wRxq*4qImh
zKkV3{S5$8hH7)Zh)LomFyaMNt71+_rD@fn3>5iWE4XodD;&O(?s`Wbdfs_(1xx1xL
z`41oj_U%@ZWOh0`0RpRN+~hoz!-CRj>g%+3O6rTP7O1})9M)B5e(^&R@>EY}G3aq%1LSWd^XC{r#Wc>Gix->LO9!+&VuS4SZ01KjEHO
zx$*NZw#!@|G_{*qVEywqoMz){b1@2w<3R1p#feLEk**Ty3f=P9n2zOLWg%&`*K4Aitt_(&
zDDS8yQqvVYqps#^2B($$?_gt?Kv~a1b1s&0oZ2WLV(mq0<`0^;6nJl4>bs|!V;)Kz
zk~SL$Gjn=rRiFWuD%3sH-Yn|)0SOFH*F*`lpEZ*4$Y0Hj`}NJn*RE+k@A2_$NXc+g{+Q?+zX7Y5!=zuBdX2&;P^L*jSNkRC@uoNwIm
zwY@zVLnktM0idvFaIn_pve
ziKdRd;u6Kem#02jg7IEAhk_uTL^@uY_=jIXwY`lg$@)ji`tB>&ho!ZrFLi$Kt><+c
zJ{H;}l~>+V>TP<1{pN$$feL>okD{aKUhrdK=SxRd-aCcdlbMl}H>p`fIU_uv#(Y)0EGXe$M6&%Z
zs*Q{m>*;s@IuPSpC%s=YLMcLO&&T&P-NA{zm&&L-_o5TtbxP}S_2<%Z6;JKEt}WIS
zRow{M%zBICt7PVOVpMwzw#|Kj^{4@VLZ+y6xBR)OCwEJSP_C*tCBFsd0p&s~0PK^4`w9mr!@$0>Cx^SuXEzkxt=
zBO~#Tx9tgawIF1@zW73G@#Shjtk*qdpvf)7dfu_dN_5`RFT!nt*6vKXw}Fg|^R9c(
zU;Ji5pEI!OUCZXxk3Ss3_rL2e9_f&jYd*eSqkduotFG!4B<-NJTGj6k>>40mR@Tr_v2U8Y23
z+S~U`;JJj$Z@N$Z9`v=0`SJQ+8*{kzzZShRev!>?_gAB;a-~2ptq)6n71#+uiDrW@yW)W!ipw
z<(-|s%FKYjtTGwAuWo$tL2ss4k`?t>C!MwL;jJg?6CEjLlHYXGW5lm&Csg)>8hBEE
zhn(D4f((sn(e)eY&)BN;#i?ah_0%>Ea37(D+$ENQ&Ft)~D*!Q*STzSS*t-6ZrY&Hp
z5*jn-;S}NuSKygYJl4HJ2Vbu>6QlAxQ__S*9J$oN9+LC&;~^w8jh|YW$=@$2M)Yd7
zQlDoU){*}HAlJG+@e4&!`kP8p{Oaourd?cRWcPdomToJ)=p8)lrJfXTI
zHs%Fq?Gx|c4~7UyFYqaQUx=$>^h>1Wff5QXUlSK>c6_w
zv~`oqGDPhDJy1WK|5&s)^$>fclJ)6bP$9VZdie8W0c#Az&e{R~txpsg(ay9Ghqe3$+4~=|2MLe~?%js53p*nLWr7+#KMY^8|5_8-JIvS4U=qPDC`swT
zP#S(VYsk%@&DT2Tk@D$0Ez6bfdV)aB!i%HQ_US7Fi>b$Bi##yU1g6e3ZSdS-+;7vC
z3YjXO+`*eYz%H*$sT$J)XkI^HS*JAX05GsPGo4KCYx_a!0{&IHI*LGh!OV>gg9a=*1_(MOAJ13^5j#v*
zDnEZdq2q-Xzl_sR*e6SDNMIw_fdKm)Ge0-i0|N>*?0{{Hbg
zu#!kKV&Wy@DTc0QB{bHPT5MJg5X371tYaKPo?h!|uN9L_peqianf!L8G0t^(4-
zAQdKJ6$IsslT4t3PVBd
zTLZ=U$EOWVO@rVkfvztPhMif(e~Lv<{^;ww3Fk2`DE*a~UxSgv1Yn0mnvr;jCAl=?
zmZ+mOttGX;Y{VLyHZ1b=ImHU_11kYPY-p}si^8XbH`2Ck=@p~7%J467uA+eZ*Y5S!
zZ~fp0g_tkH^+~fsW1ksges%A?$a$K2H_N~ks^P6;RC>cQe_z@G>%|>~?tUu9x`MpW
z!SDtQ6tKpR87Fp-5pCFlqqB1j-g_|q@67HC@Y~3<|MJ66{h>x!SbOZdw9DGCWa59B!kqw839ZunUi=zTXwB`=f>m0(QVHz
zL_Ty;KLl)xxX}X{$CUOCESwX$n7=cn&Q!12>f+*3Fx#DWgpu_w|ES5ohj4`iS
z)4;LI)E~^QSmy-j{j5DEKY1(ZBAg6oyv|Jj9R=+m^H~>(|G)PJc-%Vak)3XF`H%C#
zQmU^1pop&0yzKosdfup+9*vC0l%d)wCZ|8uu~(BBNxE3LXZ!%`(5~3y6&cS?WzU+D
z5(j6zPF4$ip{S9R65sEFqW?ic`VYEiFid(%SmzfMta^*t!$E*_PX3dJi0@V^1F21M
z$w0HJf)EsgroQm7f?gcNu;}j+>>SW*Egc@ikZwj9l6bad=J#(^sPxiW`zm;5E8t1r
z2FFM1!*ilA=3d_~af>-p%UyxBg0{+MXL9G(n921VxM1n$S|RvrbxkB$OlfAyE4z`K
z4S`pP?GD3edW#JX%r(3Ed0;CqL!`agMGVX_4rl*q3*chQ^&E(D?#}80hKS6yhGH=C
z+2b^%vj8~m#8js1PZoeh02ntg8U)wDX8=>bP1@Sp`tD-K6n7>B<(8Ce1gO#D;jOmR
zSQ*DN(PlSeVmOigOuyyg;*#1RVpAfkDNaOtPFj1ofGS&+()C^TgzQzzB;1yu)^Hn>
zh&Hc;=*Ze6E^k_janyEi#z7it;n=nQ#m5MG;4(Ncz)b7Ul_1jr83{6reh3E^^HT$u
z@7y<60!G4hAlh&O0%-b}0V#1q%hQ?5u&dlyVj|p}3Na^$EYwoUHyVg$7pn3Ovp^GX3En=Hy
zo8gPOLe7M&H9u{B_O@=S41To{?DhOL_~K*L)8{3p82yGVdW1>m?x0m6hGWEsoQ%bA
z=nE4xIfmEtBji#5)poclwS2(V%xnvpV^q}yG}KO;c02%S38CJg6m5;Et*Ie~!V%SBv6^YGQS016x`${;ZBzI{EwLL%
z&KhLDIJG_x{r&#^OoD<32T4ki>SsE^4|4N*P@pNwUyA9rcGI
zp+y}na9vCA2H|sqWJf^s1h=1$A7v=sk5n2tp%3wvho?}6T3-~H0^
z-_~*mm{fv#kkjNALR`%W69cndGSX9o%^uG3!wM&^bT;1t1l8SxH@e|oxnGU3Xf?tBXL~Dv|k30(CPwdT%e|
zNgM!q7*2z~^*!eF^-7!~1nG#REdHC|zJ2U~h^iSIbAz5BV02D2oNfL3<7wM|X-)nv
z$XPSOC6do}ee<{E+`enZxJo&;t1H8W*J-PLrXF0sk!v{7#Eu8a;c9@*g4suV!jJc>B8Rs@uJ`us+YuJcTjX86vE7bDDE&X^#EQ3BR|Ove
z(zNFRJlN!azuWt9*a<9#Y9v$+6Fn-RBDD>Logs5*%-VN@Xj6H;nn-Md6EbzveEW|e
z_LDw>V%&QO);n@MQHH9yD1ehj6+EvDyA9%w)agtO9+UtNDk{MU<(GwFcncw}Ih?l-
zSUmev3wB3%Y^;zFNOe@#mls{o=Sgnr4^>Oe2p<%QtS4ve0?eGNiPA5YAxy9!W{^Vb)tXph9#qu6p~(H#|$&!@~pqnd{4!{)zzTB4We|XbzIW
z#mhJXdVcOoICASSQy)H(?I<-AoBaL3!XPpv2aDcn@U*QLm`@NExUtfUjfY>%4T9fm
zkaZd|f0k3y1t3Y5Q+9fKy5{HtJUn4AmA148He>~V#3So-Vb!T}($O)mrv#8NK%q7?
z?|1x8`^?xb+UzYN3&?8^y&gb_^?;`49sO%k0gO
zucy~JtX$JH06H&czuHDj0jGJm0i50NyLSbJY+Ev;+iM%(VUAHcv@ifM0#e9u8Sr=W
zXtNVYYPVnEL`qD5*AIn-5m3KxfOE}qNJ+WR=8YULMa4%
zLEzP80+`u(;*rLP+EWo7TA4~XZ)1r=xQ@LLipNHb9)F1yJRLf&go
z=(9(01;*_&&#qf9BYAu`y`Bivan@Zq;Ce;X-0+xdv7@zvaClr;Xj=i7Pd-FddWXB~
z&U8KI@%sggWct@yGH&U=w0Z}Nhm3TQdY`viSiQC9?+mjJCC=3=1&b8d1sWceduU|S
zM!Zw_xOIav$-Bw%VJ%{iy#idFp8EaUas2B(;1Iq8sx|j_^l5lZ%#Msd%VV7`rY8oM
zR!=k@)@W0mUB|xq{QVO3z|<#MHnxQ(xVCX16bBs*gXNGbay4JT&|-RVAf8eC<{nF}K#YdA|Z(b=y0|XU|gCF}ANsQ_H(D<9t4vR_Dm*&L72WR;I#5ZH_W-
zbssIGqz}~oO=%r3TUe(PpRnA9v5~)Qrl2t;PIJkDG06mdg{}Pjs*)|u&AWGJU*XFS
zLplHY!-t&u`kI-#QU?LD?f6BSQMo<4ru2mJpt*n}zZ
z3lsB@Eg9i^%oSQy>Ccd7jtta!&tQs2|5qE`&Umzj;crE|q1?S|`
zU%)|7KURU5hfrGiH}LH96<_lDKU3fo@e~Sw1waV2zyQ!AK+7{RG&^zPChWXlLKhTP
zp_`5$Kkfvo9~beYD=954?r7X-dC2alaG#+QpN$9mp?&@xaRGI$jeTBgkEL!cOi)QS
zxxN&6WL^iLd}l4e4>u$v6Ba(%@SdtTSQ$#{S
z1vGOXPE%|3_#Ut$*RfAcw>TA?B-j3Mby95cA>(1VO^uzZ?3gzfK>R@oRRCn2=0d)v
z^OA%@2`j&#$-Y`kxY<$&=)H$9>=E9bJ2=_ddx46o0_SYq`t^hg>gwvcRZNV@FvpZt
zVB0g|=mGn_$X|z09mCEKuW?DwTPovw{$G82+mycEI(Bn$R)PfO%?-d#<^d}Tf^uBp62U>D
zqk~dCpaJu7&?UGcJBa=)!n7J_l0Qp-DS9r=52g%UxNsrG(SyO14^#zq$OOKVA83kQ-E@R2^v00s3E?FrG-Wt-rlG^CTb?F|G#FUF8H1d#g!L15>c!h@(Fu>aAR13A;i_Myl3U
zc1;eiHat3d6+o6iZ6%|cAu=3W+=sS-s4Z!`4EUU+M<+6oz?{}F;^lh2^h*d{Yxus)
zlSQ>bfBveT6<7JDJ-J`zk~$sVgW)xD`)fr|eDl4i(L(lHNZZ6@`kv$_;c>7TyoD~&
zJ`Xv3m==7-@iHZBuK}to50ahhNXBF1nZ=Tl65`8A7^)&MdH6l+!!=fET`Lf8+QpH&
zyYPUa*-6uz@7~tXZ)poqAxq{(;k%L9qto|8_K;5{e^h<8CS6f-Nu6p+5T-->-annG
z>_Kwx7bQVwe-34siKp!00p)vJc=KRcv&}vr3PZ4v`N0-SOG|^6hA-$g{{_{Pe6c(=
zbvf$p7v_kJ#SC0mb1U_Bmmw9m3E
zJZNf?^}!^y%cS%JG092q)5((5C^x$GFkPPQkzvwHWh(1JhLA0nOa8W{j+!64#rWIj
z#16NDSraoIrb6S^6RS++$G2&WJhj|s!*u-2+Hl7VdTY0q^j(IP^{RA+ysK|9{_#2H
z!fBu_vSw`bQZtk3;Y<5S9*d>RC3sy+Y{T%nHAx5s{5y82OY>rcfK8XKC%iDIv|I_Q
z(ZRPfh!q}0NS2h62xW1mgQ&@?0rBqc83h}*DrFw6v90p*FDqvP@7&2nR02bKi-3Rt
zQ%);t2>Gcxepdt^Gy|;+M7Oy0vsS-(GxEb_{Oj#)s=mtTlcR1ndf~LQ;q6=Jx0u2?
z*Lwp#dj^6vDXpy`zsVydei%5P_sqXYyf({4O-((4kah+goF~__pKLZf(&}dWr8KP%
z?N!9aP_-wO^W$i9U0dIy2F<@L87`ye)m?5M>>XIF
z&@9ZPy?ECL`#QKR2hU2&?AvOUMR}u>96x^kI`7GUqi-3Do3)TVdH-A4vEm|PZTCQ)
z>&RWv3%lo2ouSwc+WO|v)~EI}eP4_0cnF&x9nYLk+bI;{c#2~fzL2yq&-wbjRWC!2
zY=a3zhrsnzkI*jZ1YM6p0b8vKL7hiy={(p&XFzz5m2=r8EFxmEXw)H_^TqU+STH}X
zg4J1`{We4PHATA8yQeQnu1QVPz2LCa=;XQF=4iS+fh!!$b!QlNqQ;tZv0c4|5^cS=*p${Gwb+iQLL@42`5|zk|6CJc>g4hfth%|
zun#!h%?~`M4nuWA(zY}zijDZS!^+dL;Pi>qPd#wuu0|uXJ;k!C*q(RG^|{`&0<5r(
zzwNihonA9->h306lj@dxp})?KuK4Q7hD0O?XPY=M#CgSC*QnRtbt5oFfp&x8-F^Bt
z7q9)HPVqW?X?FDP4}Hi4o(pqMt6uo^T$t&qmphkRy>+(C+J@3gn~sGlt7E!`-C`Hp
zxybWT<%RcNNfaDb=USBl2r(KhY6T)$EU%zIk{JS><&nCmtgpY=b5~zq-w3GkwlJtHGudy1eR5-V;0;F2dut2zJy
z(4$;cK|t3RTWoD>TGl&v8O&VY>kOF@tZzT#
zj5j}|8+i&TljFlaRG)p;H8*Z9Q#iG)FjDzR)7DYVH6QGc)Xv)t!+uavX8OIpE1ldn
z$=9u8ts$P{@(oj?wQ#TTFWN%6;YlcmWg&LW5epH<5Kf?2y!Hm9V0txiCzcD
zQPe&(5yPWUfF*F~B@y_aP6H+dpE_ZLq8Is<=PC?DX@^}1Taz?pTRpACVotE@%(T)lC
z;*)i4Cu;w3>fSRE6XunWi>*Wli_eDVz!mAC+-`qROb7ucwLi2B>Ir?p(p~GKqhn(F
zKsw@3xXYrnpwqk^TUQ}VGbBcI4|r);hr1bUk!6;IV;JW+Wx@1GaCnjss7OJSqaNhuX9q%CNZwswtf4<`WXZY3ov!G^zMdR
zJil=W`#p28jUC{MK$1Po>l1)*fngv8!0Fyu5}ILTsd
zCFfK3UGCTThDpmDbQPSfi9o{=<_J(3s!gf2VR9_|(N`KLOZ+QSW`8Xt4(kj%KcS3?xM)25p->_-m3L`~t-rb79jWd2373GB
z{HBCV=F@)chZqp4eC9_vg>@oM-ER6OoNgnj_Lk<~i@vTu|Kd-1t*ob6Su5%~4C2Ay
z3ThvP8#@2KefoxX6BF(8Eay>2I9U!+ydOW_!69z;E&H1M(p+E4!3wmJbqoyXP*n;R
zM{8`K!$`8_dtaigv#|T@*oE1@5RT#$@P_09tkU0WQuYn5oDH6ySjsCOx^${-*Y%KCyyvb(xw9sal|I~rVTdYoVK
zsH!zN-#A!gb-um3l8;_RY^~~UcQg9!SH}#X#Oq&o*Qw5)@EE;QQ&ZbVJ>1*^exO?|
zc2yK&f$Z-wmn=D1i>KP<_INI^?05h&rh+0SSech!Uo`eyz-oMNjA|T!tSu?|2zMtJ
z{=X|P(`OUb2{Ycd#O!u@@a(V0gX4-}Z2=~M4ssPr|oUsL8sMr;B%`5n)w#`8RAjkNy!>~-h}LQG0r
z_{ibIdw#vnbomwU&H_r*rLNW5e3cj{AF_t@3x|S%t$)@&-tPBb_T$&G7TOQT1Vu`A
zs=MycfLsv9B=&k2f6}p6R}prTNY_&hnSA
z?T>vtqwm8`?AZVFwA9D3(K{m&0+|g_bw`B{H3i7`pW@sxdO5NocewD1>%u=)Mz{on
zgv>g9edH3i3nDpWq4YPY>xNH8I%A?#jb6FEd-uS=4~3IEV&0l8aO0VVY254ff0uhi
z;fe_*&LjK3t}Wv~*S4Ig8U6j&V%Kb#hRIzE$DdXo_Yu)hIggNiR|>}Z*E?99~bxi
zaUGxy&B@2pEN&nJ)QRa`yNYq>iZkbO`aEQrQf)F-@2GT)&UKH
z62e@Y$Vc#0((3R?M9;aRBHisgq8Bf-^!oK{4m+4N36{}cv^c_N5}fsMkx1{FI5GspqGq_7{moZPKoQ7=i$CVD-I5Km
z;$NLoQ}aFF<)D=`yXAkL#bE1Bx;0664y=m0^TX?xp-796xH)y*-TF-rCx(CW37^nQ
z-$+nB|G7#AW@g{4tgQ5Rch^(d6mp|uap8*;nmTBOtc$+w=jW1M^40wn5#84Pceah1
zp2F~1_};6;{?DsEZtM@1_lc$C@0;EAJw!FC<-adf(EF>z-1;@he76Y)hJ;+Ynsu+a
zS&wnEEZcu+NkA7W{QmuGN4ly|?8X-^LBv7#_;EgA9tC$NzdN=b>Hm4B<)+uhw?zkv
zvx{FmWX0cdQv11Go6@3}#l`kNcfx~V`FTZf$Iy?m{i3)u^VDmb
zfWRtrIW&$OxrrJN9OX;(llEW#4_ogYPxb%De;<2~l#vlhNcM;#ql_q`>=9*zY}xCe
zj3k>xk(Dhgd#@;ojO>xU_a49JTYbK_>$-l|`J*Dvd5_oo`FhU#qo{SU@g<-isrB72
za%@216Ovk@y1JHCBd0;v5d}6z9ahsYmHzK&44Ic*iyrJ*YIj89FLBU`iPAW3KfwLn
z{Plf`tOIEILQ345AW7Nk750uftibadKPH)yE}gYj}83qa12
z6g*&We>smBLYNe%KrmB~6`5^=T;7|;Mrmk6zy`~W&dcLLC8nj}A|lSPYSy4A8L0j4
zcYpslz_WPGwYml%NJ4sYkP0Wl2XWn5CXpgnR#EYWO~|6Z00*#kd;`m{ku`CX*5|NL7I^E=(VzOIm8th>%88m?^F0x!N
z7Z)$ntlcSbq)vGM?WBCp1DkBK*6>95d*ZI=qBIV!q7k76tBsNAN=L<1IDD*JA
zqy2SXr|M{XkOnF(Ty6;zf!F|6Y=H*|ntu?@P9#czq6&z#eEt6Yg@SRp*DOp9K8E4u
z99Q=e{@nH)d!5|V2Wy^Y=#ZY*%9CgD0Q!H9CVt%09f~EuM>`2W#izh1>uW=ut@4pi
z!8GBVBm1OukLouEo<^)n|5n5v{_N6^G2~`yU3bLjBh(K6jbLPv9x}l9M0~Tiv$KH!
zI6@u;1t!R$4W$Pa_p;(+9v_UJJZTI#lVtbu7NkCka3dr_#P`|Nvo&q+zFAXmDo$Oac*@ZSm!DpgSxd8$n7Nh$r#?Ww_=Q{=FFoXIR1i!1?gdu=!z2=hl
z-2fQmc0Jy&HO`04
zQjD}NTDsm}Z4fx2^95&wMv{(N?DJV{-)|-B?l@nn-Mmj7`2xFJSjL#<5w|H&52{*o
zt->+x^o+6#3&(T{!uY2R)IwNTS(7R&k^CS&5fQL|@Uf*#pAQxk@Y9Pq1HNy$3HE;I
zNyLJy3CTGmTLf_AQphF%KGW;lNC$)ke3k=(!iFm6ft6(rLtk8m{j;2%UW2#nS-H7>x<9O>AbK|R)GsQbme4zyuM@o(
zwRWAWK&c;ML1!^)Gg}$9~pMagh!Yt
z)dV#?75|8Y1Tvt1s`N{I37Hz3hZt;JmFnth(@p;-u@hufUVz+d_w;FH@&new6^_XN`#+&u4(ULcxc*tmOoC{f6y+M@^-PHg6=G^+#
z-~S!idV~uW^rqjq@ZkikG-n8$vbjBUH~HN9Xut%Q`?d=EIRqb~;xau&k#62i3@7A3
z1J{wCCx(|Cj)w|qTP`kZYg0g)mEk+xAK@R#1qEH{x*5usjvnw~!A~*B?x#Lsr*z+i
zWB%F&XI>Z#yi>?|mR}JNh>*G%bRE!JNYPkqr$_$!!&;;#%D-;o0{pcAb-8ykC2M&%pnH{#gOe5e@k1f%mN&%*c0Xi*aL4Z>c#lR2SBBM@gv}
zZ2x(fS~KqhB7hI%5|5HNP;&EW9F~u5)hQ>)W_S(YdRa9wwo1X*$WeDIAa9XX06oD$+7GK@PIOx^9Dx}y|-=GyEIN<(hePnv@_0Qt0ZHCp=PYn`7(aBJ_#
z`fJ~U)S9^z_Yb4gi^)aUO=?l;s=c+tb!Xnw({~m|2>anUdVfkj>udaN^iz+ysj`G@
znrf&dId)-ham1atBW;(cDdlw4RR=roDhBK12+Ik!n1Jj$o}YJ7kkUfhOY4EdblP|<
zj8J01DLGG-C$n_6W8kWWqFyXFMr4GhU!l$ezUPkAxF8oj@DjcVi_nBlZPB-=B$HPP
zcbm&gfBmLl^iRrcW9?g6p+kSd3vNt4!flYT>Oc*=y;9@ppPbS6e4}Pqf=uAnL9DLR
z@A(d!>pVKbUW;olX(-5tCZENBQ)rP>IrzNdipgPajzUpL)xJ9)^29O0I~%*;`XUV7
zXG>9_tM+kLDmO@h=0k(DN>jr>R*_00)e2^4zyNk=rqzSKsHHCbwb>nc_HTJ6NT2j{
z)Re}buTHW#UT;vPb+e&m+(u6g=J!`@X3Eue*W6!gK{Ir~x7J39Z40RcYw_mZT@l=H
z=`Xs0w;KoY*5v9h>#gIVx@k!`wx$Cu0AMd|L?Vc)#z=I-&V~*&AVwJ=_f}66T09gK
z6gGp}Q0_1HQ$mhYGIi2KdDLUB=0BY>#)YXzi;)jeF(1>F=cuEwcFUH_L(}CY8fE`%
z3+ay=*gXl_3DeMeBlqR-b)f8^=RP+CgW
zhv~+TJonYBL)r^(RL$X1yqTn>I>AgNuauK9*Z(84$j#;)T
zbgB(t`m-{2RC9!V-gt}5WvaR;QjaLSXS_8kAlp05YM-H28dYLOUAf+N6=}?eKI?jD
z>j`UzK1Je3E>n~Z6L(S9K7L6hGl<6y@8%lBHXuY-58&WcD@o3HURYgOY90fN(Y&29YSicFjm2aEuZd*jQPTnu|B0m;dz99X4i8PSunB7cpi{#szky1Y=(iQ1@
zex)UK9fofV?%Cj{ZOwmUax&nXo9JlT@Rj>h7`Vg7Yqas#63BW`m=d5*Fl}&9;JyGS
z`XcE30czp(If%+$!(0T?w*jc5n$mN%I;hC>`-kZm_zpGCI{XJlM{ik{h~%OY)y+Te
z?&e-4AYv}Sg(rko(sft{tpuIptbkihN@A8pN}RU<
zYl(af$-@R2M${ddqu&w=RS62d{A$DHjI%3=EER-wk6y{BUBOEN@vQ|7HEmxKo~9(0
ziZ+{01ypRh*RD4K6Te4;WODO{=r!8l_Vu+FX3id)?@3s_?uFq~xZBOZJfBW!x>!{a
z%`@(JbEo_T7{El?M7EAGm-H$6(M&6^bCcVf*g
zU0r$vaRVEoY!Je4G&a7Bd=1S64Kg*DBIy0e5P0}$?{WA#kak;UMY!b<$2}7De`ijK
z1rfY@R+{!p#k}{omy7uuWWrEyZb9Dw?-Qf8@=^hK_RmG#JS$FdLUW;W!sFO&qH6WI
z4Mmg(Bi(TBL~j_(s;MFyI9xuSLyoG^8kCKuS1N+g0lujP$Unoor>EnCov(+wH3>BX
z=BjnK#Aaf`B;V8HQHO({@)gQ2MjwfNi-wKZC@NFo+3H}>(=f-n+5IJUv>J&n9>w`R
zvFQNQk&b?xyDds(86J*yHuVpEWdqtKHtfmcqD(f{D6qX@SgUKC1~EUI3ELF&tU_of
zZb#Fm9o4cCBZ9h7&EUJX##LZ4#qx-Syz(_C>D1x-u25
zl8_TDUd;--ftSpO>-A=0yDK(-e>GV!?@Pslvs_+3J~|F*T%KoCpsJ;k>_!`A^dSaY
zb$gj1g0t&pA6R!onN2B`{!Y3+^5xO~$$c&3RQ9}#-u9i@NiOYUw?moilLbzSb^D8F
zuq&zg2!6S4S#Hw&Fith>E4~VsT>6pB^;el3%BiaLYF1?T<=8g2w;NV#x!-mrlf?sq
za#Y9BPxWU{S-zr_&8{b^R~8r}3GE#Gef5q+
zB!xIAts>9OPiLtqbZ(nnma*v!D$9!?V)4P&n!3EDKO{6!xwVyq*{ONE&gw$wU6+IR
z15^^;$Xz(dJF}T)j>-cBr)0Y0OIV5syd64gY50xslfj#?T!ugtYTMBnd|ni&N>wMN
zU)KG}qfK#N4(OI#lu0BB|Ij_|aX8plI;ngM3syqHiExL94*z7tDjQVI%^zxMC?HlM
zt@%(M^|^G}omC3@;3%r>nZxc(P7x)!$cdhZlMj{I9v&PdYryn@9VGW^g?i-Z*gntl
zuusk!wZlegkc7{B=~qda3g3^8s23|fFlkE^W~bD7ER|7(IFYS1?k&ES4Y9Qz7H3#o
zT&@2(xHgbZU{h?p|1qI&$GTy^=83~b84uUXprH%Kugohw4d^_@7_o|*MP^(%1SsQj
z<6GkJN(|T12{gsbZxuFcVxr%^rgJvyP>(lD6zZA#u4)k&(lF@nIF=i#%t5b0&hi9=sFqprX3;IRA($ZYp>SIN-pcF8~LIyByIdFMHI}xwzrYTY^(k!*PcnPWv
zmea5w)&m&{6B;s0*{N!S9A@|%CPN-mTkjJvSuQYU1a25!K|jTCPxE@j*_vBFFk{F&
zzVeagMg`H;m*DL*&M6Apj4fcNN6|i%3VS9}}ElB8C}g@QA{+bzDz#
zW&EnTKEsQC=`h3FKqZmZd|)=~^l&h`cS+~@Bh{-yzjs{3uGP4Z8t?UFLjX{2+_s*T
zzt?UL->=M=?0AeiLGOCw7mqk39&a&)DVfWkdGLxz#`sV9c2TCGyX&BJyTWm{xKlyh
zSQNlFyx}j&
z9Yt$dvbOGWIP;yL@X?>tDjh{uFKm2pJ8r*?;Spnqe;u7AXkKY+BRX=uD0$d3T8NfJ
zkaoH>!}2o@#2Q$ZJHlI|Wb+C5zr-okamIvD08jaR+>8kGKwme~cM60k)PuA7+DJ6!
z*abNg^BxWaM-EeP*NWR*Q3dYP|QzO`YYzVejNwngA3
zud@j+%L7tOX!gmuozpe87_CZwBu>yX)zp{a{>-jT!t^vI~Il1)}{!Xvj=;6`p3b1|K=fz&`
z>ijC&I40)Fuc?jQ*$eAcFyC)JK7{v%7O+*_kD;OQx8#1Fb5htLM#IMgrUCB4i{e4~l
zCGJX)^Ub9nyrpy1A;DA*w|C4oFAh!r36a|;t-id~*FJn!gSs*!E`i~SQs~j%&tJ1k
zQ(0~uAKoaDGX;Dp^JtIx%ALo!uvlN1SFd6n|1r9#o4G&`c{Vw6>6ohbF7uU=qeo=w
zpQba2Y;vK>1bZC6oMle2G*%c3|mMmw|aGumV=C@R8%yE2)^yz
z3-FkBHQISl1X#buXeMC#XeGhg5d@jh#RIVm)L&8!7wvC$8b*>`g`yNE$DA}bsaIS;
zd?UX-+>HhCBRqzp84*fzMpp=QrE@2r22C4NT-kR%&~9I=aArts?OHVj5?!;
z)H|0co;z;*Z2xu{*iIs4iG^`;mJ1A~&xgkxqI;|_3C+^-)Hb!J2i-b~mdH8!5ODAD
zX-q&|fua0y&6a``)0w_j-%k|U7wMPFW-cHQMcAP%oefy5UeF!S%Ce*s^cnYEDdGr_E0D1*^r
z?9M~*AcWr@cJ{S5oZaw$O%;_IO(k^TT;b+C-eML~o11$%q_xI27?<(&d(5Z4J9f#X
zUnk2^t7e;0ksR|CjLQ=v1UiiFT}JD2`qD8Mb{(d5?`%>qH}$A#I!u>ZjrOh&tL1bC
zXkU#zl6YuMCq@4D@Z+^QD)62t`_7O{Oc8fW3|p9=N2OpnQ}K=xzO$DE9>pu&JunRk9TaQ
zKTzZN@)!OHg1lM&))Em?P#u-Wm=(T)`k)uD)e62}G^CA6OGVGdI6h4h$;!MuBYng@
z2CGzl*bWdx(yX&3#30S)hLBt3_w$*}uOi77p`VY4
z-_Fm=>N)WlI5&6aR-CED=qB~M*>BSV1=3T^DagD(0Z8Useq%xeYzTdCx42^j)}^IW
zl9k!X85n|LhSC69TOfsGW*#vN)ApfcYwcVluPeMB#%X@~$o65&Iy7VJhyQ@bT*9Sd
z3Gab8%Q)qO#;seZh}TvrMrs$$+a08H%HE*Gt!Bl$==A)ZnbY$`Q}`W8!J95t1s!>P
z4%xO?Ox>U#x2LDV)2{XR)L!pg6~hmRkPN=LDaW827{!Lxo!oYQGKOE+ce49qEYj3K{^n8y)^%A%Oadt*R(F;V`AzK5y!mx(sO(~(^I3IyYs=-W#7QOWAL3r
z_6fbXdL&AL{i}?+OGYQ3V6SVJ?AKcLNBUs~fw8IQ%CZ
zeh)g0eCaKPQ=V=92qX#i`ot$vV%Fir;k!Yva)kGN%&+Ewr5GB;qpz(+W(}Y|79t2k0!+2*yv!T6&Zd
zChuN=GQ^0;7&5fN3+zK;jh+H^c0SGVHsnvYbT2;Wpe*o%gya=9)HaqT9urP;U_u&&
z;1G^+NK`wM
zZb>3YP-}n=d4`l!;BKRFYqh$eyNz5Dr;$r%OQ(*O6&*<~QOWkUG|iCO#;CDHOEBf)
zt1OP3{aq`+Td9pbK4Nf*#))4r
zB9%K!HWXi#8htKs+WuVw&Qy=~;<=3AzPLrGtm5P7E#2gem4GrZ2wZ$yzwYl-#R7l;oQ
z70P{luLK4t-IaCs6?3Wpt&7
zW8+D3r(jJFg^C;^1)+u2@TrA3;h5rHWZZsqCsC~cYS|~SE_dttjT=v3J_Koxg=#-J
z6_wxEm;rEy=UH6@aP3)655EAI!T;YU8s^h&0fDQke(m>gG!}XewT$tZtab;LD;dA4
z(yLcCZ8f}
zd>hN>F#LLp#hn_pP+4m_jEQK8{l1sdHy{&)ze9G*LcKhfbG;1*@mhnbWS_&7
zc0o0(_J!eW^HhouE4`#+QaoQx+fgHbM)*e>H7v;}E=SsrP5
z9WLw_$YDwCfQdbP|I64#^P7mJE;eN+NICdnFyTakzMj71Q`^(6Q~_XGNI)VoS@tHJ2F{p@$xNUO+p#~@L#mV&P5e9HdL75iD0g(SeO8@YKvD#df
z%dVCjuPb(~(x=E)?8D`&s*ZTm>;uv*2778g3yCkM?HtKR+A_}_f5ymJ7oQ-qliuhc
zxquUBaD4P?VxA)_JK>C!p3DYb_}RFs5b%q{3Xg=v3UeG?9S?B3XC1Egso?t<2+IH7
z_vf@-Ho*`0`ih3t+rZ{A6KWT~mKL?VPgh=1@dp%Y8)3Fo#zO%t5M~B+
z>kqbIS3;-x-!=`?E!Q$c5*CrK)M5{}k&#r20&D?2;8K_c&56;-Jw
z#sdc{UKF(*e9dF0Zg;Yt)|Bwq4%IB9Twd(GD_AqPe%&Z4>e@5L1?~@ZkfSV?;Q1h>
ztHf&b;EKfIzvpszN0f#PI}?Km?=xwwdjpBdkN7|!ZXg5pq3d^HXcAtXToj;-_t*TV*z&iezH#c0+)&=E&g($AXFKj8&4=7*uLjcl5$cLKYq>Yk7x+tWZZO
zTBlm=2HfC4HopOUhlt}!QM3fcy*_;v?wHey2*o>SzU%mUdVFV!F<{8RVx;msG@)Fc
zv@08*m_r?>SLB|hE!)0iKE~>e?pXN{NKSTN#YNkUulaEEvgZv(L&y4YWjp5PoFj>3
z?jAo}>_6V(SUO8LpRcf3trZ?W$krOv``+4^9V~GxRYk}0h?CaQ-_+A>Lf?6*_F8(J
z$aLE~S|l0PDRPh0(b1XJJ*C}K{aH8HO1YBgamc_JTW320waZ>`d%yM)Es7pSS@r4X
zL#rafZ$_$RLZ{Z8`RZr5veGg#AK}0Jf~@d96K(0a5aITDAveD}*t@H>^9)k+`ar6b
zH1$#~>TQkvS=pma9B~O3n!cd*V5@i3s_F*M-07ExSM~2{oiGq)a!7ykh>u_U=3)Ol
zuZNb^k?d^yiOUv6H2nxp)WGxblzS?{UUH1}49{vPF5fAFd!?nw(7q&b4xIaQQ63KS
z>}iMlHFcknW*RG4ORI&W>dh7w5Y49oKK|
z%hC;%K-6;WJ3LyYPUby3tM_)<-5u9Ir%!dRpT%3Px%-r__$lM_omC%IW-0Ixkhl68
z%o53w>9FvIaQ@ZVxhv&+cgwV#ty*Ni&)-BrN`i)#d?;;eWB4Z7lf{9&FfH|@s}gpX
z2+lmKk#!>DdEv5G-_nnh+M^cGt|Vut;@#=l?6eRVBlo&6qa&^A{ci7mgxB$_>qq|e
zzv>8NdZOMAda?WH4(n{kvK5e@rZlQM=8n3R9Mc&DRlC<`-iR*O!pJxVj5EmwpIWc4
zggCgk6VN>h${o1SuAEt8N{nVii9NG9`sAf|$H@Ix;;V(q?Ig$EXC8p}If
zjSTfmRYj$(9@{%4wRr+FNpfr}>Jpxi4?pBxx+qrU!qx4{75U)qtJS6cv%yvs&fVCR
z9tXp->pghvU3X1#s%!|K@x<3Jj$o~Ac!U(29bs3vjMc{Dye&8uO`iNwoaC`8W7
z#jRWU%jZ;@_s<_g&2CCG1)8u}596K`qhfxoy^Zhd69{5#Sg%r23JIZ5vC^-Y
zG@ORccM%D=ZZ+v${Fr2@n$fe%In7&r-JU+yI2DEBkz?beaTP`4akU=#CsN$ZdAhYf
z8o!l>CtRA}|1sEkzQMN~K806oz=>CPiNRuj!-@MU!`+ajx{s04dU^`j7Db)5yPaEau
zpu(=AqC$_L@Zj|o*VxF&U~{(fRQr`1!-f&9i`q_CL{OGgD%yHDQL4_^di=
zMNPqudRb(qiJhp_o}P5|tDUWNXCy7x6UEOcJH@+}*x*P>$^>JY2jgVny?HP#WOAT0#3n*?w!HAy>}9v2o}`4uPPQ}HtX_8-
zDF~PxG32CsB_-Alxi-Tkz~6+V*8lEpIt=i_q&Z|F7N{#QcIk`B=nLsY%)d03x7rmu
zq5XX?VOP=34~%Yo`l5VN4tWm}dSia)-=eota~1J3x@iIDn=VAF7wb60wDd)Z-^I6;
z@n{QvO7Q3Gk*e@{(WQ0pXO0AmCCa)pIeAzVIqHz4*5IffQ7`~|`uO?j6%MfY9eVLLqS8i7MTT1CHD9^E
z_Jd6*XxpTW>rD(ZJ|>F~O7w>npUQu|0D;*28Q2fxmsTjG(5lRm%%)y>C;dZ{=xrfv
z)3Zd=Gg>5Pjc9%4C6+BalhQx_DN5U?EXvOds*w(rl(c+*2ZsP^TMwi#W2FfzsJeN=
z%1#U2d-{_^lHIw8z8X?HNk--ZA!a58T^Vx3ap3gCKvP4%2UWoo-ONBE`n@z^Y>{I@
z)?Iwl2npeh)6wl)#Lo%j#X>ZR+zH5}ps_E%o}2Ba7Jc<+^ndhg&Tkc8Po}rGXz3oBl7ZzKw`_L3bZ<5ThYK?VZ85#W4-baedtxy%z8=vzio)t8WFPPim*5XTR`$C^h3
zvB6hVHS?eGmj1WSQSLHEmnn`g8yGAP*gwBv@I;6!3IdD7#-wI%^UKIJ
z=>c8xcqU(;U?M~)9(^Joa{2;h4NUgm2(ES02d!tT2^U#IMl@4zb^EBxCOJs$Ee
zN~EdFG8edq2(tru{k=rcCJw&_!cNWq9Z314F658I?GD2M6Q-j#=;Yn30RT79{@n)7Fr;O@E(nh#wH3zZv+o7a{+^G^Ie9
z1&oj)qkJ-tK0&D-+%=&x3xHiBT0d8P)3M~CsSGB;bK-W^*Xv=(y3;(b4_*d2O)b%ZdA;NK7c&rSmn>Mp$7zc;>y*fhwxwLzaS*O$Ce&x2qL
zXtp?govtQW4_6U26zKef$s9zH659I^SsYd
z20GCCOKmR+(zYuu`3~3o`@b-*Q*dU0#Uo{9a_BdHz5z}1Abji`v;#ygU*OTqO)~%w
z4xq%HH-zI1{v8ps`Crw9u=WUrRHVqAzG$Moin#I|Q3}Em#%gT`gl^$>R8-W6H$kyS}MJAT04
zAX!}yMS<3LnD2beAW8*qCd?25IaELb;sMn77D%T8L=B#4ZxA_4L^9JA+fRI=nh{^DQf}F_`4`4OAF*`tZ#;#0PSpU(Z
zH-8oIFo2tI8#+#Z0!o4SJU=#ECkvYzn0Q4f002pYCVX5#9K(PB(Sozjm6&K}yHcmD>3aJG@d0oZQo38mp
zDnC&YTbkJ?fcXxk=I{5TjGq_2FD@=~KENKyn~8lIE8x%(``-A+(~j$~fK>TukmZwP
z2SfQ}U%q^y5i&mqi2}nrqhBoJdN7z=zI}uuRs-n&e$5F4&6`!YE8_^I1a#so9rb`f
zKumXy7<#9QA5U~K1A_EX~1Zg!^7{WKPtpNf_gf1hPrQ5*ag-M!VfMVbRXr~qoKV`{MUmhc}
zuR#ZV>kOjutXXJ60$1c*3H0aJgFYHk4VUL_pao&OMmUjsj|M@uOvt=1Hy94qaDQ`7
zMqEiN=DzL7Suhx2GK}(1ldl|L8rM-MxK2?iAO(K6D_oMqWycyGC6vLICyEI+9?-9=
zhdBgJ+;f{NhOqn)HXbibO5^(^?rVaKr0Hn&P1x9yX%53GKtO23eM9j%h%{+wX{lH7
z9!5T|)JPXgu9{>$VgHZ!0*Bgo&KZUGp7;NW%=ut%sK$#C
z_!i9oZMlc*h4wK}Nf7r~-5K?w6Q_p0*yixdcTc~c*bl2nzY^5;!h5;Q0bzYw?E)r8
z(=$;RF9Od<8p?&Ck&&uT`s~eO?yD>#9aykn*Z(cp`;4gh-W)oxtHg64c&Dyr$_nb~
zV0irYfCym-^U^3nOp4x8w;k?NlEiw?@Gt|+1hDG7u>cuHcrcY6vOy?539J&N>@;+(
za#QOdXUqqKWj!l*$*~c%1~hQ1Rw=^V2qNDO!cpwpPIh)apFVwh67w7`@9M@zLxJ%(
zg~k0VIR((8S`VBM+nr^-WMd)oZa}V-v|8;UmlL+RcO=r$o5x#7of8&f97!WPSgF(5
z3V`TleF3D*ZP@WH=O}%bS5$lnlhp`s7sfwyD;yQK@)1N8TxKnY6-4eFP;pOt++f5E
zsi>;rJj+oVn0grL_SZFtBE>!3k%oTZJ+-i~Am(!1oaA!--j8VBW|aUAqPDeF-i}xZ
zTdEcY#z2pXN1hQLdPI>H#!N5r@tuK(6D^x81vo;4=&W6AF5jyU=v4rGSZ!+pIMjIw
z^5!!?801}nYxWta{{T3=$i{{TyBj_!X)`DqlB~${0O(HQcsDt7=L<|*1K_ESy~^`w
zAJOkJ@69$-y_Q$CO$%GqsI%Co
zWEK^r^r->iowBg7&~(AHDZ=N0Jn^NQHyIH1L?tECq~zp$<8M^SxhrttKfT>a&G)JT
zX~c10H1W+-bb^%!Xy+|WO&YkqHUNY{C;>oc;%i%hxvjwo2d--#`=Aiw$$%Q0oecmg
z4>OzyaGrkyc>$nq&z>vl11<;17LuBpZ!hM+o^Yx-4XBPk(31wthU6#IVPG^q|2CK9
z;pSGNIJ9+gY61C9KiKcbAz0r1H?Ibp*9be>d1G499Y&R#TyMt6>}k5)M?4MYq%K24
z=5}11x{Ep}$E4P38#@#^+ekV~1us+5D;e?h5+OZkkjJP6>fuX3#L`V$T3V8D0hO+l
z={$ZlkRf3EQxzIPmIopc*l-9N4=*^EC}d5t+~IZGwup!-0XM)DIC3mYaPTb^E7i}o
zcVl?;n;{{~X-4_kdF=`G_S8;ZD>P}ZN1j#r0jQ`VYR?hA?W2`0X8~~)fgKE!{x&Da
z9?1&xfaSWQ{Rnn4I-kcdE``u7#}*cXfPi@Q!UAj?O_1OVUf+X7XL8jq
z?f_L@1M?6krmhmHt_1)cWMA}ifo&RbGU56PxgZ2T@Kuy3450FH>)t(LdIpBO4ksvN
zfD0a|yV}uE4nhPmg$XNPaY0;_;JKf{k^@E#?5y&kZf};x{rs!Na9th)AQ)Jn!JP#N
z{sbJ=SHtJkGald|$P-BOW{CU<>)Hb)H-v|j)>#g4OBwJiDMx33C9?|5{Uoq=ufAac
z90i)=@A|X?Ec3RiYGle(Qd!3+EaFwKlO3;D3(@L(VoNu@L44(M)Z$6#;`MKDuYlgfPC+sa0K%X
zB#g?()sWGr;Wdy9`*Bf3M3Y|vczZz@I4B^%VCpA7RzPUcyPh
z4NKq;fcZ>(i6o^A?p-i|7KQxZ19y-iNWdVe5+9=^K_8vZi(365x`JrZv-(P)RzW!82{=#TMSzuxBU#l>&TeaY_n)!j{%h>&uP6xH
zfU%A&xf+iC#LdqX_Q04h0%)8lp1|8?S<@p)UoOx_F44R^KyvJc9aJTNMKMx`S
z>NpI`M5MgG;h@lS9e6iqPhhU@J8mp$3;?LNl$74EST?<)v~hG)cZi2J3|gFV^}I(p
z=JD^I=44M|(1J3oS_@*<21=}Xm?c3Fr+I*$cw9lr)%^^a1A^cqSMCf`$
zL#8gW<|SGJzp-^WBNFjkVRZ|APXKNU6s-g*g5dz>)kW5+u;EJh2pOt^LmE7_cl~+u9yb5`&!b+mx*9slAk!t4_cb0D>OP
zGdaDa2A@>ge&x5Y?Iez-6-e)Dm=mpZZN!~igS6A3!ObT
z;tIo+vcrS#HlW`UF{)Vge7FATiPI
z5bP8pX;0Y~5**z0l8Bmk1E%X?SkEggL;|MInXM0Cm^f2=1w31*oJOaRoab-MG*kpG
z8h}{&zGs;Ub%6sKqz4Ze3g!82+J=Wp`$HJ@bW72kAe=32uQw0>96tUhSby&uNravXaO>(n}pXO(y5Decn1$d?pMJFV=?nOIVx#dqP=F3ZhoSm$JnN`oqliArZ3aWrjj3!54U~L%b(o$}d6qwWhJQ
z?%mHt^6t^OSm-K(;S~hdeXwoW8VOOt*70UeT;nV4e#>C!eLOdJ2*7xVG*wazp{#gU
zTQoH#C34n{bGZPMkfZhJ=m<&m$h$%S3H&5HL}b;*#zxYglmUsqkzzW4QZ*O?I9rce
znwuwp))ig^$rAV7aewt0U$M^=)YMOH>0D4hY488ac}Gh^ckM-RaB$CH>*%jv*yy%$
z_wN7v`b|C*a*rqKN*@x98&OnmONmw-vZcY_U9nC}sKRs1acA
zVA9di!RRg9t8W|Y>*Rlfam1xae^hk?%zF@mMutPUffz;B1C6#7h*-xUj=%q&&0PDk
zi<=v@%ZW_Ybhr6XscqPo2t0ImQ4wEi_BT0X2SZ*8jF9XGr-bksjQ)nZq*{$l4%Z;_
zkU%NC@39C?2p{3V5!4(wk=A^P2G5=afuCsrTTJG5!n=1Iu}-$OSE4t#xz|xYf2mG<
z)1r+$*1CE%A3=`AaO)vqIVUa~D6ZxMVzLs3u}yz}z6d4|^SzSB{K0)CCGPwM7GdGO
zi5t|Rj+Nu~0gT}&
zY+T$V{Tff!mNOapHR6aY3*>^55m76MonZ7E4Oj{TARItY7yMZ7;qcwo7Eu-2lN3&P
zj1~D4Vq!EVJ%#~v%RdTU=@Ze8F7~Tsh~5Es2Z%%gSdukl(ZVb-5W5A-+Cj3G7uFln
z5&+R~YPnac>J4NGHMLg;x70e2+jkF8N{KlU6Gf!Uv{6P^*6*
zMkALeERHGwCfy$nH<{~7h2FZD6pz(9JP{bCft05iUn0m~s4;wki7#2R^*_<6V2dB|
zpMZ5?c3_JFa6x0Ew7j-04jMwy?$7!8Qm~J#jkv9hgMWS-rmMDBMnk}jKGhrBe=uD0
z01bm-@a$fd>)GO=|6NOL-oKAwycp^^C>aAg8;d=5c@$=eIb$Kt&T8VWXoO&v
zEmpWkDE0x0ot3pY(|#Bf)M*7x4c@-`bF|Q^gCg)j1K}ZS#~vV_$&x{kXv&W_gzZm7
z1e%f{#6)sPkj~=JqJp-h`1tq-gI)k41sFQBmOq9cTOYdE1*niP?tm{p0A}L9+f4ZP
zHh&bd8lr*SmjC+<#J`YqK)g7`1AIvKw3)r(@J)C-#X6uu!
znIVr24Siu#ROo`d14|3{+|l5`KoZc5!yS$fH|~SHxfvI@`I#)1NK#KpNxj$u@`>;?vB=Yj*cjwbl?4ZrK)`)b9**jL
z%7l)h=W>6cT=+G^X5!@9uEIy^5HsAiu*iN4{>WRk8SPI>49Po-%gc|<5Ae{ixykS|
z0#Fj{xsF3T>^K+U*4hAg_XCImu`09hW+V14^rlZ*VfW=0^6
zhs$L6Rst3+B#hX#S~@ybGD02}o2!8$^qp|WqkRXp!TIMmkWmN32l5bub_?Jg&`d)%
zXmzVkKWo4d>|eN5HNr$%SkTGdI|j<#o%Nf*uZ3n3)v6JjLK;gKs@PKhGhr1GI~Is}&d=Uxk?M
zPS=R@pI|~;F9I^M7N=2%hUg
zTo(Xo9U48lseU|0cXWH4aGc%@bPVwkW&&xt*|ah{$zbo)!3d-H8zVe`qQvV~V{klu
zaxB+C0T&c8THRe;Zy;S6IXWe@N3;^qD2z0!Bl$4sr9(-A4Zgg9q$I#VvW5r);2Pj)
zx5K|cdkT_#Mqm=)`4P|9ZZB#9B$*tz7~V!lu#{^@M{A_lhk;E7!W!X4I7CzifXM(C
z=5-3CB1i=xmX=I>7YTbpEgt4*3&FH&mS#te;5P`Ht9`1fsvaMMorhSJ1ZMFu0en+b
zt)A5chOhI}eMSKF)rT%aegcpbG5g2fK4Zgod8%0C2ce|^0r?@2<(-vL;gp%Fsf&<6
zOmJT(0Ax!rz*-vkA!lnb_=SXjJ!6Z3
zBKd+~(95}I@f!{A#w;N*8hU}891Ur8lq8VsS7$pfHEramWl=dU{`LkRGxAQgo15F8
zAfD=4*ob-LN1+}APcFWWjt(Nu1+Z#>fjs17M9v;~HIN8wa_c~h82Z10l#O;Nas)nAP(-!f7fKb)n3ws9#R~>sax(@nc8XyaYsDL!M;S!ch
z(m`z$?X&y`(jh=S&!BO}BrIo*z~gAkLfkJvVnf|W1X6Br1M|I3JW-NS9BQF)alrcl
zdq44#`VspV`^W#*Yy3b$Isa+NbLySHKOW>>jU$)gQT00f!`S4S4Q7BY_#@`IBAYVz
zJ$@)s24fN;8CT%2{VXzjNX#fcWI71EjaCt^C*bNeg1i{ZpW1*}!ecmgM#83
z5fn~RY2sr=Y?;OPXD>ypq^9iTom7Isy5e{i4+W`BXz^4d=2p$orUW$5mF;L<3Ap
z{hTr;k6_W+0!*i;r7%Cg4myg!M0i8SCF-|Jl*w4-t)TyU0ki=20exEJh&ccQt(IU2
zo+^?|%hs;iPQ%77!$`SCVhM9xcv23VM@
z@F@L(lsZyPgCdAZm8%0Hoq}ZHAZ|g@({NK%e-|hyDmFol%?5;4Uc7h#vamR?;vr)q
z3$xolQ{>OW!b^lq9*pk>^hmWq8R3zM3F{mB>pn2V!UOAcNhzZLlz@^6o&97ms)H
z`5Ksu_1*25>$o
z*MOuupW*4z1#s}C&eZR|RyBbp4mAbBQSUjK{*Yn}CaKMx(FvT+&Q4BB=7KwOk6W0!
zaIn2xsdc_XH=F+C3X0{ee9dL0Qm8ilI~=en|F1KZWtA{!1_DuP$|{5NC-#H+x;2tOdKvpSJ#nr}!cZCrgNO(ol`wtja?U+BjE2BATOJ|2}MCL
zM|{5jF40G*7aKi&N|!n_IrFD}SoHLbRk??%eB<3)$(Syn?z|ARx+Hm>hUM`RjpRC3
zJT;FPBpmMg2i)U#3W3^|5iVQ}c&Xi*{(yJC$?m%ua%AU#K^=5_je3<&I}Pq-2y@`f
zWsF34%=4cId}XoU34m?&f5&jqzQb4U33Dufb5t|f-FNhS1-X;U=}MDcW3Ye-tS1Ob
zO3Q7;oR>D@ym3QY{5WJ%77<~F@;6`dztac!@X>54oYuJMo#!VE$
z-47WtYSr#kxJTgmWM9C-=Vg$gxKD=CCC~c|+i^6-ne!^8!Y|G#;bLH5d`Gwj$S+Ed
zU+2X{Pxg81)aUZThonDH-u*#@6H5H&_v@xLFVq5HpM`3Fim$bS=|(f*b6#QQ2bmuQ
zILDU}b+A8zB|3epOzDnS;O#}TCE$^>DE+4JeA)2|PIIk>8i(P_3pyOM66Vjk=c_Fi
z%3pii?3D-l&!GDEDWP8e8Meb{sx#-2oyFERJT2%tN{LeG`hMeJiM1i@R^NI59jXEF
z1VIe(i^vt~+SGdao0zcJF*KpAXRZUDB%0>StZn0j(DX2Y7kHrX!v(oF<#EpB-YO~TAai7CaxCqia#$4
z;mo^wr{U|UkjLV?hS$iAIAGE5>6q>3ua2S6z?X*neB@sDI~LJ{U~+bnjmS0
z5s822v06AWRAxWbI)fU*0UNiB5h@)(_2C9_j#3LT3v<$|t*Z22kH<3_)+d8d6Cddh
zRR}RL2)&tvKDVB@pn
zlC}xc>9e4MQv_dt$^RNrg7ua9=2deHdh$G*qq>{!UQ=QM&@{vmxsn&(vW2%Z+3LoW
zBhgc%EkqR>9Q$YBoYI%1R#LRbc-HOl(TP4|l;5nAaUh<-U`k`$)FS#q9B
zje(YcD@x+T2Kj-CRG2i03E)=36ND`$CiXV=-xZdI!pA#d>o=R5&u!OjQQt1+Sle;B
zzb8%%@@#simN_wy;?b-QigNeb^USQK4u^sK`!2BD0j0y$VqzqeP-?(vp(0N&}&6DzYLfB9R@*4jGlb
z*Lz;2=Xu`ac#rq~qu;OY`@ZgLe7~Q0p691^w4=&WW|P>lmI3AYY)>IB*;qq`3j&|t
z9vT;LU_H6rqmuuTd!OVRSXwfeKx*+^{D!~{@{+<1ID-WB#}u-FfCKWNn`
zB=5{Xed1-d#)XO}oz~hJrC!sF
zTzx+gJLzm{f%E$0%S)lW4%AG>)2;D$HQerGP2au9V9i3jeBrpp?)9a07pZ^v`_UN)
zE@)z)r6bBKnfBkppQ`D8iTQ9evxl{FVh-GKHfwp$LK6Q}{5jehCx~aLd
zd%E9dNGB=+6ziVPVa2ZFZJ9J~(y4oA9Eh98TYA1(%Gi2_T;_cnEu*qR@el=*;4L1otC%A$mMc(_qc
zTIb@E8k=2Ey+*5f*X89K`Q*wKDq#3@Ip#kqC~7OOu)XKt79+kO@_3B!4DzGjMJPmS
z-fX<+uz&v{HnIEu{^gBR7Khl(+6z=UFoMp{@CEMVF!Pe|5RE2#hR#WT;-npn_J@Va
z8;oq%{Zzsffi!@e>YeZpBBx>k_^_Y=10gGDBF3h!TGb!Vns%vruDB`K@&U87)@@d~
zUnwu0ve4Xf&EEX_zWzfKkK{LaH}W?`2~hiXzvV39x+5X%TkP|YfwEt#Yoo7l*JJqi
z6$qmbT-Z~1|As&?yeo+wL-pj2NHb-}-nh4%j{|1DkIpIE*ziG0{KFjWUkf4SZb)|I
zU?F>MdJB90UY9*uL+jl=)>WQQS?NS
zaD+Y8)c*{2ZzrlS%G%m@JZo06WOmn-pVgdI2+CJiosli=TRaE+!XwNk-hSv-$HLzj
zp>GqJ{=rm#uaXa>$(G~yFk~^Zu(cX2AB;UiC{fxFMF$FXP2B}P7cT^!s5E|(RIoxU
z%HKb-V5sELnOjXSIIO=fg^T1}^AoL^_&#exbgzG7$F*fhnVoK}){d_UGVe4(@vX*)rd8vv3(Ed6}^MtLF%plH+ZznagbJkCZ>BM>5~!$Rv*
zJhfrpRgarp3$e(P$+{y3FC6`oB`PUZ-N(WthoUF;1T7vFH5$J~S2OlxF{1j3qm$61
zB>x@CL<3PlHJtDILnP0wL$Yv*wu)`15Cj|DFTB|xR=exWob=Oh`5GcT_V|2M%RBzf
z3YwkVizb{byIJXUPsWe3SH7ZrDc-L=Ja{{e7#O$+X8M77wC6}T=36c%9Vvj@B}kDT
zQBl!R$UvPU%KXHqw=+`S@af@Myl#GYWO1b9?TVkxuNW}r{2x5FZ<7;JF;b-r{GXTO
zDGBuwVDWD*avl;R9JdtRBh6Qb*S1B|#E49m-lT`2`R4P7{ue{3Z#-dP4z!gsoQfi!}zb*CA<9=wwuA06>7_R?>@pVN3h--n#ejQogfbYZsC
zyZwIyXaP$Me)@m}5>wnQDl=karY#4)YRirLra2bM5q6s+S|EHUiVWo
z<%Hqe*U;}w)(+75BCVu;y4hnRni{bI_(rlo^Jeb%zCPze!VFrJ({$8@y|r)mPH&Iw
zwp`*nm@M2TpJ^-%k8uSkx>y7y`96ONV$5Neu*Ark4-&)chRFtFDST-~x=9tm@|?{d
zBbg`OatZq~5oScogy0|*g-@~4wsA#Dch|Y-JWJN~@NQvkQx>Oa7TXt7k8GLeb@tP_
zwS%RI@oM``HDMniYM<^@pMU~tCa%@f#0O#T3K!ao>gzbP#u{O$GnWr1q
z0B)GC>mSK%oVp%0{`WA%OdK3(ndOE=)cUsjp?>nCK7A!HLwC8~R>&52FQ5lSDZenj
z4!xh67naKZf*!U0qvuc%-YDh7jp!s`G{q)qPV$@s^Jw0Vk?Hzb>yp!a8#g4}o$k2%
zQMJkO*S1Gn*lXHi<<8ijIbs8U=hS{vddr*Fo@*?p&b&jQ5^5+#eO!#5QfB53jgJ#z
zq#9b^Zuztg^^Zc3QjKg#PjuG1#Px!U$R_fy_(30#U<;%nCte|C?Cj~GM%TnlLdohU
zO336SY_b*)*EZ9&iWs9fbE*G}+-OnInr9EEH_R;FefaPui3%b+8<%NTlhvE`kPyX7
zXQdQI8hrpl8jVISNy%)!s(=}w7NTnvzSv#o^ryN*7fnsjzRy64i|1ppRO#s2=v*@P
zs`ZE~uIcBR_Pk4yQwx5+)#_ypy`MH%$8rs?^XH}5;x?=-(;MXaGs?3jZv67fjDXQP
zTDK6Bg_FDwejZGpjB)dN0F)LXr~EeB>k!7$1fr@$;$)t8ri=Tl1pc
z4~&J`i)2_<9ogZKmmb#osC%j9)6$a-?dkzNizs>D%sv03>s6!K_htKgU%L
zD3qHPxWC#P$=V~Uq%e7=PkXNc|D&XGvx$%uMCm*G$Q8Rde7gt*`&QYJD4+3?;oj8g
z6cr^Ut*HXOzULyl+07&6Z=ar8cR2L|ZP5;bk-l(l{L*97n$Xx`fi+YSY)kRQlnS#(
zCimA+-Vk*B#bQ=s-WN+8rik9_dmU5IP)!O%o%1_T6+glb>hC83W5|z4RbvE}&kR3N
z4R7&%ylb(<@XfpAd>=L8(9*dVA+5@|sG0VqMn3;rI~^>ZNk
zTHbiQ&WU2}2|KbU#`=JV%*x@u74oAUTC6NoKI{yQb+othR&0bsPXEcxIrZ;0i)58<
zPHm3mH=zdcmG$g1?i1=wtY=mh`t7p~I-DB5x?`8G*sk^gWp=7sTHo=R=m9Rp|J}aK3
zBN|!f3p;A}tvb%?dwo}{Pe!88Z;gWqVZXQYI4%7dyM-}la$d6?Pw$y^EkC)k;tUHw
z*n#yvhfojc=L_yA)`=>A=^C+^>0GV$JZ1LbyR?_s4~FDOuG~PpCo&~3j&2!r`yRHV
zvL?B6tgowqxP4(VpAHWGBUzQ%@Vo78Vv`N~jT^?pX0
z=r!Bk2OG2&_802kEY~-3d}d4;kyIS!@NHEQ6(O?W^Fw`fVlo}tInboYKY>Qpe$?nz
zuM>1le7RH?32?yu-)#kCz4EFLOvUdh7pFBVXlqDs&&gn34NU?RVZ{!9WQI}WPdjr3)TldJlvqeXlT!y{Rfo4>{C70}oimaeX}d;R3-7Rmbb
z&H%RC2HFK`to4p5@6e9XIXGul-@E-l$jyot(~~w%%u#m(61usSo^o#$kXre6>Ngj!
zym{r6#YSIC;&=I*rEy`m!><5{PAuIOMfdhbekCTfifQu{e|
zJ~C@pD~)J(Y6voD`N>hnDRQc7%xX~kgvQkWUq8b|_$w;P4!E$)-!U^Y5;!|!$xp;-
zZAceMH-D8hC@DK<;To%EE6wn&jCWs6@*3)}Y@j$!i&6Zf1&DF8?`=YwMUx}{i+`l?
zHs4QI>v#y&n5t(o!`q8Ihm{jQNj`lTn$M6Eoa589zULWi$MWmsrJT?DiL1evuJaW~9Md~P}jNb@mvC$jW0SDDZ+`Ujb_324v2;;DCEarL8GQAh3Su|bFY5i-Pczp`0rf&6DaF!#LLRcb{&TCL;G0?Wma;M
z9-uF%mYHDJDD3hr;G3=~$nOc#Rp&Z)N5%HY>T?dNR&2!Dp-=ieX^Agn&pcr#f*A+S
zTAm6n-0)D>y-KNyO|fUU+iT`gp?XbIF`;r@77tS
z>sq?C+0!*zJZtAE)OPH(NL$-zYL$lQz~t*W;sUjDoV9(U!|2)el$8WOwO7jO#e@PM
z{~^{uB4tIjnXNL=YUJEG4Dz5rcj|wC@r<+d79bFpU1USO-4jBN(sA>LV{bl|8i;)9
zVAOU}WY!DI38z$4Q{>v5aGE-92c16K(BlLlI)Qs81G5?&^LJ_egkDY7L^O${Qz9n(b0NG%|{cWSaL_P9%!`RSrKzo6lz=t6sdd5Ap#F+Lj8f`hX=RHAY
zwp5DsSv0zK{g$UKwwyL0M4JwT-Hjg3MoJp$CnLRXP{ux)Q}4{dbWLKPYlbEFJ+iLa}b5BV+$Kue^Vi1qUzaLSm
zx<%GjfVD$&bM&A!af@o0uDn2i^f4qWh@&PwPPJ5b(IZez`s9YUk
z{CQL3D3x+(o?XTcu9KD)CQrTYHF$8l8z%gs(UeQy)3N=2)5UWwoM#AqYDc=45U?}J
z{y@g3{rw-;{ta3Xx5QW*LUW#G)kd~>wFW^EeNmF!|bC_f4XlDvQJ??XPg8Z!;lRmZM%6`Ale(H(
zVO6MywLq}7ES&{^QS6Ss_X>;#r~Cq()h}^8xg!6)>mx^zE%A6<5C|f&R!ai7)kFSs
zyz&C%WU=5vVLf$qA17ofx?}JADO~6g_o=^aVo~X&`ADEPF2mv&UyITe-D&o3e243Z
zd|@+Nl=n!^Y9Z@?4wbNfczpZ{vQ`pVqd7@==a!#>_6xqDRLi|z81G8&XTBFUZrw=Z
zYu}kn>#I%doDCy~YNAV;G5+6UymJHDST<|;Ra2yum@}ZHqT;{TZG-PQ*IN1nm1A$e
z?)D&m)P#^&v1@<6ALbWbUe;j1TGQ`DK=-!2-8@o;l6vCVXr6vDi3oaH@}XE<-BV-~
zsCw|7@0x_%zhO5@^Y4e}>*#N4Znl718|`k~YEN;}AeJKw
zOyS^DNfSXEeECMa_;&AJi=5f=_J8h|&eL_kOu^ON451<_iI#)R#uOv*lS=x=-It>8
zZKYoXqb%a>U-$?4WRfzG0bRpxxEXRrTRX@&&QJlUA|2J2qGA*4)4`ur=w1JxgB#e<
zPsA0RcoW&cjkacjIG@XkM!o#~V20gCEqD2)pU7VyGFcoVZIEnCac&jvqUxTQ%^||X
z>vxHwzJs%2un1sJ(t|VIyYat=Hb!y-1T{#k>v^*&h9wF6&Nxn5m8B%yU;l(S|LrxS
zr*4S&8~sF#|9{;g=!%t>Qc%(l=!NBZi+yABCm)>cD1OoRM~R>{ML^X*EI_7BjbHA+
z(?or}WVZIli5BWdB^+(&*~SE)Orq8n8&LxW$y)P2C&sM!;!WnJ!`Z~{8{~mkKFjl|
zQ%>{}YN`Iu?(!GlxjL)MsRGq^mldVxxQ21-U5ak*3PY?Gq4(b()J_peT`gF;L_z*q
z?bbJ&zEURaG&_Q5uJ><2=g*I_45}x8SCvt2n3gAwSEX>rthKN_<&@=kfK-|xBpup{
zpB%V;s}}x+#Jzp04LyH?|w%rv|gN=-sTtz9eO+XA}
z8h+3!SC0Y6Pz-!c*26HvatW}a_hwJ-Uc
zq|`3o{kqEpZw8ctb9v{{Bh$_{BcR0aOnB;my*b+P_rVx`$=yxb2DkINvI?8<1S0N*
zhF$?4L~Q$?Bl`=`CJ!1(kvn`+gIogJVk};Q`%dcd0^4c*w2BOYw(;_o?rMLkzQK&E
zco4;8bj;T3y|=$#OsFRQSk0pu#+MxxOE4h7WVQz}Y7H~}^WcR{n9|9ZaCH`eY+UO!_E(e&Rh^k%7AlKi9*od_b@563{Q(G5C!Q7?AKb#GX1SST38ks_sLo)M-??+gH$Pv7B7weW
zlY$2Eib96Ah=F=RhmDL0b+f?jQ(P?A1WG7Sz)wIAI`nCeLtBb_d%NdY;7JoalZJ1Y
z#7>Vu_*FG(VX^1$XaNoU4rBsSE`Uz@vF14+G&3@CI7gfTT!{4oUTGzuch_uzioQn9
z<&$VG1=3DesE2vuMw<%{L38VbP1s<1>{#U+UX)Hd$ExJX;WKLE4@QTF@32n+Q49&3
zcNofc6}28yAVF|f{&mA~h*)9Mc1+1AT!fcuGm1s~Vu_b>jvzE*=-V9qnDi2@h219|E*!a;hbCRrC)G
zWJOUTZuSwFI#d^_BDhTi-Hs+fdw7X$%Cz%Jve9Xtu`
znF!BxvN?gxG=iR2@@I)1J7@s3e~*W$&f|cPmZW{56sXfM3(Ok`;Rk@11~T~y3{s-p
z7Z`d`hqip{0cnz%pLcQS_VBP{Nl$McTR!qSPzUT9e#B$OHVJGQu*Yg9C!iO9_z>P?
zou?Hg2R^l7frhuY96lKUWMhKy($PiohGFnB2BmNSOz79;#LLU;>h3OZf&X4u7&Tb6
z3L*!gwSW{(P-QWf{BeUNmG=+85pvjGdihdJ9YJV=ZwihY+
z&wN*)*O*wjUorb2Il1yAp!k7~y%n2o&Shm$-_eZMJM&8v!2V8XGQ7hHtQzd?=~)SG
z!aHQySMjhm*~boDI6Q;^;~fmvRpg!YxP4;A?RY)+#Q*Mha(dYy%=IPh*y+r6U;-0&
z-UMQittaCR&#jiW0uNT7ncqKt@R9&v0Cv}brF{jqmjN4QE%5ZQO;00aq5>|5ErNIQkX1b_UeI{0x&>@l>y%0CocOl>oPq_l)uh<)>$|P)}zixd3Qh
z7)%2gc@?w@I>1oLepKLX(G%~elFJS38Mw5xv_4A6A|yJ2b+Z5X@n}bo1xR7gcR;!(
zProU>waIeC-6LF{f#=vF=zsyj$>2!yLZ(R#k9&{v&HGVNB7m-~gr{A5#@;^gw}i*!
zPtiFusFc=Rq#&LE2l9?2!Jh!^{vG<>S796(;JwMo@7#kQgKQJ2|80{S@22d%__7Hn
zDp0s$cCR5C{y_Q4K%^oq39$6cUjVQHC^HKlN
z{rfi<@Q!c07eP_x1sqm2D^Vx*+Si>6krP$>Rqri9HHh`K;$d}lF%L5Yu-A2_t|Pe6
zU*$QN3RBs41YF=<9z*j$KHnlD&5Y7a$I^iD^E1
zKj7Vjw@OPlcS#8!eOYW{a^5mm=2O(w;GIj?89C`eatlkT1DM^wk_KmsRH?3SBj~#N
zrbcjXNEAa-zCF{}+zD{R`k)Mu^paO7RL#ng^9Hx^-~MBP$?&qL1+#xIAz@doB}pHM
zxmdo50*J&N0fmXs0sDO}1Kbf9D!~`x$%d)IC*=eM84!)V%xOix>h8UJ*7vc3LFCs0
zEC)-2+ZJ5ytynVg1M|IAQ9-Z{>a}p+YBJ^^d{Tk2nlU*3{~%H0>q29LKQX%YPAES?
z%AZY6POc9U16&ybVH&|8gYz5w$ibiz$OagjI-mq|==;0S`n?LIyXb(%;kkm9>vn7m
ze9gkinV$1IWAP|o|L78Q8Z}7MsagXTnB0{wpNaZIDc7b0=`dCa%
z%&j-;zylECu!d}PK$N{+`QV>hwcCsFe<`58lfM5L~u-{_~Bq?^f#pJ{?egbyrniTb;pP~qbq`$rc4G5uk)
zgGw)Act;=||F8w^oKw&&PLjJV(WA>TM0%XC$Fdo|e{RA2X0+?c5+O3Y$~zYz#6faW
zEI-_G8(7eKEd8qN+ae5il@bSr$aVn8QwqBND%-iKA?34Y(>^L3vVi<=&u>5mSQ=Na
zS`}J>aAXr2d4`W;=Z%*Ba;&@bsP|W(_#CLCzK%yN)$oD1ZOp(%(rC?KSnlqE&m1zY
zB)d=bLH_>s;2Z0B)!GW5uGc4S83!KG1*J^&G{4^4@OaYQC8z0N1~rnKZ@Z$95dM}U
zO*e4qXqbeikcwQ%Rp;@n(QXZ;G9^6?k0CJlo2Kxfkmj
zg@dN{A{{KhO=g8;k;?*GjO1VLMw8JN(l^C`ap@aqZ-~d{>_Vi`^y>5m!!*;GitZW^
z@c=LeZ;V~ai3(A*o~aKA6~W=$h>l5>niH932p1N8JT7vw8@TnIo6mme+nGvXxmCeW
z!N2+wbtvDqg4FO&bbC`r_rz^?kgki^-NB&yc+HAmTL!7~Yt{Hygz41m(`D*D_=4vk
zL#p4}=ebH2RDRn;>6oe3u3UYzEOJiyW#^-eA3xR0c08<{5<7otO2*^!*UIysmN*@kn{g`<|bKsenL+;?2
zrNsqBE6lAn*B|-@SuN-QN+fw*TeF3&VG@f!E9F}}%V~PR-Tdw5hCF}lJ0Y)^@3X|YHEq$(LZiQtD3e?Gh6(|DwxF|zk)~<2e+iWu;~|tD)fOW24WG!HM$f?
zTz~fFGZw{q{Vz?s1lgd2W$C>1VzPL#u>NjK5MOf@%TW8(BT+tcBS>MwA|u-pEGPPE
z-mfJd%lovA=IIe2yl)D*5dA9a96@Am-b(19O90Ty8p`iw%3KO5q7hj%OfO`qA`oZuZDJ3Q4
zFwE_V_j!$jE~|ZiZpjGF1O@I_OYT^n{1D2k7KR%F`-Rr{pB%_>&HoHX%cOs*uw7HA
zNx7={vQ7L@fYRS*@~d*O`6`spzchcKAgXDu
z{PeM+!g&&Rj$@2wOnjuf8kAD(`Q}jjc;z`Wn&801LFPGuARSID3UAu9sr>V2L$G5)
zA3sjTHV{OF31OsW2m_)1;POk7MeEaIpbCOelBr$*`siva;O4(zUkiT&f3oJ~BGSUw
z9CiI7^R%nk;=tcIHud8d?#sj63lG0bXI$qP*;tWzYGm=4|HRBt(~DTE1rP-T<-%v=
zq=M@UeU|RH*uPma5q|?tkfDzm1>yT?$t!^pSHLwgKigRN1u~2$a(#-S$dd|rqeJ4Y
zL0>@!76(l$T60F!@`DXJkh7hPK21V;HWE*Z8)7Bf00
z5D`K%qh`WYbL!&JJm*BrsVOSiXx(@eH43W*WWh;kX}3Q*+t};;TNxt4!uI1Y-cc(|
zPTBn|aNhPch}@N5Zt!>D^n9s8gM(wRdXJPNW{6YfaN{1ZggmJ}L}M~TQC!smTP+Pb
z+3Z<*Z?dxX{2s7|eA@uzrrytD(@jhjE)|?;m7C>-sXj^jLhpx-s_M!O7VKu$*)2}$
z+7Sz0HLic}8t1O#zE%s^1c48_eqzT9h=^o|US;qFH#~*pdo7p+uJ*P0P7wY+R0=pV
z1wS@f_e==zSnIU
zF)%bd6c1esPG_2*?RW&sSuZ$_{bY29?Hlju7aA)s)d_j
zjmZ$(?nrUz>>h4`;=Xq2%4=qSZo@qLMbO>2J(ok#uPq>V{#s)O|
zSsLcR_8_%4>?VG&5=74YW=;HuAa1~v3fr)hfHV12e!i2D`PY#6(>n$0c83N8(1WST
zzjLq`WYVk_#(w0R?ueHzKlAbEdvbl7bQ=mq!h<|KJk5D7r}d+3p?)LnJ)eV1fmTft
z0&IR+6eQE@dhgpdCW%xh)Bo^nN<7>Tb@6*IH)w1bH0N%
zypgQ7LErJZGvJGg5}2N
z-T>7uOlB7b+CX{c$G~XMr`3Z%-B#OYw;noB7x0GRh1^n=++2#)Nlh!Or%-xe9{FMN
z{P?AvX8BFe+knfcjFN9Z_s!~4_|Y$??d&p7ehl_+gS;yrpG{2Q3WDne6G4H8WnaV~
zk{=DHw#Uxg;|DnoG6@UnO3M#Cn)1rZ21M|FW*O~`?@~~X0e>zlRH8R>{oR+>_oe5L
z2H<;No_iy@Oad^Kc}p^ks{79ur9xJ;j2!i${Ce*eCzcR9NFc3NSGL0z5eHIa&-uVT
zKU~7GHenA<$|Jodk1q5|7OtG}hT<9>+g50|jYr$&H0QZ4ksAq895Q`lJB>3{s$|9v
zH&snmh*kBGX5#4?WN3N)&mOX87eawk97$w!)eJ29^;M^kWJw^7N}#;Zycbs)Z9kT<
z^N416y_qhp>SLa#YHA_%2XwD-zt6{;)_@b|omjd!23BdHTpJ448=Uitf+u$Nt+c;FA|ua!wT9bbZv1rYVr!4F2ns>Bl5Ts+MTt6fmqmZa=(#4W
z{_8qsW@o=nqT2kykDB2Y%Ilvd@hvg<&&KY?(1J+lLRR`u3WDQ48v(b%TJr{VkC#6i
z;WKh_KI4v9IwRrlleMn6Pe$XZoj;OOQpBOXcN(C_oi^W)kAKHv(lBnMm>jeLc7r*}
zD0^_pUA_)5=xvL@c#|$^392N4XgO2;5WgdjBwDlL
z_{nvn)>7D}=iaZDr24hbwwBux#MBR%2eLT}OE!Vj_xof$_%$Qms*pF*HKrlb2oub2
zydUD-G_jCuJF$@ZcyP}ATjNxz`{0~4g)c!z__{XMQ&eM|*DVAP5(7lpu2q)v(<8U%
zT>5KE%07M+g&Vg(ZcS6r2H~IBxVYjz-0ter@Z69&5c6w0jsK)wy2WjYs5J05ad8wF
z_nHx}JeR@V=G@KY&=}~S9-|3Ikk7FwH1>4@7jrtSh9|-?Sm6+K;Y~_uxsa9U{-36WwT*K
z${hjhfvktf<_;lCyIc;$=_yN8)>pc~p{uZHFcYHG?7l~KlYYHR86`Ec6bs{bumI8~
zr^8@_^}SJmXJuhYNJvnDF%NGfhZO!o$r~x>gIy=~SHB_MYK!{zjtzy=2QxGi!yZ0N
z-h4J^Yr_0nT25ynMqZS35zFU&zrBy$99D0HvXC_r$-&N7U5_E1Zmj94d=;q;e*{h4mqT1OzfXj$Y|p~isc)tw6b4yeE`=eK
zzqw~WcBosc_uMb{@fzjWHW69bV9Nmigj{M0~>nRgRxu6{W*E
zB*+q|neFlL!ey`Dh2^?M&hrS=Fjj89ePO8W7+Re5i1}QqWT&d5xr;7t{qke!hx}45
zJw5%lwzlshe3Ftzkk_k$eT$hrio`A+A|&SRb8S=GW^)<0NXMOcf31cLJKeATx@aZ0
zm(UR2Rlh57(Jgr38W&CYg(`^$d5pZB(5p+rwzoQU
zDhXOg5B8pmLB+o_ryUHd2`K0!Mn@~iMQLGCk>SjsdCi=s9Jb+M)RD16!*uoZBv8Jx
zva*sGqu5C)M;WFh?@pd
zv)gJRNEHK$esWfpArdxign|%&4hsqSE^SPexk=X9MQ?wPJ(@6pkK1y+ey!
z6YOI%o|9kcJcD0yAG_<^kXf%b+9=O9QMDP~%Es<4mBMvyiTQ{oM6$8yLE5c>`Nm7(
zArcCyR_8j6-WZ19YXaqjKZ4QBE9$*rH#dmrZvXQko|B#t1hHyLRY_4FF<0qgt$Lu;
z8VV-gaBbFSiYxwnhVb3<@kwE3l4$8xg0gz8K!qS~WIf&A_lU9=g_$wSZ@t--PEtqN
zNFPeRq$clq*WCY3p61SL{WBxGR@@x6mY%)CMqR6~_V8HJDh>wc&
z*8Ze8!CJAo80;)CY~kYa3NA5`XZf7$jlI3!$j(bW
z&0Eyx&xw-YMjWakTr0&l6=8%epn)~{=VepJKC3!)Fz2*(p%Cs=uSHX&30oSvlXlDZ
zMB!m?wN{?iTd!nh+%YEgYX`oc+$&*-&8O3K^p;Ppnfv-;%|gs8%9<-dgyKDm+R
z745%c!u){S@VH=awa-9(w!pX3igdn2aj;(O$(AIC8{B#9Qa?L^93Z+mxf)1%ySBUZLyYJ?^?%P2HF8zb~oGp6ejc
z$dY#I_6O)AQDgb*E93L?P0@y~2Uq3v&c%#4bG1By{($$EskS4HBDbe#{w|uczLcwj
z7e=y!chX!)NJ*)OmQRAzzeCu<-%k$zh#8nweLSnRVoK-YDvw9pIwgb!*DgxF;g-`L
z&H}H65)ak-P_02=!we2zKZHQz4lOS!?pyHcb^y9fcJ&00y4X7hE!4<#@ecO|V9OQ<
zNVTZezDn!}*0|}#mNOY_7BW6`=`uJ1W+-bsadL7>&{;l_R5L}zOThi|L{(YuTvV!E
zr79<-wY2_7#S`r^E^_-92cK{k7wHXc&Nnq^oey8F_jblpP9sw0C}iJb_AH-GrXBuP
zi@@(!XVt-DuZMyPs_I*!?9AeJZTBTtlX1>)DF@Z#(DBR}0#^b#nMVQw0tx?aXsRxD
zCVXamZLRK{x>j}y6Q#+@uLkHfGW*}Y%r>~^1u)kgy3W`fl!XoUp?YLP|jYild4UN)uv>eSTKY-kzi
zP3y(vc~`MXRfV$%{Jb$Or=a?Xk`P_Lk)nH2rp4)C?Y`h^sso$$+8~cH^O`Vl`PFQ2
zvCWP5)vCXz#C0~sG*=gL($bL7U3hvqnB%UqPUE{sZS~^q-aqk_#Cf$s)r(kFht$f0
z>m{|1n3yEPM8R&VKtU@Tvz-tBJ0}~_o}QiuQs
z8(;Jt@^IH3V|^>CTGtoGwFXP8&t4pR8^kUZbI;6O6?O7jIK7?e_r3}*uORU#90%2Q
zHH1%wCr&)6+mWt5>!2z}tu?h?V|tw^wsU3W_u-AUOea;EMf%UHUR2B5>R64z4LB8B
z#GM0m@j>veiMGoRRFs_ey)Ygx-yoQtAA`@$ecHgYa4&8({hA9cew=gl4w~D2-;Fxj
z5#sAS?tf1;4?e8!v|!;KdFI=WMDN13B=w4=%Kpl;uYc_!MSswL3@$yO_s?qfBZ)!7
zRs(+V3mQSKd*<|Ox2c+clp4;pGq$s@I?%d;w9PH&=zSmU=gKDdzGfC=dwT`PMJ1oL
z%i)3ucLOFONI=A`)#;P)qve}G1RCn<#x$CFXu#gqp$gx!CpkFtP_1DDwREqJY_D%A
zKVkRMw&Tyk8WVRg%e}6k8GETBXws{WTUNlxrozsZwlehe~s
z)bg5oZ>iS@#hp>j;9f|0Q(=Hwg}6mA9Xf<6q!mP985R+dh&q+H5S7y3-%_wZ1d<2Y
z?Dx8chR@I(A0Pu3*im4Nq6Y>B9Iub4&%TaZ3UfZ(`1{+s0W*1{V;qlvQ?N|*ao)g<
z`{ekL{MIr2M22eD+?bUg3BL
z%ko0=FKP^?AGqhEHhbu;UPi=$&FAm0vzyJMkFGn5DogCT
zil*Y78QD6X3B6Y5b6(t5YO}ZFZa;gvC4M;;P_Z%MUig&hDVTV;_FmG%FTWU6dtTCr
zKJnu`@Z>#bJb$|#tIWWUymx=^)6E@J^fFo(__Ny?%51ss$gT>=l#MG`e&TB{GnWi%
zKG8}b%52Aj%u$AF`eeCwQN`tP0>PxUI?deWZD9L}hun2}jrDZy8bQ4!+K$h;rLd?d
zf_MXb{#{|F+#&zn9@Fbw&#tkXUvtx>+R}`1!=kAq_T)%)kvoetl@ps;1osrd_p#oF
z#*ecw&4-&Q)^~Z;eW~;xe&x8`=Vz?XBfrPm+r8)Mw=9}oq{EZqGGjXy&s6sPWvbq=
zr
zLj|>jgK!%R`g%XHj=F%aj78N;5eQdd6j|1#BSC;Pdt=e<-~C5q{q&4B3ysIN^sp(IzPmj%~U2w&sT3z=9VN`_{l?D3U?_rQE|;J
zs+U1DqkMZ;IkC;u-Dg&~BK!AS3KqNBBbCJ{%lNxTUZ1HGiuJDOYocNrdUU97?^UTb
zI?e;alNavpSq^qo&Yaz7$L@A36T(6P;
z`x%9A>&~)hcZlluZ7)$BUZ+6I>#M{hP<WQq?(c_eT5r*rpJ(vh#?LS%SLyfp
zFma=pzwE|^%pl(KspT!sv~-Iv`-Zc=cEW-&%j5gRT7w*4-r(x%|GgE5sCqj6IJR^6
z?2?wG$fS32gd)BGhe+NA-KKd%>ATKXbZMb|<_JN9j
i>aBmz?Em{G?-JwGW95pR^hN{nbE*e5mD2Vf_xnF4it)?<

literal 127604
zcmc$`cRbbo8$XUzW<&@X5wf#6W+B<5BJ(7h4%vI%h3sTy%M79HeY))&duE616~~_6
z*Ez@i`E-9Czu))YZ;yvZ&hdV~-mllSujh4L?_doz1!4jk0xT>nV#P;tT3A?D{IIZa
z+VQS{Z=ekP>EJ(XM=ga1SVi5pmca-7SC4cZv9KsvP(Rp;T6cbf%L%QXJac-cssb^w
zgYg)f+PyI2af7`AS7TvGxIw_LFf%7(MmN|?TSthSBoq1y2>2cKH7^q*`VuD_Nv3D2
z8jP}b4rYu(Jc2y;n4}0885t!UOwA!$at|+V2cIOFES;QQL3nvxU0r!x1$gWnEO`0E
z#Kd^-@$>TYbAv0m9o=o6jNQ0x9hotU{IiannWKq=)hj0}J6lH7y2dZ;oSh_@m{1S;
z&p*sQovh6N`y^Y(i(`QU@}mC2%g1w%_djccTP0B6LbS{r?Or;gmVat%i|haV?tib>u(Wfs1GILq
zGEua3GIIbAySV$G4gUXMaj}*JFN$1$kPpKf^tXUHQUnsb|6#NgL12H;9V{#vEJeBd
zPu#HAQt&69${)5VXp+S;3dyS8SNz7!a)Y>uTR~Zd%aNO9N@|^bEJWlCZbhi+5G(Sw;
z_g+juCMVcmyNZn`gM~xthlR_C_5b=;8E)L+PxCp#^|@@j<*E$Vy8{2SE8^SlY#!xF
zR%-05q#my=i{Uc%p5fjkUp*$GCSOHF_oFs1m?YB{8M-c|LA(vWKJHFkqZjDrE6eMB
zX?eX#^v_d%{*4!Rc{`b;2g&bReu6kg*6+xPw7b&O*RV;s9}!{W?L5H=3m#SXc?WO)
z@S!V+6Ga|KluTPh=(;JEmcGc=wh!;|=Q!@V$k+ebA*r8CJ)Xo2VPb=x?e^J#5oEpK
z-j4OThCQ0L8gW=AFLdQh(RB8wmrLhvYta_L%JW|Jq+XoI=L>$>buv+2QUPqE3+ituO~XA@G7UD}@DG|cXx;Jmx@idN`Py;Hb)9tLHK
z#+-~56>CehNN!$+trGLRz}@A6w)GMOk8KjH!d^D$S{Gs=$!_ClZ(aZ0RJ|C^ryH_G
z_y3V4|I`Q1TS6#oM@!5u>sGz`m?Z9MYeesJVC=d-=~_FFh>vgj+F^0#4|
z#J+m#=iaCVL*=+_tr(F+wxvDsf8{w{3rFk7qVvN{j(XZgQSAoAb;Qs$wszMcZD!Tz
ztR7E~#13<9k**52?BCGx%ZWG5{J}ZK-e)+-g)YZvHg%VHs84m0w6Bl0neu7C<9dUi
zX7ej5UBal-UWvpgU-<2BAAZ~{iXlKVb6qh#yj9*Jy2bo`1!SINdt0BEx&B7=OjA%T
zJ~?X@pGj+#&3$IRvaPShFJrZ09L|g
zG|g;p0qtM04{oFMoG}#0BPDe|`a?P!ndOZA*WaQq`=33seD!T5T6!1}V06^xyiqow
zc8I^{f0h0wl1pDd_ZXwpaQ`btSU81`o^Q>zY`h^?Yqukb?{APkEK_%LZwaOTt5f~EktUgk
zv;2p4T=YJ>SOGs@&JJR@Td0yw+uEXXE8GY>OEF(jn>8o+Pkvd9t=r{CKT&^89S`
zyk>UfIGN>k)iW|p9kvYJ0R3uV&7nKHti0j@B1E@DdsP$o
zTf%ShF{!Wui}xQ%@~>dJ=eVFEVml#P`TXy{swN>^OQ%M09(@Fi`j<@`)H#-W3zTPP
zn9ydZj0U^4S=*DT>?(%S(K46XT0&~f$U8##bv}IWh~e$4up4pjqk4C_vgw+5)oL+(
z3Rbh66Cn8KNl7hra6&1$9ga8QaL=FqS7&)9Hps92elh-c!dBafD7>YwARJc;^FRvl
zzzPooK}z%jG%D%Xb1z!BjY?;eW6i?Y^2T0l5b)y9^(^&ePQofD1GjSdbAJ8xUmIOr5qjZ!u;7LH1nb<*0&e6GmWY@dn)u
zKIdQE6~tEtWj
z_V6Z~IEH#ij@&Ku#`O|``YB7((_myol~p6x{_BeN_^*-`+DzKdx02F37*^-^jiq46
zImJwTXK$Y1p*My(i4%=Ki*j6*tzE8MMFg8F9+VsbLyK@^^-B|kMErg?!2wS|1a~?>F_PnESA6Z2V7#K)2%6ST6h*2
zF}E~?E&3P66*T~RJib5L=YW>bPtyK0;1IMQn@nqGXBG2j+1w7FJX;#INL!aWnIQ>M
zeDlZl8GE;Ivwqkd&#e~K@~o|zOKm5|*S?n9S;4w;AHCX=1fWpLzRL1-429IvL+roq
zDV?MwS`M$Q86D~=-tqN|zgoA0eD7+~8fO1E*E-1&bLtCq@nH4;YSLC?HINgK0$pJi
zchjwd*biAxYcFL2^Zplxxhe9%{*|V()%&Bk=U;UFt^kj9?W-<|_0DHV&Z5CEQHp6&
z)4_Vy#7{~*yADTG4RefFy9qh3CvtmY&p
zyCWeFXKZqeCNJGVooq*NnGk(4r&yT;d(Bn`0jvOIYe+$%GofhGheY1Fo(Q9EVjHQs
zFoB8V>GQLD!T%Ctk6Zr0EKL_yPI+u;ztY2Y3Xiv_YxdX`yM1S8<=quyGz~2+z?0Tb
zb}Yp)G)$5Sw%@Xbj=*d6(pVFRAhik&g(p$PO)uMYjm6i?qH7mZJq?PD%Evx@zT@>@
ztAmYKChZUGTTP1V)M!_#r1)Cr4F)lOmESW^&-=v@>7)T@+I6EmrC87VR5bl%M(|^9
zQnelT!zkVr;%Ws{yp|j7FssUI8KnwYAlV8C!zWA>wP$KgBgMmP_9FOFm0@_Os1T9t
zwDi$vt%)O`uYmcH9L5or0`uuIjLZ_jNS_c6ORM~-VdoVSpQo`;ZLms|UYy%B3_s9q
z3LZjH=Wzlu?kpT6?{bTGS_`XguaI6QRNtKA76&6}j+kVC*9yV&4&jA&Ay7C$4
za4#UkYhCxuk<+v{{r*Ou0)w+nQJ(U@qGK7xJ727aI7HljKNs;mJGpllTEM)h$K+8S
z^$oMM8Vle;o_d@Q#^rG1^0&6kdFjIU$kHxnIr?>^UVai9%PNdZNAF^OagA^yfKAhnk-q*eoEv5uZEQ(|OEj?u&*=2YG#xtHg${b5ZW85Lzw
zhl((A#@&cuzB5I9gqS*`H24*oRsZBJGBp0~$c5^UvO$6IYY1*2s00|^FS(%>KYzQ-
zh+h23ebnNA&$IJBEyl@rxvcRLZ){vg??YC^Eitk=bqbQ`&ws2d8SN+a;r+%f(ScH=
zBCt->7uKl{SSRtLt$s0#b$Tz;T6I@kMYX_>gAZ%eRIqEB^W`}RPe~u*|6#p>s8Hi7
zEq7-ZncYy}$4E~)3_pfuU&+vB{Mx`Pj9SRx2;sX)YLG5S^Jp*RScZ_SecoZ?J6oDlq}0);
zf%A3l%87p1m-~&W(|;A_D56dTiAPg>xEgnf8Y6l?iP0yask?ZXq`yqZ&AXmqYy6z`
zZz>Ysi*``^eNGqjt^e{X=1M?3N@XNtzJ*Y5PW1SmA>}rMNEkLAM$)G|V(!0*!&|@e
zUVJgxabnVSdY#_=S5)21Vyupl;*;TGD7@~r0R-ecE+E!FJ>Q(=F5vG1u(rRn$ga(kCZBzo&t4Iw@C5D}AB
zB2>1?V(YPAJH5U)W>YLja~(Y66qsfpXIZ0GT-@=)t6FvkrmFqUhxJUl<84$P%j+zg
zAgg&gQtNu#`a9U83<=v8r)4cvpotq0tp@p?;ykPyEXcIWk{&CCN-?L8hASsBM9F6`NrdM*SPqACpAvQpY`33!Ab^W){nz+6M2x>^(nGYvdTY$zKstN0zx1_baD;JdxtPvesqm2ZDL`+}DM|XMh_^XS@Dr;Nw8Q2O0ShMuQ}y
zF^Vp*+`DQ{?sk4W4Sh0?I%Gzf1HSL^_;g5^bj@NQt?gY`o0Z)%Vap&w5OVdp<7&0a
zjGA-`laIe7lKGITj^M{?5qb8v5-&$H{~X$g$n2B3t$l4vj)A)=R+U>9mJd6iV1dks
z;T?9z6$^=@P2U;=2q%4x*B6Vb7ienf;1Re+ZZns2i#r}vpPwDCKD#7;q0$#B0hrB=
z+fO9p3wBw+!-41(uo>ps?-b~H@^tF_bo#s+^@m{3E7xv)ICm3>dIU1mB_|y7w7hon
z=+gyWx@ZdP*sMh+3bfTuRC_+(5qnrD^MXaoGceU<%*yufzb>nJQcIs5%&9?%zdy@z
zv^RCCTFEb%NcG&%9Waq}yBK0m!*IY7{Y^#ohA1ZBr9S^ZI|z2w2q
z*TEe7)4sm0qh|5d;UZI_QMOS)sN2@RG+lG27I*n9j_eee%*#z(DeL)EZH9=eF7}$|
z-00veF3YL_3Htd--4=qT0R;AXr2lnMWfK0IKKr#B^YKugGRNcfO5Qj48253C|FUA#
z+!i#PhmVPP*v36^JjTzup$NeG~jO>TVrn;Q3*X
z?}83tfR)vi{NE7Z>cDNy6$zq7gUed_-zd!X@Ht3P=~sFlu6B_@f==>6SJL7_Pg)uk
zchr>AfNIL#eC#8>f?o(Kn8_ZHt2<-GNc&;Ef*Ri->*$7I{7L=kBkjtwqtzfq`5BKr
z7A^@}1TcK@i)0IVtP-z&%H(qYO7+_Q-Uzd&3m~M%X0L+`ZXc}{yAo3IbRzpl)LvgD
zanXmB$BvsP!6pEHw&bz9606ym^u0&w_~ont6L$*KbG@99nvz@Zmu<~5tBB3;=abaV
zw8Ksd10AC00e?NlIa*OV^*GsTn&jr*V
z%UKy-R`@k2tJz6RBlzhP5hudSj;KAHR)l9UHg_WbX+eJoB3Jdpy^o)`eAU#<&ps2v
z!Hz;Y5~RF6wD$~{kI1z5QZ~JEKS`dkD1FFY`5o!AlPU4oJ~kQHP>R@@o@*Va2W`^b
zhg7%aVI_-ZXT$n-DM=FU4&9vDEC_lr2h)tt)B@(ekgK}qYZlT7_a^<>qb$W}vUzuA
z<8InS0tdi5)*}68&Y=K_j^e!pWNY@s-&Yxuk*#fVhQm~cQKTXdwTWy;nrFC
zh!JO~!#6+d;O(=$O8ESyrnmIL-y{xGj4@TVzOLK#U)fzZ>-Sk#-Weg+?R>=n_zLel
zKH6E1K3>s*C+dHFAwN6i@$iM=g1Ya~gi{m_MG)W)hz(@tx?DK2ukkS}V76aIe8@y6O*RLla;s
z0$7KR!43?E?Fw3}A;dCYx81(8#Agk&$V25TLKKZ`dNH?zVcBn
zlsd*2x8KHP!oDqJy!v>fcG2{n^_$S+-8ShnzS6{p+g+by2uVMY`ybeqsu^>tDKqY6omq%3IF5dowVq%4K&*XFU4})C2)p{TyAjSik9QXKlnHY{I1@
z8Gg%d*1q_0tD}9>{d5|ft#by3c*BQ#)T36W#q2g%92VC*^ps*^H&&Zb(Z~{we{QM=
z2VCfb2z@qXR^e2cP+Wz-vSYlVOTzKVqI8Hfpq
z!9BqeUgJw;DSB&lAG-`4yM*i=nr0&x~CcRD33P@qWg(goSNmb=bOr6F#j_U22+$+CcT1D
z+itvOI7z>Z^r?3AD_b|HN=dfufSl(7d$nS>=Qc3V>d4o6pYgTK5u%qSV&f}?%ox{b
zg&&~~R{cOO8&}WJv!ZrFP*=Q#GgO4t&yiTh%U#~z#NhGac+>Y>-!B&?Q}^|DF3YIT
z`Cjw+ozut?ZfOFM&)1TD&kp6?Q9)nZDF58lx0ouZ{$Hc<88fO~OQA!#+=C~njlnL#
z{uQm_Tg~K-uG8L?X0W%Lv3(PJ<95&|=UNuiW0Hi0>Fy;`eRU+*r;{>prYc
zany5W9ggeqf~iy01TpxC=JLYtP(i)yHGND!sbhv|=#S^C^+
z>C_&fI~3F*W#ne@so#<|^q^-#&RNmhI!x
zHL}8OhL5a8LHdzw{6!I2>v!ydEK!-kTAvjuCGkx)&l%mcTimK!`WS`f&yV|3g2-#s
ztX~`Jd&J`b%2!{pEt`fYo?~$!UsZtuO5Oc*y-V+m-d#N(+@67+%u<=zAMw5j=}tbJ
z+HZ*WNHfY0;@y3kc1|WF1XyvtE!g)^^=!aE;D~IPQ;o0o9?x8RE}+c`A_hc`{B`9B
z<8@Dl;q2catC~qBlGW{IXh8aF?!MRtpaR&WnC%lKZh;#s$og}0&dqs8lZ=OcH+Udyk#UrAg;se9hsbAeU|q0iKFsuCO2JuVDA)Q-sE%!}5w`MQFruaAb0C%K)Q;;&-C+5I6`fL_}&)rxdf6%^HU9LBQ-
z@7$Y(J9LcDS3QvektxhZNYW9`&jmj*>?zmm=A7s*zNI|5(6h4hFh7HPKh=9r
zcM(om!G%U2fJKXAL0L5YhCsFPk%L$vV=c`eX@4&^fQW>(zr4O{qS}$%Dw}BxauBC)
zYmdrr9+w_Qx=(=0V4TM~h*IN@k4E4dCv9wbBlXzr#PkxP_Uk!Gwo1)O6w5pfT`LT`
zs2pgC2gho>&B)#Wa)RI~r?B{Gve?|6VCr|PcZB9dkj
zR3^7@R}`@>X-HaW0X(+X3}=fXz`he|8OUmO
zauUQj9|`rNELqN$)22-tihp3@^)rOC=#>vAxz0)*9pW5V{aOSsd?MvlBYwB9{M?DK
z$HqI^YxX|XHPM>6T8ou1<$HbJ{CH;dckvNJQ%7gte%<>%2_o-cH*GnB(&-n4SyAk(
zQ`Y^SyDRf2@o<)KY3RzFxrgxgqStvTqOkJ}46S(gO@ADs0?_=W6L)!Q=u&=PX5(=U%?3|{AxvZ+mFHVJ
z<9LjuSNkrVxTG7Gk;YwbwY#kM)+V(u^0EmPZ9oZ;%+b&L43vR9u+IKn^RW5++}hTp
zRQ}XZzH2zUcix>*KbEvf=R*0Hr=-6vTB~@ymz4v=LB~ScgQ_u^JPK}z%rj`DWlL~1
zP<0a9wI5$8Rd-LTxq_YQ*Y@=T@APx^e{Oi@y?yE~cSynJQ_JrP?iGhCG$8eJ-{|r;
zq2hm^%c0ZbQqYJn0n{1-@VN57Sk+Ts{$01(gX6&e=E=%6$WnUq?Zl%8IuG--)E*|d
z4Qb|`7*;P-mN?1=66ICBMFC5O?(?w$CqAJxi#_=cAiKA}9e$AwrVrr>%G&i|cYhqn
zUx_#JV1FDRf((SHW^?*pC8qfWP@bLu`Ru?`T{-#XvoLswlwhz9R=|;&=0{?crE3ku
zYuDt_q@>0<`liLeXuU}4xRrlPIg(fm#65OrCfK#MXJ^NonOH>CsC_#V
zwAJIVKLH<0GeI|}s!r2G2)}{LsKJJF({Ba${R^NxN2I^6B8Lrn*DI%uw(7O=%MA4_
zJQcUfLW29&pq>sqADqrwLS`of3Xi+2G~#Eb#;3FXA_dQF&}O
zo~6-E8b1`&d7?(L734Dl_4Uq}22OR>FYBI_nv+sDX+dvVt?DAam0pQz7zNsT2D|Y=i03&s1c3^3F%gx#GU<4o3G+2hz@dHEf03{fZS{
zmo(_egDs^zHGYmfJ&a#-+vXy$_|No+ZV(>VcpU?C@ZM|VQbm1z^rxPk)Oua
zE+pDeo*Erok}TkO-FXq>X|rjBPQBmbYew~VR7^M^PB`f5pACehd(?nRwe}N(vElNq
zj?@r>?F5d)A36Oc`QJzAyb7!s+)=fMG6DZGl1nCYVHG~1Ju~b=@bKc6R_3jc%EWT
z`^xnu-(F`3F&#nYgE>+kU{al
z$`&NI?y|j94Zj1L&AfMS)apUwn&)FeY^*#5k}C!avWvQe$GQ6I++v^1G6Con1%Swe
zl6|V2Vyd^9Z4N#}5;XB3mNEq{v*=}W)2T8%w{+GRzCSW^n%$`X@~eCvf~D9>-dVhQ
z;dgr$$Hn!JxasmUcy=?p5m2A!1
zps~vGrauLzb`>zTV$}3Im%C6c9q($&2p$y}cP^jM4$;}oe|)7|ws7;J-uIoDYa<*>32>iWxWJ?&qm{b9`0zb*suADkzPmSud$r
zE^w{4^Ip}rIjT`2f8F~P=Wf=0On;1Cf=uRa910IREWI+HE9RZd4_tno<>k_Alf#H2
zR9Xi2-bAHAF8bzUphGRyy>KExsUm_?=V58FNRnOYv4S^B)@nliq^=Ea(9F0OZxN)s
zDMWo^9@aUO<3qZ4IB;yKHyXfsa#@y(oU1v$}-cSELne$%g
zwHYCM9~KWPZ~Bz@4V21P07Q^`QeitbRhYb`(0VgP+;KLTP1|q;B=X;z3C~b{8%ee`
zd7o}k%Qdb`WF0)s@DnWMl@BkveII()(7uJj{mRx%^ca(R!J7wcMjr)lD4i3lzxgsZ8vmEvpNKxUu1SuPhw|7Q>v_M)ScmSIIgM46
z>8jtMk1pMI*76WIbRiQt!a|b9qEKAHhIxdDq`ic?XG0b$*HJuHWx0|b=Aa0&at{J*@8G=G
z__CUMEJcaB7EyCOGHt`H9b+NDBms}C2F$gQ?~
z4(%nPs)?D)ITF+*cb{ZbLOzjW`YlK;&2YN<^~3xt+(A5Iedg8pJ(_K?2PaIYSI-EU
zjkJ>A)`1^ARc`EI#F{3$cL(K|mT>%kU}eP$yDapeY7AwLZ>v#3F+4o4rS#Xsdp6CZVe>RPciU0h7RPJ5Ap%t61f#
zp5p8vAA#fTd0yYqWe^wa*DXrgg4HPtN6!K9v>^r1PqqzTV28=KfXqRz$8RY@Wk7c(
zThvrhlwP=%y2O@T2I8h4ndsXK%W~Z^XNt_#(B#*_)*O2^RC3VXbt3H3+w;W0!aUY=s!kXDsV#g$J)0dv$TkUg%{#XhddvQslu7s_QjmUecENss#tcExhoON1R~~bULsGIRI{n
za3hxzY9w9q0pUr_Idv}%9wWsCqOF=f-Ea-IiG6uK+}ZH5y2XlP*Rw+??3$QSz+C@Ncs
zk^mOy{W1xbm}h!X-$JQ>L+(n2wmm<`*(&rrKVPiI0(h-pPZv;JkxRyOf61iB;t4*s
z%gdRr_uO}3IO+|W0NDR#1SkbH7wWO^pb+r>w@(zX8iux*L{PTn`rGh(kjev4OL0K~
z@HiUPNMEkfZo#u}SEZi!)$y8W)t!z5Z@xNP?d$fqAY$H{c~12p3r{xv5ceLIP@0&~
zgY}Rp1!(a5Dz_3B&
z`7Vl2EB+dBk1^~!B@1wt!EK=XO}dN&vK>m?Spea9tS?HRyZdlDt$x$iht1Ef_pj?E
zG89t+`nLq;Rwbi?{pc-LR`pvrh3X(xT9szj!}GUgd*|Y#`0{4Ewdu)F$ac$k2)iou
z)frey^UF_EJoBK+K!$gsxNf_{VU)jk6GTFX^&tMYy&YdMs!27&*|>!gn4l0BB0}N#>%)@~Bt`745TYqCfrQ+$ZU(XG1tS
zrmmCt%)rw;6GHqkeztP4~U$->~+(T^mX#VBGG8Qzw_oe3aVoTiQO^OJAKji&Rz9r98D$=I&~jevvs;jH%w2q_mzODElB1IF
zLh&+Gzv}7Si@As}0|2x>`iMs-Q@5OxL^+#{)3t$6WJ~$A!*s`~)S|})Rg0{hkJE4b
z_1+kE%(}4&uz7>e%|3gjY0eN{b{AC1s*TRofJwLDIRFI9*s5dYndAXR7S7(Yb+@u
zDuxMCy643Xe??CUkH4HN(^BA>%HIG5Xos0Y$(d6tV{S7aKra2NGSK++;F*6W=MTVL
zOBd*le4%!lWx4iJ8I8OJyW;&J$3APybxp2RGFVf_X@FT+-%Q7*$4G5*s^&p?RL8)`
zBfaJ5c{LERj8xi9H=CXMv@>o4z`t)Rz!iI-DfzGz`aht^=WXxsR0AQ#ZC;?x?DP;Fl|TGjn$93h+)
z==*$_K=&iBlk}&OkYn{l?AGz_{GOu^3I*SHnbtvk%=5e#QNHQD=@}sEFM{_iei|3$
z{JL(ZbN>=bQ+5RX3=T&>iuHh>d(!dKbXU-ZOUFA*Rj8|LO$y0iJ^)&EVrx>y)yWV(=aZrZmxPNd-Gg1o17=FQ&eyA$fz4M!8d?
zz+1DwK^{wO&ITYQr@NpM1DCB;tkg@E8FR~>?($fDK7G(CqSYo9UVSDTGZ7@U^Kk+d
z=qLMD$07Bbau#-ZQ~3d^-EdCIB4FqCcX2XLzD-|-ym4ITm-C;@_QldC;sV7ykL(Kf
zHXHhVgc9w(N7!G-ac0z5pMFS}pM{4vD94zZIks6eO`MsteRB0vwUqbKjc^ills+viVo>RAxsv8(3Pex|+Ki|q3!ac)m{mmjAc
zJ&}iRYN?%e)O42cVa1-5|76021j?M8SD8Q_DvX85lCsdz?lS91W9UUp4%%i5iWHl~&PNCJZ
zA3-NRED2!g4Q7VJM+@)WIA6|%i?~Fys&*_!_IxOjDEw*TDS6)V)CW=Hz~w~m50#HO
zujkr7E@)V4pwo<1=x1qglMR!)Y(16Dl2
z*D|`bs>2S9`W#ur+!S%eEOM+i$Gv4QDWraJ
zwF15S9p8#Dqm@;&iZB8GD@!UVN=9#THIW6(+q_d|q#jEjgSM|+riY&nGL+)I
zOA=DQe7?tQi_>`f`1>mTI-T9ajKL(DQU>q+Mk
zX8@4|AoUaGy#o{?=_gq(>Al?ok-qwwtDMEc0dLgy@xf}?M^x&>3qdNh#%eo2(jF01
zh{190+86bJo`pq}Jw?@Idn7?&O)W@^1VpAVP%c&rvQ;uJp3jcymh7GG1>k%N)ILU9
z#1j0;(Z~6T>b&mp)@>~RVYYPm`N4_s&dRW_ZOT+5$kHK>y~Xq{=b(a`vJw-KA-H_?
z=&gLlR}BW0gQrW99x8*%Eg-L;^4WE2&FE(1n*XUEa&1H}3d`gLrxeB5U=c2e8$wCu
zRBO-TdN;J$-tqGk{f5!6%?t4sUoGhm(s7UCef_TS@(9JH-qVJdN?k?JX!3ddl`Ks1
zpC3eGwg&<+qOYkE?(89_zGlG|f@Ndb1kY7EW@)GG!#$H?0%*A?epAj0C
zvb+OY%eUYS4X&PfzytV5=d2mk-7wOtc&U4n<
Y`mD;?{Z{wvt!u<5f|ZR1=~B}
zv+V40Z-wQ5kYpUt*8IGY{_*|mhg^nySoJrx-}k5g<~K$^>;gEIVPrzDBwu+QH2!XDKd<(6J_PqHx2z)F_e@MLlN2%H1i6wgys)p-Jqn@UIGJ{Grwv6
zP3wPm21CNWp)x~MZ_}U=3XHJ@V}5G&51DYs5-^RF-QHR27wlC+b+Y1ns$|c{jN|Yi
z-e>JJr`n;EeeQXO(${DbO!|$YSb8q`yA3Wnh|AIt-}PBiKfmh-vsCaU(6wTz>4oP+
zgbjv>++&TJ~R`Y?^=vGisv=2xZs~o$<+6v3Hb%eK$S6fV&02aYU(S}=fsH~O&BoE
z0lL`l#a1z8Y6x-qJ-PqPuQ50TmB%X4+($Pg04YO|Ju?XxGPuhp;`CR*hqx
z4Q$#T0_YmU@@1oDwFY3xM04u8TBb`kaI#zLwMHP>=R21$)F%ZJcWfFI3=^2)PMG=p
zR7xq{zCA0R#HRe+A;ey*Bq1K|$sUu*8yiSOJsbW1HBB*J=}b_Rz#nNuCvzJ!aGatJ
zMw*ame3Ea_O&J~dreUF*NES=Nsu69(>J)Uj#S3O(^1@L$Z70y5jp5Tzh1u4j!}Il1
z@g`3;_0(JcI}dT)|3=E8RGPfjD|UB+k$)@!BtWLat?k1Pc9M7>42(5|W)l>g#xSv-
zwL2O*^Ejpoa{qF3z_Y}|j-%Y6L*x`dEH)gf4=V0m{g;^U7~nu`R3MJO&KBn0|0P?@
zlaA&*s#yV?MV5hU5x5DuNK9%9&guj~@>Js#lx$
zI9KlI)FPOtISm+9T$
zsfwu!utCGYfJp0VjAqFuDlhkgl+X#OOu9K*%6LlPAGld}IP-Ip!Bj(s?%$Nc)BU{L
z_z=^(n*i*dFRI_u3!Ozlq5qg*EF3V#h?!+N)#r72o*RJF?Jc;Zg^QW+Kn{{S$`$nGyY4z+QIDhzvp@-
z6a1V^mfNFhTiU4H_$Kv?vnl?Z3L-@f?rbwIUqPR7FaR}V;~BcfBdinVU7)|b(D+J<
z8-bM-YP?YOyX+u9JM?6<)S?!22}}Q{5{y2RKRuYJ{p$3q{dBAiwi|h<#3{P7fbp>DzPE)D5(L*x+t=HRcQZeiZ(a*?}x0kI?
z$FSmuKVTugk7{tnyECV5iPisRFid*JU53DJX5Q84$m4r0C{iwj%biEGmJMc9D96j#?*MvBS-sjL3rC0!duV%x4)JkcmAn+cFxo
zjMB)aH&-3lL+8kiI(tnP&+is}A(H-x{m4-T`IYekD_gsHWCC&9gG0%FcDU%XT;
z;Z&5F!JDYMIzU9pSjo|6y6~AYZP(v*b54rzpM6Tpb=O^LY54Jeo9A@&h8H11hkkxY
z@^X-oq0Gg2!k8a$T(lB*5;5jqm09Bj@Xsw$HwL{lm^@W(3DbrjN-0-xT3`_ne*JQ&
zZNIi!({H=IJ^p0{Oi6yDb7;S93p}^+zoR_h^_Oe<-LsD$s?}^c@tU7R@$$_@9>(#c
z=owGGHd?=p?)8yLuLq{4Vg#8{jxjaQaLig;lDyUvFC}`*&phiCHff-_6-c38<#2@&
z@%cfguK)JjmAgfX`PpgnO_gr!RUk!}^#1*t!IVYAA9DbVk@Hu(onBm>wW_ojKEW+@
z1~*bszZNc$BOB!5i-vrrgJNYfam_1%DCFPYGzYq38=knukLx5&S2>25``p40s
zJW&}R_WIms*tC}R*~DtCb4I*QT^6-x3?*csa$b2P?rvQK)s8g@Hm1Z?s0wydo<>rL
zj)VOd$UZ$c7>u(z!s~SjxHsL6Rbn%v@H$Z;hdO{PWk9YZnUFBJ>EmUNq6wPc`Nco)
z@FE*S1o=>t-AUkOFdtO@_1*!Pf()e4Y5c}qq|@`|#{S@J24#zD@*(xJIY0G9J2Ss*
z$QSRCV#0*pF`&Hq$A`TTjPmmM6SRSG*$4WeU35)lKZMC!1;n$9@j2})a2~l2G{(iv
zBea||zer5y?p>Rw82v6
z$)z^U5~v>k93T=m|EWpOzU8n0zwSAg>jVR1o);s&mh8Y=aPNv*5~00?lYl=@MCi;!
zAJKgZYVSr{9cwIt&F}o%rrG25iJ#Ag5YKsE*ZrCVMRJfe8ZK_Y&wVOEj~D8F4AVUS
z)?PDwE~C01bfEuYW39BurQeF#wII^B3!!YJVjmeUlheHF@u^+8IKuElpb_1`ZOqFu
zCQ-qK{?=pgz8VydZ~8;RLWP53?2r!+XX^SISv;ZP*pH&slHB45$9I3|Fz0$k#aejs
zUKX+FzU2y_YE%`n2i34_+a<&~no{&i{J#_e3knAKg-L>*i&R<>+vFtFi#Sr?$YAN;
zC5_6TJ_jjz-He;H?WiN6jPVHQ`lv&TkD0i(q(;#QdDjwC8!E%{zY-QY~qp?6Y@UQLS$l(=pHk!e)6Ij#=~1u}0
z_7ijiJ2QYn2;8eKXWL;0lBIep`&Yp@5l@)q={`$X%<@OdVSG
zi;Y2m!OLOJeSi@sV;pB1D%jK~cVp4q!3lLMVLL%M*L79;0(;?6K`6rM4*#8m&+|=W
zE!;2d5zNqi*uwl(ql#UvUx8p!kX7Zw&(;XK)!}EEV>y%*3ZSadzY5M(B8bj_+|-l*FMI3V%>@J*1xf)|DBXAS%@r+4mHFQzW>zJKfAJ-$6
zaj_E>EB~IW`bhTDy8v0~23J|=5>?$=abxMk|GrMAdaB;j!i3MT4B4v3=bBo7w?<08X#RiLI=OPp30eyU1bPH@k&k8UgEy{d=TlvvdDt|B@zdRvE8;#b-u<7C1RBc6T9?#T=7~q`<5ZZ<}1lKDUw2`uufX
ztvcC&i5t7UBAM!I|A(-%j;gZT-ZrVEAdMg(-KnTZiIg-bAt?ePN_THix23JDOmd)M?hodS-NyFxh8^QU=5N3JM;6>jHRI50LXH4mYR$R+&*}Tu
z!u^l`^wzkw9c0y}sLy>=SQi{}`>?=Z*`F4hof<|*p--^&WE4+qH`|uKco2eFye#H3
z3iI%$xZx_XpiB|A71*gW5(6(Iw=mqH;nDa`@HcttKfdLkcwqpmjnM8pxy6BFa7;{f
z?E4lx!CjL<@S&8>UE9Pryw|ne!#j$1-ShM9d>TmaQ@(iZkTn^?htv6uFIotlp5grz
zkAY1zL!(OUX6o;q^zWO-@ALTIueRQ&Xvo8oZe6Y;XXG=UmkYczA$N9}UZU`$D=+)|
zfsD}98q!E>BQdl<09Xf)N1!ST%9@>tpCrq
zt;)^bF&OXkuAXhgtr4Cx@s4S&;XHgHh)>Ju6V2=va6)!1R493lg)7I66FzkPOjQ0L
zFZ&AJP^O=cQfM@)jOV8!zcp?5@BO8RwmC5#HJR}W8ShR&_MN;8+{?=Wir|+C@Rd7x
zoE>t>f~tCft)+jj#y?|lJ4XE5BS+f5@mO!7etj-;jay&r8;Y~=G()WZOm{1N;&<}S
z12=5DFoXP8xgw1EDHP8nIbiRDW5!|bDCFZ5jf0&%$<|7rt&oxPRKxCoWaYR1#m})1
zWomsbhQfD!Rvf1~0@k+2-q@u_3_Biuc6xSySMlp8J12bP^-M!h=keEa_vX8YyNW#i
z&h7RLGGX>tsja;xOcOmWOL2L(jZ^0T>+!uy8IocQL7vnaI5(k+=Py;@O3-mIC!IKF
zs*k7=N@ebXj_Nn(J=Xd;7^xtu4=$=78K
z0mXo)v1eRA>~L3|#{(d<9=^L$0jfEx7{cx~;8XH{T^733xc^?=y>S>&VIc6zu5-iB
zZ6V%8j#BM1kM6Qb7P(^l;wrzHXKH4oSYt&#|LfWZ!@-WFS~K$Rjly(%d@i$OgqdFA
zR3U++`)@fF8Mk{f^Nyr<5~%Np@P$5f?rzvEY6=nD?OrDKF?(*5n8M;CmHp>-b9ct6V5
z8hTl!1bl+pSjEe+TJoCZ83h>gCW8!AI(j-WG;2X*1RlPRu<`G{Yy;!H+vFMB+F)2=
z_(jj}b;s)%TpXNyez1^G+d{;42@Y)YVm$SUEHl11c5oa(Boz8Df1D*lOUj>k~_aLfHv<1zb`N9utNZ|fb&-?u>(D>NZ+2D(iB4z
zNKKKJv`JavFn>{_Y9s%lzGa?FkU?oZM&LY?nxnC!*tKhGSAajQq!UvLQ>$XvWS!5z
z(BkCKh^N)hJx6+mJrcOOT&|w=hy?B<}M5Tq%i`>P%U!Ur}^>b7yMQ+^)mD<
z@bH*+{cDGj*5khvJI=gN(TP?h|6ZWKr~DbDBg7*ER3;kJ%0GLIx9jWU=ZzUZkj=W6
zdEfqCxMy%UvZ-_02D{6~h`#A}VA6;2Ntju%@VWq)nGC~
z1HX7)bH6XQ4Ne;Nw?%Q+?)w-vul?Y<0F%R*Dyb^xr5E(+Ba-%JBOh&C#QR_wIsjP^+8YP
z4@1-3yfdVW%!}V(AYzXX;+eSGTAtLUw*`KEZC-fuk!4@K
zXdd4wy=yp?oI$Q
z(YYdhdtR%G$Ij!_Wijxr7=2MN;;{>4D!&hGZ!!jz48DS9S1CJo#{7F7cj+#l=T%67
z_44QG%mx~d^}982Z2X@C
zbr%h)|E4{ky*dygPJSv^yTFu?lWqAq-n82WzO+Z39r7!7;l@9=8g(>1YGR!do#Jd^
zZ_`vxDk+;3D?;9j9%n7#ZcXh^%hSdC#r!!5>Bs4VjwSrj#DM!idzdmLhMB>%11xS-
z955Wozi;H1S{4B6EoU$NA*~s?e}U}~QF)t3()_R@9i~2dq|bezX5D3j%li8tiU$)f
zH^h$4i=?_RVQ~iL72MRsU<~?1lWB~baUzF1z(oVHW2wdILTrV44rW0+A8UIl^>N@&
zh4A)3Ms~IBS+OdgXA|r-~-Q4Q|A!6t?lj~`TG1Y%9
zU_GosUgmM!4aXJw4Fw{tk0F({s?{tq@)wr=Bb?NyhFW-}%JM~TcB{G{f7=Mcjedc<
zHkqhXv;G3}Xmjd;)j1{K3!nd#u72!yUYpv;H
zA!9D?vcbRp2h>O9&E6tOMYYlFSmh9XD0F0tVK~J)O`IBiF6!^n4`~6x*?Mb^EkbpL
z8Z0JSO0~5C{77e7L{!Y%ODv+lSkBG4|F|Ii7IA+Oje_$1J)QRkz&L5x{P{HCum#>4
ztby7-_L{ES>W)p!cL7+~?tje6=j)l)3WF4xe0?oad(mOV1n-`t8uSb}d&a@xCFfzd
z-w`xjzJ~H^js_j(xk^W$+E5rNXBS0zd`)(KbKjH>9{+C;DSmXF0-!2B3tDK8qF7)a
z(eNM-U+@&$PyMRg+MaW(;cX=){Vn9#<9YKNy?Aww-P#(q0wY*yd!s5g5;iaS;Y$2%
zngZ1fUE$miOU3LV6yuKklb0x!O-H}1*Hq!27n9&-GUFYXW=+axwP}=I^W-xdVM#2-Fq6Udy!9(&
zq(cRLYVaihiQ*)rw9Ily?gNn&Vtu(0b0Hk(0&xTn-;tsFpLyurz}UA*`StDn2XE7>
zJagw7K(h}K6YZkb{6`wPj=@erym8a|*xScQ-J#L%H=;qt>33P*CX*g^==*X
zjFWZE>YNg7l9yp?|7e{>Ae9z)n5Su!Tyu5PoN)gd$WJ9D^=kBrof+MoL7-L-%{<_)
zhJoPlzY$l^XM-xCuU$H=$5&o^a{OO}IHfJuJ2a~z`8DXJP7S*tpDq5GF!L9|Frw{1
z#wrEfGi=+|`@rchG92gsvFz8bYn8l?rV@6}hq$(*?b*FU_;#FU
zX4>^NTpvFm$b4dBuLi$|p7|l+WvIuuK{=zA#M{UP!zsYeZ9OJNyY-(Y0`S`9k`brn
zdX2bI5q?_!_3!iM8dC%x^XeC?@v7(YYbrPFanZg=iNo@v@0~T4MvLv1oSF*o*j1Bs
z#657zqvAIXQ2sxTIqjIbYL~sq-sDG(SH@!sK|I3#C!Sk?#=gbrE0{-FOpiMPbQ+VZq;54i|?DX~P?i;V#oanHmW#jJI(i~aSYV0EZheN-F%+TefA
z*pJLAuW58vwdI|HTK*i`ujwNAH&>><*Id0egFm7oR0hUu>S2-^dIX}b+ZVQujou4D
zJw`6S(>(v|oy$edrJOVhSCI_l!w)OI(DQY=NZv!zd2@^Z&
z%m^P@*FBzH1Y^{3t>`%&%uuEUkSihprm~W6`0t$w?r^s+D}x|87ZjDSjJH$Y^?vvJ
zQm_w@cSiGJwQJ#SefP!Zo{0+?v@~6{0iwEqm-cq@?W)H^IVeG)4<~#=ho;;92MKq%
zw1==?(NKLskFw&oo78~W7XLOEtSZtODGWY%Uh@bH1xE@(KcI^)jF9<6~tDxZmn5C4zLMWL~8WExQ(q9o6
z`mQTXPCxE1l>;*~P~h4aTpD{gzE?+PW*WdfLBZW@{ky66GBhjKM30}BwcMENjcA+x
z_#PC!jRyKwKt^E5#6`Ao9RcjzYY1?>jSP+eH}ix=ai6qH-^1k3=7o{Z97h`
zgY#?me-hpWf^Vh$8z`n8fg>DVKLd_lYw*}^t0hVb!r+i2o)QAJ`VeT5*i4d6h3EGO
z`EU7fnX9vhU2l6e`zWay%b3z@-~SEwwF%n$99|)WAiM%s8nD5tpgQ6}H)Wv}Mm%wB
z`h(C{X+1yNKTMU?9&nYt>Pw0A<MFbU(Z%=3
zKFUk7(lZP-eaD^mM(A@#j2Dj1j#W&EIsxa)ethU}E={{w
zRNRzt#oZ;sH^LrTO1TdK7LLXmOylPVtC^{wc&K0d98^P8fF(I@CIyuOCw%lR7P_9#
zbyCcg*#72wjYdX4m1Y;1{UyW_4AHvM+xsT3BHe~mUDdX#FefG0_%~fvi12c^4JES&
zWeRv}42rEzHkYz*W@YDP>9|10Ur9p8=?zmKjm
zNpXf=CU$%*Wb=E`KSTq-rPzVPmvbZjh5{qnZ*tEy@c-h&HU8nljqvjvH+tI39Ycn6
zA!mUo2!(A?nK5Ky)EZ_HIP>1obWD)oi{C$+Hcp&lg?1_=MJF3@yjrIqXzrq*Fes0M
z9H@HS{%Anusk}b>#d=CbjS%CLJRPW8$|p60%~E(tNL-$F#k`leT~O>cC<2u$XuN)Q
z1Xk(6c~BA%ExQg@hqA5pmQYvBlPc_%Dibf(O*aAMh)eg0_z3VfD?|8?(G`7N)9c5;
zg|-38x)R6OE|l8yQ#{6yTS!FD1*r72<^hr#JrWdYY>FNhgfGIvr%!%3G3QSes(}Wes
zc3%bvrk(Rpb%5BK)YXa7><6ICaIb~)4NQ|IOGByZj3wS$Lfuc~LLZ_%S%OmBz`RbC
zOp#d@#ZOVPEKrfc?*sJ(+A=NQ`@_&TRQJAk6t~
zo$1K(9G<-dU<;Piejo*T*sh74Z@)D!8PeDoJpX8xeHehKC-9twG|P+)j7Lzaa52=^
z@GoayuR3@>Jti&qSqVcOpc`tjs+elM_I1uxdO!|pkhpv&4s6E3lD+*#@?Y$(>f(
z#I|e;N`S?LgDNuMv>GSdh5$rpUK*&~pYQTO2E~5@o55g|Q<)msVg0buNSeErx+z|H
zIkRzq8!_;)@fWCKLIDL1YtJBe)iMp*&e`m}X_4F%4G@uVdduO78{Rry_#Fj-$n_E_
zr!C~VznoGmM_db-HsmH>Ktfyf2DV5{9bj9i4!V3H_W9!#1=AKv?}g}NEsTz?0(@g9
zurQeTl>r2wt%i)8D*UOVu7+l5Om>>$B;YfXU~OBLdx6b9{!ZLB@#EczF511%oLC?L
z<5Cbe@SAd+us9*TvbWUZM5K5{aLRt*oRWDw?0JvKkv&vR)p01H6K7T;v1~7`RGp^o
zEW(u|cmWvOO1iFVIy0-o2CxfIUGvS{9>}qz#^upp`t!nlz-bIsg^OBskAwLp3PE*#z#h=qIPe(fVvh
z{qj5jEFSZ#s2cEiq@E!GN#Wi>v+ks_{FWsQh$I}ZC7d0jxNAX48uw;uDSluX1t&YJ
zoJxf&B6(ozeJO?+hJ3Db706moL15NP)zYIM`4LJMU#78v8^`d;>uV!>TMW6^PYqJJ
zZkD)i0StBbk(9%#lN^#3bgHoe^3&Pkg4vPgr}QOWKvB(r<6msn0)VO|lt%%K%?_mE
ztF;8z%9kREuBCB>zat}24E8>;j-f#;y=vVri+OinbLN&Ia`-&I*Bo~ARG3?C?+++1
zEt$!EZ+YC>fBs_<_~6$77K85SVhc7C`FfWs4VejNp8)r5bluFOPV2Od^{>Q_X(O+h
z3&rf`f@5Xc(jtK8LRM$BIj_G$vSvS9dJTChQt$8pwMPD>Rf{DI?>49zCGOQ>uW=Zl
zDt4%Tm1`l!GcF)&rFw9v0vp8K3Bra%XmR!5ygJ$!NM(pu7u3;P1C^}(fXi~SH+JyP
zP@&s;%DGW
z@ppxpR|`Dbj+VF=&rj|-%4Z=zB%U+wA@VtK&xZ8AEk6AmtMUErwz`hOE%>oERW}^m
z*>OimxBdK6gimX-hQEN7DYdwKl5x;J9{DTAXeG3<0t#vFTj{cA@o@G~@!BCczsBGF
zr9Khuj*ve0@WJn!=?4d7>>dxZ+~FgSUB9i#f>xT(do5l00YMGZ5ea7yYH81}pv$#-
zj?<}wE#oKjp6A)Cnz^u~w@NeewzM4H`TiohPn%Y?mHAZELP|`AgibA
z&{CPTA7h)>XJj^+{(9fEM8wn6({05)MoD<5sma+Bu-c;6UiUeyY|YKggKG9ZcN(O(
z7X4JBs&Iy>N&=xvT5#YCkSbJA8DwhJwR-Bi`cj37MqO-SIP{7
zm9V_^K#k`_Q~bDQ$B+*Ebuq#qi$^y*RQ&wG(t;~_j#Go{-5jSTSIr!!7FVYnr#9D^
zkT)WL-sUy@<`&ngAlgfm{){-lW^g*)Z{kt0Kq{33f9Y~N#_KyMQ!wO(i9u5fGL_SF
zP*uj{hWEMVY9R7WCFq31r!Gc6WpvOU^PatPdoZ(BE0qUSSV`&(CV&a*;l4XG68sa?
z;O1qD+{!M_7jt7>f>U&wzZu8SApR8Y{*&RGgYT6j=LW}1Djoy6%BUb*jxAByuV%E4
zKw@wvex+7;9^x1)p!1R
ztic>Gqfx@n#~w>r+yhv|PS58fSQlA+r!YP~wMscU#Fssr8eAm-l$9yID|7_OQhMjG
zB&pAji*7tdSuAIYb7klr8B{2cHeC{q4!@LKG!XvVdzzR_>{5^
z4C_VIaj=0XZ<{z+UTL3=%32C||Je$V;1CH$Ft_ehY8-S%I
z)fALTQDF2Ug}DW;_aT#yuVwT_D0BU7`9Yso3L!zvE*pV3uW~5lS>L|x=e&JgDuTXd
z^5`U2xHEpt&ar26YN~4S$X(PfBR=lFz>M9Ig3FJt<9f`4o;U#+=(Xwsd`S{p3zlQQ
zWa=m%%^(p+7qhZ(M|&wvZUE*MCZc4aB?ktb&&LK$j#q3J4UnfX)E@hNDd_9vk2~0D
zN{L+nuN&Q@CAnr4t&h|^dwGjEd?{pOZyu+V)CCxj(EF4@jj9(sF9C$D-%|{}q?-%G
zu+4x-P#gqH)_v**aVgAsg3HL2($CUod9(^+7X0I*^xYdXhPWF>;Ua?XxhD{I1*c-3Y1^Z+^syI)
zY*Hk)WG>2TQnX)b42RchmYk_85uJC_u<5BNwtHxV?-C>|ESJl@57mIHl)$IYTT|}Sdj=Wi?+jrHraw*Jf0#kOB{u0^#MKp+
z5q?sZm-!&p2fNhQZlNKxu!$7rM)-AP&zz`zuYNa>B|(;rb~CvnVjQ{zkQpqkewEH;
z43724`f
z8rZoFhhtEv@b-i4M6;ro-9!^1lOWm)Ji4tRJqLT%^Afk|U@In%tFAlvz+H{K0|m@R
z5c4d&k-f6;K)2#_6{b}ZJVf4labEj+5GUzHX_{wT;fh$Q5@N3rhm2V4`z1gk
zq~bVVeVSpkQP!%@ieC^Xp|QfY6n;sfF%mwco8X$|{g%zs=v4t68}zm_WWr|sU!w#}
ztwp6`pQ}xM8Ew5VpIOO2_Kbyh=@Ww+YdrQ17JM}dVr
z6+a`R&R`2vGnvgzPElPzd00FrG{1GdG&(`ds~Kf*6|@wYZNhMmDr}<7Z&pWnhR81{
zbJ42Ev7}f&Hf?<_&v%ZLc|2-`j%@WMgrS5Bfxk1?^$D09n10mMBj0yoTE-tmDV^sE
zZ*wjLeHW(?vVZleUsp@Qd&%m|&ouTir8{tOrt#J}TBkq9=6IeAqjerR>5+o}P#pW%
z!!rTw=%06C*&G%0q~XUK$4B*&ij_#6_26)Ul5_iQqg__HpcD6$jyh{FqaBR1@No8`
zk7=ser|U2;alW&8PlG#hs872}2YH?dUYKt0CDMxiqV6pB8hw@}~HD6>>jAOfmftjmn^8JJ<^p0l@;iDL*W1_Ifv1)HK{AHx_ksDz#1;
zewf%itK0+>?!i?J?Gi2gI}WRZA#RcrA`RwvjOV1hDjl=g)e1z8Dy$lM$MXcwbGfzV
zqLkWw+|N;UN^$7oS3W467?~U#J(8gW9T?QA;$Qe*2=*&}%`lFT@)Gls8RM@<=5XoROn$mhVN>I%p&h7*ct}b^i2sFsu=@Lmf={24jz*2bwb!c>
zAM>8fee9WBArWtO6^-lovBoHl7A;dZ{qUuxKE8a0^uZVs#rMTw^F{6JShUPhi*E+Z
zXNlnhuq&V*v;v1tx0N#UBa8X|ngI5XM@>PL^8N(M`GwPrd1#qh7qlEPVhQ3W1?bIg
zTQ#Pt^cxWI{+U6x)8PF`$0>6I)M8NmrMuh>$0LfMOj3I8Owp?6?-{{Y+ewD4)aDH9AHcMww~$)zvDdcjk~R4
zEBScThq2$3$XxZ;^rNGku=rP88hSZhDr&kQfzQ@KQC6*LiC3HNfXckjXZR)5$6-4e
zFOWm)+w8}+aM-z#Tz@_)=M^y9hVS7NmXRL825pn^+UT1F*>o+xk
z+*Rv=;{j`dUk2;A+Gfg!xck48(N7;Xn=7j8SrJ5#JsP1eR$Qg4wtPkt{|O3d5tr1(1;4k`
zBtoG7tfxm4C3)+oXrsW7d+6=`9iY(&k?cEye%-)q=!MzZ@hhA~WlJG@mMb^Gg+#^m
zR&JGEN%OmnU7rbcl=_YE&xA`rzq2h0uqQ8qsfQq7#!@b~IM5Kt_|tDFYFH+$h_h6@
z>^l09XcQ6=5+XWOB34{9*V=&P-Gi_7bDt`L*d^&d%=fxZS#nzI!F>ph8--oD<>@G{
zW~{9yi^sYIsgqYFdMdI&L|sh(f$T^{dtX1jj#q155}<_gaeZ=fWk!iRC?+PIf(sc5
zNm2j+K3zfrCZiA~4VL^;eT}XQfPw2;Wl~)oTdNb=*A_*0pnA}!PEb&U;iPHrcD2tg
zYceUwdVx?!Pl}~d>a#*D?=6cju7~Svz?UaZrxZUk4uWcG*;F~4m0naG)e$}G?Oe+z
zkzKN7Z$E%&Ky)R1;oesD9$KIM89^V{eKf;u5)c!~?beALaMJaWI(z3UBtT*?p$TZ!
znUbB1X6S0ESr>jbD{@1#a89RdiMZ@MLWBAPkJsH1N1XsIF-doKQu96}(FVCMoCw$;yq#f(o{Y+^ybnHTDKq5QIGy^@i+R
z_chftzceXjXJbp2Dd4hRhniJ1B}r*o`1~qz-g&VTOa=IA5jDuTG
zw+Gc>LR1!Pbl)(~Yd>X_cNoGf=jedAsyO(N;58{8MWbXHjOe0)!@6B_8H3x;`3Io%
z_X#|W)EqBue)eGn^Ft4i9pq@)2s9mrBs40LbSqsYc)cyrM8SP0(?z~zSfrC7F`JZ#
zn5(jqIf=92O@3=QZKBHI3V8%}TB=*O!`jFfs6PP9JNp^uK7jxXWVT^Sb~=$u)kuU)
ztt7!RzHypCn;hsGif$MutRh>HqC>+RWti7r*$WQ`(71p+6}#YDuKMbAw5@jF78;lr
zfFo=wJmGOT#`0dbXM|{qoj+p8Bu0;H+jLi)4eC(PxA&D_D*A;;VhE-o2DGy2ysL>>
zUm{a$Pv;GLgQhhq5sysDjs8oDNSGoBuUL^_x
z?IwW+S9x|2sNZFRdiM~gTK=dUWLp0b+uLlsZv{$un|^XJr|y_clWOYTs=RpA@(_15
zYr-14w##m9W4iNZ)3alsNp?|own(CAsKwQzuzbJ+-8`Q}RqM9m#F$H#waA_%Is@ou
zQG3cO8&HhON$at%s*Ns%SNpT6gDhv?TdhDjQPi7XV#fwtaoa%9K!xQb;#{gy`ANL}
z7@RtBcbkVBz0i?%THWb7B20Jl5iyo56@)0HO%~w$)&6VS2`3Dm40#;=3_N^D6Qy2YHM4Sb$~O&44`9*emC%
zH11CwkLUa|SYg&M5u_mdGdX9E{qJ($kcntMs1-!{_l=9}y8TyVy)baZ={9lb5MH<_
ziO;}$dGHGgb7DaxrjFTMC)VIoEjT+CtQFNOdZ|d~4CHw;O(mrE=!H`Yb
z#~Lg_Ue`!HT@5BL;=<3q+hDsgBE4ubr-EXL`T#QV60>GQ}
zgq~0awTafg_LTrFv@HqiPN*U2U;M=P+x|{&M`XtPwL%4Kf6L%C7v)0OW
zP1L6#iW{3B*Lv?w%FR34fcxP{YtEbJNYHAUy6x9$_V!=7740f37X!uis`U1
zNzkCBl&*9>8gA#@PNPY%D}7rZ#$kx=dE{DAk!)OtYy_zuVkD0SuptBF)i6nOgkGy4
zVICS5oJAx=BcG>|V{~`{!oa;_Oq_sM1mRZ&T6}ng(6!*TN$Z6Zh{57#NK5omb@F(f
zTElLl*K@NVVH7rYs%S%1f?zO*V%2C(8H=yGG%Qpr@566D_OMvF&e{m~uG1N=IR`~4K6QgGCS%l%6X8c}pG8J4gp7C3uF?XHVouqs$%Vxv
z>z$s}P*(K)=EUM4p(?gGx`yjA_jSP$tg3M30}vumu%mdr&Ch7l`pJ1->S(XGI)PmL
zRZGfJ{CIff8DMMoxOA!HAe58T60YL3zrfKTJfW4GYfJ@1le-#xtJyGh_6wv|1Vtb;
z5gqa3HahA2{>}5?qrVc>X&BzaZ6Y-%CoxSCTV1EP%QF(%2;aFAn{96M-Dv*0f9SkC
z%jmr-be#%^<=$HbwwR{Fu}#ljhi_C`iyfU4nWOtJ9_AgR$ALHIL=E*Sx3u2&otUaG
zq_xuM!IfAV7=}!(1hC^qaY3Hi;};zx?ugmeqpSR^0^#4=0(IQuvlnn`t#c&3`AwI5
zlfCVV*4KLSPv>F8Iu#JpPhP*ps#|lCzAGAe21nuo_2NxMd80<#b9oYTfhU?fngS>A
zt1|{8XC<%t=p!(vhbn@QFsKZHxo9`;;jQ@K;|*CY8bl3YhDs01Is%UY984hAvp^f$
z7Yphm50~$>n!y?S*uwRscHWQ*$}DlmO#uWA5l8b>Z8ZGiGt5zi-v&}CMx;;kqj!pb^5yB!R0%(9N@jnAh@n5;nvLjBu5GL+l``i@Lv5cjL^4wy{ICShS%
zQyhcD)b;WKs2@yFjQI0_))^u;Mmc^g13N>VI|e7{nB73W;Q?14j<&-WSBA)N|6u`)
zUiqPp0^w+EL8>%Rari~J$>0b^w(jn|W1bvS`NO?=(`L7MDZ!;x+M`>`>
zs};vy!l~sB9zU80)x5T;3y*?)TS>0bStt@r7o;qs)9C)=(ri83iz(SCH^e;MB`WZI
z`7!)?9`!8;Axl!-xX7JG_v(iIMZ{Wn8w>TyNlX%
zb~{7+OdKsuicSfcs^zg$ZXEPyNP&g?L4jN2Wm~xUg<>d~j_r$yX8p1Ulxf9P-_*bu
zF>xsCDO9;gH1=NWLuI;ADk%oC`$TEw7l#v=awCn)Ao^?&(-$Z@X0*LzvT8eA=rk8`NoiajE#ES}PR
zz@uTco6OD0&uaPNbEt`&4K7hQ6D7~)M!Zf0_oOt|UjC8{&%95KIJr8#
z=u;N^_a65^vpJ=0UJY(dQwLWT%eY<2o-O#odM{`<%xpQw{*G|AwY!uW95BW@c@j;u
zKuVP>Jr?hu`GTJ9GZ~(BE_pBpXZJ3)mOP%9TrX(2XBF+nUCksH=E;t(Xl3%GOo;)I|E@QAX~|+y|FNch$wtM8rOe
z80#A`4UFR;YJ)K!g#t`}_|ym_mFH~wA8S!QZ=vlqj((QFV6^=y6l`x~1n~V_jr0*o
z75Cvk0K`kxbL1XSR!wt4zYjw9NFp7$VW)`G;>|0;o|7SQtpQ%{1j5^O;$0w1!n8J|
z?|fC_F#woVG2;DT+SBX<7eQ}(XjqL5DZGD&OFyw7P23~uMHojz61cSESHY4Sd#$X+TpL>7-pjsHMO+Qi
z!0i)869(pMdQhyC!EM56tZn|p6J!R`_q(n0(#5V7*bX`&JGKVMtB_XIuYNB^AW>#U
zCUUPyqe;;bmKslLUXvtU0GUk1il0_oeprS&I2FVZY;n%0%j>5GF)ph8t_1ggMl^Mx
z1p91+T8jq4FCfOh3D9d^44Z&Pxz}M-GCn$^IJ9gKs(tL10Zzt#pW$B^F=Ud?(kO9*HyFtw+V_M!4K0zI!k0~EO^a62bZ}PE=|#eIXqQO-mu+P(@}?8=1$mm
z^_ZQ&ffd)Gnm9rLtQAlz8gf=N$HwygetC4mybP@igQJ9B++cA>`wBwg#Zz`5G_oJG
z7wR*rFrq1b79`Dax531D3!0}@K4h~rmpKHn5@v%94NeqU?ejt#@{7pn=z9ji45B}-
zp8y~<)jhm{Zk8}A^gsw4OGv8oq+$YYm_Of&%{aQs_7X${BU0pgB&=7MVabx%4~kd_
z`5N9uKaHJqOat|7yF=eShw$YpWx}W_M-Oq|ZGsXSHW!gE=muH0YO!l^sDbC$Wo@bj
z`Ft|^8pY4k?A~+lXX!YLw!c);h4(x>fJ3DaSO?lcgAwYs8$BCQPn58Iq+HIwqwMI|
zmA!pb3Ef@TU2?YVISWElVPs>V^(4jFIVZI(-eMAmssqpd7P_2FHC$o43Jy}BU7!1E
zcRM&thO*S#osV;&=9rI9yLUh~`A0l9mBHcLXVmqYqqUK2dX?b%X1_Rl=g&*}R)pf^
z#F%4&ebJq+dlGb(YvsYw%Tj+o?`X=&fbI-*+W;cS&=U}=&G~&O%ZEu@@2+GW5AEy$}zc;f#5U2A5bqcS+{p30&E%`p~zV<~VXT*%I)$jJ{QZVNGKm4$vlO;N(A0oaOKB#;FQMLy%Ww(jb;Nna5+{*^&O5alRZl|
zLje2Bl}$sWjH~q14X`?HeriE;$XCxcox$jTmzt+g#L8Bh=lE@iiyF-gg6&2*pJ4S4
zk$aydSyqY|<>c%j(s_Zr8C|wI
zrtSKM`~VBFwSZd8qmp_q~O4
zsPi3sW!7_o^vLx>RkJxtBs%kS%$O1aH{p6!=3|8z@(QCrzC0*%oGiD&mck=(M|>?E
z(kolb(yMZ`7_WSE-T$$hypiCQm#2%c4bae-XvM)nf{TPphStMgZW64CDh*9^-{}l)
zbS+rj$6Pcl2X9-nkZq*#hba}6WyDaRb@ul+YwVf5mF3=|cIRSiBgrMuvV7z4rn+g*
zc42At@ai^eM&7GnlSv_|3K)=OP4sSyP_Zq->UPBd>bUR+kIu)+X2V>|H?eA0Se15^
z2KL+|O~iHe@HMitO^CX3;`5V_-$gQffsRhplJEJ_WA>!#T+{6;^KsAVAHv&mhoaxH
zIrtY6n}iZoFEvgJsKn2~G|FfmGy6*DY0_hMr0LLK*^E;HiKL|uoVRa6`}<*?hkv6Y
z8?`scG?7&C(ENJ|p`S02%b@y%7oX3>p6v9=GPg?u2>?ORu^!GMCxz-$XqaOf(j9i1
znOA$Jd9sO2MMPCz&qq@JEA!xenR&nh5zElV&Gtb?m1qp!xJ;g}@!i1m{P{hV#+c6(
zdIu&e7AT|CnkcoHh`M*2qR|vhog~D_$~)36&a$Z!u1q9Sst)#zHZUhre>Y{LW&b!@cRsqX$V$BD7pOYA
zn1uIyMII@5JsOM1%Y!;;Jyl1WGc@47K|BiX14y(TyMtqN=OY>in|JlV!XE#m!2V55
z8}gyZ1I{&JJHx=hcll0!eD%m{)Av_AHC0BN-%B7wtzoP>Ig_=a*i@{gfQcWjU4F5)
z{pdZZDL*<6B7dLkN?MJjOeE1&cpO+ow+U$u4zhPRf9(~x^@FQJraRcd{8hVebYXXL
zwh%H}V%K_V2p59PPqUg|%NMad8HEKR&*xdaw)teA8Zam!xv4M-J9Kjp;bI8^1E#id
zWUny^HAlyij0WdxkVTFw%0w$lK`W{!ZP7~$M
zz?n5}`TF#S5Jfa~8I+0274SM}B*-y|M|g1oO{W7tC;lD)rWa-rXnZ#wpKcIeea4^6v9|s!st|4Yjlr4r
zi#juh_TiR8XTzX2D75^nPVbq=SrvL~%Z0ASP$*!DLg
zG8)X!UqbwXPUcjPwcN7w3;4knDyB$64`?`n9xuS=r4j)S$q-UgAVqrz(apkB*;CY*
zw6P7mXN!886aO5TXIag!pU+m&%QVN-Xy7um}fXXTzS>R3zM
zd-9qg&?K-jfkowGvEdgUn?KK@02GesEd)$!isIuDTJ
zZE$P3sp=bS95e$;U<%%GyX`N(Z&TK!YzGFMM|)w!nH;K;S3k0;_5j0iRK5c0focpO
z(aYOciT3R|NFKp{JZ*|v6HF%Ua#aN3A44Jx;qpR_
z_NJTyp3%H=&tQ9ITOj>6gAf+xnPfs&5PT8iU|zT?B(3+o)02X~##PK(Y70mR{1A&Z
z^gRK?%--s!WMwag6jn}n<+7$@$`Xt*ipcdaa+#g2k5nme>8q(n0F5qBCD)S%Kb;wn;kxj_9{gnYJcpZrz=?99~bg0<+$taDFqA&GV6~WzcdKL;b95**cNl
z&1k!X_O~(l3(5X8a;!ibR^fki0!bq}D#MO?T5m;2Y?Z|>imQ*N-BJu>aF~;l+Wd&{
zdN4lRaQ&z_ytqpYoma>wtm`|lP*(_w-w)a#H$Av<(3dM~+!S#AZF_+(Cftlwc87Wg
z9z1g&;q+dNBn3bbUMmGHXYP{hok&J|&w&lIs9cxm^NXVFJkc!v#-NOWuu=td`Cxl;
zzm-iInGt{D6J0ZUku9+gbmiuJXN+?})h7;5-aXuAZZ+39BD<5%$A@<}tr+W=_)xXd
ztDpUj{0B`mRRWPPv}F%2?2ji*P##X+1z>7ix@#*)y;aycIAqQ92|8M*j7QL2@V)V$XH>
ziOo~}xfs}>e+hn6SkZ2M&?6$#$UGA;xD*y%qOBuih6f1DkeMj#xiU@t5#kyct0^d_
z4K+{NI_ql`?q2?~m%nloy)|fXe|28ZzE28OOytjJMp&U4E{p;8-%QL2(89nl1b{Kn
zMq=LDFnE5gyjZTYAN-jyy$U&IukGDE;FB!>kKg&pA{66Ap=;9&>7#hEc9K9^kdb|-A;
z$KSI_F(1vZ$4~j4o4VW#h4lGg!*{l`Byx_@ya0topl>z9pWjoQ8-q8j*?E{e%*KQM
zx2A^x%Rdp8?$FMcqF8+Y`DbFmD(-hFNtLRI&W;BwaDpjngF>cx{7Z<2CtL?1Grd@cA5zi5n4
zXAqy5`t}=?v}x?)9v@=L1B7glftPs?iY?M?)>E%LP-7jW5xsHf-aY)2rc1$6nR^o3
zS^?R~@qGVG;c75SOMz+wpCt*U-{tbO#27{Pgz6Ni`4$Jpo*gW`3s$F|_6M7r0At=9
z)t|Nk`RJ~gKH6i-2ae|bidX+xEdN?HAk!k`ZND&TKO$G7&q@2|{~OEGeVr<*n6ck$
z9Gmvr0h1ggOZR##rV2B@3Z!5#sY_vvg4XTsfT+s^pwHYdNS!(UtYm03|M_W4s9PT^
zN&{(TONq15_eQ!gI8(p(8L2W|iel5D)#w%O*SPm45TQDC(Z65;2XAuh+YzCdy?O3v
zD;g*#pY(e3d5gq`@{tv>(&024t}B1$>z{A9oV!RyLVnLLS(v_$N;e|iC|%MW(p}PZ&wzgS
zyZ`-Nvs|MKX3l%g+5PNiyTtDQm@MqRm9VMG&)iij4^#_G7+*3Iq2$vTHc43!*&!Q-
z-$lbM?6+QuWo}8<|GW5qo+8TzqFH7s_5Nk8y-=oS>=)bj3F41#b`s`~_tq4jV|l#?
zQFF7oX@eA0LVWNZfTHKZov{m_gfo(mqG77;BtbP$4B`0aw*BALKiB~UEsU1J4|z&Q
zotl`FXh01}_T4;amNEpc0A6z}TkO{tXlE0vv*Be)LgENu$99i5o?2Uf#l7^8FGgi2
z;kW<7Z}z|1;GYAAwd10al-FhXYt;@lBt?c*v|IdRO)BwG&6e3mg86409h$Ou3Y5Uk
zNyow>A#E4Cr`Bh4)rV6ro{}F%g)t~)w1FZ~m(0J!^v^DX-v~fx$t5lq@ZH^bV{Vy<
z!P{DaMJaHj7#Jg9;t6jnbu92bCSa74-<>pxueXR2CmliT=OjhJ)#IZJ3iET5sZ%)O
ztHgERnD^~}Jp?rfai)IFTD#0zN+z;21#!j8=Me-S>>}t_ZVoC72Z9TB>J`VxdhgY#
z=VxzQba2!`c;KkTK3-5b$5JF2_+62p$!5qWF@n}aPPXf+)jy~F=Tlt%e9gsL>*$@A
z98C=&MgRA}ao`8u8eFN9u#cI5?cB`hhn)Ke5+#-TDBgUI^me4UF==*nG3lsWS(Wx#
zIq9JaH=5Z*S$=S%@3c54Zk0sDN`*9rQFl0IdCv+oJNqycLKa&P>)s
z#?Q$Cy~X&A)GW#asd1NHcdCpJG3RmR5eb9Ede9>xeaxkP&X)ci74~ddT0q;IEOZN<
z3gL%5Ek_9JuQ;-@iw5m87t%U(-1|(bd0`k}(2NZB6#zo~F&zKAfeGU^91kj8&U&v;
z42^bB%(QTSmqKl=c;b7VGYx~m?+ZRhW^kppOTkFetuj|VWdNicq^VUVlYn-^=$*!@YLRD+RWY;k0gLxq+p4tCvq252s#{
z=oYpusM@X=ro8$UR07(WWsm-#WB#oqfP+AW`2I1+y@NCH{9-CWIZEvNC*K|t?>ZK
zyg)4?63pCd2W@FAfPhFL;3tSSZk)D%{q0ZXy-U}UMjZOu{>>?e^^e#)FS#dTx?|5d
z4d&jysgV3XQb>ObVmAJL{oAqWGJ7T&&qy$P0ydB4n{YkbYuyqFUj!kUs5SmsS~Hd#
zLu$nPo#nxEZMo;5phOF1awX8I%+GF!;!>
z8q68bd|B0TT)zMV0owqm>qiufmEd2#@n6y0dxG$i-(WImH(H*$?iD?#QQ$8gL-&0%A;@;;c74G5!0lhtJDN
z$lsw280K1ej&={dfBrQl{>yE0KE;+0GCJ)lN2&jQ5Ki~K_c4n$HQPE;c)4dV26bi7
zl{QA9M3yekJ6jrv_w{dE8oV6+*Lqg*-;UA#dV?}D`V?m?5NB8?KD^C}ONK)HAR4E)
zo$$8}y~-2H1VR|hK-7Bl@QsUAIlk4F6X}sY_t!ycq@-QI=-2@|?IZO58jOE51$?&)
zpzL+u`MAVS-t5x1q!Xu%s}ETw%-Yk{PqGJ_@U&J`icn_!K_8I%kWYx0I&EncR!;HH
zeH!xNr2;uAL7xhA%A=?Emdaut#fA{IqH{0jG!m%+ENR?I;*4GE-FEiBru^@5z-JO=`dBMbx&r|MtPr*8!h$8_fB!VXr+CcsL-kn7n%o^#C>dl1w-yG>57(6YXW1Ng0^O|>caR&MzSlu;TFtMI#>HOu&
zoj+d1+Yr^khN!^i%%T%8k?I#KYE#i
z`irHa0lwOnU$oW!lFrv47QeXBg&@&7{}iBC|0uxJY1D-@xza=2P14C>o==#56XQ?6
za=(3!3@te7c=cvEvm6~YwK+cI_^ytrS-%G^VHtW`T_kCNS>dx=Mv5@d3!?ycK|l~h
zg(H9M4F8990i8fk3hGYR=zsJw()`BYX3Hr?WBIz-X$&w<6@ovJ8i>I>nQi?%3j@us
z{4GfQe`NX@+Cs5hDNq0SlTqt4(+-N!$UVwg(OgR>#KmVa(VQ$kTKB4Qp(R)xmg~Qp
z%6j0LFuglobWT^#yAik#`IAcV!=fpyVy9Og5@Am7Jtq38&J)ag)CQ#G-huXg
zR(mi*xqUS=g%$YF@|8WH!{+}geRz8YyW629iSX-EP~N`9>9RGMCGD9;w57cNmQ*eK
zO%O9!peF_w4emyf6yAHAsbbca!pqTN;50Cj|JlJgRDv`)aW_+|&u2b<-#!`>6@+*3
zr+~|6w10Y1gR;+P2dNZs{NfPePIh*7(ECn||8FQ3C3FFrXAkspw_&i=z(L1RX!x{-ISqUvw$<_J=9VQLv_)DI
zks@_gX6#b>_%uAzPOVk&)jl(r#EPeAtw8s=iLb)Qd<05^QdaNgdpJypIhXbh;HhAwAK>ERs5ONc~>5};fl?NS}_nesH=L>WH2rZNxRYj|kj`O~nWDixy
z=7c!8u0-GA5hycm4(TZz8^(1OY$=!!_z~T*w_4zuSg>Tv-97j>W%O^I_^JJ!f?(gexc>RaggGlqG~R;B*b=F&K$Acv;7~WGa*RRDr`k3MuU4
zK*F(9+no
z4%E8RWrZ{OBKDqR$ZG~8Nrc`$!k*mfhdJLd{bjYkU(t!_pf(57N>9cFR;Jzdov@_$
zl4bsI^54Rje@qUH8%I4aDzRor_PJXSX#(2;p(tv3lu$uGUB@P4FtftUFf&{!E=s2s
zpi>T#u_3AL{PKp-NuwFE;eO%Z1z(}HT9Ev8kUTp;vw`7{%ka;q|N9N>2hL~~x5j)d
z;hIjWab3KtxrpK|q%gt2wJMG)bWuyLItc1q*&2$q5;AbE7w48RdBORncA+giMl?j7
zjDhHo>q%37w*z=3X^PIwP=Ov52fz5D^*ajj+!nN8_-Hg>RgeRj;V5=c4~$aHe?cP$
z{&0%irB<|p+Q!Zjy>$Qh5TLa`!qc2+K>w4+aT>kL(+p%W7(n;zWy=Ql@64R?(2e)3
zv+wlGLPp)q47W!SSTC~~3cy1V1CSbOU|I_uU<#LtwVtYyS|84j2-wC>mjtdlr4Kq)
z*jCsS372u(_aJS^_*nU%$;I!RMH
zFiBH#{X_Sc3<(3X@lVg1tOu#B2C4{=Sgw_pk3D6P>-?RuBIAg>mtXny3QrxAn3L+C
zM0dYrky$Q?4mHV(lp|v&kY!qASC!R7)L};Tp)fDFT3&Duth!n~UcHTny8iSXzi7dH@X__!6ZWF+4AD=+W<36m_
zz33hp5=SQx0U4VLa>u?$C15HXDxK+rXS)zILu&7!BGB#M4&Zn%T>(!79y;hEO8&XN
zQsYs_o$Q3()e|PP9kW^cHEHR+{L;_Wts*$Of}@rEL31E>3>XZ*bMX=DP9Z$g2qVOU
zRUiMSj{C$e|KR*lcS%0(dbMz=b8&6-vKaSMD;BjZB>&!m$7L~FZZ>q^$K9
zEs>%{a)^fSQfNt?%i=o+OP@th9W!2O&YY??>84}DZ!J5tNML0ykU~BdXgl$#1&mtT
znE0ndsSyap$UipN9%CGhOmn{qO-x;K=gx1kE(f5hHnYqm+RPNss^0+Y$c)~6PzLfF3(4Y%#}-6CM?{`=yPAtl{3uIGX!ri?
z962>~vyy8S_6w(>BXzacoq0P;qnOX0DCjvcQ%g41JVA6r=)<#tM9s;JEtI>WNsl0&
z^*_%GFv~4u^==7vRlK9n;`LJ3;sq$5#^?QN1<;MS7bjMex}M`Iuv8rEsXf0<(wAs{
zK@<1Pw^V_*O+dnDo2msSS=w(Dw@Nn$NXo-wSY#hiOl$*?4Hm%7k_+Zsv8+XJuOQ`(
zpP5=fEtI=HNUO!z9$oZHa8*6)_Aa_z{9*?VMF&n`@^&aY><`xVtsKJUFYte?fKyzy
z{KcKkUn=HzvWv_5Hb##+s6I{5Qfa7hKN#{W3r=#jU_nV7dwhC(ogS0)#dGREerp)a
z9r8`n+-Ue%Ve#NhBvN51@13+dg~5-k684&7XexrhKqU(S>`2g4pq?6MSfP}Cgoihh
z*S0xUP&VxiyV7y&W~;+~k&xcwuFL+``bGH$uVix6$TR%zv~&O0E75G0#HKb~0l!eT
zY_b}oNvwvThM-^+lUqydLfTIbZ*%8>2_ULaWP}
zb*Me)2pM{4%Pp!j@Insm{`G!`PJxCL$lU*y5m158N4x9O2+*+2B@Po37AW0V!6`Da
z6Zc-6wmBCY6sp>{xz>JJMif-n`jg?H4kLnW*Vl>b)U6Aj{1#?Dm%=TVD-9l|jklUB
zWeRSPc#-rqiSa^b>PXr|EVoJRdKAmxq{Z9xDN@8*Kvd!QzOmQp{ERek
z6_v3(7aJG58Wbt>--aW{B6WG53cK{=J(D&r%Wj3
z`b&)r_>bL$jSApGzwIr}V=P2f&%_+$^7=1owiWA-UJ$EZ
zL0N~X+*=u1Sk~%Vchz#av|sksBA1_ZTa_KK%moT2?auox`k3g=*(x!HA|w|wkrP({
z8?3z&XpLXifVQaMz)g(fB_{{l`F3gJb*8xr4vI1nlq7um@9%*0dmHG6U_CzN7f&83
zk{JXs1g5OEAl~5z-zjBG+5s#zd>r622UP&13rf=}+_J!pix{4I%`3ZK&9^MEiKu$b
zh$FeyhblJpB?r|XnPN;i7eBe+&E@Y>m~tD%vR|ON_@aXHFzF_D0VGh!hb_7w#>6K1
z{?}GEv}%p}E${S>^wiC88fqFU<)5*CXx+t*RE7*c{Z8mm%`;iYK@(l$Fog+N`2DMnZhT^7DPu*
z1KgIAuz;(g@{RrOq$%CM>4sEOD6KH(OP%9p03Bj5I*#cr*8jQIOtvUK%PDu@LAPi#
zKWFT}X^1X<$aehKq}kTP;OH
z){kcHTmAkeI`rG+fz$T1_C3%yHU0)?r
zt*9eK88-D3_xy){uG*hD|{sKINPXIb^`Hxm|o=-x#i
zi|D9ve74*B_=b)Yz=huqnBKm#B;bYmcuA%Tq3NzC00^QKKz_}s&3ch~@1yq3y9koK
zUFH=*<=UOO|H@0Zcjf*a#KQ)kTSfq_)t@dy#2foJ0iC4=0Tff=fN<{xoUbir9usqC
zLgIUShMYK=+2*fHNz0Pvg8YRqn#eNin|5)3-Q)uW-650pg|rmek2DnSb40Ww#h+5J
zFVbaZrCG)$vTz5jlZ_0SX7g?HH{
zJDZE1X(e}=PFGjitRQL@yR6Rdj%A##rgsM_L%c=#Z7$g-PDATLCePXTC%8uf5}pe~
zcj{-~b=OwvIjZsbxC=XWqLh&&o;iFabr~}eJ9!f&3jEkY+tR#yXv)|=-z-LG!Bjqa
zpUXUgf@kwyQ4l`s2Ly*$jcQ_BjUOS4y#Vjpe1x*{H`x-6F#hB72LVP|G(YEEMs6PZ
z^0Ge5L54gz@I(=GKF+uuDK;}zF%Qk&+0gz|_y4#UFb-m#uboEwMW&Z=^Lp#{8qF%+cWq=B)vrqqVd3R`~J~g
zUzImUm5xQ5697h3x!3A1c2fg~wc|BNr|LBa_*VJ
zR{hp$M*d=6MC;_1{ht(YIIVXS4Es{|`KYt4DgHnRp-hzyvlg|F#X-kn&rnM4nahXz
z@ucBpjQLp|u5#dUe+d+2?kZ@^Q07HEyJANtHc`xZ4->gpB?$8;doy^?yof^CJ~SiA
zd0Yzc*mQDv5up1+8KV$-paSr;0BAn3CREEZi3NdDQzYzC5V{?@YaZ?$AlgOA-Z8*__tQMQBL$3IYXc==SkGlmS5APt%dH6O?b5d!6YYs~7bWiIZB!HOjO@5L@?lgrus
zWLOcJHnl2syOzh*Vxv*tclG2kyA8J)f5^6Du;7!AD0
z3jKX=_~-u?=R161xtj^n1F9j5Z+N4t7QgZQyvwBjYhql&5xfmB_3ch6o+d=+-n3
zf5{-W>i`gVW;pQRBpx*pjN{-m?jkPlP@fi6@7CvD$x8ittN2K<<7^&OO=CdY!A7V0
zxy+v=j*;vvJlEGs;We3$IR4|2Id~>ZoKO+&@h7FhN`usAL$!gMlys97naF>1g=DHb
znvnC5a-6VdkS3x_-dnp*q8wW->D9l{&SisNB`I~S4_#oiDwyygu44pf%ek;Hi!=j}
z#PO9N4~of%nHnPEZFUFT^RIq!FU?-rhmBV`S)71Az66;|*~j2)mY_Qmt%41>HDuO~
z=UYHFJ~f_u%sYk|6vGdPQsok=)rsE~Wq8cs^yD6wE{&6dyQ=9bff+X+b-nxoXkT)g_WpNpqp__L
zP4R!dHMbs5?`?7`MPp0RTywyf|IQ8)(1oe29cl*%ZIet_;t@b
z?YzsEegVic!k>h&N2kLR4NQnNw{}=}pxfv*TsxOOQm$U_$@#;#v5_9%NKY0s@a%uK
zxP}*N3oX>>^cUeS%L`fmzA33jav+vkn9c|l^ZZK?_4s1-vqx6AO0W6&;_i3(_>n;Wvlb3yxPiy6{02_(}5D
zDg+cx$RTiL!yjIzy-})ZvFh4@CGRM3Q*W0{F?3=MBA5*bt`eDO9=FpV-SV7TQ44U
zsp|uc2a-)8khYvno*KzenktqJ{c`K61%O}XzNx=U*5L10+d{YB7v}QI%k5nh!xId<
ztth`A$>aU|nerbeETfK#l?IuTMQrL%^mKpeE&3q;@Sh(kK}}71>;``1Q5V-q2L&B^
z5{-PrOGI%b#UN4kJE1_Cl%RbF$SG-x8IO|S6_5gnerQR|tI10PfAA=2gl%XInb4yK
zDeF;?-e@@r#PGZE0)5_5dpqM=CDWPW-wIiTXDpC0`AvS=eTuoUPQEu^jJ6baUP(8y
zEsR*){rrC^GRePv+698C92zT3r=hUaGjkF6`RIHVMLGplIVH5
zD#Z(zO885$9V#|n6X|SsG>BF#{iLbt2HvL3;1hO;1@_pcsU|`YTP593{V#l>EeS@B
z6gDw>y7l;A7*z%pt8{ZHH^%iknFn8ut=8#xq_`aTBQRKB=uPq!ccE$6{#@L>so
zWxXF^)fEQyJS9^-EJ>Y?e}DlnBRrT}pmQ&7X;5HJaT!OK^QQp~eP{(F>-#;_V7O|V
zf&#$!`>=^ws=Knf2fZMPWNmlzt>`|E+iwj%@YB+W-1?Ip0_<*IEY6(omUz&pB{|*H
z1u1_SPmb@YKpo@cQ7xGFt0qGDU>DAg6aH@H*J~
zf+KnKfmp=-m9#tZpCS`@63LN6rx~|u=>h@JPj|DKh>-i6_m&bSPw0ZTdlWR_$df-o
zDk>a7GdxOFS=qf+peLwN*B#Ft0VqaVZvcFh{s_~!uhfqFYg`>jz<31;*=tZ9Ts1sI
zEcGzG!t#oA%XMp{er#2Y;o&yA893wZZV
z_;0yN#BX5~2MljCm~HaSCSAuUaj1T^Os9vN2@E8Ke}Q!pc&HY)1RZUjeRMKP|C`bS
z;PON4eR&=fBc2lPgun3(HO?ED08-dYMf^-%9jd8&6RnfMWD4A%0>5#z`V+u=sXXx+
zy0IaYK5S@y-+U%EC&OR;y$fpqnB^7g(xesH{dlZ~V6?*em?8bKHtcY9HnD543x-qk`fLp1FJ7_Wpa^cq+$7c!@Z$RXzGtiJ)I_E
z2#5QsoSAr^<*FhUx&>Y95lK8$H?>mhDFN3ctiNlgg|A&MxMj%C+v)FSLE$@&FSjKW
zxlLQyPYu64CtB)^@Y_QZX6+ahF!bHVa1Ps2{vJ=ZMbNdldEX1nXQ`z8S)5~uf<29wd;MCapz6`mH9P0CGY
z%X_a`CK{Y_c{+aEQ3TSiPiBT}ls>KQvq0@yJn3(l!OxKfc_Gs(WHc
zvUAhiusZ35!*tZFID2zyc#TQx@S(L7adSA&NH)YkEGz!9R
zoHytpo{agr9Y8@_v_NP1JjNYvzZ&S^$Lr0{i}IHXAmMy(;OAcvi;|?w!@PY0slvfqU_wz4y=M#zQwy^B@Ny(0{|
z_7bGr_0PmLCP;W-BP~C*6Nu}VFk@^X4{$5G?S
zU*&eA;^4uC0KWR}ZdFa*CylS}Z@S#}_XT1yl{sTY7adL}#%$VtjWNljOEN!Q#GMsV
zJ`q;(Lc&?b>D(qM`3?k)P@EC~Gt1xU!9Sn%e$^lM;#vT1#DFm#L3PWKq_0fgDN*=R
zDSK+8H)NQhGY_BFY+4fe!wR0@RCI`@VfVX`lIOiKFN7>qq^m;K%i2Dx$wqNqmDHE6
z^*vk^xFwnT0Xlh-S#**uVBEWa1R7=*CQa16CpcuTAIQfM3J4hT^qdAXiNQh$|NBD9
ze8@cYY|Q#7kLGsXFYlK}X%!SDl$%jKU8_{AY$Hm^k6E@(q2YRoI&3f$e!w}GvTt~d
zz2_kRoYlx6{v8btH@4AST}hh$qaytqZB8AA;bU3m-B}xxAk!Wj@JI6h=&$QdSPQz2
zg}79rV`1&ZAOo`0J+5aREfOr1$=FS3ix0ymMZX6iq}uE(`cmw~14JfV9=+6xF!b`w#p#$m=`G
zzM^NhR?nh7_*8Z6sMbMQjd!{{Pw=yyem@WnbPS{I{&l;Z%TQ~fbC&a04q|_urlRXN
z)$*XTS1s?TVgkx@A&^V*$n8Q1)JRQ&(Ys^oV{rp`@+eu#iD_EW7dOxWAgBL&DL?cX
zXZhBDRMg`m(DNEin%v8p3x}qE-sWmpVZY$uD`NyMZ&~An+0o#Bn5st=YsqB|4B|ws
zNL&9wU^u+}&jq6Fg?{Qm&~J`6>FfINpWE8ubDiw|`FnBi(jujOnIiw=P$y4Y#z8{5
zYc;7^?PI0Yx)JAogBh&;PablrLmIV~wU%~~bopf#jF_a>WUs}FIc++YOIDH=eZLwH
z7fv&?jC22VwKjX!!NQK9zWDCbLp0>60s*s7nUi{l7?N`C+
z>-ek8a`PFO>C1h07lzzKi&9IzR^K*d8fft(r$3UOq74-Tp@Vk}=v-p4$9*LT)S6IP
z?Mm(2Kh8fvGCm2g0SFYfeU+7KDq!QKXA34Toh?kTbH}viCAGTt7EN+m0=$q1#P3
zTPHt%-}1Tu*B|g=H7uodu{N;88r=hpvyo3@y8u4Q8@eV0{GX>J
znc4;|YFfP9r)#U2>5|O#E7l*%>0ipH6AiCc@K7-xp3_Ps3?AvYSe6STdfMIoY6fIM
z?_;0qhRc$DZF~1j(JiSA6fKNM)4-b{{%h54sg`7m&=4~8^uqK
zr=rrLT5>+9-YwGBTw8?eAN-M878@Zx-UI!BFqM@22(NR~66RzQ&zYhHMIm8_f!imR
zTSR&>`=n*Ax#1?6!s9fmj2cT8F*cE|5ixPRpF>`=CYA}26b2QXukt&d`B5A&sGV#y
z51RB9$gJh0+UHYr0#_xB3-Rj56C>>&`ncFK}p1gO}AU9q_;t9@uTPXDoK>m#kJl+KH4
z{WmpNrzSondch*+g^H4?Feo2|3S|*eTQqtq1ZETsSDIl=K9+>A%IRy$wSQ|$@Lu|T
zgT-f2Sq+~7CtXec)r{INW}UOP#c9cB14gh3zg0KW%jA`elpOcb8oS_!=6EH{
z3}G@`_Y#8)eXVA-koq=aM58{lZpJ9S*GQzFn7Qnpdfrp}RQdEd#?!bfk7h@WZ-xnb
zSVog)2~*PfS0OcT}r*4U8W~p7vrA6iuW$?`>)x%0Ih=Qe|hh
zdfr`#O*zDd+NgikpcSsMijtkx_si5+Z?Qj3I?JHgE6NoEGaL3{vz~sD`Bryo8_AO
z6!NIUwabVTo%dRuSu)p(^1$xDgU#;BdY-VPe#)g0XL;+sOrM}Sn5Xa_$hdRBP
zdiTrj@a7DJ=-mo%KpyKKs7`7YoVdS%ef(zLE}-qS*t;&RFEKh|Lri=Y*ILUq54@U4
z7Q`zF+`>on6Kf^2@LQ!g4)@(Rmz(=<0^$EX4%j2qnv36`V0B4nWgmNLC!kH7uE^v+
z`BMQ8Y%D05N8UX{E_cA_%p5wDGupP5m6Hmr>CPP?#vmgoq
z%t1#ZoG&!XPVRRL>1vr5lcC1IVy)T5owyS*MUE&|&%tlVISYb2UH3SDzrWY7*!%Vq
zd>sv{kUGd)-J83<_7TC&>;N_8nyKkudrj$cO6jd~xARUO+T`mUUT5^<`B$%3-$e(6
zyrFPmizT(O;7NQiNF6-H4xDM|J7r5GoJfOBm+`U}W*2L6dyQ8j7?JGLsE3iP|i!yj`{IGPU
zL==Ya<^O#x%FTXuc^>t!GEr_IPX*s>z-(fSFSW#RdQIJS+{Nl1Q?i8lIA?{=j||L&
z&F4QiJOiE3Cov`(B!cQYGL9MrB|p>ZJ>JP8mlh;95vl#1$!~vEiEnsNa*(dA9$oMp
zZL}t8Eb2wCeQr!4;nF@sIbGWxNG&K6_~>E_RGGVLJpDRH_@6#2Jz%U|qp(~avainI
zA4*$v<}Nw^{H?uoUA1K*KHqGew*8?UIi|F(^7wX9&)&EP1#R%zbgqoy{AqU@Q=IXu
ziM)fm)BJG-B6nHFN7~RHp!{GIv$D$1yV8%yQ~J7ap|W?mp3-hYflTNwGkqm?uTnoJ
z(e>!Ec+Zd;FYAF1-YfjUV&tflMD)}Sua>;~KrxfxzWt1#KmXzhcBI6g810|K1s#yK
zs-aBz2k(;ta-ba&FLJ~hqrPzv!a|%ditQtS!$EV`*^EJE*)tnDR^j}$=4#x;cXLBa
zL12XL=L@@n^e6z6ug
zQNKrFf0jY2?Qdl&pGh6nC`n%@fTD&Ax6f^8M`3GHnplnOe;zV*SaL&^N~4NEzTi^X
zDcuGuHw&phQl{Txj;%Ck8AiNtA`D{!=6B;Y{#G0ta(TeIhvZYU!HV<1nODXS3P>q0
zu&_B{>BRzXuY-SS5ThU4T;C%a@zis-R2pc6#GgRtf49Y+vhRzAqSRNT2+@~B^>=ck
zp}k&aBk2<7-;Ia|L{U)Rg<@9WwGjd;FHf41iv-OUttm`LwmRV_j#5%}?mx?{5R_Os
zfzJf;|HqilTboKKJ4}=Obg%K)JY3Ceps0hu>c__r?tBS$VQ|9
zASIH%Kr43tX9a)Wgqc7t(J2kDPMWwiW{kGJ7(HKK4H`HW(BjL?$?3;34wMm)y!#$QhGf8lta%*f@q;Uk22V#rDbRxfHU9Nc9z04d
zD^tD0tWF8I&corAeK1Z&mEy;&@Qu^{4wmZFn$e#{%SY~hwi;Sew_n}Mg7qaE#ChYJ
z>L=)_&f4)Z%XL=aPBS&ds!KI{*#P?YLd0V`*zi^GA$Px(qLI~=aa67ut}=qvVE)eE
zlI!)C0}{Pa1E&0v;)gejQGvlx#P)zX3)x%%sxxi7g8Dkf9E3TH;)Lyr|SSL
zz6)Tmc#PA!uL-QwYC;v>l?q58A_0B&V^Cl%M8onbpgviDdBv!!XAfv%Wv(tSx^LRo
z7(r5WaSHicL?wPX!`TdQzsb*&l7A^|JVk7H7dpmfBY1|qdw6QhYc^T>Oga1=`nK0A
z6CkffCGuR!SK_Mf&Fw1M6apBy@nuu-{UgCT&2Nf!s3
z{`@s5GY0(j0Dl#fu{+?bP+2Dmo_qL=-VpJ<^+xHGpP!$0MycsA0H}do;ZMZg=
zjkXU;jkE+0VqSVy!u!BMw{S$MS?A&}?h&mA!9+G^z89b}k_O4rx_7Tr3fM_H$V+}N
z-b$BWSK+R@7++e{y8Li=27idMNR0Gznzw};4j)G
zKfKx;BxAH%V?{wQD>gHi6fwfo{V@k&oJx~tL^ZD|w?CO;{V@cI8iu5ZILCPtpGL!M
zyS>V?NmBmC2B~?d052yOaGQh*KH=%fqJsL5X^$gq{`I`Ccuc-Ju8UDc^tU^2A=0k`
z%n30&8LxHMG`0tnDu7E_-Z9el>E5tQYucNcQwHXmqKYv#olm5s1w_*+jJn77S
z7SQ6TwkPu>#Y>I&SVgEzmSn3&v%N_8UB1?*T($X=b+z!}Q@Q=YZK)Bj#hQske!&l$;G8;^QqrtaT?E{=a
z41hk~&{XGk9vOU!pfo65K$ZYST!ChW>~%WwJ`{_;Jm%qH59IY?;dZCegUn9fauMvj
z9gTY92y2pZ<>YFsXvz;vAqhN@7gsh!yjhHs7g)hMa|2Jyy)62QILp17oAb>`CLvgQ
zoubnj2yaq=o(1GJAkEH5A;<);6y!jki;gi2g_mgkc^nvoAAhvQ^{e~E`pXyvQOx9_
zK2UV51Wy2V8ssw>LrAR>x*Ac^Z3xqUVovJWm!IF6=Gr|f8nracnrA_w6sLMo7mVE(5r%pd8Ziv%pPn$C(Ub!=>;p~a_NiPEkdZHE$4ATB)0|X&}U9?;_MMOAk1zh;(v#n|c0p|^3R9*!zlEiSf!3WKu
z35~8zj(3V04x@uNi+^eZoDK`Wg4v=0CpkTdsx^Gr3tFTGEqo%%O%E(g1!r_wQ&8qJ
zE#_ieg2@*~PDj%1uh?&v$14=$A6n42>eN>M7U8R5v71-@L^e^gS!3cBC%U)~J9yz@
z$*Ap8XelANzL|i$cDr|>^$v9zq2)xSYX>)S7%Um}xWiw_sMC@Pbt83-H`@oVm4JWg
z@S+W(H#KK#7+Xc5GT(Z1A?eE=J25Df8vu^8D5RlxXv`*omu{q4Bcs13O)JKT^a&dW
zd@n|C>Dk?!gk&lX)Vl6u304NC{X_@>tycnI;z&)Og+{Tq09#i~GnSw%me3ANqu$el
za}5qK?Tls#PEzC43*!4AdU=e;fEIk+zjyFd7xT1Pcf1!;&KKx@F)`=FC{9_3I4sGoj<6l4^@(2YV)KJBG7cT&OrD;ANC~BX8jSz4
z2ZL8pnpWDAV58gmKueYY%=~gPgHkjzqF4BF2@$qTUJLyg;WreAFK$vTgj2!d+rq3;meHI$}}$fujEylorHW4arwv&SuG
zEDKp0kt<;cbLfHPZQ8rsUry;yXl;v)U&+*Q$wka4z23AiE7-?_7m%6DFD2!Q?>NpT
z1=mi$Ico!*|23!t>l7I^Sw*FP_0l6Kt!!-A^8~=T%tym)-2i7mB0%Z`J$Km%?=Zr4
zB`I(DU|R4*dkYB9?k3i*JjrN_;6#h2HC%^!iMDFk9F(
zc{tr<*r_Fzx`776~PjbDoh&6-ogkA7*u|#!Ekf
z?ggYKcZIsOx#jhkbfeB_G+NJ^ip#C3YK*cl_&7(-Ag|B`f7DjcJ+lbf*P`s^TiOAE
zIbKZ>UY<#BatWNeSd}@1m_v+|yNPPy_a68mq#l|k#%5Z%vJeF#mKW5$qE|Jtahwv-
ztlEG)XUip>6}`wYz0sU$a2yWMbGE3K{l@{{?UC~
zFtwn)aygkP_We=0me{zab*01;K=jy#yu1Ln8@YW0j1P~ngn#$-eK{!z
z^unjVqvZn#`)3sBH79p0Cq<7Nv7wsbalf}`W)jUdG0Aw^0B$pWwgwDiiL3wsWi3ui
zFsd>F&f(lBn)C}^RVQ?b8i4bRz*V8HTRWVHNx|z>=>%xY@DqS>twRLirA9Cjo4iiz
z)gWLGjryGcryM1u$CFFV^6kgv1cP$KHB#WyCOlX^j2+iEd3XXl)0v%_Y~F}SX7qz!
z=9zsLM=O*yttSg3eGk>o9Z}mg$}r@Q7HfC_YHwES4X$>pECN3Dt9OIb5KaT9Q8cgM
z^}!}I-Qe%(=9i*w%ZM)}qAo82AA<2rueViJr|p-B4BloFU67uX^;J-MK>EWG6ww8?z*xrI_y
z_r*o98Mz}-aS>?OGVVqpz&s_ZnJiDxD&RPC`r`8w%)19#U_9J0p44BM0!clw+(NJ-*k+kdVlziql_To%h`I9`$1*$
z0TicblW#W1<}VX)3Z`f8b#`~4;KTb0(}`{6b3hk(|2GRYO&+aC|4;7T>jj1NE!&?V
zV8~al_VRPBb$a5H!ci%@q=l|z>y521W3z
zM@CvhxN}Qi4rhy~IEhz2XQUt*T4}%>CgB>x;`Gib~ORs6HD9fp~`L
zTC{&6ECL+cv8!Q%$PT*)J*Eq2vyBI}`uCi(*SzF8LL0ZFDDR=Zf7d`o#qk6h-FP47
z4aPd$y^+Q51c)`0ft_so=OoO4j2e*>7>==8xAMgvi?hSc;YxNzzJ`2NU}LP-{i<0K
zG3Fydze7hfl1xZpop0y*tYMQMn*HyUX9`C2X{iYI(sqT-$2oud+)HkXahYDd0gJMhj?0Z-C!bMvgfH8V9R)~p)&2ZE$j*y
z;w)Y3@Zi0WA0N!6H?i6_g6Rr^jjpnDVW8PWeOTyplD?wxjXmx%!rV6UB3}af(Mj1bPurb&Wl3miHJ6ySC;78f?sIFoNy4SI7($fBAtB+Rpw#PgN$k;qM{ZVWTd2UYa%IlEMwu}ASV}>u+iIDS)xoW`*xb}=xF0ktRI-xi1Y|ZilEw77%Jpd0uXO3uMz!v@;
z#{8nRZJ1^@`KD7S0)ukzgws|5*czRD;qAPLxpM`)~n=H
zvUO6tJh}hAE$2R7!!3`-`>FSNuoQ(~M%7cKVg&==6&>|Ga3U-9ue{5+4Jlc}iV&e80*HwF$m4A
zNX!f^Fb>4HAD?Z@=@qg7F5+h}i*_(>Q>ytBx+)!f3TAqk*}g%ilr<-u>+>3R^@1>Sr=zIUYf0IspPyddS51*nd7V0{P;K0E#ADQ9>CbA
zuM(~GYR(@L%=wZPg+nUQV8OAn++kTBP*PpaS0NkI^7ydG97#+Z|93+n*4}#6|+H+=hJpEN=@r}xKN?c
zkuP6WoYr-*h8lvHSc{)3*Lqiql8}<6EZ*dru*%qpR|n=D~wzd%4Vk
zC2qWR_SVPlfW->JM4BncX(Ie(Bv0c3IQP0q_xRqzJfmdm#sk7r5)+xPz>MDuMlVM%
zzFVunE)Gkrqg&8aFntO0mZIdPd&$ExZDGtI|0j5~lS4SWcPRRpC-)UQIwGV0ZByL0
zo>?vQJ1%$Y5LYi*+?mIe^l?TrqDhWvBvzRF=uR`f{pVHY58ttr-
z_c7}1h+?uDllT9S-$&!a!NJBmG|yYRoaV
za7*!px>|3yu>Nw5TWGS%p9)M+II>T;ljQ+)^(_@NP1n$S!eVO}?{{2wPnMA=(Y5U;5g@t(nqaTYT}MtaNThC!63yl2xm$2=M?m=#ua&}Cz0WFslDsjcQdm4
zE;M6dB?*2H!al2>`=DQ84RrkfhYfoR#Ukg^)hKC*8U{YR6{3;X5hpQVSy|kocuV07*%rA*9R=VyFO7wojZR3eaSkLs`Z9yW#+0()J1G}DY7Lb
zc;*x0)&bRh+ibPiV)K!*D^=$gQWE-G=d!{xD1vC!5Z1%t>BWVW6FlW$lV0v|y`!-iddhQ0uZD#L+Z9(G>(xiw{r
zYfZE~CVkky^ToY#6+>YgsWy`B<#gv-cIu$Z?RPv%rih09x|7sJ>a{q6^QSEkqX#$C
z0N&7F9~*Yh=Y21F?z2Y?qunlAX2w
z+BW
zjUoE0INyD@#Lccf^Cj}aP
zQL0@uL*n~Ymo{6M(amu(tHl%zU%jBk#bJP`qty203`RQQIymDBJmST8vKPjDr>Iuq
z!ykyjp-~0I-?|1lo#T1u76DfNamb!8?*3$=?%MzG^_5Xou2HwJ1%ZuHf`}q1(j`ha
z(%l^@A>A!4qM*_n>6Q-ZE>Suq1qG1}(%p5R7sT&;cZ_?_@SGod)V<&5c~{Ie=Uj@Q
z6NJ_R0_BBsfkBqS-8|9}I>;2%*`{xIWdePKub0Ce=B19SsSn&zDy&JX{2bwg!XJUb
zPQP5uSL*WjKy^R9D(m7>vGhW;J(?u
zSn=Ac7~t)`?MHA?WxceNm0SLA4Pm=n92>O?g%_TTDYvkWL5(-~QU1;ZCfGAal#qTy
z3WtpUR(5a}Da)5>ou0A^5z45CTtA#5oq(LCW##ELn`v-i`T9r!bU~>2HpjaceA`Uz>(g4t==DZiPz~51<NEW{wz-RH7&gm4*eAAG8e_gy|9nj|)rvAG{issZ{}~5?9iDF&
z3SD2H>nuk6;|c3#4b%EDYGLe@v0AePcIX{%B?1lDx?i$>NlD2v39U-;AHU3SdL=uk
z`aIlQBUd{3&0qO-xlNO_Goia%PLZ9-LQa-AYp0JUMvp+L8Nb6>tx#{A!}T$kT+C$(
zm#Skv@bi8`Y0?RnwDx2?v>fS1NVK@0)iJaxT|K=iT5j}t)6O)CzL&$V`@z>6h$h9k
z&aC=RDw|vT-d)Bm+!tZT>PAz;tCJqe{Y8s+@UiV<^dE2I)9))k81=Fk+Xy7RZHVeo
zt3hXMW(5ex6zRcDDRkB$Y)tVI!Jx(tr|LNe_9)(C-J_-TELa5_iwxXtNNnH~h8p||
zUD2wF0>i^Zu@l_wjyl=g%x@;YM}wiu)oZtm7$5v-0t><=D@u$e8^s;&KC&Ds>%qOL
z*%kBI#C70{+S}AhfB=m1I)ed}LH&0PS|wQi(s159mQt~Jh;V9^+r~`T#{f=X?_6HK
zFj);YGoWG7!8nPpO=yyZO$nAV;z*zG#RC2K(^!<UcMSX?ulx4QELt%A)&cPfCt9bMcb$oc+;k7tDSJ){Uc;RXFLA2s
z_dve)&GI(n$);(LO;TN5J)<_q9W4)5*b&Y1@BpY@L`p1Gkjd1^uIBc&oYJS+$30xd
z2wVff<8?8xYbdn9Z$h9%-G~YwYdZ2z$6QNRW4R2DF-Dda*Tla#>s2Y@&tOnu=qffK#X4~<{KRpX7umQ6xYjYp0Iwx|zn>M*(
zF<=F;)XCwFz;*UeumnjO%9Yh~w-xUihD#NtgPv!`>B?R&-`r16qx&prlg!DSTQ
zpYROi9D_s}1r_WSr*^ph0(=-d32+2h@VQoTD#ObT6s;*gnwn>pW6hF2$&<)%aLLl^
z(%dHqM=@}yPisM~-%rPIMAptD7Vi~Z)GYV#O3{jwk))j*^PvI0N$SmNheoVJ=B@Y?
zrpHC3r@9enc0v+?+rn
zE8|qx9r!Ew^5B$(}vF&s!YMJ$5W
zv0}Kry*&(Aq4ptyS|b$&>46JT(hIsKP)MJ-51vwqy}I^^(5FUWXK@(n7kNpDG>@-0
z02^769(1@^7SZ?XT$6Zex@rvF?qVXYy}{r%u&TP`7#;oAIsJwcdqn>;_R?2}f)mS#
z{ImC5Sd6+y7JTfiY*5m$hyUMWQb9B7Nnev?b97^EJ#60Y;Dz3eMtV$_O8V2B6qfHw
zGYL?O?Hy3f4^^D62~o4|Cs6fx(_yefa93K
z%DY7k!Vz9>F4IZws@Mwb$trY41tbok9>A2=AFlnVtsKf#?~2?dezVQHTDPT;qn!TA
zTE^#iBPhY;yE#<)f8RWzncltL4bq|0c!O7M!6MlFLdVV8K@UC16U~R{XqqZ68(3i6R$~BI&03{y
zV(@~XNtX3xpIY&ceR@{bm;#@fZj(z!Uh7TNWy7&H(h{jmAr+A|uS0ksY({4wy@-MF
zp2?L()!>+>+ruYic&~h0AU1&#Kbk~fzcrn+*dK&?`aAH@23#b$$NjW(%^H79=p1+~M3hZuxeZgk3
zpEl?sOz3fhlr2n$E`mM0Bh-Pju+j#z1)*dAiDHKdhM@2Q6voqfp+0)xPfG~
z1Cq%E&Qvbv_Xysr40`6$d{3Aq0&E`8Woo`L&n+MKP&(x;30Tkb(0uT*olk<<7+8(n
zS+dQ3#Q8gvfAh7mscD?Hf8%w!$546(wy~c&?Sy_2;EhOEPI=cBc!SN6C-0x+k^X80
z=6Xz;u0}5#X-_I&AA?k^yyXQB3DpBU%U0U8bcR*2ZHMaGiCR
z45~RzP5JTqS?e5zpOUGg7TPm)^a-&1CXZu1awQwkuDh(6hd}ZQJi$KtKsh$Gs*ZY;
zsJfB9C+al}r_jZaFno((@(UNi5fU3_+z)1Hm&gYx4DX@ep1S<8(4(-F-VegjT4dyC
zlY;5uWUweP%nper+HuSr1}mjz>Rv|@e)P9cVeT6s40W+QhbB49k#9cSL!oI0X%1V!
zl@1QmewE<4T5T}jVkAs=D<#9vWB@o4)lbGZ^J0v!DBUX1uN~oR<8@xt3JwE!>?mSQ
zuI}`3X{76gnKx*xm}y?AHqYdneUyEAx7fc1%r@(KSJ~$SxubCHCa$_#`=O1ra^5-u?)ouJ!R_)*c%8IdBI9-LTdkw9ioi-S9Ha-&xCVfeY=W)I6OlNC%{=^b
zu~Nk&_NtFXdC-?=nWw=K%=3uv^Z9-uB7EC0f*q2>_0vtv73mS^4=h5bDL?CbpX_Mz
zAfr?#v19M3qUkYBlvR-Q&AJT?s|IFui>q5lQN}b5N)K{uXhA+QP@n1dhsMtMjGctT
zxNup^tBI$hR_&`RLlYBuL;6O_Yoo8q9|JV5F61$!J;w5Lupryd?K&Nf*_*wZlh0LO
z8VhPHJPDX#cEhq^pNH#dW@k+kUp&u-lbndN+H&Axg-O
zlFR$pExD7tvVkf(C}tNz4dv-yega*og|^~>%Bh~`O8Mq@%OdoI`%KRRszS2jU0v-`TmmY1S{W;PQ#u_Dqk$908w;68s&&^mysCRhhgz`DlcV^PcISG
zGX;mi0&6hp54G=S4IwNx<`?>=o=I3r$zQhkr(?MbOD7%`6v6IslqyJnWjpiY(I4=t}871lf$qM2q5P
zBA;soVfcRhwK>~*0A(Ps8vwyY(nFiHXcmjx(3kSWAO5X7$6BvgQn2ASS98qpg}fi6
z6i%?(qge4tqXVX()qXPE=ICyNm9QY8K#d-wU-q%GQks^;)ejT%xi4&ed6^*lQ(nI}
zxgl0UKT@ta(_x9C*w7^S^P|-Tib-+8^qc5!fj=0mmQ2B}O4|sD9wu
zHjyPyzree`5F0H24&;pDP{N$nOEfZgRa%)xX52v_o1+98NZIjkm9~b-
ztDlU1_?K!Mv7^Pgsxu0Qb=wNbhwZ)W&l@>{5CbCScsHj-Y
z9~zsPb%E0`K0Q7_+-U*cYBGRDGdmlpc(bbWv%6N{r$5b8bf-t!6;`pN+;&a8T-jhm
z30ul;*(RnpAfHbCycy9q9BAUZW42pg~2{Dk2tn2JH4
zzu==1qld{irz0!`lg+Xp;)Ge4QqdU-=pPDO~2;jb}#-jrDDx&bfR^nrD3s?lh|#Y%MppDoerL&;;~y
z87&X72+2{8gxD?QU@Jp4~|u?;|Kign%w1?U2xEo
zrwNl$dFtCF@u+Q0e~#xnO{g{qAYTByBD-hf&NBRgND+{Pg0pHn%Aw45`J?p;>dB^T
zBdOT;x761gdG~VH!{MCTpk#DkDj`8UQu0n3mF3pezyJ54rO=@j_cbkI9O3Me6E_P6
zp1_X~jy~T!<5f2TJjR-~qk}2Xsb3jqJS(&lD<2DQy9Yk@#tmQO-AOA85mNUkH0rPj
zqa~!nUwX{NNl<*p`a^8|fwuRoIfUH{C+vXW+Qz(^FJD!Vdb?V41gP^1ii(NlChrMc
zeH8xnNA4n(!6m(h2~DGClJeV6Eqc)V*)8FYGRU7}kf9Gj8`+ka{|3ZwxhkOl`R3>fosMrz_nnKklzHqLb
zuvCzCB5YbdV`qrIb21;#`dFrts#5Qe(#cmDT8O2?MM949lK%AWOw!Ps9V%*GCjobls62QoGdewJGk@uHMe?5K38i1A;%<#t
ziAje+M^CfMZ%w5qx$tx~Uq+RB*M&Td)jxR^rvsX^OEp!CUfu5M;xtr;SFZU4V5qhk
z9=Wl2YB1xNy>}A`Xx&Py<|F|9gbse!+07reIJ*hL1agVOUQt@r6B2>^XvfMF(`}hJ
zO1CI8a*IXKLzl<=ree=O6{cK@h5C`mP+Mr*_L<}r)(?-sHJEwjyLG0?FH2)%WBrDu
z6}gt38W-kG|IyI@$@|`UC;q(O<9b4Q@qKHrm&^_i8N4&$W_e+Se+^#2nha4C!RZFg
zUjjrUIVsUxRf{LGZQ-0mf~I<-!Rey$kFm;|rw}l!Kli43s6N`sV0@cW
z{>xwJbVCf~rV96@0!CGlUTRCp4b=IG0`=x7sb%Fkx4>XNE*>Y4ytgniPL-w6@h~A)
zMKCmOqx$vi+4-B`$4xyghMHj*_RHi0{h{{VzIiu;(Y}Sl+^y}>I`6`^@&vem`hG3^
zu9`5}WwiIF^o`ML%s~lxpTj_^hU)%tP=e8Gn<)4$$Dexn`2~+EsuUqJCG9U%9=NUL
zAK_*>{j%uGkvnQBe*>j&WRpjDR^{-
zlGt?i?Qd}+*k^V-i=GUPSlL0r3gepOdSeZ%M*%zu;_^`L|7}kA7C)
zTd2!TD~9hXVip$9Eo8MsGbH5rv#2UxcA!-R*;x)ng80Q|C)q8Y3~x125n}BDf0hh$
z2u??>1H{N+-uI}c+tP8$P*m6;4w^UwLZ)pzJ7=|0`0Z0oKSfnDEKu^Hc}M*05c$EN
z%3Gt2sBghdf-89ha?u
z`{SvIgzpo51rfXj3cJOr#**NJf**j;B95x2$Sm8)_POi9@$-Jxw(TvxLIv^A)912=
ziW<%!@duL+?aUY^qd?@Z_i!C8!l@YCOoxl+_t0q}S&hueqOfxYA-a#q{?)
z1Yz!Nn>6!Eg`@&eX8L=!LR8~RxE+N%7&M+MBI6}a$+L4tOB;;YDB}I`3{1)*jBeXR
zNZ$91gS@z!jy~9XT@M@^AJ-MY_wxD;(HrmRc*q1h4exW8NWU((|j1$A#vw>0sj1tIT7!o$dUi$|sbm?q?l(_=L$S;=02<
zVxfU!_-=(g1AltU3!;|#${ZFk;(E$B=I|ij>-ZvPzC(t@Ph(`&p9H|kAOMcG7Zb`+
z`_#hdVM4r27qx`cLzp0;a2p}U$zb;j>5m)x9tLe3p3X#MYQL!T$5-Hiis^;w8QyI&
zDzzr)ss3&62)3YvK*5eBEK+8^VW7=r8v|yE-&8QsDcD7#g)&<>iMCHOi#`rw*gG12
zV{Nkwgzi}X+3;HRb}jSrn|-Py2OoWrU*EEn9T$`v-#F#yD`rn`NQRg|M;YdZmO^pp
z{LLe_!X{Hrr@kBfxhZTC(Vu4skJ5fF^=D)skse}7Z-0HxIT(NM|9cxWnX!{&%o!<8
zO0H=#&vDeA8ex;0UsppFdf2%XJXiiN&7mR$&qE$KbIeyLOaV9Lzf*;%kJS>9P3xH?
zcLtW2WWalINxJLh7=5>~*-swG8S3-lhUHAzE}YWzty9_YpH(p(nb&rc)=N|rG`Wu!ZpL@k
zD+HD^Oa~J0vY*W4RYeIM>fbE9uHFPShzU8wb$!#@0)z
z38dseQBO1HHJ^J?hD~DaE2EfWQ=Dvezc4oyEd4w@!N}m|^-Hfn`qlQWp~tsff5-H!
zZpU7jSSe}2@=RC5%x&|dRv2-soc)TDg0fLY?ZEMD%d+ZG9v*5k?ECESF*_|q6D$!2{}3cP83yPsD#2gSQ&;%3o)i{rfsDM
zY((q+J@+%aGTTYG9wKnOmSj{)Ca4<=cyC->2WaKnfqNUWvX8TUyHcZLAYiPd4vm~{
zG7H9}+e2*NA+%+IXZ*_BHM3~!MzZ>^n`TFwncqzQka!s@1=&O-JPskJCI5Kw@LD7(
zScMFK?puH47xao)=388Ruw(M*y8wgHw;|8ehF7*!5{fpBS9j*5u_>+)rh^ag3%vVE
z`&uQm7uj%(7B2(V?vxlHo|U(r1Sh6=B2&7mp1G$_D=mQ>H0At#`I)WIaE!^L*OGJ@
zx_rL>;`~a?umvZ=$wQ&Rdu(_ngML=z9ct20+mf{305|ZQLGRJ&T>p8JAt7P9PF?(u
zy%ydya^(nFvG%o#xQn=UCu2|i`RndbY9->%l;Yr@+&v0C@=-U1?~|OzcTp(58_B&^
z@IG4Mf`Coz@{yq-wH%eqU?ONh0tP=!YC=|oyndgawRM6CfxvaJk&_bQJ!JH>18xX4
z1}@pQaUw9TaR945JqO>00j36~3hGPptA}ltj_=Gji5}cLp9O)bU<7nRMM=F74|C{(E;2&Z4LTjXplrzu)uSqo$^2@+!IE
z3cCdF;D#?Y#18<>zakxvPd6uLG5{ENkTAd$BuR=@EHovQUokL$4HS`stwcCyfyKe%
z!r)p*g-W`(yZS$~$_VQJ^vN+x@j^rSy(-OWuqSEFbpG453|?Q380dWJc3VA=rL&*C
zHKl`YJNHAj?&mF-OjMJtE>~yf@2TjWqm-pj9WSC63t;dvSavbF^cXMajWM>%uz|OS
zHrJQT8Y+00{SY8a4PZu8WyneljA`(Q(bgld(+pjjzw~euS#>7ozI!m+$;GRBa7E=J
zIQtRM`i9~cvp(^0km8F@N`dWi)K{hk|6Id!y`isZxWSV57F@ljUAd2p7o)|pkL}aj
z&u0vvEu!?rB`D^466%2IJ^F<8bJDAN`8CJG+>&4xH58&mO3%N^o)Q+%Ws3kdsmfS{
zVta2@Y5dS=e1p7Sd+#f1Yz#1r=}iuqN76uG=7YRx{o9w{KP>=dmu6XUEQ=Jjm-$Ss
zso~=ux7xrF+04g3yobSiS~(1LS^n0F`~?bP2dA6ODvYK6<4OOp6MNWLe6i${A_vUf
z2g~MT6T#wGQ8Tx~q%SK%3mC58eu}Y)`$^b#HxDbTBADXSdhRZV1IbM%0KTb$aPDqK
zQ3JjPBXGw51lR|ni$kfFn0H8_rYOzN*Bk4jq06tsDENGQO*Kj$)q`~
z!I=w=h4+;7(7Iy`t`AYL#$AanN@5{CYuxo`&HMM@R?vzKpG>u!BEz`UZsYpQH3Tt8
zi8(b?9E<$W%#XNnHpG}rW2pBcaXQc9vWjnop)PBW)fNZwpm~z+@@mySUHf*F66~=n
z(JDsZ^zW6fqP06f#>fIqmetZrWXC45%i3KI>PLhL@%nKO`*j*~{O8#MGNM$VaKOJs
z3%4Q2Ifd}m!|xk
z3qB12hNmrLuj;$;mhH!uisu(j6xTu|%KOtmJo_3_evz)8eh}VU{ZVfk>ga#2altb|
z+ZH<=wlKKcUAuq-
z{0jAJJ$mk%p#?IyySs0K$)om&7~lgCy#!1aX~6Eu-+E&wp$riF3_!O{by>+tPE2G5
zyFT&gvhg&E4488|HLaoy+8T#s>1!vYVOlRgnJZo)g
zPZQu13dW12zS~_zw;9gDZG6zo&o;t3MZQ0!n
z6TBz27E*&VD1uESERz0I{5%Z#Y0;vHzf%R9Vph91V}I7V7EMp40PW?t%He)~HasQ2
z8seOSwGUB^U{5r{zunMF-v!JL-A@nZO~#6!APB_f`HH}<(hr%_w*UqD+UFk7fX*9tj1L-_{>zfSDzOv+-MozUPm@u0%g9FyBD^F+|9mm(`~1^20IszAJW#O#UvpqOWBS|p^kV7FzgS=^
z6ROq_cP3zU^fHZZw*+EmiUeGB0Q&?m)J{_Z0G+X;%Kmo-u!KXvko)%ahP5uXCQrSg
zkrYtxwHIwl?iD|4!>I=P2qsHvtvWQu-)*5W;I4G6|N0DAy8Tm-*bmH1Zqg}z1k0J0
z@kB-g)I&-9e`W9)HhR}*_5-Pn=P`qi68Ro33rgxbTs-ZKI^CA(j}Npc@`{FzL}ruy
zS`QdvI7@F`L^(iRxaWIv?I=!+MSY=yVOk(s<9IsVWL6(crCucsNLm;hdw@#n0s$6g
zy3quZX%FSt(f90>f+ywJ++NOVdx@Z=UYP`X9&(w|r+L^4^&UGVdZXxkn0i&$$@sN^ChE3Y
zq}vmsp{LsP-!Uhx6tZ<9uPyXf7RG+Lto2#}GH*FHMe0x_^V^R&m>jKjO{JOII(Ilq
zVTJ$%tR|
zJYcC<7#0!*WNYO9f^sBmM!wLV%W1M}X7gYYuWy&s5A-2OpoEA9vgWv9Bv2YflY8$s
zBeHIAH_ie2Ol5E3fV&byJ(#ME#PS!f`jvM922B2zui*al6_2q}cg$oUO&lgZ1ZL}ucI+7FA@oM}E9IkydO_NiRlk9{D%B+M
zrG1JUA23EX0i*^*^dMlPDj+%9*wfgxonS94g3;_
zMT4Hb3iBqN34#AhYj;uUo*`HbS=_?U0L{Xa-tKhm>{QX8=ge%p#N>g`>5Dt$2{c!b
z{;FWK1&y#Ode{6^G9ir|;5{go8+spz0~$Y*qK+g`?a@ZkDvn6gYZfMeE!NWCD+1d>
zU?OMSkr=4(P{zWl7M{&*?#yb8mPSMGJlcDwH=LrkhqH%7#8C
z<(c`8zh5fMyoO>3_2a8%!gg&w`cKf%EPpM^kqhCJSLR
zm5EikA)Dx$^@xOTALf5)v+ZAa?g&rfH<1O^+niQ3{&dYoU#z!#9sFAl^{yN?#q0r5
zU#M(jxz>fZqcBWaGzFxx5p90%&^G3wBEu<%B@MttTAI9_VE9aDtts(4FuSG!m%~tr
zF^Mr)QER570cF{JP^F5x-@q}
z8J!jOZ+>uUaX|y^SW9Dm7fiM6
zaF(80&5%Sa$B%zoW;`m?Wyi6#YxH?DO2-LV{%tX~^>89&%N+ANi?^3Nlw**Ag9Q5U
zmIKF#Q&B|OK+>i6&qh%h`s)JmMuY;?6c7VJz+*cM&^0j1|3JDb>&hQsQOD3<>{kentA+t
zTWQa~8>v5Q;bFXcO!+c}bml(%f4HH4zZT{CL=l*EUc>iSIZa6>Jidgx{Xop%FpP-&
zw03Zl4zQKSpQ(C$DEB#BpyfH){)PmkJ$$W}{YW5~J7V0`<@7pqYDCr)d+x1*K0f(R
z23Q0l?y@(C)}bIeoY&M5kdr&V%{s$aK6G7GvjXA&HjgYajVDCh=)
z)9|()X|Np>x57=XPgl9D=qQekkB=u+k-U-Am^-``$_y*0GkpDJ1jemb>T7J+*j7^PMnE+}RXq&>zoXW!i3+FK;-`o%8?5rc
z5=ved{R_PLJ-?6QYRe3gCx!N#||pMVjN0rnigy5fQIf%ka`g_)$%E1o_whnjbe751`b
zCMH<1v07=7$RXi{bpby9Ei>phc;MML3~bs!U~=P
zZ#Y!uWQY&>p1cj?K6=TMD}sXbBz~d{g!JkR(;VSQ8CEjwz?L`8=|RE(wN5j8kbw2K
z5+me5UG@ayK*zh8iquCzdWF#+fHBC$?BFqq=hP%~g)2)|9`cXgmB0B!(JAhJ}TS&yE86%iqhpZj6QlJq(tp
zGR>G{a4_$C-6N&R#_Dp{t`JQP8%q;|?Hk!spH#LLBRDP(jSau4VXv!wxUhaFn>-F#
zX-V6C`AfH`eCyXb3>&!2u5sCYhqbH0%)j6gityT0ls*)%xb}BiDn*fmH3>WdK#^sx
zOGU_gYW%&-9C!@ybp>Ot{(#Z@L_|c))@}7hf)#K3OCx#9ei&q7VDGC8U=dzPazv9&
z2hwxB!unkuhzG^LGo%`k92zRLJhPO4$Fra%bK0~h0}WnpCtFtn=c3)!JVsL)jL2~5
zf`1Xhle~ajDE#O^^#Js9s)NHEhB4oF62|2&n-jeoO+SMK`tcIL%n6-~*v?x&Y0QJ2
z*ZA^Q&P1T}`tyLl)%qqIkn?R+k3f6afj*vDVb82a1TJb|;%=2io?dmy
zi{p(pXpzgGTtr|f?JsKo3Q$fHYo$*wCqooMHjD)3S=kEPDI0CBm9bBsOH3au(a`=L
zZ(?EADZ}hQCe}NP}fjzA~48(nT8-Jjg=3hcbklT!qD@9pNRt;42PS`f;-&agxIyu_I
z%|7nv`iwy$PMmOk2=ZnD+s%=npeoJ}xUY~v2e!TLF1`TuXazr4=2r-~aQ?6#f}*{_
zN~wdXmaEC4^RdPKvH@wutXlEuVF2WqPj{O9EBm|%aN4qwmm$%s>4gw;Z)%W#Qn4V
zHlbPY>7zk5n$T%d!5op(u@h7ZgzuI<27$q8S+k^M%P4miQ>{=EJ0&8$FZ?1T$^a((
zDfBZncS!%5;b&k~E&E*`N=WI$lEwnw`_fa=?=Re&@swc@W&l`;-@oz$7Y}RmM7xot
z@zN|7_+kxEEcosS9xZd2_ow*LLcO!m9eq@Uh*%KLu=JA0@)$dNo4kxn#5V=L9t|_F
z#5E!rm+Mn=7bu6A_8dG4fi*%DKqIbBiCi1Hmv^g58tcIqKt#YQQDKSv8{y&5V|0E{
z3TI%uw1>xf++>2ApN+6(@R{E1xmM_ujpFd7MegE&-R{}_>8FHx%R0meptTk8YYi-e
zHwJ;5DOp@%)<;Id=VH~g2w2HC=ro<0NQ6?Qke$)>y)3x~^)&?{f?kJXgT*Mfg}m#S
zS(RoW279l};8wnR9s&R()Jmlv7-{o9)SB4y{3qnju%E)%YNy4ef6xcn*Wg
z`D?e>noREolmIxj%OpnsW1f5A!omVVy%)&%1l+N2GOf3sh#ToU0$P3-IQkRy(oSgV
z0KUf7kbb>KIVh54ft}FpVrGV8fC6&H*eXFyZrg|pK;IFWcu^`kNgj*zOQh6V_~;S9
zfky1>Tassab6>kPC?#$f6Qy&p^nTQ@VG>+S3lrYQTK?gor6E`8GpSVMC7v^KkK;Jt
zi*{%&Cu!7hX^C~7#S06f94)?MmOI#<$)^b=@?aWM-oqaCSVkuzDkqG=q+~EdFY{rJ
zT2_9P4E#6&uk%*EEbJ6IR+7vdl+Cuc=fhQD*oG`P*4BV5)q{XC4cIS_N$Gp{iG(4Y
zJCuhj5J1KE3Qlnc|2#)fe8LhwC;=44=2&h!I{-LSm%@LvTsk9*1w)4Apxl0B
z5#>CG6UEtprXj{P^`i9c9j@Ly83ThPO#=sq5@gy6@YUFgx=!-oob(O4`JLHn@#3$b
zXP?p-s&+Kfz5NDgmd=B~8``AxsFTy9HH4@Y*pwL&9{#JtRl@u9z^r;En6Z_=F&l`4
zHIhng_%k{f$X1^&7d5Y7{~0qR(!{%t=2FN*H-&3x8s9m!`dkD(?A>e6rxTke2Fufh
zx!i=pf%S@|LI$+)#B0um!jukOrkOIoZPYWgLoEikw=Dwy_{mFAJcQXCC3lVXr8;+B
zn#JR#1JS$iHyCCo8R&#|y6&o_y#y+fFzcUn<2UWduF!a6ttYju)gwF8B87pJg$Dd%
zw2o2p=_)d`{gpk$uqGOtVC1L|-8IiGzv~2
za-n3X0tVNKM6Q}8!L_=#9kFE;kklUCKOQQ4K${6`N)=dipN?mKuzGs1+1xk}&rhMd
zr<8y;x!YJa>u)wYLmKaOnXAk2!7HdbOj-`qAgSBzL_(&MIs~qRY*XVP-pfu@SX{O6+jSEHDLEGQBZbK9y`T{!~1y_JpRuUFfHvaHd-
z=t0@e8<+6m(r8gkz2}PI=8h+i6@wgkLf1Ko+?m
zZufKsm0Ipz%8H@N5&-FR-k`ZQ2qynQLnIuB_E&3!C4!n~?RR
z@BtslL>yf=f`+es7qNE)8M*^33ZJGIL-V@Jej$p{q42I>IwXTY@a=LiQDcsyP8iMsA}$Z++#AIgP6P`D=u6}E^q
zHuthANFh%0VKT&75x9Req^+zQv$AL5x-8Z3wssA-`f!CMLOFK6#0~Z7C2|fjD4`H)
z)~2K`%0k}gwQQ8le=Uw11xlbexU>2FRrb%i2(kd62_tq$#ms+dM03UUIiy_!?B*jY
zoJ@krCJ~bKJ|d6Va7km(`8{~{^;+_Z5D?y-CMVil{OmS2jpts|HdGBv_?fOehKpPkcH_BYK!@GMN78nBnlu
z2Rr)qn}SLj@dZyWFZFuwnmt@EwE5AGE4cg=zrVhVjO8%v1UHxx+Adc0k`Z834=sk;
z;Yt?G^?%&w6X%sVd5*TCzjA}6gF%yapRYdv761oe*$@*pwbq%iy+*Bzb(g$8)ICMW
zvV?KQzFbw48Fl`HNg6$ZNHewYOP2*JbPROiV$vQfu0m{jC+
z^xv8RhW8v`CloIklyaL6F)42~YYUC)yCxF(=NOLhl8}%Lf}u4dVMXX;b4iUi4q4R-
z7p=2A1$}i?!jX~ec-8Zw<)3y^W?fH^_iefc>s2bP#TCh@|H3Edof$r*5reza$_kbbHV2#`(m!%cTLr}kW(hui`gMOcUQn|JtgJ(
zW5XUDKG)Tu>XRM!m1fK)~+6wQwn4H#wq;b+~-F{y2TNGlUUO68pMkDTcCrtI6&C@r~P=@^U(abPpU4qy!S9%
zky5Jz{PdC0_hZ#%WlwOTGoH+Ms#HAHkrKvP<<0djb>~-?lE@rXVURv6B;qOT964#w
z2AI+hbrrT4Dm$vAYg7czNn3#WwNa0vT-n3n1?;u?1S(_M5h#DofcWALdZv@ra|s#DRohTs$l3wF)u#DA!o_YV^=*t|f-
zL$h>@1n~GzUa^@0N%CmaD_5)3H#!+R_3C+5Nq)`VUiEx`@XFD9?(lQZ!16y;!}2|*M-?Tl=H)W`v_2}eI{En^wRW2H
zHeJHpi#i2UpR!)iy5|!pp(Hx4NvqVVBT8Q{P!VJ+rN7n0#%|
z+6_o_)0VE1s_d
z&p2$R1Q`AM?%pTPP8~EcT7|vc$HE&ez2*0+uIhwnlPw$&A<}uw{;wCTNzi$u>xio_
zoLP2K;6@X{z4Ql@4?qxD3AvaT_Mwq!^AfmWto9V05)4%Fv0FlU9%}(JL3x=J1)h&nVSwFSz;Jk}*T&
zN6BEmXL*XqyIR`@(smYZvyUC->iN4w7Krc%L@Fjn!6zm5&1Bl!D>plOI`+0h_WP(c
zy2KUBHfGsX++aa_bM3&S!TN5~53*Ih%V93`{!tua-OGSUG5X|c6fhJw{OL~(SW!p>
zcgX>WW10tJZL*wh5Zr}@0jwHYx1rxwS
zOVjAsPIfzghkUghn8`B_kDAP|78227tHXJrGjm7FJ+Pp7wDYr0K)BMpPJ&*DN8hKP
zGYQE;NvRIoqr_eR6HEY6Kc?yJ-iVv_^pK6d2+OG<2EBgosp(Hv1cwSYckNNM
zNsZCYz=;^zfi4i4dCXq@>um~{B{g-I=p~+joKaDib|1`r&&DPjz+>)uiFKncN~qT!
z7&8iru%WL`PcnIQ^P1`<|Gt8LJzv=_Nr|nKycE~Yfx)NSp*7VwfF_I{6jdcrdZtN@
zZrP5^(uJoZsa-M_-l+~lTgc10<^0U;F`G7@&3Rf{V;4?v%Ca2z+y$T-Gz43Z>fGEr
zQminUC+&=^PR_4HqRV8MeQb{|a@|!q{jd_3WFTK-IMv!6k?`5QL_`}y87_&Up@u!A
zEkj+DNMmVgHo@b(@JO$T`E
ztr0>EhYR1!^O`d6>V=uRgCKD3r8qgU!0$6^G=KZL-bq1v#yl~Tq>rT$$zKEn!oq})
z2|k=5?{{$MywYdriZjF}#M~0pXU)?8CS}A-_|~bu&QC30
zpVMe~WPT*V+_n&W#D&P(B^Vm)M8XFsd4f0%8qOaZ(W+B!|L>=1Debt^hGIg))2PIKRR^Q%zm?QXWRric#|O(uf9R7wMXB*quo8gS%BA>Nt04w+KT6Y^O*(aEx$A&AFJz6r
zJ+-Bgr$+WNt4a4sMQ;v&v+gMwy<3*)MZ++Ha~dG9tz47vqgvcXA*rY^mYfPTVQL%X
zjoBY}%(|NOe;qKB&Jb!P{!aldl@f-wtk@B_d~YO90(R4Byd=>
zRyV5N0uH8QoBqCj$gKl58BbhUJ
z9_u1Yh`wb^MOp{M-Sk!@69{9JR2%zb(U>1st?E+viE+s#pO+%}=U$NBiAGGyp)t*W
zPxv)Kzsxl~flmLIYABgm6OE!Ff2`TAPL#GCb=BmkW&xKaH}gYRXQQM5-QRXFcW{eV%N{
zy@q~tY3m6+f!6{s$=!(prw4iC$x|UEX=NqNiA$
zU(FBrJ6XS|Lue$L4onan*6~_S&qb|NWfC4clOVIpWz(EvTK)j
zzM90i6Lk%>O4QcThFSaYL}LPN*NVk%!Hm)9+~@yq_NG{b_T&0jw#sk#ZTa*H>`f9R
zsIQ9wtdMWNQ=$j-1h?tKTXYD>IKlV3Qph#Wfd4Gb#VlgVWpRY)$?RRir?-)=L{0M(VFWt*q9ND*fae5d3{UqOHj{0(V`KKG*taw3Ut?YZgZ3*pi
zz!4!a#^Rz&eBw$P;TZ+izBbU1{%2jA|9zl@=+AC3WT&T>u5ns;y5t7-0Z{4=0~dV}
zV7+lqDT4wqLqq^!({!n_ia-e1DM{&v0!L0TGBC4*qS*IAarukqp?pv(@ZaQWb)!9B
zLnq?fvJesRi*ou|ac293|3lYVKt(BqS^Z|%Z
z9EXPd>gT?)M}Ez1eKF2AM-!GH$ehveq0f_fE)*(SYVvH8yKh4MP^>LB4#KZXef1v@
zmigyPegV&IkU4B;bt^d;cv36&Btq&t|j`t%1&8>Ld)~7p$Wfz=6
zF>ZALZ1l#yKCk9K
zi2aGE&C$(hvx|h_=m%$Utff8X$=xc(f_;b!s6@GGw7VC0Qcsno@Z-~;iG+ykO~70t
zUDPuu(ieHC^x2;QuI}|l=83s^7vTO_Jpqm~s2KQh;D2MgDOgpnYFG2}?Z$x(Ndlt#
z@N?Y{C9%`(R({W2^Tt-tfAIj=cbMj*wvWy>KY!{uCx0)-Hs^raHhckm^|=4|>dg31
zsy>(kOybs4ob#1Jja^Ws|14ICw%SW$t99LVSx>g-NmjK+1UdCIJknP^TOCaI}lpb@xs`Mq-yP8K%&e;7vN$33>#vn8yaKPC{Q;rY1X52ChYI&2&XiYE%8qVPn*HZc
zrjChncu8C?p%Hfd5A&{uf9}uwOvI{*yHM-8#)=@WWG^A0L}An^d?0zc0#0;F@}S}Y
zdKpBWKuaf
zIpD$U17eB}c0K}2ee~>ktJ4Y~ce`<7>lrdvzW@iZF1db)oys`TzZZcPLIjqgpL$BY
zf$o|6eBJUp?m+sidwGI4*|K8fyCc5!ctc-lm1HicIW?OVX4&26z%l(vzy{P9wyU;&
zeF<;5IXvJ3|1oiEFu$Pd^G!vz+`K(ovw@jQgRC2m14pl`
zlS-77G~x38fY;#CW-OOQ-Gvn0FJ5(O#mIj6D)rm*XRTphAHIJ!8P?MSlc;S%#86w&8kJ5oTTjnq7dv4jH(NeW*E5ZL-#
zHp33Lxj*lN9xe{hWSE-mpX^)ix~8e@1uP|YL2j}egvjQg%olh|=;_4pTU%hWR
z@IqcK?47OZrbrd(!Kr*qy~yXsA|P3;3Uht06LO^_l0!rBaBC7sWj|09SBQB`W3ShT
z9Ucq(tj0c@HWr*(gxKj<$>Kkui*NOge3*9!BPZEuCqea=0hLQ0Gw4l?4)6KZxJI1J
z9|G7cp+pGXGkiQ%a_ous;rYeH;MF-+ijnB%UT=LrJT;Hs_{D37d%+nT3(1((t6>4C
zV&WklHTAEvQRIj|c#vRbBJDnFrY3e^n^i~Jn5rJSGv8Bx+xgF=1%?BFPNM*7Ms_%L
z8~Lq}QwUF}LuPYJ}ly3VNEP1A5O{wzpPV&d$`KL2LON({tC`W~{yjaU%n_RkWj{6e_a*`;
z-VHn1u%p~=#y@*wCp8%8J4%%Rbl0tzr_OE8{cW7oLm4|L`E7S`Iq_N$=4;!#=Jtr2
zPwxO!BT2VxH5qivuHwr!9^`f3G1Qcpcd?Ln*SB1s2>dpe0>ADE(9M88e$|v3CFOx&
zFdC*he%DLnF)#;fVrmHxkyIH%$jNn39)DDPxgJ3M`cE7r|0bZZ-u`=?0gIR@{*{M%
zG*go=#Hm1p`0_GMiNIx+Zn{Tn>E)@j^T|Yunqx3K1U7*-2bt^O3n2cBQb9Z2@Xfc}
zPy_}gTz}q2GTV?ic952Tsi^?j4zL8$>HPu*goG$i(8UV_ZA~&(&3S)5gRmB@WXWVp
zV4*6ENjva2r#dJ0Dl;xMMcm}qx#$1m*DoiN%Qvtfn4826$cS+EeLxeUbs)<4GPyi^
zQSMIbphh}Q=UIo|h}PZZrb!dB|`PS?`2b3C$BEv5J}P#!Sn
zW-?*0SQ2XUEA*E%wY2bD9N$uZ{Cz0aJz;!&=EP@OvuN}Dm(r+uqo)RQR1=z5M;*(m
zQ~GBvx&gS`Q;I)g5N0mq2zM-RThBk=(NQYk+I!(DEHTib4IM&5c%Nk}jZ6BmUjH*v
z<>XLwbaVpw23SZmsBB=^7Dzv;tmch_em@_2WUS<#J1i7i|D4uJ8Kh8E?5fH@S_6uS
z3A)NuUl7pI-c`7LHP5DGj6v5J_`oPL4vDjH#%JBlG@a%@KN;NV{%tZ0#v5@&)9Oks
z+C0tGX;WowE#~gvq7m_M%Pm_NBye4{y
zPymyKnF=|=`K6nz=O6dbod@88J!+Z6=!m=Cp_T1hKy*LJfu5g?Slp{MFCEi3yj9y0
zv!iJ%j!|%4Gd{(3(ENMBcg;Qn$gZ=(>)w
zrVb(LO*I7iWdxeyUmvfFLNR3MF^7I|6#1C)@JIkx(!n>FF+wbEoPWOD)yZSaQMB(j
z#{xWr1x*x+fsg*H^JfAE1_O&zUXR4=uU0}W6hWl+wUjx}XsKz&0N1AyYBm>W>mYg`
ztaab2yAS=d9!Hg+@qPprHC=TQ;htylHW`t>&cAlh$Bobb*kNLW``Z@_PW~l|RmUK8
zl!s?foZE?fFnu2HRR8z}7E?5#u-|T`-JL3<#b=XTjYT>PGFm(dFK?)sH@m?d-P+D|
z$sUeg_8C+sNI~_z7sbG)4ZX)006F%K_bb|m9Jf3Or(giRGKmN&tR>%7sa{_e$&9i?
zFwv)AkmAI6dxtnKy*9iE}v#j)W?Cs?B#j}3KGjAS9xTuiNNK;COHO!Dbe!v$9sG#}Dl2Jmt7($8
zucfvJWhqx(R3}t&Sz;ySqW2&?-0`c4!#aPbfEM9c>1vcQwR5*kJZ*j)=WtaMFXfx2
z^!d)rnO>~jxzicuy||P7IQL1{lLJhb?Xsi6DpNAgtf%BF+U2S47K^{TI|2!3k>1e@
z#ek06>SUyb!n?{GzB<-#)fD*o`7=Rjstj~J{p*{eBZnJMwDKlU2J#g}s)c*AZ}*zqdGIc4l*6*fB&3!?c|NQX3lQ)Pf<`z*Ry(EhYGk(wAcyNv)bhTl
zULcEC(g+j6qc{MFi=Zde7CFyrtju8PDV+md$EPC)XSEnYKv}^0)=aT=D
zVr;D}eslq0eJDz*ellr`XkLH7C{2x>kc16r0f5%Y1t>JZ+iUx6S)qqa?;Dqw9%sLR
zTs)3H&)L77mB6L$5RESRVCU5%C>5mKO@b?b@Ax>L71hqw+quIOOK8|GA4554e4NmBWv*8446!7z&p*|AK&)4UYF$}+8pO1N8+
z@a{GyLxfn$h#AxF7mVXFbjY^`PdkYNw6d31;%WjAq7Zbj5zG&my81ylZ*|@iANh)U
zllGY*b!zKe`26k{3NGRwahwB@q3{
zKggq_TD53`EdpXLw8goopLf-VYRc)*UpwB`a?0ngtZhK~&)
zMqUA=?QjadmNo6M&Z5Xcm#mM-vYkXfEE4^8vOXZA&ofL&@$yKde@avH*6+RJDbLza
zt9u(V#YS^6A{Ca%SrGP@e&7wN^0bK;_4R6MAGdzpq3CEMk8j}!+QAJxq9vVwk57~h
z7|KmrJ@n4&%zkvvnzYs<^0UaKg~M7W7!?Ix(BA2q?oFHS{-;jc+BMmhUsE$=nVI(1
ztJQDn>SMFAj6bc+qKRXFH-ddUmgskH(|$_2C{J^U=YU)!DV|;6c5&j-WMx-BleOmL
zURZpT4U2-pko=OuO@%2M)h$_k+Rjm>pZ8U{la{V0i}owOI+=295bf>liO~VDlPcGI
z^Rx5%k)y0MF$Q?-tB3K#S(zK?6`8HUuIOjM990kY^Rv`;h({zI-AQrfPyo{$;{5q3
zv4=$#jU*5JRjLGS#J3XZ$53=F-CrYdyo-cf)Yjhy(tJ=b{F6tjhYr@?78RI$V-r-@fFS1_z)~DVj
zl^G5z?k}zu^S8i&S-~rJlLBEF4fD&+XY&v__zxEDj4vm9PrAIFvqgHopEV=x-(A{F
z*y%S|R9adno$BoGKUv!ZCs@o75QWcuF*N}5Nt?)_@w6q;-8zfM(Ur}7`UZ0651fGNg#INjiNQcS-({?XZtDiKBcYit20P|5R
z!E{tLOe2BYi@L1}BiO1nlJW!HfFluPZWiy;{%GOgXRW8XuJ_UNNzSiMJnM`l&qkgl
z{p!{ouNc{bmv}3-nefq?Qn!wM?7Tx((Q)r|85vGWT!NKJdx!o8{(AXPsD8Sb+iHU7
zKs5QIADge*Nc>BGqIad)VY7|8utgPJ%q#X6J_`$9H(#H?zO$@gG$>t$M&?HY&j@OLZ#BVPYL`EeEB8G
zpz{Kur}r;)%ezse2_M%^zq2t=`nzcH-Vqn^t16f%e{2~{y4Y6wfiWEU7b4V38B94BD;2XCM|*o
zp96i02AP%O@H|Zik9hx_Y&Qj6v@8KVI`juKZ|B%M*+fBA?;mZiaP=zGujRuhr_?ky
zUO(ok&WXFJ-e$num|0K~{$LZuqt7MxyVP6t0*RY!K
zWi5i(88&Q&OZSS4oukz`H74ZgG)|DkKj1$8SU2@Cc9`pNe}A6Nv7Nr9Y4Kap7X&f4
zY_I?oLG^_DyZb|&>N*?O>V-Ovkp|gqpTJ7|a2liBtRG60zx23*wX*FzS9A|d%WpL9
z=!N(`-T0Zosh=%8l{sOkk{A9%TQOmMg0+#;Y_<|cF8u3x%e0ENwEcFU``X95t=e>)
zng>G5!KElzX!OQPRIaxc?(z}J@L}2pH)HCmSm&wfxox>gP_Lu>zIjde-%F#XSy&k-
zeQi7(d_jB0B^_!RbtE=8z((`q9nS66&4S_LPgY;0L1~h4W3n<4c=yPsOHbP_9d87n
zU*>_Z9?{D@N&pOmryHC?ZtL0zlj4#!nqrQ9ts1RdztSw3x-WyY;!kqsZ^{}3d96QI
zRjBKbI1{HfWBl67
zn>;s(K?cW?fOZm2b&t(*m2bh^xPEag4}I
zMf#(eff`70e2<)D#^L=A$E-0Q+^}laVQS|>s3!b(m^qtF{Dpuuh@zX1A%K3^J|n7_
z3{tPy{Hn4+zEQ&~ZO~2LRg_Ge+anqSJkVkw4
z`F&x_#T7S91~1%uG@-OOt~WA?WJArY?&=;?D7PD_9;Xa?#~T(7TY}wQY}wwZC=n%i
zR{{QNxL5;n)5h~!e14H%g_j`_yZWa9Ql7Ww!Gc$eVmU$--feSgarVy0C%PsRMP_C0
zd#cdvf$2xG7AJHVaPt?mVW)8$FjLP}61VpCcgEXeSfzm23_~a|^0-P=v
z3Pv15D|h^>L>J4H@=v$V_iem_>%N{pU@u6J#bJgh-I(V+Zwe4RFOub&S8Z}D6qN^|
zEQ(CjbuJ7ct8uZTy)155P4&6yeWu+JE}!Iq9*u82anY6MI$`KXJL|W|N8%%Era0VR
zQ`-qC!Gw8G7YE|VhpR&vvtBX7;=E~{J=m)!zpG@Y*JX+&8NR*dy=N^_5V7aI+EMcB
ze%Lp}x@%*IyT=M`QvO)i9i#;k%~5-3`{qbS$3#Nt9kdHfE{P0uziO)
zOD67_T9`=E?b$aB;i9^(H1QT#i|&LJfi5?O)1v|&85ckC&O1G{H*4Qaj{W=zfuD{ga|Wj*FKSDM5Bi3k91}J3s5D0_h3tOhG#Yt2a8JK{%7Lmt2LJX`yF4F7>&|0r
z0^S4SfNeu;4O7TJyvks7=5#ddlzwY7T5zh-NBAV!=S4S7A1WMHt)=~nf<
zjWd$4ChyL;#rS*$r@MBgv^6dTIs>Xl=9&q;GYP9*ub4|umQ8fZ2dXPiWFEB`>2AK5
zAb;3=szgwjlzSsERju+kF{u56v=g6f+B0Q7*>p~#e6#kwGgk&
z>G83ayHS<;Xhzp$nCKqZ8WhPaRlXW>&CB;XV6WG-_VG7UXOsIAJ^n5i{Z9M|vjuqS
zlnWdSktwMhR}8+EH|hG?LH!iTDa7miW1cRoAzgCVabQOUYzeQVbl@yzJvp9VM9rg?{i3^O>jtO<2~R@^#(kfPg%;9f#ER$%9l3mI
zwy@jHosCwNPcf>9$+9>@wj1)%)!h1?mEqT_YteJWy~lBLjvtp+)Uzf3t)(GX1liWX
zC<&Ut9w{DetX>S$~WyJy0
zZ>(RBeli_K($)-dN~SbLZ#uGC4S$bsYTYPR~Ed9huirwzP5l;v_h@^Rf(qyUL+mfpgJ
z;3%mg)_^Tu-HILA5HBlP-5aj^_wFXnEin$TZ{+bdgsx4s>A{UP-!7Mrbrygq!YH^d}$v0SMS
z4c-ue*OH53Dj+vG{__-CC4y}47PhSF!Pyxv&);@xI_
zWi=L@a;1dfOi=IZxtYJv!ZgFKyLinet2`)`v(TFuKYRJ5GdQc%Siw}GZf2(K<3K)6
zFgQ4;tC-S&H8Yw1edGEXRsjCb(j7ds2lPQg9KJ@m0|RP2_4{hJ{H8oc?^myKcK;fy
zAD9E^MqRh2zYij;S0!uo>O4wowNB;i0|(jSBh+mAH(aK0HOQ#^k-%Z&OQxv?yJ1cV
zb-(a4a-T-i^!=wpB6{I$%PyyP?2O=X`H-BA-cWvcMcns%E%)dmS4c$J(z>-_ZNbox
z&0g~vk$WBY#`}zo;4Ijk!Iy7B4^G~N_If^=BWi-F|E@N-;UFub(0lP4cSMI|&75cL
z+h*aVeiU#T{3mh$XPp3#g8l^>??sCk?jBz+
z40tRE_0IFA4LOS%^reg(;XZ6){Q@F((ITd;{v_5FHl=0rwz)6y3d|NW&ty;BKnitm
zJUYiP?^7I?e%se?RHKeG!6L~y@Li;qxzYP2!_y0|ZCMm3zbWP>VI%~+IykYNCg|8>
zMJ)P&6|NC0B}URz-1=UZu(P<+Pc;epYo{*ugi4@1nB7Xb;;?&K?o=)`j211Z7ZkvD=`!
z+vSFG43m_m>0~yk8_e&`FaXlUT_7seYrHM6_lGeZ!a``XctM_2i?%$?KF^ZX
zdbj8ola&X~loGcgSQKd33(7G|x&~$o$q~O&Bn+?HDgW#yxHS(lIu?a682sy#!p!^e
zfM`mjS#N7Q;yT<}DqCc@*@cs^Eb5zVCsv|-o9>RWb^H$&MC!8nEjsr=A|y>;XG2=H
zzBmrKYs$M@bvattZ8s=fUT5y6=O=4UcD48KnINhpAfH1LFH+3u0=A~${;*r1o@}Zh
z<-amYf*iCRO`G?r)N}d4Ybu@CFu>1A8owft{%toy!b&znd6Nw~d_3PN@TjhA#ha#D
z_MfOc8o;>B0vnJE#lPy&a#_f-piaEuf2a>it?OL`2?3*hlGzbtl_5A{8ir534N4Y?
z5@#lJs^grvQ4baix&wVfrK4FyE4$q}lWA{P8VxhYebl5m7_{4bQEc$NJ=%5oknhO&
zTv68WecP+BTx(tq&5|gw@ehyQvtntu6>Zr|H!rlCi-K`jM#C-&>aw8$0q14*N+`4b
zrxj`;OUj@{3n9?~V=x{Qzm@e*y!)?FSjrl3M78+zO=IIzmTm
ztTD7%biuk#R42o)Kip^;Pa?Bm-=0gvf!b7*xNM|wPaQGm!Fn9Nj6_W36XhND^C|@b
z_QhJCbme+E0r{Kr17>|R#X@KgQbc8O-6)jmjys(kEAdWxRMiX3t*gt^E0Hl3idk|k
zmK2*SFk)Ea`x9%aZo8^O6ThuBK5|*`WKrpS%C-7!ADi9i(;KayLA|itQnn^8A>?Dc#cB{pPdzEc69J^sO^|0$XxR7J@|H$gwqE&sLQ
z+*pK%yIAd-#t)0EVvbu&P>zQT$o44YHwjX}$s0n`c%~hZB4(|1hb4Pq=zjUaUVAyb
zXAcqwhGy)ZLPAVfySyB-1=_gVVHBvXEBk4}V!5YE+n-rJNUi@+|Ed1{=OnUN;e@hx
zRSfNO6@nZV)tR^x+=!flzHV-w>$LFU%U(sDdiTWB!%kCA?|$_Pqc1wj=_p5gu-SK;
z32OK2W?nPr?&h(`>%!Vra6F$47AHg(JR8SW<7I!yldtxuhm<|G{za|6l5I*%F*sHqzbfT-w
z0@f?wE61|PnPO)S3O@nOIS?pp_*`rot!8V<-l7sVk8v%AGIx#p=hKyxSmklLnN^33
zNXrQKjb&#O4@xIz<(2I|*|MWqzgiiOr71C&_whyA1y#Gr>^$t=?y#2yinS4xhzfgk
z-75N%1m)8_n_KWnGMtT5-_D30$&cZmhlaL%^(mprCp69Z%N6;>v6igocB3-pUAyQB
zfm&y~OYB=8e9BA@lq|eNZQ};`xa!}bD@O4XnkAooK%T}
ze#@;+Cl$ZBb}<+8V5XVh^-T2p2Z^W#x`z}##s#{dcmO+m{K@z(?9kcl`wPRw1h!qL
z)$>KHH+5ZhOdvx0eCal$O2~^Rm6qctztVu6d&?sP??#8XzaZg@|DO6$Fic}*`rb9(
zMuE)Cu*Q7Bhqj0QQ8
zzaNq`%J1$K09El_ywId%io33P=R238|K=lsZec7B;;{rb6_BB@3_3hV3x@gqS
z!7Sh~GvZ{klnd{An=fvEC+`fpnj`M;;~~B!bt2Z56x1{VUQh7I-
zv^3z{E~-aigCN#`e&r&c2!j51|HVE16kr1}1Y
zr1?m~>$1lwlhgYdY6Zp&BFfwvT3H_K76JK%xYl@?h`p|+Cl+4X(CA5QLtDGN;4it>
zV5$ur8d&Bdqt$xr_@MMImEc-y%Q(rw#z55^3`oDs_M3N>kf<5s>7tNJB
zMPmQI#xOCPUrsAoiSj^~)W`K7y7|kGA`U`Y^nP#
zs4~i%1C#Urqn!?8^yX|SJJ_&(kt`2oEB|uuq5azf_xEo6h(KEMcXU_qkiLAJ!0PWB
zu=-1{qbBa^LqiB~fEEYVm)PX|zfT1-8fnr04Tjp^4>OekZys~7KtOtbpynjaW^)J2cpq-k)VJaD+j*uUQ_%o@&8_}AaXA%AKgEH)Laq~^`xzhIl
z-!zd8vCD=AAdPtQsOPAlagYDcjg_OXBx&-AIzfW}s_ojld}yJvU`Vyk8_bDBOVKi^
z0F1MkTYH`U|D+C(W097ES@Et0JH0pJ){0Y)kxPFkD7y0gju$|9^8-#0^DetOb`58iCZfEZZ4b_}?Q2+)MDsDrfGdN?!gk
zG2)wqk!Avcye1{4UwdU}%t#A{c9R2`HlTg$nodO?xA3xJ5l
z-T@~c#iF`h`2qeX;Ch)^p$PH`2Az+wkv7%=!n9=_}RD{BoE0n&ooUBGAfgG4_tM=;-A=vF;w#s9lT{Ktz~eK03GXMbV>FP?U$J_PD)cVVG7tvMR)<}4#|Qn
zkv}f=T=VhQ7(%x`+2a<0+ctTV3h<&#|>}HTGQ9m1y>;V{^6??dxMxHtmmk9{DGKj`}{=^RT9$N()=*1ea+o2(|;
zj?*M;N}nYB57#$l8jT#z!GWp4RdOepL%+&l?Cil1eeISqS1Q(5eY(lNWAJhXJ7gmq
zw;S~L=lZAG0i|X`LPdy+vt%P_cL8*CqBDezz_98C)3br}wc^HU-SfT;WKx?n;2Rc)
zKpP`m7GTRHIW)MO9zWpd{Rq16u{6Y_O!_!B>dxO?3v)C2_Tz
zH89icz3&V~{+mb7!C#rV>!_N+EcBB0h&D27cofiAT4PpmR-7RCqhN(I;GV
zHzutG1OqpmmOyMfK%-K(!lsC-^x@)gePAih!3q$pMgf%8QT-c)?FxTjJ&-OAg1a7t
zceixYD2+G#1|Ukt16`ReKDF@sa$ji*T1N;%6%4(5>kR^+6>&juI%DYEcPk-!yYb?z
z7=cSfVG2rYvib#tjPkLpkJe3ncpOqaZqL$sKLn@y{mt)s=Rb@6MR1c1UC-c9R0#rJ
zE3a-KjiqDP`&{r-@@&j3^1cssaYOyCsjGk4Uec`VG@`lr@6HEabV;D-qwt(u+0B7}
zggW0tx<=mx{9WdRQ7N1il~H8V(}KK@Dkq=S_T^DNle9upehw1(T>=AQB*TZ#kE`&L
zok5dW0r?_tSrEteZ7aat`H30WbssllL5L)eKOu
z64<(A2*ei-(5uI34RO+Ze>np!1yH>m*?UD3}YJ2V}~xR>5cO@4Xtjl
zT#4nsQ}|rX3HocLXnprp-9OI=hO6=ZI0XnXzqKlylBw$n!S8@WL~WOR>j{d&oG14~
zjCs=VN878R@e9NCEf@2JmD3!;sd^=+6k;iL6z_R(MZs!}W%VW8_2M{rE(O6
zYab!3qFKfg)L5RleG#s6D`9@W^748Pk%aSM8ZrTwYn6F-jMO(8i^J2_F4DZ&@#o?#
zAX7+YLy}LM0BhajAT@icO_Sdv;_J`o$r|MSsNhPh^=G#rqy!rn<9vta-`&IUp41IX
zW5utuqR0DFwBmPWID=GM^;hw{FPemrzjkihEcLhIs8U22AM&Cr41+-gn;=JnAv_RZ
z6Gj*>-T{^R$ONpdAd+RIq)~|;deDqGxB+)2H}!l#ItV_?;A5
z5DvN$wO}r^(^`~n(sc$jeS!L>_6}g}?o=bxvRp+m9!_#=`6tZ+5ih^!uM)6Chxn2sh-w=U}$NW{oho0OS
zPC0MJ#lNQbhPRoY;|=78iGItV;dz9`XM$U)j}Ndrgt+d1%+fA3O{k;{Cq75>=0w;x
zm>36y+z_)i#^hreIh)r$q&A@x8rsWtSs5I;_&m97lX_!Q@|-{sA?y#RA6+VQg7>5<
zgu_??7eLm4zu_ILu*cIRp+%GRZ*~7?KAyF1H(JY+@JxxY}ha`2|?Kmcnjl-K$MLbqEluI!14AtKS((N_rF}zt$92HjHu33C*drD9
zruoOx>Uq$fBBmQ2*MB7hx(rdax6?0Ibi_|7X(9)2rl2xh>I8oQabkcidLwVP&JgT$QF-obBYmCEmjSic$%FjGaCH7)b%SKO1c`oj-_>dI&t*%6V;vSj
z3FlGEy#EOjLT^DuH>+f(efHL47{!Y(gmA9trrOdUd98G;e6x2=LLyIHfw
z)9lBuJNvZ)gv8d|os5g`0
zg>!co45u{@*2sR02Z)D$nias`5?#+A%T0!zkn{^M46d17+9Ui8k)U0ON+6}s=YLLp
zW+=S;#z_L}uBgp0M+%~6H7l$G<Vev8QlYrB+@oDwxc{c
zwQhwOJh*q}x)6xru~+vMcnNTZpA)f-4O--ckS}j$_0W1Q5RnG6Y`3-e2~kpNj~sqd
zOp_3w#FY|pEOr>#j%p${(M9-PoJGE-7U9p?d9RqnkAlZsy6F#&oe`~Ok{LkEC2Lp`
zLtCz~I&!;K=%A5uBQFbuq|#&*EI3s~nHI
zx?*Oi|6Kj%I*5(R{c0(P;`u%11MKXtU{gpdKLXWycnc2XQ=J|H&hLqKq>jzUlyGJi
zNq%6czG<-{E%J!M3UDD;b81XlugOkuIs#~$hsDW)C%5nNuk{`41-4dw{6S?h&Krio
z;2?*k-ky!yQ2_cmMRqgzkW$u8NLB@!9hrKE=8+(&cdlefdQ3%Xk!KrasB`%nAqPE+Pda
zE9rVz)c^rxiBltwRQ_?=9bWaMX(jDzJP9%153rZQ5*NUUE*7{h2rHcdySTvmrRKsWOTZD7>Jv^SGOrbChO{6o
zkMjHB>lR^fR+ealpJPvrlD-C+hnX~$G^}%N-n9SBZ_v9TG6XcgmG76ST?c$DW69)K
zJT+YtdZ^jZ#N!Ov$OKw(FX4V{h~W;u9?mbV-@KLwWTxX~z*!ZB==wLCO*|fFIXT!m
zSm8+amGhwrc|&AH$5M`SaMY~87-vz}E8#*7vq&PwK~>?}ikCP*Sneg1BBdu9KNR$s
zkeczB$FsFh2L6#9x;5M2=oSa^5FWS6_e{Svl5c@>CXS#k6q)M{BCn=R)(DS2A2{0h
zqJczEB-i#73A>%7XU4Jr=NZ<|q}i?0r2mdTMG&w#$C9-$(B<1
zVAp;u4KEC8lG8Q0ni_?JLR|J|hl$XM%AX(uht$`=fLYKP%gHS2N5=?J9^
zxhg!VuXSziX%BXOw13x8%`)>X?v3?E*l8ldwTp7&4lD|8+1FGo{>MXUXsN8=tuOed+Re5z=_>eI1=@VFw|vf-;+;W2EpGq$
zzfNqS;$UR9r}9Pq&BQxIU>Z|aA&^MFzW$WpT!d+XOWLe|zTRPSuY^IgE_4cJu)v)(vR=a)XFL%2^j2HSXHj9dz%J9M})&TtrDz~7pLXMBHWn2a&@X4(#bmMxY9?!N+C+~5DgNv1vsQY
zn29TmFopvUv*V5uv+uj7#=`vQhDhhXg!$Lt>QlacW7*7VFXNGKW^cwfu5~y)2Q=|d
zq#Z4ASRyQr{jvFvwwDQayKFr_fz3m!WON8|U@g`Ym8?CMm3($I+IY@_Sn)(f2eRrD
zjBwDU-OiQw&V!_rUH_)S5HINMi9>w~p>^G_5KYRL&kEEVtHiw&Bk2z6sbE1@EP!46
z$|3+jWy57pY*6KXl(_cEP-|O%FN4$tSK;+R6Hbsix
z{8>A8Ns%j&tq{0l;4$+GYK<%XqV^f{`|DiVvsNP@z4%_iEeQ{?Te13zy>v-WM79E}
zTa9q$c6OQ{vE75>TcSlqbYmOHXatD>=C%4GO2(tL>b2>1KsP>!g2=k7uXCu&6*ZoF
zQ3^dv7CT!B*y8Lng(aoH{JtSq)fSJ#)uyP#5EF)pCEMxdQtTt)eAAkkED4zC1f}EUbQ*H
z!O_r}x*R0OMy4M|^kR~La2Ju`1ze$$3aw&8hBQ7gcSElg7Fpl*Z1eN9wwjAm(CrYf
zWw;_7LMiY>4&*(VAlfJ}q!y2Co!Ro0Uz^w?vSJC!Zw1*4hFX?3!TfW32TVYFKvK#s
zyY=<9-K2h^;AGLr^{Yy`L@Updh>o<|q6CEz8mS0ywv{It#eUw$fvH
zSgcRiPpWbz_kAaz7M^uBjOY(X!$lSP?h+f#q^Y-#P8#-}wk}-}!7)z?=ZM&|1vuo4
zNJztB%g&EU&}@XHOd+J$JT3IBHAaWlMvQHXtggcN^b2=`E!#L_o;F(9m`LS)vrxrxMDmsE0x%D0%
ze8r?^FiW8?b4Iw1ofZ(dn|D>xmRWKM2DanH9q-(@2}
zZ2PN4J+&Kd?ZT%cF%VXFq9fVlYIv-Vzbw;Eyt-akv+;rjndpC#)L1p6fwEEHZLHaw
z7SK1ru`qDPJ)j0Kw#}m~F1n2QY#YhxfxI#tOfP(0^9b+Kt-T$ueRAQrgRU=6Oat={
zn9w4c6RUXPJu`5p&*^%i!D-&&$ch$X5n+1yH^Ro_zW$Vm&Im}#&5*U
zR4u@cHt5dtJ-pk1c}mc3yZLE;tQMx-pnPy<<7kKAv&}zF2|jW;4J?O?t_phI~
zauTn-?@OLw725jNN>#buLeDUI@uu@#Jb4cQdTXaiL^!x%b%sekTYC6OUHCwnb~UKp
zxjAehgAU99Q|puewySVeuq4HHlC#u2i%w8;ZiLTcdX!qKG0dZ-vKnNIuQ
zyvWRb(MI5@51S2Ul6!F-PBY=cfk!v2zokU?$@&CoD=2SU%rm54^77o6k#Dr1!fCAC
z=vBOB2lVZp&F6`b$KQ3hZ+yps)>ig3Di6Q+vaSElR54GICnN^%)t71^H|gCt4z%O-+x0JKJ;_&%0UB^{K*
z>XW8{9HG7pzIzy7R;cI;ZhB=<^8Zli4IVHKg5=i*=gE0e%RppKHzl0U>}xA9CSBgJ
z+&SM_0b(ex50(`g_9=|zRV1(xX^LEpWWkdsNN>~}54qKjI%+V1NFOT@6k`SV%_|AN
z{gLq*5P&b@A@GtM1>-8B?nAs}0D2RtZ*B`{mQe#Sma5-EO;nw{uBUS0Me-vxpu~ak
z*6(prZ^Br+3aVFc-pQ}EaYzQU{8lUnZ!>T)%nMz)BqdC-HTlUW0VE~OCW60FQ8ZdC
z?W|Ax@>nlGO`M+;I%-fPWaQPQqG&U#B#|XpUY}+Kti%d&
zj&>jOcF&_%Qo6UP7%$$CY*H_BKDTvueWN(Ak&KjhIdaDApi!`KCx~w4LwG`s4o~go
z+j;e%H441@3GzzA!|WCyI74NB!DsY_ZsW5Vl?S6zuU{f5w|E}&NM|A~LF%?2R}rmq
zddp}J9#tDJT3A$|)oIx2wLVBZ%qd5t|CDATBQ&fe+CqzPXT
z`T1Ivp_^!`JjTft9J2q)4EKVLhjD||$ai-ZU#&~|N-N+<$8@w_auUSV_1lhpdPYE9
zY!Zk=fQ(=pNZ}Iii?GJ|6GSZrrS{Z*Pu-~*KENQZ##Zj7{8KG-8vVjdaYeY=Vvb;NuG`Py%#iLhz2fQs^+cQYm8(`7ulAb)5
zaYGhAIay!*cuQbA%M=k&u87w)#$hl7`fc6PxtYYBorr8`@meN%E_nngcp2k
z&NYJ!OFKxqw*g`2Mzgx}PWpc8A=_RD<0qm4x>EmX&DTgy6x%U|sge*l-3Q_^P$ra*
zWQ=lcnx{(gq=R`ozBZOZtGu3?YXlnrIs?#RtE`sPM~5!gBF=nWB~fEcz}E-;*ql!C
zhYyn?)a?&Hi|3x^_@=$6247V8aEH7j3&K`r{P0O6`@|w=YFEehvBr
zupFp~{oYAYTy5PP{K4tT>k6OwARgQV7Ooer`ZjSj6}J885TJSGKTtyG^+5HHs{QjH
z8`lX6Jtps$=@b9%EZn$E5XMVmEOlQ|fRs6;8G(Y^4O)CA1(x(Nl?IgFBvwXh^41#R
zzf86tTVX?JjQLc0{DQv?^cRz6<{BZRDYLlWr>IBAp=9l8_oM9rQ~k#~AG_CJZTyk7V}0pbKw(n!Z_v3ic-F)vi_k9h$qW9=9YLDg^zG6nJv#q9
zv4QcyU;1%s6v5+Sirw7CNIr4?44Ss~0K{AaH%NF5Kprxh*2b=RW?M)7L~t>KrLFC~
zSJD6Pg?WVd57gW+V>$u#n3wX``3ZD1Ikk3<=stvx!MT0Oce?R7^yucDqnjlI!0(Ka
z|53`*@DA+$1K>p_xy1e+2>j2(!-dJ+J1BL~qmjbblXQOn|LgddHP?v@e+!;y&OUrD
z{qva;2QcdxOf0|o{SSnS3|`o1t=r!80q#nylwX}P-w#w6
z)30B)x#=7$;Z^?6qy8~$1H_>a%&m6H(Z|a(^06g?@0rA_s~`L>A$klbGd|~tP6Q|}
zQ^7MvNgeZLwJds*30n0~HeY@G^(Tl&A=5Z<8^CMS-g
zRvsx8vWHG+{YuL>oCjMru1@sYGL5%5KTAdLDk*aX?U#F?*9~O&oteUawB;2YBleVi
z#Q>UT{$Q~F*x=#Nu}d_MBt?7=muiRte;qHB@P$8Ou;Pwcyz6=Xg4jz)x!b&uMOkBp
zg93^b@8y*buH1gHsR4MNnusCnG8}b_j7?4fLXLqj@HNY31eBC{bkHUXI=vP{R-mr4vg3k>3b09%8~?`eUIkk=
z#ur(ulU@Gh<7a=6)9`neE|D&*QM-BilJQ2;&0w~lb)rhloSbc*LLc2tI_@H>-bg<^
zK8g8E@?8EO&G=Ic?rP#Cj&b3Xjqr_;_I=mp+nuyZ%)fduu3QPo)A)g7)X=)Am~EN)
z%~7+@k^fE|Yw9_FXQL
^CKiXERX>+1c3}pju+-$`2fPI0zR~CY^YCQDfcG;*8^u
z35(!rACUjAa%gZv-TMJ(^hmjc(DFVy-_f-jPrVxbCN~=O^I2e(01BC=T}boFZ
z@5tICmT;;vAJSCVF>^?FsEfryYoeR(tf;HKA3hqy7>?&4c!c>Q`0u}X?=S<9kBY$S
zS{C_BFVJ(v2Gb)lBvU)>y0q!JKpPL!{wx0+$Q5v_exPbePE#oRCKobnJWb_Ow8IZc
zO+D-Oee_y)^0nqhZoP7wSdjO-#g3MLuI#_}Op6{P`V-
zDSeAbbh{QnjDW!n44))}K3Zm~$t#A>hXZNX6cAgd8z9^7UG@O;xcD3VB{HYRHve)-8K?KJ5gyR_oOx5!xMbXC&eH+QYyzMxZmQ8W?z
z@qAtOhMlUF{PR<+4*}3${lf#9A1FZPFq*+ir_55DA$2OI7NPs)4H2UQ%Azbh*$*ue
zLG1pT9tiV*vS|4~_{Beo8sHAj61y%BUIK}Pcc2U>R)2g!W7L-kBJ8eisESU%
znnIl3BRd0hNU<`=u{Ac?q)(HV1%Ej{uD)wo^|`cEz+-F3!sRWv6`+il{gHKTJ4A|I
z`wC^V!wm1+%g-{1yQ_kUVTmEn*46*7+%c~oGfA4~J2GM1`JjHd@s;5y`!J4HKA9^+BVxqcf>u1(?!WS*;JmmLU_-w2gPXj#-8=_i)}t
zxR9exmzJ$MgSrWIP|#%wa=D#&L|FI)F@8w(?vAOqpBs5zKc;hL;u4%mm3;m9xBg{L
zWbX%Ye_%(kN5kxJgNk&&8JGXL$!5gB0_Q>35Xmuxc&Dp-7BaO{W5Xh`)ocf!cAs?1
zDsY2#o{9)=bxqKoWAx{o0u1F$8V<(OaW&^xF-OK`TpBMvsCexz(0F7Z!YrA)(fV4E
zC4r3s;VFjH9|M5DsaRl4L3O>VNYyB>_boo$HTc04?l|3Y^Cf|DZAl
zXmmfCzBH6rkQRMILjRhZUUS(DzREhj_Ss{_J4h8LLdawCEkKC)zPQV?BduVT9+rTo
z8A~uUi!)+9a9a<)rS9N<(i>D8QTBVlfy%jZoxrFP|
zi}&6#+TRWMatl8>5F~^+;(uQRe_rQG_z1ZPz6YmmC&*)*QepiRB~QT(sXNa+QahzD
z?vhKXy`VFRng||B{7m+xyfo;?PX3GzQV{;2SiX`vH$6d{i;%P({5GAYJk0=5Z06G*
zE5L%PA`<9#WLOINC>@arIG)*{@Z07Mh_5LX6_NqF7hH=IC80lQRwu=nj`esmAL><-z=Et
zr6(uv&a*TXz973+W`;>6>c%2?W*T@8RO5cXJ@|*FXIrknx_^CF0nXdMxSc+CE(=3r
z9n_B9
zA1NR?vh%8%80jysu>Puxn&YsBJVKTpfiVT;kKJX~I+9Xn&iz4I|1lCtf2iz`-2}xt
z4a@fZ8$7+TOf?I)zbE$dG^o<;e^u7DYI^|?TL=}He3jGr$JGhH8NAQv2+!r-zymjr
zwKVN2fb~17gqSD!B67wH85A$e3Y@_r|6N3?DXXI$X!r8~&eCkwRo
zjVxK=QRz7_HRQcfW+Rj_mkfnT+fUa7NO2P#AZa?&-&FM-DQ@(6ps3jrt$?kQBo9lL
zXDPsRFx_4Fpb?T2SFsS#TfJ#h!1(C?%r}bZH~oYhRLxg&r*GcV_8^K7(y~$vB0Vt!
zJmUz3$mR9n>v+N>z&$r4N$GBNe8yO$Rt1hJOs;00kR)GM>fPo&3Xd{ohyUr^81{>IBrwVAzJ_uX3v-e_(;#_3_M>y(-@GV=ykeX0}DKRGc8yv
zi`GLXQZm1^T;1S;cGeZC@hyvN_4g6~0AK+nW)MTxR_Eb@wLJA2OBR99h+$d2raK3
zmuIFsiNDvrgOQKLzK5RMdWG8O!#E}0_X->>d=w(Q^kj)=@1~|+d5U+AI*>|Gh06y9
z)LD*J35G(2IDP}a{={M?zMs}DG6}%P-2q$dJNtAJ=Tc>=5OJ*29SNx>FSc{RoBM1G
za^s5NuGBwutQC;s%s>NQ))n0_qO3tf_zbdV&?_z;TrK{?^^fie^o?zJCKhOmhpEpRVtAnICC=L=$W{P9`
zZMgmMoS$7HXs5+zjuyvitGSGii@gdXj^}(}mo*KDy-|Iju@=x*6+Q$8r(41?tbUT3nZLX~s15nU9Hw@^nhCc`(a_+hXhzD~85@{f=DTNgF?H-KrcGrvab
zs^4I8lE3OCQ`6p!x!q37E8J^YB6gT`P4(GcZse+}KFsnKJnAA++!8#v$~;(;Q%~P2
zIzQ!KG|UP+_;>y#`tvn5btcP>$NK$)CM}B9ZMz)l{Yoe0=%3ImRV{Q~6fPrZ^)AfG
z*?Fmi5gG={dF^i(GGJjVt(8Q
z&*AeSF_fReiJHSgQe_NNJLM8@D}0+1hi`M_7xc>gAmF(*3qE$LeJQH5nFT*6w9=QbeR-QWav(UGM?rdR6n+LR2UbkVl#l3a>gW<6jz#P8y5253c~|C8
zF`blo^I5s4YJaNG#)4{nLm!V_hib*GL`DTBwWUe|RUjKPAWKwM3Q8(+F5u0ov&5jb
zffGNVTR>2-Z~Rh`=nz7|Pu=Q*OX(W~!+WZY6nBwJy|87i-c#Z#Yu*h`1gXB-6T2zX
z=I1srAXt2%yTn4J63pF92SpS)V*h2kgaeo@Pz`xz`ypWX1HG`_=72zV<@meLlpW}xm1>pq8P%>g5Ih0R5h*fJ
z4ZYLVsd>nEHmJlm0EKSmUW&;XiPG6s;b@*%)>+FvqGGt&s->kKPj+c;2ob6F-Gl%5h6
zGUpG9Wx20Um2Z#~!o0a!iHO(Whu~`~2g0B=F`MQx`6>AE^0_1)0ut(|YYO%1aqrJq
zY5>aZ)LuZkbctaFKa={N-uq;NIm{(1PJV}8sE^krFka!5a&~IRF3rA8jg1A91HR3+!Zez
z=yB4smaxtTx!E^JbY2+^6hc5uvj8Y9_RV0DOk{BqgxBxr;6akGy|HS8#qQ(7$;W*&
zxcnV!gWNP+{yCiRd)FMCNcG1?K||g
zXrfQ&6Jd3a+$<4~1l^biOe0b2-I0-Z??y6Ly~)u0owC&NFVg`G)SEMr`iOyn@~;;-
z?T!1Ya@mz?%7u5!Ij~`w(MMsW(MQy-2h{e41D?Kebs8AOWb)7rL*l)+qZYE{zL)2;Wz0k
zJqgOK+Hp;x`(>fSb@?hℑFvQf;Q(s3;@s8yaYeC}EZV!r5laD3@=ZZN+;BQ}4TZ
z9vFLeAr!+Dyha8vwt`8I6E>^0x2*kP)KjQC7k8fTSNb-+?LGPLRsC9NN-SaAi1%>u
zoEq)HH>U3Wi&@C$o)ZY7n|*}Py7mFB#p$fxLCMUx&fMYmU*%8dFvxp|X(Ur+slTA^
zMa8vi8Nti@o)iqn(ex`4+n0L3n+-D<8{KH7#+Y!*lQVbm1~ixNE>mNIwCf%W`NjJ>
zsx7+IFEr7mD!ABAqxNGQEno8AGUDzLid3)3AK)~SI=J~BdzYNqA42%we?ZfSy%{x}
zVi%d3yw(tk)NAhN#|dV97!RH!Hg-A&X>Zc2H5Pq>}zYl013P}&6
zwwi+OlCa6?3^*awK!5D}hIv$oem~`r6NRfX498kQWm>_(%)a;Gs&5TZ%hN@R=YB7jGOEJufK}4PB${4H5O5O7jqSoUxvGAHPWNNOwesY!)N}s)BG3KT@8sll0vnr2zL$i=g$oTL%a(sV^uK+<8
zxpRBkg?cGHAn9Yii;qMoL)dw=B4V~)4X2cAgTj6;WEmML$C>K|Fomk|Vpm3lmqEql
z{pyu+N&rW@I+rRP@)X1;BOWJ9`f%7G3I8g|Gz)u#oZ5NhAluJ53+haQ0pg1e14{Q8
zL7!tl5vWwrXV2M1l5xSTcP-l-ClEp6j$MfT?g;Vp^X;
zyAjhCC@O~fXI`eI{M$E518pEa!yG5-WF8~w;~67ls~-c#Z*fP9*#_f2n4_0=52eT@tf=Sb?gy=D`$@
zG46b0VN@wPU}!_lowx~-$qwzjh$q0O5jxH!DQ-`@_xCq|1XK}&>OdMWwf79@&%1c`
zx}mORys&K}JgTt#;I$bwarWj!7@?FB<q|Ji-2Xf_7`^h+RX_i_!mR07xPtn
zhR|tKcZspEfoXLYmEou0%Q(4aqUc>WMi*_Y(;(|&R_H~;&TUQNJH_jLFH*tSw~GS@
z!zJdtD#2|XUPyf$Guy_6Hm~GbsX`S@zT)pB2klgpwVswv;8OkTSKW0k07M6afwleg
z*lD&WA&j0`-6>ACjTKxxrOS=UB9MsAvKCBs*_^s<>(Uo?gWSTtEx4qecmT>Jxm7DR
z+9z-z+OoQI)f5hYt?Bb~LF4*uZ51hG0sXXDF1%~Aj@h1ZJriM32mS2
zKJAT-AnjQkc0A*&$i52{&WLq=GaC@aZwYD^uNga2i_V1KlREMD&j6Zoo)DC$I<1F&
zn{SVOCwxHPpUr-5^pWU^qb5xLVvW_9uI&=$dB2ct@6$-14efY$qjwdKqwFqy^H>G5
zn#6(HSKFyoZ=1_T$<~r)+rcgztg&c-xO9?v!s%6^U=yDFH
zM{NH~%3UU6Pk;aGXOn6eqkVawD^e3S!^r|ljrF(d+>aMpkLS2uS==s6FD{(@5=C8-
zAn(~8f4v}xjJk8P=dX$fsP>y0CZII0SxS9idh&6%QomcmYy~&xPTH;pPyX=au4bs3
zf@64k@6eMMXgM>(?TOdhUe(%*sQJj7-s5umI`Kd_Pdy{LVXULajc8=b*U4Tn-FaN;
znLMCXF5FyiFR`?u-{yq~0_9lj34TYrlMx!q&#SXp{(3b-2y?Y2Ko7)mP(AAj`U`{)
z9(GIgU$EbGX78xY!vEm$CcshDRtD5E7p%UkgQ}wLT)GZ)27>5qQfhVBVv1Rms=wwJh
z6itkTm+LtxUvGDi_2`-Q7`#bURMS+9thP#g-I;IoN`2#bzq%Z|rgt(B?y>~ZfQ|CL
zRK%e#$Jo5B0&?+2qD~PoD|FI!wK+sNJWVz_`r%^Y~xp(pddMKx;LkZ|&jkWn~xsdm#
zviuRj$39X|Ki)%G&0`lurj9-)_V=+}>N-tw@IKXP+tWC^?&J7~C2UM&33c~{AL7Uh
zmwh6}qcSW|s$$I3dSRr(UT&+o{O`AvKQxm9bd`!X-b
z^nTKZqoFQ9)OE!028sDSIJ5NkU)*Iz?9AQ;0uCFDWNs0OO=sF9+2cFg%25}6rpk=y
zg)P%|VrqoZ$atkhxig#z3_+8xLNb3|SkiPjoXF^FEQI5*Fo
z?^8e9%|9lDmqgImz0sLm{0@3(Vlg#Xwv0Wf2A4NS^yL6kLJ@Y44M#-hozO~)2o
zn>w%w5v_wnDY8zvbeol)Tv#QHBouP2AE=FxhkK^gZtlOr#PuiL_n;nE19ruW#$+J&
zo^-5#|7FZwW+Z^OYjm6tH27SW3?a+oZQ0n55|pC1ur8NaWJMYA?#?(<&{XtPb@H#I
zh041fQsr5F-X8U%=aBdUAG$_$;C^^#WHEZ0K_x+i_ac~D>`3SGzi#k0isR?`(sy!R
z0ynOCz9?JJxGWEp@?X%dFGf;K@f}&@zAJGP9K-9f^@0w1yc^8wZ&|
zlC?+5?1Kx!c7`#0GGTRB{@2GdsyOkC9mu;%)g-G;IW*F=INpEpC)eF`a
zO^48g8)5;a%A;9a?Yd%F
zr-{js3|TTO*2l6Dq%28t$v^^Hj_t^BaH~`pMHI*YIEA&yOt|!g3LROzi}^QmN?#X>
zNsXfmeLJTpXdl-J_Y~7Fdt4^4sP91=g|}l)(5}6VD3C(wUzgZZ@&fs5e$DW|JqP?O
zfRQDfpC`vdqa)dhv2R{;WF|Ef@8Up9NDc_$>U3O4Sn(c@&pfCWrv!DlnhSl3FM;p$
zS2YDi(tv2K)}U2x{-;mDl+&k=as+)%MSbYP$A~mBM$q=+mI$C#QicdEnHMP6slr6(
z&IL;GlK+p-VBsX;UYdJH7@IsM^Tl)%Cg%%BWc=_CArsJGt`k{&Fzca_%Tr`nMb(ik
zC3T#sz%3p)p?}@8fF#x;lg7Xs51O9sq`O3ivE73eeV?oL|44K-BORPs$CD%*c)4B;>`F?92Yp&m_(PIT72T+o1b?
z@S{A#-_MR_!I4lDlfHl9He6I2t8zCh&&-9u$kf?!qqw3kvZCnV+3bTQpf~Wk3EV(j%B}mLRAjvp{V>#mtMK3k$_MHx6&|h4tr8c*ioZoVXjHr8^uC~s
z?QplY$#A*z`lk+_%VhnJjZ!(&C2qOF$jtjlvTt6yR@2W=N9-Lr#7yK7Sn6rwgUd7U
z0x<)`{*dd4?KIo}5CUilf?}pU+Dfu7xpW4T$C1lM2-Wh?eFh<9IJ5Rr)mu&7$6b8&
z*-4hE|KBGFCYnZuA?xS#PlAbdsi%R7cXfh}<;1j0BqMx`gHdWe@GMWp|M-5%H_rtm
z?5cNUwqSB4=Q-YeOya!^iI_|Y03X}gXFQU|I+eaP$-qx-`G1@nZEGw~pSP8GJqNwL
zj`L-R=9{M^APfNw*m;nSQpUM210T2P|M%bLAe^Qu`*UQiwT-^s#@}&=gua))a_%>O
zo-^#4YW`)mEqovZqucM`oyDPDCeMY``!Bi6ucGV~5un!j6X;7;hzH}`ITO4$zQA2|
zat&RF%qH=y5QZ%RvWqn3pw~e?o`Nuu+X0w(A;M$sX{95ll|?QA8;1(+%EIZ5qt70~
zv{bE>j|3KH`fLrcFN~l6?M3BD{@km%juKDUCG?|XklF065O883j)8awdouM_%-=BN
z)Q|5YfP>I)w+;3Qw2AtGrsfJ*GqnSwDm^Bs**m&4U&II^T_V$q~v~s#j?s
z9G0rLeq0PDSn?TSe}Uu+$U@IiYSoou=J0X88O|Ty23TKni-!I-r3Mr1mRYi7ta;6|
zo4?F~8FG9#_Q9NR{TID%Y_-7p4#>n^>bS^Pi6_!
zFC7Lu8!!??zvz;M1elZ4Y+63$F#U0lZQcz4M=s_x252tsJL_S735#jqYyA0df9j{M
ze3GR&dsawqemzmbMZ#R8hBTxvv*pPCwO`@J6y*TG273xRb9hR&o*X;<>79yX>|Y*3
z++EpMYUElhPFV2j2^BHTqqqLo@8azX{A(?wG=Nm+OSgjqRQ-{)<(1m~IY3}NUNP!w
zkX^{n0p?U32Za>mlxMlfDe10r%?bAFvH(o(xI3M1*4Z6`g
zm+eMf22KFND{^yI5Ku7=9R`ftf`+?bdLcFefLLp=L27{u4-me8wQ9
zm^R%%DG3c-u>r3+3GM_+BGThmM>57<%laUfhQ}7qLfn6mlmwsv1jtBJxfvc@1
zZQK=HNytkf7f*p=I3uKk)_@0au08`Kz-J8=30i>Os+@AutViR+eE5ZLOGwIf5(ZqqmVyiH^*d>;ZO+hk$W>_V;nPO1H`ZI6
zB&YXvi;4ZWM6{mVIYj%x_O`omE2UqsflA$H0lGk=+y;OeMAK&64mUHUI&AD4!w+bk
zJfpev)J1Ry7)qfxG3P&r@m}Xv)xi7KlOPeu&FI$YK;repsKarJ7m&2HgkCAB(%Goj
zMHGV1wELz-f}`xvPW|DEy-c=_|1tNfhEv}c!Mo@xnJS0YVP6vitM5Tbdtw!S8rtF6~QGDJxp)4ucL4oZvKG`J1VTkv@nHnA${Jv%v7qzgLo=K8%o42%%Uy%$Ci68z(W
z^d$5kAS3aQST3xh1ZW#q^KRvLo^h{o*~^3>o2@Kiy~oW5>$hX4H@1NGil!&Fx&yhI
zI9PYE(WZF3c*AtgIpBbJkfF+*aw=@rrMfNM0UU|Wh2qbL
z+3yIUu5`EW0}^Mf6J}aok)(LJGiA#0QJ^O%?<%ks3&mus_~fKK-||%Fud|5U%Eb)c
z4`Eo6q!#{A{hq?UZr2=szm0957E1T$k_?5~k5&R_7u`7lH-dQw0n
z0BSQ>VrwvM6{F)n1t?0dLh|+Oo5us0oJ%DAm~i%f`v&&`z%6fNKx!Z4{%8y8_NwzX>ARp6$Gv=xR!%_5Hsxv&>X@83Sf6u*-CJ)5sn4
zGNOi+IS+S{epIdlw>q$57_TjGpFaVVuAWJdm1vVFunN1703u*J!vl3u)kemdor?XQ
zh7W4L5SmXuIju}CBo9Ng+3KqxN`W0u$E2i-e(Kp9-8mAd=VIIawLaL~@5uAwd&UY^
zQ1&^mKQ!vttnJoa0)g-p$ltl8DQ7+>p{_c>e%Q|iO{%sX&PGQTFq|b;pwqHJc@?75
zeUY|G`3IHdSVLW=pBB{hE8#dM{Y#ol=-ZhsslXA@Ik_h0zo}~JM{MX#*2SJDh0(F(
zX0(}K`>C*WZ3+8UA>FqY?Op(Ssw4)TLjt1!^C(q1Ozlt%(5fC*v|SiWn0T?nxX(82
zvJV^FXSkY;Lg<4o-l)*fwXenUAu@4H6JZRP6V5tMI<eI@V=Fa+J;>XWy>g;(Q
zk0D8Y;pbt0dd&=l`sR_XQI)(^s*(B87EOF@u*r9Q-Q06?5EzI$UDKFN@`_FGewg3k
zz2rK2&nKgR+^KqS^u-!1xFau(*V%a8ja4GV+!xvHv#6%$;ihNsf4qir_7aoX}_f=-TPR1R~Q$7afatH-d90R)A$uzyWqE`>@)&etQ
z=ev))MOESEKFm_W3#MZU+fyE?4tX(OAVnmGdq5}
zM9R;I46~Xy3}8_dQn+2@%kgV|T<;khEzqcIl6xIvUo`jb&Ez3v#zU>-r_*s~AB})m
zc%}oOgwb99I1xDIqVGnxCiHim^5>fPewrUuanL>7vHX0Gz8W!q
z0qQoEiQ@qvIMG3ubp|pFrt?vrAxJ6MDm;PsVc-=F6I+q^JDX5!JqqkP3gtFavf6KY
z01vXahTOds=gyRaThgh|5%A8&HIgxLc@@4GQ&GFNSj)5MDca|E
zqWpZ*S*#V3(Ccc&M`=r)%AXn5V&UR
zhMBSR1aaLn8-b^WwZpFVup9^sB}Q*_D&7iK$S$d~7_J{voS8?@0_>-;s!58vev!$o
zp=zz=f{O2Q1lq;@Xv}Igx}fys)E%D-BBLfjCpa!xr#Qo~r2Ypr*qyu8psJtu+PN$@
zo83JQY+brl*rql1=LyU|;OWDjyF+p&7Y=Hc^Ot$!0iV^*Zej%SZ9Z{T5e><4UFj7|
z?nd?^c7Tb$C>ZkDXRkod%y`f%8?kuh%Ps_0Gth1I}6^88@pY)S}O|s?TXj&cF||DDM-&
z5)Un%9~qD*A{h=LJomUTs_tNxCz>I?ysuh1hjnzQ2$*eV;4dj0@Xoy?%Toq!EX~P%
zpu1ULuyqRI-J>LIR^D^2Dwk8qqkl`}Q;r*S_uL|b#W0}e@q0V!#jfUZ6EKs+Cjq~(
zn74<@NCWI#Zf#*sq#tF+(Sb`x*QoG^0TA)cspIF^I;zL**SD8nn6E!V)lX?DU_n0T
zES|2f=a-r2%CRsiUG|pHYn|7#Rv(fDtSbHaw#UE4)*m-=G18U_vQEX^!}78^4~_H(}RxD9n>u?W$Jl#zbqtd-cZoy=)BT`1{re
zO)CRQmN^*ovtek-Vlhz#TP;8^E4Wv!F1=qQ_KAc0I(8&&@b;c@jCn2DUG#|_3q8)E
zCch!E!p+z5HN8te?*C|GPs<{)FMI-0`$Nb;l0<%=$O~uls6Y*@gZX+eo$)!Y!FP)q
zp95tYSMRlLqQ;49!}7In0_Xeon0tuyM2k==iL?YBajGPly
zF6I#^L0FI{wQ@Izvd>s#TyMNYpvXMTj4~ORnLW~lgGZTudztOUV(1$Fo=)r5$p#|o
zb_30R3%A(_AD!IF$P=m-h>Uo<3v~cyJ5W|0*dA|R=V5MC-NV|mC7X*o8+W)112xjh
zYFxC$6=LmtK4lkR2UB$Xxiuhh-nh!vzL|X=o$+E52{xPMXT6Fs8
zh|ptcuNj@4Pjq#vH9emZwE!y-REtP|z%1>zo}|2rQhrt=;Ip@RZyTiaZ;#jc@PqX&
z!VXcLX#k@f{F?ZC&Pop>t(I%1{Mep{8>q>8v7yb4Dskg6+goP18teLf;T1Yo;sWO&
z3vz1osX)p*y_g;mU`pKfJ}O&H>84pUk{11JUbS2+b*auEN4RmqfxZ0>z
zF(4&$8!wOQ!#xcxe<~c3r8#6WlM}JmJ&Oz@73l$|L)WR4VhPe4c-If!F)L-!nV@bE
z2`syjSl9fggvo&JXBq?M66b44bNUqsQYX>=a&1%bqP0MVwEPdzk_rH1u&em!%ze-m
z0e@HGCVGa6RTD|kfmzeg+vfhC=nck!rh706x!u2=p}g##N*P&W=W|gOq24w%=T=g#uPqr(5hM
zM`-f)qteaJB!!WT!>wg(Y3}ICmN(K_M=Neqx1#*Y=t;O(
z{bN;iNLzXQeQ%DfF}3=k5paO4%ww9~a*K>XvtK7ydSDsv8K2I($CSL=tDzEkq6DA@
zqOpBTUG|mQkgPuZ)z$jz>hjEi5TqpSeQ_Ei7mzavr4GRsi+bJH{1pvorQK;_mpr*F
zvx3j)UgNHaP7Fc&ZdXs@O(Cpc&U?9cqfST(yd}vwMw=o=Brc5kV8$SNgm<+WTnAI#
z+a+HIitkzP*F;DjDTPV2DSAq^jo8;G>TQD0=FB)j#A06B8`8U=j#U$(y<#A9x_w1y
zug*o*YIFNuz!!ls;h4zR8u_(As&xzu%WXmh<8Xf2nAK$tvGA<4q~9Ch>`kHqs5Td`
z^~1DMc_e^#`68{|f4j8bERLYJQfpX6rV`ATDU?
z#1iQgTH-hytRUYZ>LSt44~zS%2-dmk#CEM@2lTRp9>eaJSS5QWR7Mebq`^
z-y+c^PSCj;VY{_`FWD_&GmE>HDQD=~k&=)X4EuWg)$d~yfM_npsi7^-6h=#)y#ai#
z?SdLl!W5toT9kiBe(w`%&BH2*vO;|5MT~xlB}nwF@}$stqGTqd_P&KFuFO8Ofg`7{uaJtVwbcMs0SSTSTCU(|fo;EZA|PBzS-=jAf(-EaHP#rrVluE8qK
z!^b|PIaPwlufk^(uY)P#HFufTH%G)G@O_@e$9j&9-T*e1Ej}Y>Ub`hAH1zP@mQ6ov
z#o{Zp-i6J(BSxnRf~l(d{3sXa0d7uV_d}Qu7k?8t`)JDw_{n<8dn2}$vr(}f8mOH^
z`&e3Q0b>BQxM2O{dv(mmmcECkBtKtCyBd6keMcu9Apa5a6EL3e8s9hG8OI%tx9_w3
z2oJyvVaRFUDVAjZh-&IJPn3wAoKP^?Y&tJKz#E!K9{pe~8ze$6Cck03+
z=zn&U%T63Sbm~=;(>g&jqxAZ^Zc&N=W8^SlB36Q4(%XHgF?}3eOuLX}@N1Evr~vh<
zZ~SPhqALj@vUD;Yv;cR$iX0<|AoT}K(8M(m2zXeYo%wAHLNoBu(^>Cl`Dj2NClvAs
zyI11&N3MWB&z9oWztEW|G793Bx5C7%72|++Y>ow8=X6Z$(TjQA?!ZCdQ|u9Y&POSh
zor(}(@R)fMTibjcaAeT@Z8;K0N}Njt9HXpf*xBSvLx-`za-AiCB>Ck|IgyVst*k+AW2Xx=Ip%P3Fu%bNmQlNym|j9AiVLd=!XItSYAIz
zu304`5nU!I0z1oY=u#6Vx%bHO&6v{%ATN=_9+Aeip1~?UOB%5jp8(A$)Rqo=
z*?k2K;)*Zy(3%Qp!Yn~q9iuT|7=hR(>F6c8Ogzogu0PM(XH;;~0~pIWl}ajlsXrwM
zD_Od^@3K1^U9mfj8%6s@56QZ5EqT(g8*z5KCVkF`f9*AXaX($UcO
z(D04$OVXb}0NmQPe>U1}Zu{Qpx#YuT3x4yx*GooF?3H|B+u_po9B@r^lzMrl{q!RQ
zkAU7!+@<)Db^`RX_DwD8h3yo7=Y-9jPJyD{3j6UCF}~?^xBA5oPS^}4+ztX;Z~^c5
z?qAV8`7`Xn$OgQx5TA*J>wkUhDLBR<|C;4k&(D`
zuofb9Jko}rQ{S{RL3FaB{?ImK|0V3;LB^;Ned3`+JcxYgTIZK3B*ZxYIweMN#VYch
z^37|Hd{pS|YaH=Gi&}Y0!u%tHM_coLtCjQStL@(yqUj7BZjFwI;;n-pmJ}AiQ<;>-
zR!Y^y3KBq=+p%P)?vjz)k~{#zoeYYacIy=;s9O;
zexjM0;o`ehl57^00>$ZpuwS&67r4EpJN9Ox_~wmH=El~Y;L=!yJ}roZ&rKKgQ$((;ONQh?Sz%W2fs@e
z!dXn*-akC*=&er%(?h&FmLy)h-NwRC
z&3I?4qTZ-T4jEal2XNYoE-Og3HcI_M_2TGIi~a%KEuGb10NbcD2^-_UJ0xcD!=~MQ
zBMUJoC82=0p?)RJ;fH>;pU(^ckfcoG7@elj7?s$v>PO&9QEo{lcf0Ektp0zEoq0Ue
zYx~E?7AZ!jq$69qeJNYE8Evve_Kany=7gwJgHhJ%)Tz)z2#1t?m%Xu9D3c`_WG!oU
zV;fs}t}k^?ox}6{^;ceAGrr$@x$bLuU!OZpv>Cs^(<3<6G^#6tXjaD(w;O9lq(Sk8
z{CjReASB$*xPwC0P)p<9;sdSF>JiryORm+N}Wq7CNH~T|A|6Vt&yA=8rcnXy#HNQ
z2@5~|0LIfd9jiTGL~{m>a(4EOV^Gm%p%!5zx!Ka_3drdt>#|W$Pkq5xz9qqdpQUX?
zw_3+Dr!=xAnTj#x-?akIdXEZyT=LT(gfn4P7;B~g(5aJU_;Qz{?g!A-&vFeNbXG9zvlJ~}
zH1C`@iB~hw_D{+xiKMrOXcc+xOE1YjbnEVk?5%?7*=uvfIMX~8Q3xrT7v~@ulIb|9G@KS84ZU1K=@(KIiCiV9P!+J*9z|wl{BGOE>L0c9SAmAP&|2B3X5hMzeFbz;CHcHco+(
zR1H#u@DbCFPu$_!i8Jx6%8Wc=rZZN&a3Mn@$~AC>x_}j9Q7XW?
z6=5w-V?RSh@Q8EM3}-|J+DzmufjCwIls1&#guh
z1b9A0Sv#8-_WP)wmTcY?=rdXjT$`z;k~2#-h~dYL_XO&#YiEd2c9(Cz`o7E3SZUvh
z*_LaHVf
zr%PI4rY#$%4HxPn{y>7T3>PQjS)NcwI5(>K#J^$l9HX+gD_%l%t15}fG2e5;i;_WeiO
z;!<#?c;%tB)m0Hm|JR2CWK_U
z8ybkTXYa=D(WT)D|1Ux?l+++9M3~HE_DZsS1{e^pC>P$#sMaDs?nHQFU4+cG)#p{H
zj>>$wPgBl6tR}UQh=X}LeR>ZJcD$FQ&OrH_*3b=2DP~zOK=`KR`F8o0nDn3T0UU0w
zSD(f*1+K2+mc@ka
zevFwuXe`xDotVE%S&u5$L7>uij2<-o0g=&yYNe0`AJAU@9WIdh+ur@S0+81QxD>!=
zjxa%hXivy_^47-BcNMDPO+?qDf`k)b-379A-3w^y%y!qkM_0Caar1vn*9-uiPs0?b
zi;1(V|%DOxzoKBdQlZfwK$}0BEw9o<9
zY~y5K^#*9ZnOS>_wz47262b%^ZYS+vyRENh-ePet{j!#-(wi&+?)YK#9myUK^O?S{
zNbgz39-f$Nl}_i9w`c^T`UGj{sl971rc`FxrBG&1SxgcMsLr^_czdz(E2)B0Mzi=Y
z#%hQGw9WSFWQcM4q8pWee~A4^@)y5e7YWan%Q6F;Q5^EkLcE|Fvjlr
zBXB!6BS@q4<(}hXPgl^DRd#7gDaic1{QNHZC5YN~U2$#Tt;Cs}>*(%mi05Oy)unvP
zL0_K6mY>2Z0q>LIC5hGYFn=22^sptmQIyE=eo{i)-{<|&OW?hILasQxu-3^`M`V$$
zO`NL%(>8a~b4*VK&s!Tk-}y#Hdq%f@e}RSIEB&?j;Zu5<
zDH;MIdTu|<3)j7cEN^#+D*1PeQ=}K7%P(7MB2V^N5
zx(>fDzYIK}4agDwk4Yjh%X%h~-v0T~zfW0iEOAZ`jL4PPr1;6$fj0ZLTQM#{;tfg!1AMTa*@Z#9
zHo^Do5RVe1u0Q=pLts4+PI}NJQX;kHMpCjwn*Ba$~iqfgzaHx|_FNmMMZIT3+A{muAdo
zKmz|31bP)bYcl^QHY53mm!MofcL_2-%+-c_HT~Tdq$nj>DMEUD&6+pVirchPlMm`k
z8Pgr8(A;XBoL+7KZxPkaXeX_aHy|v=H9|U(ES5Y%yttqwn$sQ^>}gH9Mj_Y%cglgF
ziI5W5^@8{Or9?5Nr*P!2HUKRW8K8nTZcIOE#g$-yI$kD($H%s4ocemVw~Yq{
zH%y81U3n|>2`)h80IkzJD$>;Qk0WvUN9@AC`;s3(Uvj#*-m{b_KUPnpq>(ay^sMXS
zkm9x*7*W;n=*%=&dXjrh4xzl)t)G?CAn>6kqI9o^`1>eV32eMex!TI%P-G}B=ZBrY~#2*v3ejE9i&%r=?RM`Y0aO0pZbNk=_g&(+)dZkZ7J=NMsKHl6f3
zhZK4XK`%x6M`4YL-W*-ivM^=wrIv}!!Q3%_Fe!Ry=vlbc?UPVF+0%MHMbSK~IA*=!
zvcdbnNS<+>LHUK)J=n*wuAJf?p5KrkZmw1_0j(xyLJ`BU+L*}VKJsrLccuhNV9p=p
zSajvu$GMmgxz4?-x7asig#Bx?+TG0=qDXcRdyZZBcyx?G*>+g4%GbKE{;b{&JWVi6
zt+R@vK?`ObO_>55dEo>^sN&y8DdM!uunP-K@=)9wS|gC1ug=TjsKOQlYrsjRwA=f!
zB3ln7`$D0wf3(3H3K~aTlnQ)86nX~MS@vk@4Oyqqbyr1Myt}?C+QyJ8Cie?mJ3Q36
ztTs_$60p{v#4$>>vS(r5qH%LB6!zDz4sw2dRmO)Ne?{P9n~3rcQPWj%4!?nMc36}v
z)P`hpd*R2nA8!I-VRIVrlU
zcaV9Hij7A`w6yAbyd1o@A}LXO%F5vb(|!{PDz%?sdyo~~+OG;{);c>+b1lXMjcA3j
zEqg4#HNp_^{Uv;~4+46hi
z)Pr1~HyQU4(DSY+VNOJC#NDIGUy2T|%WF=lcRyvi5U^JtnbXq2t#wDM10^rBwJS;_t2=Oz|Mp%c;=$CD8E5Oj6g1J-pH*zDK_`J%EIbf6aHAl
z?S@bi-oj0lv(eR(U^V0AA2OiC`@ZsIdzP_wWt1DV_(XBip2@%lIfdm>Vlg+W(BwIc
zelc4vEy>;XPRVV`oNtk4?a%62Xyh4|nw*X944I!WNbM!HBJDz}8H8j#p@=Iw{;xtR
z2`a1uM08sMbX1$N&%YX)KjoCztzTdR2lPwfUy*ou_XDe*h8b&L=O=F0ZHz2=^gtDW
zAu50LOb?5Y$&;hfn{(WdNY!R;xu`T&^@-Hc-ii#2aTFc7!XjYue_t}8=o5xVpo*^U}>fO
z=%0qi`@#;cT6H&7N%yZZl5$H4&d{Zv`<~b(Qz&_a8XnN4Sra$xr4ETsZ%+1Eq87nI
z@5aH7G~;y*Z6r83%Bt&gV2@!|MD*$3CsQetD3^kR*X*4
z6+^K8tjyh~nXh*_?&)`=hKD_Ba}brfu?xa+=EWB7kfeCC0=vnGFP&Tr@md>lbGbp9R2TJ}_FI)Owea`!l1T%6aP*^=q1I=IHDg)Mj6lYj2E
zrv^WIYvDLGybU}-AujxTq*vZ%xmSM2w_f>G2g{b90o`Dv{M0p~kW4ogmCKb%&V8(F
zR5)w#0>2gNlJEaiDf@d;uey6{F@)!L4p|MRh*8at-F#CW
zJSVKk1+@VE;Ow>5Tyu`N$35<`-zv&W5#W*Gp-?CSwDf%?6zYsO3WeQx
z_6+<+pNcCH{&mJsR_Z?L1oW+UdfmeSQVJ}=glB55|`5tfLGpP$rOr7NaNbV{f1UQBsIP4trJlJxYBRKCFz$KqEi
z+gt9!)4_7X!%+`J6QbJPYi?k{D*nI!cqdt}X~IXXgSq>0CVNmQLt4&@C?YePyd>OD
zPcYd*SJZD6VpL?69J+{p>|BK#Vr=Q+cnGBEQ)LJBZaoVwym^0zz=an@bouSu@ApyE
z?0S!FeyV;QG50`M(QUC_jb0r-jY5fY?%(Oe_!#7+P&up(6HDq5pr{!xUB*Ju5RvK<
zj~%Io@ThcJqebp-Xu%^wFWl=?LU+-(hTF>!?|dVL+j~b!%biBO)X=dHo7m}=>C~qi
zW}mxMtqh-Lyq8ArK|Loh<%B~Wc7m_}zqj~ilP65<{c9!lMT!*NFfCI{AKt@<56h>c
z&AUH7Ag`;dV`68Qe^1V*wl&{_@7pL{Zj6IMd54CCB!}EoAh~d`z0|e6G*skq;Ak;i
zu4>+wt*WLLUeh{>j;5cxh8_(snT)R_Ly61LF<_(K_-~7AP>$TwMYqe6O`HGz{{8;e
z%Ba4G+jh4@pRQB~+*v%2{wPyDrx!yn8;Zds@tXJUY8$^!ISJWR7C6b8slw&9tf7+UDPW)<0&E$BS7GEo=zwj7OVnWH7wWguo4!fm
zqpRyDo$t>}#)O3MKKg>HmO$3TH{G6)bM7(&BO6<*spIPuH@a4)SKt5MB+*w9oik2R
zug{`SFq=s!%+s%ul9RU%H`*88^NlF#=unK3!Hq&$*tCngMRtBA4;Go5Pc?-#wYA-D
z&!(Nbjpn6e6M{`MUiI-i^n&=}J$gdY~2Zeoa;P
zd@syB$EED}Rc06W#EW*W57U;uySz%9bLNlXqEO?8Uu4R0g7z%mJw+#==gsw~xww=r
z-<0pJIo{$>%~X4Dg$p-gereY8X>NPSNAAf*`Z*u&`I@baRBdaT
zVqDRJ9}X+ZeX45x|1v5GrAs)}ZsOUrOcHylS$CHPi@JPp@VgW5oT>(kLZz=>pcGE`
zJlZ$y`uJd5Qy)E_v+>~Y)Pu#ptV!dsLEWL|B9?$or`xvouT)u5*xYyC=fTwdC10E&
zD10`+gHN1np6DWtWG`|rJd>wkx-4LeALx`M8Xdg$J6x&2&HcehI2_wUw|
zNaT3<_U+q;W@ew14aG{S`5bYA`a}M=32Q#Y>hwRDs0o8NpSyf;TpBbT$TvRy7?!Tn
z&BevlU1YAF#&BFB8=bt-da(51@b7iA4u2&!!DO26`hUln6j7={SN9VqR)ktgI@;huV{mYAw20entZQ&F
zP4b-=XW*w|+8Z}oLyYijr_TBcNB_0!+x!}jf0OR&#rdqC!YtqB;ISFMKkUBIN}&l?
zztl0Lx_mh~H1x{GMM=p*PE6{QJ3$$rBke`k9&QLI3Q%_|t|u;W!Qy
z@tlje#1eMf$Q3?0C0sN|6sha+-VFE#M*uC{<*rtdS&vGQnX22yu_I<0Ew~t@uJEs4
zcn7`rPmtN4yIZnS?(NXR=JhS~#5?K3@+(2d#dL>-UM6AT*yOXwTSdggWM<}6Ynhsw
zPX8#Nf1k^?&;G9!y>!>1__&ef?bl7Lqj7OGqzXro1&yA@f#7kj5f&l-e09~D{^rf0
zn>W#ZU3)Q~#vA_^4bum1bS9{(Voph+9j@
ziN@*%{a<$>Db?R=|6U-5-zc)H6543WaeCM^Uf@~(Eo?5hbc_v)5Wg$epms6DiskBG
z?>Xz3_Ls!Csc2-dB>JNUog{4G-;2gxDFqrMpfII{Qc6ipEh`-
z61A|(uyq((Hn}SwJ!LmdcKeW?&S?DjF(Vfj=Fk0igCV;{MqV{GHrD+3_lihJhh+El
zlq|oFd0q%QNW0<1=G_uT=T?wZJdJbt^5y@Si8naxl84aW*DPJD3MRfI|_TPNVqSv2RB-7vS^Ph*aVzq1Jn`)5Baoz;wc;X}FUUuqHhUc2#Q{s6HTYs5Yp@Lsnk?3VbKI`YjC|l;!JvFQn
zp4{-FW1@Z2$797Rr^LI_GT)PFr-Dk1s#572X4;%zS6-F1dY{whuxkBgg|*R+!^
zOSeKz$Z@d)?rcBZa;=%`=g*&JBb8-_Mn=4b-_H))cRjEz(;@Bgj1+YE1lQmG9Rq9~Ba^r6^tQ
zL)<@aHVjKNJKSB{-^%gIk_lthy>gHYSy0e^R`K_4p=lS7SvO-fBdSRUwuV_f`&kqq
z&M*4x8*2?z=Bs00|MMoDa8gU7HKSZ?A|kqw6VICoH3VE&juUhUjfgObI&QoXI6Ie?
zaADtr;JToS-E>QyoZC`SpZ%!Ej!lQRzZmEH%=XGtYC|}_zQt$P}p|$Pv)DeNB
z!!PKuF9%^Iy7w2FhGbVP3%;ii;F=u|r1RJrb}C=UuDC)@4#9HY@4D9AU&Curz6B=j
z`9|JE6QTG1vF*?-y((81Iqf_{iH;%7!l%JfYf=~`Wo7@(>e!KcfEffHi9h{bC+#A7
zN?O!2VzLVLSpVzsBN~*rK%v`?<<|1>P%^qy`~Llx(Y)p#<>Lf1TsEfh@HZ&Wz=5QR
zdh9nQ6FwFQU87s0r)6JBYB*}spbgWs_Ry-;t9xa&jRqgV_`ni
z7Pk-k!P*sl+u~Dyzs7~DL|Q(d23`^yOkjyUI*U?twY9V3bzXfGNSE)rHSfCq<7Sc5
zie8+6o#d$3NtN1XjSs^WPAuHq%H({O-BMH@DVD?KF3&bvqPX{4tvr)Y;aoSgbsL;gM1mvF};SCmB
zX#Xx;8Y)?s{h9m;fT8$Mf4;GQgWby(I2_(*{kh(wmvN|3>&K;(Ikbn*TBeN2;73Li
zJ@++Z?*iZ405Z+mmy{=fU&zi
z$a4IB-A;+mE%anYq@utRkc|WYgHu>tbxuth!rD>oaj@B?AjZVNAeon!XZADc
z)nK7%GS;axjKaddR*F53_e`4Y-p+StBtzUG4xkctR`e(3=!J#Y!`2)vJJnPqVn=cS
zQtSHm3V-WpXLP}r%2N?a%%1dzH-@ci3FbRf{Tt5tXg+!JQ9hQR2E9KOp)Wj5C18uh
zw#|J3EL3{BKqQ}3AZ1Te7)ueH7#xsth0}^!s!W(lg`)+p#h^-++fHv$pB}k&_BOIw
zKE4v`N`TjX@7@)DA8!{ukc0AYu*$6jUhxC?Dm@+DECim7@?mb1wpp)}gM`}_gDFP`
zJI%Mw_vGm7TMQQdOuX{~($AKy%Xxpd6KrwD49$X%;}a8XeD|`oOXLRz20rPO<+nRT
zuxURj+wOo8q{wPi4{^K2fdc%rw2vR>fBRBxAL#iF9%-9?w;dNyWMD6qzs$g-qePEZ
zrBe;F&q#^Ju!nRj?j8LgzJMB|gPf6jQgfBhQs>J;_3Af#$$}@X!o>zu;JuH}Ul;6_
ze(->IeSN*s((c>KbCe6!J0os)yt1RXOytmix5#fld-~GXm-K>F)mHe#0;_)YuY8^6a5nrIi~iiNrE?!+&POy)
zZEZlNX^iAFdfnH)*6wxOwIa$%Xx{tzcfwgB?8eN8lp7_O7VY!3x8xH{vsbt9>BHX(B3s?cl2GMURm($+e{WU2SAQdCZ$baW
z?7ozgn=S`4bCdJz2j6$Gv9Xtg7`An$Lli`rLXPl>E?+dO52RXItJ0};
z=dPrr)RDl##PlQoJ;XK_k5l4UupBl=M5Sl@?Lgl&pwRGH%#pgKo+_`i08Omvu
zmVFs0`{iE(6XDvHHoxK_g(*;`J&2j!6>N^+C}HhOlEAOMb>0Y0no@#6zjo~Fi}zh=
z3U1F5u!&IYjvp#n-d7se`4LM3H}Yb{6N*81y~TCMud^K=?opP#ETLgwsPC_Gb28BD
zf;!uKbabmv-|Df6NkpSfo>z9D*m0@cHix*}{dDw$2V1*(qC3CrJp0UtinEO4z{E(}
z%)36mHy(i?GH<61)-M^xtjfA@IImvA+@&EejN8P(PSR~WPtt9gbczY}ia3wyqV88L
zMo!!d6dh2Mga!xy=C6Sa_X!SLhkeFNVx@8;!1Lr_RH)70-+$xjg~~@?v5lWRIdgdI
zxi>}HnjkjHWiik9R3MRO(zun(jNCR$X4HqakTy6o|
z$v2i&TWZ(&t-l!QAwda1P|k2|woaK6M1vCha}(`=zq->Ds2XPI1%EU)c36}Qf4m_T
z@D#A!K2RU0JhO;d)MkH5QL%LyX$&5a!coKtIThQBOdor#G$Y}2*lCnF!1vXwx!2^D
zj1((FB{{|}UhPB%qKX101ru
z_D$Imk7z0lCe*R<@%250Bw?hy%xP4A705fbpA+tYT_>7YayFfhdb702fivL-G-Lt+
zf$XMy?5*BxU31}$7H&xumC%OuoSNhEiR~q)z5bZ4=58Eqz=rRgMqC58p5A7sdnS4hv)_$2&D_oGTDy
z>%no~?~HnlKK)gi*RVb6Jn`Op{%kzvjqu9eL3OVap>o&7Z^&Wj(?A(w`)7()Q^D8{&K
zwDPev>jUNu--FD!I!iW8nifwN&aSKVq>uW|jQBuZolp5YW&A{G;yiqQH>`9LfI45#
z7NZIs7SsZ#PBs!wc0J+%HMt1eO|N*JR7~VUQY+Hwj(Ggbbs-DO>*Mj!fp+iGNR?ZZ
z2T8%n;f$DQ3@0B%)ZHa3FZ0Ti1_zGf6_!pI(9u73F)dQZs6TIRUSC<0;6|ttbW#=;G>VJ#6Z3
z8dgwu!97I=nHyG`09lO7z|_cqyAUmb@Q*fv8X`$dao>bH=Q^4q_SvSE>&z^QaQU@x)W}ys!*u@BW17JhwLd3r@ao
z3KEH0b3PJwjE@0Zij>(DaT_8KXM!{H7`ji>fgI9>UBALmA
zn)(48DBA?zh;#CC)s9}x0;KY%Es>nzrT`orb&BDH%OM-Fb|Ks=*k4VK9pDf)t-?mG
zB`7_Z3)&4GSE_dA+k{p&1-(v=kD{T}^E_J6&wQ9HiNKrJgjCt;`W~vIo(EZwZ2jz8
zz+O@yQT^az6BX5?^4JW95^B+*3XFQbM_qsGsidVP^UC`Q^CIx9r@%-`uJcUCxmc*lM@=j1J)Z&a&Ubux(Y~afsrEoY
zcNTnHtR_B)8VIp4FEmaLsm7zHX}Czg_BMWYEA&!y_`6#Ia;y;
z{-sDxyVBVvilR1uum)wFXE$%zs?BFzR#ryYcYaZ~T0%l10~|lxw7UFxK~1=FiG8XI
zfsgYA71k@>z%c`LR`6~V;tW=fTPts$MtupSGKYbSHzYOcxw*N+FqrWjUU1fAsLr%7
zSFKj^3Pt?~+*Yd6fJNGM9|;%&U(C72{bUxXWkm=Ns>zZB1Y|tYkhMR;9$r~>^Ele-
zV>`Hy(Q}=n&v_d*0C~Y4ex?1<1405;l_iw?TYv)&ShU;J6eTB?&JEIIc=aN#PbA1zZ=J|CE`XFDI*@K)8D(viAM5r+)L?xZbm8
zpD8_e4QgEO!fM#Me10VRDQhn2qe}aA?OJ(1sbxhQ53Wx&Uyp#&C@X+UOpjj=mQ2sU
z;L1umYv)~THJS}&#DG-`bL*b~ZTR*PDT+}60(X|!(VW7o2>{@%>&5OxPv=H!Y7h(z
ziQ$TT46k`OR9WM8F#s+L-;FvBS$qOoG0KyKJEA!?E#6fIHmZg}J~kb!_#R-8^G}EM
zJ@+IavvpOvY)smg@jwefc&hv
zN9=SKPEb`)SA~<+D_>tX*1jBlF%gJJuO?)}S$_@`n218F>${R*#K6g^nvHa0PFdvG4{F^=lpZ(Oa~J65TCny~;RrQsJz
z@h;QX*z~J)GBPq`9z5s)z!f(7D&6X}!5`EPIE%-1^XY?Vo*$g1BJR7nj?2GHqi$P|
zeMuZyNPn2zoh0G^ZB2Zc66tas-sa?BAHkUGuVg`#K+4MBg^;R0K@dKCehPJ05O-m)
zC{uKQHu>8cmDk}zkNwS`ki9-bO}VR6BKe-|6NHG4l`O;+1>Wh($-QgfvGW}FuOg>B
zc$L?wL;}hGu0)Pr6)R36L`Z}tpybjF}52p1W0uu%35!uPe%69#(_1*e$Ghvp)YtO&jap@D#jtL7hQ&V$c>Uuwy
zN=BERaXb3-y_Pr;it0{P1LT1D;vq{Gip2-N<8*B5h*P((Lp;#7e$LCory
zxU+y$DBFLpIe3~Um~c>{<8@oNW`Eu@FsQQ=C`td5JnP20u^WvK&Y_IKkCe3lwHH4d
zNCF4bc&(f!PwY|wD7hT)a-nf^_%~B1j5?t{qlhSRI=O8-*=W0=iLDByAcULrRc@f6
z)mYTCRb*uRz(5LZCL|lSkYpSm(ybZ!)HHKZy`lH^QcHfvuaA8S%2;a
z<6k2(cqnhhmao~R<9I;9n?t1)J{J`g6w-qV*I4rCt)(G#
zcr`W(1+Z%TG%BY
zd6y#dzIos&=+^~rNWSaPKVFacwg!|I)6JU=H#i4MtP<3+bu{@YNk~YnvC<1gWgn`1
zQb7-wJ1E1#Hx~=q%^=#iPDhz+e#NEIxTKConxM;&uw`qir+u}{_srLIDX`~iO89kL
z=TargD?ULr9d4tA!a_BET|WYzF|2v2Q~zXNy<@{nP{O4`?ppvi
zSAjag$F&L^mvpb{OlOari!y?e@4jlDQfiO7xMcc|79d`Pt=+At1eL$TIG%OD$G5Zb
z{MXC~6rLSWS>d&3wDwW|0){@Up7$)&(w@g6d)m~W`EO&R>gK=?R3OA}&4`_70>Qr?
z^ZDhgSEgfMo+E^8N6uEv=wZF>bFmvY?g2Nkt+D)qP&Ak4WC=o*_&I4)&+u_;IFtFR
zosq8bI)4#LQ5OI`sVONMAAzgas83E#7U|7w^cy6t7Kvh0kPw3ws-Tt2WUy4B<**<<
zJ^fx4quqQrlLu7U`O%`z(M`#Pw@KCE(p?ew(C*%~aV4t@@>-1;MQn?OHV;8b5o(?q_X5FPZLtD^5OfhPddRY(H5n&s7$OfZU~MEHP~6()K!pkd
z9KCrnZ8b^^h~hY?FVy*SpluAsTQF)dm~iT`rNm<^gUOsN1=9xGaRNoo`?>h2-2#iD
z+rX!caDmjo0sL*X!txg3v5@Cssbm1f+*)E4faHbagB5jv46-S1c?e2b&fnXe2;53I
zIZVrq%mbb%OS>f7+SA^?Fg0N`|CkjBxSq|qE?_as=b5wgYmR+N+#0Ss&F<8kR2dHf
z5`B8jY9K$=_z=pErluw(6_sv@wP6l3&Lv>1a^ESVuu-8V#$hkYj3%L9#RDCzYp*}F
z#_*ZXb*5?oB^}PHIUXr{d@y>V2FODy9vgZ66kwYWf(kf)w%B6BIy}gtJ;?_p%fl78
z-3~6J)gD{lFDqOL@PaNR*ZF%;SX0hyL6^n&TJu{z@jXd=Y7~9b+y@^{PR@X4QK-fS
zYrJZV2mAAkw)SQc%%QmV3?V+?HpdPN4NU
zvhHkMp`*1xuWqROKfwmh6|{>gCy06ohsWKvAj1uSV3(Q=<{56|#AmUM9aLo;$OjGZ
z@azUc$mmuzM{0;p1lkV9n)*12>zDq#)IBV)X?c&bCnhGih1tqjjoRY|H5I8439lQlYUCzShzrmztJF+MLHnTK|XR
zEO;kG?CrQRa=qKUC7M?j5CIc~WzEsH?Z$9BFs1;6eDq+QBH^ZeI(7sn(kRrFn&#A_
zDx^5~h=Eh=2vsB%Uus1&*<$
z3LSQlNVdq@q
zLGZS#s|9utlI9;i0%Yw~wtrj$Tql;(DU++O<2|-Dn+t))5h@M^LltJss!WB(!yus1
z-?>kJCc!2H5Eix+%+M}j#|=UZ6zaQuGla+*v}+eJxkgxcz-QH*qmh@CYc9YCX4zSJ
z=1wIRGbO;(Lxj`?RA3Rc-K%Xi2P}hVUvo=Kw@HG>lPEPn7`h%?J(SIA8A=kZ(Yyt6
zL%*?vWc#!kusE)pKc(2ZAnboc$bZ(D*x1j&rE6l^p-n@R*asEWyVNo#h=asbRQlP~
zCrG;p@cy-qd?b!H49h~&Rrr#l8ahLOg|eoRVX-m=gMu`y#a|7*FX)^Jj_vHV@{MZP
zzJ|n{0VtN(7PyFsz(291R1shrqsX>l3oM&$;<@9=bZ@S~JzbZnuvZg+IKuVlztz@y
zY%@QM2BgL(UI*?gF24Aq9=fQTkZzoHMQ`4eLZi{p42m&!RZ_yx($Z4&<>~Kw9W0fI
z^p*%ABwNjqV{5#~kda<4A!i|}(}?Q=+Y;!dtSc1~M4@|eM;x&ZE@*KA2i?xa#>Vyl
z@M%>w^gF$GzLwd|Y(g;01e>NrFG8Nqlw~jbZWITxb=d3VDBKhn{?GC5>*$0Wq|F;h
zq6QpjNb?l4)948DNyV+H6_
zdSPK5gsQ0q~$!NdEN2;REiA-Mk$B$E__-vl9
z_L$AiXXdEc+uN6w{%cOywvhB(bRxYD*BW|Ctn>j?%mNS7_ao$Hk^7$Q10bA{&IL4}
zEHy`_Y)v60>*-H?jIqprcLIg*d{ReT=ec3JJYHP4hx9wSd<8>eYBr4=pHfQzZi_
zvqq1%ODMC(DLpnH=Dc+U*N}30LnyKtr04t_2i{Nt={P|TaBt=mDEd;vN`y}iH^O76
zs`sYJk?sl-_@qh-l#c7tke&`y2i={iviNDh_{{^MCiEvC87Dyi!XqSfitWo#z6qg$
zZPsF+8MWtjtYE7Df_el+oFKdR>cd%dFeZskHRK9gMtDj&h78Tsa?0C(6q<(w2cw?1
zYK;O#rgwPvcUD$PErA$*58c6J;0YuFPBa{zAt1X79i=}FnrU8O=&p*oJBuf*fqgi*
z9|0qK6?nOGW?o@+CQ*R6E?&Dfx1$056LQs0D(7ZP>lzx!fx}%pa4~6(Hiqt;^RPS=
zhVE@c(4y+IxXHkv749{c8sQ^I0*@~EEM_st(=Zjzp}(A$5F20w3c&XtAH?h*Rzg(F
z8QtM^-?dISI9|x9nWiHAPopJNZDc*7=CID((iW%{a)d5hLs0N+Z&u7hD0&)^vhu~Pdu-o>tWWbeos^kHD<=M}5CSn#k
znNbq=?_Y&h2$!JKGDWuu0+CPcSEm6h%pT#7O(d?Qp!c3jUCX5zD99H9nbQW~fs%T9
zdatSJCzPVn5%7%Z`(>JS9gz@H935av<_2FE8;-I?kN(7X5vapsVnij*`B
za)KFjb`Uj6%3*dRa1JVy?HWKw1%B*OvL3Dp+cGHwmaEX9_RJB2F#So`eDrF-AfvgA
z{Vq`OCqkQo8UBnp_(FppkncQnk=6$3MwXxf+y$&b5s=UcgoUi;;1NAg=I{dlQFAaJ
zZJzxTI51v^`EKv(70Ahopk18r&1x3AiV}}v&Z?wmW|l#C9Vp>n%+t;cn&%*GSb(&J
zwtJOd)_sr!TbrR>3>i5Sbfpp_AC>v6fmTt2y02*%YBd5y%iXLY=L1MDW`&vh|Rod{jWC|FM>2$Kk|
zaow-k@5#X~Ct9Y141qM70hS>R7Es>lAkt;1Y7rm|das5p1fg)$94amJ<){M!CF8UF
z49oxt1F!Li&!Vz=dhvjZ?g*TEPqyw5*ZWE`Z{3FgoIob;?a$0k8PTnG
zhs66E``>3$jWhABHaJNMp^q
zZ8QQl7GQ=8#ED{UpMqxV2{6{s77RCqU~1Ih4LWe~r*(BJ{`qlls3Z7H;wt{uve~zb
z>@b1?O6+j;#rEQWQIvX`{B_Wuy^A{yYIM{GV$#;Ft2a89?(2MU%e~@VY&qQY_bY1T
zK6(!F!4-Zhh$Np>GA;?7L*&nQS=g7^n;Z`vuuOI%Et&3s5-l8885{!J6aacu>t@vUh($I8QmI2P;>xvaKQ3VX(bB^q
zl`?-(74f@kaHqQ1;9z+D8mYMisOk*|0dXIf)*je;NzI*T#q3|PxSp{_TGui&(dzN1-mH!p65-HiNrK~To
z%5RbG6s%GBi^zZTXt66n*`827Ihf;FdYAGH9XCuq0Cf%^ja2iTs_^7@;~e}i>u-McHH%Ayy2*eB_d(x5&QDX1$TAp
ziyyy^5a^_&%we+at>|FV0wJ&O-o=(eq8^PAL~r24LqM$85>g_)Z*d2kBd*53xpi1Z
zOs7)?t^TjZ{fh(@DW5DEJCjt~a7`Jm-NQx`8pa^(pgjTcY51q6_{HMU80l!HjB>Yo
zK-VLAya#kk0qpP?cGv$-5EOz&abq$B=iHHJTuoz`NmDaHTunbQLq_nrRrdds@cu&V
zBQ$fkaU7emMMv>0=2(q08(t}260jayd74`qK0Xo2p>{qQu6z#l%_sdD&!)!4v9qT@
z@0}D5nFfy=IFSEWu-DPqH`oB=~!DTO6)!jD^%-
z2f5skm!MAjDq@wFauUr~Y}&BQUHmJ(g_6-9naGaYJ_hQU8uo?HxBBIdf-M>
z3Dmem(f@nO)(;!%>TE5aO>}oQWLQoxMv{ojfDH6nEA6?fZ8-Yn}<%SrSd0{IwTc?A44j`+gqzj}Lh33;o
z@Tav_WbtKAvmG?=uX5gnX(JU&%WRMB2+T1nlX5(1acsQBBl$^PJzML4{i<|#B1!q(
zW-nf2+9N#iV=b&GubtbIXf4`|=oFS#@=@^A5%-M;Ab3=To_^7{Z{K!5!c!7|{Qypo
zC8sh8lu*pbbXIT_CLR-dqn+VU{sm3DP#ghA8UMe5_@$)I`FR&H`EUNanmQN6PO+W7
zhN%uOlMAEavq*bD~pxeOn!$HMyZ~k*oD3
z1Yttf?h^oViQj>BHQ(Qyiw2D!prZ+kJY~%B4O96$7|Uwqh1i6Y5%<1`D1ios|BdhX
z`PcOl=sbi6B^a#howhY2czkTCUlN`oCOtp1PzTctJ-{JsEe^VGKpRfM+-4y3-
zY`)^uOQekl9$j<6VlvZK*Vp~}%H`ZQ`3H7OUkB(jQd5F4u2WE_2lNCX-Lgu=)vw{HfBqy=2;4BuOSy*GZYo%v$U;qKHS<}qdHJv(OOxtBN;)61
z6E40@8+7D!Ah^Axn{?PK?m@5X@O%Xs>H5zd($Awu8EZ02IOvS$-JdhGZ+7XlGqi*GM
zO4rBVq%hfVlz3=4TpeLA-JA3-=UX7nXdP7=`>5^IVPyiX!_hR!!}CH7$K%VrLN21=
zJ3aYnsW3CSrG_G$uK4eiDrE}-dr`#RIlhY^CPb!C2{vge^kt&iO(T{L^FO-d+VNRf-_r0Wv5DD#&?I4&T^ACXl@SrJ=A?
z%hAgM5Y0$WF9D*%wUVa0XbxGyWosY3=*3qFWEfA548nib;&wzYVn(JAym_Ufj-|GN
z$8D|QsG>|UWagGL%VuQ55wREh?nM&^>XtfBpaXqacEC
z#otIPDD**Bi`S_BJOVt24AbbUXiNjzb|fEakygP1ZlUjmjuuGpw%I@f|H|cuo}0m&
zPF>EQQf~IMy?Te5*)!iWvO1bj>0j#^zo|g40eYKs`rzF`{5#X(;>&8{t@KAyN|jyp
zpnXOVr6}oLyw%p%k;`hVVK@i|MBC;V6MT7lDcZ;+*xLK2^J2;(Jbg
z#ZwfdC?Z47i&L^zXoZ-1u?U{$!{cGUjen_QHD(NfQG3VwiEd*n=8<}}pv}_Q1zU^FX5m+dOJM6BSaKko1CO(AG6O(?wTJ>3?!z~h#YvWCgGxVbGMlIJ+kmdql%pSHe|
z|7Tle;3c(9&^b}{HJE!Z-Vt4CDe$T!e!iIH?N0DIw!*b}W1Xxnn@1Z}$9VGFD6SP>
z5~$Kwq${aVeZlD`y(!UW)eH>FliZ&{xsUnf+rt
zoQE=obHkGxz5uXygrC$G!7T|X51UBj}BSX@7soI|INm*9-
zrn#fP`iP%+_3KKM*I)!kPmPfJIg56k!wg!}>J;{A98eFY;oy^`!zhS*74j@p-S~eo
z7HIn)ClpD-M7;psVYY0V-7o1WCyV$-0d^GiOG8@baP!bn@`xcV^u2yIy**|Wy1L@I
z-8@N~{#ttXajy=2E}NYp1tf
z^hiNzt!HEkA>A><kSx})zfn_>c^d2>q?JCbNCn*e;&*}TX%esg$e7P*NSdWR<^yOTM@L6bGJH^M@`
z1Jk#UIo0F41iPy~`A
z5FD)bATbXW`U$XZCbWwIYv##?cT05&2UA4~{^jpJC?z8ck|87?2-~aD#ZrI6ozUK1=!d
zdVA-dP=fBqgdJ(JAOE?3(!tm5=~05$c)ThRo48$^#}b{7xBdG*Fpk<}#FBWl=#Z9X
zm!aJwQ(Q6CVHuh*5P*RmcCD8>vVQR>GOovOz_-og?eBJ~VmvW+q}C4f9<|85zZ)i%
z?cQTa^veiNS$QyVKv(pr#QBy_tkvl<&nP?HFRn0+W{O$1=eTnHdX~jdaRzjC+{2LD
zf^-UMW(3PXlro4HR_q}mo3DY)^
z&}c9|7}hCKL*Z%|=vdg|d4kT<;&U0b{V(Pi+dLEU$jDMt!DyuwigJD(p
zA}24;z=_~%kgUSUrHs6Mu|jp(SY@xriyKImRSbMPl6@-Kkl&m<=R5nk2L=*4?IYMv
zV_mf2s~%t$0OdydX9dgWuig@eUv1k>;D(Kg$lU+B@@JcP8On3E$}1#NSC!Xd3*JlR
z%HCDJ1z$Q))OSQUri8xNiZ35|>$wQR_MjL`DB83`@Y<|6*mmRWbWq~y_1{0>taZ2(
zM5>7sA$yWgD2mv3?KgZ$)@5poalJ<)3y
z{u!ZWyZYyPSGmf2)j8*q-MJroTnLgC#$}`{^U-L_*iyKM9%>mHsP?D;X<=PF0%AbU
z1FbBeTDTRs7H69NYyv++ak#yBA@-+qd09WM_v&nsuZb=-TF6y1SBbQ|wJ|iLeanel
zns^rk&tV{8ifi){fQ1uXZr49ZC04An@4ek-=Y3c4)>40~*DpX3x~=$|FjXyrFl?yw
zTkdfUwn&s&i|kjO$vLIGo8jCxP!Dq>iI0HBJ02F%z!Qh|-s8YO4$pjEe
zOl-p&ju^$cR(X4U5j%iHocNUP7n(%upytgkVU75gFr8q&&44^K@tcic0+OELS)pD8
z{TaLWE~XzQnq$dMV)h
zla+`Ay7pMXA5ex#DTn4?mdzWlcruf?ky;m`@a`O6C~t01Y^{6*_0>QOO7RW|nPA)n
za)ItAy{htPKC5gfaq_{u#HFqO&#UzD8osDfv;2ue;ZA8EvE+217&dwJ%d1AZIWUv4O<93#D!of%AcTo^k8>`ub}`olnMG&(0)*ZvXR&G
z(D`V;uLkB8Nu}CWXOjc9DfDVwSHI!{ZNYmyMwAQ9
zmpRaT7ary$p-w0AJVeo+Fft+HXc#h3=0EO~DHdR*dHSIf6ZVGElLy%h+^CIF>N{;4byK54h52*?6#Mc(6=q
zvG0iARllR?#psjv=&SOgkF^F7(XuekBuYVMGU$ajw#@8jPAs{*{xmpBJ9|gL
z&mxQ2jJacuD)n<*=(&ZsD-K%Pgr|hsxk=7r&r!Ldf`?5|rXu~#>+0cMw{$d?#>!%(q
zZD$#D3k^Z-oL@m`OpoH=;4$qy4H4!(Og$O!Xth(a>sP-6{Y}1)ZUFQge!_6-fIWyh
zOZE;y+HzIEHig~u#&Uc(H0vil;gQ=Q`|oL$ML@Wf&!c0-*CRy)&1UR
z7i)m>^6?Pp;wC_4(Ezd+A_|I@``M32oF@q2_-d!Em_p*9EG76XEN
zdwW-f99dyx*E=YPh=Cb;j(LZ_X+Xb?h`(}F4q)#7^7_U`ROJ1AgQbf!DOQVE_iq!H
z_r7hp9Q&rR)g2@P{C3mto(%wC&NJ&_M!M5>Fx$lMXbS!CVhvQTqDzD>4|rb_hb5To?zsw6X$W
z2I+YE&vs;FFPd1Ci=JMBCrGCx_A(|Mjl
z>VE3CkoGD@HxyVa;bdJ%)SF+LV5(5nSRWs{(9zLR)6ig@d4UB4;C*ZBTRtyd`~>;Z
zX&jt^VRaA)EV|r6DMG#46LcxS{Uc0XHHE9@=v_ZPK4w2)CuwT`Ju@@Y-9Ik_50sRa
zJ`3ZNhFc;OHZyJ4525?rU+p1;xEoxT1lncz>IDHvH?N^gdjy2$ttdIi9fyf}f`Nj!
z(D=u^F5CDaUAFU{rEgj7Ht5e<2T!*L227`kBP?@
z%&*ulv+t)xR*j2q=zcXy!n`HZp|s8BeS`Aq1M-J&-@X<_U**JZJk=ZIePd5mM3KB}
zW0BVRDRrg&CM)FklDfkY*g(-F{F@L=c|d52JfL5soTOr*i{0P_Wuv;V(-flNt2b|U
zJ}U|s2B!l0tUGNsIRy=1ETrV%w55LT8i9;WckII4
zc!%V&^bd~L$ataFiWQ4tl#x!!`HOPEzHy{HP@fgEY
z<7{h&dHES>{bfl30fDoK4sxYcU;vN4xbL{fq3>|S6K}%_SNy~E6IVdWNnYmJ=cb?Y
z5ocL<BX%5a<#oq3~q>=*}`jqzFU)i3XaG6wwNLl%~np(10oKbC?-
zMMPLg0$ie5X-UVjtBcG1?WbP&XoO4dURFZ*uY`n7_<*Wv>bSCq$l;=efhbh3s&oPs
zg%c%4aYbdJRo*_L%3tkGUJBOoacFO75lxXaEy^pOX=)0I$qq0B+Z79-8MDtM#|?AB
z<9u~^$;LT>P$SX_+_3^-&}ISA(ouUwdt83|_9JuxY91aX^C&+?ORm7wuM%1BNy$C{
z`+I-w-L%Is(YhU^Fyp?#!NNCgx}E)=qN|Fp6D1T96E7=dB&N3JCXTgI4<3pTY?*)A
zt8m=>VUP^bcWxWOUS7U~Rab}ji9+t$y11_|z@0HjeFgqX^4Guz4;&X=)ej!5{Y~mr
zQzi^dyDEE04Xkw`qz0#}r%D`$9$JB;`?)I%yct0V)RO3J@SI#qH{J}aByi|k-nD_a
z{WO(I(=398&>fbu*gVPfnW;l4xma^S8u&sHiy*9%T?+nR&DWZ`IyRKIr+6-0fcuLP
z2mAJsH|P%5V=!iK?TJ^;hJjpD<$}5W*tcEwplR+d_Ec6@evR3gcWD(A?`rsa?oLk>
zZt4h&_o-CU_Uu!uAXtX>5`Bjk;stAax2+}e1*Ig8Ef|c8d-X+14Do7lqmwOH$B-0X{kftTY{sj|p(#FH^H)=<7
zP>>~+Y&j*3RiIt7gS5Q9@e5TvU=d~m!T
zg-}dtgR2`F(nAIZ5$q?SW2y-Q=PnkoL8c`>VzPa2AfIvo9B(>$=FqH@3TDeXAj0II_TsOozK52T7pgJ}HS<4iQ2nc~m_D
z!N^1pPk?D@uaNpNTFDO^xfe9F0;PRwYC60B9v0`Yl$oYXQRTsYj(ul)QQU_!%DXQw
zI*oP~xA!b>-L|c!))`srI=pWv%7&)1QfieAOH=TSqz0Yo%A=T=K$rTBI+~
zrS4NMz)ZTk>~^aJ(qD->qHbB?!i5W$IxAbzbiRhF#C~DgQd>)l9~)sCi1VVzQoBq-qxbqeENuf_B0H*RQ77hDu*K;4_+ok&T*vbdy)w)`TJiq9?_TF|pT0f|
zRyMXRh~irj!gXW%@z8q3WHnlm+vOofa@byjyxc5@gDT;%j7ZKQf2H$FGf!T)l-UNP
zxYNrYNk^>~WoUzj-8seeyC?r__ok)b_rNe&Mx&0DiAR>klcEz(D6e}FF3I;sZr5JY
zECH=(i(-S#)0i{`Wpb!Fp+Zu{^$~IvZv0BDuCM1fz|N%WfI;yNP)^KuZqjYZ1k`M>
z%!}mR8|^x-T^yL6o=((VFM{GrSVI7kD>jKieXy!I`w_|Siwb94J5u2b{j~*z%
zI*05URh1ckvn8Zs^v~>wo;l*EpHxXkn~Cz)t!j#po%KQ+u)X=y=C4Jt9!s9%-f{L0
zh_^Wf^%$Gzh8CeovOaAX!l5rEmEODh?%lge?7vhRjiZ~0TW4*q8igid;G1%90|==B
zhcRIp{;q&07~gaT{F4fFxRpHzv15#`DllkJsj5HWBR_=oD!6H-9p?5P^F6f96FCSJryHJ=e92DXojbpr
zH}r%YqTE8}VzeIQ(_bo43DI~R_N0hy3R5;V+2FjB=&LPyv$Ov~>W>RMH~E?Eb^IAL
zyr2>=kd2=na#@VAL?kdfl?WnF^6c8f(Tx@(yLF=Eubk#7M0E(iy&s=ukBAW=5fR(5
z=Uy7lk)3ta?=(@f{J5nP{f~r^&2k3B0)AZ5Tz{7E@iBhiMtmxT?P*&9sRMX7F^jjW
zn{ye$`(~#@{+pg}qWbpjs!@XN_q=DrW9FB|FX*{*u1mFVOg)A@$-yS1$$|jO2Dz`}
zu)vHV$6Cus5W)R0H!EznRucJz&3{cSBkUrE>T;VKvqfz_(s+>_qxX9cy#Q|}m7?id
zvExrSr!f$B@7(c05H@;>VnNrRueNps+|!G5#o3~Rm|-(5r-e9MNgCz6`q1P6&z|Ii
z!JtqmghZ31`FD&*``taT8CB;O7i+)bH3q6SFswh|gP5J%`Qxc`r*^8ImR4-?hb~kj
zy=Z~1b)7>cc3j>te|VmSb#r0jl8(i~yJk!KLzF6MA2l>id3@dD(L@BX96%$j_InYv
z(b)J5*!U=`swt(M$G>m)=xF@=8omy*5o(C*WS`VAw}=Knx82Y+AfTAg0#x@C&vMEs
zz*pq|S$~m$A2vd3(ZH)o{;(l_Vqz7PT8JTlcb6DO_8fh7FV=H#a9GYMtt>Qn*5*vj
z&E*<4+BG}BM%!~1U~1&M;I=4`$s}P!B~-FBtr7g>$>^6IUq!noItp8Q#;_N{+(`kyRF5XLuDsvdD*f|+|4<>@dP~wH5#EX$3
zU}q!VHbR<1bhvwfX_9#vDU{#!|E_Sr?*nWHe9$Htd|Kyaou_oEMff6AJxV6JlGOYp
zeBiy$TdYnmO=R`dLf?G`B0trcy=uxt=>7$o8$LeY4}gt4g+Q|51cI;BRLO_)S$3>_rFH;7-a2LFQhLZ1%^X1R`Jg
zpfvAAG01ee5UbechI;A~d{(`?T!C+8E|As=<8o+!0nA9wb^U-O;8UihhIb38e6W}S
z8CU_F{^xs7%*@Q}fPvw!v3AZw0o#?=G*rGiox(0J}~P->Yc~FabR?C+&#RC*Hu$4CNk0&N2NLp
z<;;oMedxWJO1))=MrwbcM-Rft}rAHt*xS0NeZ#a^I-kQ<`>`pK2_6-eudmQQX(v0^b
zV+<_9=|mm23D$g;mMGpc7mzrff%$cOjI1Iv7Z4MT3e2wNq`9BazCvsf2w-t!03l1eU(!=Y$H3Sx
zW#*(_NDLAk3Fa8**+C;W@>Rm7tAr;|aT*VW*r@;k&6FD&ezHk_+0~-k&u0X=(wI&g
z>B)Hb@Zq3kUhdGauzQ#dCf$lg-BY>wO3|~z>2%Y&RkcuVDwTvbs_NKz4Yj^7{9m&3}gngB#<<5!3Wxk
z^}mOm_ycx!s!-aFZkVq>FAJX?UpRXh6pv^d-p}H~`0z6$1xjIwXiT0hoTK2jRM0m^
zO3VgjjN`Ruqvmm59bLw`l|8s2KJ%_0LfZ_Ke*gLNLIK>nZ^0_y8DQmP-!5q9ACP8b
z!>*8VZRqUmteiPO=LT;FJSv!lqw<_AEMCA@Lyy%cDD$D?eJFv`_2}~-=m=?m3t85A
ztS&o;NAoou&Az*3F!1D=GyF(UmUSorFR$FwX!80}xuH!`ZGZma
z1qtIOcc0ML=UM@rKAgFfQZ=m{x-nwEodSg5nzCGRDxl6
zi+ruu;nZX~qWEVti-cJRf_%_~z9q3%fY}CHyt1FkhA>hC^l(^YQon?0+Fw{V6MDRF
zMYmfwSq2us0Iu-YlGb=vUEMky45`Kvpg+*=eCF3|ch>lXQ{$K0b6$_BtQW
zR@}Fb7br_P$`U8NmF9B`F-P?z&acl~0}RNUY0n8O!KP5~VIU5m#^nL1$3*}v*Y&$c
zU+-O81Q~JWQ4O#*hi%{Bk$;rkq^g`b7~j}~I-tZa3*7{XB9eg|VS?yPJM#73s?k+K_Iz
zMe=!-6YZ#Yc0lCVc7s#Ov;z6aZxC{vyXawm7A&3fOQB>le9PyT+j(Oh3_4UWWpFYo
zY0|)Q`d@y(`0_(*D^)es#t3S)zT+rYZUZCx8!^zTjGb*N
zbS{si0Sq^Y8*_7a@87oLDfLqAH`2F*nwq*g?1DuJlS})!=RRyFMMv{nGLbM=4r*(`9FU+SJQ3YWXfGDR0^OjHFo)b*U1|Gbb8Im
zP{n(JVrdu)IUp}?kdkWYG`=Fur9%-6OD@t=u0U*c`1R6>K8g_4ve&dhw7vU<vmrL#ts}$1rn)G`ByK%pkOm<
zmv^W80n;-3`iR24#WrJrXf
zKmsOh)1dSmagvY8oc=LftL1W%#({elAXxPPsj%9fz7C?~L(q9xIx`uLuY4h0Qy)OG
zNEQ=MFV)h7p^B~@fHPR*`q_%igCk6h*c*El9Un;47H+Kjvy=#>@vx!E$=a^V7(c#x
zVE_`={Z7@0K%HOcZsKZ9Q1BI(M=Yf(B30qJxjWG)_b!0kA%MOy%vsFa^VGh*hXech
ziYI^nCT#%8;UU(S3Vn^F$7O^x{d|Y#9R(NFQB#~;5uPTvDFaYWBswAp+oS8M4MXG4
zK>89Erpl_0F99kgCvV9Q<*-by7nn)ub?^Ics1o21@@jsz
ztW1UUO6VtKxXfAw*^sa>QiYg`}$ez>$kcAyEj@q7bt`sdZkhqKm?gU?P
zv!Q2ycWAI36UE?7E;->i`uJIC-Al~%Pvo!g0-_Q$k^}2<2|{|VUDhETMw*KqALXT<&xJDuVc(k1M;k+!SBRYzu&^)wVDaFnJE%IE1$N5zH
zNwoEKsrwTY8v;>Nv@K!Yest#t0}qdVm_fXf^Ag!?NX{7XeNM(~{x7VBDT^kXn){=_
z@Qod9+%h&5&%Hu=Z9X677)zcGlWn`CD*S*
zm)EafZ&3#uE5nQFjDRV%bgM5jGG0XUKS9&Hrv<6A-+s;v4jHUmTt6Su+quN>)xO^<
zKn=p`(-kDOah+X`Fsb-HFqg6Gd(5udLB9XaozY5S;*Gnqmba#BHNG2lNE2!d1grMYUj*+%W7=;+=R%`(#8YV9
z4R
zca>a45uo$V-5vG9@!}TwF1OoAqnsyd>5r-V1_WHQUPkii0xFZ``atwH>VssDLG%lO
z9b+M(1?Zf7ElwyDiS{mMzy8YIRG|A7g!S-iTRvtvp
zkj$@vv|#J;2W^d0-r22p%Fv-()@|Bp73)&KzViCUjym)5c-0AH;*B$O)5P+L8Q6SsXt_)XFt`ezEwy_5NnP0RL8&A
zv~$q=?HLucbh?A3ryDekTEzN!;xVKtdS(6`?a$A)td}-V@&9(Ci)AWp0J`nyOr})_
zoYMl+XO+z^7uC6xJ&J6|0G8|sG4Y%Fo?ARUII7X`QKJGnA1t8(Y@JlJ)J=!-){;$$
zE5S;oqNcM4mIM2dAua3R2zDPWR#ZD~o9HQBT~4<#fTHsgZop|SC@Cqif_TLyBHlSK
zqW=6+*XnZ2sQKe#oSh&VC2}jvY(CPdwezKiLt1
z=R4IQk^JiMZil5=NlZrZLEn*Fh&uFbQ?2I$cfVoof9j6h7{a)Qg24V2Z616R8iv+2
zVc5U|{mZj=?+TwNx~#`r4VB3H-GQ|G7^=uorcam#nWOb%difdCmVgHj7~1kl%1qMN
zKS&OLafW$7Qam@v&as_4EJ2XWE%~$;Fh4L!&v*|&Se+bFywaS8;`=P>flR0O0X#-R
z6aD1+!>8D6m)aa8^sm7IF$qh@m`8Z}OfA8J^(V@_%FBW{2|hxXVxEeQT5Gckix&%=
z23dm#N#9W|Q#RP&E2WGK_VxLA^WHd|T@NDoD_hVQwj1RF8SMpREvMXqm|lT-gzf~D
z3dY+Cvphd~do9{A7txgi$)snp#CX93*Hx%vm+Q8_fy>|$JlmN(2k^E@vpy1vZ7v~A
zH>28D`lUSWhvx+Du*A6kj?G*iQJm>8GW)oJ9^!!yN1a&Q70SEp+PszRpAQGi#`X9~ic
zUjPm!+n1J?cYVHyLWRjQ8mQaa+i$gsQh}$5OHfi$s^K27cYnl=7-F>0o;)Fq2`s#6
z&94?BYWb35r<`?+jM*-)$ae~NGV2Z?$X&Jt#W2K89)kwl(rFohUGi7Yp1YiXC(ZRJ
zqt&sucY4bH4~E=Oz9((uFN8Tuf2b}w?ZbZ%Cc2YxITDc`7^LP0XeB~VPj6$(w|;%E
zU48}PR_bDf=gJJ<%O(djolMQCAu(b~N*x0pfHG3Qy?}G(>AFS^gA&mZbf71q^}lT3
z*7+jGt^eh;aW>|pDTxwoW#^qbhSC)R3LpQ4_ji-?=S<2d%S{S)v8@oOn-%l+ZAMQoD=z^gP3Xhx@=f-pXSf^DIAHS)j8714BW$!(m#EdS4xmEudl)
z&I2Q>K;x@>_eLG*_eIh)3ONRVmN(}B7h{i-ySyGug#&Z~*e2UhSqalH^S(07CFitz
z_Zm4<@N29gT^oDJ;Z&6rO4cP8*`faC2(g$^v!;Uhn5qc~pU+q5Apr=cfw5Zscv~}c&r)M}bFWypDZ{Rrx
zD?!fun}5ufuaj!+=#2OaZQo!b)(WV>MVP1LVq#pKo+T0*jaQ;RU#0!`cUuK$ynGVX
zFh}dtvi$Jv7YwD^+yl{Q*I%Z>rpIAW2RNaA=1lAWkL6%P3@1!@8jH?dfkh9Kdldro
zMC627*7;*8{^>UsBoX`A!4A0=cpE4+!Zj057D6asjqXo#9l(^5Iq%6ZUM{`{I~Iqe
zvap?mg-pcGKy$cos3CY?_Gq~f?ehLv!waXuI7gWF+2!vl3#shu@87m>q3_Xp_ZOsk
zVc261)&;*-jy?d<~>y|Bus+fxce
zrBZzeE!Ye-XWrY=zxM)$3*xIkcN64|Frw=sG==OzcLGvcTd;J3fzW=<>-*!n*O3E;
z6#Qmb-0EmxqB|Uaep;{lg=}34O;#jAX)ND%bz&=(_P%dK?Piq*mK2wF$vM!L225NK
z_jx5$tr3B^XH5lLJT01otHZ_MBHB797WmQ;z=G}rKW+qi
z{j#Lt)Cr~jPghLn1YmEG)Ry&;hf)PSG#jqR*tVicj^uAvW2E|0?T^?+!xsq&K4>@h
zEH8siR*5Fn$=*YM=?C9x#|c0>0fFZs2cvC-kv586XhNcEh;o_(Y0ei3XZ-
zJMdoJaLa@hxbB~0^6@p!R@$;<|B$4-VP&fSdk~A_>
zTb26628thVjjl43ymbFtr05Js3WV*$2(0Iv^5RQD0YFWQDbed>?=
z&<$4~%+u9L23o1-YHzKvdZN$7S-Gg!Z~+)?&t=B#4P}U$XlH%NpNHy470fqS={)%K
zN$bwNdvVSaotB0*#h}&6UFm#)Y|FoU_eQ{ZbnDis0!RD=H~@EAY{dI4SU5J(Rmz7e
zRSJyGPEQ{IIb@0i3Z}rn{(%aj5m?w7wC{sbwKU&n#lz@vQgszRY-UJQruSa+9;O
z>R>V+IB)ij=#Q7Q4!6^
zuQTNsx8$2+SnkvBnHjmdgXk*>Q4OEmsKT4LlOk4gLjd6A#8(nvv&gMt$S9*tJ}wC$
zR7@)y)UV1!h}vqjQ6V`n>qHV-k&Xi>r!|kEu@DVK{-(s-L@XM9I;^~|qdSBcSXo2G
z{*u2z;a!zIK^<#FC6pUCN>L%utr(Q0tt_yx>P&P<$YPX>K?&Wd5Ih&R1CDbM%FBPV
zBeW;;cwv?!MT^5JlfTbz4E6D;)dM^;AoD+;+^1{X>be&}M&``?qRsnDt69bLV(T#bIL+d?Oi|q~%A-Kh2|7)`25&zEh
zVA?yg>)*o6O)BnKTed7RdgP&Zxz~YxUX(Q!l960oO?+&9`(r7k>H6%SD0Ls0THQ~?
zi?72RvmUHKR^6iv^C5x%LWYb0-hMerz#6#pZ>LVYb?M!o*nYdlUH_Kji#Qhz;Xq0u
z|3NzoI)!KUaqDW-WAz;!cm92kSmE#YQg49y59Vo)OWgCZmeYTvhcaIk7ZaJW{_3zdzBy
z#vGq9rq{+RUKy>jSjDTt`!PbG`%b83)N`?PgRFzIv0!cg{ZiTyRgawM?=bZ-?mqdl
zrhh5g%Z}!*d$fWR)AQhXn$*r0JjA&EX&?(_q%UR2|NKCZ_1J)dCHQ4hjBbz2G$4O6
z^GCLc*o=C;6p6>*HU7=L3=kgf^rdHHinDo*t~E!2Zdtn?^K4euVB3pdj65lWbayUS
z?2%+;b+L=)@-LS&GrC;fWSL+#O3~Nz3jfxBHT#8>_I!pEQ3gM`4Db_P;D08+hCX7~
zlev9JP}VE^;hu3raz9kxo~e*BFXm}2|F)VS^|$GD;WAS4;$FL?|DuaxiNE7IrW+z6
zB4_B?{(hTaTXuH#Gx)RGN@c3UYUc6b`9D`<{(YVgY@RCSM(M@B&#a^I{B>+MH6RQZ
zu0!{{vg9NDR#r|G^uT%KHM9mVr2l>|)}i;f2+a;&-Tl3u9L<;4`3hZa{Cj?@KTlV9
z9_SnxvEujnbzim2Md9|-76uW9^#($x8Uritl#ib^^(|dTFl3MNBb}gf12=cysYvWD
ze2Kz1o}NspzygPQ$B53X%$mIfO;!N*5${9c70xno_TQUo^p+$<7P}nc?^vGYKiqYk
z@+W`91YNhDklo{WI$x2Cnx%1xzw6xA6v*D-L0EW$sD3?ca9u*D0ly_cp*_}S+U
z209oTzhNSGv`*Vh1p|A|n05AosOysS&ma3RC3+XZtzX5zkTS|)p|w;m8F|AW*y0R(vHMAV+dA7T1w9rS7(BT&<&lK2m&unIcQjHsnqpGH%edOxTzbWilw3&d&}-YLIewFT&#n~wml{@sO{qg3
z5q2BP3Y&O9xIGJY0YKAp$^Z7CcD{y&26=r}@Z-mcZbQX!H&QGQZ?1b<-&B+N-T2M8
zD;XNJw|!vilBgXV&7PaytuhWp%Jlh%*Ed|aLF~78Fe(($r?1{i;kKot21B|XXs1xR
zdT&4R@AdXW+Pjx)pPK@%IZJgL+DsJkGp{`6dE8Y=9zET1MSI~d4S!y5SbBY?yUM+9
z`vAxT@3@}o>xoTn88znLM>F}zQ5bMQC9XKQ9?t%6vxoYDXV0){CmsGmx8xU_*NIrB
z$-^=BT3&?>`?P58>@(*H$KKVBp5I~9)tzobdp@0S?j2I%lPwhd=I5FUq(#DunYuYG
zDe_^t%Pkf0ciZ^X>YO~e9)N6o^QT!2?PZ-!t-&;23Gd1{TJ;0}F0!~wT=s8cqj#k{
zSo%yul)MzEpLp-slV7ssUFr6hHiRri#Sv!`llV2-#s-tPL2drOJ0)D=eqzU*=g#)f|A)2Dk>I8pk>?aALLB&2?=j>TQ{JZ;T
z2qq7+u(ArgEfoA8E&!H7`@RG=I%?N1H}Yns)Mp!w^4c?=H-EAmS*5JZ`S*iw)E
z{&u@YNt%qS+gjg$WwFbNdp|@((@thNUO+;<4ycI4nV{&H^uZ6oSMWQOf8?K$6dr7E
z-IArN+Dbb+4t6v*{iu4CSasBu@--O)qJxC3XDQEqswyVMGZ#dAuF?3H(qNcU!*GVv
ziy+1l_CE0UuLLb+4ao%7T^M}@%$4A%w-6{7Jwc9@HZdz&_W>gzLC*pPd;le7V!dr_eZjb~M_VZqp-
zfr({T1)GTQdF=SYJ#@>Deq`7LV=hCBV
zoV*Sj9!{g`UPIizTx^fQ4Hohyx<*p8I5{=dqfP3p5LUM$a@m84jCy7>SXK(~x-LGJ
zL#5BqEo2j@8GNNb{VR*Sp_9plk-Y4`k&(WZ?MjH?i7&OA+x}F!OAwREkRf{oJ{FID
zr&W77^zKimm$y
z0d4OW4@{MAo8QvRb+>_X6CH^C|NHGiHF&++9}aEssv{(y+3Dx{i7*fb>pqH5sTdY}
ze#mPdIh0>S3=4E|EhQO;Q1Ji8#GfVJnVLs%XM_+Y>-r*KR2DwIK+NJv{u)GZDxg9YcRZ$8
zdQeU~y7EGyYxX2p1_R^9d-(0|+f>Z0s1C$XW4xmsUC-*GRbn+JPzb#79cU^%?$v4A
z%pSB{^sriJ|K8}+GfR(Kl{lqD
zWKk%d9XSTmd6LROk_s{3GpCZ!-i{8p7syX5Je4u8M}>O4wcUc&Ya6lhw|E4!d0p~kL
zV(kYF>1;Pw6Bq??h*it}2+Yb2x~|B<$=snbo$u(JB6hOUh3T$*tiCu^Mr6%NO@;Z$
zuRo9E+*q|*bIoRUe#g6Oliq{2{Znp@vu(XIaRrnvbgH|@8VD!;!yAcAYL6lc#kJkK
zM057FZ#c>Tm1xerf3P2nw!lr=yYEBX3knV{8KE&Qaclc#9YK^3EO!fwmyAtRGK?>2
z5(g=}2nH#}d6F`8W}sands7HIVj4?h=w(iZ$6YwtsQv28{TPU^|m8=hr6
zZ8*EpE1i6mVlz3EDu(Y$AaP0H_@zzbg)RO)S|T}J-|c4kwVl%}_Z+^zEanedt8$7k
zqx_kW9gkrFdA}nmuQA^_9iS$thaL!;6|oJH#y)^zjf}>`>0QT`d&+MW|5|KfDf1@2
zS*LRG@&*9A+9D(Ku!R|YuTx1Lcn?q#x17GxmPK?DK{^yJf&?|y<4Y1|4A!h(ypJN|
zbYpHvb1^C=qKXzQ7iZzT%1EoX9FW2(^Gio>>+)lo?u#``Cke5~;=`Y7-piyu>k4^S
zvYU-)qFmW4G7p-YB+NUsSt7)O+5>
z=0z7#E>LZwAa35g39$;ZG;HThmC;*#pQ(g~tq1DwasX;?P8*yGk&3L_ae|uu%Bve|
z%6`;xq|CcKn+qBE$=*hHkj-@oc+i40C*4x!kj!t7h0P9`(%if+^{u)(L~o*l!+=t3_8QJkW-O89W&bBd>Cxe
zLn3|;)BB6XMQnH9MsSB5I~Tq`SZa}&r!Ko)%VG38$?v9A?oH1zQN_bR7S>0|U!8m>
zJZQIXpde+~H^Hnyb;WC8n+OACaO9`?t_WTg!~MKjp-!zMRagDjWE__%?bPkli%tlVb%$6or)@zK&
zGvK>}zSN*OxGn
zB<98Y@P{W?8N~~~j=@=xn2vLoW%f?C7^d@iz2!q8FzwagQcZP0r)FcVeM}Rf#ZNym
z|C8Tj2e5WC&*fdXyrpc*{=V~|dgy6JD{chGTf5|!GVw!OVc_bKadB-^)nuyq4P{DA
zp(I)i@+@HB2-@3dQNH7bhri7pt~sdV{9MO>VrT5+ZslaR*EfQs4t^Z-T1-uuIfE^9
z_vrm&RnA$S>Wj^znK%xv*D18Ix=AMbgL1IS}{;q%!Iys|bt
zjzW_nXyVLUQy0b^6#Awi+8iSth4LkfU!Fs`%BftR)nknr<+}2pxT(XyZ1vHSNh!vCAgG-|PH{&W9g7Y0fY`
zQh$^oGXV>+(PE}(qdVmnF0P71bgix2VwmV4mI5aa-IM+zqrJPe3rGzvshHiDemn6Y
ziLk>ug^5=&k(()P4dKbov}!fxLi08H3P0iIRo+A43nlJms1sCFR0y_cA0NjkDNqM_
z!R7(@_)x}-K_uC3ca$(?nQN`vj=s)KT)7!+$&)V1YHA6prn;I|va$OlW0p?s=6)jU
zq<(x{zJoS4rtdl*^(5Qp%JHng(0-NF((_i#6fbqJS#*1gAE3|-`(Q?y;ak(Yo53=V
z&vNntH9K<%?oeUG<$~qW%4(sX5YMDxrIgy!EkrRd^`=lrBVD!WQnj7s`7b5zt~jF^
zgCrf~E}CzQw+~%X$U7p!eY@Ron@NhXl1_xw*UJNeRpHmn8^CXGD#nt&dg`HhNkPHY
zN!{Uutk!kamz4*IE3yW@YiZ50zOJhdw(NFMTVG5Z_7-lXt&GQt%7!%V3km)5LWFyy
zDDSx(&KLR!w*hADZD9`|-om)p@QU%iySiwFya(~Wd9ST*tIg0J6R8LGk
zv+xk`YSY}{?eV$%<7xo4>-F)VCU+5=m$?cIm03AS4zghmuD9P?P-bj;*d3UgD_^p*
zc!|c-M>HSj=?5~)z1-YU+ddUS%y%WXrMSzhd~k(b-eF;Wo?led_bryhVK2aLH4rV`
z+}uJIdZ*(Q8#;sJMQT<3X4Da`whUf7Gl|al`%x7hvj%B$)#%)66$y^Qqa)`32q-foUWQYatNoV?zTf4tZ2gVQA^^-Iiegy*INy-sYKiJ`b2QFAt2r00^_@9$LI
zuPjzPiElFKy^eT$$?8W891%-`p#o+|6sIDpsc=Opf!{W%-wX<*dc`1Em&S(*fE?Sh
zJw&{|i`*|DEv*$H_3>SYSD8Y9Wvha3k0f4|`jBfdi$@4C4$zr0fbAA5I
z+T4sAZ*XipF5I7C3#}|`D7&}Uj2-0i8`)QlZmc>IlZlR%82RylBB*I8$P7&Q4#^?P
zzpm)zx~+w=ZZ@HKdT|O@Q%>faqb|CL`5Pa3*_Fdoyt+$lrm_1K?P!BnC^#n5P42EG
z*P)^G8pp5brtgL(+Ba11ILwP3AKysG$QNVkcVU^M6gxEv4v@v%ms2Np
z+`X^Zidp_$WYg`O4dopNB~LZ}igdgSF8*2cPMRvnxjJOsfUAsE^NNLPnEeg=hi{m#
zpvv)Ci!vef;LGZRiJSvhQqvNjR2N$*wFLQyGX2H0s8qh*^SHD|<#)($+`U7E;%mER
z-&;)~Hd2b$-^;4r!cb@OF?pm4xFOk|#2v-zqQD(;dp(h5E*;r=?Et>NccX<$*M)bl
z`tN`J7G14mH4B=g?>i#swTh~<=gD)Ihl0h5fEmothzma+qMQmh$C3s{#
zfRN7DuGfA|-udRgsV3S)i?M7AaZe+i4`i7AI1hinc{VQ8A!Vi!QcO~{oBd`g%N9$l
znsGqMpPYKyeOMr)BwgkNQt_nscUlndhWuoR3XEdjLxKAC{Z7+@UR<q>w&cK(TG0!@-;gQbuMez3J06Evo+B
zsf-RL^~LwRDr$>yK&+I!6jN7*Y&scSCSvXpi-+|-S@mrbY$}lZZnl#n3LP065Ddjz
z2cuusa<(UJt^klB%HzM3#!TNHg*8!nO|FR@5X8a5MDeNZK0aDFa%eeh9z
z{m0YKS~3r~0ed2KkwU`h=Z@vXIg
zHaVwNQUeVOd=@-0d-Gbi_giXm_$avCBW=g~p;KKyvQf0KU5qex
z#eZ)?r*G{-0$jNEPzV^CO9*b_Aun=D?l;|ckUE#qf)ZtH6`FB6DLw->T{`;qr*8lo
zUg3BD`x7vpWggIE<_Df|IYA7LU2Ff%$}?+0aRD3Pq{PJ6)XnL92>8JZalN}$2rrsB
z>hiw&C`&}HtB;HPt?UAa2fXF9m?^5x+*%>u$_jQW?bSX(n^fAM(Tiia?-9f3OpOu4
zMz}4I^JkZ>sfrobepr*^@@a)l3uYSn6BKHTmE_ilXCO#H%WDoIQ*J4$e|(*Rm?U_(
zeRQdp)PdYRmL{L%&2$-F+5i4?uD&rXZkmrugE~
z$Kpa?&xhX@^)17>$pJyGmqLc7{5#w^(25=HwCX;m)~s?xC$)#F)No%S$4N$-E$vs6
zMlbexdSvj-?@-~fiS@gA^)S7*e%%k$HyYa2I*tJTV!ey^5K&%5s?!F&S+61*8q^
z8?(fgywj75`_0)K^H7%0+Fsm+svwrN2@z#~;3(t>)
zYY4AY)Cw~T^tR%GB@i_rd<5ko*B^e@2~h{~IYMfB86ebpN}Yo6o;hEj+%&
zvW8=?VW_h%+c)NmKPMLl6lHC@YE^I4@+4GPCYmIBntFO{qB3jVu)4TN@!FG=zOi0Sg?B`@
z-3(Hp+rBkHFSE^NVVM+ni;pfcA-;gu+_XZYQOHc9R7H&
zLnS5mf1WF!sXig~zLLjz$MxfvO0*ISHWo1Jo~~N()MQ4sh~QsW4-R%XUeb>UmVJ%W%2gP5`4eJ`c-~uv$ErVgG~piu;=Tk8Lh<=3+-PG
zD_uW(ywlroB**H^#J!BGZ+6t>J#Z^kQjDZ~q^F+0?Qni@9Ve5neCHz>avxaSz(hE*
z^z-SS+nDx|U)6C*y4KW;Qyac?DBAN`>oyk*{bXAGdd)}v%lR8oI)cGvrMIu|OIX^u
z(M&<-oYdZXWj17(88WA=_HCQK4dM&=Rht-Mz4g#Q_wRfL3t?py^Du7Tv)i70AM5EI
z@;cwKiJw1DVA!Qq&${%kL8aEkZ6}^=R!})R^Rwwx>d7DD%58a^p|@{6ILpo{s@>4R
zBpDq*_p^;9TH?v2T}(&sZmKW15gYBT#A{Kf(I9Xt;7(ne)z{Sh&wuL%89cym0wpVW
zY*<=b*e=Ld;!srN7H>}D(PaJ7|D1D%Q7zklhKS%rl#)^6ru^uuE7UIB!)N<%kJTld
zPO|UMwRmFcy7{hw!|ky&-aOj-I=7B*ac{hGV_n{j-J7yk89z(W{oXqle$&pjj|e^K
zKl#+z-oC2CA#kLveB%j8t97MJAtCxc|-PV=IuXjGybz|
znp&auT%^ABO*0S8c%OS>Na%>2-4CneSni#zDZjWxXU6h^Lg!U<2EVliUlB^SEt0Y%
zPk8mOpF&>P7rxT#{!O4QDyN$)=BJwN5{w%@7j}N_r1{9m!{2yyR$uT`PdIaWZ+XuA
zQ9gNP)q63ffhyLH(T(kiq1N0hy)-ht>#P+eYrT)~t{xx9g2g6D)oy>=;q&U~(+f@6
z+{fBJ9`P+J@!vUkE=^|KCx*(|Ot|Ek51E5B1II~7-$GsVkA>`||DHoYho0kiiKqIP
zsCavg3ywFwF?;ZQ|CIN9T|$nW*7oMZ{vY3kQq|-+@Ec5So?z>$irfF_hLuvr>&B|i
znQ;zwi+z!@FDy$B;*WNGcYD&^ci#I=s~e8v{=R4IB7K45!lHemPuf_#U&lBmx3wy2
zemuN&{TrUqyOSnsBc0>}o(slaP>uh6btZ3o@{0eNL4FSIci-9jYwiEI-HZL0bys#%
zu47M>uQfg1B$){RZLz5uQqzL;b#gGLlXbRiYUJm=`(}=}Ro#2~!ZvS{Jqr^UT-5$juKP+3u
z+_KoM1&7{`7gP-5Zhc7`@X$~Dx+yw!jOEIHN|6DJP4`MWHk@qt4NQju?RSuawJbN~
zah$N0z%9Wsrs9bc`<|`s(o&-)5AgP;F_Q82if)ze-w4p)S)B3ik5~LTzw8Cq2&Lmd;DUiI0@EdYi`Y995z@#A&3ss+u9P5TB~M`kqT6>-nU<`q8bX^-0XY(g;xz{t@NtLy${Y6KMZbX762n)_raNMrNmf)~;BH^q2ElGrPnyjy_E|c$sf!
z^*V~;O}!t@0v}Sany>vSU%bX9;eE%%h|=@jic7<~t!Y0Fn>{hUs-YDgBb3s7eGTL9
zam&X5ZvPJ#pjJMh^UxW-28Z1H`|QoMY@55-a+sR-xOejPRZrD<>-k@akvN3M
zpZ;vA73SkAj?eb)ap2xf%TL&XE6LjN^E>JwjvtSX{e4V}yJ;)ePH6pRa=7aZ1?5=$
zGF#^@>L78mn}1%rOs;iEu`?Px$n@Pw$e|GZ8Md#waUS6Yt@aDfAFZVIM%p*E&s&V
zw8=@&FGfjuS~v4bWf*cAj!7J-ojo)owp*c{-1dGhjBUGH%f9ZXe}1xFo}KmLzmLqT
za$2*C&;0(!(tmHqV*kVViX)N9ACCR_X+x_gb!flIK*xdjoY~2%$zNVO4sJVH9h%LX
zo1NNQrTH$M-qjsGAfZRk`e5g|fmp+sbllJ&e!eZTkpyPxCv<8j=_ee}K5b$zbS=RDu%d%VV(
zJEsIp7c{P7*B-2XcwPLxu)ch2K^L&sdnyn0t@Y)E_G=FpQn?{@7ewf4{P-00BOW^>-b
z$favu;v-23@9+Fo&FxLGT
zyI)kd*KyQ3wY*T1`?@DgP()j|LoM<`f}wp~!o6<{yKeZ(&&4EPmyTxGDtjY_GQDMV
zvnRdkdn4V!R$qiz(!zwmOnH}Uq{-XVR=iPr6Qi~FEn7m{_VbyGv0cY)E&#+Zp
zlEZ^-Y#)XuxI;Oa`qnk9UC7G1%DQO2c0D_j@{P$gO?@3wJ-pZd?qF?loZTe}JboU_
zEt?D29o=DS8SzVGFGb1n%+KiNonPk0{VcU${o+;&-gdmoK-qI_3WGONZ+4ZrrTRT?`HNCs5YQH
zpH+Ujt&5OnUpiLun@i;S~TmxfrO^z6+O{=G_PuEmZ3ez
z$jVAsT4ZvvYyBOMWzW!eU(LbcTm?FWHhNmWsfC2Ei{9GxA?3-oo^k<^*=+uTK&p-O
z*G~N8eGz8(Q0x6W`R&%vnDp%i1@o?P+dDy-jiO|1$1X%O;te6lbwK&DBOl1B
zg7Y(SMBEn$tptGONskdcUDj>es(;sm4RAd)`cFUSPq1F>@Ejh#O4zsG-sHD!wEM|K
z>c}%<2_sKAbX@!nPDxi@=kJ_+cItLgU&>+gRi%i3e6w40OiEV8U435^@M_DYV(IC*
zH%x+SOm*|#h`tD_{0*Z*xkd36O{z=lBLM2mkA*?r)xs8qbqrCAGU$0G<`_OtkX#V`R^~T9HC%n%%@W~A6u4we&
z?F;c5o;F{ljxhP#PgWqN{t%+}rO&4aIV%ww
zsuz2KZenF!X;u_!DQ%y*ne)Sssb?qccCTm8c4YDTy;|K+P)%jkPV>*N@p?eK(DE()
zQ^!wbUbk%*y5jx&DU%E9)f>GFqn#xsXVXgWg$tLY|GhR;K9-x74rz0Zbkl>Ns_fdc
zsnTS$ZXzTx@m>Es6dZ}_lATqQ-W5n-c7dKrl&V9o
z2E7l>j9f=wb0T1!lq>VVI=1_Q<95t~R6GjVY)Gt;woVYa
zO6-CvGbVrUGubr+0#-F|PkV;5;Mcv0FI|TCTt*vTEq78)$TfRg`#g~*`plTdEA`fk
zUXgE?j}F?kUR@EXvplIVsQ9XrdI!Ua&wFv&=-ZD!n3ksNy4et?vPEZZ@ZpnUPXG8D
zEqiAdOKE;@)HFzmr19r8QVx#Eiu0SA|6+SYpp!Cp`1SOe{nd+97H>vl1Vl-v;$ayW
zxf0^Dv$I67)#*dR1*7)%mKHnAzX_QEbCWox3g3EGq;|k%&Ue*P1Eh+|cI6ASayMXiIa$^6I=sjeI5BpmR03rfL7vC;nI4cp)&mIaGoO}1;$W4n&s0Rh2~cp(F3j|C|KvGud!X<%6DFZY+%}A{
zJg?R7@w~})q48#Ns$u7IftV1hBOh;_mSzZudLv#XeIYh%^_1$*?w>z`zN_*G<`}F~
z+xcgCHOc#uy=W9|V+1rQbNp(h
zS}9-lP?{Ti#`e;1Rp!JHy=w#yH;7WKZd%4An@}|iVS_kvDzso<(ylO4xTS7b_O2%G
zLW1(wrYbs{tOHz(!WmO_>T3gnuk9|NbSe9v$vgeR#cn(>O<0}kx4>&nndWc-Zuf~d
zv^ZU&R8x(w!o5gHRFoc?*O2lL)DlJ`bmQFITvuOTAT<2Xo;w#&6pvHrJy#AaO>k_r_G?7vL+7cmN$%=!5NLCvkAqkZk#wNcPt
z$>5j-6#${Shlg)M!{w<{JvAP}8z}k8J{tzckcVsblq@&bSigmOs~c(-i1;_ZQ>2*g893fz;$_Q%J?
z-2_mQTgBip-y61Yw;sL$#J_-eze3v(t~Ft~E6e)wPCt}p-D(uTY#2@JmCkWfgdrzJ
zXsfLrK`pda)zrL!MB*q!%s}5N>Bv8G<_uvAg02@J7I${XNWmP0FcyiQz;=*>+hxy<
zf74Zc1OQ$AC~+GBNzEziUSNOK^BhA{SZ8ekQ7e;g7IVTba4@6Yi_W~A7c`WKbbx}d
z#bZY%5HYm}L->@EX8h2j9mQ1QAUKV^qI=`K1|+s`e1N%O0#GMN5-gyxOgQ@(SvfT`
zFfg#NYoJ-WOJBb!9OAu!K!abuc?$GM@bsOkh)$879xJ5_d?q3?a1sjf#pU~D0AR*H
zA_KPjdQnNgRA6@<0Piv`mAaMdbQr6!?uh9A;tC5v)lz>HqmSR#l~U`xaXe0!vs%GV
zV63VdBg54
z(8udX3!QxF;&%epAWM*aS`VogBlb*T-BfnK@QAc7-8%tIm%PaFgBk4{6pZz65;*N%q1(ZSmVg@t_=sFZ@Qy(?hrdbVG2x%c7G{adC0<-WA6`zu*QdVNKJsqaW#^62Q^87Q*Gm?*gHC
zKrCD?77`+-&jtk#{=&9=7paUGcxA(u-S$|W9cx
ze3`-vKN0^S*3Cw#57WEn`)Cx)WMj6HU|#t!t;tzss?p@)%n^zm22F&z2{9hRrUae<
z2b7k!wml;I=;-K*r;w&vL!_JA7Ln!A^(G|=&rQdSsh^pJcBtWj0}mzktY-&_B&)ej
zvmKM1uE9a6G$|RGx0~x!E`eXhe8oMJ>CKxr5DHNNRfu*9vsI|v(?Cv6DC}KmOw4uA
z#fgC3-o1M-=99DyocJ5cpTS+sIr&w`s2Cd>jt12G_I+|z%7(;b@Y-$Wn_W$jS>~LZ
zI1JK$a#|WAh?Gz`UX7`!sID8vE6h-W;%EfBl$nT5Vse^;th9dY&IZdQ%The=3Ldz{
zS(#HmQ2G0`gB(otszxXc&Tfl-lwF7JUX?t=>mSW$bI@SfNW3!@3I;7l^An$6T$S2)
zx%eKo+zp%q$sn1!d3t_oY;S8zA~|Krd8LK934?-y0>=Dp-qR1A7BNu2SX^A3@JLQ|
zH+Y{y;(?bh3knJ*;AV~5yEc}BBN3ol6tqZte|9NLi1(wMD0iz9
zm}boIVQ&1n!X<|LV3KY%T}~$bue$Cp8z_HoBp|eOy#)LR*|7|_B23mCpaXuf5aBGS
zO~L5{1-xJO#(8WQotoC$QBhYRjlei?-yq&btS=E>rj}aV~t-vr89v^7Hf8oY)59$mHNB14B>n(UNa&
zgODi6m3u{_GA@})z9G6z?qTYONaR6{$vjXp^PM*(HFfNUJ~bL@?~I;_NEh=F))%=Pzte;+bv@C2Lc
zbO)j*`_9>3@yZM4+C+^wc~RNn%7u?>7hE7H0W73%EW-R%t&6y^tHBN+>Nrl~&&N?^
z*dt1xh3GCq%Jm2FzC1+R#H|R(DCR1JRSo?1hT1T8inMq<>FG(|E+=+>aUN7*
zh01dUvg4~DiWALvsJ1R~_7Q@36e@1+&wA@xI)4S)t3&`Af_Nh2I<5Eb=K+dNB<-Vq
zzp3NfU~aHt0`C9({@uKL3l>H{4z8>h{DG|8#_7}eZ?exGKj!cJR^Tzn0U6Vo@1hq9
zSh%@H4+_#?f@ePQ{K41rt@{juR&Cd#Z^~LrDWUE%)t@b*>0&L{owUfOSTH9PUMUNr
zTJbVxKmAf3c^!uL7od0f70Jw7?3Wv#_j+;_pfKT6JKthAi=1VR{U_Dv}_#F5Rd0Ot#2T%^kQgJ9|qgj}OBqEv>`Z2?@k-VLK3AUgKFUT`Wgigi5YEkdR*WVND#*=OC(?p}^RS=nua7>pK|+Esq1|w}Lb~Nc
zSEwt=PC#@G{AA;;Zz#aaE9K0~Wig@3gEj5k>mhb)SH|fL;81P1d=|OdD5|ntg6p^O
zo~5f^nAlrkOKBKQdlph+qyCf0^w}2bvnA2fc@f>~GdPY>%@yI#vE5JkmeL5z+tt0l
zzsI=dKnN7w70pCL)7ee;_r_kshwn4_4C?*0|ZAj+?fy_gZq6tt6QR9V};*RvywRF$jx_1!9*hQrf9PQe(jj>kgqFg-pvkRXW
z7QT9We>^swSHXF`_!EW|-0U8TbTo(QcL?8-<--2B)hMZu^qMKb=R^GOP2sFbZJyMr
zn$PG|zWdfvcSYJK4V;|op8avs&tr!3OXQsZ%`%c2D?#6oI1fQspBI%?w(L=q=9kE}
zg&cQ!<4I(c!UZqm!Rm)T+_ko$j^`$_p0!v|#CRo4DP*7ls&<482C;~M*w|P*Ye)qG
zvnifd4Xd5-&W;YP_NdKlY$1{^ef|C7;xWbzA$ogGWUy{ExW8!yt~&QboUZ1H485WZ
z#xnaL??u@k4*MI$hv+5?wrmd9YWh=hVBfOY*0+ZHIml7fpBoV#E(qJ190XggysVPv
z4ppaqwrqlSL=SogUY}6uu9KHvz465;HMy-2-(3f(Q;5KE))X7pXAjKJ&5@qTc592f
zXA1^#*O{wuG|NU!z}bXdWn)xSRLJ~!oc6?wBk2z)ZY)&@$h~)JVowNHSimOfqC0&5
ze($`@s$4p={1Ot3=1C>b6&h5RcMA;pc?_lvAE!_73LZK{IkP!ftI2tlf5l8dB7521
zEA@|o&Jh;&Iu1N0=xkvL!>AqL8s#0(&cVT>{_Y&407xR#??OXC5AF>)>R~}*KD)vJ
z)!iEbzqv$-#qsKm7R6q9qERQo0Y+WB`^jgt^Pq~eIcnlAWF+%X_{cvz^Os*X)-pQ}
z^}QP;n;S4TzFh7}*y@J0@84@7ZN7K!-i96z+a|~T&}~Y1PrKqr;;V82=gyFtilK=g
zj6c7=OG8A}e(s~T_Lswihx%gk4!;qB&cv?C`FQ7j)c(noxU`<>QtC>3uB#bllK%6Q
zw`M12)s2)Hcr90uNR^~)1_zOhQ^>U1VRM|iOIMe@5nUEG;#VVp^Z}Uet@0Pk%+b(e;f`ji>DO-qR)tM|38!tiJ?Qpu
z`zkgUFAK>X;NV4#mb@$O{5JvQ5S)FZ(!}U
z_bWR2CEu>!f#z8tAk>TlPXjNs=Q;%1#cnDuZTfx=iXn_u0RaIp&Pu=2XT1%cve0bd
z2q&%Mo*qe$U<76&koW{&?wzB}FBCeUHAZ|ERa3Jubdz-!pVT^
zaeN(#Uv=xup3`e{=u>yET_d@ddee*pkMH&-ud$AuKMojZna(e09Azu6@){jg)#tdv
zgrvsnI6pssQelzj2oJsjzt9zS4fHHmg==bRN=+$Bc|R$6zf=xD-O=*YggP=*6QZ7n
zRk*L9v`r-hisV-ZfB~Pa@U30L93UQ*9CO?mN`6jp_h8(&Z$OcN*$*B}I$XcjIp@(@
zC}tTC`ThAJ_U+p@)8Blznu-D%x;f?dW*6VM`2L^{fI{zWwjbWKH*4k8f)YiLC*-zz
z+ak5zpSzOW{zGOhC${ms#<2^_>CxHwtjTW=Y*1>cd?iaxh7kQPe4=lsC!yK@Fm11=
z(1b)W=D9#k%|WIt8EXqaI3!Q^C14it6<;l>`wK}p$JQ82$0H=EYL%B|
z*_QI-;18L)=h{)Q(*5}X!v`^Qw|A{4GJFXqaC>y>V#6RXr<;RJ=T(Z$EIMVIy|@(f
z_TKW>S(0x;
zUWc#JQBJ=Vg)r(Mh4fDgP$nixX>!{z0)3
z2@sK0M(obV9|XUf7M_@TMv=2qoWXZLdY$5y#XgixRq#G%WmDREN(A>KVVYFUid%oT
zSVq9GtT{#zZxQ^Ky}elm$fy84tZqzRbSg9w4egLM
z^nWQ;hpPbXAZhnN1@US=6J{R>ufk~%t>PwvW$q!w65NO_lk^qXufG0fGwWJ1BMK?{
zU+(S4TXMF`hbI&jZTpS`@>%hdCr|o)+~E4OkeJJ#5m;SEp*5Jxb#(~kiLHQy%B|Mj
zz#-(iYM(OFp>9c>YTcL570tqE*Hf|D>^|p*!&k1SnLo_ZYPY87a>-S{Ce?-6b)uVi
z`jju!*0@0Q$N3oqp;6$me1f^*Fpre@yFb5_=|JE1(3
zi_VRrq9Wr!a#GR?9-eDLnpeot2i@MzbXiSB{h8_Mr
zp|63vq)^w59Gl2)ypL1mCHi`4J3BDg>H7ZtzGu8~x~cr?9xK=40HWN3W4r<`Afd6b
zHR>vNGJjr)q%I2-U(560bij@q#Vpx^>Y8uG$rGro3$-74C1of<`^95|YQx#z=N`oT
zP8Vm=y5GB#%X7lwYDNt=0MWmA{a3GRcSrpox`}gOvHo8yfT7s-qwbaT{-(!reF|Er
zb~S$9<*7y^EUc-bs!DgLllQY@34+YcnwZRmOOM##j`%80h{yoeu+UIsWMrIhuaN+v
zZK72kGxO&U3Z_*sb>bo^?jWQqSK>ZPTU7~M`vaVE8Sbd|6rwg130I0(=nbh50e47~
zT>m~?FA||^f8>Z5mv2od8Y?}0@qQfdZ#nk2kal;XVSoH!69i4|cHsYLfE1$d)5Y<`
z37fAp2|uWVogK>i6znU5_(gk4*afTDRd^hqhW$_~xEDw3D%FtX%a=m~TnCDIFDGIZ
zj&XqLX+8WEVxp4yCpw+o=E2PhfN<(KTZqVcAS=6=N?>KuOsd{)e9lRDmV)V-{mKHo@~J~;{igjIBW~ml9h_3U;b=4bR)^1(YX=vi>G^?OGZ{Ibs_t}Gx
zM6DH3ahl?O1?ei3QdvE%Jjvx{^~`OuRF2(>w0tX@lreu@+vuzS@^2t|j^I!UVw&CB
z+Uf+((u)@_KHS&mC4)L>&TkqxT%ZQmxVIptuUxYx0Ag;Mk`YOmdNP8#3hy-5U8`Oa~%!q)i9cBvz0<
zwQ;7P<>oIbR(F~>Awt5%8w3;@%QB6ygZ`^79T4ZBIr;OlM!dP(yXm
z0_i1UM+mn;GU5;yX9Txi6G;eEUq&UT`fpYD5}^g;$&iPmY~8w*cqW5B^Ha!C!nX2F
zor9<9ml_cYALabAgv_o}c>$Y67qX=e@F>aM#U5lI-|X0;uqfc8>bF2YD(Nq*!t=f6
zZsM2IVcQ*}?@*tuD=VC$9G274$~z%^T4&XVaW?-GthWMeU3tM$)g;>6B|fk?nsH#uI?DhVXDdi
z#hK)y@W{Dtx|AQz^P}gQ#c0bUt^}nux6xlMfN)^3hqQhDMhiet#$^{K0u
zgpu<3stw@_`wiBOl(1UaJP+Dq&lJqRa%EE#{?IaBrS0=BzI04{#=)spz~?Hz4@=Y|
zF3ORq#xGQE(cMBBtK9qJ{r1pxiCw7}-$ar}+=%_*^miWfI?PbMQ!5G;Tr^2L`7JZ1
zWp?4vYDbRE*T7Oa#w>R|_t$*2;&_vO+Ug{}nyZ2FR_9jKPMLf@vetXoZJ_O*O{ShQ9w`p|;d34GPk`K3ub@b1{=rN!dWj(e(U!*W`!Awi!MoLmH{yiaCJB
zc*(XQl_&BQd(N40rPucwwked|42<6=7mf_^6U#1JS3xZbir9Y7Ot9wW*zwWy78_<(UyIfAm;1VnW}B?phx#
zi!5b`MmP0sjmc3(RKCL%-+7%m=q|&2|GEeF2
z+pX3w+rh9by7@>7Xcgj?OaVbE_b#TDOH}Gda66k*x){PgJDO@NhPwXoqA?4f6Ph^1
zlUnQ>O=)@Kwjzp_+h2g{1^x2H!)E{BdgP=Fx4x<>{JCq?iSft~uR}wi2Q%J5iN4E&
zVlXhQxYg{+shE+jj_=z>j}BhrDCSMH{6#-{Y=l02?an|^;UwcK_BFSY-A`^6W;w|z
z7wWz)J4t_e;$&nmx47RUfzH5U#CN{rhJoKttILp?cZnAMicaq_CHVc
zc+as;UOSDWk1Gr}COp1usO23}*wg2n~lvE^3r3rZ!fi1HK=
z?Ddja?QW)BWJIO7+>S*zP*~da_m;e7FU^VBT~9eWz2wxbC^wc)H1woyOx(sm#-JXHQz@x2PTXX{h}EvW}OsSfq&X*Rx(;%!}7DW-iGH%Z0gK
zQTEGLjAS;e=5!w$6sLsvF3WP|Qsm8gV?J0^IQ`3$H!Hu!@XwyoL<%4GKWbl8O8)1w
zL{@2R@Q-1O6sL?8K4W~}6El?
ztlyM=#$QaPR>a5EW%865qc)}QjJkmxXWk?WbVhZA{8VJX4Mo~`bd&tP9o$#ryoP8)
zChe_T&u_2cx4Fwo6X^d}9LVbcd-%;qE10{iC`FVx&eT&qKNb?Q_x>m5RK>^^{p36(
z$>=@N-v3x87g
zD&-DU0X1+Uwd~vPA$@XQ+lQ`;Fz%pq98H>pKf4Ogu>Lltl=;}=pR9aGQs>>z(rlxd5V+~DMyJz{g<((fg
zE{I+Fi`_Qeooo4{g@0yi{?7RumphJY*0@uhXY2|L{N&+rn~`*1oXIlK(%NrZG9~6;
zm2V
zLHtoi>5()`MSls(u!drXXI96GL;v&E%a-2y=r3w&{u(;dX=@)=e^*MukKr^E2Z8f^
zAH@o~<-&BaN(;C&Q!VH!W1`6m5o^8H{&`NEz_Mk628@_zmgXZU-k=UmGdzXTz;P>Ky
z!F1E(Y?3QR9f^C1VJ8zjhv}cuLv)x05QWC@Qn0~^=a*OeA2ic$YXj6*1}PgZt;dqR
z>-|snEgB`u&zxGJAvUxuy0Gx;gE5o(n)Z0)BIHMIrQZY5?pCDip^?c&1I+&R;`-D_
zf6-<%3w0|BDk5M8pnUBrXH(aL0R0jq?sAY`tk#iSsxDeP&eONmhg6GC&dn(4fdNl`
zqVAsfyr8%+$M^KtWV3Y>cUfHEr*m7p9#Lh|k#*{5hVf@0q$f5r;lS3j9j{nkrbD?n
z*7;|Kp{p{yb=2deL*vq(4$=$FzRr77FX{&135A(|Xj=Y^dS&a}*2rg7$H#=Bq7WD!
z9;W{!6`_minGp1O(Z5|%Tkd*Evz<3!#|4b|gpu}~KWwH1^n&hZIL~+Q*u$P(EF+YimE)JnWV7WMZ=C(A^SaO81(vG{4L7iqg$Ou
ze%#1m`WQ9$cTLN!+I!{A3jjPL;ha&W?JI4lJb9N8+zuvX<
zFjhvb^xP>nr#z;8`NaHkrgKVdnI$xCbQ&Gy?w0Jk=12eJ=ZwfM`svvxI@0v
zG{mh+t6DmYGB;uDYSf8q<>C^SQ`kaDq1>nHPbxmx+V!BmCgxP+KiBpIE5%~V@1vaq
z!{Tcftgh|colv1m3pGDLZFmTrr0HGnxbDc#bW`GAiVpGi^XFC23=q1t3o}wUpsM3h
z2w@(CahsX%LsSWhp2K`ZtOK|TQicGoMD{MK<11uuVxXFn=kp@O9ZLhkdBWgiqrV9c
z^~;fwoJ0{A@B&f^ijH-3mih_krW%_}PfuUDQIq^7H_cS?p5jSr_#Tzv%7Ct709K0>
zndl-x*Sy?8o&MC-HI}pz{yLP;4UhOb3suQ>umAhBA5*fY`vb$6o{b{toS9+}k7Y?a
zemmi@P*;N47M`Vjp@1)@?E!GYCa+m&2@^ZMFTS%`pM8#^Z&r?>_a=DHs=&)AbXV|f
zq7Ehzu%x)qTOGM>q7bz|j2Jdsz(dQ%ccxcY>>}DwV;G$i|6n*yZfcQ%#41KM
zWr!eG(akfPJ=HTZ64KbXbCtj&d>HV!oOy!{sn-)a@dXjDOW%I)C5@P
zL!09L@~jC;Z>~MOw#@$*m6NFWDG#^TRrX)GxtD^8Fw0~2mb3Q?ym`;Pk*nzy@UNr1
zxZL1qVEJlifaH4`@MM6-aVDZKrkwCCZOkXQU|iqWbi3=myK_|jw{Nqf!$7PbiNbq=
zdFBnT^8~yATfu7mh49^gFJMx8u=(ZQ5|G0aUxneH9PI4ZZ>B+%gggdg4Pe
zuVf?{aq-Pv#faUzc4tdtm}ux(S4^_!Vo9MpD%oM4W&Sn&
zLEH{wL-lG0otsZSJHB|bq;0~)^CSO?37
zOEH6huO%PaH4oC^wXAvI03W(q+CwHLCInNx-Ic@B(-T{e2S7jhv-bd{A_vgQ#ymht
z@yTRFQKX-3Q|t!gEBzU$!n{3}U3~vpR!B&Q;HGB~u|w}T`PG?2X#I@hx6C~chKSrh
ztOw|dlX0Ge-nx|tvxK!$!j#_HmCdcyobY{c2>oX}92mJ_+g&y9-Dajvb(cnC%*`n2
zv`oU|8Gp$a=~eOU5b-|e!!bE`}XZae>?$?%i^iiEx{RhC0ztCk|$!0
zN|5K^Pb|6&*%=xNQCdQ>MeBjc`k{S`u<#B;emTco&)*49+|HbN3*Y7SX~a_t+jS~`
zvCeN{-7Y7faC>msy`u8|4Ji>Pj3Ko7=+$SPyq9R_;(jWSNPR5~
z;`0h!yBlhs0ePYA#Tb*T*u~8)CxLJ6+N@`>C#WG)Taz~L35i)4X02*`^ZIpA8!^a*
z1jxlgpmLG{UD8?GcZp4{3|hjp9yp$|W;g1Llw2<>D`Rfgk5{v9EZ^~L=iStbYwt}=
zwz9zhBaUFL$W>B1v_0Kvvh`&+^d`R=*hoy5Ug;(u-L0Sz^WI~bE3cVPGT=ilz`vzzu
zNBH8O!Hs~^HnyJn0rWGLMrT?+L>!2LNW$Ktzz}k6pT13Z+O6yUxqU{v^$!C=v-g1^_|g{KXOL~4cW72&xW^u$Drn-JEWDmF)vL&b#8oj
zx8r3I5fPX+^*D&U=C8QSv`W%rISxR;xIcw&-n=`n1YAra7X%+Pup=vq?$fAHJKUSdFN=pN=nW^^?Y=K8I1t}*x;w~
z+EeGQtNd#EZW1E74|S;pY0C`6^(hh3hl>3hdxt`n&Lj(<`gQ`W4F4E%m^#3tg>1Wo
z8saNmSy`zovJaAjZr|T?1M4b;*?ErXp|otG3yJ9_{8tkbN^){m0BiUXd)MyBkw9eW
z>yTMt+Mx;HnZ2v4ZFdRgP}ebpVjKXVQR?SRO2~fcxxGMJS{$j%Dk}0$Z5ynma@5Zi
zo{)c`%}e09^8S>aQC?5pMWGK2SJoL3`>sKr#JZp}-xa0!0gLdvgz#O7cyqrZ_
zeMdM~gGgjG>7m=uR6-KFh$_?VO{1aM6duH;KscYcT6JMpf7HLMFpBqguD
zh1|HYEYA1OQ3Qww#vwz*vlT{&mvP>iDb^Xmu#oiMIc?5TVMtScB;JQULRZB3h|{B(
z0icNw9H~pra+zAZ^2XRf`+G%wGe_}Uv>f0)1-vlBXdv7N0
zh@ggy#fXWq3UEiD5t8g1k(z=ECT}!;1gZ1c-iUp~@cpb;^ZWHfaDyp3Ri_$A2FgH!
z!+?(1Y!I(>oFqTi8vo49F4+Oz7?7b-Eg_E(kPrTB$vrRBe^tpr`+kClroq1puPU3x
zgIz;VDKCduLlQT))W00PWU94W^CsbRhV>mSm{{O4+Pua`YD?wAUat--X#5>4P}NE
zdUB{uVLN&dLzDSUE}T3(`_mne9GSIz#|TshnYKbT0>bgh1rc^8j*XzQOG!wqxp}5q=HmcwKjxhz1?K4`(MZjb9)|yvoXtV+OYE`UwWgG!toYE
zdCyyLA52hza{qQE@Nlpmvl>rwNbkQ4(~z34lT%Xx@H2Z;tb_GNhmIdBREWq=HcMVp
zU`}7i8TPbb@s*Bum=a#U)`AwPP7c?E)G=%i8v+((SBbO{s4#>4xWReke=s7s
zz=6ON@hV6b%PA5ki=WbX>ydbF
z)%Beyy>cOax;?h!t|j%`YkC)+{5yvrOCXKmwK(j$V-g@}t$B!R30fdf|a_7^%>895P
z0Ht+m$pwE#aMPyxv{UE75ux?KdnBE~_?)a&IuFo-HkA{_d$VmizLLdmG9Bq3l
z0Xrw9>*qUV%^lgZHJ3oK*t$I)iJ@oM1byBK)KZ!9VqS6GsIut=K)381Szvi6_D=>Pq-HuX*bLk_%9Y95C^3_0!k9U5H>O4
zpOQWX&|DM^00C)Ta0J2zOG3F8D|0!EXNwsw!
zDz5wh^W}a%W%<-;rPr@v>9qDD&yD!Q4OB-x5WT?U-Xxfv5r
zdt}&dr%pX-`NZX*?NMXnT>XDm@mk5@nRU9o9WGu+2lYELml#=&I7=yjsYxv5Fq{kq8GsOkZNg$!WJaLNyLC?
zI9r0eeFw650QU*Z68IS^3Q@@-S;d`=wvSrX(7qxi6&fATdh$o!QP$bH7W4U_n>RVu
z3su2ZM62=G?g{Cawac-+UzJ^;Q3zq6Bqb#guLMeJdBr&|uO37)uT=&bKir%O->uwJ
zZt-`wth~k45Ig%|L-#rfSD-P3GYNSaF5>2Q=Omt>FsQ$$tZ{UFD;PKg^Csd)ZXGaA
z320<-!F1fIJ!SF8NXge^nQ`@-rJf&G2_{~q-I3N{Do-VGQ78KJO4{wJAD*!A=YLgP
zyvAU}w?oWQ+{zG+z?v7oIp`mf}^S2yZM-GSmymI;Y?N4D9zg8Ze
zwmN)d%eK~$BS*e}+g)C$dV8`;D!V_w{e)DGE5y~|1@{n)tp0Y_QjkV7`RM(KnZf*2
zwx9I)b0CP1VcNd@!PT&omKF=lgRAW|%0GSDM^*{(h83|9Yg|JiKW>|E-uH1}R-|Y&
zeH|Rk#1|Atr2#>{da?NPN&Wp!^Kfc)Zhf5kv5#rlvL+xmChd-qRz-=NYdIcwE+%q)`9LF>
ziJyL2BBL-xzq3?m1me#PsFU{(OOrYe;^GfA&P6SMIG$8v4K(EOUG3Sa)CHNiK75-)
zZBLGbOb>TTA&m|Q49vvPo(IT7WU*bN1z!Idu#K4HD*%a!Jmw>kvxnf7DX#f_a=nt0<(@km>WN)4vaL67-&Wxf
zK&;Ifq}dY%-SmV!gMv77+{M5;QE&0$v)Q$K*DfxPpe7H-?A{lvuRou(?a{4iZvvYqU0;TE@iA0L
zlV~dPL>4Z+DP_1O{pgqDo9Ot+N01+X?JtRa_#D_)&GqPoxpFoxD}b6fT=Al`U^(J+O6#*dyi*8_$>f%Mwxw>a+w
zH`s3@BO|8P9r!QuWfh9gjvYinzn9~>Hp;bBLZJj*ZSX6ieY0H|&M&lAUD_P8
zK0f>GwFh&G%<@43lFfi8DFGd!$93nNWp$y=o}!xS$DMnM_^JN+dcCi{RXLcQVJ;vy
z1v@5M<(zxs%lPxCs6SeJcBYP$%szTwJ%wZ@4O!YBY}V_n)Vqucz3BXeK2n~eSf(N+
zhjAhOyMz{51VikpljvU!@8x;1;qT#~K4_yW&2zBTZ8}TT)r+_|H{o-M-
zb@p(_ut?Ug6N&kM0EL%Y7yqH;EuneE}$9k2r^Nrx4%Z0JHE0nDL?8xzF6ZSwZ61
zxyHxSBuc
zMTOtFbGUOOeA>y)QJej_xqJq@B@0LH?gpG*Qyr)3xcU7+Flxj@J=Qk8IDyj6vRO6o;Fy`1jNr=p(Sx8y=STb&r9y49ZB&~c
zSz7`Ax^c1vIh!g3>zk(z?SJZ&cqHze4q~ziPW#i{ujtmo9FH`#5nCJ>e-msdn$aIf
zD7$}1I@3$~#+HrQXG){DF#L1VOyVmi}H~S~&zPs4mK6sq%T1ZIg;Q*V$
z)9J&di@utlfp9c)lI_02d%{hz#tK)Cgf1N@PGyx7z=V4BCkBRxg*nu)Ffx7sFROQG
zXlV5Ab8q-yI!zP~d&w1R|)oa!q;-WH!LAQRavKg*eazTaGzhC}AfFuwV(
zHI>_Z{+v#aKzD90I5C^Pfu5?1`NqkC-=sZ{JnPB}^g~~!%5@;pc@dW0*RNmyuv8vz
z2FBRfaa7C}+2Cn}Bi?bf#ko_FIw49{=ALQ$JY~K#JyrT=nUu|&y`vbS`@-)xck;on
zWo&F)zlp40AHLrbF6LH)Jj%wyPo0n@och7XraBoaRE(5eL`-bkFKXGCDe(2e3+52L
zKYDz!?wecy4^u8>vz*-NuCfcp8;_m)^uOD&YXJek3BLnR{gl@tHU@74QB}x$jMC$6
zD7+=I_8baJ!4!$B0SLG&3bxnSxw+GMU0>wB?hU~`J-P@EDqLB)$H6e(KX6ET#LJ^^
z+}b6E=AUoVVH*KG%*|6VSvW9x8!$OYHNFY`0Z<}qLOnm~Z+(6~4rt4OD_3~N5nzTJ
zb8&_jX7t&qFSE?usQAPZTwu9xP3n{EU;Foe-kb8P&vH%tY4}|4nceoJ
zM1MERS8)Y}rqE{&70Z#-)c2mU+`ysP*6MK$mdn0ob&2aMK4tE!3snPDG~K3p?Lbp(
zwP)UcAGwrhOnh_9TQfPzx52?dj$RN|BtxIU)gsFxwD5~
zZR$W?(z9*NtO9c2l6vItW+R7)48anAD&$fsYIrF7fByPTN*W(YxiDzE^5WWRMsz{zQK48X
zlOqFVf$TT9W0rm3}A%&?4Rf886CW4%iwPRTC
z{n+iwP>#X(Oh=LvaQX6quyOR++JBS;464UbbLbnWj9nITMKa@s5lh6qdk4PdZ8_1A
z1|5VCp`08X_q`EIqt*Hd_P-RH{MIa?_3g)$rdt
zqD@H)KZgVy=iHSS2va}c=hTe(J=D4a$v=M{53GPj&m|sJuhUU?M)o&nX1dtiYG3=$s_$Ch
zqB5n|Wp9BP_D7_HYmkVKj-fpTsdJy@mnfJHkg(07muub(194I%Ev?;nOp=hdaq!HY
zgFpJ@aM6fOn4AsH_@lC!!ukKR1!*A5o{TwODx{JlZ{l(vp;?mV2Wxg_R@MzAWA6{6
zO=(T)i*+F?#xivZf`S@2Ihru}lCZV?BK!-geIGWK&Ly#_&lw(`$n*YTeI1_@v#;^Y
zzllV_RuiFfqP_Hi?{BLd-%|IYW{+|3(00^SjrU`NgIP&l638ZCuB&^^xiKvf4B!l`
z4Y?>iZT^^h_a0&b-j7V+Bc3D4vEive)(^?iabMFx`;w5CpNv+GC#E)F4mUpy3=CW<
z5O7xdV5zhfdq3M7tDK6|Si$x40a)glVw}{7$?3s(PfMvmuj1$}o}bQaWSPG37_^;p
z-rf`UsVHf;{WhW>&NXgaQL6cdJPwUDeZQI&XCKzyHlE0r-wm6N@%)D`q{Uprv?z=+
z;xZIk#cDMQ%w3d_kHuW0qL?Z{AtuA*!A+qM$_nxh3G&n$dQ=Pv9nkx}s!O4KU|1@a
z46iX#sKh-x=BWSkO5xHgumdrrrueHZ{bPS)e2m6(@aZWLcq=yArFUGtl0t1bTrflT
zpI1&Vy@GsarwE18PqUQHi)+*RQ^Jez)x}|VeT_mfmB0kO4FlXYe}BJf>GzF|rY;=?
zx0Zg7`L?OEv*f?Ok29sZM*s-1s@`|5X+&9|=tkosjgg`eF)&N6gd#2#Ou4uRi_o4%
zw;C*%G+dyMz|yZ$FXQ_^BjOpL-G!33XTlJp5v`JRM)DUlqJ${4WiIED3@&|NHu9ps
zu?B_Fza7Ks`x`jOb*43wipiJ&p
z_R<00C5gM!wcLGt@d=Jx=>r9!Ja|sJkb!7K=YVhe0{LW0xTrw}#h=$gI{ldML5$={
z)vKbSlEzKZNWB#sn}zXX%MsT*jaS>K{4-u2%DoE}2bX6cqu%#%|l
z`nKxU`<~ov`{bIo-*>^(Tzh!pl)Gbn#FUGw-r*=c(FfC0^4HEyPT*|*;<(Sy
zHqImZ**;FeZA~m2r+KzH82qT2+LJfgvfaA&efat)ebE}O-~LzfOQLirlyB3^OsOba
z-Lx89^SQ}!jk%iw4JBgJ^6nq@`)%jpx)2YIRiP^&sE~?*t
zI{1@Yb#_oCZ{e5KpRqC-Np|{)=@H|`Pm2u(48JUp-iABmiZ_Waqfn>>DZ^#LKfc_)
zJ+T=r-~^12O*T{fbqQ;}@a$NUi}|$_*YQ6Z6b+HdnVUBXUEc28ztX4Rb)RBnbkb~}
zPco
@DVi^%7fVZu8IgZaehdahuQk-uJ8`M^db|qV@E0wHu}5g0}I+lsXaB$ap2!
z&niCXV(8H^P|`lA{pnwaA$OhHtkv$m8E-y{WOpF%b?xlzdHw8ctAWw65B>uKtxp49
zWiq7OOz~9J=Jua&J2a&3Y0hh)J^1=f3U$Stce~fj*|P&R!!&7KlEZ`6R^icy_CCsav3pC#wZngN-j~f2q?oQe-$mQS5r1nuf9AH=?n1T2
z3;(3(&YvEw&7-BHxuK@IkJs9c2!uY}75tXLziQ&bTlYg6ePZof6i_he|doo`dz)6WpIFLTe?Rm+@wY1Y^TxTqaEZ>`dl|K#(CLJ%UA
zB0`J*w#xbl!4HnRZFgnJ>_&Ltk#h`c&bH0Q9M;5y;?MjQwm;-Gm}?$?5++hCA|W;6
zI>zVBOp%mnFALG;7z$ENKUwW7lk??u`$%KU&CYN3TdEzatTW%Ne+S{b>CI7k2=IDR
zIyakt_p(gZ-%+?y5kEV%)5GO-tkuoWITjmq>uRff_~qU?-hT0(pDu1)aI@-U@MGzH
z3w)!$#wFbsFE_5%^m41-Vk%5YFg!oF0@ut&>D{HA+QI8!4$s|vz!-s>s-;gwk-Q8T
z!4!;&@bWW1r;L=%h2w7+s5K~tR3yh96;pR@{D0Vb6L>7w?`!;4q-1K)phSg2RE7-6
zkV?pqNaj+Kc_Q^Dv}+X+hXTaeTBOt@^8or
zEq(hLZoj9`RVPv4RUDtVkUmey_R2S}8}w}#3JW#9w3g+ED4*;|tonL*sgL>hfIQbK
zg6`zXEq!u$ZKir1Q})rpmMkS;>10gJq9JU|L3?8rvx&sW731ZaoeJhx=$C9v=n39!
z^O2A6`75r}-6dQ%osOgx~r}F%fFFyJ=m=a$MOh3@pDLO65;$Ws&3{ntG@^Mch{7$xDWv!|Y+M3OpfwjW;qv?dg51
zrC;lI41G_%yKYo4?2I#4V$K@j;+N%ugAb3zew3d!Z)z5ja1cMKU>Y146d2pzE|u8R
zn|9v*c*9FKxk<}L9jf+r6H<9CU5eUaYbZx{x*Xt<&6r+%`6bSG;KRLf#i~m6
zNZ&M8nnv~Bkx#5QZ~bF7Pot;VI^^X`QKfYln9i;DR?3SRH%&+!9*Q1el{M2ksCQSV
z{X!(8QhkEb)~7*%rposBhmgwUKE5&fp((9hf%
zM&|@{#x0t*Ri-qZDAkesA#$gC=pFxYBL1PH18pax#lDIo%@%$BpA_eEVz$h#tT3R?
zjo|W-DYO|L<}evEpo)q@NLQc>lX
zpELMpCT@zFntX3BAo4NPL73-d&0@r|Ww@K<>(=*+AO4Q9Y44*
z7oE)_7-}{UvhB;!*GdP|^y0&nb$(B#F5WnIy3c)`ZMUyb){ln3V)mdV)2I3Wh`xEH
zswbJYL)s~>rtALK*-F2q%#K4ejxGbL(q9iuJYzI0vOaC?IwSsl=liV(s3HY1KeL4G
zo2q*NJ0Yl)&)LV*BM@rIw(iEljII4}n>GnR!7WFQVHHvN1!Slg#9-dNdrOyIvQ;kb
zDeWnA`{F-+u99Kl*~hgUI1{(9HJhAfvT8m>DZHgPw2;J}pIxCy$;zw?U
zTXx_uEHM(`?wh$c?pc`nDD`@hpwv{=v;d8JQCnUD09&LsB55qi=d%V9Q{6@G&ksKK
zrCcs{{bMIypPnv!V@k`jQ#Bw`r@u8h%QNe-TJRd>?x7!NJ9D+?-I|syTh$h5Rees9
zPqwtvSAm<7bY7p|yh)SaaCw!liLTwmPj8ygT)UT>*llLrXo-?zhgOlNli}L=%8que
zBCZ$9%6-BL?!gn!W9BWG!eK~B_V3`qdJSp9n9T^1I1lxK%=8QqpkHLK2s{;=%qjS$7uS{I~9e8JHNcFb?n=(Eh2P@#*POBdi<~cnb)^Y)lK}Tp{j8q
zwv5rNKB23BTVr`&dQqX>%Oep|yA{j~59*!!Q>Z&MASyg~eb~;MJ4GjEd7-G;Z~3dI
zF)JnY`jc}5w`udu;
zIkvH>OpKI8(ag8pLrr3@gBTyVRfO)6DBH#{?=*ASOtT602e*d=
z6n_KbZvX=$72Wx>Nw~K;Af_lkJ0uIC|BZBDotJ#(^WC%_2Lx4SgZL}taP~yL
zZ;fC!Iqed1tRY`TANN;@Z_8=_g|~&m2hoT`tQqeaxV0@cV<+?%$DxrXWn1A
zb)%ZTYKL9eoT0
zO;i2-w`S4maidg{T*qS@wrzlM;0~DE`|(g*kBV6w)Je5%g>G|
zmi#g-#3!qG)f2ZVv-@`3gas^Akt=6T{FarQyS<|owf!54SbWf_tO1L1pS-8tc1}31
z8nqmbbsn0zY_8Z;q&{K*rHjgoVuKqfqfCCB
z1Ld$ryz*`M+Km@QI~F28AKB|85EIwem14gBZIA-D+hKyRx{V^v!B2WFEnvPIK>i;ZDe+SIu?D&4=65M69*X&6WgA`JlqV(D
z@J*qGTJQO!bB^2C_!#faVpgf5{wt#S`|~fjB~NypZu->J
z_vV2L5bmkr?vG4_jSEWe61MIRBICsx0r$m-ojoc~E-y3mE!;9u&=D|w7$Y^c@A2cu
zjU;E6wBdr|n;K9SvMFod-=F~0ra_V@LFku5ZQKAOQ_z0*Shg4D<;rbQnmet2N7?Y|
z(YdlCC71ne7ukLEixEa61@fJU_|P2RT}64(_4CD)E&=_og&R+s{<+c_FWfMh_KNCD
znuGJQGsRpitE2k<6z|7h;$&!bYtsLt-W>kN@&2XCyt`hb5&^DfCle{1!35EnO%}ei
z*`lAvtRE_T67jIGe%f43Bi}XeH`RQv8JF>UQgI
zf7gGlC1;^*Y(8^giT`L?gb}>zH9VPwZq}=q^yD__x|Gc^uynpR2~!bBZod2nwV;+}
zqreYky-=xYH_Nd17LqKMrwcY-scgc!4W%SU@9nMD9={oJOb%&dXXC#gt9r0_-HT$G
zn`fgv#txXYhbE@x`5heh1&DyHG?<;-QVOK+6IxK1iL%hl0V
zvN1I1s_S|Mict{)Z%(%en0ndCFY0CYC0V$)s7wd3FI4!C`p|LHE^a+nLM6k+yi@Iq
zUy)x>e`8Mp>-J`CiKmV#@?RtBYsc3~*h$IKZtBW9HWkw~>87_p{cA8(GF8I%N8#?b
zgWqJ?R319aN*>-CK;Ji0RRNHQXk(>9q5OU-tYM0M`=C70tU_ZF7A&n(ZV}8xlu)M5
zW-qF`p~4xtJf9KI(>dmTwRJ*!;vAv&P9%B5mD3TbN|f2mD&sAmZ8RCIBQ~;!J%5t=
zQGUw&<7jD%qo5A$#-D+fJOw<%3zAC9M*@LK(-hc;UL9^f&)chV=I!T$1W}T*f}>az
zQ&}!AFNkcikYB};waZUiU9KmU=0+S{$SPp!`<`)TRkbeh_KEhSE^gv4J69x5M>Ib+
z`@*#JZQ6tkcAq+(X~wc6#1nQY>pX1{Vchp=z>7wD+t^aw(#`Mn{jZFCC&z3S8&!{+
zc^l}ChZW6T&ZtXIe=hN~o?EW>EL4_Q%2&L_PG>xXb{C&;Z|9f{zmUd94IvyMm
zFL#*O>Uv$lSE)bYPW{L25DX>2$kyz8eijKI0&L`!l+;7_G{>r4;J{Nl!p0r7FG)7A
z_wn+o1uYt_C@MHese&{nU(NzLL7Tf)dv)Zqwq2R1wLAMZA0mUvwBYuW>(f
zZ^9m*j8{-NS*HD0{XRtzB2(bkvm?(Xtjfb|beof-KLqF;}H}-bJR#xd;HhnvfRO&ecEPC-yG*o+%DX)^Pwc~U{?^AHTA;^jfp79
zK5cF1&ax1iJJQI8g)=G2`EM_sv!^qUc_lKS@oZ{ze9S-R{(Zspj1(s!+DB@8?}!o}
zydLC1bM^xB{)mcq%l!VQ_B-f=_dix?LC^frw!_E44Ab!0!XIx;$h^d-Vqsy4g=3U*
zxTH1-DnbDm9U^4HDO;+LA
z<)&!^+y%{VQwc*`BZVrqrt|@rSS5o?e|Lt3uvtAdu-r}SZXr7VF}sxWOMtL8;~i;m
zGD45JWP?THK0(2{S}V%r_2X^dMeBJ69g|GF0bugG>6)y=sR;f4!eg}MWln+lX>yoD$9-{eRPeU
zG=B!`p_6y{;*l^BPzhVx6olHwg0fG)432y-_>3w41N1r+j%o8W~()}*44MA-;(eY5sbm*+s*C_
zuB)Gzx$|do=3rjyruA-WufAm35kziJuQrLhzF}I`yd>@ZB|?jYqQEM8UgRVbOJ^=P|aTei7qz&`mZHCfAqC(3;_p|L0J#H4`V^q{|!gtOGk
z8K&d+9-atsznIhbX{S)P$hySXR@Ngt0lj`U;vfC2ST$N>yR=TR->k~+tIf^ZeavO~
zkGGP7z|9j+Q$h-IOn;?X=1DtP*QaKfbocytw;l-X%26fycuQIK>9HW;b9MXYSFd5(
zl?-!v5~>~RWU?V-wvVIN|LpMPO!s9shFy~1Y6YUl<_`SXGbaM{^14gutNfhFFP?=N
z!2y){;^nx@3R-t$X3zQd(B7ic4ekY^9IiJbcYC4Y>@7Xkz%Fm%ptanbo|$)jnbomP
z(SN3gZ&YRHWyxMH6}AZ$5}FMDc>Q_!DuKh3>Fqi>pZy$yed
z#kRlCM8TnX`1jCI3QWOGp_KqtDHhG?CJl7&LFZ1wmn&LZTbF(ma4tS=x*y{GyO$sW
z%QM@j;}z8S<=QKhwl?%6na=f3e`&m^y2LGR5Nu*Ramjvhu4~n~#P8|{3uXKgghf(J
zcaDyK4+qICXMS)yL&zk%v94WPwq>}fd%^Cvog)ThYF^3n>T^O!2&?$AZmGl>vaFI!
zKX>oei>tp5*|QNEEM$<|6!4{bVVPD@XHVaeRyMeY#^A}Xpy8}Uq~4?LLZ{%9qXo{u
zAbY-YosoV>cEgpXCV+n-9EQ?gSjS8#W0RuY84|
zP*S_XB_eKW_tvYgKW4(fZn$`;yV`8qg^V**OOD+s&hGWRrRC8%q6eDXr!BOK2XvB5
z)hwN!eNjB~Ho2oq#eC9hZ~oIKR^O)^U!Q8R>CY8;yHD!8s=)Ku-6>Y;{L=2^vtE$0
z2|Y8HxqRsR=w$H??%602cl?<5+?=5qtA;LF(c8#~=%a9coO{k&Me%@5xYeVydc&hP
zQ`FKF*j2lRisjvwB2H!Ioy>Aw415}%_{7rc?1f1gHz%sdjVq{(7~PU1ufuN3(((xK
zPxYl2CjvmKju#<3bQ943gXLEW4rntVnAuGBPbTtW_6^QStQ=EbnF+e8JJ{V+R1Yl3
z@7qdp5al59m@uuF@p5`-cX@Qr8j`RF7@|S8ZdRczC
z3^3Hej+=3;ZWByg&D0r1OP77fe>}7EwL9qN&gEq*IX>dxXFsfSt9U+B#-{h0`kluu
z$&~KX)!h6OrN@{mO7-4wH{^-O6g_U1?6|Dp6`KA_%RJvYqY>*m(to9Hd_U#+OuKHt
zf6cdH8#Hmvp#!1@a%;s0pGC6Y3Y%oBWi@w&fT6jgCGP3A#)naU{$E}f11+PXFJF1FyX6-wrZyHq#3T&gKjsRZebHTRsT+*
zF2FxXY3%&4Mx2{}W6Q9g#1b1XGr{{Z#l$ER`EAN0@mJqXs`u}nOc%VK&ZWQ|vblR8
z>8i*;JQ@}!MJy5xlyxiX2hUpdynPz}3LrqO>iwx}ts&c1I#o^~$2Onet9Fc4wCOv)
zl?HA<{lG@!-tONKVdwigS{t?FbdrX8(`tc72Io)QNID|^d|R6M{BliH>3pX2kK{}J
z1rlyl{mhMOZ$q;iR*Pn8+Ov1ijq7CS8}b%g-ux6^_cW{6tY@^eFmaMYl-h7`(z7*;L}G5gR1w|SVS!r>ER
zq4YTY`hm(f7oJ*HoPGRpj}QLbUCtR`HaC~_Doy$wI}yLiF^;xV;Eey$frJ#xU!ai0
zq5E<6C>Bcs^fJx>S_d<%{cjky!=uH6AY}Sn#A!_Ot#3`$`fNM2z2T#9WOMLJPaqq`u%t{xruWTc`bop{mX$X9Q<{~(
zy1x~AoqxbSS584@7KkP%-=011rRG)94|0R`Z
zd6CWdS|roRN}iF;cr&nR&x4@%Uo)Nzy_PVPJ$_ZlS_!{;i8i-68S_}wHjuy9BYkl}
z0a@-=I`uyY(BO5~oEzhLZ1{@$7#YeQZt
zLIxN!UNSsnk%gL>VTQ07Zo=u!
zCUpelYZ0rqT#eKKblULjT7Fz6*|@<1`i`T7=4mjcK}K@dt_Ov?(dS^%1@0!2L%ey%
z4s+!W+cLI%jO#L`{`)IsXpWQQkMXIgRJ9&`(21aU=pA_jckMo$5(0q(pI;o+LgPBQ
zv<%-t7cSn2q`bh4Ql}og82${R2C^0s(J^-hWtQ6GUt$OgJqb?mKw)$~;l;rGpW?XV
zE6)J;Y%M^Y=_dA_tSlJ#)kej~39n!7eryAaX;YNzCMI@uCr!ufHGc7F*)JLmt
zy=JTbdplmyRWgK4AP5+w!a%N_@~|@mtUC~@QunhcK(N7IC3zyyv&pri
z6$bM0xv24GT3=5lc7skm?f@_Em8C}zYWuHjlG*>04v
zIqc`;>`olm1D7I_a}S|kGQc&wKM&4!q$}reQz`lO=J+915Sc%Qlk`C}YF@v(d-rZn
zPmeJK>9=&SVCw;zKK1j9EZgOB1|%q#H}-GfQ^9}XwG1SDXDuv}0f@2e+?foDIjNR_
z;SI>+HhG0XshYYV;a&r?R>F1WHL2o(3w~u)l@hQ_u&Ez4U~CbnVrRp{!>yc_TtSwF
z@F9>SMnlMhpTq|^0Rc(_e@V*
z7?fw6#9$*(8M#=MNxu6nQ2pm%`w{#5_wPN^lVf9b_|=miHmUxG4ap6sf6fB^Xm?+}
z^>%4)0DzDB?J{glvqg7OqzTR6F|aHJ!`8DuXeoAg2hQ6Z+Ma70Epc;jzt6`gl9jVC
zGpeYfk}cJaRe0Qb;#YU~N~Z*txhy+&=>8Td1{)8OuzqanM)HfhcDX4V$TcOV;F(QA
zOFe+C+Xxeh_1~>AV~dWFkz`#*K@L^dC!?$#A>9Grf=P%(%+P*>8j$x!F=%>y;IW6l$5W@d>?5SH|V6KKvvzi(J-#CHBPu=
z>&i2TAAAOEFI648hO@6|epslxIz*`KkblYcsrq6pm^1!*Qq_3qq{0DQ%z=46+`vc(yba
zT)fR{Yzr6U@?>ipkK2G`rGv^zsw*I8)d5|Tcf6@O%Pfg3mtc?^dg#*eC-1hmI?q=b
zE=3(3&Mi$?teRVxCF@(LEq=pC>wsV`hXVFbEgDF+Hj5NMm%M9R*MM=a|N4jesv
z_$;Y1!E%a%ufCwuv0(Z6r`@+!f~rR%iEQb`E)Z&1Xo!MWzd*|E(D&*WRuevpgn2&p
zU_U&LW7;DB$shwb&T|&$aS`lANKEfN-WhWEvL|^{#OzaW&-$R!tA|o9;X
zaPop%f5RaNK{5DJgTh&Ve}7?%Z-+jXl_g=^&KzXCFNzuw=_f!IE0|~(LU8_q_loV!
zG_KlW-&glB!6SvQGWwR{SkcNNlhKsL;gYKEX7lNaptk&AcLsayxu?L$cD!=lap=9n%mZ0Lwf{^lsw7UVC+!8R~2Tv&zQ%*3;Km{Er;X!G1gVFm)
zE;onu?EsXxwJ;~6pGB%r2Oe_*;&W_lY#%-ac#%G&hG}5DVj42L-x<|X*ZcDpF}7|F
z?kDFsiorSce(*qz9QKu2MYhpF?AbO3r$vlO5VmYiKZ%BO4ZRnhJnkF$l}6ywNAm!
zl&hbg{jl*aC#}N2JrSBksHT4xfy>272>6m;T~F^h^o>dII6scRbMZFEPsHU3EV$s7
zfxN<&GRic~$|FtO`2XSn8;b<4^{Ct@Ciczmrg8pZhY>K9&0yx-nAvQR{n!AO!vU9e
z&tO;Z968dMFnNUBSGX0r^@!2+FmmBkz#{dOc?W~!o$ZCN3@aw*FqO}D2dGXU&~J7c
z!mdaFROq>ZDy?WP5t28J--n@BqhJd&I6-DqM@oc#A0J2Js*bhHAR7%HG!AN(T-op7
zyOzeUhs&5b#{tZnAVpn}0DvFQf4^z2OnT|1mVr~O4BJ7y+}Gt{??HqQgLBH&+;xUf
zF(Gaw*O&)>Kw!e6i^pzlydfjwK@!qo|CLu`4kes9+^al0k0Wa!$K6?5|1o!VQvfxC
zETrl8Dgo-4z`VW9Yx7&2;m|PQ>%LIx*8~md5JE&vM+KA(;?EWNA)_UAT*{)~H9FtG
z3dIa0XvR3TBQ0B
z0KuJAc>#!Mw|iZ(kSMKadxlYm;oRVHn0p82|Bf+)U(Jiclq|7g&bKo^eLOqwU`_{Q
z_h+xz$W}5uY^L2tDu@QJ?R7zD*2%Mr#X-hNo*xhs9?TphMTrngbDCmi9WFX#cQxhg
zA>U(D&v1roUK^43KLra0Qhep-fYWEs-NLWD)l;@T|D6;33Zdfhi>g!n|=v&0(h7`=6^@~KG%r@8>>=E
zX4AH9zZ*o*Eg)rRa_#8qr_Ih@2KBdNd8x^7`u+NYkaE9I7&!A>>`jS=Ik}3IEJfgD
zZxo?LK8S9T06^vzV`7-5w5s}J?b?|4)N9uC!z{_htOM`2b(`!2Maz)onP&~laE;X0
zqJQ%}ilaIWp_Y%JAcZt0u+N1IS`!l!d#qJjXuarmD62ik2ke6#MAoaJre>Rai#ofq
z2d!%`oBPo{73H3C531FqwAkDk#RVy`cv1#4Be7$2Wp326P>Dct+gCU44v7RwOEj0j
z>5b`)HOP6@dD8-LCL)j@ios!>2=PGNolEGFi<}YZy>{iKcEmDDHZ^3BBKpwCt%|q&#E*2=PC-I#v1(^48`gY;{UC!kRk^d~N<2Vzfc^k7NOC`=n3|k=
zL=Hy#mhdsAEv$FyoYygo=oDxwLz)bore$s(()51y^@P4pw!fXc{O3@Ss!%BydEu+E
zq{9DOi7aXvQZZw9Y#Fs7LmwlXL^!l2$
zV)j<8Sx?8rue-Uxl%5pt?oNwOXl#3v!bu(wG-D*!8Rx@qgxm&X0fGMh>ObuQduB1*
zi7wgtinQf(Aw*;{Z7atjd3LVsh*?8ZHK6087sKAWH+pUWax!A6cE8vBiAGIgxfUud3
z&GH*uI`pLsm&&N5VTiKhzH%&Rmc-9$KVTl
zu(RIzMfm$iVaA~b9843SoFz%2-+?QTNt#u`(2=;ie#TBq*p~x%be@GcrG8^fgiYSA
z!MnT&&?;;u?9?|-D#U4CKaYCxieQMI)Ym2PKw+Yraj0u>ekJr1nj{pme?>!-TD;^u
z^u0h!efZHLgs|6OEabU}7@3H7zEBU=26?`w&CQ{`K8Nrkj1fL$u7t`ncB<_wq<IqfO-N$!E5k4!uchKvkG-6)W*O2
z;v#PLFF?ER33X?uW^kTxGp$GJuQf%-Ot!>2#AT;X{`5I$hI1Eaq|KY#UNE9W0&M6j
zIracLf`6FWsv
z$!y1H!(1%bE6TlT_l#V|DQLMRx=!4zc@g`ATrm3EJ_o-n?UT;449W(X99Q&!&
zHc%hlpK5D(`Hb+V(QSuouc=iS>AC&6VkCU|q4rh5&|~q@A=|pVE^j$3a9AMs)ZGH?
z6B^HxoO>SZkK$mCd|A*g{d-tLQ_ET=Ln`8Wzp%aekS-~v2&p{A_Q
zcOBT(V`LAPk%luxWch6gs7lPHlK|wD
zyk>KzPKt6_A11FTgf8S}B_Ol&p9MivBk^d$*qGfFST58hqW5^Nsd5XvLjf0%<9pti
zHwm;AI55Gcha4`g-1z9>!{pjUKALT)!!Nx4U9nm`@O!GnYMDYQ(pU>Gd)n19Cr&VO
zG7*5jy09~d#NzoJT+-=ZNEj`$ss!F3nkxMIVz2yaGlAqHV?YlBnre_bbZJng1-PxL
zIM=QM0y5NIjh|Lt#gCz!6uZYF_9?uj6bcI*!01s>_`v;8KmCA|)Gp{kMY5NH9BhN7
zp}+mMi+=62Q@`ZlzKbf8(=?EQ7@hmqj*DVRkD=+}Y7oO70$A7OYHG>@xfHqEFhI2V
z8LD5>)b)Jd1^A%(JGm4DD4jn2+VpK?Bu8#uo>bgy4Sek1Y8INFM+*Ih7GE88ZK~&>3H8wW(zkJkr8MA2|
z7e>pt26jrj?UZ=&HFZHzRaG8-G^3KOS>`5n`AJ3K>G*w*c>DhS@vcH2pY{0|#GB{7
z$LZFJZQp|j{laT7H*)77qO#v-B;ge?lhwfTH`8k5XeubI1Io#1Iz`PlHa2$VYsJn$
z#d~mUQ}NFR$lLwvS4-wCD=PtvC*v}G`SK+tr~?n3WO>A((ol7
zsDt|FvPiJKuDFEUKL|r4*lK;6g{zb+)VD`0kHUgkLDUe<0XSu=
zX8oFl(vZ_BJR-suCa}$gi-53w@7!6{Re1kCEf61F{@(KeO^L|cNE1z4DGb!a(RL(a
z0JSn#Q)Nd7I~wl^Zx&aVpbY;pG{mo<`}*STFW=i?qVp5QgntZq<}kc+g@JJiuFq1(
z0g=Xkn3iHD0_bp&mo1{{uOat?_+cXgW{dBTBOG!Zx!6^BD*7Fw=EHFSl8!&U_9eRR
zbYkv7B`&;uKu!Q&Q>a|O<=ZD!2r8w}9Y22j5Y)j6E%-Pih<`eg?dir{jv%Bc)R87?pt7BBouxEnsV7mm^=}|ETaTmm
zeDxblD_>sdmF6=2IZSVEv@CP7$puS0CZtrE5WNlZyo<N8;OFDW$
zq;G_%#$Egg5LX!hqx--F_we%45jBNQ*DEAMt=mtyx)wHfk(I~D#KhAWQiu0U(3lQG
z%WorsY_IOZ2Vcv;@DU(B=fLzp6B!m6E5;u@Sesi=pj0QdFd~O3@C=ah`(;J}S1-ZB
zJ&dHn6_Ry-OVUi-q1`D?r#gi@je;bg6I$!%#g3zUaB_G7sFaDWeU_X?F?-`OZAH&6
zwG(*Y;a3Iu`9C3SP8eNW&GN1QIFCXdINYB2!)^%)Rt*0;@+vqmkTf=52C{qz7#Hxr
z9l?3Ix#YM$jFB>9?(yE;fo$Qhaph(hXs;o)_csqe3kq+lamV3n~1*fddt=IHBdQNWPjN0vvuh=!h8~A?s;ws#Z%0lgob)!(t9UD&3lwSEO&(v)ylwiv7CAC@h%YO=%I$jmJfPAf26owKMtokb!jpVTA*b42J(P
zZ`%gn`BN9sff<>ewtn5ji+zHy_Ypbrh8;V6K;7MquONuaAC@*2$SB?qFD4#-gMoY|*I(q!S->FJM&w9oos=y&
zlYV2A9@cUVd=OZV;x5jI`;4U?8t>32w6cW$W+3rT+_V~kO3f>ZKPVUz;nX9v9!3Ng
zp-6RYqNbfld>L~1`M(!ut2SeBK`&%cgNxuy_-Ks7*Me7U8-b;eK0|+?<`D5W
zt~SIM^M$${T!)AD~N>8ovpswvG_SqWq>eYW&V`nG$KUbq@#Yp}culNn;W(=Nb<7#4Q)IuFSsW|qe
z(~z0+iJ*wvJku+eg)1}ryt!!W)#EKPW!#OcWbWLXF#q`v@S6YL+sJ6o|Gc-|Fxyr7
zmCHH1_tnVTkyFM~msQtu$WDdsF>vgwgCtN_e`_P}iJS&S^J(lMP
z4~l?^@f^X(+9093e{UCSm%~X*@vj|%+Ye6VFKK$1ma=8=-~CMEVPUFxW0OqKrB^U?
zhLut;W%r&8RO7x__W=`_APN7QLwJw)vj_sRwzh8+oE4nJzqS>=qsQkVKhi(=kK6us
z6~3$f5api2R(oZ{w{c$jvrT4f_46$UC2fCFr!j5_F-~|W<~nnTTv@CvO4UWteLD>U
zRxU06u=Nzi&LYbMPbf3#~Nd+xR2_yJB`m+W$^JDjm!yy>yCv3qODV
zJ4k;*ZwZ7K2i{JawtIH=@4G!Jr?`lZjk;44!5l+l%DdL4yU=r#xNcUE{7z+as_;GX
zE62bG<^LRSW=@KD2q3@gA)?MH*&iXn)G}jA4;V(bhWz(d58h9gh!J^*9(}pMwoLmv
zZN1p_)$~J~(+Ux-*WM}NJ9*-i#0wQ3;)a^^c9Ia*LyJ}C-AkU^Jr!|OA5Jfgs^*sW
z?)m%tFVCj#|NVORJ5FL^2f_B4*8RYmd(Q_YX=7gutyZ(RuzT;`)hOV8QTFVK#$m#-
zSFY~H8|sp?iyB6!3*8nakimSS-@7MjjU_@!uG?b1SisVHynpqumS9;58*RU+pKr`?_w@lr&~u**y$nd15eya2n$wM5(S#*7Fno
zYis4@r;jJqA71@;lm+Wghm$WD+1bAHsYf}Ao+!!^Y0GxD0=J2sx{o^zt8%9=2F9mS-t}dgef##+#;oZ~
zcDJux=6BPl@6;}3ge8*g+=MA)E~C5~MFh!t)qs=LE?n5J(!RcPlTer|^~I-`K4C}-
z&?+1~CRaCZIQupj=><983Mur@ZaE^Taw+E~rMPHK=h;$kCK21x9<7dh#rO9K{kvmd
zx@sLbAXEKkOSH^6UKOcz7q@$C)ObjxaUfI?M6jCQadr!{(Z>U1`~nzyIsPcAw2#Rz
z%E?=E5_GLzbRxZ~5qfA5V-
zk;Tv(x$$kKE=|IfG_E<*2ba1KXy2EW(Sj1wMd38#gbk)kvR_+rNxw>R&?
z_AhR2uVvNMH^M&gHpJdfLljS(m~6x`PFj^ajk%BZUXWh3v~Vft$FdoNhaho`eg|jo
zugE;zbM$MkRh{{Fv{>)?=h9w1y^pH`L{~kicV}-XeJ_uN*?X+JMQsbXnZqZzJ*tEFZs7!pOWwux-
zZc%vD@#^z&!{sBN_5SX5=YFG`+@2|mYm6mQ#AY1`_R`F%9O)60uCR>@+v$1l3Nv5X
zNZ6&1x^~}P=T&ekU|FBB^Voy@JA8b6r4U{qBmKgs)ZhU#PmISf9qUx3@^J58&ndV4
zk|Td`Y+zrLNDw>gqc!lMEw8LJTC)ku%pYYCm`Y9~aRiGj8$*nKQJ_~HeQ`hU>XDSC
zh(+(^Bzv90Tuc6Ry0MUVf1~o`Xn~QeltEkedGaB%96R|`w&lJ2LQdfhG273lo@5bz
zc?j9hrbPEiU0cJuNAncY^k>4R8^^|HjRWI%S5{Y-#V7<|G!Bp9gjdm-nhH9BQBHvp
z!HL#wpaEP)o20y|>hdY_>M&C3KB!ToaeC43>qLm((zfF}M{3K%pEMJV-VQchvb#Q6
z8Qd%MH@YhPH!$CtOTD1}@J*Q3k58)uoWHW8BlfgkH4sNwr>_xndH5EYpT+z%x9RuU
zGMB6wRRK(Y*lFy_=B?vZyl3UmI)4bP5Q+5JqHC)?bZWyo
zuk@dI_9uRq>&~_8CVDR|-5>uM{`3IN?KAO~Vh2mbWE9Kt3LsyKCmT)gr4e?gwWZ;
zY9WPQQDQTD1q>VLjD3t|QE4$@T=rIE!G&vadk)7;sdU@0lDl;V-BETOUR)Ao=@kS813d|Sy`^SfLI4`W9LsGyo=elc*(=WIOPeKU?t9akDWQsBPYf{E}-7R9IU?bI^8EezPs|iDOa#-A-gSm
zAg_?|qvo-_($ehGOEY^xK^X-|Iu?bTJ4L(5RUD?*wx}0eP
zPolJp>1|%lNS*L4h#V=-pK-djc8w|3hgEB-J53gfb(mjE^KZ9xNI%56pL$oSvg4D#
z55R1$z~~ayQpfP43vR<=H|{7r*QKG7Gv2l*CI2U*j=M}!n*>lwv&>`q+4
zxio%a{+t)~%Y6=F$B_?ktp3;#C;Q1x@EXM;XrPoIVc|Xz6ZDE%b7o|-)7;#c)tJcJ%_3C|r-dkY;h*x=S
z2aCN#a-j{~K?_44J_jqF6uZqO&KlH?mAe_1o*oyUxxALBaMwHd%}xci)d#~{v!(f3N*kvGz0D6;5A`KjW(2#?>z^cN>3M!qT-Jp2)
zW=jjjPSC>wbB}YkMfkMX4XP@A8ctfb*13EKTj>1Q&%b%4czL_Im!om7>V>TiyJB1H
zW`7!KP`fd9ReL;^oY}i;TwZcYHSe1G+=iDY182n}3wO|!#%xjk8l`1YTe$nnYquRl
zM6?ly?k2M30)xqWm_W0IcggNa{HNbHBv6^qBB?$J8dV-;Bk@`OqlfX>utjG2dj*3Z
zTxs1KgSjZ`ZUCogk)f7i>s6CWFmXYz4YS>_C2U}1-R~$+satw{YUhnwRpp6*utCb9
zJi>-zT~AlCnebjsOpdKDk=`}G{*(Ri*Z)cidsnT^H@<~D_i&Gp`ll)<<8#W*)g?NF
z!$b!y!|fO9wHi?)yo}T#JC@5XQE`+eZTd`m`vXg^**&e%;I?(dRYGlN$zDu=2SD}#
zSS3l1Y}Y=)=npsx-QJ5JE~B(Ql%uoTBan!?ILG(P?w*#@@>8AtzSGe`fqU_hsA*}*
zE*Uv`22{Cy21sSnwK01Nb=7^e(7R9xQ5N-{E|;cmJVyIs)Wm~4_GsNhBlpa7B^p*U
zi#e|<*-s2B?rUwVqCIFo;1irbeNsM~ZY50YNpD3h_l?)x}LH|4*8)$FS<0IKQZMFI3tSOk3F1;M!AibNB@G8#beOtM&G@GC5bEEqu5mI}%?XVGb
zxZ}@ttz^j0Q8*NB2)U{~rUF*~Ovc`qVm7!DMv#;sJ)b~|-|L7#_v^_mb
zPT>5<)6)p8{%yk7n90Lj&`IFahYy{g^&#E+1Ue(zte&tWlYLb0%bD#a*S_y%oGHI6
zlNzPaX4kmB749D9Nbf=mBH99ZNU0axLZ7sAWgQWt?O
zn>TNZLvxKpqA*ht@kq>N>VUV8kL4|a)CAAosD+4|qt|O{7@kY6y^_G<=QHTEeQoDK
zB42m0xTSgfYMq>uaa)RhH?(bip)(WpC{9#TKddBLVUuJ2kxOVtkT2F>B+x-89J~Jj
z|H<30HiytSov!#ITlscpO!u$Owv-+Fc@F0E6c|Z*+$F3ki#D<9j6E}7+{?7|KzB(a
zaX|EL;u;V7G?hag8&4Z}@h$5XJ-Q^#wAzuTWUa$bbZ9x)>I`{(*0VX$
z1gqn$+ws8Q#fh{7(tj@j|Bf%JA#(3{v(vSEW-8f~axyM$TP34-o5TDadsu;+P<;Q|
z*Sy~v*`BL^Oj*!c_*toEtCp~x*X;|tx3|>+F*E)rU2Eg7MQuwtnCU%YQX}sn{QGiB
z*pBO{HTR2BD4=#
z&hBL-0%chz4sTuHxo?_d9XR*I#h!eOLA^
z+l^(JUqIuA+O9Soo|?VkIa?Msuop{IGL=rCsrhox$YtBt=9u?LzwHnKc^Ww)vKDGR
z4EcM9p+I7J`T1jv|2>91W+lR`3>feB5R*LCfU)!B`SXV#KAgauRYq`0S9T0z38Tis
zZv&LV30!_+r(=YZJ#%wnbQlP)Y2(z5O>N(Hsi-j$6}0wR+RwWGoYa*26B3`g!9tIl
zY3YLbV)W*3ZyJ8e)3)A`*5sF_4`BCjr`l%1bDf-{3KMCvfdw+Fk;lKyNZg|Xb=QP3
zu=m}M()%0O*d77$(rN7$B|2Uj#WUC@u{)V_*lO7ZeOgGEVBSh$qd%MYAyBI`H0g6z
z!>V!}jUcR&|2;bKZ%v*5e^CE*o{xHDwZ|7iam0ktHrp*
zj1o3BsT-%hrQ+DvY)jmvk-u`$y2*OXssjlf56!JXhNW*hZV#%wFA_<_gl5ZG2xJZc
zsd>mQy$h}O!oB~kiwE&7I9Dw?ilB|ykP9D=Jjzd^kfBrJOo3~FZ<0aEJN6P=*$vkN
za~oI5D#~)OSeVmXQDTngZa;d~lG#GfDrG>_X+$3IUt^2B$<^Z5(Ix4zTs2}u2QlJr
z4YGta!Y_&27xMkjuPp!Jwm7NvIZTuZd6S|Kdt?-C^!GyyOAiKglSJ_8r2bD~UjmKg
z+D84D%8+Q5oD99l6e`k$hEu&}8AGKgk)aYr85%T6b<9&}kSRliGFK#$$~@0AAwxp=
z_pNiz_pNXJ|N5Qvsx`##To-`92RYwvxr3UQUJPi@m?A2Y=6*uY9t3QL_Rq`2j1
zKkm+$;m>v~=9Wt%8Y)9dOJrjYy53Tn*g(y4n)+Wp0pn@X?S&7n2u4E!$}n|!3r^vwq{S><0Ca=yg0E#%EsTm-hvB;F!eMjJIUQS)bzB7Ac;^Z$e?iSxvFa9kv
zt!bV1;bW(_oEdR4W>UNw&XfdFr^zY@vdP`F__RGx&|zt?uq*VkW(Ii4Qq
z&rLpVr*<>=MmW2f+UbV8Sz^qz%iljnav&}tOY?D^P+(BH!w!MdAGJSuc>fwrSdg?j
z&%VtM72Q>wFmLMXLkt4{T~mFa?Crqhu$3?{4Un#5GhI5%w=ml1nzDnS%<&D}wAuPF
zeI@IeJ+u??E4WQ}_An`I7-rfOJ|xX>V8O)djpnR)U<+}69nt&GZQg=N`yG>7Bv`Ka
zGX;9~@moF7U^yVka^6}!rPD=r{a(%>)Ao(_3Mpktbp-`
z#(z(k$IjBu_61x5HjTK$!c~h+*TimR)m?Q>P~{5WCFAMnzTQisv662}9jt9um|JG{
z{VMT{`azXaGW(p@Vtf<w`{=JNVxxqC0dLPaC!BjeSp+h=w!C
zELYwdE5`aXA!^dB|CR6$$?p6ZLG^9ht2}lK3RUEF8%D}}VjgyD^Yh&i)74}xN?7&Dk3}enz5aUW;a0LFcte3{oX2=u$hO)#!Wq=||2e$BsodvqMaglY*kd>*
zL#Z*w;na(S=iVaiHpyY{`R`KVrJ>|S7h0`pg$WmxNi`(&&DM(FxMk&?)xYK4%eBgZ
z$0D(=+mY64#jSoK^3Q^Kd_y&^^0C|8>@6>^6{2=5GOOLv7-Mw}&S;bzAwP7b@I(Mk@xg^j7R
zi*R+eww{+42g1cDeO`Si>E45R)6$V&^Bbp|f6ZEe0H}`V`yWmV+*4)#;P@bODBnn5
zD3(5={Kd1)1GJ@+!~Y4OynkS_HDTO@{StaA@m8qcwi_MJ+8BzGFT+>Ix`c)>bX$Ll*s;j
zX{(n=Ua-WqO-v*;2H9FJP)0N6JPSVz$KH~1zx6>o|F2U{Df#>JB^L83Jb3eWyB)8#
z8Z$Y3xU-#NSTP=>aya*pHD^BF+1B7J({qepc4h5Gza~)i-(9d*>$&tds(eGs^B=?S
z$@h7`#`?*ZfX8akR*-7{{$eheDns(k_(2ltu35Qx_SFr=_cF9?Fkmk;rG7_UfKioEg*Ead9|LfvC>T_g8sMC{=Eqo){{G
zv7Xncrs5E$aIoVQG`
z&X!%=1U(NbE-t?Kb$+Eqf7ObXcJP?YyImFH4W3w
z!z~PWB(!p9k}kgay+#L~+mlAURq~l+C-;q5L&=h0!om4r?WW(OmZQ%mWxD#AQA@Pd
zSE&%%;1WF_Fo?~-PUP8k{?wDtq;nSW^l84r903v+2sX`;CtmR&aEzOl<
z6|IMPZTG=6YzK5D_Y8j9rm)zkfndig%u_#(;QFK)4GSZJc7Q+%p0_eUFGeP&?@)}q
zg^;4n6@#E2gokrc7l&J}6pes%0<3y2tc=jxd4bwTMgW7x
z_k4W1WF;ydd`Kn2;zg+splVH1Qxhv28&TYRg|PeX+MT>ssTswE8E$im5HZ~Ydfy7A
z*LpFrD;S(i2$Y!Jeg0crWovaaD4H4?ZnGP7qsm($fZwC0wg_GWnD)l>b81RRMur>P
zVK0U&$6a3R(3zsOrut}HWgE!rH7OQccwaXVHkYqjwPy4u*y3+Nac=GJH}0GeVr7Jq
z@qw@Jb&PiM1)~o*1!@=`%)L*&u*N7LQ3AV!Dc>2vK~96JpiE#0a%w?~AoYpiB_sE(
zJG1M(mSJs*)k6_3;AVUB-#xf7
z9rE1bk)ncvKQYc_8E(5MDwTp6sUul6eXjZC0EEJ1%N*Gw*`XTj@IR
zQj=5Aq14**VR{`oO^RkgeDZLc=|6fEbAM~UH8eJQ;JzAvIknmG;c16v3xD9K>`-JWwd$QWw%(kB6`UH?*qG>;0vYHJVugkIxY
z#!z3I#hb6y%`Je}-eP#4)x<0&REt={pdevlDL=n7+*BI7%n)#)-3WuSz6k;y-cw}YIzMb>%jR7Yp0
z*oKMG(Q6=%>WX(3&VaZ*49<9k%8{Z>>3U56#Q|mAR7e^_kRq#;J=1t-^&>mMD3zUH
z!;P62z$M2y0y`N-9CV3)Wfh?#i+y+&a<7o8&mTX&BkfrqpKHhg+tFv2=O?3`@Z4P=
zW_(UKl6^-s>>?vnF?T*nImlfE%`gjY4UR6ZkWb#m1?#iaLGpj58GdhU?EFoaXW$y%
z!y2j0Qg?FNO8j$ZA|iD!T@TNab%z;J$8o|dK*H8`4y6$@GxK^!F#}!Q>$vsDFs)EN
zI{?=-gDm&f{rl;YGomz_DSQdA-d}7V>Qi04ACV?6DaY@Y)x$?D4);qT({FJUdO&0x^qe2=4kkGmLngRoOZDJNZWQ}6wL?%BTIZ|;
z5m&azVvj1%YbcG^&yKi2XZjYrMGtUt1w}>o*JNHu9L9p-AhRt8%GAO%D|dxywmE@1
z(28NBtAMkMV%M@GuUL+UEeh;5B=jhn48o&O{Vi3*xW87&;+J47svsHb%>z4zw&|tK
z5^_%AMF?f<)BJ(+dV~NYIDV
zb7$1QyWG5a(>S}iZ-lJ=+~?2LUYyXwKrjy&>li6uChQC2eCS*R(XGA+nGFkacb{Y;
zA6nV$x$lXr(@KiFp`qbAnOJ5B|C}?(iO4424JW2y1Y4CVAO3P9j);uQhmoo9F7@cP
zS+hQ!I>(afzV5NnVk*@XY0Iq-iiEW%zzWVex30LK`9lOs<3H>QnkX#@ql#XWDQNo5
z?~swHzpwm>%a$aB2LxE03x#Okc;~%T_F)9zHx$SF8L50TGJB;G+&
zE+VW~hrzfG;X4iO{BAn^cC8K0QnmN`-^a)G2F{h5904oyFqz?9YZj{LQb&uZgh^A1
zll8|LyKkW&;F4j}fsJ!`d@QL(Qnl}aa2^kEb(l$M>FJ*XS$e(WwZQOjyYE>)hMMil
zW?IQ?O`f0fcv73U>f^MQ@_$GU``6EI^kO`6Z2J5OE%<_)vU-Eei4-+^bwAfGrIs@9
zf+gmSB4=2`B)zh`#H5Va@ArHTF7q|N6p49>N*X68r(9;jlldDB#74x-8Ye%${)7wo
zq;C*AljC2@YGD(c^iD*Dkt`ODT)SL_MQPyCa%y;dd~EYFaVVq2Iqq?pt}({Q?2cH;`GXjR-0#Ha*KiQk_z;QftA;N(<^b;+|gb
z?3N1}$n$o?Sf$e52+EM6H2?SKVm^km>_0jO${o9{t*y5Mtk12=19xFvK#V3*NMu0E
zt1GQicY~l$>jyno=aqf;y~=jbh06yit;5poSW3L!h&ZxBB<;klkurM8iUr{R)CEu<
zH~?{JW}$Gh*r;^uEptfvAF#YIR}>SA(I2ra_7(GeMx
z8>Jto-sI-!=osH@fyto{JAxVWAi?6q$oaU!)u`9Wa8G(24r!Zhh1hbuNa^&e*T5B^
z=#|Vp-i1BQBf2yaUYehkX*#oW2Bk(MkN^f6x?aRvAXNY
zSDtQoIp4+{)|v$S%V4k5Mlqst*Rf-d-i^>;MbQzcf-Rv&LfIYYD-9d<*P0}4Zn>=9
zXfdMFjV<4PTQn@mshRj~na?75N)h8nII=9Cg?2)1P)~JI{}5={fw22k?@zOB
z$eZxOJN;GTpRYbioP#P-c$8_uB4wc11KSD%jyTgdx`7A*(qoZ%Hs&q_^GR&pT+!8D
zV}H2MY(DPPY~92$UuN{>f$N8>1+`uY?Q0rP>9x3M&F*W~ugU
z++1?<#!!3)l`Gt;(9BqtmF>NKFwPHYYyJ5}2qQ(8p~8x9vbZqVunQZ6rhep+-6wRa
zg@>ArIK(ND_{0$*CYQ^&9rm*-2+3I=Q
z6gIE7KMl5Wc_^LLswAYP_d|R^i-$tqF+3brbEwPCf;(f%c=*
zlExXFItRZ#bo$n4L1vxO*X(qDdAM+Vbd=;a19i_Zz|t{(sI^rvDByyaK7K-vY4sGD
zgsk6r$y2-6Dm8%}K_g#1ZRyutC{HPk^
z%VzYK#o`Qg9*?CPow#=B$0_KFUFHnq_~xWjrP`O&Uefp-HKdz!p+*5AOCnVo)_ruU
z2MD8grwT&^)Rs4Wng0G9dvz)1ucr&27OG_tSaM}!=6*AU2nbJ9!3!#b;IqY}B+`I)P;Cm+|0V!Yx
z6-ZJ8_32zl2I590SFTvGRZ?19=a99jYV}!%?~0jjb2;?j%UOTye~=y)!!htJ2cAUp
zkx#nWO<7qvQb_`u$>hNF5lRcn4RHr`8%s9BqaPKJR4*vC%8pw$iZ`9Y4PfALUngK8
zERQ2)CjvC0-a!BsYjWFltVbKi@fre+XoM@;IAG0d5FH2DN6)wU?I-f6(9qdm;sjBL
zZa3(8IuRI2S0b6Ul{Y-_zHD4zV9XS$a-UJdasH2PBj;b{F
zRLbuU)j2#z(%g?s%6e?c#YLGcz6;&O>%~9&Q-&^5^FfEfb3D_DATa<91zPVwglYL>v8Vh^kUze0mJ1|=TC6&Wu-46@L#
zNzOm@GIN)(En0-Md0i6RuF0BO4jWA2euyjS8wmoIGOau1;f8Pxx>1UFxE(l5Yd&Qg
zFPjM3(WAYJY18>>UpcNS|fn
z;z~E%^;~yru($WyH7^!7S&#wKbRYfz@x%nTm3%HU+t2q11m)UyYeZ%!kVDJSF^%@g
zh>8YM15W`*yf+4c`6%kZ|92GStN;Z?7IiPZ4nb)NCa+h!f<-J41m&g-Sp=g$(rp{P
zOG5ga;zZXt*_`QYj8(G$2Q-Y5Js+?d1{Xitocct3q{Zn?vVq7FOtG%~W*Apu9|
zDI5}K*}1s7S)PlM;P2+7wU3*buBv8?*`Hr%(W{p(JwBdIv{ot8h9fVo8xr=u9N#?D
zhyO91Lk_5Bq=6$~AzB^sLu7XO#<>a|&02@WC<7ff+L5C>2&eXTu-oAUPV~s;7dh2Zv2anHY;57a#
z$U-57fVa1zTdffP=n{88plL&KoMw3`a)U>Q-vfVC)yGh3;lqSo9-$XqJb(v<{#qb?
z!j9BKmTs`^cxkREEX;xgFQq>e<3QFmeVLPkQ}O(VFwJa-j+fcuFkoM6L)Emc!Z|
zhTdsVr{G{M>@ou+^=g+1U<09p1gpoFHq!*oS2I+@-Y(UI#sf_29?`
zdHI#Z*PHe>?~S@`+RG%@+7^XU?9LEe-JXZ9xW;WLbYG<>Vs*`VK+Cujn$B6cMw3
zKC^^2m2T7p`KuYix_o%^|H!}NUJXL|OB)64&k)(Wdf{zP&rNKbU(AP4!+C@Vv6LB_
z&D%K6^sS7YI=he--hzVBv%aIFLtw`aK15TwLn8duyG!1^V*0L&xc>+W|BG+{A+_kC
zW|Vvi2qCG>IwP0sFlkD?9^(AqtSBJ2A!+dY%Q4SHn5e!zJ%I9+m~<7%9t98O7{dMg
ziGc8M=}?8UyLsfSe2X#wV=se~W~^KyXUD`>EOnclgs#E4*syYhE^Ec8-zR%6;JcQ=
zLEGd{NFqFtOmiTIdhdY4QGgHQkqG*#UIjmfm^0m_e|QcF1N1vp_Vjj98gYIkpSu;{
z%6kVNA0H2x63|QIA+q-V*5uqD?|kWRbYV!;W^6o8YHU1y?bMWkfs4y?pBz{JTer~7
z5_+7Gkin#k@_skq4a}xzyD9)4p}1{#Z?71FuO{49Z??c9HhBjS6Uf?@z=~bnKh?5)
zA0bn!-5W2KNDSosKykw!M5mYij4){kt+?L~oGI;GwKtyj`kbTh(3+4&02}*3y1)~w9
zv8OgAs->gj>8pF5o}Qnoi4W>aL)!&dI!K*1(%rB$Jdn1@NB2WdQ=z51pSp!0`PVwO
z5=QcY-__QxCNy#gj2r9s?c4VlK4v1lEG#Uq&a^VNwY8xGB7F$H95Q&6jKI6{fv~Ej
zVa591jXXw>kLptaM(|T0kAXg=DxLi_&{iE%U10nKJ}u}3N_)%G^BRDlLtQwu>~ba-
zARE(olC}Tzh$0gS!dQ@OC{1-tHtdg1gy2@@|XrOHTdq=}FKErJS$;6%3HVIywi0`B<9hJbc^;FE=1x2}kS`xP;{)Nm?$kses_Js90MOt>3ek!CsL9%$n9~h1y~Ru%8O)yBcgd})
z!{y{KdIm(nP3#|d`D|cRMiOfPkoY8bSyY&Qk~}i^N8M>WfUBSS<8;NZ-@Lg_Ol;mS
zqCcBNXQWVjsuK>|I5&eFzYE%>`AA-QRKNdlEkEB9y)Aom_N30*$%zLG@QSRWs1Q#fN~hljxk0?&2h5n8U}9(2rXJy8)O49D}ni4~FBmIRZ6Ma2{KaGcP4b6;r;R
zB2v`DS-mfFHl&fRw*^{Q1B%B@j33@xV>O&fs)v3qrH*e_lEI&ntgz
z2%~FgoV+)m3d~=4*NdF7iN9YgcC}|{vjhtvef<7{pjzvwc!K5%>qgSj)?Gtmh`{3d
zpR1SS>MK&bZ^Z`r0gxh^rf-W_NCqjn|7aeN^>0
zb-9;JfhrM(6sMe{XJu+Ge5yDJAinPR5zc4w?
zcG01yf$f=Zk?%R5x*H~^gk>ijZAG?rkF}P_ZogEe*sbQhDM%@tOR`Gw(oo9%ge$_&
z6LV7;C{tsJ=RU4(`H2Ze0V^nm{z&8PIhJj~K|!*+iE~z7n!cgvjH`ZP{Jp&t^Ie^E
znXL}0%c=qYtz5Or3o{=nDoILe!-kV61)O82l*+h_PD?GtbslYAGPAy@yK)&gWm^vZ
zzLc&Z|IChThl*`O6yspHet5)XIv4
zcp-}MeavPtz0WSb<}q%qh^0=5*j~Kk+mqsPP78rB8y9AzVmEWXhWGS32JWOI86H2{
zX>MdfhBtT@~RGzuB35YhD^$
z6pBhde>Tu`C^Oy?e^c>Vm5IR;is6FMg}k#dGcwr+-`hbtvi_T6{egLNI%A{E#!~FV
zz9=OXyT+H6>Zysn3b!d%p8MKY!>y)m$3I4ChL@rXe^-e7fTT!77D@b7LinuhK4XJ#
z68a3Oi=<^1#QVv=C=-$R7Dn67K66xLvT>Ga?d9P&l9G}z8CfeEJpb%IP&?c(t{oDHS4LZ8cV
za}VJ5(FOI`C%BSW{Jw~~9nKW=M?ayDGCr0o;lTycHwxUt-FULNLJ#>G5QwvmZx
zRw{>Ws>|z}M?&IT`~w2)!D#}y^3AFFdmd5;@bVn~%K4|g@Rp`&ouixL6QcBuH$G)N
zfTdcWG#s^-dB(tdNSFW2?5{~Rb@fi<%KMg)dsyQO%TiwyHR
zg~cPYVVst+meJ2>u5Q9R%fO`u@hll}Q&LK5^&#?NQypX}#bDPU$OTPBPLvsQ+0@C!
zWH=hdQdWzuGArl3ayVUhZFfZ9$CBA?MWU4pQAdj-N7P}%ObCjy<^?)YjHr4JRFkPl
z%`)SKQt|iYJ(OJbNVpr_Sj+Q7KT=({&fNPCzu7RqoH`MohLDbUT62Qp7&MPX@Fxp|
zUbupFem2LND%^#QF#nJJHMz@M=|HmYkzxDns)Tr~1|`Ym8MoO5OhYr6jC49R9T+o%
zJMQ~P>hn9}jYE~V1yIAj`8{0o0J%ys;>^LHDS=HH-IcxOp!@T3D@J4E?nd&-l8(W@WHX?5E7T2fQ<0N2x4;e+
z+v_{8)qPl@5q|m)+GNn|9%LIh41=%<7doLiM4VT7Ey+7H9nc8d5rpZjQY5`Y
zieQv`S-2=%)JZz-+QVlpQ90wrF^s38{gpVAH2RK}=BtD}fBA<*ymQtQ$Il(b6Rumc
z^M6&VD*b@5Ru>>6CnVHmRzZE_((c=%%6mJ|w%ZE#@~-ag5Zmc=nwx(|uFm4i2?nsr
z-qG`iGH&KOF%}Gei58>kv|qa&?;Mq&yDDYJx9U5=18fBNd^cQgKj>)g?ELL<+mH;p
z3@qYy;Jn?6zD>D}lE%mudXuK~ml*wVU8b;u)cL}+E96GRWPGaM8@wmtpqy4}qT56(
z&CS=`>dSpp3EPpEoI3(v%&g(J9JMe~%w;8S`O7IZ!BH?CQk?zSj`GeN{hb;EMYVIA
zi{`qzs3?BM!|?lWi8}n#R_P~B(lu2P@hwv>I>S13>AWEKTFxnR9VO_
ze`Rd6+GhK!gTp$BjIlR%jdMpgy9ySK(!-1`b9*$&yP)p2?5B6>*X7bC!=E<=$Me?d
zz6;$|pUcVaX6w|SuE@ZBW{V$VgzHiEE5Sx{IaarH*{{atP9E?+s&1G0s?cYLe@C~^
zFO$f|cFp-i#~7P;&cO1h=;X#Cw`b|`^;|wCC7*VcM6^VOylK_6{Fv^jZ!puK|I>%n
z-96JzN>FR=;{JD&Qtpv_IgG|Ku?*xGU>I$
z{w(?DBT+|Ybl4pi=>0msQQ_C?Z`?TXt$pd~%NbL72c=k)Dye9nuj$uRgX7$p73W`<
z>aa$Ui``wNK1WWE^4lHmjJ5tQ9662k^V?~9feqW3MCzVV(M^2}U-FAL*`1J3ATKi=
zHSVvn8@Fk$T1sB@;)W7}dd)>2TOHPlSoMr?foPwmD!a%Q3&Ry()(dL+DbH^P`GCP&
z)i-Ts^ts&nc6~{*3WuO||D=VvCGC_}E(Zr#tgtnrEbA=*3kS7=_3n
zeA;P3Q-@VIk&(9|L(Fcx%JQezb`8AAXKb&Jvgsx+usY$uqZ|9{wI92{vH5(-TD|e@
zcO!9k6+z9Y)-Z>dshH#>HtX||><f$()
zKBV9ol6O*@>uPXa#J8q>wX^Ef{7R`W#G`dKse2p$QmP?oIC?h3%_iy3n0tI;{jgKT1OsR0
z5$CFm7kFzY)pu!)UzE~QighFJ%5-|QK1;>WLJ!5F1#GTUl4lr=-$W=|RMkvF^xecK3Z-w+vjU@+eR)iJ9*&?TcsU9m~!v77|I)
zQRsSf)Ulq7q4ZS&n^@rfP37`0p2W9$-cw@stmk)ed|(-$88scxD%
zlE6>?HX`woXSKYfXDHRBLjGK1n{k8u{(t{kKj|i=nv^FJ|FevI1N~2py$QQbul^sD
Ct9a%B

literal 123766
zcmc$`bySsW6Fv-EkP=b4K@jOiI<_L+-5?DjAl;=RN_UsEQqm2I0@B^3ba&_P-fYi%
zVy!RM`u_M_>v&-MJa^69GuK=*vp*=xOJJfCqaz?7U`k0oQ$|3z?Tdf_`H6ZPd}2Vs
zkq-Wc=%6g|6rrexbQS!9_DWLA0RaJ@2KFDKl=8zJ@HmF0s-~l+oGibwtu>3GiLH?-
zi>vi3@H7H~pesN4(c09}kkZxqrHuo>s}R-A6a3(3*vG6?lsAt!S_x5U$|+Kc+S;2^
zK4#%!VWSd8r=+A5v^O#1SAHh`=XLO#5S4|a<12nvRu>l+78gzyTYGa>c0N8nRyGb+
z4i0AU1ha#ijiaF}vyB5ae2~A!d1mThY;XC>(bCq25;m@(k*$-X5ET_{qW}GYuhY@e
z?C+Uu9R6$zY>*Z9FRbh=Y^?tq8@wtA`;=eV)WP@7{f{$SGxv;TSipP&8rc|{9bM_X{#_Ljy{HjbwD
zV6s1N|FyvX>nHw
za#F}W3vbB5ht2c5t2mQ!v^7t_9`bnnVz&vB8Zy`BeW)vs2Yf0H_K)7_*P;vz*RlU|
zhnidWnJ^ddwo{i)w_cpaRV{n02Iqv?a?P*xoGe*#B2o5fq4m+4HJu7y-S;x$rbvaI
zkcAk8d6Jbo3L@z={iCz3VWUpF@__sHKaXxx>=$2JfB7)S@Fc6dWrdnyaQ>xC%gymZ
zrNC-w?%zked7l5iU(N;ri5_CsS>d*6bX;cr#(Z}F`bPeHzrW>Rfe2B)m|8Whk3-Yf
zKOQLPpllt!S~R>7Q~C=qoLVOJgBz)pQ~Kqs5|Ah*7t+05?RSI0T&?8Lo&1Ur>K
zlI6nk!{q>5q95oUhS9>-`k=2=p8~UE!rc>cZ_C73h{r|9CEzgv$*v
z%I^$q*`EwVHhdY;i+0Ls;jlGnjSgao)w8?t{iK2+MW?&)vL2dR%CxC_~w7>
zPx1e{x7ay7kS()c`urP&$7O3uGSzhsjmNn4PSwltQe|HCGgO}g`=Yo24G9}Av3v7t
z!>qS4ZWcTMjz7miiEcKrbbJR1w!nOBi$(ALM*(D~TLpJ1_F}5PcCHx35OJ8iClU6n
zo^+Y@n%=70-csDb&1~WPTciE#(JcBzyv`hXtS7j5-1bb4zuv-%mUe#=wqwnDtwmOn
zi3iR>rhz?G^yVD2kXCTCWOYYa(Ia5z?1&BNo?@MNSVFBI6EodH;GGdi;l3K79wF>?
zR^6W|QI{FY-*LE}8}*y*uXBb;JCE7VZGWj|cfPHT#HRXNcbbSu{ui37Z(90AJEcZW
zS?-Ll2{npuhaumbcEtl|nkGrn0y_NcrP4o~iJ3>WxJOoUm>%!$>9C5{Lic8V(UPYb~x{1qqxKC@z?kcYE28UFd#`3BhoqYjF)!MQ4$U^^$Bl?ePBSx(6M
z3YT=Fw!>9X`#tp5VG#Rge(yqZaU9jCiu&_heU`n_(75{5{>*^7U#~HyVl?zu&yKg#
zu-Xv+VUhgADZg}cPgc)|iOg7+_ls{eo^LtWHeEVvGKW};tS{cuzglXE7Rf2tePYec
zofF$#UE9t3XV%X&;E0*|25#;Hlo8v)*P9*B%0pgU1!Kog{oPBsK%e((CQ6>4o}Zj0+uLW)Fzt9RmOuq5AqBgk09;K|Ln`c>
z6B3DuwX>I)N%`*;3H(xYtKK|5E~&vHDV*N6zKp@x_{Df|
ze*RNQs++~;;Q_H~N6;AD{AM8zaoQpXqI$9ZjEm8sbM(>Y#4P#4QkD30m+cvw=C^1z
zJHNgevf2D=pLl)=oaVJknc_9(X7AWDH5ROR{yvM=$L36+1c87}?
zOegL^J%n;2Y5qtk=6OqLqA5;CDODU%`(8A2e~E2Wg9Wn5%pFqRKDh*r7VFy8oKrT^
zf8G`gbqEFliz5o@D_@U;eu*f#>QR(C$3|`66873QOg#5Fi@34JZeHk91=lla%}GfM
zzFT%b;)5X5QP=KL<5&WAKB?jR4m|E3=zZQ{k}Pf2ER#ozEV9$!|E;!AoO^xEk3#Go
zkH`@sTgVoalrTpse^n$N*CY=&YduCfNs`7g>Z1L@oYH*
z_M(C$GS5Huk$+#a-nC@Xp;u@y^o_t&l~V=**|k~5(xlz1;C!)LaG%aB3N~S?PlI{m
zQ~aDM``&q%d>*iVt)(_csg!pT;bFyk?9pz!zdBu4Rkz)E&U?h{`wr(ul<-wKp{*n@
ze-sUUwB;k0$Y>lCwH|BI8)!Cg^+awrmZiT{#lICHZ>;
z2oPok=q(R2A3?Xhg)fCkZoQ29W=MX)`L_sp;iDa*_R&_(2xN=h=$E39ANpC#8pbaD
z(2=4=`5+J=2q6-R6l+E!n?XP
z6P09Vz`fr3_8O)aej`-Sv@z}Ns1Z06m_&_UA#l>IUJ7(JFBmr^cp;r!ii31NX5<-;
zT6=|lD42{~$i14+VX0cbfy)ODApR&=x&yXmxkUsC+jC!?(r4~z^{0g{WXoc2TuFXT
zo+R2ik>lKj-+dYY5DE{tEivFiih+LcR_dToLGa&iEe$F>Lgh`0Ph?)~4b^AClxc)n
z-8PY{T{hHzXc+<$Kbha*sKIPu)r@<4=CATwE9$JQZc?H7tY{C|i|$(fy3u++vqxZe
zUAAGhaLx9>p&MYGnd>}L(CHj}R;FxhX)@VEF<#r{9Bo<&{7DT8g=e4Bs&ZjyyM6P&
z?#(`qW{Hviq-mVq)W&$(_g2}R)XeV)+l@0A^J_@B^>7ow4@`j0vfcnTe47G(RF`uW
zX?2^B(iA8|pL{r-Ry8iqFrm&z3ulqE_V;X=CB|(T7ifO)#lkJ@$`DdHo3!!5+kV2@
zCITHz8Xhpg!#@*f!zNhw8X$&GKmsPnIAFpqpa-o*$pe26TUbU0rc*b!29paBFDjrS{Y$kV5N*0_c86xq
zk#ici@74Ta5EQtg(gACN?^VJD2ZEoe!uwgJuc?IO^Wx|Qd}tWj-X`R;MRnJEq{ev)
zJ1qA&U7oGlQ2pnK#GlX9yA}CdU;U=mscEU^&w33v3fif_Gj}!_5u@Jd0p7izD+JlE
z+`h@Q)cG<@BSxztZO&tKD<}utA$q-+`#Mfa^=7j-2->el3W|}#<$O0dq%0G-dqko?
z5*>g^^z3ke_vYEWaHx0_osg_roo|N{!ng@n_l!Wn_0I3_qOAY&vrx)6(2^2|5_6*Q
z`*+}4X`u~iIcqTLO|1N;3gR2stcq`;(z8`nVW~N3C11Lxz=&E#owAh!Qb*+hrN`4Q
zvwcDTGI#fWQYyaO(B04ycSgFojR;7IvCtCl;<2(EYc{+9xDmm95I%4$g=Ng2ps1ko
zFitHDwN@i42*
zvW-Tg`-j>6bMpugG>E0~O8J*(9B_+-;1^3BK60JyU!nWm@*aE=N`Lr>j$Aou>|P;%a
z9C0|5CC;H&yti@(N%;)AYMd3P^_ymB7`H_UNRcReeBGZfSC!5)7%2Bj7E&y$!It-Kkiida
z5sN-bo|j`^Gzn8D?zzr~@J!Lf>CTyT#s5zA+-sNI-5=4hZ9!iLky#e%+GJI+z}ZUX
z^fwfWjS)@J>m!{uU9ZDod8V$7s#$NFy3I;XR&JOG?^1_zzmvyU|AMZ}23CC!DbhIM
z&0YSV)(wXvm`Xehh<-;@Z~U_{x)(KFxmUUF
z&S@TiXsOTZ6S%If(QtssrgDAa@pi>?Z~@b!a5PIzKCF+InFlvp=B6yM)UITJLpY*I
zPZ6GP3v)3~wtJ^j{mM`*L5AJ_Y&Ey8*XL^AX0zgOyYaO9QPRsfd}i%v+n~9`
zwyNtXVAj?zlq%gHd)s67+^B`KDD?MNfE!&Q?Im5^JhK~jg$@+Qpy}B#fHbuJ@~m}b
zg(fBSej+TewXPh0bcbQp`*2ue8b##fNtOL#ho)}#u+POjMO$;nZZPrV4!LArc6_xU
zo2I>(;vdaB;C^*0P-YwGZs}xuxX$cm=s`cOqy^(bcYU~pp&cn)I1KM1ZBA*}t+6`N
zxPsr_@HrxW5)@uTn;_8JiZ>4ZqzP2PLKJ$G
z%Zw65C@@9*IPi;ZZuC?Kj`AF^Z?uk|mP}jE7D91-(uNUu0q~B}J#~g(wm#=NetQy@
zHcgjD%2Fz6xx-Irb)Umf8}*GXFs@C=>S;xI`&&sf7DAWsl|$Y2$s>FsxZUNC{W_>1
zvdMKKV}n+uU05-qU8ImEY_rn$5kx*#MG4cGOqyT5ZZ=(CZ2{G+SQy)s{8cfk@1U)Q
z-$5NFI1ow`jt9Td@bDOB{(v);Lp(Tcx^9>&D8&^jGwZfzZ8%UCJH0%gy>`An?e#IT
zbU)rAW1a~{o&);@fMeCXV_@9G=TxT6BtiSZY6+_Ujcuxh0V**^TuJ+TpEyN(`IbqK
zH_CeXQ%ovGz*g0;(~N=Ki~5(%|9ffWRDNE^
z(^ctOmUXu1R5XyS>iN)TKN*VU-~M!uP5g?I6O)#c*CDOI3sf8%iD9I-Kux@CkV6bn
z^)_z?%*5u=`Y{&Q4+tHN8JAFD@%}<^nRXhj3E`RH#kWay8qsRQB;{p`jG&gN#bB-P
zO97wAF9#=T)jEC>I2=-$mgAW=c_L?S;C;FnvDeAaIB8Rtfb+KplL1CI9*CQ
zb>I5({jM9&A2jR(NwTg?fT89jPIGzT{ThemS`B@-7VGNY?@qrMh5B5s8z3K}`tMc<
zSRAbM8!9UCpr>cJx}y5^Jf$4!%v!ZZ=fzb)(7J6%g@J*Ao(2#2{O(e{>K7wuk)H9~
z@16#`Sm7Rf{zlU^R-)}&iuc*7JmY(?QLJ22gzOngpPvfd
zI6K(sLR*lkr|OJM9>R{i&kT}C5`a#*IT3-q6fSy~o^g9&*Oubq7s|BH_4%ylMAOw_
zvAkPRiMU&u)?jG5F^VEN?Ojwye2CT=5k@b~6IFX8UAOCyNSa?a4&N^kr!QS~88Kk#
zH3mzbv>~VK&U2T!547*JY5;9Edb!Bz^TGf3jSq(dkw0=TXYk`%l%8NoE;ypagU+6`
z^5_D$zu)ha)(hbI{*}?)O+oY+rNI2@bc(aSfIymgBCz-gq>h2n=u8BT&4}O9XUwF3
z?#X*#1qBrq_r4f@M{n-H?EA`PY1WgRaOY3JeS>!XcuNZ@NG+`ft?AnwUF1FhGNy!%
zr*IxQp^!N$I?jm`#;aoc_dY!q0T;odN|7)&Cmk*h?MSFHPt*YcSMAvSSI7}wz8pVp
zG>*{^k5T>Etz4Ml0$5fegLxpyruNfdO+7YdSBKVVhH=-7`+A|~rISdHww2n^wxZL(
zi&tDI_2MRPLe*b3y_IhhXxB@@b7H#9yfM`t$6K3@Z>RQqMbZ>vnW(QRzkft5
zW~oT#Fb&lC^8CpUnf#Qqqs{fo8F#Cy&C1DIm`iY49iX8OIDQRN-m!*f6Z}m_6IRRA
zOuE&5aw+`7-HDGG`J+x{49+60f*?$3c7dY#MMFjAscx-c{|-f9%KP~Wo^v=#3GHZ4
z<6B5<>wm{GX<;KbfMI@T{w$nF7{cdlch=^G`7|KNbaJE@k8RM_i}1PuQZy)!(yB{H
z14@|}Hfxa?lH5vtzFCzvIPJ9$?=64R3aUV
znCqv1&;tt3oo_4E2geQDUSdHpd$bov>DvSpr#~fvP0$h@~Y>S$G
z58rZhOhA(Woy&~d^5R86_-p?0a}Ez+TG~S>M1+@7vc>kvIZPy(idP47C3X*sfD=wz
zuAgH*@|>!+E4iqTWLw6QbE^RoGw$u4M;Sil4J-!iuD
z_d60MDZk68w2OzF*70FAmpw(V1z{FhJIwd8J89aTIZLctJbh#yf7f-to5#VpEwuX-
znTzIJyor)(ZF|;jIVFcexf?Kpk$Gon??tm23n@iUTDyR6<1wmJuhT94!q
zhrUawxAN;^BP+JbJRpSJdk;+6_v_yGQhey?669_VmtBH>4AAVi?@>5(u_s7VHhE-H
z_5!Bn2czCQ1X?F+394?FP1-hjJ<)rr4m|%F
z_c5Sg%}3bf5Er&-I>?;I3_qFD;6dBJ%>&33r5$^F&+zg>KASEz+Cz#)g^+bs_@GZT
z)9+R3JzS_1)d_$Z_a(2e{LqmA2P)Lce07qXj7!`;d#QW{=rjDwF&dk8VFUDZKp`(i
z8h)|iqq5#5HnP>O{
z&b#B2sZZ3#v39ly0nZ&Ih6mvn*YlSHosuHdLLmo_!5*`@o@(iQC~fvixZxzMslmlz
zeJ1h7ZBglkIRBcQgz+k2HJ1a-PZxV1=39p8USC~egdX?ZH|4*ZrJI>q2ZPd@j>V4B
zg$m4**y_->Pc$JKgjX*xn!&e-uYQ+f@J+kWcLS7)VBoh
zS*==F=5CsE_qXA(iw}*d1AIT)_7JqE6aGfd43_9Zv86F{a}%yY35SSe7(1Y@Rr{l$SQ?sesyh|
zGs5Z@69EXz6k>S|R+wcfW54cw0uagwA_0~mOOMNvk-1{a*Y8T3=4o%!8cc*=@u-p_cnPR)!0SDfJA~nbHY6elp`mVr_)f!OZJn#t(Iy6a
z`39E^72fS?b?uu6v0*pS9pI{UjG4ZP;pZfz=Rx8W~IGYmurKU
zucKB@#zCaB>=>LFiXu-Eb>cx`wD`jFGxfs7Tw=2G9V@{v$1@&nb9pwzSa0Wty8_iu
zC|?G2NKGQDVm0EaC#~Gei}yYBH`7Q)M59!^1@YLd(Ac|;54jOa&pB|
zr&)MGH8i%rv$CuCV>q=u>fu)Xe$VT?!<9}o3!$?llO}?*q#6-cS4$*lIuhbK)-4_P
zgmNi%#VbVyB+<4j)O>J;7N01pt4+3h>`0{R>
z6c376n&VuM9ZJR()oL|pOrEswWHw%PZ1tId@_#}YaBEXfZHnepKEhLpefvf%n^6PN
z#Pp}|@3%P0>t>sE*H`Blq0B=+pgy-OA8p!-HnoA-YUv3Fzs{5?@a}Z?ddw5?{CPe!x5Wq~O)zb=qkV>Ywd)LB`Ru?f5|`QP|79
z%a*(;V~`9WkEM{8WUEUqjz5|Q1ohj0UG!v(QTrqqciE=8m$!68GRsSl+C3x0b@0dz
z+pm+KS{Au>TaJ0sGiy5^%D+apyp0tY%Cj37r+Hy7bfxR%vYH*&n&FWheEyx{>eq!t
zl&0&-Zhmq`>3243YOJb`fZnH7WQ92{TQ2X!CT9=ztxPX`Cr?w1Keb(GCFo13y;_**
zWNq5Gcwrdy`cJCQLKAXGShezVv~T5vxTy>T)vwM4PNv4q)5tJJcew$-8RCz(zq0bH
zZmYHvMj=RrED`Kif5#FLJRB!7PyXF_xoIoVbn@`9MgyS+u(hqJ082hzDvry|JQ>ON
ziiTx^(lb%B$f{zU?j+BWsqA!9x-BHlx~&~eIrM5e1d`o~V`ALYMU
zb6{_UrVF8^BfL;ZTmVc>d4g|GC1!}o8S??Y({}B8QOWXEd{Ml}0jbwP|3{wMTaqpf
z(HuCm@{Zof7Q%_b(k8M17}p0oWCsA30|H<$Wu$7+Eb#
zMzn1}yX-Ui1Tpw9iD3S4vV~f-hL9}lyRT;dFMpB#8+FKdD)Qp_P-Xx=b1XM;?gqHr
zCGWjy?Ya!QsF9UFl@17RNtNtRZ+lm1j`VwpeKxRQAd>K3omktVYqT!pS%tYAE`CHW
ztG+1_t}EWYz9bhJwD~Yd_xa=A5u<9J0mNaM`N~H;hQRIH;s3!u(Q}nsXkqRXMPA^Tz@CR;M^dmQG3uTbenq8f=iPVCy
zH<4%f=(zMp^I^#>1ras|r$WPo(PI12@qIA?`3pPKn~d>Qu;+4^JX)qntpTuX7AQd?
zlfoSm9SJaABO%|EITO?!BypBr>OtiL!|l7DppfC#JyT(H%#H@kYqZoBKLdUfbMJ+c
zbCE%lx2d_^gk^3Gh=}MZW+{~BFZ9%G0I${)@cT~`$VCjZ;f{U|7UlV8AjJJiDnm2>
z&N$KDi_%aZm1Oq{nu4$zryrAI1bPcSLdUwfF;c{DrODkE?vrI%_j6dc04s7e
zSI#KCS-lV`ql{8GF+Rs?8M;XAFHf@sD2|}e%rdTQ^%aIx_2)7C9Ibm+`6-X9dpb0n
z1|>de7~3KBboh7Lc8Vbp-={l0oORRkfxGxkei
zwflRk!yQj+qIZ2M~YHAc4(XDOHL>F=TpET5C_
zHe+854WqehCI?~*ULQVrT9?-YeA6WVDVN6m6pOx`vVb6>isRBwmY|*%
ztVtWot)^>lhb8Mx($7CES_++&6CPGI=Pp)U9!=&Vk3M`^+;H@nZWf^GvR}u|1}j=h
z20Z|UJi=AO>N;X|jt7Qv0q2ReJ?sYd&g3uRe*lJ$`i-K#zgOUavtpI1B8
zY$OhNQa*{;hPlX)-$O;q+CGREcrw!^6MPqCk&4(UdYrOj)6cr<4B9dDJ?>{-Xb^bnZVnDuD577RAPOj
zjYkz@#)akkbO;qE>->yPk4Mfa@Oy2va(}p^cf?C4@|X
zySA9^Wofi82|x-gjmjG1wJRWJ0i`R&57#4%v8C*
zuXDPEFmv_?B6?r(U_wLv+)O!*0?mlyb=QN1DYolAfh>etfYtV&Xor6`9_wC_>|aQ6
zu?(aON;9i+F(~5T&uChey>y%Kcz6e5iXAR2(RLM2MkAwwki8hdK^KB<2o$j!mC2xS
zDujfmM`|}6j|*5a8@Y}F7Rt?c?3w4bs?K|KVaOQx&jPW7a`AW;8GP4<@$+?PDuqi|
zZUrp>>+29EbfoF{a%vl}SYuz++(L;sTfKf`7`#szq|mpqTwJrs)=nxgKAWhU0_5kV
zT4B+^4@#~L1oXcW&Ek&~*X=rw8;=`Sik^&HiU+CWD5Q~paAsQ{u=d*Ehec%b9-~_3
zDdi>!hBt9RnKD)fAAgZ5`JuPIZR4R0#UhMXUZI0%ljaRRZa&wK67Y?r4KxA86V_kO
z{bVhH?>C3x3
zBm`?PRY!mnN~P=1$8z@Yeiu6V^``&xs)vS<}=9i
z-uv=waXy!CZ8$D}!~KXSr9?!*qv`DvkhWC1lmpRsuqPV0xBM80g+GQ`-hQMjAXRBc
z(mRz);%4SoIQl@FD%q_sVYgDR<%Prx_^V6-58$9_a#IA_&t1!(=0^8Bzbvdp3kmg;
zA*T0z>M&*3=0BtA(lD{NmS5aD(+whmNbNLfNCou@sFGrlv|mnfX>E>NUZlM&`6;*Z
z$vONKgsDHY^Mm;Rly>?7^(eLrV((+O1yCe-iZsKlX`DsYjuk5{sTPjAjx^
zFmE`wk5TZ6^29iK(C3QOAp|MfeIXwL{7~)vcuw=sLAR=p{h#HW4zguw2u{~{*~#+p
zx5|~Sl>3rjHKW<|3LQIE;?=6w#0ZYEzxvIPlw{jDez_Od>~x
z6amp#@)oIVs7j>UyvyCj3w+pH79x-}k8+Yj;Lf_u8E8Tp)!qh@2u%SutH&o1Cxhq+
z9NqH%`MSY1u@{aT;1DsE4qI@cWaDx9dj~wH8t-pGFeQA+pqR(fdH0aCgM33bB@!kpdpgTpx8LzWPgqv1*t96uQ{?-rqyD$%Zb+j;xq@
zu=ttvQto8rm1z1rz{_hVE>KL$z8$229<$LG(e|>55IQRY1^dyCzK^0RVGlYPcxnV_
zYNF^ErvZvyzbmluWo&+juo}jImY6S+UaNyGm$pADIkU{(sCU~pG8}e)+7iDv22!2I
zTG{lzser%LOjaf@fq8;`DrB8ZLVCRG!(tybRkG{^5g%`+?xMOa;{EwQY(5vZpL?hd
zgQd%8k=}H!@F6{qX$LNh`)`eWk=q9d8a#~eDny7U>a9UH6?V@kKfUd5Xt8ZP&8b^X
z^)T|4p1wAYaLV1K72ZYL7q5(#YM-C2oU{>$OFj?yKwQD9LAGk!%;Qf2Ii&T4S!jWRFXWDK|BDQacgOxo=8l*w8&6^+d5%&A1om<-sa==oQ?DO
zSA?qbt-2(hL7@HR6@r#UfU7f6Y`KD^Cum~$+7WYEKFZES?E`8uC!-YCK)_5kvJF!`
zS__j(Ayu#@L)Yhga<;Z5h@fpCt07YpNf4e+@Vom^G0ZNgRUOd4?VTEpdoiUt1u;JW
z=W=Yk>qKFy)9PK)rTz8REWYapE>M_lfevXWgYvs?5;UEFtXb{j=O8wAZl536)Jja5
zVvf#1kY-4YJRJfgb;1FU)E+PeCrFA2kFNwW-rWP3%JMkwA>J*m7}+9irTT>uBlJP^
z9xzvv{ZhBOj&+Pnoc+q1bCAMPWzb&Q2edsSf9EJw9*g$3D8Z{uKmbbjMqyEiw$6@31JyJBu@gcxGIU
z#{q~N9S6iOL4IkpnWaFsQ3Tu$fSd3h+R)0phEd`B^O?(qsu96s$Man6Lj)hsLk+K%W;
zS{)%{HaHuIQ)Wf$k&Skx6l+zIAj88L?l<|pYs-($N<>>~y~UM_u4P+FQ7hyL>(bwe%8vl7-bv{IgYx5J%$O9uznDdOq_xL3{hIc`
z{CAE*gFXOck$N0wF9<+I3A<>qe#6vKlC3_2mZopWwNVjYto`(j+{0OAnPJw{&7bkrmyO~QLgk*KC1@us_kAe8VXY4J8*J2ZfdIF
zk{{S(Z<%~g>(spIC^Ms{5#nY#p9!D>$+Q>bs0e!%D
zhh=S~#CGngB+RRe2BpN{YA-HjOGY{oR^x>u&-(zsK#y%X*+HI(X|)DKLFmq!A>=~*
zwQL+jRhgH$sqUWv%R3pUE*hsq8-@)MfVqQpQk=?pG}rj#J{P;?c{eG0So%jA#?gA!
z4^%(o(nigoelA1A%y72l(nJz8(sjIU$sQfrk4ZC9%^-XYm
zH=b)=*4kQNNoPwv1M|DTeS*9NqgkV`au9?N?^(D)=zNK<@RXjQa@|Hu87$J{_}H(t
zL7cUMS`15-G3(WGP2>Q=lAsO*9KDyoG-75-tcHP_2TjctJ^+?Pi-Ed~CP%YO_-AHT
zJk9iQlNY&kgk|aYH-{pm$kmPkuBhI*N@ep()X_;0IqMHwqF~mkWZI6MAL9u1DMLE6
z2~r<=as0vA%3r)r$uTbScAh)yM0W;fV2etIv8H3=muU7{-BUmkeHj!$*C3%6DHI=&
zPh)Zz(K4?AflM3u`mrc_1PlULX6Fabe5x$7av9(OZeQ~PL>mOm)c7?$Jsi==#3(*2`qPBqr;U?0
zTD=Iez3-X5`-;e?FLTckqS|^scWZZLPA{B=I|DGa#J(~=f#tpPrklA?&I(&sa3w4Q&OZ7*|v~V|G}EhxM$N
zMs5F{zGGkM(ZssdOZ|&BHjtC+n?nlaTZ+G!^*lK?D4JwMlg8c)^c&)+w|}sVhg9Qt
z3uWm76k_a%FjHeE*3-YKbhdn3Lm(y6jRzI?Nn!teA#
z%WVW$64Mvcv(b~=G~b5GL=<8}C?FM-&#!eLI_^ZP=2^KBjpFVktF~Ef5yBGgDJSUO
z6yaP_$bJ065cG8I`X_sDp;7l;0djM=x*I=)_~BYaiEC;bsr^(fnMP;BC2%3@3w?~i
z$rOb`oCf)c9?d69k^O|;XeOQe`k;I&S$Hepv~x3zj-Jtr@Kfdlhr?1=@@%(i-#e2e
z>C?k+5`vd3abL8=rmyH4lV?LzmBz17YmyCyb+~31e%0!uQ_`1TcuDj5Jk%GnK5Hgu
zR2k!3EJ}HLU2RG%E}9q!^IOCksuRWUe5k#1?R#bu;@T^T
zgb9b05ml`k6MnfN_&HPEyK3kJq;@9ujzm;W&phaTQFDqJ#O2klre9R#%hC6$SyNuW
z${{-2x1T|?qwG_Mw0rlT58^@xW=$A;0hpJ+9s+jhioUzqoM%L_FQrp!QA4$q_f9Qdh(3LLN76
ziodX)f$yrjV=J!`#ys|5@lW@9A8{^gq-#5MR9><$x|}Y?36OovHJ-m<$SNHSCFL6s
z0y!~Q)k^0SwrzrS{cfub$QmnMERd<-=iYxwR=ljCD<&}#WAyPH^@1g1<+yZcyLh4!
z#BkygJEwR(A$_WAu*5$CBqjdF-OD_79#m?KA&n>lbuu*oVfC)|G%S?-NI9v%<05NB
zmW1V#DHU-bmTjfh0Cx717EeG1L;VIX>-Eu8)ngCWI@O5Il1lZ_A}AHHV1cjJq~FW@
z@Vn*FI+d085(+k_72G1s=sCR2eFxhNVb`7y)fKby92DX&z>4xN`mMHrHxAy4?U(TK
z>WZ}&??9OTq|y_x!}Y@3<=Me+J@k)l6h4B=bStI0^d5k@_%mqzpA+HEV~YwDzT*8pjRB3O1LMWlnNbdj9<
zrDhVY#e-HD^y=S!zty$N7?I&{;ndd0J^2#n!`4NFeRc$h4+jt?gJ%(O>qeJB!;u}V
zsAd`v>l>*aTYR}PpFsBP7Yym<%J4jXUc8hUy(QE)+c#C?kTPqBJ55p!+>1byz~|2w
z1y=w%L+5e&W_|XqxLb!qu4l9x-W?KGt@J!;@))}W)fA2lD#p0}mY(TiOxTD1+HAjF
zKDw+4D^cJPfU3Of;X$&X+O#k6ezxrVf>zH!VRgufdY(!dcAka-i|_$Y_nFSq29gx0
zbW`;d$ic?B)Yp^zMBmn56|2O33TnW0aG_x*y28)-8ZP!aZN6hV9gzNMQRquX0mcZQf6?oQj2L?Q
zz%sjO)m+FU3%-+dT^>c!6%d
zL%P?W3ZeUwtGA5L;_c}x#1PAQnM}UDM*c|fugY>t#n;dYLVo6gXWX#NoOb-zKs?5b
zbCU0mY`%QE4SQJoHB|dL4u1qYB>VoKNC6wvV}6Yn&Uv&>wHXz#(4&)u+pnd+s^Q+2DTY-ungv!u*u7@LC4LDq?Bi&P_eI
zFH{La@AX`kLWRq2=T+fXy--e4Ct-oqP}vVFLzQ{>9gsZbyne8}cy|&H
zi0Y>tY_2X&OGbdVrf>wo46IxW%6RXls8Rc4VL801lu|Je6a^Em89x=0>n{MU3jPWH
z$hEYou2i1y{;rXO#m00`;2WVLA&2=m@#h`U^joLPX$scEc5P(KBl;ejL*__NVU#MY
zlDT_IeC_UyVYx>a2b*HBEjmfz6ZNlBE-Cb|lv8fd(b?7#y=X)WZ+W{3zoF)|=_O^Y
zeNv#fUkVBI4n9X}1XrvMv|}U^kN_M1pXMZ3oj-+&tU;8vJw38w{7(*^bL8t+AH_5)
zUnV38d)2y?z+&@1kv+iNR7hOTBZT~xgc`ITeg#06Y5KBd2u7QHe0TtaJ_eCutDb#3
zjZI<2gnhWgvZ+)j39K#8xQXHg%FQr`G~olR(pc19!NIHMUHW*5yPQ@Fc|WX7ve4dK
zuL@-{J)bDIgb_DfH$}qWo%Bzrpu~k7RzM`DX3hYF#Knpjl2UeOeht=1neK3}FJT+h
z4P7U1ov%(*SS?#{6>bcH?f?PMA-a2`L_ox_NmWAW717m7Z%HD*CHhIm=(@MizNq?|
z_m3{VqxsTGY*npDT$!PI2i#kCNK0!^OX_2<@0Kug9r+c;9RO3t87)F1b^oIpD(?Dm
zppaF}(IC9o(HAO7nY|rF{^(N@T9ip`ne!=~ZpVWtmYKror)KM|Qtf8?S6eh`Vd@Nr
z>!ZCW@P=X-Uj#-+@Z*B)FLeinI-#eIKToxDLO`Ykp8|ap27MVLK5gaqZXWEQ1@iLl
zT280_HKWD6?e5`EQ09~z0*cSQ`WTQE7UW7ul(qXvq>VAK#yP{EYXKKi4A#IRmB{5K$01+-J(u#8lP`y)
znUSFx<64?}TUY_C$9B=a-Nf+Dzb09!M{*pEZ|y?c=h=>1z?@$bL?^vjL|+E1W#gb_
zfayk`@&ZB2hQ%4dV0%1SYy>{O5bN??N*Sg}SwiOkEL}4!fG)Qm$tPRr|ECjzlFf49
zVJ^GF^O?OZ5A~p_2t!@#hhJ{AG+hleFjuZ6W$WEcsEGMjBhFK=VAB+Qv)(;tK%sh?
zY@>2MK3*60v?sLO7%TbPkahohPy5$*0l)?GB@x=8c`ARs5;9EXGL!YATjcwn=2#17
z2>m6gmr+@h>KeRzQ4L3Ol_1+C;;O}w{iB{bUp6kZL$Q^EyJ{TYI|i$dMl>Sfu~jTA
zoa+kuXU{3D`kR>AssME2ee_vxVcFnv!S%Ouf};OXf&Vu8fu5h}nHER3k456yTGchC
z1o2GB;ifrAQdOp;6TK&{6M@m7**n;dQN7^jx!wP+F6=6C;`EJaf(ERb&hxZGBV(%?
zyjY&#{CzsOB0z)gfu6qz0x8a${7AYUs>wmnK$YaSkwZ(CwJGD2>`G`eNWQugip!g&
zE;S8EM+kvX7Y3;Up8Tsnqt6n};$;E5eY5WNW#xI>OaH}x>H!r@4;IA!m}(0G2oY%Y
zVzzXTH;Es9@*CMK@X&k{$ayl12uWNH_Vn
zRys;Uh;eA6MQVRKBokmxSbw#tY(9{+`@fwOIQxNO$==8upZ;W}M?=E}=?$V#u9%LjLM2qRsAH%YWLt%SjPK+#Vmv(fJJZIMg%k7%^6IxsINU
z-#%H2E#wW%=lW+a9v338`Hx(sRPW!|pf&My$tF6lKxW(_Pem#(*#T>Q;xe!3mnzbT
zxMoH+Wre@83YXE7))uP_`@@9K@&p#bMW;ahcJuRpnPm$i2=V%huKW_)W{a1C-9hv|
zCNi?8#+Ck7vdmQjY(+=+;50^`F1VdZ>t0Gaa4XI1hkZGxYEZl6n`00mK%bMuu7h`X
zdaq5E9ywf0{gD}y=y*H&&tj?1{ss%vE^h5fAZ0P?wHK9ULaUgH`o}fixH$C(>BG9P
zH6*dQxLqqWjPdfR?u`-hlSMpp`TM9K$;DvF)ST6g^VKgX
zjC*fOfx10}4-Hq4#CYExJHX3<=0@_pZvOF39sPl5A#{w8kSbHF0_F%0GS=XHAtYFt
zGr$|%Kp?H&B13oK0+C4v-J=pYmiNPq;JW^j=qhFCtXlvBF0xKmK_1$3@cSap#G-agu!
zM3|j2B&~^wwW|p5y-6(iwZd9{$F>6|;9scVg%&-|D!q-pru}k@@Ewg4=SPt;C2cv)
z&=N9eVA4|M7bcy=&jR%uE2cQC5n7;;=^XHKo1k-k6;!eEV6^@z0D$ZN;5|Yfn`8ly
zXqg7#bz;Fj6UB}it4zZfg0NhPF*z}!=&eL!J&-5yaR>3(;aPR#j
z;Mu1@6|imCzc26?B+xsx5Gs6T;R0wyfe=faC7bQ**f^g0v9Cg(kV>D_rv}_|3(1dq
zbjMgXCF>2vjcoX%fk_FN%YAjDycTSb%|6x%Qsv5S64QWvaR&lxFB)`8IezOotq$wn
zs7eOD?x1o(0oV2Dnev&)LqZNULYn@pR#zT0^Nw08u{Dbsjgh0f{u5
zqm6M()2kkotDl|rtiPWNk^TssO|#j@>?Vfof^8&D*t$u{u++IyS47
z<7OltZ~uuN{O;1enzXK}>Gj%j*qX2^b|S8O6ddY)-WWIQE>$#yjwkHA9frAd*mZj!
zozwW%`GV`jMR0xyF5HIb5FO1F4d5yq&CeO`~H+N`C!I0aQtM1`_IdVpJ
z?r|=iNymrE4-*O&r?F4Ft-XV&bym6e_o!JX-RRk1UD+_qK`jyLXjPzOsfJNX{TH^q
z!ToqRnl<*>NnDnZBp`Q?^jK!}EGx0iYb8Nnf5+&J6G?q%>_FSYI~uwqKRwLLzkVJg
zOtPz>gM;LBQP2(NjOvUIZ?C#dqnPPkGF0CmEmA6z_@bO4=OV&7YLJUYF%}{jr5?np
za}}ZAI7f2`N&olv#fAI6EW%&Vl~-bk&JTueWyk`n%Y7I~GTv
zLaImcv&}ClSp{nBWsDs|lD_XcsJJ3Z>tF;3kBm`LK
zaqnJUhC%XNRF!E_o?AvY(f691ZnEyXFA}y2#1XRqVUMe_^cH7$;m6QLAWmR*OrfCD
zZZEt3(Qd{U*+KBdM7bZizL5wR
z!uhNnt;&~L^pA)=XBQXUSry#-zY36sPeO&ttM}2*w^>m;C{0vt+O95xdm@(%0DAblo-?xa@O|9cy!^3
zB8PVqb^jmA-a4wvt?U1WO@m0cbeDukNNvI(l#-AJ36<_<1B%ihf`k%M(kUGUS|N`R0lc`uiw|}d*^@oS7gDcr*0KHG^t*v
zdM5af=mKPL?gXLd~$sWrwrKdN`ch-J+Qw5epHms)lSbr0_!
z^j~%nImVf11m*n7w%2Iy(1{)=bZ9h!5h-EAOLbK!$KNb>M?h<#t~6Vv@zj||GCMurp4#f=fMDfFCdw9HI5&}L!Do$ta!
zO)B95#laSc&~1S&bO8(Y)d$5|BJEUYzWH;&^sBG@kjnJbsUs^m4t@E
zcVl!@qak<}Kk>|RM@o8kL3#A?8~ql6%xd$m!R=YNaWU0M#v$hvuDE{(&|Ijed=mp5l+L)Q!ZD@=|gIvnDRq7^NIQ(
z$@yas8H
zwS@$
zZ!8nH`5fQQ=xt#;{>)U7W<2WnAvm#G}DCxwOlx*KubY&R`fqpHb}d3hxnhjnw1uLK*tqYfvZ8)
zy@1Dwef^5sHtwQWT6CX02iGUI78R(MtN!#Yfgh%Z;7_n?4U-Q2jA-(e+Mvf8M~u{V
zp!w0NHvE9{zu^Rx3|~~nRiet4^r)D&5RfK8Cx@)xtTm}CXUTDnIq;b
z8tI#Ig81Ge)paX=femC#uOlH)D0WKG*;tN(O=5fZvLo(CwbjoU6f-ir&pi9w7Q_9!
z%)ai0Z5ePt8aZ7`H_bkmOMIVcnY@yGgEo5HI|Y`sEpjv2(3$b8@7ew}(hL;OosBLg
zM)ytr;jjK#(AaC&{^D0z~xFKfC@^*%){P
zNT+%m2;d%P`Ltl&TFaznjq3Yk=RSPxWGj!Jj
zHJ4k332QTZ9tXx0gxF;K%O4GtWtSr7&ChrVyb4=*d#6gROsyFv4HNWyh50|$N5oCm
z-N3`6617kL1e%8@orfQ{{xe*@ejP#YAF@xF76kR%gAeH+FYoygh<1loV2@|$UBBi#
zSIVCjT#q7M%Fm)s(!@G$iDdZyIXcw-Q1O%C@EUHWbaGyws4TuBa3QWZpSR!Qj1z*oiyX!f!D{$wLp+OHQVYX^3k%d!kGUb
z=Z(;lSvg^)8DTP6=-Yeo{;f}5O;j8W*_WDJ9r>XzR!5RbN
znTmC>mRgb^_U7io3yrr42{U_~*XtH5uzb^oMuh82b>HoWze$$OJxHIXk)sNy+E~qe
zjO6q++t^MCt@FOytBZc#`uIEU{)aA?jXp8PF8r!2TiXFr6Jq{~(7eKJhvL+s;bBudjNBA1
zq-XLnt^DQhmmWG|E+saqlfPPGR5;Y(-V>p&h{f=V@3R2Nwt3k_()@RB;q{|@V@lN=
z@Y=4noN%!3I1@z7vx3M`!Bey`R?kVlAhG_J(hRr?)0_VMApVn7d>_--m0|1Vgkv-B
z5X|n!@oq4l-31;N;{~XsfLmfgG5^$MS8=&Fsg`!SzqQ`ui6(dMl$_fiyHobEdiZyM
zf`0q*CAZDc-&2!F&jUJ+?u=+grpEv&fhOzOyyS(unz1@ghPxFy(~^oEpPT2}Z&txx
zhj8>ifte7VKi9eZMd~!qnv;%Rk{P1f#vgRf>a}A#uBW|IUubIrRp90P
zUTn3kVVK`+1b(NEP1T&N*z(o9;1UF(M0kw!Yb=6_f6;`lpU42a_p{g?bqKFfg9ML?
z2u_I8BLr8~e)k+!`hPlVgRJ5oM>Uls61YS&Qgth0Sk$*G~Es@-?pQ&jo)lF|M4zn%@61PJfjyAo(Y;
z^Uoni7I}9St(Y*PGUDRi
z)h|}I0Te_FK&1@+s|<7f($GFMUhI5tJCvbdg5gX$)o%!2Xp;(Bt?qM?UYsTXULZHu
zpDPQb)L75ItDcb@NAeECpZtjG^Px$>nW%26t-N_Ds!ha^S*{?(w`3<;IKt&;&2L@7
z=~uYNE777mp8k*k$jxqz=EWWUZ`WVPbuRmFu(>2RpRQ20L*bmt>TOM_O(&4my*2vZ
zbT-EjyYFBn?_*RmAG0!%`m&!!^ovH=)U}MkwWmKCTfb>Fc04GsdpOb)lVm}W_fP}AHA(Nt8mmoB
zq5Zc)HZKw!dfB{@ijxH&FYu@7m#qEth@u
z9Ngm|u1;9+#Qy7DvVllA`N!T!ooF#OmA6+JQV`LOJrreplE
z2ArODAt}(;6^@8lS8r8>8*fGG!w%VM=2kfPv~_GQfa{*@xz1U+oEQ@D?{@>FMCtwk
zs*4pj|F%MmXZGLwSrZghg4xfHBTh%Y2*B8W99WTd!md+>^?L4vMkl`lUL=obBi+*
zBz6-B#9vq7i>+Ck#Yi&}{iE!G+}t0l=t7Z?r_e|>6U1!O8^zZDND;OkGo-8h6fyRZ
z11r=M{?Z$A3w88#CwuuW&qGu**pDvheP5F;%o~)1p8q1nAx%Oekh^#b^X79>%duXU
zjIP^Sh7fBs_WQNPip6&AHXPv@ciEbV?C)Yu-53Wn=yCdr2@~7#ti3Dyu_mBn`(L**
z3&>GY9^+Ysw5uvy8#q5G4R*#M>>Tr!FG`v$}dl&e+8k4`@KT1NGRpaLMe59
z1M6J=I6cbL`Q&VcQ0jm4$sR#o9*mb4zs;A3y{4lAZ2_}?3|wW_uhp)FtM774NATH*
z9~3m$yW)2<^kA~9MnF0f3E?K()0oH5V{$!fqW^!(P>ftxg32$X(Z7PB``A_A)}P*P
z6C(6i=bMIS4{si|x;g`b`QiMvo%Cm8it1THl6`iNR_7~XMbGA2?4A|+=&{1o~aENp5P!hAguuapQo_|
z!3_2tv~Hro3CNCw^OWbl*I(xw+x70}xk%nqf+xMabQpdtQjNry7!{mX(Yj3u?AY-=
zl)M-+-Pu99{XmrTGR^{dzzkqlM>!t&bP{7*KrE9ppewLDq{J6~cjFDo6MLj1h}155
z{k2k;Y}U5i>b^112k9l7i#Hh}{H*d%G~y=AIQxeLr?RXH#G@eZ^9B(*vK;bWIi8UZ!ezKHZ?bg*nlMvGtf<(8ON%h#&Q76-zMHA-
z9u%}%B;t(%d646G){69PWyNFaXj*h;j4VhTDD*j@JHTUKWA2gK>gev2a%#}a@OKo*QhuWSBJw-Iz9#F5wyN41RV{7
zYWQ;RZi41#J*U3SZ4)5hvjt*nyLf_26LHOgXOOp$QEhkt`U7gI%F5jy|${&Gf%clR`4e*s*nVUp@yUINq
z4U6Vh&|46Vt6{Q+x0FWs;pu%m#)^P&uw0{}p=j5>oj7Rpm{#JeC(X;Oa6z^cMYy5^
z%%Mf0Qrzvcl^TLaZfJc;;-goWE7ER2df*06xaH})`2HbMU)RFPoAnLb8-Ns@fDEPA
zI>z^FhR$I1CQ^Vp4XDZlC!lwveZCPWaJS1+uB;Ee$^oISBl&O4H3dkRbhLcpXi<_l
z(B>&HS40>Z-rkecGmXwM>=9tcKCdW^z-G%n>
z(7z|5{^vV9pL0GLq_%pBSPNR1O+2~MIQYmKb@X_wr5f>7z{p1(^*eaky;bfOXdY8Z
z2exU=I#Dd&|5K6t84PtVsR{XM8-{F&)j0c*|1skAFl!ztNTmQyO^rhk!5D>$!PM<|dSX(?6ESqAWyK
z5L-7M-Xk(@#TV(d%ogYQCReAM{sC_r}b72J64d5AkN+=07B(
z_$&drEoD+9Do!lK?C(el&FcDg%z8y0@h9Eav)CY!fyuYqRdbNaW=xtlGUnA-
z1azBEB=)}R=+pM>#x-8DlE3|WK>VY`56NhHj#7^I0`xzW@Jn2K_eftPwH})v{(>)h
zWFDz^MVnj%djKQd0ROF((TdO4OvdCLnlz_QQRe
zu*d=r|DW|Jw0wf?6|$#N)UNJncpNp!~HH>j)eM5SKWkTo&H@HCVar7W^bze
zEfJDs&*FySfUd7mS8-fm_VHJw^z!kNvMqSXaO@Z6V@QPG7Js}ntl`58!Q!>7PY>MU
zS49KBmDA(1Do4-t42{-2YZ~{OWTn(K=@PH90Ynw-%2>J%q`K$$JU`s
zN%`-z`hyj|Wq}z6gG7$^el;tX-oWFt;M1$9-3tWzG{}swTFj`SR{_I1Xlpn6S`n{MG29|P
zF$0`gyj;Q=+RS%Fjqn(st7-vX!r%tRY&ZE;bhc
zS_09T^08rxzy2M-&WT^yth${fxy?9$4*0;0;jb!(B45(ICWe1*T$NiUb^YOLPrQSQ
zEkvrbj+pWo=Ju)&SdGM0oSZ)Mxt#IZq3x@-w%rA7#t#mJ0<}$r+J`Rtd%?-rxOs1t
z;RJal1mu=l-G!@p#a1R5)_YdlPw}!bf*-vRZLg^#m>NxSm>ZWCPa`H;T|0{V-P1X!;PthL&%_YO*!7|HhE^
zQy%P3YGQ8>T;qKY|hjI}-8t%!6YqkiszjfSwp{zqqdVxZCfubToq-=xSNH
zM3K0I0D&p#;?hocS6Ir?oLl3Cfs7P@i}Zvf=iS%KFGeB=FkJR(1vVWtkE>UbY4%7p
zy2MsKYq&}wCyF3JP(20AJ8M|etgo^A@dss2yhn+c(mzl
zo>e(4?4O-QPKy{>L(caJvcD_D+?SA>lW?Y9~}Yn-oBrk?)j3AxY>nQoFm-#
zmQwU9#3H+Nmoojbxou-D_uUjlwBha}+y=$6t_}*Ly!7!C01bgx-!j}&C4AVZ9h(TQ
zr*a1RdPKSKGlQW`b)I@hQMYc+uZwk#IjMF0(nU?ZXSGp|h6&m)HPTMMonBybr!{6G
z(ON6g%A_{$CKkzNtAVIoY9%(ogxtKko=NiLHT@;S3fc37CHJ;x+W-pNi@XQA4*(@L
zqK&tbPj%k2KBbgeg$0d
z$8>j{1CoU@G}J4E@ncy^;&LfxyAdY`_zF)Rr|M$-0O(uweL3rWSj<@@aIlnFql_}?
zE*i6hz(tdNO3N7@%dV%G%hRS&Fku?oFdxEg>Z0azyLMC1OJ7@$J{sU`E&|TqYd&P3Rr1tIn%uUmzS}Cbi3nR_Zf
zvpJ~6bP}0%{$Q>#8<1aYaAY~H^)K3(SJ
zc9JZ4;U$$)tp9BGQ{zu%|WHHC{YFVX>i3)J`z%>+QmTj2gK#B1BDsLiTX?V6$*Cc
zy?Dgx3_rc91~)Z+qI|I{1OEZ5d!x}vUs|fJ8Z=AY!cBN0
zA~khJ3RQ)#jw?`^Q{J&qo*~%m9&F@tLr8z1l%rN`LZoO&F9)=eLgU4=AdeWoKeheg
zSiKW)$5i(Td0rTLb`meMzQtn*i>fDbc8#(PepYw-kKPhml8;Tp);56aMYiDY_@aTR
zbOPsx$dlf!M_3xtPkG)<;5&z9d1Uoa+j%MA5`;6o)L%&{fLbW!+~~V2rrHu98^qmm
z>|{(s+sszT<)W(~@c;~PH1qZ^^<$7hGV8q69g@hEu&skM+zK)e_}L2Dpo-}4H-3#@
zk^uui^Du8X%w~Ci=>j}eg$)jh<>HnLAQnl@ZQ9WRg_GT?P)t@fkMEaSXRPsh~II!
z#6Av)UpS8CF~tHd*
zoW`~wM!&u{2}@toyqj<@)Jk{9i^<55v=
zCSvCxH3gj*+LYqm>3T~UlH1Kis>m>jn|eEusJ#BidCf%YTdt1@$>@(#F>y$1AlP{?
z-rk@hc0ZXjcoR#<*Klfj1t#8n%nITQb2crZGkWW?JtfB62SAny5gYda)T}+M$`d<>
zhV`o+nX>=k7`Zb0>pMeF({5LBR!ICfh=Q0y?pTMu=~W!z6H!Jh;>p&w~!HV-SIaKRw#
z{<;#Eu~XK1Q#@n8jZ08cDbp~PBSQ!6|p=O*pQEi@D7x!Cv1z{627
zdU#86pmQLNqY;6@-mTk9QRN=to@HyG=EjdhVvV=b9!O&z%mqJYcvQ49o)H}vumpIp
zF=|p%a31w_^VdYDpZ#tVbA4*3lj*hxj>Cg37K(O_Q=s`UdH6jQG(g$YKu`S!0+i*>>3xpQd
zL?-hbVjzd!sX3mceX6(?#rA|dqA5(u{5jiCHWz#=cARy*rF)GmiryuktEs~kf414)
zcpxP*O3;RlX@QJ6LlRFbFU-o29baJ=^bX&Nj^a=I*+7GBm#8$l-lI1*nJWaw0w~|k
zw*Vv}2j|qfn;6zau8C@P*4jR^Y_e2ZW0Ja%Li?JxZ5NU9xW$*|RQmS2Q#ALVoVgHe
z%U{YVQj#v)`0z;m+x81}U-jqhmi#TLG`%rekn>4Ckhc=!O28&XBwD?@=YX#v^cuC|
z^&Jc)9n_wyi>BIs*zI!{+x7}?-*Grd2v}aut}7PihdjJ=y}U3^V;e>A9=CNQ==$C}
z74V38%ys5n8teZX7=SL8xyH!oXFcvea|JHw8C1I(s%EwO05Q@`Z#^z23V!GGY=D5?
zF@Ekj@k!;I1AoV(KptA_!?GAQdqdResZ1pSpm~na)oim3S#&*;^9OF4isw4dxgSdvw3zFNs9WEjhN)uLY68{zoQj`v}S50~~nKB^mQ6
z4F7|vj;Wn<<&$_)50Lw7EgitV7%B24dlc$j%9)GYI&mCs`>C2RN&%*o1YO%eXPP#s
z?&1NsN2bD~9iL}@k&d4Q8MXD29bc*D#dmsOtd2U0=K9$pw|LP96daaFzxvm`PYm%F
zG}7BQVSci#7xTVGca>3}u7PX$!_~mUHR;zWh)aD8!or0cPkTs&&yUW~R~Sv(!ZKUm
zvFkjuBi2NHngfrF?V(pmy&UtO2N|eRj%Qokg;h)jMtN=(PApqB;|-7(Z6Rbwt1II!
zWlrH@QvsT}FwJ0iqaI&PD7!I2F5)ppR7lSY0Vnly$ha`{a;lCQrlW$E6*AnRxv8$bZ~
z2(I}&<~*Y5F%~N`=7otmhtg%_MIMTij|sJ_3Zh3z62Zlz>r)M**YRI+47dRO$T+-8
z6;R1Ct|+XA`8F+wLSyq77`-H3UwHH|ZN*~+VPF+gRQhEPm~f(?OgN`IrSe*WY_a=f
z*isj=SAmDpeyPzu4vj_B*XJzc{K6cvlYM2Gagvf!gd#iJ<@1(3$(cBMDF#UF??qwK
zXH@@Zlls6W&75g9>*-1!aHJl}>bljw3dNdS!l}$d37Di8VeCgz6fNG57<65qvuE7h
zco;;xh?Soz7*Vp>vS-yF$9>*irr~ZhF^NQ>kV=vgGoTV@<3c!sB%A0>DwuNxX
zm$J2eIjWh~nXB=|q2GPyaj5?h%7638=!l#@|H0GT+h8tY;uEBA-TeD7*NnvAi%H!6
z#J$ug;3>)5zsTNx$&17}-9IVn|8$cG47{WX`f7<9{NycGi=&4fHOlLlr;$pvh5sJA>Kk*t3dKDcMVw63CTd
z@NZ{VeS|>EEO0iGp#P#+k!fPi>4JDRu#GeIK)RUZ(8np5s~Hk^B=Ol#I`^ykRf_cz
zvimGMKwxqSdJ4cotTpRWgjw`=F)q&}DD!BYf5;#a;aF+wxi*7N6lf;n60w^`Z6%15
zaJ*Ip21fLT!}YCTzv`K<&6UbGWB)~hJ
zBNJLUf!$|0n09}jo86hp=Y>AHR_<+I)WaUJK{E0HnPs4PVF%(>`EKZ!^c1wS!8?~h
z>3X7_r+W(;EU7%{*vt|0UT|CZ)~I!lvu&@)-KebktY{4428=9Oh%t
zP<(ijxbdqM3FgbwS;L9?(++kg=pLECyNOn;e5Rtj}yH_AHv0T;3MXWWNW(;|N
z#>O`{u;Q=o4yIaT|dEm8R|8aC#*;TvX51${kwWey;@ZxjJ!nll4T2K-U@nL?seR+RJmDOze?T|KwzPKxY-t?-g>D9+#Q%>?26v(xt5I2LvMFc
zyO}!YkEgI8uGLWF3KyT2Mt}+DDLX2cU+zJ5Z234{MyAQpeivd5kSl7Fa8S{^&!HYzjCoSp&%tiyjXQ_7
zgt@n&Lvt`NVz@!+z@`5ERD5~UXi8xni0leQ_`6If6wfZ~V_$uQ#mq*w-K9~~mOSXH
znp0HMqfqBDjBUeRCyu5q1*N`a1%Qp
zfaB7^0wcX(Fc%T6iRX3d*-vv3AM*4mvB;Tfrd=6ol1-?0Ef40>OTmCta}Jn2OcEii
zR~ok4zG~E#gGfPrgW5(6h__{-eec#3Rg9D3ma>{F-TY@3iGyk&LDF-JaDrkB7S$1S
z{Tn*l}43hHW74AmDTm)vz+bPl`=3CvPMg6EjTy
zqJpENR(#@i4vd)YsN6eRqcHL&!cLa(aO?-+PG>}iky<5~=)97f(HP`{rgV0(3xjzI
z2dvT^ZErEvehG3xXisZen|#m$)sV2ymG=*0mC3g!V!frdeb(5(y9sViFy-E#A=s9f7L}YmaB5q|mW|33Q)&%{;BZ#Kzs*G>a6q6N$S6nuK=b&n9L(H01GI*jZq97aj<_CA~Hl(gOACRK?%dKDyFo*_@{
z69+B>JT>nOr)-lY3JMWTSpcnG6dd&YVxdJJyBT#b-jE5fQI^lmz(-%ij|G-~N9UGI
z0~rcU3U0lpG~9y4Ef|=3?k4+m+!DJ>GeEn6Hm(maQ;s12W~RIi15iYN_uW%VZgY7K
z8EA#gxnh*dJv{H6cPskY#>ESYV}7&shBmewRJ({`Ff8jQA~cphWE`*u+2jPwMoaMD
z$c)o8Ij*oZYdG(aelt}j<#2*=k6FGo=bz=SFg=&4-tTec1^t;0eASD@
z&0M*l(N#}XtG47^+hGl1iP4IuhGX9^%9Ya|x}p;AfT6k|8Tz2s>35lSxYpB|c;s1Q
zxqDU%jOSM^YU5y-)Di*Pn!|Q4k}w>wM+?Yd{t6%sC1KL?Cd@mP)S>aXz}OIUkgEt8nq4OI;ZgxWQ27yV|62nA9OCnRD}RHUZ?ej|?$E>aRZ38o$=H-yb57FFGk=SCbz
zBS=(-YDPmG-W5bOf8-dm$oev0WU&t;XlVR$Vg-`!40v^Up=1bzE4WXUESK)QmaU@P
zn|xm^ap}eQq⁢?276EGE>AtXa@nFE7u@Jq+*ZVZ1?!b2fP|N^CmP^$Y3;S8f4kk
zx!a~2VjHhM!CY1EIiNqxJeMX4QGd@t>4Y436Cfo3x3ka6>p~`K*7a6sKAj_AkBoxh
zn)>I6_E2FQmUzELNKlO*y7Z>C=7QY7gSW^4^*#0+|8A1&1Sg>zs2y>(DKYSWU$6@%
zC1$04Q&e4CkXUr-AFzBltP%$X%fy&_30R}8aQV@Bt}<4GGHx52RY`~7x>bPY?bBDh
zalmy272im6fOA+&gEGEtb)n%pjULcOF{!<6l={XA!zj*t@?G
zKy)5^sQua4i{RF)Pmev`@=+LI{P-SBqS0;65yL3zO%?J$`?833+N!OpH5scy>N`1g
z=11h4ds))xRT8a)x;Wb*y}_|AnVylico3{>+m3VrfSIN6=eUXlnUwVC95X%YZ_1e=
z$nKY)5m;bbl+1%8hv|Gzf5)^yP({m-5F~}!o7@33r8Se5&kErxHBI5Ge#H2CX-Eeo
zM_nHzg&>)c;_Xs^N25W{w|oz)RNz9}VHF`V&%6kT#Ommme5lnJ`k9>_t5Q>t+Ybm{
z-)HpKW%lz)_33hb!Nz6+DpgldXAV&ytw3aG-CUfM>YNtlic+8>h!NY?($djfcp`g|
z2itAaZSYCq)}(-aLrxmIL;Auam)?`=%+w`@8>#26=M=&8Za5wI(P+r)dr41P6M*k>
zz+QcI$3Hg{uGYP_!yZwD*V@2eUI!rDMHuj+rV}t;sl11?@pa|8d?=NrHwD*-womxC+2Urq=Vc+Z`T4=hhb4cr^*oma*_3h4F&GENCc?
zD<&=Pb0dgGu;(L`kqjSi43R_3uEYV1@XB|T0m2Ja@b<_j$7Obcn0Z8K7c;ib1`o}+)dZYy^b3A<#D
z5}GCUk}rH}aLN01_pVM3TbV&7<@zE$ab$5Zj%_}KQ@)A%HPszBN@-M`A{9kQ5Wt6~
zr}DhF0HtVuAhc7=S#nbix}qIxQY@@UH*VtYh>^k@6WDe>1{7>Iz$XliVteXsZ3ZueE>nW;(^>
zKi3jrCPFQz!TqUjthG}5$DEkGOSpLdO@jt+k9G1wL2OjtDcK&i9JtDtC9a2;pb5>)
zER=xdYFeyVvGXdsTjgi`Ri5Y}!iVng0~CUziW<}OPWMG@HVRh^6h
zeO>*cc!Xh`6beeA1XEvs6rkcJ<-6IvwVBhG!I~gNxnt4$$f(7ztMx6u(nXmVxbeN!
zmU=)!!%{liSXvAHG~|}i_#5RCjCYOGqqFei%7E>6r#e)|?esrLW1a;wKi}02LMmk9
zGL_;mqu)rTL*H;6DJM%jyF?s-K~ju%Hr|>2pcwFB;v8RZyPA84U*~XPxlckCeX$md
zvB5HZk|>ljH)80HC96;hiZod_`dJUO2V*Wy{O=52&d%CYPLpTrkrKe|4E*;gjs#>R
zukMV%0TOg#>CrLhOyU9cWzU%^;9hy)wf+^0lrSTc?k8$oVujeRwYn;VHsMP*
zR&nU@=JTmHpX>sqO{QI>4XmWVO)djOm2YE8KEEt^dr((
zwPC9|;RCQcSQS?-sTW8I|GDRFtcIs@-Xz{+Re)q1UpZ)VN$|j6%}=JV5x0*%eko5?
z!1Cr&Pub5=%f3Wb;6uHWxY#W7&)=H@qtL3LI>M0%G6r9(Z7uspZlH#2sYJ&qL7`;5
zu$pq;wT(<$UP>D8-7Pt>RmF;$xQi3iOy(IQw$o|kD3s+sD=J@~qHtmm2BSMM1r>Nr4SFGpZw$1*Qm(Q#)tg
z7i8ma9xB*JYyc$%RVN&bpT#*1Jwh6C>oczRH*Z|>*6cC<&`^;i^So%^HquN((Ca#2
zMni1~cgdiN9iQ<&xTjm{h^mPY480U01Yx$-ugfIery~ml|t`%
z*{!Yne|~utT6159f&NOJZ%#WiB!wlfG1^5fP^G?50y~E61`>Z7C
z<&6gKh1EIBjz!Y#B4twDk3MW5(1O9(7-@?PF>tG+oh3dbyXjwrRE!yPTU}T?hvxqf
z0tZchj_w$iVJSZ%4d$7CdC8-SrFZ0c{Fp&fbiG?BG}>h2Rh;K^3uDwW*uu^r?GO|L
zAEGLP^{mM|54x!lMm0X#l2u7g{bEPC44s{LXKP}D@Au_0fb}afj3r<5ZfZ9Fq8Bx?
z;i2{Zfcx!IW}IPSL!+dXWK5M|ETh;?-elXzhSa)$xu>sYza{dz;mSf=lJNo4=!}-~
zv2HU(j7)C=#jdo~y`=K4AQ<56j^vf{S|x(GwZxDNLA7QteJO4J0c72*&5cFDjtk<9
zUG#67{Su`6_No{D5M;DdX}=Ze0z>&Kq4cWqbU;N~qKEu0ZQKs+iVcD45%IYm+HGT#J@m&Ev`+~vM+wB$t!>;nQZM-3a%TOksK+pw`GyG|HR3?s8G)Og)`@d+UMc
z6u}-Yk--!C&em>JJ#BaS{@fv#!4b-%8g=gS&6-%P^@LUUDwKSGvjv#+>+~RI
zi($gVm4c~Ht`|QUzC2+omeIkGTK<$#>?
z>uG>b>Y4KPhVd1u-WObkpBUXZ{4IzNk6q8@>A_
z4Ja^2HFpiyJ*xLYT`thvV>GnO?+e-{l~%(54W3kANRozg0oJrHy%eK?+h3&mv8K(W
z-qtE!4U^-^-jm@2?;&%jz5%{gQJ%SaZ~o`vFQ>1gnd6XmE9$qR;s@i-WSWvQU8?g5
z#!NDeodu(@uc=(#m6DkEvAGYttG7%QZAIkUN=84y4Xi1Y$Bj{{l)o{BtVl%mC-j4X8#m4zaZF6F(Snvlp5NHJzws4$b6PTh@lFM
zJx7^za!@D)+q_k0fbvt)R(cIEAh9)w$a4+q2TgFxTc2<=-P`x{v9l^Hmjke+MMd*fBD-1dB
zt5w{q`CZjm8`4_;Rp79-m!N@?(DK8dO?!$#C>kg!r|c&;9+g*!tf^MyqPMg*=nJtG
zBf*E5^AsC(PnYL0nb}jwwo<8RPE>parc$Qa|0O@
ziu>JX7fn}RtVBw9OZp*l)E<{x_7ZC+3p~{19{EVFo>hz>Mx5PO
zL;3wY%GxKKXe!sUM~}-OMc<~wH@9Vs@(Nhm44itl3imgs!r&|1)q}97KFLpPlLjW
z`V1=qB}C@+?B3{!M*qc}5|akqC9twL!}Gk5beqL)jo016irmsiy8=z!UqL
zdTxBA*s#TWBO{rWq^7h#`TplWlbY4hp5!Tg5w7UK(~w6Zg##^OLyUNo@(;(6xHA)PTT-#wloTBjJmd^fybKMgCEIt*$}gxFkdxN
zXy+kY%1z5@JTB6jd%y|LMP8yR0b&#p|J2h&`K
z!*r%8wJ80gPVD0$*iP+JkSC1abzDbFt9NGNp6hFo7bv3_OFt)Vmc|W4W|>Nfe-zCv
z0T4IQ^!md?rG4N};7NKO5+>;=)1u_Q#ynBa))ODyZ>O;f;?id^LgJf4X`RPw-FBY(
z)^tBYW;qhwVxR`WO!ta=gYJQ}Xjri~YPeb)7~mRWJIe0a5U~m$I17n)Fu=lf1S8VA
zfo$|XR#w`iqWSkOvQ#~p;N(db&)XkX<1DMwvTpL$1JSC+o@Q{|9oy#yKZ6|-fPgkz
zkcSUXlPSp7F77;~_oUJ4h)U2b2>4zkK=e6gA(kBQI>v?nw)jq68FIFsBbe+PYD*vQ
zNiZj(lfSAcr55P;U>`ZJm&!y95@L>Xe11Q7b+jsA7ADn<_-1Ruk^A`rm!|Tvkd$4Fi_QSfEV4CH82Ak|r9=ius7Z6O&92u=F!
z^Vp=-GkVU^hjC|Ky3&4
z`0s1CWni*_YI1$k>PKG)hLS$IB8tJ*huaZ_DAAzrJo5_7)Xo9Zi)hmJol(_lh}Y((>Zx9pkvGdB!IF=3AN-o
z625p(fv>}xZCuO9(@ck34dyl|16@1sU=cG0iK?oZ9L`(I00n%>o~${?Gn{Tv*GF)Y
z9-lTokd!)GBzXzoGV8SA=c`&Qf}V#jniq9S0%nCsGu$IiXWMpeA!#+GB323
zR)IkbH5tS{wMolb&%u0c%t6|luYH1fw^%^nh6t-6y6ew*Ww#I^x#`!m(n)ursf@U(
zm9YdL&oV&;iCDQ`6&z9IABLCc`<#B!TC<`*SHOS*!f8VmgBTE>JpnVelgm?XUT0NM
zi8)wNy;)NAt}sg%DMkEV(;@$wgB$f20bZ!oUb%_y-x6i$?>4
z=Jw^}T~VCXd|!ypv_^iuGfAONNz2g?km+43(ZE0N6E(g0l-xo3$P2WMU|t^ndE29)
zoH%d7En8a@Jye$OU;p2`WZ-u#9HkHr!7-No7BqD~vJs%d@I*J)@8<|Wo}!G5p;@~8
z3GyB>Y@-)np;fTnX%_Yz6O5Mq>(?zY5ekm8nwTurNFT5wD{{JQO>kIUvhSmdhZw4L
zbZU&5SkRK#hC-
z#6CvL53heALkswDIG*f8W794BI>nx$MaH=zlr#tjK{1P=Z4l%|4jvMgT^MKk*AgRF
zG`PaL4~gyV4}y|B|YwSNCYgv2k4l)Fi-wxbK@E0!>ER$11-(02Ej`N#HneI
zTRIz;B(p{>}(h*^H$SVh_)
zHq)D9-;-!6_$T&Qo5-MhKEh<&&wwjgX}nXF92NK4E8%b74x3P?-t$`+jmT|t2!*gG
zsM)Y1qpdO4x9_iwTohE{uSN9Ku(-}Z-+kbBg;;l=W85+_zFBqeRkOM+3lxJ8LB=vm
zZa-dLgAJQp5^-8q9#snot5?7FsL7?6+_Bve2;r8~w#
zLPEMjq`SLOR6>xFE@>FLQ;<%PZYf0?r0YKe_`LDG|61yr#ge)2bIv|HuYGOXJ-Kht
z*sx1EPr-xeFN_xzJ=%Gtb-53|zYrVPmpjGrctWfP`DYw8n{)Ul?`>!ms@C=zK?G+q
zq@y7FY@j!3K_@}jwFD$UMzGW3FTd>a>R{e40_o;%(AUE}cJCzs!80+iQePD&!eAUw
z_R!Y<*tHdmq6=NOOzch$>39?SCSH0Ar^i;UWl>aM-=copS*_*nR1ge!ECBu9!!gE*
zgBHKtB3zhR8yAU)#t~y^b_cI&b(Fqe@v;kX_if|3J4XWOGP{U!IzZl<9x?y#w+&7!aK(ip-5J3pgfF5M+f{#RBoeSNRP;7vFqX3Or
zQZGf`M}6}X_U|_sIB`5agTJ8cg_}&iTT^6qG|$sj-=Z=kz*2Fi&Zt_>(&EzBE5PXB
zTSPE4f8h8bBqI%7{)b{xdv!1GB>K%uF94_*vLK|k;_(23DT`MmvWTnEim!Lk$Zr)T
zMFiw$V>B5Tu4LD^lo@ic$=+DL`qy)8k#;{~d-)2A*NpAAnBw@vx({_7I{
z)gLr?&XDT1;%azny$%b-^?Cuc^z(FS^{m!z%j=mp+sx8U+QA$=9z3&b|tXSnLAPtAYcDy**&b3S#TOMn5dw!!E&
z153}^2cES?wsb(dUZYcUt+D>&Nzu?pZH3>sq}!vRMk{bjb!^jhq%uJQF{UPBi?ob&hF&ZEv7K3fw)iApWFGU<8!ee_Cx#u!alhE0K>fZqp%@l$WUC
z18p$L`eb2mdcVbaU9;>n{~9wb=wizS%BipJ`W62}3MhG@>^iMg!}!6*n=cF&w5kN!
z9Fe|BUTXTV=MeRYCp~)_@xLa~AV28flg%dW{yv3B^Oz|v!ZGio?#BZ+-Nw5LNb^Bs
z-uA_M^fBn>i3V)%wGSQs*Vd&IH1QKHD9a_ixbWbSKW=I|3y%BGGeSq4QF~Cej`#WK
z0Xb@b;=FXzEClZA5!CfD${)stinsCeqNwHc*B$E92NHUmmJ{67pgpQr`rj1TKMN#F
zL~6?+DRi@oIp(NnRSjmhp=M#dvyk~>;NjH*@h%s*{SrR9?9sC9KV1|sAYs9CN(56v
zqTPQHsG2Y7V#9Q6R4;{w^ZWDZLwetkrI$9f`-og=T%^O=*b6WLh8Ez}ODV5w;Qw)x
z2N)2Ht~B6o#VmEl)E;qj!jrx8NNl;@=8r_pI00*Kgb@
z>cvX`4JY@K^1!e`mG?-Nc`EfLO8Drj3dZ<*N@wM;>Rz3K*e`45u`f~OW7xD`fz(~u
z|J(!l7m!h1?|Lo!UY>K2!eO6IaumzvZn|>zzOzTYOg!$^!4VTneg<0U53dd|g$!{^
z3rA||KBKLt7VL0LVsmBS>D-2B28TK>zUvl^q{duCGG6==k0J6bx^WcqJLta}$aO+}p
zNH|BTp^%YYoqKKELomCRAuo>
zNQ?oWx@lNvMP*43ln?AflluY)i*PI(%qBjJTb%f|>`%Lp>J@hUQL*{;iUI^&JD^Uu
zSc{8*y{{qdczOC+jP*B)+e3@nrv
z-@ba}{K&rk6K(~2Joy`$92|%X96r}Ve5GwlD0sp?ON^-{;MZPD)54XpC6y*y7zb9c<>XB166YRb?3E
zv-m?SR?!K8o|QMfd-r>m{d-4J97v`j4+#(%>UmA2==?hfUwkd;&^CSy7nG``krQ79
zh~+F>n+xiubklt;FI&NzkH?<3w;IJJTT_AyD~dn4+kyM}Lm-
zN{RxD?*un)!!%~$vt-2LoImce6yHoOAmzJ28+NUsqW8>@Lcfw$~LInc@6-A^U
zt7^}~K2cZ&#LmuBv$j`Z`d&HW@^_&-W$!-Kmh<+NTLOjRWs7aaHT=RgngBYGZ2KCM
zGDaM`3Umyu1>Q9ZtS31w!irj8W|MYkeilsfh09lq;eTjM?oUX4d`&J~PmcU;8QZep
zFWYYk6QfN!PCAy_h8SqQhFxvgTQOgB%QQx}(<7}>3gCe0ZwEFJ95ed>k1ym*?#T`71jPm_YJ@Oq7fq;#W-c>L%exD=vVnX-w$(Z2s%?DEexfFS0OV4)|>a+dvG
zYt^j~+U8vB29p$MMqogMGlfnM?s&@R9VpcPnoD`MPRs%9xfoEaf97GK`@-KcQy
zrK`p>@?(E1RKdg7UZq?7L-#5!5T}v%#}Kh0
z$=&Q4UB^qG4g@-@i^pnHTkL$zTCS2J)_J6vnHkz@rA8sa6k<96%b`CGI@b}9(@N?8
z=~X<{0~5vUK-XUv0P(PqJaa2~bQY6+6i*gEfLY3sx+LeI{)y3(TS{Ylp+=w$75jPv
z(+L^$<4r6x%3m8Pu6+L`SN#x`q#(!H-j1YjgYiYx{`8;18*@XMJ_
zMGV#b;A$Jy1pSYnZr&ACLJWq_DRe%W@v=DnzmDifSv^pJ&)KGGdq?uXNPW*AA!TxT
zUp@`BC6BjdRbY0&sT-2pJ{Z{cRvgNiPDBj-o2pr&me1tHUniU3<~lzuJnB~ESk}UU
zoDJJ+|ljjoVA))4c~
zC~u{V)0h25;jf==8>k0sTPAdKz?)O9zK$Em7qjf032uR9S;ei2@^B+Q9Hg0Y63D1Y
z8if!>|BC|(kUzno*Xv4nl=sd7&S4ljDX>o@x?CArIv(w0-@D7g6S&CK1G=85K-Xw?
z@ErjKK+j0$=;&CTwtH33{P#->Bf?j!bt=MaD{+DSsQ7FmxrV^XS|uGN(^AP1fTl
ziQ^`%v3fsnzcnx|u;i|-de5$GKv1P%?UkhHvpam|-Y#d{jBzUu2|~Pz*HVE4A0OG~
z5mLP|KKLnwR@7i*h|0B`qa?~8NB@1@3arT~^aA^-4JUufz-8t?kU
z011>mwj5M<%hZeXe0!2agbWdmp8R1xs6gp{W9Qf_i>n>#d0{}PY%~dt8RsoFtau$L
zV`u8LvJDSW1PgLXHJFOvmJ79g7=2|O!sz7ZdAD(TtCTF$%Cj;LePwcFA*v?D4;B76s0@H94fwY7#f-sr8F+N6y@0NkQN-xonWI7O|u1C+2BBl{%-*Mk8}C&
zAI$uamY?gs;L_>+Zl;p^@Kauy?_{JvktgBPBY$BTt;*Nm;P)V_b!in_r++~B`1lqK
zMz{|tijn1YsM;V`gv4UghP`X$5&O5K&VRcEpq`Bi2`c-j$QQ?G_CUp8H=|+x#*gYq
zu?Ef#Wn%TDh5~hk+)qfiyx`nQ6m!c@or&-mT2b1izYx{&r!@KArQ0X?!eig^W~V($
z=QnBkG9?cp=l?fx_-_jLpLG#DM#|%Au<7A+1YT4UnZ@dUC>ev#lz`G`$dT%ye*88O
z482lW%qE_oW2%T1m(RM2aagWN6$!vpLB7+^Odb<8YbC{>pJI>W
zyagqm8@U*@R@h(xxnqCQ`hn-kzDz6_Nh?w7v_6;#hqK)U>B?*#2>ck9RR$-Ti
zM_`tsK(jJ>+IdzR4d)I62ukE)0da-Y$^QEHH=DQ_5(rnq7mDBE*s%cy+Zck>_zT=5
zZ}H+3Z4Yzmq^k135w`trJ5SAS$B*(v6`(nVu}xM)^;VcNYgd?3uh&+5tl{m=FpM{1
ztW^(6kXaSKRd%&~xxP)uj^wY_Z|io&Js8PaBcdpbmYY3!6kS)TAespNCRA%RXa%L}
z$>zK%e%Xh)^->-E6lO2-rSnMKaTZ*Rn*`{q&on-n*IJ$K&jtEU^66tzDjtvVeCDWJ
zGH`9qqzUwCMAc&4L+(tCoQoTqE4rmzR;L&L1@zsF8KofkkzZKY#INLT1PyyA%Bc7@
z8w1ka%jRd0A*4%4>3#qHwglw^
z#pw3kn70ktqUVw36}MR1ADC$E(@3Jo5pXf6e#FY)Q9)i&{b$0FYlR~_UH%K<`**&V
zUlajNq0~5RJq+2k0fuqOB+#}ngHC(FR-|CI5
z#>Q_(@>eZ0*=toYx;#Hx(D5g48>~GTm{a1nklG_`@*rK_dCuF
zfiWtLI~h(XMSA!Q`Q<(sdVW^!j6FmBEJr|FrnYF(5}*<6nlb1*-q9y90JjAzsQC)X
zh)jS^%MPeONx@W%DUi+K1RW>?{A6#Pw&{3qv%k!dDXf6)dHSdPgNl7z+<;TDL8S`k
zuepL(&v?{U9$!ZYz6{VJPLno5iY{`qXx6TppeJ|Q-oMCg;a{RfLd9D?dY*z*(;{5P
zg?@M@zFW&X1-B*Q%j~eVRNeNs*{!whSzlhA;Yv@rnXLgKA=j&W=_er4lm5aizz3UH
zT3x%Fu)^Gf7LU%#J+Jy{uRISar^}WeVC}G91QpJROquWm#ExJfQcYoq>IuAhnJiY+
z)2%|YA@z2Vne~r0^r&A9RhuyzZnMloHz!;!d|nUxAwVK`z;JR8Z)nHBKceVu-puT{_gPxYA6Np%sSsVw``ePfJ{+_L
zSj@fQsARPDn7k!wbTT?E%py02SK7FU*y{4%40$RV}Z1%9=qj4S<%{*
z)3=F-r7iQf370DZqeabY$I7jR#y86v28|0@S=jy>2L=q^N?k3M{+ZKgk5gje^H*;t
z=_a0<6ZTfvOmyByLLJDFjY>gkZl5h@J38N7uQ6{~Cugw7Gm^SH8d3z9&-fCP=})f~
zuYjJAPgLEK)06o7-jy{bhehYbb<9R@G`%rRU3NAbpoBsL3lYFCL^Fm&snGx-A^@6)
zU3%rft?L8;LUQ%u`+t}V-OCcZ<^4m^z6xR+c@t?&>AK5}pDbp@g}A4@Vj2y-E7S#i
zesh6^V6MY+hEOiWt3tsO8XqT+%(9M@W~+V@&8kTp{o4xrNv#Qr__xB}^u=pk
zgoWW{7L!{0XI>PfZm)oY+7qY1hxRvk9k79rkBoLRkG8M*FP{&
zTOd44B}SfXianLQyRolGtYhv3Z%A>+w%U!Kv{=<_#&u>}XvgSA>2GTOkqJCdTMgE_
zAer;b(BN%+xm}Wa
zp!qKc7q>`6>#cB(zIn^ym(SdCx`fv43v)AEoPoET~p_^LhY6E-Jp
zkubRBd&;kS)5Oa^QN+Hk6?&Y;KY_HHWSl1}7P}9h-M#dSBF$q%u;qK-?A*|-Ka=kD
zW%~|NdL4!_+R?{$jeuovgBFqNekqRB_<})G(0PM}kY4sy!n?T+r{6Qmy$fb4dBw7U
zE?WTqqx4nUU3M+158s@^SO6S)lFN
zEWAD{fLmRByCp)LIWB*Mg3d#ERvrp(HV!@3el3qpnZ3?5p&F3!SNkCv#`qf91cqGEf%!<`fLmzx!H;+{8vq6-GQNvdnf{~w;Ot!J
zv%mOZVBFoQx1jU<*RX{RK`8uQ_hX<>#R&?(W>>NC;wBmKwK5xtwb_6xD(89wm}LeL_@(H2=p
ziWdhzu5x(~4Rdl$kG}AJmzrJ}T3elYHP_^82Wske-E@e|k-Q5
z&<7qx$qa6%Bb`M!Z^pgI4$qv?XDG(997x#WZr&4CyXSpsyE0Q>i~8+v=}Vj*RPZS}
zr~>cY2k`NO?wyZlhhVw42IebuCU}3~EAqoT$A7CM@oEvZgM*|v)ntqJZ
zC)oJa{UKPcfcT*;otW7WFbI+u4Uta#*qWspOWRT5G07e*0Y$qmz?#AWM#)PBe!EJy
z!yvX)MGh+pm-#+Gtm73vz~&`~U9rYcCi8^+o!1_#8DxEaM9A@@^_i8i>2SkqowV!F
z7^GJ6b&6%rUB7U!)E&)9d*DEoRbXH!!Y=$#R4?R^V;pU
z^jME-ky}Yw;*(?ypM6j?Oi*@9S;*r00cHQ9fw@g@^`pI7Ol{sZ%^iioyJ#oyr(>1U
zFFN1fv;(=yc|+AOK+lracz0@FgZa)H&%=;)F>_u)>g9?DDm@jUU4=Y26(6OsBE=Bn
zbjqwhOj}=PqiFENASSmzi0qFIsk;IJNz#Os1(`Il&taOQu=Y$ynB(d{_R66fV0(15
zn`r(Hv0`gVt$_#GbAE-0rBIVDGXZf@r+jv)t%HXLvh3SnGT-)X1taEy=-Xm*|
z7t?Zi@9bmA^DIVYOHW#Gop~oiZby$aa|ititVB;A;`OyGJ2tojCHXwHav4@CPH~sT
zb+q%ypWjJE%kv)evz)!>x`+=yl_X5{hkksF#O#Zp>I<;E;aoDZ^MYsOT`|r($46@-
ziw%#9%(i4>a4J0G`EBp3o>$5Ly6&rFMb3wkalCvo=44}XyZA=s?7f=xv`M-&-7`lX
zUnl&@hP&P^R$52tdViC2bUsnAgHszR#4ixqhY&4jb40U|`%ZQ23Li|pxyTi>8F5-q$?lDNDL;<4eIsX!PmxPSM&syv+EXk9gH-mZ
zDY|JUtuGVh0Epd&tnW6uK1=DSwoIo;bQ?`Em_Odesa6L6Ur|++9_U%IC;(Br(hTz2
zP|#hFWY#heE#SwCV|jN%xDxx%>-ep%d0KOv7HO
zckO9L)$rHn+~RT#=Y*FdBlWVH6-PdH8;X0JHlQTgmBb6heeZelQ@Wx
zPJBv_r5o|Jrz!syn^Tz3%O>kG$}C2;W?WYkE9*w+Y7Cw}_t32U6-u#!6PmTqg_4Jj
zV|R~J<=(0L0#}H6EP1-tzW2SQnWlFx@I$WyvT2$*f~L724)jR7_Iv_ff`qb7i7u>V
zxf14Rn3SKa%ybkZ&ZY3*Cj;D^qCY~i%b$Rrbh&86I9lblopbFW76t!MOhF+$cU;3d
zu~6Y130FXg+3>!#@vooT3PmKHU4gcrCQH%2lZLi_|0~+|zCO`JtLpIQli@RQ3#w<@
z#a#3MxHe`O!ZzQyW4UaLHNBkkP4eyyu*KOf4BqvI7@Ol4I=w%7=9s(+!en|q&o4!J
z)$rG?#v~Y3P$&6*a@nT%-}e?TmB3B*AbiEs!+mS+(m#~tAjN>_B_WOwLjduV1dvbn
zw{dKsn`q7ydIx8|&M)T9z7-ohLzGq%7aaf|9HEC{djzofHx<43E(80q&FvFUgZy~8
z5>pH(m+c0sQP#=+dgUOfGU*?H3(S8YN|a{IV)QeyR(RhU``UBOv>)Fyfj_3QxXY+Z
zw$whVMQxDI_a(XwR@KXFS_T;oOv<_{nK|)$?2m^|Kg5-3A(|`z6;0-u19<)r&gENf
zEvoHper}r~-xnp#*Y`bWM|jdn8t
zP@Ii@__uf%#0BFv^$2C{{*je9=pji@0Fi(d9>VwL5l7lYs>2fgQVZF{sq?nMA?-mm
z8!Q2!qi#~;gz)|Ee@&nyrB4O6WdgkkN`c4|kca0SDBPH+OuQ*lk`)uIb6x(*)`6Ie
z4m7yFGT5c=LAP*bi^#^FrT*2&o%em5@WE@iPPU%Ot}@HS+irYa
z5PxuW&P0T;+Bf^Xgyw{n@XpHmUw?qJJcgO_i8!kDzHc0&8AwKWc`b@X3HgjE_}J(hPS
z1fSVe1Zi5$>Kw0?za~tSz1{-}G~_aClb?5rdGUFJdqo6Z{K|(QR*s7M?rA)z%A8r_Ezd`g2W|)4@9N)T
zD${%pAK(u*|01~$Vm1rEh)z8RWG)DI`q+m5ZLerj9u9cVWkj)+!`}`
zRQX{MPGbHa&?_v)sqe{f5Wwh=YtGg;8oe!woP$jQ#@)D>OMKR6jJfH#yKa`75lRWb
z?|KXXcrl@J{~UEJ(u8oCXDD={6_K?B8Ov3z%zq>h?8Ds&k!J&R8-;BXZS0fpWR3rQ
ztt0NxI9D$>LRrf9kPf^mY_`EKP?d{dGM=vaOVpb0O-S_&K7v(K+~aQ<2lf$T(d-0d
z^jtsY_0{~9EFYLeZQv1Iyu01cbjZrDceDWD^nS@eKh-S3rxE*3KXPuW_ghe}fUVG`
z+U6KQYSUe$-*T$=x1>h-4LY!&iv1_URa+5}wxoD2c3sA$iAI_bDf151LmVmZ4MX0v
zDyu`r?fVFflG{BNYz!aJc09aw@i4`)_sbVD)zWaQ6moAcEm}cU^(Wf5%U3{#xICNF
z{(;p8n>%UAdUW;GutAKJ_1!*XDP`BZyMj`s#4l=BB(uIAU+wV_1i8%J7An||6?PG|P53`Ti;VflW2LU$AS2B%Kl&tavALq-6bCDWvAC90+Ym@eGWFylJSKzZ9YY#;Y(Q6)a{1iC;Y+?q9R`8dpmtc(1W%g5}OH&qOz
z!&GbD#2<-RuV7rZj(PyiwW&$0WN8rF3Bc7S4@joXaFeL8x@ZT#`7l`|0(h-{wZ$R*oS*1CxLv#)ZF7sJobl_M_K%MSb@a|v#NgCQb(Nx2GN?90venU<5
zN}U#7?wJW@{D%_uv>JT*`J&0dD&ncv{0$BiE<|9V-pZo1wl
z7SX?+iv*`{5pNMFVo1U{S5=H4-NXeSt0RbSY?Sn}C1#lGdf5>A5X`G{yI0p;91n&u
z*Mw|~YD#1kXawpobKC5_cjR2y#(f;&xH|i=VkSrN7Y~$p*TYz3FWlpNB`)7}xptrq
zo>R6l2FT8GBP!yDJB>6e!|7zESNuE#wgfSrPy-ma{!C|LAZMi^dSjtQk$D;U2e1a7
zd+sx_&=Le*kvx_Ry+0Gl&fUIYl5DE!V{(z?G}&A`6FAgc2ob~QJyO=I2>vU@%923D
zZXQ8um{!{!@!|N&&1f^q=?Dqg?4tCO0z>X2QDrms`jA?X&#fgPEDIQZ5CPdRe(
zuv}O_U8*_g<#9Aa-u?P(hXM%D%~9gZJ4T<%G2B!Mg1&sNH}ah#D&K*No`p$WCnWsa
zZRyd@Kid>tXBP)?r|ZuYUv>ILg}k^q!{rOCideh}C+efIDJ6)cl4lg)=lM-R#SA!k
zv=A-)WaUz?L)i+7%iH*Vdm=i_|4-d7_hVBDu4>FHqF(Tnu2WUfJSjB&*It{${fqmX
z4KzW^EV#8j>;w#`$bFBa{9YRF0?F>$S)P`l#01J<=)TnbHc2!vGOww+9LwAy2gwfO
zO0``}@)guK%mkuV{oPZ(9c|dQ@9T#QuX-`Qi4+zr*{L)~E>htd_eBTt5zDX#(FGAp
zZVf|e7`&GPGl&Nk>Ux5Nw*cKC#!may(!(-UAeH1DH^hj3g01
zwno1FlzyN|lhaiTebH@fq^QQQoV^m+sqlY-e%?|0Sk8%`XD4%;DxBK5EC&o)(yFq|
zjoj7fc<9Ea#IbWkZ@$>MYSf*d<+rC+>=IIU`r(LkCon^RKIc;jhs;ZiSMh-*TW!PH
zbj0rdOv_>s^SW`^OlX7v>c)?_@zo9xSx&(SSV|b{++nk9GokU_|GW1)1SXZP(xw?*
z1cVF%Gv1y#u2>bSUKJ`c?v9dz+NCv_gRZN*rrupYl@aFRg37tLN@nMty~_UEU043Bq&NmUOy
ztAZO$!W=e8m2?UQE$2aHGru|G$GVx1kJyc+6LX}8!^+=O4Hse&WdpJp#Jy&_5U+Pn
zt(R#iBj2;#Uj7f!xD=2mO8LfA9ke?GtTuV${jhfn{g!BHul4}10rpVc>REp7tFIxJ
zTNN`LyTYGAWwlSkTZhsYjUfA@?lTng*x<#`Tp{JuM?ddHJoGK6^mRdK5S{+?w2aP*
zSqk!z_o8Ss!T&A#^AGh}j@{Jx4RY$vn(24On;*Pd`dGfR6t_26qnKDWo0jp1tS0u)
zOnp$GjEPjA!{b9-lqDVb+uq_ZSt0@-$Z~l*ao11Y7`ODv~6zs$M!?$VNX53Uw2jSB89SYU#f
zyxZDXXQLAF#hV9wpKmFl>~As^3U7HG-&5;Ep;e>eJc<2)qB!KKe55#9qpl}i^=K#P
z@f7g+jc=t_Cz(H?^5w)Ua%Cv(w!tz|e}o7#9)0S050!cX%)S(~gsil4RH%ET$_T4a
z9F2wR?*jg=WQZF%-==!?Pt4NxY;R9a0Efh0Zkg+P_-niF?f&k3w1l8ii*HQTt7S1u
zE95_y>o@0}MPq8c`JZNfIB9zSLdD%5j^wA8$E#|zsxB(8L3AIeSIj3A>~`Ievw)%S
ztvqTHEmlJZgJB^2Tcj^32GBO+;Ff!L5hsbz>*U-*zCi6XR%YC(!^L<-KgaB&@#;et
zvm|wx9CBpYaf#^v6>1#cB{!??Y2)Y7xU*Ut0akV|m7mqJjlnGyKUI$u%uwQLAJ^15
z^b2FXwQDP;{>)H{xHyPodcEEMiJ*(s~LnL1hE
z7zsX0moCdF=INy5yRLtG&A5vz?d7VybSTx%BSQVMQM?4x#MSpzjHtN7VuP!Xxz_V1vS3C!5
zfqwRQx^K`kFw{5BdHuVNlkjJEv+|pfN_+7Ru$@pQijl%e%3h5C7HP~yg}NKY_ha?Q
z`Bjm*e~y1xnoGuc&e+Bx%R4!H-aD<(SO2Wyk1f`-Sc>zdo9e|b5j!}BQ+t{s3o1&{
zpx~Hdi_UMos)XEikY<%|ybY(34?(E*Z-VB-YKqYU{z$sbq!J`ow)Wrt?WFfnzVje7
z@piUJKULzQkm_b!>M6z(1-VU;nvT?q%Ey#0>$g`8yFBAnI-zVv+hu06TC#Pf2RD~K
zkb9nZ)^xqSSYJi!oKRjh^o8T(M#8p-izhR8EmaB+6_#brxi{BiyvsC*`_!UCRYx;p
zEh>#;${OSDsJZUdBsDsmFh1z4eQ;xS#<;N8FssE)NzqYF1!%4v{ISm>7r*a;`!71u
z6AMiv82i)Or^@K3d-SCD^>*$@Bnl^GQC{M>|5n*#;&bKCh`Zd3&3P7`n#RvEWc#z<
z0gF|U@khn5t@dL2^UY7k{Rgi$IVLewu%3qNw_7tOx|em5J8u2j*jdw^!ThS3YJ6K+
z0qqyB&BfW|=^s{O*+U7Cka;>mvHWST=b7)^`oZ|4IP0l%R-076qzBP(nSz=3Zru+=
zHJRmK6#&OkRr_eS=A#Dwey=W0=xI;-Ow2a8J-N}q%qg#@8+;*r?t_^E6`S%*O
zHDj%pdwh{Jss7s_BK~e-ONHZnjpa?FkKy^Bkx`yGVlntE%W5PD#%H@~?}zZ%Szf3N
zHWP)7$C&eW0Jr6R4&-B2I`0eG*%|tS=pXcZ(V1@Iw=Pj_yaZc3t=}5ZDMi6?*b!_A
z$$$VQq0Atf&)t;6ostFTzjbe9Jd%ndi~82nIo{ek#g0uBt>0Bu%eXD?B}{6vvrddy
z@IT9bikznp9Db}@IfIy#4DEu}^`_;6aZ$QmLKkN=@^qsJ?R>P=yR?!Ly&jiU(IcV^D;^VQK4`+8*^4&r1W>Zu5h$y6v9CmS**K%`iVo6;>
zO8!mA0a;H|GaqLZcWZ*o_grnPm7=B0Qd^s!az6`ubdf^7RXC`7Jv?biQu?!LMA
z=L*liIO$mOnX4<^ov>1&m%a^`%F?c%nl+>WXgQPdhN9{~Iu4R39O0DsI-8%hD{ggr
zr!UnjjXZxzWya3%nxlw2NoScqe-_{8Xg9R<_&!l;LxawU$9YD+O6lP`5kehy&~Ks<
zc?8X#Yaz;q0!Psz)JvTbGy^aQhYKNh5n88Q9}%q6yDeXT4}VL@_=O{6SXD8a-yaqk
z*xSZ~LXPr(jDQ=)T>d@PiB(b|8D$Km;4urch$bFH
z5xknt-P_s6N=C0JwKrS=uGF`DnaRd-piU~LJ-PU#(yMKazJZb_)BL$+r&pHcg9Y!C
zH_s_9^eN8*xnjCcRih;eQ@67XQf&Uvgu`qfH`Qm13ySWC(C
z(TWw&4_}q3HWs-%MFyYs(#}u^1T0<)u_3^ry?KY|m5-ExBi`i|i3AsVdlukco>7i%
z6Ea$mI-UM*As~XK9J{BWDn-8lmi50j%wcGjDf{#IR>s8_CkH8J;v46Zc{{BOG4M7JHux7p-JR2ZSMT)Kb$8ueN@a4xJvsg{}Xf&QS{Pgw%MdYEHUv;@uKa3{J
zqJXi^2s&Ph!`>OOr^q^!CeDxiu$(V(qUmgws+sO6dlE6%L~M7C2{)urK#mMB?iLJ#
zVIVS~7?hk?1PmO23|Z=CSzJjvBs(sNxcmQHJX*?uPDLco?Qsndh=>`-T!@o5Oi?n&
z95VZSEI@)p`JVII$XQP8^v<3aG+JGZa3fDPR$pMNJ$$nprlo%J)P^Y1Jp^f=mQppzhX@nKZ@CncmPf2Q5*O!B`j6A9s#DpG28a{a~AFSJQ~hVO>l-O^=6
z%;b>&^hqao0Da@BCDi=_4LQ4zg^9%OIboT_ZDFzFZqVG``sYQl8jZXuL?@3j!9D*E
z;oO-7Wdt~9L(ZI7yoLbtr2+M>6Y-5@74iPM|rZU;
zSkV9dho#`CWIt8s#Cb21>L^_{l17&1kINPl7-&MnyTF#9^nwCDl%PYRa9XC@zIq=D
zw4C)D{=V1iripU%rObFMOs#2v&v^x~ir4|)6x|@qs_;mFMp2=YOJYGFX2JL*O_f5N
zBxFQ|Yh}8&^3TfmXR?ATG)HZG4OIN60+M#C1(ld;pFo;d%4tmh##Mu|?-PQi+Pa@%
zvYggCbcPqJAG|0}yb3=Y5}u4B>9w{e81p51`nx=C99E30w1{SvOlx)~8I~dL)kuPW
z@4=wxEW!0(|K)S$cv5N9X~tL3>}cuNAIt0$eAAL3+ZEEl%YRt@uy=t9iNNF*BE>g^
z!QEwGk`TY6&xhkY6Loo~^r(FSPPlX2IQj0}c0}@N0A>nO_@GE6KXFv~z6(qp
zAWagOzJ-Ym^-~;2jVr(Puu2&Z>gikLX4vjVcW?LfS_5AvzT(U*HP3thP_|G@lEBfj
z>0Gl*(K{{z{=GIrEK2UD^>GT)ZM5LE=!PWjXPFQae+xC;WVP3-gGm&tk0FLMK$Z;y
zy7MJdBp~OQl)s_xq3R)>*{=)Rvf4PW8{uH#&`h~xHqf9E0k~BK3U}z&h{2$f%wSS3
zy7K!bhza#CNK_yY`N8^&^AqG!tHg_MDi~H@S%Athi-T33-+eXkMH7|edEJtw>-oMj
zL|^~j*I2tvcVFqpAEq=W9CEP&Xve>UYW|>sO}hgTpNIMbX)(2^{mRur1-FeM#&x;4
zbhzX4)VH|)aTVHifl^T#c_<>A0wIeCgT-PTVmoj)EVFQxeSRf}DzA`4v--EN=ZX%7
zxe58L_v!#F=fJ1lc)l30zbc@~Fll1H!1VjtG&F$-aOU6W1mnL5;%=jmiz6mM50)DD
z`?#L%jgm(*Dk63PhVFYNyUyHOU+P0QeOw{zc63vHYr@!Zfr0PNGP&ly*CM~MPC|K-sHZiO79sTXDp!qiGg|)}xU#Y7p@LH9TY+L7
zv@u%uL0eck17P*Tyu|u~sZlb4%JB;Tka>%t^!fgjy`243(I<|wXtJ_0+=w-F^+M}QXs7KcPOx0&-LhUC-afhAwhl%~0`>YLo396V0b<=T8Zyu0R(
zv)OMH453>v#(#ov40@YkWmQzODg{lzKb|Za
zD03;I#oo0>OH)&oP=
z2Xhp+d=`^W7wPAjS1Y*R@)<1K`JHdR`HU_^}M(9y;2c(1(t)JuUq-htMLNLC&VJO
znT{41V{mexYjqfVnA+kd-Cdr~#|#;C?9aJ8iicvCJ2t87-<)M^dmzsA2Q($}O!_dR
zBtUjh>schNv5|%oB7NlawGR0*RB5^}-V|>7%uvDWq1RaiMeY<}%6@{cgtrAr^9wNl
z>N5)c)N1qeN&S7Bv-)fXIzMz0*vB+KbP0l6_+d1Z;2sb=9j%&Yr`BZAulM!z^?3bs
z#nH~m%bU?DSP}J<4Tmig9D)|VIy4(FcFQdaRdWmAVcbZ407~`=ihL2$cqkNtCn4Y^
zz5~Kj_Z(SGVJGkNnpv2kTa3wd>NH#GI?ZCEVhA+#8
z^U+PW%@!{df8VSo$-xMDuhhQAw*xgndM|9C<9lfnDVw1Rii(C{6nBYHFWGpNZF*u#Djm;bPDd+6d0J-ukc7+r0ZG}{bAAE4fC{ab!nIC6TFS8$iu4{b$~RqD
zm)i6$G%NRKBp7rii#4`>9cLT!c#TN}rCO}8n*J@&>WVG*`U9ZKreAEQ@Q!3u%P*{WJLiD?H7#k#?$UuRbyUGK~u9UF#28T;uj9DZgL3BOQ}gtv5u
z!GsC7$opz%Ty6yq$X^^y|+0_wX8F~jcOsIL2K)p!nrH{8}U
zQ+q=;{VlHeB*f1jdI4I5=;BruT-xdMXSfd~I9X$K$fj=iGBPr9+RVX%cEP~yKPyQK
za$;wTlB#(sGQeya1~rPcbcXQFFl5Gj2W=*w!SD;et&c!aM}v9i4@2*@ZZ6ml7U@5Z
zM@DbUPZV}u=<<#_-!MzRu*Xc
zvjcRaD=#PU4O&H|8y4S?TNNqm_kF^m-A1qSKu&atSPZ@nT$&y>8J^W{q4a%=9?vEu
z$$^i_nSP$7W>X&ZL7=iAgfR3f6A~XU`B4Ep7k&%;JgA5F8?rp=<*Ms1fZ?kFItHbD
zwHI#tYor5*D0+Zc4}lVsTpcOM0A>dMU}bv0$d`j&=?JiCT4d=ZAgE9ao*Uoiz@4rX
zjb>3-pciQe_&2w~70IS-2(Jbh|8=*B+~s&EPqm-3ql(l!;Vy!?2b0+u;JDs@f4gL{
z+S3yM8$cnWo>6!X8J*fzJiccsvsd{bb{xmF1GIFp`VRuvclFrWkVoO7$siXD9JlLc
ztwa}2VW61{Y666Up321u*TEY>v`_c3GrK-Lxi{fs7yr#Pb_=(QC&-)+!r7r)PmQ>5
zW~LLqQ6o+SGN=Bh&Hj@90rLo*vKzZcs3HbVM3(DsJo}W=imhia1^?4@yPTCm$phYk(fS80Ojh1~G
zK^}py%A?Y8RFetVMUC>6R?`WsqPy&K2quphGn}t(R_Al~-4bMjXf>$d$s#H9#vxkbEr8%Z`R5|(CA=z3I
zlpqAD&gVWO`Xq-&vhd3GcL
z+QD7+!oV}DB8rcUBnDZsi7bylk#_Z35W>0P!yrO2m%CHiAID?f1<)7yB0fb21do2O
zYE{W2Xr^3umwDn54QNZiTP}_|>H>bUIPf?(ltb_=0NEq{TKg@W@nVC(rQY;|kVhoK
zB)5xk@y@N}Kh)>gTqL5gMS26`<_diTA>0mVQqvH0_$9A-mlK^jYt{Cvmt^Eunj-Jr
z;;#uL##eM@Fo+j=RlfJ?8Mm?WAB;KX%4z(%wGX+QShlg$E>uo{y88*zj6nF!2OMn8
zXY>OFZ>MtxkZ#o;yrI4V1lUj&x$T*$*9>he1>asVXb&x~bH=>QwkE**EcA|F%nl0L
z%`nsNnUZ#PC0RHaYb*1SlUB7dV33I+GY;*|J32k_ej)};%DJzI)mJ3?Tk0tH`@tyc
z59+Tn@QF9i4rVL&!x>fK3Bp@VC_7+K6S{CjGb5tG0PwF&);uBMUW@m)@nG-K3uf=(
zHW_Hb#1C}W0zB}O0>9dIU3jNm22vghGJ;-VqBZ6+>sit6fijovm$qa_!Xb~JH+A@n
zgiwAE-g(RO><_kFiLmPq`lHP+hx*?fIfx^0-_lAESkwN@P&d*cW`vg#b
zU^$9}-qv;qFU8o!c%(NgPUb_fsA5NUB}-HYF&xSH?&*4d$b|djy#?8Q4p%nBQAp*-@#c{l-gC#nDK}pH~A=eosFA-4$0CHo(vz4#tKBmX1XPJ|n7=
z1(T>{yD1f`?F`K?5fiK71R|Btfw0Oz`W-^Y!2
zva+*hR`$%wiWIV7@7H}__jR4ud7c+&cO31-AagVr(aV8Ny9R0V_9@|QDe<_qtSq)|nEk!j
zzm>Vu#HjpY#JjNT>UmEd7DJXhC^al=?H-Fy_`aRz519@{XD}8S`{WcW3*7{-&x_*I
zyn7PW+<&G1+?dpgi3Clc-yl1l#SyIa4{Ul$CQpp>WXr^<{@xCzKlRsT|@kvI$$31`llc7lWaN_jV~~bKgNoTg>BQ(t10*8IGN@P
z*VBx$z5{C}2tB#@XYlpvd5&$69Dhz8YA?MR$ps5SNv&}ExM6D?c<>K)R_B<;IeRly
zi4iKYC=6aMF_M`}Lw}}NiVVJ00Cx&osqK3|Ytr@KJDn}66lK^~yKPsV
zG*F5gwqAj|@rGZ8nTPalFcnEtJ$sw6)>!uqa*HRaIKN)2ycx+Hkk6_)g{`|%GrI1C
z5;kThoeO>w%Sm9=w}WpGqGhcy8C(Z^A!tF1{dXjjwo$j~{tH+bICyu?TueGa41&N5sS|y_tD-9}AFCw?FI{
zUA(+!$CnkVT&Br9wT^)&D5MW&cTS{;+QvwO5brjLo!9yg|C}aY__(*KVDf{5Yk&Xl
zH@)RK65K_~`*9~PHsYt?+~Br>P+qF{MLu=T*eaj1+sx$
zD;btB5vJoUbpw|<$TN;=ag1UZ!Bk{Gb(tqo5sWf+UggYjeJ$a1j`yoKltC_hBm^ZQ
zO%*0RaI{rDb$k?=&-KE}uLIJfdMdX|lsvkf$?0jVn<78p*5cJ3UUif04-7pU4F&qi
zn4FrwnkP|PsL~NhlrgDk>{;p~YNt}>T`I!n=SZnRtatMvzqqj4&r&+FwgnUk&eu)B
zGh5Q|>lA8c$PNgHQA7s?i5nJ@bd2}w|ulMdWo)b<@z}T_OX>fQwZb+3T
zlH)K~{#rP0PT126Wsh;cMSA06mXe7>
z)BVSwf?mPjJ{q@ThW>Y&JCTeA_nIt?M^d_1#Di}VSnM`j@LrPX
z1yjzbnOl(14jQd6NqcGmqWAUVQNj~5Ba*mgm{ViV3Pe`S*AipZN`^eot1BmMLh4Jv7
z5@R|0(X`L6ZW7eoO!Zkq{U~63T>TX0qI_~E!Rg`>#7F{wNl#=kIigQ`b9wZ}D}kL&
z7Ez}b6yq7l5H}VqU^7&f(@QQzv~<-JJJVaGhNPP0mO6eCGw1~Ze_Mie2wV^A=N$wq
z)D9zo0;!Y!?1~bpnZ%?L2Zb5q+l{KLdnx)TlNyx{^lQ^?x|G0mFK)c30P#T0%NNfz
zW@Z(ve-V(VDQSq2U^VdezJ8W5uJ7
z+-1(WGHS|hXNA#wu@regu4Mrt-lvoJ*(+dWk%*Kc9_(D#vl8W>CzMHo!sRq2zfC0i7b^=N`o}Cl+Uxqq)N$#2!27x3W=S7Tvfq
zOW5!uREGE2YOB`=_2;CxULwVdUSS{$$y4$S_Rm^(8O>W?V!`eiB-Mm;bA@CeUwf4M
z+rD;V?=X*$db}VikSbw|AHt9Cs9eOa@H3HyexJF8?&b$ijY_K2?uN{xY3Fj#Xd&b8&6D!-oQ=@(L+6g>Pt9*0u=7yj1IarxjMdKfTuK$lj;6g5UA%sW2ql
zE#&=h8_?^!y$>fv#o~GNG*Wk_eD}6zS+3!K85hd@NMk3M)pg?CRwxrmj;!Z4xLi(+
zo!Jp7|0IK7cfL|h3u}vwcAQFDcsADE#m(3$#JyWZ?J(*R!VM|5V=l0q{>CoPpX*}_
z)B0WgcXQbIWt;=8oX?oHW@iFih08TnmZHaiCGLScwx6!pE0n6TKrhhYEA?ZXO9)AH
zsb8_s|Dqo>vtquGQA=&XG+u{fBuy_~Hd8ARm&Y5lNd_(NZL%tlSemEdu5Z8BOSh$s)j2Dp{IrW@fl7?VHtf
zp)@vJ=J_Ta@JPzzoS(1#WnGH~$={KTjT*Danya;}`{nsk(swDU>Y2oS3XjCY<*bH!
zG0BruO|D2i-P4!q8b^T526D82;+DV7NwWtEB<=Sc=sqE+-&PsMdNfHwu+Og@R2pixBFFvg+C)8`J;V%QWsDTgP_n03
z`^4wwBvi4;GB8hK-r&xKc}?}ad{4HFm;XJR
z?nx+o6Jppe423+ictck8W1(<*Hr>B6ey4+ZiSD5>g4+0ezi!YA@)+e(;~1TSOZg{|
z^0}DWNU)@KGr71k2|6-2Yov>+-R4gSw!BW3{`Ns<4a)c(hCV<$nrT!^oyHq)`%H2O
z=1K0KzEb6qZJ?9D&VgAocUwct?$xTuhym;AeW}!k70|x_#XBHN#^@6$GAGi1x@OG%
z(LP-^xrNzr2^KKo`op&tSWOBq_wJGmm2<%!`hn+5)|ii*II&MKLLz0}J8%NRi^FH)
zh!7;a0P+m_^`&nvb(-n2J=N~^JYoG%AWi|;B7F0!A#@8e4E=w8Lkws?`L4RuLn2ct
zEaD1-6jJmheYZ-{D_ENJ!TrKVeAYqi>k2be5^-zXnY*;yVblh>D=`xY187aFX~nNE
zamhMQ1apR;6&xSkzRMbb@zbn8rnqr2*5A<9;XDm}_Fw
zeAiSAb^tlrsSPQ1juUB(WO||SIgoTOZDaIAJB-P)EZga?-jAl-EJfPq*qY+BQp|n0Nl1{T8K=lD^Yv!Nd
zjFjZhGWC@o5DT^k<3y`u7`EXU$@aq4=?i4`*T$5lI81OSV^cD&BZJKCyK`Y&+Zz~B
zJtO%ItheI9EvPe=I1Vi>;?!
zQwpsZlSxk&%%FzI4E3a>>7%-q1>{gCQJ@j^JsS_6sm9dUJOr2UxrQ!ZCn}lrZ}^`Yn$HZ
zU%`G}kj37nV-|}pWe%C%fAES~%7h#elPFxm%f;9~{bP;1SXjdATSPHlkIGEmMbbBB
zE4-=p_zNUy3n@41qG4>Jbe{8x&f`W;np{u%Y@$2JqWS>UWrUX1>Fo?b~B(l{FN+i}b{ta8a~N&urcHBj3+
zc)ehbGlsrL(AP{@2BcD%AFisHwwzjSejreH1?I9@Jiqx>V0yX6roSQM6j<$cP*dJj
z0(J$TBHJEE(6&K6pd3XczU-=TcQ>DT)^>7^`C^9bep0ngQvCE|_RRJ|z8uO}_n-aL
zhih`HQR>dOb{3J#d;4+K#W%OgF{eFE^B1H03S#1@*)f(G{l{}k?~ON8Gn@`p!n&tw
zQky)mff1XDc@iVTv*g?ryxa`J`!Af{u_@AEX*3{Sl@?(bifGF)M@AF
z9RS`I5Y%+<3e*g=5;#&@n~D^QE2k)NSrE4vkVj$eI3B$&;Z3)>cMaB_
zyuo5kBfKlr3T+aph>1dEblaD!(Yzf+URl}MZ@^KLH?vuwe)j~Vu!&G*8+_-2v``xW
z;Eq>E&(A&YH%hranH0;U&^oboaQ)X-xmiXgfEeB6YrR2w`Lyo&)izJ{R2$EUO>|7eAqmm0`z-h25S~^g+^Z3lQeOJo163|!^3z;Vl7j|5UAYf;JF3CU5
z3#G_i=|Bqid)2C6PrmDY~b+X4y(eB2k
z9nJm(nPl~Hoc?Rk!$*k2=wm37yAoQ;C^=;v-&%bql6vFnjNzo4QU9b3ZQ!K@GEP*;
zt1#Cr?)4u{PPVPkdkt(O*=o-Iq+j5WazXL|Sr~v4cpi$942O&y1n7UsnPKdgB+%3;
zxyo
z0LwmT57%sT2-a)}8%HJCE~Ts_^vuRL^0cw$Mm#F3Yuwu~Cb0PI+fU_^R{2li;|4sX-FwpGPv^yfWV4~@
z))q@r{xN&jZVN2(wrI5LBNb2!l{|nVav)#_8
7s@iFTfEgcau}RX?44j%PIV^GV%M
zNJ4d1#-+M~hnMpFqvP@~E62DjJaxqlkK^(b
zIe9o+<-R4_wGl}nOfN)-=3Q<@$R;yMJ+yVlXV<_b)SE2I+%QQEOKf5bERr81KlSES
zd?WVV7$w|y7;#G1MRR}8p7}H
zpML154odWT=hf7IA;NGl`$P7fESo#N3D#Qg*S6WVC!}L|jEUYn5_lukBcFBhm!S&t
z8C*0v)@3&=RVvMnA#zEUYe$TU%Jaw4CpSZnB55{j?b6|`F<_pTcLS)oe2v1PM++9jjRWik#C5h$oS6x_`GD|=P9M`x-Gz8F5}
ziXYlPmJ|LnjJmyw?}>NAYbHzzVz?&+Z_iw+R?0uWxzymw@oQURk^f3x-w#$e|J=X$
z$?zfcgp?hyy7rBo1>&3QM&Bnt>KMJq?(XPI`o;Im%@-e$%fnt~nlTVM&D0_E?knz^1X-nrym-1BkvcuPgNs`|YLRLPzv-R`e9Xv=9$
zC}T&(QX)MX7IHZG78cS6Fm@o`mLSHw44H{KKKI>2F94LV(!Or%4Oi-$oudm@Nn$v*
ztRxWwRC!g}!6AjY6qbj1{{b%riC$=wSRsc6FUf|Hdf;*fv+(ZB(g
zfY75qF#=h`aeeqLh1)jhr1__ET!)dLWeV&uI(el^Nwjk|vFM{Lhd
z9OWha(*&}m52_-U#@-)Vh5kmJAmB4#4PC*WJl69g@BH0ReCsONqvt)b&Nww!9giEy
zeHO1B5}mL|s7{ZennkSb1K}C;FZfE+=6p<<(R7!mqYGQlvv${VXv(2=f%_G-r?+Ar)B(7eINcM-EtyVR=*G%x%W_@
zq$`O#RP&MB;-1Xqts-sdSz7_-TWAH5dYIY%0
z8iiD#kan+GyyPco#Cp68Jp8EOYYlL5)d!sVV!V8m>94$
zWs0zGZwx|`N8g=tSK^?G032{z|={l^9i-^ct2q-leT=frmfloObo%-$2E$x-cb9gFo(+
zigVw;A}JfmBx2eqMMQ}A3j_{
z5IuTmW$x|m+(-h@onFpwxNi?gb%<1?9fT*@4*DS%${S4KL^4z2KEz@E+0eIWXCj^)
z)YR1U;)M*e*T*#Le_eUVhZ$hklF&3--2O6C64F*vOxGDWQY5^8d1&32rLiF?=CPBL
zC`QuBgOcm67Z%`qiNcocD4DLfI0`T5|7YD8t*dm^NR#G5xKn3>a0c6x?(qjU1RW*#
zdWUj}=`7rvD=+u#Cp8UTPqD-jG#>T+ye&)XuXggV!T8TRkS8QZxB7TSyz<*PT5i^@
z;hOdNvQ=6S-Yh|r73v9QTXW~{N=Lqtv@M<|u?Efr0&BJbn$Eeh_g>hdjl>J))v`%S
z>Qk*!TVgRtx!qkm?j?y}dPCT_I2miIN0T%8jY~^?T^H+7#!jLRxv*0^d$F6!j+I0@U?uD!zb5m~%j4U!UI!sJu$j#XTFoi|P%PbO(Ae1}{WnUlzUvyYI)#Ro>sbPlD~;Gh3aKi{|Ofzl3(p)R?>HT4T|*s*4ZJ
z7e^;J^7UeT5|h*(9#qMP=?Cmv)__?l=~2n7x^~uG@wB7QNlz+GAtQEU_Ana8Sjb_Fnb;?o-A5OA91D=a%W0BES(0@2CBNZGU@TG?Qcdz3*VO<8
zPCi-Gc7jI7iUY{L>|;-}E#Ab$$n4x%zS7gvGX-dV!Vv<>HsCBzuZ%Y#RH8+{XLcV|
z?fRbTxq`8BJ^FC7k@pO0nQhCQzW~A&IKK7i$Y(nI5-RtJ&`i*1(
z?(0g~X~){}XS!IIpDjErC7)^j>yqXnL%WW2r}VuU*WvuOK~T_4r^<8*-#k)m%S4m;
zk?xzZsm9!jhUeBizxEvDt_D~udCscv3Lhyw-G;s#(%Z_00jQYaa&u##UlYDQEMfb(
z`njdNq>tDNZTKqa#MHgNEldLe?dVJEZbmQQdGg(0^{hehfKzJX!pFi6pm0z2Zw2iE
zxr!5MaiIfX6mgh|0IAz)&uZZI1ft&`Wn;F9
z)Ajw{9Bd?k2tyD8s-sBtcnDpV$r;}(+v91gnX3#w83-A-`b4RmMoWYd}enfZ5?$XIz
zr6N9w=Qi&f@vl)WDUmdSr|NY8O$#zKR48=Ms3(M*g=3h>`b@1oxbUXw8?DdUVNw3i
z7KXt*M3~ac<5k@(xOfp%XJ~}6-eXKWhO`ms@)3>eZD*w6Ce$?QJgE4$BK-j
zyHRuTT__jO)f2YYX9*gEOqWv;2FWJ*)iSd?$o;OJrA%5;g&0EvRpvTRFU^v%E5tN?
zpZ+;v>UaeBwbx|y{Eq%K4V3miKY1YERauhsI64QDBBr%g(R$v{+g}4ae%zSrxzblP*f7PlDBmud>
zT#ti&g}cqAR`F_J>}-P2>r^g6A0WSGdWEE{g7jRmR_?H9e(RQS!ieA#CxtA~`6k_6FZ2vt3OODqw%CCr#^Y=q;
zZbRqS$J4+ge(rr{Pxb{Wd%tQT!?FBW9zFhRQ<9C!cLcBJ%xVH{tGEh{F?FG|^&Ln@
ztqPwSR+R@3gxOs0$-Bt94Sg`_%Rr`mD!n;io>r5DmX)PWp`>;$osH5-{o*D=XgTV;
z9v+_lw29Q6;k3syuk#I=`1HdPM(hdA0~OZP*7M@>siVanW78r5&--l
z%k3Bo{AsVY%uutt4|<&2!h0C_VIsg4J{h%o^b6b*0y5qv;dRUel)NvagtP@IlX}W6
zFVwd%J>%2L2$;?`PYW%zDby
z5u6vq6yTw5SDQ>)cLBUUc7JzPuF{NpdZz1$^pYdu!D@(KF_q|8}HwceE>E8Z^n
z(pO2uy5gYAebvcNum{?tRbQ(Y9n_%#Zrfpf4u{e3a=|4$&v{@vcia1O^q#cwh{`MK
zloV!=L$y@r+{8U{6q|%%lim@d=lta<0~?IB
zw;`u~e+S;VF^Fy;J>#_PD@x)!n^X_~)z>*B>^2&^*1LDr-gKJ;F2;P8`!0OKsev-{
zN?~J~XQz&tL9E)eL)iKD{wJTA(u&2sC>?r&Sw_J6;JtVoQs{T2E-lIAb{ElkqP
z3PHkq8B9z68MyM2B|yeFCZ_c3HYK$;5u>z?i1)){WCVA=p=-Da6SHUIR_yi9)!T)e
z2I4T?rT$xWIdGzdJ(jLKiib0x46IU%iROC~!y+BMbG}%n?=to*TEb{5?Sj=hQmB}s
zQ38%b1LCaGw|+dcJlk>i4DVK<&x>GA-IJ~ut>JXy-crCsN#}IrAd8Ndl*^fXfPVD6
z)^Dq|A0=`1?^qOtJ0@AWFVOqeH91es7W&0z%S_Y(8+&qss58D_kDSGRCn>-C+b)<{
z1DDy5wnpwYspos3vTGv|_>P~*_trqn$t3HvXvi}RS`tit5ab+_zR>sd8V@9zu{TG&
zI3MaD^gyF8p!TK8zmy*{3~ZvFx?4US_FVpZyGCEIj-#n(hJD&-#Q3$C@&pWDmcC}#
zA+#L?v6sfVpI4sSeT;evGY8Jg
z|C|=6o*_ZTUX|Y3AsX=E4UEoB>Jf-*#K^krgSjobmsh!gfra{Q03mNi7SMp&S&{eI
zT>4cTBpP|Y5z`MAGXN%3u$keRxr!>++4lR#d3a06JQHlxHLpB_1{I?4$_9t@%Cd%l
zZy@S5NQQDb-?+Zc68H8^66uGC7Q=oy;f&`?$Rj=#Kt@UZZvBl%<96p@#pWo5L8Yf(
z*vnIqHZ+9VkCiJOeq>UQTC;QWjEY#P^ODw4*5@R3|RsRrZh1v4zuAj
zEmVp*+#gNHOmOjSP*09VJwZwsxM-Apu_RrlN(p%d3pN(}yG2Z?ufrs(IZNs_QHgYA
z6_scxe)Q911T)YEY^iHN(ZUa-kf_F!P8fTGeTSt`J`f8Davi7VB2Ise0`bPnM;trf
zm%nqTgNK2<2zSF*ow)alFCQ_fZi&BG|J+U%AOFcvpK=G4d>Qv`2j8Z3GwI)N1Eq-3
z9dv8ku6KSWWEIv_~zFXz2uzDLK
z<-Tym=q!kbh;4S?q{+M*cI#E|0j-?>XYY?ccSiRUAhcbBBTwR#&gx04o-C^JYp|i#
z&t5-GPdVP8V$Q@Ak=9NhRi@+K!rLhtc}vzEuNYb*(@q1N*1m(N+YpWjoc
z!TLM!c9NpknbQc4H89E_j{~HQ7*MXv$bzQ!xDcp~`K!!vFZBvkG^sDm`GikP*$C`~`k{ekY
z-Uc&*U+m{FOdo>n7|4JOM$}<%menM6dZ?&uIhrIw?G{3@*5$W($1qdDj*Uo_1Sih>&c)MCV*$gUJTq!xq7FtTBEam;dbVoZ
zb9#_1C~KPjBuT`&9dwEoG{M7-CAB{{KLKiV9LcC<))nP6&fwMHWf!=5})5PS3&q1$jQlv7cckwVQv7}FcWa_!jI^q=iB
zBpYIm#d|4DrF%*7whSj5D+$2tzN}u*vPRZtj_k@yr^mbom7N~%q1XN)p`XV371Sw<
z1sM@Mx<$AH*I{gqcA1${q%72YnBDJ>jnRC_)nt0~JOVVBlPnS_
zQzTf{(2=8_QncdYT@=rKIR^4{oY6D%&)5njUaII@+_j7)u#Qf0N&frdzm76YBY2SK
zk+XV~WmL_5v89~4O1CuX074=QN3Tw{PJ>y;w`5(Q+|{&?DS_7mqk&`?V?#UOGnSnX
zg)mbf)Lt_&e1BiYPa*n=(gW`=*DgIdi)TOu7k=gRIHo9&T@3UyI^m8C*6XBJp~k_IFNn}dVVN2&_iTl1~kH6Yg1)4E}i$~_IG?RE2Ga-4$Yh%byaHX{PI8nlU1TOyV8=@Q_^X}>J}
zoH+*4-Y65r9zOl@v)}~pxY!$>RD34Ys*PLYhT-jFujH?THf|}wMe5&cO+tp5INRu3
zfStQ~)XNdICcHOx+u0W5Dv+)>7Cv_5Jg(twYCiBdxTlWMLKW;oC+quC`sYS3A{REW
z5tZ2yANAECQGKg!LN7cGw2kM4DppA%$4VX}0z;rJ&lbf1&R0^7FsDfzWxO{?e3wX+
z1zSOQ+|Wn3?xqToMCrY)$;ye>52!B8c0HPg3IYgYR$qs!5E<=2m=cH#VQf>iL<+~*
zv?8(zv6i1~hIJeFcO;$va^^w^u1;S3qA~$1Cpv6*oo`hghhD&#kWxK+UCv0|tyk)0
zOxWzscq?g~Uu&m3&0>+`!v?u3=5x5m5?Z?hh$H1=mu{Z0pn|E!)v=GRFdv`;
z03Uu3uT&4X2D8}01jruX&?HH^=HyE2JvCf|!0x?fg~c-bw0eQNq75HAw4BLr>kJ@A
z)A>!+cYa11`K~G-mY9p-zwvxl0f$j!A^Dh0
zXA8^2@81g4D(ygO<8?DIz!D2{glm+_2s)>=K$_4M?(+kITOGVxV>gMA(++
zD?nd}db(2B!er$KOxc=#wOm)RIo`l{?mAq0g(hd$H~~DV*rakHAP-&l_&RL+w)O2+
zkWBOlNX6y{D{|{Eu&FWVvv(Fxo&7t!C^KKgl?ay`!d7Ogol(J#q9UU_NE)y(sd?e-
z^MD398m_H3paZe+sspG&JV9K&)HPU-u*RZPKy!0xFHzld9%-{!bv`up`Z7ifFi--A
zdNQiU5QuHX>m=uh#usnLBdrdExFvXLd#IyrsqpMQGMF1Fy*;|W$EkFmLa5NNcOWwl
zio`VtVXEK1e!ljJ&5lK%ec{n`&3}7temwNMNh{QvB#n4Kvm>9vdEldw3KQM;bglfH
zUpDXl48`)1#IH4O@8?cO{oXf$ijz7$158f<1FWZXo5lF#3dMk)U;)b~WI>c@a>}{gw_C$
zl!aMC<`siPFLe#T)1P0H9Kq6PiWFHs
z70f#hUjE;&t6T`0?Uk(Bnt0&2BauP0!?FuWr0y1-!_qY^ZnICgjulO7FY)r+LYn#BIJL2;L~_5$-|fFYyBy3oAu!_2RJh9xa7Q*r@?Em7l1P9p?!yCZoEK|kcf%Sy(rqB+^
zVivAB*G+-Y>cz*OTa%I6FRouRNJ?rLAx1ifz7WxG|FrZ)u;Sv{C?vsslPHj0VpZv~
z^avFclK>_rzoeEvr6$;`2ylZn8A2JRLl;nr=?SBU-Qvf7oD2%!+B9t(9=3pacYTax
za0biWz77u--_z4S*Rt?2i5r)*-)Yn}Ogwc`WR~fAgw^O@ZKXeoS7cv=2NXV@-Thd*
zQ0{;$;h76QW#^2%rjG-o_f`kU3)qoc%rfe|WwaE8FnwqMupvZa<4NMD_0E&$U}8E<
zKODLQI=a}eEDaXSE$B#g?ELQZqVEfuD3X0mRzP>d6ZgQtRmZAWXcmu?1K28mj
zl7e^aab?N1);N0WKLUH+Ga7qK;v^qk{CD>G=c|GavI1Ikg}M0S?CbI@OkC40IMK`(
z5_nhfTJ!^{X1Zl}iU6W)(LY|fn|CpD3byA9h1xqk^Kb`bL4E}I$AFeb#Zcq=Y<_N9
z3=qM^rd*vHp_pJ>aT2Z#!E#5HWWpJKi|gWJr_G%Uy?1aQ&BVbK-yqy9(E*UL0QD%yZLfdXhb&avzT^qO
z*#>FiRy*K`W2hQ;($ou;G`-h@PUSpvO6mh5>Y<5mgpO5ocf;HfRA?K#vrj?z$Z!hsxa)vH{b=y3NF=FqB
zIm2IN?4P?!eu>nng3y+LtR@Jw0*xl9xA|@lv=LoTYixuYmJ#Pbe>26BT#%sEb?ri3
zzJ9?Qkjqhd8Y7-6Uql0Q+o8c^e@D9a*xGFO4KGZ%881)`MhsNsB5_|h>^qFNEsam2
zq1Gm8aopegnKl6)HTath)i9z;+On$|l8>qVz2$v}*T=apkx=Wo%rx_@QBuct$^^v3
z#GDOvc?A_4fT$9b;0>~9vGm=wS6$pIJ`*F=guWx;?H5hesYtoKDF40n^vY5X!AivC
zLr_orP1}|C#?M~n%%+`p^`)7f^SB{=HDrHNEl`i_kk<%$^4aBFa)Dj(!Y5-je{bk6
z^0rU=lAI4X3dUY)FO^_*$Lpx-#6jdAs5r9VHJHqff
zAkOu$;=1D~Doad*_}~SNze!vYn3o`D5p?C<26hog#>a-Gy*tIMuG0CFcoU7B6g*o+
z#M#H?oV^m!CO)(k;*}rp-&rWo+To(bm?s0G7B_
zbhq0{^S2I9WjUSePKdNp;kU}KoBXE9{P7_FW5MJd$l79XG9U+6L4*KxtV$0kpFan+
zL;zoUG+bJ)4!NwCuAVe+dc$W>`RtqXOef7JS-DEp^EnV1wO@HSp0{1m4-O+xa;j~-
zL-0vMzuvDFx0*AE3w%Ik&Oyycqb8E>m^wb|D<*A>xlA5>Tf`Ee
zRjWR0se!TZ~DbtaNOQm_~QOA7Z#lr~F|j~HMyvN(4%NabV{^F5=_c;IbipwC}*?;D1^EW2$ntlm=a7PPJGZ&xV|hp
zh5FP0oG1r%L?P8-ElR#_1;Kw;(L!d*WH3vw_C41Kk;E
zngEmCZ73=nCW+&atPC>FPstyP4f+$hZ8-X>T7GN^x2Mnk-=V`o7I=2TtBe-iY1bg3
zv7l>!i$h_v0K9S@ARG)6<-oo|Jxw^omW$%rN#JM{`r+Hw>LfzCsB{H}#{d6y_9X^w
z@O3&&=>V|3b>Jb0Lzp=hfFjF!RaDRtW2*^}B5CY6!@wezd
zx4UehFY$ZLs3LsOvj172KTh30zX|$*WAYbGmLZLXzd;9|lGldrHHhK(1QSrKa0kY;v}o-XXv@^1^G-VfuC;8R3_)U=J4jM}~0;|n)bP0!;`i{Mh-?3VdH
zaK9stixo+79WA4~YBqW<2!nF7K9NSo``O0a`=&mekP?W9gYAv+_c;2smL+GX->%+y
zz_s&V3iLnY9ut-*#+_}WhAmLHav78$`Ed>SbO^gITxv*@JW9;??4ul@P}=o=w+seh
zk}@ZZw~tQu`=J6>VPEo3L$?dPI0Kh!B5-UBSVr9}q0eg37R6%3s8lVh-3pWzTxl;b
zk>i5We$7(=NznkGu#T78o{9x$5t>gVV8fhrH{#S)!(P{J%oA59E6)5ETIioE$$WvN
zEt=hmW?DrI>YJ7bhQ+FdKY!nljy0ihAz$)epGmnnd~TwqW8oRc1Z|r#60e
zW&JOZ^Uq3H9;@sOb4rJx7S3xrZ=0CMxHL!K=5Htz3CAOrTR#$^x5wEDI^N<^$JPjj
zip^ID@$yo^C6zH5Xw@^yaD$ma9Q0q`08&(RD`0Fs=;s~PM1H=x*#v{@(IBAvvB**6
z4jhU?r0bL>TlW1e!hPKaQCtVy)?6Kl0*Mf!@n#Ic=*zwc8R%t#Ir1dsKV&R)=!2KTb{CZPePPf0yJU
z%*R}vNYDt=9gHr*Y^iG&OvM8sK~#e<0y8#;3f=;S_wn_HA?3m_}>jA5-wB9
zf^P?atNH6(kVRnhqk~^%r;_}8OThh38(=o0>%`0!<&L*9t=CukJyjCu+;_I$r_IBJiqZ=ngO-`4c4Qq0m!Ne@jt7l8E>uvE`L(!n#D^rb;*~
zfKeSukEDaz0X)8{fnm?oC;$7=|5z7zjI+1pBb43RWyxdZGtY%v0Kj)oQnHJ?7Y}+U
zieOcfJU=;xad0CsUqzK}{ugSgGP5>n4wHhb9B=4GrNdzJ2h<~u+;}{c-`04j+DmSM
zLxd{ne1N_2OxbMwzw;Sp9z-$8Gi=!vjhYBRLov?*Ew+0>(Jprn)-B2z>&p87`oqD%
zf4=NWp&iS2T?dt3g`*A;Q1=gmi}^yfmLm$3&bPm!2N&t
zZjN^^Zj45CkK-LY7l|h7W#8-G&l6YvS|mSYH`afTVC4)vdm<`_vjQ9xk>0O+!uf+(
zuY3~g%4BID&4fO7xGr6A;^3M-ElN$q22hsRUO8zT_N*UzsyAlr*Ra83tz6LNI22iX3)
z6185k%F-?h{M`cxl;Ch$=%g9+TDrTWU2QObPb4>6In7sfUA5}A($vImAFBp^ca-8k
z>WsU3um-OeA5FegVJ^cRo9>s*R*aL)&lQ8*i$>HY3SqBR-#vUQcTZzo=_t*SM2%|_
zpPXKHK1%&-Q2^OqlJLG#XYfiJlop0ojz*kZew%{D+F1`pxeP9T&AY6ZtXD)y+&fAA
z&l-{iV=zgOPn`pkOOjcX3C3+RreIxFfy;I(-Dxt}V9tu9L)V&F4|Z_#u|T*+@%n^A
zi*&b{;)LldunB*?Q<6m#QMwe@tyzuO@BZnW`PRL+X}?xu`y7#S8C-Tzy4SOIwSq3|
z@-*$4M;!-bF=Piskz)DIb>hmTAxDiL%hTm{+QlFDdstQZAXxosM?kXWP8xZ*m8ejq
z9;U~du3EKSjQ?6fVV`ESA7*#3FU{i7!SxAmESY}lja-x91ZA-+|Am{5;5!haT9P#S
zCh6HFYj?w0_*^^xiTZROG~I?L!2{g&f`jYu$tOu)p{Ip>FLQDdl$T-Oz>p&SZ<
zmLCg}H-?_63_Y`Dh15F%YVSp}f4I+9c%U_SclA|ih~yMs{ODtf`lhn4^QW}f4_cgM
zswM+5#@VfL&a$D2r@2Do1TLLbozYC7R~fYIZNLWNJf`K8^Zw}WzTYEuf1Z*I0jXiW
z!3y@47gZ|}1!7yr=rX;&%kY0wO?VzJbU1gGjYI~vi&EsN60nUKO9}AfZqG>`vmEGF
zZ9Q3gIiDRe`B+GX-he9>`;C#B>WBYq97!xrjnvXU-lA(n4Jl2MsIAe}-6J
zVGB5gZ-2axH#EbfRJ*n-$v!QU>mXE=DVS{LxIb-4zwjVFiso8%$DBcRoBBsPFB*Tr
z|BtS(4ybb3-UdVj0Ywoh1*D}Lr9lPh-ZV&eNF%KxQqtYsAl;#aun~~hbVxTy!=}F3
zp!avK=YIcjj_|%S@64=O>silwmWT5}GyU>0HKU<&6@G@;JM;Q1PJ>RA*y^#hcgV$#
z|0wnH-b366{DaOwmPx1@qxo7#?g(*eRe#%J+bXI0E>GoW{IMNE1g#Q
zAO(6Zg2IC!Ze#~uil%^zr$Y)wHT*MI&3pG2qJsp50@JaR8|H~J`q;l6xmGf7fwHWQ
z#b-AxDC*W{6?Qf(Z@*0CTfvk(8`|HDb5X6~<#RdQQ5noq5R4{GtP9~QunMGa@zU4bKYz5^%f78K-q8jj-TG!GReW=Jvq9iKT
zJV9peh3`p}&#Y@FIVa)^Qqy{x
z*O7=n9}1U7B@rTwj5E7P$%iV1Tp#Chsa3BvuY17b@)cHQvbC)r7WPJ$p=}p0Btcv}
z$*zj@DJ47i8RN44g`d9dG@c5SbAirv)t#y*BNLj?y|!@pP{VMzgkq=$e@QmV_By6y
zLzp2|435*cLH5y9MVbQ12l9`06W*5SyQwGW4=?oSo2kcrpRSb8DmalGA{{Xf+G>p*
zP++k3c$}#DGEO;%d{`xqX_@)n7rwr#sIco{g$;4`0vi@_->1s8JtiB)f5tk@4?cbz
zwKc){x$unRT)*L?Bpb;GzMz;tE)UC{m(tP|0({G$1J45<`iOZ^OFH}9h<`62(`VYH8iK5zuD~!%5o61qTEqtEc
z=luPj?On<`B%BC%r3g(couS5)yE&S-2kkm(zeHxRQpW
znYlHhqQ%^i#E0hFWznVg%IkC+v?gOxJ=SpKY((4H$}g@8ga{B5;{|=dF5j;^?~Be*
z@_rj7zdY$=c!AL-Tn6)@}V5YvQR`$2)pG8!nZo6(h
zvAz3}62*g`MOg-3-lN2Kba`wcRqeqMi2?+rpbYS5k~5)?_S(*5V|
zut06RICouvIO?_+aR%MHVp
zFK3l^I3XmwumEif`mUFYMPvKdhEUb4040wOWWzi}4
z&|v5I5)mW_-)-(Q8Vd
zLK%{!U8*bnW$Z0yyYKc5^SeXT1@dniY&E*R@5>asKKk^r`^s@47k_m=6r??=qw=V<
z{dO@X{IS=PY!zwatQba}<=Rkko!R*Y1pk?E#X2=GJsRZ*3~tV>jJa*g%e9kEQenvk%RXfG+?dP^
zKq0xBFfwwt$uifhr`Ki3O?3$5`-1pGJW+F
z&?i-Yv;dANpQ?};N4nJ+`51E8LYmDNKjcNfoV;s@dUC-4(Pn9~nr5)8Du`B#9~zAl
zJUt^Yd@mlgh?nmIwJvbXji6T=fs(46AoHhTDQ4C8?_G@Y;ycr4pI#JOWh*=uc}$gR
z>&KMw8lOTp@coxomL^uTcQpi_S*-0Lg!)DV*!o5j=HIK8Hd!Nh#d9;{#TCRQB0v$r
z#lFkl4qVqnM?
zux7krqpBIxklAyp|L|aS>0oVSbfZGccX@)M(5+wMAtCSPo<^HHVFQzgvV_LWM(sY=
zX0?^;2iW{xG6%e9xq!6GzT9}*Qpe-lkcLYX@;RY|d8!BYHg*30)usANdq>cT?IN#gPeO3c{`4KZ=hSCp2s4ynU|3>x=IbJ#JQ
z$&irIDbX$JHY)uHXu1N=ZjNE`58-zzlioWJ8xoAoulk5tPqM(7k6=UtAg%+*KR54h
zO;1NgWNhgtR-oKPJU}Q#2A6gpR|TMnfjyDL+*S!d8(P8T@Z^0j%hxsAVxx6dZVr@O
z65?SF$I(Mm?C1R$+U%u^{elLkmjeL3{V)6wj^eNM6dkLlC^#L=vz7)fk8;~Ncxv^=R7Yy)Bc4m8bC~Fe7=k{Z#ieb6(g{gNm
z7s=!ORqXMBo=%@migb=~lk^AkGIHY=l9jpcYl)t6?FS}rRGO@iJlXy}$R1;VCN`b4
z(l{5SM0~bNA0BnCzNKUDLvN~dPz8fIJgms9O)8$`W&%rqjIKoLz1Y-5=~8kA1cRjW
z-klD)z=XXN&dB;aB937A*Gc6aBX&a35zA#p{VLqhc0Wix+~96a9Bnx5loDF19LKg}
z=CEtMNAH@hc45My2UT1LPzGCwECK}rGH9tll)oBy5n^MY_+uDI_66#ZvDsGRcp;&#
zq9GGHgG_y7AE;A8FjK{N0HRO)r$8T8)eBPLuRRkfx|>yc_c*
zd@C240k$5WosMKCC#KO7=Ve@N46=UZ^*d}Xb~4xM99J&UTgGuZ^(J_DSY#@cu9wtp
z&J_L1k1cYRX*xTSfSi~&)Z2JW8B^e`@o*j=6VYp;BgQ=9hHC36IpVf43kBmpBY%Z!
zV1#sYOS_$s>&`D&WBh9#D?ZWfjhvQ
zi__~Hmz05afQ`s-pr9>d@S$j0iBx}cexy7J=sN_yuo;eb@UR<+c~DQXX@5uUFV%h{
z%#N3~Eurta8X6(K6TEin4AU>pZqW@N>0kG7X}+^Ldbn7NfB$<~&N4$v9vjl+OqH!h
zm9QY>T)l|*wAwaEFv`^?IB7F}3TH1u2iw4QHzIn%uyK-2dx1i5%hN<=E*SRuO|QEi
z8d)3jjxqWIeOtARv}faS=_At1vEK_8@<`X+E|Tp=jPH3f^_TAzo$NmQTq&P5cQtBj
zbe|qCxi++~RdLlQ)_0-C`+8livoI=$qX`zqw}@kbf)Y`ExhLl%F$PvotbFF8p*d4>
z?l|6;1|?49)js~%iv{yhwgsugV#WOF;m^X?4znp56Ni1a-1Ngvz9yIGr?J2FaCmi|
zudlRu!w)79eo82;ex`tDdq})p76^xL%)Y7EQk8>EkES~@T+n!2A1=S`U|@0F|D|}H
zr)kT3!Fca!`Avoz>=X#g0!3Tx%nA->XiE2}#ch`EFF1jYGW(%O#CK09)xZ#6~``JgyVKM|%Cdmw-Ra{2N?yn4Rn
zc9!-YBu;3buP3`b&vz0TH9qMna>0GLmeBNIw8E8P7al}BIp2n<{~8a2;l!=N^QjTKS#DghYA;}
zV4b77+lGIg@aESY<&9~}ct;oInyKQ`*uC?&s#8+beNY+82{+yT;#E3qm&_sQD+5fuPa3f
z-~D!<%dgrk^(w#hCK;y~y|$S@a*k>GX_l>k4C`CIv&{!Ig#~U=PiS(<_X~d+@4xR)
zg+&#~KXYBcRD>Q{uVD+Nx+R?5c&N)YT>n~ke5UOtJ`QcWgBg665$U$t)gJhWx^prl
z*NP^`i8zYVs1_npMTDO7aWN%gy1VpSa>Zj&H2c|lTDA7CK~bUuD4nbY9U|g;KV0r&
zB-DP44k3G>v?&IUpFVBxj%EKA^x$bf;6_8clLZ9}Lp+>qVul4>8V1ZwPD7{LS%Xwk
zw`m^iy4^*&{N$W|%}g4Widj#0RooP)i`ecEpFZU=7IZ8=(R+uD*MmJ#<4ZhNW^I@k
z6xn^~7N?rZcWsT!q{Mt-KKwnhi7%v(UPV`~)-b8=DMwMtm&esg+EaR(*;#_q_01s|
zm)64{v8a7_?7-{DZUajc{vKCvrZu*o<8_vflH|PO0om;N?eZHlntXTPiYCXE_W%ax^q#yGO9SI(vO{nte_mT2;Ft|BA7u
zj9X%~xL|x}y!O5ek(e!PYAP8+No_Pu+-<)be~K0&uU5ahw(F<#O~$fZ{P|>5+_~Et
zm;2@|#O1&7$O}2=fRA4NrgyDOGj+3t!gtL&>jvx&bjm{wZEK)DNTUZd`TePvzcRYJ
z30&QYWOBMF@JC-l9z7#>2j!yaqy$p&T#RbT9wEOXhA|ri<~bjH`%YRbfZ0weEQBy`
zgB&QpyR&DMjH`uZuCYm<^nM?njJ!drOx2F744>!a5oKZm>0wE}w6N6j!u28^`w-ru
z@ubLt8Z-Z`QU|w8*xSftrRv$Xd`LTVbI@o_wVDYHY{kNW?=>`t(^nDdMXh)T@e`+y
z7j-)&m+*Z!UUuQM$I?DiG3%Tx*mjekc4QfF86NQVepz~UX>`G<6%`U>`JtN5DyAPdb2$~RH3TfK47?l3sFxOJ9}=m=CtOuj2KZuQ@)
zv2?A{ntc$j`r#&OrB&sNSOsfO%?kQhsAh`S$HehgY#~WOosFSFW^kMA6LVKnB^UZ@
z)d&!Hfe8fh1N({CDW6BtUm=ZIEZ4B0l7;wqNk^%b`bfI9(iShNzpWWtv;?KTn{P3h
zY{20tHM#PFyoBV@O$z#A(o4kj<@`+w`eQOny4qR0qOdghuKdzLG8CH4N~hSNiC?RB
zR^9kI5q``Zv*%S=bx%o|1MqkYx{69{lu>!AtmY~$#uy;5alf_pyZKK)+v!~oVY-mH
zUO5)e;#oAl9AuViUwA{K8Me1XEO7huYQ)Eg5xcF|uz?ZF2sOn-#dQ?xC<)FgYcv@?
z_jOpq9|39*uR$ZH-qyKG*8(`oH;{w)iQaMCHjjC2r$ZKXlRK`%XN%^I{Y_^*jA{>cUgOriO!9uHUg0QB+$Kw1y%sdQ!*J$Md2-IZ
zzvgvldeX?`XN=z^Cx*g#c06?F`ve-ojH%0lGhgWWegb&c_mJap!p%LT6Zlv5n}5lG
zeph0LVBy7@ns3c*gZmxvA@btAAlFQ#`bQ#gg0y6vVXD*^-?)jF8YO0!ujWLH5=6{p
z8@=i29LuG3d>U>Ay@A=)x`}H>$!A(+?b3R$9XREHbE0L~<9>O?Sfa8Z!76D)T_4FgVT?SFk^a;P^SFNK
zI>}qVk;LyRifah~A?)-&bK@dL4iaZWF%6XH&=xo)EhZZH$|^Hzu^vu()F1nJVi^O_*)@HidVR=W?EwoP^!DkEf#u(
zZ#$54sdq
znxuuw@ifB%<>Ugbxmyh<)J}cj`jcYGp7$jsG3)pHd^bPJd)f_4yJbo3Fus$qXM|c&
zs8j2_pip<2>pebc)y_=_wj#Tp)VdwU}
zgxtztZ){p7$ie*eF~H6A#vG&=N_5RRQx5?Q<+@rgeB0D*;vZd;yl)8gNs?@2ZGKof$E2$y
ze>w);Tya`zg)s7SEJ<@OZ5nPm`97Qj;Q>K~2L6R!3mUiUv4Yd|MOKYq1H3?xd3h~T
z1RNN9il#HI;aWBLi7EwPwZo$NxL!L_M1S(+U<-2?f1@#+Ax}|d?JHxIZ=|7WJAb|y
zl0FeLYTbIYnvmfnsJ7WNQJ1_1E7EU&SqUx1dDCtEmpu-I02*w?s5n5&Ay_o+7kY-!nz@
z?vLAe97Yd;PPi2iAs{)TNZI0d3)YBI-KMzK
ziHW)NBF=i8_-Gng)2X!zTJXKI$}RQSz?snDRL-fj>3s2i!6~JpX1?90ri%mGiJ}*h
zO}1H?@nvqYCwg-qq>_H@jz2@IIpf(R5RKd>9m&nFGhSSaJqjN7*z`F5t{?kr*Z6dV
z*n4*Ap{a5o%$HphDXE4HZ;XVyDaCFKXDKUm7F5RgCRqe!_`C2RW#=+w$fe5cy$R)
zqOxT>sd;`N16$+XzD7}Ir*CrML3<=sZhLZ85K$fRC%&mOj$jxkzVffrhn9-mdOtBd
zUNP`;Kz~x`!{Qs=P1}~WM$fVaHG7Un%JoVR-LWJH?xWXbAh4?kBIqnT^1v5ep}fJ;<|
zsX-W}+d>4#adl>wQj}u;%nj${o)D2HDNOczUen!p&FN8a0xx)%(QKoIgRa>_`a=K9^+hw
zbUo8vTqw|SGQ0xV=E*;UwrtelPCIwTjDKPktN8kHPdTU8{d|l0mG*Ah{y9jgrd8X+&uRUM_^?OADoN1XxYIaav#dQx>xT9G
z>aq&vMx!*Z6Tt0w2^;9hGl_@lYDhij7v1W=7cwyh#jH14@(x!>YYaCNM?u561vJ-5
z;!gw*p=rmP&!#IE%mthn<07gzP9Y8t)hiFss!`dMV9U3xWf&yr?6`XhlC`SVk7Tja$F`6HVc
zdj=KO%lwLSX(8Zq{tU`mq?C1%;J@8Mn9R@jxH2~S-=I_Bp~_Q-4~vss+?jCK_zAp3
zi2j(=D+B_V4OtXrU8Jzk8;G0ea}7qvFTU^Q?MDNk<%bifSG3-AeC>{~*x*P-;{1ro
zDq+MqGdLD0Ubw*Gp?!aK9sPdqJBPWGYcLKfPi;svPGyzeq<^L8_
zQb<3pz4HBog_cNOX+HT!!dLGAV?e+Vh}U_6Q(!=8^9aeiTm0w5uUX*7+k$o}+5|pV
zi%llChGAkcKoA|WMg0}^_eJq%R^SQBB8ZUoRmNR#qQX$oJ}mfx&;`(s?}^U%UcF}U
z4GPgXM?2O0&*w2@|M%kGV@Qzz9%A0zKqcTwD}@PVF|UG59WQ)2onGx>+n;+W+^hXb
zkb}2Ddo$UsD~IWF?0+87kLo;Az#&41aJ0gjK~I;wQl3>{j}Vg_e1a0_z_}UvgY2)V
zJA?rMiRV}q+?0bdD2VVAbbu~O49bRG&CZ(-!=zv!#`m9Zx^m|H#le(WR}{Fxx5x1Y
z9z*R1?3JjSZ$*tRUy6P&L01JlnC%I?!>}`5HU)CeE)NlkC5A&
zL(u-77*37^_#Am*)DFPM#UuX#Du8cBrsMqQU9YeTHe^E%e*5dchmMm|gYbnXjGuLp
z$yuYb94lLsaB|^RzcEpIecLr2op&^KJR>Bm8Gm2z68as9xj>iUOZt-$%$Y{`3t{Z=3{m
zcrkEipjY!`7>GX(mWxn;Mr$*hpnun(9`N)XZalCNG9wBFrfHIr#q-?l5H#=Mw6g=u
zT?atfRioC7OhV{c0lfuE#f3+{4`vTQk6I}LS=gN%OZ``{$)}u%u;LQx_&sf*5{y?!
zbX>Ztz(hc)3bgXl2ERJ)uus8=h1mZHS*
zTqLFEuH3~=w&2-gJ5Q}TN5(khm&36j*R-e(?lTIs6s(}U{lQAs&t_XmV}Dli&+AiS
zP)AUTV(fE?0hs253dC;C>FSn%K1&sJHI4kQ)9{}`r+6Tog{CBKCro7Yd$G-+zlka_sq){(R@(tNMSF2G*azYjw$PsR*1!^1C_yk-VU6(*YrorbnMk
z^xq%n|Nc6s3$gnavb!3<2aNaes`NeqfhE1eMji{favt)%-=^c=7RrnhY>f)|cAqn_
z%G5Z^BPHqPpdnNiqE5^fs4yx}YvvdO<(W;O8`6CbB!gndfvyz0t_z*{D}QCt{~P#U
zkIu0}45&s7iVX$?lTHCdQ7ceJP|^i4y51YZsz31oo0x4Mks$;Eh~&8K*ZzB(|KqtJ
zwL;(prFhN*cD1)Hy2V6wpZgN;wBG^|D3wrjdqHW+r!d)B3p&Tih2}l}|2}6B7kJ~H
z?zZuWmGT$;3AVBgsLsd%S|EuhK(#D=4bh}$^LQyacowtf7oeo4*+Ps07%Oc+kEDV+
zaA7j!y;8~d8;kHVK8ggj|2gA
zBI!gP7^6<(#M#dqf;KlE+RIV%cT?O)6mz()mwcN*C}p;4SVAUeZANr`Ybb*G*luP=
zGCOKI@Y8SK^#v8jqC9{S_i=-3++xlVpbSo}cf_iJPEaqWm1apXMqANzQVU?~PulhJ
zFKGjL$Yr(FTKEGXw=lW^#5Juh&JL>zTh57DP4bfpPv3`*ACq$7xu7ojzNYmH%zJak_tJH8me1r7IO6|pY{ZXp0~kscM%h`@
zcpL8!{~3g8fMhoEIhA+26?amadorS?6tZLTwE{=H=mF1SjDFh2bQrzYPOcrO?%5pD
zuy*h~ThpOPmn~vrD=@hW_Z2)%Jz3;#TDhsEOZluD6uWF8KvaW5d{(p
z#gnGliT7WZ9cARZCIgE>{RXJTh02TIE+GoS>p;QIB%n92BlP*KA=g5VCOu9zHo=(-
z#=0IWg3${W`aU6l56@Zwtu>+?;i*4vL6I?)est0RXzQ~k0BUdwxOV|gP0jXJ3rIk2
zCfN0Kj^ik;!wbK#Ss=6kHi5o}El?2pyv$aF75pYj@N7jkF);V|WwCaH-US7GnjgfF
z6whXoFHVh}TClmF5I*XE{-I3pbmn%E)7Dh1fjieAWFPxYk7<9siycsiTdJ8p+jL^3
zFjk+kc&(S(`#vzvbmyDkd7ov&{!rh=KG^y?Giw>J#6bGR-CU^0qfx7`e=h}+cde)w
zyropdysc+-dR;U|Nc4PaEa2hBWyBDF75g18;JTg#oRb>=xm8R*Qwj2
z6DoKZqDwk<@asJ4KJ|30f;_=Vm(F?Ez)||C`oey`&KY!neRM!NhUH5h?FcN7jL=S5
zg=;RDF0#Y~q*69P6lPISQlZ{(WVIo@9r$p&CHe
zn@R))PYdUso&dTIWnK5WcFX2XV_Qh~wi{hP^S!yLT-^V$p%FK~sqFg$U^Oc~c@cVX
zjh4e_&
zxT9V%gYQp&HD8+qQjUscz8aNO>{798ZHYh@rgOPKmoU?y)#a!6jnYMNYLk3MOUy$c
znL{0(_8FpWLSJ9bxMGkHRw7Q%<4*S?x?2QzCVm|?GF%~Z%MnX-$
zGdP|N85REMcb{kbsyg)p2C-C6WCvw)EIiCjkPQ=!W*biRqPOsgl?zd+^2r&ntagUM
znJc@@6pD>$jbg^{`S*T2JHGF?W7@^HuX0Raftl-U$CvEChv_qc>+PM|28+xxVQ6$Hy+
zSqjJUP6g1VT6TL#XazJ`!e;*qwlcejTth>}<8mr;jal
zvm>9Rempz`SBvdMaSPjyd0jcegYh3y6Jk?ey>JA?ZTa}}G`eF?ewQ7C>|XCNgVMt(
z*E)o<82?TO#bz76PGDuR>C)@Vx17Ji>m{W*QN
zK}ZKRkA^7_a%P=-3#l@t3JN4ky;tXZI)PArW*d4wQrUfpi
zgI@9mdkwgIv#m0zTT&t4fJVNyu0!YraHt&EMzSnTM}Xu=2W`6D$34M>h;lAyN~Y!A
z@H9LT_1DqVY&+*0)}D6b7G=`vU{{7AGjLF^X=VTyq!q%XJ~O{0D>{J5RNWk=Z{;A6
zY*2S|178ojSlHX(qH3{!wvewtxJp{_Z^xGG#wEWfwzer)wM=~YjZ@-qzjWTly9{FP
zT&G61U(ZR-FTFXT1q>5g1>>!lxCfM4+)>0R_dY(@@ECN
z1W$hzpb-{Y^!caD-4Zx@bqFx#MUHdY3+|?4X(u2o^WjChGF#_KkYSEcN^MePplj+!
z4W_x^9kUbR!ZeH*^qFD#xe?myaEBn!evrZFv@hd=y_ZkXs4a2l1386JKN2Mcq!K2j
zJNko}A~TTBw4HQeK!U`wEsEOggxO;|KnXmntD|;!wBwv3L7MTwz>mE+xtvMTC%(I=50
zmjrA#OAL&eqDSWfKAa#*jTeP1rV;0pR#;IEry?Z;Eqn>DO?CiMOK6)3&pj+d9_M>1C|;?NUXNtl|{%
zl`zzW3iYK=n)ZD!Vl6uDk?@m`7@m3+EE#AG-xGo0lkT947hfiB1%Ks5Mv^)8_|OisL6B`h6<>GJe@2
z9Qg44Z?D(zq%@}O07P|O8Qfo)RzSbGYv_2b)G>!k
z(nQ|6m>@hf?B?CVul+9EsoOKai7RHiCg;FMKlvFVX5z{c4%&&6!7tu`;HHc7fkg!j
zA`>U+uXO=R2=t$=Tzm^R)qUbp
zq|l#pe^DBa$$8%#?}y{K<D9W+9*Kvs!QCb0z3
zyBUM`e1?UskL<`V8(VE4_vqK?L@$?C??$;|dFcXpQ!oTIvNOW?IqTZ+F;iz*PS|+=!hjT^F^}UqK^Gh_mD|6L
zq5xGs{mKkgx3L4?&Pe_el&2^&jv-G=U*rb>iKP&F$yg4;G6dl$pg0FPhng?h?ZjsU
z8~3kJ1+65Pn_bFwhxzK-g2ecvG%j}T#yN#kLM3(c9F&7M1s9v
z3b$?wsd3@^cO>xUIRH2L-;7Mq0WZxR)t+<558btY3pj_<8blng)yMfN+2;R7tsmkkatx5@F~XR_14agOhR*
z@?^&PuRPz23rj^Fe?ZQO>4lD+0}q~!?Oum8uBBgABv(I>!sp^4UINj3a3v$6SGN|>
zG4A&_ob^}sB-)}$cIy|Xr)Tq8pa8<_P#O)hB>(g2Oi0w-TxCtX&O
z7)EO?0wT%!o**PiR|KsKp&8*0MDCr`4Q;zQhpT}akCEFQ|1naCT_o2g!Ia;GaxAcb53H%a!znDqZJNbZlCZ>GR^GHPT2)-#4-LR^iw
zVq(|E)aYmC&23Mr=0}BNzq-7!C#!?tP*v5H&W?qdfxZtxJ)WWYyGrHzDgYTu#8Tp(
zv=2AzuB}koDF`=sqWHyV$D*X|ehIy~gXD_^AOsqW2bA~C88l;s9Qg7LPWB2AxLmpI
zvy0%dIJu1<$0DN!z_|4tyapwFc0dBuz0yM7O?f!kD5pP4XU3S;_?<1KRnytQQw4J|
zh5nIP>wHjL@Ssb*FY-BGGJ)v7!YL$gA~CQPgIp;&&;e9Ok6P=ncE7MSmdjGzUXrJ4
ze#V#7aZEunt;|dtbV%pkYa3EPqD^oceh;3JC&!ux_JzT6;vpq2jP
zP580llbbK{kxCP-?=9vtd6v5KWM`x)FaXoapKozkScnF&AEMkqAB3`<>td|&1OVwD
z629-R6srUMLK1r3z(MqyP&g*#rVJ~76Sf25fS!jSM(kXE15lE;jHEcq`<~sqoxWTy
ze_2i7d&LI}wH=%SM)Cq+Bm>il=RNPAWYZ=K)(BVT$roiK&LHY`d`9(qbGzZmr>*>8
z>m$&$1tL!_ixC|9ct)}oh(g$~be4*-GXP%wv}EX2cP2OK8FC7DqZrigiq_<*BP(h3
z(hVIkcq6DEfUHrA$z0FECI`*=1#qcaa37UQ`2p3G1}ROKJCV79-OVfr1yv?B>;CW$
z32xSWYDH-m!@vysh2Q?yihOnf0qz*zL=D0;$f8qX3jqh*4#wkLM&JPfw%?Z2SAm!9Z5rFIkwc!!bW_TnD
zX0T_8d_mgd#CA8rTu6l>>;$~_?pA&d^2a6ImY4s48-M>OH-`w(UkATO1NfzbpI6V&
z2wb=0o-uufyV+H`G
zY7ED~r}Oja!lbKs4*$XB
z#WqEV>CP0MUkA-k&&moW6>Ln&S}CgnyivSY%K3^n`VU6!@@z6rPAyBdSy-!qGsI7|aL2~v7(*2+B
zWi720_ZVlw)Y&{@%#;-$^@9R3wnGN}O!1PnVF!e$cXWVep)(=KsWKGRHy3;C(`PH2
zwS#Qhts-#ME#T$jtpJmx7#nF;Lj)ql3;w8rvJ|?Ow_s((|HsOvBUV;E_+2|VXbz#x
z;xvi#1`#ZMv3yRG4$;>s5P`@iA582HmcF*)J@f^@^Fx29cBU@9{f_js+0;D8%@Fjs
zk)gjKWiWUInmw6(NkE{+q~oGeXg+|sxx}DoGP?!3lo0Gj(yP=qoh~rDsMw7fM%N%j
zEq1CbM*Gj!jRm{j)6eDh7{h?T{xHzlma;QM1zL7`)S5|40FeylrzgBw{uP=rvcJZI
zaQ@5(vVyeOY%jqmjeG5E@MaJ-P9Shf)jb4Ii8;LJ1u>We%Fiy7Z?Sr34&Jfxta&!D
zecjE);)hmJJkol&_D;tO3_kz92A=sp)LPTK{#YS0IZzGM+z#>^)j3
zOJUl%fIIvNjMyLIjc6jKtCXkXVpTo!L3PY!DRHTC)F@t6(+~v&FC1TzCq$&to>(@i
zBz2HE1L&*0)MuQKE)`6jXza}v@P;F}DHOA19Mn1Stqh~YvG{*umunpj3io^c@D%qkdyS7b8fIgu-1HfyZ(mf9H5F-sn4_al#CV+b`klYpYad-+)~De0g^~
zb>6cT0Gkh(rRjZKwFClr1*?cFcoWDv+p}T4r%wVT^zrmucPK9OzJ|pNQ{&MrOJ4u^
zsv87c!-?iVlQOGY9hS}>ZGS&7FjO4TyTcX8{!aqUp$QQDhKmEW?AI#VE{4<=NqWP@
zYaPzRO`e{02J>dbPufg*zuXo)WeLx+^S5vBM|j#!%c;Ji*KY1Tft!D&K*3D~5-V$j
zsq4{4Y_4}7U%HF{1hTwFGbb`OvtJYV?0C@9r5tYnI0>Y@I%4$vGW`2Whb^S%KUV7R
zOCK?-er`6EESD?4xdPCUAS_YPj3PL!@pMOW7!dePx{6zF@qV8K?frD`Q~`cUEpXXe
zRst!ch2jGN(OaBK!?Ge0AtK7~c#%$e10-Hd85;9Y)Lk-_#qqW5<}3ig_qiCWt7tp7jckM-3)rb178w(
zmP?4x>y_h0RCY=D2cq!I(MW&5BD`CegWem9v1`bFZi4x=@Mp+&YD`*Ti@kuv*iCvu
zHv^#`YT^JZnG6pUj5A>VptMw03slY(st5=D5mL6bARETOL$g$~loU{yRdN#bMM5;g
zLIJ=qTJ8{g4|!W+9`gZ2^6X-(Tv?jOX$7c8=j}vlI&O3?K!1@_m^FEvK{+19Z;R!B
zI0x!yVxlu_K6H1!?B=5K`TyHY?a+jj!zf!y;`*_KAk7o1$XBk094m?`y?Z%Z4qb0k
zXsWm6VjUcm2Pfj-Jva*t~TGM>UOItvrr8+5-HJ
z6`(H3dk`k&J?nY8l+bY$xwXH?w^v{U)E*`kOU6{ej?kD{d1^l11a7)TcS(RFCcA>;
zofxQGb1oa8)x=qixcMF5vb{37uI7?<@h$xm53~WknlH2T1oq<24NDk}zOijr%lz3U
z$XnbANIu`o?#!UW*mcW8rHK589s2zvl!9zL$h4;(yEFsJlEKkE-^r{p-WJQX?UDFO
zdzSNVB3|k)G7cg{sX<>JJ_ap}0uB3^VJm`qFY_o_z+N{orzL!kxc**fB(A8y)g9QK
zfBhU(zE=Aga$-XD5rVbik{j6buG6^UQxAy?IQeOeFnhERNDL5xW$IfcFJ$SD6&!_N
zDtAgpD|P}5e-ikBcM0#5>&=72wLca)o7Bq1&PR$*A1m*Lzs};xO1~{V)jYii=rh
zaMIi@-u0hA_9eI@qzYi98lgN}tGn47_!Zc|IPRwPCRaKQ8q4PI9{{J$Ouu=zle7Xp
zIuD7V_k0~hZyM1n(|xStoA7sRD(8wGVWqmIu5;M+`HRj`^CNA+!`k-z%C)|!EuNoh
zrV0AM*=a|4l=v32AZm#X3{UXqY)4qWn9SyAFFhM?hc92lS_!8prJ!7EfFMH@#I
zv@RGoinr7|1RMi;O+?Yw>%Sonp_Fs*dyDKppTp{qAzNtpHLzY&mVM6*
z3HpF!KHrVI0CI!nvR(>G5i+E*jK%tg%t?xo5#((b#hf2ZTW$Pzj6L!kJjzD?#ZT~{
zo5ifmSQ*l*A670oaUX@$bwat0uQOi#jgUK82p3wO@zxH5`f*0TK5X3LuTn
z2@j^c_x18a$t_`+Xbuv@M0Aoe*aMy-O*+Qkm!%LtS<0bs13tCEdDiE3cBAwXhR)Az
zbrxtT1=rcA5=X)B=Icfa+L1N^}Yj=qm44)P*F@xWJPiv!gi2*eGPQ~wyr
zW@UM7K3wcn+^CaE>O_@Kj`d#vEtpnCpx4xmeQkn|m42$ToaN&k)3dOZ@3WI@W|nO
z{pdb2@*!yH`HvLIV}pys&&0`ceRmj6MSLrAvSnx44Xdv1eh}9}Nre
z9vwTwZ~oB&IEwRve`q}xEcs*xI#F8nB#iXw>EO3tk)CjJ^ieq$P1i9gxtPqFmZ}7`
z?^FH@(gyWQLR;Q`yPkVRInQ7liG7->EKL(iorXxIUv-58*i~G68;FdmfwpD~*(HPQ
zj(0?(Zk%k05klgCQ1ExYOh}75gl(~V=aBe>s+HfmQ9JsMjrBjz5&}RB0sD;PNsF=C
zCd`6OoLl}GUrZO0!PAzT1G6NhFKgoF&vqp7gb_>a_Tm+Yq~)hWex~wM=%0vwBI5Kv
zr$F{2yW9wa{3t~DoQqV?HRxrr<~C!GdFJ%cQGXk-Kar3(kJPKVXyU}FYq&cFx2^Yx
z#!SB#7cAb^Y7+|!Mg|E1XLv_&D2;p8begsq|p=JQU>dUx4BQtNfsF9}g
zqy)~H6DY|kA-k#$0=w`o0k0z>Ru#bkiBq7J;IMH683ypt&nQCrY53+I
zyH$!W^mYI#T6T{=JE-<0nPI^Ilr$&+E+2^{OjvUex&5Cd03R4}NWi;u>VT2BFgQJ)
z$R~9F&gm-M&v=a{9{J^U>9BKy%$pYj`Aj68frRW6in9EM!k{cFg#=elCq>{9&OYA7
zj+wjPY3#ZF>R|3Bxaw|~{@F5v4^fEbIEJpLvHlV+w#|g=DjR7goX_YTw(?pZ)Bd)C
zzefO8dzB8k;c(@QIb^;rt$`g{0ZMG|+BMz0?gJ@CIt#STBvIgBLOU)MXSc+MJa;Yzbz6I&X18SR;EXX
z$G_ebb1^~P8NTnf_urnF*72kY{Q4KE5!m7PUQ>sAQ_&$M0v20-T+wtYbQ(c5sgvzu
z-!g^y2rwT@(@}()QfyhTD_ez2pZ_>}7tA|kP6_IJZ4phC5hSh`GvEr`f8PwmPM6__
zTU(IgAy!?&(8niGDca4NuQ!5Ab3l@;4j9iGgtqOyKf8A5u4<`)-`Z3|-O1j7Y>O+w
z)-@hJ_ERgZ*%o-C`={ZvdeKtD0V6B;Xp_x`MrA4O?{c|+Y^w#-r~>VU1Nb-62~f2(
zlUP50&2zhUD8C0u9XDq-q6SdMZ}=Rfi?E{|=M+Exkhnw1VBk?>qA2gqK~eNfY#h9G|P$ovFRQg+?K
z{ch8?=a98wuC|=;(pwTmaGWwD($9}&%TaOw*%QPfY=T~ulz$2f3m>t7HVRH>`|8WM
z;Z4UOiTmj#KRjX#r}OvCi&~KV;KJ*?U*Fw81;0!te+AMZ{{Toyq!(29FSzjyP4picm%5AK0h9Quz
z*K_SPpq;|_)jT9#fix+iZhA9-d63l5AnP!3LHS|Cb2PFw&#&Ul`&3TOlP%_yfM&c2
zQc3L}Erh`Ife}f8+=&4VjK@L9uVtFsSZqA+IptKO666gRj#AQ3X%y?49?Wml=sJrW
zX40lG?f)F=dJwp)m3+Xf`e(!a!v?aU6f>*MBMQL|+eAlg*mjBz&)UsxySUG6rtVL7
zRFGnsp+N9r9=+U(VJ*=iYA07jp(DsUd!u93UyLy&S#Jd9`1}{>3c6B$_!O|>2enY(
zfoo++Eay4Q)r0vwl>7Lr6yLGW{1r(Sy7x%qScusSK5EoEmjsLc-WnCYUhPalx)^i8
z?bSLiFnmXM@nKb375$X9&SK{p&%-?ClhEabpM$JUb)b+#{eR4X9OX^&fUDNmqx3_8
zjeELV?~F!DoJ29rZ^1aqM>Foy_97YJ<$ONwt_Yyi%4FVy6BiRAWZ!r}+9$DRAC&WY
z*NO=xTe`RJtI2jm3KR$kyZbmOtyYUvI4{U+NB%#)zB;bT?ECtHAP6cLzwK)PF`ySr6j6ai`J?(PN!5$WdANO$+;#(N%P#@}~l-hccs3~UAnIeYQ{~GWxlLX({AZ@ej9VUVHHaW1uK_P=jZ(j;p?AcdgoBQ_$fs+i`901L
z1_|CC;3t0pc+-}|!0p|2JtI%?+YSA+vH_XY>
z3wd0YS~u?y7nK&|8-SCZ7tqzI9hTaAieo4?^Q6mSbI$T`tkp~u6sUOtTz>>kdd{t)
zXf6;Zz&GfUIK9d?G}_w}5oqeTC8Z8@=T~}_czI*g&d1LANaU3Rj^V;DQX3+`-h2h>
zgN#UiNRWsRWouyT51z*r*b^IXNY6tR+sNt6THI4i-5?<4!K~DlmFb7^928{FR%{~`
z3TvIazR-+UIFTfxyb2!X4oCC-h*JIzY-KqKmpf@KP9sZpjS8%04zwLG!BZ6JF)~`U
z;tv}seM=}`xQCRaoEvukbROOwyI5JprE;aA$8~agjYpdv;tfxL+hAv~K85V07?DBc;#0gvD-$CT+x}bdfdgVJ|kVnn6&1
z9iz7Iz5kof-K_0zFXF@E%{vC|S9z_!!a>92IDO90xsYVX
zYm=%%>1D35Ex*Of{W7>I(sD2NP&nJ^iCgggR7q$d4l|nxN6JYJORlK!f`e7zxLe>n
zgv3G|ut{JRzJfg5FzW+11pETiHb7+Bm`F
zd0P3(c%g@|Ft0$Gtuxv;poHLl(mXC&eFN9DB>@iunXA_$x{04myETT
z-2`Ib|P09qsj!VK^`;;
zFf9~IOB9^c^M=}<27~q?-!42r5p1Rg@IYwpE$0Npd*)(fm{+5f
z4hldv6}+W7^RO}Sbz8CQ^icCo@7w!1dNGI~?Qxet{kbTe8f@?sM5HS%7wPrqh+s&KM~s2t}>Ee&eLN2Rl1U6Y%)pbg6kE=+)bm2Um;jO!LCLQ
z1Z3T`-&h!&WxrnIT1~xG9)S>w=$i-JaYuTwAbSn-V&6638fZ2myHyDkL9$l}jpf$>
zQ{K)f!NpKNPs(J#%YDMCKBu&QNi(PW3m1P|y7O9EIO3B)UeWX&uVIWHWO4w}zd?gL
z>(lj8610Ecr*F(9pA^#GDreB_sQSi3zbcafeg9kZ3m|wVu&4_+(@Ux5Im~{(x7Jad
zJ*z0~@4Kp*bX%MAuG`67r`CP(g2!Q(5t-8`gnSmx6RThot9QEY
zF(PV)DTPbah3I-i&lpvJGUT7_pV1c|q-<|>p%!EMTA2(Os~>o2qgz2SJY@(DkpH5!iRw~Fy`e<#{U##KlE($h
zzDjT*TwwxSQZrnKH*Y3_!)?Oi{9sjdK~!(qHYR4%Ht{`SsTv)Prxki79CVf739
z%j29{ne5|LS`|~nq2EKJU#H>CxqCa^8k_t6m<7@9_v`z@k)#_qOwB!bg@(Q2z{zG?
z@9d50NEI@_tx({!wWQ<`SrQtj${w3){`B~=Y2$-%R=rS@IVsQT4BeQ8>5rK@8%hMD
zks(l(8p2UG0buBG;dtRWrSKzok}6(V{kUGR5V?g3r=FiOLq_{k+EdoTfVvo6G3pRX
z?|ELOI26xE2Hx-@4!Oi+OcU}Vv4*BLS`0MdnFNgSiE~VGaO|bf&)>5W{42dxe4tG{
zFT||@{UXg5h$$(!FUIV&HWZDkX4%O2UC;JpX0V8~X4iI+sLSbe)b80H8#SXGt<@qp
z!Zz4c2{xQ2ybUl|CdK*t$p02RTH{`8oLHub5gO1sZFlBHH@A$ksD8b&c~~=;yC~LG
zME^2)Wej9D7cShNxt=@4lny#VPOZz!oq1=fsOBcejd*)+aSX3OYQnD|LN4J-J?4wq
zDh?cwTWp!)?d8)zJFkFmW;iW$cbqmgoj_Sitj2n#$vmoeM!(Y8~nESQ4C8X{nJZ`jK4vpSz-+e*3VsV*)@Vn*g||(J0ytGaoHqT{>6D;aVPX
z&m`CL0sT8lZ+(E~&yTHx}If6Cu|Wol5lvqRxJ
zL~N$qSBMPenuNb3}M99LYYZ?cwF)wpm}j+2&F@IZs0F
z*Jgr2c+59ztGEuKvP`OSTF;Bvc5Um~ME10gUeB!;^RvrsaCYnqFnrv?Xt6MMGp~B4
zBNrvEC;Q&gpmSq!FLCAIPT`vl79d=*^FZ#~sFUgEl)i)rN^*?OrL*+OG^D+ch9Rj7
z5^yyH^LPL7fRun^<*PcjZv-NNA>fKbkc#ia3l~IoHT?B0-&-y;Nw*^}ubpYaBpai}
z_7=P1&p!bDi>)VXoq`o}FKCpTM#v#auw37s(BV(-&n-}4TrpP*G)|*l0d(28^TKAs
z%VO&MEuULz-3!`-0%n}E0!-P;Y-GF3QepfXxn)eT26p<
z=wXD&->4{v)uFuUU@(vo65#Vi;nAohT{i)buYZ
zdDvU_eKLF8(%wEqoL+r6Yg$nM$&Vv7S?7eUP&3y^Y)fPk_?8`*i2)cWOB=W3e0DR{
z%7IYfsy>$-1Mj+hQ#V_el}neB%j;_7gk-3P&jGbbt1-*Qk81Ms*yg)^Ll&RvMR--A&mQmy-Ts82jnM
zoRX+6t65l;t%kEu=Net%*yjf}(A9b!cNJdt$R~VMc07=93g*=+>DYG|T)Z1J27&|3Gnmj_zuh_ae`d%h0`?w<5?J^<6$=aaoj}
zok?K;)ijAzDHx(xaTt#?wbdSCGN6#kQWovMox#V=9Q*15cO#?SLR4-^Os|sal_mkz
zBe}w0m&xHZVeM1P{==!q;QDL7q&gcQGny7wU)!1M!IN(nV&=JK-z3c(t7E{{I7C?^
z%x^fT6w|IBR+M*AdO$lI%uM^^zJo&2zk#@KUnb=cj@-*wyc;I+C+}Qvwc*!4cr|hK
zZ46Q8)=ExPV+q)u6H0TG2Sy5yx`saGqGmYuHYiTXndZVyA`HRxr0hQwYi|HsU3CyE
zjIVdx&3R~5NvPoYYvo&YsU5t-OQI7
zWJI@z4eH06mbOHT*e~$P0b%$-gg2#H(`jW@?*+7O=4D5&Mc!2D{@yEo5s2OF&7$sY
zb<-#h(G5=a%3e$}GGx+Yl-4z3M$AD!0=UD}g2Frkl;tf8+>SOw99IY1Kto`bRDhwV
zb%g!TwErC}40_^%25FW60;ALdcrzreNrN(Mph`qc0K?$+TDdN-hQ)*Xg?n{=ezcss
zuS97BB&ocb_?2T-4u&3!F>7FT{SqA;COGP8w*Kj9tNXR>%^SS`DOdd51Mm-5-uV~<&7ft|t8<{-VPgpj
z5eqgh!$!ycD%9mS&IIQp&jmi%9XGw9adM@*)Ne1lty)osOQcvFfwGNC>{u;5r)g)g
z%d-*_!nDG;cPp=^v)r{ysbfCs{i#xx`N*uz4Ov>kn}P(y{spLHKfi07^REghT|zD2
zx1x0GXSfGO#*Gldh{cQp0>k4z@?ayE`cG?-CJ~M+{Vf-$1qS0ikt+qDXQheSpZ{cT
z|4hx_QTRY|SI`}%1}J!|f$Y_YwRDVWV)a0{(5yb@#uH%Br+<741V;$1&#Ea^?Qa+@
z7GvW^pIZgGfnhZqn~%&ag7C+WXF@A(Ha0Bw3_``zFSxR+m0a3
zqQZqTDM&Zn>`&qZs6$=KE9Vop?SKtt%nlP5jA2Q=`DAc+QQjk>x`k20Zi(gI(CP&y
z>n^J+)wTnJjLz&*3!$F5tSdR`Z^EG2UZ39W?}(&3i_0AacZv5&2Qw}%MdcMvL*124m&Qx6szh(3hSms1u%)=^Ko_k77`FX|;D#e|Y
z0*ltY*lXGkJzw{ih!>QZ8~o`HmtuI=QFS>(jVa~yPGhwpN^az>!03d2LYY95*%zIx
z5u|J?kQPNsJsad}QMtic0lW8L7{au<}J2`&e
zmF$#4rar=e`U}7P91Hxai^Y`rimqGpgWlq=!hW)upr5qi1bN+W-aN?&>Lxl@rCGkdWRd
zp|9Cqv=-3w?#J4+&a!JJn4Ju0a!(h8>;sp?B;iYh7w#-Z#Y#Q|0wDup4eH3-1>{Da
z)npOdd-?fOH2bcys%qlmuTKH@Z~1oB*B5uF{-?tBOVsbvCGpK9XEuPtF+E*<`$`yv
zhw+~X8PGVOa91V!m0u`JKA)t4W4-lr-Y%=Nbm_jBvZC3W%E2e8U!-;xTYcdk^rVhd
zU^x34ag=ubO>JBqS88T+tqpFTL`Uuz)d4M8I-o(NLOhH$Ai?@A@PC{fe%T@LK)P=T
zkNy`@APwAN80VZ6yFTOKs}k#^l+fOtpA+vw<=<8G!enGtpHLfhgSBUS$ZPic0O!|KQQ;cYGm)x`&AcxKd%t8+T7EC?sl
zbY{x<%IV*dG>bPj5#p`VLE4xVM9pII6a-tsym`+V^zWM>Ho
zq3wafrm(Y|8LFZFeeM1FWs>?isAuwI^*2)UYg=Z&-n7WG(%L}Tsq6&aKiN5(=I
z7q1yf6{E;6PHLLy3ZqD~0`WRk!-fy3A(gbjz$C*o$et&PS7$IeW9xMn#tt_C
zE7fR0h7kWhm#3^5p)o@*puERbP)|6aq~7kB+Lg4qj1oYAaC-ZMpu
z{sxgXcwS2VX60$uax=Cu$7N-Y(o~Pub84-@F&Q+p{&GA}X{8g;^iOfx&&ifGI|t;^
z40mxOA|w5dwqr=HuRGMMboKbHzVr>as8$&GJZwN5rpvd2@L-%HY)&h{keIW5#ZC%e
zvz7XkK607Lv(qb~arvm37<4H;*aRK!Nm94}m)G(4hRZ6)1G|8f+5w_FY|{uJKiC%D
z#Z!81++WTkS#Y={k=u3jbo_?wCEV)s0@eL2w)yVuBTEXF4-0L_-Qs7;%Mnk`lF+Pw
ze%m)3^&N3ybVL=VG0F)Z>bUC(YPB
zpK{f|AvC|3Dt$u=vDBw{mTrYwTjqdD95OToYY=!sl@Q&CvfL^Utkb!&z
ztHDv*m&W=nw!*uZ_c%}X>hXI{hKJd|qUwJK;4C}CGe2581JvmFTHp*?6Hu!+flTNB
zKV6)7AnjZMLYA4!8XryYdz-aI^40@Qaml|CrnK<#dP2-tz2<{<1wK*r<*IP-0;Mw|
z>k^Nt-i!a-T!0dBjmI2<|AGPhtL|)<>hUZ~MFU@-)hC>3D*_(l3~~zKzr%#1aiV9*
z>$USy_Y81@2-;!7d-2~3Hw<>c8jJ^bSs!zo|ACX;+l#6c>M#=?tBjqm^Mlw;Q?qx8dEw
zSxXzlny+w|9>f;hH6OXd|NmJKt`F%9v=#R7nnpgSdt98Fjd(gKCD6
zGnUk7m1>v7RFN&VFYTRs>jV}I-;-(tB
zv&4@X^7m|Ee8+_sPo-@Q%J>177mJ`YUt!FbP@${`
z(DV2_zf35|c4&^skL~997l*8$lR3x%p;9&gTp+Ph&vob0MK&N_
zsNfXiGCz6^ULZ50Puh*;J6^?acQS!h_HSNq#1m`&={WHHulY~`d*vX+Jrg=;>P&r=
zadcrsPiDb^k`w1GkY(;_>|72h8otxX{+gn{cRaY>R;hi$UHTsiy)w7Ud(*@6n|Ddd
z4ZvE>dVs4-knSnwYORU|Z0)A+oS~l&@z>;hyCV*Q;O+Oa7v(IZx#Nu3qokzQWUJHL
zpcsda7w6!Q)woWJd31k#UBe7_D3o`Ru_{-EhF<>vo_t$`;gOLwJQibB+-AeCjvAs!
z>ed{Z8HZfp0<>Q=x7b-@-hbN&S6_O+6_$T}(lv?BT`t6!>PWs7xp4{g?*j$<+~a1F
zOaCW+cvI4eVyp(Y%1*M*%=>4sH#5-eT=nsGc(9l=J<3og9ZVzi|9|L6c)NS{qJl*x
z18654t8fRl@aZAsMJkRvd=8{%sL3YRS|GnX`_oMJU#35+5>I|nk~=XbPLGT>)=SF3
z7l)aQ7_y^cIHziqX4=Vt$Qu;&KghH<*1h|UwDs@Lb>R1XEX^lMv36u(y|j9-aG3qR
zdC#=qHFdY5Y%4Do4glrxPoKko8H+c(p5fhHot%?FYfmL}HWYTemcm&uQxH2R`RLD2
zLJZ0MRjdwDZ^SRDb3{qsdY+?jbuWf8UpC)Vkb3N*g*rAeBZea1<~2_?73b{jM^`Sh
zgb-b-e4_c_MV1EPgWp{+Yqcjr!=;C%N1olnZi%Hm{D}vyLcsA8GcIv8l^Q*R8#}a1m|Qr
zaLdsWfY}2gcon$a(vIu^pZPLOf}7di@57eoQEjJ_Eg;vMsl)!<@-lL7np}$GH0-pd
z(00~!eJCDE3jO5w$?WCF<;S5r$n-E{E{{Evlctk|=
z`%yV(hLFm0!YYEMBig3)Fnz9hLb_gK9Jj9{FU4CcD?}3m=5Bm@y}fgIz=MeG2*?^7
zYEOs5)m
Q4^V_#_kWtN*hK25&q==lUWw&%uUnblxVUla2Bg3SYmWgt$L)?%P!2
z;0a%K22(QuIOZOFKiMjljg6yM(2__?>&
zc)R1BXJ141Qd^Ek!^78V=0hbXKSpgD~OarhUoNyYjljtjpn4f>&Clo$X7=zeS
z|NID+_4d_SJW^wCZz1odhiV$fNkj;RpL57VwCuq@ef*hsB97nA6(x88I3rm+tHu1d
zlVaXmUPmWv)mNYe@og|~;OQ78GnrYBXx!W6$w4sOE>gPXf%8g1uc0aFrN;-%r^ZeX
z5lBuL)rd042qgNy8wOrU&OA6R0n5_xBBeIXk-Iw*5Sefpm3XU)qDal=qGxUA5IFNuItrCEFdpJ
zyMHr8jK(H_g2OQ;!5X4v4J2a91c73F0{lOf3Yk7XQIDQCaaJe03+aie;N?t$qHfl{
zixp6mNuSlQs&`oNY&dS=sNqIvmVTMZvE2K7&8ljxuyINVl(8Uv9*2k6!CpOZ@TqAg
z5vj8;$RWtmGeDiJx!1dTpq!F(RSF#eW27>c-_CFaIPeA|T?TsgKx(uuycgXwXfwQi
zJAb$IU40ITSr-O%pw+O|a4I3fikvuL>*en{268zz{@kglw8Z-!L7?Qp>JbSzdu$$+
zKyfq66nHouO`N;N%O|wXM0`BRWYP)+~4a!
zdXA28#3Bud8;ynA?RT(zH@@#@Z+OrL@Q7iUhO+7S20Njh__8)gLR$f$jQnD4R>*@w
zulUhvf`_}johHDl)Wr59KE=%j`akT>*M7cpH5@I%c=%ai`;|`g0ok!o`pF}MG->{*owg2;~4j6p~CjzaUqQc;1hFG*!vPqiFNgmCwUXG)#oKz8lC9~@cq1u!ngyrLmbQ}0{rHO
zS8ty_Gq>HWUSbrk(%d;+c)8mKL_%R~8Wnl=6If&+w%r_(e3J|7uzD5vTD`dNebXDu
z;hcIE(Wx}%AXSEXaxlsVwK-TNbkS-s`k2Gb-4{A_|!9
zjt9`GU@YWQ=lv2FE`cem2?H(-f&-wnjgZ4~t95WY!iHdK`kfswD97R2O`j
zi~@!rQYhy9rltNP?L@D7oFuSqh<%DLeLsWQZ$HE8kj}5FC}DGuYD3fNoUliazOaUX
zOZFCSgx~MVrdI8bt!2L-_BFX5Ac+y`OJwJUiEAb;>MKEXg{ZQ&^9+9GN+IgY+lgk=
z99}7{G?TBp{j`{E0;e&P%!PbY)7ZS(l@6|!s0=~Y7%sA5pj?g|oggSG;fPZ0~j4tZ~
za!-{dG3vpI2Kq)LVsbWX;4xMRnL64Q=8h=>K2?w7G-{F8?x@3T(4$R#fPq{}_}QYg
z@QMDC;7Y(|f5wKr6eLuXRJcMuDW~AE(%^9?xA%5Fs#ln8|dn4ToKf((W+#yleN(xuZ^UQe1
zc9izP$EbM#=H$C{Sy1Doh1_;Y6$;{Ce6J6zAzynXin~f<3toS6Vq=^Jbgr;Ko-t-n
zbH>7A9XlO+vj``1$)~ebE6~7Gec)j1bE8PBqtu?W{=<8?vw$6aOK=m5>cOU+B3O>U
zDj9n5k3V_utP{BZC)P{MVLCb%1^gDC?yDeU&UO+cOL)0#34ZOUKn%JRE4U}Pt`;k(
z9&O=?ySI9pxN=GfInD+0GR6qYC{(o^m4c|yXl#G7_o^dM_}Z}!U-&kFJ(((vdM)
z<;6;8>~n^c3R8IS<}gq*VYHGw5Nby8^8kpGgdB1E?bK6=5AIR_@%+)^^&NgQDI7$5
zq8T^S@Q?R6gg1+2+IKj}T?~ZDi?AttaKXC_4lA7`d6@APFH@l;LYPDOG>;|+v<2ar
z=LU&jD@|9|^P?T`p
zl60q`PF2ns!YN1!!M#3Hj3$g%_m@J@0zbEC53-3bKqI{`KU!uL!;`OEv;vGNZ5EF~
zCpQ9>HM`DQKU0h1FqbTavEaqF$pdh(N$kG3Jf@-$ZbBsD;JB1tcPcNNEWety72REU
zba@o5;W-~7#z=HtQ3A+RINU1V&>lnzl09QEL%UA_=e-DNxz()TGnujGCE}0usF{UT
z`e5%SD>H)kQx#dX!(RBfwnkbMYHms9@TBYHV%fM(_Yiis_vO_F=vb0e*#UW^+B^ob
z4i(|uwrK5JqE3|Dg5?~p4+%CRSjh*$y=)
zg3%kA^Jm3^FX&4PU)+k2FU5rmOn(WX=R_nYeTMGb75RQgkw6WTc?oB1#FS4;2-S?E
ztIiuy=d0q7guqaTI`hf9{4d28v3#;gcFf*4j!ai{Fbg{E1yZN0lQ)J!MS09iyLh5K
z4sK^jWT!S5t$rFdPTMXW0~D*Avuf)xpgKF;=_%kR>dadIIU_s2jSklVn?vX5!R
zo;G$i2y_Dhf0%VxS)J)njy{mv_Lw)W71T1i9?9mXeb9CeR4yQgnJ)B{52H(HzQ<1m
zjL#ZlU)FWu&`sgYA3XUbqqR5i0d7arz+RZyj`*8#>J-yJ);u=!o*lEufkuXQ{%q#`
z_PhjMEA5Oaxrjnw5F4$qLVXSRU$vJlC_ydKP=rzTN>~h9YYNSwR`N*;SU9gZvK(iJ
z$I)tqndB;-^aXp)qrRkcuI|jW8m)F^9n%GkKRbiQ(Oua0<#10nPbXRDqn%(iBe}M3
z8Fq|WP`u?Zc+@E~9(nTALC916_R<;0h}-NIh}vM(lc2JKCgA6A_dtRyzB!mMw@kp|
zNTIm;xc*L;E6Ka5YZ*Pj;g%X}uvOPDXt^u@eL*&&c01x}#UqnXx~x3s+S%g)=Fw36SdNM70n3wlo0wtea&kEKJ#A>2h`dS9{_qj}9
zgTu0zypR)uebUSdI5swGWQc($XTH%Tgw2eebos#M5YC3U)QnP-EzrxLO57c&0qAP;fLZ68Br#g%NCa#xm|xoST0Jb-kcyL;jv+U@g&F1?cSl$iSa;SSH^9{>2lh8_!sVp*VP)eOt#~7+M>aGQb
znmO84b>+{k=tQcHlUqGa12ud}VQSF|{6J%&<%}&Cm0cm-$x4oEHjF{Jd1>;g=g1c^
zlqy_frD_|F+{zM-VAHI+BNX6OsjPmH_{U}AQv=YLyYtqbG2Kt=pFHldL$ANOZy<6Z
z383k!HMOXv&gigVrQ<1pBxWOWn76Az5M2eiWejH&fRG}FqUY@Q{kSiSg9{hhf4Nlg
zBQpl`S(Eqy*paSp)~wwIb|@wLi>TV`5(?+YR+6NhOyT$AhoHa+a=fTrP6?p0{`C3!
zgMkkwRJSEdYUY9jB|dVR5!(fl5{a={fIh^K@c}XKtq63ji^B|ew<3FcTgW-CEr3u
ziBw#!yB{JAr8R~V)@T7`69Ug95$3IO@5H<;5~F5UrnxrMwEIuhrT{gkGY9f%Nq2%7
zP!TK6!rFl84AHvMG~?>C0^XI=;Y-bBb1XScRp_ksyE8v~U=$;T|@|VC9zr
z*I8=cB_)61E=NL#IQICMDB&U~Sjw_#D<=JMJwLTDo+c)j((B-ge1N2VV%q#7-jakeZ-mzzQt!&SdOI2?_1d$snzTJ9C;V)wV0O_kMwKTwRra!v5~`c|~?)*X`^_Ilc!(Oal!^TX;l
zG>w!$*X$fjHr+_HuZMZ!b^`gGi?AU^A9~7f
z#nlNsQixHqFUmWRt#MPpz$pTDw)La$NXM^{R)CY6u!^p?t)9^p7R9
ziTe}>F)5Y*?DoKq-R$mKjubZI0(Xu?>Z_(lOU7ApwG(pF
zrFF0mkO%N7E+da|8Vp8PB>w2W;!E~@Ey{b&)-}xVC&<4h`#z=67FSA9{t7Tc3J7?)
z1X(OE-f=nF1yy~cid*upry0U4Wpcd=To6yM{WzQX#9)6~;rtaK!Eh&4zXE$Iwrk0$
zeS#3cI#Ua!9e2lRoWJ@+yXLopkXhL*Oj@lFoR0Q@HXS9P=};dGT!LyOwyH-!X4vnl
zoeXSIHX*#la0s}w%_$dZ$D8CT_1Vj<&Qk-kUz0w;h*>#Uc~GAc!fEix96+h|R5vIx
zfU&ng-Y}b)TVPC)0k|vkt7T+Hf2^c6596JqHGmA6@3mr=M6mp&!J=%dE~Jb?tg}uz
zTAn8hGmP9Fzyr_LZmB!JS_?cSx0UyWAL}d`7th;unV`qw?=sQ;c^@gB(KPe2pefcJ
zDElOZPlZ3QVbU)@!w$q`S^xtc;#nzkpcj4195_)-+&d^fkSXJ|vu_22385D9+#>Vu
z9^PH!#)Sxx(G}_ucHR7~CR}fD@2m5GlpAQyKB(6^oE%V}2r2lV0K|RaZFEDbD}`)G
z+6L64cyu4~YaDcQO^^5%$*mnjzTS(h)-7ZzRX~|8IG~)Ll1PA`)||E?IQKyw1W@s!
z`hMbDo&#lej|adj?nsHPfl#DryldYBxZY+wLfAnON!+i;oC|I%wNsoRFmUJO#ef@7
ze>Eb*K*4{GRD_D*>0^w#=D68@M<6(KB!Rlmrf8TUzU#eJb
z`LH_cf};N^K85DH;aoD&K&pkZ`t&ZI@y-q4p`*-jgqx3nARa96r0#j!+4pBh!0no>
zZKpY?n}tS&xmt|}NLD6+7C3sixe0jWkq|+2WI7dI|+08)Itz{~Lt-
za+sV$0g;`P>z0(zt<8xBh!6-&cyeeYQcd=?!$C4uU|xG%Pd;;Yb$K1s+ehYCT{0gp
z!jxESV8#%^_2Uct6&j*AldIApX}4lP9ByU6;{hNLi7l#}0|XWdTLgg>J?F>Y>-84F
zHtV@wD5xT^>z0vS3P~CD`G!oqK#9;`Z1#=7&fKNv$hTLY-t_o+^neq&rzBA;aPt&k
z(#SvLeM2jt#3xM64N|Q_rG~M*T@vgz$tJ7L$r~lf>V`}Rsa|vm9sQtU*d4dBpHNkS
zH==>I3)~73_+=9law)^HyGqa0>Ulg+{Rj>;I)(RltJ$b7)UJ5r&6!y&OXSZ=89%P)
zO^HRP3{S7y+I4(J9nYmaa}rDDMz{1u`qO9cqfS}d5BIkEo}dI-E>F9>AML?DcG<1p
z^5ED}=t(~{2mJy})Ry+*7hYH7hNC(@bB?d6VhSh1p!8u!!HS`<5#u^q;2BfaDO|Id
z(&m(8c*uQap`o;KSyy5_U$yx6oh@P(!KrS+t+)vwaRrE)jO#s%H8BMSOdSMc=scj~xiW1tThnp*@(In>fABjvA
zEH7Cx-CIt2dH3v4LH73>3_$jKuM<<)=hC+jubC3wA0NGB5N0;dYXdh<1Vk={qH!K)9D)D*|$s*CeC{1s0$R6z$Onq?u{st-i`w
z&AGL7P^g38Clzu+tjD&T8aiiHM@b|XsWG&bgHbtX&Jj^Ym!J$n#prH$!|lNKc37BZ
z(fpRs!aL4!3Bby17uxkf2H9i3e}ZLT9?He{#+#eA39U^O=p8!utv+zgyy*TYkLUi=
z*=gUfW0U#z;ghKJD1mW3XNM2t4hsuM(_UV6l9!##RaRjf<9)m+;1ky&G!5!}ip6#!
zTYSSALDO}oBOw&+Tf1%tsygu*0og}x=ArsEZZ_N=Hz7Un5gX2^tIfh)ge~7XU^c1mr|@ayrV)_
zwcN7|fff^bBRf~irtJr$Ix6HoOa>2jmZ=c0uQY(bVTe2Q$9aAc_t5R$m+|Yc=R|yk
zVjwupIJZoD`R0$S*!U8TR6Ynyqks}VBfpJ6F^*&;d?VvXu`@wvd`0WBYtQ5N7x})z
zGPAn?wZFE1-*84C0&SepS*vJ{c4|DGycjPpsW5cXm{2*8Sv}PlhDflKP5vsVE;+#J
z0gUInCYyqBSzJyU(>OE=oo2qde1mwGMFVk#3haHx9>6V+%&-Ryu5@l&p$gc>pD6q(
z701_S0i4Q*A0^QL2t2qM(lVy^aT4xQOVvYuVf^(9>{e7f9!!M|NYkSQ4zFtS>AJHb
zrmaw1I$LVjm-4tf_-6Zh&I&td(;<5FTE_kN0qxv=0D9!F83}#g{*zF&Yj%S5wJ)t*
zas=Vj0Dg|}D*^Os73}-$xW0#a1Wo{n*wkX}VJf9Wd_<~fVEabrd6FvNMOXz)S(`ww
zChiS7ECdGu;uwvka&aA)r)tQX(4r#Rkk{CAm!5wDco1tlri4igWas4@zbrt5x0k!;
z6|rTHQWw9^FVgaUFEuu9>ZngaLCmwKb3`U=0VvR4_t(al;)|gPiFh5?NBR$mtV2zS
z%f2UI08UB2dXod`(RZ45HGJ4@7{b}dK71sHtmP*m`!>+^Y$b`iG3!)$TfAr7RUX{(
z2NPFLf23OY_@;-ocWE^#P>R+1cbx2ioAsmnbXI13X$LXtD&wk
zD>I0F&jx(RDYgfEs&6S|gyTMd80#jFLNLE;rr!AG(Ez+{7p8o?RNg>jyKGLyRGqS5
z?FZT>SA8qgfP28+R-QG+gR)!Zk0n&NxO6JVp!k?lXn__^J~$zUARA+8dT*|NBhYq>
zIWVVRD7zWa=%1Q9A@8qg1VZPgSP)*SNYMXII*{ZZVKarL|;QM45WIk1r>40kGetxpbMK4WQUpgAtiwTLoK-pl9?0*x-ud0X$_&&
zLeN95e@|iBKuE;lV&KzzfDs($MWNy*9Wq)8GNjIypNU=o;qvr@K6AVn;b+$x^Y%Lk
zK9;mcshmEp#(lt&nZ1U}y;ZMI1V{w4xQ}t+c4(nAFQQzpS8*RjIUwVz4O
z{X0xi@y!PKoe*3re}ax*=sJI+RzJTI5(R7@u7O1Lt1n=(a<>4|IucO<~T#Gq>@)o`yJb#Uj6#k()4%7eC
z%T$gY7ydYXxAe}2`6hW_0Z7}2Rhv*&OQBc6B8Zqa3k
zB0(x#Gyyo~eQ5*Ms=s6uzy5BL**Uv;c8gDQL;nt`e|`HfTp@WlI!$g{
z44t(qqf~-X>JTx{-v;GjvS-Gkz##>nNXQ-4XY|VGy?}$MYWmx6_uxPO+2`|#^Xz{r
zVoLU|#8V9yu2iP#3&XuYMo0^1F{04~=(g!{8{cog#y-Ccqjw0FwasgCl=YuIt&
zzVAl}aX{1h-j}yL`P78=z5+E?B&7R|&(!7p&uJ&l@=`>n-Iv$NLErEwZy0_S6&!%)
zEnJZVVyLbwMM*Nweo-C^m?Ef!qi)0x*x?_$qE`=$MQq3`$8(Tx;h^&KI3F(D;3g=f
zz|Y{#8kb!(v53sw5Jk841AjWr(5tSFqqhMC>gUS-KVQM4C{*iR%N4N3;d$6Wq#`>B
zZ-*IcJ`?QfT|mI{5hWgJ7OPG#vM90Av}6T5nq`3g8!JJat^d35e}D&Kgqnc+EsajS
zFW;DrlxfB&YQ!T9*ZSjNvJ&?p0uLLhaCjc7LS!jzyDn8M_+svVu5L??cLN+bo(czv
z>N=+HDR${#ihsTi1S{tvj?laC^^f`n9rg-dD}YwHL_UZED=R?
z%6)mSY@}2^3HQ3$km0bU!GZzcVNssD0FESS9Jk+A?%0gdJk`bIO-nzd9I!teHL3;j
zTZoJIAJgY_RaPVm`7kvNBYzOkW+yP_cTg$K=%MA#vng)rzTz?veYqeh70`nsqp)2SaEp>`W|4#7cV
zg4fF!YYufiwKDtt$A$XO52*VvQmi)#=wq{|vdvE$jb8=h3fmM^@ZFh@-3=4a5(jSz
z&WYDV7yP1&hb%6`tCS((Y(cON?-WPk4mixfAqKkFc~}A>_VAG%`K)nw5gFV@pZsy}ujI7(1%!m%vVXFFc&cXxP
z>gV9P3h^TYE0Ye#jZTe5xNNp$H2`gE>AVTG)u0Ahh$ZM0q04FTUu(XDFw08;9k;sm
zHWc}Ah$9Chr!kmF6_p=6WZNjd02qTM7tFv*_Q0Ka_@zB^{3UBQP#7toc9*vIp?)mi
z7-qW*9nFUA|2}$ecn5KojsiM%bqRdJ{UO>MPq{VPz*8xU_=7?gl@g?X`Enzk_wS@R
z;32G`?e*ahE}w8jT6)4-zWJbWVk3$R>h!h=3I)11Wmz786XM?v7IiH+z1Lu2Hl~Yt
za${|kpuC~K$ZK#c+#j9Ope~`?m|JdOTb~vW<)K`X@Sfi^u3A?ZNa<{DP;?-WSp_^A
zWd$I9KI};F6d$-`$*
zwk1Xq6+t^>p>_RwJ2BCTmaz0&7;@wVLe2I-O!lE30&UlxP;iC
zKHdPkqvrWZ#{9=Ks*A$WQLC+;=P+4~3h0DYtYV|dh)Ry_gbuiD{2UrxJo|Mt
zBCvJh=?l#*B5WVmFr6CkHlE*=yj2gLWfL%nY_C~XC65E7F+ch{zxN*>MY?S716s+|
z0O)PF9)n!lDZe*=O?t26Zlh|E7*3)@0KPvFz6Z*t{}GHiJqg?x)O519z9?i*-WI=~
zB#15v)bTq`4~GIv06fhD-qjjp5d_GOR=NKf8C;}*ECf%Hh{jDobeS9UqD(hw@Ql=%
z+6lmS&%yU$J5IfK9;P!j8rOKg5Tej=FTTcan-a5uf?S*5Z^tV0(>fwLh%jo4xhk%sMGIMJ_|0_5b8eguASxpVg6o2{zjC%E5NlS
zQ++IJRv3Amx1T{s?oRyB&r!Rs<^gkSIMyF<-R^k}?)JgPj)(ja%7r@efd-18(_
z$8lFu4PqKh00a09$WOiUGu6G-c5mbVSj9E?05IJs0Kf^a>%Q?(??cyo8B5oR%7nQn
z3RDW6^!4qN50#@S4!WZi<=!Fqo@__4Dl+?Tl#g`T!E`9ooyS#!F>??q-h!n)dj}XF
z2dQY=H#)XDFfO2ybOJ3sYoUfK7NExH$@yPLoNVF;(7kCAFcc?2I0AM3Wu>0CXq=UA
ze7Ya-LVz44S5SlR#P(IsB68f77{groA$a!zxRh@&@_=mo?70z6F_+(9B}vkJ?P4FP
z*~!ln?Kw|Np&T<$7i1e1p8{|-N)WWQngBSWOC_j=Re{b5iW8CG-2B&o=ffK^R;Us>
zIhd^=S(x;m+f11~d>nmN54{_s(*z4@=V@o^p8F6y+mclVH#hnce~FjuT-r!fNV3y0Ub2Tdl#}C4XG0Xz_L<>LUj+
zcxO_-eJH*pq-%LUk*^XSvQen#8&F6txq62I;wn+ICm98^lYc0_cX9XA$0Y0ejVp@+
zo0%xvI-Q2EQ4XjF-?)_jvRJ6tw-ZFZUTu83-C^hX`iuzqr*mgSSX%EVfqVj2Zmg*U-_x8{Fvyl2jk5LD;o_c1@RYmu
zUl~u93LGbz0G*OOwRzYf=vhn`_=1NVPhPh51t0m>;tw*v;gvXy@l3P^rE0eAl9=IR
zB#n%5Zt~m*;0&SaJI2mAZ|QV8*%6wvfr`UBY$U-#A1X=qI(ma6yWFM!>j_QI)vfcs
zNB8Y|_vjo1;NG3NeAcV!SrEQ+)4lU}YQ3$tg3`|GJOo~-@*TwX0|+C_JogGG=8pB1
zvkpzqWYSA4tXhj;?-m?sCbJ?4kh{vBC*eA0PYom+^R}WKUK+6LFn(u?{qL%SACal9
zSZz!`bLohpI9YIMytfo(gf98SLH=&1r7Vw!c!asK1m7+kp04?`*kf~iD?QMSQ6S;`?*h8iCE9#7lLsp*gfv{9+f=wevH|*&kYIYp>t3
z;(XxiDbWV%I?=N~ZCI0=v+?iY;vZL+KGs@qEy%EHcAu=E!-7VpsvVj;7TuX7*HFng
z{ezpNHT1M|xeaPJ=
zi$Xc}>NRrt>A+RvZ~14(HJ>U~XL2aLyiplgoJKhCTe8MXY4_xJT>3#VBbBu@gs*t&
zW7p%Mvx~HUCq->9nz1^^^EZxhj0L|rj{Kdl3$(aHyZxY)w)XjGV3}FDr?)Tjc)(4u
zc%Su-Ps>!99Ns=kP6t*a8ye(FxVCBVPnK&FYuVVII755K;#`xj(Gq_)CiJt`>x5VD
z`LXPaOqt%+r%R(=fBpvAA%rajS@|oxEACl;4ff1b?mHU&r&on@{c5>=YvZAq
z>x-)n3o$Ne3oKFq+Bw1D&xGK+O{P|@rye#3rrO%*a)_`s9NHwStk3bJ{QR85uTFgi
zsgmTtnX3QMc9THUOJkm*z&FPz
z%QI{L8=krPe+4M=u0PH_lpGP+p=GYxI(Lq*B_lg%a{rcQ!gfGV`l
zTKWaT+aeh}auY4ruQOA0l=KekYc>fB&pRlS&HXwN)G#`?
zblWoZ*MD+;by|RCT`w>&R;RV@WX;!lS}35AA0HcaBX4?EZkRguo>b#IRROl-H8Rh%
zo?pEhrUE=?P1|zy>N{6lo~H#@T3P_ptLK4vv5D)7_PqE~7#*?wYvs;cQ?}UR^89=U
znVyYRaqG72an-7PTY3CNfTNX}^_1C@fdy$Z>l;b2XN8ep_n5_c-H^NXc+)FfnRp(P
z*qV!XBQ5pAqi^JW-J$#5BJ1^ql)3X|tAhO6K+XGz1M^ZVD=(VuR=IueRbpIj><(ER
z-Jc9!M(%4*VqTf6-Br1c4t8?Azz^m
zW}AnZtJZRF&t7RXcaHAY!U98Js#(NjqaFRF`Qaw4d2&VY0&lj`S9`+CAS?Z_c9yW0
l99YdmE3}E{Hkxt#sZWW@Qr&Q0=m!H3c)I$ztaD0e0su%iQ9J+u


From ea85018ccfe21a26fe0420b4021ca96e70341837 Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Sat, 8 Sep 2018 16:13:00 -0700
Subject: [PATCH 2066/3084] clif-util: fix `clif-util pass` subcommand
 arguments

Because of the way that the `pass` subcommand orders its arguments, the
positional "single-file" input cannot be optional with a default value, because
it is followed by required positional arguments. If it were optional, that would
result in argument ambiguity where `clap` cannot tell if the optional positional
argument is supplied, or if the given argument is the next required positional
argument.

Before this commit:

```
$ cargo run --bin clif-util -- pass ./filetests/dce/basic.clif dce
   Compiling cranelift-tools v0.21.0 (file:///Users/fitzgen/src/cranelift)
    Finished dev [unoptimized + debuginfo] target(s) in 4.38s
     Running `target/debug/clif-util pass ./filetests/dce/basic.clif dce`
thread 'main' panicked at 'Found positional argument which is not required with a lower index than a required positional argument: "single-file" index 1', /Users/fitzgen/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.32.0/src/app/parser.rs:612:21
note: Run with `RUST_BACKTRACE=1` for a backtrace.
```

After this commit:

```
$ cargo run --bin clif-util -- pass ./filetests/dce/basic.clif dce
   Compiling cranelift-filetests v0.21.0 (file:///Users/fitzgen/src/cranelift/lib/filetests)
   Compiling cranelift-tools v0.21.0 (file:///Users/fitzgen/src/cranelift)
    Finished dev [unoptimized + debuginfo] target(s) in 5.96s
     Running `target/debug/clif-util pass ./filetests/dce/basic.clif dce`
1 tests
```
---
 cranelift/src/clif-util.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 925f42f3fd..f608e78e1e 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -55,9 +55,9 @@ fn add_input_file_arg<'a>() -> clap::Arg<'a, 'a> {
 
 fn add_single_input_file_arg<'a>() -> clap::Arg<'a, 'a> {
     Arg::with_name("single-file")
-        .default_value("-")
+        .required(true)
         .value_name("single-file")
-        .help("Specify a file to be used. Defaults to reading from stdin.")
+        .help("Specify a file to be used. Use '-' for stdin.")
 }
 
 fn add_pass_arg<'a>() -> clap::Arg<'a, 'a> {

From 52e6b070580a8ab8cde8a77fc9a4081dfd671844 Mon Sep 17 00:00:00 2001
From: Maddy 
Date: Tue, 11 Sep 2018 02:51:43 +0000
Subject: [PATCH 2067/3084] Update the reload pass to replace copies with
 fill/spill instructions.

---
 cranelift/filetests/regalloc/reload.clif |  27 +++++
 lib/codegen/src/regalloc/reload.rs       | 144 ++++++++++++++---------
 2 files changed, 118 insertions(+), 53 deletions(-)

diff --git a/cranelift/filetests/regalloc/reload.clif b/cranelift/filetests/regalloc/reload.clif
index 517b0990d1..0d39047dfd 100644
--- a/cranelift/filetests/regalloc/reload.clif
+++ b/cranelift/filetests/regalloc/reload.clif
@@ -17,3 +17,30 @@ ebb0:
     ; check: $(reload=$V) = fill v0
     ; check: return $reload
 }
+
+; Check that copies where the arg has been spilled are replaced with fills.
+;
+; RV32E has 6 registers for function arguments so the 7th, v6, will be placed
+; on the stack.
+function %spilled_copy_arg(i32, i32, i32, i32, i32, i32, i32) -> i32 {
+
+ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32):
+    ; not: copy
+    ; check: v10 = fill v6
+    v10 = copy v6
+    return v10
+}
+
+; Check that copies where the result has been spilled are replaced with spills.
+;
+; v1 is live across a call so it will be spilled.
+function %spilled_copy_result(i32) -> i32 {
+    fn0 = %foo(i32)
+
+ebb0(v0: i32):
+    ; not: copy
+    ; check: v1 = spill v0
+    v1 = copy v0
+    call fn0(v1)
+    return v1
+}
diff --git a/lib/codegen/src/regalloc/reload.rs b/lib/codegen/src/regalloc/reload.rs
index f07f35cea0..22a6ac5963 100644
--- a/lib/codegen/src/regalloc/reload.rs
+++ b/lib/codegen/src/regalloc/reload.rs
@@ -13,7 +13,7 @@ use cursor::{Cursor, EncCursor};
 use dominator_tree::DominatorTree;
 use entity::{SparseMap, SparseMapValue};
 use ir::{AbiParam, ArgumentLoc, InstBuilder};
-use ir::{Ebb, Function, Inst, Value};
+use ir::{Ebb, Function, Inst, InstructionData, Opcode, Value};
 use isa::RegClass;
 use isa::{ConstraintKind, EncInfo, Encoding, RecipeConstraints, TargetIsa};
 use regalloc::affinity::Affinity;
@@ -209,6 +209,84 @@ impl<'a> Context<'a> {
         debug_assert!(self.candidates.is_empty());
         self.find_candidates(inst, constraints);
 
+        if let InstructionData::Unary {
+            opcode: Opcode::Copy,
+            ..
+        } = self.cur.func.dfg[inst]
+        {
+            self.reload_copy_candidates(inst);
+        } else {
+            self.reload_inst_candidates(ebb, inst);
+        }
+
+        // TODO: Reuse reloads for future instructions.
+        self.reloads.clear();
+
+        let (_throughs, _kills, defs) =
+            tracker.process_inst(inst, &self.cur.func.dfg, self.liveness);
+
+        // Advance to the next instruction so we can insert any spills after the instruction.
+        self.cur.next_inst();
+
+        // Rewrite register defs that need to be spilled.
+        //
+        // Change:
+        //
+        // v2 = inst ...
+        //
+        // Into:
+        //
+        // v7 = inst ...
+        // v2 = spill v7
+        //
+        // That way, we don't need to rewrite all future uses of v2.
+        for (lv, op) in defs.iter().zip(constraints.outs) {
+            if lv.affinity.is_stack() && op.kind != ConstraintKind::Stack {
+                if let InstructionData::Unary {
+                    opcode: Opcode::Copy,
+                    arg,
+                } = self.cur.func.dfg[inst]
+                {
+                    self.cur.func.dfg.replace(inst).spill(arg);
+                    let ok = self.cur.func.update_encoding(inst, self.cur.isa).is_ok();
+                    debug_assert!(ok);
+                } else {
+                    let value_type = self.cur.func.dfg.value_type(lv.value);
+                    let reg = self.cur.func.dfg.replace_result(lv.value, value_type);
+                    self.liveness.create_dead(reg, inst, Affinity::new(op));
+                    self.insert_spill(ebb, lv.value, reg);
+                }
+            }
+        }
+
+        // Same thing for spilled call return values.
+        let retvals = &defs[constraints.outs.len()..];
+        if !retvals.is_empty() {
+            let sig = self
+                .cur
+                .func
+                .dfg
+                .call_signature(inst)
+                .expect("Extra results on non-call instruction");
+            for (i, lv) in retvals.iter().enumerate() {
+                let abi = self.cur.func.dfg.signatures[sig].returns[i];
+                debug_assert!(
+                    abi.location.is_reg(),
+                    "expected reg; got {:?}",
+                    abi.location
+                );
+                if lv.affinity.is_stack() {
+                    let reg = self.cur.func.dfg.replace_result(lv.value, abi.value_type);
+                    self.liveness
+                        .create_dead(reg, inst, Affinity::abi(&abi, self.cur.isa));
+                    self.insert_spill(ebb, lv.value, reg);
+                }
+            }
+        }
+    }
+
+    // Reload the current candidates for the given `inst`.
+    fn reload_inst_candidates(&mut self, ebb: Ebb, inst: Inst) {
         // Insert fill instructions before `inst` and replace `cand.value` with the filled value.
         for cand in self.candidates.iter_mut() {
             if let Some(reload) = self.reloads.get(cand.value) {
@@ -244,60 +322,20 @@ impl<'a> Context<'a> {
                 args[cand.argidx] = cand.value;
             }
         }
+    }
 
-        // TODO: Reuse reloads for future instructions.
-        self.reloads.clear();
+    // Reload the current candidates for the given copy `inst`.
+    //
+    // As an optimization, replace a copy instruction where the argument has been spilled with
+    // a fill instruction.
+    fn reload_copy_candidates(&mut self, inst: Inst) {
+        // Copy instructions can only have one argument.
+        debug_assert!(self.candidates.is_empty() || self.candidates.len() == 1);
 
-        let (_throughs, _kills, defs) =
-            tracker.process_inst(inst, &self.cur.func.dfg, self.liveness);
-
-        // Advance to the next instruction so we can insert any spills after the instruction.
-        self.cur.next_inst();
-
-        // Rewrite register defs that need to be spilled.
-        //
-        // Change:
-        //
-        // v2 = inst ...
-        //
-        // Into:
-        //
-        // v7 = inst ...
-        // v2 = spill v7
-        //
-        // That way, we don't need to rewrite all future uses of v2.
-        for (lv, op) in defs.iter().zip(constraints.outs) {
-            if lv.affinity.is_stack() && op.kind != ConstraintKind::Stack {
-                let value_type = self.cur.func.dfg.value_type(lv.value);
-                let reg = self.cur.func.dfg.replace_result(lv.value, value_type);
-                self.liveness.create_dead(reg, inst, Affinity::new(op));
-                self.insert_spill(ebb, lv.value, reg);
-            }
-        }
-
-        // Same thing for spilled call return values.
-        let retvals = &defs[constraints.outs.len()..];
-        if !retvals.is_empty() {
-            let sig = self
-                .cur
-                .func
-                .dfg
-                .call_signature(inst)
-                .expect("Extra results on non-call instruction");
-            for (i, lv) in retvals.iter().enumerate() {
-                let abi = self.cur.func.dfg.signatures[sig].returns[i];
-                debug_assert!(
-                    abi.location.is_reg(),
-                    "expected reg; got {:?}",
-                    abi.location
-                );
-                if lv.affinity.is_stack() {
-                    let reg = self.cur.func.dfg.replace_result(lv.value, abi.value_type);
-                    self.liveness
-                        .create_dead(reg, inst, Affinity::abi(&abi, self.cur.isa));
-                    self.insert_spill(ebb, lv.value, reg);
-                }
-            }
+        if let Some(cand) = self.candidates.pop() {
+            self.cur.func.dfg.replace(inst).fill(cand.value);
+            let ok = self.cur.func.update_encoding(inst, self.cur.isa).is_ok();
+            debug_assert!(ok);
         }
     }
 

From d0f703a6ff369661f4a94b99376d43c2f846726f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 12 Sep 2018 14:38:15 -0700
Subject: [PATCH 2068/3084] Fix the type of table bounds in DummyEnvironment.

Also, fix the wasm testsuite harness to run the verifier with a
TargetIsa so that it catches problems like this.
---
 lib/wasm/src/environ/dummy.rs    | 2 +-
 lib/wasm/tests/wasm_testsuite.rs | 9 +++++++--
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 229e2a7274..d1e6a8052c 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -206,7 +206,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         let bound_gv = func.create_global_value(ir::GlobalValueData::Load {
             base: vmctx,
             offset: Offset32::new(0),
-            global_type: self.pointer_type(),
+            global_type: I32,
         });
 
         func.create_table(ir::TableData {
diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs
index 06d7da4e7b..a64ec63e6b 100644
--- a/lib/wasm/tests/wasm_testsuite.rs
+++ b/lib/wasm/tests/wasm_testsuite.rs
@@ -4,6 +4,7 @@ extern crate cranelift_wasm;
 extern crate target_lexicon;
 extern crate wabt;
 
+use cranelift_codegen::isa;
 use cranelift_codegen::print_errors::pretty_verifier_error;
 use cranelift_codegen::settings::{self, Configurable, Flags};
 use cranelift_codegen::verifier;
@@ -75,9 +76,13 @@ fn handle_module(path: &Path, flags: &Flags) {
     };
     let mut dummy_environ = DummyEnvironment::with_triple_flags(triple!("riscv64"), flags.clone());
     translate_module(&data, &mut dummy_environ).unwrap();
+
+    let isa = isa::lookup(dummy_environ.info.triple)
+        .unwrap()
+        .finish(dummy_environ.info.flags);
     for func in dummy_environ.info.function_bodies.values() {
-        verifier::verify_function(func, flags)
-            .map_err(|errors| panic!(pretty_verifier_error(func, None, None, errors)))
+        verifier::verify_function(func, &*isa)
+            .map_err(|errors| panic!(pretty_verifier_error(func, Some(&*isa), None, errors)))
             .unwrap();
     }
 }

From 9e65b694bd23ad5c0426afba14b804df1204d96c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 12 Sep 2018 16:06:03 -0700
Subject: [PATCH 2069/3084] Minor code simplification; avoid an unneeded
 `match`.

---
 lib/simplejit/src/backend.rs | 89 +++++++++++++++++-------------------
 1 file changed, 43 insertions(+), 46 deletions(-)

diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index ea9dd36115..09a12dd7ad 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -350,53 +350,50 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
     ) -> Self::FinalizedData {
         use std::ptr::write_unaligned;
 
-        for record in &data.relocs {
-            match *record {
-                RelocRecord {
-                    reloc,
-                    offset,
-                    ref name,
-                    addend,
-                } => {
-                    let ptr = data.storage;
-                    debug_assert!((offset as usize) < data.size);
-                    let at = unsafe { ptr.offset(offset as isize) };
-                    let base = if namespace.is_function(name) {
-                        let (def, name_str, _signature) = namespace.get_function_definition(&name);
-                        match def {
-                            Some(compiled) => compiled.code,
-                            None => self.lookup_symbol(name_str),
-                        }
-                    } else {
-                        let (def, name_str, _writable) = namespace.get_data_definition(&name);
-                        match def {
-                            Some(compiled) => compiled.storage,
-                            None => self.lookup_symbol(name_str),
-                        }
-                    };
-                    // TODO: Handle overflow.
-                    let what = unsafe { base.offset(addend as isize) };
-                    match reloc {
-                        Reloc::Abs4 => {
-                            // TODO: Handle overflow.
-                            #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))]
-                            unsafe {
-                                write_unaligned(at as *mut u32, what as u32)
-                            };
-                        }
-                        Reloc::Abs8 => {
-                            #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))]
-                            unsafe {
-                                write_unaligned(at as *mut u64, what as u64)
-                            };
-                        }
-                        Reloc::X86PCRel4
-                        | Reloc::X86CallPCRel4
-                        | Reloc::X86GOTPCRel4
-                        | Reloc::X86CallPLTRel4 => panic!("unexpected text relocation in data"),
-                        _ => unimplemented!(),
-                    }
+        for &RelocRecord {
+            reloc,
+            offset,
+            ref name,
+            addend,
+        } in &data.relocs
+        {
+            let ptr = data.storage;
+            debug_assert!((offset as usize) < data.size);
+            let at = unsafe { ptr.offset(offset as isize) };
+            let base = if namespace.is_function(name) {
+                let (def, name_str, _signature) = namespace.get_function_definition(&name);
+                match def {
+                    Some(compiled) => compiled.code,
+                    None => self.lookup_symbol(name_str),
                 }
+            } else {
+                let (def, name_str, _writable) = namespace.get_data_definition(&name);
+                match def {
+                    Some(compiled) => compiled.storage,
+                    None => self.lookup_symbol(name_str),
+                }
+            };
+            // TODO: Handle overflow.
+            let what = unsafe { base.offset(addend as isize) };
+            match reloc {
+                Reloc::Abs4 => {
+                    // TODO: Handle overflow.
+                    #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))]
+                    unsafe {
+                        write_unaligned(at as *mut u32, what as u32)
+                    };
+                }
+                Reloc::Abs8 => {
+                    #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))]
+                    unsafe {
+                        write_unaligned(at as *mut u64, what as u64)
+                    };
+                }
+                Reloc::X86PCRel4
+                | Reloc::X86CallPCRel4
+                | Reloc::X86GOTPCRel4
+                | Reloc::X86CallPLTRel4 => panic!("unexpected text relocation in data"),
+                _ => unimplemented!(),
             }
         }
         (data.storage, data.size)

From f2177deec1d41f4a0f22d2acc9de10dbe661c258 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 12 Sep 2018 16:07:43 -0700
Subject: [PATCH 2070/3084] Improve assertion messages in Module finalization.

When an object being finalized references an object declared as needing
a definition, the definition needs to be available. Add asserts to catch
this specific case.
---
 lib/module/src/module.rs | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 909fa36c6d..0f04440f2a 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -263,7 +263,13 @@ where
         name: &ir::ExternalName,
     ) -> (Option<&B::CompiledFunction>, &str, &ir::Signature) {
         let info = self.contents.get_function_info(name);
+        debug_assert!(
+            !info.decl.linkage.is_definable() || info.compiled.is_some(),
+            "Finalization requires a definition for function {}.",
+            name,
+        );
         debug_assert_eq!(info.decl.linkage.is_definable(), info.compiled.is_some());
+
         (
             info.compiled.as_ref(),
             &info.decl.name,
@@ -278,7 +284,13 @@ where
         name: &ir::ExternalName,
     ) -> (Option<&B::CompiledData>, &str, bool) {
         let info = self.contents.get_data_info(name);
+        debug_assert!(
+            !info.decl.linkage.is_definable() || info.compiled.is_some(),
+            "Finalization requires a definition for data object {}.",
+            name,
+        );
         debug_assert_eq!(info.decl.linkage.is_definable(), info.compiled.is_some());
+
         (info.compiled.as_ref(), &info.decl.name, info.decl.writable)
     }
 

From f6519c87a4438cce05ff70fd0f4ffb13c44da192 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 12 Sep 2018 16:58:36 -0700
Subject: [PATCH 2071/3084] Improve the assertion failure message for sealing a
 block twice.

---
 lib/frontend/src/ssa.rs | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs
index b15eea172c..f41e6a32c4 100644
--- a/lib/frontend/src/ssa.rs
+++ b/lib/frontend/src/ssa.rs
@@ -442,14 +442,18 @@ impl SSABuilder {
     fn seal_one_ebb_header_block(&mut self, ebb: Ebb, func: &mut Function) {
         let block = self.header_block(ebb);
 
-        let (undef_vars, ebb): (Vec<(Variable, Value)>, Ebb) = match self.blocks[block] {
+        let undef_vars = match self.blocks[block] {
             BlockData::EbbBody { .. } => panic!("this should not happen"),
             BlockData::EbbHeader(ref mut data) => {
-                debug_assert!(!data.sealed);
+                debug_assert!(
+                    !data.sealed,
+                    "Attempting to seal {} which is already sealed.",
+                    ebb
+                );
+                debug_assert_eq!(ebb, data.ebb);
                 // Extract the undef_variables data from the block so that we
                 // can iterate over it without borrowing the whole builder.
-                let undef_variables = mem::replace(&mut data.undef_variables, Vec::new());
-                (undef_variables, data.ebb)
+                mem::replace(&mut data.undef_variables, Vec::new())
             }
         };
 

From f23ea042428f881f06acdab51cb563271b3f23b3 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 12 Sep 2018 16:59:03 -0700
Subject: [PATCH 2072/3084] Fix a panic when splitting a critical edge on a
 jump_table with nulls.

This fixes #505.
---
 lib/frontend/src/ssa.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs
index f41e6a32c4..3474b5104e 100644
--- a/lib/frontend/src/ssa.rs
+++ b/lib/frontend/src/ssa.rs
@@ -657,7 +657,7 @@ impl SSABuilder {
                 self.blocks[middle_block].add_predecessor(jump_inst_block, jump_inst);
                 self.mark_ebb_header_block_sealed(middle_block);
                 for old_dest in func.jump_tables[jt].as_mut_slice() {
-                    if old_dest.unwrap() == dest_ebb {
+                    if *old_dest == PackedOption::from(dest_ebb) {
                         *old_dest = PackedOption::from(middle_ebb);
                     }
                 }
@@ -1013,6 +1013,7 @@ mod tests {
         ssa.def_var(x_var, x1, block0);
         let mut data = JumpTableData::new();
         data.push_entry(ebb1);
+        data.set_entry(2, ebb1);
         let jt = func.create_jump_table(data);
         ssa.use_var(&mut func, x_var, I32, block0).0;
         let br_table = {

From 53a0c6c67f4bb70f9e61e55e93afac448130ca7f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 13 Sep 2018 12:59:25 -0700
Subject: [PATCH 2073/3084] Update to the rustfmt in rust 1.29, which is now
 stable.

---
 cranelift/src/clif-util.rs            | 30 ++++++++++++---------
 lib/bforest/src/lib.rs                | 20 +++++++++++---
 lib/codegen/src/isa/x86/abi.rs        |  3 +--
 lib/codegen/src/legalizer/boundary.rs |  6 ++---
 lib/codegen/src/lib.rs                | 15 ++++++++---
 lib/codegen/src/regalloc/coloring.rs  |  7 +++--
 lib/codegen/src/regalloc/liveness.rs  |  3 +--
 lib/codegen/src/regalloc/spilling.rs  |  3 +--
 lib/codegen/src/verifier/flags.rs     |  3 ++-
 lib/codegen/src/verifier/mod.rs       | 17 ++++++------
 lib/entity/src/lib.rs                 | 20 +++++++++++---
 lib/faerie/src/backend.rs             |  9 +++----
 lib/faerie/src/lib.rs                 | 27 +++++++++++++++----
 lib/filetests/src/concurrent.rs       |  6 ++---
 lib/filetests/src/lib.rs              | 13 +++++++--
 lib/filetests/src/runner.rs           |  3 +--
 lib/frontend/src/frontend.rs          |  9 +++----
 lib/frontend/src/lib.rs               | 10 +++++--
 lib/module/src/lib.rs                 | 20 +++++++++++---
 lib/module/src/module.rs              |  5 +++-
 lib/native/src/lib.rs                 | 27 +++++++++++++++----
 lib/reader/src/lexer.rs               |  3 +--
 lib/reader/src/lib.rs                 | 27 +++++++++++++++----
 lib/reader/src/parser.rs              | 39 ++++++++++++++-------------
 lib/serde/src/clif-json.rs            | 32 +++++++++++++++-------
 lib/simplejit/src/lib.rs              | 27 +++++++++++++++----
 lib/umbrella/src/lib.rs               | 27 +++++++++++++++----
 lib/wasm/src/environ/spec.rs          |  6 ++++-
 lib/wasm/src/lib.rs                   | 20 +++++++++++---
 lib/wasm/tests/wasm_testsuite.rs      |  3 +--
 30 files changed, 305 insertions(+), 135 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index f608e78e1e..d1ad95ed3d 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -1,10 +1,19 @@
 #![deny(trivial_numeric_casts)]
-#![warn(unused_import_braces, unstable_features, unused_extern_crates)]
+#![warn(
+    unused_import_braces,
+    unstable_features,
+    unused_extern_crates
+)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
-        unicode_not_nfc, use_self
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        unicode_not_nfc,
+        use_self
     )
 )]
 
@@ -153,31 +162,26 @@ fn main() {
                 .arg(add_time_flag())
                 .arg(add_input_file_arg())
                 .arg(add_debug_flag()),
-        )
-        .subcommand(
+        ).subcommand(
             SubCommand::with_name("cat")
                 .about("Outputs .clif file")
                 .arg(add_input_file_arg())
                 .arg(add_debug_flag()),
-        )
-        .subcommand(
+        ).subcommand(
             SubCommand::with_name("print-cfg")
                 .about("Prints out cfg in dot format")
                 .arg(add_input_file_arg())
                 .arg(add_debug_flag()),
-        )
-        .subcommand(
+        ).subcommand(
             add_wasm_or_compile("compile")
                 .arg(
                     Arg::with_name("just-decode")
                         .short("t")
                         .help("Just decode WebAssembly to Cranelift IR"),
-                )
-                .arg(Arg::with_name("check-translation").short("c").help(
+                ).arg(Arg::with_name("check-translation").short("c").help(
                     "Just checks the correctness of Cranelift IR translated from WebAssembly",
                 )),
-        )
-        .subcommand(add_wasm_or_compile("wasm"))
+        ).subcommand(add_wasm_or_compile("wasm"))
         .subcommand(
             SubCommand::with_name("pass")
                 .about("Run specified pass(s) on an input file.")
diff --git a/lib/bforest/src/lib.rs b/lib/bforest/src/lib.rs
index 2a1c5f8436..7971d481d8 100644
--- a/lib/bforest/src/lib.rs
+++ b/lib/bforest/src/lib.rs
@@ -16,13 +16,25 @@
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", warn(unstable_features))]
-#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
+#![cfg_attr(
+    feature = "clippy",
+    plugin(clippy(conf_file = "../../clippy.toml"))
+)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    allow(new_without_default, new_without_default_derive)
+)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
-        print_stdout, unicode_not_nfc, use_self
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        print_stdout,
+        unicode_not_nfc,
+        use_self
     )
 )]
 // Turns on no_std and alloc features if std is not available.
diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs
index 6028fa507d..af7dda69d1 100644
--- a/lib/codegen/src/isa/x86/abi.rs
+++ b/lib/codegen/src/isa/x86/abi.rs
@@ -99,8 +99,7 @@ impl ArgAssigner for Args {
                         RU::r14
                     } else {
                         RU::rsi
-                    } as RegUnit)
-                        .into()
+                    } as RegUnit).into()
                 }
                 // This is SpiderMonkey's `WasmTableCallSigReg`.
                 ArgumentPurpose::SignatureId => return ArgumentLoc::Reg(RU::r10 as RegUnit).into(),
diff --git a/lib/codegen/src/legalizer/boundary.rs b/lib/codegen/src/legalizer/boundary.rs
index e052e63be6..b8bdc83ca8 100644
--- a/lib/codegen/src/legalizer/boundary.rs
+++ b/lib/codegen/src/legalizer/boundary.rs
@@ -569,8 +569,7 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph
             rt.purpose == ArgumentPurpose::Link
                 || rt.purpose == ArgumentPurpose::StructReturn
                 || rt.purpose == ArgumentPurpose::VMContext
-        })
-        .count();
+        }).count();
     let abi_args = func.signature.returns.len() - special_args;
 
     let pos = &mut FuncCursor::new(func).at_inst(inst);
@@ -689,8 +688,7 @@ fn spill_call_arguments(pos: &mut FuncCursor) -> bool {
                     }
                     _ => None,
                 }
-            })
-            .collect::>()
+            }).collect::>()
     };
 
     if arglist.is_empty() {
diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index 10e8a7a461..ec924dc6bf 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -3,7 +3,10 @@
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", deny(unstable_features))]
-#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
+#![cfg_attr(
+    feature = "clippy",
+    plugin(clippy(conf_file = "../../clippy.toml"))
+)]
 #![cfg_attr(feature="cargo-clippy", allow(
 // This requires Rust 1.27 or later.
                 duration_subsec,
@@ -32,8 +35,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
-        print_stdout, unicode_not_nfc, use_self
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        print_stdout,
+        unicode_not_nfc,
+        use_self
     )
 )]
 // Turns on no_std and alloc features if std is not available.
diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs
index 56691eecf3..d9df465f5c 100644
--- a/lib/codegen/src/regalloc/coloring.rs
+++ b/lib/codegen/src/regalloc/coloring.rs
@@ -902,10 +902,9 @@ impl<'a> Context<'a> {
             }
             Table(jt) => {
                 let lr = &self.liveness[value];
-                !lr.is_local()
-                    && self.cur.func.jump_tables[jt]
-                        .entries()
-                        .any(|(_, ebb)| lr.is_livein(ebb, ctx))
+                !lr.is_local() && self.cur.func.jump_tables[jt]
+                    .entries()
+                    .any(|(_, ebb)| lr.is_livein(ebb, ctx))
             }
         }
     }
diff --git a/lib/codegen/src/regalloc/liveness.rs b/lib/codegen/src/regalloc/liveness.rs
index 01f72d9adb..8877179e43 100644
--- a/lib/codegen/src/regalloc/liveness.rs
+++ b/lib/codegen/src/regalloc/liveness.rs
@@ -220,8 +220,7 @@ fn get_or_create<'a>(
                         func.dfg
                             .call_signature(inst)
                             .map(|sig| Affinity::abi(&func.dfg.signatures[sig].returns[rnum], isa))
-                    })
-                    .unwrap_or_default();
+                    }).unwrap_or_default();
             }
             ValueDef::Param(ebb, num) => {
                 def = ebb.into();
diff --git a/lib/codegen/src/regalloc/spilling.rs b/lib/codegen/src/regalloc/spilling.rs
index d1244c0774..92b632c30a 100644
--- a/lib/codegen/src/regalloc/spilling.rs
+++ b/lib/codegen/src/regalloc/spilling.rs
@@ -463,8 +463,7 @@ impl<'a> Context<'a> {
                     }
                 }
                 None
-            })
-            .min_by(|&a, &b| {
+            }).min_by(|&a, &b| {
                 // Find the minimum candidate according to the RPO of their defs.
                 self.domtree.rpo_cmp(
                     self.cur.func.dfg.value_def(a),
diff --git a/lib/codegen/src/verifier/flags.rs b/lib/codegen/src/verifier/flags.rs
index 0d16d17ef5..d3fda7555f 100644
--- a/lib/codegen/src/verifier/flags.rs
+++ b/lib/codegen/src/verifier/flags.rs
@@ -113,7 +113,8 @@ impl<'a> FlagsVerifier<'a> {
                     .encinfo
                     .as_ref()
                     .and_then(|ei| ei.operand_constraints(self.func.encodings[inst]))
-                    .map_or(false, |c| c.clobbers_flags) && live_val.is_some()
+                    .map_or(false, |c| c.clobbers_flags)
+                    && live_val.is_some()
                 {
                     return fatal!(errors, inst, "encoding clobbers live CPU flags in {}", live);
                 }
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 51aa484525..40f8302592 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -941,11 +941,11 @@ impl<'a> Verifier<'a> {
                     );
                 }
                 // The defining EBB dominates the instruction using this value.
-                if is_reachable
-                    && !self
-                        .expected_domtree
-                        .dominates(ebb, loc_inst, &self.func.layout)
-                {
+                if is_reachable && !self.expected_domtree.dominates(
+                    ebb,
+                    loc_inst,
+                    &self.func.layout,
+                ) {
                     return fatal!(
                         errors,
                         loc_inst,
@@ -1039,7 +1039,8 @@ impl<'a> Verifier<'a> {
         for (&prev_ebb, &next_ebb) in domtree.cfg_postorder().iter().adjacent_pairs() {
             if self
                 .expected_domtree
-                .rpo_cmp(prev_ebb, next_ebb, &self.func.layout) != Ordering::Greater
+                .rpo_cmp(prev_ebb, next_ebb, &self.func.layout)
+                != Ordering::Greater
             {
                 return fatal!(
                     errors,
@@ -1586,8 +1587,8 @@ impl<'a> Verifier<'a> {
 
         let encoding = self.func.encodings[inst];
         if encoding.is_legal() {
-            let mut encodings =
-                isa.legal_encodings(
+            let mut encodings = isa
+                .legal_encodings(
                     &self.func,
                     &self.func.dfg[inst],
                     self.func.dfg.ctrl_typevar(inst),
diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs
index dd666c6ac7..a1d07db442 100644
--- a/lib/entity/src/lib.rs
+++ b/lib/entity/src/lib.rs
@@ -32,13 +32,25 @@
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", deny(unstable_features))]
-#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
+#![cfg_attr(
+    feature = "clippy",
+    plugin(clippy(conf_file = "../../clippy.toml"))
+)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    allow(new_without_default, new_without_default_derive)
+)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
-        print_stdout, unicode_not_nfc, use_self
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        print_stdout,
+        unicode_not_nfc,
+        use_self
     )
 )]
 // Turns on no_std and alloc features if std is not available.
diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index 6aa948703e..908eb1411d 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -228,8 +228,7 @@ impl Backend for FaerieBackend {
                     from: name,
                     to,
                     at: u64::from(offset),
-                })
-                .map_err(|e| ModuleError::Backend(e.to_string()))?;
+                }).map_err(|e| ModuleError::Backend(e.to_string()))?;
         }
         for &(offset, id, addend) in data_relocs {
             debug_assert_eq!(
@@ -242,8 +241,7 @@ impl Backend for FaerieBackend {
                     from: name,
                     to,
                     at: u64::from(offset),
-                })
-                .map_err(|e| ModuleError::Backend(e.to_string()))?;
+                }).map_err(|e| ModuleError::Backend(e.to_string()))?;
         }
 
         self.artifact
@@ -409,8 +407,7 @@ impl<'a> RelocSink for FaerieRelocSink<'a> {
                     reloc: raw_reloc,
                     addend: addend_i32,
                 },
-            )
-            .expect("faerie relocation error");
+            ).expect("faerie relocation error");
     }
 
     fn reloc_jt(&mut self, _offset: CodeOffset, _reloc: Reloc, _jt: ir::JumpTable) {
diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs
index 078488a953..0b9187a468 100644
--- a/lib/faerie/src/lib.rs
+++ b/lib/faerie/src/lib.rs
@@ -2,15 +2,32 @@
 //!
 //! Users of this module should not have to depend on faerie directly.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![deny(
+    missing_docs,
+    trivial_numeric_casts,
+    unused_extern_crates,
+    unstable_features
+)]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
+#![cfg_attr(
+    feature = "clippy",
+    plugin(clippy(conf_file = "../../clippy.toml"))
+)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    allow(new_without_default, new_without_default_derive)
+)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
-        print_stdout, unicode_not_nfc, use_self
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        print_stdout,
+        unicode_not_nfc,
+        use_self
     )
 )]
 
diff --git a/lib/filetests/src/concurrent.rs b/lib/filetests/src/concurrent.rs
index df9034618b..44963cc466 100644
--- a/lib/filetests/src/concurrent.rs
+++ b/lib/filetests/src/concurrent.rs
@@ -106,8 +106,7 @@ fn heartbeat_thread(replies: Sender) -> thread::JoinHandle<()> {
             while replies.send(Reply::Tick).is_ok() {
                 thread::sleep(Duration::from_secs(1));
             }
-        })
-        .unwrap()
+        }).unwrap()
 }
 
 /// Spawn a worker thread running tests.
@@ -154,6 +153,5 @@ fn worker_thread(
             // Timing is accumulated independently per thread.
             // Timings from this worker thread will be aggregated by `ConcurrentRunner::join()`.
             timing::take_current()
-        })
-        .unwrap()
+        }).unwrap()
 }
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index f8e796182e..24086d88ab 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -3,7 +3,12 @@
 //! This crate contains the main test driver as well as implementations of the
 //! available filetest commands.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![deny(
+    missing_docs,
+    trivial_numeric_casts,
+    unused_extern_crates,
+    unstable_features
+)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "cargo-clippy",
     allow(
@@ -13,7 +18,11 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else, unicode_not_nfc,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        unicode_not_nfc,
         use_self
     )
 )]
diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs
index a05f54c390..ff6f203590 100644
--- a/lib/filetests/src/runner.rs
+++ b/lib/filetests/src/runner.rs
@@ -315,8 +315,7 @@ impl TestRunner {
                     ..
                 } => Some(dur),
                 _ => None,
-            })
-            .collect::>();
+            }).collect::>();
 
         // Get me some real data, kid.
         let len = times.len();
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 0a6732a89b..405de81f51 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -517,11 +517,10 @@ impl<'a> FunctionBuilder<'a> {
             None => false,
             Some(entry) => self.position.ebb.unwrap() == entry,
         };
-        !is_entry && self.func_ctx.ssa.is_sealed(self.position.ebb.unwrap())
-            && !self
-                .func_ctx
-                .ssa
-                .has_any_predecessors(self.position.ebb.unwrap())
+        !is_entry && self.func_ctx.ssa.is_sealed(self.position.ebb.unwrap()) && !self
+            .func_ctx
+            .ssa
+            .has_any_predecessors(self.position.ebb.unwrap())
     }
 
     /// Returns `true` if and only if no instructions have been added since the last call to
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index aec18899cb..9b90a5df07 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -163,8 +163,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
-        print_stdout, unicode_not_nfc, use_self
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        print_stdout,
+        unicode_not_nfc,
+        use_self
     )
 )]
 #![cfg_attr(not(feature = "std"), no_std)]
diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs
index 479288df57..3c569d0ec8 100644
--- a/lib/module/src/lib.rs
+++ b/lib/module/src/lib.rs
@@ -3,13 +3,25 @@
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", deny(unstable_features))]
-#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
+#![cfg_attr(
+    feature = "clippy",
+    plugin(clippy(conf_file = "../../clippy.toml"))
+)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    allow(new_without_default, new_without_default_derive)
+)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
-        print_stdout, unicode_not_nfc, use_self
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        print_stdout,
+        unicode_not_nfc,
+        use_self
     )
 )]
 // Turns on no_std and alloc features if std is not available.
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 0f04440f2a..bdeb90c627 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -129,7 +129,10 @@ pub enum ModuleError {
     #[fail(display = "Duplicate definition of identifier: {}", _0)]
     DuplicateDefinition(String),
     /// Indicates an identifier was defined, but was declared as an import
-    #[fail(display = "Invalid to define identifier declared as an import: {}", _0)]
+    #[fail(
+        display = "Invalid to define identifier declared as an import: {}",
+        _0
+    )]
     InvalidImportDefinition(String),
     /// Wraps a `cranelift-codegen` error
     #[fail(display = "Compilation error: {}", _0)]
diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs
index 3d4e4bac2c..7d88984db5 100644
--- a/lib/native/src/lib.rs
+++ b/lib/native/src/lib.rs
@@ -1,15 +1,32 @@
 //! Performs autodetection of the host for the purposes of running
 //! Cranelift to generate code to run on the same machine.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![deny(
+    missing_docs,
+    trivial_numeric_casts,
+    unused_extern_crates,
+    unstable_features
+)]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
+#![cfg_attr(
+    feature = "clippy",
+    plugin(clippy(conf_file = "../../clippy.toml"))
+)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    allow(new_without_default, new_without_default_derive)
+)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
-        print_stdout, unicode_not_nfc, use_self
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        print_stdout,
+        unicode_not_nfc,
+        use_self
     )
 )]
 #![cfg_attr(not(feature = "std"), no_std)]
diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs
index 5a2a0124a2..c1d9a81d93 100644
--- a/lib/reader/src/lexer.rs
+++ b/lib/reader/src/lexer.rs
@@ -322,8 +322,7 @@ impl<'a> Lexer<'a> {
                 .and_then(|(prefix, number)| {
                     Self::numbered_entity(prefix, number)
                         .or_else(|| Self::value_type(text, prefix, number))
-                })
-                .unwrap_or_else(|| match text {
+                }).unwrap_or_else(|| match text {
                     "iflags" => Token::Type(types::IFLAGS),
                     "fflags" => Token::Type(types::FFLAGS),
                     _ => Token::Identifier(text),
diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs
index 2873a644ac..bfa9bc8207 100644
--- a/lib/reader/src/lib.rs
+++ b/lib/reader/src/lib.rs
@@ -3,15 +3,32 @@
 //! The `cranelift_reader` library supports reading .clif files. This functionality is needed for
 //! testing Cranelift, but is not essential for a JIT compiler.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![deny(
+    missing_docs,
+    trivial_numeric_casts,
+    unused_extern_crates,
+    unstable_features
+)]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
+#![cfg_attr(
+    feature = "clippy",
+    plugin(clippy(conf_file = "../../clippy.toml"))
+)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    allow(new_without_default, new_without_default_derive)
+)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
-        print_stdout, unicode_not_nfc, use_self
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        print_stdout,
+        unicode_not_nfc,
+        use_self
     )
 )]
 
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 4a116699b8..1594d183b6 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -969,9 +969,10 @@ impl<'a> Parser<'a> {
                         self.consume();
                         match self.token() {
                             Some(Token::Integer(index_str)) => {
-                                let index: u32 = u32::from_str_radix(index_str, 10).map_err(
-                                    |_| self.error("the integer given overflows the u32 type"),
-                                )?;
+                                let index: u32 =
+                                    u32::from_str_radix(index_str, 10).map_err(|_| {
+                                        self.error("the integer given overflows the u32 type")
+                                    })?;
                                 self.consume();
                                 Ok(ExternalName::user(namespace, index))
                             }
@@ -2557,7 +2558,7 @@ mod tests {
                                              v1 = iadd_imm v3, 17
                                            }",
         ).parse_function(None)
-            .unwrap();
+        .unwrap();
         assert_eq!(func.name.to_string(), "%qux");
         let v4 = details.map.lookup_str("v4").unwrap();
         assert_eq!(v4.to_string(), "v4");
@@ -2633,7 +2634,7 @@ mod tests {
                                        ss1 = spill_slot 1
                                      }",
         ).parse_function(None)
-            .unwrap();
+        .unwrap();
         assert_eq!(func.name.to_string(), "%foo");
         let mut iter = func.stack_slots.keys();
         let _ss0 = iter.next().unwrap();
@@ -2656,8 +2657,8 @@ mod tests {
                                     ss1  = spill_slot 1
                                 }",
             ).parse_function(None)
-                .unwrap_err()
-                .to_string(),
+            .unwrap_err()
+            .to_string(),
             "3: duplicate entity: ss1"
         );
     }
@@ -2670,7 +2671,7 @@ mod tests {
                                      ebb4(v3: i32):
                                      }",
         ).parse_function(None)
-            .unwrap();
+        .unwrap();
         assert_eq!(func.name.to_string(), "%ebbs");
 
         let mut ebbs = func.layout.ebbs();
@@ -2692,7 +2693,7 @@ mod tests {
                 ebb0:
                     return 2",
         ).parse_function(None)
-            .unwrap_err();
+        .unwrap_err();
 
         assert_eq!(location.line_number, 3);
         assert_eq!(message, "duplicate entity: ebb0");
@@ -2705,7 +2706,7 @@ mod tests {
                 jt0 = jump_table 0, 0
                 jt0 = jump_table 0, 0",
         ).parse_function(None)
-            .unwrap_err();
+        .unwrap_err();
 
         assert_eq!(location.line_number, 3);
         assert_eq!(message, "duplicate entity: jt0");
@@ -2718,7 +2719,7 @@ mod tests {
                 ss0 = explicit_slot 8
                 ss0 = explicit_slot 8",
         ).parse_function(None)
-            .unwrap_err();
+        .unwrap_err();
 
         assert_eq!(location.line_number, 3);
         assert_eq!(message, "duplicate entity: ss0");
@@ -2731,7 +2732,7 @@ mod tests {
                 gv0 = vmctx
                 gv0 = vmctx",
         ).parse_function(None)
-            .unwrap_err();
+        .unwrap_err();
 
         assert_eq!(location.line_number, 3);
         assert_eq!(message, "duplicate entity: gv0");
@@ -2744,7 +2745,7 @@ mod tests {
                 heap0 = static gv0, min 0x1000, bound 0x10_0000, guard 0x1000
                 heap0 = static gv0, min 0x1000, bound 0x10_0000, guard 0x1000",
         ).parse_function(None)
-            .unwrap_err();
+        .unwrap_err();
 
         assert_eq!(location.line_number, 3);
         assert_eq!(message, "duplicate entity: heap0");
@@ -2757,7 +2758,7 @@ mod tests {
                 sig0 = ()
                 sig0 = ()",
         ).parse_function(None)
-            .unwrap_err();
+        .unwrap_err();
 
         assert_eq!(location.line_number, 3);
         assert_eq!(message, "duplicate entity: sig0");
@@ -2771,7 +2772,7 @@ mod tests {
                 fn0 = %foo sig0
                 fn0 = %foo sig0",
         ).parse_function(None)
-            .unwrap_err();
+        .unwrap_err();
 
         assert_eq!(location.line_number, 4);
         assert_eq!(message, "duplicate entity: fn0");
@@ -2791,7 +2792,7 @@ mod tests {
                          } ; Trailing.
                          ; More trailing.",
         ).parse_function(None)
-            .unwrap();
+        .unwrap();
         assert_eq!(func.name.to_string(), "%comment");
         assert_eq!(comments.len(), 8); // no 'before' comment.
         assert_eq!(
@@ -2868,7 +2869,7 @@ mod tests {
                           isa riscv
                           function %foo() system_v {}",
         ).unwrap()
-            .isa_spec
+        .isa_spec
         {
             IsaSpec::None(_) => panic!("Expected some ISA"),
             IsaSpec::Some(v) => {
@@ -2887,8 +2888,8 @@ mod tests {
                                              trap int_divz
                                            }",
         ).parse_function(None)
-            .unwrap()
-            .0;
+        .unwrap()
+        .0;
         assert_eq!(func.name.to_string(), "u1:2");
 
         // Invalid characters in the name:
diff --git a/lib/serde/src/clif-json.rs b/lib/serde/src/clif-json.rs
index 228b13703d..bab63ace97 100644
--- a/lib/serde/src/clif-json.rs
+++ b/lib/serde/src/clif-json.rs
@@ -1,14 +1,30 @@
 //! Utility for `cranelift_serde`.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![deny(
+    missing_docs,
+    trivial_numeric_casts,
+    unused_extern_crates,
+    unstable_features
+)]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
+#![cfg_attr(
+    feature = "clippy",
+    plugin(clippy(conf_file = "../../clippy.toml"))
+)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    allow(new_without_default, new_without_default_derive)
+)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
-        unicode_not_nfc, use_self
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        unicode_not_nfc,
+        use_self
     )
 )]
 
@@ -67,8 +83,7 @@ fn main() {
                         .value_name("FILE")
                         .help("Input file for serialization"),
                 ),
-        )
-        .subcommand(
+        ).subcommand(
             SubCommand::with_name("deserialize")
                 .about("Deserializes Cranelift IR into JSON.")
                 .arg(
@@ -77,8 +92,7 @@ fn main() {
                         .value_name("FILE")
                         .help("Input file for deserialization"),
                 ),
-        )
-        .get_matches();
+        ).get_matches();
 
     let res_serde = match matches.subcommand() {
         ("serialize", Some(m)) => {
diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs
index 99297f33b0..50f84f67eb 100644
--- a/lib/simplejit/src/lib.rs
+++ b/lib/simplejit/src/lib.rs
@@ -1,14 +1,31 @@
 //! Top-level lib.rs for `cranelift_simplejit`.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![deny(
+    missing_docs,
+    trivial_numeric_casts,
+    unused_extern_crates,
+    unstable_features
+)]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
+#![cfg_attr(
+    feature = "clippy",
+    plugin(clippy(conf_file = "../../clippy.toml"))
+)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    allow(new_without_default, new_without_default_derive)
+)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
-        print_stdout, unicode_not_nfc, use_self
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        print_stdout,
+        unicode_not_nfc,
+        use_self
     )
 )]
 
diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs
index d56bdd6f71..79d262ecfd 100644
--- a/lib/umbrella/src/lib.rs
+++ b/lib/umbrella/src/lib.rs
@@ -1,14 +1,31 @@
 //! Cranelift umbrella crate, providing a convenient one-line dependency.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates, unstable_features)]
+#![deny(
+    missing_docs,
+    trivial_numeric_casts,
+    unused_extern_crates,
+    unstable_features
+)]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
+#![cfg_attr(
+    feature = "clippy",
+    plugin(clippy(conf_file = "../../clippy.toml"))
+)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    allow(new_without_default, new_without_default_derive)
+)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
-        print_stdout, unicode_not_nfc, use_self
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        print_stdout,
+        unicode_not_nfc,
+        use_self
     )
 )]
 
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index bc8e3fa9fa..31640635b5 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -35,7 +35,11 @@ pub enum WasmError {
     ///
     /// This error code is used by a WebAssembly translator when it encounters invalid WebAssembly
     /// code. This should never happen for validated WebAssembly code.
-    #[fail(display = "Invalid input WebAssembly code at offset {}: {}", _1, _0)]
+    #[fail(
+        display = "Invalid input WebAssembly code at offset {}: {}",
+        _1,
+        _0
+    )]
     InvalidWebAssembly {
         /// A string describing the validation error.
         message: &'static str,
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index a559ae941d..95667d870d 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -12,13 +12,25 @@
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", deny(unstable_features))]
-#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(feature = "cargo-clippy", allow(new_without_default, new_without_default_derive))]
+#![cfg_attr(
+    feature = "clippy",
+    plugin(clippy(conf_file = "../../clippy.toml"))
+)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    allow(new_without_default, new_without_default_derive)
+)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic, mut_mut, nonminimal_bool, option_map_unwrap_or, option_map_unwrap_or_else,
-        print_stdout, unicode_not_nfc, use_self
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        print_stdout,
+        unicode_not_nfc,
+        use_self
     )
 )]
 #![cfg_attr(not(feature = "std"), no_std)]
diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs
index a64ec63e6b..f7cc0e678e 100644
--- a/lib/wasm/tests/wasm_testsuite.rs
+++ b/lib/wasm/tests/wasm_testsuite.rs
@@ -30,8 +30,7 @@ fn testsuite() {
                 }
             }
             false
-        })
-        .collect();
+        }).collect();
     paths.sort_by_key(|dir| dir.path());
     let flags = Flags::new(settings::builder());
     for path in paths {

From 2fe96c30a6ed226898729c9f9dd7006a11bd0472 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 13 Sep 2018 15:30:39 -0700
Subject: [PATCH 2074/3084] Check in the Crane and Ferris drawing so that
 people can remix it :-).

Fixes #514.
---
 cranelift/media/crane-ferris.svg | 265 +++++++++++++++++++++++++++++++
 1 file changed, 265 insertions(+)
 create mode 100644 cranelift/media/crane-ferris.svg

diff --git a/cranelift/media/crane-ferris.svg b/cranelift/media/crane-ferris.svg
new file mode 100644
index 0000000000..808fd75c3e
--- /dev/null
+++ b/cranelift/media/crane-ferris.svg
@@ -0,0 +1,265 @@
+
+
+
+
+  
+  
+  
+    
+      
+        image/svg+xml
+        
+        
+      
+    
+  
+  
+    
+    
+    
+  
+  
+    
+      
+        
+      
+      
+        
+      
+      
+        
+      
+      
+        
+      
+      
+        
+      
+      
+        
+      
+      
+        
+      
+      
+        
+      
+      
+        
+      
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+  
+

From 3228d73f332e85eaa4750625824cd774382f9808 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 19 Sep 2018 13:49:59 -0700
Subject: [PATCH 2075/3084] Add more content to cranelift-entity's README.md.
 (#515)

* Add more content to cranelift-entity's README.md.

Summarize what cranelift-entity provides, and how it differs from
similar systems such as slotmap, which was recently highlighted in the
RustConf 2018 Closing Keynote.
---
 lib/entity/README.md | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/lib/entity/README.md b/lib/entity/README.md
index 4679e74b40..2b9ff7e042 100644
--- a/lib/entity/README.md
+++ b/lib/entity/README.md
@@ -1,2 +1,40 @@
 This crate contains array-based data structures used by the core Cranelift code
 generator which use densely numbered entity references as mapping keys.
+
+One major difference between this crate and crates like [slotmap], [slab],
+and [generational-arena] is that this crate currently provides no way to delete
+entities. This limits its use to situations where deleting isn't important,
+however this also makes it more efficient, because it doesn't need extra
+bookkeeping state to reuse the storage for deleted objects, or to ensure that
+new objects always have unique keys (eg. slotmap's and generational-arena's
+versioning).
+
+Another major difference is that this crate protects against using a key from
+one map to access an element in another. Where `SlotMap`, `Slab`, and `Arena`
+have a value type parameter, `PrimaryMap` has a key type parameter and a value
+type parameter. The crate also provides the `entity_impl` macro which makes it
+easy to declare new unique types for use as keys. Any attempt to use a key in
+a map it's not intended for is diagnosed with a type error.
+
+Another is that this crate has two core map types, `PrimaryMap` and
+`EntityMap`, which serve complementary purposes. A `PrimaryMap` creates its
+own keys when elements are inserted, while an `EntityMap` reuses the keys
+values of a `PrimaryMap`, conceptually storing additional data in the same
+index space. `EntityMap`'s values must implement `Default` and all elements
+in an `EntityMap` initially have the value of `default()`.
+
+A common way to implement `Default` is to wrap a type in `Option`, however
+this crate also provides the `PackedOption` utility which can use less memory
+in some cases.
+
+Additional utilities provided by this crate include:
+ - `EntityList`, for allocating many small arrays (such as instruction operand
+    lists in a compiler code generator).
+ - `SparseMap`: an alternative to `EntityMap` which can use less memory
+   in some situations.
+ - `EntitySet`: a specialized form of `EntityMap` using a bitvector to
+   record which entities are members of the set.
+
+[slotmap]: https://crates.io/crates/slotmap
+[slab]: https://crates.io/crates/slab
+[generational-arena]: https://crates.io/crates/generational-arena

From d514cec06590d4969d5bc1f11d89428dfa62eb53 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 19 Sep 2018 18:18:40 -0700
Subject: [PATCH 2076/3084] Avoid unneeded '&'s in let patterns.

---
 lib/bforest/src/pool.rs   | 2 +-
 lib/codegen/src/ir/dfg.rs | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/bforest/src/pool.rs b/lib/bforest/src/pool.rs
index ceab96d38c..9330ee5502 100644
--- a/lib/bforest/src/pool.rs
+++ b/lib/bforest/src/pool.rs
@@ -90,7 +90,7 @@ impl NodePool {
 
         // The root node can't be an inner node with just a single sub-tree. It should have been
         // pruned.
-        if let &NodeData::Inner { size, .. } = &self[node] {
+        if let NodeData::Inner { size, .. } = self[node] {
             assert!(size > 0, "Root must have more than one sub-tree");
         }
 
diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs
index 08bec5d624..9a6a22e9fa 100644
--- a/lib/codegen/src/ir/dfg.rs
+++ b/lib/codegen/src/ir/dfg.rs
@@ -156,10 +156,10 @@ pub struct Values<'a> {
 
 /// Check for non-values
 fn valid_valuedata(data: &ValueData) -> bool {
-    if let &ValueData::Alias {
+    if let ValueData::Alias {
         ty: types::INVALID,
         original,
-    } = data
+    } = *data
     {
         if original == Value::reserved_value() {
             return false;

From 6e9c33a1efc27dec44ee274b51534abc4f25ab38 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 19 Sep 2018 18:44:23 -0700
Subject: [PATCH 2077/3084] Mark JIT memory as readable in addition to
 executable.

While we don't currently need this, we will for jump tables and constant
pools.
---
 lib/simplejit/src/backend.rs | 2 +-
 lib/simplejit/src/memory.rs  | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index 09a12dd7ad..3fc46cd993 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -406,7 +406,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
     fn publish(&mut self) {
         // Now that we're done patching, prepare the memory for execution!
         self.readonly_memory.set_readonly();
-        self.code_memory.set_executable();
+        self.code_memory.set_readable_and_executable();
     }
 
     /// SimpleJIT emits code and data into memory as it processes them, so it
diff --git a/lib/simplejit/src/memory.rs b/lib/simplejit/src/memory.rs
index b856c19009..f5dcf42780 100644
--- a/lib/simplejit/src/memory.rs
+++ b/lib/simplejit/src/memory.rs
@@ -113,15 +113,15 @@ impl Memory {
         Ok(self.current.ptr)
     }
 
-    /// Set all memory allocated in this `Memory` up to now as executable.
-    pub fn set_executable(&mut self) {
+    /// Set all memory allocated in this `Memory` up to now as readable and executable.
+    pub fn set_readable_and_executable(&mut self) {
         self.finish_current();
 
         for &PtrLen { ptr, len } in &self.allocations[self.executable..] {
             if len != 0 {
                 unsafe {
-                    region::protect(ptr, len, region::Protection::Execute)
-                        .expect("unable to make memory executable");
+                    region::protect(ptr, len, region::Protection::ReadExecute)
+                        .expect("unable to make memory readable+executable");
                 }
             }
         }

From c840fb2f088231b3eb2d80414cad0a6b4e936582 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 19 Sep 2018 20:07:30 -0700
Subject: [PATCH 2078/3084] Move tests/moduletests.rs into lib/simplejit/tests.

These tests depend on cranelift-simplejit, which is higher-level than the
other crates they depend on, so lib/simplejit is a good place for them.
---
 lib/simplejit/Cargo.toml                                       | 2 ++
 cranelift/tests/moduletests.rs => lib/simplejit/tests/basic.rs | 0
 2 files changed, 2 insertions(+)
 rename cranelift/tests/moduletests.rs => lib/simplejit/tests/basic.rs (100%)

diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 341e27e750..d1df478224 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -23,6 +23,8 @@ winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
 cranelift = { path = "../umbrella", version = "0.21.1" }
+cranelift-frontend = { path = "../frontend", version = "0.21.1" }
+cranelift-entity = { path = "../entity", version = "0.21.1" }
 
 [features]
 default = ["std"]
diff --git a/cranelift/tests/moduletests.rs b/lib/simplejit/tests/basic.rs
similarity index 100%
rename from cranelift/tests/moduletests.rs
rename to lib/simplejit/tests/basic.rs

From e7565a889300295edc9c10562077b7963bd2d4b9 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 21 Sep 2018 19:49:24 -0700
Subject: [PATCH 2079/3084] Mention the Windows x64 calling convention support
 in README.md.

---
 cranelift/README.md | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index b325058da1..5c71313d99 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -24,10 +24,10 @@ implementation.
 
 The x86-64 backend is currently the most complete and stable; other
 architectures are in various stages of development. Cranelift currently
-supports the System V AMD64 ABI calling convention used on many
-platforms, but does not yet support the Windows x64 calling convention.
-The performance of code produced by Cranelift is not yet impressive,
-though we have plans to fix that.
+supports both the System V AMD64 ABI calling convention used on many
+platforms and the Windows x64 calling convention. The performance
+of code produced by Cranelift is not yet impressive, though we have plans
+to fix that.
 
 The core codegen crates have minimal dependencies, support
 [no\_std](#building-with-no-std) mode, and do not require any host

From fd081f2af8cef0c8a3e64002c5ca1f01c550354b Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 21 Sep 2018 19:50:05 -0700
Subject: [PATCH 2080/3084] Update README.md to mention the actual version of
 Rust supported.

---
 cranelift/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index 5c71313d99..9a409912df 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -41,7 +41,7 @@ needed before it would be ready for a production use case.
 
 Cranelift's APIs are not yet stable.
 
-Cranelift currently supports Rust 1.25.1 and later. We intend to always
+Cranelift currently supports Rust 1.25.0 and later. We intend to always
 support the latest *stable* Rust. And, we currently support the version
 of Rust in the latest Ubuntu LTS, although whether we will always do so
 is not yet determined. Cranelift requires Python 2.7 or Python 3 to

From 8d6a8e9069060b724748c8427efc6c693915a1f3 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 21 Sep 2018 20:13:33 -0700
Subject: [PATCH 2081/3084] Remove `Module`'s `finalize_function` and
 `finalize_data`. (#519)

* Remove `Module`'s `finalize_function` and `finalize_data`.

Remove the ability to finalize individiual functions and data objects,
and instead just provide a way to finalize everything that's been
defined but not yet finalized. This allows SimpleJIT to share an
allocation between multiple functions without having to worry about
individual functions being finalized and needing to be published
without the other functions in the same allocation.

Users of the return values of `Module`'s `finalize_function` and
`finalize_data` should now use `get_finalized_function` and
`get_finalized_data` to obtain these values.
---
 lib/module/src/module.rs                    | 131 ++++++--------------
 lib/simplejit/examples/simplejit-minimal.rs |   2 +-
 lib/simplejit/tests/basic.rs                |  12 +-
 3 files changed, 46 insertions(+), 99 deletions(-)

diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index bdeb90c627..bcfa7491b2 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -154,8 +154,6 @@ where
     decl: FunctionDeclaration,
     /// The compiled artifact, once it's available.
     compiled: Option,
-    /// A flag indicating whether the function has been finalized.
-    finalized: bool,
 }
 
 impl ModuleFunction
@@ -187,8 +185,6 @@ where
     decl: DataDeclaration,
     /// The "compiled" artifact, once it's available.
     compiled: Option,
-    /// A flag indicating whether the data object has been finalized.
-    finalized: bool,
 }
 
 impl ModuleData
@@ -314,6 +310,8 @@ where
 {
     names: HashMap,
     contents: ModuleContents,
+    functions_to_finalize: Vec,
+    data_objects_to_finalize: Vec,
     backend: B,
 }
 
@@ -329,6 +327,8 @@ where
                 functions: PrimaryMap::new(),
                 data_objects: PrimaryMap::new(),
             },
+            functions_to_finalize: Vec::new(),
+            data_objects_to_finalize: Vec::new(),
             backend: B::new(backend_builder),
         }
     }
@@ -407,7 +407,6 @@ where
                         signature: signature.clone(),
                     },
                     compiled: None,
-                    finalized: false,
                 });
                 entry.insert(FuncOrDataId::Func(id));
                 self.backend.declare_function(name, linkage);
@@ -447,7 +446,6 @@ where
                         writable,
                     },
                     compiled: None,
-                    finalized: false,
                 });
                 entry.insert(FuncOrDataId::Data(id));
                 self.backend.declare_data(name, linkage, writable);
@@ -523,6 +521,7 @@ where
             )?)
         };
         self.contents.functions[func].compiled = compiled;
+        self.functions_to_finalize.push(func);
         Ok(())
     }
 
@@ -546,6 +545,7 @@ where
             )?)
         };
         self.contents.data_objects[data].compiled = compiled;
+        self.data_objects_to_finalize.push(data);
         Ok(())
     }
 
@@ -592,20 +592,16 @@ where
         );
     }
 
-    /// Perform all outstanding relocations on the given function. This requires all `Local`
-    /// and `Export` entities referenced to be defined.
+    /// Finalize all functions and data objects that are defined but not yet finalized.
+    /// All symbols referenced in their bodies that are declared as needing a definition
+    /// must be defined by this point.
     ///
-    /// # Panics
-    ///
-    /// When the function has already been finalized this panics.
-    pub fn finalize_function(&mut self, func: FuncId) -> B::FinalizedFunction {
-        let output = {
+    /// Use `get_finalized_function` and `get_finalized_data` to obtain the final
+    /// artifacts.
+    pub fn finalize_definitions(&mut self) {
+        for func in self.functions_to_finalize.drain(..) {
             let info = &self.contents.functions[func];
-            debug_assert!(
-                info.decl.linkage.is_definable(),
-                "imported function cannot be finalized"
-            );
-            assert!(!info.finalized, "function can't be finalized twice");
+            debug_assert!(info.decl.linkage.is_definable());
             self.backend.finalize_function(
                 info.compiled
                     .as_ref()
@@ -613,38 +609,11 @@ where
                 &ModuleNamespace:: {
                     contents: &self.contents,
                 },
-            )
-        };
-        self.contents.functions[func].finalized = true;
-        self.backend.publish();
-        output
-    }
-
-    /// Return the finalized artifact from the backend, if it provides one.
-    pub fn get_finalized_function(&mut self, func: FuncId) -> B::FinalizedFunction {
-        let info = &self.contents.functions[func];
-        debug_assert!(info.finalized, "data object not yet finalized");
-        self.backend.get_finalized_function(
-            info.compiled
-                .as_ref()
-                .expect("function must be compiled before it can be finalized"),
-        )
-    }
-
-    /// Perform all outstanding relocations on the given data object. This requires all
-    /// `Local` and `Export` entities referenced to be defined.
-    ///
-    /// # Panics
-    ///
-    /// When the data object has already been finalized this panics.
-    pub fn finalize_data(&mut self, data: DataId) -> B::FinalizedData {
-        let output = {
-            let info = &self.contents.data_objects[data];
-            debug_assert!(
-                info.decl.linkage.is_definable(),
-                "imported data cannot be finalized"
             );
-            assert!(!info.finalized, "data object can't be finalized twice");
+        }
+        for data in self.data_objects_to_finalize.drain(..) {
+            let info = &self.contents.data_objects[data];
+            debug_assert!(info.decl.linkage.is_definable());
             self.backend.finalize_data(
                 info.compiled
                     .as_ref()
@@ -652,17 +621,32 @@ where
                 &ModuleNamespace:: {
                     contents: &self.contents,
                 },
-            )
-        };
-        self.contents.data_objects[data].finalized = true;
+            );
+        }
         self.backend.publish();
-        output
+    }
+
+    /// Return the finalized artifact from the backend, if it provides one.
+    pub fn get_finalized_function(&mut self, func: FuncId) -> B::FinalizedFunction {
+        let info = &self.contents.functions[func];
+        debug_assert!(
+            !self.functions_to_finalize.iter().any(|x| *x == func),
+            "function not yet finalized"
+        );
+        self.backend.get_finalized_function(
+            info.compiled
+                .as_ref()
+                .expect("function must be compiled before it can be finalized"),
+        )
     }
 
     /// Return the finalized artifact from the backend, if it provides one.
     pub fn get_finalized_data(&mut self, data: DataId) -> B::FinalizedData {
         let info = &self.contents.data_objects[data];
-        debug_assert!(info.finalized, "data object not yet finalized");
+        debug_assert!(
+            !self.data_objects_to_finalize.iter().any(|x| *x == data),
+            "data object not yet finalized"
+        );
         self.backend.get_finalized_data(
             info.compiled
                 .as_ref()
@@ -670,45 +654,6 @@ where
         )
     }
 
-    /// Finalize all functions and data objects. Note that this doesn't return the
-    /// final artifacts returned from `finalize_function` or `finalize_data`. Use
-    /// `get_finalized_function` and `get_finalized_data` to obtain the final
-    /// artifacts.
-    pub fn finalize_all(&mut self) {
-        // TODO: Could we use something like `into_iter()` here?
-        for info in self.contents.functions.values() {
-            if info.decl.linkage.is_definable() && !info.finalized {
-                self.backend.finalize_function(
-                    info.compiled
-                        .as_ref()
-                        .expect("function must be compiled before it can be finalized"),
-                    &ModuleNamespace:: {
-                        contents: &self.contents,
-                    },
-                );
-            }
-        }
-        for info in self.contents.functions.values_mut() {
-            info.finalized = true;
-        }
-        for info in self.contents.data_objects.values() {
-            if info.decl.linkage.is_definable() && !info.finalized {
-                self.backend.finalize_data(
-                    info.compiled
-                        .as_ref()
-                        .expect("data object must be compiled before it can be finalized"),
-                    &ModuleNamespace:: {
-                        contents: &self.contents,
-                    },
-                );
-            }
-        }
-        for info in self.contents.data_objects.values_mut() {
-            info.finalized = true;
-        }
-        self.backend.publish();
-    }
-
     /// Consume the module and return the resulting `Product`. Some `Backend`
     /// implementations may provide additional functionality available after
     /// a `Module` is complete.
diff --git a/lib/simplejit/examples/simplejit-minimal.rs b/lib/simplejit/examples/simplejit-minimal.rs
index e393363768..0a736a8746 100644
--- a/lib/simplejit/examples/simplejit-minimal.rs
+++ b/lib/simplejit/examples/simplejit-minimal.rs
@@ -67,7 +67,7 @@ fn main() {
     module.clear_context(&mut ctx);
 
     // Perform linking.
-    module.finalize_all();
+    module.finalize_definitions();
 
     // Get a raw pointer to the generated code.
     let code_b = module.get_finalized_function(func_b);
diff --git a/lib/simplejit/tests/basic.rs b/lib/simplejit/tests/basic.rs
index a4aea4b389..7363fee4e9 100644
--- a/lib/simplejit/tests/basic.rs
+++ b/lib/simplejit/tests/basic.rs
@@ -57,13 +57,15 @@ fn define_simple_function(module: &mut Module) -> FuncId {
 }
 
 #[test]
-#[should_panic(expected = "function can't be finalized twice")]
-fn panic_on_double_finalize() {
+fn double_finalize() {
     let mut module: Module = Module::new(SimpleJITBuilder::new());
 
     let func_id = define_simple_function(&mut module);
-    module.finalize_function(func_id);
-    module.finalize_function(func_id);
+    module.finalize_definitions();
+
+    // Calling `finalize_definitions` a second time without any new definitions
+    // should have no effect.
+    module.finalize_definitions();
 }
 
 #[test]
@@ -72,6 +74,6 @@ fn panic_on_define_after_finalize() {
     let mut module: Module = Module::new(SimpleJITBuilder::new());
 
     let func_id = define_simple_function(&mut module);
-    module.finalize_function(func_id);
+    module.finalize_definitions();
     define_simple_function(&mut module);
 }

From ea0c196c11e44688aed713a202bc9fb8ec82c8d0 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 21 Sep 2018 20:16:55 -0700
Subject: [PATCH 2082/3084] More introductory documentation (#520)

* Add a comment to .rustfmt.toml explaning why it's here.

* Use `
` for specialized information in README.md.

* Describe a more elaborate issue-labelling system.
---
 cranelift/.rustfmt.toml   |  1 +
 cranelift/CONTRIBUTING.md | 22 +++++++++++++++++-----
 cranelift/README.md       | 23 ++++++++++++++---------
 3 files changed, 32 insertions(+), 14 deletions(-)

diff --git a/cranelift/.rustfmt.toml b/cranelift/.rustfmt.toml
index e69de29bb2..8148fc6346 100644
--- a/cranelift/.rustfmt.toml
+++ b/cranelift/.rustfmt.toml
@@ -0,0 +1 @@
+# This file tells tools we use rustfmt. We use the default settings.
diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
index b5223d006b..a5f5aad334 100644
--- a/cranelift/CONTRIBUTING.md
+++ b/cranelift/CONTRIBUTING.md
@@ -27,12 +27,24 @@ We're happy to mentor people, whether you're learning Rust, learning about
 compiler backends, learning about machine code, learning about how Cranelift
 does things, or all together at once.
 
-We tag issues in the issue tracker marked [good first issue] when we can, so
-that's sometimes a good place to get started. Also, we encourage people to just
-look around and find things they're interested in. This a good time to get
-involved, as there aren't a lot of things set in stone yet.
+We categorize issues in the issue tracker using a tag scheme inspired by
+[Rust's issue tags]. For example, the [E-easy] marks good beginner issues,
+and [E-rust] marks issues which likely require some familiarity with Rust,
+though not necessarily Cranelift-specific or even compiler-specific
+experience. [E-compiler-easy] marks issues good for beginners who have
+some familiarity with compilers, or are interested in gaining some :-).
 
-[good first issue]: https://github.com/CraneStation/cranelift/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22
+See also the [full list of labels].
+
+Also, we encourage people to just look around and find things they're
+interested in. This a good time to get involved, as there aren't a lot of
+things set in stone yet.
+
+[Rust's issue tags]: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#issue-triage
+[E-easy]: https://github.com/CraneStation/cranelift/labels/E-easy
+[E-rust]: https://github.com/CraneStation/cranelift/labels/E-rust
+[E-compiler-easy]: https://github.com/CraneStation/cranelift/labels/E-compiler-easy
+[full list of labels]: https://github.com/CraneStation/cranelift/labels
 
 ### Code of Conduct
 
diff --git a/cranelift/README.md b/cranelift/README.md
index 9a409912df..0dc6b8d61f 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -29,9 +29,8 @@ platforms and the Windows x64 calling convention. The performance
 of code produced by Cranelift is not yet impressive, though we have plans
 to fix that.
 
-The core codegen crates have minimal dependencies, support
-[no\_std](#building-with-no-std) mode, and do not require any host
-floating-point support.
+The core codegen crates have minimal dependencies, support no\_std mode
+(see below), and do not require any host floating-point support.
 
 Cranelift does not yet perform mitigations for Spectre or related
 security issues, though it may do so in the future. It does not
@@ -74,10 +73,10 @@ to tell cargo to visit all of the crates.
 `test-all.sh` at the top level is a script which runs all the cargo
 tests and also performs code format, lint, and documentation checks.
 
-Building with no\_std
----------------------
+

+Building with no_std
 
-The following crates support \`no\_std\`:
+The following crates support \`no\_std\`, although they do depend on liballoc:
  - cranelift-entity
  - cranelift-bforest
  - cranelift-codegen
@@ -123,13 +122,19 @@ called hashmap\_core is pulled in (via the core feature). This is mostly
 the same as std::collections::HashMap, except that it doesn't have DOS
 protection. Just something to think about.
 
-Building the documentation
---------------------------
+

 
-To build the Cranelift documentation, you need the [Sphinx documentation
+

+Building the documentation
+
+Cranelift's documentation is [published online](https://cranelift.readthedocs.io/).
+
+To build the documentation locally, you need the [Sphinx documentation
 generator](http://www.sphinx-doc.org/) as well as Python 3::
 
     $ pip install sphinx sphinx-autobuild sphinx_rtd_theme
     $ cd cranelift/docs
     $ make html
     $ open _build/html/index.html
+
+


From 8055abf90d372b5615e9d561ef10e19aa7ca8de7 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 21 Sep 2018 20:52:59 -0700
Subject: [PATCH 2083/3084] Add a mention of simplejit-demo to the README.

---
 cranelift/README.md | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/cranelift/README.md b/cranelift/README.md
index 0dc6b8d61f..b008ba7296 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -13,6 +13,11 @@ into executable machine code.
 For more information, see [the
 documentation](https://cranelift.readthedocs.io/en/latest/?badge=latest).
 
+For an example of how to use the JIT, see the [SimpleJIT Demo], which
+implements a toy language.
+
+[SimpleJIT Demo]: https://github.com/CraneStation/simplejit-demo
+
 Status
 ------
 

From 46beddecde1a735f4d56726a09bf5b93872fa886 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 21 Sep 2018 20:57:40 -0700
Subject: [PATCH 2084/3084] Update to capstone 0.5.0.

---
 cranelift/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 4ae89e3e35..4488566448 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -30,7 +30,7 @@ filecheck = "0.3.0"
 clap = "2.32.0"
 serde = "1.0.8"
 term = "0.5.1"
-capstone = { version = "0.4", optional = true }
+capstone = { version = "0.5.0", optional = true }
 wabt = { version = "0.5", optional = true }
 target-lexicon = "0.0.3"
 pretty_env_logger = "0.2.4"

From cb6b1a76dbe99e992ea34977f788ccc107835959 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 21 Sep 2018 21:14:06 -0700
Subject: [PATCH 2085/3084] Add a `use std::vec::Vec;` to fix the no_std build.

---
 lib/module/src/module.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index bcfa7491b2..626c34e4f5 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -11,6 +11,7 @@ use data_context::DataContext;
 use std::borrow::ToOwned;
 use std::collections::HashMap;
 use std::string::String;
+use std::vec::Vec;
 use Backend;
 
 /// A function identifier for use in the `Module` interface.

From 10edc1b6abbb56fb335adca3253e40c18ecbb3c7 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 21 Sep 2018 21:29:14 -0700
Subject: [PATCH 2086/3084] Format src/wasm.rs with newer rustfmt.

---
 cranelift/src/wasm.rs | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index f125096525..11b859d52b 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -2,7 +2,10 @@
 //! crate.
 //!
 //! Reads Wasm binary files, translates the functions' code to Cranelift IR.
-#![cfg_attr(feature = "cargo-clippy", allow(too_many_arguments, cyclomatic_complexity))]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    allow(too_many_arguments, cyclomatic_complexity)
+)]
 
 use cranelift_codegen::print_errors::{pretty_error, pretty_verifier_error};
 use cranelift_codegen::settings::FlagsOrIsa;

From ab997209599822ee9d620c92a9c31ede446ef4c7 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 21 Sep 2018 21:39:41 -0700
Subject: [PATCH 2087/3084] Bump version to 0.22.0

---
 cranelift/Cargo.toml        | 26 +++++++++++++-------------
 cranelift/publish-all.sh    |  2 +-
 lib/bforest/Cargo.toml      |  4 ++--
 lib/codegen/Cargo.toml      |  8 ++++----
 lib/codegen/meta/Cargo.toml |  2 +-
 lib/entity/Cargo.toml       |  2 +-
 lib/faerie/Cargo.toml       |  6 +++---
 lib/filetests/Cargo.toml    |  6 +++---
 lib/frontend/Cargo.toml     |  4 ++--
 lib/module/Cargo.toml       |  6 +++---
 lib/native/Cargo.toml       |  4 ++--
 lib/reader/Cargo.toml       |  4 ++--
 lib/serde/Cargo.toml        |  6 +++---
 lib/simplejit/Cargo.toml    | 14 +++++++-------
 lib/umbrella/Cargo.toml     |  6 +++---
 lib/wasm/Cargo.toml         |  8 ++++----
 16 files changed, 54 insertions(+), 54 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 4488566448..59e940b5d8 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.21.1"
+version = "0.22.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,18 +14,18 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.21.1" }
-cranelift-entity = { path = "lib/entity", version = "0.21.1" }
-cranelift-reader = { path = "lib/reader", version = "0.21.1" }
-cranelift-frontend = { path = "lib/frontend", version = "0.21.1" }
-cranelift-serde = { path = "lib/serde", version = "0.21.1", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.21.1", optional = true }
-cranelift-native = { path = "lib/native", version = "0.21.1" }
-cranelift-filetests = { path = "lib/filetests", version = "0.21.1" }
-cranelift-module = { path = "lib/module", version = "0.21.1" }
-cranelift-faerie = { path = "lib/faerie", version = "0.21.1" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.21.1" }
-cranelift = { path = "lib/umbrella", version = "0.21.1" }
+cranelift-codegen = { path = "lib/codegen", version = "0.22.0" }
+cranelift-entity = { path = "lib/entity", version = "0.22.0" }
+cranelift-reader = { path = "lib/reader", version = "0.22.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.22.0" }
+cranelift-serde = { path = "lib/serde", version = "0.22.0", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.22.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.22.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.22.0" }
+cranelift-module = { path = "lib/module", version = "0.22.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.22.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.22.0" }
+cranelift = { path = "lib/umbrella", version = "0.22.0" }
 filecheck = "0.3.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 70ad24c09f..6106ae9d6e 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.21.1"
+version="0.22.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index a4a71c48f7..8b08504993 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.21.1"
+version = "0.22.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 keywords = ["btree", "forest", "set", "map"]
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.21.1", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 4dc06b7c44..2e38b99274 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.21.1"
+version = "0.22.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.21.1", default-features = false }
-cranelift-bforest = { path = "../bforest", version = "0.21.1", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
+cranelift-bforest = { path = "../bforest", version = "0.22.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -25,7 +25,7 @@ log = { version = "0.4.4", default-features = false, features = ["release_max_le
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.21.1" }
+cranelift-codegen-meta = { path = "meta", version = "0.22.0" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 01f19298a5..78eed62419 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.21.1"
+version = "0.22.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index 35c0c69df5..3e7718ad7c 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.21.1"
+version = "0.22.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 1576dce164..b19c5c1775 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.21.1"
+version = "0.22.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.1" }
-cranelift-module = { path = "../module", version = "0.21.1" }
+cranelift-codegen = { path = "../codegen", version = "0.22.0" }
+cranelift-module = { path = "../module", version = "0.22.0" }
 faerie = "0.5.0"
 goblin = "0.0.17"
 failure = "0.1.2"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 6ef936a4b9..edb7f99ef5 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.21.1"
+version = "0.22.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,8 +9,8 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.1" }
-cranelift-reader = { path = "../reader", version = "0.21.1" }
+cranelift-codegen = { path = "../codegen", version = "0.22.0" }
+cranelift-reader = { path = "../reader", version = "0.22.0" }
 file-per-thread-logger = "0.1.1"
 filecheck = "0.3.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 22b4e7076c..d8b156cb86 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.21.1"
+version = "0.22.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 03885eaaf9..b9d320a55c 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.21.1"
+version = "0.22.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.21.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 0bd870ccdc..da585be597 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.21.1"
+version = "0.22.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 000f4a6660..878520a472 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.21.1"
+version = "0.22.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.1" }
+cranelift-codegen = { path = "../codegen", version = "0.22.0" }
 target-lexicon = "0.0.3"
 
 [badges]
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index 10777476b0..c1623cd7ac 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.21.1"
+version = "0.22.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
-cranelift-reader = { path = "../reader", version = "0.21.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
+cranelift-reader = { path = "../reader", version = "0.22.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index d1df478224..b8ed24ed3c 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.21.1"
+version = "0.22.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
-cranelift-module = { path = "../module", version = "0.21.1", default-features = false }
-cranelift-native = { path = "../native", version = "0.21.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.22.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.22.0", default-features = false }
 region = "0.3.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.0.3", default-features = false }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../umbrella", version = "0.21.1" }
-cranelift-frontend = { path = "../frontend", version = "0.21.1" }
-cranelift-entity = { path = "../entity", version = "0.21.1" }
+cranelift = { path = "../umbrella", version = "0.22.0" }
+cranelift-frontend = { path = "../frontend", version = "0.22.0" }
+cranelift-entity = { path = "../entity", version = "0.22.0" }
 
 [features]
 default = ["std"]
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index a15adc2390..7a6241ea60 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.21.1"
+version = "0.22.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,8 +11,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.21.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.22.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index e17b7d73b4..c36f04e876 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.21.1"
+version = "0.22.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,9 +11,9 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.17.2", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.21.1", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.21.1", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.21.1", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.22.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 15a237520ee2bf6cd4c4781105bcdff3cebc3a3e Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Sat, 22 Sep 2018 20:59:54 -0700
Subject: [PATCH 2088/3084] Depend on scroll 0.9.0 to fix Rust 1.25
 compatibility.

---
 lib/faerie/Cargo.toml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index b19c5c1775..9d57533e20 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -16,6 +16,12 @@ goblin = "0.0.17"
 failure = "0.1.2"
 target-lexicon = "0.0.3"
 
+# Temporarily depend on 0.9.0 since 0.9.1 uses i128 which isn't
+# supported in Rust 1.25.0. And scroll 0.9.0 doesn't work with
+# scroll_derive 0.9.5. See https://github.com/m4b/scroll/issues/40 .
+scroll = { version = "=0.9.0", default-features = false }
+scroll_derive = { version = "=0.9.4", default-features = false }
+
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }

From c187e7a345ba7ce310081f7fd0337f159d6786e0 Mon Sep 17 00:00:00 2001
From: Aaron Power 
Date: Sun, 23 Sep 2018 02:41:11 +0100
Subject: [PATCH 2089/3084] Replaced LibCall's fmt::Display implementation

---
 lib/codegen/src/ir/libcall.rs | 14 +-------------
 1 file changed, 1 insertion(+), 13 deletions(-)

diff --git a/lib/codegen/src/ir/libcall.rs b/lib/codegen/src/ir/libcall.rs
index 8bf8cc08c1..95c8f7f067 100644
--- a/lib/codegen/src/ir/libcall.rs
+++ b/lib/codegen/src/ir/libcall.rs
@@ -40,21 +40,9 @@ pub enum LibCall {
     NearestF64,
 }
 
-const NAME: [&str; 9] = [
-    "Probestack",
-    "CeilF32",
-    "CeilF64",
-    "FloorF32",
-    "FloorF64",
-    "TruncF32",
-    "TruncF64",
-    "NearestF32",
-    "NearestF64",
-];
-
 impl fmt::Display for LibCall {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        f.write_str(NAME[*self as usize])
+        fmt::Debug::fmt(self, f)
     }
 }
 

From 3e996c198f443f9ef2aff29d22b487b4bf404d11 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Sat, 22 Sep 2018 22:22:54 -0700
Subject: [PATCH 2090/3084] Remove the scroll 0.9.0 dependency.

Scroll 0.9.2 is now published which fixes the issue in 0.9.1, so
we can drop our fixed dependency on 0.9.0.
---
 lib/faerie/Cargo.toml | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 9d57533e20..b19c5c1775 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -16,12 +16,6 @@ goblin = "0.0.17"
 failure = "0.1.2"
 target-lexicon = "0.0.3"
 
-# Temporarily depend on 0.9.0 since 0.9.1 uses i128 which isn't
-# supported in Rust 1.25.0. And scroll 0.9.0 doesn't work with
-# scroll_derive 0.9.5. See https://github.com/m4b/scroll/issues/40 .
-scroll = { version = "=0.9.0", default-features = false }
-scroll_derive = { version = "=0.9.4", default-features = false }
-
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }

From 09f2b78b14cb9bcf0f0a536edcf2bb0d5a492338 Mon Sep 17 00:00:00 2001
From: Aaron Power 
Date: Sun, 9 Sep 2018 10:28:05 +0100
Subject: [PATCH 2091/3084] Added FunctionBuilder::{call_memcpy, call_memset,
 call_memmove}

---
 lib/codegen/src/ir/libcall.rs |   9 +++
 lib/faerie/src/backend.rs     |   3 +
 lib/frontend/Cargo.toml       |   1 +
 lib/frontend/src/frontend.rs  | 141 +++++++++++++++++++++++++++++++++-
 lib/frontend/src/lib.rs       |   2 +
 5 files changed, 153 insertions(+), 3 deletions(-)

diff --git a/lib/codegen/src/ir/libcall.rs b/lib/codegen/src/ir/libcall.rs
index 95c8f7f067..6e749c72d6 100644
--- a/lib/codegen/src/ir/libcall.rs
+++ b/lib/codegen/src/ir/libcall.rs
@@ -38,6 +38,12 @@ pub enum LibCall {
     NearestF32,
     /// nearest.f64
     NearestF64,
+    /// libc.memcpy
+    Memcpy,
+    /// libc.memset
+    Memset,
+    /// libc.memmove
+    Memmove,
 }
 
 impl fmt::Display for LibCall {
@@ -60,6 +66,9 @@ impl FromStr for LibCall {
             "TruncF64" => Ok(LibCall::TruncF64),
             "NearestF32" => Ok(LibCall::NearestF32),
             "NearestF64" => Ok(LibCall::NearestF64),
+            "Memcpy" => Ok(LibCall::Memcpy),
+            "Memset" => Ok(LibCall::Memset),
+            "Memmove" => Ok(LibCall::Memmove),
             _ => Err(()),
         }
     }
diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index 908eb1411d..d51ad6e489 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -80,6 +80,9 @@ impl FaerieBuilder {
             ir::LibCall::TruncF64 => "trunc".to_owned(),
             ir::LibCall::NearestF32 => "nearbyintf".to_owned(),
             ir::LibCall::NearestF64 => "nearbyint".to_owned(),
+            ir::LibCall::Memcpy => "memcpy".to_owned(),
+            ir::LibCall::Memset => "memset".to_owned(),
+            ir::LibCall::Memmove => "memmove".to_owned(),
         })
     }
 }
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index d8b156cb86..c838682f57 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -11,6 +11,7 @@ readme = "README.md"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
+target-lexicon = { version = "0.0.3", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 405de81f51..d32e1892e7 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -4,9 +4,9 @@ use cranelift_codegen::entity::{EntityMap, EntitySet};
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::function::DisplayFunction;
 use cranelift_codegen::ir::{
-    DataFlowGraph, Ebb, ExtFuncData, FuncRef, Function, GlobalValue, GlobalValueData, Heap,
-    HeapData, Inst, InstBuilderBase, InstructionData, JumpTable, JumpTableData, SigRef, Signature,
-    StackSlot, StackSlotData, Type, Value,
+    types, AbiParam, DataFlowGraph, Ebb, ExtFuncData, ExternalName, FuncRef, Function, GlobalValue,
+    GlobalValueData, Heap, HeapData, Inst, InstBuilder, InstBuilderBase, InstructionData,
+    JumpTable, JumpTableData, LibCall, SigRef, Signature, StackSlot, StackSlotData, Type, Value,
 };
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::packed_option::PackedOption;
@@ -545,6 +545,80 @@ impl<'a> FunctionBuilder<'a> {
     }
 }
 
+/// Helper functions
+impl<'a> FunctionBuilder<'a> {
+    /// Calls libc.memcpy
+    ///
+    /// Copies the `size` bytes from `src` to `dest`, assumes that `src + size`
+    /// won't overlap onto `dest`. If `dest` and `src` overlap, the behavior is
+    /// undefined. Applications in which `dest` and `src` might overlap should
+    /// use `call_memmove` instead.
+    pub fn call_memcpy(&mut self, isa: &TargetIsa, dest: Value, src: Value, size: Value) {
+        let pointer_type = isa.pointer_type();
+        let signature = {
+            let mut s = Signature::new(isa.flags().call_conv());
+            s.params.push(AbiParam::new(pointer_type));
+            s.params.push(AbiParam::new(pointer_type));
+            s.params.push(AbiParam::new(pointer_type));
+            self.import_signature(s)
+        };
+
+        let libc_memcpy = self.import_function(ExtFuncData {
+            name: ExternalName::LibCall(LibCall::Memcpy),
+            signature,
+            colocated: false,
+        });
+
+        self.ins().call(libc_memcpy, &[dest, src, size]);
+    }
+
+    /// Calls libc.memset
+    ///
+    /// Writes `len` bytes of value `ch` to memory starting at `buffer`.
+    pub fn call_memset(&mut self, isa: &TargetIsa, buffer: Value, ch: Value, len: Value) {
+        let pointer_type = isa.pointer_type();
+        let signature = {
+            let mut s = Signature::new(isa.flags().call_conv());
+            s.params.push(AbiParam::new(pointer_type));
+            s.params.push(AbiParam::new(types::I32));
+            s.params.push(AbiParam::new(pointer_type));
+            self.import_signature(s)
+        };
+
+        let libc_memset = self.import_function(ExtFuncData {
+            name: ExternalName::LibCall(LibCall::Memset),
+            signature,
+            colocated: false,
+        });
+
+        self.ins().uextend(types::I32, ch);
+        self.ins().call(libc_memset, &[buffer, ch, len]);
+    }
+
+    /// Calls libc.memmove
+    ///
+    /// Copies `len` bytes from memory starting at `source` to memory starting
+    /// at `dest`. `source` is always read before writing to `dest`.
+    pub fn call_memmove(&mut self, isa: &TargetIsa, dest: Value, source: Value, num: Value) {
+        let pointer_type = isa.pointer_type();
+        let signature = {
+            let mut s = Signature::new(isa.flags().call_conv());
+            s.params.push(AbiParam::new(pointer_type));
+            s.params.push(AbiParam::new(pointer_type));
+            s.params.push(AbiParam::new(pointer_type));
+            self.import_signature(s)
+        };
+
+        let libc_memmove = self.import_function(ExtFuncData {
+            name: ExternalName::LibCall(LibCall::Memmove),
+            signature,
+            colocated: false,
+        });
+
+        self.ins().call(libc_memmove, &[dest, source, num]);
+    }
+}
+
 // Helper functions
 impl<'a> FunctionBuilder<'a> {
     fn move_to_next_basic_block(&mut self) {
@@ -691,4 +765,65 @@ mod tests {
     fn sample_with_lazy_seal() {
         sample_function(true)
     }
+
+    #[test]
+    fn memcpy() {
+        use cranelift_codegen::{isa, settings};
+        use std::str::FromStr;
+
+        let shared_builder = settings::builder();
+        let shared_flags = settings::Flags::new(shared_builder);
+
+        let triple = ::target_lexicon::Triple::from_str("arm").expect("Couldn't create arm triple");
+
+        let target = isa::lookup(triple)
+            .ok()
+            .map(|b| b.finish(shared_flags))
+            .expect("This test requires arm support.");
+
+        let mut sig = Signature::new(target.flags().call_conv());
+        sig.returns.push(AbiParam::new(I32));
+
+        let mut fn_ctx = FunctionBuilderContext::new();
+        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+        {
+            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
+
+            let block0 = builder.create_ebb();
+            let x = Variable::new(0);
+            let y = Variable::new(1);
+            let z = Variable::new(2);
+            builder.declare_var(x, target.pointer_type());
+            builder.declare_var(y, target.pointer_type());
+            builder.declare_var(z, I32);
+            builder.append_ebb_params_for_function_params(block0);
+            builder.switch_to_block(block0);
+
+            let src = builder.use_var(x);
+            let dest = builder.use_var(y);
+            let size = builder.use_var(y);
+            builder.call_memcpy(&*target, dest, src, size);
+            builder.ins().return_(&[size]);
+
+            builder.seal_all_blocks();
+            builder.finalize();
+        }
+
+        assert_eq!(
+            func.display(None).to_string(),
+            "function %sample() -> i32 fast {
+    sig0 = (i32, i32, i32) fast
+    fn0 = %Memcpy sig0
+
+ebb0:
+    v3 = iconst.i32 0
+    v1 -> v3
+    v2 = iconst.i32 0
+    v0 -> v2
+    call fn0(v1, v0, v1)
+    return v1
+}
+"
+        );
+    }
 }
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index 9b90a5df07..73c4abe042 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -177,6 +177,8 @@
 #![cfg_attr(not(feature = "std"), feature(alloc))]
 
 extern crate cranelift_codegen;
+#[cfg(test)]
+extern crate target_lexicon;
 
 pub use frontend::{FunctionBuilder, FunctionBuilderContext};
 pub use variable::Variable;

From 2a7cc7e644d87b4e802e1c4828f1a746789bf9d9 Mon Sep 17 00:00:00 2001
From: Sergey Pepyakin 
Date: Sun, 23 Sep 2018 15:48:17 +0100
Subject: [PATCH 2092/3084] Bump wabt version to 0.6.

---
 cranelift/Cargo.toml | 2 +-
 lib/wasm/Cargo.toml  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 59e940b5d8..39e3ef182a 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -31,7 +31,7 @@ clap = "2.32.0"
 serde = "1.0.8"
 term = "0.5.1"
 capstone = { version = "0.5.0", optional = true }
-wabt = { version = "0.5", optional = true }
+wabt = { version = "0.6", optional = true }
 target-lexicon = "0.0.3"
 pretty_env_logger = "0.2.4"
 file-per-thread-logger = "0.1.1"
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index c36f04e876..bd2c12d46e 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -21,7 +21,7 @@ target-lexicon = { version = "0.0.3", default-features = false }
 log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
 
 [dev-dependencies]
-wabt = "0.5.0"
+wabt = "0.6.0"
 
 [features]
 default = ["std"]

From e07e159bdab6fd8f081c5eb7beb8f761bb267f08 Mon Sep 17 00:00:00 2001
From: Bruce Mitchener 
Date: Sun, 23 Sep 2018 14:21:28 +0700
Subject: [PATCH 2093/3084] filetests: run_passes can take &str, not &String.

---
 cranelift/src/clif-util.rs | 2 +-
 lib/filetests/src/lib.rs   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index d1ad95ed3d..9d87983b40 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -217,7 +217,7 @@ fn main() {
                 rest_cmd.is_present("verbose"),
                 &get_vec(rest_cmd.values_of("pass")),
                 target_val,
-                &rest_cmd.value_of("single-file").unwrap().to_string(),
+                rest_cmd.value_of("single-file").unwrap(),
             ).map(|_time| ())
         }
         ("print-cfg", Some(rest_cmd)) => {
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index 24086d88ab..6d48eb9636 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -93,7 +93,7 @@ pub fn run(verbose: bool, files: &[String]) -> TestResult {
 ///
 /// Directories are scanned recursively for test cases ending in `.clif`.
 ///
-pub fn run_passes(verbose: bool, passes: &[String], target: &str, file: &String) -> TestResult {
+pub fn run_passes(verbose: bool, passes: &[String], target: &str, file: &str) -> TestResult {
     let mut runner = TestRunner::new(verbose);
 
     let path = Path::new(file);

From c8a65a721c881a3e164c41be759fc302343bd307 Mon Sep 17 00:00:00 2001
From: Bruce Mitchener 
Date: Sun, 23 Sep 2018 14:26:28 +0700
Subject: [PATCH 2094/3084] clippy: Remove useless format call.

---
 cranelift/src/clif-util.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 9d87983b40..b38d4c587a 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -264,7 +264,7 @@ fn main() {
 
             result
         }
-        _ => Err(format!("Invalid subcommand.")),
+        _ => Err("Invalid subcommand.".to_owned()),
     };
 
     if let Err(mut msg) = res_util {

From 81d6731e7676465d0a0867d13b0f27876dc65d18 Mon Sep 17 00:00:00 2001
From: Bruce Mitchener 
Date: Sun, 23 Sep 2018 14:27:00 +0700
Subject: [PATCH 2095/3084] clippy: Remove explicit return statements.

---
 lib/codegen/src/ir/dfg.rs | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs
index 9a6a22e9fa..bb8ddbe9c4 100644
--- a/lib/codegen/src/ir/dfg.rs
+++ b/lib/codegen/src/ir/dfg.rs
@@ -165,19 +165,18 @@ fn valid_valuedata(data: &ValueData) -> bool {
             return false;
         }
     }
-    return true;
+    true
 }
 
 impl<'a> Iterator for Values<'a> {
     type Item = Value;
 
     fn next(&mut self) -> Option {
-        return self
-            .inner
+        self.inner
             .by_ref()
             .filter(|kv| valid_valuedata(kv.1))
             .next()
-            .map(|kv| kv.0);
+            .map(|kv| kv.0)
     }
 }
 

From 2c53e2102c43a7b9b1369df4e387c1549000ead1 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Tue, 25 Sep 2018 16:34:32 +0200
Subject: [PATCH 2096/3084] Add a sparse Switch usable instead of JumpTable to
 cranelift-frontend (#517)

* Add a sparse Switch usable instead of JumpTable to cranelift-frontend (fixes #438)
---
 lib/frontend/Cargo.toml    |   1 +
 lib/frontend/src/lib.rs    |   4 +
 lib/frontend/src/switch.rs | 350 +++++++++++++++++++++++++++++++++++++
 3 files changed, 355 insertions(+)
 create mode 100644 lib/frontend/src/switch.rs

diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index d8b156cb86..6c813b396d 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -11,6 +11,7 @@ readme = "README.md"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
+log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
 
 [features]
 default = ["std"]
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index 9b90a5df07..d1f8436a11 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -177,12 +177,16 @@
 #![cfg_attr(not(feature = "std"), feature(alloc))]
 
 extern crate cranelift_codegen;
+#[macro_use]
+extern crate log;
 
 pub use frontend::{FunctionBuilder, FunctionBuilderContext};
+pub use switch::Switch;
 pub use variable::Variable;
 
 mod frontend;
 mod ssa;
+mod switch;
 mod variable;
 
 #[cfg(not(feature = "std"))]
diff --git a/lib/frontend/src/switch.rs b/lib/frontend/src/switch.rs
new file mode 100644
index 0000000000..e86c49bddb
--- /dev/null
+++ b/lib/frontend/src/switch.rs
@@ -0,0 +1,350 @@
+use cranelift_codegen::ir::condcodes::IntCC;
+use cranelift_codegen::ir::*;
+use frontend::FunctionBuilder;
+use std::collections::HashMap;
+
+type EntryIndex = u64;
+
+/// Unlike with `br_table`, `Switch` cases may be sparse or non-0-based.
+/// They emit efficient code using branches, jump tables, or a combination of both.
+#[derive(Debug)]
+pub struct Switch {
+    cases: HashMap,
+}
+
+impl Switch {
+    /// Create a new empty switch
+    pub fn new() -> Self {
+        Switch {
+            cases: HashMap::new(),
+        }
+    }
+
+    /// Set a switch entry
+    pub fn set_entry(&mut self, index: EntryIndex, ebb: Ebb) {
+        let prev = self.cases.insert(index, ebb);
+        assert!(
+            prev.is_none(),
+            "Tried to set the same entry {} twice",
+            index
+        );
+    }
+
+    fn collect_contiguous_case_ranges(self) -> Vec<(EntryIndex, Vec)> {
+        debug!("build_contiguous_case_ranges before: {:#?}", self.cases);
+        let mut cases = self.cases.into_iter().collect::>();
+        cases.sort_by_key(|&(index, _)| index);
+
+        let mut contiguous_case_ranges: Vec<(EntryIndex, Vec)> = vec![];
+        let mut last_index = None;
+        for (index, ebb) in cases {
+            match last_index {
+                None => contiguous_case_ranges.push((index, vec![])),
+                Some(last_index) => {
+                    if index > last_index + 1 {
+                        contiguous_case_ranges.push((index, vec![]));
+                    }
+                }
+            }
+            contiguous_case_ranges.last_mut().unwrap().1.push(ebb);
+            last_index = Some(index);
+        }
+
+        debug!(
+            "build_contiguous_case_ranges after: {:#?}",
+            contiguous_case_ranges
+        );
+
+        contiguous_case_ranges
+    }
+
+    fn build_search_tree(
+        bx: &mut FunctionBuilder,
+        val: Value,
+        otherwise: Ebb,
+        contiguous_case_ranges: Vec<(EntryIndex, Vec)>,
+    ) -> Vec<(EntryIndex, Ebb, Vec)> {
+        let mut cases_and_jt_ebbs = Vec::new();
+
+        // Avoid allocation in the common case
+        if contiguous_case_ranges.len() <= 3 {
+            Self::build_search_branches(
+                bx,
+                val,
+                otherwise,
+                contiguous_case_ranges,
+                &mut cases_and_jt_ebbs,
+            );
+            return cases_and_jt_ebbs;
+        }
+
+        let mut stack: Vec<(Option, Vec<(EntryIndex, Vec)>)> = Vec::new();
+        stack.push((None, contiguous_case_ranges));
+
+        while let Some((ebb, contiguous_case_ranges)) = stack.pop() {
+            if let Some(ebb) = ebb {
+                bx.switch_to_block(ebb);
+            }
+
+            if contiguous_case_ranges.len() <= 3 {
+                Self::build_search_branches(
+                    bx,
+                    val,
+                    otherwise,
+                    contiguous_case_ranges,
+                    &mut cases_and_jt_ebbs,
+                );
+            } else {
+                let split_point = contiguous_case_ranges.len() / 2;
+                let mut left = contiguous_case_ranges;
+                let right = left.split_off(split_point);
+
+                let left_ebb = bx.create_ebb();
+                let right_ebb = bx.create_ebb();
+
+                let should_take_right_side =
+                    bx.ins()
+                        .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, val, right[0].0 as i64);
+                bx.ins().brnz(should_take_right_side, right_ebb, &[]);
+                bx.ins().jump(left_ebb, &[]);
+
+                stack.push((Some(left_ebb), left));
+                stack.push((Some(right_ebb), right));
+            }
+        }
+
+        cases_and_jt_ebbs
+    }
+
+    fn build_search_branches(
+        bx: &mut FunctionBuilder,
+        val: Value,
+        otherwise: Ebb,
+        contiguous_case_ranges: Vec<(EntryIndex, Vec)>,
+        cases_and_jt_ebbs: &mut Vec<(EntryIndex, Ebb, Vec)>,
+    ) {
+        for (first_index, ebbs) in contiguous_case_ranges.into_iter().rev() {
+            if ebbs.len() == 1 {
+                let is_good_val = bx.ins().icmp_imm(IntCC::Equal, val, first_index as i64);
+                bx.ins().brnz(is_good_val, ebbs[0], &[]);
+            } else {
+                let jt_ebb = bx.create_ebb();
+                let is_good_val =
+                    bx.ins()
+                        .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, val, first_index as i64);
+                bx.ins().brnz(is_good_val, jt_ebb, &[]);
+                cases_and_jt_ebbs.push((first_index, jt_ebb, ebbs));
+            }
+        }
+
+        bx.ins().jump(otherwise, &[]);
+    }
+
+    fn build_jump_tables(
+        bx: &mut FunctionBuilder,
+        val: Value,
+        otherwise: Ebb,
+        cases_and_jt_ebbs: Vec<(EntryIndex, Ebb, Vec)>,
+    ) {
+        for (first_index, jt_ebb, ebbs) in cases_and_jt_ebbs.into_iter().rev() {
+            let mut jt_data = JumpTableData::new();
+            for ebb in ebbs {
+                jt_data.push_entry(ebb);
+            }
+            let jump_table = bx.create_jump_table(jt_data);
+
+            bx.switch_to_block(jt_ebb);
+            let discr = bx.ins().iadd_imm(val, (first_index as i64).wrapping_neg());
+            bx.ins().br_table(discr, jump_table);
+            bx.ins().jump(otherwise, &[]);
+        }
+    }
+
+    /// Build the switch
+    ///
+    /// # Arguments
+    ///
+    /// * The function builder to emit to
+    /// * The value to switch on
+    /// * The default ebb
+    pub fn emit(self, bx: &mut FunctionBuilder, val: Value, otherwise: Ebb) {
+        // FIXME icmp(_imm) doesn't have encodings for i8 and i16 on x86(_64) yet
+        let val = match bx.func.dfg.value_type(val) {
+            types::I8 | types::I16 => bx.ins().uextend(types::I32, val),
+            _ => val,
+        };
+
+        let contiguous_case_ranges = self.collect_contiguous_case_ranges();
+        let cases_and_jt_ebbs = Self::build_search_tree(bx, val, otherwise, contiguous_case_ranges);
+        Self::build_jump_tables(bx, val, otherwise, cases_and_jt_ebbs);
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use cranelift_codegen::ir::Function;
+    use frontend::FunctionBuilderContext;
+
+    macro_rules! setup {
+        ($default:expr, [$($index:expr,)*]) => {{
+            let mut func = Function::new();
+            let mut func_ctx = FunctionBuilderContext::new();
+            {
+                let mut bx = FunctionBuilder::new(&mut func, &mut func_ctx);
+                let ebb = bx.create_ebb();
+                bx.switch_to_block(ebb);
+                let val = bx.ins().iconst(types::I8, 0);
+                let mut switch = Switch::new();
+                $(
+                    let ebb = bx.create_ebb();
+                    switch.set_entry($index, ebb);
+                )*
+                switch.emit(&mut bx, val, Ebb::with_number($default).unwrap());
+            }
+            func
+                .to_string()
+                .trim_left_matches("function u0:0() fast {\n")
+                .trim_right_matches("\n}\n")
+                .to_string()
+        }};
+    }
+
+    #[test]
+    fn switch_zero() {
+        let func = setup!(0, [0,]);
+        assert_eq!(
+            func,
+            "ebb0:
+    v0 = iconst.i8 0
+    v1 = uextend.i32 v0
+    v2 = icmp_imm eq v1, 0
+    brnz v2, ebb1
+    jump ebb0"
+        );
+    }
+
+    #[test]
+    fn switch_single() {
+        let func = setup!(0, [1,]);
+        assert_eq!(
+            func,
+            "ebb0:
+    v0 = iconst.i8 0
+    v1 = uextend.i32 v0
+    v2 = icmp_imm eq v1, 1
+    brnz v2, ebb1
+    jump ebb0"
+        );
+    }
+
+    #[test]
+    fn switch_bool() {
+        let func = setup!(0, [0, 1,]);
+        assert_eq!(
+            func,
+            "    jt0 = jump_table ebb1, ebb2
+
+ebb0:
+    v0 = iconst.i8 0
+    v1 = uextend.i32 v0
+    v2 = icmp_imm uge v1, 0
+    brnz v2, ebb3
+    jump ebb0
+
+ebb3:
+    v3 = iadd_imm.i32 v1, 0
+    br_table v3, jt0
+    jump ebb0"
+        );
+    }
+
+    #[test]
+    fn switch_two_gap() {
+        let func = setup!(0, [0, 2,]);
+        assert_eq!(
+            func,
+            "ebb0:
+    v0 = iconst.i8 0
+    v1 = uextend.i32 v0
+    v2 = icmp_imm eq v1, 2
+    brnz v2, ebb2
+    v3 = icmp_imm eq v1, 0
+    brnz v3, ebb1
+    jump ebb0"
+        );
+    }
+
+    #[test]
+    fn switch_many() {
+        let func = setup!(0, [0, 1, 5, 7, 10, 11, 12,]);
+        assert_eq!(
+            func,
+            "    jt0 = jump_table ebb1, ebb2
+    jt1 = jump_table ebb5, ebb6, ebb7
+
+ebb0:
+    v0 = iconst.i8 0
+    v1 = uextend.i32 v0
+    v2 = icmp_imm uge v1, 7
+    brnz v2, ebb9
+    jump ebb8
+
+ebb9:
+    v3 = icmp_imm.i32 uge v1, 10
+    brnz v3, ebb10
+    v4 = icmp_imm.i32 eq v1, 7
+    brnz v4, ebb4
+    jump ebb0
+
+ebb8:
+    v5 = icmp_imm.i32 eq v1, 5
+    brnz v5, ebb3
+    v6 = icmp_imm.i32 uge v1, 0
+    brnz v6, ebb11
+    jump ebb0
+
+ebb11:
+    v7 = iadd_imm.i32 v1, 0
+    br_table v7, jt0
+    jump ebb0
+
+ebb10:
+    v8 = iadd_imm.i32 v1, -10
+    br_table v8, jt1
+    jump ebb0"
+        );
+    }
+
+    #[test]
+    fn switch_min_index_value() {
+        let func = setup!(0, [::std::i64::MIN as u64, 1,]);
+        assert_eq!(
+            func,
+            "ebb0:
+    v0 = iconst.i8 0
+    v1 = uextend.i32 v0
+    v2 = icmp_imm eq v1, 0x8000_0000_0000_0000
+    brnz v2, ebb1
+    v3 = icmp_imm eq v1, 1
+    brnz v3, ebb2
+    jump ebb0"
+        );
+    }
+
+    #[test]
+    fn switch_max_index_value() {
+        let func = setup!(0, [::std::i64::MAX as u64, 1,]);
+        assert_eq!(
+            func,
+            "ebb0:
+    v0 = iconst.i8 0
+    v1 = uextend.i32 v0
+    v2 = icmp_imm eq v1, 0x7fff_ffff_ffff_ffff
+    brnz v2, ebb1
+    v3 = icmp_imm eq v1, 1
+    brnz v3, ebb2
+    jump ebb0"
+        )
+    }
+}

From 2eec1469a83f426d5c4c2c6ec94f2033d23542e5 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 26 Sep 2018 01:10:23 +0200
Subject: [PATCH 2097/3084] Legalize some more i8/i16 intructions (#524)

* Legalize some more i8/i16 intructions
---
 .../isa/x86/legalize-byte-ops-i8.clif         |  36 +++
 .../filetests/isa/x86/legalize-icmp-i8.clif   |  19 ++
 .../filetests/isa/x86/legalize-shlr-i8.clif   |  24 ++
 .../filetests/isa/x86/legalize-urem-i8.clif   |  15 ++
 cranelift/filetests/legalizer/bitrev.clif     | 169 +++++++++-----
 lib/codegen/meta-python/base/legalize.py      | 217 +++++++++++++-----
 lib/codegen/meta-python/cdsl/predicates.py    |   2 +-
 lib/codegen/src/isa/riscv/enc_tables.rs       |   1 -
 lib/codegen/src/isa/x86/enc_tables.rs         |   1 -
 9 files changed, 368 insertions(+), 116 deletions(-)
 create mode 100644 cranelift/filetests/isa/x86/legalize-byte-ops-i8.clif
 create mode 100644 cranelift/filetests/isa/x86/legalize-icmp-i8.clif
 create mode 100644 cranelift/filetests/isa/x86/legalize-shlr-i8.clif
 create mode 100644 cranelift/filetests/isa/x86/legalize-urem-i8.clif

diff --git a/cranelift/filetests/isa/x86/legalize-byte-ops-i8.clif b/cranelift/filetests/isa/x86/legalize-byte-ops-i8.clif
new file mode 100644
index 0000000000..b0a318b8d4
--- /dev/null
+++ b/cranelift/filetests/isa/x86/legalize-byte-ops-i8.clif
@@ -0,0 +1,36 @@
+test compile
+target x86_64
+
+; regex: V=v\d+
+
+function u0:0(i8, i8) fast {
+fn0 = %black_box(i8)
+ss0 = explicit_slot 1 ; black box
+
+ebb0(v0: i8, v1: i8):
+    v99 = stack_addr.i64 ss0
+
+    ; check: istore8 $(V), $(V)
+
+    v2 = band v0, v1
+    store v2, v99
+    v3 = bor v0, v1
+    store v3, v99
+    v4 = bxor v0, v1
+    store v4, v99
+    v5 = bnot v0
+    store v5, v99
+    v6 = band_not v0, v1
+    store v6, v99
+    v7 = bor_not v0, v1
+    store v7, v99
+    v8 = bxor_not v0, v1
+    store v8, v99
+    v9 = band_imm v0, 42
+    store v9, v99
+    v10 = bor_imm v0, 42
+    store v10, v99
+    v11 = bxor_imm v0, 42
+    store v11, v99
+    return
+}
diff --git a/cranelift/filetests/isa/x86/legalize-icmp-i8.clif b/cranelift/filetests/isa/x86/legalize-icmp-i8.clif
new file mode 100644
index 0000000000..41bd27950f
--- /dev/null
+++ b/cranelift/filetests/isa/x86/legalize-icmp-i8.clif
@@ -0,0 +1,19 @@
+test compile
+target x86_64
+
+; regex: V=v\d+
+
+function u0:0(i8, i8) -> i8 fast {
+ebb0(v0: i8, v1: i8):
+    v2 = icmp_imm sle v0, 0
+    ; check: $(e1=$V) = sextend.i32 v0
+    ; nextln: v2 = icmp_imm sle $e1, 0
+    v3 = bint.i8 v2
+    v4 = icmp eq v0, v1
+    ; check: $(e2=$V) = uextend.i32 v0
+    ; nextln: $(e3=$V) = uextend.i32 v1
+    ; nextln: v4 = icmp eq $e2, $e3
+    v5 = bint.i8 v4
+    v6 = iadd v3, v5
+    return v6
+}
diff --git a/cranelift/filetests/isa/x86/legalize-shlr-i8.clif b/cranelift/filetests/isa/x86/legalize-shlr-i8.clif
new file mode 100644
index 0000000000..dbd0da1204
--- /dev/null
+++ b/cranelift/filetests/isa/x86/legalize-shlr-i8.clif
@@ -0,0 +1,24 @@
+test compile
+target x86_64
+
+; regex: V=v\d+
+
+function u0:0(i8, i8) -> i8 fast {
+ebb0(v0: i8, v1: i8):
+    v2 = ishl v0, v1
+    ; check: $(e1=$V) = uextend.i32 v0
+    ; check: $(r1=$V) = ishl $e1, v1
+    ; check v2 = ireduce.i8 $r1
+    v3 = ushr v0, v1
+    ; check: $(e2=$V) = uextend.i32 v0
+    ; check: $(r2=$V) = ushr $e2, v1
+    ; check v2 = ireduce.i8 $r2
+    v4 = sshr v0, v1
+    ; check: $(e3=$V) = sextend.i32 v0
+    ; check: $(r3=$V) = sshr $e3, v1
+    ; check v2 = ireduce.i8 $r3
+
+    v5 = iadd v2, v3
+    v6 = iadd v4, v5
+    return v6
+}
diff --git a/cranelift/filetests/isa/x86/legalize-urem-i8.clif b/cranelift/filetests/isa/x86/legalize-urem-i8.clif
new file mode 100644
index 0000000000..0c66a3f580
--- /dev/null
+++ b/cranelift/filetests/isa/x86/legalize-urem-i8.clif
@@ -0,0 +1,15 @@
+test compile
+target x86_64
+
+; regex: V=v\d+
+
+function u0:0(i8, i8) -> i8 fast {
+ebb0(v0: i8, v1: i8):
+    v2 = urem v0, v1
+    ; check: $(a=$V) = uextend.i32 v0
+    ; nextln: $(b=$V) = uextend.i32 v1
+    ; nextln: $(c=$V) = iconst.i32 0
+    ; nextln: $(V), $(r=$V) = x86_udivmodx $a, $c, $b
+    ; nextln: v2 = ireduce.i8 $r
+    return v2
+}
diff --git a/cranelift/filetests/legalizer/bitrev.clif b/cranelift/filetests/legalizer/bitrev.clif
index 0f75c373a9..b4098b87b4 100644
--- a/cranelift/filetests/legalizer/bitrev.clif
+++ b/cranelift/filetests/legalizer/bitrev.clif
@@ -6,68 +6,125 @@ ebb0(v0: i8):
     v1 = bitrev.i8 v0
     return v1
 }
-; check: v2 = band_imm v0, 170
-; check: v3 = ushr_imm v2, 1
-; check: v4 = band_imm v0, 85
-; check: v5 = ishl_imm v4, 1
-; check: v16 = uextend.i32 v3
-; check: v17 = uextend.i32 v5
-; check: v18 = bor v16, v17
-; check: v6 = ireduce.i8 v18
-; check: v7 = band_imm v6, 204
-; check: v8 = ushr_imm v7, 2
-; check: v9 = band_imm v6, 51
-; check: v10 = ushr_imm v9, 2
-; check: v19 = uextend.i32 v8
-; check: v20 = uextend.i32 v10
-; check: v21 = bor v19, v20
-; check: v11 = ireduce.i8 v21
-; check: v12 = band_imm v11, 240
-; check: v13 = ushr_imm v12, 4
-; check: v14 = band_imm v11, 15
-; check: v15 = ishl_imm v14, 4
-; check: v22 = uextend.i32 v13
-; check: v23 = uextend.i32 v15
-; check: v24 = bor v22, v23
-; check: v1 = ireduce.i8 v24
+; check: v16 = uextend.i32 v0
+; check: v17 = band_imm v16, 170
+; check: v2 = ireduce.i8 v17
+; check: v18 = uextend.i32 v2
+; check: v19 = ushr_imm v18, 1
+; check: v3 = ireduce.i8 v19
+; check: v20 = uextend.i32 v0
+; check: v21 = band_imm v20, 85
+; check: v4 = ireduce.i8 v21
+; check: v22 = uextend.i32 v4
+; check: v23 = ishl_imm v22, 1
+; check: v5 = ireduce.i8 v23
+; check: v24 = uextend.i32 v3
+; check: v25 = uextend.i32 v5
+; check: v26 = bor v24, v25
+; check: v6 = ireduce.i8 v26
+; check: v27 = uextend.i32 v6
+; check: v28 = band_imm v27, 204
+; check: v7 = ireduce.i8 v28
+; check: v29 = uextend.i32 v7
+; check: v30 = ushr_imm v29, 2
+; check: v8 = ireduce.i8 v30
+; check: v31 = uextend.i32 v6
+; check: v32 = band_imm v31, 51
+; check: v9 = ireduce.i8 v32
+; check: v33 = uextend.i32 v9
+; check: v34 = ushr_imm v33, 2
+; check: v10 = ireduce.i8 v34
+; check: v35 = uextend.i32 v8
+; check: v36 = uextend.i32 v10
+; check: v37 = bor v35, v36
+; check: v11 = ireduce.i8 v37
+; check: v38 = uextend.i32 v11
+; check: v39 = band_imm v38, 240
+; check: v12 = ireduce.i8 v39
+; check: v40 = uextend.i32 v12
+; check: v41 = ushr_imm v40, 4
+; check: v13 = ireduce.i8 v41
+; check: v42 = uextend.i32 v11
+; check: v43 = band_imm v42, 15
+; check: v14 = ireduce.i8 v43
+; check: v44 = uextend.i32 v14
+; check: v45 = ishl_imm v44, 4
+; check: v15 = ireduce.i8 v45
+; check: v46 = uextend.i32 v13
+; check: v47 = uextend.i32 v15
+; check: v48 = bor v46, v47
+; check: v1 = ireduce.i8 v48
+; check: return v1
 
 function %reverse_bits_16(i16) -> i16 {
 ebb0(v0: i16):
     v1 = bitrev.i16 v0
     return v1
 }
-; check: v2 = band_imm v0, 0xaaaa
-; check: v3 = ushr_imm v2, 1
-; check: v4 = band_imm v0, 0x5555
-; check: v5 = ishl_imm v4, 1
-; check: v21 = uextend.i32 v3
-; check: v22 = uextend.i32 v5
-; check: v23 = bor v21, v22
-; check: v6 = ireduce.i16 v23
-; check: v7 = band_imm v6, 0xcccc
-; check: v8 = ushr_imm v7, 2
-; check: v9 = band_imm v6, 0x3333
-; check: v10 = ushr_imm v9, 2
-; check: v24 = uextend.i32 v8
-; check: v25 = uextend.i32 v10
-; check: v26 = bor v24, v25
-; check: v11 = ireduce.i16 v26
-; check: v12 = band_imm v11, 0xf0f0
-; check: v13 = ushr_imm v12, 4
-; check: v14 = band_imm v11, 3855
-; check: v15 = ishl_imm v14, 4
-; check: v27 = uextend.i32 v13
-; check: v28 = uextend.i32 v15
-; check: v29 = bor v27, v28
-; check: v16 = ireduce.i16 v29
-; check: v17 = band_imm v16, 0xff00
-; check: v18 = ushr_imm v17, 8
-; check: v19 = band_imm v16, 255
-; check: v20 = ishl_imm v19, 8
-; check: v30 = uextend.i32 v18
-; check: v31 = uextend.i32 v20
-; check: v32 = bor v30, v31
-; check: v1 = ireduce.i16 v32
+; check: v21 = uextend.i32 v0
+; check: v22 = band_imm v21, 0xaaaa
+; check: v2 = ireduce.i16 v22
+; check: v23 = uextend.i32 v2
+; check: v24 = ushr_imm v23, 1
+; check: v3 = ireduce.i16 v24
+; check: v25 = uextend.i32 v0
+; check: v26 = band_imm v25, 0x5555
+; check: v4 = ireduce.i16 v26
+; check: v27 = uextend.i32 v4
+; check: v28 = ishl_imm v27, 1
+; check: v5 = ireduce.i16 v28
+; check: v29 = uextend.i32 v3
+; check: v30 = uextend.i32 v5
+; check: v31 = bor v29, v30
+; check: v6 = ireduce.i16 v31
+; check: v32 = uextend.i32 v6
+; check: v33 = band_imm v32, 0xcccc
+; check: v7 = ireduce.i16 v33
+; check: v34 = uextend.i32 v7
+; check: v35 = ushr_imm v34, 2
+; check: v8 = ireduce.i16 v35
+; check: v36 = uextend.i32 v6
+; check: v37 = band_imm v36, 0x3333
+; check: v9 = ireduce.i16 v37
+; check: v38 = uextend.i32 v9
+; check: v39 = ushr_imm v38, 2
+; check: v10 = ireduce.i16 v39
+; check: v40 = uextend.i32 v8
+; check: v41 = uextend.i32 v10
+; check: v42 = bor v40, v41
+; check: v11 = ireduce.i16 v42
+; check: v43 = uextend.i32 v11
+; check: v44 = band_imm v43, 0xf0f0
+; check: v12 = ireduce.i16 v44
+; check: v45 = uextend.i32 v12
+; check: v46 = ushr_imm v45, 4
+; check: v13 = ireduce.i16 v46
+; check: v47 = uextend.i32 v11
+; check: v48 = band_imm v47, 3855
+; check: v14 = ireduce.i16 v48
+; check: v49 = uextend.i32 v14
+; check: v50 = ishl_imm v49, 4
+; check: v15 = ireduce.i16 v50
+; check: v51 = uextend.i32 v13
+; check: v52 = uextend.i32 v15
+; check: v53 = bor v51, v52
+; check: v16 = ireduce.i16 v53
+; check: v54 = uextend.i32 v16
+; check: v55 = band_imm v54, 0xff00
+; check: v17 = ireduce.i16 v55
+; check: v56 = uextend.i32 v17
+; check: v57 = ushr_imm v56, 8
+; check: v18 = ireduce.i16 v57
+; check: v58 = uextend.i32 v16
+; check: v59 = band_imm v58, 255
+; check: v19 = ireduce.i16 v59
+; check: v60 = uextend.i32 v19
+; check: v61 = ishl_imm v60, 8
+; check: v20 = ireduce.i16 v61
+; check: v62 = uextend.i32 v18
+; check: v63 = uextend.i32 v20
+; check: v64 = bor v62, v63
+; check: v1 = ireduce.i16 v64
 ; check: return v1
 
 function %reverse_bits_32(i32) -> i32 {
diff --git a/lib/codegen/meta-python/base/legalize.py b/lib/codegen/meta-python/base/legalize.py
index 9f6175f138..ec82ed9d3f 100644
--- a/lib/codegen/meta-python/base/legalize.py
+++ b/lib/codegen/meta-python/base/legalize.py
@@ -30,6 +30,13 @@ from .instructions import bitrev
 from cdsl.ast import Var
 from cdsl.xform import Rtl, XFormGroup
 
+try:
+    from typing import TYPE_CHECKING # noqa
+    if TYPE_CHECKING:
+        from cdsl.instructions import Instruction # noqa
+except ImportError:
+    TYPE_CHECKING = False
+
 
 narrow = XFormGroup('narrow', """
         Legalize instructions by narrowing.
@@ -89,6 +96,7 @@ expand.custom_legalize(insts.stack_store, 'expand_stack_store')
 
 x = Var('x')
 y = Var('y')
+z = Var('z')
 a = Var('a')
 a1 = Var('a1')
 a2 = Var('a2')
@@ -174,6 +182,92 @@ narrow.legalize(
             a << iconcat(al, ah)
         ))
 
+
+def widen_one_arg(signed, op):
+    # type: (bool, Instruction) -> None
+    for int_ty in [types.i8, types.i16]:
+        if signed:
+            widen.legalize(
+                a << op.bind(int_ty)(b),
+                Rtl(
+                    x << sextend.i32(b),
+                    z << op.i32(x),
+                    a << ireduce.bind(int_ty)(z)
+                ))
+        else:
+            widen.legalize(
+                a << op.bind(int_ty)(b),
+                Rtl(
+                    x << uextend.i32(b),
+                    z << op.i32(x),
+                    a << ireduce.bind(int_ty)(z)
+                ))
+
+
+def widen_two_arg(signed, op):
+    # type: (bool, Instruction) -> None
+    for int_ty in [types.i8, types.i16]:
+        if signed:
+            widen.legalize(
+                a << op.bind(int_ty)(b, c),
+                Rtl(
+                    x << sextend.i32(b),
+                    y << sextend.i32(c),
+                    z << op.i32(x, y),
+                    a << ireduce.bind(int_ty)(z)
+                ))
+        else:
+            widen.legalize(
+                a << op.bind(int_ty)(b, c),
+                Rtl(
+                    x << uextend.i32(b),
+                    y << uextend.i32(c),
+                    z << op.i32(x, y),
+                    a << ireduce.bind(int_ty)(z)
+                ))
+
+
+def widen_imm(signed, op):
+    # type: (bool, Instruction) -> None
+    for int_ty in [types.i8, types.i16]:
+        if signed:
+            widen.legalize(
+                a << op.bind(int_ty)(b, c),
+                Rtl(
+                    x << sextend.i32(b),
+                    z << op.i32(x, c),
+                    a << ireduce.bind(int_ty)(z)
+                ))
+        else:
+            widen.legalize(
+                a << op.bind(int_ty)(b, c),
+                Rtl(
+                    x << uextend.i32(b),
+                    z << op.i32(x, c),
+                    a << ireduce.bind(int_ty)(z)
+                ))
+
+
+for binop in [iadd, isub, imul, udiv, urem]:
+    widen_two_arg(False, binop)
+
+widen_two_arg(True, sdiv)
+
+widen_one_arg(False, bnot)
+
+for binop in [iadd_imm, imul_imm, udiv_imm, urem_imm]:
+    widen_imm(False, binop)
+
+for binop in [sdiv_imm, srem_imm]:
+    widen_imm(True, binop)
+
+# bit ops
+for binop in [band, bor, bxor, band_not, bor_not, bxor_not]:
+    widen_two_arg(False, binop)
+
+for binop in [band_imm, bor_imm, bxor_imm]:
+    widen_imm(False, binop)
+
 for int_ty in [types.i8, types.i16]:
     widen.legalize(
         a << iconst.bind(int_ty)(b),
@@ -210,63 +304,6 @@ widen.legalize(
         a << ireduce(b)
     ))
 
-for binop in [iadd, isub, imul, udiv, band, bor, bxor]:
-    for int_ty in [types.i8, types.i16]:
-        widen.legalize(
-            a << binop.bind(int_ty)(x, y),
-            Rtl(
-                b << uextend.i32(x),
-                c << uextend.i32(y),
-                d << binop(b, c),
-                a << ireduce(d)
-            )
-        )
-
-for binop in [sdiv]:
-    for int_ty in [types.i8, types.i16]:
-        widen.legalize(
-            a << binop.bind(int_ty)(x, y),
-            Rtl(
-                b << sextend.i32(x),
-                c << sextend.i32(y),
-                d << binop(b, c),
-                a << ireduce(d)
-            )
-        )
-
-for unop in [bnot]:
-    for int_ty in [types.i8, types.i16]:
-        widen.legalize(
-            a << unop.bind(int_ty)(x),
-            Rtl(
-                b << sextend.i32(x),
-                d << unop(b),
-                a << ireduce(d)
-            )
-        )
-
-for binop in [iadd_imm, imul_imm, udiv_imm]:
-    for int_ty in [types.i8, types.i16]:
-        widen.legalize(
-            a << binop.bind(int_ty)(x, y),
-            Rtl(
-                b << uextend.i32(x),
-                c << binop(b, y),
-                a << ireduce(c)
-            )
-        )
-
-for binop in [sdiv_imm]:
-    for int_ty in [types.i8, types.i16]:
-        widen.legalize(
-            a << binop.bind(int_ty)(x, y),
-            Rtl(
-                b << sextend.i32(x),
-                c << binop(b, y),
-                a << ireduce(c)
-            )
-        )
-
 for int_ty in [types.i8, types.i16]:
     widen.legalize(
         br_table.bind(int_ty)(x, y),
@@ -285,6 +322,72 @@ for int_ty in [types.i8, types.i16]:
         )
     )
 
+for int_ty in [types.i8, types.i16]:
+    for op in [ushr_imm, ishl_imm]:
+        widen.legalize(
+            a << op.bind(int_ty)(b, c),
+            Rtl(
+                x << uextend.i32(b),
+                z << op.i32(x, c),
+                a << ireduce.bind(int_ty)(z)
+            ))
+
+    widen.legalize(
+        a << ishl.bind(int_ty)(b, c),
+        Rtl(
+            x << uextend.i32(b),
+            z << ishl.i32(x, c),
+            a << ireduce.bind(int_ty)(z)
+        ))
+
+    widen.legalize(
+        a << ushr.bind(int_ty)(b, c),
+        Rtl(
+            x << uextend.i32(b),
+            z << ushr.i32(x, c),
+            a << ireduce.bind(int_ty)(z)
+        ))
+
+    widen.legalize(
+        a << sshr.bind(int_ty)(b, c),
+        Rtl(
+            x << sextend.i32(b),
+            z << sshr.i32(x, c),
+            a << ireduce.bind(int_ty)(z)
+        ))
+
+    for w_cc in [
+        intcc.eq, intcc.ne, intcc.ugt, intcc.ult, intcc.uge, intcc.ule
+    ]:
+        widen.legalize(
+            a << insts.icmp_imm.bind(int_ty)(w_cc, b, c),
+            Rtl(
+                x << uextend.i32(b),
+                a << insts.icmp_imm(w_cc, x, c)
+            ))
+        widen.legalize(
+            a << insts.icmp.bind(int_ty)(w_cc, b, c),
+            Rtl(
+                x << uextend.i32(b),
+                y << uextend.i32(c),
+                a << insts.icmp.i32(w_cc, x, y)
+            ))
+    for w_cc in [intcc.sgt, intcc.slt, intcc.sge, intcc.sle]:
+        widen.legalize(
+            a << insts.icmp_imm.bind(int_ty)(w_cc, b, c),
+            Rtl(
+                x << sextend.i32(b),
+                a << insts.icmp_imm(w_cc, x, c)
+            ))
+        widen.legalize(
+            a << insts.icmp.bind(int_ty)(w_cc, b, c),
+            Rtl(
+                x << sextend.i32(b),
+                y << sextend.i32(c),
+                a << insts.icmp(w_cc, x, y)
+            )
+        )
+
 # Expand integer operations with carry for RISC architectures that don't have
 # the flags.
 expand.legalize(
diff --git a/lib/codegen/meta-python/cdsl/predicates.py b/lib/codegen/meta-python/cdsl/predicates.py
index ee7b81f3da..c81173ad5c 100644
--- a/lib/codegen/meta-python/cdsl/predicates.py
+++ b/lib/codegen/meta-python/cdsl/predicates.py
@@ -244,7 +244,7 @@ class FieldPredicate(object):
         """
         # Prepend `field` to the predicate function arguments.
         args = (self.field.rust_name(),) + tuple(map(str, self.args))
-        return 'predicates::{}({})'.format(self.function, ', '.join(args))
+        return '::predicates::{}({})'.format(self.function, ', '.join(args))
 
 
 class IsEqual(FieldPredicate):
diff --git a/lib/codegen/src/isa/riscv/enc_tables.rs b/lib/codegen/src/isa/riscv/enc_tables.rs
index 46b2458b49..5f7c084a67 100644
--- a/lib/codegen/src/isa/riscv/enc_tables.rs
+++ b/lib/codegen/src/isa/riscv/enc_tables.rs
@@ -6,7 +6,6 @@ use isa;
 use isa::constraints::*;
 use isa::enc_tables::*;
 use isa::encoding::RecipeSizing;
-use predicates;
 
 // Include the generated encoding tables:
 // - `LEVEL1_RV32`
diff --git a/lib/codegen/src/isa/x86/enc_tables.rs b/lib/codegen/src/isa/x86/enc_tables.rs
index ebaaa526f7..65b7d7c38e 100644
--- a/lib/codegen/src/isa/x86/enc_tables.rs
+++ b/lib/codegen/src/isa/x86/enc_tables.rs
@@ -10,7 +10,6 @@ use isa;
 use isa::constraints::*;
 use isa::enc_tables::*;
 use isa::encoding::RecipeSizing;
-use predicates;
 
 include!(concat!(env!("OUT_DIR"), "/encoding-x86.rs"));
 include!(concat!(env!("OUT_DIR"), "/legalize-x86.rs"));

From d266b1a42db10e997954f4aefa5c664c47e384b2 Mon Sep 17 00:00:00 2001
From: Muhammad Mominul Huque 
Date: Thu, 27 Sep 2018 01:03:44 +0600
Subject: [PATCH 2098/3084] Rename `EntityMap` to `SecondaryMap` (#528)

* Rename `EntityMap` to `SecondaryMap`
---
 lib/codegen/src/dominator_tree.rs    | 10 +++++-----
 lib/codegen/src/flowgraph.rs         |  6 +++---
 lib/codegen/src/ir/dfg.rs            | 10 +++++-----
 lib/codegen/src/ir/function.rs       | 10 +++++-----
 lib/codegen/src/ir/layout.rs         | 10 +++++-----
 lib/codegen/src/ir/mod.rs            | 10 +++++-----
 lib/codegen/src/loop_analysis.rs     |  6 +++---
 lib/codegen/src/print_errors.rs      |  6 +++---
 lib/codegen/src/regalloc/virtregs.rs | 10 +++++-----
 lib/codegen/src/verifier/flags.rs    |  6 +++---
 lib/codegen/src/write.rs             | 16 ++++++++--------
 lib/entity/README.md                 | 12 ++++++------
 lib/entity/src/lib.rs                |  6 +++---
 lib/entity/src/map.rs                | 20 ++++++++++----------
 lib/entity/src/set.rs                |  2 +-
 lib/entity/src/sparse.rs             | 20 ++++++++++----------
 lib/frontend/src/frontend.rs         | 10 +++++-----
 lib/frontend/src/ssa.rs              | 12 ++++++------
 18 files changed, 91 insertions(+), 91 deletions(-)

diff --git a/lib/codegen/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs
index a246f671bc..2198fbf2bf 100644
--- a/lib/codegen/src/dominator_tree.rs
+++ b/lib/codegen/src/dominator_tree.rs
@@ -1,6 +1,6 @@
 //! A Dominator Tree represented as mappings of Ebbs to their immediate dominator.
 
-use entity::EntityMap;
+use entity::SecondaryMap;
 use flowgraph::{BasicBlock, ControlFlowGraph};
 use ir::instructions::BranchInfo;
 use ir::{Ebb, ExpandedProgramPoint, Function, Inst, Layout, ProgramOrder, Value};
@@ -38,7 +38,7 @@ struct DomNode {
 
 /// The dominator tree for a single function.
 pub struct DominatorTree {
-    nodes: EntityMap,
+    nodes: SecondaryMap,
 
     /// CFG post-order of all reachable EBBs.
     postorder: Vec,
@@ -217,7 +217,7 @@ impl DominatorTree {
     /// function.
     pub fn new() -> Self {
         Self {
-            nodes: EntityMap::new(),
+            nodes: SecondaryMap::new(),
             postorder: Vec::new(),
             stack: Vec::new(),
             valid: false,
@@ -505,7 +505,7 @@ impl DominatorTree {
 /// The information in this auxillary data structure is not easy to update when the control flow
 /// graph changes, which is why it is kept separate.
 pub struct DominatorTreePreorder {
-    nodes: EntityMap,
+    nodes: SecondaryMap,
 
     // Scratch memory used by `compute_postorder()`.
     stack: Vec,
@@ -533,7 +533,7 @@ impl DominatorTreePreorder {
     /// Create a new blank `DominatorTreePreorder`.
     pub fn new() -> Self {
         Self {
-            nodes: EntityMap::new(),
+            nodes: SecondaryMap::new(),
             stack: Vec::new(),
         }
     }
diff --git a/lib/codegen/src/flowgraph.rs b/lib/codegen/src/flowgraph.rs
index b84da7cd0b..97c3f932f5 100644
--- a/lib/codegen/src/flowgraph.rs
+++ b/lib/codegen/src/flowgraph.rs
@@ -24,7 +24,7 @@
 //! and `(Ebb0, jmp Ebb2)` respectively.
 
 use bforest;
-use entity::EntityMap;
+use entity::SecondaryMap;
 use ir::instructions::BranchInfo;
 use ir::{Ebb, Function, Inst};
 use std::mem;
@@ -72,7 +72,7 @@ struct CFGNode {
 /// and successors where predecessors are basic blocks and successors are
 /// extended basic blocks.
 pub struct ControlFlowGraph {
-    data: EntityMap,
+    data: SecondaryMap,
     pred_forest: bforest::MapForest,
     succ_forest: bforest::SetForest,
     valid: bool,
@@ -82,7 +82,7 @@ impl ControlFlowGraph {
     /// Allocate a new blank control flow graph.
     pub fn new() -> Self {
         Self {
-            data: EntityMap::new(),
+            data: SecondaryMap::new(),
             valid: false,
             pred_forest: bforest::MapForest::new(),
             succ_forest: bforest::SetForest::new(),
diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs
index bb8ddbe9c4..7bcf17e949 100644
--- a/lib/codegen/src/ir/dfg.rs
+++ b/lib/codegen/src/ir/dfg.rs
@@ -1,6 +1,6 @@
 //! Data flow graph tracking Instructions, Values, and EBBs.
 
-use entity::{self, EntityMap, PrimaryMap};
+use entity::{self, PrimaryMap, SecondaryMap};
 use ir;
 use ir::builder::ReplaceBuilder;
 use ir::extfunc::ExtFuncData;
@@ -34,7 +34,7 @@ pub struct DataFlowGraph {
     ///
     /// This map gets resized automatically by `make_inst()` so it is always in sync with the
     /// primary `insts` map.
-    results: EntityMap,
+    results: SecondaryMap,
 
     /// Extended basic blocks in the function and their parameters.
     ///
@@ -67,7 +67,7 @@ impl DataFlowGraph {
     pub fn new() -> Self {
         Self {
             insts: PrimaryMap::new(),
-            results: EntityMap::new(),
+            results: SecondaryMap::new(),
             ebbs: PrimaryMap::new(),
             value_lists: ValueListPool::new(),
             values: PrimaryMap::new(),
@@ -90,7 +90,7 @@ impl DataFlowGraph {
     /// Get the total number of instructions created in this function, whether they are currently
     /// inserted in the layout or not.
     ///
-    /// This is intended for use with `EntityMap::with_capacity`.
+    /// This is intended for use with `SecondaryMap::with_capacity`.
     pub fn num_insts(&self) -> usize {
         self.insts.len()
     }
@@ -103,7 +103,7 @@ impl DataFlowGraph {
     /// Get the total number of extended basic blocks created in this function, whether they are
     /// currently inserted in the layout or not.
     ///
-    /// This is intended for use with `EntityMap::with_capacity`.
+    /// This is intended for use with `SecondaryMap::with_capacity`.
     pub fn num_ebbs(&self) -> usize {
         self.ebbs.len()
     }
diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs
index cf9cbdbe29..d15de64c3a 100644
--- a/lib/codegen/src/ir/function.rs
+++ b/lib/codegen/src/ir/function.rs
@@ -4,7 +4,7 @@
 //! instructions.
 
 use binemit::CodeOffset;
-use entity::{EntityMap, PrimaryMap};
+use entity::{PrimaryMap, SecondaryMap};
 use ir;
 use ir::{DataFlowGraph, ExternalName, Layout, Signature};
 use ir::{
@@ -84,10 +84,10 @@ impl Function {
             jump_tables: PrimaryMap::new(),
             dfg: DataFlowGraph::new(),
             layout: Layout::new(),
-            encodings: EntityMap::new(),
-            locations: EntityMap::new(),
-            offsets: EntityMap::new(),
-            srclocs: EntityMap::new(),
+            encodings: SecondaryMap::new(),
+            locations: SecondaryMap::new(),
+            offsets: SecondaryMap::new(),
+            srclocs: SecondaryMap::new(),
         }
     }
 
diff --git a/lib/codegen/src/ir/layout.rs b/lib/codegen/src/ir/layout.rs
index a9dc08d786..a251822847 100644
--- a/lib/codegen/src/ir/layout.rs
+++ b/lib/codegen/src/ir/layout.rs
@@ -3,7 +3,7 @@
 //! The order of extended basic blocks in a function and the order of instructions in an EBB is
 //! determined by the `Layout` data structure defined in this module.
 
-use entity::EntityMap;
+use entity::SecondaryMap;
 use ir::progpoint::{ExpandedProgramPoint, ProgramOrder};
 use ir::{Ebb, Inst};
 use packed_option::PackedOption;
@@ -28,11 +28,11 @@ use timing;
 pub struct Layout {
     /// Linked list nodes for the layout order of EBBs Forms a doubly linked list, terminated in
     /// both ends by `None`.
-    ebbs: EntityMap,
+    ebbs: SecondaryMap,
 
     /// Linked list nodes for the layout order of instructions. Forms a double linked list per EBB,
     /// terminated in both ends by `None`.
-    insts: EntityMap,
+    insts: SecondaryMap,
 
     /// First EBB in the layout order, or `None` when no EBBs have been laid out.
     first_ebb: Option,
@@ -45,8 +45,8 @@ impl Layout {
     /// Create a new empty `Layout`.
     pub fn new() -> Self {
         Self {
-            ebbs: EntityMap::new(),
-            insts: EntityMap::new(),
+            ebbs: SecondaryMap::new(),
+            insts: SecondaryMap::new(),
             first_ebb: None,
             last_ebb: None,
         }
diff --git a/lib/codegen/src/ir/mod.rs b/lib/codegen/src/ir/mod.rs
index 5421911886..a9c035703e 100644
--- a/lib/codegen/src/ir/mod.rs
+++ b/lib/codegen/src/ir/mod.rs
@@ -47,20 +47,20 @@ pub use ir::types::Type;
 pub use ir::valueloc::{ArgumentLoc, ValueLoc};
 
 use binemit;
-use entity::{EntityMap, PrimaryMap};
+use entity::{PrimaryMap, SecondaryMap};
 use isa;
 
 /// Map of value locations.
-pub type ValueLocations = EntityMap;
+pub type ValueLocations = SecondaryMap;
 
 /// Map of jump tables.
 pub type JumpTables = PrimaryMap;
 
 /// Map of instruction encodings.
-pub type InstEncodings = EntityMap;
+pub type InstEncodings = SecondaryMap;
 
 /// Code offsets for EBBs.
-pub type EbbOffsets = EntityMap;
+pub type EbbOffsets = SecondaryMap;
 
 /// Source locations for instructions.
-pub type SourceLocs = EntityMap;
+pub type SourceLocs = SecondaryMap;
diff --git a/lib/codegen/src/loop_analysis.rs b/lib/codegen/src/loop_analysis.rs
index fc1591b0cf..ded382f3f7 100644
--- a/lib/codegen/src/loop_analysis.rs
+++ b/lib/codegen/src/loop_analysis.rs
@@ -2,7 +2,7 @@
 //! and parent in the loop tree.
 
 use dominator_tree::DominatorTree;
-use entity::EntityMap;
+use entity::SecondaryMap;
 use entity::{Keys, PrimaryMap};
 use flowgraph::{BasicBlock, ControlFlowGraph};
 use ir::{Ebb, Function, Layout};
@@ -21,7 +21,7 @@ entity_impl!(Loop, "loop");
 /// its eventual parent in the loop tree and all the EBB belonging to the loop.
 pub struct LoopAnalysis {
     loops: PrimaryMap,
-    ebb_loop_map: EntityMap>,
+    ebb_loop_map: SecondaryMap>,
     valid: bool,
 }
 
@@ -48,7 +48,7 @@ impl LoopAnalysis {
         Self {
             valid: false,
             loops: PrimaryMap::new(),
-            ebb_loop_map: EntityMap::new(),
+            ebb_loop_map: SecondaryMap::new(),
         }
     }
 
diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index db557c865e..217d09f902 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -1,6 +1,6 @@
 //! Utility routines for pretty-printing error messages.
 
-use entity::EntityMap;
+use entity::SecondaryMap;
 use ir;
 use ir::entities::{AnyEntity, Inst, Value};
 use ir::function::Function;
@@ -40,7 +40,7 @@ impl<'a> FuncWriter for PrettyVerifierError<'a> {
         &mut self,
         w: &mut Write,
         func: &Function,
-        aliases: &EntityMap>,
+        aliases: &SecondaryMap>,
         isa: Option<&TargetIsa>,
         inst: Inst,
         indent: usize,
@@ -63,7 +63,7 @@ impl<'a> FuncWriter for PrettyVerifierError<'a> {
 fn pretty_instruction_error(
     w: &mut Write,
     func: &Function,
-    aliases: &EntityMap>,
+    aliases: &SecondaryMap>,
     isa: Option<&TargetIsa>,
     cur_inst: Inst,
     indent: usize,
diff --git a/lib/codegen/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs
index 0e4bcdf5d0..7d2b87153b 100644
--- a/lib/codegen/src/regalloc/virtregs.rs
+++ b/lib/codegen/src/regalloc/virtregs.rs
@@ -15,7 +15,7 @@ use dbg::DisplayList;
 use dominator_tree::DominatorTreePreorder;
 use entity::EntityRef;
 use entity::{EntityList, ListPool};
-use entity::{EntityMap, Keys, PrimaryMap};
+use entity::{Keys, PrimaryMap, SecondaryMap};
 use ir::{Function, Value};
 use packed_option::PackedOption;
 use ref_slice::ref_slice;
@@ -45,10 +45,10 @@ pub struct VirtRegs {
     unused_vregs: Vec,
 
     /// Each value belongs to at most one virtual register.
-    value_vregs: EntityMap>,
+    value_vregs: SecondaryMap>,
 
     /// Table used during the union-find phase while `vregs` is empty.
-    union_find: EntityMap,
+    union_find: SecondaryMap,
 
     /// Values that have been activated in the `union_find` table, but not yet added to any virtual
     /// registers by the `finish_union_find()` function.
@@ -62,8 +62,8 @@ impl VirtRegs {
             pool: ListPool::new(),
             vregs: PrimaryMap::new(),
             unused_vregs: Vec::new(),
-            value_vregs: EntityMap::new(),
-            union_find: EntityMap::new(),
+            value_vregs: SecondaryMap::new(),
+            union_find: SecondaryMap::new(),
             pending_values: Vec::new(),
         }
     }
diff --git a/lib/codegen/src/verifier/flags.rs b/lib/codegen/src/verifier/flags.rs
index d3fda7555f..cf9c924244 100644
--- a/lib/codegen/src/verifier/flags.rs
+++ b/lib/codegen/src/verifier/flags.rs
@@ -1,6 +1,6 @@
 //! Verify CPU flags values.
 
-use entity::{EntityMap, SparseSet};
+use entity::{SecondaryMap, SparseSet};
 use flowgraph::{BasicBlock, ControlFlowGraph};
 use ir;
 use ir::instructions::BranchInfo;
@@ -32,7 +32,7 @@ pub fn verify_flags(
         func,
         cfg,
         encinfo: isa.map(|isa| isa.encoding_info()),
-        livein: EntityMap::new(),
+        livein: SecondaryMap::new(),
     };
     verifier.check(errors)
 }
@@ -43,7 +43,7 @@ struct FlagsVerifier<'a> {
     encinfo: Option,
 
     /// The single live-in flags value (if any) for each EBB.
-    livein: EntityMap>,
+    livein: SecondaryMap>,
 }
 
 impl<'a> FlagsVerifier<'a> {
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index 0c3b7cbae9..743540bf67 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -3,7 +3,7 @@
 //! The `write` module provides the `write_function` function which converts an IR `Function` to an
 //! equivalent textual form. This textual form can be read back by the `cranelift-reader` crate.
 
-use entity::EntityMap;
+use entity::SecondaryMap;
 use ir::entities::AnyEntity;
 use ir::{DataFlowGraph, Ebb, Function, Inst, SigRef, Type, Value, ValueDef};
 use isa::{RegInfo, TargetIsa};
@@ -19,7 +19,7 @@ pub trait FuncWriter {
         &mut self,
         w: &mut Write,
         func: &Function,
-        aliases: &EntityMap>,
+        aliases: &SecondaryMap>,
         isa: Option<&TargetIsa>,
         inst: Inst,
         ident: usize,
@@ -101,7 +101,7 @@ impl FuncWriter for PlainWriter {
         &mut self,
         w: &mut Write,
         func: &Function,
-        aliases: &EntityMap>,
+        aliases: &SecondaryMap>,
         isa: Option<&TargetIsa>,
         inst: Inst,
         indent: usize,
@@ -117,8 +117,8 @@ pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) -
 }
 
 /// Create a reverse-alias map from a value to all aliases having that value as a direct target
-fn alias_map(func: &Function) -> EntityMap> {
-    let mut aliases = EntityMap::<_, Vec<_>>::new();
+fn alias_map(func: &Function) -> SecondaryMap> {
+    let mut aliases = SecondaryMap::<_, Vec<_>>::new();
     for v in func.dfg.values() {
         // VADFS returns the immediate target of an alias
         if let Some(k) = func.dfg.value_alias_dest_for_serialization(v) {
@@ -216,7 +216,7 @@ fn decorate_ebb(
     func_w: &mut FW,
     w: &mut Write,
     func: &Function,
-    aliases: &EntityMap>,
+    aliases: &SecondaryMap>,
     isa: Option<&TargetIsa>,
     ebb: Ebb,
 ) -> fmt::Result {
@@ -279,7 +279,7 @@ fn type_suffix(func: &Function, inst: Inst) -> Option {
 /// Write out any aliases to the given target, including indirect aliases
 fn write_value_aliases(
     w: &mut Write,
-    aliases: &EntityMap>,
+    aliases: &SecondaryMap>,
     target: Value,
     indent: usize,
 ) -> fmt::Result {
@@ -293,7 +293,7 @@ fn write_value_aliases(
 fn write_instruction(
     w: &mut Write,
     func: &Function,
-    aliases: &EntityMap>,
+    aliases: &SecondaryMap>,
     isa: Option<&TargetIsa>,
     inst: Inst,
     indent: usize,
diff --git a/lib/entity/README.md b/lib/entity/README.md
index 2b9ff7e042..f840b142e6 100644
--- a/lib/entity/README.md
+++ b/lib/entity/README.md
@@ -17,11 +17,11 @@ easy to declare new unique types for use as keys. Any attempt to use a key in
 a map it's not intended for is diagnosed with a type error.
 
 Another is that this crate has two core map types, `PrimaryMap` and
-`EntityMap`, which serve complementary purposes. A `PrimaryMap` creates its
-own keys when elements are inserted, while an `EntityMap` reuses the keys
+`SecondaryMap`, which serve complementary purposes. A `PrimaryMap` creates its
+own keys when elements are inserted, while an `SecondaryMap` reuses the keys
 values of a `PrimaryMap`, conceptually storing additional data in the same
-index space. `EntityMap`'s values must implement `Default` and all elements
-in an `EntityMap` initially have the value of `default()`.
+index space. `SecondaryMap`'s values must implement `Default` and all elements
+in an `SecondaryMap` initially have the value of `default()`.
 
 A common way to implement `Default` is to wrap a type in `Option`, however
 this crate also provides the `PackedOption` utility which can use less memory
@@ -30,9 +30,9 @@ in some cases.
 Additional utilities provided by this crate include:
  - `EntityList`, for allocating many small arrays (such as instruction operand
     lists in a compiler code generator).
- - `SparseMap`: an alternative to `EntityMap` which can use less memory
+ - `SparseMap`: an alternative to `SecondaryMap` which can use less memory
    in some situations.
- - `EntitySet`: a specialized form of `EntityMap` using a bitvector to
+ - `EntitySet`: a specialized form of `SecondaryMap` using a bitvector to
    record which entities are members of the set.
 
 [slotmap]: https://crates.io/crates/slotmap
diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs
index a1d07db442..d8e4fcac18 100644
--- a/lib/entity/src/lib.rs
+++ b/lib/entity/src/lib.rs
@@ -15,7 +15,7 @@
 //!
 //! - [`PrimaryMap`](struct.PrimaryMap.html) is used to keep track of a vector of entities,
 //!   assigning a unique entity reference to each.
-//! - [`EntityMap`](struct.EntityMap.html) is used to associate secondary information to an entity.
+//! - [`SecondaryMap`](struct.SecondaryMap.html) is used to associate secondary information to an entity.
 //!   The map is implemented as a simple vector, so it does not keep track of which entities have
 //!   been inserted. Instead, any unknown entities map to the default value.
 //! - [`SparseMap`](struct.SparseMap.html) is used to associate secondary information to a small
@@ -70,7 +70,7 @@ mod std {
 pub extern crate core as __core;
 
 /// A type wrapping a small integer index should implement `EntityRef` so it can be used as the key
-/// of an `EntityMap` or `SparseMap`.
+/// of an `SecondaryMap` or `SparseMap`.
 pub trait EntityRef: Copy + Eq {
     /// Create a new entity reference from a small integer.
     /// This should crash if the requested index is not representable.
@@ -135,7 +135,7 @@ mod sparse;
 pub use self::iter::{Iter, IterMut};
 pub use self::keys::Keys;
 pub use self::list::{EntityList, ListPool};
-pub use self::map::EntityMap;
+pub use self::map::SecondaryMap;
 pub use self::primary::PrimaryMap;
 pub use self::set::EntitySet;
 pub use self::sparse::{SparseMap, SparseMapValue, SparseSet};
diff --git a/lib/entity/src/map.rs b/lib/entity/src/map.rs
index 285e61e4c6..87a478e570 100644
--- a/lib/entity/src/map.rs
+++ b/lib/entity/src/map.rs
@@ -8,14 +8,14 @@ use {EntityRef, Iter, IterMut, Keys};
 
 /// A mapping `K -> V` for densely indexed entity references.
 ///
-/// The `EntityMap` data structure uses the dense index space to implement a map with a vector.
-/// Unlike `PrimaryMap`, an `EntityMap` can't be used to allocate entity references. It is used to
+/// The `SecondaryMap` data structure uses the dense index space to implement a map with a vector.
+/// Unlike `PrimaryMap`, an `SecondaryMap` can't be used to allocate entity references. It is used to
 /// associate secondary information with entities.
 ///
 /// The map does not track if an entry for a key has been inserted or not. Instead it behaves as if
 /// all keys have a default entry from the beginning.
 #[derive(Debug, Clone)]
-pub struct EntityMap
+pub struct SecondaryMap
 where
     K: EntityRef,
     V: Clone,
@@ -25,8 +25,8 @@ where
     unused: PhantomData,
 }
 
-/// Shared `EntityMap` implementation for all value types.
-impl EntityMap
+/// Shared `SecondaryMap` implementation for all value types.
+impl SecondaryMap
 where
     K: EntityRef,
     V: Clone,
@@ -100,10 +100,10 @@ where
     }
 }
 
-/// Immutable indexing into an `EntityMap`.
+/// Immutable indexing into an `SecondaryMap`.
 ///
 /// All keys are permitted. Untouched entries have the default value.
-impl Index for EntityMap
+impl Index for SecondaryMap
 where
     K: EntityRef,
     V: Clone,
@@ -115,10 +115,10 @@ where
     }
 }
 
-/// Mutable indexing into an `EntityMap`.
+/// Mutable indexing into an `SecondaryMap`.
 ///
 /// The map grows as needed to accommodate new keys.
-impl IndexMut for EntityMap
+impl IndexMut for SecondaryMap
 where
     K: EntityRef,
     V: Clone,
@@ -154,7 +154,7 @@ mod tests {
         let r0 = E(0);
         let r1 = E(1);
         let r2 = E(2);
-        let mut m = EntityMap::new();
+        let mut m = SecondaryMap::new();
 
         let v: Vec = m.keys().collect();
         assert_eq!(v, []);
diff --git a/lib/entity/src/set.rs b/lib/entity/src/set.rs
index 7644db49ad..4f7b631c93 100644
--- a/lib/entity/src/set.rs
+++ b/lib/entity/src/set.rs
@@ -7,7 +7,7 @@ use {EntityRef, Keys};
 /// A set of `K` for densely indexed entity references.
 ///
 /// The `EntitySet` data structure uses the dense index space to implement a set with a bitvector.
-/// Like `EntityMap`, an `EntitySet` is used to associate secondary information with entities.
+/// Like `SecondaryMap`, an `EntitySet` is used to associate secondary information with entities.
 #[derive(Debug, Clone)]
 pub struct EntitySet
 where
diff --git a/lib/entity/src/sparse.rs b/lib/entity/src/sparse.rs
index 28cbd80ad8..1a6de3e39d 100644
--- a/lib/entity/src/sparse.rs
+++ b/lib/entity/src/sparse.rs
@@ -11,7 +11,7 @@ use std::mem;
 use std::slice;
 use std::u32;
 use std::vec::Vec;
-use {EntityMap, EntityRef};
+use {EntityRef, SecondaryMap};
 
 /// Trait for extracting keys from values stored in a `SparseMap`.
 ///
@@ -27,16 +27,16 @@ pub trait SparseMapValue {
 ///
 /// A `SparseMap` map provides:
 ///
-/// - Memory usage equivalent to `EntityMap` + `Vec`, so much smaller than
-///   `EntityMap` for sparse mappings of larger `V` types.
-/// - Constant time lookup, slightly slower than `EntityMap`.
+/// - Memory usage equivalent to `SecondaryMap` + `Vec`, so much smaller than
+///   `SecondaryMap` for sparse mappings of larger `V` types.
+/// - Constant time lookup, slightly slower than `SecondaryMap`.
 /// - A very fast, constant time `clear()` operation.
 /// - Fast insert and erase operations.
 /// - Stable iteration that is as fast as a `Vec`.
 ///
-/// # Compared to `EntityMap`
+/// # Compared to `SecondaryMap`
 ///
-/// When should we use a `SparseMap` instead of a secondary `EntityMap`? First of all, `SparseMap`
+/// When should we use a `SparseMap` instead of a secondary `SecondaryMap`? First of all, `SparseMap`
 /// does not provide the functionality of a `PrimaryMap` which can allocate and assign entity
 /// references to objects as they are pushed onto the map. It is only the secondary entity maps
 /// that can be replaced with a `SparseMap`.
@@ -44,10 +44,10 @@ pub trait SparseMapValue {
 /// - A secondary entity map assigns a default mapping to all keys. It doesn't distinguish between
 ///   an unmapped key and one that maps to the default value. `SparseMap` does not require
 ///   `Default` values, and it tracks accurately if a key has been mapped or not.
-/// - Iterating over the contents of an `EntityMap` is linear in the size of the *key space*, while
+/// - Iterating over the contents of an `SecondaryMap` is linear in the size of the *key space*, while
 ///   iterating over a `SparseMap` is linear in the number of elements in the mapping. This is an
 ///   advantage precisely when the mapping is sparse.
-/// - `SparseMap::clear()` is constant time and super-fast. `EntityMap::clear()` is linear in the
+/// - `SparseMap::clear()` is constant time and super-fast. `SecondaryMap::clear()` is linear in the
 ///   size of the key space. (Or, rather the required `resize()` call following the `clear()` is).
 /// - `SparseMap` requires the values to implement `SparseMapValue` which means that they must
 ///   contain their own key.
@@ -56,7 +56,7 @@ where
     K: EntityRef,
     V: SparseMapValue,
 {
-    sparse: EntityMap,
+    sparse: SecondaryMap,
     dense: Vec,
 }
 
@@ -68,7 +68,7 @@ where
     /// Create a new empty mapping.
     pub fn new() -> Self {
         Self {
-            sparse: EntityMap::new(),
+            sparse: SecondaryMap::new(),
             dense: Vec::new(),
         }
     }
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index d32e1892e7..98cdc1c753 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -1,6 +1,6 @@
 //! A frontend for building Cranelift IR from other languages.
 use cranelift_codegen::cursor::{Cursor, FuncCursor};
-use cranelift_codegen::entity::{EntityMap, EntitySet};
+use cranelift_codegen::entity::{EntitySet, SecondaryMap};
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::function::DisplayFunction;
 use cranelift_codegen::ir::{
@@ -24,8 +24,8 @@ use variable::Variable;
 /// use here, `variable::Variable` can be used.
 pub struct FunctionBuilderContext {
     ssa: SSABuilder,
-    ebbs: EntityMap,
-    types: EntityMap,
+    ebbs: SecondaryMap,
+    types: SecondaryMap,
 }
 
 /// Temporary object used to build a single Cranelift IR `Function`.
@@ -79,8 +79,8 @@ impl FunctionBuilderContext {
     pub fn new() -> Self {
         Self {
             ssa: SSABuilder::new(),
-            ebbs: EntityMap::new(),
-            types: EntityMap::new(),
+            ebbs: SecondaryMap::new(),
+            types: SecondaryMap::new(),
         }
     }
 
diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs
index 3474b5104e..2cb6839b76 100644
--- a/lib/frontend/src/ssa.rs
+++ b/lib/frontend/src/ssa.rs
@@ -6,7 +6,7 @@
 //! Lecture Notes in Computer Science, vol 7791. Springer, Berlin, Heidelberg
 
 use cranelift_codegen::cursor::{Cursor, FuncCursor};
-use cranelift_codegen::entity::{EntityMap, EntityRef, PrimaryMap};
+use cranelift_codegen::entity::{EntityRef, PrimaryMap, SecondaryMap};
 use cranelift_codegen::ir::immediates::{Ieee32, Ieee64};
 use cranelift_codegen::ir::instructions::BranchInfo;
 use cranelift_codegen::ir::types::{F32, F64};
@@ -36,13 +36,13 @@ use Variable;
 pub struct SSABuilder {
     // Records for every variable and for every relevant block, the last definition of
     // the variable in the block.
-    // TODO: Consider a sparse representation rather than EntityMap-of-EntityMap.
-    variables: EntityMap>>,
+    // TODO: Consider a sparse representation rather than SecondaryMap-of-SecondaryMap.
+    variables: SecondaryMap>>,
     // Records the position of the basic blocks and the list of values used but not defined in the
     // block.
     blocks: PrimaryMap,
     // Records the basic blocks at the beginning of the `Ebb`s.
-    ebb_headers: EntityMap>,
+    ebb_headers: SecondaryMap>,
 
     // Call and result stacks for use in the `use_var`/`predecessors_lookup` state machine.
     calls: Vec,
@@ -158,9 +158,9 @@ impl SSABuilder {
     /// Allocate a new blank SSA builder struct. Use the API function to interact with the struct.
     pub fn new() -> Self {
         Self {
-            variables: EntityMap::with_default(EntityMap::new()),
+            variables: SecondaryMap::with_default(SecondaryMap::new()),
             blocks: PrimaryMap::new(),
-            ebb_headers: EntityMap::new(),
+            ebb_headers: SecondaryMap::new(),
             calls: Vec::new(),
             results: Vec::new(),
             side_effects: SideEffects::new(),

From 87e43ccb9402d83f5a741eb0f1ac44167d8ec113 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 25 Sep 2018 16:19:11 -0700
Subject: [PATCH 2099/3084] Mention in the README that we avoid using callstack
 recursion.

Callstack recursion has the property that the maximum stack depth can
grow significantly, depending on the input program. Cranelift uses
several recursive algorithms, however it uses explicit heap-based
stacks to do so.
---
 cranelift/README.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index b008ba7296..facc032d6b 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -35,7 +35,8 @@ of code produced by Cranelift is not yet impressive, though we have plans
 to fix that.
 
 The core codegen crates have minimal dependencies, support no\_std mode
-(see below), and do not require any host floating-point support.
+(see below), and do not require any host floating-point support, and
+do not use callstack recursion.
 
 Cranelift does not yet perform mitigations for Spectre or related
 security issues, though it may do so in the future. It does not

From 6fdd58d1433ac4b2fa4260b6650ed8af53e96023 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 27 Sep 2018 09:06:40 -0700
Subject: [PATCH 2100/3084] Tidy up one last mention of `EntityMap`.

---
 cranelift/rustc.md | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/cranelift/rustc.md b/cranelift/rustc.md
index 22d1ce8238..1f2749e323 100644
--- a/cranelift/rustc.md
+++ b/cranelift/rustc.md
@@ -51,13 +51,14 @@ such a thing might look like, including:
  - Build an optimizer IR without the constraints of fast-debug-build
    compilation. Cranelift's base IR is focused on Codegen, so a
    full-strength optimizer would either use an IR layer on top of it
-   (possibly using Cranelift's flexible EntityMap system), or possibly
-   an independent IR that could be translated to/from the base IR.
-   Either way, this overall architecture would keep the optimizer out
-   of the way of the non-optimizing build path, which keeps that path
-   fast and simple, and gives the optimizer more flexibility. If we
-   then want to base the IR on a powerful data structure like the Value
-   State Dependence Graph (VSDG), we can do so with fewer compromises.
+   (possibly using cranelift-entity's flexible `SecondaryMap`s), or
+   possibly an independent IR that could be translated to/from the base
+   IR. Either way, this overall architecture would keep the optimizer
+   out of the way of the non-optimizing build path, which keeps that
+   path fast and simple, and gives the optimizer more flexibility. If we
+   then want to base the IR on a powerful data structure like the
+   Value State Dependence Graph (VSDG), we can do so with fewer
+   compromises.
 
 And, these ideas build on each other. For example, one of the challenges
 for dependence-graph-oriented IRs like the VSDG is getting good enough

From aa5ba71c97457ae2723144e9482c468a020b7ea6 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 27 Sep 2018 11:43:41 -0700
Subject: [PATCH 2101/3084] Clarify the meaning of the symbol_value
 instruction.

---
 lib/codegen/meta-python/base/instructions.py | 6 +++---
 lib/codegen/src/ir/globalvalue.rs            | 9 +++++++--
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/lib/codegen/meta-python/base/instructions.py b/lib/codegen/meta-python/base/instructions.py
index 0b48fb84f2..83459a17bc 100644
--- a/lib/codegen/meta-python/base/instructions.py
+++ b/lib/codegen/meta-python/base/instructions.py
@@ -502,15 +502,15 @@ global_value = Instruction(
         'global_value', r"""
         Compute the value of global GV.
         """,
-        ins=GV, outs=addr)
+        ins=GV, outs=a)
 
 # A specialized form of global_value instructions that only handles
 # symbolic names.
 symbol_value = Instruction(
         'symbol_value', r"""
-        Compute the value of global GV, which is a symbolic address.
+        Compute the value of global GV, which is a symbolic value.
         """,
-        ins=GV, outs=addr)
+        ins=GV, outs=a)
 
 #
 # WebAssembly bounds-checked heap accesses.
diff --git a/lib/codegen/src/ir/globalvalue.rs b/lib/codegen/src/ir/globalvalue.rs
index 97ebdae262..47696fe3d1 100644
--- a/lib/codegen/src/ir/globalvalue.rs
+++ b/lib/codegen/src/ir/globalvalue.rs
@@ -39,8 +39,13 @@ pub enum GlobalValueData {
         global_type: Type,
     },
 
-    /// Value is a symbolic address. Cranelift itself does not interpret this name;
-    /// it's used by embedders to link with other data structures.
+    /// Value is symbolic, meaning it's a name which will be resolved to an
+    /// actual value later (eg. by linking). Cranelift itself does not interpret
+    /// this name; it's used by embedders to link with other data structures.
+    ///
+    /// For now, symbolic values always have pointer type, and represent
+    /// addresses, however in the future they could be used to represent other
+    /// things as well.
     Symbol {
         /// The symbolic name.
         name: ExternalName,

From 44f1ee5518627474757bc0ba58f710ea9c6a2491 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 27 Sep 2018 16:50:01 -0700
Subject: [PATCH 2102/3084] Comment that test-all.sh runs checks on the Python
 files.

---
 cranelift/test-all.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index 3547f16000..83db4528ec 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -4,6 +4,7 @@ set -euo pipefail
 # This is the top-level test script:
 #
 # - Check code formatting.
+# - Perform checks on Python code.
 # - Make a debug build.
 # - Make a release build.
 # - Run unit tests for all Rust crates (including the filetests)

From 75e4ff62a056139fbeb4d5c6320458f5c50ffbb0 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 27 Sep 2018 17:30:16 -0700
Subject: [PATCH 2103/3084] Update to filecheck 0.4.0.

---
 cranelift/Cargo.toml     | 2 +-
 lib/filetests/Cargo.toml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 39e3ef182a..e97cbbc000 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -26,7 +26,7 @@ cranelift-module = { path = "lib/module", version = "0.22.0" }
 cranelift-faerie = { path = "lib/faerie", version = "0.22.0" }
 cranelift-simplejit = { path = "lib/simplejit", version = "0.22.0" }
 cranelift = { path = "lib/umbrella", version = "0.22.0" }
-filecheck = "0.3.0"
+filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
 term = "0.5.1"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index edb7f99ef5..59c4309563 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -12,6 +12,6 @@ publish = false
 cranelift-codegen = { path = "../codegen", version = "0.22.0" }
 cranelift-reader = { path = "../reader", version = "0.22.0" }
 file-per-thread-logger = "0.1.1"
-filecheck = "0.3.0"
+filecheck = "0.4.0"
 num_cpus = "1.8.0"
 log = "0.4.4"

From 52aa1d292a6a08a1102c6e78573ed2febfc23430 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 28 Sep 2018 14:46:26 -0700
Subject: [PATCH 2104/3084] Update no_std support in cranelift-frontend.

---
 lib/frontend/Cargo.toml      |  3 ++-
 lib/frontend/src/frontend.rs |  2 +-
 lib/frontend/src/lib.rs      | 15 ++++++++++++---
 lib/frontend/src/switch.rs   |  2 ++
 4 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 171c158b8e..90f58edc1d 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -13,11 +13,12 @@ readme = "README.md"
 cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
 log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
+hashmap_core = { version = "0.1.9", optional = true }
 
 [features]
 default = ["std"]
 std = ["cranelift-codegen/std"]
-core = ["cranelift-codegen/core"]
+core = ["hashmap_core", "cranelift-codegen/core"]
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 98cdc1c753..930466da5b 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -653,7 +653,6 @@ impl<'a> FunctionBuilder<'a> {
 
 #[cfg(test)]
 mod tests {
-
     use cranelift_codegen::entity::EntityRef;
     use cranelift_codegen::ir::types::*;
     use cranelift_codegen::ir::{AbiParam, ExternalName, Function, InstBuilder, Signature};
@@ -661,6 +660,7 @@ mod tests {
     use cranelift_codegen::settings::CallConv;
     use cranelift_codegen::verifier::verify_function;
     use frontend::{FunctionBuilder, FunctionBuilderContext};
+    use std::string::ToString;
     use Variable;
 
     fn sample_function(lazy_seal: bool) {
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index 1b5eacd939..045705d723 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -176,6 +176,9 @@
 #![cfg_attr(not(feature = "std"), no_std)]
 #![cfg_attr(not(feature = "std"), feature(alloc))]
 
+#[cfg(not(feature = "std"))]
+#[macro_use]
+extern crate alloc;
 extern crate cranelift_codegen;
 #[cfg(test)]
 extern crate target_lexicon;
@@ -191,10 +194,16 @@ mod ssa;
 mod switch;
 mod variable;
 
+/// This replaces `std` in builds with `core`.
 #[cfg(not(feature = "std"))]
 mod std {
-    extern crate alloc;
-
-    pub use self::alloc::vec;
+    pub use alloc::{string, vec};
     pub use core::*;
+    pub mod collections {
+        #[allow(unused_extern_crates)]
+        extern crate hashmap_core;
+
+        pub use self::hashmap_core::map as hash_map;
+        pub use self::hashmap_core::{HashMap, HashSet};
+    }
 }
diff --git a/lib/frontend/src/switch.rs b/lib/frontend/src/switch.rs
index e86c49bddb..829ae24b7a 100644
--- a/lib/frontend/src/switch.rs
+++ b/lib/frontend/src/switch.rs
@@ -2,6 +2,7 @@ use cranelift_codegen::ir::condcodes::IntCC;
 use cranelift_codegen::ir::*;
 use frontend::FunctionBuilder;
 use std::collections::HashMap;
+use std::vec::Vec;
 
 type EntryIndex = u64;
 
@@ -185,6 +186,7 @@ mod tests {
     use super::*;
     use cranelift_codegen::ir::Function;
     use frontend::FunctionBuilderContext;
+    use std::string::ToString;
 
     macro_rules! setup {
         ($default:expr, [$($index:expr,)*]) => {{

From b94cf6c65be5ac9a0da2a00df45032b342f932c5 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 28 Sep 2018 14:50:01 -0700
Subject: [PATCH 2105/3084] Rename test modules to "tests" for consistency.

The majority of the test modules were already named "tests", and that's
what the example in the Rust book uses, so switch to that for all test
modules, for consistency.
---
 lib/bforest/src/lib.rs               | 2 +-
 lib/bforest/src/map.rs               | 2 +-
 lib/bforest/src/node.rs              | 2 +-
 lib/bforest/src/path.rs              | 2 +-
 lib/bforest/src/set.rs               | 2 +-
 lib/codegen/src/dominator_tree.rs    | 2 +-
 lib/codegen/src/ir/libcall.rs        | 2 +-
 lib/codegen/src/loop_analysis.rs     | 3 +--
 lib/codegen/src/regalloc/virtregs.rs | 2 +-
 lib/codegen/src/timing.rs            | 2 +-
 lib/codegen/src/topo_order.rs        | 2 +-
 11 files changed, 11 insertions(+), 12 deletions(-)

diff --git a/lib/bforest/src/lib.rs b/lib/bforest/src/lib.rs
index 7971d481d8..0fce1c11b6 100644
--- a/lib/bforest/src/lib.rs
+++ b/lib/bforest/src/lib.rs
@@ -158,7 +158,7 @@ fn slice_shift(s: &mut [T], n: usize) {
 }
 
 #[cfg(test)]
-mod test {
+mod tests {
     use super::*;
     use entity::EntityRef;
 
diff --git a/lib/bforest/src/map.rs b/lib/bforest/src/map.rs
index 497ca2c112..636cee2df5 100644
--- a/lib/bforest/src/map.rs
+++ b/lib/bforest/src/map.rs
@@ -429,7 +429,7 @@ where
 }
 
 #[cfg(test)]
-mod test {
+mod tests {
     use super::super::NodeData;
     use super::*;
     use std::mem;
diff --git a/lib/bforest/src/node.rs b/lib/bforest/src/node.rs
index de5337053d..768db40174 100644
--- a/lib/bforest/src/node.rs
+++ b/lib/bforest/src/node.rs
@@ -582,7 +582,7 @@ where
 }
 
 #[cfg(test)]
-mod test {
+mod tests {
     use super::*;
     use std::mem;
     use std::string::ToString;
diff --git a/lib/bforest/src/path.rs b/lib/bforest/src/path.rs
index 153c487547..196094c241 100644
--- a/lib/bforest/src/path.rs
+++ b/lib/bforest/src/path.rs
@@ -703,7 +703,7 @@ impl fmt::Display for Path {
 }
 
 #[cfg(test)]
-mod test {
+mod tests {
     use super::super::{Forest, NodeData, NodePool};
     use super::*;
     use std::cmp::Ordering;
diff --git a/lib/bforest/src/set.rs b/lib/bforest/src/set.rs
index 911907ee61..cf4acdf9e3 100644
--- a/lib/bforest/src/set.rs
+++ b/lib/bforest/src/set.rs
@@ -357,7 +357,7 @@ where
 }
 
 #[cfg(test)]
-mod test {
+mod tests {
     use super::super::NodeData;
     use super::*;
     use std::mem;
diff --git a/lib/codegen/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs
index 2198fbf2bf..f8ca5d546c 100644
--- a/lib/codegen/src/dominator_tree.rs
+++ b/lib/codegen/src/dominator_tree.rs
@@ -666,7 +666,7 @@ impl DominatorTreePreorder {
 }
 
 #[cfg(test)]
-mod test {
+mod tests {
     use super::*;
     use cursor::{Cursor, FuncCursor};
     use flowgraph::ControlFlowGraph;
diff --git a/lib/codegen/src/ir/libcall.rs b/lib/codegen/src/ir/libcall.rs
index 6e749c72d6..61766dc8ae 100644
--- a/lib/codegen/src/ir/libcall.rs
+++ b/lib/codegen/src/ir/libcall.rs
@@ -190,7 +190,7 @@ fn make_funcref(libcall: LibCall, func: &mut Function, sig: Signature, isa: &Tar
 }
 
 #[cfg(test)]
-mod test {
+mod tests {
     use super::*;
     use std::string::ToString;
 
diff --git a/lib/codegen/src/loop_analysis.rs b/lib/codegen/src/loop_analysis.rs
index ded382f3f7..21bf57f9aa 100644
--- a/lib/codegen/src/loop_analysis.rs
+++ b/lib/codegen/src/loop_analysis.rs
@@ -230,8 +230,7 @@ impl LoopAnalysis {
 }
 
 #[cfg(test)]
-mod test {
-
+mod tests {
     use cursor::{Cursor, FuncCursor};
     use dominator_tree::DominatorTree;
     use flowgraph::ControlFlowGraph;
diff --git a/lib/codegen/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs
index 7d2b87153b..5d8992a179 100644
--- a/lib/codegen/src/regalloc/virtregs.rs
+++ b/lib/codegen/src/regalloc/virtregs.rs
@@ -396,7 +396,7 @@ impl VirtRegs {
 }
 
 #[cfg(test)]
-mod test {
+mod tests {
     use super::*;
     use entity::EntityRef;
     use ir::Value;
diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs
index b340796578..e5327071c9 100644
--- a/lib/codegen/src/timing.rs
+++ b/lib/codegen/src/timing.rs
@@ -242,7 +242,7 @@ mod details {
 }
 
 #[cfg(test)]
-mod test {
+mod tests {
     use super::*;
     use std::string::ToString;
 
diff --git a/lib/codegen/src/topo_order.rs b/lib/codegen/src/topo_order.rs
index a43d0fcc93..aaedaf0e10 100644
--- a/lib/codegen/src/topo_order.rs
+++ b/lib/codegen/src/topo_order.rs
@@ -88,7 +88,7 @@ impl TopoOrder {
 }
 
 #[cfg(test)]
-mod test {
+mod tests {
     use super::*;
     use cursor::{Cursor, FuncCursor};
     use dominator_tree::DominatorTree;

From e8fc612dc24ddff24ab1a7acc4ce9277a54aa018 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 28 Sep 2018 16:56:04 -0700
Subject: [PATCH 2106/3084] Update to wasmparser 0.19.0.

---
 lib/wasm/Cargo.toml             |  2 +-
 lib/wasm/src/code_translator.rs | 15 +++++++++------
 2 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index bd2c12d46e..beada39d7d 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -10,7 +10,7 @@ readme = "README.md"
 keywords = ["webassembly", "wasm"]
 
 [dependencies]
-wasmparser = { version = "0.17.2", default-features = false }
+wasmparser = { version = "0.19.0", default-features = false }
 cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
 cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
 cranelift-frontend = { path = "../frontend", version = "0.22.0", default-features = false }
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index 92619f1bf9..5a74db5bad 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -255,7 +255,7 @@ pub fn translate_operator(
         Operator::BrTable { table } => {
             let (depths, default) = table.read_table();
             let mut min_depth = default;
-            for depth in &depths {
+            for depth in &*depths {
                 if *depth < min_depth {
                     min_depth = *depth;
                 }
@@ -273,9 +273,9 @@ pub fn translate_operator(
             let mut data = JumpTableData::with_capacity(depths.len());
             if jump_args_count == 0 {
                 // No jump arguments
-                for depth in depths {
+                for depth in &*depths {
                     let ebb = {
-                        let i = state.control_stack.len() - 1 - (depth as usize);
+                        let i = state.control_stack.len() - 1 - (*depth as usize);
                         let frame = &mut state.control_stack[i];
                         frame.set_branched_to_exit();
                         frame.br_destination()
@@ -297,12 +297,12 @@ pub fn translate_operator(
                 let return_count = jump_args_count;
                 let mut dest_ebb_sequence = Vec::new();
                 let mut dest_ebb_map = HashMap::new();
-                for depth in depths {
-                    let branch_ebb = match dest_ebb_map.entry(depth as usize) {
+                for depth in &*depths {
+                    let branch_ebb = match dest_ebb_map.entry(*depth as usize) {
                         hash_map::Entry::Occupied(entry) => *entry.get(),
                         hash_map::Entry::Vacant(entry) => {
                             let ebb = builder.create_ebb();
-                            dest_ebb_sequence.push((depth as usize, ebb));
+                            dest_ebb_sequence.push((*depth as usize, ebb));
                             *entry.insert(ebb)
                         }
                     };
@@ -893,6 +893,9 @@ pub fn translate_operator(
         | Operator::I64AtomicRmw32UCmpxchg { .. } => {
             return Err(WasmError::Unsupported("proposed thread operators"));
         }
+        Operator::RefNull | Operator::RefIsNull { .. } => {
+            return Err(WasmError::Unsupported("proposed reference-type operators"));
+        }
     };
     Ok(())
 }

From 0b3d3ac880018071ce020fd6cc70d5bedba63a4a Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 2 Oct 2018 19:45:23 +0200
Subject: [PATCH 2107/3084] Remove logging levels restrictions (#538)

* Fixes #537: Remove release mode logging levels restrictions;

* Add information about log's logging levels in the README;
---
 cranelift/README.md              | 16 ++++++++++++++++
 lib/codegen/Cargo.toml           |  2 +-
 lib/codegen/meta/src/cdsl/mod.rs |  2 +-
 lib/codegen/src/isa/registers.rs |  2 +-
 lib/frontend/Cargo.toml          |  2 +-
 lib/module/Cargo.toml            |  2 +-
 lib/wasm/Cargo.toml              |  2 +-
 7 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index facc032d6b..c58e48a9cc 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -130,6 +130,22 @@ protection. Just something to think about.
 
 

 
+

+Log configuration
+
+Cranelift uses the `log` crate to log messages at various levels. It doesn't
+specify any maximal logging level, so embedders can choose what it should be;
+however, this can have an impact of Cranelift's code size. You can use `log`
+features to reduce the maximum logging level. For instance if you want to limit
+the level of logging to `warn` messages and above in release mode:
+
+```
+[dependency.log]
+...
+features = ["release_max_level_warn"]
+```
+

+
 

 Building the documentation
 
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 2e38b99274..b1757f95b5 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -18,7 +18,7 @@ failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 target-lexicon = { version = "0.0.3", default-features = false }
-log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
+log = { version = "0.4.4", default-features = false }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
 # machine code. Integration tests that need external dependencies can be
diff --git a/lib/codegen/meta/src/cdsl/mod.rs b/lib/codegen/meta/src/cdsl/mod.rs
index 8f087a7c3d..6a004b2220 100644
--- a/lib/codegen/meta/src/cdsl/mod.rs
+++ b/lib/codegen/meta/src/cdsl/mod.rs
@@ -1,7 +1,7 @@
 //! Cranelift DSL classes.
 //!
 //! This module defines the classes that are used to define Cranelift
-//! instructions and other entitties.
+//! instructions and other entities.
 
 pub mod types;
 
diff --git a/lib/codegen/src/isa/registers.rs b/lib/codegen/src/isa/registers.rs
index 9296b70af5..03376e727d 100644
--- a/lib/codegen/src/isa/registers.rs
+++ b/lib/codegen/src/isa/registers.rs
@@ -151,7 +151,7 @@ pub struct RegClassData {
     /// first register unit in each allocatable register.
     pub mask: RegUnitMask,
 
-    /// The global `RegInfo` instance containing that this register class.
+    /// The global `RegInfo` instance containing this register class.
     pub info: &'static RegInfo,
 }
 
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 90f58edc1d..269f085a57 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -12,7 +12,7 @@ readme = "README.md"
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
-log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
+log = { version = "0.4.4", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 
 [features]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index b9d320a55c..52e58d33e6 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -14,7 +14,7 @@ cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features
 cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
-log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
+log = { version = "0.4.4", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index beada39d7d..2469716d3c 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -18,7 +18,7 @@ hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 target-lexicon = { version = "0.0.3", default-features = false }
-log = { version = "0.4.4", default-features = false, features = ["release_max_level_warn"] }
+log = { version = "0.4.4", default-features = false }
 
 [dev-dependencies]
 wabt = "0.6.0"

From 8b296e4874de57bc11d6941fa42f16c83ca5d191 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 2 Oct 2018 15:05:16 +0200
Subject: [PATCH 2108/3084] Fixes #504: Implement Display/Fail for SetError and
 LookupError;

---
 lib/codegen/src/isa/mod.rs  | 4 +++-
 lib/codegen/src/settings.rs | 5 ++++-
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs
index b95c32f967..33e78c96da 100644
--- a/lib/codegen/src/isa/mod.rs
+++ b/lib/codegen/src/isa/mod.rs
@@ -120,12 +120,14 @@ pub fn lookup(triple: Triple) -> Result {
 }
 
 /// Describes reason for target lookup failure
-#[derive(PartialEq, Eq, Copy, Clone, Debug)]
+#[derive(Fail, PartialEq, Eq, Copy, Clone, Debug)]
 pub enum LookupError {
     /// Support for this target was disabled in the current build.
+    #[fail(display = "Support for this target is disabled")]
     SupportDisabled,
 
     /// Support for this target has not yet been implemented.
+    #[fail(display = "Support for this target has not been implemented yet")]
     Unsupported,
 }
 
diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs
index da0ac133aa..15274d3811 100644
--- a/lib/codegen/src/settings.rs
+++ b/lib/codegen/src/settings.rs
@@ -147,15 +147,18 @@ impl Configurable for Builder {
 }
 
 /// An error produced when changing a setting.
-#[derive(Debug, PartialEq, Eq)]
+#[derive(Fail, Debug, PartialEq, Eq)]
 pub enum SetError {
     /// No setting by this name exists.
+    #[fail(display = "No existing setting with this name")]
     BadName,
 
     /// Type mismatch for setting (e.g., setting an enum setting as a bool).
+    #[fail(display = "Trying to set a setting with the wrong type")]
     BadType,
 
     /// This is not a valid value for this setting.
+    #[fail(display = "Unexpected value for a setting")]
     BadValue,
 }
 

From de1d82b4ba8cf71117bfb36ce913a8d19b172626 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 3 Oct 2018 14:58:07 +0200
Subject: [PATCH 2109/3084] Make SettingsError easier to diagnose;

---
 lib/codegen/src/settings.rs | 42 ++++++++++++++++++++++++-------------
 lib/reader/src/isaspec.rs   | 13 +++++++++---
 2 files changed, 38 insertions(+), 17 deletions(-)

diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs
index 15274d3811..81f1ae2102 100644
--- a/lib/codegen/src/settings.rs
+++ b/lib/codegen/src/settings.rs
@@ -85,7 +85,7 @@ impl Builder {
     /// Look up a descriptor by name.
     fn lookup(&self, name: &str) -> SetResult<(usize, detail::Detail)> {
         match probe(self.template, name, simple_hash(name)) {
-            Err(_) => Err(SetError::BadName),
+            Err(_) => Err(SetError::BadName(name.to_string())),
             Ok(entry) => {
                 let d = &self.template.descriptors[self.template.hash_table[entry] as usize];
                 Ok((d.offset as usize, d.detail))
@@ -98,14 +98,17 @@ fn parse_bool_value(value: &str) -> SetResult {
     match value {
         "true" | "on" | "yes" | "1" => Ok(true),
         "false" | "off" | "no" | "0" => Ok(false),
-        _ => Err(SetError::BadValue),
+        _ => Err(SetError::BadValue("bool".to_string())),
     }
 }
 
 fn parse_enum_value(value: &str, choices: &[&str]) -> SetResult {
     match choices.iter().position(|&tag| tag == value) {
         Some(idx) => Ok(idx as u8),
-        None => Err(SetError::BadValue),
+        None => Err(SetError::BadValue(format!(
+            "any among {}",
+            choices.join(", ")
+        ))),
     }
 }
 
@@ -134,13 +137,15 @@ impl Configurable for Builder {
                 self.set_bit(offset, bit, parse_bool_value(value)?);
             }
             Detail::Num => {
-                self.bytes[offset] = value.parse().map_err(|_| SetError::BadValue)?;
+                self.bytes[offset] = value
+                    .parse()
+                    .map_err(|_| SetError::BadValue("number".to_string()))?;
             }
             Detail::Enum { last, enumerators } => {
                 self.bytes[offset] =
                     parse_enum_value(value, self.template.enums(last, enumerators))?;
             }
-            Detail::Preset => return Err(SetError::BadName),
+            Detail::Preset => return Err(SetError::BadName(name.to_string())),
         }
         Ok(())
     }
@@ -150,16 +155,16 @@ impl Configurable for Builder {
 #[derive(Fail, Debug, PartialEq, Eq)]
 pub enum SetError {
     /// No setting by this name exists.
-    #[fail(display = "No existing setting with this name")]
-    BadName,
+    #[fail(display = "No existing setting named '{}'", _0)]
+    BadName(String),
 
     /// Type mismatch for setting (e.g., setting an enum setting as a bool).
     #[fail(display = "Trying to set a setting with the wrong type")]
     BadType,
 
     /// This is not a valid value for this setting.
-    #[fail(display = "Unexpected value for a setting")]
-    BadValue,
+    #[fail(display = "Unexpected value for a setting, expected {}", _0)]
+    BadValue(String),
 }
 
 /// A result returned when changing a setting.
@@ -385,7 +390,7 @@ mod tests {
     #[test]
     fn modify_bool() {
         let mut b = builder();
-        assert_eq!(b.enable("not_there"), Err(BadName));
+        assert_eq!(b.enable("not_there"), Err(BadName("not_there".to_string())));
         assert_eq!(b.enable("enable_simd"), Ok(()));
         assert_eq!(b.set("enable_simd", "false"), Ok(()));
 
@@ -396,10 +401,19 @@ mod tests {
     #[test]
     fn modify_string() {
         let mut b = builder();
-        assert_eq!(b.set("not_there", "true"), Err(BadName));
-        assert_eq!(b.set("enable_simd", ""), Err(BadValue));
-        assert_eq!(b.set("enable_simd", "best"), Err(BadValue));
-        assert_eq!(b.set("opt_level", "true"), Err(BadValue));
+        assert_eq!(
+            b.set("not_there", "true"),
+            Err(BadName("not_there".to_string()))
+        );
+        assert_eq!(b.set("enable_simd", ""), Err(BadValue("bool".to_string())));
+        assert_eq!(
+            b.set("enable_simd", "best"),
+            Err(BadValue("bool".to_string()))
+        );
+        assert_eq!(
+            b.set("opt_level", "true"),
+            Err(BadValue("any among default, best, fastest".to_string()))
+        );
         assert_eq!(b.set("opt_level", "best"), Ok(()));
         assert_eq!(b.set("enable_simd", "0"), Ok(()));
 
diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs
index 8a0e1f07b8..9132395162 100644
--- a/lib/reader/src/isaspec.rs
+++ b/lib/reader/src/isaspec.rs
@@ -43,14 +43,21 @@ where
         match opt {
             TestOption::Flag(name) => match config.enable(name) {
                 Ok(_) => {}
-                Err(SetError::BadName) => return err!(loc, "unknown flag '{}'", opt),
+                Err(SetError::BadName(name)) => return err!(loc, "unknown flag '{}'", name),
                 Err(_) => return err!(loc, "not a boolean flag: '{}'", opt),
             },
             TestOption::Value(name, value) => match config.set(name, value) {
                 Ok(_) => {}
-                Err(SetError::BadName) => return err!(loc, "unknown setting '{}'", opt),
+                Err(SetError::BadName(name)) => return err!(loc, "unknown setting '{}'", name),
                 Err(SetError::BadType) => return err!(loc, "invalid setting type: '{}'", opt),
-                Err(SetError::BadValue) => return err!(loc, "invalid setting value: '{}'", opt),
+                Err(SetError::BadValue(expected)) => {
+                    return err!(
+                        loc,
+                        "invalid setting value for '{}', expected {}",
+                        opt,
+                        expected
+                    )
+                }
             },
         }
     }

From 79cea5e18b43fe2c4a089fd27e35627f66392925 Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Wed, 3 Oct 2018 11:04:21 -0600
Subject: [PATCH 2110/3084] Implement jump tables (#453)

* Add 'jump_table_entry' and 'indirect_jump' instructions.

* Update CodeSink to keep track of code size. Pretty up clif-util's disassembly output.

* Only disassemble the machine portion of output. Pretty print the read-only data after it.

* Update switch frontend code to use new br_table instruction w/ default.
---
 cranelift/filetests/isa/x86/binary64.clif     |  38 +++++++
 .../filetests/isa/x86/legalize-br-table.clif  |   4 +-
 cranelift/filetests/parser/branch.clif        |  12 +-
 cranelift/filetests/verifier/type_check.clif  |   6 +-
 cranelift/filetests/wasm/control.clif         |   4 +-
 cranelift/src/compile.rs                      |  62 +++++++++-
 lib/codegen/meta-python/base/formats.py       |   5 +-
 lib/codegen/meta-python/base/instructions.py  |  41 ++++++-
 lib/codegen/meta-python/base/legalize.py      |   4 +-
 lib/codegen/meta-python/base/settings.py      |   9 ++
 lib/codegen/meta-python/cdsl/instructions.py  |   3 +
 lib/codegen/meta-python/cdsl/isa.py           |   2 +-
 lib/codegen/meta-python/isa/x86/encodings.py  |  11 ++
 lib/codegen/meta-python/isa/x86/recipes.py    |  45 ++++++++
 lib/codegen/src/binemit/memorysink.rs         |   7 ++
 lib/codegen/src/binemit/mod.rs                |  19 ++++
 lib/codegen/src/binemit/relaxation.rs         |   7 ++
 lib/codegen/src/cfg_printer.rs                |   7 +-
 lib/codegen/src/dominator_tree.rs             |  25 +++--
 lib/codegen/src/flowgraph.rs                  |   5 +-
 lib/codegen/src/ir/function.rs                |   7 +-
 lib/codegen/src/ir/instructions.rs            |  11 +-
 lib/codegen/src/ir/jumptable.rs               |   5 +
 lib/codegen/src/ir/mod.rs                     |   3 +
 lib/codegen/src/isa/x86/binemit.rs            |   8 +-
 lib/codegen/src/legalizer/mod.rs              |  77 ++++++++++++-
 lib/codegen/src/regalloc/coloring.rs          |  12 +-
 lib/codegen/src/settings.rs                   |   3 +-
 lib/codegen/src/verifier/flags.rs             |   7 +-
 lib/codegen/src/verifier/locations.rs         |  14 ++-
 lib/codegen/src/verifier/mod.rs               |  19 +++-
 lib/codegen/src/write.rs                      |  12 +-
 lib/filetests/src/test_binemit.rs             |  23 +++-
 lib/filetests/src/test_compile.rs             |   1 +
 lib/frontend/src/ssa.rs                       | 106 +++++++++++++-----
 lib/frontend/src/switch.rs                    |  12 +-
 lib/reader/src/parser.rs                      |  37 +++++-
 lib/serde/src/serde_clif_json.rs              |  47 +++++++-
 lib/wasm/src/code_translator.rs               |   7 +-
 39 files changed, 627 insertions(+), 100 deletions(-)

diff --git a/cranelift/filetests/isa/x86/binary64.clif b/cranelift/filetests/isa/x86/binary64.clif
index b50e0d33db..9ea203b799 100644
--- a/cranelift/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/isa/x86/binary64.clif
@@ -1402,3 +1402,41 @@ ebb0:
 
     trap user0                                          ; bin: user0 0f 0b
 }
+
+; Tests for i64 jump table instructions.
+function %I64_JT(i64 [%rdi]) {
+    jt0 = jump_table ebb1, ebb2, ebb3
+
+ebb0(v0: i64 [%rdi]):
+    ; Note: The next two lines will need to change whenever instructions are
+    ;        added or removed from this test.
+    [-, %rax]           v1 = jump_table_base.i64 jt0    ; bin: 48 8d 05 00000039
+    [-, %r10]           v2 = jump_table_base.i64 jt0    ; bin: 4c 8d 15 00000032
+
+    [-, %rbx]           v10 = iconst.i64 1
+    [-, %r13]           v11 = iconst.i64 2
+
+    [-, %rax]           v20 = jump_table_entry.i64 v10, v1, 4, jt0      ; bin: 48 63 04 98
+    [-, %rax]           v21 = jump_table_entry.i64 v10, v2, 4, jt0      ; bin: 49 63 04 9a
+    [-, %rax]           v22 = jump_table_entry.i64 v11, v1, 4, jt0      ; bin: 4a 63 04 a8
+    [-, %rax]           v23 = jump_table_entry.i64 v11, v2, 4, jt0      ; bin: 4b 63 04 aa
+
+    [-, %r10]           v30 = jump_table_entry.i64 v10, v1, 4, jt0      ; bin: 4c 63 14 98
+    [-, %r10]           v31 = jump_table_entry.i64 v10, v2, 4, jt0      ; bin: 4d 63 14 9a
+    [-, %r10]           v32 = jump_table_entry.i64 v11, v1, 4, jt0      ; bin: 4e 63 14 a8
+    [-, %r10]           v33 = jump_table_entry.i64 v11, v2, 4, jt0      ; bin: 4f 63 14 aa
+
+    fallthrough ebb10
+
+ebb10:
+    indirect_jump_table_br v10, jt0             ; bin: ff e3
+ebb11:
+    indirect_jump_table_br v11, jt0             ; bin: 41 ff e5
+
+ebb1:
+    fallthrough ebb2
+ebb2:
+    fallthrough ebb3
+ebb3:
+    trap user0
+}
diff --git a/cranelift/filetests/isa/x86/legalize-br-table.clif b/cranelift/filetests/isa/x86/legalize-br-table.clif
index 3047acd9d1..b600de147c 100644
--- a/cranelift/filetests/isa/x86/legalize-br-table.clif
+++ b/cranelift/filetests/isa/x86/legalize-br-table.clif
@@ -9,7 +9,9 @@ function u0:0(i64) system_v {
 ebb0(v0: i64):
     v1 = stack_addr.i64 ss0
     v2 = load.i8 v1
-    br_table v2, jt0
+    br_table v2, ebb2, jt0
+
+ebb2:
     jump ebb1
 
 ebb1:
diff --git a/cranelift/filetests/parser/branch.clif b/cranelift/filetests/parser/branch.clif
index 2969c8f384..ba68fc590d 100644
--- a/cranelift/filetests/parser/branch.clif
+++ b/cranelift/filetests/parser/branch.clif
@@ -85,21 +85,22 @@ function %jumptable(i32) {
     jt2 = jump_table 0, 0, ebb10, ebb40, ebb20, ebb30
 
 ebb10(v3: i32):
-    br_table v3, jt2
-    trap user1
+    br_table v3, ebb50, jt2
+
 ebb20:
     trap user2
 ebb30:
     trap user3
 ebb40:
     trap user4
+ebb50:
+    trap user1    
 }
 ; sameln: function %jumptable(i32) fast {
 ; check:      jt2 = jump_table 0, 0, ebb10, ebb40, ebb20, ebb30
 ; check:      jt200 = jump_table 0
 ; check:  ebb10(v3: i32):
-; nextln:     br_table v3, jt2
-; nextln:     trap user1
+; nextln:     br_table v3, ebb50, jt2
 ; nextln: 
 ; nextln: ebb20:
 ; nextln:     trap user2
@@ -109,4 +110,7 @@ ebb40:
 ; nextln: 
 ; nextln: ebb40:
 ; nextln:     trap user4
+; nextln: 
+; nextln: ebb50:
+; nextln:     trap user1
 ; nextln: }
diff --git a/cranelift/filetests/verifier/type_check.clif b/cranelift/filetests/verifier/type_check.clif
index 9cd9c9f8b0..9fa8fa9cca 100644
--- a/cranelift/filetests/verifier/type_check.clif
+++ b/cranelift/filetests/verifier/type_check.clif
@@ -71,10 +71,12 @@ function %jump_table_args() {
     jt1 = jump_table ebb1
     ebb0:
         v0 = iconst.i32 0
-        br_table v0, jt1 ; error: takes no arguments, but had target ebb1 with 1 arguments
-        return
+        br_table v0, ebb2, jt1 ; error: takes no arguments, but had target ebb1 with 1 arguments
+
     ebb1(v5: i32):
         return
+    ebb2:
+        return
 }
 
 function %jump_args() {
diff --git a/cranelift/filetests/wasm/control.clif b/cranelift/filetests/wasm/control.clif
index a34bbeb935..2e9f9a17ad 100644
--- a/cranelift/filetests/wasm/control.clif
+++ b/cranelift/filetests/wasm/control.clif
@@ -51,7 +51,9 @@ function %br_table(i32) {
 jt0 = jump_table ebb3, ebb1, 0, ebb2
 
 ebb0(v0: i32):
-    br_table v0, jt0
+    br_table v0, ebb4, jt0
+
+ebb4:
     trap oob
 
 ebb1:
diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs
index c49b5d3aef..708763e7b3 100644
--- a/cranelift/src/compile.rs
+++ b/cranelift/src/compile.rs
@@ -97,12 +97,20 @@ fn handle_module(
         context.func = func;
 
         // Compile and encode the result to machine code.
+        let total_size = context
+            .compile(isa)
+            .map_err(|err| pretty_error(&context.func, Some(isa), err))?;
+
         let mut mem = Vec::new();
+        mem.resize(total_size as usize, 0);
+
         let mut relocs = PrintRelocs { flag_print };
         let mut traps = PrintTraps { flag_print };
-        context
-            .compile_and_emit(isa, &mut mem, &mut relocs, &mut traps)
-            .map_err(|err| pretty_error(&context.func, Some(isa), err))?;
+        let mut code_sink: binemit::MemoryCodeSink;
+        unsafe {
+            code_sink = binemit::MemoryCodeSink::new(mem.as_mut_ptr(), &mut relocs, &mut traps);
+        }
+        isa.emit_function_to_memory(&context.func, &mut code_sink);
 
         if flag_print {
             println!("{}", context.func.display(isa));
@@ -121,17 +129,41 @@ fn handle_module(
             }
 
             println!();
-            print_disassembly(isa, &mem)?;
+            print_disassembly(isa, &mem[0..code_sink.code_size as usize])?;
+            print_readonly_data(&mem[code_sink.code_size as usize..total_size as usize]);
         }
     }
 
     Ok(())
 }
 
+fn print_readonly_data(mem: &[u8]) {
+    if mem.len() == 0 {
+        return;
+    }
+
+    println!("\nFollowed by {} bytes of read-only data:", mem.len());
+
+    for (i, byte) in mem.iter().enumerate() {
+        if i % 16 == 0 {
+            if i != 0 {
+                println!();
+            }
+            print!("{:4}: ", i);
+        }
+        if i % 4 == 0 {
+            print!(" ");
+        }
+        print!("{:02x} ", byte);
+    }
+    println!();
+}
+
 cfg_if! {
     if #[cfg(feature = "disas")] {
         use capstone::prelude::*;
         use target_lexicon::Architecture;
+        use std::fmt::Write;
 
         fn get_disassembler(isa: &TargetIsa) -> Result {
             let cs = match isa.triple().architecture {
@@ -168,10 +200,28 @@ cfg_if! {
         fn print_disassembly(isa: &TargetIsa, mem: &[u8]) -> Result<(), String> {
             let mut cs = get_disassembler(isa)?;
 
-            println!("\nDisassembly:");
+            println!("\nDisassembly of {} bytes:", mem.len());
             let insns = cs.disasm_all(&mem, 0x0).unwrap();
             for i in insns.iter() {
-                println!("{}", i);
+                let mut line = String::new();
+
+                write!(&mut line, "{:4x}:\t", i.address()).unwrap();
+
+                let mut bytes_str = String::new();
+                for b in i.bytes() {
+                    write!(&mut bytes_str, "{:02x} ", b).unwrap();
+                }
+                write!(&mut line, "{:21}\t", bytes_str).unwrap();
+
+                if let Some(s) = i.mnemonic() {
+                    write!(&mut line, "{}\t", s).unwrap();
+                }
+
+                if let Some(s) = i.op_str() {
+                    write!(&mut line, "{}", s).unwrap();
+                }
+
+                println!("{}", line);
             }
             Ok(())
         }
diff --git a/lib/codegen/meta-python/base/formats.py b/lib/codegen/meta-python/base/formats.py
index c008306409..b63863eed0 100644
--- a/lib/codegen/meta-python/base/formats.py
+++ b/lib/codegen/meta-python/base/formats.py
@@ -50,7 +50,10 @@ Branch = InstructionFormat(VALUE, ebb, VARIABLE_ARGS)
 BranchInt = InstructionFormat(intcc, VALUE, ebb, VARIABLE_ARGS)
 BranchFloat = InstructionFormat(floatcc, VALUE, ebb, VARIABLE_ARGS)
 BranchIcmp = InstructionFormat(intcc, VALUE, VALUE, ebb, VARIABLE_ARGS)
-BranchTable = InstructionFormat(VALUE, entities.jump_table)
+BranchTable = InstructionFormat(VALUE, ebb, entities.jump_table)
+BranchTableEntry = InstructionFormat(VALUE, VALUE, uimm8, entities.jump_table)
+BranchTableBase = InstructionFormat(entities.jump_table)
+IndirectJump = InstructionFormat(VALUE, entities.jump_table)
 
 Call = InstructionFormat(func_ref, VARIABLE_ARGS)
 CallIndirect = InstructionFormat(sig_ref, VALUE, VARIABLE_ARGS)
diff --git a/lib/codegen/meta-python/base/instructions.py b/lib/codegen/meta-python/base/instructions.py
index 83459a17bc..c22b6b9b77 100644
--- a/lib/codegen/meta-python/base/instructions.py
+++ b/lib/codegen/meta-python/base/instructions.py
@@ -130,6 +130,8 @@ brff = Instruction(
         ins=(Cond, f, EBB, args), is_branch=True)
 
 x = Operand('x', iB, doc='index into jump table')
+Entry = TypeVar('Entry', 'A scalar integer type', ints=True)
+entry = Operand('entry', Entry, doc='entry of jump table')
 JT = Operand('JT', entities.jump_table)
 br_table = Instruction(
         'br_table', r"""
@@ -137,12 +139,47 @@ br_table = Instruction(
 
         Use ``x`` as an unsigned index into the jump table ``JT``. If a jump
         table entry is found, branch to the corresponding EBB. If no entry was
-        found fall through to the next instruction.
+        found or the index is out-of-bounds, branch to the given default EBB.
 
         Note that this branch instruction can't pass arguments to the targeted
         blocks. Split critical edges as needed to work around this.
         """,
-        ins=(x, JT), is_branch=True)
+        ins=(x, EBB, JT), is_branch=True, is_terminator=True)
+
+Size = Operand('Size', uimm8, 'Size in bytes')
+jump_table_entry = Instruction(
+    'jump_table_entry', r"""
+    Get an entry from a jump table.
+
+    Load a serialized ``entry`` from a jump table ``JT`` at a given index
+    ``addr`` with a specific ``Size``. The retrieved entry may need to be
+    decoded after loading, depending upon the jump table type used.
+
+    Currently, the only type supported is entries which are relative to the
+    base of the jump table.
+    """,
+    ins=(x, addr, Size, JT), outs=entry)
+
+jump_table_base = Instruction(
+    'jump_table_base', r"""
+    Get the absolute base address of a jump table.
+
+    This is used for jump tables wherein the entries are stored relative to
+    the base of jump table. In order to use these, generated code should first
+    load an entry using ``jump_table_entry``, then use this instruction to add
+    the relative base back to it.
+    """,
+    ins=JT, outs=addr)
+
+indirect_jump_table_br = Instruction(
+    'indirect_jump_table_br', r"""
+    Branch indirectly via a jump table entry.
+
+    Unconditionally jump via a jump table entry that was previously loaded
+    with the ``jump_table_entry`` instruction.
+    """,
+    ins=(addr, JT),
+    is_branch=True, is_indirect_branch=True, is_terminator=True)
 
 code = Operand('code', trapcode)
 trap = Instruction(
diff --git a/lib/codegen/meta-python/base/legalize.py b/lib/codegen/meta-python/base/legalize.py
index ec82ed9d3f..a1c6883356 100644
--- a/lib/codegen/meta-python/base/legalize.py
+++ b/lib/codegen/meta-python/base/legalize.py
@@ -306,10 +306,10 @@ widen.legalize(
 
 for int_ty in [types.i8, types.i16]:
     widen.legalize(
-        br_table.bind(int_ty)(x, y),
+        br_table.bind(int_ty)(x, y, z),
         Rtl(
             b << uextend.i32(x),
-            br_table(b, y),
+            br_table(b, y, z),
         )
     )
 
diff --git a/lib/codegen/meta-python/base/settings.py b/lib/codegen/meta-python/base/settings.py
index 7e116b1668..e43e455693 100644
--- a/lib/codegen/meta-python/base/settings.py
+++ b/lib/codegen/meta-python/base/settings.py
@@ -166,4 +166,13 @@ probestack_size_log2 = NumSetting(
         """,
         default=12)
 
+#
+# Jump table options.
+#
+jump_tables_enabled = BoolSetting(
+        """
+        Enable the use of jump tables in generated machine code.
+        """,
+        default=True)
+
 group.close(globals())
diff --git a/lib/codegen/meta-python/cdsl/instructions.py b/lib/codegen/meta-python/cdsl/instructions.py
index f972f82bc7..e4c7186333 100644
--- a/lib/codegen/meta-python/cdsl/instructions.py
+++ b/lib/codegen/meta-python/cdsl/instructions.py
@@ -89,6 +89,7 @@ class Instruction(object):
     :param constraints: Tuple of instruction-specific TypeConstraints.
     :param is_terminator: This is a terminator instruction.
     :param is_branch: This is a branch instruction.
+    :param is_indirect_branch: This is an indirect branch instruction.
     :param is_call: This is a call instruction.
     :param is_return: This is a return instruction.
     :param can_trap: This instruction can trap.
@@ -103,6 +104,8 @@ class Instruction(object):
     ATTRIBS = {
             'is_terminator': 'True for instructions that terminate the EBB.',
             'is_branch': 'True for all branch or jump instructions.',
+            'is_indirect_branch':
+            'True for all indirect branch or jump instructions.',
             'is_call': 'Is this a call instruction?',
             'is_return': 'Is this a return instruction?',
             'can_load': 'Can this instruction read from memory?',
diff --git a/lib/codegen/meta-python/cdsl/isa.py b/lib/codegen/meta-python/cdsl/isa.py
index bcff050b1b..5c57c52fbd 100644
--- a/lib/codegen/meta-python/cdsl/isa.py
+++ b/lib/codegen/meta-python/cdsl/isa.py
@@ -469,7 +469,7 @@ class Encoding(object):
                 "Format {} must match recipe: {}".format(
                     self.inst.format, recipe.format))
 
-        if self.inst.is_branch:
+        if self.inst.is_branch and not self.inst.is_indirect_branch:
             assert recipe.branch_range, (
                     'Recipe {} for {} must have a branch_range'
                     .format(recipe, self.inst.name))
diff --git a/lib/codegen/meta-python/isa/x86/encodings.py b/lib/codegen/meta-python/isa/x86/encodings.py
index a5101261c6..5db4e63b9a 100644
--- a/lib/codegen/meta-python/isa/x86/encodings.py
+++ b/lib/codegen/meta-python/isa/x86/encodings.py
@@ -507,6 +507,17 @@ enc_both(base.brz.b1, r.t8jccd_abcd, 0x84)
 enc_both(base.brnz.b1, r.t8jccb_abcd, 0x75)
 enc_both(base.brnz.b1, r.t8jccd_abcd, 0x85)
 
+#
+# Jump tables
+#
+X86_64.enc(base.jump_table_entry.i64.any.any, *r.jt_entry.rex(0x63, w=1))
+X86_32.enc(base.jump_table_entry.i32.any.any, *r.jt_entry(0x8b))
+
+X86_64.enc(base.jump_table_base.i64, *r.jt_base.rex(0x8d, w=1))
+X86_32.enc(base.jump_table_base.i32, *r.jt_base(0x8d))
+
+enc_x86_64(base.indirect_jump_table_br.i64, r.indirect_jmp, 0xff, rrr=4)
+X86_32.enc(base.indirect_jump_table_br.i32, *r.indirect_jmp(0xff, rrr=4))
 #
 # Trap as ud2
 #
diff --git a/lib/codegen/meta-python/isa/x86/recipes.py b/lib/codegen/meta-python/isa/x86/recipes.py
index c2baa515e7..f033cc2090 100644
--- a/lib/codegen/meta-python/isa/x86/recipes.py
+++ b/lib/codegen/meta-python/isa/x86/recipes.py
@@ -14,6 +14,7 @@ from base.formats import IntCompare, IntCompareImm, FloatCompare
 from base.formats import IntCond, FloatCond
 from base.formats import IntSelect, IntCondTrap, FloatCondTrap
 from base.formats import Jump, Branch, BranchInt, BranchFloat
+from base.formats import BranchTableEntry, BranchTableBase, IndirectJump
 from base.formats import Ternary, FuncAddr, UnaryGlobalValue
 from base.formats import RegMove, RegSpill, RegFill, CopySpecial
 from base.formats import LoadComplex, StoreComplex
@@ -276,6 +277,18 @@ def floatccs(iform):
     return Or(*(IsEqual(iform.cond, cc) for cc in supported_floatccs))
 
 
+def valid_scale(iform):
+    # type: (InstructionFormat) -> PredNode
+    """
+    Return an instruction predicate that checks if `iform.imm` is a valid
+    `scale` for a SIB byte.
+    """
+    return Or(IsEqual(iform.imm, 1),
+              IsEqual(iform.imm, 2),
+              IsEqual(iform.imm, 4),
+              IsEqual(iform.imm, 8))
+
+
 # A null unary instruction that takes a GPR register. Can be used for identity
 # copies and no-op conversions.
 null = EncRecipe('null', Unary, size=0, ins=GPR, outs=0, emit='')
@@ -1473,6 +1486,38 @@ brfd = TailRecipe(
         disp4(destination, func, sink);
         ''')
 
+indirect_jmp = TailRecipe(
+        'indirect_jmp', IndirectJump, size=1, ins=GPR, outs=(),
+        clobbers_flags=False,
+        emit='''
+        PUT_OP(bits, rex1(in_reg0), sink);
+        modrm_r_bits(in_reg0, bits, sink);
+        ''')
+
+jt_entry = TailRecipe(
+        'jt_entry', BranchTableEntry, size=2,
+        ins=(GPR_DEREF_SAFE, GPR_ZERO_DEREF_SAFE),
+        outs=(GPR),
+        clobbers_flags=False,
+        instp=valid_scale(BranchTableEntry),
+        emit='''
+        PUT_OP(bits, rex3(in_reg1, out_reg0, in_reg0), sink);
+        modrm_sib(out_reg0, sink);
+        sib(imm.trailing_zeros() as u8, in_reg0, in_reg1, sink);
+        ''')
+
+jt_base = TailRecipe(
+        'jt_base', BranchTableBase, size=5, ins=(), outs=(GPR),
+        clobbers_flags=False,
+        emit='''
+        // No reloc is needed here as the jump table is emitted directly after
+        // the function body.
+        PUT_OP(bits, rex2(0, out_reg0), sink);
+        modrm_riprel(out_reg0, sink);
+
+        jt_disp4(table, func, sink);
+        ''')
+
 #
 # Test flags and set a register.
 #
diff --git a/lib/codegen/src/binemit/memorysink.rs b/lib/codegen/src/binemit/memorysink.rs
index d3b98b63cb..c7ec60de89 100644
--- a/lib/codegen/src/binemit/memorysink.rs
+++ b/lib/codegen/src/binemit/memorysink.rs
@@ -32,6 +32,8 @@ use std::ptr::write_unaligned;
 pub struct MemoryCodeSink<'a> {
     data: *mut u8,
     offset: isize,
+    /// Size of the machine code portion of output
+    pub code_size: isize,
     relocs: &'a mut RelocSink,
     traps: &'a mut TrapSink,
 }
@@ -49,6 +51,7 @@ impl<'a> MemoryCodeSink<'a> {
         MemoryCodeSink {
             data,
             offset: 0,
+            code_size: 0,
             relocs,
             traps,
         }
@@ -131,6 +134,10 @@ impl<'a> CodeSink for MemoryCodeSink<'a> {
         let ofs = self.offset();
         self.traps.trap(ofs, srcloc, code);
     }
+
+    fn begin_rodata(&mut self) {
+        self.code_size = self.offset;
+    }
 }
 
 /// A `TrapSink` implementation that does nothing, which is convenient when
diff --git a/lib/codegen/src/binemit/mod.rs b/lib/codegen/src/binemit/mod.rs
index fd8cff031a..42b16d442b 100644
--- a/lib/codegen/src/binemit/mod.rs
+++ b/lib/codegen/src/binemit/mod.rs
@@ -94,6 +94,9 @@ pub trait CodeSink {
 
     /// Add trap information for the current offset.
     fn trap(&mut self, TrapCode, SourceLoc);
+
+    /// Code output is complete, read-only data may follow.
+    fn begin_rodata(&mut self);
 }
 
 /// Report a bad encoding error.
@@ -123,4 +126,20 @@ where
             emit_inst(func, inst, &mut divert, sink);
         }
     }
+
+    sink.begin_rodata();
+
+    // output jump tables
+    for (jt, jt_data) in func.jump_tables.iter() {
+        let jt_offset = func.jt_offsets[jt];
+        for idx in 0..jt_data.len() {
+            match jt_data.get_entry(idx) {
+                Some(ebb) => {
+                    let rel_offset: i32 = func.offsets[ebb] as i32 - jt_offset as i32;
+                    sink.put4(rel_offset as u32)
+                }
+                None => sink.put4(0),
+            }
+        }
+    }
 }
diff --git a/lib/codegen/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs
index ad13dc21c1..3b45a1c7b3 100644
--- a/lib/codegen/src/binemit/relaxation.rs
+++ b/lib/codegen/src/binemit/relaxation.rs
@@ -95,6 +95,13 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> CodegenResult CFGPrinter<'a> {
                     BranchInfo::SingleDest(dest, _) => {
                         write!(w, " | <{}>{} {}", inst, idata.opcode(), dest)?
                     }
-                    BranchInfo::Table(table) => {
-                        write!(w, " | <{}>{} {}", inst, idata.opcode(), table)?
+                    BranchInfo::Table(table, dest) => {
+                        write!(w, " | <{}>{} {}", inst, idata.opcode(), table)?;
+                        if let Some(dest) = dest {
+                            write!(w, " {}", dest)?
+                        }
                     }
                     BranchInfo::NotABranch => {}
                 }
diff --git a/lib/codegen/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs
index f8ca5d546c..9883e34418 100644
--- a/lib/codegen/src/dominator_tree.rs
+++ b/lib/codegen/src/dominator_tree.rs
@@ -345,18 +345,13 @@ impl DominatorTree {
     fn push_successors(&mut self, func: &Function, ebb: Ebb) {
         for inst in func.layout.ebb_insts(ebb) {
             match func.dfg.analyze_branch(inst) {
-                BranchInfo::SingleDest(succ, _) => {
-                    if self.nodes[succ].rpo_number == 0 {
-                        self.nodes[succ].rpo_number = SEEN;
-                        self.stack.push(succ);
-                    }
-                }
-                BranchInfo::Table(jt) => {
+                BranchInfo::SingleDest(succ, _) => self.push_if_unseen(succ),
+                BranchInfo::Table(jt, dest) => {
                     for (_, succ) in func.jump_tables[jt].entries() {
-                        if self.nodes[succ].rpo_number == 0 {
-                            self.nodes[succ].rpo_number = SEEN;
-                            self.stack.push(succ);
-                        }
+                        self.push_if_unseen(succ);
+                    }
+                    if let Some(dest) = dest {
+                        self.push_if_unseen(dest);
                     }
                 }
                 BranchInfo::NotABranch => {}
@@ -364,6 +359,14 @@ impl DominatorTree {
         }
     }
 
+    /// Push `ebb` onto `self.stack` if it has not already been seen.
+    fn push_if_unseen(&mut self, ebb: Ebb) {
+        if self.nodes[ebb].rpo_number == 0 {
+            self.nodes[ebb].rpo_number = SEEN;
+            self.stack.push(ebb);
+        }
+    }
+
     /// Build a dominator tree from a control flow graph using Keith D. Cooper's
     /// "Simple, Fast Dominator Algorithm."
     fn compute_domtree(&mut self, func: &Function, cfg: &ControlFlowGraph) {
diff --git a/lib/codegen/src/flowgraph.rs b/lib/codegen/src/flowgraph.rs
index 97c3f932f5..48b328cb51 100644
--- a/lib/codegen/src/flowgraph.rs
+++ b/lib/codegen/src/flowgraph.rs
@@ -125,7 +125,10 @@ impl ControlFlowGraph {
                 BranchInfo::SingleDest(dest, _) => {
                     self.add_edge(ebb, inst, dest);
                 }
-                BranchInfo::Table(jt) => {
+                BranchInfo::Table(jt, dest) => {
+                    if let Some(dest) = dest {
+                        self.add_edge(ebb, inst, dest);
+                    }
                     for (_, dest) in func.jump_tables[jt].entries() {
                         self.add_edge(ebb, inst, dest);
                     }
diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs
index d15de64c3a..26e860dbe4 100644
--- a/lib/codegen/src/ir/function.rs
+++ b/lib/codegen/src/ir/function.rs
@@ -11,7 +11,8 @@ use ir::{
     Ebb, ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, JumpTable,
     JumpTableData, SigRef, StackSlot, StackSlotData, Table, TableData,
 };
-use ir::{EbbOffsets, InstEncodings, JumpTables, SourceLocs, StackSlots, ValueLocations};
+use ir::{EbbOffsets, InstEncodings, SourceLocs, StackSlots, ValueLocations};
+use ir::{JumpTableOffsets, JumpTables};
 use isa::{EncInfo, Encoding, Legalize, TargetIsa};
 use settings::CallConv;
 use std::fmt;
@@ -64,6 +65,9 @@ pub struct Function {
     /// in the textual IR format.
     pub offsets: EbbOffsets,
 
+    /// Code offsets of Jump Table headers.
+    pub jt_offsets: JumpTableOffsets,
+
     /// Source locations.
     ///
     /// Track the original source location for each instruction. The source locations are not
@@ -87,6 +91,7 @@ impl Function {
             encodings: SecondaryMap::new(),
             locations: SecondaryMap::new(),
             offsets: SecondaryMap::new(),
+            jt_offsets: SecondaryMap::new(),
             srclocs: SecondaryMap::new(),
         }
     }
diff --git a/lib/codegen/src/ir/instructions.rs b/lib/codegen/src/ir/instructions.rs
index e709aa01c1..84e34b1f39 100644
--- a/lib/codegen/src/ir/instructions.rs
+++ b/lib/codegen/src/ir/instructions.rs
@@ -194,7 +194,10 @@ impl InstructionData {
                 ref args,
                 ..
             } => BranchInfo::SingleDest(destination, &args.as_slice(pool)[2..]),
-            InstructionData::BranchTable { table, .. } => BranchInfo::Table(table),
+            InstructionData::BranchTable {
+                table, destination, ..
+            } => BranchInfo::Table(table, Some(destination)),
+            InstructionData::IndirectJump { table, .. } => BranchInfo::Table(table, None),
             _ => {
                 debug_assert!(!self.opcode().is_branch());
                 BranchInfo::NotABranch
@@ -213,7 +216,7 @@ impl InstructionData {
             | InstructionData::BranchInt { destination, .. }
             | InstructionData::BranchFloat { destination, .. }
             | InstructionData::BranchIcmp { destination, .. } => Some(destination),
-            InstructionData::BranchTable { .. } => None,
+            InstructionData::BranchTable { .. } | InstructionData::IndirectJump { .. } => None,
             _ => {
                 debug_assert!(!self.opcode().is_branch());
                 None
@@ -284,8 +287,8 @@ pub enum BranchInfo<'a> {
     /// This is a branch or jump to a single destination EBB, possibly taking value arguments.
     SingleDest(Ebb, &'a [Value]),
 
-    /// This is a jump table branch which can have many destination EBBs.
-    Table(JumpTable),
+    /// This is a jump table branch which can have many destination EBBs and maybe one default EBB.
+    Table(JumpTable, Option),
 }
 
 /// Information about call instructions.
diff --git a/lib/codegen/src/ir/jumptable.rs b/lib/codegen/src/ir/jumptable.rs
index 17706d5959..aff6503a57 100644
--- a/lib/codegen/src/ir/jumptable.rs
+++ b/lib/codegen/src/ir/jumptable.rs
@@ -45,6 +45,11 @@ impl JumpTableData {
         self.table.len()
     }
 
+    /// Boolean that is false if the table has missing entries.
+    pub fn fully_dense(&self) -> bool {
+        self.holes == 0
+    }
+
     /// Set a table entry.
     ///
     /// The table will grow as needed to fit `idx`.
diff --git a/lib/codegen/src/ir/mod.rs b/lib/codegen/src/ir/mod.rs
index a9c035703e..716bf99b85 100644
--- a/lib/codegen/src/ir/mod.rs
+++ b/lib/codegen/src/ir/mod.rs
@@ -62,5 +62,8 @@ pub type InstEncodings = SecondaryMap;
 /// Code offsets for EBBs.
 pub type EbbOffsets = SecondaryMap;
 
+/// Code offsets for Jump Tables.
+pub type JumpTableOffsets = SecondaryMap;
+
 /// Source locations for instructions.
 pub type SourceLocs = SecondaryMap;
diff --git a/lib/codegen/src/isa/x86/binemit.rs b/lib/codegen/src/isa/x86/binemit.rs
index 6b9c709bf2..9095512eda 100644
--- a/lib/codegen/src/isa/x86/binemit.rs
+++ b/lib/codegen/src/isa/x86/binemit.rs
@@ -3,7 +3,7 @@
 use super::registers::RU;
 use binemit::{bad_encoding, CodeSink, Reloc};
 use ir::condcodes::{CondCode, FloatCC, IntCC};
-use ir::{Ebb, Function, Inst, InstructionData, Opcode, TrapCode};
+use ir::{Ebb, Function, Inst, InstructionData, JumpTable, Opcode, TrapCode};
 use isa::{RegUnit, StackBase, StackBaseMask, StackRef};
 use regalloc::RegDiversions;
 
@@ -249,6 +249,7 @@ fn sib_noindex(base: RegUnit, sink: &mut CS) {
     sink.put1(b);
 }
 
+/// Emit a SIB byte with a scale, base, and index.
 fn sib(scale: u8, index: RegUnit, base: RegUnit, sink: &mut CS) {
     // SIB        SS_III_BBB.
     debug_assert_eq!(scale & !0x03, 0, "Scale out of range");
@@ -332,3 +333,8 @@ fn disp4(destination: Ebb, func: &Function, sink: &mut CS
     let delta = func.offsets[destination].wrapping_sub(sink.offset() + 4);
     sink.put4(delta);
 }
+
+fn jt_disp4(jt: JumpTable, func: &Function, sink: &mut CS) {
+    let delta = func.jt_offsets[jt].wrapping_sub(sink.offset() + 4);
+    sink.put4(delta);
+}
diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs
index a7645fb643..9fb3a59554 100644
--- a/lib/codegen/src/legalizer/mod.rs
+++ b/lib/codegen/src/legalizer/mod.rs
@@ -16,6 +16,7 @@
 use bitset::BitSet;
 use cursor::{Cursor, FuncCursor};
 use flowgraph::ControlFlowGraph;
+use ir::types::I32;
 use ir::{self, InstBuilder, MemFlags};
 use isa::TargetIsa;
 use timing;
@@ -170,6 +171,72 @@ fn expand_cond_trap(
 
 /// Jump tables.
 fn expand_br_table(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    cfg: &mut ControlFlowGraph,
+    isa: &TargetIsa,
+) {
+    if isa.flags().jump_tables_enabled() {
+        expand_br_table_jt(inst, func, cfg, isa);
+    } else {
+        expand_br_table_conds(inst, func, cfg, isa);
+    }
+}
+
+/// Expand br_table to jump table.
+fn expand_br_table_jt(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    cfg: &mut ControlFlowGraph,
+    isa: &TargetIsa,
+) {
+    use ir::condcodes::IntCC;
+
+    let (arg, default_ebb, table) = match func.dfg[inst] {
+        ir::InstructionData::BranchTable {
+            opcode: ir::Opcode::BrTable,
+            arg,
+            destination,
+            table,
+        } => (arg, destination, table),
+        _ => panic!("Expected br_table: {}", func.dfg.display_inst(inst, None)),
+    };
+
+    let table_size = func.jump_tables[table].len();
+    let table_is_fully_dense = func.jump_tables[table].fully_dense();
+    let addr_ty = isa.pointer_type();
+    let entry_ty = I32;
+
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+    pos.use_srcloc(inst);
+
+    // Bounds check
+    let oob = pos
+        .ins()
+        .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, arg, table_size as i64);
+
+    pos.ins().brnz(oob, default_ebb, &[]);
+
+    let base_addr = pos.ins().jump_table_base(addr_ty, table);
+    let entry = pos
+        .ins()
+        .jump_table_entry(addr_ty, arg, base_addr, entry_ty.bytes() as u8, table);
+
+    // If the table isn't fully dense, zero-check the entry.
+    if !table_is_fully_dense {
+        pos.ins().brz(entry, default_ebb, &[]);
+    }
+
+    let addr = pos.ins().iadd(base_addr, entry);
+    pos.ins().indirect_jump_table_br(addr, table);
+
+    let ebb = pos.current_ebb().unwrap();
+    pos.remove_inst();
+    cfg.recompute_ebb(pos.func, ebb);
+}
+
+/// Expand br_table to series of conditionals.
+fn expand_br_table_conds(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
@@ -177,17 +244,17 @@ fn expand_br_table(
 ) {
     use ir::condcodes::IntCC;
 
-    let (arg, table) = match func.dfg[inst] {
+    let (arg, default_ebb, table) = match func.dfg[inst] {
         ir::InstructionData::BranchTable {
             opcode: ir::Opcode::BrTable,
             arg,
+            destination,
             table,
-        } => (arg, table),
+        } => (arg, destination, table),
         _ => panic!("Expected br_table: {}", func.dfg.display_inst(inst, None)),
     };
 
     // This is a poor man's jump table using just a sequence of conditional branches.
-    // TODO: Lower into a jump table load and indirect branch.
     let table_size = func.jump_tables[table].len();
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
@@ -199,7 +266,9 @@ fn expand_br_table(
         }
     }
 
-    // `br_table` falls through when nothing matches.
+    // `br_table` jumps to the default destination if nothing matches
+    pos.ins().jump(default_ebb, &[]);
+
     let ebb = pos.current_ebb().unwrap();
     pos.remove_inst();
     cfg.recompute_ebb(pos.func, ebb);
diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs
index d9df465f5c..0859bec657 100644
--- a/lib/codegen/src/regalloc/coloring.rs
+++ b/lib/codegen/src/regalloc/coloring.rs
@@ -900,11 +900,15 @@ impl<'a> Context<'a> {
                 let lr = &self.liveness[value];
                 lr.is_livein(ebb, ctx)
             }
-            Table(jt) => {
+            Table(jt, ebb) => {
                 let lr = &self.liveness[value];
-                !lr.is_local() && self.cur.func.jump_tables[jt]
-                    .entries()
-                    .any(|(_, ebb)| lr.is_livein(ebb, ctx))
+                !lr.is_local()
+                    && (ebb.map_or(false, |ebb| lr.is_livein(ebb, ctx)) || self
+                        .cur
+                        .func
+                        .jump_tables[jt]
+                        .entries()
+                        .any(|(_, ebb)| lr.is_livein(ebb, ctx)))
             }
         }
     }
diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs
index 81f1ae2102..743bd29268 100644
--- a/lib/codegen/src/settings.rs
+++ b/lib/codegen/src/settings.rs
@@ -380,7 +380,8 @@ mod tests {
              allones_funcaddrs = false\n\
              probestack_enabled = true\n\
              probestack_func_adjusts_sp = false\n\
-             probestack_size_log2 = 12\n"
+             probestack_size_log2 = 12\n\
+             jump_tables_enabled = true\n"
         );
         assert_eq!(f.opt_level(), super::OptLevel::Default);
         assert_eq!(f.enable_simd(), true);
diff --git a/lib/codegen/src/verifier/flags.rs b/lib/codegen/src/verifier/flags.rs
index cf9c924244..39cdb30660 100644
--- a/lib/codegen/src/verifier/flags.rs
+++ b/lib/codegen/src/verifier/flags.rs
@@ -135,7 +135,12 @@ impl<'a> FlagsVerifier<'a> {
                         merge(&mut live_val, val, inst, errors)?;
                     }
                 }
-                BranchInfo::Table(jt) => {
+                BranchInfo::Table(jt, dest) => {
+                    if let Some(dest) = dest {
+                        if let Some(val) = self.livein[dest].expand() {
+                            merge(&mut live_val, val, inst, errors)?;
+                        }
+                    }
                     for (_, dest) in self.func.jump_tables[jt].entries() {
                         if let Some(val) = self.livein[dest].expand() {
                             merge(&mut live_val, val, inst, errors)?;
diff --git a/lib/codegen/src/verifier/locations.rs b/lib/codegen/src/verifier/locations.rs
index 73997d0a92..c1ed8bd1b9 100644
--- a/lib/codegen/src/verifier/locations.rs
+++ b/lib/codegen/src/verifier/locations.rs
@@ -332,9 +332,21 @@ impl<'a> LocationVerifier<'a> {
                     }
                 }
             }
-            Table(jt) => {
+            Table(jt, ebb) => {
                 for d in divert.all() {
                     let lr = &liveness[d.value];
+                    if let Some(ebb) = ebb {
+                        if lr.is_livein(ebb, liveness.context(&self.func.layout)) {
+                            return fatal!(
+                                errors,
+                                inst,
+                                "{} is diverted to {} and live in to {}",
+                                d.value,
+                                d.to.display(&self.reginfo),
+                                ebb
+                            );
+                        }
+                    }
                     for (_, ebb) in self.func.jump_tables[jt].entries() {
                         if lr.is_livein(ebb, liveness.context(&self.func.layout)) {
                             return fatal!(
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 40f8302592..01baa66c11 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -657,7 +657,10 @@ impl<'a> Verifier<'a> {
                 self.verify_ebb(inst, destination, errors)?;
                 self.verify_value_list(inst, args, errors)?;
             }
-            BranchTable { table, .. } => {
+            BranchTable { table, .. }
+            | BranchTableBase { table, .. }
+            | BranchTableEntry { table, .. }
+            | IndirectJump { table, .. } => {
                 self.verify_jump_table(inst, table, errors)?;
             }
             Call {
@@ -1213,7 +1216,19 @@ impl<'a> Verifier<'a> {
                     .map(|&v| self.func.dfg.value_type(v));
                 self.typecheck_variable_args_iterator(inst, iter, errors)?;
             }
-            BranchInfo::Table(table) => {
+            BranchInfo::Table(table, ebb) => {
+                if let Some(ebb) = ebb {
+                    let arg_count = self.func.dfg.num_ebb_params(ebb);
+                    if arg_count != 0 {
+                        return nonfatal!(
+                            errors,
+                            inst,
+                            "takes no arguments, but had target {} with {} arguments",
+                            ebb,
+                            arg_count
+                        );
+                    }
+                }
                 for (_, ebb) in self.func.jump_tables[table].entries() {
                     let arg_count = self.func.dfg.num_ebb_params(ebb);
                     if arg_count != 0 {
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index 743540bf67..24b8499f39 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -443,7 +443,17 @@ pub fn write_operands(
             write!(w, " {} {}, {}, {}", cond, args[0], args[1], destination)?;
             write_ebb_args(w, &args[2..])
         }
-        BranchTable { arg, table, .. } => write!(w, " {}, {}", arg, table),
+        BranchTable {
+            arg,
+            destination,
+            table,
+            ..
+        } => write!(w, " {}, {}, {}", arg, destination, table),
+        BranchTableBase { table, .. } => write!(w, " {}", table),
+        BranchTableEntry {
+            args, imm, table, ..
+        } => write!(w, " {}, {}, {}, {}", args[0], args[1], imm, table),
+        IndirectJump { arg, table, .. } => write!(w, " {}, {}", arg, table),
         Call {
             func_ref, ref args, ..
         } => write!(w, " {}({})", func_ref, DisplayValues(args.as_slice(pool))),
diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs
index ef8e93936a..2f0db746bd 100644
--- a/lib/filetests/src/test_binemit.rs
+++ b/lib/filetests/src/test_binemit.rs
@@ -4,7 +4,7 @@
 //! functions and compares the results to the expected output.
 
 use cranelift_codegen::binemit;
-use cranelift_codegen::binemit::RegDiversions;
+use cranelift_codegen::binemit::{CodeSink, RegDiversions};
 use cranelift_codegen::dbg::DisplayList;
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::entities::AnyEntity;
@@ -30,6 +30,7 @@ pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
 
 /// Code sink that generates text.
 struct TextSink {
+    code_size: binemit::CodeOffset,
     offset: binemit::CodeOffset,
     text: String,
 }
@@ -38,6 +39,7 @@ impl TextSink {
     /// Create a new empty TextSink.
     pub fn new() -> Self {
         Self {
+            code_size: 0,
             offset: 0,
             text: String::new(),
         }
@@ -93,6 +95,10 @@ impl binemit::CodeSink for TextSink {
     fn trap(&mut self, code: ir::TrapCode, _srcloc: ir::SourceLoc) {
         write!(self.text, "{} ", code).unwrap();
     }
+
+    fn begin_rodata(&mut self) {
+        self.code_size = self.offset
+    }
 }
 
 impl SubTest for TestBinEmit {
@@ -281,6 +287,21 @@ impl SubTest for TestBinEmit {
             }
         }
 
+        sink.begin_rodata();
+
+        for (jt, jt_data) in func.jump_tables.iter() {
+            let jt_offset = func.jt_offsets[jt];
+            for idx in 0..jt_data.len() {
+                match jt_data.get_entry(idx) {
+                    Some(ebb) => {
+                        let rel_offset: i32 = func.offsets[ebb] as i32 - jt_offset as i32;
+                        sink.put4(rel_offset as u32)
+                    }
+                    None => sink.put4(0),
+                }
+            }
+        }
+
         if sink.offset != code_size {
             return Err(format!(
                 "Expected code size {}, got {}",
diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs
index 1fb6d0e4c0..4ff112eab9 100644
--- a/lib/filetests/src/test_compile.rs
+++ b/lib/filetests/src/test_compile.rs
@@ -104,4 +104,5 @@ impl binemit::CodeSink for SizeSink {
     }
     fn reloc_jt(&mut self, _reloc: binemit::Reloc, _jt: ir::JumpTable) {}
     fn trap(&mut self, _code: ir::TrapCode, _srcloc: ir::SourceLoc) {}
+    fn begin_rodata(&mut self) {}
 }
diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs
index 2cb6839b76..d13cc0531a 100644
--- a/lib/frontend/src/ssa.rs
+++ b/lib/frontend/src/ssa.rs
@@ -10,7 +10,7 @@ use cranelift_codegen::entity::{EntityRef, PrimaryMap, SecondaryMap};
 use cranelift_codegen::ir::immediates::{Ieee32, Ieee64};
 use cranelift_codegen::ir::instructions::BranchInfo;
 use cranelift_codegen::ir::types::{F32, F64};
-use cranelift_codegen::ir::{Ebb, Function, Inst, InstBuilder, Type, Value};
+use cranelift_codegen::ir::{Ebb, Function, Inst, InstBuilder, InstructionData, Type, Value};
 use cranelift_codegen::packed_option::PackedOption;
 use cranelift_codegen::packed_option::ReservedValue;
 use std::mem;
@@ -647,7 +647,7 @@ impl SSABuilder {
                 func.dfg.append_inst_arg(jump_inst, val);
                 None
             }
-            BranchInfo::Table(jt) => {
+            BranchInfo::Table(jt, default_ebb) => {
                 // In the case of a jump table, the situation is tricky because br_table doesn't
                 // support arguments.
                 // We have to split the critical edge
@@ -656,6 +656,21 @@ impl SSABuilder {
                 let middle_block = self.declare_ebb_header_block(middle_ebb);
                 self.blocks[middle_block].add_predecessor(jump_inst_block, jump_inst);
                 self.mark_ebb_header_block_sealed(middle_block);
+
+                if let Some(default_ebb) = default_ebb {
+                    if dest_ebb == default_ebb {
+                        match func.dfg[jump_inst] {
+                            InstructionData::BranchTable {
+                                destination: ref mut dest,
+                                ..
+                            } => {
+                                *dest = middle_ebb;
+                            }
+                            _ => panic!("should not happen"),
+                        }
+                    }
+                }
+
                 for old_dest in func.jump_tables[jt].as_mut_slice() {
                     if *old_dest == PackedOption::from(dest_ebb) {
                         *old_dest = PackedOption::from(middle_ebb);
@@ -986,20 +1001,31 @@ mod tests {
     #[test]
     fn br_table_with_args() {
         // This tests the on-demand splitting of critical edges for br_table with jump arguments
-        let mut func = Function::new();
-        let mut ssa = SSABuilder::new();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb1 = func.dfg.make_ebb();
+        //
         // Here is the pseudo-program we want to translate:
+        //
+        // function %f {
+        // jt = jump_table ebb2, 0, ebb1
         // ebb0:
-        //    x = 0;
-        //    br_table x ebb1
-        //    x = 1
-        //    jump ebb1
+        //    x = 1;
+        //    br_table x, ebb2, jt
         // ebb1:
+        //    x = 2
+        //    jump ebb2
+        // ebb2:
         //    x = x + 1
         //    return
-        //
+        // }
+
+        let mut func = Function::new();
+        let mut ssa = SSABuilder::new();
+        let mut jump_table = JumpTableData::new();
+        let ebb0 = func.dfg.make_ebb();
+        let ebb1 = func.dfg.make_ebb();
+        let ebb2 = func.dfg.make_ebb();
+
+        // ebb0:
+        //    x = 1;
         let block0 = ssa.declare_ebb_header_block(ebb0);
         ssa.seal_ebb_header_block(ebb0, &mut func);
         let x_var = Variable::new(0);
@@ -1007,42 +1033,60 @@ mod tests {
             let mut cur = FuncCursor::new(&mut func);
             cur.insert_ebb(ebb0);
             cur.insert_ebb(ebb1);
+            cur.insert_ebb(ebb2);
             cur.goto_bottom(ebb0);
             cur.ins().iconst(I32, 1)
         };
         ssa.def_var(x_var, x1, block0);
-        let mut data = JumpTableData::new();
-        data.push_entry(ebb1);
-        data.set_entry(2, ebb1);
-        let jt = func.create_jump_table(data);
+
+        // jt = jump_table ebb2, 0, ebb1
+        jump_table.push_entry(ebb2);
+        jump_table.set_entry(2, ebb1);
+        let jt = func.create_jump_table(jump_table);
+
+        // ebb0:
+        //    ...
+        //    br_table x, ebb2, jt
         ssa.use_var(&mut func, x_var, I32, block0).0;
         let br_table = {
             let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
-            cur.ins().br_table(x1, jt)
+            cur.ins().br_table(x1, ebb2, jt)
         };
-        let block1 = ssa.declare_ebb_body_block(block0);
-        let x3 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
+
+        // ebb1:
+        //    x = 2
+        //    jump ebb2
+        let block1 = ssa.declare_ebb_header_block(ebb1);
+        ssa.seal_ebb_header_block(ebb1, &mut func);
+        let x2 = {
+            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1);
             cur.ins().iconst(I32, 2)
         };
-        ssa.def_var(x_var, x3, block1);
+        ssa.def_var(x_var, x2, block1);
         let jump_inst = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
-            cur.ins().jump(ebb1, &[])
-        };
-        let block2 = ssa.declare_ebb_header_block(ebb1);
-        ssa.declare_ebb_predecessor(ebb1, block1, jump_inst);
-        ssa.declare_ebb_predecessor(ebb1, block0, br_table);
-        ssa.seal_ebb_header_block(ebb1, &mut func);
-        let x4 = ssa.use_var(&mut func, x_var, I32, block2).0;
-        {
             let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1);
-            cur.ins().iadd_imm(x4, 1)
+            cur.ins().jump(ebb2, &[])
         };
+
+        // ebb2:
+        //    x = x + 1
+        //    return
+        let block3 = ssa.declare_ebb_header_block(ebb2);
+        ssa.declare_ebb_predecessor(ebb2, block1, jump_inst);
+        ssa.declare_ebb_predecessor(ebb2, block0, br_table);
+        ssa.seal_ebb_header_block(ebb2, &mut func);
+        let block4 = ssa.declare_ebb_body_block(block3);
+        let x3 = ssa.use_var(&mut func, x_var, I32, block4).0;
+        let x4 = {
+            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb2);
+            cur.ins().iadd_imm(x3, 1)
+        };
+        ssa.def_var(x_var, x4, block4);
         {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb2);
             cur.ins().return_(&[])
         };
+
         let flags = settings::Flags::new(settings::builder());
         match verify_function(&func, &flags) {
             Ok(()) => {}
diff --git a/lib/frontend/src/switch.rs b/lib/frontend/src/switch.rs
index 829ae24b7a..7e8db1183d 100644
--- a/lib/frontend/src/switch.rs
+++ b/lib/frontend/src/switch.rs
@@ -156,8 +156,7 @@ impl Switch {
 
             bx.switch_to_block(jt_ebb);
             let discr = bx.ins().iadd_imm(val, (first_index as i64).wrapping_neg());
-            bx.ins().br_table(discr, jump_table);
-            bx.ins().jump(otherwise, &[]);
+            bx.ins().br_table(discr, otherwise, jump_table);
         }
     }
 
@@ -256,8 +255,7 @@ ebb0:
 
 ebb3:
     v3 = iadd_imm.i32 v1, 0
-    br_table v3, jt0
-    jump ebb0"
+    br_table v3, ebb0, jt0"
         );
     }
 
@@ -308,13 +306,11 @@ ebb8:
 
 ebb11:
     v7 = iadd_imm.i32 v1, 0
-    br_table v7, jt0
-    jump ebb0
+    br_table v7, ebb0, jt0
 
 ebb10:
     v8 = iadd_imm.i32 v1, -10
-    br_table v8, jt1
-    jump ebb0"
+    br_table v8, ebb0, jt1"
         );
     }
 
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 1594d183b6..8809774047 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -2225,9 +2225,44 @@ impl<'a> Parser<'a> {
             InstructionFormat::BranchTable => {
                 let arg = self.match_value("expected SSA value operand")?;
                 self.match_token(Token::Comma, "expected ',' between operands")?;
+                let ebb_num = self.match_ebb("expected branch destination EBB")?;
+                self.match_token(Token::Comma, "expected ',' between operands")?;
                 let table = self.match_jt()?;
                 ctx.check_jt(table, self.loc)?;
-                InstructionData::BranchTable { opcode, arg, table }
+                InstructionData::BranchTable {
+                    opcode,
+                    arg,
+                    destination: ebb_num,
+                    table,
+                }
+            }
+            InstructionFormat::BranchTableBase => {
+                let table = self.match_jt()?;
+                ctx.check_jt(table, self.loc)?;
+                InstructionData::BranchTableBase { opcode, table }
+            }
+            InstructionFormat::BranchTableEntry => {
+                let index = self.match_value("expected SSA value operand")?;
+                self.match_token(Token::Comma, "expected ',' between operands")?;
+                let base = self.match_value("expected SSA value operand")?;
+                self.match_token(Token::Comma, "expected ',' between operands")?;
+                let imm = self.match_uimm8("expected width")?;
+                self.match_token(Token::Comma, "expected ',' between operands")?;
+                let table = self.match_jt()?;
+                ctx.check_jt(table, self.loc)?;
+                InstructionData::BranchTableEntry {
+                    opcode,
+                    args: [index, base],
+                    imm,
+                    table,
+                }
+            }
+            InstructionFormat::IndirectJump => {
+                let arg = self.match_value("expected SSA value operand")?;
+                self.match_token(Token::Comma, "expected ',' between operands")?;
+                let table = self.match_jt()?;
+                ctx.check_jt(table, self.loc)?;
+                InstructionData::IndirectJump { opcode, arg, table }
             }
             InstructionFormat::InsertLane => {
                 let lhs = self.match_value("expected SSA value first operand")?;
diff --git a/lib/serde/src/serde_clif_json.rs b/lib/serde/src/serde_clif_json.rs
index 7ad03df646..536dda6a3b 100644
--- a/lib/serde/src/serde_clif_json.rs
+++ b/lib/serde/src/serde_clif_json.rs
@@ -117,6 +117,22 @@ pub enum SerInstData {
         destination: String,
     },
     BranchTable {
+        opcode: String,
+        arg: String,
+        destination: String,
+        table: String,
+    },
+    BranchTableEntry {
+        opcode: String,
+        args: [String; 2],
+        imm: String,
+        table: String,
+    },
+    BranchTableBase {
+        opcode: String,
+        table: String,
+    },
+    IndirectJump {
         opcode: String,
         arg: String,
         table: String,
@@ -445,7 +461,36 @@ pub fn get_inst_data(inst_index: Inst, func: &Function) -> SerInstData {
                 destination: destination.to_string(),
             }
         }
-        InstructionData::BranchTable { opcode, arg, table } => SerInstData::BranchTable {
+        InstructionData::BranchTable {
+            opcode,
+            arg,
+            destination,
+            table,
+        } => SerInstData::BranchTable {
+            opcode: opcode.to_string(),
+            arg: arg.to_string(),
+            destination: destination.to_string(),
+            table: table.to_string(),
+        },
+        InstructionData::BranchTableBase { opcode, table } => SerInstData::BranchTableBase {
+            opcode: opcode.to_string(),
+            table: table.to_string(),
+        },
+        InstructionData::BranchTableEntry {
+            opcode,
+            args,
+            imm,
+            table,
+        } => {
+            let hold_args = [args[0].to_string(), args[1].to_string()];
+            SerInstData::BranchTableEntry {
+                opcode: opcode.to_string(),
+                args: hold_args,
+                imm: imm.to_string(),
+                table: table.to_string(),
+            }
+        }
+        InstructionData::IndirectJump { opcode, arg, table } => SerInstData::IndirectJump {
             opcode: opcode.to_string(),
             arg: arg.to_string(),
             table: table.to_string(),
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index 5a74db5bad..9b2f62777b 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -283,14 +283,13 @@ pub fn translate_operator(
                     data.push_entry(ebb);
                 }
                 let jt = builder.create_jump_table(data);
-                builder.ins().br_table(val, jt);
                 let ebb = {
                     let i = state.control_stack.len() - 1 - (default as usize);
                     let frame = &mut state.control_stack[i];
                     frame.set_branched_to_exit();
                     frame.br_destination()
                 };
-                builder.ins().jump(ebb, &[]);
+                builder.ins().br_table(val, ebb, jt);
             } else {
                 // Here we have jump arguments, but Cranelift's br_table doesn't support them
                 // We then proceed to split the edges going out of the br_table
@@ -309,14 +308,14 @@ pub fn translate_operator(
                     data.push_entry(branch_ebb);
                 }
                 let jt = builder.create_jump_table(data);
-                builder.ins().br_table(val, jt);
                 let default_ebb = {
                     let i = state.control_stack.len() - 1 - (default as usize);
                     let frame = &mut state.control_stack[i];
                     frame.set_branched_to_exit();
                     frame.br_destination()
                 };
-                builder.ins().jump(default_ebb, state.peekn(return_count));
+                dest_ebb_sequence.push((default as usize, default_ebb));
+                builder.ins().br_table(val, default_ebb, jt);
                 for (depth, dest_ebb) in dest_ebb_sequence {
                     builder.switch_to_block(dest_ebb);
                     builder.seal_block(dest_ebb);

From 17859a87ff6af67a0559c23fae92ba3cc44940b9 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 28 Sep 2018 22:50:31 -0700
Subject: [PATCH 2111/3084] Fix a typo in a comment.

---
 lib/codegen/src/ir/immediates.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/codegen/src/ir/immediates.rs b/lib/codegen/src/ir/immediates.rs
index 3297769ac4..2db7123942 100644
--- a/lib/codegen/src/ir/immediates.rs
+++ b/lib/codegen/src/ir/immediates.rs
@@ -215,7 +215,7 @@ impl Offset32 {
         Offset32(x)
     }
 
-    /// Create a new `Offset32` representing the signed numver `x` if possible.
+    /// Create a new `Offset32` representing the signed number `x` if possible.
     pub fn try_from_i64(x: i64) -> Option {
         let casted = x as i32;
         if casted as i64 == x {

From ddf8fd23b524c7be566c668ecf58696005278b97 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 28 Sep 2018 22:50:46 -0700
Subject: [PATCH 2112/3084] Remove obsolete TODO comments.

---
 lib/wasm/src/code_translator.rs | 2 --
 1 file changed, 2 deletions(-)

diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index 9b2f62777b..3c615568ca 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -417,7 +417,6 @@ pub fn translate_operator(
         /******************************* Load instructions ***********************************
          * Wasm specifies an integer alignment flag but we drop it in Cranelift.
          * The memory base address is provided by the environment.
-         * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not
          ************************************************************************************/
         Operator::I32Load8U {
             memarg: MemoryImmediate { flags: _, offset },
@@ -492,7 +491,6 @@ pub fn translate_operator(
         /****************************** Store instructions ***********************************
          * Wasm specifies an integer alignment flag but we drop it in Cranelift.
          * The memory base address is provided by the environment.
-         * TODO: differentiate between 32 bit and 64 bit architecture, to put the uextend or not
          ************************************************************************************/
         Operator::I32Store {
             memarg: MemoryImmediate { flags: _, offset },

From b2a28d69e60b3cc9c1888bd3f246fc3fc152dcaf Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 3 Oct 2018 23:43:59 +0200
Subject: [PATCH 2113/3084] Add encodings for i8 and i16 copy, spill, fill,
 ireduce.i8.i16 (#534)

* Add encodings for i8 and i16 copy, spill, fill, ireduce.i8.i16

Also adds legalization for srem, irsub_imm, {u,s}extend.i16.i8

Fixes #477 cc #466

* Legalize popcnt, clz and ctz for i8 and i16

* Fix bug in call_memset
---
 .../filetests/isa/x86/ireduce-i16-to-i8.clif  |  8 +++
 cranelift/filetests/isa/x86/isub_imm-i8.clif  | 13 +++++
 .../isa/x86/legalize-clz-ctz-i8.clif          | 25 ++++++++
 .../filetests/isa/x86/legalize-popcnt-i8.clif |  9 +++
 .../filetests/isa/x86/uextend-i8-to-i16.clif  | 14 +++++
 lib/codegen/meta-python/base/legalize.py      | 58 ++++++++++++++++++-
 lib/codegen/meta-python/isa/x86/encodings.py  | 22 ++++---
 lib/frontend/src/frontend.rs                  |  2 +-
 8 files changed, 139 insertions(+), 12 deletions(-)
 create mode 100644 cranelift/filetests/isa/x86/ireduce-i16-to-i8.clif
 create mode 100644 cranelift/filetests/isa/x86/isub_imm-i8.clif
 create mode 100644 cranelift/filetests/isa/x86/legalize-clz-ctz-i8.clif
 create mode 100644 cranelift/filetests/isa/x86/legalize-popcnt-i8.clif
 create mode 100644 cranelift/filetests/isa/x86/uextend-i8-to-i16.clif

diff --git a/cranelift/filetests/isa/x86/ireduce-i16-to-i8.clif b/cranelift/filetests/isa/x86/ireduce-i16-to-i8.clif
new file mode 100644
index 0000000000..0f8303dfc4
--- /dev/null
+++ b/cranelift/filetests/isa/x86/ireduce-i16-to-i8.clif
@@ -0,0 +1,8 @@
+test compile
+target x86_64
+
+function u0:0(i16) -> i8 fast {
+ebb0(v0: i16):
+    v1 = ireduce.i8 v0
+    return v1
+}
diff --git a/cranelift/filetests/isa/x86/isub_imm-i8.clif b/cranelift/filetests/isa/x86/isub_imm-i8.clif
new file mode 100644
index 0000000000..8958b1afa4
--- /dev/null
+++ b/cranelift/filetests/isa/x86/isub_imm-i8.clif
@@ -0,0 +1,13 @@
+test compile
+target x86_64
+
+function u0:0(i8) -> i8 fast {
+ebb0(v0: i8):
+    v1 = iconst.i8 0
+    v2 = isub v1, v0
+    ; check: v4 = uextend.i32 v0
+    ; nextln: v6 = iconst.i32 0
+    ; nextln = isub v6, v4
+    ; nextln = ireduce.i8 v5
+    return v2
+}
diff --git a/cranelift/filetests/isa/x86/legalize-clz-ctz-i8.clif b/cranelift/filetests/isa/x86/legalize-clz-ctz-i8.clif
new file mode 100644
index 0000000000..914bcb0e30
--- /dev/null
+++ b/cranelift/filetests/isa/x86/legalize-clz-ctz-i8.clif
@@ -0,0 +1,25 @@
+test compile
+target x86_64
+
+; regex: V=v\d+
+
+function u0:0(i8) -> i8, i8 fast {
+ebb0(v0: i8):
+    v1 = clz v0
+    ; check: v3 = uextend.i32 v0
+    ; nextln: v6 = iconst.i32 -1
+    ; nextln: v7 = iconst.i32 31
+    ; nextln: v8, v9 = x86_bsr v3
+    ; nextln: v10 = selectif.i32 eq v9, v6, v8
+    ; nextln: v4 = isub v7, v10
+    ; nextln: v5 = iadd_imm v4, -24
+    ; nextln: v1 = ireduce.i8 v5
+    v2 = ctz v0
+    ; nextln: v11 = uextend.i32 v0
+    ; nextln: v12 = bor_imm v11, 256
+    ; nextln: v14 = iconst.i32 32
+    ; nextln: v15, v16 = x86_bsf v12
+    ; nextln: v13 = selectif.i32 eq v16, v14, v15
+    ; nextln: v2 = ireduce.i8 v13
+    return v1, v2
+}
diff --git a/cranelift/filetests/isa/x86/legalize-popcnt-i8.clif b/cranelift/filetests/isa/x86/legalize-popcnt-i8.clif
new file mode 100644
index 0000000000..e761a2c7ca
--- /dev/null
+++ b/cranelift/filetests/isa/x86/legalize-popcnt-i8.clif
@@ -0,0 +1,9 @@
+test compile
+target x86_64
+
+function u0:0(i8) -> i8 fast {
+ebb0(v0: i8):
+    v1 = popcnt v0
+    ; check-not: sextend.i32 v0
+    return v1
+}
diff --git a/cranelift/filetests/isa/x86/uextend-i8-to-i16.clif b/cranelift/filetests/isa/x86/uextend-i8-to-i16.clif
new file mode 100644
index 0000000000..d92da90343
--- /dev/null
+++ b/cranelift/filetests/isa/x86/uextend-i8-to-i16.clif
@@ -0,0 +1,14 @@
+test compile
+target x86_64
+
+function u0:0(i8) -> i16 fast {
+ebb0(v0: i8):
+    v1 = uextend.i16 v0
+    return v1
+}
+
+function u0:1(i8) -> i16 fast {
+ebb0(v0: i8):
+    v1 = sextend.i16 v0
+    return v1
+}
diff --git a/lib/codegen/meta-python/base/legalize.py b/lib/codegen/meta-python/base/legalize.py
index a1c6883356..0625bb3963 100644
--- a/lib/codegen/meta-python/base/legalize.py
+++ b/lib/codegen/meta-python/base/legalize.py
@@ -248,12 +248,12 @@ def widen_imm(signed, op):
                 ))
 
 
+# int ops
 for binop in [iadd, isub, imul, udiv, urem]:
     widen_two_arg(False, binop)
 
-widen_two_arg(True, sdiv)
-
-widen_one_arg(False, bnot)
+for binop in [sdiv, srem]:
+    widen_two_arg(True, binop)
 
 for binop in [iadd_imm, imul_imm, udiv_imm, urem_imm]:
     widen_imm(False, binop)
@@ -261,13 +261,50 @@ for binop in [iadd_imm, imul_imm, udiv_imm, urem_imm]:
 for binop in [sdiv_imm, srem_imm]:
     widen_imm(True, binop)
 
+widen_imm(False, irsub_imm)
+
 # bit ops
+widen_one_arg(False, bnot)
+
 for binop in [band, bor, bxor, band_not, bor_not, bxor_not]:
     widen_two_arg(False, binop)
 
 for binop in [band_imm, bor_imm, bxor_imm]:
     widen_imm(False, binop)
 
+widen_one_arg(False, insts.popcnt)
+
+for (int_ty, num) in [(types.i8, 24), (types.i16, 16)]:
+    widen.legalize(
+        a << insts.clz.bind(int_ty)(b),
+        Rtl(
+            c << uextend.i32(b),
+            d << insts.clz.i32(c),
+            e << iadd_imm(d, imm64(-num)),
+            a << ireduce.bind(int_ty)(e)
+        ))
+
+    widen.legalize(
+        a << insts.cls.bind(int_ty)(b),
+        Rtl(
+            c << sextend.i32(b),
+            d << insts.cls.i32(c),
+            e << iadd_imm(d, imm64(-num)),
+            a << ireduce.bind(int_ty)(e)
+        ))
+
+for (int_ty, num) in [(types.i8, 1 << 8), (types.i16, 1 << 16)]:
+    widen.legalize(
+        a << insts.ctz.bind(int_ty)(b),
+        Rtl(
+            c << uextend.i32(b),
+            # When `b` is zero, returns the size of x in bits.
+            d << bor_imm(c, imm64(num)),
+            e << insts.ctz.i32(d),
+            a << ireduce.bind(int_ty)(e)
+        ))
+
+# iconst
 for int_ty in [types.i8, types.i16]:
     widen.legalize(
         a << iconst.bind(int_ty)(b),
@@ -276,6 +313,21 @@ for int_ty in [types.i8, types.i16]:
             a << ireduce.bind(int_ty)(c)
         ))
 
+widen.legalize(
+    a << uextend.i16.i8(b),
+    Rtl(
+        c << uextend.i32(b),
+        a << ireduce(c)
+    ))
+
+widen.legalize(
+    a << sextend.i16.i8(b),
+    Rtl(
+        c << sextend.i32(b),
+        a << ireduce(c)
+    ))
+
+
 widen.legalize(
     store.i8(flags, a, ptr, offset),
     Rtl(
diff --git a/lib/codegen/meta-python/isa/x86/encodings.py b/lib/codegen/meta-python/isa/x86/encodings.py
index 5db4e63b9a..d07e003c3c 100644
--- a/lib/codegen/meta-python/isa/x86/encodings.py
+++ b/lib/codegen/meta-python/isa/x86/encodings.py
@@ -173,7 +173,8 @@ enc_i32_i64(x86.smulx, r.mulx, 0xf7, rrr=5)
 enc_i32_i64(x86.umulx, r.mulx, 0xf7, rrr=4)
 
 enc_i32_i64(base.copy, r.umr, 0x89)
-enc_both(base.copy.b1, r.umr, 0x89)
+for ty in [types.b1, types.i8, types.i16]:
+    enc_both(base.copy.bind(ty), r.umr, 0x89)
 
 # For x86-64, only define REX forms for now, since we can't describe the
 # special regunit immediate operands with the current constraint language.
@@ -301,11 +302,12 @@ for recipe in [r.st_abcd, r.stDisp8_abcd, r.stDisp32_abcd]:
 enc_i32_i64(base.spill, r.spillSib32, 0x89)
 enc_i32_i64(base.regspill, r.regspill32, 0x89)
 
-# Use a 32-bit write for spilling `b1` to avoid constraining the permitted
-# registers.
+# Use a 32-bit write for spilling `b1`, `i8` and `i16` to avoid
+# constraining the permitted registers.
 # See MIN_SPILL_SLOT_SIZE which makes this safe.
-enc_both(base.spill.b1, r.spillSib32, 0x89)
-enc_both(base.regspill.b1, r.regspill32, 0x89)
+for ty in [types.b1, types.i8, types.i16]:
+    enc_both(base.spill.bind(ty), r.spillSib32, 0x89)
+    enc_both(base.regspill.bind(ty), r.regspill32, 0x89)
 
 for recipe in [r.ld, r.ldDisp8, r.ldDisp32]:
     enc_i32_i64_ld_st(base.load, True, recipe, 0x8b)
@@ -319,9 +321,10 @@ for recipe in [r.ld, r.ldDisp8, r.ldDisp32]:
 enc_i32_i64(base.fill, r.fillSib32, 0x8b)
 enc_i32_i64(base.regfill, r.regfill32, 0x8b)
 
-# Load 32 bits from `b1` spill slots. See `spill.b1` above.
-enc_both(base.fill.b1, r.fillSib32, 0x8b)
-enc_both(base.regfill.b1, r.regfill32, 0x8b)
+# Load 32 bits from `b1`, `i8` and `i16` spill slots. See `spill.b1` above.
+for ty in [types.b1, types.i8, types.i16]:
+    enc_both(base.fill.bind(ty), r.fillSib32, 0x8b)
+    enc_both(base.regfill.bind(ty), r.regfill32, 0x8b)
 
 # Push and Pop
 X86_32.enc(x86.push.i32, *r.pushq(0x50))
@@ -578,8 +581,11 @@ X86_64.enc(base.bint.i32.b1, *r.urm_noflags_abcd(0x0f, 0xb6))
 # Numerical conversions.
 
 # Reducing an integer is a no-op.
+X86_32.enc(base.ireduce.i8.i16, r.null, 0)
 X86_32.enc(base.ireduce.i8.i32, r.null, 0)
 X86_32.enc(base.ireduce.i16.i32, r.null, 0)
+
+X86_64.enc(base.ireduce.i8.i16, r.null, 0)
 X86_64.enc(base.ireduce.i8.i32, r.null, 0)
 X86_64.enc(base.ireduce.i16.i32, r.null, 0)
 X86_64.enc(base.ireduce.i8.i64, r.null, 0)
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 930466da5b..c2befee997 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -591,7 +591,7 @@ impl<'a> FunctionBuilder<'a> {
             colocated: false,
         });
 
-        self.ins().uextend(types::I32, ch);
+        let ch = self.ins().uextend(types::I32, ch);
         self.ins().call(libc_memset, &[buffer, ch, len]);
     }
 

From 2eb9ae45aa76b7771c847cee2cdb4e846520d8ef Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 4 Oct 2018 09:19:09 -0700
Subject: [PATCH 2114/3084] Tidy up some comments.

---
 lib/codegen/meta-python/isa/x86/recipes.py | 4 ++--
 lib/codegen/src/isa/x86/binemit.rs         | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/lib/codegen/meta-python/isa/x86/recipes.py b/lib/codegen/meta-python/isa/x86/recipes.py
index f033cc2090..8a204728d9 100644
--- a/lib/codegen/meta-python/isa/x86/recipes.py
+++ b/lib/codegen/meta-python/isa/x86/recipes.py
@@ -1510,11 +1510,11 @@ jt_base = TailRecipe(
         'jt_base', BranchTableBase, size=5, ins=(), outs=(GPR),
         clobbers_flags=False,
         emit='''
-        // No reloc is needed here as the jump table is emitted directly after
-        // the function body.
         PUT_OP(bits, rex2(0, out_reg0), sink);
         modrm_riprel(out_reg0, sink);
 
+        // No reloc is needed here as the jump table is emitted directly after
+        // the function body.
         jt_disp4(table, func, sink);
         ''')
 
diff --git a/lib/codegen/src/isa/x86/binemit.rs b/lib/codegen/src/isa/x86/binemit.rs
index 9095512eda..2389595b2e 100644
--- a/lib/codegen/src/isa/x86/binemit.rs
+++ b/lib/codegen/src/isa/x86/binemit.rs
@@ -328,12 +328,13 @@ fn disp1(destination: Ebb, func: &Function, sink: &mut CS
     sink.put1(delta as u8);
 }
 
-/// Emit a single-byte branch displacement to `destination`.
+/// Emit a four-byte branch displacement to `destination`.
 fn disp4(destination: Ebb, func: &Function, sink: &mut CS) {
     let delta = func.offsets[destination].wrapping_sub(sink.offset() + 4);
     sink.put4(delta);
 }
 
+/// Emit a four-byte displacement to jump table `jt`.
 fn jt_disp4(jt: JumpTable, func: &Function, sink: &mut CS) {
     let delta = func.jt_offsets[jt].wrapping_sub(sink.offset() + 4);
     sink.put4(delta);

From bed073fac0e5f75fe877e27c8499b9cede47e11f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 3 Oct 2018 22:42:54 -0700
Subject: [PATCH 2115/3084] Fix copy+pastos in debug messages.

---
 cranelift/src/compile.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs
index 708763e7b3..5a93daa5a2 100644
--- a/cranelift/src/compile.rs
+++ b/cranelift/src/compile.rs
@@ -34,13 +34,13 @@ impl binemit::RelocSink for PrintRelocs {
         addend: binemit::Addend,
     ) {
         if self.flag_print {
-            println!("reloc_ebb: {} {} {} at {}", r, name, addend, where_);
+            println!("reloc_external: {} {} {} at {}", r, name, addend, where_);
         }
     }
 
     fn reloc_jt(&mut self, where_: binemit::CodeOffset, r: binemit::Reloc, jt: ir::JumpTable) {
         if self.flag_print {
-            println!("reloc_ebb: {} {} at {}", r, jt, where_);
+            println!("reloc_jt: {} {} at {}", r, jt, where_);
         }
     }
 }

From dc9221a70c3b9d1573005dacbdae5d097425c5f7 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 28 Sep 2018 23:22:01 -0700
Subject: [PATCH 2116/3084] Fix unused variable warnings.

---
 lib/simplejit/tests/basic.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/simplejit/tests/basic.rs b/lib/simplejit/tests/basic.rs
index 7363fee4e9..3c79e55492 100644
--- a/lib/simplejit/tests/basic.rs
+++ b/lib/simplejit/tests/basic.rs
@@ -60,7 +60,7 @@ fn define_simple_function(module: &mut Module) -> FuncId {
 fn double_finalize() {
     let mut module: Module = Module::new(SimpleJITBuilder::new());
 
-    let func_id = define_simple_function(&mut module);
+    define_simple_function(&mut module);
     module.finalize_definitions();
 
     // Calling `finalize_definitions` a second time without any new definitions
@@ -73,7 +73,7 @@ fn double_finalize() {
 fn panic_on_define_after_finalize() {
     let mut module: Module = Module::new(SimpleJITBuilder::new());
 
-    let func_id = define_simple_function(&mut module);
+    define_simple_function(&mut module);
     module.finalize_definitions();
     define_simple_function(&mut module);
 }

From 1098eafb45427f247e4d652cd02196b7b71e9a40 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 4 Oct 2018 10:35:31 -0700
Subject: [PATCH 2117/3084] Remove the concept of non-dense jump tables.

WebAssembly doesn't have non-dense jump tables, and higher-level users
are better served by the facilities in lib/frontend/src/switch.rs for
working with non-dense switches.

This eliminates the concept of "absent" jump table entries, which
were represented as "0" in the text format.

Also, jump table contents are now enclosed in `[` and `]`, so that
we can unambiguously display empty jump tables. Previously, empty jump
tables were displayed as if they had a single absent entry.
---
 cranelift/docs/ir.rst                         |   5 +-
 cranelift/filetests/isa/x86/binary64.clif     |   2 +-
 .../filetests/isa/x86/legalize-br-table.clif  |   2 +-
 cranelift/filetests/parser/branch.clif        |   8 +-
 cranelift/filetests/verifier/type_check.clif  |   2 +-
 cranelift/filetests/wasm/control.clif         |   2 +-
 lib/codegen/src/binemit/mod.rs                |  11 +-
 lib/codegen/src/dominator_tree.rs             |   4 +-
 lib/codegen/src/flowgraph.rs                  |   4 +-
 lib/codegen/src/ir/function.rs                |   5 -
 lib/codegen/src/ir/jumptable.rs               | 140 +++++-------------
 lib/codegen/src/legalizer/mod.rs              |  13 +-
 lib/codegen/src/regalloc/coloring.rs          |   4 +-
 lib/codegen/src/verifier/flags.rs             |   4 +-
 lib/codegen/src/verifier/locations.rs         |   4 +-
 lib/codegen/src/verifier/mod.rs               |   4 +-
 lib/filetests/src/test_binemit.rs             |  11 +-
 lib/frontend/src/frontend.rs                  |  12 +-
 lib/frontend/src/ssa.rs                       |  10 +-
 lib/frontend/src/switch.rs                    |   6 +-
 lib/reader/src/parser.rs                      |  66 ++++-----
 lib/reader/src/sourcemap.rs                   |   2 +-
 22 files changed, 113 insertions(+), 208 deletions(-)

diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index 8bd17cacbb..c452dfb6bc 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -360,13 +360,12 @@ instruction in the EBB.
 .. autoinst:: brff
 .. autoinst:: br_table
 
-.. inst:: JT = jump_table EBB0, EBB1, ..., EBBn
+.. inst:: JT = jump_table [EBB0, EBB1, ..., EBBn]
 
     Declare a jump table in the :term:`function preamble`.
 
     This declares a jump table for use by the :inst:`br_table` indirect branch
-    instruction. Entries in the table are either EBB names, or ``0`` which
-    indicates an absent entry.
+    instruction. Entries in the table are EBB names.
 
     The EBBs listed must belong to the current function, and they can't have
     any arguments.
diff --git a/cranelift/filetests/isa/x86/binary64.clif b/cranelift/filetests/isa/x86/binary64.clif
index 9ea203b799..dca189d406 100644
--- a/cranelift/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/isa/x86/binary64.clif
@@ -1405,7 +1405,7 @@ ebb0:
 
 ; Tests for i64 jump table instructions.
 function %I64_JT(i64 [%rdi]) {
-    jt0 = jump_table ebb1, ebb2, ebb3
+    jt0 = jump_table [ebb1, ebb2, ebb3]
 
 ebb0(v0: i64 [%rdi]):
     ; Note: The next two lines will need to change whenever instructions are
diff --git a/cranelift/filetests/isa/x86/legalize-br-table.clif b/cranelift/filetests/isa/x86/legalize-br-table.clif
index b600de147c..959d3028b2 100644
--- a/cranelift/filetests/isa/x86/legalize-br-table.clif
+++ b/cranelift/filetests/isa/x86/legalize-br-table.clif
@@ -4,7 +4,7 @@ target x86_64
 
 function u0:0(i64) system_v {
     ss0 = explicit_slot 1
-    jt0 = jump_table ebb1
+    jt0 = jump_table [ebb1]
 
 ebb0(v0: i64):
     v1 = stack_addr.i64 ss0
diff --git a/cranelift/filetests/parser/branch.clif b/cranelift/filetests/parser/branch.clif
index ba68fc590d..4404feba14 100644
--- a/cranelift/filetests/parser/branch.clif
+++ b/cranelift/filetests/parser/branch.clif
@@ -81,8 +81,8 @@ ebb1(v92: i32, v93: f32):
 ; nextln: }
 
 function %jumptable(i32) {
-    jt200 = jump_table 0, 0
-    jt2 = jump_table 0, 0, ebb10, ebb40, ebb20, ebb30
+    jt200 = jump_table []
+    jt2 = jump_table [ebb10, ebb40, ebb20, ebb30]
 
 ebb10(v3: i32):
     br_table v3, ebb50, jt2
@@ -97,8 +97,8 @@ ebb50:
     trap user1    
 }
 ; sameln: function %jumptable(i32) fast {
-; check:      jt2 = jump_table 0, 0, ebb10, ebb40, ebb20, ebb30
-; check:      jt200 = jump_table 0
+; check:      jt2 = jump_table [ebb10, ebb40, ebb20, ebb30]
+; check:      jt200 = jump_table []
 ; check:  ebb10(v3: i32):
 ; nextln:     br_table v3, ebb50, jt2
 ; nextln: 
diff --git a/cranelift/filetests/verifier/type_check.clif b/cranelift/filetests/verifier/type_check.clif
index 9fa8fa9cca..ce61b81a10 100644
--- a/cranelift/filetests/verifier/type_check.clif
+++ b/cranelift/filetests/verifier/type_check.clif
@@ -68,7 +68,7 @@ function %fn_call_incorrect_arg_type(i64) {
 ; TODO: Should we instead just verify that jump tables contain no EBBs that take arguments? This
 ; error doesn't occur if no instruction uses the jump table.
 function %jump_table_args() {
-    jt1 = jump_table ebb1
+    jt1 = jump_table [ebb1]
     ebb0:
         v0 = iconst.i32 0
         br_table v0, ebb2, jt1 ; error: takes no arguments, but had target ebb1 with 1 arguments
diff --git a/cranelift/filetests/wasm/control.clif b/cranelift/filetests/wasm/control.clif
index 2e9f9a17ad..66c82df937 100644
--- a/cranelift/filetests/wasm/control.clif
+++ b/cranelift/filetests/wasm/control.clif
@@ -48,7 +48,7 @@ ebb0:
 }
 
 function %br_table(i32) {
-jt0 = jump_table ebb3, ebb1, 0, ebb2
+jt0 = jump_table [ebb3, ebb1, ebb2]
 
 ebb0(v0: i32):
     br_table v0, ebb4, jt0
diff --git a/lib/codegen/src/binemit/mod.rs b/lib/codegen/src/binemit/mod.rs
index 42b16d442b..661297a9ee 100644
--- a/lib/codegen/src/binemit/mod.rs
+++ b/lib/codegen/src/binemit/mod.rs
@@ -132,14 +132,9 @@ where
     // output jump tables
     for (jt, jt_data) in func.jump_tables.iter() {
         let jt_offset = func.jt_offsets[jt];
-        for idx in 0..jt_data.len() {
-            match jt_data.get_entry(idx) {
-                Some(ebb) => {
-                    let rel_offset: i32 = func.offsets[ebb] as i32 - jt_offset as i32;
-                    sink.put4(rel_offset as u32)
-                }
-                None => sink.put4(0),
-            }
+        for ebb in jt_data.iter() {
+            let rel_offset: i32 = func.offsets[*ebb] as i32 - jt_offset as i32;
+            sink.put4(rel_offset as u32)
         }
     }
 }
diff --git a/lib/codegen/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs
index 9883e34418..e25e11990c 100644
--- a/lib/codegen/src/dominator_tree.rs
+++ b/lib/codegen/src/dominator_tree.rs
@@ -347,8 +347,8 @@ impl DominatorTree {
             match func.dfg.analyze_branch(inst) {
                 BranchInfo::SingleDest(succ, _) => self.push_if_unseen(succ),
                 BranchInfo::Table(jt, dest) => {
-                    for (_, succ) in func.jump_tables[jt].entries() {
-                        self.push_if_unseen(succ);
+                    for succ in func.jump_tables[jt].iter() {
+                        self.push_if_unseen(*succ);
                     }
                     if let Some(dest) = dest {
                         self.push_if_unseen(dest);
diff --git a/lib/codegen/src/flowgraph.rs b/lib/codegen/src/flowgraph.rs
index 48b328cb51..3f61f6a852 100644
--- a/lib/codegen/src/flowgraph.rs
+++ b/lib/codegen/src/flowgraph.rs
@@ -129,8 +129,8 @@ impl ControlFlowGraph {
                     if let Some(dest) = dest {
                         self.add_edge(ebb, inst, dest);
                     }
-                    for (_, dest) in func.jump_tables[jt].entries() {
-                        self.add_edge(ebb, inst, dest);
+                    for dest in func.jump_tables[jt].iter() {
+                        self.add_edge(ebb, inst, *dest);
                     }
                 }
                 BranchInfo::NotABranch => {}
diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs
index 26e860dbe4..99516d9711 100644
--- a/lib/codegen/src/ir/function.rs
+++ b/lib/codegen/src/ir/function.rs
@@ -122,11 +122,6 @@ impl Function {
         self.jump_tables.push(data)
     }
 
-    /// Inserts an entry in a previously declared jump table.
-    pub fn insert_jump_table_entry(&mut self, jt: JumpTable, index: usize, ebb: Ebb) {
-        self.jump_tables[jt].set_entry(index, ebb);
-    }
-
     /// Creates a stack slot in the function, to be used by `stack_load`, `stack_store` and
     /// `stack_addr` instructions.
     pub fn create_stack_slot(&mut self, data: StackSlotData) -> StackSlot {
diff --git a/lib/codegen/src/ir/jumptable.rs b/lib/codegen/src/ir/jumptable.rs
index aff6503a57..7ba0a7d13c 100644
--- a/lib/codegen/src/ir/jumptable.rs
+++ b/lib/codegen/src/ir/jumptable.rs
@@ -4,39 +4,29 @@
 //! The actual table of destinations is stored in a `JumpTableData` struct defined in this module.
 
 use ir::entities::Ebb;
-use packed_option::PackedOption;
 use std::fmt::{self, Display, Formatter};
-use std::iter;
-use std::slice;
+use std::slice::{Iter, IterMut};
 use std::vec::Vec;
 
 /// Contents of a jump table.
 ///
-/// All jump tables use 0-based indexing and are expected to be densely populated. They don't need
-/// to be completely populated, though. Individual entries can be missing.
+/// All jump tables use 0-based indexing and densely populated.
 #[derive(Clone)]
 pub struct JumpTableData {
-    // Table entries, using `None` as a placeholder for missing entries.
-    table: Vec>,
-
-    // How many `None` holes in table?
-    holes: usize,
+    // Table entries.
+    table: Vec,
 }
 
 impl JumpTableData {
     /// Create a new empty jump table.
     pub fn new() -> Self {
-        Self {
-            table: Vec::new(),
-            holes: 0,
-        }
+        Self { table: Vec::new() }
     }
 
     /// Create a new empty jump table with the specified capacity.
     pub fn with_capacity(capacity: usize) -> Self {
         Self {
             table: Vec::with_capacity(capacity),
-            holes: 0,
         }
     }
 
@@ -45,100 +35,48 @@ impl JumpTableData {
         self.table.len()
     }
 
-    /// Boolean that is false if the table has missing entries.
-    pub fn fully_dense(&self) -> bool {
-        self.holes == 0
-    }
-
-    /// Set a table entry.
-    ///
-    /// The table will grow as needed to fit `idx`.
-    pub fn set_entry(&mut self, idx: usize, dest: Ebb) {
-        // Resize table to fit `idx`.
-        if idx >= self.table.len() {
-            self.holes += idx - self.table.len();
-            self.table.resize(idx + 1, None.into());
-        } else if self.table[idx].is_none() {
-            // We're filling in an existing hole.
-            self.holes -= 1;
-        }
-        self.table[idx] = dest.into();
-    }
-
     /// Append a table entry.
     pub fn push_entry(&mut self, dest: Ebb) {
-        self.table.push(dest.into())
-    }
-
-    /// Clear a table entry.
-    ///
-    /// The `br_table` instruction will fall through if given an index corresponding to a cleared
-    /// table entry.
-    pub fn clear_entry(&mut self, idx: usize) {
-        if idx < self.table.len() && self.table[idx].is_some() {
-            self.holes += 1;
-            self.table[idx] = None.into();
-        }
-    }
-
-    /// Get the entry for `idx`, or `None`.
-    pub fn get_entry(&self, idx: usize) -> Option {
-        self.table.get(idx).and_then(|e| e.expand())
-    }
-
-    /// Enumerate over all `(idx, dest)` pairs in the table in order.
-    ///
-    /// This returns an iterator that skips any empty slots in the table.
-    pub fn entries(&self) -> Entries {
-        Entries(self.table.iter().cloned().enumerate())
+        self.table.push(dest)
     }
 
     /// Checks if any of the entries branch to `ebb`.
     pub fn branches_to(&self, ebb: Ebb) -> bool {
-        self.table
-            .iter()
-            .any(|target_ebb| target_ebb.expand() == Some(ebb))
+        self.table.iter().any(|target_ebb| *target_ebb == ebb)
+    }
+
+    /// Access the whole table as a slice.
+    pub fn as_slice(&self) -> &[Ebb] {
+        self.table.as_slice()
     }
 
     /// Access the whole table as a mutable slice.
-    pub fn as_mut_slice(&mut self) -> &mut [PackedOption] {
+    pub fn as_mut_slice(&mut self) -> &mut [Ebb] {
         self.table.as_mut_slice()
     }
-}
 
-/// Enumerate `(idx, dest)` pairs in order.
-pub struct Entries<'a>(iter::Enumerate>>>);
+    /// Returns an iterator over the table.
+    pub fn iter(&self) -> Iter {
+        self.table.iter()
+    }
 
-impl<'a> Iterator for Entries<'a> {
-    type Item = (usize, Ebb);
-
-    fn next(&mut self) -> Option {
-        loop {
-            if let Some((idx, dest)) = self.0.next() {
-                if let Some(ebb) = dest.expand() {
-                    return Some((idx, ebb));
-                }
-            } else {
-                return None;
-            }
-        }
+    /// Returns an iterator that allows modifying each value.
+    pub fn iter_mut(&mut self) -> IterMut {
+        self.table.iter_mut()
     }
 }
 
 impl Display for JumpTableData {
     fn fmt(&self, fmt: &mut Formatter) -> fmt::Result {
-        match self.table.first().and_then(|e| e.expand()) {
-            None => write!(fmt, "jump_table 0")?,
-            Some(first) => write!(fmt, "jump_table {}", first)?,
+        write!(fmt, "jump_table [")?;
+        match self.table.first() {
+            None => (),
+            Some(first) => write!(fmt, "{}", first)?,
         }
-
-        for dest in self.table.iter().skip(1).map(|e| e.expand()) {
-            match dest {
-                None => write!(fmt, ", 0")?,
-                Some(ebb) => write!(fmt, ", {}", ebb)?,
-            }
+        for ebb in self.table.iter().skip(1) {
+            write!(fmt, ", {}", ebb)?;
         }
-        Ok(())
+        write!(fmt, "]")
     }
 }
 
@@ -148,18 +86,17 @@ mod tests {
     use entity::EntityRef;
     use ir::Ebb;
     use std::string::ToString;
-    use std::vec::Vec;
 
     #[test]
     fn empty() {
         let jt = JumpTableData::new();
 
-        assert_eq!(jt.get_entry(0), None);
-        assert_eq!(jt.get_entry(10), None);
+        assert_eq!(jt.as_slice().get(0), None);
+        assert_eq!(jt.as_slice().get(10), None);
 
-        assert_eq!(jt.to_string(), "jump_table 0");
+        assert_eq!(jt.to_string(), "jump_table []");
 
-        let v: Vec<(usize, Ebb)> = jt.entries().collect();
+        let v = jt.as_slice();
         assert_eq!(v, []);
     }
 
@@ -170,16 +107,13 @@ mod tests {
 
         let mut jt = JumpTableData::new();
 
-        jt.set_entry(0, e1);
-        jt.set_entry(0, e2);
-        jt.set_entry(10, e1);
+        jt.push_entry(e1);
+        jt.push_entry(e2);
+        jt.push_entry(e1);
 
-        assert_eq!(
-            jt.to_string(),
-            "jump_table ebb2, 0, 0, 0, 0, 0, 0, 0, 0, 0, ebb1"
-        );
+        assert_eq!(jt.to_string(), "jump_table [ebb1, ebb2, ebb1]");
 
-        let v: Vec<(usize, Ebb)> = jt.entries().collect();
-        assert_eq!(v, [(0, e2), (10, e1)]);
+        let v = jt.as_slice();
+        assert_eq!(v, [e1, e2, e1]);
     }
 }
diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs
index 9fb3a59554..e52889c672 100644
--- a/lib/codegen/src/legalizer/mod.rs
+++ b/lib/codegen/src/legalizer/mod.rs
@@ -203,7 +203,6 @@ fn expand_br_table_jt(
     };
 
     let table_size = func.jump_tables[table].len();
-    let table_is_fully_dense = func.jump_tables[table].fully_dense();
     let addr_ty = isa.pointer_type();
     let entry_ty = I32;
 
@@ -222,11 +221,6 @@ fn expand_br_table_jt(
         .ins()
         .jump_table_entry(addr_ty, arg, base_addr, entry_ty.bytes() as u8, table);
 
-    // If the table isn't fully dense, zero-check the entry.
-    if !table_is_fully_dense {
-        pos.ins().brz(entry, default_ebb, &[]);
-    }
-
     let addr = pos.ins().iadd(base_addr, entry);
     pos.ins().indirect_jump_table_br(addr, table);
 
@@ -260,10 +254,9 @@ fn expand_br_table_conds(
     pos.use_srcloc(inst);
 
     for i in 0..table_size {
-        if let Some(dest) = pos.func.jump_tables[table].get_entry(i) {
-            let t = pos.ins().icmp_imm(IntCC::Equal, arg, i as i64);
-            pos.ins().brnz(t, dest, &[]);
-        }
+        let dest = pos.func.jump_tables[table].as_slice()[i];
+        let t = pos.ins().icmp_imm(IntCC::Equal, arg, i as i64);
+        pos.ins().brnz(t, dest, &[]);
     }
 
     // `br_table` jumps to the default destination if nothing matches
diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs
index 0859bec657..3c1a11b3fb 100644
--- a/lib/codegen/src/regalloc/coloring.rs
+++ b/lib/codegen/src/regalloc/coloring.rs
@@ -907,8 +907,8 @@ impl<'a> Context<'a> {
                         .cur
                         .func
                         .jump_tables[jt]
-                        .entries()
-                        .any(|(_, ebb)| lr.is_livein(ebb, ctx)))
+                        .iter()
+                        .any(|ebb| lr.is_livein(*ebb, ctx)))
             }
         }
     }
diff --git a/lib/codegen/src/verifier/flags.rs b/lib/codegen/src/verifier/flags.rs
index 39cdb30660..022ba5e1f3 100644
--- a/lib/codegen/src/verifier/flags.rs
+++ b/lib/codegen/src/verifier/flags.rs
@@ -141,8 +141,8 @@ impl<'a> FlagsVerifier<'a> {
                             merge(&mut live_val, val, inst, errors)?;
                         }
                     }
-                    for (_, dest) in self.func.jump_tables[jt].entries() {
-                        if let Some(val) = self.livein[dest].expand() {
+                    for dest in self.func.jump_tables[jt].iter() {
+                        if let Some(val) = self.livein[*dest].expand() {
                             merge(&mut live_val, val, inst, errors)?;
                         }
                     }
diff --git a/lib/codegen/src/verifier/locations.rs b/lib/codegen/src/verifier/locations.rs
index c1ed8bd1b9..4962ef916b 100644
--- a/lib/codegen/src/verifier/locations.rs
+++ b/lib/codegen/src/verifier/locations.rs
@@ -347,8 +347,8 @@ impl<'a> LocationVerifier<'a> {
                             );
                         }
                     }
-                    for (_, ebb) in self.func.jump_tables[jt].entries() {
-                        if lr.is_livein(ebb, liveness.context(&self.func.layout)) {
+                    for ebb in self.func.jump_tables[jt].iter() {
+                        if lr.is_livein(*ebb, liveness.context(&self.func.layout)) {
                             return fatal!(
                                 errors,
                                 inst,
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 01baa66c11..2daa1d5440 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -1229,8 +1229,8 @@ impl<'a> Verifier<'a> {
                         );
                     }
                 }
-                for (_, ebb) in self.func.jump_tables[table].entries() {
-                    let arg_count = self.func.dfg.num_ebb_params(ebb);
+                for ebb in self.func.jump_tables[table].iter() {
+                    let arg_count = self.func.dfg.num_ebb_params(*ebb);
                     if arg_count != 0 {
                         return nonfatal!(
                             errors,
diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs
index 2f0db746bd..d8dfcb617d 100644
--- a/lib/filetests/src/test_binemit.rs
+++ b/lib/filetests/src/test_binemit.rs
@@ -291,14 +291,9 @@ impl SubTest for TestBinEmit {
 
         for (jt, jt_data) in func.jump_tables.iter() {
             let jt_offset = func.jt_offsets[jt];
-            for idx in 0..jt_data.len() {
-                match jt_data.get_entry(idx) {
-                    Some(ebb) => {
-                        let rel_offset: i32 = func.offsets[ebb] as i32 - jt_offset as i32;
-                        sink.put4(rel_offset as u32)
-                    }
-                    None => sink.put4(0),
-                }
+            for ebb in jt_data.iter() {
+                let rel_offset: i32 = func.offsets[*ebb] as i32 - jt_offset as i32;
+                sink.put4(rel_offset as u32)
             }
         }
 
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index c2befee997..e637d6bf18 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -152,12 +152,11 @@ impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
                             .jump_tables
                             .get(table)
                             .expect("you are referencing an undeclared jump table")
-                            .entries()
-                            .map(|(_, ebb)| ebb)
-                            .filter(|dest_ebb| unique.insert(*dest_ebb))
+                            .iter()
+                            .filter(|&dest_ebb| unique.insert(*dest_ebb))
                         {
                             self.builder.func_ctx.ssa.declare_ebb_predecessor(
-                                dest_ebb,
+                                *dest_ebb,
                                 self.builder.position.basic_block.unwrap(),
                                 inst,
                             )
@@ -336,11 +335,6 @@ impl<'a> FunctionBuilder<'a> {
         self.func.create_jump_table(data)
     }
 
-    /// Inserts an entry in a previously declared jump table.
-    pub fn insert_jump_table_entry(&mut self, jt: JumpTable, index: usize, ebb: Ebb) {
-        self.func.insert_jump_table_entry(jt, index, ebb)
-    }
-
     /// Creates a stack slot in the function, to be used by `stack_load`, `stack_store` and
     /// `stack_addr` instructions.
     pub fn create_stack_slot(&mut self, data: StackSlotData) -> StackSlot {
diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs
index d13cc0531a..a78ed0b890 100644
--- a/lib/frontend/src/ssa.rs
+++ b/lib/frontend/src/ssa.rs
@@ -672,8 +672,8 @@ impl SSABuilder {
                 }
 
                 for old_dest in func.jump_tables[jt].as_mut_slice() {
-                    if *old_dest == PackedOption::from(dest_ebb) {
-                        *old_dest = PackedOption::from(middle_ebb);
+                    if *old_dest == dest_ebb {
+                        *old_dest = middle_ebb;
                     }
                 }
                 let mut cur = FuncCursor::new(func).at_bottom(middle_ebb);
@@ -1005,7 +1005,7 @@ mod tests {
         // Here is the pseudo-program we want to translate:
         //
         // function %f {
-        // jt = jump_table ebb2, 0, ebb1
+        // jt = jump_table [ebb2, ebb1]
         // ebb0:
         //    x = 1;
         //    br_table x, ebb2, jt
@@ -1039,9 +1039,9 @@ mod tests {
         };
         ssa.def_var(x_var, x1, block0);
 
-        // jt = jump_table ebb2, 0, ebb1
+        // jt = jump_table [ebb2, ebb1]
         jump_table.push_entry(ebb2);
-        jump_table.set_entry(2, ebb1);
+        jump_table.push_entry(ebb1);
         let jt = func.create_jump_table(jump_table);
 
         // ebb0:
diff --git a/lib/frontend/src/switch.rs b/lib/frontend/src/switch.rs
index 7e8db1183d..f01a9f620b 100644
--- a/lib/frontend/src/switch.rs
+++ b/lib/frontend/src/switch.rs
@@ -244,7 +244,7 @@ mod tests {
         let func = setup!(0, [0, 1,]);
         assert_eq!(
             func,
-            "    jt0 = jump_table ebb1, ebb2
+            "    jt0 = jump_table [ebb1, ebb2]
 
 ebb0:
     v0 = iconst.i8 0
@@ -280,8 +280,8 @@ ebb3:
         let func = setup!(0, [0, 1, 5, 7, 10, 11, 12,]);
         assert_eq!(
             func,
-            "    jt0 = jump_table ebb1, ebb2
-    jt1 = jump_table ebb5, ebb6, ebb7
+            "    jt0 = jump_table [ebb1, ebb2]
+    jt1 = jump_table [ebb5, ebb6, ebb7]
 
 ebb0:
     v0 = iconst.i8 0
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 8809774047..4840387e5a 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -1494,48 +1494,48 @@ impl<'a> Parser<'a> {
 
     // Parse a jump table decl.
     //
-    // jump-table-decl ::= * JumpTable(jt) "=" "jump_table" jt-entry {"," jt-entry}
+    // jump-table-decl ::= * JumpTable(jt) "=" "jump_table" "[" jt-entry {"," jt-entry} "]"
     fn parse_jump_table_decl(&mut self) -> ParseResult<(JumpTable, JumpTableData)> {
         let jt = self.match_jt()?;
         self.match_token(Token::Equal, "expected '=' in jump_table decl")?;
         self.match_identifier("jump_table", "expected 'jump_table'")?;
+        self.match_token(Token::LBracket, "expected '[' before jump table contents")?;
 
         let mut data = JumpTableData::new();
 
-        // jump-table-decl ::= JumpTable(jt) "=" "jump_table" * jt-entry {"," jt-entry}
-        for idx in 0_usize.. {
-            if let Some(dest) = self.parse_jump_table_entry()? {
-                data.set_entry(idx, dest);
-            }
-            if !self.optional(Token::Comma) {
-                // Collect any trailing comments.
-                self.token();
-                self.claim_gathered_comments(jt);
-
-                return Ok((jt, data));
-            }
-        }
-
-        err!(self.loc, "jump_table too long")
-    }
-
-    // jt-entry ::= * Ebb(dest) | "0"
-    fn parse_jump_table_entry(&mut self) -> ParseResult> {
+        // jump-table-decl ::= JumpTable(jt) "=" "jump_table" "[" * Ebb(dest) {"," Ebb(dest)} "]"
         match self.token() {
-            Some(Token::Integer(s)) => {
-                if s == "0" {
-                    self.consume();
-                    Ok(None)
-                } else {
-                    err!(self.loc, "invalid jump_table entry '{}'", s)
-                }
-            }
             Some(Token::Ebb(dest)) => {
                 self.consume();
-                Ok(Some(dest))
+                data.push_entry(dest);
+
+                loop {
+                    match self.token() {
+                        Some(Token::Comma) => {
+                            self.consume();
+                            if let Some(Token::Ebb(dest)) = self.token() {
+                                self.consume();
+                                data.push_entry(dest);
+                            } else {
+                                return err!(self.loc, "expected jump_table entry");
+                            }
+                        }
+                        Some(Token::RBracket) => break,
+                        _ => return err!(self.loc, "expected ']' after jump table contents"),
+                    }
+                }
             }
-            _ => err!(self.loc, "expected jump_table entry"),
+            Some(Token::RBracket) => (),
+            _ => return err!(self.loc, "expected jump_table entry"),
         }
+
+        self.consume();
+
+        // Collect any trailing comments.
+        self.token();
+        self.claim_gathered_comments(jt);
+
+        Ok((jt, data))
     }
 
     // Parse a function body, add contents to `ctx`.
@@ -2738,8 +2738,8 @@ mod tests {
     fn duplicate_jt() {
         let ParseError { location, message } = Parser::new(
             "function %ebbs() system_v {
-                jt0 = jump_table 0, 0
-                jt0 = jump_table 0, 0",
+                jt0 = jump_table []
+                jt0 = jump_table []",
         ).parse_function(None)
         .unwrap_err();
 
@@ -2820,7 +2820,7 @@ mod tests {
                          function %comment() system_v { ; decl
                             ss10  = outgoing_arg 13 ; stackslot.
                             ; Still stackslot.
-                            jt10 = jump_table ebb0
+                            jt10 = jump_table [ebb0]
                             ; Jumptable
                          ebb0: ; Basic block
                          trap user42; Instruction
diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs
index ca653d21b5..7e4da9de80 100644
--- a/lib/reader/src/sourcemap.rs
+++ b/lib/reader/src/sourcemap.rs
@@ -218,7 +218,7 @@ mod tests {
         let tf = parse_test(
             "function %detail() {
                                ss10 = incoming_arg 13
-                               jt10 = jump_table ebb0
+                               jt10 = jump_table [ebb0]
                              ebb0(v4: i32, v7: i32):
                                v10 = iadd v4, v7
                              }",

From 20eea311a39ed43163f247eaf4e645e351276fb0 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 28 Sep 2018 22:55:06 -0700
Subject: [PATCH 2118/3084] Miscellaenous code cleanups.

---
 lib/codegen/src/postopt.rs | 81 +++++++++++++++++---------------------
 1 file changed, 36 insertions(+), 45 deletions(-)

diff --git a/lib/codegen/src/postopt.rs b/lib/codegen/src/postopt.rs
index 178f143c6f..fdfc358697 100644
--- a/lib/codegen/src/postopt.rs
+++ b/lib/codegen/src/postopt.rs
@@ -171,17 +171,15 @@ fn optimize_cpu_flags(
 
 struct MemOpInfo {
     opcode: Opcode,
-    inst: Inst,
     itype: Type,
     arg: Value,
     st_arg: Option,
     flags: MemFlags,
     offset: Offset32,
-    add_args: Option<[Value; 2]>,
 }
 
 fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa) {
-    let mut info = match pos.func.dfg[inst] {
+    let info = match pos.func.dfg[inst] {
         InstructionData::Load {
             opcode,
             arg,
@@ -189,13 +187,11 @@ fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa)
             offset,
         } => MemOpInfo {
             opcode,
-            inst,
             itype: pos.func.dfg.ctrl_typevar(inst),
             arg,
             st_arg: None,
             flags,
             offset,
-            add_args: None,
         },
         InstructionData::Store {
             opcode,
@@ -204,118 +200,113 @@ fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa)
             offset,
         } => MemOpInfo {
             opcode,
-            inst,
             itype: pos.func.dfg.ctrl_typevar(inst),
             arg: args[1],
             st_arg: Some(args[0]),
             flags,
             offset,
-            add_args: None,
         },
         _ => return,
     };
 
-    if let ValueDef::Result(result_inst, _) = pos.func.dfg.value_def(info.arg) {
+    let add_args = if let ValueDef::Result(result_inst, _) = pos.func.dfg.value_def(info.arg) {
         match pos.func.dfg[result_inst] {
-            InstructionData::Binary { opcode, args } if opcode == Opcode::Iadd => {
-                info.add_args = Some(args);
-            }
+            InstructionData::Binary {
+                opcode: Opcode::Iadd,
+                args,
+            } => args,
             _ => return,
         }
     } else {
         return;
-    }
+    };
 
     match info.opcode {
         Opcode::Load => {
-            pos.func.dfg.replace(info.inst).load_complex(
-                info.itype,
-                info.flags,
-                &info.add_args.unwrap(),
-                info.offset,
-            );
+            pos.func
+                .dfg
+                .replace(inst)
+                .load_complex(info.itype, info.flags, &add_args, info.offset);
         }
         Opcode::Uload8 => {
-            pos.func.dfg.replace(info.inst).uload8_complex(
+            pos.func.dfg.replace(inst).uload8_complex(
                 info.itype,
                 info.flags,
-                &info.add_args.unwrap(),
+                &add_args,
                 info.offset,
             );
         }
         Opcode::Sload8 => {
-            pos.func.dfg.replace(info.inst).sload8_complex(
+            pos.func.dfg.replace(inst).sload8_complex(
                 info.itype,
                 info.flags,
-                &info.add_args.unwrap(),
+                &add_args,
                 info.offset,
             );
         }
         Opcode::Uload16 => {
-            pos.func.dfg.replace(info.inst).uload16_complex(
+            pos.func.dfg.replace(inst).uload16_complex(
                 info.itype,
                 info.flags,
-                &info.add_args.unwrap(),
+                &add_args,
                 info.offset,
             );
         }
         Opcode::Sload16 => {
-            pos.func.dfg.replace(info.inst).sload16_complex(
+            pos.func.dfg.replace(inst).sload16_complex(
                 info.itype,
                 info.flags,
-                &info.add_args.unwrap(),
+                &add_args,
                 info.offset,
             );
         }
         Opcode::Uload32 => {
-            pos.func.dfg.replace(info.inst).uload32_complex(
-                info.flags,
-                &info.add_args.unwrap(),
-                info.offset,
-            );
+            pos.func
+                .dfg
+                .replace(inst)
+                .uload32_complex(info.flags, &add_args, info.offset);
         }
         Opcode::Sload32 => {
-            pos.func.dfg.replace(info.inst).sload32_complex(
-                info.flags,
-                &info.add_args.unwrap(),
-                info.offset,
-            );
+            pos.func
+                .dfg
+                .replace(inst)
+                .sload32_complex(info.flags, &add_args, info.offset);
         }
         Opcode::Store => {
-            pos.func.dfg.replace(info.inst).store_complex(
+            pos.func.dfg.replace(inst).store_complex(
                 info.flags,
                 info.st_arg.unwrap(),
-                &info.add_args.unwrap(),
+                &add_args,
                 info.offset,
             );
         }
         Opcode::Istore8 => {
-            pos.func.dfg.replace(info.inst).istore8_complex(
+            pos.func.dfg.replace(inst).istore8_complex(
                 info.flags,
                 info.st_arg.unwrap(),
-                &info.add_args.unwrap(),
+                &add_args,
                 info.offset,
             );
         }
         Opcode::Istore16 => {
-            pos.func.dfg.replace(info.inst).istore16_complex(
+            pos.func.dfg.replace(inst).istore16_complex(
                 info.flags,
                 info.st_arg.unwrap(),
-                &info.add_args.unwrap(),
+                &add_args,
                 info.offset,
             );
         }
         Opcode::Istore32 => {
-            pos.func.dfg.replace(info.inst).istore32_complex(
+            pos.func.dfg.replace(inst).istore32_complex(
                 info.flags,
                 info.st_arg.unwrap(),
-                &info.add_args.unwrap(),
+                &add_args,
                 info.offset,
             );
         }
         _ => return,
     }
-    let ok = pos.func.update_encoding(info.inst, isa).is_ok();
+    let ok = pos.func.update_encoding(inst, isa).is_ok();
     debug_assert!(ok);
 }
 

From 06bbd3e39373fa2a84ef0ef8cfb91dc9602ad245 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 28 Sep 2018 23:19:15 -0700
Subject: [PATCH 2119/3084] Pass the target isa when printing a function as a
 string.

This allows encodings and register names to be printed correctly.
---
 lib/filetests/src/test_dce.rs        | 2 +-
 lib/filetests/src/test_licm.rs       | 2 +-
 lib/filetests/src/test_postopt.rs    | 2 +-
 lib/filetests/src/test_preopt.rs     | 2 +-
 lib/filetests/src/test_shrink.rs     | 2 +-
 lib/filetests/src/test_simple_gvn.rs | 2 +-
 6 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/lib/filetests/src/test_dce.rs b/lib/filetests/src/test_dce.rs
index e5a4e79f98..6c1645d5fc 100644
--- a/lib/filetests/src/test_dce.rs
+++ b/lib/filetests/src/test_dce.rs
@@ -41,7 +41,7 @@ impl SubTest for TestDCE {
             .dce(context.flags_or_isa())
             .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?;
 
-        let text = comp_ctx.func.to_string();
+        let text = comp_ctx.func.display(context.isa).to_string();
         run_filecheck(&text, context)
     }
 }
diff --git a/lib/filetests/src/test_licm.rs b/lib/filetests/src/test_licm.rs
index 149cf788cf..57523cdf59 100644
--- a/lib/filetests/src/test_licm.rs
+++ b/lib/filetests/src/test_licm.rs
@@ -41,7 +41,7 @@ impl SubTest for TestLICM {
             .licm(context.flags_or_isa())
             .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?;
 
-        let text = comp_ctx.func.to_string();
+        let text = comp_ctx.func.display(context.isa).to_string();
         run_filecheck(&text, context)
     }
 }
diff --git a/lib/filetests/src/test_postopt.rs b/lib/filetests/src/test_postopt.rs
index 248528c2b6..4789fd00c7 100644
--- a/lib/filetests/src/test_postopt.rs
+++ b/lib/filetests/src/test_postopt.rs
@@ -38,7 +38,7 @@ impl SubTest for TestPostopt {
             .postopt(isa)
             .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?;
 
-        let text = comp_ctx.func.to_string();
+        let text = comp_ctx.func.display(isa).to_string();
         run_filecheck(&text, context)
     }
 }
diff --git a/lib/filetests/src/test_preopt.rs b/lib/filetests/src/test_preopt.rs
index 7e4ae15db7..86f8f87de7 100644
--- a/lib/filetests/src/test_preopt.rs
+++ b/lib/filetests/src/test_preopt.rs
@@ -38,7 +38,7 @@ impl SubTest for TestPreopt {
             .preopt(isa)
             .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?;
 
-        let text = &comp_ctx.func.to_string();
+        let text = &comp_ctx.func.display(isa).to_string();
         run_filecheck(&text, context)
     }
 }
diff --git a/lib/filetests/src/test_shrink.rs b/lib/filetests/src/test_shrink.rs
index 69eaabc007..7d909c6187 100644
--- a/lib/filetests/src/test_shrink.rs
+++ b/lib/filetests/src/test_shrink.rs
@@ -40,7 +40,7 @@ impl SubTest for TestShrink {
             .shrink_instructions(isa)
             .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?;
 
-        let text = comp_ctx.func.to_string();
+        let text = comp_ctx.func.display(isa).to_string();
         run_filecheck(&text, context)
     }
 }
diff --git a/lib/filetests/src/test_simple_gvn.rs b/lib/filetests/src/test_simple_gvn.rs
index d6d06f49ce..5b664a4f74 100644
--- a/lib/filetests/src/test_simple_gvn.rs
+++ b/lib/filetests/src/test_simple_gvn.rs
@@ -40,7 +40,7 @@ impl SubTest for TestSimpleGVN {
             .simple_gvn(context.flags_or_isa())
             .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?;
 
-        let text = comp_ctx.func.to_string();
+        let text = comp_ctx.func.display(context.isa).to_string();
         run_filecheck(&text, context)
     }
 }

From c61722f83ffc8d7f81477d886c51ac71ff557586 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Sun, 30 Sep 2018 15:34:14 -0700
Subject: [PATCH 2120/3084] Remove obsolete clippy scripts.

Now that clippy is installable via rustup and is generally more stable,
we no longer need special scripts. `rustup component add clippy-preview` is
sufficient to install clippy, and `cargo clippy` is sufficient to run it.

Also, don't run clippy in test-all.sh. We do generally want to fix things
clippy reports, however it's not a requirement that the code be kept
clippy-warning-free at all times.
---
 check-clippy.sh       | 10 ----------
 clippy-all.sh         |  7 -------
 cranelift/test-all.sh | 10 +---------
 3 files changed, 1 insertion(+), 26 deletions(-)
 delete mode 100755 check-clippy.sh
 delete mode 100755 clippy-all.sh

diff --git a/check-clippy.sh b/check-clippy.sh
deleted file mode 100755
index 072069cb9b..0000000000
--- a/check-clippy.sh
+++ /dev/null
@@ -1,10 +0,0 @@
-#!/bin/bash
-set -euo pipefail
-
-# Usage: check-clippy.sh
-
-if cargo install --list | tee /dev/null | grep -q "^clippy v0"; then
-    exit 0
-else
-    exit 1
-fi
diff --git a/clippy-all.sh b/clippy-all.sh
deleted file mode 100755
index ba840b1d00..0000000000
--- a/clippy-all.sh
+++ /dev/null
@@ -1,7 +0,0 @@
-#!/bin/bash
-set -euo pipefail
-
-# Check all sources with clippy.
-# In the clif-util crate (root dir) clippy will only work with nightly cargo -
-# there is a bug where it will reject the commands passed to it by cargo 0.25.0
-exec cargo +nightly clippy --all
diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index 83db4528ec..46a985a4e7 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -9,7 +9,7 @@ set -euo pipefail
 # - Make a release build.
 # - Run unit tests for all Rust crates (including the filetests)
 # - Build API documentation.
-# - Optionally, run clippy and fuzzing.
+# - Optionally, run fuzzing.
 #
 # All tests run by this script should be passing at all times.
 
@@ -70,14 +70,6 @@ cargo test --all
 banner "Rust documentation: $topdir/target/doc/cranelift/index.html"
 cargo doc
 
-# Run clippy if we have it.
-banner "Rust linter"
-if "$topdir/check-clippy.sh"; then
-    "$topdir/clippy-all.sh"
-else
-    echo "\`cargo +nightly install clippy\` for optional rust linting"
-fi
-
 # Ensure fuzzer works by running it with a single input
 # Note LSAN is disabled due to https://github.com/google/sanitizers/issues/764
 banner "cargo fuzz check"

From bf041e3ae2ee2080fa54e79ac2842c1f6ff7f65a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 5 Oct 2018 06:43:22 -0700
Subject: [PATCH 2121/3084] Move `return_at_end` out of Settings and into the
 wasm FuncEnvironment. (#547)

* Move `return_at_end` out of Settings and into the wasm FuncEnvironment.

The `return_at_end` flag supports users that want to append a custom
epilogue to Cranelift-produced functions. It arranges for functions to
always return via a single return statement at the end, and users are
expected to remove this return to append their code.

This patch makes two changes:
 - First, introduce a `fallthrough_return` instruction and use that
   instead of adding a `return` at the end. That's simpler than having
   users remove the `return` themselves.

 - Second, move this setting out of the Settings and into the wasm
   FuncEnvironment. This flag isn't something the code generator uses,
   it's something that the wasm translator uses. The code generator
   needs to preserve the property, however we can give the
   `fallthrough_return` instruction properties to ensure this as needed,
   such as marking it non-cloneable.
---
 .../filetests/verifier/return_at_end.clif     | 22 -----------
 cranelift/src/wasm.rs                         | 11 ++++--
 ..._at_end.wat => use_fallthrough_return.wat} |  0
 lib/codegen/meta-python/base/instructions.py  | 10 +++++
 lib/codegen/meta-python/base/settings.py      | 10 -----
 lib/codegen/src/settings.rs                   |  1 -
 lib/codegen/src/verifier/mod.rs               | 30 ++-------------
 lib/wasm/src/code_translator.rs               | 11 +++---
 lib/wasm/src/environ/dummy.rs                 | 37 +++++++++++++++----
 lib/wasm/src/environ/mod.rs                   |  2 +-
 lib/wasm/src/environ/spec.rs                  | 14 +++++++
 lib/wasm/src/func_translator.rs               |  7 +++-
 lib/wasm/src/lib.rs                           |  3 +-
 lib/wasm/tests/wasm_testsuite.rs              | 24 +++++++-----
 14 files changed, 91 insertions(+), 91 deletions(-)
 delete mode 100644 cranelift/filetests/verifier/return_at_end.clif
 rename cranelift/wasmtests/{return_at_end.wat => use_fallthrough_return.wat} (100%)

diff --git a/cranelift/filetests/verifier/return_at_end.clif b/cranelift/filetests/verifier/return_at_end.clif
deleted file mode 100644
index fcbae57070..0000000000
--- a/cranelift/filetests/verifier/return_at_end.clif
+++ /dev/null
@@ -1,22 +0,0 @@
-test verifier
-set return_at_end
-
-function %ok(i32) {
-ebb0(v0: i32):
-    brnz v0, ebb1
-    trap int_divz
-
-ebb1:
-    trapz v0, user5
-    return
-}
-
-function %bad(i32) {
-ebb0(v0: i32):
-    brnz v0, ebb1
-    return ; error: Internal return not allowed
-
-ebb1:
-    trapz v0, user6
-    return
-}
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 11b859d52b..3e91c3e7b0 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -11,7 +11,9 @@ use cranelift_codegen::print_errors::{pretty_error, pretty_verifier_error};
 use cranelift_codegen::settings::FlagsOrIsa;
 use cranelift_codegen::Context;
 use cranelift_entity::EntityRef;
-use cranelift_wasm::{translate_module, DummyEnvironment, FuncIndex, ModuleEnvironment};
+use cranelift_wasm::{
+    translate_module, DummyEnvironment, FuncIndex, ModuleEnvironment, ReturnMode,
+};
 use std::error::Error;
 use std::path::Path;
 use std::path::PathBuf;
@@ -102,8 +104,11 @@ fn handle_module(
         }
     };
 
-    let mut dummy_environ =
-        DummyEnvironment::with_triple_flags(isa.triple().clone(), fisa.flags.clone());
+    let mut dummy_environ = DummyEnvironment::with_triple_flags(
+        isa.triple().clone(),
+        fisa.flags.clone(),
+        ReturnMode::NormalReturns,
+    );
     translate_module(&module_binary, &mut dummy_environ).map_err(|e| e.to_string())?;
 
     let _ = terminal.fg(term::color::GREEN);
diff --git a/cranelift/wasmtests/return_at_end.wat b/cranelift/wasmtests/use_fallthrough_return.wat
similarity index 100%
rename from cranelift/wasmtests/return_at_end.wat
rename to cranelift/wasmtests/use_fallthrough_return.wat
diff --git a/lib/codegen/meta-python/base/instructions.py b/lib/codegen/meta-python/base/instructions.py
index c22b6b9b77..2993d6de49 100644
--- a/lib/codegen/meta-python/base/instructions.py
+++ b/lib/codegen/meta-python/base/instructions.py
@@ -234,6 +234,16 @@ x_return = Instruction(
         """,
         ins=rvals, is_return=True, is_terminator=True)
 
+fallthrough_return = Instruction(
+        'fallthrough_return', r"""
+        Return from the function by fallthrough.
+
+        This is a specialized instruction for use where one wants to append
+        a custom epilogue, which will then perform the real return. This
+        instruction has no encoding.
+        """,
+        ins=rvals, is_return=True, is_terminator=True)
+
 FN = Operand(
         'FN',
         entities.func_ref,
diff --git a/lib/codegen/meta-python/base/settings.py b/lib/codegen/meta-python/base/settings.py
index e43e455693..4eb7541840 100644
--- a/lib/codegen/meta-python/base/settings.py
+++ b/lib/codegen/meta-python/base/settings.py
@@ -65,16 +65,6 @@ colocated_libcalls = BoolSetting(
         they can use more efficient addressing.
         """)
 
-return_at_end = BoolSetting(
-        """
-        Generate functions with at most a single return instruction at the
-        end of the function.
-
-        This guarantees that functions do not have any internal return
-        instructions. Either they never return, or they have a single return
-        instruction at the end.
-        """)
-
 avoid_div_traps = BoolSetting(
         """
         Generate explicit checks around native division instructions to avoid
diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs
index 743bd29268..c9374d9e47 100644
--- a/lib/codegen/src/settings.rs
+++ b/lib/codegen/src/settings.rs
@@ -370,7 +370,6 @@ mod tests {
              call_conv = \"fast\"\n\
              is_pic = false\n\
              colocated_libcalls = false\n\
-             return_at_end = false\n\
              avoid_div_traps = false\n\
              enable_float = true\n\
              enable_nan_canonicalization = false\n\
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 2daa1d5440..ba396a8b10 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -307,7 +307,6 @@ struct Verifier<'a> {
     func: &'a Function,
     expected_cfg: ControlFlowGraph,
     expected_domtree: DominatorTree,
-    flags: &'a Flags,
     isa: Option<&'a TargetIsa>,
 }
 
@@ -319,7 +318,6 @@ impl<'a> Verifier<'a> {
             func,
             expected_cfg,
             expected_domtree,
-            flags: fisa.flags,
             isa: fisa.isa,
         }
     }
@@ -1654,9 +1652,9 @@ impl<'a> Verifier<'a> {
         // Instructions with side effects are not allowed to be ghost instructions.
         let opcode = self.func.dfg[inst].opcode();
 
-        // The `fallthrough` instruction is marked as a terminator and a branch, but it is not
-        // required to have an encoding.
-        if opcode == Opcode::Fallthrough {
+        // The `fallthrough` and `fallthrough_return` instructions are marked as terminators and
+        // branches, but they are not required to have an encoding.
+        if opcode == Opcode::Fallthrough || opcode == Opcode::FallthroughReturn {
             return Ok(());
         }
 
@@ -1696,24 +1694,6 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
-    /// Verify the `return_at_end` property which requires that there are no internal return
-    /// instructions.
-    fn verify_return_at_end(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
-        for ebb in self.func.layout.ebbs() {
-            let inst = self.func.layout.last_inst(ebb).unwrap();
-            if self.func.dfg[inst].opcode().is_return() && Some(ebb) != self.func.layout.last_ebb()
-            {
-                report!(
-                    errors,
-                    inst,
-                    "Internal return not allowed with return_at_end=1"
-                );
-            }
-        }
-
-        errors.as_result()
-    }
-
     pub fn run(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         self.verify_global_values(errors)?;
         self.verify_heaps(errors)?;
@@ -1729,10 +1709,6 @@ impl<'a> Verifier<'a> {
             }
         }
 
-        if self.flags.return_at_end() {
-            self.verify_return_at_end(errors)?;
-        }
-
         verify_flags(self.func, &self.expected_cfg, self.isa, errors)?;
 
         Ok(())
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index 3c615568ca..152c20ca20 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -28,7 +28,7 @@ use cranelift_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags};
 use cranelift_codegen::packed_option::ReservedValue;
 use cranelift_entity::EntityRef;
 use cranelift_frontend::{FunctionBuilder, Variable};
-use environ::{FuncEnvironment, GlobalVariable, WasmError, WasmResult};
+use environ::{FuncEnvironment, GlobalVariable, ReturnMode, WasmError, WasmResult};
 use state::{ControlStackFrame, TranslationState};
 use std::collections::{hash_map, HashMap};
 use std::vec::Vec;
@@ -340,11 +340,10 @@ pub fn translate_operator(
             };
             {
                 let args = state.peekn(return_count);
-                if environ.flags().return_at_end() {
-                    builder.ins().jump(br_destination, args);
-                } else {
-                    builder.ins().return_(args);
-                }
+                match environ.return_mode() {
+                    ReturnMode::NormalReturns => builder.ins().return_(args),
+                    ReturnMode::FallthroughReturn => builder.ins().jump(br_destination, args),
+                };
             }
             state.popn(return_count);
             state.reachable = false;
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index d1e6a8052c..ae0edb96e3 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -7,7 +7,7 @@ use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::settings;
 use cranelift_entity::{EntityRef, PrimaryMap};
-use environ::{FuncEnvironment, GlobalVariable, ModuleEnvironment, WasmResult};
+use environ::{FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmResult};
 use func_translator::FuncTranslator;
 use std::string::String;
 use std::vec::Vec;
@@ -105,38 +105,55 @@ pub struct DummyEnvironment {
 
     /// Vector of wasm bytecode size for each function.
     pub func_bytecode_sizes: Vec,
+
+    /// How to return from functions.
+    return_mode: ReturnMode,
 }
 
 impl DummyEnvironment {
     /// Allocates the data structures with default flags.
     pub fn with_triple(triple: Triple) -> Self {
-        Self::with_triple_flags(triple, settings::Flags::new(settings::builder()))
+        Self::with_triple_flags(
+            triple,
+            settings::Flags::new(settings::builder()),
+            ReturnMode::NormalReturns,
+        )
     }
 
-    /// Allocates the data structures with the given flags.
-    pub fn with_triple_flags(triple: Triple, flags: settings::Flags) -> Self {
+    /// Allocates the data structures with the given triple.
+    pub fn with_triple_flags(
+        triple: Triple,
+        flags: settings::Flags,
+        return_mode: ReturnMode,
+    ) -> Self {
         Self {
             info: DummyModuleInfo::with_triple_flags(triple, flags),
             trans: FuncTranslator::new(),
             func_bytecode_sizes: Vec::new(),
+            return_mode,
         }
     }
 
     /// Return a `DummyFuncEnvironment` for translating functions within this
     /// `DummyEnvironment`.
     pub fn func_env(&self) -> DummyFuncEnvironment {
-        DummyFuncEnvironment::new(&self.info)
+        DummyFuncEnvironment::new(&self.info, self.return_mode)
     }
 }
 
 /// The `FuncEnvironment` implementation for use by the `DummyEnvironment`.
 pub struct DummyFuncEnvironment<'dummy_environment> {
     pub mod_info: &'dummy_environment DummyModuleInfo,
+
+    return_mode: ReturnMode,
 }
 
 impl<'dummy_environment> DummyFuncEnvironment<'dummy_environment> {
-    pub fn new(mod_info: &'dummy_environment DummyModuleInfo) -> Self {
-        Self { mod_info }
+    pub fn new(mod_info: &'dummy_environment DummyModuleInfo, return_mode: ReturnMode) -> Self {
+        Self {
+            mod_info,
+            return_mode,
+        }
     }
 
     // Create a signature for `sigidx` amended with a `vmctx` argument after the standard wasm
@@ -321,6 +338,10 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
     ) -> WasmResult {
         Ok(pos.ins().iconst(I32, -1))
     }
+
+    fn return_mode(&self) -> ReturnMode {
+        self.return_mode
+    }
 }
 
 impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
@@ -433,7 +454,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
 
     fn define_function_body(&mut self, body_bytes: &'data [u8]) -> WasmResult<()> {
         let func = {
-            let mut func_environ = DummyFuncEnvironment::new(&self.info);
+            let mut func_environ = DummyFuncEnvironment::new(&self.info, self.return_mode);
             let func_index =
                 FuncIndex::new(self.get_num_func_imports() + self.info.function_bodies.len());
             let name = get_func_name(func_index);
diff --git a/lib/wasm/src/environ/mod.rs b/lib/wasm/src/environ/mod.rs
index 0ca34575f7..d44011fd8c 100644
--- a/lib/wasm/src/environ/mod.rs
+++ b/lib/wasm/src/environ/mod.rs
@@ -5,5 +5,5 @@ mod spec;
 
 pub use environ::dummy::DummyEnvironment;
 pub use environ::spec::{
-    FuncEnvironment, GlobalVariable, ModuleEnvironment, WasmError, WasmResult,
+    FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmError, WasmResult,
 };
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 31640635b5..872ec28475 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -74,6 +74,15 @@ impl WasmError {
 /// A convenient alias for a `Result` that uses `WasmError` as the error type.
 pub type WasmResult = Result;
 
+/// How to return from functions.
+#[derive(Copy, Clone, PartialEq, Eq, Debug)]
+pub enum ReturnMode {
+    /// Use normal return instructions as needed.
+    NormalReturns,
+    /// Use a single fallthrough return at the end of the function.
+    FallthroughReturn,
+}
+
 /// Environment affecting the translation of a single WebAssembly function.
 ///
 /// A `FuncEnvironment` trait object is required to translate a WebAssembly function to Cranelift
@@ -216,6 +225,11 @@ pub trait FuncEnvironment {
     fn translate_loop_header(&mut self, _pos: FuncCursor) {
         // By default, don't emit anything.
     }
+
+    /// Should the code be structured to use a single `fallthrough_return` instruction at the end
+    /// of the function body, rather than `return` instructions as needed? This is used by VMs
+    /// to append custom epilogues.
+    fn return_mode(&self) -> ReturnMode;
 }
 
 /// An object satisfying the `ModuleEnvironment` trait can be passed as argument to the
diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs
index 0f946751db..0d99754c79 100644
--- a/lib/wasm/src/func_translator.rs
+++ b/lib/wasm/src/func_translator.rs
@@ -9,7 +9,7 @@ use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir::{self, Ebb, InstBuilder};
 use cranelift_codegen::timing;
 use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
-use environ::{FuncEnvironment, WasmError, WasmResult};
+use environ::{FuncEnvironment, ReturnMode, WasmError, WasmResult};
 use state::TranslationState;
 use wasmparser::{self, BinaryReader};
 
@@ -209,7 +209,10 @@ fn parse_function_body(
     if state.reachable {
         debug_assert!(builder.is_pristine());
         if !builder.is_unreachable() {
-            builder.ins().return_(&state.stack);
+            match environ.return_mode() {
+                ReturnMode::NormalReturns => builder.ins().return_(&state.stack),
+                ReturnMode::FallthroughReturn => builder.ins().fallthrough_return(&state.stack),
+            };
         }
     }
 
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index 95667d870d..21b757f9c8 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -59,7 +59,8 @@ mod state;
 mod translation_utils;
 
 pub use environ::{
-    DummyEnvironment, FuncEnvironment, GlobalVariable, ModuleEnvironment, WasmError, WasmResult,
+    DummyEnvironment, FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmError,
+    WasmResult,
 };
 pub use func_translator::FuncTranslator;
 pub use module_translator::translate_module;
diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs
index f7cc0e678e..be9c836463 100644
--- a/lib/wasm/tests/wasm_testsuite.rs
+++ b/lib/wasm/tests/wasm_testsuite.rs
@@ -6,9 +6,9 @@ extern crate wabt;
 
 use cranelift_codegen::isa;
 use cranelift_codegen::print_errors::pretty_verifier_error;
-use cranelift_codegen::settings::{self, Configurable, Flags};
+use cranelift_codegen::settings::{self, Flags};
 use cranelift_codegen::verifier;
-use cranelift_wasm::{translate_module, DummyEnvironment};
+use cranelift_wasm::{translate_module, DummyEnvironment, ReturnMode};
 use std::fs;
 use std::fs::File;
 use std::io;
@@ -35,16 +35,18 @@ fn testsuite() {
     let flags = Flags::new(settings::builder());
     for path in paths {
         let path = path.path();
-        handle_module(&path, &flags);
+        handle_module(&path, &flags, ReturnMode::NormalReturns);
     }
 }
 
 #[test]
-fn return_at_end() {
-    let mut flag_builder = settings::builder();
-    flag_builder.enable("return_at_end").unwrap();
-    let flags = Flags::new(flag_builder);
-    handle_module(Path::new("../../wasmtests/return_at_end.wat"), &flags);
+fn use_fallthrough_return() {
+    let flags = Flags::new(settings::builder());
+    handle_module(
+        Path::new("../../wasmtests/use_fallthrough_return.wat"),
+        &flags,
+        ReturnMode::FallthroughReturn,
+    );
 }
 
 fn read_file(path: &Path) -> io::Result> {
@@ -54,7 +56,7 @@ fn read_file(path: &Path) -> io::Result> {
     Ok(buf)
 }
 
-fn handle_module(path: &Path, flags: &Flags) {
+fn handle_module(path: &Path, flags: &Flags, return_mode: ReturnMode) {
     let data = match path.extension() {
         None => {
             panic!("the file extension is not wasm or wat");
@@ -73,7 +75,9 @@ fn handle_module(path: &Path, flags: &Flags) {
             None | Some(&_) => panic!("the file extension for {:?} is not wasm or wat", path),
         },
     };
-    let mut dummy_environ = DummyEnvironment::with_triple_flags(triple!("riscv64"), flags.clone());
+    let mut dummy_environ =
+        DummyEnvironment::with_triple_flags(triple!("riscv64"), flags.clone(), return_mode);
+
     translate_module(&data, &mut dummy_environ).unwrap();
 
     let isa = isa::lookup(dummy_environ.info.triple)

From 652e526bb631376ad0bef490e999be3747d15ce7 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 5 Oct 2018 09:12:47 -0700
Subject: [PATCH 2122/3084] Put TargetIsa's emit_inst under a "testing_hooks"
 feature. (#531)

* Put TargetIsa's emit_inst under a "testing_hooks" feature.

In practice, TargetIsa's emit_inst pulls in its own instantiation
of the target-specifi `emit_inst` functions, which can be quite
large, and LTO doesn't eliminate them because they're held live
by TargetIsa's vtable.

Fortunately, this function is only used by tests, so we can put
it behind a feature flag.

Fixes #530.

* Add comments for `emit_inst` to clarify its purpose.
---
 lib/codegen/Cargo.toml           | 3 +++
 lib/codegen/src/isa/arm32/mod.rs | 5 ++++-
 lib/codegen/src/isa/arm64/mod.rs | 5 ++++-
 lib/codegen/src/isa/mod.rs       | 6 ++++--
 lib/codegen/src/isa/riscv/mod.rs | 5 ++++-
 lib/codegen/src/isa/x86/mod.rs   | 5 ++++-
 lib/filetests/Cargo.toml         | 2 +-
 7 files changed, 24 insertions(+), 7 deletions(-)

diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index b1757f95b5..7c1d2d0e9a 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -34,6 +34,9 @@ cranelift-codegen-meta = { path = "meta", version = "0.22.0" }
 default = ["std"]
 std = ["cranelift-entity/std", "cranelift-bforest/std", "target-lexicon/std"]
 core = ["hashmap_core"]
+# This enables some additional functions useful for writing tests, but which
+# can significantly increase the size of the library.
+testing_hooks = []
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/codegen/src/isa/arm32/mod.rs b/lib/codegen/src/isa/arm32/mod.rs
index 3606c18676..cee753a5f0 100644
--- a/lib/codegen/src/isa/arm32/mod.rs
+++ b/lib/codegen/src/isa/arm32/mod.rs
@@ -7,7 +7,9 @@ mod registers;
 pub mod settings;
 
 use super::super::settings as shared_settings;
-use binemit::{emit_function, CodeSink, MemoryCodeSink};
+#[cfg(feature = "testing_hooks")]
+use binemit::CodeSink;
+use binemit::{emit_function, MemoryCodeSink};
 use ir;
 use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings};
 use isa::Builder as IsaBuilder;
@@ -111,6 +113,7 @@ impl TargetIsa for Isa {
         abi::allocatable_registers(func)
     }
 
+    #[cfg(feature = "testing_hooks")]
     fn emit_inst(
         &self,
         func: &ir::Function,
diff --git a/lib/codegen/src/isa/arm64/mod.rs b/lib/codegen/src/isa/arm64/mod.rs
index 96e62a9e37..830a109b52 100644
--- a/lib/codegen/src/isa/arm64/mod.rs
+++ b/lib/codegen/src/isa/arm64/mod.rs
@@ -7,7 +7,9 @@ mod registers;
 pub mod settings;
 
 use super::super::settings as shared_settings;
-use binemit::{emit_function, CodeSink, MemoryCodeSink};
+#[cfg(feature = "testing_hooks")]
+use binemit::CodeSink;
+use binemit::{emit_function, MemoryCodeSink};
 use ir;
 use isa::enc_tables::{lookup_enclist, Encodings};
 use isa::Builder as IsaBuilder;
@@ -98,6 +100,7 @@ impl TargetIsa for Isa {
         abi::allocatable_registers(func)
     }
 
+    #[cfg(feature = "testing_hooks")]
     fn emit_inst(
         &self,
         func: &ir::Function,
diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs
index 33e78c96da..c3766346b7 100644
--- a/lib/codegen/src/isa/mod.rs
+++ b/lib/codegen/src/isa/mod.rs
@@ -307,6 +307,10 @@ pub trait TargetIsa: fmt::Display {
     ///
     /// Note that this will call `put*` methods on the `sink` trait object via its vtable which
     /// is not the fastest way of emitting code.
+    ///
+    /// This function is under the "testing_hooks" feature, and is only suitable for use by
+    /// test harnesses. It increases code size, and is inefficient.
+    #[cfg(feature = "testing_hooks")]
     fn emit_inst(
         &self,
         func: &ir::Function,
@@ -316,7 +320,5 @@ pub trait TargetIsa: fmt::Display {
     );
 
     /// Emit a whole function into memory.
-    ///
-    /// This is more performant than calling `emit_inst` for each instruction.
     fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut binemit::MemoryCodeSink);
 }
diff --git a/lib/codegen/src/isa/riscv/mod.rs b/lib/codegen/src/isa/riscv/mod.rs
index 1a09f89bc1..393219571f 100644
--- a/lib/codegen/src/isa/riscv/mod.rs
+++ b/lib/codegen/src/isa/riscv/mod.rs
@@ -7,7 +7,9 @@ mod registers;
 pub mod settings;
 
 use super::super::settings as shared_settings;
-use binemit::{emit_function, CodeSink, MemoryCodeSink};
+#[cfg(feature = "testing_hooks")]
+use binemit::CodeSink;
+use binemit::{emit_function, MemoryCodeSink};
 use ir;
 use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings};
 use isa::Builder as IsaBuilder;
@@ -105,6 +107,7 @@ impl TargetIsa for Isa {
         abi::allocatable_registers(func, &self.isa_flags)
     }
 
+    #[cfg(feature = "testing_hooks")]
     fn emit_inst(
         &self,
         func: &ir::Function,
diff --git a/lib/codegen/src/isa/x86/mod.rs b/lib/codegen/src/isa/x86/mod.rs
index 0546c648cd..1dfa62bf61 100644
--- a/lib/codegen/src/isa/x86/mod.rs
+++ b/lib/codegen/src/isa/x86/mod.rs
@@ -7,7 +7,9 @@ mod registers;
 pub mod settings;
 
 use super::super::settings as shared_settings;
-use binemit::{emit_function, CodeSink, MemoryCodeSink};
+#[cfg(feature = "testing_hooks")]
+use binemit::CodeSink;
+use binemit::{emit_function, MemoryCodeSink};
 use ir;
 use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings};
 use isa::Builder as IsaBuilder;
@@ -115,6 +117,7 @@ impl TargetIsa for Isa {
         abi::allocatable_registers(func, &self.triple)
     }
 
+    #[cfg(feature = "testing_hooks")]
     fn emit_inst(
         &self,
         func: &ir::Function,
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 59c4309563..1ed2daf9e0 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.22.0" }
+cranelift-codegen = { path = "../codegen", version = "0.22.0", features = ["testing_hooks"] }
 cranelift-reader = { path = "../reader", version = "0.22.0" }
 file-per-thread-logger = "0.1.1"
 filecheck = "0.4.0"

From 9a1e966156c0649ff1dca2761baf70299b203466 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 5 Oct 2018 13:46:38 -0700
Subject: [PATCH 2123/3084] Fix an unused import warning.

---
 lib/codegen/src/verifier/mod.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index ba396a8b10..49deccd777 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -70,7 +70,7 @@ use ir::{
 };
 use isa::TargetIsa;
 use iterators::IteratorExtras;
-use settings::{Flags, FlagsOrIsa};
+use settings::FlagsOrIsa;
 use std::cmp::Ordering;
 use std::collections::BTreeSet;
 use std::fmt::{self, Display, Formatter, Write};

From 17a96319811558f5a8c6a0724fceb1b669224891 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 5 Oct 2018 16:40:50 -0700
Subject: [PATCH 2124/3084] Use more `Self` keywords instead of repeating the
 type name.

---
 lib/bforest/src/map.rs                |  8 ++------
 lib/bforest/src/set.rs                |  8 ++------
 lib/codegen/meta/src/cdsl/types.rs    |  4 ++--
 lib/codegen/meta/src/srcgen.rs        |  4 ++--
 lib/codegen/src/binemit/memorysink.rs |  8 ++------
 lib/codegen/src/cfg_printer.rs        |  4 ++--
 lib/codegen/src/cursor.rs             |  8 ++++----
 lib/codegen/src/ir/builder.rs         |  8 ++++----
 lib/codegen/src/ir/layout.rs          |  4 ++--
 lib/codegen/src/regalloc/liverange.rs |  7 ++-----
 lib/codegen/src/settings.rs           |  2 +-
 lib/codegen/src/verifier/mod.rs       |  4 ++--
 lib/frontend/src/frontend.rs          | 11 ++++-------
 lib/reader/src/lexer.rs               |  4 ++--
 lib/reader/src/parser.rs              |  8 ++++----
 lib/reader/src/testcommand.rs         |  6 +++---
 16 files changed, 40 insertions(+), 58 deletions(-)

diff --git a/lib/bforest/src/map.rs b/lib/bforest/src/map.rs
index 636cee2df5..97ab7e6d90 100644
--- a/lib/bforest/src/map.rs
+++ b/lib/bforest/src/map.rs
@@ -266,12 +266,8 @@ where
     C: Comparator,
 {
     /// Create a cursor with a default (off-the-end) location.
-    fn new(
-        container: &'a mut Map,
-        forest: &'a mut MapForest,
-        comp: &'a C,
-    ) -> MapCursor<'a, K, V, C> {
-        MapCursor {
+    fn new(container: &'a mut Map, forest: &'a mut MapForest, comp: &'a C) -> Self {
+        Self {
             root: &mut container.root,
             pool: &mut forest.nodes,
             comp,
diff --git a/lib/bforest/src/set.rs b/lib/bforest/src/set.rs
index cf4acdf9e3..f950e1e3cb 100644
--- a/lib/bforest/src/set.rs
+++ b/lib/bforest/src/set.rs
@@ -207,12 +207,8 @@ where
     C: Comparator,
 {
     /// Create a cursor with a default (invalid) location.
-    fn new(
-        container: &'a mut Set,
-        forest: &'a mut SetForest,
-        comp: &'a C,
-    ) -> SetCursor<'a, K, C> {
-        SetCursor {
+    fn new(container: &'a mut Set, forest: &'a mut SetForest, comp: &'a C) -> Self {
+        Self {
             root: &mut container.root,
             pool: &mut forest.nodes,
             comp,
diff --git a/lib/codegen/meta/src/cdsl/types.rs b/lib/codegen/meta/src/cdsl/types.rs
index 90438e9179..9d2e9a16dc 100644
--- a/lib/codegen/meta/src/cdsl/types.rs
+++ b/lib/codegen/meta/src/cdsl/types.rs
@@ -300,8 +300,8 @@ pub struct VectorType {
 
 impl VectorType {
     /// Initialize a new integer type with `n` bits.
-    pub fn new(base: LaneType, lanes: u64) -> VectorType {
-        VectorType { base, lanes }
+    pub fn new(base: LaneType, lanes: u64) -> Self {
+        Self { base, lanes }
     }
 
     /// Return a string containing the documentation comment for this vector type.
diff --git a/lib/codegen/meta/src/srcgen.rs b/lib/codegen/meta/src/srcgen.rs
index 53a4dc5f75..2e2babe512 100644
--- a/lib/codegen/meta/src/srcgen.rs
+++ b/lib/codegen/meta/src/srcgen.rs
@@ -38,8 +38,8 @@ pub struct Formatter {
 impl Formatter {
     /// Source code formatter class. Used to collect source code to be written
     /// to a file, and keep track of indentation.
-    pub fn new() -> Formatter {
-        Formatter {
+    pub fn new() -> Self {
+        Self {
             indent: 0,
             lines: Vec::new(),
         }
diff --git a/lib/codegen/src/binemit/memorysink.rs b/lib/codegen/src/binemit/memorysink.rs
index c7ec60de89..1e8a6ab15f 100644
--- a/lib/codegen/src/binemit/memorysink.rs
+++ b/lib/codegen/src/binemit/memorysink.rs
@@ -43,12 +43,8 @@ impl<'a> MemoryCodeSink<'a> {
     ///
     /// This function is unsafe since `MemoryCodeSink` does not perform bounds checking on the
     /// memory buffer, and it can't guarantee that the `data` pointer is valid.
-    pub unsafe fn new<'sink>(
-        data: *mut u8,
-        relocs: &'sink mut RelocSink,
-        traps: &'sink mut TrapSink,
-    ) -> MemoryCodeSink<'sink> {
-        MemoryCodeSink {
+    pub unsafe fn new(data: *mut u8, relocs: &'a mut RelocSink, traps: &'a mut TrapSink) -> Self {
+        Self {
             data,
             offset: 0,
             code_size: 0,
diff --git a/lib/codegen/src/cfg_printer.rs b/lib/codegen/src/cfg_printer.rs
index ebc2058489..382c83c658 100644
--- a/lib/codegen/src/cfg_printer.rs
+++ b/lib/codegen/src/cfg_printer.rs
@@ -15,8 +15,8 @@ pub struct CFGPrinter<'a> {
 /// A utility for pretty-printing the CFG of a `Function`.
 impl<'a> CFGPrinter<'a> {
     /// Create a new CFGPrinter.
-    pub fn new(func: &'a Function) -> CFGPrinter<'a> {
-        CFGPrinter {
+    pub fn new(func: &'a Function) -> Self {
+        Self {
             func,
             cfg: ControlFlowGraph::with_function(func),
         }
diff --git a/lib/codegen/src/cursor.rs b/lib/codegen/src/cursor.rs
index 6d475331ad..daa23308f2 100644
--- a/lib/codegen/src/cursor.rs
+++ b/lib/codegen/src/cursor.rs
@@ -580,8 +580,8 @@ pub struct FuncCursor<'f> {
 
 impl<'f> FuncCursor<'f> {
     /// Create a new `FuncCursor` pointing nowhere.
-    pub fn new(func: &'f mut ir::Function) -> FuncCursor<'f> {
-        FuncCursor {
+    pub fn new(func: &'f mut ir::Function) -> Self {
+        Self {
             pos: CursorPosition::Nowhere,
             srcloc: Default::default(),
             func,
@@ -662,8 +662,8 @@ pub struct EncCursor<'f> {
 
 impl<'f> EncCursor<'f> {
     /// Create a new `EncCursor` pointing nowhere.
-    pub fn new(func: &'f mut ir::Function, isa: &'f TargetIsa) -> EncCursor<'f> {
-        EncCursor {
+    pub fn new(func: &'f mut ir::Function, isa: &'f TargetIsa) -> Self {
+        Self {
             pos: CursorPosition::Nowhere,
             srcloc: Default::default(),
             built_inst: None,
diff --git a/lib/codegen/src/ir/builder.rs b/lib/codegen/src/ir/builder.rs
index 9128092f67..4e3866ad17 100644
--- a/lib/codegen/src/ir/builder.rs
+++ b/lib/codegen/src/ir/builder.rs
@@ -74,8 +74,8 @@ pub struct InsertBuilder<'f, IIB: InstInserterBase<'f>> {
 impl<'f, IIB: InstInserterBase<'f>> InsertBuilder<'f, IIB> {
     /// Create a new builder which inserts instructions at `pos`.
     /// The `dfg` and `pos.layout` references should be from the same `Function`.
-    pub fn new(inserter: IIB) -> InsertBuilder<'f, IIB> {
-        InsertBuilder {
+    pub fn new(inserter: IIB) -> Self {
+        Self {
             inserter,
             unused: PhantomData,
         }
@@ -185,8 +185,8 @@ pub struct ReplaceBuilder<'f> {
 
 impl<'f> ReplaceBuilder<'f> {
     /// Create a `ReplaceBuilder` that will overwrite `inst`.
-    pub fn new(dfg: &'f mut DataFlowGraph, inst: Inst) -> ReplaceBuilder {
-        ReplaceBuilder { dfg, inst }
+    pub fn new(dfg: &'f mut DataFlowGraph, inst: Inst) -> Self {
+        Self { dfg, inst }
     }
 }
 
diff --git a/lib/codegen/src/ir/layout.rs b/lib/codegen/src/ir/layout.rs
index a251822847..f598246b84 100644
--- a/lib/codegen/src/ir/layout.rs
+++ b/lib/codegen/src/ir/layout.rs
@@ -782,8 +782,8 @@ mod tests {
     impl<'f> LayoutCursor<'f> {
         /// Create a new `LayoutCursor` for `layout`.
         /// The cursor holds a mutable reference to `layout` for its entire lifetime.
-        pub fn new(layout: &'f mut Layout) -> LayoutCursor<'f> {
-            LayoutCursor {
+        pub fn new(layout: &'f mut Layout) -> Self {
+            Self {
                 layout,
                 pos: CursorPosition::Nowhere,
             }
diff --git a/lib/codegen/src/regalloc/liverange.rs b/lib/codegen/src/regalloc/liverange.rs
index cc6a985641..7e56b88fb9 100644
--- a/lib/codegen/src/regalloc/liverange.rs
+++ b/lib/codegen/src/regalloc/liverange.rs
@@ -188,11 +188,8 @@ pub struct LiveRangeContext<'a, PO: 'a + ProgramOrder> {
 
 impl<'a, PO: ProgramOrder> LiveRangeContext<'a, PO> {
     /// Make a new context.
-    pub fn new(
-        order: &'a PO,
-        forest: &'a bforest::MapForest,
-    ) -> LiveRangeContext<'a, PO> {
-        LiveRangeContext { order, forest }
+    pub fn new(order: &'a PO, forest: &'a bforest::MapForest) -> Self {
+        Self { order, forest }
     }
 }
 
diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs
index c9374d9e47..3f7d684a87 100644
--- a/lib/codegen/src/settings.rs
+++ b/lib/codegen/src/settings.rs
@@ -182,7 +182,7 @@ impl<'a> PredicateView<'a> {
     /// Create a new view of a precomputed predicate vector.
     ///
     /// See the `predicate_view()` method on the various `Flags` types defined for each ISA.
-    pub fn new(bits: &'a [u8]) -> PredicateView {
+    pub fn new(bits: &'a [u8]) -> Self {
         PredicateView(bits)
     }
 
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 49deccd777..0219b18db4 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -311,10 +311,10 @@ struct Verifier<'a> {
 }
 
 impl<'a> Verifier<'a> {
-    pub fn new(func: &'a Function, fisa: FlagsOrIsa<'a>) -> Verifier<'a> {
+    pub fn new(func: &'a Function, fisa: FlagsOrIsa<'a>) -> Self {
         let expected_cfg = ControlFlowGraph::with_function(func);
         let expected_domtree = DominatorTree::with_function(func, &expected_cfg);
-        Verifier {
+        Self {
             func,
             expected_cfg,
             expected_domtree,
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index e637d6bf18..716f5cf3de 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -103,8 +103,8 @@ pub struct FuncInstBuilder<'short, 'long: 'short> {
 }
 
 impl<'short, 'long> FuncInstBuilder<'short, 'long> {
-    fn new<'s, 'l>(builder: &'s mut FunctionBuilder<'l>, ebb: Ebb) -> FuncInstBuilder<'s, 'l> {
-        FuncInstBuilder { builder, ebb }
+    fn new(builder: &'short mut FunctionBuilder<'long>, ebb: Ebb) -> Self {
+        Self { builder, ebb }
     }
 }
 
@@ -210,12 +210,9 @@ impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
 impl<'a> FunctionBuilder<'a> {
     /// Creates a new FunctionBuilder structure that will operate on a `Function` using a
     /// `FunctionBuilderContext`.
-    pub fn new(
-        func: &'a mut Function,
-        func_ctx: &'a mut FunctionBuilderContext,
-    ) -> FunctionBuilder<'a> {
+    pub fn new(func: &'a mut Function, func_ctx: &'a mut FunctionBuilderContext) -> Self {
         debug_assert!(func_ctx.is_empty());
-        FunctionBuilder {
+        Self {
             func,
             srcloc: Default::default(),
             func_ctx,
diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs
index c1d9a81d93..5ac84ab773 100644
--- a/lib/reader/src/lexer.rs
+++ b/lib/reader/src/lexer.rs
@@ -128,8 +128,8 @@ pub struct Lexer<'a> {
 }
 
 impl<'a> Lexer<'a> {
-    pub fn new(s: &'a str) -> Lexer {
-        let mut lex = Lexer {
+    pub fn new(s: &'a str) -> Self {
+        let mut lex = Self {
             source: s,
             chars: s.char_indices(),
             lookahead: None,
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 4840387e5a..8951fb8dab 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -117,8 +117,8 @@ struct Context<'a> {
 }
 
 impl<'a> Context<'a> {
-    fn new(f: Function, unique_isa: Option<&'a TargetIsa>) -> Context<'a> {
-        Context {
+    fn new(f: Function, unique_isa: Option<&'a TargetIsa>) -> Self {
+        Self {
             function: f,
             map: SourceMap::new(),
             unique_isa,
@@ -309,8 +309,8 @@ impl<'a> Context<'a> {
 
 impl<'a> Parser<'a> {
     /// Create a new `Parser` which reads `text`. The referenced text must outlive the parser.
-    pub fn new(text: &'a str) -> Parser {
-        Parser {
+    pub fn new(text: &'a str) -> Self {
+        Self {
             lex: Lexer::new(text),
             lex_error: None,
             lookahead: None,
diff --git a/lib/reader/src/testcommand.rs b/lib/reader/src/testcommand.rs
index 229957b17e..17896a070c 100644
--- a/lib/reader/src/testcommand.rs
+++ b/lib/reader/src/testcommand.rs
@@ -35,10 +35,10 @@ pub enum TestOption<'a> {
 impl<'a> TestCommand<'a> {
     /// Create a new TestCommand by parsing `s`.
     /// The returned command contains references into `s`.
-    pub fn new(s: &'a str) -> TestCommand<'a> {
+    pub fn new(s: &'a str) -> Self {
         let mut parts = s.split_whitespace();
         let cmd = parts.next().unwrap_or("");
-        TestCommand {
+        Self {
             command: cmd,
             options: parts
                 .filter(|s| !s.is_empty())
@@ -61,7 +61,7 @@ impl<'a> Display for TestCommand<'a> {
 impl<'a> TestOption<'a> {
     /// Create a new TestOption by parsing `s`.
     /// The returned option contains references into `s`.
-    pub fn new(s: &'a str) -> TestOption<'a> {
+    pub fn new(s: &'a str) -> Self {
         match s.find('=') {
             None => TestOption::Flag(s),
             Some(p) => TestOption::Value(&s[0..p], &s[p + 1..]),

From 3861dcf38cf16f384b6775c7292fa28dbe7eba62 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 5 Oct 2018 16:44:13 -0700
Subject: [PATCH 2125/3084] Move URLs out of line and wrap long lines.

---
 cranelift/FUZZING.md | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/cranelift/FUZZING.md b/cranelift/FUZZING.md
index 55643a73b9..a2e8eb7d51 100644
--- a/cranelift/FUZZING.md
+++ b/cranelift/FUZZING.md
@@ -1,6 +1,13 @@
 # Fuzzing
 
-This document describes how to fuzz cranelift with [`cargo-fuzz`](https://github.com/rust-fuzz/cargo-fuzz). The fuzz targets use `wasm-opt` from [`binaryen-rs`](https://github.com/pepyakin/binaryen-rs) to generate valid WebAssembly modules from the fuzzed input supplied by `cargo-fuzz` (via [libfuzzer](http://llvm.org/docs/LibFuzzer.html)). In this scheme coverage feedback from both cranelift and the `wasm-opt` input generation code is used to inform the fuzzer.
+This document describes how to fuzz cranelift with [`cargo-fuzz`]. The fuzz targets use `wasm-opt`
+from [`binaryen-rs`] to generate valid WebAssembly modules from the fuzzed input supplied by
+`cargo-fuzz` (via [libfuzzer]). In this scheme coverage feedback from both cranelift and the
+`wasm-opt` input generation code is used to inform the fuzzer.
+
+[`cargo-fuzz`]: https://github.com/rust-fuzz/cargo-fuzz
+[`binaryen-rs`]: https://github.com/pepyakin/binaryen-rs
+[libfuzzer]: http://llvm.org/docs/LibFuzzer.html
 
 # Usage
 

From be9d1c6d2797c38fa19674cda83abe19e3b3e5ed Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 5 Oct 2018 16:50:54 -0700
Subject: [PATCH 2126/3084] Add Emacs backup and autosave files to .gitignore.

---
 cranelift/.gitignore | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/cranelift/.gitignore b/cranelift/.gitignore
index 1714fd39af..98cb9403a1 100644
--- a/cranelift/.gitignore
+++ b/cranelift/.gitignore
@@ -11,3 +11,5 @@ cranelift.dbg*
 .mypy_cache
 rusty-tags.*
 docs/_build
+*~
+\#*\#

From 8bd0cf25336a085a3de2c87460b5cee891aea6aa Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 5 Oct 2018 16:52:13 -0700
Subject: [PATCH 2127/3084] Update to raw-cpuid 6.0.0.

---
 lib/native/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index da585be597..58a5234d59 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -13,7 +13,7 @@ cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features
 target-lexicon = { version = "0.0.3", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
-raw-cpuid = "5.0.0"
+raw-cpuid = "6.0.0"
 
 [features]
 default = ["std"]

From 7f90414ae1b816ffa41a4cd43be1e863e0563a29 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 5 Oct 2018 17:16:51 -0700
Subject: [PATCH 2128/3084] Update no_std support.

---
 lib/codegen/src/lib.rs      | 4 +++-
 lib/codegen/src/settings.rs | 5 +++++
 lib/wasm/Cargo.toml         | 2 +-
 3 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index ec924dc6bf..24c381a6eb 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -48,6 +48,8 @@
 // Turns on no_std and alloc features if std is not available.
 #![cfg_attr(not(feature = "std"), no_std)]
 #![cfg_attr(not(feature = "std"), feature(alloc))]
+// TODO: Remove this workaround once https://github.com/rust-lang/rust/issues/27747 is done.
+#![cfg_attr(not(feature = "std"), feature(slice_concat_ext))]
 
 #[cfg(not(feature = "std"))]
 #[macro_use]
@@ -119,7 +121,7 @@ pub use result::{CodegenError, CodegenResult};
 /// This replaces `std` in builds with `core`.
 #[cfg(not(feature = "std"))]
 mod std {
-    pub use alloc::{boxed, string, vec};
+    pub use alloc::{boxed, slice, string, vec};
     pub use core::*;
     pub mod collections {
         #[allow(unused_extern_crates)]
diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs
index 3f7d684a87..cd135eba4b 100644
--- a/lib/codegen/src/settings.rs
+++ b/lib/codegen/src/settings.rs
@@ -25,6 +25,11 @@ use isa::TargetIsa;
 use std::boxed::Box;
 use std::fmt;
 use std::str;
+use std::string::{String, ToString};
+
+// TODO: Remove this workaround once https://github.com/rust-lang/rust/issues/27747 is done.
+#[cfg(not(feature = "std"))]
+use std::slice::SliceConcatExt;
 
 /// A string-based configurator for settings groups.
 ///
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 2469716d3c..d298060f76 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -10,7 +10,7 @@ readme = "README.md"
 keywords = ["webassembly", "wasm"]
 
 [dependencies]
-wasmparser = { version = "0.19.0", default-features = false }
+wasmparser = { version = "0.19.1", default-features = false }
 cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
 cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
 cranelift-frontend = { path = "../frontend", version = "0.22.0", default-features = false }

From c2069762efd805be14f895afae0e1426ddf6151b Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 10 Oct 2018 00:38:16 +0200
Subject: [PATCH 2129/3084] Declare br_table otherwise ebb as predecessor
 (#551)

* Declare br_table otherwise ebb as predecessor (fixes #545)
---
 lib/frontend/src/frontend.rs | 12 +++++-
 lib/simplejit/tests/basic.rs | 73 ++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+), 2 deletions(-)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 716f5cf3de..be178d554c 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -141,7 +141,10 @@ impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
                 None => {
                     // branch_destination() doesn't detect jump_tables
                     // If jump table we declare all entries successor
-                    if let InstructionData::BranchTable { table, .. } = data {
+                    if let InstructionData::BranchTable {
+                        table, destination, ..
+                    } = data
+                    {
                         // Unlike all other jumps/branches, jump tables are
                         // capable of having the same successor appear
                         // multiple times, so we must deduplicate.
@@ -159,8 +162,13 @@ impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
                                 *dest_ebb,
                                 self.builder.position.basic_block.unwrap(),
                                 inst,
-                            )
+                            );
                         }
+                        self.builder.func_ctx.ssa.declare_ebb_predecessor(
+                            destination,
+                            self.builder.position.basic_block.unwrap(),
+                            inst,
+                        );
                     }
                 }
             }
diff --git a/lib/simplejit/tests/basic.rs b/lib/simplejit/tests/basic.rs
index 3c79e55492..e76282e74e 100644
--- a/lib/simplejit/tests/basic.rs
+++ b/lib/simplejit/tests/basic.rs
@@ -77,3 +77,76 @@ fn panic_on_define_after_finalize() {
     module.finalize_definitions();
     define_simple_function(&mut module);
 }
+
+#[test]
+fn switch_error() {
+    use cranelift_codegen::settings;
+
+    let sig = Signature {
+        params: vec![AbiParam::new(types::I32)],
+        returns: vec![AbiParam::new(types::I32)],
+        call_conv: CallConv::SystemV,
+    };
+
+    let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
+
+    let mut func_ctx = FunctionBuilderContext::new();
+    {
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut func, &mut func_ctx);
+        let start = bcx.create_ebb();
+        let bb0 = bcx.create_ebb();
+        let bb1 = bcx.create_ebb();
+        let bb2 = bcx.create_ebb();
+        let bb3 = bcx.create_ebb();
+        println!("{} {} {} {} {}", start, bb0, bb1, bb2, bb3);
+
+        bcx.declare_var(Variable::new(0), types::I32);
+        bcx.declare_var(Variable::new(1), types::I32);
+        let in_val = bcx.append_ebb_param(start, types::I32);
+        bcx.switch_to_block(start);
+        bcx.def_var(Variable::new(0), in_val);
+        bcx.ins().jump(bb0, &[]);
+
+        bcx.switch_to_block(bb0);
+        let discr = bcx.use_var(Variable::new(0));
+        let mut switch = cranelift_frontend::Switch::new();
+        for &(index, bb) in &[
+            (9, bb1),
+            (13, bb1),
+            (10, bb1),
+            (92, bb1),
+            (39, bb1),
+            (34, bb1),
+        ] {
+            switch.set_entry(index, bb);
+        }
+        switch.emit(&mut bcx, discr, bb2);
+
+        bcx.switch_to_block(bb1);
+        let v = bcx.use_var(Variable::new(0));
+        bcx.def_var(Variable::new(1), v);
+        bcx.ins().jump(bb3, &[]);
+
+        bcx.switch_to_block(bb2);
+        let v = bcx.use_var(Variable::new(0));
+        bcx.def_var(Variable::new(1), v);
+        bcx.ins().jump(bb3, &[]);
+
+        bcx.switch_to_block(bb3);
+        let r = bcx.use_var(Variable::new(1));
+        bcx.ins().return_(&[r]);
+
+        bcx.seal_all_blocks();
+        bcx.finalize();
+    }
+
+    let flags = settings::Flags::new(settings::builder());
+    match cranelift_codegen::verify_function(&func, &flags) {
+        Ok(_) => {}
+        Err(err) => {
+            let pretty_error =
+                cranelift_codegen::print_errors::pretty_verifier_error(&func, None, None, err);
+            panic!("pretty_error:\n{}", pretty_error);
+        }
+    }
+}

From 9d6821d6d999e62aae49526ef0eca66387c29c37 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 16 Oct 2018 00:43:38 +0200
Subject: [PATCH 2130/3084] Fix #335: Introduce variable size recipes and
 remove GPR_SAFE reg classes (#552)

* Rename size to base_size and introduce a compute_size function;

* Add infra to inspect in/outs registers when computing the size of an instruction;

* Remove the GPR_SAFE_DEREF and GPR_ZERO_DEREF_SAFE register classes on x86 (fixes #335);
---
 .../regalloc/gpr-deref-safe-335.clif          |  44 ++
 lib/codegen/meta-python/cdsl/isa.py           |  30 +-
 lib/codegen/meta-python/gen_encoding.py       |   3 +-
 lib/codegen/meta-python/isa/riscv/recipes.py  |  34 +-
 lib/codegen/meta-python/isa/x86/recipes.py    | 470 +++++++++++-------
 lib/codegen/meta-python/isa/x86/registers.py  |   7 -
 lib/codegen/src/binemit/relaxation.rs         |  15 +-
 lib/codegen/src/binemit/shrink.rs             |   6 +-
 lib/codegen/src/context.rs                    |   1 -
 lib/codegen/src/ir/function.rs                |  21 +-
 lib/codegen/src/ir/stackslot.rs               |   2 +-
 lib/codegen/src/isa/encoding.rs               |  33 +-
 lib/codegen/src/isa/mod.rs                    |   4 +-
 lib/codegen/src/isa/riscv/enc_tables.rs       |   2 +-
 lib/codegen/src/isa/x86/binemit.rs            |   1 +
 lib/codegen/src/isa/x86/enc_tables.rs         |  60 ++-
 lib/filetests/src/test_binemit.rs             |   5 +-
 17 files changed, 498 insertions(+), 240 deletions(-)
 create mode 100644 cranelift/filetests/regalloc/gpr-deref-safe-335.clif

diff --git a/cranelift/filetests/regalloc/gpr-deref-safe-335.clif b/cranelift/filetests/regalloc/gpr-deref-safe-335.clif
new file mode 100644
index 0000000000..8e8b2260cf
--- /dev/null
+++ b/cranelift/filetests/regalloc/gpr-deref-safe-335.clif
@@ -0,0 +1,44 @@
+test regalloc
+target x86_64
+
+function u0:587() fast {
+ebb0:
+    v97 = iconst.i32 0
+    v169 = iconst.i32 0
+    v1729 = iconst.i32 0
+    jump ebb100(v97, v97, v97, v97, v97)
+
+ebb100(v1758: i32, v1784: i32, v1845: i32, v1856: i32, v1870: i32):
+    v1762 = iconst.i32 0
+    v1769 = iconst.i32 0
+    v1774 = iconst.i32 0
+    v1864 = iconst.i32 0
+    v1897 = iconst.i32 0
+    jump ebb102(v1774, v1784, v1845, v1856, v1870, v1758, v1762, v169, v1729, v97, v169, v169, v169, v169)
+
+ebb102(v1785: i32, v1789: i32, v1843: i32, v1854: i32, v1868: i32, v1882: i32, v1890: i32, v1901: i32, v1921: i32, v1933: i32, v2058: i32, v2124: i32, v2236: i32, v2366: i32):
+    v1929 = iconst.i32 0
+    v1943 = iconst.i32 0
+    v1949 = iconst.i32 0
+    jump ebb123(v1897, v1769)
+
+ebb123(v1950: i32, v1979: i32):
+    v1955 = iconst.i32 0
+    brz v1955, ebb125
+    jump ebb122(v1929, v1843, v1864, v2058, v1882, v1897, v1943, v1868, v2124, v1901)
+
+ebb125:
+    v1961 = iadd_imm.i32 v1949, 0
+    v1952 = iconst.i32 0
+    v1962 = iconst.i64 0
+    v1963 = load.i32 v1962
+    brz v1963, ebb123(v1952, v1961)
+    jump ebb127
+
+ebb127:
+    v1966 = iconst.i32 0
+    jump ebb122(v1963, v1966, v1966, v1966, v1966, v1966, v1966, v1966, v1966, v1966)
+
+ebb122(v1967: i32, v1971: i32, v1972: i32, v1978: i32, v2032: i32, v2041: i32, v2053: i32, v2076: i32, v2085: i32, v2096: i32):
+    trap user0
+}
diff --git a/lib/codegen/meta-python/cdsl/isa.py b/lib/codegen/meta-python/cdsl/isa.py
index 5c57c52fbd..90a9610d62 100644
--- a/lib/codegen/meta-python/cdsl/isa.py
+++ b/lib/codegen/meta-python/cdsl/isa.py
@@ -316,7 +316,8 @@ class EncRecipe(object):
     :param name: Short mnemonic name for this recipe.
     :param format: All encoded instructions must have this
             :py:class:`InstructionFormat`.
-    :param size: Number of bytes in the binary encoded instruction.
+    :param base_size: Base number of bytes in the binary encoded instruction.
+    :param compute_size: Function name to use when computing actual size.
     :param ins: Tuple of register constraints for value operands.
     :param outs: Tuple of register constraints for results.
     :param branch_range: `(origin, bits)` range for branches.
@@ -328,22 +329,25 @@ class EncRecipe(object):
 
     def __init__(
             self,
-            name,                 # type: str
-            format,               # type: InstructionFormat
-            size,                 # type: int
-            ins,                  # type: ConstraintSeq
-            outs,                 # type: ConstraintSeq
-            branch_range=None,    # type: BranchRange
-            clobbers_flags=True,  # type: bool
-            instp=None,           # type: PredNode
-            isap=None,            # type: PredNode
-            emit=None             # type: str
+            name,                     # type: str
+            format,                   # type: InstructionFormat
+            base_size,                # type: int
+            ins,                      # type: ConstraintSeq
+            outs,                     # type: ConstraintSeq
+            compute_size=None,        # type: str
+            branch_range=None,        # type: BranchRange
+            clobbers_flags=True,      # type: bool
+            instp=None,               # type: PredNode
+            isap=None,                # type: PredNode
+            emit=None                 # type: str
             ):
         # type: (...) -> None
         self.name = name
         self.format = format
-        assert size >= 0
-        self.size = size
+        assert base_size >= 0
+        self.base_size = base_size
+        self.compute_size = compute_size if compute_size is not None \
+            else 'base_size'
         self.branch_range = branch_range
         self.clobbers_flags = clobbers_flags
         self.instp = instp
diff --git a/lib/codegen/meta-python/gen_encoding.py b/lib/codegen/meta-python/gen_encoding.py
index 59875bf489..6c5d7367ed 100644
--- a/lib/codegen/meta-python/gen_encoding.py
+++ b/lib/codegen/meta-python/gen_encoding.py
@@ -832,7 +832,8 @@ def emit_recipe_sizing(isa, fmt):
         for r in isa.all_recipes:
             fmt.comment('Code size information for recipe {}:'.format(r.name))
             with fmt.indented('RecipeSizing {', '},'):
-                fmt.format('bytes: {},', r.size)
+                fmt.format('base_size: {},', r.base_size)
+                fmt.format('compute_size: {},', r.compute_size)
                 if r.branch_range:
                     fmt.format(
                         'branch_range: '
diff --git a/lib/codegen/meta-python/isa/riscv/recipes.py b/lib/codegen/meta-python/isa/riscv/recipes.py
index 170dd914d3..9808892b3c 100644
--- a/lib/codegen/meta-python/isa/riscv/recipes.py
+++ b/lib/codegen/meta-python/isa/riscv/recipes.py
@@ -93,33 +93,33 @@ def LUI():
 # R-type 32-bit instructions: These are mostly binary arithmetic instructions.
 # The encbits are `opcode[6:2] | (funct3 << 5) | (funct7 << 8)
 R = EncRecipe(
-        'R', Binary, size=4, ins=(GPR, GPR), outs=GPR,
+        'R', Binary, base_size=4, ins=(GPR, GPR), outs=GPR,
         emit='put_r(bits, in_reg0, in_reg1, out_reg0, sink);')
 
 # R-type with an immediate shift amount instead of rs2.
 Rshamt = EncRecipe(
-        'Rshamt', BinaryImm, size=4, ins=GPR, outs=GPR,
+        'Rshamt', BinaryImm, base_size=4, ins=GPR, outs=GPR,
         emit='put_rshamt(bits, in_reg0, imm.into(), out_reg0, sink);')
 
 # R-type encoding of an integer comparison.
 Ricmp = EncRecipe(
-        'Ricmp', IntCompare, size=4, ins=(GPR, GPR), outs=GPR,
+        'Ricmp', IntCompare, base_size=4, ins=(GPR, GPR), outs=GPR,
         emit='put_r(bits, in_reg0, in_reg1, out_reg0, sink);')
 
 Ii = EncRecipe(
-        'Ii', BinaryImm, size=4, ins=GPR, outs=GPR,
+        'Ii', BinaryImm, base_size=4, ins=GPR, outs=GPR,
         instp=IsSignedInt(BinaryImm.imm, 12),
         emit='put_i(bits, in_reg0, imm.into(), out_reg0, sink);')
 
 # I-type instruction with a hardcoded %x0 rs1.
 Iz = EncRecipe(
-        'Iz', UnaryImm, size=4, ins=(), outs=GPR,
+        'Iz', UnaryImm, base_size=4, ins=(), outs=GPR,
         instp=IsSignedInt(UnaryImm.imm, 12),
         emit='put_i(bits, 0, imm.into(), out_reg0, sink);')
 
 # I-type encoding of an integer comparison.
 Iicmp = EncRecipe(
-        'Iicmp', IntCompareImm, size=4, ins=GPR, outs=GPR,
+        'Iicmp', IntCompareImm, base_size=4, ins=GPR, outs=GPR,
         instp=IsSignedInt(IntCompareImm.imm, 12),
         emit='put_i(bits, in_reg0, imm.into(), out_reg0, sink);')
 
@@ -127,7 +127,7 @@ Iicmp = EncRecipe(
 # immediate offset.
 # The variable return values are not encoded.
 Iret = EncRecipe(
-        'Iret', MultiAry, size=4, ins=(), outs=(),
+        'Iret', MultiAry, base_size=4, ins=(), outs=(),
         emit='''
         // Return instructions are always a jalr to %x1.
         // The return address is provided as a special-purpose link argument.
@@ -142,7 +142,7 @@ Iret = EncRecipe(
 
 # I-type encoding for `jalr` as a call_indirect.
 Icall = EncRecipe(
-        'Icall', CallIndirect, size=4, ins=GPR, outs=(),
+        'Icall', CallIndirect, base_size=4, ins=GPR, outs=(),
         emit='''
         // call_indirect instructions are jalr with rd=%x1.
         put_i(
@@ -157,23 +157,23 @@ Icall = EncRecipe(
 
 # Copy of a GPR is implemented as addi x, 0.
 Icopy = EncRecipe(
-        'Icopy', Unary, size=4, ins=GPR, outs=GPR,
+        'Icopy', Unary, base_size=4, ins=GPR, outs=GPR,
         emit='put_i(bits, in_reg0, 0, out_reg0, sink);')
 
 # Same for a GPR regmove.
 Irmov = EncRecipe(
-        'Irmov', RegMove, size=4, ins=GPR, outs=(),
+        'Irmov', RegMove, base_size=4, ins=GPR, outs=(),
         emit='put_i(bits, src, 0, dst, sink);')
 
 # U-type instructions have a 20-bit immediate that targets bits 12-31.
 U = EncRecipe(
-        'U', UnaryImm, size=4, ins=(), outs=GPR,
+        'U', UnaryImm, base_size=4, ins=(), outs=GPR,
         instp=IsSignedInt(UnaryImm.imm, 32, 12),
         emit='put_u(bits, imm.into(), out_reg0, sink);')
 
 # UJ-type unconditional branch instructions.
 UJ = EncRecipe(
-        'UJ', Jump, size=4, ins=(), outs=(), branch_range=(0, 21),
+        'UJ', Jump, base_size=4, ins=(), outs=(), branch_range=(0, 21),
         emit='''
         let dest = i64::from(func.offsets[destination]);
         let disp = dest - i64::from(sink.offset());
@@ -181,7 +181,7 @@ UJ = EncRecipe(
         ''')
 
 UJcall = EncRecipe(
-        'UJcall', Call, size=4, ins=(), outs=(),
+        'UJcall', Call, base_size=4, ins=(), outs=(),
         emit='''
         sink.reloc_external(Reloc::RiscvCall,
                             &func.dfg.ext_funcs[func_ref].name,
@@ -192,7 +192,7 @@ UJcall = EncRecipe(
 
 # SB-type branch instructions.
 SB = EncRecipe(
-        'SB', BranchIcmp, size=4,
+        'SB', BranchIcmp, base_size=4,
         ins=(GPR, GPR), outs=(),
         branch_range=(0, 13),
         emit='''
@@ -203,7 +203,7 @@ SB = EncRecipe(
 
 # SB-type branch instruction with rs2 fixed to zero.
 SBzero = EncRecipe(
-        'SBzero', Branch, size=4,
+        'SBzero', Branch, base_size=4,
         ins=(GPR), outs=(),
         branch_range=(0, 13),
         emit='''
@@ -214,12 +214,12 @@ SBzero = EncRecipe(
 
 # Spill of a GPR.
 GPsp = EncRecipe(
-        'GPsp', Unary, size=4,
+        'GPsp', Unary, base_size=4,
         ins=GPR, outs=Stack(GPR),
         emit='unimplemented!();')
 
 # Fill of a GPR.
 GPfi = EncRecipe(
-        'GPfi', Unary, size=4,
+        'GPfi', Unary, base_size=4,
         ins=Stack(GPR), outs=GPR,
         emit='unimplemented!();')
diff --git a/lib/codegen/meta-python/isa/x86/recipes.py b/lib/codegen/meta-python/isa/x86/recipes.py
index 8a204728d9..4c495807b7 100644
--- a/lib/codegen/meta-python/isa/x86/recipes.py
+++ b/lib/codegen/meta-python/isa/x86/recipes.py
@@ -19,8 +19,8 @@ from base.formats import Ternary, FuncAddr, UnaryGlobalValue
 from base.formats import RegMove, RegSpill, RegFill, CopySpecial
 from base.formats import LoadComplex, StoreComplex
 from base.formats import StackLoad
-from .registers import GPR, ABCD, FPR, GPR_DEREF_SAFE, GPR_ZERO_DEREF_SAFE
-from .registers import GPR8, FPR8, GPR8_DEREF_SAFE, GPR8_ZERO_DEREF_SAFE, FLAG
+from .registers import GPR, ABCD, FPR
+from .registers import GPR8, FPR8, FLAG
 from .registers import StackGPR32, StackFPR32
 from .defs import supported_floatccs
 from .settings import use_sse41
@@ -113,8 +113,6 @@ def replace_put_op(emit, prefix):
 # Register class mapping for no-REX instructions.
 NOREX_MAP = {
         GPR: GPR8,
-        GPR_DEREF_SAFE: GPR8_DEREF_SAFE,
-        GPR_ZERO_DEREF_SAFE: GPR8_ZERO_DEREF_SAFE,
         FPR: FPR8
     }
 
@@ -156,7 +154,7 @@ class TailRecipe:
             self,
             name,                   # type: str
             format,                 # type: InstructionFormat
-            size,                   # type: int
+            base_size,              # type: int
             ins,                    # type: ConstraintSeq
             outs,                   # type: ConstraintSeq
             branch_range=None,      # type: int
@@ -165,12 +163,13 @@ class TailRecipe:
             isap=None,              # type: PredNode
             when_prefixed=None,     # type: TailRecipe
             requires_prefix=False,  # type: bool
-            emit=None               # type: str
+            emit=None,              # type: str
+            compute_size=None       # type: str
             ):
         # type: (...) -> None
         self.name = name
         self.format = format
-        self.size = size
+        self.base_size = base_size
         self.ins = ins
         self.outs = outs
         self.branch_range = branch_range
@@ -180,6 +179,7 @@ class TailRecipe:
         self.when_prefixed = when_prefixed
         self.requires_prefix = requires_prefix
         self.emit = emit
+        self.compute_size = compute_size
 
         # Cached recipes, keyed by name prefix.
         self.recipes = dict()  # type: Dict[str, EncRecipe]
@@ -194,25 +194,26 @@ class TailRecipe:
         rrr = kwargs.get('rrr', 0)
         w = kwargs.get('w', 0)
         name, bits = decode_ops(ops, rrr, w)
-        size = len(ops) + self.size
+        base_size = len(ops) + self.base_size
 
         # All branch ranges are relative to the end of the instruction.
         branch_range = None  # type BranchRange
         if self.branch_range is not None:
-            branch_range = (size, self.branch_range)
+            branch_range = (base_size, self.branch_range)
 
         if name not in self.recipes:
             recipe = EncRecipe(
                 name + self.name,
                 self.format,
-                size,
+                base_size,
                 ins=self.ins,
                 outs=self.outs,
                 branch_range=branch_range,
                 clobbers_flags=self.clobbers_flags,
                 instp=self.instp,
                 isap=self.isap,
-                emit=replace_put_op(self.emit, name))
+                emit=replace_put_op(self.emit, name),
+                compute_size=self.compute_size)
 
             recipe.ins = map_regs_norex(recipe.ins)
             recipe.outs = map_regs_norex(recipe.outs)
@@ -237,25 +238,26 @@ class TailRecipe:
         w = kwargs.get('w', 0)
         name, bits = decode_ops(ops, rrr, w)
         name = 'Rex' + name
-        size = 1 + len(ops) + self.size
+        base_size = 1 + len(ops) + self.base_size
 
         # All branch ranges are relative to the end of the instruction.
         branch_range = None  # type BranchRange
         if self.branch_range is not None:
-            branch_range = (size, self.branch_range)
+            branch_range = (base_size, self.branch_range)
 
         if name not in self.recipes:
             recipe = EncRecipe(
                 name + self.name,
                 self.format,
-                size,
+                base_size,
                 ins=self.ins,
                 outs=self.outs,
                 branch_range=branch_range,
                 clobbers_flags=self.clobbers_flags,
                 instp=self.instp,
                 isap=self.isap,
-                emit=replace_put_op(self.emit, name))
+                emit=replace_put_op(self.emit, name),
+                compute_size=self.compute_size)
             self.recipes[name] = recipe
 
         return (self.recipes[name], bits)
@@ -291,11 +293,11 @@ def valid_scale(iform):
 
 # A null unary instruction that takes a GPR register. Can be used for identity
 # copies and no-op conversions.
-null = EncRecipe('null', Unary, size=0, ins=GPR, outs=0, emit='')
+null = EncRecipe('null', Unary, base_size=0, ins=GPR, outs=0, emit='')
 
 # XX opcode, no ModR/M.
 trap = TailRecipe(
-        'trap', Trap, size=0, ins=(), outs=(),
+        'trap', Trap, base_size=0, ins=(), outs=(),
         emit='''
         sink.trap(code, func.srclocs[inst]);
         PUT_OP(bits, BASE_REX, sink);
@@ -303,7 +305,7 @@ trap = TailRecipe(
 
 # Macro: conditional jump over a ud2.
 trapif = EncRecipe(
-        'trapif', IntCondTrap, size=4, ins=FLAG.rflags, outs=(),
+        'trapif', IntCondTrap, base_size=4, ins=FLAG.rflags, outs=(),
         clobbers_flags=False,
         emit='''
         // Jump over a 2-byte ud2.
@@ -316,7 +318,7 @@ trapif = EncRecipe(
         ''')
 
 trapff = EncRecipe(
-        'trapff', FloatCondTrap, size=4, ins=FLAG.rflags, outs=(),
+        'trapff', FloatCondTrap, base_size=4, ins=FLAG.rflags, outs=(),
         clobbers_flags=False,
         instp=floatccs(FloatCondTrap),
         emit='''
@@ -332,7 +334,7 @@ trapff = EncRecipe(
 
 # XX /r
 rr = TailRecipe(
-        'rr', Binary, size=1, ins=(GPR, GPR), outs=0,
+        'rr', Binary, base_size=1, ins=(GPR, GPR), outs=0,
         emit='''
         PUT_OP(bits, rex2(in_reg0, in_reg1), sink);
         modrm_rr(in_reg0, in_reg1, sink);
@@ -340,7 +342,7 @@ rr = TailRecipe(
 
 # XX /r with operands swapped. (RM form).
 rrx = TailRecipe(
-        'rrx', Binary, size=1, ins=(GPR, GPR), outs=0,
+        'rrx', Binary, base_size=1, ins=(GPR, GPR), outs=0,
         emit='''
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
         modrm_rr(in_reg1, in_reg0, sink);
@@ -348,7 +350,7 @@ rrx = TailRecipe(
 
 # XX /r with FPR ins and outs. A form.
 fa = TailRecipe(
-        'fa', Binary, size=1, ins=(FPR, FPR), outs=0,
+        'fa', Binary, base_size=1, ins=(FPR, FPR), outs=0,
         emit='''
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
         modrm_rr(in_reg1, in_reg0, sink);
@@ -356,7 +358,7 @@ fa = TailRecipe(
 
 # XX /r with FPR ins and outs. A form with input operands swapped.
 fax = TailRecipe(
-        'fax', Binary, size=1, ins=(FPR, FPR), outs=1,
+        'fax', Binary, base_size=1, ins=(FPR, FPR), outs=1,
         emit='''
         PUT_OP(bits, rex2(in_reg0, in_reg1), sink);
         modrm_rr(in_reg0, in_reg1, sink);
@@ -364,7 +366,7 @@ fax = TailRecipe(
 
 # XX /n for a unary operation with extension bits.
 ur = TailRecipe(
-        'ur', Unary, size=1, ins=GPR, outs=0,
+        'ur', Unary, base_size=1, ins=GPR, outs=0,
         emit='''
         PUT_OP(bits, rex1(in_reg0), sink);
         modrm_r_bits(in_reg0, bits, sink);
@@ -373,7 +375,7 @@ ur = TailRecipe(
 # XX /r, but for a unary operator with separate input/output register, like
 # copies. MR form, preserving flags.
 umr = TailRecipe(
-        'umr', Unary, size=1, ins=GPR, outs=GPR,
+        'umr', Unary, base_size=1, ins=GPR, outs=GPR,
         clobbers_flags=False,
         emit='''
         PUT_OP(bits, rex2(out_reg0, in_reg0), sink);
@@ -382,7 +384,7 @@ umr = TailRecipe(
 
 # Same as umr, but with FPR -> GPR registers.
 rfumr = TailRecipe(
-        'rfumr', Unary, size=1, ins=FPR, outs=GPR,
+        'rfumr', Unary, base_size=1, ins=FPR, outs=GPR,
         clobbers_flags=False,
         emit='''
         PUT_OP(bits, rex2(out_reg0, in_reg0), sink);
@@ -392,7 +394,7 @@ rfumr = TailRecipe(
 # XX /r, but for a unary operator with separate input/output register.
 # RM form. Clobbers FLAGS.
 urm = TailRecipe(
-        'urm', Unary, size=1, ins=GPR, outs=GPR,
+        'urm', Unary, base_size=1, ins=GPR, outs=GPR,
         emit='''
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
         modrm_rr(in_reg0, out_reg0, sink);
@@ -400,7 +402,7 @@ urm = TailRecipe(
 
 # XX /r. Same as urm, but doesn't clobber FLAGS.
 urm_noflags = TailRecipe(
-        'urm_noflags', Unary, size=1, ins=GPR, outs=GPR,
+        'urm_noflags', Unary, base_size=1, ins=GPR, outs=GPR,
         clobbers_flags=False,
         emit='''
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
@@ -409,7 +411,7 @@ urm_noflags = TailRecipe(
 
 # XX /r. Same as urm_noflags, but input limited to ABCD.
 urm_noflags_abcd = TailRecipe(
-        'urm_noflags_abcd', Unary, size=1, ins=ABCD, outs=GPR,
+        'urm_noflags_abcd', Unary, base_size=1, ins=ABCD, outs=GPR,
         when_prefixed=urm_noflags,
         clobbers_flags=False,
         emit='''
@@ -419,7 +421,7 @@ urm_noflags_abcd = TailRecipe(
 
 # XX /r, RM form, FPR -> FPR.
 furm = TailRecipe(
-        'furm', Unary, size=1, ins=FPR, outs=FPR,
+        'furm', Unary, base_size=1, ins=FPR, outs=FPR,
         clobbers_flags=False,
         emit='''
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
@@ -428,7 +430,7 @@ furm = TailRecipe(
 
 # XX /r, RM form, GPR -> FPR.
 frurm = TailRecipe(
-        'frurm', Unary, size=1, ins=GPR, outs=FPR,
+        'frurm', Unary, base_size=1, ins=GPR, outs=FPR,
         clobbers_flags=False,
         emit='''
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
@@ -437,7 +439,7 @@ frurm = TailRecipe(
 
 # XX /r, RM form, FPR -> GPR.
 rfurm = TailRecipe(
-        'rfurm', Unary, size=1, ins=FPR, outs=GPR,
+        'rfurm', Unary, base_size=1, ins=FPR, outs=GPR,
         clobbers_flags=False,
         emit='''
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
@@ -446,7 +448,7 @@ rfurm = TailRecipe(
 
 # XX /r, RMI form for one of the roundXX SSE 4.1 instructions.
 furmi_rnd = TailRecipe(
-        'furmi_rnd', Unary, size=2, ins=FPR, outs=FPR,
+        'furmi_rnd', Unary, base_size=2, ins=FPR, outs=FPR,
         isap=use_sse41,
         emit='''
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
@@ -462,7 +464,7 @@ furmi_rnd = TailRecipe(
 
 # XX /r, for regmove instructions.
 rmov = TailRecipe(
-        'rmov', RegMove, size=1, ins=GPR, outs=(),
+        'rmov', RegMove, base_size=1, ins=GPR, outs=(),
         clobbers_flags=False,
         emit='''
         PUT_OP(bits, rex2(dst, src), sink);
@@ -471,7 +473,7 @@ rmov = TailRecipe(
 
 # XX /r, for regmove instructions (FPR version, RM encoded).
 frmov = TailRecipe(
-        'frmov', RegMove, size=1, ins=FPR, outs=(),
+        'frmov', RegMove, base_size=1, ins=FPR, outs=(),
         clobbers_flags=False,
         emit='''
         PUT_OP(bits, rex2(src, dst), sink);
@@ -480,7 +482,7 @@ frmov = TailRecipe(
 
 # XX /n with one arg in %rcx, for shifts.
 rc = TailRecipe(
-        'rc', Binary, size=1, ins=(GPR, GPR.rcx), outs=0,
+        'rc', Binary, base_size=1, ins=(GPR, GPR.rcx), outs=0,
         emit='''
         PUT_OP(bits, rex1(in_reg0), sink);
         modrm_r_bits(in_reg0, bits, sink);
@@ -488,7 +490,7 @@ rc = TailRecipe(
 
 # XX /n for division: inputs in %rax, %rdx, r. Outputs in %rax, %rdx.
 div = TailRecipe(
-        'div', Ternary, size=1,
+        'div', Ternary, base_size=1,
         ins=(GPR.rax, GPR.rdx, GPR), outs=(GPR.rax, GPR.rdx),
         emit='''
         sink.trap(TrapCode::IntegerDivisionByZero, func.srclocs[inst]);
@@ -498,7 +500,7 @@ div = TailRecipe(
 
 # XX /n for {s,u}mulx: inputs in %rax, r. Outputs in %rdx(hi):%rax(lo)
 mulx = TailRecipe(
-        'mulx', Binary, size=1,
+        'mulx', Binary, base_size=1,
         ins=(GPR.rax, GPR), outs=(GPR.rax, GPR.rdx),
         emit='''
         PUT_OP(bits, rex1(in_reg1), sink);
@@ -507,7 +509,7 @@ mulx = TailRecipe(
 
 # XX /n ib with 8-bit immediate sign-extended.
 r_ib = TailRecipe(
-        'r_ib', BinaryImm, size=2, ins=GPR, outs=0,
+        'r_ib', BinaryImm, base_size=2, ins=GPR, outs=0,
         instp=IsSignedInt(BinaryImm.imm, 8),
         emit='''
         PUT_OP(bits, rex1(in_reg0), sink);
@@ -518,7 +520,7 @@ r_ib = TailRecipe(
 
 # XX /n id with 32-bit immediate sign-extended.
 r_id = TailRecipe(
-        'r_id', BinaryImm, size=5, ins=GPR, outs=0,
+        'r_id', BinaryImm, base_size=5, ins=GPR, outs=0,
         instp=IsSignedInt(BinaryImm.imm, 32),
         emit='''
         PUT_OP(bits, rex1(in_reg0), sink);
@@ -529,7 +531,7 @@ r_id = TailRecipe(
 
 # XX /n id with 32-bit immediate sign-extended. UnaryImm version.
 u_id = TailRecipe(
-        'u_id', UnaryImm, size=5, ins=(), outs=GPR,
+        'u_id', UnaryImm, base_size=5, ins=(), outs=GPR,
         instp=IsSignedInt(UnaryImm.imm, 32),
         emit='''
         PUT_OP(bits, rex1(out_reg0), sink);
@@ -540,7 +542,7 @@ u_id = TailRecipe(
 
 # XX+rd id unary with 32-bit immediate. Note no recipe predicate.
 pu_id = TailRecipe(
-        'pu_id', UnaryImm, size=4, ins=(), outs=GPR,
+        'pu_id', UnaryImm, base_size=4, ins=(), outs=GPR,
         emit='''
         // The destination register is encoded in the low bits of the opcode.
         // No ModR/M.
@@ -551,7 +553,7 @@ pu_id = TailRecipe(
 
 # XX+rd id unary with bool immediate. Note no recipe predicate.
 pu_id_bool = TailRecipe(
-        'pu_id_bool', UnaryBool, size=4, ins=(), outs=GPR,
+        'pu_id_bool', UnaryBool, base_size=4, ins=(), outs=GPR,
         emit='''
         // The destination register is encoded in the low bits of the opcode.
         // No ModR/M.
@@ -562,7 +564,7 @@ pu_id_bool = TailRecipe(
 
 # XX+rd iq unary with 64-bit immediate.
 pu_iq = TailRecipe(
-        'pu_iq', UnaryImm, size=8, ins=(), outs=GPR,
+        'pu_iq', UnaryImm, base_size=8, ins=(), outs=GPR,
         emit='''
         PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
         let imm: i64 = imm.into();
@@ -571,7 +573,7 @@ pu_iq = TailRecipe(
 
 # XX /n Unary with floating point 32-bit immediate equal to zero.
 f32imm_z = TailRecipe(
-    'f32imm_z', UnaryIeee32, size=1, ins=(), outs=FPR,
+    'f32imm_z', UnaryIeee32, base_size=1, ins=(), outs=FPR,
     instp=IsZero32BitFloat(UnaryIeee32.imm),
     emit='''
         PUT_OP(bits, rex2(out_reg0, out_reg0), sink);
@@ -580,7 +582,7 @@ f32imm_z = TailRecipe(
 
 # XX /n Unary with floating point 64-bit immediate equal to zero.
 f64imm_z = TailRecipe(
-    'f64imm_z', UnaryIeee64, size=1, ins=(), outs=FPR,
+    'f64imm_z', UnaryIeee64, base_size=1, ins=(), outs=FPR,
     instp=IsZero64BitFloat(UnaryIeee64.imm),
     emit='''
         PUT_OP(bits, rex2(out_reg0, out_reg0), sink);
@@ -588,21 +590,21 @@ f64imm_z = TailRecipe(
     ''')
 
 pushq = TailRecipe(
-    'pushq', Unary, size=0, ins=GPR, outs=(),
+    'pushq', Unary, base_size=0, ins=GPR, outs=(),
     emit='''
     sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
     PUT_OP(bits | (in_reg0 & 7), rex1(in_reg0), sink);
     ''')
 
 popq = TailRecipe(
-    'popq', NullAry, size=0, ins=(), outs=GPR,
+    'popq', NullAry, base_size=0, ins=(), outs=GPR,
     emit='''
     PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
     ''')
 
 # XX /r, for regmove instructions.
 copysp = TailRecipe(
-        'copysp', CopySpecial, size=1, ins=(), outs=(),
+        'copysp', CopySpecial, base_size=1, ins=(), outs=(),
         clobbers_flags=False,
         emit='''
         PUT_OP(bits, rex2(dst, src), sink);
@@ -610,14 +612,14 @@ copysp = TailRecipe(
         ''')
 
 adjustsp = TailRecipe(
-    'adjustsp', Unary, size=1, ins=(GPR), outs=(),
+    'adjustsp', Unary, base_size=1, ins=(GPR), outs=(),
     emit='''
     PUT_OP(bits, rex2(RU::rsp.into(), in_reg0), sink);
     modrm_rr(RU::rsp.into(), in_reg0, sink);
     ''')
 
 adjustsp_ib = TailRecipe(
-    'adjustsp_ib', UnaryImm, size=2, ins=(), outs=(),
+    'adjustsp_ib', UnaryImm, base_size=2, ins=(), outs=(),
     instp=IsSignedInt(UnaryImm.imm, 8),
     emit='''
     PUT_OP(bits, rex1(RU::rsp.into()), sink);
@@ -627,7 +629,7 @@ adjustsp_ib = TailRecipe(
     ''')
 
 adjustsp_id = TailRecipe(
-    'adjustsp_id', UnaryImm, size=5, ins=(), outs=(),
+    'adjustsp_id', UnaryImm, base_size=5, ins=(), outs=(),
     instp=IsSignedInt(UnaryImm.imm, 32),
     emit='''
     PUT_OP(bits, rex1(RU::rsp.into()), sink);
@@ -639,7 +641,7 @@ adjustsp_id = TailRecipe(
 
 # XX+rd id with Abs4 function relocation.
 fnaddr4 = TailRecipe(
-        'fnaddr4', FuncAddr, size=4, ins=(), outs=GPR,
+        'fnaddr4', FuncAddr, base_size=4, ins=(), outs=GPR,
         emit='''
         PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
         sink.reloc_external(Reloc::Abs4,
@@ -650,7 +652,7 @@ fnaddr4 = TailRecipe(
 
 # XX+rd iq with Abs8 function relocation.
 fnaddr8 = TailRecipe(
-        'fnaddr8', FuncAddr, size=8, ins=(), outs=GPR,
+        'fnaddr8', FuncAddr, base_size=8, ins=(), outs=GPR,
         emit='''
         PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
         sink.reloc_external(Reloc::Abs8,
@@ -661,7 +663,7 @@ fnaddr8 = TailRecipe(
 
 # Similar to fnaddr4, but writes !0 (this is used by BaldrMonkey).
 allones_fnaddr4 = TailRecipe(
-        'allones_fnaddr4', FuncAddr, size=4, ins=(), outs=GPR,
+        'allones_fnaddr4', FuncAddr, base_size=4, ins=(), outs=GPR,
         emit='''
         PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
         sink.reloc_external(Reloc::Abs4,
@@ -673,7 +675,7 @@ allones_fnaddr4 = TailRecipe(
 
 # Similar to fnaddr8, but writes !0 (this is used by BaldrMonkey).
 allones_fnaddr8 = TailRecipe(
-        'allones_fnaddr8', FuncAddr, size=8, ins=(), outs=GPR,
+        'allones_fnaddr8', FuncAddr, base_size=8, ins=(), outs=GPR,
         emit='''
         PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
         sink.reloc_external(Reloc::Abs8,
@@ -684,7 +686,7 @@ allones_fnaddr8 = TailRecipe(
         ''')
 
 pcrel_fnaddr8 = TailRecipe(
-        'pcrel_fnaddr8', FuncAddr, size=5, ins=(), outs=GPR,
+        'pcrel_fnaddr8', FuncAddr, base_size=5, ins=(), outs=GPR,
         # rex2 gets passed 0 for r/m register because the upper bit of
         # r/m doesnt get decoded when in rip-relative addressing mode.
         emit='''
@@ -699,7 +701,7 @@ pcrel_fnaddr8 = TailRecipe(
         ''')
 
 got_fnaddr8 = TailRecipe(
-        'got_fnaddr8', FuncAddr, size=5, ins=(), outs=GPR,
+        'got_fnaddr8', FuncAddr, base_size=5, ins=(), outs=GPR,
         # rex2 gets passed 0 for r/m register because the upper bit of
         # r/m doesnt get decoded when in rip-relative addressing mode.
         emit='''
@@ -716,7 +718,7 @@ got_fnaddr8 = TailRecipe(
 
 # XX+rd id with Abs4 globalsym relocation.
 gvaddr4 = TailRecipe(
-        'gvaddr4', UnaryGlobalValue, size=4, ins=(), outs=GPR,
+        'gvaddr4', UnaryGlobalValue, base_size=4, ins=(), outs=GPR,
         emit='''
         PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
         sink.reloc_external(Reloc::Abs4,
@@ -727,7 +729,7 @@ gvaddr4 = TailRecipe(
 
 # XX+rd iq with Abs8 globalsym relocation.
 gvaddr8 = TailRecipe(
-        'gvaddr8', UnaryGlobalValue, size=8, ins=(), outs=GPR,
+        'gvaddr8', UnaryGlobalValue, base_size=8, ins=(), outs=GPR,
         emit='''
         PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
         sink.reloc_external(Reloc::Abs8,
@@ -738,7 +740,7 @@ gvaddr8 = TailRecipe(
 
 # XX+rd iq with PCRel4 globalsym relocation.
 pcrel_gvaddr8 = TailRecipe(
-        'pcrel_gvaddr8', UnaryGlobalValue, size=5, ins=(), outs=GPR,
+        'pcrel_gvaddr8', UnaryGlobalValue, base_size=5, ins=(), outs=GPR,
         emit='''
         PUT_OP(bits, rex2(0, out_reg0), sink);
         modrm_rm(5, out_reg0, sink);
@@ -752,7 +754,7 @@ pcrel_gvaddr8 = TailRecipe(
 
 # XX+rd iq with Abs8 globalsym relocation.
 got_gvaddr8 = TailRecipe(
-        'got_gvaddr8', UnaryGlobalValue, size=5, ins=(), outs=GPR,
+        'got_gvaddr8', UnaryGlobalValue, base_size=5, ins=(), outs=GPR,
         emit='''
         PUT_OP(bits, rex2(0, out_reg0), sink);
         modrm_rm(5, out_reg0, sink);
@@ -771,7 +773,7 @@ got_gvaddr8 = TailRecipe(
 #
 
 spaddr4_id = TailRecipe(
-        'spaddr4_id', StackLoad, size=6, ins=(), outs=GPR,
+        'spaddr4_id', StackLoad, base_size=6, ins=(), outs=GPR,
         emit='''
         let sp = StackRef::sp(stack_slot, &func.stack_slots);
         let base = stk_base(sp.base);
@@ -783,7 +785,7 @@ spaddr4_id = TailRecipe(
         ''')
 
 spaddr8_id = TailRecipe(
-        'spaddr8_id', StackLoad, size=6, ins=(), outs=GPR,
+        'spaddr8_id', StackLoad, base_size=6, ins=(), outs=GPR,
         emit='''
         let sp = StackRef::sp(stack_slot, &func.stack_slots);
         let base = stk_base(sp.base);
@@ -801,37 +803,50 @@ spaddr8_id = TailRecipe(
 
 # XX /r register-indirect store with no offset.
 st = TailRecipe(
-        'st', Store, size=1, ins=(GPR, GPR_ZERO_DEREF_SAFE), outs=(),
+        'st', Store, base_size=1, ins=(GPR, GPR), outs=(),
         instp=IsEqual(Store.offset, 0),
         clobbers_flags=False,
+        compute_size="size_plus_maybe_offset_for_in_reg_1",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        modrm_rm(in_reg1, in_reg0, sink);
+        if needs_offset(in_reg1) {
+            modrm_disp8(in_reg1, in_reg0, sink);
+            sink.put1(0);
+        } else {
+            modrm_rm(in_reg1, in_reg0, sink);
+        }
         ''')
 
 # XX /r register-indirect store with index and no offset.
 stWithIndex = TailRecipe(
-    'stWithIndex', StoreComplex, size=2,
-    ins=(GPR, GPR_ZERO_DEREF_SAFE, GPR_DEREF_SAFE),
+    'stWithIndex', StoreComplex, base_size=2,
+    ins=(GPR, GPR, GPR),
     outs=(),
     instp=IsEqual(StoreComplex.offset, 0),
     clobbers_flags=False,
+    compute_size="size_plus_maybe_offset_for_in_reg_1",
     emit='''
     if !flags.notrap() {
         sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
     }
     PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
-    modrm_sib(in_reg0, sink);
-    sib(0, in_reg2, in_reg1, sink);
+    if needs_offset(in_reg1) {
+        modrm_sib_disp8(in_reg0, sink);
+        sib(0, in_reg2, in_reg1, sink);
+        sink.put1(0);
+    } else {
+        modrm_sib(in_reg0, sink);
+        sib(0, in_reg2, in_reg1, sink);
+    }
     ''')
 
 # XX /r register-indirect store with no offset.
 # Only ABCD allowed for stored value. This is for byte stores with no REX.
 st_abcd = TailRecipe(
-        'st_abcd', Store, size=1, ins=(ABCD, GPR), outs=(),
+        'st_abcd', Store, base_size=1, ins=(ABCD, GPR), outs=(),
         instp=IsEqual(Store.offset, 0),
         when_prefixed=st,
         clobbers_flags=False,
@@ -846,66 +861,92 @@ st_abcd = TailRecipe(
 # XX /r register-indirect store with index and no offset.
 # Only ABCD allowed for stored value. This is for byte stores with no REX.
 stWithIndex_abcd = TailRecipe(
-    'stWithIndex_abcd', StoreComplex, size=2,
-    ins=(ABCD, GPR_ZERO_DEREF_SAFE, GPR_DEREF_SAFE),
+    'stWithIndex_abcd', StoreComplex, base_size=2,
+    ins=(ABCD, GPR, GPR),
     outs=(),
     instp=IsEqual(StoreComplex.offset, 0),
     clobbers_flags=False,
+    compute_size="size_plus_maybe_offset_for_in_reg_1",
     emit='''
     if !flags.notrap() {
         sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
     }
     PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
-    modrm_sib(in_reg0, sink);
-    sib(0, in_reg2, in_reg1, sink);
+    if needs_offset(in_reg1) {
+        modrm_sib_disp8(in_reg0, sink);
+        sib(0, in_reg2, in_reg1, sink);
+        sink.put1(0);
+    } else {
+        modrm_sib(in_reg0, sink);
+        sib(0, in_reg2, in_reg1, sink);
+    }
     ''')
 
 # XX /r register-indirect store of FPR with no offset.
 fst = TailRecipe(
-        'fst', Store, size=1, ins=(FPR, GPR_ZERO_DEREF_SAFE), outs=(),
+        'fst', Store, base_size=1, ins=(FPR, GPR), outs=(),
         instp=IsEqual(Store.offset, 0),
         clobbers_flags=False,
+        compute_size="size_plus_maybe_offset_for_in_reg_1",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        modrm_rm(in_reg1, in_reg0, sink);
+        if needs_offset(in_reg1) {
+            modrm_disp8(in_reg1, in_reg0, sink);
+            sink.put1(0);
+        } else {
+            modrm_rm(in_reg1, in_reg0, sink);
+        }
         ''')
 # XX /r register-indirect store with index and no offset of FPR.
 fstWithIndex = TailRecipe(
-        'fstWithIndex', StoreComplex, size=2,
-        ins=(FPR, GPR_ZERO_DEREF_SAFE, GPR_DEREF_SAFE), outs=(),
+        'fstWithIndex', StoreComplex, base_size=2,
+        ins=(FPR, GPR, GPR), outs=(),
         instp=IsEqual(StoreComplex.offset, 0),
         clobbers_flags=False,
+        compute_size="size_plus_maybe_offset_for_in_reg_1",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
-        modrm_sib(in_reg0, sink);
-        sib(0, in_reg2, in_reg1, sink);
+        if needs_offset(in_reg1) {
+            modrm_sib_disp8(in_reg0, sink);
+            sib(0, in_reg2, in_reg1, sink);
+            sink.put1(0);
+        } else {
+            modrm_sib(in_reg0, sink);
+            sib(0, in_reg2, in_reg1, sink);
+        }
         ''')
 
 # XX /r register-indirect store with 8-bit offset.
 stDisp8 = TailRecipe(
-        'stDisp8', Store, size=2, ins=(GPR, GPR_DEREF_SAFE), outs=(),
+        'stDisp8', Store, base_size=2, ins=(GPR, GPR), outs=(),
         instp=IsSignedInt(Store.offset, 8),
         clobbers_flags=False,
+        compute_size="size_plus_maybe_sib_for_in_reg_1",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        modrm_disp8(in_reg1, in_reg0, sink);
+        if needs_sib_byte(in_reg1) {
+            modrm_sib_disp8(in_reg0, sink);
+            sib_noindex(in_reg1, sink);
+        } else {
+            modrm_disp8(in_reg1, in_reg0, sink);
+        }
         let offset: i32 = offset.into();
         sink.put1(offset as u8);
         ''')
 
 # XX /r register-indirect store with index and 8-bit offset.
 stWithIndexDisp8 = TailRecipe(
-    'stWithIndexDisp8', StoreComplex, size=3,
-    ins=(GPR, GPR, GPR_DEREF_SAFE),
+    'stWithIndexDisp8', StoreComplex, base_size=3,
+    ins=(GPR, GPR, GPR),
     outs=(),
     instp=IsSignedInt(StoreComplex.offset, 8),
     clobbers_flags=False,
@@ -923,7 +964,7 @@ stWithIndexDisp8 = TailRecipe(
 # XX /r register-indirect store with 8-bit offset.
 # Only ABCD allowed for stored value. This is for byte stores with no REX.
 stDisp8_abcd = TailRecipe(
-        'stDisp8_abcd', Store, size=2, ins=(ABCD, GPR), outs=(),
+        'stDisp8_abcd', Store, base_size=2, ins=(ABCD, GPR), outs=(),
         instp=IsSignedInt(Store.offset, 8),
         when_prefixed=stDisp8,
         clobbers_flags=False,
@@ -940,8 +981,8 @@ stDisp8_abcd = TailRecipe(
 # XX /r register-indirect store with index and 8-bit offset.
 # Only ABCD allowed for stored value. This is for byte stores with no REX.
 stWithIndexDisp8_abcd = TailRecipe(
-    'stWithIndexDisp8_abcd', StoreComplex, size=3,
-    ins=(ABCD, GPR, GPR_DEREF_SAFE),
+    'stWithIndexDisp8_abcd', StoreComplex, base_size=3,
+    ins=(ABCD, GPR, GPR),
     outs=(),
     instp=IsSignedInt(StoreComplex.offset, 8),
     clobbers_flags=False,
@@ -958,23 +999,29 @@ stWithIndexDisp8_abcd = TailRecipe(
 
 # XX /r register-indirect store with 8-bit offset of FPR.
 fstDisp8 = TailRecipe(
-        'fstDisp8', Store, size=2, ins=(FPR, GPR_DEREF_SAFE), outs=(),
+        'fstDisp8', Store, base_size=2, ins=(FPR, GPR), outs=(),
         instp=IsSignedInt(Store.offset, 8),
         clobbers_flags=False,
+        compute_size='size_plus_maybe_sib_for_in_reg_1',
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        modrm_disp8(in_reg1, in_reg0, sink);
+        if needs_sib_byte(in_reg1) {
+            modrm_sib_disp8(in_reg0, sink);
+            sib_noindex(in_reg1, sink);
+        } else {
+            modrm_disp8(in_reg1, in_reg0, sink);
+        }
         let offset: i32 = offset.into();
         sink.put1(offset as u8);
         ''')
 
 # XX /r register-indirect store with index and 8-bit offset of FPR.
 fstWithIndexDisp8 = TailRecipe(
-    'fstWithIndexDisp8', StoreComplex, size=3,
-    ins=(FPR, GPR, GPR_DEREF_SAFE),
+    'fstWithIndexDisp8', StoreComplex, base_size=3,
+    ins=(FPR, GPR, GPR),
     outs=(),
     instp=IsSignedInt(StoreComplex.offset, 8),
     clobbers_flags=False,
@@ -991,22 +1038,28 @@ fstWithIndexDisp8 = TailRecipe(
 
 # XX /r register-indirect store with 32-bit offset.
 stDisp32 = TailRecipe(
-        'stDisp32', Store, size=5, ins=(GPR, GPR_DEREF_SAFE), outs=(),
+        'stDisp32', Store, base_size=5, ins=(GPR, GPR), outs=(),
         clobbers_flags=False,
+        compute_size='size_plus_maybe_sib_for_in_reg_1',
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        modrm_disp32(in_reg1, in_reg0, sink);
+        if needs_sib_byte(in_reg1) {
+            modrm_sib_disp32(in_reg0, sink);
+            sib_noindex(in_reg1, sink);
+        } else {
+            modrm_disp32(in_reg1, in_reg0, sink);
+        }
         let offset: i32 = offset.into();
         sink.put4(offset as u32);
         ''')
 
 # XX /r register-indirect store with index and 32-bit offset.
 stWithIndexDisp32 = TailRecipe(
-    'stWithIndexDisp32', StoreComplex, size=6,
-    ins=(GPR, GPR, GPR_DEREF_SAFE),
+    'stWithIndexDisp32', StoreComplex, base_size=6,
+    ins=(GPR, GPR, GPR),
     outs=(),
     instp=IsSignedInt(StoreComplex.offset, 32),
     clobbers_flags=False,
@@ -1024,7 +1077,7 @@ stWithIndexDisp32 = TailRecipe(
 # XX /r register-indirect store with 32-bit offset.
 # Only ABCD allowed for stored value. This is for byte stores with no REX.
 stDisp32_abcd = TailRecipe(
-        'stDisp32_abcd', Store, size=5, ins=(ABCD, GPR), outs=(),
+        'stDisp32_abcd', Store, base_size=5, ins=(ABCD, GPR), outs=(),
         when_prefixed=stDisp32,
         clobbers_flags=False,
         emit='''
@@ -1040,8 +1093,8 @@ stDisp32_abcd = TailRecipe(
 # XX /r register-indirect store with index and 32-bit offset.
 # Only ABCD allowed for stored value. This is for byte stores with no REX.
 stWithIndexDisp32_abcd = TailRecipe(
-    'stWithIndexDisp32_abcd', StoreComplex, size=6,
-    ins=(ABCD, GPR, GPR_DEREF_SAFE),
+    'stWithIndexDisp32_abcd', StoreComplex, base_size=6,
+    ins=(ABCD, GPR, GPR),
     outs=(),
     instp=IsSignedInt(StoreComplex.offset, 32),
     clobbers_flags=False,
@@ -1058,22 +1111,28 @@ stWithIndexDisp32_abcd = TailRecipe(
 
 # XX /r register-indirect store with 32-bit offset of FPR.
 fstDisp32 = TailRecipe(
-        'fstDisp32', Store, size=5, ins=(FPR, GPR_DEREF_SAFE), outs=(),
+        'fstDisp32', Store, base_size=5, ins=(FPR, GPR), outs=(),
         clobbers_flags=False,
+        compute_size='size_plus_maybe_sib_for_in_reg_1',
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        modrm_disp32(in_reg1, in_reg0, sink);
+        if needs_sib_byte(in_reg1) {
+            modrm_sib_disp32(in_reg0, sink);
+            sib_noindex(in_reg1, sink);
+        } else {
+            modrm_disp32(in_reg1, in_reg0, sink);
+        }
         let offset: i32 = offset.into();
         sink.put4(offset as u32);
         ''')
 
 # XX /r register-indirect store with index and 32-bit offset of FPR.
 fstWithIndexDisp32 = TailRecipe(
-    'fstWithIndexDisp32', StoreComplex, size=6,
-    ins=(FPR, GPR, GPR_DEREF_SAFE),
+    'fstWithIndexDisp32', StoreComplex, base_size=6,
+    ins=(FPR, GPR, GPR),
     outs=(),
     instp=IsSignedInt(StoreComplex.offset, 32),
     clobbers_flags=False,
@@ -1090,7 +1149,7 @@ fstWithIndexDisp32 = TailRecipe(
 
 # Unary spill with SIB and 32-bit displacement.
 spillSib32 = TailRecipe(
-        'spillSib32', Unary, size=6, ins=GPR, outs=StackGPR32,
+        'spillSib32', Unary, base_size=6, ins=GPR, outs=StackGPR32,
         clobbers_flags=False,
         emit='''
         sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
@@ -1103,7 +1162,7 @@ spillSib32 = TailRecipe(
 
 # Like spillSib32, but targeting an FPR rather than a GPR.
 fspillSib32 = TailRecipe(
-        'fspillSib32', Unary, size=6, ins=FPR, outs=StackFPR32,
+        'fspillSib32', Unary, base_size=6, ins=FPR, outs=StackFPR32,
         clobbers_flags=False,
         emit='''
         sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
@@ -1116,7 +1175,7 @@ fspillSib32 = TailRecipe(
 
 # Regspill using RSP-relative addressing.
 regspill32 = TailRecipe(
-        'regspill32', RegSpill, size=6, ins=GPR, outs=(),
+        'regspill32', RegSpill, base_size=6, ins=GPR, outs=(),
         clobbers_flags=False,
         emit='''
         sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
@@ -1130,7 +1189,7 @@ regspill32 = TailRecipe(
 
 # Like regspill32, but targeting an FPR rather than a GPR.
 fregspill32 = TailRecipe(
-        'fregspill32', RegSpill, size=6, ins=FPR, outs=(),
+        'fregspill32', RegSpill, base_size=6, ins=FPR, outs=(),
         clobbers_flags=False,
         emit='''
         sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
@@ -1148,81 +1207,113 @@ fregspill32 = TailRecipe(
 
 # XX /r load with no offset.
 ld = TailRecipe(
-        'ld', Load, size=1, ins=(GPR_ZERO_DEREF_SAFE), outs=(GPR),
+        'ld', Load, base_size=1, ins=(GPR), outs=(GPR),
         instp=IsEqual(Load.offset, 0),
         clobbers_flags=False,
+        compute_size="size_plus_maybe_offset_for_in_reg_0",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_rm(in_reg0, out_reg0, sink);
+        if needs_offset(in_reg0) {
+            modrm_disp8(in_reg0, out_reg0, sink);
+            sink.put1(0);
+        } else {
+            modrm_rm(in_reg0, out_reg0, sink);
+        }
         ''')
 
 # XX /r load with index and no offset.
 ldWithIndex = TailRecipe(
-    'ldWithIndex', LoadComplex, size=2,
-    ins=(GPR_ZERO_DEREF_SAFE, GPR_DEREF_SAFE),
+    'ldWithIndex', LoadComplex, base_size=2,
+    ins=(GPR, GPR),
     outs=(GPR),
     instp=IsEqual(LoadComplex.offset, 0),
     clobbers_flags=False,
+    compute_size="size_plus_maybe_offset_for_in_reg_0",
     emit='''
     if !flags.notrap() {
         sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
     }
     PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
-    modrm_sib(out_reg0, sink);
-    sib(0, in_reg1, in_reg0, sink);
+    if needs_offset(in_reg0) {
+        modrm_sib_disp8(out_reg0, sink);
+        sib(0, in_reg1, in_reg0, sink);
+        sink.put1(0);
+    } else {
+        modrm_sib(out_reg0, sink);
+        sib(0, in_reg1, in_reg0, sink);
+    }
     ''')
 
 # XX /r float load with no offset.
 fld = TailRecipe(
-        'fld', Load, size=1, ins=(GPR_ZERO_DEREF_SAFE), outs=(FPR),
+        'fld', Load, base_size=1, ins=(GPR), outs=(FPR),
         instp=IsEqual(Load.offset, 0),
         clobbers_flags=False,
+        compute_size="size_plus_maybe_offset_for_in_reg_0",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_rm(in_reg0, out_reg0, sink);
+        if needs_offset(in_reg0) {
+            modrm_disp8(in_reg0, out_reg0, sink);
+            sink.put1(0);
+        } else {
+            modrm_rm(in_reg0, out_reg0, sink);
+        }
         ''')
 
 # XX /r float load with index and no offset.
 fldWithIndex = TailRecipe(
-    'fldWithIndex', LoadComplex, size=2,
-    ins=(GPR_ZERO_DEREF_SAFE, GPR_DEREF_SAFE),
+    'fldWithIndex', LoadComplex, base_size=2,
+    ins=(GPR, GPR),
     outs=(FPR),
     instp=IsEqual(LoadComplex.offset, 0),
     clobbers_flags=False,
+    compute_size="size_plus_maybe_offset_for_in_reg_0",
     emit='''
     if !flags.notrap() {
         sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
     }
     PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
-    modrm_sib(out_reg0, sink);
-    sib(0, in_reg1, in_reg0, sink);
+    if needs_offset(in_reg0) {
+        modrm_sib_disp8(out_reg0, sink);
+        sib(0, in_reg1, in_reg0, sink);
+        sink.put1(0);
+    } else {
+        modrm_sib(out_reg0, sink);
+        sib(0, in_reg1, in_reg0, sink);
+    }
     ''')
 
 # XX /r load with 8-bit offset.
 ldDisp8 = TailRecipe(
-        'ldDisp8', Load, size=2, ins=(GPR_DEREF_SAFE), outs=(GPR),
+        'ldDisp8', Load, base_size=2, ins=(GPR), outs=(GPR),
         instp=IsSignedInt(Load.offset, 8),
         clobbers_flags=False,
+        compute_size="size_plus_maybe_sib_for_in_reg_0",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_disp8(in_reg0, out_reg0, sink);
+        if needs_sib_byte(in_reg0) {
+            modrm_sib_disp8(out_reg0, sink);
+            sib_noindex(in_reg0, sink);
+        } else {
+            modrm_disp8(in_reg0, out_reg0, sink);
+        }
         let offset: i32 = offset.into();
         sink.put1(offset as u8);
         ''')
 
 # XX /r load with index and 8-bit offset.
 ldWithIndexDisp8 = TailRecipe(
-    'ldWithIndexDisp8', LoadComplex, size=3,
-    ins=(GPR, GPR_DEREF_SAFE),
+    'ldWithIndexDisp8', LoadComplex, base_size=3,
+    ins=(GPR, GPR),
     outs=(GPR),
     instp=IsSignedInt(LoadComplex.offset, 8),
     clobbers_flags=False,
@@ -1239,23 +1330,29 @@ ldWithIndexDisp8 = TailRecipe(
 
 # XX /r float load with 8-bit offset.
 fldDisp8 = TailRecipe(
-        'fldDisp8', Load, size=2, ins=(GPR_DEREF_SAFE), outs=(FPR),
+        'fldDisp8', Load, base_size=2, ins=(GPR), outs=(FPR),
         instp=IsSignedInt(Load.offset, 8),
         clobbers_flags=False,
+        compute_size="size_plus_maybe_sib_for_in_reg_0",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_disp8(in_reg0, out_reg0, sink);
+        if needs_sib_byte(in_reg0) {
+            modrm_sib_disp8(out_reg0, sink);
+            sib_noindex(in_reg0, sink);
+        } else {
+            modrm_disp8(in_reg0, out_reg0, sink);
+        }
         let offset: i32 = offset.into();
         sink.put1(offset as u8);
         ''')
 
 # XX /r float load with 8-bit offset.
 fldWithIndexDisp8 = TailRecipe(
-    'fldWithIndexDisp8', LoadComplex, size=3,
-    ins=(GPR, GPR_DEREF_SAFE),
+    'fldWithIndexDisp8', LoadComplex, base_size=3,
+    ins=(GPR, GPR),
     outs=(FPR),
     instp=IsSignedInt(LoadComplex.offset, 8),
     clobbers_flags=False,
@@ -1272,23 +1369,29 @@ fldWithIndexDisp8 = TailRecipe(
 
 # XX /r load with 32-bit offset.
 ldDisp32 = TailRecipe(
-        'ldDisp32', Load, size=5, ins=(GPR_DEREF_SAFE), outs=(GPR),
+        'ldDisp32', Load, base_size=5, ins=(GPR), outs=(GPR),
         instp=IsSignedInt(Load.offset, 32),
         clobbers_flags=False,
+        compute_size='size_plus_maybe_sib_for_in_reg_0',
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_disp32(in_reg0, out_reg0, sink);
+        if needs_sib_byte(in_reg0) {
+            modrm_sib_disp32(out_reg0, sink);
+            sib_noindex(in_reg0, sink);
+        } else {
+            modrm_disp32(in_reg0, out_reg0, sink);
+        }
         let offset: i32 = offset.into();
         sink.put4(offset as u32);
         ''')
 
 # XX /r load with index and 32-bit offset.
 ldWithIndexDisp32 = TailRecipe(
-    'ldWithIndexDisp32', LoadComplex, size=6,
-    ins=(GPR, GPR_DEREF_SAFE),
+    'ldWithIndexDisp32', LoadComplex, base_size=6,
+    ins=(GPR, GPR),
     outs=(GPR),
     instp=IsSignedInt(LoadComplex.offset, 32),
     clobbers_flags=False,
@@ -1305,23 +1408,29 @@ ldWithIndexDisp32 = TailRecipe(
 
 # XX /r float load with 32-bit offset.
 fldDisp32 = TailRecipe(
-        'fldDisp32', Load, size=5, ins=(GPR_DEREF_SAFE), outs=(FPR),
+        'fldDisp32', Load, base_size=5, ins=(GPR), outs=(FPR),
         instp=IsSignedInt(Load.offset, 32),
         clobbers_flags=False,
+        compute_size="size_plus_maybe_sib_for_in_reg_0",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_disp32(in_reg0, out_reg0, sink);
+        if needs_sib_byte(in_reg0) {
+            modrm_sib_disp32(out_reg0, sink);
+            sib_noindex(in_reg0, sink);
+        } else {
+            modrm_disp32(in_reg0, out_reg0, sink);
+        }
         let offset: i32 = offset.into();
         sink.put4(offset as u32);
         ''')
 
 # XX /r float load with index and 32-bit offset.
 fldWithIndexDisp32 = TailRecipe(
-    'fldWithIndexDisp32', LoadComplex, size=6,
-    ins=(GPR, GPR_DEREF_SAFE),
+    'fldWithIndexDisp32', LoadComplex, base_size=6,
+    ins=(GPR, GPR),
     outs=(FPR),
     instp=IsSignedInt(LoadComplex.offset, 32),
     clobbers_flags=False,
@@ -1338,7 +1447,7 @@ fldWithIndexDisp32 = TailRecipe(
 
 # Unary fill with SIB and 32-bit displacement.
 fillSib32 = TailRecipe(
-        'fillSib32', Unary, size=6, ins=StackGPR32, outs=GPR,
+        'fillSib32', Unary, base_size=6, ins=StackGPR32, outs=GPR,
         clobbers_flags=False,
         emit='''
         let base = stk_base(in_stk0.base);
@@ -1350,7 +1459,7 @@ fillSib32 = TailRecipe(
 
 # Like fillSib32, but targeting an FPR rather than a GPR.
 ffillSib32 = TailRecipe(
-        'ffillSib32', Unary, size=6, ins=StackFPR32, outs=FPR,
+        'ffillSib32', Unary, base_size=6, ins=StackFPR32, outs=FPR,
         clobbers_flags=False,
         emit='''
         let base = stk_base(in_stk0.base);
@@ -1362,7 +1471,7 @@ ffillSib32 = TailRecipe(
 
 # Regfill with RSP-relative 32-bit displacement.
 regfill32 = TailRecipe(
-        'regfill32', RegFill, size=6, ins=StackGPR32, outs=(),
+        'regfill32', RegFill, base_size=6, ins=StackGPR32, outs=(),
         clobbers_flags=False,
         emit='''
         let src = StackRef::sp(src, &func.stack_slots);
@@ -1375,7 +1484,7 @@ regfill32 = TailRecipe(
 
 # Like regfill32, but targeting an FPR rather than a GPR.
 fregfill32 = TailRecipe(
-        'fregfill32', RegFill, size=6, ins=StackFPR32, outs=(),
+        'fregfill32', RegFill, base_size=6, ins=StackFPR32, outs=(),
         clobbers_flags=False,
         emit='''
         let src = StackRef::sp(src, &func.stack_slots);
@@ -1390,7 +1499,7 @@ fregfill32 = TailRecipe(
 # Call/return
 #
 call_id = TailRecipe(
-        'call_id', Call, size=4, ins=(), outs=(),
+        'call_id', Call, base_size=4, ins=(), outs=(),
         emit='''
         sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
         PUT_OP(bits, BASE_REX, sink);
@@ -1403,7 +1512,7 @@ call_id = TailRecipe(
         ''')
 
 call_plt_id = TailRecipe(
-        'call_plt_id', Call, size=4, ins=(), outs=(),
+        'call_plt_id', Call, base_size=4, ins=(), outs=(),
         emit='''
         sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
         PUT_OP(bits, BASE_REX, sink);
@@ -1414,7 +1523,7 @@ call_plt_id = TailRecipe(
         ''')
 
 call_r = TailRecipe(
-        'call_r', CallIndirect, size=1, ins=GPR, outs=(),
+        'call_r', CallIndirect, base_size=1, ins=GPR, outs=(),
         emit='''
         sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
         PUT_OP(bits, rex1(in_reg0), sink);
@@ -1422,7 +1531,7 @@ call_r = TailRecipe(
         ''')
 
 ret = TailRecipe(
-        'ret', MultiAry, size=0, ins=(), outs=(),
+        'ret', MultiAry, base_size=0, ins=(), outs=(),
         emit='''
         PUT_OP(bits, BASE_REX, sink);
         ''')
@@ -1431,7 +1540,7 @@ ret = TailRecipe(
 # Branches
 #
 jmpb = TailRecipe(
-        'jmpb', Jump, size=1, ins=(), outs=(),
+        'jmpb', Jump, base_size=1, ins=(), outs=(),
         branch_range=8,
         clobbers_flags=False,
         emit='''
@@ -1440,7 +1549,7 @@ jmpb = TailRecipe(
         ''')
 
 jmpd = TailRecipe(
-        'jmpd', Jump, size=4, ins=(), outs=(),
+        'jmpd', Jump, base_size=4, ins=(), outs=(),
         branch_range=32,
         clobbers_flags=False,
         emit='''
@@ -1449,7 +1558,7 @@ jmpd = TailRecipe(
         ''')
 
 brib = TailRecipe(
-        'brib', BranchInt, size=1, ins=FLAG.rflags, outs=(),
+        'brib', BranchInt, base_size=1, ins=FLAG.rflags, outs=(),
         branch_range=8,
         clobbers_flags=False,
         emit='''
@@ -1458,7 +1567,7 @@ brib = TailRecipe(
         ''')
 
 brid = TailRecipe(
-        'brid', BranchInt, size=4, ins=FLAG.rflags, outs=(),
+        'brid', BranchInt, base_size=4, ins=FLAG.rflags, outs=(),
         branch_range=32,
         clobbers_flags=False,
         emit='''
@@ -1467,7 +1576,7 @@ brid = TailRecipe(
         ''')
 
 brfb = TailRecipe(
-        'brfb', BranchFloat, size=1, ins=FLAG.rflags, outs=(),
+        'brfb', BranchFloat, base_size=1, ins=FLAG.rflags, outs=(),
         branch_range=8,
         clobbers_flags=False,
         instp=floatccs(BranchFloat),
@@ -1477,7 +1586,7 @@ brfb = TailRecipe(
         ''')
 
 brfd = TailRecipe(
-        'brfd', BranchFloat, size=4, ins=FLAG.rflags, outs=(),
+        'brfd', BranchFloat, base_size=4, ins=FLAG.rflags, outs=(),
         branch_range=32,
         clobbers_flags=False,
         instp=floatccs(BranchFloat),
@@ -1487,7 +1596,7 @@ brfd = TailRecipe(
         ''')
 
 indirect_jmp = TailRecipe(
-        'indirect_jmp', IndirectJump, size=1, ins=GPR, outs=(),
+        'indirect_jmp', IndirectJump, base_size=1, ins=GPR, outs=(),
         clobbers_flags=False,
         emit='''
         PUT_OP(bits, rex1(in_reg0), sink);
@@ -1495,19 +1604,26 @@ indirect_jmp = TailRecipe(
         ''')
 
 jt_entry = TailRecipe(
-        'jt_entry', BranchTableEntry, size=2,
-        ins=(GPR_DEREF_SAFE, GPR_ZERO_DEREF_SAFE),
+        'jt_entry', BranchTableEntry, base_size=2,
+        ins=(GPR, GPR),
         outs=(GPR),
         clobbers_flags=False,
         instp=valid_scale(BranchTableEntry),
+        compute_size="size_plus_maybe_offset_for_in_reg_1",
         emit='''
         PUT_OP(bits, rex3(in_reg1, out_reg0, in_reg0), sink);
-        modrm_sib(out_reg0, sink);
-        sib(imm.trailing_zeros() as u8, in_reg0, in_reg1, sink);
+        if needs_offset(in_reg1) {
+            modrm_sib_disp8(out_reg0, sink);
+            sib(imm.trailing_zeros() as u8, in_reg0, in_reg1, sink);
+            sink.put1(0);
+        } else {
+            modrm_sib(out_reg0, sink);
+            sib(imm.trailing_zeros() as u8, in_reg0, in_reg1, sink);
+        }
         ''')
 
 jt_base = TailRecipe(
-        'jt_base', BranchTableBase, size=5, ins=(), outs=(GPR),
+        'jt_base', BranchTableBase, base_size=5, ins=(), outs=(GPR),
         clobbers_flags=False,
         emit='''
         PUT_OP(bits, rex2(0, out_reg0), sink);
@@ -1529,7 +1645,7 @@ jt_base = TailRecipe(
 #
 
 seti = TailRecipe(
-        'seti', IntCond, size=1, ins=FLAG.rflags, outs=GPR,
+        'seti', IntCond, base_size=1, ins=FLAG.rflags, outs=GPR,
         requires_prefix=True,
         clobbers_flags=False,
         emit='''
@@ -1537,7 +1653,7 @@ seti = TailRecipe(
         modrm_r_bits(out_reg0, bits, sink);
         ''')
 seti_abcd = TailRecipe(
-        'seti_abcd', IntCond, size=1, ins=FLAG.rflags, outs=ABCD,
+        'seti_abcd', IntCond, base_size=1, ins=FLAG.rflags, outs=ABCD,
         when_prefixed=seti,
         clobbers_flags=False,
         emit='''
@@ -1546,7 +1662,7 @@ seti_abcd = TailRecipe(
         ''')
 
 setf = TailRecipe(
-        'setf', FloatCond, size=1, ins=FLAG.rflags, outs=GPR,
+        'setf', FloatCond, base_size=1, ins=FLAG.rflags, outs=GPR,
         requires_prefix=True,
         clobbers_flags=False,
         emit='''
@@ -1554,7 +1670,7 @@ setf = TailRecipe(
         modrm_r_bits(out_reg0, bits, sink);
         ''')
 setf_abcd = TailRecipe(
-        'setf_abcd', FloatCond, size=1, ins=FLAG.rflags, outs=ABCD,
+        'setf_abcd', FloatCond, base_size=1, ins=FLAG.rflags, outs=ABCD,
         when_prefixed=setf,
         clobbers_flags=False,
         emit='''
@@ -1568,7 +1684,7 @@ setf_abcd = TailRecipe(
 # 1 byte, modrm(r,r), is after the opcode
 #
 cmov = TailRecipe(
-        'cmov', IntSelect, size=1, ins=(FLAG.rflags, GPR, GPR), outs=2,
+        'cmov', IntSelect, base_size=1, ins=(FLAG.rflags, GPR, GPR), outs=2,
         requires_prefix=False,
         clobbers_flags=False,
         emit='''
@@ -1580,7 +1696,7 @@ cmov = TailRecipe(
 # Bit scan forwards and reverse
 #
 bsf_and_bsr = TailRecipe(
-        'bsf_and_bsr', Unary, size=1, ins=GPR, outs=(GPR, FLAG.rflags),
+        'bsf_and_bsr', Unary, base_size=1, ins=GPR, outs=(GPR, FLAG.rflags),
         requires_prefix=False,
         clobbers_flags=True,
         emit='''
@@ -1594,7 +1710,7 @@ bsf_and_bsr = TailRecipe(
 
 # XX /r, MR form. Compare two GPR registers and set flags.
 rcmp = TailRecipe(
-        'rcmp', Binary, size=1, ins=(GPR, GPR), outs=FLAG.rflags,
+        'rcmp', Binary, base_size=1, ins=(GPR, GPR), outs=FLAG.rflags,
         emit='''
         PUT_OP(bits, rex2(in_reg0, in_reg1), sink);
         modrm_rr(in_reg0, in_reg1, sink);
@@ -1602,7 +1718,7 @@ rcmp = TailRecipe(
 
 # XX /r, RM form. Compare two FPR registers and set flags.
 fcmp = TailRecipe(
-        'fcmp', Binary, size=1, ins=(FPR, FPR), outs=FLAG.rflags,
+        'fcmp', Binary, base_size=1, ins=(FPR, FPR), outs=FLAG.rflags,
         emit='''
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
         modrm_rr(in_reg1, in_reg0, sink);
@@ -1610,7 +1726,7 @@ fcmp = TailRecipe(
 
 # XX /n, MI form with imm8.
 rcmp_ib = TailRecipe(
-        'rcmp_ib', BinaryImm, size=2, ins=GPR, outs=FLAG.rflags,
+        'rcmp_ib', BinaryImm, base_size=2, ins=GPR, outs=FLAG.rflags,
         instp=IsSignedInt(BinaryImm.imm, 8),
         emit='''
         PUT_OP(bits, rex1(in_reg0), sink);
@@ -1621,7 +1737,7 @@ rcmp_ib = TailRecipe(
 
 # XX /n, MI form with imm32.
 rcmp_id = TailRecipe(
-        'rcmp_id', BinaryImm, size=5, ins=GPR, outs=FLAG.rflags,
+        'rcmp_id', BinaryImm, base_size=5, ins=GPR, outs=FLAG.rflags,
         instp=IsSignedInt(BinaryImm.imm, 32),
         emit='''
         PUT_OP(bits, rex1(in_reg0), sink);
@@ -1632,7 +1748,7 @@ rcmp_id = TailRecipe(
 
 # Same as rcmp, but second operand is the stack pointer.
 rcmp_sp = TailRecipe(
-        'rcmp_sp', Unary, size=1, ins=GPR, outs=FLAG.rflags,
+        'rcmp_sp', Unary, base_size=1, ins=GPR, outs=FLAG.rflags,
         emit='''
         PUT_OP(bits, rex2(in_reg0, RU::rsp.into()), sink);
         modrm_rr(in_reg0, RU::rsp.into(), sink);
@@ -1652,7 +1768,7 @@ rcmp_sp = TailRecipe(
 # Bits 0-7 are the Jcc opcode.
 # Bits 8-15 control the test instruction which always has opcode byte 0x85.
 tjccb = TailRecipe(
-        'tjccb', Branch, size=1 + 2, ins=GPR, outs=(),
+        'tjccb', Branch, base_size=1 + 2, ins=GPR, outs=(),
         branch_range=8,
         emit='''
         // test r, r.
@@ -1664,7 +1780,7 @@ tjccb = TailRecipe(
         ''')
 
 tjccd = TailRecipe(
-        'tjccd', Branch, size=1 + 6, ins=GPR, outs=(),
+        'tjccd', Branch, base_size=1 + 6, ins=GPR, outs=(),
         branch_range=32,
         emit='''
         // test r, r.
@@ -1681,7 +1797,7 @@ tjccd = TailRecipe(
 # Same as tjccb, but only looks at the low 8 bits of the register, for b1
 # types.
 t8jccb = TailRecipe(
-        't8jccb', Branch, size=1 + 2, ins=GPR, outs=(),
+        't8jccb', Branch, base_size=1 + 2, ins=GPR, outs=(),
         branch_range=8,
         requires_prefix=True,
         emit='''
@@ -1693,7 +1809,7 @@ t8jccb = TailRecipe(
         disp1(destination, func, sink);
         ''')
 t8jccb_abcd = TailRecipe(
-        't8jccb_abcd', Branch, size=1 + 2, ins=ABCD, outs=(),
+        't8jccb_abcd', Branch, base_size=1 + 2, ins=ABCD, outs=(),
         branch_range=8,
         when_prefixed=t8jccb,
         emit='''
@@ -1706,7 +1822,7 @@ t8jccb_abcd = TailRecipe(
         ''')
 
 t8jccd = TailRecipe(
-        't8jccd', Branch, size=1 + 6, ins=GPR, outs=(),
+        't8jccd', Branch, base_size=1 + 6, ins=GPR, outs=(),
         branch_range=32,
         requires_prefix=True,
         emit='''
@@ -1719,7 +1835,7 @@ t8jccd = TailRecipe(
         disp4(destination, func, sink);
         ''')
 t8jccd_abcd = TailRecipe(
-        't8jccd_abcd', Branch, size=1 + 6, ins=ABCD, outs=(),
+        't8jccd_abcd', Branch, base_size=1 + 6, ins=ABCD, outs=(),
         branch_range=32,
         when_prefixed=t8jccd,
         emit='''
@@ -1738,7 +1854,7 @@ t8jccd_abcd = TailRecipe(
 # any register, but is is larger because it uses a 32-bit test instruction with
 # a 0xff immediate.
 t8jccd_long = TailRecipe(
-        't8jccd_long', Branch, size=5 + 6, ins=GPR, outs=(),
+        't8jccd_long', Branch, base_size=5 + 6, ins=GPR, outs=(),
         branch_range=32,
         emit='''
         // test32 r, 0xff.
@@ -1769,7 +1885,7 @@ t8jccd_long = TailRecipe(
 # instruction, so it is limited to the `ABCD` register class for booleans.
 # The omission of a `when_prefixed` alternative is deliberate here.
 icscc = TailRecipe(
-        'icscc', IntCompare, size=1 + 3, ins=(GPR, GPR), outs=ABCD,
+        'icscc', IntCompare, base_size=1 + 3, ins=(GPR, GPR), outs=ABCD,
         emit='''
         // Comparison instruction.
         PUT_OP(bits, rex2(in_reg0, in_reg1), sink);
@@ -1794,7 +1910,7 @@ icscc = TailRecipe(
         ''')
 
 icscc_ib = TailRecipe(
-        'icscc_ib', IntCompareImm, size=2 + 3, ins=GPR, outs=ABCD,
+        'icscc_ib', IntCompareImm, base_size=2 + 3, ins=GPR, outs=ABCD,
         instp=IsSignedInt(IntCompareImm.imm, 8),
         emit='''
         // Comparison instruction.
@@ -1822,7 +1938,7 @@ icscc_ib = TailRecipe(
         ''')
 
 icscc_id = TailRecipe(
-        'icscc_id', IntCompareImm, size=5 + 3, ins=GPR, outs=ABCD,
+        'icscc_id', IntCompareImm, base_size=5 + 3, ins=GPR, outs=ABCD,
         instp=IsSignedInt(IntCompareImm.imm, 32),
         emit='''
         // Comparison instruction.
@@ -1864,7 +1980,7 @@ icscc_id = TailRecipe(
 # Not all floating point condition codes are supported.
 # The omission of a `when_prefixed` alternative is deliberate here.
 fcscc = TailRecipe(
-        'fcscc', FloatCompare, size=1 + 3, ins=(FPR, FPR), outs=ABCD,
+        'fcscc', FloatCompare, base_size=1 + 3, ins=(FPR, FPR), outs=ABCD,
         instp=floatccs(FloatCompare),
         emit='''
         // Comparison instruction.
diff --git a/lib/codegen/meta-python/isa/x86/registers.py b/lib/codegen/meta-python/isa/x86/registers.py
index f463b93a46..3cca0bc377 100644
--- a/lib/codegen/meta-python/isa/x86/registers.py
+++ b/lib/codegen/meta-python/isa/x86/registers.py
@@ -46,14 +46,7 @@ FlagRegs = RegBank(
         names=['rflags'])
 
 GPR = RegClass(IntRegs)
-# Certain types of deref encodings cannot be used with all registers.
-#   R13/RBP cannot be used with zero-offset load or store instructions.
-#   R12 cannot be used with a non-SIB-byte encoding of all derefs.
-GPR_DEREF_SAFE = GPR.without(GPR.rsp, GPR.r12)
-GPR_ZERO_DEREF_SAFE = GPR_DEREF_SAFE.without(GPR.rbp, GPR.r13)
 GPR8 = GPR[0:8]
-GPR8_DEREF_SAFE = GPR8.without(GPR.rsp)
-GPR8_ZERO_DEREF_SAFE = GPR8_DEREF_SAFE.without(GPR.rbp)
 ABCD = GPR[0:4]
 FPR = RegClass(FloatRegs)
 FPR8 = FPR[0:8]
diff --git a/lib/codegen/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs
index 3b45a1c7b3..8616dccbbd 100644
--- a/lib/codegen/src/binemit/relaxation.rs
+++ b/lib/codegen/src/binemit/relaxation.rs
@@ -32,6 +32,7 @@ use cursor::{Cursor, FuncCursor};
 use ir::{Function, InstructionData, Opcode};
 use isa::{EncInfo, TargetIsa};
 use iterators::IteratorExtras;
+use regalloc::RegDiversions;
 use timing;
 use CodegenResult;
 
@@ -51,6 +52,7 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> CodegenResult CodegenResult CodegenResult CodegenResult(&'a self, ebb: Ebb, encinfo: &EncInfo) -> InstOffsetIter<'a> {
+    pub fn inst_offsets<'a>(
+        &'a self,
+        func: &'a Function,
+        ebb: Ebb,
+        encinfo: &EncInfo,
+    ) -> InstOffsetIter<'a> {
         assert!(
             !self.offsets.is_empty(),
             "Code layout must be computed first"
         );
         InstOffsetIter {
             encinfo: encinfo.clone(),
+            func,
+            divert: RegDiversions::new(),
             encodings: &self.encodings,
             offset: self.offsets[ebb],
             iter: self.layout.ebb_insts(ebb),
@@ -226,6 +234,8 @@ impl fmt::Debug for Function {
 /// Iterator returning instruction offsets and sizes: `(offset, inst, size)`.
 pub struct InstOffsetIter<'a> {
     encinfo: EncInfo,
+    divert: RegDiversions,
+    func: &'a Function,
     encodings: &'a InstEncodings,
     offset: CodeOffset,
     iter: ir::layout::Insts<'a>,
@@ -236,10 +246,13 @@ impl<'a> Iterator for InstOffsetIter<'a> {
 
     fn next(&mut self) -> Option {
         self.iter.next().map(|inst| {
-            let size = self.encinfo.bytes(self.encodings[inst]);
+            self.divert.apply(&self.func.dfg[inst]);
+            let byte_size =
+                self.encinfo
+                    .byte_size(self.encodings[inst], inst, &self.divert, self.func);
             let offset = self.offset;
-            self.offset += size;
-            (offset, inst, size)
+            self.offset += byte_size;
+            (offset, inst, byte_size)
         })
     }
 }
diff --git a/lib/codegen/src/ir/stackslot.rs b/lib/codegen/src/ir/stackslot.rs
index 41913ac0f0..c574986d46 100644
--- a/lib/codegen/src/ir/stackslot.rs
+++ b/lib/codegen/src/ir/stackslot.rs
@@ -63,7 +63,7 @@ pub enum StackSlotKind {
     /// An emergency spill slot.
     ///
     /// Emergency slots are allocated late when the register's constraint solver needs extra space
-    /// to shuffle registers around. The are only used briefly, and can be reused.
+    /// to shuffle registers around. They are only used briefly, and can be reused.
     EmergencySlot,
 }
 
diff --git a/lib/codegen/src/isa/encoding.rs b/lib/codegen/src/isa/encoding.rs
index 6bb7e30aec..589069b311 100644
--- a/lib/codegen/src/isa/encoding.rs
+++ b/lib/codegen/src/isa/encoding.rs
@@ -1,7 +1,9 @@
 //! The `Encoding` struct.
 
 use binemit::CodeOffset;
+use ir::{Function, Inst};
 use isa::constraints::{BranchRange, RecipeConstraints};
+use regalloc::RegDiversions;
 use std::fmt;
 
 /// Bits needed to encode an instruction as binary machine code.
@@ -78,12 +80,24 @@ impl fmt::Display for DisplayEncoding {
     }
 }
 
+type SizeCalculatorFn = fn(&RecipeSizing, Inst, &RegDiversions, &Function) -> u8;
+
+/// Returns the base size of the Recipe, assuming it's fixed. This is the default for most
+/// encodings; others can be variable and longer than this base size, depending on the registers
+/// they're using and use a different function, specific per platform.
+pub fn base_size(sizing: &RecipeSizing, _: Inst, _2: &RegDiversions, _3: &Function) -> u8 {
+    sizing.base_size
+}
+
 /// Code size information for an encoding recipe.
 ///
 /// All encoding recipes correspond to an exact instruction size.
 pub struct RecipeSizing {
     /// Size in bytes of instructions encoded with this recipe.
-    pub bytes: u8,
+    pub base_size: u8,
+
+    /// Method computing the real instruction's size, given inputs and outputs.
+    pub compute_size: SizeCalculatorFn,
 
     /// Allowed branch range in this recipe, if any.
     ///
@@ -118,13 +132,20 @@ impl EncInfo {
         }
     }
 
-    /// Get the exact size in bytes of instructions encoded with `enc`.
+    /// Get the precise size in bytes of instructions encoded with `enc`.
     ///
     /// Returns 0 for illegal encodings.
-    pub fn bytes(&self, enc: Encoding) -> CodeOffset {
-        self.sizing
-            .get(enc.recipe())
-            .map_or(0, |s| CodeOffset::from(s.bytes))
+    pub fn byte_size(
+        &self,
+        enc: Encoding,
+        inst: Inst,
+        divert: &RegDiversions,
+        func: &Function,
+    ) -> CodeOffset {
+        self.sizing.get(enc.recipe()).map_or(0, |s| {
+            let compute_size = s.compute_size;
+            CodeOffset::from(compute_size(&s, inst, divert, func))
+        })
     }
 
     /// Get the branch range that is supported by `enc`, if any.
diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs
index c3766346b7..678395f43c 100644
--- a/lib/codegen/src/isa/mod.rs
+++ b/lib/codegen/src/isa/mod.rs
@@ -47,7 +47,7 @@
 //! concurrent function compilations.
 
 pub use isa::constraints::{BranchRange, ConstraintKind, OperandConstraint, RecipeConstraints};
-pub use isa::encoding::{EncInfo, Encoding};
+pub use isa::encoding::{base_size, EncInfo, Encoding};
 pub use isa::registers::{regs_overlap, RegClass, RegClassIndex, RegInfo, RegUnit};
 pub use isa::stack::{StackBase, StackBaseMask, StackRef};
 
@@ -204,7 +204,7 @@ pub trait TargetIsa: fmt::Display {
     /// Get a data structure describing the registers in this ISA.
     fn register_info(&self) -> RegInfo;
 
-    /// Returns an iterartor over legal encodings for the instruction.
+    /// Returns an iterator over legal encodings for the instruction.
     fn legal_encodings<'a>(
         &'a self,
         func: &'a ir::Function,
diff --git a/lib/codegen/src/isa/riscv/enc_tables.rs b/lib/codegen/src/isa/riscv/enc_tables.rs
index 5f7c084a67..bbb7492805 100644
--- a/lib/codegen/src/isa/riscv/enc_tables.rs
+++ b/lib/codegen/src/isa/riscv/enc_tables.rs
@@ -5,7 +5,7 @@ use ir;
 use isa;
 use isa::constraints::*;
 use isa::enc_tables::*;
-use isa::encoding::RecipeSizing;
+use isa::encoding::{base_size, RecipeSizing};
 
 // Include the generated encoding tables:
 // - `LEVEL1_RV32`
diff --git a/lib/codegen/src/isa/x86/binemit.rs b/lib/codegen/src/isa/x86/binemit.rs
index 2389595b2e..c930af39fd 100644
--- a/lib/codegen/src/isa/x86/binemit.rs
+++ b/lib/codegen/src/isa/x86/binemit.rs
@@ -1,5 +1,6 @@
 //! Emitting binary x86 machine code.
 
+use super::enc_tables::{needs_offset, needs_sib_byte};
 use super::registers::RU;
 use binemit::{bad_encoding, CodeSink, Reloc};
 use ir::condcodes::{CondCode, FloatCC, IntCC};
diff --git a/lib/codegen/src/isa/x86/enc_tables.rs b/lib/codegen/src/isa/x86/enc_tables.rs
index 65b7d7c38e..9c52563a27 100644
--- a/lib/codegen/src/isa/x86/enc_tables.rs
+++ b/lib/codegen/src/isa/x86/enc_tables.rs
@@ -5,15 +5,73 @@ use bitset::BitSet;
 use cursor::{Cursor, FuncCursor};
 use flowgraph::ControlFlowGraph;
 use ir::condcodes::IntCC;
-use ir::{self, InstBuilder};
+use ir::{self, Function, Inst, InstBuilder};
 use isa;
 use isa::constraints::*;
 use isa::enc_tables::*;
+use isa::encoding::base_size;
 use isa::encoding::RecipeSizing;
+use isa::RegUnit;
+use regalloc::RegDiversions;
 
 include!(concat!(env!("OUT_DIR"), "/encoding-x86.rs"));
 include!(concat!(env!("OUT_DIR"), "/legalize-x86.rs"));
 
+pub fn needs_sib_byte(reg: RegUnit) -> bool {
+    reg == RU::r12 as RegUnit || reg == RU::rsp as RegUnit
+}
+pub fn needs_offset(reg: RegUnit) -> bool {
+    reg == RU::r13 as RegUnit || reg == RU::rbp as RegUnit
+}
+
+fn additional_size_if(
+    op_index: usize,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+    condition_func: fn(RegUnit) -> bool,
+) -> u8 {
+    let addr_reg = divert.reg(func.dfg.inst_args(inst)[op_index], &func.locations);
+    if condition_func(addr_reg) {
+        1
+    } else {
+        0
+    }
+}
+
+fn size_plus_maybe_offset_for_in_reg_0(
+    sizing: &RecipeSizing,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    sizing.base_size + additional_size_if(0, inst, divert, func, needs_offset)
+}
+fn size_plus_maybe_offset_for_in_reg_1(
+    sizing: &RecipeSizing,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    sizing.base_size + additional_size_if(1, inst, divert, func, needs_offset)
+}
+fn size_plus_maybe_sib_for_in_reg_0(
+    sizing: &RecipeSizing,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    sizing.base_size + additional_size_if(0, inst, divert, func, needs_sib_byte)
+}
+fn size_plus_maybe_sib_for_in_reg_1(
+    sizing: &RecipeSizing,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    sizing.base_size + additional_size_if(1, inst, divert, func, needs_sib_byte)
+}
+
 /// Expand the `sdiv` and `srem` instructions using `x86_sdivmodx`.
 fn expand_sdivrem(
     inst: ir::Inst,
diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs
index d8dfcb617d..da0f0eec09 100644
--- a/lib/filetests/src/test_binemit.rs
+++ b/lib/filetests/src/test_binemit.rs
@@ -149,7 +149,8 @@ impl SubTest for TestBinEmit {
 
                         if opt_level == OptLevel::Best {
                             // Get the smallest legal encoding
-                            legal_encodings.min_by_key(|&e| encinfo.bytes(e))
+                            legal_encodings
+                                .min_by_key(|&e| encinfo.byte_size(e, inst, &divert, &func))
                         } else {
                             // If not optimizing, just use the first encoding.
                             legal_encodings.next()
@@ -204,7 +205,7 @@ impl SubTest for TestBinEmit {
                 "Inconsistent {} header offset",
                 ebb
             );
-            for (offset, inst, enc_bytes) in func.inst_offsets(ebb, &encinfo) {
+            for (offset, inst, enc_bytes) in func.inst_offsets(&func, ebb, &encinfo) {
                 assert_eq!(sink.offset, offset);
                 sink.text.clear();
                 let enc = func.encodings[inst];

From 58229e10bfc18a49b7c10cd3bb648191a6ed5aa1 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 15 Oct 2018 16:01:49 -0700
Subject: [PATCH 2131/3084] Pin mypy to 0.630.

Pin mypy to 0.630 to work around errors reported in mypy 0.641.
---
 .travis.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index a62423fb7a..7113ae2d66 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -23,7 +23,9 @@ addons:
         packages:
             - python3-pip
 install:
-    - pip3 install --user --upgrade mypy flake8
+    # Fix the mypy version at 0.630 since version 0.641 is more strict and the
+    # code is not yet updated.
+    - pip3 install --user --upgrade mypy==0.630 flake8
     - mypy --version
 before_script:
     # If an old version of rustfmt from cargo is already installed, uninstall

From af0a239539e2e08222f8daa14ae7cb18b3a016c1 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 15 Oct 2018 20:17:25 +0200
Subject: [PATCH 2132/3084] Revive the -T aka --time-passes argument to report
 run times on the CLI;

---
 cranelift/src/clif-util.rs  |  9 +++++++--
 cranelift/src/compile.rs    | 15 ++++++++++++++-
 cranelift/src/wasm.rs       |  8 ++++++++
 lib/filetests/src/lib.rs    | 21 ++++++++++++++++-----
 lib/filetests/src/runner.rs | 10 +++++++++-
 5 files changed, 54 insertions(+), 9 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index b38d4c587a..13a6ba7d06 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -199,10 +199,12 @@ fn main() {
         }
         ("test", Some(rest_cmd)) => {
             handle_debug_flag(rest_cmd.is_present("debug"));
-            cranelift_filetests::run(
+            let result = cranelift_filetests::run(
                 rest_cmd.is_present("verbose"),
+                rest_cmd.is_present("time-passes"),
                 &get_vec(rest_cmd.values_of("file")),
-            ).map(|_time| ())
+            ).map(|_time| ());
+            result
         }
         ("pass", Some(rest_cmd)) => {
             handle_debug_flag(rest_cmd.is_present("debug"));
@@ -215,6 +217,7 @@ fn main() {
             // Can be unwrapped because 'single-file' is required
             cranelift_filetests::run_passes(
                 rest_cmd.is_present("verbose"),
+                rest_cmd.is_present("time-passes"),
                 &get_vec(rest_cmd.values_of("pass")),
                 target_val,
                 rest_cmd.value_of("single-file").unwrap(),
@@ -235,6 +238,7 @@ fn main() {
             compile::run(
                 get_vec(rest_cmd.values_of("file")),
                 rest_cmd.is_present("print"),
+                rest_cmd.is_present("time-passes"),
                 &get_vec(rest_cmd.values_of("set")),
                 target_val,
             )
@@ -257,6 +261,7 @@ fn main() {
                 &get_vec(rest_cmd.values_of("set")),
                 target_val,
                 rest_cmd.is_present("print-size"),
+                rest_cmd.is_present("time-passes"),
             );
 
             #[cfg(not(feature = "wasm"))]
diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs
index 5a93daa5a2..59402ff97f 100644
--- a/cranelift/src/compile.rs
+++ b/cranelift/src/compile.rs
@@ -3,6 +3,7 @@
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_codegen::settings::FlagsOrIsa;
+use cranelift_codegen::timing;
 use cranelift_codegen::Context;
 use cranelift_codegen::{binemit, ir};
 use cranelift_reader::parse_test;
@@ -60,6 +61,7 @@ impl binemit::TrapSink for PrintTraps {
 pub fn run(
     files: Vec,
     flag_print: bool,
+    flag_report_times: bool,
     flag_set: &[String],
     flag_isa: &str,
 ) -> Result<(), String> {
@@ -68,13 +70,20 @@ pub fn run(
     for filename in files {
         let path = Path::new(&filename);
         let name = String::from(path.as_os_str().to_string_lossy());
-        handle_module(flag_print, &path.to_path_buf(), &name, parsed.as_fisa())?;
+        handle_module(
+            flag_print,
+            flag_report_times,
+            &path.to_path_buf(),
+            &name,
+            parsed.as_fisa(),
+        )?;
     }
     Ok(())
 }
 
 fn handle_module(
     flag_print: bool,
+    flag_report_times: bool,
     path: &PathBuf,
     name: &str,
     fisa: FlagsOrIsa,
@@ -134,6 +143,10 @@ fn handle_module(
         }
     }
 
+    if flag_report_times {
+        print!("{}", timing::take_current());
+    }
+
     Ok(())
 }
 
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 3e91c3e7b0..3dd1269980 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -10,6 +10,7 @@
 use cranelift_codegen::print_errors::{pretty_error, pretty_verifier_error};
 use cranelift_codegen::settings::FlagsOrIsa;
 use cranelift_codegen::Context;
+use cranelift_codegen::timing;
 use cranelift_entity::EntityRef;
 use cranelift_wasm::{
     translate_module, DummyEnvironment, FuncIndex, ModuleEnvironment, ReturnMode,
@@ -46,6 +47,7 @@ pub fn run(
     flag_set: &[String],
     flag_triple: &str,
     flag_print_size: bool,
+    flag_report_times: bool,
 ) -> Result<(), String> {
     let parsed = parse_sets_and_triple(flag_set, flag_triple)?;
 
@@ -58,6 +60,7 @@ pub fn run(
             flag_check_translation,
             flag_print,
             flag_print_size,
+            flag_report_times,
             &path.to_path_buf(),
             &name,
             parsed.as_fisa(),
@@ -72,6 +75,7 @@ fn handle_module(
     flag_check_translation: bool,
     flag_print: bool,
     flag_print_size: bool,
+    flag_report_times: bool,
     path: &PathBuf,
     name: &str,
     fisa: FlagsOrIsa,
@@ -209,6 +213,10 @@ fn handle_module(
         println!("Total module bytecode size: {} bytes", total_bytecode_size);
     }
 
+    if flag_report_times {
+        println!("{}", timing::take_current());
+    }
+
     let _ = terminal.fg(term::color::GREEN);
     vprintln!(flag_verbose, "ok");
     let _ = terminal.reset();
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index 6d48eb9636..0032bc59db 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -35,6 +35,7 @@ extern crate num_cpus;
 #[macro_use]
 extern crate log;
 
+use cranelift_codegen::timing;
 use cranelift_reader::TestCommand;
 use runner::TestRunner;
 use std::path::Path;
@@ -73,8 +74,8 @@ type TestResult = Result;
 /// Directories are scanned recursively for test cases ending in `.clif`. These test cases are
 /// executed on background threads.
 ///
-pub fn run(verbose: bool, files: &[String]) -> TestResult {
-    let mut runner = TestRunner::new(verbose);
+pub fn run(verbose: bool, report_times: bool, files: &[String]) -> TestResult {
+    let mut runner = TestRunner::new(verbose, report_times);
 
     for path in files.iter().map(Path::new) {
         if path.is_file() {
@@ -93,8 +94,14 @@ pub fn run(verbose: bool, files: &[String]) -> TestResult {
 ///
 /// Directories are scanned recursively for test cases ending in `.clif`.
 ///
-pub fn run_passes(verbose: bool, passes: &[String], target: &str, file: &str) -> TestResult {
-    let mut runner = TestRunner::new(verbose);
+pub fn run_passes(
+    verbose: bool,
+    report_times: bool,
+    passes: &[String],
+    target: &str,
+    file: &str,
+) -> TestResult {
+    let mut runner = TestRunner::new(verbose, /* report_times */ false);
 
     let path = Path::new(file);
     if path == Path::new("-") || path.is_file() {
@@ -103,7 +110,11 @@ pub fn run_passes(verbose: bool, passes: &[String], target: &str, file: &str) ->
         runner.push_dir(path);
     }
 
-    runner.run_passes(passes, target)
+    let result = runner.run_passes(passes, target);
+    if report_times {
+        println!("{}", timing::take_current());
+    }
+    result
 }
 
 /// Create a new subcommand trait object to match `parsed.command`.
diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs
index ff6f203590..7840c0dfff 100644
--- a/lib/filetests/src/runner.rs
+++ b/lib/filetests/src/runner.rs
@@ -4,6 +4,7 @@
 //! scanning directories for tests.
 
 use concurrent::{ConcurrentRunner, Reply};
+use cranelift_codegen::timing;
 use std::error::Error;
 use std::ffi::OsStr;
 use std::fmt::{self, Display};
@@ -62,6 +63,9 @@ impl Display for QueueEntry {
 pub struct TestRunner {
     verbose: bool,
 
+    // Should we print the timings out?
+    report_times: bool,
+
     // Directories that have not yet been scanned.
     dir_stack: Vec,
 
@@ -85,9 +89,10 @@ pub struct TestRunner {
 
 impl TestRunner {
     /// Create a new blank TrstRunner.
-    pub fn new(verbose: bool) -> Self {
+    pub fn new(verbose: bool, report_times: bool) -> Self {
         Self {
             verbose,
+            report_times,
             dir_stack: Vec::new(),
             tests: Vec::new(),
             new_tests: 0,
@@ -300,6 +305,9 @@ impl TestRunner {
                 }
             }
             conc.join();
+            if self.report_times {
+                println!("{}", timing::take_current());
+            }
         }
     }
 

From e10a3434b8d84874bc7369bb8f73e52e1682e856 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 16 Oct 2018 10:03:48 -0700
Subject: [PATCH 2133/3084] Update test files for the cranelift_filetests::run
 API change.

---
 cranelift/tests/filetests.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/cranelift/tests/filetests.rs b/cranelift/tests/filetests.rs
index 5eb4ca3638..89646a2d3d 100644
--- a/cranelift/tests/filetests.rs
+++ b/cranelift/tests/filetests.rs
@@ -3,5 +3,6 @@ extern crate cranelift_filetests;
 #[test]
 fn filetests() {
     // Run all the filetests in the following directories.
-    cranelift_filetests::run(false, &["filetests".into(), "docs".into()]).expect("test harness");
+    cranelift_filetests::run(false, false, &["filetests".into(), "docs".into()])
+        .expect("test harness");
 }

From 709eed21c1ae35b362be741afbf8a087cf15b24a Mon Sep 17 00:00:00 2001
From: oooooba 
Date: Sat, 20 Oct 2018 02:49:41 +0900
Subject: [PATCH 2134/3084] Use types to represent wasm
 global/table/memory/signature indices (#560)

* Use a type to represent wasm table indices.

* Use a type to represent wasm global variable indices.

* Use a type to represent wasm memory indices.

* Use a type to represent wasm signature indices.

* Use PrimaryMap instead of Vec to protect against using wrong indices.
---
 lib/wasm/src/code_translator.rs     |  8 ++---
 lib/wasm/src/environ/dummy.rs       | 18 +++++------
 lib/wasm/src/sections_translator.rs | 48 ++++++++++++++++++++---------
 lib/wasm/src/state.rs               |  8 ++---
 lib/wasm/src/translation_utils.rs   | 27 +++++++++++-----
 5 files changed, 70 insertions(+), 39 deletions(-)

diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index 152c20ca20..e7a62b65fe 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -380,9 +380,9 @@ pub fn translate_operator(
             let callee = state.pop1();
             let call = environ.translate_call_indirect(
                 builder.cursor(),
-                table_index as TableIndex,
+                TableIndex::new(table_index as usize),
                 table,
-                index as SignatureIndex,
+                SignatureIndex::new(index as usize),
                 sigref,
                 callee,
                 state.peekn(num_args),
@@ -403,13 +403,13 @@ pub fn translate_operator(
         Operator::MemoryGrow { reserved } => {
             // The WebAssembly MVP only supports one linear memory, but we expect the reserved
             // argument to be a memory index.
-            let heap_index = reserved as MemoryIndex;
+            let heap_index = MemoryIndex::new(reserved as usize);
             let heap = state.get_heap(builder.func, reserved, environ);
             let val = state.pop1();
             state.push1(environ.translate_memory_grow(builder.cursor(), heap_index, heap, val)?)
         }
         Operator::MemorySize { reserved } => {
-            let heap_index = reserved as MemoryIndex;
+            let heap_index = MemoryIndex::new(reserved as usize);
             let heap = state.get_heap(builder.func, reserved, environ);
             state.push1(environ.translate_memory_size(builder.cursor(), heap_index, heap)?);
         }
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index ae0edb96e3..5608abec6d 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -51,7 +51,7 @@ pub struct DummyModuleInfo {
     pub flags: settings::Flags,
 
     /// Signatures as provided by `declare_signature`.
-    pub signatures: Vec,
+    pub signatures: PrimaryMap,
 
     /// Module and field names of imported functions as provided by `declare_func_import`.
     pub imported_funcs: Vec<(String, String)>,
@@ -63,13 +63,13 @@ pub struct DummyModuleInfo {
     pub function_bodies: PrimaryMap,
 
     /// Tables as provided by `declare_table`.
-    pub tables: Vec>,
+    pub tables: PrimaryMap>,
 
     /// Memories as provided by `declare_memory`.
-    pub memories: Vec>,
+    pub memories: PrimaryMap>,
 
     /// Globals as provided by `declare_global`.
-    pub globals: Vec>,
+    pub globals: PrimaryMap>,
 
     /// The start function.
     pub start_func: Option,
@@ -81,13 +81,13 @@ impl DummyModuleInfo {
         Self {
             triple,
             flags,
-            signatures: Vec::new(),
+            signatures: PrimaryMap::new(),
             imported_funcs: Vec::new(),
             functions: PrimaryMap::new(),
             function_bodies: PrimaryMap::new(),
-            tables: Vec::new(),
-            memories: Vec::new(),
-            globals: Vec::new(),
+            tables: PrimaryMap::new(),
+            memories: PrimaryMap::new(),
+            globals: PrimaryMap::new(),
             start_func: None,
         }
     }
@@ -179,7 +179,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
 
     fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalVariable {
         // Just create a dummy `vmctx` global.
-        let offset = ((index * 8) as i64 + 8).into();
+        let offset = ((index.index() * 8) as i64 + 8).into();
         let vmctx = func.create_global_value(ir::GlobalValueData::VMContext {});
         let iadd = func.create_global_value(ir::GlobalValueData::IAddImm {
             base: vmctx,
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 1ea3ce26ec..fa4e288d04 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -72,7 +72,11 @@ pub fn parse_import_section<'data>(
                 // becomes a concern here.
                 let module_name = from_utf8(module).unwrap();
                 let field_name = from_utf8(field).unwrap();
-                environ.declare_func_import(sig as SignatureIndex, module_name, field_name);
+                environ.declare_func_import(
+                    SignatureIndex::new(sig as usize),
+                    module_name,
+                    field_name,
+                );
             }
             ParserState::ImportSectionEntry {
                 ty:
@@ -125,7 +129,7 @@ pub fn parse_function_section(
     loop {
         match *parser.read() {
             ParserState::FunctionSectionEntry(sigindex) => {
-                environ.declare_func_type(sigindex as SignatureIndex);
+                environ.declare_func_type(SignatureIndex::new(sigindex as usize));
             }
             ParserState::EndSection => break,
             ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
@@ -151,12 +155,20 @@ pub fn parse_export_section<'data>(
                 // assume valid UTF-8 and use `from_utf8_unchecked` if performance
                 // becomes a concern here.
                 let name = from_utf8(field).unwrap();
-                let func_index = FuncIndex::new(index as usize);
+                let index = index as usize;
                 match *kind {
-                    ExternalKind::Function => environ.declare_func_export(func_index, name),
-                    ExternalKind::Table => environ.declare_table_export(func_index.index(), name),
-                    ExternalKind::Memory => environ.declare_memory_export(func_index.index(), name),
-                    ExternalKind::Global => environ.declare_global_export(func_index.index(), name),
+                    ExternalKind::Function => {
+                        environ.declare_func_export(FuncIndex::new(index), name)
+                    }
+                    ExternalKind::Table => {
+                        environ.declare_table_export(TableIndex::new(index), name)
+                    }
+                    ExternalKind::Memory => {
+                        environ.declare_memory_export(MemoryIndex::new(index), name)
+                    }
+                    ExternalKind::Global => {
+                        environ.declare_global_export(GlobalIndex::new(index), name)
+                    }
                 }
             }
             ParserState::EndSection => break,
@@ -235,7 +247,7 @@ pub fn parse_global_section(
                 GlobalInit::F64Const(value.bits())
             }
             ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => {
-                GlobalInit::GlobalRef(global_index as GlobalIndex)
+                GlobalInit::GlobalRef(GlobalIndex::new(global_index as usize))
             }
             ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
             ref s => panic!("unexpected section content: {:?}", s),
@@ -281,9 +293,12 @@ pub fn parse_data_section<'data>(
                 (None, value as u32 as usize)
             }
             ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => {
-                match environ.get_global(global_index as GlobalIndex).initializer {
+                match environ
+                    .get_global(GlobalIndex::new(global_index as usize))
+                    .initializer
+                {
                     GlobalInit::I32Const(value) => (None, value as u32 as usize),
-                    GlobalInit::Import() => (Some(global_index as GlobalIndex), 0),
+                    GlobalInit::Import() => (Some(GlobalIndex::new(global_index as usize)), 0),
                     _ => panic!("should not happen"),
                 }
             }
@@ -309,7 +324,7 @@ pub fn parse_data_section<'data>(
                 ref s => panic!("unexpected section content: {:?}", s),
             };
             environ.declare_data_initialization(
-                memory_index as MemoryIndex,
+                MemoryIndex::new(memory_index as usize),
                 base,
                 running_offset,
                 data,
@@ -352,7 +367,9 @@ pub fn parse_elements_section(
 ) -> WasmResult<()> {
     loop {
         let table_index = match *parser.read() {
-            ParserState::BeginElementSectionEntry(table_index) => table_index as TableIndex,
+            ParserState::BeginElementSectionEntry(table_index) => {
+                TableIndex::new(table_index as usize)
+            }
             ParserState::EndSection => break,
             ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
             ref s => panic!("unexpected section content: {:?}", s),
@@ -367,9 +384,12 @@ pub fn parse_elements_section(
                 (None, value as u32 as usize)
             }
             ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => {
-                match environ.get_global(global_index as GlobalIndex).initializer {
+                match environ
+                    .get_global(GlobalIndex::new(global_index as usize))
+                    .initializer
+                {
                     GlobalInit::I32Const(value) => (None, value as u32 as usize),
-                    GlobalInit::Import() => (Some(global_index as GlobalIndex), 0),
+                    GlobalInit::Import() => (Some(GlobalIndex::new(global_index as usize)), 0),
                     _ => panic!("should not happen"),
                 }
             }
diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs
index 2d6d3610af..df909d8b38 100644
--- a/lib/wasm/src/state.rs
+++ b/lib/wasm/src/state.rs
@@ -287,7 +287,7 @@ impl TranslationState {
         index: u32,
         environ: &mut FE,
     ) -> GlobalVariable {
-        let index = index as GlobalIndex;
+        let index = GlobalIndex::new(index as usize);
         *self
             .globals
             .entry(index)
@@ -302,7 +302,7 @@ impl TranslationState {
         index: u32,
         environ: &mut FE,
     ) -> ir::Heap {
-        let index = index as MemoryIndex;
+        let index = MemoryIndex::new(index as usize);
         *self
             .heaps
             .entry(index)
@@ -317,7 +317,7 @@ impl TranslationState {
         index: u32,
         environ: &mut FE,
     ) -> ir::Table {
-        let index = index as TableIndex;
+        let index = TableIndex::new(index as usize);
         *self
             .tables
             .entry(index)
@@ -334,7 +334,7 @@ impl TranslationState {
         index: u32,
         environ: &mut FE,
     ) -> (ir::SigRef, usize) {
-        let index = index as SignatureIndex;
+        let index = SignatureIndex::new(index as usize);
         *self.signatures.entry(index).or_insert_with(|| {
             let sig = environ.make_indirect_sig(func, index);
             (sig, normal_args(&func.dfg.signatures[sig]))
diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index 5d40539720..d76d55494f 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -13,14 +13,25 @@ entity_impl!(FuncIndex);
 pub struct DefinedFuncIndex(u32);
 entity_impl!(DefinedFuncIndex);
 
-/// Index of a table (imported or defined) inside the WebAssembly module.
-pub type TableIndex = usize;
-/// Index of a global variable (imported or defined) inside the WebAssembly module.
-pub type GlobalIndex = usize;
-/// Index of a linear memory (imported or defined) inside the WebAssembly module.
-pub type MemoryIndex = usize;
-/// Index of a signature (imported or defined) inside the WebAssembly module.
-pub type SignatureIndex = usize;
+/// Index type of a table (imported or defined) inside the WebAssembly module.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
+pub struct TableIndex(u32);
+entity_impl!(TableIndex);
+
+/// Index type of a global variable (imported or defined) inside the WebAssembly module.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
+pub struct GlobalIndex(u32);
+entity_impl!(GlobalIndex);
+
+/// Index type of a linear memory (imported or defined) inside the WebAssembly module.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
+pub struct MemoryIndex(u32);
+entity_impl!(MemoryIndex);
+
+/// Index type of a signature (imported or defined) inside the WebAssembly module.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
+pub struct SignatureIndex(u32);
+entity_impl!(SignatureIndex);
 
 /// WebAssembly global.
 #[derive(Debug, Clone, Copy)]

From 54ab1ea5334509a2dcb21697fd3d06ce05c96582 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 28 Sep 2018 22:56:39 -0700
Subject: [PATCH 2135/3084] Optimize load/store with an iadd_imm operand.

Fold the immediate into the load/store offset when possible.
---
 .../postopt/fold_offset_into_address.clif     |  18 ++
 lib/codegen/src/ir/immediates.rs              |  10 +
 lib/codegen/src/postopt.rs                    | 220 +++++++++++-------
 3 files changed, 159 insertions(+), 89 deletions(-)
 create mode 100644 cranelift/filetests/postopt/fold_offset_into_address.clif

diff --git a/cranelift/filetests/postopt/fold_offset_into_address.clif b/cranelift/filetests/postopt/fold_offset_into_address.clif
new file mode 100644
index 0000000000..45ce7bcd0a
--- /dev/null
+++ b/cranelift/filetests/postopt/fold_offset_into_address.clif
@@ -0,0 +1,18 @@
+test postopt
+target x86_64
+
+; Fold the immediate of an iadd_imm into an address offset.
+
+function u0:0(i64 vmctx) -> i64 {
+ebb0(v0: i64):
+                     v1 = iadd_imm.i64 v0, 16
+[RexOp1ldDisp8#808b] v2 = load.i64 notrap aligned v1
+[Op1ret#c3]          return v2
+}
+
+; sameln: function u0:0(i64 vmctx) -> i64 fast {
+; nextln: ebb0(v0: i64):
+; nextln:                                     v1 = iadd_imm v0, 16
+; nextln: [RexOp1ldDisp8#808b]                v2 = load.i64 notrap aligned v0+16
+; nextln: [Op1ret#c3]                         return v2
+; nextln: }
diff --git a/lib/codegen/src/ir/immediates.rs b/lib/codegen/src/ir/immediates.rs
index 2db7123942..3d8247845a 100644
--- a/lib/codegen/src/ir/immediates.rs
+++ b/lib/codegen/src/ir/immediates.rs
@@ -224,6 +224,16 @@ impl Offset32 {
             None
         }
     }
+
+    /// Add in the signed number `x` if possible.
+    pub fn try_add_i64(self, x: i64) -> Option {
+        let casted = x as i32;
+        if casted as i64 == x {
+            self.0.checked_add(casted).map(Self::new)
+        } else {
+            None
+        }
+    }
 }
 
 impl Into for Offset32 {
diff --git a/lib/codegen/src/postopt.rs b/lib/codegen/src/postopt.rs
index fdfc358697..8c64b9bb04 100644
--- a/lib/codegen/src/postopt.rs
+++ b/lib/codegen/src/postopt.rs
@@ -179,6 +179,7 @@ struct MemOpInfo {
 }
 
 fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa) {
+    // Look for simple loads and stores we can optimize.
     let info = match pos.func.dfg[inst] {
         InstructionData::Load {
             opcode,
@@ -209,103 +210,144 @@ fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa)
         _ => return,
     };
 
-    let add_args = if let ValueDef::Result(result_inst, _) = pos.func.dfg.value_def(info.arg) {
+    // Examine the instruction that defines the address operand.
+    if let ValueDef::Result(result_inst, _) = pos.func.dfg.value_def(info.arg) {
         match pos.func.dfg[result_inst] {
             InstructionData::Binary {
                 opcode: Opcode::Iadd,
                 args,
-            } => args,
-            _ => return,
+            } => match info.opcode {
+                // Operand is an iadd. Fold it into a memory address with a complex address mode.
+                Opcode::Load => {
+                    pos.func.dfg.replace(inst).load_complex(
+                        info.itype,
+                        info.flags,
+                        &args,
+                        info.offset,
+                    );
+                }
+                Opcode::Uload8 => {
+                    pos.func.dfg.replace(inst).uload8_complex(
+                        info.itype,
+                        info.flags,
+                        &args,
+                        info.offset,
+                    );
+                }
+                Opcode::Sload8 => {
+                    pos.func.dfg.replace(inst).sload8_complex(
+                        info.itype,
+                        info.flags,
+                        &args,
+                        info.offset,
+                    );
+                }
+                Opcode::Uload16 => {
+                    pos.func.dfg.replace(inst).uload16_complex(
+                        info.itype,
+                        info.flags,
+                        &args,
+                        info.offset,
+                    );
+                }
+                Opcode::Sload16 => {
+                    pos.func.dfg.replace(inst).sload16_complex(
+                        info.itype,
+                        info.flags,
+                        &args,
+                        info.offset,
+                    );
+                }
+                Opcode::Uload32 => {
+                    pos.func
+                        .dfg
+                        .replace(inst)
+                        .uload32_complex(info.flags, &args, info.offset);
+                }
+                Opcode::Sload32 => {
+                    pos.func
+                        .dfg
+                        .replace(inst)
+                        .sload32_complex(info.flags, &args, info.offset);
+                }
+                Opcode::Store => {
+                    pos.func.dfg.replace(inst).store_complex(
+                        info.flags,
+                        info.st_arg.unwrap(),
+                        &args,
+                        info.offset,
+                    );
+                }
+                Opcode::Istore8 => {
+                    pos.func.dfg.replace(inst).istore8_complex(
+                        info.flags,
+                        info.st_arg.unwrap(),
+                        &args,
+                        info.offset,
+                    );
+                }
+                Opcode::Istore16 => {
+                    pos.func.dfg.replace(inst).istore16_complex(
+                        info.flags,
+                        info.st_arg.unwrap(),
+                        &args,
+                        info.offset,
+                    );
+                }
+                Opcode::Istore32 => {
+                    pos.func.dfg.replace(inst).istore32_complex(
+                        info.flags,
+                        info.st_arg.unwrap(),
+                        &args,
+                        info.offset,
+                    );
+                }
+                _ => panic!("Unsupported load or store opcode"),
+            },
+            InstructionData::BinaryImm {
+                opcode: Opcode::IaddImm,
+                arg,
+                imm,
+            } => match pos.func.dfg[inst] {
+                // Operand is an iadd_imm. Fold the immediate into the offset if possible.
+                InstructionData::Load {
+                    arg: ref mut load_arg,
+                    ref mut offset,
+                    ..
+                } => {
+                    if let Some(imm) = offset.try_add_i64(imm.into()) {
+                        *load_arg = arg;
+                        *offset = imm;
+                    } else {
+                        // Overflow.
+                        return;
+                    }
+                }
+                InstructionData::Store {
+                    args: ref mut store_args,
+                    ref mut offset,
+                    ..
+                } => {
+                    if let Some(imm) = offset.try_add_i64(imm.into()) {
+                        store_args[0] = arg;
+                        *offset = imm;
+                    } else {
+                        // Overflow.
+                        return;
+                    }
+                }
+                _ => panic!(),
+            },
+            _ => {
+                // Address value is defined by some other kind of instruction.
+                return;
+            }
         }
     } else {
+        // Address value is not the result of an instruction.
         return;
-    };
-
-    match info.opcode {
-        Opcode::Load => {
-            pos.func
-                .dfg
-                .replace(inst)
-                .load_complex(info.itype, info.flags, &add_args, info.offset);
-        }
-        Opcode::Uload8 => {
-            pos.func.dfg.replace(inst).uload8_complex(
-                info.itype,
-                info.flags,
-                &add_args,
-                info.offset,
-            );
-        }
-        Opcode::Sload8 => {
-            pos.func.dfg.replace(inst).sload8_complex(
-                info.itype,
-                info.flags,
-                &add_args,
-                info.offset,
-            );
-        }
-        Opcode::Uload16 => {
-            pos.func.dfg.replace(inst).uload16_complex(
-                info.itype,
-                info.flags,
-                &add_args,
-                info.offset,
-            );
-        }
-        Opcode::Sload16 => {
-            pos.func.dfg.replace(inst).sload16_complex(
-                info.itype,
-                info.flags,
-                &add_args,
-                info.offset,
-            );
-        }
-        Opcode::Uload32 => {
-            pos.func
-                .dfg
-                .replace(inst)
-                .uload32_complex(info.flags, &add_args, info.offset);
-        }
-        Opcode::Sload32 => {
-            pos.func
-                .dfg
-                .replace(inst)
-                .sload32_complex(info.flags, &add_args, info.offset);
-        }
-        Opcode::Store => {
-            pos.func.dfg.replace(inst).store_complex(
-                info.flags,
-                info.st_arg.unwrap(),
-                &add_args,
-                info.offset,
-            );
-        }
-        Opcode::Istore8 => {
-            pos.func.dfg.replace(inst).istore8_complex(
-                info.flags,
-                info.st_arg.unwrap(),
-                &add_args,
-                info.offset,
-            );
-        }
-        Opcode::Istore16 => {
-            pos.func.dfg.replace(inst).istore16_complex(
-                info.flags,
-                info.st_arg.unwrap(),
-                &add_args,
-                info.offset,
-            );
-        }
-        Opcode::Istore32 => {
-            pos.func.dfg.replace(inst).istore32_complex(
-                info.flags,
-                info.st_arg.unwrap(),
-                &add_args,
-                info.offset,
-            );
-        }
-        _ => return,
     }
+
     let ok = pos.func.update_encoding(inst, isa).is_ok();
     debug_assert!(ok);
 }

From 8cd1b879175067cd019ec79a951da4b448d801f5 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 18 Oct 2018 22:44:51 -0700
Subject: [PATCH 2136/3084] Rename parse_elements_section.

In WebAssembly documentation, it's the "element" section.
---
 lib/wasm/src/module_translator.rs   | 4 ++--
 lib/wasm/src/sections_translator.rs | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs
index 023d14dba1..6b7c854f27 100644
--- a/lib/wasm/src/module_translator.rs
+++ b/lib/wasm/src/module_translator.rs
@@ -3,7 +3,7 @@
 use cranelift_codegen::timing;
 use environ::{ModuleEnvironment, WasmError, WasmResult};
 use sections_translator::{
-    parse_code_section, parse_data_section, parse_elements_section, parse_export_section,
+    parse_code_section, parse_data_section, parse_element_section, parse_export_section,
     parse_function_section, parse_function_signatures, parse_global_section, parse_import_section,
     parse_memory_section, parse_start_section, parse_table_section,
 };
@@ -88,7 +88,7 @@ pub fn translate_module<'data>(
                 code: SectionCode::Element,
                 ..
             } => {
-                parse_elements_section(&mut parser, environ)?;
+                parse_element_section(&mut parser, environ)?;
                 next_input = ParserInput::Default;
             }
             ParserState::BeginSection {
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index fa4e288d04..1a09033705 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -360,8 +360,8 @@ pub fn parse_table_section(parser: &mut Parser, environ: &mut ModuleEnvironment)
     Ok(())
 }
 
-/// Retrieves the tables from the table section
-pub fn parse_elements_section(
+/// Retrieves the elements from the element section
+pub fn parse_element_section(
     parser: &mut Parser,
     environ: &mut ModuleEnvironment,
 ) -> WasmResult<()> {

From fcb1151dd17f8ec66e615b5fd879b7830b55270b Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 19 Oct 2018 11:05:35 -0700
Subject: [PATCH 2137/3084] Add a badge indicating Rust 1.25 support.

---
 cranelift/README.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cranelift/README.md b/cranelift/README.md
index c58e48a9cc..8e65e72ebf 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -9,6 +9,7 @@ into executable machine code.
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
 [![Build Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
+![Minimum rustc 1.25](https://img.shields.io/badge/rustc-1.25+-red.svg)
 
 For more information, see [the
 documentation](https://cranelift.readthedocs.io/en/latest/?badge=latest).

From 3ec21459c5039bfa74a87a4b2017e9d831a434ab Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 4 Oct 2018 10:56:05 +0200
Subject: [PATCH 2138/3084] Remove verify! macro that's used only once;

---
 lib/codegen/src/verifier/mod.rs | 59 +++++----------------------------
 1 file changed, 8 insertions(+), 51 deletions(-)

diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 0219b18db4..215508ce18 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -120,53 +120,6 @@ macro_rules! nonfatal {
     });
 }
 
-/// Shorthand syntax for calling functions of the form
-/// `verify_foo(a, b, &mut VerifierErrors) -> VerifierStepResult`
-/// as if they had the form `verify_foo(a, b) -> VerifierResult`.
-///
-/// This syntax also ensures that no errors whatsoever were reported,
-/// even if they were not fatal.
-///
-/// # Example
-/// ```rust,ignore
-/// verify!(verify_context, func, cfg, domtree, fisa)
-///
-/// // ... is equivalent to...
-///
-/// let mut errors = VerifierErrors::new();
-/// let result = verify_context(func, cfg, domtree, fisa, &mut errors);
-///
-/// if errors.is_empty() {
-///     Ok(result.unwrap())
-/// } else {
-///     Err(errors)
-/// }
-/// ```
-#[macro_export]
-macro_rules! verify {
-    ( $verifier: expr; $fun: ident $(, $arg: expr )* ) => ({
-        let mut errors = $crate::verifier::VerifierErrors::default();
-        let result = $verifier.$fun( $( $arg, )* &mut errors);
-
-        if errors.is_empty() {
-            Ok(result.unwrap())
-        } else {
-            Err(errors)
-        }
-    });
-
-    ( $fun: path, $(, $arg: expr )* ) => ({
-        let mut errors = $crate::verifier::VerifierErrors::default();
-        let result = $fun( $( $arg, )* &mut errors);
-
-        if errors.is_empty() {
-            Ok(result.unwrap())
-        } else {
-            Err(errors)
-        }
-    });
-}
-
 mod cssa;
 mod flags;
 mod liveness;
@@ -203,9 +156,6 @@ pub type VerifierStepResult = Result;
 ///
 /// Unlike `VerifierStepResult<()>` which may be `Ok` while still having reported
 /// errors, this type always returns `Err` if an error (fatal or not) was reported.
-///
-/// Typically, this error will be constructed by using `verify!` on a function
-/// that returns `VerifierStepResult`.
 pub type VerifierResult = Result;
 
 /// List of verifier errors.
@@ -280,7 +230,14 @@ pub fn verify_function<'a, FOI: Into>>(
     fisa: FOI,
 ) -> VerifierResult<()> {
     let _tt = timing::verifier();
-    verify!(Verifier::new(func, fisa.into()); run)
+    let mut errors = VerifierErrors::default();
+    let verifier = Verifier::new(func, fisa.into());
+    let result = verifier.run(&mut errors);
+    if errors.is_empty() {
+        Ok(result.unwrap())
+    } else {
+        Err(errors)
+    }
 }
 
 /// Verify `func` after checking the integrity of associated context data structures `cfg` and

From 586a8835e9608b0bbaf8d2c62d7039da6fe93dc0 Mon Sep 17 00:00:00 2001
From: Lachlan Sneff 
Date: Tue, 23 Oct 2018 00:50:09 -0400
Subject: [PATCH 2139/3084] Add a `readonly` flag for loads (#562)

* Add readonly MemFlag

* Add readonly flag verifier check

* Make global loads readonly

* Fix gvn to consider readonly loads
---
 cranelift/docs/ir.rst                        | 13 +++++-----
 cranelift/filetests/simple_gvn/readonly.clif | 25 +++++++++++++++++++
 lib/codegen/src/ir/globalvalue.rs            | 16 ++++++++++--
 lib/codegen/src/ir/memflags.rs               | 17 ++++++++++++-
 lib/codegen/src/legalizer/globalvalue.rs     |  9 +++++--
 lib/codegen/src/simple_gvn.rs                | 20 ++++++++++++++-
 lib/codegen/src/verifier/mod.rs              | 26 ++++++++++++++++++++
 lib/reader/src/parser.rs                     |  7 +++---
 lib/wasm/src/environ/dummy.rs                |  3 +++
 9 files changed, 120 insertions(+), 16 deletions(-)
 create mode 100644 cranelift/filetests/simple_gvn/readonly.clif

diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index c452dfb6bc..5925f4e601 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -502,12 +502,13 @@ Memory operation flags
 Loads and stores can have flags that loosen their semantics in order to enable
 optimizations.
 
-======= ===========================================
-Flag    Description
-======= ===========================================
-notrap  Memory is assumed to be :term:`accessible`.
-aligned Trapping allowed for misaligned accesses.
-======= ===========================================
+=======  ===========================================
+Flag     Description
+=======  ===========================================
+notrap   Memory is assumed to be :term:`accessible`.
+aligned  Trapping allowed for misaligned accesses.
+readonly The data at the specified address will not modified between when this function is called and exited.
+=======  ===========================================
 
 When the ``accessible`` flag is set, the behavior is undefined if the memory
 is not :term:`accessible`.
diff --git a/cranelift/filetests/simple_gvn/readonly.clif b/cranelift/filetests/simple_gvn/readonly.clif
new file mode 100644
index 0000000000..28eacce61c
--- /dev/null
+++ b/cranelift/filetests/simple_gvn/readonly.clif
@@ -0,0 +1,25 @@
+test simple-gvn
+
+target x86_64
+
+function %eliminate_redundant_global_loads(i32, i64 vmctx) {
+    gv0 = vmctx
+    gv1 = load.i64 notrap aligned readonly gv0
+    heap0 = static gv1, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000, index_type i32
+
+ebb0(v0: i32, v1: i64):
+    v2 = heap_addr.i64 heap0, v0, 1
+    v3 = heap_addr.i64 heap0, v0, 1
+
+    v4 = iconst.i32 0
+    store.i32 notrap aligned v4, v2
+    store.i32 notrap aligned v4, v3
+
+    return
+}
+; check: v2 = heap_addr.i64 heap0, v0, 1
+; check: v3 -> v2
+; check: v4 = iconst.i32 0
+; check: store notrap aligned v4, v2
+; check: store notrap aligned v4, v2
+; check: return
diff --git a/lib/codegen/src/ir/globalvalue.rs b/lib/codegen/src/ir/globalvalue.rs
index 47696fe3d1..b46a2ab4e6 100644
--- a/lib/codegen/src/ir/globalvalue.rs
+++ b/lib/codegen/src/ir/globalvalue.rs
@@ -15,7 +15,8 @@ pub enum GlobalValueData {
     ///
     /// The `base` global value is assumed to contain a pointer. This global value is computed
     /// by loading from memory at that pointer value. The memory must be accessible, and
-    /// naturally aligned to hold a value of the type.
+    /// naturally aligned to hold a value of the type. The data at this address is assumed
+    /// to never change while the current function is executing.
     Load {
         /// The base pointer global value.
         base: GlobalValue,
@@ -25,6 +26,9 @@ pub enum GlobalValueData {
 
         /// Type of the loaded value.
         global_type: Type,
+
+        /// Specifies whether the memory that this refers to is readonly, allowing for the elimination of redundant loads.
+        readonly: bool,
     },
 
     /// Value is an offset from another global value.
@@ -90,7 +94,15 @@ impl fmt::Display for GlobalValueData {
                 base,
                 offset,
                 global_type,
-            } => write!(f, "load.{} notrap aligned {}{}", global_type, base, offset),
+                readonly,
+            } => write!(
+                f,
+                "load.{} notrap aligned {}{}{}",
+                global_type,
+                if readonly { "readonly " } else { "" },
+                base,
+                offset
+            ),
             GlobalValueData::IAddImm {
                 global_type,
                 base,
diff --git a/lib/codegen/src/ir/memflags.rs b/lib/codegen/src/ir/memflags.rs
index b05cea2913..f459abdb3f 100644
--- a/lib/codegen/src/ir/memflags.rs
+++ b/lib/codegen/src/ir/memflags.rs
@@ -5,9 +5,10 @@ use std::fmt;
 enum FlagBit {
     Notrap,
     Aligned,
+    Readonly,
 }
 
-const NAMES: [&str; 2] = ["notrap", "aligned"];
+const NAMES: [&str; 3] = ["notrap", "aligned", "readonly"];
 
 /// Flags for memory operations like load/store.
 ///
@@ -79,6 +80,20 @@ impl MemFlags {
     pub fn set_aligned(&mut self) {
         self.set(FlagBit::Aligned)
     }
+
+    /// Test if the `readonly` flag is set.
+    ///
+    /// Loads with this flag have no memory dependendies.
+    /// This results in indefined behavior if the dereferenced memory is mutated at any time
+    /// between when the function is called and when it is exited.
+    pub fn readonly(self) -> bool {
+        self.read(FlagBit::Readonly)
+    }
+
+    /// Set the `readonly` flag.
+    pub fn set_readonly(&mut self) {
+        self.set(FlagBit::Readonly)
+    }
 }
 
 impl fmt::Display for MemFlags {
diff --git a/lib/codegen/src/legalizer/globalvalue.rs b/lib/codegen/src/legalizer/globalvalue.rs
index 352c03e13b..1807734c53 100644
--- a/lib/codegen/src/legalizer/globalvalue.rs
+++ b/lib/codegen/src/legalizer/globalvalue.rs
@@ -38,7 +38,8 @@ pub fn expand_global_value(
             base,
             offset,
             global_type,
-        } => load_addr(inst, func, base, offset, global_type, isa),
+            readonly,
+        } => load_addr(inst, func, base, offset, global_type, readonly, isa),
         ir::GlobalValueData::Symbol { .. } => symbol(inst, func, gv, isa),
     }
 }
@@ -88,6 +89,7 @@ fn load_addr(
     base: ir::GlobalValue,
     offset: ir::immediates::Offset32,
     global_type: ir::Type,
+    readonly: bool,
     isa: &TargetIsa,
 ) {
     // We need to load a pointer from the `base` global value, so insert a new `global_value`
@@ -107,10 +109,13 @@ fn load_addr(
         pos.ins().global_value(ptr_ty, base)
     };
 
-    // Global-value loads are always notrap and aligned.
+    // Global-value loads are always notrap and aligned. They may be readonly.
     let mut mflags = ir::MemFlags::new();
     mflags.set_notrap();
     mflags.set_aligned();
+    if readonly {
+        mflags.set_readonly();
+    }
 
     // Perform the load.
     pos.func
diff --git a/lib/codegen/src/simple_gvn.rs b/lib/codegen/src/simple_gvn.rs
index 79560af674..54d1040e9d 100644
--- a/lib/codegen/src/simple_gvn.rs
+++ b/lib/codegen/src/simple_gvn.rs
@@ -18,10 +18,19 @@ fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool {
         || opcode.can_trap()
         || opcode.other_side_effects()
         || opcode.can_store()
-        || opcode.can_load()
         || opcode.writes_cpu_flags()
 }
 
+/// Test that, if the specified instruction is a load, it doesn't have the `readonly` memflag.
+fn is_load_and_not_readonly(inst_data: &InstructionData) -> bool {
+    match *inst_data {
+        InstructionData::Load { flags, .. } | InstructionData::LoadComplex { flags, .. } => {
+            !flags.readonly()
+        }
+        _ => inst_data.opcode().can_load(),
+    }
+}
+
 /// Wrapper around `InstructionData` which implements `Eq` and `Hash`
 #[derive(Clone)]
 struct HashKey<'a, 'f: 'a> {
@@ -91,14 +100,23 @@ pub fn do_simple_gvn(func: &mut Function, domtree: &mut DominatorTree) {
             let func = Ref::map(pos.borrow(), |pos| &pos.func);
 
             let opcode = func.dfg[inst].opcode();
+
             if opcode.is_branch() && !opcode.is_terminator() {
                 scope_stack.push(func.layout.next_inst(inst).unwrap());
                 visible_values.increment_depth();
             }
+
             if trivially_unsafe_for_gvn(opcode) {
                 continue;
             }
 
+            let inst_data = func.dfg[inst].clone();
+
+            // These are split up to separate concerns.
+            if is_load_and_not_readonly(&inst_data) {
+                continue;
+            }
+
             let ctrl_typevar = func.dfg.ctrl_typevar(inst);
             let key = HashKey {
                 inst: func.dfg[inst].clone(),
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 215508ce18..1e4fd631d1 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -1651,6 +1651,31 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
+    fn immediate_constraints(
+        &self,
+        inst: Inst,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
+        let inst_data = &self.func.dfg[inst];
+
+        // If this is some sort of a store instruction, get the memflags, else, just return.
+        let memflags = match *inst_data {
+            ir::InstructionData::Store { flags, .. }
+            | ir::InstructionData::StoreComplex { flags, .. } => flags,
+            _ => return Ok(()),
+        };
+
+        if memflags.readonly() {
+            fatal!(
+                errors,
+                inst,
+                "A store instruction cannot have the `readonly` MemFlag"
+            )
+        } else {
+            Ok(())
+        }
+    }
+
     pub fn run(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         self.verify_global_values(errors)?;
         self.verify_heaps(errors)?;
@@ -1663,6 +1688,7 @@ impl<'a> Verifier<'a> {
                 self.instruction_integrity(inst, errors)?;
                 self.typecheck(inst, errors)?;
                 self.verify_encoding(inst, errors)?;
+                self.immediate_constraints(inst, errors)?;
             }
         }
 
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 8951fb8dab..c129b76e64 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -1226,16 +1226,15 @@ impl<'a> Parser<'a> {
                 let flags = self.optional_memflags();
                 let base = self.match_gv("expected global value: gv«n»")?;
                 let offset = self.optional_offset32()?;
-                let mut expected_flags = MemFlags::new();
-                expected_flags.set_notrap();
-                expected_flags.set_aligned();
-                if flags != expected_flags {
+
+                if !(flags.notrap() && flags.aligned()) {
                     return err!(self.loc, "global-value load must be notrap and aligned");
                 }
                 GlobalValueData::Load {
                     base,
                     offset,
                     global_type,
+                    readonly: flags.readonly(),
                 }
             }
             "iadd_imm" => {
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 5608abec6d..20e1462f50 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -199,6 +199,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
             base: addr,
             offset: Offset32::new(0),
             global_type: self.pointer_type(),
+            readonly: true,
         });
 
         func.create_heap(ir::HeapData {
@@ -219,11 +220,13 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
             base: vmctx,
             offset: Offset32::new(0),
             global_type: self.pointer_type(),
+            readonly: true, // when tables in wasm become "growable", revisit whether this can be readonly or not.
         });
         let bound_gv = func.create_global_value(ir::GlobalValueData::Load {
             base: vmctx,
             offset: Offset32::new(0),
             global_type: I32,
+            readonly: true,
         });
 
         func.create_table(ir::TableData {

From b288c6001a8e963739519114d6d69ec18e53b14c Mon Sep 17 00:00:00 2001
From: Boris-Chengbiao Zhou 
Date: Tue, 23 Oct 2018 06:52:35 +0200
Subject: [PATCH 2140/3084] Fix all clippy warnings (#564)

* Fix all clippy warnings

* Revert usage of inclusive ranges

* Remove redundant function argument

* Revert use of unavailable pointer methods

* Introduce ContiguousCaseRange
---
 cranelift/src/clif-util.rs         |  7 ++---
 cranelift/src/compile.rs           |  2 +-
 lib/bforest/src/map.rs             |  1 +
 lib/bforest/src/set.rs             |  1 +
 lib/codegen/meta/src/cdsl/types.rs | 32 ++++++++++-----------
 lib/codegen/meta/src/gen_types.rs  |  3 +-
 lib/codegen/src/ir/dfg.rs          |  5 ++--
 lib/codegen/src/ir/function.rs     |  9 ++----
 lib/codegen/src/isa/encoding.rs    |  2 +-
 lib/codegen/src/print_errors.rs    |  2 +-
 lib/codegen/src/verifier/mod.rs    | 41 +++++++++++----------------
 lib/filetests/src/runner.rs        |  2 +-
 lib/filetests/src/test_binemit.rs  |  2 +-
 lib/frontend/src/switch.rs         | 45 ++++++++++++++++++++----------
 lib/reader/src/parser.rs           | 41 +++++++++++++--------------
 lib/simplejit/src/backend.rs       |  2 +-
 lib/wasm/src/environ/spec.rs       |  1 +
 17 files changed, 100 insertions(+), 98 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 13a6ba7d06..5d5ca2b5d4 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -115,7 +115,7 @@ fn add_debug_flag<'a>() -> clap::Arg<'a, 'a> {
 }
 
 /// Returns a vector of clap value options and changes these options into a vector of strings
-fn get_vec<'a>(argument_vec: Option>) -> Vec {
+fn get_vec(argument_vec: Option) -> Vec {
     let mut ret_vec: Vec = Vec::new();
     if let Some(clap_vec) = argument_vec {
         for val in clap_vec {
@@ -199,12 +199,11 @@ fn main() {
         }
         ("test", Some(rest_cmd)) => {
             handle_debug_flag(rest_cmd.is_present("debug"));
-            let result = cranelift_filetests::run(
+            cranelift_filetests::run(
                 rest_cmd.is_present("verbose"),
                 rest_cmd.is_present("time-passes"),
                 &get_vec(rest_cmd.values_of("file")),
-            ).map(|_time| ());
-            result
+            ).map(|_time| ())
         }
         ("pass", Some(rest_cmd)) => {
             handle_debug_flag(rest_cmd.is_present("debug"));
diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs
index 59402ff97f..0babf6f83e 100644
--- a/cranelift/src/compile.rs
+++ b/cranelift/src/compile.rs
@@ -151,7 +151,7 @@ fn handle_module(
 }
 
 fn print_readonly_data(mem: &[u8]) {
-    if mem.len() == 0 {
+    if mem.is_empty() {
         return;
     }
 
diff --git a/lib/bforest/src/map.rs b/lib/bforest/src/map.rs
index 97ab7e6d90..9b1c78f906 100644
--- a/lib/bforest/src/map.rs
+++ b/lib/bforest/src/map.rs
@@ -284,6 +284,7 @@ where
     ///
     /// If the cursor reaches the end, return `None` and leave the cursor at the off-the-end
     /// position.
+    #[cfg_attr(feature = "cargo-clippy", allow(should_implement_trait))]
     pub fn next(&mut self) -> Option<(K, V)> {
         self.path.next(self.pool)
     }
diff --git a/lib/bforest/src/set.rs b/lib/bforest/src/set.rs
index f950e1e3cb..f73a6b06d3 100644
--- a/lib/bforest/src/set.rs
+++ b/lib/bforest/src/set.rs
@@ -225,6 +225,7 @@ where
     ///
     /// If the cursor reaches the end, return `None` and leave the cursor at the off-the-end
     /// position.
+    #[cfg_attr(feature = "cargo-clippy", allow(should_implement_trait))]
     pub fn next(&mut self) -> Option {
         self.path.next(self.pool).map(|(k, _)| k)
     }
diff --git a/lib/codegen/meta/src/cdsl/types.rs b/lib/codegen/meta/src/cdsl/types.rs
index 9d2e9a16dc..6f191a10a9 100644
--- a/lib/codegen/meta/src/cdsl/types.rs
+++ b/lib/codegen/meta/src/cdsl/types.rs
@@ -161,8 +161,8 @@ pub enum LaneType {
 
 impl LaneType {
     /// Return a string containing the documentation comment for this lane type.
-    pub fn doc(&self) -> String {
-        match *self {
+    pub fn doc(self) -> String {
+        match self {
             LaneType::BoolType(_) => format!("A boolean type with {} bits.", self.lane_bits()),
             LaneType::FloatType(base_types::Float::F32) => String::from(
                 "A 32-bit floating point type represented in the IEEE 754-2008
@@ -185,8 +185,8 @@ impl LaneType {
     }
 
     /// Return the number of bits in a lane.
-    pub fn lane_bits(&self) -> u64 {
-        match *self {
+    pub fn lane_bits(self) -> u64 {
+        match self {
             LaneType::BoolType(ref b) => *b as u64,
             LaneType::FloatType(ref f) => *f as u64,
             LaneType::IntType(ref i) => *i as u64,
@@ -194,8 +194,8 @@ impl LaneType {
     }
 
     /// Get the name of this lane type.
-    pub fn name(&self) -> String {
-        match *self {
+    pub fn name(self) -> String {
+        match self {
             LaneType::BoolType(_) => format!("b{}", self.lane_bits()),
             LaneType::FloatType(_) => format!("f{}", self.lane_bits()),
             LaneType::IntType(_) => format!("i{}", self.lane_bits()),
@@ -203,8 +203,8 @@ impl LaneType {
     }
 
     /// Find the unique number associated with this lane type.
-    pub fn number(&self) -> u8 {
-        LANE_BASE + match *self {
+    pub fn number(self) -> u8 {
+        LANE_BASE + match self {
             LaneType::BoolType(base_types::Bool::B1) => 0,
             LaneType::BoolType(base_types::Bool::B8) => 1,
             LaneType::BoolType(base_types::Bool::B16) => 2,
@@ -394,8 +394,8 @@ pub enum SpecialType {
 
 impl SpecialType {
     /// Return a string containing the documentation comment for this special type.
-    pub fn doc(&self) -> String {
-        match *self {
+    pub fn doc(self) -> String {
+        match self {
             SpecialType::Flag(base_types::Flag::IFlags) => String::from(
                 "CPU flags representing the result of an integer comparison. These flags
                 can be tested with an :type:`intcc` condition code.",
@@ -408,23 +408,23 @@ impl SpecialType {
     }
 
     /// Return the number of bits in a lane.
-    pub fn lane_bits(&self) -> u64 {
-        match *self {
+    pub fn lane_bits(self) -> u64 {
+        match self {
             SpecialType::Flag(_) => 0,
         }
     }
 
     /// Get the name of this special type.
-    pub fn name(&self) -> String {
-        match *self {
+    pub fn name(self) -> String {
+        match self {
             SpecialType::Flag(base_types::Flag::IFlags) => "iflags".to_string(),
             SpecialType::Flag(base_types::Flag::FFlags) => "fflags".to_string(),
         }
     }
 
     /// Find the unique number associated with this special type.
-    pub fn number(&self) -> u8 {
-        match *self {
+    pub fn number(self) -> u8 {
+        match self {
             SpecialType::Flag(base_types::Flag::IFlags) => 1,
             SpecialType::Flag(base_types::Flag::FFlags) => 2,
         }
diff --git a/lib/codegen/meta/src/gen_types.rs b/lib/codegen/meta/src/gen_types.rs
index 1055f0de0e..1bbcbfe687 100644
--- a/lib/codegen/meta/src/gen_types.rs
+++ b/lib/codegen/meta/src/gen_types.rs
@@ -47,8 +47,7 @@ fn emit_vectors(bits: u64, fmt: &mut srcgen::Formatter) -> Result<(), error::Err
 /// Emit types using the given formatter object.
 fn emit_types(fmt: &mut srcgen::Formatter) -> Result<(), error::Error> {
     // Emit all of the special types, such as types for CPU flags.
-    for spec in cdsl_types::ValueType::all_special_types().map(|ty| cdsl_types::ValueType::from(ty))
-    {
+    for spec in cdsl_types::ValueType::all_special_types().map(cdsl_types::ValueType::from) {
         emit_type(&spec, fmt)?;
     }
 
diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs
index 7bcf17e949..6d92f70667 100644
--- a/lib/codegen/src/ir/dfg.rs
+++ b/lib/codegen/src/ir/dfg.rs
@@ -127,7 +127,7 @@ fn maybe_resolve_aliases(values: &PrimaryMap, value: Value) ->
     let mut v = value;
 
     // Note that values may be empty here.
-    for _ in 0..1 + values.len() {
+    for _ in 0..values.len() + 1 {
         if let ValueData::Alias { original, .. } = values[v] {
             v = original;
         } else {
@@ -174,8 +174,7 @@ impl<'a> Iterator for Values<'a> {
     fn next(&mut self) -> Option {
         self.inner
             .by_ref()
-            .filter(|kv| valid_valuedata(kv.1))
-            .next()
+            .find(|kv| valid_valuedata(kv.1))
             .map(|kv| kv.0)
     }
 }
diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs
index 800c9f2e2f..f2db834eab 100644
--- a/lib/codegen/src/ir/function.rs
+++ b/lib/codegen/src/ir/function.rs
@@ -178,19 +178,14 @@ impl Function {
     ///
     /// This function can only be used after the code layout has been computed by the
     /// `binemit::relax_branches()` function.
-    pub fn inst_offsets<'a>(
-        &'a self,
-        func: &'a Function,
-        ebb: Ebb,
-        encinfo: &EncInfo,
-    ) -> InstOffsetIter<'a> {
+    pub fn inst_offsets<'a>(&'a self, ebb: Ebb, encinfo: &EncInfo) -> InstOffsetIter<'a> {
         assert!(
             !self.offsets.is_empty(),
             "Code layout must be computed first"
         );
         InstOffsetIter {
             encinfo: encinfo.clone(),
-            func,
+            func: self,
             divert: RegDiversions::new(),
             encodings: &self.encodings,
             offset: self.offsets[ebb],
diff --git a/lib/codegen/src/isa/encoding.rs b/lib/codegen/src/isa/encoding.rs
index 589069b311..ee643c508d 100644
--- a/lib/codegen/src/isa/encoding.rs
+++ b/lib/codegen/src/isa/encoding.rs
@@ -85,7 +85,7 @@ type SizeCalculatorFn = fn(&RecipeSizing, Inst, &RegDiversions, &Function) -> u8
 /// Returns the base size of the Recipe, assuming it's fixed. This is the default for most
 /// encodings; others can be variable and longer than this base size, depending on the registers
 /// they're using and use a different function, specific per platform.
-pub fn base_size(sizing: &RecipeSizing, _: Inst, _2: &RegDiversions, _3: &Function) -> u8 {
+pub fn base_size(sizing: &RecipeSizing, _: Inst, _: &RegDiversions, _: &Function) -> u8 {
     sizing.base_size
 }
 
diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 217d09f902..5944e407ff 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -25,7 +25,7 @@ pub fn pretty_verifier_error<'a>(
     let mut w = String::new();
 
     decorate_function(
-        &mut PrettyVerifierError(func_w.unwrap_or(Box::new(PlainWriter)), &mut errors),
+        &mut PrettyVerifierError(func_w.unwrap_or_else(|| Box::new(PlainWriter)), &mut errors),
         &mut w,
         func,
         isa,
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 1e4fd631d1..6fedcc5af2 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -234,7 +234,8 @@ pub fn verify_function<'a, FOI: Into>>(
     let verifier = Verifier::new(func, fisa.into());
     let result = verifier.run(&mut errors);
     if errors.is_empty() {
-        Ok(result.unwrap())
+        result.unwrap();
+        Ok(())
     } else {
         Err(errors)
     }
@@ -392,30 +393,22 @@ impl<'a> Verifier<'a> {
                     );
                 }
 
-                match heap_data.style {
-                    ir::HeapStyle::Dynamic { bound_gv, .. } => {
-                        if !self.func.global_values.is_valid(bound_gv) {
-                            return nonfatal!(
-                                errors,
-                                heap,
-                                "invalid bound global value {}",
-                                bound_gv
-                            );
-                        }
-
-                        let index_type = heap_data.index_type;
-                        let bound_type = self.func.global_values[bound_gv].global_type(isa);
-                        if index_type != bound_type {
-                            report!(
-                                errors,
-                                heap,
-                                "heap index type {} differs from the type of its bound, {}",
-                                index_type,
-                                bound_type
-                            );
-                        }
+                if let ir::HeapStyle::Dynamic { bound_gv, .. } = heap_data.style {
+                    if !self.func.global_values.is_valid(bound_gv) {
+                        return nonfatal!(errors, heap, "invalid bound global value {}", bound_gv);
+                    }
+
+                    let index_type = heap_data.index_type;
+                    let bound_type = self.func.global_values[bound_gv].global_type(isa);
+                    if index_type != bound_type {
+                        report!(
+                            errors,
+                            heap,
+                            "heap index type {} differs from the type of its bound, {}",
+                            index_type,
+                            bound_type
+                        );
                     }
-                    _ => {}
                 }
             }
         }
diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs
index 7840c0dfff..fb2a0d4177 100644
--- a/lib/filetests/src/runner.rs
+++ b/lib/filetests/src/runner.rs
@@ -31,7 +31,7 @@ enum State {
     Done(TestResult),
 }
 
-#[derive(PartialEq, Eq, Debug)]
+#[derive(PartialEq, Eq, Debug, Clone, Copy)]
 pub enum IsPass {
     Pass,
     NotPass,
diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs
index da0f0eec09..da8310bfe6 100644
--- a/lib/filetests/src/test_binemit.rs
+++ b/lib/filetests/src/test_binemit.rs
@@ -205,7 +205,7 @@ impl SubTest for TestBinEmit {
                 "Inconsistent {} header offset",
                 ebb
             );
-            for (offset, inst, enc_bytes) in func.inst_offsets(&func, ebb, &encinfo) {
+            for (offset, inst, enc_bytes) in func.inst_offsets(ebb, &encinfo) {
                 assert_eq!(sink.offset, offset);
                 sink.text.clear();
                 let enc = func.encodings[inst];
diff --git a/lib/frontend/src/switch.rs b/lib/frontend/src/switch.rs
index f01a9f620b..ce3a2d943f 100644
--- a/lib/frontend/src/switch.rs
+++ b/lib/frontend/src/switch.rs
@@ -8,7 +8,7 @@ type EntryIndex = u64;
 
 /// Unlike with `br_table`, `Switch` cases may be sparse or non-0-based.
 /// They emit efficient code using branches, jump tables, or a combination of both.
-#[derive(Debug)]
+#[derive(Debug, Default)]
 pub struct Switch {
     cases: HashMap,
 }
@@ -16,7 +16,7 @@ pub struct Switch {
 impl Switch {
     /// Create a new empty switch
     pub fn new() -> Self {
-        Switch {
+        Self {
             cases: HashMap::new(),
         }
     }
@@ -31,23 +31,23 @@ impl Switch {
         );
     }
 
-    fn collect_contiguous_case_ranges(self) -> Vec<(EntryIndex, Vec)> {
+    fn collect_contiguous_case_ranges(self) -> Vec {
         debug!("build_contiguous_case_ranges before: {:#?}", self.cases);
         let mut cases = self.cases.into_iter().collect::>();
         cases.sort_by_key(|&(index, _)| index);
 
-        let mut contiguous_case_ranges: Vec<(EntryIndex, Vec)> = vec![];
+        let mut contiguous_case_ranges: Vec = vec![];
         let mut last_index = None;
         for (index, ebb) in cases {
             match last_index {
-                None => contiguous_case_ranges.push((index, vec![])),
+                None => contiguous_case_ranges.push(ContiguousCaseRange::new(index)),
                 Some(last_index) => {
                     if index > last_index + 1 {
-                        contiguous_case_ranges.push((index, vec![]));
+                        contiguous_case_ranges.push(ContiguousCaseRange::new(index));
                     }
                 }
             }
-            contiguous_case_ranges.last_mut().unwrap().1.push(ebb);
+            contiguous_case_ranges.last_mut().unwrap().ebbs.push(ebb);
             last_index = Some(index);
         }
 
@@ -63,7 +63,7 @@ impl Switch {
         bx: &mut FunctionBuilder,
         val: Value,
         otherwise: Ebb,
-        contiguous_case_ranges: Vec<(EntryIndex, Vec)>,
+        contiguous_case_ranges: Vec,
     ) -> Vec<(EntryIndex, Ebb, Vec)> {
         let mut cases_and_jt_ebbs = Vec::new();
 
@@ -79,7 +79,7 @@ impl Switch {
             return cases_and_jt_ebbs;
         }
 
-        let mut stack: Vec<(Option, Vec<(EntryIndex, Vec)>)> = Vec::new();
+        let mut stack: Vec<(Option, Vec)> = Vec::new();
         stack.push((None, contiguous_case_ranges));
 
         while let Some((ebb, contiguous_case_ranges)) = stack.pop() {
@@ -103,9 +103,11 @@ impl Switch {
                 let left_ebb = bx.create_ebb();
                 let right_ebb = bx.create_ebb();
 
-                let should_take_right_side =
-                    bx.ins()
-                        .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, val, right[0].0 as i64);
+                let should_take_right_side = bx.ins().icmp_imm(
+                    IntCC::UnsignedGreaterThanOrEqual,
+                    val,
+                    right[0].first_index as i64,
+                );
                 bx.ins().brnz(should_take_right_side, right_ebb, &[]);
                 bx.ins().jump(left_ebb, &[]);
 
@@ -121,10 +123,10 @@ impl Switch {
         bx: &mut FunctionBuilder,
         val: Value,
         otherwise: Ebb,
-        contiguous_case_ranges: Vec<(EntryIndex, Vec)>,
+        contiguous_case_ranges: Vec,
         cases_and_jt_ebbs: &mut Vec<(EntryIndex, Ebb, Vec)>,
     ) {
-        for (first_index, ebbs) in contiguous_case_ranges.into_iter().rev() {
+        for ContiguousCaseRange { first_index, ebbs } in contiguous_case_ranges.into_iter().rev() {
             if ebbs.len() == 1 {
                 let is_good_val = bx.ins().icmp_imm(IntCC::Equal, val, first_index as i64);
                 bx.ins().brnz(is_good_val, ebbs[0], &[]);
@@ -180,6 +182,21 @@ impl Switch {
     }
 }
 
+#[derive(Debug)]
+struct ContiguousCaseRange {
+    first_index: EntryIndex,
+    ebbs: Vec,
+}
+
+impl ContiguousCaseRange {
+    fn new(first_index: EntryIndex) -> Self {
+        Self {
+            first_index,
+            ebbs: Vec::new(),
+        }
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index c129b76e64..937edbee02 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -775,29 +775,26 @@ impl<'a> Parser<'a> {
         let mut targets = Vec::new();
         let flag_builder = settings::builder();
 
-        match target_pass {
-            Some(targ) => {
-                let loc = self.loc;
-                let triple = match Triple::from_str(targ) {
-                    Ok(triple) => triple,
-                    Err(err) => return err!(loc, err),
-                };
-                let mut isa_builder = match isa::lookup(triple) {
-                    Err(isa::LookupError::SupportDisabled) => {
-                        return err!(loc, "support disabled target '{}'", targ)
-                    }
-                    Err(isa::LookupError::Unsupported) => {
-                        return err!(loc, "unsupported target '{}'", targ)
-                    }
-                    Ok(b) => b,
-                };
-                specified_target = true;
+        if let Some(targ) = target_pass {
+            let loc = self.loc;
+            let triple = match Triple::from_str(targ) {
+                Ok(triple) => triple,
+                Err(err) => return err!(loc, err),
+            };
+            let mut isa_builder = match isa::lookup(triple) {
+                Err(isa::LookupError::SupportDisabled) => {
+                    return err!(loc, "support disabled target '{}'", targ)
+                }
+                Err(isa::LookupError::Unsupported) => {
+                    return err!(loc, "unsupported target '{}'", targ)
+                }
+                Ok(b) => b,
+            };
+            specified_target = true;
 
-                // Construct a trait object with the aggregate settings.
-                targets.push(isa_builder.finish(settings::Flags::new(flag_builder.clone())));
-            }
-            None => (),
-        };
+            // Construct a trait object with the aggregate settings.
+            targets.push(isa_builder.finish(settings::Flags::new(flag_builder.clone())));
+        }
 
         if !specified_target {
             // No `target` commands.
diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index 3fc46cd993..68cc7e8e92 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -411,7 +411,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
 
     /// SimpleJIT emits code and data into memory as it processes them, so it
     /// doesn't need to provide anything after the `Module` is complete.
-    fn finish(self) -> () {}
+    fn finish(self) {}
 }
 
 #[cfg(not(windows))]
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 872ec28475..9f0dc14964 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -161,6 +161,7 @@ pub trait FuncEnvironment {
     /// The signature `sig_ref` was previously created by `make_indirect_sig()`.
     ///
     /// Return the call instruction whose results are the WebAssembly return values.
+    #[cfg_attr(feature = "cargo-clippy", allow(too_many_arguments))]
     fn translate_call_indirect(
         &mut self,
         pos: FuncCursor,

From ed8ecb99df1ceff64f910a7c4571dc07c98cfbab Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 22 Oct 2018 08:58:37 -0700
Subject: [PATCH 2141/3084] Remove an obsolete comment.

---
 lib/frontend/src/frontend.rs | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index be178d554c..e04d831efe 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -18,10 +18,6 @@ use variable::Variable;
 /// In order to reduce memory reallocations when compiling multiple functions,
 /// `FunctionBuilderContext` holds various data structures which are cleared between
 /// functions, rather than dropped, preserving the underlying allocations.
-///
-/// The `Variable` parameter can be any index-like type that can be made to
-/// implement `EntityRef`. For frontends that don't have an obvious type to
-/// use here, `variable::Variable` can be used.
 pub struct FunctionBuilderContext {
     ssa: SSABuilder,
     ebbs: SecondaryMap,

From 6a234893eb224025da080d04c986156a01c1c35a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 23 Oct 2018 17:33:20 -0700
Subject: [PATCH 2142/3084] Remove use of deprecated APIs.

std::error::Error's description() function is [deprecated]; follow the
recommended advice and use to_string() instead.

[deprecated]: https://doc.rust-lang.org/std/error/trait.Error.html#method.description
---
 cranelift/src/wasm.rs | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 3dd1269980..0012f61f68 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -9,8 +9,8 @@
 
 use cranelift_codegen::print_errors::{pretty_error, pretty_verifier_error};
 use cranelift_codegen::settings::FlagsOrIsa;
-use cranelift_codegen::Context;
 use cranelift_codegen::timing;
+use cranelift_codegen::Context;
 use cranelift_entity::EntityRef;
 use cranelift_wasm::{
     translate_module, DummyEnvironment, FuncIndex, ModuleEnvironment, ReturnMode,
@@ -89,13 +89,12 @@ fn handle_module(
     vprint!(flag_verbose, "Translating... ");
     let _ = terminal.reset();
 
-    let mut module_binary =
-        read_to_end(path.clone()).map_err(|err| String::from(err.description()))?;
+    let mut module_binary = read_to_end(path.clone()).map_err(|err| err.to_string())?;
 
     if !module_binary.starts_with(&[b'\0', b'a', b's', b'm']) {
         module_binary = match wat2wasm(&module_binary) {
             Ok(data) => data,
-            Err(e) => return Err(String::from(e.description())),
+            Err(e) => return Err(e.to_string()),
         };
     }
 

From a2fcb32245df463cf445e2e36bcaa7f4c2dd0cc2 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 24 Oct 2018 10:00:42 -0700
Subject: [PATCH 2143/3084] Escape backslashes in Python comments.

The latest version of flake8 diagnoses these as invalid escape
sequences, so properly escape them.
---
 lib/codegen/meta-python/base/instructions.py  | 2 +-
 lib/codegen/meta-python/cdsl/instructions.py  | 2 +-
 lib/codegen/meta-python/cdsl/test_ti.py       | 2 +-
 lib/codegen/meta-python/cdsl/ti.py            | 4 ++--
 lib/codegen/meta-python/cdsl/typevar.py       | 2 +-
 lib/codegen/meta-python/cdsl/xform.py         | 2 +-
 lib/codegen/meta-python/semantics/__init__.py | 6 +++---
 7 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/lib/codegen/meta-python/base/instructions.py b/lib/codegen/meta-python/base/instructions.py
index 2993d6de49..498a75727f 100644
--- a/lib/codegen/meta-python/base/instructions.py
+++ b/lib/codegen/meta-python/base/instructions.py
@@ -1134,7 +1134,7 @@ srem_imm = Instruction(
 
 irsub_imm = Instruction(
         'irsub_imm', """
-        Immediate reverse wrapping subtraction: :math:`a := Y - x \pmod{2^B}`.
+        Immediate reverse wrapping subtraction: :math:`a := Y - x \\pmod{2^B}`.
 
         Also works as integer negation when :math:`Y = 0`. Use :inst:`iadd_imm`
         with a negative immediate operand for the reverse immediate
diff --git a/lib/codegen/meta-python/cdsl/instructions.py b/lib/codegen/meta-python/cdsl/instructions.py
index e4c7186333..f158ecbd4b 100644
--- a/lib/codegen/meta-python/cdsl/instructions.py
+++ b/lib/codegen/meta-python/cdsl/instructions.py
@@ -345,7 +345,7 @@ class Instruction(object):
         `(inst, typevars)` pair.
 
         This version in `Instruction` itself allows non-polymorphic
-        instructions to duck-type as `BoundInstruction`\s.
+        instructions to duck-type as `BoundInstruction`\\s.
         """
         assert not self.is_polymorphic, self
         return (self, ())
diff --git a/lib/codegen/meta-python/cdsl/test_ti.py b/lib/codegen/meta-python/cdsl/test_ti.py
index d3baa4d3c5..0e7a01023e 100644
--- a/lib/codegen/meta-python/cdsl/test_ti.py
+++ b/lib/codegen/meta-python/cdsl/test_ti.py
@@ -98,7 +98,7 @@ def check_concrete_typing_rtl(var_types, rtl):
     Check that a concrete type assignment var_types (Dict[Var, TypeVar]) is
     valid for an Rtl rtl. Specifically check that:
 
-    1) For each Var v \in rtl, v is defined in var_types
+    1) For each Var v \\in rtl, v is defined in var_types
 
     2) For all v, var_types[v] is a singleton type
 
diff --git a/lib/codegen/meta-python/cdsl/ti.py b/lib/codegen/meta-python/cdsl/ti.py
index bc2c16d5b1..0a7df053ca 100644
--- a/lib/codegen/meta-python/cdsl/ti.py
+++ b/lib/codegen/meta-python/cdsl/ti.py
@@ -419,7 +419,7 @@ class TypeEnv(object):
         E.g. if we have a root of the tree that looks like:
 
           typeof_a   typeof_b
-                 \  /
+                 \\  /
               typeof_x
                   |
                 half_width(1)
@@ -430,7 +430,7 @@ class TypeEnv(object):
         resulting graph is:
 
           typeof_a   typeof_b
-                 \  /
+                 \\  /
               typeof_x
         """
         source_tvs = set([v.get_typevar() for v in self.vars])
diff --git a/lib/codegen/meta-python/cdsl/typevar.py b/lib/codegen/meta-python/cdsl/typevar.py
index d1d7c7137d..9d2dace044 100644
--- a/lib/codegen/meta-python/cdsl/typevar.py
+++ b/lib/codegen/meta-python/cdsl/typevar.py
@@ -120,7 +120,7 @@ def legal_bool(bits):
     # type: (int) -> bool
     """
     True iff bits is a legal bit width for a bool type.
-    bits == 1 || bits \in { 8, 16, .. MAX_BITS }
+    bits == 1 || bits \\in { 8, 16, .. MAX_BITS }
     """
     return bits == 1 or \
         (bits >= 8 and bits <= MAX_BITS and is_power_of_two(bits))
diff --git a/lib/codegen/meta-python/cdsl/xform.py b/lib/codegen/meta-python/cdsl/xform.py
index 607c1776a2..5b2ee24b55 100644
--- a/lib/codegen/meta-python/cdsl/xform.py
+++ b/lib/codegen/meta-python/cdsl/xform.py
@@ -113,7 +113,7 @@ class Rtl(object):
         # type: (Rtl) -> None
         """
         Given that there is only 1 possible concrete typing T for self, assign
-        a singleton TV with type t=T[v] for each Var v \in self. Its an error
+        a singleton TV with type t=T[v] for each Var v \\in self. Its an error
         to call this on an Rtl with more than 1 possible typing. This modifies
         the Rtl in-place.
         """
diff --git a/lib/codegen/meta-python/semantics/__init__.py b/lib/codegen/meta-python/semantics/__init__.py
index 260ecae331..6b3586db1f 100644
--- a/lib/codegen/meta-python/semantics/__init__.py
+++ b/lib/codegen/meta-python/semantics/__init__.py
@@ -19,11 +19,11 @@ def verify_semantics(inst, src, xforms):
     Verify that the semantics transforms in xforms correctly describe the
     instruction described by the src Rtl. This involves checking that:
         0) src is a single instance of inst
-        1) For all x\in xforms x.src is a single instance of inst
+        1) For all x \\in xforms x.src is a single instance of inst
         2) For any concrete values V of Literals in inst:
             For all concrete typing T of inst:
-                Exists single x \in xforms that applies to src conretazied to V
-                and T
+                Exists single x \\in xforms that applies to src conretazied to
+                V and T
     """
     # 0) The source rtl is always a single instance of inst
     assert len(src.rtl) == 1 and src.rtl[0].expr.inst == inst

From 07c98f533001924099bdc1a927acdf36dc89822a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 24 Oct 2018 11:12:24 -0700
Subject: [PATCH 2144/3084] Fix an unused import warning.

---
 cranelift/src/wasm.rs | 1 -
 1 file changed, 1 deletion(-)

diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 0012f61f68..bab6e1394a 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -15,7 +15,6 @@ use cranelift_entity::EntityRef;
 use cranelift_wasm::{
     translate_module, DummyEnvironment, FuncIndex, ModuleEnvironment, ReturnMode,
 };
-use std::error::Error;
 use std::path::Path;
 use std::path::PathBuf;
 use term;

From e999c53eaacd681c0c54dee213c9346154a44b2e Mon Sep 17 00:00:00 2001
From: Aaron Power 
Date: Thu, 18 Oct 2018 15:07:08 +0100
Subject: [PATCH 2145/3084] Provide optimised codegen for small libc calls

---
 lib/frontend/src/frontend.rs | 168 +++++++++++++++++++++++++++++++++--
 1 file changed, 161 insertions(+), 7 deletions(-)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index e04d831efe..66ae1762a9 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -6,7 +6,8 @@ use cranelift_codegen::ir::function::DisplayFunction;
 use cranelift_codegen::ir::{
     types, AbiParam, DataFlowGraph, Ebb, ExtFuncData, ExternalName, FuncRef, Function, GlobalValue,
     GlobalValueData, Heap, HeapData, Inst, InstBuilder, InstBuilderBase, InstructionData,
-    JumpTable, JumpTableData, LibCall, SigRef, Signature, StackSlot, StackSlotData, Type, Value,
+    JumpTable, JumpTableData, LibCall, MemFlags, SigRef, Signature, StackSlot, StackSlotData, Type,
+    Value,
 };
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::packed_option::PackedOption;
@@ -567,10 +568,51 @@ impl<'a> FunctionBuilder<'a> {
         self.ins().call(libc_memcpy, &[dest, src, size]);
     }
 
+    /// Optimised memcpy for small copys.
+    pub fn emit_small_memcpy(
+        &mut self,
+        isa: &TargetIsa,
+        dest: Value,
+        src: Value,
+        size: u64,
+        dest_align: u8,
+        src_align: u8,
+    ) {
+        // Currently the result of guess work, not actual profiling.
+        const THRESHOLD: u64 = 4;
+
+        let access_size = greatest_divisible_power_of_two(size);
+        assert!(
+            access_size.is_power_of_two(),
+            "`size` is not a power of two"
+        );
+        assert!(
+            access_size >= ::std::cmp::min(src_align, dest_align) as u64,
+            "`size` is smaller than `dest` and `src`'s alignment value."
+        );
+        let load_and_store_amount = size / access_size;
+
+        if load_and_store_amount > THRESHOLD {
+            let size_value = self.ins().iconst(isa.pointer_type(), size as i64);
+            self.call_memcpy(isa, dest, src, size_value);
+            return;
+        }
+
+        let mut flags = MemFlags::new();
+        flags.set_aligned();
+
+        for i in 0..load_and_store_amount {
+            let offset = (access_size * i) as i32;
+            let int_type = Type::int(access_size as u16).unwrap();
+            let value = self.ins().load(int_type, flags, src, offset);
+            self.ins().store(flags, value, dest, offset);
+        }
+    }
+
     /// Calls libc.memset
     ///
-    /// Writes `len` bytes of value `ch` to memory starting at `buffer`.
-    pub fn call_memset(&mut self, isa: &TargetIsa, buffer: Value, ch: Value, len: Value) {
+    /// Writes `size` bytes of value `ch` to memory starting at `buffer`.
+    pub fn call_memset(&mut self, isa: &TargetIsa, buffer: Value, ch: Value, size: Value) {
         let pointer_type = isa.pointer_type();
         let signature = {
             let mut s = Signature::new(isa.flags().call_conv());
@@ -587,14 +629,68 @@ impl<'a> FunctionBuilder<'a> {
         });
 
         let ch = self.ins().uextend(types::I32, ch);
-        self.ins().call(libc_memset, &[buffer, ch, len]);
+        self.ins().call(libc_memset, &[buffer, ch, size]);
+    }
+
+    /// Calls libc.memset
+    ///
+    /// Writes `size` bytes of value `ch` to memory starting at `buffer`.
+    pub fn emit_small_memset(
+        &mut self,
+        isa: &TargetIsa,
+        buffer: Value,
+        ch: u32,
+        size: u64,
+        buffer_align: u8,
+    ) {
+        // Currently the result of guess work, not actual profiling.
+        const THRESHOLD: u64 = 4;
+
+        let access_size = greatest_divisible_power_of_two(size);
+        assert!(
+            access_size.is_power_of_two(),
+            "`size` is not a power of two"
+        );
+        assert!(
+            access_size >= buffer_align as u64,
+            "`size` is smaller than `dest` and `src`'s alignment value."
+        );
+        let load_and_store_amount = size / access_size;
+
+        if load_and_store_amount > THRESHOLD {
+            let ch = self.ins().iconst(types::I32, ch as i64);
+            let size = self.ins().iconst(isa.pointer_type(), size as i64);
+            self.call_memset(isa, buffer, ch, size);
+        } else {
+            let mut flags = MemFlags::new();
+            flags.set_aligned();
+
+            let ch = ch as u64;
+            let int_type = Type::int(access_size as u16).unwrap();
+            let raw_value = if int_type == types::I64 {
+                (ch << 32) | (ch << 16) | (ch << 8) | ch
+            } else if int_type == types::I32 {
+                (ch << 16) | (ch << 8) | ch
+            } else if int_type == types::I16 {
+                (ch << 8) | ch
+            } else {
+                assert_eq!(int_type, types::I8);
+                ch
+            };
+
+            let value = self.ins().iconst(int_type, raw_value as i64);
+            for i in 0..load_and_store_amount {
+                let offset = (access_size * i) as i32;
+                self.ins().store(flags, value, buffer, offset);
+            }
+        }
     }
 
     /// Calls libc.memmove
     ///
-    /// Copies `len` bytes from memory starting at `source` to memory starting
+    /// Copies `size` bytes from memory starting at `source` to memory starting
     /// at `dest`. `source` is always read before writing to `dest`.
-    pub fn call_memmove(&mut self, isa: &TargetIsa, dest: Value, source: Value, num: Value) {
+    pub fn call_memmove(&mut self, isa: &TargetIsa, dest: Value, source: Value, size: Value) {
         let pointer_type = isa.pointer_type();
         let signature = {
             let mut s = Signature::new(isa.flags().call_conv());
@@ -610,8 +706,57 @@ impl<'a> FunctionBuilder<'a> {
             colocated: false,
         });
 
-        self.ins().call(libc_memmove, &[dest, source, num]);
+        self.ins().call(libc_memmove, &[dest, source, size]);
     }
+
+    /// Optimised memmove for small moves.
+    pub fn emit_small_memmove(
+        &mut self,
+        isa: &TargetIsa,
+        dest: Value,
+        src: Value,
+        size: u64,
+        dest_align: u8,
+        src_align: u8,
+    ) {
+        // Currently the result of guess work, not actual profiling.
+        const THRESHOLD: u64 = 4;
+
+        let access_size = greatest_divisible_power_of_two(size);
+        assert!(
+            access_size.is_power_of_two(),
+            "`size` is not a power of two"
+        );
+        assert!(
+            access_size >= ::std::cmp::min(src_align, dest_align) as u64,
+            "`size` is smaller than `dest` and `src`'s alignment value."
+        );
+        let load_and_store_amount = size / access_size;
+
+        if load_and_store_amount > THRESHOLD {
+            let size_value = self.ins().iconst(isa.pointer_type(), size as i64);
+            self.call_memmove(isa, dest, src, size_value);
+            return;
+        }
+
+        let mut flags = MemFlags::new();
+        flags.set_aligned();
+
+        // Load all of the memory first in case `dest` overlaps.
+        let registers: Vec<_> = (0..load_and_store_amount)
+            .map(|i| {
+                let offset = (access_size * i) as i32;
+                (self.ins().load(types::I8, flags, src, offset), offset)
+            }).collect();
+
+        for (value, offset) in registers {
+            self.ins().store(flags, value, dest, offset);
+        }
+    }
+}
+
+fn greatest_divisible_power_of_two(size: u64) -> u64 {
+    (size as i64 & -(size as i64)) as u64
 }
 
 // Helper functions
@@ -648,6 +793,7 @@ impl<'a> FunctionBuilder<'a> {
 
 #[cfg(test)]
 mod tests {
+    use super::greatest_divisible_power_of_two;
     use cranelift_codegen::entity::EntityRef;
     use cranelift_codegen::ir::types::*;
     use cranelift_codegen::ir::{AbiParam, ExternalName, Function, InstBuilder, Signature};
@@ -821,4 +967,12 @@ ebb0:
 "
         );
     }
+
+    #[test]
+    fn test_greatest_divisible_power_of_two() {
+        assert_eq!(64, greatest_divisible_power_of_two(64));
+        assert_eq!(16, greatest_divisible_power_of_two(48));
+        assert_eq!(8, greatest_divisible_power_of_two(24));
+        assert_eq!(1, greatest_divisible_power_of_two(25));
+    }
 }

From 1b21d3a90ebf9b4496bd40569ebd6519e31015de Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 26 Oct 2018 11:22:43 -0700
Subject: [PATCH 2146/3084] Fix the path to cdsl/types.py in a comment.

---
 lib/codegen/src/ir/types.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/codegen/src/ir/types.rs b/lib/codegen/src/ir/types.rs
index 1d19f5c6d7..e1dcf0d177 100644
--- a/lib/codegen/src/ir/types.rs
+++ b/lib/codegen/src/ir/types.rs
@@ -24,7 +24,7 @@ pub struct Type(u8);
 /// Not a valid type. Can't be loaded or stored. Can't be part of a SIMD vector.
 pub const INVALID: Type = Type(0);
 
-/// Start of the lane types. See also `meta-python/cdsl.types.py`.
+/// Start of the lane types. See also `meta-python/cdsl/types.py`.
 const LANE_BASE: u8 = 0x70;
 
 /// Start of the 2-lane vector types.

From 87e683b1f47a4527c812b0e3754d0fb48467f04a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 26 Oct 2018 11:23:04 -0700
Subject: [PATCH 2147/3084] Update no_std support.

---
 lib/frontend/src/frontend.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 66ae1762a9..4014072e4a 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -12,6 +12,7 @@ use cranelift_codegen::ir::{
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::packed_option::PackedOption;
 use ssa::{Block, SSABuilder, SideEffects};
+use std::vec::Vec;
 use variable::Variable;
 
 /// Structure used for translating a series of functions into Cranelift IR.

From bf569b70dcd22bbad59d608bcb04c9f2846f8223 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 26 Oct 2018 11:24:43 -0700
Subject: [PATCH 2148/3084] Make cranelift-wasm's `type_to_type` easier to
 optimize.

---
 lib/wasm/src/translation_utils.rs | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index d76d55494f..cc2926dad1 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -92,13 +92,13 @@ pub struct Memory {
 
 /// Helper function translating wasmparser types to Cranelift types when possible.
 pub fn type_to_type(ty: wasmparser::Type) -> Result {
-    match ty {
-        wasmparser::Type::I32 => Ok(ir::types::I32),
-        wasmparser::Type::I64 => Ok(ir::types::I64),
-        wasmparser::Type::F32 => Ok(ir::types::F32),
-        wasmparser::Type::F64 => Ok(ir::types::F64),
-        _ => Err(()),
-    }
+    Ok(match ty {
+        wasmparser::Type::I32 => ir::types::I32,
+        wasmparser::Type::I64 => ir::types::I64,
+        wasmparser::Type::F32 => ir::types::F32,
+        wasmparser::Type::F64 => ir::types::F64,
+        _ => return Err(()),
+    })
 }
 
 /// Turns a `wasmparser` `f32` into a `Cranelift` one.

From a19c6088f08f8dcf0c90f1c832c337ed59d90944 Mon Sep 17 00:00:00 2001
From: Dan Gieschen Knutson 
Date: Sun, 28 Oct 2018 22:26:44 -0500
Subject: [PATCH 2149/3084] use iterative rather than recursive method for
 following aliases (#573)

* use iterative rather than recursive method for following aliases

* this avoids consuming stack via plain recursive calls
---
 lib/codegen/src/write.rs | 10 +++++++---
 lib/entity/src/map.rs    |  4 ++--
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index 24b8499f39..d790efa920 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -283,10 +283,14 @@ fn write_value_aliases(
     target: Value,
     indent: usize,
 ) -> fmt::Result {
-    for &a in &aliases[target] {
-        writeln!(w, "{1:0$}{2} -> {3}", indent, "", a, target)?;
-        write_value_aliases(w, aliases, a, indent)?;
+    let mut todo_stack = vec![target];
+    while let Some(target) = todo_stack.pop() {
+        for &a in &aliases[target] {
+            writeln!(w, "{1:0$}{2} -> {3}", indent, "", a, target)?;
+            todo_stack.push(a);
+        }
     }
+
     Ok(())
 }
 
diff --git a/lib/entity/src/map.rs b/lib/entity/src/map.rs
index 87a478e570..df8ac115a2 100644
--- a/lib/entity/src/map.rs
+++ b/lib/entity/src/map.rs
@@ -84,12 +84,12 @@ where
         Keys::with_len(self.elems.len())
     }
 
-    /// Iterate over all the keys in this map.
+    /// Iterate over all the values in this map.
     pub fn values(&self) -> slice::Iter {
         self.elems.iter()
     }
 
-    /// Iterate over all the keys in this map, mutable edition.
+    /// Iterate over all the values in this map, mutable edition.
     pub fn values_mut(&mut self) -> slice::IterMut {
         self.elems.iter_mut()
     }

From 2de5542139c3bef1dd75b82ea039693df25378c6 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 26 Oct 2018 17:02:22 -0700
Subject: [PATCH 2150/3084] Make `FuncEnvironment::return_mode()` default to
 `NormalReturns`.

`NormalReturns` is what we expect most users will normally want.
---
 lib/wasm/src/environ/spec.rs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 9f0dc14964..db6d0761d4 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -230,7 +230,9 @@ pub trait FuncEnvironment {
     /// Should the code be structured to use a single `fallthrough_return` instruction at the end
     /// of the function body, rather than `return` instructions as needed? This is used by VMs
     /// to append custom epilogues.
-    fn return_mode(&self) -> ReturnMode;
+    fn return_mode(&self) -> ReturnMode {
+        ReturnMode::NormalReturns
+    }
 }
 
 /// An object satisfying the `ModuleEnvironment` trait can be passed as argument to the

From f1db50aa760c518abdf3706c96286cba794b05d7 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 26 Oct 2018 06:05:54 -0700
Subject: [PATCH 2151/3084] Fix constant-offset folding to use the correct
 operand for stores.

---
 .../postopt/fold_offset_into_address.clif          | 14 ++++++++++++++
 lib/codegen/src/postopt.rs                         |  2 +-
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/cranelift/filetests/postopt/fold_offset_into_address.clif b/cranelift/filetests/postopt/fold_offset_into_address.clif
index 45ce7bcd0a..52379f4a50 100644
--- a/cranelift/filetests/postopt/fold_offset_into_address.clif
+++ b/cranelift/filetests/postopt/fold_offset_into_address.clif
@@ -16,3 +16,17 @@ ebb0(v0: i64):
 ; nextln: [RexOp1ldDisp8#808b]                v2 = load.i64 notrap aligned v0+16
 ; nextln: [Op1ret#c3]                         return v2
 ; nextln: }
+
+function u0:1(i64, i64 vmctx) {
+ebb0(v3: i64, v0: i64):
+                     v1 = iadd_imm.i64 v0, 16
+[RexOp1stDisp8#8089] store.i64 notrap aligned v3, v1
+[Op1ret#c3]          return
+}
+
+; sameln: function u0:1(i64, i64 vmctx) fast {
+; nextln: ebb0(v3: i64, v0: i64):
+; nextln:                                     v1 = iadd_imm v0, 16
+; nextln: [RexOp1stDisp8#8089]                store notrap aligned v3, v0+16
+; nextln: [Op1ret#c3]                         return
+; nextln: }
diff --git a/lib/codegen/src/postopt.rs b/lib/codegen/src/postopt.rs
index 8c64b9bb04..a57497b331 100644
--- a/lib/codegen/src/postopt.rs
+++ b/lib/codegen/src/postopt.rs
@@ -329,7 +329,7 @@ fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa)
                     ..
                 } => {
                     if let Some(imm) = offset.try_add_i64(imm.into()) {
-                        store_args[0] = arg;
+                        store_args[1] = arg;
                         *offset = imm;
                     } else {
                         // Overflow.

From d2b3ff618361eff0b438356fddea12a8f015530b Mon Sep 17 00:00:00 2001
From: theJosher 
Date: Sun, 28 Oct 2018 22:58:56 -0400
Subject: [PATCH 2152/3084] Errors reported by mypy 0.641 #558

---
 lib/codegen/meta-python/cdsl/ti.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/lib/codegen/meta-python/cdsl/ti.py b/lib/codegen/meta-python/cdsl/ti.py
index 0a7df053ca..64791ac1cd 100644
--- a/lib/codegen/meta-python/cdsl/ti.py
+++ b/lib/codegen/meta-python/cdsl/ti.py
@@ -26,6 +26,14 @@ class TypeConstraint(object):
     """
     Base class for all runtime-emittable type constraints.
     """
+
+    def __init__(self, tv, tc):
+        # type: (TypeVar, Union[TypeVar, TypeSet]) -> None
+        """
+        Abstract "constructor" for linters
+        """
+        assert False, "Abstract"
+
     def translate(self, m):
         # type: (Union[TypeEnv, TypeMap]) -> TypeConstraint
         """
@@ -75,7 +83,7 @@ class TypeConstraint(object):
         """
         Return the typevars contained in this constraint.
         """
-        return filter(lambda x:  isinstance(x, TypeVar), self._args())
+        return list(filter(lambda x: isinstance(x, TypeVar), self._args()))
 
     def is_trivial(self):
         # type: () -> bool

From 2a5f245b1d25baddd7173437d7de80a90ddada4c Mon Sep 17 00:00:00 2001
From: theJosher 
Date: Mon, 29 Oct 2018 22:19:05 -0400
Subject: [PATCH 2153/3084] #558 undoing the workaround - removing mypy 0.630
 pin

---
 .travis.yml | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 7113ae2d66..a62423fb7a 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -23,9 +23,7 @@ addons:
         packages:
             - python3-pip
 install:
-    # Fix the mypy version at 0.630 since version 0.641 is more strict and the
-    # code is not yet updated.
-    - pip3 install --user --upgrade mypy==0.630 flake8
+    - pip3 install --user --upgrade mypy flake8
     - mypy --version
 before_script:
     # If an old version of rustfmt from cargo is already installed, uninstall

From 0b769f5020d2e45517dfa777784dd1cc66e93e51 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 29 Oct 2018 13:31:50 -0700
Subject: [PATCH 2154/3084] Use `Display` rather than having an explicit
 `name()` function for types.

This is more idiomatic Rust.
---
 lib/codegen/meta/src/cdsl/types.rs | 79 +++++++++++++++---------------
 lib/codegen/meta/src/gen_types.rs  |  2 +-
 2 files changed, 40 insertions(+), 41 deletions(-)

diff --git a/lib/codegen/meta/src/cdsl/types.rs b/lib/codegen/meta/src/cdsl/types.rs
index 6f191a10a9..60e9888e78 100644
--- a/lib/codegen/meta/src/cdsl/types.rs
+++ b/lib/codegen/meta/src/cdsl/types.rs
@@ -79,16 +79,6 @@ impl ValueType {
         self.width() / 8
     }
 
-    /// Get the name of this type.
-    pub fn name(&self) -> String {
-        match *self {
-            ValueType::BV(ref b) => b.name(),
-            ValueType::Lane(l) => l.name(),
-            ValueType::Special(s) => s.name(),
-            ValueType::Vector(ref v) => v.name(),
-        }
-    }
-
     /// Find the unique number associated with this type.
     pub fn number(&self) -> Option {
         match *self {
@@ -101,7 +91,7 @@ impl ValueType {
 
     /// Return the name of this type for generated Rust source files.
     pub fn _rust_name(&self) -> String {
-        format!("{}{}", _RUST_NAME_PREFIX, self.name().to_uppercase())
+        format!("{}{}", _RUST_NAME_PREFIX, self.to_string().to_uppercase())
     }
 
     /// Return true iff:
@@ -119,7 +109,12 @@ impl ValueType {
 
 impl fmt::Display for ValueType {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        write!(f, "{}", self.name())
+        match *self {
+            ValueType::BV(ref b) => b.fmt(f),
+            ValueType::Lane(l) => l.fmt(f),
+            ValueType::Special(s) => s.fmt(f),
+            ValueType::Vector(ref v) => v.fmt(f),
+        }
     }
 }
 
@@ -193,15 +188,6 @@ impl LaneType {
         }
     }
 
-    /// Get the name of this lane type.
-    pub fn name(self) -> String {
-        match self {
-            LaneType::BoolType(_) => format!("b{}", self.lane_bits()),
-            LaneType::FloatType(_) => format!("f{}", self.lane_bits()),
-            LaneType::IntType(_) => format!("i{}", self.lane_bits()),
-        }
-    }
-
     /// Find the unique number associated with this lane type.
     pub fn number(self) -> u8 {
         LANE_BASE + match self {
@@ -220,6 +206,16 @@ impl LaneType {
     }
 }
 
+impl fmt::Display for LaneType {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        match *self {
+            LaneType::BoolType(_) => write!(f, "b{}", self.lane_bits()),
+            LaneType::FloatType(_) => write!(f, "f{}", self.lane_bits()),
+            LaneType::IntType(_) => write!(f, "i{}", self.lane_bits()),
+        }
+    }
+}
+
 impl fmt::Debug for LaneType {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         let inner_msg = format!("bits={}", self.lane_bits());
@@ -309,7 +305,7 @@ impl VectorType {
         format!(
             "A SIMD vector with {} lanes containing a `{}` each.",
             self.lane_count(),
-            self.base.name()
+            self.base
         )
     }
 
@@ -323,11 +319,6 @@ impl VectorType {
         self.lanes
     }
 
-    /// Get the name of this vector type.
-    pub fn name(&self) -> String {
-        format!("{}x{}", self.base.name(), self.lane_count())
-    }
-
     /// Find the unique number associated with this vector type.
     ///
     /// Vector types are encoded with the lane type in the low 4 bits and
@@ -340,12 +331,18 @@ impl VectorType {
     }
 }
 
+impl fmt::Display for VectorType {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "{}x{}", self.base, self.lane_count())
+    }
+}
+
 impl fmt::Debug for VectorType {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         write!(
             f,
             "VectorType(base={}, lanes={})",
-            self.base.name(),
+            self.base,
             self.lane_count()
         )
     }
@@ -371,10 +368,11 @@ impl BVType {
     pub fn lane_bits(&self) -> u64 {
         self.bits
     }
+}
 
-    /// Get the name of this bitvector type.
-    pub fn name(&self) -> String {
-        format!("bv{}", self.bits)
+impl fmt::Display for BVType {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "bv{}", self.bits)
     }
 }
 
@@ -414,14 +412,6 @@ impl SpecialType {
         }
     }
 
-    /// Get the name of this special type.
-    pub fn name(self) -> String {
-        match self {
-            SpecialType::Flag(base_types::Flag::IFlags) => "iflags".to_string(),
-            SpecialType::Flag(base_types::Flag::FFlags) => "fflags".to_string(),
-        }
-    }
-
     /// Find the unique number associated with this special type.
     pub fn number(self) -> u8 {
         match self {
@@ -431,13 +421,22 @@ impl SpecialType {
     }
 }
 
+impl fmt::Display for SpecialType {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        match *self {
+            SpecialType::Flag(base_types::Flag::IFlags) => write!(f, "iflags"),
+            SpecialType::Flag(base_types::Flag::FFlags) => write!(f, "fflags"),
+        }
+    }
+}
+
 impl fmt::Debug for SpecialType {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         write!(
             f,
             "{}",
             match *self {
-                SpecialType::Flag(_) => format!("FlagsType({})", self.name()),
+                SpecialType::Flag(_) => format!("FlagsType({})", self),
             }
         )
     }
diff --git a/lib/codegen/meta/src/gen_types.rs b/lib/codegen/meta/src/gen_types.rs
index 1bbcbfe687..8dbd647bee 100644
--- a/lib/codegen/meta/src/gen_types.rs
+++ b/lib/codegen/meta/src/gen_types.rs
@@ -13,7 +13,7 @@ use srcgen;
 
 /// Emit a constant definition of a single value type.
 fn emit_type(ty: &cdsl_types::ValueType, fmt: &mut srcgen::Formatter) -> Result<(), error::Error> {
-    let name = ty.name().to_uppercase();
+    let name = ty.to_string().to_uppercase();
     let number = ty.number().ok_or_else(|| {
         error::Error::with_msg(format!(
             "Could not emit type `{}` which has no number.",

From 5d9591d1eff5cf91ee1a7a910f209af331d094e6 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 30 Oct 2018 11:28:08 -0700
Subject: [PATCH 2155/3084] Adjust whitespace to match the upstream exception
 text.

This is a whitespace-only change.
---
 cranelift/LICENSE | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cranelift/LICENSE b/cranelift/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/cranelift/LICENSE
+++ b/cranelift/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+

From 62e55f63e6b335b3bc653fef4b694d04990fc98f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 24 Oct 2018 09:33:13 -0700
Subject: [PATCH 2156/3084] Update the minimum supported Rust version to 1.29.

We no longer need the Ubuntu LTS restriction, so now the only only
constraint I'm aware of is Firefox's policy. Fortunately, that tracks
the latest stable delayed by only two weeks. So this puts is at
Rust 1.29 now.
---
 .travis.yml               | 6 +++---
 cranelift/CONTRIBUTING.md | 9 +++------
 cranelift/README.md       | 9 +++------
 3 files changed, 9 insertions(+), 15 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index a62423fb7a..1020d2b8e3 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,9 +2,9 @@
 
 language: rust
 rust:
-    - stable
-    # The version of rust in the latest Ubuntu LTS, currently Bionic.
-    - 1.25.0
+    # The oldest version we currently support. See
+    # CONTRIBUTING.md#rustc-version-support for details.
+    - 1.29.0
     - beta
     - nightly
 matrix:
diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
index a5f5aad334..89aff16fec 100644
--- a/cranelift/CONTRIBUTING.md
+++ b/cranelift/CONTRIBUTING.md
@@ -75,17 +75,14 @@ which may be convenient when there are multiple versions installed.
 
 ### Rustc version support
 
-Our current policy is to support the version of Rustc that ships with the
-latest Ubuntu LTS release, as well as the current stable version. This means
-we don't use some of the very latest released Rust features.
+Cranelift supports stable Rust, and follows the
+[Rust Update Policy for Firefox].
 
 Some of the developer scripts depend on nightly Rust, for example to run
 clippy and other tools, however we avoid depending on these for the main
 build.
 
-That said, if there are any new Rust features that would be particularly
-valuable to use, please bring them up, as we may be able to find ways to
-accommodate them.
+[Rust Update Policy for Firefox]: https://wiki.mozilla.org/Rust_Update_Policy_for_Firefox#Schedule
 
 ### Python
 
diff --git a/cranelift/README.md b/cranelift/README.md
index 8e65e72ebf..cb567cd5a5 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -9,7 +9,7 @@ into executable machine code.
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
 [![Build Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
-![Minimum rustc 1.25](https://img.shields.io/badge/rustc-1.25+-red.svg)
+![Minimum rustc 1.29](https://img.shields.io/badge/rustc-1.29+-red.svg)
 
 For more information, see [the
 documentation](https://cranelift.readthedocs.io/en/latest/?badge=latest).
@@ -47,11 +47,8 @@ needed before it would be ready for a production use case.
 
 Cranelift's APIs are not yet stable.
 
-Cranelift currently supports Rust 1.25.0 and later. We intend to always
-support the latest *stable* Rust. And, we currently support the version
-of Rust in the latest Ubuntu LTS, although whether we will always do so
-is not yet determined. Cranelift requires Python 2.7 or Python 3 to
-build.
+Cranelift currently requires Rust 1.29 or later, and Python 2.7 or 3
+to build.
 
 Planned uses
 ------------

From 9471c06da4efeb2e5dfd59bab371ede93ee004da Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 24 Oct 2018 11:10:20 -0700
Subject: [PATCH 2157/3084] Update to use newer Rust features.

This re-introduces several cleanups that we previously deferred for not
supporting Rust 1.25.
---
 lib/bforest/src/node.rs     | 10 +++++-----
 lib/codegen/src/context.rs  |  9 +--------
 lib/codegen/src/ir/dfg.rs   |  2 +-
 lib/codegen/src/lib.rs      |  2 --
 lib/codegen/src/timing.rs   |  2 +-
 lib/entity/src/list.rs      |  2 +-
 lib/faerie/Cargo.toml       |  2 +-
 lib/filetests/src/lib.rs    |  6 +-----
 lib/filetests/src/runner.rs |  8 +-------
 lib/simplejit/Cargo.toml    |  2 +-
 lib/simplejit/src/memory.rs |  2 +-
 11 files changed, 14 insertions(+), 33 deletions(-)

diff --git a/lib/bforest/src/node.rs b/lib/bforest/src/node.rs
index 768db40174..583fe0f78e 100644
--- a/lib/bforest/src/node.rs
+++ b/lib/bforest/src/node.rs
@@ -105,7 +105,7 @@ impl NodeData {
                 let size = usize::from(size);
                 // TODO: We could probably use `get_unchecked()` here since `size` is always in
                 // range.
-                (&keys[0..size], &tree[0..size + 1])
+                (&keys[0..size], &tree[0..=size])
             }
             _ => panic!("Expected inner node"),
         }
@@ -175,10 +175,10 @@ impl NodeData {
                 debug_assert!(sz <= keys.len());
                 debug_assert!(index <= sz, "Can't insert at {} with {} keys", index, sz);
 
-                if let Some(ks) = keys.get_mut(0..sz + 1) {
+                if let Some(ks) = keys.get_mut(0..=sz) {
                     *size = (sz + 1) as u8;
                     slice_insert(ks, index, key);
-                    slice_insert(&mut tree[1..sz + 2], index, node);
+                    slice_insert(&mut tree[1..=sz + 1], index, node);
                     true
                 } else {
                     false
@@ -203,10 +203,10 @@ impl NodeData {
                 debug_assert!(sz <= keys.len());
                 debug_assert!(index <= sz);
 
-                if let Some(ks) = keys.get_mut(0..sz + 1) {
+                if let Some(ks) = keys.get_mut(0..=sz) {
                     *size = (sz + 1) as u8;
                     slice_insert(ks, index, key);
-                    slice_insert(&mut vals[0..sz + 1], index, value);
+                    slice_insert(&mut vals[0..=sz], index, value);
                     true
                 } else {
                     false
diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs
index e0f0de2c07..fa9c78d7f5 100644
--- a/lib/codegen/src/context.rs
+++ b/lib/codegen/src/context.rs
@@ -101,14 +101,7 @@ impl Context {
         let code_size = self.compile(isa)?;
         let old_len = mem.len();
         mem.resize(old_len + code_size as usize, 0);
-        unsafe {
-            self.emit_to_memory(
-                isa,
-                mem.as_mut_ptr().offset(old_len as isize),
-                relocs,
-                traps,
-            )
-        };
+        unsafe { self.emit_to_memory(isa, mem.as_mut_ptr().add(old_len), relocs, traps) };
         Ok(())
     }
 
diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs
index 6d92f70667..ecb22237f9 100644
--- a/lib/codegen/src/ir/dfg.rs
+++ b/lib/codegen/src/ir/dfg.rs
@@ -127,7 +127,7 @@ fn maybe_resolve_aliases(values: &PrimaryMap, value: Value) ->
     let mut v = value;
 
     // Note that values may be empty here.
-    for _ in 0..values.len() + 1 {
+    for _ in 0..=values.len() {
         if let ValueData::Alias { original, .. } = values[v] {
             v = original;
         } else {
diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index 24c381a6eb..ff9ca2d073 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -8,8 +8,6 @@
     plugin(clippy(conf_file = "../../clippy.toml"))
 )]
 #![cfg_attr(feature="cargo-clippy", allow(
-// This requires Rust 1.27 or later.
-                duration_subsec,
 // Produces only a false positive:
                 while_let_loop,
 // Produces many false positives, but did produce some valid lints, now fixed:
diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs
index e5327071c9..4ac9e153ed 100644
--- a/lib/codegen/src/timing.rs
+++ b/lib/codegen/src/timing.rs
@@ -154,7 +154,7 @@ mod details {
                 fn fmtdur(mut dur: Duration, f: &mut fmt::Formatter) -> fmt::Result {
                     // Round to nearest ms by adding 500us.
                     dur += Duration::new(0, 500_000);
-                    let ms = dur.subsec_nanos() / 1_000_000;
+                    let ms = dur.subsec_millis();
                     write!(f, "{:4}.{:03} ", dur.as_secs(), ms)
                 }
 
diff --git a/lib/entity/src/list.rs b/lib/entity/src/list.rs
index c061419eab..c2b399da2c 100644
--- a/lib/entity/src/list.rs
+++ b/lib/entity/src/list.rs
@@ -234,7 +234,7 @@ impl EntityList {
 
         let block = pool.alloc(sclass_for_length(len));
         pool.data[block] = T::new(len);
-        pool.data[block + 1..block + len + 1].copy_from_slice(slice);
+        pool.data[block + 1..=block + len].copy_from_slice(slice);
 
         Self {
             index: (block + 1) as u32,
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index b19c5c1775..7b84b67fa6 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -12,7 +12,7 @@ readme = "README.md"
 cranelift-codegen = { path = "../codegen", version = "0.22.0" }
 cranelift-module = { path = "../module", version = "0.22.0" }
 faerie = "0.5.0"
-goblin = "0.0.17"
+goblin = "0.0.19"
 failure = "0.1.2"
 target-lexicon = "0.0.3"
 
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index 0032bc59db..56c7e66ff3 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -10,11 +10,7 @@
     unstable_features
 )]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "cargo-clippy",
-    allow(
-          type_complexity,
-          // This requires Rust 1.27 or later.
-          duration_subsec))]
+#![cfg_attr(feature = "cargo-clippy", allow(type_complexity))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs
index fb2a0d4177..b2b007803f 100644
--- a/lib/filetests/src/runner.rs
+++ b/lib/filetests/src/runner.rs
@@ -47,13 +47,7 @@ impl Display for QueueEntry {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         let p = self.path.to_string_lossy();
         match self.state {
-            State::Done(Ok(dur)) => write!(
-                f,
-                "{}.{:03} {}",
-                dur.as_secs(),
-                dur.subsec_nanos() / 1_000_000,
-                p
-            ),
+            State::Done(Ok(dur)) => write!(f, "{}.{:03} {}", dur.as_secs(), dur.subsec_millis(), p),
             State::Done(Err(ref e)) => write!(f, "FAIL {}: {}", p, e),
             _ => write!(f, "{}", p),
         }
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index b8ed24ed3c..4d85eb1bce 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -13,7 +13,7 @@ readme = "README.md"
 cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
 cranelift-module = { path = "../module", version = "0.22.0", default-features = false }
 cranelift-native = { path = "../native", version = "0.22.0", default-features = false }
-region = "0.3.0"
+region = "1.0.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
 target-lexicon = { version = "0.0.3", default-features = false }
diff --git a/lib/simplejit/src/memory.rs b/lib/simplejit/src/memory.rs
index f5dcf42780..9720e2ae17 100644
--- a/lib/simplejit/src/memory.rs
+++ b/lib/simplejit/src/memory.rs
@@ -100,7 +100,7 @@ impl Memory {
     pub fn allocate(&mut self, size: usize) -> Result<*mut u8, String> {
         if size <= self.current.len - self.position {
             // TODO: Ensure overflow is not possible.
-            let ptr = unsafe { self.current.ptr.offset(self.position as isize) };
+            let ptr = unsafe { self.current.ptr.add(self.position) };
             self.position += size;
             return Ok(ptr);
         }

From 6f8c3012c662ec11fc38f99e7914391d4309933e Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Fri, 2 Nov 2018 20:57:14 +0100
Subject: [PATCH 2158/3084] SimpleJit perf map (#571)

* SimpleJit perf map
---
 lib/simplejit/src/backend.rs | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index 68cc7e8e92..f774adbbaa 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -11,6 +11,7 @@ use libc;
 use memory::Memory;
 use std::collections::HashMap;
 use std::ffi::CString;
+use std::io::Write;
 use std::ptr;
 use target_lexicon::PointerWidth;
 #[cfg(windows)]
@@ -167,7 +168,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
 
     fn define_function(
         &mut self,
-        _name: &str,
+        name: &str,
         ctx: &cranelift_codegen::Context,
         _namespace: &ModuleNamespace,
         code_size: u32,
@@ -177,6 +178,17 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
             .code_memory
             .allocate(size)
             .expect("TODO: handle OOM etc.");
+
+        if cfg!(target_os = "linux") && ::std::env::var_os("PERF_BUILDID_DIR").is_some() {
+            let mut map_file = ::std::fs::OpenOptions::new()
+                .create(true)
+                .append(true)
+                .open(format!("/tmp/perf-{}.map", ::std::process::id()))
+                .unwrap();
+
+            writeln!(map_file, "{:x} {:x} {}", ptr as usize, code_size, name);
+        }
+
         let mut reloc_sink = SimpleJITRelocSink::new();
         // Ignore traps for now. For now, frontends should just avoid generating code
         // that traps.

From 7094d9f47094e84f01c788a28b5a85a785be4507 Mon Sep 17 00:00:00 2001
From: Joe Howarth 
Date: Fri, 2 Nov 2018 16:47:39 -0400
Subject: [PATCH 2159/3084] Prefix verifier errors with ; (#585)

* Prefix verifier errors with ;

extract arrow drawing code into helper
print ; at beginning of line followed by arrow and error body
---
 lib/codegen/src/print_errors.rs | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 5944e407ff..8e764065a7 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -84,11 +84,7 @@ fn pretty_instruction_error(
                     printed_instr = true;
                 }
 
-                write!(w, "{1:0$}^", indent, "")?;
-                for _c in cur_inst.to_string().chars() {
-                    write!(w, "~")?;
-                }
-                writeln!(w, " verifier {}", err.to_string())?;
+                print_error(w, indent, cur_inst.to_string(), err)?;
             }
             ir::entities::AnyEntity::Inst(_) => i += 1,
             _ => unreachable!(),
@@ -133,11 +129,7 @@ fn pretty_preamble_error(
                 printed_entity = true;
             }
 
-            write!(w, "{1:0$}^", indent, "")?;
-            for _c in entity.to_string().chars() {
-                write!(w, "~")?;
-            }
-            writeln!(w, " verifier {}", err.to_string())?;
+            print_error(w, indent, entity.to_string(), err)?;
         } else {
             i += 1
         }
@@ -152,6 +144,18 @@ fn pretty_preamble_error(
     Ok(())
 }
 
+/// Prints ;   ^~~~~~ verifier [ERROR BODY]
+fn print_error(w: &mut Write, indent: usize, s: String, err: VerifierError) -> fmt::Result {
+    let indent = if indent < 1 { 0 } else { indent - 1 };
+
+    write!(w, ";{1:0$}^", indent, "")?;
+    for _c in s.chars() {
+        write!(w, "~")?;
+    }
+    writeln!(w, " verifier {}", err.to_string())?;
+    Ok(())
+}
+
 /// Pretty-print a Cranelift error.
 pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CodegenError) -> String {
     if let CodegenError::Verifier(e) = err {

From d4f8eb7453c8e8a060f26ddc9ac7983eb0bbb8be Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 2 Nov 2018 13:51:42 -0700
Subject: [PATCH 2160/3084] Introduce a `TargetFrontendConfig` type. (#570)

* Introduce a `TargetFrontendConfig` type.

`TargetFrontendConfig` is information specific to the target which is
provided to frontends to allow them to produce Cranelift IR for the
target. Currently this includes the pointer size and the default calling
convention.

The default calling convention is now inferred from the target, rather
than being a setting. cranelift-native is now just a provider of target
information, rather than also being a provider of settings, which gives
it a clearer role.

And instead of having cranelift-frontend routines require the whole
`TargetIsa`, just require the `TargetFrontendConfig`, and add a way to
get the `TargetFrontendConfig` from a `Module`.

Fixes #529.
Fixes #555.
---
 .../filetests/isa/x86/legalize-libcall.clif   |  1 -
 cranelift/fuzz/Cargo.toml                     |  1 +
 cranelift/fuzz/fuzz_translate_module.rs       | 11 ++-
 cranelift/src/wasm.rs                         |  6 +-
 lib/codegen/meta-python/base/settings.py      | 24 ------
 lib/codegen/src/ir/extfunc.rs                 |  3 +-
 lib/codegen/src/ir/function.rs                |  3 +-
 lib/codegen/src/ir/libcall.rs                 |  6 +-
 lib/codegen/src/isa/call_conv.rs              | 74 +++++++++++++++++++
 lib/codegen/src/isa/mod.rs                    | 56 +++++++++++++-
 lib/codegen/src/isa/x86/abi.rs                |  9 +--
 lib/codegen/src/settings.rs                   |  1 -
 lib/frontend/src/frontend.rs                  | 66 +++++++++++------
 lib/frontend/src/lib.rs                       |  3 +-
 lib/module/src/module.rs                      | 17 +++--
 lib/native/src/lib.rs                         | 47 ++++++++----
 lib/reader/src/parser.rs                      |  5 +-
 lib/simplejit/src/backend.rs                  |  5 +-
 lib/simplejit/tests/basic.rs                  |  2 +-
 lib/umbrella/src/lib.rs                       |  2 +-
 lib/wasm/Cargo.toml                           |  4 +-
 lib/wasm/src/environ/dummy.rs                 | 48 ++++--------
 lib/wasm/src/environ/spec.rs                  | 18 ++---
 lib/wasm/src/func_translator.rs               | 42 ++++++++---
 lib/wasm/src/lib.rs                           |  1 +
 lib/wasm/src/sections_translator.rs           |  2 +-
 lib/wasm/tests/wasm_testsuite.rs              |  8 +-
 27 files changed, 297 insertions(+), 168 deletions(-)
 create mode 100644 lib/codegen/src/isa/call_conv.rs

diff --git a/cranelift/filetests/isa/x86/legalize-libcall.clif b/cranelift/filetests/isa/x86/legalize-libcall.clif
index 3605f2ff7b..e28bebd668 100644
--- a/cranelift/filetests/isa/x86/legalize-libcall.clif
+++ b/cranelift/filetests/isa/x86/legalize-libcall.clif
@@ -2,7 +2,6 @@ test legalizer
 
 ; Pre-SSE 4.1, we need to use runtime library calls for floating point rounding operations.
 set is_pic
-set call_conv=system_v
 target x86_64
 
 function %floor(f32) -> f32 {
diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
index af5f216836..3c2930ae07 100644
--- a/cranelift/fuzz/Cargo.toml
+++ b/cranelift/fuzz/Cargo.toml
@@ -11,6 +11,7 @@ cargo-fuzz = true
 cargo-fuzz = "*"
 binaryen = { git = "https://github.com/pepyakin/binaryen-rs.git" }
 libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" }
+cranelift-codegen = { path = "../lib/codegen" }
 cranelift-wasm = { path = "../lib/wasm" }
 cranelift-reader = { path = "../lib/reader" }
 target-lexicon = "0.0.3"
diff --git a/cranelift/fuzz/fuzz_translate_module.rs b/cranelift/fuzz/fuzz_translate_module.rs
index df339be545..abc742a62f 100644
--- a/cranelift/fuzz/fuzz_translate_module.rs
+++ b/cranelift/fuzz/fuzz_translate_module.rs
@@ -1,11 +1,15 @@
 #![no_main]
+
 #[macro_use]
 extern crate libfuzzer_sys;
 extern crate binaryen;
+extern crate cranelift_codegen;
 extern crate cranelift_wasm;
 #[macro_use]
 extern crate target_lexicon;
-use cranelift_wasm::{translate_module, DummyEnvironment};
+
+use cranelift_codegen::{isa, settings};
+use cranelift_wasm::{translate_module, DummyEnvironment, ReturnMode};
 use std::str::FromStr;
 
 fuzz_target!(|data: &[u8]| {
@@ -13,6 +17,9 @@ fuzz_target!(|data: &[u8]| {
 
     let wasm = binaryen_module.write();
 
-    let mut dummy_environ = DummyEnvironment::with_triple(triple!("x86_64"));
+    let flags = settings::Flags::new(settings::builder());
+    let triple = triple!("x86_64");
+    let isa = isa::lookup(triple).unwrap().finish(flags);
+    let mut dummy_environ = DummyEnvironment::new(isa.frontend_config(), ReturnMode::NormalReturns);
     translate_module(&wasm, &mut dummy_environ).unwrap();
 });
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index bab6e1394a..3eeddf5241 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -106,11 +106,7 @@ fn handle_module(
         }
     };
 
-    let mut dummy_environ = DummyEnvironment::with_triple_flags(
-        isa.triple().clone(),
-        fisa.flags.clone(),
-        ReturnMode::NormalReturns,
-    );
+    let mut dummy_environ = DummyEnvironment::new(isa.frontend_config(), ReturnMode::NormalReturns);
     translate_module(&module_binary, &mut dummy_environ).map_err(|e| e.to_string())?;
 
     let _ = terminal.fg(term::color::GREEN);
diff --git a/lib/codegen/meta-python/base/settings.py b/lib/codegen/meta-python/base/settings.py
index 4eb7541840..534ef28830 100644
--- a/lib/codegen/meta-python/base/settings.py
+++ b/lib/codegen/meta-python/base/settings.py
@@ -27,30 +27,6 @@ enable_verifier = BoolSetting(
         """,
         default=True)
 
-call_conv = EnumSetting(
-        """
-        Default calling convention:
-
-        - fast: not-ABI-stable convention for best performance
-        - cold: not-ABI-stable convention for infrequently executed code
-        - system_v: System V-style convention used on many platforms
-        - windows_fastcall: Windows "fastcall" convention, also used for
-                            x64 and ARM
-        - baldrdash: SpiderMonkey WebAssembly convention
-        - probestack: specialized convention for the probestack function
-
-        The default calling convention may be overridden by individual
-        functions.
-        """,
-
-        'fast',
-        'cold',
-        'system_v',
-        'windows_fastcall',
-        'baldrdash',
-        'probestack'
-)
-
 # Note that Cranelift doesn't currently need an is_pie flag, because PIE is
 # just PIC where symbols can't be pre-empted, which can be expressed with the
 # `colocated` flag on external functions and global values.
diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs
index 451a90a7f9..7ee4fbe8e2 100644
--- a/lib/codegen/src/ir/extfunc.rs
+++ b/lib/codegen/src/ir/extfunc.rs
@@ -6,8 +6,7 @@
 //! This module declares the data types used to represent external functions and call signatures.
 
 use ir::{ArgumentLoc, ExternalName, SigRef, Type};
-use isa::{RegInfo, RegUnit};
-use settings::CallConv;
+use isa::{CallConv, RegInfo, RegUnit};
 use std::fmt;
 use std::str::FromStr;
 use std::vec::Vec;
diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs
index f2db834eab..629741b4ae 100644
--- a/lib/codegen/src/ir/function.rs
+++ b/lib/codegen/src/ir/function.rs
@@ -13,9 +13,8 @@ use ir::{
 };
 use ir::{EbbOffsets, InstEncodings, SourceLocs, StackSlots, ValueLocations};
 use ir::{JumpTableOffsets, JumpTables};
-use isa::{EncInfo, Encoding, Legalize, TargetIsa};
+use isa::{CallConv, EncInfo, Encoding, Legalize, TargetIsa};
 use regalloc::RegDiversions;
-use settings::CallConv;
 use std::fmt;
 use write::write_function;
 
diff --git a/lib/codegen/src/ir/libcall.rs b/lib/codegen/src/ir/libcall.rs
index 61766dc8ae..1e5297022e 100644
--- a/lib/codegen/src/ir/libcall.rs
+++ b/lib/codegen/src/ir/libcall.rs
@@ -4,8 +4,7 @@ use ir::{
     types, AbiParam, ArgumentPurpose, ExtFuncData, ExternalName, FuncRef, Function, Inst, Opcode,
     Signature, Type,
 };
-use isa::{RegUnit, TargetIsa};
-use settings::CallConv;
+use isa::{CallConv, RegUnit, TargetIsa};
 use std::fmt;
 use std::str::FromStr;
 
@@ -166,8 +165,7 @@ fn make_funcref_for_inst(
     inst: Inst,
     isa: &TargetIsa,
 ) -> FuncRef {
-    // Start with a fast calling convention. We'll give the ISA a chance to change it.
-    let mut sig = Signature::new(isa.flags().call_conv());
+    let mut sig = Signature::new(isa.default_call_conv());
     for &v in func.dfg.inst_args(inst) {
         sig.params.push(AbiParam::new(func.dfg.value_type(v)));
     }
diff --git a/lib/codegen/src/isa/call_conv.rs b/lib/codegen/src/isa/call_conv.rs
new file mode 100644
index 0000000000..ce5b10b315
--- /dev/null
+++ b/lib/codegen/src/isa/call_conv.rs
@@ -0,0 +1,74 @@
+use std::fmt;
+use std::str;
+use target_lexicon::{OperatingSystem, Triple};
+
+/// Calling convention identifiers.
+#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]
+pub enum CallConv {
+    /// Best performance, not ABI-stable
+    Fast,
+    /// Smallest caller code size, not ABI-stable
+    Cold,
+    /// System V-style convention used on many platforms
+    SystemV,
+    /// Windows "fastcall" convention, also used for x64 and ARM
+    WindowsFastcall,
+    /// SpiderMonkey WebAssembly convention
+    Baldrdash,
+    /// Specialized convention for the probestack function
+    Probestack,
+}
+
+impl CallConv {
+    /// Return the default calling convention for the given target triple.
+    pub fn default_for_triple(triple: &Triple) -> Self {
+        match triple.operating_system {
+            OperatingSystem::Unknown
+            | OperatingSystem::Bitrig
+            | OperatingSystem::Cloudabi
+            | OperatingSystem::Darwin
+            | OperatingSystem::Dragonfly
+            | OperatingSystem::Freebsd
+            | OperatingSystem::Fuchsia
+            | OperatingSystem::Haiku
+            | OperatingSystem::Ios
+            | OperatingSystem::L4re
+            | OperatingSystem::Linux
+            | OperatingSystem::Nebulet
+            | OperatingSystem::Netbsd
+            | OperatingSystem::Openbsd
+            | OperatingSystem::Redox
+            | OperatingSystem::Solaris => CallConv::SystemV,
+            OperatingSystem::Windows => CallConv::WindowsFastcall,
+            os => panic!("unsupported operating system: {}", os),
+        }
+    }
+}
+
+impl fmt::Display for CallConv {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        f.write_str(match *self {
+            CallConv::Fast => "fast",
+            CallConv::Cold => "cold",
+            CallConv::SystemV => "system_v",
+            CallConv::WindowsFastcall => "windows_fastcall",
+            CallConv::Baldrdash => "baldrdash",
+            CallConv::Probestack => "probestack",
+        })
+    }
+}
+
+impl str::FromStr for CallConv {
+    type Err = ();
+    fn from_str(s: &str) -> Result {
+        match s {
+            "fast" => Ok(CallConv::Fast),
+            "cold" => Ok(CallConv::Cold),
+            "system_v" => Ok(CallConv::SystemV),
+            "windows_fastcall" => Ok(CallConv::WindowsFastcall),
+            "baldrdash" => Ok(CallConv::Baldrdash),
+            "probestack" => Ok(CallConv::Probestack),
+            _ => Err(()),
+        }
+    }
+}
diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs
index 678395f43c..20469bb6c0 100644
--- a/lib/codegen/src/isa/mod.rs
+++ b/lib/codegen/src/isa/mod.rs
@@ -46,6 +46,7 @@
 //! The configured target ISA trait object is a `Box` which can be used for multiple
 //! concurrent function compilations.
 
+pub use isa::call_conv::CallConv;
 pub use isa::constraints::{BranchRange, ConstraintKind, OperandConstraint, RecipeConstraints};
 pub use isa::encoding::{base_size, EncInfo, Encoding};
 pub use isa::registers::{regs_overlap, RegClass, RegClassIndex, RegInfo, RegUnit};
@@ -58,10 +59,10 @@ use isa::enc_tables::Encodings;
 use regalloc;
 use result::CodegenResult;
 use settings;
-use settings::{CallConv, SetResult};
+use settings::SetResult;
 use std::boxed::Box;
 use std::fmt;
-use target_lexicon::{Architecture, Triple};
+use target_lexicon::{Architecture, PointerWidth, Triple};
 use timing;
 
 #[cfg(build_riscv)]
@@ -76,6 +77,7 @@ mod arm32;
 #[cfg(build_arm64)]
 mod arm64;
 
+mod call_conv;
 mod constraints;
 mod enc_tables;
 mod encoding;
@@ -164,6 +166,34 @@ impl settings::Configurable for Builder {
 pub type Legalize =
     fn(ir::Inst, &mut ir::Function, &mut flowgraph::ControlFlowGraph, &TargetIsa) -> bool;
 
+/// This struct provides information that a frontend may need to know about a target to
+/// produce Cranelift IR for the target.
+#[derive(Clone, Copy)]
+pub struct TargetFrontendConfig {
+    /// The default calling convention of the target.
+    pub default_call_conv: CallConv,
+
+    /// The pointer width of the target.
+    pub pointer_width: PointerWidth,
+}
+
+impl TargetFrontendConfig {
+    /// Get the pointer type of this target.
+    pub fn pointer_type(&self) -> ir::Type {
+        ir::Type::int(u16::from(self.pointer_bits())).unwrap()
+    }
+
+    /// Get the width of pointers on this target, in units of bits.
+    pub fn pointer_bits(&self) -> u8 {
+        self.pointer_width.bits()
+    }
+
+    /// Get the width of pointers on this target, in units of bytes.
+    pub fn pointer_bytes(&self) -> u8 {
+        self.pointer_width.bytes()
+    }
+}
+
 /// Methods that are specialized to a target ISA. Implies a Display trait that shows the
 /// shared flags, as well as any isa-specific flags.
 pub trait TargetIsa: fmt::Display {
@@ -176,19 +206,37 @@ pub trait TargetIsa: fmt::Display {
     /// Get the ISA-independent flags that were used to make this trait object.
     fn flags(&self) -> &settings::Flags;
 
+    /// Get the default calling convention of this target.
+    fn default_call_conv(&self) -> CallConv {
+        CallConv::default_for_triple(self.triple())
+    }
+
     /// Get the pointer type of this ISA.
     fn pointer_type(&self) -> ir::Type {
         ir::Type::int(u16::from(self.pointer_bits())).unwrap()
     }
 
+    /// Get the width of pointers on this ISA.
+    fn pointer_width(&self) -> PointerWidth {
+        self.triple().pointer_width().unwrap()
+    }
+
     /// Get the width of pointers on this ISA, in units of bits.
     fn pointer_bits(&self) -> u8 {
-        self.triple().pointer_width().unwrap().bits()
+        self.pointer_width().bits()
     }
 
     /// Get the width of pointers on this ISA, in units of bytes.
     fn pointer_bytes(&self) -> u8 {
-        self.triple().pointer_width().unwrap().bytes()
+        self.pointer_width().bytes()
+    }
+
+    /// Get the information needed by frontends producing Cranelift IR.
+    fn frontend_config(&self) -> TargetFrontendConfig {
+        TargetFrontendConfig {
+            default_call_conv: self.default_call_conv(),
+            pointer_width: self.pointer_width(),
+        }
     }
 
     /// Does the CPU implement scalar comparisons using a CPU flags register?
diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs
index af7dda69d1..67fe3a486a 100644
--- a/lib/codegen/src/isa/x86/abi.rs
+++ b/lib/codegen/src/isa/x86/abi.rs
@@ -10,10 +10,9 @@ use ir::{
     get_probestack_funcref, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, InstBuilder,
     ValueLoc,
 };
-use isa::{RegClass, RegUnit, TargetIsa};
+use isa::{CallConv, RegClass, RegUnit, TargetIsa};
 use regalloc::RegisterSet;
 use result::CodegenResult;
-use settings::CallConv;
 use stack_layout::layout_stack;
 use std::i32;
 use target_lexicon::{PointerWidth, Triple};
@@ -189,12 +188,12 @@ pub fn allocatable_registers(_func: &ir::Function, triple: &Triple) -> RegisterS
 }
 
 /// Get the set of callee-saved registers.
-fn callee_saved_gprs(isa: &TargetIsa) -> &'static [RU] {
+fn callee_saved_gprs(isa: &TargetIsa, call_conv: CallConv) -> &'static [RU] {
     match isa.triple().pointer_width().unwrap() {
         PointerWidth::U16 => panic!(),
         PointerWidth::U32 => &[RU::rbx, RU::rsi, RU::rdi],
         PointerWidth::U64 => {
-            if isa.flags().call_conv() == CallConv::WindowsFastcall {
+            if call_conv == CallConv::WindowsFastcall {
                 // "registers RBX, RBP, RDI, RSI, RSP, R12, R13, R14, R15 are considered nonvolatile
                 //  and must be saved and restored by a function that uses them."
                 // as per https://msdn.microsoft.com/en-us/library/6t169e9c.aspx
@@ -219,7 +218,7 @@ fn callee_saved_gprs(isa: &TargetIsa) -> &'static [RU] {
 /// Get the set of callee-saved registers that are used.
 fn callee_saved_gprs_used(isa: &TargetIsa, func: &ir::Function) -> RegisterSet {
     let mut all_callee_saved = RegisterSet::empty();
-    for reg in callee_saved_gprs(isa) {
+    for reg in callee_saved_gprs(isa, func.signature.call_conv) {
         all_callee_saved.free(GPR, *reg as RegUnit);
     }
 
diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs
index cd135eba4b..50cd38b7fa 100644
--- a/lib/codegen/src/settings.rs
+++ b/lib/codegen/src/settings.rs
@@ -372,7 +372,6 @@ mod tests {
             "[shared]\n\
              opt_level = \"default\"\n\
              enable_verifier = true\n\
-             call_conv = \"fast\"\n\
              is_pic = false\n\
              colocated_libcalls = false\n\
              avoid_div_traps = false\n\
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 4014072e4a..ff6efa6835 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -9,7 +9,7 @@ use cranelift_codegen::ir::{
     JumpTable, JumpTableData, LibCall, MemFlags, SigRef, Signature, StackSlot, StackSlotData, Type,
     Value,
 };
-use cranelift_codegen::isa::TargetIsa;
+use cranelift_codegen::isa::{TargetFrontendConfig, TargetIsa};
 use cranelift_codegen::packed_option::PackedOption;
 use ssa::{Block, SSABuilder, SideEffects};
 use std::vec::Vec;
@@ -550,10 +550,16 @@ impl<'a> FunctionBuilder<'a> {
     /// won't overlap onto `dest`. If `dest` and `src` overlap, the behavior is
     /// undefined. Applications in which `dest` and `src` might overlap should
     /// use `call_memmove` instead.
-    pub fn call_memcpy(&mut self, isa: &TargetIsa, dest: Value, src: Value, size: Value) {
-        let pointer_type = isa.pointer_type();
+    pub fn call_memcpy(
+        &mut self,
+        config: &TargetFrontendConfig,
+        dest: Value,
+        src: Value,
+        size: Value,
+    ) {
+        let pointer_type = config.pointer_type();
         let signature = {
-            let mut s = Signature::new(isa.flags().call_conv());
+            let mut s = Signature::new(config.default_call_conv);
             s.params.push(AbiParam::new(pointer_type));
             s.params.push(AbiParam::new(pointer_type));
             s.params.push(AbiParam::new(pointer_type));
@@ -572,7 +578,7 @@ impl<'a> FunctionBuilder<'a> {
     /// Optimised memcpy for small copys.
     pub fn emit_small_memcpy(
         &mut self,
-        isa: &TargetIsa,
+        config: &TargetFrontendConfig,
         dest: Value,
         src: Value,
         size: u64,
@@ -594,8 +600,8 @@ impl<'a> FunctionBuilder<'a> {
         let load_and_store_amount = size / access_size;
 
         if load_and_store_amount > THRESHOLD {
-            let size_value = self.ins().iconst(isa.pointer_type(), size as i64);
-            self.call_memcpy(isa, dest, src, size_value);
+            let size_value = self.ins().iconst(config.pointer_type(), size as i64);
+            self.call_memcpy(config, dest, src, size_value);
             return;
         }
 
@@ -613,10 +619,16 @@ impl<'a> FunctionBuilder<'a> {
     /// Calls libc.memset
     ///
     /// Writes `size` bytes of value `ch` to memory starting at `buffer`.
-    pub fn call_memset(&mut self, isa: &TargetIsa, buffer: Value, ch: Value, size: Value) {
-        let pointer_type = isa.pointer_type();
+    pub fn call_memset(
+        &mut self,
+        config: &TargetFrontendConfig,
+        buffer: Value,
+        ch: Value,
+        size: Value,
+    ) {
+        let pointer_type = config.pointer_type();
         let signature = {
-            let mut s = Signature::new(isa.flags().call_conv());
+            let mut s = Signature::new(config.default_call_conv);
             s.params.push(AbiParam::new(pointer_type));
             s.params.push(AbiParam::new(types::I32));
             s.params.push(AbiParam::new(pointer_type));
@@ -638,7 +650,7 @@ impl<'a> FunctionBuilder<'a> {
     /// Writes `size` bytes of value `ch` to memory starting at `buffer`.
     pub fn emit_small_memset(
         &mut self,
-        isa: &TargetIsa,
+        config: &TargetFrontendConfig,
         buffer: Value,
         ch: u32,
         size: u64,
@@ -660,8 +672,8 @@ impl<'a> FunctionBuilder<'a> {
 
         if load_and_store_amount > THRESHOLD {
             let ch = self.ins().iconst(types::I32, ch as i64);
-            let size = self.ins().iconst(isa.pointer_type(), size as i64);
-            self.call_memset(isa, buffer, ch, size);
+            let size = self.ins().iconst(config.pointer_type(), size as i64);
+            self.call_memset(config, buffer, ch, size);
         } else {
             let mut flags = MemFlags::new();
             flags.set_aligned();
@@ -691,10 +703,16 @@ impl<'a> FunctionBuilder<'a> {
     ///
     /// Copies `size` bytes from memory starting at `source` to memory starting
     /// at `dest`. `source` is always read before writing to `dest`.
-    pub fn call_memmove(&mut self, isa: &TargetIsa, dest: Value, source: Value, size: Value) {
-        let pointer_type = isa.pointer_type();
+    pub fn call_memmove(
+        &mut self,
+        config: &TargetFrontendConfig,
+        dest: Value,
+        source: Value,
+        size: Value,
+    ) {
+        let pointer_type = config.pointer_type();
         let signature = {
-            let mut s = Signature::new(isa.flags().call_conv());
+            let mut s = Signature::new(config.default_call_conv);
             s.params.push(AbiParam::new(pointer_type));
             s.params.push(AbiParam::new(pointer_type));
             s.params.push(AbiParam::new(pointer_type));
@@ -713,7 +731,7 @@ impl<'a> FunctionBuilder<'a> {
     /// Optimised memmove for small moves.
     pub fn emit_small_memmove(
         &mut self,
-        isa: &TargetIsa,
+        config: &TargetFrontendConfig,
         dest: Value,
         src: Value,
         size: u64,
@@ -735,8 +753,8 @@ impl<'a> FunctionBuilder<'a> {
         let load_and_store_amount = size / access_size;
 
         if load_and_store_amount > THRESHOLD {
-            let size_value = self.ins().iconst(isa.pointer_type(), size as i64);
-            self.call_memmove(isa, dest, src, size_value);
+            let size_value = self.ins().iconst(config.pointer_type(), size as i64);
+            self.call_memmove(config, dest, src, size_value);
             return;
         }
 
@@ -798,8 +816,8 @@ mod tests {
     use cranelift_codegen::entity::EntityRef;
     use cranelift_codegen::ir::types::*;
     use cranelift_codegen::ir::{AbiParam, ExternalName, Function, InstBuilder, Signature};
+    use cranelift_codegen::isa::CallConv;
     use cranelift_codegen::settings;
-    use cranelift_codegen::settings::CallConv;
     use cranelift_codegen::verifier::verify_function;
     use frontend::{FunctionBuilder, FunctionBuilderContext};
     use std::string::ToString;
@@ -923,7 +941,7 @@ mod tests {
             .map(|b| b.finish(shared_flags))
             .expect("This test requires arm support.");
 
-        let mut sig = Signature::new(target.flags().call_conv());
+        let mut sig = Signature::new(target.default_call_conv());
         sig.returns.push(AbiParam::new(I32));
 
         let mut fn_ctx = FunctionBuilderContext::new();
@@ -944,7 +962,7 @@ mod tests {
             let src = builder.use_var(x);
             let dest = builder.use_var(y);
             let size = builder.use_var(y);
-            builder.call_memcpy(&*target, dest, src, size);
+            builder.call_memcpy(&target.frontend_config(), dest, src, size);
             builder.ins().return_(&[size]);
 
             builder.seal_all_blocks();
@@ -953,8 +971,8 @@ mod tests {
 
         assert_eq!(
             func.display(None).to_string(),
-            "function %sample() -> i32 fast {
-    sig0 = (i32, i32, i32) fast
+            "function %sample() -> i32 system_v {
+    sig0 = (i32, i32, i32) system_v
     fn0 = %Memcpy sig0
 
 ebb0:
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index 045705d723..01dcdc4a2a 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -68,7 +68,8 @@
 //! use cranelift_codegen::entity::EntityRef;
 //! use cranelift_codegen::ir::types::*;
 //! use cranelift_codegen::ir::{AbiParam, ExternalName, Function, InstBuilder, Signature};
-//! use cranelift_codegen::settings::{self, CallConv};
+//! use cranelift_codegen::isa::CallConv;
+//! use cranelift_codegen::settings;
 //! use cranelift_codegen::verifier::verify_function;
 //! use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
 //!
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 626c34e4f5..47daccd768 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -6,7 +6,7 @@
 // shared with `DataContext`?
 
 use cranelift_codegen::entity::{EntityRef, PrimaryMap};
-use cranelift_codegen::{binemit, ir, CodegenError, Context};
+use cranelift_codegen::{binemit, ir, isa, CodegenError, Context};
 use data_context::DataContext;
 use std::borrow::ToOwned;
 use std::collections::HashMap;
@@ -340,9 +340,10 @@ where
         self.names.get(name).cloned()
     }
 
-    /// Return then pointer type for the current target.
-    pub fn pointer_type(&self) -> ir::types::Type {
-        self.backend.isa().pointer_type()
+    /// Return the target information needed by frontends to produce Cranelift IR
+    /// for the current target.
+    pub fn target_config(&self) -> isa::TargetFrontendConfig {
+        self.backend.isa().frontend_config()
     }
 
     /// Create a new `Context` initialized for use with this `Module`.
@@ -351,7 +352,7 @@ where
     /// convention for the `TargetIsa`.
     pub fn make_context(&self) -> Context {
         let mut ctx = Context::new();
-        ctx.func.signature.call_conv = self.backend.isa().flags().call_conv();
+        ctx.func.signature.call_conv = self.backend.isa().default_call_conv();
         ctx
     }
 
@@ -361,14 +362,14 @@ where
     /// convention for the `TargetIsa`.
     pub fn clear_context(&self, ctx: &mut Context) {
         ctx.clear();
-        ctx.func.signature.call_conv = self.backend.isa().flags().call_conv();
+        ctx.func.signature.call_conv = self.backend.isa().default_call_conv();
     }
 
     /// Create a new empty `Signature` with the default calling convention for
     /// the `TargetIsa`, to which parameter and return types can be added for
     /// declaring a function to be called by this `Module`.
     pub fn make_signature(&self) -> ir::Signature {
-        ir::Signature::new(self.backend.isa().flags().call_conv())
+        ir::Signature::new(self.backend.isa().default_call_conv())
     }
 
     /// Clear the given `Signature` and reset for use with a new function.
@@ -376,7 +377,7 @@ where
     /// This ensures that the `Signature` is initialized with the default
     /// calling convention for the `TargetIsa`.
     pub fn clear_signature(&self, sig: &mut ir::Signature) {
-        sig.clear(self.backend.isa().flags().call_conv());
+        sig.clear(self.backend.isa().default_call_conv());
     }
 
     /// Declare a function in this module.
diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs
index 7d88984db5..57fa5a4cd9 100644
--- a/lib/native/src/lib.rs
+++ b/lib/native/src/lib.rs
@@ -37,26 +37,16 @@ extern crate raw_cpuid;
 extern crate target_lexicon;
 
 use cranelift_codegen::isa;
-use cranelift_codegen::settings::{self, Configurable};
+use cranelift_codegen::settings::Configurable;
 use target_lexicon::Triple;
 
 #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
 use raw_cpuid::CpuId;
 
-/// Return `settings` and `isa` builders configured for the current host
+/// Return an `isa` builder configured for the current host
 /// machine, or `Err(())` if the host machine is not supported
 /// in the current configuration.
-pub fn builders() -> Result<(settings::Builder, isa::Builder), &'static str> {
-    let mut flag_builder = settings::builder();
-
-    if cfg!(any(unix, target_os = "nebulet")) {
-        flag_builder.set("call_conv", "system_v").unwrap();
-    } else if cfg!(windows) {
-        flag_builder.set("call_conv", "windows_fastcall").unwrap();
-    } else {
-        return Err("unrecognized environment");
-    }
-
+pub fn builder() -> Result {
     let mut isa_builder = isa::lookup(Triple::host()).map_err(|err| match err {
         isa::LookupError::SupportDisabled => "support for architecture disabled at compile time",
         isa::LookupError::Unsupported => "unsupported architecture",
@@ -66,7 +56,7 @@ pub fn builders() -> Result<(settings::Builder, isa::Builder), &'static str> {
         parse_x86_cpuid(&mut isa_builder)?;
     }
 
-    Ok((flag_builder, isa_builder))
+    Ok(isa_builder)
 }
 
 #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
@@ -108,3 +98,32 @@ fn parse_x86_cpuid(isa_builder: &mut isa::Builder) -> Result<(), &'static str> {
     }
     Ok(())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::builder;
+    use cranelift_codegen::isa::CallConv;
+    use cranelift_codegen::settings;
+
+    #[test]
+    fn test() {
+        if let Ok(isa_builder) = builder() {
+            let flag_builder = settings::builder();
+            let isa = isa_builder.finish(settings::Flags::new(flag_builder));
+            if cfg!(any(unix, target_os = "nebulet")) {
+                assert_eq!(isa.default_call_conv(), CallConv::SystemV);
+            } else if cfg!(windows) {
+                assert_eq!(isa.default_call_conv(), CallConv::WindowsFastcall);
+            }
+            if cfg!(target_pointer_width = "64") {
+                assert_eq!(isa.pointer_bits(), 64);
+            }
+            if cfg!(target_pointer_width = "32") {
+                assert_eq!(isa.pointer_bits(), 32);
+            }
+            if cfg!(target_pointer_width = "16") {
+                assert_eq!(isa.pointer_bits(), 16);
+            }
+        }
+    }
+}
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 937edbee02..650efe3ea0 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -12,9 +12,8 @@ use cranelift_codegen::ir::{
     Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind, Table, TableData, Type,
     Value, ValueLoc,
 };
-use cranelift_codegen::isa::{self, Encoding, RegUnit, TargetIsa};
+use cranelift_codegen::isa::{self, CallConv, Encoding, RegUnit, TargetIsa};
 use cranelift_codegen::packed_option::ReservedValue;
-use cranelift_codegen::settings::CallConv;
 use cranelift_codegen::{settings, timing};
 use error::{Location, ParseError, ParseResult};
 use isaspec;
@@ -2562,7 +2561,7 @@ mod tests {
     use cranelift_codegen::ir::types;
     use cranelift_codegen::ir::StackSlotKind;
     use cranelift_codegen::ir::{ArgumentExtension, ArgumentPurpose};
-    use cranelift_codegen::settings::CallConv;
+    use cranelift_codegen::isa::CallConv;
     use error::ParseError;
     use isaspec::IsaSpec;
     use testfile::{Comment, Details};
diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index f774adbbaa..ccf113ce1c 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -26,8 +26,9 @@ pub struct SimpleJITBuilder {
 impl SimpleJITBuilder {
     /// Create a new `SimpleJITBuilder`.
     pub fn new() -> Self {
-        let (flag_builder, isa_builder) = cranelift_native::builders().unwrap_or_else(|_| {
-            panic!("host machine is not a supported target");
+        let flag_builder = settings::builder();
+        let isa_builder = cranelift_native::builder().unwrap_or_else(|msg| {
+            panic!("host machine is not supported: {}", msg);
         });
         let isa = isa_builder.finish(settings::Flags::new(flag_builder));
         Self::with_isa(isa)
diff --git a/lib/simplejit/tests/basic.rs b/lib/simplejit/tests/basic.rs
index e76282e74e..1614686911 100644
--- a/lib/simplejit/tests/basic.rs
+++ b/lib/simplejit/tests/basic.rs
@@ -5,7 +5,7 @@ extern crate cranelift_module;
 extern crate cranelift_simplejit;
 
 use cranelift_codegen::ir::*;
-use cranelift_codegen::settings::*;
+use cranelift_codegen::isa::CallConv;
 use cranelift_codegen::Context;
 use cranelift_entity::EntityRef;
 use cranelift_frontend::*;
diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs
index 79d262ecfd..7a3a8f00dc 100644
--- a/lib/umbrella/src/lib.rs
+++ b/lib/umbrella/src/lib.rs
@@ -46,7 +46,7 @@ pub mod prelude {
         MemFlags, Signature, StackSlotData, StackSlotKind, TrapCode, Type, Value,
     };
     pub use codegen::isa;
-    pub use codegen::settings::{self, CallConv, Configurable};
+    pub use codegen::settings::{self, Configurable};
 
     pub use frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
 }
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index d298060f76..28547d784f 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -17,15 +17,15 @@ cranelift-frontend = { path = "../frontend", version = "0.22.0", default-feature
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
-target-lexicon = { version = "0.0.3", default-features = false }
 log = { version = "0.4.4", default-features = false }
 
 [dev-dependencies]
 wabt = "0.6.0"
+target-lexicon = "0.0.3"
 
 [features]
 default = ["std"]
-std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std", "target-lexicon/std"]
+std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std"]
 core = ["hashmap_core", "cranelift-codegen/core", "cranelift-frontend/core", "wasmparser/core"]
 
 [badges]
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 20e1462f50..ae2183efb9 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -5,13 +5,12 @@ use cranelift_codegen::cursor::FuncCursor;
 use cranelift_codegen::ir::immediates::{Imm64, Offset32};
 use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{self, InstBuilder};
-use cranelift_codegen::settings;
+use cranelift_codegen::isa::TargetFrontendConfig;
 use cranelift_entity::{EntityRef, PrimaryMap};
 use environ::{FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmResult};
 use func_translator::FuncTranslator;
 use std::string::String;
 use std::vec::Vec;
-use target_lexicon::Triple;
 use translation_utils::{
     DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table,
     TableIndex,
@@ -44,11 +43,8 @@ impl Exportable {
 /// `DummyEnvironment` to allow it to be borrowed separately from the
 /// `FuncTranslator` field.
 pub struct DummyModuleInfo {
-    /// Target description.
-    pub triple: Triple,
-
-    /// Compilation setting flags.
-    pub flags: settings::Flags,
+    /// Target description relevant to frontends producing Cranelift IR.
+    config: TargetFrontendConfig,
 
     /// Signatures as provided by `declare_signature`.
     pub signatures: PrimaryMap,
@@ -76,11 +72,10 @@ pub struct DummyModuleInfo {
 }
 
 impl DummyModuleInfo {
-    /// Allocates the data structures with the given flags.
-    pub fn with_triple_flags(triple: Triple, flags: settings::Flags) -> Self {
+    /// Creates a new `DummyModuleInfo` instance.
+    pub fn new(config: TargetFrontendConfig) -> Self {
         Self {
-            triple,
-            flags,
+            config,
             signatures: PrimaryMap::new(),
             imported_funcs: Vec::new(),
             functions: PrimaryMap::new(),
@@ -111,23 +106,10 @@ pub struct DummyEnvironment {
 }
 
 impl DummyEnvironment {
-    /// Allocates the data structures with default flags.
-    pub fn with_triple(triple: Triple) -> Self {
-        Self::with_triple_flags(
-            triple,
-            settings::Flags::new(settings::builder()),
-            ReturnMode::NormalReturns,
-        )
-    }
-
-    /// Allocates the data structures with the given triple.
-    pub fn with_triple_flags(
-        triple: Triple,
-        flags: settings::Flags,
-        return_mode: ReturnMode,
-    ) -> Self {
+    /// Creates a new `DummyEnvironment` instance.
+    pub fn new(config: TargetFrontendConfig, return_mode: ReturnMode) -> Self {
         Self {
-            info: DummyModuleInfo::with_triple_flags(triple, flags),
+            info: DummyModuleInfo::new(config),
             trans: FuncTranslator::new(),
             func_bytecode_sizes: Vec::new(),
             return_mode,
@@ -169,12 +151,8 @@ impl<'dummy_environment> DummyFuncEnvironment<'dummy_environment> {
 }
 
 impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environment> {
-    fn triple(&self) -> &Triple {
-        &self.mod_info.triple
-    }
-
-    fn flags(&self) -> &settings::Flags {
-        &self.mod_info.flags
+    fn target_config(&self) -> TargetFrontendConfig {
+        self.mod_info.config
     }
 
     fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalVariable {
@@ -348,8 +326,8 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
 }
 
 impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
-    fn flags(&self) -> &settings::Flags {
-        &self.info.flags
+    fn target_config(&self) -> &TargetFrontendConfig {
+        &self.info.config
     }
 
     fn get_func_name(&self, func_index: FuncIndex) -> ir::ExternalName {
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index db6d0761d4..7f48ef99d2 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -2,9 +2,8 @@
 //! traits `FunctionEnvironment` and `ModuleEnvironment`.
 use cranelift_codegen::cursor::FuncCursor;
 use cranelift_codegen::ir::{self, InstBuilder};
-use cranelift_codegen::settings::Flags;
+use cranelift_codegen::isa::TargetFrontendConfig;
 use std::vec::Vec;
-use target_lexicon::Triple;
 use translation_utils::{
     FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
 };
@@ -89,22 +88,19 @@ pub enum ReturnMode {
 /// IR. The function environment provides information about the WebAssembly module as well as the
 /// runtime environment.
 pub trait FuncEnvironment {
-    /// Get the triple for the current compilation.
-    fn triple(&self) -> &Triple;
-
-    /// Get the flags for the current compilation.
-    fn flags(&self) -> &Flags;
+    /// Get the information needed to produce Cranelift IR for the given target.
+    fn target_config(&self) -> TargetFrontendConfig;
 
     /// Get the Cranelift integer type to use for native pointers.
     ///
     /// This returns `I64` for 64-bit architectures and `I32` for 32-bit architectures.
     fn pointer_type(&self) -> ir::Type {
-        ir::Type::int(u16::from(self.triple().pointer_width().unwrap().bits())).unwrap()
+        ir::Type::int(u16::from(self.target_config().pointer_bits())).unwrap()
     }
 
     /// Get the size of a native pointer, in bytes.
     fn pointer_bytes(&self) -> u8 {
-        self.triple().pointer_width().unwrap().bytes()
+        self.target_config().pointer_bytes()
     }
 
     /// Set up the necessary preamble definitions in `func` to access the global variable
@@ -239,8 +235,8 @@ pub trait FuncEnvironment {
 /// [`translate_module`](fn.translate_module.html) function. These methods should not be called
 /// by the user, they are only for `cranelift-wasm` internal use.
 pub trait ModuleEnvironment<'data> {
-    /// Get the flags for the current compilation.
-    fn flags(&self) -> &Flags;
+    /// Get the information needed to produce Cranelift IR for the current target.
+    fn target_config(&self) -> &TargetFrontendConfig;
 
     /// Return the name for the given function index.
     fn get_func_name(&self, func_index: FuncIndex) -> ir::ExternalName;
diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs
index 0d99754c79..5b7a116454 100644
--- a/lib/wasm/src/func_translator.rs
+++ b/lib/wasm/src/func_translator.rs
@@ -236,11 +236,11 @@ fn cur_srcloc(reader: &BinaryReader) -> ir::SourceLoc {
 
 #[cfg(test)]
 mod tests {
-    use super::FuncTranslator;
+    use super::{FuncTranslator, ReturnMode};
     use cranelift_codegen::ir::types::I32;
-    use cranelift_codegen::{ir, Context};
-    use environ::{DummyEnvironment, FuncEnvironment};
-    use target_lexicon::Triple;
+    use cranelift_codegen::{ir, isa, settings, Context};
+    use environ::DummyEnvironment;
+    use target_lexicon::PointerWidth;
 
     #[test]
     fn small1() {
@@ -258,7 +258,15 @@ mod tests {
         ];
 
         let mut trans = FuncTranslator::new();
-        let runtime = DummyEnvironment::with_triple(Triple::default());
+        let flags = settings::Flags::new(settings::builder());
+        let runtime = DummyEnvironment::new(
+            isa::TargetFrontendConfig {
+                default_call_conv: isa::CallConv::Fast,
+                pointer_width: PointerWidth::U64,
+            },
+            ReturnMode::NormalReturns,
+        );
+
         let mut ctx = Context::new();
 
         ctx.func.name = ir::ExternalName::testcase("small1");
@@ -269,7 +277,7 @@ mod tests {
             .translate(&BODY, &mut ctx.func, &mut runtime.func_env())
             .unwrap();
         debug!("{}", ctx.func.display(None));
-        ctx.verify(runtime.func_env().flags()).unwrap();
+        ctx.verify(&flags).unwrap();
     }
 
     #[test]
@@ -289,7 +297,14 @@ mod tests {
         ];
 
         let mut trans = FuncTranslator::new();
-        let runtime = DummyEnvironment::with_triple(Triple::default());
+        let flags = settings::Flags::new(settings::builder());
+        let runtime = DummyEnvironment::new(
+            isa::TargetFrontendConfig {
+                default_call_conv: isa::CallConv::Fast,
+                pointer_width: PointerWidth::U64,
+            },
+            ReturnMode::NormalReturns,
+        );
         let mut ctx = Context::new();
 
         ctx.func.name = ir::ExternalName::testcase("small2");
@@ -300,7 +315,7 @@ mod tests {
             .translate(&BODY, &mut ctx.func, &mut runtime.func_env())
             .unwrap();
         debug!("{}", ctx.func.display(None));
-        ctx.verify(runtime.func_env().flags()).unwrap();
+        ctx.verify(&flags).unwrap();
     }
 
     #[test]
@@ -329,7 +344,14 @@ mod tests {
         ];
 
         let mut trans = FuncTranslator::new();
-        let runtime = DummyEnvironment::with_triple(Triple::default());
+        let flags = settings::Flags::new(settings::builder());
+        let runtime = DummyEnvironment::new(
+            isa::TargetFrontendConfig {
+                default_call_conv: isa::CallConv::Fast,
+                pointer_width: PointerWidth::U64,
+            },
+            ReturnMode::NormalReturns,
+        );
         let mut ctx = Context::new();
 
         ctx.func.name = ir::ExternalName::testcase("infloop");
@@ -339,6 +361,6 @@ mod tests {
             .translate(&BODY, &mut ctx.func, &mut runtime.func_env())
             .unwrap();
         debug!("{}", ctx.func.display(None));
-        ctx.verify(runtime.func_env().flags()).unwrap();
+        ctx.verify(&flags).unwrap();
     }
 }
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index 21b757f9c8..76da0113ac 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -40,6 +40,7 @@ extern crate cranelift_codegen;
 #[macro_use]
 extern crate cranelift_entity;
 extern crate cranelift_frontend;
+#[cfg(test)]
 extern crate target_lexicon;
 extern crate wasmparser;
 
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 1a09033705..3865a9c026 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -35,7 +35,7 @@ pub fn parse_function_signatures(
                 ref params,
                 ref returns,
             }) => {
-                let mut sig = Signature::new(environ.flags().call_conv());
+                let mut sig = Signature::new(environ.target_config().default_call_conv);
                 sig.params.extend(params.iter().map(|ty| {
                     let cret_arg: ir::Type = type_to_type(*ty)
                         .expect("only numeric types are supported in function signatures");
diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs
index be9c836463..bf0dee04dc 100644
--- a/lib/wasm/tests/wasm_testsuite.rs
+++ b/lib/wasm/tests/wasm_testsuite.rs
@@ -75,14 +75,12 @@ fn handle_module(path: &Path, flags: &Flags, return_mode: ReturnMode) {
             None | Some(&_) => panic!("the file extension for {:?} is not wasm or wat", path),
         },
     };
-    let mut dummy_environ =
-        DummyEnvironment::with_triple_flags(triple!("riscv64"), flags.clone(), return_mode);
+    let triple = triple!("riscv64");
+    let isa = isa::lookup(triple).unwrap().finish(flags.clone());
+    let mut dummy_environ = DummyEnvironment::new(isa.frontend_config(), return_mode);
 
     translate_module(&data, &mut dummy_environ).unwrap();
 
-    let isa = isa::lookup(dummy_environ.info.triple)
-        .unwrap()
-        .finish(dummy_environ.info.flags);
     for func in dummy_environ.info.function_bodies.values() {
         verifier::verify_function(func, &*isa)
             .map_err(|errors| panic!(pretty_verifier_error(func, Some(&*isa), None, errors)))

From a5cad9a748819597a4b3a8cbd7881cdc6b47fe82 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 2 Nov 2018 13:33:30 -0700
Subject: [PATCH 2161/3084] Fix a typo in a comment.

---
 lib/codegen/src/regalloc/virtregs.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/codegen/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs
index 5d8992a179..55fece93c1 100644
--- a/lib/codegen/src/regalloc/virtregs.rs
+++ b/lib/codegen/src/regalloc/virtregs.rs
@@ -280,7 +280,7 @@ impl UFEntry {
 /// 2. When done, call `finish_union_find()` to construct the virtual register sets based on the
 ///    `union()` calls.
 ///
-/// The values that were passed to `union(a, b)` mist not belong to any existing virtual registers
+/// The values that were passed to `union(a, b)` must not belong to any existing virtual registers
 /// by the time `finish_union_find()` is called.
 ///
 /// For more information on the algorithm implemented here, see Chapter 21 "Data Structures for

From 3ff8867e5785bf7ea74bd60b4cdeacf040da7c8c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 26 Oct 2018 06:20:08 -0700
Subject: [PATCH 2162/3084] Split the default edge of a br_table.

When splitting critical edges for a br_table to handle arguments being
passed, split the default edge along with the normal table edges.
---
 cranelift/wasmtests/br_table.wat | 30 ++++++++++++++++++++++++++++++
 lib/wasm/src/code_translator.rs  | 17 +++++++++--------
 2 files changed, 39 insertions(+), 8 deletions(-)
 create mode 100644 cranelift/wasmtests/br_table.wat

diff --git a/cranelift/wasmtests/br_table.wat b/cranelift/wasmtests/br_table.wat
new file mode 100644
index 0000000000..75444fa49c
--- /dev/null
+++ b/cranelift/wasmtests/br_table.wat
@@ -0,0 +1,30 @@
+(module
+  (func (result i32)
+    (block (result i32)
+      (block (result i32)
+        (block (result i32)
+          (br_table 0 1 2 3 (i32.const 42) (i32.const 0))
+        )
+      )
+    )
+  )
+  (func (result i32)
+    (block (result i32)
+      (block (result i32)
+        (block (result i32)
+          (br_table 3 2 1 0 (i32.const 42) (i32.const 0))
+        )
+      )
+    )
+  )
+  (func (result i32)
+    (block (result i32)
+      (br_table 0 0 1 1 (i32.const 42) (i32.const 0))
+    )
+  )
+  (func (result i32)
+    (block (result i32)
+      (br_table 1 1 0 0 (i32.const 42) (i32.const 0))
+    )
+  )
+)
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index e7a62b65fe..8d5fee979a 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -307,15 +307,16 @@ pub fn translate_operator(
                     };
                     data.push_entry(branch_ebb);
                 }
-                let jt = builder.create_jump_table(data);
-                let default_ebb = {
-                    let i = state.control_stack.len() - 1 - (default as usize);
-                    let frame = &mut state.control_stack[i];
-                    frame.set_branched_to_exit();
-                    frame.br_destination()
+                let default_branch_ebb = match dest_ebb_map.entry(default as usize) {
+                    hash_map::Entry::Occupied(entry) => *entry.get(),
+                    hash_map::Entry::Vacant(entry) => {
+                        let ebb = builder.create_ebb();
+                        dest_ebb_sequence.push((default as usize, ebb));
+                        *entry.insert(ebb)
+                    }
                 };
-                dest_ebb_sequence.push((default as usize, default_ebb));
-                builder.ins().br_table(val, default_ebb, jt);
+                let jt = builder.create_jump_table(data);
+                builder.ins().br_table(val, default_branch_ebb, jt);
                 for (depth, dest_ebb) in dest_ebb_sequence {
                     builder.switch_to_block(dest_ebb);
                     builder.seal_block(dest_ebb);

From 8d7538049cbda1efcef0595478a35a00e7192750 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 5 Nov 2018 12:51:39 -0800
Subject: [PATCH 2163/3084] Pass TargetFrontendConfig by value rather than by
 reference.

Passing it by reference was an artifact of an earlier version of the
TargetFrontendConfig code, but it's no longer needed, as
TargetFrontendConfig is now a Copy type.
---
 lib/frontend/src/frontend.rs | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index ff6efa6835..0c595a585f 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -552,7 +552,7 @@ impl<'a> FunctionBuilder<'a> {
     /// use `call_memmove` instead.
     pub fn call_memcpy(
         &mut self,
-        config: &TargetFrontendConfig,
+        config: TargetFrontendConfig,
         dest: Value,
         src: Value,
         size: Value,
@@ -578,7 +578,7 @@ impl<'a> FunctionBuilder<'a> {
     /// Optimised memcpy for small copys.
     pub fn emit_small_memcpy(
         &mut self,
-        config: &TargetFrontendConfig,
+        config: TargetFrontendConfig,
         dest: Value,
         src: Value,
         size: u64,
@@ -621,7 +621,7 @@ impl<'a> FunctionBuilder<'a> {
     /// Writes `size` bytes of value `ch` to memory starting at `buffer`.
     pub fn call_memset(
         &mut self,
-        config: &TargetFrontendConfig,
+        config: TargetFrontendConfig,
         buffer: Value,
         ch: Value,
         size: Value,
@@ -650,7 +650,7 @@ impl<'a> FunctionBuilder<'a> {
     /// Writes `size` bytes of value `ch` to memory starting at `buffer`.
     pub fn emit_small_memset(
         &mut self,
-        config: &TargetFrontendConfig,
+        config: TargetFrontendConfig,
         buffer: Value,
         ch: u32,
         size: u64,
@@ -705,7 +705,7 @@ impl<'a> FunctionBuilder<'a> {
     /// at `dest`. `source` is always read before writing to `dest`.
     pub fn call_memmove(
         &mut self,
-        config: &TargetFrontendConfig,
+        config: TargetFrontendConfig,
         dest: Value,
         source: Value,
         size: Value,
@@ -731,7 +731,7 @@ impl<'a> FunctionBuilder<'a> {
     /// Optimised memmove for small moves.
     pub fn emit_small_memmove(
         &mut self,
-        config: &TargetFrontendConfig,
+        config: TargetFrontendConfig,
         dest: Value,
         src: Value,
         size: u64,
@@ -962,7 +962,7 @@ mod tests {
             let src = builder.use_var(x);
             let dest = builder.use_var(y);
             let size = builder.use_var(y);
-            builder.call_memcpy(&target.frontend_config(), dest, src, size);
+            builder.call_memcpy(target.frontend_config(), dest, src, size);
             builder.ins().return_(&[size]);
 
             builder.seal_all_blocks();

From 17e88ed1c5e6182eb5d9857919b69004507bb10f Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 16 Oct 2018 16:47:06 +0200
Subject: [PATCH 2164/3084] [build] Use the Rust-generated types files in place
 of the Python one;

---
 lib/codegen/build.rs                 |  2 +-
 lib/codegen/meta-python/build.py     |  2 -
 lib/codegen/meta-python/gen_types.py | 64 ----------------------------
 lib/codegen/src/ir/types.rs          |  2 +-
 4 files changed, 2 insertions(+), 68 deletions(-)
 delete mode 100644 lib/codegen/meta-python/gen_types.py

diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index 92770e9b16..7cb20c4a14 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -80,7 +80,7 @@ fn main() {
     // Now that the Python build process is complete, generate files that are
     // emitted by the `meta` crate.
     // ------------------------------------------------------------------------
-    if let Err(err) = meta::gen_types::generate("new_types.rs", &out_dir) {
+    if let Err(err) = meta::gen_types::generate("types.rs", &out_dir) {
         eprintln!("Error: {}", err);
         process::exit(1);
     }
diff --git a/lib/codegen/meta-python/build.py b/lib/codegen/meta-python/build.py
index 9d3d43f95d..b88653822c 100644
--- a/lib/codegen/meta-python/build.py
+++ b/lib/codegen/meta-python/build.py
@@ -5,7 +5,6 @@
 from __future__ import absolute_import
 import argparse
 import isa
-import gen_types
 import gen_instr
 import gen_settings
 import gen_build_deps
@@ -26,7 +25,6 @@ def main():
 
     isas = isa.all_isas()
 
-    gen_types.generate(out_dir)
     gen_instr.generate(isas, out_dir)
     gen_settings.generate(isas, out_dir)
     gen_encoding.generate(isas, out_dir)
diff --git a/lib/codegen/meta-python/gen_types.py b/lib/codegen/meta-python/gen_types.py
deleted file mode 100644
index 4199efa51f..0000000000
--- a/lib/codegen/meta-python/gen_types.py
+++ /dev/null
@@ -1,64 +0,0 @@
-"""
-Generate sources with type info.
-
-This generates a `types.rs` file which is included in
-`lib/codegen/ir/types.rs`. The file provides constant definitions for the most
-commonly used types, including all of the scalar types.
-
-This ensures that Python and Rust use the same type numbering.
-"""
-from __future__ import absolute_import
-import srcgen
-from cdsl.types import ValueType
-import base.types  # noqa
-
-try:
-    from typing import Iterable  # noqa
-except ImportError:
-    pass
-
-
-def emit_type(ty, fmt):
-    # type: (ValueType, srcgen.Formatter) -> None
-    """
-    Emit a constant definition of a single value type.
-    """
-    name = ty.name.upper()
-    fmt.doc_comment(ty.__doc__)
-    fmt.line(
-            'pub const {}: Type = Type({:#x});'
-            .format(name, ty.number))
-    fmt.line()
-
-
-def emit_vectors(bits, fmt):
-    # type: (int, srcgen.Formatter) -> None
-    """
-    Emit definition for all vector types with `bits` total size.
-    """
-    size = bits // 8
-    for ty in ValueType.all_lane_types:
-        mb = ty.membytes
-        if mb == 0 or mb >= size:
-            continue
-        emit_type(ty.by(size // mb), fmt)
-
-
-def emit_types(fmt):
-    # type: (srcgen.Formatter) -> None
-    for spec in ValueType.all_special_types:
-        emit_type(spec, fmt)
-    for ty in ValueType.all_lane_types:
-        emit_type(ty, fmt)
-    # Emit vector definitions for common SIMD sizes.
-    emit_vectors(64, fmt)
-    emit_vectors(128, fmt)
-    emit_vectors(256, fmt)
-    emit_vectors(512, fmt)
-
-
-def generate(out_dir):
-    # type: (str) -> None
-    fmt = srcgen.Formatter()
-    emit_types(fmt)
-    fmt.update_file('types.rs', out_dir)
diff --git a/lib/codegen/src/ir/types.rs b/lib/codegen/src/ir/types.rs
index e1dcf0d177..c7f6d12f23 100644
--- a/lib/codegen/src/ir/types.rs
+++ b/lib/codegen/src/ir/types.rs
@@ -30,7 +30,7 @@ const LANE_BASE: u8 = 0x70;
 /// Start of the 2-lane vector types.
 const VECTOR_BASE: u8 = LANE_BASE + 16;
 
-// Include code generated by `lib/codegen/meta-python/gen_types.py`. This file contains constant
+// Include code generated by `lib/codegen/meta/gen_types.rs`. This file contains constant
 // definitions for all the scalar types as well as common vector types for 64, 128, 256, and
 // 512-bit SIMD vectors.
 include!(concat!(env!("OUT_DIR"), "/types.rs"));

From 4f2d7dd54f7c95624ed4b5bd73684df715dd5ef3 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 16 Oct 2018 17:00:01 +0200
Subject: [PATCH 2165/3084] [build] Move Isa enum to the meta library;

---
 lib/codegen/build.rs            | 55 +--------------------------------
 lib/codegen/meta/src/isa/mod.rs | 53 +++++++++++++++++++++++++++++++
 lib/codegen/meta/src/lib.rs     |  1 +
 3 files changed, 55 insertions(+), 54 deletions(-)
 create mode 100644 lib/codegen/meta/src/isa/mod.rs

diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index 7cb20c4a14..39f6ebc62a 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -20,6 +20,7 @@
 
 extern crate cranelift_codegen_meta as meta;
 
+use meta::isa::Isa;
 use std::env;
 use std::process;
 
@@ -99,60 +100,6 @@ fn identify_python() -> &'static str {
     panic!("The Cranelift build requires Python (version 2.7 or version 3)");
 }
 
-/// Represents known ISA target.
-#[derive(Copy, Clone)]
-enum Isa {
-    Riscv,
-    X86,
-    Arm32,
-    Arm64,
-}
-
-impl Isa {
-    /// Creates isa target using name.
-    fn new(name: &str) -> Option {
-        Isa::all()
-            .iter()
-            .cloned()
-            .filter(|isa| isa.name() == name)
-            .next()
-    }
-
-    /// Creates isa target from arch.
-    fn from_arch(arch: &str) -> Option {
-        Isa::all()
-            .iter()
-            .cloned()
-            .filter(|isa| isa.is_arch_applicable(arch))
-            .next()
-    }
-
-    /// Returns all supported isa targets.
-    fn all() -> [Isa; 4] {
-        [Isa::Riscv, Isa::X86, Isa::Arm32, Isa::Arm64]
-    }
-
-    /// Returns name of the isa target.
-    fn name(&self) -> &'static str {
-        match *self {
-            Isa::Riscv => "riscv",
-            Isa::X86 => "x86",
-            Isa::Arm32 => "arm32",
-            Isa::Arm64 => "arm64",
-        }
-    }
-
-    /// Checks if arch is applicable for the isa target.
-    fn is_arch_applicable(&self, arch: &str) -> bool {
-        match *self {
-            Isa::Riscv => arch == "riscv",
-            Isa::X86 => ["x86_64", "i386", "i586", "i686"].contains(&arch),
-            Isa::Arm32 => arch.starts_with("arm") || arch.starts_with("thumb"),
-            Isa::Arm64 => arch == "aarch64",
-        }
-    }
-}
-
 /// Returns isa targets to configure conditional compilation.
 fn isa_targets(cranelift_targets: Option<&str>, target_triple: &str) -> Result, String> {
     match cranelift_targets {
diff --git a/lib/codegen/meta/src/isa/mod.rs b/lib/codegen/meta/src/isa/mod.rs
new file mode 100644
index 0000000000..85141bb714
--- /dev/null
+++ b/lib/codegen/meta/src/isa/mod.rs
@@ -0,0 +1,53 @@
+/// Represents known ISA target.
+#[derive(Copy, Clone)]
+pub enum Isa {
+    Riscv,
+    X86,
+    Arm32,
+    Arm64,
+}
+
+impl Isa {
+    /// Creates isa target using name.
+    pub fn new(name: &str) -> Option {
+        Isa::all()
+            .iter()
+            .cloned()
+            .filter(|isa| isa.name() == name)
+            .next()
+    }
+
+    /// Creates isa target from arch.
+    pub fn from_arch(arch: &str) -> Option {
+        Isa::all()
+            .iter()
+            .cloned()
+            .filter(|isa| isa.is_arch_applicable(arch))
+            .next()
+    }
+
+    /// Returns all supported isa targets.
+    pub fn all() -> [Isa; 4] {
+        [Isa::Riscv, Isa::X86, Isa::Arm32, Isa::Arm64]
+    }
+
+    /// Returns name of the isa target.
+    pub fn name(&self) -> &'static str {
+        match *self {
+            Isa::Riscv => "riscv",
+            Isa::X86 => "x86",
+            Isa::Arm32 => "arm32",
+            Isa::Arm64 => "arm64",
+        }
+    }
+
+    /// Checks if arch is applicable for the isa target.
+    fn is_arch_applicable(&self, arch: &str) -> bool {
+        match *self {
+            Isa::Riscv => arch == "riscv",
+            Isa::X86 => ["x86_64", "i386", "i586", "i686"].contains(&arch),
+            Isa::Arm32 => arch.starts_with("arm") || arch.starts_with("thumb"),
+            Isa::Arm64 => arch == "aarch64",
+        }
+    }
+}
diff --git a/lib/codegen/meta/src/lib.rs b/lib/codegen/meta/src/lib.rs
index f7533e9709..5bf0646f35 100644
--- a/lib/codegen/meta/src/lib.rs
+++ b/lib/codegen/meta/src/lib.rs
@@ -1,5 +1,6 @@
 pub mod error;
 pub mod gen_types;
+pub mod isa;
 
 mod base;
 mod cdsl;

From b7f2acf0ea62228e76ae57596542e2d400ee3d6b Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 16 Oct 2018 16:42:40 +0200
Subject: [PATCH 2166/3084] [build] Implement registers code generation in the
 Rust meta crate;

---
 lib/codegen/build.rs                      |  26 ++-
 lib/codegen/meta-python/cdsl/registers.py |   3 +-
 lib/codegen/meta/Cargo.toml               |   3 +
 lib/codegen/meta/src/cdsl/isa.rs          | 142 +++++++++++++++
 lib/codegen/meta/src/cdsl/mod.rs          |   2 +
 lib/codegen/meta/src/cdsl/regs.rs         | 199 ++++++++++++++++++++++
 lib/codegen/meta/src/gen_registers.rs     | 140 +++++++++++++++
 lib/codegen/meta/src/isa/arm32/mod.rs     |  39 +++++
 lib/codegen/meta/src/isa/arm64/mod.rs     |  35 ++++
 lib/codegen/meta/src/isa/mod.rs           |  44 +++--
 lib/codegen/meta/src/isa/riscv/mod.rs     |  24 +++
 lib/codegen/meta/src/isa/x86/mod.rs       |  43 +++++
 lib/codegen/meta/src/lib.rs               |   4 +
 lib/codegen/meta/src/srcgen.rs            |  51 ++----
 lib/entity/src/primary.rs                 |  10 ++
 15 files changed, 714 insertions(+), 51 deletions(-)
 create mode 100644 lib/codegen/meta/src/cdsl/isa.rs
 create mode 100644 lib/codegen/meta/src/cdsl/regs.rs
 create mode 100644 lib/codegen/meta/src/gen_registers.rs
 create mode 100644 lib/codegen/meta/src/isa/arm32/mod.rs
 create mode 100644 lib/codegen/meta/src/isa/arm64/mod.rs
 create mode 100644 lib/codegen/meta/src/isa/riscv/mod.rs
 create mode 100644 lib/codegen/meta/src/isa/x86/mod.rs

diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index 39f6ebc62a..8eae5faf94 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -24,7 +24,11 @@ use meta::isa::Isa;
 use std::env;
 use std::process;
 
+use std::time::Instant;
+
 fn main() {
+    let start_time = Instant::now();
+
     let out_dir = env::var("OUT_DIR").expect("The OUT_DIR environment variable must be set");
     let target_triple = env::var("TARGET").expect("The TARGET environment variable must be set");
     let cranelift_targets = env::var("CRANELIFT_TARGETS").ok();
@@ -35,7 +39,7 @@ fn main() {
     match isa_targets(cranelift_targets, &target_triple) {
         Ok(isa_targets) => {
             for isa in &isa_targets {
-                println!("cargo:rustc-cfg=build_{}", isa.name());
+                println!("cargo:rustc-cfg=build_{}", isa.to_string());
             }
         }
         Err(err) => {
@@ -44,8 +48,6 @@ fn main() {
         }
     }
 
-    println!("Build script generating files in {}", out_dir);
-
     let cur_dir = env::current_dir().expect("Can't access current working directory");
     let crate_dir = cur_dir.as_path();
 
@@ -81,10 +83,28 @@ fn main() {
     // Now that the Python build process is complete, generate files that are
     // emitted by the `meta` crate.
     // ------------------------------------------------------------------------
+    let isas = meta::isa::define_all();
+
     if let Err(err) = meta::gen_types::generate("types.rs", &out_dir) {
         eprintln!("Error: {}", err);
         process::exit(1);
     }
+
+    for isa in isas {
+        if let Err(err) = meta::gen_registers::generate(isa, "new_registers", &out_dir) {
+            eprintln!("Error: {}", err);
+            process::exit(1);
+        }
+    }
+
+    println!(
+        "cargo:warning=Cranelift meta-build step took {:?}",
+        Instant::now() - start_time
+    );
+    println!(
+        "cargo:warning=Meta-build script generated files in {}",
+        out_dir
+    );
 }
 
 fn identify_python() -> &'static str {
diff --git a/lib/codegen/meta-python/cdsl/registers.py b/lib/codegen/meta-python/cdsl/registers.py
index 0c96583db0..1e4ffe75b1 100644
--- a/lib/codegen/meta-python/cdsl/registers.py
+++ b/lib/codegen/meta-python/cdsl/registers.py
@@ -87,7 +87,6 @@ class RegBank(object):
         self.names = names
         self.classes = list()  # type: List[RegClass]
         self.toprcs = list()  # type: List[RegClass]
-        self.first_toprc_index = None  # type: int
 
         assert len(names) <= units
 
@@ -248,7 +247,7 @@ class RegClass(object):
     def intersect(self, other):
         # type: (RegClass) -> RCTup
         """
-        Get a tuple representing the intersction of two register classes.
+        Get a tuple representing the intersection of two register classes.
 
         Returns `None` if the two classes are disjoint.
         """
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 78eed62419..98c22d24c1 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -7,6 +7,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
+[dependencies]
+cranelift-entity = { path = "../../entity", version = "0.22.0", default-features = false }
+
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/codegen/meta/src/cdsl/isa.rs b/lib/codegen/meta/src/cdsl/isa.rs
new file mode 100644
index 0000000000..4a6a819683
--- /dev/null
+++ b/lib/codegen/meta/src/cdsl/isa.rs
@@ -0,0 +1,142 @@
+use cranelift_entity::PrimaryMap;
+
+use super::regs::{
+    RegBank, RegBankBuilder, RegBankIndex, RegClass, RegClassBuilder, RegClassIndex,
+};
+
+pub struct TargetIsa {
+    pub name: &'static str,
+    pub reg_banks: PrimaryMap,
+    pub reg_classes: PrimaryMap,
+}
+
+impl TargetIsa {
+    pub fn new(name: &'static str) -> Self {
+        Self {
+            name,
+            reg_banks: PrimaryMap::new(),
+            reg_classes: PrimaryMap::new(),
+        }
+    }
+
+    pub fn add_reg_bank(&mut self, builder: RegBankBuilder) -> RegBankIndex {
+        let first_unit = if self.reg_banks.len() == 0 {
+            0
+        } else {
+            let last = &self.reg_banks.last().unwrap();
+            let first_available_unit = (last.first_unit + last.units) as i8;
+            let units = builder.units;
+            let align = if units.is_power_of_two() {
+                units
+            } else {
+                units.next_power_of_two()
+            } as i8;
+            (first_available_unit + align - 1) & -align
+        } as u8;
+
+        self.reg_banks.push(RegBank::new(
+            builder.name,
+            first_unit,
+            builder.units,
+            builder.names,
+            builder.prefix,
+            builder
+                .pressure_tracking
+                .expect("Pressure tracking must be explicitly set"),
+        ))
+    }
+
+    pub fn add_reg_class(&mut self, builder: RegClassBuilder) -> RegClassIndex {
+        let reg_bank_units = self.reg_banks.get(builder.bank).unwrap().units;
+
+        let start = builder.start;
+        assert!(start < reg_bank_units);
+
+        let count = if builder.count != 0 {
+            builder.count
+        } else {
+            reg_bank_units / builder.width
+        };
+
+        let reg_class_index = builder.index;
+        assert!(
+            self.reg_classes.next_key() == reg_class_index,
+            "should have inserted RegClass where expected"
+        );
+
+        let reg_class = RegClass::new(
+            builder.name,
+            reg_class_index,
+            builder.width,
+            builder.bank,
+            builder.toprc,
+            count,
+            start,
+        );
+        self.reg_classes.push(reg_class);
+
+        let reg_bank = self.reg_banks.get_mut(builder.bank).unwrap();
+        reg_bank.classes.push(reg_class_index);
+
+        reg_class_index
+    }
+
+    /// Checks that the set of register classes satisfies:
+    ///
+    /// 1. Closed under intersection: The intersection of any two register
+    ///    classes in the set is either empty or identical to a member of the
+    ///    set.
+    /// 2. There are no identical classes under different names.
+    /// 3. Classes are sorted topologically such that all subclasses have a
+    ///    higher index that the superclass.
+    pub fn check(&self) {
+        for reg_bank in self.reg_banks.values() {
+            for i1 in reg_bank.classes.iter() {
+                for i2 in reg_bank.classes.iter() {
+                    if i1 >= i2 {
+                        continue;
+                    }
+
+                    let rc1 = self.reg_classes.get(*i1).unwrap();
+                    let rc2 = self.reg_classes.get(*i2).unwrap();
+
+                    let rc1_mask = rc1.mask(0);
+                    let rc2_mask = rc2.mask(0);
+
+                    assert!(
+                        rc1.width != rc2.width || rc1_mask != rc2_mask,
+                        "no duplicates"
+                    );
+                    if rc1.width != rc2.width {
+                        continue;
+                    }
+
+                    let mut intersect = Vec::new();
+                    for (a, b) in rc1_mask.iter().zip(rc2_mask.iter()) {
+                        intersect.push(a & b);
+                    }
+                    if intersect == vec![0; intersect.len()] {
+                        continue;
+                    }
+
+                    // Classes must be topologically ordered, so the intersection can't be the
+                    // superclass.
+                    assert!(intersect != rc1_mask);
+
+                    // If the intersection is the second one, then it must be a subclass.
+                    if intersect == rc2_mask {
+                        assert!(
+                            self.reg_classes
+                                .get(*i1)
+                                .unwrap()
+                                .subclasses
+                                .iter()
+                                .find(|x| **x == *i2)
+                                .is_some()
+                        );
+                    }
+                }
+            }
+        }
+    }
+}
diff --git a/lib/codegen/meta/src/cdsl/mod.rs b/lib/codegen/meta/src/cdsl/mod.rs
index 6a004b2220..4d29421e0b 100644
--- a/lib/codegen/meta/src/cdsl/mod.rs
+++ b/lib/codegen/meta/src/cdsl/mod.rs
@@ -3,6 +3,8 @@
 //! This module defines the classes that are used to define Cranelift
 //! instructions and other entities.
 
+pub mod isa;
+pub mod regs;
 pub mod types;
 
 /// Convert the string `s` to CamelCase.
diff --git a/lib/codegen/meta/src/cdsl/regs.rs b/lib/codegen/meta/src/cdsl/regs.rs
new file mode 100644
index 0000000000..bc40ffaf69
--- /dev/null
+++ b/lib/codegen/meta/src/cdsl/regs.rs
@@ -0,0 +1,199 @@
+use cranelift_entity::EntityRef;
+
+use super::isa::TargetIsa;
+
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct RegBankIndex(u32);
+entity_impl!(RegBankIndex);
+
+pub struct RegBank {
+    pub name: &'static str,
+    pub first_unit: u8,
+    pub units: u8,
+    pub names: Vec<&'static str>,
+    pub prefix: &'static str,
+    pub pressure_tracking: bool,
+    pub toprcs: Vec,
+    pub classes: Vec,
+}
+
+impl RegBank {
+    pub fn new(
+        name: &'static str,
+        first_unit: u8,
+        units: u8,
+        names: Vec<&'static str>,
+        prefix: &'static str,
+        pressure_tracking: bool,
+    ) -> Self {
+        RegBank {
+            name,
+            first_unit,
+            units,
+            names,
+            prefix,
+            pressure_tracking,
+            toprcs: Vec::new(),
+            classes: Vec::new(),
+        }
+    }
+}
+
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct RegClassIndex(u32);
+entity_impl!(RegClassIndex);
+
+pub struct RegClass {
+    pub name: &'static str,
+    pub index: RegClassIndex,
+    pub width: u8,
+    pub bank: RegBankIndex,
+    pub toprc: RegClassIndex,
+    pub count: u8,
+    pub start: u8,
+    pub subclasses: Vec,
+}
+
+impl RegClass {
+    pub fn new(
+        name: &'static str,
+        index: RegClassIndex,
+        width: u8,
+        bank: RegBankIndex,
+        toprc: RegClassIndex,
+        count: u8,
+        start: u8,
+    ) -> Self {
+        Self {
+            name,
+            index,
+            width,
+            bank,
+            toprc,
+            count,
+            start,
+            subclasses: Vec::new(),
+        }
+    }
+
+    /// Compute a bit-mask of subclasses, including self.
+    pub fn subclass_mask(&self) -> u64 {
+        let mut m = 1 << self.index.index();
+        for rc in self.subclasses.iter() {
+            m |= 1 << rc.index();
+        }
+        m
+    }
+
+    /// Compute a bit-mask of the register units allocated by this register class.
+    pub fn mask(&self, bank_first_unit: u8) -> Vec {
+        let mut u = (self.start + bank_first_unit) as usize;
+        let mut out_mask = vec![0, 0, 0];
+        for _ in 0..self.count {
+            out_mask[u / 32] |= 1 << (u % 32);
+            u += self.width as usize;
+        }
+        out_mask
+    }
+}
+
+pub struct RegClassBuilder {
+    pub name: &'static str,
+    pub index: RegClassIndex,
+    pub width: u8,
+    pub bank: RegBankIndex,
+    pub toprc: RegClassIndex,
+    pub count: u8,
+    pub start: u8,
+}
+
+impl RegClassBuilder {
+    pub fn new_toplevel(isa: &mut TargetIsa, name: &'static str, bank: RegBankIndex) -> Self {
+        let index = isa.reg_classes.next_key();
+
+        // Add it to the top-level register classes of the register bank.
+        isa.reg_banks.get_mut(bank).unwrap().toprcs.push(index);
+
+        Self {
+            name,
+            index,
+            width: 1,
+            bank,
+            toprc: index,
+            count: 0,
+            start: 0,
+        }
+    }
+
+    pub fn subclass_of(
+        isa: &mut TargetIsa,
+        name: &'static str,
+        parent_index: RegClassIndex,
+        start: u8,
+        stop: u8,
+    ) -> Self {
+        assert!(stop >= start);
+
+        let index = isa.reg_classes.next_key();
+
+        let toprc = isa.reg_classes.get(parent_index).unwrap().toprc;
+        for reg_class in isa.reg_classes.values_mut() {
+            if reg_class.toprc == toprc {
+                reg_class.subclasses.push(index);
+            }
+        }
+
+        let parent = &isa.reg_classes.get(parent_index).unwrap();
+        Self {
+            name,
+            count: stop - start,
+            width: parent.width,
+            start: parent.start + start * parent.width,
+            bank: parent.bank,
+            toprc: parent.toprc,
+            index,
+        }
+    }
+
+    pub fn count(mut self, count: u8) -> Self {
+        self.count = count;
+        self
+    }
+
+    pub fn width(mut self, width: u8) -> Self {
+        self.width = width;
+        self
+    }
+}
+
+pub struct RegBankBuilder {
+    pub name: &'static str,
+    pub units: u8,
+    pub names: Vec<&'static str>,
+    pub prefix: &'static str,
+    pub pressure_tracking: Option,
+}
+
+impl RegBankBuilder {
+    pub fn new(name: &'static str, prefix: &'static str) -> Self {
+        Self {
+            name,
+            units: 0,
+            names: vec![],
+            prefix,
+            pressure_tracking: None,
+        }
+    }
+    pub fn units(mut self, units: u8) -> Self {
+        self.units = units;
+        self
+    }
+    pub fn names(mut self, names: Vec<&'static str>) -> Self {
+        self.names = names;
+        self
+    }
+    pub fn track_pressure(mut self, track: bool) -> Self {
+        self.pressure_tracking = Some(track);
+        self
+    }
+}
diff --git a/lib/codegen/meta/src/gen_registers.rs b/lib/codegen/meta/src/gen_registers.rs
new file mode 100644
index 0000000000..637a3ea3fa
--- /dev/null
+++ b/lib/codegen/meta/src/gen_registers.rs
@@ -0,0 +1,140 @@
+use cdsl::isa::TargetIsa;
+use cdsl::regs::{RegBank, RegClass};
+use cranelift_entity::EntityRef;
+use error;
+use srcgen::Formatter;
+
+fn gen_regbank(fmt: &mut Formatter, reg_bank: &RegBank) {
+    let names = if reg_bank.names.len() > 0 {
+        format!(r#""{}""#, reg_bank.names.join(r#"", ""#))
+    } else {
+        "".to_string()
+    };
+    fmt.line("RegBank {");
+    fmt.indent(|fmt| {
+        fmt.line(&format!(r#"name: "{}","#, reg_bank.name));
+        fmt.line(&format!("first_unit: {},", reg_bank.first_unit));
+        fmt.line(&format!("units: {},", reg_bank.units));
+        fmt.line(&format!("names: &[{}],", names));
+        fmt.line(&format!(r#"prefix: "{}","#, reg_bank.prefix));
+        fmt.line(&format!("first_toprc: {},", reg_bank.toprcs[0].index()));
+        fmt.line(&format!("num_toprcs: {},", reg_bank.toprcs.len()));
+        fmt.line(&format!(
+            "pressure_tracking: {},",
+            if reg_bank.pressure_tracking {
+                "true"
+            } else {
+                "false"
+            }
+        ));
+    });
+    fmt.line("},");
+}
+
+fn gen_regclass(isa: &TargetIsa, reg_class: &RegClass, fmt: &mut Formatter) {
+    let reg_bank = isa.reg_banks.get(reg_class.bank).unwrap();
+
+    let mask: Vec = reg_class
+        .mask(reg_bank.first_unit)
+        .iter()
+        .map(|x| format!("0x{:08x}", x))
+        .collect();
+    let mask = mask.join(", ");
+
+    fmt.line(&format!(
+        "pub static {}_DATA: RegClassData = RegClassData {{",
+        reg_class.name
+    ));
+    fmt.indent(|fmt| {
+        fmt.line(&format!(r#"name: "{}","#, reg_class.name));
+        fmt.line(&format!("index: {},", reg_class.index.index()));
+        fmt.line(&format!("width: {},", reg_class.width));
+        fmt.line(&format!("bank: {},", reg_class.bank.index()));
+        fmt.line(&format!("toprc: {},", reg_class.toprc.index()));
+        fmt.line(&format!(
+            "first: {},",
+            reg_bank.first_unit + reg_class.start
+        ));
+        fmt.line(&format!("subclasses: {:#x},", reg_class.subclass_mask()));
+        fmt.line(&format!("mask: [{}],", mask));
+        fmt.line("info: &INFO,");
+    });
+    fmt.line("};");
+    fmt.line("#[allow(dead_code)]");
+    fmt.line(&format!(
+        "pub static {}: RegClass = &{}_DATA;",
+        reg_class.name, reg_class.name
+    ));
+}
+
+fn gen_regbank_units(reg_bank: &RegBank, fmt: &mut Formatter) {
+    for unit in 0..reg_bank.units {
+        let v = unit + reg_bank.first_unit;
+        if (unit as usize) < reg_bank.names.len() {
+            fmt.line(&format!("{} = {},", reg_bank.names[unit as usize], v));
+            continue;
+        }
+        fmt.line(&format!("{}{} = {},", reg_bank.prefix, unit, v));
+    }
+}
+
+fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) -> Result<(), error::Error> {
+    // Emit RegInfo.
+    fmt.line("pub static INFO: RegInfo = RegInfo {");
+
+    fmt.indent(|fmt| {
+        fmt.line("banks: &[");
+        // Bank descriptors.
+        fmt.indent(|fmt| {
+            for reg_bank in isa.reg_banks.values() {
+                gen_regbank(fmt, ®_bank);
+            }
+        });
+        fmt.line("],");
+        // References to register classes.
+        fmt.line("classes: &[");
+        fmt.indent(|fmt| {
+            for reg_class in isa.reg_classes.values() {
+                fmt.line(&format!("&{}_DATA,", reg_class.name));
+            }
+        });
+        fmt.line("],");
+    });
+    fmt.line("};");
+
+    // Register class descriptors.
+    for rc in isa.reg_classes.values() {
+        gen_regclass(&isa, rc, fmt);
+    }
+
+    // Emit constants for all the register units.
+    fmt.line("#[allow(dead_code, non_camel_case_types)]");
+    fmt.line("#[derive(Clone, Copy)]");
+    fmt.line("pub enum RU {");
+    fmt.indent(|fmt| {
+        for reg_bank in isa.reg_banks.values() {
+            gen_regbank_units(reg_bank, fmt);
+        }
+    });
+    fmt.line("}");
+
+    // Emit Into conversion for the RU class.
+    fmt.line("impl Into for RU {");
+    fmt.indent(|fmt| {
+        fmt.line("fn into(self) -> RegUnit {");
+        fmt.indent(|fmt| {
+            fmt.line("self as RegUnit");
+        });
+        fmt.line("}")
+    });
+    fmt.line("}");
+
+    Ok(())
+}
+
+pub fn generate(isa: TargetIsa, base_filename: &str, out_dir: &str) -> Result<(), error::Error> {
+    let mut fmt = Formatter::new();
+    gen_isa(&isa, &mut fmt)?;
+    fmt.update_file(&format!("{}-{}.rs", base_filename, isa.name), out_dir)?;
+    Ok(())
+}
diff --git a/lib/codegen/meta/src/isa/arm32/mod.rs b/lib/codegen/meta/src/isa/arm32/mod.rs
new file mode 100644
index 0000000000..ce33422276
--- /dev/null
+++ b/lib/codegen/meta/src/isa/arm32/mod.rs
@@ -0,0 +1,39 @@
+use cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use isa;
+
+pub fn define() -> isa::TargetIsa {
+    let mut isa = isa::TargetIsa::new("arm32");
+
+    let builder = RegBankBuilder::new("FloatRegs", "s")
+        .units(64)
+        .track_pressure(true);
+    let float_regs = isa.add_reg_bank(builder);
+
+    let builder = RegBankBuilder::new("IntRegs", "r")
+        .units(16)
+        .track_pressure(true);
+    let int_regs = isa.add_reg_bank(builder);
+
+    let builder = RegBankBuilder::new("FlagRegs", "")
+        .units(1)
+        .names(vec!["nzcv"])
+        .track_pressure(false);
+    let flag_reg = isa.add_reg_bank(builder);
+
+    let builder = RegClassBuilder::new_toplevel(&mut isa, "S", float_regs).count(32);
+    isa.add_reg_class(builder);
+
+    let builder = RegClassBuilder::new_toplevel(&mut isa, "D", float_regs).width(2);
+    isa.add_reg_class(builder);
+
+    let builder = RegClassBuilder::new_toplevel(&mut isa, "Q", float_regs).width(4);
+    isa.add_reg_class(builder);
+
+    let builder = RegClassBuilder::new_toplevel(&mut isa, "GPR", int_regs);
+    isa.add_reg_class(builder);
+
+    let builder = RegClassBuilder::new_toplevel(&mut isa, "FLAG", flag_reg);
+    isa.add_reg_class(builder);
+
+    isa
+}
diff --git a/lib/codegen/meta/src/isa/arm64/mod.rs b/lib/codegen/meta/src/isa/arm64/mod.rs
new file mode 100644
index 0000000000..8e13d28321
--- /dev/null
+++ b/lib/codegen/meta/src/isa/arm64/mod.rs
@@ -0,0 +1,35 @@
+use cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use isa;
+
+pub fn define() -> isa::TargetIsa {
+    let mut isa = isa::TargetIsa::new("arm64");
+
+    // The `x31` regunit serves as the stack pointer / zero register depending on context. We
+    // reserve it and don't model the difference.
+    let builder = RegBankBuilder::new("IntRegs", "x")
+        .units(32)
+        .track_pressure(true);
+    let int_regs = isa.add_reg_bank(builder);
+
+    let builder = RegBankBuilder::new("FloatRegs", "v")
+        .units(32)
+        .track_pressure(true);
+    let float_regs = isa.add_reg_bank(builder);
+
+    let builder = RegBankBuilder::new("FlagRegs", "")
+        .units(1)
+        .names(vec!["nzcv"])
+        .track_pressure(false);
+    let flag_reg = isa.add_reg_bank(builder);
+
+    let builder = RegClassBuilder::new_toplevel(&mut isa, "GPR", int_regs);
+    isa.add_reg_class(builder);
+
+    let builder = RegClassBuilder::new_toplevel(&mut isa, "FPR", float_regs);
+    isa.add_reg_class(builder);
+
+    let builder = RegClassBuilder::new_toplevel(&mut isa, "FLAG", flag_reg);
+    isa.add_reg_class(builder);
+
+    isa
+}
diff --git a/lib/codegen/meta/src/isa/mod.rs b/lib/codegen/meta/src/isa/mod.rs
index 85141bb714..ffb13fb151 100644
--- a/lib/codegen/meta/src/isa/mod.rs
+++ b/lib/codegen/meta/src/isa/mod.rs
@@ -1,3 +1,11 @@
+use cdsl::isa::TargetIsa;
+use std::fmt;
+
+mod arm32;
+mod arm64;
+mod riscv;
+mod x86;
+
 /// Represents known ISA target.
 #[derive(Copy, Clone)]
 pub enum Isa {
@@ -13,7 +21,7 @@ impl Isa {
         Isa::all()
             .iter()
             .cloned()
-            .filter(|isa| isa.name() == name)
+            .filter(|isa| isa.to_string() == name)
             .next()
     }
 
@@ -31,16 +39,6 @@ impl Isa {
         [Isa::Riscv, Isa::X86, Isa::Arm32, Isa::Arm64]
     }
 
-    /// Returns name of the isa target.
-    pub fn name(&self) -> &'static str {
-        match *self {
-            Isa::Riscv => "riscv",
-            Isa::X86 => "x86",
-            Isa::Arm32 => "arm32",
-            Isa::Arm64 => "arm64",
-        }
-    }
-
     /// Checks if arch is applicable for the isa target.
     fn is_arch_applicable(&self, arch: &str) -> bool {
         match *self {
@@ -51,3 +49,27 @@ impl Isa {
         }
     }
 }
+
+impl fmt::Display for Isa {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        match *self {
+            Isa::Riscv => write!(f, "riscv"),
+            Isa::X86 => write!(f, "x86"),
+            Isa::Arm32 => write!(f, "arm32"),
+            Isa::Arm64 => write!(f, "arm64"),
+        }
+    }
+}
+
+pub fn define_all() -> Vec {
+    let isas = vec![
+        riscv::define(),
+        arm32::define(),
+        arm64::define(),
+        x86::define(),
+    ];
+    for isa in isas.iter() {
+        isa.check();
+    }
+    isas
+}
diff --git a/lib/codegen/meta/src/isa/riscv/mod.rs b/lib/codegen/meta/src/isa/riscv/mod.rs
new file mode 100644
index 0000000000..315ee42233
--- /dev/null
+++ b/lib/codegen/meta/src/isa/riscv/mod.rs
@@ -0,0 +1,24 @@
+use cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use isa;
+
+pub fn define() -> isa::TargetIsa {
+    let mut isa = isa::TargetIsa::new("riscv");
+
+    let builder = RegBankBuilder::new("IntRegs", "x")
+        .units(32)
+        .track_pressure(true);
+    let int_regs = isa.add_reg_bank(builder);
+
+    let builder = RegBankBuilder::new("FloatRegs", "f")
+        .units(32)
+        .track_pressure(true);
+    let float_regs = isa.add_reg_bank(builder);
+
+    let builder = RegClassBuilder::new_toplevel(&mut isa, "GPR", int_regs);
+    isa.add_reg_class(builder);
+
+    let builder = RegClassBuilder::new_toplevel(&mut isa, "FPR", float_regs);
+    isa.add_reg_class(builder);
+
+    isa
+}
diff --git a/lib/codegen/meta/src/isa/x86/mod.rs b/lib/codegen/meta/src/isa/x86/mod.rs
new file mode 100644
index 0000000000..43fd4ed67d
--- /dev/null
+++ b/lib/codegen/meta/src/isa/x86/mod.rs
@@ -0,0 +1,43 @@
+use cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use isa;
+
+pub fn define() -> isa::TargetIsa {
+    let mut isa = isa::TargetIsa::new("x86");
+
+    let builder = RegBankBuilder::new("IntRegs", "r")
+        .units(16)
+        .names(vec!["rax", "rcx", "rdx", "rbx", "rsp", "rbp", "rsi", "rdi"])
+        .track_pressure(true);
+    let int_regs = isa.add_reg_bank(builder);
+
+    let builder = RegBankBuilder::new("FloatRegs", "xmm")
+        .units(16)
+        .track_pressure(true);
+    let float_regs = isa.add_reg_bank(builder);
+
+    let builder = RegBankBuilder::new("FlagRegs", "")
+        .units(1)
+        .names(vec!["rflags"])
+        .track_pressure(false);
+    let flag_reg = isa.add_reg_bank(builder);
+
+    let builder = RegClassBuilder::new_toplevel(&mut isa, "GPR", int_regs);
+    let gpr = isa.add_reg_class(builder);
+
+    let builder = RegClassBuilder::new_toplevel(&mut isa, "FPR", float_regs);
+    let fpr = isa.add_reg_class(builder);
+
+    let builder = RegClassBuilder::new_toplevel(&mut isa, "FLAG", flag_reg);
+    isa.add_reg_class(builder);
+
+    let builder = RegClassBuilder::subclass_of(&mut isa, "GPR8", gpr, 0, 8);
+    let gpr8 = isa.add_reg_class(builder);
+
+    let builder = RegClassBuilder::subclass_of(&mut isa, "ABCD", gpr8, 0, 4);
+    isa.add_reg_class(builder);
+
+    let builder = RegClassBuilder::subclass_of(&mut isa, "FPR8", fpr, 0, 8);
+    isa.add_reg_class(builder);
+
+    isa
+}
diff --git a/lib/codegen/meta/src/lib.rs b/lib/codegen/meta/src/lib.rs
index 5bf0646f35..233138e35d 100644
--- a/lib/codegen/meta/src/lib.rs
+++ b/lib/codegen/meta/src/lib.rs
@@ -1,4 +1,8 @@
+#[macro_use]
+extern crate cranelift_entity;
+
 pub mod error;
+pub mod gen_registers;
 pub mod gen_types;
 pub mod isa;
 
diff --git a/lib/codegen/meta/src/srcgen.rs b/lib/codegen/meta/src/srcgen.rs
index 2e2babe512..bd404fbc06 100644
--- a/lib/codegen/meta/src/srcgen.rs
+++ b/lib/codegen/meta/src/srcgen.rs
@@ -12,24 +12,6 @@ use error;
 
 static SHIFTWIDTH: usize = 4;
 
-struct _IndentedScope {
-    fmt: Formatter,
-    after: Option,
-}
-
-impl _IndentedScope {
-    fn _enter(&mut self) {
-        self.fmt._indent_push();
-    }
-
-    fn _exit(&mut self) {
-        self.fmt._indent_pop();
-        if let Some(ref s) = self.after {
-            self.fmt.line(&s);
-        }
-    }
-}
-
 pub struct Formatter {
     indent: usize,
     lines: Vec,
@@ -46,16 +28,23 @@ impl Formatter {
     }
 
     /// Increase current indentation level by one.
-    pub fn _indent_push(&mut self) {
+    pub fn indent_push(&mut self) {
         self.indent += 1;
     }
 
     /// Decrease indentation by one level.
-    pub fn _indent_pop(&mut self) {
+    pub fn indent_pop(&mut self) {
         assert!(self.indent > 0, "Already at top level indentation");
         self.indent -= 1;
     }
 
+    pub fn indent T>(&mut self, f: F) -> T {
+        self.indent_push();
+        let ret = f(self);
+        self.indent_pop();
+        ret
+    }
+
     /// Get the current whitespace indentation in the form of a String.
     fn get_indent(&self) -> String {
         if self.indent == 0 {
@@ -68,9 +57,9 @@ impl Formatter {
     /// Get a string containing whitespace outdented one level. Used for
     /// lines of code that are inside a single indented block.
     fn _get_outdent(&mut self) -> String {
-        self._indent_push();
+        self.indent_push();
         let s = self.get_indent();
-        self._indent_pop();
+        self.indent_pop();
         s
     }
 
@@ -103,13 +92,6 @@ impl Formatter {
         Ok(())
     }
 
-    /// Return a scope object for use with a `with` statement.
-    /// The optional `before` and `after` parameters are surrounding lines
-    /// which are *not* indented.
-    fn _indented(&self, _before: Option<&str>, _after: Option<&str>) -> _IndentedScope {
-        unimplemented!();
-    }
-
     /// Add one or more lines after stripping common indentation.
     pub fn _multi_line(&mut self, s: &str) {
         parse_multiline(s).into_iter().for_each(|l| self.line(&l));
@@ -158,7 +140,6 @@ fn parse_multiline(s: &str) -> Vec {
         .iter()
         .skip(1)
         .map(|l| l.len() - l.trim_left().len())
-        .filter(|&i| i > 0)
         .min();
 
     // Strip off leading blank lines.
@@ -257,9 +238,9 @@ mod srcgen_tests {
     fn formatter_basic_example_works() {
         let mut fmt = Formatter::new();
         fmt.line("Hello line 1");
-        fmt._indent_push();
+        fmt.indent_push();
         fmt._comment("Nested comment");
-        fmt._indent_pop();
+        fmt.indent_pop();
         fmt.line("Back home again");
         let expected_lines = vec![
             "Hello line 1\n",
@@ -277,9 +258,9 @@ mod srcgen_tests {
         let actual_results = Vec::with_capacity(4);
         (0..3).for_each(|_| {
             fmt.get_indent();
-            fmt._indent_push();
+            fmt.indent_push();
         });
-        (0..3).for_each(|_| fmt._indent_pop());
+        (0..3).for_each(|_| fmt.indent_pop());
         fmt.get_indent();
 
         actual_results
@@ -300,7 +281,7 @@ mod srcgen_tests {
     fn fmt_can_add_indented_line() {
         let mut fmt = Formatter::new();
         fmt.line("hello");
-        fmt._indent_push();
+        fmt.indent_push();
         fmt.line("world");
         let expected_lines = vec!["hello\n", "    world\n"];
         assert_eq!(fmt.lines, expected_lines);
diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs
index 0902621c22..e23e8e2a6b 100644
--- a/lib/entity/src/primary.rs
+++ b/lib/entity/src/primary.rs
@@ -50,6 +50,11 @@ where
         self.elems.get(k.index())
     }
 
+    /// Get the element at `k` if it exists, mutable version.
+    pub fn get_mut(&mut self, k: K) -> Option<&mut V> {
+        self.elems.get_mut(k.index())
+    }
+
     /// Is this map completely empty?
     pub fn is_empty(&self) -> bool {
         self.elems.is_empty()
@@ -101,6 +106,11 @@ where
         self.elems.push(v);
         k
     }
+
+    /// Returns the last element that was inserted in the map.
+    pub fn last(&self) -> Option<&V> {
+        self.elems.last()
+    }
 }
 
 /// Immutable indexing into an `PrimaryMap`.

From bcbb2d01ccf3c38c25d59c363c7dd4cb6f9a02e7 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 30 Oct 2018 16:43:07 +0100
Subject: [PATCH 2167/3084] [build] Stop using Python code to generate the
 register files;

---
 lib/codegen/build.rs                     |   2 +-
 lib/codegen/meta-python/build.py         |   2 -
 lib/codegen/meta-python/gen_registers.py | 109 -----------------------
 3 files changed, 1 insertion(+), 112 deletions(-)
 delete mode 100644 lib/codegen/meta-python/gen_registers.py

diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index 8eae5faf94..f5dc654273 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -91,7 +91,7 @@ fn main() {
     }
 
     for isa in isas {
-        if let Err(err) = meta::gen_registers::generate(isa, "new_registers", &out_dir) {
+        if let Err(err) = meta::gen_registers::generate(isa, "registers", &out_dir) {
             eprintln!("Error: {}", err);
             process::exit(1);
         }
diff --git a/lib/codegen/meta-python/build.py b/lib/codegen/meta-python/build.py
index b88653822c..60fe124e21 100644
--- a/lib/codegen/meta-python/build.py
+++ b/lib/codegen/meta-python/build.py
@@ -10,7 +10,6 @@ import gen_settings
 import gen_build_deps
 import gen_encoding
 import gen_legalizer
-import gen_registers
 import gen_binemit
 
 
@@ -29,7 +28,6 @@ def main():
     gen_settings.generate(isas, out_dir)
     gen_encoding.generate(isas, out_dir)
     gen_legalizer.generate(isas, out_dir)
-    gen_registers.generate(isas, out_dir)
     gen_binemit.generate(isas, out_dir)
     gen_build_deps.generate()
 
diff --git a/lib/codegen/meta-python/gen_registers.py b/lib/codegen/meta-python/gen_registers.py
deleted file mode 100644
index 166bc611da..0000000000
--- a/lib/codegen/meta-python/gen_registers.py
+++ /dev/null
@@ -1,109 +0,0 @@
-"""
-Generate register bank descriptions for each ISA.
-"""
-
-from __future__ import absolute_import
-import srcgen
-
-try:
-    from typing import Sequence, List  # noqa
-    from cdsl.isa import TargetISA  # noqa
-    from cdsl.registers import RegBank, RegClass  # noqa
-except ImportError:
-    pass
-
-
-def gen_regbank(regbank, fmt):
-    # type: (RegBank, srcgen.Formatter) -> None
-    """
-    Emit a static data definition for regbank.
-    """
-    with fmt.indented('RegBank {', '},'):
-        fmt.format('name: "{}",', regbank.name)
-        fmt.format('first_unit: {},', regbank.first_unit)
-        fmt.format('units: {},', regbank.units)
-        fmt.format(
-                'names: &[{}],',
-                ', '.join('"{}"'.format(n) for n in regbank.names))
-        fmt.format('prefix: "{}",', regbank.prefix)
-        fmt.format('first_toprc: {},', regbank.toprcs[0].index)
-        fmt.format('num_toprcs: {},', len(regbank.toprcs))
-        fmt.format(
-                'pressure_tracking: {},',
-                'true' if regbank.pressure_tracking else 'false')
-
-
-def gen_regbank_units(regbank, fmt):
-    # type: (RegBank, srcgen.Formatter) -> None
-    """
-    Emit constants for all the register units in `regbank`.
-    """
-    for unit in range(regbank.units):
-        v = unit + regbank.first_unit
-        if unit < len(regbank.names):
-            fmt.format("{} = {},", regbank.names[unit], v)
-        else:
-            fmt.format("{}{} = {},", regbank.prefix, unit, v)
-
-
-def gen_regclass(rc, fmt):
-    # type: (RegClass, srcgen.Formatter) -> None
-    """
-    Emit a static data definition for a register class.
-    """
-    with fmt.indented(
-            'pub static {}_DATA: RegClassData = RegClassData {{'
-            .format(rc.name), '};'):
-        fmt.format('name: "{}",', rc.name)
-        fmt.format('index: {},', rc.index)
-        fmt.format('width: {},', rc.width)
-        fmt.format('bank: {},', rc.bank.index)
-        fmt.format('toprc: {},', rc.toprc.index)
-        fmt.format('first: {},', rc.bank.first_unit + rc.start())
-        fmt.format('subclasses: 0x{:x},', rc.subclass_mask())
-        mask = ', '.join('0x{:08x}'.format(x) for x in rc.mask())
-        fmt.format('mask: [{}],', mask)
-        fmt.line('info: &INFO,')
-    # Also emit a convenient reference for use by hand-written code.
-    fmt.line('#[allow(dead_code)]')
-    fmt.format('pub static {0}: RegClass = &{0}_DATA;', rc.name)
-
-
-def gen_isa(isa, fmt):
-    # type: (TargetISA, srcgen.Formatter) -> None
-    """
-    Generate register tables for isa.
-    """
-    if not isa.regbanks:
-        print('cargo:warning={} has no register banks'.format(isa.name))
-
-    with fmt.indented('pub static INFO: RegInfo = RegInfo {', '};'):
-        # Bank descriptors.
-        with fmt.indented('banks: &[', '],'):
-            for regbank in isa.regbanks:
-                gen_regbank(regbank, fmt)
-        with fmt.indented('classes: &[', '],'):
-            for rc in isa.regclasses:
-                fmt.format('&{}_DATA,', rc.name)
-
-    # Register class descriptors.
-    for rc in isa.regclasses:
-        gen_regclass(rc, fmt)
-
-    # Emit constants for all the register units.
-    fmt.line('#[allow(dead_code, non_camel_case_types)]')
-    fmt.line('#[derive(Clone, Copy)]')
-    with fmt.indented('pub enum RU {', '}'):
-        for regbank in isa.regbanks:
-            gen_regbank_units(regbank, fmt)
-    with fmt.indented('impl Into for RU {', '}'):
-        with fmt.indented('fn into(self) -> RegUnit {', '}'):
-            fmt.line('self as RegUnit')
-
-
-def generate(isas, out_dir):
-    # type: (Sequence[TargetISA], str) -> None
-    for isa in isas:
-        fmt = srcgen.Formatter()
-        gen_isa(isa, fmt)
-        fmt.update_file('registers-{}.rs'.format(isa.name), out_dir)

From 9e084dbadce53c4735aec39bdf44c8d2b65080be Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 6 Nov 2018 13:24:06 -0800
Subject: [PATCH 2168/3084] Update to wabt 0.7.0.

---
 cranelift/Cargo.toml | 2 +-
 lib/wasm/Cargo.toml  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index e97cbbc000..51a714af46 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -31,7 +31,7 @@ clap = "2.32.0"
 serde = "1.0.8"
 term = "0.5.1"
 capstone = { version = "0.5.0", optional = true }
-wabt = { version = "0.6", optional = true }
+wabt = { version = "0.7.0", optional = true }
 target-lexicon = "0.0.3"
 pretty_env_logger = "0.2.4"
 file-per-thread-logger = "0.1.1"
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 28547d784f..1a09c44e5e 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -20,7 +20,7 @@ failure_derive = { version = "0.1.1", default-features = false }
 log = { version = "0.4.4", default-features = false }
 
 [dev-dependencies]
-wabt = "0.6.0"
+wabt = "0.7.0"
 target-lexicon = "0.0.3"
 
 [features]

From 5ea6c57b955e987022484c891b77ef899f7e219a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 23 Oct 2018 16:34:29 -0700
Subject: [PATCH 2169/3084] Update to the new wasmparser and port to the new
 readers API.

The new wasmparser API provides dedicated reader types for each section
type, which significantly simplifies the code.

This also changes WasmError::from_binary_reader_error into a From
trait so that we don't have to do .map_err(from_binary_reader_error)
throughout the code.
---
 lib/wasm/Cargo.toml                 |   2 +-
 lib/wasm/src/environ/spec.rs        |   5 +-
 lib/wasm/src/func_translator.rs     |  14 +-
 lib/wasm/src/lib.rs                 |   1 +
 lib/wasm/src/module_translator.rs   | 226 +++++++------
 lib/wasm/src/sections_translator.rs | 501 ++++++++++------------------
 6 files changed, 319 insertions(+), 430 deletions(-)

diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 1a09c44e5e..e674945ac2 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -10,7 +10,7 @@ readme = "README.md"
 keywords = ["webassembly", "wasm"]
 
 [dependencies]
-wasmparser = { version = "0.19.1", default-features = false }
+wasmparser = { version = "0.21.4", default-features = false }
 cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
 cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
 cranelift-frontend = { path = "../frontend", version = "0.22.0", default-features = false }
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 7f48ef99d2..d5518e037a 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -3,6 +3,7 @@
 use cranelift_codegen::cursor::FuncCursor;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::isa::TargetFrontendConfig;
+use std::convert::From;
 use std::vec::Vec;
 use translation_utils::{
     FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
@@ -62,9 +63,9 @@ pub enum WasmError {
     ImplLimitExceeded,
 }
 
-impl WasmError {
+impl From for WasmError {
     /// Convert from a `BinaryReaderError` to a `WasmError`.
-    pub fn from_binary_reader_error(e: BinaryReaderError) -> Self {
+    fn from(e: BinaryReaderError) -> Self {
         let BinaryReaderError { message, offset } = e;
         WasmError::InvalidWebAssembly { message, offset }
     }
diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs
index 5b7a116454..4b9e5fe7ca 100644
--- a/lib/wasm/src/func_translator.rs
+++ b/lib/wasm/src/func_translator.rs
@@ -9,7 +9,7 @@ use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir::{self, Ebb, InstBuilder};
 use cranelift_codegen::timing;
 use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
-use environ::{FuncEnvironment, ReturnMode, WasmError, WasmResult};
+use environ::{FuncEnvironment, ReturnMode, WasmResult};
 use state::TranslationState;
 use wasmparser::{self, BinaryReader};
 
@@ -135,16 +135,12 @@ fn parse_local_decls(
     num_params: usize,
 ) -> WasmResult<()> {
     let mut next_local = num_params;
-    let local_count = reader
-        .read_local_count()
-        .map_err(WasmError::from_binary_reader_error)?;
+    let local_count = reader.read_local_count()?;
 
     let mut locals_total = 0;
     for _ in 0..local_count {
         builder.set_srcloc(cur_srcloc(reader));
-        let (count, ty) = reader
-            .read_local_decl(&mut locals_total)
-            .map_err(WasmError::from_binary_reader_error)?;
+        let (count, ty) = reader.read_local_decl(&mut locals_total)?;
         declare_locals(builder, count, ty, &mut next_local);
     }
 
@@ -195,9 +191,7 @@ fn parse_function_body(
     // Keep going until the final `End` operator which pops the outermost block.
     while !state.control_stack.is_empty() {
         builder.set_srcloc(cur_srcloc(&reader));
-        let op = reader
-            .read_operator()
-            .map_err(WasmError::from_binary_reader_error)?;
+        let op = reader.read_operator()?;
         translate_operator(op, builder, state, environ)?;
     }
 
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index 76da0113ac..b119d2df8f 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -76,6 +76,7 @@ mod std {
 
     pub use self::alloc::string;
     pub use self::alloc::vec;
+    pub use core::convert;
     pub use core::fmt;
     pub use core::option;
     pub use core::{cmp, i32, str, u32};
diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs
index 6b7c854f27..7a7ce91b9b 100644
--- a/lib/wasm/src/module_translator.rs
+++ b/lib/wasm/src/module_translator.rs
@@ -1,119 +1,143 @@
 //! Translation skeleton that traverses the whole WebAssembly module and call helper functions
 //! to deal with each part of it.
 use cranelift_codegen::timing;
-use environ::{ModuleEnvironment, WasmError, WasmResult};
+use environ::{ModuleEnvironment, WasmResult};
 use sections_translator::{
     parse_code_section, parse_data_section, parse_element_section, parse_export_section,
-    parse_function_section, parse_function_signatures, parse_global_section, parse_import_section,
-    parse_memory_section, parse_start_section, parse_table_section,
+    parse_function_section, parse_global_section, parse_import_section, parse_memory_section,
+    parse_start_section, parse_table_section, parse_type_section,
 };
-use wasmparser::{Parser, ParserInput, ParserState, SectionCode, WasmDecoder};
+use wasmparser::{ModuleReader, SectionCode};
 
 /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cranelift IR
 /// [`Function`](../codegen/ir/function/struct.Function.html).
-/// Returns the functions and also the mappings for imported functions and signature between the
-/// indexes in the wasm module and the indexes inside each functions.
 pub fn translate_module<'data>(
     data: &'data [u8],
     environ: &mut ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
     let _tt = timing::wasm_translate_module();
-    let mut parser = Parser::new(data);
-    match *parser.read() {
-        ParserState::BeginWasm { .. } => {}
-        ParserState::Error(e) => {
-            return Err(WasmError::from_binary_reader_error(e));
+    let mut reader = ModuleReader::new(data)?;
+
+    reader.skip_custom_sections()?;
+    if reader.eof() {
+        return Ok(());
+    }
+    let mut section = reader.read()?;
+
+    if let SectionCode::Type = section.code {
+        let types = section.get_type_section_reader()?;
+        parse_type_section(types, environ)?;
+
+        reader.skip_custom_sections()?;
+        if reader.eof() {
+            return Ok(());
         }
-        ref s => panic!("modules should begin properly: {:?}", s),
+        section = reader.read()?;
     }
-    let mut next_input = ParserInput::Default;
-    loop {
-        match *parser.read_with_input(next_input) {
-            ParserState::BeginSection {
-                code: SectionCode::Type,
-                ..
-            } => {
-                parse_function_signatures(&mut parser, environ)?;
-                next_input = ParserInput::Default;
-            }
-            ParserState::BeginSection {
-                code: SectionCode::Import,
-                ..
-            } => {
-                parse_import_section(&mut parser, environ)?;
-                next_input = ParserInput::Default;
-            }
-            ParserState::BeginSection {
-                code: SectionCode::Function,
-                ..
-            } => {
-                parse_function_section(&mut parser, environ)?;
-                next_input = ParserInput::Default;
-            }
-            ParserState::BeginSection {
-                code: SectionCode::Table,
-                ..
-            } => {
-                parse_table_section(&mut parser, environ)?;
-            }
-            ParserState::BeginSection {
-                code: SectionCode::Memory,
-                ..
-            } => {
-                parse_memory_section(&mut parser, environ)?;
-                next_input = ParserInput::Default;
-            }
-            ParserState::BeginSection {
-                code: SectionCode::Global,
-                ..
-            } => {
-                parse_global_section(&mut parser, environ)?;
-                next_input = ParserInput::Default;
-            }
-            ParserState::BeginSection {
-                code: SectionCode::Export,
-                ..
-            } => {
-                parse_export_section(&mut parser, environ)?;
-                next_input = ParserInput::Default;
-            }
-            ParserState::BeginSection {
-                code: SectionCode::Start,
-                ..
-            } => {
-                parse_start_section(&mut parser, environ)?;
-                next_input = ParserInput::Default;
-            }
-            ParserState::BeginSection {
-                code: SectionCode::Element,
-                ..
-            } => {
-                parse_element_section(&mut parser, environ)?;
-                next_input = ParserInput::Default;
-            }
-            ParserState::BeginSection {
-                code: SectionCode::Code,
-                ..
-            } => parse_code_section(&mut parser, environ)?,
-            ParserState::EndSection => {
-                next_input = ParserInput::Default;
-            }
-            ParserState::EndWasm => return Ok(()),
-            ParserState::BeginSection {
-                code: SectionCode::Data,
-                ..
-            } => {
-                parse_data_section(&mut parser, environ)?;
-            }
-            ParserState::BeginSection {
-                code: SectionCode::Custom { .. },
-                ..
-            } => {
-                // Ignore unknown custom sections.
-                next_input = ParserInput::SkipSection;
-            }
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            _ => panic!("wrong content in the preamble"),
-        };
+
+    if let SectionCode::Import = section.code {
+        let imports = section.get_import_section_reader()?;
+        parse_import_section(imports, environ)?;
+
+        reader.skip_custom_sections()?;
+        if reader.eof() {
+            return Ok(());
+        }
+        section = reader.read()?;
     }
+
+    if let SectionCode::Function = section.code {
+        let functions = section.get_function_section_reader()?;
+        parse_function_section(functions, environ)?;
+
+        reader.skip_custom_sections()?;
+        if reader.eof() {
+            return Ok(());
+        }
+        section = reader.read()?;
+    }
+
+    if let SectionCode::Table = section.code {
+        let tables = section.get_table_section_reader()?;
+        parse_table_section(tables, environ)?;
+
+        reader.skip_custom_sections()?;
+        if reader.eof() {
+            return Ok(());
+        }
+        section = reader.read()?;
+    }
+
+    if let SectionCode::Memory = section.code {
+        let memories = section.get_memory_section_reader()?;
+        parse_memory_section(memories, environ)?;
+
+        reader.skip_custom_sections()?;
+        if reader.eof() {
+            return Ok(());
+        }
+        section = reader.read()?;
+    }
+
+    if let SectionCode::Global = section.code {
+        let globals = section.get_global_section_reader()?;
+        parse_global_section(globals, environ)?;
+
+        reader.skip_custom_sections()?;
+        if reader.eof() {
+            return Ok(());
+        }
+        section = reader.read()?;
+    }
+
+    if let SectionCode::Export = section.code {
+        let exports = section.get_export_section_reader()?;
+        parse_export_section(exports, environ)?;
+
+        reader.skip_custom_sections()?;
+        if reader.eof() {
+            return Ok(());
+        }
+        section = reader.read()?;
+    }
+
+    if let SectionCode::Start = section.code {
+        let start = section.get_start_section_content()?;
+        parse_start_section(start, environ)?;
+
+        reader.skip_custom_sections()?;
+        if reader.eof() {
+            return Ok(());
+        }
+        section = reader.read()?;
+    }
+
+    if let SectionCode::Element = section.code {
+        let elements = section.get_element_section_reader()?;
+        parse_element_section(elements, environ)?;
+
+        reader.skip_custom_sections()?;
+        if reader.eof() {
+            return Ok(());
+        }
+        section = reader.read()?;
+    }
+
+    if let SectionCode::Code = section.code {
+        let code = section.get_code_section_reader()?;
+        parse_code_section(code, environ)?;
+
+        reader.skip_custom_sections()?;
+        if reader.eof() {
+            return Ok(());
+        }
+        section = reader.read()?;
+    }
+
+    if let SectionCode::Data = section.code {
+        let data = section.get_data_section_reader()?;
+        parse_data_section(data, environ)?;
+    }
+
+    Ok(())
 }
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 3865a9c026..e8468d9d4c 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -9,32 +9,32 @@
 //! interpreted on the fly.
 use cranelift_codegen::ir::{self, AbiParam, Signature};
 use cranelift_entity::EntityRef;
-use environ::{ModuleEnvironment, WasmError, WasmResult};
+use environ::{ModuleEnvironment, WasmResult};
 use std::str::from_utf8;
 use std::vec::Vec;
 use translation_utils::{
     type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex,
     Table, TableElementType, TableIndex,
 };
-use wasmparser;
 use wasmparser::{
-    ExternalKind, FuncType, ImportSectionEntryType, MemoryType, Operator, Parser, ParserState,
-    WasmDecoder,
+    self, CodeSectionReader, Data, DataSectionReader, Element, ElementSectionReader, Export,
+    ExportSectionReader, ExternalKind, FuncType, FunctionSectionReader, GlobalSectionReader,
+    GlobalType, Import, ImportSectionEntryType, ImportSectionReader, MemorySectionReader,
+    MemoryType, Operator, TableSectionReader, TypeSectionReader,
 };
 
-/// Reads the Type Section of the wasm module and returns the corresponding function signatures.
-pub fn parse_function_signatures(
-    parser: &mut Parser,
+/// Parses the Type section of the wasm module.
+pub fn parse_type_section(
+    types: TypeSectionReader,
     environ: &mut ModuleEnvironment,
 ) -> WasmResult<()> {
-    loop {
-        match *parser.read() {
-            ParserState::EndSection => break,
-            ParserState::TypeSectionEntry(FuncType {
+    for entry in types {
+        match entry? {
+            FuncType {
                 form: wasmparser::Type::Func,
                 ref params,
                 ref returns,
-            }) => {
+            } => {
                 let mut sig = Signature::new(environ.target_config().default_call_conv);
                 sig.params.extend(params.iter().map(|ty| {
                     let cret_arg: ir::Type = type_to_type(*ty)
@@ -48,24 +48,23 @@ pub fn parse_function_signatures(
                 }));
                 environ.declare_signature(&sig);
             }
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
+            ref s => panic!("unsupported type: {:?}", s),
         }
     }
     Ok(())
 }
 
-/// Retrieves the imports from the imports section of the binary.
+/// Parses the Import section of the wasm module.
 pub fn parse_import_section<'data>(
-    parser: &mut Parser<'data>,
+    imports: ImportSectionReader<'data>,
     environ: &mut ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
-    loop {
-        match *parser.read() {
-            ParserState::ImportSectionEntry {
-                ty: ImportSectionEntryType::Function(sig),
+    for entry in imports {
+        match entry? {
+            Import {
                 module,
                 field,
+                ty: ImportSectionEntryType::Function(sig),
             } => {
                 // The input has already been validated, so we should be able to
                 // assume valid UTF-8 and use `from_utf8_unchecked` if performance
@@ -78,7 +77,7 @@ pub fn parse_import_section<'data>(
                     field_name,
                 );
             }
-            ParserState::ImportSectionEntry {
+            Import {
                 ty:
                     ImportSectionEntryType::Memory(MemoryType {
                         limits: ref memlimits,
@@ -92,7 +91,7 @@ pub fn parse_import_section<'data>(
                     shared,
                 });
             }
-            ParserState::ImportSectionEntry {
+            Import {
                 ty: ImportSectionEntryType::Global(ref ty),
                 ..
             } => {
@@ -102,7 +101,7 @@ pub fn parse_import_section<'data>(
                     initializer: GlobalInit::Import(),
                 });
             }
-            ParserState::ImportSectionEntry {
+            Import {
                 ty: ImportSectionEntryType::Table(ref tab),
                 ..
             } => environ.declare_table(Table {
@@ -113,333 +112,203 @@ pub fn parse_import_section<'data>(
                 size: tab.limits.initial as usize,
                 maximum: tab.limits.maximum.map(|x| x as usize),
             }),
-            ParserState::EndSection => break,
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-    }
-    Ok(())
-}
-
-/// Retrieves the correspondences between functions and signatures from the function section
-pub fn parse_function_section(
-    parser: &mut Parser,
-    environ: &mut ModuleEnvironment,
-) -> WasmResult<()> {
-    loop {
-        match *parser.read() {
-            ParserState::FunctionSectionEntry(sigindex) => {
-                environ.declare_func_type(SignatureIndex::new(sigindex as usize));
-            }
-            ParserState::EndSection => break,
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-    }
-    Ok(())
-}
-
-/// Retrieves the names of the functions from the export section
-pub fn parse_export_section<'data>(
-    parser: &mut Parser<'data>,
-    environ: &mut ModuleEnvironment<'data>,
-) -> WasmResult<()> {
-    loop {
-        match *parser.read() {
-            ParserState::ExportSectionEntry {
-                field,
-                ref kind,
-                index,
-            } => {
-                // The input has already been validated, so we should be able to
-                // assume valid UTF-8 and use `from_utf8_unchecked` if performance
-                // becomes a concern here.
-                let name = from_utf8(field).unwrap();
-                let index = index as usize;
-                match *kind {
-                    ExternalKind::Function => {
-                        environ.declare_func_export(FuncIndex::new(index), name)
-                    }
-                    ExternalKind::Table => {
-                        environ.declare_table_export(TableIndex::new(index), name)
-                    }
-                    ExternalKind::Memory => {
-                        environ.declare_memory_export(MemoryIndex::new(index), name)
-                    }
-                    ExternalKind::Global => {
-                        environ.declare_global_export(GlobalIndex::new(index), name)
-                    }
-                }
-            }
-            ParserState::EndSection => break,
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-    }
-    Ok(())
-}
-
-/// Retrieves the start function index from the start section
-pub fn parse_start_section(parser: &mut Parser, environ: &mut ModuleEnvironment) -> WasmResult<()> {
-    loop {
-        match *parser.read() {
-            ParserState::StartSectionEntry(index) => {
-                environ.declare_start_func(FuncIndex::new(index as usize));
-            }
-            ParserState::EndSection => break,
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-    }
-    Ok(())
-}
-
-/// Retrieves the size and maximum fields of memories from the memory section
-pub fn parse_memory_section(
-    parser: &mut Parser,
-    environ: &mut ModuleEnvironment,
-) -> WasmResult<()> {
-    loop {
-        match *parser.read() {
-            ParserState::MemorySectionEntry(ref ty) => {
-                environ.declare_memory(Memory {
-                    pages_count: ty.limits.initial as usize,
-                    maximum: ty.limits.maximum.map(|x| x as usize),
-                    shared: ty.shared,
-                });
-            }
-            ParserState::EndSection => break,
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-    }
-    Ok(())
-}
-
-/// Retrieves the size and maximum fields of memories from the memory section
-pub fn parse_global_section(
-    parser: &mut Parser,
-    environ: &mut ModuleEnvironment,
-) -> WasmResult<()> {
-    loop {
-        let (content_type, mutability) = match *parser.read() {
-            ParserState::BeginGlobalSectionEntry(ref ty) => (ty.content_type, ty.mutable),
-            ParserState::EndSection => break,
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-        match *parser.read() {
-            ParserState::BeginInitExpressionBody => (),
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
         }
-        let initializer = match *parser.read() {
-            ParserState::InitExpressionOperator(Operator::I32Const { value }) => {
-                GlobalInit::I32Const(value)
-            }
-            ParserState::InitExpressionOperator(Operator::I64Const { value }) => {
-                GlobalInit::I64Const(value)
-            }
-            ParserState::InitExpressionOperator(Operator::F32Const { value }) => {
-                GlobalInit::F32Const(value.bits())
-            }
-            ParserState::InitExpressionOperator(Operator::F64Const { value }) => {
-                GlobalInit::F64Const(value.bits())
-            }
-            ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => {
+    }
+    Ok(())
+}
+
+/// Parses the Function section of the wasm module.
+pub fn parse_function_section(
+    functions: FunctionSectionReader,
+    environ: &mut ModuleEnvironment,
+) -> WasmResult<()> {
+    for entry in functions {
+        let sigindex = entry?;
+        environ.declare_func_type(SignatureIndex::new(sigindex as usize));
+    }
+    Ok(())
+}
+
+/// Parses the Table section of the wasm module.
+pub fn parse_table_section(
+    tables: TableSectionReader,
+    environ: &mut ModuleEnvironment,
+) -> WasmResult<()> {
+    for entry in tables {
+        let table = entry?;
+        environ.declare_table(Table {
+            ty: match type_to_type(table.element_type) {
+                Ok(t) => TableElementType::Val(t),
+                Err(()) => TableElementType::Func(),
+            },
+            size: table.limits.initial as usize,
+            maximum: table.limits.maximum.map(|x| x as usize),
+        });
+    }
+    Ok(())
+}
+
+/// Parses the Memory section of the wasm module.
+pub fn parse_memory_section(
+    memories: MemorySectionReader,
+    environ: &mut ModuleEnvironment,
+) -> WasmResult<()> {
+    for entry in memories {
+        let memory = entry?;
+        environ.declare_memory(Memory {
+            pages_count: memory.limits.initial as usize,
+            maximum: memory.limits.maximum.map(|x| x as usize),
+            shared: memory.shared,
+        });
+    }
+    Ok(())
+}
+
+/// Parses the Global section of the wasm module.
+pub fn parse_global_section(
+    globals: GlobalSectionReader,
+    environ: &mut ModuleEnvironment,
+) -> WasmResult<()> {
+    for entry in globals {
+        let wasmparser::Global {
+            ty: GlobalType {
+                content_type,
+                mutable,
+            },
+            init_expr,
+        } = entry?;
+        let mut init_expr_reader = init_expr.get_binary_reader();
+        let initializer = match init_expr_reader.read_operator()? {
+            Operator::I32Const { value } => GlobalInit::I32Const(value),
+            Operator::I64Const { value } => GlobalInit::I64Const(value),
+            Operator::F32Const { value } => GlobalInit::F32Const(value.bits()),
+            Operator::F64Const { value } => GlobalInit::F64Const(value.bits()),
+            Operator::GetGlobal { global_index } => {
                 GlobalInit::GlobalRef(GlobalIndex::new(global_index as usize))
             }
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
+            ref s => panic!("unsupported init expr in global section: {:?}", s),
         };
-        match *parser.read() {
-            ParserState::EndInitExpressionBody => (),
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        }
         let global = Global {
             ty: type_to_type(content_type).unwrap(),
-            mutability,
+            mutability: mutable,
             initializer,
         };
         environ.declare_global(global);
-        match *parser.read() {
-            ParserState::EndGlobalSectionEntry => (),
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        }
     }
     Ok(())
 }
 
-pub fn parse_data_section<'data>(
-    parser: &mut Parser<'data>,
+/// Parses the Export section of the wasm module.
+pub fn parse_export_section<'data>(
+    exports: ExportSectionReader<'data>,
     environ: &mut ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
-    loop {
-        let memory_index = match *parser.read() {
-            ParserState::BeginDataSectionEntry(memory_index) => memory_index,
-            ParserState::EndSection => break,
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-        match *parser.read() {
-            ParserState::BeginInitExpressionBody => (),
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-        let (base, offset) = match *parser.read() {
-            ParserState::InitExpressionOperator(Operator::I32Const { value }) => {
-                (None, value as u32 as usize)
-            }
-            ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => {
-                match environ
-                    .get_global(GlobalIndex::new(global_index as usize))
-                    .initializer
-                {
-                    GlobalInit::I32Const(value) => (None, value as u32 as usize),
-                    GlobalInit::Import() => (Some(GlobalIndex::new(global_index as usize)), 0),
-                    _ => panic!("should not happen"),
-                }
-            }
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-        match *parser.read() {
-            ParserState::EndInitExpressionBody => (),
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-        match *parser.read() {
-            ParserState::BeginDataSectionEntryBody(_) => (),
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-        let mut running_offset = offset;
-        loop {
-            let data = match *parser.read() {
-                ParserState::DataSectionEntryBodyChunk(data) => data,
-                ParserState::EndDataSectionEntryBody => break,
-                ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-                ref s => panic!("unexpected section content: {:?}", s),
-            };
-            environ.declare_data_initialization(
-                MemoryIndex::new(memory_index as usize),
-                base,
-                running_offset,
-                data,
-            );
-            running_offset += data.len();
+    for entry in exports {
+        let Export {
+            field,
+            ref kind,
+            index,
+        } = entry?;
+
+        // The input has already been validated, so we should be able to
+        // assume valid UTF-8 and use `from_utf8_unchecked` if performance
+        // becomes a concern here.
+        let name = from_utf8(field).unwrap();
+        let index = index as usize;
+        match *kind {
+            ExternalKind::Function => environ.declare_func_export(FuncIndex::new(index), name),
+            ExternalKind::Table => environ.declare_table_export(TableIndex::new(index), name),
+            ExternalKind::Memory => environ.declare_memory_export(MemoryIndex::new(index), name),
+            ExternalKind::Global => environ.declare_global_export(GlobalIndex::new(index), name),
         }
-        match *parser.read() {
-            ParserState::EndDataSectionEntry => (),
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
     }
     Ok(())
 }
 
-/// Retrieves the tables from the table section
-pub fn parse_table_section(parser: &mut Parser, environ: &mut ModuleEnvironment) -> WasmResult<()> {
-    loop {
-        match *parser.read() {
-            ParserState::TableSectionEntry(ref table) => environ.declare_table(Table {
-                ty: match type_to_type(table.element_type) {
-                    Ok(t) => TableElementType::Val(t),
-                    Err(()) => TableElementType::Func(),
-                },
-                size: table.limits.initial as usize,
-                maximum: table.limits.maximum.map(|x| x as usize),
-            }),
-            ParserState::EndSection => break,
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-    }
+/// Parses the Start section of the wasm module.
+pub fn parse_start_section(index: u32, environ: &mut ModuleEnvironment) -> WasmResult<()> {
+    environ.declare_start_func(FuncIndex::new(index as usize));
     Ok(())
 }
 
-/// Retrieves the elements from the element section
-pub fn parse_element_section(
-    parser: &mut Parser,
+/// Parses the Element section of the wasm module.
+pub fn parse_element_section<'data>(
+    elements: ElementSectionReader<'data>,
     environ: &mut ModuleEnvironment,
 ) -> WasmResult<()> {
-    loop {
-        let table_index = match *parser.read() {
-            ParserState::BeginElementSectionEntry(table_index) => {
-                TableIndex::new(table_index as usize)
-            }
-            ParserState::EndSection => break,
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-        match *parser.read() {
-            ParserState::BeginInitExpressionBody => (),
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-        let (base, offset) = match *parser.read() {
-            ParserState::InitExpressionOperator(Operator::I32Const { value }) => {
-                (None, value as u32 as usize)
-            }
-            ParserState::InitExpressionOperator(Operator::GetGlobal { global_index }) => {
-                match environ
-                    .get_global(GlobalIndex::new(global_index as usize))
-                    .initializer
-                {
-                    GlobalInit::I32Const(value) => (None, value as u32 as usize),
-                    GlobalInit::Import() => (Some(GlobalIndex::new(global_index as usize)), 0),
-                    _ => panic!("should not happen"),
-                }
-            }
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-        match *parser.read() {
-            ParserState::EndInitExpressionBody => (),
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-        match *parser.read() {
-            ParserState::ElementSectionEntryBody(ref elements) => {
-                let elems: Vec = elements
-                    .iter()
-                    .map(|&x| FuncIndex::new(x as usize))
-                    .collect();
-                environ.declare_table_elements(table_index, base, offset, elems)
-            }
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
-        };
-        match *parser.read() {
-            ParserState::EndElementSectionEntry => (),
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("unexpected section content: {:?}", s),
+    for entry in elements {
+        let Element {
+            table_index,
+            init_expr,
+            items,
+        } = entry?;
+        let mut init_expr_reader = init_expr.get_binary_reader();
+        let (base, offset) = match init_expr_reader.read_operator()? {
+            Operator::I32Const { value } => (None, value as u32 as usize),
+            Operator::GetGlobal { global_index } => match environ
+                .get_global(GlobalIndex::new(global_index as usize))
+                .initializer
+            {
+                GlobalInit::I32Const(value) => (None, value as u32 as usize),
+                GlobalInit::Import() => (Some(GlobalIndex::new(global_index as usize)), 0),
+                _ => panic!("should not happen"),
+            },
+            ref s => panic!("unsupported init expr in element section: {:?}", s),
         };
+        let items_reader = items.get_items_reader()?;
+        let mut elems = Vec::new();
+        for item in items_reader {
+            let x = item?;
+            elems.push(FuncIndex::new(x as usize));
+        }
+        environ.declare_table_elements(TableIndex::new(table_index as usize), base, offset, elems)
     }
     Ok(())
 }
 
-/// Parses every function body in the code section and defines the corresponding function.
+/// Parses the Code section of the wasm module.
 pub fn parse_code_section<'data>(
-    parser: &mut Parser<'data>,
+    code: CodeSectionReader<'data>,
     environ: &mut ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
-    loop {
-        match *parser.read() {
-            ParserState::BeginFunctionBody { .. } => {}
-            ParserState::EndSection => break,
-            ParserState::Error(e) => return Err(WasmError::from_binary_reader_error(e)),
-            ref s => panic!("wrong content in code section: {:?}", s),
-        }
-        let mut reader = parser.create_binary_reader();
+    for body in code {
+        let mut reader = body?.get_binary_reader();
         let size = reader.bytes_remaining();
-        environ.define_function_body(
-            reader
-                .read_bytes(size)
-                .map_err(WasmError::from_binary_reader_error)?,
-        )?;
+        environ.define_function_body(reader.read_bytes(size)?)?;
+    }
+    Ok(())
+}
+
+/// Parses the Data section of the wasm module.
+pub fn parse_data_section<'data>(
+    data: DataSectionReader<'data>,
+    environ: &mut ModuleEnvironment<'data>,
+) -> WasmResult<()> {
+    for entry in data {
+        let Data {
+            memory_index,
+            init_expr,
+            data,
+        } = entry?;
+        let mut init_expr_reader = init_expr.get_binary_reader();
+        let (base, offset) = match init_expr_reader.read_operator()? {
+            Operator::I32Const { value } => (None, value as u32 as usize),
+            Operator::GetGlobal { global_index } => match environ
+                .get_global(GlobalIndex::new(global_index as usize))
+                .initializer
+            {
+                GlobalInit::I32Const(value) => (None, value as u32 as usize),
+                GlobalInit::Import() => (Some(GlobalIndex::new(global_index as usize)), 0),
+                _ => panic!("should not happen"),
+            },
+            ref s => panic!("unsupported init expr in data section: {:?}", s),
+        };
+        environ.declare_data_initialization(
+            MemoryIndex::new(memory_index as usize),
+            base,
+            offset,
+            data,
+        );
     }
     Ok(())
 }

From 05c0b3bdd174d25b5cd3a340cc1310898c9458eb Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 19 Oct 2018 11:09:39 -0700
Subject: [PATCH 2170/3084] Insert copies to support a value being used as
 multiple return values.

When one value is used multiple times for separate return values, we
need to copy it to produce a new value, so that each value can be
allocated a different register.
---
 .../filetests/regalloc/multiple-returns.clif  | 13 +++++++
 lib/codegen/src/regalloc/spilling.rs          | 36 +++++++++++++++++--
 2 files changed, 47 insertions(+), 2 deletions(-)
 create mode 100644 cranelift/filetests/regalloc/multiple-returns.clif

diff --git a/cranelift/filetests/regalloc/multiple-returns.clif b/cranelift/filetests/regalloc/multiple-returns.clif
new file mode 100644
index 0000000000..6e36c9b9b8
--- /dev/null
+++ b/cranelift/filetests/regalloc/multiple-returns.clif
@@ -0,0 +1,13 @@
+test regalloc
+target x86_64
+
+; Return the same value twice. This needs a copy so that each value can be
+; allocated its own register.
+function %multiple_returns() -> i64, i64 {
+ebb0:
+    v2 = iconst.i64 0
+    return v2, v2
+}
+; check: v2 = iconst.i64 0
+; check: v3 = copy v2
+; check: return v2, v3
diff --git a/lib/codegen/src/regalloc/spilling.rs b/lib/codegen/src/regalloc/spilling.rs
index 92b632c30a..132c98be92 100644
--- a/lib/codegen/src/regalloc/spilling.rs
+++ b/lib/codegen/src/regalloc/spilling.rs
@@ -17,8 +17,8 @@
 
 use cursor::{Cursor, EncCursor};
 use dominator_tree::DominatorTree;
-use ir::{Ebb, Function, Inst, InstBuilder, SigRef, Value, ValueLoc};
-use isa::registers::{RegClassIndex, RegClassMask};
+use ir::{ArgumentLoc, Ebb, Function, Inst, InstBuilder, SigRef, Value, ValueLoc};
+use isa::registers::{RegClass, RegClassIndex, RegClassMask, RegUnit};
 use isa::{ConstraintKind, EncInfo, RecipeConstraints, RegInfo, TargetIsa};
 use regalloc::affinity::Affinity;
 use regalloc::live_value_tracker::{LiveValue, LiveValueTracker};
@@ -30,6 +30,15 @@ use std::vec::Vec;
 use timing;
 use topo_order::TopoOrder;
 
+/// Return a top-level register class which contains `unit`.
+fn toprc_containing_regunit(unit: RegUnit, reginfo: &RegInfo) -> RegClass {
+    let bank = reginfo.bank_containing_regunit(unit).unwrap();
+    reginfo.classes[bank.first_toprc..(bank.first_toprc + bank.num_toprcs)]
+        .iter()
+        .find(|&rc| rc.contains(unit))
+        .expect("reg unit should be in a toprc")
+}
+
 /// Persistent data structures for the spilling pass.
 pub struct Spilling {
     spills: Vec,
@@ -335,6 +344,29 @@ impl<'a> Context<'a> {
                 self.reg_uses.push(reguse);
             }
         }
+
+        // Similarly, for return instructions, collect uses of ABI-defined
+        // return values.
+        if self.cur.func.dfg[inst].opcode().is_return() {
+            for (ret_idx, (ret, &arg)) in
+                self.cur.func.signature.returns.iter().zip(args).enumerate()
+            {
+                let idx = constraints.ins.len() + ret_idx;
+                let unit = match ret.location {
+                    ArgumentLoc::Unassigned => {
+                        panic!("function return signature should be legalized")
+                    }
+                    ArgumentLoc::Reg(unit) => unit,
+                    ArgumentLoc::Stack(_) => continue,
+                };
+                let toprc = toprc_containing_regunit(unit, &self.reginfo);
+                let mut reguse = RegUse::new(arg, idx, toprc.into());
+                reguse.fixed = true;
+
+                debug!("  reguse: {}", reguse);
+                self.reg_uses.push(reguse);
+            }
+        }
     }
 
     // Collect register uses from the ABI input constraints.

From 681cb5e20abfa7ffb186a22adaeb4666d8113406 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 6 Nov 2018 12:08:34 -0800
Subject: [PATCH 2171/3084] Assert that the non-fixed inputs to return match
 the function's signature.

---
 lib/codegen/src/regalloc/spilling.rs | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/codegen/src/regalloc/spilling.rs b/lib/codegen/src/regalloc/spilling.rs
index 132c98be92..1ce0649fed 100644
--- a/lib/codegen/src/regalloc/spilling.rs
+++ b/lib/codegen/src/regalloc/spilling.rs
@@ -348,6 +348,11 @@ impl<'a> Context<'a> {
         // Similarly, for return instructions, collect uses of ABI-defined
         // return values.
         if self.cur.func.dfg[inst].opcode().is_return() {
+            debug_assert_eq!(
+                self.cur.func.dfg.inst_variable_args(inst).len(),
+                self.cur.func.signature.returns.len(),
+                "The non-fixed arguments in a return should follow the function's signature."
+            );
             for (ret_idx, (ret, &arg)) in
                 self.cur.func.signature.returns.iter().zip(args).enumerate()
             {

From 88bbbca6cdb49cfdf58ee445c1847ba7d742334b Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 26 Oct 2018 06:39:44 -0700
Subject: [PATCH 2172/3084] Make regalloc visit fallthrough_return
 instructions.

Add an explicit "is_ghost" property to selected instructions, and use
that to determine whether reload and coloring should visit instructions.
This allows them to visit fallthrough_return instructions and insert
fills and register moves as needed.
---
 .../regalloc/fallthrough-return.clif          | 23 +++++++
 lib/codegen/meta-python/base/instructions.py  |  8 +--
 lib/codegen/meta-python/cdsl/instructions.py  |  2 +
 lib/codegen/src/regalloc/coloring.rs          | 59 +++++++++-------
 lib/codegen/src/regalloc/reload.rs            | 69 ++++++++++---------
 5 files changed, 101 insertions(+), 60 deletions(-)
 create mode 100644 cranelift/filetests/regalloc/fallthrough-return.clif

diff --git a/cranelift/filetests/regalloc/fallthrough-return.clif b/cranelift/filetests/regalloc/fallthrough-return.clif
new file mode 100644
index 0000000000..557710eb5a
--- /dev/null
+++ b/cranelift/filetests/regalloc/fallthrough-return.clif
@@ -0,0 +1,23 @@
+test regalloc
+target x86_64
+
+; Test that fallthrough returns are visited by reload and coloring.
+
+function %foo() -> f64 {
+  fn0 = %bar()
+
+ebb0:
+  v0 = f64const 0.0
+  call fn0()
+  fallthrough_return v0
+}
+; check: fill v0
+
+function %foo() -> f64 {
+  fn0 = %bar() -> f64, f64
+
+ebb0:
+  v0, v1 = call fn0()
+  fallthrough_return v1
+}
+; check: regmove v1, %xmm1 -> %xmm0
diff --git a/lib/codegen/meta-python/base/instructions.py b/lib/codegen/meta-python/base/instructions.py
index 498a75727f..718bb33f45 100644
--- a/lib/codegen/meta-python/base/instructions.py
+++ b/lib/codegen/meta-python/base/instructions.py
@@ -844,7 +844,7 @@ vsplit = Instruction(
         the lanes from ``x``. The result may be two scalars if ``x`` only had
         two lanes.
         """,
-        ins=x, outs=(lo, hi))
+        ins=x, outs=(lo, hi), is_ghost=True)
 
 Any128 = TypeVar(
         'Any128', 'Any scalar or vector type with as most 128 lanes',
@@ -864,7 +864,7 @@ vconcat = Instruction(
 
         It is possible to form a vector by concatenating two scalars.
         """,
-        ins=(x, y), outs=a)
+        ins=(x, y), outs=a, is_ghost=True)
 
 c = Operand('c', TxN.as_bool(), doc='Controlling vector')
 x = Operand('x', TxN, doc='Value to use where `c` is true')
@@ -2009,7 +2009,7 @@ isplit = Instruction(
         Returns the low half of `x` and the high half of `x` as two independent
         values.
         """,
-        ins=x, outs=(lo, hi))
+        ins=x, outs=(lo, hi), is_ghost=True)
 
 
 NarrowInt = TypeVar(
@@ -2029,6 +2029,6 @@ iconcat = Instruction(
         the same number of lanes as the inputs, but the lanes are twice the
         size.
         """,
-        ins=(lo, hi), outs=a)
+        ins=(lo, hi), outs=a, is_ghost=True)
 
 GROUP.close()
diff --git a/lib/codegen/meta-python/cdsl/instructions.py b/lib/codegen/meta-python/cdsl/instructions.py
index f158ecbd4b..30bdc306e6 100644
--- a/lib/codegen/meta-python/cdsl/instructions.py
+++ b/lib/codegen/meta-python/cdsl/instructions.py
@@ -92,6 +92,7 @@ class Instruction(object):
     :param is_indirect_branch: This is an indirect branch instruction.
     :param is_call: This is a call instruction.
     :param is_return: This is a return instruction.
+    :param is_ghost: This is a ghost instruction.
     :param can_trap: This instruction can trap.
     :param can_load: This instruction can load from memory.
     :param can_store: This instruction can store to memory.
@@ -108,6 +109,7 @@ class Instruction(object):
             'True for all indirect branch or jump instructions.',
             'is_call': 'Is this a call instruction?',
             'is_return': 'Is this a return instruction?',
+            'is_ghost': 'Is this a ghost instruction?',
             'can_load': 'Can this instruction read from memory?',
             'can_store': 'Can this instruction write to memory?',
             'can_trap': 'Can this instruction cause a trap?',
diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs
index 3c1a11b3fb..f0b96e79c9 100644
--- a/lib/codegen/src/regalloc/coloring.rs
+++ b/lib/codegen/src/regalloc/coloring.rs
@@ -165,8 +165,9 @@ impl<'a> Context<'a> {
         self.cur.goto_top(ebb);
         while let Some(inst) = self.cur.next_inst() {
             self.cur.use_srcloc(inst);
-            let enc = self.cur.func.encodings[inst];
-            if let Some(constraints) = self.encinfo.operand_constraints(enc) {
+            if !self.cur.func.dfg[inst].opcode().is_ghost() {
+                let enc = self.cur.func.encodings[inst];
+                let constraints = self.encinfo.operand_constraints(enc);
                 if self.visit_inst(inst, constraints, tracker, &mut regs) {
                     self.replace_global_defines(inst, tracker);
                     // Restore cursor location after `replace_global_defines` moves it.
@@ -290,7 +291,7 @@ impl<'a> Context<'a> {
     fn visit_inst(
         &mut self,
         inst: Inst,
-        constraints: &RecipeConstraints,
+        constraints: Option<&RecipeConstraints>,
         tracker: &mut LiveValueTracker,
         regs: &mut AvailableRegs,
     ) -> bool {
@@ -306,7 +307,9 @@ impl<'a> Context<'a> {
 
         // Program the solver with register constraints for the input side.
         self.solver.reset(®s.input);
-        self.program_input_constraints(inst, constraints.ins);
+        if let Some(constraints) = constraints {
+            self.program_input_constraints(inst, constraints.ins);
+        }
         let call_sig = self.cur.func.dfg.call_signature(inst);
         if let Some(sig) = call_sig {
             program_input_abi(
@@ -390,14 +393,16 @@ impl<'a> Context<'a> {
         // Program the fixed output constraints before the general defines. This allows us to
         // detect conflicts between fixed outputs and tied operands where the input value hasn't
         // been converted to a solver variable.
-        if constraints.fixed_outs {
-            self.program_fixed_outputs(
-                constraints.outs,
-                defs,
-                throughs,
-                &mut replace_global_defines,
-                ®s.global,
-            );
+        if let Some(constraints) = constraints {
+            if constraints.fixed_outs {
+                self.program_fixed_outputs(
+                    constraints.outs,
+                    defs,
+                    throughs,
+                    &mut replace_global_defines,
+                    ®s.global,
+                );
+            }
         }
         if let Some(sig) = call_sig {
             self.program_output_abi(
@@ -408,13 +413,15 @@ impl<'a> Context<'a> {
                 ®s.global,
             );
         }
-        self.program_output_constraints(
-            inst,
-            constraints.outs,
-            defs,
-            &mut replace_global_defines,
-            ®s.global,
-        );
+        if let Some(constraints) = constraints {
+            self.program_output_constraints(
+                inst,
+                constraints.outs,
+                defs,
+                &mut replace_global_defines,
+                ®s.global,
+            );
+        }
 
         // Finally, we've fully programmed the constraint solver.
         // We expect a quick solution in most cases.
@@ -440,12 +447,14 @@ impl<'a> Context<'a> {
 
         // Tied defs are not part of the solution above.
         // Copy register assignments from tied inputs to tied outputs.
-        if constraints.tied_ops {
-            for (op, lv) in constraints.outs.iter().zip(defs) {
-                if let ConstraintKind::Tied(num) = op.kind {
-                    let arg = self.cur.func.dfg.inst_args(inst)[num as usize];
-                    let reg = self.divert.reg(arg, &self.cur.func.locations);
-                    self.cur.func.locations[lv.value] = ValueLoc::Reg(reg);
+        if let Some(constraints) = constraints {
+            if constraints.tied_ops {
+                for (op, lv) in constraints.outs.iter().zip(defs) {
+                    if let ConstraintKind::Tied(num) = op.kind {
+                        let arg = self.cur.func.dfg.inst_args(inst)[num as usize];
+                        let reg = self.divert.reg(arg, &self.cur.func.locations);
+                        self.cur.func.locations[lv.value] = ValueLoc::Reg(reg);
+                    }
                 }
             }
         }
diff --git a/lib/codegen/src/regalloc/reload.rs b/lib/codegen/src/regalloc/reload.rs
index 22a6ac5963..c767993578 100644
--- a/lib/codegen/src/regalloc/reload.rs
+++ b/lib/codegen/src/regalloc/reload.rs
@@ -125,8 +125,8 @@ impl<'a> Context<'a> {
 
         // visit_ebb_header() places us at the first interesting instruction in the EBB.
         while let Some(inst) = self.cur.current_inst() {
-            let encoding = self.cur.func.encodings[inst];
-            if encoding.is_legal() {
+            if !self.cur.func.dfg[inst].opcode().is_ghost() {
+                let encoding = self.cur.func.encodings[inst];
                 self.visit_inst(ebb, inst, encoding, tracker);
                 tracker.drop_dead(inst);
             } else {
@@ -200,10 +200,7 @@ impl<'a> Context<'a> {
         self.cur.use_srcloc(inst);
 
         // Get the operand constraints for `inst` that we are trying to satisfy.
-        let constraints = self
-            .encinfo
-            .operand_constraints(encoding)
-            .expect("Missing instruction encoding");
+        let constraints = self.encinfo.operand_constraints(encoding);
 
         // Identify reload candidates.
         debug_assert!(self.candidates.is_empty());
@@ -240,27 +237,32 @@ impl<'a> Context<'a> {
         // v2 = spill v7
         //
         // That way, we don't need to rewrite all future uses of v2.
-        for (lv, op) in defs.iter().zip(constraints.outs) {
-            if lv.affinity.is_stack() && op.kind != ConstraintKind::Stack {
-                if let InstructionData::Unary {
-                    opcode: Opcode::Copy,
-                    arg,
-                } = self.cur.func.dfg[inst]
-                {
-                    self.cur.func.dfg.replace(inst).spill(arg);
-                    let ok = self.cur.func.update_encoding(inst, self.cur.isa).is_ok();
-                    debug_assert!(ok);
-                } else {
-                    let value_type = self.cur.func.dfg.value_type(lv.value);
-                    let reg = self.cur.func.dfg.replace_result(lv.value, value_type);
-                    self.liveness.create_dead(reg, inst, Affinity::new(op));
-                    self.insert_spill(ebb, lv.value, reg);
+        if let Some(constraints) = constraints {
+            for (lv, op) in defs.iter().zip(constraints.outs) {
+                if lv.affinity.is_stack() && op.kind != ConstraintKind::Stack {
+                    if let InstructionData::Unary {
+                        opcode: Opcode::Copy,
+                        arg,
+                    } = self.cur.func.dfg[inst]
+                    {
+                        self.cur.func.dfg.replace(inst).spill(arg);
+                        let ok = self.cur.func.update_encoding(inst, self.cur.isa).is_ok();
+                        debug_assert!(ok);
+                    } else {
+                        let value_type = self.cur.func.dfg.value_type(lv.value);
+                        let reg = self.cur.func.dfg.replace_result(lv.value, value_type);
+                        self.liveness.create_dead(reg, inst, Affinity::new(op));
+                        self.insert_spill(ebb, lv.value, reg);
+                    }
                 }
             }
         }
 
         // Same thing for spilled call return values.
-        let retvals = &defs[constraints.outs.len()..];
+        let retvals = &defs[self.cur.func.dfg[inst]
+                                .opcode()
+                                .constraints()
+                                .fixed_results()..];
         if !retvals.is_empty() {
             let sig = self
                 .cur
@@ -342,21 +344,26 @@ impl<'a> Context<'a> {
     // Find reload candidates for `inst` and add them to `self.candidates`.
     //
     // These are uses of spilled values where the operand constraint requires a register.
-    fn find_candidates(&mut self, inst: Inst, constraints: &RecipeConstraints) {
+    fn find_candidates(&mut self, inst: Inst, constraints: Option<&RecipeConstraints>) {
         let args = self.cur.func.dfg.inst_args(inst);
 
-        for (argidx, (op, &arg)) in constraints.ins.iter().zip(args).enumerate() {
-            if op.kind != ConstraintKind::Stack && self.liveness[arg].affinity.is_stack() {
-                self.candidates.push(ReloadCandidate {
-                    argidx,
-                    value: arg,
-                    regclass: op.regclass,
-                })
+        if let Some(constraints) = constraints {
+            for (argidx, (op, &arg)) in constraints.ins.iter().zip(args).enumerate() {
+                if op.kind != ConstraintKind::Stack && self.liveness[arg].affinity.is_stack() {
+                    self.candidates.push(ReloadCandidate {
+                        argidx,
+                        value: arg,
+                        regclass: op.regclass,
+                    })
+                }
             }
         }
 
         // If we only have the fixed arguments, we're done now.
-        let offset = constraints.ins.len();
+        let offset = self.cur.func.dfg[inst]
+            .opcode()
+            .constraints()
+            .fixed_value_arguments();
         if args.len() == offset {
             return;
         }

From 26da67b39461c8f88f27819c158ef80820fe3519 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 6 Nov 2018 10:38:29 -0800
Subject: [PATCH 2173/3084] Add a verifier check that ghost instructions don't
 have encodings.

---
 lib/codegen/src/verifier/mod.rs | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 6fedcc5af2..31a7dfc029 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -1550,6 +1550,15 @@ impl<'a> Verifier<'a> {
 
         let encoding = self.func.encodings[inst];
         if encoding.is_legal() {
+            if self.func.dfg[inst].opcode().is_ghost() {
+                return nonfatal!(
+                    errors,
+                    inst,
+                    "Ghost instruction has an encoding: {}",
+                    isa.encoding_info().display(encoding)
+                );
+            }
+
             let mut encodings = isa
                 .legal_encodings(
                     &self.func,

From 997424a4c5805cbf488164c1afe16d757a1706e1 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 6 Nov 2018 10:44:28 -0800
Subject: [PATCH 2174/3084] Add more comments explaining ghost instructions.

---
 lib/codegen/meta-python/cdsl/instructions.py | 3 ++-
 lib/codegen/src/regalloc/coloring.rs         | 4 +++-
 lib/codegen/src/regalloc/reload.rs           | 4 ++++
 3 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/lib/codegen/meta-python/cdsl/instructions.py b/lib/codegen/meta-python/cdsl/instructions.py
index 30bdc306e6..5b2a04d7be 100644
--- a/lib/codegen/meta-python/cdsl/instructions.py
+++ b/lib/codegen/meta-python/cdsl/instructions.py
@@ -92,7 +92,8 @@ class Instruction(object):
     :param is_indirect_branch: This is an indirect branch instruction.
     :param is_call: This is a call instruction.
     :param is_return: This is a return instruction.
-    :param is_ghost: This is a ghost instruction.
+    :param is_ghost: This is a ghost instruction, which has no encoding and no
+                     other register allocation constraints.
     :param can_trap: This instruction can trap.
     :param can_load: This instruction can load from memory.
     :param can_store: This instruction can store to memory.
diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs
index f0b96e79c9..5e77a42f94 100644
--- a/lib/codegen/src/regalloc/coloring.rs
+++ b/lib/codegen/src/regalloc/coloring.rs
@@ -166,6 +166,8 @@ impl<'a> Context<'a> {
         while let Some(inst) = self.cur.next_inst() {
             self.cur.use_srcloc(inst);
             if !self.cur.func.dfg[inst].opcode().is_ghost() {
+                // This is an instruction which either has an encoding or carries ABI-related
+                // register allocation constraints.
                 let enc = self.cur.func.encodings[inst];
                 let constraints = self.encinfo.operand_constraints(enc);
                 if self.visit_inst(inst, constraints, tracker, &mut regs) {
@@ -175,7 +177,7 @@ impl<'a> Context<'a> {
                     self.cur.goto_inst(inst);
                 }
             } else {
-                // This is a ghost instruction with no encoding.
+                // This is a ghost instruction with no encoding and no extra constraints.
                 let (_throughs, kills) = tracker.process_ghost(inst);
                 self.process_ghost_kills(kills, &mut regs);
             }
diff --git a/lib/codegen/src/regalloc/reload.rs b/lib/codegen/src/regalloc/reload.rs
index c767993578..e22fefc0fc 100644
--- a/lib/codegen/src/regalloc/reload.rs
+++ b/lib/codegen/src/regalloc/reload.rs
@@ -126,10 +126,14 @@ impl<'a> Context<'a> {
         // visit_ebb_header() places us at the first interesting instruction in the EBB.
         while let Some(inst) = self.cur.current_inst() {
             if !self.cur.func.dfg[inst].opcode().is_ghost() {
+                // This instruction either has an encoding or has ABI constraints, so visit it to
+                // insert spills and fills as needed.
                 let encoding = self.cur.func.encodings[inst];
                 self.visit_inst(ebb, inst, encoding, tracker);
                 tracker.drop_dead(inst);
             } else {
+                // This is a ghost instruction with no encoding and no extra constraints, so we can
+                // just skip over it.
                 self.cur.next_inst();
             }
         }

From bdcc06eb15ad86c5725176b8b2cd6ad28b6c3e20 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 26 Oct 2018 06:39:30 -0700
Subject: [PATCH 2175/3084] When lowering br_table to branches, delete the old
 jump tables.

This prevents codegen from emitting unused readonly data.
---
 .../filetests/legalizer/br_table_cond.clif    | 40 +++++++++++++++++++
 lib/codegen/src/legalizer/mod.rs              |  5 +++
 2 files changed, 45 insertions(+)
 create mode 100644 cranelift/filetests/legalizer/br_table_cond.clif

diff --git a/cranelift/filetests/legalizer/br_table_cond.clif b/cranelift/filetests/legalizer/br_table_cond.clif
new file mode 100644
index 0000000000..5b7beb3253
--- /dev/null
+++ b/cranelift/filetests/legalizer/br_table_cond.clif
@@ -0,0 +1,40 @@
+test legalizer
+set probestack_enabled=false
+set jump_tables_enabled=false
+target x86_64
+
+; Test that when jump_tables_enables is false, all jump tables are eliminated.
+
+function u0:0(i64 vmctx) baldrdash {
+    gv0 = vmctx
+    gv1 = iadd_imm.i64 gv0, 48
+    jt0 = jump_table [ebb2, ebb2, ebb7]
+    jt1 = jump_table [ebb8, ebb8]
+
+ebb0(v0: i64):
+    jump ebb5
+
+ebb5:
+    v1 = global_value.i64 gv1
+    v2 = load.i64 v1
+    trapnz v2, interrupt
+    v3 = iconst.i32 0
+    br_table v3, ebb3, jt0
+
+ebb7:
+    v4 = iconst.i32 0
+    br_table v4, ebb3, jt1
+
+ebb8:
+    jump ebb5
+
+ebb3:
+    jump ebb2
+
+ebb2:
+    jump ebb1
+
+ebb1:
+    fallthrough_return
+}
+; not: jump_table
diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs
index e52889c672..b552934289 100644
--- a/lib/codegen/src/legalizer/mod.rs
+++ b/lib/codegen/src/legalizer/mod.rs
@@ -110,6 +110,11 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
             }
         }
     }
+
+    // Now that we've lowered all br_tables, we don't need the jump tables anymore.
+    if !isa.flags().jump_tables_enabled() {
+        pos.func.jump_tables.clear();
+    }
 }
 
 // Include legalization patterns that were generated by `gen_legalizer.py` from the `XForms` in

From 3409af7c0730dcf8b6a9e1a7d9ac105164e47b82 Mon Sep 17 00:00:00 2001
From: Lachlan Sneff 
Date: Wed, 7 Nov 2018 18:59:29 -0500
Subject: [PATCH 2176/3084] Add early-stage optimization crate (#556)

* Add simple constant folding and folding tests
---
 cranelift/Cargo.toml                          |   1 +
 cranelift/filetests/preopt/branch.clif        |  54 ++++
 cranelift/filetests/preopt/numerical.clif     |  36 +++
 .../div_by_const_indirect.clif                |   2 +-
 .../div_by_const_non_power_of_2.clif          |   2 +-
 .../div_by_const_power_of_2.clif              |   2 +-
 .../rem_by_const_non_power_of_2.clif          |   2 +-
 .../rem_by_const_power_of_2.clif              |   2 +-
 .../{preopt => simple_preopt}/simplify.clif   |   2 +-
 lib/codegen/src/context.rs                    |   2 +-
 lib/codegen/src/lib.rs                        |   2 +-
 .../src/{preopt.rs => simple_preopt.rs}       |   0
 lib/codegen/src/timing.rs                     |  15 +-
 lib/filetests/Cargo.toml                      |   1 +
 lib/filetests/src/lib.rs                      |   5 +-
 lib/filetests/src/test_preopt.rs              |  22 +-
 lib/filetests/src/test_simple_preopt.rs       |  44 +++
 lib/preopt/Cargo.toml                         |  27 ++
 lib/preopt/LICENSE                            | 219 +++++++++++++++
 lib/preopt/README.md                          |   1 +
 lib/preopt/src/constant_folding.rs            | 254 ++++++++++++++++++
 lib/preopt/src/lib.rs                         |  71 +++++
 22 files changed, 745 insertions(+), 21 deletions(-)
 create mode 100644 cranelift/filetests/preopt/branch.clif
 create mode 100644 cranelift/filetests/preopt/numerical.clif
 rename cranelift/filetests/{preopt => simple_preopt}/div_by_const_indirect.clif (98%)
 rename cranelift/filetests/{preopt => simple_preopt}/div_by_const_non_power_of_2.clif (99%)
 rename cranelift/filetests/{preopt => simple_preopt}/div_by_const_power_of_2.clif (99%)
 rename cranelift/filetests/{preopt => simple_preopt}/rem_by_const_non_power_of_2.clif (99%)
 rename cranelift/filetests/{preopt => simple_preopt}/rem_by_const_power_of_2.clif (99%)
 rename cranelift/filetests/{preopt => simple_preopt}/simplify.clif (98%)
 rename lib/codegen/src/{preopt.rs => simple_preopt.rs} (100%)
 create mode 100644 lib/filetests/src/test_simple_preopt.rs
 create mode 100644 lib/preopt/Cargo.toml
 create mode 100644 lib/preopt/LICENSE
 create mode 100644 lib/preopt/README.md
 create mode 100644 lib/preopt/src/constant_folding.rs
 create mode 100644 lib/preopt/src/lib.rs

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 51a714af46..e347ce3efc 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -25,6 +25,7 @@ cranelift-filetests = { path = "lib/filetests", version = "0.22.0" }
 cranelift-module = { path = "lib/module", version = "0.22.0" }
 cranelift-faerie = { path = "lib/faerie", version = "0.22.0" }
 cranelift-simplejit = { path = "lib/simplejit", version = "0.22.0" }
+cranelift-preopt = { path = "lib/preopt", version = "0.22.0" }
 cranelift = { path = "lib/umbrella", version = "0.22.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
diff --git a/cranelift/filetests/preopt/branch.clif b/cranelift/filetests/preopt/branch.clif
new file mode 100644
index 0000000000..2b696fa2ef
--- /dev/null
+++ b/cranelift/filetests/preopt/branch.clif
@@ -0,0 +1,54 @@
+test preopt
+target x86_64
+
+function %brz_fold() -> i32 {
+ebb0:
+    v0 = bconst.b1 false
+    brz v0, ebb2
+    jump ebb1
+ebb1:
+    v1 = iconst.i32 42
+    return v1
+ebb2:
+    v2 = iconst.i32 24
+    return v2
+}
+; sameln: function %brz_fold
+; nextln: ebb0:
+; nextln:     v0 = bconst.b1 false
+; nextln:     jump ebb2
+; nextln: 
+; nextln: ebb1:
+; nextln:     v1 = iconst.i32 42
+; nextln:     return v1
+; nextln: 
+; nextln: ebb2:
+; nextln:     v2 = iconst.i32 24
+; nextln:     return v2
+; nextln: }
+
+function %brnz_fold() -> i32 {
+ebb0:
+    v0 = bconst.b1 true
+    brnz v0, ebb2
+    jump ebb1
+ebb1:
+    v1 = iconst.i32 42
+    return v1
+ebb2:
+    v2 = iconst.i32 24
+    return v2
+}
+; sameln: function %brnz_fold
+; nextln: ebb0:
+; nextln:     v0 = bconst.b1 true
+; nextln:     jump ebb2
+; nextln: 
+; nextln: ebb1:
+; nextln:     v1 = iconst.i32 42
+; nextln:     return v1
+; nextln: 
+; nextln: ebb2:
+; nextln:     v2 = iconst.i32 24
+; nextln:     return v2
+; nextln: }
diff --git a/cranelift/filetests/preopt/numerical.clif b/cranelift/filetests/preopt/numerical.clif
new file mode 100644
index 0000000000..27fdaec0f7
--- /dev/null
+++ b/cranelift/filetests/preopt/numerical.clif
@@ -0,0 +1,36 @@
+test preopt
+target x86_64
+
+function %iadd_fold() -> i32 {
+ebb0:
+    v0 = iconst.i32 37
+    v1 = iconst.i32 5
+    v2 = iadd v0, v1
+    v3 = iconst.i32 8
+    v4 = iadd v2, v3
+    return v4
+}
+; sameln: function %iadd_fold
+; nextln: ebb0:
+; nextln:     v0 = iconst.i32 37
+; nextln:     v1 = iconst.i32 5
+; nextln:     v2 = iconst.i32 42
+; nextln:     v3 = iconst.i32 8
+; nextln:     v4 = iconst.i32 50
+; nextln:     return v4
+; nextln: }
+
+function %isub_fold() -> i32 {
+ebb0:
+    v0 = iconst.i32 42
+    v1 = iconst.i32 1
+    v2 = isub v0, v1
+    return v2
+}
+; sameln: function %isub_fold
+; nextln: ebb0:
+; nextln:     v0 = iconst.i32 42
+; nextln:     v1 = iconst.i32 1
+; nextln:     v2 = iconst.i32 41
+; nextln:     return v2
+; nextln: }
\ No newline at end of file
diff --git a/cranelift/filetests/preopt/div_by_const_indirect.clif b/cranelift/filetests/simple_preopt/div_by_const_indirect.clif
similarity index 98%
rename from cranelift/filetests/preopt/div_by_const_indirect.clif
rename to cranelift/filetests/simple_preopt/div_by_const_indirect.clif
index 5d3266d26a..fa66337fc2 100644
--- a/cranelift/filetests/preopt/div_by_const_indirect.clif
+++ b/cranelift/filetests/simple_preopt/div_by_const_indirect.clif
@@ -1,4 +1,4 @@
-test preopt
+test simple_preopt
 target i686 baseline
 
 ; Cases where the denominator is created by an iconst
diff --git a/cranelift/filetests/preopt/div_by_const_non_power_of_2.clif b/cranelift/filetests/simple_preopt/div_by_const_non_power_of_2.clif
similarity index 99%
rename from cranelift/filetests/preopt/div_by_const_non_power_of_2.clif
rename to cranelift/filetests/simple_preopt/div_by_const_non_power_of_2.clif
index 1d05d4a53c..fa0ac41bff 100644
--- a/cranelift/filetests/preopt/div_by_const_non_power_of_2.clif
+++ b/cranelift/filetests/simple_preopt/div_by_const_non_power_of_2.clif
@@ -1,4 +1,4 @@
-test preopt
+test simple_preopt
 target i686 baseline
 
 ; -------- U32 --------
diff --git a/cranelift/filetests/preopt/div_by_const_power_of_2.clif b/cranelift/filetests/simple_preopt/div_by_const_power_of_2.clif
similarity index 99%
rename from cranelift/filetests/preopt/div_by_const_power_of_2.clif
rename to cranelift/filetests/simple_preopt/div_by_const_power_of_2.clif
index a047107c26..5a959750b9 100644
--- a/cranelift/filetests/preopt/div_by_const_power_of_2.clif
+++ b/cranelift/filetests/simple_preopt/div_by_const_power_of_2.clif
@@ -1,4 +1,4 @@
-test preopt
+test simple_preopt
 target i686 baseline
 
 ; -------- U32 --------
diff --git a/cranelift/filetests/preopt/rem_by_const_non_power_of_2.clif b/cranelift/filetests/simple_preopt/rem_by_const_non_power_of_2.clif
similarity index 99%
rename from cranelift/filetests/preopt/rem_by_const_non_power_of_2.clif
rename to cranelift/filetests/simple_preopt/rem_by_const_non_power_of_2.clif
index f440aa4a6e..40c5e1d828 100644
--- a/cranelift/filetests/preopt/rem_by_const_non_power_of_2.clif
+++ b/cranelift/filetests/simple_preopt/rem_by_const_non_power_of_2.clif
@@ -1,4 +1,4 @@
-test preopt
+test simple_preopt
 target i686 baseline
 
 ; -------- U32 --------
diff --git a/cranelift/filetests/preopt/rem_by_const_power_of_2.clif b/cranelift/filetests/simple_preopt/rem_by_const_power_of_2.clif
similarity index 99%
rename from cranelift/filetests/preopt/rem_by_const_power_of_2.clif
rename to cranelift/filetests/simple_preopt/rem_by_const_power_of_2.clif
index 70bd1bbd4e..09eebfa684 100644
--- a/cranelift/filetests/preopt/rem_by_const_power_of_2.clif
+++ b/cranelift/filetests/simple_preopt/rem_by_const_power_of_2.clif
@@ -1,4 +1,4 @@
-test preopt
+test simple_preopt
 target i686 baseline
 
 ; -------- U32 --------
diff --git a/cranelift/filetests/preopt/simplify.clif b/cranelift/filetests/simple_preopt/simplify.clif
similarity index 98%
rename from cranelift/filetests/preopt/simplify.clif
rename to cranelift/filetests/simple_preopt/simplify.clif
index 4426b98186..3df0f3355e 100644
--- a/cranelift/filetests/preopt/simplify.clif
+++ b/cranelift/filetests/simple_preopt/simplify.clif
@@ -1,4 +1,4 @@
-test preopt
+test simple_preopt
 target i686
 
 function %iadd_imm(i32) -> i32 {
diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs
index fa9c78d7f5..ffff2e25ab 100644
--- a/lib/codegen/src/context.rs
+++ b/lib/codegen/src/context.rs
@@ -22,11 +22,11 @@ use licm::do_licm;
 use loop_analysis::LoopAnalysis;
 use nan_canonicalization::do_nan_canonicalization;
 use postopt::do_postopt;
-use preopt::do_preopt;
 use regalloc;
 use result::CodegenResult;
 use settings::{FlagsOrIsa, OptLevel};
 use simple_gvn::do_simple_gvn;
+use simple_preopt::do_preopt;
 use std::vec::Vec;
 use timing;
 use unreachable_code::eliminate_unreachable_code;
diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index ff9ca2d073..6ece592ea7 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -104,12 +104,12 @@ mod nan_canonicalization;
 mod partition_slice;
 mod postopt;
 mod predicates;
-mod preopt;
 mod ref_slice;
 mod regalloc;
 mod result;
 mod scoped_hash_map;
 mod simple_gvn;
+mod simple_preopt;
 mod stack_layout;
 mod topo_order;
 mod unreachable_code;
diff --git a/lib/codegen/src/preopt.rs b/lib/codegen/src/simple_preopt.rs
similarity index 100%
rename from lib/codegen/src/preopt.rs
rename to lib/codegen/src/simple_preopt.rs
diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs
index 4ac9e153ed..68ca4850c0 100644
--- a/lib/codegen/src/timing.rs
+++ b/lib/codegen/src/timing.rs
@@ -124,7 +124,7 @@ mod details {
     }
 
     /// Accumulated timing information for a single pass.
-    #[derive(Default)]
+    #[derive(Default, Copy, Clone)]
     struct PassTime {
         /// Total time spent running this pass including children.
         total: Duration,
@@ -134,17 +134,24 @@ mod details {
     }
 
     /// Accumulated timing for all passes.
-    #[derive(Default)]
     pub struct PassTimes {
         pass: [PassTime; NUM_PASSES],
     }
 
+    impl Default for PassTimes {
+        fn default() -> Self {
+            PassTimes {
+                pass: [Default::default(); NUM_PASSES],
+            }
+        }
+    }
+
     impl fmt::Display for PassTimes {
         fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
             writeln!(f, "======== ========  ==================================")?;
             writeln!(f, "   Total     Self  Pass")?;
             writeln!(f, "-------- --------  ----------------------------------")?;
-            for (time, desc) in self.pass.iter().zip(&DESCRIPTIONS) {
+            for (time, desc) in self.pass.iter().zip(&DESCRIPTIONS[..]) {
                 // Omit passes that haven't run.
                 if time.total == Duration::default() {
                     continue;
@@ -212,7 +219,7 @@ mod details {
     /// Add `timings` to the accumulated timings for the current thread.
     pub fn add_to_current(times: &PassTimes) {
         PASS_TIME.with(|rc| {
-            for (a, b) in rc.borrow_mut().pass.iter_mut().zip(×.pass) {
+            for (a, b) in rc.borrow_mut().pass.iter_mut().zip(×.pass[..]) {
                 a.total += b.total;
                 a.child += b.child;
             }
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 1ed2daf9e0..c3f587e569 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -11,6 +11,7 @@ publish = false
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.22.0", features = ["testing_hooks"] }
 cranelift-reader = { path = "../reader", version = "0.22.0" }
+cranelift-preopt = { path = "../preopt", version = "0.22.0" }
 file-per-thread-logger = "0.1.1"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index 56c7e66ff3..3a6581d602 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -24,6 +24,7 @@
 )]
 
 extern crate cranelift_codegen;
+extern crate cranelift_preopt;
 extern crate cranelift_reader;
 extern crate file_per_thread_logger;
 extern crate filecheck;
@@ -56,6 +57,7 @@ mod test_print_cfg;
 mod test_regalloc;
 mod test_shrink;
 mod test_simple_gvn;
+mod test_simple_preopt;
 mod test_verifier;
 
 /// The result of running the test in a file.
@@ -127,12 +129,13 @@ fn new_subtest(parsed: &TestCommand) -> subtest::SubtestResult test_legalizer::subtest(parsed),
         "licm" => test_licm::subtest(parsed),
         "postopt" => test_postopt::subtest(parsed),
-        "preopt" => test_preopt::subtest(parsed),
+        "simple_preopt" => test_simple_preopt::subtest(parsed),
         "print-cfg" => test_print_cfg::subtest(parsed),
         "regalloc" => test_regalloc::subtest(parsed),
         "shrink" => test_shrink::subtest(parsed),
         "simple-gvn" => test_simple_gvn::subtest(parsed),
         "verifier" => test_verifier::subtest(parsed),
+        "preopt" => test_preopt::subtest(parsed),
         _ => Err(format!("unknown test command '{}'", parsed.command)),
     }
 }
diff --git a/lib/filetests/src/test_preopt.rs b/lib/filetests/src/test_preopt.rs
index 86f8f87de7..43566bdfa5 100644
--- a/lib/filetests/src/test_preopt.rs
+++ b/lib/filetests/src/test_preopt.rs
@@ -1,10 +1,14 @@
-//! Test command for testing the preopt pass.
+//! Test command for testing the constant folding pass.
+//!
+//! The `dce` test command runs each function through the constant folding pass after ensuring
+//! that all instructions are legal for the target.
 //!
 //! The resulting function is sent to `filecheck`.
 
 use cranelift_codegen;
 use cranelift_codegen::ir::Function;
 use cranelift_codegen::print_errors::pretty_error;
+use cranelift_preopt::optimize;
 use cranelift_reader::TestCommand;
 use std::borrow::Cow;
 use subtest::{run_filecheck, Context, SubTest, SubtestResult};
@@ -29,16 +33,18 @@ impl SubTest for TestPreopt {
         true
     }
 
-    fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> {
-        let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned());
-        let isa = context.isa.expect("preopt needs an ISA");
+    fn needs_isa(&self) -> bool {
+        true
+    }
 
-        comp_ctx.flowgraph();
-        comp_ctx
-            .preopt(isa)
+    fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> {
+        let isa = context.isa.expect("compile needs an ISA");
+        let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned());
+
+        optimize(&mut comp_ctx, isa)
             .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?;
 
-        let text = &comp_ctx.func.display(isa).to_string();
+        let text = comp_ctx.func.display(context.isa).to_string();
         run_filecheck(&text, context)
     }
 }
diff --git a/lib/filetests/src/test_simple_preopt.rs b/lib/filetests/src/test_simple_preopt.rs
new file mode 100644
index 0000000000..803aa944e7
--- /dev/null
+++ b/lib/filetests/src/test_simple_preopt.rs
@@ -0,0 +1,44 @@
+//! Test command for testing the preopt pass.
+//!
+//! The resulting function is sent to `filecheck`.
+
+use cranelift_codegen;
+use cranelift_codegen::ir::Function;
+use cranelift_codegen::print_errors::pretty_error;
+use cranelift_reader::TestCommand;
+use std::borrow::Cow;
+use subtest::{run_filecheck, Context, SubTest, SubtestResult};
+
+struct TestSimplePreopt;
+
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+    assert_eq!(parsed.command, "simple_preopt");
+    if !parsed.options.is_empty() {
+        Err(format!("No options allowed on {}", parsed))
+    } else {
+        Ok(Box::new(TestSimplePreopt))
+    }
+}
+
+impl SubTest for TestSimplePreopt {
+    fn name(&self) -> &'static str {
+        "simple_preopt"
+    }
+
+    fn is_mutating(&self) -> bool {
+        true
+    }
+
+    fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> {
+        let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned());
+        let isa = context.isa.expect("preopt needs an ISA");
+
+        comp_ctx.flowgraph();
+        comp_ctx
+            .preopt(isa)
+            .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?;
+
+        let text = &comp_ctx.func.display(isa).to_string();
+        run_filecheck(&text, context)
+    }
+}
diff --git a/lib/preopt/Cargo.toml b/lib/preopt/Cargo.toml
new file mode 100644
index 0000000000..d88ce942bc
--- /dev/null
+++ b/lib/preopt/Cargo.toml
@@ -0,0 +1,27 @@
+[package]
+authors = ["The Cranelift Project Developers"]
+name = "cranelift-preopt"
+version = "0.22.0"
+description = "Support for optimizations in Cranelift"
+license = "Apache-2.0 WITH LLVM-exception"
+documentation = "https://cranelift.readthedocs.io/"
+repository = "https://github.com/CraneStation/cranelift"
+categories = ["no_std"]
+readme = "README.md"
+keywords = ["optimize", "compile", "compiler", "jit"]
+
+[dependencies]
+cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
+# This is commented out because it doesn't build on Rust 1.25.0, which
+# cranelift currently supports.
+# rustc_apfloat = { version = "0.1.2", default-features = false }
+
+[features]
+default = ["std"]
+std = ["cranelift-codegen/std", "cranelift-entity/std"]
+core = ["cranelift-codegen/core"]
+
+[badges]
+maintenance = { status = "experimental" }
+travis-ci = { repository = "CraneStation/cranelift" }
\ No newline at end of file
diff --git a/lib/preopt/LICENSE b/lib/preopt/LICENSE
new file mode 100644
index 0000000000..be1d7c438a
--- /dev/null
+++ b/lib/preopt/LICENSE
@@ -0,0 +1,219 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
diff --git a/lib/preopt/README.md b/lib/preopt/README.md
new file mode 100644
index 0000000000..12608b7adb
--- /dev/null
+++ b/lib/preopt/README.md
@@ -0,0 +1 @@
+This crate performes early-stage optimizations on [Cranelift](https://crates.io/crates/cranelift) IR.
\ No newline at end of file
diff --git a/lib/preopt/src/constant_folding.rs b/lib/preopt/src/constant_folding.rs
new file mode 100644
index 0000000000..1fb5d116eb
--- /dev/null
+++ b/lib/preopt/src/constant_folding.rs
@@ -0,0 +1,254 @@
+//! Fold operations on constants at compile time.
+
+use cranelift_codegen::{
+    cursor::{Cursor, FuncCursor},
+    ir::{self, InstBuilder},
+};
+// use rustc_apfloat::{
+//     ieee::{Double, Single},
+//     Float,
+// };
+
+enum ConstImm {
+    Bool(bool),
+    I64(i64),
+    Ieee32(f32), // Ieee32 and Ieee64 will be replaced with `Single` and `Double` from the rust_apfloat library eventually.
+    Ieee64(f64),
+}
+
+impl ConstImm {
+    fn unwrap_i64(self) -> i64 {
+        if let ConstImm::I64(imm) = self {
+            imm
+        } else {
+            panic!("self did not contain an `i64`.")
+        }
+    }
+
+    fn evaluate_truthiness(self) -> bool {
+        match self {
+            ConstImm::Bool(b) => b,
+            ConstImm::I64(imm) => imm != 0,
+            _ => panic!(
+                "Only a `ConstImm::Bool` and `ConstImm::I64` can be evaluated for \"truthiness\""
+            ),
+        }
+    }
+}
+
+/// Fold operations on constants.
+///
+/// It's important to note that this will not remove unused constants. It's
+/// assumed that the DCE pass will take care of them.
+pub fn fold_constants(func: &mut ir::Function) {
+    let mut pos = FuncCursor::new(func);
+
+    while let Some(_ebb) = pos.next_ebb() {
+        while let Some(inst) = pos.next_inst() {
+            use self::ir::InstructionData::*;
+            match pos.func.dfg[inst] {
+                Binary { opcode, args } => {
+                    fold_binary(&mut pos.func.dfg, inst, opcode, args);
+                }
+                Unary { opcode, arg } => {
+                    fold_unary(&mut pos.func.dfg, inst, opcode, arg);
+                }
+                Branch {
+                    opcode,
+                    args: _,
+                    destination: _,
+                } => {
+                    fold_branch(&mut pos, inst, opcode);
+                }
+                _ => {}
+            }
+        }
+    }
+}
+
+fn resolve_value_to_imm(dfg: &ir::DataFlowGraph, value: ir::Value) -> Option {
+    let original = dfg.resolve_aliases(value);
+
+    let inst = dfg.value_def(original).unwrap_inst();
+
+    use self::ir::{InstructionData::*, Opcode::*};
+    match dfg[inst] {
+        UnaryImm {
+            opcode: Iconst,
+            imm,
+        } => Some(ConstImm::I64(imm.into())),
+        UnaryIeee32 {
+            opcode: F32const,
+            imm,
+        } => {
+            // See https://doc.rust-lang.org/std/primitive.f32.html#method.from_bits for caveats.
+            let ieee_f32 = f32::from_bits(imm.bits());
+            Some(ConstImm::Ieee32(ieee_f32))
+        }
+        UnaryIeee64 {
+            opcode: F64const,
+            imm,
+        } => {
+            // See https://doc.rust-lang.org/std/primitive.f32.html#method.from_bits for caveats.
+            let ieee_f64 = f64::from_bits(imm.bits());
+            Some(ConstImm::Ieee64(ieee_f64))
+        }
+        UnaryBool {
+            opcode: Bconst,
+            imm,
+        } => Some(ConstImm::Bool(imm)),
+        _ => None,
+    }
+}
+
+fn evaluate_binary(opcode: ir::Opcode, imm0: ConstImm, imm1: ConstImm) -> Option {
+    use std::num::Wrapping;
+
+    match opcode {
+        ir::Opcode::Iadd => {
+            let imm0 = Wrapping(imm0.unwrap_i64());
+            let imm1 = Wrapping(imm1.unwrap_i64());
+            Some(ConstImm::I64((imm0 + imm1).0))
+        }
+        ir::Opcode::Isub => {
+            let imm0 = Wrapping(imm0.unwrap_i64());
+            let imm1 = Wrapping(imm1.unwrap_i64());
+            Some(ConstImm::I64((imm0 - imm1).0))
+        }
+        ir::Opcode::Imul => {
+            let imm0 = Wrapping(imm0.unwrap_i64());
+            let imm1 = Wrapping(imm1.unwrap_i64());
+            Some(ConstImm::I64((imm0 * imm1).0))
+        }
+        ir::Opcode::Udiv => {
+            let imm0 = Wrapping(imm0.unwrap_i64());
+            let imm1 = Wrapping(imm1.unwrap_i64());
+            if imm1.0 == 0 {
+                panic!("Cannot divide by a zero.")
+            }
+            Some(ConstImm::I64((imm0 / imm1).0))
+        }
+        ir::Opcode::Fadd => match (imm0, imm1) {
+            (ConstImm::Ieee32(imm0), ConstImm::Ieee32(imm1)) => Some(ConstImm::Ieee32(imm0 + imm1)),
+            (ConstImm::Ieee64(imm0), ConstImm::Ieee64(imm1)) => Some(ConstImm::Ieee64(imm0 + imm1)),
+            _ => unreachable!(),
+        },
+        ir::Opcode::Fsub => match (imm0, imm1) {
+            (ConstImm::Ieee32(imm0), ConstImm::Ieee32(imm1)) => Some(ConstImm::Ieee32(imm0 - imm1)),
+            (ConstImm::Ieee64(imm0), ConstImm::Ieee64(imm1)) => Some(ConstImm::Ieee64(imm0 - imm1)),
+            _ => unreachable!(),
+        },
+        ir::Opcode::Fmul => match (imm0, imm1) {
+            (ConstImm::Ieee32(imm0), ConstImm::Ieee32(imm1)) => Some(ConstImm::Ieee32(imm0 * imm1)),
+            (ConstImm::Ieee64(imm0), ConstImm::Ieee64(imm1)) => Some(ConstImm::Ieee64(imm0 * imm1)),
+            _ => unreachable!(),
+        },
+        ir::Opcode::Fdiv => match (imm0, imm1) {
+            (ConstImm::Ieee32(imm0), ConstImm::Ieee32(imm1)) => Some(ConstImm::Ieee32(imm0 / imm1)),
+            (ConstImm::Ieee64(imm0), ConstImm::Ieee64(imm1)) => Some(ConstImm::Ieee64(imm0 / imm1)),
+            _ => unreachable!(),
+        },
+        _ => None,
+    }
+}
+
+fn evaluate_unary(opcode: ir::Opcode, imm: ConstImm) -> Option {
+    match opcode {
+        ir::Opcode::Fneg => match imm {
+            ConstImm::Ieee32(imm) => Some(ConstImm::Ieee32(-imm)),
+            ConstImm::Ieee64(imm) => Some(ConstImm::Ieee64(-imm)),
+            _ => unreachable!(),
+        },
+        ir::Opcode::Fabs => match imm {
+            ConstImm::Ieee32(imm) => Some(ConstImm::Ieee32(imm.abs())),
+            ConstImm::Ieee64(imm) => Some(ConstImm::Ieee64(imm.abs())),
+            _ => unreachable!(),
+        },
+        _ => None,
+    }
+}
+
+fn replace_inst(dfg: &mut ir::DataFlowGraph, inst: ir::Inst, const_imm: ConstImm) {
+    use self::ConstImm::*;
+    match const_imm {
+        I64(imm) => {
+            let typevar = dfg.ctrl_typevar(inst);
+            dfg.replace(inst).iconst(typevar, imm);
+        }
+        Ieee32(imm) => {
+            dfg.replace(inst)
+                .f32const(ir::immediates::Ieee32::with_bits(imm.to_bits()));
+        }
+        Ieee64(imm) => {
+            dfg.replace(inst)
+                .f64const(ir::immediates::Ieee64::with_bits(imm.to_bits()));
+        }
+        Bool(imm) => {
+            let typevar = dfg.ctrl_typevar(inst);
+            dfg.replace(inst).bconst(typevar, imm);
+        }
+    }
+}
+
+/// Fold a binary instruction.
+fn fold_binary(
+    dfg: &mut ir::DataFlowGraph,
+    inst: ir::Inst,
+    opcode: ir::Opcode,
+    args: [ir::Value; 2],
+) {
+    let (imm0, imm1) = if let (Some(imm0), Some(imm1)) = (
+        resolve_value_to_imm(dfg, args[0]),
+        resolve_value_to_imm(dfg, args[1]),
+    ) {
+        (imm0, imm1)
+    } else {
+        return;
+    };
+
+    if let Some(const_imm) = evaluate_binary(opcode, imm0, imm1) {
+        replace_inst(dfg, inst, const_imm);
+    }
+}
+
+/// Fold a unary instruction.
+fn fold_unary(dfg: &mut ir::DataFlowGraph, inst: ir::Inst, opcode: ir::Opcode, arg: ir::Value) {
+    let imm = if let Some(imm) = resolve_value_to_imm(dfg, arg) {
+        imm
+    } else {
+        return;
+    };
+
+    if let Some(const_imm) = evaluate_unary(opcode, imm) {
+        replace_inst(dfg, inst, const_imm);
+    }
+}
+
+fn fold_branch(pos: &mut FuncCursor, inst: ir::Inst, opcode: ir::Opcode) {
+    let (cond, ebb, args) = {
+        let values = pos.func.dfg.inst_args(inst);
+        let inst_data = &pos.func.dfg[inst];
+        (
+            resolve_value_to_imm(&pos.func.dfg, values[0]).unwrap(),
+            inst_data.branch_destination().unwrap(),
+            values[1..].to_vec(),
+        )
+    };
+
+    let truthiness = cond.evaluate_truthiness();
+    let branch_if_zero = match opcode {
+        ir::Opcode::Brz => true,
+        ir::Opcode::Brnz => false,
+        _ => unreachable!(),
+    };
+
+    if (branch_if_zero && !truthiness) || (!branch_if_zero && truthiness) {
+        pos.func.dfg.replace(inst).jump(ebb, &args);
+        // remove the rest of the ebb to avoid verifier errors
+        while let Some(next_inst) = pos.func.layout.next_inst(inst) {
+            pos.func.layout.remove_inst(next_inst);
+        }
+    } else {
+        pos.remove_inst_and_step_back();
+    }
+}
diff --git a/lib/preopt/src/lib.rs b/lib/preopt/src/lib.rs
new file mode 100644
index 0000000000..bf2ffc1a8c
--- /dev/null
+++ b/lib/preopt/src/lib.rs
@@ -0,0 +1,71 @@
+//! Performes early-stage optimizations on Cranelift IR.
+
+#![deny(
+    missing_docs,
+    trivial_numeric_casts,
+    unused_extern_crates,
+    unstable_features
+)]
+#![warn(unused_import_braces)]
+#![cfg_attr(
+    feature = "clippy",
+    plugin(clippy(conf_file = "../../clippy.toml"))
+)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    allow(new_without_default, new_without_default_derive)
+)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    warn(
+        float_arithmetic,
+        mut_mut,
+        nonminimal_bool,
+        option_map_unwrap_or,
+        option_map_unwrap_or_else,
+        print_stdout,
+        unicode_not_nfc,
+        use_self
+    )
+)]
+#![cfg_attr(not(feature = "std"), no_std)]
+#![cfg_attr(not(feature = "std"), alloc)]
+
+#[cfg(not(feature = "std"))]
+extern crate alloc;
+
+extern crate cranelift_codegen;
+// extern crate rustc_apfloat;
+
+mod constant_folding;
+
+use cranelift_codegen::{isa::TargetIsa, settings::FlagsOrIsa, CodegenResult, Context};
+
+/// Optimize the function with available optimizations.
+///
+/// Since this can be resource intensive (and code-size inflating),
+/// it is separated from `Context::compile` to allow DCE to remove it
+/// if it's not used.
+pub fn optimize(ctx: &mut Context, isa: &TargetIsa) -> CodegenResult<()> {
+    ctx.verify_if(isa)?;
+    fold_constants(ctx, isa)?;
+
+    Ok(())
+}
+
+/// Fold constants
+pub fn fold_constants<'a, FOI>(ctx: &mut Context, fisa: FOI) -> CodegenResult<()>
+where
+    FOI: Into>,
+{
+    constant_folding::fold_constants(&mut ctx.func);
+    ctx.verify_if(fisa)?;
+    Ok(())
+}
+
+/// This replaces `std` in builds with `core`.
+#[cfg(not(feature = "std"))]
+mod std {
+    pub use alloc::{boxed, slice, string, vec};
+    pub use core::*;
+}

From b476f823d4fa3a2b0a407db4ddebbf1531ce8d01 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 7 Nov 2018 14:25:03 -0800
Subject: [PATCH 2177/3084] Update to target_lexicon 0.2.0.

---
 cranelift/Cargo.toml             |  2 +-
 cranelift/fuzz/Cargo.toml        |  2 +-
 lib/codegen/Cargo.toml           |  2 +-
 lib/codegen/src/isa/call_conv.rs | 26 ++++++--------------------
 lib/faerie/Cargo.toml            |  4 ++--
 lib/frontend/Cargo.toml          |  2 +-
 lib/native/Cargo.toml            |  2 +-
 lib/reader/Cargo.toml            |  2 +-
 lib/simplejit/Cargo.toml         |  2 +-
 lib/wasm/Cargo.toml              |  2 +-
 10 files changed, 16 insertions(+), 30 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index e347ce3efc..e401e954c5 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -33,7 +33,7 @@ serde = "1.0.8"
 term = "0.5.1"
 capstone = { version = "0.5.0", optional = true }
 wabt = { version = "0.7.0", optional = true }
-target-lexicon = "0.0.3"
+target-lexicon = "0.2.0"
 pretty_env_logger = "0.2.4"
 file-per-thread-logger = "0.1.1"
 
diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
index 3c2930ae07..a78d0ed3fc 100644
--- a/cranelift/fuzz/Cargo.toml
+++ b/cranelift/fuzz/Cargo.toml
@@ -14,7 +14,7 @@ libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" }
 cranelift-codegen = { path = "../lib/codegen" }
 cranelift-wasm = { path = "../lib/wasm" }
 cranelift-reader = { path = "../lib/reader" }
-target-lexicon = "0.0.3"
+target-lexicon = "0.2.0"
 
 # Prevent this from interfering with workspaces
 [workspace]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 7c1d2d0e9a..baaa059f3f 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -17,7 +17,7 @@ cranelift-bforest = { path = "../bforest", version = "0.22.0", default-features
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
-target-lexicon = { version = "0.0.3", default-features = false }
+target-lexicon = { version = "0.2.0", default-features = false }
 log = { version = "0.4.4", default-features = false }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
diff --git a/lib/codegen/src/isa/call_conv.rs b/lib/codegen/src/isa/call_conv.rs
index ce5b10b315..0fe242b958 100644
--- a/lib/codegen/src/isa/call_conv.rs
+++ b/lib/codegen/src/isa/call_conv.rs
@@ -1,6 +1,6 @@
 use std::fmt;
 use std::str;
-use target_lexicon::{OperatingSystem, Triple};
+use target_lexicon::{CallingConvention, Triple};
 
 /// Calling convention identifiers.
 #[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]
@@ -22,25 +22,11 @@ pub enum CallConv {
 impl CallConv {
     /// Return the default calling convention for the given target triple.
     pub fn default_for_triple(triple: &Triple) -> Self {
-        match triple.operating_system {
-            OperatingSystem::Unknown
-            | OperatingSystem::Bitrig
-            | OperatingSystem::Cloudabi
-            | OperatingSystem::Darwin
-            | OperatingSystem::Dragonfly
-            | OperatingSystem::Freebsd
-            | OperatingSystem::Fuchsia
-            | OperatingSystem::Haiku
-            | OperatingSystem::Ios
-            | OperatingSystem::L4re
-            | OperatingSystem::Linux
-            | OperatingSystem::Nebulet
-            | OperatingSystem::Netbsd
-            | OperatingSystem::Openbsd
-            | OperatingSystem::Redox
-            | OperatingSystem::Solaris => CallConv::SystemV,
-            OperatingSystem::Windows => CallConv::WindowsFastcall,
-            os => panic!("unsupported operating system: {}", os),
+        match triple.default_calling_convention() {
+            // Default to System V for unknown targets because most everything
+            // uses System V.
+            Ok(CallingConvention::SystemV) | Err(()) => CallConv::SystemV,
+            Ok(CallingConvention::WindowsFastcall) => CallConv::WindowsFastcall,
         }
     }
 }
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 7b84b67fa6..40f60549e1 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -11,10 +11,10 @@ readme = "README.md"
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.22.0" }
 cranelift-module = { path = "../module", version = "0.22.0" }
-faerie = "0.5.0"
+faerie = "0.6.0"
 goblin = "0.0.19"
 failure = "0.1.2"
-target-lexicon = "0.0.3"
+target-lexicon = "0.2.0"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 269f085a57..000c759438 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -11,7 +11,7 @@ readme = "README.md"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
-target-lexicon = { version = "0.0.3", default-features = false }
+target-lexicon = { version = "0.2.0", default-features = false }
 log = { version = "0.4.4", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 58a5234d59..5c680720dd 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -10,7 +10,7 @@ readme = "README.md"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
-target-lexicon = { version = "0.0.3", default-features = false }
+target-lexicon = { version = "0.2.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
 raw-cpuid = "6.0.0"
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 878520a472..1158ecaa06 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -10,7 +10,7 @@ readme = "README.md"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.22.0" }
-target-lexicon = "0.0.3"
+target-lexicon = "0.2.0"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 4d85eb1bce..9046a16dee 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -16,7 +16,7 @@ cranelift-native = { path = "../native", version = "0.22.0", default-features =
 region = "1.0.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
-target-lexicon = { version = "0.0.3", default-features = false }
+target-lexicon = { version = "0.2.0", default-features = false }
 
 [target.'cfg(target_os = "windows")'.dependencies]
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index e674945ac2..ee2b1e45f1 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -21,7 +21,7 @@ log = { version = "0.4.4", default-features = false }
 
 [dev-dependencies]
 wabt = "0.7.0"
-target-lexicon = "0.0.3"
+target-lexicon = "0.2.0"
 
 [features]
 default = ["std"]

From fd875c3f5a027945095cde8605f775da9da1b839 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 7 Nov 2018 15:52:52 -0800
Subject: [PATCH 2178/3084] Pass TargetFrontendConfig by value rather than by
 reference.

---
 lib/codegen/src/isa/mod.rs | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs
index 20469bb6c0..f304cf09c3 100644
--- a/lib/codegen/src/isa/mod.rs
+++ b/lib/codegen/src/isa/mod.rs
@@ -179,17 +179,17 @@ pub struct TargetFrontendConfig {
 
 impl TargetFrontendConfig {
     /// Get the pointer type of this target.
-    pub fn pointer_type(&self) -> ir::Type {
+    pub fn pointer_type(self) -> ir::Type {
         ir::Type::int(u16::from(self.pointer_bits())).unwrap()
     }
 
     /// Get the width of pointers on this target, in units of bits.
-    pub fn pointer_bits(&self) -> u8 {
+    pub fn pointer_bits(self) -> u8 {
         self.pointer_width.bits()
     }
 
     /// Get the width of pointers on this target, in units of bytes.
-    pub fn pointer_bytes(&self) -> u8 {
+    pub fn pointer_bytes(self) -> u8 {
         self.pointer_width.bytes()
     }
 }

From 19dc75f45a688a9299be5dfcfcffa9e860695dd4 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 7 Nov 2018 16:02:17 -0800
Subject: [PATCH 2179/3084] Add preopt to the maintainer scripts.

---
 cranelift/publish-all.sh | 1 +
 cranelift/test-no_std.sh | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 6106ae9d6e..4198a4c0db 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -37,6 +37,7 @@ echo git commit -a -m "\"Bump version to $version"\"
 echo git push
 for crate in \
     entity bforest codegen/meta codegen frontend native \
+    preopt \
     reader wasm module \
     faerie umbrella simplejit
 do
diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh
index 568dffb638..32c354c065 100755
--- a/cranelift/test-no_std.sh
+++ b/cranelift/test-no_std.sh
@@ -13,7 +13,7 @@ function banner {
 }
 
 # Test those packages which have no_std support.
-LIBS="codegen frontend wasm native module simplejit umbrella"
+LIBS="codegen frontend wasm native preopt module simplejit umbrella"
 for LIB in $LIBS; do
     banner "Rust unit tests in $LIB"
     pushd "lib/$LIB" >/dev/null

From 38e8667f9759d4e4bb6d171db5615ad6aa838772 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 7 Nov 2018 16:02:57 -0800
Subject: [PATCH 2180/3084] Adjust whitespace to match the upstream exception
 text.

This is a whitespace-only change.
---
 lib/bforest/LICENSE      | 1 +
 lib/codegen/LICENSE      | 1 +
 lib/codegen/meta/LICENSE | 1 +
 lib/entity/LICENSE       | 1 +
 lib/faerie/LICENSE       | 1 +
 lib/filetests/LICENSE    | 1 +
 lib/frontend/LICENSE     | 1 +
 lib/module/LICENSE       | 1 +
 lib/native/LICENSE       | 1 +
 lib/preopt/LICENSE       | 1 +
 lib/reader/LICENSE       | 1 +
 lib/serde/LICENSE        | 1 +
 lib/simplejit/LICENSE    | 1 +
 lib/umbrella/LICENSE     | 1 +
 lib/wasm/LICENSE         | 1 +
 15 files changed, 15 insertions(+)

diff --git a/lib/bforest/LICENSE b/lib/bforest/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/bforest/LICENSE
+++ b/lib/bforest/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/codegen/LICENSE b/lib/codegen/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/codegen/LICENSE
+++ b/lib/codegen/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/codegen/meta/LICENSE b/lib/codegen/meta/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/codegen/meta/LICENSE
+++ b/lib/codegen/meta/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/entity/LICENSE b/lib/entity/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/entity/LICENSE
+++ b/lib/entity/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/faerie/LICENSE b/lib/faerie/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/faerie/LICENSE
+++ b/lib/faerie/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/filetests/LICENSE b/lib/filetests/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/filetests/LICENSE
+++ b/lib/filetests/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/frontend/LICENSE b/lib/frontend/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/frontend/LICENSE
+++ b/lib/frontend/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/module/LICENSE b/lib/module/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/module/LICENSE
+++ b/lib/module/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/native/LICENSE b/lib/native/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/native/LICENSE
+++ b/lib/native/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/preopt/LICENSE b/lib/preopt/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/preopt/LICENSE
+++ b/lib/preopt/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/reader/LICENSE b/lib/reader/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/reader/LICENSE
+++ b/lib/reader/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/serde/LICENSE b/lib/serde/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/serde/LICENSE
+++ b/lib/serde/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/simplejit/LICENSE b/lib/simplejit/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/simplejit/LICENSE
+++ b/lib/simplejit/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/umbrella/LICENSE b/lib/umbrella/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/umbrella/LICENSE
+++ b/lib/umbrella/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+
diff --git a/lib/wasm/LICENSE b/lib/wasm/LICENSE
index be1d7c438a..f9d81955f4 100644
--- a/lib/wasm/LICENSE
+++ b/lib/wasm/LICENSE
@@ -217,3 +217,4 @@ conflicts with the conditions of the GPLv2, you may retroactively and
 prospectively choose to deem waived or otherwise exclude such Section(s) of
 the License, but only in their entirety and only with respect to the Combined
 Software.
+

From ad8a19c7ff9fce503c6457acff29705f2d6c28b9 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 7 Nov 2018 16:13:51 -0800
Subject: [PATCH 2181/3084] Update no_std support.

---
 lib/preopt/src/lib.rs | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/lib/preopt/src/lib.rs b/lib/preopt/src/lib.rs
index bf2ffc1a8c..789b308afb 100644
--- a/lib/preopt/src/lib.rs
+++ b/lib/preopt/src/lib.rs
@@ -1,12 +1,8 @@
 //! Performes early-stage optimizations on Cranelift IR.
 
-#![deny(
-    missing_docs,
-    trivial_numeric_casts,
-    unused_extern_crates,
-    unstable_features
-)]
+#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates,)]
 #![warn(unused_import_braces)]
+#![cfg_attr(feature = "std", deny(unstable_features))]
 #![cfg_attr(
     feature = "clippy",
     plugin(clippy(conf_file = "../../clippy.toml"))
@@ -29,7 +25,7 @@
     )
 )]
 #![cfg_attr(not(feature = "std"), no_std)]
-#![cfg_attr(not(feature = "std"), alloc)]
+#![cfg_attr(not(feature = "std"), feature(alloc))]
 
 #[cfg(not(feature = "std"))]
 extern crate alloc;

From 14dfc0d1b618ef41a4fa2e65950db82a53e852fd Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 7 Nov 2018 15:53:07 -0800
Subject: [PATCH 2182/3084] Bump version to 0.23.0.

---
 cranelift/Cargo.toml        | 28 ++++++++++++++--------------
 cranelift/publish-all.sh    |  2 +-
 lib/bforest/Cargo.toml      |  4 ++--
 lib/codegen/Cargo.toml      |  8 ++++----
 lib/codegen/meta/Cargo.toml |  4 ++--
 lib/entity/Cargo.toml       |  2 +-
 lib/faerie/Cargo.toml       |  6 +++---
 lib/filetests/Cargo.toml    |  8 ++++----
 lib/frontend/Cargo.toml     |  4 ++--
 lib/module/Cargo.toml       |  6 +++---
 lib/native/Cargo.toml       |  4 ++--
 lib/preopt/Cargo.toml       |  6 +++---
 lib/reader/Cargo.toml       |  4 ++--
 lib/serde/Cargo.toml        |  6 +++---
 lib/simplejit/Cargo.toml    | 14 +++++++-------
 lib/umbrella/Cargo.toml     |  6 +++---
 lib/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index e401e954c5..8aeb8538d6 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.22.0"
+version = "0.23.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,19 +14,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.22.0" }
-cranelift-entity = { path = "lib/entity", version = "0.22.0" }
-cranelift-reader = { path = "lib/reader", version = "0.22.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.22.0" }
-cranelift-serde = { path = "lib/serde", version = "0.22.0", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.22.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.22.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.22.0" }
-cranelift-module = { path = "lib/module", version = "0.22.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.22.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.22.0" }
-cranelift-preopt = { path = "lib/preopt", version = "0.22.0" }
-cranelift = { path = "lib/umbrella", version = "0.22.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.23.0" }
+cranelift-entity = { path = "lib/entity", version = "0.23.0" }
+cranelift-reader = { path = "lib/reader", version = "0.23.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.23.0" }
+cranelift-serde = { path = "lib/serde", version = "0.23.0", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.23.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.23.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.23.0" }
+cranelift-module = { path = "lib/module", version = "0.23.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.23.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.23.0" }
+cranelift-preopt = { path = "lib/preopt", version = "0.23.0" }
+cranelift = { path = "lib/umbrella", version = "0.23.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 4198a4c0db..3643cbec42 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.22.0"
+version="0.23.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index 8b08504993..291fdf63ee 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.22.0"
+version = "0.23.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 keywords = ["btree", "forest", "set", "map"]
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.23.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index baaa059f3f..679555bc03 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.22.0"
+version = "0.23.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
-cranelift-bforest = { path = "../bforest", version = "0.22.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.23.0", default-features = false }
+cranelift-bforest = { path = "../bforest", version = "0.23.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -25,7 +25,7 @@ log = { version = "0.4.4", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.22.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.23.0" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 98c22d24c1..57f701205e 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,14 +1,14 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.22.0"
+version = "0.23.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-entity = { path = "../../entity", version = "0.22.0", default-features = false }
+cranelift-entity = { path = "../../entity", version = "0.23.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index 3e7718ad7c..a7fe521acc 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.22.0"
+version = "0.23.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 40f60549e1..334cf488a5 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.22.0"
+version = "0.23.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.22.0" }
-cranelift-module = { path = "../module", version = "0.22.0" }
+cranelift-codegen = { path = "../codegen", version = "0.23.0" }
+cranelift-module = { path = "../module", version = "0.23.0" }
 faerie = "0.6.0"
 goblin = "0.0.19"
 failure = "0.1.2"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index c3f587e569..fc1ddff8a3 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.22.0"
+version = "0.23.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,9 +9,9 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.22.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../reader", version = "0.22.0" }
-cranelift-preopt = { path = "../preopt", version = "0.22.0" }
+cranelift-codegen = { path = "../codegen", version = "0.23.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../reader", version = "0.23.0" }
+cranelift-preopt = { path = "../preopt", version = "0.23.0" }
 file-per-thread-logger = "0.1.1"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 000c759438..215fb3bdf3 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.22.0"
+version = "0.23.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 log = { version = "0.4.4", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 52e58d33e6..55bf9d5721 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.22.0"
+version = "0.23.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.23.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 log = { version = "0.4.4", default-features = false }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 5c680720dd..595f5ea20a 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.22.0"
+version = "0.23.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/preopt/Cargo.toml b/lib/preopt/Cargo.toml
index d88ce942bc..52dd0d8902 100644
--- a/lib/preopt/Cargo.toml
+++ b/lib/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.22.0"
+version = "0.23.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,8 +11,8 @@ readme = "README.md"
 keywords = ["optimize", "compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.23.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 1158ecaa06..b577923bc7 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.22.0"
+version = "0.23.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.22.0" }
+cranelift-codegen = { path = "../codegen", version = "0.23.0" }
 target-lexicon = "0.2.0"
 
 [badges]
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index c1623cd7ac..0b571006c2 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.22.0"
+version = "0.23.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
-cranelift-reader = { path = "../reader", version = "0.22.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
+cranelift-reader = { path = "../reader", version = "0.23.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 9046a16dee..730ceacf85 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.22.0"
+version = "0.23.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.22.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.22.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.23.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.23.0", default-features = false }
 region = "1.0.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.2.0", default-features = false }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../umbrella", version = "0.22.0" }
-cranelift-frontend = { path = "../frontend", version = "0.22.0" }
-cranelift-entity = { path = "../entity", version = "0.22.0" }
+cranelift = { path = "../umbrella", version = "0.23.0" }
+cranelift-frontend = { path = "../frontend", version = "0.23.0" }
+cranelift-entity = { path = "../entity", version = "0.23.0" }
 
 [features]
 default = ["std"]
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index 7a6241ea60..ca6e554591 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.22.0"
+version = "0.23.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,8 +11,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.22.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.23.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index ee2b1e45f1..70bbb6bbe9 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.22.0"
+version = "0.23.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,9 +11,9 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.21.4", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.22.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.22.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.22.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.23.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.23.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 1d14f5ddbd6549ecbde1b46b687e88ff7062f019 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 7 Nov 2018 16:24:19 -0800
Subject: [PATCH 2183/3084] Use cranelift-entity with default features.

The meta crate doesn't need to support no_std, so it can just use
default settings.
---
 lib/codegen/meta/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 57f701205e..70073e4492 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -8,7 +8,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-entity = { path = "../../entity", version = "0.23.0", default-features = false }
+cranelift-entity = { path = "../../entity", version = "0.23.0" }
 
 [badges]
 maintenance = { status = "experimental" }

From 47a33474ef483bdc337b7ca859b7b624343c365c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 7 Nov 2018 16:28:37 -0800
Subject: [PATCH 2184/3084] Fix a typo in the category name.

---
 lib/preopt/Cargo.toml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/preopt/Cargo.toml b/lib/preopt/Cargo.toml
index 52dd0d8902..c2c6ee46fd 100644
--- a/lib/preopt/Cargo.toml
+++ b/lib/preopt/Cargo.toml
@@ -6,7 +6,7 @@ description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
 repository = "https://github.com/CraneStation/cranelift"
-categories = ["no_std"]
+categories = ["no-std"]
 readme = "README.md"
 keywords = ["optimize", "compile", "compiler", "jit"]
 
@@ -24,4 +24,4 @@ core = ["cranelift-codegen/core"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
\ No newline at end of file
+travis-ci = { repository = "CraneStation/cranelift" }

From 8b5ad467c6a3a268a90afc4ded845cae4a9a0f84 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 7 Nov 2018 16:45:36 -0800
Subject: [PATCH 2185/3084] Fix missing newline.

---
 lib/preopt/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/preopt/README.md b/lib/preopt/README.md
index 12608b7adb..cb1560e718 100644
--- a/lib/preopt/README.md
+++ b/lib/preopt/README.md
@@ -1 +1 @@
-This crate performes early-stage optimizations on [Cranelift](https://crates.io/crates/cranelift) IR.
\ No newline at end of file
+This crate performes early-stage optimizations on [Cranelift](https://crates.io/crates/cranelift) IR.

From e13b0886dc6cbe70a0c45564a424299d1d2b5086 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 8 Nov 2018 16:41:00 +0100
Subject: [PATCH 2186/3084] SimpleJIT: Ignore result of writeln! to avoid a
 compiler warning when writing the perf map;

---
 lib/simplejit/src/backend.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index ccf113ce1c..b105e70e3c 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -187,7 +187,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
                 .open(format!("/tmp/perf-{}.map", ::std::process::id()))
                 .unwrap();
 
-            writeln!(map_file, "{:x} {:x} {}", ptr as usize, code_size, name);
+            let _ = writeln!(map_file, "{:x} {:x} {}", ptr as usize, code_size, name);
         }
 
         let mut reloc_sink = SimpleJITRelocSink::new();

From f896bfb94653e78b245682d0ec624b6fe36f443a Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 8 Nov 2018 16:48:20 +0100
Subject: [PATCH 2187/3084] Prefix fixed_results/fixed_value_arguments with num
 to indicate they return a usize;

---
 lib/codegen/src/ir/dfg.rs             | 48 ++++++++++++++++++---------
 lib/codegen/src/ir/instructions.rs    | 34 +++++++++----------
 lib/codegen/src/legalizer/boundary.rs | 22 +++++++-----
 lib/codegen/src/legalizer/split.rs    | 16 +++++----
 lib/codegen/src/regalloc/reload.rs    |  4 +--
 lib/codegen/src/regalloc/spilling.rs  |  6 ++--
 lib/codegen/src/verifier/mod.rs       |  4 +--
 7 files changed, 80 insertions(+), 54 deletions(-)

diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs
index ecb22237f9..2ac9969d97 100644
--- a/lib/codegen/src/ir/dfg.rs
+++ b/lib/codegen/src/ir/dfg.rs
@@ -430,26 +430,38 @@ impl DataFlowGraph {
 
     /// Get the fixed value arguments on `inst` as a slice.
     pub fn inst_fixed_args(&self, inst: Inst) -> &[Value] {
-        let fixed_args = self[inst].opcode().constraints().fixed_value_arguments();
-        &self.inst_args(inst)[..fixed_args]
+        let num_fixed_args = self[inst]
+            .opcode()
+            .constraints()
+            .num_fixed_value_arguments();
+        &self.inst_args(inst)[..num_fixed_args]
     }
 
     /// Get the fixed value arguments on `inst` as a mutable slice.
     pub fn inst_fixed_args_mut(&mut self, inst: Inst) -> &mut [Value] {
-        let fixed_args = self[inst].opcode().constraints().fixed_value_arguments();
-        &mut self.inst_args_mut(inst)[..fixed_args]
+        let num_fixed_args = self[inst]
+            .opcode()
+            .constraints()
+            .num_fixed_value_arguments();
+        &mut self.inst_args_mut(inst)[..num_fixed_args]
     }
 
     /// Get the variable value arguments on `inst` as a slice.
     pub fn inst_variable_args(&self, inst: Inst) -> &[Value] {
-        let fixed_args = self[inst].opcode().constraints().fixed_value_arguments();
-        &self.inst_args(inst)[fixed_args..]
+        let num_fixed_args = self[inst]
+            .opcode()
+            .constraints()
+            .num_fixed_value_arguments();
+        &self.inst_args(inst)[num_fixed_args..]
     }
 
     /// Get the variable value arguments on `inst` as a mutable slice.
     pub fn inst_variable_args_mut(&mut self, inst: Inst) -> &mut [Value] {
-        let fixed_args = self[inst].opcode().constraints().fixed_value_arguments();
-        &mut self.inst_args_mut(inst)[fixed_args..]
+        let num_fixed_args = self[inst]
+            .opcode()
+            .constraints()
+            .num_fixed_value_arguments();
+        &mut self.inst_args_mut(inst)[num_fixed_args..]
     }
 
     /// Create result values for an instruction that produces multiple results.
@@ -489,7 +501,10 @@ impl DataFlowGraph {
         // Get the call signature if this is a function call.
         if let Some(sig) = self.call_signature(inst) {
             // Create result values corresponding to the call return types.
-            debug_assert_eq!(self.insts[inst].opcode().constraints().fixed_results(), 0);
+            debug_assert_eq!(
+                self.insts[inst].opcode().constraints().num_fixed_results(),
+                0
+            );
             let num_results = self.signatures[sig].returns.len();
             for res_idx in 0..num_results {
                 let ty = self.signatures[sig].returns[res_idx].value_type;
@@ -504,7 +519,7 @@ impl DataFlowGraph {
         } else {
             // Create result values corresponding to the opcode's constraints.
             let constraints = self.insts[inst].opcode().constraints();
-            let num_results = constraints.fixed_results();
+            let num_results = constraints.num_fixed_results();
             for res_idx in 0..num_results {
                 let ty = constraints.result_type(res_idx, ctrl_typevar);
                 if let Some(Some(v)) = reuse.next() {
@@ -662,9 +677,9 @@ impl DataFlowGraph {
         ctrl_typevar: Type,
     ) -> Option {
         let constraints = self.insts[inst].opcode().constraints();
-        let fixed_results = constraints.fixed_results();
+        let num_fixed_results = constraints.num_fixed_results();
 
-        if result_idx < fixed_results {
+        if result_idx < num_fixed_results {
             return Some(constraints.result_type(result_idx, ctrl_typevar));
         }
 
@@ -672,7 +687,7 @@ impl DataFlowGraph {
         self.call_signature(inst).and_then(|sigref| {
             self.signatures[sigref]
                 .returns
-                .get(result_idx - fixed_results)
+                .get(result_idx - num_fixed_results)
                 .map(|&arg| arg.value_type)
         })
     }
@@ -934,7 +949,10 @@ impl DataFlowGraph {
     ) -> usize {
         // Get the call signature if this is a function call.
         if let Some(sig) = self.call_signature(inst) {
-            assert_eq!(self.insts[inst].opcode().constraints().fixed_results(), 0);
+            assert_eq!(
+                self.insts[inst].opcode().constraints().num_fixed_results(),
+                0
+            );
             for res_idx in 0..self.signatures[sig].returns.len() {
                 let ty = self.signatures[sig].returns[res_idx].value_type;
                 if let Some(v) = reuse.get(res_idx) {
@@ -943,7 +961,7 @@ impl DataFlowGraph {
             }
         } else {
             let constraints = self.insts[inst].opcode().constraints();
-            for res_idx in 0..constraints.fixed_results() {
+            for res_idx in 0..constraints.num_fixed_results() {
                 let ty = constraints.result_type(res_idx, ctrl_typevar);
                 if let Some(v) = reuse.get(res_idx) {
                     self.set_value_type_for_parser(*v, ty);
diff --git a/lib/codegen/src/ir/instructions.rs b/lib/codegen/src/ir/instructions.rs
index 84e34b1f39..b01525ca80 100644
--- a/lib/codegen/src/ir/instructions.rs
+++ b/lib/codegen/src/ir/instructions.rs
@@ -335,8 +335,8 @@ pub struct OpcodeConstraints {
     typeset_offset: u8,
 
     /// Offset into `OPERAND_CONSTRAINT` table of the descriptors for this opcode. The first
-    /// `fixed_results()` entries describe the result constraints, then follows constraints for the
-    /// fixed `Value` input operands. (`fixed_value_arguments()` of them).
+    /// `num_fixed_results()` entries describe the result constraints, then follows constraints for the
+    /// fixed `Value` input operands. (`num_fixed_value_arguments()` of them).
     constraint_offset: u16,
 }
 
@@ -360,7 +360,7 @@ impl OpcodeConstraints {
 
     /// Get the number of *fixed* result values produced by this opcode.
     /// This does not include `variable_args` produced by calls.
-    pub fn fixed_results(self) -> usize {
+    pub fn num_fixed_results(self) -> usize {
         (self.flags & 0x7) as usize
     }
 
@@ -371,7 +371,7 @@ impl OpcodeConstraints {
     /// The number of fixed input values is usually implied by the instruction format, but
     /// instruction formats that use a `ValueList` put both fixed and variable arguments in the
     /// list. This method returns the *minimum* number of values required in the value list.
-    pub fn fixed_value_arguments(self) -> usize {
+    pub fn num_fixed_value_arguments(self) -> usize {
         ((self.flags >> 5) & 0x7) as usize
     }
 
@@ -394,7 +394,7 @@ impl OpcodeConstraints {
     /// Get the value type of result number `n`, having resolved the controlling type variable to
     /// `ctrl_type`.
     pub fn result_type(self, n: usize, ctrl_type: Type) -> Type {
-        debug_assert!(n < self.fixed_results(), "Invalid result index");
+        debug_assert!(n < self.num_fixed_results(), "Invalid result index");
         if let ResolvedConstraint::Bound(t) =
             OPERAND_CONSTRAINTS[self.constraint_offset() + n].resolve(ctrl_type)
         {
@@ -411,10 +411,10 @@ impl OpcodeConstraints {
     /// `ValueTypeSet`. This is represented with the `ArgumentConstraint::Free` variant.
     pub fn value_argument_constraint(self, n: usize, ctrl_type: Type) -> ResolvedConstraint {
         debug_assert!(
-            n < self.fixed_value_arguments(),
+            n < self.num_fixed_value_arguments(),
             "Invalid value argument index"
         );
-        let offset = self.constraint_offset() + self.fixed_results();
+        let offset = self.constraint_offset() + self.num_fixed_results();
         OPERAND_CONSTRAINTS[offset + n].resolve(ctrl_type)
     }
 
@@ -603,8 +603,8 @@ mod tests {
         let a = Opcode::Iadd.constraints();
         assert!(a.use_typevar_operand());
         assert!(!a.requires_typevar_operand());
-        assert_eq!(a.fixed_results(), 1);
-        assert_eq!(a.fixed_value_arguments(), 2);
+        assert_eq!(a.num_fixed_results(), 1);
+        assert_eq!(a.num_fixed_value_arguments(), 2);
         assert_eq!(a.result_type(0, types::I32), types::I32);
         assert_eq!(a.result_type(0, types::I8), types::I8);
         assert_eq!(
@@ -619,8 +619,8 @@ mod tests {
         let b = Opcode::Bitcast.constraints();
         assert!(!b.use_typevar_operand());
         assert!(!b.requires_typevar_operand());
-        assert_eq!(b.fixed_results(), 1);
-        assert_eq!(b.fixed_value_arguments(), 1);
+        assert_eq!(b.num_fixed_results(), 1);
+        assert_eq!(b.num_fixed_value_arguments(), 1);
         assert_eq!(b.result_type(0, types::I32), types::I32);
         assert_eq!(b.result_type(0, types::I8), types::I8);
         match b.value_argument_constraint(0, types::I32) {
@@ -629,18 +629,18 @@ mod tests {
         }
 
         let c = Opcode::Call.constraints();
-        assert_eq!(c.fixed_results(), 0);
-        assert_eq!(c.fixed_value_arguments(), 0);
+        assert_eq!(c.num_fixed_results(), 0);
+        assert_eq!(c.num_fixed_value_arguments(), 0);
 
         let i = Opcode::CallIndirect.constraints();
-        assert_eq!(i.fixed_results(), 0);
-        assert_eq!(i.fixed_value_arguments(), 1);
+        assert_eq!(i.num_fixed_results(), 0);
+        assert_eq!(i.num_fixed_value_arguments(), 1);
 
         let cmp = Opcode::Icmp.constraints();
         assert!(cmp.use_typevar_operand());
         assert!(cmp.requires_typevar_operand());
-        assert_eq!(cmp.fixed_results(), 1);
-        assert_eq!(cmp.fixed_value_arguments(), 2);
+        assert_eq!(cmp.num_fixed_results(), 1);
+        assert_eq!(cmp.num_fixed_value_arguments(), 2);
     }
 
     #[test]
diff --git a/lib/codegen/src/legalizer/boundary.rs b/lib/codegen/src/legalizer/boundary.rs
index b8bdc83ca8..e337704f66 100644
--- a/lib/codegen/src/legalizer/boundary.rs
+++ b/lib/codegen/src/legalizer/boundary.rs
@@ -201,8 +201,14 @@ where
 
     // We theoretically allow for call instructions that return a number of fixed results before
     // the call return values. In practice, it doesn't happen.
-    let fixed_results = pos.func.dfg[call].opcode().constraints().fixed_results();
-    debug_assert_eq!(fixed_results, 0, "Fixed results on calls not supported");
+    debug_assert_eq!(
+        pos.func.dfg[call]
+            .opcode()
+            .constraints()
+            .num_fixed_results(),
+        0,
+        "Fixed results on calls not supported"
+    );
 
     let results = pos.func.dfg.detach_results(call);
     let mut next_res = 0;
@@ -440,11 +446,11 @@ fn legalize_inst_arguments(
     // The value list contains all arguments to the instruction, including the callee on an
     // indirect call which isn't part of the call arguments that must match the ABI signature.
     // Figure out how many fixed values are at the front of the list. We won't touch those.
-    let fixed_values = pos.func.dfg[inst]
+    let num_fixed_values = pos.func.dfg[inst]
         .opcode()
         .constraints()
-        .fixed_value_arguments();
-    let have_args = vlist.len(&pos.func.dfg.value_lists) - fixed_values;
+        .num_fixed_value_arguments();
+    let have_args = vlist.len(&pos.func.dfg.value_lists) - num_fixed_values;
 
     // Grow the value list to the right size and shift all the existing arguments to the right.
     // This lets us write the new argument values into the list without overwriting the old
@@ -472,11 +478,11 @@ fn legalize_inst_arguments(
     //   [FFFFNNNNNNNNNNNNNNNNNNNN]
     //
     vlist.grow_at(
-        fixed_values,
+        num_fixed_values,
         abi_args - have_args,
         &mut pos.func.dfg.value_lists,
     );
-    let old_arg_offset = fixed_values + abi_args - have_args;
+    let old_arg_offset = num_fixed_values + abi_args - have_args;
 
     let mut abi_arg = 0;
     for old_arg in 0..have_args {
@@ -487,7 +493,7 @@ fn legalize_inst_arguments(
             let abi_type = get_abi_type(func, abi_arg);
             if func.dfg.value_type(arg) == abi_type.value_type {
                 // This is the argument type we need.
-                vlist.as_mut_slice(&mut func.dfg.value_lists)[fixed_values + abi_arg] = arg;
+                vlist.as_mut_slice(&mut func.dfg.value_lists)[num_fixed_values + abi_arg] = arg;
                 abi_arg += 1;
                 Ok(())
             } else {
diff --git a/lib/codegen/src/legalizer/split.rs b/lib/codegen/src/legalizer/split.rs
index 1cce097f26..46f530bf61 100644
--- a/lib/codegen/src/legalizer/split.rs
+++ b/lib/codegen/src/legalizer/split.rs
@@ -133,14 +133,14 @@ fn split_any(
                 "Predecessor not a branch: {}",
                 pos.func.dfg.display_inst(inst, None)
             );
-            let fixed_args = branch_opc.constraints().fixed_value_arguments();
+            let num_fixed_args = branch_opc.constraints().num_fixed_value_arguments();
             let mut args = pos.func.dfg[inst]
                 .take_value_list()
                 .expect("Branches must have value lists.");
             let num_args = args.len(&pos.func.dfg.value_lists);
             // Get the old value passed to the EBB argument we're repairing.
             let old_arg = args
-                .get(fixed_args + repair.num, &pos.func.dfg.value_lists)
+                .get(num_fixed_args + repair.num, &pos.func.dfg.value_lists)
                 .expect("Too few branch arguments");
 
             // It's possible that the CFG's predecessor list has duplicates. Detect them here.
@@ -155,21 +155,23 @@ fn split_any(
 
             // The `lo` part replaces the original argument.
             *args
-                .get_mut(fixed_args + repair.num, &mut pos.func.dfg.value_lists)
+                .get_mut(num_fixed_args + repair.num, &mut pos.func.dfg.value_lists)
                 .unwrap() = lo;
 
             // The `hi` part goes at the end. Since multiple repairs may have been scheduled to the
             // same EBB, there could be multiple arguments missing.
-            if num_args > fixed_args + repair.hi_num {
+            if num_args > num_fixed_args + repair.hi_num {
                 *args
-                    .get_mut(fixed_args + repair.hi_num, &mut pos.func.dfg.value_lists)
-                    .unwrap() = hi;
+                    .get_mut(
+                        num_fixed_args + repair.hi_num,
+                        &mut pos.func.dfg.value_lists,
+                    ).unwrap() = hi;
             } else {
                 // We need to append one or more arguments. If we're adding more than one argument,
                 // there must be pending repairs on the stack that will fill in the correct values
                 // instead of `hi`.
                 args.extend(
-                    iter::repeat(hi).take(1 + fixed_args + repair.hi_num - num_args),
+                    iter::repeat(hi).take(1 + num_fixed_args + repair.hi_num - num_args),
                     &mut pos.func.dfg.value_lists,
                 );
             }
diff --git a/lib/codegen/src/regalloc/reload.rs b/lib/codegen/src/regalloc/reload.rs
index e22fefc0fc..6d16ec5d3f 100644
--- a/lib/codegen/src/regalloc/reload.rs
+++ b/lib/codegen/src/regalloc/reload.rs
@@ -266,7 +266,7 @@ impl<'a> Context<'a> {
         let retvals = &defs[self.cur.func.dfg[inst]
                                 .opcode()
                                 .constraints()
-                                .fixed_results()..];
+                                .num_fixed_results()..];
         if !retvals.is_empty() {
             let sig = self
                 .cur
@@ -367,7 +367,7 @@ impl<'a> Context<'a> {
         let offset = self.cur.func.dfg[inst]
             .opcode()
             .constraints()
-            .fixed_value_arguments();
+            .num_fixed_value_arguments();
         if args.len() == offset {
             return;
         }
diff --git a/lib/codegen/src/regalloc/spilling.rs b/lib/codegen/src/regalloc/spilling.rs
index 1ce0649fed..e4b1d52e48 100644
--- a/lib/codegen/src/regalloc/spilling.rs
+++ b/lib/codegen/src/regalloc/spilling.rs
@@ -376,10 +376,10 @@ impl<'a> Context<'a> {
 
     // Collect register uses from the ABI input constraints.
     fn collect_abi_reg_uses(&mut self, inst: Inst, sig: SigRef) {
-        let fixed_args = self.cur.func.dfg[inst]
+        let num_fixed_args = self.cur.func.dfg[inst]
             .opcode()
             .constraints()
-            .fixed_value_arguments();
+            .num_fixed_value_arguments();
         let args = self.cur.func.dfg.inst_variable_args(inst);
         for (idx, (abi, &arg)) in self.cur.func.dfg.signatures[sig]
             .params
@@ -396,7 +396,7 @@ impl<'a> Context<'a> {
                     ),
                     Affinity::Unassigned => panic!("Missing affinity for {}", arg),
                 };
-                let mut reguse = RegUse::new(arg, fixed_args + idx, rci);
+                let mut reguse = RegUse::new(arg, num_fixed_args + idx, rci);
                 reguse.fixed = true;
                 reguse.spilled = spilled;
                 self.reg_uses.push(reguse);
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 31a7dfc029..1d25cd9b23 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -524,12 +524,12 @@ impl<'a> Verifier<'a> {
             );
         }
 
-        let fixed_results = inst_data.opcode().constraints().fixed_results();
+        let num_fixed_results = inst_data.opcode().constraints().num_fixed_results();
         // var_results is 0 if we aren't a call instruction
         let var_results = dfg
             .call_signature(inst)
             .map_or(0, |sig| dfg.signatures[sig].returns.len());
-        let total_results = fixed_results + var_results;
+        let total_results = num_fixed_results + var_results;
 
         // All result values for multi-valued instructions are created
         let got_results = dfg.inst_results(inst).len();

From add4043bb5eef31374f29ad9f1e22ad708e0691e Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 8 Nov 2018 18:41:15 +0100
Subject: [PATCH 2188/3084] [build] Introduce a TargetIsaBuilder to better
 encapsulate the TargetIsa;

It was the caller's responsibility to call TargetIsa::check() before;
now one can't manipulate a TargetIsa without calling the
TargetIsaBuilder::finish() method, which is less error-prone and more in
line with what's coming for other things we're going to generate in the
meta crate.

Also splits the construction of a RegClass in two parts: a prototype is
made first when declaring the RegClass, and missing bits are filled in
when adding it to the TargetIsa(Builder). This avoids an awkward passing
of the isa to the RegClass ctor.
---
 lib/codegen/meta/src/cdsl/isa.rs      | 90 ++++++++++++++++++---------
 lib/codegen/meta/src/cdsl/regs.rs     | 50 +++++----------
 lib/codegen/meta/src/isa/arm32/mod.rs | 18 +++---
 lib/codegen/meta/src/isa/arm64/mod.rs | 14 ++---
 lib/codegen/meta/src/isa/mod.rs       |  8 +--
 lib/codegen/meta/src/isa/riscv/mod.rs | 12 ++--
 lib/codegen/meta/src/isa/x86/mod.rs   | 20 +++---
 7 files changed, 108 insertions(+), 104 deletions(-)

diff --git a/lib/codegen/meta/src/cdsl/isa.rs b/lib/codegen/meta/src/cdsl/isa.rs
index 4a6a819683..d1855176c9 100644
--- a/lib/codegen/meta/src/cdsl/isa.rs
+++ b/lib/codegen/meta/src/cdsl/isa.rs
@@ -1,7 +1,7 @@
 use cranelift_entity::PrimaryMap;
 
 use super::regs::{
-    RegBank, RegBankBuilder, RegBankIndex, RegClass, RegClassBuilder, RegClassIndex,
+    RegBank, RegBankBuilder, RegBankIndex, RegClass, RegClassBuilder, RegClassIndex, RegClassProto,
 };
 
 pub struct TargetIsa {
@@ -18,12 +18,24 @@ impl TargetIsa {
             reg_classes: PrimaryMap::new(),
         }
     }
+}
+
+pub struct TargetIsaBuilder {
+    isa: TargetIsa,
+}
+
+impl TargetIsaBuilder {
+    pub fn new(name: &'static str) -> Self {
+        Self {
+            isa: TargetIsa::new(name),
+        }
+    }
 
     pub fn add_reg_bank(&mut self, builder: RegBankBuilder) -> RegBankIndex {
-        let first_unit = if self.reg_banks.len() == 0 {
+        let first_unit = if self.isa.reg_banks.len() == 0 {
             0
         } else {
-            let last = &self.reg_banks.last().unwrap();
+            let last = &self.isa.reg_banks.last().unwrap();
             let first_available_unit = (last.first_unit + last.units) as i8;
             let units = builder.units;
             let align = if units.is_power_of_two() {
@@ -34,7 +46,7 @@ impl TargetIsa {
             (first_available_unit + align - 1) & -align
         } as u8;
 
-        self.reg_banks.push(RegBank::new(
+        self.isa.reg_banks.push(RegBank::new(
             builder.name,
             first_unit,
             builder.units,
@@ -47,38 +59,51 @@ impl TargetIsa {
     }
 
     pub fn add_reg_class(&mut self, builder: RegClassBuilder) -> RegClassIndex {
-        let reg_bank_units = self.reg_banks.get(builder.bank).unwrap().units;
+        let class_index = self.isa.reg_classes.next_key();
 
-        let start = builder.start;
+        // Finish delayed construction of RegClass.
+        let (bank, toprc, start, width) = match builder.proto {
+            RegClassProto::TopLevel(bank_index) => {
+                self.isa
+                    .reg_banks
+                    .get_mut(bank_index)
+                    .unwrap()
+                    .toprcs
+                    .push(class_index);
+                (bank_index, class_index, builder.start, builder.width)
+            }
+            RegClassProto::SubClass(parent_class_index) => {
+                assert!(builder.width == 0);
+                let (bank, toprc, start, width) = {
+                    let parent = self.isa.reg_classes.get(parent_class_index).unwrap();
+                    (parent.bank, parent.toprc, parent.start, parent.width)
+                };
+                for reg_class in self.isa.reg_classes.values_mut() {
+                    if reg_class.toprc == toprc {
+                        reg_class.subclasses.push(class_index);
+                    }
+                }
+                let subclass_start = start + builder.start * width;
+                (bank, toprc, subclass_start, width)
+            }
+        };
+
+        let reg_bank_units = self.isa.reg_banks.get(bank).unwrap().units;
         assert!(start < reg_bank_units);
 
         let count = if builder.count != 0 {
             builder.count
         } else {
-            reg_bank_units / builder.width
+            reg_bank_units / width
         };
 
-        let reg_class_index = builder.index;
-        assert!(
-            self.reg_classes.next_key() == reg_class_index,
-            "should have inserted RegClass where expected"
-        );
+        let reg_class = RegClass::new(builder.name, class_index, width, bank, toprc, count, start);
+        self.isa.reg_classes.push(reg_class);
 
-        let reg_class = RegClass::new(
-            builder.name,
-            reg_class_index,
-            builder.width,
-            builder.bank,
-            builder.toprc,
-            count,
-            start,
-        );
-        self.reg_classes.push(reg_class);
+        let reg_bank = self.isa.reg_banks.get_mut(bank).unwrap();
+        reg_bank.classes.push(class_index);
 
-        let reg_bank = self.reg_banks.get_mut(builder.bank).unwrap();
-        reg_bank.classes.push(reg_class_index);
-
-        reg_class_index
+        class_index
     }
 
     /// Checks that the set of register classes satisfies:
@@ -89,16 +114,16 @@ impl TargetIsa {
     /// 2. There are no identical classes under different names.
     /// 3. Classes are sorted topologically such that all subclasses have a
     ///    higher index that the superclass.
-    pub fn check(&self) {
-        for reg_bank in self.reg_banks.values() {
+    pub fn finish(self) -> TargetIsa {
+        for reg_bank in self.isa.reg_banks.values() {
             for i1 in reg_bank.classes.iter() {
                 for i2 in reg_bank.classes.iter() {
                     if i1 >= i2 {
                         continue;
                     }
 
-                    let rc1 = self.reg_classes.get(*i1).unwrap();
-                    let rc2 = self.reg_classes.get(*i2).unwrap();
+                    let rc1 = self.isa.reg_classes.get(*i1).unwrap();
+                    let rc2 = self.isa.reg_classes.get(*i2).unwrap();
 
                     let rc1_mask = rc1.mask(0);
                     let rc2_mask = rc2.mask(0);
@@ -126,7 +151,8 @@ impl TargetIsa {
                     // If the intersection is the second one, then it must be a subclass.
                     if intersect == rc2_mask {
                         assert!(
-                            self.reg_classes
+                            self.isa
+                                .reg_classes
                                 .get(*i1)
                                 .unwrap()
                                 .subclasses
@@ -138,5 +164,7 @@ impl TargetIsa {
                 }
             }
         }
+
+        self.isa
     }
 }
diff --git a/lib/codegen/meta/src/cdsl/regs.rs b/lib/codegen/meta/src/cdsl/regs.rs
index bc40ffaf69..4e8a34001b 100644
--- a/lib/codegen/meta/src/cdsl/regs.rs
+++ b/lib/codegen/meta/src/cdsl/regs.rs
@@ -1,7 +1,5 @@
 use cranelift_entity::EntityRef;
 
-use super::isa::TargetIsa;
-
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct RegBankIndex(u32);
 entity_impl!(RegBankIndex);
@@ -97,71 +95,53 @@ impl RegClass {
     }
 }
 
+pub enum RegClassProto {
+    TopLevel(RegBankIndex),
+    SubClass(RegClassIndex),
+}
+
 pub struct RegClassBuilder {
     pub name: &'static str,
-    pub index: RegClassIndex,
     pub width: u8,
-    pub bank: RegBankIndex,
-    pub toprc: RegClassIndex,
     pub count: u8,
     pub start: u8,
+    pub proto: RegClassProto,
 }
 
 impl RegClassBuilder {
-    pub fn new_toplevel(isa: &mut TargetIsa, name: &'static str, bank: RegBankIndex) -> Self {
-        let index = isa.reg_classes.next_key();
-
-        // Add it to the top-level register classes of the register bank.
-        isa.reg_banks.get_mut(bank).unwrap().toprcs.push(index);
-
+    pub fn new_toplevel(name: &'static str, bank: RegBankIndex) -> Self {
         Self {
             name,
-            index,
             width: 1,
-            bank,
-            toprc: index,
             count: 0,
             start: 0,
+            proto: RegClassProto::TopLevel(bank),
         }
     }
-
     pub fn subclass_of(
-        isa: &mut TargetIsa,
         name: &'static str,
         parent_index: RegClassIndex,
         start: u8,
         stop: u8,
     ) -> Self {
         assert!(stop >= start);
-
-        let index = isa.reg_classes.next_key();
-
-        let toprc = isa.reg_classes.get(parent_index).unwrap().toprc;
-        for reg_class in isa.reg_classes.values_mut() {
-            if reg_class.toprc == toprc {
-                reg_class.subclasses.push(index);
-            }
-        }
-
-        let parent = &isa.reg_classes.get(parent_index).unwrap();
         Self {
             name,
+            width: 0,
             count: stop - start,
-            width: parent.width,
-            start: parent.start + start * parent.width,
-            bank: parent.bank,
-            toprc: parent.toprc,
-            index,
+            start: start,
+            proto: RegClassProto::SubClass(parent_index),
         }
     }
-
     pub fn count(mut self, count: u8) -> Self {
         self.count = count;
         self
     }
-
     pub fn width(mut self, width: u8) -> Self {
-        self.width = width;
+        match self.proto {
+            RegClassProto::TopLevel(_) => self.width = width,
+            RegClassProto::SubClass(_) => panic!("Subclasses inherit their parent's width."),
+        }
         self
     }
 }
diff --git a/lib/codegen/meta/src/isa/arm32/mod.rs b/lib/codegen/meta/src/isa/arm32/mod.rs
index ce33422276..e0e47c2e2e 100644
--- a/lib/codegen/meta/src/isa/arm32/mod.rs
+++ b/lib/codegen/meta/src/isa/arm32/mod.rs
@@ -1,8 +1,8 @@
+use cdsl::isa::{TargetIsa, TargetIsaBuilder};
 use cdsl::regs::{RegBankBuilder, RegClassBuilder};
-use isa;
 
-pub fn define() -> isa::TargetIsa {
-    let mut isa = isa::TargetIsa::new("arm32");
+pub fn define() -> TargetIsa {
+    let mut isa = TargetIsaBuilder::new("arm32");
 
     let builder = RegBankBuilder::new("FloatRegs", "s")
         .units(64)
@@ -20,20 +20,20 @@ pub fn define() -> isa::TargetIsa {
         .track_pressure(false);
     let flag_reg = isa.add_reg_bank(builder);
 
-    let builder = RegClassBuilder::new_toplevel(&mut isa, "S", float_regs).count(32);
+    let builder = RegClassBuilder::new_toplevel("S", float_regs).count(32);
     isa.add_reg_class(builder);
 
-    let builder = RegClassBuilder::new_toplevel(&mut isa, "D", float_regs).width(2);
+    let builder = RegClassBuilder::new_toplevel("D", float_regs).width(2);
     isa.add_reg_class(builder);
 
-    let builder = RegClassBuilder::new_toplevel(&mut isa, "Q", float_regs).width(4);
+    let builder = RegClassBuilder::new_toplevel("Q", float_regs).width(4);
     isa.add_reg_class(builder);
 
-    let builder = RegClassBuilder::new_toplevel(&mut isa, "GPR", int_regs);
+    let builder = RegClassBuilder::new_toplevel("GPR", int_regs);
     isa.add_reg_class(builder);
 
-    let builder = RegClassBuilder::new_toplevel(&mut isa, "FLAG", flag_reg);
+    let builder = RegClassBuilder::new_toplevel("FLAG", flag_reg);
     isa.add_reg_class(builder);
 
-    isa
+    isa.finish()
 }
diff --git a/lib/codegen/meta/src/isa/arm64/mod.rs b/lib/codegen/meta/src/isa/arm64/mod.rs
index 8e13d28321..2435f3676e 100644
--- a/lib/codegen/meta/src/isa/arm64/mod.rs
+++ b/lib/codegen/meta/src/isa/arm64/mod.rs
@@ -1,8 +1,8 @@
+use cdsl::isa::{TargetIsa, TargetIsaBuilder};
 use cdsl::regs::{RegBankBuilder, RegClassBuilder};
-use isa;
 
-pub fn define() -> isa::TargetIsa {
-    let mut isa = isa::TargetIsa::new("arm64");
+pub fn define() -> TargetIsa {
+    let mut isa = TargetIsaBuilder::new("arm64");
 
     // The `x31` regunit serves as the stack pointer / zero register depending on context. We
     // reserve it and don't model the difference.
@@ -22,14 +22,14 @@ pub fn define() -> isa::TargetIsa {
         .track_pressure(false);
     let flag_reg = isa.add_reg_bank(builder);
 
-    let builder = RegClassBuilder::new_toplevel(&mut isa, "GPR", int_regs);
+    let builder = RegClassBuilder::new_toplevel("GPR", int_regs);
     isa.add_reg_class(builder);
 
-    let builder = RegClassBuilder::new_toplevel(&mut isa, "FPR", float_regs);
+    let builder = RegClassBuilder::new_toplevel("FPR", float_regs);
     isa.add_reg_class(builder);
 
-    let builder = RegClassBuilder::new_toplevel(&mut isa, "FLAG", flag_reg);
+    let builder = RegClassBuilder::new_toplevel("FLAG", flag_reg);
     isa.add_reg_class(builder);
 
-    isa
+    isa.finish()
 }
diff --git a/lib/codegen/meta/src/isa/mod.rs b/lib/codegen/meta/src/isa/mod.rs
index ffb13fb151..18f75c42e5 100644
--- a/lib/codegen/meta/src/isa/mod.rs
+++ b/lib/codegen/meta/src/isa/mod.rs
@@ -62,14 +62,10 @@ impl fmt::Display for Isa {
 }
 
 pub fn define_all() -> Vec {
-    let isas = vec![
+    vec![
         riscv::define(),
         arm32::define(),
         arm64::define(),
         x86::define(),
-    ];
-    for isa in isas.iter() {
-        isa.check();
-    }
-    isas
+    ]
 }
diff --git a/lib/codegen/meta/src/isa/riscv/mod.rs b/lib/codegen/meta/src/isa/riscv/mod.rs
index 315ee42233..67f504c0f8 100644
--- a/lib/codegen/meta/src/isa/riscv/mod.rs
+++ b/lib/codegen/meta/src/isa/riscv/mod.rs
@@ -1,8 +1,8 @@
+use cdsl::isa::{TargetIsa, TargetIsaBuilder};
 use cdsl::regs::{RegBankBuilder, RegClassBuilder};
-use isa;
 
-pub fn define() -> isa::TargetIsa {
-    let mut isa = isa::TargetIsa::new("riscv");
+pub fn define() -> TargetIsa {
+    let mut isa = TargetIsaBuilder::new("riscv");
 
     let builder = RegBankBuilder::new("IntRegs", "x")
         .units(32)
@@ -14,11 +14,11 @@ pub fn define() -> isa::TargetIsa {
         .track_pressure(true);
     let float_regs = isa.add_reg_bank(builder);
 
-    let builder = RegClassBuilder::new_toplevel(&mut isa, "GPR", int_regs);
+    let builder = RegClassBuilder::new_toplevel("GPR", int_regs);
     isa.add_reg_class(builder);
 
-    let builder = RegClassBuilder::new_toplevel(&mut isa, "FPR", float_regs);
+    let builder = RegClassBuilder::new_toplevel("FPR", float_regs);
     isa.add_reg_class(builder);
 
-    isa
+    isa.finish()
 }
diff --git a/lib/codegen/meta/src/isa/x86/mod.rs b/lib/codegen/meta/src/isa/x86/mod.rs
index 43fd4ed67d..7970ea06db 100644
--- a/lib/codegen/meta/src/isa/x86/mod.rs
+++ b/lib/codegen/meta/src/isa/x86/mod.rs
@@ -1,8 +1,8 @@
+use cdsl::isa::{TargetIsa, TargetIsaBuilder};
 use cdsl::regs::{RegBankBuilder, RegClassBuilder};
-use isa;
 
-pub fn define() -> isa::TargetIsa {
-    let mut isa = isa::TargetIsa::new("x86");
+pub fn define() -> TargetIsa {
+    let mut isa = TargetIsaBuilder::new("x86");
 
     let builder = RegBankBuilder::new("IntRegs", "r")
         .units(16)
@@ -21,23 +21,23 @@ pub fn define() -> isa::TargetIsa {
         .track_pressure(false);
     let flag_reg = isa.add_reg_bank(builder);
 
-    let builder = RegClassBuilder::new_toplevel(&mut isa, "GPR", int_regs);
+    let builder = RegClassBuilder::new_toplevel("GPR", int_regs);
     let gpr = isa.add_reg_class(builder);
 
-    let builder = RegClassBuilder::new_toplevel(&mut isa, "FPR", float_regs);
+    let builder = RegClassBuilder::new_toplevel("FPR", float_regs);
     let fpr = isa.add_reg_class(builder);
 
-    let builder = RegClassBuilder::new_toplevel(&mut isa, "FLAG", flag_reg);
+    let builder = RegClassBuilder::new_toplevel("FLAG", flag_reg);
     isa.add_reg_class(builder);
 
-    let builder = RegClassBuilder::subclass_of(&mut isa, "GPR8", gpr, 0, 8);
+    let builder = RegClassBuilder::subclass_of("GPR8", gpr, 0, 8);
     let gpr8 = isa.add_reg_class(builder);
 
-    let builder = RegClassBuilder::subclass_of(&mut isa, "ABCD", gpr8, 0, 4);
+    let builder = RegClassBuilder::subclass_of("ABCD", gpr8, 0, 4);
     isa.add_reg_class(builder);
 
-    let builder = RegClassBuilder::subclass_of(&mut isa, "FPR8", fpr, 0, 8);
+    let builder = RegClassBuilder::subclass_of("FPR8", fpr, 0, 8);
     isa.add_reg_class(builder);
 
-    isa
+    isa.finish()
 }

From ce2364ddd93b0e3859dc49b66575f3af925bb692 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 9 Nov 2018 15:59:40 +0100
Subject: [PATCH 2189/3084] [build] Remove dead code in Python and move
 assertions to the Rust code generator;

---
 lib/codegen/meta-python/cdsl/isa.py | 45 -----------------------------
 lib/codegen/meta/src/cdsl/isa.rs    | 19 ++++++++++++
 2 files changed, 19 insertions(+), 45 deletions(-)

diff --git a/lib/codegen/meta-python/cdsl/isa.py b/lib/codegen/meta-python/cdsl/isa.py
index 90a9610d62..3e3ac0b4fe 100644
--- a/lib/codegen/meta-python/cdsl/isa.py
+++ b/lib/codegen/meta-python/cdsl/isa.py
@@ -48,7 +48,6 @@ class TargetISA(object):
         self.instruction_groups = instruction_groups
         self.cpumodes = list()  # type: List[CPUMode]
         self.regbanks = list()  # type: List[RegBank]
-        self.regclasses = list()  # type: List[RegClass]
         self.legalize_codes = OrderedDict()  # type: OrderedDict[XFormGroup, int]  # noqa
         # Unique copies of all predicates.
         self._predicates = dict()  # type: Dict[PredKey, PredNode]
@@ -74,7 +73,6 @@ class TargetISA(object):
         """
         self._collect_encoding_recipes()
         self._collect_predicates()
-        self._collect_regclasses()
         self._collect_legalize_codes()
         return self
 
@@ -122,49 +120,6 @@ class TargetISA(object):
                 if enc.isap:
                     self.settings.number_predicate(enc.isap)
 
-    def _collect_regclasses(self):
-        # type: () -> None
-        """
-        Collect and number register classes.
-
-        Every register class needs a unique index, and the classes need to be
-        topologically ordered.
-
-        We also want all the top-level register classes to be first.
-        """
-        # Compute subclasses and top-level classes in each bank.
-        # Collect the top-level classes so they get numbered consecutively.
-        for bank in self.regbanks:
-            bank.finish_regclasses()
-            # Always get the pressure tracking classes in first.
-            if bank.pressure_tracking:
-                self.regclasses.extend(bank.toprcs)
-
-        # The limit on the number of top-level register classes can be raised.
-        # This should be coordinated with the `MAX_TRACKED_TOPRCS` constant in
-        # `isa/registers.rs`.
-        assert len(self.regclasses) <= 4, "Too many top-level register classes"
-
-        # Get the remaining top-level register classes which may exceed
-        # `MAX_TRACKED_TOPRCS`.
-        for bank in self.regbanks:
-            if not bank.pressure_tracking:
-                self.regclasses.extend(bank.toprcs)
-
-        # Collect all of the non-top-level register classes.
-        # They are numbered strictly after the top-level classes.
-        for bank in self.regbanks:
-            self.regclasses.extend(
-                    rc for rc in bank.classes if not rc.is_toprc())
-
-        for idx, rc in enumerate(self.regclasses):
-            rc.index = idx
-
-        # The limit on the number of register classes can be changed. It should
-        # be coordinated with the `RegClassMask` and `RegClassIndex` types in
-        # `isa/registers.rs`.
-        assert len(self.regclasses) <= 32, "Too many register classes"
-
     def _collect_legalize_codes(self):
         # type: () -> None
         """
diff --git a/lib/codegen/meta/src/cdsl/isa.rs b/lib/codegen/meta/src/cdsl/isa.rs
index d1855176c9..7168d32c85 100644
--- a/lib/codegen/meta/src/cdsl/isa.rs
+++ b/lib/codegen/meta/src/cdsl/isa.rs
@@ -165,6 +165,25 @@ impl TargetIsaBuilder {
             }
         }
 
+        // This limit should be coordinated with the `RegClassMask` and `RegClassIndex` types in
+        // isa/registers.rs of the non-meta code.
+        assert!(
+            self.isa.reg_classes.len() <= 32,
+            "Too many register classes"
+        );
+
+        // The maximum number of top-level register classes which have pressure tracking should be
+        // kept in sync with the MAX_TRACKED_TOPRCS constant in isa/registers.rs of the non-meta
+        // code.
+        let num_toplevel = self
+            .isa
+            .reg_classes
+            .values()
+            .filter(|x| {
+                x.toprc == x.index && self.isa.reg_banks.get(x.bank).unwrap().pressure_tracking
+            }).count();
+        assert!(num_toplevel <= 4, "Too many top-level register classes");
+
         self.isa
     }
 }

From d54569cbdd56b91892fca7d02a7f6c046011655c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 9 Nov 2018 15:10:17 -0800
Subject: [PATCH 2190/3084] Fix a typo in a comment.

---
 lib/codegen/src/isa/enc_tables.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/codegen/src/isa/enc_tables.rs b/lib/codegen/src/isa/enc_tables.rs
index 1d8ed94780..8cc259420f 100644
--- a/lib/codegen/src/isa/enc_tables.rs
+++ b/lib/codegen/src/isa/enc_tables.rs
@@ -40,7 +40,7 @@ pub type LegalizeCode = u8;
 /// size of the `LEVEL2` table.
 ///
 /// Empty entries are encoded with a `!0` value for `log2len` which will always be out of range.
-/// Entries that have a `legalize` value but no level 2 table have an `offset` field that is out f
+/// Entries that have a `legalize` value but no level 2 table have an `offset` field that is out of
 /// bounds.
 pub struct Level1Entry + Copy> {
     pub ty: Type,

From a0c2b40c1759d73ab725fa98c6988c596734935c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 9 Nov 2018 15:11:03 -0800
Subject: [PATCH 2191/3084] Remove unnecessary `pub` keywords.

---
 lib/codegen/src/isa/x86/abi.rs | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs
index 67fe3a486a..2542eb3eb9 100644
--- a/lib/codegen/src/isa/x86/abi.rs
+++ b/lib/codegen/src/isa/x86/abi.rs
@@ -270,7 +270,7 @@ pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenRes
     }
 }
 
-pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> {
+fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> {
     debug_assert!(
         !isa.flags().probestack_enabled(),
         "baldrdash does not expect cranelift to emit stack probes"
@@ -291,7 +291,7 @@ pub fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) ->
 
 /// Implementation of the fastcall-based Win64 calling convention described at [1]
 /// [1] https://msdn.microsoft.com/en-us/library/ms235286.aspx
-pub fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> {
+fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> {
     if isa.triple().pointer_width().unwrap() != PointerWidth::U64 {
         panic!("TODO: windows-fastcall: x86-32 not implemented yet");
     }
@@ -363,7 +363,7 @@ pub fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> C
 }
 
 /// Insert a System V-compatible prologue and epilogue.
-pub fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> {
+fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> {
     // The original 32-bit x86 ELF ABI had a 4-byte aligned stack pointer, but
     // newer versions use a 16-byte aligned stack pointer.
     let stack_align = 16;

From 7b27fdbf5439a9a05c603e10f49c9374a8e06223 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 9 Nov 2018 15:13:42 -0800
Subject: [PATCH 2192/3084] Avoid cloning an `InstructionData` when it isn't
 needed.

---
 lib/codegen/src/simple_gvn.rs | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/lib/codegen/src/simple_gvn.rs b/lib/codegen/src/simple_gvn.rs
index 54d1040e9d..d60bdb6899 100644
--- a/lib/codegen/src/simple_gvn.rs
+++ b/lib/codegen/src/simple_gvn.rs
@@ -110,10 +110,8 @@ pub fn do_simple_gvn(func: &mut Function, domtree: &mut DominatorTree) {
                 continue;
             }
 
-            let inst_data = func.dfg[inst].clone();
-
             // These are split up to separate concerns.
-            if is_load_and_not_readonly(&inst_data) {
+            if is_load_and_not_readonly(&func.dfg[inst]) {
                 continue;
             }
 
@@ -123,9 +121,8 @@ pub fn do_simple_gvn(func: &mut Function, domtree: &mut DominatorTree) {
                 ty: ctrl_typevar,
                 pos: &pos,
             };
-            let entry = visible_values.entry(key);
             use scoped_hash_map::Entry::*;
-            match entry {
+            match visible_values.entry(key) {
                 Occupied(entry) => {
                     debug_assert!(domtree.dominates(*entry.get(), inst, &func.layout));
                     // If the redundant instruction is representing the current

From 076850549de1f48574270c292f57e6786f4490c3 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 12 Nov 2018 06:35:44 -0800
Subject: [PATCH 2193/3084] Clarify an assertion error message.

---
 lib/codegen/src/regalloc/register_set.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/codegen/src/regalloc/register_set.rs b/lib/codegen/src/regalloc/register_set.rs
index b297578cae..d3a11a9d45 100644
--- a/lib/codegen/src/regalloc/register_set.rs
+++ b/lib/codegen/src/regalloc/register_set.rs
@@ -72,7 +72,7 @@ impl RegisterSet {
         let (idx, bits) = bitmask(rc, reg);
         debug_assert!(
             (self.avail[idx] & bits) == 0,
-            "{}:{} not allocated in {}",
+            "{}:{} is already free in {}",
             rc,
             rc.info.display_regunit(reg),
             self.display(rc.info)

From f3ea2d5d66b2f40128a05cff25467f9fcbe3ee81 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 12 Nov 2018 06:35:23 -0800
Subject: [PATCH 2194/3084] Tidy up documentation comment syntax.

---
 lib/codegen/src/regalloc/coalescing.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/codegen/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs
index 053d7f7dee..cfdfd984fd 100644
--- a/lib/codegen/src/regalloc/coalescing.rs
+++ b/lib/codegen/src/regalloc/coalescing.rs
@@ -375,7 +375,7 @@ impl<'a> Context<'a> {
     }
 
     /// Finish the union-find part of the coalescing algorithm.
-    ///     ///
+    ///
     /// This builds the initial set of virtual registers as the transitive/reflexive/symmetric
     /// closure of the relation formed by EBB parameter-argument pairs found by `union_find_ebb()`.
     fn finish_union_find(&mut self) {

From ef2e11265cf28bfc4b58a26ee1e4f5a75d1b641d Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 12 Nov 2018 07:13:53 -0800
Subject: [PATCH 2195/3084] Update to Rust 1.30.

---
 .travis.yml         | 2 +-
 cranelift/README.md | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 1020d2b8e3..8e90dd1c5d 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -4,7 +4,7 @@ language: rust
 rust:
     # The oldest version we currently support. See
     # CONTRIBUTING.md#rustc-version-support for details.
-    - 1.29.0
+    - 1.30.1
     - beta
     - nightly
 matrix:
diff --git a/cranelift/README.md b/cranelift/README.md
index cb567cd5a5..b0e37978f9 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -9,7 +9,7 @@ into executable machine code.
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
 [![Build Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
-![Minimum rustc 1.29](https://img.shields.io/badge/rustc-1.29+-red.svg)
+![Minimum rustc 1.30](https://img.shields.io/badge/rustc-1.30+-red.svg)
 
 For more information, see [the
 documentation](https://cranelift.readthedocs.io/en/latest/?badge=latest).
@@ -47,7 +47,7 @@ needed before it would be ready for a production use case.
 
 Cranelift's APIs are not yet stable.
 
-Cranelift currently requires Rust 1.29 or later, and Python 2.7 or 3
+Cranelift currently requires Rust 1.30 or later, and Python 2.7 or 3
 to build.
 
 Planned uses

From cd7c57e598f19ddc0fe0e54596c3f6f226f1541a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 6 Nov 2018 11:14:14 -0800
Subject: [PATCH 2196/3084] Make spilling visit fallthrough_return instructions
 too.

This is a followup to af2a952aabd82cf401cc664d0262b139ff92d86b. It
teaches the spilling pass to use the is_ghost() property to test whether
to visit instructions. This fixes a bug handling multiple return values
with fallthrough_return.
---
 .../filetests/regalloc/multiple-returns.clif  |  10 ++
 lib/codegen/src/regalloc/spilling.rs          | 104 +++++++++---------
 2 files changed, 64 insertions(+), 50 deletions(-)

diff --git a/cranelift/filetests/regalloc/multiple-returns.clif b/cranelift/filetests/regalloc/multiple-returns.clif
index 6e36c9b9b8..6514e3e030 100644
--- a/cranelift/filetests/regalloc/multiple-returns.clif
+++ b/cranelift/filetests/regalloc/multiple-returns.clif
@@ -11,3 +11,13 @@ ebb0:
 ; check: v2 = iconst.i64 0
 ; check: v3 = copy v2
 ; check: return v2, v3
+
+; Same thing, now with a fallthrough_return.
+function %multiple_returns() -> i64, i64 {
+ebb0:
+    v2 = iconst.i64 0
+    fallthrough_return v2, v2
+}
+; check: v2 = iconst.i64 0
+; check: v3 = copy v2
+; check: fallthrough_return v2, v3
diff --git a/lib/codegen/src/regalloc/spilling.rs b/lib/codegen/src/regalloc/spilling.rs
index e4b1d52e48..168956703c 100644
--- a/lib/codegen/src/regalloc/spilling.rs
+++ b/lib/codegen/src/regalloc/spilling.rs
@@ -134,11 +134,8 @@ impl<'a> Context<'a> {
         self.process_spills(tracker);
 
         while let Some(inst) = self.cur.next_inst() {
-            if let Some(constraints) = self
-                .encinfo
-                .operand_constraints(self.cur.func.encodings[inst])
-            {
-                self.visit_inst(inst, ebb, constraints, tracker);
+            if !self.cur.func.dfg[inst].opcode().is_ghost() {
+                self.visit_inst(inst, ebb, tracker);
             } else {
                 let (_throughs, kills) = tracker.process_ghost(inst);
                 self.free_regs(kills);
@@ -237,17 +234,15 @@ impl<'a> Context<'a> {
         self.free_dead_regs(params);
     }
 
-    fn visit_inst(
-        &mut self,
-        inst: Inst,
-        ebb: Ebb,
-        constraints: &RecipeConstraints,
-        tracker: &mut LiveValueTracker,
-    ) {
+    fn visit_inst(&mut self, inst: Inst, ebb: Ebb, tracker: &mut LiveValueTracker) {
         debug!("Inst {}, {}", self.cur.display_inst(inst), self.pressure);
         debug_assert_eq!(self.cur.current_inst(), Some(inst));
         debug_assert_eq!(self.cur.current_ebb(), Some(ebb));
 
+        let constraints = self
+            .encinfo
+            .operand_constraints(self.cur.func.encodings[inst]);
+
         // We may need to resolve register constraints if there are any noteworthy uses.
         debug_assert!(self.reg_uses.is_empty());
         self.collect_reg_uses(inst, ebb, constraints);
@@ -282,23 +277,25 @@ impl<'a> Context<'a> {
         // Make sure we have enough registers for the register defs.
         // Dead defs are included here. They need a register too.
         // No need to process call return values, they are in fixed registers.
-        for op in constraints.outs {
-            if op.kind != ConstraintKind::Stack {
-                // Add register def to pressure, spill if needed.
-                while let Err(mask) = self.pressure.take_transient(op.regclass) {
-                    debug!("Need {} reg from {} throughs", op.regclass, throughs.len());
-                    match self.spill_candidate(mask, throughs) {
-                        Some(cand) => self.spill_reg(cand),
-                        None => panic!(
-                            "Ran out of {} registers for {}",
-                            op.regclass,
-                            self.cur.display_inst(inst)
-                        ),
+        if let Some(constraints) = constraints {
+            for op in constraints.outs {
+                if op.kind != ConstraintKind::Stack {
+                    // Add register def to pressure, spill if needed.
+                    while let Err(mask) = self.pressure.take_transient(op.regclass) {
+                        debug!("Need {} reg from {} throughs", op.regclass, throughs.len());
+                        match self.spill_candidate(mask, throughs) {
+                            Some(cand) => self.spill_reg(cand),
+                            None => panic!(
+                                "Ran out of {} registers for {}",
+                                op.regclass,
+                                self.cur.display_inst(inst)
+                            ),
+                        }
                     }
                 }
             }
+            self.pressure.reset_transient();
         }
-        self.pressure.reset_transient();
 
         // Restore pressure state, compute pressure with affinities from `defs`.
         // Exclude dead defs. Includes call return values.
@@ -315,35 +312,42 @@ impl<'a> Context<'a> {
     // We are assuming here that if a value is used both by a fixed register operand and a register
     // class operand, they two are compatible. We are also assuming that two register class
     // operands are always compatible.
-    fn collect_reg_uses(&mut self, inst: Inst, ebb: Ebb, constraints: &RecipeConstraints) {
+    fn collect_reg_uses(&mut self, inst: Inst, ebb: Ebb, constraints: Option<&RecipeConstraints>) {
         let args = self.cur.func.dfg.inst_args(inst);
-        for (idx, (op, &arg)) in constraints.ins.iter().zip(args).enumerate() {
-            let mut reguse = RegUse::new(arg, idx, op.regclass.into());
-            let lr = &self.liveness[arg];
-            let ctx = self.liveness.context(&self.cur.func.layout);
-            match op.kind {
-                ConstraintKind::Stack => continue,
-                ConstraintKind::FixedReg(_) => reguse.fixed = true,
-                ConstraintKind::Tied(_) => {
-                    // A tied operand must kill the used value.
-                    reguse.tied = !lr.killed_at(inst, ebb, ctx);
+        let num_fixed_ins = if let Some(constraints) = constraints {
+            for (idx, (op, &arg)) in constraints.ins.iter().zip(args).enumerate() {
+                let mut reguse = RegUse::new(arg, idx, op.regclass.into());
+                let lr = &self.liveness[arg];
+                let ctx = self.liveness.context(&self.cur.func.layout);
+                match op.kind {
+                    ConstraintKind::Stack => continue,
+                    ConstraintKind::FixedReg(_) => reguse.fixed = true,
+                    ConstraintKind::Tied(_) => {
+                        // A tied operand must kill the used value.
+                        reguse.tied = !lr.killed_at(inst, ebb, ctx);
+                    }
+                    ConstraintKind::FixedTied(_) => {
+                        reguse.fixed = true;
+                        reguse.tied = !lr.killed_at(inst, ebb, ctx);
+                    }
+                    ConstraintKind::Reg => {}
                 }
-                ConstraintKind::FixedTied(_) => {
-                    reguse.fixed = true;
-                    reguse.tied = !lr.killed_at(inst, ebb, ctx);
+                if lr.affinity.is_stack() {
+                    reguse.spilled = true;
                 }
-                ConstraintKind::Reg => {}
-            }
-            if lr.affinity.is_stack() {
-                reguse.spilled = true;
-            }
 
-            // Only collect the interesting register uses.
-            if reguse.fixed || reguse.tied || reguse.spilled {
-                debug!("  reguse: {}", reguse);
-                self.reg_uses.push(reguse);
+                // Only collect the interesting register uses.
+                if reguse.fixed || reguse.tied || reguse.spilled {
+                    debug!("  reguse: {}", reguse);
+                    self.reg_uses.push(reguse);
+                }
             }
-        }
+            constraints.ins.len()
+        } else {
+            // A non-ghost instruction with no constraints can't have any
+            // fixed operands.
+            0
+        };
 
         // Similarly, for return instructions, collect uses of ABI-defined
         // return values.
@@ -356,7 +360,7 @@ impl<'a> Context<'a> {
             for (ret_idx, (ret, &arg)) in
                 self.cur.func.signature.returns.iter().zip(args).enumerate()
             {
-                let idx = constraints.ins.len() + ret_idx;
+                let idx = num_fixed_ins + ret_idx;
                 let unit = match ret.location {
                     ArgumentLoc::Unassigned => {
                         panic!("function return signature should be legalized")

From c2f5bc00a5441ed8edd0aadb58804f745358e476 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 12 Nov 2018 15:15:02 -0800
Subject: [PATCH 2197/3084] Fix typos in comments.

---
 lib/codegen/src/ir/extname.rs | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/codegen/src/ir/extname.rs b/lib/codegen/src/ir/extname.rs
index 05f1adc54a..b35c9aeb05 100644
--- a/lib/codegen/src/ir/extname.rs
+++ b/lib/codegen/src/ir/extname.rs
@@ -13,7 +13,7 @@ const TESTCASE_NAME_LENGTH: usize = 16;
 
 /// The name of an external is either a reference to a user-defined symbol
 /// table, or a short sequence of ascii bytes so that test cases do not have
-/// to keep track of a sy mbol table.
+/// to keep track of a symbol table.
 ///
 /// External names are primarily used as keys by code using Cranelift to map
 /// from a `cranelift_codegen::ir::FuncRef` or similar to additional associated
@@ -68,12 +68,12 @@ impl ExternalName {
         }
     }
 
-    /// Create a new external name from user-provided integer indicies.
+    /// Create a new external name from user-provided integer indices.
     ///
     /// # Examples
     /// ```rust
     /// # use cranelift_codegen::ir::ExternalName;
-    /// // Create `ExternalName` from integer indicies
+    /// // Create `ExternalName` from integer indices
     /// let name = ExternalName::user(123, 456);
     /// assert_eq!(name.to_string(), "u123:456");
     /// ```

From 039188a4bfebbc83680d0585c1cb31eae9e3ed34 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 13 Nov 2018 16:08:13 -0800
Subject: [PATCH 2198/3084] Move Vim integration files into their own repo.

https://github.com/CraneStation/cranelift.vim
---
 cranelift/README.md        |  7 +++++++
 misc/vim/ftdetect/clif.vim |  1 -
 misc/vim/syntax/clif.vim   | 43 --------------------------------------
 3 files changed, 7 insertions(+), 44 deletions(-)
 delete mode 100644 misc/vim/ftdetect/clif.vim
 delete mode 100644 misc/vim/syntax/clif.vim

diff --git a/cranelift/README.md b/cranelift/README.md
index b0e37978f9..b2fda81b5b 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -158,3 +158,10 @@ generator](http://www.sphinx-doc.org/) as well as Python 3::
     $ open _build/html/index.html
 
 

+
+Editor Support
+--------------
+
+Editor support for working with Cranelift IR (clif) files:
+
+ - Vim: https://github.com/CraneStation/cranelift.vim
diff --git a/misc/vim/ftdetect/clif.vim b/misc/vim/ftdetect/clif.vim
deleted file mode 100644
index 58312284a1..0000000000
--- a/misc/vim/ftdetect/clif.vim
+++ /dev/null
@@ -1 +0,0 @@
-au BufRead,BufNewFile *.clif set filetype=clif
diff --git a/misc/vim/syntax/clif.vim b/misc/vim/syntax/clif.vim
deleted file mode 100644
index 337e3a78a7..0000000000
--- a/misc/vim/syntax/clif.vim
+++ /dev/null
@@ -1,43 +0,0 @@
-" Vim syntax file
-" Language:     Cranelift
-" Maintainer:   The Cranelift Project Developers
-
-if version < 600
-  syntax clear
-elseif exists("b:current_syntax")
-  finish
-endif
-
-" Disable spell checking even in comments.
-" They tend to refer to weird stuff like assembler mnemonics anyway.
-syn spell notoplevel
-
-syn keyword clifHeader test target set
-syn keyword clifDecl function jump_table incoming_arg outgoing_arg spill_slot explicit_slot emergency_slot
-syn keyword clifFilecheck check sameln nextln unordered not regex contained
-
-syn match clifType  /\<\([bif]\d\+\(x\d\+\)\?\)\|[if]flags\>/
-syn match clifEntity /\<\(v\|ss\|jt\|fn\|sig\)\d\+\>/
-syn match clifLabel /\/
-syn match clifName /%\w\+\>/
-
-syn match clifNumber /-\?\<[0-9_]\+\>/
-syn match clifNumber /-\?\<0x[0-9a-fA-F_]\+\(\.[0-9a-fA-F_]*\)\?\(p[+-]\?\d\+\)\?\>/
-syn match clifHexSeq /#\x\+\>/
-syn match clifSourceLoc /@[0-9a-f]\+\>/
-
-syn region clifCommentLine start=";" end="$" contains=clifFilecheck
-
-hi def link clifHeader        Keyword
-hi def link clifDecl          Keyword
-hi def link clifType          Type
-hi def link clifEntity        Identifier
-hi def link clifLabel         Label
-hi def link clifName          String
-hi def link clifNumber        Number
-hi def link clifHexSeq        Number
-hi def link clifCommentLine   Comment
-hi def link clifFilecheck     SpecialComment
-hi def link clifSourceLoc     LineNr
-
-let b:current_syntax = "clif"

From 5baeed06bb2359822dd69a494269ff33e1eadbec Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 14 Nov 2018 19:41:56 +0100
Subject: [PATCH 2199/3084] Better incompatible declaration error (#605)

* Better incompatible declaration error

* Change display of IncompatibleSignature error
---
 lib/module/src/module.rs | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 47daccd768..d6328dab73 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -123,9 +123,18 @@ pub enum ModuleError {
     /// Indicates an identifier was used before it was declared
     #[fail(display = "Undeclared identifier: {}", _0)]
     Undeclared(String),
-    /// Indicates an identifier was used contrary to the way it was declared
+    /// Indicates an identifier was used as data/function first, but then used as the other
     #[fail(display = "Incompatible declaration of identifier: {}", _0)]
     IncompatibleDeclaration(String),
+    /// Indicates a function identifier was declared with a
+    /// different signature than declared previously
+    #[fail(
+        display = "Function {} signature {:?} is incompatible with previous declaration {:?}",
+        _0,
+        _2,
+        _1
+    )]
+    IncompatibleSignature(String, ir::Signature, ir::Signature),
     /// Indicates an identifier was defined more than once
     #[fail(display = "Duplicate definition of identifier: {}", _0)]
     DuplicateDefinition(String),
@@ -164,7 +173,11 @@ where
     fn merge(&mut self, linkage: Linkage, sig: &ir::Signature) -> Result<(), ModuleError> {
         self.decl.linkage = Linkage::merge(self.decl.linkage, linkage);
         if &self.decl.signature != sig {
-            return Err(ModuleError::IncompatibleDeclaration(self.decl.name.clone()));
+            return Err(ModuleError::IncompatibleSignature(
+                self.decl.name.clone(),
+                self.decl.signature.clone(),
+                sig.clone(),
+            ));
         }
         Ok(())
     }

From 155fd4c72a813585f7535ed2bb6509f5153e0b90 Mon Sep 17 00:00:00 2001
From: oooooba 
Date: Thu, 15 Nov 2018 05:58:50 +0900
Subject: [PATCH 2200/3084] Eliminate call stack recursion in VirtRegs::find
 (#584)

* Eliminate call stack recursion in VirtRegs::find
---
 lib/codegen/src/regalloc/virtregs.rs | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/lib/codegen/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs
index 55fece93c1..13ef4a5381 100644
--- a/lib/codegen/src/regalloc/virtregs.rs
+++ b/lib/codegen/src/regalloc/virtregs.rs
@@ -291,20 +291,22 @@ impl UFEntry {
 impl VirtRegs {
     /// Find the leader value and rank of the set containing `v`.
     /// Compress the path if needed.
-    fn find(&mut self, val: Value) -> (Value, u32) {
-        match UFEntry::decode(self.union_find[val]) {
-            UFEntry::Rank(rank) => (val, rank),
-            UFEntry::Link(parent) => {
-                // TODO: This recursion would be more efficient as an iteration that pushes
-                // elements onto a SmallVector.
-                let found = self.find(parent);
-                // Compress the path if needed.
-                if found.0 != parent {
-                    self.union_find[val] = UFEntry::encode_link(found.0);
+    fn find(&mut self, mut val: Value) -> (Value, u32) {
+        let mut val_stack = vec![];
+        let found = loop {
+            match UFEntry::decode(self.union_find[val]) {
+                UFEntry::Rank(rank) => break (val, rank),
+                UFEntry::Link(parent) => {
+                    val_stack.push(val);
+                    val = parent;
                 }
-                found
             }
+        };
+        // Compress the path
+        while let Some(val) = val_stack.pop() {
+            self.union_find[val] = UFEntry::encode_link(found.0);
         }
+        found
     }
 
     /// Union the two sets containing `a` and `b`.

From f6617afcdd1c8d762cb2fd5af9ffdd99c9f64b58 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 13 Nov 2018 16:29:08 -0800
Subject: [PATCH 2201/3084] Remove the build time and directory from
 cranelift-codegen-meta's output.

This makes its output nondeterministic, making it friendier for
compilation caching tools.
---
 lib/codegen/build.rs | 13 -------------
 1 file changed, 13 deletions(-)

diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index f5dc654273..1faedddd06 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -24,11 +24,7 @@ use meta::isa::Isa;
 use std::env;
 use std::process;
 
-use std::time::Instant;
-
 fn main() {
-    let start_time = Instant::now();
-
     let out_dir = env::var("OUT_DIR").expect("The OUT_DIR environment variable must be set");
     let target_triple = env::var("TARGET").expect("The TARGET environment variable must be set");
     let cranelift_targets = env::var("CRANELIFT_TARGETS").ok();
@@ -96,15 +92,6 @@ fn main() {
             process::exit(1);
         }
     }
-
-    println!(
-        "cargo:warning=Cranelift meta-build step took {:?}",
-        Instant::now() - start_time
-    );
-    println!(
-        "cargo:warning=Meta-build script generated files in {}",
-        out_dir
-    );
 }
 
 fn identify_python() -> &'static str {

From b170b74b65f984fa65b7ed313c497001a78f79a4 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 13 Nov 2018 19:04:13 +0100
Subject: [PATCH 2202/3084] Fixes #600: Add a SIB byte when encoding a
 non-indexed load/store into r12/rsp;

Memory access instructions which took the GPR_ZERO_DEREF_SAFE register
class (that was removed in #600) should check for the need of either an
offset (r13/rbp) or the SIB byte (r12/rsp). Some load/store instructions
would already take an index, thus already contain the SIB byte in this
case (see instructions which have a comment telling that the else branch
already contains an SIB byte). Non-indexed memory accesses lacked the
SIB byte check, which this patch adds.
---
 lib/codegen/meta-python/isa/x86/recipes.py | 33 ++++++++++++++++------
 lib/codegen/src/isa/x86/enc_tables.rs      | 19 +++++++++++++
 2 files changed, 44 insertions(+), 8 deletions(-)

diff --git a/lib/codegen/meta-python/isa/x86/recipes.py b/lib/codegen/meta-python/isa/x86/recipes.py
index 4c495807b7..9ed9e09e3a 100644
--- a/lib/codegen/meta-python/isa/x86/recipes.py
+++ b/lib/codegen/meta-python/isa/x86/recipes.py
@@ -806,13 +806,16 @@ st = TailRecipe(
         'st', Store, base_size=1, ins=(GPR, GPR), outs=(),
         instp=IsEqual(Store.offset, 0),
         clobbers_flags=False,
-        compute_size="size_plus_maybe_offset_for_in_reg_1",
+        compute_size="size_plus_maybe_sib_or_offset_for_in_reg_1",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        if needs_offset(in_reg1) {
+        if needs_sib_byte(in_reg1) {
+            modrm_sib(in_reg0, sink);
+            sib_noindex(in_reg1, sink);
+        } else if needs_offset(in_reg1) {
             modrm_disp8(in_reg1, in_reg0, sink);
             sink.put1(0);
         } else {
@@ -833,6 +836,7 @@ stWithIndex = TailRecipe(
         sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
     }
     PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
+    // The else branch always inserts an SIB byte.
     if needs_offset(in_reg1) {
         modrm_sib_disp8(in_reg0, sink);
         sib(0, in_reg2, in_reg1, sink);
@@ -872,6 +876,7 @@ stWithIndex_abcd = TailRecipe(
         sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
     }
     PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
+    // The else branch always inserts an SIB byte.
     if needs_offset(in_reg1) {
         modrm_sib_disp8(in_reg0, sink);
         sib(0, in_reg2, in_reg1, sink);
@@ -887,13 +892,16 @@ fst = TailRecipe(
         'fst', Store, base_size=1, ins=(FPR, GPR), outs=(),
         instp=IsEqual(Store.offset, 0),
         clobbers_flags=False,
-        compute_size="size_plus_maybe_offset_for_in_reg_1",
+        compute_size="size_plus_maybe_sib_or_offset_for_in_reg_1",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        if needs_offset(in_reg1) {
+        if needs_sib_byte(in_reg1) {
+            modrm_sib(in_reg0, sink);
+            sib_noindex(in_reg1, sink);
+        } else if needs_offset(in_reg1) {
             modrm_disp8(in_reg1, in_reg0, sink);
             sink.put1(0);
         } else {
@@ -912,6 +920,7 @@ fstWithIndex = TailRecipe(
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
+        // The else branch always inserts an SIB byte.
         if needs_offset(in_reg1) {
             modrm_sib_disp8(in_reg0, sink);
             sib(0, in_reg2, in_reg1, sink);
@@ -1210,13 +1219,16 @@ ld = TailRecipe(
         'ld', Load, base_size=1, ins=(GPR), outs=(GPR),
         instp=IsEqual(Load.offset, 0),
         clobbers_flags=False,
-        compute_size="size_plus_maybe_offset_for_in_reg_0",
+        compute_size="size_plus_maybe_sib_or_offset_for_in_reg_0",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        if needs_offset(in_reg0) {
+        if needs_sib_byte(in_reg0) {
+            modrm_sib(out_reg0, sink);
+            sib_noindex(in_reg0, sink);
+        } else if needs_offset(in_reg0) {
             modrm_disp8(in_reg0, out_reg0, sink);
             sink.put1(0);
         } else {
@@ -1237,6 +1249,7 @@ ldWithIndex = TailRecipe(
         sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
     }
     PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
+    // The else branch always inserts an SIB byte.
     if needs_offset(in_reg0) {
         modrm_sib_disp8(out_reg0, sink);
         sib(0, in_reg1, in_reg0, sink);
@@ -1252,13 +1265,16 @@ fld = TailRecipe(
         'fld', Load, base_size=1, ins=(GPR), outs=(FPR),
         instp=IsEqual(Load.offset, 0),
         clobbers_flags=False,
-        compute_size="size_plus_maybe_offset_for_in_reg_0",
+        compute_size="size_plus_maybe_sib_or_offset_for_in_reg_0",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        if needs_offset(in_reg0) {
+        if needs_sib_byte(in_reg0) {
+            modrm_sib(out_reg0, sink);
+            sib_noindex(in_reg0, sink);
+        } else if needs_offset(in_reg0) {
             modrm_disp8(in_reg0, out_reg0, sink);
             sink.put1(0);
         } else {
@@ -1279,6 +1295,7 @@ fldWithIndex = TailRecipe(
         sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
     }
     PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
+    // The else branch always inserts an SIB byte.
     if needs_offset(in_reg0) {
         modrm_sib_disp8(out_reg0, sink);
         sib(0, in_reg1, in_reg0, sink);
diff --git a/lib/codegen/src/isa/x86/enc_tables.rs b/lib/codegen/src/isa/x86/enc_tables.rs
index 9c52563a27..a709d03b76 100644
--- a/lib/codegen/src/isa/x86/enc_tables.rs
+++ b/lib/codegen/src/isa/x86/enc_tables.rs
@@ -23,6 +23,9 @@ pub fn needs_sib_byte(reg: RegUnit) -> bool {
 pub fn needs_offset(reg: RegUnit) -> bool {
     reg == RU::r13 as RegUnit || reg == RU::rbp as RegUnit
 }
+pub fn needs_sib_byte_or_offset(reg: RegUnit) -> bool {
+    needs_sib_byte(reg) || needs_offset(reg)
+}
 
 fn additional_size_if(
     op_index: usize,
@@ -71,6 +74,22 @@ fn size_plus_maybe_sib_for_in_reg_1(
 ) -> u8 {
     sizing.base_size + additional_size_if(1, inst, divert, func, needs_sib_byte)
 }
+fn size_plus_maybe_sib_or_offset_for_in_reg_0(
+    sizing: &RecipeSizing,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    sizing.base_size + additional_size_if(0, inst, divert, func, needs_sib_byte_or_offset)
+}
+fn size_plus_maybe_sib_or_offset_for_in_reg_1(
+    sizing: &RecipeSizing,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    sizing.base_size + additional_size_if(1, inst, divert, func, needs_sib_byte_or_offset)
+}
 
 /// Expand the `sdiv` and `srem` instructions using `x86_sdivmodx`.
 fn expand_sdivrem(

From d45e8b5830f18add73e0b573bc8390e890e44f3b Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 14 Nov 2018 11:49:34 +0100
Subject: [PATCH 2203/3084] Add SIB/offset for ABCD stores too;

---
 lib/codegen/meta-python/isa/x86/recipes.py | 27 +++++++++++++++++++---
 1 file changed, 24 insertions(+), 3 deletions(-)

diff --git a/lib/codegen/meta-python/isa/x86/recipes.py b/lib/codegen/meta-python/isa/x86/recipes.py
index 9ed9e09e3a..534804e0e2 100644
--- a/lib/codegen/meta-python/isa/x86/recipes.py
+++ b/lib/codegen/meta-python/isa/x86/recipes.py
@@ -854,12 +854,21 @@ st_abcd = TailRecipe(
         instp=IsEqual(Store.offset, 0),
         when_prefixed=st,
         clobbers_flags=False,
+        compute_size="size_plus_maybe_sib_or_offset_for_in_reg_1",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        modrm_rm(in_reg1, in_reg0, sink);
+        if needs_sib_byte(in_reg1) {
+            modrm_sib(in_reg0, sink);
+            sib_noindex(in_reg1, sink);
+        } else if needs_offset(in_reg1) {
+            modrm_disp8(in_reg1, in_reg0, sink);
+            sink.put1(0);
+        } else {
+            modrm_rm(in_reg1, in_reg0, sink);
+        }
         ''')
 
 # XX /r register-indirect store with index and no offset.
@@ -977,12 +986,18 @@ stDisp8_abcd = TailRecipe(
         instp=IsSignedInt(Store.offset, 8),
         when_prefixed=stDisp8,
         clobbers_flags=False,
+        compute_size="size_plus_maybe_sib_for_in_reg_1",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        modrm_disp8(in_reg1, in_reg0, sink);
+        if needs_sib_byte(in_reg1) {
+            modrm_sib_disp8(in_reg0, sink);
+            sib_noindex(in_reg1, sink);
+        } else {
+            modrm_disp8(in_reg1, in_reg0, sink);
+        }
         let offset: i32 = offset.into();
         sink.put1(offset as u8);
         ''')
@@ -1089,12 +1104,18 @@ stDisp32_abcd = TailRecipe(
         'stDisp32_abcd', Store, base_size=5, ins=(ABCD, GPR), outs=(),
         when_prefixed=stDisp32,
         clobbers_flags=False,
+        compute_size="size_plus_maybe_sib_for_in_reg_1",
         emit='''
         if !flags.notrap() {
             sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
         }
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        modrm_disp32(in_reg1, in_reg0, sink);
+        if needs_sib_byte(in_reg1) {
+            modrm_sib_disp32(in_reg0, sink);
+            sib_noindex(in_reg1, sink);
+        } else {
+            modrm_disp32(in_reg1, in_reg0, sink);
+        }
         let offset: i32 = offset.into();
         sink.put4(offset as u32);
         ''')

From b41bc55007e13d03770624006f2b34ce9f18c089 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 14 Nov 2018 13:04:37 +0100
Subject: [PATCH 2204/3084] Pretty-print errors for extended basic blocks too;

---
 lib/codegen/src/print_errors.rs | 51 +++++++++++++++++++++++++++++----
 lib/codegen/src/write.rs        | 25 ++++++++++++++--
 2 files changed, 69 insertions(+), 7 deletions(-)

diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 8e764065a7..43bbc2cea4 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -2,7 +2,7 @@
 
 use entity::SecondaryMap;
 use ir;
-use ir::entities::{AnyEntity, Inst, Value};
+use ir::entities::{AnyEntity, Ebb, Inst, Value};
 use ir::function::Function;
 use isa::TargetIsa;
 use result::CodegenError;
@@ -36,6 +36,17 @@ pub fn pretty_verifier_error<'a>(
 struct PrettyVerifierError<'a>(Box, &'a mut Vec);
 
 impl<'a> FuncWriter for PrettyVerifierError<'a> {
+    fn write_ebb_header(
+        &mut self,
+        w: &mut Write,
+        func: &Function,
+        isa: Option<&TargetIsa>,
+        ebb: Ebb,
+        indent: usize,
+    ) -> fmt::Result {
+        pretty_ebb_header_error(w, func, isa, ebb, indent, &mut *self.0, self.1)
+    }
+
     fn write_instruction(
         &mut self,
         w: &mut Write,
@@ -59,7 +70,39 @@ impl<'a> FuncWriter for PrettyVerifierError<'a> {
     }
 }
 
-/// Pretty-print a function verifier error.
+/// Pretty-print a function verifier error for a given EBB.
+fn pretty_ebb_header_error(
+    w: &mut Write,
+    func: &Function,
+    isa: Option<&TargetIsa>,
+    cur_ebb: Ebb,
+    indent: usize,
+    func_w: &mut FuncWriter,
+    errors: &mut Vec,
+) -> fmt::Result {
+    let mut i = 0;
+    let mut printed_ebb = false;
+
+    while i < errors.len() {
+        match errors[i].location {
+            ir::entities::AnyEntity::Ebb(ebb) if ebb == cur_ebb => {
+                if !printed_ebb {
+                    func_w.write_ebb_header(w, func, isa, cur_ebb, indent)?;
+                    printed_ebb = true;
+                }
+                let err = errors.remove(i);
+                print_error(w, indent, cur_ebb.to_string(), err)?;
+            }
+            _ => {
+                i += 1;
+            }
+        }
+    }
+
+    Ok(())
+}
+
+/// Pretty-print a function verifier error for a given instruction.
 fn pretty_instruction_error(
     w: &mut Write,
     func: &Function,
@@ -77,13 +120,11 @@ fn pretty_instruction_error(
     while i != errors.len() {
         match errors[i].location {
             ir::entities::AnyEntity::Inst(inst) if inst == cur_inst => {
-                let err = errors.remove(i);
-
                 if !printed_instr {
                     func_w.write_instruction(w, func, aliases, isa, cur_inst, indent)?;
                     printed_instr = true;
                 }
-
+                let err = errors.remove(i);
                 print_error(w, indent, cur_inst.to_string(), err)?;
             }
             ir::entities::AnyEntity::Inst(_) => i += 1,
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index d790efa920..b4f9b1c76d 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -14,6 +14,16 @@ use std::vec::Vec;
 
 /// A `FuncWriter` used to decorate functions during printing.
 pub trait FuncWriter {
+    /// Write the extended basic block header for the current function.
+    fn write_ebb_header(
+        &mut self,
+        w: &mut Write,
+        func: &Function,
+        isa: Option<&TargetIsa>,
+        ebb: Ebb,
+        indent: usize,
+    ) -> fmt::Result;
+
     /// Write the given `inst` to `w`.
     fn write_instruction(
         &mut self,
@@ -22,7 +32,7 @@ pub trait FuncWriter {
         aliases: &SecondaryMap>,
         isa: Option<&TargetIsa>,
         inst: Inst,
-        ident: usize,
+        indent: usize,
     ) -> fmt::Result;
 
     /// Write the preamble to `w`. By default, this uses `write_entity_definition`.
@@ -108,6 +118,17 @@ impl FuncWriter for PlainWriter {
     ) -> fmt::Result {
         write_instruction(w, func, aliases, isa, inst, indent)
     }
+
+    fn write_ebb_header(
+        &mut self,
+        w: &mut Write,
+        func: &Function,
+        isa: Option<&TargetIsa>,
+        ebb: Ebb,
+        indent: usize,
+    ) -> fmt::Result {
+        write_ebb_header(w, func, isa, ebb, indent)
+    }
 }
 
 /// Write `func` to `w` as equivalent text.
@@ -227,7 +248,7 @@ fn decorate_ebb(
         36
     };
 
-    write_ebb_header(w, func, isa, ebb, indent)?;
+    func_w.write_ebb_header(w, func, isa, ebb, indent)?;
     for a in func.dfg.ebb_params(ebb).iter().cloned() {
         write_value_aliases(w, aliases, a, indent)?;
     }

From 86cc33673befd10feac837986766a0632f2c87b1 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 14 Nov 2018 13:40:22 +0100
Subject: [PATCH 2205/3084] Add binary tests for codegen of loads/stores
 involving r12/r13 on x86;

---
 cranelift/filetests/isa/x86/binary64.clif | 128 ++++++++++++++++++++++
 1 file changed, 128 insertions(+)

diff --git a/cranelift/filetests/isa/x86/binary64.clif b/cranelift/filetests/isa/x86/binary64.clif
index dca189d406..1c43280cb2 100644
--- a/cranelift/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/isa/x86/binary64.clif
@@ -1440,3 +1440,131 @@ ebb2:
 ebb3:
     trap user0
 }
+
+function %r12_r13_loads() {
+ebb0:
+    [-,%r12]            v1 = iconst.i64 0x0123_4567_89ab_cdef
+    [-,%r13]            v2 = iconst.i64 0xfedc_ba98_7654_3210
+    [-,%rax]            v3 = iconst.i64 0x1
+
+    ;; Simple GPR load.
+    ; asm: movq (%r12), %rdx
+    [-,%rdx]            v4 = load.i64 notrap v1 ; bin: 49 8b 14 24
+    ; asm: movq (%r13), %rdx
+    [-,%rdx]            v5 = load.i64 notrap v2 ; bin: 49 8b 55 00
+
+    ;; Load with disp8.
+    ; asm: movq 0x1(%r12), %rdx
+    [-,%rdx]            v6 = load.i64 notrap v1+1 ; bin: 49 8b 54 24 01
+    ; asm: movq 0x1(%r13), %rdx
+    [-,%rdx]            v7 = load.i64 notrap v2+1 ; bin: 49 8b 55 01
+
+    ;; Load with disp32.
+    ; asm: movq 0x100(%r12), %rdx
+    [-,%rdx]            v8 = load.i64 notrap v1+256 ; bin: 49 8b 94 24 00000100
+    ; asm: movq 0x100(%r13), %rdx
+    [-,%rdx]            v9 = load.i64 notrap v2+256 ; bin: 49 8b 95 00000100
+
+    ;; Load for base+index.
+    ; asm: movq (%r12, %rax, 1), %rdx
+    [-,%rdx]            v10 = load_complex.i64 notrap v1+v3 ; bin: 49 8b 14 04
+    ; asm: movq (%r13, %rax, 1), %rdx
+    [-,%rdx]            v11 = load_complex.i64 notrap v2+v3 ; bin: 49 8b 54 05 00
+
+    ;; Now for FP values.
+    ; asm: movss (%r12), %xmm0
+    [-,%xmm0]            v12 = load.f32 notrap v1 ; bin: f3 41 0f 10 04 24
+    ; asm: movss (%r13), %xmm0
+    [-,%xmm0]            v13 = load.f32 notrap v2 ; bin: f3 41 0f 10 45 00
+
+    ;; Load with disp8.
+    ; asm: movss 0x1(%r12), %xmm0
+    [-,%xmm0]            v14 = load.f32 notrap v1+1 ; bin: f3 41 0f 10 44 24 01
+    ; asm: movss 0x1(%r13), %xmm0
+    [-,%xmm0]            v15 = load.f32 notrap v2+1 ; bin: f3 41 0f 10 45 01
+
+    ;; Load with disp32.
+    ; asm: movss 0x100(%r12), %xmm0
+    [-,%xmm0]            v16 = load.f32 notrap v1+256 ; bin: f3 41 0f 10 84 24 00000100
+    ; asm: movss 0x100(%r13), %xmm0
+    [-,%xmm0]            v17 = load.f32 notrap v2+256 ; bin: f3 41 0f 10 85 00000100
+
+    ;; Load for base+index.
+    ; asm: movss (%r12, %rax, 1), %xmm0
+    [-,%xmm0]            v18 = load_complex.f32 notrap v1+v3 ; bin: f3 41 0f 10 04 04
+    ; asm: movss (%r13, %rax, 1), %xmm0
+    [-,%xmm0]            v19 = load_complex.f32 notrap v2+v3 ; bin: f3 41 0f 10 44 05 00
+
+    return
+}
+
+function %r12_r13_stores() {
+ebb0:
+    [-,%r12]            v1 = iconst.i64 0x0123_4567_89ab_cdef
+    [-,%r13]            v2 = iconst.i64 0xfedc_ba98_7654_3210
+    [-,%rax]            v3 = iconst.i64 0x1
+    [-,%xmm0]           v4 = f32const 0x1.0
+
+    ;; Simple GPR load.
+    ; asm: movq %rax, (%r12)
+    store notrap v3, v1; bin: 49 89 04 24
+    ; asm: movq (%r13), %rdx
+    store notrap v3, v2; bin: 49 89 45 00
+
+    ; asm: movq %rax, 0x1(%r12)
+    store notrap v3, v1+1; bin: 49 89 44 24 01
+    ; asm: movq %rax, 0x1(%r13)
+    store notrap v3, v2+1; bin: 49 89 45 01
+
+    ; asm: movq %rax, 0x100(%r12)
+    store notrap v3, v1+256; bin: 49 89 84 24 00000100
+    ; asm: movq %rax, 0x100(%r13)
+    store notrap v3, v2+256; bin: 49 89 85 00000100
+
+    ; asm: movq %rax, (%r12, %rax, 1)
+    store_complex notrap v3, v1+v3; bin: 49 89 04 04
+    ; asm: movq %rax, (%r13, %rax, 1)
+    store_complex notrap v3, v2+v3; bin: 49 89 44 05 00
+
+    ; asm: movb %al, (%r12)
+    istore8 notrap v3, v1; bin: 41 88 04 24
+    ; asm: movb %al, (%r13)
+    istore8 notrap v3, v2; bin: 41 88 45 00
+
+    ; asm: movb %al, 0x1(%r12)
+    istore8 notrap v3, v1+1; bin: 41 88 44 24 01
+    ; asm: movb %al, 0x1(%r13)
+    istore8 notrap v3, v2+1; bin: 41 88 45 01
+
+    ; asm: movb %al, 0x100(%r12)
+    istore8 notrap v3, v1+256; bin: 41 88 84 24 00000100
+    ; asm: movb %al, 0x100(%r13)
+    istore8 notrap v3, v2+256; bin: 41 88 85 00000100
+
+    ; asm: movb %al, (%r12, %rax, 1)
+    istore8_complex notrap v3, v1+v3; bin: 41 88 04 04
+    ; asm: movb %al, (%r13, %rax, 1)
+    istore8_complex notrap v3, v2+v3; bin: 41 88 44 05 00
+
+    ; asm: movss %xmm0, (%r12)
+    store notrap v4, v1; bin: f3 41 0f 11 04 24
+    ; asm: movss %xmm0, (%r13)
+    store notrap v4, v2; bin: f3 41 0f 11 45 00
+
+    ; asm: movss %xmm0, 0x1(%r12)
+    store notrap v4, v1+1; bin: f3 41 0f 11 44 24 01
+    ; asm: movss %xmm0, 0x1(%r13)
+    store notrap v4, v2+1; bin: f3 41 0f 11 45 01
+
+    ; asm: movss %xmm0, 0x100(%r12)
+    store notrap v4, v1+256; bin: f3 41 0f 11 84 24 00000100
+    ; asm: movss %xmm0, 0x100(%r13)
+    store notrap v4, v2+256; bin: f3 41 0f 11 85 00000100
+
+    ; asm: movss %xmm0, (%r12, %rax, 1)
+    store_complex notrap v4, v1+v3; bin: f3 41 0f 11 04 04
+    ; asm: movss %xmm0, (%r13, %rax, 1)
+    store_complex notrap v4, v2+v3; bin: f3 41 0f 11 44 05 00
+
+    return
+}

From c7bc1b7c56657a72a13b98a26516791d2dd4d14a Mon Sep 17 00:00:00 2001
From: Dan Robertson 
Date: Thu, 15 Nov 2018 00:56:59 +0000
Subject: [PATCH 2206/3084] Fix fuzz_reader_parse_test

Fix compilation issues with fuzz_reader_parse_test.
---
 cranelift/fuzz/fuzz_reader_parse_test.rs | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/cranelift/fuzz/fuzz_reader_parse_test.rs b/cranelift/fuzz/fuzz_reader_parse_test.rs
index 8ef582af05..cccf135916 100644
--- a/cranelift/fuzz/fuzz_reader_parse_test.rs
+++ b/cranelift/fuzz/fuzz_reader_parse_test.rs
@@ -4,6 +4,8 @@ extern crate libfuzzer_sys;
 extern crate cranelift_reader;
 use std::str;
 
-fuzz_target!(|data: &[u8]| if let Ok(s) = str::from_utf8(data) {
-    let _ = cranelift_reader::parse_test(s);
+fuzz_target!(|data: &[u8]| {
+    if let Ok(s) = str::from_utf8(data) {
+        let _ = cranelift_reader::parse_test(s, None, None);
+    }
 });

From 641771ac6ae6def74193dfda3e33e26dce398e64 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 12 Nov 2018 05:13:43 -0800
Subject: [PATCH 2207/3084] Add encodings to the instructions in LICM's
 generated preheaders.

When LICM generates a preheader, it needs to add encodings to the
instructions it generates, since it runs after legalization.
---
 cranelift/filetests/licm/basic.clif           |  1 +
 cranelift/filetests/licm/complex.clif         |  1 +
 cranelift/filetests/licm/encoding.clif        | 36 +++++++++++++++++++
 cranelift/filetests/licm/multiple-blocks.clif |  1 +
 cranelift/filetests/licm/nested_loops.clif    |  1 +
 cranelift/filetests/licm/reject.clif          |  5 +--
 lib/codegen/src/context.rs                    |  5 +--
 lib/codegen/src/licm.rs                       |  9 +++--
 lib/filetests/src/test_licm.rs                |  3 +-
 9 files changed, 54 insertions(+), 8 deletions(-)
 create mode 100644 cranelift/filetests/licm/encoding.clif

diff --git a/cranelift/filetests/licm/basic.clif b/cranelift/filetests/licm/basic.clif
index 84d9242169..f409523c70 100644
--- a/cranelift/filetests/licm/basic.clif
+++ b/cranelift/filetests/licm/basic.clif
@@ -1,4 +1,5 @@
 test licm
+target riscv32
 
 function %simple_loop(i32) -> i32 {
 
diff --git a/cranelift/filetests/licm/complex.clif b/cranelift/filetests/licm/complex.clif
index 3b339574a8..0f825e3a8e 100644
--- a/cranelift/filetests/licm/complex.clif
+++ b/cranelift/filetests/licm/complex.clif
@@ -1,4 +1,5 @@
 test licm
+target riscv32
 
 function %complex(i32) -> i32 system_v {
 ebb0(v0: i32):
diff --git a/cranelift/filetests/licm/encoding.clif b/cranelift/filetests/licm/encoding.clif
new file mode 100644
index 0000000000..27a965f55b
--- /dev/null
+++ b/cranelift/filetests/licm/encoding.clif
@@ -0,0 +1,36 @@
+test licm
+target riscv32
+
+; Ensure that instructions emitted by LICM get encodings.
+
+function %simple_loop(i32) -> i32 {
+            ebb0(v0: i32):
+[UJ#1b]       jump ebb1(v0)
+
+            ebb1(v1: i32):
+[Iz#04,%x0]   v2 = iconst.i32 1
+[Iz#04,%x1]   v3 = iconst.i32 2
+[R#0c,%x2]    v4 = iadd v2, v3
+[SBzero#18]   brz v1, ebb2(v1)
+[R#200c,%x5]  v5 = isub v1, v2
+[UJ#1b]       jump ebb1(v5)
+
+            ebb2(v6: i32):
+[Iret#19]     return v6
+}
+
+; check: function
+; nextln:                                 ebb0(v0: i32):
+; nextln: [Iz#04,%x0]                         v2 = iconst.i32 1
+; nextln: [Iz#04,%x1]                         v3 = iconst.i32 2
+; nextln: [R#0c,%x2]                          v4 = iadd v2, v3
+; nextln: [UJ#1b]                             jump ebb1(v0)
+; nextln: 
+; nextln:                                 ebb1(v1: i32):
+; nextln: [SBzero#18]                         brz v1, ebb2(v1)
+; nextln: [R#200c,%x5]                        v5 = isub v1, v2
+; nextln: [UJ#1b]                             jump ebb1(v5)
+; nextln: 
+; nextln:                                 ebb2(v6: i32):
+; nextln: [Iret#19]                           return v6
+; nextln: }
diff --git a/cranelift/filetests/licm/multiple-blocks.clif b/cranelift/filetests/licm/multiple-blocks.clif
index d47f8658d2..de5af884e8 100644
--- a/cranelift/filetests/licm/multiple-blocks.clif
+++ b/cranelift/filetests/licm/multiple-blocks.clif
@@ -1,4 +1,5 @@
 test licm
+target riscv32
 
 function %multiple_blocks(i32) -> i32 {
 
diff --git a/cranelift/filetests/licm/nested_loops.clif b/cranelift/filetests/licm/nested_loops.clif
index 9fb3c08806..f9ade38cf2 100644
--- a/cranelift/filetests/licm/nested_loops.clif
+++ b/cranelift/filetests/licm/nested_loops.clif
@@ -1,4 +1,5 @@
 test licm
+target riscv32
 
 function %nested_loops(i32) -> i32 {
 
diff --git a/cranelift/filetests/licm/reject.clif b/cranelift/filetests/licm/reject.clif
index 052ae2b6f2..a23decb1eb 100644
--- a/cranelift/filetests/licm/reject.clif
+++ b/cranelift/filetests/licm/reject.clif
@@ -1,4 +1,5 @@
 test licm
+target riscv32
 
 function %other_side_effects(i32) -> i32 {
 
@@ -6,9 +7,9 @@ ebb0(v0: i32):
     jump ebb1(v0)
 
 ebb1(v1: i32):
-    regmove.i32 v0, %10 -> %20
+    regmove.i32 v0, %x10 -> %x20
 ; check: ebb1(v1: i32):
-; check: regmove.i32 v0, %10 -> %20
+; check: regmove.i32 v0, %x10 -> %x20
     v2 = iconst.i32 1
     brz v1, ebb2(v1)
     v5 = isub v1, v2
diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs
index ffff2e25ab..a0bd1e680d 100644
--- a/lib/codegen/src/context.rs
+++ b/lib/codegen/src/context.rs
@@ -275,14 +275,15 @@ impl Context {
     }
 
     /// Perform LICM on the function.
-    pub fn licm<'a, FOI: Into>>(&mut self, fisa: FOI) -> CodegenResult<()> {
+    pub fn licm(&mut self, isa: &TargetIsa) -> CodegenResult<()> {
         do_licm(
+            isa,
             &mut self.func,
             &mut self.cfg,
             &mut self.domtree,
             &mut self.loop_analysis,
         );
-        self.verify_if(fisa)
+        self.verify_if(isa)
     }
 
     /// Perform unreachable code elimination.
diff --git a/lib/codegen/src/licm.rs b/lib/codegen/src/licm.rs
index f9954087b4..0cba66b53b 100644
--- a/lib/codegen/src/licm.rs
+++ b/lib/codegen/src/licm.rs
@@ -1,11 +1,12 @@
 //! A Loop Invariant Code Motion optimization pass
 
-use cursor::{Cursor, FuncCursor};
+use cursor::{Cursor, FuncCursor, EncCursor};
 use dominator_tree::DominatorTree;
 use entity::{EntityList, ListPool};
 use flowgraph::{BasicBlock, ControlFlowGraph};
 use fx::FxHashSet;
 use ir::{DataFlowGraph, Ebb, Function, Inst, InstBuilder, Layout, Opcode, Type, Value};
+use isa::TargetIsa;
 use loop_analysis::{Loop, LoopAnalysis};
 use std::vec::Vec;
 use timing;
@@ -14,6 +15,7 @@ use timing;
 /// loop-invariant instructions out of them.
 /// Changes the CFG and domtree in-place during the operation.
 pub fn do_licm(
+    isa: &TargetIsa,
     func: &mut Function,
     cfg: &mut ControlFlowGraph,
     domtree: &mut DominatorTree,
@@ -36,7 +38,7 @@ pub fn do_licm(
             match has_pre_header(&func.layout, cfg, domtree, loop_analysis.loop_header(lp)) {
                 None => {
                     let pre_header =
-                        create_pre_header(loop_analysis.loop_header(lp), func, cfg, domtree);
+                        create_pre_header(isa, loop_analysis.loop_header(lp), func, cfg, domtree);
                     pos = FuncCursor::new(func).at_last_inst(pre_header);
                 }
                 // If there is a natural pre-header we insert new instructions just before the
@@ -60,6 +62,7 @@ pub fn do_licm(
 // Insert a pre-header before the header, modifying the function layout and CFG to reflect it.
 // A jump instruction to the header is placed at the end of the pre-header.
 fn create_pre_header(
+    isa: &TargetIsa,
     header: Ebb,
     func: &mut Function,
     cfg: &mut ControlFlowGraph,
@@ -87,7 +90,7 @@ fn create_pre_header(
         }
     }
     {
-        let mut pos = FuncCursor::new(func).at_top(header);
+        let mut pos = EncCursor::new(func, isa).at_top(header);
         // Inserts the pre-header at the right place in the layout.
         pos.insert_ebb(pre_header);
         pos.next_inst();
diff --git a/lib/filetests/src/test_licm.rs b/lib/filetests/src/test_licm.rs
index 57523cdf59..6c4ef0795d 100644
--- a/lib/filetests/src/test_licm.rs
+++ b/lib/filetests/src/test_licm.rs
@@ -33,12 +33,13 @@ impl SubTest for TestLICM {
     }
 
     fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> {
+        let isa = context.isa.expect("LICM needs an ISA");
         let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned());
 
         comp_ctx.flowgraph();
         comp_ctx.compute_loop_analysis();
         comp_ctx
-            .licm(context.flags_or_isa())
+            .licm(isa)
             .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?;
 
         let text = comp_ctx.func.display(context.isa).to_string();

From 0fed78e06365f5df225278f4ab8d513d08c7009c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 12 Nov 2018 05:17:54 -0800
Subject: [PATCH 2208/3084] Don't allow loop preheaders to have critical edges.

If the block which would be a preheader for a loop has other successors,
don't hoist instructions into it. Instead create a dedicated preheader.
---
 cranelift/filetests/licm/complex.clif       | 76 +++++++++++----------
 cranelift/filetests/licm/critical-edge.clif | 42 ++++++++++++
 lib/codegen/src/licm.rs                     | 19 +++---
 3 files changed, 94 insertions(+), 43 deletions(-)
 create mode 100644 cranelift/filetests/licm/critical-edge.clif

diff --git a/cranelift/filetests/licm/complex.clif b/cranelift/filetests/licm/complex.clif
index 0f825e3a8e..c7d26f9a33 100644
--- a/cranelift/filetests/licm/complex.clif
+++ b/cranelift/filetests/licm/complex.clif
@@ -3,43 +3,43 @@ target riscv32
 
 function %complex(i32) -> i32 system_v {
 ebb0(v0: i32):
-    jump ebb1(v0)
+[UJ#1b]    jump ebb1(v0)
 
-ebb1(v1: i32):
-    v2 = iconst.i32 1
-    v3 = iconst.i32 4
-    v4 = iadd v2, v1
-    brz v1, ebb2(v2)
-    jump ebb4(v4)
+         ebb1(v1: i32):
+            v2 = iconst.i32 1
+            v3 = iconst.i32 4
+            v4 = iadd v2, v1
+[SBzero#18] brz v1, ebb2(v2)
+[UJ#1b]     jump ebb4(v4)
 
-ebb2(v5: i32):
-    v6 = iconst.i32 2
-    v7 = iadd v5, v4
-    v8 = iadd v6, v1
-    jump ebb3(v8)
+         ebb2(v5: i32):
+            v6 = iconst.i32 2
+            v7 = iadd v5, v4
+            v8 = iadd v6, v1
+[UJ#1b]   jump ebb3(v8)
 
-ebb3(v9: i32):
-    v10 = iadd v9, v5
-    v11 = iadd.i32 v1, v4
-    brz.i32 v1, ebb2(v9)
-    jump ebb6(v10)
+         ebb3(v9: i32):
+            v10 = iadd v9, v5
+            v11 = iadd.i32 v1, v4
+[SBzero#18] brz.i32 v1, ebb2(v9)
+[UJ#1b]   jump ebb6(v10)
 
-ebb4(v12: i32):
-    v13 = iconst.i32 3
-    v14 = iadd v12, v13
-    v15 = iadd.i32 v4, v13
-    jump ebb5(v13)
+         ebb4(v12: i32):
+            v13 = iconst.i32 3
+            v14 = iadd v12, v13
+            v15 = iadd.i32 v4, v13
+[UJ#1b]   jump ebb5(v13)
 
-ebb5(v16: i32):
-    v17 = iadd.i32 v14, v4
-    brz.i32 v1, ebb4(v16)
-    jump ebb6(v16)
+         ebb5(v16: i32):
+            v17 = iadd.i32 v14, v4
+[SBzero#18] brz.i32 v1, ebb4(v16)
+[UJ#1b]   jump ebb6(v16)
 
-ebb6(v18: i32):
-    v19 = iadd v18, v2
-    v20 = iadd.i32 v2, v3
-    brz.i32 v1, ebb1(v20)
-    return v19
+         ebb6(v18: i32):
+            v19 = iadd v18, v2
+            v20 = iadd.i32 v2, v3
+[SBzero#18] brz.i32 v1, ebb1(v20)
+[Iret#19]   return v19
 }
 
 ; sameln: function %complex
@@ -53,11 +53,13 @@ ebb6(v18: i32):
 ; nextln: 
 ; nextln: ebb1(v1: i32):
 ; nextln:     v4 = iadd.i32 v2, v1
+; nextln:     brz v1, ebb7(v2)
+; nextln:     jump ebb8(v4)
+; nextln: 
+; nextln: ebb7(v21: i32):
 ; nextln:     v8 = iadd.i32 v6, v1
-; nextln:     v11 = iadd v1, v4
-; nextln:     brz v1, ebb2(v2)
-; nextln:     v15 = iadd v4, v13
-; nextln:     jump ebb4(v4)
+; nextln:     v11 = iadd.i32 v1, v4
+; nextln:     jump ebb2(v21)
 ; nextln: 
 ; nextln: ebb2(v5: i32):
 ; nextln:     v7 = iadd v5, v4
@@ -68,6 +70,10 @@ ebb6(v18: i32):
 ; nextln:     brz.i32 v1, ebb2(v9)
 ; nextln:     jump ebb6(v10)
 ; nextln: 
+; nextln: ebb8(v22: i32):
+; nextln:     v15 = iadd.i32 v4, v13
+; nextln:     jump ebb4(v22)
+; nextln: 
 ; nextln: ebb4(v12: i32):
 ; nextln:     v14 = iadd v12, v13
 ; nextln:     jump ebb5(v13)
diff --git a/cranelift/filetests/licm/critical-edge.clif b/cranelift/filetests/licm/critical-edge.clif
new file mode 100644
index 0000000000..1c69f6364e
--- /dev/null
+++ b/cranelift/filetests/licm/critical-edge.clif
@@ -0,0 +1,42 @@
+test licm
+target riscv32
+
+; The loop in this function is entered from a critical edge.
+
+function %critical_edge(i32, i32) -> i32 {
+
+            ebb0(v0: i32, v7: i32):
+[SBzero#38]   brnz v7, ebb1(v0)
+[Iret#19]     return v0
+
+            ebb1(v1: i32):
+              v2 = iconst.i32 1
+              v3 = iconst.i32 2
+              v4 = iadd v2, v3
+[SBzero#18]   brz v1, ebb2(v1)
+              v5 = isub v1, v2
+[UJ#1b]       jump ebb1(v5)
+
+            ebb2(v6: i32):
+[Iret#19]     return v6
+
+}
+; sameln: function %critical_edge
+; nextln: ebb0(v0: i32, v7: i32):
+; nextln:     brnz v7, ebb3(v0)
+; nextln:     return v0
+; nextln: 
+; nextln: ebb3(v8: i32):
+; nextln:     v2 = iconst.i32 1
+; nextln:     v3 = iconst.i32 2
+; nextln:     v4 = iadd v2, v3
+; nextln:     jump ebb1(v8)
+; nextln: 
+; nextln: ebb1(v1: i32):
+; nextln:     brz v1, ebb2(v1)
+; nextln:     v5 = isub v1, v2
+; nextln:     jump ebb1(v5)
+; nextln: 
+; nextln: ebb2(v6: i32):
+; nextln:     return v6
+; nextln: }
diff --git a/lib/codegen/src/licm.rs b/lib/codegen/src/licm.rs
index 0cba66b53b..b203fdfe9e 100644
--- a/lib/codegen/src/licm.rs
+++ b/lib/codegen/src/licm.rs
@@ -1,6 +1,6 @@
 //! A Loop Invariant Code Motion optimization pass
 
-use cursor::{Cursor, FuncCursor, EncCursor};
+use cursor::{Cursor, EncCursor, FuncCursor};
 use dominator_tree::DominatorTree;
 use entity::{EntityList, ListPool};
 use flowgraph::{BasicBlock, ControlFlowGraph};
@@ -111,21 +111,24 @@ fn has_pre_header(
     header: Ebb,
 ) -> Option<(Ebb, Inst)> {
     let mut result = None;
-    let mut found = false;
     for BasicBlock {
         ebb: pred_ebb,
-        inst: last_inst,
+        inst: branch_inst,
     } in cfg.pred_iter(header)
     {
         // We only count normal edges (not the back edges)
-        if !domtree.dominates(header, last_inst, layout) {
-            if found {
+        if !domtree.dominates(header, branch_inst, layout) {
+            if result.is_some() {
                 // We have already found one, there are more than one
                 return None;
-            } else {
-                result = Some((pred_ebb, last_inst));
-                found = true;
             }
+            if branch_inst != layout.last_inst(pred_ebb).unwrap()
+                || cfg.succ_iter(pred_ebb).nth(1).is_some()
+            {
+                // It's along a critical edge, so don't use it.
+                return None;
+            }
+            result = Some((pred_ebb, branch_inst));
         }
     }
     result

From 48186b3199c00bd88cd0acf1a1bff3326fc648b1 Mon Sep 17 00:00:00 2001
From: Syrus Akbary 
Date: Sun, 18 Nov 2018 23:47:58 -0800
Subject: [PATCH 2209/3084] Make TargetIsa thread-safe

---
 lib/codegen/src/isa/mod.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs
index f304cf09c3..b4bf97d334 100644
--- a/lib/codegen/src/isa/mod.rs
+++ b/lib/codegen/src/isa/mod.rs
@@ -196,7 +196,7 @@ impl TargetFrontendConfig {
 
 /// Methods that are specialized to a target ISA. Implies a Display trait that shows the
 /// shared flags, as well as any isa-specific flags.
-pub trait TargetIsa: fmt::Display {
+pub trait TargetIsa: fmt::Display + Sync {
     /// Get the name of this ISA.
     fn name(&self) -> &'static str;
 

From b11646aba3541c7ce26995992748653b317c5642 Mon Sep 17 00:00:00 2001
From: Aaron Power 
Date: Fri, 9 Nov 2018 12:32:29 +0000
Subject: [PATCH 2210/3084] Fix bit count bug in emit_small_{memcpy, memset,
 memmove}.

---
 lib/frontend/src/frontend.rs | 66 ++++++++++++++++++++++++++++++++++--
 1 file changed, 63 insertions(+), 3 deletions(-)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 0c595a585f..692bc48de6 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -605,12 +605,12 @@ impl<'a> FunctionBuilder<'a> {
             return;
         }
 
+        let int_type = Type::int((access_size * 8) as u16).unwrap();
         let mut flags = MemFlags::new();
         flags.set_aligned();
 
         for i in 0..load_and_store_amount {
             let offset = (access_size * i) as i32;
-            let int_type = Type::int(access_size as u16).unwrap();
             let value = self.ins().load(int_type, flags, src, offset);
             self.ins().store(flags, value, dest, offset);
         }
@@ -679,7 +679,7 @@ impl<'a> FunctionBuilder<'a> {
             flags.set_aligned();
 
             let ch = ch as u64;
-            let int_type = Type::int(access_size as u16).unwrap();
+            let int_type = Type::int((access_size * 8) as u16).unwrap();
             let raw_value = if int_type == types::I64 {
                 (ch << 32) | (ch << 16) | (ch << 8) | ch
             } else if int_type == types::I32 {
@@ -765,7 +765,10 @@ impl<'a> FunctionBuilder<'a> {
         let registers: Vec<_> = (0..load_and_store_amount)
             .map(|i| {
                 let offset = (access_size * i) as i32;
-                (self.ins().load(types::I8, flags, src, offset), offset)
+                (
+                    self.ins().load(config.pointer_type(), flags, src, offset),
+                    offset,
+                )
             }).collect();
 
         for (value, offset) in registers {
@@ -987,6 +990,63 @@ ebb0:
         );
     }
 
+    #[test]
+    fn small_memcpy() {
+        use cranelift_codegen::{isa, settings};
+        use std::str::FromStr;
+
+        let shared_builder = settings::builder();
+        let shared_flags = settings::Flags::new(shared_builder);
+
+        let triple = ::target_lexicon::Triple::from_str("arm").expect("Couldn't create arm triple");
+
+        let target = isa::lookup(triple)
+            .ok()
+            .map(|b| b.finish(shared_flags))
+            .expect("This test requires arm support.");
+
+        let mut sig = Signature::new(target.default_call_conv());
+        sig.returns.push(AbiParam::new(I32));
+
+        let mut fn_ctx = FunctionBuilderContext::new();
+        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+        {
+            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
+
+            let block0 = builder.create_ebb();
+            let x = Variable::new(0);
+            let y = Variable::new(16);
+            builder.declare_var(x, target.pointer_type());
+            builder.declare_var(y, target.pointer_type());
+            builder.append_ebb_params_for_function_params(block0);
+            builder.switch_to_block(block0);
+
+            let src = builder.use_var(x);
+            let dest = builder.use_var(y);
+            let size = 8;
+            builder.emit_small_memcpy(target.frontend_config(), dest, src, size, 8, 8);
+            builder.ins().return_(&[dest]);
+
+            builder.seal_all_blocks();
+            builder.finalize();
+        }
+
+        assert_eq!(
+            func.display(None).to_string(),
+            "function %sample() -> i32 system_v {
+ebb0:
+    v4 = iconst.i32 0
+    v1 -> v4
+    v3 = iconst.i32 0
+    v0 -> v3
+    v2 = load.i64 aligned v0
+    store aligned v2, v1
+    return v1
+}
+"
+        );
+    }
+
     #[test]
     fn test_greatest_divisible_power_of_two() {
         assert_eq!(64, greatest_divisible_power_of_two(64));

From 737fde04a88bbcc424a8f2b2672353e641437830 Mon Sep 17 00:00:00 2001
From: Syrus Akbary 
Date: Sun, 25 Nov 2018 05:10:58 -0800
Subject: [PATCH 2211/3084] Improved imports (#612)

* Impoved support for wasm global imports

* Refactored parse_import_section improving readability

* Improved support for wasm table imports

* Improved support for wasm memory imports

* Improved formatting

* Added DefinedGlobalIndex, DefinedMemoryIndex, DefinedTableIndex structs
---
 lib/wasm/src/environ/dummy.rs       | 36 ++++++++++++
 lib/wasm/src/environ/spec.rs        | 11 ++++
 lib/wasm/src/lib.rs                 |  4 +-
 lib/wasm/src/sections_translator.rs | 89 +++++++++++++++--------------
 lib/wasm/src/translation_utils.rs   | 15 +++++
 5 files changed, 110 insertions(+), 45 deletions(-)

diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index ae2183efb9..7ed6e02969 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -52,6 +52,15 @@ pub struct DummyModuleInfo {
     /// Module and field names of imported functions as provided by `declare_func_import`.
     pub imported_funcs: Vec<(String, String)>,
 
+    /// Module and field names of imported globals as provided by `declare_global_import`.
+    pub imported_globals: Vec<(String, String)>,
+
+    /// Module and field names of imported tables as provided by `declare_table_import`.
+    pub imported_tables: Vec<(String, String)>,
+
+    /// Module and field names of imported memories as provided by `declare_memory_import`.
+    pub imported_memories: Vec<(String, String)>,
+
     /// Functions, imported and local.
     pub functions: PrimaryMap>,
 
@@ -78,6 +87,9 @@ impl DummyModuleInfo {
             config,
             signatures: PrimaryMap::new(),
             imported_funcs: Vec::new(),
+            imported_globals: Vec::new(),
+            imported_tables: Vec::new(),
+            imported_memories: Vec::new(),
             functions: PrimaryMap::new(),
             function_bodies: PrimaryMap::new(),
             tables: PrimaryMap::new(),
@@ -375,6 +387,13 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         self.info.globals.push(Exportable::new(global));
     }
 
+    fn declare_global_import(&mut self, global: Global, module: &'data str, field: &'data str) {
+        self.info.globals.push(Exportable::new(global));
+        self.info
+            .imported_globals
+            .push((String::from(module), String::from(field)));
+    }
+
     fn get_global(&self, global_index: GlobalIndex) -> &Global {
         &self.info.globals[global_index].entity
     }
@@ -382,6 +401,14 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
     fn declare_table(&mut self, table: Table) {
         self.info.tables.push(Exportable::new(table));
     }
+
+    fn declare_table_import(&mut self, table: Table, module: &'data str, field: &'data str) {
+        self.info.tables.push(Exportable::new(table));
+        self.info
+            .imported_tables
+            .push((String::from(module), String::from(field)));
+    }
+
     fn declare_table_elements(
         &mut self,
         _table_index: TableIndex,
@@ -391,9 +418,18 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
     ) {
         // We do nothing
     }
+
     fn declare_memory(&mut self, memory: Memory) {
         self.info.memories.push(Exportable::new(memory));
     }
+
+    fn declare_memory_import(&mut self, memory: Memory, module: &'data str, field: &'data str) {
+        self.info.memories.push(Exportable::new(memory));
+        self.info
+            .imported_memories
+            .push((String::from(module), String::from(field)));
+    }
+
     fn declare_data_initialization(
         &mut self,
         _memory_index: MemoryIndex,
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index d5518e037a..705c540052 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -268,11 +268,18 @@ pub trait ModuleEnvironment<'data> {
     /// Declares a global to the environment.
     fn declare_global(&mut self, global: Global);
 
+    /// Declares a global import to the environment.
+    fn declare_global_import(&mut self, global: Global, module: &'data str, field: &'data str);
+
     /// Return the global for the given global index.
     fn get_global(&self, global_index: GlobalIndex) -> &Global;
 
     /// Declares a table to the environment.
     fn declare_table(&mut self, table: Table);
+
+    /// Declares a table import to the environment.
+    fn declare_table_import(&mut self, table: Table, module: &'data str, field: &'data str);
+
     /// Fills a declared table with references to functions in the module.
     fn declare_table_elements(
         &mut self,
@@ -283,6 +290,10 @@ pub trait ModuleEnvironment<'data> {
     );
     /// Declares a memory to the environment
     fn declare_memory(&mut self, memory: Memory);
+
+    /// Declares a memory import to the environment.
+    fn declare_memory_import(&mut self, memory: Memory, module: &'data str, field: &'data str);
+
     /// Fills a declared memory with bytes at module instantiation.
     fn declare_data_initialization(
         &mut self,
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index b119d2df8f..86f7c2a769 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -66,8 +66,8 @@ pub use environ::{
 pub use func_translator::FuncTranslator;
 pub use module_translator::translate_module;
 pub use translation_utils::{
-    DefinedFuncIndex, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex,
-    SignatureIndex, Table, TableIndex,
+    DefinedFuncIndex, DefinedGlobalIndex, DefinedMemoryIndex, DefinedTableIndex, FuncIndex, Global,
+    GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
 };
 
 #[cfg(not(feature = "std"))]
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index e8468d9d4c..1fbb41fcd6 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -60,58 +60,61 @@ pub fn parse_import_section<'data>(
     environ: &mut ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
     for entry in imports {
-        match entry? {
-            Import {
-                module,
-                field,
-                ty: ImportSectionEntryType::Function(sig),
-            } => {
-                // The input has already been validated, so we should be able to
-                // assume valid UTF-8 and use `from_utf8_unchecked` if performance
-                // becomes a concern here.
-                let module_name = from_utf8(module).unwrap();
-                let field_name = from_utf8(field).unwrap();
+        let import = entry?;
+
+        // The input has already been validated, so we should be able to
+        // assume valid UTF-8 and use `from_utf8_unchecked` if performance
+        // becomes a concern here.
+        let module_name = from_utf8(import.module).unwrap();
+        let field_name = from_utf8(import.field).unwrap();
+
+        match import.ty {
+            ImportSectionEntryType::Function(sig) => {
                 environ.declare_func_import(
                     SignatureIndex::new(sig as usize),
                     module_name,
                     field_name,
                 );
             }
-            Import {
-                ty:
-                    ImportSectionEntryType::Memory(MemoryType {
-                        limits: ref memlimits,
+            ImportSectionEntryType::Memory(MemoryType {
+                limits: ref memlimits,
+                shared,
+            }) => {
+                environ.declare_memory_import(
+                    Memory {
+                        pages_count: memlimits.initial as usize,
+                        maximum: memlimits.maximum.map(|x| x as usize),
                         shared,
-                    }),
-                ..
-            } => {
-                environ.declare_memory(Memory {
-                    pages_count: memlimits.initial as usize,
-                    maximum: memlimits.maximum.map(|x| x as usize),
-                    shared,
-                });
+                    },
+                    module_name,
+                    field_name,
+                );
             }
-            Import {
-                ty: ImportSectionEntryType::Global(ref ty),
-                ..
-            } => {
-                environ.declare_global(Global {
-                    ty: type_to_type(ty.content_type).unwrap(),
-                    mutability: ty.mutable,
-                    initializer: GlobalInit::Import(),
-                });
+            ImportSectionEntryType::Global(ref ty) => {
+                environ.declare_global_import(
+                    Global {
+                        ty: type_to_type(ty.content_type).unwrap(),
+                        mutability: ty.mutable,
+                        initializer: GlobalInit::Import(),
+                    },
+                    module_name,
+                    field_name,
+                );
+            }
+            ImportSectionEntryType::Table(ref tab) => {
+                environ.declare_table_import(
+                    Table {
+                        ty: match type_to_type(tab.element_type) {
+                            Ok(t) => TableElementType::Val(t),
+                            Err(()) => TableElementType::Func(),
+                        },
+                        size: tab.limits.initial as usize,
+                        maximum: tab.limits.maximum.map(|x| x as usize),
+                    },
+                    module_name,
+                    field_name,
+                );
             }
-            Import {
-                ty: ImportSectionEntryType::Table(ref tab),
-                ..
-            } => environ.declare_table(Table {
-                ty: match type_to_type(tab.element_type) {
-                    Ok(t) => TableElementType::Val(t),
-                    Err(()) => TableElementType::Func(),
-                },
-                size: tab.limits.initial as usize,
-                maximum: tab.limits.maximum.map(|x| x as usize),
-            }),
         }
     }
     Ok(())
diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index cc2926dad1..95b24b9f0f 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -13,6 +13,21 @@ entity_impl!(FuncIndex);
 pub struct DefinedFuncIndex(u32);
 entity_impl!(DefinedFuncIndex);
 
+/// Index type of a defined table inside the WebAssembly module.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
+pub struct DefinedTableIndex(u32);
+entity_impl!(DefinedTableIndex);
+
+/// Index type of a defined memory inside the WebAssembly module.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
+pub struct DefinedMemoryIndex(u32);
+entity_impl!(DefinedMemoryIndex);
+
+/// Index type of a defined global inside the WebAssembly module.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
+pub struct DefinedGlobalIndex(u32);
+entity_impl!(DefinedGlobalIndex);
+
 /// Index type of a table (imported or defined) inside the WebAssembly module.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
 pub struct TableIndex(u32);

From 54c388d870d8c26e1a4fb5412f806b3f6a614d95 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 26 Nov 2018 22:18:18 -0800
Subject: [PATCH 2212/3084] Make the verifier output even prettier (#599)

* Fix verifier printing to print instruction encodings consistently.

Use `FuncWriter::write_instruction` for all instructions so that
encodings are printed consistently.

* Make use-before-def errors mention the relevant value.

* When there are verifier errors, print a message at the end.

* Make verifier errors prettier.

Fix the length of the "^~~~~" to match the printed entity, and print the
error messsage on its own line.

* Clean up "test verifier" failure messages.

* Tidy the uses-value-from-itself error.

The use instruction is the same as the def instruction, so don't print
both. Also, the use instruction is already being printed at the
beginning, so don't print it again at the end.
---
 .../verifier/defs_dominates_uses.clif         |   4 +-
 lib/codegen/src/print_errors.rs               | 112 ++++++++++--------
 lib/codegen/src/verifier/mod.rs               |  11 +-
 lib/filetests/src/test_verifier.rs            |   8 +-
 4 files changed, 74 insertions(+), 61 deletions(-)

diff --git a/cranelift/filetests/verifier/defs_dominates_uses.clif b/cranelift/filetests/verifier/defs_dominates_uses.clif
index ba3dc98eb4..fcbeb14816 100644
--- a/cranelift/filetests/verifier/defs_dominates_uses.clif
+++ b/cranelift/filetests/verifier/defs_dominates_uses.clif
@@ -4,13 +4,13 @@ test verifier
 
 function %non_dominating(i32) -> i32 system_v {
 ebb0(v0: i32):
-    v1 = iadd.i32 v2, v0   ; error: uses value from non-dominating
+    v1 = iadd.i32 v2, v0   ; error: uses value v2 from non-dominating
     v2 = iadd.i32 v1, v0
     return v2
 }
 
 function %inst_uses_its_own_values(i32) -> i32 system_v {
 ebb0(v0: i32):
-    v1 = iadd.i32 v1, v0   ; error: uses value from itself
+    v1 = iadd.i32 v1, v0   ; error: uses value v1 from itself
     return v1
 }
diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 43bbc2cea4..73c8cf809a 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -23,6 +23,7 @@ pub fn pretty_verifier_error<'a>(
 ) -> String {
     let mut errors = errors.0;
     let mut w = String::new();
+    let num_errors = errors.len();
 
     decorate_function(
         &mut PrettyVerifierError(func_w.unwrap_or_else(|| Box::new(PlainWriter)), &mut errors),
@@ -30,6 +31,14 @@ pub fn pretty_verifier_error<'a>(
         func,
         isa,
     ).unwrap();
+
+    writeln!(
+        w,
+        "\n; {} verifier error{} detected (see above). Compilation aborted.",
+        num_errors,
+        if num_errors == 1 { "" } else { "s" }
+    ).unwrap();
+
     w
 }
 
@@ -80,25 +89,31 @@ fn pretty_ebb_header_error(
     func_w: &mut FuncWriter,
     errors: &mut Vec,
 ) -> fmt::Result {
-    let mut i = 0;
-    let mut printed_ebb = false;
+    let mut s = String::new();
+    func_w.write_ebb_header(&mut s, func, isa, cur_ebb, indent)?;
+    write!(w, "{}", s)?;
 
-    while i < errors.len() {
+    // TODO: Use drain_filter here when it gets stabilized
+    let mut i = 0;
+    let mut printed_error = false;
+    while i != errors.len() {
         match errors[i].location {
             ir::entities::AnyEntity::Ebb(ebb) if ebb == cur_ebb => {
-                if !printed_ebb {
-                    func_w.write_ebb_header(w, func, isa, cur_ebb, indent)?;
-                    printed_ebb = true;
+                if !printed_error {
+                    print_arrow(w, &s)?;
+                    printed_error = true;
                 }
                 let err = errors.remove(i);
-                print_error(w, indent, cur_ebb.to_string(), err)?;
-            }
-            _ => {
-                i += 1;
+                print_error(w, err)?;
             }
+            _ => i += 1,
         }
     }
 
+    if printed_error {
+        w.write_char('\n')?;
+    }
+
     Ok(())
 }
 
@@ -113,35 +128,29 @@ fn pretty_instruction_error(
     func_w: &mut FuncWriter,
     errors: &mut Vec,
 ) -> fmt::Result {
+    let mut s = String::new();
+    func_w.write_instruction(&mut s, func, aliases, isa, cur_inst, indent)?;
+    write!(w, "{}", s)?;
+
     // TODO: Use drain_filter here when it gets stabilized
     let mut i = 0;
-    let mut printed_instr = false;
-
+    let mut printed_error = false;
     while i != errors.len() {
         match errors[i].location {
             ir::entities::AnyEntity::Inst(inst) if inst == cur_inst => {
-                if !printed_instr {
-                    func_w.write_instruction(w, func, aliases, isa, cur_inst, indent)?;
-                    printed_instr = true;
+                if !printed_error {
+                    print_arrow(w, &s)?;
+                    printed_error = true;
                 }
                 let err = errors.remove(i);
-                print_error(w, indent, cur_inst.to_string(), err)?;
+                print_error(w, err)?;
             }
-            ir::entities::AnyEntity::Inst(_) => i += 1,
-            _ => unreachable!(),
+            _ => i += 1,
         }
     }
 
-    if printed_instr {
+    if printed_error {
         w.write_char('\n')?;
-    } else {
-        writeln!(
-            w,
-            "{1:0$}{2}",
-            indent,
-            "",
-            func.dfg.display_inst(cur_inst, isa)
-        )?;
     }
 
     Ok(())
@@ -155,45 +164,54 @@ fn pretty_preamble_error(
     func_w: &mut FuncWriter,
     errors: &mut Vec,
 ) -> fmt::Result {
+    let mut s = String::new();
+    func_w.write_entity_definition(&mut s, func, entity, value)?;
+    write!(w, "{}", s)?;
+
     // TODO: Use drain_filter here when it gets stabilized
-    let indent = 4;
-
     let mut i = 0;
-    let mut printed_entity = false;
-
+    let mut printed_error = false;
     while i != errors.len() {
         if entity == errors[i].location {
-            let err = errors.remove(i);
-
-            if !printed_entity {
-                func_w.write_entity_definition(w, func, entity, value)?;
-                printed_entity = true;
+            if !printed_error {
+                print_arrow(w, &s)?;
+                printed_error = true;
             }
-
-            print_error(w, indent, entity.to_string(), err)?;
+            let err = errors.remove(i);
+            print_error(w, err)?;
         } else {
             i += 1
         }
     }
 
-    if printed_entity {
+    if printed_error {
         w.write_char('\n')?;
-    } else {
-        func_w.write_entity_definition(w, func, entity, value)?;
     }
 
     Ok(())
 }
 
-/// Prints ;   ^~~~~~ verifier [ERROR BODY]
-fn print_error(w: &mut Write, indent: usize, s: String, err: VerifierError) -> fmt::Result {
-    let indent = if indent < 1 { 0 } else { indent - 1 };
+/// Prints:
+///    ;   ^~~~~~
+fn print_arrow(w: &mut Write, entity: &str) -> fmt::Result {
+    write!(w, ";")?;
 
-    write!(w, ";{1:0$}^", indent, "")?;
-    for _c in s.chars() {
+    let indent = entity.len() - entity.trim_start().len();
+    if indent != 0 {
+        write!(w, "{1:0$}^", indent - 1, "")?;
+    }
+
+    for _ in 0..entity.trim().len() - 1 {
         write!(w, "~")?;
     }
-    writeln!(w, " verifier {}", err.to_string())?;
+
+    writeln!(w)
+}
+
+/// Prints:
+///    ; error: [ERROR BODY]
+fn print_error(w: &mut Write, err: VerifierError) -> fmt::Result {
+    writeln!(w, "; error: {}", err.to_string())?;
     Ok(())
 }
 
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 1d25cd9b23..84244cc590 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -861,18 +861,13 @@ impl<'a> Verifier<'a> {
                         return fatal!(
                             errors,
                             loc_inst,
-                            "uses value from non-dominating {}",
+                            "uses value {} from non-dominating {}",
+                            v,
                             def_inst
                         );
                     }
                     if def_inst == loc_inst {
-                        return fatal!(
-                            errors,
-                            loc_inst,
-                            "uses value from itself {},  {}",
-                            def_inst,
-                            loc_inst
-                        );
+                        return fatal!(errors, loc_inst, "uses value {} from itself", v);
                     }
                 }
             }
diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs
index 842e720cc4..90dcad7e20 100644
--- a/lib/filetests/src/test_verifier.rs
+++ b/lib/filetests/src/test_verifier.rs
@@ -62,7 +62,7 @@ impl SubTest for TestVerifier {
                 let mut errors = errors.0;
                 let mut msg = String::new();
 
-                // for each expected error, find a suitable match
+                // For each expected error, find a suitable match.
                 for expect in expected {
                     let pos = errors
                         .iter()
@@ -70,7 +70,7 @@ impl SubTest for TestVerifier {
 
                     match pos {
                         None => {
-                            write!(msg, "expected error {}", expect.0).unwrap();
+                            writeln!(msg, "  expected error {}: {}", expect.0, expect.1).unwrap();
                         }
                         Some(pos) => {
                             errors.swap_remove(pos);
@@ -78,9 +78,9 @@ impl SubTest for TestVerifier {
                     }
                 }
 
-                // report remaining errors
+                // Report remaining errors.
                 for err in errors {
-                    write!(msg, "unexpected error {}", err).unwrap();
+                    writeln!(msg, "unexpected error {}", err).unwrap();
                 }
 
                 if msg.is_empty() {

From c17579e7ece1ed0e4748330e78818270c0ab13e2 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 16 Nov 2018 07:03:44 -0800
Subject: [PATCH 2213/3084] Bump version to 0.24.0

---
 cranelift/Cargo.toml        | 28 ++++++++++++++--------------
 cranelift/publish-all.sh    |  2 +-
 lib/bforest/Cargo.toml      |  4 ++--
 lib/codegen/Cargo.toml      |  8 ++++----
 lib/codegen/meta/Cargo.toml |  4 ++--
 lib/entity/Cargo.toml       |  2 +-
 lib/faerie/Cargo.toml       |  6 +++---
 lib/filetests/Cargo.toml    |  8 ++++----
 lib/frontend/Cargo.toml     |  4 ++--
 lib/module/Cargo.toml       |  6 +++---
 lib/native/Cargo.toml       |  4 ++--
 lib/preopt/Cargo.toml       |  6 +++---
 lib/reader/Cargo.toml       |  4 ++--
 lib/serde/Cargo.toml        |  6 +++---
 lib/simplejit/Cargo.toml    | 14 +++++++-------
 lib/umbrella/Cargo.toml     |  6 +++---
 lib/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 8aeb8538d6..eab8e80f5b 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.23.0"
+version = "0.24.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,19 +14,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.23.0" }
-cranelift-entity = { path = "lib/entity", version = "0.23.0" }
-cranelift-reader = { path = "lib/reader", version = "0.23.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.23.0" }
-cranelift-serde = { path = "lib/serde", version = "0.23.0", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.23.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.23.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.23.0" }
-cranelift-module = { path = "lib/module", version = "0.23.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.23.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.23.0" }
-cranelift-preopt = { path = "lib/preopt", version = "0.23.0" }
-cranelift = { path = "lib/umbrella", version = "0.23.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.24.0" }
+cranelift-entity = { path = "lib/entity", version = "0.24.0" }
+cranelift-reader = { path = "lib/reader", version = "0.24.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.24.0" }
+cranelift-serde = { path = "lib/serde", version = "0.24.0", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.24.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.24.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.24.0" }
+cranelift-module = { path = "lib/module", version = "0.24.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.24.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.24.0" }
+cranelift-preopt = { path = "lib/preopt", version = "0.24.0" }
+cranelift = { path = "lib/umbrella", version = "0.24.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 3643cbec42..6e6282b98b 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.23.0"
+version="0.24.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index 291fdf63ee..ea4b726711 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.23.0"
+version = "0.24.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 keywords = ["btree", "forest", "set", "map"]
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.23.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.24.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 679555bc03..5735f8efbc 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.23.0"
+version = "0.24.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.23.0", default-features = false }
-cranelift-bforest = { path = "../bforest", version = "0.23.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.24.0", default-features = false }
+cranelift-bforest = { path = "../bforest", version = "0.24.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -25,7 +25,7 @@ log = { version = "0.4.4", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.23.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.24.0" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 70073e4492..0de1d9fb72 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,14 +1,14 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.23.0"
+version = "0.24.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-entity = { path = "../../entity", version = "0.23.0" }
+cranelift-entity = { path = "../../entity", version = "0.24.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index a7fe521acc..b9c1f5125a 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.23.0"
+version = "0.24.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 334cf488a5..b60435e3ec 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.23.0"
+version = "0.24.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.23.0" }
-cranelift-module = { path = "../module", version = "0.23.0" }
+cranelift-codegen = { path = "../codegen", version = "0.24.0" }
+cranelift-module = { path = "../module", version = "0.24.0" }
 faerie = "0.6.0"
 goblin = "0.0.19"
 failure = "0.1.2"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index fc1ddff8a3..293e23b68c 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.23.0"
+version = "0.24.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,9 +9,9 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.23.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../reader", version = "0.23.0" }
-cranelift-preopt = { path = "../preopt", version = "0.23.0" }
+cranelift-codegen = { path = "../codegen", version = "0.24.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../reader", version = "0.24.0" }
+cranelift-preopt = { path = "../preopt", version = "0.24.0" }
 file-per-thread-logger = "0.1.1"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 215fb3bdf3..bbf02f0ee8 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.23.0"
+version = "0.24.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 log = { version = "0.4.4", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 55bf9d5721..0cb6f96628 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.23.0"
+version = "0.24.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.23.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.24.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 log = { version = "0.4.4", default-features = false }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 595f5ea20a..1e5b036e27 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.23.0"
+version = "0.24.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/preopt/Cargo.toml b/lib/preopt/Cargo.toml
index c2c6ee46fd..a27ed328ac 100644
--- a/lib/preopt/Cargo.toml
+++ b/lib/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.23.0"
+version = "0.24.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,8 +11,8 @@ readme = "README.md"
 keywords = ["optimize", "compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.23.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.24.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index b577923bc7..4da0b882a7 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.23.0"
+version = "0.24.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.23.0" }
+cranelift-codegen = { path = "../codegen", version = "0.24.0" }
 target-lexicon = "0.2.0"
 
 [badges]
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index 0b571006c2..e525e501c2 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.23.0"
+version = "0.24.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
-cranelift-reader = { path = "../reader", version = "0.23.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
+cranelift-reader = { path = "../reader", version = "0.24.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 730ceacf85..1b1ad63cd2 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.23.0"
+version = "0.24.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.23.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.23.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.24.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.24.0", default-features = false }
 region = "1.0.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.2.0", default-features = false }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../umbrella", version = "0.23.0" }
-cranelift-frontend = { path = "../frontend", version = "0.23.0" }
-cranelift-entity = { path = "../entity", version = "0.23.0" }
+cranelift = { path = "../umbrella", version = "0.24.0" }
+cranelift-frontend = { path = "../frontend", version = "0.24.0" }
+cranelift-entity = { path = "../entity", version = "0.24.0" }
 
 [features]
 default = ["std"]
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index ca6e554591..aa65b9ab85 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.23.0"
+version = "0.24.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,8 +11,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.23.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.24.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 70bbb6bbe9..13d7c8c5d5 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.23.0"
+version = "0.24.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,9 +11,9 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.21.4", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.23.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.23.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.23.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.24.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.24.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 367f3cd5d3ff712c1a74fc5637056ce353c8ad33 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 19 Nov 2018 07:04:38 -0600
Subject: [PATCH 2214/3084] Implement `reserve` and `reserve_exact` for
 `PrimaryMap`.

---
 lib/entity/src/primary.rs | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs
index e23e8e2a6b..9b87098239 100644
--- a/lib/entity/src/primary.rs
+++ b/lib/entity/src/primary.rs
@@ -111,6 +111,16 @@ where
     pub fn last(&self) -> Option<&V> {
         self.elems.last()
     }
+
+    /// Reserves capacity for at least `additional` more elements to be inserted.
+    pub fn reserve(&mut self, additional: usize) {
+        self.elems.reserve(additional)
+    }
+
+    /// Reserves the minimum capacity for exactly `additional` more elements to be inserted.
+    pub fn reserve_exact(&mut self, additional: usize) {
+        self.elems.reserve_exact(additional)
+    }
 }
 
 /// Immutable indexing into an `PrimaryMap`.

From f6a6ce319933ca31c1b9e246cc59fabd15460112 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 19 Nov 2018 07:05:05 -0600
Subject: [PATCH 2215/3084] Make ModuleEnvironment's target_config() return by
 value.

This makes it more consistent with the other target_config() functions
which were all changed to return by value.
---
 lib/wasm/src/environ/dummy.rs | 4 ++--
 lib/wasm/src/environ/spec.rs  | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 7ed6e02969..926c87cf9e 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -338,8 +338,8 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
 }
 
 impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
-    fn target_config(&self) -> &TargetFrontendConfig {
-        &self.info.config
+    fn target_config(&self) -> TargetFrontendConfig {
+        self.info.config
     }
 
     fn get_func_name(&self, func_index: FuncIndex) -> ir::ExternalName {
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 705c540052..d963038f19 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -237,7 +237,7 @@ pub trait FuncEnvironment {
 /// by the user, they are only for `cranelift-wasm` internal use.
 pub trait ModuleEnvironment<'data> {
     /// Get the information needed to produce Cranelift IR for the current target.
-    fn target_config(&self) -> &TargetFrontendConfig;
+    fn target_config(&self) -> TargetFrontendConfig;
 
     /// Return the name for the given function index.
     fn get_func_name(&self, func_index: FuncIndex) -> ir::ExternalName;

From f0695a79d14c28fb35602c1f65714011932df17d Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 26 Nov 2018 04:43:42 -0800
Subject: [PATCH 2216/3084] Add an autoinst line for fallthrough_return so that
 it's included in the docs.

---
 cranelift/docs/ir.rst | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index 5925f4e601..c86e7fb3a0 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -460,6 +460,7 @@ preamble`:
 
 .. autoinst:: call
 .. autoinst:: x_return
+.. autoinst:: fallthrough_return
 
 This simple example illustrates direct function calls and signatures:
 

From 09e5b339160982b074ef1923aad085fbf51fe955 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 26 Nov 2018 22:08:15 -0800
Subject: [PATCH 2217/3084] Update to wasmparser 0.22.

---
 lib/wasm/Cargo.toml                 | 2 +-
 lib/wasm/src/code_translator.rs     | 2 +-
 lib/wasm/src/sections_translator.rs | 4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 13d7c8c5d5..31a90762db 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -10,7 +10,7 @@ readme = "README.md"
 keywords = ["webassembly", "wasm"]
 
 [dependencies]
-wasmparser = { version = "0.21.4", default-features = false }
+wasmparser = { version = "0.22.0", default-features = false }
 cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
 cranelift-entity = { path = "../entity", version = "0.24.0", default-features = false }
 cranelift-frontend = { path = "../frontend", version = "0.24.0", default-features = false }
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index 8d5fee979a..8a270e687e 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -253,7 +253,7 @@ pub fn translate_operator(
         }
         Operator::BrIf { relative_depth } => translate_br_if(relative_depth, builder, state),
         Operator::BrTable { table } => {
-            let (depths, default) = table.read_table();
+            let (depths, default) = table.read_table()?;
             let mut min_depth = default;
             for depth in &*depths {
                 if *depth < min_depth {
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 1fbb41fcd6..bab2ac1f90 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -19,8 +19,8 @@ use translation_utils::{
 use wasmparser::{
     self, CodeSectionReader, Data, DataSectionReader, Element, ElementSectionReader, Export,
     ExportSectionReader, ExternalKind, FuncType, FunctionSectionReader, GlobalSectionReader,
-    GlobalType, Import, ImportSectionEntryType, ImportSectionReader, MemorySectionReader,
-    MemoryType, Operator, TableSectionReader, TypeSectionReader,
+    GlobalType, ImportSectionEntryType, ImportSectionReader, MemorySectionReader, MemoryType,
+    Operator, TableSectionReader, TypeSectionReader,
 };
 
 /// Parses the Type section of the wasm module.

From 30654a6a7d888ce2f9f90c2cd477149d1a7fde69 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 26 Nov 2018 22:31:07 -0800
Subject: [PATCH 2218/3084] Bump version to 0.25.0

---
 cranelift/Cargo.toml        | 28 ++++++++++++++--------------
 cranelift/publish-all.sh    |  2 +-
 lib/bforest/Cargo.toml      |  4 ++--
 lib/codegen/Cargo.toml      |  8 ++++----
 lib/codegen/meta/Cargo.toml |  4 ++--
 lib/entity/Cargo.toml       |  2 +-
 lib/faerie/Cargo.toml       |  6 +++---
 lib/filetests/Cargo.toml    |  8 ++++----
 lib/frontend/Cargo.toml     |  4 ++--
 lib/module/Cargo.toml       |  6 +++---
 lib/native/Cargo.toml       |  4 ++--
 lib/preopt/Cargo.toml       |  6 +++---
 lib/reader/Cargo.toml       |  4 ++--
 lib/serde/Cargo.toml        |  6 +++---
 lib/simplejit/Cargo.toml    | 14 +++++++-------
 lib/umbrella/Cargo.toml     |  6 +++---
 lib/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index eab8e80f5b..a20993880c 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.24.0"
+version = "0.25.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,19 +14,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.24.0" }
-cranelift-entity = { path = "lib/entity", version = "0.24.0" }
-cranelift-reader = { path = "lib/reader", version = "0.24.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.24.0" }
-cranelift-serde = { path = "lib/serde", version = "0.24.0", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.24.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.24.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.24.0" }
-cranelift-module = { path = "lib/module", version = "0.24.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.24.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.24.0" }
-cranelift-preopt = { path = "lib/preopt", version = "0.24.0" }
-cranelift = { path = "lib/umbrella", version = "0.24.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.25.0" }
+cranelift-entity = { path = "lib/entity", version = "0.25.0" }
+cranelift-reader = { path = "lib/reader", version = "0.25.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.25.0" }
+cranelift-serde = { path = "lib/serde", version = "0.25.0", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.25.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.25.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.25.0" }
+cranelift-module = { path = "lib/module", version = "0.25.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.25.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.25.0" }
+cranelift-preopt = { path = "lib/preopt", version = "0.25.0" }
+cranelift = { path = "lib/umbrella", version = "0.25.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 6e6282b98b..67f29244e0 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.24.0"
+version="0.25.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index ea4b726711..17529993be 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.24.0"
+version = "0.25.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 keywords = ["btree", "forest", "set", "map"]
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.24.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.25.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 5735f8efbc..35561ac788 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.24.0"
+version = "0.25.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.24.0", default-features = false }
-cranelift-bforest = { path = "../bforest", version = "0.24.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.25.0", default-features = false }
+cranelift-bforest = { path = "../bforest", version = "0.25.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -25,7 +25,7 @@ log = { version = "0.4.4", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.24.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.25.0" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 0de1d9fb72..674af361fa 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,14 +1,14 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.24.0"
+version = "0.25.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-entity = { path = "../../entity", version = "0.24.0" }
+cranelift-entity = { path = "../../entity", version = "0.25.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index b9c1f5125a..b77ffe7e07 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.24.0"
+version = "0.25.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index b60435e3ec..ef6b354818 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.24.0"
+version = "0.25.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.24.0" }
-cranelift-module = { path = "../module", version = "0.24.0" }
+cranelift-codegen = { path = "../codegen", version = "0.25.0" }
+cranelift-module = { path = "../module", version = "0.25.0" }
 faerie = "0.6.0"
 goblin = "0.0.19"
 failure = "0.1.2"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 293e23b68c..294b1b4e17 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.24.0"
+version = "0.25.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,9 +9,9 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.24.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../reader", version = "0.24.0" }
-cranelift-preopt = { path = "../preopt", version = "0.24.0" }
+cranelift-codegen = { path = "../codegen", version = "0.25.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../reader", version = "0.25.0" }
+cranelift-preopt = { path = "../preopt", version = "0.25.0" }
 file-per-thread-logger = "0.1.1"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index bbf02f0ee8..973ec60e35 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.24.0"
+version = "0.25.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 log = { version = "0.4.4", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 0cb6f96628..6b21bfdc1b 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.24.0"
+version = "0.25.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.24.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.25.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 log = { version = "0.4.4", default-features = false }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 1e5b036e27..cd991470c3 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.24.0"
+version = "0.25.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/preopt/Cargo.toml b/lib/preopt/Cargo.toml
index a27ed328ac..601e26d19a 100644
--- a/lib/preopt/Cargo.toml
+++ b/lib/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.24.0"
+version = "0.25.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,8 +11,8 @@ readme = "README.md"
 keywords = ["optimize", "compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.24.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.25.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 4da0b882a7..20fbeb7469 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.24.0"
+version = "0.25.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.24.0" }
+cranelift-codegen = { path = "../codegen", version = "0.25.0" }
 target-lexicon = "0.2.0"
 
 [badges]
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index e525e501c2..b07fd3cbc2 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.24.0"
+version = "0.25.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
-cranelift-reader = { path = "../reader", version = "0.24.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
+cranelift-reader = { path = "../reader", version = "0.25.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 1b1ad63cd2..f4d94d3d0b 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.24.0"
+version = "0.25.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.24.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.24.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.25.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.25.0", default-features = false }
 region = "1.0.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.2.0", default-features = false }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../umbrella", version = "0.24.0" }
-cranelift-frontend = { path = "../frontend", version = "0.24.0" }
-cranelift-entity = { path = "../entity", version = "0.24.0" }
+cranelift = { path = "../umbrella", version = "0.25.0" }
+cranelift-frontend = { path = "../frontend", version = "0.25.0" }
+cranelift-entity = { path = "../entity", version = "0.25.0" }
 
 [features]
 default = ["std"]
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index aa65b9ab85..6a24c72c3f 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.24.0"
+version = "0.25.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,8 +11,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.24.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.25.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 31a90762db..6a768c87ba 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.24.0"
+version = "0.25.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,9 +11,9 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.22.0", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.24.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.24.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.24.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.25.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.25.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From f028c53479c9eb71d7acab7b26c91bd21f11f28a Mon Sep 17 00:00:00 2001
From: Zach Reizner 
Date: Sun, 25 Nov 2018 17:42:10 -0800
Subject: [PATCH 2219/3084] Fix incorrect wasm subcommand 'about' text.

---
 cranelift/src/clif-util.rs | 2 +-
 cranelift/src/wasm.rs      | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 5d5ca2b5d4..7f5d521f41 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -128,7 +128,7 @@ fn get_vec(argument_vec: Option) -> Vec {
 
 fn add_wasm_or_compile<'a>(cmd: &str) -> clap::App<'a, 'a> {
     let about_str = match cmd {
-        "wasm" => "Compiles Cranelift IR into target language",
+        "wasm" => "Compiles Wasm binary/text into Cranelift IR and then into target language",
         "compile" => "Compiles Cranelift IR into target language",
         _ => panic!("Invalid command"),
     };
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 3eeddf5241..9e22279751 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -1,7 +1,7 @@
 //! CLI tool to use the functions provided by the [cranelift-wasm](../cranelift_wasm/index.html)
 //! crate.
 //!
-//! Reads Wasm binary files, translates the functions' code to Cranelift IR.
+//! Reads Wasm binary/text files, translates the functions' code to Cranelift IR.
 #![cfg_attr(
     feature = "cargo-clippy",
     allow(too_many_arguments, cyclomatic_complexity)

From 7c03ba43becade3860048971f40813097814b31c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 28 Nov 2018 06:04:31 -0800
Subject: [PATCH 2220/3084] Document which instructions are meant for producers
 to use.

This reorganizes some things in ir.rst to put all instructions not meant
for frontends to worry about in a dedicated section.

Fixes #282.
---
 cranelift/docs/ir.rst | 134 +++++++++++++++++++++++-------------------
 1 file changed, 74 insertions(+), 60 deletions(-)

diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index c86e7fb3a0..bdbcef7f5c 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -352,12 +352,9 @@ condition is satisfied, otherwise execution continues at the following
 instruction in the EBB.
 
 .. autoinst:: jump
-.. autoinst:: fallthrough
 .. autoinst:: brz
 .. autoinst:: brnz
 .. autoinst:: br_icmp
-.. autoinst:: brif
-.. autoinst:: brff
 .. autoinst:: br_table
 
 .. inst:: JT = jump_table [EBB0, EBB1, ..., EBBn]
@@ -384,8 +381,6 @@ is zero.
 .. autoinst:: trap
 .. autoinst:: trapz
 .. autoinst:: trapnz
-.. autoinst:: trapif
-.. autoinst:: trapff
 
 
 Function calls
@@ -803,34 +798,6 @@ load a constant into an SSA value.
 .. autoinst:: f64const
 .. autoinst:: bconst
 
-Live range splitting
---------------------
-
-Cranelift's register allocator assigns each SSA value to a register or a spill
-slot on the stack for its entire live range. Since the live range of an SSA
-value can be quite large, it is sometimes beneficial to split the live range
-into smaller parts.
-
-A live range is split by creating new SSA values that are copies or the
-original value or each other. The copies are created by inserting :inst:`copy`,
-:inst:`spill`, or :inst:`fill` instructions, depending on whether the values
-are assigned to registers or stack slots.
-
-This approach permits SSA form to be preserved throughout the register
-allocation pass and beyond.
-
-.. autoinst:: copy
-.. autoinst:: spill
-.. autoinst:: fill
-
-Register values can be temporarily diverted to other registers by the
-:inst:`regmove` instruction, and to and from stack slots by :inst:`regspill`
-and :inst:`regfill`.
-
-.. autoinst:: regmove
-.. autoinst:: regspill
-.. autoinst:: regfill
-
 Vector operations
 -----------------
 
@@ -988,12 +955,6 @@ represented as a floating point number.
 .. autoinst:: trunc
 .. autoinst:: nearest
 
-CPU flag operations
--------------------
-
-.. autoinst:: trueif
-.. autoinst:: trueff
-
 Conversion operations
 ---------------------
 
@@ -1014,27 +975,6 @@ Conversion operations
 .. autoinst:: fcvt_from_uint
 .. autoinst:: fcvt_from_sint
 
-Legalization operations
------------------------
-
-These instructions are used as helpers when legalizing types and operations for
-the target ISA.
-
-.. autoinst:: isplit
-.. autoinst:: iconcat
-
-Special register operations
----------------------------
-
-The prologue and epilogue of a function needs to manipulate special registers like the stack
-pointer and the frame pointer. These instructions should not be used in regular code.
-
-.. autoinst:: adjust_sp_down
-.. autoinst:: adjust_sp_up_imm
-.. autoinst:: adjust_sp_down_imm
-.. autoinst:: ifcmp_sp
-.. autoinst:: copy_special
-
 .. _extload-truncstore:
 
 Extending loads and truncating stores
@@ -1082,6 +1022,80 @@ Instructions that can only be used by the x86 target ISA.
 .. autoinst:: isa.x86.instructions.push
 .. autoinst:: isa.x86.instructions.pop
 
+Codegen implementation instructions
+===================================
+
+Frontends don't need to emit the instructions in this section themselves;
+Cranelift will generate them automatically as needed.
+
+Legalization operations
+-----------------------
+
+These instructions are used as helpers when legalizing types and operations for
+the target ISA.
+
+.. autoinst:: isplit
+.. autoinst:: iconcat
+
+Special register operations
+---------------------------
+
+The prologue and epilogue of a function needs to manipulate special registers like the stack
+pointer and the frame pointer. These instructions should not be used in regular code.
+
+.. autoinst:: adjust_sp_down
+.. autoinst:: adjust_sp_up_imm
+.. autoinst:: adjust_sp_down_imm
+.. autoinst:: ifcmp_sp
+.. autoinst:: copy_special
+
+Low-level control flow operations
+---------------------------------
+
+.. autoinst:: fallthrough
+
+CPU flag operations
+-------------------
+
+These operations are for working with the "flags" registers of some CPU
+architectures.
+
+.. autoinst:: trueif
+.. autoinst:: trueff
+.. autoinst:: trapif
+.. autoinst:: trapff
+.. autoinst:: brif
+.. autoinst:: brff
+
+Live range splitting
+--------------------
+
+Cranelift's register allocator assigns each SSA value to a register or a spill
+slot on the stack for its entire live range. Since the live range of an SSA
+value can be quite large, it is sometimes beneficial to split the live range
+into smaller parts.
+
+A live range is split by creating new SSA values that are copies or the
+original value or each other. The copies are created by inserting :inst:`copy`,
+:inst:`spill`, or :inst:`fill` instructions, depending on whether the values
+are assigned to registers or stack slots.
+
+This approach permits SSA form to be preserved throughout the register
+allocation pass and beyond.
+
+.. autoinst:: copy
+.. autoinst:: spill
+.. autoinst:: fill
+
+Register values can be temporarily diverted to other registers by the
+:inst:`regmove` instruction, and to and from stack slots by :inst:`regspill`
+and :inst:`regfill`.
+
+.. autoinst:: regmove
+.. autoinst:: regspill
+.. autoinst:: regfill
+
+
 Instruction groups
 ==================
 

From ef21fffa1c6ba9e4a2b4820bc000aeaa19776dda Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 28 Nov 2018 08:54:40 -0800
Subject: [PATCH 2221/3084] Clarify Cranelift's design with respect to
 mid-level optimization. (#619)

* Clarify Cranelift's design with respect to mid-level optimization.

Cranelift doesn't currently do much mid-level optimization, however it
is something we're thinking about, so remove text describing it as out of
scope, and add more text explaining the vision for how it would fit into
the overall system.
---
 cranelift/docs/compare-llvm.rst | 45 ++++++++++++++++++++-------------
 1 file changed, 28 insertions(+), 17 deletions(-)

diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst
index 6ca562bcd3..e23f9f7890 100644
--- a/cranelift/docs/compare-llvm.rst
+++ b/cranelift/docs/compare-llvm.rst
@@ -21,18 +21,7 @@ highlighting some of the differences and similarities. Both projects:
 - Can target multiple ISAs.
 - Can cross-compile by default without rebuilding the code generator.
 
-Cranelift's scope is much smaller than that of LLVM. The classical three main
-parts of a compiler are:
-
-1. The language-dependent front end parses and type-checks the input program.
-2. Common optimizations that are independent of both the input language and the
-   target ISA.
-3. The code generator which depends strongly on the target ISA.
-
-LLVM provides both common optimizations *and* a code generator. Cranelift only
-provides the last part, the code generator. LLVM additionally provides
-infrastructure for building assemblers and disassemblers. Cranelift does not
-handle assembly at all---it only generates binary machine code.
+However, there are also some major differences, described in the following sections.
 
 Intermediate representations
 ============================
@@ -103,7 +92,24 @@ smaller scope.
   is annotated with an assigned ISA register or stack slot.
 
 The Cranelift intermediate representation is similar to LLVM IR, but at a slightly
-lower level of abstraction.
+lower level of abstraction, to allow it to be used all the way through the
+codegen process.
+
+This design tradeoff does mean that Cranelift IR is less friendly for mid-level
+optimizations. Cranelift doesn't currently perform mid-level optimizations,
+however if it should grow to where this becomes important, the vision is that
+Cranelift would add a separate IR layer, or possibly an separate IR, to support
+this. Instead of frontends producing optimizer IR which is then translated to
+codegen IR, Cranelift would have frontends producing codegen IR, which can be
+translated to optimizer IR and back.
+
+This biases the overall system towards fast compilation when mid-level
+optimization is not needed, such as when emitting unoptimized code for or when
+low-level optimizations are sufficient.
+
+And, it removes some constraints in the mid-level optimize IR design space,
+making it more feasible to consider ideas such as using a
+[VSDG-based IR](https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-705.pdf).
 
 Program structure
 -----------------
@@ -112,10 +118,15 @@ In LLVM IR, the largest representable unit is the *module* which corresponds
 more or less to a C translation unit. It is a collection of functions and
 global variables that may contain references to external symbols too.
 
-In Cranelift IR, the largest representable unit is the *function*. This is so
-that functions can easily be compiled in parallel without worrying about
-references to shared data structures. Cranelift does not have any
-inter-procedural optimizations like inlining.
+In `Cranelift's IR` `_,
+used by the `cranelift-codegen `_ crate,
+functions are self-contained, allowing them to be compiled independently. At
+this level, there is no explicit module that contains the functions.
+
+Module functionality in Cranelift is provided as an optional library layer, in
+the `cranelift-module `_ crate. It provides
+facilities for working with modules, which can contain multiple functions as
+well as data objects, and it links them together.
 
 An LLVM IR function is a graph of *basic blocks*. A Cranelift IR function is a
 graph of *extended basic blocks* that may contain internal branch instructions.

From d8b2def52ec1dd27256a09272515a237d0ea87ad Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 28 Nov 2018 14:31:53 -0800
Subject: [PATCH 2222/3084] Add preliminary appveyor support.

---
 appveyor.yml | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
 create mode 100644 appveyor.yml

diff --git a/appveyor.yml b/appveyor.yml
new file mode 100644
index 0000000000..9fdcaa8f16
--- /dev/null
+++ b/appveyor.yml
@@ -0,0 +1,26 @@
+environment:
+  matrix:
+  - TARGET: x86_64-pc-windows-gnu
+    BITS: 64
+    MSYS2: 1
+  - TARGET: x86_64-pc-windows-msvc
+    BITS: 64
+  - TARGET: i686-pc-windows-gnu
+    BITS: 32
+    MSYS2: 1
+  - TARGET: i686-pc-windows-msvc
+    BITS: 32
+install:
+  - curl -sSf -o rustup-init.exe https://win.rustup.rs/
+  - rustup-init.exe -y --default-host %TARGET%
+  - set PATH=%PATH%;C:\Users\appveyor\.cargo\bin
+  - if defined MSYS2 set PATH=C:\msys64\mingw%BITS%\bin;%PATH%
+  - rustc -V
+  - cargo -V
+build: false
+test_script:
+  - cargo build --verbose --all
+  - cargo test --verbose --all
+branches:
+  only:
+    - master

From 0e601ac4a52a68db698ddcd4d2d6843d0ec2788a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 28 Nov 2018 15:46:31 -0800
Subject: [PATCH 2223/3084] Add an appveyor build badge.

---
 cranelift/README.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index b2fda81b5b..900eca3e3d 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -7,7 +7,8 @@ representation](https://cranelift.readthedocs.io/en/latest/ir.html)
 into executable machine code.
 
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
-[![Build Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
+[![Travis Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
+[![Appveyor Status](https://ci.appveyor.com/api/projects/status/oub7wrrb59utuv8x?svg=true)](https://ci.appveyor.com/project/CraneStation/cranelift)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
 ![Minimum rustc 1.30](https://img.shields.io/badge/rustc-1.30+-red.svg)
 

From eaca8d3f2e110da4093af6fcb0bc5bd4fcf557b6 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 28 Nov 2018 15:47:03 -0800
Subject: [PATCH 2224/3084] Enable OSX in Travis CI.

---
 .travis.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index 8e90dd1c5d..ec928fecff 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,5 +1,8 @@
 # Travis CI script. See https://travis-ci.org/ for more info.
 
+os:
+    - linux
+    - osx
 language: rust
 rust:
     # The oldest version we currently support. See

From 9f7a7fa389ed84911be5212f8480002ca0a88a7a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 28 Nov 2018 15:47:15 -0800
Subject: [PATCH 2225/3084] Disable wabt and disassembler features on appveyor
 for now.

---
 appveyor.yml | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/appveyor.yml b/appveyor.yml
index 9fdcaa8f16..59aade2e57 100644
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -19,8 +19,10 @@ install:
   - cargo -V
 build: false
 test_script:
-  - cargo build --verbose --all
-  - cargo test --verbose --all
+  # Add --no-default-features because wabt and disass don't currently build
+  # on appveyor.
+  - cargo build --verbose --all --no-default-features
+  - cargo test --verbose --all --no-default-features
 branches:
   only:
     - master

From f8754c2cf8fb5f8b88d1b5bca3ad5d86a5845b5e Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 28 Nov 2018 16:08:26 -0800
Subject: [PATCH 2226/3084] Fix warnings when the "wasm" feature is disabled.

---
 cranelift/src/clif-util.rs | 35 +++++++++++++++++++----------------
 cranelift/src/utils.rs     |  1 +
 2 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 7f5d521f41..cedd823267 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -24,6 +24,7 @@ extern crate cfg_if;
 extern crate capstone;
 extern crate clap;
 extern crate cranelift_codegen;
+#[cfg(feature = "wasm")]
 extern crate cranelift_entity;
 extern crate cranelift_filetests;
 extern crate cranelift_reader;
@@ -245,23 +246,25 @@ fn main() {
         ("wasm", Some(rest_cmd)) => {
             handle_debug_flag(rest_cmd.is_present("debug"));
 
-            let mut target_val: &str = "";
-            if let Some(clap_target) = rest_cmd.value_of("target") {
-                target_val = clap_target;
-            }
-
             #[cfg(feature = "wasm")]
-            let result = wasm::run(
-                get_vec(rest_cmd.values_of("file")),
-                rest_cmd.is_present("verbose"),
-                rest_cmd.is_present("just-decode"),
-                rest_cmd.is_present("check-translation"),
-                rest_cmd.is_present("print"),
-                &get_vec(rest_cmd.values_of("set")),
-                target_val,
-                rest_cmd.is_present("print-size"),
-                rest_cmd.is_present("time-passes"),
-            );
+            let result = {
+                let mut target_val: &str = "";
+                if let Some(clap_target) = rest_cmd.value_of("target") {
+                    target_val = clap_target;
+                }
+
+                wasm::run(
+                    get_vec(rest_cmd.values_of("file")),
+                    rest_cmd.is_present("verbose"),
+                    rest_cmd.is_present("just-decode"),
+                    rest_cmd.is_present("check-translation"),
+                    rest_cmd.is_present("print"),
+                    &get_vec(rest_cmd.values_of("set")),
+                    target_val,
+                    rest_cmd.is_present("print-size"),
+                    rest_cmd.is_present("time-passes"),
+                )
+            };
 
             #[cfg(not(feature = "wasm"))]
             let result = Err("Error: clif-util was compiled without wasm support.".to_owned());
diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs
index 9315ff33d3..9a6ec437bc 100644
--- a/cranelift/src/utils.rs
+++ b/cranelift/src/utils.rs
@@ -25,6 +25,7 @@ pub fn read_to_string>(path: P) -> io::Result {
 }
 
 /// Read an entire file into a vector of bytes.
+#[cfg(feature = "wasm")]
 pub fn read_to_end>(path: P) -> io::Result> {
     let mut buffer = Vec::new();
     if path.as_ref() == Path::new("-") {

From 06d269be638535ce24a5e5e77bcdae29a291a74a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 28 Nov 2018 16:25:01 -0800
Subject: [PATCH 2227/3084] Disable mypy on OSX on Travis.

---
 lib/codegen/meta-python/check.sh | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/lib/codegen/meta-python/check.sh b/lib/codegen/meta-python/check.sh
index 7b9a9e7760..e34aa068f6 100755
--- a/lib/codegen/meta-python/check.sh
+++ b/lib/codegen/meta-python/check.sh
@@ -17,7 +17,11 @@ function runif {
 runif flake8 .
 
 # Type checking.
-runif mypy --py2 build.py
+# Disable mypy checking on osx on travis for now, since pip installs mypy into
+# a directory which is not in the PATH.
+if [ "$TRAVIS_OS_NAME" != "osx" ]; then
+    runif mypy --py2 build.py
+fi
 
 # Python unit tests.
 runif python2.7 -m unittest discover

From 324c821adf3bbd5856c200c207644dc94e028d20 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 28 Nov 2018 16:46:42 -0800
Subject: [PATCH 2228/3084] Replace 'mypy --version' with adding '--verbose' to
 the pip3 install command-line.

---
 .travis.yml | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index ec928fecff..c80acf071c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -26,8 +26,7 @@ addons:
         packages:
             - python3-pip
 install:
-    - pip3 install --user --upgrade mypy flake8
-    - mypy --version
+    - pip3 install --verbose --user --upgrade mypy flake8
 before_script:
     # If an old version of rustfmt from cargo is already installed, uninstall
     # it, since it can prevent the installation of the new version from rustup.

From ba48fd22231412b1b5fa1b1011cca430b9f6b912 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 28 Nov 2018 17:18:12 -0800
Subject: [PATCH 2229/3084] Add TODO comments noting temporary bug workarounds.

---
 appveyor.yml                     | 4 ++--
 lib/codegen/meta-python/check.sh | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/appveyor.yml b/appveyor.yml
index 59aade2e57..5b95b731d1 100644
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -19,8 +19,8 @@ install:
   - cargo -V
 build: false
 test_script:
-  # Add --no-default-features because wabt and disass don't currently build
-  # on appveyor.
+  # TODO: Remove --no-default-features here. wabt and disass currently
+  # don't build on appveyor due to cmake issues.
   - cargo build --verbose --all --no-default-features
   - cargo test --verbose --all --no-default-features
 branches:
diff --git a/lib/codegen/meta-python/check.sh b/lib/codegen/meta-python/check.sh
index e34aa068f6..665e74fc33 100755
--- a/lib/codegen/meta-python/check.sh
+++ b/lib/codegen/meta-python/check.sh
@@ -17,8 +17,8 @@ function runif {
 runif flake8 .
 
 # Type checking.
-# Disable mypy checking on osx on travis for now, since pip installs mypy into
-# a directory which is not in the PATH.
+# TODO: Re-enable mypy on Travis osx. Pip currently installs mypy into a
+# directory which is not in the PATH.
 if [ "$TRAVIS_OS_NAME" != "osx" ]; then
     runif mypy --py2 build.py
 fi

From 416f8c094d1db22adcc2f0fc95aafbf181418531 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 29 Nov 2018 05:48:24 -0800
Subject: [PATCH 2230/3084] Name Table and Memory's initial field `minimum` and
 make it u32.

The spec has switched to calling the "initial" field the "minimum" field
because when linear memory is imported, it may initially be greater than
the "initial" value.

Making it u32 instead of usize will help avoid accidental host-specific
behavior.
---
 lib/wasm/src/sections_translator.rs | 16 ++++++++--------
 lib/wasm/src/translation_utils.rs   |  8 ++++----
 2 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index bab2ac1f90..511f7d07ee 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -82,8 +82,8 @@ pub fn parse_import_section<'data>(
             }) => {
                 environ.declare_memory_import(
                     Memory {
-                        pages_count: memlimits.initial as usize,
-                        maximum: memlimits.maximum.map(|x| x as usize),
+                        minimum: memlimits.initial,
+                        maximum: memlimits.maximum,
                         shared,
                     },
                     module_name,
@@ -108,8 +108,8 @@ pub fn parse_import_section<'data>(
                             Ok(t) => TableElementType::Val(t),
                             Err(()) => TableElementType::Func(),
                         },
-                        size: tab.limits.initial as usize,
-                        maximum: tab.limits.maximum.map(|x| x as usize),
+                        minimum: tab.limits.initial,
+                        maximum: tab.limits.maximum,
                     },
                     module_name,
                     field_name,
@@ -144,8 +144,8 @@ pub fn parse_table_section(
                 Ok(t) => TableElementType::Val(t),
                 Err(()) => TableElementType::Func(),
             },
-            size: table.limits.initial as usize,
-            maximum: table.limits.maximum.map(|x| x as usize),
+            minimum: table.limits.initial,
+            maximum: table.limits.maximum,
         });
     }
     Ok(())
@@ -159,8 +159,8 @@ pub fn parse_memory_section(
     for entry in memories {
         let memory = entry?;
         environ.declare_memory(Memory {
-            pages_count: memory.limits.initial as usize,
-            maximum: memory.limits.maximum.map(|x| x as usize),
+            minimum: memory.limits.initial,
+            maximum: memory.limits.maximum,
             shared: memory.shared,
         });
     }
diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index 95b24b9f0f..2ebf4c3879 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -82,9 +82,9 @@ pub struct Table {
     /// The type of data stored in elements of the table.
     pub ty: TableElementType,
     /// The minimum number of elements in the table.
-    pub size: usize,
+    pub minimum: u32,
     /// The maximum number of elements in the table.
-    pub maximum: Option,
+    pub maximum: Option,
 }
 
 /// WebAssembly table element. Can be a function or a scalar type.
@@ -98,9 +98,9 @@ pub enum TableElementType {
 #[derive(Debug, Clone, Copy)]
 pub struct Memory {
     /// The minimum number of pages in the memory.
-    pub pages_count: usize,
+    pub minimum: u32,
     /// The maximum number of pages in the memory.
-    pub maximum: Option,
+    pub maximum: Option,
     /// Whether the memory may be shared between multiple threads.
     pub shared: bool,
 }

From 9947c315b70fa58853e26f61c33a44f711dc5ee7 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 30 Nov 2018 05:42:57 -0800
Subject: [PATCH 2231/3084] Fix Python check script to work outside of Travis.

---
 lib/codegen/meta-python/check.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/codegen/meta-python/check.sh b/lib/codegen/meta-python/check.sh
index 665e74fc33..fec63798df 100755
--- a/lib/codegen/meta-python/check.sh
+++ b/lib/codegen/meta-python/check.sh
@@ -19,7 +19,7 @@ runif flake8 .
 # Type checking.
 # TODO: Re-enable mypy on Travis osx. Pip currently installs mypy into a
 # directory which is not in the PATH.
-if [ "$TRAVIS_OS_NAME" != "osx" ]; then
+if [ "${TRAVIS_OS_NAME:-other}" != "osx" ]; then
     runif mypy --py2 build.py
 fi
 

From d15e56a9faa3b179d9d929d4eba8994403e162e7 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 20 Nov 2018 15:40:39 +0100
Subject: [PATCH 2232/3084] Add build warning only if CRANELIFT_VERBOSE is set
 to anything;

---
 lib/codegen/build.rs | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index 1faedddd06..67bbdfe175 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -23,8 +23,11 @@ extern crate cranelift_codegen_meta as meta;
 use meta::isa::Isa;
 use std::env;
 use std::process;
+use std::time::Instant;
 
 fn main() {
+    let start_time = Instant::now();
+
     let out_dir = env::var("OUT_DIR").expect("The OUT_DIR environment variable must be set");
     let target_triple = env::var("TARGET").expect("The TARGET environment variable must be set");
     let cranelift_targets = env::var("CRANELIFT_TARGETS").ok();
@@ -92,6 +95,14 @@ fn main() {
             process::exit(1);
         }
     }
+
+    if let Ok(_) = env::var("CRANELIFT_VERBOSE") {
+        println!(
+            "cargo:warning=Build step took {:?}.",
+            Instant::now() - start_time
+        );
+        println!("cargo:warning=Generated files are in {}", out_dir);
+    }
 }
 
 fn identify_python() -> &'static str {

From 12df9434095814bcd4ee2c867dfce3a1ebdfe2ec Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 21 Nov 2018 15:36:22 +0100
Subject: [PATCH 2233/3084] [meta] Add support to generate Match expressions in
 Rust;

---
 lib/codegen/meta/src/srcgen.rs | 95 +++++++++++++++++++++++++++-------
 1 file changed, 75 insertions(+), 20 deletions(-)

diff --git a/lib/codegen/meta/src/srcgen.rs b/lib/codegen/meta/src/srcgen.rs
index bd404fbc06..70bcfe6ef7 100644
--- a/lib/codegen/meta/src/srcgen.rs
+++ b/lib/codegen/meta/src/srcgen.rs
@@ -3,7 +3,7 @@
 //! The `srcgen` module contains generic helper routines and classes for
 //! generating source code.
 
-use std::collections::{BTreeMap, HashSet};
+use std::collections::{BTreeMap, BTreeSet};
 use std::fs;
 use std::io::Write;
 use std::path;
@@ -112,8 +112,29 @@ impl Formatter {
     }
 
     /// Add a match expression.
-    fn _add_match(&mut self, _m: &_Match) {
-        unimplemented!();
+    pub fn add_match(&mut self, m: Match) {
+        self.line(&format!("match {} {{", m.expr));
+        self.indent(|fmt| {
+            for ((fields, body), names) in m.arms.iter() {
+                // name { fields } | name { fields } => { body }
+                let conditions: Vec = names
+                    .iter()
+                    .map(|name| {
+                        if fields.len() > 0 {
+                            format!("{} {{ {} }}", name, fields.join(", "))
+                        } else {
+                            name.clone()
+                        }
+                    }).collect();
+                let lhs = conditions.join(" | ");
+                fmt.line(&format!("{} => {{", lhs));
+                fmt.indent(|fmt| {
+                    fmt.line(body);
+                });
+                fmt.line("}");
+            }
+        });
+        self.line("}");
     }
 }
 
@@ -186,44 +207,78 @@ fn parse_multiline(s: &str) -> Vec {
 /// expression, automatically deduplicating overlapping identical arms.
 ///
 /// Note that this class is ignorant of Rust types, and considers two fields
-/// with the same name to be equivalent. A BTreeMap is used to represent the
-/// arms in order to make the order deterministic.
-struct _Match<'a> {
-    _expr: &'a str,
-    arms: BTreeMap<(Vec<&'a str>, &'a str), HashSet<&'a str>>,
+/// with the same name to be equivalent. BTreeMap/BTreeSet are used to
+/// represent the arms in order to make the order deterministic.
+pub struct Match {
+    expr: String,
+    arms: BTreeMap<(Vec, String), BTreeSet>,
 }
 
-impl<'a> _Match<'a> {
+impl Match {
     /// Create a new match statement on `expr`.
-    fn _new(expr: &'a str) -> Self {
+    pub fn new>(expr: T) -> Self {
         Self {
-            _expr: expr,
+            expr: expr.into(),
             arms: BTreeMap::new(),
         }
     }
 
     /// Add an arm to the Match statement.
-    fn _arm(&mut self, name: &'a str, fields: Vec<&'a str>, body: &'a str) {
+    pub fn arm>(&mut self, name: T, fields: Vec, body: T) {
         // let key = (fields, body);
-        let match_arm = self.arms.entry((fields, body)).or_insert_with(HashSet::new);
-        match_arm.insert(name);
+        let body = body.into();
+        let fields = fields.into_iter().map(|x| x.into()).collect();
+        let match_arm = self
+            .arms
+            .entry((fields, body))
+            .or_insert_with(BTreeSet::new);
+        match_arm.insert(name.into());
     }
 }
 
 #[cfg(test)]
 mod srcgen_tests {
-    use super::_Match;
     use super::parse_multiline;
     use super::Formatter;
+    use super::Match;
+
+    fn from_raw_string(s: impl Into) -> Vec {
+        s.into()
+            .trim()
+            .split("\n")
+            .into_iter()
+            .map(|x| format!("{}\n", x))
+            .collect()
+    }
 
     #[test]
     fn adding_arms_works() {
-        let mut m = _Match::_new("x");
-        m._arm("Orange", vec!["a", "b"], "some body");
-        m._arm("Yellow", vec!["a", "b"], "some body");
-        m._arm("Green", vec!["a", "b"], "different body");
-        m._arm("Blue", vec!["x", "y"], "some body");
+        let mut m = Match::new("x");
+        m.arm("Orange", vec!["a", "b"], "some body");
+        m.arm("Yellow", vec!["a", "b"], "some body");
+        m.arm("Green", vec!["a", "b"], "different body");
+        m.arm("Blue", vec!["x", "y"], "some body");
         assert_eq!(m.arms.len(), 3);
+
+        let mut fmt = Formatter::new();
+        fmt.add_match(m);
+
+        let expected_lines = from_raw_string(
+            r#"
+match x {
+    Green { a, b } => {
+        different body
+    }
+    Orange { a, b } | Yellow { a, b } => {
+        some body
+    }
+    Blue { x, y } => {
+        some body
+    }
+}
+        "#,
+        );
+        assert_eq!(fmt.lines, expected_lines);
     }
 
     #[test]

From 4c8f1e7a5a4d02d50dc6a474d12d2a052efa0f07 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 21 Nov 2018 15:36:53 +0100
Subject: [PATCH 2234/3084] [meta] Ignore empty lines to compute indent when
 parsing multiple lines;

---
 lib/codegen/meta/src/srcgen.rs | 36 ++++++++++++++++++++++++++++++----
 1 file changed, 32 insertions(+), 4 deletions(-)

diff --git a/lib/codegen/meta/src/srcgen.rs b/lib/codegen/meta/src/srcgen.rs
index 70bcfe6ef7..a7941e9bf8 100644
--- a/lib/codegen/meta/src/srcgen.rs
+++ b/lib/codegen/meta/src/srcgen.rs
@@ -3,6 +3,7 @@
 //! The `srcgen` module contains generic helper routines and classes for
 //! generating source code.
 
+use std::cmp;
 use std::collections::{BTreeMap, BTreeSet};
 use std::fs;
 use std::io::Write;
@@ -107,8 +108,13 @@ impl Formatter {
     pub fn doc_comment(&mut self, contents: &str) {
         parse_multiline(contents)
             .iter()
-            .map(|l| format!("/// {}", l))
-            .for_each(|s| self.line(s.as_str()));
+            .map(|l| {
+                if l.len() == 0 {
+                    "///".into()
+                } else {
+                    format!("/// {}", l)
+                }
+            }).for_each(|s| self.line(s.as_str()));
     }
 
     /// Add a match expression.
@@ -156,10 +162,11 @@ fn parse_multiline(s: &str) -> Vec {
     let expanded_tab = format!("{:-1$}", " ", SHIFTWIDTH);
     let lines: Vec = s.lines().map(|l| l.replace("\t", &expanded_tab)).collect();
 
-    // Determine minimum indentation, ignoring the first line.
+    // Determine minimum indentation, ignoring the first line and empty lines.
     let indent = lines
         .iter()
         .skip(1)
+        .filter(|l| !l.trim().is_empty())
         .map(|l| l.len() - l.trim_left().len())
         .min();
 
@@ -174,8 +181,9 @@ fn parse_multiline(s: &str) -> Vec {
 
     // Remove trailing whitespace from other lines.
     let mut other_lines = if let Some(indent) = indent {
+        // Note that empty lines may have fewer than `indent` chars.
         lines_iter
-            .map(|l| &l[indent..])
+            .map(|l| &l[cmp::min(indent, l.len())..])
             .map(|l| l.trim_right())
             .map(|l| l.to_string())
             .collect::>()
@@ -349,4 +357,24 @@ match x {
         let expected_lines = vec!["/// documentation\n", "/// is\n", "/// good\n"];
         assert_eq!(fmt.lines, expected_lines);
     }
+
+    #[test]
+    fn fmt_can_add_doc_comments_with_empty_lines() {
+        let mut fmt = Formatter::new();
+        fmt.doc_comment(
+            r#"documentation
+        can be really good.
+
+        If you stick to writing it.
+"#,
+        );
+        let expected_lines = from_raw_string(
+            r#"
+/// documentation
+/// can be really good.
+///
+/// If you stick to writing it."#,
+        );
+        assert_eq!(fmt.lines, expected_lines);
+    }
 }

From d94e027c2ac8cbfc2907b8eb39692c6382b0edb0 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 21 Nov 2018 15:38:12 +0100
Subject: [PATCH 2235/3084] [build] Move settings generation from Python to
 Rust code;

---
 lib/codegen/build.rs                     |  24 +-
 lib/codegen/meta-python/cdsl/settings.py |   2 +-
 lib/codegen/meta-python/gen_settings.py  |   6 +-
 lib/codegen/meta/src/base/mod.rs         |   1 +
 lib/codegen/meta/src/base/settings.rs    | 164 +++++++++
 lib/codegen/meta/src/cdsl/isa.rs         |   9 +-
 lib/codegen/meta/src/cdsl/mod.rs         |  32 +-
 lib/codegen/meta/src/cdsl/settings.rs    | 385 +++++++++++++++++++
 lib/codegen/meta/src/constant_hash.rs    |  50 +++
 lib/codegen/meta/src/gen_registers.rs    |   8 +-
 lib/codegen/meta/src/gen_settings.rs     | 447 +++++++++++++++++++++++
 lib/codegen/meta/src/isa/arm32/mod.rs    |  10 +-
 lib/codegen/meta/src/isa/arm64/mod.rs    |  10 +-
 lib/codegen/meta/src/isa/mod.rs          |  11 +-
 lib/codegen/meta/src/isa/riscv/mod.rs    |  57 ++-
 lib/codegen/meta/src/isa/x86/mod.rs      |  77 +++-
 lib/codegen/meta/src/lib.rs              |   7 +-
 lib/codegen/meta/src/srcgen.rs           |   4 +-
 lib/codegen/meta/src/unique_table.rs     |  68 ++++
 lib/codegen/src/constant_hash.rs         |   2 +-
 20 files changed, 1334 insertions(+), 40 deletions(-)
 create mode 100644 lib/codegen/meta/src/base/settings.rs
 create mode 100644 lib/codegen/meta/src/cdsl/settings.rs
 create mode 100644 lib/codegen/meta/src/constant_hash.rs
 create mode 100644 lib/codegen/meta/src/gen_settings.rs
 create mode 100644 lib/codegen/meta/src/unique_table.rs

diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index 67bbdfe175..122d441c45 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -82,20 +82,12 @@ fn main() {
     // Now that the Python build process is complete, generate files that are
     // emitted by the `meta` crate.
     // ------------------------------------------------------------------------
-    let isas = meta::isa::define_all();
 
-    if let Err(err) = meta::gen_types::generate("types.rs", &out_dir) {
+    if let Err(err) = generate_meta(&out_dir) {
         eprintln!("Error: {}", err);
         process::exit(1);
     }
 
-    for isa in isas {
-        if let Err(err) = meta::gen_registers::generate(isa, "registers", &out_dir) {
-            eprintln!("Error: {}", err);
-            process::exit(1);
-        }
-    }
-
     if let Ok(_) = env::var("CRANELIFT_VERBOSE") {
         println!(
             "cargo:warning=Build step took {:?}.",
@@ -105,6 +97,20 @@ fn main() {
     }
 }
 
+fn generate_meta(out_dir: &str) -> Result<(), meta::error::Error> {
+    let shared_settings = meta::gen_settings::generate_common("new_settings.rs", &out_dir)?;
+    let isas = meta::isa::define_all(&shared_settings);
+
+    meta::gen_types::generate("types.rs", &out_dir)?;
+
+    for isa in &isas {
+        meta::gen_registers::generate(&isa, "registers", &out_dir)?;
+        meta::gen_settings::generate(&isa, "new_settings", &out_dir)?;
+    }
+
+    Ok(())
+}
+
 fn identify_python() -> &'static str {
     for python in &["python", "python3", "python2.7"] {
         if process::Command::new(python)
diff --git a/lib/codegen/meta-python/cdsl/settings.py b/lib/codegen/meta-python/cdsl/settings.py
index cbcac2d98d..28ff0f0b3e 100644
--- a/lib/codegen/meta-python/cdsl/settings.py
+++ b/lib/codegen/meta-python/cdsl/settings.py
@@ -340,7 +340,7 @@ class SettingGroup(object):
         precomputed predicates.
 
         This is the size of the byte-sized settings plus all the numbered
-        predcate bits rounded up to a whole number of bytes.
+        predicate bits rounded up to a whole number of bytes.
         """
         return self.boolean_offset + (len(self.predicate_number) + 7) // 8
 
diff --git a/lib/codegen/meta-python/gen_settings.py b/lib/codegen/meta-python/gen_settings.py
index 287f1a2a39..88d2a9e0cd 100644
--- a/lib/codegen/meta-python/gen_settings.py
+++ b/lib/codegen/meta-python/gen_settings.py
@@ -255,8 +255,8 @@ def gen_display(sgrp, fmt):
             fmt.line('Ok(())')
 
 
-def gen_constructor(sgrp, parent, fmt):
-    # type: (SettingGroup, PredContext, srcgen.Formatter) -> None
+def gen_constructor(sgrp, fmt):
+    # type: (SettingGroup, srcgen.Formatter) -> None
     """
     Generate a Flags constructor.
     """
@@ -310,7 +310,7 @@ def gen_group(sgrp, fmt):
     with fmt.indented('pub struct Flags {', '}'):
         fmt.line('bytes: [u8; {}],'.format(sgrp.byte_size()))
 
-    gen_constructor(sgrp, None, fmt)
+    gen_constructor(sgrp, fmt)
     gen_enum_types(sgrp, fmt)
     gen_getters(sgrp, fmt)
     gen_descriptors(sgrp, fmt)
diff --git a/lib/codegen/meta/src/base/mod.rs b/lib/codegen/meta/src/base/mod.rs
index a75614d9c8..1d596a1306 100644
--- a/lib/codegen/meta/src/base/mod.rs
+++ b/lib/codegen/meta/src/base/mod.rs
@@ -1,3 +1,4 @@
 //! Definitions for the base Cranelift language.
 
+pub mod settings;
 pub mod types;
diff --git a/lib/codegen/meta/src/base/settings.rs b/lib/codegen/meta/src/base/settings.rs
new file mode 100644
index 0000000000..c89a4e20ff
--- /dev/null
+++ b/lib/codegen/meta/src/base/settings.rs
@@ -0,0 +1,164 @@
+use cdsl::settings::{SettingGroup, SettingGroupBuilder};
+
+pub fn generate() -> SettingGroup {
+    let mut settings = SettingGroupBuilder::new("shared");
+
+    settings.add_enum(
+        "opt_level",
+        r#"
+        Optimization level:
+
+        - default: Very profitable optimizations enabled, none slow.
+        - best: Enable all optimizations
+        - fastest: Optimize for compile time by disabling most optimizations.
+        "#,
+        vec!["default", "best", "fastest"],
+    );
+
+    settings.add_bool(
+        "enable_verifier",
+        r#"
+        Run the Cranelift IR verifier at strategic times during compilation.
+
+        This makes compilation slower but catches many bugs. The verifier is
+        disabled by default, except when reading Cranelift IR from a text file.
+        "#,
+        true,
+    );
+
+    // Note that Cranelift doesn't currently need an is_pie flag, because PIE is
+    // just PIC where symbols can't be pre-empted, which can be expressed with the
+    // `colocated` flag on external functions and global values.
+    settings.add_bool(
+        "is_pic",
+        "Enable Position-Independent Code generation",
+        false,
+    );
+
+    settings.add_bool(
+        "colocated_libcalls",
+        r#"
+            Use colocated libcalls.
+
+            Generate code that assumes that libcalls can be declared "colocated",
+            meaning they will be defined along with the current function, such that
+            they can use more efficient addressing.
+            "#,
+        false,
+    );
+
+    settings.add_bool(
+        "avoid_div_traps",
+        r#"
+            Generate explicit checks around native division instructions to avoid
+            their trapping.
+
+            This is primarily used by SpiderMonkey which doesn't install a signal
+            handler for SIGFPE, but expects a SIGILL trap for division by zero.
+
+            On ISAs like ARM where the native division instructions don't trap,
+            this setting has no effect - explicit checks are always inserted.
+            "#,
+        false,
+    );
+
+    settings.add_bool(
+        "enable_float",
+        r#"
+            Enable the use of floating-point instructions
+
+            Disabling use of floating-point instructions is not yet implemented.
+            "#,
+        true,
+    );
+
+    settings.add_bool(
+        "enable_nan_canonicalization",
+        r#"
+            Enable NaN canonicalization
+
+            This replaces NaNs with a single canonical value, for users requiring
+            entirely deterministic WebAssembly computation. This is not required
+            by the WebAssembly spec, so it is not enabled by default.
+            "#,
+        false,
+    );
+
+    settings.add_bool("enable_simd", "Enable the use of SIMD instructions.", true);
+
+    settings.add_bool(
+        "enable_atomics",
+        "Enable the use of atomic instructions",
+        true,
+    );
+
+    // Settings specific to the `baldrdash` calling convention.
+
+    settings.add_num(
+        "baldrdash_prologue_words",
+        r#"
+            Number of pointer-sized words pushed by the baldrdash prologue.
+
+            Functions with the `baldrdash` calling convention don't generate their
+            own prologue and epilogue. They depend on externally generated code
+            that pushes a fixed number of words in the prologue and restores them
+            in the epilogue.
+
+            This setting configures the number of pointer-sized words pushed on the
+            stack when the Cranelift-generated code is entered. This includes the
+            pushed return address on x86.
+            "#,
+        0,
+    );
+
+    // BaldrMonkey requires that not-yet-relocated function addresses be encoded
+    // as all-ones bitpatterns.
+    settings.add_bool(
+        "allones_funcaddrs",
+        "Emit not-yet-relocated function addresses as all-ones bit patterns.",
+        false,
+    );
+
+    // Stack probing options.
+
+    settings.add_bool(
+        "probestack_enabled",
+        r#"
+            Enable the use of stack probes, for calling conventions which support this
+            functionality.
+            "#,
+        true,
+    );
+
+    settings.add_bool(
+        "probestack_func_adjusts_sp",
+        r#"
+            Set this to true of the stack probe function modifies the stack pointer
+            itself.
+            "#,
+        false,
+    );
+
+    settings.add_num(
+        "probestack_size_log2",
+        r#"
+            The log2 of the size of the stack guard region.
+
+            Stack frames larger than this size will have stack overflow checked
+            by calling the probestack function.
+
+            The default is 12, which translates to a size of 4096.
+            "#,
+        12,
+    );
+
+    // Jump table options.
+
+    settings.add_bool(
+        "jump_tables_enabled",
+        "Enable the use of jump tables in generated machine code.",
+        true,
+    );
+
+    settings.finish()
+}
diff --git a/lib/codegen/meta/src/cdsl/isa.rs b/lib/codegen/meta/src/cdsl/isa.rs
index 7168d32c85..50e4d842c8 100644
--- a/lib/codegen/meta/src/cdsl/isa.rs
+++ b/lib/codegen/meta/src/cdsl/isa.rs
@@ -3,19 +3,22 @@ use cranelift_entity::PrimaryMap;
 use super::regs::{
     RegBank, RegBankBuilder, RegBankIndex, RegClass, RegClassBuilder, RegClassIndex, RegClassProto,
 };
+use super::settings::SettingGroup;
 
 pub struct TargetIsa {
     pub name: &'static str,
     pub reg_banks: PrimaryMap,
     pub reg_classes: PrimaryMap,
+    pub settings: SettingGroup,
 }
 
 impl TargetIsa {
-    pub fn new(name: &'static str) -> Self {
+    pub fn new(name: &'static str, settings: SettingGroup) -> Self {
         Self {
             name,
             reg_banks: PrimaryMap::new(),
             reg_classes: PrimaryMap::new(),
+            settings,
         }
     }
 }
@@ -25,9 +28,9 @@ pub struct TargetIsaBuilder {
 }
 
 impl TargetIsaBuilder {
-    pub fn new(name: &'static str) -> Self {
+    pub fn new(name: &'static str, settings: SettingGroup) -> Self {
         Self {
-            isa: TargetIsa::new(name),
+            isa: TargetIsa::new(name, settings),
         }
     }
 
diff --git a/lib/codegen/meta/src/cdsl/mod.rs b/lib/codegen/meta/src/cdsl/mod.rs
index 4d29421e0b..87c615d70e 100644
--- a/lib/codegen/meta/src/cdsl/mod.rs
+++ b/lib/codegen/meta/src/cdsl/mod.rs
@@ -5,10 +5,38 @@
 
 pub mod isa;
 pub mod regs;
+pub mod settings;
 pub mod types;
 
+/// A macro that converts boolean settings into predicates to look more natural.
+#[macro_export]
+macro_rules! predicate {
+    ($a:ident && $($b:tt)*) => {
+        PredicateNode::And(Box::new($a.into()), Box::new(predicate!($($b)*)))
+    };
+    ($a:ident) => {
+        $a.into()
+    };
+}
+
+#[macro_export]
+macro_rules! preset {
+    () => {
+        vec![]
+    };
+    ($($x:ident)&&*) => {
+        {
+            let mut v = Vec::new();
+            $(
+                v.push($x.into());
+            )*
+            v
+        }
+    };
+}
+
 /// Convert the string `s` to CamelCase.
-fn _camel_case(s: &str) -> String {
+pub fn camel_case(s: &str) -> String {
     let mut output_chars = String::with_capacity(s.len());
 
     let mut capitalize = true;
@@ -30,7 +58,7 @@ fn _camel_case(s: &str) -> String {
 
 #[cfg(test)]
 mod tests {
-    use super::_camel_case as camel_case;
+    use super::camel_case;
 
     #[test]
     fn camel_case_works() {
diff --git a/lib/codegen/meta/src/cdsl/settings.rs b/lib/codegen/meta/src/cdsl/settings.rs
new file mode 100644
index 0000000000..f5e4c9b368
--- /dev/null
+++ b/lib/codegen/meta/src/cdsl/settings.rs
@@ -0,0 +1,385 @@
+use std::iter;
+
+#[derive(Clone, Copy, Hash, PartialEq, Eq)]
+pub struct BoolSettingIndex(usize);
+
+#[derive(Hash, PartialEq, Eq)]
+pub struct BoolSetting {
+    pub default: bool,
+    pub bit_offset: u8,
+    pub predicate_number: u8,
+}
+
+#[derive(Hash, PartialEq, Eq)]
+pub enum SpecificSetting {
+    Bool(BoolSetting),
+    Enum(Vec<&'static str>),
+    Num(u8),
+}
+
+#[derive(Hash, PartialEq, Eq)]
+pub struct Setting {
+    pub name: &'static str,
+    pub comment: &'static str,
+    pub specific: SpecificSetting,
+    pub byte_offset: u8,
+}
+
+impl Setting {
+    pub fn default_byte(&self) -> u8 {
+        match self.specific {
+            SpecificSetting::Bool(BoolSetting {
+                default,
+                bit_offset,
+                ..
+            }) => {
+                if default {
+                    1 << bit_offset
+                } else {
+                    0
+                }
+            }
+            SpecificSetting::Enum(_) => 0,
+            SpecificSetting::Num(default) => default,
+        }
+    }
+
+    fn byte_for_value(&self, v: bool) -> u8 {
+        match self.specific {
+            SpecificSetting::Bool(BoolSetting { bit_offset, .. }) => {
+                if v {
+                    1 << bit_offset
+                } else {
+                    0
+                }
+            }
+            _ => panic!("byte_for_value shouldn't be used for non-boolean settings."),
+        }
+    }
+
+    fn byte_mask(&self) -> u8 {
+        match self.specific {
+            SpecificSetting::Bool(BoolSetting { bit_offset, .. }) => 1 << bit_offset,
+            _ => panic!("byte_for_value shouldn't be used for non-boolean settings."),
+        }
+    }
+}
+
+#[derive(Hash, PartialEq, Eq)]
+pub struct PresetIndex(usize);
+
+#[derive(Hash, PartialEq, Eq)]
+pub enum PresetType {
+    BoolSetting(BoolSettingIndex),
+    OtherPreset(PresetIndex),
+}
+
+impl Into for BoolSettingIndex {
+    fn into(self) -> PresetType {
+        PresetType::BoolSetting(self)
+    }
+}
+impl Into for PresetIndex {
+    fn into(self) -> PresetType {
+        PresetType::OtherPreset(self)
+    }
+}
+
+#[derive(Hash, PartialEq, Eq)]
+pub struct Preset {
+    pub name: &'static str,
+    values: Vec,
+}
+
+impl Preset {
+    pub fn layout(&self, group: &SettingGroup) -> Vec<(u8, u8)> {
+        let mut layout: Vec<(u8, u8)> = iter::repeat((0, 0))
+            .take(group.settings_size as usize)
+            .collect();
+        for bool_index in &self.values {
+            let setting = &group.settings[bool_index.0];
+            let mask = setting.byte_mask();
+            let val = setting.byte_for_value(true);
+            assert!((val & !mask) == 0);
+            let (l_mask, l_val) = layout.get_mut(setting.byte_offset as usize).unwrap();
+            *l_mask |= mask;
+            *l_val = (*l_val & !mask) | val;
+        }
+        layout
+    }
+}
+
+pub struct SettingGroup {
+    pub name: &'static str,
+    pub settings: Vec,
+    pub bool_start_byte_offset: u8,
+    pub settings_size: u8,
+    pub presets: Vec,
+    pub predicates: Vec,
+}
+
+impl SettingGroup {
+    fn num_bool_settings(&self) -> u8 {
+        self.settings
+            .iter()
+            .filter(|s| {
+                if let SpecificSetting::Bool(_) = s.specific {
+                    true
+                } else {
+                    false
+                }
+            }).count() as u8
+    }
+
+    pub fn byte_size(&self) -> u8 {
+        let num_predicates = self.num_bool_settings() + (self.predicates.len() as u8);
+        self.bool_start_byte_offset + (num_predicates + 7) / 8
+    }
+
+    pub fn get_bool(&self, name: &'static str) -> (BoolSettingIndex, &Self) {
+        for (i, s) in self.settings.iter().enumerate() {
+            if let SpecificSetting::Bool(_) = s.specific {
+                if s.name == name {
+                    return (BoolSettingIndex(i), self);
+                }
+            }
+        }
+        panic!("Should have found bool setting by name.");
+    }
+}
+
+/// This is the basic information needed to track the specific parts of a setting when building
+/// them.
+pub enum ProtoSpecificSetting {
+    Bool(bool, u8),
+    Enum(Vec<&'static str>),
+    Num(u8),
+}
+
+/// This is the information provided during building for a setting.
+struct ProtoSetting {
+    name: &'static str,
+    comment: &'static str,
+    specific: ProtoSpecificSetting,
+}
+
+#[derive(Hash, PartialEq, Eq)]
+pub enum PredicateNode {
+    OwnedBool(BoolSettingIndex),
+    SharedBool(&'static str, &'static str),
+    And(Box, Box),
+}
+
+impl Into for BoolSettingIndex {
+    fn into(self) -> PredicateNode {
+        PredicateNode::OwnedBool(self)
+    }
+}
+impl<'a> Into for (BoolSettingIndex, &'a SettingGroup) {
+    fn into(self) -> PredicateNode {
+        let (index, group) = (self.0, self.1);
+        let setting = &group.settings[index.0];
+        PredicateNode::SharedBool(group.name, setting.name)
+    }
+}
+
+impl PredicateNode {
+    fn render(&self, group: &SettingGroup) -> String {
+        match self {
+            PredicateNode::OwnedBool(bool_setting_index) => format!(
+                "{}.{}()",
+                group.name, group.settings[bool_setting_index.0].name
+            ),
+            PredicateNode::SharedBool(group_name, bool_name) => {
+                format!("{}.{}()", group_name, bool_name)
+            }
+            PredicateNode::And(lhs, rhs) => {
+                format!("{} && {}", lhs.render(group), rhs.render(group))
+            }
+        }
+    }
+}
+
+pub struct Predicate {
+    pub name: &'static str,
+    node: PredicateNode,
+    pub number: u8,
+}
+
+impl Predicate {
+    pub fn render(&self, group: &SettingGroup) -> String {
+        self.node.render(group)
+    }
+}
+
+pub struct SettingGroupBuilder {
+    name: &'static str,
+    settings: Vec,
+    presets: Vec,
+    predicates: Vec,
+    predicate_number: u8,
+}
+
+impl SettingGroupBuilder {
+    pub fn new(name: &'static str) -> Self {
+        Self {
+            name,
+            settings: Vec::new(),
+            presets: Vec::new(),
+            predicates: Vec::new(),
+            predicate_number: 0,
+        }
+    }
+
+    fn add_setting(
+        &mut self,
+        name: &'static str,
+        comment: &'static str,
+        specific: ProtoSpecificSetting,
+    ) {
+        self.settings.push(ProtoSetting {
+            name,
+            comment,
+            specific,
+        })
+    }
+
+    pub fn add_bool(
+        &mut self,
+        name: &'static str,
+        comment: &'static str,
+        default: bool,
+    ) -> BoolSettingIndex {
+        assert!(
+            self.predicates.len() == 0,
+            "predicates must be added after the boolean settings"
+        );
+        let predicate_number = self.predicate_number;
+        self.predicate_number += 1;
+        self.add_setting(
+            name,
+            comment,
+            ProtoSpecificSetting::Bool(default, predicate_number),
+        );
+        BoolSettingIndex(self.settings.len() - 1)
+    }
+
+    pub fn add_enum(
+        &mut self,
+        name: &'static str,
+        comment: &'static str,
+        values: Vec<&'static str>,
+    ) {
+        self.add_setting(name, comment, ProtoSpecificSetting::Enum(values));
+    }
+
+    pub fn add_num(&mut self, name: &'static str, comment: &'static str, default: u8) {
+        self.add_setting(name, comment, ProtoSpecificSetting::Num(default));
+    }
+
+    pub fn add_predicate(&mut self, name: &'static str, node: PredicateNode) {
+        let number = self.predicate_number;
+        self.predicate_number += 1;
+        self.predicates.push(Predicate { name, node, number });
+    }
+
+    pub fn add_preset(&mut self, name: &'static str, args: Vec) -> PresetIndex {
+        let mut values = Vec::new();
+        for arg in args {
+            match arg {
+                PresetType::OtherPreset(index) => {
+                    values.extend(self.presets[index.0].values.iter());
+                }
+                PresetType::BoolSetting(index) => values.push(index),
+            }
+        }
+        self.presets.push(Preset { name, values });
+        PresetIndex(self.presets.len() - 1)
+    }
+
+    /// Compute the layout of the byte vector used to represent this settings
+    /// group.
+    ///
+    /// The byte vector contains the following entries in order:
+    ///
+    /// 1. Byte-sized settings like `NumSetting` and `EnumSetting`.
+    /// 2. `BoolSetting` settings.
+    /// 3. Precomputed named predicates.
+    /// 4. Other numbered predicates, including anonymous predicates and parent
+    ///    predicates that need to be accessible by number.
+    ///
+    /// Set `self.settings_size` to the length of the byte vector prefix that
+    /// contains the settings. All bytes after that are computed, not
+    /// configured.
+    ///
+    /// Set `self.boolean_offset` to the beginning of the numbered predicates,
+    /// 2. in the list above.
+    ///
+    /// Assign `byte_offset` and `bit_offset` fields in all settings.
+    ///
+    /// After calling this method, no more settings can be added, but
+    /// additional predicates can be made accessible with `number_predicate()`.
+    pub fn finish(self) -> SettingGroup {
+        let mut group = SettingGroup {
+            name: self.name,
+            settings: Vec::new(),
+            bool_start_byte_offset: 0,
+            settings_size: 0,
+            presets: Vec::new(),
+            predicates: Vec::new(),
+        };
+
+        let mut byte_offset = 0;
+
+        // Assign the non-boolean settings first.
+        for s in &self.settings {
+            let specific = match s.specific {
+                ProtoSpecificSetting::Bool(..) => continue,
+                ProtoSpecificSetting::Enum(ref values) => SpecificSetting::Enum(values.clone()),
+                ProtoSpecificSetting::Num(default) => SpecificSetting::Num(default),
+            };
+
+            group.settings.push(Setting {
+                name: s.name,
+                comment: s.comment,
+                byte_offset,
+                specific,
+            });
+
+            byte_offset += 1;
+        }
+
+        group.bool_start_byte_offset = byte_offset;
+
+        // Then the boolean settings.
+        for s in &self.settings {
+            let (default, predicate_number) = match s.specific {
+                ProtoSpecificSetting::Bool(default, predicate_number) => {
+                    (default, predicate_number)
+                }
+                ProtoSpecificSetting::Enum(_) | ProtoSpecificSetting::Num(_) => continue,
+            };
+            group.settings.push(Setting {
+                name: s.name,
+                comment: s.comment,
+                byte_offset: byte_offset + predicate_number / 8,
+                specific: SpecificSetting::Bool(BoolSetting {
+                    default,
+                    bit_offset: predicate_number % 8,
+                    predicate_number,
+                }),
+            });
+        }
+
+        assert!(
+            group.predicates.len() == 0,
+            "settings_size is the byte size before adding predicates"
+        );
+        group.settings_size = group.byte_size();
+
+        group.predicates.extend(self.predicates);
+        group.presets.extend(self.presets);
+
+        group
+    }
+}
diff --git a/lib/codegen/meta/src/constant_hash.rs b/lib/codegen/meta/src/constant_hash.rs
new file mode 100644
index 0000000000..84b5513d5a
--- /dev/null
+++ b/lib/codegen/meta/src/constant_hash.rs
@@ -0,0 +1,50 @@
+pub fn simple_hash(s: &str) -> usize {
+    let mut h: u32 = 5381;
+    for c in s.chars() {
+        h = (h ^ c as u32).wrapping_add(h.rotate_right(6));
+    }
+    h as usize
+}
+
+/// Compute an open addressed, quadratically probed hash table containing
+/// `items`. The returned table is a list containing the elements of the
+/// iterable `items` and `None` in unused slots.
+pub fn generate_table usize>(items: &Vec, hash_function: H) -> Vec> {
+    let size = (1.20 * items.len() as f64) as usize;
+    // TODO do we really need the multiply by two here?
+    let size = if size.is_power_of_two() {
+        size * 2
+    } else {
+        size.next_power_of_two()
+    };
+
+    let mut table: Vec> = Vec::new();
+    table.resize(size, None);
+
+    for i in items {
+        let mut h = hash_function(i) % size;
+        let mut s = 0;
+        while table[h].is_some() {
+            s += 1;
+            h = (h + s) % size;
+        }
+        table[h] = Some(i);
+    }
+
+    table
+}
+
+#[test]
+fn test_generate_table() {
+    let v = vec!["Hello".to_string(), "world".to_string()];
+    let table = generate_table(&v, |s| simple_hash(&s));
+    assert_eq!(
+        table,
+        vec![
+            None,
+            Some(&"Hello".to_string()),
+            Some(&"world".to_string()),
+            None
+        ]
+    );
+}
diff --git a/lib/codegen/meta/src/gen_registers.rs b/lib/codegen/meta/src/gen_registers.rs
index 637a3ea3fa..9234dc5c67 100644
--- a/lib/codegen/meta/src/gen_registers.rs
+++ b/lib/codegen/meta/src/gen_registers.rs
@@ -78,7 +78,7 @@ fn gen_regbank_units(reg_bank: &RegBank, fmt: &mut Formatter) {
     }
 }
 
-fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) -> Result<(), error::Error> {
+fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) {
     // Emit RegInfo.
     fmt.line("pub static INFO: RegInfo = RegInfo {");
 
@@ -128,13 +128,11 @@ fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) -> Result<(), error::Error> {
         fmt.line("}")
     });
     fmt.line("}");
-
-    Ok(())
 }
 
-pub fn generate(isa: TargetIsa, base_filename: &str, out_dir: &str) -> Result<(), error::Error> {
+pub fn generate(isa: &TargetIsa, base_filename: &str, out_dir: &str) -> Result<(), error::Error> {
     let mut fmt = Formatter::new();
-    gen_isa(&isa, &mut fmt)?;
+    gen_isa(&isa, &mut fmt);
     fmt.update_file(&format!("{}-{}.rs", base_filename, isa.name), out_dir)?;
     Ok(())
 }
diff --git a/lib/codegen/meta/src/gen_settings.rs b/lib/codegen/meta/src/gen_settings.rs
new file mode 100644
index 0000000000..cd50a57f88
--- /dev/null
+++ b/lib/codegen/meta/src/gen_settings.rs
@@ -0,0 +1,447 @@
+use base;
+use cdsl::camel_case;
+use cdsl::isa::TargetIsa;
+use cdsl::settings::{BoolSetting, Predicate, Preset, Setting, SettingGroup, SpecificSetting};
+use constant_hash::{generate_table, simple_hash};
+use error;
+use srcgen::{Formatter, Match};
+use std::collections::HashMap;
+use unique_table::UniqueTable;
+
+enum ParentGroup {
+    None,
+    Shared,
+}
+
+/// Emits the constructor of the Flags structure.
+fn gen_constructor(group: &SettingGroup, parent: ParentGroup, fmt: &mut Formatter) {
+    let args = match parent {
+        ParentGroup::None => "builder: Builder",
+        ParentGroup::Shared => "shared: &settings::Flags, builder: Builder",
+    };
+    fmt.line("impl Flags {");
+    fmt.indent(|fmt| {
+        fmt.doc_comment(&format!("Create flags {} settings group.", group.name));
+        fmt.line("#[allow(unused_variables)]");
+        fmt.line(&format!("pub fn new({}) -> Self {{", args));
+        fmt.indent(|fmt| {
+            fmt.line(&format!(
+                "let bvec = builder.state_for(\"{}\");",
+                group.name
+            ));
+            fmt.line(&format!(
+                "let mut {} = Self {{ bytes: [0; {}] }};",
+                group.name,
+                group.byte_size()
+            ));
+            fmt.line(&format!(
+                "debug_assert_eq!(bvec.len(), {});",
+                group.settings_size
+            ));
+            fmt.line(&format!(
+                "{}.bytes[0..{}].copy_from_slice(&bvec);",
+                group.name, group.settings_size
+            ));
+
+            // Now compute the predicates.
+            for p in &group.predicates {
+                fmt.comment(&format!("Precompute #{}.", p.number));
+                fmt.line(&format!("if {} {{", p.render(group)));
+                fmt.indent(|fmt| {
+                    fmt.line(&format!(
+                        "{}.bytes[{}] |= 1 << {};",
+                        group.name,
+                        group.bool_start_byte_offset + p.number / 8,
+                        p.number % 8
+                    ));
+                });
+                fmt.line("}");
+            }
+
+            fmt.line(group.name);
+        });
+        fmt.line("}");
+    });
+    fmt.line("}");
+}
+
+/// Emit Display and FromStr implementations for enum settings.
+fn gen_to_and_from_str(name: &str, values: &[&'static str], fmt: &mut Formatter) {
+    fmt.line(&format!("impl fmt::Display for {} {{", name));
+    fmt.indent(|fmt| {
+        fmt.line("fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {");
+        fmt.indent(|fmt| {
+            fmt.line("f.write_str(match *self {");
+            fmt.indent(|fmt| {
+                for v in values.iter() {
+                    fmt.line(&format!("{}::{} => \"{}\",", name, camel_case(v), v));
+                }
+            });
+            fmt.line("})");
+        });
+        fmt.line("}");
+    });
+    fmt.line("}");
+
+    fmt.line(&format!("impl str::FromStr for {} {{", name));
+    fmt.indent(|fmt| {
+        fmt.line("type Err = ();");
+        fmt.line("fn from_str(s: &str) -> Result {");
+        fmt.indent(|fmt| {
+            fmt.line("match s {");
+            fmt.indent(|fmt| {
+                for v in values.iter() {
+                    fmt.line(&format!("\"{}\" => Ok({}::{}),", v, name, camel_case(v)));
+                }
+                fmt.line("_ => Err(()),");
+            });
+            fmt.line("}");
+        });
+        fmt.line("}");
+    });
+    fmt.line("}");
+}
+
+/// Emit real enum for the Enum settings.
+fn gen_enum_types(group: &SettingGroup, fmt: &mut Formatter) {
+    for setting in group.settings.iter() {
+        let values = match setting.specific {
+            SpecificSetting::Bool(_) | SpecificSetting::Num(_) => continue,
+            SpecificSetting::Enum(ref values) => values,
+        };
+        let name = camel_case(setting.name);
+
+        fmt.doc_comment(&format!("Values for `{}.{}`.", group.name, setting.name));
+        fmt.line("#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]");
+        fmt.line(&format!("pub enum {} {{", name));
+        fmt.indent(|fmt| {
+            for v in values.iter() {
+                fmt.doc_comment(&format!("`{}`.", v));
+                fmt.line(&format!("{},", camel_case(v)));
+            }
+        });
+        fmt.line("}");
+
+        gen_to_and_from_str(&name, values, fmt);
+    }
+}
+
+/// Emit a getter function for `setting`.
+fn gen_getter(setting: &Setting, fmt: &mut Formatter) {
+    fmt.doc_comment(setting.comment);
+    match setting.specific {
+        SpecificSetting::Bool(BoolSetting {
+            predicate_number, ..
+        }) => {
+            fmt.line(&format!("pub fn {}(&self) -> bool {{", setting.name));
+            fmt.indent(|fmt| {
+                fmt.line(&format!("self.numbered_predicate({})", predicate_number));
+            });
+            fmt.line("}");
+        }
+        SpecificSetting::Enum(ref values) => {
+            let ty = camel_case(setting.name);
+            fmt.line(&format!("pub fn {}(&self) -> {} {{", setting.name, ty));
+            fmt.indent(|fmt| {
+                let mut m = Match::new(format!("self.bytes[{}]", setting.byte_offset));
+                for (i, v) in values.iter().enumerate() {
+                    m.arm(
+                        format!("{}", i),
+                        vec![],
+                        format!("{}::{}", ty, camel_case(v)),
+                    );
+                }
+                m.arm("_", vec![], "panic!(\"Invalid enum value\")");
+                fmt.add_match(m);
+            });
+            fmt.line("}");
+        }
+        SpecificSetting::Num(_) => {
+            fmt.line(&format!("pub fn {}(&self) -> u8 {{", setting.name));
+            fmt.indent(|fmt| {
+                fmt.line(&format!("self.bytes[{}]", setting.byte_offset));
+            });
+            fmt.line("}");
+        }
+    }
+}
+
+fn gen_pred_getter(predicate: &Predicate, group: &SettingGroup, fmt: &mut Formatter) {
+    fmt.doc_comment(&format!(
+        "Computed predicate `{}`.",
+        predicate.render(group)
+    ));
+    fmt.line(&format!("pub fn {}(&self) -> bool {{", predicate.name));
+    fmt.indent(|fmt| {
+        fmt.line(&format!("self.numbered_predicate({})", predicate.number));
+    });
+    fmt.line("}");
+}
+
+/// Emits getters for each setting value.
+fn gen_getters(group: &SettingGroup, fmt: &mut Formatter) {
+    fmt.doc_comment("User-defined settings.");
+    fmt.line("#[allow(dead_code)]");
+    fmt.line("impl Flags {");
+    fmt.indent(|fmt| {
+        fmt.doc_comment("Get a view of the boolean predicates.");
+        fmt.line("pub fn predicate_view(&self) -> ::settings::PredicateView {");
+        fmt.indent(|fmt| {
+            fmt.line(&format!(
+                "::settings::PredicateView::new(&self.bytes[{}..])",
+                group.bool_start_byte_offset
+            ));
+        });
+        fmt.line("}");
+
+        if group.settings.len() > 0 {
+            fmt.doc_comment("Dynamic numbered predicate getter.");
+            fmt.line("fn numbered_predicate(&self, p: usize) -> bool {");
+            fmt.indent(|fmt| {
+                fmt.line(&format!(
+                    "self.bytes[{} + p / 8] & (1 << (p % 8)) != 0",
+                    group.bool_start_byte_offset
+                ));
+            });
+            fmt.line("}");
+        }
+
+        for setting in &group.settings {
+            gen_getter(&setting, fmt);
+        }
+        for predicate in &group.predicates {
+            gen_pred_getter(&predicate, &group, fmt);
+        }
+    });
+    fmt.line("}");
+}
+
+#[derive(Hash, PartialEq, Eq)]
+enum SettingOrPreset<'a> {
+    Setting(&'a Setting),
+    Preset(&'a Preset),
+}
+
+impl<'a> SettingOrPreset<'a> {
+    fn name(&self) -> &str {
+        match self {
+            SettingOrPreset::Setting(s) => s.name,
+            SettingOrPreset::Preset(p) => p.name,
+        }
+    }
+}
+
+/// Emits DESCRIPTORS, ENUMERATORS, HASH_TABLE and PRESETS.
+fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
+    let mut enum_table: UniqueTable<&'static str> = UniqueTable::new();
+
+    let mut descriptor_index_map: HashMap = HashMap::new();
+
+    // Generate descriptors.
+    fmt.line(&format!(
+        "static DESCRIPTORS: [detail::Descriptor; {}] = [",
+        group.settings.len() + group.presets.len()
+    ));
+    fmt.indent(|fmt| {
+        for (idx, setting) in group.settings.iter().enumerate() {
+            fmt.line("detail::Descriptor {");
+            fmt.indent(|fmt| {
+                fmt.line(&format!("name: \"{}\",", setting.name));
+                fmt.line(&format!("offset: {},", setting.byte_offset));
+                match &setting.specific {
+                    SpecificSetting::Bool(BoolSetting { bit_offset, .. }) => {
+                        fmt.line(&format!(
+                            "detail: detail::Detail::Bool {{ bit: {} }},",
+                            bit_offset
+                        ));
+                    }
+                    SpecificSetting::Enum(values) => {
+                        let offset = enum_table.add(values);
+                        fmt.line(&format!(
+                            "detail: detail::Detail::Enum {{ last: {}, enumerators: {} }},",
+                            values.len() - 1,
+                            offset
+                        ));
+                    }
+                    SpecificSetting::Num(_) => {
+                        fmt.line("detail: detail::Detail::Num,");
+                    }
+                }
+
+                descriptor_index_map.insert(SettingOrPreset::Setting(setting), idx);
+            });
+            fmt.line("},");
+        }
+
+        for (idx, preset) in group.presets.iter().enumerate() {
+            fmt.line("detail::Descriptor {");
+            fmt.indent(|fmt| {
+                fmt.line(&format!("name: \"{}\",", preset.name));
+                fmt.line(&format!("offset: {},", (idx as u8) * group.settings_size));
+                fmt.line("detail: detail::Detail::Preset,");
+            });
+            fmt.line("},");
+
+            descriptor_index_map.insert(SettingOrPreset::Preset(preset), idx);
+        }
+    });
+    fmt.line("];");
+
+    // Generate enumerators.
+    fmt.line(&format!(
+        "static ENUMERATORS: [&str; {}] = [",
+        enum_table.len()
+    ));
+    fmt.indent(|fmt| {
+        for enum_val in enum_table.iter() {
+            fmt.line(&format!("\"{}\",", enum_val));
+        }
+    });
+    fmt.line("];");
+
+    // Generate hash table.
+    let mut hash_entries: Vec = Vec::new();
+    hash_entries.extend(
+        group
+            .settings
+            .iter()
+            .map(|x| SettingOrPreset::Setting(x))
+            .collect::>(),
+    );
+    hash_entries.extend(
+        group
+            .presets
+            .iter()
+            .map(|x| SettingOrPreset::Preset(x))
+            .collect::>(),
+    );
+    let hash_table = generate_table(&hash_entries, |entry| simple_hash(entry.name()));
+    fmt.line(&format!(
+        "static HASH_TABLE: [u16; {}] = [",
+        hash_table.len()
+    ));
+    fmt.indent(|fmt| {
+        for h in &hash_table {
+            match h {
+                Some(setting_or_preset) => fmt.line(&format!(
+                    "{},",
+                    &descriptor_index_map
+                        .get(setting_or_preset)
+                        .unwrap()
+                        .to_string()
+                )),
+                None => fmt.line("0xffff,"),
+            }
+        }
+    });
+    fmt.line("];");
+
+    // Generate presets.
+    fmt.line(&format!(
+        "static PRESETS: [(u8, u8); {}] = [",
+        group.presets.len()
+    ));
+    fmt.indent(|fmt| {
+        for preset in &group.presets {
+            fmt.comment(preset.name);
+            for (mask, value) in preset.layout(&group) {
+                fmt.line(&format!("(0b{:08b}, 0b{:08b}),", mask, value));
+            }
+        }
+    });
+    fmt.line("];");
+}
+
+fn gen_template(group: &SettingGroup, fmt: &mut Formatter) {
+    let mut default_bytes: Vec = Vec::new();
+    default_bytes.resize(group.settings_size as usize, 0);
+    for setting in &group.settings {
+        *default_bytes.get_mut(setting.byte_offset as usize).unwrap() |= setting.default_byte();
+    }
+
+    let default_bytes: Vec = default_bytes
+        .iter()
+        .map(|x| format!("{:#04x}", x))
+        .collect();
+    let default_bytes_str = default_bytes.join(", ");
+
+    fmt.line("static TEMPLATE: detail::Template = detail::Template {");
+    fmt.indent(|fmt| {
+        fmt.line(&format!("name: \"{}\",", group.name));
+        fmt.line("descriptors: &DESCRIPTORS,");
+        fmt.line("enumerators: &ENUMERATORS,");
+        fmt.line("hash_table: &HASH_TABLE,");
+        fmt.line(&format!("defaults: &[{}],", default_bytes_str));
+        fmt.line("presets: &PRESETS,");
+    });
+    fmt.line("};");
+
+    fmt.doc_comment(&format!(
+        "Create a `settings::Builder` for the {} settings group.",
+        group.name
+    ));
+    fmt.line("pub fn builder() -> Builder {");
+    fmt.indent(|fmt| {
+        fmt.line("Builder::new(&TEMPLATE)");
+    });
+    fmt.line("}");
+}
+
+fn gen_display(group: &SettingGroup, fmt: &mut Formatter) {
+    fmt.line("impl fmt::Display for Flags {");
+    fmt.indent(|fmt| {
+        fmt.line("fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {");
+        fmt.indent(|fmt| {
+            fmt.line(&format!("writeln!(f, \"[{}]\")?;", group.name));
+            fmt.line("for d in &DESCRIPTORS {");
+            fmt.indent(|fmt| {
+                fmt.line("if !d.detail.is_preset() {");
+                fmt.indent(|fmt| {
+                    fmt.line("write!(f, \"{} = \", d.name)?;");
+                    fmt.line(
+                        "TEMPLATE.format_toml_value(d.detail, self.bytes[d.offset as usize], f)?;",
+                    );
+                    fmt.line("writeln!(f)?;");
+                });
+                fmt.line("}");
+            });
+            fmt.line("}");
+            fmt.line("Ok(())");
+        });
+        fmt.line("}")
+    });
+    fmt.line("}");
+}
+
+fn gen_group(group: &SettingGroup, parent: ParentGroup, fmt: &mut Formatter) {
+    // Generate struct.
+    fmt.line("#[derive(Clone)]");
+    fmt.doc_comment(&format!("Flags group `{}`.", group.name));
+    fmt.line("pub struct Flags {");
+    fmt.indent(|fmt| {
+        fmt.line(&format!("bytes: [u8; {}],", group.byte_size()));
+    });
+    fmt.line("}");
+
+    gen_constructor(group, parent, fmt);
+    gen_enum_types(group, fmt);
+    gen_getters(group, fmt);
+    gen_descriptors(group, fmt);
+    gen_template(group, fmt);
+    gen_display(group, fmt);
+}
+
+pub fn generate_common(filename: &str, out_dir: &str) -> Result {
+    let settings = base::settings::generate();
+    let mut fmt = Formatter::new();
+    gen_group(&settings, ParentGroup::None, &mut fmt);
+    fmt.update_file(filename, out_dir)?;
+    Ok(settings)
+}
+
+pub fn generate(isa: &TargetIsa, prefix: &str, out_dir: &str) -> Result<(), error::Error> {
+    let mut fmt = Formatter::new();
+    gen_group(&isa.settings, ParentGroup::Shared, &mut fmt);
+    fmt.update_file(&format!("{}-{}.rs", prefix, isa.name), out_dir)?;
+    Ok(())
+}
diff --git a/lib/codegen/meta/src/isa/arm32/mod.rs b/lib/codegen/meta/src/isa/arm32/mod.rs
index e0e47c2e2e..1a4fe471d7 100644
--- a/lib/codegen/meta/src/isa/arm32/mod.rs
+++ b/lib/codegen/meta/src/isa/arm32/mod.rs
@@ -1,8 +1,14 @@
 use cdsl::isa::{TargetIsa, TargetIsaBuilder};
 use cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use cdsl::settings::{SettingGroup, SettingGroupBuilder};
 
-pub fn define() -> TargetIsa {
-    let mut isa = TargetIsaBuilder::new("arm32");
+fn define_settings(_shared: &SettingGroup) -> SettingGroup {
+    let setting = SettingGroupBuilder::new("arm32");
+    setting.finish()
+}
+
+pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
+    let mut isa = TargetIsaBuilder::new("arm32", define_settings(shared_settings));
 
     let builder = RegBankBuilder::new("FloatRegs", "s")
         .units(64)
diff --git a/lib/codegen/meta/src/isa/arm64/mod.rs b/lib/codegen/meta/src/isa/arm64/mod.rs
index 2435f3676e..63f5fa661e 100644
--- a/lib/codegen/meta/src/isa/arm64/mod.rs
+++ b/lib/codegen/meta/src/isa/arm64/mod.rs
@@ -1,8 +1,14 @@
 use cdsl::isa::{TargetIsa, TargetIsaBuilder};
 use cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use cdsl::settings::{SettingGroup, SettingGroupBuilder};
 
-pub fn define() -> TargetIsa {
-    let mut isa = TargetIsaBuilder::new("arm64");
+fn define_settings(_shared: &SettingGroup) -> SettingGroup {
+    let setting = SettingGroupBuilder::new("arm64");
+    setting.finish()
+}
+
+pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
+    let mut isa = TargetIsaBuilder::new("arm64", define_settings(shared_settings));
 
     // The `x31` regunit serves as the stack pointer / zero register depending on context. We
     // reserve it and don't model the difference.
diff --git a/lib/codegen/meta/src/isa/mod.rs b/lib/codegen/meta/src/isa/mod.rs
index 18f75c42e5..f6d012fe78 100644
--- a/lib/codegen/meta/src/isa/mod.rs
+++ b/lib/codegen/meta/src/isa/mod.rs
@@ -1,4 +1,5 @@
 use cdsl::isa::TargetIsa;
+use cdsl::settings::SettingGroup;
 use std::fmt;
 
 mod arm32;
@@ -61,11 +62,11 @@ impl fmt::Display for Isa {
     }
 }
 
-pub fn define_all() -> Vec {
+pub fn define_all(shared_settings: &SettingGroup) -> Vec {
     vec![
-        riscv::define(),
-        arm32::define(),
-        arm64::define(),
-        x86::define(),
+        riscv::define(shared_settings),
+        arm32::define(shared_settings),
+        arm64::define(shared_settings),
+        x86::define(shared_settings),
     ]
 }
diff --git a/lib/codegen/meta/src/isa/riscv/mod.rs b/lib/codegen/meta/src/isa/riscv/mod.rs
index 67f504c0f8..9d82997e94 100644
--- a/lib/codegen/meta/src/isa/riscv/mod.rs
+++ b/lib/codegen/meta/src/isa/riscv/mod.rs
@@ -1,8 +1,61 @@
 use cdsl::isa::{TargetIsa, TargetIsaBuilder};
 use cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 
-pub fn define() -> TargetIsa {
-    let mut isa = TargetIsaBuilder::new("riscv");
+fn define_settings(shared: &SettingGroup) -> SettingGroup {
+    let mut setting = SettingGroupBuilder::new("riscv");
+
+    let supports_m = setting.add_bool(
+        "supports_m",
+        "CPU supports the 'M' extension (mul/div)",
+        false,
+    );
+    let supports_a = setting.add_bool(
+        "supports_a",
+        "CPU supports the 'A' extension (atomics)",
+        false,
+    );
+    let supports_f = setting.add_bool(
+        "supports_f",
+        "CPU supports the 'F' extension (float)",
+        false,
+    );
+    let supports_d = setting.add_bool(
+        "supports_d",
+        "CPU supports the 'D' extension (double)",
+        false,
+    );
+
+    let enable_m = setting.add_bool(
+        "enable_m",
+        "Enable the use of 'M' instructions if available",
+        true,
+    );
+
+    setting.add_bool(
+        "enable_e",
+        "Enable the 'RV32E' instruction set with only 16 registers",
+        true,
+    );
+
+    let shared_enable_atomics = shared.get_bool("enable_atomics");
+    let shared_enable_float = shared.get_bool("enable_float");
+    let shared_enable_simd = shared.get_bool("enable_simd");
+
+    setting.add_predicate("use_m", predicate!(supports_m && enable_m));
+    setting.add_predicate("use_a", predicate!(supports_a && shared_enable_atomics));
+    setting.add_predicate("use_f", predicate!(supports_f && shared_enable_float));
+    setting.add_predicate("use_d", predicate!(supports_d && shared_enable_float));
+    setting.add_predicate(
+        "full_float",
+        predicate!(shared_enable_simd && supports_f && supports_d),
+    );
+
+    setting.finish()
+}
+
+pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
+    let mut isa = TargetIsaBuilder::new("riscv", define_settings(shared_settings));
 
     let builder = RegBankBuilder::new("IntRegs", "x")
         .units(32)
diff --git a/lib/codegen/meta/src/isa/x86/mod.rs b/lib/codegen/meta/src/isa/x86/mod.rs
index 7970ea06db..962446c9be 100644
--- a/lib/codegen/meta/src/isa/x86/mod.rs
+++ b/lib/codegen/meta/src/isa/x86/mod.rs
@@ -1,9 +1,74 @@
 use cdsl::isa::{TargetIsa, TargetIsaBuilder};
 use cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 
-pub fn define() -> TargetIsa {
-    let mut isa = TargetIsaBuilder::new("x86");
+pub fn define_settings(_shared: &SettingGroup) -> SettingGroup {
+    let mut settings = SettingGroupBuilder::new("x86");
 
+    // CPUID.01H:ECX
+    let has_sse3 = settings.add_bool("has_sse3", "SSE3: CPUID.01H:ECX.SSE3[bit 0]", false);
+    let has_ssse3 = settings.add_bool("has_ssse3", "SSSE3: CPUID.01H:ECX.SSSE3[bit 9]", false);
+    let has_sse41 = settings.add_bool("has_sse41", "SSE4.1: CPUID.01H:ECX.SSE4_1[bit 19]", false);
+    let has_sse42 = settings.add_bool("has_sse42", "SSE4.2: CPUID.01H:ECX.SSE4_2[bit 20]", false);
+    let has_popcnt = settings.add_bool("has_popcnt", "POPCNT: CPUID.01H:ECX.POPCNT[bit 23]", false);
+    settings.add_bool("has_avx", "AVX: CPUID.01H:ECX.AVX[bit 28]", false);
+
+    // CPUID.(EAX=07H, ECX=0H):EBX
+    let has_bmi1 = settings.add_bool(
+        "has_bmi1",
+        "BMI1: CPUID.(EAX=07H, ECX=0H):EBX.BMI1[bit 3]",
+        false,
+    );
+    let has_bmi2 = settings.add_bool(
+        "has_bmi2",
+        "BMI2: CPUID.(EAX=07H, ECX=0H):EBX.BMI2[bit 8]",
+        false,
+    );
+
+    // CPUID.EAX=80000001H:ECX
+    let has_lzcnt = settings.add_bool(
+        "has_lzcnt",
+        "LZCNT: CPUID.EAX=80000001H:ECX.LZCNT[bit 5]",
+        false,
+    );
+
+    settings.add_predicate("use_sse41", predicate!(has_sse41));
+    settings.add_predicate("use_sse42", predicate!(has_sse41 && has_sse42));
+    settings.add_predicate("use_popcnt", predicate!(has_popcnt && has_sse42));
+    settings.add_predicate("use_bmi1", predicate!(has_bmi1));
+    settings.add_predicate("use_lznct", predicate!(has_lzcnt));
+
+    settings.add_preset("baseline", preset!());
+    let nehalem = settings.add_preset(
+        "nehalem",
+        preset!(has_sse3 && has_ssse3 && has_sse41 && has_sse42 && has_popcnt),
+    );
+    let haswell = settings.add_preset(
+        "haswell",
+        preset!(nehalem && has_bmi1 && has_bmi2 && has_lzcnt),
+    );
+    let broadwell = settings.add_preset("broadwell", preset!(haswell));
+    let skylake = settings.add_preset("skylake", preset!(broadwell));
+    let cannonlake = settings.add_preset("cannonlake", preset!(skylake));
+    settings.add_preset("icelake", preset!(cannonlake));
+    settings.add_preset(
+        "znver1",
+        preset!(
+            has_sse3
+                && has_ssse3
+                && has_sse41
+                && has_sse42
+                && has_popcnt
+                && has_bmi1
+                && has_bmi2
+                && has_lzcnt
+        ),
+    );
+
+    settings.finish()
+}
+
+fn define_registers(isa: &mut TargetIsaBuilder) {
     let builder = RegBankBuilder::new("IntRegs", "r")
         .units(16)
         .names(vec!["rax", "rcx", "rdx", "rbx", "rsp", "rbp", "rsi", "rdi"])
@@ -38,6 +103,14 @@ pub fn define() -> TargetIsa {
 
     let builder = RegClassBuilder::subclass_of("FPR8", fpr, 0, 8);
     isa.add_reg_class(builder);
+}
+
+pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
+    let settings = define_settings(shared_settings);
+
+    let mut isa = TargetIsaBuilder::new("x86", settings);
+
+    define_registers(&mut isa);
 
     isa.finish()
 }
diff --git a/lib/codegen/meta/src/lib.rs b/lib/codegen/meta/src/lib.rs
index 233138e35d..e393079f45 100644
--- a/lib/codegen/meta/src/lib.rs
+++ b/lib/codegen/meta/src/lib.rs
@@ -1,11 +1,16 @@
 #[macro_use]
 extern crate cranelift_entity;
 
+#[macro_use]
+mod cdsl;
+
 pub mod error;
 pub mod gen_registers;
+pub mod gen_settings;
 pub mod gen_types;
 pub mod isa;
 
 mod base;
-mod cdsl;
+mod constant_hash;
 mod srcgen;
+mod unique_table;
diff --git a/lib/codegen/meta/src/srcgen.rs b/lib/codegen/meta/src/srcgen.rs
index a7941e9bf8..db3eac85e8 100644
--- a/lib/codegen/meta/src/srcgen.rs
+++ b/lib/codegen/meta/src/srcgen.rs
@@ -99,7 +99,7 @@ impl Formatter {
     }
 
     /// Add a comment line.
-    pub fn _comment(&mut self, s: &str) {
+    pub fn comment(&mut self, s: &str) {
         let commented_line = format!("// {}", s);
         self.line(&commented_line);
     }
@@ -302,7 +302,7 @@ match x {
         let mut fmt = Formatter::new();
         fmt.line("Hello line 1");
         fmt.indent_push();
-        fmt._comment("Nested comment");
+        fmt.comment("Nested comment");
         fmt.indent_pop();
         fmt.line("Back home again");
         let expected_lines = vec![
diff --git a/lib/codegen/meta/src/unique_table.rs b/lib/codegen/meta/src/unique_table.rs
new file mode 100644
index 0000000000..8e325cdeed
--- /dev/null
+++ b/lib/codegen/meta/src/unique_table.rs
@@ -0,0 +1,68 @@
+use std::slice;
+
+/// A table of sequences which tries to avoid common subsequences.
+pub struct UniqueTable {
+    table: Vec,
+}
+
+impl UniqueTable {
+    pub fn new() -> Self {
+        Self { table: Vec::new() }
+    }
+    pub fn add(&mut self, values: &Vec) -> usize {
+        if let Some(offset) = find_subsequence(values, &self.table) {
+            offset
+        } else {
+            let offset = self.table.len();
+            self.table.extend((*values).clone());
+            offset
+        }
+    }
+    pub fn len(&self) -> usize {
+        self.table.len()
+    }
+    pub fn iter(&self) -> slice::Iter {
+        self.table.iter()
+    }
+}
+
+/// Try to find the subsequence `sub` in the `whole` sequence. Returns None if
+/// it's not been found, or Some(index) if it has been. Naive implementation
+/// until proven we need something better.
+fn find_subsequence(sub: &Vec, whole: &Vec) -> Option {
+    assert!(sub.len() > 0);
+    // We want i + sub.len() <= whole.len(), i.e. i < whole.len() + 1 - sub.len().
+    if whole.len() < sub.len() {
+        return None;
+    }
+    let max = whole.len() + 1 - sub.len();
+    for i in 0..max {
+        let mut found: Option = Some(i);
+        for j in 0..sub.len() {
+            if sub[j] != whole[i + j] {
+                found = None;
+                break;
+            }
+        }
+        if found.is_some() {
+            return found;
+        }
+    }
+    return None;
+}
+
+#[test]
+fn test_find_subsequence() {
+    assert_eq!(find_subsequence(&vec![1], &vec![4]), None);
+    assert_eq!(find_subsequence(&vec![1], &vec![1]), Some(0));
+    assert_eq!(find_subsequence(&vec![1, 2], &vec![1]), None);
+    assert_eq!(find_subsequence(&vec![1, 2], &vec![1, 2]), Some(0));
+    assert_eq!(find_subsequence(&vec![1, 2], &vec![1, 3]), None);
+    assert_eq!(find_subsequence(&vec![1, 2], &vec![0, 1, 2]), Some(1));
+    assert_eq!(find_subsequence(&vec![1, 2], &vec![0, 1, 3, 1]), None);
+    assert_eq!(find_subsequence(&vec![1, 2], &vec![0, 1, 3, 1, 2]), Some(3));
+    assert_eq!(
+        find_subsequence(&vec![1, 1, 3], &vec![1, 1, 1, 3, 3]),
+        Some(1)
+    );
+}
diff --git a/lib/codegen/src/constant_hash.rs b/lib/codegen/src/constant_hash.rs
index 0aaa3c51c2..2c3bca1a20 100644
--- a/lib/codegen/src/constant_hash.rs
+++ b/lib/codegen/src/constant_hash.rs
@@ -56,7 +56,7 @@ pub fn probe + ?Sized>(
 }
 
 /// A primitive hash function for matching opcodes.
-/// Must match `lib/codegen/meta-python/constant_hash.py`.
+/// Must match `lib/codegen/meta-python/constant_hash.py` and `lib/codegen/meta/constant_hash.rs`.
 pub fn simple_hash(s: &str) -> usize {
     let mut h: u32 = 5381;
     for c in s.chars() {

From 990e1386f5275b467c7287c09739f54c42bd4b96 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 26 Nov 2018 03:50:50 -0800
Subject: [PATCH 2236/3084] Check for `cargo fmt` rather than checking for
 `rustfmt` directly.

---
 cranelift/test-all.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index 46a985a4e7..8afe66eebd 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -27,13 +27,13 @@ function banner {
 
 # Run rustfmt if we have it.
 banner "Rust formatting"
-if type rustfmt > /dev/null; then
+if cargo +stable fmt -- --version > /dev/null ; then
     if ! "$topdir/format-all.sh" --check ; then
         echo "Formatting diffs detected! Run \"cargo fmt --all\" to correct."
         exit 1
     fi
 else
-    echo "rustfmt not available; formatting not checked!"
+    echo "cargo-fmt not available; formatting not checked!"
     echo
     echo "If you are using rustup, rustfmt can be installed via"
     echo "\"rustup component add --toolchain=stable rustfmt-preview\", or see"

From fc28923b6e64890c6e2072ac1dac47cd86775c42 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 5 Dec 2018 18:48:56 -0500
Subject: [PATCH 2237/3084] The Gitter chat is currently shared for all
 CraneStation projects.

---
 cranelift/CONTRIBUTING.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
index 89aff16fec..105568991f 100644
--- a/cranelift/CONTRIBUTING.md
+++ b/cranelift/CONTRIBUTING.md
@@ -13,11 +13,11 @@ is reflected in the code or documentation yet. If you see things that seem
 missing or that don't make sense, or even that just don't work the way you
 expect them to, we're interested to hear about it!
 
-We have a [Cranelift chat on Gitter], and questions are also welcome as issues
+We have a [CraneStation chat on Gitter], and questions are also welcome as issues
 in the [Cranelift issue tracker]. Some folks also hang out in the #cranelift
 IRC channel on [irc.mozilla.org].
 
-[Cranelift chat on Gitter]: https://gitter.im/CraneStation/Lobby
+[CraneStation chat on Gitter]: https://gitter.im/CraneStation/Lobby
 [Cranelift issue tracker]: https://github.com/CraneStation/cranelift/issues/new
 [irc.mozilla.org]: https://wiki.mozilla.org/IRC
 

From d3eb06209b4095b55bae93a6b95e62c4788ab816 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 5 Dec 2018 18:49:23 -0500
Subject: [PATCH 2238/3084] Enable RUST_BACKTRACE=1 when running "cargo test".

---
 cranelift/test-all.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index 8afe66eebd..ca6744ec1c 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -64,7 +64,7 @@ cargo build
 
 # Run the tests. We run these in debug mode so that assertions are enabled.
 banner "Rust unit tests"
-cargo test --all
+RUST_BACKTRACE=1 cargo test --all
 
 # Make sure the documentation builds.
 banner "Rust documentation: $topdir/target/doc/cranelift/index.html"

From 323a9925e539fb27ef63418e4006a6b342ae087b Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 5 Dec 2018 18:49:59 -0500
Subject: [PATCH 2239/3084] Change the Rust version banner to green, because
 that looks nicer here.

---
 cranelift/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index 900eca3e3d..e695d28fad 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -10,7 +10,7 @@ into executable machine code.
 [![Travis Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
 [![Appveyor Status](https://ci.appveyor.com/api/projects/status/oub7wrrb59utuv8x?svg=true)](https://ci.appveyor.com/project/CraneStation/cranelift)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
-![Minimum rustc 1.30](https://img.shields.io/badge/rustc-1.30+-red.svg)
+![Minimum rustc 1.30](https://img.shields.io/badge/rustc-1.30+-green.svg)
 
 For more information, see [the
 documentation](https://cranelift.readthedocs.io/en/latest/?badge=latest).

From 5adab629f22961bf0fb19bfa3c333c96e2e68620 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 6 Dec 2018 16:15:48 -0500
Subject: [PATCH 2240/3084] Update to the rustfmt in rust 1.31, which is now
 stable.

---
 cranelift/src/clif-util.rs               | 27 +++++-----
 cranelift/src/utils.rs                   |  3 +-
 lib/bforest/src/lib.rs                   |  5 +-
 lib/codegen/meta/src/cdsl/isa.rs         | 22 ++++-----
 lib/codegen/meta/src/cdsl/settings.rs    |  3 +-
 lib/codegen/meta/src/cdsl/types.rs       | 27 +++++-----
 lib/codegen/meta/src/srcgen.rs           |  6 ++-
 lib/codegen/src/binemit/relaxation.rs    |  3 +-
 lib/codegen/src/ir/dfg.rs                |  6 +--
 lib/codegen/src/isa/registers.rs         |  3 +-
 lib/codegen/src/isa/x86/abi.rs           |  3 +-
 lib/codegen/src/legalizer/boundary.rs    |  6 ++-
 lib/codegen/src/legalizer/split.rs       |  3 +-
 lib/codegen/src/lib.rs                   |  5 +-
 lib/codegen/src/print_errors.rs          |  6 ++-
 lib/codegen/src/regalloc/coloring.rs     | 10 ++--
 lib/codegen/src/regalloc/context.rs      | 12 +++--
 lib/codegen/src/regalloc/liveness.rs     |  3 +-
 lib/codegen/src/regalloc/register_set.rs |  3 +-
 lib/codegen/src/regalloc/reload.rs       |  6 +--
 lib/codegen/src/regalloc/spilling.rs     |  3 +-
 lib/codegen/src/timing.rs                |  4 +-
 lib/codegen/src/verifier/mod.rs          | 13 ++---
 lib/entity/src/lib.rs                    |  5 +-
 lib/faerie/src/backend.rs                |  9 ++--
 lib/faerie/src/lib.rs                    |  5 +-
 lib/filetests/src/concurrent.rs          |  6 ++-
 lib/filetests/src/runner.rs              |  3 +-
 lib/frontend/src/frontend.rs             | 13 +++--
 lib/module/src/lib.rs                    |  5 +-
 lib/module/src/module.rs                 |  9 +---
 lib/native/src/lib.rs                    |  5 +-
 lib/preopt/src/lib.rs                    |  7 +--
 lib/reader/src/lexer.rs                  |  3 +-
 lib/reader/src/lib.rs                    |  5 +-
 lib/reader/src/parser.rs                 | 63 ++++++++++++++----------
 lib/reader/src/sourcemap.rs              |  3 +-
 lib/serde/src/clif-json.rs               | 11 ++---
 lib/simplejit/src/lib.rs                 |  5 +-
 lib/umbrella/src/lib.rs                  |  5 +-
 lib/wasm/src/environ/spec.rs             |  6 +--
 lib/wasm/src/lib.rs                      |  5 +-
 lib/wasm/tests/wasm_testsuite.rs         |  3 +-
 43 files changed, 181 insertions(+), 177 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index cedd823267..2396af17e4 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -1,9 +1,5 @@
 #![deny(trivial_numeric_casts)]
-#![warn(
-    unused_import_braces,
-    unstable_features,
-    unused_extern_crates
-)]
+#![warn(unused_import_braces, unstable_features, unused_extern_crates)]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
@@ -163,26 +159,31 @@ fn main() {
                 .arg(add_time_flag())
                 .arg(add_input_file_arg())
                 .arg(add_debug_flag()),
-        ).subcommand(
+        )
+        .subcommand(
             SubCommand::with_name("cat")
                 .about("Outputs .clif file")
                 .arg(add_input_file_arg())
                 .arg(add_debug_flag()),
-        ).subcommand(
+        )
+        .subcommand(
             SubCommand::with_name("print-cfg")
                 .about("Prints out cfg in dot format")
                 .arg(add_input_file_arg())
                 .arg(add_debug_flag()),
-        ).subcommand(
+        )
+        .subcommand(
             add_wasm_or_compile("compile")
                 .arg(
                     Arg::with_name("just-decode")
                         .short("t")
                         .help("Just decode WebAssembly to Cranelift IR"),
-                ).arg(Arg::with_name("check-translation").short("c").help(
+                )
+                .arg(Arg::with_name("check-translation").short("c").help(
                     "Just checks the correctness of Cranelift IR translated from WebAssembly",
                 )),
-        ).subcommand(add_wasm_or_compile("wasm"))
+        )
+        .subcommand(add_wasm_or_compile("wasm"))
         .subcommand(
             SubCommand::with_name("pass")
                 .about("Run specified pass(s) on an input file.")
@@ -204,7 +205,8 @@ fn main() {
                 rest_cmd.is_present("verbose"),
                 rest_cmd.is_present("time-passes"),
                 &get_vec(rest_cmd.values_of("file")),
-            ).map(|_time| ())
+            )
+            .map(|_time| ())
         }
         ("pass", Some(rest_cmd)) => {
             handle_debug_flag(rest_cmd.is_present("debug"));
@@ -221,7 +223,8 @@ fn main() {
                 &get_vec(rest_cmd.values_of("pass")),
                 target_val,
                 rest_cmd.value_of("single-file").unwrap(),
-            ).map(|_time| ())
+            )
+            .map(|_time| ())
         }
         ("print-cfg", Some(rest_cmd)) => {
             handle_debug_flag(rest_cmd.is_present("debug"));
diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs
index 9a6ec437bc..a05c0ac3b0 100644
--- a/cranelift/src/utils.rs
+++ b/cranelift/src/utils.rs
@@ -65,7 +65,8 @@ pub fn parse_sets_and_triple(
         flag_set.iter().map(|x| x.as_str()),
         &mut flag_builder,
         Location { line_number: 0 },
-    ).map_err(|err| err.to_string())?;
+    )
+    .map_err(|err| err.to_string())?;
 
     let mut words = flag_triple.trim().split_whitespace();
     // Look for `target foo`.
diff --git a/lib/bforest/src/lib.rs b/lib/bforest/src/lib.rs
index 0fce1c11b6..1bc1d2a091 100644
--- a/lib/bforest/src/lib.rs
+++ b/lib/bforest/src/lib.rs
@@ -16,10 +16,7 @@
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", warn(unstable_features))]
-#![cfg_attr(
-    feature = "clippy",
-    plugin(clippy(conf_file = "../../clippy.toml"))
-)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
     allow(new_without_default, new_without_default_derive)
diff --git a/lib/codegen/meta/src/cdsl/isa.rs b/lib/codegen/meta/src/cdsl/isa.rs
index 50e4d842c8..65edbbbb8b 100644
--- a/lib/codegen/meta/src/cdsl/isa.rs
+++ b/lib/codegen/meta/src/cdsl/isa.rs
@@ -153,16 +153,15 @@ impl TargetIsaBuilder {
 
                     // If the intersection is the second one, then it must be a subclass.
                     if intersect == rc2_mask {
-                        assert!(
-                            self.isa
-                                .reg_classes
-                                .get(*i1)
-                                .unwrap()
-                                .subclasses
-                                .iter()
-                                .find(|x| **x == *i2)
-                                .is_some()
-                        );
+                        assert!(self
+                            .isa
+                            .reg_classes
+                            .get(*i1)
+                            .unwrap()
+                            .subclasses
+                            .iter()
+                            .find(|x| **x == *i2)
+                            .is_some());
                     }
                 }
             }
@@ -184,7 +183,8 @@ impl TargetIsaBuilder {
             .values()
             .filter(|x| {
                 x.toprc == x.index && self.isa.reg_banks.get(x.bank).unwrap().pressure_tracking
-            }).count();
+            })
+            .count();
         assert!(num_toplevel <= 4, "Too many top-level register classes");
 
         self.isa
diff --git a/lib/codegen/meta/src/cdsl/settings.rs b/lib/codegen/meta/src/cdsl/settings.rs
index f5e4c9b368..253f7680d6 100644
--- a/lib/codegen/meta/src/cdsl/settings.rs
+++ b/lib/codegen/meta/src/cdsl/settings.rs
@@ -128,7 +128,8 @@ impl SettingGroup {
                 } else {
                     false
                 }
-            }).count() as u8
+            })
+            .count() as u8
     }
 
     pub fn byte_size(&self) -> u8 {
diff --git a/lib/codegen/meta/src/cdsl/types.rs b/lib/codegen/meta/src/cdsl/types.rs
index 60e9888e78..9fb7754c3c 100644
--- a/lib/codegen/meta/src/cdsl/types.rs
+++ b/lib/codegen/meta/src/cdsl/types.rs
@@ -190,19 +190,20 @@ impl LaneType {
 
     /// Find the unique number associated with this lane type.
     pub fn number(self) -> u8 {
-        LANE_BASE + match self {
-            LaneType::BoolType(base_types::Bool::B1) => 0,
-            LaneType::BoolType(base_types::Bool::B8) => 1,
-            LaneType::BoolType(base_types::Bool::B16) => 2,
-            LaneType::BoolType(base_types::Bool::B32) => 3,
-            LaneType::BoolType(base_types::Bool::B64) => 4,
-            LaneType::IntType(base_types::Int::I8) => 5,
-            LaneType::IntType(base_types::Int::I16) => 6,
-            LaneType::IntType(base_types::Int::I32) => 7,
-            LaneType::IntType(base_types::Int::I64) => 8,
-            LaneType::FloatType(base_types::Float::F32) => 9,
-            LaneType::FloatType(base_types::Float::F64) => 10,
-        }
+        LANE_BASE
+            + match self {
+                LaneType::BoolType(base_types::Bool::B1) => 0,
+                LaneType::BoolType(base_types::Bool::B8) => 1,
+                LaneType::BoolType(base_types::Bool::B16) => 2,
+                LaneType::BoolType(base_types::Bool::B32) => 3,
+                LaneType::BoolType(base_types::Bool::B64) => 4,
+                LaneType::IntType(base_types::Int::I8) => 5,
+                LaneType::IntType(base_types::Int::I16) => 6,
+                LaneType::IntType(base_types::Int::I32) => 7,
+                LaneType::IntType(base_types::Int::I64) => 8,
+                LaneType::FloatType(base_types::Float::F32) => 9,
+                LaneType::FloatType(base_types::Float::F64) => 10,
+            }
     }
 }
 
diff --git a/lib/codegen/meta/src/srcgen.rs b/lib/codegen/meta/src/srcgen.rs
index db3eac85e8..36270b9c70 100644
--- a/lib/codegen/meta/src/srcgen.rs
+++ b/lib/codegen/meta/src/srcgen.rs
@@ -114,7 +114,8 @@ impl Formatter {
                 } else {
                     format!("/// {}", l)
                 }
-            }).for_each(|s| self.line(s.as_str()));
+            })
+            .for_each(|s| self.line(s.as_str()));
     }
 
     /// Add a match expression.
@@ -131,7 +132,8 @@ impl Formatter {
                         } else {
                             name.clone()
                         }
-                    }).collect();
+                    })
+                    .collect();
                 let lhs = conditions.join(" | ");
                 fmt.line(&format!("{} => {{", lhs));
                 fmt.indent(|fmt| {
diff --git a/lib/codegen/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs
index 8616dccbbd..f658fb70b1 100644
--- a/lib/codegen/src/binemit/relaxation.rs
+++ b/lib/codegen/src/binemit/relaxation.rs
@@ -186,7 +186,8 @@ fn relax_branch(
                 debug!("  trying [{}]: OK", encinfo.display(enc));
                 true
             }
-        }) {
+        })
+    {
         cur.func.encodings[inst] = enc;
         return encinfo.byte_size(enc, inst, &divert, &cur.func);
     }
diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs
index 2ac9969d97..905238275a 100644
--- a/lib/codegen/src/ir/dfg.rs
+++ b/lib/codegen/src/ir/dfg.rs
@@ -802,9 +802,9 @@ impl DataFlowGraph {
             .remove(num as usize, &mut self.value_lists);
         for index in num..(self.num_ebb_params(ebb) as u16) {
             match self.values[self.ebbs[ebb]
-                                  .params
-                                  .get(index as usize, &self.value_lists)
-                                  .unwrap()]
+                .params
+                .get(index as usize, &self.value_lists)
+                .unwrap()]
             {
                 ValueData::Param { ref mut num, .. } => {
                     *num -= 1;
diff --git a/lib/codegen/src/isa/registers.rs b/lib/codegen/src/isa/registers.rs
index 03376e727d..a11a3fbd52 100644
--- a/lib/codegen/src/isa/registers.rs
+++ b/lib/codegen/src/isa/registers.rs
@@ -89,7 +89,8 @@ impl RegBank {
                     None
                 }
             }
-        }.and_then(|offset| {
+        }
+        .and_then(|offset| {
             if offset < self.units {
                 Some(offset + self.first_unit)
             } else {
diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs
index 2542eb3eb9..1f385af419 100644
--- a/lib/codegen/src/isa/x86/abi.rs
+++ b/lib/codegen/src/isa/x86/abi.rs
@@ -98,7 +98,8 @@ impl ArgAssigner for Args {
                         RU::r14
                     } else {
                         RU::rsi
-                    } as RegUnit).into()
+                    } as RegUnit)
+                    .into()
                 }
                 // This is SpiderMonkey's `WasmTableCallSigReg`.
                 ArgumentPurpose::SignatureId => return ArgumentLoc::Reg(RU::r10 as RegUnit).into(),
diff --git a/lib/codegen/src/legalizer/boundary.rs b/lib/codegen/src/legalizer/boundary.rs
index e337704f66..8b7b6c564c 100644
--- a/lib/codegen/src/legalizer/boundary.rs
+++ b/lib/codegen/src/legalizer/boundary.rs
@@ -575,7 +575,8 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph
             rt.purpose == ArgumentPurpose::Link
                 || rt.purpose == ArgumentPurpose::StructReturn
                 || rt.purpose == ArgumentPurpose::VMContext
-        }).count();
+        })
+        .count();
     let abi_args = func.signature.returns.len() - special_args;
 
     let pos = &mut FuncCursor::new(func).at_inst(inst);
@@ -694,7 +695,8 @@ fn spill_call_arguments(pos: &mut FuncCursor) -> bool {
                     }
                     _ => None,
                 }
-            }).collect::>()
+            })
+            .collect::>()
     };
 
     if arglist.is_empty() {
diff --git a/lib/codegen/src/legalizer/split.rs b/lib/codegen/src/legalizer/split.rs
index 46f530bf61..6de664863a 100644
--- a/lib/codegen/src/legalizer/split.rs
+++ b/lib/codegen/src/legalizer/split.rs
@@ -165,7 +165,8 @@ fn split_any(
                     .get_mut(
                         num_fixed_args + repair.hi_num,
                         &mut pos.func.dfg.value_lists,
-                    ).unwrap() = hi;
+                    )
+                    .unwrap() = hi;
             } else {
                 // We need to append one or more arguments. If we're adding more than one argument,
                 // there must be pending repairs on the stack that will fill in the correct values
diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index 6ece592ea7..036e096220 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -3,10 +3,7 @@
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", deny(unstable_features))]
-#![cfg_attr(
-    feature = "clippy",
-    plugin(clippy(conf_file = "../../clippy.toml"))
-)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature="cargo-clippy", allow(
 // Produces only a false positive:
                 while_let_loop,
diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index 73c8cf809a..de9fef8844 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -30,14 +30,16 @@ pub fn pretty_verifier_error<'a>(
         &mut w,
         func,
         isa,
-    ).unwrap();
+    )
+    .unwrap();
 
     writeln!(
         w,
         "\n; {} verifier error{} detected (see above). Compilation aborted.",
         num_errors,
         if num_errors == 1 { "" } else { "s" }
-    ).unwrap();
+    )
+    .unwrap();
 
     w
 }
diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs
index 5e77a42f94..8ce42737d6 100644
--- a/lib/codegen/src/regalloc/coloring.rs
+++ b/lib/codegen/src/regalloc/coloring.rs
@@ -914,12 +914,10 @@ impl<'a> Context<'a> {
             Table(jt, ebb) => {
                 let lr = &self.liveness[value];
                 !lr.is_local()
-                    && (ebb.map_or(false, |ebb| lr.is_livein(ebb, ctx)) || self
-                        .cur
-                        .func
-                        .jump_tables[jt]
-                        .iter()
-                        .any(|ebb| lr.is_livein(*ebb, ctx)))
+                    && (ebb.map_or(false, |ebb| lr.is_livein(ebb, ctx))
+                        || self.cur.func.jump_tables[jt]
+                            .iter()
+                            .any(|ebb| lr.is_livein(*ebb, ctx)))
             }
         }
     }
diff --git a/lib/codegen/src/regalloc/context.rs b/lib/codegen/src/regalloc/context.rs
index 08e658bd4c..c297250ea7 100644
--- a/lib/codegen/src/regalloc/context.rs
+++ b/lib/codegen/src/regalloc/context.rs
@@ -116,7 +116,8 @@ impl Context {
                     &self.liveness,
                     &self.virtregs,
                     &mut errors,
-                ).is_ok();
+                )
+                .is_ok();
 
             if !ok {
                 return Err(errors.into());
@@ -144,7 +145,8 @@ impl Context {
                     &self.liveness,
                     &self.virtregs,
                     &mut errors,
-                ).is_ok();
+                )
+                .is_ok();
 
             if !ok {
                 return Err(errors.into());
@@ -171,7 +173,8 @@ impl Context {
                     &self.liveness,
                     &self.virtregs,
                     &mut errors,
-                ).is_ok();
+                )
+                .is_ok();
 
             if !ok {
                 return Err(errors.into());
@@ -193,7 +196,8 @@ impl Context {
                     &self.liveness,
                     &self.virtregs,
                     &mut errors,
-                ).is_ok();
+                )
+                .is_ok();
 
             if !ok {
                 return Err(errors.into());
diff --git a/lib/codegen/src/regalloc/liveness.rs b/lib/codegen/src/regalloc/liveness.rs
index 8877179e43..01f72d9adb 100644
--- a/lib/codegen/src/regalloc/liveness.rs
+++ b/lib/codegen/src/regalloc/liveness.rs
@@ -220,7 +220,8 @@ fn get_or_create<'a>(
                         func.dfg
                             .call_signature(inst)
                             .map(|sig| Affinity::abi(&func.dfg.signatures[sig].returns[rnum], isa))
-                    }).unwrap_or_default();
+                    })
+                    .unwrap_or_default();
             }
             ValueDef::Param(ebb, num) => {
                 def = ebb.into();
diff --git a/lib/codegen/src/regalloc/register_set.rs b/lib/codegen/src/regalloc/register_set.rs
index d3a11a9d45..99a3f3e416 100644
--- a/lib/codegen/src/regalloc/register_set.rs
+++ b/lib/codegen/src/regalloc/register_set.rs
@@ -207,7 +207,8 @@ impl<'a> fmt::Display for DisplayRegisterSet<'a> {
                                     .unwrap_or_else(|| char::from_digit(
                                         u32::from(offset % 10),
                                         10
-                                    ).unwrap())
+                                    )
+                                    .unwrap())
                             )?;
                         }
                     }
diff --git a/lib/codegen/src/regalloc/reload.rs b/lib/codegen/src/regalloc/reload.rs
index 6d16ec5d3f..8240d7fa51 100644
--- a/lib/codegen/src/regalloc/reload.rs
+++ b/lib/codegen/src/regalloc/reload.rs
@@ -264,9 +264,9 @@ impl<'a> Context<'a> {
 
         // Same thing for spilled call return values.
         let retvals = &defs[self.cur.func.dfg[inst]
-                                .opcode()
-                                .constraints()
-                                .num_fixed_results()..];
+            .opcode()
+            .constraints()
+            .num_fixed_results()..];
         if !retvals.is_empty() {
             let sig = self
                 .cur
diff --git a/lib/codegen/src/regalloc/spilling.rs b/lib/codegen/src/regalloc/spilling.rs
index 168956703c..108d08b095 100644
--- a/lib/codegen/src/regalloc/spilling.rs
+++ b/lib/codegen/src/regalloc/spilling.rs
@@ -504,7 +504,8 @@ impl<'a> Context<'a> {
                     }
                 }
                 None
-            }).min_by(|&a, &b| {
+            })
+            .min_by(|&a, &b| {
                 // Find the minimum candidate according to the RPO of their defs.
                 self.domtree.rpo_cmp(
                     self.cur.func.dfg.value_def(a),
diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs
index 68ca4850c0..3edca7f73f 100644
--- a/lib/codegen/src/timing.rs
+++ b/lib/codegen/src/timing.rs
@@ -37,7 +37,7 @@ macro_rules! define_passes {
 }
 
 // Pass definitions.
-define_passes!{
+define_passes! {
     Pass, NUM_PASSES, DESCRIPTIONS;
 
     process_file: "Processing test file",
@@ -176,7 +176,7 @@ mod details {
     }
 
     /// Information about passes in a single thread.
-    thread_local!{
+    thread_local! {
         static CURRENT_PASS: Cell = Cell::new(Pass::None);
         static PASS_TIME: RefCell = RefCell::new(Default::default());
     }
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 84244cc590..adcf9d5a93 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -887,11 +887,11 @@ impl<'a> Verifier<'a> {
                     );
                 }
                 // The defining EBB dominates the instruction using this value.
-                if is_reachable && !self.expected_domtree.dominates(
-                    ebb,
-                    loc_inst,
-                    &self.func.layout,
-                ) {
+                if is_reachable
+                    && !self
+                        .expected_domtree
+                        .dominates(ebb, loc_inst, &self.func.layout)
+                {
                     return fatal!(
                         errors,
                         loc_inst,
@@ -1559,7 +1559,8 @@ impl<'a> Verifier<'a> {
                     &self.func,
                     &self.func.dfg[inst],
                     self.func.dfg.ctrl_typevar(inst),
-                ).peekable();
+                )
+                .peekable();
 
             if encodings.peek().is_none() {
                 return nonfatal!(
diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs
index d8e4fcac18..8f22caaaa7 100644
--- a/lib/entity/src/lib.rs
+++ b/lib/entity/src/lib.rs
@@ -32,10 +32,7 @@
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", deny(unstable_features))]
-#![cfg_attr(
-    feature = "clippy",
-    plugin(clippy(conf_file = "../../clippy.toml"))
-)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
     allow(new_without_default, new_without_default_derive)
diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index d51ad6e489..7028efbb01 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -231,7 +231,8 @@ impl Backend for FaerieBackend {
                     from: name,
                     to,
                     at: u64::from(offset),
-                }).map_err(|e| ModuleError::Backend(e.to_string()))?;
+                })
+                .map_err(|e| ModuleError::Backend(e.to_string()))?;
         }
         for &(offset, id, addend) in data_relocs {
             debug_assert_eq!(
@@ -244,7 +245,8 @@ impl Backend for FaerieBackend {
                     from: name,
                     to,
                     at: u64::from(offset),
-                }).map_err(|e| ModuleError::Backend(e.to_string()))?;
+                })
+                .map_err(|e| ModuleError::Backend(e.to_string()))?;
         }
 
         self.artifact
@@ -410,7 +412,8 @@ impl<'a> RelocSink for FaerieRelocSink<'a> {
                     reloc: raw_reloc,
                     addend: addend_i32,
                 },
-            ).expect("faerie relocation error");
+            )
+            .expect("faerie relocation error");
     }
 
     fn reloc_jt(&mut self, _offset: CodeOffset, _reloc: Reloc, _jt: ir::JumpTable) {
diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs
index 0b9187a468..51da811fae 100644
--- a/lib/faerie/src/lib.rs
+++ b/lib/faerie/src/lib.rs
@@ -9,10 +9,7 @@
     unstable_features
 )]
 #![warn(unused_import_braces)]
-#![cfg_attr(
-    feature = "clippy",
-    plugin(clippy(conf_file = "../../clippy.toml"))
-)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
     allow(new_without_default, new_without_default_derive)
diff --git a/lib/filetests/src/concurrent.rs b/lib/filetests/src/concurrent.rs
index 44963cc466..df9034618b 100644
--- a/lib/filetests/src/concurrent.rs
+++ b/lib/filetests/src/concurrent.rs
@@ -106,7 +106,8 @@ fn heartbeat_thread(replies: Sender) -> thread::JoinHandle<()> {
             while replies.send(Reply::Tick).is_ok() {
                 thread::sleep(Duration::from_secs(1));
             }
-        }).unwrap()
+        })
+        .unwrap()
 }
 
 /// Spawn a worker thread running tests.
@@ -153,5 +154,6 @@ fn worker_thread(
             // Timing is accumulated independently per thread.
             // Timings from this worker thread will be aggregated by `ConcurrentRunner::join()`.
             timing::take_current()
-        }).unwrap()
+        })
+        .unwrap()
 }
diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs
index b2b007803f..778150fa7c 100644
--- a/lib/filetests/src/runner.rs
+++ b/lib/filetests/src/runner.rs
@@ -317,7 +317,8 @@ impl TestRunner {
                     ..
                 } => Some(dur),
                 _ => None,
-            }).collect::>();
+            })
+            .collect::>();
 
         // Get me some real data, kid.
         let len = times.len();
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 692bc48de6..2e29db56c0 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -514,10 +514,12 @@ impl<'a> FunctionBuilder<'a> {
             None => false,
             Some(entry) => self.position.ebb.unwrap() == entry,
         };
-        !is_entry && self.func_ctx.ssa.is_sealed(self.position.ebb.unwrap()) && !self
-            .func_ctx
-            .ssa
-            .has_any_predecessors(self.position.ebb.unwrap())
+        !is_entry
+            && self.func_ctx.ssa.is_sealed(self.position.ebb.unwrap())
+            && !self
+                .func_ctx
+                .ssa
+                .has_any_predecessors(self.position.ebb.unwrap())
     }
 
     /// Returns `true` if and only if no instructions have been added since the last call to
@@ -769,7 +771,8 @@ impl<'a> FunctionBuilder<'a> {
                     self.ins().load(config.pointer_type(), flags, src, offset),
                     offset,
                 )
-            }).collect();
+            })
+            .collect();
 
         for (value, offset) in registers {
             self.ins().store(flags, value, dest, offset);
diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs
index 3c569d0ec8..e9e2131b41 100644
--- a/lib/module/src/lib.rs
+++ b/lib/module/src/lib.rs
@@ -3,10 +3,7 @@
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", deny(unstable_features))]
-#![cfg_attr(
-    feature = "clippy",
-    plugin(clippy(conf_file = "../../clippy.toml"))
-)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
     allow(new_without_default, new_without_default_derive)
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index d6328dab73..372fddd7ae 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -130,19 +130,14 @@ pub enum ModuleError {
     /// different signature than declared previously
     #[fail(
         display = "Function {} signature {:?} is incompatible with previous declaration {:?}",
-        _0,
-        _2,
-        _1
+        _0, _2, _1
     )]
     IncompatibleSignature(String, ir::Signature, ir::Signature),
     /// Indicates an identifier was defined more than once
     #[fail(display = "Duplicate definition of identifier: {}", _0)]
     DuplicateDefinition(String),
     /// Indicates an identifier was defined, but was declared as an import
-    #[fail(
-        display = "Invalid to define identifier declared as an import: {}",
-        _0
-    )]
+    #[fail(display = "Invalid to define identifier declared as an import: {}", _0)]
     InvalidImportDefinition(String),
     /// Wraps a `cranelift-codegen` error
     #[fail(display = "Compilation error: {}", _0)]
diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs
index 57fa5a4cd9..f8732714a0 100644
--- a/lib/native/src/lib.rs
+++ b/lib/native/src/lib.rs
@@ -8,10 +8,7 @@
     unstable_features
 )]
 #![warn(unused_import_braces)]
-#![cfg_attr(
-    feature = "clippy",
-    plugin(clippy(conf_file = "../../clippy.toml"))
-)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
     allow(new_without_default, new_without_default_derive)
diff --git a/lib/preopt/src/lib.rs b/lib/preopt/src/lib.rs
index 789b308afb..c55c70df74 100644
--- a/lib/preopt/src/lib.rs
+++ b/lib/preopt/src/lib.rs
@@ -1,12 +1,9 @@
 //! Performes early-stage optimizations on Cranelift IR.
 
-#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates,)]
+#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", deny(unstable_features))]
-#![cfg_attr(
-    feature = "clippy",
-    plugin(clippy(conf_file = "../../clippy.toml"))
-)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
     allow(new_without_default, new_without_default_derive)
diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs
index 5ac84ab773..1a77a1814e 100644
--- a/lib/reader/src/lexer.rs
+++ b/lib/reader/src/lexer.rs
@@ -322,7 +322,8 @@ impl<'a> Lexer<'a> {
                 .and_then(|(prefix, number)| {
                     Self::numbered_entity(prefix, number)
                         .or_else(|| Self::value_type(text, prefix, number))
-                }).unwrap_or_else(|| match text {
+                })
+                .unwrap_or_else(|| match text {
                     "iflags" => Token::Type(types::IFLAGS),
                     "fflags" => Token::Type(types::FFLAGS),
                     _ => Token::Identifier(text),
diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs
index bfa9bc8207..3b8222947a 100644
--- a/lib/reader/src/lib.rs
+++ b/lib/reader/src/lib.rs
@@ -10,10 +10,7 @@
     unstable_features
 )]
 #![warn(unused_import_braces)]
-#![cfg_attr(
-    feature = "clippy",
-    plugin(clippy(conf_file = "../../clippy.toml"))
-)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
     allow(new_without_default, new_without_default_derive)
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 650efe3ea0..a53f734792 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -2587,7 +2587,8 @@ mod tests {
                                              v3 -> v4
                                              v1 = iadd_imm v3, 17
                                            }",
-        ).parse_function(None)
+        )
+        .parse_function(None)
         .unwrap();
         assert_eq!(func.name.to_string(), "%qux");
         let v4 = details.map.lookup_str("v4").unwrap();
@@ -2663,7 +2664,8 @@ mod tests {
                                        ss3 = incoming_arg 13
                                        ss1 = spill_slot 1
                                      }",
-        ).parse_function(None)
+        )
+        .parse_function(None)
         .unwrap();
         assert_eq!(func.name.to_string(), "%foo");
         let mut iter = func.stack_slots.keys();
@@ -2686,7 +2688,8 @@ mod tests {
                                     ss1  = spill_slot 13
                                     ss1  = spill_slot 1
                                 }",
-            ).parse_function(None)
+            )
+            .parse_function(None)
             .unwrap_err()
             .to_string(),
             "3: duplicate entity: ss1"
@@ -2700,7 +2703,8 @@ mod tests {
                                      ebb0:
                                      ebb4(v3: i32):
                                      }",
-        ).parse_function(None)
+        )
+        .parse_function(None)
         .unwrap();
         assert_eq!(func.name.to_string(), "%ebbs");
 
@@ -2722,7 +2726,8 @@ mod tests {
                 ebb0:
                 ebb0:
                     return 2",
-        ).parse_function(None)
+        )
+        .parse_function(None)
         .unwrap_err();
 
         assert_eq!(location.line_number, 3);
@@ -2735,7 +2740,8 @@ mod tests {
             "function %ebbs() system_v {
                 jt0 = jump_table []
                 jt0 = jump_table []",
-        ).parse_function(None)
+        )
+        .parse_function(None)
         .unwrap_err();
 
         assert_eq!(location.line_number, 3);
@@ -2748,7 +2754,8 @@ mod tests {
             "function %ebbs() system_v {
                 ss0 = explicit_slot 8
                 ss0 = explicit_slot 8",
-        ).parse_function(None)
+        )
+        .parse_function(None)
         .unwrap_err();
 
         assert_eq!(location.line_number, 3);
@@ -2761,7 +2768,8 @@ mod tests {
             "function %ebbs() system_v {
                 gv0 = vmctx
                 gv0 = vmctx",
-        ).parse_function(None)
+        )
+        .parse_function(None)
         .unwrap_err();
 
         assert_eq!(location.line_number, 3);
@@ -2774,7 +2782,8 @@ mod tests {
             "function %ebbs() system_v {
                 heap0 = static gv0, min 0x1000, bound 0x10_0000, guard 0x1000
                 heap0 = static gv0, min 0x1000, bound 0x10_0000, guard 0x1000",
-        ).parse_function(None)
+        )
+        .parse_function(None)
         .unwrap_err();
 
         assert_eq!(location.line_number, 3);
@@ -2787,7 +2796,8 @@ mod tests {
             "function %ebbs() system_v {
                 sig0 = ()
                 sig0 = ()",
-        ).parse_function(None)
+        )
+        .parse_function(None)
         .unwrap_err();
 
         assert_eq!(location.line_number, 3);
@@ -2801,7 +2811,8 @@ mod tests {
                 sig0 = ()
                 fn0 = %foo sig0
                 fn0 = %foo sig0",
-        ).parse_function(None)
+        )
+        .parse_function(None)
         .unwrap_err();
 
         assert_eq!(location.line_number, 4);
@@ -2821,7 +2832,8 @@ mod tests {
                          trap user42; Instruction
                          } ; Trailing.
                          ; More trailing.",
-        ).parse_function(None)
+        )
+        .parse_function(None)
         .unwrap();
         assert_eq!(func.name.to_string(), "%comment");
         assert_eq!(comments.len(), 8); // no 'before' comment.
@@ -2858,7 +2870,8 @@ mod tests {
                              function %comment() system_v {}",
             None,
             None,
-        ).unwrap();
+        )
+        .unwrap();
         assert_eq!(tf.commands.len(), 2);
         assert_eq!(tf.commands[0].command, "cfg");
         assert_eq!(tf.commands[1].command, "verify");
@@ -2879,26 +2892,25 @@ mod tests {
     #[test]
     #[cfg(build_riscv)]
     fn isa_spec() {
-        assert!(
-            parse_test(
-                "target
+        assert!(parse_test(
+            "target
                             function %foo() system_v {}",
-            ).is_err()
-        );
+        )
+        .is_err());
 
-        assert!(
-            parse_test(
-                "target riscv32
+        assert!(parse_test(
+            "target riscv32
                             set enable_float=false
                             function %foo() system_v {}",
-            ).is_err()
-        );
+        )
+        .is_err());
 
         match parse_test(
             "set enable_float=false
                           isa riscv
                           function %foo() system_v {}",
-        ).unwrap()
+        )
+        .unwrap()
         .isa_spec
         {
             IsaSpec::None(_) => panic!("Expected some ISA"),
@@ -2917,7 +2929,8 @@ mod tests {
                                            ebb0:
                                              trap int_divz
                                            }",
-        ).parse_function(None)
+        )
+        .parse_function(None)
         .unwrap()
         .0;
         assert_eq!(func.name.to_string(), "u1:2");
diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs
index 7e4da9de80..f3f0423e43 100644
--- a/lib/reader/src/sourcemap.rs
+++ b/lib/reader/src/sourcemap.rs
@@ -224,7 +224,8 @@ mod tests {
                              }",
             None,
             None,
-        ).unwrap();
+        )
+        .unwrap();
         let map = &tf.functions[0].1.map;
 
         assert_eq!(map.lookup_str("v0"), None);
diff --git a/lib/serde/src/clif-json.rs b/lib/serde/src/clif-json.rs
index bab63ace97..77498ca06e 100644
--- a/lib/serde/src/clif-json.rs
+++ b/lib/serde/src/clif-json.rs
@@ -7,10 +7,7 @@
     unstable_features
 )]
 #![warn(unused_import_braces)]
-#![cfg_attr(
-    feature = "clippy",
-    plugin(clippy(conf_file = "../../clippy.toml"))
-)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
     allow(new_without_default, new_without_default_derive)
@@ -83,7 +80,8 @@ fn main() {
                         .value_name("FILE")
                         .help("Input file for serialization"),
                 ),
-        ).subcommand(
+        )
+        .subcommand(
             SubCommand::with_name("deserialize")
                 .about("Deserializes Cranelift IR into JSON.")
                 .arg(
@@ -92,7 +90,8 @@ fn main() {
                         .value_name("FILE")
                         .help("Input file for deserialization"),
                 ),
-        ).get_matches();
+        )
+        .get_matches();
 
     let res_serde = match matches.subcommand() {
         ("serialize", Some(m)) => {
diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs
index 50f84f67eb..f50cb826c6 100644
--- a/lib/simplejit/src/lib.rs
+++ b/lib/simplejit/src/lib.rs
@@ -7,10 +7,7 @@
     unstable_features
 )]
 #![warn(unused_import_braces)]
-#![cfg_attr(
-    feature = "clippy",
-    plugin(clippy(conf_file = "../../clippy.toml"))
-)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
     allow(new_without_default, new_without_default_derive)
diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs
index 7a3a8f00dc..41a55f6fe4 100644
--- a/lib/umbrella/src/lib.rs
+++ b/lib/umbrella/src/lib.rs
@@ -7,10 +7,7 @@
     unstable_features
 )]
 #![warn(unused_import_braces)]
-#![cfg_attr(
-    feature = "clippy",
-    plugin(clippy(conf_file = "../../clippy.toml"))
-)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
     allow(new_without_default, new_without_default_derive)
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index d963038f19..c7910f65f1 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -35,11 +35,7 @@ pub enum WasmError {
     ///
     /// This error code is used by a WebAssembly translator when it encounters invalid WebAssembly
     /// code. This should never happen for validated WebAssembly code.
-    #[fail(
-        display = "Invalid input WebAssembly code at offset {}: {}",
-        _1,
-        _0
-    )]
+    #[fail(display = "Invalid input WebAssembly code at offset {}: {}", _1, _0)]
     InvalidWebAssembly {
         /// A string describing the validation error.
         message: &'static str,
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index 86f7c2a769..e528d5f966 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -12,10 +12,7 @@
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", deny(unstable_features))]
-#![cfg_attr(
-    feature = "clippy",
-    plugin(clippy(conf_file = "../../clippy.toml"))
-)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
     allow(new_without_default, new_without_default_derive)
diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs
index bf0dee04dc..18b8c08c11 100644
--- a/lib/wasm/tests/wasm_testsuite.rs
+++ b/lib/wasm/tests/wasm_testsuite.rs
@@ -30,7 +30,8 @@ fn testsuite() {
                 }
             }
             false
-        }).collect();
+        })
+        .collect();
     paths.sort_by_key(|dir| dir.path());
     let flags = Flags::new(settings::builder());
     for path in paths {

From 5d7ece3d158432efe6b5606136d9e9dfb58f193e Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 6 Dec 2018 16:20:10 -0500
Subject: [PATCH 2241/3084] Fix clippy warning namespaces.

---
 cranelift/src/clif-util.rs | 14 +++++++-------
 lib/bforest/src/lib.rs     | 16 ++++++++--------
 lib/codegen/src/lib.rs     | 16 ++++++++--------
 lib/entity/src/lib.rs      | 16 ++++++++--------
 lib/faerie/src/lib.rs      | 16 ++++++++--------
 lib/filetests/src/lib.rs   | 12 ++++++------
 lib/frontend/src/lib.rs    | 16 ++++++++--------
 lib/module/src/lib.rs      | 16 ++++++++--------
 lib/native/src/lib.rs      | 16 ++++++++--------
 lib/preopt/src/lib.rs      | 16 ++++++++--------
 lib/reader/src/lib.rs      | 16 ++++++++--------
 lib/serde/src/clif-json.rs | 14 +++++++-------
 lib/simplejit/src/lib.rs   | 16 ++++++++--------
 lib/umbrella/src/lib.rs    | 16 ++++++++--------
 lib/wasm/src/lib.rs        | 16 ++++++++--------
 15 files changed, 116 insertions(+), 116 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 2396af17e4..1b9255db4b 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -3,13 +3,13 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 
diff --git a/lib/bforest/src/lib.rs b/lib/bforest/src/lib.rs
index 1bc1d2a091..4b478f189c 100644
--- a/lib/bforest/src/lib.rs
+++ b/lib/bforest/src/lib.rs
@@ -24,14 +24,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        print_stdout,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 // Turns on no_std and alloc features if std is not available.
diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index 036e096220..c8d4e4434f 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -30,14 +30,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        print_stdout,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 // Turns on no_std and alloc features if std is not available.
diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs
index 8f22caaaa7..10386e9902 100644
--- a/lib/entity/src/lib.rs
+++ b/lib/entity/src/lib.rs
@@ -40,14 +40,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        print_stdout,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 // Turns on no_std and alloc features if std is not available.
diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs
index 51da811fae..f9de8e7ab6 100644
--- a/lib/faerie/src/lib.rs
+++ b/lib/faerie/src/lib.rs
@@ -17,14 +17,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        print_stdout,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index 3a6581d602..ccdaab791a 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -14,12 +14,12 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        unicode_not_nfc,
-        use_self
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index 01dcdc4a2a..97514a75f2 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -164,14 +164,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        print_stdout,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 #![cfg_attr(not(feature = "std"), no_std)]
diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs
index e9e2131b41..7e31dc9f3c 100644
--- a/lib/module/src/lib.rs
+++ b/lib/module/src/lib.rs
@@ -11,14 +11,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        print_stdout,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 // Turns on no_std and alloc features if std is not available.
diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs
index f8732714a0..c62322afe9 100644
--- a/lib/native/src/lib.rs
+++ b/lib/native/src/lib.rs
@@ -16,14 +16,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        print_stdout,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 #![cfg_attr(not(feature = "std"), no_std)]
diff --git a/lib/preopt/src/lib.rs b/lib/preopt/src/lib.rs
index c55c70df74..c896a0c01c 100644
--- a/lib/preopt/src/lib.rs
+++ b/lib/preopt/src/lib.rs
@@ -11,14 +11,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        print_stdout,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 #![cfg_attr(not(feature = "std"), no_std)]
diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs
index 3b8222947a..ca79ea2099 100644
--- a/lib/reader/src/lib.rs
+++ b/lib/reader/src/lib.rs
@@ -18,14 +18,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        print_stdout,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 
diff --git a/lib/serde/src/clif-json.rs b/lib/serde/src/clif-json.rs
index 77498ca06e..f06a9e552b 100644
--- a/lib/serde/src/clif-json.rs
+++ b/lib/serde/src/clif-json.rs
@@ -15,13 +15,13 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 
diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs
index f50cb826c6..c1a44f672d 100644
--- a/lib/simplejit/src/lib.rs
+++ b/lib/simplejit/src/lib.rs
@@ -15,14 +15,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        print_stdout,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 
diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs
index 41a55f6fe4..fab42da984 100644
--- a/lib/umbrella/src/lib.rs
+++ b/lib/umbrella/src/lib.rs
@@ -15,14 +15,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        print_stdout,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index e528d5f966..6c23f97640 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -20,14 +20,14 @@
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
-        float_arithmetic,
-        mut_mut,
-        nonminimal_bool,
-        option_map_unwrap_or,
-        option_map_unwrap_or_else,
-        print_stdout,
-        unicode_not_nfc,
-        use_self
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
     )
 )]
 #![cfg_attr(not(feature = "std"), no_std)]

From f065b39d652a99ab78421da400b173b9035c2c71 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 7 Dec 2018 01:01:12 -0500
Subject: [PATCH 2242/3084] Add Wasmtime as another example user of Cranelift.

---
 cranelift/README.md | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/cranelift/README.md b/cranelift/README.md
index e695d28fad..800dc6cc02 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -20,6 +20,11 @@ implements a toy language.
 
 [SimpleJIT Demo]: https://github.com/CraneStation/simplejit-demo
 
+For an example of how to use Cranelift to run WebAssembly code, see
+[Wasmtime], which implements a standalone VM using Cranelift.
+
+[Wasmtime]: https://github.com/CraneStation/wasmtime
+
 Status
 ------
 

From 93696a80bb1deb47135134fc0e117abbaa758be9 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 5 Dec 2018 17:30:22 +0100
Subject: [PATCH 2243/3084] Fixes #614: Implement a debug trap;

---
 cranelift/filetests/isa/x86/binary64.clif    | 3 +++
 lib/codegen/meta-python/base/instructions.py | 4 ++++
 lib/codegen/meta-python/isa/x86/encodings.py | 5 +++++
 lib/codegen/meta-python/isa/x86/recipes.py   | 5 +++++
 4 files changed, 17 insertions(+)

diff --git a/cranelift/filetests/isa/x86/binary64.clif b/cranelift/filetests/isa/x86/binary64.clif
index 1c43280cb2..e876cd59c0 100644
--- a/cranelift/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/isa/x86/binary64.clif
@@ -784,6 +784,9 @@ ebb1:
     ; asm: jnbe .+4; ud2
     trapif ule v11, user0                          ; bin: 77 02 user0 0f 0b
 
+    ; Debug trap.
+    debugtrap ; bin: cc
+
     ; Stack check.
     ; asm: cmpq %rsp, %rcx
     [-,%rflags]         v40 = ifcmp_sp v1       ; bin: 48 39 e1
diff --git a/lib/codegen/meta-python/base/instructions.py b/lib/codegen/meta-python/base/instructions.py
index 718bb33f45..53d1586abf 100644
--- a/lib/codegen/meta-python/base/instructions.py
+++ b/lib/codegen/meta-python/base/instructions.py
@@ -181,6 +181,10 @@ indirect_jump_table_br = Instruction(
     ins=(addr, JT),
     is_branch=True, is_indirect_branch=True, is_terminator=True)
 
+debugtrap = Instruction('debugtrap', r"""
+    Encodes an assembly debug trap.
+    """, can_load=True, can_store=True, other_side_effects=True)
+
 code = Operand('code', trapcode)
 trap = Instruction(
         'trap', r"""
diff --git a/lib/codegen/meta-python/isa/x86/encodings.py b/lib/codegen/meta-python/isa/x86/encodings.py
index d07e003c3c..4ae37c48a2 100644
--- a/lib/codegen/meta-python/isa/x86/encodings.py
+++ b/lib/codegen/meta-python/isa/x86/encodings.py
@@ -521,12 +521,17 @@ X86_32.enc(base.jump_table_base.i32, *r.jt_base(0x8d))
 
 enc_x86_64(base.indirect_jump_table_br.i64, r.indirect_jmp, 0xff, rrr=4)
 X86_32.enc(base.indirect_jump_table_br.i32, *r.indirect_jmp(0xff, rrr=4))
+
 #
 # Trap as ud2
 #
 X86_32.enc(base.trap, *r.trap(0x0f, 0x0b))
 X86_64.enc(base.trap, *r.trap(0x0f, 0x0b))
 
+# Debug trap as int3
+X86_32.enc(base.debugtrap, r.debugtrap, 0)
+X86_64.enc(base.debugtrap, r.debugtrap, 0)
+
 # Using a standard EncRecipe, not the TailRecipe.
 X86_32.enc(base.trapif, r.trapif, 0)
 X86_64.enc(base.trapif, r.trapif, 0)
diff --git a/lib/codegen/meta-python/isa/x86/recipes.py b/lib/codegen/meta-python/isa/x86/recipes.py
index 534804e0e2..484f61ee6f 100644
--- a/lib/codegen/meta-python/isa/x86/recipes.py
+++ b/lib/codegen/meta-python/isa/x86/recipes.py
@@ -295,6 +295,11 @@ def valid_scale(iform):
 # copies and no-op conversions.
 null = EncRecipe('null', Unary, base_size=0, ins=GPR, outs=0, emit='')
 
+debugtrap = EncRecipe('debugtrap', NullAry, base_size=1, ins=(), outs=(),
+                      emit='''
+                      sink.put1(0xcc);
+                      ''')
+
 # XX opcode, no ModR/M.
 trap = TailRecipe(
         'trap', Trap, base_size=0, ins=(), outs=(),

From a20c852148e3995c283568c9d47bd80326c26c89 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 29 Nov 2018 04:53:30 -0800
Subject: [PATCH 2244/3084] Support heaps with no offset-guard pages.

Also, say "guard-offset pages" rather than just "guard pages" to describe the
region of a heap which is never accessible and which exists to support
optimizations for heap accesses with offsets.

And, introduce a `Uimm64` immediate type, and make all heap fields use
`Uimm64` instead of `Imm64` since they really are unsigned.
---
 cranelift/docs/heapex-dyn.clif                |   2 +-
 cranelift/docs/heapex-sm32.clif               |   2 +-
 cranelift/docs/heapex-sm64.clif               |   2 +-
 cranelift/docs/ir.rst                         |  37 ++--
 .../filetests/isa/x86/legalize-heaps.clif     |  32 ++--
 .../filetests/isa/x86/legalize-memory.clif    |   6 +-
 cranelift/filetests/parser/memory.clif        |  16 +-
 cranelift/filetests/regalloc/aliases.clif     |   2 +-
 .../filetests/regalloc/coalescing-207.clif    |   4 +-
 .../filetests/regalloc/coloring-227.clif      |   2 +-
 cranelift/filetests/regalloc/reload-208.clif  |   2 +-
 cranelift/filetests/simple_gvn/readonly.clif  |   2 +-
 cranelift/filetests/verifier/heap.clif        |  10 +-
 cranelift/filetests/wasm/f32-memory64.clif    |   4 +-
 cranelift/filetests/wasm/f64-memory64.clif    |   4 +-
 cranelift/filetests/wasm/i32-memory64.clif    |  16 +-
 cranelift/filetests/wasm/i64-memory64.clif    |  22 +--
 lib/codegen/src/ir/heap.rs                    |  18 +-
 lib/codegen/src/ir/immediates.rs              | 174 +++++++++++++++---
 lib/codegen/src/ir/table.rs                   |   6 +-
 lib/codegen/src/ir/trapcode.rs                |   3 +-
 lib/codegen/src/legalizer/heap.rs             |  16 +-
 lib/codegen/src/legalizer/table.rs            |  12 +-
 lib/reader/src/parser.rs                      |  46 +++--
 lib/umbrella/src/lib.rs                       |   2 +-
 lib/wasm/src/code_translator.rs               |  24 +--
 lib/wasm/src/environ/dummy.rs                 |   8 +-
 27 files changed, 302 insertions(+), 172 deletions(-)

diff --git a/cranelift/docs/heapex-dyn.clif b/cranelift/docs/heapex-dyn.clif
index 276cbdd491..24fc254d98 100644
--- a/cranelift/docs/heapex-dyn.clif
+++ b/cranelift/docs/heapex-dyn.clif
@@ -4,7 +4,7 @@ function %add_members(i32, i64 vmctx) -> f32 baldrdash {
     gv0 = vmctx
     gv1 = load.i64 notrap aligned gv0+64
     gv2 = load.i32 notrap aligned gv0+72
-    heap0 = dynamic gv1, min 0x1000, bound gv2, guard 0
+    heap0 = dynamic gv1, min 0x1000, bound gv2, offset_guard 0
 
 ebb0(v0: i32, v6: i64):
     v1 = heap_addr.i64 heap0, v0, 20
diff --git a/cranelift/docs/heapex-sm32.clif b/cranelift/docs/heapex-sm32.clif
index 88b78e9ea7..9c9c35e8f9 100644
--- a/cranelift/docs/heapex-sm32.clif
+++ b/cranelift/docs/heapex-sm32.clif
@@ -3,7 +3,7 @@ test verifier
 function %add_members(i32, i32 vmctx) -> f32 baldrdash {
     gv0 = vmctx
     gv1 = load.i32 notrap aligned gv0+64
-    heap0 = static gv1, min 0x1000, bound 0x10_0000, guard 0x1000
+    heap0 = static gv1, min 0x1000, bound 0x10_0000, offset_guard 0x1000
 
 ebb0(v0: i32, v5: i32):
     v1 = heap_addr.i32 heap0, v0, 1
diff --git a/cranelift/docs/heapex-sm64.clif b/cranelift/docs/heapex-sm64.clif
index 1c14ea6392..22752eb401 100644
--- a/cranelift/docs/heapex-sm64.clif
+++ b/cranelift/docs/heapex-sm64.clif
@@ -3,7 +3,7 @@ test verifier
 function %add_members(i32, i64 vmctx) -> f32 baldrdash {
     gv0 = vmctx
     gv1 = load.i64 notrap aligned gv0+64
-    heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000
+    heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v5: i64):
     v1 = heap_addr.i64 heap0, v0, 1
diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index bdbcef7f5c..faf941863c 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -651,7 +651,7 @@ architecture.
         fontsize=10,
         fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans"
     ]
-    "static" [label="mapped\npages|unmapped\npages|guard\npages"]
+    "static" [label="mapped\npages|unmapped\npages|offset_guard\npages"]
 
 A heap appears as three consecutive ranges of address space:
 
@@ -661,9 +661,9 @@ A heap appears as three consecutive ranges of address space:
 2. The *unmapped pages* is a possibly empty range of address space that may be
    mapped in the future when the heap is grown. They are :term:`addressable` but
    not :term:`accessible`.
-3. The *guard pages* is a range of address space that is guaranteed to cause a
-   trap when accessed. It is used to optimize bounds checking for heap accesses
-   with a shared base pointer. They are :term:`addressable` but
+3. The *offset-guard pages* is a range of address space that is guaranteed to
+   always cause a trap when accessed. It is used to optimize bounds checking for
+   heap accesses with a shared base pointer. They are :term:`addressable` but
    not :term:`accessible`.
 
 The *heap bound* is the total size of the mapped and unmapped pages. This is
@@ -683,10 +683,10 @@ A *static heap* starts out with all the address space it will ever need, so it
 never moves to a different address. At the base address is a number of mapped
 pages corresponding to the heap's current size. Then follows a number of
 unmapped pages where the heap can grow up to its maximum size. After the
-unmapped pages follow the guard pages which are also guaranteed to generate a
-trap when accessed.
+unmapped pages follow the offset-guard pages which are also guaranteed to
+generate a trap when accessed.
 
-.. inst:: H = static Base, min MinBytes, bound BoundBytes, guard GuardBytes
+.. inst:: H = static Base, min MinBytes, bound BoundBytes, offset_guard OffsetGuardBytes
 
     Declare a static heap in the preamble.
 
@@ -694,17 +694,18 @@ trap when accessed.
     :arg MinBytes: Guaranteed minimum heap size in bytes. Accesses below this
             size will never trap.
     :arg BoundBytes: Fixed heap bound in bytes. This defines the amount of
-            address space reserved for the heap, not including the guard pages.
-    :arg GuardBytes: Size of the guard pages in bytes.
+            address space reserved for the heap, not including the offset-guard
+            pages.
+    :arg OffsetGuardBytes: Size of the offset-guard pages in bytes.
 
 Dynamic heaps
 ~~~~~~~~~~~~~
 
 A *dynamic heap* can be relocated to a different base address when it is
-resized, and its bound can move dynamically. The guard pages move when the heap
-is resized. The bound of a dynamic heap is stored in a global value.
+resized, and its bound can move dynamically. The offset-guard pages move when
+the heap is resized. The bound of a dynamic heap is stored in a global value.
 
-.. inst:: H = dynamic Base, min MinBytes, bound BoundGV, guard GuardBytes
+.. inst:: H = dynamic Base, min MinBytes, bound BoundGV, offset_guard OffsetGuardBytes
 
     Declare a dynamic heap in the preamble.
 
@@ -712,14 +713,14 @@ is resized. The bound of a dynamic heap is stored in a global value.
     :arg MinBytes: Guaranteed minimum heap size in bytes. Accesses below this
             size will never trap.
     :arg BoundGV: Global value containing the current heap bound in bytes.
-    :arg GuardBytes: Size of the guard pages in bytes.
+    :arg OffsetGuardBytes: Size of the offset-guard pages in bytes.
 
 Heap examples
 ~~~~~~~~~~~~~
 
 The SpiderMonkey VM prefers to use fixed heaps with a 4 GB bound and 2 GB of
-guard pages when running WebAssembly code on 64-bit CPUs. The combination of a
-4 GB fixed bound and 1-byte bounds checks means that no code needs to be
+offset-guard pages when running WebAssembly code on 64-bit CPUs. The combination
+of a 4 GB fixed bound and 1-byte bounds checks means that no code needs to be
 generated for bounds checks at all:
 
 .. literalinclude:: heapex-sm64.clif
@@ -728,7 +729,7 @@ generated for bounds checks at all:
 
 A static heap can also be used for 32-bit code when the WebAssembly module
 declares a small upper bound on its memory. A 1 MB static bound with a single 4
-KB guard page still has opportunities for sharing bounds checking code:
+KB offset-guard page still has opportunities for sharing bounds checking code:
 
 .. literalinclude:: heapex-sm32.clif
     :language: clif
@@ -738,8 +739,8 @@ If the upper bound on the heap size is too large, a dynamic heap is required
 instead.
 
 Finally, a runtime environment that simply allocates a heap with
-:c:func:`malloc()` may not have any guard pages at all. In that case, full
-bounds checking is required for each access:
+:c:func:`malloc()` may not have any offset-guard pages at all. In that case,
+full bounds checking is required for each access:
 
 .. literalinclude:: heapex-dyn.clif
     :language: clif
diff --git a/cranelift/filetests/isa/x86/legalize-heaps.clif b/cranelift/filetests/isa/x86/legalize-heaps.clif
index df2be3f287..5f60c7b43f 100644
--- a/cranelift/filetests/isa/x86/legalize-heaps.clif
+++ b/cranelift/filetests/isa/x86/legalize-heaps.clif
@@ -10,23 +10,23 @@ function %heap_addrs(i32, i64, i64 vmctx) {
     gv2 = iadd_imm.i64 gv4, 80
     gv3 = load.i32 notrap aligned gv4+88
 
-    heap0 = static gv0, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000, index_type i32
-    heap1 = static gv0, guard 0x1000, bound 0x1_0000, index_type i32
-    heap2 = static gv0, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000, index_type i64
-    heap3 = static gv0, guard 0x1000, bound 0x1_0000, index_type i64
-    heap4 = dynamic gv1, min 0x1_0000, bound gv3, guard 0x8000_0000, index_type i32
-    heap5 = dynamic gv1, bound gv3, guard 0x1000, index_type i32
-    heap6 = dynamic gv1, min 0x1_0000, bound gv2, guard 0x8000_0000, index_type i64
-    heap7 = dynamic gv1, bound gv2, guard 0x1000, index_type i64
+    heap0 = static gv0, min 0x1_0000, bound 0x1_0000_0000, offset_guard 0x8000_0000, index_type i32
+    heap1 = static gv0, offset_guard 0x1000, bound 0x1_0000, index_type i32
+    heap2 = static gv0, min 0x1_0000, bound 0x1_0000_0000, offset_guard 0x8000_0000, index_type i64
+    heap3 = static gv0, offset_guard 0x1000, bound 0x1_0000, index_type i64
+    heap4 = dynamic gv1, min 0x1_0000, bound gv3, offset_guard 0x8000_0000, index_type i32
+    heap5 = dynamic gv1, bound gv3, offset_guard 0x1000, index_type i32
+    heap6 = dynamic gv1, min 0x1_0000, bound gv2, offset_guard 0x8000_0000, index_type i64
+    heap7 = dynamic gv1, bound gv2, offset_guard 0x1000, index_type i64
 
-    ; check: heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000, index_type i32
-    ; check: heap1 = static gv0, min 0, bound 0x0001_0000, guard 4096, index_type i32
-    ; check: heap2 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000, index_type i64
-    ; check: heap3 = static gv0, min 0, bound 0x0001_0000, guard 4096, index_type i64
-    ; check: heap4 = dynamic gv1, min 0x0001_0000, bound gv3, guard 0x8000_0000, index_type i32
-    ; check: heap5 = dynamic gv1, min 0, bound gv3, guard 4096, index_type i32
-    ; check: heap6 = dynamic gv1, min 0x0001_0000, bound gv2, guard 0x8000_0000, index_type i64
-    ; check: heap7 = dynamic gv1, min 0, bound gv2, guard 4096, index_type i64
+    ; check: heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000, index_type i32
+    ; check: heap1 = static gv0, min 0, bound 0x0001_0000, offset_guard 4096, index_type i32
+    ; check: heap2 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000, index_type i64
+    ; check: heap3 = static gv0, min 0, bound 0x0001_0000, offset_guard 4096, index_type i64
+    ; check: heap4 = dynamic gv1, min 0x0001_0000, bound gv3, offset_guard 0x8000_0000, index_type i32
+    ; check: heap5 = dynamic gv1, min 0, bound gv3, offset_guard 4096, index_type i32
+    ; check: heap6 = dynamic gv1, min 0x0001_0000, bound gv2, offset_guard 0x8000_0000, index_type i64
+    ; check: heap7 = dynamic gv1, min 0, bound gv2, offset_guard 4096, index_type i64
 
 ebb0(v0: i32, v1: i64, v3: i64):
     ; The fast-path; 32-bit index, static heap with a sufficient bound, no bounds check needed!
diff --git a/cranelift/filetests/isa/x86/legalize-memory.clif b/cranelift/filetests/isa/x86/legalize-memory.clif
index fbb9ca1f37..eb24523b14 100644
--- a/cranelift/filetests/isa/x86/legalize-memory.clif
+++ b/cranelift/filetests/isa/x86/legalize-memory.clif
@@ -47,7 +47,7 @@ ebb1:
 function %staticheap_sm64(i32, i64 vmctx) -> f32 baldrdash {
     gv0 = vmctx
     gv1 = iadd_imm.i64 gv0, 64
-    heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000
+    heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v999: i64):
     ; check: ebb0(
@@ -68,7 +68,7 @@ ebb0(v0: i32, v999: i64):
 function %staticheap_static_oob_sm64(i32, i64 vmctx) -> f32 baldrdash {
     gv0 = vmctx
     gv1 = iadd_imm.i64 gv0, 64
-    heap0 = static gv1, min 0x1000, bound 0x1000_0000, guard 0x8000_0000
+    heap0 = static gv1, min 0x1000, bound 0x1000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v999: i64):
     ; Everything after the obviously OOB access should be eliminated, leaving
@@ -92,7 +92,7 @@ ebb0(v0: i32, v999: i64):
 function %staticheap_sm64(i32, i64 vmctx) -> f32 baldrdash {
     gv0 = vmctx
     gv1 = iadd_imm.i64 gv0, 64
-    heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, guard 0x8000_0000
+    heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v999: i64):
     ; check: ebb0(
diff --git a/cranelift/filetests/parser/memory.clif b/cranelift/filetests/parser/memory.clif
index d406007dfe..4e763f2b4d 100644
--- a/cranelift/filetests/parser/memory.clif
+++ b/cranelift/filetests/parser/memory.clif
@@ -52,13 +52,13 @@ ebb0:
 
 ; Declare static heaps.
 function %sheap(i32, i64 vmctx) -> i64 {
-    heap1 = static gv5, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000
-    heap2 = static gv5, guard 0x1000, bound 0x1_0000
+    heap1 = static gv5, min 0x1_0000, bound 0x1_0000_0000, offset_guard 0x8000_0000
+    heap2 = static gv5, offset_guard 0x1000, bound 0x1_0000
     gv4 = vmctx
     gv5 = iadd_imm.i64 gv4, 64
 
-    ; check: heap1 = static gv5, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
-    ; check: heap2 = static gv5, min 0, bound 0x0001_0000, guard 4096
+    ; check: heap1 = static gv5, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
+    ; check: heap2 = static gv5, min 0, bound 0x0001_0000, offset_guard 4096
 ebb0(v1: i32, v2: i64):
     v3 = heap_addr.i64 heap1, v1, 0
     ; check: v3 = heap_addr.i64 heap1, v1, 0
@@ -67,14 +67,14 @@ ebb0(v1: i32, v2: i64):
 
 ; Declare dynamic heaps.
 function %dheap(i32, i64 vmctx) -> i64 {
-    heap1 = dynamic gv5, min 0x1_0000, bound gv6, guard 0x8000_0000
-    heap2 = dynamic gv5, bound gv6, guard 0x1000
+    heap1 = dynamic gv5, min 0x1_0000, bound gv6, offset_guard 0x8000_0000
+    heap2 = dynamic gv5, bound gv6, offset_guard 0x1000
     gv4 = vmctx
     gv5 = iadd_imm.i64 gv4, 64
     gv6 = iadd_imm.i64 gv4, 72
 
-    ; check: heap1 = dynamic gv5, min 0x0001_0000, bound gv6, guard 0x8000_0000
-    ; check: heap2 = dynamic gv5, min 0, bound gv6, guard 4096
+    ; check: heap1 = dynamic gv5, min 0x0001_0000, bound gv6, offset_guard 0x8000_0000
+    ; check: heap2 = dynamic gv5, min 0, bound gv6, offset_guard 4096
 ebb0(v1: i32, v2: i64):
     v3 = heap_addr.i64 heap2, v1, 0
     ; check: v3 = heap_addr.i64 heap2, v1, 0
diff --git a/cranelift/filetests/regalloc/aliases.clif b/cranelift/filetests/regalloc/aliases.clif
index b9b76c3f1d..2a94192415 100644
--- a/cranelift/filetests/regalloc/aliases.clif
+++ b/cranelift/filetests/regalloc/aliases.clif
@@ -3,7 +3,7 @@ target x86_64 haswell
 
 function %value_aliases(i32, f32, i64 vmctx) baldrdash {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: f32, v2: i64):
     v3 = iconst.i32 0
diff --git a/cranelift/filetests/regalloc/coalescing-207.clif b/cranelift/filetests/regalloc/coalescing-207.clif
index 37b028af03..30b33ee44a 100644
--- a/cranelift/filetests/regalloc/coalescing-207.clif
+++ b/cranelift/filetests/regalloc/coalescing-207.clif
@@ -7,7 +7,7 @@ target x86_64 haswell
 function %pr207(i64 vmctx, i32, i32) -> i32 system_v {
     gv1 = vmctx
     gv0 = iadd_imm.i64 gv1, -8
-    heap0 = static gv0, min 0, bound 0x5000, guard 0x0040_0000
+    heap0 = static gv0, min 0, bound 0x5000, offset_guard 0x0040_0000
     sig0 = (i64 vmctx, i32, i32) -> i32 system_v
     sig1 = (i64 vmctx, i32, i32, i32) -> i32 system_v
     sig2 = (i64 vmctx, i32, i32, i32) -> i32 system_v
@@ -1036,7 +1036,7 @@ ebb92(v767: i32):
 ; Same problem from musl.wasm.
 function %musl(f64 [%xmm0], i64 vmctx [%rdi]) -> f64 [%xmm0] system_v {
     gv0 = vmctx
-    heap0 = static gv0, min 0, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0, bound 0x0001_0000_0000, offset_guard 0x8000_0000
     sig0 = (f64 [%xmm0], i32 [%rdi], i64 vmctx [%rsi]) -> f64 [%xmm0] system_v
     fn0 = u0:517 sig0
 
diff --git a/cranelift/filetests/regalloc/coloring-227.clif b/cranelift/filetests/regalloc/coloring-227.clif
index e5aeb3aa82..41d6817ffd 100644
--- a/cranelift/filetests/regalloc/coloring-227.clif
+++ b/cranelift/filetests/regalloc/coloring-227.clif
@@ -3,7 +3,7 @@ target x86_64 haswell
 
 function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8]) system_v {
     gv0 = vmctx
-    heap0 = static gv0, min 0, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
                           ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i64):
 [RexOp1pu_id#b8]              v5 = iconst.i32 0
diff --git a/cranelift/filetests/regalloc/reload-208.clif b/cranelift/filetests/regalloc/reload-208.clif
index 629d984f62..116e5b719f 100644
--- a/cranelift/filetests/regalloc/reload-208.clif
+++ b/cranelift/filetests/regalloc/reload-208.clif
@@ -13,7 +13,7 @@ target x86_64 haswell
 function %pr208(i64 vmctx [%rdi]) system_v {
     gv1 = vmctx
     gv0 = iadd_imm.i64 gv1, -8
-    heap0 = static gv0, min 0, bound 0x5000, guard 0x0040_0000
+    heap0 = static gv0, min 0, bound 0x5000, offset_guard 0x0040_0000
     sig0 = (i64 vmctx [%rdi]) -> i32 [%rax] system_v
     sig1 = (i64 vmctx [%rdi], i32 [%rsi]) system_v
     fn0 = u0:1 sig0
diff --git a/cranelift/filetests/simple_gvn/readonly.clif b/cranelift/filetests/simple_gvn/readonly.clif
index 28eacce61c..3c01299064 100644
--- a/cranelift/filetests/simple_gvn/readonly.clif
+++ b/cranelift/filetests/simple_gvn/readonly.clif
@@ -5,7 +5,7 @@ target x86_64
 function %eliminate_redundant_global_loads(i32, i64 vmctx) {
     gv0 = vmctx
     gv1 = load.i64 notrap aligned readonly gv0
-    heap0 = static gv1, min 0x1_0000, bound 0x1_0000_0000, guard 0x8000_0000, index_type i32
+    heap0 = static gv1, min 0x1_0000, bound 0x1_0000_0000, offset_guard 0x8000_0000, index_type i32
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
diff --git a/cranelift/filetests/verifier/heap.clif b/cranelift/filetests/verifier/heap.clif
index a436c623d8..2c73b726ba 100644
--- a/cranelift/filetests/verifier/heap.clif
+++ b/cranelift/filetests/verifier/heap.clif
@@ -4,7 +4,7 @@ target x86_64
 function %heap_base_type(i64 vmctx) {
     gv0 = vmctx
     gv1 = load.i32 notrap aligned gv0
-    heap0 = static gv1, guard 0x1000, bound 0x1_0000, index_type i32 ; error: heap base has type i32, which is not the pointer type i64
+    heap0 = static gv1, offset_guard 0x1000, bound 0x1_0000, index_type i32 ; error: heap base has type i32, which is not the pointer type i64
 
 ebb0(v0: i64):
     return
@@ -12,7 +12,7 @@ ebb0(v0: i64):
 
 function %invalid_base(i64 vmctx) {
     gv0 = vmctx
-    heap0 = dynamic gv1, bound gv0, guard 0x1000, index_type i64 ; error: invalid base global value gv1
+    heap0 = dynamic gv1, bound gv0, offset_guard 0x1000, index_type i64 ; error: invalid base global value gv1
 
 ebb0(v0: i64):
     return
@@ -20,7 +20,7 @@ ebb0(v0: i64):
 
 function %invalid_bound(i64 vmctx) {
     gv0 = vmctx
-    heap0 = dynamic gv0, bound gv1, guard 0x1000, index_type i64 ; error: invalid bound global value gv1
+    heap0 = dynamic gv0, bound gv1, offset_guard 0x1000, index_type i64 ; error: invalid bound global value gv1
 
 ebb0(v0: i64):
     return
@@ -29,7 +29,7 @@ ebb0(v0: i64):
 function %heap_bound_type(i64 vmctx) {
     gv0 = vmctx
     gv1 = load.i16 notrap aligned gv0
-    heap0 = dynamic gv0, bound gv1, guard 0x1000, index_type i32 ; error: heap index type i32 differs from the type of its bound, i16
+    heap0 = dynamic gv0, bound gv1, offset_guard 0x1000, index_type i32 ; error: heap index type i32 differs from the type of its bound, i16
 
 ebb0(v0: i64):
     return
@@ -37,7 +37,7 @@ ebb0(v0: i64):
 
 function %heap_addr_index_type(i64 vmctx, i64) {
     gv0 = vmctx
-    heap0 = static gv0, guard 0x1000, bound 0x1_0000, index_type i32
+    heap0 = static gv0, offset_guard 0x1000, bound 0x1_0000, index_type i32
 
 ebb0(v0: i64, v1: i64):
     v2 = heap_addr.i64 heap0, v1, 0; error: index type i64 differs from heap index type i32
diff --git a/cranelift/filetests/wasm/f32-memory64.clif b/cranelift/filetests/wasm/f32-memory64.clif
index 42b183311b..edc5c22780 100644
--- a/cranelift/filetests/wasm/f32-memory64.clif
+++ b/cranelift/filetests/wasm/f32-memory64.clif
@@ -7,7 +7,7 @@ target x86_64 haswell
 
 function %f32_load(i32, i64 vmctx) -> f32 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -17,7 +17,7 @@ ebb0(v0: i32, v1: i64):
 
 function %f32_store(f32, i32, i64 vmctx) {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: f32, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
diff --git a/cranelift/filetests/wasm/f64-memory64.clif b/cranelift/filetests/wasm/f64-memory64.clif
index b495dfd344..85351d9d8d 100644
--- a/cranelift/filetests/wasm/f64-memory64.clif
+++ b/cranelift/filetests/wasm/f64-memory64.clif
@@ -7,7 +7,7 @@ target x86_64 haswell
 
 function %f64_load(i32, i64 vmctx) -> f64 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -17,7 +17,7 @@ ebb0(v0: i32, v1: i64):
 
 function %f64_store(f64, i32, i64 vmctx) {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: f64, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
diff --git a/cranelift/filetests/wasm/i32-memory64.clif b/cranelift/filetests/wasm/i32-memory64.clif
index b32325561a..306c4e4aa5 100644
--- a/cranelift/filetests/wasm/i32-memory64.clif
+++ b/cranelift/filetests/wasm/i32-memory64.clif
@@ -7,7 +7,7 @@ target x86_64 haswell
 
 function %i32_load(i32, i64 vmctx) -> i32 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -17,7 +17,7 @@ ebb0(v0: i32, v1: i64):
 
 function %i32_store(i32, i32, i64 vmctx) {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
@@ -27,7 +27,7 @@ ebb0(v0: i32, v1: i32, v2: i64):
 
 function %i32_load8_s(i32, i64 vmctx) -> i32 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -37,7 +37,7 @@ ebb0(v0: i32, v1: i64):
 
 function %i32_load8_u(i32, i64 vmctx) -> i32 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -47,7 +47,7 @@ ebb0(v0: i32, v1: i64):
 
 function %i32_store8(i32, i32, i64 vmctx) {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
@@ -57,7 +57,7 @@ ebb0(v0: i32, v1: i32, v2: i64):
 
 function %i32_load16_s(i32, i64 vmctx) -> i32 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -67,7 +67,7 @@ ebb0(v0: i32, v1: i64):
 
 function %i32_load16_u(i32, i64 vmctx) -> i32 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -77,7 +77,7 @@ ebb0(v0: i32, v1: i64):
 
 function %i32_store16(i32, i32, i64 vmctx) {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
diff --git a/cranelift/filetests/wasm/i64-memory64.clif b/cranelift/filetests/wasm/i64-memory64.clif
index 76a1a02e4b..edea6da503 100644
--- a/cranelift/filetests/wasm/i64-memory64.clif
+++ b/cranelift/filetests/wasm/i64-memory64.clif
@@ -7,7 +7,7 @@ target x86_64 haswell
 
 function %i64_load(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -17,7 +17,7 @@ ebb0(v0: i32, v1: i64):
 
 function %i64_store(i64, i32, i64 vmctx) {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i64, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
@@ -27,7 +27,7 @@ ebb0(v0: i64, v1: i32, v2: i64):
 
 function %i64_load8_s(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -37,7 +37,7 @@ ebb0(v0: i32, v1: i64):
 
 function %i64_load8_u(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -47,7 +47,7 @@ ebb0(v0: i32, v1: i64):
 
 function %i64_store8(i64, i32, i64 vmctx) {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i64, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
@@ -57,7 +57,7 @@ ebb0(v0: i64, v1: i32, v2: i64):
 
 function %i64_load16_s(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -67,7 +67,7 @@ ebb0(v0: i32, v1: i64):
 
 function %i64_load16_u(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -77,7 +77,7 @@ ebb0(v0: i32, v1: i64):
 
 function %i64_store16(i64, i32, i64 vmctx) {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i64, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
@@ -87,7 +87,7 @@ ebb0(v0: i64, v1: i32, v2: i64):
 
 function %i64_load32_s(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -97,7 +97,7 @@ ebb0(v0: i32, v1: i64):
 
 function %i64_load32_u(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
@@ -107,7 +107,7 @@ ebb0(v0: i32, v1: i64):
 
 function %i64_store32(i64, i32, i64 vmctx) {
     gv0 = vmctx
-    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, guard 0x8000_0000
+    heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
 ebb0(v0: i64, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
diff --git a/lib/codegen/src/ir/heap.rs b/lib/codegen/src/ir/heap.rs
index d31c4c1f15..f8d1b78cb0 100644
--- a/lib/codegen/src/ir/heap.rs
+++ b/lib/codegen/src/ir/heap.rs
@@ -1,6 +1,6 @@
 //! Heaps.
 
-use ir::immediates::Imm64;
+use ir::immediates::Uimm64;
 use ir::{GlobalValue, Type};
 use std::fmt;
 
@@ -12,10 +12,10 @@ pub struct HeapData {
 
     /// Guaranteed minimum heap size in bytes. Heap accesses before `min_size` don't need bounds
     /// checking.
-    pub min_size: Imm64,
+    pub min_size: Uimm64,
 
-    /// Size in bytes of the guard pages following the heap.
-    pub guard_size: Imm64,
+    /// Size in bytes of the offset-guard pages following the heap.
+    pub offset_guard_size: Uimm64,
 
     /// Heap style, with additional style-specific info.
     pub style: HeapStyle,
@@ -34,10 +34,10 @@ pub enum HeapStyle {
     },
 
     /// A static heap has a fixed base address and a number of not-yet-allocated pages before the
-    /// guard pages.
+    /// offset-guard pages.
     Static {
-        /// Heap bound in bytes. The guard pages are allocated after the bound.
-        bound: Imm64,
+        /// Heap bound in bytes. The offset-guard pages are allocated after the bound.
+        bound: Uimm64,
     },
 }
 
@@ -55,8 +55,8 @@ impl fmt::Display for HeapData {
         }
         write!(
             f,
-            ", guard {}, index_type {}",
-            self.guard_size, self.index_type
+            ", offset_guard {}, index_type {}",
+            self.offset_guard_size, self.index_type
         )
     }
 }
diff --git a/lib/codegen/src/ir/immediates.rs b/lib/codegen/src/ir/immediates.rs
index 3d8247845a..9badf2fcfd 100644
--- a/lib/codegen/src/ir/immediates.rs
+++ b/lib/codegen/src/ir/immediates.rs
@@ -9,7 +9,7 @@ use std::mem;
 use std::str::FromStr;
 use std::{i32, u32};
 
-/// 64-bit immediate integer operand.
+/// 64-bit immediate signed integer operand.
 ///
 /// An `Imm64` operand can also be used to represent immediate values of smaller integer types by
 /// sign-extending to `i64`.
@@ -40,13 +40,87 @@ impl From for Imm64 {
     }
 }
 
+impl Display for Imm64 {
+    fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+        let x = self.0;
+        if -10_000 < x && x < 10_000 {
+            // Use decimal for small numbers.
+            write!(f, "{}", x)
+        } else {
+            write_hex(x as u64, f)
+        }
+    }
+}
+
+/// Parse a 64-bit signed number.
+fn parse_i64(s: &str) -> Result {
+    let negative = s.starts_with('-');
+    let s2 = if negative || s.starts_with('+') {
+        &s[1..]
+    } else {
+        s
+    };
+
+    let mut value = parse_u64(s2)?;
+
+    // We support the range-and-a-half from -2^63 .. 2^64-1.
+    if negative {
+        value = value.wrapping_neg();
+        // Don't allow large negative values to wrap around and become positive.
+        if value as i64 > 0 {
+            return Err("Negative number too small");
+        }
+    }
+    Ok(value as i64)
+}
+
+impl FromStr for Imm64 {
+    type Err = &'static str;
+
+    // Parse a decimal or hexadecimal `Imm64`, formatted as above.
+    fn from_str(s: &str) -> Result {
+        parse_i64(s).map(Self::new)
+    }
+}
+
+/// 64-bit immediate unsigned integer operand.
+///
+/// A `Uimm64` operand can also be used to represent immediate values of smaller integer types by
+/// zero-extending to `i64`.
+#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)]
+pub struct Uimm64(u64);
+
+impl Uimm64 {
+    /// Create a new `Uimm64` representing the unsigned number `x`.
+    pub fn new(x: u64) -> Self {
+        Uimm64(x)
+    }
+
+    /// Return self negated.
+    pub fn wrapping_neg(self) -> Self {
+        Uimm64(self.0.wrapping_neg())
+    }
+}
+
+impl Into for Uimm64 {
+    fn into(self) -> u64 {
+        self.0
+    }
+}
+
+impl From for Uimm64 {
+    fn from(x: u64) -> Self {
+        Uimm64(x)
+    }
+}
+
 /// Hexadecimal with a multiple of 4 digits and group separators:
 ///
 ///   0xfff0
 ///   0x0001_ffff
 ///   0xffff_ffff_fff8_4400
 ///
-fn write_hex(x: i64, f: &mut Formatter) -> fmt::Result {
+fn write_hex(x: u64, f: &mut Formatter) -> fmt::Result {
     let mut pos = (64 - x.leading_zeros() - 1) & 0xf0;
     write!(f, "0x{:04x}", (x >> pos) & 0xffff)?;
     while pos > 0 {
@@ -56,10 +130,10 @@ fn write_hex(x: i64, f: &mut Formatter) -> fmt::Result {
     Ok(())
 }
 
-impl Display for Imm64 {
+impl Display for Uimm64 {
     fn fmt(&self, f: &mut Formatter) -> fmt::Result {
         let x = self.0;
-        if -10_000 < x && x < 10_000 {
+        if x < 10_000 {
             // Use decimal for small numbers.
             write!(f, "{}", x)
         } else {
@@ -68,20 +142,16 @@ impl Display for Imm64 {
     }
 }
 
-/// Parse a 64-bit number.
-fn parse_i64(s: &str) -> Result {
+/// Parse a 64-bit unsigned number.
+fn parse_u64(s: &str) -> Result {
     let mut value: u64 = 0;
     let mut digits = 0;
-    let negative = s.starts_with('-');
-    let s2 = if negative || s.starts_with('+') {
-        &s[1..]
-    } else {
-        s
-    };
 
-    if s2.starts_with("0x") {
+    if s.starts_with("-0x") {
+        return Err("Invalid character in hexadecimal number");
+    } else if s.starts_with("0x") {
         // Hexadecimal.
-        for ch in s2[2..].chars() {
+        for ch in s[2..].chars() {
             match ch.to_digit(16) {
                 Some(digit) => {
                     digits += 1;
@@ -101,7 +171,7 @@ fn parse_i64(s: &str) -> Result {
         }
     } else {
         // Decimal number, possibly negative.
-        for ch in s2.chars() {
+        for ch in s.chars() {
             match ch.to_digit(16) {
                 Some(digit) => {
                     digits += 1;
@@ -128,23 +198,15 @@ fn parse_i64(s: &str) -> Result {
         return Err("No digits in number");
     }
 
-    // We support the range-and-a-half from -2^63 .. 2^64-1.
-    if negative {
-        value = value.wrapping_neg();
-        // Don't allow large negative values to wrap around and become positive.
-        if value as i64 > 0 {
-            return Err("Negative number too small");
-        }
-    }
-    Ok(value as i64)
+    Ok(value)
 }
 
-impl FromStr for Imm64 {
+impl FromStr for Uimm64 {
     type Err = &'static str;
 
-    // Parse a decimal or hexadecimal `Imm64`, formatted as above.
+    // Parse a decimal or hexadecimal `Uimm64`, formatted as above.
     fn from_str(s: &str) -> Result {
-        parse_i64(s).map(Self::new)
+        parse_u64(s).map(Self::new)
     }
 }
 
@@ -182,7 +244,7 @@ impl Display for Uimm32 {
         if self.0 < 10_000 {
             write!(f, "{}", self.0)
         } else {
-            write_hex(i64::from(self.0), f)
+            write_hex(u64::from(self.0), f)
         }
     }
 }
@@ -268,7 +330,7 @@ impl Display for Offset32 {
         if val < 10_000 {
             write!(f, "{}", val)
         } else {
-            write_hex(val, f)
+            write_hex(val as u64, f)
         }
     }
 }
@@ -683,6 +745,20 @@ mod tests {
         assert_eq!(Imm64(0x10000).to_string(), "0x0001_0000");
     }
 
+    #[test]
+    fn format_uimm64() {
+        assert_eq!(Uimm64(0).to_string(), "0");
+        assert_eq!(Uimm64(9999).to_string(), "9999");
+        assert_eq!(Uimm64(10000).to_string(), "0x2710");
+        assert_eq!(Uimm64(-9999i64 as u64).to_string(), "0xffff_ffff_ffff_d8f1");
+        assert_eq!(
+            Uimm64(-10000i64 as u64).to_string(),
+            "0xffff_ffff_ffff_d8f0"
+        );
+        assert_eq!(Uimm64(0xffff).to_string(), "0xffff");
+        assert_eq!(Uimm64(0x10000).to_string(), "0x0001_0000");
+    }
+
     // Verify that `text` can be parsed as a `T` into a value that displays as `want`.
     fn parse_ok(text: &str, want: &str)
     where
@@ -750,6 +826,46 @@ mod tests {
         parse_err::("0x0_0000_0000_0000_0000", "Too many hexadecimal digits");
     }
 
+    #[test]
+    fn parse_uimm64() {
+        parse_ok::("0", "0");
+        parse_ok::("1", "1");
+        parse_ok::("0x0", "0");
+        parse_ok::("0xf", "15");
+        parse_ok::("0xffffffff_fffffff7", "0xffff_ffff_ffff_fff7");
+
+        // Probe limits.
+        parse_ok::("0xffffffff_ffffffff", "0xffff_ffff_ffff_ffff");
+        parse_ok::("0x80000000_00000000", "0x8000_0000_0000_0000");
+        parse_ok::("18446744073709551615", "0xffff_ffff_ffff_ffff");
+        // Overflow both the `checked_add` and `checked_mul`.
+        parse_err::("18446744073709551616", "Too large decimal number");
+        parse_err::("184467440737095516100", "Too large decimal number");
+
+        // Underscores are allowed where digits go.
+        parse_ok::("0_0", "0");
+        parse_ok::("_10_", "10");
+        parse_ok::("0x97_88_bb", "0x0097_88bb");
+        parse_ok::("0x_97_", "151");
+
+        parse_err::("", "No digits in number");
+        parse_err::("_", "No digits in number");
+        parse_err::("0x", "No digits in number");
+        parse_err::("0x_", "No digits in number");
+        parse_err::("-", "Invalid character in decimal number");
+        parse_err::("-0x", "Invalid character in hexadecimal number");
+        parse_err::(" ", "Invalid character in decimal number");
+        parse_err::("0 ", "Invalid character in decimal number");
+        parse_err::(" 0", "Invalid character in decimal number");
+        parse_err::("--", "Invalid character in decimal number");
+        parse_err::("-0x-", "Invalid character in hexadecimal number");
+        parse_err::("-0", "Invalid character in decimal number");
+        parse_err::("-1", "Invalid character in decimal number");
+
+        // Hex count overflow.
+        parse_err::("0x0_0000_0000_0000_0000", "Too many hexadecimal digits");
+    }
+
     #[test]
     fn format_offset32() {
         assert_eq!(Offset32(0).to_string(), "");
diff --git a/lib/codegen/src/ir/table.rs b/lib/codegen/src/ir/table.rs
index 5db2402793..0b019d81cd 100644
--- a/lib/codegen/src/ir/table.rs
+++ b/lib/codegen/src/ir/table.rs
@@ -1,6 +1,6 @@
 //! Tables.
 
-use ir::immediates::Imm64;
+use ir::immediates::Uimm64;
 use ir::{GlobalValue, Type};
 use std::fmt;
 
@@ -12,13 +12,13 @@ pub struct TableData {
 
     /// Guaranteed minimum table size in elements. Table accesses before `min_size` don't need
     /// bounds checking.
-    pub min_size: Imm64,
+    pub min_size: Uimm64,
 
     /// Global value giving the current bound of the table, in elements.
     pub bound_gv: GlobalValue,
 
     /// The size of a table element, in bytes.
-    pub element_size: Imm64,
+    pub element_size: Uimm64,
 
     /// The index type for the table.
     pub index_type: Type,
diff --git a/lib/codegen/src/ir/trapcode.rs b/lib/codegen/src/ir/trapcode.rs
index 5bdb5a541d..fc16dddb92 100644
--- a/lib/codegen/src/ir/trapcode.rs
+++ b/lib/codegen/src/ir/trapcode.rs
@@ -17,7 +17,8 @@ pub enum TrapCode {
     /// A `heap_addr` instruction detected an out-of-bounds error.
     ///
     /// Note that not all out-of-bounds heap accesses are reported this way;
-    /// some are detected by a segmentation fault on the heap guard pages.
+    /// some are detected by a segmentation fault on the heap unmapped or
+    /// offset-guard pages.
     HeapOutOfBounds,
 
     /// A `table_addr` instruction detected an out-of-bounds error.
diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs
index 70e926920a..2edf2fae37 100644
--- a/lib/codegen/src/legalizer/heap.rs
+++ b/lib/codegen/src/legalizer/heap.rs
@@ -49,7 +49,7 @@ fn dynamic_addr(
     bound_gv: ir::GlobalValue,
     func: &mut ir::Function,
 ) {
-    let access_size = i64::from(access_size);
+    let access_size = u64::from(access_size);
     let offset_ty = func.dfg.value_type(offset);
     let addr_ty = func.dfg.value_type(func.dfg.first_result(inst));
     let min_size = func.heaps[heap].min_size.into();
@@ -67,13 +67,13 @@ fn dynamic_addr(
     } else if access_size <= min_size {
         // We know that bound >= min_size, so here we can compare `offset > bound - access_size`
         // without wrapping.
-        let adj_bound = pos.ins().iadd_imm(bound, -access_size);
+        let adj_bound = pos.ins().iadd_imm(bound, -(access_size as i64));
         oob = pos
             .ins()
             .icmp(IntCC::UnsignedGreaterThan, offset, adj_bound);
     } else {
         // We need an overflow check for the adjusted offset.
-        let access_size_val = pos.ins().iconst(offset_ty, access_size);
+        let access_size_val = pos.ins().iconst(offset_ty, access_size as i64);
         let (adj_offset, overflow) = pos.ins().iadd_cout(offset, access_size_val);
         pos.ins().trapnz(overflow, ir::TrapCode::HeapOutOfBounds);
         oob = pos
@@ -91,11 +91,11 @@ fn static_addr(
     heap: ir::Heap,
     offset: ir::Value,
     access_size: u32,
-    bound: i64,
+    bound: u64,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
 ) {
-    let access_size = i64::from(access_size);
+    let access_size = u64::from(access_size);
     let offset_ty = func.dfg.value_type(offset);
     let addr_ty = func.dfg.value_type(func.dfg.first_result(inst));
     let mut pos = FuncCursor::new(func).at_inst(inst);
@@ -117,7 +117,7 @@ fn static_addr(
     }
 
     // Check `offset > limit` which is now known non-negative.
-    let limit = bound - access_size;
+    let limit = bound - u64::from(access_size);
 
     // We may be able to omit the check entirely for 32-bit offsets if the heap bound is 4 GB or
     // more.
@@ -126,10 +126,10 @@ fn static_addr(
             // Prefer testing `offset >= limit - 1` when limit is odd because an even number is
             // likely to be a convenient constant on ARM and other RISC architectures.
             pos.ins()
-                .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, offset, limit - 1)
+                .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, offset, limit as i64 - 1)
         } else {
             pos.ins()
-                .icmp_imm(IntCC::UnsignedGreaterThan, offset, limit)
+                .icmp_imm(IntCC::UnsignedGreaterThan, offset, limit as i64)
         };
         pos.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds);
     }
diff --git a/lib/codegen/src/legalizer/table.rs b/lib/codegen/src/legalizer/table.rs
index d7633780f2..f6005f1518 100644
--- a/lib/codegen/src/legalizer/table.rs
+++ b/lib/codegen/src/legalizer/table.rs
@@ -92,17 +92,15 @@ fn compute_addr(
 
     let element_size = pos.func.tables[table].element_size;
     let mut offset;
-    let element_size_i64: i64 = element_size.into();
-    debug_assert!(element_size_i64 >= 0);
-    let element_size_u64 = element_size_i64 as u64;
-    if element_size_u64 == 1 {
+    let element_size: u64 = element_size.into();
+    if element_size == 1 {
         offset = index;
-    } else if element_size_u64.is_power_of_two() {
+    } else if element_size.is_power_of_two() {
         offset = pos
             .ins()
-            .ishl_imm(index, i64::from(element_size_u64.trailing_zeros()));
+            .ishl_imm(index, i64::from(element_size.trailing_zeros()));
     } else {
-        offset = pos.ins().imul_imm(index, element_size);
+        offset = pos.ins().imul_imm(index, element_size as i64);
     }
 
     if element_offset == Offset32::new(0) {
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index a53f734792..6cae92c0b0 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -3,7 +3,7 @@
 use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::entities::AnyEntity;
-use cranelift_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32};
+use cranelift_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32, Uimm64};
 use cranelift_codegen::ir::instructions::{InstructionData, InstructionFormat, VariableArgs};
 use cranelift_codegen::ir::types::INVALID;
 use cranelift_codegen::ir::{
@@ -188,10 +188,10 @@ impl<'a> Context<'a> {
         while self.function.heaps.next_key().index() <= heap.index() {
             self.function.create_heap(HeapData {
                 base: GlobalValue::reserved_value(),
-                min_size: Imm64::new(0),
-                guard_size: Imm64::new(0),
+                min_size: Uimm64::new(0),
+                offset_guard_size: Uimm64::new(0),
                 style: HeapStyle::Static {
-                    bound: Imm64::new(0),
+                    bound: Uimm64::new(0),
                 },
                 index_type: INVALID,
             });
@@ -214,9 +214,9 @@ impl<'a> Context<'a> {
         while self.function.tables.next_key().index() <= table.index() {
             self.function.create_table(TableData {
                 base_gv: GlobalValue::reserved_value(),
-                min_size: Imm64::new(0),
+                min_size: Uimm64::new(0),
                 bound_gv: GlobalValue::reserved_value(),
-                element_size: Imm64::new(0),
+                element_size: Uimm64::new(0),
                 index_type: INVALID,
             });
         }
@@ -544,6 +544,19 @@ impl<'a> Parser<'a> {
         }
     }
 
+    // Match and consume a Uimm64 immediate.
+    fn match_uimm64(&mut self, err_msg: &str) -> ParseResult {
+        if let Some(Token::Integer(text)) = self.token() {
+            self.consume();
+            // Lexer just gives us raw text that looks like an integer.
+            // Parse it as an Uimm64 to check for overflow and other issues.
+            text.parse()
+                .map_err(|_| self.error("expected u64 decimal immediate"))
+        } else {
+            err!(self.loc, err_msg)
+        }
+    }
+
     // Match and consume a Uimm32 immediate.
     fn match_uimm32(&mut self, err_msg: &str) -> ParseResult {
         if let Some(Token::Integer(text)) = self.token() {
@@ -1279,7 +1292,7 @@ impl<'a> Parser<'a> {
     // heap-base ::= GlobalValue(base)
     // heap-attr ::= "min" Imm64(bytes)
     //             | "bound" Imm64(bytes)
-    //             | "guard" Imm64(bytes)
+    //             | "offset_guard" Imm64(bytes)
     //             | "index_type" type
     //
     fn parse_heap_decl(&mut self) -> ParseResult<(Heap, HeapData)> {
@@ -1302,7 +1315,7 @@ impl<'a> Parser<'a> {
         let mut data = HeapData {
             base,
             min_size: 0.into(),
-            guard_size: 0.into(),
+            offset_guard_size: 0.into(),
             style: HeapStyle::Static { bound: 0.into() },
             index_type: ir::types::I32,
         };
@@ -1311,7 +1324,7 @@ impl<'a> Parser<'a> {
         while self.optional(Token::Comma) {
             match self.match_any_identifier("expected heap attribute name")? {
                 "min" => {
-                    data.min_size = self.match_imm64("expected integer min size")?;
+                    data.min_size = self.match_uimm64("expected integer min size")?;
                 }
                 "bound" => {
                     data.style = match style_name {
@@ -1319,13 +1332,14 @@ impl<'a> Parser<'a> {
                             bound_gv: self.match_gv("expected gv bound")?,
                         },
                         "static" => HeapStyle::Static {
-                            bound: self.match_imm64("expected integer bound")?,
+                            bound: self.match_uimm64("expected integer bound")?,
                         },
                         t => return err!(self.loc, "unknown heap style '{}'", t),
                     };
                 }
-                "guard" => {
-                    data.guard_size = self.match_imm64("expected integer guard size")?;
+                "offset_guard" => {
+                    data.offset_guard_size =
+                        self.match_uimm64("expected integer offset-guard size")?;
                 }
                 "index_type" => {
                     data.index_type = self.match_type("expected index type")?;
@@ -1381,7 +1395,7 @@ impl<'a> Parser<'a> {
         while self.optional(Token::Comma) {
             match self.match_any_identifier("expected table attribute name")? {
                 "min" => {
-                    data.min_size = self.match_imm64("expected integer min size")?;
+                    data.min_size = self.match_uimm64("expected integer min size")?;
                 }
                 "bound" => {
                     data.bound_gv = match style_name {
@@ -1390,7 +1404,7 @@ impl<'a> Parser<'a> {
                     };
                 }
                 "element_size" => {
-                    data.element_size = self.match_imm64("expected integer element size")?;
+                    data.element_size = self.match_uimm64("expected integer element size")?;
                 }
                 "index_type" => {
                     data.index_type = self.match_type("expected index type")?;
@@ -2780,8 +2794,8 @@ mod tests {
     fn duplicate_heap() {
         let ParseError { location, message } = Parser::new(
             "function %ebbs() system_v {
-                heap0 = static gv0, min 0x1000, bound 0x10_0000, guard 0x1000
-                heap0 = static gv0, min 0x1000, bound 0x10_0000, guard 0x1000",
+                heap0 = static gv0, min 0x1000, bound 0x10_0000, offset_guard 0x1000
+                heap0 = static gv0, min 0x1000, bound 0x10_0000, offset_guard 0x1000",
         )
         .parse_function(None)
         .unwrap_err();
diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs
index fab42da984..b3aaa3e7ca 100644
--- a/lib/umbrella/src/lib.rs
+++ b/lib/umbrella/src/lib.rs
@@ -36,7 +36,7 @@ pub mod prelude {
     pub use codegen;
     pub use codegen::entity::EntityRef;
     pub use codegen::ir::condcodes::{FloatCC, IntCC};
-    pub use codegen::ir::immediates::{Ieee32, Ieee64, Imm64};
+    pub use codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Uimm64};
     pub use codegen::ir::types;
     pub use codegen::ir::{
         AbiParam, Ebb, ExtFuncData, ExternalName, GlobalValueData, InstBuilder, JumpTableData,
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index 8a270e687e..8866eeef36 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -992,20 +992,20 @@ fn get_heap_addr(
 ) -> (ir::Value, i32) {
     use std::cmp::min;
 
-    let guard_size: i64 = builder.func.heaps[heap].guard_size.into();
-    debug_assert!(guard_size > 0, "Heap guard pages currently required");
+    let mut adjusted_offset = u64::from(offset);
+    let offset_guard_size: u64 = builder.func.heaps[heap].offset_guard_size.into();
 
     // Generate `heap_addr` instructions that are friendly to CSE by checking offsets that are
-    // multiples of the guard size. Add one to make sure that we check the pointer itself is in
-    // bounds.
-    //
-    // For accesses on the outer skirts of the guard pages, we expect that we get a trap
-    // even if the access goes beyond the guard pages. This is because the first byte pointed to is
-    // inside the guard pages.
-    let check_size = min(
-        i64::from(u32::MAX),
-        1 + (i64::from(offset) / guard_size) * guard_size,
-    ) as u32;
+    // multiples of the offset-guard size. Add one to make sure that we check the pointer itself
+    // is in bounds.
+    if offset_guard_size != 0 {
+        adjusted_offset = adjusted_offset / offset_guard_size * offset_guard_size;
+    }
+
+    // For accesses on the outer skirts of the offset-guard pages, we expect that we get a trap
+    // even if the access goes beyond the offset-guard pages. This is because the first byte
+    // pointed to is inside the offset-guard pages.
+    let check_size = min(u64::from(u32::MAX), 1 + adjusted_offset) as u32;
     let base = builder.ins().heap_addr(addr_ty, heap, addr32, check_size);
 
     // Native load/store instructions take a signed `Offset32` immediate, so adjust the base
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 926c87cf9e..4ff36e6f58 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -2,7 +2,7 @@
 //! wasm translation.
 
 use cranelift_codegen::cursor::FuncCursor;
-use cranelift_codegen::ir::immediates::{Imm64, Offset32};
+use cranelift_codegen::ir::immediates::{Offset32, Uimm64};
 use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::isa::TargetFrontendConfig;
@@ -195,7 +195,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         func.create_heap(ir::HeapData {
             base: gv,
             min_size: 0.into(),
-            guard_size: 0x8000_0000.into(),
+            offset_guard_size: 0x8000_0000.into(),
             style: ir::HeapStyle::Static {
                 bound: 0x1_0000_0000.into(),
             },
@@ -221,9 +221,9 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
 
         func.create_table(ir::TableData {
             base_gv,
-            min_size: Imm64::new(0),
+            min_size: Uimm64::new(0),
             bound_gv,
-            element_size: Imm64::new(i64::from(self.pointer_bytes()) * 2),
+            element_size: Uimm64::from(u64::from(self.pointer_bytes()) * 2),
             index_type: I32,
         })
     }

From 954573440d83af0ae58c09cd4143371b2b060a78 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 30 Nov 2018 16:11:59 -0800
Subject: [PATCH 2245/3084] Remove ModuleEnvironment::get_func_name, which is
 unused.

---
 lib/wasm/src/environ/dummy.rs | 4 ----
 lib/wasm/src/environ/spec.rs  | 3 ---
 2 files changed, 7 deletions(-)

diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 4ff36e6f58..22edbc998e 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -342,10 +342,6 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         self.info.config
     }
 
-    fn get_func_name(&self, func_index: FuncIndex) -> ir::ExternalName {
-        get_func_name(func_index)
-    }
-
     fn declare_signature(&mut self, sig: &ir::Signature) {
         self.info.signatures.push(sig.clone());
     }
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index c7910f65f1..ab2bc62a9f 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -235,9 +235,6 @@ pub trait ModuleEnvironment<'data> {
     /// Get the information needed to produce Cranelift IR for the current target.
     fn target_config(&self) -> TargetFrontendConfig;
 
-    /// Return the name for the given function index.
-    fn get_func_name(&self, func_index: FuncIndex) -> ir::ExternalName;
-
     /// Declares a function signature to the environment.
     fn declare_signature(&mut self, sig: &ir::Signature);
 

From 91477d21c8f4536231504ac7b9796b547585e361 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 30 Nov 2018 16:16:35 -0800
Subject: [PATCH 2246/3084] Add an `as_u32()` member to `entity_impl` types.

This allows us to avoid a lot of casting indices back to u32.
---
 lib/codegen/src/regalloc/virtregs.rs        | 2 +-
 lib/entity/src/lib.rs                       | 7 +++++++
 lib/module/src/module.rs                    | 8 ++++----
 lib/simplejit/examples/simplejit-minimal.rs | 4 ++--
 lib/simplejit/tests/basic.rs                | 2 +-
 lib/wasm/src/environ/dummy.rs               | 2 +-
 6 files changed, 16 insertions(+), 9 deletions(-)

diff --git a/lib/codegen/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs
index 13ef4a5381..ac4856261f 100644
--- a/lib/codegen/src/regalloc/virtregs.rs
+++ b/lib/codegen/src/regalloc/virtregs.rs
@@ -266,7 +266,7 @@ impl UFEntry {
 
     /// Encode a link entry.
     fn encode_link(v: Value) -> i32 {
-        !(v.index() as i32)
+        !(v.as_u32() as i32)
     }
 }
 
diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs
index 10386e9902..191a9c09e7 100644
--- a/lib/entity/src/lib.rs
+++ b/lib/entity/src/lib.rs
@@ -98,6 +98,13 @@ macro_rules! entity_impl {
                 $entity($crate::__core::u32::MAX)
             }
         }
+
+        impl $entity {
+            /// Return the underlying index value as a `u32`.
+            pub fn as_u32(self) -> u32 {
+                self.0
+            }
+        }
     };
 
     // Include basic `Display` impl using the given display prefix.
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 372fddd7ae..18f6167a2f 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -473,7 +473,7 @@ where
         let signature = in_func.import_signature(decl.signature.clone());
         let colocated = decl.linkage.is_final();
         in_func.import_function(ir::ExtFuncData {
-            name: ir::ExternalName::user(0, func.index() as u32),
+            name: ir::ExternalName::user(0, func.as_u32()),
             signature,
             colocated,
         })
@@ -486,7 +486,7 @@ where
         let decl = &self.contents.data_objects[data].decl;
         let colocated = decl.linkage.is_final();
         func.create_global_value(ir::GlobalValueData::Symbol {
-            name: ir::ExternalName::user(1, data.index() as u32),
+            name: ir::ExternalName::user(1, data.as_u32()),
             offset: ir::immediates::Imm64::new(0),
             colocated,
         })
@@ -494,12 +494,12 @@ where
 
     /// TODO: Same as above.
     pub fn declare_func_in_data(&self, func: FuncId, ctx: &mut DataContext) -> ir::FuncRef {
-        ctx.import_function(ir::ExternalName::user(0, func.index() as u32))
+        ctx.import_function(ir::ExternalName::user(0, func.as_u32()))
     }
 
     /// TODO: Same as above.
     pub fn declare_data_in_data(&self, data: DataId, ctx: &mut DataContext) -> ir::GlobalValue {
-        ctx.import_global_value(ir::ExternalName::user(1, data.index() as u32))
+        ctx.import_global_value(ir::ExternalName::user(1, data.as_u32()))
     }
 
     /// Define a function, producing the function body from the given `Context`.
diff --git a/lib/simplejit/examples/simplejit-minimal.rs b/lib/simplejit/examples/simplejit-minimal.rs
index 0a736a8746..4e2d156bc8 100644
--- a/lib/simplejit/examples/simplejit-minimal.rs
+++ b/lib/simplejit/examples/simplejit-minimal.rs
@@ -27,7 +27,7 @@ fn main() {
         .unwrap();
 
     ctx.func.signature = sig_a;
-    ctx.func.name = ExternalName::user(0, func_a.index() as u32);
+    ctx.func.name = ExternalName::user(0, func_a.as_u32());
     {
         let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
         let ebb = bcx.create_ebb();
@@ -45,7 +45,7 @@ fn main() {
     module.clear_context(&mut ctx);
 
     ctx.func.signature = sig_b;
-    ctx.func.name = ExternalName::user(0, func_b.index() as u32);
+    ctx.func.name = ExternalName::user(0, func_b.as_u32());
     {
         let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
         let ebb = bcx.create_ebb();
diff --git a/lib/simplejit/tests/basic.rs b/lib/simplejit/tests/basic.rs
index 1614686911..9128ef6068 100644
--- a/lib/simplejit/tests/basic.rs
+++ b/lib/simplejit/tests/basic.rs
@@ -42,7 +42,7 @@ fn define_simple_function(module: &mut Module) -> FuncId {
         .unwrap();
 
     let mut ctx = Context::new();
-    ctx.func = Function::with_name_signature(ExternalName::user(0, func_id.index() as u32), sig);
+    ctx.func = Function::with_name_signature(ExternalName::user(0, func_id.as_u32()), sig);
     let mut func_ctx = FunctionBuilderContext::new();
     {
         let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 22edbc998e..70c13fd0e6 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -18,7 +18,7 @@ use translation_utils::{
 
 /// Compute a `ir::ExternalName` for a given wasm function index.
 fn get_func_name(func_index: FuncIndex) -> ir::ExternalName {
-    ir::ExternalName::user(0, func_index.index() as u32)
+    ir::ExternalName::user(0, func_index.as_u32())
 }
 
 /// A collection of names under which a given entity is exported.

From e623da51c6865c820db54c19217ea4e29095e112 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 30 Nov 2018 16:27:00 -0800
Subject: [PATCH 2247/3084] Suppress dead-code warnings on functions expanded
 from `entity_impl!`.

---
 lib/entity/src/lib.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs
index 191a9c09e7..7d63eaee16 100644
--- a/lib/entity/src/lib.rs
+++ b/lib/entity/src/lib.rs
@@ -101,6 +101,7 @@ macro_rules! entity_impl {
 
         impl $entity {
             /// Return the underlying index value as a `u32`.
+            #[allow(dead_code)]
             pub fn as_u32(self) -> u32 {
                 self.0
             }

From c2872317204cb93d84d258fa07c4b7ad51b80d85 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 30 Nov 2018 16:31:00 -0800
Subject: [PATCH 2248/3084] Implement FromIterator for PrimaryMap.

---
 lib/entity/src/primary.rs | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs
index 9b87098239..ae1097f6aa 100644
--- a/lib/entity/src/primary.rs
+++ b/lib/entity/src/primary.rs
@@ -1,4 +1,5 @@
 //! Densely numbered entity references as mapping keys.
+use std::iter::FromIterator;
 use std::marker::PhantomData;
 use std::ops::{Index, IndexMut};
 use std::slice;
@@ -170,6 +171,21 @@ where
     }
 }
 
+impl FromIterator for PrimaryMap
+where
+    K: EntityRef,
+{
+    fn from_iter(iter: T) -> Self
+    where
+        T: IntoIterator,
+    {
+        Self {
+            elems: Vec::from_iter(iter),
+            unused: PhantomData,
+        }
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -346,4 +362,15 @@ mod tests {
             }
         }
     }
+
+    fn from_iter() {
+        let mut m: PrimaryMap = PrimaryMap::new();
+        m.push(12);
+        m.push(33);
+        let n = m.values().collect::>();
+        assert!(m.len() == n.len());
+        for (me, ne) in m.values().zip(n.values()) {
+            assert!(*me == **ne);
+        }
+    }
 }

From f0dbba0ec8b188e0eab701c1cccd616ab4e076c4 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 30 Nov 2018 16:37:00 -0800
Subject: [PATCH 2249/3084] Change TableElementType::Func() to
 TableElementType::Func.

---
 lib/wasm/src/sections_translator.rs | 4 ++--
 lib/wasm/src/translation_utils.rs   | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 511f7d07ee..15a1dfc0da 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -106,7 +106,7 @@ pub fn parse_import_section<'data>(
                     Table {
                         ty: match type_to_type(tab.element_type) {
                             Ok(t) => TableElementType::Val(t),
-                            Err(()) => TableElementType::Func(),
+                            Err(()) => TableElementType::Func,
                         },
                         minimum: tab.limits.initial,
                         maximum: tab.limits.maximum,
@@ -142,7 +142,7 @@ pub fn parse_table_section(
         environ.declare_table(Table {
             ty: match type_to_type(table.element_type) {
                 Ok(t) => TableElementType::Val(t),
-                Err(()) => TableElementType::Func(),
+                Err(()) => TableElementType::Func,
             },
             minimum: table.limits.initial,
             maximum: table.limits.maximum,
diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index 2ebf4c3879..9b81c28a2d 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -91,7 +91,7 @@ pub struct Table {
 #[derive(Debug, Clone, Copy)]
 pub enum TableElementType {
     Val(ir::Type),
-    Func(),
+    Func,
 }
 
 /// WebAssembly linear memory.

From 6e17bebf4419a648234346cdc3ed871dd3a7ec41 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 30 Nov 2018 16:41:54 -0800
Subject: [PATCH 2250/3084] Publish `TableElementType` for library users.

---
 lib/wasm/src/lib.rs               | 3 ++-
 lib/wasm/src/translation_utils.rs | 2 ++
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index 6c23f97640..354a62f9e8 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -64,7 +64,8 @@ pub use func_translator::FuncTranslator;
 pub use module_translator::translate_module;
 pub use translation_utils::{
     DefinedFuncIndex, DefinedGlobalIndex, DefinedMemoryIndex, DefinedTableIndex, FuncIndex, Global,
-    GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
+    GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex, Table, TableElementType,
+    TableIndex,
 };
 
 #[cfg(not(feature = "std"))]
diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index 9b81c28a2d..02e03cb386 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -90,7 +90,9 @@ pub struct Table {
 /// WebAssembly table element. Can be a function or a scalar type.
 #[derive(Debug, Clone, Copy)]
 pub enum TableElementType {
+    /// A scalar type.
     Val(ir::Type),
+    /// A function.
     Func,
 }
 

From 42e35c4c578b667933f4ff15853ee45285986cfd Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 30 Nov 2018 16:59:35 -0800
Subject: [PATCH 2251/3084] Implement `PrimaryMap::with_capacity`.

---
 lib/entity/src/primary.rs | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs
index ae1097f6aa..2a4c37154a 100644
--- a/lib/entity/src/primary.rs
+++ b/lib/entity/src/primary.rs
@@ -41,6 +41,14 @@ where
         }
     }
 
+    /// Create a new empty map with the given capacity.
+    pub fn with_capacity(capacity: usize) -> Self {
+        Self {
+            elems: Vec::with_capacity(capacity),
+            unused: PhantomData,
+        }
+    }
+
     /// Check if `k` is a valid key in the map.
     pub fn is_valid(&self, k: K) -> bool {
         k.index() < self.elems.len()

From 76583192a88146640199b21623da3893e47c6877 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 6 Dec 2018 09:36:48 -0500
Subject: [PATCH 2252/3084] Change ZeroOneOrMore::Zero() to
 TableElementType::Zero.

---
 lib/frontend/src/ssa.rs | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs
index a78ed0b890..15ba1c98ca 100644
--- a/lib/frontend/src/ssa.rs
+++ b/lib/frontend/src/ssa.rs
@@ -192,9 +192,9 @@ impl SSABuilder {
 /// Small enum used for clarity in some functions.
 #[derive(Debug)]
 enum ZeroOneOrMore {
-    Zero(),
+    Zero,
     One(T),
-    More(),
+    More,
 }
 
 #[derive(Debug)]
@@ -526,7 +526,7 @@ impl SSABuilder {
         temp_arg_var: Variable,
         dest_ebb: Ebb,
     ) {
-        let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero();
+        let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero;
 
         // Iterate over the predecessors.
         for _ in 0..self.predecessors(dest_ebb).len() {
@@ -534,21 +534,21 @@ impl SSABuilder {
             // to var and we put it as an argument of the branch instruction.
             let pred_val = self.results.pop().unwrap();
             match pred_values {
-                ZeroOneOrMore::Zero() => {
+                ZeroOneOrMore::Zero => {
                     if pred_val != temp_arg_val {
                         pred_values = ZeroOneOrMore::One(pred_val);
                     }
                 }
                 ZeroOneOrMore::One(old_val) => {
                     if pred_val != temp_arg_val && pred_val != old_val {
-                        pred_values = ZeroOneOrMore::More();
+                        pred_values = ZeroOneOrMore::More;
                     }
                 }
-                ZeroOneOrMore::More() => {}
+                ZeroOneOrMore::More => {}
             }
         }
         let result_val = match pred_values {
-            ZeroOneOrMore::Zero() => {
+            ZeroOneOrMore::Zero => {
                 // The variable is used but never defined before. This is an irregularity in the
                 // code, but rather than throwing an error we silently initialize the variable to
                 // 0. This will have no effect since this situation happens in unreachable code.
@@ -583,7 +583,7 @@ impl SSABuilder {
                 func.dfg.change_to_alias(temp_arg_val, resolved);
                 resolved
             }
-            ZeroOneOrMore::More() => {
+            ZeroOneOrMore::More => {
                 // There is disagreement in the predecessors on which value to use so we have
                 // to keep the ebb argument. To avoid borrowing `self` for the whole loop,
                 // temporarily detach the predecessors list and replace it with an empty list.

From 38d1d58e06ddfb4d18b7aa00752c044ba29e1679 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 6 Dec 2018 09:37:34 -0500
Subject: [PATCH 2253/3084] Change GlobalInit::Import() to GlobalInit::Import.

---
 lib/wasm/src/sections_translator.rs | 6 +++---
 lib/wasm/src/translation_utils.rs   | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 15a1dfc0da..ea3c6c6525 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -95,7 +95,7 @@ pub fn parse_import_section<'data>(
                     Global {
                         ty: type_to_type(ty.content_type).unwrap(),
                         mutability: ty.mutable,
-                        initializer: GlobalInit::Import(),
+                        initializer: GlobalInit::Import,
                     },
                     module_name,
                     field_name,
@@ -253,7 +253,7 @@ pub fn parse_element_section<'data>(
                 .initializer
             {
                 GlobalInit::I32Const(value) => (None, value as u32 as usize),
-                GlobalInit::Import() => (Some(GlobalIndex::new(global_index as usize)), 0),
+                GlobalInit::Import => (Some(GlobalIndex::new(global_index as usize)), 0),
                 _ => panic!("should not happen"),
             },
             ref s => panic!("unsupported init expr in element section: {:?}", s),
@@ -301,7 +301,7 @@ pub fn parse_data_section<'data>(
                 .initializer
             {
                 GlobalInit::I32Const(value) => (None, value as u32 as usize),
-                GlobalInit::Import() => (Some(GlobalIndex::new(global_index as usize)), 0),
+                GlobalInit::Import => (Some(GlobalIndex::new(global_index as usize)), 0),
                 _ => panic!("should not happen"),
             },
             ref s => panic!("unsupported init expr in data section: {:?}", s),
diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index 02e03cb386..899523f17a 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -73,7 +73,7 @@ pub enum GlobalInit {
     /// A `get_global` of another global.
     GlobalRef(GlobalIndex),
     ///< The global is imported from, and thus initialized by, a different module.
-    Import(),
+    Import,
 }
 
 /// WebAssembly table.

From 566c160c379609a6937d6720e50c9919208f0e9e Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 6 Dec 2018 10:50:12 -0500
Subject: [PATCH 2254/3084] Add a function to return the pointer type for a
 given triple.

Also use a slightly tidier naming convention for such functions.
---
 lib/codegen/src/ir/types.rs      | 11 +++++++++++
 lib/codegen/src/isa/call_conv.rs |  2 +-
 lib/codegen/src/isa/mod.rs       |  2 +-
 3 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/lib/codegen/src/ir/types.rs b/lib/codegen/src/ir/types.rs
index c7f6d12f23..ceffe04e9c 100644
--- a/lib/codegen/src/ir/types.rs
+++ b/lib/codegen/src/ir/types.rs
@@ -2,6 +2,7 @@
 
 use std::default::Default;
 use std::fmt::{self, Debug, Display, Formatter};
+use target_lexicon::{PointerWidth, Triple};
 
 /// The type of an SSA value.
 ///
@@ -271,6 +272,16 @@ impl Type {
     pub fn wider_or_equal(self, other: Self) -> bool {
         self.lane_count() == other.lane_count() && self.lane_bits() >= other.lane_bits()
     }
+
+    /// Return the pointer type for the given target triple.
+    pub fn triple_pointer_type(triple: &Triple) -> Self {
+        match triple.pointer_width() {
+            Ok(PointerWidth::U16) => I16,
+            Ok(PointerWidth::U32) => I32,
+            Ok(PointerWidth::U64) => I64,
+            Err(()) => panic!("unable to determine architecture pointer width"),
+        }
+    }
 }
 
 impl Display for Type {
diff --git a/lib/codegen/src/isa/call_conv.rs b/lib/codegen/src/isa/call_conv.rs
index 0fe242b958..9caff3ebaa 100644
--- a/lib/codegen/src/isa/call_conv.rs
+++ b/lib/codegen/src/isa/call_conv.rs
@@ -21,7 +21,7 @@ pub enum CallConv {
 
 impl CallConv {
     /// Return the default calling convention for the given target triple.
-    pub fn default_for_triple(triple: &Triple) -> Self {
+    pub fn triple_default(triple: &Triple) -> Self {
         match triple.default_calling_convention() {
             // Default to System V for unknown targets because most everything
             // uses System V.
diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs
index b4bf97d334..d8eba6ccc3 100644
--- a/lib/codegen/src/isa/mod.rs
+++ b/lib/codegen/src/isa/mod.rs
@@ -208,7 +208,7 @@ pub trait TargetIsa: fmt::Display + Sync {
 
     /// Get the default calling convention of this target.
     fn default_call_conv(&self) -> CallConv {
-        CallConv::default_for_triple(self.triple())
+        CallConv::triple_default(self.triple())
     }
 
     /// Get the pointer type of this ISA.

From 30a089031919caa8686d3ecefda85e96ec7e98da Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 6 Dec 2018 16:00:25 -0500
Subject: [PATCH 2255/3084] Rename GlobalInit::GlobalRef to
 GlobalInit::GetGlobal.

"Ref" is an increasingly overloaded term.
---
 lib/wasm/src/sections_translator.rs | 2 +-
 lib/wasm/src/translation_utils.rs   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index ea3c6c6525..10ec04880b 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -187,7 +187,7 @@ pub fn parse_global_section(
             Operator::F32Const { value } => GlobalInit::F32Const(value.bits()),
             Operator::F64Const { value } => GlobalInit::F64Const(value.bits()),
             Operator::GetGlobal { global_index } => {
-                GlobalInit::GlobalRef(GlobalIndex::new(global_index as usize))
+                GlobalInit::GetGlobal(GlobalIndex::new(global_index as usize))
             }
             ref s => panic!("unsupported init expr in global section: {:?}", s),
         };
diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index 899523f17a..9dc8a2ffd6 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -71,7 +71,7 @@ pub enum GlobalInit {
     /// An `f64.const`.
     F64Const(u64),
     /// A `get_global` of another global.
-    GlobalRef(GlobalIndex),
+    GetGlobal(GlobalIndex),
     ///< The global is imported from, and thus initialized by, a different module.
     Import,
 }

From 605f34257bb7020979964933f5a27fe4daec5235 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 6 Dec 2018 16:24:54 -0500
Subject: [PATCH 2256/3084] Fix a missing `#[test]`.

---
 lib/entity/src/primary.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs
index 2a4c37154a..6fe70523fe 100644
--- a/lib/entity/src/primary.rs
+++ b/lib/entity/src/primary.rs
@@ -371,6 +371,7 @@ mod tests {
         }
     }
 
+    #[test]
     fn from_iter() {
         let mut m: PrimaryMap = PrimaryMap::new();
         m.push(12);

From 7b51195f496fbea4be29a3d7e306aa5e23440be1 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 6 Dec 2018 17:54:10 -0500
Subject: [PATCH 2257/3084] Add an offset to cranelift-wasm's `GlobalVariable`.

---
 lib/wasm/Cargo.toml             |  1 +
 lib/wasm/src/code_translator.rs |  9 +++++----
 lib/wasm/src/environ/dummy.rs   | 11 ++++-------
 lib/wasm/src/environ/spec.rs    |  3 +++
 lib/wasm/src/lib.rs             |  1 +
 5 files changed, 14 insertions(+), 11 deletions(-)

diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 6a768c87ba..a88550d1e1 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -18,6 +18,7 @@ hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 log = { version = "0.4.4", default-features = false }
+cast = { version = "0.2.2" }
 
 [dev-dependencies]
 wabt = "0.7.0"
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index 8866eeef36..f62ac89f77 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -75,12 +75,12 @@ pub fn translate_operator(
         Operator::GetGlobal { global_index } => {
             let val = match state.get_global(builder.func, global_index, environ) {
                 GlobalVariable::Const(val) => val,
-                GlobalVariable::Memory { gv, ty } => {
+                GlobalVariable::Memory { gv, offset, ty } => {
                     let addr = builder.ins().global_value(environ.pointer_type(), gv);
                     let mut flags = ir::MemFlags::new();
                     flags.set_notrap();
                     flags.set_aligned();
-                    builder.ins().load(ty, flags, addr, 0)
+                    builder.ins().load(ty, flags, addr, offset)
                 }
             };
             state.push1(val);
@@ -88,13 +88,14 @@ pub fn translate_operator(
         Operator::SetGlobal { global_index } => {
             match state.get_global(builder.func, global_index, environ) {
                 GlobalVariable::Const(_) => panic!("global #{} is a constant", global_index),
-                GlobalVariable::Memory { gv, .. } => {
+                GlobalVariable::Memory { gv, offset, ty } => {
                     let addr = builder.ins().global_value(environ.pointer_type(), gv);
                     let mut flags = ir::MemFlags::new();
                     flags.set_notrap();
                     flags.set_aligned();
                     let val = state.pop1();
-                    builder.ins().store(flags, val, addr, 0);
+                    debug_assert_eq!(ty, builder.func.dfg.value_type(val));
+                    builder.ins().store(flags, val, addr, offset);
                 }
             }
         }
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 70c13fd0e6..0a31b17b6a 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -1,6 +1,7 @@
 //! "Dummy" implementations of `ModuleEnvironment` and `FuncEnvironment` for testing
 //! wasm translation.
 
+use cast;
 use cranelift_codegen::cursor::FuncCursor;
 use cranelift_codegen::ir::immediates::{Offset32, Uimm64};
 use cranelift_codegen::ir::types::*;
@@ -169,15 +170,11 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
 
     fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalVariable {
         // Just create a dummy `vmctx` global.
-        let offset = ((index.index() * 8) as i64 + 8).into();
+        let offset = cast::i32((index.index() * 8) + 8).unwrap().into();
         let vmctx = func.create_global_value(ir::GlobalValueData::VMContext {});
-        let iadd = func.create_global_value(ir::GlobalValueData::IAddImm {
-            base: vmctx,
-            offset,
-            global_type: self.pointer_type(),
-        });
         GlobalVariable::Memory {
-            gv: iadd,
+            gv: vmctx,
+            offset: offset,
             ty: self.mod_info.globals[index].entity.ty,
         }
     }
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index ab2bc62a9f..7aeeb43c92 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -1,6 +1,7 @@
 //! All the runtime support necessary for the wasm to cranelift translation is formalized by the
 //! traits `FunctionEnvironment` and `ModuleEnvironment`.
 use cranelift_codegen::cursor::FuncCursor;
+use cranelift_codegen::ir::immediates::Offset32;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::isa::TargetFrontendConfig;
 use std::convert::From;
@@ -20,6 +21,8 @@ pub enum GlobalVariable {
     Memory {
         /// The address of the global variable storage.
         gv: ir::GlobalValue,
+        /// An offset to add to the address.
+        offset: Offset32,
         /// The global variable's type.
         ty: ir::Type,
     },
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index 354a62f9e8..acd63bedb4 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -36,6 +36,7 @@
 extern crate cranelift_codegen;
 #[macro_use]
 extern crate cranelift_entity;
+extern crate cast;
 extern crate cranelift_frontend;
 #[cfg(test)]
 extern crate target_lexicon;

From 95a6a25db388eb96357b383291bff84a79ab9456 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 6 Dec 2018 17:55:24 -0500
Subject: [PATCH 2258/3084] Update to pretty_env_logger 0.3.0.

---
 cranelift/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index a20993880c..709db5bf2d 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -34,7 +34,7 @@ term = "0.5.1"
 capstone = { version = "0.5.0", optional = true }
 wabt = { version = "0.7.0", optional = true }
 target-lexicon = "0.2.0"
-pretty_env_logger = "0.2.4"
+pretty_env_logger = "0.3.0"
 file-per-thread-logger = "0.1.1"
 
 [features]

From 5b03581442febc7a2aa57a7227e0051915706c7c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 6 Dec 2018 17:59:58 -0500
Subject: [PATCH 2259/3084] Tidy up `use` declaration syntax.

---
 lib/entity/src/map.rs           | 4 +++-
 lib/entity/src/primary.rs       | 4 +++-
 lib/entity/src/set.rs           | 3 ++-
 lib/entity/src/sparse.rs        | 3 ++-
 lib/filetests/src/concurrent.rs | 3 ++-
 lib/filetests/src/runner.rs     | 3 ++-
 lib/module/src/data_context.rs  | 2 +-
 7 files changed, 15 insertions(+), 7 deletions(-)

diff --git a/lib/entity/src/map.rs b/lib/entity/src/map.rs
index df8ac115a2..dbec7b2f11 100644
--- a/lib/entity/src/map.rs
+++ b/lib/entity/src/map.rs
@@ -1,10 +1,12 @@
 //! Densely numbered entity references as mapping keys.
 
+use iter::{Iter, IterMut};
+use keys::Keys;
 use std::marker::PhantomData;
 use std::ops::{Index, IndexMut};
 use std::slice;
 use std::vec::Vec;
-use {EntityRef, Iter, IterMut, Keys};
+use EntityRef;
 
 /// A mapping `K -> V` for densely indexed entity references.
 ///
diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs
index 6fe70523fe..813fdeca7f 100644
--- a/lib/entity/src/primary.rs
+++ b/lib/entity/src/primary.rs
@@ -1,10 +1,12 @@
 //! Densely numbered entity references as mapping keys.
+use iter::{Iter, IterMut};
+use keys::Keys;
 use std::iter::FromIterator;
 use std::marker::PhantomData;
 use std::ops::{Index, IndexMut};
 use std::slice;
 use std::vec::Vec;
-use {EntityRef, Iter, IterMut, Keys};
+use EntityRef;
 
 /// A primary mapping `K -> V` allocating dense entity references.
 ///
diff --git a/lib/entity/src/set.rs b/lib/entity/src/set.rs
index 4f7b631c93..58f1a7da0a 100644
--- a/lib/entity/src/set.rs
+++ b/lib/entity/src/set.rs
@@ -1,8 +1,9 @@
 //! Densely numbered entity references as set keys.
 
+use keys::Keys;
 use std::marker::PhantomData;
 use std::vec::Vec;
-use {EntityRef, Keys};
+use EntityRef;
 
 /// A set of `K` for densely indexed entity references.
 ///
diff --git a/lib/entity/src/sparse.rs b/lib/entity/src/sparse.rs
index 1a6de3e39d..b1af505304 100644
--- a/lib/entity/src/sparse.rs
+++ b/lib/entity/src/sparse.rs
@@ -7,11 +7,12 @@
 //! > Briggs, Torczon, *An efficient representation for sparse sets*,
 //!   ACM Letters on Programming Languages and Systems, Volume 2, Issue 1-4, March-Dec. 1993.
 
+use map::SecondaryMap;
 use std::mem;
 use std::slice;
 use std::u32;
 use std::vec::Vec;
-use {EntityRef, SecondaryMap};
+use EntityRef;
 
 /// Trait for extracting keys from values stored in a `SparseMap`.
 ///
diff --git a/lib/filetests/src/concurrent.rs b/lib/filetests/src/concurrent.rs
index df9034618b..ae04a3b6c5 100644
--- a/lib/filetests/src/concurrent.rs
+++ b/lib/filetests/src/concurrent.rs
@@ -7,13 +7,14 @@ use cranelift_codegen::dbg::LOG_FILENAME_PREFIX;
 use cranelift_codegen::timing;
 use file_per_thread_logger;
 use num_cpus;
+use runone;
 use std::panic::catch_unwind;
 use std::path::{Path, PathBuf};
 use std::sync::mpsc::{channel, Receiver, Sender};
 use std::sync::{Arc, Mutex};
 use std::thread;
 use std::time::Duration;
-use {runone, TestResult};
+use TestResult;
 
 /// Request sent to worker threads contains jobid and path.
 struct Request(usize, PathBuf);
diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs
index 778150fa7c..37224c2dd3 100644
--- a/lib/filetests/src/runner.rs
+++ b/lib/filetests/src/runner.rs
@@ -5,12 +5,13 @@
 
 use concurrent::{ConcurrentRunner, Reply};
 use cranelift_codegen::timing;
+use runone;
 use std::error::Error;
 use std::ffi::OsStr;
 use std::fmt::{self, Display};
 use std::path::{Path, PathBuf};
 use std::time;
-use {runone, TestResult};
+use TestResult;
 
 /// Timeout in seconds when we're not making progress.
 const TIMEOUT_PANIC: usize = 10;
diff --git a/lib/module/src/data_context.rs b/lib/module/src/data_context.rs
index 4a7fdd7a6e..d941e9c411 100644
--- a/lib/module/src/data_context.rs
+++ b/lib/module/src/data_context.rs
@@ -132,8 +132,8 @@ impl DataContext {
 
 #[cfg(test)]
 mod tests {
+    use super::{DataContext, Init};
     use cranelift_codegen::ir;
-    use {DataContext, Init};
 
     #[test]
     fn basic_data_context() {

From b8416525391033caba82b99c7c80f8cdcc482cd1 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 10 Dec 2018 07:12:20 -0800
Subject: [PATCH 2260/3084] Add a `Slice` to supplement `PrimaryMap`.

---
 lib/entity/src/boxed_slice.rs | 315 ++++++++++++++++++++++++++++++++++
 lib/entity/src/lib.rs         |   2 +
 lib/entity/src/primary.rs     |  10 +-
 3 files changed, 326 insertions(+), 1 deletion(-)
 create mode 100644 lib/entity/src/boxed_slice.rs

diff --git a/lib/entity/src/boxed_slice.rs b/lib/entity/src/boxed_slice.rs
new file mode 100644
index 0000000000..6f9890e7d7
--- /dev/null
+++ b/lib/entity/src/boxed_slice.rs
@@ -0,0 +1,315 @@
+//! Boxed slices for `PrimaryMap`.
+
+use iter::{Iter, IterMut};
+use keys::Keys;
+use std::marker::PhantomData;
+use std::ops::{Index, IndexMut};
+use std::slice;
+use EntityRef;
+
+/// A slice mapping `K -> V` allocating dense entity references.
+///
+/// The `BoxedSlice` data structure uses the dense index space to implement a map with a boxed
+/// slice.
+#[derive(Debug, Clone)]
+pub struct BoxedSlice
+where
+    K: EntityRef,
+{
+    elems: Box<[V]>,
+    unused: PhantomData,
+}
+
+impl BoxedSlice
+where
+    K: EntityRef,
+{
+    /// Create a new slice from a raw pointer. A safer way to create slices is
+    /// to use `PrimaryMap::into_boxed_slice()`.
+    pub unsafe fn from_raw(raw: *mut [V]) -> Self {
+        Self {
+            elems: Box::from_raw(raw),
+            unused: PhantomData,
+        }
+    }
+
+    /// Check if `k` is a valid key in the map.
+    pub fn is_valid(&self, k: K) -> bool {
+        k.index() < self.elems.len()
+    }
+
+    /// Get the element at `k` if it exists.
+    pub fn get(&self, k: K) -> Option<&V> {
+        self.elems.get(k.index())
+    }
+
+    /// Get the element at `k` if it exists, mutable version.
+    pub fn get_mut(&mut self, k: K) -> Option<&mut V> {
+        self.elems.get_mut(k.index())
+    }
+
+    /// Is this map completely empty?
+    pub fn is_empty(&self) -> bool {
+        self.elems.is_empty()
+    }
+
+    /// Get the total number of entity references created.
+    pub fn len(&self) -> usize {
+        self.elems.len()
+    }
+
+    /// Iterate over all the keys in this map.
+    pub fn keys(&self) -> Keys {
+        Keys::with_len(self.elems.len())
+    }
+
+    /// Iterate over all the values in this map.
+    pub fn values(&self) -> slice::Iter {
+        self.elems.iter()
+    }
+
+    /// Iterate over all the values in this map, mutable edition.
+    pub fn values_mut(&mut self) -> slice::IterMut {
+        self.elems.iter_mut()
+    }
+
+    /// Iterate over all the keys and values in this map.
+    pub fn iter(&self) -> Iter {
+        Iter::new(self.elems.iter())
+    }
+
+    /// Iterate over all the keys and values in this map, mutable edition.
+    pub fn iter_mut(&mut self) -> IterMut {
+        IterMut::new(self.elems.iter_mut())
+    }
+
+    /// Get the key that will be assigned to the next pushed value.
+    pub fn next_key(&self) -> K {
+        K::new(self.elems.len())
+    }
+
+    /// Returns the last element that was inserted in the map.
+    pub fn last(&self) -> Option<&V> {
+        self.elems.last()
+    }
+}
+
+/// Immutable indexing into a `BoxedSlice`.
+/// The indexed value must be in the map.
+impl Index for BoxedSlice
+where
+    K: EntityRef,
+{
+    type Output = V;
+
+    fn index(&self, k: K) -> &V {
+        &self.elems[k.index()]
+    }
+}
+
+/// Mutable indexing into a `BoxedSlice`.
+impl IndexMut for BoxedSlice
+where
+    K: EntityRef,
+{
+    fn index_mut(&mut self, k: K) -> &mut V {
+        &mut self.elems[k.index()]
+    }
+}
+
+impl<'a, K, V> IntoIterator for &'a BoxedSlice
+where
+    K: EntityRef,
+{
+    type Item = (K, &'a V);
+    type IntoIter = Iter<'a, K, V>;
+
+    fn into_iter(self) -> Self::IntoIter {
+        Iter::new(self.elems.iter())
+    }
+}
+
+impl<'a, K, V> IntoIterator for &'a mut BoxedSlice
+where
+    K: EntityRef,
+{
+    type Item = (K, &'a mut V);
+    type IntoIter = IterMut<'a, K, V>;
+
+    fn into_iter(self) -> Self::IntoIter {
+        IterMut::new(self.elems.iter_mut())
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use primary::PrimaryMap;
+
+    // `EntityRef` impl for testing.
+    #[derive(Clone, Copy, Debug, PartialEq, Eq)]
+    struct E(u32);
+
+    impl EntityRef for E {
+        fn new(i: usize) -> Self {
+            E(i as u32)
+        }
+        fn index(self) -> usize {
+            self.0 as usize
+        }
+    }
+
+    #[test]
+    fn basic() {
+        let r0 = E(0);
+        let r1 = E(1);
+        let p = PrimaryMap::::new();
+        let m = p.into_boxed_slice();
+
+        let v: Vec = m.keys().collect();
+        assert_eq!(v, []);
+
+        assert!(!m.is_valid(r0));
+        assert!(!m.is_valid(r1));
+    }
+
+    #[test]
+    fn iter() {
+        let mut p: PrimaryMap = PrimaryMap::new();
+        p.push(12);
+        p.push(33);
+        let mut m = p.into_boxed_slice();
+
+        let mut i = 0;
+        for (key, value) in &m {
+            assert_eq!(key.index(), i);
+            match i {
+                0 => assert_eq!(*value, 12),
+                1 => assert_eq!(*value, 33),
+                _ => panic!(),
+            }
+            i += 1;
+        }
+        i = 0;
+        for (key_mut, value_mut) in m.iter_mut() {
+            assert_eq!(key_mut.index(), i);
+            match i {
+                0 => assert_eq!(*value_mut, 12),
+                1 => assert_eq!(*value_mut, 33),
+                _ => panic!(),
+            }
+            i += 1;
+        }
+    }
+
+    #[test]
+    fn iter_rev() {
+        let mut p: PrimaryMap = PrimaryMap::new();
+        p.push(12);
+        p.push(33);
+        let mut m = p.into_boxed_slice();
+
+        let mut i = 2;
+        for (key, value) in m.iter().rev() {
+            i -= 1;
+            assert_eq!(key.index(), i);
+            match i {
+                0 => assert_eq!(*value, 12),
+                1 => assert_eq!(*value, 33),
+                _ => panic!(),
+            }
+        }
+
+        i = 2;
+        for (key, value) in m.iter_mut().rev() {
+            i -= 1;
+            assert_eq!(key.index(), i);
+            match i {
+                0 => assert_eq!(*value, 12),
+                1 => assert_eq!(*value, 33),
+                _ => panic!(),
+            }
+        }
+    }
+    #[test]
+    fn keys() {
+        let mut p: PrimaryMap = PrimaryMap::new();
+        p.push(12);
+        p.push(33);
+        let m = p.into_boxed_slice();
+
+        let mut i = 0;
+        for key in m.keys() {
+            assert_eq!(key.index(), i);
+            i += 1;
+        }
+    }
+
+    #[test]
+    fn keys_rev() {
+        let mut p: PrimaryMap = PrimaryMap::new();
+        p.push(12);
+        p.push(33);
+        let m = p.into_boxed_slice();
+
+        let mut i = 2;
+        for key in m.keys().rev() {
+            i -= 1;
+            assert_eq!(key.index(), i);
+        }
+    }
+
+    #[test]
+    fn values() {
+        let mut p: PrimaryMap = PrimaryMap::new();
+        p.push(12);
+        p.push(33);
+        let mut m = p.into_boxed_slice();
+
+        let mut i = 0;
+        for value in m.values() {
+            match i {
+                0 => assert_eq!(*value, 12),
+                1 => assert_eq!(*value, 33),
+                _ => panic!(),
+            }
+            i += 1;
+        }
+        i = 0;
+        for value_mut in m.values_mut() {
+            match i {
+                0 => assert_eq!(*value_mut, 12),
+                1 => assert_eq!(*value_mut, 33),
+                _ => panic!(),
+            }
+            i += 1;
+        }
+    }
+
+    #[test]
+    fn values_rev() {
+        let mut p: PrimaryMap = PrimaryMap::new();
+        p.push(12);
+        p.push(33);
+        let mut m = p.into_boxed_slice();
+
+        let mut i = 2;
+        for value in m.values().rev() {
+            i -= 1;
+            match i {
+                0 => assert_eq!(*value, 12),
+                1 => assert_eq!(*value, 33),
+                _ => panic!(),
+            }
+        }
+        i = 2;
+        for value_mut in m.values_mut().rev() {
+            i -= 1;
+            match i {
+                0 => assert_eq!(*value_mut, 12),
+                1 => assert_eq!(*value_mut, 33),
+                _ => panic!(),
+            }
+        }
+    }
+}
diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs
index 7d63eaee16..e445c0d8a4 100644
--- a/lib/entity/src/lib.rs
+++ b/lib/entity/src/lib.rs
@@ -129,6 +129,7 @@ macro_rules! entity_impl {
 
 pub mod packed_option;
 
+mod boxed_slice;
 mod iter;
 mod keys;
 mod list;
@@ -137,6 +138,7 @@ mod primary;
 mod set;
 mod sparse;
 
+pub use self::boxed_slice::BoxedSlice;
 pub use self::iter::{Iter, IterMut};
 pub use self::keys::Keys;
 pub use self::list::{EntityList, ListPool};
diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs
index 813fdeca7f..6d710c2e4f 100644
--- a/lib/entity/src/primary.rs
+++ b/lib/entity/src/primary.rs
@@ -1,4 +1,5 @@
 //! Densely numbered entity references as mapping keys.
+use boxed_slice::BoxedSlice;
 use iter::{Iter, IterMut};
 use keys::Keys;
 use std::iter::FromIterator;
@@ -21,7 +22,8 @@ use EntityRef;
 /// Note that `PrimaryMap` doesn't implement `Deref` or `DerefMut`, which would allow
 /// `&PrimaryMap` to convert to `&[V]`. One of the main advantages of `PrimaryMap` is
 /// that it only allows indexing with the distinct `EntityRef` key type, so converting to a
-/// plain slice would make it easier to use incorrectly.
+/// plain slice would make it easier to use incorrectly. To make a slice of a `PrimaryMap`, use
+/// `into_boxed_slice`.
 #[derive(Debug, Clone)]
 pub struct PrimaryMap
 where
@@ -132,6 +134,11 @@ where
     pub fn reserve_exact(&mut self, additional: usize) {
         self.elems.reserve_exact(additional)
     }
+
+    /// Consumes this `PrimaryMap` and produces a `BoxedSlice`.
+    pub fn into_boxed_slice(self) -> BoxedSlice {
+        unsafe { BoxedSlice::::from_raw(Box::<[V]>::into_raw(self.elems.into_boxed_slice())) }
+    }
 }
 
 /// Immutable indexing into an `PrimaryMap`.
@@ -378,6 +385,7 @@ mod tests {
         let mut m: PrimaryMap = PrimaryMap::new();
         m.push(12);
         m.push(33);
+
         let n = m.values().collect::>();
         assert!(m.len() == n.len());
         for (me, ne) in m.values().zip(n.values()) {

From 68a13646f9e5b46e402f2a8a77d720317d0d3399 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 10 Dec 2018 07:17:39 -0800
Subject: [PATCH 2261/3084] Add a from_u32 function to EntityRef.

This eliminates the need for casting in common cases.
---
 lib/entity/src/lib.rs | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs
index e445c0d8a4..e86ea0d8fd 100644
--- a/lib/entity/src/lib.rs
+++ b/lib/entity/src/lib.rs
@@ -100,6 +100,13 @@ macro_rules! entity_impl {
         }
 
         impl $entity {
+            /// Return the underlying index value as a `u32`.
+            #[allow(dead_code)]
+            pub fn from_u32(x: u32) -> Self {
+                debug_assert!(x < $crate::__core::u32::MAX);
+                $entity(x)
+            }
+
             /// Return the underlying index value as a `u32`.
             #[allow(dead_code)]
             pub fn as_u32(self) -> u32 {

From a55c933f192e4b29f7e250156da3abd1c9f64cd2 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 11 Dec 2018 05:44:26 -0800
Subject: [PATCH 2262/3084] Factor out a `MemFlags` constructor for trusted
 notrap/aligned accesses.

---
 lib/codegen/src/ir/memflags.rs           | 9 +++++++++
 lib/codegen/src/legalizer/globalvalue.rs | 4 +---
 lib/codegen/src/legalizer/mod.rs         | 4 +---
 lib/wasm/src/code_translator.rs          | 8 ++------
 lib/wasm/src/environ/dummy.rs            | 4 +---
 5 files changed, 14 insertions(+), 15 deletions(-)

diff --git a/lib/codegen/src/ir/memflags.rs b/lib/codegen/src/ir/memflags.rs
index f459abdb3f..6acbe167ac 100644
--- a/lib/codegen/src/ir/memflags.rs
+++ b/lib/codegen/src/ir/memflags.rs
@@ -26,6 +26,15 @@ impl MemFlags {
         Self { bits: 0 }
     }
 
+    /// Create a set of flags representing an access from a "trusted" address, meaning it's
+    /// known to be aligned and non-trapping.
+    pub fn trusted() -> Self {
+        let mut result = Self::new();
+        result.set_notrap();
+        result.set_aligned();
+        result
+    }
+
     /// Read a flag bit.
     fn read(self, bit: FlagBit) -> bool {
         self.bits & (1 << bit as usize) != 0
diff --git a/lib/codegen/src/legalizer/globalvalue.rs b/lib/codegen/src/legalizer/globalvalue.rs
index 1807734c53..e6b8dce39e 100644
--- a/lib/codegen/src/legalizer/globalvalue.rs
+++ b/lib/codegen/src/legalizer/globalvalue.rs
@@ -110,9 +110,7 @@ fn load_addr(
     };
 
     // Global-value loads are always notrap and aligned. They may be readonly.
-    let mut mflags = ir::MemFlags::new();
-    mflags.set_notrap();
-    mflags.set_aligned();
+    let mut mflags = ir::MemFlags::trusted();
     if readonly {
         mflags.set_readonly();
     }
diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs
index b552934289..f2a4636719 100644
--- a/lib/codegen/src/legalizer/mod.rs
+++ b/lib/codegen/src/legalizer/mod.rs
@@ -400,10 +400,8 @@ fn expand_stack_load(
 
     let addr = pos.ins().stack_addr(addr_ty, stack_slot, offset);
 
-    let mut mflags = MemFlags::new();
     // Stack slots are required to be accessible and aligned.
-    mflags.set_notrap();
-    mflags.set_aligned();
+    let mflags = MemFlags::trusted();
     pos.func.dfg.replace(inst).load(ty, mflags, addr, 0);
 }
 
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index f62ac89f77..8ca304f802 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -77,9 +77,7 @@ pub fn translate_operator(
                 GlobalVariable::Const(val) => val,
                 GlobalVariable::Memory { gv, offset, ty } => {
                     let addr = builder.ins().global_value(environ.pointer_type(), gv);
-                    let mut flags = ir::MemFlags::new();
-                    flags.set_notrap();
-                    flags.set_aligned();
+                    let flags = ir::MemFlags::trusted();
                     builder.ins().load(ty, flags, addr, offset)
                 }
             };
@@ -90,9 +88,7 @@ pub fn translate_operator(
                 GlobalVariable::Const(_) => panic!("global #{} is a constant", global_index),
                 GlobalVariable::Memory { gv, offset, ty } => {
                     let addr = builder.ins().global_value(environ.pointer_type(), gv);
-                    let mut flags = ir::MemFlags::new();
-                    flags.set_notrap();
-                    flags.set_aligned();
+                    let flags = ir::MemFlags::trusted();
                     let val = state.pop1();
                     debug_assert_eq!(ty, builder.func.dfg.value_type(val));
                     builder.ins().store(flags, val, addr, offset);
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 0a31b17b6a..a31a750b56 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -270,9 +270,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
             let ext = pos.ins().uextend(I64, callee);
             pos.ins().imul_imm(ext, 4)
         };
-        let mut mflags = ir::MemFlags::new();
-        mflags.set_notrap();
-        mflags.set_aligned();
+        let mflags = ir::MemFlags::trusted();
         let func_ptr = pos.ins().load(ptr, mflags, callee_offset, 0);
 
         // Build a value list for the indirect call instruction containing the callee, call_args,

From bc18085ad19805c8a14df80a45510f7c3d8231f8 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 11 Dec 2018 12:25:06 -0800
Subject: [PATCH 2263/3084] Use EntityRef::from_u32 to reduce casting.

---
 lib/codegen/src/ir/progpoint.rs      |  4 ++--
 lib/codegen/src/regalloc/virtregs.rs |  3 +--
 lib/module/src/module.rs             |  6 +++---
 lib/wasm/src/code_translator.rs      | 11 +++++------
 lib/wasm/src/sections_translator.rs  | 26 +++++++++++---------------
 lib/wasm/src/state.rs                | 11 +++++------
 6 files changed, 27 insertions(+), 34 deletions(-)

diff --git a/lib/codegen/src/ir/progpoint.rs b/lib/codegen/src/ir/progpoint.rs
index b505af05a0..19d14892b1 100644
--- a/lib/codegen/src/ir/progpoint.rs
+++ b/lib/codegen/src/ir/progpoint.rs
@@ -85,9 +85,9 @@ impl From for ExpandedProgramPoint {
 impl From for ExpandedProgramPoint {
     fn from(pp: ProgramPoint) -> Self {
         if pp.0 & 1 == 0 {
-            ExpandedProgramPoint::Inst(Inst::new((pp.0 / 2) as usize))
+            ExpandedProgramPoint::Inst(Inst::from_u32(pp.0 / 2))
         } else {
-            ExpandedProgramPoint::Ebb(Ebb::new((pp.0 / 2) as usize))
+            ExpandedProgramPoint::Ebb(Ebb::from_u32(pp.0 / 2))
         }
     }
 }
diff --git a/lib/codegen/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs
index ac4856261f..63c2e7a413 100644
--- a/lib/codegen/src/regalloc/virtregs.rs
+++ b/lib/codegen/src/regalloc/virtregs.rs
@@ -13,7 +13,6 @@
 
 use dbg::DisplayList;
 use dominator_tree::DominatorTreePreorder;
-use entity::EntityRef;
 use entity::{EntityList, ListPool};
 use entity::{Keys, PrimaryMap, SecondaryMap};
 use ir::{Function, Value};
@@ -258,7 +257,7 @@ impl UFEntry {
     /// Decode a table entry.
     fn decode(x: i32) -> Self {
         if x < 0 {
-            UFEntry::Link(Value::new((!x) as usize))
+            UFEntry::Link(Value::from_u32((!x) as u32))
         } else {
             UFEntry::Rank(x as u32)
         }
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 18f6167a2f..e070514c8a 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -5,7 +5,7 @@
 // TODO: Factor out `ir::Function`'s `ext_funcs` and `global_values` into a struct
 // shared with `DataContext`?
 
-use cranelift_codegen::entity::{EntityRef, PrimaryMap};
+use cranelift_codegen::entity::PrimaryMap;
 use cranelift_codegen::{binemit, ir, isa, CodegenError, Context};
 use data_context::DataContext;
 use std::borrow::ToOwned;
@@ -222,7 +222,7 @@ where
     fn get_function_info(&self, name: &ir::ExternalName) -> &ModuleFunction {
         if let ir::ExternalName::User { namespace, index } = *name {
             debug_assert_eq!(namespace, 0);
-            let func = FuncId::new(index as usize);
+            let func = FuncId::from_u32(index);
             &self.functions[func]
         } else {
             panic!("unexpected ExternalName kind {}", name)
@@ -233,7 +233,7 @@ where
     fn get_data_info(&self, name: &ir::ExternalName) -> &ModuleData {
         if let ir::ExternalName::User { namespace, index } = *name {
             debug_assert_eq!(namespace, 1);
-            let data = DataId::new(index as usize);
+            let data = DataId::from_u32(index);
             &self.data_objects[data]
         } else {
             panic!("unexpected ExternalName kind {}", name)
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index 8ca304f802..34aa37ef5e 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -26,7 +26,6 @@ use cranelift_codegen::ir::condcodes::{FloatCC, IntCC};
 use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags};
 use cranelift_codegen::packed_option::ReservedValue;
-use cranelift_entity::EntityRef;
 use cranelift_frontend::{FunctionBuilder, Variable};
 use environ::{FuncEnvironment, GlobalVariable, ReturnMode, WasmError, WasmResult};
 use state::{ControlStackFrame, TranslationState};
@@ -355,7 +354,7 @@ pub fn translate_operator(
             let (fref, num_args) = state.get_direct_func(builder.func, function_index, environ);
             let call = environ.translate_call(
                 builder.cursor(),
-                FuncIndex::new(function_index as usize),
+                FuncIndex::from_u32(function_index),
                 fref,
                 state.peekn(num_args),
             )?;
@@ -378,9 +377,9 @@ pub fn translate_operator(
             let callee = state.pop1();
             let call = environ.translate_call_indirect(
                 builder.cursor(),
-                TableIndex::new(table_index as usize),
+                TableIndex::from_u32(table_index),
                 table,
-                SignatureIndex::new(index as usize),
+                SignatureIndex::from_u32(index),
                 sigref,
                 callee,
                 state.peekn(num_args),
@@ -401,13 +400,13 @@ pub fn translate_operator(
         Operator::MemoryGrow { reserved } => {
             // The WebAssembly MVP only supports one linear memory, but we expect the reserved
             // argument to be a memory index.
-            let heap_index = MemoryIndex::new(reserved as usize);
+            let heap_index = MemoryIndex::from_u32(reserved);
             let heap = state.get_heap(builder.func, reserved, environ);
             let val = state.pop1();
             state.push1(environ.translate_memory_grow(builder.cursor(), heap_index, heap, val)?)
         }
         Operator::MemorySize { reserved } => {
-            let heap_index = MemoryIndex::new(reserved as usize);
+            let heap_index = MemoryIndex::from_u32(reserved);
             let heap = state.get_heap(builder.func, reserved, environ);
             state.push1(environ.translate_memory_size(builder.cursor(), heap_index, heap)?);
         }
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 10ec04880b..4ea84d51f2 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -70,11 +70,7 @@ pub fn parse_import_section<'data>(
 
         match import.ty {
             ImportSectionEntryType::Function(sig) => {
-                environ.declare_func_import(
-                    SignatureIndex::new(sig as usize),
-                    module_name,
-                    field_name,
-                );
+                environ.declare_func_import(SignatureIndex::from_u32(sig), module_name, field_name);
             }
             ImportSectionEntryType::Memory(MemoryType {
                 limits: ref memlimits,
@@ -127,7 +123,7 @@ pub fn parse_function_section(
 ) -> WasmResult<()> {
     for entry in functions {
         let sigindex = entry?;
-        environ.declare_func_type(SignatureIndex::new(sigindex as usize));
+        environ.declare_func_type(SignatureIndex::from_u32(sigindex));
     }
     Ok(())
 }
@@ -187,7 +183,7 @@ pub fn parse_global_section(
             Operator::F32Const { value } => GlobalInit::F32Const(value.bits()),
             Operator::F64Const { value } => GlobalInit::F64Const(value.bits()),
             Operator::GetGlobal { global_index } => {
-                GlobalInit::GetGlobal(GlobalIndex::new(global_index as usize))
+                GlobalInit::GetGlobal(GlobalIndex::from_u32(global_index))
             }
             ref s => panic!("unsupported init expr in global section: {:?}", s),
         };
@@ -230,7 +226,7 @@ pub fn parse_export_section<'data>(
 
 /// Parses the Start section of the wasm module.
 pub fn parse_start_section(index: u32, environ: &mut ModuleEnvironment) -> WasmResult<()> {
-    environ.declare_start_func(FuncIndex::new(index as usize));
+    environ.declare_start_func(FuncIndex::from_u32(index));
     Ok(())
 }
 
@@ -249,11 +245,11 @@ pub fn parse_element_section<'data>(
         let (base, offset) = match init_expr_reader.read_operator()? {
             Operator::I32Const { value } => (None, value as u32 as usize),
             Operator::GetGlobal { global_index } => match environ
-                .get_global(GlobalIndex::new(global_index as usize))
+                .get_global(GlobalIndex::from_u32(global_index))
                 .initializer
             {
                 GlobalInit::I32Const(value) => (None, value as u32 as usize),
-                GlobalInit::Import => (Some(GlobalIndex::new(global_index as usize)), 0),
+                GlobalInit::Import => (Some(GlobalIndex::from_u32(global_index)), 0),
                 _ => panic!("should not happen"),
             },
             ref s => panic!("unsupported init expr in element section: {:?}", s),
@@ -262,9 +258,9 @@ pub fn parse_element_section<'data>(
         let mut elems = Vec::new();
         for item in items_reader {
             let x = item?;
-            elems.push(FuncIndex::new(x as usize));
+            elems.push(FuncIndex::from_u32(x));
         }
-        environ.declare_table_elements(TableIndex::new(table_index as usize), base, offset, elems)
+        environ.declare_table_elements(TableIndex::from_u32(table_index), base, offset, elems)
     }
     Ok(())
 }
@@ -297,17 +293,17 @@ pub fn parse_data_section<'data>(
         let (base, offset) = match init_expr_reader.read_operator()? {
             Operator::I32Const { value } => (None, value as u32 as usize),
             Operator::GetGlobal { global_index } => match environ
-                .get_global(GlobalIndex::new(global_index as usize))
+                .get_global(GlobalIndex::from_u32(global_index))
                 .initializer
             {
                 GlobalInit::I32Const(value) => (None, value as u32 as usize),
-                GlobalInit::Import => (Some(GlobalIndex::new(global_index as usize)), 0),
+                GlobalInit::Import => (Some(GlobalIndex::from_u32(global_index)), 0),
                 _ => panic!("should not happen"),
             },
             ref s => panic!("unsupported init expr in data section: {:?}", s),
         };
         environ.declare_data_initialization(
-            MemoryIndex::new(memory_index as usize),
+            MemoryIndex::from_u32(memory_index),
             base,
             offset,
             data,
diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs
index df909d8b38..02efc65ae3 100644
--- a/lib/wasm/src/state.rs
+++ b/lib/wasm/src/state.rs
@@ -4,7 +4,6 @@
 //! value and control stacks during the translation of a single function.
 
 use cranelift_codegen::ir::{self, Ebb, Inst, Value};
-use cranelift_entity::EntityRef;
 use environ::{FuncEnvironment, GlobalVariable};
 use std::collections::HashMap;
 use std::vec::Vec;
@@ -287,7 +286,7 @@ impl TranslationState {
         index: u32,
         environ: &mut FE,
     ) -> GlobalVariable {
-        let index = GlobalIndex::new(index as usize);
+        let index = GlobalIndex::from_u32(index);
         *self
             .globals
             .entry(index)
@@ -302,7 +301,7 @@ impl TranslationState {
         index: u32,
         environ: &mut FE,
     ) -> ir::Heap {
-        let index = MemoryIndex::new(index as usize);
+        let index = MemoryIndex::from_u32(index);
         *self
             .heaps
             .entry(index)
@@ -317,7 +316,7 @@ impl TranslationState {
         index: u32,
         environ: &mut FE,
     ) -> ir::Table {
-        let index = TableIndex::new(index as usize);
+        let index = TableIndex::from_u32(index);
         *self
             .tables
             .entry(index)
@@ -334,7 +333,7 @@ impl TranslationState {
         index: u32,
         environ: &mut FE,
     ) -> (ir::SigRef, usize) {
-        let index = SignatureIndex::new(index as usize);
+        let index = SignatureIndex::from_u32(index);
         *self.signatures.entry(index).or_insert_with(|| {
             let sig = environ.make_indirect_sig(func, index);
             (sig, normal_args(&func.dfg.signatures[sig]))
@@ -351,7 +350,7 @@ impl TranslationState {
         index: u32,
         environ: &mut FE,
     ) -> (ir::FuncRef, usize) {
-        let index = FuncIndex::new(index as usize);
+        let index = FuncIndex::from_u32(index);
         *self.functions.entry(index).or_insert_with(|| {
             let fref = environ.make_direct_func(func, index);
             let sig = func.dfg.ext_funcs[fref].signature;

From c8e457e8343dc9dd10a9ecdccd52966754089664 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 11 Dec 2018 12:54:23 -0800
Subject: [PATCH 2264/3084] Bump version to 0.26.0

---
 cranelift/Cargo.toml        | 28 ++++++++++++++--------------
 cranelift/publish-all.sh    |  2 +-
 lib/bforest/Cargo.toml      |  4 ++--
 lib/codegen/Cargo.toml      |  8 ++++----
 lib/codegen/meta/Cargo.toml |  4 ++--
 lib/entity/Cargo.toml       |  2 +-
 lib/faerie/Cargo.toml       |  6 +++---
 lib/filetests/Cargo.toml    |  8 ++++----
 lib/frontend/Cargo.toml     |  4 ++--
 lib/module/Cargo.toml       |  6 +++---
 lib/native/Cargo.toml       |  4 ++--
 lib/preopt/Cargo.toml       |  6 +++---
 lib/reader/Cargo.toml       |  4 ++--
 lib/serde/Cargo.toml        |  6 +++---
 lib/simplejit/Cargo.toml    | 14 +++++++-------
 lib/umbrella/Cargo.toml     |  6 +++---
 lib/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 709db5bf2d..e2afd0799f 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.25.0"
+version = "0.26.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -14,19 +14,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.25.0" }
-cranelift-entity = { path = "lib/entity", version = "0.25.0" }
-cranelift-reader = { path = "lib/reader", version = "0.25.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.25.0" }
-cranelift-serde = { path = "lib/serde", version = "0.25.0", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.25.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.25.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.25.0" }
-cranelift-module = { path = "lib/module", version = "0.25.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.25.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.25.0" }
-cranelift-preopt = { path = "lib/preopt", version = "0.25.0" }
-cranelift = { path = "lib/umbrella", version = "0.25.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.26.0" }
+cranelift-entity = { path = "lib/entity", version = "0.26.0" }
+cranelift-reader = { path = "lib/reader", version = "0.26.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.26.0" }
+cranelift-serde = { path = "lib/serde", version = "0.26.0", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.26.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.26.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.26.0" }
+cranelift-module = { path = "lib/module", version = "0.26.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.26.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.26.0" }
+cranelift-preopt = { path = "lib/preopt", version = "0.26.0" }
+cranelift = { path = "lib/umbrella", version = "0.26.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 67f29244e0..ba9f196ce2 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.25.0"
+version="0.26.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index 17529993be..c2da583ccc 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.25.0"
+version = "0.26.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 keywords = ["btree", "forest", "set", "map"]
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.25.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 35561ac788..2ab25583cb 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.25.0"
+version = "0.26.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.25.0", default-features = false }
-cranelift-bforest = { path = "../bforest", version = "0.25.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
+cranelift-bforest = { path = "../bforest", version = "0.26.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -25,7 +25,7 @@ log = { version = "0.4.4", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.25.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.26.0" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 674af361fa..92e4f34f20 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,14 +1,14 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.25.0"
+version = "0.26.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-entity = { path = "../../entity", version = "0.25.0" }
+cranelift-entity = { path = "../../entity", version = "0.26.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index b77ffe7e07..a7eeae5787 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.25.0"
+version = "0.26.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index ef6b354818..003ec93846 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.25.0"
+version = "0.26.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.25.0" }
-cranelift-module = { path = "../module", version = "0.25.0" }
+cranelift-codegen = { path = "../codegen", version = "0.26.0" }
+cranelift-module = { path = "../module", version = "0.26.0" }
 faerie = "0.6.0"
 goblin = "0.0.19"
 failure = "0.1.2"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 294b1b4e17..84701f2e83 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.25.0"
+version = "0.26.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -9,9 +9,9 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.25.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../reader", version = "0.25.0" }
-cranelift-preopt = { path = "../preopt", version = "0.25.0" }
+cranelift-codegen = { path = "../codegen", version = "0.26.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../reader", version = "0.26.0" }
+cranelift-preopt = { path = "../preopt", version = "0.26.0" }
 file-per-thread-logger = "0.1.1"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 973ec60e35..87791458e9 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.25.0"
+version = "0.26.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 log = { version = "0.4.4", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 6b21bfdc1b..1694d065b2 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.25.0"
+version = "0.26.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.25.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
 log = { version = "0.4.4", default-features = false }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index cd991470c3..20fc2911df 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.25.0"
+version = "0.26.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/preopt/Cargo.toml b/lib/preopt/Cargo.toml
index 601e26d19a..f9337faa74 100644
--- a/lib/preopt/Cargo.toml
+++ b/lib/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.25.0"
+version = "0.26.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,8 +11,8 @@ readme = "README.md"
 keywords = ["optimize", "compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.25.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 20fbeb7469..8d4b70b975 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.25.0"
+version = "0.26.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -9,7 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.25.0" }
+cranelift-codegen = { path = "../codegen", version = "0.26.0" }
 target-lexicon = "0.2.0"
 
 [badges]
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index b07fd3cbc2..086cfbe5a5 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.25.0"
+version = "0.26.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
-cranelift-reader = { path = "../reader", version = "0.25.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
+cranelift-reader = { path = "../reader", version = "0.26.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index f4d94d3d0b..9ddbed39e8 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.25.0"
+version = "0.26.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.25.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.25.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
+cranelift-module = { path = "../module", version = "0.26.0", default-features = false }
+cranelift-native = { path = "../native", version = "0.26.0", default-features = false }
 region = "1.0.0"
 libc = { version = "0.2.42", default-features = false }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.2.0", default-features = false }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../umbrella", version = "0.25.0" }
-cranelift-frontend = { path = "../frontend", version = "0.25.0" }
-cranelift-entity = { path = "../entity", version = "0.25.0" }
+cranelift = { path = "../umbrella", version = "0.26.0" }
+cranelift-frontend = { path = "../frontend", version = "0.26.0" }
+cranelift-entity = { path = "../entity", version = "0.26.0" }
 
 [features]
 default = ["std"]
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index 6a24c72c3f..1ca70dcac1 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.25.0"
+version = "0.26.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,8 +11,8 @@ readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.25.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.26.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index a88550d1e1..cf14471a40 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.25.0"
+version = "0.26.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,9 +11,9 @@ keywords = ["webassembly", "wasm"]
 
 [dependencies]
 wasmparser = { version = "0.22.0", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.25.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.25.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.25.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.26.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 8bd35e154b9497be6d1194c299d463ce706322bb Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 11 Dec 2018 13:22:28 -0800
Subject: [PATCH 2265/3084] Tidy up some match bindings to be more consistent
 with the rest of the codebase.

---
 lib/codegen/meta/src/cdsl/settings.rs | 9 +++++----
 lib/codegen/meta/src/gen_settings.rs  | 8 ++++----
 lib/codegen/meta/src/srcgen.rs        | 4 ++--
 3 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/lib/codegen/meta/src/cdsl/settings.rs b/lib/codegen/meta/src/cdsl/settings.rs
index 253f7680d6..805dfd2a7a 100644
--- a/lib/codegen/meta/src/cdsl/settings.rs
+++ b/lib/codegen/meta/src/cdsl/settings.rs
@@ -101,7 +101,8 @@ impl Preset {
             let mask = setting.byte_mask();
             let val = setting.byte_for_value(true);
             assert!((val & !mask) == 0);
-            let (l_mask, l_val) = layout.get_mut(setting.byte_offset as usize).unwrap();
+            let (ref mut l_mask, ref mut l_val) =
+                *layout.get_mut(setting.byte_offset as usize).unwrap();
             *l_mask |= mask;
             *l_val = (*l_val & !mask) | val;
         }
@@ -186,15 +187,15 @@ impl<'a> Into for (BoolSettingIndex, &'a SettingGroup) {
 
 impl PredicateNode {
     fn render(&self, group: &SettingGroup) -> String {
-        match self {
+        match *self {
             PredicateNode::OwnedBool(bool_setting_index) => format!(
                 "{}.{}()",
                 group.name, group.settings[bool_setting_index.0].name
             ),
-            PredicateNode::SharedBool(group_name, bool_name) => {
+            PredicateNode::SharedBool(ref group_name, ref bool_name) => {
                 format!("{}.{}()", group_name, bool_name)
             }
-            PredicateNode::And(lhs, rhs) => {
+            PredicateNode::And(ref lhs, ref rhs) => {
                 format!("{} && {}", lhs.render(group), rhs.render(group))
             }
         }
diff --git a/lib/codegen/meta/src/gen_settings.rs b/lib/codegen/meta/src/gen_settings.rs
index cd50a57f88..93fddce210 100644
--- a/lib/codegen/meta/src/gen_settings.rs
+++ b/lib/codegen/meta/src/gen_settings.rs
@@ -224,7 +224,7 @@ enum SettingOrPreset<'a> {
 
 impl<'a> SettingOrPreset<'a> {
     fn name(&self) -> &str {
-        match self {
+        match *self {
             SettingOrPreset::Setting(s) => s.name,
             SettingOrPreset::Preset(p) => p.name,
         }
@@ -248,14 +248,14 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
             fmt.indent(|fmt| {
                 fmt.line(&format!("name: \"{}\",", setting.name));
                 fmt.line(&format!("offset: {},", setting.byte_offset));
-                match &setting.specific {
+                match setting.specific {
                     SpecificSetting::Bool(BoolSetting { bit_offset, .. }) => {
                         fmt.line(&format!(
                             "detail: detail::Detail::Bool {{ bit: {} }},",
                             bit_offset
                         ));
                     }
-                    SpecificSetting::Enum(values) => {
+                    SpecificSetting::Enum(ref values) => {
                         let offset = enum_table.add(values);
                         fmt.line(&format!(
                             "detail: detail::Detail::Enum {{ last: {}, enumerators: {} }},",
@@ -322,7 +322,7 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
     ));
     fmt.indent(|fmt| {
         for h in &hash_table {
-            match h {
+            match *h {
                 Some(setting_or_preset) => fmt.line(&format!(
                     "{},",
                     &descriptor_index_map
diff --git a/lib/codegen/meta/src/srcgen.rs b/lib/codegen/meta/src/srcgen.rs
index 36270b9c70..295567de80 100644
--- a/lib/codegen/meta/src/srcgen.rs
+++ b/lib/codegen/meta/src/srcgen.rs
@@ -122,7 +122,7 @@ impl Formatter {
     pub fn add_match(&mut self, m: Match) {
         self.line(&format!("match {} {{", m.expr));
         self.indent(|fmt| {
-            for ((fields, body), names) in m.arms.iter() {
+            for (&(ref fields, ref body), ref names) in m.arms.iter() {
                 // name { fields } | name { fields } => { body }
                 let conditions: Vec = names
                     .iter()
@@ -252,7 +252,7 @@ mod srcgen_tests {
     use super::Formatter;
     use super::Match;
 
-    fn from_raw_string(s: impl Into) -> Vec {
+    fn from_raw_string>(s: S) -> Vec {
         s.into()
             .trim()
             .split("\n")

From bee4ee4b6e9532a89ae67356fb50aa45e8be587d Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 13 Dec 2018 06:03:28 -0800
Subject: [PATCH 2266/3084] Clarify that cranelift-wasm is just one component.

For a complete WebAssembly implementation, see Wasmtime.
---
 lib/wasm/README.md | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/lib/wasm/README.md b/lib/wasm/README.md
index 981ab41423..a554f3ac2a 100644
--- a/lib/wasm/README.md
+++ b/lib/wasm/README.md
@@ -1,2 +1,8 @@
 This crate performs the translation from a wasm module in binary format to the
-in-memory form of the [Cranelift](https://crates.io/crates/cranelift) IR.
+in-memory form of the [Cranelift IR].
+
+If you're looking for a complete WebAssembly implementation that uses this
+library, see [Wasmtime].
+
+[Wasmtime]: https://github.com/CraneStation/wasmtime
+[Cranelift IR]: https://cranelift.readthedocs.io/en/latest/ir.html

From c9666381f6493ee760f7dfcdff24c4fad79dc591 Mon Sep 17 00:00:00 2001
From: Vincent Esche 
Date: Thu, 13 Dec 2018 14:17:18 +0100
Subject: [PATCH 2267/3084] =?UTF-8?q?Fixed=20links=20in=20`=E2=80=A6/docs/?=
 =?UTF-8?q?compare-llvm.rst`?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

---
 cranelift/docs/compare-llvm.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/docs/compare-llvm.rst b/cranelift/docs/compare-llvm.rst
index e23f9f7890..ad0e71eaf4 100644
--- a/cranelift/docs/compare-llvm.rst
+++ b/cranelift/docs/compare-llvm.rst
@@ -109,7 +109,7 @@ low-level optimizations are sufficient.
 
 And, it removes some constraints in the mid-level optimize IR design space,
 making it more feasible to consider ideas such as using a
-[VSDG-based IR](https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-705.pdf).
+`VSDG-based IR `_.
 
 Program structure
 -----------------
@@ -118,7 +118,7 @@ In LLVM IR, the largest representable unit is the *module* which corresponds
 more or less to a C translation unit. It is a collection of functions and
 global variables that may contain references to external symbols too.
 
-In `Cranelift's IR` `_,
+In `Cranelift's IR `_,
 used by the `cranelift-codegen `_ crate,
 functions are self-contained, allowing them to be compiled independently. At
 this level, there is no explicit module that contains the functions.

From 46d9a3cd1a88bf4308d5155e066410466e0cc732 Mon Sep 17 00:00:00 2001
From: Nicholas Nethercote 
Date: Fri, 14 Dec 2018 09:34:23 +1100
Subject: [PATCH 2268/3084] Use an `FxHashMap` in `RegDiversions`.

Because it's hot and the number of entries can reach the 1000s, so
linear insertion and search is bad.

This reduces runtime for `sqlite` and `UE4Game-HTML5-Shipping` by 3-4%,
and a couple of other benchmarks (`sqlite`, `godot`, `clang`) by smaller
amounts.

It also increases runtime for `mono` and `tanks` by about 1%; this seems
to be due to incidental changes in which functions are inlined more than
algorithmic changes.
---
 lib/codegen/src/regalloc/coloring.rs  |  8 ++---
 lib/codegen/src/regalloc/diversion.rs | 52 ++++++++++++++-------------
 lib/codegen/src/verifier/locations.rs | 14 ++++----
 lib/entity/src/map.rs                 |  2 ++
 4 files changed, 40 insertions(+), 36 deletions(-)

diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs
index 8ce42737d6..d040140984 100644
--- a/lib/codegen/src/regalloc/coloring.rs
+++ b/lib/codegen/src/regalloc/coloring.rs
@@ -654,10 +654,10 @@ impl<'a> Context<'a> {
     where
         Pred: FnMut(&LiveRange, LiveRangeContext) -> bool,
     {
-        for rdiv in self.divert.all() {
+        for (&value, rdiv) in self.divert.iter() {
             let lr = self
                 .liveness
-                .get(rdiv.value)
+                .get(value)
                 .expect("Missing live range for diverted register");
             if pred(lr, self.liveness.context(&self.cur.func.layout)) {
                 if let Affinity::Reg(rci) = lr.affinity {
@@ -665,7 +665,7 @@ impl<'a> Context<'a> {
                     // Stack diversions should not be possible here. The only live transiently
                     // during `shuffle_inputs()`.
                     self.solver.reassign_in(
-                        rdiv.value,
+                        value,
                         rc,
                         rdiv.to.unwrap_reg(),
                         rdiv.from.unwrap_reg(),
@@ -673,7 +673,7 @@ impl<'a> Context<'a> {
                 } else {
                     panic!(
                         "Diverted register {} with {} affinity",
-                        rdiv.value,
+                        value,
                         lr.affinity.display(&self.reginfo)
                     );
                 }
diff --git a/lib/codegen/src/regalloc/diversion.rs b/lib/codegen/src/regalloc/diversion.rs
index 3572acd543..090bf936c9 100644
--- a/lib/codegen/src/regalloc/diversion.rs
+++ b/lib/codegen/src/regalloc/diversion.rs
@@ -7,11 +7,12 @@
 //! These register diversions are local to an EBB. No values can be diverted when entering a new
 //! EBB.
 
+use fx::FxHashMap;
 use ir::{InstructionData, Opcode};
 use ir::{StackSlot, Value, ValueLoc, ValueLocations};
 use isa::{RegInfo, RegUnit};
+use std::collections::hash_map::{Entry, Iter};
 use std::fmt;
-use std::vec::Vec;
 
 /// A diversion of a value from its original location to a new register or stack location.
 ///
@@ -22,8 +23,6 @@ use std::vec::Vec;
 /// the current one.
 #[derive(Clone, Copy, Debug, PartialEq, Eq)]
 pub struct Diversion {
-    /// The value that is diverted.
-    pub value: Value,
     /// The original value location.
     pub from: ValueLoc,
     /// The current value location.
@@ -32,22 +31,22 @@ pub struct Diversion {
 
 impl Diversion {
     /// Make a new diversion.
-    pub fn new(value: Value, from: ValueLoc, to: ValueLoc) -> Self {
+    pub fn new(from: ValueLoc, to: ValueLoc) -> Self {
         debug_assert!(from.is_assigned() && to.is_assigned());
-        Self { value, from, to }
+        Self { from, to }
     }
 }
 
 /// Keep track of diversions in an EBB.
 pub struct RegDiversions {
-    current: Vec,
+    current: FxHashMap,
 }
 
 impl RegDiversions {
     /// Create a new empty diversion tracker.
     pub fn new() -> Self {
         Self {
-            current: Vec::new(),
+            current: FxHashMap::default(),
         }
     }
 
@@ -63,12 +62,12 @@ impl RegDiversions {
 
     /// Get the current diversion of `value`, if any.
     pub fn diversion(&self, value: Value) -> Option<&Diversion> {
-        self.current.iter().find(|d| d.value == value)
+        self.current.get(&value)
     }
 
     /// Get all current diversions.
-    pub fn all(&self) -> &[Diversion] {
-        self.current.as_slice()
+    pub fn iter(&self) -> Iter<'_, Value, Diversion> {
+        self.current.iter()
     }
 
     /// Get the current location for `value`. Fall back to the assignment map for non-diverted
@@ -95,15 +94,22 @@ impl RegDiversions {
     /// The `from` location must match an existing `to` location, if any.
     pub fn divert(&mut self, value: Value, from: ValueLoc, to: ValueLoc) {
         debug_assert!(from.is_assigned() && to.is_assigned());
-        if let Some(i) = self.current.iter().position(|d| d.value == value) {
-            debug_assert_eq!(self.current[i].to, from, "Bad regmove chain for {}", value);
-            if self.current[i].from != to {
-                self.current[i].to = to;
-            } else {
-                self.current.swap_remove(i);
+        match self.current.entry(value) {
+            Entry::Occupied(mut e) => {
+                // TODO: non-lexical lifetimes should allow removal of the scope and early return.
+                {
+                    let d = e.get_mut();
+                    debug_assert_eq!(d.to, from, "Bad regmove chain for {}", value);
+                    if d.from != to {
+                        d.to = to;
+                        return;
+                    }
+                }
+                e.remove();
+            }
+            Entry::Vacant(e) => {
+                e.insert(Diversion::new(from, to));
             }
-        } else {
-            self.current.push(Diversion::new(value, from, to));
         }
     }
 
@@ -154,10 +160,7 @@ impl RegDiversions {
     ///
     /// Returns the `to` location of the removed diversion.
     pub fn remove(&mut self, value: Value) -> Option {
-        self.current
-            .iter()
-            .position(|d| d.value == value)
-            .map(|i| self.current.swap_remove(i).to)
+        self.current.remove(&value).map(|d| d.to)
     }
 
     /// Return an object that can display the diversions.
@@ -172,11 +175,11 @@ pub struct DisplayDiversions<'a>(&'a RegDiversions, Option<&'a RegInfo>);
 impl<'a> fmt::Display for DisplayDiversions<'a> {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         write!(f, "{{")?;
-        for div in self.0.all() {
+        for (value, div) in self.0.iter() {
             write!(
                 f,
                 " {}: {} -> {}",
-                div.value,
+                value,
                 div.from.display(self.1),
                 div.to.display(self.1)
             )?
@@ -201,7 +204,6 @@ mod tests {
         assert_eq!(
             divs.diversion(v1),
             Some(&Diversion {
-                value: v1,
                 from: ValueLoc::Reg(10),
                 to: ValueLoc::Reg(12),
             })
diff --git a/lib/codegen/src/verifier/locations.rs b/lib/codegen/src/verifier/locations.rs
index 4962ef916b..f03a2bed37 100644
--- a/lib/codegen/src/verifier/locations.rs
+++ b/lib/codegen/src/verifier/locations.rs
@@ -318,14 +318,14 @@ impl<'a> LocationVerifier<'a> {
                 dfg.display_inst(inst, self.isa)
             ),
             SingleDest(ebb, _) => {
-                for d in divert.all() {
-                    let lr = &liveness[d.value];
+                for (&value, d) in divert.iter() {
+                    let lr = &liveness[value];
                     if lr.is_livein(ebb, liveness.context(&self.func.layout)) {
                         return fatal!(
                             errors,
                             inst,
                             "{} is diverted to {} and live in to {}",
-                            d.value,
+                            value,
                             d.to.display(&self.reginfo),
                             ebb
                         );
@@ -333,15 +333,15 @@ impl<'a> LocationVerifier<'a> {
                 }
             }
             Table(jt, ebb) => {
-                for d in divert.all() {
-                    let lr = &liveness[d.value];
+                for (&value, d) in divert.iter() {
+                    let lr = &liveness[value];
                     if let Some(ebb) = ebb {
                         if lr.is_livein(ebb, liveness.context(&self.func.layout)) {
                             return fatal!(
                                 errors,
                                 inst,
                                 "{} is diverted to {} and live in to {}",
-                                d.value,
+                                value,
                                 d.to.display(&self.reginfo),
                                 ebb
                             );
@@ -353,7 +353,7 @@ impl<'a> LocationVerifier<'a> {
                                 errors,
                                 inst,
                                 "{} is diverted to {} and live in to {}",
-                                d.value,
+                                value,
                                 d.to.display(&self.reginfo),
                                 ebb
                             );
diff --git a/lib/entity/src/map.rs b/lib/entity/src/map.rs
index dbec7b2f11..c7ced088f7 100644
--- a/lib/entity/src/map.rs
+++ b/lib/entity/src/map.rs
@@ -97,6 +97,7 @@ where
     }
 
     /// Resize the map to have `n` entries by adding default entries as needed.
+    #[inline]
     pub fn resize(&mut self, n: usize) {
         self.elems.resize(n, self.default.clone());
     }
@@ -125,6 +126,7 @@ where
     K: EntityRef,
     V: Clone,
 {
+    #[inline]
     fn index_mut(&mut self, k: K) -> &mut V {
         let i = k.index();
         if i >= self.elems.len() {

From 95118e7244cd3ae6bdbbfd33af41c77f48d1e36b Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Mon, 24 Dec 2018 15:02:38 +0100
Subject: [PATCH 2269/3084] Nicer panic when trying to insert instruction
 before calling switch_to_block

---
 lib/frontend/src/frontend.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 2e29db56c0..ad1e872172 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -367,7 +367,8 @@ impl<'a> FunctionBuilder<'a> {
     /// Returns an object with the [`InstBuilder`](../codegen/ir/builder/trait.InstBuilder.html)
     /// trait that allows to conveniently append an instruction to the current `Ebb` being built.
     pub fn ins<'short>(&'short mut self) -> FuncInstBuilder<'short, 'a> {
-        let ebb = self.position.ebb.unwrap();
+        let ebb = self.position.ebb
+            .expect("Please call switch_to_block before inserting instructions");
         FuncInstBuilder::new(self, ebb)
     }
 

From 998a7d2b6d086cc717df111e2fcb2828e39e6616 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 20 Dec 2018 15:02:10 -0800
Subject: [PATCH 2270/3084] Mention Wasmtime as a complete implementation of
 the Environment traits.

---
 cranelift/README.md           | 2 +-
 lib/wasm/src/environ/dummy.rs | 6 +++++-
 lib/wasm/src/environ/spec.rs  | 6 ++++++
 3 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index 800dc6cc02..82ac7912dd 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -21,7 +21,7 @@ implements a toy language.
 [SimpleJIT Demo]: https://github.com/CraneStation/simplejit-demo
 
 For an example of how to use Cranelift to run WebAssembly code, see
-[Wasmtime], which implements a standalone VM using Cranelift.
+[Wasmtime], which implements a standalone, embeddable, VM using Cranelift.
 
 [Wasmtime]: https://github.com/CraneStation/wasmtime
 
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index a31a750b56..fc7db3af38 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -1,5 +1,9 @@
 //! "Dummy" implementations of `ModuleEnvironment` and `FuncEnvironment` for testing
-//! wasm translation.
+//! wasm translation. For complete implementations of `ModuleEnvironment` and
+//! `FuncEnvironment`, see [wasmtime-environ] in [Wasmtime].
+//!
+//! [wasmtime-environ]: https://crates.io/crates/wasmtime-environ
+//! [Wasmtime]: https://github.com/CraneStation/wasmtime
 
 use cast;
 use cranelift_codegen::cursor::FuncCursor;
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 7aeeb43c92..9f3c2ba165 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -1,5 +1,11 @@
 //! All the runtime support necessary for the wasm to cranelift translation is formalized by the
 //! traits `FunctionEnvironment` and `ModuleEnvironment`.
+//!
+//! There are skeleton implementations of these traits in the `dummy` module, and complete
+//! implementations in [Wasmtime].
+//!
+//! [Wasmtime]: https://github.com/CraneStation/wasmtime
+
 use cranelift_codegen::cursor::FuncCursor;
 use cranelift_codegen::ir::immediates::Offset32;
 use cranelift_codegen::ir::{self, InstBuilder};

From 887451a2eb87d3f6aada3670964bbd074c3d0b1a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 24 Dec 2018 09:32:50 -0800
Subject: [PATCH 2271/3084] Fix formatting.

---
 lib/frontend/src/frontend.rs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index ad1e872172..74a3e5990b 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -367,7 +367,9 @@ impl<'a> FunctionBuilder<'a> {
     /// Returns an object with the [`InstBuilder`](../codegen/ir/builder/trait.InstBuilder.html)
     /// trait that allows to conveniently append an instruction to the current `Ebb` being built.
     pub fn ins<'short>(&'short mut self) -> FuncInstBuilder<'short, 'a> {
-        let ebb = self.position.ebb
+        let ebb = self
+            .position
+            .ebb
             .expect("Please call switch_to_block before inserting instructions");
         FuncInstBuilder::new(self, ebb)
     }

From 4224a95f0dfee4679123d2dab5e919de5062f3ef Mon Sep 17 00:00:00 2001
From: Bruce Mitchener 
Date: Mon, 24 Dec 2018 10:41:08 -0500
Subject: [PATCH 2272/3084] Update wasmparser to 0.23.

---
 lib/wasm/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index cf14471a40..64a4716981 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -10,7 +10,7 @@ readme = "README.md"
 keywords = ["webassembly", "wasm"]
 
 [dependencies]
-wasmparser = { version = "0.22.0", default-features = false }
+wasmparser = { version = "0.23.0", default-features = false }
 cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
 cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
 cranelift-frontend = { path = "../frontend", version = "0.26.0", default-features = false }

From 4f8753fa11289ba85c78752ffbf64f9c03e84d2b Mon Sep 17 00:00:00 2001
From: Bruce Mitchener 
Date: Mon, 24 Dec 2018 10:20:32 -0500
Subject: [PATCH 2273/3084] Fix typos.

---
 cranelift/docs/clif_domain.py                   |  4 ++--
 cranelift/filetests/isa/x86/binary32-float.clif |  4 ++--
 cranelift/filetests/isa/x86/binary64-float.clif |  4 ++--
 lib/bforest/src/path.rs                         |  2 +-
 lib/codegen/meta-python/cdsl/__init__.py        |  2 +-
 lib/codegen/meta-python/cdsl/ast.py             | 10 +++++-----
 lib/codegen/meta-python/cdsl/test_ti.py         |  2 +-
 lib/codegen/meta-python/cdsl/test_typevar.py    |  2 +-
 lib/codegen/meta-python/cdsl/ti.py              |  8 ++++----
 lib/codegen/meta-python/cdsl/xform.py           |  2 +-
 lib/codegen/meta-python/gen_legalizer.py        |  4 ++--
 lib/codegen/meta-python/semantics/smtlib.py     |  2 +-
 lib/codegen/src/binemit/mod.rs                  |  2 +-
 lib/codegen/src/dominator_tree.rs               |  2 +-
 lib/codegen/src/ir/builder.rs                   |  2 +-
 lib/codegen/src/ir/memflags.rs                  |  4 ++--
 lib/codegen/src/loop_analysis.rs                |  4 ++--
 lib/codegen/src/timing.rs                       |  2 +-
 lib/codegen/src/write.rs                        |  2 +-
 lib/entity/src/keys.rs                          |  4 ++--
 lib/faerie/src/backend.rs                       |  2 +-
 lib/frontend/src/frontend.rs                    |  2 +-
 lib/preopt/README.md                            |  2 +-
 lib/preopt/src/lib.rs                           |  2 +-
 lib/reader/src/parser.rs                        |  4 ++--
 lib/wasm/src/code_translator.rs                 |  2 +-
 26 files changed, 41 insertions(+), 41 deletions(-)

diff --git a/cranelift/docs/clif_domain.py b/cranelift/docs/clif_domain.py
index 80db764bd5..eb9de01575 100644
--- a/cranelift/docs/clif_domain.py
+++ b/cranelift/docs/clif_domain.py
@@ -42,7 +42,7 @@ class ClifObject(ObjectDescription):
 
     def add_target_and_index(self, name, sig, signode):
         """
-        Add ``name`` the the index.
+        Add ``name`` to the index.
 
         :param name: The object name returned by :func:`handle_signature`.
         :param sig: The signature text.
@@ -76,7 +76,7 @@ def parse_type(name, signode):
     """
     Parse a type with embedded type vars and append to signode.
 
-    Return a a string that can be compiled into a regular expression matching
+    Return a string that can be compiled into a regular expression matching
     the type.
     """
 
diff --git a/cranelift/filetests/isa/x86/binary32-float.clif b/cranelift/filetests/isa/x86/binary32-float.clif
index 53bbcbff8e..0b171388ae 100644
--- a/cranelift/filetests/isa/x86/binary32-float.clif
+++ b/cranelift/filetests/isa/x86/binary32-float.clif
@@ -193,7 +193,7 @@ ebb0:
     ; Comparisons.
     ;
     ; Only `supported_floatccs` are tested here. Others are handled by
-    ; legalization paterns.
+    ; legalization patterns.
 
     ; asm: ucomiss %xmm2, %xmm5
     ; asm: setnp %bl
@@ -434,7 +434,7 @@ ebb0:
     ; Comparisons.
     ;
     ; Only `supported_floatccs` are tested here. Others are handled by
-    ; legalization paterns.
+    ; legalization patterns.
 
     ; asm: ucomisd %xmm2, %xmm5
     ; asm: setnp %bl
diff --git a/cranelift/filetests/isa/x86/binary64-float.clif b/cranelift/filetests/isa/x86/binary64-float.clif
index 12beb1fc78..582f4e9c45 100644
--- a/cranelift/filetests/isa/x86/binary64-float.clif
+++ b/cranelift/filetests/isa/x86/binary64-float.clif
@@ -206,7 +206,7 @@ ebb0:
     ; Comparisons.
     ;
     ; Only `supported_floatccs` are tested here. Others are handled by
-    ; legalization paterns.
+    ; legalization patterns.
 
     ; asm: ucomiss %xmm10, %xmm5
     ; asm: setnp %bl
@@ -469,7 +469,7 @@ ebb0:
     ; Comparisons.
     ;
     ; Only `supported_floatccs` are tested here. Others are handled by
-    ; legalization paterns.
+    ; legalization patterns.
 
     ; asm: ucomisd %xmm10, %xmm5
     ; asm: setnp %bl
diff --git a/lib/bforest/src/path.rs b/lib/bforest/src/path.rs
index 196094c241..7ccb9ee6f5 100644
--- a/lib/bforest/src/path.rs
+++ b/lib/bforest/src/path.rs
@@ -476,7 +476,7 @@ impl Path {
         match status {
             Removed::Healthy => {}
             Removed::Rightmost => {
-                // The rightmost entry was removed from the curent node, so move the path so it
+                // The rightmost entry was removed from the current node, so move the path so it
                 // points at the first entry of the next node at this level.
                 debug_assert_eq!(
                     usize::from(self.entry[level]),
diff --git a/lib/codegen/meta-python/cdsl/__init__.py b/lib/codegen/meta-python/cdsl/__init__.py
index 43314a6f18..a0b5d4c13b 100644
--- a/lib/codegen/meta-python/cdsl/__init__.py
+++ b/lib/codegen/meta-python/cdsl/__init__.py
@@ -2,7 +2,7 @@
 Cranelift DSL classes.
 
 This module defines the classes that are used to define Cranelift instructions
-and other entitties.
+and other entities.
 """
 from __future__ import absolute_import
 import re
diff --git a/lib/codegen/meta-python/cdsl/ast.py b/lib/codegen/meta-python/cdsl/ast.py
index 75716d36a9..a524ba7cf6 100644
--- a/lib/codegen/meta-python/cdsl/ast.py
+++ b/lib/codegen/meta-python/cdsl/ast.py
@@ -2,7 +2,7 @@
 Abstract syntax trees.
 
 This module defines classes that can be used to create abstract syntax trees
-for patern matching an rewriting of cranelift instructions.
+for pattern matching an rewriting of cranelift instructions.
 """
 from __future__ import absolute_import
 from . import instructions
@@ -79,7 +79,7 @@ class Def(object):
         # type: (VarAtomMap) -> Def
         """
         Return a copy of this Def with vars replaced with fresh variables,
-        in accordance with the map m. Update m as neccessary.
+        in accordance with the map m. Update m as necessary.
         """
         new_expr = self.expr.copy(m)
         new_defs = []  # type: List[Var]
@@ -423,7 +423,7 @@ class Apply(Expr):
         # type: (VarAtomMap) -> Apply
         """
         Return a copy of this Expr with vars replaced with fresh variables,
-        in accordance with the map m. Update m as neccessary.
+        in accordance with the map m. Update m as necessary.
         """
         return Apply(self.inst, tuple(map(lambda e: replace_var(e, m),
                                           self.args)))
@@ -441,7 +441,7 @@ class Apply(Expr):
     def substitution(self, other, s):
         # type: (Apply, VarAtomMap) -> Optional[VarAtomMap]
         """
-        If there is a substituion from Var->Atom that converts self to other,
+        If there is a substitution from Var->Atom that converts self to other,
         return it, otherwise return None. Note that this is strictly weaker
         than unification (see TestXForm.test_subst_enum_bad_var_const for
         example).
@@ -513,7 +513,7 @@ class ConstantInt(Literal):
     A value of an integer immediate operand.
 
     Immediate operands like `imm64` or `offset32` can be specified in AST
-    expressions using the call syntax: `imm64(5)` which greates a `ConstantInt`
+    expressions using the call syntax: `imm64(5)` which creates a `ConstantInt`
     node.
     """
 
diff --git a/lib/codegen/meta-python/cdsl/test_ti.py b/lib/codegen/meta-python/cdsl/test_ti.py
index 0e7a01023e..b88113a20f 100644
--- a/lib/codegen/meta-python/cdsl/test_ti.py
+++ b/lib/codegen/meta-python/cdsl/test_ti.py
@@ -502,7 +502,7 @@ class TestXForm(TypeCheckingBaseTest):
             # For any patterns where the type env includes constraints, at
             # least one of the "theoretically possible" concrete typings must
             # be prevented by the constraints. (i.e. we are not emitting
-            # unneccessary constraints).
+            # unnecessary constraints).
             # We check that by asserting that the number of concrete typings is
             # less than the number of all possible free typevar assignments
             if (len(xform.ti.constraints) > 0):
diff --git a/lib/codegen/meta-python/cdsl/test_typevar.py b/lib/codegen/meta-python/cdsl/test_typevar.py
index dca5ba1b48..48806cc4aa 100644
--- a/lib/codegen/meta-python/cdsl/test_typevar.py
+++ b/lib/codegen/meta-python/cdsl/test_typevar.py
@@ -233,7 +233,7 @@ class TestTypeVar(TestCase):
             intersect = ts1.copy()
             intersect &= ts2
 
-            # Propagate instersections backward
+            # Propagate intersections backward
             ts1_src = reduce(lambda ts, func:   ts.preimage(func),
                              reversed(i1),
                              intersect)
diff --git a/lib/codegen/meta-python/cdsl/ti.py b/lib/codegen/meta-python/cdsl/ti.py
index 64791ac1cd..26f01f9e6b 100644
--- a/lib/codegen/meta-python/cdsl/ti.py
+++ b/lib/codegen/meta-python/cdsl/ti.py
@@ -276,7 +276,7 @@ class SameWidth(TypeConstraint):
 
 class TypeEnv(object):
     """
-    Class encapsulating the neccessary book keeping for type inference.
+    Class encapsulating the necessary book keeping for type inference.
         :attribute type_map: dict holding the equivalence relations between tvs
         :attribute constraints: a list of accumulated constraints - tuples
                             (tv1, tv2)) where tv1 and tv2 are equal
@@ -331,7 +331,7 @@ class TypeEnv(object):
         """
         Record a that the free tv1 is part of the same equivalence class as
         tv2. The canonical representative of the merged class is tv2's
-        cannonical representative.
+        canonical representative.
         """
         assert not tv1.is_derived
         assert self[tv1] == tv1
@@ -376,7 +376,7 @@ class TypeEnv(object):
         non-derived TVs implicitly get the lowest rank (0). Derived variables
         get their rank from their free typevar. Singletons have the highest
         rank. TVs associated with vars in a source pattern have a higher rank
-        than TVs associted with temporary vars.
+        than TVs associated with temporary vars.
         """
         default_rank = TypeEnv.RANK_INTERNAL if tv.singleton_type() is None \
             else TypeEnv.RANK_SINGLETON
@@ -837,7 +837,7 @@ def ti_def(definition, typ):
         fresh_formal_tvs = move_first(fresh_formal_tvs, idx)
         actual_tvs = move_first(actual_tvs, idx)
 
-    # Unify each actual typevar with the correpsonding fresh formal tv
+    # Unify each actual typevar with the corresponding fresh formal tv
     for (actual_tv, formal_tv) in zip(actual_tvs, fresh_formal_tvs):
         typ_or_err = unify(actual_tv, formal_tv, typ)
         err = get_error(typ_or_err)
diff --git a/lib/codegen/meta-python/cdsl/xform.py b/lib/codegen/meta-python/cdsl/xform.py
index 5b2ee24b55..0e4deb9910 100644
--- a/lib/codegen/meta-python/cdsl/xform.py
+++ b/lib/codegen/meta-python/cdsl/xform.py
@@ -51,7 +51,7 @@ class Rtl(object):
         # type: (VarAtomMap) -> Rtl
         """
         Return a copy of this rtl with all Vars substituted with copies or
-        according to m. Update m as neccessary.
+        according to m. Update m as necessary.
         """
         return Rtl(*[d.copy(m) for d in self.rtl])
 
diff --git a/lib/codegen/meta-python/gen_legalizer.py b/lib/codegen/meta-python/gen_legalizer.py
index c0b664d214..7e22f8377e 100644
--- a/lib/codegen/meta-python/gen_legalizer.py
+++ b/lib/codegen/meta-python/gen_legalizer.py
@@ -32,10 +32,10 @@ except ImportError:
 def get_runtime_typechecks(xform):
     # type: (XForm) -> List[TypeConstraint]
     """
-    Given a XForm build a list of runtime type checks neccessary to determine
+    Given a XForm build a list of runtime type checks necessary to determine
     if it applies. We have 2 types of runtime checks:
         1) typevar tv belongs to typeset T - needed for free tvs whose
-               typeset is constrainted by their use in the dst pattern
+               typeset is constrained by their use in the dst pattern
 
         2) tv1 == tv2 where tv1 and tv2 are derived TVs - caused by unification
                 of non-bijective functions
diff --git a/lib/codegen/meta-python/semantics/smtlib.py b/lib/codegen/meta-python/semantics/smtlib.py
index 40b0694d3b..9ae9fbfaa7 100644
--- a/lib/codegen/meta-python/semantics/smtlib.py
+++ b/lib/codegen/meta-python/semantics/smtlib.py
@@ -201,7 +201,7 @@ def equivalent(r1, r2, inp_m, out_m):
         assert isinstance(v2, Var)
         results_eq_exp.append(mk_eq(m1[v1], m2[v2]))
 
-    # Put the whole query toghether
+    # Put the whole query together
     return q1 + q2 + args_eq_exp + [Not(And(*results_eq_exp))]
 
 
diff --git a/lib/codegen/src/binemit/mod.rs b/lib/codegen/src/binemit/mod.rs
index 661297a9ee..9143af14e6 100644
--- a/lib/codegen/src/binemit/mod.rs
+++ b/lib/codegen/src/binemit/mod.rs
@@ -49,7 +49,7 @@ pub enum Reloc {
 
 impl fmt::Display for Reloc {
     /// Display trait implementation drops the arch, since its used in contexts where the arch is
-    /// already unambigious, e.g. clif syntax with isa specified. In other contexts, use Debug.
+    /// already unambiguous, e.g. clif syntax with isa specified. In other contexts, use Debug.
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
             Reloc::Abs4 => write!(f, "Abs4"),
diff --git a/lib/codegen/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs
index e25e11990c..c8c5132d73 100644
--- a/lib/codegen/src/dominator_tree.rs
+++ b/lib/codegen/src/dominator_tree.rs
@@ -505,7 +505,7 @@ impl DominatorTree {
 /// - An ordering of EBBs according to a dominator tree pre-order.
 /// - Constant time dominance checks at the EBB granularity.
 ///
-/// The information in this auxillary data structure is not easy to update when the control flow
+/// The information in this auxiliary data structure is not easy to update when the control flow
 /// graph changes, which is why it is kept separate.
 pub struct DominatorTreePreorder {
     nodes: SecondaryMap,
diff --git a/lib/codegen/src/ir/builder.rs b/lib/codegen/src/ir/builder.rs
index 4e3866ad17..ed5675eedf 100644
--- a/lib/codegen/src/ir/builder.rs
+++ b/lib/codegen/src/ir/builder.rs
@@ -162,7 +162,7 @@ where
         {
             let dfg = self.inserter.data_flow_graph_mut();
             inst = dfg.make_inst(data);
-            // Make an `Interator>`.
+            // Make an `Iterator>`.
             let ru = self.reuse.as_ref().iter().cloned();
             dfg.make_inst_results_reusing(inst, ctrl_typevar, ru);
         }
diff --git a/lib/codegen/src/ir/memflags.rs b/lib/codegen/src/ir/memflags.rs
index 6acbe167ac..1a1f48d015 100644
--- a/lib/codegen/src/ir/memflags.rs
+++ b/lib/codegen/src/ir/memflags.rs
@@ -92,8 +92,8 @@ impl MemFlags {
 
     /// Test if the `readonly` flag is set.
     ///
-    /// Loads with this flag have no memory dependendies.
-    /// This results in indefined behavior if the dereferenced memory is mutated at any time
+    /// Loads with this flag have no memory dependencies.
+    /// This results in undefined behavior if the dereferenced memory is mutated at any time
     /// between when the function is called and when it is exited.
     pub fn readonly(self) -> bool {
         self.read(FlagBit::Readonly)
diff --git a/lib/codegen/src/loop_analysis.rs b/lib/codegen/src/loop_analysis.rs
index 21bf57f9aa..3f6490f5fb 100644
--- a/lib/codegen/src/loop_analysis.rs
+++ b/lib/codegen/src/loop_analysis.rs
@@ -118,7 +118,7 @@ impl LoopAnalysis {
         self.valid
     }
 
-    /// Clear all the data structures contanted in the loop analysis. This will leave the
+    /// Clear all the data structures contained in the loop analysis. This will leave the
     /// analysis in a similar state to a context returned by `new()` except that allocated
     /// memory be retained.
     pub fn clear(&mut self) {
@@ -191,7 +191,7 @@ impl LoopAnalysis {
                         let mut node_loop_parent_option = self.loops[node_loop].parent;
                         while let Some(node_loop_parent) = node_loop_parent_option.expand() {
                             if node_loop_parent == lp {
-                                // We have encounterd lp so we stop (already visited)
+                                // We have encountered lp so we stop (already visited)
                                 break;
                             } else {
                                 //
diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs
index 3edca7f73f..285e4102a1 100644
--- a/lib/codegen/src/timing.rs
+++ b/lib/codegen/src/timing.rs
@@ -157,7 +157,7 @@ mod details {
                     continue;
                 }
 
-                // Write a duration as secs.milis, trailing space.
+                // Write a duration as secs.millis, trailing space.
                 fn fmtdur(mut dur: Duration, f: &mut fmt::Formatter) -> fmt::Result {
                     // Round to nearest ms by adding 500us.
                     dur += Duration::new(0, 500_000);
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index b4f9b1c76d..4a70574caa 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -377,7 +377,7 @@ fn write_instruction(
     write_operands(w, &func.dfg, isa, inst)?;
     writeln!(w)?;
 
-    // Value aliases come out on lines after the instruction defining the referrent.
+    // Value aliases come out on lines after the instruction defining the referent.
     for r in func.dfg.inst_results(inst) {
         write_value_aliases(w, aliases, *r, indent)?;
     }
diff --git a/lib/entity/src/keys.rs b/lib/entity/src/keys.rs
index 5014dc3be5..e0165f21b7 100644
--- a/lib/entity/src/keys.rs
+++ b/lib/entity/src/keys.rs
@@ -1,7 +1,7 @@
 //! A double-ended iterator over entity references.
 //!
-//! When `std::iter::Step` is stablized, `Keys` could be implemented as a wrapper around
-//! `std::ops::Range`, but for now, we implment it manually.
+//! When `std::iter::Step` is stabilized, `Keys` could be implemented as a wrapper around
+//! `std::ops::Range`, but for now, we implement it manually.
 
 use std::marker::PhantomData;
 use EntityRef;
diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index 7028efbb01..164d722396 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -107,7 +107,7 @@ impl Backend for FaerieBackend {
     type CompiledFunction = FaerieCompiledFunction;
     type CompiledData = FaerieCompiledData;
 
-    // There's no need to return invidual artifacts; we're writing them into
+    // There's no need to return individual artifacts; we're writing them into
     // the output file instead.
     type FinalizedFunction = ();
     type FinalizedData = ();
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 74a3e5990b..d3c364b0d6 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -580,7 +580,7 @@ impl<'a> FunctionBuilder<'a> {
         self.ins().call(libc_memcpy, &[dest, src, size]);
     }
 
-    /// Optimised memcpy for small copys.
+    /// Optimised memcpy for small copies.
     pub fn emit_small_memcpy(
         &mut self,
         config: TargetFrontendConfig,
diff --git a/lib/preopt/README.md b/lib/preopt/README.md
index cb1560e718..1c4f04dc64 100644
--- a/lib/preopt/README.md
+++ b/lib/preopt/README.md
@@ -1 +1 @@
-This crate performes early-stage optimizations on [Cranelift](https://crates.io/crates/cranelift) IR.
+This crate performs early-stage optimizations on [Cranelift](https://crates.io/crates/cranelift) IR.
diff --git a/lib/preopt/src/lib.rs b/lib/preopt/src/lib.rs
index c896a0c01c..9f307c4dfd 100644
--- a/lib/preopt/src/lib.rs
+++ b/lib/preopt/src/lib.rs
@@ -1,4 +1,4 @@
-//! Performes early-stage optimizations on Cranelift IR.
+//! Performs early-stage optimizations on Cranelift IR.
 
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 6cae92c0b0..3bcac20aca 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -2034,7 +2034,7 @@ impl<'a> Parser<'a> {
                     opcode
                 );
             }
-        // Treat it as a syntax error to speficy a typevar on a non-polymorphic opcode.
+        // Treat it as a syntax error to specify a typevar on a non-polymorphic opcode.
         } else if ctrl_type != INVALID {
             return err!(self.loc, "{} does not take a typevar", opcode);
         }
@@ -2080,7 +2080,7 @@ impl<'a> Parser<'a> {
         Ok(args)
     }
 
-    // Parse an optional value list enclosed in parantheses.
+    // Parse an optional value list enclosed in parentheses.
     fn parse_opt_value_list(&mut self) -> ParseResult {
         if !self.optional(Token::LPar) {
             return Ok(VariableArgs::new());
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index 34aa37ef5e..c7683d8dab 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -897,7 +897,7 @@ pub fn translate_operator(
 #[cfg_attr(feature = "cargo-clippy", allow(unneeded_field_pattern))]
 /// Deals with a Wasm instruction located in an unreachable portion of the code. Most of them
 /// are dropped but special ones like `End` or `Else` signal the potential end of the unreachable
-/// portion so the translation state muts be updated accordingly.
+/// portion so the translation state must be updated accordingly.
 fn translate_unreachable_operator(
     op: &Operator,
     builder: &mut FunctionBuilder,

From e3db942b0cec38e4cf5af7d6ca6ce33d7f79d666 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 26 Dec 2018 11:35:55 +0100
Subject: [PATCH 2274/3084] Put default implementations of FuncWriter methods
 in a seperate method

This is useful when you want to write extra information
---
 lib/codegen/src/write.rs | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index 4a70574caa..6629f0f3be 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -41,6 +41,16 @@ pub trait FuncWriter {
         w: &mut Write,
         func: &Function,
         regs: Option<&RegInfo>,
+    ) -> Result {
+        self.super_preamble(w, func, regs)
+    }
+
+    /// Default impl of `write_preamble`
+    fn super_preamble(
+        &mut self,
+        w: &mut Write,
+        func: &Function,
+        regs: Option<&RegInfo>,
     ) -> Result {
         let mut any = false;
 
@@ -91,13 +101,24 @@ pub trait FuncWriter {
     }
 
     /// Write an entity definition defined in the preamble to `w`.
-    #[allow(unused_variables)]
     fn write_entity_definition(
         &mut self,
         w: &mut Write,
         func: &Function,
         entity: AnyEntity,
         value: &fmt::Display,
+    ) -> fmt::Result {
+        self.super_entity_definition(w, func, entity, value)
+    }
+
+    /// Default impl of `write_entity_definition`
+    #[allow(unused_variables)]
+    fn super_entity_definition(
+        &mut self,
+        w: &mut Write,
+        func: &Function,
+        entity: AnyEntity,
+        value: &fmt::Display,
     ) -> fmt::Result {
         writeln!(w, "    {} = {}", entity, value)
     }

From effe6c04e44426444feb3794e534ba53a65b9997 Mon Sep 17 00:00:00 2001
From: Muhammad Mominul Huque 
Date: Wed, 26 Dec 2018 23:49:05 +0600
Subject: [PATCH 2275/3084] Update to Rust 2018 edition (#632)

* initial cargo fix run

* Upgrade cranelift-entity crate

* Upgrade bforest crate

* Upgrade the codegen crate

* Upgrade the faerie crate

* Upgrade the filetests crate

* Upgrade the codegen-meta crate

* Upgrade the frontend crate

* Upgrade the cranelift-module crate

* Upgrade the cranelift-native crate

* Upgrade the cranelift-preopt crate

* Upgrade the cranelift-reader crate

* Upgrade the cranelift-serde crate

* Upgrade the cranelift-simplejit crate

* Upgrade the cranelift or cranelift-umbrella crate

* Upgrade the cranelift-wasm crate

* Upgrade cranelift-tools crate

* Use new import style on remaining files

* run format-all.sh

* run test-all.sh, update Readme and travis ci configuration
fixed an AssertionError also

* Remove deprecated functions
---
 .travis.yml                                   |  2 +-
 cranelift/Cargo.toml                          |  1 +
 cranelift/README.md                           |  2 +-
 cranelift/src/cat.rs                          |  4 +-
 cranelift/src/clif-util.rs                    | 17 +------
 cranelift/src/compile.rs                      |  3 +-
 cranelift/src/print_cfg.rs                    |  4 +-
 cranelift/src/wasm.rs                         |  2 +-
 cranelift/tests/filetests.rs                  |  2 -
 lib/bforest/Cargo.toml                        |  1 +
 lib/bforest/src/lib.rs                        |  4 +-
 lib/bforest/src/map.rs                        |  2 +-
 lib/bforest/src/pool.rs                       |  4 +-
 lib/bforest/src/set.rs                        |  2 +-
 lib/codegen/Cargo.toml                        |  1 +
 lib/codegen/build.rs                          |  4 +-
 lib/codegen/meta-python/cdsl/predicates.py    |  3 +-
 lib/codegen/meta-python/cdsl/xform.py         |  2 +-
 lib/codegen/meta-python/gen_encoding.py       | 16 +++---
 lib/codegen/meta-python/gen_instr.py          |  2 +-
 lib/codegen/meta-python/gen_legalizer.py      | 28 ++++++-----
 lib/codegen/meta-python/gen_settings.py       |  6 +--
 lib/codegen/meta-python/isa/x86/recipes.py    |  8 +--
 lib/codegen/meta-python/test_gen_legalizer.py |  4 +-
 lib/codegen/meta/Cargo.toml                   |  1 +
 lib/codegen/meta/src/base/settings.rs         |  2 +-
 lib/codegen/meta/src/cdsl/regs.rs             |  1 +
 lib/codegen/meta/src/cdsl/types.rs            |  2 +-
 lib/codegen/meta/src/gen_registers.rs         |  8 +--
 lib/codegen/meta/src/gen_settings.rs          | 18 ++++---
 lib/codegen/meta/src/gen_types.rs             |  6 +--
 lib/codegen/meta/src/isa/arm32/mod.rs         |  6 +--
 lib/codegen/meta/src/isa/arm64/mod.rs         |  6 +--
 lib/codegen/meta/src/isa/mod.rs               |  4 +-
 lib/codegen/meta/src/isa/riscv/mod.rs         |  6 +--
 lib/codegen/meta/src/isa/x86/mod.rs           |  6 +--
 lib/codegen/meta/src/lib.rs                   |  3 --
 lib/codegen/meta/src/srcgen.rs                | 10 ++--
 lib/codegen/src/abi.rs                        |  6 +--
 lib/codegen/src/binemit/memorysink.rs         | 10 ++--
 lib/codegen/src/binemit/mod.rs                | 20 ++++----
 lib/codegen/src/binemit/relaxation.rs         | 17 ++++---
 lib/codegen/src/binemit/shrink.rs             | 11 ++--
 lib/codegen/src/cfg_printer.rs                |  6 +--
 lib/codegen/src/context.rs                    | 38 +++++++-------
 lib/codegen/src/cursor.rs                     |  4 +-
 lib/codegen/src/dce.rs                        | 12 ++---
 lib/codegen/src/dominator_tree.rs             | 24 ++++-----
 lib/codegen/src/flowgraph.rs                  | 14 +++---
 lib/codegen/src/ir/builder.rs                 | 18 +++----
 lib/codegen/src/ir/dfg.rs                     | 30 +++++------
 lib/codegen/src/ir/entities.rs                |  3 +-
 lib/codegen/src/ir/extfunc.rs                 |  6 +--
 lib/codegen/src/ir/extname.rs                 |  4 +-
 lib/codegen/src/ir/function.rs                | 20 ++++----
 lib/codegen/src/ir/globalvalue.rs             |  6 +--
 lib/codegen/src/ir/heap.rs                    |  4 +-
 lib/codegen/src/ir/instructions.rs            | 18 +++----
 lib/codegen/src/ir/jumptable.rs               |  6 +--
 lib/codegen/src/ir/layout.rs                  | 17 ++++---
 lib/codegen/src/ir/libcall.rs                 |  4 +-
 lib/codegen/src/ir/mod.rs                     | 50 ++++++++++---------
 lib/codegen/src/ir/progpoint.rs               |  8 +--
 lib/codegen/src/ir/sourceloc.rs               |  2 +-
 lib/codegen/src/ir/stackslot.rs               | 10 ++--
 lib/codegen/src/ir/table.rs                   |  4 +-
 lib/codegen/src/ir/valueloc.rs                |  4 +-
 lib/codegen/src/isa/arm32/abi.rs              |  8 +--
 lib/codegen/src/isa/arm32/binemit.rs          |  6 +--
 lib/codegen/src/isa/arm32/enc_tables.rs       |  9 ++--
 lib/codegen/src/isa/arm32/mod.rs              | 14 +++---
 lib/codegen/src/isa/arm32/registers.rs        |  6 +--
 lib/codegen/src/isa/arm32/settings.rs         |  2 +-
 lib/codegen/src/isa/arm64/abi.rs              |  8 +--
 lib/codegen/src/isa/arm64/binemit.rs          |  6 +--
 lib/codegen/src/isa/arm64/enc_tables.rs       |  9 ++--
 lib/codegen/src/isa/arm64/mod.rs              | 14 +++---
 lib/codegen/src/isa/arm64/registers.rs        |  4 +-
 lib/codegen/src/isa/arm64/settings.rs         |  2 +-
 lib/codegen/src/isa/constraints.rs            |  8 +--
 lib/codegen/src/isa/enc_tables.rs             |  8 +--
 lib/codegen/src/isa/encoding.rs               |  8 +--
 lib/codegen/src/isa/mod.rs                    | 35 +++++++------
 lib/codegen/src/isa/registers.rs              |  2 +-
 lib/codegen/src/isa/riscv/abi.rs              |  8 +--
 lib/codegen/src/isa/riscv/binemit.rs          | 10 ++--
 lib/codegen/src/isa/riscv/enc_tables.rs       | 10 ++--
 lib/codegen/src/isa/riscv/mod.rs              | 24 ++++-----
 lib/codegen/src/isa/riscv/registers.rs        |  4 +-
 lib/codegen/src/isa/riscv/settings.rs         |  4 +-
 lib/codegen/src/isa/stack.rs                  |  4 +-
 lib/codegen/src/isa/x86/abi.rs                | 22 ++++----
 lib/codegen/src/isa/x86/binemit.rs            | 14 +++---
 lib/codegen/src/isa/x86/enc_tables.rs         | 44 ++++++++--------
 lib/codegen/src/isa/x86/mod.rs                | 18 +++----
 lib/codegen/src/isa/x86/registers.rs          |  4 +-
 lib/codegen/src/isa/x86/settings.rs           |  4 +-
 lib/codegen/src/legalizer/boundary.rs         | 15 +++---
 lib/codegen/src/legalizer/call.rs             |  8 +--
 lib/codegen/src/legalizer/globalvalue.rs      |  8 +--
 lib/codegen/src/legalizer/heap.rs             | 10 ++--
 lib/codegen/src/legalizer/libcall.rs          |  8 +--
 lib/codegen/src/legalizer/mod.rs              | 18 +++----
 lib/codegen/src/legalizer/split.rs            |  6 +--
 lib/codegen/src/legalizer/table.rs            | 12 ++---
 lib/codegen/src/lib.rs                        | 25 +++-------
 lib/codegen/src/licm.rs                       | 18 +++----
 lib/codegen/src/loop_analysis.rs              | 25 +++++-----
 lib/codegen/src/nan_canonicalization.rs       | 14 +++---
 lib/codegen/src/postopt.rs                    | 16 +++---
 lib/codegen/src/predicates.rs                 |  4 +-
 lib/codegen/src/print_errors.rs               | 16 +++---
 lib/codegen/src/regalloc/affinity.rs          |  4 +-
 lib/codegen/src/regalloc/coalescing.rs        | 25 +++++-----
 lib/codegen/src/regalloc/coloring.rs          | 35 ++++++-------
 lib/codegen/src/regalloc/context.rs           | 32 ++++++------
 lib/codegen/src/regalloc/diversion.rs         | 12 ++---
 .../src/regalloc/live_value_tracker.rs        | 16 +++---
 lib/codegen/src/regalloc/liveness.rs          | 16 +++---
 lib/codegen/src/regalloc/liverange.rs         | 16 +++---
 lib/codegen/src/regalloc/pressure.rs          | 14 +++---
 lib/codegen/src/regalloc/register_set.rs      |  4 +-
 lib/codegen/src/regalloc/reload.rs            | 25 +++++-----
 lib/codegen/src/regalloc/solver.rs            | 25 +++++-----
 lib/codegen/src/regalloc/spilling.rs          | 25 +++++-----
 lib/codegen/src/regalloc/virtregs.rs          | 19 +++----
 lib/codegen/src/result.rs                     |  3 +-
 lib/codegen/src/scoped_hash_map.rs            |  2 +-
 lib/codegen/src/settings.rs                   |  7 +--
 lib/codegen/src/simple_gvn.rs                 | 12 ++---
 lib/codegen/src/simple_preopt.rs              | 18 +++----
 lib/codegen/src/stack_layout.rs               | 14 +++---
 lib/codegen/src/timing.rs                     |  1 +
 lib/codegen/src/topo_order.rs                 | 14 +++---
 lib/codegen/src/unreachable_code.rs           | 11 ++--
 lib/codegen/src/verifier/cssa.rs              | 16 +++---
 lib/codegen/src/verifier/flags.rs             | 16 +++---
 lib/codegen/src/verifier/liveness.rs          | 16 +++---
 lib/codegen/src/verifier/locations.rs         | 14 +++---
 lib/codegen/src/verifier/mod.rs               | 39 ++++++++-------
 lib/codegen/src/write.rs                      | 20 ++++----
 lib/entity/Cargo.toml                         |  1 +
 lib/entity/src/boxed_slice.rs                 |  8 +--
 lib/entity/src/iter.rs                        |  2 +-
 lib/entity/src/keys.rs                        |  2 +-
 lib/entity/src/lib.rs                         |  2 +-
 lib/entity/src/list.rs                        |  4 +-
 lib/entity/src/map.rs                         |  6 +--
 lib/entity/src/primary.rs                     |  8 +--
 lib/entity/src/set.rs                         |  4 +-
 lib/entity/src/sparse.rs                      |  6 +--
 lib/faerie/Cargo.toml                         |  1 +
 lib/faerie/src/backend.rs                     |  4 +-
 lib/faerie/src/lib.rs                         | 11 +---
 lib/filetests/Cargo.toml                      |  1 +
 lib/filetests/src/concurrent.rs               |  5 +-
 lib/filetests/src/lib.rs                      | 11 +---
 lib/filetests/src/match_directive.rs          |  2 +-
 lib/filetests/src/runner.rs                   |  6 +--
 lib/filetests/src/runone.rs                   |  5 +-
 lib/filetests/src/test_binemit.rs             |  4 +-
 lib/filetests/src/test_cat.rs                 |  2 +-
 lib/filetests/src/test_compile.rs             |  3 +-
 lib/filetests/src/test_dce.rs                 |  2 +-
 lib/filetests/src/test_domtree.rs             |  4 +-
 lib/filetests/src/test_legalizer.rs           |  2 +-
 lib/filetests/src/test_licm.rs                |  2 +-
 lib/filetests/src/test_postopt.rs             |  2 +-
 lib/filetests/src/test_preopt.rs              |  2 +-
 lib/filetests/src/test_print_cfg.rs           |  2 +-
 lib/filetests/src/test_regalloc.rs            |  2 +-
 lib/filetests/src/test_shrink.rs              |  2 +-
 lib/filetests/src/test_simple_gvn.rs          |  2 +-
 lib/filetests/src/test_simple_preopt.rs       |  2 +-
 lib/filetests/src/test_verifier.rs            |  4 +-
 lib/frontend/Cargo.toml                       |  1 +
 lib/frontend/src/frontend.rs                  |  8 +--
 lib/frontend/src/lib.rs                       | 11 ++--
 lib/frontend/src/ssa.rs                       |  6 +--
 lib/frontend/src/switch.rs                    |  5 +-
 lib/module/Cargo.toml                         |  1 +
 lib/module/src/backend.rs                     | 10 ++--
 lib/module/src/lib.rs                         | 14 ++----
 lib/module/src/module.rs                      |  8 +--
 lib/native/Cargo.toml                         |  1 +
 lib/native/src/lib.rs                         |  5 --
 lib/preopt/Cargo.toml                         |  1 +
 lib/preopt/src/lib.rs                         |  3 --
 lib/reader/Cargo.toml                         |  1 +
 lib/reader/src/isaspec.rs                     |  4 +-
 lib/reader/src/lexer.rs                       |  4 +-
 lib/reader/src/lib.rs                         | 15 +++---
 lib/reader/src/parser.rs                      | 20 ++++----
 lib/reader/src/sourcemap.rs                   |  6 +--
 lib/reader/src/testfile.rs                    |  8 +--
 lib/serde/Cargo.toml                          |  1 +
 lib/serde/src/clif-json.rs                    |  7 ---
 lib/serde/src/serde_clif_json.rs              |  1 +
 lib/simplejit/Cargo.toml                      |  1 +
 lib/simplejit/examples/simplejit-minimal.rs   |  4 --
 lib/simplejit/src/backend.rs                  |  2 +-
 lib/simplejit/src/lib.rs                      | 13 +----
 lib/simplejit/tests/basic.rs                  |  6 ---
 lib/umbrella/Cargo.toml                       |  1 +
 lib/umbrella/src/lib.rs                       | 22 ++++----
 lib/wasm/Cargo.toml                           |  1 +
 lib/wasm/src/code_translator.rs               |  8 +--
 lib/wasm/src/environ/dummy.rs                 | 12 ++---
 lib/wasm/src/environ/mod.rs                   |  4 +-
 lib/wasm/src/environ/spec.rs                  |  7 +--
 lib/wasm/src/func_translator.rs               | 10 ++--
 lib/wasm/src/lib.rs                           | 24 ++-------
 lib/wasm/src/module_translator.rs             |  6 +--
 lib/wasm/src/sections_translator.rs           | 12 ++---
 lib/wasm/src/state.rs                         |  4 +-
 lib/wasm/src/translation_utils.rs             |  1 +
 lib/wasm/tests/wasm_testsuite.rs              |  7 +--
 217 files changed, 963 insertions(+), 1021 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index c80acf071c..bc772c38a2 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,7 +7,7 @@ language: rust
 rust:
     # The oldest version we currently support. See
     # CONTRIBUTING.md#rustc-version-support for details.
-    - 1.30.1
+    - 1.31.1
     - beta
     - nightly
 matrix:
diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index e2afd0799f..bbe81eaaed 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -7,6 +7,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
 repository = "https://github.com/CraneStation/cranelift"
 publish = false
+edition = "2018"
 
 [[bin]]
 name = "clif-util"
diff --git a/cranelift/README.md b/cranelift/README.md
index 82ac7912dd..5f579b0eb6 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -10,7 +10,7 @@ into executable machine code.
 [![Travis Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
 [![Appveyor Status](https://ci.appveyor.com/api/projects/status/oub7wrrb59utuv8x?svg=true)](https://ci.appveyor.com/project/CraneStation/cranelift)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
-![Minimum rustc 1.30](https://img.shields.io/badge/rustc-1.30+-green.svg)
+![Minimum rustc 1.31](https://img.shields.io/badge/rustc-1.31+-green.svg)
 
 For more information, see [the
 documentation](https://cranelift.readthedocs.io/en/latest/?badge=latest).
diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs
index cb824a6eb7..a98e509698 100644
--- a/cranelift/src/cat.rs
+++ b/cranelift/src/cat.rs
@@ -3,9 +3,9 @@
 //! Read a sequence of Cranelift IR files and print them again to stdout. This has the effect of
 //! normalizing formatting and removing comments.
 
+use crate::utils::read_to_string;
+use crate::CommandResult;
 use cranelift_reader::parse_functions;
-use utils::read_to_string;
-use CommandResult;
 
 pub fn run(files: &[String]) -> CommandResult {
     for (i, f) in files.into_iter().enumerate() {
diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 1b9255db4b..4c53cc8828 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -13,28 +13,13 @@
     )
 )]
 
-extern crate file_per_thread_logger;
-#[macro_use]
-extern crate cfg_if;
-#[cfg(feature = "disas")]
-extern crate capstone;
-extern crate clap;
-extern crate cranelift_codegen;
-#[cfg(feature = "wasm")]
-extern crate cranelift_entity;
-extern crate cranelift_filetests;
-extern crate cranelift_reader;
-extern crate pretty_env_logger;
+use cfg_if::cfg_if;
 
 cfg_if! {
     if #[cfg(feature = "wasm")] {
-        extern crate cranelift_wasm;
-        extern crate term;
-        extern crate wabt;
         mod wasm;
     }
 }
-extern crate target_lexicon;
 
 use clap::{App, Arg, SubCommand};
 use cranelift_codegen::dbg::LOG_FILENAME_PREFIX;
diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs
index 0babf6f83e..50b21faf15 100644
--- a/cranelift/src/compile.rs
+++ b/cranelift/src/compile.rs
@@ -1,5 +1,7 @@
 //! CLI tool to read Cranelift IR files and compile them into native code.
 
+use crate::utils::{parse_sets_and_triple, read_to_string};
+use cfg_if::cfg_if;
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_codegen::settings::FlagsOrIsa;
@@ -9,7 +11,6 @@ use cranelift_codegen::{binemit, ir};
 use cranelift_reader::parse_test;
 use std::path::Path;
 use std::path::PathBuf;
-use utils::{parse_sets_and_triple, read_to_string};
 
 struct PrintRelocs {
     flag_print: bool,
diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs
index 6d16c77262..30d91b5862 100644
--- a/cranelift/src/print_cfg.rs
+++ b/cranelift/src/print_cfg.rs
@@ -3,10 +3,10 @@
 //! Read a series of Cranelift IR files and print their control flow graphs
 //! in graphviz format.
 
+use crate::utils::read_to_string;
+use crate::CommandResult;
 use cranelift_codegen::cfg_printer::CFGPrinter;
 use cranelift_reader::parse_functions;
-use utils::read_to_string;
-use CommandResult;
 
 pub fn run(files: &[String]) -> CommandResult {
     for (i, f) in files.into_iter().enumerate() {
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 9e22279751..9ffea3adca 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -18,7 +18,7 @@ use cranelift_wasm::{
 use std::path::Path;
 use std::path::PathBuf;
 use term;
-use utils::{parse_sets_and_triple, read_to_end};
+use crate::utils::{parse_sets_and_triple, read_to_end};
 use wabt::wat2wasm;
 
 macro_rules! vprintln {
diff --git a/cranelift/tests/filetests.rs b/cranelift/tests/filetests.rs
index 89646a2d3d..a633461109 100644
--- a/cranelift/tests/filetests.rs
+++ b/cranelift/tests/filetests.rs
@@ -1,5 +1,3 @@
-extern crate cranelift_filetests;
-
 #[test]
 fn filetests() {
     // Run all the filetests in the following directories.
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index c2da583ccc..0181e32a83 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -9,6 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 categories = ["no-std"]
 readme = "README.md"
 keywords = ["btree", "forest", "set", "map"]
+edition = "2018"
 
 [dependencies]
 cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
diff --git a/lib/bforest/src/lib.rs b/lib/bforest/src/lib.rs
index 4b478f189c..ee9213b00a 100644
--- a/lib/bforest/src/lib.rs
+++ b/lib/bforest/src/lib.rs
@@ -48,7 +48,7 @@ mod std {
 
 #[macro_use]
 extern crate cranelift_entity as entity;
-use entity::packed_option;
+use crate::entity::packed_option;
 
 use std::borrow::BorrowMut;
 use std::cmp::Ordering;
@@ -157,7 +157,7 @@ fn slice_shift(s: &mut [T], n: usize) {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use entity::EntityRef;
+    use crate::entity::EntityRef;
 
     /// An opaque reference to an extended basic block in a function.
     #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
diff --git a/lib/bforest/src/map.rs b/lib/bforest/src/map.rs
index 9b1c78f906..a809282833 100644
--- a/lib/bforest/src/map.rs
+++ b/lib/bforest/src/map.rs
@@ -1,7 +1,7 @@
 //! Forest of maps.
 
 use super::{Comparator, Forest, Node, NodeData, NodePool, Path, INNER_SIZE};
-use packed_option::PackedOption;
+use crate::packed_option::PackedOption;
 #[cfg(test)]
 use std::fmt;
 use std::marker::PhantomData;
diff --git a/lib/bforest/src/pool.rs b/lib/bforest/src/pool.rs
index 9330ee5502..cec5e40a7d 100644
--- a/lib/bforest/src/pool.rs
+++ b/lib/bforest/src/pool.rs
@@ -3,7 +3,7 @@
 #[cfg(test)]
 use super::Comparator;
 use super::{Forest, Node, NodeData};
-use entity::PrimaryMap;
+use crate::entity::PrimaryMap;
 #[cfg(test)]
 use std::fmt;
 use std::ops::{Index, IndexMut};
@@ -83,7 +83,7 @@ impl NodePool {
         NodeData: fmt::Display,
         F::Key: fmt::Display,
     {
-        use entity::SparseSet;
+        use crate::entity::SparseSet;
         use std::borrow::Borrow;
         use std::cmp::Ordering;
         use std::vec::Vec;
diff --git a/lib/bforest/src/set.rs b/lib/bforest/src/set.rs
index f73a6b06d3..0ca5d92457 100644
--- a/lib/bforest/src/set.rs
+++ b/lib/bforest/src/set.rs
@@ -1,7 +1,7 @@
 //! Forest of sets.
 
 use super::{Comparator, Forest, Node, NodeData, NodePool, Path, SetValue, INNER_SIZE};
-use packed_option::PackedOption;
+use crate::packed_option::PackedOption;
 #[cfg(test)]
 use std::fmt;
 use std::marker::PhantomData;
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 2ab25583cb..cddb573715 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -10,6 +10,7 @@ categories = ["no-std"]
 readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
 build = "build.rs"
+edition = "2018"
 
 [dependencies]
 cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
diff --git a/lib/codegen/build.rs b/lib/codegen/build.rs
index 122d441c45..e8a57cfc26 100644
--- a/lib/codegen/build.rs
+++ b/lib/codegen/build.rs
@@ -18,9 +18,9 @@
 // The build script expects to be run from the directory where this build.rs file lives. The
 // current directory is used to find the sources.
 
-extern crate cranelift_codegen_meta as meta;
+use cranelift_codegen_meta as meta;
 
-use meta::isa::Isa;
+use crate::meta::isa::Isa;
 use std::env;
 use std::process;
 use std::time::Instant;
diff --git a/lib/codegen/meta-python/cdsl/predicates.py b/lib/codegen/meta-python/cdsl/predicates.py
index c81173ad5c..0177c09fff 100644
--- a/lib/codegen/meta-python/cdsl/predicates.py
+++ b/lib/codegen/meta-python/cdsl/predicates.py
@@ -244,7 +244,8 @@ class FieldPredicate(object):
         """
         # Prepend `field` to the predicate function arguments.
         args = (self.field.rust_name(),) + tuple(map(str, self.args))
-        return '::predicates::{}({})'.format(self.function, ', '.join(args))
+        return 'crate::predicates::{}({})'\
+            .format(self.function, ', '.join(args))
 
 
 class IsEqual(FieldPredicate):
diff --git a/lib/codegen/meta-python/cdsl/xform.py b/lib/codegen/meta-python/cdsl/xform.py
index 0e4deb9910..27aa515f94 100644
--- a/lib/codegen/meta-python/cdsl/xform.py
+++ b/lib/codegen/meta-python/cdsl/xform.py
@@ -394,7 +394,7 @@ class XFormGroup(object):
             # table referring to it.
             return self.name
         else:
-            return '::legalizer::{}'.format(self.name)
+            return 'crate::legalizer::{}'.format(self.name)
 
     def legalize(self, src, dst):
         # type: (Union[Def, Apply], Rtl) -> None
diff --git a/lib/codegen/meta-python/gen_encoding.py b/lib/codegen/meta-python/gen_encoding.py
index 6c5d7367ed..5f70c97eca 100644
--- a/lib/codegen/meta-python/gen_encoding.py
+++ b/lib/codegen/meta-python/gen_encoding.py
@@ -110,7 +110,7 @@ def emit_instp(instp, fmt, has_func=False):
     fields = ', '.join(sorted(fnames))
 
     with fmt.indented(
-            'if let ir::InstructionData::{} {{ {}, .. }} = *inst {{'
+            'if let crate::ir::InstructionData::{} {{ {}, .. }} = *inst {{'
             .format(iform.name, fields), '}'):
         if has_type_check:
             # We could implement this if we need to.
@@ -132,7 +132,8 @@ def emit_inst_predicates(instps, fmt):
     for instp, number in instps.items():
         name = 'inst_predicate_{}'.format(number)
         with fmt.indented(
-                'fn {}(func: &ir::Function, inst: &ir::InstructionData)'
+                'fn {}(func: &crate::ir::Function, '
+                'inst: &crate::ir::InstructionData)'
                 '-> bool {{'.format(name), '}'):
             emit_instp(instp, fmt, has_func=True)
 
@@ -168,7 +169,7 @@ def emit_recipe_predicates(isa, fmt):
 
         # Generate the predicate function.
         with fmt.indented(
-                'fn {}({}: ::settings::PredicateView, '
+                'fn {}({}: crate::settings::PredicateView, '
                 '{}: &ir::InstructionData) -> bool {{'
                 .format(
                     name,
@@ -657,7 +658,8 @@ def emit_level2_hashtables(level2_hashtables, offt, level2_doc, fmt):
             if entry:
                 fmt.line(
                         'Level2Entry ' +
-                        '{{ opcode: Some(ir::Opcode::{}), offset: {:#08x} }},'
+                        '{{ opcode: Some(crate::ir::Opcode::{}), '
+                        'offset: {:#08x} }},'
                         .format(entry.inst.camel_name, entry.offset))
             else:
                 fmt.line(
@@ -682,15 +684,15 @@ def emit_level1_hashtable(cpumode, level1, offt, fmt):
             # Empty hash table entry. Include the default legalization action.
             if not level2:
                 fmt.format(
-                        'Level1Entry {{ ty: ir::types::INVALID, log2len: !0, '
-                        'offset: 0, legalize: {} }},',
+                        'Level1Entry {{ ty: crate::ir::types::INVALID, '
+                        'log2len: !0, offset: 0, legalize: {} }},',
                         level1.legalize_code)
                 continue
 
             if level2.ty is not None:
                 tyname = level2.ty.rust_name()
             else:
-                tyname = 'ir::types::INVALID'
+                tyname = 'crate::ir::types::INVALID'
 
             lcode = cpumode.isa.legalize_code(level2.legalize)
 
diff --git a/lib/codegen/meta-python/gen_instr.py b/lib/codegen/meta-python/gen_instr.py
index 2f5eeae93d..57be62a5d4 100644
--- a/lib/codegen/meta-python/gen_instr.py
+++ b/lib/codegen/meta-python/gen_instr.py
@@ -661,7 +661,7 @@ def gen_inst_builder(inst, fmt):
     # The controlling type variable will be inferred from the input values if
     # possible. Otherwise, it is the first method argument.
     if inst.is_polymorphic and not inst.use_typevar_operand:
-        args.append('{}: ir::Type'.format(inst.ctrl_typevar.name))
+        args.append('{}: crate::ir::Type'.format(inst.ctrl_typevar.name))
 
     tmpl_types = list()  # type: List[str]
     into_args = list()  # type: List[str]
diff --git a/lib/codegen/meta-python/gen_legalizer.py b/lib/codegen/meta-python/gen_legalizer.py
index 7e22f8377e..2f78d086f3 100644
--- a/lib/codegen/meta-python/gen_legalizer.py
+++ b/lib/codegen/meta-python/gen_legalizer.py
@@ -103,19 +103,21 @@ def emit_runtime_typecheck(check, fmt, type_sets):
 
         base_exp = build_derived_expr(tv.base)
         if (tv.derived_func == TypeVar.LANEOF):
-            return "{}.map(|t: ir::Type| t.lane_type())".format(base_exp)
+            return "{}.map(|t: crate::ir::Type| t.lane_type())"\
+                .format(base_exp)
         elif (tv.derived_func == TypeVar.ASBOOL):
-            return "{}.map(|t: ir::Type| t.as_bool())".format(base_exp)
+            return "{}.map(|t: crate::ir::Type| t.as_bool())".format(base_exp)
         elif (tv.derived_func == TypeVar.HALFWIDTH):
-            return "{}.and_then(|t: ir::Type| t.half_width())".format(base_exp)
+            return "{}.and_then(|t: crate::ir::Type| t.half_width())"\
+                .format(base_exp)
         elif (tv.derived_func == TypeVar.DOUBLEWIDTH):
-            return "{}.and_then(|t: ir::Type| t.double_width())"\
+            return "{}.and_then(|t: crate::ir::Type| t.double_width())"\
                 .format(base_exp)
         elif (tv.derived_func == TypeVar.HALFVECTOR):
-            return "{}.and_then(|t: ir::Type| t.half_vector())"\
+            return "{}.and_then(|t: crate::ir::Type| t.half_vector())"\
                 .format(base_exp)
         elif (tv.derived_func == TypeVar.DOUBLEVECTOR):
-            return "{}.and_then(|t: ir::Type| t.by(2))".format(base_exp)
+            return "{}.and_then(|t: crate::ir::Type| t.by(2))".format(base_exp)
         else:
             assert False, "Unknown derived function {}".format(tv.derived_func)
 
@@ -174,7 +176,7 @@ def unwrap_inst(iref, node, fmt):
     arg_names = tuple(
             arg.name if isinstance(arg, Var) else '_' for arg in expr.args)
     with fmt.indented(
-            'let ({}, predicate) = if let ir::InstructionData::{} {{'
+            'let ({}, predicate) = if let crate::ir::InstructionData::{} {{'
             .format(', '.join(map(str, arg_names)), iform.name), '};'):
         # Fields are encoded directly.
         for f in iform.imm_fields:
@@ -359,13 +361,13 @@ def gen_xform_group(xgrp, fmt, type_sets):
     fmt.doc_comment("Legalize `inst`.")
     fmt.line('#[allow(unused_variables,unused_assignments,non_snake_case)]')
     with fmt.indented('pub fn {}('.format(xgrp.name)):
-        fmt.line('inst: ir::Inst,')
-        fmt.line('func: &mut ir::Function,')
-        fmt.line('cfg: &mut ::flowgraph::ControlFlowGraph,')
-        fmt.line('isa: &::isa::TargetIsa,')
+        fmt.line('inst: crate::ir::Inst,')
+        fmt.line('func: &mut crate::ir::Function,')
+        fmt.line('cfg: &mut crate::flowgraph::ControlFlowGraph,')
+        fmt.line('isa: &crate::isa::TargetIsa,')
     with fmt.indented(') -> bool {', '}'):
-        fmt.line('use ir::InstBuilder;')
-        fmt.line('use cursor::{Cursor, FuncCursor};')
+        fmt.line('use crate::ir::InstBuilder;')
+        fmt.line('use crate::cursor::{Cursor, FuncCursor};')
         fmt.line('let mut pos = FuncCursor::new(func).at_inst(inst);')
         fmt.line('pos.use_srcloc(inst);')
 
diff --git a/lib/codegen/meta-python/gen_settings.py b/lib/codegen/meta-python/gen_settings.py
index 88d2a9e0cd..cb9d3615e3 100644
--- a/lib/codegen/meta-python/gen_settings.py
+++ b/lib/codegen/meta-python/gen_settings.py
@@ -117,10 +117,10 @@ def gen_getters(sgrp, fmt):
     with fmt.indented('impl Flags {', '}'):
         fmt.doc_comment('Get a view of the boolean predicates.')
         with fmt.indented(
-                'pub fn predicate_view(&self) -> ::settings::PredicateView {',
-                '}'):
+                'pub fn predicate_view(&self) -> '
+                'crate::settings::PredicateView {', '}'):
             fmt.format(
-                    '::settings::PredicateView::new(&self.bytes[{}..])',
+                    'crate::settings::PredicateView::new(&self.bytes[{}..])',
                     sgrp.boolean_offset)
         if sgrp.settings:
             fmt.doc_comment('Dynamic numbered predicate getter.')
diff --git a/lib/codegen/meta-python/isa/x86/recipes.py b/lib/codegen/meta-python/isa/x86/recipes.py
index 484f61ee6f..76544b0e28 100644
--- a/lib/codegen/meta-python/isa/x86/recipes.py
+++ b/lib/codegen/meta-python/isa/x86/recipes.py
@@ -1934,7 +1934,7 @@ icscc = TailRecipe(
         PUT_OP(bits, rex2(in_reg0, in_reg1), sink);
         modrm_rr(in_reg0, in_reg1, sink);
         // `setCC` instruction, no REX.
-        use ir::condcodes::IntCC::*;
+        use crate::ir::condcodes::IntCC::*;
         let setcc = match cond {
             Equal => 0x94,
             NotEqual => 0x95,
@@ -1962,7 +1962,7 @@ icscc_ib = TailRecipe(
         let imm: i64 = imm.into();
         sink.put1(imm as u8);
         // `setCC` instruction, no REX.
-        use ir::condcodes::IntCC::*;
+        use crate::ir::condcodes::IntCC::*;
         let setcc = match cond {
             Equal => 0x94,
             NotEqual => 0x95,
@@ -1990,7 +1990,7 @@ icscc_id = TailRecipe(
         let imm: i64 = imm.into();
         sink.put4(imm as u32);
         // `setCC` instruction, no REX.
-        use ir::condcodes::IntCC::*;
+        use crate::ir::condcodes::IntCC::*;
         let setcc = match cond {
             Equal => 0x94,
             NotEqual => 0x95,
@@ -2030,7 +2030,7 @@ fcscc = TailRecipe(
         PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
         modrm_rr(in_reg1, in_reg0, sink);
         // `setCC` instruction, no REX.
-        use ir::condcodes::FloatCC::*;
+        use crate::ir::condcodes::FloatCC::*;
         let setcc = match cond {
             Ordered                    => 0x9b, // EQ|LT|GT => setnp (P=0)
             Unordered                  => 0x9a, // UN       => setp  (P=1)
diff --git a/lib/codegen/meta-python/test_gen_legalizer.py b/lib/codegen/meta-python/test_gen_legalizer.py
index 3882bd0bf9..2b1dbf4aca 100644
--- a/lib/codegen/meta-python/test_gen_legalizer.py
+++ b/lib/codegen/meta-python/test_gen_legalizer.py
@@ -148,9 +148,9 @@ class TestRuntimeChecks(TestCase):
                 self.v5 << vselect(self.v1, self.v3, self.v4),
         )
         x = XForm(r, r)
-        tv2_exp = 'Some({}).map(|t: ir::Type| t.as_bool())'\
+        tv2_exp = 'Some({}).map(|t: crate::ir::Type| t.as_bool())'\
             .format(self.v2.get_typevar().name)
-        tv3_exp = 'Some({}).map(|t: ir::Type| t.as_bool())'\
+        tv3_exp = 'Some({}).map(|t: crate::ir::Type| t.as_bool())'\
             .format(self.v3.get_typevar().name)
 
         self.check_yo_check(
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 92e4f34f20..2b443cb52e 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -6,6 +6,7 @@ description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
+edition = "2018"
 
 [dependencies]
 cranelift-entity = { path = "../../entity", version = "0.26.0" }
diff --git a/lib/codegen/meta/src/base/settings.rs b/lib/codegen/meta/src/base/settings.rs
index c89a4e20ff..b792a72283 100644
--- a/lib/codegen/meta/src/base/settings.rs
+++ b/lib/codegen/meta/src/base/settings.rs
@@ -1,4 +1,4 @@
-use cdsl::settings::{SettingGroup, SettingGroupBuilder};
+use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
 
 pub fn generate() -> SettingGroup {
     let mut settings = SettingGroupBuilder::new("shared");
diff --git a/lib/codegen/meta/src/cdsl/regs.rs b/lib/codegen/meta/src/cdsl/regs.rs
index 4e8a34001b..11605de563 100644
--- a/lib/codegen/meta/src/cdsl/regs.rs
+++ b/lib/codegen/meta/src/cdsl/regs.rs
@@ -1,3 +1,4 @@
+use cranelift_entity::entity_impl;
 use cranelift_entity::EntityRef;
 
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
diff --git a/lib/codegen/meta/src/cdsl/types.rs b/lib/codegen/meta/src/cdsl/types.rs
index 9fb7754c3c..a4e6443e47 100644
--- a/lib/codegen/meta/src/cdsl/types.rs
+++ b/lib/codegen/meta/src/cdsl/types.rs
@@ -5,7 +5,7 @@
 
 use std::fmt;
 
-use base::types as base_types;
+use crate::base::types as base_types;
 
 // Numbering scheme for value types:
 //
diff --git a/lib/codegen/meta/src/gen_registers.rs b/lib/codegen/meta/src/gen_registers.rs
index 9234dc5c67..5fb326e84e 100644
--- a/lib/codegen/meta/src/gen_registers.rs
+++ b/lib/codegen/meta/src/gen_registers.rs
@@ -1,8 +1,8 @@
-use cdsl::isa::TargetIsa;
-use cdsl::regs::{RegBank, RegClass};
+use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::regs::{RegBank, RegClass};
+use crate::error;
+use crate::srcgen::Formatter;
 use cranelift_entity::EntityRef;
-use error;
-use srcgen::Formatter;
 
 fn gen_regbank(fmt: &mut Formatter, reg_bank: &RegBank) {
     let names = if reg_bank.names.len() > 0 {
diff --git a/lib/codegen/meta/src/gen_settings.rs b/lib/codegen/meta/src/gen_settings.rs
index 93fddce210..497c1cc2f1 100644
--- a/lib/codegen/meta/src/gen_settings.rs
+++ b/lib/codegen/meta/src/gen_settings.rs
@@ -1,12 +1,14 @@
-use base;
-use cdsl::camel_case;
-use cdsl::isa::TargetIsa;
-use cdsl::settings::{BoolSetting, Predicate, Preset, Setting, SettingGroup, SpecificSetting};
-use constant_hash::{generate_table, simple_hash};
-use error;
-use srcgen::{Formatter, Match};
+use crate::base;
+use crate::cdsl::camel_case;
+use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::settings::{
+    BoolSetting, Predicate, Preset, Setting, SettingGroup, SpecificSetting,
+};
+use crate::constant_hash::{generate_table, simple_hash};
+use crate::error;
+use crate::srcgen::{Formatter, Match};
+use crate::unique_table::UniqueTable;
 use std::collections::HashMap;
-use unique_table::UniqueTable;
 
 enum ParentGroup {
     None,
diff --git a/lib/codegen/meta/src/gen_types.rs b/lib/codegen/meta/src/gen_types.rs
index 8dbd647bee..5a9f9f8c5e 100644
--- a/lib/codegen/meta/src/gen_types.rs
+++ b/lib/codegen/meta/src/gen_types.rs
@@ -7,9 +7,9 @@
 //! This ensures that the metaprogram and the generated program see the same
 //! type numbering.
 
-use cdsl::types as cdsl_types;
-use error;
-use srcgen;
+use crate::cdsl::types as cdsl_types;
+use crate::error;
+use crate::srcgen;
 
 /// Emit a constant definition of a single value type.
 fn emit_type(ty: &cdsl_types::ValueType, fmt: &mut srcgen::Formatter) -> Result<(), error::Error> {
diff --git a/lib/codegen/meta/src/isa/arm32/mod.rs b/lib/codegen/meta/src/isa/arm32/mod.rs
index 1a4fe471d7..ad615dfad2 100644
--- a/lib/codegen/meta/src/isa/arm32/mod.rs
+++ b/lib/codegen/meta/src/isa/arm32/mod.rs
@@ -1,6 +1,6 @@
-use cdsl::isa::{TargetIsa, TargetIsaBuilder};
-use cdsl::regs::{RegBankBuilder, RegClassBuilder};
-use cdsl::settings::{SettingGroup, SettingGroupBuilder};
+use crate::cdsl::isa::{TargetIsa, TargetIsaBuilder};
+use crate::cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
 
 fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     let setting = SettingGroupBuilder::new("arm32");
diff --git a/lib/codegen/meta/src/isa/arm64/mod.rs b/lib/codegen/meta/src/isa/arm64/mod.rs
index 63f5fa661e..7abdd51ca3 100644
--- a/lib/codegen/meta/src/isa/arm64/mod.rs
+++ b/lib/codegen/meta/src/isa/arm64/mod.rs
@@ -1,6 +1,6 @@
-use cdsl::isa::{TargetIsa, TargetIsaBuilder};
-use cdsl::regs::{RegBankBuilder, RegClassBuilder};
-use cdsl::settings::{SettingGroup, SettingGroupBuilder};
+use crate::cdsl::isa::{TargetIsa, TargetIsaBuilder};
+use crate::cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
 
 fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     let setting = SettingGroupBuilder::new("arm64");
diff --git a/lib/codegen/meta/src/isa/mod.rs b/lib/codegen/meta/src/isa/mod.rs
index f6d012fe78..eaa359b801 100644
--- a/lib/codegen/meta/src/isa/mod.rs
+++ b/lib/codegen/meta/src/isa/mod.rs
@@ -1,5 +1,5 @@
-use cdsl::isa::TargetIsa;
-use cdsl::settings::SettingGroup;
+use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::settings::SettingGroup;
 use std::fmt;
 
 mod arm32;
diff --git a/lib/codegen/meta/src/isa/riscv/mod.rs b/lib/codegen/meta/src/isa/riscv/mod.rs
index 9d82997e94..dbf0e16e8a 100644
--- a/lib/codegen/meta/src/isa/riscv/mod.rs
+++ b/lib/codegen/meta/src/isa/riscv/mod.rs
@@ -1,6 +1,6 @@
-use cdsl::isa::{TargetIsa, TargetIsaBuilder};
-use cdsl::regs::{RegBankBuilder, RegClassBuilder};
-use cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
+use crate::cdsl::isa::{TargetIsa, TargetIsaBuilder};
+use crate::cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 
 fn define_settings(shared: &SettingGroup) -> SettingGroup {
     let mut setting = SettingGroupBuilder::new("riscv");
diff --git a/lib/codegen/meta/src/isa/x86/mod.rs b/lib/codegen/meta/src/isa/x86/mod.rs
index 962446c9be..63087a2697 100644
--- a/lib/codegen/meta/src/isa/x86/mod.rs
+++ b/lib/codegen/meta/src/isa/x86/mod.rs
@@ -1,6 +1,6 @@
-use cdsl::isa::{TargetIsa, TargetIsaBuilder};
-use cdsl::regs::{RegBankBuilder, RegClassBuilder};
-use cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
+use crate::cdsl::isa::{TargetIsa, TargetIsaBuilder};
+use crate::cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 
 pub fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     let mut settings = SettingGroupBuilder::new("x86");
diff --git a/lib/codegen/meta/src/lib.rs b/lib/codegen/meta/src/lib.rs
index e393079f45..d35d8cddcc 100644
--- a/lib/codegen/meta/src/lib.rs
+++ b/lib/codegen/meta/src/lib.rs
@@ -1,6 +1,3 @@
-#[macro_use]
-extern crate cranelift_entity;
-
 #[macro_use]
 mod cdsl;
 
diff --git a/lib/codegen/meta/src/srcgen.rs b/lib/codegen/meta/src/srcgen.rs
index 295567de80..164c477997 100644
--- a/lib/codegen/meta/src/srcgen.rs
+++ b/lib/codegen/meta/src/srcgen.rs
@@ -9,7 +9,7 @@ use std::fs;
 use std::io::Write;
 use std::path;
 
-use error;
+use crate::error;
 
 static SHIFTWIDTH: usize = 4;
 
@@ -151,7 +151,7 @@ fn _indent(s: &str) -> Option {
     if s.is_empty() {
         None
     } else {
-        let t = s.trim_left();
+        let t = s.trim_start();
         Some(s.len() - t.len())
     }
 }
@@ -169,7 +169,7 @@ fn parse_multiline(s: &str) -> Vec {
         .iter()
         .skip(1)
         .filter(|l| !l.trim().is_empty())
-        .map(|l| l.len() - l.trim_left().len())
+        .map(|l| l.len() - l.trim_start().len())
         .min();
 
     // Strip off leading blank lines.
@@ -186,12 +186,12 @@ fn parse_multiline(s: &str) -> Vec {
         // Note that empty lines may have fewer than `indent` chars.
         lines_iter
             .map(|l| &l[cmp::min(indent, l.len())..])
-            .map(|l| l.trim_right())
+            .map(|l| l.trim_end())
             .map(|l| l.to_string())
             .collect::>()
     } else {
         lines_iter
-            .map(|l| l.trim_right())
+            .map(|l| l.trim_end())
             .map(|l| l.to_string())
             .collect::>()
     };
diff --git a/lib/codegen/src/abi.rs b/lib/codegen/src/abi.rs
index 1a86295b8b..954c40b643 100644
--- a/lib/codegen/src/abi.rs
+++ b/lib/codegen/src/abi.rs
@@ -3,7 +3,7 @@
 //! This module provides functions and data structures that are useful for implementing the
 //! `TargetIsa::legalize_signature()` method.
 
-use ir::{AbiParam, ArgumentExtension, ArgumentLoc, Type};
+use crate::ir::{AbiParam, ArgumentExtension, ArgumentLoc, Type};
 use std::cmp::Ordering;
 use std::vec::Vec;
 
@@ -182,8 +182,8 @@ pub fn legalize_abi_value(have: Type, arg: &AbiParam) -> ValueConversion {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use ir::types;
-    use ir::AbiParam;
+    use crate::ir::types;
+    use crate::ir::AbiParam;
 
     #[test]
     fn legalize() {
diff --git a/lib/codegen/src/binemit/memorysink.rs b/lib/codegen/src/binemit/memorysink.rs
index 1e8a6ab15f..7f6c62f451 100644
--- a/lib/codegen/src/binemit/memorysink.rs
+++ b/lib/codegen/src/binemit/memorysink.rs
@@ -15,7 +15,7 @@
 //! `CodeSink::put*` methods, so the performance impact of the virtual callbacks is less severe.
 
 use super::{Addend, CodeOffset, CodeSink, Reloc};
-use ir::{ExternalName, JumpTable, SourceLoc, TrapCode};
+use crate::ir::{ExternalName, JumpTable, SourceLoc, TrapCode};
 use std::ptr::write_unaligned;
 
 /// A `CodeSink` that writes binary machine code directly into memory.
@@ -57,13 +57,13 @@ impl<'a> MemoryCodeSink<'a> {
 /// A trait for receiving relocations for code that is emitted directly into memory.
 pub trait RelocSink {
     /// Add a relocation referencing an EBB at the current offset.
-    fn reloc_ebb(&mut self, CodeOffset, Reloc, CodeOffset);
+    fn reloc_ebb(&mut self, _: CodeOffset, _: Reloc, _: CodeOffset);
 
     /// Add a relocation referencing an external symbol at the current offset.
-    fn reloc_external(&mut self, CodeOffset, Reloc, &ExternalName, Addend);
+    fn reloc_external(&mut self, _: CodeOffset, _: Reloc, _: &ExternalName, _: Addend);
 
     /// Add a relocation referencing a jump table.
-    fn reloc_jt(&mut self, CodeOffset, Reloc, JumpTable);
+    fn reloc_jt(&mut self, _: CodeOffset, _: Reloc, _: JumpTable);
 }
 
 /// A trait for receiving trap codes and offsets.
@@ -72,7 +72,7 @@ pub trait RelocSink {
 /// [`NullTrapSink`](binemit/trait.TrapSink.html) implementation.
 pub trait TrapSink {
     /// Add trap information for a specific offset.
-    fn trap(&mut self, CodeOffset, SourceLoc, TrapCode);
+    fn trap(&mut self, _: CodeOffset, _: SourceLoc, _: TrapCode);
 }
 
 impl<'a> CodeSink for MemoryCodeSink<'a> {
diff --git a/lib/codegen/src/binemit/mod.rs b/lib/codegen/src/binemit/mod.rs
index 9143af14e6..c4ad46c7df 100644
--- a/lib/codegen/src/binemit/mod.rs
+++ b/lib/codegen/src/binemit/mod.rs
@@ -10,9 +10,9 @@ mod shrink;
 pub use self::memorysink::{MemoryCodeSink, NullTrapSink, RelocSink, TrapSink};
 pub use self::relaxation::relax_branches;
 pub use self::shrink::shrink_instructions;
-pub use regalloc::RegDiversions;
+pub use crate::regalloc::RegDiversions;
 
-use ir::{ExternalName, Function, Inst, JumpTable, SourceLoc, TrapCode};
+use crate::ir::{ExternalName, Function, Inst, JumpTable, SourceLoc, TrapCode};
 use std::fmt;
 
 /// Offset in bytes from the beginning of the function.
@@ -72,28 +72,28 @@ pub trait CodeSink {
     fn offset(&self) -> CodeOffset;
 
     /// Add 1 byte to the code section.
-    fn put1(&mut self, u8);
+    fn put1(&mut self, _: u8);
 
     /// Add 2 bytes to the code section.
-    fn put2(&mut self, u16);
+    fn put2(&mut self, _: u16);
 
     /// Add 4 bytes to the code section.
-    fn put4(&mut self, u32);
+    fn put4(&mut self, _: u32);
 
     /// Add 8 bytes to the code section.
-    fn put8(&mut self, u64);
+    fn put8(&mut self, _: u64);
 
     /// Add a relocation referencing an EBB at the current offset.
-    fn reloc_ebb(&mut self, Reloc, CodeOffset);
+    fn reloc_ebb(&mut self, _: Reloc, _: CodeOffset);
 
     /// Add a relocation referencing an external symbol plus the addend at the current offset.
-    fn reloc_external(&mut self, Reloc, &ExternalName, Addend);
+    fn reloc_external(&mut self, _: Reloc, _: &ExternalName, _: Addend);
 
     /// Add a relocation referencing a jump table.
-    fn reloc_jt(&mut self, Reloc, JumpTable);
+    fn reloc_jt(&mut self, _: Reloc, _: JumpTable);
 
     /// Add trap information for the current offset.
-    fn trap(&mut self, TrapCode, SourceLoc);
+    fn trap(&mut self, _: TrapCode, _: SourceLoc);
 
     /// Code output is complete, read-only data may follow.
     fn begin_rodata(&mut self);
diff --git a/lib/codegen/src/binemit/relaxation.rs b/lib/codegen/src/binemit/relaxation.rs
index f658fb70b1..2ef2167d2e 100644
--- a/lib/codegen/src/binemit/relaxation.rs
+++ b/lib/codegen/src/binemit/relaxation.rs
@@ -27,14 +27,15 @@
 //! ebb23:
 //! ```
 
-use binemit::CodeOffset;
-use cursor::{Cursor, FuncCursor};
-use ir::{Function, InstructionData, Opcode};
-use isa::{EncInfo, TargetIsa};
-use iterators::IteratorExtras;
-use regalloc::RegDiversions;
-use timing;
-use CodegenResult;
+use crate::binemit::CodeOffset;
+use crate::cursor::{Cursor, FuncCursor};
+use crate::ir::{Function, InstructionData, Opcode};
+use crate::isa::{EncInfo, TargetIsa};
+use crate::iterators::IteratorExtras;
+use crate::regalloc::RegDiversions;
+use crate::timing;
+use crate::CodegenResult;
+use log::debug;
 
 /// Relax branches and compute the final layout of EBB headers in `func`.
 ///
diff --git a/lib/codegen/src/binemit/shrink.rs b/lib/codegen/src/binemit/shrink.rs
index 92d2dd050f..b3cfdfa6d1 100644
--- a/lib/codegen/src/binemit/shrink.rs
+++ b/lib/codegen/src/binemit/shrink.rs
@@ -5,11 +5,12 @@
 //! flexibility. However, once register allocation is done, this is no longer important, and we
 //! can switch to smaller encodings when possible.
 
-use ir::instructions::InstructionData;
-use ir::Function;
-use isa::TargetIsa;
-use regalloc::RegDiversions;
-use timing;
+use crate::ir::instructions::InstructionData;
+use crate::ir::Function;
+use crate::isa::TargetIsa;
+use crate::regalloc::RegDiversions;
+use crate::timing;
+use log::debug;
 
 /// Pick the smallest valid encodings for instructions.
 pub fn shrink_instructions(func: &mut Function, isa: &TargetIsa) {
diff --git a/lib/codegen/src/cfg_printer.rs b/lib/codegen/src/cfg_printer.rs
index 382c83c658..c15ec134b9 100644
--- a/lib/codegen/src/cfg_printer.rs
+++ b/lib/codegen/src/cfg_printer.rs
@@ -2,9 +2,9 @@
 
 use std::fmt::{Display, Formatter, Result, Write};
 
-use flowgraph::{BasicBlock, ControlFlowGraph};
-use ir::instructions::BranchInfo;
-use ir::Function;
+use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::ir::instructions::BranchInfo;
+use crate::ir::Function;
 
 /// A utility for pretty-printing the CFG of a `Function`.
 pub struct CFGPrinter<'a> {
diff --git a/lib/codegen/src/context.rs b/lib/codegen/src/context.rs
index a0bd1e680d..a78553aa9e 100644
--- a/lib/codegen/src/context.rs
+++ b/lib/codegen/src/context.rs
@@ -9,28 +9,28 @@
 //! contexts concurrently. Typically, you would have one context per compilation thread and only a
 //! single ISA instance.
 
-use binemit::{
+use crate::binemit::{
     relax_branches, shrink_instructions, CodeOffset, MemoryCodeSink, RelocSink, TrapSink,
 };
-use dce::do_dce;
-use dominator_tree::DominatorTree;
-use flowgraph::ControlFlowGraph;
-use ir::Function;
-use isa::TargetIsa;
-use legalize_function;
-use licm::do_licm;
-use loop_analysis::LoopAnalysis;
-use nan_canonicalization::do_nan_canonicalization;
-use postopt::do_postopt;
-use regalloc;
-use result::CodegenResult;
-use settings::{FlagsOrIsa, OptLevel};
-use simple_gvn::do_simple_gvn;
-use simple_preopt::do_preopt;
+use crate::dce::do_dce;
+use crate::dominator_tree::DominatorTree;
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir::Function;
+use crate::isa::TargetIsa;
+use crate::legalize_function;
+use crate::licm::do_licm;
+use crate::loop_analysis::LoopAnalysis;
+use crate::nan_canonicalization::do_nan_canonicalization;
+use crate::postopt::do_postopt;
+use crate::regalloc;
+use crate::result::CodegenResult;
+use crate::settings::{FlagsOrIsa, OptLevel};
+use crate::simple_gvn::do_simple_gvn;
+use crate::simple_preopt::do_preopt;
+use crate::timing;
+use crate::unreachable_code::eliminate_unreachable_code;
+use crate::verifier::{verify_context, verify_locations, VerifierErrors, VerifierResult};
 use std::vec::Vec;
-use timing;
-use unreachable_code::eliminate_unreachable_code;
-use verifier::{verify_context, verify_locations, VerifierErrors, VerifierResult};
 
 /// Persistent data structures and compilation pipeline.
 pub struct Context {
diff --git a/lib/codegen/src/cursor.rs b/lib/codegen/src/cursor.rs
index daa23308f2..eae5fc0b1f 100644
--- a/lib/codegen/src/cursor.rs
+++ b/lib/codegen/src/cursor.rs
@@ -2,8 +2,8 @@
 //!
 //! This module defines cursor data types that can be used for inserting instructions.
 
-use ir;
-use isa::TargetIsa;
+use crate::ir;
+use crate::isa::TargetIsa;
 
 /// The possible positions of a cursor.
 #[derive(Clone, Copy, PartialEq, Eq, Debug)]
diff --git a/lib/codegen/src/dce.rs b/lib/codegen/src/dce.rs
index 2c5d5e2a32..d42c36870e 100644
--- a/lib/codegen/src/dce.rs
+++ b/lib/codegen/src/dce.rs
@@ -3,13 +3,13 @@
 //! Dead code here means instructions that have no side effects and have no
 //! result values used by other instructions.
 
-use cursor::{Cursor, FuncCursor};
-use dominator_tree::DominatorTree;
-use entity::EntityRef;
-use ir::instructions::InstructionData;
-use ir::{DataFlowGraph, Function, Inst, Opcode};
+use crate::cursor::{Cursor, FuncCursor};
+use crate::dominator_tree::DominatorTree;
+use crate::entity::EntityRef;
+use crate::ir::instructions::InstructionData;
+use crate::ir::{DataFlowGraph, Function, Inst, Opcode};
+use crate::timing;
 use std::vec::Vec;
-use timing;
 
 /// Test whether the given opcode is unsafe to even consider for DCE.
 fn trivially_unsafe_for_dce(opcode: Opcode) -> bool {
diff --git a/lib/codegen/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs
index c8c5132d73..9d20eab0d4 100644
--- a/lib/codegen/src/dominator_tree.rs
+++ b/lib/codegen/src/dominator_tree.rs
@@ -1,15 +1,15 @@
 //! A Dominator Tree represented as mappings of Ebbs to their immediate dominator.
 
-use entity::SecondaryMap;
-use flowgraph::{BasicBlock, ControlFlowGraph};
-use ir::instructions::BranchInfo;
-use ir::{Ebb, ExpandedProgramPoint, Function, Inst, Layout, ProgramOrder, Value};
-use packed_option::PackedOption;
+use crate::entity::SecondaryMap;
+use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::ir::instructions::BranchInfo;
+use crate::ir::{Ebb, ExpandedProgramPoint, Function, Inst, Layout, ProgramOrder, Value};
+use crate::packed_option::PackedOption;
+use crate::timing;
 use std::cmp;
 use std::cmp::Ordering;
 use std::mem;
 use std::vec::Vec;
-use timing;
 
 /// RPO numbers are not first assigned in a contiguous way but as multiples of STRIDE, to leave
 /// room for modifications of the dominator tree.
@@ -671,12 +671,12 @@ impl DominatorTreePreorder {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use cursor::{Cursor, FuncCursor};
-    use flowgraph::ControlFlowGraph;
-    use ir::types::*;
-    use ir::{Function, InstBuilder, TrapCode};
-    use settings;
-    use verifier::{verify_context, VerifierErrors};
+    use crate::cursor::{Cursor, FuncCursor};
+    use crate::flowgraph::ControlFlowGraph;
+    use crate::ir::types::*;
+    use crate::ir::{Function, InstBuilder, TrapCode};
+    use crate::settings;
+    use crate::verifier::{verify_context, VerifierErrors};
 
     #[test]
     fn empty() {
diff --git a/lib/codegen/src/flowgraph.rs b/lib/codegen/src/flowgraph.rs
index 3f61f6a852..4412e53599 100644
--- a/lib/codegen/src/flowgraph.rs
+++ b/lib/codegen/src/flowgraph.rs
@@ -23,12 +23,12 @@
 //! Here `Ebb1` and `Ebb2` would each have a single predecessor denoted as `(Ebb0, brz)`
 //! and `(Ebb0, jmp Ebb2)` respectively.
 
-use bforest;
-use entity::SecondaryMap;
-use ir::instructions::BranchInfo;
-use ir::{Ebb, Function, Inst};
+use crate::bforest;
+use crate::entity::SecondaryMap;
+use crate::ir::instructions::BranchInfo;
+use crate::ir::{Ebb, Function, Inst};
+use crate::timing;
 use std::mem;
-use timing;
 
 /// A basic block denoted by its enclosing Ebb and last instruction.
 #[derive(PartialEq, Eq)]
@@ -212,8 +212,8 @@ pub type SuccIter<'a> = bforest::SetIter<'a, Ebb>;
 #[cfg(test)]
 mod tests {
     use super::*;
-    use cursor::{Cursor, FuncCursor};
-    use ir::{types, Function, InstBuilder};
+    use crate::cursor::{Cursor, FuncCursor};
+    use crate::ir::{types, Function, InstBuilder};
     use std::vec::Vec;
 
     #[test]
diff --git a/lib/codegen/src/ir/builder.rs b/lib/codegen/src/ir/builder.rs
index ed5675eedf..7ed9a8e79e 100644
--- a/lib/codegen/src/ir/builder.rs
+++ b/lib/codegen/src/ir/builder.rs
@@ -3,11 +3,11 @@
 //! A `Builder` provides a convenient interface for inserting instructions into a Cranelift
 //! function. Many of its methods are generated from the meta language instruction definitions.
 
-use ir;
-use ir::types;
-use ir::{DataFlowGraph, InstructionData};
-use ir::{Inst, Opcode, Type, Value};
-use isa;
+use crate::ir;
+use crate::ir::types;
+use crate::ir::{DataFlowGraph, InstructionData};
+use crate::ir::{Inst, Opcode, Type, Value};
+use crate::isa;
 
 /// Base trait for instruction builders.
 ///
@@ -215,10 +215,10 @@ impl<'f> InstBuilderBase<'f> for ReplaceBuilder<'f> {
 
 #[cfg(test)]
 mod tests {
-    use cursor::{Cursor, FuncCursor};
-    use ir::condcodes::*;
-    use ir::types::*;
-    use ir::{Function, InstBuilder, ValueDef};
+    use crate::cursor::{Cursor, FuncCursor};
+    use crate::ir::condcodes::*;
+    use crate::ir::types::*;
+    use crate::ir::{Function, InstBuilder, ValueDef};
 
     #[test]
     fn types() {
diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs
index 905238275a..7d3e5c94ea 100644
--- a/lib/codegen/src/ir/dfg.rs
+++ b/lib/codegen/src/ir/dfg.rs
@@ -1,20 +1,20 @@
 //! Data flow graph tracking Instructions, Values, and EBBs.
 
-use entity::{self, PrimaryMap, SecondaryMap};
-use ir;
-use ir::builder::ReplaceBuilder;
-use ir::extfunc::ExtFuncData;
-use ir::instructions::{BranchInfo, CallInfo, InstructionData};
-use ir::types;
-use ir::{Ebb, FuncRef, Inst, SigRef, Signature, Type, Value, ValueList, ValueListPool};
-use isa::TargetIsa;
-use packed_option::ReservedValue;
+use crate::entity::{self, PrimaryMap, SecondaryMap};
+use crate::ir;
+use crate::ir::builder::ReplaceBuilder;
+use crate::ir::extfunc::ExtFuncData;
+use crate::ir::instructions::{BranchInfo, CallInfo, InstructionData};
+use crate::ir::types;
+use crate::ir::{Ebb, FuncRef, Inst, SigRef, Signature, Type, Value, ValueList, ValueListPool};
+use crate::isa::TargetIsa;
+use crate::packed_option::ReservedValue;
+use crate::write::write_operands;
 use std::fmt;
 use std::iter;
 use std::mem;
 use std::ops::{Index, IndexMut};
 use std::u16;
-use write::write_operands;
 
 /// A data flow graph defines all instructions and extended basic blocks in a function as well as
 /// the data flow dependencies between them. The DFG also tracks values which can be either
@@ -1063,9 +1063,9 @@ impl DataFlowGraph {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use cursor::{Cursor, FuncCursor};
-    use ir::types;
-    use ir::{Function, InstructionData, Opcode, TrapCode};
+    use crate::cursor::{Cursor, FuncCursor};
+    use crate::ir::types;
+    use crate::ir::{Function, InstructionData, Opcode, TrapCode};
     use std::string::ToString;
 
     #[test]
@@ -1218,8 +1218,8 @@ mod tests {
 
     #[test]
     fn aliases() {
-        use ir::condcodes::IntCC;
-        use ir::InstBuilder;
+        use crate::ir::condcodes::IntCC;
+        use crate::ir::InstBuilder;
 
         let mut func = Function::new();
         let ebb0 = func.dfg.make_ebb();
diff --git a/lib/codegen/src/ir/entities.rs b/lib/codegen/src/ir/entities.rs
index 185af8b9e7..b669cf77b6 100644
--- a/lib/codegen/src/ir/entities.rs
+++ b/lib/codegen/src/ir/entities.rs
@@ -19,6 +19,7 @@
 //! The entity references all implement the `Display` trait in a way that matches the textual IR
 //! format.
 
+use crate::entity::entity_impl;
 use std::fmt;
 use std::u32;
 
@@ -318,7 +319,7 @@ mod tests {
 
     #[test]
     fn memory() {
-        use packed_option::PackedOption;
+        use crate::packed_option::PackedOption;
         use std::mem;
         // This is the whole point of `PackedOption`.
         assert_eq!(
diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs
index 7ee4fbe8e2..9c17f34556 100644
--- a/lib/codegen/src/ir/extfunc.rs
+++ b/lib/codegen/src/ir/extfunc.rs
@@ -5,8 +5,8 @@
 //!
 //! This module declares the data types used to represent external functions and call signatures.
 
-use ir::{ArgumentLoc, ExternalName, SigRef, Type};
-use isa::{CallConv, RegInfo, RegUnit};
+use crate::ir::{ArgumentLoc, ExternalName, SigRef, Type};
+use crate::isa::{CallConv, RegInfo, RegUnit};
 use std::fmt;
 use std::str::FromStr;
 use std::vec::Vec;
@@ -334,7 +334,7 @@ impl fmt::Display for ExtFuncData {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use ir::types::{B8, F32, I32};
+    use crate::ir::types::{B8, F32, I32};
     use std::string::ToString;
 
     #[test]
diff --git a/lib/codegen/src/ir/extname.rs b/lib/codegen/src/ir/extname.rs
index b35c9aeb05..7a9da78f7f 100644
--- a/lib/codegen/src/ir/extname.rs
+++ b/lib/codegen/src/ir/extname.rs
@@ -4,7 +4,7 @@
 //! function. The name of an external declaration doesn't have any meaning to
 //! Cranelift, which compiles functions independently.
 
-use ir::LibCall;
+use crate::ir::LibCall;
 use std::cmp;
 use std::fmt::{self, Write};
 use std::str::FromStr;
@@ -119,7 +119,7 @@ impl FromStr for ExternalName {
 #[cfg(test)]
 mod tests {
     use super::ExternalName;
-    use ir::LibCall;
+    use crate::ir::LibCall;
     use std::string::ToString;
     use std::u32;
 
diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs
index 629741b4ae..85bfdb5207 100644
--- a/lib/codegen/src/ir/function.rs
+++ b/lib/codegen/src/ir/function.rs
@@ -3,20 +3,20 @@
 //! The `Function` struct defined in this module owns all of its extended basic blocks and
 //! instructions.
 
-use binemit::CodeOffset;
-use entity::{PrimaryMap, SecondaryMap};
-use ir;
-use ir::{DataFlowGraph, ExternalName, Layout, Signature};
-use ir::{
+use crate::binemit::CodeOffset;
+use crate::entity::{PrimaryMap, SecondaryMap};
+use crate::ir;
+use crate::ir::{DataFlowGraph, ExternalName, Layout, Signature};
+use crate::ir::{
     Ebb, ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, JumpTable,
     JumpTableData, SigRef, StackSlot, StackSlotData, Table, TableData,
 };
-use ir::{EbbOffsets, InstEncodings, SourceLocs, StackSlots, ValueLocations};
-use ir::{JumpTableOffsets, JumpTables};
-use isa::{CallConv, EncInfo, Encoding, Legalize, TargetIsa};
-use regalloc::RegDiversions;
+use crate::ir::{EbbOffsets, InstEncodings, SourceLocs, StackSlots, ValueLocations};
+use crate::ir::{JumpTableOffsets, JumpTables};
+use crate::isa::{CallConv, EncInfo, Encoding, Legalize, TargetIsa};
+use crate::regalloc::RegDiversions;
+use crate::write::write_function;
 use std::fmt;
-use write::write_function;
 
 /// A function.
 ///
diff --git a/lib/codegen/src/ir/globalvalue.rs b/lib/codegen/src/ir/globalvalue.rs
index b46a2ab4e6..aedfa1dfc8 100644
--- a/lib/codegen/src/ir/globalvalue.rs
+++ b/lib/codegen/src/ir/globalvalue.rs
@@ -1,8 +1,8 @@
 //! Global values.
 
-use ir::immediates::{Imm64, Offset32};
-use ir::{ExternalName, GlobalValue, Type};
-use isa::TargetIsa;
+use crate::ir::immediates::{Imm64, Offset32};
+use crate::ir::{ExternalName, GlobalValue, Type};
+use crate::isa::TargetIsa;
 use std::fmt;
 
 /// Information about a global value declaration.
diff --git a/lib/codegen/src/ir/heap.rs b/lib/codegen/src/ir/heap.rs
index f8d1b78cb0..8495fd51e3 100644
--- a/lib/codegen/src/ir/heap.rs
+++ b/lib/codegen/src/ir/heap.rs
@@ -1,7 +1,7 @@
 //! Heaps.
 
-use ir::immediates::Uimm64;
-use ir::{GlobalValue, Type};
+use crate::ir::immediates::Uimm64;
+use crate::ir::{GlobalValue, Type};
 use std::fmt;
 
 /// Information about a heap declaration.
diff --git a/lib/codegen/src/ir/instructions.rs b/lib/codegen/src/ir/instructions.rs
index b01525ca80..f0ac7c24c1 100644
--- a/lib/codegen/src/ir/instructions.rs
+++ b/lib/codegen/src/ir/instructions.rs
@@ -11,14 +11,14 @@ use std::ops::{Deref, DerefMut};
 use std::str::FromStr;
 use std::vec::Vec;
 
-use ir;
-use ir::types;
-use ir::{Ebb, FuncRef, JumpTable, SigRef, Type, Value};
-use isa;
+use crate::ir;
+use crate::ir::types;
+use crate::ir::{Ebb, FuncRef, JumpTable, SigRef, Type, Value};
+use crate::isa;
 
-use bitset::BitSet;
-use entity;
-use ref_slice::{ref_slice, ref_slice_mut};
+use crate::bitset::BitSet;
+use crate::entity;
+use crate::ref_slice::{ref_slice, ref_slice_mut};
 
 /// Some instructions use an external list of argument values because there is not enough space in
 /// the 16-byte `InstructionData` struct. These value lists are stored in a memory pool in
@@ -73,7 +73,7 @@ impl FromStr for Opcode {
 
     /// Parse an Opcode name from a string.
     fn from_str(s: &str) -> Result {
-        use constant_hash::{probe, simple_hash, Table};
+        use crate::constant_hash::{probe, simple_hash, Table};
 
         impl<'a> Table<&'a str> for [Option] {
             fn len(&self) -> usize {
@@ -645,7 +645,7 @@ mod tests {
 
     #[test]
     fn value_set() {
-        use ir::types::*;
+        use crate::ir::types::*;
 
         let vts = ValueTypeSet {
             lanes: BitSet16::from_range(0, 8),
diff --git a/lib/codegen/src/ir/jumptable.rs b/lib/codegen/src/ir/jumptable.rs
index 7ba0a7d13c..e436ea412f 100644
--- a/lib/codegen/src/ir/jumptable.rs
+++ b/lib/codegen/src/ir/jumptable.rs
@@ -3,7 +3,7 @@
 //! Jump tables are declared in the preamble and assigned an `ir::entities::JumpTable` reference.
 //! The actual table of destinations is stored in a `JumpTableData` struct defined in this module.
 
-use ir::entities::Ebb;
+use crate::ir::entities::Ebb;
 use std::fmt::{self, Display, Formatter};
 use std::slice::{Iter, IterMut};
 use std::vec::Vec;
@@ -83,8 +83,8 @@ impl Display for JumpTableData {
 #[cfg(test)]
 mod tests {
     use super::JumpTableData;
-    use entity::EntityRef;
-    use ir::Ebb;
+    use crate::entity::EntityRef;
+    use crate::ir::Ebb;
     use std::string::ToString;
 
     #[test]
diff --git a/lib/codegen/src/ir/layout.rs b/lib/codegen/src/ir/layout.rs
index f598246b84..748217847f 100644
--- a/lib/codegen/src/ir/layout.rs
+++ b/lib/codegen/src/ir/layout.rs
@@ -3,13 +3,14 @@
 //! The order of extended basic blocks in a function and the order of instructions in an EBB is
 //! determined by the `Layout` data structure defined in this module.
 
-use entity::SecondaryMap;
-use ir::progpoint::{ExpandedProgramPoint, ProgramOrder};
-use ir::{Ebb, Inst};
-use packed_option::PackedOption;
+use crate::entity::SecondaryMap;
+use crate::ir::progpoint::{ExpandedProgramPoint, ProgramOrder};
+use crate::ir::{Ebb, Inst};
+use crate::packed_option::PackedOption;
+use crate::timing;
+use log::debug;
 use std::cmp;
 use std::iter::{IntoIterator, Iterator};
-use timing;
 
 /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not
 /// contain definitions of instructions or EBBs, but depends on `Inst` and `Ebb` entity references
@@ -741,9 +742,9 @@ impl<'f> DoubleEndedIterator for Insts<'f> {
 #[cfg(test)]
 mod tests {
     use super::Layout;
-    use cursor::{Cursor, CursorPosition};
-    use entity::EntityRef;
-    use ir::{Ebb, Inst, ProgramOrder, SourceLoc};
+    use crate::cursor::{Cursor, CursorPosition};
+    use crate::entity::EntityRef;
+    use crate::ir::{Ebb, Inst, ProgramOrder, SourceLoc};
     use std::cmp::Ordering;
     use std::vec::Vec;
 
diff --git a/lib/codegen/src/ir/libcall.rs b/lib/codegen/src/ir/libcall.rs
index 1e5297022e..c282de34e4 100644
--- a/lib/codegen/src/ir/libcall.rs
+++ b/lib/codegen/src/ir/libcall.rs
@@ -1,10 +1,10 @@
 //! Naming well-known routines in the runtime library.
 
-use ir::{
+use crate::ir::{
     types, AbiParam, ArgumentPurpose, ExtFuncData, ExternalName, FuncRef, Function, Inst, Opcode,
     Signature, Type,
 };
-use isa::{CallConv, RegUnit, TargetIsa};
+use crate::isa::{CallConv, RegUnit, TargetIsa};
 use std::fmt;
 use std::str::FromStr;
 
diff --git a/lib/codegen/src/ir/mod.rs b/lib/codegen/src/ir/mod.rs
index 716bf99b85..8167d3b607 100644
--- a/lib/codegen/src/ir/mod.rs
+++ b/lib/codegen/src/ir/mod.rs
@@ -23,32 +23,36 @@ mod trapcode;
 pub mod types;
 mod valueloc;
 
-pub use ir::builder::{InsertBuilder, InstBuilder, InstBuilderBase, InstInserterBase};
-pub use ir::dfg::{DataFlowGraph, ValueDef};
-pub use ir::entities::{
+pub use crate::ir::builder::{InsertBuilder, InstBuilder, InstBuilderBase, InstInserterBase};
+pub use crate::ir::dfg::{DataFlowGraph, ValueDef};
+pub use crate::ir::entities::{
     Ebb, FuncRef, GlobalValue, Heap, Inst, JumpTable, SigRef, StackSlot, Table, Value,
 };
-pub use ir::extfunc::{AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData, Signature};
-pub use ir::extname::ExternalName;
-pub use ir::function::Function;
-pub use ir::globalvalue::GlobalValueData;
-pub use ir::heap::{HeapData, HeapStyle};
-pub use ir::instructions::{InstructionData, Opcode, ValueList, ValueListPool, VariableArgs};
-pub use ir::jumptable::JumpTableData;
-pub use ir::layout::Layout;
-pub use ir::libcall::{get_libcall_funcref, get_probestack_funcref, LibCall};
-pub use ir::memflags::MemFlags;
-pub use ir::progpoint::{ExpandedProgramPoint, ProgramOrder, ProgramPoint};
-pub use ir::sourceloc::SourceLoc;
-pub use ir::stackslot::{StackSlotData, StackSlotKind, StackSlots};
-pub use ir::table::TableData;
-pub use ir::trapcode::TrapCode;
-pub use ir::types::Type;
-pub use ir::valueloc::{ArgumentLoc, ValueLoc};
+pub use crate::ir::extfunc::{
+    AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData, Signature,
+};
+pub use crate::ir::extname::ExternalName;
+pub use crate::ir::function::Function;
+pub use crate::ir::globalvalue::GlobalValueData;
+pub use crate::ir::heap::{HeapData, HeapStyle};
+pub use crate::ir::instructions::{
+    InstructionData, Opcode, ValueList, ValueListPool, VariableArgs,
+};
+pub use crate::ir::jumptable::JumpTableData;
+pub use crate::ir::layout::Layout;
+pub use crate::ir::libcall::{get_libcall_funcref, get_probestack_funcref, LibCall};
+pub use crate::ir::memflags::MemFlags;
+pub use crate::ir::progpoint::{ExpandedProgramPoint, ProgramOrder, ProgramPoint};
+pub use crate::ir::sourceloc::SourceLoc;
+pub use crate::ir::stackslot::{StackSlotData, StackSlotKind, StackSlots};
+pub use crate::ir::table::TableData;
+pub use crate::ir::trapcode::TrapCode;
+pub use crate::ir::types::Type;
+pub use crate::ir::valueloc::{ArgumentLoc, ValueLoc};
 
-use binemit;
-use entity::{PrimaryMap, SecondaryMap};
-use isa;
+use crate::binemit;
+use crate::entity::{PrimaryMap, SecondaryMap};
+use crate::isa;
 
 /// Map of value locations.
 pub type ValueLocations = SecondaryMap;
diff --git a/lib/codegen/src/ir/progpoint.rs b/lib/codegen/src/ir/progpoint.rs
index 19d14892b1..abe2dd786f 100644
--- a/lib/codegen/src/ir/progpoint.rs
+++ b/lib/codegen/src/ir/progpoint.rs
@@ -1,7 +1,7 @@
 //! Program points.
 
-use entity::EntityRef;
-use ir::{Ebb, Inst, ValueDef};
+use crate::entity::EntityRef;
+use crate::ir::{Ebb, Inst, ValueDef};
 use std::cmp;
 use std::fmt;
 use std::u32;
@@ -146,8 +146,8 @@ pub trait ProgramOrder {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use entity::EntityRef;
-    use ir::{Ebb, Inst};
+    use crate::entity::EntityRef;
+    use crate::ir::{Ebb, Inst};
     use std::string::ToString;
 
     #[test]
diff --git a/lib/codegen/src/ir/sourceloc.rs b/lib/codegen/src/ir/sourceloc.rs
index b6bea0e7d9..3f2add7346 100644
--- a/lib/codegen/src/ir/sourceloc.rs
+++ b/lib/codegen/src/ir/sourceloc.rs
@@ -50,7 +50,7 @@ impl fmt::Display for SourceLoc {
 
 #[cfg(test)]
 mod tests {
-    use ir::SourceLoc;
+    use crate::ir::SourceLoc;
     use std::string::ToString;
 
     #[test]
diff --git a/lib/codegen/src/ir/stackslot.rs b/lib/codegen/src/ir/stackslot.rs
index c574986d46..bf2d555bf2 100644
--- a/lib/codegen/src/ir/stackslot.rs
+++ b/lib/codegen/src/ir/stackslot.rs
@@ -3,9 +3,9 @@
 //! The `StackSlotData` struct keeps track of a single stack slot in a function.
 //!
 
-use entity::{Iter, IterMut, Keys, PrimaryMap};
-use ir::{StackSlot, Type};
-use packed_option::PackedOption;
+use crate::entity::{Iter, IterMut, Keys, PrimaryMap};
+use crate::ir::{StackSlot, Type};
+use crate::packed_option::PackedOption;
 use std::cmp;
 use std::fmt;
 use std::ops::{Index, IndexMut};
@@ -340,8 +340,8 @@ impl StackSlots {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use ir::types;
-    use ir::Function;
+    use crate::ir::types;
+    use crate::ir::Function;
     use std::string::ToString;
 
     #[test]
diff --git a/lib/codegen/src/ir/table.rs b/lib/codegen/src/ir/table.rs
index 0b019d81cd..003eb710b9 100644
--- a/lib/codegen/src/ir/table.rs
+++ b/lib/codegen/src/ir/table.rs
@@ -1,7 +1,7 @@
 //! Tables.
 
-use ir::immediates::Uimm64;
-use ir::{GlobalValue, Type};
+use crate::ir::immediates::Uimm64;
+use crate::ir::{GlobalValue, Type};
 use std::fmt;
 
 /// Information about a table declaration.
diff --git a/lib/codegen/src/ir/valueloc.rs b/lib/codegen/src/ir/valueloc.rs
index a88a3af4de..f2c5151f09 100644
--- a/lib/codegen/src/ir/valueloc.rs
+++ b/lib/codegen/src/ir/valueloc.rs
@@ -3,8 +3,8 @@
 //! The register allocator assigns every SSA value to either a register or a stack slot. This
 //! assignment is represented by a `ValueLoc` object.
 
-use ir::StackSlot;
-use isa::{RegInfo, RegUnit};
+use crate::ir::StackSlot;
+use crate::isa::{RegInfo, RegUnit};
 use std::fmt;
 
 /// Value location.
diff --git a/lib/codegen/src/isa/arm32/abi.rs b/lib/codegen/src/isa/arm32/abi.rs
index 1305685825..a1890fa329 100644
--- a/lib/codegen/src/isa/arm32/abi.rs
+++ b/lib/codegen/src/isa/arm32/abi.rs
@@ -1,10 +1,10 @@
 //! ARM ABI implementation.
 
 use super::registers::{D, GPR, Q, S};
-use ir;
-use isa::RegClass;
-use regalloc::RegisterSet;
-use settings as shared_settings;
+use crate::ir;
+use crate::isa::RegClass;
+use crate::regalloc::RegisterSet;
+use crate::settings as shared_settings;
 
 /// Legalize `sig`.
 pub fn legalize_signature(
diff --git a/lib/codegen/src/isa/arm32/binemit.rs b/lib/codegen/src/isa/arm32/binemit.rs
index 9f50e63993..2ca78b4d6f 100644
--- a/lib/codegen/src/isa/arm32/binemit.rs
+++ b/lib/codegen/src/isa/arm32/binemit.rs
@@ -1,7 +1,7 @@
 //! Emitting binary ARM32 machine code.
 
-use binemit::{bad_encoding, CodeSink};
-use ir::{Function, Inst};
-use regalloc::RegDiversions;
+use crate::binemit::{bad_encoding, CodeSink};
+use crate::ir::{Function, Inst};
+use crate::regalloc::RegDiversions;
 
 include!(concat!(env!("OUT_DIR"), "/binemit-arm32.rs"));
diff --git a/lib/codegen/src/isa/arm32/enc_tables.rs b/lib/codegen/src/isa/arm32/enc_tables.rs
index f71dd33f87..e3833eecf7 100644
--- a/lib/codegen/src/isa/arm32/enc_tables.rs
+++ b/lib/codegen/src/isa/arm32/enc_tables.rs
@@ -1,10 +1,9 @@
 //! Encoding tables for ARM32 ISA.
 
-use ir;
-use isa;
-use isa::constraints::*;
-use isa::enc_tables::*;
-use isa::encoding::RecipeSizing;
+use crate::isa;
+use crate::isa::constraints::*;
+use crate::isa::enc_tables::*;
+use crate::isa::encoding::RecipeSizing;
 
 include!(concat!(env!("OUT_DIR"), "/encoding-arm32.rs"));
 include!(concat!(env!("OUT_DIR"), "/legalize-arm32.rs"));
diff --git a/lib/codegen/src/isa/arm32/mod.rs b/lib/codegen/src/isa/arm32/mod.rs
index cee753a5f0..a184061ac0 100644
--- a/lib/codegen/src/isa/arm32/mod.rs
+++ b/lib/codegen/src/isa/arm32/mod.rs
@@ -8,13 +8,13 @@ pub mod settings;
 
 use super::super::settings as shared_settings;
 #[cfg(feature = "testing_hooks")]
-use binemit::CodeSink;
-use binemit::{emit_function, MemoryCodeSink};
-use ir;
-use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings};
-use isa::Builder as IsaBuilder;
-use isa::{EncInfo, RegClass, RegInfo, TargetIsa};
-use regalloc;
+use crate::binemit::CodeSink;
+use crate::binemit::{emit_function, MemoryCodeSink};
+use crate::ir;
+use crate::isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings};
+use crate::isa::Builder as IsaBuilder;
+use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
+use crate::regalloc;
 use std::boxed::Box;
 use std::fmt;
 use target_lexicon::{Architecture, Triple};
diff --git a/lib/codegen/src/isa/arm32/registers.rs b/lib/codegen/src/isa/arm32/registers.rs
index 305890b29e..70549528cf 100644
--- a/lib/codegen/src/isa/arm32/registers.rs
+++ b/lib/codegen/src/isa/arm32/registers.rs
@@ -1,13 +1,13 @@
 //! ARM32 register descriptions.
 
-use isa::registers::{RegBank, RegClass, RegClassData, RegInfo, RegUnit};
+use crate::isa::registers::{RegBank, RegClass, RegClassData, RegInfo, RegUnit};
 
 include!(concat!(env!("OUT_DIR"), "/registers-arm32.rs"));
 
 #[cfg(test)]
 mod tests {
     use super::{D, GPR, INFO, S};
-    use isa::RegUnit;
+    use crate::isa::RegUnit;
     use std::string::{String, ToString};
 
     #[test]
@@ -34,7 +34,7 @@ mod tests {
     #[test]
     fn overlaps() {
         // arm32 has the most interesting register geometries, so test `regs_overlap()` here.
-        use isa::regs_overlap;
+        use crate::isa::regs_overlap;
 
         let r0 = GPR.unit(0);
         let r1 = GPR.unit(1);
diff --git a/lib/codegen/src/isa/arm32/settings.rs b/lib/codegen/src/isa/arm32/settings.rs
index a5075597d9..d32483100f 100644
--- a/lib/codegen/src/isa/arm32/settings.rs
+++ b/lib/codegen/src/isa/arm32/settings.rs
@@ -1,6 +1,6 @@
 //! ARM32 Settings.
 
-use settings::{self, detail, Builder};
+use crate::settings::{self, detail, Builder};
 use std::fmt;
 
 // Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
diff --git a/lib/codegen/src/isa/arm64/abi.rs b/lib/codegen/src/isa/arm64/abi.rs
index 79b7a97cd7..efc1f6125d 100644
--- a/lib/codegen/src/isa/arm64/abi.rs
+++ b/lib/codegen/src/isa/arm64/abi.rs
@@ -1,10 +1,10 @@
 //! ARM 64 ABI implementation.
 
 use super::registers::{FPR, GPR};
-use ir;
-use isa::RegClass;
-use regalloc::RegisterSet;
-use settings as shared_settings;
+use crate::ir;
+use crate::isa::RegClass;
+use crate::regalloc::RegisterSet;
+use crate::settings as shared_settings;
 
 /// Legalize `sig`.
 pub fn legalize_signature(
diff --git a/lib/codegen/src/isa/arm64/binemit.rs b/lib/codegen/src/isa/arm64/binemit.rs
index dd0654067e..05df67e5bb 100644
--- a/lib/codegen/src/isa/arm64/binemit.rs
+++ b/lib/codegen/src/isa/arm64/binemit.rs
@@ -1,7 +1,7 @@
 //! Emitting binary ARM64 machine code.
 
-use binemit::{bad_encoding, CodeSink};
-use ir::{Function, Inst};
-use regalloc::RegDiversions;
+use crate::binemit::{bad_encoding, CodeSink};
+use crate::ir::{Function, Inst};
+use crate::regalloc::RegDiversions;
 
 include!(concat!(env!("OUT_DIR"), "/binemit-arm64.rs"));
diff --git a/lib/codegen/src/isa/arm64/enc_tables.rs b/lib/codegen/src/isa/arm64/enc_tables.rs
index 6007450cb5..6e1b73e0e9 100644
--- a/lib/codegen/src/isa/arm64/enc_tables.rs
+++ b/lib/codegen/src/isa/arm64/enc_tables.rs
@@ -1,10 +1,9 @@
 //! Encoding tables for ARM64 ISA.
 
-use ir;
-use isa;
-use isa::constraints::*;
-use isa::enc_tables::*;
-use isa::encoding::RecipeSizing;
+use crate::isa;
+use crate::isa::constraints::*;
+use crate::isa::enc_tables::*;
+use crate::isa::encoding::RecipeSizing;
 
 include!(concat!(env!("OUT_DIR"), "/encoding-arm64.rs"));
 include!(concat!(env!("OUT_DIR"), "/legalize-arm64.rs"));
diff --git a/lib/codegen/src/isa/arm64/mod.rs b/lib/codegen/src/isa/arm64/mod.rs
index 830a109b52..b438e49949 100644
--- a/lib/codegen/src/isa/arm64/mod.rs
+++ b/lib/codegen/src/isa/arm64/mod.rs
@@ -8,13 +8,13 @@ pub mod settings;
 
 use super::super::settings as shared_settings;
 #[cfg(feature = "testing_hooks")]
-use binemit::CodeSink;
-use binemit::{emit_function, MemoryCodeSink};
-use ir;
-use isa::enc_tables::{lookup_enclist, Encodings};
-use isa::Builder as IsaBuilder;
-use isa::{EncInfo, RegClass, RegInfo, TargetIsa};
-use regalloc;
+use crate::binemit::CodeSink;
+use crate::binemit::{emit_function, MemoryCodeSink};
+use crate::ir;
+use crate::isa::enc_tables::{lookup_enclist, Encodings};
+use crate::isa::Builder as IsaBuilder;
+use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
+use crate::regalloc;
 use std::boxed::Box;
 use std::fmt;
 use target_lexicon::Triple;
diff --git a/lib/codegen/src/isa/arm64/registers.rs b/lib/codegen/src/isa/arm64/registers.rs
index 2c1d85e091..91ca256bbe 100644
--- a/lib/codegen/src/isa/arm64/registers.rs
+++ b/lib/codegen/src/isa/arm64/registers.rs
@@ -1,13 +1,13 @@
 //! ARM64 register descriptions.
 
-use isa::registers::{RegBank, RegClass, RegClassData, RegInfo, RegUnit};
+use crate::isa::registers::{RegBank, RegClass, RegClassData, RegInfo, RegUnit};
 
 include!(concat!(env!("OUT_DIR"), "/registers-arm64.rs"));
 
 #[cfg(test)]
 mod tests {
     use super::INFO;
-    use isa::RegUnit;
+    use crate::isa::RegUnit;
     use std::string::{String, ToString};
 
     #[test]
diff --git a/lib/codegen/src/isa/arm64/settings.rs b/lib/codegen/src/isa/arm64/settings.rs
index 6d51aed6ad..6ef1fcb258 100644
--- a/lib/codegen/src/isa/arm64/settings.rs
+++ b/lib/codegen/src/isa/arm64/settings.rs
@@ -1,6 +1,6 @@
 //! ARM64 Settings.
 
-use settings::{self, detail, Builder};
+use crate::settings::{self, detail, Builder};
 use std::fmt;
 
 // Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
diff --git a/lib/codegen/src/isa/constraints.rs b/lib/codegen/src/isa/constraints.rs
index 7c5da6032d..d6e8097385 100644
--- a/lib/codegen/src/isa/constraints.rs
+++ b/lib/codegen/src/isa/constraints.rs
@@ -7,10 +7,10 @@
 //! It is the register allocator's job to make sure that the register constraints on value operands
 //! are satisfied.
 
-use binemit::CodeOffset;
-use ir::{Function, Inst, ValueLoc};
-use isa::{RegClass, RegUnit};
-use regalloc::RegDiversions;
+use crate::binemit::CodeOffset;
+use crate::ir::{Function, Inst, ValueLoc};
+use crate::isa::{RegClass, RegUnit};
+use crate::regalloc::RegDiversions;
 
 /// Register constraint for a single value operand or instruction result.
 #[derive(PartialEq, Debug)]
diff --git a/lib/codegen/src/isa/enc_tables.rs b/lib/codegen/src/isa/enc_tables.rs
index 8cc259420f..ccdbc28ebd 100644
--- a/lib/codegen/src/isa/enc_tables.rs
+++ b/lib/codegen/src/isa/enc_tables.rs
@@ -3,10 +3,10 @@
 //! This module contains types and functions for working with the encoding tables generated by
 //! `lib/codegen/meta-python/gen_encoding.py`.
 
-use constant_hash::{probe, Table};
-use ir::{Function, InstructionData, Opcode, Type};
-use isa::{Encoding, Legalize};
-use settings::PredicateView;
+use crate::constant_hash::{probe, Table};
+use crate::ir::{Function, InstructionData, Opcode, Type};
+use crate::isa::{Encoding, Legalize};
+use crate::settings::PredicateView;
 use std::ops::Range;
 
 /// A recipe predicate.
diff --git a/lib/codegen/src/isa/encoding.rs b/lib/codegen/src/isa/encoding.rs
index ee643c508d..bf6a001b07 100644
--- a/lib/codegen/src/isa/encoding.rs
+++ b/lib/codegen/src/isa/encoding.rs
@@ -1,9 +1,9 @@
 //! The `Encoding` struct.
 
-use binemit::CodeOffset;
-use ir::{Function, Inst};
-use isa::constraints::{BranchRange, RecipeConstraints};
-use regalloc::RegDiversions;
+use crate::binemit::CodeOffset;
+use crate::ir::{Function, Inst};
+use crate::isa::constraints::{BranchRange, RecipeConstraints};
+use crate::regalloc::RegDiversions;
 use std::fmt;
 
 /// Bits needed to encode an instruction as binary machine code.
diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs
index d8eba6ccc3..291fdfb76c 100644
--- a/lib/codegen/src/isa/mod.rs
+++ b/lib/codegen/src/isa/mod.rs
@@ -46,24 +46,27 @@
 //! The configured target ISA trait object is a `Box` which can be used for multiple
 //! concurrent function compilations.
 
-pub use isa::call_conv::CallConv;
-pub use isa::constraints::{BranchRange, ConstraintKind, OperandConstraint, RecipeConstraints};
-pub use isa::encoding::{base_size, EncInfo, Encoding};
-pub use isa::registers::{regs_overlap, RegClass, RegClassIndex, RegInfo, RegUnit};
-pub use isa::stack::{StackBase, StackBaseMask, StackRef};
+pub use crate::isa::call_conv::CallConv;
+pub use crate::isa::constraints::{
+    BranchRange, ConstraintKind, OperandConstraint, RecipeConstraints,
+};
+pub use crate::isa::encoding::{base_size, EncInfo, Encoding};
+pub use crate::isa::registers::{regs_overlap, RegClass, RegClassIndex, RegInfo, RegUnit};
+pub use crate::isa::stack::{StackBase, StackBaseMask, StackRef};
 
-use binemit;
-use flowgraph;
-use ir;
-use isa::enc_tables::Encodings;
-use regalloc;
-use result::CodegenResult;
-use settings;
-use settings::SetResult;
+use crate::binemit;
+use crate::flowgraph;
+use crate::ir;
+use crate::isa::enc_tables::Encodings;
+use crate::regalloc;
+use crate::result::CodegenResult;
+use crate::settings;
+use crate::settings::SetResult;
+use crate::timing;
+use failure_derive::Fail;
 use std::boxed::Box;
 use std::fmt;
 use target_lexicon::{Architecture, PointerWidth, Triple};
-use timing;
 
 #[cfg(build_riscv)]
 mod riscv;
@@ -334,8 +337,8 @@ pub trait TargetIsa: fmt::Display + Sync {
     fn prologue_epilogue(&self, func: &mut ir::Function) -> CodegenResult<()> {
         let _tt = timing::prologue_epilogue();
         // This default implementation is unlikely to be good enough.
-        use ir::stackslot::{StackOffset, StackSize};
-        use stack_layout::layout_stack;
+        use crate::ir::stackslot::{StackOffset, StackSize};
+        use crate::stack_layout::layout_stack;
 
         let word_size = StackSize::from(self.pointer_bytes());
 
diff --git a/lib/codegen/src/isa/registers.rs b/lib/codegen/src/isa/registers.rs
index a11a3fbd52..4941712618 100644
--- a/lib/codegen/src/isa/registers.rs
+++ b/lib/codegen/src/isa/registers.rs
@@ -1,6 +1,6 @@
 //! Data structures describing the registers in an ISA.
 
-use entity::EntityRef;
+use crate::entity::EntityRef;
 use std::fmt;
 
 /// Register units are the smallest units of register allocation.
diff --git a/lib/codegen/src/isa/riscv/abi.rs b/lib/codegen/src/isa/riscv/abi.rs
index ed7230d2cb..cd33de404e 100644
--- a/lib/codegen/src/isa/riscv/abi.rs
+++ b/lib/codegen/src/isa/riscv/abi.rs
@@ -7,10 +7,10 @@
 
 use super::registers::{FPR, GPR};
 use super::settings;
-use abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion};
-use ir::{self, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, Type};
-use isa::RegClass;
-use regalloc::RegisterSet;
+use crate::abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion};
+use crate::ir::{self, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, Type};
+use crate::isa::RegClass;
+use crate::regalloc::RegisterSet;
 use std::i32;
 use target_lexicon::Triple;
 
diff --git a/lib/codegen/src/isa/riscv/binemit.rs b/lib/codegen/src/isa/riscv/binemit.rs
index 120324d1be..cd153f4ac1 100644
--- a/lib/codegen/src/isa/riscv/binemit.rs
+++ b/lib/codegen/src/isa/riscv/binemit.rs
@@ -1,10 +1,10 @@
 //! Emitting binary RISC-V machine code.
 
-use binemit::{bad_encoding, CodeSink, Reloc};
-use ir::{Function, Inst, InstructionData};
-use isa::{RegUnit, StackBaseMask, StackRef};
-use predicates::is_signed_int;
-use regalloc::RegDiversions;
+use crate::binemit::{bad_encoding, CodeSink, Reloc};
+use crate::ir::{Function, Inst, InstructionData};
+use crate::isa::{RegUnit, StackBaseMask, StackRef};
+use crate::predicates::is_signed_int;
+use crate::regalloc::RegDiversions;
 use std::u32;
 
 include!(concat!(env!("OUT_DIR"), "/binemit-riscv.rs"));
diff --git a/lib/codegen/src/isa/riscv/enc_tables.rs b/lib/codegen/src/isa/riscv/enc_tables.rs
index bbb7492805..19488003a3 100644
--- a/lib/codegen/src/isa/riscv/enc_tables.rs
+++ b/lib/codegen/src/isa/riscv/enc_tables.rs
@@ -1,11 +1,11 @@
 //! Encoding tables for RISC-V.
 
 use super::registers::*;
-use ir;
-use isa;
-use isa::constraints::*;
-use isa::enc_tables::*;
-use isa::encoding::{base_size, RecipeSizing};
+use crate::ir;
+use crate::isa;
+use crate::isa::constraints::*;
+use crate::isa::enc_tables::*;
+use crate::isa::encoding::{base_size, RecipeSizing};
 
 // Include the generated encoding tables:
 // - `LEVEL1_RV32`
diff --git a/lib/codegen/src/isa/riscv/mod.rs b/lib/codegen/src/isa/riscv/mod.rs
index 393219571f..7382809893 100644
--- a/lib/codegen/src/isa/riscv/mod.rs
+++ b/lib/codegen/src/isa/riscv/mod.rs
@@ -8,13 +8,13 @@ pub mod settings;
 
 use super::super::settings as shared_settings;
 #[cfg(feature = "testing_hooks")]
-use binemit::CodeSink;
-use binemit::{emit_function, MemoryCodeSink};
-use ir;
-use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings};
-use isa::Builder as IsaBuilder;
-use isa::{EncInfo, RegClass, RegInfo, TargetIsa};
-use regalloc;
+use crate::binemit::CodeSink;
+use crate::binemit::{emit_function, MemoryCodeSink};
+use crate::ir;
+use crate::isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings};
+use crate::isa::Builder as IsaBuilder;
+use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
+use crate::regalloc;
 use std::boxed::Box;
 use std::fmt;
 use target_lexicon::{PointerWidth, Triple};
@@ -125,13 +125,13 @@ impl TargetIsa for Isa {
 
 #[cfg(test)]
 mod tests {
-    use ir::{immediates, types};
-    use ir::{Function, InstructionData, Opcode};
-    use isa;
-    use settings::{self, Configurable};
+    use crate::ir::{immediates, types};
+    use crate::ir::{Function, InstructionData, Opcode};
+    use crate::isa;
+    use crate::settings::{self, Configurable};
     use std::str::FromStr;
     use std::string::{String, ToString};
-    use target_lexicon;
+    use target_lexicon::triple;
 
     fn encstr(isa: &isa::TargetIsa, enc: Result) -> String {
         match enc {
diff --git a/lib/codegen/src/isa/riscv/registers.rs b/lib/codegen/src/isa/riscv/registers.rs
index d6254fe38e..4f167bae8f 100644
--- a/lib/codegen/src/isa/riscv/registers.rs
+++ b/lib/codegen/src/isa/riscv/registers.rs
@@ -1,13 +1,13 @@
 //! RISC-V register descriptions.
 
-use isa::registers::{RegBank, RegClass, RegClassData, RegInfo, RegUnit};
+use crate::isa::registers::{RegBank, RegClass, RegClassData, RegInfo, RegUnit};
 
 include!(concat!(env!("OUT_DIR"), "/registers-riscv.rs"));
 
 #[cfg(test)]
 mod tests {
     use super::{FPR, GPR, INFO};
-    use isa::RegUnit;
+    use crate::isa::RegUnit;
     use std::string::{String, ToString};
 
     #[test]
diff --git a/lib/codegen/src/isa/riscv/settings.rs b/lib/codegen/src/isa/riscv/settings.rs
index 972f5c6b5f..2530c4a860 100644
--- a/lib/codegen/src/isa/riscv/settings.rs
+++ b/lib/codegen/src/isa/riscv/settings.rs
@@ -1,6 +1,6 @@
 //! RISC-V Settings.
 
-use settings::{self, detail, Builder};
+use crate::settings::{self, detail, Builder};
 use std::fmt;
 
 // Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
@@ -11,7 +11,7 @@ include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs"));
 #[cfg(test)]
 mod tests {
     use super::{builder, Flags};
-    use settings::{self, Configurable};
+    use crate::settings::{self, Configurable};
     use std::string::ToString;
 
     #[test]
diff --git a/lib/codegen/src/isa/stack.rs b/lib/codegen/src/isa/stack.rs
index df61f77edc..cc7b578039 100644
--- a/lib/codegen/src/isa/stack.rs
+++ b/lib/codegen/src/isa/stack.rs
@@ -4,8 +4,8 @@
 //! defined in this module expresses the low-level details of accessing a stack slot from an
 //! encoded instruction.
 
-use ir::stackslot::{StackOffset, StackSlotKind, StackSlots};
-use ir::StackSlot;
+use crate::ir::stackslot::{StackOffset, StackSlotKind, StackSlots};
+use crate::ir::StackSlot;
 
 /// A method for referencing a stack slot in the current stack frame.
 ///
diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs
index 1f385af419..fd09a9511c 100644
--- a/lib/codegen/src/isa/x86/abi.rs
+++ b/lib/codegen/src/isa/x86/abi.rs
@@ -1,19 +1,19 @@
 //! x86 ABI implementation.
 
 use super::registers::{FPR, GPR, RU};
-use abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion};
-use cursor::{Cursor, CursorPosition, EncCursor};
-use ir;
-use ir::immediates::Imm64;
-use ir::stackslot::{StackOffset, StackSize};
-use ir::{
+use crate::abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion};
+use crate::cursor::{Cursor, CursorPosition, EncCursor};
+use crate::ir;
+use crate::ir::immediates::Imm64;
+use crate::ir::stackslot::{StackOffset, StackSize};
+use crate::ir::{
     get_probestack_funcref, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, InstBuilder,
     ValueLoc,
 };
-use isa::{CallConv, RegClass, RegUnit, TargetIsa};
-use regalloc::RegisterSet;
-use result::CodegenResult;
-use stack_layout::layout_stack;
+use crate::isa::{CallConv, RegClass, RegUnit, TargetIsa};
+use crate::regalloc::RegisterSet;
+use crate::result::CodegenResult;
+use crate::stack_layout::layout_stack;
 use std::i32;
 use target_lexicon::{PointerWidth, Triple};
 
@@ -511,7 +511,7 @@ fn insert_common_prologue(
 /// Insert a check that generates a trap if the stack pointer goes
 /// below a value in `stack_limit_arg`.
 fn insert_stack_check(pos: &mut EncCursor, stack_size: i64, stack_limit_arg: ir::Value) {
-    use ir::condcodes::IntCC;
+    use crate::ir::condcodes::IntCC;
 
     // Copy `stack_limit_arg` into a %rax and use it for calculating
     // a SP threshold.
diff --git a/lib/codegen/src/isa/x86/binemit.rs b/lib/codegen/src/isa/x86/binemit.rs
index c930af39fd..513d6347fa 100644
--- a/lib/codegen/src/isa/x86/binemit.rs
+++ b/lib/codegen/src/isa/x86/binemit.rs
@@ -2,11 +2,11 @@
 
 use super::enc_tables::{needs_offset, needs_sib_byte};
 use super::registers::RU;
-use binemit::{bad_encoding, CodeSink, Reloc};
-use ir::condcodes::{CondCode, FloatCC, IntCC};
-use ir::{Ebb, Function, Inst, InstructionData, JumpTable, Opcode, TrapCode};
-use isa::{RegUnit, StackBase, StackBaseMask, StackRef};
-use regalloc::RegDiversions;
+use crate::binemit::{bad_encoding, CodeSink, Reloc};
+use crate::ir::condcodes::{CondCode, FloatCC, IntCC};
+use crate::ir::{Ebb, Function, Inst, InstructionData, JumpTable, Opcode, TrapCode};
+use crate::isa::{RegUnit, StackBase, StackBaseMask, StackRef};
+use crate::regalloc::RegDiversions;
 
 include!(concat!(env!("OUT_DIR"), "/binemit-x86.rs"));
 
@@ -270,7 +270,7 @@ fn sib(scale: u8, index: RegUnit, base: RegUnit, sink: &m
 /// 0x0f 0x90: SetCC.
 ///
 fn icc2opc(cond: IntCC) -> u16 {
-    use ir::condcodes::IntCC::*;
+    use crate::ir::condcodes::IntCC::*;
     match cond {
         // 0x0 = Overflow.
         // 0x1 = !Overflow.
@@ -303,7 +303,7 @@ fn icc2opc(cond: IntCC) -> u16 {
 ///
 /// Not all floating point condition codes are supported.
 fn fcc2opc(cond: FloatCC) -> u16 {
-    use ir::condcodes::FloatCC::*;
+    use crate::ir::condcodes::FloatCC::*;
     match cond {
         Ordered                    => 0xb, // EQ|LT|GT => *np (P=0)
         Unordered                  => 0xa, // UN       => *p  (P=1)
diff --git a/lib/codegen/src/isa/x86/enc_tables.rs b/lib/codegen/src/isa/x86/enc_tables.rs
index a709d03b76..bdd3458f14 100644
--- a/lib/codegen/src/isa/x86/enc_tables.rs
+++ b/lib/codegen/src/isa/x86/enc_tables.rs
@@ -1,18 +1,18 @@
 //! Encoding tables for x86 ISAs.
 
 use super::registers::*;
-use bitset::BitSet;
-use cursor::{Cursor, FuncCursor};
-use flowgraph::ControlFlowGraph;
-use ir::condcodes::IntCC;
-use ir::{self, Function, Inst, InstBuilder};
-use isa;
-use isa::constraints::*;
-use isa::enc_tables::*;
-use isa::encoding::base_size;
-use isa::encoding::RecipeSizing;
-use isa::RegUnit;
-use regalloc::RegDiversions;
+use crate::bitset::BitSet;
+use crate::cursor::{Cursor, FuncCursor};
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir::condcodes::IntCC;
+use crate::ir::{self, Function, Inst, InstBuilder};
+use crate::isa;
+use crate::isa::constraints::*;
+use crate::isa::enc_tables::*;
+use crate::isa::encoding::base_size;
+use crate::isa::encoding::RecipeSizing;
+use crate::isa::RegUnit;
+use crate::regalloc::RegDiversions;
 
 include!(concat!(env!("OUT_DIR"), "/encoding-x86.rs"));
 include!(concat!(env!("OUT_DIR"), "/legalize-x86.rs"));
@@ -228,7 +228,7 @@ fn expand_minmax(
     cfg: &mut ControlFlowGraph,
     _isa: &isa::TargetIsa,
 ) {
-    use ir::condcodes::FloatCC;
+    use crate::ir::condcodes::FloatCC;
 
     let (x, y, x86_opc, bitwise_opc) = match func.dfg[inst] {
         ir::InstructionData::Binary {
@@ -322,7 +322,7 @@ fn expand_fcvt_from_uint(
     cfg: &mut ControlFlowGraph,
     _isa: &isa::TargetIsa,
 ) {
-    use ir::condcodes::IntCC;
+    use crate::ir::condcodes::IntCC;
 
     let x;
     match func.dfg[inst] {
@@ -395,8 +395,8 @@ fn expand_fcvt_to_sint(
     cfg: &mut ControlFlowGraph,
     _isa: &isa::TargetIsa,
 ) {
-    use ir::condcodes::{FloatCC, IntCC};
-    use ir::immediates::{Ieee32, Ieee64};
+    use crate::ir::condcodes::{FloatCC, IntCC};
+    use crate::ir::immediates::{Ieee32, Ieee64};
 
     let x = match func.dfg[inst] {
         ir::InstructionData::Unary {
@@ -491,8 +491,8 @@ fn expand_fcvt_to_sint_sat(
     cfg: &mut ControlFlowGraph,
     _isa: &isa::TargetIsa,
 ) {
-    use ir::condcodes::{FloatCC, IntCC};
-    use ir::immediates::{Ieee32, Ieee64};
+    use crate::ir::condcodes::{FloatCC, IntCC};
+    use crate::ir::immediates::{Ieee32, Ieee64};
 
     let x = match func.dfg[inst] {
         ir::InstructionData::Unary {
@@ -611,8 +611,8 @@ fn expand_fcvt_to_uint(
     cfg: &mut ControlFlowGraph,
     _isa: &isa::TargetIsa,
 ) {
-    use ir::condcodes::{FloatCC, IntCC};
-    use ir::immediates::{Ieee32, Ieee64};
+    use crate::ir::condcodes::{FloatCC, IntCC};
+    use crate::ir::immediates::{Ieee32, Ieee64};
 
     let x = match func.dfg[inst] {
         ir::InstructionData::Unary {
@@ -693,8 +693,8 @@ fn expand_fcvt_to_uint_sat(
     cfg: &mut ControlFlowGraph,
     _isa: &isa::TargetIsa,
 ) {
-    use ir::condcodes::{FloatCC, IntCC};
-    use ir::immediates::{Ieee32, Ieee64};
+    use crate::ir::condcodes::{FloatCC, IntCC};
+    use crate::ir::immediates::{Ieee32, Ieee64};
 
     let x = match func.dfg[inst] {
         ir::InstructionData::Unary {
diff --git a/lib/codegen/src/isa/x86/mod.rs b/lib/codegen/src/isa/x86/mod.rs
index 1dfa62bf61..10ec0e5269 100644
--- a/lib/codegen/src/isa/x86/mod.rs
+++ b/lib/codegen/src/isa/x86/mod.rs
@@ -8,18 +8,18 @@ pub mod settings;
 
 use super::super::settings as shared_settings;
 #[cfg(feature = "testing_hooks")]
-use binemit::CodeSink;
-use binemit::{emit_function, MemoryCodeSink};
-use ir;
-use isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings};
-use isa::Builder as IsaBuilder;
-use isa::{EncInfo, RegClass, RegInfo, TargetIsa};
-use regalloc;
-use result::CodegenResult;
+use crate::binemit::CodeSink;
+use crate::binemit::{emit_function, MemoryCodeSink};
+use crate::ir;
+use crate::isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings};
+use crate::isa::Builder as IsaBuilder;
+use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
+use crate::regalloc;
+use crate::result::CodegenResult;
+use crate::timing;
 use std::boxed::Box;
 use std::fmt;
 use target_lexicon::{PointerWidth, Triple};
-use timing;
 
 #[allow(dead_code)]
 struct Isa {
diff --git a/lib/codegen/src/isa/x86/registers.rs b/lib/codegen/src/isa/x86/registers.rs
index 5ca0c4912c..a5ebf438d4 100644
--- a/lib/codegen/src/isa/x86/registers.rs
+++ b/lib/codegen/src/isa/x86/registers.rs
@@ -1,13 +1,13 @@
 //! x86 register descriptions.
 
-use isa::registers::{RegBank, RegClass, RegClassData, RegInfo, RegUnit};
+use crate::isa::registers::{RegBank, RegClass, RegClassData, RegInfo, RegUnit};
 
 include!(concat!(env!("OUT_DIR"), "/registers-x86.rs"));
 
 #[cfg(test)]
 mod tests {
     use super::*;
-    use isa::RegUnit;
+    use crate::isa::RegUnit;
     use std::string::{String, ToString};
 
     #[test]
diff --git a/lib/codegen/src/isa/x86/settings.rs b/lib/codegen/src/isa/x86/settings.rs
index 66699b1289..52eccaf9f8 100644
--- a/lib/codegen/src/isa/x86/settings.rs
+++ b/lib/codegen/src/isa/x86/settings.rs
@@ -1,6 +1,6 @@
 //! x86 Settings.
 
-use settings::{self, detail, Builder};
+use crate::settings::{self, detail, Builder};
 use std::fmt;
 
 // Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
@@ -11,7 +11,7 @@ include!(concat!(env!("OUT_DIR"), "/settings-x86.rs"));
 #[cfg(test)]
 mod tests {
     use super::{builder, Flags};
-    use settings::{self, Configurable};
+    use crate::settings::{self, Configurable};
 
     #[test]
     fn presets() {
diff --git a/lib/codegen/src/legalizer/boundary.rs b/lib/codegen/src/legalizer/boundary.rs
index 8b7b6c564c..c4bbcc9db6 100644
--- a/lib/codegen/src/legalizer/boundary.rs
+++ b/lib/codegen/src/legalizer/boundary.rs
@@ -17,16 +17,17 @@
 //! Between the two phases, preamble signatures and call/return arguments don't match. This
 //! intermediate state doesn't type check.
 
-use abi::{legalize_abi_value, ValueConversion};
-use cursor::{Cursor, FuncCursor};
-use flowgraph::ControlFlowGraph;
-use ir::instructions::CallInfo;
-use ir::{
+use crate::abi::{legalize_abi_value, ValueConversion};
+use crate::cursor::{Cursor, FuncCursor};
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir::instructions::CallInfo;
+use crate::ir::{
     AbiParam, ArgumentLoc, ArgumentPurpose, DataFlowGraph, Ebb, Function, Inst, InstBuilder,
     SigRef, Signature, Type, Value, ValueLoc,
 };
-use isa::TargetIsa;
-use legalizer::split::{isplit, vsplit};
+use crate::isa::TargetIsa;
+use crate::legalizer::split::{isplit, vsplit};
+use log::debug;
 use std::vec::Vec;
 
 /// Legalize all the function signatures in `func`.
diff --git a/lib/codegen/src/legalizer/call.rs b/lib/codegen/src/legalizer/call.rs
index 21753d19db..6122937f3c 100644
--- a/lib/codegen/src/legalizer/call.rs
+++ b/lib/codegen/src/legalizer/call.rs
@@ -3,10 +3,10 @@
 //! This module exports the `expand_call` function which transforms a `call`
 //! instruction into `func_addr` and `call_indirect` instructions.
 
-use cursor::{Cursor, FuncCursor};
-use flowgraph::ControlFlowGraph;
-use ir::{self, InstBuilder};
-use isa::TargetIsa;
+use crate::cursor::{Cursor, FuncCursor};
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir::{self, InstBuilder};
+use crate::isa::TargetIsa;
 
 /// Expand a `call` instruction. This lowers it to a `call_indirect`, which
 /// is only done if the ABI doesn't support direct calls.
diff --git a/lib/codegen/src/legalizer/globalvalue.rs b/lib/codegen/src/legalizer/globalvalue.rs
index e6b8dce39e..344a8b8d0f 100644
--- a/lib/codegen/src/legalizer/globalvalue.rs
+++ b/lib/codegen/src/legalizer/globalvalue.rs
@@ -3,10 +3,10 @@
 //! This module exports the `expand_global_value` function which transforms a `global_value`
 //! instruction into code that depends on the kind of global value referenced.
 
-use cursor::{Cursor, FuncCursor};
-use flowgraph::ControlFlowGraph;
-use ir::{self, InstBuilder};
-use isa::TargetIsa;
+use crate::cursor::{Cursor, FuncCursor};
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir::{self, InstBuilder};
+use crate::isa::TargetIsa;
 
 /// Expand a `global_value` instruction according to the definition of the global value.
 pub fn expand_global_value(
diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs
index 2edf2fae37..41d9778421 100644
--- a/lib/codegen/src/legalizer/heap.rs
+++ b/lib/codegen/src/legalizer/heap.rs
@@ -3,11 +3,11 @@
 //! This module exports the `expand_heap_addr` function which transforms a `heap_addr`
 //! instruction into code that depends on the kind of heap referenced.
 
-use cursor::{Cursor, FuncCursor};
-use flowgraph::ControlFlowGraph;
-use ir::condcodes::IntCC;
-use ir::{self, InstBuilder};
-use isa::TargetIsa;
+use crate::cursor::{Cursor, FuncCursor};
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir::condcodes::IntCC;
+use crate::ir::{self, InstBuilder};
+use crate::isa::TargetIsa;
 
 /// Expand a `heap_addr` instruction according to the definition of the heap.
 pub fn expand_heap_addr(
diff --git a/lib/codegen/src/legalizer/libcall.rs b/lib/codegen/src/legalizer/libcall.rs
index fb1ba6e313..c4cbac30ee 100644
--- a/lib/codegen/src/legalizer/libcall.rs
+++ b/lib/codegen/src/legalizer/libcall.rs
@@ -1,9 +1,9 @@
 //! Expanding instructions as runtime library calls.
 
-use ir;
-use ir::{get_libcall_funcref, InstBuilder};
-use isa::TargetIsa;
-use legalizer::boundary::legalize_libcall_signature;
+use crate::ir;
+use crate::ir::{get_libcall_funcref, InstBuilder};
+use crate::isa::TargetIsa;
+use crate::legalizer::boundary::legalize_libcall_signature;
 use std::vec::Vec;
 
 /// Try to expand `inst` as a library call, returning true is successful.
diff --git a/lib/codegen/src/legalizer/mod.rs b/lib/codegen/src/legalizer/mod.rs
index f2a4636719..db0fb107fa 100644
--- a/lib/codegen/src/legalizer/mod.rs
+++ b/lib/codegen/src/legalizer/mod.rs
@@ -13,13 +13,13 @@
 //! The legalizer does not deal with register allocation constraints. These constraints are derived
 //! from the encoding recipes, and solved later by the register allocator.
 
-use bitset::BitSet;
-use cursor::{Cursor, FuncCursor};
-use flowgraph::ControlFlowGraph;
-use ir::types::I32;
-use ir::{self, InstBuilder, MemFlags};
-use isa::TargetIsa;
-use timing;
+use crate::bitset::BitSet;
+use crate::cursor::{Cursor, FuncCursor};
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir::types::I32;
+use crate::ir::{self, InstBuilder, MemFlags};
+use crate::isa::TargetIsa;
+use crate::timing;
 
 mod boundary;
 mod call;
@@ -195,7 +195,7 @@ fn expand_br_table_jt(
     cfg: &mut ControlFlowGraph,
     isa: &TargetIsa,
 ) {
-    use ir::condcodes::IntCC;
+    use crate::ir::condcodes::IntCC;
 
     let (arg, default_ebb, table) = match func.dfg[inst] {
         ir::InstructionData::BranchTable {
@@ -241,7 +241,7 @@ fn expand_br_table_conds(
     cfg: &mut ControlFlowGraph,
     _isa: &TargetIsa,
 ) {
-    use ir::condcodes::IntCC;
+    use crate::ir::condcodes::IntCC;
 
     let (arg, default_ebb, table) = match func.dfg[inst] {
         ir::InstructionData::BranchTable {
diff --git a/lib/codegen/src/legalizer/split.rs b/lib/codegen/src/legalizer/split.rs
index 6de664863a..9e27ae2397 100644
--- a/lib/codegen/src/legalizer/split.rs
+++ b/lib/codegen/src/legalizer/split.rs
@@ -64,9 +64,9 @@
 //! It is possible to have circular dependencies of EBB arguments that are never used by any real
 //! instructions. These loops will remain in the program.
 
-use cursor::{Cursor, CursorPosition, FuncCursor};
-use flowgraph::{BasicBlock, ControlFlowGraph};
-use ir::{self, Ebb, Inst, InstBuilder, InstructionData, Opcode, Type, Value, ValueDef};
+use crate::cursor::{Cursor, CursorPosition, FuncCursor};
+use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::ir::{self, Ebb, Inst, InstBuilder, InstructionData, Opcode, Type, Value, ValueDef};
 use std::iter;
 use std::vec::Vec;
 
diff --git a/lib/codegen/src/legalizer/table.rs b/lib/codegen/src/legalizer/table.rs
index f6005f1518..14da52612f 100644
--- a/lib/codegen/src/legalizer/table.rs
+++ b/lib/codegen/src/legalizer/table.rs
@@ -3,12 +3,12 @@
 //! This module exports the `expand_table_addr` function which transforms a `table_addr`
 //! instruction into code that depends on the kind of table referenced.
 
-use cursor::{Cursor, FuncCursor};
-use flowgraph::ControlFlowGraph;
-use ir::condcodes::IntCC;
-use ir::immediates::Offset32;
-use ir::{self, InstBuilder};
-use isa::TargetIsa;
+use crate::cursor::{Cursor, FuncCursor};
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir::condcodes::IntCC;
+use crate::ir::immediates::Offset32;
+use crate::ir::{self, InstBuilder};
+use crate::isa::TargetIsa;
 
 /// Expand a `table_addr` instruction according to the definition of the table.
 pub fn expand_table_addr(
diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index c8d4e4434f..737aca4140 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -49,26 +49,17 @@
 #[cfg(not(feature = "std"))]
 #[macro_use]
 extern crate alloc;
-extern crate failure;
-#[macro_use]
-extern crate failure_derive;
-#[cfg_attr(test, macro_use)]
-extern crate target_lexicon;
 
-#[macro_use]
-extern crate log;
-
-pub use context::Context;
-pub use legalizer::legalize_function;
-pub use verifier::verify_function;
-pub use write::write_function;
+pub use crate::context::Context;
+pub use crate::legalizer::legalize_function;
+pub use crate::verifier::verify_function;
+pub use crate::write::write_function;
 
 /// Version number of the cranelift-codegen crate.
 pub const VERSION: &str = env!("CARGO_PKG_VERSION");
 
-#[macro_use]
-pub extern crate cranelift_entity as entity;
-pub extern crate cranelift_bforest as bforest;
+pub use cranelift_bforest as bforest;
+pub use cranelift_entity as entity;
 
 pub mod binemit;
 pub mod cfg_printer;
@@ -85,7 +76,7 @@ pub mod timing;
 pub mod verifier;
 pub mod write;
 
-pub use entity::packed_option;
+pub use crate::entity::packed_option;
 
 mod abi;
 mod bitset;
@@ -111,7 +102,7 @@ mod stack_layout;
 mod topo_order;
 mod unreachable_code;
 
-pub use result::{CodegenError, CodegenResult};
+pub use crate::result::{CodegenError, CodegenResult};
 
 /// This replaces `std` in builds with `core`.
 #[cfg(not(feature = "std"))]
diff --git a/lib/codegen/src/licm.rs b/lib/codegen/src/licm.rs
index b203fdfe9e..101eb47d36 100644
--- a/lib/codegen/src/licm.rs
+++ b/lib/codegen/src/licm.rs
@@ -1,15 +1,15 @@
 //! A Loop Invariant Code Motion optimization pass
 
-use cursor::{Cursor, EncCursor, FuncCursor};
-use dominator_tree::DominatorTree;
-use entity::{EntityList, ListPool};
-use flowgraph::{BasicBlock, ControlFlowGraph};
-use fx::FxHashSet;
-use ir::{DataFlowGraph, Ebb, Function, Inst, InstBuilder, Layout, Opcode, Type, Value};
-use isa::TargetIsa;
-use loop_analysis::{Loop, LoopAnalysis};
+use crate::cursor::{Cursor, EncCursor, FuncCursor};
+use crate::dominator_tree::DominatorTree;
+use crate::entity::{EntityList, ListPool};
+use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::fx::FxHashSet;
+use crate::ir::{DataFlowGraph, Ebb, Function, Inst, InstBuilder, Layout, Opcode, Type, Value};
+use crate::isa::TargetIsa;
+use crate::loop_analysis::{Loop, LoopAnalysis};
+use crate::timing;
 use std::vec::Vec;
-use timing;
 
 /// Performs the LICM pass by detecting loops within the CFG and moving
 /// loop-invariant instructions out of them.
diff --git a/lib/codegen/src/loop_analysis.rs b/lib/codegen/src/loop_analysis.rs
index 3f6490f5fb..ea26332b40 100644
--- a/lib/codegen/src/loop_analysis.rs
+++ b/lib/codegen/src/loop_analysis.rs
@@ -1,14 +1,15 @@
 //! A loop analysis represented as mappings of loops to their header Ebb
 //! and parent in the loop tree.
 
-use dominator_tree::DominatorTree;
-use entity::SecondaryMap;
-use entity::{Keys, PrimaryMap};
-use flowgraph::{BasicBlock, ControlFlowGraph};
-use ir::{Ebb, Function, Layout};
-use packed_option::PackedOption;
+use crate::dominator_tree::DominatorTree;
+use crate::entity::entity_impl;
+use crate::entity::SecondaryMap;
+use crate::entity::{Keys, PrimaryMap};
+use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::ir::{Ebb, Function, Layout};
+use crate::packed_option::PackedOption;
+use crate::timing;
 use std::vec::Vec;
-use timing;
 
 /// A opaque reference to a code loop.
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
@@ -231,11 +232,11 @@ impl LoopAnalysis {
 
 #[cfg(test)]
 mod tests {
-    use cursor::{Cursor, FuncCursor};
-    use dominator_tree::DominatorTree;
-    use flowgraph::ControlFlowGraph;
-    use ir::{types, Function, InstBuilder};
-    use loop_analysis::{Loop, LoopAnalysis};
+    use crate::cursor::{Cursor, FuncCursor};
+    use crate::dominator_tree::DominatorTree;
+    use crate::flowgraph::ControlFlowGraph;
+    use crate::ir::{types, Function, InstBuilder};
+    use crate::loop_analysis::{Loop, LoopAnalysis};
     use std::vec::Vec;
 
     #[test]
diff --git a/lib/codegen/src/nan_canonicalization.rs b/lib/codegen/src/nan_canonicalization.rs
index 0f11c39fe8..235203b764 100644
--- a/lib/codegen/src/nan_canonicalization.rs
+++ b/lib/codegen/src/nan_canonicalization.rs
@@ -2,13 +2,13 @@
 //! instructions that may return a NaN result with a sequence of operations
 //! that will replace nondeterministic NaN's with a single canonical NaN value.
 
-use cursor::{Cursor, FuncCursor};
-use ir::condcodes::FloatCC;
-use ir::immediates::{Ieee32, Ieee64};
-use ir::types;
-use ir::types::Type;
-use ir::{Function, Inst, InstBuilder, InstructionData, Opcode, Value};
-use timing;
+use crate::cursor::{Cursor, FuncCursor};
+use crate::ir::condcodes::FloatCC;
+use crate::ir::immediates::{Ieee32, Ieee64};
+use crate::ir::types;
+use crate::ir::types::Type;
+use crate::ir::{Function, Inst, InstBuilder, InstructionData, Opcode, Value};
+use crate::timing;
 
 // Canonical 32-bit and 64-bit NaN values.
 static CANON_32BIT_NAN: u32 = 0b01111111110000000000000000000000;
diff --git a/lib/codegen/src/postopt.rs b/lib/codegen/src/postopt.rs
index a57497b331..591b92ed03 100644
--- a/lib/codegen/src/postopt.rs
+++ b/lib/codegen/src/postopt.rs
@@ -2,14 +2,14 @@
 
 #![allow(non_snake_case)]
 
-use cursor::{Cursor, EncCursor};
-use ir::condcodes::{CondCode, FloatCC, IntCC};
-use ir::dfg::ValueDef;
-use ir::immediates::{Imm64, Offset32};
-use ir::instructions::{Opcode, ValueList};
-use ir::{Ebb, Function, Inst, InstBuilder, InstructionData, MemFlags, Type, Value};
-use isa::TargetIsa;
-use timing;
+use crate::cursor::{Cursor, EncCursor};
+use crate::ir::condcodes::{CondCode, FloatCC, IntCC};
+use crate::ir::dfg::ValueDef;
+use crate::ir::immediates::{Imm64, Offset32};
+use crate::ir::instructions::{Opcode, ValueList};
+use crate::ir::{Ebb, Function, Inst, InstBuilder, InstructionData, MemFlags, Type, Value};
+use crate::isa::TargetIsa;
+use crate::timing;
 
 /// Information collected about a compare+branch sequence.
 struct CmpBrInfo {
diff --git a/lib/codegen/src/predicates.rs b/lib/codegen/src/predicates.rs
index eca8f2318c..d35b54c6c3 100644
--- a/lib/codegen/src/predicates.rs
+++ b/lib/codegen/src/predicates.rs
@@ -9,7 +9,7 @@
 //! Some of these predicates may be unused in certain ISA configurations, so we suppress the
 //! dead code warning.
 
-use ir;
+use crate::ir;
 
 /// Check that a 64-bit floating point value is zero.
 #[allow(dead_code)]
@@ -93,7 +93,7 @@ mod tests {
 
     #[test]
     fn cvt_imm64() {
-        use ir::immediates::Imm64;
+        use crate::ir::immediates::Imm64;
 
         let x1 = Imm64::new(-8);
         let x2 = Imm64::new(8);
diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index de9fef8844..c4f48ef300 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -1,18 +1,18 @@
 //! Utility routines for pretty-printing error messages.
 
-use entity::SecondaryMap;
-use ir;
-use ir::entities::{AnyEntity, Ebb, Inst, Value};
-use ir::function::Function;
-use isa::TargetIsa;
-use result::CodegenError;
+use crate::entity::SecondaryMap;
+use crate::ir;
+use crate::ir::entities::{AnyEntity, Ebb, Inst, Value};
+use crate::ir::function::Function;
+use crate::isa::TargetIsa;
+use crate::result::CodegenError;
+use crate::verifier::{VerifierError, VerifierErrors};
+use crate::write::{decorate_function, FuncWriter, PlainWriter};
 use std::boxed::Box;
 use std::fmt;
 use std::fmt::Write;
 use std::string::{String, ToString};
 use std::vec::Vec;
-use verifier::{VerifierError, VerifierErrors};
-use write::{decorate_function, FuncWriter, PlainWriter};
 
 /// Pretty-print a verifier error.
 pub fn pretty_verifier_error<'a>(
diff --git a/lib/codegen/src/regalloc/affinity.rs b/lib/codegen/src/regalloc/affinity.rs
index 44d3dbb11b..39d23b31bb 100644
--- a/lib/codegen/src/regalloc/affinity.rs
+++ b/lib/codegen/src/regalloc/affinity.rs
@@ -8,8 +8,8 @@
 //! subclass. This is just a hint, and the register allocator is allowed to pick a register from a
 //! larger register class instead.
 
-use ir::{AbiParam, ArgumentLoc};
-use isa::{ConstraintKind, OperandConstraint, RegClassIndex, RegInfo, TargetIsa};
+use crate::ir::{AbiParam, ArgumentLoc};
+use crate::isa::{ConstraintKind, OperandConstraint, RegClassIndex, RegInfo, TargetIsa};
 use std::fmt;
 
 /// Preferred register allocation for an SSA value.
diff --git a/lib/codegen/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs
index cfdfd984fd..cf290e1f31 100644
--- a/lib/codegen/src/regalloc/coalescing.rs
+++ b/lib/codegen/src/regalloc/coalescing.rs
@@ -5,23 +5,24 @@
 //! possible and inserting copies where necessary such that all argument values passed to an EBB
 //! parameter will belong to the same virtual register as the EBB parameter value itself.
 
-use cursor::{Cursor, EncCursor};
-use dbg::DisplayList;
-use dominator_tree::{DominatorTree, DominatorTreePreorder};
-use flowgraph::{BasicBlock, ControlFlowGraph};
-use fx::FxHashMap;
-use ir::{self, InstBuilder, ProgramOrder};
-use ir::{Ebb, ExpandedProgramPoint, Function, Inst, Value};
-use isa::{EncInfo, TargetIsa};
-use regalloc::affinity::Affinity;
-use regalloc::liveness::Liveness;
-use regalloc::virtregs::{VirtReg, VirtRegs};
+use crate::cursor::{Cursor, EncCursor};
+use crate::dbg::DisplayList;
+use crate::dominator_tree::{DominatorTree, DominatorTreePreorder};
+use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::fx::FxHashMap;
+use crate::ir::{self, InstBuilder, ProgramOrder};
+use crate::ir::{Ebb, ExpandedProgramPoint, Function, Inst, Value};
+use crate::isa::{EncInfo, TargetIsa};
+use crate::regalloc::affinity::Affinity;
+use crate::regalloc::liveness::Liveness;
+use crate::regalloc::virtregs::{VirtReg, VirtRegs};
+use crate::timing;
+use log::debug;
 use std::cmp;
 use std::fmt;
 use std::iter;
 use std::slice;
 use std::vec::Vec;
-use timing;
 
 // # Implementation
 //
diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs
index d040140984..566dd7dee3 100644
--- a/lib/codegen/src/regalloc/coloring.rs
+++ b/lib/codegen/src/regalloc/coloring.rs
@@ -42,22 +42,23 @@
 //!
 //! The exception is the entry block whose arguments are colored from the ABI requirements.
 
-use cursor::{Cursor, EncCursor};
-use dominator_tree::DominatorTree;
-use ir::{AbiParam, ArgumentLoc, InstBuilder, ValueDef};
-use ir::{Ebb, Function, Inst, Layout, SigRef, Value, ValueLoc};
-use isa::{regs_overlap, RegClass, RegInfo, RegUnit};
-use isa::{ConstraintKind, EncInfo, OperandConstraint, RecipeConstraints, TargetIsa};
-use packed_option::PackedOption;
-use regalloc::affinity::Affinity;
-use regalloc::live_value_tracker::{LiveValue, LiveValueTracker};
-use regalloc::liveness::Liveness;
-use regalloc::liverange::{LiveRange, LiveRangeContext};
-use regalloc::register_set::RegisterSet;
-use regalloc::solver::{Solver, SolverError};
-use regalloc::RegDiversions;
+use crate::cursor::{Cursor, EncCursor};
+use crate::dominator_tree::DominatorTree;
+use crate::ir::{AbiParam, ArgumentLoc, InstBuilder, ValueDef};
+use crate::ir::{Ebb, Function, Inst, Layout, SigRef, Value, ValueLoc};
+use crate::isa::{regs_overlap, RegClass, RegInfo, RegUnit};
+use crate::isa::{ConstraintKind, EncInfo, OperandConstraint, RecipeConstraints, TargetIsa};
+use crate::packed_option::PackedOption;
+use crate::regalloc::affinity::Affinity;
+use crate::regalloc::live_value_tracker::{LiveValue, LiveValueTracker};
+use crate::regalloc::liveness::Liveness;
+use crate::regalloc::liverange::{LiveRange, LiveRangeContext};
+use crate::regalloc::register_set::RegisterSet;
+use crate::regalloc::solver::{Solver, SolverError};
+use crate::regalloc::RegDiversions;
+use crate::timing;
+use log::debug;
 use std::mem;
-use timing;
 
 /// Data structures for the coloring pass.
 ///
@@ -901,7 +902,7 @@ impl<'a> Context<'a> {
     /// branch destinations. Branch arguments and EBB parameters are not considered live on the
     /// edge.
     fn is_live_on_outgoing_edge(&self, value: Value) -> bool {
-        use ir::instructions::BranchInfo::*;
+        use crate::ir::instructions::BranchInfo::*;
 
         let inst = self.cur.current_inst().expect("Not on an instruction");
         let ctx = self.liveness.context(&self.cur.func.layout);
@@ -930,7 +931,7 @@ impl<'a> Context<'a> {
     ///
     /// The solver needs to be reminded of the available registers before any moves are inserted.
     fn shuffle_inputs(&mut self, regs: &mut RegisterSet) {
-        use regalloc::solver::Move::*;
+        use crate::regalloc::solver::Move::*;
 
         let spills = self.solver.schedule_moves(regs);
 
diff --git a/lib/codegen/src/regalloc/context.rs b/lib/codegen/src/regalloc/context.rs
index c297250ea7..7ad78743db 100644
--- a/lib/codegen/src/regalloc/context.rs
+++ b/lib/codegen/src/regalloc/context.rs
@@ -4,21 +4,23 @@
 //! the register allocator algorithm. This doesn't preserve any data between functions, but it
 //! avoids allocating data structures independently for each function begin compiled.
 
-use dominator_tree::DominatorTree;
-use flowgraph::ControlFlowGraph;
-use ir::Function;
-use isa::TargetIsa;
-use regalloc::coalescing::Coalescing;
-use regalloc::coloring::Coloring;
-use regalloc::live_value_tracker::LiveValueTracker;
-use regalloc::liveness::Liveness;
-use regalloc::reload::Reload;
-use regalloc::spilling::Spilling;
-use regalloc::virtregs::VirtRegs;
-use result::CodegenResult;
-use timing;
-use topo_order::TopoOrder;
-use verifier::{verify_context, verify_cssa, verify_liveness, verify_locations, VerifierErrors};
+use crate::dominator_tree::DominatorTree;
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir::Function;
+use crate::isa::TargetIsa;
+use crate::regalloc::coalescing::Coalescing;
+use crate::regalloc::coloring::Coloring;
+use crate::regalloc::live_value_tracker::LiveValueTracker;
+use crate::regalloc::liveness::Liveness;
+use crate::regalloc::reload::Reload;
+use crate::regalloc::spilling::Spilling;
+use crate::regalloc::virtregs::VirtRegs;
+use crate::result::CodegenResult;
+use crate::timing;
+use crate::topo_order::TopoOrder;
+use crate::verifier::{
+    verify_context, verify_cssa, verify_liveness, verify_locations, VerifierErrors,
+};
 
 /// Persistent memory allocations for register allocation.
 pub struct Context {
diff --git a/lib/codegen/src/regalloc/diversion.rs b/lib/codegen/src/regalloc/diversion.rs
index 090bf936c9..516f95c610 100644
--- a/lib/codegen/src/regalloc/diversion.rs
+++ b/lib/codegen/src/regalloc/diversion.rs
@@ -7,10 +7,10 @@
 //! These register diversions are local to an EBB. No values can be diverted when entering a new
 //! EBB.
 
-use fx::FxHashMap;
-use ir::{InstructionData, Opcode};
-use ir::{StackSlot, Value, ValueLoc, ValueLocations};
-use isa::{RegInfo, RegUnit};
+use crate::fx::FxHashMap;
+use crate::ir::{InstructionData, Opcode};
+use crate::ir::{StackSlot, Value, ValueLoc, ValueLocations};
+use crate::isa::{RegInfo, RegUnit};
 use std::collections::hash_map::{Entry, Iter};
 use std::fmt;
 
@@ -191,8 +191,8 @@ impl<'a> fmt::Display for DisplayDiversions<'a> {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use entity::EntityRef;
-    use ir::Value;
+    use crate::entity::EntityRef;
+    use crate::ir::Value;
 
     #[test]
     fn inserts() {
diff --git a/lib/codegen/src/regalloc/live_value_tracker.rs b/lib/codegen/src/regalloc/live_value_tracker.rs
index 9fe6e2615a..1b08862e37 100644
--- a/lib/codegen/src/regalloc/live_value_tracker.rs
+++ b/lib/codegen/src/regalloc/live_value_tracker.rs
@@ -4,14 +4,14 @@
 //! The sets of live values are computed on the fly as the tracker is moved from instruction to
 //! instruction, starting at the EBB header.
 
-use dominator_tree::DominatorTree;
-use entity::{EntityList, ListPool};
-use fx::FxHashMap;
-use ir::{DataFlowGraph, Ebb, ExpandedProgramPoint, Inst, Layout, Value};
-use partition_slice::partition_slice;
-use regalloc::affinity::Affinity;
-use regalloc::liveness::Liveness;
-use regalloc::liverange::LiveRange;
+use crate::dominator_tree::DominatorTree;
+use crate::entity::{EntityList, ListPool};
+use crate::fx::FxHashMap;
+use crate::ir::{DataFlowGraph, Ebb, ExpandedProgramPoint, Inst, Layout, Value};
+use crate::partition_slice::partition_slice;
+use crate::regalloc::affinity::Affinity;
+use crate::regalloc::liveness::Liveness;
+use crate::regalloc::liverange::LiveRange;
 use std::vec::Vec;
 
 type ValueList = EntityList;
diff --git a/lib/codegen/src/regalloc/liveness.rs b/lib/codegen/src/regalloc/liveness.rs
index 01f72d9adb..186aaadd60 100644
--- a/lib/codegen/src/regalloc/liveness.rs
+++ b/lib/codegen/src/regalloc/liveness.rs
@@ -175,17 +175,17 @@
 //!
 //! There is some room for improvement.
 
-use entity::SparseMap;
-use flowgraph::{BasicBlock, ControlFlowGraph};
-use ir::dfg::ValueDef;
-use ir::{Ebb, Function, Inst, Layout, ProgramPoint, Value};
-use isa::{EncInfo, OperandConstraint, TargetIsa};
-use regalloc::affinity::Affinity;
-use regalloc::liverange::{LiveRange, LiveRangeContext, LiveRangeForest};
+use crate::entity::SparseMap;
+use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::ir::dfg::ValueDef;
+use crate::ir::{Ebb, Function, Inst, Layout, ProgramPoint, Value};
+use crate::isa::{EncInfo, OperandConstraint, TargetIsa};
+use crate::regalloc::affinity::Affinity;
+use crate::regalloc::liverange::{LiveRange, LiveRangeContext, LiveRangeForest};
+use crate::timing;
 use std::mem;
 use std::ops::Index;
 use std::vec::Vec;
-use timing;
 
 /// A set of live ranges, indexed by value number.
 type LiveRangeSet = SparseMap;
diff --git a/lib/codegen/src/regalloc/liverange.rs b/lib/codegen/src/regalloc/liverange.rs
index 7e56b88fb9..a1012ef1c4 100644
--- a/lib/codegen/src/regalloc/liverange.rs
+++ b/lib/codegen/src/regalloc/liverange.rs
@@ -107,10 +107,10 @@
 //! of coalescing, so we would need to roll our own.
 //!
 
-use bforest;
-use entity::SparseMapValue;
-use ir::{Ebb, ExpandedProgramPoint, Inst, Layout, ProgramOrder, ProgramPoint, Value};
-use regalloc::affinity::Affinity;
+use crate::bforest;
+use crate::entity::SparseMapValue;
+use crate::ir::{Ebb, ExpandedProgramPoint, Inst, Layout, ProgramOrder, ProgramPoint, Value};
+use crate::regalloc::affinity::Affinity;
 use std::cmp::Ordering;
 use std::marker::PhantomData;
 
@@ -457,10 +457,10 @@ impl SparseMapValue for GenLiveRange {
 #[cfg(test)]
 mod tests {
     use super::{GenLiveRange, LiveRangeContext};
-    use bforest;
-    use entity::EntityRef;
-    use ir::{Ebb, Inst, Value};
-    use ir::{ExpandedProgramPoint, ProgramOrder};
+    use crate::bforest;
+    use crate::entity::EntityRef;
+    use crate::ir::{Ebb, Inst, Value};
+    use crate::ir::{ExpandedProgramPoint, ProgramOrder};
     use std::cmp::Ordering;
     use std::vec::Vec;
 
diff --git a/lib/codegen/src/regalloc/pressure.rs b/lib/codegen/src/regalloc/pressure.rs
index a70407bad1..1e45942585 100644
--- a/lib/codegen/src/regalloc/pressure.rs
+++ b/lib/codegen/src/regalloc/pressure.rs
@@ -36,8 +36,8 @@
 // Remove once we're using the pressure tracker.
 #![allow(dead_code)]
 
-use isa::registers::{RegClass, RegClassMask, RegInfo, MAX_TRACKED_TOPRCS};
-use regalloc::RegisterSet;
+use crate::isa::registers::{RegClass, RegClassMask, RegInfo, MAX_TRACKED_TOPRCS};
+use crate::regalloc::RegisterSet;
 use std::cmp::min;
 use std::fmt;
 use std::iter::ExactSizeIterator;
@@ -273,17 +273,17 @@ impl fmt::Display for Pressure {
 #[cfg(build_arm32)]
 mod tests {
     use super::Pressure;
-    use isa::{RegClass, TargetIsa};
-    use regalloc::RegisterSet;
+    use crate::isa::{RegClass, TargetIsa};
+    use crate::regalloc::RegisterSet;
     use std::borrow::Borrow;
     use std::boxed::Box;
     use std::str::FromStr;
-    use target_lexicon;
+    use target_lexicon::triple;
 
     // Make an arm32 `TargetIsa`, if possible.
     fn arm32() -> Option> {
-        use isa;
-        use settings;
+        use crate::isa;
+        use crate::settings;
 
         let shared_builder = settings::builder();
         let shared_flags = settings::Flags::new(shared_builder);
diff --git a/lib/codegen/src/regalloc/register_set.rs b/lib/codegen/src/regalloc/register_set.rs
index 99a3f3e416..df73a3cda0 100644
--- a/lib/codegen/src/regalloc/register_set.rs
+++ b/lib/codegen/src/regalloc/register_set.rs
@@ -5,7 +5,7 @@
 //! "register unit" abstraction. Every register contains one or more register units. Registers that
 //! share a register unit can't be in use at the same time.
 
-use isa::registers::{RegClass, RegInfo, RegUnit, RegUnitMask};
+use crate::isa::registers::{RegClass, RegInfo, RegUnit, RegUnitMask};
 use std::char;
 use std::fmt;
 use std::iter::ExactSizeIterator;
@@ -228,7 +228,7 @@ impl fmt::Display for RegisterSet {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use isa::registers::{RegClass, RegClassData};
+    use crate::isa::registers::{RegClass, RegClassData};
     use std::vec::Vec;
 
     // Register classes for testing.
diff --git a/lib/codegen/src/regalloc/reload.rs b/lib/codegen/src/regalloc/reload.rs
index 8240d7fa51..efbe2204bd 100644
--- a/lib/codegen/src/regalloc/reload.rs
+++ b/lib/codegen/src/regalloc/reload.rs
@@ -9,19 +9,20 @@
 //! possible to minimize the number of `fill` instructions needed. This must not cause the register
 //! pressure limits to be exceeded.
 
-use cursor::{Cursor, EncCursor};
-use dominator_tree::DominatorTree;
-use entity::{SparseMap, SparseMapValue};
-use ir::{AbiParam, ArgumentLoc, InstBuilder};
-use ir::{Ebb, Function, Inst, InstructionData, Opcode, Value};
-use isa::RegClass;
-use isa::{ConstraintKind, EncInfo, Encoding, RecipeConstraints, TargetIsa};
-use regalloc::affinity::Affinity;
-use regalloc::live_value_tracker::{LiveValue, LiveValueTracker};
-use regalloc::liveness::Liveness;
+use crate::cursor::{Cursor, EncCursor};
+use crate::dominator_tree::DominatorTree;
+use crate::entity::{SparseMap, SparseMapValue};
+use crate::ir::{AbiParam, ArgumentLoc, InstBuilder};
+use crate::ir::{Ebb, Function, Inst, InstructionData, Opcode, Value};
+use crate::isa::RegClass;
+use crate::isa::{ConstraintKind, EncInfo, Encoding, RecipeConstraints, TargetIsa};
+use crate::regalloc::affinity::Affinity;
+use crate::regalloc::live_value_tracker::{LiveValue, LiveValueTracker};
+use crate::regalloc::liveness::Liveness;
+use crate::timing;
+use crate::topo_order::TopoOrder;
+use log::debug;
 use std::vec::Vec;
-use timing;
-use topo_order::TopoOrder;
 
 /// Reusable data structures for the reload pass.
 pub struct Reload {
diff --git a/lib/codegen/src/regalloc/solver.rs b/lib/codegen/src/regalloc/solver.rs
index 283a461378..0b433a4587 100644
--- a/lib/codegen/src/regalloc/solver.rs
+++ b/lib/codegen/src/regalloc/solver.rs
@@ -99,11 +99,12 @@
 //! over.
 
 use super::RegisterSet;
-use dbg::DisplayList;
-use entity::{SparseMap, SparseMapValue};
-use ir::Value;
-use isa::{RegClass, RegUnit};
-use regalloc::register_set::RegSetIter;
+use crate::dbg::DisplayList;
+use crate::entity::{SparseMap, SparseMapValue};
+use crate::ir::Value;
+use crate::isa::{RegClass, RegUnit};
+use crate::regalloc::register_set::RegSetIter;
+use log::debug;
 use std::cmp;
 use std::fmt;
 use std::mem;
@@ -1129,18 +1130,18 @@ impl fmt::Display for Solver {
 #[cfg(build_arm32)]
 mod tests {
     use super::{Move, Solver};
-    use entity::EntityRef;
-    use ir::Value;
-    use isa::{RegClass, RegInfo, RegUnit, TargetIsa};
-    use regalloc::RegisterSet;
+    use crate::entity::EntityRef;
+    use crate::ir::Value;
+    use crate::isa::{RegClass, RegInfo, RegUnit, TargetIsa};
+    use crate::regalloc::RegisterSet;
     use std::boxed::Box;
     use std::str::FromStr;
-    use target_lexicon;
+    use target_lexicon::triple;
 
     // Make an arm32 `TargetIsa`, if possible.
     fn arm32() -> Option> {
-        use isa;
-        use settings;
+        use crate::isa;
+        use crate::settings;
 
         let shared_builder = settings::builder();
         let shared_flags = settings::Flags::new(shared_builder);
diff --git a/lib/codegen/src/regalloc/spilling.rs b/lib/codegen/src/regalloc/spilling.rs
index 108d08b095..7bbf26422c 100644
--- a/lib/codegen/src/regalloc/spilling.rs
+++ b/lib/codegen/src/regalloc/spilling.rs
@@ -15,20 +15,21 @@
 //!    be compatible. Otherwise, the value must be copied into a new register for some of the
 //!    operands.
 
-use cursor::{Cursor, EncCursor};
-use dominator_tree::DominatorTree;
-use ir::{ArgumentLoc, Ebb, Function, Inst, InstBuilder, SigRef, Value, ValueLoc};
-use isa::registers::{RegClass, RegClassIndex, RegClassMask, RegUnit};
-use isa::{ConstraintKind, EncInfo, RecipeConstraints, RegInfo, TargetIsa};
-use regalloc::affinity::Affinity;
-use regalloc::live_value_tracker::{LiveValue, LiveValueTracker};
-use regalloc::liveness::Liveness;
-use regalloc::pressure::Pressure;
-use regalloc::virtregs::VirtRegs;
+use crate::cursor::{Cursor, EncCursor};
+use crate::dominator_tree::DominatorTree;
+use crate::ir::{ArgumentLoc, Ebb, Function, Inst, InstBuilder, SigRef, Value, ValueLoc};
+use crate::isa::registers::{RegClass, RegClassIndex, RegClassMask, RegUnit};
+use crate::isa::{ConstraintKind, EncInfo, RecipeConstraints, RegInfo, TargetIsa};
+use crate::regalloc::affinity::Affinity;
+use crate::regalloc::live_value_tracker::{LiveValue, LiveValueTracker};
+use crate::regalloc::liveness::Liveness;
+use crate::regalloc::pressure::Pressure;
+use crate::regalloc::virtregs::VirtRegs;
+use crate::timing;
+use crate::topo_order::TopoOrder;
+use log::debug;
 use std::fmt;
 use std::vec::Vec;
-use timing;
-use topo_order::TopoOrder;
 
 /// Return a top-level register class which contains `unit`.
 fn toprc_containing_regunit(unit: RegUnit, reginfo: &RegInfo) -> RegClass {
diff --git a/lib/codegen/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs
index 63c2e7a413..deb3b5f20b 100644
--- a/lib/codegen/src/regalloc/virtregs.rs
+++ b/lib/codegen/src/regalloc/virtregs.rs
@@ -11,13 +11,14 @@
 //! If any values in a virtual register are spilled, they will use the same stack slot. This avoids
 //! memory-to-memory copies when a spilled value is passed as an EBB argument.
 
-use dbg::DisplayList;
-use dominator_tree::DominatorTreePreorder;
-use entity::{EntityList, ListPool};
-use entity::{Keys, PrimaryMap, SecondaryMap};
-use ir::{Function, Value};
-use packed_option::PackedOption;
-use ref_slice::ref_slice;
+use crate::dbg::DisplayList;
+use crate::dominator_tree::DominatorTreePreorder;
+use crate::entity::entity_impl;
+use crate::entity::{EntityList, ListPool};
+use crate::entity::{Keys, PrimaryMap, SecondaryMap};
+use crate::ir::{Function, Value};
+use crate::packed_option::PackedOption;
+use crate::ref_slice::ref_slice;
 use std::cmp::Ordering;
 use std::fmt;
 use std::vec::Vec;
@@ -399,8 +400,8 @@ impl VirtRegs {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use entity::EntityRef;
-    use ir::Value;
+    use crate::entity::EntityRef;
+    use crate::ir::Value;
 
     #[test]
     fn empty_union_find() {
diff --git a/lib/codegen/src/result.rs b/lib/codegen/src/result.rs
index dace896049..5beb416a94 100644
--- a/lib/codegen/src/result.rs
+++ b/lib/codegen/src/result.rs
@@ -1,6 +1,7 @@
 //! Result and error types representing the outcome of compiling a function.
 
-use verifier::VerifierErrors;
+use crate::verifier::VerifierErrors;
+use failure_derive::Fail;
 
 /// A compilation error.
 ///
diff --git a/lib/codegen/src/scoped_hash_map.rs b/lib/codegen/src/scoped_hash_map.rs
index f7fd992ac0..97d073151b 100644
--- a/lib/codegen/src/scoped_hash_map.rs
+++ b/lib/codegen/src/scoped_hash_map.rs
@@ -4,7 +4,7 @@
 //! container that has a concept of scopes that can be entered and exited, such that
 //! values inserted while inside a scope aren't visible outside the scope.
 
-use fx::FxHashMap;
+use crate::fx::FxHashMap;
 use std::collections::hash_map;
 use std::hash::Hash;
 use std::mem;
diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs
index 50cd38b7fa..a8ce0166f3 100644
--- a/lib/codegen/src/settings.rs
+++ b/lib/codegen/src/settings.rs
@@ -20,8 +20,9 @@
 //! assert_eq!(f.opt_level(), settings::OptLevel::Fastest);
 //! ```
 
-use constant_hash::{probe, simple_hash};
-use isa::TargetIsa;
+use crate::constant_hash::{probe, simple_hash};
+use crate::isa::TargetIsa;
+use failure_derive::Fail;
 use std::boxed::Box;
 use std::fmt;
 use std::str;
@@ -202,7 +203,7 @@ impl<'a> PredicateView<'a> {
 /// This module holds definitions that need to be public so the can be instantiated by generated
 /// code in other modules.
 pub mod detail {
-    use constant_hash;
+    use crate::constant_hash;
     use std::fmt;
 
     /// An instruction group template.
diff --git a/lib/codegen/src/simple_gvn.rs b/lib/codegen/src/simple_gvn.rs
index d60bdb6899..576934071d 100644
--- a/lib/codegen/src/simple_gvn.rs
+++ b/lib/codegen/src/simple_gvn.rs
@@ -1,13 +1,13 @@
 //! A simple GVN pass.
 
-use cursor::{Cursor, FuncCursor};
-use dominator_tree::DominatorTree;
-use ir::{Function, Inst, InstructionData, Opcode, Type};
-use scoped_hash_map::ScopedHashMap;
+use crate::cursor::{Cursor, FuncCursor};
+use crate::dominator_tree::DominatorTree;
+use crate::ir::{Function, Inst, InstructionData, Opcode, Type};
+use crate::scoped_hash_map::ScopedHashMap;
+use crate::timing;
 use std::cell::{Ref, RefCell};
 use std::hash::{Hash, Hasher};
 use std::vec::Vec;
-use timing;
 
 /// Test whether the given opcode is unsafe to even consider for GVN.
 fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool {
@@ -121,7 +121,7 @@ pub fn do_simple_gvn(func: &mut Function, domtree: &mut DominatorTree) {
                 ty: ctrl_typevar,
                 pos: &pos,
             };
-            use scoped_hash_map::Entry::*;
+            use crate::scoped_hash_map::Entry::*;
             match visible_values.entry(key) {
                 Occupied(entry) => {
                     debug_assert!(domtree.dominates(*entry.get(), inst, &func.layout));
diff --git a/lib/codegen/src/simple_preopt.rs b/lib/codegen/src/simple_preopt.rs
index e252659d35..793def27ef 100644
--- a/lib/codegen/src/simple_preopt.rs
+++ b/lib/codegen/src/simple_preopt.rs
@@ -2,15 +2,15 @@
 
 #![allow(non_snake_case)]
 
-use cursor::{Cursor, FuncCursor};
-use divconst_magic_numbers::{magicS32, magicS64, magicU32, magicU64};
-use divconst_magic_numbers::{MS32, MS64, MU32, MU64};
-use ir::dfg::ValueDef;
-use ir::instructions::Opcode;
-use ir::types::{I32, I64};
-use ir::Inst;
-use ir::{DataFlowGraph, Function, InstBuilder, InstructionData, Type, Value};
-use timing;
+use crate::cursor::{Cursor, FuncCursor};
+use crate::divconst_magic_numbers::{magicS32, magicS64, magicU32, magicU64};
+use crate::divconst_magic_numbers::{MS32, MS64, MU32, MU64};
+use crate::ir::dfg::ValueDef;
+use crate::ir::instructions::Opcode;
+use crate::ir::types::{I32, I64};
+use crate::ir::Inst;
+use crate::ir::{DataFlowGraph, Function, InstBuilder, InstructionData, Type, Value};
+use crate::timing;
 
 //----------------------------------------------------------------------
 //
diff --git a/lib/codegen/src/stack_layout.rs b/lib/codegen/src/stack_layout.rs
index 9a34e06374..54eff6a759 100644
--- a/lib/codegen/src/stack_layout.rs
+++ b/lib/codegen/src/stack_layout.rs
@@ -1,8 +1,8 @@
 //! Computing stack layout.
 
-use ir::stackslot::{StackOffset, StackSize, StackSlotKind};
-use ir::StackSlots;
-use result::{CodegenError, CodegenResult};
+use crate::ir::stackslot::{StackOffset, StackSize, StackSlotKind};
+use crate::ir::StackSlots;
+use crate::result::{CodegenError, CodegenResult};
 use std::cmp::{max, min};
 
 /// Compute the stack frame layout.
@@ -111,10 +111,10 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CodegenResu
 #[cfg(test)]
 mod tests {
     use super::layout_stack;
-    use ir::stackslot::StackOffset;
-    use ir::types;
-    use ir::{StackSlotData, StackSlotKind, StackSlots};
-    use result::CodegenError;
+    use crate::ir::stackslot::StackOffset;
+    use crate::ir::types;
+    use crate::ir::{StackSlotData, StackSlotKind, StackSlots};
+    use crate::result::CodegenError;
 
     #[test]
     fn layout() {
diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs
index 285e4102a1..b3aaab1f41 100644
--- a/lib/codegen/src/timing.rs
+++ b/lib/codegen/src/timing.rs
@@ -102,6 +102,7 @@ impl fmt::Display for Pass {
 #[cfg(feature = "std")]
 mod details {
     use super::{Pass, DESCRIPTIONS, NUM_PASSES};
+    use log::debug;
     use std::cell::{Cell, RefCell};
     use std::fmt;
     use std::mem;
diff --git a/lib/codegen/src/topo_order.rs b/lib/codegen/src/topo_order.rs
index aaedaf0e10..9486b054f4 100644
--- a/lib/codegen/src/topo_order.rs
+++ b/lib/codegen/src/topo_order.rs
@@ -1,8 +1,8 @@
 //! Topological order of EBBs, according to the dominator tree.
 
-use dominator_tree::DominatorTree;
-use entity::SparseSet;
-use ir::{Ebb, Layout};
+use crate::dominator_tree::DominatorTree;
+use crate::entity::SparseSet;
+use crate::ir::{Ebb, Layout};
 use std::vec::Vec;
 
 /// Present EBBs in a topological order such that all dominating EBBs are guaranteed to be visited
@@ -90,10 +90,10 @@ impl TopoOrder {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use cursor::{Cursor, FuncCursor};
-    use dominator_tree::DominatorTree;
-    use flowgraph::ControlFlowGraph;
-    use ir::{Function, InstBuilder};
+    use crate::cursor::{Cursor, FuncCursor};
+    use crate::dominator_tree::DominatorTree;
+    use crate::flowgraph::ControlFlowGraph;
+    use crate::ir::{Function, InstBuilder};
     use std::iter;
 
     #[test]
diff --git a/lib/codegen/src/unreachable_code.rs b/lib/codegen/src/unreachable_code.rs
index 74cc6f4ea4..95e2955245 100644
--- a/lib/codegen/src/unreachable_code.rs
+++ b/lib/codegen/src/unreachable_code.rs
@@ -1,10 +1,11 @@
 //! Unreachable code elimination.
 
-use cursor::{Cursor, FuncCursor};
-use dominator_tree::DominatorTree;
-use flowgraph::ControlFlowGraph;
-use ir;
-use timing;
+use crate::cursor::{Cursor, FuncCursor};
+use crate::dominator_tree::DominatorTree;
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir;
+use crate::timing;
+use log::debug;
 
 /// Eliminate unreachable code.
 ///
diff --git a/lib/codegen/src/verifier/cssa.rs b/lib/codegen/src/verifier/cssa.rs
index 3162bfb5e9..6014ff8030 100644
--- a/lib/codegen/src/verifier/cssa.rs
+++ b/lib/codegen/src/verifier/cssa.rs
@@ -1,13 +1,13 @@
 //! Verify conventional SSA form.
 
-use dbg::DisplayList;
-use dominator_tree::{DominatorTree, DominatorTreePreorder};
-use flowgraph::{BasicBlock, ControlFlowGraph};
-use ir::{ExpandedProgramPoint, Function};
-use regalloc::liveness::Liveness;
-use regalloc::virtregs::VirtRegs;
-use timing;
-use verifier::{VerifierErrors, VerifierStepResult};
+use crate::dbg::DisplayList;
+use crate::dominator_tree::{DominatorTree, DominatorTreePreorder};
+use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::ir::{ExpandedProgramPoint, Function};
+use crate::regalloc::liveness::Liveness;
+use crate::regalloc::virtregs::VirtRegs;
+use crate::timing;
+use crate::verifier::{VerifierErrors, VerifierStepResult};
 
 /// Verify conventional SSA form for `func`.
 ///
diff --git a/lib/codegen/src/verifier/flags.rs b/lib/codegen/src/verifier/flags.rs
index 022ba5e1f3..9be8d48280 100644
--- a/lib/codegen/src/verifier/flags.rs
+++ b/lib/codegen/src/verifier/flags.rs
@@ -1,13 +1,13 @@
 //! Verify CPU flags values.
 
-use entity::{SecondaryMap, SparseSet};
-use flowgraph::{BasicBlock, ControlFlowGraph};
-use ir;
-use ir::instructions::BranchInfo;
-use isa;
-use packed_option::PackedOption;
-use timing;
-use verifier::{VerifierErrors, VerifierStepResult};
+use crate::entity::{SecondaryMap, SparseSet};
+use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::ir;
+use crate::ir::instructions::BranchInfo;
+use crate::isa;
+use crate::packed_option::PackedOption;
+use crate::timing;
+use crate::verifier::{VerifierErrors, VerifierStepResult};
 
 /// Verify that CPU flags are used correctly.
 ///
diff --git a/lib/codegen/src/verifier/liveness.rs b/lib/codegen/src/verifier/liveness.rs
index e5fae0585f..c77ac82e16 100644
--- a/lib/codegen/src/verifier/liveness.rs
+++ b/lib/codegen/src/verifier/liveness.rs
@@ -1,14 +1,14 @@
 //! Liveness verifier.
 
-use flowgraph::{BasicBlock, ControlFlowGraph};
-use ir::entities::AnyEntity;
-use ir::{ExpandedProgramPoint, Function, Inst, ProgramOrder, ProgramPoint, Value};
-use isa::TargetIsa;
-use regalloc::liveness::Liveness;
-use regalloc::liverange::LiveRange;
+use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::ir::entities::AnyEntity;
+use crate::ir::{ExpandedProgramPoint, Function, Inst, ProgramOrder, ProgramPoint, Value};
+use crate::isa::TargetIsa;
+use crate::regalloc::liveness::Liveness;
+use crate::regalloc::liverange::LiveRange;
+use crate::timing;
+use crate::verifier::{VerifierErrors, VerifierStepResult};
 use std::cmp::Ordering;
-use timing;
-use verifier::{VerifierErrors, VerifierStepResult};
 
 /// Verify liveness information for `func`.
 ///
diff --git a/lib/codegen/src/verifier/locations.rs b/lib/codegen/src/verifier/locations.rs
index f03a2bed37..26e5edfaf1 100644
--- a/lib/codegen/src/verifier/locations.rs
+++ b/lib/codegen/src/verifier/locations.rs
@@ -1,11 +1,11 @@
 //! Verify value locations.
 
-use ir;
-use isa;
-use regalloc::liveness::Liveness;
-use regalloc::RegDiversions;
-use timing;
-use verifier::{VerifierErrors, VerifierStepResult};
+use crate::ir;
+use crate::isa;
+use crate::regalloc::liveness::Liveness;
+use crate::regalloc::RegDiversions;
+use crate::timing;
+use crate::verifier::{VerifierErrors, VerifierStepResult};
 
 /// Verify value locations for `func`.
 ///
@@ -303,7 +303,7 @@ impl<'a> LocationVerifier<'a> {
         divert: &RegDiversions,
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
-        use ir::instructions::BranchInfo::*;
+        use crate::ir::instructions::BranchInfo::*;
 
         // We can only check CFG edges if we have a liveness analysis.
         let liveness = match self.liveness {
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index adcf9d5a93..1268ed489f 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -57,26 +57,27 @@
 //!   of arguments must match the destination type, and the lane indexes must be in range.
 
 use self::flags::verify_flags;
-use dbg::DisplayList;
-use dominator_tree::DominatorTree;
-use entity::SparseSet;
-use flowgraph::{BasicBlock, ControlFlowGraph};
-use ir;
-use ir::entities::AnyEntity;
-use ir::instructions::{BranchInfo, CallInfo, InstructionFormat, ResolvedConstraint};
-use ir::{
+use crate::dbg::DisplayList;
+use crate::dominator_tree::DominatorTree;
+use crate::entity::SparseSet;
+use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::ir;
+use crate::ir::entities::AnyEntity;
+use crate::ir::instructions::{BranchInfo, CallInfo, InstructionFormat, ResolvedConstraint};
+use crate::ir::{
     types, ArgumentLoc, Ebb, FuncRef, Function, GlobalValue, Inst, JumpTable, Opcode, SigRef,
     StackSlot, StackSlotKind, Type, Value, ValueDef, ValueList, ValueLoc,
 };
-use isa::TargetIsa;
-use iterators::IteratorExtras;
-use settings::FlagsOrIsa;
+use crate::isa::TargetIsa;
+use crate::iterators::IteratorExtras;
+use crate::settings::FlagsOrIsa;
+use crate::timing;
+use failure_derive::Fail;
 use std::cmp::Ordering;
 use std::collections::BTreeSet;
 use std::fmt::{self, Display, Formatter, Write};
 use std::string::String;
 use std::vec::Vec;
-use timing;
 
 pub use self::cssa::verify_cssa;
 pub use self::liveness::verify_liveness;
@@ -90,14 +91,14 @@ pub use self::locations::verify_locations;
 /// as the error message.
 macro_rules! report {
     ( $errors: expr, $loc: expr, $msg: tt ) => {
-        $errors.0.push(::verifier::VerifierError {
+        $errors.0.push(crate::verifier::VerifierError {
             location: $loc.into(),
             message: String::from($msg),
         })
     };
 
     ( $errors: expr, $loc: expr, $fmt: tt, $( $arg: expr ),+ ) => {
-        $errors.0.push(::verifier::VerifierError {
+        $errors.0.push(crate::verifier::VerifierError {
             location: $loc.into(),
             message: format!( $fmt, $( $arg ),+ ),
         })
@@ -551,7 +552,7 @@ impl<'a> Verifier<'a> {
         inst: Inst,
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
-        use ir::instructions::InstructionData::*;
+        use crate::ir::instructions::InstructionData::*;
 
         for &arg in self.func.dfg.inst_args(inst) {
             self.verify_inst_arg(inst, arg, errors)?;
@@ -1699,10 +1700,10 @@ impl<'a> Verifier<'a> {
 #[cfg(test)]
 mod tests {
     use super::{Verifier, VerifierError, VerifierErrors};
-    use entity::EntityList;
-    use ir::instructions::{InstructionData, Opcode};
-    use ir::Function;
-    use settings;
+    use crate::entity::EntityList;
+    use crate::ir::instructions::{InstructionData, Opcode};
+    use crate::ir::Function;
+    use crate::settings;
 
     macro_rules! assert_err_with_msg {
         ($e:expr, $msg:expr) => {
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index 6629f0f3be..c01e7b8bd4 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -3,11 +3,11 @@
 //! The `write` module provides the `write_function` function which converts an IR `Function` to an
 //! equivalent textual form. This textual form can be read back by the `cranelift-reader` crate.
 
-use entity::SecondaryMap;
-use ir::entities::AnyEntity;
-use ir::{DataFlowGraph, Ebb, Function, Inst, SigRef, Type, Value, ValueDef};
-use isa::{RegInfo, TargetIsa};
-use packed_option::ReservedValue;
+use crate::entity::SecondaryMap;
+use crate::ir::entities::AnyEntity;
+use crate::ir::{DataFlowGraph, Ebb, Function, Inst, SigRef, Type, Value, ValueDef};
+use crate::isa::{RegInfo, TargetIsa};
+use crate::packed_option::ReservedValue;
 use std::fmt::{self, Write};
 use std::string::String;
 use std::vec::Vec;
@@ -413,7 +413,7 @@ pub fn write_operands(
     inst: Inst,
 ) -> fmt::Result {
     let pool = &dfg.value_lists;
-    use ir::instructions::InstructionData::*;
+    use crate::ir::instructions::InstructionData::*;
     match dfg[inst] {
         Unary { arg, .. } => write!(w, " {}", arg),
         UnaryImm { imm, .. } => write!(w, " {}", imm),
@@ -663,9 +663,9 @@ impl<'a> fmt::Display for DisplayValuesWithDelimiter<'a> {
 
 #[cfg(test)]
 mod tests {
-    use cursor::{Cursor, CursorPosition, FuncCursor};
-    use ir::types;
-    use ir::{ExternalName, Function, InstBuilder, StackSlotData, StackSlotKind};
+    use crate::cursor::{Cursor, CursorPosition, FuncCursor};
+    use crate::ir::types;
+    use crate::ir::{ExternalName, Function, InstBuilder, StackSlotData, StackSlotKind};
     use std::string::ToString;
 
     #[test]
@@ -714,7 +714,7 @@ mod tests {
 
     #[test]
     fn aliases() {
-        use ir::InstBuilder;
+        use crate::ir::InstBuilder;
 
         let mut func = Function::new();
         {
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index a7eeae5787..ca28428af9 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -9,6 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 categories = ["no-std"]
 readme = "README.md"
 keywords = ["entity", "set", "map"]
+edition = "2018"
 
 [features]
 default = ["std"]
diff --git a/lib/entity/src/boxed_slice.rs b/lib/entity/src/boxed_slice.rs
index 6f9890e7d7..c97522467c 100644
--- a/lib/entity/src/boxed_slice.rs
+++ b/lib/entity/src/boxed_slice.rs
@@ -1,11 +1,11 @@
 //! Boxed slices for `PrimaryMap`.
 
-use iter::{Iter, IterMut};
-use keys::Keys;
+use crate::iter::{Iter, IterMut};
+use crate::keys::Keys;
+use crate::EntityRef;
 use std::marker::PhantomData;
 use std::ops::{Index, IndexMut};
 use std::slice;
-use EntityRef;
 
 /// A slice mapping `K -> V` allocating dense entity references.
 ///
@@ -144,7 +144,7 @@ where
 #[cfg(test)]
 mod tests {
     use super::*;
-    use primary::PrimaryMap;
+    use crate::primary::PrimaryMap;
 
     // `EntityRef` impl for testing.
     #[derive(Clone, Copy, Debug, PartialEq, Eq)]
diff --git a/lib/entity/src/iter.rs b/lib/entity/src/iter.rs
index f818bb0cf5..27053a0962 100644
--- a/lib/entity/src/iter.rs
+++ b/lib/entity/src/iter.rs
@@ -1,9 +1,9 @@
 //! A double-ended iterator over entity references and entities.
 
+use crate::EntityRef;
 use std::iter::Enumerate;
 use std::marker::PhantomData;
 use std::slice;
-use EntityRef;
 
 /// Iterate over all keys in order.
 pub struct Iter<'a, K: EntityRef, V>
diff --git a/lib/entity/src/keys.rs b/lib/entity/src/keys.rs
index e0165f21b7..ea5bd89441 100644
--- a/lib/entity/src/keys.rs
+++ b/lib/entity/src/keys.rs
@@ -3,8 +3,8 @@
 //! When `std::iter::Step` is stabilized, `Keys` could be implemented as a wrapper around
 //! `std::ops::Range`, but for now, we implement it manually.
 
+use crate::EntityRef;
 use std::marker::PhantomData;
-use EntityRef;
 
 /// Iterate over all keys in order.
 pub struct Keys {
diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs
index e86ea0d8fd..cde226690b 100644
--- a/lib/entity/src/lib.rs
+++ b/lib/entity/src/lib.rs
@@ -71,7 +71,7 @@ pub extern crate core as __core;
 pub trait EntityRef: Copy + Eq {
     /// Create a new entity reference from a small integer.
     /// This should crash if the requested index is not representable.
-    fn new(usize) -> Self;
+    fn new(_: usize) -> Self;
 
     /// Get the index that was used to create this entity reference.
     fn index(self) -> usize;
diff --git a/lib/entity/src/list.rs b/lib/entity/src/list.rs
index c2b399da2c..37ac92ad0b 100644
--- a/lib/entity/src/list.rs
+++ b/lib/entity/src/list.rs
@@ -1,8 +1,8 @@
 //! Small lists of entity references.
+use crate::EntityRef;
 use std::marker::PhantomData;
 use std::mem;
 use std::vec::Vec;
-use EntityRef;
 
 /// A small list of entity references allocated from a pool.
 ///
@@ -483,7 +483,7 @@ impl EntityList {
 mod tests {
     use super::*;
     use super::{sclass_for_length, sclass_size};
-    use EntityRef;
+    use crate::EntityRef;
 
     /// An opaque reference to an instruction in a function.
     #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
diff --git a/lib/entity/src/map.rs b/lib/entity/src/map.rs
index c7ced088f7..47e40d0a74 100644
--- a/lib/entity/src/map.rs
+++ b/lib/entity/src/map.rs
@@ -1,12 +1,12 @@
 //! Densely numbered entity references as mapping keys.
 
-use iter::{Iter, IterMut};
-use keys::Keys;
+use crate::iter::{Iter, IterMut};
+use crate::keys::Keys;
+use crate::EntityRef;
 use std::marker::PhantomData;
 use std::ops::{Index, IndexMut};
 use std::slice;
 use std::vec::Vec;
-use EntityRef;
 
 /// A mapping `K -> V` for densely indexed entity references.
 ///
diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs
index 6d710c2e4f..7a45fa9582 100644
--- a/lib/entity/src/primary.rs
+++ b/lib/entity/src/primary.rs
@@ -1,13 +1,13 @@
 //! Densely numbered entity references as mapping keys.
-use boxed_slice::BoxedSlice;
-use iter::{Iter, IterMut};
-use keys::Keys;
+use crate::boxed_slice::BoxedSlice;
+use crate::iter::{Iter, IterMut};
+use crate::keys::Keys;
+use crate::EntityRef;
 use std::iter::FromIterator;
 use std::marker::PhantomData;
 use std::ops::{Index, IndexMut};
 use std::slice;
 use std::vec::Vec;
-use EntityRef;
 
 /// A primary mapping `K -> V` allocating dense entity references.
 ///
diff --git a/lib/entity/src/set.rs b/lib/entity/src/set.rs
index 58f1a7da0a..80765130e6 100644
--- a/lib/entity/src/set.rs
+++ b/lib/entity/src/set.rs
@@ -1,9 +1,9 @@
 //! Densely numbered entity references as set keys.
 
-use keys::Keys;
+use crate::keys::Keys;
+use crate::EntityRef;
 use std::marker::PhantomData;
 use std::vec::Vec;
-use EntityRef;
 
 /// A set of `K` for densely indexed entity references.
 ///
diff --git a/lib/entity/src/sparse.rs b/lib/entity/src/sparse.rs
index b1af505304..a07dc589b1 100644
--- a/lib/entity/src/sparse.rs
+++ b/lib/entity/src/sparse.rs
@@ -7,12 +7,12 @@
 //! > Briggs, Torczon, *An efficient representation for sparse sets*,
 //!   ACM Letters on Programming Languages and Systems, Volume 2, Issue 1-4, March-Dec. 1993.
 
-use map::SecondaryMap;
+use crate::map::SecondaryMap;
+use crate::EntityRef;
 use std::mem;
 use std::slice;
 use std::u32;
 use std::vec::Vec;
-use EntityRef;
 
 /// Trait for extracting keys from values stored in a `SparseMap`.
 ///
@@ -230,7 +230,7 @@ pub type SparseSet = SparseMap;
 #[cfg(test)]
 mod tests {
     use super::*;
-    use EntityRef;
+    use crate::EntityRef;
 
     /// An opaque reference to an instruction in a function.
     #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 003ec93846..0edc0c723c 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -7,6 +7,7 @@ repository = "https://github.com/CraneStation/cranelift"
 documentation = "https://cranelift.readthedocs.io/"
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
+edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.26.0" }
diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index 164d722396..6941fb6312 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -1,6 +1,7 @@
 //! Defines `FaerieBackend`.
 
-use container;
+use crate::container;
+use crate::traps::{FaerieTrapManifest, FaerieTrapSink};
 use cranelift_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSink};
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::{self, binemit, ir};
@@ -12,7 +13,6 @@ use faerie;
 use failure::Error;
 use std::fs::File;
 use target_lexicon::Triple;
-use traps::{FaerieTrapManifest, FaerieTrapSink};
 
 #[derive(Debug)]
 /// Setting to enable collection of traps. Setting this to `Enabled` in
diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs
index f9de8e7ab6..1f7167a8ce 100644
--- a/lib/faerie/src/lib.rs
+++ b/lib/faerie/src/lib.rs
@@ -28,16 +28,9 @@
     )
 )]
 
-extern crate cranelift_codegen;
-extern crate cranelift_module;
-extern crate faerie;
-extern crate failure;
-extern crate goblin;
-extern crate target_lexicon;
-
 mod backend;
 mod container;
 pub mod traps;
 
-pub use backend::{FaerieBackend, FaerieBuilder, FaerieProduct, FaerieTrapCollection};
-pub use container::Format;
+pub use crate::backend::{FaerieBackend, FaerieBuilder, FaerieProduct, FaerieTrapCollection};
+pub use crate::container::Format;
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 84701f2e83..2dc722be82 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -7,6 +7,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
 repository = "https://github.com/CraneStation/cranelift"
 publish = false
+edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.26.0", features = ["testing_hooks"] }
diff --git a/lib/filetests/src/concurrent.rs b/lib/filetests/src/concurrent.rs
index ae04a3b6c5..30e2c94cfe 100644
--- a/lib/filetests/src/concurrent.rs
+++ b/lib/filetests/src/concurrent.rs
@@ -3,18 +3,19 @@
 //! This module provides the `ConcurrentRunner` struct which uses a pool of threads to run tests
 //! concurrently.
 
+use crate::runone;
+use crate::TestResult;
 use cranelift_codegen::dbg::LOG_FILENAME_PREFIX;
 use cranelift_codegen::timing;
 use file_per_thread_logger;
+use log::error;
 use num_cpus;
-use runone;
 use std::panic::catch_unwind;
 use std::path::{Path, PathBuf};
 use std::sync::mpsc::{channel, Receiver, Sender};
 use std::sync::{Arc, Mutex};
 use std::thread;
 use std::time::Duration;
-use TestResult;
 
 /// Request sent to worker threads contains jobid and path.
 struct Request(usize, PathBuf);
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index ccdaab791a..084620e3a0 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -23,18 +23,9 @@
     )
 )]
 
-extern crate cranelift_codegen;
-extern crate cranelift_preopt;
-extern crate cranelift_reader;
-extern crate file_per_thread_logger;
-extern crate filecheck;
-extern crate num_cpus;
-#[macro_use]
-extern crate log;
-
+use crate::runner::TestRunner;
 use cranelift_codegen::timing;
 use cranelift_reader::TestCommand;
-use runner::TestRunner;
 use std::path::Path;
 use std::time;
 
diff --git a/lib/filetests/src/match_directive.rs b/lib/filetests/src/match_directive.rs
index 9a8d94a4df..29c9fa6bae 100644
--- a/lib/filetests/src/match_directive.rs
+++ b/lib/filetests/src/match_directive.rs
@@ -9,7 +9,7 @@ pub fn match_directive<'a>(comment: &'a str, directive: &str) -> Option<&'a str>
         directive.ends_with(':'),
         "Directive must include trailing colon"
     );
-    let text = comment.trim_left_matches(';').trim_left();
+    let text = comment.trim_left_matches(';').trim_start();
     if text.starts_with(directive) {
         Some(text[directive.len()..].trim())
     } else {
diff --git a/lib/filetests/src/runner.rs b/lib/filetests/src/runner.rs
index 37224c2dd3..4c84551dd9 100644
--- a/lib/filetests/src/runner.rs
+++ b/lib/filetests/src/runner.rs
@@ -3,15 +3,15 @@
 //! This module implements the `TestRunner` struct which manages executing tests as well as
 //! scanning directories for tests.
 
-use concurrent::{ConcurrentRunner, Reply};
+use crate::concurrent::{ConcurrentRunner, Reply};
+use crate::runone;
+use crate::TestResult;
 use cranelift_codegen::timing;
-use runone;
 use std::error::Error;
 use std::ffi::OsStr;
 use std::fmt::{self, Display};
 use std::path::{Path, PathBuf};
 use std::time;
-use TestResult;
 
 /// Timeout in seconds when we're not making progress.
 const TIMEOUT_PANIC: usize = 10;
diff --git a/lib/filetests/src/runone.rs b/lib/filetests/src/runone.rs
index 04c1e22f3b..a7b95dc7b5 100644
--- a/lib/filetests/src/runone.rs
+++ b/lib/filetests/src/runone.rs
@@ -1,5 +1,7 @@
 //! Run the tests in a single test file.
 
+use crate::subtest::{Context, SubTest, SubtestResult};
+use crate::{new_subtest, TestResult};
 use cranelift_codegen::ir::Function;
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::print_errors::pretty_verifier_error;
@@ -8,13 +10,12 @@ use cranelift_codegen::timing;
 use cranelift_codegen::verify_function;
 use cranelift_reader::parse_test;
 use cranelift_reader::IsaSpec;
+use log::info;
 use std::borrow::Cow;
 use std::fs;
 use std::io::{self, Read};
 use std::path::Path;
 use std::time;
-use subtest::{Context, SubTest, SubtestResult};
-use {new_subtest, TestResult};
 
 /// Read an entire file into a string.
 fn read_to_string>(path: P) -> io::Result {
diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs
index da8310bfe6..47fa59a2c0 100644
--- a/lib/filetests/src/test_binemit.rs
+++ b/lib/filetests/src/test_binemit.rs
@@ -3,6 +3,8 @@
 //! The `binemit` test command generates binary machine code for every instruction in the input
 //! functions and compares the results to the expected output.
 
+use crate::match_directive::match_directive;
+use crate::subtest::{Context, SubTest, SubtestResult};
 use cranelift_codegen::binemit;
 use cranelift_codegen::binemit::{CodeSink, RegDiversions};
 use cranelift_codegen::dbg::DisplayList;
@@ -11,11 +13,9 @@ use cranelift_codegen::ir::entities::AnyEntity;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_codegen::settings::OptLevel;
 use cranelift_reader::TestCommand;
-use match_directive::match_directive;
 use std::borrow::Cow;
 use std::collections::HashMap;
 use std::fmt::Write;
-use subtest::{Context, SubTest, SubtestResult};
 
 struct TestBinEmit;
 
diff --git a/lib/filetests/src/test_cat.rs b/lib/filetests/src/test_cat.rs
index ad37b444e6..88ebadf11e 100644
--- a/lib/filetests/src/test_cat.rs
+++ b/lib/filetests/src/test_cat.rs
@@ -1,9 +1,9 @@
 //! The `cat` subtest.
 
+use crate::subtest::{self, Context, SubTest, SubtestResult};
 use cranelift_codegen::ir::Function;
 use cranelift_reader::TestCommand;
 use std::borrow::Cow;
-use subtest::{self, Context, SubTest, SubtestResult};
 
 /// Object implementing the `test cat` sub-test.
 ///
diff --git a/lib/filetests/src/test_compile.rs b/lib/filetests/src/test_compile.rs
index 4ff112eab9..7cb9c702a4 100644
--- a/lib/filetests/src/test_compile.rs
+++ b/lib/filetests/src/test_compile.rs
@@ -2,12 +2,13 @@
 //!
 //! The `compile` test command runs each function through the full code generator pipeline
 
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use cranelift_codegen;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_codegen::{binemit, ir};
 use cranelift_reader::TestCommand;
+use log::info;
 use std::borrow::Cow;
-use subtest::{run_filecheck, Context, SubTest, SubtestResult};
 
 struct TestCompile;
 
diff --git a/lib/filetests/src/test_dce.rs b/lib/filetests/src/test_dce.rs
index 6c1645d5fc..2b07654f64 100644
--- a/lib/filetests/src/test_dce.rs
+++ b/lib/filetests/src/test_dce.rs
@@ -5,12 +5,12 @@
 //!
 //! The resulting function is sent to `filecheck`.
 
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use cranelift_codegen;
 use cranelift_codegen::ir::Function;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_reader::TestCommand;
 use std::borrow::Cow;
-use subtest::{run_filecheck, Context, SubTest, SubtestResult};
 
 struct TestDCE;
 
diff --git a/lib/filetests/src/test_domtree.rs b/lib/filetests/src/test_domtree.rs
index 93a1b74a9d..c2683591f2 100644
--- a/lib/filetests/src/test_domtree.rs
+++ b/lib/filetests/src/test_domtree.rs
@@ -12,16 +12,16 @@
 //! We verify that the dominator tree annotations are complete and correct.
 //!
 
+use crate::match_directive::match_directive;
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use cranelift_codegen::dominator_tree::{DominatorTree, DominatorTreePreorder};
 use cranelift_codegen::flowgraph::ControlFlowGraph;
 use cranelift_codegen::ir::entities::AnyEntity;
 use cranelift_codegen::ir::Function;
 use cranelift_reader::TestCommand;
-use match_directive::match_directive;
 use std::borrow::{Borrow, Cow};
 use std::collections::HashMap;
 use std::fmt::{self, Write};
-use subtest::{run_filecheck, Context, SubTest, SubtestResult};
 
 struct TestDomtree;
 
diff --git a/lib/filetests/src/test_legalizer.rs b/lib/filetests/src/test_legalizer.rs
index 10821ee6ab..cb9ab8858c 100644
--- a/lib/filetests/src/test_legalizer.rs
+++ b/lib/filetests/src/test_legalizer.rs
@@ -3,12 +3,12 @@
 //! The `test legalizer` test command runs each function through `legalize_function()` and sends
 //! the result to filecheck.
 
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use cranelift_codegen;
 use cranelift_codegen::ir::Function;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_reader::TestCommand;
 use std::borrow::Cow;
-use subtest::{run_filecheck, Context, SubTest, SubtestResult};
 
 struct TestLegalizer;
 
diff --git a/lib/filetests/src/test_licm.rs b/lib/filetests/src/test_licm.rs
index 6c4ef0795d..32360cadc7 100644
--- a/lib/filetests/src/test_licm.rs
+++ b/lib/filetests/src/test_licm.rs
@@ -5,12 +5,12 @@
 //!
 //! The resulting function is sent to `filecheck`.
 
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use cranelift_codegen;
 use cranelift_codegen::ir::Function;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_reader::TestCommand;
 use std::borrow::Cow;
-use subtest::{run_filecheck, Context, SubTest, SubtestResult};
 
 struct TestLICM;
 
diff --git a/lib/filetests/src/test_postopt.rs b/lib/filetests/src/test_postopt.rs
index 4789fd00c7..e7919b13ca 100644
--- a/lib/filetests/src/test_postopt.rs
+++ b/lib/filetests/src/test_postopt.rs
@@ -2,12 +2,12 @@
 //!
 //! The resulting function is sent to `filecheck`.
 
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use cranelift_codegen;
 use cranelift_codegen::ir::Function;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_reader::TestCommand;
 use std::borrow::Cow;
-use subtest::{run_filecheck, Context, SubTest, SubtestResult};
 
 struct TestPostopt;
 
diff --git a/lib/filetests/src/test_preopt.rs b/lib/filetests/src/test_preopt.rs
index 43566bdfa5..2da3005df6 100644
--- a/lib/filetests/src/test_preopt.rs
+++ b/lib/filetests/src/test_preopt.rs
@@ -5,13 +5,13 @@
 //!
 //! The resulting function is sent to `filecheck`.
 
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use cranelift_codegen;
 use cranelift_codegen::ir::Function;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_preopt::optimize;
 use cranelift_reader::TestCommand;
 use std::borrow::Cow;
-use subtest::{run_filecheck, Context, SubTest, SubtestResult};
 
 struct TestPreopt;
 
diff --git a/lib/filetests/src/test_print_cfg.rs b/lib/filetests/src/test_print_cfg.rs
index 4de4a3e7f4..7053234dc3 100644
--- a/lib/filetests/src/test_print_cfg.rs
+++ b/lib/filetests/src/test_print_cfg.rs
@@ -5,10 +5,10 @@
 
 use std::borrow::Cow;
 
+use crate::subtest::{self, Context, SubTest, SubtestResult};
 use cranelift_codegen::cfg_printer::CFGPrinter;
 use cranelift_codegen::ir::Function;
 use cranelift_reader::TestCommand;
-use subtest::{self, Context, SubTest, SubtestResult};
 
 /// Object implementing the `test print-cfg` sub-test.
 struct TestPrintCfg;
diff --git a/lib/filetests/src/test_regalloc.rs b/lib/filetests/src/test_regalloc.rs
index e09623a691..a813b43e2f 100644
--- a/lib/filetests/src/test_regalloc.rs
+++ b/lib/filetests/src/test_regalloc.rs
@@ -5,12 +5,12 @@
 //!
 //! The resulting function is sent to `filecheck`.
 
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use cranelift_codegen;
 use cranelift_codegen::ir::Function;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_reader::TestCommand;
 use std::borrow::Cow;
-use subtest::{run_filecheck, Context, SubTest, SubtestResult};
 
 struct TestRegalloc;
 
diff --git a/lib/filetests/src/test_shrink.rs b/lib/filetests/src/test_shrink.rs
index 7d909c6187..2cbd2b614f 100644
--- a/lib/filetests/src/test_shrink.rs
+++ b/lib/filetests/src/test_shrink.rs
@@ -5,12 +5,12 @@
 //!
 //! The resulting function is sent to `filecheck`.
 
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use cranelift_codegen;
 use cranelift_codegen::ir::Function;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_reader::TestCommand;
 use std::borrow::Cow;
-use subtest::{run_filecheck, Context, SubTest, SubtestResult};
 
 struct TestShrink;
 
diff --git a/lib/filetests/src/test_simple_gvn.rs b/lib/filetests/src/test_simple_gvn.rs
index 5b664a4f74..3752129a92 100644
--- a/lib/filetests/src/test_simple_gvn.rs
+++ b/lib/filetests/src/test_simple_gvn.rs
@@ -5,12 +5,12 @@
 //!
 //! The resulting function is sent to `filecheck`.
 
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use cranelift_codegen;
 use cranelift_codegen::ir::Function;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_reader::TestCommand;
 use std::borrow::Cow;
-use subtest::{run_filecheck, Context, SubTest, SubtestResult};
 
 struct TestSimpleGVN;
 
diff --git a/lib/filetests/src/test_simple_preopt.rs b/lib/filetests/src/test_simple_preopt.rs
index 803aa944e7..8b2ea6999c 100644
--- a/lib/filetests/src/test_simple_preopt.rs
+++ b/lib/filetests/src/test_simple_preopt.rs
@@ -2,12 +2,12 @@
 //!
 //! The resulting function is sent to `filecheck`.
 
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use cranelift_codegen;
 use cranelift_codegen::ir::Function;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_reader::TestCommand;
 use std::borrow::Cow;
-use subtest::{run_filecheck, Context, SubTest, SubtestResult};
 
 struct TestSimplePreopt;
 
diff --git a/lib/filetests/src/test_verifier.rs b/lib/filetests/src/test_verifier.rs
index 90dcad7e20..7e057248fb 100644
--- a/lib/filetests/src/test_verifier.rs
+++ b/lib/filetests/src/test_verifier.rs
@@ -9,13 +9,13 @@
 //! This annotation means that the verifier is expected to given an error for the jump instruction
 //! containing the substring "jump to non-existent EBB".
 
+use crate::match_directive::match_directive;
+use crate::subtest::{Context, SubTest, SubtestResult};
 use cranelift_codegen::ir::Function;
 use cranelift_codegen::verify_function;
 use cranelift_reader::TestCommand;
-use match_directive::match_directive;
 use std::borrow::{Borrow, Cow};
 use std::fmt::Write;
-use subtest::{Context, SubTest, SubtestResult};
 
 struct TestVerifier;
 
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 87791458e9..826edf278e 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -8,6 +8,7 @@ documentation = "https://cranelift.readthedocs.io/"
 categories = ["no-std"]
 repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
+edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index d3c364b0d6..2ed09fbd01 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -1,4 +1,6 @@
 //! A frontend for building Cranelift IR from other languages.
+use crate::ssa::{Block, SSABuilder, SideEffects};
+use crate::variable::Variable;
 use cranelift_codegen::cursor::{Cursor, FuncCursor};
 use cranelift_codegen::entity::{EntitySet, SecondaryMap};
 use cranelift_codegen::ir;
@@ -11,9 +13,7 @@ use cranelift_codegen::ir::{
 };
 use cranelift_codegen::isa::{TargetFrontendConfig, TargetIsa};
 use cranelift_codegen::packed_option::PackedOption;
-use ssa::{Block, SSABuilder, SideEffects};
 use std::vec::Vec;
-use variable::Variable;
 
 /// Structure used for translating a series of functions into Cranelift IR.
 ///
@@ -822,15 +822,15 @@ impl<'a> FunctionBuilder<'a> {
 #[cfg(test)]
 mod tests {
     use super::greatest_divisible_power_of_two;
+    use crate::frontend::{FunctionBuilder, FunctionBuilderContext};
+    use crate::Variable;
     use cranelift_codegen::entity::EntityRef;
     use cranelift_codegen::ir::types::*;
     use cranelift_codegen::ir::{AbiParam, ExternalName, Function, InstBuilder, Signature};
     use cranelift_codegen::isa::CallConv;
     use cranelift_codegen::settings;
     use cranelift_codegen::verifier::verify_function;
-    use frontend::{FunctionBuilder, FunctionBuilderContext};
     use std::string::ToString;
-    use Variable;
 
     fn sample_function(lazy_seal: bool) {
         let mut sig = Signature::new(CallConv::SystemV);
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index 97514a75f2..3f8ec616cc 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -180,15 +180,10 @@
 #[cfg(not(feature = "std"))]
 #[macro_use]
 extern crate alloc;
-extern crate cranelift_codegen;
-#[cfg(test)]
-extern crate target_lexicon;
-#[macro_use]
-extern crate log;
 
-pub use frontend::{FunctionBuilder, FunctionBuilderContext};
-pub use switch::Switch;
-pub use variable::Variable;
+pub use crate::frontend::{FunctionBuilder, FunctionBuilderContext};
+pub use crate::switch::Switch;
+pub use crate::variable::Variable;
 
 mod frontend;
 mod ssa;
diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs
index 15ba1c98ca..1046e960a2 100644
--- a/lib/frontend/src/ssa.rs
+++ b/lib/frontend/src/ssa.rs
@@ -5,6 +5,7 @@
 //! In: Jhala R., De Bosschere K. (eds) Compiler Construction. CC 2013.
 //! Lecture Notes in Computer Science, vol 7791. Springer, Berlin, Heidelberg
 
+use crate::Variable;
 use cranelift_codegen::cursor::{Cursor, FuncCursor};
 use cranelift_codegen::entity::{EntityRef, PrimaryMap, SecondaryMap};
 use cranelift_codegen::ir::immediates::{Ieee32, Ieee64};
@@ -16,7 +17,6 @@ use cranelift_codegen::packed_option::ReservedValue;
 use std::mem;
 use std::u32;
 use std::vec::Vec;
-use Variable;
 
 /// Structure containing the data relevant the construction of SSA for a given function.
 ///
@@ -749,6 +749,8 @@ impl SSABuilder {
 
 #[cfg(test)]
 mod tests {
+    use crate::ssa::SSABuilder;
+    use crate::Variable;
     use cranelift_codegen::cursor::{Cursor, FuncCursor};
     use cranelift_codegen::entity::EntityRef;
     use cranelift_codegen::ir::instructions::BranchInfo;
@@ -756,8 +758,6 @@ mod tests {
     use cranelift_codegen::ir::{Function, Inst, InstBuilder, JumpTableData, Opcode};
     use cranelift_codegen::settings;
     use cranelift_codegen::verify_function;
-    use ssa::SSABuilder;
-    use Variable;
 
     #[test]
     fn simple_block() {
diff --git a/lib/frontend/src/switch.rs b/lib/frontend/src/switch.rs
index ce3a2d943f..d5af64ed8c 100644
--- a/lib/frontend/src/switch.rs
+++ b/lib/frontend/src/switch.rs
@@ -1,6 +1,7 @@
+use crate::frontend::FunctionBuilder;
 use cranelift_codegen::ir::condcodes::IntCC;
 use cranelift_codegen::ir::*;
-use frontend::FunctionBuilder;
+use log::debug;
 use std::collections::HashMap;
 use std::vec::Vec;
 
@@ -200,8 +201,8 @@ impl ContiguousCaseRange {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::frontend::FunctionBuilderContext;
     use cranelift_codegen::ir::Function;
-    use frontend::FunctionBuilderContext;
     use std::string::ToString;
 
     macro_rules! setup {
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 1694d065b2..826a00cf72 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -8,6 +8,7 @@ documentation = "https://cranelift.readthedocs.io/"
 categories = ["no-std"]
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
+edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs
index 8138009341..37e235edb7 100644
--- a/lib/module/src/backend.rs
+++ b/lib/module/src/backend.rs
@@ -1,13 +1,13 @@
 //! Defines the `Backend` trait.
 
+use crate::DataContext;
+use crate::Linkage;
+use crate::ModuleNamespace;
+use crate::ModuleResult;
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::Context;
 use cranelift_codegen::{binemit, ir};
 use std::marker;
-use DataContext;
-use Linkage;
-use ModuleNamespace;
-use ModuleResult;
 
 /// A `Backend` implements the functionality needed to support a `Module`.
 ///
@@ -46,7 +46,7 @@ where
     type Product;
 
     /// Create a new `Backend` instance.
-    fn new(Self::Builder) -> Self;
+    fn new(_: Self::Builder) -> Self;
 
     /// Return the `TargetIsa` to compile for.
     fn isa(&self) -> &TargetIsa;
diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs
index 7e31dc9f3c..bd6ca28c8a 100644
--- a/lib/module/src/lib.rs
+++ b/lib/module/src/lib.rs
@@ -29,21 +29,13 @@
 #[cfg_attr(test, macro_use)]
 extern crate alloc;
 
-extern crate cranelift_codegen;
-#[macro_use]
-extern crate cranelift_entity;
-#[macro_use]
-extern crate failure;
-#[macro_use]
-extern crate log;
-
 mod backend;
 mod data_context;
 mod module;
 
-pub use backend::Backend;
-pub use data_context::{DataContext, DataDescription, Init};
-pub use module::{
+pub use crate::backend::Backend;
+pub use crate::data_context::{DataContext, DataDescription, Init};
+pub use crate::module::{
     DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleError, ModuleNamespace, ModuleResult,
 };
 
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index e070514c8a..1d5a33130a 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -5,14 +5,16 @@
 // TODO: Factor out `ir::Function`'s `ext_funcs` and `global_values` into a struct
 // shared with `DataContext`?
 
-use cranelift_codegen::entity::PrimaryMap;
+use crate::data_context::DataContext;
+use crate::Backend;
+use cranelift_codegen::entity::{entity_impl, PrimaryMap};
 use cranelift_codegen::{binemit, ir, isa, CodegenError, Context};
-use data_context::DataContext;
+use failure::Fail;
+use log::info;
 use std::borrow::ToOwned;
 use std::collections::HashMap;
 use std::string::String;
 use std::vec::Vec;
-use Backend;
 
 /// A function identifier for use in the `Module` interface.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 20fc2911df..f73357d938 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -7,6 +7,7 @@ repository = "https://github.com/CraneStation/cranelift"
 categories = ["no-std"]
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
+edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs
index c62322afe9..24650ca28c 100644
--- a/lib/native/src/lib.rs
+++ b/lib/native/src/lib.rs
@@ -28,11 +28,6 @@
 )]
 #![cfg_attr(not(feature = "std"), no_std)]
 
-extern crate cranelift_codegen;
-#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
-extern crate raw_cpuid;
-extern crate target_lexicon;
-
 use cranelift_codegen::isa;
 use cranelift_codegen::settings::Configurable;
 use target_lexicon::Triple;
diff --git a/lib/preopt/Cargo.toml b/lib/preopt/Cargo.toml
index f9337faa74..08c492f2f0 100644
--- a/lib/preopt/Cargo.toml
+++ b/lib/preopt/Cargo.toml
@@ -9,6 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 categories = ["no-std"]
 readme = "README.md"
 keywords = ["optimize", "compile", "compiler", "jit"]
+edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
diff --git a/lib/preopt/src/lib.rs b/lib/preopt/src/lib.rs
index 9f307c4dfd..321b7a58b6 100644
--- a/lib/preopt/src/lib.rs
+++ b/lib/preopt/src/lib.rs
@@ -27,9 +27,6 @@
 #[cfg(not(feature = "std"))]
 extern crate alloc;
 
-extern crate cranelift_codegen;
-// extern crate rustc_apfloat;
-
 mod constant_folding;
 
 use cranelift_codegen::{isa::TargetIsa, settings::FlagsOrIsa, CodegenResult, Context};
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 8d4b70b975..c183e689b3 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -7,6 +7,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
 repository = "https://github.com/CraneStation/cranelift"
 readme = "README.md"
+edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.26.0" }
diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs
index 9132395162..6b58982c3a 100644
--- a/lib/reader/src/isaspec.rs
+++ b/lib/reader/src/isaspec.rs
@@ -6,10 +6,10 @@
 //! If a test case file contains `isa` commands, the tests will only be run against the specified
 //! ISAs. If the file contains no `isa` commands, the tests will be run against all supported ISAs.
 
+use crate::error::{Location, ParseResult};
+use crate::testcommand::TestOption;
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::settings::{Configurable, Flags, SetError};
-use error::{Location, ParseResult};
-use testcommand::TestOption;
 
 /// The ISA specifications in a `.clif` file.
 pub enum IsaSpec {
diff --git a/lib/reader/src/lexer.rs b/lib/reader/src/lexer.rs
index 1a77a1814e..8673b2b922 100644
--- a/lib/reader/src/lexer.rs
+++ b/lib/reader/src/lexer.rs
@@ -1,8 +1,8 @@
 //! Lexical analysis for .clif files.
 
+use crate::error::Location;
 use cranelift_codegen::ir::types;
 use cranelift_codegen::ir::{Ebb, Value};
-use error::Location;
 #[allow(unused_imports, deprecated)]
 use std::ascii::AsciiExt;
 use std::str::CharIndices;
@@ -481,9 +481,9 @@ impl<'a> Lexer<'a> {
 mod tests {
     use super::trailing_digits;
     use super::*;
+    use crate::error::Location;
     use cranelift_codegen::ir::types;
     use cranelift_codegen::ir::{Ebb, Value};
-    use error::Location;
 
     #[test]
     fn digits() {
diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs
index ca79ea2099..d2e6a96e8b 100644
--- a/lib/reader/src/lib.rs
+++ b/lib/reader/src/lib.rs
@@ -29,15 +29,12 @@
     )
 )]
 
-extern crate cranelift_codegen;
-extern crate target_lexicon;
-
-pub use error::{Location, ParseError, ParseResult};
-pub use isaspec::{parse_options, IsaSpec};
-pub use parser::{parse_functions, parse_test};
-pub use sourcemap::SourceMap;
-pub use testcommand::{TestCommand, TestOption};
-pub use testfile::{Comment, Details, TestFile};
+pub use crate::error::{Location, ParseError, ParseResult};
+pub use crate::isaspec::{parse_options, IsaSpec};
+pub use crate::parser::{parse_functions, parse_test};
+pub use crate::sourcemap::SourceMap;
+pub use crate::testcommand::{TestCommand, TestOption};
+pub use crate::testfile::{Comment, Details, TestFile};
 
 mod error;
 mod isaspec;
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 3bcac20aca..7503805bb6 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -1,5 +1,11 @@
 //! Parser for .clif files.
 
+use crate::error::{Location, ParseError, ParseResult};
+use crate::isaspec;
+use crate::lexer::{LexError, Lexer, LocatedError, LocatedToken, Token};
+use crate::sourcemap::SourceMap;
+use crate::testcommand::TestCommand;
+use crate::testfile::{Comment, Details, TestFile};
 use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::entities::AnyEntity;
@@ -15,16 +21,10 @@ use cranelift_codegen::ir::{
 use cranelift_codegen::isa::{self, CallConv, Encoding, RegUnit, TargetIsa};
 use cranelift_codegen::packed_option::ReservedValue;
 use cranelift_codegen::{settings, timing};
-use error::{Location, ParseError, ParseResult};
-use isaspec;
-use lexer::{LexError, Lexer, LocatedError, LocatedToken, Token};
-use sourcemap::SourceMap;
 use std::mem;
 use std::str::FromStr;
 use std::{u16, u32};
 use target_lexicon::Triple;
-use testcommand::TestCommand;
-use testfile::{Comment, Details, TestFile};
 
 /// Parse the entire `text` into a list of functions.
 ///
@@ -793,7 +793,7 @@ impl<'a> Parser<'a> {
                 Ok(triple) => triple,
                 Err(err) => return err!(loc, err),
             };
-            let mut isa_builder = match isa::lookup(triple) {
+            let isa_builder = match isa::lookup(triple) {
                 Err(isa::LookupError::SupportDisabled) => {
                     return err!(loc, "support disabled target '{}'", targ)
                 }
@@ -2571,14 +2571,14 @@ impl<'a> Parser<'a> {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::error::ParseError;
+    use crate::isaspec::IsaSpec;
+    use crate::testfile::{Comment, Details};
     use cranelift_codegen::ir::entities::AnyEntity;
     use cranelift_codegen::ir::types;
     use cranelift_codegen::ir::StackSlotKind;
     use cranelift_codegen::ir::{ArgumentExtension, ArgumentPurpose};
     use cranelift_codegen::isa::CallConv;
-    use error::ParseError;
-    use isaspec::IsaSpec;
-    use testfile::{Comment, Details};
 
     #[test]
     fn argument_type() {
diff --git a/lib/reader/src/sourcemap.rs b/lib/reader/src/sourcemap.rs
index f3f0423e43..a398dd5849 100644
--- a/lib/reader/src/sourcemap.rs
+++ b/lib/reader/src/sourcemap.rs
@@ -6,12 +6,12 @@
 //! The `SourceMap` struct defined in this module makes this mapping available
 //! to parser clients.
 
+use crate::error::{Location, ParseResult};
+use crate::lexer::split_entity_name;
 use cranelift_codegen::ir::entities::AnyEntity;
 use cranelift_codegen::ir::{
     Ebb, FuncRef, GlobalValue, Heap, JumpTable, SigRef, StackSlot, Table, Value,
 };
-use error::{Location, ParseResult};
-use lexer::split_entity_name;
 use std::collections::HashMap;
 
 /// Mapping from entity names to source locations.
@@ -211,7 +211,7 @@ impl SourceMap {
 
 #[cfg(test)]
 mod tests {
-    use parse_test;
+    use crate::parse_test;
 
     #[test]
     fn details() {
diff --git a/lib/reader/src/testfile.rs b/lib/reader/src/testfile.rs
index 685b7d99e6..506694586f 100644
--- a/lib/reader/src/testfile.rs
+++ b/lib/reader/src/testfile.rs
@@ -4,12 +4,12 @@
 //! file-based test case.
 //!
 
+use crate::error::Location;
+use crate::isaspec::IsaSpec;
+use crate::sourcemap::SourceMap;
+use crate::testcommand::TestCommand;
 use cranelift_codegen::ir::entities::AnyEntity;
 use cranelift_codegen::ir::Function;
-use error::Location;
-use isaspec::IsaSpec;
-use sourcemap::SourceMap;
-use testcommand::TestCommand;
 
 /// A parsed test case.
 ///
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index 086cfbe5a5..a8a34e64b2 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -8,6 +8,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 categories = ["no-std"]
 readme = "README.md"
 keywords = ["webassembly", "serde"]
+edition = "2018"
 
 [[bin]]
 name = "clif-json"
diff --git a/lib/serde/src/clif-json.rs b/lib/serde/src/clif-json.rs
index f06a9e552b..fa25a723ed 100644
--- a/lib/serde/src/clif-json.rs
+++ b/lib/serde/src/clif-json.rs
@@ -25,13 +25,6 @@
     )
 )]
 
-extern crate clap;
-extern crate cranelift_reader;
-extern crate serde_json;
-#[macro_use]
-extern crate serde_derive;
-extern crate cranelift_codegen;
-
 use clap::{App, Arg, SubCommand};
 use cranelift_reader::parse_functions;
 use std::fs::File;
diff --git a/lib/serde/src/serde_clif_json.rs b/lib/serde/src/serde_clif_json.rs
index 536dda6a3b..f771fc7438 100644
--- a/lib/serde/src/serde_clif_json.rs
+++ b/lib/serde/src/serde_clif_json.rs
@@ -1,4 +1,5 @@
 use cranelift_codegen::ir::{Ebb, Function, Inst, InstructionData, Signature};
+use serde_derive::{Deserialize, Serialize};
 
 /// Serializable version of the original Cranelift IR
 #[derive(Clone, Debug, Hash, PartialEq, Eq, Serialize, Deserialize)]
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 9ddbed39e8..440c4072f1 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -8,6 +8,7 @@ documentation = "https://cranelift.readthedocs.io/"
 categories = ["no-std"]
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
+edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
diff --git a/lib/simplejit/examples/simplejit-minimal.rs b/lib/simplejit/examples/simplejit-minimal.rs
index 4e2d156bc8..eb1d3257ac 100644
--- a/lib/simplejit/examples/simplejit-minimal.rs
+++ b/lib/simplejit/examples/simplejit-minimal.rs
@@ -1,7 +1,3 @@
-extern crate cranelift;
-extern crate cranelift_module;
-extern crate cranelift_simplejit;
-
 use cranelift::prelude::*;
 use cranelift_module::{Linkage, Module};
 use cranelift_simplejit::{SimpleJITBackend, SimpleJITBuilder};
diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index b105e70e3c..b7582c9c7b 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -1,5 +1,6 @@
 //! Defines `SimpleJITBackend`.
 
+use crate::memory::Memory;
 use cranelift_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSink};
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::{self, ir, settings};
@@ -8,7 +9,6 @@ use cranelift_module::{
 };
 use cranelift_native;
 use libc;
-use memory::Memory;
 use std::collections::HashMap;
 use std::ffi::CString;
 use std::io::Write;
diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs
index c1a44f672d..eabeb7ac0f 100644
--- a/lib/simplejit/src/lib.rs
+++ b/lib/simplejit/src/lib.rs
@@ -26,18 +26,7 @@
     )
 )]
 
-extern crate cranelift_codegen;
-extern crate cranelift_module;
-extern crate cranelift_native;
-extern crate errno;
-extern crate libc;
-extern crate region;
-extern crate target_lexicon;
-
-#[cfg(target_os = "windows")]
-extern crate winapi;
-
 mod backend;
 mod memory;
 
-pub use backend::{SimpleJITBackend, SimpleJITBuilder};
+pub use crate::backend::{SimpleJITBackend, SimpleJITBuilder};
diff --git a/lib/simplejit/tests/basic.rs b/lib/simplejit/tests/basic.rs
index 9128ef6068..38da80c9a2 100644
--- a/lib/simplejit/tests/basic.rs
+++ b/lib/simplejit/tests/basic.rs
@@ -1,9 +1,3 @@
-extern crate cranelift_codegen;
-extern crate cranelift_entity;
-extern crate cranelift_frontend;
-extern crate cranelift_module;
-extern crate cranelift_simplejit;
-
 use cranelift_codegen::ir::*;
 use cranelift_codegen::isa::CallConv;
 use cranelift_codegen::Context;
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index 1ca70dcac1..eb80082dd6 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -9,6 +9,7 @@ repository = "https://github.com/CraneStation/cranelift"
 categories = ["no-std"]
 readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
+edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs
index b3aaa3e7ca..c14134e4f4 100644
--- a/lib/umbrella/src/lib.rs
+++ b/lib/umbrella/src/lib.rs
@@ -27,23 +27,23 @@
 )]
 
 /// Provide these crates, renamed to reduce stutter.
-pub extern crate cranelift_codegen as codegen;
-pub extern crate cranelift_frontend as frontend;
+pub use cranelift_codegen as codegen;
+pub use cranelift_frontend as frontend;
 
 /// A prelude providing convenient access to commonly-used cranelift features. Use
 /// as `use cranelift::prelude::*`.
 pub mod prelude {
-    pub use codegen;
-    pub use codegen::entity::EntityRef;
-    pub use codegen::ir::condcodes::{FloatCC, IntCC};
-    pub use codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Uimm64};
-    pub use codegen::ir::types;
-    pub use codegen::ir::{
+    pub use crate::codegen;
+    pub use crate::codegen::entity::EntityRef;
+    pub use crate::codegen::ir::condcodes::{FloatCC, IntCC};
+    pub use crate::codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Uimm64};
+    pub use crate::codegen::ir::types;
+    pub use crate::codegen::ir::{
         AbiParam, Ebb, ExtFuncData, ExternalName, GlobalValueData, InstBuilder, JumpTableData,
         MemFlags, Signature, StackSlotData, StackSlotKind, TrapCode, Type, Value,
     };
-    pub use codegen::isa;
-    pub use codegen::settings::{self, Configurable};
+    pub use crate::codegen::isa;
+    pub use crate::codegen::settings::{self, Configurable};
 
-    pub use frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
+    pub use crate::frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
 }
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 64a4716981..71b4b2547e 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -8,6 +8,7 @@ license = "Apache-2.0 WITH LLVM-exception"
 categories = ["no-std", "wasm"]
 readme = "README.md"
 keywords = ["webassembly", "wasm"]
+edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.23.0", default-features = false }
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index c7683d8dab..a7b61fc580 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -22,18 +22,18 @@
 //!
 //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as
 //! argument.
+use crate::environ::{FuncEnvironment, GlobalVariable, ReturnMode, WasmError, WasmResult};
+use crate::state::{ControlStackFrame, TranslationState};
+use crate::translation_utils::{f32_translation, f64_translation, num_return_values, type_to_type};
+use crate::translation_utils::{FuncIndex, MemoryIndex, SignatureIndex, TableIndex};
 use cranelift_codegen::ir::condcodes::{FloatCC, IntCC};
 use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags};
 use cranelift_codegen::packed_option::ReservedValue;
 use cranelift_frontend::{FunctionBuilder, Variable};
-use environ::{FuncEnvironment, GlobalVariable, ReturnMode, WasmError, WasmResult};
-use state::{ControlStackFrame, TranslationState};
 use std::collections::{hash_map, HashMap};
 use std::vec::Vec;
 use std::{i32, u32};
-use translation_utils::{f32_translation, f64_translation, num_return_values, type_to_type};
-use translation_utils::{FuncIndex, MemoryIndex, SignatureIndex, TableIndex};
 use wasmparser::{MemoryImmediate, Operator};
 
 // Clippy warns about "flags: _" but its important to document that the flags field is ignored
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index fc7db3af38..81212fa379 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -5,6 +5,12 @@
 //! [wasmtime-environ]: https://crates.io/crates/wasmtime-environ
 //! [Wasmtime]: https://github.com/CraneStation/wasmtime
 
+use crate::environ::{FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmResult};
+use crate::func_translator::FuncTranslator;
+use crate::translation_utils::{
+    DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table,
+    TableIndex,
+};
 use cast;
 use cranelift_codegen::cursor::FuncCursor;
 use cranelift_codegen::ir::immediates::{Offset32, Uimm64};
@@ -12,14 +18,8 @@ use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::isa::TargetFrontendConfig;
 use cranelift_entity::{EntityRef, PrimaryMap};
-use environ::{FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmResult};
-use func_translator::FuncTranslator;
 use std::string::String;
 use std::vec::Vec;
-use translation_utils::{
-    DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table,
-    TableIndex,
-};
 
 /// Compute a `ir::ExternalName` for a given wasm function index.
 fn get_func_name(func_index: FuncIndex) -> ir::ExternalName {
diff --git a/lib/wasm/src/environ/mod.rs b/lib/wasm/src/environ/mod.rs
index d44011fd8c..831fa0ef5e 100644
--- a/lib/wasm/src/environ/mod.rs
+++ b/lib/wasm/src/environ/mod.rs
@@ -3,7 +3,7 @@
 mod dummy;
 mod spec;
 
-pub use environ::dummy::DummyEnvironment;
-pub use environ::spec::{
+pub use crate::environ::dummy::DummyEnvironment;
+pub use crate::environ::spec::{
     FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmError, WasmResult,
 };
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 9f3c2ba165..cfd943ed78 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -6,15 +6,16 @@
 //!
 //! [Wasmtime]: https://github.com/CraneStation/wasmtime
 
+use crate::translation_utils::{
+    FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
+};
 use cranelift_codegen::cursor::FuncCursor;
 use cranelift_codegen::ir::immediates::Offset32;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::isa::TargetFrontendConfig;
+use failure_derive::Fail;
 use std::convert::From;
 use std::vec::Vec;
-use translation_utils::{
-    FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
-};
 use wasmparser::BinaryReaderError;
 
 /// The value of a WebAssembly global variable.
diff --git a/lib/wasm/src/func_translator.rs b/lib/wasm/src/func_translator.rs
index 4b9e5fe7ca..7e5255499d 100644
--- a/lib/wasm/src/func_translator.rs
+++ b/lib/wasm/src/func_translator.rs
@@ -4,13 +4,14 @@
 //! function to Cranelift IR guided by a `FuncEnvironment` which provides information about the
 //! WebAssembly module and the runtime environment.
 
-use code_translator::translate_operator;
+use crate::code_translator::translate_operator;
+use crate::environ::{FuncEnvironment, ReturnMode, WasmResult};
+use crate::state::TranslationState;
 use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir::{self, Ebb, InstBuilder};
 use cranelift_codegen::timing;
 use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
-use environ::{FuncEnvironment, ReturnMode, WasmResult};
-use state::TranslationState;
+use log::info;
 use wasmparser::{self, BinaryReader};
 
 /// WebAssembly to Cranelift IR function translator.
@@ -231,9 +232,10 @@ fn cur_srcloc(reader: &BinaryReader) -> ir::SourceLoc {
 #[cfg(test)]
 mod tests {
     use super::{FuncTranslator, ReturnMode};
+    use crate::environ::DummyEnvironment;
     use cranelift_codegen::ir::types::I32;
     use cranelift_codegen::{ir, isa, settings, Context};
-    use environ::DummyEnvironment;
+    use log::debug;
     use target_lexicon::PointerWidth;
 
     #[test]
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index acd63bedb4..2b75ce9e4c 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -33,22 +33,6 @@
 #![cfg_attr(not(feature = "std"), no_std)]
 #![cfg_attr(not(feature = "std"), feature(alloc))]
 
-extern crate cranelift_codegen;
-#[macro_use]
-extern crate cranelift_entity;
-extern crate cast;
-extern crate cranelift_frontend;
-#[cfg(test)]
-extern crate target_lexicon;
-extern crate wasmparser;
-
-extern crate failure;
-#[macro_use]
-extern crate failure_derive;
-
-#[macro_use]
-extern crate log;
-
 mod code_translator;
 mod environ;
 mod func_translator;
@@ -57,13 +41,13 @@ mod sections_translator;
 mod state;
 mod translation_utils;
 
-pub use environ::{
+pub use crate::environ::{
     DummyEnvironment, FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmError,
     WasmResult,
 };
-pub use func_translator::FuncTranslator;
-pub use module_translator::translate_module;
-pub use translation_utils::{
+pub use crate::func_translator::FuncTranslator;
+pub use crate::module_translator::translate_module;
+pub use crate::translation_utils::{
     DefinedFuncIndex, DefinedGlobalIndex, DefinedMemoryIndex, DefinedTableIndex, FuncIndex, Global,
     GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex, Table, TableElementType,
     TableIndex,
diff --git a/lib/wasm/src/module_translator.rs b/lib/wasm/src/module_translator.rs
index 7a7ce91b9b..1cf0f4cd2d 100644
--- a/lib/wasm/src/module_translator.rs
+++ b/lib/wasm/src/module_translator.rs
@@ -1,12 +1,12 @@
 //! Translation skeleton that traverses the whole WebAssembly module and call helper functions
 //! to deal with each part of it.
-use cranelift_codegen::timing;
-use environ::{ModuleEnvironment, WasmResult};
-use sections_translator::{
+use crate::environ::{ModuleEnvironment, WasmResult};
+use crate::sections_translator::{
     parse_code_section, parse_data_section, parse_element_section, parse_export_section,
     parse_function_section, parse_global_section, parse_import_section, parse_memory_section,
     parse_start_section, parse_table_section, parse_type_section,
 };
+use cranelift_codegen::timing;
 use wasmparser::{ModuleReader, SectionCode};
 
 /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cranelift IR
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 4ea84d51f2..c1bdda1460 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -7,15 +7,15 @@
 //! The special case of the initialize expressions for table elements offsets or global variables
 //! is handled, according to the semantics of WebAssembly, to only specific expressions that are
 //! interpreted on the fly.
-use cranelift_codegen::ir::{self, AbiParam, Signature};
-use cranelift_entity::EntityRef;
-use environ::{ModuleEnvironment, WasmResult};
-use std::str::from_utf8;
-use std::vec::Vec;
-use translation_utils::{
+use crate::environ::{ModuleEnvironment, WasmResult};
+use crate::translation_utils::{
     type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex,
     Table, TableElementType, TableIndex,
 };
+use cranelift_codegen::ir::{self, AbiParam, Signature};
+use cranelift_entity::EntityRef;
+use std::str::from_utf8;
+use std::vec::Vec;
 use wasmparser::{
     self, CodeSectionReader, Data, DataSectionReader, Element, ElementSectionReader, Export,
     ExportSectionReader, ExternalKind, FuncType, FunctionSectionReader, GlobalSectionReader,
diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs
index 02efc65ae3..80d1f5fe23 100644
--- a/lib/wasm/src/state.rs
+++ b/lib/wasm/src/state.rs
@@ -3,11 +3,11 @@
 //! The `TranslationState` struct defined in this module is used to keep track of the WebAssembly
 //! value and control stacks during the translation of a single function.
 
+use crate::environ::{FuncEnvironment, GlobalVariable};
+use crate::translation_utils::{FuncIndex, GlobalIndex, MemoryIndex, SignatureIndex, TableIndex};
 use cranelift_codegen::ir::{self, Ebb, Inst, Value};
-use environ::{FuncEnvironment, GlobalVariable};
 use std::collections::HashMap;
 use std::vec::Vec;
-use translation_utils::{FuncIndex, GlobalIndex, MemoryIndex, SignatureIndex, TableIndex};
 
 /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following
 /// fields:
diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index 9dc8a2ffd6..cdfe276d46 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -1,4 +1,5 @@
 //! Helper functions and structures for the translation.
+use cranelift_codegen::entity::entity_impl;
 use cranelift_codegen::ir;
 use std::u32;
 use wasmparser;
diff --git a/lib/wasm/tests/wasm_testsuite.rs b/lib/wasm/tests/wasm_testsuite.rs
index 18b8c08c11..30428954bd 100644
--- a/lib/wasm/tests/wasm_testsuite.rs
+++ b/lib/wasm/tests/wasm_testsuite.rs
@@ -1,9 +1,3 @@
-extern crate cranelift_codegen;
-extern crate cranelift_wasm;
-#[macro_use]
-extern crate target_lexicon;
-extern crate wabt;
-
 use cranelift_codegen::isa;
 use cranelift_codegen::print_errors::pretty_verifier_error;
 use cranelift_codegen::settings::{self, Flags};
@@ -15,6 +9,7 @@ use std::io;
 use std::io::prelude::*;
 use std::path::Path;
 use std::str::FromStr;
+use target_lexicon::triple;
 use wabt::wat2wasm;
 
 #[test]

From ab72eeadf969e73aac9983213248ab048d4d165a Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Tue, 1 Jan 2019 14:00:18 +0100
Subject: [PATCH 2276/3084] Bump faerie to 0.7.0

---
 lib/faerie/Cargo.toml     | 2 +-
 lib/faerie/src/backend.rs | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 0edc0c723c..4ef81063e9 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -12,7 +12,7 @@ edition = "2018"
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.26.0" }
 cranelift-module = { path = "../module", version = "0.26.0" }
-faerie = "0.6.0"
+faerie = "0.7.0"
 goblin = "0.0.19"
 failure = "0.1.2"
 target-lexicon = "0.2.0"
diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index 6941fb6312..db1b13d596 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -408,7 +408,7 @@ impl<'a> RelocSink for FaerieRelocSink<'a> {
                     to: &ref_name,
                     at: u64::from(offset),
                 },
-                faerie::RelocOverride {
+                faerie::Reloc::Raw {
                     reloc: raw_reloc,
                     addend: addend_i32,
                 },

From b096d0606b8b28d66ea0aea7b2474115def2977f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 2 Jan 2019 11:46:15 -0800
Subject: [PATCH 2277/3084] Move ifcmp, icmp_imm, and ffcmp into the CPU flags
 doc section.

---
 cranelift/docs/ir.rst | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index faf941863c..550d42a81c 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -814,8 +814,6 @@ Integer operations
 
 .. autoinst:: icmp
 .. autoinst:: icmp_imm
-.. autoinst:: ifcmp
-.. autoinst:: ifcmp_imm
 .. autoinst:: iadd
 .. autoinst:: iadd_imm
 .. autoinst:: iadd_cin
@@ -914,7 +912,6 @@ Floating point operations
 These operations generally follow IEEE 754-2008 semantics.
 
 .. autoinst:: fcmp
-.. autoinst:: ffcmp
 .. autoinst:: fadd
 .. autoinst:: fsub
 .. autoinst:: fmul
@@ -1061,6 +1058,9 @@ CPU flag operations
 These operations are for working with the "flags" registers of some CPU
 architectures.
 
+.. autoinst:: ifcmp
+.. autoinst:: ifcmp_imm
+.. autoinst:: ffcmp
 .. autoinst:: trueif
 .. autoinst:: trueff
 .. autoinst:: trapif

From 8a11bd6af79dc921fe55b353bc85e1f097cb99b4 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 2 Jan 2019 11:58:44 -0800
Subject: [PATCH 2278/3084] Remove `BoxedSlice`'s `next_key()` function, which
 is unneeded.

`BoxedSlice` does not support appending elements, so it isn't useful to
look up the next key.
---
 lib/entity/src/boxed_slice.rs | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/lib/entity/src/boxed_slice.rs b/lib/entity/src/boxed_slice.rs
index c97522467c..e619238387 100644
--- a/lib/entity/src/boxed_slice.rs
+++ b/lib/entity/src/boxed_slice.rs
@@ -83,11 +83,6 @@ where
         IterMut::new(self.elems.iter_mut())
     }
 
-    /// Get the key that will be assigned to the next pushed value.
-    pub fn next_key(&self) -> K {
-        K::new(self.elems.len())
-    }
-
     /// Returns the last element that was inserted in the map.
     pub fn last(&self) -> Option<&V> {
         self.elems.last()

From 29edc3fe5012fa1a6eb23deffd5ffb17ebebec7a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 2 Jan 2019 11:59:22 -0800
Subject: [PATCH 2279/3084] Make `ListPool`'s `alloc` fill new memory with
 reserved values.

This makes it harder to misuse, as 0 could be confused for a valid value.
---
 lib/entity/src/list.rs | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)

diff --git a/lib/entity/src/list.rs b/lib/entity/src/list.rs
index 37ac92ad0b..617ea2db32 100644
--- a/lib/entity/src/list.rs
+++ b/lib/entity/src/list.rs
@@ -1,4 +1,5 @@
 //! Small lists of entity references.
+use crate::packed_option::ReservedValue;
 use crate::EntityRef;
 use std::marker::PhantomData;
 use std::mem;
@@ -59,13 +60,13 @@ use std::vec::Vec;
 /// The index stored in an `EntityList` points to part 2, the list elements. The value 0 is
 /// reserved for the empty list which isn't allocated in the vector.
 #[derive(Clone, Debug)]
-pub struct EntityList {
+pub struct EntityList {
     index: u32,
     unused: PhantomData,
 }
 
 /// Create an empty list.
-impl Default for EntityList {
+impl Default for EntityList {
     fn default() -> Self {
         Self {
             index: 0,
@@ -76,7 +77,7 @@ impl Default for EntityList {
 
 /// A memory pool for storing lists of `T`.
 #[derive(Clone, Debug)]
-pub struct ListPool {
+pub struct ListPool {
     // The main array containing the lists.
     data: Vec,
 
@@ -105,7 +106,7 @@ fn is_sclass_min_length(len: usize) -> bool {
     len > 3 && len.is_power_of_two()
 }
 
-impl ListPool {
+impl ListPool {
     /// Create a new list pool.
     pub fn new() -> Self {
         Self {
@@ -140,7 +141,8 @@ impl ListPool {
     /// Allocate a storage block with a size given by `sclass`.
     ///
     /// Returns the first index of an available segment of `self.data` containing
-    /// `sclass_size(sclass)` elements.
+    /// `sclass_size(sclass)` elements. The allocated memory is filled with reserved
+    /// values.
     fn alloc(&mut self, sclass: SizeClass) -> usize {
         // First try the free list for this size class.
         match self.free.get(sclass as usize).cloned() {
@@ -155,9 +157,8 @@ impl ListPool {
             _ => {
                 // Nothing on the free list. Allocate more memory.
                 let offset = self.data.len();
-                // We don't want to mess around with uninitialized data.
-                // Just fill it up with nulls.
-                self.data.resize(offset + sclass_size(sclass), T::new(0));
+                self.data
+                    .resize(offset + sclass_size(sclass), T::reserved_value());
                 offset
             }
         }
@@ -219,7 +220,7 @@ impl ListPool {
     }
 }
 
-impl EntityList {
+impl EntityList {
     /// Create a new empty list.
     pub fn new() -> Self {
         Default::default()
@@ -351,7 +352,7 @@ impl EntityList {
         }
     }
 
-    /// Grow list by adding `count` uninitialized elements at the end.
+    /// Grow list by adding `count` reserved-value elements at the end.
     ///
     /// Returns a mutable slice representing the whole list.
     fn grow<'a>(&'a mut self, count: usize, pool: &'a mut ListPool) -> &'a mut [T] {
@@ -483,6 +484,7 @@ impl EntityList {
 mod tests {
     use super::*;
     use super::{sclass_for_length, sclass_size};
+    use crate::packed_option::ReservedValue;
     use crate::EntityRef;
 
     /// An opaque reference to an instruction in a function.

From cd4f96c066117d00c241135d9939fd527809ae8a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 2 Jan 2019 11:59:57 -0800
Subject: [PATCH 2280/3084] Use `Vec::with_capacity` when we know the eventual
 size of the `Vec`.

---
 lib/wasm/src/sections_translator.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index c1bdda1460..2ee60c19a9 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -255,7 +255,7 @@ pub fn parse_element_section<'data>(
             ref s => panic!("unsupported init expr in element section: {:?}", s),
         };
         let items_reader = items.get_items_reader()?;
-        let mut elems = Vec::new();
+        let mut elems = Vec::with_capacity(cast::usize(items_reader.get_count()));
         for item in items_reader {
             let x = item?;
             elems.push(FuncIndex::from_u32(x));

From 831d0334710b98abbc10d0c230059f98201d8164 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 2 Jan 2019 12:01:36 -0800
Subject: [PATCH 2281/3084] Add a `shrink_to_fit` function to `PrimaryMap`.

---
 lib/entity/src/primary.rs | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs
index 7a45fa9582..8f1daed7e3 100644
--- a/lib/entity/src/primary.rs
+++ b/lib/entity/src/primary.rs
@@ -135,6 +135,11 @@ where
         self.elems.reserve_exact(additional)
     }
 
+    /// Shrinks the capacity of the `PrimaryMap` as much as possible.
+    pub fn shrink_to_fit(&mut self) {
+        self.elems.shrink_to_fit()
+    }
+
     /// Consumes this `PrimaryMap` and produces a `BoxedSlice`.
     pub fn into_boxed_slice(self) -> BoxedSlice {
         unsafe { BoxedSlice::::from_raw(Box::<[V]>::into_raw(self.elems.into_boxed_slice())) }

From c8e47177fccb1af0c0f082a2836f53c86d3a652d Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 2 Jan 2019 12:05:32 -0800
Subject: [PATCH 2282/3084] Add an assert to
 `append_ebb_params_for_function_params`.

---
 lib/frontend/src/frontend.rs | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 2ed09fbd01..d769b678f8 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -405,6 +405,11 @@ impl<'a> FunctionBuilder<'a> {
     /// parameters. This can be used to set up the ebb parameters for the
     /// entry block.
     pub fn append_ebb_params_for_function_params(&mut self, ebb: Ebb) {
+        debug_assert!(
+            !self.func_ctx.ssa.has_any_predecessors(ebb),
+            "ebb parameters for function parameters should only be added to the entry block"
+        );
+
         // These parameters count as "user" parameters here because they aren't
         // inserted by the SSABuilder.
         let user_param_count = &mut self.func_ctx.ebbs[ebb].user_param_count;

From e9cb50313dc3a8cf5ad87a64116857f4d338255d Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Mon, 24 Dec 2018 15:31:31 +0100
Subject: [PATCH 2283/3084] Fix emit_small_mem{cpy,set} when size == 0 or
 access_size > 8

---
 lib/frontend/src/frontend.rs | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index d769b678f8..1d454661c9 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -598,6 +598,10 @@ impl<'a> FunctionBuilder<'a> {
         // Currently the result of guess work, not actual profiling.
         const THRESHOLD: u64 = 4;
 
+        if size == 0 {
+            return;
+        }
+
         let access_size = greatest_divisible_power_of_two(size);
         assert!(
             access_size.is_power_of_two(),
@@ -607,6 +611,13 @@ impl<'a> FunctionBuilder<'a> {
             access_size >= ::std::cmp::min(src_align, dest_align) as u64,
             "`size` is smaller than `dest` and `src`'s alignment value."
         );
+
+        let (access_size, int_type) = if access_size > 8 {
+            (access_size, Type::int((access_size * 8) as u16).unwrap())
+        } else {
+            (8, types::I64)
+        };
+
         let load_and_store_amount = size / access_size;
 
         if load_and_store_amount > THRESHOLD {
@@ -615,7 +626,6 @@ impl<'a> FunctionBuilder<'a> {
             return;
         }
 
-        let int_type = Type::int((access_size * 8) as u16).unwrap();
         let mut flags = MemFlags::new();
         flags.set_aligned();
 
@@ -669,6 +679,10 @@ impl<'a> FunctionBuilder<'a> {
         // Currently the result of guess work, not actual profiling.
         const THRESHOLD: u64 = 4;
 
+        if size == 0 {
+            return;
+        }
+
         let access_size = greatest_divisible_power_of_two(size);
         assert!(
             access_size.is_power_of_two(),
@@ -678,6 +692,13 @@ impl<'a> FunctionBuilder<'a> {
             access_size >= buffer_align as u64,
             "`size` is smaller than `dest` and `src`'s alignment value."
         );
+
+        let (access_size, int_type) = if access_size > 8 {
+            (access_size, Type::int((access_size * 8) as u16).unwrap())
+        } else {
+            (8, types::I64)
+        };
+
         let load_and_store_amount = size / access_size;
 
         if load_and_store_amount > THRESHOLD {
@@ -689,7 +710,6 @@ impl<'a> FunctionBuilder<'a> {
             flags.set_aligned();
 
             let ch = ch as u64;
-            let int_type = Type::int((access_size * 8) as u16).unwrap();
             let raw_value = if int_type == types::I64 {
                 (ch << 32) | (ch << 16) | (ch << 8) | ch
             } else if int_type == types::I32 {

From 7e42966c68dba1cfdfcef116b3613a6cfeb48318 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 15 Nov 2018 12:44:14 +0100
Subject: [PATCH 2284/3084] Use snake case for magic numbers computation;

---
 lib/codegen/src/divconst_magic_numbers.rs | 376 +++++++++++-----------
 lib/codegen/src/simple_preopt.rs          |  86 ++---
 2 files changed, 230 insertions(+), 232 deletions(-)

diff --git a/lib/codegen/src/divconst_magic_numbers.rs b/lib/codegen/src/divconst_magic_numbers.rs
index 32a342703c..89c686498e 100644
--- a/lib/codegen/src/divconst_magic_numbers.rs
+++ b/lib/codegen/src/divconst_magic_numbers.rs
@@ -8,39 +8,37 @@
 //! so as to emphasise the similarity of the U32 and U64 cases and the S32
 //! and S64 cases.
 
-#![allow(non_snake_case)]
-
 // Structures to hold the "magic numbers" computed.
 
 #[derive(PartialEq, Debug)]
 pub struct MU32 {
-    pub mulBy: u32,
-    pub doAdd: bool,
-    pub shiftBy: i32,
+    pub mul_by: u32,
+    pub do_add: bool,
+    pub shift_by: i32,
 }
 
 #[derive(PartialEq, Debug)]
 pub struct MU64 {
-    pub mulBy: u64,
-    pub doAdd: bool,
-    pub shiftBy: i32,
+    pub mul_by: u64,
+    pub do_add: bool,
+    pub shift_by: i32,
 }
 
 #[derive(PartialEq, Debug)]
 pub struct MS32 {
-    pub mulBy: i32,
-    pub shiftBy: i32,
+    pub mul_by: i32,
+    pub shift_by: i32,
 }
 
 #[derive(PartialEq, Debug)]
 pub struct MS64 {
-    pub mulBy: i64,
-    pub shiftBy: i32,
+    pub mul_by: i64,
+    pub shift_by: i32,
 }
 
 // The actual "magic number" generators follow.
 
-pub fn magicU32(d: u32) -> MU32 {
+pub fn magic_u32(d: u32) -> MU32 {
     debug_assert_ne!(d, 0);
     debug_assert_ne!(d, 1); // d==1 generates out of range shifts.
 
@@ -80,13 +78,13 @@ pub fn magicU32(d: u32) -> MU32 {
     }
 
     MU32 {
-        mulBy: q2 + 1,
-        doAdd: do_add,
-        shiftBy: p - 32,
+        mul_by: q2 + 1,
+        do_add: do_add,
+        shift_by: p - 32,
     }
 }
 
-pub fn magicU64(d: u64) -> MU64 {
+pub fn magic_u64(d: u64) -> MU64 {
     debug_assert_ne!(d, 0);
     debug_assert_ne!(d, 1); // d==1 generates out of range shifts.
 
@@ -126,13 +124,13 @@ pub fn magicU64(d: u64) -> MU64 {
     }
 
     MU64 {
-        mulBy: q2 + 1,
-        doAdd: do_add,
-        shiftBy: p - 64,
+        mul_by: q2 + 1,
+        do_add: do_add,
+        shift_by: p - 64,
     }
 }
 
-pub fn magicS32(d: i32) -> MS32 {
+pub fn magic_s32(d: i32) -> MS32 {
     debug_assert_ne!(d, -1);
     debug_assert_ne!(d, 0);
     debug_assert_ne!(d, 1);
@@ -166,16 +164,16 @@ pub fn magicS32(d: i32) -> MS32 {
     }
 
     MS32 {
-        mulBy: (if d < 0 {
+        mul_by: (if d < 0 {
             u32::wrapping_neg(q2 + 1)
         } else {
             q2 + 1
         }) as i32,
-        shiftBy: p - 32,
+        shift_by: p - 32,
     }
 }
 
-pub fn magicS64(d: i64) -> MS64 {
+pub fn magic_s64(d: i64) -> MS64 {
     debug_assert_ne!(d, -1);
     debug_assert_ne!(d, 0);
     debug_assert_ne!(d, 1);
@@ -209,266 +207,266 @@ pub fn magicS64(d: i64) -> MS64 {
     }
 
     MS64 {
-        mulBy: (if d < 0 {
+        mul_by: (if d < 0 {
             u64::wrapping_neg(q2 + 1)
         } else {
             q2 + 1
         }) as i64,
-        shiftBy: p - 64,
+        shift_by: p - 64,
     }
 }
 
 #[cfg(test)]
 mod tests {
-    use super::{magicS32, magicS64, magicU32, magicU64};
+    use super::{magic_s32, magic_s64, magic_u32, magic_u64};
     use super::{MS32, MS64, MU32, MU64};
 
-    fn mkMU32(mulBy: u32, doAdd: bool, shiftBy: i32) -> MU32 {
+    fn mkMU32(mul_by: u32, do_add: bool, shift_by: i32) -> MU32 {
         MU32 {
-            mulBy,
-            doAdd,
-            shiftBy,
+            mul_by,
+            do_add,
+            shift_by,
         }
     }
 
-    fn mkMU64(mulBy: u64, doAdd: bool, shiftBy: i32) -> MU64 {
+    fn mkMU64(mul_by: u64, do_add: bool, shift_by: i32) -> MU64 {
         MU64 {
-            mulBy,
-            doAdd,
-            shiftBy,
+            mul_by,
+            do_add,
+            shift_by,
         }
     }
 
-    fn mkMS32(mulBy: i32, shiftBy: i32) -> MS32 {
-        MS32 { mulBy, shiftBy }
+    fn mkMS32(mul_by: i32, shift_by: i32) -> MS32 {
+        MS32 { mul_by, shift_by }
     }
 
-    fn mkMS64(mulBy: i64, shiftBy: i32) -> MS64 {
-        MS64 { mulBy, shiftBy }
+    fn mkMS64(mul_by: i64, shift_by: i32) -> MS64 {
+        MS64 { mul_by, shift_by }
     }
 
     #[test]
     fn test_magicU32() {
-        assert_eq!(magicU32(2u32), mkMU32(0x80000000u32, false, 0));
-        assert_eq!(magicU32(3u32), mkMU32(0xaaaaaaabu32, false, 1));
-        assert_eq!(magicU32(4u32), mkMU32(0x40000000u32, false, 0));
-        assert_eq!(magicU32(5u32), mkMU32(0xcccccccdu32, false, 2));
-        assert_eq!(magicU32(6u32), mkMU32(0xaaaaaaabu32, false, 2));
-        assert_eq!(magicU32(7u32), mkMU32(0x24924925u32, true, 3));
-        assert_eq!(magicU32(9u32), mkMU32(0x38e38e39u32, false, 1));
-        assert_eq!(magicU32(10u32), mkMU32(0xcccccccdu32, false, 3));
-        assert_eq!(magicU32(11u32), mkMU32(0xba2e8ba3u32, false, 3));
-        assert_eq!(magicU32(12u32), mkMU32(0xaaaaaaabu32, false, 3));
-        assert_eq!(magicU32(25u32), mkMU32(0x51eb851fu32, false, 3));
-        assert_eq!(magicU32(125u32), mkMU32(0x10624dd3u32, false, 3));
-        assert_eq!(magicU32(625u32), mkMU32(0xd1b71759u32, false, 9));
-        assert_eq!(magicU32(1337u32), mkMU32(0x88233b2bu32, true, 11));
-        assert_eq!(magicU32(65535u32), mkMU32(0x80008001u32, false, 15));
-        assert_eq!(magicU32(65536u32), mkMU32(0x00010000u32, false, 0));
-        assert_eq!(magicU32(65537u32), mkMU32(0xffff0001u32, false, 16));
-        assert_eq!(magicU32(31415927u32), mkMU32(0x445b4553u32, false, 23));
-        assert_eq!(magicU32(0xdeadbeefu32), mkMU32(0x93275ab3u32, false, 31));
-        assert_eq!(magicU32(0xfffffffdu32), mkMU32(0x40000001u32, false, 30));
-        assert_eq!(magicU32(0xfffffffeu32), mkMU32(0x00000003u32, true, 32));
-        assert_eq!(magicU32(0xffffffffu32), mkMU32(0x80000001u32, false, 31));
+        assert_eq!(magic_u32(2u32), mkMU32(0x80000000u32, false, 0));
+        assert_eq!(magic_u32(3u32), mkMU32(0xaaaaaaabu32, false, 1));
+        assert_eq!(magic_u32(4u32), mkMU32(0x40000000u32, false, 0));
+        assert_eq!(magic_u32(5u32), mkMU32(0xcccccccdu32, false, 2));
+        assert_eq!(magic_u32(6u32), mkMU32(0xaaaaaaabu32, false, 2));
+        assert_eq!(magic_u32(7u32), mkMU32(0x24924925u32, true, 3));
+        assert_eq!(magic_u32(9u32), mkMU32(0x38e38e39u32, false, 1));
+        assert_eq!(magic_u32(10u32), mkMU32(0xcccccccdu32, false, 3));
+        assert_eq!(magic_u32(11u32), mkMU32(0xba2e8ba3u32, false, 3));
+        assert_eq!(magic_u32(12u32), mkMU32(0xaaaaaaabu32, false, 3));
+        assert_eq!(magic_u32(25u32), mkMU32(0x51eb851fu32, false, 3));
+        assert_eq!(magic_u32(125u32), mkMU32(0x10624dd3u32, false, 3));
+        assert_eq!(magic_u32(625u32), mkMU32(0xd1b71759u32, false, 9));
+        assert_eq!(magic_u32(1337u32), mkMU32(0x88233b2bu32, true, 11));
+        assert_eq!(magic_u32(65535u32), mkMU32(0x80008001u32, false, 15));
+        assert_eq!(magic_u32(65536u32), mkMU32(0x00010000u32, false, 0));
+        assert_eq!(magic_u32(65537u32), mkMU32(0xffff0001u32, false, 16));
+        assert_eq!(magic_u32(31415927u32), mkMU32(0x445b4553u32, false, 23));
+        assert_eq!(magic_u32(0xdeadbeefu32), mkMU32(0x93275ab3u32, false, 31));
+        assert_eq!(magic_u32(0xfffffffdu32), mkMU32(0x40000001u32, false, 30));
+        assert_eq!(magic_u32(0xfffffffeu32), mkMU32(0x00000003u32, true, 32));
+        assert_eq!(magic_u32(0xffffffffu32), mkMU32(0x80000001u32, false, 31));
     }
     #[test]
     fn test_magicU64() {
-        assert_eq!(magicU64(2u64), mkMU64(0x8000000000000000u64, false, 0));
-        assert_eq!(magicU64(3u64), mkMU64(0xaaaaaaaaaaaaaaabu64, false, 1));
-        assert_eq!(magicU64(4u64), mkMU64(0x4000000000000000u64, false, 0));
-        assert_eq!(magicU64(5u64), mkMU64(0xcccccccccccccccdu64, false, 2));
-        assert_eq!(magicU64(6u64), mkMU64(0xaaaaaaaaaaaaaaabu64, false, 2));
-        assert_eq!(magicU64(7u64), mkMU64(0x2492492492492493u64, true, 3));
-        assert_eq!(magicU64(9u64), mkMU64(0xe38e38e38e38e38fu64, false, 3));
-        assert_eq!(magicU64(10u64), mkMU64(0xcccccccccccccccdu64, false, 3));
-        assert_eq!(magicU64(11u64), mkMU64(0x2e8ba2e8ba2e8ba3u64, false, 1));
-        assert_eq!(magicU64(12u64), mkMU64(0xaaaaaaaaaaaaaaabu64, false, 3));
-        assert_eq!(magicU64(25u64), mkMU64(0x47ae147ae147ae15u64, true, 5));
-        assert_eq!(magicU64(125u64), mkMU64(0x0624dd2f1a9fbe77u64, true, 7));
-        assert_eq!(magicU64(625u64), mkMU64(0x346dc5d63886594bu64, false, 7));
-        assert_eq!(magicU64(1337u64), mkMU64(0xc4119d952866a139u64, false, 10));
+        assert_eq!(magic_u64(2u64), mkMU64(0x8000000000000000u64, false, 0));
+        assert_eq!(magic_u64(3u64), mkMU64(0xaaaaaaaaaaaaaaabu64, false, 1));
+        assert_eq!(magic_u64(4u64), mkMU64(0x4000000000000000u64, false, 0));
+        assert_eq!(magic_u64(5u64), mkMU64(0xcccccccccccccccdu64, false, 2));
+        assert_eq!(magic_u64(6u64), mkMU64(0xaaaaaaaaaaaaaaabu64, false, 2));
+        assert_eq!(magic_u64(7u64), mkMU64(0x2492492492492493u64, true, 3));
+        assert_eq!(magic_u64(9u64), mkMU64(0xe38e38e38e38e38fu64, false, 3));
+        assert_eq!(magic_u64(10u64), mkMU64(0xcccccccccccccccdu64, false, 3));
+        assert_eq!(magic_u64(11u64), mkMU64(0x2e8ba2e8ba2e8ba3u64, false, 1));
+        assert_eq!(magic_u64(12u64), mkMU64(0xaaaaaaaaaaaaaaabu64, false, 3));
+        assert_eq!(magic_u64(25u64), mkMU64(0x47ae147ae147ae15u64, true, 5));
+        assert_eq!(magic_u64(125u64), mkMU64(0x0624dd2f1a9fbe77u64, true, 7));
+        assert_eq!(magic_u64(625u64), mkMU64(0x346dc5d63886594bu64, false, 7));
+        assert_eq!(magic_u64(1337u64), mkMU64(0xc4119d952866a139u64, false, 10));
         assert_eq!(
-            magicU64(31415927u64),
+            magic_u64(31415927u64),
             mkMU64(0x116d154b9c3d2f85u64, true, 25)
         );
         assert_eq!(
-            magicU64(0x00000000deadbeefu64),
+            magic_u64(0x00000000deadbeefu64),
             mkMU64(0x93275ab2dfc9094bu64, false, 31)
         );
         assert_eq!(
-            magicU64(0x00000000fffffffdu64),
+            magic_u64(0x00000000fffffffdu64),
             mkMU64(0x8000000180000005u64, false, 31)
         );
         assert_eq!(
-            magicU64(0x00000000fffffffeu64),
+            magic_u64(0x00000000fffffffeu64),
             mkMU64(0x0000000200000005u64, true, 32)
         );
         assert_eq!(
-            magicU64(0x00000000ffffffffu64),
+            magic_u64(0x00000000ffffffffu64),
             mkMU64(0x8000000080000001u64, false, 31)
         );
         assert_eq!(
-            magicU64(0x0000000100000000u64),
+            magic_u64(0x0000000100000000u64),
             mkMU64(0x0000000100000000u64, false, 0)
         );
         assert_eq!(
-            magicU64(0x0000000100000001u64),
+            magic_u64(0x0000000100000001u64),
             mkMU64(0xffffffff00000001u64, false, 32)
         );
         assert_eq!(
-            magicU64(0x0ddc0ffeebadf00du64),
+            magic_u64(0x0ddc0ffeebadf00du64),
             mkMU64(0x2788e9d394b77da1u64, true, 60)
         );
         assert_eq!(
-            magicU64(0xfffffffffffffffdu64),
+            magic_u64(0xfffffffffffffffdu64),
             mkMU64(0x4000000000000001u64, false, 62)
         );
         assert_eq!(
-            magicU64(0xfffffffffffffffeu64),
+            magic_u64(0xfffffffffffffffeu64),
             mkMU64(0x0000000000000003u64, true, 64)
         );
         assert_eq!(
-            magicU64(0xffffffffffffffffu64),
+            magic_u64(0xffffffffffffffffu64),
             mkMU64(0x8000000000000001u64, false, 63)
         );
     }
     #[test]
     fn test_magicS32() {
-        assert_eq!(magicS32(-0x80000000i32), mkMS32(0x7fffffffu32 as i32, 30));
-        assert_eq!(magicS32(-0x7FFFFFFFi32), mkMS32(0xbfffffffu32 as i32, 29));
-        assert_eq!(magicS32(-0x7FFFFFFEi32), mkMS32(0x7ffffffdu32 as i32, 30));
-        assert_eq!(magicS32(-31415927i32), mkMS32(0xbba4baadu32 as i32, 23));
-        assert_eq!(magicS32(-1337i32), mkMS32(0x9df73135u32 as i32, 9));
-        assert_eq!(magicS32(-256i32), mkMS32(0x7fffffffu32 as i32, 7));
-        assert_eq!(magicS32(-5i32), mkMS32(0x99999999u32 as i32, 1));
-        assert_eq!(magicS32(-3i32), mkMS32(0x55555555u32 as i32, 1));
-        assert_eq!(magicS32(-2i32), mkMS32(0x7fffffffu32 as i32, 0));
-        assert_eq!(magicS32(2i32), mkMS32(0x80000001u32 as i32, 0));
-        assert_eq!(magicS32(3i32), mkMS32(0x55555556u32 as i32, 0));
-        assert_eq!(magicS32(4i32), mkMS32(0x80000001u32 as i32, 1));
-        assert_eq!(magicS32(5i32), mkMS32(0x66666667u32 as i32, 1));
-        assert_eq!(magicS32(6i32), mkMS32(0x2aaaaaabu32 as i32, 0));
-        assert_eq!(magicS32(7i32), mkMS32(0x92492493u32 as i32, 2));
-        assert_eq!(magicS32(9i32), mkMS32(0x38e38e39u32 as i32, 1));
-        assert_eq!(magicS32(10i32), mkMS32(0x66666667u32 as i32, 2));
-        assert_eq!(magicS32(11i32), mkMS32(0x2e8ba2e9u32 as i32, 1));
-        assert_eq!(magicS32(12i32), mkMS32(0x2aaaaaabu32 as i32, 1));
-        assert_eq!(magicS32(25i32), mkMS32(0x51eb851fu32 as i32, 3));
-        assert_eq!(magicS32(125i32), mkMS32(0x10624dd3u32 as i32, 3));
-        assert_eq!(magicS32(625i32), mkMS32(0x68db8badu32 as i32, 8));
-        assert_eq!(magicS32(1337i32), mkMS32(0x6208cecbu32 as i32, 9));
-        assert_eq!(magicS32(31415927i32), mkMS32(0x445b4553u32 as i32, 23));
-        assert_eq!(magicS32(0x7ffffffei32), mkMS32(0x80000003u32 as i32, 30));
-        assert_eq!(magicS32(0x7fffffffi32), mkMS32(0x40000001u32 as i32, 29));
+        assert_eq!(magic_s32(-0x80000000i32), mkMS32(0x7fffffffu32 as i32, 30));
+        assert_eq!(magic_s32(-0x7FFFFFFFi32), mkMS32(0xbfffffffu32 as i32, 29));
+        assert_eq!(magic_s32(-0x7FFFFFFEi32), mkMS32(0x7ffffffdu32 as i32, 30));
+        assert_eq!(magic_s32(-31415927i32), mkMS32(0xbba4baadu32 as i32, 23));
+        assert_eq!(magic_s32(-1337i32), mkMS32(0x9df73135u32 as i32, 9));
+        assert_eq!(magic_s32(-256i32), mkMS32(0x7fffffffu32 as i32, 7));
+        assert_eq!(magic_s32(-5i32), mkMS32(0x99999999u32 as i32, 1));
+        assert_eq!(magic_s32(-3i32), mkMS32(0x55555555u32 as i32, 1));
+        assert_eq!(magic_s32(-2i32), mkMS32(0x7fffffffu32 as i32, 0));
+        assert_eq!(magic_s32(2i32), mkMS32(0x80000001u32 as i32, 0));
+        assert_eq!(magic_s32(3i32), mkMS32(0x55555556u32 as i32, 0));
+        assert_eq!(magic_s32(4i32), mkMS32(0x80000001u32 as i32, 1));
+        assert_eq!(magic_s32(5i32), mkMS32(0x66666667u32 as i32, 1));
+        assert_eq!(magic_s32(6i32), mkMS32(0x2aaaaaabu32 as i32, 0));
+        assert_eq!(magic_s32(7i32), mkMS32(0x92492493u32 as i32, 2));
+        assert_eq!(magic_s32(9i32), mkMS32(0x38e38e39u32 as i32, 1));
+        assert_eq!(magic_s32(10i32), mkMS32(0x66666667u32 as i32, 2));
+        assert_eq!(magic_s32(11i32), mkMS32(0x2e8ba2e9u32 as i32, 1));
+        assert_eq!(magic_s32(12i32), mkMS32(0x2aaaaaabu32 as i32, 1));
+        assert_eq!(magic_s32(25i32), mkMS32(0x51eb851fu32 as i32, 3));
+        assert_eq!(magic_s32(125i32), mkMS32(0x10624dd3u32 as i32, 3));
+        assert_eq!(magic_s32(625i32), mkMS32(0x68db8badu32 as i32, 8));
+        assert_eq!(magic_s32(1337i32), mkMS32(0x6208cecbu32 as i32, 9));
+        assert_eq!(magic_s32(31415927i32), mkMS32(0x445b4553u32 as i32, 23));
+        assert_eq!(magic_s32(0x7ffffffei32), mkMS32(0x80000003u32 as i32, 30));
+        assert_eq!(magic_s32(0x7fffffffi32), mkMS32(0x40000001u32 as i32, 29));
     }
     #[test]
     fn test_magicS64() {
         assert_eq!(
-            magicS64(-0x8000000000000000i64),
+            magic_s64(-0x8000000000000000i64),
             mkMS64(0x7fffffffffffffffu64 as i64, 62)
         );
         assert_eq!(
-            magicS64(-0x7FFFFFFFFFFFFFFFi64),
+            magic_s64(-0x7FFFFFFFFFFFFFFFi64),
             mkMS64(0xbfffffffffffffffu64 as i64, 61)
         );
         assert_eq!(
-            magicS64(-0x7FFFFFFFFFFFFFFEi64),
+            magic_s64(-0x7FFFFFFFFFFFFFFEi64),
             mkMS64(0x7ffffffffffffffdu64 as i64, 62)
         );
         assert_eq!(
-            magicS64(-0x0ddC0ffeeBadF00di64),
+            magic_s64(-0x0ddC0ffeeBadF00di64),
             mkMS64(0x6c3b8b1635a4412fu64 as i64, 59)
         );
         assert_eq!(
-            magicS64(-0x100000001i64),
+            magic_s64(-0x100000001i64),
             mkMS64(0x800000007fffffffu64 as i64, 31)
         );
         assert_eq!(
-            magicS64(-0x100000000i64),
+            magic_s64(-0x100000000i64),
             mkMS64(0x7fffffffffffffffu64 as i64, 31)
         );
         assert_eq!(
-            magicS64(-0xFFFFFFFFi64),
+            magic_s64(-0xFFFFFFFFi64),
             mkMS64(0x7fffffff7fffffffu64 as i64, 31)
         );
         assert_eq!(
-            magicS64(-0xFFFFFFFEi64),
+            magic_s64(-0xFFFFFFFEi64),
             mkMS64(0x7ffffffefffffffdu64 as i64, 31)
         );
         assert_eq!(
-            magicS64(-0xFFFFFFFDi64),
+            magic_s64(-0xFFFFFFFDi64),
             mkMS64(0x7ffffffe7ffffffbu64 as i64, 31)
         );
         assert_eq!(
-            magicS64(-0xDeadBeefi64),
+            magic_s64(-0xDeadBeefi64),
             mkMS64(0x6cd8a54d2036f6b5u64 as i64, 31)
         );
         assert_eq!(
-            magicS64(-31415927i64),
+            magic_s64(-31415927i64),
             mkMS64(0x7749755a31e1683du64 as i64, 24)
         );
-        assert_eq!(magicS64(-1337i64), mkMS64(0x9df731356bccaf63u64 as i64, 9));
-        assert_eq!(magicS64(-256i64), mkMS64(0x7fffffffffffffffu64 as i64, 7));
-        assert_eq!(magicS64(-5i64), mkMS64(0x9999999999999999u64 as i64, 1));
-        assert_eq!(magicS64(-3i64), mkMS64(0x5555555555555555u64 as i64, 1));
-        assert_eq!(magicS64(-2i64), mkMS64(0x7fffffffffffffffu64 as i64, 0));
-        assert_eq!(magicS64(2i64), mkMS64(0x8000000000000001u64 as i64, 0));
-        assert_eq!(magicS64(3i64), mkMS64(0x5555555555555556u64 as i64, 0));
-        assert_eq!(magicS64(4i64), mkMS64(0x8000000000000001u64 as i64, 1));
-        assert_eq!(magicS64(5i64), mkMS64(0x6666666666666667u64 as i64, 1));
-        assert_eq!(magicS64(6i64), mkMS64(0x2aaaaaaaaaaaaaabu64 as i64, 0));
-        assert_eq!(magicS64(7i64), mkMS64(0x4924924924924925u64 as i64, 1));
-        assert_eq!(magicS64(9i64), mkMS64(0x1c71c71c71c71c72u64 as i64, 0));
-        assert_eq!(magicS64(10i64), mkMS64(0x6666666666666667u64 as i64, 2));
-        assert_eq!(magicS64(11i64), mkMS64(0x2e8ba2e8ba2e8ba3u64 as i64, 1));
-        assert_eq!(magicS64(12i64), mkMS64(0x2aaaaaaaaaaaaaabu64 as i64, 1));
-        assert_eq!(magicS64(25i64), mkMS64(0xa3d70a3d70a3d70bu64 as i64, 4));
-        assert_eq!(magicS64(125i64), mkMS64(0x20c49ba5e353f7cfu64 as i64, 4));
-        assert_eq!(magicS64(625i64), mkMS64(0x346dc5d63886594bu64 as i64, 7));
-        assert_eq!(magicS64(1337i64), mkMS64(0x6208ceca9433509du64 as i64, 9));
+        assert_eq!(magic_s64(-1337i64), mkMS64(0x9df731356bccaf63u64 as i64, 9));
+        assert_eq!(magic_s64(-256i64), mkMS64(0x7fffffffffffffffu64 as i64, 7));
+        assert_eq!(magic_s64(-5i64), mkMS64(0x9999999999999999u64 as i64, 1));
+        assert_eq!(magic_s64(-3i64), mkMS64(0x5555555555555555u64 as i64, 1));
+        assert_eq!(magic_s64(-2i64), mkMS64(0x7fffffffffffffffu64 as i64, 0));
+        assert_eq!(magic_s64(2i64), mkMS64(0x8000000000000001u64 as i64, 0));
+        assert_eq!(magic_s64(3i64), mkMS64(0x5555555555555556u64 as i64, 0));
+        assert_eq!(magic_s64(4i64), mkMS64(0x8000000000000001u64 as i64, 1));
+        assert_eq!(magic_s64(5i64), mkMS64(0x6666666666666667u64 as i64, 1));
+        assert_eq!(magic_s64(6i64), mkMS64(0x2aaaaaaaaaaaaaabu64 as i64, 0));
+        assert_eq!(magic_s64(7i64), mkMS64(0x4924924924924925u64 as i64, 1));
+        assert_eq!(magic_s64(9i64), mkMS64(0x1c71c71c71c71c72u64 as i64, 0));
+        assert_eq!(magic_s64(10i64), mkMS64(0x6666666666666667u64 as i64, 2));
+        assert_eq!(magic_s64(11i64), mkMS64(0x2e8ba2e8ba2e8ba3u64 as i64, 1));
+        assert_eq!(magic_s64(12i64), mkMS64(0x2aaaaaaaaaaaaaabu64 as i64, 1));
+        assert_eq!(magic_s64(25i64), mkMS64(0xa3d70a3d70a3d70bu64 as i64, 4));
+        assert_eq!(magic_s64(125i64), mkMS64(0x20c49ba5e353f7cfu64 as i64, 4));
+        assert_eq!(magic_s64(625i64), mkMS64(0x346dc5d63886594bu64 as i64, 7));
+        assert_eq!(magic_s64(1337i64), mkMS64(0x6208ceca9433509du64 as i64, 9));
         assert_eq!(
-            magicS64(31415927i64),
+            magic_s64(31415927i64),
             mkMS64(0x88b68aa5ce1e97c3u64 as i64, 24)
         );
         assert_eq!(
-            magicS64(0x00000000deadbeefi64),
+            magic_s64(0x00000000deadbeefi64),
             mkMS64(0x93275ab2dfc9094bu64 as i64, 31)
         );
         assert_eq!(
-            magicS64(0x00000000fffffffdi64),
+            magic_s64(0x00000000fffffffdi64),
             mkMS64(0x8000000180000005u64 as i64, 31)
         );
         assert_eq!(
-            magicS64(0x00000000fffffffei64),
+            magic_s64(0x00000000fffffffei64),
             mkMS64(0x8000000100000003u64 as i64, 31)
         );
         assert_eq!(
-            magicS64(0x00000000ffffffffi64),
+            magic_s64(0x00000000ffffffffi64),
             mkMS64(0x8000000080000001u64 as i64, 31)
         );
         assert_eq!(
-            magicS64(0x0000000100000000i64),
+            magic_s64(0x0000000100000000i64),
             mkMS64(0x8000000000000001u64 as i64, 31)
         );
         assert_eq!(
-            magicS64(0x0000000100000001i64),
+            magic_s64(0x0000000100000001i64),
             mkMS64(0x7fffffff80000001u64 as i64, 31)
         );
         assert_eq!(
-            magicS64(0x0ddc0ffeebadf00di64),
+            magic_s64(0x0ddc0ffeebadf00di64),
             mkMS64(0x93c474e9ca5bbed1u64 as i64, 59)
         );
         assert_eq!(
-            magicS64(0x7ffffffffffffffdi64),
+            magic_s64(0x7ffffffffffffffdi64),
             mkMS64(0x2000000000000001u64 as i64, 60)
         );
         assert_eq!(
-            magicS64(0x7ffffffffffffffei64),
+            magic_s64(0x7ffffffffffffffei64),
             mkMS64(0x8000000000000003u64 as i64, 62)
         );
         assert_eq!(
-            magicS64(0x7fffffffffffffffi64),
+            magic_s64(0x7fffffffffffffffi64),
             mkMS64(0x4000000000000001u64 as i64, 61)
         );
     }
@@ -478,67 +476,67 @@ mod tests {
         // don't panic with integer wraparounds, especially at boundary
         // cases for their arguments. The actual results are thrown away.
         let mut total: u64 = 0;
-        // Testing UP magicU32
+        // Testing UP magic_u32
         for x in 2..(200 * 1000u32) {
-            let m = magicU32(x);
-            total = total ^ (m.mulBy as u64);
-            total = total + (m.shiftBy as u64);
-            total = total - (if m.doAdd { 123 } else { 456 });
+            let m = magic_u32(x);
+            total = total ^ (m.mul_by as u64);
+            total = total + (m.shift_by as u64);
+            total = total - (if m.do_add { 123 } else { 456 });
         }
         assert_eq!(total, 1747815691);
-        // Testing DOWN magicU32
+        // Testing DOWN magic_u32
         for x in 0..(200 * 1000u32) {
-            let m = magicU32(0xFFFF_FFFFu32 - x);
-            total = total ^ (m.mulBy as u64);
-            total = total + (m.shiftBy as u64);
-            total = total - (if m.doAdd { 123 } else { 456 });
+            let m = magic_u32(0xFFFF_FFFFu32 - x);
+            total = total ^ (m.mul_by as u64);
+            total = total + (m.shift_by as u64);
+            total = total - (if m.do_add { 123 } else { 456 });
         }
         assert_eq!(total, 2210292772);
 
-        // Testing UP magicU64
+        // Testing UP magic_u64
         for x in 2..(200 * 1000u64) {
-            let m = magicU64(x);
-            total = total ^ m.mulBy;
-            total = total + (m.shiftBy as u64);
-            total = total - (if m.doAdd { 123 } else { 456 });
+            let m = magic_u64(x);
+            total = total ^ m.mul_by;
+            total = total + (m.shift_by as u64);
+            total = total - (if m.do_add { 123 } else { 456 });
         }
         assert_eq!(total, 7430004084791260605);
-        // Testing DOWN magicU64
+        // Testing DOWN magic_u64
         for x in 0..(200 * 1000u64) {
-            let m = magicU64(0xFFFF_FFFF_FFFF_FFFFu64 - x);
-            total = total ^ m.mulBy;
-            total = total + (m.shiftBy as u64);
-            total = total - (if m.doAdd { 123 } else { 456 });
+            let m = magic_u64(0xFFFF_FFFF_FFFF_FFFFu64 - x);
+            total = total ^ m.mul_by;
+            total = total + (m.shift_by as u64);
+            total = total - (if m.do_add { 123 } else { 456 });
         }
         assert_eq!(total, 7547519887519825919);
 
-        // Testing UP magicS32
+        // Testing UP magic_s32
         for x in 0..(200 * 1000i32) {
-            let m = magicS32(-0x8000_0000i32 + x);
-            total = total ^ (m.mulBy as u64);
-            total = total + (m.shiftBy as u64);
+            let m = magic_s32(-0x8000_0000i32 + x);
+            total = total ^ (m.mul_by as u64);
+            total = total + (m.shift_by as u64);
         }
         assert_eq!(total, 10899224186731671235);
-        // Testing DOWN magicS32
+        // Testing DOWN magic_s32
         for x in 0..(200 * 1000i32) {
-            let m = magicS32(0x7FFF_FFFFi32 - x);
-            total = total ^ (m.mulBy as u64);
-            total = total + (m.shiftBy as u64);
+            let m = magic_s32(0x7FFF_FFFFi32 - x);
+            total = total ^ (m.mul_by as u64);
+            total = total + (m.shift_by as u64);
         }
         assert_eq!(total, 7547519887517897369);
 
-        // Testing UP magicS64
+        // Testing UP magic_s64
         for x in 0..(200 * 1000i64) {
-            let m = magicS64(-0x8000_0000_0000_0000i64 + x);
-            total = total ^ (m.mulBy as u64);
-            total = total + (m.shiftBy as u64);
+            let m = magic_s64(-0x8000_0000_0000_0000i64 + x);
+            total = total ^ (m.mul_by as u64);
+            total = total + (m.shift_by as u64);
         }
         assert_eq!(total, 8029756891368555163);
-        // Testing DOWN magicS64
+        // Testing DOWN magic_s64
         for x in 0..(200 * 1000i64) {
-            let m = magicS64(0x7FFF_FFFF_FFFF_FFFFi64 - x);
-            total = total ^ (m.mulBy as u64);
-            total = total + (m.shiftBy as u64);
+            let m = magic_s64(0x7FFF_FFFF_FFFF_FFFFi64 - x);
+            total = total ^ (m.mul_by as u64);
+            total = total + (m.shift_by as u64);
         }
         // Force `total` -- and hence, the entire computation -- to
         // be used, so that rustc can't optimise it out.
diff --git a/lib/codegen/src/simple_preopt.rs b/lib/codegen/src/simple_preopt.rs
index 793def27ef..b71d4a31c4 100644
--- a/lib/codegen/src/simple_preopt.rs
+++ b/lib/codegen/src/simple_preopt.rs
@@ -3,7 +3,7 @@
 #![allow(non_snake_case)]
 
 use crate::cursor::{Cursor, FuncCursor};
-use crate::divconst_magic_numbers::{magicS32, magicS64, magicU32, magicU64};
+use crate::divconst_magic_numbers::{magic_s32, magic_s64, magic_u32, magic_u64};
 use crate::divconst_magic_numbers::{MS32, MS64, MU32, MU64};
 use crate::ir::dfg::ValueDef;
 use crate::ir::instructions::Opcode;
@@ -183,27 +183,27 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
         DivRemByConstInfo::DivU32(n1, d) | DivRemByConstInfo::RemU32(n1, d) => {
             debug_assert!(d >= 3);
             let MU32 {
-                mulBy,
-                doAdd,
-                shiftBy,
-            } = magicU32(d);
+                mul_by,
+                do_add,
+                shift_by,
+            } = magic_u32(d);
             let qf; // final quotient
-            let q0 = pos.ins().iconst(I32, mulBy as i64);
+            let q0 = pos.ins().iconst(I32, mul_by as i64);
             let q1 = pos.ins().umulhi(n1, q0);
-            if doAdd {
-                debug_assert!(shiftBy >= 1 && shiftBy <= 32);
+            if do_add {
+                debug_assert!(shift_by >= 1 && shift_by <= 32);
                 let t1 = pos.ins().isub(n1, q1);
                 let t2 = pos.ins().ushr_imm(t1, 1);
                 let t3 = pos.ins().iadd(t2, q1);
-                // I never found any case where shiftBy == 1 here.
+                // I never found any case where shift_by == 1 here.
                 // So there's no attempt to fold out a zero shift.
-                debug_assert_ne!(shiftBy, 1);
-                qf = pos.ins().ushr_imm(t3, (shiftBy - 1) as i64);
+                debug_assert_ne!(shift_by, 1);
+                qf = pos.ins().ushr_imm(t3, (shift_by - 1) as i64);
             } else {
-                debug_assert!(shiftBy >= 0 && shiftBy <= 31);
-                // Whereas there are known cases here for shiftBy == 0.
-                if shiftBy > 0 {
-                    qf = pos.ins().ushr_imm(q1, shiftBy as i64);
+                debug_assert!(shift_by >= 0 && shift_by <= 31);
+                // Whereas there are known cases here for shift_by == 0.
+                if shift_by > 0 {
+                    qf = pos.ins().ushr_imm(q1, shift_by as i64);
                 } else {
                     qf = q1;
                 }
@@ -253,27 +253,27 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
         DivRemByConstInfo::DivU64(n1, d) | DivRemByConstInfo::RemU64(n1, d) => {
             debug_assert!(d >= 3);
             let MU64 {
-                mulBy,
-                doAdd,
-                shiftBy,
-            } = magicU64(d);
+                mul_by,
+                do_add,
+                shift_by,
+            } = magic_u64(d);
             let qf; // final quotient
-            let q0 = pos.ins().iconst(I64, mulBy as i64);
+            let q0 = pos.ins().iconst(I64, mul_by as i64);
             let q1 = pos.ins().umulhi(n1, q0);
-            if doAdd {
-                debug_assert!(shiftBy >= 1 && shiftBy <= 64);
+            if do_add {
+                debug_assert!(shift_by >= 1 && shift_by <= 64);
                 let t1 = pos.ins().isub(n1, q1);
                 let t2 = pos.ins().ushr_imm(t1, 1);
                 let t3 = pos.ins().iadd(t2, q1);
-                // I never found any case where shiftBy == 1 here.
+                // I never found any case where shift_by == 1 here.
                 // So there's no attempt to fold out a zero shift.
-                debug_assert_ne!(shiftBy, 1);
-                qf = pos.ins().ushr_imm(t3, (shiftBy - 1) as i64);
+                debug_assert_ne!(shift_by, 1);
+                qf = pos.ins().ushr_imm(t3, (shift_by - 1) as i64);
             } else {
-                debug_assert!(shiftBy >= 0 && shiftBy <= 63);
-                // Whereas there are known cases here for shiftBy == 0.
-                if shiftBy > 0 {
-                    qf = pos.ins().ushr_imm(q1, shiftBy as i64);
+                debug_assert!(shift_by >= 0 && shift_by <= 63);
+                // Whereas there are known cases here for shift_by == 0.
+                if shift_by > 0 {
+                    qf = pos.ins().ushr_imm(q1, shift_by as i64);
                 } else {
                     qf = q1;
                 }
@@ -334,21 +334,21 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             } else {
                 // S32 div, rem by a non-power-of-2
                 debug_assert!(d < -2 || d > 2);
-                let MS32 { mulBy, shiftBy } = magicS32(d);
-                let q0 = pos.ins().iconst(I32, mulBy as i64);
+                let MS32 { mul_by, shift_by } = magic_s32(d);
+                let q0 = pos.ins().iconst(I32, mul_by as i64);
                 let q1 = pos.ins().smulhi(n1, q0);
-                let q2 = if d > 0 && mulBy < 0 {
+                let q2 = if d > 0 && mul_by < 0 {
                     pos.ins().iadd(q1, n1)
-                } else if d < 0 && mulBy > 0 {
+                } else if d < 0 && mul_by > 0 {
                     pos.ins().isub(q1, n1)
                 } else {
                     q1
                 };
-                debug_assert!(shiftBy >= 0 && shiftBy <= 31);
-                let q3 = if shiftBy == 0 {
+                debug_assert!(shift_by >= 0 && shift_by <= 31);
+                let q3 = if shift_by == 0 {
                     q2
                 } else {
-                    pos.ins().sshr_imm(q2, shiftBy as i64)
+                    pos.ins().sshr_imm(q2, shift_by as i64)
                 };
                 let t1 = pos.ins().ushr_imm(q3, 31);
                 let qf = pos.ins().iadd(q3, t1);
@@ -409,21 +409,21 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             } else {
                 // S64 div, rem by a non-power-of-2
                 debug_assert!(d < -2 || d > 2);
-                let MS64 { mulBy, shiftBy } = magicS64(d);
-                let q0 = pos.ins().iconst(I64, mulBy);
+                let MS64 { mul_by, shift_by } = magic_s64(d);
+                let q0 = pos.ins().iconst(I64, mul_by);
                 let q1 = pos.ins().smulhi(n1, q0);
-                let q2 = if d > 0 && mulBy < 0 {
+                let q2 = if d > 0 && mul_by < 0 {
                     pos.ins().iadd(q1, n1)
-                } else if d < 0 && mulBy > 0 {
+                } else if d < 0 && mul_by > 0 {
                     pos.ins().isub(q1, n1)
                 } else {
                     q1
                 };
-                debug_assert!(shiftBy >= 0 && shiftBy <= 63);
-                let q3 = if shiftBy == 0 {
+                debug_assert!(shift_by >= 0 && shift_by <= 63);
+                let q3 = if shift_by == 0 {
                     q2
                 } else {
-                    pos.ins().sshr_imm(q2, shiftBy as i64)
+                    pos.ins().sshr_imm(q2, shift_by as i64)
                 };
                 let t1 = pos.ins().ushr_imm(q3, 63);
                 let qf = pos.ins().iadd(q3, t1);

From 813810008b4c57d1f134bff7ae7e3fae3d4dc48c Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 3 Jan 2019 15:10:25 +0100
Subject: [PATCH 2285/3084] Replace deprecated trim_left_matches by
 trim_start_matches;

---
 lib/filetests/src/match_directive.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/filetests/src/match_directive.rs b/lib/filetests/src/match_directive.rs
index 29c9fa6bae..a3f951f3b4 100644
--- a/lib/filetests/src/match_directive.rs
+++ b/lib/filetests/src/match_directive.rs
@@ -9,7 +9,7 @@ pub fn match_directive<'a>(comment: &'a str, directive: &str) -> Option<&'a str>
         directive.ends_with(':'),
         "Directive must include trailing colon"
     );
-    let text = comment.trim_left_matches(';').trim_start();
+    let text = comment.trim_start_matches(';').trim_start();
     if text.starts_with(directive) {
         Some(text[directive.len()..].trim())
     } else {

From c78a9a4a7c47176bbfd7011a7b58a3bc6d828812 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 3 Jan 2019 15:55:33 +0100
Subject: [PATCH 2286/3084] Bump log and file-per-thread-logger;

---
 cranelift/Cargo.toml     | 2 +-
 lib/codegen/Cargo.toml   | 2 +-
 lib/filetests/Cargo.toml | 4 ++--
 lib/frontend/Cargo.toml  | 2 +-
 lib/module/Cargo.toml    | 2 +-
 lib/wasm/Cargo.toml      | 2 +-
 6 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index bbe81eaaed..04c2168578 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -36,7 +36,7 @@ capstone = { version = "0.5.0", optional = true }
 wabt = { version = "0.7.0", optional = true }
 target-lexicon = "0.2.0"
 pretty_env_logger = "0.3.0"
-file-per-thread-logger = "0.1.1"
+file-per-thread-logger = "0.1.2"
 
 [features]
 default = ["disas", "wasm"]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index cddb573715..6281a77644 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -19,7 +19,7 @@ failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 target-lexicon = { version = "0.2.0", default-features = false }
-log = { version = "0.4.4", default-features = false }
+log = { version = "0.4.6", default-features = false }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
 # machine code. Integration tests that need external dependencies can be
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 2dc722be82..77505a10e5 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -13,7 +13,7 @@ edition = "2018"
 cranelift-codegen = { path = "../codegen", version = "0.26.0", features = ["testing_hooks"] }
 cranelift-reader = { path = "../reader", version = "0.26.0" }
 cranelift-preopt = { path = "../preopt", version = "0.26.0" }
-file-per-thread-logger = "0.1.1"
+file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
-log = "0.4.4"
+log = "0.4.6"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 826edf278e..00391d83bf 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -13,7 +13,7 @@ edition = "2018"
 [dependencies]
 cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
-log = { version = "0.4.4", default-features = false }
+log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 
 [features]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 826a00cf72..9a564c8b31 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -15,7 +15,7 @@ cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features
 cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = "0.1.1"
-log = { version = "0.4.4", default-features = false }
+log = { version = "0.4.6", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 71b4b2547e..45c2419bf7 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -18,7 +18,7 @@ cranelift-frontend = { path = "../frontend", version = "0.26.0", default-feature
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
-log = { version = "0.4.4", default-features = false }
+log = { version = "0.4.6", default-features = false }
 cast = { version = "0.2.2" }
 
 [dev-dependencies]

From 03c22ed67a2d48378952a352851866db82e74688 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 3 Jan 2019 18:09:38 +0100
Subject: [PATCH 2287/3084] Remove warnings in tests;

---
 lib/entity/src/list.rs           | 1 -
 lib/frontend/src/switch.rs       | 4 ++--
 lib/serde/src/clif-json.rs       | 3 +--
 lib/serde/src/serde_clif_json.rs | 4 ++--
 4 files changed, 5 insertions(+), 7 deletions(-)

diff --git a/lib/entity/src/list.rs b/lib/entity/src/list.rs
index 617ea2db32..4dc3ce575d 100644
--- a/lib/entity/src/list.rs
+++ b/lib/entity/src/list.rs
@@ -484,7 +484,6 @@ impl EntityList {
 mod tests {
     use super::*;
     use super::{sclass_for_length, sclass_size};
-    use crate::packed_option::ReservedValue;
     use crate::EntityRef;
 
     /// An opaque reference to an instruction in a function.
diff --git a/lib/frontend/src/switch.rs b/lib/frontend/src/switch.rs
index d5af64ed8c..5ecb4edc38 100644
--- a/lib/frontend/src/switch.rs
+++ b/lib/frontend/src/switch.rs
@@ -223,8 +223,8 @@ mod tests {
             }
             func
                 .to_string()
-                .trim_left_matches("function u0:0() fast {\n")
-                .trim_right_matches("\n}\n")
+                .trim_start_matches("function u0:0() fast {\n")
+                .trim_end_matches("\n}\n")
                 .to_string()
         }};
     }
diff --git a/lib/serde/src/clif-json.rs b/lib/serde/src/clif-json.rs
index fa25a723ed..9253d08d1c 100644
--- a/lib/serde/src/clif-json.rs
+++ b/lib/serde/src/clif-json.rs
@@ -100,8 +100,7 @@ fn main() {
             }
         }
         ("deserialize", Some(m)) => {
-            let mut file =
-                File::open(m.value_of("FILE").unwrap()).expect("Unable to open the file");
+            let file = File::open(m.value_of("FILE").unwrap()).expect("Unable to open the file");
             call_de(&file)
         }
         _ => Err("Invalid subcommand.".to_string()),
diff --git a/lib/serde/src/serde_clif_json.rs b/lib/serde/src/serde_clif_json.rs
index f771fc7438..50655b732d 100644
--- a/lib/serde/src/serde_clif_json.rs
+++ b/lib/serde/src/serde_clif_json.rs
@@ -745,7 +745,7 @@ pub fn populate_inst(func: &Function, ebb: Ebb) -> Vec {
     let mut ser_vec: Vec = Vec::new();
     let ret_iter = func.layout.ebb_insts(ebb);
     for inst in ret_iter {
-        let mut ser_inst: SerInst = SerInst::new(inst, &func);
+        let ser_inst: SerInst = SerInst::new(inst, &func);
         ser_vec.push(ser_inst);
     }
     ser_vec
@@ -867,7 +867,7 @@ impl SerObj {
     pub fn new(funcs: &[Function]) -> Self {
         let mut func_vec: Vec = Vec::new();
         for func in funcs {
-            let mut ser_func: SerFunction = SerFunction::new(&func);
+            let ser_func: SerFunction = SerFunction::new(&func);
             func_vec.push(ser_func);
         }
         Self::create_new(func_vec)

From 7f250e340c4781529d3e66c1ae8bb72c2b893109 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 18 Dec 2018 12:11:32 -0800
Subject: [PATCH 2288/3084] Simplify `ModuleEnvironment`.

Remove some unneeded functions, and remove the `GlobalInit` special case
for data and elem initializer offsets; implementations that want that
information can provide it for themselves.
---
 cranelift/src/wasm.rs               |  6 ++----
 lib/wasm/src/environ/dummy.rs       | 25 +++++++++----------------
 lib/wasm/src/environ/spec.rs        | 13 +------------
 lib/wasm/src/sections_translator.rs | 18 ++----------------
 4 files changed, 14 insertions(+), 48 deletions(-)

diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 9ffea3adca..00574f6259 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -7,18 +7,16 @@
     allow(too_many_arguments, cyclomatic_complexity)
 )]
 
+use crate::utils::{parse_sets_and_triple, read_to_end};
 use cranelift_codegen::print_errors::{pretty_error, pretty_verifier_error};
 use cranelift_codegen::settings::FlagsOrIsa;
 use cranelift_codegen::timing;
 use cranelift_codegen::Context;
 use cranelift_entity::EntityRef;
-use cranelift_wasm::{
-    translate_module, DummyEnvironment, FuncIndex, ModuleEnvironment, ReturnMode,
-};
+use cranelift_wasm::{translate_module, DummyEnvironment, FuncIndex, ReturnMode};
 use std::path::Path;
 use std::path::PathBuf;
 use term;
-use crate::utils::{parse_sets_and_triple, read_to_end};
 use wabt::wat2wasm;
 
 macro_rules! vprintln {
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 81212fa379..290aa38c0d 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -138,6 +138,15 @@ impl DummyEnvironment {
     pub fn func_env(&self) -> DummyFuncEnvironment {
         DummyFuncEnvironment::new(&self.info, self.return_mode)
     }
+
+    fn get_func_type(&self, func_index: FuncIndex) -> SignatureIndex {
+        self.info.functions[func_index].entity
+    }
+
+    /// Return the number of imported functions within this `DummyEnvironment`.
+    pub fn get_num_func_imports(&self) -> usize {
+        self.info.imported_funcs.len()
+    }
 }
 
 /// The `FuncEnvironment` implementation for use by the `DummyEnvironment`.
@@ -345,10 +354,6 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         self.info.signatures.push(sig.clone());
     }
 
-    fn get_signature(&self, sig_index: SignatureIndex) -> &ir::Signature {
-        &self.info.signatures[sig_index]
-    }
-
     fn declare_func_import(
         &mut self,
         sig_index: SignatureIndex,
@@ -366,18 +371,10 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
             .push((String::from(module), String::from(field)));
     }
 
-    fn get_num_func_imports(&self) -> usize {
-        self.info.imported_funcs.len()
-    }
-
     fn declare_func_type(&mut self, sig_index: SignatureIndex) {
         self.info.functions.push(Exportable::new(sig_index));
     }
 
-    fn get_func_type(&self, func_index: FuncIndex) -> SignatureIndex {
-        self.info.functions[func_index].entity
-    }
-
     fn declare_global(&mut self, global: Global) {
         self.info.globals.push(Exportable::new(global));
     }
@@ -389,10 +386,6 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
             .push((String::from(module), String::from(field)));
     }
 
-    fn get_global(&self, global_index: GlobalIndex) -> &Global {
-        &self.info.globals[global_index].entity
-    }
-
     fn declare_table(&mut self, table: Table) {
         self.info.tables.push(Exportable::new(table));
     }
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index cfd943ed78..9ad951890b 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -248,9 +248,6 @@ pub trait ModuleEnvironment<'data> {
     /// Declares a function signature to the environment.
     fn declare_signature(&mut self, sig: &ir::Signature);
 
-    /// Return the signature with the given index.
-    fn get_signature(&self, sig_index: SignatureIndex) -> &ir::Signature;
-
     /// Declares a function import to the environment.
     fn declare_func_import(
         &mut self,
@@ -259,24 +256,15 @@ pub trait ModuleEnvironment<'data> {
         field: &'data str,
     );
 
-    /// Return the number of imported funcs.
-    fn get_num_func_imports(&self) -> usize;
-
     /// Declares the type (signature) of a local function in the module.
     fn declare_func_type(&mut self, sig_index: SignatureIndex);
 
-    /// Return the signature index for the given function index.
-    fn get_func_type(&self, func_index: FuncIndex) -> SignatureIndex;
-
     /// Declares a global to the environment.
     fn declare_global(&mut self, global: Global);
 
     /// Declares a global import to the environment.
     fn declare_global_import(&mut self, global: Global, module: &'data str, field: &'data str);
 
-    /// Return the global for the given global index.
-    fn get_global(&self, global_index: GlobalIndex) -> &Global;
-
     /// Declares a table to the environment.
     fn declare_table(&mut self, table: Table);
 
@@ -291,6 +279,7 @@ pub trait ModuleEnvironment<'data> {
         offset: usize,
         elements: Vec,
     );
+
     /// Declares a memory to the environment
     fn declare_memory(&mut self, memory: Memory);
 
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 2ee60c19a9..89c1750c82 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -244,14 +244,7 @@ pub fn parse_element_section<'data>(
         let mut init_expr_reader = init_expr.get_binary_reader();
         let (base, offset) = match init_expr_reader.read_operator()? {
             Operator::I32Const { value } => (None, value as u32 as usize),
-            Operator::GetGlobal { global_index } => match environ
-                .get_global(GlobalIndex::from_u32(global_index))
-                .initializer
-            {
-                GlobalInit::I32Const(value) => (None, value as u32 as usize),
-                GlobalInit::Import => (Some(GlobalIndex::from_u32(global_index)), 0),
-                _ => panic!("should not happen"),
-            },
+            Operator::GetGlobal { global_index } => (Some(GlobalIndex::from_u32(global_index)), 0),
             ref s => panic!("unsupported init expr in element section: {:?}", s),
         };
         let items_reader = items.get_items_reader()?;
@@ -292,14 +285,7 @@ pub fn parse_data_section<'data>(
         let mut init_expr_reader = init_expr.get_binary_reader();
         let (base, offset) = match init_expr_reader.read_operator()? {
             Operator::I32Const { value } => (None, value as u32 as usize),
-            Operator::GetGlobal { global_index } => match environ
-                .get_global(GlobalIndex::from_u32(global_index))
-                .initializer
-            {
-                GlobalInit::I32Const(value) => (None, value as u32 as usize),
-                GlobalInit::Import => (Some(GlobalIndex::from_u32(global_index)), 0),
-                _ => panic!("should not happen"),
-            },
+            Operator::GetGlobal { global_index } => (Some(GlobalIndex::from_u32(global_index)), 0),
             ref s => panic!("unsupported init expr in data section: {:?}", s),
         };
         environ.declare_data_initialization(

From 054e6fcf07f486769c539ae63ffa1271b6affc09 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 18 Dec 2018 12:55:11 -0800
Subject: [PATCH 2289/3084] Add functions to `ModuleEnvironment` to support
 reserving buffers up front.

These default to doing nothing, but implementations can override them to
preallocate buffers.

Also, reorder the functions in `ModuleEnvironment` to more closely
reflect the sequence from the WebAssembly module layout.
---
 lib/wasm/src/environ/spec.rs | 95 ++++++++++++++++++++++++++----------
 1 file changed, 70 insertions(+), 25 deletions(-)

diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 9ad951890b..37d741a4f9 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -245,9 +245,17 @@ pub trait ModuleEnvironment<'data> {
     /// Get the information needed to produce Cranelift IR for the current target.
     fn target_config(&self) -> TargetFrontendConfig;
 
+    /// Provides the number of signatures up front. By default this does nothing, but
+    /// implementations can use this to preallocate memory if desired.
+    fn reserve_signatures(&mut self, _num: u32) {}
+
     /// Declares a function signature to the environment.
     fn declare_signature(&mut self, sig: &ir::Signature);
 
+    /// Provides the number of imports up front. By default this does nothing, but
+    /// implementations can use this to preallocate memory if desired.
+    fn reserve_imports(&mut self, _num: u32) {}
+
     /// Declares a function import to the environment.
     fn declare_func_import(
         &mut self,
@@ -256,20 +264,71 @@ pub trait ModuleEnvironment<'data> {
         field: &'data str,
     );
 
-    /// Declares the type (signature) of a local function in the module.
-    fn declare_func_type(&mut self, sig_index: SignatureIndex);
+    /// Declares a table import to the environment.
+    fn declare_table_import(&mut self, table: Table, module: &'data str, field: &'data str);
 
-    /// Declares a global to the environment.
-    fn declare_global(&mut self, global: Global);
+    /// Declares a memory import to the environment.
+    fn declare_memory_import(&mut self, memory: Memory, module: &'data str, field: &'data str);
 
     /// Declares a global import to the environment.
     fn declare_global_import(&mut self, global: Global, module: &'data str, field: &'data str);
 
+    /// Notifies the implementation that all imports have been declared.
+    fn finish_imports(&mut self) {}
+
+    /// Provides the number of defined functions up front. By default this does nothing, but
+    /// implementations can use this to preallocate memory if desired.
+    fn reserve_func_types(&mut self, _num: u32) {}
+
+    /// Declares the type (signature) of a local function in the module.
+    fn declare_func_type(&mut self, sig_index: SignatureIndex);
+
+    /// Provides the number of defined tables up front. By default this does nothing, but
+    /// implementations can use this to preallocate memory if desired.
+    fn reserve_tables(&mut self, _num: u32) {}
+
     /// Declares a table to the environment.
     fn declare_table(&mut self, table: Table);
 
-    /// Declares a table import to the environment.
-    fn declare_table_import(&mut self, table: Table, module: &'data str, field: &'data str);
+    /// Provides the number of defined memories up front. By default this does nothing, but
+    /// implementations can use this to preallocate memory if desired.
+    fn reserve_memories(&mut self, _num: u32) {}
+
+    /// Declares a memory to the environment
+    fn declare_memory(&mut self, memory: Memory);
+
+    /// Provides the number of defined globals up front. By default this does nothing, but
+    /// implementations can use this to preallocate memory if desired.
+    fn reserve_globals(&mut self, _num: u32) {}
+
+    /// Declares a global to the environment.
+    fn declare_global(&mut self, global: Global);
+
+    /// Provides the number of exports up front. By default this does nothing, but
+    /// implementations can use this to preallocate memory if desired.
+    fn reserve_exports(&mut self, _num: u32) {}
+
+    /// Declares a function export to the environment.
+    fn declare_func_export(&mut self, func_index: FuncIndex, name: &'data str);
+
+    /// Declares a table export to the environment.
+    fn declare_table_export(&mut self, table_index: TableIndex, name: &'data str);
+
+    /// Declares a memory export to the environment.
+    fn declare_memory_export(&mut self, memory_index: MemoryIndex, name: &'data str);
+
+    /// Declares a global export to the environment.
+    fn declare_global_export(&mut self, global_index: GlobalIndex, name: &'data str);
+
+    /// Notifies the implementation that all exports have been declared.
+    fn finish_exports(&mut self) {}
+
+    /// Declares the optional start function.
+    fn declare_start_func(&mut self, index: FuncIndex);
+
+    /// Provides the number of element initializers up front. By default this does nothing, but
+    /// implementations can use this to preallocate memory if desired.
+    fn reserve_table_elements(&mut self, _num: u32) {}
 
     /// Fills a declared table with references to functions in the module.
     fn declare_table_elements(
@@ -280,11 +339,12 @@ pub trait ModuleEnvironment<'data> {
         elements: Vec,
     );
 
-    /// Declares a memory to the environment
-    fn declare_memory(&mut self, memory: Memory);
+    /// Provides the contents of a function body.
+    fn define_function_body(&mut self, body_bytes: &'data [u8]) -> WasmResult<()>;
 
-    /// Declares a memory import to the environment.
-    fn declare_memory_import(&mut self, memory: Memory, module: &'data str, field: &'data str);
+    /// Provides the number of data initializers up front. By default this does nothing, but
+    /// implementations can use this to preallocate memory if desired.
+    fn reserve_data_initializers(&mut self, _num: u32) {}
 
     /// Fills a declared memory with bytes at module instantiation.
     fn declare_data_initialization(
@@ -294,19 +354,4 @@ pub trait ModuleEnvironment<'data> {
         offset: usize,
         data: &'data [u8],
     );
-
-    /// Declares a function export to the environment.
-    fn declare_func_export(&mut self, func_index: FuncIndex, name: &'data str);
-    /// Declares a table export to the environment.
-    fn declare_table_export(&mut self, table_index: TableIndex, name: &'data str);
-    /// Declares a memory export to the environment.
-    fn declare_memory_export(&mut self, memory_index: MemoryIndex, name: &'data str);
-    /// Declares a global export to the environment.
-    fn declare_global_export(&mut self, global_index: GlobalIndex, name: &'data str);
-
-    /// Declares a start function.
-    fn declare_start_func(&mut self, index: FuncIndex);
-
-    /// Provides the contents of a function body.
-    fn define_function_body(&mut self, body_bytes: &'data [u8]) -> WasmResult<()>;
 }

From 712063336310c5a107ef06a34f0419e46f4b036b Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 18 Dec 2018 14:13:46 -0800
Subject: [PATCH 2290/3084] Change the `elems` parameter of
 `declare_table_elements` to a boxed slice.

Implementations that want a full `Vec` can use `into_vec` to convert it
back.
---
 lib/wasm/src/environ/dummy.rs       | 2 +-
 lib/wasm/src/environ/spec.rs        | 4 ++--
 lib/wasm/src/sections_translator.rs | 7 ++++++-
 3 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 290aa38c0d..5bf1961683 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -402,7 +402,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         _table_index: TableIndex,
         _base: Option,
         _offset: usize,
-        _elements: Vec,
+        _elements: Box<[FuncIndex]>,
     ) {
         // We do nothing
     }
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 37d741a4f9..92c7fa9e2b 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -14,8 +14,8 @@ use cranelift_codegen::ir::immediates::Offset32;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::isa::TargetFrontendConfig;
 use failure_derive::Fail;
+use std::boxed::Box;
 use std::convert::From;
-use std::vec::Vec;
 use wasmparser::BinaryReaderError;
 
 /// The value of a WebAssembly global variable.
@@ -336,7 +336,7 @@ pub trait ModuleEnvironment<'data> {
         table_index: TableIndex,
         base: Option,
         offset: usize,
-        elements: Vec,
+        elements: Box<[FuncIndex]>,
     );
 
     /// Provides the contents of a function body.
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 89c1750c82..6192ec7b0f 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -253,7 +253,12 @@ pub fn parse_element_section<'data>(
             let x = item?;
             elems.push(FuncIndex::from_u32(x));
         }
-        environ.declare_table_elements(TableIndex::from_u32(table_index), base, offset, elems)
+        environ.declare_table_elements(
+            TableIndex::from_u32(table_index),
+            base,
+            offset,
+            elems.into_boxed_slice(),
+        )
     }
     Ok(())
 }

From 6ec0b3c6bfb2d3c9231d00181bdd69f9fc4f9f88 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 3 Jan 2019 11:43:02 -0800
Subject: [PATCH 2291/3084] Add a comment.

---
 lib/wasm/src/environ/spec.rs | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 92c7fa9e2b..93a82cca88 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -340,6 +340,9 @@ pub trait ModuleEnvironment<'data> {
     );
 
     /// Provides the contents of a function body.
+    ///
+    /// Note there's no `reserve_function_bodies` function because the number of
+    /// functions is already provided by `reserve_func_types`.
     fn define_function_body(&mut self, body_bytes: &'data [u8]) -> WasmResult<()>;
 
     /// Provides the number of data initializers up front. By default this does nothing, but

From 13138b65f7161c7572aa2c3064be47b1ff4b2cfc Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 3 Jan 2019 11:43:15 -0800
Subject: [PATCH 2292/3084] Call the new ModuleEnvironment reserve functions.

---
 lib/wasm/src/sections_translator.rs | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 6192ec7b0f..e3f36b754d 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -28,6 +28,8 @@ pub fn parse_type_section(
     types: TypeSectionReader,
     environ: &mut ModuleEnvironment,
 ) -> WasmResult<()> {
+    environ.reserve_signatures(types.get_count());
+
     for entry in types {
         match entry? {
             FuncType {
@@ -59,6 +61,8 @@ pub fn parse_import_section<'data>(
     imports: ImportSectionReader<'data>,
     environ: &mut ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
+    environ.reserve_imports(imports.get_count());
+
     for entry in imports {
         let import = entry?;
 
@@ -113,6 +117,8 @@ pub fn parse_import_section<'data>(
             }
         }
     }
+
+    environ.finish_imports();
     Ok(())
 }
 
@@ -121,10 +127,13 @@ pub fn parse_function_section(
     functions: FunctionSectionReader,
     environ: &mut ModuleEnvironment,
 ) -> WasmResult<()> {
+    environ.reserve_func_types(functions.get_count());
+
     for entry in functions {
         let sigindex = entry?;
         environ.declare_func_type(SignatureIndex::from_u32(sigindex));
     }
+
     Ok(())
 }
 
@@ -133,6 +142,8 @@ pub fn parse_table_section(
     tables: TableSectionReader,
     environ: &mut ModuleEnvironment,
 ) -> WasmResult<()> {
+    environ.reserve_tables(tables.get_count());
+
     for entry in tables {
         let table = entry?;
         environ.declare_table(Table {
@@ -144,6 +155,7 @@ pub fn parse_table_section(
             maximum: table.limits.maximum,
         });
     }
+
     Ok(())
 }
 
@@ -152,6 +164,8 @@ pub fn parse_memory_section(
     memories: MemorySectionReader,
     environ: &mut ModuleEnvironment,
 ) -> WasmResult<()> {
+    environ.reserve_memories(memories.get_count());
+
     for entry in memories {
         let memory = entry?;
         environ.declare_memory(Memory {
@@ -160,6 +174,7 @@ pub fn parse_memory_section(
             shared: memory.shared,
         });
     }
+
     Ok(())
 }
 
@@ -168,6 +183,8 @@ pub fn parse_global_section(
     globals: GlobalSectionReader,
     environ: &mut ModuleEnvironment,
 ) -> WasmResult<()> {
+    environ.reserve_globals(globals.get_count());
+
     for entry in globals {
         let wasmparser::Global {
             ty: GlobalType {
@@ -194,6 +211,7 @@ pub fn parse_global_section(
         };
         environ.declare_global(global);
     }
+
     Ok(())
 }
 
@@ -202,6 +220,8 @@ pub fn parse_export_section<'data>(
     exports: ExportSectionReader<'data>,
     environ: &mut ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
+    environ.reserve_exports(exports.get_count());
+
     for entry in exports {
         let Export {
             field,
@@ -221,6 +241,8 @@ pub fn parse_export_section<'data>(
             ExternalKind::Global => environ.declare_global_export(GlobalIndex::new(index), name),
         }
     }
+
+    environ.finish_exports();
     Ok(())
 }
 
@@ -235,6 +257,8 @@ pub fn parse_element_section<'data>(
     elements: ElementSectionReader<'data>,
     environ: &mut ModuleEnvironment,
 ) -> WasmResult<()> {
+    environ.reserve_table_elements(elements.get_count());
+
     for entry in elements {
         let Element {
             table_index,
@@ -281,6 +305,8 @@ pub fn parse_data_section<'data>(
     data: DataSectionReader<'data>,
     environ: &mut ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
+    environ.reserve_data_initializers(data.get_count());
+
     for entry in data {
         let Data {
             memory_index,
@@ -300,5 +326,6 @@ pub fn parse_data_section<'data>(
             data,
         );
     }
+
     Ok(())
 }

From a7aee246e9389fc16765c35fc899387023c06972 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 7 Jan 2019 12:34:21 -0800
Subject: [PATCH 2293/3084] Pass `declare_signature`'s argument by value.

`parse_type_section` doesn't need the value after passing it, and this
allows callees to avoid cloning if they need their own copy.
---
 lib/wasm/src/environ/dummy.rs       | 4 ++--
 lib/wasm/src/environ/spec.rs        | 2 +-
 lib/wasm/src/sections_translator.rs | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 5bf1961683..081244ca49 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -350,8 +350,8 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         self.info.config
     }
 
-    fn declare_signature(&mut self, sig: &ir::Signature) {
-        self.info.signatures.push(sig.clone());
+    fn declare_signature(&mut self, sig: ir::Signature) {
+        self.info.signatures.push(sig);
     }
 
     fn declare_func_import(
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 93a82cca88..27249c1234 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -250,7 +250,7 @@ pub trait ModuleEnvironment<'data> {
     fn reserve_signatures(&mut self, _num: u32) {}
 
     /// Declares a function signature to the environment.
-    fn declare_signature(&mut self, sig: &ir::Signature);
+    fn declare_signature(&mut self, sig: ir::Signature);
 
     /// Provides the number of imports up front. By default this does nothing, but
     /// implementations can use this to preallocate memory if desired.
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index e3f36b754d..0a3aff5803 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -48,7 +48,7 @@ pub fn parse_type_section(
                         .expect("only numeric types are supported in function signatures");
                     AbiParam::new(cret_arg)
                 }));
-                environ.declare_signature(&sig);
+                environ.declare_signature(sig);
             }
             ref s => panic!("unsupported type: {:?}", s),
         }

From 9eba81a8d9323e57d81493e03adafefc09ece717 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 7 Jan 2019 11:21:28 -0800
Subject: [PATCH 2294/3084] Update clippy settings and fix a few clippy
 warnings.

---
 cranelift/src/compile.rs                  |   4 +-
 cranelift/src/wasm.rs                     |   2 +-
 lib/bforest/src/lib.rs                    |   5 +-
 lib/bforest/src/map.rs                    |   2 +-
 lib/bforest/src/pool.rs                   |   2 +-
 lib/bforest/src/set.rs                    |   2 +-
 lib/codegen/meta/src/constant_hash.rs     |   3 +-
 lib/codegen/meta/src/gen_settings.rs      |   3 +-
 lib/codegen/src/binemit/memorysink.rs     |   6 +-
 lib/codegen/src/cursor.rs                 |   2 +-
 lib/codegen/src/dce.rs                    |   5 +-
 lib/codegen/src/divconst_magic_numbers.rs | 281 +++++++++++++---------
 lib/codegen/src/ir/globalvalue.rs         |   3 +-
 lib/codegen/src/ir/instructions.rs        |   4 +-
 lib/codegen/src/legalizer/heap.rs         |   2 +-
 lib/codegen/src/lib.rs                    |  32 +--
 lib/codegen/src/licm.rs                   |   2 +-
 lib/codegen/src/regalloc/solver.rs        |   2 +-
 lib/codegen/src/regalloc/virtregs.rs      |   2 +-
 lib/entity/src/lib.rs                     |   8 +-
 lib/entity/src/map.rs                     |   4 +-
 lib/entity/src/sparse.rs                  |  19 +-
 lib/faerie/src/backend.rs                 |   3 +-
 lib/faerie/src/lib.rs                     |   2 +-
 lib/filetests/src/lib.rs                  |   2 +-
 lib/frontend/src/frontend.rs              |  12 +-
 lib/frontend/src/lib.rs                   |   2 +-
 lib/module/src/lib.rs                     |   5 +-
 lib/native/src/lib.rs                     |   5 +-
 lib/preopt/src/lib.rs                     |   5 +-
 lib/reader/src/lib.rs                     |   5 +-
 lib/reader/src/parser.rs                  |   5 +-
 lib/serde/src/clif-json.rs                |   5 +-
 lib/simplejit/src/backend.rs              |  10 +-
 lib/simplejit/src/lib.rs                  |   5 +-
 lib/umbrella/src/lib.rs                   |   5 +-
 lib/wasm/src/code_translator.rs           |   4 +-
 lib/wasm/src/environ/dummy.rs             |   2 +-
 lib/wasm/src/environ/spec.rs              |   2 +-
 lib/wasm/src/lib.rs                       |   5 +-
 40 files changed, 246 insertions(+), 233 deletions(-)

diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs
index 50b21faf15..9903dd6161 100644
--- a/cranelift/src/compile.rs
+++ b/cranelift/src/compile.rs
@@ -111,9 +111,7 @@ fn handle_module(
             .compile(isa)
             .map_err(|err| pretty_error(&context.func, Some(isa), err))?;
 
-        let mut mem = Vec::new();
-        mem.resize(total_size as usize, 0);
-
+        let mut mem = vec![0; total_size as usize];
         let mut relocs = PrintRelocs { flag_print };
         let mut traps = PrintTraps { flag_print };
         let mut code_sink: binemit::MemoryCodeSink;
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 00574f6259..af8e818697 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -4,7 +4,7 @@
 //! Reads Wasm binary/text files, translates the functions' code to Cranelift IR.
 #![cfg_attr(
     feature = "cargo-clippy",
-    allow(too_many_arguments, cyclomatic_complexity)
+    allow(clippy::too_many_arguments, clippy::cyclomatic_complexity)
 )]
 
 use crate::utils::{parse_sets_and_triple, read_to_end};
diff --git a/lib/bforest/src/lib.rs b/lib/bforest/src/lib.rs
index ee9213b00a..e78292600e 100644
--- a/lib/bforest/src/lib.rs
+++ b/lib/bforest/src/lib.rs
@@ -17,10 +17,7 @@
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", warn(unstable_features))]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(
-    feature = "cargo-clippy",
-    allow(new_without_default, new_without_default_derive)
-)]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/bforest/src/map.rs b/lib/bforest/src/map.rs
index a809282833..313b870f49 100644
--- a/lib/bforest/src/map.rs
+++ b/lib/bforest/src/map.rs
@@ -284,7 +284,7 @@ where
     ///
     /// If the cursor reaches the end, return `None` and leave the cursor at the off-the-end
     /// position.
-    #[cfg_attr(feature = "cargo-clippy", allow(should_implement_trait))]
+    #[cfg_attr(feature = "cargo-clippy", allow(clippy::should_implement_trait))]
     pub fn next(&mut self) -> Option<(K, V)> {
         self.path.next(self.pool)
     }
diff --git a/lib/bforest/src/pool.rs b/lib/bforest/src/pool.rs
index cec5e40a7d..0eb873ccab 100644
--- a/lib/bforest/src/pool.rs
+++ b/lib/bforest/src/pool.rs
@@ -63,7 +63,7 @@ impl NodePool {
     pub fn free_tree(&mut self, node: Node) {
         if let NodeData::Inner { size, tree, .. } = self[node] {
             // Note that we have to capture `tree` by value to avoid borrow checker trouble.
-            #[cfg_attr(feature = "cargo-clippy", allow(needless_range_loop))]
+            #[cfg_attr(feature = "cargo-clippy", allow(clippy::needless_range_loop))]
             for i in 0..usize::from(size + 1) {
                 // Recursively free sub-trees. This recursion can never be deeper than `MAX_PATH`,
                 // and since most trees have less than a handful of nodes, it is worthwhile to
diff --git a/lib/bforest/src/set.rs b/lib/bforest/src/set.rs
index 0ca5d92457..e73171d368 100644
--- a/lib/bforest/src/set.rs
+++ b/lib/bforest/src/set.rs
@@ -225,7 +225,7 @@ where
     ///
     /// If the cursor reaches the end, return `None` and leave the cursor at the off-the-end
     /// position.
-    #[cfg_attr(feature = "cargo-clippy", allow(should_implement_trait))]
+    #[cfg_attr(feature = "cargo-clippy", allow(clippy::should_implement_trait))]
     pub fn next(&mut self) -> Option {
         self.path.next(self.pool).map(|(k, _)| k)
     }
diff --git a/lib/codegen/meta/src/constant_hash.rs b/lib/codegen/meta/src/constant_hash.rs
index 84b5513d5a..7f629d0d8b 100644
--- a/lib/codegen/meta/src/constant_hash.rs
+++ b/lib/codegen/meta/src/constant_hash.rs
@@ -18,8 +18,7 @@ pub fn generate_table usize>(items: &Vec, hash_function: H) -
         size.next_power_of_two()
     };
 
-    let mut table: Vec> = Vec::new();
-    table.resize(size, None);
+    let mut table: Vec> = vec![None; size];
 
     for i in items {
         let mut h = hash_function(i) % size;
diff --git a/lib/codegen/meta/src/gen_settings.rs b/lib/codegen/meta/src/gen_settings.rs
index 497c1cc2f1..bdcd206e13 100644
--- a/lib/codegen/meta/src/gen_settings.rs
+++ b/lib/codegen/meta/src/gen_settings.rs
@@ -355,8 +355,7 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
 }
 
 fn gen_template(group: &SettingGroup, fmt: &mut Formatter) {
-    let mut default_bytes: Vec = Vec::new();
-    default_bytes.resize(group.settings_size as usize, 0);
+    let mut default_bytes: Vec = vec![0; group.settings_size as usize];
     for setting in &group.settings {
         *default_bytes.get_mut(setting.byte_offset as usize).unwrap() |= setting.default_byte();
     }
diff --git a/lib/codegen/src/binemit/memorysink.rs b/lib/codegen/src/binemit/memorysink.rs
index 7f6c62f451..3d7d673719 100644
--- a/lib/codegen/src/binemit/memorysink.rs
+++ b/lib/codegen/src/binemit/memorysink.rs
@@ -89,7 +89,7 @@ impl<'a> CodeSink for MemoryCodeSink<'a> {
 
     fn put2(&mut self, x: u16) {
         unsafe {
-            #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))]
+            #[cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
             write_unaligned(self.data.offset(self.offset) as *mut u16, x);
         }
         self.offset += 2;
@@ -97,7 +97,7 @@ impl<'a> CodeSink for MemoryCodeSink<'a> {
 
     fn put4(&mut self, x: u32) {
         unsafe {
-            #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))]
+            #[cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
             write_unaligned(self.data.offset(self.offset) as *mut u32, x);
         }
         self.offset += 4;
@@ -105,7 +105,7 @@ impl<'a> CodeSink for MemoryCodeSink<'a> {
 
     fn put8(&mut self, x: u64) {
         unsafe {
-            #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))]
+            #[cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
             write_unaligned(self.data.offset(self.offset) as *mut u64, x);
         }
         self.offset += 8;
diff --git a/lib/codegen/src/cursor.rs b/lib/codegen/src/cursor.rs
index eae5fc0b1f..ba7bc92b13 100644
--- a/lib/codegen/src/cursor.rs
+++ b/lib/codegen/src/cursor.rs
@@ -751,7 +751,7 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut EncCursor<'f> {
         }
         // Assign an encoding.
         // XXX Is there a way to describe this error to the user?
-        #[cfg_attr(feature = "cargo-clippy", allow(match_wild_err_arm))]
+        #[cfg_attr(feature = "cargo-clippy", allow(clippy::match_wild_err_arm))]
         match self
             .isa
             .encode(&self.func, &self.func.dfg[inst], ctrl_typevar)
diff --git a/lib/codegen/src/dce.rs b/lib/codegen/src/dce.rs
index d42c36870e..0aa9adc15b 100644
--- a/lib/codegen/src/dce.rs
+++ b/lib/codegen/src/dce.rs
@@ -9,7 +9,6 @@ use crate::entity::EntityRef;
 use crate::ir::instructions::InstructionData;
 use crate::ir::{DataFlowGraph, Function, Inst, Opcode};
 use crate::timing;
-use std::vec::Vec;
 
 /// Test whether the given opcode is unsafe to even consider for DCE.
 fn trivially_unsafe_for_dce(opcode: Opcode) -> bool {
@@ -46,9 +45,7 @@ pub fn do_dce(func: &mut Function, domtree: &mut DominatorTree) {
     let _tt = timing::dce();
     debug_assert!(domtree.is_valid());
 
-    let mut live = Vec::with_capacity(func.dfg.num_values());
-    live.resize(func.dfg.num_values(), false);
-
+    let mut live = vec![false; func.dfg.num_values()];
     for &ebb in domtree.cfg_postorder() {
         let mut pos = FuncCursor::new(func).at_bottom(ebb);
         while let Some(inst) = pos.prev_inst() {
diff --git a/lib/codegen/src/divconst_magic_numbers.rs b/lib/codegen/src/divconst_magic_numbers.rs
index 89c686498e..ebf1ae3f12 100644
--- a/lib/codegen/src/divconst_magic_numbers.rs
+++ b/lib/codegen/src/divconst_magic_numbers.rs
@@ -221,7 +221,7 @@ mod tests {
     use super::{magic_s32, magic_s64, magic_u32, magic_u64};
     use super::{MS32, MS64, MU32, MU64};
 
-    fn mkMU32(mul_by: u32, do_add: bool, shift_by: i32) -> MU32 {
+    fn make_mu32(mul_by: u32, do_add: bool, shift_by: i32) -> MU32 {
         MU32 {
             mul_by,
             do_add,
@@ -229,7 +229,7 @@ mod tests {
         }
     }
 
-    fn mkMU64(mul_by: u64, do_add: bool, shift_by: i32) -> MU64 {
+    fn make_mu64(mul_by: u64, do_add: bool, shift_by: i32) -> MU64 {
         MU64 {
             mul_by,
             do_add,
@@ -237,237 +237,282 @@ mod tests {
         }
     }
 
-    fn mkMS32(mul_by: i32, shift_by: i32) -> MS32 {
+    fn make_ms32(mul_by: i32, shift_by: i32) -> MS32 {
         MS32 { mul_by, shift_by }
     }
 
-    fn mkMS64(mul_by: i64, shift_by: i32) -> MS64 {
+    fn make_ms64(mul_by: i64, shift_by: i32) -> MS64 {
         MS64 { mul_by, shift_by }
     }
 
     #[test]
     fn test_magicU32() {
-        assert_eq!(magic_u32(2u32), mkMU32(0x80000000u32, false, 0));
-        assert_eq!(magic_u32(3u32), mkMU32(0xaaaaaaabu32, false, 1));
-        assert_eq!(magic_u32(4u32), mkMU32(0x40000000u32, false, 0));
-        assert_eq!(magic_u32(5u32), mkMU32(0xcccccccdu32, false, 2));
-        assert_eq!(magic_u32(6u32), mkMU32(0xaaaaaaabu32, false, 2));
-        assert_eq!(magic_u32(7u32), mkMU32(0x24924925u32, true, 3));
-        assert_eq!(magic_u32(9u32), mkMU32(0x38e38e39u32, false, 1));
-        assert_eq!(magic_u32(10u32), mkMU32(0xcccccccdu32, false, 3));
-        assert_eq!(magic_u32(11u32), mkMU32(0xba2e8ba3u32, false, 3));
-        assert_eq!(magic_u32(12u32), mkMU32(0xaaaaaaabu32, false, 3));
-        assert_eq!(magic_u32(25u32), mkMU32(0x51eb851fu32, false, 3));
-        assert_eq!(magic_u32(125u32), mkMU32(0x10624dd3u32, false, 3));
-        assert_eq!(magic_u32(625u32), mkMU32(0xd1b71759u32, false, 9));
-        assert_eq!(magic_u32(1337u32), mkMU32(0x88233b2bu32, true, 11));
-        assert_eq!(magic_u32(65535u32), mkMU32(0x80008001u32, false, 15));
-        assert_eq!(magic_u32(65536u32), mkMU32(0x00010000u32, false, 0));
-        assert_eq!(magic_u32(65537u32), mkMU32(0xffff0001u32, false, 16));
-        assert_eq!(magic_u32(31415927u32), mkMU32(0x445b4553u32, false, 23));
-        assert_eq!(magic_u32(0xdeadbeefu32), mkMU32(0x93275ab3u32, false, 31));
-        assert_eq!(magic_u32(0xfffffffdu32), mkMU32(0x40000001u32, false, 30));
-        assert_eq!(magic_u32(0xfffffffeu32), mkMU32(0x00000003u32, true, 32));
-        assert_eq!(magic_u32(0xffffffffu32), mkMU32(0x80000001u32, false, 31));
+        assert_eq!(magic_u32(2u32), make_mu32(0x80000000u32, false, 0));
+        assert_eq!(magic_u32(3u32), make_mu32(0xaaaaaaabu32, false, 1));
+        assert_eq!(magic_u32(4u32), make_mu32(0x40000000u32, false, 0));
+        assert_eq!(magic_u32(5u32), make_mu32(0xcccccccdu32, false, 2));
+        assert_eq!(magic_u32(6u32), make_mu32(0xaaaaaaabu32, false, 2));
+        assert_eq!(magic_u32(7u32), make_mu32(0x24924925u32, true, 3));
+        assert_eq!(magic_u32(9u32), make_mu32(0x38e38e39u32, false, 1));
+        assert_eq!(magic_u32(10u32), make_mu32(0xcccccccdu32, false, 3));
+        assert_eq!(magic_u32(11u32), make_mu32(0xba2e8ba3u32, false, 3));
+        assert_eq!(magic_u32(12u32), make_mu32(0xaaaaaaabu32, false, 3));
+        assert_eq!(magic_u32(25u32), make_mu32(0x51eb851fu32, false, 3));
+        assert_eq!(magic_u32(125u32), make_mu32(0x10624dd3u32, false, 3));
+        assert_eq!(magic_u32(625u32), make_mu32(0xd1b71759u32, false, 9));
+        assert_eq!(magic_u32(1337u32), make_mu32(0x88233b2bu32, true, 11));
+        assert_eq!(magic_u32(65535u32), make_mu32(0x80008001u32, false, 15));
+        assert_eq!(magic_u32(65536u32), make_mu32(0x00010000u32, false, 0));
+        assert_eq!(magic_u32(65537u32), make_mu32(0xffff0001u32, false, 16));
+        assert_eq!(magic_u32(31415927u32), make_mu32(0x445b4553u32, false, 23));
+        assert_eq!(
+            magic_u32(0xdeadbeefu32),
+            make_mu32(0x93275ab3u32, false, 31)
+        );
+        assert_eq!(
+            magic_u32(0xfffffffdu32),
+            make_mu32(0x40000001u32, false, 30)
+        );
+        assert_eq!(magic_u32(0xfffffffeu32), make_mu32(0x00000003u32, true, 32));
+        assert_eq!(
+            magic_u32(0xffffffffu32),
+            make_mu32(0x80000001u32, false, 31)
+        );
     }
     #[test]
     fn test_magicU64() {
-        assert_eq!(magic_u64(2u64), mkMU64(0x8000000000000000u64, false, 0));
-        assert_eq!(magic_u64(3u64), mkMU64(0xaaaaaaaaaaaaaaabu64, false, 1));
-        assert_eq!(magic_u64(4u64), mkMU64(0x4000000000000000u64, false, 0));
-        assert_eq!(magic_u64(5u64), mkMU64(0xcccccccccccccccdu64, false, 2));
-        assert_eq!(magic_u64(6u64), mkMU64(0xaaaaaaaaaaaaaaabu64, false, 2));
-        assert_eq!(magic_u64(7u64), mkMU64(0x2492492492492493u64, true, 3));
-        assert_eq!(magic_u64(9u64), mkMU64(0xe38e38e38e38e38fu64, false, 3));
-        assert_eq!(magic_u64(10u64), mkMU64(0xcccccccccccccccdu64, false, 3));
-        assert_eq!(magic_u64(11u64), mkMU64(0x2e8ba2e8ba2e8ba3u64, false, 1));
-        assert_eq!(magic_u64(12u64), mkMU64(0xaaaaaaaaaaaaaaabu64, false, 3));
-        assert_eq!(magic_u64(25u64), mkMU64(0x47ae147ae147ae15u64, true, 5));
-        assert_eq!(magic_u64(125u64), mkMU64(0x0624dd2f1a9fbe77u64, true, 7));
-        assert_eq!(magic_u64(625u64), mkMU64(0x346dc5d63886594bu64, false, 7));
-        assert_eq!(magic_u64(1337u64), mkMU64(0xc4119d952866a139u64, false, 10));
+        assert_eq!(magic_u64(2u64), make_mu64(0x8000000000000000u64, false, 0));
+        assert_eq!(magic_u64(3u64), make_mu64(0xaaaaaaaaaaaaaaabu64, false, 1));
+        assert_eq!(magic_u64(4u64), make_mu64(0x4000000000000000u64, false, 0));
+        assert_eq!(magic_u64(5u64), make_mu64(0xcccccccccccccccdu64, false, 2));
+        assert_eq!(magic_u64(6u64), make_mu64(0xaaaaaaaaaaaaaaabu64, false, 2));
+        assert_eq!(magic_u64(7u64), make_mu64(0x2492492492492493u64, true, 3));
+        assert_eq!(magic_u64(9u64), make_mu64(0xe38e38e38e38e38fu64, false, 3));
+        assert_eq!(magic_u64(10u64), make_mu64(0xcccccccccccccccdu64, false, 3));
+        assert_eq!(magic_u64(11u64), make_mu64(0x2e8ba2e8ba2e8ba3u64, false, 1));
+        assert_eq!(magic_u64(12u64), make_mu64(0xaaaaaaaaaaaaaaabu64, false, 3));
+        assert_eq!(magic_u64(25u64), make_mu64(0x47ae147ae147ae15u64, true, 5));
+        assert_eq!(magic_u64(125u64), make_mu64(0x0624dd2f1a9fbe77u64, true, 7));
+        assert_eq!(
+            magic_u64(625u64),
+            make_mu64(0x346dc5d63886594bu64, false, 7)
+        );
+        assert_eq!(
+            magic_u64(1337u64),
+            make_mu64(0xc4119d952866a139u64, false, 10)
+        );
         assert_eq!(
             magic_u64(31415927u64),
-            mkMU64(0x116d154b9c3d2f85u64, true, 25)
+            make_mu64(0x116d154b9c3d2f85u64, true, 25)
         );
         assert_eq!(
             magic_u64(0x00000000deadbeefu64),
-            mkMU64(0x93275ab2dfc9094bu64, false, 31)
+            make_mu64(0x93275ab2dfc9094bu64, false, 31)
         );
         assert_eq!(
             magic_u64(0x00000000fffffffdu64),
-            mkMU64(0x8000000180000005u64, false, 31)
+            make_mu64(0x8000000180000005u64, false, 31)
         );
         assert_eq!(
             magic_u64(0x00000000fffffffeu64),
-            mkMU64(0x0000000200000005u64, true, 32)
+            make_mu64(0x0000000200000005u64, true, 32)
         );
         assert_eq!(
             magic_u64(0x00000000ffffffffu64),
-            mkMU64(0x8000000080000001u64, false, 31)
+            make_mu64(0x8000000080000001u64, false, 31)
         );
         assert_eq!(
             magic_u64(0x0000000100000000u64),
-            mkMU64(0x0000000100000000u64, false, 0)
+            make_mu64(0x0000000100000000u64, false, 0)
         );
         assert_eq!(
             magic_u64(0x0000000100000001u64),
-            mkMU64(0xffffffff00000001u64, false, 32)
+            make_mu64(0xffffffff00000001u64, false, 32)
         );
         assert_eq!(
             magic_u64(0x0ddc0ffeebadf00du64),
-            mkMU64(0x2788e9d394b77da1u64, true, 60)
+            make_mu64(0x2788e9d394b77da1u64, true, 60)
         );
         assert_eq!(
             magic_u64(0xfffffffffffffffdu64),
-            mkMU64(0x4000000000000001u64, false, 62)
+            make_mu64(0x4000000000000001u64, false, 62)
         );
         assert_eq!(
             magic_u64(0xfffffffffffffffeu64),
-            mkMU64(0x0000000000000003u64, true, 64)
+            make_mu64(0x0000000000000003u64, true, 64)
         );
         assert_eq!(
             magic_u64(0xffffffffffffffffu64),
-            mkMU64(0x8000000000000001u64, false, 63)
+            make_mu64(0x8000000000000001u64, false, 63)
         );
     }
     #[test]
     fn test_magicS32() {
-        assert_eq!(magic_s32(-0x80000000i32), mkMS32(0x7fffffffu32 as i32, 30));
-        assert_eq!(magic_s32(-0x7FFFFFFFi32), mkMS32(0xbfffffffu32 as i32, 29));
-        assert_eq!(magic_s32(-0x7FFFFFFEi32), mkMS32(0x7ffffffdu32 as i32, 30));
-        assert_eq!(magic_s32(-31415927i32), mkMS32(0xbba4baadu32 as i32, 23));
-        assert_eq!(magic_s32(-1337i32), mkMS32(0x9df73135u32 as i32, 9));
-        assert_eq!(magic_s32(-256i32), mkMS32(0x7fffffffu32 as i32, 7));
-        assert_eq!(magic_s32(-5i32), mkMS32(0x99999999u32 as i32, 1));
-        assert_eq!(magic_s32(-3i32), mkMS32(0x55555555u32 as i32, 1));
-        assert_eq!(magic_s32(-2i32), mkMS32(0x7fffffffu32 as i32, 0));
-        assert_eq!(magic_s32(2i32), mkMS32(0x80000001u32 as i32, 0));
-        assert_eq!(magic_s32(3i32), mkMS32(0x55555556u32 as i32, 0));
-        assert_eq!(magic_s32(4i32), mkMS32(0x80000001u32 as i32, 1));
-        assert_eq!(magic_s32(5i32), mkMS32(0x66666667u32 as i32, 1));
-        assert_eq!(magic_s32(6i32), mkMS32(0x2aaaaaabu32 as i32, 0));
-        assert_eq!(magic_s32(7i32), mkMS32(0x92492493u32 as i32, 2));
-        assert_eq!(magic_s32(9i32), mkMS32(0x38e38e39u32 as i32, 1));
-        assert_eq!(magic_s32(10i32), mkMS32(0x66666667u32 as i32, 2));
-        assert_eq!(magic_s32(11i32), mkMS32(0x2e8ba2e9u32 as i32, 1));
-        assert_eq!(magic_s32(12i32), mkMS32(0x2aaaaaabu32 as i32, 1));
-        assert_eq!(magic_s32(25i32), mkMS32(0x51eb851fu32 as i32, 3));
-        assert_eq!(magic_s32(125i32), mkMS32(0x10624dd3u32 as i32, 3));
-        assert_eq!(magic_s32(625i32), mkMS32(0x68db8badu32 as i32, 8));
-        assert_eq!(magic_s32(1337i32), mkMS32(0x6208cecbu32 as i32, 9));
-        assert_eq!(magic_s32(31415927i32), mkMS32(0x445b4553u32 as i32, 23));
-        assert_eq!(magic_s32(0x7ffffffei32), mkMS32(0x80000003u32 as i32, 30));
-        assert_eq!(magic_s32(0x7fffffffi32), mkMS32(0x40000001u32 as i32, 29));
+        assert_eq!(
+            magic_s32(-0x80000000i32),
+            make_ms32(0x7fffffffu32 as i32, 30)
+        );
+        assert_eq!(
+            magic_s32(-0x7FFFFFFFi32),
+            make_ms32(0xbfffffffu32 as i32, 29)
+        );
+        assert_eq!(
+            magic_s32(-0x7FFFFFFEi32),
+            make_ms32(0x7ffffffdu32 as i32, 30)
+        );
+        assert_eq!(magic_s32(-31415927i32), make_ms32(0xbba4baadu32 as i32, 23));
+        assert_eq!(magic_s32(-1337i32), make_ms32(0x9df73135u32 as i32, 9));
+        assert_eq!(magic_s32(-256i32), make_ms32(0x7fffffffu32 as i32, 7));
+        assert_eq!(magic_s32(-5i32), make_ms32(0x99999999u32 as i32, 1));
+        assert_eq!(magic_s32(-3i32), make_ms32(0x55555555u32 as i32, 1));
+        assert_eq!(magic_s32(-2i32), make_ms32(0x7fffffffu32 as i32, 0));
+        assert_eq!(magic_s32(2i32), make_ms32(0x80000001u32 as i32, 0));
+        assert_eq!(magic_s32(3i32), make_ms32(0x55555556u32 as i32, 0));
+        assert_eq!(magic_s32(4i32), make_ms32(0x80000001u32 as i32, 1));
+        assert_eq!(magic_s32(5i32), make_ms32(0x66666667u32 as i32, 1));
+        assert_eq!(magic_s32(6i32), make_ms32(0x2aaaaaabu32 as i32, 0));
+        assert_eq!(magic_s32(7i32), make_ms32(0x92492493u32 as i32, 2));
+        assert_eq!(magic_s32(9i32), make_ms32(0x38e38e39u32 as i32, 1));
+        assert_eq!(magic_s32(10i32), make_ms32(0x66666667u32 as i32, 2));
+        assert_eq!(magic_s32(11i32), make_ms32(0x2e8ba2e9u32 as i32, 1));
+        assert_eq!(magic_s32(12i32), make_ms32(0x2aaaaaabu32 as i32, 1));
+        assert_eq!(magic_s32(25i32), make_ms32(0x51eb851fu32 as i32, 3));
+        assert_eq!(magic_s32(125i32), make_ms32(0x10624dd3u32 as i32, 3));
+        assert_eq!(magic_s32(625i32), make_ms32(0x68db8badu32 as i32, 8));
+        assert_eq!(magic_s32(1337i32), make_ms32(0x6208cecbu32 as i32, 9));
+        assert_eq!(magic_s32(31415927i32), make_ms32(0x445b4553u32 as i32, 23));
+        assert_eq!(
+            magic_s32(0x7ffffffei32),
+            make_ms32(0x80000003u32 as i32, 30)
+        );
+        assert_eq!(
+            magic_s32(0x7fffffffi32),
+            make_ms32(0x40000001u32 as i32, 29)
+        );
     }
     #[test]
     fn test_magicS64() {
         assert_eq!(
             magic_s64(-0x8000000000000000i64),
-            mkMS64(0x7fffffffffffffffu64 as i64, 62)
+            make_ms64(0x7fffffffffffffffu64 as i64, 62)
         );
         assert_eq!(
             magic_s64(-0x7FFFFFFFFFFFFFFFi64),
-            mkMS64(0xbfffffffffffffffu64 as i64, 61)
+            make_ms64(0xbfffffffffffffffu64 as i64, 61)
         );
         assert_eq!(
             magic_s64(-0x7FFFFFFFFFFFFFFEi64),
-            mkMS64(0x7ffffffffffffffdu64 as i64, 62)
+            make_ms64(0x7ffffffffffffffdu64 as i64, 62)
         );
         assert_eq!(
             magic_s64(-0x0ddC0ffeeBadF00di64),
-            mkMS64(0x6c3b8b1635a4412fu64 as i64, 59)
+            make_ms64(0x6c3b8b1635a4412fu64 as i64, 59)
         );
         assert_eq!(
             magic_s64(-0x100000001i64),
-            mkMS64(0x800000007fffffffu64 as i64, 31)
+            make_ms64(0x800000007fffffffu64 as i64, 31)
         );
         assert_eq!(
             magic_s64(-0x100000000i64),
-            mkMS64(0x7fffffffffffffffu64 as i64, 31)
+            make_ms64(0x7fffffffffffffffu64 as i64, 31)
         );
         assert_eq!(
             magic_s64(-0xFFFFFFFFi64),
-            mkMS64(0x7fffffff7fffffffu64 as i64, 31)
+            make_ms64(0x7fffffff7fffffffu64 as i64, 31)
         );
         assert_eq!(
             magic_s64(-0xFFFFFFFEi64),
-            mkMS64(0x7ffffffefffffffdu64 as i64, 31)
+            make_ms64(0x7ffffffefffffffdu64 as i64, 31)
         );
         assert_eq!(
             magic_s64(-0xFFFFFFFDi64),
-            mkMS64(0x7ffffffe7ffffffbu64 as i64, 31)
+            make_ms64(0x7ffffffe7ffffffbu64 as i64, 31)
         );
         assert_eq!(
             magic_s64(-0xDeadBeefi64),
-            mkMS64(0x6cd8a54d2036f6b5u64 as i64, 31)
+            make_ms64(0x6cd8a54d2036f6b5u64 as i64, 31)
         );
         assert_eq!(
             magic_s64(-31415927i64),
-            mkMS64(0x7749755a31e1683du64 as i64, 24)
+            make_ms64(0x7749755a31e1683du64 as i64, 24)
+        );
+        assert_eq!(
+            magic_s64(-1337i64),
+            make_ms64(0x9df731356bccaf63u64 as i64, 9)
+        );
+        assert_eq!(
+            magic_s64(-256i64),
+            make_ms64(0x7fffffffffffffffu64 as i64, 7)
+        );
+        assert_eq!(magic_s64(-5i64), make_ms64(0x9999999999999999u64 as i64, 1));
+        assert_eq!(magic_s64(-3i64), make_ms64(0x5555555555555555u64 as i64, 1));
+        assert_eq!(magic_s64(-2i64), make_ms64(0x7fffffffffffffffu64 as i64, 0));
+        assert_eq!(magic_s64(2i64), make_ms64(0x8000000000000001u64 as i64, 0));
+        assert_eq!(magic_s64(3i64), make_ms64(0x5555555555555556u64 as i64, 0));
+        assert_eq!(magic_s64(4i64), make_ms64(0x8000000000000001u64 as i64, 1));
+        assert_eq!(magic_s64(5i64), make_ms64(0x6666666666666667u64 as i64, 1));
+        assert_eq!(magic_s64(6i64), make_ms64(0x2aaaaaaaaaaaaaabu64 as i64, 0));
+        assert_eq!(magic_s64(7i64), make_ms64(0x4924924924924925u64 as i64, 1));
+        assert_eq!(magic_s64(9i64), make_ms64(0x1c71c71c71c71c72u64 as i64, 0));
+        assert_eq!(magic_s64(10i64), make_ms64(0x6666666666666667u64 as i64, 2));
+        assert_eq!(magic_s64(11i64), make_ms64(0x2e8ba2e8ba2e8ba3u64 as i64, 1));
+        assert_eq!(magic_s64(12i64), make_ms64(0x2aaaaaaaaaaaaaabu64 as i64, 1));
+        assert_eq!(magic_s64(25i64), make_ms64(0xa3d70a3d70a3d70bu64 as i64, 4));
+        assert_eq!(
+            magic_s64(125i64),
+            make_ms64(0x20c49ba5e353f7cfu64 as i64, 4)
+        );
+        assert_eq!(
+            magic_s64(625i64),
+            make_ms64(0x346dc5d63886594bu64 as i64, 7)
+        );
+        assert_eq!(
+            magic_s64(1337i64),
+            make_ms64(0x6208ceca9433509du64 as i64, 9)
         );
-        assert_eq!(magic_s64(-1337i64), mkMS64(0x9df731356bccaf63u64 as i64, 9));
-        assert_eq!(magic_s64(-256i64), mkMS64(0x7fffffffffffffffu64 as i64, 7));
-        assert_eq!(magic_s64(-5i64), mkMS64(0x9999999999999999u64 as i64, 1));
-        assert_eq!(magic_s64(-3i64), mkMS64(0x5555555555555555u64 as i64, 1));
-        assert_eq!(magic_s64(-2i64), mkMS64(0x7fffffffffffffffu64 as i64, 0));
-        assert_eq!(magic_s64(2i64), mkMS64(0x8000000000000001u64 as i64, 0));
-        assert_eq!(magic_s64(3i64), mkMS64(0x5555555555555556u64 as i64, 0));
-        assert_eq!(magic_s64(4i64), mkMS64(0x8000000000000001u64 as i64, 1));
-        assert_eq!(magic_s64(5i64), mkMS64(0x6666666666666667u64 as i64, 1));
-        assert_eq!(magic_s64(6i64), mkMS64(0x2aaaaaaaaaaaaaabu64 as i64, 0));
-        assert_eq!(magic_s64(7i64), mkMS64(0x4924924924924925u64 as i64, 1));
-        assert_eq!(magic_s64(9i64), mkMS64(0x1c71c71c71c71c72u64 as i64, 0));
-        assert_eq!(magic_s64(10i64), mkMS64(0x6666666666666667u64 as i64, 2));
-        assert_eq!(magic_s64(11i64), mkMS64(0x2e8ba2e8ba2e8ba3u64 as i64, 1));
-        assert_eq!(magic_s64(12i64), mkMS64(0x2aaaaaaaaaaaaaabu64 as i64, 1));
-        assert_eq!(magic_s64(25i64), mkMS64(0xa3d70a3d70a3d70bu64 as i64, 4));
-        assert_eq!(magic_s64(125i64), mkMS64(0x20c49ba5e353f7cfu64 as i64, 4));
-        assert_eq!(magic_s64(625i64), mkMS64(0x346dc5d63886594bu64 as i64, 7));
-        assert_eq!(magic_s64(1337i64), mkMS64(0x6208ceca9433509du64 as i64, 9));
         assert_eq!(
             magic_s64(31415927i64),
-            mkMS64(0x88b68aa5ce1e97c3u64 as i64, 24)
+            make_ms64(0x88b68aa5ce1e97c3u64 as i64, 24)
         );
         assert_eq!(
             magic_s64(0x00000000deadbeefi64),
-            mkMS64(0x93275ab2dfc9094bu64 as i64, 31)
+            make_ms64(0x93275ab2dfc9094bu64 as i64, 31)
         );
         assert_eq!(
             magic_s64(0x00000000fffffffdi64),
-            mkMS64(0x8000000180000005u64 as i64, 31)
+            make_ms64(0x8000000180000005u64 as i64, 31)
         );
         assert_eq!(
             magic_s64(0x00000000fffffffei64),
-            mkMS64(0x8000000100000003u64 as i64, 31)
+            make_ms64(0x8000000100000003u64 as i64, 31)
         );
         assert_eq!(
             magic_s64(0x00000000ffffffffi64),
-            mkMS64(0x8000000080000001u64 as i64, 31)
+            make_ms64(0x8000000080000001u64 as i64, 31)
         );
         assert_eq!(
             magic_s64(0x0000000100000000i64),
-            mkMS64(0x8000000000000001u64 as i64, 31)
+            make_ms64(0x8000000000000001u64 as i64, 31)
         );
         assert_eq!(
             magic_s64(0x0000000100000001i64),
-            mkMS64(0x7fffffff80000001u64 as i64, 31)
+            make_ms64(0x7fffffff80000001u64 as i64, 31)
         );
         assert_eq!(
             magic_s64(0x0ddc0ffeebadf00di64),
-            mkMS64(0x93c474e9ca5bbed1u64 as i64, 59)
+            make_ms64(0x93c474e9ca5bbed1u64 as i64, 59)
         );
         assert_eq!(
             magic_s64(0x7ffffffffffffffdi64),
-            mkMS64(0x2000000000000001u64 as i64, 60)
+            make_ms64(0x2000000000000001u64 as i64, 60)
         );
         assert_eq!(
             magic_s64(0x7ffffffffffffffei64),
-            mkMS64(0x8000000000000003u64 as i64, 62)
+            make_ms64(0x8000000000000003u64 as i64, 62)
         );
         assert_eq!(
             magic_s64(0x7fffffffffffffffi64),
-            mkMS64(0x4000000000000001u64 as i64, 61)
+            make_ms64(0x4000000000000001u64 as i64, 61)
         );
     }
     #[test]
diff --git a/lib/codegen/src/ir/globalvalue.rs b/lib/codegen/src/ir/globalvalue.rs
index aedfa1dfc8..6c7d8032f6 100644
--- a/lib/codegen/src/ir/globalvalue.rs
+++ b/lib/codegen/src/ir/globalvalue.rs
@@ -27,7 +27,8 @@ pub enum GlobalValueData {
         /// Type of the loaded value.
         global_type: Type,
 
-        /// Specifies whether the memory that this refers to is readonly, allowing for the elimination of redundant loads.
+        /// Specifies whether the memory that this refers to is readonly, allowing for the
+        /// elimination of redundant loads.
         readonly: bool,
     },
 
diff --git a/lib/codegen/src/ir/instructions.rs b/lib/codegen/src/ir/instructions.rs
index f0ac7c24c1..4f9cf5125e 100644
--- a/lib/codegen/src/ir/instructions.rs
+++ b/lib/codegen/src/ir/instructions.rs
@@ -335,8 +335,8 @@ pub struct OpcodeConstraints {
     typeset_offset: u8,
 
     /// Offset into `OPERAND_CONSTRAINT` table of the descriptors for this opcode. The first
-    /// `num_fixed_results()` entries describe the result constraints, then follows constraints for the
-    /// fixed `Value` input operands. (`num_fixed_value_arguments()` of them).
+    /// `num_fixed_results()` entries describe the result constraints, then follows constraints for
+    /// the fixed `Value` input operands. (`num_fixed_value_arguments()` of them).
     constraint_offset: u16,
 }
 
diff --git a/lib/codegen/src/legalizer/heap.rs b/lib/codegen/src/legalizer/heap.rs
index 41d9778421..8ffd02ec5f 100644
--- a/lib/codegen/src/legalizer/heap.rs
+++ b/lib/codegen/src/legalizer/heap.rs
@@ -117,7 +117,7 @@ fn static_addr(
     }
 
     // Check `offset > limit` which is now known non-negative.
-    let limit = bound - u64::from(access_size);
+    let limit = bound - access_size;
 
     // We may be able to omit the check entirely for 32-bit offsets if the heap bound is 4 GB or
     // more.
diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index 737aca4140..eb61f62f09 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -6,27 +6,27 @@
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature="cargo-clippy", allow(
 // Produces only a false positive:
-                while_let_loop,
+                clippy::while_let_loop,
 // Produces many false positives, but did produce some valid lints, now fixed:
-                needless_lifetimes,
+                clippy::needless_lifetimes,
 // Generated code makes some style transgressions, but readability doesn't suffer much:
-                many_single_char_names,
-                identity_op,
-                needless_borrow,
-                cast_lossless,
-                unreadable_literal,
-                assign_op_pattern,
-                empty_line_after_outer_attr,
+                clippy::many_single_char_names,
+                clippy::identity_op,
+                clippy::needless_borrow,
+                clippy::cast_lossless,
+                clippy::unreadable_literal,
+                clippy::assign_op_pattern,
+                clippy::empty_line_after_outer_attr,
 // Hard to avoid in generated code:
-                cyclomatic_complexity,
-                too_many_arguments,
+                clippy::cyclomatic_complexity,
+                clippy::too_many_arguments,
 // Code generator doesn't have a way to collapse identical arms:
-                match_same_arms,
+                clippy::match_same_arms,
 // These are relatively minor style issues, but would be easy to fix:
-                new_without_default,
-                new_without_default_derive,
-                should_implement_trait,
-                len_without_is_empty))]
+                clippy::new_without_default,
+                clippy::new_without_default_derive,
+                clippy::should_implement_trait,
+                clippy::len_without_is_empty))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/codegen/src/licm.rs b/lib/codegen/src/licm.rs
index 101eb47d36..cb3dbd87a9 100644
--- a/lib/codegen/src/licm.rs
+++ b/lib/codegen/src/licm.rs
@@ -191,7 +191,7 @@ fn remove_loop_invariant_instructions(
             loop_values.insert(*val);
         }
         pos.goto_top(*ebb);
-        #[cfg_attr(feature = "cargo-clippy", allow(block_in_if_condition_stmt))]
+        #[cfg_attr(feature = "cargo-clippy", allow(clippy::block_in_if_condition_stmt))]
         while let Some(inst) = pos.next_inst() {
             if is_loop_invariant(inst, &pos.func.dfg, &loop_values) {
                 // If all the instruction's argument are defined outside the loop
diff --git a/lib/codegen/src/regalloc/solver.rs b/lib/codegen/src/regalloc/solver.rs
index 0b433a4587..ceb6cdfb4b 100644
--- a/lib/codegen/src/regalloc/solver.rs
+++ b/lib/codegen/src/regalloc/solver.rs
@@ -295,7 +295,7 @@ impl Move {
     }
 
     /// Get the "from" register and register class, if possible.
-    #[cfg_attr(feature = "cargo-clippy", allow(wrong_self_convention))]
+    #[cfg_attr(feature = "cargo-clippy", allow(clippy::wrong_self_convention))]
     fn from_reg(&self) -> Option<(RegClass, RegUnit)> {
         match *self {
             Move::Reg { rc, from, .. } | Move::Spill { rc, from, .. } => Some((rc, from)),
diff --git a/lib/codegen/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs
index deb3b5f20b..3fe55474de 100644
--- a/lib/codegen/src/regalloc/virtregs.rs
+++ b/lib/codegen/src/regalloc/virtregs.rs
@@ -97,7 +97,7 @@ impl VirtRegs {
     ///
     /// If `value` belongs to a virtual register, the congruence class is the values of the virtual
     /// register. Otherwise it is just the value itself.
-    #[cfg_attr(feature = "cargo-clippy", allow(trivially_copy_pass_by_ref))]
+    #[cfg_attr(feature = "cargo-clippy", allow(clippy::trivially_copy_pass_by_ref))]
     pub fn congruence_class<'a, 'b>(&'a self, value: &'b Value) -> &'b [Value]
     where
         'a: 'b,
diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs
index cde226690b..ce34c1e50f 100644
--- a/lib/entity/src/lib.rs
+++ b/lib/entity/src/lib.rs
@@ -15,9 +15,9 @@
 //!
 //! - [`PrimaryMap`](struct.PrimaryMap.html) is used to keep track of a vector of entities,
 //!   assigning a unique entity reference to each.
-//! - [`SecondaryMap`](struct.SecondaryMap.html) is used to associate secondary information to an entity.
-//!   The map is implemented as a simple vector, so it does not keep track of which entities have
-//!   been inserted. Instead, any unknown entities map to the default value.
+//! - [`SecondaryMap`](struct.SecondaryMap.html) is used to associate secondary information to an
+//!   entity. The map is implemented as a simple vector, so it does not keep track of which
+//!   entities have been inserted. Instead, any unknown entities map to the default value.
 //! - [`SparseMap`](struct.SparseMap.html) is used to associate secondary information to a small
 //!   number of entities. It tracks accurately which entities have been inserted. This is a
 //!   specialized data structure which can use a lot of memory, so read the documentation before
@@ -35,7 +35,7 @@
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
-    allow(new_without_default, new_without_default_derive)
+    allow(clippy::new_without_default, clippy::new_without_default_derive)
 )]
 #![cfg_attr(
     feature = "cargo-clippy",
diff --git a/lib/entity/src/map.rs b/lib/entity/src/map.rs
index 47e40d0a74..e0f331106b 100644
--- a/lib/entity/src/map.rs
+++ b/lib/entity/src/map.rs
@@ -11,8 +11,8 @@ use std::vec::Vec;
 /// A mapping `K -> V` for densely indexed entity references.
 ///
 /// The `SecondaryMap` data structure uses the dense index space to implement a map with a vector.
-/// Unlike `PrimaryMap`, an `SecondaryMap` can't be used to allocate entity references. It is used to
-/// associate secondary information with entities.
+/// Unlike `PrimaryMap`, an `SecondaryMap` can't be used to allocate entity references. It is used
+/// to associate secondary information with entities.
 ///
 /// The map does not track if an entry for a key has been inserted or not. Instead it behaves as if
 /// all keys have a default entry from the beginning.
diff --git a/lib/entity/src/sparse.rs b/lib/entity/src/sparse.rs
index a07dc589b1..a6a43db5e0 100644
--- a/lib/entity/src/sparse.rs
+++ b/lib/entity/src/sparse.rs
@@ -37,19 +37,20 @@ pub trait SparseMapValue {
 ///
 /// # Compared to `SecondaryMap`
 ///
-/// When should we use a `SparseMap` instead of a secondary `SecondaryMap`? First of all, `SparseMap`
-/// does not provide the functionality of a `PrimaryMap` which can allocate and assign entity
-/// references to objects as they are pushed onto the map. It is only the secondary entity maps
-/// that can be replaced with a `SparseMap`.
+/// When should we use a `SparseMap` instead of a secondary `SecondaryMap`? First of all,
+/// `SparseMap` does not provide the functionality of a `PrimaryMap` which can allocate and assign
+/// entity references to objects as they are pushed onto the map. It is only the secondary entity
+/// maps that can be replaced with a `SparseMap`.
 ///
 /// - A secondary entity map assigns a default mapping to all keys. It doesn't distinguish between
 ///   an unmapped key and one that maps to the default value. `SparseMap` does not require
 ///   `Default` values, and it tracks accurately if a key has been mapped or not.
-/// - Iterating over the contents of an `SecondaryMap` is linear in the size of the *key space*, while
-///   iterating over a `SparseMap` is linear in the number of elements in the mapping. This is an
-///   advantage precisely when the mapping is sparse.
-/// - `SparseMap::clear()` is constant time and super-fast. `SecondaryMap::clear()` is linear in the
-///   size of the key space. (Or, rather the required `resize()` call following the `clear()` is).
+/// - Iterating over the contents of an `SecondaryMap` is linear in the size of the *key space*,
+///   while iterating over a `SparseMap` is linear in the number of elements in the mapping. This
+///   is an advantage precisely when the mapping is sparse.
+/// - `SparseMap::clear()` is constant time and super-fast. `SecondaryMap::clear()` is linear in
+///   the size of the key space. (Or, rather the required `resize()` call following the `clear()`
+///   is).
 /// - `SparseMap` requires the values to implement `SparseMapValue` which means that they must
 ///   contain their own key.
 pub struct SparseMap
diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index db1b13d596..699311441d 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -152,8 +152,7 @@ impl Backend for FaerieBackend {
         namespace: &ModuleNamespace,
         code_size: u32,
     ) -> ModuleResult {
-        let mut code: Vec = Vec::with_capacity(code_size as usize);
-        code.resize(code_size as usize, 0);
+        let mut code: Vec = vec![0; code_size as usize];
 
         // Non-lexical lifetimes would obviate the braces here.
         {
diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs
index 1f7167a8ce..096faf1148 100644
--- a/lib/faerie/src/lib.rs
+++ b/lib/faerie/src/lib.rs
@@ -12,7 +12,7 @@
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
-    allow(new_without_default, new_without_default_derive)
+    allow(clippy::new_without_default, clippy::new_without_default_derive)
 )]
 #![cfg_attr(
     feature = "cargo-clippy",
diff --git a/lib/filetests/src/lib.rs b/lib/filetests/src/lib.rs
index 084620e3a0..6284e945fa 100644
--- a/lib/filetests/src/lib.rs
+++ b/lib/filetests/src/lib.rs
@@ -10,7 +10,7 @@
     unstable_features
 )]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "cargo-clippy", allow(type_complexity))]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::type_complexity))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 1d454661c9..8867f9ad69 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -546,7 +546,7 @@ impl<'a> FunctionBuilder<'a> {
     ///
     /// Useful for debug purposes. Use it with `None` for standard printing.
     // Clippy thinks the lifetime that follows is needless, but rustc needs it
-    #[cfg_attr(feature = "cargo-clippy", allow(needless_lifetimes))]
+    #[cfg_attr(feature = "cargo-clippy", allow(clippy::needless_lifetimes))]
     pub fn display<'b, I: Into>>(&'b self, isa: I) -> DisplayFunction {
         self.func.display(isa)
     }
@@ -608,7 +608,7 @@ impl<'a> FunctionBuilder<'a> {
             "`size` is not a power of two"
         );
         assert!(
-            access_size >= ::std::cmp::min(src_align, dest_align) as u64,
+            access_size >= u64::from(::std::cmp::min(src_align, dest_align)),
             "`size` is smaller than `dest` and `src`'s alignment value."
         );
 
@@ -689,7 +689,7 @@ impl<'a> FunctionBuilder<'a> {
             "`size` is not a power of two"
         );
         assert!(
-            access_size >= buffer_align as u64,
+            access_size >= u64::from(buffer_align),
             "`size` is smaller than `dest` and `src`'s alignment value."
         );
 
@@ -702,14 +702,14 @@ impl<'a> FunctionBuilder<'a> {
         let load_and_store_amount = size / access_size;
 
         if load_and_store_amount > THRESHOLD {
-            let ch = self.ins().iconst(types::I32, ch as i64);
+            let ch = self.ins().iconst(types::I32, i64::from(ch));
             let size = self.ins().iconst(config.pointer_type(), size as i64);
             self.call_memset(config, buffer, ch, size);
         } else {
             let mut flags = MemFlags::new();
             flags.set_aligned();
 
-            let ch = ch as u64;
+            let ch = u64::from(ch);
             let raw_value = if int_type == types::I64 {
                 (ch << 32) | (ch << 16) | (ch << 8) | ch
             } else if int_type == types::I32 {
@@ -777,7 +777,7 @@ impl<'a> FunctionBuilder<'a> {
             "`size` is not a power of two"
         );
         assert!(
-            access_size >= ::std::cmp::min(src_align, dest_align) as u64,
+            access_size >= u64::from(::std::cmp::min(src_align, dest_align)),
             "`size` is smaller than `dest` and `src`'s alignment value."
         );
         let load_and_store_amount = size / access_size;
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index 3f8ec616cc..0eb2b47692 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -160,7 +160,7 @@
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", deny(unstable_features))]
-#![cfg_attr(feature = "cargo-clippy", allow(new_without_default))]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs
index bd6ca28c8a..a62a6d2e65 100644
--- a/lib/module/src/lib.rs
+++ b/lib/module/src/lib.rs
@@ -4,10 +4,7 @@
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", deny(unstable_features))]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(
-    feature = "cargo-clippy",
-    allow(new_without_default, new_without_default_derive)
-)]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs
index 24650ca28c..a780405288 100644
--- a/lib/native/src/lib.rs
+++ b/lib/native/src/lib.rs
@@ -9,10 +9,7 @@
 )]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(
-    feature = "cargo-clippy",
-    allow(new_without_default, new_without_default_derive)
-)]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/preopt/src/lib.rs b/lib/preopt/src/lib.rs
index 321b7a58b6..b93e914cb2 100644
--- a/lib/preopt/src/lib.rs
+++ b/lib/preopt/src/lib.rs
@@ -4,10 +4,7 @@
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", deny(unstable_features))]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(
-    feature = "cargo-clippy",
-    allow(new_without_default, new_without_default_derive)
-)]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/reader/src/lib.rs b/lib/reader/src/lib.rs
index d2e6a96e8b..803c17c62d 100644
--- a/lib/reader/src/lib.rs
+++ b/lib/reader/src/lib.rs
@@ -11,10 +11,7 @@
 )]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(
-    feature = "cargo-clippy",
-    allow(new_without_default, new_without_default_derive)
-)]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 7503805bb6..3aefb575f3 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -51,7 +51,8 @@ pub fn parse_test<'a>(
     let isa_spec: isaspec::IsaSpec;
     let commands: Vec>;
 
-    // Check for specified passes and target, if present throw out test commands/targets specified in file.
+    // Check for specified passes and target, if present throw out test commands/targets specified
+    // in file.
     match passes {
         Some(pass_vec) => {
             parser.parse_test_commands();
@@ -338,7 +339,7 @@ impl<'a> Parser<'a> {
         // clippy says self.lookahead is immutable so this loop is either infinite or never
         // running. I don't think this is true - self.lookahead is mutated in the loop body - so
         // maybe this is a clippy bug? Either way, disable clippy for this.
-        #[cfg_attr(feature = "cargo-clippy", allow(while_immutable_condition))]
+        #[cfg_attr(feature = "cargo-clippy", allow(clippy::while_immutable_condition))]
         while self.lookahead == None {
             match self.lex.next() {
                 Some(Ok(LocatedToken { token, location })) => {
diff --git a/lib/serde/src/clif-json.rs b/lib/serde/src/clif-json.rs
index 9253d08d1c..34f97cc971 100644
--- a/lib/serde/src/clif-json.rs
+++ b/lib/serde/src/clif-json.rs
@@ -8,10 +8,7 @@
 )]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(
-    feature = "cargo-clippy",
-    allow(new_without_default, new_without_default_derive)
-)]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/simplejit/src/backend.rs b/lib/simplejit/src/backend.rs
index b7582c9c7b..8dc3ee09d0 100644
--- a/lib/simplejit/src/backend.rs
+++ b/lib/simplejit/src/backend.rs
@@ -326,13 +326,13 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
             match reloc {
                 Reloc::Abs4 => {
                     // TODO: Handle overflow.
-                    #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))]
+                    #[cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
                     unsafe {
                         write_unaligned(at as *mut u32, what as u32)
                     };
                 }
                 Reloc::Abs8 => {
-                    #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))]
+                    #[cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
                     unsafe {
                         write_unaligned(at as *mut u64, what as u64)
                     };
@@ -340,7 +340,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
                 Reloc::X86PCRel4 | Reloc::X86CallPCRel4 => {
                     // TODO: Handle overflow.
                     let pcrel = ((what as isize) - (at as isize)) as i32;
-                    #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))]
+                    #[cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
                     unsafe {
                         write_unaligned(at as *mut i32, pcrel)
                     };
@@ -391,13 +391,13 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
             match reloc {
                 Reloc::Abs4 => {
                     // TODO: Handle overflow.
-                    #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))]
+                    #[cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
                     unsafe {
                         write_unaligned(at as *mut u32, what as u32)
                     };
                 }
                 Reloc::Abs8 => {
-                    #[cfg_attr(feature = "cargo-clippy", allow(cast_ptr_alignment))]
+                    #[cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
                     unsafe {
                         write_unaligned(at as *mut u64, what as u64)
                     };
diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs
index eabeb7ac0f..57fc809b08 100644
--- a/lib/simplejit/src/lib.rs
+++ b/lib/simplejit/src/lib.rs
@@ -8,10 +8,7 @@
 )]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(
-    feature = "cargo-clippy",
-    allow(new_without_default, new_without_default_derive)
-)]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs
index c14134e4f4..90927323a5 100644
--- a/lib/umbrella/src/lib.rs
+++ b/lib/umbrella/src/lib.rs
@@ -8,10 +8,7 @@
 )]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(
-    feature = "cargo-clippy",
-    allow(new_without_default, new_without_default_derive)
-)]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index a7b61fc580..f956efab31 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -37,7 +37,7 @@ use std::{i32, u32};
 use wasmparser::{MemoryImmediate, Operator};
 
 // Clippy warns about "flags: _" but its important to document that the flags field is ignored
-#[cfg_attr(feature = "cargo-clippy", allow(unneeded_field_pattern))]
+#[cfg_attr(feature = "cargo-clippy", allow(clippy::unneeded_field_pattern))]
 /// Translates wasm operators into Cranelift IR instructions. Returns `true` if it inserted
 /// a return.
 pub fn translate_operator(
@@ -894,7 +894,7 @@ pub fn translate_operator(
 }
 
 // Clippy warns us of some fields we are deliberately ignoring
-#[cfg_attr(feature = "cargo-clippy", allow(unneeded_field_pattern))]
+#[cfg_attr(feature = "cargo-clippy", allow(clippy::unneeded_field_pattern))]
 /// Deals with a Wasm instruction located in an unreachable portion of the code. Most of them
 /// are dropped but special ones like `End` or `Else` signal the potential end of the unreachable
 /// portion so the translation state must be updated accordingly.
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index 081244ca49..b52b5bfc1d 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -187,7 +187,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         let vmctx = func.create_global_value(ir::GlobalValueData::VMContext {});
         GlobalVariable::Memory {
             gv: vmctx,
-            offset: offset,
+            offset,
             ty: self.mod_info.globals[index].entity.ty,
         }
     }
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index 27249c1234..f79152f5be 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -164,7 +164,7 @@ pub trait FuncEnvironment {
     /// The signature `sig_ref` was previously created by `make_indirect_sig()`.
     ///
     /// Return the call instruction whose results are the WebAssembly return values.
-    #[cfg_attr(feature = "cargo-clippy", allow(too_many_arguments))]
+    #[cfg_attr(feature = "cargo-clippy", allow(clippy::too_many_arguments))]
     fn translate_call_indirect(
         &mut self,
         pos: FuncCursor,
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index 2b75ce9e4c..854fb65dce 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -13,10 +13,7 @@
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "std", deny(unstable_features))]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(
-    feature = "cargo-clippy",
-    allow(new_without_default, new_without_default_derive)
-)]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(

From a5bd3924e3c4e6e1f35824fa6aa421d5b76a4fb8 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 7 Jan 2019 11:51:47 -0800
Subject: [PATCH 2295/3084] Add crate version identifiers.

---
 lib/codegen/src/lib.rs   | 6 +++---
 lib/faerie/src/lib.rs    | 3 +++
 lib/frontend/src/lib.rs  | 3 +++
 lib/module/src/lib.rs    | 3 +++
 lib/native/src/lib.rs    | 3 +++
 lib/simplejit/src/lib.rs | 3 +++
 lib/umbrella/src/lib.rs  | 3 +++
 lib/wasm/src/lib.rs      | 3 +++
 8 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index eb61f62f09..f73167d7fc 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -55,9 +55,6 @@ pub use crate::legalizer::legalize_function;
 pub use crate::verifier::verify_function;
 pub use crate::write::write_function;
 
-/// Version number of the cranelift-codegen crate.
-pub const VERSION: &str = env!("CARGO_PKG_VERSION");
-
 pub use cranelift_bforest as bforest;
 pub use cranelift_entity as entity;
 
@@ -118,3 +115,6 @@ mod std {
         pub use alloc::collections::BTreeSet;
     }
 }
+
+/// Version number of this crate.
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/lib/faerie/src/lib.rs b/lib/faerie/src/lib.rs
index 096faf1148..2bf2f2b599 100644
--- a/lib/faerie/src/lib.rs
+++ b/lib/faerie/src/lib.rs
@@ -34,3 +34,6 @@ pub mod traps;
 
 pub use crate::backend::{FaerieBackend, FaerieBuilder, FaerieProduct, FaerieTrapCollection};
 pub use crate::container::Format;
+
+/// Version number of this crate.
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index 0eb2b47692..90c5b5d908 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -203,3 +203,6 @@ mod std {
         pub use self::hashmap_core::{HashMap, HashSet};
     }
 }
+
+/// Version number of this crate.
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs
index a62a6d2e65..db9b793369 100644
--- a/lib/module/src/lib.rs
+++ b/lib/module/src/lib.rs
@@ -49,3 +49,6 @@ mod std {
         pub use self::hashmap_core::{HashMap, HashSet};
     }
 }
+
+/// Version number of this crate.
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs
index a780405288..4322512ec3 100644
--- a/lib/native/src/lib.rs
+++ b/lib/native/src/lib.rs
@@ -116,3 +116,6 @@ mod tests {
         }
     }
 }
+
+/// Version number of this crate.
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/lib/simplejit/src/lib.rs b/lib/simplejit/src/lib.rs
index 57fc809b08..d907964cff 100644
--- a/lib/simplejit/src/lib.rs
+++ b/lib/simplejit/src/lib.rs
@@ -27,3 +27,6 @@ mod backend;
 mod memory;
 
 pub use crate::backend::{SimpleJITBackend, SimpleJITBuilder};
+
+/// Version number of this crate.
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs
index 90927323a5..6db0fab1f8 100644
--- a/lib/umbrella/src/lib.rs
+++ b/lib/umbrella/src/lib.rs
@@ -44,3 +44,6 @@ pub mod prelude {
 
     pub use crate::frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
 }
+
+/// Version number of this crate.
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index 854fb65dce..4bfaf93012 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -67,3 +67,6 @@ mod std {
         pub use self::hashmap_core::{map as hash_map, HashMap};
     }
 }
+
+/// Version number of this crate.
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");

From 3809e7252d10e9a10d6d75d904d3cf5673a7b9e5 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 29 Dec 2018 12:16:13 +0100
Subject: [PATCH 2296/3084] Remove outdated comment

cranelift-frontend now inserts libcalls for mem{cpy,set,move}
---
 lib/faerie/src/backend.rs | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/lib/faerie/src/backend.rs b/lib/faerie/src/backend.rs
index 699311441d..92cf1e4ba4 100644
--- a/lib/faerie/src/backend.rs
+++ b/lib/faerie/src/backend.rs
@@ -67,8 +67,7 @@ impl FaerieBuilder {
     }
 
     /// Default names for `ir::LibCall`s. A function by this name is imported into the object as
-    /// part of the translation of a `ir::ExternalName::LibCall` variant. Calls to a LibCall should
-    /// only be inserted into the IR by the `cranelift_codegen` legalizer pass.
+    /// part of the translation of a `ir::ExternalName::LibCall` variant.
     pub fn default_libcall_names() -> Box String> {
         Box::new(move |libcall| match libcall {
             ir::LibCall::Probestack => "__cranelift_probestack".to_owned(),

From 2f0e0dd995762dda2d8d8ab8ac1ca229c275e7a7 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Fri, 11 Jan 2019 13:45:10 +0100
Subject: [PATCH 2297/3084] Really fix emit_small_memcpy this time!

---
 lib/frontend/src/frontend.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 8867f9ad69..06fbbecec5 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -612,7 +612,7 @@ impl<'a> FunctionBuilder<'a> {
             "`size` is smaller than `dest` and `src`'s alignment value."
         );
 
-        let (access_size, int_type) = if access_size > 8 {
+        let (access_size, int_type) = if access_size <= 8 {
             (access_size, Type::int((access_size * 8) as u16).unwrap())
         } else {
             (8, types::I64)
@@ -693,7 +693,7 @@ impl<'a> FunctionBuilder<'a> {
             "`size` is smaller than `dest` and `src`'s alignment value."
         );
 
-        let (access_size, int_type) = if access_size > 8 {
+        let (access_size, int_type) = if access_size <= 8 {
             (access_size, Type::int((access_size * 8) as u16).unwrap())
         } else {
             (8, types::I64)

From 50a045363c9890051524b1ba23b2c757baf16f67 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Sun, 6 Jan 2019 13:00:08 -0800
Subject: [PATCH 2298/3084] Add an explicit trap code for wasm `unreachable`.

---
 lib/codegen/src/ir/trapcode.rs  | 9 ++++++++-
 lib/wasm/src/code_translator.rs | 4 +---
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/lib/codegen/src/ir/trapcode.rs b/lib/codegen/src/ir/trapcode.rs
index fc16dddb92..e0d7ddfe55 100644
--- a/lib/codegen/src/ir/trapcode.rs
+++ b/lib/codegen/src/ir/trapcode.rs
@@ -42,6 +42,9 @@ pub enum TrapCode {
     /// Failed float-to-int conversion.
     BadConversionToInteger,
 
+    /// Code that was supposed to have been unreachable was reached.
+    UnreachableCodeReached,
+
     /// Execution has potentially run too long and may be interrupted.
     /// This trap is resumable.
     Interrupt,
@@ -63,6 +66,7 @@ impl Display for TrapCode {
             IntegerOverflow => "int_ovf",
             IntegerDivisionByZero => "int_divz",
             BadConversionToInteger => "bad_toint",
+            UnreachableCodeReached => "unreachable",
             Interrupt => "interrupt",
             User(x) => return write!(f, "user{}", x),
         };
@@ -85,6 +89,7 @@ impl FromStr for TrapCode {
             "int_ovf" => Ok(IntegerOverflow),
             "int_divz" => Ok(IntegerDivisionByZero),
             "bad_toint" => Ok(BadConversionToInteger),
+            "unreachable" => Ok(UnreachableCodeReached),
             "interrupt" => Ok(Interrupt),
             _ if s.starts_with("user") => s[4..].parse().map(User).map_err(|_| ()),
             _ => Err(()),
@@ -98,7 +103,7 @@ mod tests {
     use std::string::ToString;
 
     // Everything but user-defined codes.
-    const CODES: [TrapCode; 9] = [
+    const CODES: [TrapCode; 11] = [
         TrapCode::StackOverflow,
         TrapCode::HeapOutOfBounds,
         TrapCode::TableOutOfBounds,
@@ -108,6 +113,8 @@ mod tests {
         TrapCode::IntegerOverflow,
         TrapCode::IntegerDivisionByZero,
         TrapCode::BadConversionToInteger,
+        TrapCode::UnreachableCodeReached,
+        TrapCode::Interrupt,
     ];
 
     #[test]
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index f956efab31..43b4af8f9e 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -108,9 +108,7 @@ pub fn translate_operator(
             // We do nothing
         }
         Operator::Unreachable => {
-            // We use `trap user0` to indicate a user-generated trap.
-            // We could make the trap code configurable if need be.
-            builder.ins().trap(ir::TrapCode::User(0));
+            builder.ins().trap(ir::TrapCode::UnreachableCodeReached);
             state.reachable = false;
         }
         /***************************** Control flow blocks **********************************

From aeb9161e2cd244a23d3ff8bbee9b63d434d86695 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 7 Jan 2019 11:04:58 -0800
Subject: [PATCH 2299/3084] Update `no_std` support for Rust 2018 Edition.

With Rust 2018 Edition, the `mod std` trick to alias `core` names to
`std` no longer works, so switch to just having the code use `core`
explicitly.

So instead, switch to just using `core::*` for things that in core.
This is more consistent with other Rust no_std code. And it allows
us to enable `no_std` mode unconditionally in the crates that support
it, which makes testing a little easier.

There actually three cases:

 - For things in std and also in core, like `cmp`: Just use them via
   `core::*`.

 - For things in std and also in alloc, like `Vec`: Import alloc as std, as
   use them from std. This allows them to work on both stable (which
   doesn't provide alloc, but we don't support no_std mode anyway) and
   nightly.

 - For HashMap and similar which are not in core or alloc, import them in
   the top-level lib.rs files from either std or the third-party hashmap_core
   crate, and then have the code use super::hashmap_core.

Also, no_std support continues to be "best effort" at this time and not
something most people need to be testing.
---
 cranelift/README.md                      |  2 +-
 cranelift/fuzz/Cargo.toml                |  1 +
 cranelift/test-no_std.sh                 |  2 +-
 lib/bforest/Cargo.toml                   |  1 +
 lib/bforest/src/lib.rs                   | 20 +++++++--------
 lib/bforest/src/map.rs                   |  6 ++---
 lib/bforest/src/node.rs                  |  6 ++---
 lib/bforest/src/path.rs                  |  8 +++---
 lib/bforest/src/pool.rs                  |  8 +++---
 lib/bforest/src/set.rs                   |  6 ++---
 lib/codegen/meta-python/gen_instr.py     | 16 ++++++------
 lib/codegen/src/abi.rs                   |  2 +-
 lib/codegen/src/binemit/memorysink.rs    |  2 +-
 lib/codegen/src/binemit/mod.rs           |  2 +-
 lib/codegen/src/bitset.rs                |  6 ++---
 lib/codegen/src/cfg_printer.rs           |  2 +-
 lib/codegen/src/dbg.rs                   |  2 +-
 lib/codegen/src/dominator_tree.rs        |  6 ++---
 lib/codegen/src/flowgraph.rs             |  2 +-
 lib/codegen/src/fx.rs                    |  8 +++---
 lib/codegen/src/ir/builder.rs            |  2 +-
 lib/codegen/src/ir/condcodes.rs          |  4 +--
 lib/codegen/src/ir/dfg.rs                | 10 ++++----
 lib/codegen/src/ir/entities.rs           |  8 +++---
 lib/codegen/src/ir/extfunc.rs            |  4 +--
 lib/codegen/src/ir/extname.rs            |  8 +++---
 lib/codegen/src/ir/function.rs           |  2 +-
 lib/codegen/src/ir/globalvalue.rs        |  2 +-
 lib/codegen/src/ir/heap.rs               |  2 +-
 lib/codegen/src/ir/immediates.rs         | 14 +++++------
 lib/codegen/src/ir/instructions.rs       | 10 ++++----
 lib/codegen/src/ir/jumptable.rs          |  4 +--
 lib/codegen/src/ir/layout.rs             |  6 ++---
 lib/codegen/src/ir/libcall.rs            |  4 +--
 lib/codegen/src/ir/memflags.rs           |  2 +-
 lib/codegen/src/ir/progpoint.rs          |  6 ++---
 lib/codegen/src/ir/sourceloc.rs          |  2 +-
 lib/codegen/src/ir/stackslot.rs          | 10 ++++----
 lib/codegen/src/ir/table.rs              |  2 +-
 lib/codegen/src/ir/trapcode.rs           |  4 +--
 lib/codegen/src/ir/types.rs              |  4 +--
 lib/codegen/src/ir/valueloc.rs           |  2 +-
 lib/codegen/src/isa/arm32/mod.rs         |  2 +-
 lib/codegen/src/isa/arm32/settings.rs    |  2 +-
 lib/codegen/src/isa/arm64/mod.rs         |  2 +-
 lib/codegen/src/isa/arm64/settings.rs    |  2 +-
 lib/codegen/src/isa/call_conv.rs         |  4 +--
 lib/codegen/src/isa/enc_tables.rs        |  2 +-
 lib/codegen/src/isa/encoding.rs          |  2 +-
 lib/codegen/src/isa/mod.rs               |  2 +-
 lib/codegen/src/isa/registers.rs         |  2 +-
 lib/codegen/src/isa/riscv/abi.rs         |  2 +-
 lib/codegen/src/isa/riscv/binemit.rs     |  2 +-
 lib/codegen/src/isa/riscv/mod.rs         |  4 +--
 lib/codegen/src/isa/riscv/settings.rs    |  2 +-
 lib/codegen/src/isa/x86/abi.rs           |  2 +-
 lib/codegen/src/isa/x86/mod.rs           |  2 +-
 lib/codegen/src/isa/x86/settings.rs      |  2 +-
 lib/codegen/src/legalizer/split.rs       |  2 +-
 lib/codegen/src/lib.rs                   | 30 ++++++++--------------
 lib/codegen/src/partition_slice.rs       |  2 +-
 lib/codegen/src/print_errors.rs          |  4 +--
 lib/codegen/src/ref_slice.rs             |  2 +-
 lib/codegen/src/regalloc/affinity.rs     |  2 +-
 lib/codegen/src/regalloc/coalescing.rs   |  8 +++---
 lib/codegen/src/regalloc/coloring.rs     |  2 +-
 lib/codegen/src/regalloc/diversion.rs    |  4 +--
 lib/codegen/src/regalloc/liveness.rs     |  4 +--
 lib/codegen/src/regalloc/liverange.rs    |  6 ++---
 lib/codegen/src/regalloc/pressure.rs     | 10 ++++----
 lib/codegen/src/regalloc/register_set.rs |  8 +++---
 lib/codegen/src/regalloc/solver.rs       | 10 ++++----
 lib/codegen/src/regalloc/spilling.rs     |  2 +-
 lib/codegen/src/regalloc/virtregs.rs     |  4 +--
 lib/codegen/src/scoped_hash_map.rs       | 19 +++++++-------
 lib/codegen/src/settings.rs              | 29 +++++++++++++--------
 lib/codegen/src/simple_gvn.rs            |  4 +--
 lib/codegen/src/stack_layout.rs          |  2 +-
 lib/codegen/src/timing.rs                |  2 +-
 lib/codegen/src/topo_order.rs            |  2 +-
 lib/codegen/src/verifier/liveness.rs     |  2 +-
 lib/codegen/src/verifier/mod.rs          |  4 +--
 lib/codegen/src/write.rs                 |  2 +-
 lib/entity/Cargo.toml                    |  1 +
 lib/entity/src/boxed_slice.rs            |  8 +++---
 lib/entity/src/iter.rs                   |  6 ++---
 lib/entity/src/keys.rs                   |  6 ++---
 lib/entity/src/lib.rs                    | 14 +++++------
 lib/entity/src/list.rs                   |  4 +--
 lib/entity/src/map.rs                    |  6 ++---
 lib/entity/src/packed_option.rs          |  4 +--
 lib/entity/src/primary.rs                |  9 ++++---
 lib/entity/src/set.rs                    |  4 +--
 lib/entity/src/sparse.rs                 |  6 ++---
 lib/frontend/src/frontend.rs             |  8 +++---
 lib/frontend/src/lib.rs                  | 26 ++++++++-----------
 lib/frontend/src/ssa.rs                  |  4 +--
 lib/frontend/src/switch.rs               |  6 ++---
 lib/frontend/src/variable.rs             |  2 +-
 lib/module/Cargo.toml                    |  2 +-
 lib/module/src/backend.rs                |  2 +-
 lib/module/src/data_context.rs           |  4 +--
 lib/module/src/lib.rs                    | 29 ++++++++-------------
 lib/module/src/module.rs                 |  6 ++---
 lib/native/src/lib.rs                    |  2 +-
 lib/preopt/src/constant_folding.rs       |  2 +-
 lib/preopt/src/lib.rs                    | 15 +++++------
 lib/serde/Cargo.toml                     | 10 ++------
 lib/simplejit/Cargo.toml                 | 16 ++++--------
 lib/umbrella/src/lib.rs                  |  1 +
 lib/wasm/Cargo.toml                      |  2 +-
 lib/wasm/src/code_translator.rs          |  9 +++----
 lib/wasm/src/environ/dummy.rs            |  1 +
 lib/wasm/src/environ/spec.rs             |  2 +-
 lib/wasm/src/lib.rs                      | 32 ++++++++++--------------
 lib/wasm/src/sections_translator.rs      |  2 +-
 lib/wasm/src/state.rs                    |  2 +-
 lib/wasm/src/translation_utils.rs        |  2 +-
 118 files changed, 322 insertions(+), 355 deletions(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index 5f579b0eb6..f8bd1e8c15 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -94,7 +94,7 @@ The following crates support \`no\_std\`, although they do depend on liballoc:
  - cranelift-native
  - cranelift-wasm
  - cranelift-module
- - cranelift-simplejit
+ - cranelift-preopt
  - cranelift
 
 To use no\_std mode, disable the std feature and enable the core
diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
index a78d0ed3fc..0285feaa51 100644
--- a/cranelift/fuzz/Cargo.toml
+++ b/cranelift/fuzz/Cargo.toml
@@ -3,6 +3,7 @@ name = "clif-wasm-fuzz"
 version = "0.0.1"
 authors = ["foote@fastly.com"]
 publish = false
+edition = "2018"
 
 [package.metadata]
 cargo-fuzz = true
diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh
index 32c354c065..dab7e3afbd 100755
--- a/cranelift/test-no_std.sh
+++ b/cranelift/test-no_std.sh
@@ -13,7 +13,7 @@ function banner {
 }
 
 # Test those packages which have no_std support.
-LIBS="codegen frontend wasm native preopt module simplejit umbrella"
+LIBS="codegen frontend wasm native preopt module entity bforest umbrella"
 for LIB in $LIBS; do
     banner "Rust unit tests in $LIB"
     pushd "lib/$LIB" >/dev/null
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index 0181e32a83..36ec8a752c 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -17,6 +17,7 @@ cranelift-entity = { path = "../entity", version = "0.26.0", default-features =
 [features]
 default = ["std"]
 std = ["cranelift-entity/std"]
+core = []
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/bforest/src/lib.rs b/lib/bforest/src/lib.rs
index e78292600e..7b6beb2599 100644
--- a/lib/bforest/src/lib.rs
+++ b/lib/bforest/src/lib.rs
@@ -31,24 +31,24 @@
         clippy::use_self
     )
 )]
-// Turns on no_std and alloc features if std is not available.
-#![cfg_attr(not(feature = "std"), no_std)]
+#![no_std]
 #![cfg_attr(not(feature = "std"), feature(alloc))]
 
-/// This replaces `std` in builds with `core`.
+#[cfg(test)]
 #[cfg(not(feature = "std"))]
-mod std {
-    extern crate alloc;
-    pub use self::alloc::{boxed, string, vec};
-    pub use core::*;
-}
+#[macro_use]
+extern crate alloc as std;
+#[cfg(test)]
+#[cfg(feature = "std")]
+#[macro_use]
+extern crate std;
 
 #[macro_use]
 extern crate cranelift_entity as entity;
 use crate::entity::packed_option;
 
-use std::borrow::BorrowMut;
-use std::cmp::Ordering;
+use core::borrow::BorrowMut;
+use core::cmp::Ordering;
 
 mod map;
 mod node;
diff --git a/lib/bforest/src/map.rs b/lib/bforest/src/map.rs
index 313b870f49..3cf8421a30 100644
--- a/lib/bforest/src/map.rs
+++ b/lib/bforest/src/map.rs
@@ -3,8 +3,8 @@
 use super::{Comparator, Forest, Node, NodeData, NodePool, Path, INNER_SIZE};
 use crate::packed_option::PackedOption;
 #[cfg(test)]
-use std::fmt;
-use std::marker::PhantomData;
+use core::fmt;
+use core::marker::PhantomData;
 #[cfg(test)]
 use std::string::String;
 
@@ -429,7 +429,7 @@ where
 mod tests {
     use super::super::NodeData;
     use super::*;
-    use std::mem;
+    use core::mem;
     use std::vec::Vec;
 
     #[test]
diff --git a/lib/bforest/src/node.rs b/lib/bforest/src/node.rs
index 583fe0f78e..f0dbc4d7f5 100644
--- a/lib/bforest/src/node.rs
+++ b/lib/bforest/src/node.rs
@@ -1,8 +1,8 @@
 //! B+-tree nodes.
 
 use super::{slice_insert, slice_shift, Forest, Node, SetValue, INNER_SIZE};
-use std::borrow::{Borrow, BorrowMut};
-use std::fmt;
+use core::borrow::{Borrow, BorrowMut};
+use core::fmt;
 
 /// B+-tree node.
 ///
@@ -584,7 +584,7 @@ where
 #[cfg(test)]
 mod tests {
     use super::*;
-    use std::mem;
+    use core::mem;
     use std::string::ToString;
 
     // Forest impl for a set implementation.
diff --git a/lib/bforest/src/path.rs b/lib/bforest/src/path.rs
index 7ccb9ee6f5..1df66b7ef7 100644
--- a/lib/bforest/src/path.rs
+++ b/lib/bforest/src/path.rs
@@ -2,11 +2,11 @@
 
 use super::node::Removed;
 use super::{slice_insert, slice_shift, Comparator, Forest, Node, NodeData, NodePool, MAX_PATH};
-use std::borrow::Borrow;
-use std::marker::PhantomData;
+use core::borrow::Borrow;
+use core::marker::PhantomData;
 
 #[cfg(test)]
-use std::fmt;
+use core::fmt;
 
 pub(super) struct Path {
     /// Number of path entries including the root and leaf nodes.
@@ -706,7 +706,7 @@ impl fmt::Display for Path {
 mod tests {
     use super::super::{Forest, NodeData, NodePool};
     use super::*;
-    use std::cmp::Ordering;
+    use core::cmp::Ordering;
 
     struct TC();
 
diff --git a/lib/bforest/src/pool.rs b/lib/bforest/src/pool.rs
index 0eb873ccab..1b51cfe56b 100644
--- a/lib/bforest/src/pool.rs
+++ b/lib/bforest/src/pool.rs
@@ -5,8 +5,8 @@ use super::Comparator;
 use super::{Forest, Node, NodeData};
 use crate::entity::PrimaryMap;
 #[cfg(test)]
-use std::fmt;
-use std::ops::{Index, IndexMut};
+use core::fmt;
+use core::ops::{Index, IndexMut};
 
 /// A pool of nodes, including a free list.
 pub(super) struct NodePool {
@@ -84,8 +84,8 @@ impl NodePool {
         F::Key: fmt::Display,
     {
         use crate::entity::SparseSet;
-        use std::borrow::Borrow;
-        use std::cmp::Ordering;
+        use core::borrow::Borrow;
+        use core::cmp::Ordering;
         use std::vec::Vec;
 
         // The root node can't be an inner node with just a single sub-tree. It should have been
diff --git a/lib/bforest/src/set.rs b/lib/bforest/src/set.rs
index e73171d368..72e517561f 100644
--- a/lib/bforest/src/set.rs
+++ b/lib/bforest/src/set.rs
@@ -3,8 +3,8 @@
 use super::{Comparator, Forest, Node, NodeData, NodePool, Path, SetValue, INNER_SIZE};
 use crate::packed_option::PackedOption;
 #[cfg(test)]
-use std::fmt;
-use std::marker::PhantomData;
+use core::fmt;
+use core::marker::PhantomData;
 #[cfg(test)]
 use std::string::String;
 
@@ -357,7 +357,7 @@ where
 mod tests {
     use super::super::NodeData;
     use super::*;
-    use std::mem;
+    use core::mem;
     use std::vec::Vec;
 
     #[test]
diff --git a/lib/codegen/meta-python/gen_instr.py b/lib/codegen/meta-python/gen_instr.py
index 57be62a5d4..1c6905f1cd 100644
--- a/lib/codegen/meta-python/gen_instr.py
+++ b/lib/codegen/meta-python/gen_instr.py
@@ -257,8 +257,8 @@ def gen_instruction_data_impl(fmt):
                 'pub fn eq(&self, other: &Self, pool: &ir::ValueListPool)'
                 ' -> bool {',
                 '}'):
-            with fmt.indented('if ::std::mem::discriminant(self) != '
-                              '::std::mem::discriminant(other) {', '}'):
+            with fmt.indented('if ::core::mem::discriminant(self) != '
+                              '::core::mem::discriminant(other) {', '}'):
                 fmt.line('return false;')
             with fmt.indented('match (self, other) {', '}'):
                 for f in InstructionFormat.all_formats:
@@ -301,7 +301,7 @@ def gen_instruction_data_impl(fmt):
                 hash the contents of any `ValueLists`.
                 """)
         with fmt.indented(
-                'pub fn hash'
+                'pub fn hash'
                 '(&self, state: &mut H, pool: &ir::ValueListPool) {',
                 '}'):
             with fmt.indented('match *self {', '}'):
@@ -323,13 +323,13 @@ def gen_instruction_data_impl(fmt):
                         members.append(field.member)
                     pat = n + ' { ' + ', '.join(members) + ' }'
                     with fmt.indented(pat + ' => {', '}'):
-                        fmt.line('::std::hash::Hash::hash( '
-                                 '&::std::mem::discriminant(self), state);')
-                        fmt.line('::std::hash::Hash::hash(&opcode, state);')
+                        fmt.line('::core::hash::Hash::hash( '
+                                 '&::core::mem::discriminant(self), state);')
+                        fmt.line('::core::hash::Hash::hash(&opcode, state);')
                         for field in f.imm_fields:
-                            fmt.line('::std::hash::Hash::hash(&{}, state);'
+                            fmt.line('::core::hash::Hash::hash(&{}, state);'
                                      .format(field.member))
-                        fmt.line('::std::hash::Hash::hash({}, state);'
+                        fmt.line('::core::hash::Hash::hash({}, state);'
                                  .format(args))
 
 
diff --git a/lib/codegen/src/abi.rs b/lib/codegen/src/abi.rs
index 954c40b643..4c0b8d4dbc 100644
--- a/lib/codegen/src/abi.rs
+++ b/lib/codegen/src/abi.rs
@@ -4,7 +4,7 @@
 //! `TargetIsa::legalize_signature()` method.
 
 use crate::ir::{AbiParam, ArgumentExtension, ArgumentLoc, Type};
-use std::cmp::Ordering;
+use core::cmp::Ordering;
 use std::vec::Vec;
 
 /// Legalization action to perform on a single argument or return value when converting a
diff --git a/lib/codegen/src/binemit/memorysink.rs b/lib/codegen/src/binemit/memorysink.rs
index 3d7d673719..18aa578f09 100644
--- a/lib/codegen/src/binemit/memorysink.rs
+++ b/lib/codegen/src/binemit/memorysink.rs
@@ -16,7 +16,7 @@
 
 use super::{Addend, CodeOffset, CodeSink, Reloc};
 use crate::ir::{ExternalName, JumpTable, SourceLoc, TrapCode};
-use std::ptr::write_unaligned;
+use core::ptr::write_unaligned;
 
 /// A `CodeSink` that writes binary machine code directly into memory.
 ///
diff --git a/lib/codegen/src/binemit/mod.rs b/lib/codegen/src/binemit/mod.rs
index c4ad46c7df..63fa7ef40b 100644
--- a/lib/codegen/src/binemit/mod.rs
+++ b/lib/codegen/src/binemit/mod.rs
@@ -13,7 +13,7 @@ pub use self::shrink::shrink_instructions;
 pub use crate::regalloc::RegDiversions;
 
 use crate::ir::{ExternalName, Function, Inst, JumpTable, SourceLoc, TrapCode};
-use std::fmt;
+use core::fmt;
 
 /// Offset in bytes from the beginning of the function.
 ///
diff --git a/lib/codegen/src/bitset.rs b/lib/codegen/src/bitset.rs
index 10d624d8a2..ccbe413175 100644
--- a/lib/codegen/src/bitset.rs
+++ b/lib/codegen/src/bitset.rs
@@ -5,9 +5,9 @@
 //!
 //! If you would like to add support for larger bitsets in the future, you need to change the trait
 //! bound Into and the u32 in the implementation of `max_bits()`.
-use std::convert::{From, Into};
-use std::mem::size_of;
-use std::ops::{Add, BitOr, Shl, Sub};
+use core::convert::{From, Into};
+use core::mem::size_of;
+use core::ops::{Add, BitOr, Shl, Sub};
 
 /// A small bitset built on a single primitive integer type
 #[derive(Clone, Copy, Debug, PartialEq, Eq)]
diff --git a/lib/codegen/src/cfg_printer.rs b/lib/codegen/src/cfg_printer.rs
index c15ec134b9..6a9932febe 100644
--- a/lib/codegen/src/cfg_printer.rs
+++ b/lib/codegen/src/cfg_printer.rs
@@ -1,6 +1,6 @@
 //! The `CFGPrinter` utility.
 
-use std::fmt::{Display, Formatter, Result, Write};
+use core::fmt::{Display, Formatter, Result, Write};
 
 use crate::flowgraph::{BasicBlock, ControlFlowGraph};
 use crate::ir::instructions::BranchInfo;
diff --git a/lib/codegen/src/dbg.rs b/lib/codegen/src/dbg.rs
index f19c222caf..1d814ceedb 100644
--- a/lib/codegen/src/dbg.rs
+++ b/lib/codegen/src/dbg.rs
@@ -1,5 +1,5 @@
 //! Debug tracing helpers.
-use std::fmt;
+use core::fmt;
 
 /// Prefix added to the log file names, just before the thread name or id.
 pub static LOG_FILENAME_PREFIX: &str = "cranelift.dbg.";
diff --git a/lib/codegen/src/dominator_tree.rs b/lib/codegen/src/dominator_tree.rs
index 9d20eab0d4..feea33a0ef 100644
--- a/lib/codegen/src/dominator_tree.rs
+++ b/lib/codegen/src/dominator_tree.rs
@@ -6,9 +6,9 @@ use crate::ir::instructions::BranchInfo;
 use crate::ir::{Ebb, ExpandedProgramPoint, Function, Inst, Layout, ProgramOrder, Value};
 use crate::packed_option::PackedOption;
 use crate::timing;
-use std::cmp;
-use std::cmp::Ordering;
-use std::mem;
+use core::cmp;
+use core::cmp::Ordering;
+use core::mem;
 use std::vec::Vec;
 
 /// RPO numbers are not first assigned in a contiguous way but as multiples of STRIDE, to leave
diff --git a/lib/codegen/src/flowgraph.rs b/lib/codegen/src/flowgraph.rs
index 4412e53599..335ca15ce4 100644
--- a/lib/codegen/src/flowgraph.rs
+++ b/lib/codegen/src/flowgraph.rs
@@ -28,7 +28,7 @@ use crate::entity::SecondaryMap;
 use crate::ir::instructions::BranchInfo;
 use crate::ir::{Ebb, Function, Inst};
 use crate::timing;
-use std::mem;
+use core::mem;
 
 /// A basic block denoted by its enclosing Ebb and last instruction.
 #[derive(PartialEq, Eq)]
diff --git a/lib/codegen/src/fx.rs b/lib/codegen/src/fx.rs
index eb3b7f2b36..36eb62df90 100644
--- a/lib/codegen/src/fx.rs
+++ b/lib/codegen/src/fx.rs
@@ -10,10 +10,10 @@
 // option. This file may not be copied, modified, or distributed
 // except according to those terms.
 
-use std::collections::{HashMap, HashSet};
-use std::default::Default;
-use std::hash::{BuildHasherDefault, Hash, Hasher};
-use std::ops::BitXor;
+use super::{HashMap, HashSet};
+use core::default::Default;
+use core::hash::{BuildHasherDefault, Hash, Hasher};
+use core::ops::BitXor;
 
 pub type FxHashMap = HashMap>;
 pub type FxHashSet = HashSet>;
diff --git a/lib/codegen/src/ir/builder.rs b/lib/codegen/src/ir/builder.rs
index 7ed9a8e79e..f84e23eab8 100644
--- a/lib/codegen/src/ir/builder.rs
+++ b/lib/codegen/src/ir/builder.rs
@@ -59,7 +59,7 @@ pub trait InstInserterBase<'f>: Sized {
     fn insert_built_inst(self, inst: Inst, ctrl_typevar: Type) -> &'f mut DataFlowGraph;
 }
 
-use std::marker::PhantomData;
+use core::marker::PhantomData;
 
 /// Builder that inserts an instruction at the current position.
 ///
diff --git a/lib/codegen/src/ir/condcodes.rs b/lib/codegen/src/ir/condcodes.rs
index d7208278c7..743e30954d 100644
--- a/lib/codegen/src/ir/condcodes.rs
+++ b/lib/codegen/src/ir/condcodes.rs
@@ -4,8 +4,8 @@
 //! are different rules for comparing integers and floating point numbers, so they use different
 //! condition codes.
 
-use std::fmt::{self, Display, Formatter};
-use std::str::FromStr;
+use core::fmt::{self, Display, Formatter};
+use core::str::FromStr;
 
 /// Common traits of condition codes.
 pub trait CondCode: Copy {
diff --git a/lib/codegen/src/ir/dfg.rs b/lib/codegen/src/ir/dfg.rs
index 7d3e5c94ea..93ea2bbe8c 100644
--- a/lib/codegen/src/ir/dfg.rs
+++ b/lib/codegen/src/ir/dfg.rs
@@ -10,11 +10,11 @@ use crate::ir::{Ebb, FuncRef, Inst, SigRef, Signature, Type, Value, ValueList, V
 use crate::isa::TargetIsa;
 use crate::packed_option::ReservedValue;
 use crate::write::write_operands;
-use std::fmt;
-use std::iter;
-use std::mem;
-use std::ops::{Index, IndexMut};
-use std::u16;
+use core::fmt;
+use core::iter;
+use core::mem;
+use core::ops::{Index, IndexMut};
+use core::u16;
 
 /// A data flow graph defines all instructions and extended basic blocks in a function as well as
 /// the data flow dependencies between them. The DFG also tracks values which can be either
diff --git a/lib/codegen/src/ir/entities.rs b/lib/codegen/src/ir/entities.rs
index b669cf77b6..a0b694ad84 100644
--- a/lib/codegen/src/ir/entities.rs
+++ b/lib/codegen/src/ir/entities.rs
@@ -20,8 +20,8 @@
 //! format.
 
 use crate::entity::entity_impl;
-use std::fmt;
-use std::u32;
+use core::fmt;
+use core::u32;
 
 /// An opaque reference to an extended basic block in a function.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
@@ -305,8 +305,8 @@ impl From
 for AnyEntity {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use core::u32;
     use std::string::ToString;
-    use std::u32;
 
     #[test]
     fn value_with_number() {
@@ -320,7 +320,7 @@ mod tests {
     #[test]
     fn memory() {
         use crate::packed_option::PackedOption;
-        use std::mem;
+        use core::mem;
         // This is the whole point of `PackedOption`.
         assert_eq!(
             mem::size_of::(),
diff --git a/lib/codegen/src/ir/extfunc.rs b/lib/codegen/src/ir/extfunc.rs
index 9c17f34556..ada7bab8ca 100644
--- a/lib/codegen/src/ir/extfunc.rs
+++ b/lib/codegen/src/ir/extfunc.rs
@@ -7,8 +7,8 @@
 
 use crate::ir::{ArgumentLoc, ExternalName, SigRef, Type};
 use crate::isa::{CallConv, RegInfo, RegUnit};
-use std::fmt;
-use std::str::FromStr;
+use core::fmt;
+use core::str::FromStr;
 use std::vec::Vec;
 
 /// Function signature.
diff --git a/lib/codegen/src/ir/extname.rs b/lib/codegen/src/ir/extname.rs
index 7a9da78f7f..5fd8600882 100644
--- a/lib/codegen/src/ir/extname.rs
+++ b/lib/codegen/src/ir/extname.rs
@@ -5,9 +5,9 @@
 //! Cranelift, which compiles functions independently.
 
 use crate::ir::LibCall;
-use std::cmp;
-use std::fmt::{self, Write};
-use std::str::FromStr;
+use core::cmp;
+use core::fmt::{self, Write};
+use core::str::FromStr;
 
 const TESTCASE_NAME_LENGTH: usize = 16;
 
@@ -120,8 +120,8 @@ impl FromStr for ExternalName {
 mod tests {
     use super::ExternalName;
     use crate::ir::LibCall;
+    use core::u32;
     use std::string::ToString;
-    use std::u32;
 
     #[test]
     fn display_testcase() {
diff --git a/lib/codegen/src/ir/function.rs b/lib/codegen/src/ir/function.rs
index 85bfdb5207..e0ad323d0a 100644
--- a/lib/codegen/src/ir/function.rs
+++ b/lib/codegen/src/ir/function.rs
@@ -16,7 +16,7 @@ use crate::ir::{JumpTableOffsets, JumpTables};
 use crate::isa::{CallConv, EncInfo, Encoding, Legalize, TargetIsa};
 use crate::regalloc::RegDiversions;
 use crate::write::write_function;
-use std::fmt;
+use core::fmt;
 
 /// A function.
 ///
diff --git a/lib/codegen/src/ir/globalvalue.rs b/lib/codegen/src/ir/globalvalue.rs
index 6c7d8032f6..5dc7792867 100644
--- a/lib/codegen/src/ir/globalvalue.rs
+++ b/lib/codegen/src/ir/globalvalue.rs
@@ -3,7 +3,7 @@
 use crate::ir::immediates::{Imm64, Offset32};
 use crate::ir::{ExternalName, GlobalValue, Type};
 use crate::isa::TargetIsa;
-use std::fmt;
+use core::fmt;
 
 /// Information about a global value declaration.
 #[derive(Clone)]
diff --git a/lib/codegen/src/ir/heap.rs b/lib/codegen/src/ir/heap.rs
index 8495fd51e3..8a4b4e84b9 100644
--- a/lib/codegen/src/ir/heap.rs
+++ b/lib/codegen/src/ir/heap.rs
@@ -2,7 +2,7 @@
 
 use crate::ir::immediates::Uimm64;
 use crate::ir::{GlobalValue, Type};
-use std::fmt;
+use core::fmt;
 
 /// Information about a heap declaration.
 #[derive(Clone)]
diff --git a/lib/codegen/src/ir/immediates.rs b/lib/codegen/src/ir/immediates.rs
index 9badf2fcfd..21d8376e37 100644
--- a/lib/codegen/src/ir/immediates.rs
+++ b/lib/codegen/src/ir/immediates.rs
@@ -4,10 +4,10 @@
 //! Each type here should have a corresponding definition in the `cranelift.immediates` Python
 //! module in the meta language.
 
-use std::fmt::{self, Display, Formatter};
-use std::mem;
-use std::str::FromStr;
-use std::{i32, u32};
+use core::fmt::{self, Display, Formatter};
+use core::mem;
+use core::str::FromStr;
+use core::{i32, u32};
 
 /// 64-bit immediate signed integer operand.
 ///
@@ -729,10 +729,10 @@ impl FromStr for Ieee64 {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use std::fmt::Display;
-    use std::str::FromStr;
+    use core::fmt::Display;
+    use core::str::FromStr;
+    use core::{f32, f64};
     use std::string::ToString;
-    use std::{f32, f64};
 
     #[test]
     fn format_imm64() {
diff --git a/lib/codegen/src/ir/instructions.rs b/lib/codegen/src/ir/instructions.rs
index 4f9cf5125e..ee4a0b416a 100644
--- a/lib/codegen/src/ir/instructions.rs
+++ b/lib/codegen/src/ir/instructions.rs
@@ -6,9 +6,9 @@
 //! A large part of this module is auto-generated from the instruction descriptions in the meta
 //! directory.
 
-use std::fmt::{self, Display, Formatter};
-use std::ops::{Deref, DerefMut};
-use std::str::FromStr;
+use core::fmt::{self, Display, Formatter};
+use core::ops::{Deref, DerefMut};
+use core::str::FromStr;
 use std::vec::Vec;
 
 use crate::ir;
@@ -561,7 +561,7 @@ mod tests {
 
     #[test]
     fn opcodes() {
-        use std::mem;
+        use core::mem;
 
         let x = Opcode::Iadd;
         let mut y = Opcode::Isub;
@@ -590,7 +590,7 @@ mod tests {
 
     #[test]
     fn instruction_data() {
-        use std::mem;
+        use core::mem;
         // The size of the `InstructionData` enum is important for performance. It should not
         // exceed 16 bytes. Use `Box` out-of-line payloads for instruction formats that
         // require more space than that. It would be fine with a data structure smaller than 16
diff --git a/lib/codegen/src/ir/jumptable.rs b/lib/codegen/src/ir/jumptable.rs
index e436ea412f..a40ee48e74 100644
--- a/lib/codegen/src/ir/jumptable.rs
+++ b/lib/codegen/src/ir/jumptable.rs
@@ -4,8 +4,8 @@
 //! The actual table of destinations is stored in a `JumpTableData` struct defined in this module.
 
 use crate::ir::entities::Ebb;
-use std::fmt::{self, Display, Formatter};
-use std::slice::{Iter, IterMut};
+use core::fmt::{self, Display, Formatter};
+use core::slice::{Iter, IterMut};
 use std::vec::Vec;
 
 /// Contents of a jump table.
diff --git a/lib/codegen/src/ir/layout.rs b/lib/codegen/src/ir/layout.rs
index 748217847f..e1015f7f3f 100644
--- a/lib/codegen/src/ir/layout.rs
+++ b/lib/codegen/src/ir/layout.rs
@@ -8,9 +8,9 @@ use crate::ir::progpoint::{ExpandedProgramPoint, ProgramOrder};
 use crate::ir::{Ebb, Inst};
 use crate::packed_option::PackedOption;
 use crate::timing;
+use core::cmp;
+use core::iter::{IntoIterator, Iterator};
 use log::debug;
-use std::cmp;
-use std::iter::{IntoIterator, Iterator};
 
 /// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not
 /// contain definitions of instructions or EBBs, but depends on `Inst` and `Ebb` entity references
@@ -745,7 +745,7 @@ mod tests {
     use crate::cursor::{Cursor, CursorPosition};
     use crate::entity::EntityRef;
     use crate::ir::{Ebb, Inst, ProgramOrder, SourceLoc};
-    use std::cmp::Ordering;
+    use core::cmp::Ordering;
     use std::vec::Vec;
 
     struct LayoutCursor<'f> {
diff --git a/lib/codegen/src/ir/libcall.rs b/lib/codegen/src/ir/libcall.rs
index c282de34e4..3617f906d6 100644
--- a/lib/codegen/src/ir/libcall.rs
+++ b/lib/codegen/src/ir/libcall.rs
@@ -5,8 +5,8 @@ use crate::ir::{
     Signature, Type,
 };
 use crate::isa::{CallConv, RegUnit, TargetIsa};
-use std::fmt;
-use std::str::FromStr;
+use core::fmt;
+use core::str::FromStr;
 
 /// The name of a runtime library routine.
 ///
diff --git a/lib/codegen/src/ir/memflags.rs b/lib/codegen/src/ir/memflags.rs
index 1a1f48d015..87fd6bf3ab 100644
--- a/lib/codegen/src/ir/memflags.rs
+++ b/lib/codegen/src/ir/memflags.rs
@@ -1,6 +1,6 @@
 //! Memory operation flags.
 
-use std::fmt;
+use core::fmt;
 
 enum FlagBit {
     Notrap,
diff --git a/lib/codegen/src/ir/progpoint.rs b/lib/codegen/src/ir/progpoint.rs
index abe2dd786f..bd03c120cd 100644
--- a/lib/codegen/src/ir/progpoint.rs
+++ b/lib/codegen/src/ir/progpoint.rs
@@ -2,9 +2,9 @@
 
 use crate::entity::EntityRef;
 use crate::ir::{Ebb, Inst, ValueDef};
-use std::cmp;
-use std::fmt;
-use std::u32;
+use core::cmp;
+use core::fmt;
+use core::u32;
 
 /// A `ProgramPoint` represents a position in a function where the live range of an SSA value can
 /// begin or end. It can be either:
diff --git a/lib/codegen/src/ir/sourceloc.rs b/lib/codegen/src/ir/sourceloc.rs
index 3f2add7346..ab5722a57d 100644
--- a/lib/codegen/src/ir/sourceloc.rs
+++ b/lib/codegen/src/ir/sourceloc.rs
@@ -3,7 +3,7 @@
 //! Cranelift tracks the original source location of each instruction, and preserves the source
 //! location when instructions are transformed.
 
-use std::fmt;
+use core::fmt;
 
 /// A source location.
 ///
diff --git a/lib/codegen/src/ir/stackslot.rs b/lib/codegen/src/ir/stackslot.rs
index bf2d555bf2..4c94a6b012 100644
--- a/lib/codegen/src/ir/stackslot.rs
+++ b/lib/codegen/src/ir/stackslot.rs
@@ -6,11 +6,11 @@
 use crate::entity::{Iter, IterMut, Keys, PrimaryMap};
 use crate::ir::{StackSlot, Type};
 use crate::packed_option::PackedOption;
-use std::cmp;
-use std::fmt;
-use std::ops::{Index, IndexMut};
-use std::slice;
-use std::str::FromStr;
+use core::cmp;
+use core::fmt;
+use core::ops::{Index, IndexMut};
+use core::slice;
+use core::str::FromStr;
 use std::vec::Vec;
 
 /// The size of an object on the stack, or the size of a stack frame.
diff --git a/lib/codegen/src/ir/table.rs b/lib/codegen/src/ir/table.rs
index 003eb710b9..9e436cca64 100644
--- a/lib/codegen/src/ir/table.rs
+++ b/lib/codegen/src/ir/table.rs
@@ -2,7 +2,7 @@
 
 use crate::ir::immediates::Uimm64;
 use crate::ir::{GlobalValue, Type};
-use std::fmt;
+use core::fmt;
 
 /// Information about a table declaration.
 #[derive(Clone)]
diff --git a/lib/codegen/src/ir/trapcode.rs b/lib/codegen/src/ir/trapcode.rs
index e0d7ddfe55..273a77670c 100644
--- a/lib/codegen/src/ir/trapcode.rs
+++ b/lib/codegen/src/ir/trapcode.rs
@@ -1,7 +1,7 @@
 //! Trap codes describing the reason for a trap.
 
-use std::fmt::{self, Display, Formatter};
-use std::str::FromStr;
+use core::fmt::{self, Display, Formatter};
+use core::str::FromStr;
 
 /// A trap code describing the reason for a trap.
 ///
diff --git a/lib/codegen/src/ir/types.rs b/lib/codegen/src/ir/types.rs
index ceffe04e9c..2d946e04f8 100644
--- a/lib/codegen/src/ir/types.rs
+++ b/lib/codegen/src/ir/types.rs
@@ -1,7 +1,7 @@
 //! Common types for the Cranelift code generator.
 
-use std::default::Default;
-use std::fmt::{self, Debug, Display, Formatter};
+use core::default::Default;
+use core::fmt::{self, Debug, Display, Formatter};
 use target_lexicon::{PointerWidth, Triple};
 
 /// The type of an SSA value.
diff --git a/lib/codegen/src/ir/valueloc.rs b/lib/codegen/src/ir/valueloc.rs
index f2c5151f09..0cbcd5d2c3 100644
--- a/lib/codegen/src/ir/valueloc.rs
+++ b/lib/codegen/src/ir/valueloc.rs
@@ -5,7 +5,7 @@
 
 use crate::ir::StackSlot;
 use crate::isa::{RegInfo, RegUnit};
-use std::fmt;
+use core::fmt;
 
 /// Value location.
 #[derive(Copy, Clone, Debug, PartialEq, Eq)]
diff --git a/lib/codegen/src/isa/arm32/mod.rs b/lib/codegen/src/isa/arm32/mod.rs
index a184061ac0..7ceacf036b 100644
--- a/lib/codegen/src/isa/arm32/mod.rs
+++ b/lib/codegen/src/isa/arm32/mod.rs
@@ -15,8 +15,8 @@ use crate::isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encoding
 use crate::isa::Builder as IsaBuilder;
 use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
+use core::fmt;
 use std::boxed::Box;
-use std::fmt;
 use target_lexicon::{Architecture, Triple};
 
 #[allow(dead_code)]
diff --git a/lib/codegen/src/isa/arm32/settings.rs b/lib/codegen/src/isa/arm32/settings.rs
index d32483100f..5490c8c2b3 100644
--- a/lib/codegen/src/isa/arm32/settings.rs
+++ b/lib/codegen/src/isa/arm32/settings.rs
@@ -1,7 +1,7 @@
 //! ARM32 Settings.
 
 use crate::settings::{self, detail, Builder};
-use std::fmt;
+use core::fmt;
 
 // Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
 // `Flags` struct with an impl for all of the settings defined in
diff --git a/lib/codegen/src/isa/arm64/mod.rs b/lib/codegen/src/isa/arm64/mod.rs
index b438e49949..84277301c3 100644
--- a/lib/codegen/src/isa/arm64/mod.rs
+++ b/lib/codegen/src/isa/arm64/mod.rs
@@ -15,8 +15,8 @@ use crate::isa::enc_tables::{lookup_enclist, Encodings};
 use crate::isa::Builder as IsaBuilder;
 use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
+use core::fmt;
 use std::boxed::Box;
-use std::fmt;
 use target_lexicon::Triple;
 
 #[allow(dead_code)]
diff --git a/lib/codegen/src/isa/arm64/settings.rs b/lib/codegen/src/isa/arm64/settings.rs
index 6ef1fcb258..e7d0abdf6c 100644
--- a/lib/codegen/src/isa/arm64/settings.rs
+++ b/lib/codegen/src/isa/arm64/settings.rs
@@ -1,7 +1,7 @@
 //! ARM64 Settings.
 
 use crate::settings::{self, detail, Builder};
-use std::fmt;
+use core::fmt;
 
 // Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
 // `Flags` struct with an impl for all of the settings defined in
diff --git a/lib/codegen/src/isa/call_conv.rs b/lib/codegen/src/isa/call_conv.rs
index 9caff3ebaa..d4b4d3999b 100644
--- a/lib/codegen/src/isa/call_conv.rs
+++ b/lib/codegen/src/isa/call_conv.rs
@@ -1,5 +1,5 @@
-use std::fmt;
-use std::str;
+use core::fmt;
+use core::str;
 use target_lexicon::{CallingConvention, Triple};
 
 /// Calling convention identifiers.
diff --git a/lib/codegen/src/isa/enc_tables.rs b/lib/codegen/src/isa/enc_tables.rs
index ccdbc28ebd..323473fe2b 100644
--- a/lib/codegen/src/isa/enc_tables.rs
+++ b/lib/codegen/src/isa/enc_tables.rs
@@ -7,7 +7,7 @@ use crate::constant_hash::{probe, Table};
 use crate::ir::{Function, InstructionData, Opcode, Type};
 use crate::isa::{Encoding, Legalize};
 use crate::settings::PredicateView;
-use std::ops::Range;
+use core::ops::Range;
 
 /// A recipe predicate.
 ///
diff --git a/lib/codegen/src/isa/encoding.rs b/lib/codegen/src/isa/encoding.rs
index bf6a001b07..118bfa2006 100644
--- a/lib/codegen/src/isa/encoding.rs
+++ b/lib/codegen/src/isa/encoding.rs
@@ -4,7 +4,7 @@ use crate::binemit::CodeOffset;
 use crate::ir::{Function, Inst};
 use crate::isa::constraints::{BranchRange, RecipeConstraints};
 use crate::regalloc::RegDiversions;
-use std::fmt;
+use core::fmt;
 
 /// Bits needed to encode an instruction as binary machine code.
 ///
diff --git a/lib/codegen/src/isa/mod.rs b/lib/codegen/src/isa/mod.rs
index 291fdfb76c..ab77269ea5 100644
--- a/lib/codegen/src/isa/mod.rs
+++ b/lib/codegen/src/isa/mod.rs
@@ -63,9 +63,9 @@ use crate::result::CodegenResult;
 use crate::settings;
 use crate::settings::SetResult;
 use crate::timing;
+use core::fmt;
 use failure_derive::Fail;
 use std::boxed::Box;
-use std::fmt;
 use target_lexicon::{Architecture, PointerWidth, Triple};
 
 #[cfg(build_riscv)]
diff --git a/lib/codegen/src/isa/registers.rs b/lib/codegen/src/isa/registers.rs
index 4941712618..6baebd7237 100644
--- a/lib/codegen/src/isa/registers.rs
+++ b/lib/codegen/src/isa/registers.rs
@@ -1,7 +1,7 @@
 //! Data structures describing the registers in an ISA.
 
 use crate::entity::EntityRef;
-use std::fmt;
+use core::fmt;
 
 /// Register units are the smallest units of register allocation.
 ///
diff --git a/lib/codegen/src/isa/riscv/abi.rs b/lib/codegen/src/isa/riscv/abi.rs
index cd33de404e..59b266f7ca 100644
--- a/lib/codegen/src/isa/riscv/abi.rs
+++ b/lib/codegen/src/isa/riscv/abi.rs
@@ -11,7 +11,7 @@ use crate::abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion};
 use crate::ir::{self, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, Type};
 use crate::isa::RegClass;
 use crate::regalloc::RegisterSet;
-use std::i32;
+use core::i32;
 use target_lexicon::Triple;
 
 struct Args {
diff --git a/lib/codegen/src/isa/riscv/binemit.rs b/lib/codegen/src/isa/riscv/binemit.rs
index cd153f4ac1..64f3c00298 100644
--- a/lib/codegen/src/isa/riscv/binemit.rs
+++ b/lib/codegen/src/isa/riscv/binemit.rs
@@ -5,7 +5,7 @@ use crate::ir::{Function, Inst, InstructionData};
 use crate::isa::{RegUnit, StackBaseMask, StackRef};
 use crate::predicates::is_signed_int;
 use crate::regalloc::RegDiversions;
-use std::u32;
+use core::u32;
 
 include!(concat!(env!("OUT_DIR"), "/binemit-riscv.rs"));
 
diff --git a/lib/codegen/src/isa/riscv/mod.rs b/lib/codegen/src/isa/riscv/mod.rs
index 7382809893..60154cf162 100644
--- a/lib/codegen/src/isa/riscv/mod.rs
+++ b/lib/codegen/src/isa/riscv/mod.rs
@@ -15,8 +15,8 @@ use crate::isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encoding
 use crate::isa::Builder as IsaBuilder;
 use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
+use core::fmt;
 use std::boxed::Box;
-use std::fmt;
 use target_lexicon::{PointerWidth, Triple};
 
 #[allow(dead_code)]
@@ -129,7 +129,7 @@ mod tests {
     use crate::ir::{Function, InstructionData, Opcode};
     use crate::isa;
     use crate::settings::{self, Configurable};
-    use std::str::FromStr;
+    use core::str::FromStr;
     use std::string::{String, ToString};
     use target_lexicon::triple;
 
diff --git a/lib/codegen/src/isa/riscv/settings.rs b/lib/codegen/src/isa/riscv/settings.rs
index 2530c4a860..660c1ef669 100644
--- a/lib/codegen/src/isa/riscv/settings.rs
+++ b/lib/codegen/src/isa/riscv/settings.rs
@@ -1,7 +1,7 @@
 //! RISC-V Settings.
 
 use crate::settings::{self, detail, Builder};
-use std::fmt;
+use core::fmt;
 
 // Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
 // `Flags` struct with an impl for all of the settings defined in
diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs
index fd09a9511c..39852fce90 100644
--- a/lib/codegen/src/isa/x86/abi.rs
+++ b/lib/codegen/src/isa/x86/abi.rs
@@ -14,7 +14,7 @@ use crate::isa::{CallConv, RegClass, RegUnit, TargetIsa};
 use crate::regalloc::RegisterSet;
 use crate::result::CodegenResult;
 use crate::stack_layout::layout_stack;
-use std::i32;
+use core::i32;
 use target_lexicon::{PointerWidth, Triple};
 
 /// Argument registers for x86-64
diff --git a/lib/codegen/src/isa/x86/mod.rs b/lib/codegen/src/isa/x86/mod.rs
index 10ec0e5269..acefe0e786 100644
--- a/lib/codegen/src/isa/x86/mod.rs
+++ b/lib/codegen/src/isa/x86/mod.rs
@@ -17,8 +17,8 @@ use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
 use crate::result::CodegenResult;
 use crate::timing;
+use core::fmt;
 use std::boxed::Box;
-use std::fmt;
 use target_lexicon::{PointerWidth, Triple};
 
 #[allow(dead_code)]
diff --git a/lib/codegen/src/isa/x86/settings.rs b/lib/codegen/src/isa/x86/settings.rs
index 52eccaf9f8..b638ea7ee8 100644
--- a/lib/codegen/src/isa/x86/settings.rs
+++ b/lib/codegen/src/isa/x86/settings.rs
@@ -1,7 +1,7 @@
 //! x86 Settings.
 
 use crate::settings::{self, detail, Builder};
-use std::fmt;
+use core::fmt;
 
 // Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
 // `Flags` struct with an impl for all of the settings defined in
diff --git a/lib/codegen/src/legalizer/split.rs b/lib/codegen/src/legalizer/split.rs
index 9e27ae2397..67d947f4a9 100644
--- a/lib/codegen/src/legalizer/split.rs
+++ b/lib/codegen/src/legalizer/split.rs
@@ -67,7 +67,7 @@
 use crate::cursor::{Cursor, CursorPosition, FuncCursor};
 use crate::flowgraph::{BasicBlock, ControlFlowGraph};
 use crate::ir::{self, Ebb, Inst, InstBuilder, InstructionData, Opcode, Type, Value, ValueDef};
-use std::iter;
+use core::iter;
 use std::vec::Vec;
 
 /// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values
diff --git a/lib/codegen/src/lib.rs b/lib/codegen/src/lib.rs
index f73167d7fc..48f5a27902 100644
--- a/lib/codegen/src/lib.rs
+++ b/lib/codegen/src/lib.rs
@@ -40,15 +40,20 @@
         clippy::use_self
     )
 )]
-// Turns on no_std and alloc features if std is not available.
-#![cfg_attr(not(feature = "std"), no_std)]
+#![no_std]
 #![cfg_attr(not(feature = "std"), feature(alloc))]
-// TODO: Remove this workaround once https://github.com/rust-lang/rust/issues/27747 is done.
-#![cfg_attr(not(feature = "std"), feature(slice_concat_ext))]
 
 #[cfg(not(feature = "std"))]
 #[macro_use]
-extern crate alloc;
+extern crate alloc as std;
+#[cfg(feature = "std")]
+#[macro_use]
+extern crate std;
+
+#[cfg(not(feature = "std"))]
+use hashmap_core::{map as hash_map, HashMap, HashSet};
+#[cfg(feature = "std")]
+use std::collections::{hash_map, HashMap, HashSet};
 
 pub use crate::context::Context;
 pub use crate::legalizer::legalize_function;
@@ -101,20 +106,5 @@ mod unreachable_code;
 
 pub use crate::result::{CodegenError, CodegenResult};
 
-/// This replaces `std` in builds with `core`.
-#[cfg(not(feature = "std"))]
-mod std {
-    pub use alloc::{boxed, slice, string, vec};
-    pub use core::*;
-    pub mod collections {
-        #[allow(unused_extern_crates)]
-        extern crate hashmap_core;
-
-        pub use self::hashmap_core::map as hash_map;
-        pub use self::hashmap_core::{HashMap, HashSet};
-        pub use alloc::collections::BTreeSet;
-    }
-}
-
 /// Version number of this crate.
 pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/lib/codegen/src/partition_slice.rs b/lib/codegen/src/partition_slice.rs
index b1e0f95f85..a4ee129008 100644
--- a/lib/codegen/src/partition_slice.rs
+++ b/lib/codegen/src/partition_slice.rs
@@ -1,6 +1,6 @@
 //! Rearrange the elements in a slice according to a predicate.
 
-use std::mem;
+use core::mem;
 
 /// Rearrange the elements of the mutable slice `s` such that elements where `p(t)` is true precede
 /// the elements where `p(t)` is false.
diff --git a/lib/codegen/src/print_errors.rs b/lib/codegen/src/print_errors.rs
index c4f48ef300..2361cb714a 100644
--- a/lib/codegen/src/print_errors.rs
+++ b/lib/codegen/src/print_errors.rs
@@ -8,9 +8,9 @@ use crate::isa::TargetIsa;
 use crate::result::CodegenError;
 use crate::verifier::{VerifierError, VerifierErrors};
 use crate::write::{decorate_function, FuncWriter, PlainWriter};
+use core::fmt;
+use core::fmt::Write;
 use std::boxed::Box;
-use std::fmt;
-use std::fmt::Write;
 use std::string::{String, ToString};
 use std::vec::Vec;
 
diff --git a/lib/codegen/src/ref_slice.rs b/lib/codegen/src/ref_slice.rs
index 251b601d00..2fad921cea 100644
--- a/lib/codegen/src/ref_slice.rs
+++ b/lib/codegen/src/ref_slice.rs
@@ -7,7 +7,7 @@
 //!
 //! Despite their using an unsafe block, these functions are completely safe.
 
-use std::slice;
+use core::slice;
 
 pub fn ref_slice(s: &T) -> &[T] {
     unsafe { slice::from_raw_parts(s, 1) }
diff --git a/lib/codegen/src/regalloc/affinity.rs b/lib/codegen/src/regalloc/affinity.rs
index 39d23b31bb..bbd29f8f2e 100644
--- a/lib/codegen/src/regalloc/affinity.rs
+++ b/lib/codegen/src/regalloc/affinity.rs
@@ -10,7 +10,7 @@
 
 use crate::ir::{AbiParam, ArgumentLoc};
 use crate::isa::{ConstraintKind, OperandConstraint, RegClassIndex, RegInfo, TargetIsa};
-use std::fmt;
+use core::fmt;
 
 /// Preferred register allocation for an SSA value.
 #[derive(Clone, Copy, Debug)]
diff --git a/lib/codegen/src/regalloc/coalescing.rs b/lib/codegen/src/regalloc/coalescing.rs
index cf290e1f31..f59435d170 100644
--- a/lib/codegen/src/regalloc/coalescing.rs
+++ b/lib/codegen/src/regalloc/coalescing.rs
@@ -17,11 +17,11 @@ use crate::regalloc::affinity::Affinity;
 use crate::regalloc::liveness::Liveness;
 use crate::regalloc::virtregs::{VirtReg, VirtRegs};
 use crate::timing;
+use core::cmp;
+use core::fmt;
+use core::iter;
+use core::slice;
 use log::debug;
-use std::cmp;
-use std::fmt;
-use std::iter;
-use std::slice;
 use std::vec::Vec;
 
 // # Implementation
diff --git a/lib/codegen/src/regalloc/coloring.rs b/lib/codegen/src/regalloc/coloring.rs
index 566dd7dee3..12ef018329 100644
--- a/lib/codegen/src/regalloc/coloring.rs
+++ b/lib/codegen/src/regalloc/coloring.rs
@@ -57,8 +57,8 @@ use crate::regalloc::register_set::RegisterSet;
 use crate::regalloc::solver::{Solver, SolverError};
 use crate::regalloc::RegDiversions;
 use crate::timing;
+use core::mem;
 use log::debug;
-use std::mem;
 
 /// Data structures for the coloring pass.
 ///
diff --git a/lib/codegen/src/regalloc/diversion.rs b/lib/codegen/src/regalloc/diversion.rs
index 516f95c610..6e9b1f23cd 100644
--- a/lib/codegen/src/regalloc/diversion.rs
+++ b/lib/codegen/src/regalloc/diversion.rs
@@ -8,11 +8,11 @@
 //! EBB.
 
 use crate::fx::FxHashMap;
+use crate::hash_map::{Entry, Iter};
 use crate::ir::{InstructionData, Opcode};
 use crate::ir::{StackSlot, Value, ValueLoc, ValueLocations};
 use crate::isa::{RegInfo, RegUnit};
-use std::collections::hash_map::{Entry, Iter};
-use std::fmt;
+use core::fmt;
 
 /// A diversion of a value from its original location to a new register or stack location.
 ///
diff --git a/lib/codegen/src/regalloc/liveness.rs b/lib/codegen/src/regalloc/liveness.rs
index 186aaadd60..5e21c14c90 100644
--- a/lib/codegen/src/regalloc/liveness.rs
+++ b/lib/codegen/src/regalloc/liveness.rs
@@ -183,8 +183,8 @@ use crate::isa::{EncInfo, OperandConstraint, TargetIsa};
 use crate::regalloc::affinity::Affinity;
 use crate::regalloc::liverange::{LiveRange, LiveRangeContext, LiveRangeForest};
 use crate::timing;
-use std::mem;
-use std::ops::Index;
+use core::mem;
+use core::ops::Index;
 use std::vec::Vec;
 
 /// A set of live ranges, indexed by value number.
diff --git a/lib/codegen/src/regalloc/liverange.rs b/lib/codegen/src/regalloc/liverange.rs
index a1012ef1c4..5629ea40ec 100644
--- a/lib/codegen/src/regalloc/liverange.rs
+++ b/lib/codegen/src/regalloc/liverange.rs
@@ -111,8 +111,8 @@ use crate::bforest;
 use crate::entity::SparseMapValue;
 use crate::ir::{Ebb, ExpandedProgramPoint, Inst, Layout, ProgramOrder, ProgramPoint, Value};
 use crate::regalloc::affinity::Affinity;
-use std::cmp::Ordering;
-use std::marker::PhantomData;
+use core::cmp::Ordering;
+use core::marker::PhantomData;
 
 /// Global live range of a single SSA value.
 ///
@@ -461,7 +461,7 @@ mod tests {
     use crate::entity::EntityRef;
     use crate::ir::{Ebb, Inst, Value};
     use crate::ir::{ExpandedProgramPoint, ProgramOrder};
-    use std::cmp::Ordering;
+    use core::cmp::Ordering;
     use std::vec::Vec;
 
     // Dummy program order which simply compares indexes.
diff --git a/lib/codegen/src/regalloc/pressure.rs b/lib/codegen/src/regalloc/pressure.rs
index 1e45942585..cd1ef956a5 100644
--- a/lib/codegen/src/regalloc/pressure.rs
+++ b/lib/codegen/src/regalloc/pressure.rs
@@ -38,9 +38,9 @@
 
 use crate::isa::registers::{RegClass, RegClassMask, RegInfo, MAX_TRACKED_TOPRCS};
 use crate::regalloc::RegisterSet;
-use std::cmp::min;
-use std::fmt;
-use std::iter::ExactSizeIterator;
+use core::cmp::min;
+use core::fmt;
+use core::iter::ExactSizeIterator;
 
 /// Information per top-level register class.
 ///
@@ -275,9 +275,9 @@ mod tests {
     use super::Pressure;
     use crate::isa::{RegClass, TargetIsa};
     use crate::regalloc::RegisterSet;
-    use std::borrow::Borrow;
+    use core::borrow::Borrow;
+    use core::str::FromStr;
     use std::boxed::Box;
-    use std::str::FromStr;
     use target_lexicon::triple;
 
     // Make an arm32 `TargetIsa`, if possible.
diff --git a/lib/codegen/src/regalloc/register_set.rs b/lib/codegen/src/regalloc/register_set.rs
index df73a3cda0..29b6df7625 100644
--- a/lib/codegen/src/regalloc/register_set.rs
+++ b/lib/codegen/src/regalloc/register_set.rs
@@ -6,10 +6,10 @@
 //! share a register unit can't be in use at the same time.
 
 use crate::isa::registers::{RegClass, RegInfo, RegUnit, RegUnitMask};
-use std::char;
-use std::fmt;
-use std::iter::ExactSizeIterator;
-use std::mem::size_of_val;
+use core::char;
+use core::fmt;
+use core::iter::ExactSizeIterator;
+use core::mem::size_of_val;
 
 /// Set of registers available for allocation.
 #[derive(Clone)]
diff --git a/lib/codegen/src/regalloc/solver.rs b/lib/codegen/src/regalloc/solver.rs
index ceb6cdfb4b..be1c9a2d27 100644
--- a/lib/codegen/src/regalloc/solver.rs
+++ b/lib/codegen/src/regalloc/solver.rs
@@ -104,11 +104,11 @@ use crate::entity::{SparseMap, SparseMapValue};
 use crate::ir::Value;
 use crate::isa::{RegClass, RegUnit};
 use crate::regalloc::register_set::RegSetIter;
+use core::cmp;
+use core::fmt;
+use core::mem;
+use core::u16;
 use log::debug;
-use std::cmp;
-use std::fmt;
-use std::mem;
-use std::u16;
 use std::vec::Vec;
 
 /// A variable in the constraint problem.
@@ -1134,8 +1134,8 @@ mod tests {
     use crate::ir::Value;
     use crate::isa::{RegClass, RegInfo, RegUnit, TargetIsa};
     use crate::regalloc::RegisterSet;
+    use core::str::FromStr;
     use std::boxed::Box;
-    use std::str::FromStr;
     use target_lexicon::triple;
 
     // Make an arm32 `TargetIsa`, if possible.
diff --git a/lib/codegen/src/regalloc/spilling.rs b/lib/codegen/src/regalloc/spilling.rs
index 7bbf26422c..61458f4066 100644
--- a/lib/codegen/src/regalloc/spilling.rs
+++ b/lib/codegen/src/regalloc/spilling.rs
@@ -27,8 +27,8 @@ use crate::regalloc::pressure::Pressure;
 use crate::regalloc::virtregs::VirtRegs;
 use crate::timing;
 use crate::topo_order::TopoOrder;
+use core::fmt;
 use log::debug;
-use std::fmt;
 use std::vec::Vec;
 
 /// Return a top-level register class which contains `unit`.
diff --git a/lib/codegen/src/regalloc/virtregs.rs b/lib/codegen/src/regalloc/virtregs.rs
index 3fe55474de..584ad9c53a 100644
--- a/lib/codegen/src/regalloc/virtregs.rs
+++ b/lib/codegen/src/regalloc/virtregs.rs
@@ -19,8 +19,8 @@ use crate::entity::{Keys, PrimaryMap, SecondaryMap};
 use crate::ir::{Function, Value};
 use crate::packed_option::PackedOption;
 use crate::ref_slice::ref_slice;
-use std::cmp::Ordering;
-use std::fmt;
+use core::cmp::Ordering;
+use core::fmt;
 use std::vec::Vec;
 
 /// A virtual register reference.
diff --git a/lib/codegen/src/scoped_hash_map.rs b/lib/codegen/src/scoped_hash_map.rs
index 97d073151b..fe22e26ce3 100644
--- a/lib/codegen/src/scoped_hash_map.rs
+++ b/lib/codegen/src/scoped_hash_map.rs
@@ -1,13 +1,12 @@
 //! `ScopedHashMap`
 //!
-//! This module defines a struct `ScopedHashMap` which defines a `HashMap`-like
+//! This module defines a struct `ScopedHashMap` which defines a `FxHashMap`-like
 //! container that has a concept of scopes that can be entered and exited, such that
 //! values inserted while inside a scope aren't visible outside the scope.
 
 use crate::fx::FxHashMap;
-use std::collections::hash_map;
-use std::hash::Hash;
-use std::mem;
+use core::hash::Hash;
+use core::mem;
 
 struct Val {
     value: V,
@@ -17,7 +16,7 @@ struct Val {
 
 /// A view into an occupied entry in a `ScopedHashMap`. It is part of the `Entry` enum.
 pub struct OccupiedEntry<'a, K: 'a, V: 'a> {
-    entry: hash_map::OccupiedEntry<'a, K, Val>,
+    entry: super::hash_map::OccupiedEntry<'a, K, Val>,
 }
 
 impl<'a, K, V> OccupiedEntry<'a, K, V> {
@@ -29,7 +28,7 @@ impl<'a, K, V> OccupiedEntry<'a, K, V> {
 
 /// A view into a vacant entry in a `ScopedHashMap`. It is part of the `Entry` enum.
 pub struct VacantEntry<'a, K: 'a, V: 'a> {
-    entry: hash_map::VacantEntry<'a, K, Val>,
+    entry: super::hash_map::VacantEntry<'a, K, Val>,
     next_key: Option,
     depth: usize,
 }
@@ -53,7 +52,7 @@ pub enum Entry<'a, K: 'a, V: 'a> {
     Vacant(VacantEntry<'a, K, V>),
 }
 
-/// A wrapper around a `HashMap` which adds the concept of scopes. Items inserted
+/// A wrapper around a `FxHashMap` which adds the concept of scopes. Items inserted
 /// within a scope are removed when the scope is exited.
 ///
 /// Shadowing, where one scope has entries with the same keys as a containing scope,
@@ -77,10 +76,10 @@ where
         }
     }
 
-    /// Similar to `HashMap::entry`, gets the given key's corresponding entry in the map for
+    /// Similar to `FxHashMap::entry`, gets the given key's corresponding entry in the map for
     /// in-place manipulation.
     pub fn entry(&mut self, key: K) -> Entry {
-        use self::hash_map::Entry::*;
+        use super::hash_map::Entry::*;
         match self.map.entry(key) {
             Occupied(entry) => Entry::Occupied(OccupiedEntry { entry }),
             Vacant(entry) => {
@@ -104,7 +103,7 @@ where
     pub fn decrement_depth(&mut self) {
         // Remove all elements inserted at the current depth.
         while let Some(key) = self.last_insert.clone() {
-            use self::hash_map::Entry::*;
+            use crate::hash_map::Entry::*;
             match self.map.entry(key) {
                 Occupied(entry) => {
                     if entry.get().depth != self.current_depth {
diff --git a/lib/codegen/src/settings.rs b/lib/codegen/src/settings.rs
index a8ce0166f3..d0f5d5b13e 100644
--- a/lib/codegen/src/settings.rs
+++ b/lib/codegen/src/settings.rs
@@ -22,16 +22,12 @@
 
 use crate::constant_hash::{probe, simple_hash};
 use crate::isa::TargetIsa;
+use core::fmt;
+use core::str;
 use failure_derive::Fail;
 use std::boxed::Box;
-use std::fmt;
-use std::str;
 use std::string::{String, ToString};
 
-// TODO: Remove this workaround once https://github.com/rust-lang/rust/issues/27747 is done.
-#[cfg(not(feature = "std"))]
-use std::slice::SliceConcatExt;
-
 /// A string-based configurator for settings groups.
 ///
 /// The `Configurable` protocol allows settings to be modified by name before a finished `Flags`
@@ -111,10 +107,21 @@ fn parse_bool_value(value: &str) -> SetResult {
 fn parse_enum_value(value: &str, choices: &[&str]) -> SetResult {
     match choices.iter().position(|&tag| tag == value) {
         Some(idx) => Ok(idx as u8),
-        None => Err(SetError::BadValue(format!(
-            "any among {}",
-            choices.join(", ")
-        ))),
+        None => {
+            // TODO: Use `join` instead of this code, once
+            // https://github.com/rust-lang/rust/issues/27747 is resolved.
+            let mut all_choices = String::new();
+            let mut first = true;
+            for choice in choices {
+                if first {
+                    first = false
+                } else {
+                    all_choices += ", ";
+                }
+                all_choices += choice;
+            }
+            Err(SetError::BadValue(format!("any among {}", all_choices)))
+        }
     }
 }
 
@@ -204,7 +211,7 @@ impl<'a> PredicateView<'a> {
 /// code in other modules.
 pub mod detail {
     use crate::constant_hash;
-    use std::fmt;
+    use core::fmt;
 
     /// An instruction group template.
     pub struct Template {
diff --git a/lib/codegen/src/simple_gvn.rs b/lib/codegen/src/simple_gvn.rs
index 576934071d..21a0b9beb9 100644
--- a/lib/codegen/src/simple_gvn.rs
+++ b/lib/codegen/src/simple_gvn.rs
@@ -5,8 +5,8 @@ use crate::dominator_tree::DominatorTree;
 use crate::ir::{Function, Inst, InstructionData, Opcode, Type};
 use crate::scoped_hash_map::ScopedHashMap;
 use crate::timing;
-use std::cell::{Ref, RefCell};
-use std::hash::{Hash, Hasher};
+use core::cell::{Ref, RefCell};
+use core::hash::{Hash, Hasher};
 use std::vec::Vec;
 
 /// Test whether the given opcode is unsafe to even consider for GVN.
diff --git a/lib/codegen/src/stack_layout.rs b/lib/codegen/src/stack_layout.rs
index 54eff6a759..0f5c8ae639 100644
--- a/lib/codegen/src/stack_layout.rs
+++ b/lib/codegen/src/stack_layout.rs
@@ -3,7 +3,7 @@
 use crate::ir::stackslot::{StackOffset, StackSize, StackSlotKind};
 use crate::ir::StackSlots;
 use crate::result::{CodegenError, CodegenResult};
-use std::cmp::{max, min};
+use core::cmp::{max, min};
 
 /// Compute the stack frame layout.
 ///
diff --git a/lib/codegen/src/timing.rs b/lib/codegen/src/timing.rs
index b3aaab1f41..16f729f71d 100644
--- a/lib/codegen/src/timing.rs
+++ b/lib/codegen/src/timing.rs
@@ -2,7 +2,7 @@
 //!
 //! This modules provides facilities for timing the execution of individual compilation passes.
 
-use std::fmt;
+use core::fmt;
 
 pub use self::details::{add_to_current, take_current, PassTimes, TimingToken};
 
diff --git a/lib/codegen/src/topo_order.rs b/lib/codegen/src/topo_order.rs
index 9486b054f4..7dcec7274b 100644
--- a/lib/codegen/src/topo_order.rs
+++ b/lib/codegen/src/topo_order.rs
@@ -94,7 +94,7 @@ mod tests {
     use crate::dominator_tree::DominatorTree;
     use crate::flowgraph::ControlFlowGraph;
     use crate::ir::{Function, InstBuilder};
-    use std::iter;
+    use core::iter;
 
     #[test]
     fn empty() {
diff --git a/lib/codegen/src/verifier/liveness.rs b/lib/codegen/src/verifier/liveness.rs
index c77ac82e16..0d18504d31 100644
--- a/lib/codegen/src/verifier/liveness.rs
+++ b/lib/codegen/src/verifier/liveness.rs
@@ -8,7 +8,7 @@ use crate::regalloc::liveness::Liveness;
 use crate::regalloc::liverange::LiveRange;
 use crate::timing;
 use crate::verifier::{VerifierErrors, VerifierStepResult};
-use std::cmp::Ordering;
+use core::cmp::Ordering;
 
 /// Verify liveness information for `func`.
 ///
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index 1268ed489f..d139c54681 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -72,10 +72,10 @@ use crate::isa::TargetIsa;
 use crate::iterators::IteratorExtras;
 use crate::settings::FlagsOrIsa;
 use crate::timing;
+use core::cmp::Ordering;
+use core::fmt::{self, Display, Formatter, Write};
 use failure_derive::Fail;
-use std::cmp::Ordering;
 use std::collections::BTreeSet;
-use std::fmt::{self, Display, Formatter, Write};
 use std::string::String;
 use std::vec::Vec;
 
diff --git a/lib/codegen/src/write.rs b/lib/codegen/src/write.rs
index c01e7b8bd4..7731918c59 100644
--- a/lib/codegen/src/write.rs
+++ b/lib/codegen/src/write.rs
@@ -8,7 +8,7 @@ use crate::ir::entities::AnyEntity;
 use crate::ir::{DataFlowGraph, Ebb, Function, Inst, SigRef, Type, Value, ValueDef};
 use crate::isa::{RegInfo, TargetIsa};
 use crate::packed_option::ReservedValue;
-use std::fmt::{self, Write};
+use core::fmt::{self, Write};
 use std::string::String;
 use std::vec::Vec;
 
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index ca28428af9..798a3caacf 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -14,6 +14,7 @@ edition = "2018"
 [features]
 default = ["std"]
 std = []
+core = []
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/entity/src/boxed_slice.rs b/lib/entity/src/boxed_slice.rs
index e619238387..2030c6b536 100644
--- a/lib/entity/src/boxed_slice.rs
+++ b/lib/entity/src/boxed_slice.rs
@@ -3,9 +3,10 @@
 use crate::iter::{Iter, IterMut};
 use crate::keys::Keys;
 use crate::EntityRef;
-use std::marker::PhantomData;
-use std::ops::{Index, IndexMut};
-use std::slice;
+use core::marker::PhantomData;
+use core::ops::{Index, IndexMut};
+use core::slice;
+use std::boxed::Box;
 
 /// A slice mapping `K -> V` allocating dense entity references.
 ///
@@ -140,6 +141,7 @@ where
 mod tests {
     use super::*;
     use crate::primary::PrimaryMap;
+    use std::vec::Vec;
 
     // `EntityRef` impl for testing.
     #[derive(Clone, Copy, Debug, PartialEq, Eq)]
diff --git a/lib/entity/src/iter.rs b/lib/entity/src/iter.rs
index 27053a0962..8c681023d2 100644
--- a/lib/entity/src/iter.rs
+++ b/lib/entity/src/iter.rs
@@ -1,9 +1,9 @@
 //! A double-ended iterator over entity references and entities.
 
 use crate::EntityRef;
-use std::iter::Enumerate;
-use std::marker::PhantomData;
-use std::slice;
+use core::iter::Enumerate;
+use core::marker::PhantomData;
+use core::slice;
 
 /// Iterate over all keys in order.
 pub struct Iter<'a, K: EntityRef, V>
diff --git a/lib/entity/src/keys.rs b/lib/entity/src/keys.rs
index ea5bd89441..bfbaa0cb90 100644
--- a/lib/entity/src/keys.rs
+++ b/lib/entity/src/keys.rs
@@ -1,10 +1,10 @@
 //! A double-ended iterator over entity references.
 //!
-//! When `std::iter::Step` is stabilized, `Keys` could be implemented as a wrapper around
-//! `std::ops::Range`, but for now, we implement it manually.
+//! When `core::iter::Step` is stabilized, `Keys` could be implemented as a wrapper around
+//! `core::ops::Range`, but for now, we implement it manually.
 
 use crate::EntityRef;
-use std::marker::PhantomData;
+use core::marker::PhantomData;
 
 /// Iterate over all keys in order.
 pub struct Keys {
diff --git a/lib/entity/src/lib.rs b/lib/entity/src/lib.rs
index ce34c1e50f..8d5579000b 100644
--- a/lib/entity/src/lib.rs
+++ b/lib/entity/src/lib.rs
@@ -50,17 +50,15 @@
         clippy::use_self
     )
 )]
-// Turns on no_std and alloc features if std is not available.
-#![cfg_attr(not(feature = "std"), no_std)]
+#![no_std]
 #![cfg_attr(not(feature = "std"), feature(alloc))]
 
-/// This replaces `std` in builds with `core`.
 #[cfg(not(feature = "std"))]
-mod std {
-    extern crate alloc;
-    pub use self::alloc::{boxed, string, vec};
-    pub use core::*;
-}
+#[macro_use]
+extern crate alloc as std;
+#[cfg(feature = "std")]
+#[macro_use]
+extern crate std;
 
 // Re-export core so that the macros works with both std and no_std crates
 #[doc(hidden)]
diff --git a/lib/entity/src/list.rs b/lib/entity/src/list.rs
index 4dc3ce575d..009b3d70b1 100644
--- a/lib/entity/src/list.rs
+++ b/lib/entity/src/list.rs
@@ -1,8 +1,8 @@
 //! Small lists of entity references.
 use crate::packed_option::ReservedValue;
 use crate::EntityRef;
-use std::marker::PhantomData;
-use std::mem;
+use core::marker::PhantomData;
+use core::mem;
 use std::vec::Vec;
 
 /// A small list of entity references allocated from a pool.
diff --git a/lib/entity/src/map.rs b/lib/entity/src/map.rs
index e0f331106b..9410cc2f90 100644
--- a/lib/entity/src/map.rs
+++ b/lib/entity/src/map.rs
@@ -3,9 +3,9 @@
 use crate::iter::{Iter, IterMut};
 use crate::keys::Keys;
 use crate::EntityRef;
-use std::marker::PhantomData;
-use std::ops::{Index, IndexMut};
-use std::slice;
+use core::marker::PhantomData;
+use core::ops::{Index, IndexMut};
+use core::slice;
 use std::vec::Vec;
 
 /// A mapping `K -> V` for densely indexed entity references.
diff --git a/lib/entity/src/packed_option.rs b/lib/entity/src/packed_option.rs
index cf246a2498..0757e9e19b 100644
--- a/lib/entity/src/packed_option.rs
+++ b/lib/entity/src/packed_option.rs
@@ -7,8 +7,8 @@
 //! This module provides a `PackedOption` for types that have a reserved value that can be used
 //! to represent `None`.
 
-use std::fmt;
-use std::mem;
+use core::fmt;
+use core::mem;
 
 /// Types that have a reserved value which can't be created any other way.
 pub trait ReservedValue: Eq {
diff --git a/lib/entity/src/primary.rs b/lib/entity/src/primary.rs
index 8f1daed7e3..d3fc7470c0 100644
--- a/lib/entity/src/primary.rs
+++ b/lib/entity/src/primary.rs
@@ -3,10 +3,11 @@ use crate::boxed_slice::BoxedSlice;
 use crate::iter::{Iter, IterMut};
 use crate::keys::Keys;
 use crate::EntityRef;
-use std::iter::FromIterator;
-use std::marker::PhantomData;
-use std::ops::{Index, IndexMut};
-use std::slice;
+use core::iter::FromIterator;
+use core::marker::PhantomData;
+use core::ops::{Index, IndexMut};
+use core::slice;
+use std::boxed::Box;
 use std::vec::Vec;
 
 /// A primary mapping `K -> V` allocating dense entity references.
diff --git a/lib/entity/src/set.rs b/lib/entity/src/set.rs
index 80765130e6..f89e96538a 100644
--- a/lib/entity/src/set.rs
+++ b/lib/entity/src/set.rs
@@ -2,7 +2,7 @@
 
 use crate::keys::Keys;
 use crate::EntityRef;
-use std::marker::PhantomData;
+use core::marker::PhantomData;
 use std::vec::Vec;
 
 /// A set of `K` for densely indexed entity references.
@@ -80,7 +80,7 @@ where
 #[cfg(test)]
 mod tests {
     use super::*;
-    use std::u32;
+    use core::u32;
 
     // `EntityRef` impl for testing.
     #[derive(Clone, Copy, Debug, PartialEq, Eq)]
diff --git a/lib/entity/src/sparse.rs b/lib/entity/src/sparse.rs
index a6a43db5e0..83c519f47f 100644
--- a/lib/entity/src/sparse.rs
+++ b/lib/entity/src/sparse.rs
@@ -9,9 +9,9 @@
 
 use crate::map::SecondaryMap;
 use crate::EntityRef;
-use std::mem;
-use std::slice;
-use std::u32;
+use core::mem;
+use core::slice;
+use core::u32;
 use std::vec::Vec;
 
 /// Trait for extracting keys from values stored in a `SparseMap`.
diff --git a/lib/frontend/src/frontend.rs b/lib/frontend/src/frontend.rs
index 06fbbecec5..c1b3e52114 100644
--- a/lib/frontend/src/frontend.rs
+++ b/lib/frontend/src/frontend.rs
@@ -608,7 +608,7 @@ impl<'a> FunctionBuilder<'a> {
             "`size` is not a power of two"
         );
         assert!(
-            access_size >= u64::from(::std::cmp::min(src_align, dest_align)),
+            access_size >= u64::from(::core::cmp::min(src_align, dest_align)),
             "`size` is smaller than `dest` and `src`'s alignment value."
         );
 
@@ -777,7 +777,7 @@ impl<'a> FunctionBuilder<'a> {
             "`size` is not a power of two"
         );
         assert!(
-            access_size >= u64::from(::std::cmp::min(src_align, dest_align)),
+            access_size >= u64::from(::core::cmp::min(src_align, dest_align)),
             "`size` is smaller than `dest` and `src`'s alignment value."
         );
         let load_and_store_amount = size / access_size;
@@ -962,8 +962,8 @@ mod tests {
 
     #[test]
     fn memcpy() {
+        use core::str::FromStr;
         use cranelift_codegen::{isa, settings};
-        use std::str::FromStr;
 
         let shared_builder = settings::builder();
         let shared_flags = settings::Flags::new(shared_builder);
@@ -1023,8 +1023,8 @@ ebb0:
 
     #[test]
     fn small_memcpy() {
+        use core::str::FromStr;
         use cranelift_codegen::{isa, settings};
-        use std::str::FromStr;
 
         let shared_builder = settings::builder();
         let shared_flags = settings::Flags::new(shared_builder);
diff --git a/lib/frontend/src/lib.rs b/lib/frontend/src/lib.rs
index 90c5b5d908..7379503769 100644
--- a/lib/frontend/src/lib.rs
+++ b/lib/frontend/src/lib.rs
@@ -174,12 +174,20 @@
         clippy::use_self
     )
 )]
-#![cfg_attr(not(feature = "std"), no_std)]
+#![no_std]
 #![cfg_attr(not(feature = "std"), feature(alloc))]
 
 #[cfg(not(feature = "std"))]
 #[macro_use]
-extern crate alloc;
+extern crate alloc as std;
+#[cfg(feature = "std")]
+#[macro_use]
+extern crate std;
+
+#[cfg(not(feature = "std"))]
+use hashmap_core::HashMap;
+#[cfg(feature = "std")]
+use std::collections::HashMap;
 
 pub use crate::frontend::{FunctionBuilder, FunctionBuilderContext};
 pub use crate::switch::Switch;
@@ -190,19 +198,5 @@ mod ssa;
 mod switch;
 mod variable;
 
-/// This replaces `std` in builds with `core`.
-#[cfg(not(feature = "std"))]
-mod std {
-    pub use alloc::{string, vec};
-    pub use core::*;
-    pub mod collections {
-        #[allow(unused_extern_crates)]
-        extern crate hashmap_core;
-
-        pub use self::hashmap_core::map as hash_map;
-        pub use self::hashmap_core::{HashMap, HashSet};
-    }
-}
-
 /// Version number of this crate.
 pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/lib/frontend/src/ssa.rs b/lib/frontend/src/ssa.rs
index 1046e960a2..0d48c314af 100644
--- a/lib/frontend/src/ssa.rs
+++ b/lib/frontend/src/ssa.rs
@@ -6,6 +6,8 @@
 //! Lecture Notes in Computer Science, vol 7791. Springer, Berlin, Heidelberg
 
 use crate::Variable;
+use core::mem;
+use core::u32;
 use cranelift_codegen::cursor::{Cursor, FuncCursor};
 use cranelift_codegen::entity::{EntityRef, PrimaryMap, SecondaryMap};
 use cranelift_codegen::ir::immediates::{Ieee32, Ieee64};
@@ -14,8 +16,6 @@ use cranelift_codegen::ir::types::{F32, F64};
 use cranelift_codegen::ir::{Ebb, Function, Inst, InstBuilder, InstructionData, Type, Value};
 use cranelift_codegen::packed_option::PackedOption;
 use cranelift_codegen::packed_option::ReservedValue;
-use std::mem;
-use std::u32;
 use std::vec::Vec;
 
 /// Structure containing the data relevant the construction of SSA for a given function.
diff --git a/lib/frontend/src/switch.rs b/lib/frontend/src/switch.rs
index 5ecb4edc38..653afbaed1 100644
--- a/lib/frontend/src/switch.rs
+++ b/lib/frontend/src/switch.rs
@@ -1,8 +1,8 @@
+use super::HashMap;
 use crate::frontend::FunctionBuilder;
 use cranelift_codegen::ir::condcodes::IntCC;
 use cranelift_codegen::ir::*;
 use log::debug;
-use std::collections::HashMap;
 use std::vec::Vec;
 
 type EntryIndex = u64;
@@ -334,7 +334,7 @@ ebb10:
 
     #[test]
     fn switch_min_index_value() {
-        let func = setup!(0, [::std::i64::MIN as u64, 1,]);
+        let func = setup!(0, [::core::i64::MIN as u64, 1,]);
         assert_eq!(
             func,
             "ebb0:
@@ -350,7 +350,7 @@ ebb10:
 
     #[test]
     fn switch_max_index_value() {
-        let func = setup!(0, [::std::i64::MAX as u64, 1,]);
+        let func = setup!(0, [::core::i64::MAX as u64, 1,]);
         assert_eq!(
             func,
             "ebb0:
diff --git a/lib/frontend/src/variable.rs b/lib/frontend/src/variable.rs
index ad60006dd7..dddcd7490b 100644
--- a/lib/frontend/src/variable.rs
+++ b/lib/frontend/src/variable.rs
@@ -5,8 +5,8 @@
 //! their own index types to use them directly. Frontends which don't
 //! can use the `Variable` defined here.
 
+use core::u32;
 use cranelift_codegen::entity::EntityRef;
-use std::u32;
 
 ///! An opaque reference to a variable.
 #[derive(Copy, Clone, PartialEq, Eq, Debug)]
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index 9a564c8b31..dbd9f95335 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -14,7 +14,7 @@ edition = "2018"
 cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
 cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
-failure = "0.1.1"
+failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
 
 [features]
diff --git a/lib/module/src/backend.rs b/lib/module/src/backend.rs
index 37e235edb7..956c206948 100644
--- a/lib/module/src/backend.rs
+++ b/lib/module/src/backend.rs
@@ -4,10 +4,10 @@ use crate::DataContext;
 use crate::Linkage;
 use crate::ModuleNamespace;
 use crate::ModuleResult;
+use core::marker;
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::Context;
 use cranelift_codegen::{binemit, ir};
-use std::marker;
 
 /// A `Backend` implements the functionality needed to support a `Module`.
 ///
diff --git a/lib/module/src/data_context.rs b/lib/module/src/data_context.rs
index d941e9c411..335dd9a18d 100644
--- a/lib/module/src/data_context.rs
+++ b/lib/module/src/data_context.rs
@@ -61,8 +61,8 @@ impl DataContext {
                 init: Init::Uninitialized,
                 function_decls: PrimaryMap::new(),
                 data_decls: PrimaryMap::new(),
-                function_relocs: Vec::new(),
-                data_relocs: Vec::new(),
+                function_relocs: vec![],
+                data_relocs: vec![],
             },
         }
     }
diff --git a/lib/module/src/lib.rs b/lib/module/src/lib.rs
index db9b793369..ce6f00aff0 100644
--- a/lib/module/src/lib.rs
+++ b/lib/module/src/lib.rs
@@ -18,13 +18,20 @@
         clippy::use_self
     )
 )]
-// Turns on no_std and alloc features if std is not available.
-#![cfg_attr(not(feature = "std"), no_std)]
+#![no_std]
 #![cfg_attr(not(feature = "std"), feature(alloc))]
 
 #[cfg(not(feature = "std"))]
-#[cfg_attr(test, macro_use)]
-extern crate alloc;
+#[macro_use]
+extern crate alloc as std;
+#[cfg(feature = "std")]
+#[macro_use]
+extern crate std;
+
+#[cfg(not(feature = "std"))]
+use hashmap_core::{map as hash_map, HashMap};
+#[cfg(feature = "std")]
+use std::collections::{hash_map, HashMap};
 
 mod backend;
 mod data_context;
@@ -36,19 +43,5 @@ pub use crate::module::{
     DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleError, ModuleNamespace, ModuleResult,
 };
 
-/// This replaces `std` in builds with `core`.
-#[cfg(not(feature = "std"))]
-mod std {
-    pub use alloc::{borrow, boxed, string, vec};
-    pub use core::*;
-    pub mod collections {
-        #[allow(unused_extern_crates)]
-        extern crate hashmap_core;
-
-        pub use self::hashmap_core::map as hash_map;
-        pub use self::hashmap_core::{HashMap, HashSet};
-    }
-}
-
 /// Version number of this crate.
 pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 1d5a33130a..0fec7d39ce 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -5,6 +5,7 @@
 // TODO: Factor out `ir::Function`'s `ext_funcs` and `global_values` into a struct
 // shared with `DataContext`?
 
+use super::HashMap;
 use crate::data_context::DataContext;
 use crate::Backend;
 use cranelift_codegen::entity::{entity_impl, PrimaryMap};
@@ -12,7 +13,6 @@ use cranelift_codegen::{binemit, ir, isa, CodegenError, Context};
 use failure::Fail;
 use log::info;
 use std::borrow::ToOwned;
-use std::collections::HashMap;
 use std::string::String;
 use std::vec::Vec;
 
@@ -398,7 +398,7 @@ where
         signature: &ir::Signature,
     ) -> ModuleResult {
         // TODO: Can we avoid allocating names so often?
-        use std::collections::hash_map::Entry::*;
+        use super::hash_map::Entry::*;
         match self.names.entry(name.to_owned()) {
             Occupied(entry) => match *entry.get() {
                 FuncOrDataId::Func(id) => {
@@ -435,7 +435,7 @@ where
         writable: bool,
     ) -> ModuleResult {
         // TODO: Can we avoid allocating names so often?
-        use std::collections::hash_map::Entry::*;
+        use super::hash_map::Entry::*;
         match self.names.entry(name.to_owned()) {
             Occupied(entry) => match *entry.get() {
                 FuncOrDataId::Data(id) => {
diff --git a/lib/native/src/lib.rs b/lib/native/src/lib.rs
index 4322512ec3..010acb25ac 100644
--- a/lib/native/src/lib.rs
+++ b/lib/native/src/lib.rs
@@ -23,7 +23,7 @@
         clippy::use_self
     )
 )]
-#![cfg_attr(not(feature = "std"), no_std)]
+#![no_std]
 
 use cranelift_codegen::isa;
 use cranelift_codegen::settings::Configurable;
diff --git a/lib/preopt/src/constant_folding.rs b/lib/preopt/src/constant_folding.rs
index 1fb5d116eb..2d7c1c2dbb 100644
--- a/lib/preopt/src/constant_folding.rs
+++ b/lib/preopt/src/constant_folding.rs
@@ -102,7 +102,7 @@ fn resolve_value_to_imm(dfg: &ir::DataFlowGraph, value: ir::Value) -> Option Option {
-    use std::num::Wrapping;
+    use core::num::Wrapping;
 
     match opcode {
         ir::Opcode::Iadd => {
diff --git a/lib/preopt/src/lib.rs b/lib/preopt/src/lib.rs
index b93e914cb2..310d1ba614 100644
--- a/lib/preopt/src/lib.rs
+++ b/lib/preopt/src/lib.rs
@@ -18,11 +18,15 @@
         clippy::use_self
     )
 )]
-#![cfg_attr(not(feature = "std"), no_std)]
+#![no_std]
 #![cfg_attr(not(feature = "std"), feature(alloc))]
 
 #[cfg(not(feature = "std"))]
-extern crate alloc;
+#[macro_use]
+extern crate alloc as std;
+#[cfg(feature = "std")]
+#[macro_use]
+extern crate std;
 
 mod constant_folding;
 
@@ -49,10 +53,3 @@ where
     ctx.verify_if(fisa)?;
     Ok(())
 }
-
-/// This replaces `std` in builds with `core`.
-#[cfg(not(feature = "std"))]
-mod std {
-    pub use alloc::{boxed, slice, string, vec};
-    pub use core::*;
-}
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index a8a34e64b2..24aa976671 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -5,7 +5,6 @@ authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
-categories = ["no-std"]
 readme = "README.md"
 keywords = ["webassembly", "serde"]
 edition = "2018"
@@ -19,13 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
-cranelift-reader = { path = "../reader", version = "0.26.0", default-features = false }
-
-[features]
-default = ["std"]
-std = ["cranelift-codegen/std"]
-core = ["cranelift-codegen/core"]
+cranelift-codegen = { path = "../codegen", version = "0.26.0" }
+cranelift-reader = { path = "../reader", version = "0.26.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 440c4072f1..9fb1d3abd4 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -5,19 +5,18 @@ authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
 documentation = "https://cranelift.readthedocs.io/"
-categories = ["no-std"]
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
-cranelift-module = { path = "../module", version = "0.26.0", default-features = false }
-cranelift-native = { path = "../native", version = "0.26.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.26.0" }
+cranelift-module = { path = "../module", version = "0.26.0" }
+cranelift-native = { path = "../native", version = "0.26.0" }
 region = "1.0.0"
-libc = { version = "0.2.42", default-features = false }
+libc = { version = "0.2.42" }
 errno = "0.2.4"
-target-lexicon = { version = "0.2.0", default-features = false }
+target-lexicon = { version = "0.2.0" }
 
 [target.'cfg(target_os = "windows")'.dependencies]
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
@@ -27,11 +26,6 @@ cranelift = { path = "../umbrella", version = "0.26.0" }
 cranelift-frontend = { path = "../frontend", version = "0.26.0" }
 cranelift-entity = { path = "../entity", version = "0.26.0" }
 
-[features]
-default = ["std"]
-std = ["libc/use_std", "cranelift-codegen/std", "cranelift-module/std", "cranelift-native/std", "target-lexicon/std"]
-core = ["cranelift-codegen/core", "cranelift-module/core", "cranelift-native/core"]
-
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/lib/umbrella/src/lib.rs b/lib/umbrella/src/lib.rs
index 6db0fab1f8..f97ed443ce 100644
--- a/lib/umbrella/src/lib.rs
+++ b/lib/umbrella/src/lib.rs
@@ -22,6 +22,7 @@
         clippy::use_self
     )
 )]
+#![no_std]
 
 /// Provide these crates, renamed to reduce stutter.
 pub use cranelift_codegen as codegen;
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 45c2419bf7..88aae1cecf 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -19,7 +19,7 @@ hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
-cast = { version = "0.2.2" }
+cast = { version = "0.2.2", default-features = false }
 
 [dev-dependencies]
 wabt = "0.7.0"
diff --git a/lib/wasm/src/code_translator.rs b/lib/wasm/src/code_translator.rs
index 43b4af8f9e..ad0d75e2da 100644
--- a/lib/wasm/src/code_translator.rs
+++ b/lib/wasm/src/code_translator.rs
@@ -22,18 +22,17 @@
 //!
 //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as
 //! argument.
+use super::{hash_map, HashMap};
 use crate::environ::{FuncEnvironment, GlobalVariable, ReturnMode, WasmError, WasmResult};
 use crate::state::{ControlStackFrame, TranslationState};
 use crate::translation_utils::{f32_translation, f64_translation, num_return_values, type_to_type};
 use crate::translation_utils::{FuncIndex, MemoryIndex, SignatureIndex, TableIndex};
+use core::{i32, u32};
 use cranelift_codegen::ir::condcodes::{FloatCC, IntCC};
 use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags};
 use cranelift_codegen::packed_option::ReservedValue;
 use cranelift_frontend::{FunctionBuilder, Variable};
-use std::collections::{hash_map, HashMap};
-use std::vec::Vec;
-use std::{i32, u32};
 use wasmparser::{MemoryImmediate, Operator};
 
 // Clippy warns about "flags: _" but its important to document that the flags field is ignored
@@ -288,7 +287,7 @@ pub fn translate_operator(
                 // Here we have jump arguments, but Cranelift's br_table doesn't support them
                 // We then proceed to split the edges going out of the br_table
                 let return_count = jump_args_count;
-                let mut dest_ebb_sequence = Vec::new();
+                let mut dest_ebb_sequence = vec![];
                 let mut dest_ebb_map = HashMap::new();
                 for depth in &*depths {
                     let branch_ebb = match dest_ebb_map.entry(*depth as usize) {
@@ -984,7 +983,7 @@ fn get_heap_addr(
     addr_ty: Type,
     builder: &mut FunctionBuilder,
 ) -> (ir::Value, i32) {
-    use std::cmp::min;
+    use core::cmp::min;
 
     let mut adjusted_offset = u64::from(offset);
     let offset_guard_size: u64 = builder.func.heaps[heap].offset_guard_size.into();
diff --git a/lib/wasm/src/environ/dummy.rs b/lib/wasm/src/environ/dummy.rs
index b52b5bfc1d..28cea89e77 100644
--- a/lib/wasm/src/environ/dummy.rs
+++ b/lib/wasm/src/environ/dummy.rs
@@ -18,6 +18,7 @@ use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::isa::TargetFrontendConfig;
 use cranelift_entity::{EntityRef, PrimaryMap};
+use std::boxed::Box;
 use std::string::String;
 use std::vec::Vec;
 
diff --git a/lib/wasm/src/environ/spec.rs b/lib/wasm/src/environ/spec.rs
index f79152f5be..50ecca02c6 100644
--- a/lib/wasm/src/environ/spec.rs
+++ b/lib/wasm/src/environ/spec.rs
@@ -9,13 +9,13 @@
 use crate::translation_utils::{
     FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
 };
+use core::convert::From;
 use cranelift_codegen::cursor::FuncCursor;
 use cranelift_codegen::ir::immediates::Offset32;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::isa::TargetFrontendConfig;
 use failure_derive::Fail;
 use std::boxed::Box;
-use std::convert::From;
 use wasmparser::BinaryReaderError;
 
 /// The value of a WebAssembly global variable.
diff --git a/lib/wasm/src/lib.rs b/lib/wasm/src/lib.rs
index 4bfaf93012..edd5c3215f 100644
--- a/lib/wasm/src/lib.rs
+++ b/lib/wasm/src/lib.rs
@@ -27,9 +27,21 @@
         clippy::use_self
     )
 )]
-#![cfg_attr(not(feature = "std"), no_std)]
+#![no_std]
 #![cfg_attr(not(feature = "std"), feature(alloc))]
 
+#[cfg(not(feature = "std"))]
+#[macro_use]
+extern crate alloc as std;
+#[cfg(feature = "std")]
+#[macro_use]
+extern crate std;
+
+#[cfg(not(feature = "std"))]
+use hashmap_core::{map as hash_map, HashMap};
+#[cfg(feature = "std")]
+use std::collections::{hash_map, HashMap};
+
 mod code_translator;
 mod environ;
 mod func_translator;
@@ -50,23 +62,5 @@ pub use crate::translation_utils::{
     TableIndex,
 };
 
-#[cfg(not(feature = "std"))]
-mod std {
-    extern crate alloc;
-
-    pub use self::alloc::string;
-    pub use self::alloc::vec;
-    pub use core::convert;
-    pub use core::fmt;
-    pub use core::option;
-    pub use core::{cmp, i32, str, u32};
-    pub mod collections {
-        #[allow(unused_extern_crates)]
-        extern crate hashmap_core;
-
-        pub use self::hashmap_core::{map as hash_map, HashMap};
-    }
-}
-
 /// Version number of this crate.
 pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/lib/wasm/src/sections_translator.rs b/lib/wasm/src/sections_translator.rs
index 0a3aff5803..2c40010169 100644
--- a/lib/wasm/src/sections_translator.rs
+++ b/lib/wasm/src/sections_translator.rs
@@ -12,9 +12,9 @@ use crate::translation_utils::{
     type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex,
     Table, TableElementType, TableIndex,
 };
+use core::str::from_utf8;
 use cranelift_codegen::ir::{self, AbiParam, Signature};
 use cranelift_entity::EntityRef;
-use std::str::from_utf8;
 use std::vec::Vec;
 use wasmparser::{
     self, CodeSectionReader, Data, DataSectionReader, Element, ElementSectionReader, Export,
diff --git a/lib/wasm/src/state.rs b/lib/wasm/src/state.rs
index 80d1f5fe23..f392631237 100644
--- a/lib/wasm/src/state.rs
+++ b/lib/wasm/src/state.rs
@@ -3,10 +3,10 @@
 //! The `TranslationState` struct defined in this module is used to keep track of the WebAssembly
 //! value and control stacks during the translation of a single function.
 
+use super::HashMap;
 use crate::environ::{FuncEnvironment, GlobalVariable};
 use crate::translation_utils::{FuncIndex, GlobalIndex, MemoryIndex, SignatureIndex, TableIndex};
 use cranelift_codegen::ir::{self, Ebb, Inst, Value};
-use std::collections::HashMap;
 use std::vec::Vec;
 
 /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following
diff --git a/lib/wasm/src/translation_utils.rs b/lib/wasm/src/translation_utils.rs
index cdfe276d46..7ea730c9cb 100644
--- a/lib/wasm/src/translation_utils.rs
+++ b/lib/wasm/src/translation_utils.rs
@@ -1,7 +1,7 @@
 //! Helper functions and structures for the translation.
+use core::u32;
 use cranelift_codegen::entity::entity_impl;
 use cranelift_codegen::ir;
-use std::u32;
 use wasmparser;
 
 /// Index type of a function (imported or defined) inside the WebAssembly module.

From eedc7cabf17eecef2d4f3d3cb0da65b55ed9065d Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 9 Jan 2019 14:48:01 +0100
Subject: [PATCH 2300/3084] Allow peeking at the compiled function

---
 lib/module/src/module.rs | 39 ++++++++++++++++++++++++++++-----------
 1 file changed, 28 insertions(+), 11 deletions(-)

diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 0fec7d39ce..1c495927f4 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -506,7 +506,18 @@ where
 
     /// Define a function, producing the function body from the given `Context`.
     pub fn define_function(&mut self, func: FuncId, ctx: &mut Context) -> ModuleResult<()> {
-        let compiled = {
+        self.define_function_peek_compiled(func, ctx, |_, _| ())
+    }
+
+    /// Define a function, allowing to peek at the compiled function and producing the
+    /// function body from the given `Context`.
+    pub fn define_function_peek_compiled(
+        &mut self,
+        func: FuncId,
+        ctx: &mut Context,
+        peek_compiled: impl FnOnce(u32, &Context) -> T,
+    ) -> ModuleResult {
+        let (compiled, peek_res) = {
             let code_size = ctx.compile(self.backend.isa()).map_err(|e| {
                 info!(
                     "defining function {}: {}",
@@ -523,18 +534,24 @@ where
             if !info.decl.linkage.is_definable() {
                 return Err(ModuleError::InvalidImportDefinition(info.decl.name.clone()));
             }
-            Some(self.backend.define_function(
-                &info.decl.name,
-                ctx,
-                &ModuleNamespace:: {
-                    contents: &self.contents,
-                },
-                code_size,
-            )?)
+
+            let peek_res = peek_compiled(code_size, &ctx);
+
+            (
+                self.backend.define_function(
+                    &info.decl.name,
+                    ctx,
+                    &ModuleNamespace:: {
+                        contents: &self.contents,
+                    },
+                    code_size,
+                )?,
+                peek_res,
+            )
         };
-        self.contents.functions[func].compiled = compiled;
+        self.contents.functions[func].compiled = Some(compiled);
         self.functions_to_finalize.push(func);
-        Ok(())
+        Ok(peek_res)
     }
 
     /// Define a function, producing the data contents from the given `DataContext`.

From 80031cb2b4122b230972092021ae65aa12cf5c18 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Fri, 11 Jan 2019 13:39:13 +0100
Subject: [PATCH 2301/3084] Add TargetIsa

---
 lib/module/src/module.rs | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 1c495927f4..85d12f0f45 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -506,7 +506,7 @@ where
 
     /// Define a function, producing the function body from the given `Context`.
     pub fn define_function(&mut self, func: FuncId, ctx: &mut Context) -> ModuleResult<()> {
-        self.define_function_peek_compiled(func, ctx, |_, _| ())
+        self.define_function_peek_compiled(func, ctx, |_, _, _| ())
     }
 
     /// Define a function, allowing to peek at the compiled function and producing the
@@ -515,7 +515,7 @@ where
         &mut self,
         func: FuncId,
         ctx: &mut Context,
-        peek_compiled: impl FnOnce(u32, &Context) -> T,
+        peek_compiled: impl FnOnce(u32, &Context, &isa::TargetIsa) -> T,
     ) -> ModuleResult {
         let (compiled, peek_res) = {
             let code_size = ctx.compile(self.backend.isa()).map_err(|e| {
@@ -535,7 +535,7 @@ where
                 return Err(ModuleError::InvalidImportDefinition(info.decl.name.clone()));
             }
 
-            let peek_res = peek_compiled(code_size, &ctx);
+            let peek_res = peek_compiled(code_size, &ctx, self.backend.isa());
 
             (
                 self.backend.define_function(

From eb032fd0f4c689496c5d58bccb719b81a9677af7 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Tue, 15 Jan 2019 17:30:40 +0100
Subject: [PATCH 2302/3084] Simplify

---
 lib/module/src/module.rs | 30 +++++++++++++-----------------
 1 file changed, 13 insertions(+), 17 deletions(-)

diff --git a/lib/module/src/module.rs b/lib/module/src/module.rs
index 85d12f0f45..b043ee4754 100644
--- a/lib/module/src/module.rs
+++ b/lib/module/src/module.rs
@@ -517,8 +517,9 @@ where
         ctx: &mut Context,
         peek_compiled: impl FnOnce(u32, &Context, &isa::TargetIsa) -> T,
     ) -> ModuleResult {
-        let (compiled, peek_res) = {
-            let code_size = ctx.compile(self.backend.isa()).map_err(|e| {
+        let code_size;
+        let compiled = {
+            code_size = ctx.compile(self.backend.isa()).map_err(|e| {
                 info!(
                     "defining function {}: {}",
                     func,
@@ -535,23 +536,18 @@ where
                 return Err(ModuleError::InvalidImportDefinition(info.decl.name.clone()));
             }
 
-            let peek_res = peek_compiled(code_size, &ctx, self.backend.isa());
-
-            (
-                self.backend.define_function(
-                    &info.decl.name,
-                    ctx,
-                    &ModuleNamespace:: {
-                        contents: &self.contents,
-                    },
-                    code_size,
-                )?,
-                peek_res,
-            )
+            Some(self.backend.define_function(
+                &info.decl.name,
+                ctx,
+                &ModuleNamespace:: {
+                    contents: &self.contents,
+                },
+                code_size,
+            )?)
         };
-        self.contents.functions[func].compiled = Some(compiled);
+        self.contents.functions[func].compiled = compiled;
         self.functions_to_finalize.push(func);
-        Ok(peek_res)
+        Ok(peek_compiled(code_size, &ctx, self.backend.isa()))
     }
 
     /// Define a function, producing the data contents from the given `DataContext`.

From 06a072ead0ee0e4b6d3ca138e814536fdb42a442 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 16 Jan 2019 14:13:21 -0800
Subject: [PATCH 2303/3084] Mention wasmtime as a planned use.

---
 cranelift/README.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cranelift/README.md b/cranelift/README.md
index f8bd1e8c15..51e7f6a264 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -68,6 +68,7 @@ affected its design are:
  - [Backend for the IonMonkey JavaScript JIT compiler in
     Firefox](spidermonkey.md#phase-2-ionmonkey).
  - [Debug build backend for the Rust compiler](rustc.md).
+ - [Wasmtime non-Web wasm engine](https://github.com/CraneStation/wasmtime).
 
 Building Cranelift
 ------------------

From 363eea6960f3c4f78cf48b7e6552b4b63298ff8a Mon Sep 17 00:00:00 2001
From: Nathan Froyd 
Date: Tue, 22 Jan 2019 15:58:22 -0500
Subject: [PATCH 2304/3084] avoid directories for cargo build dependencies

We are facing peculiar Windows-only regressions in build times in
https://bugzilla.mozilla.org/show_bug.cgi?id=1506511 and while the build
times might just be Windows being slow, putting directories in
`rerun-if-changed` might also be causing problems.  The build only
depends on the files, anyway, so let's just say that.
---
 lib/codegen/meta-python/gen_build_deps.py | 19 ++++---------------
 1 file changed, 4 insertions(+), 15 deletions(-)

diff --git a/lib/codegen/meta-python/gen_build_deps.py b/lib/codegen/meta-python/gen_build_deps.py
index 03d9ee72d0..8423e72060 100644
--- a/lib/codegen/meta-python/gen_build_deps.py
+++ b/lib/codegen/meta-python/gen_build_deps.py
@@ -22,22 +22,11 @@ except ImportError:
     pass
 
 
-def source_files(top):
-    # type: (str) -> Iterable[str]
-    """
-    Recursively find all interesting source files and directories in the
-    directory tree starting at top. Yield a path to each file.
-    """
-    for (dirpath, dirnames, filenames) in os.walk(top):
-        yield dirpath
-        for f in filenames:
-            if f.endswith('.py'):
-                yield join(dirpath, f)
-
-
 def generate():
     # type: () -> None
     print("Dependencies from meta language directory:")
     meta = dirname(abspath(__file__))
-    for path in source_files(meta):
-        print("cargo:rerun-if-changed=" + path)
+    for (dirpath, _, filenames) in os.walk(meta):
+        for f in filenames:
+            if f.endswith('.py'):
+                print("cargo:rerun-if-changed=" + join(dirpath, f))

From 1fdf1c92d95f085d5363cc7e715bcbe54abd90ad Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 22 Jan 2019 13:27:03 -0800
Subject: [PATCH 2305/3084] Update to the rustfmt in rust 1.32, which is now
 stable.

---
 lib/codegen/src/isa/x86/abi.rs       |  2 +-
 lib/codegen/src/verifier/liveness.rs |  6 +++---
 lib/codegen/src/verifier/mod.rs      |  2 +-
 lib/filetests/src/test_binemit.rs    |  2 +-
 lib/filetests/src/test_domtree.rs    |  2 +-
 lib/reader/src/isaspec.rs            |  2 +-
 lib/reader/src/parser.rs             | 10 +++++-----
 7 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/lib/codegen/src/isa/x86/abi.rs b/lib/codegen/src/isa/x86/abi.rs
index 39852fce90..0321e36e85 100644
--- a/lib/codegen/src/isa/x86/abi.rs
+++ b/lib/codegen/src/isa/x86/abi.rs
@@ -99,7 +99,7 @@ impl ArgAssigner for Args {
                     } else {
                         RU::rsi
                     } as RegUnit)
-                    .into()
+                    .into();
                 }
                 // This is SpiderMonkey's `WasmTableCallSigReg`.
                 ArgumentPurpose::SignatureId => return ArgumentLoc::Reg(RU::r10 as RegUnit).into(),
diff --git a/lib/codegen/src/verifier/liveness.rs b/lib/codegen/src/verifier/liveness.rs
index 0d18504d31..9556ae3703 100644
--- a/lib/codegen/src/verifier/liveness.rs
+++ b/lib/codegen/src/verifier/liveness.rs
@@ -186,7 +186,7 @@ impl<'a> LivenessVerifier<'a> {
                     "Def local range for {} can't end at {}",
                     val,
                     e
-                )
+                );
             }
             ExpandedProgramPoint::Inst(i) => {
                 if self.func.layout.inst_ebb(i) != Some(def_ebb) {
@@ -216,7 +216,7 @@ impl<'a> LivenessVerifier<'a> {
                         val,
                         ebb,
                         end
-                    )
+                    );
                 }
             };
 
@@ -247,7 +247,7 @@ impl<'a> LivenessVerifier<'a> {
                             "end of {} livein ({}) never reached",
                             val,
                             end_ebb
-                        )
+                        );
                     }
                 };
             }
diff --git a/lib/codegen/src/verifier/mod.rs b/lib/codegen/src/verifier/mod.rs
index d139c54681..0aeb3ad8a9 100644
--- a/lib/codegen/src/verifier/mod.rs
+++ b/lib/codegen/src/verifier/mod.rs
@@ -1641,7 +1641,7 @@ impl<'a> Verifier<'a> {
                         "{} must have an encoding (e.g., {})",
                         text,
                         isa.encoding_info().display(enc)
-                    )
+                    );
                 }
                 Err(_) => return nonfatal!(errors, inst, "{} must have an encoding", text),
             }
diff --git a/lib/filetests/src/test_binemit.rs b/lib/filetests/src/test_binemit.rs
index 47fa59a2c0..3c959a09bc 100644
--- a/lib/filetests/src/test_binemit.rs
+++ b/lib/filetests/src/test_binemit.rs
@@ -186,7 +186,7 @@ impl SubTest for TestBinEmit {
                         return Err(format!(
                             "'bin:' directive on non-inst {}: {}",
                             comment.entity, comment.text
-                        ))
+                        ));
                     }
                 }
             }
diff --git a/lib/filetests/src/test_domtree.rs b/lib/filetests/src/test_domtree.rs
index c2683591f2..e8aeca9bfa 100644
--- a/lib/filetests/src/test_domtree.rs
+++ b/lib/filetests/src/test_domtree.rs
@@ -55,7 +55,7 @@ impl SubTest for TestDomtree {
                         return Err(format!(
                             "annotation on non-inst {}: {}",
                             comment.entity, comment.text
-                        ))
+                        ));
                     }
                 };
                 for src_ebb in tail.split_whitespace() {
diff --git a/lib/reader/src/isaspec.rs b/lib/reader/src/isaspec.rs
index 6b58982c3a..89dac510bb 100644
--- a/lib/reader/src/isaspec.rs
+++ b/lib/reader/src/isaspec.rs
@@ -56,7 +56,7 @@ where
                         "invalid setting value for '{}', expected {}",
                         opt,
                         expected
-                    )
+                    );
                 }
             },
         }
diff --git a/lib/reader/src/parser.rs b/lib/reader/src/parser.rs
index 3aefb575f3..4f5acc05f4 100644
--- a/lib/reader/src/parser.rs
+++ b/lib/reader/src/parser.rs
@@ -796,10 +796,10 @@ impl<'a> Parser<'a> {
             };
             let isa_builder = match isa::lookup(triple) {
                 Err(isa::LookupError::SupportDisabled) => {
-                    return err!(loc, "support disabled target '{}'", targ)
+                    return err!(loc, "support disabled target '{}'", targ);
                 }
                 Err(isa::LookupError::Unsupported) => {
-                    return err!(loc, "unsupported target '{}'", targ)
+                    return err!(loc, "unsupported target '{}'", targ);
                 }
                 Ok(b) => b,
             };
@@ -859,7 +859,7 @@ impl<'a> Parser<'a> {
                             continue;
                         }
                         Err(isa::LookupError::Unsupported) => {
-                            return err!(loc, "unsupported target '{}'", target_name)
+                            return err!(loc, "unsupported target '{}'", target_name);
                         }
                         Ok(b) => b,
                     };
@@ -1480,7 +1480,7 @@ impl<'a> Parser<'a> {
             Some(Token::SigRef(sig_src)) => {
                 let sig = match SigRef::with_number(sig_src) {
                     None => {
-                        return err!(self.loc, "attempted to use invalid signature ss{}", sig_src)
+                        return err!(self.loc, "attempted to use invalid signature ss{}", sig_src);
                     }
                     Some(sig) => sig,
                 };
@@ -1712,7 +1712,7 @@ impl<'a> Parser<'a> {
                             self.loc,
                             "attempted to use invalid stack slot ss{}",
                             src_num
-                        )
+                        );
                     }
                     Some(ss) => ss,
                 };

From 1bbd2ebb77b69e6fe2c7b2b62ac999b59a20cf3e Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 22 Jan 2019 20:03:28 -0800
Subject: [PATCH 2306/3084] Update to region 2.0.0.

---
 lib/simplejit/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 9fb1d3abd4..dd448e9990 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -13,7 +13,7 @@ edition = "2018"
 cranelift-codegen = { path = "../codegen", version = "0.26.0" }
 cranelift-module = { path = "../module", version = "0.26.0" }
 cranelift-native = { path = "../native", version = "0.26.0" }
-region = "1.0.0"
+region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
 target-lexicon = { version = "0.2.0" }

From c8febc1a372e1d677913cf120a161f4c02be087b Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 22 Jan 2019 20:06:34 -0800
Subject: [PATCH 2307/3084] Bump version to 0.27.0.

Major API-incompatible changes include:
 - Introduce TrapCode::UnreachableCodeReached, used for unreachable in wasm.
 - cranelift-wasm's `declare_signature` now takes its signature by value
 - cranelift-wasm's `declare_table_elements` `elems` parameter now takes a boxed slice
 - Remove cranelift-wasm's `ModuleEnvironment`'s `get_signature`,
   `get_num_func_imports`, `get_func_type`, and `get_global`.
---
 cranelift/Cargo.toml        | 28 ++++++++++++++--------------
 cranelift/publish-all.sh    |  2 +-
 lib/bforest/Cargo.toml      |  4 ++--
 lib/codegen/Cargo.toml      |  8 ++++----
 lib/codegen/meta/Cargo.toml |  4 ++--
 lib/entity/Cargo.toml       |  2 +-
 lib/faerie/Cargo.toml       |  6 +++---
 lib/filetests/Cargo.toml    |  8 ++++----
 lib/frontend/Cargo.toml     |  4 ++--
 lib/module/Cargo.toml       |  6 +++---
 lib/native/Cargo.toml       |  4 ++--
 lib/preopt/Cargo.toml       |  6 +++---
 lib/reader/Cargo.toml       |  4 ++--
 lib/serde/Cargo.toml        |  6 +++---
 lib/simplejit/Cargo.toml    | 14 +++++++-------
 lib/umbrella/Cargo.toml     |  6 +++---
 lib/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 04c2168578..f696524ba3 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.26.0"
+version = "0.27.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -15,19 +15,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.26.0" }
-cranelift-entity = { path = "lib/entity", version = "0.26.0" }
-cranelift-reader = { path = "lib/reader", version = "0.26.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.26.0" }
-cranelift-serde = { path = "lib/serde", version = "0.26.0", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.26.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.26.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.26.0" }
-cranelift-module = { path = "lib/module", version = "0.26.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.26.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.26.0" }
-cranelift-preopt = { path = "lib/preopt", version = "0.26.0" }
-cranelift = { path = "lib/umbrella", version = "0.26.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.27.0" }
+cranelift-entity = { path = "lib/entity", version = "0.27.0" }
+cranelift-reader = { path = "lib/reader", version = "0.27.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.27.0" }
+cranelift-serde = { path = "lib/serde", version = "0.27.0", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.27.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.27.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.27.0" }
+cranelift-module = { path = "lib/module", version = "0.27.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.27.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.27.0" }
+cranelift-preopt = { path = "lib/preopt", version = "0.27.0" }
+cranelift = { path = "lib/umbrella", version = "0.27.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index ba9f196ce2..bcdc1abc5a 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.26.0"
+version="0.27.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index 36ec8a752c..a6b61db391 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.26.0"
+version = "0.27.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.27.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 6281a77644..2065105d8b 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.26.0"
+version = "0.27.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
-cranelift-bforest = { path = "../bforest", version = "0.26.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.27.0", default-features = false }
+cranelift-bforest = { path = "../bforest", version = "0.27.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -26,7 +26,7 @@ log = { version = "0.4.6", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.26.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.27.0" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 2b443cb52e..4a70592e0f 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.26.0"
+version = "0.27.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../entity", version = "0.26.0" }
+cranelift-entity = { path = "../../entity", version = "0.27.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index 798a3caacf..876a0de960 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.26.0"
+version = "0.27.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 4ef81063e9..66523dc1e5 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.26.0"
+version = "0.27.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.26.0" }
-cranelift-module = { path = "../module", version = "0.26.0" }
+cranelift-codegen = { path = "../codegen", version = "0.27.0" }
+cranelift-module = { path = "../module", version = "0.27.0" }
 faerie = "0.7.0"
 goblin = "0.0.19"
 failure = "0.1.2"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 77505a10e5..5e4ed222ed 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.26.0"
+version = "0.27.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.26.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../reader", version = "0.26.0" }
-cranelift-preopt = { path = "../preopt", version = "0.26.0" }
+cranelift-codegen = { path = "../codegen", version = "0.27.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../reader", version = "0.27.0" }
+cranelift-preopt = { path = "../preopt", version = "0.27.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 00391d83bf..9cde5f699a 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.26.0"
+version = "0.27.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.27.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index dbd9f95335..f6a25835f9 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.26.0"
+version = "0.27.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.27.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.27.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index f73357d938..4fb5d94a15 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.26.0"
+version = "0.27.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.27.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/preopt/Cargo.toml b/lib/preopt/Cargo.toml
index 08c492f2f0..34567a7d5d 100644
--- a/lib/preopt/Cargo.toml
+++ b/lib/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.26.0"
+version = "0.27.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.27.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.27.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index c183e689b3..346cea6cb4 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.26.0"
+version = "0.27.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.26.0" }
+cranelift-codegen = { path = "../codegen", version = "0.27.0" }
 target-lexicon = "0.2.0"
 
 [badges]
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index 24aa976671..69ea062477 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.26.0"
+version = "0.27.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../codegen", version = "0.26.0" }
-cranelift-reader = { path = "../reader", version = "0.26.0" }
+cranelift-codegen = { path = "../codegen", version = "0.27.0" }
+cranelift-reader = { path = "../reader", version = "0.27.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index dd448e9990..4e71bd7af2 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.26.0"
+version = "0.27.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.26.0" }
-cranelift-module = { path = "../module", version = "0.26.0" }
-cranelift-native = { path = "../native", version = "0.26.0" }
+cranelift-codegen = { path = "../codegen", version = "0.27.0" }
+cranelift-module = { path = "../module", version = "0.27.0" }
+cranelift-native = { path = "../native", version = "0.27.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.2.0" }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../umbrella", version = "0.26.0" }
-cranelift-frontend = { path = "../frontend", version = "0.26.0" }
-cranelift-entity = { path = "../entity", version = "0.26.0" }
+cranelift = { path = "../umbrella", version = "0.27.0" }
+cranelift-frontend = { path = "../frontend", version = "0.27.0" }
+cranelift-entity = { path = "../entity", version = "0.27.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index eb80082dd6..83567bc0c1 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.26.0"
+version = "0.27.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.26.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.27.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.27.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index 88aae1cecf..cbce200f83 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.26.0"
+version = "0.27.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.23.0", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.26.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.26.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.26.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.27.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.27.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.27.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 6326140227741f323c47a83b4d6cab1667b865cf Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 22 Jan 2019 23:44:10 -0800
Subject: [PATCH 2308/3084] Fix legalization of bxor_imm.

---
 cranelift/filetests/legalizer/bxor_imm.clif | 11 +++++++++++
 lib/codegen/meta-python/base/legalize.py    |  2 +-
 2 files changed, 12 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/filetests/legalizer/bxor_imm.clif

diff --git a/cranelift/filetests/legalizer/bxor_imm.clif b/cranelift/filetests/legalizer/bxor_imm.clif
new file mode 100644
index 0000000000..19372613ff
--- /dev/null
+++ b/cranelift/filetests/legalizer/bxor_imm.clif
@@ -0,0 +1,11 @@
+test legalizer
+target x86_64
+
+function %foo(i64, i64) -> i64 {
+ebb0(v0: i64, v1: i64):
+    v2 = bxor_imm.i64 v0, 0x100000000
+    return v2
+}
+
+; check: v3 = iconst.i64 0x0001_0000_0000
+; check: v2 = bxor v0, v3
diff --git a/lib/codegen/meta-python/base/legalize.py b/lib/codegen/meta-python/base/legalize.py
index 0625bb3963..163379cfac 100644
--- a/lib/codegen/meta-python/base/legalize.py
+++ b/lib/codegen/meta-python/base/legalize.py
@@ -500,7 +500,7 @@ for inst_imm,      inst in [
         (urem_imm, urem),
         (band_imm, band),
         (bor_imm, bor),
-        (bxor_imm, bor),
+        (bxor_imm, bxor),
         (ifcmp_imm, ifcmp)]:
     expand.legalize(
             a << inst_imm(x, y),

From 45cd3f6193f3ac4e90cac5fa158dfe5e3744fa33 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 22 Jan 2019 23:50:12 -0800
Subject: [PATCH 2309/3084] Bump version to 0.28.0

---
 cranelift/Cargo.toml        | 28 ++++++++++++++--------------
 cranelift/publish-all.sh    |  2 +-
 lib/bforest/Cargo.toml      |  4 ++--
 lib/codegen/Cargo.toml      |  8 ++++----
 lib/codegen/meta/Cargo.toml |  4 ++--
 lib/entity/Cargo.toml       |  2 +-
 lib/faerie/Cargo.toml       |  6 +++---
 lib/filetests/Cargo.toml    |  8 ++++----
 lib/frontend/Cargo.toml     |  4 ++--
 lib/module/Cargo.toml       |  6 +++---
 lib/native/Cargo.toml       |  4 ++--
 lib/preopt/Cargo.toml       |  6 +++---
 lib/reader/Cargo.toml       |  4 ++--
 lib/serde/Cargo.toml        |  6 +++---
 lib/simplejit/Cargo.toml    | 14 +++++++-------
 lib/umbrella/Cargo.toml     |  6 +++---
 lib/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index f696524ba3..bad9123866 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.27.0"
+version = "0.28.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -15,19 +15,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.27.0" }
-cranelift-entity = { path = "lib/entity", version = "0.27.0" }
-cranelift-reader = { path = "lib/reader", version = "0.27.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.27.0" }
-cranelift-serde = { path = "lib/serde", version = "0.27.0", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.27.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.27.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.27.0" }
-cranelift-module = { path = "lib/module", version = "0.27.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.27.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.27.0" }
-cranelift-preopt = { path = "lib/preopt", version = "0.27.0" }
-cranelift = { path = "lib/umbrella", version = "0.27.0" }
+cranelift-codegen = { path = "lib/codegen", version = "0.28.0" }
+cranelift-entity = { path = "lib/entity", version = "0.28.0" }
+cranelift-reader = { path = "lib/reader", version = "0.28.0" }
+cranelift-frontend = { path = "lib/frontend", version = "0.28.0" }
+cranelift-serde = { path = "lib/serde", version = "0.28.0", optional = true }
+cranelift-wasm = { path = "lib/wasm", version = "0.28.0", optional = true }
+cranelift-native = { path = "lib/native", version = "0.28.0" }
+cranelift-filetests = { path = "lib/filetests", version = "0.28.0" }
+cranelift-module = { path = "lib/module", version = "0.28.0" }
+cranelift-faerie = { path = "lib/faerie", version = "0.28.0" }
+cranelift-simplejit = { path = "lib/simplejit", version = "0.28.0" }
+cranelift-preopt = { path = "lib/preopt", version = "0.28.0" }
+cranelift = { path = "lib/umbrella", version = "0.28.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index bcdc1abc5a..74e91feeec 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.27.0"
+version="0.28.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/lib/bforest/Cargo.toml b/lib/bforest/Cargo.toml
index a6b61db391..19d3591483 100644
--- a/lib/bforest/Cargo.toml
+++ b/lib/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.27.0"
+version = "0.28.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.27.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.28.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/codegen/Cargo.toml b/lib/codegen/Cargo.toml
index 2065105d8b..b576b90c5f 100644
--- a/lib/codegen/Cargo.toml
+++ b/lib/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.27.0"
+version = "0.28.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.27.0", default-features = false }
-cranelift-bforest = { path = "../bforest", version = "0.27.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.28.0", default-features = false }
+cranelift-bforest = { path = "../bforest", version = "0.28.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -26,7 +26,7 @@ log = { version = "0.4.6", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.27.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.28.0" }
 
 [features]
 # The "std" feature enables use of libstd. The "core" feature enables use
diff --git a/lib/codegen/meta/Cargo.toml b/lib/codegen/meta/Cargo.toml
index 4a70592e0f..3fdfbb9305 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/lib/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.27.0"
+version = "0.28.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../entity", version = "0.27.0" }
+cranelift-entity = { path = "../../entity", version = "0.28.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/entity/Cargo.toml b/lib/entity/Cargo.toml
index 876a0de960..994670997a 100644
--- a/lib/entity/Cargo.toml
+++ b/lib/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.27.0"
+version = "0.28.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/lib/faerie/Cargo.toml b/lib/faerie/Cargo.toml
index 66523dc1e5..ce11f80570 100644
--- a/lib/faerie/Cargo.toml
+++ b/lib/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.27.0"
+version = "0.28.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.27.0" }
-cranelift-module = { path = "../module", version = "0.27.0" }
+cranelift-codegen = { path = "../codegen", version = "0.28.0" }
+cranelift-module = { path = "../module", version = "0.28.0" }
 faerie = "0.7.0"
 goblin = "0.0.19"
 failure = "0.1.2"
diff --git a/lib/filetests/Cargo.toml b/lib/filetests/Cargo.toml
index 5e4ed222ed..874e8ce098 100644
--- a/lib/filetests/Cargo.toml
+++ b/lib/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.27.0"
+version = "0.28.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.27.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../reader", version = "0.27.0" }
-cranelift-preopt = { path = "../preopt", version = "0.27.0" }
+cranelift-codegen = { path = "../codegen", version = "0.28.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../reader", version = "0.28.0" }
+cranelift-preopt = { path = "../preopt", version = "0.28.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/lib/frontend/Cargo.toml b/lib/frontend/Cargo.toml
index 9cde5f699a..9a5512199b 100644
--- a/lib/frontend/Cargo.toml
+++ b/lib/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.27.0"
+version = "0.28.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.27.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.28.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/lib/module/Cargo.toml b/lib/module/Cargo.toml
index f6a25835f9..a19cea5d27 100644
--- a/lib/module/Cargo.toml
+++ b/lib/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.27.0"
+version = "0.28.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.27.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.27.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.28.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.28.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/lib/native/Cargo.toml b/lib/native/Cargo.toml
index 4fb5d94a15..235cb2ee60 100644
--- a/lib/native/Cargo.toml
+++ b/lib/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.27.0"
+version = "0.28.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.27.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.28.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/preopt/Cargo.toml b/lib/preopt/Cargo.toml
index 34567a7d5d..260017f103 100644
--- a/lib/preopt/Cargo.toml
+++ b/lib/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.27.0"
+version = "0.28.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.27.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.27.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.28.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.28.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/lib/reader/Cargo.toml b/lib/reader/Cargo.toml
index 346cea6cb4..fc40aaec25 100644
--- a/lib/reader/Cargo.toml
+++ b/lib/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.27.0"
+version = "0.28.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.27.0" }
+cranelift-codegen = { path = "../codegen", version = "0.28.0" }
 target-lexicon = "0.2.0"
 
 [badges]
diff --git a/lib/serde/Cargo.toml b/lib/serde/Cargo.toml
index 69ea062477..b9355ed933 100644
--- a/lib/serde/Cargo.toml
+++ b/lib/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.27.0"
+version = "0.28.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../codegen", version = "0.27.0" }
-cranelift-reader = { path = "../reader", version = "0.27.0" }
+cranelift-codegen = { path = "../codegen", version = "0.28.0" }
+cranelift-reader = { path = "../reader", version = "0.28.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/simplejit/Cargo.toml b/lib/simplejit/Cargo.toml
index 4e71bd7af2..1f1e8af119 100644
--- a/lib/simplejit/Cargo.toml
+++ b/lib/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.27.0"
+version = "0.28.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.27.0" }
-cranelift-module = { path = "../module", version = "0.27.0" }
-cranelift-native = { path = "../native", version = "0.27.0" }
+cranelift-codegen = { path = "../codegen", version = "0.28.0" }
+cranelift-module = { path = "../module", version = "0.28.0" }
+cranelift-native = { path = "../native", version = "0.28.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.2.0" }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../umbrella", version = "0.27.0" }
-cranelift-frontend = { path = "../frontend", version = "0.27.0" }
-cranelift-entity = { path = "../entity", version = "0.27.0" }
+cranelift = { path = "../umbrella", version = "0.28.0" }
+cranelift-frontend = { path = "../frontend", version = "0.28.0" }
+cranelift-entity = { path = "../entity", version = "0.28.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/umbrella/Cargo.toml b/lib/umbrella/Cargo.toml
index 83567bc0c1..cf4b94a139 100644
--- a/lib/umbrella/Cargo.toml
+++ b/lib/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.27.0"
+version = "0.28.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.27.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.27.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.28.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.28.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/wasm/Cargo.toml b/lib/wasm/Cargo.toml
index cbce200f83..fce084d13d 100644
--- a/lib/wasm/Cargo.toml
+++ b/lib/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.27.0"
+version = "0.28.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.23.0", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.27.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.27.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.27.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.28.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.28.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.28.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 54959cf5bb2ec6fd7c498b048b69c05b8ae5a0b6 Mon Sep 17 00:00:00 2001
From: Lukas Bergdoll 
Date: Mon, 28 Jan 2019 23:00:49 +0100
Subject: [PATCH 2310/3084] Fix link to cargo workspace documentation

---
 cranelift/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index 51e7f6a264..96878451fd 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -77,7 +77,7 @@ Cranelift uses a [conventional Cargo build
 process](https://doc.rust-lang.org/cargo/guide/working-on-an-existing-project.html).
 
 Cranelift consists of a collection of crates, and uses a [Cargo
-Workspace](https://doc.rust-lang.org/book/second-edition/ch14-03-cargo-workspaces.html),
+Workspace](https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html),
 so for some cargo commands, such as `cargo test`, the `--all` is needed
 to tell cargo to visit all of the crates.
 

From 747ad3c4c58151d3ed24840a0c8afbee1335fa21 Mon Sep 17 00:00:00 2001
From: lazypassion <25536767+lazypassion@users.noreply.github.com>
Date: Mon, 28 Jan 2019 18:56:54 -0500
Subject: [PATCH 2311/3084] moved crates in lib/ to src/, renamed crates,
 modified some files' text (#660)

moved crates in lib/ to src/, renamed crates, modified some files' text (#660)
---
 cranelift/CONTRIBUTING.md                     |  2 +-
 cranelift/Cargo.toml                          | 26 +++++++++----------
 cranelift/README.md                           |  2 +-
 {lib => cranelift}/bforest/Cargo.toml         |  2 +-
 {lib => cranelift}/bforest/LICENSE            |  0
 {lib => cranelift}/bforest/README.md          |  0
 {lib => cranelift}/bforest/src/lib.rs         |  0
 {lib => cranelift}/bforest/src/map.rs         |  0
 {lib => cranelift}/bforest/src/node.rs        |  0
 {lib => cranelift}/bforest/src/path.rs        |  0
 {lib => cranelift}/bforest/src/pool.rs        |  0
 {lib => cranelift}/bforest/src/set.rs         |  0
 {lib => cranelift}/codegen/Cargo.toml         |  4 +--
 {lib => cranelift}/codegen/LICENSE            |  0
 {lib => cranelift}/codegen/README.md          |  0
 {lib => cranelift}/codegen/build.rs           |  4 +--
 .../codegen/meta-python/base/__init__.py      |  0
 .../codegen/meta-python/base/entities.py      |  0
 .../codegen/meta-python/base/formats.py       |  0
 .../codegen/meta-python/base/immediates.py    |  0
 .../codegen/meta-python/base/instructions.py  |  0
 .../codegen/meta-python/base/legalize.py      |  0
 .../codegen/meta-python/base/predicates.py    |  0
 .../codegen/meta-python/base/semantics.py     |  0
 .../codegen/meta-python/base/settings.py      |  0
 .../codegen/meta-python/base/types.py         |  0
 .../codegen/meta-python/build.py              |  2 +-
 .../codegen/meta-python/cdsl/__init__.py      |  0
 .../codegen/meta-python/cdsl/ast.py           |  0
 .../codegen/meta-python/cdsl/formats.py       |  0
 .../codegen/meta-python/cdsl/instructions.py  |  0
 .../codegen/meta-python/cdsl/isa.py           |  0
 .../codegen/meta-python/cdsl/operands.py      |  0
 .../codegen/meta-python/cdsl/predicates.py    |  0
 .../codegen/meta-python/cdsl/registers.py     |  0
 .../codegen/meta-python/cdsl/settings.py      |  0
 .../codegen/meta-python/cdsl/test_ast.py      |  0
 .../codegen/meta-python/cdsl/test_package.py  |  0
 .../codegen/meta-python/cdsl/test_ti.py       |  0
 .../codegen/meta-python/cdsl/test_typevar.py  |  0
 .../codegen/meta-python/cdsl/test_xform.py    |  0
 .../codegen/meta-python/cdsl/ti.py            |  0
 .../codegen/meta-python/cdsl/types.py         |  0
 .../codegen/meta-python/cdsl/typevar.py       |  0
 .../codegen/meta-python/cdsl/xform.py         |  0
 .../codegen/meta-python/check.sh              |  0
 .../codegen/meta-python/constant_hash.py      |  0
 .../codegen/meta-python/gen_binemit.py        |  0
 .../codegen/meta-python/gen_build_deps.py     |  2 +-
 .../codegen/meta-python/gen_encoding.py       |  0
 .../codegen/meta-python/gen_instr.py          |  0
 .../codegen/meta-python/gen_legalizer.py      |  0
 .../codegen/meta-python/gen_settings.py       |  0
 .../codegen/meta-python/isa/__init__.py       |  0
 .../codegen/meta-python/isa/arm32/__init__.py |  0
 .../codegen/meta-python/isa/arm32/defs.py     |  0
 .../meta-python/isa/arm32/registers.py        |  0
 .../codegen/meta-python/isa/arm32/settings.py |  0
 .../codegen/meta-python/isa/arm64/__init__.py |  0
 .../codegen/meta-python/isa/arm64/defs.py     |  0
 .../meta-python/isa/arm64/registers.py        |  0
 .../codegen/meta-python/isa/arm64/settings.py |  0
 .../codegen/meta-python/isa/riscv/__init__.py |  0
 .../codegen/meta-python/isa/riscv/defs.py     |  0
 .../meta-python/isa/riscv/encodings.py        |  0
 .../codegen/meta-python/isa/riscv/recipes.py  |  0
 .../meta-python/isa/riscv/registers.py        |  0
 .../codegen/meta-python/isa/riscv/settings.py |  0
 .../codegen/meta-python/isa/x86/__init__.py   |  0
 .../codegen/meta-python/isa/x86/defs.py       |  0
 .../codegen/meta-python/isa/x86/encodings.py  |  0
 .../meta-python/isa/x86/instructions.py       |  0
 .../codegen/meta-python/isa/x86/legalize.py   |  0
 .../codegen/meta-python/isa/x86/recipes.py    |  0
 .../codegen/meta-python/isa/x86/registers.py  |  0
 .../codegen/meta-python/isa/x86/settings.py   |  0
 .../codegen/meta-python/mypy.ini              |  0
 .../codegen/meta-python/semantics/__init__.py |  0
 .../meta-python/semantics/elaborate.py        |  0
 .../codegen/meta-python/semantics/macros.py   |  0
 .../meta-python/semantics/primitives.py       |  0
 .../codegen/meta-python/semantics/smtlib.py   |  0
 .../meta-python/semantics/test_elaborate.py   |  0
 .../codegen/meta-python/srcgen.py             |  0
 .../codegen/meta-python/stubs/z3/__init__.pyi |  0
 .../codegen/meta-python/stubs/z3/z3core.pyi   |  0
 .../codegen/meta-python/stubs/z3/z3types.pyi  |  0
 .../codegen/meta-python/test_constant_hash.py |  0
 .../codegen/meta-python/test_gen_legalizer.py |  0
 .../codegen/meta-python/test_srcgen.py        |  0
 .../codegen/meta-python/unique_table.py       |  0
 {lib => cranelift}/codegen/meta/Cargo.toml    |  2 +-
 {lib => cranelift}/codegen/meta/LICENSE       |  0
 {lib => cranelift}/codegen/meta/README.md     |  0
 .../codegen/meta/src/base/mod.rs              |  0
 .../codegen/meta/src/base/settings.rs         |  0
 .../codegen/meta/src/base/types.rs            |  0
 .../codegen/meta/src/cdsl/isa.rs              |  0
 .../codegen/meta/src/cdsl/mod.rs              |  0
 .../codegen/meta/src/cdsl/regs.rs             |  0
 .../codegen/meta/src/cdsl/settings.rs         |  0
 .../codegen/meta/src/cdsl/types.rs            |  0
 .../codegen/meta/src/constant_hash.rs         |  0
 {lib => cranelift}/codegen/meta/src/error.rs  |  0
 .../codegen/meta/src/gen_registers.rs         |  0
 .../codegen/meta/src/gen_settings.rs          |  0
 .../codegen/meta/src/gen_types.rs             |  2 +-
 .../codegen/meta/src/isa/arm32/mod.rs         |  0
 .../codegen/meta/src/isa/arm64/mod.rs         |  0
 .../codegen/meta/src/isa/mod.rs               |  0
 .../codegen/meta/src/isa/riscv/mod.rs         |  0
 .../codegen/meta/src/isa/x86/mod.rs           |  0
 {lib => cranelift}/codegen/meta/src/lib.rs    |  0
 {lib => cranelift}/codegen/meta/src/srcgen.rs |  0
 .../codegen/meta/src/unique_table.rs          |  0
 {lib => cranelift}/codegen/src/abi.rs         |  0
 .../codegen/src/binemit/memorysink.rs         |  0
 {lib => cranelift}/codegen/src/binemit/mod.rs |  0
 .../codegen/src/binemit/relaxation.rs         |  0
 .../codegen/src/binemit/shrink.rs             |  0
 {lib => cranelift}/codegen/src/bitset.rs      |  0
 {lib => cranelift}/codegen/src/cfg_printer.rs |  0
 .../codegen/src/constant_hash.rs              |  4 +--
 {lib => cranelift}/codegen/src/context.rs     |  0
 {lib => cranelift}/codegen/src/cursor.rs      |  0
 {lib => cranelift}/codegen/src/dbg.rs         |  0
 {lib => cranelift}/codegen/src/dce.rs         |  0
 .../codegen/src/divconst_magic_numbers.rs     |  0
 .../codegen/src/dominator_tree.rs             |  0
 {lib => cranelift}/codegen/src/flowgraph.rs   |  0
 {lib => cranelift}/codegen/src/fx.rs          |  0
 {lib => cranelift}/codegen/src/ir/builder.rs  |  2 +-
 .../codegen/src/ir/condcodes.rs               |  0
 {lib => cranelift}/codegen/src/ir/dfg.rs      |  0
 {lib => cranelift}/codegen/src/ir/entities.rs |  0
 {lib => cranelift}/codegen/src/ir/extfunc.rs  |  0
 {lib => cranelift}/codegen/src/ir/extname.rs  |  0
 {lib => cranelift}/codegen/src/ir/function.rs |  0
 .../codegen/src/ir/globalvalue.rs             |  0
 {lib => cranelift}/codegen/src/ir/heap.rs     |  0
 .../codegen/src/ir/immediates.rs              |  0
 .../codegen/src/ir/instructions.rs            |  4 +--
 .../codegen/src/ir/jumptable.rs               |  0
 {lib => cranelift}/codegen/src/ir/layout.rs   |  0
 {lib => cranelift}/codegen/src/ir/libcall.rs  |  0
 {lib => cranelift}/codegen/src/ir/memflags.rs |  0
 {lib => cranelift}/codegen/src/ir/mod.rs      |  0
 .../codegen/src/ir/progpoint.rs               |  0
 .../codegen/src/ir/sourceloc.rs               |  0
 .../codegen/src/ir/stackslot.rs               |  0
 {lib => cranelift}/codegen/src/ir/table.rs    |  0
 {lib => cranelift}/codegen/src/ir/trapcode.rs |  0
 {lib => cranelift}/codegen/src/ir/types.rs    |  2 +-
 {lib => cranelift}/codegen/src/ir/valueloc.rs |  0
 .../codegen/src/isa/arm32/abi.rs              |  0
 .../codegen/src/isa/arm32/binemit.rs          |  0
 .../codegen/src/isa/arm32/enc_tables.rs       |  0
 .../codegen/src/isa/arm32/mod.rs              |  0
 .../codegen/src/isa/arm32/registers.rs        |  0
 .../codegen/src/isa/arm32/settings.rs         |  4 +--
 .../codegen/src/isa/arm64/abi.rs              |  0
 .../codegen/src/isa/arm64/binemit.rs          |  0
 .../codegen/src/isa/arm64/enc_tables.rs       |  0
 .../codegen/src/isa/arm64/mod.rs              |  0
 .../codegen/src/isa/arm64/registers.rs        |  0
 .../codegen/src/isa/arm64/settings.rs         |  4 +--
 .../codegen/src/isa/call_conv.rs              |  0
 .../codegen/src/isa/constraints.rs            |  0
 .../codegen/src/isa/enc_tables.rs             |  2 +-
 .../codegen/src/isa/encoding.rs               |  0
 {lib => cranelift}/codegen/src/isa/mod.rs     |  0
 .../codegen/src/isa/registers.rs              |  0
 .../codegen/src/isa/riscv/abi.rs              |  0
 .../codegen/src/isa/riscv/binemit.rs          |  0
 .../codegen/src/isa/riscv/enc_tables.rs       |  0
 .../codegen/src/isa/riscv/mod.rs              |  0
 .../codegen/src/isa/riscv/registers.rs        |  0
 .../codegen/src/isa/riscv/settings.rs         |  4 +--
 {lib => cranelift}/codegen/src/isa/stack.rs   |  0
 {lib => cranelift}/codegen/src/isa/x86/abi.rs |  0
 .../codegen/src/isa/x86/binemit.rs            |  0
 .../codegen/src/isa/x86/enc_tables.rs         |  0
 {lib => cranelift}/codegen/src/isa/x86/mod.rs |  0
 .../codegen/src/isa/x86/registers.rs          |  0
 .../codegen/src/isa/x86/settings.rs           |  4 +--
 {lib => cranelift}/codegen/src/iterators.rs   |  0
 .../codegen/src/legalizer/boundary.rs         |  0
 .../codegen/src/legalizer/call.rs             |  0
 .../codegen/src/legalizer/globalvalue.rs      |  0
 .../codegen/src/legalizer/heap.rs             |  0
 .../codegen/src/legalizer/libcall.rs          |  0
 .../codegen/src/legalizer/mod.rs              |  2 +-
 .../codegen/src/legalizer/split.rs            |  0
 .../codegen/src/legalizer/table.rs            |  0
 {lib => cranelift}/codegen/src/lib.rs         |  0
 {lib => cranelift}/codegen/src/licm.rs        |  0
 .../codegen/src/loop_analysis.rs              |  0
 .../codegen/src/nan_canonicalization.rs       |  0
 .../codegen/src/partition_slice.rs            |  0
 {lib => cranelift}/codegen/src/postopt.rs     |  0
 {lib => cranelift}/codegen/src/predicates.rs  |  2 +-
 .../codegen/src/print_errors.rs               |  0
 {lib => cranelift}/codegen/src/ref_slice.rs   |  0
 .../codegen/src/regalloc/affinity.rs          |  0
 .../codegen/src/regalloc/coalescing.rs        |  0
 .../codegen/src/regalloc/coloring.rs          |  0
 .../codegen/src/regalloc/context.rs           |  0
 .../codegen/src/regalloc/diversion.rs         |  0
 .../src/regalloc/live_value_tracker.rs        |  0
 .../codegen/src/regalloc/liveness.rs          |  0
 .../codegen/src/regalloc/liverange.rs         |  0
 .../codegen/src/regalloc/mod.rs               |  0
 .../codegen/src/regalloc/pressure.rs          |  0
 .../codegen/src/regalloc/register_set.rs      |  0
 .../codegen/src/regalloc/reload.rs            |  0
 .../codegen/src/regalloc/solver.rs            |  0
 .../codegen/src/regalloc/spilling.rs          |  0
 .../codegen/src/regalloc/virtregs.rs          |  0
 {lib => cranelift}/codegen/src/result.rs      |  0
 .../codegen/src/scoped_hash_map.rs            |  0
 {lib => cranelift}/codegen/src/settings.rs    |  2 +-
 {lib => cranelift}/codegen/src/simple_gvn.rs  |  0
 .../codegen/src/simple_preopt.rs              |  0
 .../codegen/src/stack_layout.rs               |  0
 {lib => cranelift}/codegen/src/timing.rs      |  0
 {lib => cranelift}/codegen/src/topo_order.rs  |  0
 .../codegen/src/unreachable_code.rs           |  0
 .../codegen/src/verifier/cssa.rs              |  0
 .../codegen/src/verifier/flags.rs             |  0
 .../codegen/src/verifier/liveness.rs          |  0
 .../codegen/src/verifier/locations.rs         |  0
 .../codegen/src/verifier/mod.rs               |  0
 {lib => cranelift}/codegen/src/write.rs       |  0
 cranelift/docs/Makefile                       |  2 +-
 cranelift/docs/conf.py                        |  2 +-
 cranelift/docs/meta.rst                       | 10 +++----
 cranelift/docs/testing.rst                    | 10 +++----
 {lib => cranelift}/entity/Cargo.toml          |  0
 {lib => cranelift}/entity/LICENSE             |  0
 {lib => cranelift}/entity/README.md           |  0
 {lib => cranelift}/entity/src/boxed_slice.rs  |  0
 {lib => cranelift}/entity/src/iter.rs         |  0
 {lib => cranelift}/entity/src/keys.rs         |  0
 {lib => cranelift}/entity/src/lib.rs          |  0
 {lib => cranelift}/entity/src/list.rs         |  0
 {lib => cranelift}/entity/src/map.rs          |  0
 .../entity/src/packed_option.rs               |  0
 {lib => cranelift}/entity/src/primary.rs      |  0
 {lib => cranelift}/entity/src/set.rs          |  0
 {lib => cranelift}/entity/src/sparse.rs       |  0
 {lib => cranelift}/faerie/Cargo.toml          |  4 +--
 {lib => cranelift}/faerie/LICENSE             |  0
 {lib => cranelift}/faerie/README.md           |  0
 {lib => cranelift}/faerie/src/backend.rs      |  0
 {lib => cranelift}/faerie/src/container.rs    |  0
 {lib => cranelift}/faerie/src/lib.rs          |  0
 {lib => cranelift}/faerie/src/traps.rs        |  0
 {lib => cranelift}/filetests/Cargo.toml       |  6 ++---
 {lib => cranelift}/filetests/LICENSE          |  0
 .../filetests/{ => filetests}/cfg/loop.clif   |  0
 .../{ => filetests}/cfg/traps_early.clif      |  0
 .../{ => filetests}/cfg/unused_node.clif      |  0
 .../filetests/{ => filetests}/dce/basic.clif  |  0
 .../{ => filetests}/domtree/basic.clif        |  0
 .../{ => filetests}/domtree/loops.clif        |  0
 .../{ => filetests}/domtree/loops2.clif       |  0
 .../{ => filetests}/domtree/tall-tree.clif    |  0
 .../{ => filetests}/domtree/wide-tree.clif    |  0
 .../{ => filetests}/isa/riscv/abi-e.clif      |  0
 .../{ => filetests}/isa/riscv/abi.clif        |  0
 .../{ => filetests}/isa/riscv/binary32.clif   |  0
 .../{ => filetests}/isa/riscv/encoding.clif   |  0
 .../{ => filetests}/isa/riscv/expand-i32.clif |  0
 .../isa/riscv/legalize-abi.clif               |  0
 .../isa/riscv/legalize-i64.clif               |  0
 .../isa/riscv/parse-encoding.clif             |  0
 .../{ => filetests}/isa/riscv/regmove.clif    |  0
 .../{ => filetests}/isa/riscv/split-args.clif |  0
 .../isa/riscv/verify-encoding.clif            |  0
 .../{ => filetests}/isa/x86/abcd.clif         |  0
 .../{ => filetests}/isa/x86/abi-bool.clif     |  0
 .../{ => filetests}/isa/x86/abi32.clif        |  0
 .../{ => filetests}/isa/x86/abi64.clif        |  0
 .../isa/x86/allones_funcaddrs32.clif          |  0
 .../isa/x86/allones_funcaddrs64.clif          |  0
 .../isa/x86/baseline_clz_ctz_popcount.clif    |  0
 .../baseline_clz_ctz_popcount_encoding.clif   |  0
 .../isa/x86/binary32-float.clif               |  0
 .../{ => filetests}/isa/x86/binary32.clif     |  0
 .../isa/x86/binary64-float.clif               |  0
 .../{ => filetests}/isa/x86/binary64-pic.clif |  0
 .../{ => filetests}/isa/x86/binary64.clif     |  0
 .../isa/x86/ireduce-i16-to-i8.clif            |  0
 .../{ => filetests}/isa/x86/isub_imm-i8.clif  |  0
 .../isa/x86/legalize-bint-i8.clif             |  0
 .../isa/x86/legalize-bnot.clif                |  0
 .../isa/x86/legalize-br-icmp.clif             |  0
 .../isa/x86/legalize-br-table.clif            |  0
 .../isa/x86/legalize-byte-ops-i8.clif         |  0
 .../isa/x86/legalize-call.clif                |  0
 .../isa/x86/legalize-clz-ctz-i8.clif          |  0
 .../isa/x86/legalize-custom.clif              |  0
 .../isa/x86/legalize-div-traps.clif           |  0
 .../{ => filetests}/isa/x86/legalize-div.clif |  0
 .../isa/x86/legalize-heaps.clif               |  0
 .../isa/x86/legalize-icmp-i8.clif             |  0
 .../isa/x86/legalize-iconst-i8.clif           |  0
 .../isa/x86/legalize-imul-i8.clif             |  0
 .../isa/x86/legalize-imul-imm-i8.clif         |  0
 .../isa/x86/legalize-libcall.clif             |  0
 .../isa/x86/legalize-load-store-i8.clif       |  0
 .../isa/x86/legalize-memory.clif              |  0
 .../isa/x86/legalize-mulhi.clif               |  0
 .../isa/x86/legalize-popcnt-i8.clif           |  0
 .../isa/x86/legalize-regmove-i8.clif          |  0
 .../isa/x86/legalize-shlr-i8.clif             |  0
 .../isa/x86/legalize-tables.clif              |  0
 .../isa/x86/legalize-urem-i8.clif             |  0
 .../{ => filetests}/isa/x86/nop.clif          |  0
 .../x86/optimized-zero-constants-32bit.clif   |  0
 .../isa/x86/optimized-zero-constants.clif     |  0
 .../isa/x86/probestack-adjusts-sp.clif        |  0
 .../isa/x86/probestack-disabled.clif          |  0
 .../isa/x86/probestack-noncolocated.clif      |  0
 .../isa/x86/probestack-size.clif              |  0
 .../{ => filetests}/isa/x86/probestack.clif   |  0
 .../isa/x86/prologue-epilogue.clif            |  0
 .../isa/x86/shrink-multiple-uses.clif         |  0
 .../{ => filetests}/isa/x86/shrink.clif       |  0
 .../{ => filetests}/isa/x86/stack-addr64.clif |  0
 .../isa/x86/stack-load-store64.clif           |  0
 .../isa/x86/uextend-i8-to-i16.clif            |  0
 .../isa/x86/windows_fastcall_x64.clif         |  0
 .../{ => filetests}/legalizer/bitrev.clif     |  0
 .../legalizer/br_table_cond.clif              |  0
 .../{ => filetests}/legalizer/bxor_imm.clif   |  0
 .../filetests/{ => filetests}/licm/basic.clif |  0
 .../{ => filetests}/licm/complex.clif         |  0
 .../{ => filetests}/licm/critical-edge.clif   |  0
 .../{ => filetests}/licm/encoding.clif        |  0
 .../{ => filetests}/licm/multiple-blocks.clif |  0
 .../{ => filetests}/licm/nested_loops.clif    |  0
 .../{ => filetests}/licm/reject.clif          |  0
 .../{ => filetests}/parser/alias.clif         |  0
 .../{ => filetests}/parser/branch.clif        |  0
 .../{ => filetests}/parser/call.clif          |  0
 .../{ => filetests}/parser/flags.clif         |  0
 .../parser/instruction_encoding.clif          |  0
 .../{ => filetests}/parser/keywords.clif      |  0
 .../{ => filetests}/parser/memory.clif        |  0
 .../{ => filetests}/parser/rewrite.clif       |  0
 .../{ => filetests}/parser/ternary.clif       |  0
 .../{ => filetests}/parser/tiny.clif          |  0
 .../{ => filetests}/postopt/basic.clif        |  0
 .../postopt/complex_memory_ops.clif           |  0
 .../postopt/fold_offset_into_address.clif     |  0
 .../{ => filetests}/preopt/branch.clif        |  0
 .../{ => filetests}/preopt/numerical.clif     |  0
 .../{ => filetests}/regalloc/aliases.clif     |  0
 .../{ => filetests}/regalloc/basic.clif       |  0
 .../{ => filetests}/regalloc/coalesce.clif    |  0
 .../regalloc/coalescing-207.clif              |  0
 .../regalloc/coalescing-216.clif              |  0
 .../regalloc/coloring-227.clif                |  0
 .../{ => filetests}/regalloc/constraints.clif |  0
 .../regalloc/fallthrough-return.clif          |  0
 .../{ => filetests}/regalloc/ghost-param.clif |  0
 .../regalloc/global-constraints.clif          |  0
 .../regalloc/global-fixed.clif                |  0
 .../regalloc/gpr-deref-safe-335.clif          |  0
 .../regalloc/infinite-interference.clif       |  0
 .../{ => filetests}/regalloc/iterate.clif     |  0
 .../regalloc/multi-constraints.clif           |  0
 .../regalloc/multiple-returns.clif            |  0
 .../regalloc/output-interference.clif         |  0
 .../{ => filetests}/regalloc/reload-208.clif  |  0
 .../{ => filetests}/regalloc/reload.clif      |  0
 .../regalloc/schedule-moves.clif              |  0
 .../regalloc/spill-noregs.clif                |  2 +-
 .../{ => filetests}/regalloc/spill.clif       |  0
 .../regalloc/unreachable_code.clif            |  0
 .../{ => filetests}/regalloc/x86-regres.clif  |  0
 .../{ => filetests}/simple_gvn/basic.clif     |  0
 .../{ => filetests}/simple_gvn/readonly.clif  |  0
 .../{ => filetests}/simple_gvn/reject.clif    |  0
 .../{ => filetests}/simple_gvn/scopes.clif    |  0
 .../simple_preopt/div_by_const_indirect.clif  |  0
 .../div_by_const_non_power_of_2.clif          |  0
 .../div_by_const_power_of_2.clif              |  0
 .../rem_by_const_non_power_of_2.clif          |  0
 .../rem_by_const_power_of_2.clif              |  0
 .../simple_preopt/simplify.clif               |  0
 .../{ => filetests}/verifier/bad_layout.clif  |  0
 .../verifier/defs_dominates_uses.clif         |  0
 .../{ => filetests}/verifier/flags.clif       |  0
 .../{ => filetests}/verifier/globals.clif     |  0
 .../{ => filetests}/verifier/heap.clif        |  0
 .../{ => filetests}/verifier/memory.clif      |  0
 .../{ => filetests}/verifier/table.clif       |  0
 .../{ => filetests}/verifier/type_check.clif  |  0
 .../verifier/undeclared_vmctx.clif            |  0
 .../verifier/unreachable_code.clif            |  0
 .../{ => filetests}/wasm/control.clif         |  0
 .../{ => filetests}/wasm/conversions.clif     |  0
 .../{ => filetests}/wasm/f32-arith.clif       |  0
 .../{ => filetests}/wasm/f32-compares.clif    |  0
 .../{ => filetests}/wasm/f32-memory64.clif    |  0
 .../{ => filetests}/wasm/f64-arith.clif       |  0
 .../{ => filetests}/wasm/f64-compares.clif    |  0
 .../{ => filetests}/wasm/f64-memory64.clif    |  0
 .../{ => filetests}/wasm/i32-arith.clif       |  0
 .../{ => filetests}/wasm/i32-compares.clif    |  0
 .../{ => filetests}/wasm/i32-memory64.clif    |  0
 .../{ => filetests}/wasm/i64-arith.clif       |  0
 .../{ => filetests}/wasm/i64-compares.clif    |  0
 .../{ => filetests}/wasm/i64-memory64.clif    |  0
 .../{ => filetests}/wasm/select.clif          |  0
 .../filetests/src/concurrent.rs               |  0
 {lib => cranelift}/filetests/src/lib.rs       |  0
 .../filetests/src/match_directive.rs          |  0
 {lib => cranelift}/filetests/src/runner.rs    |  0
 {lib => cranelift}/filetests/src/runone.rs    |  0
 {lib => cranelift}/filetests/src/subtest.rs   |  0
 .../filetests/src/test_binemit.rs             |  0
 {lib => cranelift}/filetests/src/test_cat.rs  |  0
 .../filetests/src/test_compile.rs             |  0
 {lib => cranelift}/filetests/src/test_dce.rs  |  0
 .../filetests/src/test_domtree.rs             |  0
 .../filetests/src/test_legalizer.rs           |  0
 {lib => cranelift}/filetests/src/test_licm.rs |  0
 .../filetests/src/test_postopt.rs             |  0
 .../filetests/src/test_preopt.rs              |  0
 .../filetests/src/test_print_cfg.rs           |  0
 .../filetests/src/test_regalloc.rs            |  0
 .../filetests/src/test_shrink.rs              |  0
 .../filetests/src/test_simple_gvn.rs          |  0
 .../filetests/src/test_simple_preopt.rs       |  0
 .../filetests/src/test_verifier.rs            |  0
 {lib => cranelift}/frontend/Cargo.toml        |  2 +-
 {lib => cranelift}/frontend/LICENSE           |  0
 {lib => cranelift}/frontend/README.md         |  0
 {lib => cranelift}/frontend/src/frontend.rs   |  0
 {lib => cranelift}/frontend/src/lib.rs        |  0
 {lib => cranelift}/frontend/src/ssa.rs        |  0
 {lib => cranelift}/frontend/src/switch.rs     |  0
 {lib => cranelift}/frontend/src/variable.rs   |  0
 cranelift/fuzz/Cargo.toml                     |  6 ++---
 {lib => cranelift}/module/Cargo.toml          |  4 +--
 {lib => cranelift}/module/LICENSE             |  0
 {lib => cranelift}/module/README.md           |  0
 {lib => cranelift}/module/src/backend.rs      |  0
 {lib => cranelift}/module/src/data_context.rs |  0
 {lib => cranelift}/module/src/lib.rs          |  0
 {lib => cranelift}/module/src/module.rs       |  0
 {lib => cranelift}/native/Cargo.toml          |  2 +-
 {lib => cranelift}/native/LICENSE             |  0
 {lib => cranelift}/native/README.md           |  0
 {lib => cranelift}/native/src/lib.rs          |  0
 {lib => cranelift}/preopt/Cargo.toml          |  4 +--
 {lib => cranelift}/preopt/LICENSE             |  0
 {lib => cranelift}/preopt/README.md           |  0
 .../preopt/src/constant_folding.rs            |  0
 {lib => cranelift}/preopt/src/lib.rs          |  0
 cranelift/publish-all.sh                      |  4 +--
 {lib => cranelift}/reader/Cargo.toml          |  2 +-
 {lib => cranelift}/reader/LICENSE             |  0
 {lib => cranelift}/reader/README.md           |  0
 {lib => cranelift}/reader/src/error.rs        |  0
 {lib => cranelift}/reader/src/isaspec.rs      |  0
 {lib => cranelift}/reader/src/lexer.rs        |  0
 {lib => cranelift}/reader/src/lib.rs          |  0
 {lib => cranelift}/reader/src/parser.rs       |  0
 {lib => cranelift}/reader/src/sourcemap.rs    |  0
 {lib => cranelift}/reader/src/testcommand.rs  |  0
 {lib => cranelift}/reader/src/testfile.rs     |  0
 {lib => cranelift}/serde/Cargo.toml           |  4 +--
 {lib => cranelift}/serde/LICENSE              |  0
 {lib => cranelift}/serde/README.md            |  2 +-
 {lib => cranelift}/serde/src/clif-json.rs     |  0
 .../serde/src/serde_clif_json.rs              |  0
 {lib => cranelift}/simplejit/Cargo.toml       | 12 ++++-----
 {lib => cranelift}/simplejit/LICENSE          |  0
 {lib => cranelift}/simplejit/README.md        |  2 +-
 .../simplejit/examples/simplejit-minimal.rs   |  0
 {lib => cranelift}/simplejit/src/backend.rs   |  0
 {lib => cranelift}/simplejit/src/lib.rs       |  0
 {lib => cranelift}/simplejit/src/memory.rs    |  0
 {lib => cranelift}/simplejit/tests/basic.rs   |  0
 cranelift/test-all.sh                         |  2 +-
 cranelift/test-no_std.sh                      |  6 +++--
 {lib => cranelift}/umbrella/Cargo.toml        |  4 +--
 {lib => cranelift}/umbrella/LICENSE           |  0
 {lib => cranelift}/umbrella/README.md         |  0
 {lib => cranelift}/umbrella/src/lib.rs        |  0
 {lib => cranelift}/wasm/Cargo.toml            |  6 ++---
 {lib => cranelift}/wasm/LICENSE               |  0
 {lib => cranelift}/wasm/README.md             |  0
 .../wasm/src/code_translator.rs               |  0
 {lib => cranelift}/wasm/src/environ/dummy.rs  |  0
 {lib => cranelift}/wasm/src/environ/mod.rs    |  0
 {lib => cranelift}/wasm/src/environ/spec.rs   |  0
 .../wasm/src/func_translator.rs               |  0
 {lib => cranelift}/wasm/src/lib.rs            |  0
 .../wasm/src/module_translator.rs             |  0
 .../wasm/src/sections_translator.rs           |  0
 {lib => cranelift}/wasm/src/state.rs          |  0
 .../wasm/src/translation_utils.rs             |  0
 .../wasm/tests/wasm_testsuite.rs              |  4 +--
 508 files changed, 94 insertions(+), 92 deletions(-)
 rename {lib => cranelift}/bforest/Cargo.toml (85%)
 rename {lib => cranelift}/bforest/LICENSE (100%)
 rename {lib => cranelift}/bforest/README.md (100%)
 rename {lib => cranelift}/bforest/src/lib.rs (100%)
 rename {lib => cranelift}/bforest/src/map.rs (100%)
 rename {lib => cranelift}/bforest/src/node.rs (100%)
 rename {lib => cranelift}/bforest/src/path.rs (100%)
 rename {lib => cranelift}/bforest/src/pool.rs (100%)
 rename {lib => cranelift}/bforest/src/set.rs (100%)
 rename {lib => cranelift}/codegen/Cargo.toml (89%)
 rename {lib => cranelift}/codegen/LICENSE (100%)
 rename {lib => cranelift}/codegen/README.md (100%)
 rename {lib => cranelift}/codegen/build.rs (96%)
 rename {lib => cranelift}/codegen/meta-python/base/__init__.py (100%)
 rename {lib => cranelift}/codegen/meta-python/base/entities.py (100%)
 rename {lib => cranelift}/codegen/meta-python/base/formats.py (100%)
 rename {lib => cranelift}/codegen/meta-python/base/immediates.py (100%)
 rename {lib => cranelift}/codegen/meta-python/base/instructions.py (100%)
 rename {lib => cranelift}/codegen/meta-python/base/legalize.py (100%)
 rename {lib => cranelift}/codegen/meta-python/base/predicates.py (100%)
 rename {lib => cranelift}/codegen/meta-python/base/semantics.py (100%)
 rename {lib => cranelift}/codegen/meta-python/base/settings.py (100%)
 rename {lib => cranelift}/codegen/meta-python/base/types.py (100%)
 rename {lib => cranelift}/codegen/meta-python/build.py (90%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/__init__.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/ast.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/formats.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/instructions.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/isa.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/operands.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/predicates.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/registers.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/settings.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/test_ast.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/test_package.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/test_ti.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/test_typevar.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/test_xform.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/ti.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/types.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/typevar.py (100%)
 rename {lib => cranelift}/codegen/meta-python/cdsl/xform.py (100%)
 rename {lib => cranelift}/codegen/meta-python/check.sh (100%)
 rename {lib => cranelift}/codegen/meta-python/constant_hash.py (100%)
 rename {lib => cranelift}/codegen/meta-python/gen_binemit.py (100%)
 rename {lib => cranelift}/codegen/meta-python/gen_build_deps.py (91%)
 rename {lib => cranelift}/codegen/meta-python/gen_encoding.py (100%)
 rename {lib => cranelift}/codegen/meta-python/gen_instr.py (100%)
 rename {lib => cranelift}/codegen/meta-python/gen_legalizer.py (100%)
 rename {lib => cranelift}/codegen/meta-python/gen_settings.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/__init__.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/arm32/__init__.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/arm32/defs.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/arm32/registers.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/arm32/settings.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/arm64/__init__.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/arm64/defs.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/arm64/registers.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/arm64/settings.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/riscv/__init__.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/riscv/defs.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/riscv/encodings.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/riscv/recipes.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/riscv/registers.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/riscv/settings.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/x86/__init__.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/x86/defs.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/x86/encodings.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/x86/instructions.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/x86/legalize.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/x86/recipes.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/x86/registers.py (100%)
 rename {lib => cranelift}/codegen/meta-python/isa/x86/settings.py (100%)
 rename {lib => cranelift}/codegen/meta-python/mypy.ini (100%)
 rename {lib => cranelift}/codegen/meta-python/semantics/__init__.py (100%)
 rename {lib => cranelift}/codegen/meta-python/semantics/elaborate.py (100%)
 rename {lib => cranelift}/codegen/meta-python/semantics/macros.py (100%)
 rename {lib => cranelift}/codegen/meta-python/semantics/primitives.py (100%)
 rename {lib => cranelift}/codegen/meta-python/semantics/smtlib.py (100%)
 rename {lib => cranelift}/codegen/meta-python/semantics/test_elaborate.py (100%)
 rename {lib => cranelift}/codegen/meta-python/srcgen.py (100%)
 rename {lib => cranelift}/codegen/meta-python/stubs/z3/__init__.pyi (100%)
 rename {lib => cranelift}/codegen/meta-python/stubs/z3/z3core.pyi (100%)
 rename {lib => cranelift}/codegen/meta-python/stubs/z3/z3types.pyi (100%)
 rename {lib => cranelift}/codegen/meta-python/test_constant_hash.py (100%)
 rename {lib => cranelift}/codegen/meta-python/test_gen_legalizer.py (100%)
 rename {lib => cranelift}/codegen/meta-python/test_srcgen.py (100%)
 rename {lib => cranelift}/codegen/meta-python/unique_table.py (100%)
 rename {lib => cranelift}/codegen/meta/Cargo.toml (85%)
 rename {lib => cranelift}/codegen/meta/LICENSE (100%)
 rename {lib => cranelift}/codegen/meta/README.md (100%)
 rename {lib => cranelift}/codegen/meta/src/base/mod.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/base/settings.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/base/types.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/cdsl/isa.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/cdsl/mod.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/cdsl/regs.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/cdsl/settings.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/cdsl/types.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/constant_hash.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/error.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/gen_registers.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/gen_settings.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/gen_types.rs (96%)
 rename {lib => cranelift}/codegen/meta/src/isa/arm32/mod.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/isa/arm64/mod.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/isa/mod.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/isa/riscv/mod.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/isa/x86/mod.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/lib.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/srcgen.rs (100%)
 rename {lib => cranelift}/codegen/meta/src/unique_table.rs (100%)
 rename {lib => cranelift}/codegen/src/abi.rs (100%)
 rename {lib => cranelift}/codegen/src/binemit/memorysink.rs (100%)
 rename {lib => cranelift}/codegen/src/binemit/mod.rs (100%)
 rename {lib => cranelift}/codegen/src/binemit/relaxation.rs (100%)
 rename {lib => cranelift}/codegen/src/binemit/shrink.rs (100%)
 rename {lib => cranelift}/codegen/src/bitset.rs (100%)
 rename {lib => cranelift}/codegen/src/cfg_printer.rs (100%)
 rename {lib => cranelift}/codegen/src/constant_hash.rs (91%)
 rename {lib => cranelift}/codegen/src/context.rs (100%)
 rename {lib => cranelift}/codegen/src/cursor.rs (100%)
 rename {lib => cranelift}/codegen/src/dbg.rs (100%)
 rename {lib => cranelift}/codegen/src/dce.rs (100%)
 rename {lib => cranelift}/codegen/src/divconst_magic_numbers.rs (100%)
 rename {lib => cranelift}/codegen/src/dominator_tree.rs (100%)
 rename {lib => cranelift}/codegen/src/flowgraph.rs (100%)
 rename {lib => cranelift}/codegen/src/fx.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/builder.rs (99%)
 rename {lib => cranelift}/codegen/src/ir/condcodes.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/dfg.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/entities.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/extfunc.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/extname.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/function.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/globalvalue.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/heap.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/immediates.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/instructions.rs (99%)
 rename {lib => cranelift}/codegen/src/ir/jumptable.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/layout.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/libcall.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/memflags.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/mod.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/progpoint.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/sourceloc.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/stackslot.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/table.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/trapcode.rs (100%)
 rename {lib => cranelift}/codegen/src/ir/types.rs (99%)
 rename {lib => cranelift}/codegen/src/ir/valueloc.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/arm32/abi.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/arm32/binemit.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/arm32/enc_tables.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/arm32/mod.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/arm32/registers.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/arm32/settings.rs (55%)
 rename {lib => cranelift}/codegen/src/isa/arm64/abi.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/arm64/binemit.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/arm64/enc_tables.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/arm64/mod.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/arm64/registers.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/arm64/settings.rs (55%)
 rename {lib => cranelift}/codegen/src/isa/call_conv.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/constraints.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/enc_tables.rs (99%)
 rename {lib => cranelift}/codegen/src/isa/encoding.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/mod.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/registers.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/riscv/abi.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/riscv/binemit.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/riscv/enc_tables.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/riscv/mod.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/riscv/registers.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/riscv/settings.rs (90%)
 rename {lib => cranelift}/codegen/src/isa/stack.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/x86/abi.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/x86/binemit.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/x86/enc_tables.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/x86/mod.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/x86/registers.rs (100%)
 rename {lib => cranelift}/codegen/src/isa/x86/settings.rs (89%)
 rename {lib => cranelift}/codegen/src/iterators.rs (100%)
 rename {lib => cranelift}/codegen/src/legalizer/boundary.rs (100%)
 rename {lib => cranelift}/codegen/src/legalizer/call.rs (100%)
 rename {lib => cranelift}/codegen/src/legalizer/globalvalue.rs (100%)
 rename {lib => cranelift}/codegen/src/legalizer/heap.rs (100%)
 rename {lib => cranelift}/codegen/src/legalizer/libcall.rs (100%)
 rename {lib => cranelift}/codegen/src/legalizer/mod.rs (99%)
 rename {lib => cranelift}/codegen/src/legalizer/split.rs (100%)
 rename {lib => cranelift}/codegen/src/legalizer/table.rs (100%)
 rename {lib => cranelift}/codegen/src/lib.rs (100%)
 rename {lib => cranelift}/codegen/src/licm.rs (100%)
 rename {lib => cranelift}/codegen/src/loop_analysis.rs (100%)
 rename {lib => cranelift}/codegen/src/nan_canonicalization.rs (100%)
 rename {lib => cranelift}/codegen/src/partition_slice.rs (100%)
 rename {lib => cranelift}/codegen/src/postopt.rs (100%)
 rename {lib => cranelift}/codegen/src/predicates.rs (98%)
 rename {lib => cranelift}/codegen/src/print_errors.rs (100%)
 rename {lib => cranelift}/codegen/src/ref_slice.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/affinity.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/coalescing.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/coloring.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/context.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/diversion.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/live_value_tracker.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/liveness.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/liverange.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/mod.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/pressure.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/register_set.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/reload.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/solver.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/spilling.rs (100%)
 rename {lib => cranelift}/codegen/src/regalloc/virtregs.rs (100%)
 rename {lib => cranelift}/codegen/src/result.rs (100%)
 rename {lib => cranelift}/codegen/src/scoped_hash_map.rs (100%)
 rename {lib => cranelift}/codegen/src/settings.rs (99%)
 rename {lib => cranelift}/codegen/src/simple_gvn.rs (100%)
 rename {lib => cranelift}/codegen/src/simple_preopt.rs (100%)
 rename {lib => cranelift}/codegen/src/stack_layout.rs (100%)
 rename {lib => cranelift}/codegen/src/timing.rs (100%)
 rename {lib => cranelift}/codegen/src/topo_order.rs (100%)
 rename {lib => cranelift}/codegen/src/unreachable_code.rs (100%)
 rename {lib => cranelift}/codegen/src/verifier/cssa.rs (100%)
 rename {lib => cranelift}/codegen/src/verifier/flags.rs (100%)
 rename {lib => cranelift}/codegen/src/verifier/liveness.rs (100%)
 rename {lib => cranelift}/codegen/src/verifier/locations.rs (100%)
 rename {lib => cranelift}/codegen/src/verifier/mod.rs (100%)
 rename {lib => cranelift}/codegen/src/write.rs (100%)
 rename {lib => cranelift}/entity/Cargo.toml (100%)
 rename {lib => cranelift}/entity/LICENSE (100%)
 rename {lib => cranelift}/entity/README.md (100%)
 rename {lib => cranelift}/entity/src/boxed_slice.rs (100%)
 rename {lib => cranelift}/entity/src/iter.rs (100%)
 rename {lib => cranelift}/entity/src/keys.rs (100%)
 rename {lib => cranelift}/entity/src/lib.rs (100%)
 rename {lib => cranelift}/entity/src/list.rs (100%)
 rename {lib => cranelift}/entity/src/map.rs (100%)
 rename {lib => cranelift}/entity/src/packed_option.rs (100%)
 rename {lib => cranelift}/entity/src/primary.rs (100%)
 rename {lib => cranelift}/entity/src/set.rs (100%)
 rename {lib => cranelift}/entity/src/sparse.rs (100%)
 rename {lib => cranelift}/faerie/Cargo.toml (79%)
 rename {lib => cranelift}/faerie/LICENSE (100%)
 rename {lib => cranelift}/faerie/README.md (100%)
 rename {lib => cranelift}/faerie/src/backend.rs (100%)
 rename {lib => cranelift}/faerie/src/container.rs (100%)
 rename {lib => cranelift}/faerie/src/lib.rs (100%)
 rename {lib => cranelift}/faerie/src/traps.rs (100%)
 rename {lib => cranelift}/filetests/Cargo.toml (66%)
 rename {lib => cranelift}/filetests/LICENSE (100%)
 rename cranelift/filetests/{ => filetests}/cfg/loop.clif (100%)
 rename cranelift/filetests/{ => filetests}/cfg/traps_early.clif (100%)
 rename cranelift/filetests/{ => filetests}/cfg/unused_node.clif (100%)
 rename cranelift/filetests/{ => filetests}/dce/basic.clif (100%)
 rename cranelift/filetests/{ => filetests}/domtree/basic.clif (100%)
 rename cranelift/filetests/{ => filetests}/domtree/loops.clif (100%)
 rename cranelift/filetests/{ => filetests}/domtree/loops2.clif (100%)
 rename cranelift/filetests/{ => filetests}/domtree/tall-tree.clif (100%)
 rename cranelift/filetests/{ => filetests}/domtree/wide-tree.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/riscv/abi-e.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/riscv/abi.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/riscv/binary32.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/riscv/encoding.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/riscv/expand-i32.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/riscv/legalize-abi.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/riscv/legalize-i64.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/riscv/parse-encoding.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/riscv/regmove.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/riscv/split-args.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/riscv/verify-encoding.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/abcd.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/abi-bool.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/abi32.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/abi64.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/allones_funcaddrs32.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/allones_funcaddrs64.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/baseline_clz_ctz_popcount.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/baseline_clz_ctz_popcount_encoding.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/binary32-float.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/binary32.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/binary64-float.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/binary64-pic.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/binary64.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/ireduce-i16-to-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/isub_imm-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-bint-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-bnot.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-br-icmp.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-br-table.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-byte-ops-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-call.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-clz-ctz-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-custom.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-div-traps.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-div.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-heaps.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-icmp-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-iconst-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-imul-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-imul-imm-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-libcall.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-load-store-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-memory.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-mulhi.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-popcnt-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-regmove-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-shlr-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-tables.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/legalize-urem-i8.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/nop.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/optimized-zero-constants-32bit.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/optimized-zero-constants.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/probestack-adjusts-sp.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/probestack-disabled.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/probestack-noncolocated.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/probestack-size.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/probestack.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/prologue-epilogue.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/shrink-multiple-uses.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/shrink.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/stack-addr64.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/stack-load-store64.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/uextend-i8-to-i16.clif (100%)
 rename cranelift/filetests/{ => filetests}/isa/x86/windows_fastcall_x64.clif (100%)
 rename cranelift/filetests/{ => filetests}/legalizer/bitrev.clif (100%)
 rename cranelift/filetests/{ => filetests}/legalizer/br_table_cond.clif (100%)
 rename cranelift/filetests/{ => filetests}/legalizer/bxor_imm.clif (100%)
 rename cranelift/filetests/{ => filetests}/licm/basic.clif (100%)
 rename cranelift/filetests/{ => filetests}/licm/complex.clif (100%)
 rename cranelift/filetests/{ => filetests}/licm/critical-edge.clif (100%)
 rename cranelift/filetests/{ => filetests}/licm/encoding.clif (100%)
 rename cranelift/filetests/{ => filetests}/licm/multiple-blocks.clif (100%)
 rename cranelift/filetests/{ => filetests}/licm/nested_loops.clif (100%)
 rename cranelift/filetests/{ => filetests}/licm/reject.clif (100%)
 rename cranelift/filetests/{ => filetests}/parser/alias.clif (100%)
 rename cranelift/filetests/{ => filetests}/parser/branch.clif (100%)
 rename cranelift/filetests/{ => filetests}/parser/call.clif (100%)
 rename cranelift/filetests/{ => filetests}/parser/flags.clif (100%)
 rename cranelift/filetests/{ => filetests}/parser/instruction_encoding.clif (100%)
 rename cranelift/filetests/{ => filetests}/parser/keywords.clif (100%)
 rename cranelift/filetests/{ => filetests}/parser/memory.clif (100%)
 rename cranelift/filetests/{ => filetests}/parser/rewrite.clif (100%)
 rename cranelift/filetests/{ => filetests}/parser/ternary.clif (100%)
 rename cranelift/filetests/{ => filetests}/parser/tiny.clif (100%)
 rename cranelift/filetests/{ => filetests}/postopt/basic.clif (100%)
 rename cranelift/filetests/{ => filetests}/postopt/complex_memory_ops.clif (100%)
 rename cranelift/filetests/{ => filetests}/postopt/fold_offset_into_address.clif (100%)
 rename cranelift/filetests/{ => filetests}/preopt/branch.clif (100%)
 rename cranelift/filetests/{ => filetests}/preopt/numerical.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/aliases.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/basic.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/coalesce.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/coalescing-207.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/coalescing-216.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/coloring-227.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/constraints.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/fallthrough-return.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/ghost-param.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/global-constraints.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/global-fixed.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/gpr-deref-safe-335.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/infinite-interference.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/iterate.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/multi-constraints.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/multiple-returns.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/output-interference.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/reload-208.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/reload.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/schedule-moves.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/spill-noregs.clif (98%)
 rename cranelift/filetests/{ => filetests}/regalloc/spill.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/unreachable_code.clif (100%)
 rename cranelift/filetests/{ => filetests}/regalloc/x86-regres.clif (100%)
 rename cranelift/filetests/{ => filetests}/simple_gvn/basic.clif (100%)
 rename cranelift/filetests/{ => filetests}/simple_gvn/readonly.clif (100%)
 rename cranelift/filetests/{ => filetests}/simple_gvn/reject.clif (100%)
 rename cranelift/filetests/{ => filetests}/simple_gvn/scopes.clif (100%)
 rename cranelift/filetests/{ => filetests}/simple_preopt/div_by_const_indirect.clif (100%)
 rename cranelift/filetests/{ => filetests}/simple_preopt/div_by_const_non_power_of_2.clif (100%)
 rename cranelift/filetests/{ => filetests}/simple_preopt/div_by_const_power_of_2.clif (100%)
 rename cranelift/filetests/{ => filetests}/simple_preopt/rem_by_const_non_power_of_2.clif (100%)
 rename cranelift/filetests/{ => filetests}/simple_preopt/rem_by_const_power_of_2.clif (100%)
 rename cranelift/filetests/{ => filetests}/simple_preopt/simplify.clif (100%)
 rename cranelift/filetests/{ => filetests}/verifier/bad_layout.clif (100%)
 rename cranelift/filetests/{ => filetests}/verifier/defs_dominates_uses.clif (100%)
 rename cranelift/filetests/{ => filetests}/verifier/flags.clif (100%)
 rename cranelift/filetests/{ => filetests}/verifier/globals.clif (100%)
 rename cranelift/filetests/{ => filetests}/verifier/heap.clif (100%)
 rename cranelift/filetests/{ => filetests}/verifier/memory.clif (100%)
 rename cranelift/filetests/{ => filetests}/verifier/table.clif (100%)
 rename cranelift/filetests/{ => filetests}/verifier/type_check.clif (100%)
 rename cranelift/filetests/{ => filetests}/verifier/undeclared_vmctx.clif (100%)
 rename cranelift/filetests/{ => filetests}/verifier/unreachable_code.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/control.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/conversions.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/f32-arith.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/f32-compares.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/f32-memory64.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/f64-arith.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/f64-compares.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/f64-memory64.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/i32-arith.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/i32-compares.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/i32-memory64.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/i64-arith.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/i64-compares.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/i64-memory64.clif (100%)
 rename cranelift/filetests/{ => filetests}/wasm/select.clif (100%)
 rename {lib => cranelift}/filetests/src/concurrent.rs (100%)
 rename {lib => cranelift}/filetests/src/lib.rs (100%)
 rename {lib => cranelift}/filetests/src/match_directive.rs (100%)
 rename {lib => cranelift}/filetests/src/runner.rs (100%)
 rename {lib => cranelift}/filetests/src/runone.rs (100%)
 rename {lib => cranelift}/filetests/src/subtest.rs (100%)
 rename {lib => cranelift}/filetests/src/test_binemit.rs (100%)
 rename {lib => cranelift}/filetests/src/test_cat.rs (100%)
 rename {lib => cranelift}/filetests/src/test_compile.rs (100%)
 rename {lib => cranelift}/filetests/src/test_dce.rs (100%)
 rename {lib => cranelift}/filetests/src/test_domtree.rs (100%)
 rename {lib => cranelift}/filetests/src/test_legalizer.rs (100%)
 rename {lib => cranelift}/filetests/src/test_licm.rs (100%)
 rename {lib => cranelift}/filetests/src/test_postopt.rs (100%)
 rename {lib => cranelift}/filetests/src/test_preopt.rs (100%)
 rename {lib => cranelift}/filetests/src/test_print_cfg.rs (100%)
 rename {lib => cranelift}/filetests/src/test_regalloc.rs (100%)
 rename {lib => cranelift}/filetests/src/test_shrink.rs (100%)
 rename {lib => cranelift}/filetests/src/test_simple_gvn.rs (100%)
 rename {lib => cranelift}/filetests/src/test_simple_preopt.rs (100%)
 rename {lib => cranelift}/filetests/src/test_verifier.rs (100%)
 rename {lib => cranelift}/frontend/Cargo.toml (88%)
 rename {lib => cranelift}/frontend/LICENSE (100%)
 rename {lib => cranelift}/frontend/README.md (100%)
 rename {lib => cranelift}/frontend/src/frontend.rs (100%)
 rename {lib => cranelift}/frontend/src/lib.rs (100%)
 rename {lib => cranelift}/frontend/src/ssa.rs (100%)
 rename {lib => cranelift}/frontend/src/switch.rs (100%)
 rename {lib => cranelift}/frontend/src/variable.rs (100%)
 rename {lib => cranelift}/module/Cargo.toml (80%)
 rename {lib => cranelift}/module/LICENSE (100%)
 rename {lib => cranelift}/module/README.md (100%)
 rename {lib => cranelift}/module/src/backend.rs (100%)
 rename {lib => cranelift}/module/src/data_context.rs (100%)
 rename {lib => cranelift}/module/src/lib.rs (100%)
 rename {lib => cranelift}/module/src/module.rs (100%)
 rename {lib => cranelift}/native/Cargo.toml (90%)
 rename {lib => cranelift}/native/LICENSE (100%)
 rename {lib => cranelift}/native/README.md (100%)
 rename {lib => cranelift}/native/src/lib.rs (100%)
 rename {lib => cranelift}/preopt/Cargo.toml (80%)
 rename {lib => cranelift}/preopt/LICENSE (100%)
 rename {lib => cranelift}/preopt/README.md (100%)
 rename {lib => cranelift}/preopt/src/constant_folding.rs (100%)
 rename {lib => cranelift}/preopt/src/lib.rs (100%)
 rename {lib => cranelift}/reader/Cargo.toml (86%)
 rename {lib => cranelift}/reader/LICENSE (100%)
 rename {lib => cranelift}/reader/README.md (100%)
 rename {lib => cranelift}/reader/src/error.rs (100%)
 rename {lib => cranelift}/reader/src/isaspec.rs (100%)
 rename {lib => cranelift}/reader/src/lexer.rs (100%)
 rename {lib => cranelift}/reader/src/lib.rs (100%)
 rename {lib => cranelift}/reader/src/parser.rs (100%)
 rename {lib => cranelift}/reader/src/sourcemap.rs (100%)
 rename {lib => cranelift}/reader/src/testcommand.rs (100%)
 rename {lib => cranelift}/reader/src/testfile.rs (100%)
 rename {lib => cranelift}/serde/Cargo.toml (80%)
 rename {lib => cranelift}/serde/LICENSE (100%)
 rename {lib => cranelift}/serde/README.md (97%)
 rename {lib => cranelift}/serde/src/clif-json.rs (100%)
 rename {lib => cranelift}/serde/src/serde_clif_json.rs (100%)
 rename {lib => cranelift}/simplejit/Cargo.toml (62%)
 rename {lib => cranelift}/simplejit/LICENSE (100%)
 rename {lib => cranelift}/simplejit/README.md (82%)
 rename {lib => cranelift}/simplejit/examples/simplejit-minimal.rs (100%)
 rename {lib => cranelift}/simplejit/src/backend.rs (100%)
 rename {lib => cranelift}/simplejit/src/lib.rs (100%)
 rename {lib => cranelift}/simplejit/src/memory.rs (100%)
 rename {lib => cranelift}/simplejit/tests/basic.rs (100%)
 rename {lib => cranelift}/umbrella/Cargo.toml (77%)
 rename {lib => cranelift}/umbrella/LICENSE (100%)
 rename {lib => cranelift}/umbrella/README.md (100%)
 rename {lib => cranelift}/umbrella/src/lib.rs (100%)
 rename {lib => cranelift}/wasm/Cargo.toml (78%)
 rename {lib => cranelift}/wasm/LICENSE (100%)
 rename {lib => cranelift}/wasm/README.md (100%)
 rename {lib => cranelift}/wasm/src/code_translator.rs (100%)
 rename {lib => cranelift}/wasm/src/environ/dummy.rs (100%)
 rename {lib => cranelift}/wasm/src/environ/mod.rs (100%)
 rename {lib => cranelift}/wasm/src/environ/spec.rs (100%)
 rename {lib => cranelift}/wasm/src/func_translator.rs (100%)
 rename {lib => cranelift}/wasm/src/lib.rs (100%)
 rename {lib => cranelift}/wasm/src/module_translator.rs (100%)
 rename {lib => cranelift}/wasm/src/sections_translator.rs (100%)
 rename {lib => cranelift}/wasm/src/state.rs (100%)
 rename {lib => cranelift}/wasm/src/translation_utils.rs (100%)
 rename {lib => cranelift}/wasm/tests/wasm_testsuite.rs (95%)

diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
index 105568991f..403e912e60 100644
--- a/cranelift/CONTRIBUTING.md
+++ b/cranelift/CONTRIBUTING.md
@@ -88,7 +88,7 @@ build.
 
 Our Python code is checked with [mypy](http://mypy-lang.org/) and
 [flake8](http://flake8.pycqa.org/en/latest/); see the
-[check.sh](https://github.com/CraneStation/cranelift/blob/master/lib/codegen/meta-python/check.sh)
+[check.sh](https://github.com/CraneStation/cranelift/blob/master/cranelift-codegen/meta-python/check.sh)
 file for details. The versions available in common package repositories such
 as Ubuntu or Homebrew typically work fine.
 
diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index bad9123866..36c85ba347 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -15,19 +15,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "lib/codegen", version = "0.28.0" }
-cranelift-entity = { path = "lib/entity", version = "0.28.0" }
-cranelift-reader = { path = "lib/reader", version = "0.28.0" }
-cranelift-frontend = { path = "lib/frontend", version = "0.28.0" }
-cranelift-serde = { path = "lib/serde", version = "0.28.0", optional = true }
-cranelift-wasm = { path = "lib/wasm", version = "0.28.0", optional = true }
-cranelift-native = { path = "lib/native", version = "0.28.0" }
-cranelift-filetests = { path = "lib/filetests", version = "0.28.0" }
-cranelift-module = { path = "lib/module", version = "0.28.0" }
-cranelift-faerie = { path = "lib/faerie", version = "0.28.0" }
-cranelift-simplejit = { path = "lib/simplejit", version = "0.28.0" }
-cranelift-preopt = { path = "lib/preopt", version = "0.28.0" }
-cranelift = { path = "lib/umbrella", version = "0.28.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.28.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.28.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.28.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.28.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.28.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.28.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.28.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.28.0" }
+cranelift-module = { path = "cranelift-module", version = "0.28.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.28.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.28.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.28.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.28.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/README.md b/cranelift/README.md
index 96878451fd..51aefca247 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -104,7 +104,7 @@ feature. This currently requires nightly rust.
 For example, to build \`cranelift-codegen\`:
 
 ``` {.sourceCode .sh}
-cd lib/codegen
+cd cranelift-codegen
 cargo build --no-default-features --features core
 ```
 
diff --git a/lib/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
similarity index 85%
rename from lib/bforest/Cargo.toml
rename to cranelift/bforest/Cargo.toml
index 19d3591483..b650e9cc83 100644
--- a/lib/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.28.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.28.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/bforest/LICENSE b/cranelift/bforest/LICENSE
similarity index 100%
rename from lib/bforest/LICENSE
rename to cranelift/bforest/LICENSE
diff --git a/lib/bforest/README.md b/cranelift/bforest/README.md
similarity index 100%
rename from lib/bforest/README.md
rename to cranelift/bforest/README.md
diff --git a/lib/bforest/src/lib.rs b/cranelift/bforest/src/lib.rs
similarity index 100%
rename from lib/bforest/src/lib.rs
rename to cranelift/bforest/src/lib.rs
diff --git a/lib/bforest/src/map.rs b/cranelift/bforest/src/map.rs
similarity index 100%
rename from lib/bforest/src/map.rs
rename to cranelift/bforest/src/map.rs
diff --git a/lib/bforest/src/node.rs b/cranelift/bforest/src/node.rs
similarity index 100%
rename from lib/bforest/src/node.rs
rename to cranelift/bforest/src/node.rs
diff --git a/lib/bforest/src/path.rs b/cranelift/bforest/src/path.rs
similarity index 100%
rename from lib/bforest/src/path.rs
rename to cranelift/bforest/src/path.rs
diff --git a/lib/bforest/src/pool.rs b/cranelift/bforest/src/pool.rs
similarity index 100%
rename from lib/bforest/src/pool.rs
rename to cranelift/bforest/src/pool.rs
diff --git a/lib/bforest/src/set.rs b/cranelift/bforest/src/set.rs
similarity index 100%
rename from lib/bforest/src/set.rs
rename to cranelift/bforest/src/set.rs
diff --git a/lib/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
similarity index 89%
rename from lib/codegen/Cargo.toml
rename to cranelift/codegen/Cargo.toml
index b576b90c5f..0448d8072e 100644
--- a/lib/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../entity", version = "0.28.0", default-features = false }
-cranelift-bforest = { path = "../bforest", version = "0.28.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.28.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.28.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/lib/codegen/LICENSE b/cranelift/codegen/LICENSE
similarity index 100%
rename from lib/codegen/LICENSE
rename to cranelift/codegen/LICENSE
diff --git a/lib/codegen/README.md b/cranelift/codegen/README.md
similarity index 100%
rename from lib/codegen/README.md
rename to cranelift/codegen/README.md
diff --git a/lib/codegen/build.rs b/cranelift/codegen/build.rs
similarity index 96%
rename from lib/codegen/build.rs
rename to cranelift/codegen/build.rs
index e8a57cfc26..c4b1a12de6 100644
--- a/lib/codegen/build.rs
+++ b/cranelift/codegen/build.rs
@@ -1,7 +1,7 @@
 // Build script.
 //
-// This program is run by Cargo when building lib/codegen. It is used to generate Rust code from
-// the language definitions in the lib/codegen/meta directory.
+// This program is run by Cargo when building cranelift-codegen. It is used to generate Rust code from
+// the language definitions in the cranelift-codegen/meta directory.
 //
 // Environment:
 //
diff --git a/lib/codegen/meta-python/base/__init__.py b/cranelift/codegen/meta-python/base/__init__.py
similarity index 100%
rename from lib/codegen/meta-python/base/__init__.py
rename to cranelift/codegen/meta-python/base/__init__.py
diff --git a/lib/codegen/meta-python/base/entities.py b/cranelift/codegen/meta-python/base/entities.py
similarity index 100%
rename from lib/codegen/meta-python/base/entities.py
rename to cranelift/codegen/meta-python/base/entities.py
diff --git a/lib/codegen/meta-python/base/formats.py b/cranelift/codegen/meta-python/base/formats.py
similarity index 100%
rename from lib/codegen/meta-python/base/formats.py
rename to cranelift/codegen/meta-python/base/formats.py
diff --git a/lib/codegen/meta-python/base/immediates.py b/cranelift/codegen/meta-python/base/immediates.py
similarity index 100%
rename from lib/codegen/meta-python/base/immediates.py
rename to cranelift/codegen/meta-python/base/immediates.py
diff --git a/lib/codegen/meta-python/base/instructions.py b/cranelift/codegen/meta-python/base/instructions.py
similarity index 100%
rename from lib/codegen/meta-python/base/instructions.py
rename to cranelift/codegen/meta-python/base/instructions.py
diff --git a/lib/codegen/meta-python/base/legalize.py b/cranelift/codegen/meta-python/base/legalize.py
similarity index 100%
rename from lib/codegen/meta-python/base/legalize.py
rename to cranelift/codegen/meta-python/base/legalize.py
diff --git a/lib/codegen/meta-python/base/predicates.py b/cranelift/codegen/meta-python/base/predicates.py
similarity index 100%
rename from lib/codegen/meta-python/base/predicates.py
rename to cranelift/codegen/meta-python/base/predicates.py
diff --git a/lib/codegen/meta-python/base/semantics.py b/cranelift/codegen/meta-python/base/semantics.py
similarity index 100%
rename from lib/codegen/meta-python/base/semantics.py
rename to cranelift/codegen/meta-python/base/semantics.py
diff --git a/lib/codegen/meta-python/base/settings.py b/cranelift/codegen/meta-python/base/settings.py
similarity index 100%
rename from lib/codegen/meta-python/base/settings.py
rename to cranelift/codegen/meta-python/base/settings.py
diff --git a/lib/codegen/meta-python/base/types.py b/cranelift/codegen/meta-python/base/types.py
similarity index 100%
rename from lib/codegen/meta-python/base/types.py
rename to cranelift/codegen/meta-python/base/types.py
diff --git a/lib/codegen/meta-python/build.py b/cranelift/codegen/meta-python/build.py
similarity index 90%
rename from lib/codegen/meta-python/build.py
rename to cranelift/codegen/meta-python/build.py
index 60fe124e21..37c23554b8 100644
--- a/lib/codegen/meta-python/build.py
+++ b/cranelift/codegen/meta-python/build.py
@@ -1,6 +1,6 @@
 # Second-level build script.
 #
-# This script is run from lib/codegen/build.rs to generate Rust files.
+# This script is run from cranelift-codegen/build.rs to generate Rust files.
 
 from __future__ import absolute_import
 import argparse
diff --git a/lib/codegen/meta-python/cdsl/__init__.py b/cranelift/codegen/meta-python/cdsl/__init__.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/__init__.py
rename to cranelift/codegen/meta-python/cdsl/__init__.py
diff --git a/lib/codegen/meta-python/cdsl/ast.py b/cranelift/codegen/meta-python/cdsl/ast.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/ast.py
rename to cranelift/codegen/meta-python/cdsl/ast.py
diff --git a/lib/codegen/meta-python/cdsl/formats.py b/cranelift/codegen/meta-python/cdsl/formats.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/formats.py
rename to cranelift/codegen/meta-python/cdsl/formats.py
diff --git a/lib/codegen/meta-python/cdsl/instructions.py b/cranelift/codegen/meta-python/cdsl/instructions.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/instructions.py
rename to cranelift/codegen/meta-python/cdsl/instructions.py
diff --git a/lib/codegen/meta-python/cdsl/isa.py b/cranelift/codegen/meta-python/cdsl/isa.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/isa.py
rename to cranelift/codegen/meta-python/cdsl/isa.py
diff --git a/lib/codegen/meta-python/cdsl/operands.py b/cranelift/codegen/meta-python/cdsl/operands.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/operands.py
rename to cranelift/codegen/meta-python/cdsl/operands.py
diff --git a/lib/codegen/meta-python/cdsl/predicates.py b/cranelift/codegen/meta-python/cdsl/predicates.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/predicates.py
rename to cranelift/codegen/meta-python/cdsl/predicates.py
diff --git a/lib/codegen/meta-python/cdsl/registers.py b/cranelift/codegen/meta-python/cdsl/registers.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/registers.py
rename to cranelift/codegen/meta-python/cdsl/registers.py
diff --git a/lib/codegen/meta-python/cdsl/settings.py b/cranelift/codegen/meta-python/cdsl/settings.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/settings.py
rename to cranelift/codegen/meta-python/cdsl/settings.py
diff --git a/lib/codegen/meta-python/cdsl/test_ast.py b/cranelift/codegen/meta-python/cdsl/test_ast.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/test_ast.py
rename to cranelift/codegen/meta-python/cdsl/test_ast.py
diff --git a/lib/codegen/meta-python/cdsl/test_package.py b/cranelift/codegen/meta-python/cdsl/test_package.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/test_package.py
rename to cranelift/codegen/meta-python/cdsl/test_package.py
diff --git a/lib/codegen/meta-python/cdsl/test_ti.py b/cranelift/codegen/meta-python/cdsl/test_ti.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/test_ti.py
rename to cranelift/codegen/meta-python/cdsl/test_ti.py
diff --git a/lib/codegen/meta-python/cdsl/test_typevar.py b/cranelift/codegen/meta-python/cdsl/test_typevar.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/test_typevar.py
rename to cranelift/codegen/meta-python/cdsl/test_typevar.py
diff --git a/lib/codegen/meta-python/cdsl/test_xform.py b/cranelift/codegen/meta-python/cdsl/test_xform.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/test_xform.py
rename to cranelift/codegen/meta-python/cdsl/test_xform.py
diff --git a/lib/codegen/meta-python/cdsl/ti.py b/cranelift/codegen/meta-python/cdsl/ti.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/ti.py
rename to cranelift/codegen/meta-python/cdsl/ti.py
diff --git a/lib/codegen/meta-python/cdsl/types.py b/cranelift/codegen/meta-python/cdsl/types.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/types.py
rename to cranelift/codegen/meta-python/cdsl/types.py
diff --git a/lib/codegen/meta-python/cdsl/typevar.py b/cranelift/codegen/meta-python/cdsl/typevar.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/typevar.py
rename to cranelift/codegen/meta-python/cdsl/typevar.py
diff --git a/lib/codegen/meta-python/cdsl/xform.py b/cranelift/codegen/meta-python/cdsl/xform.py
similarity index 100%
rename from lib/codegen/meta-python/cdsl/xform.py
rename to cranelift/codegen/meta-python/cdsl/xform.py
diff --git a/lib/codegen/meta-python/check.sh b/cranelift/codegen/meta-python/check.sh
similarity index 100%
rename from lib/codegen/meta-python/check.sh
rename to cranelift/codegen/meta-python/check.sh
diff --git a/lib/codegen/meta-python/constant_hash.py b/cranelift/codegen/meta-python/constant_hash.py
similarity index 100%
rename from lib/codegen/meta-python/constant_hash.py
rename to cranelift/codegen/meta-python/constant_hash.py
diff --git a/lib/codegen/meta-python/gen_binemit.py b/cranelift/codegen/meta-python/gen_binemit.py
similarity index 100%
rename from lib/codegen/meta-python/gen_binemit.py
rename to cranelift/codegen/meta-python/gen_binemit.py
diff --git a/lib/codegen/meta-python/gen_build_deps.py b/cranelift/codegen/meta-python/gen_build_deps.py
similarity index 91%
rename from lib/codegen/meta-python/gen_build_deps.py
rename to cranelift/codegen/meta-python/gen_build_deps.py
index 8423e72060..637865a55c 100644
--- a/lib/codegen/meta-python/gen_build_deps.py
+++ b/cranelift/codegen/meta-python/gen_build_deps.py
@@ -1,7 +1,7 @@
 """
 Generate build dependencies for Cargo.
 
-The `build.py` script is invoked by cargo when building lib/codegen to
+The `build.py` script is invoked by cargo when building cranelift-codegen to
 generate Rust code from the instruction descriptions. Cargo needs to know when
 it is necessary to rerun the build script.
 
diff --git a/lib/codegen/meta-python/gen_encoding.py b/cranelift/codegen/meta-python/gen_encoding.py
similarity index 100%
rename from lib/codegen/meta-python/gen_encoding.py
rename to cranelift/codegen/meta-python/gen_encoding.py
diff --git a/lib/codegen/meta-python/gen_instr.py b/cranelift/codegen/meta-python/gen_instr.py
similarity index 100%
rename from lib/codegen/meta-python/gen_instr.py
rename to cranelift/codegen/meta-python/gen_instr.py
diff --git a/lib/codegen/meta-python/gen_legalizer.py b/cranelift/codegen/meta-python/gen_legalizer.py
similarity index 100%
rename from lib/codegen/meta-python/gen_legalizer.py
rename to cranelift/codegen/meta-python/gen_legalizer.py
diff --git a/lib/codegen/meta-python/gen_settings.py b/cranelift/codegen/meta-python/gen_settings.py
similarity index 100%
rename from lib/codegen/meta-python/gen_settings.py
rename to cranelift/codegen/meta-python/gen_settings.py
diff --git a/lib/codegen/meta-python/isa/__init__.py b/cranelift/codegen/meta-python/isa/__init__.py
similarity index 100%
rename from lib/codegen/meta-python/isa/__init__.py
rename to cranelift/codegen/meta-python/isa/__init__.py
diff --git a/lib/codegen/meta-python/isa/arm32/__init__.py b/cranelift/codegen/meta-python/isa/arm32/__init__.py
similarity index 100%
rename from lib/codegen/meta-python/isa/arm32/__init__.py
rename to cranelift/codegen/meta-python/isa/arm32/__init__.py
diff --git a/lib/codegen/meta-python/isa/arm32/defs.py b/cranelift/codegen/meta-python/isa/arm32/defs.py
similarity index 100%
rename from lib/codegen/meta-python/isa/arm32/defs.py
rename to cranelift/codegen/meta-python/isa/arm32/defs.py
diff --git a/lib/codegen/meta-python/isa/arm32/registers.py b/cranelift/codegen/meta-python/isa/arm32/registers.py
similarity index 100%
rename from lib/codegen/meta-python/isa/arm32/registers.py
rename to cranelift/codegen/meta-python/isa/arm32/registers.py
diff --git a/lib/codegen/meta-python/isa/arm32/settings.py b/cranelift/codegen/meta-python/isa/arm32/settings.py
similarity index 100%
rename from lib/codegen/meta-python/isa/arm32/settings.py
rename to cranelift/codegen/meta-python/isa/arm32/settings.py
diff --git a/lib/codegen/meta-python/isa/arm64/__init__.py b/cranelift/codegen/meta-python/isa/arm64/__init__.py
similarity index 100%
rename from lib/codegen/meta-python/isa/arm64/__init__.py
rename to cranelift/codegen/meta-python/isa/arm64/__init__.py
diff --git a/lib/codegen/meta-python/isa/arm64/defs.py b/cranelift/codegen/meta-python/isa/arm64/defs.py
similarity index 100%
rename from lib/codegen/meta-python/isa/arm64/defs.py
rename to cranelift/codegen/meta-python/isa/arm64/defs.py
diff --git a/lib/codegen/meta-python/isa/arm64/registers.py b/cranelift/codegen/meta-python/isa/arm64/registers.py
similarity index 100%
rename from lib/codegen/meta-python/isa/arm64/registers.py
rename to cranelift/codegen/meta-python/isa/arm64/registers.py
diff --git a/lib/codegen/meta-python/isa/arm64/settings.py b/cranelift/codegen/meta-python/isa/arm64/settings.py
similarity index 100%
rename from lib/codegen/meta-python/isa/arm64/settings.py
rename to cranelift/codegen/meta-python/isa/arm64/settings.py
diff --git a/lib/codegen/meta-python/isa/riscv/__init__.py b/cranelift/codegen/meta-python/isa/riscv/__init__.py
similarity index 100%
rename from lib/codegen/meta-python/isa/riscv/__init__.py
rename to cranelift/codegen/meta-python/isa/riscv/__init__.py
diff --git a/lib/codegen/meta-python/isa/riscv/defs.py b/cranelift/codegen/meta-python/isa/riscv/defs.py
similarity index 100%
rename from lib/codegen/meta-python/isa/riscv/defs.py
rename to cranelift/codegen/meta-python/isa/riscv/defs.py
diff --git a/lib/codegen/meta-python/isa/riscv/encodings.py b/cranelift/codegen/meta-python/isa/riscv/encodings.py
similarity index 100%
rename from lib/codegen/meta-python/isa/riscv/encodings.py
rename to cranelift/codegen/meta-python/isa/riscv/encodings.py
diff --git a/lib/codegen/meta-python/isa/riscv/recipes.py b/cranelift/codegen/meta-python/isa/riscv/recipes.py
similarity index 100%
rename from lib/codegen/meta-python/isa/riscv/recipes.py
rename to cranelift/codegen/meta-python/isa/riscv/recipes.py
diff --git a/lib/codegen/meta-python/isa/riscv/registers.py b/cranelift/codegen/meta-python/isa/riscv/registers.py
similarity index 100%
rename from lib/codegen/meta-python/isa/riscv/registers.py
rename to cranelift/codegen/meta-python/isa/riscv/registers.py
diff --git a/lib/codegen/meta-python/isa/riscv/settings.py b/cranelift/codegen/meta-python/isa/riscv/settings.py
similarity index 100%
rename from lib/codegen/meta-python/isa/riscv/settings.py
rename to cranelift/codegen/meta-python/isa/riscv/settings.py
diff --git a/lib/codegen/meta-python/isa/x86/__init__.py b/cranelift/codegen/meta-python/isa/x86/__init__.py
similarity index 100%
rename from lib/codegen/meta-python/isa/x86/__init__.py
rename to cranelift/codegen/meta-python/isa/x86/__init__.py
diff --git a/lib/codegen/meta-python/isa/x86/defs.py b/cranelift/codegen/meta-python/isa/x86/defs.py
similarity index 100%
rename from lib/codegen/meta-python/isa/x86/defs.py
rename to cranelift/codegen/meta-python/isa/x86/defs.py
diff --git a/lib/codegen/meta-python/isa/x86/encodings.py b/cranelift/codegen/meta-python/isa/x86/encodings.py
similarity index 100%
rename from lib/codegen/meta-python/isa/x86/encodings.py
rename to cranelift/codegen/meta-python/isa/x86/encodings.py
diff --git a/lib/codegen/meta-python/isa/x86/instructions.py b/cranelift/codegen/meta-python/isa/x86/instructions.py
similarity index 100%
rename from lib/codegen/meta-python/isa/x86/instructions.py
rename to cranelift/codegen/meta-python/isa/x86/instructions.py
diff --git a/lib/codegen/meta-python/isa/x86/legalize.py b/cranelift/codegen/meta-python/isa/x86/legalize.py
similarity index 100%
rename from lib/codegen/meta-python/isa/x86/legalize.py
rename to cranelift/codegen/meta-python/isa/x86/legalize.py
diff --git a/lib/codegen/meta-python/isa/x86/recipes.py b/cranelift/codegen/meta-python/isa/x86/recipes.py
similarity index 100%
rename from lib/codegen/meta-python/isa/x86/recipes.py
rename to cranelift/codegen/meta-python/isa/x86/recipes.py
diff --git a/lib/codegen/meta-python/isa/x86/registers.py b/cranelift/codegen/meta-python/isa/x86/registers.py
similarity index 100%
rename from lib/codegen/meta-python/isa/x86/registers.py
rename to cranelift/codegen/meta-python/isa/x86/registers.py
diff --git a/lib/codegen/meta-python/isa/x86/settings.py b/cranelift/codegen/meta-python/isa/x86/settings.py
similarity index 100%
rename from lib/codegen/meta-python/isa/x86/settings.py
rename to cranelift/codegen/meta-python/isa/x86/settings.py
diff --git a/lib/codegen/meta-python/mypy.ini b/cranelift/codegen/meta-python/mypy.ini
similarity index 100%
rename from lib/codegen/meta-python/mypy.ini
rename to cranelift/codegen/meta-python/mypy.ini
diff --git a/lib/codegen/meta-python/semantics/__init__.py b/cranelift/codegen/meta-python/semantics/__init__.py
similarity index 100%
rename from lib/codegen/meta-python/semantics/__init__.py
rename to cranelift/codegen/meta-python/semantics/__init__.py
diff --git a/lib/codegen/meta-python/semantics/elaborate.py b/cranelift/codegen/meta-python/semantics/elaborate.py
similarity index 100%
rename from lib/codegen/meta-python/semantics/elaborate.py
rename to cranelift/codegen/meta-python/semantics/elaborate.py
diff --git a/lib/codegen/meta-python/semantics/macros.py b/cranelift/codegen/meta-python/semantics/macros.py
similarity index 100%
rename from lib/codegen/meta-python/semantics/macros.py
rename to cranelift/codegen/meta-python/semantics/macros.py
diff --git a/lib/codegen/meta-python/semantics/primitives.py b/cranelift/codegen/meta-python/semantics/primitives.py
similarity index 100%
rename from lib/codegen/meta-python/semantics/primitives.py
rename to cranelift/codegen/meta-python/semantics/primitives.py
diff --git a/lib/codegen/meta-python/semantics/smtlib.py b/cranelift/codegen/meta-python/semantics/smtlib.py
similarity index 100%
rename from lib/codegen/meta-python/semantics/smtlib.py
rename to cranelift/codegen/meta-python/semantics/smtlib.py
diff --git a/lib/codegen/meta-python/semantics/test_elaborate.py b/cranelift/codegen/meta-python/semantics/test_elaborate.py
similarity index 100%
rename from lib/codegen/meta-python/semantics/test_elaborate.py
rename to cranelift/codegen/meta-python/semantics/test_elaborate.py
diff --git a/lib/codegen/meta-python/srcgen.py b/cranelift/codegen/meta-python/srcgen.py
similarity index 100%
rename from lib/codegen/meta-python/srcgen.py
rename to cranelift/codegen/meta-python/srcgen.py
diff --git a/lib/codegen/meta-python/stubs/z3/__init__.pyi b/cranelift/codegen/meta-python/stubs/z3/__init__.pyi
similarity index 100%
rename from lib/codegen/meta-python/stubs/z3/__init__.pyi
rename to cranelift/codegen/meta-python/stubs/z3/__init__.pyi
diff --git a/lib/codegen/meta-python/stubs/z3/z3core.pyi b/cranelift/codegen/meta-python/stubs/z3/z3core.pyi
similarity index 100%
rename from lib/codegen/meta-python/stubs/z3/z3core.pyi
rename to cranelift/codegen/meta-python/stubs/z3/z3core.pyi
diff --git a/lib/codegen/meta-python/stubs/z3/z3types.pyi b/cranelift/codegen/meta-python/stubs/z3/z3types.pyi
similarity index 100%
rename from lib/codegen/meta-python/stubs/z3/z3types.pyi
rename to cranelift/codegen/meta-python/stubs/z3/z3types.pyi
diff --git a/lib/codegen/meta-python/test_constant_hash.py b/cranelift/codegen/meta-python/test_constant_hash.py
similarity index 100%
rename from lib/codegen/meta-python/test_constant_hash.py
rename to cranelift/codegen/meta-python/test_constant_hash.py
diff --git a/lib/codegen/meta-python/test_gen_legalizer.py b/cranelift/codegen/meta-python/test_gen_legalizer.py
similarity index 100%
rename from lib/codegen/meta-python/test_gen_legalizer.py
rename to cranelift/codegen/meta-python/test_gen_legalizer.py
diff --git a/lib/codegen/meta-python/test_srcgen.py b/cranelift/codegen/meta-python/test_srcgen.py
similarity index 100%
rename from lib/codegen/meta-python/test_srcgen.py
rename to cranelift/codegen/meta-python/test_srcgen.py
diff --git a/lib/codegen/meta-python/unique_table.py b/cranelift/codegen/meta-python/unique_table.py
similarity index 100%
rename from lib/codegen/meta-python/unique_table.py
rename to cranelift/codegen/meta-python/unique_table.py
diff --git a/lib/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
similarity index 85%
rename from lib/codegen/meta/Cargo.toml
rename to cranelift/codegen/meta/Cargo.toml
index 3fdfbb9305..0cb6b40abc 100644
--- a/lib/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../entity", version = "0.28.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.28.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/codegen/meta/LICENSE b/cranelift/codegen/meta/LICENSE
similarity index 100%
rename from lib/codegen/meta/LICENSE
rename to cranelift/codegen/meta/LICENSE
diff --git a/lib/codegen/meta/README.md b/cranelift/codegen/meta/README.md
similarity index 100%
rename from lib/codegen/meta/README.md
rename to cranelift/codegen/meta/README.md
diff --git a/lib/codegen/meta/src/base/mod.rs b/cranelift/codegen/meta/src/base/mod.rs
similarity index 100%
rename from lib/codegen/meta/src/base/mod.rs
rename to cranelift/codegen/meta/src/base/mod.rs
diff --git a/lib/codegen/meta/src/base/settings.rs b/cranelift/codegen/meta/src/base/settings.rs
similarity index 100%
rename from lib/codegen/meta/src/base/settings.rs
rename to cranelift/codegen/meta/src/base/settings.rs
diff --git a/lib/codegen/meta/src/base/types.rs b/cranelift/codegen/meta/src/base/types.rs
similarity index 100%
rename from lib/codegen/meta/src/base/types.rs
rename to cranelift/codegen/meta/src/base/types.rs
diff --git a/lib/codegen/meta/src/cdsl/isa.rs b/cranelift/codegen/meta/src/cdsl/isa.rs
similarity index 100%
rename from lib/codegen/meta/src/cdsl/isa.rs
rename to cranelift/codegen/meta/src/cdsl/isa.rs
diff --git a/lib/codegen/meta/src/cdsl/mod.rs b/cranelift/codegen/meta/src/cdsl/mod.rs
similarity index 100%
rename from lib/codegen/meta/src/cdsl/mod.rs
rename to cranelift/codegen/meta/src/cdsl/mod.rs
diff --git a/lib/codegen/meta/src/cdsl/regs.rs b/cranelift/codegen/meta/src/cdsl/regs.rs
similarity index 100%
rename from lib/codegen/meta/src/cdsl/regs.rs
rename to cranelift/codegen/meta/src/cdsl/regs.rs
diff --git a/lib/codegen/meta/src/cdsl/settings.rs b/cranelift/codegen/meta/src/cdsl/settings.rs
similarity index 100%
rename from lib/codegen/meta/src/cdsl/settings.rs
rename to cranelift/codegen/meta/src/cdsl/settings.rs
diff --git a/lib/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
similarity index 100%
rename from lib/codegen/meta/src/cdsl/types.rs
rename to cranelift/codegen/meta/src/cdsl/types.rs
diff --git a/lib/codegen/meta/src/constant_hash.rs b/cranelift/codegen/meta/src/constant_hash.rs
similarity index 100%
rename from lib/codegen/meta/src/constant_hash.rs
rename to cranelift/codegen/meta/src/constant_hash.rs
diff --git a/lib/codegen/meta/src/error.rs b/cranelift/codegen/meta/src/error.rs
similarity index 100%
rename from lib/codegen/meta/src/error.rs
rename to cranelift/codegen/meta/src/error.rs
diff --git a/lib/codegen/meta/src/gen_registers.rs b/cranelift/codegen/meta/src/gen_registers.rs
similarity index 100%
rename from lib/codegen/meta/src/gen_registers.rs
rename to cranelift/codegen/meta/src/gen_registers.rs
diff --git a/lib/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
similarity index 100%
rename from lib/codegen/meta/src/gen_settings.rs
rename to cranelift/codegen/meta/src/gen_settings.rs
diff --git a/lib/codegen/meta/src/gen_types.rs b/cranelift/codegen/meta/src/gen_types.rs
similarity index 96%
rename from lib/codegen/meta/src/gen_types.rs
rename to cranelift/codegen/meta/src/gen_types.rs
index 5a9f9f8c5e..6332525306 100644
--- a/lib/codegen/meta/src/gen_types.rs
+++ b/cranelift/codegen/meta/src/gen_types.rs
@@ -1,7 +1,7 @@
 //! Generate sources with type info.
 //!
 //! This generates a `types.rs` file which is included in
-//! `lib/codegen/ir/types.rs`. The file provides constant definitions for the
+//! `cranelift-codegen/ir/types.rs`. The file provides constant definitions for the
 //! most commonly used types, including all of the scalar types.
 //!
 //! This ensures that the metaprogram and the generated program see the same
diff --git a/lib/codegen/meta/src/isa/arm32/mod.rs b/cranelift/codegen/meta/src/isa/arm32/mod.rs
similarity index 100%
rename from lib/codegen/meta/src/isa/arm32/mod.rs
rename to cranelift/codegen/meta/src/isa/arm32/mod.rs
diff --git a/lib/codegen/meta/src/isa/arm64/mod.rs b/cranelift/codegen/meta/src/isa/arm64/mod.rs
similarity index 100%
rename from lib/codegen/meta/src/isa/arm64/mod.rs
rename to cranelift/codegen/meta/src/isa/arm64/mod.rs
diff --git a/lib/codegen/meta/src/isa/mod.rs b/cranelift/codegen/meta/src/isa/mod.rs
similarity index 100%
rename from lib/codegen/meta/src/isa/mod.rs
rename to cranelift/codegen/meta/src/isa/mod.rs
diff --git a/lib/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
similarity index 100%
rename from lib/codegen/meta/src/isa/riscv/mod.rs
rename to cranelift/codegen/meta/src/isa/riscv/mod.rs
diff --git a/lib/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
similarity index 100%
rename from lib/codegen/meta/src/isa/x86/mod.rs
rename to cranelift/codegen/meta/src/isa/x86/mod.rs
diff --git a/lib/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
similarity index 100%
rename from lib/codegen/meta/src/lib.rs
rename to cranelift/codegen/meta/src/lib.rs
diff --git a/lib/codegen/meta/src/srcgen.rs b/cranelift/codegen/meta/src/srcgen.rs
similarity index 100%
rename from lib/codegen/meta/src/srcgen.rs
rename to cranelift/codegen/meta/src/srcgen.rs
diff --git a/lib/codegen/meta/src/unique_table.rs b/cranelift/codegen/meta/src/unique_table.rs
similarity index 100%
rename from lib/codegen/meta/src/unique_table.rs
rename to cranelift/codegen/meta/src/unique_table.rs
diff --git a/lib/codegen/src/abi.rs b/cranelift/codegen/src/abi.rs
similarity index 100%
rename from lib/codegen/src/abi.rs
rename to cranelift/codegen/src/abi.rs
diff --git a/lib/codegen/src/binemit/memorysink.rs b/cranelift/codegen/src/binemit/memorysink.rs
similarity index 100%
rename from lib/codegen/src/binemit/memorysink.rs
rename to cranelift/codegen/src/binemit/memorysink.rs
diff --git a/lib/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
similarity index 100%
rename from lib/codegen/src/binemit/mod.rs
rename to cranelift/codegen/src/binemit/mod.rs
diff --git a/lib/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
similarity index 100%
rename from lib/codegen/src/binemit/relaxation.rs
rename to cranelift/codegen/src/binemit/relaxation.rs
diff --git a/lib/codegen/src/binemit/shrink.rs b/cranelift/codegen/src/binemit/shrink.rs
similarity index 100%
rename from lib/codegen/src/binemit/shrink.rs
rename to cranelift/codegen/src/binemit/shrink.rs
diff --git a/lib/codegen/src/bitset.rs b/cranelift/codegen/src/bitset.rs
similarity index 100%
rename from lib/codegen/src/bitset.rs
rename to cranelift/codegen/src/bitset.rs
diff --git a/lib/codegen/src/cfg_printer.rs b/cranelift/codegen/src/cfg_printer.rs
similarity index 100%
rename from lib/codegen/src/cfg_printer.rs
rename to cranelift/codegen/src/cfg_printer.rs
diff --git a/lib/codegen/src/constant_hash.rs b/cranelift/codegen/src/constant_hash.rs
similarity index 91%
rename from lib/codegen/src/constant_hash.rs
rename to cranelift/codegen/src/constant_hash.rs
index 2c3bca1a20..d6afc5eed0 100644
--- a/lib/codegen/src/constant_hash.rs
+++ b/cranelift/codegen/src/constant_hash.rs
@@ -1,6 +1,6 @@
 //! Runtime support for precomputed constant hash tables.
 //!
-//! The `lib/codegen/meta-python/constant_hash.py` Python module can generate constant hash tables
+//! The `cranelift-codegen/meta-python/constant_hash.py` Python module can generate constant hash tables
 //! using open addressing and quadratic probing. The hash tables are arrays that are guaranteed to:
 //!
 //! - Have a power-of-two size.
@@ -56,7 +56,7 @@ pub fn probe + ?Sized>(
 }
 
 /// A primitive hash function for matching opcodes.
-/// Must match `lib/codegen/meta-python/constant_hash.py` and `lib/codegen/meta/constant_hash.rs`.
+/// Must match `cranelift-codegen/meta-python/constant_hash.py` and `cranelift-codegen/meta/constant_hash.rs`.
 pub fn simple_hash(s: &str) -> usize {
     let mut h: u32 = 5381;
     for c in s.chars() {
diff --git a/lib/codegen/src/context.rs b/cranelift/codegen/src/context.rs
similarity index 100%
rename from lib/codegen/src/context.rs
rename to cranelift/codegen/src/context.rs
diff --git a/lib/codegen/src/cursor.rs b/cranelift/codegen/src/cursor.rs
similarity index 100%
rename from lib/codegen/src/cursor.rs
rename to cranelift/codegen/src/cursor.rs
diff --git a/lib/codegen/src/dbg.rs b/cranelift/codegen/src/dbg.rs
similarity index 100%
rename from lib/codegen/src/dbg.rs
rename to cranelift/codegen/src/dbg.rs
diff --git a/lib/codegen/src/dce.rs b/cranelift/codegen/src/dce.rs
similarity index 100%
rename from lib/codegen/src/dce.rs
rename to cranelift/codegen/src/dce.rs
diff --git a/lib/codegen/src/divconst_magic_numbers.rs b/cranelift/codegen/src/divconst_magic_numbers.rs
similarity index 100%
rename from lib/codegen/src/divconst_magic_numbers.rs
rename to cranelift/codegen/src/divconst_magic_numbers.rs
diff --git a/lib/codegen/src/dominator_tree.rs b/cranelift/codegen/src/dominator_tree.rs
similarity index 100%
rename from lib/codegen/src/dominator_tree.rs
rename to cranelift/codegen/src/dominator_tree.rs
diff --git a/lib/codegen/src/flowgraph.rs b/cranelift/codegen/src/flowgraph.rs
similarity index 100%
rename from lib/codegen/src/flowgraph.rs
rename to cranelift/codegen/src/flowgraph.rs
diff --git a/lib/codegen/src/fx.rs b/cranelift/codegen/src/fx.rs
similarity index 100%
rename from lib/codegen/src/fx.rs
rename to cranelift/codegen/src/fx.rs
diff --git a/lib/codegen/src/ir/builder.rs b/cranelift/codegen/src/ir/builder.rs
similarity index 99%
rename from lib/codegen/src/ir/builder.rs
rename to cranelift/codegen/src/ir/builder.rs
index f84e23eab8..43578d5378 100644
--- a/lib/codegen/src/ir/builder.rs
+++ b/cranelift/codegen/src/ir/builder.rs
@@ -32,7 +32,7 @@ pub trait InstBuilderBase<'f>: Sized {
     fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'f mut DataFlowGraph);
 }
 
-// Include trait code generated by `lib/codegen/meta-python/gen_instr.py`.
+// Include trait code generated by `cranelift-codegen/meta-python/gen_instr.py`.
 //
 // This file defines the `InstBuilder` trait as an extension of `InstBuilderBase` with methods per
 // instruction format and per opcode.
diff --git a/lib/codegen/src/ir/condcodes.rs b/cranelift/codegen/src/ir/condcodes.rs
similarity index 100%
rename from lib/codegen/src/ir/condcodes.rs
rename to cranelift/codegen/src/ir/condcodes.rs
diff --git a/lib/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
similarity index 100%
rename from lib/codegen/src/ir/dfg.rs
rename to cranelift/codegen/src/ir/dfg.rs
diff --git a/lib/codegen/src/ir/entities.rs b/cranelift/codegen/src/ir/entities.rs
similarity index 100%
rename from lib/codegen/src/ir/entities.rs
rename to cranelift/codegen/src/ir/entities.rs
diff --git a/lib/codegen/src/ir/extfunc.rs b/cranelift/codegen/src/ir/extfunc.rs
similarity index 100%
rename from lib/codegen/src/ir/extfunc.rs
rename to cranelift/codegen/src/ir/extfunc.rs
diff --git a/lib/codegen/src/ir/extname.rs b/cranelift/codegen/src/ir/extname.rs
similarity index 100%
rename from lib/codegen/src/ir/extname.rs
rename to cranelift/codegen/src/ir/extname.rs
diff --git a/lib/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
similarity index 100%
rename from lib/codegen/src/ir/function.rs
rename to cranelift/codegen/src/ir/function.rs
diff --git a/lib/codegen/src/ir/globalvalue.rs b/cranelift/codegen/src/ir/globalvalue.rs
similarity index 100%
rename from lib/codegen/src/ir/globalvalue.rs
rename to cranelift/codegen/src/ir/globalvalue.rs
diff --git a/lib/codegen/src/ir/heap.rs b/cranelift/codegen/src/ir/heap.rs
similarity index 100%
rename from lib/codegen/src/ir/heap.rs
rename to cranelift/codegen/src/ir/heap.rs
diff --git a/lib/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
similarity index 100%
rename from lib/codegen/src/ir/immediates.rs
rename to cranelift/codegen/src/ir/immediates.rs
diff --git a/lib/codegen/src/ir/instructions.rs b/cranelift/codegen/src/ir/instructions.rs
similarity index 99%
rename from lib/codegen/src/ir/instructions.rs
rename to cranelift/codegen/src/ir/instructions.rs
index ee4a0b416a..ae227dc5f7 100644
--- a/lib/codegen/src/ir/instructions.rs
+++ b/cranelift/codegen/src/ir/instructions.rs
@@ -28,7 +28,7 @@ pub type ValueList = entity::EntityList;
 /// Memory pool for holding value lists. See `ValueList`.
 pub type ValueListPool = entity::ListPool;
 
-// Include code generated by `lib/codegen/meta-python/gen_instr.py`. This file contains:
+// Include code generated by `cranelift-codegen/meta-python/gen_instr.py`. This file contains:
 //
 // - The `pub enum InstructionFormat` enum with all the instruction formats.
 // - The `pub enum InstructionData` enum with all the instruction data fields.
@@ -64,7 +64,7 @@ impl Opcode {
     }
 }
 
-// This trait really belongs in lib/reader where it is used by the `.clif` file parser, but since
+// This trait really belongs in cranelift-reader where it is used by the `.clif` file parser, but since
 // it critically depends on the `opcode_name()` function which is needed here anyway, it lives in
 // this module. This also saves us from running the build script twice to generate code for the two
 // separate crates.
diff --git a/lib/codegen/src/ir/jumptable.rs b/cranelift/codegen/src/ir/jumptable.rs
similarity index 100%
rename from lib/codegen/src/ir/jumptable.rs
rename to cranelift/codegen/src/ir/jumptable.rs
diff --git a/lib/codegen/src/ir/layout.rs b/cranelift/codegen/src/ir/layout.rs
similarity index 100%
rename from lib/codegen/src/ir/layout.rs
rename to cranelift/codegen/src/ir/layout.rs
diff --git a/lib/codegen/src/ir/libcall.rs b/cranelift/codegen/src/ir/libcall.rs
similarity index 100%
rename from lib/codegen/src/ir/libcall.rs
rename to cranelift/codegen/src/ir/libcall.rs
diff --git a/lib/codegen/src/ir/memflags.rs b/cranelift/codegen/src/ir/memflags.rs
similarity index 100%
rename from lib/codegen/src/ir/memflags.rs
rename to cranelift/codegen/src/ir/memflags.rs
diff --git a/lib/codegen/src/ir/mod.rs b/cranelift/codegen/src/ir/mod.rs
similarity index 100%
rename from lib/codegen/src/ir/mod.rs
rename to cranelift/codegen/src/ir/mod.rs
diff --git a/lib/codegen/src/ir/progpoint.rs b/cranelift/codegen/src/ir/progpoint.rs
similarity index 100%
rename from lib/codegen/src/ir/progpoint.rs
rename to cranelift/codegen/src/ir/progpoint.rs
diff --git a/lib/codegen/src/ir/sourceloc.rs b/cranelift/codegen/src/ir/sourceloc.rs
similarity index 100%
rename from lib/codegen/src/ir/sourceloc.rs
rename to cranelift/codegen/src/ir/sourceloc.rs
diff --git a/lib/codegen/src/ir/stackslot.rs b/cranelift/codegen/src/ir/stackslot.rs
similarity index 100%
rename from lib/codegen/src/ir/stackslot.rs
rename to cranelift/codegen/src/ir/stackslot.rs
diff --git a/lib/codegen/src/ir/table.rs b/cranelift/codegen/src/ir/table.rs
similarity index 100%
rename from lib/codegen/src/ir/table.rs
rename to cranelift/codegen/src/ir/table.rs
diff --git a/lib/codegen/src/ir/trapcode.rs b/cranelift/codegen/src/ir/trapcode.rs
similarity index 100%
rename from lib/codegen/src/ir/trapcode.rs
rename to cranelift/codegen/src/ir/trapcode.rs
diff --git a/lib/codegen/src/ir/types.rs b/cranelift/codegen/src/ir/types.rs
similarity index 99%
rename from lib/codegen/src/ir/types.rs
rename to cranelift/codegen/src/ir/types.rs
index 2d946e04f8..eee39c83e4 100644
--- a/lib/codegen/src/ir/types.rs
+++ b/cranelift/codegen/src/ir/types.rs
@@ -31,7 +31,7 @@ const LANE_BASE: u8 = 0x70;
 /// Start of the 2-lane vector types.
 const VECTOR_BASE: u8 = LANE_BASE + 16;
 
-// Include code generated by `lib/codegen/meta/gen_types.rs`. This file contains constant
+// Include code generated by `cranelift-codegen/meta/gen_types.rs`. This file contains constant
 // definitions for all the scalar types as well as common vector types for 64, 128, 256, and
 // 512-bit SIMD vectors.
 include!(concat!(env!("OUT_DIR"), "/types.rs"));
diff --git a/lib/codegen/src/ir/valueloc.rs b/cranelift/codegen/src/ir/valueloc.rs
similarity index 100%
rename from lib/codegen/src/ir/valueloc.rs
rename to cranelift/codegen/src/ir/valueloc.rs
diff --git a/lib/codegen/src/isa/arm32/abi.rs b/cranelift/codegen/src/isa/arm32/abi.rs
similarity index 100%
rename from lib/codegen/src/isa/arm32/abi.rs
rename to cranelift/codegen/src/isa/arm32/abi.rs
diff --git a/lib/codegen/src/isa/arm32/binemit.rs b/cranelift/codegen/src/isa/arm32/binemit.rs
similarity index 100%
rename from lib/codegen/src/isa/arm32/binemit.rs
rename to cranelift/codegen/src/isa/arm32/binemit.rs
diff --git a/lib/codegen/src/isa/arm32/enc_tables.rs b/cranelift/codegen/src/isa/arm32/enc_tables.rs
similarity index 100%
rename from lib/codegen/src/isa/arm32/enc_tables.rs
rename to cranelift/codegen/src/isa/arm32/enc_tables.rs
diff --git a/lib/codegen/src/isa/arm32/mod.rs b/cranelift/codegen/src/isa/arm32/mod.rs
similarity index 100%
rename from lib/codegen/src/isa/arm32/mod.rs
rename to cranelift/codegen/src/isa/arm32/mod.rs
diff --git a/lib/codegen/src/isa/arm32/registers.rs b/cranelift/codegen/src/isa/arm32/registers.rs
similarity index 100%
rename from lib/codegen/src/isa/arm32/registers.rs
rename to cranelift/codegen/src/isa/arm32/registers.rs
diff --git a/lib/codegen/src/isa/arm32/settings.rs b/cranelift/codegen/src/isa/arm32/settings.rs
similarity index 55%
rename from lib/codegen/src/isa/arm32/settings.rs
rename to cranelift/codegen/src/isa/arm32/settings.rs
index 5490c8c2b3..084c142399 100644
--- a/lib/codegen/src/isa/arm32/settings.rs
+++ b/cranelift/codegen/src/isa/arm32/settings.rs
@@ -3,7 +3,7 @@
 use crate::settings::{self, detail, Builder};
 use core::fmt;
 
-// Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
+// Include code generated by `cranelift-codegen/meta-python/gen_settings.py`. This file contains a public
 // `Flags` struct with an impl for all of the settings defined in
-// `lib/codegen/meta-python/isa/arm32/settings.py`.
+// `cranelift-codegen/meta-python/isa/arm32/settings.py`.
 include!(concat!(env!("OUT_DIR"), "/settings-arm32.rs"));
diff --git a/lib/codegen/src/isa/arm64/abi.rs b/cranelift/codegen/src/isa/arm64/abi.rs
similarity index 100%
rename from lib/codegen/src/isa/arm64/abi.rs
rename to cranelift/codegen/src/isa/arm64/abi.rs
diff --git a/lib/codegen/src/isa/arm64/binemit.rs b/cranelift/codegen/src/isa/arm64/binemit.rs
similarity index 100%
rename from lib/codegen/src/isa/arm64/binemit.rs
rename to cranelift/codegen/src/isa/arm64/binemit.rs
diff --git a/lib/codegen/src/isa/arm64/enc_tables.rs b/cranelift/codegen/src/isa/arm64/enc_tables.rs
similarity index 100%
rename from lib/codegen/src/isa/arm64/enc_tables.rs
rename to cranelift/codegen/src/isa/arm64/enc_tables.rs
diff --git a/lib/codegen/src/isa/arm64/mod.rs b/cranelift/codegen/src/isa/arm64/mod.rs
similarity index 100%
rename from lib/codegen/src/isa/arm64/mod.rs
rename to cranelift/codegen/src/isa/arm64/mod.rs
diff --git a/lib/codegen/src/isa/arm64/registers.rs b/cranelift/codegen/src/isa/arm64/registers.rs
similarity index 100%
rename from lib/codegen/src/isa/arm64/registers.rs
rename to cranelift/codegen/src/isa/arm64/registers.rs
diff --git a/lib/codegen/src/isa/arm64/settings.rs b/cranelift/codegen/src/isa/arm64/settings.rs
similarity index 55%
rename from lib/codegen/src/isa/arm64/settings.rs
rename to cranelift/codegen/src/isa/arm64/settings.rs
index e7d0abdf6c..b7be97e368 100644
--- a/lib/codegen/src/isa/arm64/settings.rs
+++ b/cranelift/codegen/src/isa/arm64/settings.rs
@@ -3,7 +3,7 @@
 use crate::settings::{self, detail, Builder};
 use core::fmt;
 
-// Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
+// Include code generated by `cranelift-codegen/meta-python/gen_settings.py`. This file contains a public
 // `Flags` struct with an impl for all of the settings defined in
-// `lib/codegen/meta-python/isa/arm64/settings.py`.
+// `cranelift-codegen/meta-python/isa/arm64/settings.py`.
 include!(concat!(env!("OUT_DIR"), "/settings-arm64.rs"));
diff --git a/lib/codegen/src/isa/call_conv.rs b/cranelift/codegen/src/isa/call_conv.rs
similarity index 100%
rename from lib/codegen/src/isa/call_conv.rs
rename to cranelift/codegen/src/isa/call_conv.rs
diff --git a/lib/codegen/src/isa/constraints.rs b/cranelift/codegen/src/isa/constraints.rs
similarity index 100%
rename from lib/codegen/src/isa/constraints.rs
rename to cranelift/codegen/src/isa/constraints.rs
diff --git a/lib/codegen/src/isa/enc_tables.rs b/cranelift/codegen/src/isa/enc_tables.rs
similarity index 99%
rename from lib/codegen/src/isa/enc_tables.rs
rename to cranelift/codegen/src/isa/enc_tables.rs
index 323473fe2b..45ad86441b 100644
--- a/lib/codegen/src/isa/enc_tables.rs
+++ b/cranelift/codegen/src/isa/enc_tables.rs
@@ -1,7 +1,7 @@
 //! Support types for generated encoding tables.
 //!
 //! This module contains types and functions for working with the encoding tables generated by
-//! `lib/codegen/meta-python/gen_encoding.py`.
+//! `cranelift-codegen/meta-python/gen_encoding.py`.
 
 use crate::constant_hash::{probe, Table};
 use crate::ir::{Function, InstructionData, Opcode, Type};
diff --git a/lib/codegen/src/isa/encoding.rs b/cranelift/codegen/src/isa/encoding.rs
similarity index 100%
rename from lib/codegen/src/isa/encoding.rs
rename to cranelift/codegen/src/isa/encoding.rs
diff --git a/lib/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
similarity index 100%
rename from lib/codegen/src/isa/mod.rs
rename to cranelift/codegen/src/isa/mod.rs
diff --git a/lib/codegen/src/isa/registers.rs b/cranelift/codegen/src/isa/registers.rs
similarity index 100%
rename from lib/codegen/src/isa/registers.rs
rename to cranelift/codegen/src/isa/registers.rs
diff --git a/lib/codegen/src/isa/riscv/abi.rs b/cranelift/codegen/src/isa/riscv/abi.rs
similarity index 100%
rename from lib/codegen/src/isa/riscv/abi.rs
rename to cranelift/codegen/src/isa/riscv/abi.rs
diff --git a/lib/codegen/src/isa/riscv/binemit.rs b/cranelift/codegen/src/isa/riscv/binemit.rs
similarity index 100%
rename from lib/codegen/src/isa/riscv/binemit.rs
rename to cranelift/codegen/src/isa/riscv/binemit.rs
diff --git a/lib/codegen/src/isa/riscv/enc_tables.rs b/cranelift/codegen/src/isa/riscv/enc_tables.rs
similarity index 100%
rename from lib/codegen/src/isa/riscv/enc_tables.rs
rename to cranelift/codegen/src/isa/riscv/enc_tables.rs
diff --git a/lib/codegen/src/isa/riscv/mod.rs b/cranelift/codegen/src/isa/riscv/mod.rs
similarity index 100%
rename from lib/codegen/src/isa/riscv/mod.rs
rename to cranelift/codegen/src/isa/riscv/mod.rs
diff --git a/lib/codegen/src/isa/riscv/registers.rs b/cranelift/codegen/src/isa/riscv/registers.rs
similarity index 100%
rename from lib/codegen/src/isa/riscv/registers.rs
rename to cranelift/codegen/src/isa/riscv/registers.rs
diff --git a/lib/codegen/src/isa/riscv/settings.rs b/cranelift/codegen/src/isa/riscv/settings.rs
similarity index 90%
rename from lib/codegen/src/isa/riscv/settings.rs
rename to cranelift/codegen/src/isa/riscv/settings.rs
index 660c1ef669..26999b87ee 100644
--- a/lib/codegen/src/isa/riscv/settings.rs
+++ b/cranelift/codegen/src/isa/riscv/settings.rs
@@ -3,9 +3,9 @@
 use crate::settings::{self, detail, Builder};
 use core::fmt;
 
-// Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
+// Include code generated by `cranelift-codegen/meta-python/gen_settings.py`. This file contains a public
 // `Flags` struct with an impl for all of the settings defined in
-// `lib/codegen/meta-python/isa/riscv/settings.py`.
+// `cranelift-codegen/meta-python/isa/riscv/settings.py`.
 include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs"));
 
 #[cfg(test)]
diff --git a/lib/codegen/src/isa/stack.rs b/cranelift/codegen/src/isa/stack.rs
similarity index 100%
rename from lib/codegen/src/isa/stack.rs
rename to cranelift/codegen/src/isa/stack.rs
diff --git a/lib/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
similarity index 100%
rename from lib/codegen/src/isa/x86/abi.rs
rename to cranelift/codegen/src/isa/x86/abi.rs
diff --git a/lib/codegen/src/isa/x86/binemit.rs b/cranelift/codegen/src/isa/x86/binemit.rs
similarity index 100%
rename from lib/codegen/src/isa/x86/binemit.rs
rename to cranelift/codegen/src/isa/x86/binemit.rs
diff --git a/lib/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
similarity index 100%
rename from lib/codegen/src/isa/x86/enc_tables.rs
rename to cranelift/codegen/src/isa/x86/enc_tables.rs
diff --git a/lib/codegen/src/isa/x86/mod.rs b/cranelift/codegen/src/isa/x86/mod.rs
similarity index 100%
rename from lib/codegen/src/isa/x86/mod.rs
rename to cranelift/codegen/src/isa/x86/mod.rs
diff --git a/lib/codegen/src/isa/x86/registers.rs b/cranelift/codegen/src/isa/x86/registers.rs
similarity index 100%
rename from lib/codegen/src/isa/x86/registers.rs
rename to cranelift/codegen/src/isa/x86/registers.rs
diff --git a/lib/codegen/src/isa/x86/settings.rs b/cranelift/codegen/src/isa/x86/settings.rs
similarity index 89%
rename from lib/codegen/src/isa/x86/settings.rs
rename to cranelift/codegen/src/isa/x86/settings.rs
index b638ea7ee8..ab5e080662 100644
--- a/lib/codegen/src/isa/x86/settings.rs
+++ b/cranelift/codegen/src/isa/x86/settings.rs
@@ -3,9 +3,9 @@
 use crate::settings::{self, detail, Builder};
 use core::fmt;
 
-// Include code generated by `lib/codegen/meta-python/gen_settings.py`. This file contains a public
+// Include code generated by `cranelift-codegen/meta-python/gen_settings.py`. This file contains a public
 // `Flags` struct with an impl for all of the settings defined in
-// `lib/codegen/meta-python/isa/x86/settings.py`.
+// `cranelift-codegen/meta-python/isa/x86/settings.py`.
 include!(concat!(env!("OUT_DIR"), "/settings-x86.rs"));
 
 #[cfg(test)]
diff --git a/lib/codegen/src/iterators.rs b/cranelift/codegen/src/iterators.rs
similarity index 100%
rename from lib/codegen/src/iterators.rs
rename to cranelift/codegen/src/iterators.rs
diff --git a/lib/codegen/src/legalizer/boundary.rs b/cranelift/codegen/src/legalizer/boundary.rs
similarity index 100%
rename from lib/codegen/src/legalizer/boundary.rs
rename to cranelift/codegen/src/legalizer/boundary.rs
diff --git a/lib/codegen/src/legalizer/call.rs b/cranelift/codegen/src/legalizer/call.rs
similarity index 100%
rename from lib/codegen/src/legalizer/call.rs
rename to cranelift/codegen/src/legalizer/call.rs
diff --git a/lib/codegen/src/legalizer/globalvalue.rs b/cranelift/codegen/src/legalizer/globalvalue.rs
similarity index 100%
rename from lib/codegen/src/legalizer/globalvalue.rs
rename to cranelift/codegen/src/legalizer/globalvalue.rs
diff --git a/lib/codegen/src/legalizer/heap.rs b/cranelift/codegen/src/legalizer/heap.rs
similarity index 100%
rename from lib/codegen/src/legalizer/heap.rs
rename to cranelift/codegen/src/legalizer/heap.rs
diff --git a/lib/codegen/src/legalizer/libcall.rs b/cranelift/codegen/src/legalizer/libcall.rs
similarity index 100%
rename from lib/codegen/src/legalizer/libcall.rs
rename to cranelift/codegen/src/legalizer/libcall.rs
diff --git a/lib/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
similarity index 99%
rename from lib/codegen/src/legalizer/mod.rs
rename to cranelift/codegen/src/legalizer/mod.rs
index db0fb107fa..2e1db93d42 100644
--- a/lib/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -118,7 +118,7 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
 }
 
 // Include legalization patterns that were generated by `gen_legalizer.py` from the `XForms` in
-// `lib/codegen/meta-python/base/legalize.py`.
+// `cranelift-codegen/meta-python/base/legalize.py`.
 //
 // Concretely, this defines private functions `narrow()`, and `expand()`.
 include!(concat!(env!("OUT_DIR"), "/legalizer.rs"));
diff --git a/lib/codegen/src/legalizer/split.rs b/cranelift/codegen/src/legalizer/split.rs
similarity index 100%
rename from lib/codegen/src/legalizer/split.rs
rename to cranelift/codegen/src/legalizer/split.rs
diff --git a/lib/codegen/src/legalizer/table.rs b/cranelift/codegen/src/legalizer/table.rs
similarity index 100%
rename from lib/codegen/src/legalizer/table.rs
rename to cranelift/codegen/src/legalizer/table.rs
diff --git a/lib/codegen/src/lib.rs b/cranelift/codegen/src/lib.rs
similarity index 100%
rename from lib/codegen/src/lib.rs
rename to cranelift/codegen/src/lib.rs
diff --git a/lib/codegen/src/licm.rs b/cranelift/codegen/src/licm.rs
similarity index 100%
rename from lib/codegen/src/licm.rs
rename to cranelift/codegen/src/licm.rs
diff --git a/lib/codegen/src/loop_analysis.rs b/cranelift/codegen/src/loop_analysis.rs
similarity index 100%
rename from lib/codegen/src/loop_analysis.rs
rename to cranelift/codegen/src/loop_analysis.rs
diff --git a/lib/codegen/src/nan_canonicalization.rs b/cranelift/codegen/src/nan_canonicalization.rs
similarity index 100%
rename from lib/codegen/src/nan_canonicalization.rs
rename to cranelift/codegen/src/nan_canonicalization.rs
diff --git a/lib/codegen/src/partition_slice.rs b/cranelift/codegen/src/partition_slice.rs
similarity index 100%
rename from lib/codegen/src/partition_slice.rs
rename to cranelift/codegen/src/partition_slice.rs
diff --git a/lib/codegen/src/postopt.rs b/cranelift/codegen/src/postopt.rs
similarity index 100%
rename from lib/codegen/src/postopt.rs
rename to cranelift/codegen/src/postopt.rs
diff --git a/lib/codegen/src/predicates.rs b/cranelift/codegen/src/predicates.rs
similarity index 98%
rename from lib/codegen/src/predicates.rs
rename to cranelift/codegen/src/predicates.rs
index d35b54c6c3..da01ad6b2b 100644
--- a/lib/codegen/src/predicates.rs
+++ b/cranelift/codegen/src/predicates.rs
@@ -1,7 +1,7 @@
 //! Predicate functions for testing instruction fields.
 //!
 //! This module defines functions that are used by the instruction predicates defined by
-//! `lib/codegen/meta-python/cdsl/predicates.py` classes.
+//! `cranelift-codegen/meta-python/cdsl/predicates.py` classes.
 //!
 //! The predicates the operate on integer fields use `Into` as a shared trait bound. This
 //! bound is implemented by all the native integer types as well as `Imm64`.
diff --git a/lib/codegen/src/print_errors.rs b/cranelift/codegen/src/print_errors.rs
similarity index 100%
rename from lib/codegen/src/print_errors.rs
rename to cranelift/codegen/src/print_errors.rs
diff --git a/lib/codegen/src/ref_slice.rs b/cranelift/codegen/src/ref_slice.rs
similarity index 100%
rename from lib/codegen/src/ref_slice.rs
rename to cranelift/codegen/src/ref_slice.rs
diff --git a/lib/codegen/src/regalloc/affinity.rs b/cranelift/codegen/src/regalloc/affinity.rs
similarity index 100%
rename from lib/codegen/src/regalloc/affinity.rs
rename to cranelift/codegen/src/regalloc/affinity.rs
diff --git a/lib/codegen/src/regalloc/coalescing.rs b/cranelift/codegen/src/regalloc/coalescing.rs
similarity index 100%
rename from lib/codegen/src/regalloc/coalescing.rs
rename to cranelift/codegen/src/regalloc/coalescing.rs
diff --git a/lib/codegen/src/regalloc/coloring.rs b/cranelift/codegen/src/regalloc/coloring.rs
similarity index 100%
rename from lib/codegen/src/regalloc/coloring.rs
rename to cranelift/codegen/src/regalloc/coloring.rs
diff --git a/lib/codegen/src/regalloc/context.rs b/cranelift/codegen/src/regalloc/context.rs
similarity index 100%
rename from lib/codegen/src/regalloc/context.rs
rename to cranelift/codegen/src/regalloc/context.rs
diff --git a/lib/codegen/src/regalloc/diversion.rs b/cranelift/codegen/src/regalloc/diversion.rs
similarity index 100%
rename from lib/codegen/src/regalloc/diversion.rs
rename to cranelift/codegen/src/regalloc/diversion.rs
diff --git a/lib/codegen/src/regalloc/live_value_tracker.rs b/cranelift/codegen/src/regalloc/live_value_tracker.rs
similarity index 100%
rename from lib/codegen/src/regalloc/live_value_tracker.rs
rename to cranelift/codegen/src/regalloc/live_value_tracker.rs
diff --git a/lib/codegen/src/regalloc/liveness.rs b/cranelift/codegen/src/regalloc/liveness.rs
similarity index 100%
rename from lib/codegen/src/regalloc/liveness.rs
rename to cranelift/codegen/src/regalloc/liveness.rs
diff --git a/lib/codegen/src/regalloc/liverange.rs b/cranelift/codegen/src/regalloc/liverange.rs
similarity index 100%
rename from lib/codegen/src/regalloc/liverange.rs
rename to cranelift/codegen/src/regalloc/liverange.rs
diff --git a/lib/codegen/src/regalloc/mod.rs b/cranelift/codegen/src/regalloc/mod.rs
similarity index 100%
rename from lib/codegen/src/regalloc/mod.rs
rename to cranelift/codegen/src/regalloc/mod.rs
diff --git a/lib/codegen/src/regalloc/pressure.rs b/cranelift/codegen/src/regalloc/pressure.rs
similarity index 100%
rename from lib/codegen/src/regalloc/pressure.rs
rename to cranelift/codegen/src/regalloc/pressure.rs
diff --git a/lib/codegen/src/regalloc/register_set.rs b/cranelift/codegen/src/regalloc/register_set.rs
similarity index 100%
rename from lib/codegen/src/regalloc/register_set.rs
rename to cranelift/codegen/src/regalloc/register_set.rs
diff --git a/lib/codegen/src/regalloc/reload.rs b/cranelift/codegen/src/regalloc/reload.rs
similarity index 100%
rename from lib/codegen/src/regalloc/reload.rs
rename to cranelift/codegen/src/regalloc/reload.rs
diff --git a/lib/codegen/src/regalloc/solver.rs b/cranelift/codegen/src/regalloc/solver.rs
similarity index 100%
rename from lib/codegen/src/regalloc/solver.rs
rename to cranelift/codegen/src/regalloc/solver.rs
diff --git a/lib/codegen/src/regalloc/spilling.rs b/cranelift/codegen/src/regalloc/spilling.rs
similarity index 100%
rename from lib/codegen/src/regalloc/spilling.rs
rename to cranelift/codegen/src/regalloc/spilling.rs
diff --git a/lib/codegen/src/regalloc/virtregs.rs b/cranelift/codegen/src/regalloc/virtregs.rs
similarity index 100%
rename from lib/codegen/src/regalloc/virtregs.rs
rename to cranelift/codegen/src/regalloc/virtregs.rs
diff --git a/lib/codegen/src/result.rs b/cranelift/codegen/src/result.rs
similarity index 100%
rename from lib/codegen/src/result.rs
rename to cranelift/codegen/src/result.rs
diff --git a/lib/codegen/src/scoped_hash_map.rs b/cranelift/codegen/src/scoped_hash_map.rs
similarity index 100%
rename from lib/codegen/src/scoped_hash_map.rs
rename to cranelift/codegen/src/scoped_hash_map.rs
diff --git a/lib/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
similarity index 99%
rename from lib/codegen/src/settings.rs
rename to cranelift/codegen/src/settings.rs
index d0f5d5b13e..404bf58cd5 100644
--- a/lib/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -333,7 +333,7 @@ pub mod detail {
 
 // Include code generated by `meta-python/gen_settings.py`. This file contains a public `Flags`
 // struct with an impl for all of the settings defined in
-// `lib/codegen/meta-python/base/settings.py`.
+// `cranelift-codegen/meta-python/base/settings.py`.
 include!(concat!(env!("OUT_DIR"), "/settings.rs"));
 
 /// Wrapper containing flags and optionally a `TargetIsa` trait object.
diff --git a/lib/codegen/src/simple_gvn.rs b/cranelift/codegen/src/simple_gvn.rs
similarity index 100%
rename from lib/codegen/src/simple_gvn.rs
rename to cranelift/codegen/src/simple_gvn.rs
diff --git a/lib/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
similarity index 100%
rename from lib/codegen/src/simple_preopt.rs
rename to cranelift/codegen/src/simple_preopt.rs
diff --git a/lib/codegen/src/stack_layout.rs b/cranelift/codegen/src/stack_layout.rs
similarity index 100%
rename from lib/codegen/src/stack_layout.rs
rename to cranelift/codegen/src/stack_layout.rs
diff --git a/lib/codegen/src/timing.rs b/cranelift/codegen/src/timing.rs
similarity index 100%
rename from lib/codegen/src/timing.rs
rename to cranelift/codegen/src/timing.rs
diff --git a/lib/codegen/src/topo_order.rs b/cranelift/codegen/src/topo_order.rs
similarity index 100%
rename from lib/codegen/src/topo_order.rs
rename to cranelift/codegen/src/topo_order.rs
diff --git a/lib/codegen/src/unreachable_code.rs b/cranelift/codegen/src/unreachable_code.rs
similarity index 100%
rename from lib/codegen/src/unreachable_code.rs
rename to cranelift/codegen/src/unreachable_code.rs
diff --git a/lib/codegen/src/verifier/cssa.rs b/cranelift/codegen/src/verifier/cssa.rs
similarity index 100%
rename from lib/codegen/src/verifier/cssa.rs
rename to cranelift/codegen/src/verifier/cssa.rs
diff --git a/lib/codegen/src/verifier/flags.rs b/cranelift/codegen/src/verifier/flags.rs
similarity index 100%
rename from lib/codegen/src/verifier/flags.rs
rename to cranelift/codegen/src/verifier/flags.rs
diff --git a/lib/codegen/src/verifier/liveness.rs b/cranelift/codegen/src/verifier/liveness.rs
similarity index 100%
rename from lib/codegen/src/verifier/liveness.rs
rename to cranelift/codegen/src/verifier/liveness.rs
diff --git a/lib/codegen/src/verifier/locations.rs b/cranelift/codegen/src/verifier/locations.rs
similarity index 100%
rename from lib/codegen/src/verifier/locations.rs
rename to cranelift/codegen/src/verifier/locations.rs
diff --git a/lib/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
similarity index 100%
rename from lib/codegen/src/verifier/mod.rs
rename to cranelift/codegen/src/verifier/mod.rs
diff --git a/lib/codegen/src/write.rs b/cranelift/codegen/src/write.rs
similarity index 100%
rename from lib/codegen/src/write.rs
rename to cranelift/codegen/src/write.rs
diff --git a/cranelift/docs/Makefile b/cranelift/docs/Makefile
index 082189bfa3..dbb8bd0e2c 100644
--- a/cranelift/docs/Makefile
+++ b/cranelift/docs/Makefile
@@ -14,7 +14,7 @@ help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
 autohtml: html
-	$(SPHINXABUILD) -z ../lib/codegen/meta-python --ignore '.*' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	$(SPHINXABUILD) -z ../cranelift-codegen/meta-python --ignore '.*' -b html -E $(ALLSPHINXOPTS) $(BUILDDIR)/html
 
 .PHONY: help Makefile
 
diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py
index b2987be684..09fd164537 100644
--- a/cranelift/docs/conf.py
+++ b/cranelift/docs/conf.py
@@ -23,7 +23,7 @@ sys.path.insert(0, os.path.abspath('.'))
 
 # Also add the meta-python directory to sys.path so autodoc can find the Cranelift meta
 # language definitions.
-sys.path.insert(0, os.path.abspath('../lib/codegen/meta-python'))
+sys.path.insert(0, os.path.abspath('../cranelift-codegen/meta-python'))
 
 # -- General configuration ------------------------------------------------
 
diff --git a/cranelift/docs/meta.rst b/cranelift/docs/meta.rst
index f805e08875..67457835b5 100644
--- a/cranelift/docs/meta.rst
+++ b/cranelift/docs/meta.rst
@@ -11,7 +11,7 @@ domain specific language embedded in Python. This document describes the Python
 modules that form the embedded DSL.
 
 The meta language descriptions are Python modules under the
-:file:`lib/codegen/meta-python` directory. The descriptions are processed in two
+:file:`cranelift-codegen/meta-python` directory. The descriptions are processed in two
 steps:
 
 1. The Python modules are imported. This has the effect of building static data
@@ -23,8 +23,8 @@ steps:
    constant tables.
 
 The main driver for this source code generation process is the
-:file:`lib/codegen/meta-python/build.py` script which is invoked as part of the build
-process if anything in the :file:`lib/codegen/meta-python` directory has changed
+:file:`cranelift-codegen/meta-python/build.py` script which is invoked as part of the build
+process if anything in the :file:`cranelift-codegen/meta-python` directory has changed
 since the last build.
 
 
@@ -38,7 +38,7 @@ of code generation. Each setting is defined in the meta language so a compact
 and consistent Rust representation can be generated. Shared settings are defined
 in the :mod:`base.settings` module. Some settings are specific to a target ISA,
 and defined in a :file:`settings.py` module under the appropriate
-:file:`lib/codegen/meta-python/isa/*` directory.
+:file:`cranelift-codegen/meta-python/isa/*` directory.
 
 Settings can take boolean on/off values, small numbers, or explicitly enumerated
 symbolic values. Each type is represented by a sub-class of :class:`Setting`:
@@ -433,7 +433,7 @@ architectures. Each ISA is represented by a :py:class:`cdsl.isa.TargetISA` insta
 .. autoclass:: TargetISA
 
 The definitions for each supported target live in a package under
-:file:`lib/codegen/meta-python/isa`.
+:file:`cranelift-codegen/meta-python/isa`.
 
 .. automodule:: isa
     :members:
diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst
index 747ba53d54..af2819cfc3 100644
--- a/cranelift/docs/testing.rst
+++ b/cranelift/docs/testing.rst
@@ -73,10 +73,10 @@ test.
 
 These tests are usually found in the :file:`tests` top-level directory where
 they have access to all the crates in the Cranelift repository. The
-:file:`lib/codegen` and :file:`lib/reader` crates have no external
+:file:`cranelift-codegen` and :file:`cranelift-reader` crates have no external
 dependencies, which can make testing tedious. Integration tests that don't need
-to depend on other crates can be placed in :file:`lib/codegen/tests` and
-:file:`lib/reader/tests`.
+to depend on other crates can be placed in :file:`cranelift-codegen/tests` and
+:file:`cranelift-reader/tests`.
 
 File tests
 ==========
@@ -109,7 +109,7 @@ header:
     isa_spec      : "isa" isa_name { `option` } "\n"
 
 The options given on the ``isa`` line modify the ISA-specific settings defined in
-:file:`lib/codegen/meta-python/isa/*/settings.py`.
+:file:`cranelift-codegen/meta-python/isa/*/settings.py`.
 
 All types of tests allow shared Cranelift settings to be modified:
 
@@ -119,7 +119,7 @@ All types of tests allow shared Cranelift settings to be modified:
     option        : flag | setting "=" value
 
 The shared settings available for all target ISAs are defined in
-:file:`lib/codegen/meta-python/base/settings.py`.
+:file:`cranelift-codegen/meta-python/base/settings.py`.
 
 The ``set`` lines apply settings cumulatively::
 
diff --git a/lib/entity/Cargo.toml b/cranelift/entity/Cargo.toml
similarity index 100%
rename from lib/entity/Cargo.toml
rename to cranelift/entity/Cargo.toml
diff --git a/lib/entity/LICENSE b/cranelift/entity/LICENSE
similarity index 100%
rename from lib/entity/LICENSE
rename to cranelift/entity/LICENSE
diff --git a/lib/entity/README.md b/cranelift/entity/README.md
similarity index 100%
rename from lib/entity/README.md
rename to cranelift/entity/README.md
diff --git a/lib/entity/src/boxed_slice.rs b/cranelift/entity/src/boxed_slice.rs
similarity index 100%
rename from lib/entity/src/boxed_slice.rs
rename to cranelift/entity/src/boxed_slice.rs
diff --git a/lib/entity/src/iter.rs b/cranelift/entity/src/iter.rs
similarity index 100%
rename from lib/entity/src/iter.rs
rename to cranelift/entity/src/iter.rs
diff --git a/lib/entity/src/keys.rs b/cranelift/entity/src/keys.rs
similarity index 100%
rename from lib/entity/src/keys.rs
rename to cranelift/entity/src/keys.rs
diff --git a/lib/entity/src/lib.rs b/cranelift/entity/src/lib.rs
similarity index 100%
rename from lib/entity/src/lib.rs
rename to cranelift/entity/src/lib.rs
diff --git a/lib/entity/src/list.rs b/cranelift/entity/src/list.rs
similarity index 100%
rename from lib/entity/src/list.rs
rename to cranelift/entity/src/list.rs
diff --git a/lib/entity/src/map.rs b/cranelift/entity/src/map.rs
similarity index 100%
rename from lib/entity/src/map.rs
rename to cranelift/entity/src/map.rs
diff --git a/lib/entity/src/packed_option.rs b/cranelift/entity/src/packed_option.rs
similarity index 100%
rename from lib/entity/src/packed_option.rs
rename to cranelift/entity/src/packed_option.rs
diff --git a/lib/entity/src/primary.rs b/cranelift/entity/src/primary.rs
similarity index 100%
rename from lib/entity/src/primary.rs
rename to cranelift/entity/src/primary.rs
diff --git a/lib/entity/src/set.rs b/cranelift/entity/src/set.rs
similarity index 100%
rename from lib/entity/src/set.rs
rename to cranelift/entity/src/set.rs
diff --git a/lib/entity/src/sparse.rs b/cranelift/entity/src/sparse.rs
similarity index 100%
rename from lib/entity/src/sparse.rs
rename to cranelift/entity/src/sparse.rs
diff --git a/lib/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
similarity index 79%
rename from lib/faerie/Cargo.toml
rename to cranelift/faerie/Cargo.toml
index ce11f80570..d15ebf4aa9 100644
--- a/lib/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.28.0" }
-cranelift-module = { path = "../module", version = "0.28.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.28.0" }
 faerie = "0.7.0"
 goblin = "0.0.19"
 failure = "0.1.2"
diff --git a/lib/faerie/LICENSE b/cranelift/faerie/LICENSE
similarity index 100%
rename from lib/faerie/LICENSE
rename to cranelift/faerie/LICENSE
diff --git a/lib/faerie/README.md b/cranelift/faerie/README.md
similarity index 100%
rename from lib/faerie/README.md
rename to cranelift/faerie/README.md
diff --git a/lib/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
similarity index 100%
rename from lib/faerie/src/backend.rs
rename to cranelift/faerie/src/backend.rs
diff --git a/lib/faerie/src/container.rs b/cranelift/faerie/src/container.rs
similarity index 100%
rename from lib/faerie/src/container.rs
rename to cranelift/faerie/src/container.rs
diff --git a/lib/faerie/src/lib.rs b/cranelift/faerie/src/lib.rs
similarity index 100%
rename from lib/faerie/src/lib.rs
rename to cranelift/faerie/src/lib.rs
diff --git a/lib/faerie/src/traps.rs b/cranelift/faerie/src/traps.rs
similarity index 100%
rename from lib/faerie/src/traps.rs
rename to cranelift/faerie/src/traps.rs
diff --git a/lib/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
similarity index 66%
rename from lib/filetests/Cargo.toml
rename to cranelift/filetests/Cargo.toml
index 874e8ce098..c89da76f84 100644
--- a/lib/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.28.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../reader", version = "0.28.0" }
-cranelift-preopt = { path = "../preopt", version = "0.28.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../cranelift-reader", version = "0.28.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.28.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/lib/filetests/LICENSE b/cranelift/filetests/LICENSE
similarity index 100%
rename from lib/filetests/LICENSE
rename to cranelift/filetests/LICENSE
diff --git a/cranelift/filetests/cfg/loop.clif b/cranelift/filetests/filetests/cfg/loop.clif
similarity index 100%
rename from cranelift/filetests/cfg/loop.clif
rename to cranelift/filetests/filetests/cfg/loop.clif
diff --git a/cranelift/filetests/cfg/traps_early.clif b/cranelift/filetests/filetests/cfg/traps_early.clif
similarity index 100%
rename from cranelift/filetests/cfg/traps_early.clif
rename to cranelift/filetests/filetests/cfg/traps_early.clif
diff --git a/cranelift/filetests/cfg/unused_node.clif b/cranelift/filetests/filetests/cfg/unused_node.clif
similarity index 100%
rename from cranelift/filetests/cfg/unused_node.clif
rename to cranelift/filetests/filetests/cfg/unused_node.clif
diff --git a/cranelift/filetests/dce/basic.clif b/cranelift/filetests/filetests/dce/basic.clif
similarity index 100%
rename from cranelift/filetests/dce/basic.clif
rename to cranelift/filetests/filetests/dce/basic.clif
diff --git a/cranelift/filetests/domtree/basic.clif b/cranelift/filetests/filetests/domtree/basic.clif
similarity index 100%
rename from cranelift/filetests/domtree/basic.clif
rename to cranelift/filetests/filetests/domtree/basic.clif
diff --git a/cranelift/filetests/domtree/loops.clif b/cranelift/filetests/filetests/domtree/loops.clif
similarity index 100%
rename from cranelift/filetests/domtree/loops.clif
rename to cranelift/filetests/filetests/domtree/loops.clif
diff --git a/cranelift/filetests/domtree/loops2.clif b/cranelift/filetests/filetests/domtree/loops2.clif
similarity index 100%
rename from cranelift/filetests/domtree/loops2.clif
rename to cranelift/filetests/filetests/domtree/loops2.clif
diff --git a/cranelift/filetests/domtree/tall-tree.clif b/cranelift/filetests/filetests/domtree/tall-tree.clif
similarity index 100%
rename from cranelift/filetests/domtree/tall-tree.clif
rename to cranelift/filetests/filetests/domtree/tall-tree.clif
diff --git a/cranelift/filetests/domtree/wide-tree.clif b/cranelift/filetests/filetests/domtree/wide-tree.clif
similarity index 100%
rename from cranelift/filetests/domtree/wide-tree.clif
rename to cranelift/filetests/filetests/domtree/wide-tree.clif
diff --git a/cranelift/filetests/isa/riscv/abi-e.clif b/cranelift/filetests/filetests/isa/riscv/abi-e.clif
similarity index 100%
rename from cranelift/filetests/isa/riscv/abi-e.clif
rename to cranelift/filetests/filetests/isa/riscv/abi-e.clif
diff --git a/cranelift/filetests/isa/riscv/abi.clif b/cranelift/filetests/filetests/isa/riscv/abi.clif
similarity index 100%
rename from cranelift/filetests/isa/riscv/abi.clif
rename to cranelift/filetests/filetests/isa/riscv/abi.clif
diff --git a/cranelift/filetests/isa/riscv/binary32.clif b/cranelift/filetests/filetests/isa/riscv/binary32.clif
similarity index 100%
rename from cranelift/filetests/isa/riscv/binary32.clif
rename to cranelift/filetests/filetests/isa/riscv/binary32.clif
diff --git a/cranelift/filetests/isa/riscv/encoding.clif b/cranelift/filetests/filetests/isa/riscv/encoding.clif
similarity index 100%
rename from cranelift/filetests/isa/riscv/encoding.clif
rename to cranelift/filetests/filetests/isa/riscv/encoding.clif
diff --git a/cranelift/filetests/isa/riscv/expand-i32.clif b/cranelift/filetests/filetests/isa/riscv/expand-i32.clif
similarity index 100%
rename from cranelift/filetests/isa/riscv/expand-i32.clif
rename to cranelift/filetests/filetests/isa/riscv/expand-i32.clif
diff --git a/cranelift/filetests/isa/riscv/legalize-abi.clif b/cranelift/filetests/filetests/isa/riscv/legalize-abi.clif
similarity index 100%
rename from cranelift/filetests/isa/riscv/legalize-abi.clif
rename to cranelift/filetests/filetests/isa/riscv/legalize-abi.clif
diff --git a/cranelift/filetests/isa/riscv/legalize-i64.clif b/cranelift/filetests/filetests/isa/riscv/legalize-i64.clif
similarity index 100%
rename from cranelift/filetests/isa/riscv/legalize-i64.clif
rename to cranelift/filetests/filetests/isa/riscv/legalize-i64.clif
diff --git a/cranelift/filetests/isa/riscv/parse-encoding.clif b/cranelift/filetests/filetests/isa/riscv/parse-encoding.clif
similarity index 100%
rename from cranelift/filetests/isa/riscv/parse-encoding.clif
rename to cranelift/filetests/filetests/isa/riscv/parse-encoding.clif
diff --git a/cranelift/filetests/isa/riscv/regmove.clif b/cranelift/filetests/filetests/isa/riscv/regmove.clif
similarity index 100%
rename from cranelift/filetests/isa/riscv/regmove.clif
rename to cranelift/filetests/filetests/isa/riscv/regmove.clif
diff --git a/cranelift/filetests/isa/riscv/split-args.clif b/cranelift/filetests/filetests/isa/riscv/split-args.clif
similarity index 100%
rename from cranelift/filetests/isa/riscv/split-args.clif
rename to cranelift/filetests/filetests/isa/riscv/split-args.clif
diff --git a/cranelift/filetests/isa/riscv/verify-encoding.clif b/cranelift/filetests/filetests/isa/riscv/verify-encoding.clif
similarity index 100%
rename from cranelift/filetests/isa/riscv/verify-encoding.clif
rename to cranelift/filetests/filetests/isa/riscv/verify-encoding.clif
diff --git a/cranelift/filetests/isa/x86/abcd.clif b/cranelift/filetests/filetests/isa/x86/abcd.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/abcd.clif
rename to cranelift/filetests/filetests/isa/x86/abcd.clif
diff --git a/cranelift/filetests/isa/x86/abi-bool.clif b/cranelift/filetests/filetests/isa/x86/abi-bool.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/abi-bool.clif
rename to cranelift/filetests/filetests/isa/x86/abi-bool.clif
diff --git a/cranelift/filetests/isa/x86/abi32.clif b/cranelift/filetests/filetests/isa/x86/abi32.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/abi32.clif
rename to cranelift/filetests/filetests/isa/x86/abi32.clif
diff --git a/cranelift/filetests/isa/x86/abi64.clif b/cranelift/filetests/filetests/isa/x86/abi64.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/abi64.clif
rename to cranelift/filetests/filetests/isa/x86/abi64.clif
diff --git a/cranelift/filetests/isa/x86/allones_funcaddrs32.clif b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/allones_funcaddrs32.clif
rename to cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif
diff --git a/cranelift/filetests/isa/x86/allones_funcaddrs64.clif b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/allones_funcaddrs64.clif
rename to cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif
diff --git a/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount.clif b/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/baseline_clz_ctz_popcount.clif
rename to cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount.clif
diff --git a/cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif b/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif
rename to cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif
diff --git a/cranelift/filetests/isa/x86/binary32-float.clif b/cranelift/filetests/filetests/isa/x86/binary32-float.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/binary32-float.clif
rename to cranelift/filetests/filetests/isa/x86/binary32-float.clif
diff --git a/cranelift/filetests/isa/x86/binary32.clif b/cranelift/filetests/filetests/isa/x86/binary32.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/binary32.clif
rename to cranelift/filetests/filetests/isa/x86/binary32.clif
diff --git a/cranelift/filetests/isa/x86/binary64-float.clif b/cranelift/filetests/filetests/isa/x86/binary64-float.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/binary64-float.clif
rename to cranelift/filetests/filetests/isa/x86/binary64-float.clif
diff --git a/cranelift/filetests/isa/x86/binary64-pic.clif b/cranelift/filetests/filetests/isa/x86/binary64-pic.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/binary64-pic.clif
rename to cranelift/filetests/filetests/isa/x86/binary64-pic.clif
diff --git a/cranelift/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/binary64.clif
rename to cranelift/filetests/filetests/isa/x86/binary64.clif
diff --git a/cranelift/filetests/isa/x86/ireduce-i16-to-i8.clif b/cranelift/filetests/filetests/isa/x86/ireduce-i16-to-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/ireduce-i16-to-i8.clif
rename to cranelift/filetests/filetests/isa/x86/ireduce-i16-to-i8.clif
diff --git a/cranelift/filetests/isa/x86/isub_imm-i8.clif b/cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/isub_imm-i8.clif
rename to cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif
diff --git a/cranelift/filetests/isa/x86/legalize-bint-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-bint-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-bint-i8.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-bint-i8.clif
diff --git a/cranelift/filetests/isa/x86/legalize-bnot.clif b/cranelift/filetests/filetests/isa/x86/legalize-bnot.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-bnot.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-bnot.clif
diff --git a/cranelift/filetests/isa/x86/legalize-br-icmp.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-br-icmp.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif
diff --git a/cranelift/filetests/isa/x86/legalize-br-table.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-br-table.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
diff --git a/cranelift/filetests/isa/x86/legalize-byte-ops-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-byte-ops-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-byte-ops-i8.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-byte-ops-i8.clif
diff --git a/cranelift/filetests/isa/x86/legalize-call.clif b/cranelift/filetests/filetests/isa/x86/legalize-call.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-call.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-call.clif
diff --git a/cranelift/filetests/isa/x86/legalize-clz-ctz-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-clz-ctz-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-clz-ctz-i8.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-clz-ctz-i8.clif
diff --git a/cranelift/filetests/isa/x86/legalize-custom.clif b/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-custom.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-custom.clif
diff --git a/cranelift/filetests/isa/x86/legalize-div-traps.clif b/cranelift/filetests/filetests/isa/x86/legalize-div-traps.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-div-traps.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-div-traps.clif
diff --git a/cranelift/filetests/isa/x86/legalize-div.clif b/cranelift/filetests/filetests/isa/x86/legalize-div.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-div.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-div.clif
diff --git a/cranelift/filetests/isa/x86/legalize-heaps.clif b/cranelift/filetests/filetests/isa/x86/legalize-heaps.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-heaps.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-heaps.clif
diff --git a/cranelift/filetests/isa/x86/legalize-icmp-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-icmp-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-icmp-i8.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-icmp-i8.clif
diff --git a/cranelift/filetests/isa/x86/legalize-iconst-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-iconst-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-iconst-i8.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-iconst-i8.clif
diff --git a/cranelift/filetests/isa/x86/legalize-imul-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-imul-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-imul-i8.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-imul-i8.clif
diff --git a/cranelift/filetests/isa/x86/legalize-imul-imm-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-imul-imm-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-imul-imm-i8.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-imul-imm-i8.clif
diff --git a/cranelift/filetests/isa/x86/legalize-libcall.clif b/cranelift/filetests/filetests/isa/x86/legalize-libcall.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-libcall.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-libcall.clif
diff --git a/cranelift/filetests/isa/x86/legalize-load-store-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-load-store-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-load-store-i8.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-load-store-i8.clif
diff --git a/cranelift/filetests/isa/x86/legalize-memory.clif b/cranelift/filetests/filetests/isa/x86/legalize-memory.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-memory.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-memory.clif
diff --git a/cranelift/filetests/isa/x86/legalize-mulhi.clif b/cranelift/filetests/filetests/isa/x86/legalize-mulhi.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-mulhi.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-mulhi.clif
diff --git a/cranelift/filetests/isa/x86/legalize-popcnt-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-popcnt-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-popcnt-i8.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-popcnt-i8.clif
diff --git a/cranelift/filetests/isa/x86/legalize-regmove-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-regmove-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-regmove-i8.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-regmove-i8.clif
diff --git a/cranelift/filetests/isa/x86/legalize-shlr-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-shlr-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-shlr-i8.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-shlr-i8.clif
diff --git a/cranelift/filetests/isa/x86/legalize-tables.clif b/cranelift/filetests/filetests/isa/x86/legalize-tables.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-tables.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-tables.clif
diff --git a/cranelift/filetests/isa/x86/legalize-urem-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-urem-i8.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/legalize-urem-i8.clif
rename to cranelift/filetests/filetests/isa/x86/legalize-urem-i8.clif
diff --git a/cranelift/filetests/isa/x86/nop.clif b/cranelift/filetests/filetests/isa/x86/nop.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/nop.clif
rename to cranelift/filetests/filetests/isa/x86/nop.clif
diff --git a/cranelift/filetests/isa/x86/optimized-zero-constants-32bit.clif b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/optimized-zero-constants-32bit.clif
rename to cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif
diff --git a/cranelift/filetests/isa/x86/optimized-zero-constants.clif b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/optimized-zero-constants.clif
rename to cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif
diff --git a/cranelift/filetests/isa/x86/probestack-adjusts-sp.clif b/cranelift/filetests/filetests/isa/x86/probestack-adjusts-sp.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/probestack-adjusts-sp.clif
rename to cranelift/filetests/filetests/isa/x86/probestack-adjusts-sp.clif
diff --git a/cranelift/filetests/isa/x86/probestack-disabled.clif b/cranelift/filetests/filetests/isa/x86/probestack-disabled.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/probestack-disabled.clif
rename to cranelift/filetests/filetests/isa/x86/probestack-disabled.clif
diff --git a/cranelift/filetests/isa/x86/probestack-noncolocated.clif b/cranelift/filetests/filetests/isa/x86/probestack-noncolocated.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/probestack-noncolocated.clif
rename to cranelift/filetests/filetests/isa/x86/probestack-noncolocated.clif
diff --git a/cranelift/filetests/isa/x86/probestack-size.clif b/cranelift/filetests/filetests/isa/x86/probestack-size.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/probestack-size.clif
rename to cranelift/filetests/filetests/isa/x86/probestack-size.clif
diff --git a/cranelift/filetests/isa/x86/probestack.clif b/cranelift/filetests/filetests/isa/x86/probestack.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/probestack.clif
rename to cranelift/filetests/filetests/isa/x86/probestack.clif
diff --git a/cranelift/filetests/isa/x86/prologue-epilogue.clif b/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/prologue-epilogue.clif
rename to cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
diff --git a/cranelift/filetests/isa/x86/shrink-multiple-uses.clif b/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/shrink-multiple-uses.clif
rename to cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
diff --git a/cranelift/filetests/isa/x86/shrink.clif b/cranelift/filetests/filetests/isa/x86/shrink.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/shrink.clif
rename to cranelift/filetests/filetests/isa/x86/shrink.clif
diff --git a/cranelift/filetests/isa/x86/stack-addr64.clif b/cranelift/filetests/filetests/isa/x86/stack-addr64.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/stack-addr64.clif
rename to cranelift/filetests/filetests/isa/x86/stack-addr64.clif
diff --git a/cranelift/filetests/isa/x86/stack-load-store64.clif b/cranelift/filetests/filetests/isa/x86/stack-load-store64.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/stack-load-store64.clif
rename to cranelift/filetests/filetests/isa/x86/stack-load-store64.clif
diff --git a/cranelift/filetests/isa/x86/uextend-i8-to-i16.clif b/cranelift/filetests/filetests/isa/x86/uextend-i8-to-i16.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/uextend-i8-to-i16.clif
rename to cranelift/filetests/filetests/isa/x86/uextend-i8-to-i16.clif
diff --git a/cranelift/filetests/isa/x86/windows_fastcall_x64.clif b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
similarity index 100%
rename from cranelift/filetests/isa/x86/windows_fastcall_x64.clif
rename to cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
diff --git a/cranelift/filetests/legalizer/bitrev.clif b/cranelift/filetests/filetests/legalizer/bitrev.clif
similarity index 100%
rename from cranelift/filetests/legalizer/bitrev.clif
rename to cranelift/filetests/filetests/legalizer/bitrev.clif
diff --git a/cranelift/filetests/legalizer/br_table_cond.clif b/cranelift/filetests/filetests/legalizer/br_table_cond.clif
similarity index 100%
rename from cranelift/filetests/legalizer/br_table_cond.clif
rename to cranelift/filetests/filetests/legalizer/br_table_cond.clif
diff --git a/cranelift/filetests/legalizer/bxor_imm.clif b/cranelift/filetests/filetests/legalizer/bxor_imm.clif
similarity index 100%
rename from cranelift/filetests/legalizer/bxor_imm.clif
rename to cranelift/filetests/filetests/legalizer/bxor_imm.clif
diff --git a/cranelift/filetests/licm/basic.clif b/cranelift/filetests/filetests/licm/basic.clif
similarity index 100%
rename from cranelift/filetests/licm/basic.clif
rename to cranelift/filetests/filetests/licm/basic.clif
diff --git a/cranelift/filetests/licm/complex.clif b/cranelift/filetests/filetests/licm/complex.clif
similarity index 100%
rename from cranelift/filetests/licm/complex.clif
rename to cranelift/filetests/filetests/licm/complex.clif
diff --git a/cranelift/filetests/licm/critical-edge.clif b/cranelift/filetests/filetests/licm/critical-edge.clif
similarity index 100%
rename from cranelift/filetests/licm/critical-edge.clif
rename to cranelift/filetests/filetests/licm/critical-edge.clif
diff --git a/cranelift/filetests/licm/encoding.clif b/cranelift/filetests/filetests/licm/encoding.clif
similarity index 100%
rename from cranelift/filetests/licm/encoding.clif
rename to cranelift/filetests/filetests/licm/encoding.clif
diff --git a/cranelift/filetests/licm/multiple-blocks.clif b/cranelift/filetests/filetests/licm/multiple-blocks.clif
similarity index 100%
rename from cranelift/filetests/licm/multiple-blocks.clif
rename to cranelift/filetests/filetests/licm/multiple-blocks.clif
diff --git a/cranelift/filetests/licm/nested_loops.clif b/cranelift/filetests/filetests/licm/nested_loops.clif
similarity index 100%
rename from cranelift/filetests/licm/nested_loops.clif
rename to cranelift/filetests/filetests/licm/nested_loops.clif
diff --git a/cranelift/filetests/licm/reject.clif b/cranelift/filetests/filetests/licm/reject.clif
similarity index 100%
rename from cranelift/filetests/licm/reject.clif
rename to cranelift/filetests/filetests/licm/reject.clif
diff --git a/cranelift/filetests/parser/alias.clif b/cranelift/filetests/filetests/parser/alias.clif
similarity index 100%
rename from cranelift/filetests/parser/alias.clif
rename to cranelift/filetests/filetests/parser/alias.clif
diff --git a/cranelift/filetests/parser/branch.clif b/cranelift/filetests/filetests/parser/branch.clif
similarity index 100%
rename from cranelift/filetests/parser/branch.clif
rename to cranelift/filetests/filetests/parser/branch.clif
diff --git a/cranelift/filetests/parser/call.clif b/cranelift/filetests/filetests/parser/call.clif
similarity index 100%
rename from cranelift/filetests/parser/call.clif
rename to cranelift/filetests/filetests/parser/call.clif
diff --git a/cranelift/filetests/parser/flags.clif b/cranelift/filetests/filetests/parser/flags.clif
similarity index 100%
rename from cranelift/filetests/parser/flags.clif
rename to cranelift/filetests/filetests/parser/flags.clif
diff --git a/cranelift/filetests/parser/instruction_encoding.clif b/cranelift/filetests/filetests/parser/instruction_encoding.clif
similarity index 100%
rename from cranelift/filetests/parser/instruction_encoding.clif
rename to cranelift/filetests/filetests/parser/instruction_encoding.clif
diff --git a/cranelift/filetests/parser/keywords.clif b/cranelift/filetests/filetests/parser/keywords.clif
similarity index 100%
rename from cranelift/filetests/parser/keywords.clif
rename to cranelift/filetests/filetests/parser/keywords.clif
diff --git a/cranelift/filetests/parser/memory.clif b/cranelift/filetests/filetests/parser/memory.clif
similarity index 100%
rename from cranelift/filetests/parser/memory.clif
rename to cranelift/filetests/filetests/parser/memory.clif
diff --git a/cranelift/filetests/parser/rewrite.clif b/cranelift/filetests/filetests/parser/rewrite.clif
similarity index 100%
rename from cranelift/filetests/parser/rewrite.clif
rename to cranelift/filetests/filetests/parser/rewrite.clif
diff --git a/cranelift/filetests/parser/ternary.clif b/cranelift/filetests/filetests/parser/ternary.clif
similarity index 100%
rename from cranelift/filetests/parser/ternary.clif
rename to cranelift/filetests/filetests/parser/ternary.clif
diff --git a/cranelift/filetests/parser/tiny.clif b/cranelift/filetests/filetests/parser/tiny.clif
similarity index 100%
rename from cranelift/filetests/parser/tiny.clif
rename to cranelift/filetests/filetests/parser/tiny.clif
diff --git a/cranelift/filetests/postopt/basic.clif b/cranelift/filetests/filetests/postopt/basic.clif
similarity index 100%
rename from cranelift/filetests/postopt/basic.clif
rename to cranelift/filetests/filetests/postopt/basic.clif
diff --git a/cranelift/filetests/postopt/complex_memory_ops.clif b/cranelift/filetests/filetests/postopt/complex_memory_ops.clif
similarity index 100%
rename from cranelift/filetests/postopt/complex_memory_ops.clif
rename to cranelift/filetests/filetests/postopt/complex_memory_ops.clif
diff --git a/cranelift/filetests/postopt/fold_offset_into_address.clif b/cranelift/filetests/filetests/postopt/fold_offset_into_address.clif
similarity index 100%
rename from cranelift/filetests/postopt/fold_offset_into_address.clif
rename to cranelift/filetests/filetests/postopt/fold_offset_into_address.clif
diff --git a/cranelift/filetests/preopt/branch.clif b/cranelift/filetests/filetests/preopt/branch.clif
similarity index 100%
rename from cranelift/filetests/preopt/branch.clif
rename to cranelift/filetests/filetests/preopt/branch.clif
diff --git a/cranelift/filetests/preopt/numerical.clif b/cranelift/filetests/filetests/preopt/numerical.clif
similarity index 100%
rename from cranelift/filetests/preopt/numerical.clif
rename to cranelift/filetests/filetests/preopt/numerical.clif
diff --git a/cranelift/filetests/regalloc/aliases.clif b/cranelift/filetests/filetests/regalloc/aliases.clif
similarity index 100%
rename from cranelift/filetests/regalloc/aliases.clif
rename to cranelift/filetests/filetests/regalloc/aliases.clif
diff --git a/cranelift/filetests/regalloc/basic.clif b/cranelift/filetests/filetests/regalloc/basic.clif
similarity index 100%
rename from cranelift/filetests/regalloc/basic.clif
rename to cranelift/filetests/filetests/regalloc/basic.clif
diff --git a/cranelift/filetests/regalloc/coalesce.clif b/cranelift/filetests/filetests/regalloc/coalesce.clif
similarity index 100%
rename from cranelift/filetests/regalloc/coalesce.clif
rename to cranelift/filetests/filetests/regalloc/coalesce.clif
diff --git a/cranelift/filetests/regalloc/coalescing-207.clif b/cranelift/filetests/filetests/regalloc/coalescing-207.clif
similarity index 100%
rename from cranelift/filetests/regalloc/coalescing-207.clif
rename to cranelift/filetests/filetests/regalloc/coalescing-207.clif
diff --git a/cranelift/filetests/regalloc/coalescing-216.clif b/cranelift/filetests/filetests/regalloc/coalescing-216.clif
similarity index 100%
rename from cranelift/filetests/regalloc/coalescing-216.clif
rename to cranelift/filetests/filetests/regalloc/coalescing-216.clif
diff --git a/cranelift/filetests/regalloc/coloring-227.clif b/cranelift/filetests/filetests/regalloc/coloring-227.clif
similarity index 100%
rename from cranelift/filetests/regalloc/coloring-227.clif
rename to cranelift/filetests/filetests/regalloc/coloring-227.clif
diff --git a/cranelift/filetests/regalloc/constraints.clif b/cranelift/filetests/filetests/regalloc/constraints.clif
similarity index 100%
rename from cranelift/filetests/regalloc/constraints.clif
rename to cranelift/filetests/filetests/regalloc/constraints.clif
diff --git a/cranelift/filetests/regalloc/fallthrough-return.clif b/cranelift/filetests/filetests/regalloc/fallthrough-return.clif
similarity index 100%
rename from cranelift/filetests/regalloc/fallthrough-return.clif
rename to cranelift/filetests/filetests/regalloc/fallthrough-return.clif
diff --git a/cranelift/filetests/regalloc/ghost-param.clif b/cranelift/filetests/filetests/regalloc/ghost-param.clif
similarity index 100%
rename from cranelift/filetests/regalloc/ghost-param.clif
rename to cranelift/filetests/filetests/regalloc/ghost-param.clif
diff --git a/cranelift/filetests/regalloc/global-constraints.clif b/cranelift/filetests/filetests/regalloc/global-constraints.clif
similarity index 100%
rename from cranelift/filetests/regalloc/global-constraints.clif
rename to cranelift/filetests/filetests/regalloc/global-constraints.clif
diff --git a/cranelift/filetests/regalloc/global-fixed.clif b/cranelift/filetests/filetests/regalloc/global-fixed.clif
similarity index 100%
rename from cranelift/filetests/regalloc/global-fixed.clif
rename to cranelift/filetests/filetests/regalloc/global-fixed.clif
diff --git a/cranelift/filetests/regalloc/gpr-deref-safe-335.clif b/cranelift/filetests/filetests/regalloc/gpr-deref-safe-335.clif
similarity index 100%
rename from cranelift/filetests/regalloc/gpr-deref-safe-335.clif
rename to cranelift/filetests/filetests/regalloc/gpr-deref-safe-335.clif
diff --git a/cranelift/filetests/regalloc/infinite-interference.clif b/cranelift/filetests/filetests/regalloc/infinite-interference.clif
similarity index 100%
rename from cranelift/filetests/regalloc/infinite-interference.clif
rename to cranelift/filetests/filetests/regalloc/infinite-interference.clif
diff --git a/cranelift/filetests/regalloc/iterate.clif b/cranelift/filetests/filetests/regalloc/iterate.clif
similarity index 100%
rename from cranelift/filetests/regalloc/iterate.clif
rename to cranelift/filetests/filetests/regalloc/iterate.clif
diff --git a/cranelift/filetests/regalloc/multi-constraints.clif b/cranelift/filetests/filetests/regalloc/multi-constraints.clif
similarity index 100%
rename from cranelift/filetests/regalloc/multi-constraints.clif
rename to cranelift/filetests/filetests/regalloc/multi-constraints.clif
diff --git a/cranelift/filetests/regalloc/multiple-returns.clif b/cranelift/filetests/filetests/regalloc/multiple-returns.clif
similarity index 100%
rename from cranelift/filetests/regalloc/multiple-returns.clif
rename to cranelift/filetests/filetests/regalloc/multiple-returns.clif
diff --git a/cranelift/filetests/regalloc/output-interference.clif b/cranelift/filetests/filetests/regalloc/output-interference.clif
similarity index 100%
rename from cranelift/filetests/regalloc/output-interference.clif
rename to cranelift/filetests/filetests/regalloc/output-interference.clif
diff --git a/cranelift/filetests/regalloc/reload-208.clif b/cranelift/filetests/filetests/regalloc/reload-208.clif
similarity index 100%
rename from cranelift/filetests/regalloc/reload-208.clif
rename to cranelift/filetests/filetests/regalloc/reload-208.clif
diff --git a/cranelift/filetests/regalloc/reload.clif b/cranelift/filetests/filetests/regalloc/reload.clif
similarity index 100%
rename from cranelift/filetests/regalloc/reload.clif
rename to cranelift/filetests/filetests/regalloc/reload.clif
diff --git a/cranelift/filetests/regalloc/schedule-moves.clif b/cranelift/filetests/filetests/regalloc/schedule-moves.clif
similarity index 100%
rename from cranelift/filetests/regalloc/schedule-moves.clif
rename to cranelift/filetests/filetests/regalloc/schedule-moves.clif
diff --git a/cranelift/filetests/regalloc/spill-noregs.clif b/cranelift/filetests/filetests/regalloc/spill-noregs.clif
similarity index 98%
rename from cranelift/filetests/regalloc/spill-noregs.clif
rename to cranelift/filetests/filetests/regalloc/spill-noregs.clif
index 6e0b2baabb..c470b3355b 100644
--- a/cranelift/filetests/regalloc/spill-noregs.clif
+++ b/cranelift/filetests/filetests/regalloc/spill-noregs.clif
@@ -5,7 +5,7 @@ target x86_64
 ;
 ; The spiller panics with a
 ; 'Ran out of GPR registers when inserting copy before v68 = icmp.i32 eq v66, v67',
-; lib/codegen/src/regalloc/spilling.rs:425:28 message.
+; cranelift-codegen/src/regalloc/spilling.rs:425:28 message.
 ;
 ; The process_reg_uses() function is trying to insert a copy before the icmp instruction in ebb4
 ; and runs out of registers to spill. Note that ebb7 has a lot of dead parameter values.
diff --git a/cranelift/filetests/regalloc/spill.clif b/cranelift/filetests/filetests/regalloc/spill.clif
similarity index 100%
rename from cranelift/filetests/regalloc/spill.clif
rename to cranelift/filetests/filetests/regalloc/spill.clif
diff --git a/cranelift/filetests/regalloc/unreachable_code.clif b/cranelift/filetests/filetests/regalloc/unreachable_code.clif
similarity index 100%
rename from cranelift/filetests/regalloc/unreachable_code.clif
rename to cranelift/filetests/filetests/regalloc/unreachable_code.clif
diff --git a/cranelift/filetests/regalloc/x86-regres.clif b/cranelift/filetests/filetests/regalloc/x86-regres.clif
similarity index 100%
rename from cranelift/filetests/regalloc/x86-regres.clif
rename to cranelift/filetests/filetests/regalloc/x86-regres.clif
diff --git a/cranelift/filetests/simple_gvn/basic.clif b/cranelift/filetests/filetests/simple_gvn/basic.clif
similarity index 100%
rename from cranelift/filetests/simple_gvn/basic.clif
rename to cranelift/filetests/filetests/simple_gvn/basic.clif
diff --git a/cranelift/filetests/simple_gvn/readonly.clif b/cranelift/filetests/filetests/simple_gvn/readonly.clif
similarity index 100%
rename from cranelift/filetests/simple_gvn/readonly.clif
rename to cranelift/filetests/filetests/simple_gvn/readonly.clif
diff --git a/cranelift/filetests/simple_gvn/reject.clif b/cranelift/filetests/filetests/simple_gvn/reject.clif
similarity index 100%
rename from cranelift/filetests/simple_gvn/reject.clif
rename to cranelift/filetests/filetests/simple_gvn/reject.clif
diff --git a/cranelift/filetests/simple_gvn/scopes.clif b/cranelift/filetests/filetests/simple_gvn/scopes.clif
similarity index 100%
rename from cranelift/filetests/simple_gvn/scopes.clif
rename to cranelift/filetests/filetests/simple_gvn/scopes.clif
diff --git a/cranelift/filetests/simple_preopt/div_by_const_indirect.clif b/cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif
similarity index 100%
rename from cranelift/filetests/simple_preopt/div_by_const_indirect.clif
rename to cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif
diff --git a/cranelift/filetests/simple_preopt/div_by_const_non_power_of_2.clif b/cranelift/filetests/filetests/simple_preopt/div_by_const_non_power_of_2.clif
similarity index 100%
rename from cranelift/filetests/simple_preopt/div_by_const_non_power_of_2.clif
rename to cranelift/filetests/filetests/simple_preopt/div_by_const_non_power_of_2.clif
diff --git a/cranelift/filetests/simple_preopt/div_by_const_power_of_2.clif b/cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif
similarity index 100%
rename from cranelift/filetests/simple_preopt/div_by_const_power_of_2.clif
rename to cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif
diff --git a/cranelift/filetests/simple_preopt/rem_by_const_non_power_of_2.clif b/cranelift/filetests/filetests/simple_preopt/rem_by_const_non_power_of_2.clif
similarity index 100%
rename from cranelift/filetests/simple_preopt/rem_by_const_non_power_of_2.clif
rename to cranelift/filetests/filetests/simple_preopt/rem_by_const_non_power_of_2.clif
diff --git a/cranelift/filetests/simple_preopt/rem_by_const_power_of_2.clif b/cranelift/filetests/filetests/simple_preopt/rem_by_const_power_of_2.clif
similarity index 100%
rename from cranelift/filetests/simple_preopt/rem_by_const_power_of_2.clif
rename to cranelift/filetests/filetests/simple_preopt/rem_by_const_power_of_2.clif
diff --git a/cranelift/filetests/simple_preopt/simplify.clif b/cranelift/filetests/filetests/simple_preopt/simplify.clif
similarity index 100%
rename from cranelift/filetests/simple_preopt/simplify.clif
rename to cranelift/filetests/filetests/simple_preopt/simplify.clif
diff --git a/cranelift/filetests/verifier/bad_layout.clif b/cranelift/filetests/filetests/verifier/bad_layout.clif
similarity index 100%
rename from cranelift/filetests/verifier/bad_layout.clif
rename to cranelift/filetests/filetests/verifier/bad_layout.clif
diff --git a/cranelift/filetests/verifier/defs_dominates_uses.clif b/cranelift/filetests/filetests/verifier/defs_dominates_uses.clif
similarity index 100%
rename from cranelift/filetests/verifier/defs_dominates_uses.clif
rename to cranelift/filetests/filetests/verifier/defs_dominates_uses.clif
diff --git a/cranelift/filetests/verifier/flags.clif b/cranelift/filetests/filetests/verifier/flags.clif
similarity index 100%
rename from cranelift/filetests/verifier/flags.clif
rename to cranelift/filetests/filetests/verifier/flags.clif
diff --git a/cranelift/filetests/verifier/globals.clif b/cranelift/filetests/filetests/verifier/globals.clif
similarity index 100%
rename from cranelift/filetests/verifier/globals.clif
rename to cranelift/filetests/filetests/verifier/globals.clif
diff --git a/cranelift/filetests/verifier/heap.clif b/cranelift/filetests/filetests/verifier/heap.clif
similarity index 100%
rename from cranelift/filetests/verifier/heap.clif
rename to cranelift/filetests/filetests/verifier/heap.clif
diff --git a/cranelift/filetests/verifier/memory.clif b/cranelift/filetests/filetests/verifier/memory.clif
similarity index 100%
rename from cranelift/filetests/verifier/memory.clif
rename to cranelift/filetests/filetests/verifier/memory.clif
diff --git a/cranelift/filetests/verifier/table.clif b/cranelift/filetests/filetests/verifier/table.clif
similarity index 100%
rename from cranelift/filetests/verifier/table.clif
rename to cranelift/filetests/filetests/verifier/table.clif
diff --git a/cranelift/filetests/verifier/type_check.clif b/cranelift/filetests/filetests/verifier/type_check.clif
similarity index 100%
rename from cranelift/filetests/verifier/type_check.clif
rename to cranelift/filetests/filetests/verifier/type_check.clif
diff --git a/cranelift/filetests/verifier/undeclared_vmctx.clif b/cranelift/filetests/filetests/verifier/undeclared_vmctx.clif
similarity index 100%
rename from cranelift/filetests/verifier/undeclared_vmctx.clif
rename to cranelift/filetests/filetests/verifier/undeclared_vmctx.clif
diff --git a/cranelift/filetests/verifier/unreachable_code.clif b/cranelift/filetests/filetests/verifier/unreachable_code.clif
similarity index 100%
rename from cranelift/filetests/verifier/unreachable_code.clif
rename to cranelift/filetests/filetests/verifier/unreachable_code.clif
diff --git a/cranelift/filetests/wasm/control.clif b/cranelift/filetests/filetests/wasm/control.clif
similarity index 100%
rename from cranelift/filetests/wasm/control.clif
rename to cranelift/filetests/filetests/wasm/control.clif
diff --git a/cranelift/filetests/wasm/conversions.clif b/cranelift/filetests/filetests/wasm/conversions.clif
similarity index 100%
rename from cranelift/filetests/wasm/conversions.clif
rename to cranelift/filetests/filetests/wasm/conversions.clif
diff --git a/cranelift/filetests/wasm/f32-arith.clif b/cranelift/filetests/filetests/wasm/f32-arith.clif
similarity index 100%
rename from cranelift/filetests/wasm/f32-arith.clif
rename to cranelift/filetests/filetests/wasm/f32-arith.clif
diff --git a/cranelift/filetests/wasm/f32-compares.clif b/cranelift/filetests/filetests/wasm/f32-compares.clif
similarity index 100%
rename from cranelift/filetests/wasm/f32-compares.clif
rename to cranelift/filetests/filetests/wasm/f32-compares.clif
diff --git a/cranelift/filetests/wasm/f32-memory64.clif b/cranelift/filetests/filetests/wasm/f32-memory64.clif
similarity index 100%
rename from cranelift/filetests/wasm/f32-memory64.clif
rename to cranelift/filetests/filetests/wasm/f32-memory64.clif
diff --git a/cranelift/filetests/wasm/f64-arith.clif b/cranelift/filetests/filetests/wasm/f64-arith.clif
similarity index 100%
rename from cranelift/filetests/wasm/f64-arith.clif
rename to cranelift/filetests/filetests/wasm/f64-arith.clif
diff --git a/cranelift/filetests/wasm/f64-compares.clif b/cranelift/filetests/filetests/wasm/f64-compares.clif
similarity index 100%
rename from cranelift/filetests/wasm/f64-compares.clif
rename to cranelift/filetests/filetests/wasm/f64-compares.clif
diff --git a/cranelift/filetests/wasm/f64-memory64.clif b/cranelift/filetests/filetests/wasm/f64-memory64.clif
similarity index 100%
rename from cranelift/filetests/wasm/f64-memory64.clif
rename to cranelift/filetests/filetests/wasm/f64-memory64.clif
diff --git a/cranelift/filetests/wasm/i32-arith.clif b/cranelift/filetests/filetests/wasm/i32-arith.clif
similarity index 100%
rename from cranelift/filetests/wasm/i32-arith.clif
rename to cranelift/filetests/filetests/wasm/i32-arith.clif
diff --git a/cranelift/filetests/wasm/i32-compares.clif b/cranelift/filetests/filetests/wasm/i32-compares.clif
similarity index 100%
rename from cranelift/filetests/wasm/i32-compares.clif
rename to cranelift/filetests/filetests/wasm/i32-compares.clif
diff --git a/cranelift/filetests/wasm/i32-memory64.clif b/cranelift/filetests/filetests/wasm/i32-memory64.clif
similarity index 100%
rename from cranelift/filetests/wasm/i32-memory64.clif
rename to cranelift/filetests/filetests/wasm/i32-memory64.clif
diff --git a/cranelift/filetests/wasm/i64-arith.clif b/cranelift/filetests/filetests/wasm/i64-arith.clif
similarity index 100%
rename from cranelift/filetests/wasm/i64-arith.clif
rename to cranelift/filetests/filetests/wasm/i64-arith.clif
diff --git a/cranelift/filetests/wasm/i64-compares.clif b/cranelift/filetests/filetests/wasm/i64-compares.clif
similarity index 100%
rename from cranelift/filetests/wasm/i64-compares.clif
rename to cranelift/filetests/filetests/wasm/i64-compares.clif
diff --git a/cranelift/filetests/wasm/i64-memory64.clif b/cranelift/filetests/filetests/wasm/i64-memory64.clif
similarity index 100%
rename from cranelift/filetests/wasm/i64-memory64.clif
rename to cranelift/filetests/filetests/wasm/i64-memory64.clif
diff --git a/cranelift/filetests/wasm/select.clif b/cranelift/filetests/filetests/wasm/select.clif
similarity index 100%
rename from cranelift/filetests/wasm/select.clif
rename to cranelift/filetests/filetests/wasm/select.clif
diff --git a/lib/filetests/src/concurrent.rs b/cranelift/filetests/src/concurrent.rs
similarity index 100%
rename from lib/filetests/src/concurrent.rs
rename to cranelift/filetests/src/concurrent.rs
diff --git a/lib/filetests/src/lib.rs b/cranelift/filetests/src/lib.rs
similarity index 100%
rename from lib/filetests/src/lib.rs
rename to cranelift/filetests/src/lib.rs
diff --git a/lib/filetests/src/match_directive.rs b/cranelift/filetests/src/match_directive.rs
similarity index 100%
rename from lib/filetests/src/match_directive.rs
rename to cranelift/filetests/src/match_directive.rs
diff --git a/lib/filetests/src/runner.rs b/cranelift/filetests/src/runner.rs
similarity index 100%
rename from lib/filetests/src/runner.rs
rename to cranelift/filetests/src/runner.rs
diff --git a/lib/filetests/src/runone.rs b/cranelift/filetests/src/runone.rs
similarity index 100%
rename from lib/filetests/src/runone.rs
rename to cranelift/filetests/src/runone.rs
diff --git a/lib/filetests/src/subtest.rs b/cranelift/filetests/src/subtest.rs
similarity index 100%
rename from lib/filetests/src/subtest.rs
rename to cranelift/filetests/src/subtest.rs
diff --git a/lib/filetests/src/test_binemit.rs b/cranelift/filetests/src/test_binemit.rs
similarity index 100%
rename from lib/filetests/src/test_binemit.rs
rename to cranelift/filetests/src/test_binemit.rs
diff --git a/lib/filetests/src/test_cat.rs b/cranelift/filetests/src/test_cat.rs
similarity index 100%
rename from lib/filetests/src/test_cat.rs
rename to cranelift/filetests/src/test_cat.rs
diff --git a/lib/filetests/src/test_compile.rs b/cranelift/filetests/src/test_compile.rs
similarity index 100%
rename from lib/filetests/src/test_compile.rs
rename to cranelift/filetests/src/test_compile.rs
diff --git a/lib/filetests/src/test_dce.rs b/cranelift/filetests/src/test_dce.rs
similarity index 100%
rename from lib/filetests/src/test_dce.rs
rename to cranelift/filetests/src/test_dce.rs
diff --git a/lib/filetests/src/test_domtree.rs b/cranelift/filetests/src/test_domtree.rs
similarity index 100%
rename from lib/filetests/src/test_domtree.rs
rename to cranelift/filetests/src/test_domtree.rs
diff --git a/lib/filetests/src/test_legalizer.rs b/cranelift/filetests/src/test_legalizer.rs
similarity index 100%
rename from lib/filetests/src/test_legalizer.rs
rename to cranelift/filetests/src/test_legalizer.rs
diff --git a/lib/filetests/src/test_licm.rs b/cranelift/filetests/src/test_licm.rs
similarity index 100%
rename from lib/filetests/src/test_licm.rs
rename to cranelift/filetests/src/test_licm.rs
diff --git a/lib/filetests/src/test_postopt.rs b/cranelift/filetests/src/test_postopt.rs
similarity index 100%
rename from lib/filetests/src/test_postopt.rs
rename to cranelift/filetests/src/test_postopt.rs
diff --git a/lib/filetests/src/test_preopt.rs b/cranelift/filetests/src/test_preopt.rs
similarity index 100%
rename from lib/filetests/src/test_preopt.rs
rename to cranelift/filetests/src/test_preopt.rs
diff --git a/lib/filetests/src/test_print_cfg.rs b/cranelift/filetests/src/test_print_cfg.rs
similarity index 100%
rename from lib/filetests/src/test_print_cfg.rs
rename to cranelift/filetests/src/test_print_cfg.rs
diff --git a/lib/filetests/src/test_regalloc.rs b/cranelift/filetests/src/test_regalloc.rs
similarity index 100%
rename from lib/filetests/src/test_regalloc.rs
rename to cranelift/filetests/src/test_regalloc.rs
diff --git a/lib/filetests/src/test_shrink.rs b/cranelift/filetests/src/test_shrink.rs
similarity index 100%
rename from lib/filetests/src/test_shrink.rs
rename to cranelift/filetests/src/test_shrink.rs
diff --git a/lib/filetests/src/test_simple_gvn.rs b/cranelift/filetests/src/test_simple_gvn.rs
similarity index 100%
rename from lib/filetests/src/test_simple_gvn.rs
rename to cranelift/filetests/src/test_simple_gvn.rs
diff --git a/lib/filetests/src/test_simple_preopt.rs b/cranelift/filetests/src/test_simple_preopt.rs
similarity index 100%
rename from lib/filetests/src/test_simple_preopt.rs
rename to cranelift/filetests/src/test_simple_preopt.rs
diff --git a/lib/filetests/src/test_verifier.rs b/cranelift/filetests/src/test_verifier.rs
similarity index 100%
rename from lib/filetests/src/test_verifier.rs
rename to cranelift/filetests/src/test_verifier.rs
diff --git a/lib/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
similarity index 88%
rename from lib/frontend/Cargo.toml
rename to cranelift/frontend/Cargo.toml
index 9a5512199b..4ea74a9b13 100644
--- a/lib/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.28.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/lib/frontend/LICENSE b/cranelift/frontend/LICENSE
similarity index 100%
rename from lib/frontend/LICENSE
rename to cranelift/frontend/LICENSE
diff --git a/lib/frontend/README.md b/cranelift/frontend/README.md
similarity index 100%
rename from lib/frontend/README.md
rename to cranelift/frontend/README.md
diff --git a/lib/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
similarity index 100%
rename from lib/frontend/src/frontend.rs
rename to cranelift/frontend/src/frontend.rs
diff --git a/lib/frontend/src/lib.rs b/cranelift/frontend/src/lib.rs
similarity index 100%
rename from lib/frontend/src/lib.rs
rename to cranelift/frontend/src/lib.rs
diff --git a/lib/frontend/src/ssa.rs b/cranelift/frontend/src/ssa.rs
similarity index 100%
rename from lib/frontend/src/ssa.rs
rename to cranelift/frontend/src/ssa.rs
diff --git a/lib/frontend/src/switch.rs b/cranelift/frontend/src/switch.rs
similarity index 100%
rename from lib/frontend/src/switch.rs
rename to cranelift/frontend/src/switch.rs
diff --git a/lib/frontend/src/variable.rs b/cranelift/frontend/src/variable.rs
similarity index 100%
rename from lib/frontend/src/variable.rs
rename to cranelift/frontend/src/variable.rs
diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
index 0285feaa51..d064d0bb81 100644
--- a/cranelift/fuzz/Cargo.toml
+++ b/cranelift/fuzz/Cargo.toml
@@ -12,9 +12,9 @@ cargo-fuzz = true
 cargo-fuzz = "*"
 binaryen = { git = "https://github.com/pepyakin/binaryen-rs.git" }
 libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" }
-cranelift-codegen = { path = "../lib/codegen" }
-cranelift-wasm = { path = "../lib/wasm" }
-cranelift-reader = { path = "../lib/reader" }
+cranelift-codegen = { path = "../cranelift-codegen" }
+cranelift-wasm = { path = "../cranelift-wasm" }
+cranelift-reader = { path = "../cranelift-reader" }
 target-lexicon = "0.2.0"
 
 # Prevent this from interfering with workspaces
diff --git a/lib/module/Cargo.toml b/cranelift/module/Cargo.toml
similarity index 80%
rename from lib/module/Cargo.toml
rename to cranelift/module/Cargo.toml
index a19cea5d27..ac95debee2 100644
--- a/lib/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.28.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.28.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.28.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/lib/module/LICENSE b/cranelift/module/LICENSE
similarity index 100%
rename from lib/module/LICENSE
rename to cranelift/module/LICENSE
diff --git a/lib/module/README.md b/cranelift/module/README.md
similarity index 100%
rename from lib/module/README.md
rename to cranelift/module/README.md
diff --git a/lib/module/src/backend.rs b/cranelift/module/src/backend.rs
similarity index 100%
rename from lib/module/src/backend.rs
rename to cranelift/module/src/backend.rs
diff --git a/lib/module/src/data_context.rs b/cranelift/module/src/data_context.rs
similarity index 100%
rename from lib/module/src/data_context.rs
rename to cranelift/module/src/data_context.rs
diff --git a/lib/module/src/lib.rs b/cranelift/module/src/lib.rs
similarity index 100%
rename from lib/module/src/lib.rs
rename to cranelift/module/src/lib.rs
diff --git a/lib/module/src/module.rs b/cranelift/module/src/module.rs
similarity index 100%
rename from lib/module/src/module.rs
rename to cranelift/module/src/module.rs
diff --git a/lib/native/Cargo.toml b/cranelift/native/Cargo.toml
similarity index 90%
rename from lib/native/Cargo.toml
rename to cranelift/native/Cargo.toml
index 235cb2ee60..73961a7ddc 100644
--- a/lib/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.28.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/lib/native/LICENSE b/cranelift/native/LICENSE
similarity index 100%
rename from lib/native/LICENSE
rename to cranelift/native/LICENSE
diff --git a/lib/native/README.md b/cranelift/native/README.md
similarity index 100%
rename from lib/native/README.md
rename to cranelift/native/README.md
diff --git a/lib/native/src/lib.rs b/cranelift/native/src/lib.rs
similarity index 100%
rename from lib/native/src/lib.rs
rename to cranelift/native/src/lib.rs
diff --git a/lib/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
similarity index 80%
rename from lib/preopt/Cargo.toml
rename to cranelift/preopt/Cargo.toml
index 260017f103..6a050d984a 100644
--- a/lib/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.28.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.28.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.28.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/lib/preopt/LICENSE b/cranelift/preopt/LICENSE
similarity index 100%
rename from lib/preopt/LICENSE
rename to cranelift/preopt/LICENSE
diff --git a/lib/preopt/README.md b/cranelift/preopt/README.md
similarity index 100%
rename from lib/preopt/README.md
rename to cranelift/preopt/README.md
diff --git a/lib/preopt/src/constant_folding.rs b/cranelift/preopt/src/constant_folding.rs
similarity index 100%
rename from lib/preopt/src/constant_folding.rs
rename to cranelift/preopt/src/constant_folding.rs
diff --git a/lib/preopt/src/lib.rs b/cranelift/preopt/src/lib.rs
similarity index 100%
rename from lib/preopt/src/lib.rs
rename to cranelift/preopt/src/lib.rs
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 74e91feeec..45ec385069 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -15,7 +15,7 @@ version="0.28.0"
 #
 # The main Cargo.toml in the top-level directory is the cranelift-tools crate which we don't publish.
 echo "Updating crate versions to $version"
-for crate in . lib/* lib/codegen/meta; do
+for crate in . * cranelift-codegen/meta; do
     # Update the version number of this crate to $version.
     sed -i.bk -e "s/^version = .*/version = \"$version\"/" \
         "$crate/Cargo.toml"
@@ -41,5 +41,5 @@ for crate in \
     reader wasm module \
     faerie umbrella simplejit
 do
-    echo cargo publish --manifest-path "lib/$crate/Cargo.toml"
+    echo cargo publish --manifest-path "$crate/Cargo.toml"
 done
diff --git a/lib/reader/Cargo.toml b/cranelift/reader/Cargo.toml
similarity index 86%
rename from lib/reader/Cargo.toml
rename to cranelift/reader/Cargo.toml
index fc40aaec25..a3ddb9c835 100644
--- a/lib/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.28.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0" }
 target-lexicon = "0.2.0"
 
 [badges]
diff --git a/lib/reader/LICENSE b/cranelift/reader/LICENSE
similarity index 100%
rename from lib/reader/LICENSE
rename to cranelift/reader/LICENSE
diff --git a/lib/reader/README.md b/cranelift/reader/README.md
similarity index 100%
rename from lib/reader/README.md
rename to cranelift/reader/README.md
diff --git a/lib/reader/src/error.rs b/cranelift/reader/src/error.rs
similarity index 100%
rename from lib/reader/src/error.rs
rename to cranelift/reader/src/error.rs
diff --git a/lib/reader/src/isaspec.rs b/cranelift/reader/src/isaspec.rs
similarity index 100%
rename from lib/reader/src/isaspec.rs
rename to cranelift/reader/src/isaspec.rs
diff --git a/lib/reader/src/lexer.rs b/cranelift/reader/src/lexer.rs
similarity index 100%
rename from lib/reader/src/lexer.rs
rename to cranelift/reader/src/lexer.rs
diff --git a/lib/reader/src/lib.rs b/cranelift/reader/src/lib.rs
similarity index 100%
rename from lib/reader/src/lib.rs
rename to cranelift/reader/src/lib.rs
diff --git a/lib/reader/src/parser.rs b/cranelift/reader/src/parser.rs
similarity index 100%
rename from lib/reader/src/parser.rs
rename to cranelift/reader/src/parser.rs
diff --git a/lib/reader/src/sourcemap.rs b/cranelift/reader/src/sourcemap.rs
similarity index 100%
rename from lib/reader/src/sourcemap.rs
rename to cranelift/reader/src/sourcemap.rs
diff --git a/lib/reader/src/testcommand.rs b/cranelift/reader/src/testcommand.rs
similarity index 100%
rename from lib/reader/src/testcommand.rs
rename to cranelift/reader/src/testcommand.rs
diff --git a/lib/reader/src/testfile.rs b/cranelift/reader/src/testfile.rs
similarity index 100%
rename from lib/reader/src/testfile.rs
rename to cranelift/reader/src/testfile.rs
diff --git a/lib/serde/Cargo.toml b/cranelift/serde/Cargo.toml
similarity index 80%
rename from lib/serde/Cargo.toml
rename to cranelift/serde/Cargo.toml
index b9355ed933..c88f646195 100644
--- a/lib/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../codegen", version = "0.28.0" }
-cranelift-reader = { path = "../reader", version = "0.28.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.28.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/serde/LICENSE b/cranelift/serde/LICENSE
similarity index 100%
rename from lib/serde/LICENSE
rename to cranelift/serde/LICENSE
diff --git a/lib/serde/README.md b/cranelift/serde/README.md
similarity index 97%
rename from lib/serde/README.md
rename to cranelift/serde/README.md
index 081573e413..b6ce8c62fd 100644
--- a/lib/serde/README.md
+++ b/cranelift/serde/README.md
@@ -25,7 +25,7 @@ Where the -p flag outputs Cranelift IR as pretty JSON.
 For example to build and use clif-json:
 
 ``` {.sourceCode .sh}
-cd lib/serde
+cd cranelift-serde
 cargo build
 clif-json serialize -p test.clif
 ```
diff --git a/lib/serde/src/clif-json.rs b/cranelift/serde/src/clif-json.rs
similarity index 100%
rename from lib/serde/src/clif-json.rs
rename to cranelift/serde/src/clif-json.rs
diff --git a/lib/serde/src/serde_clif_json.rs b/cranelift/serde/src/serde_clif_json.rs
similarity index 100%
rename from lib/serde/src/serde_clif_json.rs
rename to cranelift/serde/src/serde_clif_json.rs
diff --git a/lib/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
similarity index 62%
rename from lib/simplejit/Cargo.toml
rename to cranelift/simplejit/Cargo.toml
index 1f1e8af119..c4042006fa 100644
--- a/lib/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.28.0" }
-cranelift-module = { path = "../module", version = "0.28.0" }
-cranelift-native = { path = "../native", version = "0.28.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.28.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.28.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.2.0" }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../umbrella", version = "0.28.0" }
-cranelift-frontend = { path = "../frontend", version = "0.28.0" }
-cranelift-entity = { path = "../entity", version = "0.28.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.28.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.28.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.28.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/lib/simplejit/LICENSE b/cranelift/simplejit/LICENSE
similarity index 100%
rename from lib/simplejit/LICENSE
rename to cranelift/simplejit/LICENSE
diff --git a/lib/simplejit/README.md b/cranelift/simplejit/README.md
similarity index 82%
rename from lib/simplejit/README.md
rename to cranelift/simplejit/README.md
index 3d31bb98e9..f37d2f0e14 100644
--- a/lib/simplejit/README.md
+++ b/cranelift/simplejit/README.md
@@ -5,4 +5,4 @@ This crate is extremely experimental.
 
 See the [example program] for a brief overview of how to use this.
 
-[example program]: https://github.com/CraneStation/cranelift/tree/master/lib/simplejit/examples/simplejit-minimal.rs
+[example program]: https://github.com/CraneStation/cranelift/tree/master/cranelift-simplejit/examples/simplejit-minimal.rs
diff --git a/lib/simplejit/examples/simplejit-minimal.rs b/cranelift/simplejit/examples/simplejit-minimal.rs
similarity index 100%
rename from lib/simplejit/examples/simplejit-minimal.rs
rename to cranelift/simplejit/examples/simplejit-minimal.rs
diff --git a/lib/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
similarity index 100%
rename from lib/simplejit/src/backend.rs
rename to cranelift/simplejit/src/backend.rs
diff --git a/lib/simplejit/src/lib.rs b/cranelift/simplejit/src/lib.rs
similarity index 100%
rename from lib/simplejit/src/lib.rs
rename to cranelift/simplejit/src/lib.rs
diff --git a/lib/simplejit/src/memory.rs b/cranelift/simplejit/src/memory.rs
similarity index 100%
rename from lib/simplejit/src/memory.rs
rename to cranelift/simplejit/src/memory.rs
diff --git a/lib/simplejit/tests/basic.rs b/cranelift/simplejit/tests/basic.rs
similarity index 100%
rename from lib/simplejit/tests/basic.rs
rename to cranelift/simplejit/tests/basic.rs
diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index ca6744ec1c..0ee57a0c82 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -42,7 +42,7 @@ fi
 
 # Check if any Python files have changed since we last checked them.
 tsfile="$topdir/target/meta-checked"
-meta_python="$topdir/lib/codegen/meta-python"
+meta_python="$topdir/cranelift-codegen/meta-python"
 if [ -f "$tsfile" ]; then
     needcheck=$(find "$meta_python" -name '*.py' -newer "$tsfile")
 else
diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh
index dab7e3afbd..43750a38ce 100755
--- a/cranelift/test-no_std.sh
+++ b/cranelift/test-no_std.sh
@@ -13,10 +13,12 @@ function banner {
 }
 
 # Test those packages which have no_std support.
-LIBS="codegen frontend wasm native preopt module entity bforest umbrella"
+LIBS="cranelift-codegen cranelift-frontend cranelift-wasm \
+cranelift-native cranelift-preopt cranelift-module \
+cranelift-entity cranelift-bforest cranelift-umbrella"
 for LIB in $LIBS; do
     banner "Rust unit tests in $LIB"
-    pushd "lib/$LIB" >/dev/null
+    pushd "$LIB" >/dev/null
 
     # Test with just "core" enabled.
     cargo +nightly test --no-default-features --features core
diff --git a/lib/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
similarity index 77%
rename from lib/umbrella/Cargo.toml
rename to cranelift/umbrella/Cargo.toml
index cf4b94a139..6d729708a8 100644
--- a/lib/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../codegen", version = "0.28.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.28.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.28.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/lib/umbrella/LICENSE b/cranelift/umbrella/LICENSE
similarity index 100%
rename from lib/umbrella/LICENSE
rename to cranelift/umbrella/LICENSE
diff --git a/lib/umbrella/README.md b/cranelift/umbrella/README.md
similarity index 100%
rename from lib/umbrella/README.md
rename to cranelift/umbrella/README.md
diff --git a/lib/umbrella/src/lib.rs b/cranelift/umbrella/src/lib.rs
similarity index 100%
rename from lib/umbrella/src/lib.rs
rename to cranelift/umbrella/src/lib.rs
diff --git a/lib/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
similarity index 78%
rename from lib/wasm/Cargo.toml
rename to cranelift/wasm/Cargo.toml
index fce084d13d..fb0dc9a752 100644
--- a/lib/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.23.0", default-features = false }
-cranelift-codegen = { path = "../codegen", version = "0.28.0", default-features = false }
-cranelift-entity = { path = "../entity", version = "0.28.0", default-features = false }
-cranelift-frontend = { path = "../frontend", version = "0.28.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.28.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.28.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
diff --git a/lib/wasm/LICENSE b/cranelift/wasm/LICENSE
similarity index 100%
rename from lib/wasm/LICENSE
rename to cranelift/wasm/LICENSE
diff --git a/lib/wasm/README.md b/cranelift/wasm/README.md
similarity index 100%
rename from lib/wasm/README.md
rename to cranelift/wasm/README.md
diff --git a/lib/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
similarity index 100%
rename from lib/wasm/src/code_translator.rs
rename to cranelift/wasm/src/code_translator.rs
diff --git a/lib/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
similarity index 100%
rename from lib/wasm/src/environ/dummy.rs
rename to cranelift/wasm/src/environ/dummy.rs
diff --git a/lib/wasm/src/environ/mod.rs b/cranelift/wasm/src/environ/mod.rs
similarity index 100%
rename from lib/wasm/src/environ/mod.rs
rename to cranelift/wasm/src/environ/mod.rs
diff --git a/lib/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
similarity index 100%
rename from lib/wasm/src/environ/spec.rs
rename to cranelift/wasm/src/environ/spec.rs
diff --git a/lib/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
similarity index 100%
rename from lib/wasm/src/func_translator.rs
rename to cranelift/wasm/src/func_translator.rs
diff --git a/lib/wasm/src/lib.rs b/cranelift/wasm/src/lib.rs
similarity index 100%
rename from lib/wasm/src/lib.rs
rename to cranelift/wasm/src/lib.rs
diff --git a/lib/wasm/src/module_translator.rs b/cranelift/wasm/src/module_translator.rs
similarity index 100%
rename from lib/wasm/src/module_translator.rs
rename to cranelift/wasm/src/module_translator.rs
diff --git a/lib/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
similarity index 100%
rename from lib/wasm/src/sections_translator.rs
rename to cranelift/wasm/src/sections_translator.rs
diff --git a/lib/wasm/src/state.rs b/cranelift/wasm/src/state.rs
similarity index 100%
rename from lib/wasm/src/state.rs
rename to cranelift/wasm/src/state.rs
diff --git a/lib/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
similarity index 100%
rename from lib/wasm/src/translation_utils.rs
rename to cranelift/wasm/src/translation_utils.rs
diff --git a/lib/wasm/tests/wasm_testsuite.rs b/cranelift/wasm/tests/wasm_testsuite.rs
similarity index 95%
rename from lib/wasm/tests/wasm_testsuite.rs
rename to cranelift/wasm/tests/wasm_testsuite.rs
index 30428954bd..c496470ec0 100644
--- a/lib/wasm/tests/wasm_testsuite.rs
+++ b/cranelift/wasm/tests/wasm_testsuite.rs
@@ -14,7 +14,7 @@ use wabt::wat2wasm;
 
 #[test]
 fn testsuite() {
-    let mut paths: Vec<_> = fs::read_dir("../../wasmtests")
+    let mut paths: Vec<_> = fs::read_dir("../wasmtests")
         .unwrap()
         .map(|r| r.unwrap())
         .filter(|p| {
@@ -39,7 +39,7 @@ fn testsuite() {
 fn use_fallthrough_return() {
     let flags = Flags::new(settings::builder());
     handle_module(
-        Path::new("../../wasmtests/use_fallthrough_return.wat"),
+        Path::new("../wasmtests/use_fallthrough_return.wat"),
         &flags,
         ReturnMode::FallthroughReturn,
     );

From 084722a1187bf6cbd03246b60172f6c97a6cc4bd Mon Sep 17 00:00:00 2001
From: Rett Berg 
Date: Wed, 6 Feb 2019 16:20:42 -0800
Subject: [PATCH 2312/3084] fix typos in cranelift-frontend/src/lib.rs

---
 cranelift/frontend/src/lib.rs | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/cranelift/frontend/src/lib.rs b/cranelift/frontend/src/lib.rs
index 7379503769..4cfb49939a 100644
--- a/cranelift/frontend/src/lib.rs
+++ b/cranelift/frontend/src/lib.rs
@@ -20,8 +20,8 @@
 //!
 //! This API has been designed to help you translate your mutable variables into
 //! [`SSA`](https://en.wikipedia.org/wiki/Static_single_assignment_form) form.
-//! [`use_var`](struct.FunctionBuilder.html#method.use_var) will returns the Cranelift IR value
-//! that corresponds to your mutable variable at a precise point in the program. However, you know
+//! [`use_var`](struct.FunctionBuilder.html#method.use_var) will return the Cranelift IR value
+//! that corresponds to your mutable variable at a precise point in the program. However, if you know
 //! beforehand that one of your variables is defined only once, for instance if it is the result
 //! of an intermediate expression in an expression-based language, then you can translate it
 //! directly by the Cranelift IR value returned by the instruction builder. Using the
@@ -30,7 +30,7 @@
 //! beforehand if a variable is immutable or not).
 //!
 //! The moral is that you should use these three functions to handle all your mutable variables,
-//! even those that are not present in the source code but artefacts of the translation. It is up
+//! even those that are not present in the source code but artifacts of the translation. It is up
 //! to you to keep a mapping between the mutable variables of your language and their `Variable`
 //! index that is used by Cranelift. Caution: as the `Variable` is used by Cranelift to index an
 //! array containing information about your mutable variables, when you create a new `Variable`

From d2082abb731c02e5c3c9756243936aeb3606e086 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 31 Jan 2019 15:33:51 -0800
Subject: [PATCH 2313/3084] Look for Cargo.toml files only in crate
 directories.

---
 cranelift/publish-all.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 45ec385069..7f2ccf90e9 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -15,7 +15,7 @@ version="0.28.0"
 #
 # The main Cargo.toml in the top-level directory is the cranelift-tools crate which we don't publish.
 echo "Updating crate versions to $version"
-for crate in . * cranelift-codegen/meta; do
+for crate in . cranelift-* cranelift-codegen/meta; do
     # Update the version number of this crate to $version.
     sed -i.bk -e "s/^version = .*/version = \"$version\"/" \
         "$crate/Cargo.toml"

From 9f9c3060919257263a46424f07ff1dee7b279a98 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 26 Feb 2019 15:59:00 -0800
Subject: [PATCH 2314/3084] Fix indentation warnings reported by flake8.

---
 cranelift/codegen/meta-python/semantics/__init__.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/meta-python/semantics/__init__.py b/cranelift/codegen/meta-python/semantics/__init__.py
index 6b3586db1f..1ce6b46712 100644
--- a/cranelift/codegen/meta-python/semantics/__init__.py
+++ b/cranelift/codegen/meta-python/semantics/__init__.py
@@ -48,8 +48,8 @@ def verify_semantics(inst, src, xforms):
             arg = rtl_var.rtl[0].expr.args[i]
             assert isinstance(arg, Var)
             for val in op.kind.possible_values():
-                    s[arg] = val
-                    new_variants.append(rtl_var.copy(s))
+                s[arg] = val
+                new_variants.append(rtl_var.copy(s))
         variants = new_variants
 
     # For any possible version of the src with concrete enumerated immediates
@@ -65,7 +65,7 @@ def verify_semantics(inst, src, xforms):
             matching_xforms = []  # type: List[XForm]
             for x in xforms:
                 if src.substitution(x.src, {}) is None:
-                        continue
+                    continue
 
                 # Translate t using x.symtab
                 t = {x.symtab[str(v)]:  tv for (v, tv) in t.items()}

From 68479e61155014eb131f0f45083d20109e46f4d8 Mon Sep 17 00:00:00 2001
From: Rett Berg 
Date: Wed, 6 Feb 2019 22:05:06 -0800
Subject: [PATCH 2315/3084] Clarify `br_table`

From comments in
https://github.com/CraneStation/cranelift/issues/101#issuecomment-461284555
---
 cranelift/codegen/meta-python/base/instructions.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/cranelift/codegen/meta-python/base/instructions.py b/cranelift/codegen/meta-python/base/instructions.py
index 53d1586abf..ebb7257f65 100644
--- a/cranelift/codegen/meta-python/base/instructions.py
+++ b/cranelift/codegen/meta-python/base/instructions.py
@@ -143,6 +143,12 @@ br_table = Instruction(
 
         Note that this branch instruction can't pass arguments to the targeted
         blocks. Split critical edges as needed to work around this.
+
+        Do not confuse this with "tables" in WebAssembly. ``br_table`` is for
+        jump tables with destinations within the current function only -- think
+        of a ``match`` in Rust or a ``switch`` in C.  If you want to call a
+        function in a dynamic library, that will typically use
+        ``call_indirect``.
         """,
         ins=(x, EBB, JT), is_branch=True, is_terminator=True)
 

From afa4a749c5a628692886f402cd870df0d4688ac8 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 5 Feb 2019 21:27:17 +0100
Subject: [PATCH 2316/3084] Fix #666: Change the way we consider a block has
 been visited in relaxation;

This was previously using the following condition to decide that a block
hadn't been visited yet: either dest_offset is non-0 or the block isn't
the entry block. Unfortunately, this didn't work when the first block
would be non-empty but wouldn't generate code at all.

Since the original code would do at least one pass over the entire code,
the first pass that determines initial EBB offsets is done separately,
without considering branch relaxation. This ensures that all EBBs have
been visited and have correct initial offsets, and doesn't require a
special check to know whether an EBB has been visited or not.
---
 cranelift/codegen/src/binemit/relaxation.rs   |  26 ++--
 .../filetests/isa/x86/relax_branch.clif       | 123 ++++++++++++++++++
 2 files changed, 141 insertions(+), 8 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/relax_branch.clif

diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index 2ef2167d2e..5bfa323646 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -55,13 +55,26 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> CodegenResult CodegenResult i64 uext [%rax] baldrdash {
+    ss0 = incoming_arg 24, offset -24
+    gv0 = vmctx
+    gv1 = iadd_imm.i64 gv0, 48
+    gv2 = load.i64 notrap aligned readonly gv0
+    heap0 = static gv2, min 0xd839_6000, bound 0x0001_0000_0000, offset_guard 0x8000_0000, index_type i32
+
+                                ebb0(v0: i32 [%rdi], v1: i32 [%rsi], v2: i64 [%r14]):
+@0005 [-]                           fallthrough ebb3(v0, v1)
+
+                                ebb3(v8: i32 [%rdi], v19: i32 [%rsi]):
+@0005 [RexOp1ldDisp8#808b,%rax]     v7 = load.i64 v2+48
+@0005 [RexOp1rcmp_ib#f083,%rflags]  v91 = ifcmp_imm v7, 0
+@0005 [trapif#00]                   trapif ne v91, interrupt
+[Op1umr#89,%rax]                    v105 = copy v8
+@000b [Op1r_ib#83,%rax]             v10 = iadd_imm v105, 1
+                                    v80 -> v10
+@0010 [Op1umr#89,%rcx]              v92 = uextend.i64 v8
+@0010 [RexOp1ld#808b,%rdx]          v93 = load.i64 notrap aligned readonly v2
+                                    v95 -> v93
+@0010 [Op2ldWithIndex#4be,%rcx]     v12 = sload8_complex.i32 v93+v92
+[Op1umr#89,%rbx]                    v106 = copy v12
+@0017 [Op1r_ib#40c1,%rbx]           v14 = ishl_imm v106, 24
+@001a [Op1r_ib#70c1,%rbx]           v16 = sshr_imm v14, 24
+[Op1umr#89,%rdi]                    v107 = copy v16
+@001f [Op1r_ib#83,%rdi]             v18 = iadd_imm v107, 32
+[RexOp1umr#89,%r8]                  v108 = copy v19
+@0026 [RexOp1r_ib#83,%r8]           v21 = iadd_imm v108, 1
+                                    v82 -> v21
+@002b [Op1umr#89,%rsi]              v94 = uextend.i64 v19
+@002b [Op2ldWithIndex#4be,%rdx]     v23 = sload8_complex.i32 v93+v94
+                                    v55 -> v23
+[Op1umr#89,%rsi]                    v109 = copy v23
+@0032 [Op1r_ib#40c1,%rsi]           v25 = ishl_imm v109, 24
+@0035 [Op1r_ib#70c1,%rsi]           v27 = sshr_imm v25, 24
+                                    v69 -> v27
+[RexOp1umr#89,%r9]                  v110 = copy v27
+@003a [RexOp1r_ib#83,%r9]           v29 = iadd_imm v110, 32
+                                    v68 -> v29
+@0042 [Op1r_ib#83,%rcx]             v31 = iadd_imm v12, -65
+@0045 [Op1r_ib#40c1,%rcx]           v33 = ishl_imm v31, 24
+@0048 [Op1r_ib#70c1,%rcx]           v35 = sshr_imm v33, 24
+@004c [Op1r_id#4081,%rcx]           v37 = band_imm v35, 255
+[Op1rcmp_ib#7083,%rflags]           v97 = ifcmp_imm v37, 26
+@0050 [Op1brib#70]                  brif sge v97, ebb6
+[Op1umr#89,%rcx]                    v101 = copy v18
+@0054 [Op1jmpb#eb]                  jump ebb5(v18, v101)
+
+                                ebb6:
+[Op1umr#89,%rcx]                    v102 = copy.i32 v16
+@0059 [RexOp1rmov#89]               regmove v102, %rcx -> %rdi
+@0059 [RexOp1rmov#89]               regmove.i32 v16, %rbx -> %rcx
+@0059 [-]                           fallthrough ebb5(v102, v16)
+
+                                ebb5(v41: i32 [%rdi], v84: i32 [%rcx]):
+                                    v83 -> v84
+@005d [Op1r_id#4081,%rdi]           v43 = band_imm v41, 255
+@0062 [Op1r_ib#40c1,%rdi]           v45 = ishl_imm v43, 24
+                                    v52 -> v45
+@0065 [RexOp1rmov#89]               regmove v45, %rdi -> %rbx
+@0065 [Op1r_ib#70c1,%rbx]           v47 = sshr_imm v45, 24
+                                    v54 -> v47
+@0068 [RexOp1rmov#89]               regmove v47, %rbx -> %rdi
+@0068 [Op1icscc_ib#7083,%rbx]       v49 = icmp_imm ne v47, 0
+@0068 [RexOp2urm_noflags#4b6,%r10]  v50 = bint.i32 v49
+@0076 [Op1r_ib#83,%rdx]             v57 = iadd_imm.i32 v23, -65
+@0079 [Op1r_ib#40c1,%rdx]           v59 = ishl_imm v57, 24
+@007c [Op1r_ib#70c1,%rdx]           v61 = sshr_imm v59, 24
+@0080 [Op1r_id#4081,%rdx]           v63 = band_imm v61, 255
+[Op1rcmp_ib#7083,%rflags]           v98 = ifcmp_imm v63, 26
+@0084 [RexOp1rmov#89]               regmove v47, %rdi -> %rbx
+@0084 [Op1brib#70]                  brif sge v98, ebb8
+[RexOp1umr#89,%rdx]                 v103 = copy.i32 v29
+@0088 [Op1jmpb#eb]                  jump ebb7(v29, v10, v21, v103)
+
+                                ebb8:
+[Op1umr#89,%rdx]                    v104 = copy.i32 v27
+@008d [RexOp1rmov#89]               regmove v104, %rdx -> %r9
+@008d [RexOp1rmov#89]               regmove.i32 v27, %rsi -> %rdx
+@008d [-]                           fallthrough ebb7(v104, v10, v21, v27)
+
+                                ebb7(v67: i32 [%r9], v79: i32 [%rax], v81: i32 [%r8], v87: i32 [%rdx]):
+@0091 [RexOp1r_id#4081,%r9]         v71 = band_imm v67, 255
+@0094 [RexOp1r_ib#40c1,%r9]         v73 = ishl_imm v71, 24
+@0097 [RexOp1r_ib#70c1,%r9]         v75 = sshr_imm v73, 24
+@0098 [RexOp1icscc#39,%rbx]         v76 = icmp.i32 eq v47, v75
+@0098 [Op2urm_noflags_abcd#4b6,%rbx] v77 = bint.i32 v76
+@0099 [RexOp1rr#21,%r10]            v78 = band.i32 v50, v77
+@009a [RexOp1tjccb#74]              brz v78, ebb9
+[RexOp1umr#89,%rcx]                 v99 = copy v81
+[Op1umr#89,%rdx]                    v100 = copy v79
+@00a4 [RexOp1rmov#89]               regmove v100, %rdx -> %rdi
+@00a4 [RexOp1rmov#89]               regmove v99, %rcx -> %rsi
+@00a4 [RexOp1jmpd#e9]               jump ebb3(v100, v99); bin: 40 e9 ffffff2c
+
+                                ebb9:
+@00a7 [-]                           fallthrough ebb4
+
+                                ebb4:
+@00ad [Op1r_id#4081,%rcx]           v86 = band_imm.i32 v84, 255
+@00b3 [Op1r_id#4081,%rdx]           v89 = band_imm.i32 v87, 255
+@00b4 [Op1rr#29,%rcx]               v90 = isub v86, v89
+@00b5 [-]                           fallthrough ebb2(v90)
+
+                                ebb2(v5: i32 [%rcx]):
+@00b6 [-]                           fallthrough ebb1(v5)
+
+                                ebb1(v3: i32 [%rcx]):
+@00b6 [Op1umr#89,%rax]              v96 = uextend.i64 v3
+@00b6 [-]                           fallthrough_return v96
+}

From 25fdda6134787bf53b69e6913c6ed2d4796b2f03 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 8 Feb 2019 19:34:55 +0100
Subject: [PATCH 2317/3084] [meta] Move source generation responsibility into
 the meta crate itself;

---
 cranelift/codegen/build.rs            | 85 ++++++++-------------------
 cranelift/codegen/meta/src/isa/mod.rs | 42 ++++++-------
 cranelift/codegen/meta/src/lib.rs     | 64 +++++++++++++++++++-
 3 files changed, 102 insertions(+), 89 deletions(-)

diff --git a/cranelift/codegen/build.rs b/cranelift/codegen/build.rs
index c4b1a12de6..6aeaff6834 100644
--- a/cranelift/codegen/build.rs
+++ b/cranelift/codegen/build.rs
@@ -20,7 +20,6 @@
 
 use cranelift_codegen_meta as meta;
 
-use crate::meta::isa::Isa;
 use std::env;
 use std::process;
 use std::time::Instant;
@@ -30,29 +29,31 @@ fn main() {
 
     let out_dir = env::var("OUT_DIR").expect("The OUT_DIR environment variable must be set");
     let target_triple = env::var("TARGET").expect("The TARGET environment variable must be set");
-    let cranelift_targets = env::var("CRANELIFT_TARGETS").ok();
-    let cranelift_targets = cranelift_targets.as_ref().map(|s| s.as_ref());
-    let python = identify_python();
 
     // Configure isa targets cfg.
-    match isa_targets(cranelift_targets, &target_triple) {
-        Ok(isa_targets) => {
-            for isa in &isa_targets {
-                println!("cargo:rustc-cfg=build_{}", isa.to_string());
-            }
-        }
-        Err(err) => {
-            eprintln!("Error: {}", err);
-            process::exit(1);
-        }
+    let cranelift_targets = env::var("CRANELIFT_TARGETS").ok();
+    let cranelift_targets = cranelift_targets
+        .as_ref()
+        .map(|s| s.as_ref())
+        .filter(|s: &&str| s.len() > 0);
+
+    let isas = match cranelift_targets {
+        Some("native") => meta::isa_from_arch(&target_triple.split('-').next().unwrap()),
+        Some(targets) => meta::isas_from_targets(targets.split(',').collect::>()),
+        None => meta::all_isas(),
+    }
+    .expect("Error when identifying CRANELIFT_TARGETS and TARGET");
+
+    for isa in &isas {
+        println!("cargo:rustc-cfg=build_{}", isa.to_string());
     }
 
     let cur_dir = env::current_dir().expect("Can't access current working directory");
     let crate_dir = cur_dir.as_path();
 
-    // Make sure we rebuild if this build script changes.
-    // I guess that won't happen if you have non-UTF8 bytes in your path names.
-    // The `build.py` script prints out its own dependencies.
+    // Make sure we rebuild if this build script changes (will not happen with
+    // if the path to this file contains non-UTF8 bytes). The `build.py` script
+    // prints out its own dependencies.
     println!(
         "cargo:rerun-if-changed={}",
         crate_dir.join("build.rs").to_str().unwrap()
@@ -65,6 +66,7 @@ fn main() {
     // Launch build script with Python. We'll just find python in the path.
     // Use -B to disable .pyc files, because they cause trouble for vendoring
     // scripts, and this is a build step that isn't run very often anyway.
+    let python = identify_python();
     let status = process::Command::new(python)
         .current_dir(crate_dir)
         .arg("-B")
@@ -83,12 +85,15 @@ fn main() {
     // emitted by the `meta` crate.
     // ------------------------------------------------------------------------
 
-    if let Err(err) = generate_meta(&out_dir) {
+    if let Err(err) = meta::generate(&isas, &out_dir) {
         eprintln!("Error: {}", err);
         process::exit(1);
     }
 
     if let Ok(_) = env::var("CRANELIFT_VERBOSE") {
+        for isa in &isas {
+            println!("cargo:warning=Includes support for {} ISA", isa.to_string());
+        }
         println!(
             "cargo:warning=Build step took {:?}.",
             Instant::now() - start_time
@@ -97,20 +102,6 @@ fn main() {
     }
 }
 
-fn generate_meta(out_dir: &str) -> Result<(), meta::error::Error> {
-    let shared_settings = meta::gen_settings::generate_common("new_settings.rs", &out_dir)?;
-    let isas = meta::isa::define_all(&shared_settings);
-
-    meta::gen_types::generate("types.rs", &out_dir)?;
-
-    for isa in &isas {
-        meta::gen_registers::generate(&isa, "registers", &out_dir)?;
-        meta::gen_settings::generate(&isa, "new_settings", &out_dir)?;
-    }
-
-    Ok(())
-}
-
 fn identify_python() -> &'static str {
     for python in &["python", "python3", "python2.7"] {
         if process::Command::new(python)
@@ -123,33 +114,3 @@ fn identify_python() -> &'static str {
     }
     panic!("The Cranelift build requires Python (version 2.7 or version 3)");
 }
-
-/// Returns isa targets to configure conditional compilation.
-fn isa_targets(cranelift_targets: Option<&str>, target_triple: &str) -> Result, String> {
-    match cranelift_targets {
-        Some("native") => Isa::from_arch(target_triple.split('-').next().unwrap())
-            .map(|isa| vec![isa])
-            .ok_or_else(|| {
-                format!(
-                    "no supported isa found for target triple `{}`",
-                    target_triple
-                )
-            }),
-        Some(targets) => {
-            let unknown_isa_targets = targets
-                .split(',')
-                .filter(|target| Isa::new(target).is_none())
-                .collect::>();
-            let isa_targets = targets.split(',').flat_map(Isa::new).collect::>();
-            match (unknown_isa_targets.is_empty(), isa_targets.is_empty()) {
-                (true, true) => Ok(Isa::all().to_vec()),
-                (true, _) => Ok(isa_targets),
-                (_, _) => Err(format!(
-                    "unknown isa targets: `{}`",
-                    unknown_isa_targets.join(", ")
-                )),
-            }
-        }
-        None => Ok(Isa::all().to_vec()),
-    }
-}
diff --git a/cranelift/codegen/meta/src/isa/mod.rs b/cranelift/codegen/meta/src/isa/mod.rs
index eaa359b801..c06326689b 100644
--- a/cranelift/codegen/meta/src/isa/mod.rs
+++ b/cranelift/codegen/meta/src/isa/mod.rs
@@ -18,7 +18,7 @@ pub enum Isa {
 
 impl Isa {
     /// Creates isa target using name.
-    pub fn new(name: &str) -> Option {
+    pub fn from_name(name: &str) -> Option {
         Isa::all()
             .iter()
             .cloned()
@@ -27,28 +27,20 @@ impl Isa {
     }
 
     /// Creates isa target from arch.
-    pub fn from_arch(arch: &str) -> Option {
-        Isa::all()
-            .iter()
-            .cloned()
-            .filter(|isa| isa.is_arch_applicable(arch))
-            .next()
+    pub fn from_arch(arch: &str) -> Option {
+        match arch {
+            "riscv" => Some(Isa::Riscv),
+            "aarch64" => Some(Isa::Arm64),
+            x if ["x86_64", "i386", "i586", "i686"].contains(&x) => Some(Isa::X86),
+            x if x.starts_with("arm") || arch.starts_with("thumb") => Some(Isa::Arm32),
+            _ => None,
+        }
     }
 
     /// Returns all supported isa targets.
     pub fn all() -> [Isa; 4] {
         [Isa::Riscv, Isa::X86, Isa::Arm32, Isa::Arm64]
     }
-
-    /// Checks if arch is applicable for the isa target.
-    fn is_arch_applicable(&self, arch: &str) -> bool {
-        match *self {
-            Isa::Riscv => arch == "riscv",
-            Isa::X86 => ["x86_64", "i386", "i586", "i686"].contains(&arch),
-            Isa::Arm32 => arch.starts_with("arm") || arch.starts_with("thumb"),
-            Isa::Arm64 => arch == "aarch64",
-        }
-    }
 }
 
 impl fmt::Display for Isa {
@@ -62,11 +54,13 @@ impl fmt::Display for Isa {
     }
 }
 
-pub fn define_all(shared_settings: &SettingGroup) -> Vec {
-    vec![
-        riscv::define(shared_settings),
-        arm32::define(shared_settings),
-        arm64::define(shared_settings),
-        x86::define(shared_settings),
-    ]
+pub fn define(isas: &Vec, shared_settings: &SettingGroup) -> Vec {
+    isas.iter()
+        .map(|isa| match isa {
+            Isa::Riscv => riscv::define(shared_settings),
+            Isa::X86 => x86::define(shared_settings),
+            Isa::Arm32 => arm32::define(shared_settings),
+            Isa::Arm64 => arm64::define(shared_settings),
+        })
+        .collect()
 }
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index d35d8cddcc..f7f15b8d09 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -2,12 +2,70 @@
 mod cdsl;
 
 pub mod error;
-pub mod gen_registers;
-pub mod gen_settings;
-pub mod gen_types;
 pub mod isa;
 
+mod gen_registers;
+mod gen_settings;
+mod gen_types;
+
 mod base;
 mod constant_hash;
 mod srcgen;
 mod unique_table;
+
+pub fn isa_from_arch(arch: &str) -> Result, String> {
+    isa::Isa::from_arch(arch)
+        .ok_or_else(|| format!("no supported isa found for arch `{}`", arch))
+        .and_then(|isa| Ok(vec![isa]))
+}
+
+pub fn isas_from_targets(targets: Vec<&str>) -> Result, String> {
+    type R<'a> = Vec<(&'a str, Option)>;
+
+    let (known, unknown): (R, R) = targets
+        .into_iter()
+        .map(|target| (target, isa::Isa::from_name(target)))
+        .partition(|(_, opt_isa)| opt_isa.is_some());
+
+    if !unknown.is_empty() {
+        let unknown_targets = unknown
+            .into_iter()
+            .map(|(target, _)| target)
+            .collect::>()
+            .join(", ");
+        return Err(format!("unknown isa targets: {}", unknown_targets));
+    }
+
+    let isas = if known.is_empty() {
+        isa::Isa::all().to_vec()
+    } else {
+        known
+            .into_iter()
+            .map(|(_, opt_isa)| opt_isa.unwrap())
+            .collect()
+    };
+
+    Ok(isas)
+}
+
+pub fn all_isas() -> Result, String> {
+    isas_from_targets(vec![])
+}
+
+/// Generates all the Rust source files used in Cranelift from the meta-language.
+pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error> {
+    // Common definitions.
+    let shared_settings = gen_settings::generate_common("new_settings.rs", &out_dir)?;
+
+    gen_types::generate("types.rs", &out_dir)?;
+
+    // Per ISA definitions.
+    let isas = isa::define(isas, &shared_settings);
+
+    for isa in isas {
+        gen_registers::generate(&isa, "registers", &out_dir)?;
+        gen_settings::generate(&isa, "new_settings", &out_dir)?;
+    }
+
+    Ok(())
+}

From f78a61b9986581fff2cbc13322889ab1c1d001de Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Feb 2019 11:24:53 +0100
Subject: [PATCH 2318/3084] [meta] Rename base/ to shared/ in the Rust meta
 crate;

---
 cranelift/codegen/meta/src/base/mod.rs        |  4 -
 cranelift/codegen/meta/src/cdsl/types.rs      | 80 +++++++++----------
 cranelift/codegen/meta/src/gen_settings.rs    |  4 +-
 cranelift/codegen/meta/src/lib.rs             |  2 +-
 cranelift/codegen/meta/src/shared/mod.rs      |  4 +
 .../meta/src/{base => shared}/settings.rs     |  0
 .../meta/src/{base => shared}/types.rs        |  0
 7 files changed, 47 insertions(+), 47 deletions(-)
 delete mode 100644 cranelift/codegen/meta/src/base/mod.rs
 create mode 100644 cranelift/codegen/meta/src/shared/mod.rs
 rename cranelift/codegen/meta/src/{base => shared}/settings.rs (100%)
 rename cranelift/codegen/meta/src/{base => shared}/types.rs (100%)

diff --git a/cranelift/codegen/meta/src/base/mod.rs b/cranelift/codegen/meta/src/base/mod.rs
deleted file mode 100644
index 1d596a1306..0000000000
--- a/cranelift/codegen/meta/src/base/mod.rs
+++ /dev/null
@@ -1,4 +0,0 @@
-//! Definitions for the base Cranelift language.
-
-pub mod settings;
-pub mod types;
diff --git a/cranelift/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
index a4e6443e47..898c048b8b 100644
--- a/cranelift/codegen/meta/src/cdsl/types.rs
+++ b/cranelift/codegen/meta/src/cdsl/types.rs
@@ -5,7 +5,7 @@
 
 use std::fmt;
 
-use crate::base::types as base_types;
+use crate::shared::types as shared_types;
 
 // Numbering scheme for value types:
 //
@@ -149,9 +149,9 @@ impl From for ValueType {
 /// A concrete scalar type that can appear as a vector lane too.
 #[derive(Clone, Copy)]
 pub enum LaneType {
-    BoolType(base_types::Bool),
-    FloatType(base_types::Float),
-    IntType(base_types::Int),
+    BoolType(shared_types::Bool),
+    FloatType(shared_types::Float),
+    IntType(shared_types::Int),
 }
 
 impl LaneType {
@@ -159,12 +159,12 @@ impl LaneType {
     pub fn doc(self) -> String {
         match self {
             LaneType::BoolType(_) => format!("A boolean type with {} bits.", self.lane_bits()),
-            LaneType::FloatType(base_types::Float::F32) => String::from(
+            LaneType::FloatType(shared_types::Float::F32) => String::from(
                 "A 32-bit floating point type represented in the IEEE 754-2008
                 *binary32* interchange format. This corresponds to the :c:type:`float`
                 type in most C implementations.",
             ),
-            LaneType::FloatType(base_types::Float::F64) => String::from(
+            LaneType::FloatType(shared_types::Float::F64) => String::from(
                 "A 64-bit floating point type represented in the IEEE 754-2008
                 *binary64* interchange format. This corresponds to the :c:type:`double`
                 type in most C implementations.",
@@ -192,17 +192,17 @@ impl LaneType {
     pub fn number(self) -> u8 {
         LANE_BASE
             + match self {
-                LaneType::BoolType(base_types::Bool::B1) => 0,
-                LaneType::BoolType(base_types::Bool::B8) => 1,
-                LaneType::BoolType(base_types::Bool::B16) => 2,
-                LaneType::BoolType(base_types::Bool::B32) => 3,
-                LaneType::BoolType(base_types::Bool::B64) => 4,
-                LaneType::IntType(base_types::Int::I8) => 5,
-                LaneType::IntType(base_types::Int::I16) => 6,
-                LaneType::IntType(base_types::Int::I32) => 7,
-                LaneType::IntType(base_types::Int::I64) => 8,
-                LaneType::FloatType(base_types::Float::F32) => 9,
-                LaneType::FloatType(base_types::Float::F64) => 10,
+                LaneType::BoolType(shared_types::Bool::B1) => 0,
+                LaneType::BoolType(shared_types::Bool::B8) => 1,
+                LaneType::BoolType(shared_types::Bool::B16) => 2,
+                LaneType::BoolType(shared_types::Bool::B32) => 3,
+                LaneType::BoolType(shared_types::Bool::B64) => 4,
+                LaneType::IntType(shared_types::Int::I8) => 5,
+                LaneType::IntType(shared_types::Int::I16) => 6,
+                LaneType::IntType(shared_types::Int::I32) => 7,
+                LaneType::IntType(shared_types::Int::I64) => 8,
+                LaneType::FloatType(shared_types::Float::F32) => 9,
+                LaneType::FloatType(shared_types::Float::F64) => 10,
             }
     }
 }
@@ -233,40 +233,40 @@ impl fmt::Debug for LaneType {
 }
 
 /// Create a LaneType from a given bool variant.
-impl From for LaneType {
-    fn from(b: base_types::Bool) -> Self {
+impl From for LaneType {
+    fn from(b: shared_types::Bool) -> Self {
         LaneType::BoolType(b)
     }
 }
 
 /// Create a LaneType from a given float variant.
-impl From for LaneType {
-    fn from(f: base_types::Float) -> Self {
+impl From for LaneType {
+    fn from(f: shared_types::Float) -> Self {
         LaneType::FloatType(f)
     }
 }
 
 /// Create a LaneType from a given int variant.
-impl From for LaneType {
-    fn from(i: base_types::Int) -> Self {
+impl From for LaneType {
+    fn from(i: shared_types::Int) -> Self {
         LaneType::IntType(i)
     }
 }
 
 /// An iterator for different lane types.
 pub struct LaneTypeIterator {
-    bool_iter: base_types::BoolIterator,
-    int_iter: base_types::IntIterator,
-    float_iter: base_types::FloatIterator,
+    bool_iter: shared_types::BoolIterator,
+    int_iter: shared_types::IntIterator,
+    float_iter: shared_types::FloatIterator,
 }
 
 impl LaneTypeIterator {
     /// Create a new lane type iterator.
     fn new() -> Self {
         Self {
-            bool_iter: base_types::BoolIterator::new(),
-            int_iter: base_types::IntIterator::new(),
-            float_iter: base_types::FloatIterator::new(),
+            bool_iter: shared_types::BoolIterator::new(),
+            int_iter: shared_types::IntIterator::new(),
+            float_iter: shared_types::FloatIterator::new(),
         }
     }
 }
@@ -388,18 +388,18 @@ impl fmt::Debug for BVType {
 /// Special types cannot be used to form vectors.
 #[derive(Clone, Copy)]
 pub enum SpecialType {
-    Flag(base_types::Flag),
+    Flag(shared_types::Flag),
 }
 
 impl SpecialType {
     /// Return a string containing the documentation comment for this special type.
     pub fn doc(self) -> String {
         match self {
-            SpecialType::Flag(base_types::Flag::IFlags) => String::from(
+            SpecialType::Flag(shared_types::Flag::IFlags) => String::from(
                 "CPU flags representing the result of an integer comparison. These flags
                 can be tested with an :type:`intcc` condition code.",
             ),
-            SpecialType::Flag(base_types::Flag::FFlags) => String::from(
+            SpecialType::Flag(shared_types::Flag::FFlags) => String::from(
                 "CPU flags representing the result of a floating point comparison. These
                 flags can be tested with a :type:`floatcc` condition code.",
             ),
@@ -416,8 +416,8 @@ impl SpecialType {
     /// Find the unique number associated with this special type.
     pub fn number(self) -> u8 {
         match self {
-            SpecialType::Flag(base_types::Flag::IFlags) => 1,
-            SpecialType::Flag(base_types::Flag::FFlags) => 2,
+            SpecialType::Flag(shared_types::Flag::IFlags) => 1,
+            SpecialType::Flag(shared_types::Flag::FFlags) => 2,
         }
     }
 }
@@ -425,8 +425,8 @@ impl SpecialType {
 impl fmt::Display for SpecialType {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
-            SpecialType::Flag(base_types::Flag::IFlags) => write!(f, "iflags"),
-            SpecialType::Flag(base_types::Flag::FFlags) => write!(f, "fflags"),
+            SpecialType::Flag(shared_types::Flag::IFlags) => write!(f, "iflags"),
+            SpecialType::Flag(shared_types::Flag::FFlags) => write!(f, "fflags"),
         }
     }
 }
@@ -443,20 +443,20 @@ impl fmt::Debug for SpecialType {
     }
 }
 
-impl From for SpecialType {
-    fn from(f: base_types::Flag) -> Self {
+impl From for SpecialType {
+    fn from(f: shared_types::Flag) -> Self {
         SpecialType::Flag(f)
     }
 }
 
 pub struct SpecialTypeIterator {
-    flag_iter: base_types::FlagIterator,
+    flag_iter: shared_types::FlagIterator,
 }
 
 impl SpecialTypeIterator {
     fn new() -> Self {
         Self {
-            flag_iter: base_types::FlagIterator::new(),
+            flag_iter: shared_types::FlagIterator::new(),
         }
     }
 }
diff --git a/cranelift/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
index bdcd206e13..99b9f52689 100644
--- a/cranelift/codegen/meta/src/gen_settings.rs
+++ b/cranelift/codegen/meta/src/gen_settings.rs
@@ -1,4 +1,3 @@
-use crate::base;
 use crate::cdsl::camel_case;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::settings::{
@@ -6,6 +5,7 @@ use crate::cdsl::settings::{
 };
 use crate::constant_hash::{generate_table, simple_hash};
 use crate::error;
+use crate::shared;
 use crate::srcgen::{Formatter, Match};
 use crate::unique_table::UniqueTable;
 use std::collections::HashMap;
@@ -433,7 +433,7 @@ fn gen_group(group: &SettingGroup, parent: ParentGroup, fmt: &mut Formatter) {
 }
 
 pub fn generate_common(filename: &str, out_dir: &str) -> Result {
-    let settings = base::settings::generate();
+    let settings = shared::settings::generate();
     let mut fmt = Formatter::new();
     gen_group(&settings, ParentGroup::None, &mut fmt);
     fmt.update_file(filename, out_dir)?;
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index f7f15b8d09..610f2cd4a2 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -8,8 +8,8 @@ mod gen_registers;
 mod gen_settings;
 mod gen_types;
 
-mod base;
 mod constant_hash;
+mod shared;
 mod srcgen;
 mod unique_table;
 
diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
new file mode 100644
index 0000000000..bdd9a14f2c
--- /dev/null
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -0,0 +1,4 @@
+//! Shared definitions for the Cranelift intermediate language.
+
+pub mod settings;
+pub mod types;
diff --git a/cranelift/codegen/meta/src/base/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
similarity index 100%
rename from cranelift/codegen/meta/src/base/settings.rs
rename to cranelift/codegen/meta/src/shared/settings.rs
diff --git a/cranelift/codegen/meta/src/base/types.rs b/cranelift/codegen/meta/src/shared/types.rs
similarity index 100%
rename from cranelift/codegen/meta/src/base/types.rs
rename to cranelift/codegen/meta/src/shared/types.rs

From 049f0671686110fb39f6d745b45a59a0a4868b96 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Feb 2019 12:34:44 +0100
Subject: [PATCH 2319/3084] [meta] Build registers with their own builder and
 immutably construct the TargetIsa;

---
 cranelift/codegen/meta/src/cdsl/isa.rs      | 182 +-------------------
 cranelift/codegen/meta/src/cdsl/regs.rs     | 179 ++++++++++++++++++-
 cranelift/codegen/meta/src/gen_registers.rs |  10 +-
 cranelift/codegen/meta/src/isa/arm32/mod.rs |  32 ++--
 cranelift/codegen/meta/src/isa/arm64/mod.rs |  28 +--
 cranelift/codegen/meta/src/isa/riscv/mod.rs |  24 ++-
 cranelift/codegen/meta/src/isa/x86/mod.rs   |  38 ++--
 7 files changed, 256 insertions(+), 237 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/isa.rs b/cranelift/codegen/meta/src/cdsl/isa.rs
index 65edbbbb8b..86b40a12c3 100644
--- a/cranelift/codegen/meta/src/cdsl/isa.rs
+++ b/cranelift/codegen/meta/src/cdsl/isa.rs
@@ -1,192 +1,18 @@
-use cranelift_entity::PrimaryMap;
-
-use super::regs::{
-    RegBank, RegBankBuilder, RegBankIndex, RegClass, RegClassBuilder, RegClassIndex, RegClassProto,
-};
+use super::regs::IsaRegs;
 use super::settings::SettingGroup;
 
 pub struct TargetIsa {
     pub name: &'static str,
-    pub reg_banks: PrimaryMap,
-    pub reg_classes: PrimaryMap,
     pub settings: SettingGroup,
+    pub regs: IsaRegs,
 }
 
 impl TargetIsa {
-    pub fn new(name: &'static str, settings: SettingGroup) -> Self {
+    pub fn new(name: &'static str, settings: SettingGroup, regs: IsaRegs) -> Self {
         Self {
             name,
-            reg_banks: PrimaryMap::new(),
-            reg_classes: PrimaryMap::new(),
             settings,
+            regs,
         }
     }
 }
-
-pub struct TargetIsaBuilder {
-    isa: TargetIsa,
-}
-
-impl TargetIsaBuilder {
-    pub fn new(name: &'static str, settings: SettingGroup) -> Self {
-        Self {
-            isa: TargetIsa::new(name, settings),
-        }
-    }
-
-    pub fn add_reg_bank(&mut self, builder: RegBankBuilder) -> RegBankIndex {
-        let first_unit = if self.isa.reg_banks.len() == 0 {
-            0
-        } else {
-            let last = &self.isa.reg_banks.last().unwrap();
-            let first_available_unit = (last.first_unit + last.units) as i8;
-            let units = builder.units;
-            let align = if units.is_power_of_two() {
-                units
-            } else {
-                units.next_power_of_two()
-            } as i8;
-            (first_available_unit + align - 1) & -align
-        } as u8;
-
-        self.isa.reg_banks.push(RegBank::new(
-            builder.name,
-            first_unit,
-            builder.units,
-            builder.names,
-            builder.prefix,
-            builder
-                .pressure_tracking
-                .expect("Pressure tracking must be explicitly set"),
-        ))
-    }
-
-    pub fn add_reg_class(&mut self, builder: RegClassBuilder) -> RegClassIndex {
-        let class_index = self.isa.reg_classes.next_key();
-
-        // Finish delayed construction of RegClass.
-        let (bank, toprc, start, width) = match builder.proto {
-            RegClassProto::TopLevel(bank_index) => {
-                self.isa
-                    .reg_banks
-                    .get_mut(bank_index)
-                    .unwrap()
-                    .toprcs
-                    .push(class_index);
-                (bank_index, class_index, builder.start, builder.width)
-            }
-            RegClassProto::SubClass(parent_class_index) => {
-                assert!(builder.width == 0);
-                let (bank, toprc, start, width) = {
-                    let parent = self.isa.reg_classes.get(parent_class_index).unwrap();
-                    (parent.bank, parent.toprc, parent.start, parent.width)
-                };
-                for reg_class in self.isa.reg_classes.values_mut() {
-                    if reg_class.toprc == toprc {
-                        reg_class.subclasses.push(class_index);
-                    }
-                }
-                let subclass_start = start + builder.start * width;
-                (bank, toprc, subclass_start, width)
-            }
-        };
-
-        let reg_bank_units = self.isa.reg_banks.get(bank).unwrap().units;
-        assert!(start < reg_bank_units);
-
-        let count = if builder.count != 0 {
-            builder.count
-        } else {
-            reg_bank_units / width
-        };
-
-        let reg_class = RegClass::new(builder.name, class_index, width, bank, toprc, count, start);
-        self.isa.reg_classes.push(reg_class);
-
-        let reg_bank = self.isa.reg_banks.get_mut(bank).unwrap();
-        reg_bank.classes.push(class_index);
-
-        class_index
-    }
-
-    /// Checks that the set of register classes satisfies:
-    ///
-    /// 1. Closed under intersection: The intersection of any two register
-    ///    classes in the set is either empty or identical to a member of the
-    ///    set.
-    /// 2. There are no identical classes under different names.
-    /// 3. Classes are sorted topologically such that all subclasses have a
-    ///    higher index that the superclass.
-    pub fn finish(self) -> TargetIsa {
-        for reg_bank in self.isa.reg_banks.values() {
-            for i1 in reg_bank.classes.iter() {
-                for i2 in reg_bank.classes.iter() {
-                    if i1 >= i2 {
-                        continue;
-                    }
-
-                    let rc1 = self.isa.reg_classes.get(*i1).unwrap();
-                    let rc2 = self.isa.reg_classes.get(*i2).unwrap();
-
-                    let rc1_mask = rc1.mask(0);
-                    let rc2_mask = rc2.mask(0);
-
-                    assert!(
-                        rc1.width != rc2.width || rc1_mask != rc2_mask,
-                        "no duplicates"
-                    );
-                    if rc1.width != rc2.width {
-                        continue;
-                    }
-
-                    let mut intersect = Vec::new();
-                    for (a, b) in rc1_mask.iter().zip(rc2_mask.iter()) {
-                        intersect.push(a & b);
-                    }
-                    if intersect == vec![0; intersect.len()] {
-                        continue;
-                    }
-
-                    // Classes must be topologically ordered, so the intersection can't be the
-                    // superclass.
-                    assert!(intersect != rc1_mask);
-
-                    // If the intersection is the second one, then it must be a subclass.
-                    if intersect == rc2_mask {
-                        assert!(self
-                            .isa
-                            .reg_classes
-                            .get(*i1)
-                            .unwrap()
-                            .subclasses
-                            .iter()
-                            .find(|x| **x == *i2)
-                            .is_some());
-                    }
-                }
-            }
-        }
-
-        // This limit should be coordinated with the `RegClassMask` and `RegClassIndex` types in
-        // isa/registers.rs of the non-meta code.
-        assert!(
-            self.isa.reg_classes.len() <= 32,
-            "Too many register classes"
-        );
-
-        // The maximum number of top-level register classes which have pressure tracking should be
-        // kept in sync with the MAX_TRACKED_TOPRCS constant in isa/registers.rs of the non-meta
-        // code.
-        let num_toplevel = self
-            .isa
-            .reg_classes
-            .values()
-            .filter(|x| {
-                x.toprc == x.index && self.isa.reg_banks.get(x.bank).unwrap().pressure_tracking
-            })
-            .count();
-        assert!(num_toplevel <= 4, "Too many top-level register classes");
-
-        self.isa
-    }
-}
diff --git a/cranelift/codegen/meta/src/cdsl/regs.rs b/cranelift/codegen/meta/src/cdsl/regs.rs
index 11605de563..22d7d3a6c1 100644
--- a/cranelift/codegen/meta/src/cdsl/regs.rs
+++ b/cranelift/codegen/meta/src/cdsl/regs.rs
@@ -1,5 +1,4 @@
-use cranelift_entity::entity_impl;
-use cranelift_entity::EntityRef;
+use cranelift_entity::{entity_impl, EntityRef, PrimaryMap};
 
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct RegBankIndex(u32);
@@ -178,3 +177,179 @@ impl RegBankBuilder {
         self
     }
 }
+
+pub struct IsaRegsBuilder {
+    pub banks: PrimaryMap,
+    pub classes: PrimaryMap,
+}
+
+impl IsaRegsBuilder {
+    pub fn new() -> Self {
+        Self {
+            banks: PrimaryMap::new(),
+            classes: PrimaryMap::new(),
+        }
+    }
+
+    pub fn add_bank(&mut self, builder: RegBankBuilder) -> RegBankIndex {
+        let first_unit = if self.banks.len() == 0 {
+            0
+        } else {
+            let last = &self.banks.last().unwrap();
+            let first_available_unit = (last.first_unit + last.units) as i8;
+            let units = builder.units;
+            let align = if units.is_power_of_two() {
+                units
+            } else {
+                units.next_power_of_two()
+            } as i8;
+            (first_available_unit + align - 1) & -align
+        } as u8;
+
+        self.banks.push(RegBank::new(
+            builder.name,
+            first_unit,
+            builder.units,
+            builder.names,
+            builder.prefix,
+            builder
+                .pressure_tracking
+                .expect("Pressure tracking must be explicitly set"),
+        ))
+    }
+
+    pub fn add_class(&mut self, builder: RegClassBuilder) -> RegClassIndex {
+        let class_index = self.classes.next_key();
+
+        // Finish delayed construction of RegClass.
+        let (bank, toprc, start, width) = match builder.proto {
+            RegClassProto::TopLevel(bank_index) => {
+                self.banks
+                    .get_mut(bank_index)
+                    .unwrap()
+                    .toprcs
+                    .push(class_index);
+                (bank_index, class_index, builder.start, builder.width)
+            }
+            RegClassProto::SubClass(parent_class_index) => {
+                assert!(builder.width == 0);
+                let (bank, toprc, start, width) = {
+                    let parent = self.classes.get(parent_class_index).unwrap();
+                    (parent.bank, parent.toprc, parent.start, parent.width)
+                };
+                for reg_class in self.classes.values_mut() {
+                    if reg_class.toprc == toprc {
+                        reg_class.subclasses.push(class_index);
+                    }
+                }
+                let subclass_start = start + builder.start * width;
+                (bank, toprc, subclass_start, width)
+            }
+        };
+
+        let reg_bank_units = self.banks.get(bank).unwrap().units;
+        assert!(start < reg_bank_units);
+
+        let count = if builder.count != 0 {
+            builder.count
+        } else {
+            reg_bank_units / width
+        };
+
+        let reg_class = RegClass::new(builder.name, class_index, width, bank, toprc, count, start);
+        self.classes.push(reg_class);
+
+        let reg_bank = self.banks.get_mut(bank).unwrap();
+        reg_bank.classes.push(class_index);
+
+        class_index
+    }
+
+    /// Checks that the set of register classes satisfies:
+    ///
+    /// 1. Closed under intersection: The intersection of any two register
+    ///    classes in the set is either empty or identical to a member of the
+    ///    set.
+    /// 2. There are no identical classes under different names.
+    /// 3. Classes are sorted topologically such that all subclasses have a
+    ///    higher index that the superclass.
+    pub fn finish(self) -> IsaRegs {
+        for reg_bank in self.banks.values() {
+            for i1 in reg_bank.classes.iter() {
+                for i2 in reg_bank.classes.iter() {
+                    if i1 >= i2 {
+                        continue;
+                    }
+
+                    let rc1 = self.classes.get(*i1).unwrap();
+                    let rc2 = self.classes.get(*i2).unwrap();
+
+                    let rc1_mask = rc1.mask(0);
+                    let rc2_mask = rc2.mask(0);
+
+                    assert!(
+                        rc1.width != rc2.width || rc1_mask != rc2_mask,
+                        "no duplicates"
+                    );
+                    if rc1.width != rc2.width {
+                        continue;
+                    }
+
+                    let mut intersect = Vec::new();
+                    for (a, b) in rc1_mask.iter().zip(rc2_mask.iter()) {
+                        intersect.push(a & b);
+                    }
+                    if intersect == vec![0; intersect.len()] {
+                        continue;
+                    }
+
+                    // Classes must be topologically ordered, so the intersection can't be the
+                    // superclass.
+                    assert!(intersect != rc1_mask);
+
+                    // If the intersection is the second one, then it must be a subclass.
+                    if intersect == rc2_mask {
+                        assert!(self
+                            .classes
+                            .get(*i1)
+                            .unwrap()
+                            .subclasses
+                            .iter()
+                            .find(|x| **x == *i2)
+                            .is_some());
+                    }
+                }
+            }
+        }
+
+        // This limit should be coordinated with the `RegClassMask` and `RegClassIndex` types in
+        // isa/registers.rs of the non-meta code.
+        assert!(self.classes.len() <= 32, "Too many register classes");
+
+        // The maximum number of top-level register classes which have pressure tracking should be
+        // kept in sync with the MAX_TRACKED_TOPRCS constant in isa/registers.rs of the non-meta
+        // code.
+        let num_toplevel = self
+            .classes
+            .values()
+            .filter(|x| x.toprc == x.index && self.banks.get(x.bank).unwrap().pressure_tracking)
+            .count();
+        assert!(num_toplevel <= 4, "Too many top-level register classes");
+
+        IsaRegs::new(self.banks, self.classes)
+    }
+}
+
+pub struct IsaRegs {
+    pub banks: PrimaryMap,
+    pub classes: PrimaryMap,
+}
+
+impl IsaRegs {
+    fn new(
+        banks: PrimaryMap,
+        classes: PrimaryMap,
+    ) -> Self {
+        Self { banks, classes }
+    }
+}
diff --git a/cranelift/codegen/meta/src/gen_registers.rs b/cranelift/codegen/meta/src/gen_registers.rs
index 5fb326e84e..8488dab228 100644
--- a/cranelift/codegen/meta/src/gen_registers.rs
+++ b/cranelift/codegen/meta/src/gen_registers.rs
@@ -32,7 +32,7 @@ fn gen_regbank(fmt: &mut Formatter, reg_bank: &RegBank) {
 }
 
 fn gen_regclass(isa: &TargetIsa, reg_class: &RegClass, fmt: &mut Formatter) {
-    let reg_bank = isa.reg_banks.get(reg_class.bank).unwrap();
+    let reg_bank = isa.regs.banks.get(reg_class.bank).unwrap();
 
     let mask: Vec = reg_class
         .mask(reg_bank.first_unit)
@@ -86,7 +86,7 @@ fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) {
         fmt.line("banks: &[");
         // Bank descriptors.
         fmt.indent(|fmt| {
-            for reg_bank in isa.reg_banks.values() {
+            for reg_bank in isa.regs.banks.values() {
                 gen_regbank(fmt, ®_bank);
             }
         });
@@ -94,7 +94,7 @@ fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) {
         // References to register classes.
         fmt.line("classes: &[");
         fmt.indent(|fmt| {
-            for reg_class in isa.reg_classes.values() {
+            for reg_class in isa.regs.classes.values() {
                 fmt.line(&format!("&{}_DATA,", reg_class.name));
             }
         });
@@ -103,7 +103,7 @@ fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) {
     fmt.line("};");
 
     // Register class descriptors.
-    for rc in isa.reg_classes.values() {
+    for rc in isa.regs.classes.values() {
         gen_regclass(&isa, rc, fmt);
     }
 
@@ -112,7 +112,7 @@ fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) {
     fmt.line("#[derive(Clone, Copy)]");
     fmt.line("pub enum RU {");
     fmt.indent(|fmt| {
-        for reg_bank in isa.reg_banks.values() {
+        for reg_bank in isa.regs.banks.values() {
             gen_regbank_units(reg_bank, fmt);
         }
     });
diff --git a/cranelift/codegen/meta/src/isa/arm32/mod.rs b/cranelift/codegen/meta/src/isa/arm32/mod.rs
index ad615dfad2..e3decd4da0 100644
--- a/cranelift/codegen/meta/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm32/mod.rs
@@ -1,5 +1,5 @@
-use crate::cdsl::isa::{TargetIsa, TargetIsaBuilder};
-use crate::cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
 
 fn define_settings(_shared: &SettingGroup) -> SettingGroup {
@@ -7,39 +7,45 @@ fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     setting.finish()
 }
 
-pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
-    let mut isa = TargetIsaBuilder::new("arm32", define_settings(shared_settings));
+fn define_regs() -> IsaRegs {
+    let mut regs = IsaRegsBuilder::new();
 
     let builder = RegBankBuilder::new("FloatRegs", "s")
         .units(64)
         .track_pressure(true);
-    let float_regs = isa.add_reg_bank(builder);
+    let float_regs = regs.add_bank(builder);
 
     let builder = RegBankBuilder::new("IntRegs", "r")
         .units(16)
         .track_pressure(true);
-    let int_regs = isa.add_reg_bank(builder);
+    let int_regs = regs.add_bank(builder);
 
     let builder = RegBankBuilder::new("FlagRegs", "")
         .units(1)
         .names(vec!["nzcv"])
         .track_pressure(false);
-    let flag_reg = isa.add_reg_bank(builder);
+    let flag_reg = regs.add_bank(builder);
 
     let builder = RegClassBuilder::new_toplevel("S", float_regs).count(32);
-    isa.add_reg_class(builder);
+    regs.add_class(builder);
 
     let builder = RegClassBuilder::new_toplevel("D", float_regs).width(2);
-    isa.add_reg_class(builder);
+    regs.add_class(builder);
 
     let builder = RegClassBuilder::new_toplevel("Q", float_regs).width(4);
-    isa.add_reg_class(builder);
+    regs.add_class(builder);
 
     let builder = RegClassBuilder::new_toplevel("GPR", int_regs);
-    isa.add_reg_class(builder);
+    regs.add_class(builder);
 
     let builder = RegClassBuilder::new_toplevel("FLAG", flag_reg);
-    isa.add_reg_class(builder);
+    regs.add_class(builder);
 
-    isa.finish()
+    regs.finish()
+}
+
+pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
+    let settings = define_settings(shared_settings);
+    let regs = define_regs();
+    TargetIsa::new("arm32", settings, regs)
 }
diff --git a/cranelift/codegen/meta/src/isa/arm64/mod.rs b/cranelift/codegen/meta/src/isa/arm64/mod.rs
index 7abdd51ca3..baca5226ab 100644
--- a/cranelift/codegen/meta/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm64/mod.rs
@@ -1,5 +1,5 @@
-use crate::cdsl::isa::{TargetIsa, TargetIsaBuilder};
-use crate::cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
 
 fn define_settings(_shared: &SettingGroup) -> SettingGroup {
@@ -7,35 +7,41 @@ fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     setting.finish()
 }
 
-pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
-    let mut isa = TargetIsaBuilder::new("arm64", define_settings(shared_settings));
+fn define_registers() -> IsaRegs {
+    let mut regs = IsaRegsBuilder::new();
 
     // The `x31` regunit serves as the stack pointer / zero register depending on context. We
     // reserve it and don't model the difference.
     let builder = RegBankBuilder::new("IntRegs", "x")
         .units(32)
         .track_pressure(true);
-    let int_regs = isa.add_reg_bank(builder);
+    let int_regs = regs.add_bank(builder);
 
     let builder = RegBankBuilder::new("FloatRegs", "v")
         .units(32)
         .track_pressure(true);
-    let float_regs = isa.add_reg_bank(builder);
+    let float_regs = regs.add_bank(builder);
 
     let builder = RegBankBuilder::new("FlagRegs", "")
         .units(1)
         .names(vec!["nzcv"])
         .track_pressure(false);
-    let flag_reg = isa.add_reg_bank(builder);
+    let flag_reg = regs.add_bank(builder);
 
     let builder = RegClassBuilder::new_toplevel("GPR", int_regs);
-    isa.add_reg_class(builder);
+    regs.add_class(builder);
 
     let builder = RegClassBuilder::new_toplevel("FPR", float_regs);
-    isa.add_reg_class(builder);
+    regs.add_class(builder);
 
     let builder = RegClassBuilder::new_toplevel("FLAG", flag_reg);
-    isa.add_reg_class(builder);
+    regs.add_class(builder);
 
-    isa.finish()
+    regs.finish()
+}
+
+pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
+    let settings = define_settings(shared_settings);
+    let regs = define_registers();
+    TargetIsa::new("arm64", settings, regs)
 }
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index dbf0e16e8a..040d088386 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -1,5 +1,5 @@
-use crate::cdsl::isa::{TargetIsa, TargetIsaBuilder};
-use crate::cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 
 fn define_settings(shared: &SettingGroup) -> SettingGroup {
@@ -54,24 +54,30 @@ fn define_settings(shared: &SettingGroup) -> SettingGroup {
     setting.finish()
 }
 
-pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
-    let mut isa = TargetIsaBuilder::new("riscv", define_settings(shared_settings));
+fn define_registers() -> IsaRegs {
+    let mut regs = IsaRegsBuilder::new();
 
     let builder = RegBankBuilder::new("IntRegs", "x")
         .units(32)
         .track_pressure(true);
-    let int_regs = isa.add_reg_bank(builder);
+    let int_regs = regs.add_bank(builder);
 
     let builder = RegBankBuilder::new("FloatRegs", "f")
         .units(32)
         .track_pressure(true);
-    let float_regs = isa.add_reg_bank(builder);
+    let float_regs = regs.add_bank(builder);
 
     let builder = RegClassBuilder::new_toplevel("GPR", int_regs);
-    isa.add_reg_class(builder);
+    regs.add_class(builder);
 
     let builder = RegClassBuilder::new_toplevel("FPR", float_regs);
-    isa.add_reg_class(builder);
+    regs.add_class(builder);
 
-    isa.finish()
+    regs.finish()
+}
+
+pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
+    let settings = define_settings(shared_settings);
+    let regs = define_registers();
+    TargetIsa::new("riscv", settings, regs)
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index 63087a2697..da64f3cc50 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -1,8 +1,8 @@
-use crate::cdsl::isa::{TargetIsa, TargetIsaBuilder};
-use crate::cdsl::regs::{RegBankBuilder, RegClassBuilder};
+use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 
-pub fn define_settings(_shared: &SettingGroup) -> SettingGroup {
+fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     let mut settings = SettingGroupBuilder::new("x86");
 
     // CPUID.01H:ECX
@@ -68,49 +68,49 @@ pub fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     settings.finish()
 }
 
-fn define_registers(isa: &mut TargetIsaBuilder) {
+fn define_registers() -> IsaRegs {
+    let mut regs = IsaRegsBuilder::new();
+
     let builder = RegBankBuilder::new("IntRegs", "r")
         .units(16)
         .names(vec!["rax", "rcx", "rdx", "rbx", "rsp", "rbp", "rsi", "rdi"])
         .track_pressure(true);
-    let int_regs = isa.add_reg_bank(builder);
+    let int_regs = regs.add_bank(builder);
 
     let builder = RegBankBuilder::new("FloatRegs", "xmm")
         .units(16)
         .track_pressure(true);
-    let float_regs = isa.add_reg_bank(builder);
+    let float_regs = regs.add_bank(builder);
 
     let builder = RegBankBuilder::new("FlagRegs", "")
         .units(1)
         .names(vec!["rflags"])
         .track_pressure(false);
-    let flag_reg = isa.add_reg_bank(builder);
+    let flag_reg = regs.add_bank(builder);
 
     let builder = RegClassBuilder::new_toplevel("GPR", int_regs);
-    let gpr = isa.add_reg_class(builder);
+    let gpr = regs.add_class(builder);
 
     let builder = RegClassBuilder::new_toplevel("FPR", float_regs);
-    let fpr = isa.add_reg_class(builder);
+    let fpr = regs.add_class(builder);
 
     let builder = RegClassBuilder::new_toplevel("FLAG", flag_reg);
-    isa.add_reg_class(builder);
+    regs.add_class(builder);
 
     let builder = RegClassBuilder::subclass_of("GPR8", gpr, 0, 8);
-    let gpr8 = isa.add_reg_class(builder);
+    let gpr8 = regs.add_class(builder);
 
     let builder = RegClassBuilder::subclass_of("ABCD", gpr8, 0, 4);
-    isa.add_reg_class(builder);
+    regs.add_class(builder);
 
     let builder = RegClassBuilder::subclass_of("FPR8", fpr, 0, 8);
-    isa.add_reg_class(builder);
+    regs.add_class(builder);
+
+    regs.finish()
 }
 
 pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
     let settings = define_settings(shared_settings);
-
-    let mut isa = TargetIsaBuilder::new("x86", settings);
-
-    define_registers(&mut isa);
-
-    isa.finish()
+    let regs = define_registers();
+    TargetIsa::new("x86", settings, regs)
 }

From a45b814de880137453ca78f1e55b103c99d9a149 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Feb 2019 15:01:10 +0100
Subject: [PATCH 2320/3084] Fixes #13: Enable conditional compilation of ISAs
 through features;

---
 cranelift/codegen/Cargo.toml               | 10 ++-
 cranelift/codegen/build.rs                 | 36 ++++++-----
 cranelift/codegen/meta/src/isa/mod.rs      |  1 +
 cranelift/codegen/meta/src/lib.rs          | 39 +-----------
 cranelift/codegen/src/isa/mod.rs           | 24 ++++----
 cranelift/codegen/src/regalloc/pressure.rs |  2 +-
 cranelift/codegen/src/regalloc/solver.rs   |  2 +-
 cranelift/filetests/src/runner.rs          |  2 +-
 cranelift/filetests/src/runone.rs          | 16 ++++-
 cranelift/reader/src/error.rs              | 14 +++++
 cranelift/reader/src/isaspec.rs            |  2 +-
 cranelift/reader/src/parser.rs             | 71 +++++++++++++++++-----
 12 files changed, 129 insertions(+), 90 deletions(-)

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 0448d8072e..8ffee131fb 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -29,16 +29,24 @@ log = { version = "0.4.6", default-features = false }
 cranelift-codegen-meta = { path = "meta", version = "0.28.0" }
 
 [features]
+default = ["std", "x86", "arm32", "arm64", "riscv"]
+
 # The "std" feature enables use of libstd. The "core" feature enables use
 # of some minimal std-like replacement libraries. At least one of these two
 # features need to be enabled.
-default = ["std"]
 std = ["cranelift-entity/std", "cranelift-bforest/std", "target-lexicon/std"]
 core = ["hashmap_core"]
+
 # This enables some additional functions useful for writing tests, but which
 # can significantly increase the size of the library.
 testing_hooks = []
 
+# ISA targets for which we should build.
+x86 = []
+arm32 = []
+arm64 = []
+riscv = []
+
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/cranelift/codegen/build.rs b/cranelift/codegen/build.rs
index 6aeaff6834..733a1c782f 100644
--- a/cranelift/codegen/build.rs
+++ b/cranelift/codegen/build.rs
@@ -11,10 +11,6 @@
 // TARGET
 //     Target triple provided by Cargo.
 //
-// CRANELIFT_TARGETS (Optional)
-//     A setting for conditional compilation of isa targets. Possible values can be "native" or
-//     known isa targets separated by ','.
-//
 // The build script expects to be run from the directory where this build.rs file lives. The
 // current directory is used to find the sources.
 
@@ -31,22 +27,24 @@ fn main() {
     let target_triple = env::var("TARGET").expect("The TARGET environment variable must be set");
 
     // Configure isa targets cfg.
-    let cranelift_targets = env::var("CRANELIFT_TARGETS").ok();
-    let cranelift_targets = cranelift_targets
-        .as_ref()
-        .map(|s| s.as_ref())
-        .filter(|s: &&str| s.len() > 0);
+    let isa_targets = meta::isa::Isa::all()
+        .into_iter()
+        .cloned()
+        .filter(|isa| {
+            let env_key = format!("CARGO_FEATURE_{}", isa.to_string().to_uppercase());
+            env::var(env_key).is_ok()
+        })
+        .collect::>();
 
-    let isas = match cranelift_targets {
-        Some("native") => meta::isa_from_arch(&target_triple.split('-').next().unwrap()),
-        Some(targets) => meta::isas_from_targets(targets.split(',').collect::>()),
-        None => meta::all_isas(),
-    }
-    .expect("Error when identifying CRANELIFT_TARGETS and TARGET");
-
-    for isa in &isas {
-        println!("cargo:rustc-cfg=build_{}", isa.to_string());
-    }
+    let isas = if isa_targets.is_empty() {
+        // Try to match native target.
+        let target_name = target_triple.split('-').next().unwrap();
+        let isa = meta::isa_from_arch(&target_name).expect("error when identifying target");
+        println!("cargo:rustc-cfg=feature=\"{}\"", isa);
+        vec![isa]
+    } else {
+        isa_targets
+    };
 
     let cur_dir = env::current_dir().expect("Can't access current working directory");
     let crate_dir = cur_dir.as_path();
diff --git a/cranelift/codegen/meta/src/isa/mod.rs b/cranelift/codegen/meta/src/isa/mod.rs
index c06326689b..9662435867 100644
--- a/cranelift/codegen/meta/src/isa/mod.rs
+++ b/cranelift/codegen/meta/src/isa/mod.rs
@@ -44,6 +44,7 @@ impl Isa {
 }
 
 impl fmt::Display for Isa {
+    // These names should be kept in sync with the crate features.
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
             Isa::Riscv => write!(f, "riscv"),
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index 610f2cd4a2..b617d31c16 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -13,43 +13,8 @@ mod shared;
 mod srcgen;
 mod unique_table;
 
-pub fn isa_from_arch(arch: &str) -> Result, String> {
-    isa::Isa::from_arch(arch)
-        .ok_or_else(|| format!("no supported isa found for arch `{}`", arch))
-        .and_then(|isa| Ok(vec![isa]))
-}
-
-pub fn isas_from_targets(targets: Vec<&str>) -> Result, String> {
-    type R<'a> = Vec<(&'a str, Option)>;
-
-    let (known, unknown): (R, R) = targets
-        .into_iter()
-        .map(|target| (target, isa::Isa::from_name(target)))
-        .partition(|(_, opt_isa)| opt_isa.is_some());
-
-    if !unknown.is_empty() {
-        let unknown_targets = unknown
-            .into_iter()
-            .map(|(target, _)| target)
-            .collect::>()
-            .join(", ");
-        return Err(format!("unknown isa targets: {}", unknown_targets));
-    }
-
-    let isas = if known.is_empty() {
-        isa::Isa::all().to_vec()
-    } else {
-        known
-            .into_iter()
-            .map(|(_, opt_isa)| opt_isa.unwrap())
-            .collect()
-    };
-
-    Ok(isas)
-}
-
-pub fn all_isas() -> Result, String> {
-    isas_from_targets(vec![])
+pub fn isa_from_arch(arch: &str) -> Result {
+    isa::Isa::from_arch(arch).ok_or_else(|| format!("no supported isa found for arch `{}`", arch))
 }
 
 /// Generates all the Rust source files used in Cranelift from the meta-language.
diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index ab77269ea5..b3e4d81421 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -68,16 +68,16 @@ use failure_derive::Fail;
 use std::boxed::Box;
 use target_lexicon::{Architecture, PointerWidth, Triple};
 
-#[cfg(build_riscv)]
+#[cfg(feature = "riscv")]
 mod riscv;
 
-#[cfg(build_x86)]
+#[cfg(feature = "x86")]
 mod x86;
 
-#[cfg(build_arm32)]
+#[cfg(feature = "arm32")]
 mod arm32;
 
-#[cfg(build_arm64)]
+#[cfg(feature = "arm64")]
 mod arm64;
 
 mod call_conv;
@@ -90,12 +90,12 @@ mod stack;
 /// Returns a builder that can create a corresponding `TargetIsa`
 /// or `Err(LookupError::Unsupported)` if not enabled.
 macro_rules! isa_builder {
-    ($module:ident, $name:ident) => {{
-        #[cfg($name)]
+    ($name:ident, $feature:tt) => {{
+        #[cfg(feature = $feature)]
         fn $name(triple: Triple) -> Result {
-            Ok($module::isa_builder(triple))
+            Ok($name::isa_builder(triple))
         };
-        #[cfg(not($name))]
+        #[cfg(not(feature = $feature))]
         fn $name(_triple: Triple) -> Result {
             Err(LookupError::Unsupported)
         }
@@ -107,9 +107,9 @@ macro_rules! isa_builder {
 /// Return a builder that can create a corresponding `TargetIsa`.
 pub fn lookup(triple: Triple) -> Result {
     match triple.architecture {
-        Architecture::Riscv32 | Architecture::Riscv64 => isa_builder!(riscv, build_riscv)(triple),
+        Architecture::Riscv32 | Architecture::Riscv64 => isa_builder!(riscv, "riscv")(triple),
         Architecture::I386 | Architecture::I586 | Architecture::I686 | Architecture::X86_64 => {
-            isa_builder!(x86, build_x86)(triple)
+            isa_builder!(x86, "x86")(triple)
         }
         Architecture::Thumbv6m
         | Architecture::Thumbv7em
@@ -118,8 +118,8 @@ pub fn lookup(triple: Triple) -> Result {
         | Architecture::Armv4t
         | Architecture::Armv5te
         | Architecture::Armv7
-        | Architecture::Armv7s => isa_builder!(arm32, build_arm32)(triple),
-        Architecture::Aarch64 => isa_builder!(arm64, build_arm64)(triple),
+        | Architecture::Armv7s => isa_builder!(arm32, "arm32")(triple),
+        Architecture::Aarch64 => isa_builder!(arm64, "arm64")(triple),
         _ => Err(LookupError::Unsupported),
     }
 }
diff --git a/cranelift/codegen/src/regalloc/pressure.rs b/cranelift/codegen/src/regalloc/pressure.rs
index cd1ef956a5..2db3ec3d03 100644
--- a/cranelift/codegen/src/regalloc/pressure.rs
+++ b/cranelift/codegen/src/regalloc/pressure.rs
@@ -270,7 +270,7 @@ impl fmt::Display for Pressure {
 }
 
 #[cfg(test)]
-#[cfg(build_arm32)]
+#[cfg(feature = "arm32")]
 mod tests {
     use super::Pressure;
     use crate::isa::{RegClass, TargetIsa};
diff --git a/cranelift/codegen/src/regalloc/solver.rs b/cranelift/codegen/src/regalloc/solver.rs
index be1c9a2d27..96ccdd2841 100644
--- a/cranelift/codegen/src/regalloc/solver.rs
+++ b/cranelift/codegen/src/regalloc/solver.rs
@@ -1127,7 +1127,7 @@ impl fmt::Display for Solver {
 }
 
 #[cfg(test)]
-#[cfg(build_arm32)]
+#[cfg(feature = "arm32")]
 mod tests {
     use super::{Move, Solver};
     use crate::entity::EntityRef;
diff --git a/cranelift/filetests/src/runner.rs b/cranelift/filetests/src/runner.rs
index 4c84551dd9..d11ffab79e 100644
--- a/cranelift/filetests/src/runner.rs
+++ b/cranelift/filetests/src/runner.rs
@@ -83,7 +83,7 @@ pub struct TestRunner {
 }
 
 impl TestRunner {
-    /// Create a new blank TrstRunner.
+    /// Create a new blank TestRunner.
     pub fn new(verbose: bool, report_times: bool) -> Self {
         Self {
             verbose,
diff --git a/cranelift/filetests/src/runone.rs b/cranelift/filetests/src/runone.rs
index a7b95dc7b5..323442d4b5 100644
--- a/cranelift/filetests/src/runone.rs
+++ b/cranelift/filetests/src/runone.rs
@@ -33,7 +33,21 @@ pub fn run(path: &Path, passes: Option<&[String]>, target: Option<&str>) -> Test
     info!("---\nFile: {}", path.to_string_lossy());
     let started = time::Instant::now();
     let buffer = read_to_string(path).map_err(|e| e.to_string())?;
-    let testfile = parse_test(&buffer, passes, target).map_err(|e| e.to_string())?;
+
+    let testfile = match parse_test(&buffer, passes, target) {
+        Ok(testfile) => testfile,
+        Err(e) => {
+            if e.is_warning {
+                println!(
+                    "skipping test {:?} (line {}): {}",
+                    path, e.location.line_number, e.message
+                );
+                return Ok(started.elapsed());
+            }
+            return Err(e.to_string());
+        }
+    };
+
     if testfile.functions.is_empty() {
         return Err("no functions found".to_string());
     }
diff --git a/cranelift/reader/src/error.rs b/cranelift/reader/src/error.rs
index e006ad5c3b..bed2352e5c 100644
--- a/cranelift/reader/src/error.rs
+++ b/cranelift/reader/src/error.rs
@@ -19,6 +19,8 @@ pub struct ParseError {
     pub location: Location,
     /// Error message.
     pub message: String,
+    /// Whether it's a warning or a plain error.
+    pub is_warning: bool,
 }
 
 impl fmt::Display for ParseError {
@@ -40,6 +42,7 @@ macro_rules! err {
         Err($crate::ParseError {
             location: $loc.clone(),
             message: $msg.to_string(),
+            is_warning: false,
         })
     };
 
@@ -47,6 +50,17 @@ macro_rules! err {
         Err($crate::ParseError {
             location: $loc.clone(),
             message: format!( $fmt, $( $arg ),+ ),
+            is_warning: false,
+        })
+    };
+}
+
+macro_rules! warn {
+    ( $loc:expr, $fmt:expr, $( $arg:expr ),+ ) => {
+        Err($crate::ParseError {
+            location: $loc.clone(),
+            message: format!($fmt, $( $arg ),+ ),
+            is_warning: true,
         })
     };
 }
diff --git a/cranelift/reader/src/isaspec.rs b/cranelift/reader/src/isaspec.rs
index 89dac510bb..6aefce354a 100644
--- a/cranelift/reader/src/isaspec.rs
+++ b/cranelift/reader/src/isaspec.rs
@@ -17,7 +17,7 @@ pub enum IsaSpec {
     /// which are reflected in the finished `Flags` object.
     None(Flags),
 
-    /// The parsed file does contains `isa` commands.
+    /// The parsed file does contain `isa` commands.
     /// Each `isa` command is used to configure a `TargetIsa` trait object.
     Some(Vec>),
 }
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 4f5acc05f4..42dfd55f01 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -530,6 +530,7 @@ impl<'a> Parser<'a> {
         ParseError {
             location: self.loc,
             message: message.to_string(),
+            is_warning: false,
         }
     }
 
@@ -778,10 +779,7 @@ impl<'a> Parser<'a> {
     ///
     /// Accept the target from the command line for pass command.
     ///
-    pub fn parse_cmdline_target(
-        &mut self,
-        target_pass: Option<&str>,
-    ) -> ParseResult {
+    fn parse_cmdline_target(&mut self, target_pass: Option<&str>) -> ParseResult {
         // Were there any `target` commands specified?
         let mut specified_target = false;
 
@@ -799,7 +797,7 @@ impl<'a> Parser<'a> {
                     return err!(loc, "support disabled target '{}'", targ);
                 }
                 Err(isa::LookupError::Unsupported) => {
-                    return err!(loc, "unsupported target '{}'", targ);
+                    return warn!(loc, "unsupported target '{}'", targ);
                 }
                 Ok(b) => b,
             };
@@ -821,7 +819,7 @@ impl<'a> Parser<'a> {
     ///
     /// Accept a mix of `target` and `set` command lines. The `set` commands are cumulative.
     ///
-    pub fn parse_target_specs(&mut self) -> ParseResult {
+    fn parse_target_specs(&mut self) -> ParseResult {
         // Were there any `target` commands?
         let mut seen_target = false;
         // Location of last `set` command since the last `target`.
@@ -859,7 +857,7 @@ impl<'a> Parser<'a> {
                             continue;
                         }
                         Err(isa::LookupError::Unsupported) => {
-                            return err!(loc, "unsupported target '{}'", target_name);
+                            return warn!(loc, "unsupported target '{}'", target_name);
                         }
                         Ok(b) => b,
                     };
@@ -874,6 +872,7 @@ impl<'a> Parser<'a> {
                 _ => break,
             }
         }
+
         if !seen_target {
             // No `target` commands, but we allow for `set` commands.
             Ok(isaspec::IsaSpec::None(settings::Flags::new(flag_builder)))
@@ -2588,9 +2587,14 @@ mod tests {
         assert_eq!(arg.value_type, types::I32);
         assert_eq!(arg.extension, ArgumentExtension::Sext);
         assert_eq!(arg.purpose, ArgumentPurpose::Normal);
-        let ParseError { location, message } = p.parse_abi_param(None).unwrap_err();
+        let ParseError {
+            location,
+            message,
+            is_warning,
+        } = p.parse_abi_param(None).unwrap_err();
         assert_eq!(location.line_number, 1);
         assert_eq!(message, "expected parameter type");
+        assert!(!is_warning);
     }
 
     #[test]
@@ -2736,7 +2740,11 @@ mod tests {
 
     #[test]
     fn duplicate_ebb() {
-        let ParseError { location, message } = Parser::new(
+        let ParseError {
+            location,
+            message,
+            is_warning,
+        } = Parser::new(
             "function %ebbs() system_v {
                 ebb0:
                 ebb0:
@@ -2747,11 +2755,16 @@ mod tests {
 
         assert_eq!(location.line_number, 3);
         assert_eq!(message, "duplicate entity: ebb0");
+        assert!(!is_warning);
     }
 
     #[test]
     fn duplicate_jt() {
-        let ParseError { location, message } = Parser::new(
+        let ParseError {
+            location,
+            message,
+            is_warning,
+        } = Parser::new(
             "function %ebbs() system_v {
                 jt0 = jump_table []
                 jt0 = jump_table []",
@@ -2761,11 +2774,16 @@ mod tests {
 
         assert_eq!(location.line_number, 3);
         assert_eq!(message, "duplicate entity: jt0");
+        assert!(!is_warning);
     }
 
     #[test]
     fn duplicate_ss() {
-        let ParseError { location, message } = Parser::new(
+        let ParseError {
+            location,
+            message,
+            is_warning,
+        } = Parser::new(
             "function %ebbs() system_v {
                 ss0 = explicit_slot 8
                 ss0 = explicit_slot 8",
@@ -2775,11 +2793,16 @@ mod tests {
 
         assert_eq!(location.line_number, 3);
         assert_eq!(message, "duplicate entity: ss0");
+        assert!(!is_warning);
     }
 
     #[test]
     fn duplicate_gv() {
-        let ParseError { location, message } = Parser::new(
+        let ParseError {
+            location,
+            message,
+            is_warning,
+        } = Parser::new(
             "function %ebbs() system_v {
                 gv0 = vmctx
                 gv0 = vmctx",
@@ -2789,11 +2812,16 @@ mod tests {
 
         assert_eq!(location.line_number, 3);
         assert_eq!(message, "duplicate entity: gv0");
+        assert!(!is_warning);
     }
 
     #[test]
     fn duplicate_heap() {
-        let ParseError { location, message } = Parser::new(
+        let ParseError {
+            location,
+            message,
+            is_warning,
+        } = Parser::new(
             "function %ebbs() system_v {
                 heap0 = static gv0, min 0x1000, bound 0x10_0000, offset_guard 0x1000
                 heap0 = static gv0, min 0x1000, bound 0x10_0000, offset_guard 0x1000",
@@ -2803,11 +2831,16 @@ mod tests {
 
         assert_eq!(location.line_number, 3);
         assert_eq!(message, "duplicate entity: heap0");
+        assert!(!is_warning);
     }
 
     #[test]
     fn duplicate_sig() {
-        let ParseError { location, message } = Parser::new(
+        let ParseError {
+            location,
+            message,
+            is_warning,
+        } = Parser::new(
             "function %ebbs() system_v {
                 sig0 = ()
                 sig0 = ()",
@@ -2817,11 +2850,16 @@ mod tests {
 
         assert_eq!(location.line_number, 3);
         assert_eq!(message, "duplicate entity: sig0");
+        assert!(!is_warning);
     }
 
     #[test]
     fn duplicate_fn() {
-        let ParseError { location, message } = Parser::new(
+        let ParseError {
+            location,
+            message,
+            is_warning,
+        } = Parser::new(
             "function %ebbs() system_v {
                 sig0 = ()
                 fn0 = %foo sig0
@@ -2832,6 +2870,7 @@ mod tests {
 
         assert_eq!(location.line_number, 4);
         assert_eq!(message, "duplicate entity: fn0");
+        assert!(!is_warning);
     }
 
     #[test]
@@ -2905,7 +2944,7 @@ mod tests {
     }
 
     #[test]
-    #[cfg(build_riscv)]
+    #[cfg(feature = "riscv")]
     fn isa_spec() {
         assert!(parse_test(
             "target

From 615499bae88c89b7b108bad0afc4b85b0ce0afb4 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Fri, 18 Jan 2019 17:26:28 +0100
Subject: [PATCH 2321/3084] Remove define_function_peek_compiled

---
 cranelift/module/src/module.rs | 72 +++++++++++++++++-----------------
 1 file changed, 36 insertions(+), 36 deletions(-)

diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index b043ee4754..3b0f4673e0 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -505,49 +505,44 @@ where
     }
 
     /// Define a function, producing the function body from the given `Context`.
-    pub fn define_function(&mut self, func: FuncId, ctx: &mut Context) -> ModuleResult<()> {
-        self.define_function_peek_compiled(func, ctx, |_, _, _| ())
-    }
-
-    /// Define a function, allowing to peek at the compiled function and producing the
-    /// function body from the given `Context`.
-    pub fn define_function_peek_compiled(
+    ///
+    /// Returns the size of the function's code.
+    ///
+    /// Note: After calling this function the given `Context` will contain the compiled function.
+    pub fn define_function(
         &mut self,
         func: FuncId,
         ctx: &mut Context,
-        peek_compiled: impl FnOnce(u32, &Context, &isa::TargetIsa) -> T,
-    ) -> ModuleResult {
-        let code_size;
-        let compiled = {
-            code_size = ctx.compile(self.backend.isa()).map_err(|e| {
-                info!(
-                    "defining function {}: {}",
-                    func,
-                    ctx.func.display(self.backend.isa())
-                );
-                ModuleError::Compilation(e)
-            })?;
+    ) -> ModuleResult {
+        let code_size = ctx.compile(self.backend.isa()).map_err(|e| {
+            info!(
+                "defining function {}: {}",
+                func,
+                ctx.func.display(self.backend.isa())
+            );
+            ModuleError::Compilation(e)
+        })?;
 
-            let info = &self.contents.functions[func];
-            if info.compiled.is_some() {
-                return Err(ModuleError::DuplicateDefinition(info.decl.name.clone()));
-            }
-            if !info.decl.linkage.is_definable() {
-                return Err(ModuleError::InvalidImportDefinition(info.decl.name.clone()));
-            }
+        let info = &self.contents.functions[func];
+        if info.compiled.is_some() {
+            return Err(ModuleError::DuplicateDefinition(info.decl.name.clone()));
+        }
+        if !info.decl.linkage.is_definable() {
+            return Err(ModuleError::InvalidImportDefinition(info.decl.name.clone()));
+        }
+
+        let compiled = Some(self.backend.define_function(
+            &info.decl.name,
+            ctx,
+            &ModuleNamespace:: {
+                contents: &self.contents,
+            },
+            code_size,
+        )?);
 
-            Some(self.backend.define_function(
-                &info.decl.name,
-                ctx,
-                &ModuleNamespace:: {
-                    contents: &self.contents,
-                },
-                code_size,
-            )?)
-        };
         self.contents.functions[func].compiled = compiled;
         self.functions_to_finalize.push(func);
-        Ok(peek_compiled(code_size, &ctx, self.backend.isa()))
+        Ok(code_size)
     }
 
     /// Define a function, producing the data contents from the given `DataContext`.
@@ -679,6 +674,11 @@ where
         )
     }
 
+    /// Return the target isa
+    pub fn isa(&self) -> &isa::TargetIsa {
+        self.backend.isa()
+    }
+
     /// Consume the module and return the resulting `Product`. Some `Backend`
     /// implementations may provide additional functionality available after
     /// a `Module` is complete.

From 90028a6d0dbfa844b69eafbb37c15044ae553429 Mon Sep 17 00:00:00 2001
From: YISH 
Date: Wed, 20 Feb 2019 03:44:39 +0800
Subject: [PATCH 2322/3084] impl Ieee64::from(f64) and Ieee32::from(f32)

---
 cranelift/codegen/src/ir/immediates.rs | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index 21d8376e37..36cc470aef 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -665,6 +665,12 @@ impl FromStr for Ieee32 {
     }
 }
 
+impl From for Ieee32 {
+    fn from(x: f32) -> Self {
+        Ieee32::with_float(x)
+    }
+}
+
 impl Ieee64 {
     /// Create a new `Ieee64` containing the bits of `x`.
     pub fn with_bits(x: u64) -> Self {
@@ -726,6 +732,12 @@ impl FromStr for Ieee64 {
     }
 }
 
+impl From for Ieee64 {
+    fn from(x: f64) -> Self {
+        Ieee64::with_float(x)
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;

From c8e09cb37f97ec19ad210f4bd293c2b9a9a4e2e9 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 8 Feb 2019 18:28:58 +0100
Subject: [PATCH 2323/3084] [meta] Use AsRef to allow passing String or
 &str to Formatter functions;

---
 cranelift/codegen/meta/src/gen_registers.rs |  47 +++++----
 cranelift/codegen/meta/src/gen_settings.rs  | 100 +++++++++-----------
 cranelift/codegen/meta/src/gen_types.rs     |   4 +-
 cranelift/codegen/meta/src/srcgen.rs        |  29 +++---
 4 files changed, 86 insertions(+), 94 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_registers.rs b/cranelift/codegen/meta/src/gen_registers.rs
index 8488dab228..11d80ff21e 100644
--- a/cranelift/codegen/meta/src/gen_registers.rs
+++ b/cranelift/codegen/meta/src/gen_registers.rs
@@ -12,14 +12,14 @@ fn gen_regbank(fmt: &mut Formatter, reg_bank: &RegBank) {
     };
     fmt.line("RegBank {");
     fmt.indent(|fmt| {
-        fmt.line(&format!(r#"name: "{}","#, reg_bank.name));
-        fmt.line(&format!("first_unit: {},", reg_bank.first_unit));
-        fmt.line(&format!("units: {},", reg_bank.units));
-        fmt.line(&format!("names: &[{}],", names));
-        fmt.line(&format!(r#"prefix: "{}","#, reg_bank.prefix));
-        fmt.line(&format!("first_toprc: {},", reg_bank.toprcs[0].index()));
-        fmt.line(&format!("num_toprcs: {},", reg_bank.toprcs.len()));
-        fmt.line(&format!(
+        fmt.line(format!(r#"name: "{}","#, reg_bank.name));
+        fmt.line(format!("first_unit: {},", reg_bank.first_unit));
+        fmt.line(format!("units: {},", reg_bank.units));
+        fmt.line(format!("names: &[{}],", names));
+        fmt.line(format!(r#"prefix: "{}","#, reg_bank.prefix));
+        fmt.line(format!("first_toprc: {},", reg_bank.toprcs[0].index()));
+        fmt.line(format!("num_toprcs: {},", reg_bank.toprcs.len()));
+        fmt.line(format!(
             "pressure_tracking: {},",
             if reg_bank.pressure_tracking {
                 "true"
@@ -41,27 +41,24 @@ fn gen_regclass(isa: &TargetIsa, reg_class: &RegClass, fmt: &mut Formatter) {
         .collect();
     let mask = mask.join(", ");
 
-    fmt.line(&format!(
+    fmt.line(format!(
         "pub static {}_DATA: RegClassData = RegClassData {{",
         reg_class.name
     ));
     fmt.indent(|fmt| {
-        fmt.line(&format!(r#"name: "{}","#, reg_class.name));
-        fmt.line(&format!("index: {},", reg_class.index.index()));
-        fmt.line(&format!("width: {},", reg_class.width));
-        fmt.line(&format!("bank: {},", reg_class.bank.index()));
-        fmt.line(&format!("toprc: {},", reg_class.toprc.index()));
-        fmt.line(&format!(
-            "first: {},",
-            reg_bank.first_unit + reg_class.start
-        ));
-        fmt.line(&format!("subclasses: {:#x},", reg_class.subclass_mask()));
-        fmt.line(&format!("mask: [{}],", mask));
+        fmt.line(format!(r#"name: "{}","#, reg_class.name));
+        fmt.line(format!("index: {},", reg_class.index.index()));
+        fmt.line(format!("width: {},", reg_class.width));
+        fmt.line(format!("bank: {},", reg_class.bank.index()));
+        fmt.line(format!("toprc: {},", reg_class.toprc.index()));
+        fmt.line(format!("first: {},", reg_bank.first_unit + reg_class.start));
+        fmt.line(format!("subclasses: {:#x},", reg_class.subclass_mask()));
+        fmt.line(format!("mask: [{}],", mask));
         fmt.line("info: &INFO,");
     });
     fmt.line("};");
     fmt.line("#[allow(dead_code)]");
-    fmt.line(&format!(
+    fmt.line(format!(
         "pub static {}: RegClass = &{}_DATA;",
         reg_class.name, reg_class.name
     ));
@@ -71,10 +68,10 @@ fn gen_regbank_units(reg_bank: &RegBank, fmt: &mut Formatter) {
     for unit in 0..reg_bank.units {
         let v = unit + reg_bank.first_unit;
         if (unit as usize) < reg_bank.names.len() {
-            fmt.line(&format!("{} = {},", reg_bank.names[unit as usize], v));
+            fmt.line(format!("{} = {},", reg_bank.names[unit as usize], v));
             continue;
         }
-        fmt.line(&format!("{}{} = {},", reg_bank.prefix, unit, v));
+        fmt.line(format!("{}{} = {},", reg_bank.prefix, unit, v));
     }
 }
 
@@ -95,7 +92,7 @@ fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) {
         fmt.line("classes: &[");
         fmt.indent(|fmt| {
             for reg_class in isa.regs.classes.values() {
-                fmt.line(&format!("&{}_DATA,", reg_class.name));
+                fmt.line(format!("&{}_DATA,", reg_class.name));
             }
         });
         fmt.line("],");
@@ -133,6 +130,6 @@ fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) {
 pub fn generate(isa: &TargetIsa, base_filename: &str, out_dir: &str) -> Result<(), error::Error> {
     let mut fmt = Formatter::new();
     gen_isa(&isa, &mut fmt);
-    fmt.update_file(&format!("{}-{}.rs", base_filename, isa.name), out_dir)?;
+    fmt.update_file(format!("{}-{}.rs", base_filename, isa.name), out_dir)?;
     Ok(())
 }
diff --git a/cranelift/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
index 99b9f52689..183c8643a2 100644
--- a/cranelift/codegen/meta/src/gen_settings.rs
+++ b/cranelift/codegen/meta/src/gen_settings.rs
@@ -23,34 +23,31 @@ fn gen_constructor(group: &SettingGroup, parent: ParentGroup, fmt: &mut Formatte
     };
     fmt.line("impl Flags {");
     fmt.indent(|fmt| {
-        fmt.doc_comment(&format!("Create flags {} settings group.", group.name));
+        fmt.doc_comment(format!("Create flags {} settings group.", group.name));
         fmt.line("#[allow(unused_variables)]");
-        fmt.line(&format!("pub fn new({}) -> Self {{", args));
+        fmt.line(format!("pub fn new({}) -> Self {{", args));
         fmt.indent(|fmt| {
-            fmt.line(&format!(
-                "let bvec = builder.state_for(\"{}\");",
-                group.name
-            ));
-            fmt.line(&format!(
+            fmt.line(format!("let bvec = builder.state_for(\"{}\");", group.name));
+            fmt.line(format!(
                 "let mut {} = Self {{ bytes: [0; {}] }};",
                 group.name,
                 group.byte_size()
             ));
-            fmt.line(&format!(
+            fmt.line(format!(
                 "debug_assert_eq!(bvec.len(), {});",
                 group.settings_size
             ));
-            fmt.line(&format!(
+            fmt.line(format!(
                 "{}.bytes[0..{}].copy_from_slice(&bvec);",
                 group.name, group.settings_size
             ));
 
             // Now compute the predicates.
             for p in &group.predicates {
-                fmt.comment(&format!("Precompute #{}.", p.number));
-                fmt.line(&format!("if {} {{", p.render(group)));
+                fmt.comment(format!("Precompute #{}.", p.number));
+                fmt.line(format!("if {} {{", p.render(group)));
                 fmt.indent(|fmt| {
-                    fmt.line(&format!(
+                    fmt.line(format!(
                         "{}.bytes[{}] |= 1 << {};",
                         group.name,
                         group.bool_start_byte_offset + p.number / 8,
@@ -69,14 +66,14 @@ fn gen_constructor(group: &SettingGroup, parent: ParentGroup, fmt: &mut Formatte
 
 /// Emit Display and FromStr implementations for enum settings.
 fn gen_to_and_from_str(name: &str, values: &[&'static str], fmt: &mut Formatter) {
-    fmt.line(&format!("impl fmt::Display for {} {{", name));
+    fmt.line(format!("impl fmt::Display for {} {{", name));
     fmt.indent(|fmt| {
         fmt.line("fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {");
         fmt.indent(|fmt| {
             fmt.line("f.write_str(match *self {");
             fmt.indent(|fmt| {
                 for v in values.iter() {
-                    fmt.line(&format!("{}::{} => \"{}\",", name, camel_case(v), v));
+                    fmt.line(format!("{}::{} => \"{}\",", name, camel_case(v), v));
                 }
             });
             fmt.line("})");
@@ -85,7 +82,7 @@ fn gen_to_and_from_str(name: &str, values: &[&'static str], fmt: &mut Formatter)
     });
     fmt.line("}");
 
-    fmt.line(&format!("impl str::FromStr for {} {{", name));
+    fmt.line(format!("impl str::FromStr for {} {{", name));
     fmt.indent(|fmt| {
         fmt.line("type Err = ();");
         fmt.line("fn from_str(s: &str) -> Result {");
@@ -93,7 +90,7 @@ fn gen_to_and_from_str(name: &str, values: &[&'static str], fmt: &mut Formatter)
             fmt.line("match s {");
             fmt.indent(|fmt| {
                 for v in values.iter() {
-                    fmt.line(&format!("\"{}\" => Ok({}::{}),", v, name, camel_case(v)));
+                    fmt.line(format!("\"{}\" => Ok({}::{}),", v, name, camel_case(v)));
                 }
                 fmt.line("_ => Err(()),");
             });
@@ -113,13 +110,13 @@ fn gen_enum_types(group: &SettingGroup, fmt: &mut Formatter) {
         };
         let name = camel_case(setting.name);
 
-        fmt.doc_comment(&format!("Values for `{}.{}`.", group.name, setting.name));
+        fmt.doc_comment(format!("Values for `{}.{}`.", group.name, setting.name));
         fmt.line("#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]");
-        fmt.line(&format!("pub enum {} {{", name));
+        fmt.line(format!("pub enum {} {{", name));
         fmt.indent(|fmt| {
             for v in values.iter() {
-                fmt.doc_comment(&format!("`{}`.", v));
-                fmt.line(&format!("{},", camel_case(v)));
+                fmt.doc_comment(format!("`{}`.", v));
+                fmt.line(format!("{},", camel_case(v)));
             }
         });
         fmt.line("}");
@@ -135,15 +132,15 @@ fn gen_getter(setting: &Setting, fmt: &mut Formatter) {
         SpecificSetting::Bool(BoolSetting {
             predicate_number, ..
         }) => {
-            fmt.line(&format!("pub fn {}(&self) -> bool {{", setting.name));
+            fmt.line(format!("pub fn {}(&self) -> bool {{", setting.name));
             fmt.indent(|fmt| {
-                fmt.line(&format!("self.numbered_predicate({})", predicate_number));
+                fmt.line(format!("self.numbered_predicate({})", predicate_number));
             });
             fmt.line("}");
         }
         SpecificSetting::Enum(ref values) => {
             let ty = camel_case(setting.name);
-            fmt.line(&format!("pub fn {}(&self) -> {} {{", setting.name, ty));
+            fmt.line(format!("pub fn {}(&self) -> {} {{", setting.name, ty));
             fmt.indent(|fmt| {
                 let mut m = Match::new(format!("self.bytes[{}]", setting.byte_offset));
                 for (i, v) in values.iter().enumerate() {
@@ -159,9 +156,9 @@ fn gen_getter(setting: &Setting, fmt: &mut Formatter) {
             fmt.line("}");
         }
         SpecificSetting::Num(_) => {
-            fmt.line(&format!("pub fn {}(&self) -> u8 {{", setting.name));
+            fmt.line(format!("pub fn {}(&self) -> u8 {{", setting.name));
             fmt.indent(|fmt| {
-                fmt.line(&format!("self.bytes[{}]", setting.byte_offset));
+                fmt.line(format!("self.bytes[{}]", setting.byte_offset));
             });
             fmt.line("}");
         }
@@ -169,13 +166,10 @@ fn gen_getter(setting: &Setting, fmt: &mut Formatter) {
 }
 
 fn gen_pred_getter(predicate: &Predicate, group: &SettingGroup, fmt: &mut Formatter) {
-    fmt.doc_comment(&format!(
-        "Computed predicate `{}`.",
-        predicate.render(group)
-    ));
-    fmt.line(&format!("pub fn {}(&self) -> bool {{", predicate.name));
+    fmt.doc_comment(format!("Computed predicate `{}`.", predicate.render(group)));
+    fmt.line(format!("pub fn {}(&self) -> bool {{", predicate.name));
     fmt.indent(|fmt| {
-        fmt.line(&format!("self.numbered_predicate({})", predicate.number));
+        fmt.line(format!("self.numbered_predicate({})", predicate.number));
     });
     fmt.line("}");
 }
@@ -189,7 +183,7 @@ fn gen_getters(group: &SettingGroup, fmt: &mut Formatter) {
         fmt.doc_comment("Get a view of the boolean predicates.");
         fmt.line("pub fn predicate_view(&self) -> ::settings::PredicateView {");
         fmt.indent(|fmt| {
-            fmt.line(&format!(
+            fmt.line(format!(
                 "::settings::PredicateView::new(&self.bytes[{}..])",
                 group.bool_start_byte_offset
             ));
@@ -200,7 +194,7 @@ fn gen_getters(group: &SettingGroup, fmt: &mut Formatter) {
             fmt.doc_comment("Dynamic numbered predicate getter.");
             fmt.line("fn numbered_predicate(&self, p: usize) -> bool {");
             fmt.indent(|fmt| {
-                fmt.line(&format!(
+                fmt.line(format!(
                     "self.bytes[{} + p / 8] & (1 << (p % 8)) != 0",
                     group.bool_start_byte_offset
                 ));
@@ -240,7 +234,7 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
     let mut descriptor_index_map: HashMap = HashMap::new();
 
     // Generate descriptors.
-    fmt.line(&format!(
+    fmt.line(format!(
         "static DESCRIPTORS: [detail::Descriptor; {}] = [",
         group.settings.len() + group.presets.len()
     ));
@@ -248,18 +242,18 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
         for (idx, setting) in group.settings.iter().enumerate() {
             fmt.line("detail::Descriptor {");
             fmt.indent(|fmt| {
-                fmt.line(&format!("name: \"{}\",", setting.name));
-                fmt.line(&format!("offset: {},", setting.byte_offset));
+                fmt.line(format!("name: \"{}\",", setting.name));
+                fmt.line(format!("offset: {},", setting.byte_offset));
                 match setting.specific {
                     SpecificSetting::Bool(BoolSetting { bit_offset, .. }) => {
-                        fmt.line(&format!(
+                        fmt.line(format!(
                             "detail: detail::Detail::Bool {{ bit: {} }},",
                             bit_offset
                         ));
                     }
                     SpecificSetting::Enum(ref values) => {
                         let offset = enum_table.add(values);
-                        fmt.line(&format!(
+                        fmt.line(format!(
                             "detail: detail::Detail::Enum {{ last: {}, enumerators: {} }},",
                             values.len() - 1,
                             offset
@@ -278,8 +272,8 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
         for (idx, preset) in group.presets.iter().enumerate() {
             fmt.line("detail::Descriptor {");
             fmt.indent(|fmt| {
-                fmt.line(&format!("name: \"{}\",", preset.name));
-                fmt.line(&format!("offset: {},", (idx as u8) * group.settings_size));
+                fmt.line(format!("name: \"{}\",", preset.name));
+                fmt.line(format!("offset: {},", (idx as u8) * group.settings_size));
                 fmt.line("detail: detail::Detail::Preset,");
             });
             fmt.line("},");
@@ -290,13 +284,13 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
     fmt.line("];");
 
     // Generate enumerators.
-    fmt.line(&format!(
+    fmt.line(format!(
         "static ENUMERATORS: [&str; {}] = [",
         enum_table.len()
     ));
     fmt.indent(|fmt| {
         for enum_val in enum_table.iter() {
-            fmt.line(&format!("\"{}\",", enum_val));
+            fmt.line(format!("\"{}\",", enum_val));
         }
     });
     fmt.line("];");
@@ -318,14 +312,14 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
             .collect::>(),
     );
     let hash_table = generate_table(&hash_entries, |entry| simple_hash(entry.name()));
-    fmt.line(&format!(
+    fmt.line(format!(
         "static HASH_TABLE: [u16; {}] = [",
         hash_table.len()
     ));
     fmt.indent(|fmt| {
         for h in &hash_table {
             match *h {
-                Some(setting_or_preset) => fmt.line(&format!(
+                Some(setting_or_preset) => fmt.line(format!(
                     "{},",
                     &descriptor_index_map
                         .get(setting_or_preset)
@@ -339,7 +333,7 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
     fmt.line("];");
 
     // Generate presets.
-    fmt.line(&format!(
+    fmt.line(format!(
         "static PRESETS: [(u8, u8); {}] = [",
         group.presets.len()
     ));
@@ -347,7 +341,7 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
         for preset in &group.presets {
             fmt.comment(preset.name);
             for (mask, value) in preset.layout(&group) {
-                fmt.line(&format!("(0b{:08b}, 0b{:08b}),", mask, value));
+                fmt.line(format!("(0b{:08b}, 0b{:08b}),", mask, value));
             }
         }
     });
@@ -368,16 +362,16 @@ fn gen_template(group: &SettingGroup, fmt: &mut Formatter) {
 
     fmt.line("static TEMPLATE: detail::Template = detail::Template {");
     fmt.indent(|fmt| {
-        fmt.line(&format!("name: \"{}\",", group.name));
+        fmt.line(format!("name: \"{}\",", group.name));
         fmt.line("descriptors: &DESCRIPTORS,");
         fmt.line("enumerators: &ENUMERATORS,");
         fmt.line("hash_table: &HASH_TABLE,");
-        fmt.line(&format!("defaults: &[{}],", default_bytes_str));
+        fmt.line(format!("defaults: &[{}],", default_bytes_str));
         fmt.line("presets: &PRESETS,");
     });
     fmt.line("};");
 
-    fmt.doc_comment(&format!(
+    fmt.doc_comment(format!(
         "Create a `settings::Builder` for the {} settings group.",
         group.name
     ));
@@ -393,7 +387,7 @@ fn gen_display(group: &SettingGroup, fmt: &mut Formatter) {
     fmt.indent(|fmt| {
         fmt.line("fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {");
         fmt.indent(|fmt| {
-            fmt.line(&format!("writeln!(f, \"[{}]\")?;", group.name));
+            fmt.line(format!("writeln!(f, \"[{}]\")?;", group.name));
             fmt.line("for d in &DESCRIPTORS {");
             fmt.indent(|fmt| {
                 fmt.line("if !d.detail.is_preset() {");
@@ -417,10 +411,10 @@ fn gen_display(group: &SettingGroup, fmt: &mut Formatter) {
 fn gen_group(group: &SettingGroup, parent: ParentGroup, fmt: &mut Formatter) {
     // Generate struct.
     fmt.line("#[derive(Clone)]");
-    fmt.doc_comment(&format!("Flags group `{}`.", group.name));
+    fmt.doc_comment(format!("Flags group `{}`.", group.name));
     fmt.line("pub struct Flags {");
     fmt.indent(|fmt| {
-        fmt.line(&format!("bytes: [u8; {}],", group.byte_size()));
+        fmt.line(format!("bytes: [u8; {}],", group.byte_size()));
     });
     fmt.line("}");
 
@@ -443,6 +437,6 @@ pub fn generate_common(filename: &str, out_dir: &str) -> Result Result<(), error::Error> {
     let mut fmt = Formatter::new();
     gen_group(&isa.settings, ParentGroup::Shared, &mut fmt);
-    fmt.update_file(&format!("{}-{}.rs", prefix, isa.name), out_dir)?;
+    fmt.update_file(format!("{}-{}.rs", prefix, isa.name), out_dir)?;
     Ok(())
 }
diff --git a/cranelift/codegen/meta/src/gen_types.rs b/cranelift/codegen/meta/src/gen_types.rs
index 6332525306..5b48d17a27 100644
--- a/cranelift/codegen/meta/src/gen_types.rs
+++ b/cranelift/codegen/meta/src/gen_types.rs
@@ -21,10 +21,8 @@ fn emit_type(ty: &cdsl_types::ValueType, fmt: &mut srcgen::Formatter) -> Result<
         ))
     })?;
 
-    let definition = format!("pub const {}: Type = Type({:#x});\n", name, number);
-
     fmt.doc_comment(&ty.doc());
-    fmt.line(&definition);
+    fmt.line(format!("pub const {}: Type = Type({:#x});\n", name, number));
 
     Ok(())
 }
diff --git a/cranelift/codegen/meta/src/srcgen.rs b/cranelift/codegen/meta/src/srcgen.rs
index 164c477997..4e86e0b8c7 100644
--- a/cranelift/codegen/meta/src/srcgen.rs
+++ b/cranelift/codegen/meta/src/srcgen.rs
@@ -65,8 +65,8 @@ impl Formatter {
     }
 
     /// Add an indented line.
-    pub fn line(&mut self, contents: &str) {
-        let indented_line = format!("{}{}\n", self.get_indent(), contents);
+    pub fn line(&mut self, contents: impl AsRef) {
+        let indented_line = format!("{}{}\n", self.get_indent(), contents.as_ref());
         self.lines.push(indented_line);
     }
 
@@ -77,11 +77,15 @@ impl Formatter {
     }
 
     /// Write `self.lines` to a file.
-    pub fn update_file(&self, filename: &str, directory: &str) -> Result<(), error::Error> {
+    pub fn update_file(
+        &self,
+        filename: impl AsRef,
+        directory: &str,
+    ) -> Result<(), error::Error> {
         #[cfg(target_family = "windows")]
-        let path_str = format!("{}\\{}", directory, filename);
+        let path_str = format!("{}\\{}", directory, filename.as_ref());
         #[cfg(not(target_family = "windows"))]
-        let path_str = format!("{}/{}", directory, filename);
+        let path_str = format!("{}/{}", directory, filename.as_ref());
 
         let path = path::Path::new(&path_str);
         let mut f = fs::File::create(path)?;
@@ -99,14 +103,13 @@ impl Formatter {
     }
 
     /// Add a comment line.
-    pub fn comment(&mut self, s: &str) {
-        let commented_line = format!("// {}", s);
-        self.line(&commented_line);
+    pub fn comment(&mut self, s: impl AsRef) {
+        self.line(format!("// {}", s.as_ref()));
     }
 
     /// Add a (multi-line) documentation comment.
-    pub fn doc_comment(&mut self, contents: &str) {
-        parse_multiline(contents)
+    pub fn doc_comment(&mut self, contents: impl AsRef) {
+        parse_multiline(contents.as_ref())
             .iter()
             .map(|l| {
                 if l.len() == 0 {
@@ -120,7 +123,7 @@ impl Formatter {
 
     /// Add a match expression.
     pub fn add_match(&mut self, m: Match) {
-        self.line(&format!("match {} {{", m.expr));
+        self.line(format!("match {} {{", m.expr));
         self.indent(|fmt| {
             for (&(ref fields, ref body), ref names) in m.arms.iter() {
                 // name { fields } | name { fields } => { body }
@@ -135,7 +138,7 @@ impl Formatter {
                     })
                     .collect();
                 let lhs = conditions.join(" | ");
-                fmt.line(&format!("{} => {{", lhs));
+                fmt.line(format!("{} => {{", lhs));
                 fmt.indent(|fmt| {
                     fmt.line(body);
                 });
@@ -337,7 +340,7 @@ match x {
     #[test]
     fn fmt_can_add_type_to_lines() {
         let mut fmt = Formatter::new();
-        fmt.line(&format!("pub const {}: Type = Type({:#x});", "example", 0,));
+        fmt.line(format!("pub const {}: Type = Type({:#x});", "example", 0,));
         let expected_lines = vec!["pub const example: Type = Type(0x0);\n"];
         assert_eq!(fmt.lines, expected_lines);
     }

From 603d80615f2e52c4a62c597418c67b879a52cfcf Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 13 Feb 2019 17:27:33 +0100
Subject: [PATCH 2324/3084] [meta] Introduce the fmtln! macro to ease writing
 formatted strings;

---
 cranelift/codegen/meta/src/gen_registers.rs |  95 +++----
 cranelift/codegen/meta/src/gen_settings.rs  | 265 +++++++++++---------
 cranelift/codegen/meta/src/gen_types.rs     |   2 +-
 cranelift/codegen/meta/src/lib.rs           |   2 +-
 cranelift/codegen/meta/src/srcgen.rs        |  20 +-
 5 files changed, 211 insertions(+), 173 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_registers.rs b/cranelift/codegen/meta/src/gen_registers.rs
index 11d80ff21e..3a75c7147a 100644
--- a/cranelift/codegen/meta/src/gen_registers.rs
+++ b/cranelift/codegen/meta/src/gen_registers.rs
@@ -10,25 +10,26 @@ fn gen_regbank(fmt: &mut Formatter, reg_bank: &RegBank) {
     } else {
         "".to_string()
     };
-    fmt.line("RegBank {");
+    fmtln!(fmt, "RegBank {");
     fmt.indent(|fmt| {
-        fmt.line(format!(r#"name: "{}","#, reg_bank.name));
-        fmt.line(format!("first_unit: {},", reg_bank.first_unit));
-        fmt.line(format!("units: {},", reg_bank.units));
-        fmt.line(format!("names: &[{}],", names));
-        fmt.line(format!(r#"prefix: "{}","#, reg_bank.prefix));
-        fmt.line(format!("first_toprc: {},", reg_bank.toprcs[0].index()));
-        fmt.line(format!("num_toprcs: {},", reg_bank.toprcs.len()));
-        fmt.line(format!(
+        fmtln!(fmt, r#"name: "{}","#, reg_bank.name);
+        fmtln!(fmt, "first_unit: {},", reg_bank.first_unit);
+        fmtln!(fmt, "units: {},", reg_bank.units);
+        fmtln!(fmt, "names: &[{}],", names);
+        fmtln!(fmt, r#"prefix: "{}","#, reg_bank.prefix);
+        fmtln!(fmt, "first_toprc: {},", reg_bank.toprcs[0].index());
+        fmtln!(fmt, "num_toprcs: {},", reg_bank.toprcs.len());
+        fmtln!(
+            fmt,
             "pressure_tracking: {},",
             if reg_bank.pressure_tracking {
                 "true"
             } else {
                 "false"
             }
-        ));
+        );
     });
-    fmt.line("},");
+    fmtln!(fmt, "},");
 }
 
 fn gen_regclass(isa: &TargetIsa, reg_class: &RegClass, fmt: &mut Formatter) {
@@ -41,63 +42,67 @@ fn gen_regclass(isa: &TargetIsa, reg_class: &RegClass, fmt: &mut Formatter) {
         .collect();
     let mask = mask.join(", ");
 
-    fmt.line(format!(
+    fmtln!(
+        fmt,
         "pub static {}_DATA: RegClassData = RegClassData {{",
         reg_class.name
-    ));
+    );
     fmt.indent(|fmt| {
-        fmt.line(format!(r#"name: "{}","#, reg_class.name));
-        fmt.line(format!("index: {},", reg_class.index.index()));
-        fmt.line(format!("width: {},", reg_class.width));
-        fmt.line(format!("bank: {},", reg_class.bank.index()));
-        fmt.line(format!("toprc: {},", reg_class.toprc.index()));
-        fmt.line(format!("first: {},", reg_bank.first_unit + reg_class.start));
-        fmt.line(format!("subclasses: {:#x},", reg_class.subclass_mask()));
-        fmt.line(format!("mask: [{}],", mask));
-        fmt.line("info: &INFO,");
+        fmtln!(fmt, r#"name: "{}","#, reg_class.name);
+        fmtln!(fmt, "index: {},", reg_class.index.index());
+        fmtln!(fmt, "width: {},", reg_class.width);
+        fmtln!(fmt, "bank: {},", reg_class.bank.index());
+        fmtln!(fmt, "toprc: {},", reg_class.toprc.index());
+        fmtln!(fmt, "first: {},", reg_bank.first_unit + reg_class.start);
+        fmtln!(fmt, "subclasses: {:#x},", reg_class.subclass_mask());
+        fmtln!(fmt, "mask: [{}],", mask);
+        fmtln!(fmt, "info: &INFO,");
     });
-    fmt.line("};");
-    fmt.line("#[allow(dead_code)]");
-    fmt.line(format!(
+    fmtln!(fmt, "};");
+
+    fmtln!(fmt, "#[allow(dead_code)]");
+    fmtln!(
+        fmt,
         "pub static {}: RegClass = &{}_DATA;",
-        reg_class.name, reg_class.name
-    ));
+        reg_class.name,
+        reg_class.name
+    );
 }
 
 fn gen_regbank_units(reg_bank: &RegBank, fmt: &mut Formatter) {
     for unit in 0..reg_bank.units {
         let v = unit + reg_bank.first_unit;
         if (unit as usize) < reg_bank.names.len() {
-            fmt.line(format!("{} = {},", reg_bank.names[unit as usize], v));
+            fmtln!(fmt, "{} = {},", reg_bank.names[unit as usize], v);
             continue;
         }
-        fmt.line(format!("{}{} = {},", reg_bank.prefix, unit, v));
+        fmtln!(fmt, "{}{} = {},", reg_bank.prefix, unit, v);
     }
 }
 
 fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) {
     // Emit RegInfo.
-    fmt.line("pub static INFO: RegInfo = RegInfo {");
+    fmtln!(fmt, "pub static INFO: RegInfo = RegInfo {");
 
     fmt.indent(|fmt| {
-        fmt.line("banks: &[");
+        fmtln!(fmt, "banks: &[");
         // Bank descriptors.
         fmt.indent(|fmt| {
             for reg_bank in isa.regs.banks.values() {
                 gen_regbank(fmt, ®_bank);
             }
         });
-        fmt.line("],");
+        fmtln!(fmt, "],");
         // References to register classes.
-        fmt.line("classes: &[");
+        fmtln!(fmt, "classes: &[");
         fmt.indent(|fmt| {
             for reg_class in isa.regs.classes.values() {
-                fmt.line(format!("&{}_DATA,", reg_class.name));
+                fmtln!(fmt, "&{}_DATA,", reg_class.name);
             }
         });
-        fmt.line("],");
+        fmtln!(fmt, "],");
     });
-    fmt.line("};");
+    fmtln!(fmt, "};");
 
     // Register class descriptors.
     for rc in isa.regs.classes.values() {
@@ -105,26 +110,26 @@ fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) {
     }
 
     // Emit constants for all the register units.
-    fmt.line("#[allow(dead_code, non_camel_case_types)]");
-    fmt.line("#[derive(Clone, Copy)]");
-    fmt.line("pub enum RU {");
+    fmtln!(fmt, "#[allow(dead_code, non_camel_case_types)]");
+    fmtln!(fmt, "#[derive(Clone, Copy)]");
+    fmtln!(fmt, "pub enum RU {");
     fmt.indent(|fmt| {
         for reg_bank in isa.regs.banks.values() {
             gen_regbank_units(reg_bank, fmt);
         }
     });
-    fmt.line("}");
+    fmtln!(fmt, "}");
 
     // Emit Into conversion for the RU class.
-    fmt.line("impl Into for RU {");
+    fmtln!(fmt, "impl Into for RU {");
     fmt.indent(|fmt| {
-        fmt.line("fn into(self) -> RegUnit {");
+        fmtln!(fmt, "fn into(self) -> RegUnit {");
         fmt.indent(|fmt| {
-            fmt.line("self as RegUnit");
+            fmtln!(fmt, "self as RegUnit");
         });
-        fmt.line("}")
+        fmtln!(fmt, "}");
     });
-    fmt.line("}");
+    fmtln!(fmt, "}");
 }
 
 pub fn generate(isa: &TargetIsa, base_filename: &str, out_dir: &str) -> Result<(), error::Error> {
diff --git a/cranelift/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
index 183c8643a2..a3c97ebab8 100644
--- a/cranelift/codegen/meta/src/gen_settings.rs
+++ b/cranelift/codegen/meta/src/gen_settings.rs
@@ -21,84 +21,92 @@ fn gen_constructor(group: &SettingGroup, parent: ParentGroup, fmt: &mut Formatte
         ParentGroup::None => "builder: Builder",
         ParentGroup::Shared => "shared: &settings::Flags, builder: Builder",
     };
-    fmt.line("impl Flags {");
+    fmtln!(fmt, "impl Flags {");
     fmt.indent(|fmt| {
         fmt.doc_comment(format!("Create flags {} settings group.", group.name));
-        fmt.line("#[allow(unused_variables)]");
-        fmt.line(format!("pub fn new({}) -> Self {{", args));
+        fmtln!(fmt, "#[allow(unused_variables)]");
+        fmtln!(fmt, "pub fn new({}) -> Self {{", args);
         fmt.indent(|fmt| {
-            fmt.line(format!("let bvec = builder.state_for(\"{}\");", group.name));
-            fmt.line(format!(
+            fmtln!(fmt, "let bvec = builder.state_for(\"{}\");", group.name);
+            fmtln!(
+                fmt,
                 "let mut {} = Self {{ bytes: [0; {}] }};",
                 group.name,
                 group.byte_size()
-            ));
-            fmt.line(format!(
+            );
+            fmtln!(
+                fmt,
                 "debug_assert_eq!(bvec.len(), {});",
                 group.settings_size
-            ));
-            fmt.line(format!(
+            );
+            fmtln!(
+                fmt,
                 "{}.bytes[0..{}].copy_from_slice(&bvec);",
-                group.name, group.settings_size
-            ));
+                group.name,
+                group.settings_size
+            );
 
             // Now compute the predicates.
             for p in &group.predicates {
                 fmt.comment(format!("Precompute #{}.", p.number));
-                fmt.line(format!("if {} {{", p.render(group)));
+                fmtln!(fmt, "if {} {{", p.render(group));
                 fmt.indent(|fmt| {
-                    fmt.line(format!(
+                    fmtln!(
+                        fmt,
                         "{}.bytes[{}] |= 1 << {};",
                         group.name,
                         group.bool_start_byte_offset + p.number / 8,
                         p.number % 8
-                    ));
+                    );
                 });
-                fmt.line("}");
+                fmtln!(fmt, "}");
             }
 
-            fmt.line(group.name);
+            fmtln!(fmt, group.name);
         });
-        fmt.line("}");
+        fmtln!(fmt, "}");
     });
-    fmt.line("}");
+    fmtln!(fmt, "}");
 }
 
 /// Emit Display and FromStr implementations for enum settings.
 fn gen_to_and_from_str(name: &str, values: &[&'static str], fmt: &mut Formatter) {
-    fmt.line(format!("impl fmt::Display for {} {{", name));
+    fmtln!(fmt, "impl fmt::Display for {} {{", name);
     fmt.indent(|fmt| {
-        fmt.line("fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {");
+        fmtln!(
+            fmt,
+            "fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {"
+        );
         fmt.indent(|fmt| {
-            fmt.line("f.write_str(match *self {");
+            fmtln!(fmt, "f.write_str(match *self {");
             fmt.indent(|fmt| {
                 for v in values.iter() {
-                    fmt.line(format!("{}::{} => \"{}\",", name, camel_case(v), v));
+                    fmtln!(fmt, "{}::{} => \"{}\",", name, camel_case(v), v);
                 }
             });
-            fmt.line("})");
+            fmtln!(fmt, "})");
         });
-        fmt.line("}");
+        fmtln!(fmt, "}");
     });
-    fmt.line("}");
+    fmtln!(fmt, "}");
 
-    fmt.line(format!("impl str::FromStr for {} {{", name));
+    fmtln!(fmt, "impl str::FromStr for {} {{", name);
     fmt.indent(|fmt| {
-        fmt.line("type Err = ();");
-        fmt.line("fn from_str(s: &str) -> Result {");
+        fmtln!(fmt, "type Err = ();");
+        fmtln!(fmt, "fn from_str(s: &str) -> Result {");
         fmt.indent(|fmt| {
-            fmt.line("match s {");
+            fmtln!(fmt, "match s {");
             fmt.indent(|fmt| {
                 for v in values.iter() {
-                    fmt.line(format!("\"{}\" => Ok({}::{}),", v, name, camel_case(v)));
+                    fmtln!(fmt, "\"{}\" => Ok({}::{}),", v, name, camel_case(v));
                 }
-                fmt.line("_ => Err(()),");
+                fmtln!(fmt, "_ => Err(()),");
             });
-            fmt.line("}");
+            fmtln!(fmt, "}");
         });
-        fmt.line("}");
+        fmtln!(fmt, "}");
     });
-    fmt.line("}");
+    fmtln!(fmt, "}");
 }
 
 /// Emit real enum for the Enum settings.
@@ -111,15 +119,15 @@ fn gen_enum_types(group: &SettingGroup, fmt: &mut Formatter) {
         let name = camel_case(setting.name);
 
         fmt.doc_comment(format!("Values for `{}.{}`.", group.name, setting.name));
-        fmt.line("#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]");
-        fmt.line(format!("pub enum {} {{", name));
+        fmtln!(fmt, "#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]");
+        fmtln!(fmt, "pub enum {} {{", name);
         fmt.indent(|fmt| {
             for v in values.iter() {
                 fmt.doc_comment(format!("`{}`.", v));
-                fmt.line(format!("{},", camel_case(v)));
+                fmtln!(fmt, "{},", camel_case(v));
             }
         });
-        fmt.line("}");
+        fmtln!(fmt, "}");
 
         gen_to_and_from_str(&name, values, fmt);
     }
@@ -132,15 +140,15 @@ fn gen_getter(setting: &Setting, fmt: &mut Formatter) {
         SpecificSetting::Bool(BoolSetting {
             predicate_number, ..
         }) => {
-            fmt.line(format!("pub fn {}(&self) -> bool {{", setting.name));
+            fmtln!(fmt, "pub fn {}(&self) -> bool {{", setting.name);
             fmt.indent(|fmt| {
-                fmt.line(format!("self.numbered_predicate({})", predicate_number));
+                fmtln!(fmt, "self.numbered_predicate({})", predicate_number);
             });
-            fmt.line("}");
+            fmtln!(fmt, "}");
         }
         SpecificSetting::Enum(ref values) => {
             let ty = camel_case(setting.name);
-            fmt.line(format!("pub fn {}(&self) -> {} {{", setting.name, ty));
+            fmtln!(fmt, "pub fn {}(&self) -> {} {{", setting.name, ty);
             fmt.indent(|fmt| {
                 let mut m = Match::new(format!("self.bytes[{}]", setting.byte_offset));
                 for (i, v) in values.iter().enumerate() {
@@ -153,53 +161,58 @@ fn gen_getter(setting: &Setting, fmt: &mut Formatter) {
                 m.arm("_", vec![], "panic!(\"Invalid enum value\")");
                 fmt.add_match(m);
             });
-            fmt.line("}");
+            fmtln!(fmt, "}");
         }
         SpecificSetting::Num(_) => {
-            fmt.line(format!("pub fn {}(&self) -> u8 {{", setting.name));
+            fmtln!(fmt, "pub fn {}(&self) -> u8 {{", setting.name);
             fmt.indent(|fmt| {
-                fmt.line(format!("self.bytes[{}]", setting.byte_offset));
+                fmtln!(fmt, "self.bytes[{}]", setting.byte_offset);
             });
-            fmt.line("}");
+            fmtln!(fmt, "}");
         }
     }
 }
 
 fn gen_pred_getter(predicate: &Predicate, group: &SettingGroup, fmt: &mut Formatter) {
     fmt.doc_comment(format!("Computed predicate `{}`.", predicate.render(group)));
-    fmt.line(format!("pub fn {}(&self) -> bool {{", predicate.name));
+    fmtln!(fmt, "pub fn {}(&self) -> bool {{", predicate.name);
     fmt.indent(|fmt| {
-        fmt.line(format!("self.numbered_predicate({})", predicate.number));
+        fmtln!(fmt, "self.numbered_predicate({})", predicate.number);
     });
-    fmt.line("}");
+    fmtln!(fmt, "}");
 }
 
 /// Emits getters for each setting value.
 fn gen_getters(group: &SettingGroup, fmt: &mut Formatter) {
     fmt.doc_comment("User-defined settings.");
-    fmt.line("#[allow(dead_code)]");
-    fmt.line("impl Flags {");
+    fmtln!(fmt, "#[allow(dead_code)]");
+    fmtln!(fmt, "impl Flags {");
     fmt.indent(|fmt| {
         fmt.doc_comment("Get a view of the boolean predicates.");
-        fmt.line("pub fn predicate_view(&self) -> ::settings::PredicateView {");
+        fmtln!(
+            fmt,
+            "pub fn predicate_view(&self) -> ::settings::PredicateView {"
+        );
         fmt.indent(|fmt| {
-            fmt.line(format!(
+            fmtln!(
+                fmt,
                 "::settings::PredicateView::new(&self.bytes[{}..])",
                 group.bool_start_byte_offset
-            ));
+            );
         });
-        fmt.line("}");
+        fmtln!(fmt, "}");
 
         if group.settings.len() > 0 {
             fmt.doc_comment("Dynamic numbered predicate getter.");
-            fmt.line("fn numbered_predicate(&self, p: usize) -> bool {");
+            fmtln!(fmt, "fn numbered_predicate(&self, p: usize) -> bool {");
             fmt.indent(|fmt| {
-                fmt.line(format!(
+                fmtln!(
+                    fmt,
                     "self.bytes[{} + p / 8] & (1 << (p % 8)) != 0",
                     group.bool_start_byte_offset
-                ));
+                );
             });
-            fmt.line("}");
+            fmtln!(fmt, "}");
         }
 
         for setting in &group.settings {
@@ -209,7 +222,7 @@ fn gen_getters(group: &SettingGroup, fmt: &mut Formatter) {
             gen_pred_getter(&predicate, &group, fmt);
         }
     });
-    fmt.line("}");
+    fmtln!(fmt, "}");
 }
 
 #[derive(Hash, PartialEq, Eq)]
@@ -234,66 +247,66 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
     let mut descriptor_index_map: HashMap = HashMap::new();
 
     // Generate descriptors.
-    fmt.line(format!(
+    fmtln!(
+        fmt,
         "static DESCRIPTORS: [detail::Descriptor; {}] = [",
         group.settings.len() + group.presets.len()
-    ));
+    );
     fmt.indent(|fmt| {
         for (idx, setting) in group.settings.iter().enumerate() {
-            fmt.line("detail::Descriptor {");
+            fmtln!(fmt, "detail::Descriptor {");
             fmt.indent(|fmt| {
-                fmt.line(format!("name: \"{}\",", setting.name));
-                fmt.line(format!("offset: {},", setting.byte_offset));
+                fmtln!(fmt, "name: \"{}\",", setting.name);
+                fmtln!(fmt, "offset: {},", setting.byte_offset);
                 match setting.specific {
                     SpecificSetting::Bool(BoolSetting { bit_offset, .. }) => {
-                        fmt.line(format!(
+                        fmtln!(
+                            fmt,
                             "detail: detail::Detail::Bool {{ bit: {} }},",
                             bit_offset
-                        ));
+                        );
                     }
                     SpecificSetting::Enum(ref values) => {
                         let offset = enum_table.add(values);
-                        fmt.line(format!(
+                        fmtln!(
+                            fmt,
                             "detail: detail::Detail::Enum {{ last: {}, enumerators: {} }},",
                             values.len() - 1,
                             offset
-                        ));
+                        );
                     }
                     SpecificSetting::Num(_) => {
-                        fmt.line("detail: detail::Detail::Num,");
+                        fmtln!(fmt, "detail: detail::Detail::Num,");
                     }
                 }
 
                 descriptor_index_map.insert(SettingOrPreset::Setting(setting), idx);
             });
-            fmt.line("},");
+            fmtln!(fmt, "},");
         }
 
         for (idx, preset) in group.presets.iter().enumerate() {
-            fmt.line("detail::Descriptor {");
+            fmtln!(fmt, "detail::Descriptor {");
             fmt.indent(|fmt| {
-                fmt.line(format!("name: \"{}\",", preset.name));
-                fmt.line(format!("offset: {},", (idx as u8) * group.settings_size));
-                fmt.line("detail: detail::Detail::Preset,");
+                fmtln!(fmt, "name: \"{}\",", preset.name);
+                fmtln!(fmt, "offset: {},", (idx as u8) * group.settings_size);
+                fmtln!(fmt, "detail: detail::Detail::Preset,");
             });
-            fmt.line("},");
+            fmtln!(fmt, "},");
 
             descriptor_index_map.insert(SettingOrPreset::Preset(preset), idx);
         }
     });
-    fmt.line("];");
+    fmtln!(fmt, "];");
 
     // Generate enumerators.
-    fmt.line(format!(
-        "static ENUMERATORS: [&str; {}] = [",
-        enum_table.len()
-    ));
+    fmtln!(fmt, "static ENUMERATORS: [&str; {}] = [", enum_table.len());
     fmt.indent(|fmt| {
         for enum_val in enum_table.iter() {
-            fmt.line(format!("\"{}\",", enum_val));
+            fmtln!(fmt, "\"{}\",", enum_val);
         }
     });
-    fmt.line("];");
+    fmtln!(fmt, "];");
 
     // Generate hash table.
     let mut hash_entries: Vec = Vec::new();
@@ -312,40 +325,39 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
             .collect::>(),
     );
     let hash_table = generate_table(&hash_entries, |entry| simple_hash(entry.name()));
-    fmt.line(format!(
-        "static HASH_TABLE: [u16; {}] = [",
-        hash_table.len()
-    ));
+    fmtln!(fmt, "static HASH_TABLE: [u16; {}] = [", hash_table.len());
     fmt.indent(|fmt| {
         for h in &hash_table {
             match *h {
-                Some(setting_or_preset) => fmt.line(format!(
+                Some(setting_or_preset) => fmtln!(
+                    fmt,
                     "{},",
                     &descriptor_index_map
                         .get(setting_or_preset)
                         .unwrap()
                         .to_string()
-                )),
-                None => fmt.line("0xffff,"),
+                ),
+                None => fmtln!(fmt, "0xffff,"),
             }
         }
     });
-    fmt.line("];");
+    fmtln!(fmt, "];");
 
     // Generate presets.
-    fmt.line(format!(
+    fmtln!(
+        fmt,
         "static PRESETS: [(u8, u8); {}] = [",
         group.presets.len()
-    ));
+    );
     fmt.indent(|fmt| {
         for preset in &group.presets {
             fmt.comment(preset.name);
             for (mask, value) in preset.layout(&group) {
-                fmt.line(format!("(0b{:08b}, 0b{:08b}),", mask, value));
+                fmtln!(fmt, "(0b{:08b}, 0b{:08b}),", mask, value);
             }
         }
     });
-    fmt.line("];");
+    fmtln!(fmt, "];");
 }
 
 fn gen_template(group: &SettingGroup, fmt: &mut Formatter) {
@@ -360,63 +372,70 @@ fn gen_template(group: &SettingGroup, fmt: &mut Formatter) {
         .collect();
     let default_bytes_str = default_bytes.join(", ");
 
-    fmt.line("static TEMPLATE: detail::Template = detail::Template {");
+    fmtln!(
+        fmt,
+        "static TEMPLATE: detail::Template = detail::Template {"
+    );
     fmt.indent(|fmt| {
-        fmt.line(format!("name: \"{}\",", group.name));
-        fmt.line("descriptors: &DESCRIPTORS,");
-        fmt.line("enumerators: &ENUMERATORS,");
-        fmt.line("hash_table: &HASH_TABLE,");
-        fmt.line(format!("defaults: &[{}],", default_bytes_str));
-        fmt.line("presets: &PRESETS,");
+        fmtln!(fmt, "name: \"{}\",", group.name);
+        fmtln!(fmt, "descriptors: &DESCRIPTORS,");
+        fmtln!(fmt, "enumerators: &ENUMERATORS,");
+        fmtln!(fmt, "hash_table: &HASH_TABLE,");
+        fmtln!(fmt, "defaults: &[{}],", default_bytes_str);
+        fmtln!(fmt, "presets: &PRESETS,");
     });
-    fmt.line("};");
+    fmtln!(fmt, "};");
 
     fmt.doc_comment(format!(
         "Create a `settings::Builder` for the {} settings group.",
         group.name
     ));
-    fmt.line("pub fn builder() -> Builder {");
+    fmtln!(fmt, "pub fn builder() -> Builder {");
     fmt.indent(|fmt| {
-        fmt.line("Builder::new(&TEMPLATE)");
+        fmtln!(fmt, "Builder::new(&TEMPLATE)");
     });
-    fmt.line("}");
+    fmtln!(fmt, "}");
 }
 
 fn gen_display(group: &SettingGroup, fmt: &mut Formatter) {
-    fmt.line("impl fmt::Display for Flags {");
+    fmtln!(fmt, "impl fmt::Display for Flags {");
     fmt.indent(|fmt| {
-        fmt.line("fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {");
+        fmtln!(
+            fmt,
+            "fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {"
+        );
         fmt.indent(|fmt| {
-            fmt.line(format!("writeln!(f, \"[{}]\")?;", group.name));
-            fmt.line("for d in &DESCRIPTORS {");
+            fmtln!(fmt, "writeln!(f, \"[{}]\")?;", group.name);
+            fmtln!(fmt, "for d in &DESCRIPTORS {");
             fmt.indent(|fmt| {
-                fmt.line("if !d.detail.is_preset() {");
+                fmtln!(fmt, "if !d.detail.is_preset() {");
                 fmt.indent(|fmt| {
-                    fmt.line("write!(f, \"{} = \", d.name)?;");
-                    fmt.line(
+                    fmtln!(fmt, "write!(f, \"{} = \", d.name)?;");
+                    fmtln!(
+                        fmt,
                         "TEMPLATE.format_toml_value(d.detail, self.bytes[d.offset as usize], f)?;",
                     );
-                    fmt.line("writeln!(f)?;");
+                    fmtln!(fmt, "writeln!(f)?;");
                 });
-                fmt.line("}");
+                fmtln!(fmt, "}");
             });
-            fmt.line("}");
-            fmt.line("Ok(())");
+            fmtln!(fmt, "}");
+            fmtln!(fmt, "Ok(())");
         });
-        fmt.line("}")
+        fmtln!(fmt, "}")
     });
-    fmt.line("}");
+    fmtln!(fmt, "}");
 }
 
 fn gen_group(group: &SettingGroup, parent: ParentGroup, fmt: &mut Formatter) {
     // Generate struct.
-    fmt.line("#[derive(Clone)]");
+    fmtln!(fmt, "#[derive(Clone)]");
     fmt.doc_comment(format!("Flags group `{}`.", group.name));
-    fmt.line("pub struct Flags {");
+    fmtln!(fmt, "pub struct Flags {");
     fmt.indent(|fmt| {
-        fmt.line(format!("bytes: [u8; {}],", group.byte_size()));
+        fmtln!(fmt, "bytes: [u8; {}],", group.byte_size());
     });
-    fmt.line("}");
+    fmtln!(fmt, "}");
 
     gen_constructor(group, parent, fmt);
     gen_enum_types(group, fmt);
diff --git a/cranelift/codegen/meta/src/gen_types.rs b/cranelift/codegen/meta/src/gen_types.rs
index 5b48d17a27..0a52eb371e 100644
--- a/cranelift/codegen/meta/src/gen_types.rs
+++ b/cranelift/codegen/meta/src/gen_types.rs
@@ -22,7 +22,7 @@ fn emit_type(ty: &cdsl_types::ValueType, fmt: &mut srcgen::Formatter) -> Result<
     })?;
 
     fmt.doc_comment(&ty.doc());
-    fmt.line(format!("pub const {}: Type = Type({:#x});\n", name, number));
+    fmtln!(fmt, "pub const {}: Type = Type({:#x});\n", name, number);
 
     Ok(())
 }
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index b617d31c16..fe65e19d2b 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -1,5 +1,6 @@
 #[macro_use]
 mod cdsl;
+mod srcgen;
 
 pub mod error;
 pub mod isa;
@@ -10,7 +11,6 @@ mod gen_types;
 
 mod constant_hash;
 mod shared;
-mod srcgen;
 mod unique_table;
 
 pub fn isa_from_arch(arch: &str) -> Result {
diff --git a/cranelift/codegen/meta/src/srcgen.rs b/cranelift/codegen/meta/src/srcgen.rs
index 4e86e0b8c7..dc7f622940 100644
--- a/cranelift/codegen/meta/src/srcgen.rs
+++ b/cranelift/codegen/meta/src/srcgen.rs
@@ -3,6 +3,8 @@
 //! The `srcgen` module contains generic helper routines and classes for
 //! generating source code.
 
+#![macro_use]
+
 use std::cmp;
 use std::collections::{BTreeMap, BTreeSet};
 use std::fs;
@@ -13,6 +15,18 @@ use crate::error;
 
 static SHIFTWIDTH: usize = 4;
 
+/// A macro that simplifies the usage of the Formatter by allowing format
+/// strings.
+macro_rules! fmtln {
+    ($fmt:ident, $fmtstring:expr, $($fmtargs:expr),*) => {
+        $fmt.line(format!($fmtstring, $($fmtargs),*));
+    };
+
+    ($fmt:ident, $arg:expr) => {
+        $fmt.line($arg);
+    };
+}
+
 pub struct Formatter {
     indent: usize,
     lines: Vec,
@@ -104,7 +118,7 @@ impl Formatter {
 
     /// Add a comment line.
     pub fn comment(&mut self, s: impl AsRef) {
-        self.line(format!("// {}", s.as_ref()));
+        fmtln!(self, "// {}", s.as_ref());
     }
 
     /// Add a (multi-line) documentation comment.
@@ -123,7 +137,7 @@ impl Formatter {
 
     /// Add a match expression.
     pub fn add_match(&mut self, m: Match) {
-        self.line(format!("match {} {{", m.expr));
+        fmtln!(self, "match {} {{", m.expr);
         self.indent(|fmt| {
             for (&(ref fields, ref body), ref names) in m.arms.iter() {
                 // name { fields } | name { fields } => { body }
@@ -138,7 +152,7 @@ impl Formatter {
                     })
                     .collect();
                 let lhs = conditions.join(" | ");
-                fmt.line(format!("{} => {{", lhs));
+                fmtln!(fmt, "{} => {{", lhs);
                 fmt.indent(|fmt| {
                     fmt.line(body);
                 });

From 851f125edacdca704d87be0bad3fa717c8ee12b6 Mon Sep 17 00:00:00 2001
From: robojumper 
Date: Fri, 22 Feb 2019 22:45:32 +0100
Subject: [PATCH 2325/3084] Fix bitrev intrinsic

---
 cranelift/codegen/meta-python/base/legalize.py      | 8 ++++----
 cranelift/filetests/filetests/legalizer/bitrev.clif | 8 ++++----
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/cranelift/codegen/meta-python/base/legalize.py b/cranelift/codegen/meta-python/base/legalize.py
index 163379cfac..1902dea080 100644
--- a/cranelift/codegen/meta-python/base/legalize.py
+++ b/cranelift/codegen/meta-python/base/legalize.py
@@ -570,7 +570,7 @@ widen.legalize(
             b1 << band_imm(b, imm64(0xcc)),
             b2 << ushr_imm(b1, imm64(2)),
             b3 << band_imm(b, imm64(0x33)),
-            b4 << ushr_imm(b3, imm64(2)),
+            b4 << ishl_imm(b3, imm64(2)),
             c << bor(b2, b4),
             c1 << band_imm(c, imm64(0xf0)),
             c2 << ushr_imm(c1, imm64(4)),
@@ -590,7 +590,7 @@ widen.legalize(
             b1 << band_imm(b, imm64(0xcccc)),
             b2 << ushr_imm(b1, imm64(2)),
             b3 << band_imm(b, imm64(0x3333)),
-            b4 << ushr_imm(b3, imm64(2)),
+            b4 << ishl_imm(b3, imm64(2)),
             c << bor(b2, b4),
             c1 << band_imm(c, imm64(0xf0f0)),
             c2 << ushr_imm(c1, imm64(4)),
@@ -615,7 +615,7 @@ expand.legalize(
             b1 << band_imm(b, imm64(0xcccccccc)),
             b2 << ushr_imm(b1, imm64(2)),
             b3 << band_imm(b, imm64(0x33333333)),
-            b4 << ushr_imm(b3, imm64(2)),
+            b4 << ishl_imm(b3, imm64(2)),
             c << bor(b2, b4),
             c1 << band_imm(c, imm64(0xf0f0f0f0)),
             c2 << ushr_imm(c1, imm64(4)),
@@ -643,7 +643,7 @@ expand.legalize(
             b1 << band_imm(b, imm64(0xcccccccccccccccc)),
             b2 << ushr_imm(b1, imm64(2)),
             b3 << band_imm(b, imm64(0x3333333333333333)),
-            b4 << ushr_imm(b3, imm64(2)),
+            b4 << ishl_imm(b3, imm64(2)),
             c << bor(b2, b4),
             c1 << band_imm(c, imm64(0xf0f0f0f0f0f0f0f0)),
             c2 << ushr_imm(c1, imm64(4)),
diff --git a/cranelift/filetests/filetests/legalizer/bitrev.clif b/cranelift/filetests/filetests/legalizer/bitrev.clif
index b4098b87b4..b7ee07735e 100644
--- a/cranelift/filetests/filetests/legalizer/bitrev.clif
+++ b/cranelift/filetests/filetests/legalizer/bitrev.clif
@@ -32,7 +32,7 @@ ebb0(v0: i8):
 ; check: v32 = band_imm v31, 51
 ; check: v9 = ireduce.i8 v32
 ; check: v33 = uextend.i32 v9
-; check: v34 = ushr_imm v33, 2
+; check: v34 = ishl_imm v33, 2
 ; check: v10 = ireduce.i8 v34
 ; check: v35 = uextend.i32 v8
 ; check: v36 = uextend.i32 v10
@@ -87,7 +87,7 @@ ebb0(v0: i16):
 ; check: v37 = band_imm v36, 0x3333
 ; check: v9 = ireduce.i16 v37
 ; check: v38 = uextend.i32 v9
-; check: v39 = ushr_imm v38, 2
+; check: v39 = ishl_imm v38, 2
 ; check: v10 = ireduce.i16 v39
 ; check: v40 = uextend.i32 v8
 ; check: v41 = uextend.i32 v10
@@ -142,7 +142,7 @@ ebb0(v0: i32):
 ; check: v7 = band v6, v25
 ; check: v8 = ushr_imm v7, 2
 ; check: v9 = band_imm v6, 0x3333_3333
-; check: v10 = ushr_imm v9, 2
+; check: v10 = ishl_imm v9, 2
 ; check: v11 = bor v8, v10
 ; check: v26 = iconst.i32 0xf0f0_f0f0
 ; check: v12 = band v11, v26
@@ -178,7 +178,7 @@ ebb0(v0: i64):
 ; check: v8 = ushr_imm v7, 2
 ; check: v32 = iconst.i64 0x3333_3333_3333_3333
 ; check: v9 = band v6, v32
-; check: v10 = ushr_imm v9, 2
+; check: v10 = ishl_imm v9, 2
 ; check: v11 = bor v8, v10
 ; check: v33 = iconst.i64 0xf0f0_f0f0_f0f0_f0f0
 ; check: v12 = band v11, v33

From 83edf959f20e6637985ae9dbfc6fd0014f629bb8 Mon Sep 17 00:00:00 2001
From: kryptan 
Date: Sun, 24 Feb 2019 12:31:32 +0300
Subject: [PATCH 2326/3084] Fix link to issues in rustc.md

---
 cranelift/rustc.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/rustc.md b/cranelift/rustc.md
index 1f2749e323..90e9fb03cc 100644
--- a/cranelift/rustc.md
+++ b/cranelift/rustc.md
@@ -22,7 +22,7 @@ There's plenty of work to do to achieve these goals, and if we achieve
 them, we'll have enabled a Rust compiler written entirely in Rust, and
 enabled faster Rust compile times for important use cases.
 
-See [issues tagged "rustc"](https://github.com/CraneStation/cranelift/labels/rustc)
+See [issues tagged "rustc"](https://github.com/CraneStation/cranelift/labels/FA-rustc)
 for a list of some of the things that will be needed.
 
 With all that said, there is a potential goal beyond that, which is to

From 166c11af11af5741322f3f12790c7494d224143d Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 23 Feb 2019 11:13:24 +0100
Subject: [PATCH 2327/3084] Fix global value colocated printing

It used to print `gv0 = colocated symbol u1:1` while cranelift-reader
expects `gv0 = symbol colocated u1:1`.
---
 cranelift/codegen/src/ir/globalvalue.rs | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/src/ir/globalvalue.rs b/cranelift/codegen/src/ir/globalvalue.rs
index 5dc7792867..b67c26e1a2 100644
--- a/cranelift/codegen/src/ir/globalvalue.rs
+++ b/cranelift/codegen/src/ir/globalvalue.rs
@@ -114,10 +114,12 @@ impl fmt::Display for GlobalValueData {
                 offset,
                 colocated,
             } => {
-                if colocated {
-                    write!(f, "colocated ")?;
-                }
-                write!(f, "symbol {}", name)?;
+                write!(
+                    f,
+                    "symbol {}{}",
+                    if colocated { "colocated " } else { "" },
+                    name
+                )?;
                 let offset_val: i64 = offset.into();
                 if offset_val > 0 {
                     write!(f, "+")?;

From 6267e1ded9192b4a7a6500e1482791150696d750 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Mon, 25 Feb 2019 19:04:32 +0100
Subject: [PATCH 2328/3084] Implement sshr_imm.i8 (#687)

---
 .../codegen/meta-python/base/legalize.py      | 33 +++++--------------
 1 file changed, 9 insertions(+), 24 deletions(-)

diff --git a/cranelift/codegen/meta-python/base/legalize.py b/cranelift/codegen/meta-python/base/legalize.py
index 1902dea080..423309dab7 100644
--- a/cranelift/codegen/meta-python/base/legalize.py
+++ b/cranelift/codegen/meta-python/base/legalize.py
@@ -375,7 +375,7 @@ for int_ty in [types.i8, types.i16]:
     )
 
 for int_ty in [types.i8, types.i16]:
-    for op in [ushr_imm, ishl_imm]:
+    for op in [ishl, ishl_imm, ushr, ushr_imm]:
         widen.legalize(
             a << op.bind(int_ty)(b, c),
             Rtl(
@@ -384,29 +384,14 @@ for int_ty in [types.i8, types.i16]:
                 a << ireduce.bind(int_ty)(z)
             ))
 
-    widen.legalize(
-        a << ishl.bind(int_ty)(b, c),
-        Rtl(
-            x << uextend.i32(b),
-            z << ishl.i32(x, c),
-            a << ireduce.bind(int_ty)(z)
-        ))
-
-    widen.legalize(
-        a << ushr.bind(int_ty)(b, c),
-        Rtl(
-            x << uextend.i32(b),
-            z << ushr.i32(x, c),
-            a << ireduce.bind(int_ty)(z)
-        ))
-
-    widen.legalize(
-        a << sshr.bind(int_ty)(b, c),
-        Rtl(
-            x << sextend.i32(b),
-            z << sshr.i32(x, c),
-            a << ireduce.bind(int_ty)(z)
-        ))
+    for op in [sshr, sshr_imm]:
+        widen.legalize(
+            a << op.bind(int_ty)(b, c),
+            Rtl(
+                x << sextend.i32(b),
+                z << op.i32(x, c),
+                a << ireduce.bind(int_ty)(z)
+            ))
 
     for w_cc in [
         intcc.eq, intcc.ne, intcc.ugt, intcc.ult, intcc.uge, intcc.ule

From d2894f9f45207ca02d3bda68eb46bdd870afc7a3 Mon Sep 17 00:00:00 2001
From: Pat Hickey 
Date: Thu, 21 Feb 2019 15:48:03 -0800
Subject: [PATCH 2329/3084] cranelift-faerie: convert to use new extensible
 decl format

Bump faerie to 0.8.0 and goblin to 0.0.21.

* Preemptible linkages are now weak symbols
* Faerie will put read-only data in .rodata for elf
---
 cranelift/faerie/Cargo.toml     |  4 ++--
 cranelift/faerie/src/backend.rs | 25 +++++++++----------------
 2 files changed, 11 insertions(+), 18 deletions(-)

diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index d15ebf4aa9..c0f1f95bd1 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -12,8 +12,8 @@ edition = "2018"
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0" }
 cranelift-module = { path = "../cranelift-module", version = "0.28.0" }
-faerie = "0.7.0"
-goblin = "0.0.19"
+faerie = "0.8.0"
+goblin = "0.0.21"
 failure = "0.1.2"
 target-lexicon = "0.2.0"
 
diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index 92cf1e4ba4..f413879ac2 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -334,26 +334,19 @@ impl FaerieProduct {
 
 fn translate_function_linkage(linkage: Linkage) -> faerie::Decl {
     match linkage {
-        Linkage::Import => faerie::Decl::FunctionImport,
-        Linkage::Local => faerie::Decl::Function { global: false },
-        Linkage::Preemptible | Linkage::Export => faerie::Decl::Function { global: true },
+        Linkage::Import => faerie::Decl::function_import().into(),
+        Linkage::Local => faerie::Decl::function().into(),
+        Linkage::Export => faerie::Decl::function().global().into(),
+        Linkage::Preemptible => faerie::Decl::function().weak().into(),
     }
 }
 
 fn translate_data_linkage(linkage: Linkage, writable: bool) -> faerie::Decl {
     match linkage {
-        Linkage::Import => faerie::Decl::DataImport,
-        Linkage::Local => faerie::Decl::Data {
-            global: false,
-            writable,
-        },
-        Linkage::Export => faerie::Decl::Data {
-            global: true,
-            writable,
-        },
-        Linkage::Preemptible => {
-            unimplemented!("faerie doesn't support preemptible globals yet");
-        }
+        Linkage::Import => faerie::Decl::data_import().into(),
+        Linkage::Local => faerie::Decl::data().with_writable(writable).into(),
+        Linkage::Export => faerie::Decl::data().global().with_writable(writable).into(),
+        Linkage::Preemptible => faerie::Decl::data().weak().with_writable(writable).into(),
     }
 }
 
@@ -388,7 +381,7 @@ impl<'a> RelocSink for FaerieRelocSink<'a> {
             ir::ExternalName::LibCall(ref libcall) => {
                 let sym = (self.libcall_names)(*libcall);
                 self.artifact
-                    .declare(sym.clone(), faerie::Decl::FunctionImport)
+                    .declare(sym.clone(), faerie::Decl::function_import())
                     .expect("faerie declaration of libcall");
                 sym
             }

From 2f2626f4b3a1578fb8007f7d82d6794dec74fd15 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 1 Mar 2019 15:17:31 -0800
Subject: [PATCH 2330/3084] Remove the empty `[workspace]` declaration.

It doesn't appear to be needed; the crate directories having Cargo.toml
files appears to be sufficient.
---
 cranelift/Cargo.toml | 2 --
 1 file changed, 2 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 36c85ba347..f4c25a168e 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -43,8 +43,6 @@ default = ["disas", "wasm"]
 disas = ["capstone"]
 wasm = ["wabt", "cranelift-wasm"]
 
-[workspace]
-
 # We want debug symbols on release binaries by default since it allows profiling
 # tools to give more accurate information. We can always strip them out later if
 # necessary.

From 2a519092a013e6a3347d85f3925e159e91480186 Mon Sep 17 00:00:00 2001
From: Steffen Butzer 
Date: Tue, 5 Mar 2019 12:17:41 +0100
Subject: [PATCH 2331/3084] =?UTF-8?q?Use=20single=20index=20for=20param=20?=
 =?UTF-8?q?register=20allocation=20for=20windows=20callconv=20(=E2=80=A6?=
 =?UTF-8?q?=20(#693)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* Use single index for param register allocation for windows callconv (#691)

The used registers depend entirely on the parameter index (1st, 2nd, 3rd, 4th, ... param)
and we cannot shift unused registers to other indexes, if they are not designated for
the use for that parameter index.
---
 cranelift/codegen/src/isa/x86/abi.rs              | 15 ++++++++++++---
 .../filetests/isa/x86/windows_fastcall_x64.clif   |  6 ++++++
 2 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 0321e36e85..b25c17e1f6 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -115,9 +115,18 @@ impl ArgAssigner for Args {
         }
 
         // Try to use an FPR.
-        if ty.is_float() && self.fpr_used < self.fpr_limit {
-            let reg = FPR.unit(self.fpr_used);
-            self.fpr_used += 1;
+        let fpr_offset = if self.call_conv == CallConv::WindowsFastcall {
+            // Float and general registers on windows share the same parameter index.
+            // The used register depends entirely on the parameter index: Even if XMM0
+            // is not used for the first parameter, it cannot be used for the second parameter.
+            debug_assert_eq!(self.fpr_limit, self.gpr.len());
+            &mut self.gpr_used
+        } else {
+            &mut self.fpr_used
+        };
+        if ty.is_float() && *fpr_offset < self.fpr_limit {
+            let reg = FPR.unit(*fpr_offset);
+            *fpr_offset += 1;
             return ArgumentLoc::Reg(reg).into();
         }
 
diff --git a/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
index 7af721ed8e..a84c17fb9e 100644
--- a/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
+++ b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
@@ -31,3 +31,9 @@ ebb0(v0: i64, v1: i64, v2: i64, v3: i64, v4: i64):
     return
 }
 ; check: function %five_args(i64 [%rcx], i64 [%rdx], i64 [%r8], i64 [%r9], i64 [32], i64 fp [%rbp]) -> i64 fp [%rbp] windows_fastcall {
+
+function %mixed_int_float(i64, f64, i64, f32) windows_fastcall {
+ebb0(v0: i64, v1: f64, v2: i64, v3: f32):
+    return
+}
+; check: function %mixed_int_float(i64 [%rcx], f64 [%xmm1], i64 [%r8], f32 [%xmm3], i64 fp [%rbp]) -> i64 fp [%rbp] windows_fastcall {
\ No newline at end of file

From 27b0933a4a895d39c30e390fa0aa610feb25cd64 Mon Sep 17 00:00:00 2001
From: Yury Delendik 
Date: Tue, 22 Jan 2019 16:48:31 -0600
Subject: [PATCH 2332/3084] Preserve global wasm module offset in SourceLoc.

---
 cranelift/wasm/src/environ/dummy.rs       |  8 ++++++--
 cranelift/wasm/src/environ/spec.rs        |  6 +++++-
 cranelift/wasm/src/func_translator.rs     | 21 ++++++++++++---------
 cranelift/wasm/src/sections_translator.rs |  3 ++-
 4 files changed, 25 insertions(+), 13 deletions(-)

diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index 28cea89e77..70b592bbda 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -458,7 +458,11 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         self.info.start_func = Some(func_index);
     }
 
-    fn define_function_body(&mut self, body_bytes: &'data [u8]) -> WasmResult<()> {
+    fn define_function_body(
+        &mut self,
+        body_bytes: &'data [u8],
+        body_offset: usize,
+    ) -> WasmResult<()> {
         let func = {
             let mut func_environ = DummyFuncEnvironment::new(&self.info, self.return_mode);
             let func_index =
@@ -467,7 +471,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
             let sig = func_environ.vmctx_sig(self.get_func_type(func_index));
             let mut func = ir::Function::with_name_signature(name, sig);
             self.trans
-                .translate(body_bytes, &mut func, &mut func_environ)?;
+                .translate(body_bytes, body_offset, &mut func, &mut func_environ)?;
             func
         };
         self.func_bytecode_sizes.push(body_bytes.len());
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 50ecca02c6..9b96d42edf 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -343,7 +343,11 @@ pub trait ModuleEnvironment<'data> {
     ///
     /// Note there's no `reserve_function_bodies` function because the number of
     /// functions is already provided by `reserve_func_types`.
-    fn define_function_body(&mut self, body_bytes: &'data [u8]) -> WasmResult<()>;
+    fn define_function_body(
+        &mut self,
+        body_bytes: &'data [u8],
+        body_offset: usize,
+    ) -> WasmResult<()>;
 
     /// Provides the number of data initializers up front. By default this does nothing, but
     /// implementations can use this to preallocate memory if desired.
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index 7e5255499d..6a003f06f8 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -54,10 +54,15 @@ impl FuncTranslator {
     pub fn translate(
         &mut self,
         code: &[u8],
+        code_offset: usize,
         func: &mut ir::Function,
         environ: &mut FE,
     ) -> WasmResult<()> {
-        self.translate_from_reader(BinaryReader::new(code), func, environ)
+        self.translate_from_reader(
+            BinaryReader::new_with_offset(code, code_offset),
+            func,
+            environ,
+        )
     }
 
     /// Translate a binary WebAssembly function from a `BinaryReader`.
@@ -222,11 +227,9 @@ fn parse_function_body(
 
 /// Get the current source location from a reader.
 fn cur_srcloc(reader: &BinaryReader) -> ir::SourceLoc {
-    // We record source locations as byte code offsets relative to the beginning of the function.
-    // This will wrap around of a single function's byte code is larger than 4 GB, but a) the
-    // WebAssembly format doesn't allow for that, and b) that would hit other Cranelift
-    // implementation limits anyway.
-    ir::SourceLoc::new(reader.current_position() as u32)
+    // We record source locations as byte code offsets relative to the beginning of the file.
+    // This will wrap around if byte code is larger than 4 GB.
+    ir::SourceLoc::new(reader.original_position() as u32)
 }
 
 #[cfg(test)]
@@ -270,7 +273,7 @@ mod tests {
         ctx.func.signature.returns.push(ir::AbiParam::new(I32));
 
         trans
-            .translate(&BODY, &mut ctx.func, &mut runtime.func_env())
+            .translate(&BODY, 0, &mut ctx.func, &mut runtime.func_env())
             .unwrap();
         debug!("{}", ctx.func.display(None));
         ctx.verify(&flags).unwrap();
@@ -308,7 +311,7 @@ mod tests {
         ctx.func.signature.returns.push(ir::AbiParam::new(I32));
 
         trans
-            .translate(&BODY, &mut ctx.func, &mut runtime.func_env())
+            .translate(&BODY, 0, &mut ctx.func, &mut runtime.func_env())
             .unwrap();
         debug!("{}", ctx.func.display(None));
         ctx.verify(&flags).unwrap();
@@ -354,7 +357,7 @@ mod tests {
         ctx.func.signature.returns.push(ir::AbiParam::new(I32));
 
         trans
-            .translate(&BODY, &mut ctx.func, &mut runtime.func_env())
+            .translate(&BODY, 0, &mut ctx.func, &mut runtime.func_env())
             .unwrap();
         debug!("{}", ctx.func.display(None));
         ctx.verify(&flags).unwrap();
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index 2c40010169..37c5d34672 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -295,7 +295,8 @@ pub fn parse_code_section<'data>(
     for body in code {
         let mut reader = body?.get_binary_reader();
         let size = reader.bytes_remaining();
-        environ.define_function_body(reader.read_bytes(size)?)?;
+        let offset = reader.original_position();
+        environ.define_function_body(reader.read_bytes(size)?, offset)?;
     }
     Ok(())
 }

From ec521088b736b49dc1f3d11a3bdc989678ca0310 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 5 Mar 2019 15:07:21 +0100
Subject: [PATCH 2333/3084] Fixes #695: Reintroduce a workspace declaration to
 run all tests with cargo;

---
 cranelift/Cargo.toml | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index f4c25a168e..650551c380 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -9,6 +9,10 @@ repository = "https://github.com/CraneStation/cranelift"
 publish = false
 edition = "2018"
 
+# Present here only to make sure that cargo test --all runs tests for all
+# the crates.
+[workspace]
+
 [[bin]]
 name = "clif-util"
 path = "src/clif-util.rs"

From a06d25725364cc697847e0971111a083a12f36c0 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 25 Feb 2019 19:03:37 +0100
Subject: [PATCH 2334/3084] Add a command to push a tag to the repository

The tag is named `v$version` and will thus appear in the tags/releases tabs of the repository, where we can add text explaining new features, API changes, etc.
---
 cranelift/publish-all.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 7f2ccf90e9..918bcd59ce 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -34,7 +34,9 @@ cargo update
 # Note that libraries need to be published in topological order.
 
 echo git commit -a -m "\"Bump version to $version"\"
+echo git tag v$version
 echo git push
+echo git push v$version
 for crate in \
     entity bforest codegen/meta codegen frontend native \
     preopt \

From 2d2b041766178f23938d5ab8ac79c6ccb409bf5c Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 2 Mar 2019 20:19:11 +0100
Subject: [PATCH 2335/3084] Fix emit_small_memset

---
 cranelift/frontend/src/frontend.rs | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index c1b3e52114..9d1eec4b22 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -638,7 +638,7 @@ impl<'a> FunctionBuilder<'a> {
 
     /// Calls libc.memset
     ///
-    /// Writes `size` bytes of value `ch` to memory starting at `buffer`.
+    /// Writes `size` bytes of i8 value `ch` to memory starting at `buffer`.
     pub fn call_memset(
         &mut self,
         config: TargetFrontendConfig,
@@ -672,7 +672,7 @@ impl<'a> FunctionBuilder<'a> {
         &mut self,
         config: TargetFrontendConfig,
         buffer: Value,
-        ch: u32,
+        ch: u8,
         size: u64,
         buffer_align: u8,
     ) {
@@ -702,7 +702,7 @@ impl<'a> FunctionBuilder<'a> {
         let load_and_store_amount = size / access_size;
 
         if load_and_store_amount > THRESHOLD {
-            let ch = self.ins().iconst(types::I32, i64::from(ch));
+            let ch = self.ins().iconst(types::I8, i64::from(ch));
             let size = self.ins().iconst(config.pointer_type(), size as i64);
             self.call_memset(config, buffer, ch, size);
         } else {

From 395de457c0a53edc0bec5496c3a132e0b975d111 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 4 Mar 2019 23:18:28 -0800
Subject: [PATCH 2336/3084] Add more tests for emit_small_memset and
 emit_small_memcpy. (#1)

---
 cranelift/frontend/src/frontend.rs | 169 +++++++++++++++++++++++++++++
 1 file changed, 169 insertions(+)

diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 9d1eec4b22..02b1043b0a 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -1078,6 +1078,175 @@ ebb0:
         );
     }
 
+    #[test]
+    fn not_so_small_memcpy() {
+        use core::str::FromStr;
+        use cranelift_codegen::{isa, settings};
+
+        let shared_builder = settings::builder();
+        let shared_flags = settings::Flags::new(shared_builder);
+
+        let triple = ::target_lexicon::Triple::from_str("arm").expect("Couldn't create arm triple");
+
+        let target = isa::lookup(triple)
+            .ok()
+            .map(|b| b.finish(shared_flags))
+            .expect("This test requires arm support.");
+
+        let mut sig = Signature::new(target.default_call_conv());
+        sig.returns.push(AbiParam::new(I32));
+
+        let mut fn_ctx = FunctionBuilderContext::new();
+        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+        {
+            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
+
+            let block0 = builder.create_ebb();
+            let x = Variable::new(0);
+            let y = Variable::new(16);
+            builder.declare_var(x, target.pointer_type());
+            builder.declare_var(y, target.pointer_type());
+            builder.append_ebb_params_for_function_params(block0);
+            builder.switch_to_block(block0);
+
+            let src = builder.use_var(x);
+            let dest = builder.use_var(y);
+            let size = 8192;
+            builder.emit_small_memcpy(target.frontend_config(), dest, src, size, 8, 8);
+            builder.ins().return_(&[dest]);
+
+            builder.seal_all_blocks();
+            builder.finalize();
+        }
+
+        assert_eq!(
+            func.display(None).to_string(),
+            "function %sample() -> i32 system_v {
+    sig0 = (i32, i32, i32) system_v
+    fn0 = %Memcpy sig0
+
+ebb0:
+    v4 = iconst.i32 0
+    v1 -> v4
+    v3 = iconst.i32 0
+    v0 -> v3
+    v2 = iconst.i32 8192
+    call fn0(v1, v0, v2)
+    return v1
+}
+"
+        );
+    }
+
+    #[test]
+    fn small_memset() {
+        use core::str::FromStr;
+        use cranelift_codegen::{isa, settings};
+
+        let shared_builder = settings::builder();
+        let shared_flags = settings::Flags::new(shared_builder);
+
+        let triple = ::target_lexicon::Triple::from_str("arm").expect("Couldn't create arm triple");
+
+        let target = isa::lookup(triple)
+            .ok()
+            .map(|b| b.finish(shared_flags))
+            .expect("This test requires arm support.");
+
+        let mut sig = Signature::new(target.default_call_conv());
+        sig.returns.push(AbiParam::new(I32));
+
+        let mut fn_ctx = FunctionBuilderContext::new();
+        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+        {
+            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
+
+            let block0 = builder.create_ebb();
+            let y = Variable::new(16);
+            builder.declare_var(y, target.pointer_type());
+            builder.append_ebb_params_for_function_params(block0);
+            builder.switch_to_block(block0);
+
+            let dest = builder.use_var(y);
+            let size = 8;
+            builder.emit_small_memset(target.frontend_config(), dest, 1, size, 8);
+            builder.ins().return_(&[dest]);
+
+            builder.seal_all_blocks();
+            builder.finalize();
+        }
+
+        assert_eq!(
+            func.display(None).to_string(),
+            "function %sample() -> i32 system_v {
+ebb0:
+    v2 = iconst.i32 0
+    v0 -> v2
+    v1 = iconst.i64 0x0001_0001_0101
+    store aligned v1, v0
+    return v0
+}
+"
+        );
+    }
+
+    #[test]
+    fn not_so_small_memset() {
+        use core::str::FromStr;
+        use cranelift_codegen::{isa, settings};
+
+        let shared_builder = settings::builder();
+        let shared_flags = settings::Flags::new(shared_builder);
+
+        let triple = ::target_lexicon::Triple::from_str("arm").expect("Couldn't create arm triple");
+
+        let target = isa::lookup(triple)
+            .ok()
+            .map(|b| b.finish(shared_flags))
+            .expect("This test requires arm support.");
+
+        let mut sig = Signature::new(target.default_call_conv());
+        sig.returns.push(AbiParam::new(I32));
+
+        let mut fn_ctx = FunctionBuilderContext::new();
+        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+        {
+            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
+
+            let block0 = builder.create_ebb();
+            let y = Variable::new(16);
+            builder.declare_var(y, target.pointer_type());
+            builder.append_ebb_params_for_function_params(block0);
+            builder.switch_to_block(block0);
+
+            let dest = builder.use_var(y);
+            let size = 8192;
+            builder.emit_small_memset(target.frontend_config(), dest, 1, size, 8);
+            builder.ins().return_(&[dest]);
+
+            builder.seal_all_blocks();
+            builder.finalize();
+        }
+
+        assert_eq!(
+            func.display(None).to_string(),
+            "function %sample() -> i32 system_v {
+    sig0 = (i32, i32, i32) system_v
+    fn0 = %Memset sig0
+
+ebb0:
+    v4 = iconst.i32 0
+    v0 -> v4
+    v1 = iconst.i32 1
+    v2 = iconst.i32 8192
+    v3 = uextend.i32 v1
+    call fn0(v0, v3, v2)
+    return v0
+}
+"
+        );
+    }
+
     #[test]
     fn test_greatest_divisible_power_of_two() {
         assert_eq!(64, greatest_divisible_power_of_two(64));

From b4608c02c7b1217782708af0a64b0ee94e706b3a Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Tue, 5 Mar 2019 08:20:21 +0100
Subject: [PATCH 2337/3084] Fixed test

---
 cranelift/frontend/src/frontend.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 02b1043b0a..cefdb5bfe7 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -1237,7 +1237,7 @@ ebb0:
 ebb0:
     v4 = iconst.i32 0
     v0 -> v4
-    v1 = iconst.i32 1
+    v1 = iconst.i8 1
     v2 = iconst.i32 8192
     v3 = uextend.i32 v1
     call fn0(v0, v3, v2)

From 34aba7fe6664b5f06535c7d3caffca8f7b7a5be7 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 5 Mar 2019 06:36:34 -0800
Subject: [PATCH 2338/3084] Bump version to 0.29.0

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    |  8 ++++----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 650551c380..f72510e956 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.28.0"
+version = "0.29.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.28.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.28.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.28.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.28.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.28.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.28.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.28.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.28.0" }
-cranelift-module = { path = "cranelift-module", version = "0.28.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.28.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.28.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.28.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.28.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.29.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.29.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.29.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.29.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.29.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.29.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.29.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.29.0" }
+cranelift-module = { path = "cranelift-module", version = "0.29.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.29.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.29.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.29.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.29.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index b650e9cc83..f8f1d957fd 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.28.0"
+version = "0.29.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.28.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.29.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 8ffee131fb..c327308cb0 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.28.0"
+version = "0.29.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.28.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.28.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.29.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.29.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -26,7 +26,7 @@ log = { version = "0.4.6", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.28.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.29.0" }
 
 [features]
 default = ["std", "x86", "arm32", "arm64", "riscv"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 0cb6b40abc..1253c89e37 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.28.0"
+version = "0.29.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.28.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.29.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 994670997a..0b87d23b81 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.28.0"
+version = "0.29.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index c0f1f95bd1..77c329f1f8 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.28.0"
+version = "0.29.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.28.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.29.0" }
 faerie = "0.8.0"
 goblin = "0.0.21"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index c89da76f84..77a82d46e8 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.28.0"
+version = "0.29.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../cranelift-reader", version = "0.28.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.28.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../cranelift-reader", version = "0.29.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.29.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 4ea74a9b13..c2f0870378 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.28.0"
+version = "0.29.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index ac95debee2..d209ce86dd 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.28.0"
+version = "0.29.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.28.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.29.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 73961a7ddc..49e04a9163 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.28.0"
+version = "0.29.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
 target-lexicon = { version = "0.2.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 6a050d984a..3cee488618 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.28.0"
+version = "0.29.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.28.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.29.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 918bcd59ce..d893361501 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.28.0"
+version="0.29.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index a3ddb9c835..111f24ff46 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.28.0"
+version = "0.29.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0" }
 target-lexicon = "0.2.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index c88f646195..90253d4b30 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.28.0"
+version = "0.29.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.28.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.29.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index c4042006fa..b663148cb5 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.28.0"
+version = "0.29.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.28.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.28.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.29.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.29.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.2.0" }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.28.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.28.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.28.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.29.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.29.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.29.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 6d729708a8..b0138952e9 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.28.0"
+version = "0.29.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.28.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.29.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index fb0dc9a752..9be8c2a9aa 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.28.0"
+version = "0.29.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.23.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.28.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.28.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.28.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.29.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.29.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 75312a645613b5f6f31d7194549a3e90720e6037 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Thu, 7 Mar 2019 19:02:24 +0100
Subject: [PATCH 2339/3084] Update faerie to 0.9.1

---
 cranelift/Cargo.toml           | 2 +-
 cranelift/codegen/Cargo.toml   | 2 +-
 cranelift/faerie/Cargo.toml    | 4 ++--
 cranelift/frontend/Cargo.toml  | 2 +-
 cranelift/fuzz/Cargo.toml      | 2 +-
 cranelift/native/Cargo.toml    | 2 +-
 cranelift/reader/Cargo.toml    | 2 +-
 cranelift/simplejit/Cargo.toml | 2 +-
 cranelift/wasm/Cargo.toml      | 2 +-
 9 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index f72510e956..f6d3384286 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -38,7 +38,7 @@ serde = "1.0.8"
 term = "0.5.1"
 capstone = { version = "0.5.0", optional = true }
 wabt = { version = "0.7.0", optional = true }
-target-lexicon = "0.2.0"
+target-lexicon = "0.3.0"
 pretty_env_logger = "0.3.0"
 file-per-thread-logger = "0.1.2"
 
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index c327308cb0..8d906d0de7 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -18,7 +18,7 @@ cranelift-bforest = { path = "../cranelift-bforest", version = "0.29.0", default
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
-target-lexicon = { version = "0.2.0", default-features = false }
+target-lexicon = { version = "0.3.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 77c329f1f8..3355788fd4 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -12,10 +12,10 @@ edition = "2018"
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0" }
 cranelift-module = { path = "../cranelift-module", version = "0.29.0" }
-faerie = "0.8.0"
+faerie = "0.9.1"
 goblin = "0.0.21"
 failure = "0.1.2"
-target-lexicon = "0.2.0"
+target-lexicon = "0.3.0"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index c2f0870378..9cec2f2877 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -12,7 +12,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
-target-lexicon = { version = "0.2.0", default-features = false }
+target-lexicon = { version = "0.3.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 
diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
index d064d0bb81..0a2acc410f 100644
--- a/cranelift/fuzz/Cargo.toml
+++ b/cranelift/fuzz/Cargo.toml
@@ -15,7 +15,7 @@ libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" }
 cranelift-codegen = { path = "../cranelift-codegen" }
 cranelift-wasm = { path = "../cranelift-wasm" }
 cranelift-reader = { path = "../cranelift-reader" }
-target-lexicon = "0.2.0"
+target-lexicon = "0.3.0"
 
 # Prevent this from interfering with workspaces
 [workspace]
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 49e04a9163..b5ba3f20b2 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -11,7 +11,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
-target-lexicon = { version = "0.2.0", default-features = false }
+target-lexicon = { version = "0.3.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
 raw-cpuid = "6.0.0"
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 111f24ff46..6445590442 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -11,7 +11,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0" }
-target-lexicon = "0.2.0"
+target-lexicon = "0.3.0"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index b663148cb5..9ed2402683 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -16,7 +16,7 @@ cranelift-native = { path = "../cranelift-native", version = "0.29.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
-target-lexicon = { version = "0.2.0" }
+target-lexicon = { version = "0.3.0" }
 
 [target.'cfg(target_os = "windows")'.dependencies]
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 9be8c2a9aa..0462fc8a68 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -23,7 +23,7 @@ cast = { version = "0.2.2", default-features = false }
 
 [dev-dependencies]
 wabt = "0.7.0"
-target-lexicon = "0.2.0"
+target-lexicon = "0.3.0"
 
 [features]
 default = ["std"]

From ebf8bc33e8c56659ff03702bce0c50e1dfcdbba8 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Fri, 8 Mar 2019 08:06:09 +0100
Subject: [PATCH 2340/3084] Rustup stable to 1.32.0

---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index bc772c38a2..1540cd5407 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,7 +7,7 @@ language: rust
 rust:
     # The oldest version we currently support. See
     # CONTRIBUTING.md#rustc-version-support for details.
-    - 1.31.1
+    - 1.32.0
     - beta
     - nightly
 matrix:

From 00cfe62380cda6fd45dff0a8a578ac4e25d46d25 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Fri, 8 Mar 2019 16:39:17 +0100
Subject: [PATCH 2341/3084] Update README.md

---
 cranelift/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index 51aefca247..166ad1a1b8 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -53,7 +53,7 @@ needed before it would be ready for a production use case.
 
 Cranelift's APIs are not yet stable.
 
-Cranelift currently requires Rust 1.30 or later, and Python 2.7 or 3
+Cranelift currently requires Rust 1.32 or later, and Python 2.7 or 3
 to build.
 
 Planned uses

From 92b3987e547c38e89ed7d7933b430fe35401cd92 Mon Sep 17 00:00:00 2001
From: Steffen Butzer 
Date: Sat, 9 Mar 2019 21:27:55 +0100
Subject: [PATCH 2342/3084] windows/x64 call convention: only use XMM0 for
 float return values (#691)

---
 cranelift/codegen/src/isa/x86/abi.rs                     | 9 +++++----
 .../filetests/isa/x86/windows_fastcall_x64.clif          | 8 +++++++-
 2 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index b25c17e1f6..37d9ffb257 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -161,13 +161,14 @@ pub fn legalize_signature(sig: &mut ir::Signature, triple: &Triple, _current: bo
 
     legalize_args(&mut sig.params, &mut args);
 
-    let regs = if sig.call_conv == CallConv::WindowsFastcall {
-        &RET_GPRS_WIN_FASTCALL_X64[..]
+    let (regs, fpr_limit) = if sig.call_conv == CallConv::WindowsFastcall {
+        // windows-x64 calling convention only uses XMM0 or RAX for return values
+        (&RET_GPRS_WIN_FASTCALL_X64[..], 1)
     } else {
-        &RET_GPRS[..]
+        (&RET_GPRS[..], 2)
     };
 
-    let mut rets = Args::new(bits, regs, 2, sig.call_conv);
+    let mut rets = Args::new(bits, regs, fpr_limit, sig.call_conv);
     legalize_args(&mut sig.returns, &mut rets);
 }
 
diff --git a/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
index a84c17fb9e..14b46e579f 100644
--- a/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
+++ b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
@@ -36,4 +36,10 @@ function %mixed_int_float(i64, f64, i64, f32) windows_fastcall {
 ebb0(v0: i64, v1: f64, v2: i64, v3: f32):
     return
 }
-; check: function %mixed_int_float(i64 [%rcx], f64 [%xmm1], i64 [%r8], f32 [%xmm3], i64 fp [%rbp]) -> i64 fp [%rbp] windows_fastcall {
\ No newline at end of file
+; check: function %mixed_int_float(i64 [%rcx], f64 [%xmm1], i64 [%r8], f32 [%xmm3], i64 fp [%rbp]) -> i64 fp [%rbp] windows_fastcall {
+
+function %ret_val_float(f32, f64, i64, i64) -> f64 windows_fastcall {
+ebb0(v0: f32, v1: f64, v2: i64, v3: i64):
+    return v1
+}
+; check: function %ret_val_float(f32 [%xmm0], f64 [%xmm1], i64 [%r8], i64 [%r9], i64 fp [%rbp]) -> f64 [%xmm0], i64 fp [%rbp] windows_fastcall {

From b7dfe8aaa63119717e2b89ee965b093f24b72bdb Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Mar 2019 20:12:15 +0100
Subject: [PATCH 2343/3084] Fix build warnings for cranelift-codegen tests;

---
 cranelift/codegen/src/divconst_magic_numbers.rs | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/src/divconst_magic_numbers.rs b/cranelift/codegen/src/divconst_magic_numbers.rs
index ebf1ae3f12..30d8287747 100644
--- a/cranelift/codegen/src/divconst_magic_numbers.rs
+++ b/cranelift/codegen/src/divconst_magic_numbers.rs
@@ -246,7 +246,7 @@ mod tests {
     }
 
     #[test]
-    fn test_magicU32() {
+    fn test_magic_u32() {
         assert_eq!(magic_u32(2u32), make_mu32(0x80000000u32, false, 0));
         assert_eq!(magic_u32(3u32), make_mu32(0xaaaaaaabu32, false, 1));
         assert_eq!(magic_u32(4u32), make_mu32(0x40000000u32, false, 0));
@@ -279,8 +279,9 @@ mod tests {
             make_mu32(0x80000001u32, false, 31)
         );
     }
+
     #[test]
-    fn test_magicU64() {
+    fn test_magic_u64() {
         assert_eq!(magic_u64(2u64), make_mu64(0x8000000000000000u64, false, 0));
         assert_eq!(magic_u64(3u64), make_mu64(0xaaaaaaaaaaaaaaabu64, false, 1));
         assert_eq!(magic_u64(4u64), make_mu64(0x4000000000000000u64, false, 0));
@@ -346,8 +347,9 @@ mod tests {
             make_mu64(0x8000000000000001u64, false, 63)
         );
     }
+
     #[test]
-    fn test_magicS32() {
+    fn test_magic_s32() {
         assert_eq!(
             magic_s32(-0x80000000i32),
             make_ms32(0x7fffffffu32 as i32, 30)
@@ -390,8 +392,9 @@ mod tests {
             make_ms32(0x40000001u32 as i32, 29)
         );
     }
+
     #[test]
-    fn test_magicS64() {
+    fn test_magic_s64() {
         assert_eq!(
             magic_s64(-0x8000000000000000i64),
             make_ms64(0x7fffffffffffffffu64 as i64, 62)
@@ -515,6 +518,7 @@ mod tests {
             make_ms64(0x4000000000000001u64 as i64, 61)
         );
     }
+
     #[test]
     fn test_magic_generators_dont_panic() {
         // The point of this is to check that the magic number generators

From 5e12f4f2b843f0dda601ef7c2dacfb6dfa8c4dae Mon Sep 17 00:00:00 2001
From: Yury Delendik 
Date: Tue, 22 Jan 2019 18:55:44 -0600
Subject: [PATCH 2344/3084] Add missing use_srcloc in postopt and regalloc
 passes

---
 cranelift/codegen/src/postopt.rs             | 1 +
 cranelift/codegen/src/regalloc/coalescing.rs | 4 ++++
 cranelift/codegen/src/regalloc/spilling.rs   | 1 +
 3 files changed, 6 insertions(+)

diff --git a/cranelift/codegen/src/postopt.rs b/cranelift/codegen/src/postopt.rs
index 591b92ed03..a82a3e59d3 100644
--- a/cranelift/codegen/src/postopt.rs
+++ b/cranelift/codegen/src/postopt.rs
@@ -128,6 +128,7 @@ fn optimize_cpu_flags(
     // We found a compare+branch pattern. Transform it to use flags.
     let args = info.args.as_slice(&pos.func.dfg.value_lists)[1..].to_vec();
     pos.goto_inst(info.cmp_inst);
+    pos.use_srcloc(info.cmp_inst);
     match info.kind {
         CmpBrKind::Icmp { mut cond, arg } => {
             let flags = pos.ins().ifcmp(info.cmp_arg, arg);
diff --git a/cranelift/codegen/src/regalloc/coalescing.rs b/cranelift/codegen/src/regalloc/coalescing.rs
index f59435d170..401b795eea 100644
--- a/cranelift/codegen/src/regalloc/coalescing.rs
+++ b/cranelift/codegen/src/regalloc/coalescing.rs
@@ -292,6 +292,9 @@ impl<'a> Context<'a> {
 
         // Insert a copy instruction at the top of `ebb`.
         let mut pos = EncCursor::new(self.func, self.isa).at_first_inst(ebb);
+        if let Some(inst) = pos.current_inst() {
+            pos.use_srcloc(inst);
+        }
         pos.ins().with_result(param).copy(new_val);
         let inst = pos.built_inst();
         self.liveness.move_def_locally(param, inst);
@@ -347,6 +350,7 @@ impl<'a> Context<'a> {
         pred_val: Value,
     ) -> Value {
         let mut pos = EncCursor::new(self.func, self.isa).at_inst(pred_inst);
+        pos.use_srcloc(pred_inst);
         let copy = pos.ins().copy(pred_val);
         let inst = pos.built_inst();
 
diff --git a/cranelift/codegen/src/regalloc/spilling.rs b/cranelift/codegen/src/regalloc/spilling.rs
index 61458f4066..9065566003 100644
--- a/cranelift/codegen/src/regalloc/spilling.rs
+++ b/cranelift/codegen/src/regalloc/spilling.rs
@@ -420,6 +420,7 @@ impl<'a> Context<'a> {
         // secondary `opidx` key makes it possible to use an unstable (non-allocating) sort.
         self.reg_uses.sort_unstable_by_key(|u| (u.value, u.opidx));
 
+        self.cur.use_srcloc(inst);
         for i in 0..self.reg_uses.len() {
             let ru = self.reg_uses[i];
 

From 45efbf797385a5730dfe317c2bfc0c5a81b0eaba Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Tue, 26 Mar 2019 11:04:02 +0100
Subject: [PATCH 2345/3084] Optimize switch codegen and improve docs (#712)

---
 cranelift/frontend/src/switch.rs | 141 +++++++++++++++++++++++++------
 1 file changed, 114 insertions(+), 27 deletions(-)

diff --git a/cranelift/frontend/src/switch.rs b/cranelift/frontend/src/switch.rs
index 653afbaed1..0db3ac348e 100644
--- a/cranelift/frontend/src/switch.rs
+++ b/cranelift/frontend/src/switch.rs
@@ -9,6 +9,36 @@ type EntryIndex = u64;
 
 /// Unlike with `br_table`, `Switch` cases may be sparse or non-0-based.
 /// They emit efficient code using branches, jump tables, or a combination of both.
+///
+/// # Example
+///
+/// ```rust
+/// # use cranelift_codegen::ir::types::*;
+/// # use cranelift_codegen::ir::{ExternalName, Function, Signature, InstBuilder};
+/// # use cranelift_codegen::isa::CallConv;
+/// # use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Switch};
+/// #
+/// # let mut sig = Signature::new(CallConv::SystemV);
+/// # let mut fn_builder_ctx = FunctionBuilderContext::new();
+/// # let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
+/// # let mut builder = FunctionBuilder::new(&mut func, &mut fn_builder_ctx);
+/// #
+/// # let entry = builder.create_ebb();
+/// # builder.switch_to_block(entry);
+/// #
+/// let block0 = builder.create_ebb();
+/// let block1 = builder.create_ebb();
+/// let block2 = builder.create_ebb();
+/// let fallback = builder.create_ebb();
+///
+/// let val = builder.ins().iconst(I32, 1);
+///
+/// let mut switch = Switch::new();
+/// switch.set_entry(0, block0);
+/// switch.set_entry(1, block1);
+/// switch.set_entry(7, block2);
+/// switch.emit(&mut builder, val, fallback);
+/// ```
 #[derive(Debug, Default)]
 pub struct Switch {
     cases: HashMap,
@@ -32,6 +62,14 @@ impl Switch {
         );
     }
 
+    /// Turn the `cases` `HashMap` into a list of `ContiguousCaseRange`s.
+    ///
+    /// # Postconditions
+    ///
+    /// * Every entry will be represented.
+    /// * The `ContiguousCaseRange`s will not overlap.
+    /// * Between two `ContiguousCaseRange`s there will be at least one entry index.
+    /// * No `ContiguousCaseRange`s will be empty.
     fn collect_contiguous_case_ranges(self) -> Vec {
         debug!("build_contiguous_case_ranges before: {:#?}", self.cases);
         let mut cases = self.cases.into_iter().collect::>();
@@ -60,6 +98,7 @@ impl Switch {
         contiguous_case_ranges
     }
 
+    /// Binary search for the right `ContiguousCaseRange`.
     fn build_search_tree(
         bx: &mut FunctionBuilder,
         val: Value,
@@ -120,6 +159,7 @@ impl Switch {
         cases_and_jt_ebbs
     }
 
+    /// Linear search for the right `ContiguousCaseRange`.
     fn build_search_branches(
         bx: &mut FunctionBuilder,
         val: Value,
@@ -128,22 +168,41 @@ impl Switch {
         cases_and_jt_ebbs: &mut Vec<(EntryIndex, Ebb, Vec)>,
     ) {
         for ContiguousCaseRange { first_index, ebbs } in contiguous_case_ranges.into_iter().rev() {
-            if ebbs.len() == 1 {
-                let is_good_val = bx.ins().icmp_imm(IntCC::Equal, val, first_index as i64);
-                bx.ins().brnz(is_good_val, ebbs[0], &[]);
-            } else {
-                let jt_ebb = bx.create_ebb();
-                let is_good_val =
-                    bx.ins()
-                        .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, val, first_index as i64);
-                bx.ins().brnz(is_good_val, jt_ebb, &[]);
-                cases_and_jt_ebbs.push((first_index, jt_ebb, ebbs));
+            match (ebbs.len(), first_index) {
+                (1, 0) => {
+                    bx.ins().brz(val, ebbs[0], &[]);
+                }
+                (1, _) => {
+                    let is_good_val = bx.ins().icmp_imm(IntCC::Equal, val, first_index as i64);
+                    bx.ins().brnz(is_good_val, ebbs[0], &[]);
+                }
+                (_, 0) => {
+                    // if `first_index` is 0, then `icmp_imm uge val, first_index` is trivially true
+                    let jt_ebb = bx.create_ebb();
+                    bx.ins().jump(jt_ebb, &[]);
+                    cases_and_jt_ebbs.push((first_index, jt_ebb, ebbs));
+                    // `jump otherwise` below must not be hit, because the current block has been
+                    // filled above. This is the last iteration anyway, as 0 is the smallest
+                    // unsigned int, so just return here.
+                    return;
+                }
+                (_, _) => {
+                    let jt_ebb = bx.create_ebb();
+                    let is_good_val = bx.ins().icmp_imm(
+                        IntCC::UnsignedGreaterThanOrEqual,
+                        val,
+                        first_index as i64,
+                    );
+                    bx.ins().brnz(is_good_val, jt_ebb, &[]);
+                    cases_and_jt_ebbs.push((first_index, jt_ebb, ebbs));
+                }
             }
         }
 
         bx.ins().jump(otherwise, &[]);
     }
 
+    /// For every item in `cases_and_jt_ebbs` this will create a jump table in the specified ebb.
     fn build_jump_tables(
         bx: &mut FunctionBuilder,
         val: Value,
@@ -158,7 +217,11 @@ impl Switch {
             let jump_table = bx.create_jump_table(jt_data);
 
             bx.switch_to_block(jt_ebb);
-            let discr = bx.ins().iadd_imm(val, (first_index as i64).wrapping_neg());
+            let discr = if first_index == 0 {
+                val
+            } else {
+                bx.ins().iadd_imm(val, (first_index as i64).wrapping_neg())
+            };
             bx.ins().br_table(discr, otherwise, jump_table);
         }
     }
@@ -183,9 +246,22 @@ impl Switch {
     }
 }
 
+/// This represents a contiguous range of cases to switch on.
+///
+/// For example 10 => ebb1, 11 => ebb2, 12 => ebb7 will be represented as:
+///
+/// ```plain
+/// ContiguousCaseRange {
+///     first_index: 10,
+///     ebbs: vec![Ebb::from_u32(1), Ebb::from_u32(2), Ebb::from_u32(7)]
+/// }
+/// ```
 #[derive(Debug)]
 struct ContiguousCaseRange {
+    /// The entry index of the first case. Eg. 10 when the entry indexes are 10, 11, 12 and 13.
     first_index: EntryIndex,
+
+    /// The ebbs to jump to sorted in ascending order of entry index.
     ebbs: Vec,
 }
 
@@ -237,8 +313,7 @@ mod tests {
             "ebb0:
     v0 = iconst.i8 0
     v1 = uextend.i32 v0
-    v2 = icmp_imm eq v1, 0
-    brnz v2, ebb1
+    brz v1, ebb1
     jump ebb0"
         );
     }
@@ -267,13 +342,10 @@ mod tests {
 ebb0:
     v0 = iconst.i8 0
     v1 = uextend.i32 v0
-    v2 = icmp_imm uge v1, 0
-    brnz v2, ebb3
-    jump ebb0
+    jump ebb3
 
 ebb3:
-    v3 = iadd_imm.i32 v1, 0
-    br_table v3, ebb0, jt0"
+    br_table.i32 v1, ebb0, jt0"
         );
     }
 
@@ -287,8 +359,7 @@ ebb3:
     v1 = uextend.i32 v0
     v2 = icmp_imm eq v1, 2
     brnz v2, ebb2
-    v3 = icmp_imm eq v1, 0
-    brnz v3, ebb1
+    brz v1, ebb1
     jump ebb0"
         );
     }
@@ -318,17 +389,14 @@ ebb9:
 ebb8:
     v5 = icmp_imm.i32 eq v1, 5
     brnz v5, ebb3
-    v6 = icmp_imm.i32 uge v1, 0
-    brnz v6, ebb11
-    jump ebb0
+    jump ebb11
 
 ebb11:
-    v7 = iadd_imm.i32 v1, 0
-    br_table v7, ebb0, jt0
+    br_table.i32 v1, ebb0, jt0
 
 ebb10:
-    v8 = iadd_imm.i32 v1, -10
-    br_table v8, ebb0, jt1"
+    v6 = iadd_imm.i32 v1, -10
+    br_table v6, ebb0, jt1"
         );
     }
 
@@ -363,4 +431,23 @@ ebb10:
     jump ebb0"
         )
     }
+
+    #[test]
+    fn switch_optimal_codegen() {
+        let func = setup!(0, [-1i64 as u64, 0, 1,]);
+        assert_eq!(
+            func,
+            "    jt0 = jump_table [ebb2, ebb3]
+
+ebb0:
+    v0 = iconst.i8 0
+    v1 = uextend.i32 v0
+    v2 = icmp_imm eq v1, -1
+    brnz v2, ebb1
+    jump ebb4
+
+ebb4:
+    br_table.i32 v1, ebb0, jt0"
+        );
+    }
 }

From 975b7f2e8d5e9cd0b92ecde8a58d232edb1c284c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 26 Mar 2019 08:40:06 -0700
Subject: [PATCH 2346/3084] Put a space between "//!" and "```".

---
 cranelift/codegen/src/legalizer/split.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/src/legalizer/split.rs b/cranelift/codegen/src/legalizer/split.rs
index 67d947f4a9..773df13216 100644
--- a/cranelift/codegen/src/legalizer/split.rs
+++ b/cranelift/codegen/src/legalizer/split.rs
@@ -14,13 +14,13 @@
 //!
 //! When legalizing a single instruction, it is wrapped in splits and concatenations:
 //!
-//!```clif
+//! ```clif
 //!     v1 = bxor.i64 v2, v3
 //! ```
 //!
 //! becomes:
 //!
-//!```clif
+//! ```clif
 //!     v20, v21 = isplit v2
 //!     v30, v31 = isplit v3
 //!     v10 = bxor.i32 v20, v30

From 6b85df016894a8c1fa944130f93631e2c784396c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 26 Mar 2019 09:06:41 -0700
Subject: [PATCH 2347/3084] Update to wasmparser 0.29.2.

---
 cranelift/wasm/Cargo.toml                 |   2 +-
 cranelift/wasm/src/code_translator.rs     | 155 ++++++++++++++++++++++
 cranelift/wasm/src/sections_translator.rs | 109 ++++++++-------
 3 files changed, 214 insertions(+), 52 deletions(-)

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 0462fc8a68..a0ed9fb37c 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -11,7 +11,7 @@ keywords = ["webassembly", "wasm"]
 edition = "2018"
 
 [dependencies]
-wasmparser = { version = "0.23.0", default-features = false }
+wasmparser = { version = "0.29.2", default-features = false }
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.29.0", default-features = false }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.29.0", default-features = false }
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index ad0d75e2da..f83b462cef 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -886,6 +886,161 @@ pub fn translate_operator(
         Operator::RefNull | Operator::RefIsNull { .. } => {
             return Err(WasmError::Unsupported("proposed reference-type operators"));
         }
+        Operator::MemoryInit { .. }
+        | Operator::DataDrop { .. }
+        | Operator::MemoryCopy
+        | Operator::MemoryFill
+        | Operator::TableInit { .. }
+        | Operator::ElemDrop { .. }
+        | Operator::TableCopy
+        | Operator::TableGet { .. }
+        | Operator::TableSet { .. }
+        | Operator::TableGrow { .. }
+        | Operator::TableSize { .. } => {
+            return Err(WasmError::Unsupported("proposed bulk memory operators"));
+        }
+        Operator::V128Load { .. }
+        | Operator::V128Store { .. }
+        | Operator::V128Const { .. }
+        | Operator::V8x16Shuffle { .. }
+        | Operator::I8x16Splat
+        | Operator::I8x16ExtractLaneS { .. }
+        | Operator::I8x16ExtractLaneU { .. }
+        | Operator::I8x16ReplaceLane { .. }
+        | Operator::I16x8Splat
+        | Operator::I16x8ExtractLaneS { .. }
+        | Operator::I16x8ExtractLaneU { .. }
+        | Operator::I16x8ReplaceLane { .. }
+        | Operator::I32x4Splat
+        | Operator::I32x4ExtractLane { .. }
+        | Operator::I32x4ReplaceLane { .. }
+        | Operator::I64x2Splat
+        | Operator::I64x2ExtractLane { .. }
+        | Operator::I64x2ReplaceLane { .. }
+        | Operator::F32x4Splat
+        | Operator::F32x4ExtractLane { .. }
+        | Operator::F32x4ReplaceLane { .. }
+        | Operator::F64x2Splat
+        | Operator::F64x2ExtractLane { .. }
+        | Operator::F64x2ReplaceLane { .. }
+        | Operator::I8x16Eq
+        | Operator::I8x16Ne
+        | Operator::I8x16LtS
+        | Operator::I8x16LtU
+        | Operator::I8x16GtS
+        | Operator::I8x16GtU
+        | Operator::I8x16LeS
+        | Operator::I8x16LeU
+        | Operator::I8x16GeS
+        | Operator::I8x16GeU
+        | Operator::I16x8Eq
+        | Operator::I16x8Ne
+        | Operator::I16x8LtS
+        | Operator::I16x8LtU
+        | Operator::I16x8GtS
+        | Operator::I16x8GtU
+        | Operator::I16x8LeS
+        | Operator::I16x8LeU
+        | Operator::I16x8GeS
+        | Operator::I16x8GeU
+        | Operator::I32x4Eq
+        | Operator::I32x4Ne
+        | Operator::I32x4LtS
+        | Operator::I32x4LtU
+        | Operator::I32x4GtS
+        | Operator::I32x4GtU
+        | Operator::I32x4LeS
+        | Operator::I32x4LeU
+        | Operator::I32x4GeS
+        | Operator::I32x4GeU
+        | Operator::F32x4Eq
+        | Operator::F32x4Ne
+        | Operator::F32x4Lt
+        | Operator::F32x4Gt
+        | Operator::F32x4Le
+        | Operator::F32x4Ge
+        | Operator::F64x2Eq
+        | Operator::F64x2Ne
+        | Operator::F64x2Lt
+        | Operator::F64x2Gt
+        | Operator::F64x2Le
+        | Operator::F64x2Ge
+        | Operator::V128Not
+        | Operator::V128And
+        | Operator::V128Or
+        | Operator::V128Xor
+        | Operator::V128Bitselect
+        | Operator::I8x16Neg
+        | Operator::I8x16AnyTrue
+        | Operator::I8x16AllTrue
+        | Operator::I8x16Shl
+        | Operator::I8x16ShrS
+        | Operator::I8x16ShrU
+        | Operator::I8x16Add
+        | Operator::I8x16AddSaturateS
+        | Operator::I8x16AddSaturateU
+        | Operator::I8x16Sub
+        | Operator::I8x16SubSaturateS
+        | Operator::I8x16SubSaturateU
+        | Operator::I8x16Mul
+        | Operator::I16x8Neg
+        | Operator::I16x8AnyTrue
+        | Operator::I16x8AllTrue
+        | Operator::I16x8Shl
+        | Operator::I16x8ShrS
+        | Operator::I16x8ShrU
+        | Operator::I16x8Add
+        | Operator::I16x8AddSaturateS
+        | Operator::I16x8AddSaturateU
+        | Operator::I16x8Sub
+        | Operator::I16x8SubSaturateS
+        | Operator::I16x8SubSaturateU
+        | Operator::I16x8Mul
+        | Operator::I32x4Neg
+        | Operator::I32x4AnyTrue
+        | Operator::I32x4AllTrue
+        | Operator::I32x4Shl
+        | Operator::I32x4ShrS
+        | Operator::I32x4ShrU
+        | Operator::I32x4Add
+        | Operator::I32x4Sub
+        | Operator::I32x4Mul
+        | Operator::I64x2Neg
+        | Operator::I64x2AnyTrue
+        | Operator::I64x2AllTrue
+        | Operator::I64x2Shl
+        | Operator::I64x2ShrS
+        | Operator::I64x2ShrU
+        | Operator::I64x2Add
+        | Operator::I64x2Sub
+        | Operator::F32x4Abs
+        | Operator::F32x4Neg
+        | Operator::F32x4Sqrt
+        | Operator::F32x4Add
+        | Operator::F32x4Sub
+        | Operator::F32x4Mul
+        | Operator::F32x4Div
+        | Operator::F32x4Min
+        | Operator::F32x4Max
+        | Operator::F64x2Abs
+        | Operator::F64x2Neg
+        | Operator::F64x2Sqrt
+        | Operator::F64x2Add
+        | Operator::F64x2Sub
+        | Operator::F64x2Mul
+        | Operator::F64x2Div
+        | Operator::F64x2Min
+        | Operator::F64x2Max
+        | Operator::I32x4TruncSF32x4Sat
+        | Operator::I32x4TruncUF32x4Sat
+        | Operator::I64x2TruncSF64x2Sat
+        | Operator::I64x2TruncUF64x2Sat
+        | Operator::F32x4ConvertSI32x4
+        | Operator::F32x4ConvertUI32x4
+        | Operator::F64x2ConvertSI64x2
+        | Operator::F64x2ConvertUI64x2 => {
+            return Err(WasmError::Unsupported("proposed SIMD operators"));
+        }
     };
     Ok(())
 }
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index 37c5d34672..e406ec3eee 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -12,15 +12,15 @@ use crate::translation_utils::{
     type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex,
     Table, TableElementType, TableIndex,
 };
-use core::str::from_utf8;
 use cranelift_codegen::ir::{self, AbiParam, Signature};
 use cranelift_entity::EntityRef;
 use std::vec::Vec;
 use wasmparser::{
-    self, CodeSectionReader, Data, DataSectionReader, Element, ElementSectionReader, Export,
-    ExportSectionReader, ExternalKind, FuncType, FunctionSectionReader, GlobalSectionReader,
-    GlobalType, ImportSectionEntryType, ImportSectionReader, MemorySectionReader, MemoryType,
-    Operator, TableSectionReader, TypeSectionReader,
+    self, CodeSectionReader, Data, DataKind, DataSectionReader, Element, ElementKind,
+    ElementSectionReader, Export, ExportSectionReader, ExternalKind, FuncType,
+    FunctionSectionReader, GlobalSectionReader, GlobalType, ImportSectionEntryType,
+    ImportSectionReader, MemorySectionReader, MemoryType, Operator, TableSectionReader,
+    TypeSectionReader,
 };
 
 /// Parses the Type section of the wasm module.
@@ -65,12 +65,8 @@ pub fn parse_import_section<'data>(
 
     for entry in imports {
         let import = entry?;
-
-        // The input has already been validated, so we should be able to
-        // assume valid UTF-8 and use `from_utf8_unchecked` if performance
-        // becomes a concern here.
-        let module_name = from_utf8(import.module).unwrap();
-        let field_name = from_utf8(import.field).unwrap();
+        let module_name = import.module;
+        let field_name = import.field;
 
         match import.ty {
             ImportSectionEntryType::Function(sig) => {
@@ -232,13 +228,12 @@ pub fn parse_export_section<'data>(
         // The input has already been validated, so we should be able to
         // assume valid UTF-8 and use `from_utf8_unchecked` if performance
         // becomes a concern here.
-        let name = from_utf8(field).unwrap();
         let index = index as usize;
         match *kind {
-            ExternalKind::Function => environ.declare_func_export(FuncIndex::new(index), name),
-            ExternalKind::Table => environ.declare_table_export(TableIndex::new(index), name),
-            ExternalKind::Memory => environ.declare_memory_export(MemoryIndex::new(index), name),
-            ExternalKind::Global => environ.declare_global_export(GlobalIndex::new(index), name),
+            ExternalKind::Function => environ.declare_func_export(FuncIndex::new(index), field),
+            ExternalKind::Table => environ.declare_table_export(TableIndex::new(index), field),
+            ExternalKind::Memory => environ.declare_memory_export(MemoryIndex::new(index), field),
+            ExternalKind::Global => environ.declare_global_export(GlobalIndex::new(index), field),
         }
     }
 
@@ -260,29 +255,35 @@ pub fn parse_element_section<'data>(
     environ.reserve_table_elements(elements.get_count());
 
     for entry in elements {
-        let Element {
+        let Element { kind, items } = entry?;
+        if let ElementKind::Active {
             table_index,
             init_expr,
-            items,
-        } = entry?;
-        let mut init_expr_reader = init_expr.get_binary_reader();
-        let (base, offset) = match init_expr_reader.read_operator()? {
-            Operator::I32Const { value } => (None, value as u32 as usize),
-            Operator::GetGlobal { global_index } => (Some(GlobalIndex::from_u32(global_index)), 0),
-            ref s => panic!("unsupported init expr in element section: {:?}", s),
-        };
-        let items_reader = items.get_items_reader()?;
-        let mut elems = Vec::with_capacity(cast::usize(items_reader.get_count()));
-        for item in items_reader {
-            let x = item?;
-            elems.push(FuncIndex::from_u32(x));
+        } = kind
+        {
+            let mut init_expr_reader = init_expr.get_binary_reader();
+            let (base, offset) = match init_expr_reader.read_operator()? {
+                Operator::I32Const { value } => (None, value as u32 as usize),
+                Operator::GetGlobal { global_index } => {
+                    (Some(GlobalIndex::from_u32(global_index)), 0)
+                }
+                ref s => panic!("unsupported init expr in element section: {:?}", s),
+            };
+            let items_reader = items.get_items_reader()?;
+            let mut elems = Vec::with_capacity(cast::usize(items_reader.get_count()));
+            for item in items_reader {
+                let x = item?;
+                elems.push(FuncIndex::from_u32(x));
+            }
+            environ.declare_table_elements(
+                TableIndex::from_u32(table_index),
+                base,
+                offset,
+                elems.into_boxed_slice(),
+            )
+        } else {
+            panic!("unsupported passive elements section");
         }
-        environ.declare_table_elements(
-            TableIndex::from_u32(table_index),
-            base,
-            offset,
-            elems.into_boxed_slice(),
-        )
     }
     Ok(())
 }
@@ -309,23 +310,29 @@ pub fn parse_data_section<'data>(
     environ.reserve_data_initializers(data.get_count());
 
     for entry in data {
-        let Data {
+        let Data { kind, data } = entry?;
+        if let DataKind::Active {
             memory_index,
             init_expr,
-            data,
-        } = entry?;
-        let mut init_expr_reader = init_expr.get_binary_reader();
-        let (base, offset) = match init_expr_reader.read_operator()? {
-            Operator::I32Const { value } => (None, value as u32 as usize),
-            Operator::GetGlobal { global_index } => (Some(GlobalIndex::from_u32(global_index)), 0),
-            ref s => panic!("unsupported init expr in data section: {:?}", s),
-        };
-        environ.declare_data_initialization(
-            MemoryIndex::from_u32(memory_index),
-            base,
-            offset,
-            data,
-        );
+        } = kind
+        {
+            let mut init_expr_reader = init_expr.get_binary_reader();
+            let (base, offset) = match init_expr_reader.read_operator()? {
+                Operator::I32Const { value } => (None, value as u32 as usize),
+                Operator::GetGlobal { global_index } => {
+                    (Some(GlobalIndex::from_u32(global_index)), 0)
+                }
+                ref s => panic!("unsupported init expr in data section: {:?}", s),
+            };
+            environ.declare_data_initialization(
+                MemoryIndex::from_u32(memory_index),
+                base,
+                offset,
+                data,
+            );
+        } else {
+            panic!("unsupported passive data section");
+        }
     }
 
     Ok(())

From 6b854381bbc339358e4a2dfe232cad3e5166ac4a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 26 Mar 2019 09:11:57 -0700
Subject: [PATCH 2348/3084] Fix a documentation-comment warning.

---
 cranelift/codegen/src/timing.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/timing.rs b/cranelift/codegen/src/timing.rs
index 16f729f71d..2a2d393803 100644
--- a/cranelift/codegen/src/timing.rs
+++ b/cranelift/codegen/src/timing.rs
@@ -176,7 +176,7 @@ mod details {
         }
     }
 
-    /// Information about passes in a single thread.
+    // Information about passes in a single thread.
     thread_local! {
         static CURRENT_PASS: Cell = Cell::new(Pass::None);
         static PASS_TIME: RefCell = RefCell::new(Default::default());

From ec0b10cd34c3cdd67ac058e3e62d518215903fa7 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 26 Mar 2019 09:36:22 -0700
Subject: [PATCH 2349/3084] Bump version to 0.30.0

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    |  8 ++++----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index f6d3384286..068326a646 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.29.0"
+version = "0.30.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.29.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.29.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.29.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.29.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.29.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.29.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.29.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.29.0" }
-cranelift-module = { path = "cranelift-module", version = "0.29.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.29.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.29.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.29.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.29.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.30.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.30.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.30.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.30.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.30.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.30.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.30.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.30.0" }
+cranelift-module = { path = "cranelift-module", version = "0.30.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.30.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.30.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.30.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.30.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index f8f1d957fd..d5bd3565e4 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.29.0"
+version = "0.30.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.29.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.30.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 8d906d0de7..0808b0fb65 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.29.0"
+version = "0.30.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.29.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.29.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.30.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.30.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -26,7 +26,7 @@ log = { version = "0.4.6", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.29.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.30.0" }
 
 [features]
 default = ["std", "x86", "arm32", "arm64", "riscv"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 1253c89e37..8c263d7ea0 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.29.0"
+version = "0.30.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.29.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.30.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 0b87d23b81..f050953a94 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.29.0"
+version = "0.30.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 3355788fd4..0abcdb8de1 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.29.0"
+version = "0.30.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.29.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.30.0" }
 faerie = "0.9.1"
 goblin = "0.0.21"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 77a82d46e8..dfd63ac0e6 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.29.0"
+version = "0.30.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../cranelift-reader", version = "0.29.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.29.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../cranelift-reader", version = "0.30.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.30.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 9cec2f2877..7d08cff58a 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.29.0"
+version = "0.30.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
 target-lexicon = { version = "0.3.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index d209ce86dd..9ac859d901 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.29.0"
+version = "0.30.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.29.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.30.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index b5ba3f20b2..ee692f289e 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.29.0"
+version = "0.30.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
 target-lexicon = { version = "0.3.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 3cee488618..9cf1ab4e57 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.29.0"
+version = "0.30.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.29.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.30.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index d893361501..f4805aee08 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.29.0"
+version="0.30.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 6445590442..c1602171b4 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.29.0"
+version = "0.30.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0" }
 target-lexicon = "0.3.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 90253d4b30..ab035ae3fe 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.29.0"
+version = "0.30.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.29.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.30.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 9ed2402683..646c96ffa9 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.29.0"
+version = "0.30.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.29.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.29.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.30.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.30.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.3.0" }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.29.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.29.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.29.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.30.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.30.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.30.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index b0138952e9..0528694a8f 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.29.0"
+version = "0.30.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.29.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.30.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index a0ed9fb37c..c39368eb2c 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.29.0"
+version = "0.30.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.29.2", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.29.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.29.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.29.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.30.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.30.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 4dcbabf3557f00c28d05d0b50d4320a5d2d277e7 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 26 Mar 2019 09:48:06 -0700
Subject: [PATCH 2350/3084] Update crate paths in publish-all.sh.

---
 cranelift/publish-all.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index f4805aee08..5e080a4b4c 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -43,5 +43,5 @@ for crate in \
     reader wasm module \
     faerie umbrella simplejit
 do
-    echo cargo publish --manifest-path "$crate/Cargo.toml"
+    echo cargo publish --manifest-path "cranelift-$crate/Cargo.toml"
 done

From 1784060baf2da5d91ec68c1587917fd4186c65e3 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 26 Mar 2019 11:07:20 -0700
Subject: [PATCH 2351/3084] Add a sleep command between cargo publish
 invocations.

Crates.io no longer synchronously updates the crate index, so sleep for
a few seconds between cargo publish invocations to give it time to
update.
---
 cranelift/publish-all.sh | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 5e080a4b4c..7792f1e9f8 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -44,4 +44,8 @@ for crate in \
     faerie umbrella simplejit
 do
     echo cargo publish --manifest-path "cranelift-$crate/Cargo.toml"
+
+    # Sleep for a few seconds to allow the server to update the index.
+    # https://internals.rust-lang.org/t/changes-to-how-crates-io-handles-index-updates/9608
+    echo sleep 3
 done

From 82c68671551499aba3e5c400e469de6887fb7ecd Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 26 Mar 2019 11:28:39 -0700
Subject: [PATCH 2352/3084] Work around Cargo #4866. (#699)

* Work around Cargo build #4866.

This fixes #697.

* Rename "cargo4866workaround" to "core" per review feedback.
---
 cranelift/codegen/Cargo.toml      | 17 ++++++++++++++---
 cranelift/codegen/meta/Cargo.toml |  8 +++++++-
 2 files changed, 21 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 0808b0fb65..da9b5e5d8f 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -26,7 +26,7 @@ log = { version = "0.4.6", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.30.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.30.0", default-features = false }
 
 [features]
 default = ["std", "x86", "arm32", "arm64", "riscv"]
@@ -34,8 +34,19 @@ default = ["std", "x86", "arm32", "arm64", "riscv"]
 # The "std" feature enables use of libstd. The "core" feature enables use
 # of some minimal std-like replacement libraries. At least one of these two
 # features need to be enabled.
-std = ["cranelift-entity/std", "cranelift-bforest/std", "target-lexicon/std"]
-core = ["hashmap_core"]
+std = [
+    "cranelift-entity/std",
+    "cranelift-bforest/std",
+    "target-lexicon/std",
+    "cranelift-codegen-meta/std"
+]
+
+# The "core" features enables use of "hashmap_core" since core doesn't have
+# a HashMap implementation, and a workaround for Cargo #4866.
+core = [
+    "hashmap_core",
+    "cranelift-codegen-meta/core"
+]
 
 # This enables some additional functions useful for writing tests, but which
 # can significantly increase the size of the library.
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 8c263d7ea0..a5f734494d 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -9,8 +9,14 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.30.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.30.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }
+
+[features]
+default = ["std"]
+std = ["cranelift-entity/std"]
+# The "core" feature enables a workaround for Cargo #4866.
+core = ["cranelift-entity/core"]

From 141ccb9e9dba855ce118c5d5a14455088d51b567 Mon Sep 17 00:00:00 2001
From: Lars T Hansen 
Date: Wed, 27 Mar 2019 12:57:13 +0100
Subject: [PATCH 2353/3084] Add a --disasm option to clif-util `wasm` and
 `compile` (#713)

- Both the `wasm` and `compile` commands get this new subcommand, and it defaults to false.  This means that test runs with `wasm` can request disassembly (the main reason I am doing this) while test runs with `compile` now must request it, this changes current behavior.
- Switch to using context.compile_and_emit directly, and make the reloc and trap printers just accumulate output, not print it.  This allows us to factor the printing code into the disasm module.
---
 cranelift/codegen/src/context.rs |  23 ++--
 cranelift/src/clif-util.rs       |  18 +++
 cranelift/src/compile.rs         | 186 ++------------------------
 cranelift/src/disasm.rs          | 215 +++++++++++++++++++++++++++++++
 cranelift/src/wasm.rs            |  27 +++-
 5 files changed, 283 insertions(+), 186 deletions(-)
 create mode 100644 cranelift/src/disasm.rs

diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index a78553aa9e..67e676581f 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -91,18 +91,21 @@ impl Context {
     ///
     /// This function calls `compile` and `emit_to_memory`, taking care to resize `mem` as
     /// needed, so it provides a safe interface.
+    ///
+    /// Returns the size of the function's code and the size of the read-only data.
     pub fn compile_and_emit(
         &mut self,
         isa: &TargetIsa,
         mem: &mut Vec,
         relocs: &mut RelocSink,
         traps: &mut TrapSink,
-    ) -> CodegenResult<()> {
-        let code_size = self.compile(isa)?;
+    ) -> CodegenResult<(CodeOffset, CodeOffset)> {
+        let total_size = self.compile(isa)?;
         let old_len = mem.len();
-        mem.resize(old_len + code_size as usize, 0);
-        unsafe { self.emit_to_memory(isa, mem.as_mut_ptr().add(old_len), relocs, traps) };
-        Ok(())
+        mem.resize(old_len + total_size as usize, 0);
+        let code_size =
+            unsafe { self.emit_to_memory(isa, mem.as_mut_ptr().add(old_len), relocs, traps) };
+        Ok((code_size, total_size - code_size))
     }
 
     /// Compile the function.
@@ -111,7 +114,7 @@ impl Context {
     /// represented by `isa`. This does not include the final step of emitting machine code into a
     /// code sink.
     ///
-    /// Returns the size of the function's code.
+    /// Returns the size of the function's code and read-only data.
     pub fn compile(&mut self, isa: &TargetIsa) -> CodegenResult {
         let _tt = timing::compile();
         self.verify_if(isa)?;
@@ -155,15 +158,19 @@ impl Context {
     ///
     /// This function is unsafe since it does not perform bounds checking on the memory buffer,
     /// and it can't guarantee that the `mem` pointer is valid.
+    ///
+    /// Returns the size of the function's code.
     pub unsafe fn emit_to_memory(
         &self,
         isa: &TargetIsa,
         mem: *mut u8,
         relocs: &mut RelocSink,
         traps: &mut TrapSink,
-    ) {
+    ) -> CodeOffset {
         let _tt = timing::binemit();
-        isa.emit_function_to_memory(&self.func, &mut MemoryCodeSink::new(mem, relocs, traps));
+        let mut sink = MemoryCodeSink::new(mem, relocs, traps);
+        isa.emit_function_to_memory(&self.func, &mut sink);
+        sink.code_size as CodeOffset
     }
 
     /// Run the verifier on the function.
diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 4c53cc8828..d51687b4fe 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -30,6 +30,7 @@ use std::process;
 
 mod cat;
 mod compile;
+mod disasm;
 mod print_cfg;
 mod utils;
 
@@ -69,6 +70,19 @@ fn add_time_flag<'a>() -> clap::Arg<'a, 'a> {
         .help("Print pass timing report for test")
 }
 
+fn add_size_flag<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("print-size")
+        .short("X")
+        .help("Print bytecode size")
+}
+
+fn add_disasm_flag<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("disasm")
+        .long("disasm")
+        .short("D")
+        .help("Print machine code disassembly")
+}
+
 fn add_set_flag<'a>() -> clap::Arg<'a, 'a> {
     Arg::with_name("set")
         .long("set")
@@ -120,6 +134,8 @@ fn add_wasm_or_compile<'a>(cmd: &str) -> clap::App<'a, 'a> {
         .arg(add_verbose_flag())
         .arg(add_print_flag())
         .arg(add_time_flag())
+        .arg(add_size_flag())
+        .arg(add_disasm_flag())
         .arg(add_set_flag())
         .arg(add_target_flag())
         .arg(add_input_file_arg())
@@ -226,6 +242,7 @@ fn main() {
             compile::run(
                 get_vec(rest_cmd.values_of("file")),
                 rest_cmd.is_present("print"),
+                rest_cmd.is_present("disasm"),
                 rest_cmd.is_present("time-passes"),
                 &get_vec(rest_cmd.values_of("set")),
                 target_val,
@@ -247,6 +264,7 @@ fn main() {
                     rest_cmd.is_present("just-decode"),
                     rest_cmd.is_present("check-translation"),
                     rest_cmd.is_present("print"),
+                    rest_cmd.is_present("disasm"),
                     &get_vec(rest_cmd.values_of("set")),
                     target_val,
                     rest_cmd.is_present("print-size"),
diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs
index 9903dd6161..f5e34c6417 100644
--- a/cranelift/src/compile.rs
+++ b/cranelift/src/compile.rs
@@ -1,67 +1,19 @@
 //! CLI tool to read Cranelift IR files and compile them into native code.
 
+use crate::disasm::{print_all, PrintRelocs, PrintTraps};
 use crate::utils::{parse_sets_and_triple, read_to_string};
-use cfg_if::cfg_if;
-use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_codegen::settings::FlagsOrIsa;
 use cranelift_codegen::timing;
 use cranelift_codegen::Context;
-use cranelift_codegen::{binemit, ir};
 use cranelift_reader::parse_test;
 use std::path::Path;
 use std::path::PathBuf;
 
-struct PrintRelocs {
-    flag_print: bool,
-}
-
-impl binemit::RelocSink for PrintRelocs {
-    fn reloc_ebb(
-        &mut self,
-        where_: binemit::CodeOffset,
-        r: binemit::Reloc,
-        offset: binemit::CodeOffset,
-    ) {
-        if self.flag_print {
-            println!("reloc_ebb: {} {} at {}", r, offset, where_);
-        }
-    }
-
-    fn reloc_external(
-        &mut self,
-        where_: binemit::CodeOffset,
-        r: binemit::Reloc,
-        name: &ir::ExternalName,
-        addend: binemit::Addend,
-    ) {
-        if self.flag_print {
-            println!("reloc_external: {} {} {} at {}", r, name, addend, where_);
-        }
-    }
-
-    fn reloc_jt(&mut self, where_: binemit::CodeOffset, r: binemit::Reloc, jt: ir::JumpTable) {
-        if self.flag_print {
-            println!("reloc_jt: {} {} at {}", r, jt, where_);
-        }
-    }
-}
-
-struct PrintTraps {
-    flag_print: bool,
-}
-
-impl binemit::TrapSink for PrintTraps {
-    fn trap(&mut self, offset: binemit::CodeOffset, _srcloc: ir::SourceLoc, code: ir::TrapCode) {
-        if self.flag_print {
-            println!("trap: {} at {}", code, offset);
-        }
-    }
-}
-
 pub fn run(
     files: Vec,
     flag_print: bool,
+    flag_disasm: bool,
     flag_report_times: bool,
     flag_set: &[String],
     flag_isa: &str,
@@ -73,6 +25,7 @@ pub fn run(
         let name = String::from(path.as_os_str().to_string_lossy());
         handle_module(
             flag_print,
+            flag_disasm,
             flag_report_times,
             &path.to_path_buf(),
             &name,
@@ -84,6 +37,7 @@ pub fn run(
 
 fn handle_module(
     flag_print: bool,
+    flag_disasm: bool,
     flag_report_times: bool,
     path: &PathBuf,
     name: &str,
@@ -106,39 +60,21 @@ fn handle_module(
         let mut context = Context::new();
         context.func = func;
 
-        // Compile and encode the result to machine code.
-        let total_size = context
-            .compile(isa)
-            .map_err(|err| pretty_error(&context.func, Some(isa), err))?;
+        let mut relocs = PrintRelocs::new(flag_print);
+        let mut traps = PrintTraps::new(flag_print);
+        let mut mem = vec![];
 
-        let mut mem = vec![0; total_size as usize];
-        let mut relocs = PrintRelocs { flag_print };
-        let mut traps = PrintTraps { flag_print };
-        let mut code_sink: binemit::MemoryCodeSink;
-        unsafe {
-            code_sink = binemit::MemoryCodeSink::new(mem.as_mut_ptr(), &mut relocs, &mut traps);
-        }
-        isa.emit_function_to_memory(&context.func, &mut code_sink);
+        // Compile and encode the result to machine code.
+        let (code_size, rodata_size) = context
+            .compile_and_emit(isa, &mut mem, &mut relocs, &mut traps)
+            .map_err(|err| pretty_error(&context.func, Some(isa), err))?;
 
         if flag_print {
             println!("{}", context.func.display(isa));
         }
 
-        if flag_print {
-            print!(".byte ");
-            let mut first = true;
-            for byte in &mem {
-                if first {
-                    first = false;
-                } else {
-                    print!(", ");
-                }
-                print!("{}", byte);
-            }
-
-            println!();
-            print_disassembly(isa, &mem[0..code_sink.code_size as usize])?;
-            print_readonly_data(&mem[code_sink.code_size as usize..total_size as usize]);
+        if flag_disasm {
+            print_all(isa, &mem, code_size, rodata_size, &relocs, &traps)?;
         }
     }
 
@@ -148,99 +84,3 @@ fn handle_module(
 
     Ok(())
 }
-
-fn print_readonly_data(mem: &[u8]) {
-    if mem.is_empty() {
-        return;
-    }
-
-    println!("\nFollowed by {} bytes of read-only data:", mem.len());
-
-    for (i, byte) in mem.iter().enumerate() {
-        if i % 16 == 0 {
-            if i != 0 {
-                println!();
-            }
-            print!("{:4}: ", i);
-        }
-        if i % 4 == 0 {
-            print!(" ");
-        }
-        print!("{:02x} ", byte);
-    }
-    println!();
-}
-
-cfg_if! {
-    if #[cfg(feature = "disas")] {
-        use capstone::prelude::*;
-        use target_lexicon::Architecture;
-        use std::fmt::Write;
-
-        fn get_disassembler(isa: &TargetIsa) -> Result {
-            let cs = match isa.triple().architecture {
-                Architecture::Riscv32 | Architecture::Riscv64 => {
-                    return Err(String::from("No disassembler for RiscV"))
-                }
-                Architecture::I386 | Architecture::I586 | Architecture::I686 => Capstone::new()
-                    .x86()
-                    .mode(arch::x86::ArchMode::Mode32)
-                    .build(),
-                Architecture::X86_64 => Capstone::new()
-                    .x86()
-                    .mode(arch::x86::ArchMode::Mode64)
-                    .build(),
-                Architecture::Arm
-                | Architecture::Armv4t
-                | Architecture::Armv5te
-                | Architecture::Armv7
-                | Architecture::Armv7s => Capstone::new().arm().mode(arch::arm::ArchMode::Arm).build(),
-                Architecture::Thumbv6m | Architecture::Thumbv7em | Architecture::Thumbv7m => Capstone::new(
-                ).arm()
-                    .mode(arch::arm::ArchMode::Thumb)
-                    .build(),
-                Architecture::Aarch64 => Capstone::new()
-                    .arm64()
-                    .mode(arch::arm64::ArchMode::Arm)
-                    .build(),
-                _ => return Err(String::from("Unknown ISA")),
-            };
-
-            cs.map_err(|err| err.to_string())
-        }
-
-        fn print_disassembly(isa: &TargetIsa, mem: &[u8]) -> Result<(), String> {
-            let mut cs = get_disassembler(isa)?;
-
-            println!("\nDisassembly of {} bytes:", mem.len());
-            let insns = cs.disasm_all(&mem, 0x0).unwrap();
-            for i in insns.iter() {
-                let mut line = String::new();
-
-                write!(&mut line, "{:4x}:\t", i.address()).unwrap();
-
-                let mut bytes_str = String::new();
-                for b in i.bytes() {
-                    write!(&mut bytes_str, "{:02x} ", b).unwrap();
-                }
-                write!(&mut line, "{:21}\t", bytes_str).unwrap();
-
-                if let Some(s) = i.mnemonic() {
-                    write!(&mut line, "{}\t", s).unwrap();
-                }
-
-                if let Some(s) = i.op_str() {
-                    write!(&mut line, "{}", s).unwrap();
-                }
-
-                println!("{}", line);
-            }
-            Ok(())
-        }
-    } else {
-        fn print_disassembly(_: &TargetIsa, _: &[u8]) -> Result<(), String> {
-            println!("\nNo disassembly available.");
-            Ok(())
-        }
-    }
-}
diff --git a/cranelift/src/disasm.rs b/cranelift/src/disasm.rs
new file mode 100644
index 0000000000..fa55ab58a2
--- /dev/null
+++ b/cranelift/src/disasm.rs
@@ -0,0 +1,215 @@
+use cfg_if::cfg_if;
+use cranelift_codegen::isa::TargetIsa;
+use cranelift_codegen::{binemit, ir};
+use std::fmt::Write;
+
+pub struct PrintRelocs {
+    pub flag_print: bool,
+    pub text: String,
+}
+
+impl PrintRelocs {
+    pub fn new(flag_print: bool) -> PrintRelocs {
+        Self {
+            flag_print,
+            text: String::new(),
+        }
+    }
+}
+
+impl binemit::RelocSink for PrintRelocs {
+    fn reloc_ebb(
+        &mut self,
+        where_: binemit::CodeOffset,
+        r: binemit::Reloc,
+        offset: binemit::CodeOffset,
+    ) {
+        if self.flag_print {
+            write!(
+                &mut self.text,
+                "reloc_ebb: {} {} at {}\n",
+                r, offset, where_
+            )
+            .unwrap();
+        }
+    }
+
+    fn reloc_external(
+        &mut self,
+        where_: binemit::CodeOffset,
+        r: binemit::Reloc,
+        name: &ir::ExternalName,
+        addend: binemit::Addend,
+    ) {
+        if self.flag_print {
+            write!(
+                &mut self.text,
+                "reloc_external: {} {} {} at {}\n",
+                r, name, addend, where_
+            )
+            .unwrap();
+        }
+    }
+
+    fn reloc_jt(&mut self, where_: binemit::CodeOffset, r: binemit::Reloc, jt: ir::JumpTable) {
+        if self.flag_print {
+            write!(&mut self.text, "reloc_jt: {} {} at {}\n", r, jt, where_).unwrap();
+        }
+    }
+}
+
+pub struct PrintTraps {
+    pub flag_print: bool,
+    pub text: String,
+}
+
+impl PrintTraps {
+    pub fn new(flag_print: bool) -> PrintTraps {
+        Self {
+            flag_print,
+            text: String::new(),
+        }
+    }
+}
+
+impl binemit::TrapSink for PrintTraps {
+    fn trap(&mut self, offset: binemit::CodeOffset, _srcloc: ir::SourceLoc, code: ir::TrapCode) {
+        if self.flag_print {
+            write!(&mut self.text, "trap: {} at {}\n", code, offset).unwrap();
+        }
+    }
+}
+
+cfg_if! {
+    if #[cfg(feature = "disas")] {
+        use capstone::prelude::*;
+        use target_lexicon::Architecture;
+
+        fn get_disassembler(isa: &TargetIsa) -> Result {
+            let cs = match isa.triple().architecture {
+                Architecture::Riscv32 | Architecture::Riscv64 => {
+                    return Err(String::from("No disassembler for RiscV"))
+                }
+                Architecture::I386 | Architecture::I586 | Architecture::I686 => Capstone::new()
+                    .x86()
+                    .mode(arch::x86::ArchMode::Mode32)
+                    .build(),
+                Architecture::X86_64 => Capstone::new()
+                    .x86()
+                    .mode(arch::x86::ArchMode::Mode64)
+                    .build(),
+                Architecture::Arm
+                | Architecture::Armv4t
+                | Architecture::Armv5te
+                | Architecture::Armv7
+                | Architecture::Armv7s => Capstone::new().arm().mode(arch::arm::ArchMode::Arm).build(),
+                Architecture::Thumbv6m | Architecture::Thumbv7em | Architecture::Thumbv7m => Capstone::new(
+                ).arm()
+                    .mode(arch::arm::ArchMode::Thumb)
+                    .build(),
+                Architecture::Aarch64 => Capstone::new()
+                    .arm64()
+                    .mode(arch::arm64::ArchMode::Arm)
+                    .build(),
+                _ => return Err(String::from("Unknown ISA")),
+            };
+
+            cs.map_err(|err| err.to_string())
+        }
+
+        pub fn print_disassembly(isa: &TargetIsa, mem: &[u8]) -> Result<(), String> {
+            let mut cs = get_disassembler(isa)?;
+
+            println!("\nDisassembly of {} bytes:", mem.len());
+            let insns = cs.disasm_all(&mem, 0x0).unwrap();
+            for i in insns.iter() {
+                let mut line = String::new();
+
+                write!(&mut line, "{:4x}:\t", i.address()).unwrap();
+
+                let mut bytes_str = String::new();
+                let mut len = 0;
+                let mut first = true;
+                for b in i.bytes() {
+                    write!(&mut bytes_str, "{:02x}", b).unwrap();
+                    if !first {
+                        write!(&mut bytes_str, " ").unwrap();
+                    }
+                    len += 1;
+                    first = false;
+                }
+                write!(&mut line, "{:21}\t", bytes_str).unwrap();
+                if len > 8 {
+                    write!(&mut line, "\n\t\t\t\t").unwrap();
+                }
+
+                if let Some(s) = i.mnemonic() {
+                    write!(&mut line, "{}\t", s).unwrap();
+                }
+
+                if let Some(s) = i.op_str() {
+                    write!(&mut line, "{}", s).unwrap();
+                }
+
+                println!("{}", line);
+            }
+            Ok(())
+        }
+    } else {
+        pub fn print_disassembly(_: &TargetIsa, _: &[u8]) -> Result<(), String> {
+            println!("\nNo disassembly available.");
+            Ok(())
+        }
+    }
+}
+
+pub fn print_all(
+    isa: &TargetIsa,
+    mem: &[u8],
+    code_size: u32,
+    rodata_size: u32,
+    relocs: &PrintRelocs,
+    traps: &PrintTraps,
+) -> Result<(), String> {
+    print_bytes(&mem);
+    print_disassembly(isa, &mem[0..code_size as usize])?;
+    print_readonly_data(&mem[code_size as usize..(code_size + rodata_size) as usize]);
+    println!("\n{}\n{}", &relocs.text, &traps.text);
+    Ok(())
+}
+
+pub fn print_bytes(mem: &[u8]) {
+    print!(".byte ");
+    let mut first = true;
+    for byte in mem.iter() {
+        if first {
+            first = false;
+        } else {
+            print!(", ");
+        }
+        print!("{}", byte);
+    }
+    println!();
+}
+
+pub fn print_readonly_data(mem: &[u8]) {
+    if mem.is_empty() {
+        return;
+    }
+
+    println!("\nFollowed by {} bytes of read-only data:", mem.len());
+
+    for (i, byte) in mem.iter().enumerate() {
+        if i % 16 == 0 {
+            if i != 0 {
+                println!();
+            }
+            print!("{:4}: ", i);
+        }
+        if i % 4 == 0 {
+            print!(" ");
+        }
+        print!("{:02x} ", byte);
+    }
+    println!();
+}
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index af8e818697..67d0225f49 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -7,6 +7,7 @@
     allow(clippy::too_many_arguments, clippy::cyclomatic_complexity)
 )]
 
+use crate::disasm::{print_all, PrintRelocs, PrintTraps};
 use crate::utils::{parse_sets_and_triple, read_to_end};
 use cranelift_codegen::print_errors::{pretty_error, pretty_verifier_error};
 use cranelift_codegen::settings::FlagsOrIsa;
@@ -41,6 +42,7 @@ pub fn run(
     flag_just_decode: bool,
     flag_check_translation: bool,
     flag_print: bool,
+    flag_print_disasm: bool,
     flag_set: &[String],
     flag_triple: &str,
     flag_print_size: bool,
@@ -57,6 +59,7 @@ pub fn run(
             flag_check_translation,
             flag_print,
             flag_print_size,
+            flag_print_disasm,
             flag_report_times,
             &path.to_path_buf(),
             &name,
@@ -72,6 +75,7 @@ fn handle_module(
     flag_check_translation: bool,
     flag_print: bool,
     flag_print_size: bool,
+    flag_print_disasm: bool,
     flag_report_times: bool,
     path: &PathBuf,
     name: &str,
@@ -100,7 +104,7 @@ fn handle_module(
         None => {
             return Err(String::from(
                 "Error: the wasm command requires an explicit isa.",
-            ))
+            ));
         }
     };
 
@@ -157,27 +161,36 @@ fn handle_module(
     for (def_index, func) in dummy_environ.info.function_bodies.iter() {
         context.func = func.clone();
 
+        let mut saved_sizes = None;
         let func_index = num_func_imports + def_index.index();
+        let mut mem = vec![];
+        let mut relocs = PrintRelocs::new(flag_print);
+        let mut traps = PrintTraps::new(flag_print);
         if flag_check_translation {
             if let Err(errors) = context.verify(fisa) {
                 return Err(pretty_verifier_error(&context.func, fisa.isa, None, errors));
             }
         } else {
-            let compiled_size = context
-                .compile(isa)
+            let (code_size, rodata_size) = context
+                .compile_and_emit(isa, &mut mem, &mut relocs, &mut traps)
                 .map_err(|err| pretty_error(&context.func, fisa.isa, err))?;
+
             if flag_print_size {
                 println!(
                     "Function #{} code size: {} bytes",
-                    func_index, compiled_size
+                    func_index, code_size + rodata_size
                 );
-                total_module_code_size += compiled_size;
+                total_module_code_size += code_size + rodata_size;
                 println!(
                     "Function #{} bytecode size: {} bytes",
                     func_index,
                     dummy_environ.func_bytecode_sizes[def_index.index()]
                 );
             }
+
+            if flag_print_disasm {
+                saved_sizes = Some((code_size, rodata_size));
+            }
         }
 
         if flag_print {
@@ -196,6 +209,10 @@ fn handle_module(
             vprintln!(flag_verbose, "");
         }
 
+        if let Some((code_size, rodata_size)) = saved_sizes {
+            print_all(isa, &mem, code_size, rodata_size, &relocs, &traps)?;
+        }
+
         context.clear();
     }
 

From 68bda3a42df2c6a7edeb6b1838861e050cc019e3 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Mar 2019 16:49:27 +0100
Subject: [PATCH 2354/3084] [meta-python] Fix typos;

---
 cranelift/codegen/meta-python/base/entities.py     | 2 +-
 cranelift/codegen/meta-python/cdsl/instructions.py | 2 +-
 cranelift/codegen/meta-python/gen_instr.py         | 4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/meta-python/base/entities.py b/cranelift/codegen/meta-python/base/entities.py
index 7b11810f48..0226b18ca4 100644
--- a/cranelift/codegen/meta-python/base/entities.py
+++ b/cranelift/codegen/meta-python/base/entities.py
@@ -19,7 +19,7 @@ stack_slot = EntityRefKind('stack_slot', 'A stack slot.')
 #: A reference to a global value.
 global_value = EntityRefKind('global_value', 'A global value.')
 
-#: A reference to a function sugnature declared in the function preamble.
+#: A reference to a function signature declared in the function preamble.
 #: This is used to provide the call signature in a call_indirect instruction.
 sig_ref = EntityRefKind('sig_ref', 'A function signature.')
 
diff --git a/cranelift/codegen/meta-python/cdsl/instructions.py b/cranelift/codegen/meta-python/cdsl/instructions.py
index 5b2a04d7be..34741991da 100644
--- a/cranelift/codegen/meta-python/cdsl/instructions.py
+++ b/cranelift/codegen/meta-python/cdsl/instructions.py
@@ -252,7 +252,7 @@ class Instruction(object):
         for opnum in self.value_opnums:
             typ = self.ins[opnum].typevar
             tv = typ.free_typevar()
-            # Non-polymorphic or derived form ctrl_typevar is OK.
+            # Non-polymorphic or derived from ctrl_typevar is OK.
             if tv is None or tv is ctrl_typevar:
                 continue
             # No other derived typevars allowed.
diff --git a/cranelift/codegen/meta-python/gen_instr.py b/cranelift/codegen/meta-python/gen_instr.py
index 1c6905f1cd..94de93b000 100644
--- a/cranelift/codegen/meta-python/gen_instr.py
+++ b/cranelift/codegen/meta-python/gen_instr.py
@@ -364,7 +364,7 @@ def gen_opcodes(groups, fmt):
     # We explicitly set the discriminant of the first variant to 1, which
     # allows us to take advantage of the NonZero optimization, meaning that
     # wrapping enums can use the 0 discriminant instead of increasing the size
-    # if the whole type, and so SIZEOF(Option>) == SIZEOF(Opcode)
+    # of the whole type, and so SIZEOF(Option>) == SIZEOF(Opcode)
     is_first_opcode = True
     with fmt.indented('pub enum Opcode {', '}'):
         for g in groups:
@@ -505,7 +505,7 @@ def gen_type_constraints(fmt, instrs):
     # Table of operand constraint sequences (as tuples). Each operand
     # constraint is represented as a string, one of:
     # - `Concrete(vt)`, where `vt` is a value type name.
-    # - `Free(idx)` where `idx` isan index into `type_sets`.
+    # - `Free(idx)` where `idx` is an index into `type_sets`.
     # - `Same`, `Lane`, `AsBool` for controlling typevar-derived constraints.
     operand_seqs = UniqueSeqTable()
 

From 72b0d26ee93e2bfb577b6095d6a3cb993c0ef8fb Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Mar 2019 17:12:03 +0100
Subject: [PATCH 2355/3084] [meta] Add features to srcgen;

- Adds a compiler warning when the fmtln! macro isn't correctly used.
- Allow to add an empty line.
- Make the output of generated matches more beautiful, by having one
struct per line on the clause.
- Add a method to add match with doesn't read any data from fields.
- Make sure that the placeholder clause of a match is put at the end.
---
 cranelift/codegen/meta/src/gen_settings.rs |   8 +-
 cranelift/codegen/meta/src/srcgen.rs       | 107 ++++++++++++++++++---
 2 files changed, 98 insertions(+), 17 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
index a3c97ebab8..20fc6d1fc3 100644
--- a/cranelift/codegen/meta/src/gen_settings.rs
+++ b/cranelift/codegen/meta/src/gen_settings.rs
@@ -152,13 +152,9 @@ fn gen_getter(setting: &Setting, fmt: &mut Formatter) {
             fmt.indent(|fmt| {
                 let mut m = Match::new(format!("self.bytes[{}]", setting.byte_offset));
                 for (i, v) in values.iter().enumerate() {
-                    m.arm(
-                        format!("{}", i),
-                        vec![],
-                        format!("{}::{}", ty, camel_case(v)),
-                    );
+                    m.arm_no_fields(format!("{}", i), format!("{}::{}", ty, camel_case(v)));
                 }
-                m.arm("_", vec![], "panic!(\"Invalid enum value\")");
+                m.arm_no_fields("_", "panic!(\"Invalid enum value\")");
                 fmt.add_match(m);
             });
             fmtln!(fmt, "}");
diff --git a/cranelift/codegen/meta/src/srcgen.rs b/cranelift/codegen/meta/src/srcgen.rs
index dc7f622940..9a6c4af038 100644
--- a/cranelift/codegen/meta/src/srcgen.rs
+++ b/cranelift/codegen/meta/src/srcgen.rs
@@ -25,6 +25,14 @@ macro_rules! fmtln {
     ($fmt:ident, $arg:expr) => {
         $fmt.line($arg);
     };
+
+    ($_:tt, $($args:expr),+) => {
+        compile_error!("This macro requires at least two arguments: the Formatter instance and a format string.");
+    };
+
+    ($_:tt) => {
+        compile_error!("This macro requires at least two arguments: the Formatter instance and a format string.");
+    };
 }
 
 pub struct Formatter {
@@ -84,6 +92,11 @@ impl Formatter {
         self.lines.push(indented_line);
     }
 
+    /// Pushes an empty line.
+    pub fn empty_line(&mut self) {
+        self.lines.push("\n".to_string());
+    }
+
     /// Emit a line outdented one level.
     pub fn _outdented_line(&mut self, s: &str) {
         let new_line = format!("{}{}", self._get_outdent(), s);
@@ -112,7 +125,7 @@ impl Formatter {
     }
 
     /// Add one or more lines after stripping common indentation.
-    pub fn _multi_line(&mut self, s: &str) {
+    pub fn multi_line(&mut self, s: &str) {
         parse_multiline(s).into_iter().for_each(|l| self.line(&l));
     }
 
@@ -141,7 +154,7 @@ impl Formatter {
         self.indent(|fmt| {
             for (&(ref fields, ref body), ref names) in m.arms.iter() {
                 // name { fields } | name { fields } => { body }
-                let conditions: Vec = names
+                let conditions = names
                     .iter()
                     .map(|name| {
                         if fields.len() > 0 {
@@ -150,9 +163,20 @@ impl Formatter {
                             name.clone()
                         }
                     })
-                    .collect();
-                let lhs = conditions.join(" | ");
-                fmtln!(fmt, "{} => {{", lhs);
+                    .collect::>()
+                    .join(" |\n")
+                    + " => {";
+
+                fmt.multi_line(&conditions);
+                fmt.indent(|fmt| {
+                    fmt.line(body);
+                });
+                fmt.line("}");
+            }
+
+            // Make sure to include the catch all clause last.
+            if let Some(body) = m.catch_all {
+                fmt.line("_ => {");
                 fmt.indent(|fmt| {
                     fmt.line(body);
                 });
@@ -239,27 +263,57 @@ fn parse_multiline(s: &str) -> Vec {
 pub struct Match {
     expr: String,
     arms: BTreeMap<(Vec, String), BTreeSet>,
+    /// The clause for the placeholder pattern _.
+    catch_all: Option,
 }
 
 impl Match {
     /// Create a new match statement on `expr`.
-    pub fn new>(expr: T) -> Self {
+    pub fn new(expr: impl Into) -> Self {
         Self {
             expr: expr.into(),
             arms: BTreeMap::new(),
+            catch_all: None,
         }
     }
 
-    /// Add an arm to the Match statement.
-    pub fn arm>(&mut self, name: T, fields: Vec, body: T) {
-        // let key = (fields, body);
+    fn set_catch_all(&mut self, clause: String) {
+        assert!(self.catch_all.is_none());
+        self.catch_all = Some(clause);
+    }
+
+    /// Add an arm that reads fields to the Match statement.
+    pub fn arm, S: Into>(&mut self, name: T, fields: Vec, body: T) {
+        let name = name.into();
+        assert!(
+            name != "_",
+            "catch all clause can't extract fields, use arm_no_fields instead."
+        );
+
         let body = body.into();
         let fields = fields.into_iter().map(|x| x.into()).collect();
         let match_arm = self
             .arms
             .entry((fields, body))
             .or_insert_with(BTreeSet::new);
-        match_arm.insert(name.into());
+        match_arm.insert(name);
+    }
+
+    /// Adds an arm that doesn't read anythings from the fields to the Match statement.
+    pub fn arm_no_fields(&mut self, name: impl Into, body: impl Into) {
+        let body = body.into();
+
+        let name = name.into();
+        if name == "_" {
+            self.set_catch_all(body);
+            return;
+        }
+
+        let match_arm = self
+            .arms
+            .entry((Vec::new(), body))
+            .or_insert_with(BTreeSet::new);
+        match_arm.insert(name);
     }
 }
 
@@ -296,7 +350,8 @@ match x {
     Green { a, b } => {
         different body
     }
-    Orange { a, b } | Yellow { a, b } => {
+    Orange { a, b } |
+    Yellow { a, b } => {
         some body
     }
     Blue { x, y } => {
@@ -308,6 +363,36 @@ match x {
         assert_eq!(fmt.lines, expected_lines);
     }
 
+    #[test]
+    fn match_with_catchall_order() {
+        // The catchall placeholder must be placed after other clauses.
+        let mut m = Match::new("x");
+        m.arm("Orange", vec!["a", "b"], "some body");
+        m.arm("Green", vec!["a", "b"], "different body");
+        m.arm_no_fields("_", "unreachable!()");
+        assert_eq!(m.arms.len(), 2); // catchall is not counted
+
+        let mut fmt = Formatter::new();
+        fmt.add_match(m);
+
+        let expected_lines = from_raw_string(
+            r#"
+match x {
+    Green { a, b } => {
+        different body
+    }
+    Orange { a, b } => {
+        some body
+    }
+    _ => {
+        unreachable!()
+    }
+}
+        "#,
+        );
+        assert_eq!(fmt.lines, expected_lines);
+    }
+
     #[test]
     fn parse_multiline_works() {
         let input = "\n    hello\n    world\n";

From 393b88af6e104d3a027e2b0d3d21f4dc8df69489 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Mar 2019 17:20:50 +0100
Subject: [PATCH 2356/3084] [meta] Implement UniqueTable in the Rust crate;

... and also rename the previously-named UniqueTable to UniqueSeqTable,
which is the name used in the Python code.
---
 cranelift/codegen/meta/src/gen_settings.rs |  4 +-
 cranelift/codegen/meta/src/unique_table.rs | 43 +++++++++++++++++++++-
 2 files changed, 43 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
index 20fc6d1fc3..a797544534 100644
--- a/cranelift/codegen/meta/src/gen_settings.rs
+++ b/cranelift/codegen/meta/src/gen_settings.rs
@@ -7,7 +7,7 @@ use crate::constant_hash::{generate_table, simple_hash};
 use crate::error;
 use crate::shared;
 use crate::srcgen::{Formatter, Match};
-use crate::unique_table::UniqueTable;
+use crate::unique_table::UniqueSeqTable;
 use std::collections::HashMap;
 
 enum ParentGroup {
@@ -238,7 +238,7 @@ impl<'a> SettingOrPreset<'a> {
 
 /// Emits DESCRIPTORS, ENUMERATORS, HASH_TABLE and PRESETS.
 fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
-    let mut enum_table: UniqueTable<&'static str> = UniqueTable::new();
+    let mut enum_table = UniqueSeqTable::new();
 
     let mut descriptor_index_map: HashMap = HashMap::new();
 
diff --git a/cranelift/codegen/meta/src/unique_table.rs b/cranelift/codegen/meta/src/unique_table.rs
index 8e325cdeed..d4f4fa00cf 100644
--- a/cranelift/codegen/meta/src/unique_table.rs
+++ b/cranelift/codegen/meta/src/unique_table.rs
@@ -1,15 +1,54 @@
+use std::collections::HashMap;
+use std::hash::Hash;
 use std::slice;
 
+/// Collect items into the `table` list, removing duplicates.
+pub struct UniqueTable<'entries, T: Eq + Hash> {
+    table: Vec<&'entries T>,
+    map: HashMap<&'entries T, usize>,
+}
+
+impl<'entries, T: Eq + Hash> UniqueTable<'entries, T> {
+    pub fn new() -> Self {
+        Self {
+            table: Vec::new(),
+            map: HashMap::new(),
+        }
+    }
+
+    pub fn add(&mut self, entry: &'entries T) -> usize {
+        match self.map.get(&entry) {
+            None => {
+                let i = self.table.len();
+                self.table.push(entry);
+                self.map.insert(entry, i);
+                i
+            }
+            Some(&i) => i,
+        }
+    }
+
+    pub fn len(&self) -> usize {
+        self.table.len()
+    }
+    pub fn iter(&self) -> slice::Iter<&'entries T> {
+        self.table.iter()
+    }
+}
+
 /// A table of sequences which tries to avoid common subsequences.
-pub struct UniqueTable {
+pub struct UniqueSeqTable {
     table: Vec,
 }
 
-impl UniqueTable {
+impl UniqueSeqTable {
     pub fn new() -> Self {
         Self { table: Vec::new() }
     }
     pub fn add(&mut self, values: &Vec) -> usize {
+        if values.len() == 0 {
+            return 0;
+        }
         if let Some(offset) = find_subsequence(values, &self.table) {
             offset
         } else {

From f3f449b45be02ee42997f0afa278068d003df056 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Mar 2019 19:14:07 +0100
Subject: [PATCH 2357/3084] [meta] Tweak generation of settings/registers;

---
 cranelift/codegen/meta/src/gen_registers.rs   |  4 +--
 cranelift/codegen/meta/src/gen_settings.rs    | 25 ++++++++-----------
 cranelift/codegen/meta/src/lib.rs             | 19 +++++++++++---
 cranelift/codegen/meta/src/shared/settings.rs |  2 +-
 cranelift/codegen/meta/src/shared/types.rs    |  2 +-
 5 files changed, 29 insertions(+), 23 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_registers.rs b/cranelift/codegen/meta/src/gen_registers.rs
index 3a75c7147a..5e0fdac57b 100644
--- a/cranelift/codegen/meta/src/gen_registers.rs
+++ b/cranelift/codegen/meta/src/gen_registers.rs
@@ -132,9 +132,9 @@ fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) {
     fmtln!(fmt, "}");
 }
 
-pub fn generate(isa: &TargetIsa, base_filename: &str, out_dir: &str) -> Result<(), error::Error> {
+pub fn generate(isa: &TargetIsa, filename: &str, out_dir: &str) -> Result<(), error::Error> {
     let mut fmt = Formatter::new();
     gen_isa(&isa, &mut fmt);
-    fmt.update_file(format!("{}-{}.rs", base_filename, isa.name), out_dir)?;
+    fmt.update_file(filename, out_dir)?;
     Ok(())
 }
diff --git a/cranelift/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
index a797544534..b57e7a4f51 100644
--- a/cranelift/codegen/meta/src/gen_settings.rs
+++ b/cranelift/codegen/meta/src/gen_settings.rs
@@ -1,16 +1,14 @@
 use crate::cdsl::camel_case;
-use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::settings::{
     BoolSetting, Predicate, Preset, Setting, SettingGroup, SpecificSetting,
 };
 use crate::constant_hash::{generate_table, simple_hash};
 use crate::error;
-use crate::shared;
 use crate::srcgen::{Formatter, Match};
 use crate::unique_table::UniqueSeqTable;
 use std::collections::HashMap;
 
-enum ParentGroup {
+pub enum ParentGroup {
     None,
     Shared,
 }
@@ -187,12 +185,12 @@ fn gen_getters(group: &SettingGroup, fmt: &mut Formatter) {
         fmt.doc_comment("Get a view of the boolean predicates.");
         fmtln!(
             fmt,
-            "pub fn predicate_view(&self) -> ::settings::PredicateView {"
+            "pub fn predicate_view(&self) -> crate::settings::PredicateView {"
         );
         fmt.indent(|fmt| {
             fmtln!(
                 fmt,
-                "::settings::PredicateView::new(&self.bytes[{}..])",
+                "crate::settings::PredicateView::new(&self.bytes[{}..])",
                 group.bool_start_byte_offset
             );
         });
@@ -441,17 +439,14 @@ fn gen_group(group: &SettingGroup, parent: ParentGroup, fmt: &mut Formatter) {
     gen_display(group, fmt);
 }
 
-pub fn generate_common(filename: &str, out_dir: &str) -> Result {
-    let settings = shared::settings::generate();
+pub fn generate(
+    settings: &SettingGroup,
+    parent_group: ParentGroup,
+    filename: &str,
+    out_dir: &str,
+) -> Result<(), error::Error> {
     let mut fmt = Formatter::new();
-    gen_group(&settings, ParentGroup::None, &mut fmt);
+    gen_group(&settings, parent_group, &mut fmt);
     fmt.update_file(filename, out_dir)?;
-    Ok(settings)
-}
-
-pub fn generate(isa: &TargetIsa, prefix: &str, out_dir: &str) -> Result<(), error::Error> {
-    let mut fmt = Formatter::new();
-    gen_group(&isa.settings, ParentGroup::Shared, &mut fmt);
-    fmt.update_file(format!("{}-{}.rs", prefix, isa.name), out_dir)?;
     Ok(())
 }
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index fe65e19d2b..1599829dd9 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -20,16 +20,27 @@ pub fn isa_from_arch(arch: &str) -> Result {
 /// Generates all the Rust source files used in Cranelift from the meta-language.
 pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error> {
     // Common definitions.
-    let shared_settings = gen_settings::generate_common("new_settings.rs", &out_dir)?;
+    let settings = shared::settings::define();
 
+    gen_settings::generate(
+        &settings,
+        gen_settings::ParentGroup::None,
+        "new_settings.rs",
+        &out_dir,
+    )?;
     gen_types::generate("types.rs", &out_dir)?;
 
     // Per ISA definitions.
-    let isas = isa::define(isas, &shared_settings);
+    let isas = isa::define(isas, &settings);
 
     for isa in isas {
-        gen_registers::generate(&isa, "registers", &out_dir)?;
-        gen_settings::generate(&isa, "new_settings", &out_dir)?;
+        gen_registers::generate(&isa, &format!("registers-{}.rs", isa.name), &out_dir)?;
+        gen_settings::generate(
+            &isa.settings,
+            gen_settings::ParentGroup::Shared,
+            &format!("new_settings-{}", isa.name),
+            &out_dir,
+        )?;
     }
 
     Ok(())
diff --git a/cranelift/codegen/meta/src/shared/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
index b792a72283..c89efa6b02 100644
--- a/cranelift/codegen/meta/src/shared/settings.rs
+++ b/cranelift/codegen/meta/src/shared/settings.rs
@@ -1,6 +1,6 @@
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
 
-pub fn generate() -> SettingGroup {
+pub fn define() -> SettingGroup {
     let mut settings = SettingGroupBuilder::new("shared");
 
     settings.add_enum(
diff --git a/cranelift/codegen/meta/src/shared/types.rs b/cranelift/codegen/meta/src/shared/types.rs
index 7ce42b1978..aff837252b 100644
--- a/cranelift/codegen/meta/src/shared/types.rs
+++ b/cranelift/codegen/meta/src/shared/types.rs
@@ -113,7 +113,7 @@ impl Iterator for FloatIterator {
 /// A type representing CPU flags.
 ///
 /// Flags can't be stored in memory.
-#[derive(Debug, Clone, Copy, Eq, PartialEq)]
+#[derive(Debug, Clone, Copy, Eq, PartialEq, Hash)]
 pub enum Flag {
     /// CPU flags from an integer comparison.
     IFlags,

From 146e0bd2f54c77a7f54bf2504295e87a7839d1f7 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Mar 2019 19:26:32 +0100
Subject: [PATCH 2358/3084] [meta] Port Typevar to the Rust crate;

---
 cranelift/codegen/meta/src/cdsl/mod.rs     |   1 +
 cranelift/codegen/meta/src/cdsl/types.rs   |  54 +-
 cranelift/codegen/meta/src/cdsl/typevar.rs | 738 +++++++++++++++++++++
 3 files changed, 788 insertions(+), 5 deletions(-)
 create mode 100644 cranelift/codegen/meta/src/cdsl/typevar.rs

diff --git a/cranelift/codegen/meta/src/cdsl/mod.rs b/cranelift/codegen/meta/src/cdsl/mod.rs
index 87c615d70e..dd3a7ec859 100644
--- a/cranelift/codegen/meta/src/cdsl/mod.rs
+++ b/cranelift/codegen/meta/src/cdsl/mod.rs
@@ -7,6 +7,7 @@ pub mod isa;
 pub mod regs;
 pub mod settings;
 pub mod types;
+pub mod typevar;
 
 /// A macro that converts boolean settings into predicates to look more natural.
 #[macro_export]
diff --git a/cranelift/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
index 898c048b8b..643ecdc4b6 100644
--- a/cranelift/codegen/meta/src/cdsl/types.rs
+++ b/cranelift/codegen/meta/src/cdsl/types.rs
@@ -27,7 +27,7 @@ static _RUST_NAME_PREFIX: &'static str = "ir::types::";
 ///
 /// All SSA values have a type that is described by an instance of `ValueType`
 /// or one of its subclasses.
-#[derive(Debug)]
+#[derive(Clone, Debug)]
 pub enum ValueType {
     BV(BVType),
     Lane(LaneType),
@@ -90,7 +90,7 @@ impl ValueType {
     }
 
     /// Return the name of this type for generated Rust source files.
-    pub fn _rust_name(&self) -> String {
+    pub fn rust_name(&self) -> String {
         format!("{}{}", _RUST_NAME_PREFIX, self.to_string().to_uppercase())
     }
 
@@ -205,6 +205,43 @@ impl LaneType {
                 LaneType::FloatType(shared_types::Float::F64) => 10,
             }
     }
+
+    pub fn bool_from_bits(num_bits: u16) -> LaneType {
+        LaneType::BoolType(match num_bits {
+            1 => shared_types::Bool::B1,
+            8 => shared_types::Bool::B8,
+            16 => shared_types::Bool::B16,
+            32 => shared_types::Bool::B32,
+            64 => shared_types::Bool::B64,
+            _ => unreachable!("unxpected num bits for bool"),
+        })
+    }
+
+    pub fn int_from_bits(num_bits: u16) -> LaneType {
+        LaneType::IntType(match num_bits {
+            8 => shared_types::Int::I8,
+            16 => shared_types::Int::I16,
+            32 => shared_types::Int::I32,
+            64 => shared_types::Int::I64,
+            _ => unreachable!("unxpected num bits for int"),
+        })
+    }
+
+    pub fn float_from_bits(num_bits: u16) -> LaneType {
+        LaneType::FloatType(match num_bits {
+            32 => shared_types::Float::F32,
+            64 => shared_types::Float::F64,
+            _ => unreachable!("unxpected num bits for float"),
+        })
+    }
+
+    pub fn by(&self, lanes: u16) -> ValueType {
+        if lanes == 1 {
+            (*self).into()
+        } else {
+            ValueType::Vector(VectorType::new(*self, lanes.into()))
+        }
+    }
 }
 
 impl fmt::Display for LaneType {
@@ -290,6 +327,7 @@ impl Iterator for LaneTypeIterator {
 ///
 /// A vector type has a lane type which is an instance of `LaneType`,
 /// and a positive number of lanes.
+#[derive(Clone)]
 pub struct VectorType {
     base: LaneType,
     lanes: u64,
@@ -320,6 +358,11 @@ impl VectorType {
         self.lanes
     }
 
+    /// Return the lane type.
+    pub fn lane_type(&self) -> LaneType {
+        self.base
+    }
+
     /// Find the unique number associated with this vector type.
     ///
     /// Vector types are encoded with the lane type in the low 4 bits and
@@ -350,14 +393,15 @@ impl fmt::Debug for VectorType {
 }
 
 /// A flat bitvector type. Used for semantics description only.
+#[derive(Clone)]
 pub struct BVType {
     bits: u64,
 }
 
 impl BVType {
     /// Initialize a new bitvector type with `n` bits.
-    pub fn _new(bits: u64) -> Self {
-        Self { bits }
+    pub fn new(bits: u16) -> Self {
+        Self { bits: bits.into() }
     }
 
     /// Return a string containing the documentation comment for this bitvector type.
@@ -386,7 +430,7 @@ impl fmt::Debug for BVType {
 /// A concrete scalar type that is neither a vector nor a lane type.
 ///
 /// Special types cannot be used to form vectors.
-#[derive(Clone, Copy)]
+#[derive(Clone, Copy, PartialEq, Eq, Hash)]
 pub enum SpecialType {
     Flag(shared_types::Flag),
 }
diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
new file mode 100644
index 0000000000..4db5c96e2a
--- /dev/null
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -0,0 +1,738 @@
+use std::collections::BTreeSet;
+use std::iter::FromIterator;
+use std::ops;
+use std::rc::Rc;
+
+use crate::cdsl::types::{BVType, LaneType, SpecialType, ValueType};
+
+const MAX_LANES: u16 = 256;
+const MAX_BITS: u16 = 64;
+const MAX_BITVEC: u16 = MAX_BITS * MAX_LANES;
+
+/// Type variables can be used in place of concrete types when defining
+/// instructions. This makes the instructions *polymorphic*.
+///
+/// A type variable is restricted to vary over a subset of the value types.
+/// This subset is specified by a set of flags that control the permitted base
+/// types and whether the type variable can assume scalar or vector types, or
+/// both.
+#[derive(Debug)]
+pub struct TypeVarContent {
+    /// Short name of type variable used in instruction descriptions.
+    pub name: String,
+
+    /// Documentation string.
+    pub doc: String,
+
+    /// Type set associated to the type variable.
+    pub type_set: Rc,
+
+    pub base: Option,
+}
+
+#[derive(Clone, Debug)]
+pub struct TypeVar {
+    content: Rc,
+}
+
+impl TypeVar {
+    fn get_typeset(&self) -> Rc {
+        match &self.content.base {
+            Some(base) => Rc::new(base.type_var.get_typeset().image(base.derived_func)),
+            None => self.content.type_set.clone(),
+        }
+    }
+
+    /// If the associated typeset has a single type return it. Otherwise return None.
+    pub fn singleton_type(&self) -> Option {
+        let type_set = self.get_typeset();
+        if type_set.size() == 1 {
+            Some(type_set.get_singleton())
+        } else {
+            None
+        }
+    }
+
+    /// Get the free type variable controlling this one.
+    pub fn free_typevar(&self) -> Option {
+        match &self.content.base {
+            Some(base) => base.type_var.free_typevar(),
+            None => {
+                match self.singleton_type() {
+                    // A singleton type isn't a proper free variable.
+                    Some(_) => None,
+                    None => Some(self.clone()),
+                }
+            }
+        }
+    }
+
+    /// Create a type variable that is a function of another.
+    fn derived(&self, derived_func: DerivedFunc) -> TypeVar {
+        let ts = self.content.type_set.clone();
+
+        // Safety checks to avoid over/underflows.
+        assert!(ts.specials.len() == 0, "can't derive from special types");
+        match derived_func {
+            DerivedFunc::HalfWidth => {
+                assert!(
+                    ts.ints.len() == 0 || *ts.ints.iter().min().unwrap() > 8,
+                    "can't halve all integer types"
+                );
+                assert!(
+                    ts.floats.len() == 0 || *ts.floats.iter().min().unwrap() > 32,
+                    "can't halve all float types"
+                );
+                assert!(
+                    ts.bools.len() == 0 || *ts.bools.iter().min().unwrap() > 8,
+                    "can't halve all boolean types"
+                );
+            }
+            DerivedFunc::DoubleWidth => {
+                assert!(
+                    ts.ints.len() == 0 || *ts.ints.iter().max().unwrap() < MAX_BITS,
+                    "can't double all integer types"
+                );
+                assert!(
+                    ts.floats.len() == 0 || *ts.floats.iter().max().unwrap() < MAX_BITS,
+                    "can't double all float types"
+                );
+                assert!(
+                    ts.bools.len() == 0 || *ts.bools.iter().max().unwrap() < MAX_BITS,
+                    "can't double all boolean types"
+                );
+            }
+            DerivedFunc::HalfVector => {
+                assert!(
+                    *ts.lanes.iter().min().unwrap() > 1,
+                    "can't halve a scalar type"
+                );
+            }
+            DerivedFunc::DoubleVector => {
+                assert!(
+                    *ts.lanes.iter().max().unwrap() < MAX_LANES,
+                    "can't double 256 lanes"
+                );
+            }
+            DerivedFunc::LaneOf | DerivedFunc::ToBitVec | DerivedFunc::AsBool => {
+                /* no particular assertions */
+            }
+        }
+
+        return TypeVar {
+            content: Rc::new(TypeVarContent {
+                name: "".into(), // XXX Python passes None to these two fields
+                doc: "".into(),
+                type_set: ts,
+                base: Some(TypeVarParent {
+                    type_var: self.clone(),
+                    derived_func,
+                }),
+            }),
+        };
+    }
+
+    pub fn lane_of(&self) -> TypeVar {
+        return self.derived(DerivedFunc::LaneOf);
+    }
+    pub fn as_bool(&self) -> TypeVar {
+        return self.derived(DerivedFunc::AsBool);
+    }
+    pub fn half_width(&self) -> TypeVar {
+        return self.derived(DerivedFunc::HalfWidth);
+    }
+    pub fn double_width(&self) -> TypeVar {
+        return self.derived(DerivedFunc::DoubleWidth);
+    }
+    pub fn half_vector(&self) -> TypeVar {
+        return self.derived(DerivedFunc::HalfVector);
+    }
+    pub fn double_vector(&self) -> TypeVar {
+        return self.derived(DerivedFunc::DoubleVector);
+    }
+    pub fn to_bitvec(&self) -> TypeVar {
+        return self.derived(DerivedFunc::ToBitVec);
+    }
+}
+
+impl Into for &TypeVar {
+    fn into(self) -> TypeVar {
+        self.clone()
+    }
+}
+impl Into for ValueType {
+    fn into(self) -> TypeVar {
+        TypeVarBuilder::singleton(self)
+    }
+}
+
+impl PartialEq for TypeVar {
+    fn eq(&self, other: &TypeVar) -> bool {
+        match (&self.content.base, &other.content.base) {
+            (Some(base1), Some(base2)) => base1.type_var.eq(&base2.type_var),
+            (None, None) => Rc::ptr_eq(&self.content, &other.content),
+            _ => false,
+        }
+    }
+}
+
+impl ops::Deref for TypeVar {
+    type Target = TypeVarContent;
+    fn deref(&self) -> &Self::Target {
+        &*self.content
+    }
+}
+
+#[derive(Clone, Copy, Debug)]
+pub enum DerivedFunc {
+    LaneOf,
+    AsBool,
+    HalfWidth,
+    DoubleWidth,
+    HalfVector,
+    DoubleVector,
+    ToBitVec,
+}
+
+impl DerivedFunc {
+    pub fn name(&self) -> &'static str {
+        match self {
+            DerivedFunc::LaneOf => "lane_of",
+            DerivedFunc::AsBool => "as_bool",
+            DerivedFunc::HalfWidth => "half_width",
+            DerivedFunc::DoubleWidth => "double_width",
+            DerivedFunc::HalfVector => "half_vector",
+            DerivedFunc::DoubleVector => "double_vector",
+            DerivedFunc::ToBitVec => "to_bitvec",
+        }
+    }
+}
+
+#[derive(Debug)]
+pub struct TypeVarParent {
+    pub type_var: TypeVar,
+    pub derived_func: DerivedFunc,
+}
+
+/// A set of types.
+///
+/// We don't allow arbitrary subsets of types, but use a parametrized approach
+/// instead.
+///
+/// Objects of this class can be used as dictionary keys.
+///
+/// Parametrized type sets are specified in terms of ranges:
+/// - The permitted range of vector lanes, where 1 indicates a scalar type.
+/// - The permitted range of integer types.
+/// - The permitted range of floating point types, and
+/// - The permitted range of boolean types.
+///
+/// The ranges are inclusive from smallest bit-width to largest bit-width.
+///
+/// Finally, a type set can contain special types (derived from `SpecialType`)
+/// which can't appear as lane types.
+
+type RangeBound = u16;
+type Range = ops::Range;
+type NumSet = BTreeSet;
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct TypeSet {
+    pub lanes: NumSet,
+    pub ints: NumSet,
+    pub floats: NumSet,
+    pub bools: NumSet,
+    pub bitvecs: NumSet,
+    pub specials: Vec,
+}
+
+impl TypeSet {
+    fn new(
+        lanes: NumSet,
+        ints: NumSet,
+        floats: NumSet,
+        bools: NumSet,
+        bitvecs: NumSet,
+        specials: Vec,
+    ) -> Self {
+        Self {
+            lanes,
+            ints,
+            floats,
+            bools,
+            bitvecs,
+            specials,
+        }
+    }
+
+    /// Return the number of concrete types represented by this typeset.
+    fn size(&self) -> usize {
+        self.lanes.len()
+            * (self.ints.len() + self.floats.len() + self.bools.len() + self.bitvecs.len())
+            + self.specials.len()
+    }
+
+    /// Return the image of self across the derived function func.
+    fn image(&self, derived_func: DerivedFunc) -> TypeSet {
+        match derived_func {
+            DerivedFunc::LaneOf => self.lane_of(),
+            DerivedFunc::AsBool => self.as_bool(),
+            DerivedFunc::HalfWidth => self.half_width(),
+            DerivedFunc::DoubleWidth => self.double_width(),
+            DerivedFunc::HalfVector => self.half_vector(),
+            DerivedFunc::DoubleVector => self.double_vector(),
+            DerivedFunc::ToBitVec => self.to_bitvec(),
+        }
+    }
+
+    /// Return a TypeSet describing the image of self across lane_of.
+    fn lane_of(&self) -> TypeSet {
+        let mut copy = self.clone();
+        copy.lanes = NumSet::from_iter(vec![1]);
+        copy.bitvecs = NumSet::new();
+        copy
+    }
+
+    /// Return a TypeSet describing the image of self across as_bool.
+    fn as_bool(&self) -> TypeSet {
+        let mut copy = self.clone();
+        copy.ints = NumSet::new();
+        copy.floats = NumSet::new();
+        copy.bitvecs = NumSet::new();
+        if (&self.lanes - &NumSet::from_iter(vec![1])).len() > 0 {
+            copy.bools = &self.ints | &self.floats;
+            copy.bools = ©.bools | &self.bools;
+        }
+        if self.lanes.contains(&1) {
+            copy.bools.insert(1);
+        }
+        copy
+    }
+
+    /// Return a TypeSet describing the image of self across halfwidth.
+    fn half_width(&self) -> TypeSet {
+        let mut copy = self.clone();
+        copy.ints = NumSet::from_iter(self.ints.iter().filter(|&&x| x > 8).map(|&x| x / 2));
+        copy.floats = NumSet::from_iter(self.floats.iter().filter(|&&x| x > 32).map(|&x| x / 2));
+        copy.bools = NumSet::from_iter(self.bools.iter().filter(|&&x| x > 8).map(|&x| x / 2));
+        copy.bitvecs = NumSet::from_iter(self.bitvecs.iter().filter(|&&x| x > 1).map(|&x| x / 2));
+        copy.specials = Vec::new();
+        copy
+    }
+
+    /// Return a TypeSet describing the image of self across doublewidth.
+    fn double_width(&self) -> TypeSet {
+        let mut copy = self.clone();
+        copy.ints = NumSet::from_iter(self.ints.iter().filter(|&&x| x < MAX_BITS).map(|&x| x * 2));
+        copy.floats = NumSet::from_iter(
+            self.floats
+                .iter()
+                .filter(|&&x| x < MAX_BITS)
+                .map(|&x| x * 2),
+        );
+        copy.bools = NumSet::from_iter(
+            self.bools
+                .iter()
+                .filter(|&&x| x < MAX_BITS)
+                .map(|&x| x * 2)
+                .filter(legal_bool),
+        );
+        copy.bitvecs = NumSet::from_iter(
+            self.bitvecs
+                .iter()
+                .filter(|&&x| x < MAX_BITVEC)
+                .map(|&x| x * 2),
+        );
+        copy.specials = Vec::new();
+        copy
+    }
+
+    /// Return a TypeSet describing the image of self across halfvector.
+    fn half_vector(&self) -> TypeSet {
+        let mut copy = self.clone();
+        copy.bitvecs = NumSet::new();
+        copy.lanes = NumSet::from_iter(self.lanes.iter().filter(|&&x| x > 1).map(|&x| x / 2));
+        copy.specials = Vec::new();
+        copy
+    }
+
+    /// Return a TypeSet describing the image of self across doublevector.
+    fn double_vector(&self) -> TypeSet {
+        let mut copy = self.clone();
+        copy.bitvecs = NumSet::new();
+        copy.lanes = NumSet::from_iter(
+            self.lanes
+                .iter()
+                .filter(|&&x| x < MAX_LANES)
+                .map(|&x| x * 2),
+        );
+        copy.specials = Vec::new();
+        copy
+    }
+
+    /// Return a TypeSet describing the image of self across to_bitvec.
+    fn to_bitvec(&self) -> TypeSet {
+        assert!(self.bitvecs.is_empty());
+        let all_scalars = &(&self.ints | &self.floats) | &self.bools;
+
+        let mut copy = self.clone();
+        copy.lanes = NumSet::from_iter(vec![1]);
+        copy.ints = NumSet::new();
+        copy.bools = NumSet::new();
+        copy.floats = NumSet::new();
+        copy.bitvecs = self
+            .lanes
+            .iter()
+            .cycle()
+            .zip(all_scalars.iter())
+            .map(|(num_lanes, lane_width)| num_lanes * lane_width)
+            .collect();
+
+        copy.specials = Vec::new();
+        copy
+    }
+
+    fn concrete_types(&self) -> Vec {
+        let mut ret = Vec::new();
+        for &num_lanes in &self.lanes {
+            for &bits in &self.ints {
+                ret.push(LaneType::int_from_bits(bits).by(num_lanes));
+            }
+            for &bits in &self.floats {
+                ret.push(LaneType::float_from_bits(bits).by(num_lanes));
+            }
+            for &bits in &self.bools {
+                ret.push(LaneType::bool_from_bits(bits).by(num_lanes));
+            }
+            for &bits in &self.bitvecs {
+                assert_eq!(num_lanes, 1);
+                ret.push(BVType::new(bits).into());
+            }
+        }
+        for &special in &self.specials {
+            ret.push(special.into());
+        }
+        ret
+    }
+
+    /// Return the singleton type represented by self. Can only call on typesets containing 1 type.
+    fn get_singleton(&self) -> ValueType {
+        let mut types = self.concrete_types();
+        assert_eq!(types.len(), 1);
+        return types.remove(0);
+    }
+}
+
+#[derive(PartialEq)]
+pub enum Interval {
+    None,
+    All,
+    Range(Range),
+}
+
+impl Interval {
+    fn to_range(&self, full_range: Range, default: Option) -> Option {
+        match self {
+            Interval::None => {
+                if let Some(default_val) = default {
+                    Some(default_val..default_val)
+                } else {
+                    None
+                }
+            }
+
+            Interval::All => Some(full_range),
+
+            Interval::Range(range) => {
+                let (low, high) = (range.start, range.end);
+                assert!(low.is_power_of_two());
+                assert!(high.is_power_of_two());
+                assert!(low <= high);
+                assert!(low >= full_range.start);
+                assert!(high <= full_range.end);
+                Some(low..high)
+            }
+        }
+    }
+}
+
+impl Into for Range {
+    fn into(self) -> Interval {
+        Interval::Range(self)
+    }
+}
+
+pub struct TypeVarBuilder {
+    name: String,
+    doc: String,
+    ints: Interval,
+    floats: Interval,
+    bools: Interval,
+    bitvecs: Interval,
+    includes_scalars: bool,
+    simd_lanes: Interval,
+    specials: Vec,
+}
+
+impl TypeVarBuilder {
+    pub fn new(name: impl Into, doc: impl Into) -> Self {
+        Self {
+            name: name.into(),
+            doc: doc.into(),
+            ints: Interval::None,
+            floats: Interval::None,
+            bools: Interval::None,
+            bitvecs: Interval::None,
+            includes_scalars: true,
+            simd_lanes: Interval::None,
+            specials: Vec::new(),
+        }
+    }
+
+    pub fn singleton(value_type: ValueType) -> TypeVar {
+        let mut builder = TypeVarBuilder::new(value_type.to_string(), value_type.doc());
+
+        let (scalar_type, num_lanes) = match value_type {
+            ValueType::BV(bitvec_type) => {
+                let bits = bitvec_type.lane_bits() as RangeBound;
+                return builder.bitvecs(bits..bits).finish();
+            }
+            ValueType::Special(special_type) => {
+                return builder.specials(vec![special_type]).finish();
+            }
+            ValueType::Lane(lane_type) => (lane_type, 1),
+            ValueType::Vector(vec_type) => {
+                (vec_type.lane_type(), vec_type.lane_count() as RangeBound)
+            }
+        };
+
+        builder = builder.simd_lanes(num_lanes..num_lanes);
+
+        match scalar_type {
+            LaneType::IntType(int_type) => {
+                let bits = int_type as RangeBound;
+                return builder.ints(bits..bits).finish();
+            }
+            LaneType::FloatType(float_type) => {
+                let bits = float_type as RangeBound;
+                return builder.floats(bits..bits).finish();
+            }
+            LaneType::BoolType(bool_type) => {
+                let bits = bool_type as RangeBound;
+                return builder.bools(bits..bits).finish();
+            }
+        }
+    }
+
+    pub fn ints(mut self, interval: impl Into) -> Self {
+        assert!(self.ints == Interval::None);
+        self.ints = interval.into();
+        self
+    }
+    pub fn floats(mut self, interval: impl Into) -> Self {
+        assert!(self.floats == Interval::None);
+        self.floats = interval.into();
+        self
+    }
+    pub fn bools(mut self, interval: impl Into) -> Self {
+        assert!(self.bools == Interval::None);
+        self.bools = interval.into();
+        self
+    }
+    pub fn includes_scalars(mut self, includes_scalars: bool) -> Self {
+        self.includes_scalars = includes_scalars;
+        self
+    }
+    pub fn simd_lanes(mut self, interval: impl Into) -> Self {
+        assert!(self.simd_lanes == Interval::None);
+        self.simd_lanes = interval.into();
+        self
+    }
+    pub fn bitvecs(mut self, interval: impl Into) -> Self {
+        assert!(self.bitvecs == Interval::None);
+        self.bitvecs = interval.into();
+        self
+    }
+    pub fn specials(mut self, specials: Vec) -> Self {
+        assert!(self.specials.is_empty());
+        self.specials = specials;
+        self
+    }
+
+    pub fn finish(self) -> TypeVar {
+        let min_lanes = if self.includes_scalars { 1 } else { 2 };
+
+        let bools = range_to_set(self.bools.to_range(1..MAX_BITS, None))
+            .into_iter()
+            .filter(legal_bool)
+            .collect();
+
+        let type_set = Rc::new(TypeSet::new(
+            range_to_set(self.simd_lanes.to_range(min_lanes..MAX_LANES, Some(1))),
+            range_to_set(self.ints.to_range(8..MAX_BITS, None)),
+            range_to_set(self.floats.to_range(32..64, None)),
+            bools,
+            range_to_set(self.bitvecs.to_range(1..MAX_BITVEC, None)),
+            self.specials,
+        ));
+
+        TypeVar {
+            content: Rc::new(TypeVarContent {
+                name: self.name.to_string(),
+                doc: self.doc.to_string(),
+                type_set,
+                base: None,
+            }),
+        }
+    }
+}
+
+fn legal_bool(bits: &RangeBound) -> bool {
+    // Only allow legal bit widths for bool types.
+    *bits == 1 || (*bits >= 8 && *bits <= MAX_BITS && bits.is_power_of_two())
+}
+
+/// Generates a set with all the powers of two included in the range.
+fn range_to_set(range: Option) -> NumSet {
+    let mut set = NumSet::new();
+
+    let (low, high) = match range {
+        Some(range) => (range.start, range.end),
+        None => return set,
+    };
+
+    assert!(low.is_power_of_two());
+    assert!(high.is_power_of_two());
+    assert!(low <= high);
+
+    for i in low.trailing_zeros()..high.trailing_zeros() + 1 {
+        assert!(1 << i <= RangeBound::max_value());
+        set.insert(1 << i);
+    }
+    set
+}
+
+#[test]
+fn test_typevar_builder() {
+    let typevar = TypeVarBuilder::new("test", "scalar integers")
+        .ints(Interval::All)
+        .finish();
+    assert_eq!(typevar.type_set.lanes, NumSet::from_iter(vec![1]));
+    assert!(typevar.type_set.floats.is_empty());
+    assert_eq!(
+        typevar.type_set.ints,
+        NumSet::from_iter(vec![8, 16, 32, 64])
+    );
+    assert!(typevar.type_set.bools.is_empty());
+    assert!(typevar.type_set.bitvecs.is_empty());
+    assert!(typevar.type_set.specials.is_empty());
+
+    let typevar = TypeVarBuilder::new("test", "scalar bools")
+        .bools(Interval::All)
+        .finish();
+    assert_eq!(typevar.type_set.lanes, NumSet::from_iter(vec![1]));
+    assert!(typevar.type_set.floats.is_empty());
+    assert!(typevar.type_set.ints.is_empty());
+    assert_eq!(
+        typevar.type_set.bools,
+        NumSet::from_iter(vec![1, 8, 16, 32, 64])
+    );
+    assert!(typevar.type_set.bitvecs.is_empty());
+    assert!(typevar.type_set.specials.is_empty());
+
+    let typevar = TypeVarBuilder::new("test", "scalar floats")
+        .floats(Interval::All)
+        .finish();
+    assert_eq!(typevar.type_set.lanes, NumSet::from_iter(vec![1]));
+    assert_eq!(typevar.type_set.floats, NumSet::from_iter(vec![32, 64]));
+    assert!(typevar.type_set.ints.is_empty());
+    assert!(typevar.type_set.bools.is_empty());
+    assert!(typevar.type_set.bitvecs.is_empty());
+    assert!(typevar.type_set.specials.is_empty());
+
+    let typevar = TypeVarBuilder::new("test", "float vectors (but not scalars)")
+        .floats(Interval::All)
+        .simd_lanes(Interval::All)
+        .includes_scalars(false)
+        .finish();
+    assert_eq!(
+        typevar.type_set.lanes,
+        NumSet::from_iter(vec![2, 4, 8, 16, 32, 64, 128, 256])
+    );
+    assert_eq!(typevar.type_set.floats, NumSet::from_iter(vec![32, 64]));
+    assert!(typevar.type_set.ints.is_empty());
+    assert!(typevar.type_set.bools.is_empty());
+    assert!(typevar.type_set.bitvecs.is_empty());
+    assert!(typevar.type_set.specials.is_empty());
+
+    let typevar = TypeVarBuilder::new("test", "float vectors and scalars")
+        .floats(Interval::All)
+        .simd_lanes(Interval::All)
+        .includes_scalars(true)
+        .finish();
+    assert_eq!(
+        typevar.type_set.lanes,
+        NumSet::from_iter(vec![1, 2, 4, 8, 16, 32, 64, 128, 256])
+    );
+    assert_eq!(typevar.type_set.floats, NumSet::from_iter(vec![32, 64]));
+    assert!(typevar.type_set.ints.is_empty());
+    assert!(typevar.type_set.bools.is_empty());
+    assert!(typevar.type_set.bitvecs.is_empty());
+    assert!(typevar.type_set.specials.is_empty());
+
+    let typevar = TypeVarBuilder::new("test", "range of ints")
+        .ints(16..64)
+        .finish();
+    assert_eq!(typevar.type_set.lanes, NumSet::from_iter(vec![1]));
+    assert_eq!(typevar.type_set.ints, NumSet::from_iter(vec![16, 32, 64]));
+    assert!(typevar.type_set.floats.is_empty());
+    assert!(typevar.type_set.bools.is_empty());
+    assert!(typevar.type_set.bitvecs.is_empty());
+    assert!(typevar.type_set.specials.is_empty());
+}
+
+#[test]
+#[should_panic]
+fn test_typevar_builder_too_high_bound_panic() {
+    TypeVarBuilder::new("test", "invalid range of ints")
+        .ints(16..2 * MAX_BITS)
+        .finish();
+}
+
+#[test]
+#[should_panic]
+fn test_typevar_builder_inverted_bounds_panic() {
+    TypeVarBuilder::new("test", "inverted bounds")
+        .ints(32..16)
+        .finish();
+}
+
+#[test]
+fn test_singleton() {
+    use crate::cdsl::types::VectorType;
+    use crate::shared::types as shared_types;
+
+    // Test i32.
+    let typevar =
+        TypeVarBuilder::singleton(ValueType::Lane(LaneType::IntType(shared_types::Int::I32)));
+    assert_eq!(typevar.name, "i32");
+    assert_eq!(typevar.type_set.ints, NumSet::from_iter(vec![32]));
+    assert!(typevar.type_set.floats.is_empty());
+    assert!(typevar.type_set.bools.is_empty());
+    assert!(typevar.type_set.bitvecs.is_empty());
+    assert!(typevar.type_set.specials.is_empty());
+    assert_eq!(typevar.type_set.lanes, NumSet::from_iter(vec![1]));
+
+    // Test f32x4.
+    let typevar = TypeVarBuilder::singleton(ValueType::Vector(VectorType::new(
+        LaneType::FloatType(shared_types::Float::F32),
+        4,
+    )));
+    assert_eq!(typevar.name, "f32x4");
+    assert!(typevar.type_set.ints.is_empty());
+    assert_eq!(typevar.type_set.floats, NumSet::from_iter(vec![32]));
+    assert_eq!(typevar.type_set.lanes, NumSet::from_iter(vec![4]));
+    assert!(typevar.type_set.bools.is_empty());
+    assert!(typevar.type_set.bitvecs.is_empty());
+    assert!(typevar.type_set.specials.is_empty());
+}

From d59bef1902a9be700d9d16f258c419483503b2b1 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Mar 2019 19:25:11 +0100
Subject: [PATCH 2359/3084] [meta] Port Formats and Operands to the Rust crate;

---
 cranelift/codegen/meta/src/cdsl/formats.rs    | 246 +++++++++++++++
 cranelift/codegen/meta/src/cdsl/mod.rs        |   2 +
 cranelift/codegen/meta/src/cdsl/operands.rs   | 285 ++++++++++++++++++
 cranelift/codegen/meta/src/shared/entities.rs |  65 ++++
 cranelift/codegen/meta/src/shared/formats.rs  | 184 +++++++++++
 .../codegen/meta/src/shared/immediates.rs     | 145 +++++++++
 cranelift/codegen/meta/src/shared/mod.rs      |  51 ++++
 7 files changed, 978 insertions(+)
 create mode 100644 cranelift/codegen/meta/src/cdsl/formats.rs
 create mode 100644 cranelift/codegen/meta/src/cdsl/operands.rs
 create mode 100644 cranelift/codegen/meta/src/shared/entities.rs
 create mode 100644 cranelift/codegen/meta/src/shared/formats.rs
 create mode 100644 cranelift/codegen/meta/src/shared/immediates.rs

diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
new file mode 100644
index 0000000000..837440d2c0
--- /dev/null
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -0,0 +1,246 @@
+use crate::cdsl::operands::{Operand, OperandKind};
+
+use std::collections::{HashMap, HashSet};
+use std::fmt;
+use std::slice;
+
+use cranelift_entity::{entity_impl, PrimaryMap};
+
+/// An immediate field in an instruction format.
+///
+/// This corresponds to a single member of a variant of the `InstructionData`
+/// data type.
+///
+/// :param iform: Parent `InstructionFormat`.
+/// :param kind: Immediate Operand kind.
+/// :param member: Member name in `InstructionData` variant.
+#[derive(Debug)]
+pub struct FormatField {
+    /// Immediate operand number in parent.
+    immnum: usize,
+
+    /// Immediate operand kind.
+    pub kind: OperandKind,
+
+    /// Member name in InstructionDate variant.
+    pub member: &'static str,
+}
+
+/// Every instruction opcode has a corresponding instruction format which
+/// determines the number of operands and their kinds. Instruction formats are
+/// identified structurally, i.e., the format of an instruction is derived from
+/// the kinds of operands used in its declaration.
+///
+/// The instruction format stores two separate lists of operands: Immediates
+/// and values. Immediate operands (including entity references) are
+/// represented as explicit members in the `InstructionData` variants. The
+/// value operands are stored differently, depending on how many there are.
+/// Beyond a certain point, instruction formats switch to an external value
+/// list for storing value arguments. Value lists can hold an arbitrary number
+/// of values.
+///
+/// All instruction formats must be predefined in the meta shared/formats module.
+///
+/// :param kinds: List of `OperandKind` objects describing the operands.
+/// :param name: Instruction format name in CamelCase. This is used as a Rust
+///     variant name in both the `InstructionData` and `InstructionFormat`
+///     enums.
+/// :param typevar_operand: Index of the value input operand that is used to
+///     infer the controlling type variable. By default, this is `0`, the first
+///     `value` operand. The index is relative to the values only, ignoring
+///     immediate operands.
+#[derive(Debug)]
+pub struct InstructionFormat {
+    pub name: &'static str,
+    pub num_value_operands: usize,
+    pub has_value_list: bool,
+    pub imm_fields: Vec,
+    pub typevar_operand: Option,
+}
+
+impl fmt::Display for InstructionFormat {
+    fn fmt(&self, fmt: &mut fmt::Formatter) -> Result<(), fmt::Error> {
+        let args = self
+            .imm_fields
+            .iter()
+            .map(|field| format!("{}: {}", field.member, field.kind.name))
+            .collect::>()
+            .join(", ");
+        fmt.write_fmt(format_args!(
+            "{}(imms=({}), vals={})",
+            self.name, args, self.num_value_operands
+        ))?;
+        Ok(())
+    }
+}
+
+pub struct InstructionFormatBuilder {
+    name: &'static str,
+    num_value_operands: usize,
+    has_value_list: bool,
+    imm_fields: Vec,
+    typevar_operand: Option,
+}
+
+pub struct ImmParameter {
+    kind: OperandKind,
+    member: &'static str,
+}
+impl Into for (&'static str, &OperandKind) {
+    fn into(self) -> ImmParameter {
+        ImmParameter {
+            kind: self.1.clone(),
+            member: self.0,
+        }
+    }
+}
+impl Into for &OperandKind {
+    fn into(self) -> ImmParameter {
+        ImmParameter {
+            kind: self.clone(),
+            member: self.default_member.unwrap(),
+        }
+    }
+}
+
+impl InstructionFormatBuilder {
+    pub fn new(name: &'static str) -> Self {
+        Self {
+            name,
+            num_value_operands: 0,
+            has_value_list: false,
+            imm_fields: Vec::new(),
+            typevar_operand: None,
+        }
+    }
+
+    pub fn value(mut self) -> Self {
+        self.num_value_operands += 1;
+        self
+    }
+
+    pub fn varargs(mut self) -> Self {
+        self.has_value_list = true;
+        self
+    }
+
+    pub fn imm(mut self, param: impl Into) -> Self {
+        let imm_param = param.into();
+        let field = FormatField {
+            immnum: self.imm_fields.len(),
+            kind: imm_param.kind,
+            member: imm_param.member,
+        };
+        self.imm_fields.push(field);
+        self
+    }
+
+    pub fn typevar_operand(mut self, operand_index: usize) -> Self {
+        assert!(self.typevar_operand.is_none());
+        assert!(self.has_value_list || operand_index < self.num_value_operands);
+        self.typevar_operand = Some(operand_index);
+        self
+    }
+
+    pub fn finish(self) -> InstructionFormat {
+        let typevar_operand = if self.typevar_operand.is_some() {
+            self.typevar_operand
+        } else if self.has_value_list || self.num_value_operands > 0 {
+            // Default to the first value operand, if there's one.
+            Some(0)
+        } else {
+            None
+        };
+
+        InstructionFormat {
+            name: self.name,
+            num_value_operands: self.num_value_operands,
+            has_value_list: self.has_value_list,
+            imm_fields: self.imm_fields,
+            typevar_operand,
+        }
+    }
+}
+
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct InstructionFormatIndex(u32);
+entity_impl!(InstructionFormatIndex);
+
+pub struct FormatRegistry {
+    /// Map (immediate kinds names, number of values, has varargs) to an instruction format index
+    /// in the actual map.
+    sig_to_index: HashMap<(Vec, usize, bool), InstructionFormatIndex>,
+    map: PrimaryMap,
+    name_set: HashSet<&'static str>,
+}
+
+impl FormatRegistry {
+    pub fn new() -> Self {
+        Self {
+            sig_to_index: HashMap::new(),
+            map: PrimaryMap::new(),
+            name_set: HashSet::new(),
+        }
+    }
+
+    /// Find an existing instruction format that matches the given lists of instruction inputs and
+    /// outputs.
+    pub fn lookup(&self, operands_in: &Vec) -> InstructionFormatIndex {
+        let mut imm_keys = Vec::new();
+        let mut num_values = 0;
+        let mut has_varargs = false;
+
+        for operand in operands_in.iter() {
+            if operand.is_value() {
+                num_values += 1;
+            }
+            has_varargs = has_varargs || operand.is_varargs();
+            if let Some(imm_key) = operand.kind.imm_key() {
+                imm_keys.push(imm_key);
+            }
+        }
+
+        let sig = (imm_keys, num_values, has_varargs);
+        *self
+            .sig_to_index
+            .get(&sig)
+            .expect("unknown InstructionFormat; please define it in shared/formats.rs first")
+    }
+
+    pub fn get(&self, index: InstructionFormatIndex) -> &InstructionFormat {
+        self.map.get(index).unwrap()
+    }
+
+    pub fn insert(&mut self, inst_format: InstructionFormatBuilder) {
+        let name = &inst_format.name;
+        if !self.name_set.insert(name) {
+            panic!(
+                "Trying to add an InstructionFormat named {}, but it already exists!",
+                name
+            );
+        }
+
+        let format = inst_format.finish();
+
+        // Compute key.
+        let imm_keys = format
+            .imm_fields
+            .iter()
+            .map(|field| field.kind.imm_key().unwrap())
+            .collect();
+        let key = (imm_keys, format.num_value_operands, format.has_value_list);
+
+        let index = self.map.push(format);
+        if let Some(already_inserted) = self.sig_to_index.insert(key, index) {
+            panic!(
+                "duplicate InstructionFormat: trying to insert '{}' while '{}' already has the same structure.",
+                self.map.get(index).unwrap().name,
+                self.map.get(already_inserted).unwrap().name
+            );
+        }
+    }
+
+    pub fn iter(&self) -> slice::Iter {
+        self.map.values()
+    }
+}
diff --git a/cranelift/codegen/meta/src/cdsl/mod.rs b/cranelift/codegen/meta/src/cdsl/mod.rs
index dd3a7ec859..fcfe977d14 100644
--- a/cranelift/codegen/meta/src/cdsl/mod.rs
+++ b/cranelift/codegen/meta/src/cdsl/mod.rs
@@ -3,7 +3,9 @@
 //! This module defines the classes that are used to define Cranelift
 //! instructions and other entities.
 
+pub mod formats;
 pub mod isa;
+pub mod operands;
 pub mod regs;
 pub mod settings;
 pub mod types;
diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
new file mode 100644
index 0000000000..f460225bfe
--- /dev/null
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -0,0 +1,285 @@
+use std::collections::HashMap;
+
+use crate::cdsl::camel_case;
+use crate::cdsl::typevar::TypeVar;
+
+/// An instruction operand can be an *immediate*, an *SSA value*, or an *entity reference*. The
+/// type of the operand is one of:
+///
+/// 1. A `ValueType` instance indicates an SSA value operand with a concrete type.
+///
+/// 2. A `TypeVar` instance indicates an SSA value operand, and the instruction is polymorphic over
+///    the possible concrete types that the type variable can assume.
+///
+/// 3. An `ImmediateKind` instance indicates an immediate operand whose value is encoded in the
+///    instruction itself rather than being passed as an SSA value.
+///
+/// 4. An `EntityRefKind` instance indicates an operand that references another entity in the
+///    function, typically something declared in the function preamble.
+#[derive(Clone, Debug)]
+pub struct Operand {
+    pub name: &'static str,
+    pub doc: Option,
+    pub kind: OperandKind,
+}
+
+impl Operand {
+    pub fn is_value(&self) -> bool {
+        match self.kind.fields {
+            OperandKindFields::TypeVar(_) => true,
+            _ => false,
+        }
+    }
+
+    pub fn type_var(&self) -> Option<&TypeVar> {
+        match &self.kind.fields {
+            OperandKindFields::TypeVar(typevar) => Some(typevar),
+            _ => None,
+        }
+    }
+
+    pub fn is_varargs(&self) -> bool {
+        match self.kind.fields {
+            OperandKindFields::VariableArgs => true,
+            _ => false,
+        }
+    }
+
+    /// Returns true if the operand has an immediate kind or is an EntityRef.
+    // TODO inherited name from the python, rename to is_immediate_or_entityref later.
+    pub fn is_immediate(&self) -> bool {
+        match self.kind.fields {
+            OperandKindFields::ImmEnum(_)
+            | OperandKindFields::ImmValue
+            | OperandKindFields::EntityRef => true,
+            _ => false,
+        }
+    }
+
+    /// Returns true if the operand has an immediate kind.
+    pub fn is_pure_immediate(&self) -> bool {
+        match self.kind.fields {
+            OperandKindFields::ImmEnum(_) | OperandKindFields::ImmValue => true,
+            _ => false,
+        }
+    }
+
+    pub fn is_cpu_flags(&self) -> bool {
+        match &self.kind.fields {
+            OperandKindFields::TypeVar(type_var)
+                if type_var.name == "iflags" || type_var.name == "fflags" =>
+            {
+                true
+            }
+            _ => false,
+        }
+    }
+}
+
+pub struct OperandBuilder {
+    name: &'static str,
+    doc: Option,
+    kind: OperandKind,
+}
+
+impl OperandBuilder {
+    pub fn new(name: &'static str, kind: OperandKind) -> Self {
+        Self {
+            name,
+            doc: None,
+            kind,
+        }
+    }
+    pub fn doc(mut self, doc: impl Into) -> Self {
+        assert!(self.doc.is_none());
+        self.doc = Some(doc.into());
+        self
+    }
+    pub fn finish(self) -> Operand {
+        let doc = match self.doc {
+            Some(doc) => Some(doc),
+            None => match &self.kind.fields {
+                OperandKindFields::TypeVar(tvar) => Some(tvar.doc.clone()),
+                _ => self.kind.doc.clone(),
+            },
+        };
+
+        Operand {
+            name: self.name,
+            doc,
+            kind: self.kind,
+        }
+    }
+}
+
+type EnumValues = HashMap<&'static str, &'static str>;
+
+#[derive(Clone, Debug)]
+pub enum OperandKindFields {
+    EntityRef,
+    VariableArgs,
+    ImmValue,
+    ImmEnum(EnumValues),
+    TypeVar(TypeVar),
+}
+
+#[derive(Clone, Debug)]
+pub struct OperandKind {
+    pub name: &'static str,
+
+    doc: Option,
+
+    pub default_member: Option<&'static str>,
+
+    /// The camel-cased name of an operand kind is also the Rust type used to represent it.
+    pub rust_type: String,
+
+    fields: OperandKindFields,
+}
+
+impl OperandKind {
+    pub fn imm_key(&self) -> Option {
+        match self.fields {
+            OperandKindFields::ImmEnum(_)
+            | OperandKindFields::ImmValue
+            | OperandKindFields::EntityRef => Some(self.name.to_string()),
+            _ => None,
+        }
+    }
+
+    pub fn type_var(&self) -> TypeVar {
+        match &self.fields {
+            OperandKindFields::TypeVar(tvar) => tvar.clone(),
+            _ => panic!("not a typevar"),
+        }
+    }
+}
+
+pub struct OperandKindBuilder {
+    name: &'static str,
+
+    doc: Option,
+
+    default_member: Option<&'static str>,
+
+    /// The camel-cased name of an operand kind is also the Rust type used to represent it.
+    rust_type: Option,
+
+    fields: OperandKindFields,
+}
+
+impl OperandKindBuilder {
+    pub fn new(name: &'static str, fields: OperandKindFields) -> Self {
+        Self {
+            name,
+            doc: None,
+            default_member: None,
+            rust_type: None,
+            fields,
+        }
+    }
+
+    pub fn new_imm(name: &'static str) -> Self {
+        Self {
+            name,
+            doc: None,
+            default_member: None,
+            rust_type: None,
+            fields: OperandKindFields::ImmValue,
+        }
+    }
+
+    pub fn new_enum(name: &'static str, values: EnumValues) -> Self {
+        Self {
+            name,
+            doc: None,
+            default_member: None,
+            rust_type: None,
+            fields: OperandKindFields::ImmEnum(values),
+        }
+    }
+
+    pub fn doc(mut self, doc: &'static str) -> Self {
+        assert!(self.doc.is_none());
+        self.doc = Some(doc.to_string());
+        self
+    }
+    pub fn default_member(mut self, default_member: &'static str) -> Self {
+        assert!(self.default_member.is_none());
+        self.default_member = Some(default_member);
+        self
+    }
+    pub fn rust_type(mut self, rust_type: &'static str) -> Self {
+        assert!(self.rust_type.is_none());
+        self.rust_type = Some(rust_type.to_string());
+        self
+    }
+
+    pub fn finish(self) -> OperandKind {
+        let default_member = match self.default_member {
+            Some(default_member) => Some(default_member),
+            None => match &self.fields {
+                OperandKindFields::ImmEnum(_) | OperandKindFields::ImmValue => Some("imm"),
+                OperandKindFields::TypeVar(_) | OperandKindFields::EntityRef => Some(self.name),
+                OperandKindFields::VariableArgs => None,
+            },
+        };
+
+        let rust_type = match self.rust_type {
+            Some(rust_type) => rust_type.to_string(),
+            None => match &self.fields {
+                OperandKindFields::ImmEnum(_) | OperandKindFields::ImmValue => {
+                    format!("ir::immediates::{}", camel_case(self.name))
+                }
+                OperandKindFields::VariableArgs => "&[Value]".to_string(),
+                OperandKindFields::TypeVar(_) | OperandKindFields::EntityRef => {
+                    format!("ir::{}", camel_case(self.name))
+                }
+            },
+        };
+
+        let doc = match self.doc {
+            Some(doc) => Some(doc),
+            None => match &self.fields {
+                OperandKindFields::TypeVar(type_var) => Some(type_var.doc.clone()),
+                OperandKindFields::ImmEnum(_)
+                | OperandKindFields::ImmValue
+                | OperandKindFields::EntityRef
+                | OperandKindFields::VariableArgs => None,
+            },
+        };
+
+        OperandKind {
+            name: self.name,
+            doc,
+            default_member,
+            rust_type,
+            fields: self.fields,
+        }
+    }
+}
+
+impl Into for &TypeVar {
+    fn into(self) -> OperandKind {
+        OperandKindBuilder::new("value", OperandKindFields::TypeVar(self.into())).finish()
+    }
+}
+impl Into for &OperandKind {
+    fn into(self) -> OperandKind {
+        self.clone()
+    }
+}
+
+/// Helper to create an operand in definitions files.
+pub fn create_operand(name: &'static str, kind: impl Into) -> Operand {
+    OperandBuilder::new(name, kind.into()).finish()
+}
+
+/// Helper to create an operand with a documentation in definitions files.
+pub fn create_operand_doc(
+    name: &'static str,
+    kind: impl Into,
+    doc: &'static str,
+) -> Operand {
+    OperandBuilder::new(name, kind.into()).doc(doc).finish()
+}
diff --git a/cranelift/codegen/meta/src/shared/entities.rs b/cranelift/codegen/meta/src/shared/entities.rs
new file mode 100644
index 0000000000..e3ff8ac705
--- /dev/null
+++ b/cranelift/codegen/meta/src/shared/entities.rs
@@ -0,0 +1,65 @@
+use crate::cdsl::operands::{OperandKind, OperandKindBuilder as Builder, OperandKindFields};
+
+/// Small helper to initialize an OperandBuilder with the right kind, for a given name and doc.
+fn create(name: &'static str, doc: &'static str) -> Builder {
+    Builder::new(name, OperandKindFields::EntityRef).doc(doc)
+}
+
+pub fn define() -> Vec {
+    let mut kinds = Vec::new();
+
+    // A reference to an extended basic block in the same function.
+    // This is primarliy used in control flow instructions.
+    let ebb = create("ebb", "An extended basic block in the same function.")
+        .default_member("destination")
+        .finish();
+    kinds.push(ebb);
+
+    // A reference to a stack slot declared in the function preamble.
+    let stack_slot = create("stack_slot", "A stack slot").finish();
+    kinds.push(stack_slot);
+
+    // A reference to a global value.
+    let global_value = create("global_value", "A global value.").finish();
+    kinds.push(global_value);
+
+    // A reference to a function signature declared in the function preamble.
+    // This is used to provide the call signature in a call_indirect instruction.
+    let sig_ref = create("sig_ref", "A function signature.").finish();
+    kinds.push(sig_ref);
+
+    // A reference to an external function declared in the function preamble.
+    // This is used to provide the callee and signature in a call instruction.
+    let func_ref = create("func_ref", "An external function.").finish();
+    kinds.push(func_ref);
+
+    // A reference to a jump table declared in the function preamble.
+    let jump_table = create("jump_table", "A jump table.")
+        .default_member("table")
+        .finish();
+    kinds.push(jump_table);
+
+    // A reference to a heap declared in the function preamble.
+    let heap = create("heap", "A heap.").finish();
+    kinds.push(heap);
+
+    // A reference to a table declared in the function preamble.
+    let table = create("table", "A table.").finish();
+    kinds.push(table);
+
+    // A variable-sized list of value operands. Use for Ebb and function call arguments.
+    let varargs = Builder::new("variable_args", OperandKindFields::VariableArgs)
+        .doc(
+            r#"
+            A variable size list of `value` operands.
+
+            Use this to represent arguments passed to a function call, arguments
+            passed to an extended basic block, or a variable number of results
+            returned from an instruction.
+        "#,
+        )
+        .finish();
+    kinds.push(varargs);
+
+    return kinds;
+}
diff --git a/cranelift/codegen/meta/src/shared/formats.rs b/cranelift/codegen/meta/src/shared/formats.rs
new file mode 100644
index 0000000000..0af3c264ae
--- /dev/null
+++ b/cranelift/codegen/meta/src/shared/formats.rs
@@ -0,0 +1,184 @@
+use crate::cdsl::formats::{FormatRegistry, InstructionFormatBuilder as Builder};
+use crate::shared::OperandKinds;
+
+pub fn define(immediates: &OperandKinds, entities: &OperandKinds) -> FormatRegistry {
+    // Shorthands for immediates.
+    let uimm8 = immediates.by_name("uimm8");
+    let uimm32 = immediates.by_name("uimm32");
+    let imm64 = immediates.by_name("imm64");
+    let ieee32 = immediates.by_name("ieee32");
+    let ieee64 = immediates.by_name("ieee64");
+    let boolean = immediates.by_name("boolean");
+    let intcc = immediates.by_name("intcc");
+    let floatcc = immediates.by_name("floatcc");
+    let memflags = immediates.by_name("memflags");
+    let offset32 = immediates.by_name("offset32");
+    let trapcode = immediates.by_name("trapcode");
+    let regunit = immediates.by_name("regunit");
+
+    // Shorthands for entities.
+    let global_value = entities.by_name("global_value");
+    let ebb = entities.by_name("ebb");
+    let jump_table = entities.by_name("jump_table");
+    let func_ref = entities.by_name("func_ref");
+    let sig_ref = entities.by_name("sig_ref");
+    let stack_slot = entities.by_name("stack_slot");
+    let heap = entities.by_name("heap");
+    let table = entities.by_name("table");
+
+    let mut registry = FormatRegistry::new();
+
+    registry.insert(Builder::new("Unary").value());
+    registry.insert(Builder::new("UnaryImm").imm(imm64));
+    registry.insert(Builder::new("UnaryIeee32").imm(ieee32));
+    registry.insert(Builder::new("UnaryIeee64").imm(ieee64));
+    registry.insert(Builder::new("UnaryBool").imm(boolean));
+    registry.insert(Builder::new("UnaryGlobalValue").imm(global_value));
+
+    registry.insert(Builder::new("Binary").value().value());
+    registry.insert(Builder::new("BinaryImm").value().imm(imm64));
+
+    // The select instructions are controlled by the second VALUE operand.
+    // The first VALUE operand is the controlling flag which has a derived type.
+    // The fma instruction has the same constraint on all inputs.
+    registry.insert(
+        Builder::new("Ternary")
+            .value()
+            .value()
+            .value()
+            .typevar_operand(1),
+    );
+
+    // Catch-all for instructions with many outputs and inputs and no immediate
+    // operands.
+    registry.insert(Builder::new("MultiAry").varargs());
+
+    registry.insert(Builder::new("NullAry"));
+
+    registry.insert(
+        Builder::new("InsertLane")
+            .value()
+            .imm(("lane", uimm8))
+            .value(),
+    );
+    registry.insert(Builder::new("ExtractLane").value().imm(("lane", uimm8)));
+
+    registry.insert(Builder::new("IntCompare").imm(intcc).value().value());
+    registry.insert(Builder::new("IntCompareImm").imm(intcc).value().imm(imm64));
+    registry.insert(Builder::new("IntCond").imm(intcc).value());
+
+    registry.insert(Builder::new("FloatCompare").imm(floatcc).value().value());
+    registry.insert(Builder::new("FloatCond").imm(floatcc).value());;
+
+    registry.insert(Builder::new("IntSelect").imm(intcc).value().value().value());
+
+    registry.insert(Builder::new("Jump").imm(ebb).varargs());
+    registry.insert(Builder::new("Branch").value().imm(ebb).varargs());
+    registry.insert(
+        Builder::new("BranchInt")
+            .imm(intcc)
+            .value()
+            .imm(ebb)
+            .varargs(),
+    );
+    registry.insert(
+        Builder::new("BranchFloat")
+            .imm(floatcc)
+            .value()
+            .imm(ebb)
+            .varargs(),
+    );
+    registry.insert(
+        Builder::new("BranchIcmp")
+            .imm(intcc)
+            .value()
+            .value()
+            .imm(ebb)
+            .varargs(),
+    );
+    registry.insert(Builder::new("BranchTable").value().imm(ebb).imm(jump_table));
+    registry.insert(
+        Builder::new("BranchTableEntry")
+            .value()
+            .value()
+            .imm(uimm8)
+            .imm(jump_table),
+    );
+    registry.insert(Builder::new("BranchTableBase").imm(jump_table));
+    registry.insert(Builder::new("IndirectJump").value().imm(jump_table));
+
+    registry.insert(Builder::new("Call").imm(func_ref).varargs());
+    registry.insert(Builder::new("CallIndirect").imm(sig_ref).value().varargs());
+    registry.insert(Builder::new("FuncAddr").imm(func_ref));
+
+    registry.insert(Builder::new("Load").imm(memflags).value().imm(offset32));
+    registry.insert(
+        Builder::new("LoadComplex")
+            .imm(memflags)
+            .varargs()
+            .imm(offset32),
+    );
+    registry.insert(
+        Builder::new("Store")
+            .imm(memflags)
+            .value()
+            .value()
+            .imm(offset32),
+    );
+    registry.insert(
+        Builder::new("StoreComplex")
+            .imm(memflags)
+            .value()
+            .varargs()
+            .imm(offset32),
+    );
+    registry.insert(Builder::new("StackLoad").imm(stack_slot).imm(offset32));
+    registry.insert(
+        Builder::new("StackStore")
+            .value()
+            .imm(stack_slot)
+            .imm(offset32),
+    );
+
+    // Accessing a WebAssembly heap.
+    registry.insert(Builder::new("HeapAddr").imm(heap).value().imm(uimm32));
+
+    // Accessing a WebAssembly table.
+    registry.insert(Builder::new("TableAddr").imm(table).value().imm(offset32));
+
+    registry.insert(
+        Builder::new("RegMove")
+            .value()
+            .imm(("src", regunit))
+            .imm(("dst", regunit)),
+    );
+    registry.insert(
+        Builder::new("CopySpecial")
+            .imm(("src", regunit))
+            .imm(("dst", regunit)),
+    );
+    registry.insert(
+        Builder::new("RegSpill")
+            .value()
+            .imm(("src", regunit))
+            .imm(("dst", stack_slot)),
+    );
+    registry.insert(
+        Builder::new("RegFill")
+            .value()
+            .imm(("src", stack_slot))
+            .imm(("dst", regunit)),
+    );
+
+    registry.insert(Builder::new("Trap").imm(trapcode));
+    registry.insert(Builder::new("CondTrap").value().imm(trapcode));
+    registry.insert(Builder::new("IntCondTrap").imm(intcc).value().imm(trapcode));
+    registry.insert(
+        Builder::new("FloatCondTrap")
+            .imm(floatcc)
+            .value()
+            .imm(trapcode),
+    );
+
+    registry
+}
diff --git a/cranelift/codegen/meta/src/shared/immediates.rs b/cranelift/codegen/meta/src/shared/immediates.rs
new file mode 100644
index 0000000000..077fe0a511
--- /dev/null
+++ b/cranelift/codegen/meta/src/shared/immediates.rs
@@ -0,0 +1,145 @@
+use crate::cdsl::operands::{OperandKind, OperandKindBuilder as Builder};
+
+use std::collections::HashMap;
+
+pub fn define() -> Vec {
+    let mut kinds = Vec::new();
+
+    // A 64-bit immediate integer operand.
+    //
+    // This type of immediate integer can interact with SSA values with any
+    // IntType type.
+    let imm64 = Builder::new_imm("imm64")
+        .doc("A 64-bit immediate integer.")
+        .finish();
+    kinds.push(imm64);
+
+    // An unsigned 8-bit immediate integer operand.
+    //
+    // This small operand is used to indicate lane indexes in SIMD vectors and
+    // immediate bit counts on shift instructions.
+    let uimm8 = Builder::new_imm("uimm8")
+        .doc("An 8-bit immediate unsigned integer.")
+        .finish();
+    kinds.push(uimm8);
+
+    // An unsigned 32-bit immediate integer operand.
+    let uimm32 = Builder::new_imm("uimm32")
+        .doc("A 32-bit immediate unsigned integer.")
+        .finish();
+    kinds.push(uimm32);
+
+    // A 32-bit immediate signed offset.
+    //
+    // This is used to represent an immediate address offset in load/store
+    // instructions.
+    let offset32 = Builder::new_imm("offset32")
+        .doc("A 32-bit immediate signed offset.")
+        .default_member("offset")
+        .finish();
+    kinds.push(offset32);
+
+    // A 32-bit immediate floating point operand.
+    //
+    // IEEE 754-2008 binary32 interchange format.
+    let ieee32 = Builder::new_imm("ieee32")
+        .doc("A 32-bit immediate floating point number.")
+        .finish();
+    kinds.push(ieee32);
+
+    // A 64-bit immediate floating point operand.
+    //
+    // IEEE 754-2008 binary64 interchange format.
+    let ieee64 = Builder::new_imm("ieee64")
+        .doc("A 64-bit immediate floating point number.")
+        .finish();
+    kinds.push(ieee64);
+
+    // An immediate boolean operand.
+    //
+    // This type of immediate boolean can interact with SSA values with any
+    // BoolType type.
+    let boolean = Builder::new_imm("boolean")
+        .doc("An immediate boolean.")
+        .rust_type("bool")
+        .finish();
+    kinds.push(boolean);
+
+    // A condition code for comparing integer values.
+    // This enumerated operand kind is used for the `icmp` instruction and corresponds to the
+    // condcodes::IntCC` Rust type.
+    let mut intcc_values = HashMap::new();
+    intcc_values.insert("eq", "Equal");
+    intcc_values.insert("ne", "NotEqual");
+    intcc_values.insert("sge", "UnsignedGreaterThanOrEqual");
+    intcc_values.insert("sgt", "UnsignedGreaterThan");
+    intcc_values.insert("sle", "UnsignedLessThanOrEqual");
+    intcc_values.insert("slt", "UnsignedLessThan");
+    intcc_values.insert("uge", "UnsignedGreaterThanOrEqual");
+    intcc_values.insert("ugt", "UnsignedGreaterThan");
+    intcc_values.insert("ule", "UnsignedLessThanOrEqual");
+    intcc_values.insert("ult", "UnsignedLessThan");
+    let intcc = Builder::new_enum("intcc", intcc_values)
+        .doc("An integer comparison condition code.")
+        .default_member("cond")
+        .rust_type("ir::condcodes::IntCC")
+        .finish();
+    kinds.push(intcc);
+
+    // A condition code for comparing floating point values.  This enumerated operand kind is used
+    // for the `fcmp` instruction and corresponds to the `condcodes::FloatCC` Rust type.
+    let mut floatcc_values = HashMap::new();
+    floatcc_values.insert("ord", "Ordered");
+    floatcc_values.insert("uno", "Unordered");
+    floatcc_values.insert("eq", "Equal");
+    floatcc_values.insert("ne", "NotEqual");
+    floatcc_values.insert("one", "OrderedNotEqual");
+    floatcc_values.insert("ueq", "UnorderedOrEqual");
+    floatcc_values.insert("lt", "LessThan");
+    floatcc_values.insert("le", "LessThanOrEqual");
+    floatcc_values.insert("gt", "GreaterThan");
+    floatcc_values.insert("ge", "GreaterThanOrEqual");
+    floatcc_values.insert("ult", "UnorderedOrLessThan");
+    floatcc_values.insert("ule", "UnorderedOrLessThanOrEqual");
+    floatcc_values.insert("ugt", "UnorderedOrGreaterThan");
+    floatcc_values.insert("uge", "UnorderedOrGreaterThanOrEqual");
+    let floatcc = Builder::new_enum("floatcc", floatcc_values)
+        .doc("A floating point comparison condition code")
+        .default_member("cond")
+        .rust_type("ir::condcodes::FloatCC")
+        .finish();
+    kinds.push(floatcc);
+
+    // Flags for memory operations like :clif:inst:`load` and :clif:inst:`store`.
+    let memflags = Builder::new_imm("memflags")
+        .doc("Memory operation flags")
+        .default_member("flags")
+        .rust_type("ir::MemFlags")
+        .finish();
+    kinds.push(memflags);
+
+    // A register unit in the current target ISA.
+    let regunit = Builder::new_imm("regunit")
+        .doc("A register unit in the target ISA")
+        .rust_type("isa::RegUnit")
+        .finish();
+    kinds.push(regunit);
+
+    // A trap code indicating the reason for trapping.
+    //
+    // The Rust enum type also has a `User(u16)` variant for user-provided trap
+    // codes.
+    let mut trapcode_values = HashMap::new();
+    trapcode_values.insert("stk_ovf", "StackOverflow");
+    trapcode_values.insert("heap_oob", "HeapOutOfBounds");
+    trapcode_values.insert("int_ovf", "IntegerOverflow");
+    trapcode_values.insert("int_divz", "IntegerDivisionByZero");
+    let trapcode = Builder::new_enum("trapcode", trapcode_values)
+        .doc("A trap reason code.")
+        .default_member("code")
+        .rust_type("ir::TrapCode")
+        .finish();
+    kinds.push(trapcode);
+
+    return kinds;
+}
diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
index bdd9a14f2c..c8dd17ff4b 100644
--- a/cranelift/codegen/meta/src/shared/mod.rs
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -1,4 +1,55 @@
 //! Shared definitions for the Cranelift intermediate language.
 
+pub mod entities;
+pub mod formats;
+pub mod immediates;
 pub mod settings;
 pub mod types;
+
+use crate::cdsl::formats::FormatRegistry;
+use crate::cdsl::operands::OperandKind;
+use crate::cdsl::settings::SettingGroup;
+
+pub struct Definitions {
+    pub settings: SettingGroup,
+    pub operand_kinds: OperandKinds,
+    pub format_registry: FormatRegistry,
+}
+
+pub struct OperandKinds(Vec);
+
+impl OperandKinds {
+    pub fn new() -> Self {
+        Self(Vec::new())
+    }
+
+    pub fn by_name(&self, name: &'static str) -> &OperandKind {
+        self.0
+            .iter()
+            .find(|op| op.name == name)
+            .expect(&format!("unknown Operand name: {}", name))
+    }
+
+    pub fn push(&mut self, operand_kind: OperandKind) {
+        assert!(
+            self.0
+                .iter()
+                .find(|existing| existing.name == operand_kind.name)
+                .is_none(),
+            "trying to insert operand kind '{}' for the second time",
+            operand_kind.name
+        );
+        self.0.push(operand_kind);
+    }
+}
+
+pub fn define() -> Definitions {
+    let immediates = OperandKinds(immediates::define());
+    let entities = OperandKinds(entities::define());
+    let format_registry = formats::define(&immediates, &entities);
+    Definitions {
+        settings: settings::define(),
+        operand_kinds: immediates,
+        format_registry,
+    }
+}

From 208c4e6da64abd6b9ac74b0e20743f029ba45035 Mon Sep 17 00:00:00 2001
From: Lars T Hansen 
Date: Thu, 28 Mar 2019 13:50:45 +0100
Subject: [PATCH 2360/3084] Fix a disasm formatting error

---
 cranelift/src/disasm.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/src/disasm.rs b/cranelift/src/disasm.rs
index fa55ab58a2..65fb33ec06 100644
--- a/cranelift/src/disasm.rs
+++ b/cranelift/src/disasm.rs
@@ -131,10 +131,10 @@ cfg_if! {
                 let mut len = 0;
                 let mut first = true;
                 for b in i.bytes() {
-                    write!(&mut bytes_str, "{:02x}", b).unwrap();
                     if !first {
                         write!(&mut bytes_str, " ").unwrap();
                     }
+                    write!(&mut bytes_str, "{:02x}", b).unwrap();
                     len += 1;
                     first = false;
                 }

From 3c31eac48c706c6be24ca5c58ba6ac988c894232 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Mar 2019 19:36:45 +0100
Subject: [PATCH 2361/3084] [meta] Port Instruction/InstructionGroup to the
 Rust meta crate;

---
 cranelift/codegen/meta/src/cdsl/inst.rs       | 458 ++++++++++++++++++
 cranelift/codegen/meta/src/cdsl/isa.rs        |  14 +-
 cranelift/codegen/meta/src/cdsl/mod.rs        |   2 +
 .../codegen/meta/src/cdsl/type_inference.rs   |   5 +
 cranelift/codegen/meta/src/isa/arm32/mod.rs   |  11 +-
 cranelift/codegen/meta/src/isa/arm64/mod.rs   |  11 +-
 cranelift/codegen/meta/src/isa/mod.rs         |  12 +-
 cranelift/codegen/meta/src/isa/riscv/mod.rs   |  11 +-
 .../codegen/meta/src/isa/x86/instructions.rs  | 244 ++++++++++
 cranelift/codegen/meta/src/isa/x86/mod.rs     |  13 +-
 cranelift/codegen/meta/src/lib.rs             |   6 +-
 11 files changed, 763 insertions(+), 24 deletions(-)
 create mode 100644 cranelift/codegen/meta/src/cdsl/inst.rs
 create mode 100644 cranelift/codegen/meta/src/cdsl/type_inference.rs
 create mode 100644 cranelift/codegen/meta/src/isa/x86/instructions.rs

diff --git a/cranelift/codegen/meta/src/cdsl/inst.rs b/cranelift/codegen/meta/src/cdsl/inst.rs
new file mode 100644
index 0000000000..4199d6424d
--- /dev/null
+++ b/cranelift/codegen/meta/src/cdsl/inst.rs
@@ -0,0 +1,458 @@
+use crate::cdsl::camel_case;
+use crate::cdsl::formats::{FormatRegistry, InstructionFormat, InstructionFormatIndex};
+use crate::cdsl::operands::Operand;
+use crate::cdsl::type_inference::Constraint;
+use crate::cdsl::typevar::TypeVar;
+
+use std::fmt;
+use std::slice;
+
+/// Every instruction must belong to exactly one instruction group. A given
+/// target architecture can support instructions from multiple groups, and it
+/// does not necessarily support all instructions in a group.
+pub struct InstructionGroup {
+    _name: &'static str,
+    _doc: &'static str,
+    instructions: Vec,
+}
+
+impl InstructionGroup {
+    pub fn new(name: &'static str, doc: &'static str) -> Self {
+        Self {
+            _name: name,
+            _doc: doc,
+            instructions: Vec::new(),
+        }
+    }
+
+    pub fn push(&mut self, inst: Instruction) {
+        self.instructions.push(inst);
+    }
+
+    pub fn iter(&self) -> slice::Iter {
+        self.instructions.iter()
+    }
+}
+
+pub struct PolymorphicInfo {
+    pub use_typevar_operand: bool,
+    pub ctrl_typevar: TypeVar,
+    pub other_typevars: Vec,
+}
+
+pub struct Instruction {
+    /// Instruction mnemonic, also becomes opcode name.
+    pub name: &'static str,
+    pub camel_name: String,
+
+    /// Documentation string.
+    doc: &'static str,
+
+    /// Input operands. This can be a mix of SSA value operands and other operand kinds.
+    pub operands_in: Vec,
+    /// Output operands. The output operands must be SSA values or `variable_args`.
+    pub operands_out: Vec,
+    /// Instruction-specific TypeConstraints.
+    _constraints: Vec,
+
+    /// Instruction format, automatically derived from the input operands.
+    pub format: InstructionFormatIndex,
+
+    /// One of the input or output operands is a free type variable. None if the instruction is not
+    /// polymorphic, set otherwise.
+    pub polymorphic_info: Option,
+
+    pub value_opnums: Vec,
+    pub value_results: Vec,
+    pub imm_opnums: Vec,
+
+    /// True for instructions that terminate the EBB.
+    pub is_terminator: bool,
+    /// True for all branch or jump instructions.
+    pub is_branch: bool,
+    /// True for all indirect branch or jump instructions.',
+    pub is_indirect_branch: bool,
+    /// Is this a call instruction?
+    pub is_call: bool,
+    /// Is this a return instruction?
+    pub is_return: bool,
+    /// Is this a ghost instruction?
+    pub is_ghost: bool,
+    /// Can this instruction read from memory?
+    pub can_load: bool,
+    /// Can this instruction write to memory?
+    pub can_store: bool,
+    /// Can this instruction cause a trap?
+    pub can_trap: bool,
+    /// Does this instruction have other side effects besides can_* flags?
+    pub other_side_effects: bool,
+    /// Does this instruction write to CPU flags?
+    pub writes_cpu_flags: bool,
+}
+
+impl Instruction {
+    pub fn snake_name(&self) -> &'static str {
+        if self.name == "return" {
+            "return_"
+        } else {
+            self.name
+        }
+    }
+
+    pub fn doc_comment_first_line(&self) -> &'static str {
+        for line in self.doc.split("\n") {
+            let stripped = line.trim();
+            if stripped.len() > 0 {
+                return stripped;
+            }
+        }
+        ""
+    }
+}
+
+impl fmt::Display for Instruction {
+    fn fmt(&self, fmt: &mut fmt::Formatter) -> Result<(), fmt::Error> {
+        if self.operands_out.len() > 0 {
+            let operands_out = self
+                .operands_out
+                .iter()
+                .map(|op| op.name)
+                .collect::>()
+                .join(", ");
+            fmt.write_str(&operands_out)?;
+            fmt.write_str(" = ")?;
+        }
+
+        fmt.write_str(self.name)?;
+
+        if self.operands_in.len() > 0 {
+            let operands_in = self
+                .operands_in
+                .iter()
+                .map(|op| op.name)
+                .collect::>()
+                .join(", ");
+            fmt.write_str(" ")?;
+            fmt.write_str(&operands_in)?;
+        }
+
+        Ok(())
+    }
+}
+
+pub struct InstructionBuilder {
+    name: &'static str,
+    doc: &'static str,
+    operands_in: Option>,
+    operands_out: Option>,
+    constraints: Option>,
+
+    // See Instruction comments for the meaning of these fields.
+    is_terminator: bool,
+    is_branch: bool,
+    is_indirect_branch: bool,
+    is_call: bool,
+    is_return: bool,
+    is_ghost: bool,
+    can_load: bool,
+    can_store: bool,
+    can_trap: bool,
+    other_side_effects: bool,
+}
+
+impl InstructionBuilder {
+    pub fn new(name: &'static str, doc: &'static str) -> Self {
+        Self {
+            name,
+            doc,
+            operands_in: None,
+            operands_out: None,
+            constraints: None,
+
+            is_terminator: false,
+            is_branch: false,
+            is_indirect_branch: false,
+            is_call: false,
+            is_return: false,
+            is_ghost: false,
+            can_load: false,
+            can_store: false,
+            can_trap: false,
+            other_side_effects: false,
+        }
+    }
+
+    pub fn operands_in(mut self, operands: Vec<&Operand>) -> Self {
+        assert!(self.operands_in.is_none());
+        self.operands_in = Some(operands.iter().map(|x| (*x).clone()).collect());
+        self
+    }
+    pub fn operands_out(mut self, operands: Vec<&Operand>) -> Self {
+        assert!(self.operands_out.is_none());
+        self.operands_out = Some(operands.iter().map(|x| (*x).clone()).collect());
+        self
+    }
+    pub fn constraints(mut self, constraints: Vec) -> Self {
+        assert!(self.constraints.is_none());
+        self.constraints = Some(constraints);
+        self
+    }
+
+    pub fn is_terminator(mut self, val: bool) -> Self {
+        self.is_terminator = val;
+        self
+    }
+    pub fn is_branch(mut self, val: bool) -> Self {
+        self.is_branch = val;
+        self
+    }
+    pub fn is_indirect_branch(mut self, val: bool) -> Self {
+        self.is_indirect_branch = val;
+        self
+    }
+    pub fn is_call(mut self, val: bool) -> Self {
+        self.is_call = val;
+        self
+    }
+    pub fn is_return(mut self, val: bool) -> Self {
+        self.is_return = val;
+        self
+    }
+    pub fn is_ghost(mut self, val: bool) -> Self {
+        self.is_ghost = val;
+        self
+    }
+    pub fn can_load(mut self, val: bool) -> Self {
+        self.can_load = val;
+        self
+    }
+    pub fn can_store(mut self, val: bool) -> Self {
+        self.can_store = val;
+        self
+    }
+    pub fn can_trap(mut self, val: bool) -> Self {
+        self.can_trap = val;
+        self
+    }
+    pub fn other_side_effects(mut self, val: bool) -> Self {
+        self.other_side_effects = val;
+        self
+    }
+
+    pub fn finish(self, format_registry: &FormatRegistry) -> Instruction {
+        let operands_in = self.operands_in.unwrap_or_else(Vec::new);
+        let operands_out = self.operands_out.unwrap_or_else(Vec::new);
+
+        let format_index = format_registry.lookup(&operands_in);
+
+        let mut value_opnums = Vec::new();
+        let mut imm_opnums = Vec::new();
+        for (i, op) in operands_in.iter().enumerate() {
+            if op.is_value() {
+                value_opnums.push(i);
+            } else if op.is_immediate() {
+                imm_opnums.push(i);
+            } else {
+                assert!(op.is_varargs());
+            }
+        }
+
+        let mut value_results = Vec::new();
+        for (i, op) in operands_out.iter().enumerate() {
+            if op.is_value() {
+                value_results.push(i);
+            }
+        }
+
+        let format = format_registry.get(format_index);
+        let polymorphic_info =
+            verify_polymorphic(&operands_in, &operands_out, &format, &value_opnums);
+
+        // Infer from output operands whether an instruciton clobbers CPU flags or not.
+        let writes_cpu_flags = operands_out.iter().any(|op| op.is_cpu_flags());
+
+        Instruction {
+            name: self.name,
+            camel_name: camel_case(self.name),
+            doc: self.doc,
+            operands_in,
+            operands_out,
+            _constraints: self.constraints.unwrap_or_else(Vec::new),
+            format: format_index,
+            polymorphic_info,
+            value_opnums,
+            value_results,
+            imm_opnums,
+            is_terminator: self.is_terminator,
+            is_branch: self.is_branch,
+            is_indirect_branch: self.is_indirect_branch,
+            is_call: self.is_call,
+            is_return: self.is_return,
+            is_ghost: self.is_ghost,
+            can_load: self.can_load,
+            can_store: self.can_store,
+            can_trap: self.can_trap,
+            other_side_effects: self.other_side_effects,
+            writes_cpu_flags,
+        }
+    }
+}
+
+/// Check if this instruction is polymorphic, and verify its use of type variables.
+fn verify_polymorphic(
+    operands_in: &Vec,
+    operands_out: &Vec,
+    format: &InstructionFormat,
+    value_opnums: &Vec,
+) -> Option {
+    // The instruction is polymorphic if it has one free input or output operand.
+    let is_polymorphic = operands_in
+        .iter()
+        .any(|op| op.is_value() && op.type_var().unwrap().free_typevar().is_some())
+        || operands_out
+            .iter()
+            .any(|op| op.is_value() && op.type_var().unwrap().free_typevar().is_some());
+
+    if !is_polymorphic {
+        return None;
+    }
+
+    // Verify the use of type variables.
+    let mut use_typevar_operand = false;
+    let mut ctrl_typevar = None;
+    let mut other_typevars = None;
+    let mut maybe_error_message = None;
+
+    let tv_op = format.typevar_operand;
+    if let Some(tv_op) = tv_op {
+        if tv_op < value_opnums.len() {
+            let op_num = value_opnums[tv_op];
+            let tv = operands_in[op_num].type_var().unwrap();
+            let free_typevar = tv.free_typevar();
+            if (free_typevar.is_some() && tv == &free_typevar.unwrap())
+                || !tv.singleton_type().is_none()
+            {
+                match verify_ctrl_typevar(tv, &value_opnums, &operands_in, &operands_out) {
+                    Ok(typevars) => {
+                        other_typevars = Some(typevars);
+                        ctrl_typevar = Some(tv.clone());
+                        use_typevar_operand = true;
+                    }
+                    Err(error_message) => {
+                        maybe_error_message = Some(error_message);
+                    }
+                }
+            }
+        }
+    };
+
+    if !use_typevar_operand {
+        if operands_out.len() == 0 {
+            match maybe_error_message {
+                Some(msg) => panic!(msg),
+                None => panic!("typevar_operand must be a free type variable"),
+            }
+        }
+
+        let tv = operands_out[0].type_var().unwrap();
+        let free_typevar = tv.free_typevar();
+        if free_typevar.is_some() && tv != &free_typevar.unwrap() {
+            panic!("first result must be a free type variable");
+        }
+
+        other_typevars =
+            Some(verify_ctrl_typevar(tv, &value_opnums, &operands_in, &operands_out).unwrap());
+        ctrl_typevar = Some(tv.clone());
+    }
+
+    // rustc is not capable to determine this statically, so enforce it with options.
+    assert!(ctrl_typevar.is_some());
+    assert!(other_typevars.is_some());
+
+    Some(PolymorphicInfo {
+        use_typevar_operand,
+        ctrl_typevar: ctrl_typevar.unwrap(),
+        other_typevars: other_typevars.unwrap(),
+    })
+}
+
+/// Verify that the use of TypeVars is consistent with `ctrl_typevar` as the controlling type
+/// variable.
+///
+/// All polymorhic inputs must either be derived from `ctrl_typevar` or be independent free type
+/// variables only used once.
+///
+/// All polymorphic results must be derived from `ctrl_typevar`.
+///
+/// Return a vector of other type variables used, or panics.
+fn verify_ctrl_typevar(
+    ctrl_typevar: &TypeVar,
+    value_opnums: &Vec,
+    operands_in: &Vec,
+    operands_out: &Vec,
+) -> Result, String> {
+    let mut other_typevars = Vec::new();
+
+    // Check value inputs.
+    for &op_num in value_opnums {
+        let typ = operands_in[op_num].type_var();
+
+        let tv = if let Some(typ) = typ {
+            typ.free_typevar()
+        } else {
+            None
+        };
+
+        // Non-polymorphic or derived from ctrl_typevar is OK.
+        let tv = match tv {
+            Some(tv) => {
+                if &tv == ctrl_typevar {
+                    continue;
+                }
+                tv
+            }
+            None => continue,
+        };
+
+        // No other derived typevars allowed.
+        if typ.is_some() && typ.unwrap() != &tv {
+            return Err(format!(
+                "{:?}: type variable {} must be derived from {:?}",
+                operands_in[op_num],
+                typ.unwrap().name,
+                ctrl_typevar
+            ));
+        }
+
+        // Other free type variables can only be used once each.
+        for other_tv in &other_typevars {
+            if &tv == other_tv {
+                return Err(format!(
+                    "type variable {} can't be used more than once",
+                    tv.name
+                ));
+            }
+        }
+
+        other_typevars.push(tv);
+    }
+
+    // Check outputs.
+    for result in operands_out {
+        if !result.is_value() {
+            continue;
+        }
+
+        let typ = result.type_var().unwrap();
+        let tv = typ.free_typevar();
+
+        // Non-polymorphic or derived form ctrl_typevar is OK.
+        if tv.is_none() || &tv.unwrap() == ctrl_typevar {
+            continue;
+        }
+
+        return Err("type variable in output not derived from ctrl_typevar".into());
+    }
+
+    Ok(other_typevars)
+}
diff --git a/cranelift/codegen/meta/src/cdsl/isa.rs b/cranelift/codegen/meta/src/cdsl/isa.rs
index 86b40a12c3..1091f6db80 100644
--- a/cranelift/codegen/meta/src/cdsl/isa.rs
+++ b/cranelift/codegen/meta/src/cdsl/isa.rs
@@ -1,16 +1,24 @@
-use super::regs::IsaRegs;
-use super::settings::SettingGroup;
+use crate::cdsl::inst::InstructionGroup;
+use crate::cdsl::regs::IsaRegs;
+use crate::cdsl::settings::SettingGroup;
 
 pub struct TargetIsa {
     pub name: &'static str,
+    pub instructions: InstructionGroup,
     pub settings: SettingGroup,
     pub regs: IsaRegs,
 }
 
 impl TargetIsa {
-    pub fn new(name: &'static str, settings: SettingGroup, regs: IsaRegs) -> Self {
+    pub fn new(
+        name: &'static str,
+        instructions: InstructionGroup,
+        settings: SettingGroup,
+        regs: IsaRegs,
+    ) -> Self {
         Self {
             name,
+            instructions,
             settings,
             regs,
         }
diff --git a/cranelift/codegen/meta/src/cdsl/mod.rs b/cranelift/codegen/meta/src/cdsl/mod.rs
index fcfe977d14..540370624a 100644
--- a/cranelift/codegen/meta/src/cdsl/mod.rs
+++ b/cranelift/codegen/meta/src/cdsl/mod.rs
@@ -4,10 +4,12 @@
 //! instructions and other entities.
 
 pub mod formats;
+pub mod inst;
 pub mod isa;
 pub mod operands;
 pub mod regs;
 pub mod settings;
+pub mod type_inference;
 pub mod types;
 pub mod typevar;
 
diff --git a/cranelift/codegen/meta/src/cdsl/type_inference.rs b/cranelift/codegen/meta/src/cdsl/type_inference.rs
new file mode 100644
index 0000000000..de104f4b47
--- /dev/null
+++ b/cranelift/codegen/meta/src/cdsl/type_inference.rs
@@ -0,0 +1,5 @@
+use crate::cdsl::typevar::TypeVar;
+
+pub enum Constraint {
+    WiderOrEq(TypeVar, TypeVar),
+}
diff --git a/cranelift/codegen/meta/src/isa/arm32/mod.rs b/cranelift/codegen/meta/src/isa/arm32/mod.rs
index e3decd4da0..8c32cdcdcd 100644
--- a/cranelift/codegen/meta/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm32/mod.rs
@@ -1,6 +1,8 @@
+use crate::cdsl::inst::InstructionGroup;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
+use crate::shared::Definitions as SharedDefinitions;
 
 fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     let setting = SettingGroupBuilder::new("arm32");
@@ -44,8 +46,11 @@ fn define_regs() -> IsaRegs {
     regs.finish()
 }
 
-pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
-    let settings = define_settings(shared_settings);
+pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
+    let settings = define_settings(&shared_defs.settings);
     let regs = define_regs();
-    TargetIsa::new("arm32", settings, regs)
+
+    let inst_group = InstructionGroup::new("arm32", "arm32 specific instruction set");
+
+    TargetIsa::new("arm32", inst_group, settings, regs)
 }
diff --git a/cranelift/codegen/meta/src/isa/arm64/mod.rs b/cranelift/codegen/meta/src/isa/arm64/mod.rs
index baca5226ab..46baac90e6 100644
--- a/cranelift/codegen/meta/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm64/mod.rs
@@ -1,6 +1,8 @@
+use crate::cdsl::inst::InstructionGroup;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
+use crate::shared::Definitions as SharedDefinitions;
 
 fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     let setting = SettingGroupBuilder::new("arm64");
@@ -40,8 +42,11 @@ fn define_registers() -> IsaRegs {
     regs.finish()
 }
 
-pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
-    let settings = define_settings(shared_settings);
+pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
+    let settings = define_settings(&shared_defs.settings);
     let regs = define_registers();
-    TargetIsa::new("arm64", settings, regs)
+
+    let inst_group = InstructionGroup::new("arm64", "arm64 specific instruction set");
+
+    TargetIsa::new("arm64", inst_group, settings, regs)
 }
diff --git a/cranelift/codegen/meta/src/isa/mod.rs b/cranelift/codegen/meta/src/isa/mod.rs
index 9662435867..0c2bb5eed0 100644
--- a/cranelift/codegen/meta/src/isa/mod.rs
+++ b/cranelift/codegen/meta/src/isa/mod.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::isa::TargetIsa;
-use crate::cdsl::settings::SettingGroup;
+use crate::shared::Definitions as SharedDefinitions;
 use std::fmt;
 
 mod arm32;
@@ -55,13 +55,13 @@ impl fmt::Display for Isa {
     }
 }
 
-pub fn define(isas: &Vec, shared_settings: &SettingGroup) -> Vec {
+pub fn define(isas: &Vec, shared_defs: &mut SharedDefinitions) -> Vec {
     isas.iter()
         .map(|isa| match isa {
-            Isa::Riscv => riscv::define(shared_settings),
-            Isa::X86 => x86::define(shared_settings),
-            Isa::Arm32 => arm32::define(shared_settings),
-            Isa::Arm64 => arm64::define(shared_settings),
+            Isa::Riscv => riscv::define(shared_defs),
+            Isa::X86 => x86::define(shared_defs),
+            Isa::Arm32 => arm32::define(shared_defs),
+            Isa::Arm64 => arm64::define(shared_defs),
         })
         .collect()
 }
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index 040d088386..9d5a78a5a0 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -1,6 +1,8 @@
+use crate::cdsl::inst::InstructionGroup;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
+use crate::shared::Definitions as SharedDefinitions;
 
 fn define_settings(shared: &SettingGroup) -> SettingGroup {
     let mut setting = SettingGroupBuilder::new("riscv");
@@ -76,8 +78,11 @@ fn define_registers() -> IsaRegs {
     regs.finish()
 }
 
-pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
-    let settings = define_settings(shared_settings);
+pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
+    let settings = define_settings(&shared_defs.settings);
     let regs = define_registers();
-    TargetIsa::new("riscv", settings, regs)
+
+    let inst_group = InstructionGroup::new("riscv", "riscv specific instruction set");
+
+    TargetIsa::new("riscv", inst_group, settings, regs)
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
new file mode 100644
index 0000000000..68c3b56b85
--- /dev/null
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -0,0 +1,244 @@
+#![allow(non_snake_case)]
+
+use crate::cdsl::formats::FormatRegistry;
+use crate::cdsl::inst::{InstructionBuilder as Inst, InstructionGroup};
+use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
+use crate::cdsl::types::ValueType;
+use crate::cdsl::typevar::{Interval, TypeVar, TypeVarBuilder};
+use crate::shared::types;
+
+pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
+    let mut ig = InstructionGroup::new("x86", "x86 specific instruction set");
+
+    let iflags: &TypeVar = &ValueType::Special(types::Flag::IFlags.into()).into();
+
+    let iWord = &TypeVarBuilder::new("iWord", "A scalar integer machine word")
+        .ints(32..64)
+        .finish();
+    let nlo = &operand_doc("nlo", iWord, "Low part of numerator");
+    let nhi = &operand_doc("nhi", iWord, "High part of numerator");
+    let d = &operand_doc("d", iWord, "Denominator");
+    let q = &operand_doc("q", iWord, "Quotient");
+    let r = &operand_doc("r", iWord, "Remainder");
+
+    ig.push(
+        Inst::new(
+            "x86_udivmodx",
+            r#"
+        Extended unsigned division.
+
+        Concatenate the bits in `nhi` and `nlo` to form the numerator.
+        Interpret the bits as an unsigned number and divide by the unsigned
+        denominator `d`. Trap when `d` is zero or if the quotient is larger
+        than the range of the output.
+
+        Return both quotient and remainder.
+        "#,
+        )
+        .operands_in(vec![nlo, nhi, d])
+        .operands_out(vec![q, r])
+        .can_trap(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "x86_sdivmodx",
+            r#"
+        Extended signed division.
+
+        Concatenate the bits in `nhi` and `nlo` to form the numerator.
+        Interpret the bits as a signed number and divide by the signed
+        denominator `d`. Trap when `d` is zero or if the quotient is outside
+        the range of the output.
+
+        Return both quotient and remainder.
+        "#,
+        )
+        .operands_in(vec![nlo, nhi, d])
+        .operands_out(vec![q, r])
+        .can_trap(true)
+        .finish(format_registry),
+    );
+
+    let argL = &operand("argL", iWord);
+    let argR = &operand("argR", iWord);
+    let resLo = &operand("resLo", iWord);
+    let resHi = &operand("resHi", iWord);
+
+    ig.push(
+        Inst::new(
+            "x86_umulx",
+            r#"
+        Unsigned integer multiplication, producing a double-length result.
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![argL, argR])
+        .operands_out(vec![resLo, resHi])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "x86_smulx",
+            r#"
+        Signed integer multiplication, producing a double-length result.
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![argL, argR])
+        .operands_out(vec![resLo, resHi])
+        .finish(format_registry),
+    );
+
+    let Float = &TypeVarBuilder::new("Float", "A scalar or vector floating point number")
+        .floats(Interval::All)
+        .simd_lanes(Interval::All)
+        .finish();
+    let IntTo = &TypeVarBuilder::new("IntTo", "An integer type with the same number of lanes")
+        .ints(32..64)
+        .simd_lanes(Interval::All)
+        .finish();
+    let x = &operand("x", Float);
+    let a = &operand("a", IntTo);
+
+    ig.push(
+        Inst::new(
+            "x86_cvtt2si",
+            r#"
+        Convert with truncation floating point to signed integer.
+
+        The source floating point operand is converted to a signed integer by
+        rounding towards zero. If the result can't be represented in the output
+        type, returns the smallest signed value the output type can represent.
+
+        This instruction does not trap.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let x = &operand("x", Float);
+    let a = &operand("a", Float);
+    let y = &operand("y", Float);
+
+    ig.push(
+        Inst::new(
+            "x86_fmin",
+            r#"
+        Floating point minimum with x86 semantics.
+
+        This is equivalent to the C ternary operator `x < y ? x : y` which
+        differs from :inst:`fmin` when either operand is NaN or when comparing
+        +0.0 to -0.0.
+
+        When the two operands don't compare as LT, `y` is returned unchanged,
+        even if it is a signalling NaN.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "x86_fmax",
+            r#"
+        Floating point maximum with x86 semantics.
+
+        This is equivalent to the C ternary operator `x > y ? x : y` which
+        differs from :inst:`fmax` when either operand is NaN or when comparing
+        +0.0 to -0.0.
+
+        When the two operands don't compare as GT, `y` is returned unchanged,
+        even if it is a signalling NaN.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let x = &operand("x", iWord);
+
+    ig.push(
+        Inst::new(
+            "x86_push",
+            r#"
+    Pushes a value onto the stack.
+
+    Decrements the stack pointer and stores the specified value on to the top.
+
+    This is polymorphic in i32 and i64. However, it is only implemented for i64
+    in 64-bit mode, and only for i32 in 32-bit mode.
+    "#,
+        )
+        .operands_in(vec![x])
+        .other_side_effects(true)
+        .can_store(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "x86_pop",
+            r#"
+    Pops a value from the stack.
+
+    Loads a value from the top of the stack and then increments the stack
+    pointer.
+
+    This is polymorphic in i32 and i64. However, it is only implemented for i64
+    in 64-bit mode, and only for i32 in 32-bit mode.
+    "#,
+        )
+        .operands_out(vec![x])
+        .other_side_effects(true)
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    let y = &operand("y", iWord);
+    let rflags = &operand("rflags", iflags);
+
+    ig.push(
+        Inst::new(
+            "x86_bsr",
+            r#"
+    Bit Scan Reverse -- returns the bit-index of the most significant 1
+    in the word. Result is undefined if the argument is zero. However, it
+    sets the Z flag depending on the argument, so it is at least easy to
+    detect and handle that case.
+
+    This is polymorphic in i32 and i64. It is implemented for both i64 and
+    i32 in 64-bit mode, and only for i32 in 32-bit mode.
+    "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![y, rflags])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "x86_bsf",
+            r#"
+    Bit Scan Forwards -- returns the bit-index of the least significant 1
+    in the word. Is otherwise identical to 'bsr', just above.
+    "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![y, rflags])
+        .finish(format_registry),
+    );
+
+    ig
+}
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index da64f3cc50..6c5fe4d097 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -1,7 +1,11 @@
+mod instructions;
+
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 
+use crate::shared::Definitions as SharedDefinitions;
+
 fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     let mut settings = SettingGroupBuilder::new("x86");
 
@@ -109,8 +113,11 @@ fn define_registers() -> IsaRegs {
     regs.finish()
 }
 
-pub fn define(shared_settings: &SettingGroup) -> TargetIsa {
-    let settings = define_settings(shared_settings);
+pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
+    let settings = define_settings(&shared_defs.settings);
     let regs = define_registers();
-    TargetIsa::new("x86", settings, regs)
+
+    let inst_group = instructions::define(&shared_defs.format_registry);
+
+    TargetIsa::new("x86", inst_group, settings, regs)
 }
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index 1599829dd9..9b50fe1232 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -20,10 +20,10 @@ pub fn isa_from_arch(arch: &str) -> Result {
 /// Generates all the Rust source files used in Cranelift from the meta-language.
 pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error> {
     // Common definitions.
-    let settings = shared::settings::define();
+    let mut shared_defs = shared::define();
 
     gen_settings::generate(
-        &settings,
+        &shared_defs.settings,
         gen_settings::ParentGroup::None,
         "new_settings.rs",
         &out_dir,
@@ -31,7 +31,7 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
     gen_types::generate("types.rs", &out_dir)?;
 
     // Per ISA definitions.
-    let isas = isa::define(isas, &settings);
+    let isas = isa::define(isas, &mut shared_defs);
 
     for isa in isas {
         gen_registers::generate(&isa, &format!("registers-{}.rs", isa.name), &out_dir)?;

From 86430abc4ce78d143c7bdc322495f018f768d877 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Mar 2019 19:37:00 +0100
Subject: [PATCH 2362/3084] [meta] Port all the Cranelift instructions to the
 meta crate;

---
 .../codegen/meta/src/shared/instructions.rs   | 3107 +++++++++++++++++
 cranelift/codegen/meta/src/shared/mod.rs      |    4 +
 2 files changed, 3111 insertions(+)
 create mode 100644 cranelift/codegen/meta/src/shared/instructions.rs

diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
new file mode 100644
index 0000000000..a83515ee04
--- /dev/null
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -0,0 +1,3107 @@
+#![allow(non_snake_case)]
+
+use crate::cdsl::formats::FormatRegistry;
+use crate::cdsl::inst::{InstructionBuilder as Inst, InstructionGroup};
+use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
+use crate::cdsl::type_inference::Constraint::WiderOrEq;
+use crate::cdsl::types::{LaneType, ValueType};
+use crate::cdsl::typevar::{Interval, TypeVar, TypeVarBuilder};
+use crate::shared::{types, OperandKinds};
+
+pub fn define(
+    format_registry: &FormatRegistry,
+    immediates: &OperandKinds,
+    entities: &OperandKinds,
+) -> InstructionGroup {
+    let mut ig = InstructionGroup::new("base", "Shared base instruction set");
+
+    // Operand kind shorthands.
+    let intcc = immediates.by_name("intcc");
+    let floatcc = immediates.by_name("floatcc");
+    let trapcode = immediates.by_name("trapcode");
+    let uimm8 = immediates.by_name("uimm8");
+    let uimm32 = immediates.by_name("uimm32");
+    let imm64 = immediates.by_name("imm64");
+    let offset32 = immediates.by_name("offset32");
+    let memflags = immediates.by_name("memflags");
+    let ieee32 = immediates.by_name("ieee32");
+    let ieee64 = immediates.by_name("ieee64");
+    let boolean = immediates.by_name("boolean");
+    let regunit = immediates.by_name("regunit");
+
+    let ebb = entities.by_name("ebb");
+    let jump_table = entities.by_name("jump_table");
+    let variable_args = entities.by_name("variable_args");
+    let func_ref = entities.by_name("func_ref");
+    let sig_ref = entities.by_name("sig_ref");
+    let stack_slot = entities.by_name("stack_slot");
+    let global_value = entities.by_name("global_value");
+    let heap = entities.by_name("heap");
+    let table = entities.by_name("table");
+
+    let iflags: &TypeVar = &ValueType::Special(types::Flag::IFlags.into()).into();
+    let fflags: &TypeVar = &ValueType::Special(types::Flag::FFlags.into()).into();
+
+    let b1: &TypeVar = &ValueType::from(LaneType::from(types::Bool::B1)).into();
+    let f32_: &TypeVar = &ValueType::from(LaneType::from(types::Float::F32)).into();
+    let f64_: &TypeVar = &ValueType::from(LaneType::from(types::Float::F64)).into();
+
+    // Starting definitions.
+    let Int = &TypeVarBuilder::new("Int", "A scalar or vector integer type")
+        .ints(Interval::All)
+        .simd_lanes(Interval::All)
+        .finish();
+
+    let Bool = &TypeVarBuilder::new("Bool", "A scalar or vector boolean type")
+        .bools(Interval::All)
+        .simd_lanes(Interval::All)
+        .finish();
+
+    let iB = &TypeVarBuilder::new("iB", "A scalar integer type")
+        .ints(Interval::All)
+        .finish();
+
+    let iAddr = &TypeVarBuilder::new("iAddr", "An integer address type")
+        .ints(32..64)
+        .finish();
+
+    let Testable = &TypeVarBuilder::new("Testable", "A scalar boolean or integer type")
+        .ints(Interval::All)
+        .bools(Interval::All)
+        .finish();
+
+    let TxN = &TypeVarBuilder::new("TxN", "A SIMD vector type")
+        .ints(Interval::All)
+        .floats(Interval::All)
+        .bools(Interval::All)
+        .simd_lanes(Interval::All)
+        .includes_scalars(false)
+        .finish();
+
+    let Any = &TypeVarBuilder::new(
+        "Any",
+        "Any integer, float, or boolean scalar or vector type",
+    )
+    .ints(Interval::All)
+    .floats(Interval::All)
+    .bools(Interval::All)
+    .simd_lanes(Interval::All)
+    .includes_scalars(true)
+    .finish();
+
+    let Mem = &TypeVarBuilder::new("Mem", "Any type that can be stored in memory")
+        .ints(Interval::All)
+        .floats(Interval::All)
+        .simd_lanes(Interval::All)
+        .finish();
+
+    let MemTo = &TypeVarBuilder::new("MemTo", "Any type that can be stored in memory")
+        .ints(Interval::All)
+        .floats(Interval::All)
+        .simd_lanes(Interval::All)
+        .finish();
+
+    let addr = &operand("addr", iAddr);
+    let c = &operand_doc("c", Testable, "Controlling value to test");
+    let Cond = &operand("Cond", intcc);
+    let x = &operand("x", iB);
+    let y = &operand("y", iB);
+    let EBB = &operand_doc("EBB", ebb, "Destination extended basic block");
+    let args = &operand_doc("args", variable_args, "EBB arguments");
+
+    ig.push(
+        Inst::new(
+            "jump",
+            r#"
+        Jump.
+
+        Unconditionally jump to an extended basic block, passing the specified
+        EBB arguments. The number and types of arguments must match the
+        destination EBB.
+        "#,
+        )
+        .operands_in(vec![EBB, args])
+        .is_terminator(true)
+        .is_branch(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "fallthrough",
+            r#"
+        Fall through to the next EBB.
+
+        This is the same as :inst:`jump`, except the destination EBB must be
+        the next one in the layout.
+
+        Jumps are turned into fall-through instructions by the branch
+        relaxation pass. There is no reason to use this instruction outside
+        that pass.
+        "#,
+        )
+        .operands_in(vec![EBB, args])
+        .is_terminator(true)
+        .is_branch(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "brz",
+            r#"
+        Branch when zero.
+
+        If ``c`` is a :type:`b1` value, take the branch when ``c`` is false. If
+        ``c`` is an integer value, take the branch when ``c = 0``.
+        "#,
+        )
+        .operands_in(vec![c, EBB, args])
+        .is_branch(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "brnz",
+            r#"
+        Branch when non-zero.
+
+        If ``c`` is a :type:`b1` value, take the branch when ``c`` is true. If
+        ``c`` is an integer value, take the branch when ``c != 0``.
+        "#,
+        )
+        .operands_in(vec![c, EBB, args])
+        .is_branch(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "br_icmp",
+            r#"
+        Compare scalar integers and branch.
+
+        Compare ``x`` and ``y`` in the same way as the :inst:`icmp` instruction
+        and take the branch if the condition is true::
+
+            br_icmp ugt v1, v2, ebb4(v5, v6)
+
+        is semantically equivalent to::
+
+            v10 = icmp ugt, v1, v2
+            brnz v10, ebb4(v5, v6)
+
+        Some RISC architectures like MIPS and RISC-V provide instructions that
+        implement all or some of the condition codes. The instruction can also
+        be used to represent *macro-op fusion* on architectures like Intel's.
+        "#,
+        )
+        .operands_in(vec![Cond, x, y, EBB, args])
+        .is_branch(true)
+        .finish(format_registry),
+    );
+
+    let f = &operand("f", iflags);
+
+    ig.push(
+        Inst::new(
+            "brif",
+            r#"
+        Branch when condition is true in integer CPU flags.
+        "#,
+        )
+        .operands_in(vec![Cond, f, EBB, args])
+        .is_branch(true)
+        .finish(format_registry),
+    );
+
+    let Cond = &operand("Cond", floatcc);
+    let f = &operand("f", fflags);
+
+    ig.push(
+        Inst::new(
+            "brff",
+            r#"
+        Branch when condition is true in floating point CPU flags.
+        "#,
+        )
+        .operands_in(vec![Cond, f, EBB, args])
+        .is_branch(true)
+        .finish(format_registry),
+    );
+
+    let x = &operand_doc("x", iB, "index into jump table");
+
+    let Entry = &TypeVarBuilder::new("Entry", "A scalar integer type")
+        .ints(Interval::All)
+        .finish();
+
+    let entry = &operand_doc("entry", Entry, "entry of jump table");
+    let JT = &operand("JT", jump_table);
+
+    ig.push(
+        Inst::new(
+            "br_table",
+            r#"
+        Indirect branch via jump table.
+
+        Use ``x`` as an unsigned index into the jump table ``JT``. If a jump
+        table entry is found, branch to the corresponding EBB. If no entry was
+        found or the index is out-of-bounds, branch to the given default EBB.
+
+        Note that this branch instruction can't pass arguments to the targeted
+        blocks. Split critical edges as needed to work around this.
+
+        Do not confuse this with "tables" in WebAssembly. ``br_table`` is for
+        jump tables with destinations within the current function only -- think
+        of a ``match`` in Rust or a ``switch`` in C.  If you want to call a
+        function in a dynamic library, that will typically use
+        ``call_indirect``.
+        "#,
+        )
+        .operands_in(vec![x, EBB, JT])
+        .is_terminator(true)
+        .is_branch(true)
+        .finish(format_registry),
+    );
+
+    let Size = &operand_doc("Size", uimm8, "Size in bytes");
+
+    ig.push(
+        Inst::new(
+            "jump_table_entry",
+            r#"
+    Get an entry from a jump table.
+
+    Load a serialized ``entry`` from a jump table ``JT`` at a given index
+    ``addr`` with a specific ``Size``. The retrieved entry may need to be
+    decoded after loading, depending upon the jump table type used.
+
+    Currently, the only type supported is entries which are relative to the
+    base of the jump table.
+    "#,
+        )
+        .operands_in(vec![x, addr, Size, JT])
+        .operands_out(vec![entry])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "jump_table_base",
+            r#"
+    Get the absolute base address of a jump table.
+
+    This is used for jump tables wherein the entries are stored relative to
+    the base of jump table. In order to use these, generated code should first
+    load an entry using ``jump_table_entry``, then use this instruction to add
+    the relative base back to it.
+    "#,
+        )
+        .operands_in(vec![JT])
+        .operands_out(vec![addr])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "indirect_jump_table_br",
+            r#"
+    Branch indirectly via a jump table entry.
+
+    Unconditionally jump via a jump table entry that was previously loaded
+    with the ``jump_table_entry`` instruction.
+    "#,
+        )
+        .operands_in(vec![addr, JT])
+        .is_indirect_branch(true)
+        .is_terminator(true)
+        .is_branch(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "debugtrap",
+            r#"
+    Encodes an assembly debug trap.
+    "#,
+        )
+        .other_side_effects(true)
+        .can_load(true)
+        .can_store(true)
+        .finish(format_registry),
+    );
+
+    let code = &operand("code", trapcode);
+
+    ig.push(
+        Inst::new(
+            "trap",
+            r#"
+        Terminate execution unconditionally.
+        "#,
+        )
+        .operands_in(vec![code])
+        .can_trap(true)
+        .is_terminator(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "trapz",
+            r#"
+        Trap when zero.
+
+        if ``c`` is non-zero, execution continues at the following instruction.
+        "#,
+        )
+        .operands_in(vec![c, code])
+        .can_trap(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "trapnz",
+            r#"
+        Trap when non-zero.
+
+        if ``c`` is zero, execution continues at the following instruction.
+        "#,
+        )
+        .operands_in(vec![c, code])
+        .can_trap(true)
+        .finish(format_registry),
+    );
+
+    let Cond = &operand("Cond", intcc);
+    let f = &operand("f", iflags);
+
+    ig.push(
+        Inst::new(
+            "trapif",
+            r#"
+        Trap when condition is true in integer CPU flags.
+        "#,
+        )
+        .operands_in(vec![Cond, f, code])
+        .can_trap(true)
+        .finish(format_registry),
+    );
+
+    let Cond = &operand("Cond", floatcc);
+    let f = &operand("f", fflags);
+
+    ig.push(
+        Inst::new(
+            "trapff",
+            r#"
+        Trap when condition is true in floating point CPU flags.
+        "#,
+        )
+        .operands_in(vec![Cond, f, code])
+        .can_trap(true)
+        .finish(format_registry),
+    );
+
+    let rvals = &operand_doc("rvals", variable_args, "return values");
+
+    ig.push(
+        Inst::new(
+            "return",
+            r#"
+        Return from the function.
+
+        Unconditionally transfer control to the calling function, passing the
+        provided return values. The list of return values must match the
+        function signature's return types.
+        "#,
+        )
+        .operands_in(vec![rvals])
+        .is_return(true)
+        .is_terminator(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "fallthrough_return",
+            r#"
+        Return from the function by fallthrough.
+
+        This is a specialized instruction for use where one wants to append
+        a custom epilogue, which will then perform the real return. This
+        instruction has no encoding.
+        "#,
+        )
+        .operands_in(vec![rvals])
+        .is_return(true)
+        .is_terminator(true)
+        .finish(format_registry),
+    );
+
+    let FN = &operand_doc(
+        "FN",
+        func_ref,
+        "function to call, declared by :inst:`function`",
+    );
+    let args = &operand_doc("args", variable_args, "call arguments");
+
+    ig.push(
+        Inst::new(
+            "call",
+            r#"
+        Direct function call.
+
+        Call a function which has been declared in the preamble. The argument
+        types must match the function's signature.
+        "#,
+        )
+        .operands_in(vec![FN, args])
+        .operands_out(vec![rvals])
+        .is_call(true)
+        .finish(format_registry),
+    );
+
+    let SIG = &operand_doc("SIG", sig_ref, "function signature");
+    let callee = &operand_doc("callee", iAddr, "address of function to call");
+
+    ig.push(
+        Inst::new(
+            "call_indirect",
+            r#"
+        Indirect function call.
+
+        Call the function pointed to by `callee` with the given arguments. The
+        called function must match the specified signature.
+
+        Note that this is different from WebAssembly's ``call_indirect``; the
+        callee is a native address, rather than a table index. For WebAssembly,
+        :inst:`table_addr` and :inst:`load` are used to obtain a native address
+        from a table.
+        "#,
+        )
+        .operands_in(vec![SIG, callee, args])
+        .operands_out(vec![rvals])
+        .is_call(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "func_addr",
+            r#"
+        Get the address of a function.
+
+        Compute the absolute address of a function declared in the preamble.
+        The returned address can be used as a ``callee`` argument to
+        :inst:`call_indirect`. This is also a method for calling functions that
+        are too far away to be addressable by a direct :inst:`call`
+        instruction.
+        "#,
+        )
+        .operands_in(vec![FN])
+        .operands_out(vec![addr])
+        .finish(format_registry),
+    );
+
+    let SS = &operand("SS", stack_slot);
+    let Offset = &operand_doc("Offset", offset32, "Byte offset from base address");
+    let x = &operand_doc("x", Mem, "Value to be stored");
+    let a = &operand_doc("a", Mem, "Value loaded");
+    let p = &operand("p", iAddr);
+    let MemFlags = &operand("MemFlags", memflags);
+    let args = &operand_doc("args", variable_args, "Address arguments");
+
+    ig.push(
+        Inst::new(
+            "load",
+            r#"
+        Load from memory at ``p + Offset``.
+
+        This is a polymorphic instruction that can load any value type which
+        has a memory representation.
+        "#,
+        )
+        .operands_in(vec![MemFlags, p, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "load_complex",
+            r#"
+        Load from memory at ``sum(args) + Offset``.
+
+        This is a polymorphic instruction that can load any value type which
+        has a memory representation.
+        "#,
+        )
+        .operands_in(vec![MemFlags, args, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "store",
+            r#"
+        Store ``x`` to memory at ``p + Offset``.
+
+        This is a polymorphic instruction that can store any value type with a
+        memory representation.
+        "#,
+        )
+        .operands_in(vec![MemFlags, x, p, Offset])
+        .can_store(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "store_complex",
+            r#"
+        Store ``x`` to memory at ``sum(args) + Offset``.
+
+        This is a polymorphic instruction that can store any value type with a
+        memory representation.
+        "#,
+        )
+        .operands_in(vec![MemFlags, x, args, Offset])
+        .can_store(true)
+        .finish(format_registry),
+    );
+
+    let iExt8 = &TypeVarBuilder::new("iExt8", "An integer type with more than 8 bits")
+        .ints(16..64)
+        .finish();
+    let x = &operand("x", iExt8);
+    let a = &operand("a", iExt8);
+
+    ig.push(
+        Inst::new(
+            "uload8",
+            r#"
+        Load 8 bits from memory at ``p + Offset`` and zero-extend.
+
+        This is equivalent to ``load.i8`` followed by ``uextend``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, p, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "uload8_complex",
+            r#"
+        Load 8 bits from memory at ``sum(args) + Offset`` and zero-extend.
+
+        This is equivalent to ``load.i8`` followed by ``uextend``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, args, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "sload8",
+            r#"
+        Load 8 bits from memory at ``p + Offset`` and sign-extend.
+
+        This is equivalent to ``load.i8`` followed by ``sextend``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, p, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "sload8_complex",
+            r#"
+        Load 8 bits from memory at ``sum(args) + Offset`` and sign-extend.
+
+        This is equivalent to ``load.i8`` followed by ``sextend``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, args, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "istore8",
+            r#"
+        Store the low 8 bits of ``x`` to memory at ``p + Offset``.
+
+        This is equivalent to ``ireduce.i8`` followed by ``store.i8``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, x, p, Offset])
+        .can_store(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "istore8_complex",
+            r#"
+        Store the low 8 bits of ``x`` to memory at ``sum(args) + Offset``.
+
+        This is equivalent to ``ireduce.i8`` followed by ``store.i8``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, x, args, Offset])
+        .can_store(true)
+        .finish(format_registry),
+    );
+
+    let iExt16 = &TypeVarBuilder::new("iExt16", "An integer type with more than 16 bits")
+        .ints(32..64)
+        .finish();
+    let x = &operand("x", iExt16);
+    let a = &operand("a", iExt16);
+
+    ig.push(
+        Inst::new(
+            "uload16",
+            r#"
+        Load 16 bits from memory at ``p + Offset`` and zero-extend.
+
+        This is equivalent to ``load.i16`` followed by ``uextend``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, p, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "uload16_complex",
+            r#"
+        Load 16 bits from memory at ``sum(args) + Offset`` and zero-extend.
+
+        This is equivalent to ``load.i16`` followed by ``uextend``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, args, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "sload16",
+            r#"
+        Load 16 bits from memory at ``p + Offset`` and sign-extend.
+
+        This is equivalent to ``load.i16`` followed by ``sextend``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, p, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "sload16_complex",
+            r#"
+        Load 16 bits from memory at ``sum(args) + Offset`` and sign-extend.
+
+        This is equivalent to ``load.i16`` followed by ``sextend``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, args, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "istore16",
+            r#"
+        Store the low 16 bits of ``x`` to memory at ``p + Offset``.
+
+        This is equivalent to ``ireduce.i16`` followed by ``store.i16``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, x, p, Offset])
+        .can_store(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "istore16_complex",
+            r#"
+        Store the low 16 bits of ``x`` to memory at ``sum(args) + Offset``.
+
+        This is equivalent to ``ireduce.i16`` followed by ``store.i16``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, x, args, Offset])
+        .can_store(true)
+        .finish(format_registry),
+    );
+
+    let iExt32 = &TypeVarBuilder::new("iExt32", "An integer type with more than 32 bits")
+        .ints(64..64)
+        .finish();
+    let x = &operand("x", iExt32);
+    let a = &operand("a", iExt32);
+
+    ig.push(
+        Inst::new(
+            "uload32",
+            r#"
+        Load 32 bits from memory at ``p + Offset`` and zero-extend.
+
+        This is equivalent to ``load.i32`` followed by ``uextend``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, p, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "uload32_complex",
+            r#"
+        Load 32 bits from memory at ``sum(args) + Offset`` and zero-extend.
+
+        This is equivalent to ``load.i32`` followed by ``uextend``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, args, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "sload32",
+            r#"
+        Load 32 bits from memory at ``p + Offset`` and sign-extend.
+
+        This is equivalent to ``load.i32`` followed by ``sextend``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, p, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "sload32_complex",
+            r#"
+        Load 32 bits from memory at ``sum(args) + Offset`` and sign-extend.
+
+        This is equivalent to ``load.i32`` followed by ``sextend``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, args, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "istore32",
+            r#"
+        Store the low 32 bits of ``x`` to memory at ``p + Offset``.
+
+        This is equivalent to ``ireduce.i32`` followed by ``store.i32``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, x, p, Offset])
+        .can_store(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "istore32_complex",
+            r#"
+        Store the low 32 bits of ``x`` to memory at ``sum(args) + Offset``.
+
+        This is equivalent to ``ireduce.i32`` followed by ``store.i32``.
+        "#,
+        )
+        .operands_in(vec![MemFlags, x, args, Offset])
+        .can_store(true)
+        .finish(format_registry),
+    );
+
+    let x = &operand_doc("x", Mem, "Value to be stored");
+    let a = &operand_doc("a", Mem, "Value loaded");
+    let Offset = &operand_doc("Offset", offset32, "In-bounds offset into stack slot");
+
+    ig.push(
+        Inst::new(
+            "stack_load",
+            r#"
+        Load a value from a stack slot at the constant offset.
+
+        This is a polymorphic instruction that can load any value type which
+        has a memory representation.
+
+        The offset is an immediate constant, not an SSA value. The memory
+        access cannot go out of bounds, i.e.
+        :math:`sizeof(a) + Offset <= sizeof(SS)`.
+        "#,
+        )
+        .operands_in(vec![SS, Offset])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "stack_store",
+            r#"
+        Store a value to a stack slot at a constant offset.
+
+        This is a polymorphic instruction that can store any value type with a
+        memory representation.
+
+        The offset is an immediate constant, not an SSA value. The memory
+        access cannot go out of bounds, i.e.
+        :math:`sizeof(a) + Offset <= sizeof(SS)`.
+        "#,
+        )
+        .operands_in(vec![x, SS, Offset])
+        .can_store(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "stack_addr",
+            r#"
+        Get the address of a stack slot.
+
+        Compute the absolute address of a byte in a stack slot. The offset must
+        refer to a byte inside the stack slot:
+        :math:`0 <= Offset < sizeof(SS)`.
+        "#,
+        )
+        .operands_in(vec![SS, Offset])
+        .operands_out(vec![addr])
+        .finish(format_registry),
+    );
+
+    let GV = &operand("GV", global_value);
+
+    ig.push(
+        Inst::new(
+            "global_value",
+            r#"
+        Compute the value of global GV.
+        "#,
+        )
+        .operands_in(vec![GV])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "symbol_value",
+            r#"
+        Compute the value of global GV, which is a symbolic value.
+        "#,
+        )
+        .operands_in(vec![GV])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let HeapOffset = &TypeVarBuilder::new("HeapOffset", "An unsigned heap offset")
+        .ints(32..64)
+        .finish();
+
+    let H = &operand("H", heap);
+    let p = &operand("p", HeapOffset);
+    let Size = &operand_doc("Size", uimm32, "Size in bytes");
+
+    ig.push(
+        Inst::new(
+            "heap_addr",
+            r#"
+        Bounds check and compute absolute address of heap memory.
+
+        Verify that the offset range ``p .. p + Size - 1`` is in bounds for the
+        heap H, and generate an absolute address that is safe to dereference.
+
+        1. If ``p + Size`` is not greater than the heap bound, return an
+           absolute address corresponding to a byte offset of ``p`` from the
+           heap's base address.
+        2. If ``p + Size`` is greater than the heap bound, generate a trap.
+        "#,
+        )
+        .operands_in(vec![H, p, Size])
+        .operands_out(vec![addr])
+        .finish(format_registry),
+    );
+
+    let TableOffset = &TypeVarBuilder::new("TableOffset", "An unsigned table offset")
+        .ints(32..64)
+        .finish();
+    let T = &operand("T", table);
+    let p = &operand("p", TableOffset);
+    let Offset = &operand_doc("Offset", offset32, "Byte offset from element address");
+
+    ig.push(
+        Inst::new(
+            "table_addr",
+            r#"
+        Bounds check and compute absolute address of a table entry.
+
+        Verify that the offset ``p`` is in bounds for the table T, and generate
+        an absolute address that is safe to dereference.
+
+        ``Offset`` must be less than the size of a table element.
+
+        1. If ``p`` is not greater than the table bound, return an absolute
+           address corresponding to a byte offset of ``p`` from the table's
+           base address.
+        2. If ``p`` is greater than the table bound, generate a trap.
+        "#,
+        )
+        .operands_in(vec![T, p, Offset])
+        .operands_out(vec![addr])
+        .finish(format_registry),
+    );
+
+    let N = &operand("N", imm64);
+    let a = &operand_doc("a", Int, "A constant integer scalar or vector value");
+
+    ig.push(
+        Inst::new(
+            "iconst",
+            r#"
+        Integer constant.
+
+        Create a scalar integer SSA value with an immediate constant value, or
+        an integer vector where all the lanes have the same value.
+        "#,
+        )
+        .operands_in(vec![N])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let N = &operand("N", ieee32);
+    let a = &operand_doc("a", f32_, "A constant f32 scalar value");
+
+    ig.push(
+        Inst::new(
+            "f32const",
+            r#"
+        Floating point constant.
+
+        Create a :type:`f32` SSA value with an immediate constant value.
+        "#,
+        )
+        .operands_in(vec![N])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let N = &operand("N", ieee64);
+    let a = &operand_doc("a", f64_, "A constant f64 scalar value");
+
+    ig.push(
+        Inst::new(
+            "f64const",
+            r#"
+        Floating point constant.
+
+        Create a :type:`f64` SSA value with an immediate constant value.
+        "#,
+        )
+        .operands_in(vec![N])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let N = &operand("N", boolean);
+    let a = &operand_doc("a", Bool, "A constant boolean scalar or vector value");
+
+    ig.push(
+        Inst::new(
+            "bconst",
+            r#"
+        Boolean constant.
+
+        Create a scalar boolean SSA value with an immediate constant value, or
+        a boolean vector where all the lanes have the same value.
+        "#,
+        )
+        .operands_in(vec![N])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "nop",
+            r#"
+        Just a dummy instruction
+
+        Note: this doesn't compile to a machine code nop
+        "#,
+        )
+        .finish(format_registry),
+    );
+
+    let c = &operand_doc("c", Testable, "Controlling value to test");
+    let x = &operand_doc("x", Any, "Value to use when `c` is true");
+    let y = &operand_doc("y", Any, "Value to use when `c` is false");
+    let a = &operand("a", Any);
+
+    ig.push(
+        Inst::new(
+            "select",
+            r#"
+        Conditional select.
+
+        This instruction selects whole values. Use :inst:`vselect` for
+        lane-wise selection.
+        "#,
+        )
+        .operands_in(vec![c, x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let cc = &operand_doc("cc", intcc, "Controlling condition code");
+    let flags = &operand_doc("flags", iflags, "The machine's flag register");
+
+    ig.push(
+        Inst::new(
+            "selectif",
+            r#"
+        Conditional select, dependent on integer condition codes.
+        "#,
+        )
+        .operands_in(vec![cc, flags, x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let x = &operand("x", Any);
+
+    ig.push(
+        Inst::new(
+            "copy",
+            r#"
+        Register-register copy.
+
+        This instruction copies its input, preserving the value type.
+
+        A pure SSA-form program does not need to copy values, but this
+        instruction is useful for representing intermediate stages during
+        instruction transformations, and the register allocator needs a way of
+        representing register copies.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "spill",
+            r#"
+        Spill a register value to a stack slot.
+
+        This instruction behaves exactly like :inst:`copy`, but the result
+        value is assigned to a spill slot.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .can_store(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "fill",
+            r#"
+        Load a register value from a stack slot.
+
+        This instruction behaves exactly like :inst:`copy`, but creates a new
+        SSA value for the spilled input value.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .can_load(true)
+        .finish(format_registry),
+    );
+
+    let src = &operand("src", regunit);
+    let dst = &operand("dst", regunit);
+
+    ig.push(
+        Inst::new(
+            "regmove",
+            r#"
+        Temporarily divert ``x`` from ``src`` to ``dst``.
+
+        This instruction moves the location of a value from one register to
+        another without creating a new SSA value. It is used by the register
+        allocator to temporarily rearrange register assignments in order to
+        satisfy instruction constraints.
+
+        The register diversions created by this instruction must be undone
+        before the value leaves the EBB. At the entry to a new EBB, all live
+        values must be in their originally assigned registers.
+        "#,
+        )
+        .operands_in(vec![x, src, dst])
+        .other_side_effects(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "copy_special",
+            r#"
+        Copies the contents of ''src'' register to ''dst'' register.
+
+        This instructions copies the contents of one register to another
+        register without involving any SSA values. This is used for copying
+        special registers, e.g. copying the stack register to the frame
+        register in a function prologue.
+        "#,
+        )
+        .operands_in(vec![src, dst])
+        .other_side_effects(true)
+        .finish(format_registry),
+    );
+
+    let delta = &operand("delta", Int);
+
+    ig.push(
+        Inst::new(
+            "adjust_sp_down",
+            r#"
+    Subtracts ``delta`` offset value from the stack pointer register.
+
+    This instruction is used to adjust the stack pointer by a dynamic amount.
+    "#,
+        )
+        .operands_in(vec![delta])
+        .other_side_effects(true)
+        .finish(format_registry),
+    );
+
+    let Offset = &operand_doc("Offset", imm64, "Offset from current stack pointer");
+
+    ig.push(
+        Inst::new(
+            "adjust_sp_up_imm",
+            r#"
+    Adds ``Offset`` immediate offset value to the stack pointer register.
+
+    This instruction is used to adjust the stack pointer, primarily in function
+    prologues and epilogues. ``Offset`` is constrained to the size of a signed
+    32-bit integer.
+    "#,
+        )
+        .operands_in(vec![Offset])
+        .other_side_effects(true)
+        .finish(format_registry),
+    );
+
+    let Offset = &operand_doc("Offset", imm64, "Offset from current stack pointer");
+
+    ig.push(
+        Inst::new(
+            "adjust_sp_down_imm",
+            r#"
+    Subtracts ``Offset`` immediate offset value from the stack pointer
+    register.
+
+    This instruction is used to adjust the stack pointer, primarily in function
+    prologues and epilogues. ``Offset`` is constrained to the size of a signed
+    32-bit integer.
+    "#,
+        )
+        .operands_in(vec![Offset])
+        .other_side_effects(true)
+        .finish(format_registry),
+    );
+
+    let f = &operand("f", iflags);
+
+    ig.push(
+        Inst::new(
+            "ifcmp_sp",
+            r#"
+    Compare ``addr`` with the stack pointer and set the CPU flags.
+
+    This is like :inst:`ifcmp` where ``addr`` is the LHS operand and the stack
+    pointer is the RHS.
+    "#,
+        )
+        .operands_in(vec![addr])
+        .operands_out(vec![f])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "regspill",
+            r#"
+        Temporarily divert ``x`` from ``src`` to ``SS``.
+
+        This instruction moves the location of a value from a register to a
+        stack slot without creating a new SSA value. It is used by the register
+        allocator to temporarily rearrange register assignments in order to
+        satisfy instruction constraints.
+
+        See also :inst:`regmove`.
+        "#,
+        )
+        .operands_in(vec![x, src, SS])
+        .other_side_effects(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "regfill",
+            r#"
+        Temporarily divert ``x`` from ``SS`` to ``dst``.
+
+        This instruction moves the location of a value from a stack slot to a
+        register without creating a new SSA value. It is used by the register
+        allocator to temporarily rearrange register assignments in order to
+        satisfy instruction constraints.
+
+        See also :inst:`regmove`.
+        "#,
+        )
+        .operands_in(vec![x, SS, dst])
+        .other_side_effects(true)
+        .finish(format_registry),
+    );
+
+    let x = &operand_doc("x", TxN, "Vector to split");
+    let lo = &operand_doc("lo", &TxN.half_vector(), "Low-numbered lanes of `x`");
+    let hi = &operand_doc("hi", &TxN.half_vector(), "High-numbered lanes of `x`");
+
+    ig.push(
+        Inst::new(
+            "vsplit",
+            r#"
+        Split a vector into two halves.
+
+        Split the vector `x` into two separate values, each containing half of
+        the lanes from ``x``. The result may be two scalars if ``x`` only had
+        two lanes.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![lo, hi])
+        .is_ghost(true)
+        .finish(format_registry),
+    );
+
+    let Any128 = &TypeVarBuilder::new("Any128", "Any scalar or vector type with as most 128 lanes")
+        .ints(Interval::All)
+        .floats(Interval::All)
+        .bools(Interval::All)
+        .simd_lanes(1..128)
+        .includes_scalars(true)
+        .finish();
+
+    let x = &operand_doc("x", Any128, "Low-numbered lanes");
+    let y = &operand_doc("y", Any128, "High-numbered lanes");
+    let a = &operand_doc("a", &Any128.double_vector(), "Concatenation of `x` and `y`");
+
+    ig.push(
+        Inst::new(
+            "vconcat",
+            r#"
+        Vector concatenation.
+
+        Return a vector formed by concatenating ``x`` and ``y``. The resulting
+        vector type has twice as many lanes as each of the inputs. The lanes of
+        ``x`` appear as the low-numbered lanes, and the lanes of ``y`` become
+        the high-numbered lanes of ``a``.
+
+        It is possible to form a vector by concatenating two scalars.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .is_ghost(true)
+        .finish(format_registry),
+    );
+
+    let c = &operand_doc("c", &TxN.as_bool(), "Controlling vector");
+    let x = &operand_doc("x", TxN, "Value to use where `c` is true");
+    let y = &operand_doc("y", TxN, "Value to use where `c` is false");
+    let a = &operand("a", TxN);
+
+    ig.push(
+        Inst::new(
+            "vselect",
+            r#"
+        Vector lane select.
+
+        Select lanes from ``x`` or ``y`` controlled by the lanes of the boolean
+        vector ``c``.
+        "#,
+        )
+        .operands_in(vec![c, x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let x = &operand("x", &TxN.lane_of());
+
+    ig.push(
+        Inst::new(
+            "splat",
+            r#"
+        Vector splat.
+
+        Return a vector whose lanes are all ``x``.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let x = &operand_doc("x", TxN, "SIMD vector to modify");
+    let y = &operand_doc("y", &TxN.lane_of(), "New lane value");
+    let Idx = &operand_doc("Idx", uimm8, "Lane index");
+
+    ig.push(
+        Inst::new(
+            "insertlane",
+            r#"
+        Insert ``y`` as lane ``Idx`` in x.
+
+        The lane index, ``Idx``, is an immediate value, not an SSA value. It
+        must indicate a valid lane index for the type of ``x``.
+        "#,
+        )
+        .operands_in(vec![x, Idx, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let x = &operand("x", TxN);
+    let a = &operand("a", &TxN.lane_of());
+
+    ig.push(
+        Inst::new(
+            "extractlane",
+            r#"
+        Extract lane ``Idx`` from ``x``.
+
+        The lane index, ``Idx``, is an immediate value, not an SSA value. It
+        must indicate a valid lane index for the type of ``x``.
+        "#,
+        )
+        .operands_in(vec![x, Idx])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let a = &operand("a", &Int.as_bool());
+    let Cond = &operand("Cond", intcc);
+    let x = &operand("x", Int);
+    let y = &operand("y", Int);
+
+    ig.push(
+        Inst::new(
+            "icmp",
+            r#"
+        Integer comparison.
+
+        The condition code determines if the operands are interpreted as signed
+        or unsigned integers.
+
+        ====== ======== =========
+        Signed Unsigned Condition
+        ====== ======== =========
+        eq     eq       Equal
+        ne     ne       Not equal
+        slt    ult      Less than
+        sge    uge      Greater than or equal
+        sgt    ugt      Greater than
+        sle    ule      Less than or equal
+        ====== ======== =========
+
+        When this instruction compares integer vectors, it returns a boolean
+        vector of lane-wise comparisons.
+        "#,
+        )
+        .operands_in(vec![Cond, x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let a = &operand("a", b1);
+    let x = &operand("x", iB);
+    let Y = &operand("Y", imm64);
+
+    ig.push(
+        Inst::new(
+            "icmp_imm",
+            r#"
+        Compare scalar integer to a constant.
+
+        This is the same as the :inst:`icmp` instruction, except one operand is
+        an immediate constant.
+
+        This instruction can only compare scalars. Use :inst:`icmp` for
+        lane-wise vector comparisons.
+        "#,
+        )
+        .operands_in(vec![Cond, x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let f = &operand("f", iflags);
+    let x = &operand("x", iB);
+    let y = &operand("y", iB);
+
+    ig.push(
+        Inst::new(
+            "ifcmp",
+            r#"
+        Compare scalar integers and return flags.
+
+        Compare two scalar integer values and return integer CPU flags
+        representing the result.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![f])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "ifcmp_imm",
+            r#"
+        Compare scalar integer to a constant and return flags.
+
+        Like :inst:`icmp_imm`, but returns integer CPU flags instead of testing
+        a specific condition code.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![f])
+        .finish(format_registry),
+    );
+
+    let a = &operand("a", Int);
+    let x = &operand("x", Int);
+    let y = &operand("y", Int);
+
+    ig.push(
+        Inst::new(
+            "iadd",
+            r#"
+        Wrapping integer addition: :math:`a := x + y \pmod{2^B}`.
+
+        This instruction does not depend on the signed/unsigned interpretation
+        of the operands.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "isub",
+            r#"
+        Wrapping integer subtraction: :math:`a := x - y \pmod{2^B}`.
+
+        This instruction does not depend on the signed/unsigned interpretation
+        of the operands.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "imul",
+            r#"
+        Wrapping integer multiplication: :math:`a := x y \pmod{2^B}`.
+
+        This instruction does not depend on the signed/unsigned interpretation
+        of the
+        operands.
+
+        Polymorphic over all integer types (vector and scalar).
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "umulhi",
+            r#"
+        Unsigned integer multiplication, producing the high half of a
+        double-length result.
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "smulhi",
+            r#"
+        Signed integer multiplication, producing the high half of a
+        double-length result.
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "udiv",
+            r#"
+        Unsigned integer division: :math:`a := \lfloor {x \over y} \rfloor`.
+
+        This operation traps if the divisor is zero.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .can_trap(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "sdiv",
+            r#"
+        Signed integer division rounded toward zero: :math:`a := sign(xy)
+        \lfloor {|x| \over |y|}\rfloor`.
+
+        This operation traps if the divisor is zero, or if the result is not
+        representable in :math:`B` bits two's complement. This only happens
+        when :math:`x = -2^{B-1}, y = -1`.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .can_trap(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "urem",
+            r#"
+        Unsigned integer remainder.
+
+        This operation traps if the divisor is zero.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .can_trap(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "srem",
+            r#"
+        Signed integer remainder. The result has the sign of the dividend.
+
+        This operation traps if the divisor is zero.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .can_trap(true)
+        .finish(format_registry),
+    );
+
+    let a = &operand("a", iB);
+    let x = &operand("x", iB);
+    let Y = &operand("Y", imm64);
+
+    ig.push(
+        Inst::new(
+            "iadd_imm",
+            r#"
+        Add immediate integer.
+
+        Same as :inst:`iadd`, but one operand is an immediate constant.
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "imul_imm",
+            r#"
+        Integer multiplication by immediate constant.
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "udiv_imm",
+            r#"
+        Unsigned integer division by an immediate constant.
+
+        This operation traps if the divisor is zero.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "sdiv_imm",
+            r#"
+        Signed integer division by an immediate constant.
+
+        This operation traps if the divisor is zero, or if the result is not
+        representable in :math:`B` bits two's complement. This only happens
+        when :math:`x = -2^{B-1}, Y = -1`.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "urem_imm",
+            r#"
+        Unsigned integer remainder with immediate divisor.
+
+        This operation traps if the divisor is zero.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "srem_imm",
+            r#"
+        Signed integer remainder with immediate divisor.
+
+        This operation traps if the divisor is zero.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "irsub_imm",
+            r#"
+        Immediate reverse wrapping subtraction: :math:`a := Y - x \pmod{2^B}`.
+
+        Also works as integer negation when :math:`Y = 0`. Use :inst:`iadd_imm`
+        with a negative immediate operand for the reverse immediate
+        subtraction.
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let a = &operand("a", iB);
+    let x = &operand("x", iB);
+    let y = &operand("y", iB);
+    let c_in = &operand_doc("c_in", b1, "Input carry flag");
+    let c_out = &operand_doc("c_out", b1, "Output carry flag");
+    let b_in = &operand_doc("b_in", b1, "Input borrow flag");
+    let b_out = &operand_doc("b_out", b1, "Output borrow flag");
+
+    ig.push(
+        Inst::new(
+            "iadd_cin",
+            r#"
+        Add integers with carry in.
+
+        Same as :inst:`iadd` with an additional carry input. Computes:
+
+        .. math::
+
+            a = x + y + c_{in} \pmod 2^B
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y, c_in])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "iadd_cout",
+            r#"
+        Add integers with carry out.
+
+        Same as :inst:`iadd` with an additional carry output.
+
+        .. math::
+
+            a &= x + y \pmod 2^B \\
+            c_{out} &= x+y >= 2^B
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a, c_out])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "iadd_carry",
+            r#"
+        Add integers with carry in and out.
+
+        Same as :inst:`iadd` with an additional carry input and output.
+
+        .. math::
+
+            a &= x + y + c_{in} \pmod 2^B \\
+            c_{out} &= x + y + c_{in} >= 2^B
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y, c_in])
+        .operands_out(vec![a, c_out])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "isub_bin",
+            r#"
+        Subtract integers with borrow in.
+
+        Same as :inst:`isub` with an additional borrow flag input. Computes:
+
+        .. math::
+
+            a = x - (y + b_{in}) \pmod 2^B
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y, b_in])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "isub_bout",
+            r#"
+        Subtract integers with borrow out.
+
+        Same as :inst:`isub` with an additional borrow flag output.
+
+        .. math::
+
+            a &= x - y \pmod 2^B \\
+            b_{out} &= x < y
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a, b_out])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "isub_borrow",
+            r#"
+        Subtract integers with borrow in and out.
+
+        Same as :inst:`isub` with an additional borrow flag input and output.
+
+        .. math::
+
+            a &= x - (y + b_{in}) \pmod 2^B \\
+            b_{out} &= x < y + b_{in}
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y, b_in])
+        .operands_out(vec![a, b_out])
+        .finish(format_registry),
+    );
+
+    let bits = &TypeVarBuilder::new(
+        "bits",
+        "Any integer, float, or boolean scalar or vector type",
+    )
+    .ints(Interval::All)
+    .floats(Interval::All)
+    .bools(Interval::All)
+    .simd_lanes(Interval::All)
+    .includes_scalars(true)
+    .finish();
+    let x = &operand("x", bits);
+    let y = &operand("y", bits);
+    let a = &operand("a", bits);
+
+    ig.push(
+        Inst::new(
+            "band",
+            r#"
+        Bitwise and.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "bor",
+            r#"
+        Bitwise or.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "bxor",
+            r#"
+        Bitwise xor.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "bnot",
+            r#"
+        Bitwise not.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "band_not",
+            r#"
+        Bitwise and not.
+
+        Computes `x & ~y`.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "bor_not",
+            r#"
+        Bitwise or not.
+
+        Computes `x | ~y`.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "bxor_not",
+            r#"
+        Bitwise xor not.
+
+        Computes `x ^ ~y`.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let x = &operand("x", iB);
+    let Y = &operand("Y", imm64);
+    let a = &operand("a", iB);
+
+    ig.push(
+        Inst::new(
+            "band_imm",
+            r#"
+        Bitwise and with immediate.
+
+        Same as :inst:`band`, but one operand is an immediate constant.
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "bor_imm",
+            r#"
+        Bitwise or with immediate.
+
+        Same as :inst:`bor`, but one operand is an immediate constant.
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "bxor_imm",
+            r#"
+        Bitwise xor with immediate.
+
+        Same as :inst:`bxor`, but one operand is an immediate constant.
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let x = &operand_doc("x", Int, "Scalar or vector value to shift");
+    let y = &operand_doc("y", iB, "Number of bits to shift");
+    let Y = &operand("Y", imm64);
+    let a = &operand("a", Int);
+
+    ig.push(
+        Inst::new(
+            "rotl",
+            r#"
+        Rotate left.
+
+        Rotate the bits in ``x`` by ``y`` places.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "rotr",
+            r#"
+        Rotate right.
+
+        Rotate the bits in ``x`` by ``y`` places.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "rotl_imm",
+            r#"
+        Rotate left by immediate.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "rotr_imm",
+            r#"
+        Rotate right by immediate.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "ishl",
+            r#"
+        Integer shift left. Shift the bits in ``x`` towards the MSB by ``y``
+        places. Shift in zero bits to the LSB.
+
+        The shift amount is masked to the size of ``x``.
+
+        When shifting a B-bits integer type, this instruction computes:
+
+        .. math::
+            s &:= y \pmod B,                \\
+            a &:= x \cdot 2^s \pmod{2^B}.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "ushr",
+            r#"
+        Unsigned shift right. Shift bits in ``x`` towards the LSB by ``y``
+        places, shifting in zero bits to the MSB. Also called a *logical
+        shift*.
+
+        The shift amount is masked to the size of the register.
+
+        When shifting a B-bits integer type, this instruction computes:
+
+        .. math::
+            s &:= y \pmod B,                \\
+            a &:= \lfloor x \cdot 2^{-s} \rfloor.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "sshr",
+            r#"
+        Signed shift right. Shift bits in ``x`` towards the LSB by ``y``
+        places, shifting in sign bits to the MSB. Also called an *arithmetic
+        shift*.
+
+        The shift amount is masked to the size of the register.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "ishl_imm",
+            r#"
+        Integer shift left by immediate.
+
+        The shift amount is masked to the size of ``x``.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "ushr_imm",
+            r#"
+        Unsigned shift right by immediate.
+
+        The shift amount is masked to the size of the register.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "sshr_imm",
+            r#"
+        Signed shift right by immediate.
+
+        The shift amount is masked to the size of the register.
+        "#,
+        )
+        .operands_in(vec![x, Y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let x = &operand("x", iB);
+    let a = &operand("a", iB);
+
+    ig.push(
+        Inst::new(
+            "bitrev",
+            r#"
+        Reverse the bits of a integer.
+
+        Reverses the bits in ``x``.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "clz",
+            r#"
+        Count leading zero bits.
+
+        Starting from the MSB in ``x``, count the number of zero bits before
+        reaching the first one bit. When ``x`` is zero, returns the size of x
+        in bits.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "cls",
+            r#"
+        Count leading sign bits.
+
+        Starting from the MSB after the sign bit in ``x``, count the number of
+        consecutive bits identical to the sign bit. When ``x`` is 0 or -1,
+        returns one less than the size of x in bits.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "ctz",
+            r#"
+        Count trailing zeros.
+
+        Starting from the LSB in ``x``, count the number of zero bits before
+        reaching the first one bit. When ``x`` is zero, returns the size of x
+        in bits.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "popcnt",
+            r#"
+        Population count
+
+        Count the number of one bits in ``x``.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let Float = &TypeVarBuilder::new("Float", "A scalar or vector floating point number")
+        .floats(Interval::All)
+        .simd_lanes(Interval::All)
+        .finish();
+    let Cond = &operand("Cond", floatcc);
+    let x = &operand("x", Float);
+    let y = &operand("y", Float);
+    let a = &operand("a", &Float.as_bool());
+
+    ig.push(
+        Inst::new(
+            "fcmp",
+            r#"
+        Floating point comparison.
+
+        Two IEEE 754-2008 floating point numbers, `x` and `y`, relate to each
+        other in exactly one of four ways:
+
+        == ==========================================
+        UN Unordered when one or both numbers is NaN.
+        EQ When :math:`x = y`. (And :math:`0.0 = -0.0`).
+        LT When :math:`x < y`.
+        GT When :math:`x > y`.
+        == ==========================================
+
+        The 14 :type:`floatcc` condition codes each correspond to a subset of
+        the four relations, except for the empty set which would always be
+        false, and the full set which would always be true.
+
+        The condition codes are divided into 7 'ordered' conditions which don't
+        include UN, and 7 unordered conditions which all include UN.
+
+        +-------+------------+---------+------------+-------------------------+
+        |Ordered             |Unordered             |Condition                |
+        +=======+============+=========+============+=========================+
+        |ord    |EQ | LT | GT|uno      |UN          |NaNs absent / present.   |
+        +-------+------------+---------+------------+-------------------------+
+        |eq     |EQ          |ueq      |UN | EQ     |Equal                    |
+        +-------+------------+---------+------------+-------------------------+
+        |one    |LT | GT     |ne       |UN | LT | GT|Not equal                |
+        +-------+------------+---------+------------+-------------------------+
+        |lt     |LT          |ult      |UN | LT     |Less than                |
+        +-------+------------+---------+------------+-------------------------+
+        |le     |LT | EQ     |ule      |UN | LT | EQ|Less than or equal       |
+        +-------+------------+---------+------------+-------------------------+
+        |gt     |GT          |ugt      |UN | GT     |Greater than             |
+        +-------+------------+---------+------------+-------------------------+
+        |ge     |GT | EQ     |uge      |UN | GT | EQ|Greater than or equal    |
+        +-------+------------+---------+------------+-------------------------+
+
+        The standard C comparison operators, `<, <=, >, >=`, are all ordered,
+        so they are false if either operand is NaN. The C equality operator,
+        `==`, is ordered, and since inequality is defined as the logical
+        inverse it is *unordered*. They map to the :type:`floatcc` condition
+        codes as follows:
+
+        ==== ====== ============
+        C    `Cond` Subset
+        ==== ====== ============
+        `==` eq     EQ
+        `!=` ne     UN | LT | GT
+        `<`  lt     LT
+        `<=` le     LT | EQ
+        `>`  gt     GT
+        `>=` ge     GT | EQ
+        ==== ====== ============
+
+        This subset of condition codes also corresponds to the WebAssembly
+        floating point comparisons of the same name.
+
+        When this instruction compares floating point vectors, it returns a
+        boolean vector with the results of lane-wise comparisons.
+        "#,
+        )
+        .operands_in(vec![Cond, x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let f = &operand("f", fflags);
+
+    ig.push(
+        Inst::new(
+            "ffcmp",
+            r#"
+        Floating point comparison returning flags.
+
+        Compares two numbers like :inst:`fcmp`, but returns floating point CPU
+        flags instead of testing a specific condition.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![f])
+        .finish(format_registry),
+    );
+
+    let x = &operand("x", Float);
+    let y = &operand("y", Float);
+    let z = &operand("z", Float);
+    let a = &operand_doc("a", Float, "Result of applying operator to each lane");
+
+    ig.push(
+        Inst::new(
+            "fadd",
+            r#"
+        Floating point addition.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "fsub",
+            r#"
+        Floating point subtraction.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "fmul",
+            r#"
+        Floating point multiplication.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "fdiv",
+            r#"
+        Floating point division.
+
+        Unlike the integer division instructions :clif:inst:`sdiv` and
+        :clif:inst:`udiv`, this can't trap. Division by zero is infinity or
+        NaN, depending on the dividend.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "sqrt",
+            r#"
+        Floating point square root.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "fma",
+            r#"
+        Floating point fused multiply-and-add.
+
+        Computes :math:`a := xy+z` without any intermediate rounding of the
+        product.
+        "#,
+        )
+        .operands_in(vec![x, y, z])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let a = &operand_doc("a", Float, "``x`` with its sign bit inverted");
+
+    ig.push(
+        Inst::new(
+            "fneg",
+            r#"
+        Floating point negation.
+
+        Note that this is a pure bitwise operation.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let a = &operand_doc("a", Float, "``x`` with its sign bit cleared");
+
+    ig.push(
+        Inst::new(
+            "fabs",
+            r#"
+        Floating point absolute value.
+
+        Note that this is a pure bitwise operation.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let a = &operand_doc(
+        "a",
+        Float,
+        "``x`` with its sign bit changed to that of ``y``",
+    );
+
+    ig.push(
+        Inst::new(
+            "fcopysign",
+            r#"
+        Floating point copy sign.
+
+        Note that this is a pure bitwise operation. The sign bit from ``y`` is
+        copied to the sign bit of ``x``.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let a = &operand_doc("a", Float, "The smaller of ``x`` and ``y``");
+
+    ig.push(
+        Inst::new(
+            "fmin",
+            r#"
+        Floating point minimum, propagating NaNs.
+
+        If either operand is NaN, this returns a NaN.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let a = &operand_doc("a", Float, "The larger of ``x`` and ``y``");
+
+    ig.push(
+        Inst::new(
+            "fmax",
+            r#"
+        Floating point maximum, propagating NaNs.
+
+        If either operand is NaN, this returns a NaN.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let a = &operand_doc("a", Float, "``x`` rounded to integral value");
+
+    ig.push(
+        Inst::new(
+            "ceil",
+            r#"
+        Round floating point round to integral, towards positive infinity.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "floor",
+            r#"
+        Round floating point round to integral, towards negative infinity.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "trunc",
+            r#"
+        Round floating point round to integral, towards zero.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "nearest",
+            r#"
+        Round floating point round to integral, towards nearest with ties to
+        even.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let Cond = &operand("Cond", intcc);
+    let f = &operand("f", iflags);
+    let a = &operand("a", b1);
+
+    ig.push(
+        Inst::new(
+            "trueif",
+            r#"
+        Test integer CPU flags for a specific condition.
+
+        Check the CPU flags in ``f`` against the ``Cond`` condition code and
+        return true when the condition code is satisfied.
+        "#,
+        )
+        .operands_in(vec![Cond, f])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let Cond = &operand("Cond", floatcc);
+    let f = &operand("f", fflags);
+
+    ig.push(
+        Inst::new(
+            "trueff",
+            r#"
+        Test floating point CPU flags for a specific condition.
+
+        Check the CPU flags in ``f`` against the ``Cond`` condition code and
+        return true when the condition code is satisfied.
+        "#,
+        )
+        .operands_in(vec![Cond, f])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let x = &operand("x", Mem);
+    let a = &operand_doc("a", MemTo, "Bits of `x` reinterpreted");
+
+    ig.push(
+        Inst::new(
+            "bitcast",
+            r#"
+        Reinterpret the bits in `x` as a different type.
+
+        The input and output types must be storable to memory and of the same
+        size. A bitcast is equivalent to storing one type and loading the other
+        type from the same address.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let Bool = &TypeVarBuilder::new("Bool", "A scalar or vector boolean type")
+        .bools(Interval::All)
+        .simd_lanes(Interval::All)
+        .finish();
+
+    let BoolTo = &TypeVarBuilder::new(
+        "BoolTo",
+        "A smaller boolean type with the same number of lanes",
+    )
+    .bools(Interval::All)
+    .simd_lanes(Interval::All)
+    .finish();
+
+    let x = &operand("x", Bool);
+    let a = &operand("a", BoolTo);
+
+    ig.push(
+        Inst::new(
+            "breduce",
+            r#"
+        Convert `x` to a smaller boolean type in the platform-defined way.
+
+        The result type must have the same number of vector lanes as the input,
+        and each lane must not have more bits that the input lanes. If the
+        input and output types are the same, this is a no-op.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .constraints(vec![WiderOrEq(Bool.clone(), BoolTo.clone())])
+        .finish(format_registry),
+    );
+
+    let BoolTo = &TypeVarBuilder::new(
+        "BoolTo",
+        "A larger boolean type with the same number of lanes",
+    )
+    .bools(Interval::All)
+    .simd_lanes(Interval::All)
+    .finish();
+    let x = &operand("x", Bool);
+    let a = &operand("a", BoolTo);
+
+    ig.push(
+        Inst::new(
+            "bextend",
+            r#"
+        Convert `x` to a larger boolean type in the platform-defined way.
+
+        The result type must have the same number of vector lanes as the input,
+        and each lane must not have fewer bits that the input lanes. If the
+        input and output types are the same, this is a no-op.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .constraints(vec![WiderOrEq(BoolTo.clone(), Bool.clone())])
+        .finish(format_registry),
+    );
+
+    let IntTo = &TypeVarBuilder::new("IntTo", "An integer type with the same number of lanes")
+        .ints(Interval::All)
+        .simd_lanes(Interval::All)
+        .finish();
+    let x = &operand("x", Bool);
+    let a = &operand("a", IntTo);
+
+    ig.push(
+        Inst::new(
+            "bint",
+            r#"
+        Convert `x` to an integer.
+
+        True maps to 1 and false maps to 0. The result type must have the same
+        number of vector lanes as the input.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "bmask",
+            r#"
+        Convert `x` to an integer mask.
+
+        True maps to all 1s and false maps to all 0s. The result type must have
+        the same number of vector lanes as the input.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let Int = &TypeVarBuilder::new("Int", "A scalar or vector integer type")
+        .ints(Interval::All)
+        .simd_lanes(Interval::All)
+        .finish();
+
+    let IntTo = &TypeVarBuilder::new(
+        "IntTo",
+        "A smaller integer type with the same number of lanes",
+    )
+    .ints(Interval::All)
+    .simd_lanes(Interval::All)
+    .finish();
+    let x = &operand("x", Int);
+    let a = &operand("a", IntTo);
+
+    ig.push(
+        Inst::new(
+            "ireduce",
+            r#"
+        Convert `x` to a smaller integer type by dropping high bits.
+
+        Each lane in `x` is converted to a smaller integer type by discarding
+        the most significant bits. This is the same as reducing modulo
+        :math:`2^n`.
+
+        The result type must have the same number of vector lanes as the input,
+        and each lane must not have more bits that the input lanes. If the
+        input and output types are the same, this is a no-op.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .constraints(vec![WiderOrEq(Int.clone(), IntTo.clone())])
+        .finish(format_registry),
+    );
+
+    let IntTo = &TypeVarBuilder::new(
+        "IntTo",
+        "A larger integer type with the same number of lanes",
+    )
+    .ints(Interval::All)
+    .simd_lanes(Interval::All)
+    .finish();
+    let x = &operand("x", Int);
+    let a = &operand("a", IntTo);
+
+    ig.push(
+        Inst::new(
+            "uextend",
+            r#"
+        Convert `x` to a larger integer type by zero-extending.
+
+        Each lane in `x` is converted to a larger integer type by adding
+        zeroes. The result has the same numerical value as `x` when both are
+        interpreted as unsigned integers.
+
+        The result type must have the same number of vector lanes as the input,
+        and each lane must not have fewer bits that the input lanes. If the
+        input and output types are the same, this is a no-op.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .constraints(vec![WiderOrEq(IntTo.clone(), Int.clone())])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "sextend",
+            r#"
+        Convert `x` to a larger integer type by sign-extending.
+
+        Each lane in `x` is converted to a larger integer type by replicating
+        the sign bit. The result has the same numerical value as `x` when both
+        are interpreted as signed integers.
+
+        The result type must have the same number of vector lanes as the input,
+        and each lane must not have fewer bits that the input lanes. If the
+        input and output types are the same, this is a no-op.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .constraints(vec![WiderOrEq(IntTo.clone(), Int.clone())])
+        .finish(format_registry),
+    );
+
+    let FloatTo = &TypeVarBuilder::new("FloatTo", "A scalar or vector floating point number")
+        .floats(Interval::All)
+        .simd_lanes(Interval::All)
+        .finish();
+    let x = &operand("x", Float);
+    let a = &operand("a", FloatTo);
+
+    ig.push(
+        Inst::new(
+            "fpromote",
+            r#"
+        Convert `x` to a larger floating point format.
+
+        Each lane in `x` is converted to the destination floating point format.
+        This is an exact operation.
+
+        Cranelift currently only supports two floating point formats
+        - :type:`f32` and :type:`f64`. This may change in the future.
+
+        The result type must have the same number of vector lanes as the input,
+        and the result lanes must not have fewer bits than the input lanes. If
+        the input and output types are the same, this is a no-op.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .constraints(vec![WiderOrEq(FloatTo.clone(), Float.clone())])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "fdemote",
+            r#"
+        Convert `x` to a smaller floating point format.
+
+        Each lane in `x` is converted to the destination floating point format
+        by rounding to nearest, ties to even.
+
+        Cranelift currently only supports two floating point formats
+        - :type:`f32` and :type:`f64`. This may change in the future.
+
+        The result type must have the same number of vector lanes as the input,
+        and the result lanes must not have more bits than the input lanes. If
+        the input and output types are the same, this is a no-op.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .constraints(vec![WiderOrEq(Float.clone(), FloatTo.clone())])
+        .finish(format_registry),
+    );
+
+    let x = &operand("x", Float);
+    let a = &operand("a", IntTo);
+
+    ig.push(
+        Inst::new(
+            "fcvt_to_uint",
+            r#"
+        Convert floating point to unsigned integer.
+
+        Each lane in `x` is converted to an unsigned integer by rounding
+        towards zero. If `x` is NaN or if the unsigned integral value cannot be
+        represented in the result type, this instruction traps.
+
+        The result type must have the same number of vector lanes as the input.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .can_trap(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "fcvt_to_uint_sat",
+            r#"
+        Convert floating point to unsigned integer as fcvt_to_uint does, but
+        saturates the input instead of trapping. NaN and negative values are
+        converted to 0.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "fcvt_to_sint",
+            r#"
+        Convert floating point to signed integer.
+
+        Each lane in `x` is converted to a signed integer by rounding towards
+        zero. If `x` is NaN or if the signed integral value cannot be
+        represented in the result type, this instruction traps.
+
+        The result type must have the same number of vector lanes as the input.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .can_trap(true)
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "fcvt_to_sint_sat",
+            r#"
+        Convert floating point to signed integer as fcvt_to_sint does, but
+        saturates the input instead of trapping. NaN values are converted to 0.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let x = &operand("x", Int);
+    let a = &operand("a", FloatTo);
+
+    ig.push(
+        Inst::new(
+            "fcvt_from_uint",
+            r#"
+        Convert unsigned integer to floating point.
+
+        Each lane in `x` is interpreted as an unsigned integer and converted to
+        floating point using round to nearest, ties to even.
+
+        The result type must have the same number of vector lanes as the input.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    ig.push(
+        Inst::new(
+            "fcvt_from_sint",
+            r#"
+        Convert signed integer to floating point.
+
+        Each lane in `x` is interpreted as a signed integer and converted to
+        floating point using round to nearest, ties to even.
+
+        The result type must have the same number of vector lanes as the input.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
+    let WideInt = &TypeVarBuilder::new("WideInt", "An integer type with lanes from `i16` upwards")
+        .ints(16..64)
+        .simd_lanes(Interval::All)
+        .finish();
+    let x = &operand("x", WideInt);
+    let lo = &operand_doc("lo", &WideInt.half_width(), "The low bits of `x`");
+    let hi = &operand_doc("hi", &WideInt.half_width(), "The high bits of `x`");
+
+    ig.push(
+        Inst::new(
+            "isplit",
+            r#"
+        Split an integer into low and high parts.
+
+        Vectors of integers are split lane-wise, so the results have the same
+        number of lanes as the input, but the lanes are half the size.
+
+        Returns the low half of `x` and the high half of `x` as two independent
+        values.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![lo, hi])
+        .is_ghost(true)
+        .finish(format_registry),
+    );
+
+    let NarrowInt = &TypeVarBuilder::new("NarrowInt", "An integer type with lanes type to `i32`")
+        .ints(8..32)
+        .simd_lanes(Interval::All)
+        .finish();
+
+    let lo = &operand("lo", NarrowInt);
+    let hi = &operand("hi", NarrowInt);
+    let a = &operand_doc(
+        "a",
+        &NarrowInt.double_width(),
+        "The concatenation of `lo` and `hi`",
+    );
+
+    ig.push(
+        Inst::new(
+            "iconcat",
+            r#"
+        Concatenate low and high bits to form a larger integer type.
+
+        Vectors of integers are concatenated lane-wise such that the result has
+        the same number of lanes as the inputs, but the lanes are twice the
+        size.
+        "#,
+        )
+        .operands_in(vec![lo, hi])
+        .operands_out(vec![a])
+        .is_ghost(true)
+        .finish(format_registry),
+    );
+
+    ig
+}
diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
index c8dd17ff4b..0e6a11f0e1 100644
--- a/cranelift/codegen/meta/src/shared/mod.rs
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -3,15 +3,18 @@
 pub mod entities;
 pub mod formats;
 pub mod immediates;
+pub mod instructions;
 pub mod settings;
 pub mod types;
 
 use crate::cdsl::formats::FormatRegistry;
+use crate::cdsl::inst::InstructionGroup;
 use crate::cdsl::operands::OperandKind;
 use crate::cdsl::settings::SettingGroup;
 
 pub struct Definitions {
     pub settings: SettingGroup,
+    pub instructions: InstructionGroup,
     pub operand_kinds: OperandKinds,
     pub format_registry: FormatRegistry,
 }
@@ -49,6 +52,7 @@ pub fn define() -> Definitions {
     let format_registry = formats::define(&immediates, &entities);
     Definitions {
         settings: settings::define(),
+        instructions: instructions::define(&format_registry, &immediates, &entities),
         operand_kinds: immediates,
         format_registry,
     }

From 27f4163b89ff57a5f9ac2dd6ffcb993813493534 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 28 Mar 2019 12:42:37 +0100
Subject: [PATCH 2363/3084] Fixes #716: Stub out parse_x86_cpuid in the non-x86
 case;

---
 cranelift/native/src/lib.rs | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/cranelift/native/src/lib.rs b/cranelift/native/src/lib.rs
index 010acb25ac..0687e70173 100644
--- a/cranelift/native/src/lib.rs
+++ b/cranelift/native/src/lib.rs
@@ -88,6 +88,11 @@ fn parse_x86_cpuid(isa_builder: &mut isa::Builder) -> Result<(), &'static str> {
     Ok(())
 }
 
+#[cfg(not(any(target_arch = "x86", target_arch = "x86_64")))]
+fn parse_x86_cpuid(_isa_builder: &mut isa::Builder) -> Result<(), &'static str> {
+    unreachable!();
+}
+
 #[cfg(test)]
 mod tests {
     use super::builder;

From d1bb93294e896d4ab87c0c36889f8959a858cefc Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Sun, 31 Mar 2019 06:31:58 -0700
Subject: [PATCH 2364/3084] Update Travis from trusty to xenial, and to Rust
 1.33.

---
 .travis.yml         | 4 ++--
 cranelift/README.md | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 1540cd5407..f1460003c7 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,7 +7,7 @@ language: rust
 rust:
     # The oldest version we currently support. See
     # CONTRIBUTING.md#rustc-version-support for details.
-    - 1.32.0
+    - 1.33.0
     - beta
     - nightly
 matrix:
@@ -19,7 +19,7 @@ matrix:
     # Similarly, we don't need to hold up people using stable while we wait
     # for the results which may fail.
     fast_finish: true
-dist: trusty
+dist: xenial
 sudo: false
 addons:
     apt:
diff --git a/cranelift/README.md b/cranelift/README.md
index 166ad1a1b8..c68452f2c0 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -10,7 +10,7 @@ into executable machine code.
 [![Travis Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
 [![Appveyor Status](https://ci.appveyor.com/api/projects/status/oub7wrrb59utuv8x?svg=true)](https://ci.appveyor.com/project/CraneStation/cranelift)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
-![Minimum rustc 1.31](https://img.shields.io/badge/rustc-1.31+-green.svg)
+![Minimum rustc 1.33](https://img.shields.io/badge/rustc-1.33+-green.svg)
 
 For more information, see [the
 documentation](https://cranelift.readthedocs.io/en/latest/?badge=latest).

From 3e7543df7972c5d8ebfc927f77ad931a1bfa5297 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 29 Mar 2019 11:45:15 +0100
Subject: [PATCH 2365/3084] [meta] Introduce the num_set! macro to create
 NumSet as vec! does;

---
 cranelift/codegen/meta/src/cdsl/typevar.rs | 50 +++++++++++-----------
 1 file changed, 25 insertions(+), 25 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 4db5c96e2a..5e722f0063 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -236,6 +236,12 @@ type RangeBound = u16;
 type Range = ops::Range;
 type NumSet = BTreeSet;
 
+macro_rules! num_set {
+    ($($expr:expr),*) => {
+        NumSet::from_iter(vec![$($expr),*])
+    };
+}
+
 #[derive(Debug, Clone, PartialEq, Eq, Hash)]
 pub struct TypeSet {
     pub lanes: NumSet,
@@ -288,7 +294,7 @@ impl TypeSet {
     /// Return a TypeSet describing the image of self across lane_of.
     fn lane_of(&self) -> TypeSet {
         let mut copy = self.clone();
-        copy.lanes = NumSet::from_iter(vec![1]);
+        copy.lanes = num_set![1];
         copy.bitvecs = NumSet::new();
         copy
     }
@@ -299,7 +305,7 @@ impl TypeSet {
         copy.ints = NumSet::new();
         copy.floats = NumSet::new();
         copy.bitvecs = NumSet::new();
-        if (&self.lanes - &NumSet::from_iter(vec![1])).len() > 0 {
+        if (&self.lanes - &num_set![1]).len() > 0 {
             copy.bools = &self.ints | &self.floats;
             copy.bools = ©.bools | &self.bools;
         }
@@ -376,7 +382,7 @@ impl TypeSet {
         let all_scalars = &(&self.ints | &self.floats) | &self.bools;
 
         let mut copy = self.clone();
-        copy.lanes = NumSet::from_iter(vec![1]);
+        copy.lanes = num_set![1];
         copy.ints = NumSet::new();
         copy.bools = NumSet::new();
         copy.floats = NumSet::new();
@@ -617,12 +623,9 @@ fn test_typevar_builder() {
     let typevar = TypeVarBuilder::new("test", "scalar integers")
         .ints(Interval::All)
         .finish();
-    assert_eq!(typevar.type_set.lanes, NumSet::from_iter(vec![1]));
+    assert_eq!(typevar.type_set.lanes, num_set![1]);
     assert!(typevar.type_set.floats.is_empty());
-    assert_eq!(
-        typevar.type_set.ints,
-        NumSet::from_iter(vec![8, 16, 32, 64])
-    );
+    assert_eq!(typevar.type_set.ints, num_set![8, 16, 32, 64]);
     assert!(typevar.type_set.bools.is_empty());
     assert!(typevar.type_set.bitvecs.is_empty());
     assert!(typevar.type_set.specials.is_empty());
@@ -630,21 +633,18 @@ fn test_typevar_builder() {
     let typevar = TypeVarBuilder::new("test", "scalar bools")
         .bools(Interval::All)
         .finish();
-    assert_eq!(typevar.type_set.lanes, NumSet::from_iter(vec![1]));
+    assert_eq!(typevar.type_set.lanes, num_set![1]);
     assert!(typevar.type_set.floats.is_empty());
     assert!(typevar.type_set.ints.is_empty());
-    assert_eq!(
-        typevar.type_set.bools,
-        NumSet::from_iter(vec![1, 8, 16, 32, 64])
-    );
+    assert_eq!(typevar.type_set.bools, num_set![1, 8, 16, 32, 64]);
     assert!(typevar.type_set.bitvecs.is_empty());
     assert!(typevar.type_set.specials.is_empty());
 
     let typevar = TypeVarBuilder::new("test", "scalar floats")
         .floats(Interval::All)
         .finish();
-    assert_eq!(typevar.type_set.lanes, NumSet::from_iter(vec![1]));
-    assert_eq!(typevar.type_set.floats, NumSet::from_iter(vec![32, 64]));
+    assert_eq!(typevar.type_set.lanes, num_set![1]);
+    assert_eq!(typevar.type_set.floats, num_set![32, 64]);
     assert!(typevar.type_set.ints.is_empty());
     assert!(typevar.type_set.bools.is_empty());
     assert!(typevar.type_set.bitvecs.is_empty());
@@ -657,9 +657,9 @@ fn test_typevar_builder() {
         .finish();
     assert_eq!(
         typevar.type_set.lanes,
-        NumSet::from_iter(vec![2, 4, 8, 16, 32, 64, 128, 256])
+        num_set![2, 4, 8, 16, 32, 64, 128, 256]
     );
-    assert_eq!(typevar.type_set.floats, NumSet::from_iter(vec![32, 64]));
+    assert_eq!(typevar.type_set.floats, num_set![32, 64]);
     assert!(typevar.type_set.ints.is_empty());
     assert!(typevar.type_set.bools.is_empty());
     assert!(typevar.type_set.bitvecs.is_empty());
@@ -672,9 +672,9 @@ fn test_typevar_builder() {
         .finish();
     assert_eq!(
         typevar.type_set.lanes,
-        NumSet::from_iter(vec![1, 2, 4, 8, 16, 32, 64, 128, 256])
+        num_set![1, 2, 4, 8, 16, 32, 64, 128, 256]
     );
-    assert_eq!(typevar.type_set.floats, NumSet::from_iter(vec![32, 64]));
+    assert_eq!(typevar.type_set.floats, num_set![32, 64]);
     assert!(typevar.type_set.ints.is_empty());
     assert!(typevar.type_set.bools.is_empty());
     assert!(typevar.type_set.bitvecs.is_empty());
@@ -683,8 +683,8 @@ fn test_typevar_builder() {
     let typevar = TypeVarBuilder::new("test", "range of ints")
         .ints(16..64)
         .finish();
-    assert_eq!(typevar.type_set.lanes, NumSet::from_iter(vec![1]));
-    assert_eq!(typevar.type_set.ints, NumSet::from_iter(vec![16, 32, 64]));
+    assert_eq!(typevar.type_set.lanes, num_set![1]);
+    assert_eq!(typevar.type_set.ints, num_set![16, 32, 64]);
     assert!(typevar.type_set.floats.is_empty());
     assert!(typevar.type_set.bools.is_empty());
     assert!(typevar.type_set.bitvecs.is_empty());
@@ -716,12 +716,12 @@ fn test_singleton() {
     let typevar =
         TypeVarBuilder::singleton(ValueType::Lane(LaneType::IntType(shared_types::Int::I32)));
     assert_eq!(typevar.name, "i32");
-    assert_eq!(typevar.type_set.ints, NumSet::from_iter(vec![32]));
+    assert_eq!(typevar.type_set.ints, num_set![32]);
     assert!(typevar.type_set.floats.is_empty());
     assert!(typevar.type_set.bools.is_empty());
     assert!(typevar.type_set.bitvecs.is_empty());
     assert!(typevar.type_set.specials.is_empty());
-    assert_eq!(typevar.type_set.lanes, NumSet::from_iter(vec![1]));
+    assert_eq!(typevar.type_set.lanes, num_set![1]);
 
     // Test f32x4.
     let typevar = TypeVarBuilder::singleton(ValueType::Vector(VectorType::new(
@@ -730,8 +730,8 @@ fn test_singleton() {
     )));
     assert_eq!(typevar.name, "f32x4");
     assert!(typevar.type_set.ints.is_empty());
-    assert_eq!(typevar.type_set.floats, NumSet::from_iter(vec![32]));
-    assert_eq!(typevar.type_set.lanes, NumSet::from_iter(vec![4]));
+    assert_eq!(typevar.type_set.floats, num_set![32]);
+    assert_eq!(typevar.type_set.lanes, num_set![4]);
     assert!(typevar.type_set.bools.is_empty());
     assert!(typevar.type_set.bitvecs.is_empty());
     assert!(typevar.type_set.specials.is_empty());

From 605320112886cf619f27df6ef321b5f59a914f55 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 29 Mar 2019 15:02:13 +0100
Subject: [PATCH 2366/3084] [meta] Move the TypeSet building out of the TypeVar
 builder so as to test it;

---
 cranelift/codegen/meta/src/cdsl/typevar.rs    | 362 +++++++++---------
 .../codegen/meta/src/isa/x86/instructions.rs  |  34 +-
 .../codegen/meta/src/shared/instructions.rs   | 328 ++++++++++------
 3 files changed, 397 insertions(+), 327 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 5e722f0063..fb5064f912 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -36,6 +36,54 @@ pub struct TypeVar {
 }
 
 impl TypeVar {
+    pub fn new(name: impl Into, doc: impl Into, type_set: TypeSet) -> Self {
+        Self {
+            content: Rc::new(TypeVarContent {
+                name: name.into(),
+                doc: doc.into(),
+                type_set: Rc::new(type_set),
+                base: None,
+            }),
+        }
+    }
+
+    pub fn new_singleton(value_type: ValueType) -> Self {
+        let (name, doc) = (value_type.to_string(), value_type.doc());
+        let mut builder = TypeSetBuilder::new();
+
+        let (scalar_type, num_lanes) = match value_type {
+            ValueType::BV(bitvec_type) => {
+                let bits = bitvec_type.lane_bits() as RangeBound;
+                return TypeVar::new(name, doc, builder.bitvecs(bits..bits).finish());
+            }
+            ValueType::Special(special_type) => {
+                return TypeVar::new(name, doc, builder.specials(vec![special_type]).finish());
+            }
+            ValueType::Lane(lane_type) => (lane_type, 1),
+            ValueType::Vector(vec_type) => {
+                (vec_type.lane_type(), vec_type.lane_count() as RangeBound)
+            }
+        };
+
+        builder = builder.simd_lanes(num_lanes..num_lanes);
+
+        let builder = match scalar_type {
+            LaneType::IntType(int_type) => {
+                let bits = int_type as RangeBound;
+                builder.ints(bits..bits)
+            }
+            LaneType::FloatType(float_type) => {
+                let bits = float_type as RangeBound;
+                builder.floats(bits..bits)
+            }
+            LaneType::BoolType(bool_type) => {
+                let bits = bool_type as RangeBound;
+                builder.bools(bits..bits)
+            }
+        };
+        TypeVar::new(name, doc, builder.finish())
+    }
+
     fn get_typeset(&self) -> Rc {
         match &self.content.base {
             Some(base) => Rc::new(base.type_var.get_typeset().image(base.derived_func)),
@@ -162,7 +210,7 @@ impl Into for &TypeVar {
 }
 impl Into for ValueType {
     fn into(self) -> TypeVar {
-        TypeVarBuilder::singleton(self)
+        TypeVar::new_singleton(self)
     }
 }
 
@@ -429,6 +477,83 @@ impl TypeSet {
     }
 }
 
+pub struct TypeSetBuilder {
+    ints: Interval,
+    floats: Interval,
+    bools: Interval,
+    bitvecs: Interval,
+    includes_scalars: bool,
+    simd_lanes: Interval,
+    specials: Vec,
+}
+
+impl TypeSetBuilder {
+    pub fn new() -> Self {
+        Self {
+            ints: Interval::None,
+            floats: Interval::None,
+            bools: Interval::None,
+            bitvecs: Interval::None,
+            includes_scalars: true,
+            simd_lanes: Interval::None,
+            specials: Vec::new(),
+        }
+    }
+
+    pub fn ints(mut self, interval: impl Into) -> Self {
+        assert!(self.ints == Interval::None);
+        self.ints = interval.into();
+        self
+    }
+    pub fn floats(mut self, interval: impl Into) -> Self {
+        assert!(self.floats == Interval::None);
+        self.floats = interval.into();
+        self
+    }
+    pub fn bools(mut self, interval: impl Into) -> Self {
+        assert!(self.bools == Interval::None);
+        self.bools = interval.into();
+        self
+    }
+    pub fn includes_scalars(mut self, includes_scalars: bool) -> Self {
+        self.includes_scalars = includes_scalars;
+        self
+    }
+    pub fn simd_lanes(mut self, interval: impl Into) -> Self {
+        assert!(self.simd_lanes == Interval::None);
+        self.simd_lanes = interval.into();
+        self
+    }
+    pub fn bitvecs(mut self, interval: impl Into) -> Self {
+        assert!(self.bitvecs == Interval::None);
+        self.bitvecs = interval.into();
+        self
+    }
+    pub fn specials(mut self, specials: Vec) -> Self {
+        assert!(self.specials.is_empty());
+        self.specials = specials;
+        self
+    }
+
+    pub fn finish(self) -> TypeSet {
+        let min_lanes = if self.includes_scalars { 1 } else { 2 };
+;
+        let bools = range_to_set(self.bools.to_range(1..MAX_BITS, None))
+            .into_iter()
+            .filter(legal_bool)
+            .collect();
+
+        TypeSet::new(
+            range_to_set(self.simd_lanes.to_range(min_lanes..MAX_LANES, Some(1))),
+            range_to_set(self.ints.to_range(8..MAX_BITS, None)),
+            range_to_set(self.floats.to_range(32..64, None)),
+            bools,
+            range_to_set(self.bitvecs.to_range(1..MAX_BITVEC, None)),
+            self.specials,
+        )
+    }
+}
+
 #[derive(PartialEq)]
 pub enum Interval {
     None,
@@ -468,131 +593,6 @@ impl Into for Range {
     }
 }
 
-pub struct TypeVarBuilder {
-    name: String,
-    doc: String,
-    ints: Interval,
-    floats: Interval,
-    bools: Interval,
-    bitvecs: Interval,
-    includes_scalars: bool,
-    simd_lanes: Interval,
-    specials: Vec,
-}
-
-impl TypeVarBuilder {
-    pub fn new(name: impl Into, doc: impl Into) -> Self {
-        Self {
-            name: name.into(),
-            doc: doc.into(),
-            ints: Interval::None,
-            floats: Interval::None,
-            bools: Interval::None,
-            bitvecs: Interval::None,
-            includes_scalars: true,
-            simd_lanes: Interval::None,
-            specials: Vec::new(),
-        }
-    }
-
-    pub fn singleton(value_type: ValueType) -> TypeVar {
-        let mut builder = TypeVarBuilder::new(value_type.to_string(), value_type.doc());
-
-        let (scalar_type, num_lanes) = match value_type {
-            ValueType::BV(bitvec_type) => {
-                let bits = bitvec_type.lane_bits() as RangeBound;
-                return builder.bitvecs(bits..bits).finish();
-            }
-            ValueType::Special(special_type) => {
-                return builder.specials(vec![special_type]).finish();
-            }
-            ValueType::Lane(lane_type) => (lane_type, 1),
-            ValueType::Vector(vec_type) => {
-                (vec_type.lane_type(), vec_type.lane_count() as RangeBound)
-            }
-        };
-
-        builder = builder.simd_lanes(num_lanes..num_lanes);
-
-        match scalar_type {
-            LaneType::IntType(int_type) => {
-                let bits = int_type as RangeBound;
-                return builder.ints(bits..bits).finish();
-            }
-            LaneType::FloatType(float_type) => {
-                let bits = float_type as RangeBound;
-                return builder.floats(bits..bits).finish();
-            }
-            LaneType::BoolType(bool_type) => {
-                let bits = bool_type as RangeBound;
-                return builder.bools(bits..bits).finish();
-            }
-        }
-    }
-
-    pub fn ints(mut self, interval: impl Into) -> Self {
-        assert!(self.ints == Interval::None);
-        self.ints = interval.into();
-        self
-    }
-    pub fn floats(mut self, interval: impl Into) -> Self {
-        assert!(self.floats == Interval::None);
-        self.floats = interval.into();
-        self
-    }
-    pub fn bools(mut self, interval: impl Into) -> Self {
-        assert!(self.bools == Interval::None);
-        self.bools = interval.into();
-        self
-    }
-    pub fn includes_scalars(mut self, includes_scalars: bool) -> Self {
-        self.includes_scalars = includes_scalars;
-        self
-    }
-    pub fn simd_lanes(mut self, interval: impl Into) -> Self {
-        assert!(self.simd_lanes == Interval::None);
-        self.simd_lanes = interval.into();
-        self
-    }
-    pub fn bitvecs(mut self, interval: impl Into) -> Self {
-        assert!(self.bitvecs == Interval::None);
-        self.bitvecs = interval.into();
-        self
-    }
-    pub fn specials(mut self, specials: Vec) -> Self {
-        assert!(self.specials.is_empty());
-        self.specials = specials;
-        self
-    }
-
-    pub fn finish(self) -> TypeVar {
-        let min_lanes = if self.includes_scalars { 1 } else { 2 };
-
-        let bools = range_to_set(self.bools.to_range(1..MAX_BITS, None))
-            .into_iter()
-            .filter(legal_bool)
-            .collect();
-
-        let type_set = Rc::new(TypeSet::new(
-            range_to_set(self.simd_lanes.to_range(min_lanes..MAX_LANES, Some(1))),
-            range_to_set(self.ints.to_range(8..MAX_BITS, None)),
-            range_to_set(self.floats.to_range(32..64, None)),
-            bools,
-            range_to_set(self.bitvecs.to_range(1..MAX_BITVEC, None)),
-            self.specials,
-        ));
-
-        TypeVar {
-            content: Rc::new(TypeVarContent {
-                name: self.name.to_string(),
-                doc: self.doc.to_string(),
-                type_set,
-                base: None,
-            }),
-        }
-    }
-}
-
 fn legal_bool(bits: &RangeBound) -> bool {
     // Only allow legal bit widths for bool types.
     *bits == 1 || (*bits >= 8 && *bits <= MAX_BITS && bits.is_power_of_two())
@@ -620,91 +620,73 @@ fn range_to_set(range: Option) -> NumSet {
 
 #[test]
 fn test_typevar_builder() {
-    let typevar = TypeVarBuilder::new("test", "scalar integers")
-        .ints(Interval::All)
-        .finish();
-    assert_eq!(typevar.type_set.lanes, num_set![1]);
-    assert!(typevar.type_set.floats.is_empty());
-    assert_eq!(typevar.type_set.ints, num_set![8, 16, 32, 64]);
-    assert!(typevar.type_set.bools.is_empty());
-    assert!(typevar.type_set.bitvecs.is_empty());
-    assert!(typevar.type_set.specials.is_empty());
+    let type_set = TypeSetBuilder::new().ints(Interval::All).finish();
+    assert_eq!(type_set.lanes, num_set![1]);
+    assert!(type_set.floats.is_empty());
+    assert_eq!(type_set.ints, num_set![8, 16, 32, 64]);
+    assert!(type_set.bools.is_empty());
+    assert!(type_set.bitvecs.is_empty());
+    assert!(type_set.specials.is_empty());
 
-    let typevar = TypeVarBuilder::new("test", "scalar bools")
-        .bools(Interval::All)
-        .finish();
-    assert_eq!(typevar.type_set.lanes, num_set![1]);
-    assert!(typevar.type_set.floats.is_empty());
-    assert!(typevar.type_set.ints.is_empty());
-    assert_eq!(typevar.type_set.bools, num_set![1, 8, 16, 32, 64]);
-    assert!(typevar.type_set.bitvecs.is_empty());
-    assert!(typevar.type_set.specials.is_empty());
+    let type_set = TypeSetBuilder::new().bools(Interval::All).finish();
+    assert_eq!(type_set.lanes, num_set![1]);
+    assert!(type_set.floats.is_empty());
+    assert!(type_set.ints.is_empty());
+    assert_eq!(type_set.bools, num_set![1, 8, 16, 32, 64]);
+    assert!(type_set.bitvecs.is_empty());
+    assert!(type_set.specials.is_empty());
 
-    let typevar = TypeVarBuilder::new("test", "scalar floats")
-        .floats(Interval::All)
-        .finish();
-    assert_eq!(typevar.type_set.lanes, num_set![1]);
-    assert_eq!(typevar.type_set.floats, num_set![32, 64]);
-    assert!(typevar.type_set.ints.is_empty());
-    assert!(typevar.type_set.bools.is_empty());
-    assert!(typevar.type_set.bitvecs.is_empty());
-    assert!(typevar.type_set.specials.is_empty());
+    let type_set = TypeSetBuilder::new().floats(Interval::All).finish();
+    assert_eq!(type_set.lanes, num_set![1]);
+    assert_eq!(type_set.floats, num_set![32, 64]);
+    assert!(type_set.ints.is_empty());
+    assert!(type_set.bools.is_empty());
+    assert!(type_set.bitvecs.is_empty());
+    assert!(type_set.specials.is_empty());
 
-    let typevar = TypeVarBuilder::new("test", "float vectors (but not scalars)")
+    let type_set = TypeSetBuilder::new()
         .floats(Interval::All)
         .simd_lanes(Interval::All)
         .includes_scalars(false)
         .finish();
-    assert_eq!(
-        typevar.type_set.lanes,
-        num_set![2, 4, 8, 16, 32, 64, 128, 256]
-    );
-    assert_eq!(typevar.type_set.floats, num_set![32, 64]);
-    assert!(typevar.type_set.ints.is_empty());
-    assert!(typevar.type_set.bools.is_empty());
-    assert!(typevar.type_set.bitvecs.is_empty());
-    assert!(typevar.type_set.specials.is_empty());
+    assert_eq!(type_set.lanes, num_set![2, 4, 8, 16, 32, 64, 128, 256]);
+    assert_eq!(type_set.floats, num_set![32, 64]);
+    assert!(type_set.ints.is_empty());
+    assert!(type_set.bools.is_empty());
+    assert!(type_set.bitvecs.is_empty());
+    assert!(type_set.specials.is_empty());
 
-    let typevar = TypeVarBuilder::new("test", "float vectors and scalars")
+    let type_set = TypeSetBuilder::new()
         .floats(Interval::All)
         .simd_lanes(Interval::All)
         .includes_scalars(true)
         .finish();
-    assert_eq!(
-        typevar.type_set.lanes,
-        num_set![1, 2, 4, 8, 16, 32, 64, 128, 256]
-    );
-    assert_eq!(typevar.type_set.floats, num_set![32, 64]);
-    assert!(typevar.type_set.ints.is_empty());
-    assert!(typevar.type_set.bools.is_empty());
-    assert!(typevar.type_set.bitvecs.is_empty());
-    assert!(typevar.type_set.specials.is_empty());
+    assert_eq!(type_set.lanes, num_set![1, 2, 4, 8, 16, 32, 64, 128, 256]);
+    assert_eq!(type_set.floats, num_set![32, 64]);
+    assert!(type_set.ints.is_empty());
+    assert!(type_set.bools.is_empty());
+    assert!(type_set.bitvecs.is_empty());
+    assert!(type_set.specials.is_empty());
 
-    let typevar = TypeVarBuilder::new("test", "range of ints")
-        .ints(16..64)
-        .finish();
-    assert_eq!(typevar.type_set.lanes, num_set![1]);
-    assert_eq!(typevar.type_set.ints, num_set![16, 32, 64]);
-    assert!(typevar.type_set.floats.is_empty());
-    assert!(typevar.type_set.bools.is_empty());
-    assert!(typevar.type_set.bitvecs.is_empty());
-    assert!(typevar.type_set.specials.is_empty());
+    let type_set = TypeSetBuilder::new().ints(16..64).finish();
+    assert_eq!(type_set.lanes, num_set![1]);
+    assert_eq!(type_set.ints, num_set![16, 32, 64]);
+    assert!(type_set.floats.is_empty());
+    assert!(type_set.bools.is_empty());
+    assert!(type_set.bitvecs.is_empty());
+    assert!(type_set.specials.is_empty());
 }
 
 #[test]
 #[should_panic]
 fn test_typevar_builder_too_high_bound_panic() {
-    TypeVarBuilder::new("test", "invalid range of ints")
-        .ints(16..2 * MAX_BITS)
-        .finish();
+    TypeSetBuilder::new().ints(16..2 * MAX_BITS).finish();
 }
 
 #[test]
 #[should_panic]
 fn test_typevar_builder_inverted_bounds_panic() {
-    TypeVarBuilder::new("test", "inverted bounds")
-        .ints(32..16)
-        .finish();
+    TypeSetBuilder::new().ints(32..16).finish();
 }
 
 #[test]
@@ -714,7 +696,7 @@ fn test_singleton() {
 
     // Test i32.
     let typevar =
-        TypeVarBuilder::singleton(ValueType::Lane(LaneType::IntType(shared_types::Int::I32)));
+        TypeVar::new_singleton(ValueType::Lane(LaneType::IntType(shared_types::Int::I32)));
     assert_eq!(typevar.name, "i32");
     assert_eq!(typevar.type_set.ints, num_set![32]);
     assert!(typevar.type_set.floats.is_empty());
@@ -724,7 +706,7 @@ fn test_singleton() {
     assert_eq!(typevar.type_set.lanes, num_set![1]);
 
     // Test f32x4.
-    let typevar = TypeVarBuilder::singleton(ValueType::Vector(VectorType::new(
+    let typevar = TypeVar::new_singleton(ValueType::Vector(VectorType::new(
         LaneType::FloatType(shared_types::Float::F32),
         4,
     )));
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 68c3b56b85..077860cc46 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -4,7 +4,7 @@ use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::inst::{InstructionBuilder as Inst, InstructionGroup};
 use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
 use crate::cdsl::types::ValueType;
-use crate::cdsl::typevar::{Interval, TypeVar, TypeVarBuilder};
+use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
 use crate::shared::types;
 
 pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
@@ -12,9 +12,11 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
 
     let iflags: &TypeVar = &ValueType::Special(types::Flag::IFlags.into()).into();
 
-    let iWord = &TypeVarBuilder::new("iWord", "A scalar integer machine word")
-        .ints(32..64)
-        .finish();
+    let iWord = &TypeVar::new(
+        "iWord",
+        "A scalar integer machine word",
+        TypeSetBuilder::new().ints(32..64).finish(),
+    );
     let nlo = &operand_doc("nlo", iWord, "Low part of numerator");
     let nhi = &operand_doc("nhi", iWord, "High part of numerator");
     let d = &operand_doc("d", iWord, "Denominator");
@@ -96,14 +98,22 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
         .finish(format_registry),
     );
 
-    let Float = &TypeVarBuilder::new("Float", "A scalar or vector floating point number")
-        .floats(Interval::All)
-        .simd_lanes(Interval::All)
-        .finish();
-    let IntTo = &TypeVarBuilder::new("IntTo", "An integer type with the same number of lanes")
-        .ints(32..64)
-        .simd_lanes(Interval::All)
-        .finish();
+    let Float = &TypeVar::new(
+        "Float",
+        "A scalar or vector floating point number",
+        TypeSetBuilder::new()
+            .floats(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
+    let IntTo = &TypeVar::new(
+        "IntTo",
+        "An integer type with the same number of lanes",
+        TypeSetBuilder::new()
+            .ints(32..64)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
     let x = &operand("x", Float);
     let a = &operand("a", IntTo);
 
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index a83515ee04..b0c89b510e 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -5,7 +5,7 @@ use crate::cdsl::inst::{InstructionBuilder as Inst, InstructionGroup};
 use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
 use crate::cdsl::type_inference::Constraint::WiderOrEq;
 use crate::cdsl::types::{LaneType, ValueType};
-use crate::cdsl::typevar::{Interval, TypeVar, TypeVarBuilder};
+use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
 use crate::shared::{types, OperandKinds};
 
 pub fn define(
@@ -47,59 +47,88 @@ pub fn define(
     let f64_: &TypeVar = &ValueType::from(LaneType::from(types::Float::F64)).into();
 
     // Starting definitions.
-    let Int = &TypeVarBuilder::new("Int", "A scalar or vector integer type")
-        .ints(Interval::All)
-        .simd_lanes(Interval::All)
-        .finish();
+    let Int = &TypeVar::new(
+        "Int",
+        "A scalar or vector integer type",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
 
-    let Bool = &TypeVarBuilder::new("Bool", "A scalar or vector boolean type")
-        .bools(Interval::All)
-        .simd_lanes(Interval::All)
-        .finish();
+    let Bool = &TypeVar::new(
+        "Bool",
+        "A scalar or vector boolean type",
+        TypeSetBuilder::new()
+            .bools(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
 
-    let iB = &TypeVarBuilder::new("iB", "A scalar integer type")
-        .ints(Interval::All)
-        .finish();
+    let iB = &TypeVar::new(
+        "iB",
+        "A scalar integer type",
+        TypeSetBuilder::new().ints(Interval::All).finish(),
+    );
 
-    let iAddr = &TypeVarBuilder::new("iAddr", "An integer address type")
-        .ints(32..64)
-        .finish();
+    let iAddr = &TypeVar::new(
+        "iAddr",
+        "An integer address type",
+        TypeSetBuilder::new().ints(32..64).finish(),
+    );
 
-    let Testable = &TypeVarBuilder::new("Testable", "A scalar boolean or integer type")
-        .ints(Interval::All)
-        .bools(Interval::All)
-        .finish();
+    let Testable = &TypeVar::new(
+        "Testable",
+        "A scalar boolean or integer type",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .bools(Interval::All)
+            .finish(),
+    );
 
-    let TxN = &TypeVarBuilder::new("TxN", "A SIMD vector type")
-        .ints(Interval::All)
-        .floats(Interval::All)
-        .bools(Interval::All)
-        .simd_lanes(Interval::All)
-        .includes_scalars(false)
-        .finish();
+    let TxN = &TypeVar::new(
+        "TxN",
+        "A SIMD vector type",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .floats(Interval::All)
+            .bools(Interval::All)
+            .simd_lanes(Interval::All)
+            .includes_scalars(false)
+            .finish(),
+    );
 
-    let Any = &TypeVarBuilder::new(
+    let Any = &TypeVar::new(
         "Any",
         "Any integer, float, or boolean scalar or vector type",
-    )
-    .ints(Interval::All)
-    .floats(Interval::All)
-    .bools(Interval::All)
-    .simd_lanes(Interval::All)
-    .includes_scalars(true)
-    .finish();
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .floats(Interval::All)
+            .bools(Interval::All)
+            .simd_lanes(Interval::All)
+            .includes_scalars(true)
+            .finish(),
+    );
 
-    let Mem = &TypeVarBuilder::new("Mem", "Any type that can be stored in memory")
-        .ints(Interval::All)
-        .floats(Interval::All)
-        .simd_lanes(Interval::All)
-        .finish();
+    let Mem = &TypeVar::new(
+        "Mem",
+        "Any type that can be stored in memory",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .floats(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
 
-    let MemTo = &TypeVarBuilder::new("MemTo", "Any type that can be stored in memory")
-        .ints(Interval::All)
-        .floats(Interval::All)
-        .simd_lanes(Interval::All)
-        .finish();
+    let MemTo = &TypeVar::new(
+        "MemTo",
+        "Any type that can be stored in memory",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .floats(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
 
     let addr = &operand("addr", iAddr);
     let c = &operand_doc("c", Testable, "Controlling value to test");
@@ -233,9 +262,11 @@ pub fn define(
 
     let x = &operand_doc("x", iB, "index into jump table");
 
-    let Entry = &TypeVarBuilder::new("Entry", "A scalar integer type")
-        .ints(Interval::All)
-        .finish();
+    let Entry = &TypeVar::new(
+        "Entry",
+        "A scalar integer type",
+        TypeSetBuilder::new().ints(Interval::All).finish(),
+    );
 
     let entry = &operand_doc("entry", Entry, "entry of jump table");
     let JT = &operand("JT", jump_table);
@@ -578,9 +609,11 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let iExt8 = &TypeVarBuilder::new("iExt8", "An integer type with more than 8 bits")
-        .ints(16..64)
-        .finish();
+    let iExt8 = &TypeVar::new(
+        "iExt8",
+        "An integer type with more than 8 bits",
+        TypeSetBuilder::new().ints(16..64).finish(),
+    );
     let x = &operand("x", iExt8);
     let a = &operand("a", iExt8);
 
@@ -672,9 +705,11 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let iExt16 = &TypeVarBuilder::new("iExt16", "An integer type with more than 16 bits")
-        .ints(32..64)
-        .finish();
+    let iExt16 = &TypeVar::new(
+        "iExt16",
+        "An integer type with more than 16 bits",
+        TypeSetBuilder::new().ints(32..64).finish(),
+    );
     let x = &operand("x", iExt16);
     let a = &operand("a", iExt16);
 
@@ -766,9 +801,11 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let iExt32 = &TypeVarBuilder::new("iExt32", "An integer type with more than 32 bits")
-        .ints(64..64)
-        .finish();
+    let iExt32 = &TypeVar::new(
+        "iExt32",
+        "An integer type with more than 32 bits",
+        TypeSetBuilder::new().ints(64..64).finish(),
+    );
     let x = &operand("x", iExt32);
     let a = &operand("a", iExt32);
 
@@ -945,9 +982,11 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let HeapOffset = &TypeVarBuilder::new("HeapOffset", "An unsigned heap offset")
-        .ints(32..64)
-        .finish();
+    let HeapOffset = &TypeVar::new(
+        "HeapOffset",
+        "An unsigned heap offset",
+        TypeSetBuilder::new().ints(32..64).finish(),
+    );
 
     let H = &operand("H", heap);
     let p = &operand("p", HeapOffset);
@@ -973,9 +1012,11 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let TableOffset = &TypeVarBuilder::new("TableOffset", "An unsigned table offset")
-        .ints(32..64)
-        .finish();
+    let TableOffset = &TypeVar::new(
+        "TableOffset",
+        "An unsigned table offset",
+        TypeSetBuilder::new().ints(32..64).finish(),
+    );
     let T = &operand("T", table);
     let p = &operand("p", TableOffset);
     let Offset = &operand_doc("Offset", offset32, "Byte offset from element address");
@@ -1342,13 +1383,17 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let Any128 = &TypeVarBuilder::new("Any128", "Any scalar or vector type with as most 128 lanes")
-        .ints(Interval::All)
-        .floats(Interval::All)
-        .bools(Interval::All)
-        .simd_lanes(1..128)
-        .includes_scalars(true)
-        .finish();
+    let Any128 = &TypeVar::new(
+        "Any128",
+        "Any scalar or vector type with as most 128 lanes",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .floats(Interval::All)
+            .bools(Interval::All)
+            .simd_lanes(1..128)
+            .includes_scalars(true)
+            .finish(),
+    );
 
     let x = &operand_doc("x", Any128, "Low-numbered lanes");
     let y = &operand_doc("y", Any128, "High-numbered lanes");
@@ -1935,16 +1980,17 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let bits = &TypeVarBuilder::new(
+    let bits = &TypeVar::new(
         "bits",
         "Any integer, float, or boolean scalar or vector type",
-    )
-    .ints(Interval::All)
-    .floats(Interval::All)
-    .bools(Interval::All)
-    .simd_lanes(Interval::All)
-    .includes_scalars(true)
-    .finish();
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .floats(Interval::All)
+            .bools(Interval::All)
+            .simd_lanes(Interval::All)
+            .includes_scalars(true)
+            .finish(),
+    );
     let x = &operand("x", bits);
     let y = &operand("y", bits);
     let a = &operand("a", bits);
@@ -2331,10 +2377,14 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let Float = &TypeVarBuilder::new("Float", "A scalar or vector floating point number")
-        .floats(Interval::All)
-        .simd_lanes(Interval::All)
-        .finish();
+    let Float = &TypeVar::new(
+        "Float",
+        "A scalar or vector floating point number",
+        TypeSetBuilder::new()
+            .floats(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
     let Cond = &operand("Cond", floatcc);
     let x = &operand("x", Float);
     let y = &operand("y", Float);
@@ -2703,18 +2753,23 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let Bool = &TypeVarBuilder::new("Bool", "A scalar or vector boolean type")
-        .bools(Interval::All)
-        .simd_lanes(Interval::All)
-        .finish();
+    let Bool = &TypeVar::new(
+        "Bool",
+        "A scalar or vector boolean type",
+        TypeSetBuilder::new()
+            .bools(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
 
-    let BoolTo = &TypeVarBuilder::new(
+    let BoolTo = &TypeVar::new(
         "BoolTo",
         "A smaller boolean type with the same number of lanes",
-    )
-    .bools(Interval::All)
-    .simd_lanes(Interval::All)
-    .finish();
+        TypeSetBuilder::new()
+            .bools(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
 
     let x = &operand("x", Bool);
     let a = &operand("a", BoolTo);
@@ -2736,13 +2791,14 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let BoolTo = &TypeVarBuilder::new(
+    let BoolTo = &TypeVar::new(
         "BoolTo",
         "A larger boolean type with the same number of lanes",
-    )
-    .bools(Interval::All)
-    .simd_lanes(Interval::All)
-    .finish();
+        TypeSetBuilder::new()
+            .bools(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
     let x = &operand("x", Bool);
     let a = &operand("a", BoolTo);
 
@@ -2763,10 +2819,14 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let IntTo = &TypeVarBuilder::new("IntTo", "An integer type with the same number of lanes")
-        .ints(Interval::All)
-        .simd_lanes(Interval::All)
-        .finish();
+    let IntTo = &TypeVar::new(
+        "IntTo",
+        "An integer type with the same number of lanes",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
     let x = &operand("x", Bool);
     let a = &operand("a", IntTo);
 
@@ -2800,18 +2860,23 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let Int = &TypeVarBuilder::new("Int", "A scalar or vector integer type")
-        .ints(Interval::All)
-        .simd_lanes(Interval::All)
-        .finish();
+    let Int = &TypeVar::new(
+        "Int",
+        "A scalar or vector integer type",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
 
-    let IntTo = &TypeVarBuilder::new(
+    let IntTo = &TypeVar::new(
         "IntTo",
         "A smaller integer type with the same number of lanes",
-    )
-    .ints(Interval::All)
-    .simd_lanes(Interval::All)
-    .finish();
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
     let x = &operand("x", Int);
     let a = &operand("a", IntTo);
 
@@ -2836,13 +2901,14 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let IntTo = &TypeVarBuilder::new(
+    let IntTo = &TypeVar::new(
         "IntTo",
         "A larger integer type with the same number of lanes",
-    )
-    .ints(Interval::All)
-    .simd_lanes(Interval::All)
-    .finish();
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
     let x = &operand("x", Int);
     let a = &operand("a", IntTo);
 
@@ -2888,10 +2954,14 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let FloatTo = &TypeVarBuilder::new("FloatTo", "A scalar or vector floating point number")
-        .floats(Interval::All)
-        .simd_lanes(Interval::All)
-        .finish();
+    let FloatTo = &TypeVar::new(
+        "FloatTo",
+        "A scalar or vector floating point number",
+        TypeSetBuilder::new()
+            .floats(Interval::All)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
     let x = &operand("x", Float);
     let a = &operand("a", FloatTo);
 
@@ -3046,10 +3116,14 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let WideInt = &TypeVarBuilder::new("WideInt", "An integer type with lanes from `i16` upwards")
-        .ints(16..64)
-        .simd_lanes(Interval::All)
-        .finish();
+    let WideInt = &TypeVar::new(
+        "WideInt",
+        "An integer type with lanes from `i16` upwards",
+        TypeSetBuilder::new()
+            .ints(16..64)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
     let x = &operand("x", WideInt);
     let lo = &operand_doc("lo", &WideInt.half_width(), "The low bits of `x`");
     let hi = &operand_doc("hi", &WideInt.half_width(), "The high bits of `x`");
@@ -3073,10 +3147,14 @@ pub fn define(
         .finish(format_registry),
     );
 
-    let NarrowInt = &TypeVarBuilder::new("NarrowInt", "An integer type with lanes type to `i32`")
-        .ints(8..32)
-        .simd_lanes(Interval::All)
-        .finish();
+    let NarrowInt = &TypeVar::new(
+        "NarrowInt",
+        "An integer type with lanes type to `i32`",
+        TypeSetBuilder::new()
+            .ints(8..32)
+            .simd_lanes(Interval::All)
+            .finish(),
+    );
 
     let lo = &operand("lo", NarrowInt);
     let hi = &operand("hi", NarrowInt);

From ac37cb9bf5d62384642a1317e18101a10acd0f66 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 29 Mar 2019 15:28:23 +0100
Subject: [PATCH 2367/3084] [meta] Add test for {half/double}_{vector/width}
 and as_bool;

---
 cranelift/codegen/meta/src/cdsl/typevar.rs | 105 +++++++++++++++++++++
 1 file changed, 105 insertions(+)

diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index fb5064f912..54e969b3cf 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -689,6 +689,111 @@ fn test_typevar_builder_inverted_bounds_panic() {
     TypeSetBuilder::new().ints(32..16).finish();
 }
 
+#[test]
+fn test_as_bool() {
+    let a = TypeSetBuilder::new()
+        .simd_lanes(2..8)
+        .ints(8..8)
+        .floats(32..32)
+        .finish();
+    assert_eq!(
+        a.lane_of(),
+        TypeSetBuilder::new().ints(8..8).floats(32..32).finish()
+    );
+
+    // Test as_bool with disjoint intervals.
+    let mut a_as_bool = TypeSetBuilder::new().simd_lanes(2..8).finish();
+    a_as_bool.bools = num_set![8, 32];
+    assert_eq!(a.as_bool(), a_as_bool);
+
+    let b = TypeSetBuilder::new()
+        .simd_lanes(1..8)
+        .ints(8..8)
+        .floats(32..32)
+        .finish();
+    let mut b_as_bool = TypeSetBuilder::new().simd_lanes(1..8).finish();
+    b_as_bool.bools = num_set![1, 8, 32];
+    assert_eq!(b.as_bool(), b_as_bool);
+}
+
+#[test]
+fn test_forward_images() {
+    let empty_set = TypeSetBuilder::new().finish();
+
+    // Half vector.
+    assert_eq!(
+        TypeSetBuilder::new()
+            .simd_lanes(1..32)
+            .finish()
+            .half_vector(),
+        TypeSetBuilder::new().simd_lanes(1..16).finish()
+    );
+
+    // Double vector.
+    assert_eq!(
+        TypeSetBuilder::new()
+            .simd_lanes(1..32)
+            .finish()
+            .double_vector(),
+        TypeSetBuilder::new().simd_lanes(2..64).finish()
+    );
+    assert_eq!(
+        TypeSetBuilder::new()
+            .simd_lanes(128..256)
+            .finish()
+            .double_vector(),
+        TypeSetBuilder::new().simd_lanes(256..256).finish()
+    );
+
+    // Half width.
+    assert_eq!(
+        TypeSetBuilder::new().ints(8..32).finish().half_width(),
+        TypeSetBuilder::new().ints(8..16).finish()
+    );
+    assert_eq!(
+        TypeSetBuilder::new().floats(32..32).finish().half_width(),
+        empty_set
+    );
+    assert_eq!(
+        TypeSetBuilder::new().floats(32..64).finish().half_width(),
+        TypeSetBuilder::new().floats(32..32).finish()
+    );
+    assert_eq!(
+        TypeSetBuilder::new().bools(1..8).finish().half_width(),
+        empty_set
+    );
+    assert_eq!(
+        TypeSetBuilder::new().bools(1..32).finish().half_width(),
+        TypeSetBuilder::new().bools(8..16).finish()
+    );
+
+    // Double width.
+    assert_eq!(
+        TypeSetBuilder::new().ints(8..32).finish().double_width(),
+        TypeSetBuilder::new().ints(16..64).finish()
+    );
+    assert_eq!(
+        TypeSetBuilder::new().ints(32..64).finish().double_width(),
+        TypeSetBuilder::new().ints(64..64).finish()
+    );
+    assert_eq!(
+        TypeSetBuilder::new().floats(32..32).finish().double_width(),
+        TypeSetBuilder::new().floats(64..64).finish()
+    );
+    assert_eq!(
+        TypeSetBuilder::new().floats(32..64).finish().double_width(),
+        TypeSetBuilder::new().floats(64..64).finish()
+    );
+    assert_eq!(
+        TypeSetBuilder::new().bools(1..16).finish().double_width(),
+        TypeSetBuilder::new().bools(16..32).finish()
+    );
+    assert_eq!(
+        TypeSetBuilder::new().bools(32..64).finish().double_width(),
+        TypeSetBuilder::new().bools(64..64).finish()
+    );
+}
+
 #[test]
 fn test_singleton() {
     use crate::cdsl::types::VectorType;

From dd63ebfb68e1a4da2ec5c016eb3c8e773d1fd8c6 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 29 Mar 2019 15:51:14 +0100
Subject: [PATCH 2368/3084] [meta] Port typeset_singleton tests to the Rust
 crate;

---
 cranelift/codegen/meta/src/cdsl/types.rs   | 10 ++---
 cranelift/codegen/meta/src/cdsl/typevar.rs | 50 +++++++++++++++++++++-
 2 files changed, 54 insertions(+), 6 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
index 643ecdc4b6..a8aa4020d8 100644
--- a/cranelift/codegen/meta/src/cdsl/types.rs
+++ b/cranelift/codegen/meta/src/cdsl/types.rs
@@ -21,13 +21,13 @@ static LANE_BASE: u8 = 0x70;
 // Rust name prefix used for the `rust_name` method.
 static _RUST_NAME_PREFIX: &'static str = "ir::types::";
 
-// ValueType variants (i8, i32, ...) are provided in `base::types.rs`.
+// ValueType variants (i8, i32, ...) are provided in `shared::types.rs`.
 
 /// A concrete SSA value type.
 ///
 /// All SSA values have a type that is described by an instance of `ValueType`
 /// or one of its subclasses.
-#[derive(Clone, Debug)]
+#[derive(Clone, Debug, PartialEq)]
 pub enum ValueType {
     BV(BVType),
     Lane(LaneType),
@@ -147,7 +147,7 @@ impl From for ValueType {
 }
 
 /// A concrete scalar type that can appear as a vector lane too.
-#[derive(Clone, Copy)]
+#[derive(Clone, Copy, PartialEq)]
 pub enum LaneType {
     BoolType(shared_types::Bool),
     FloatType(shared_types::Float),
@@ -327,7 +327,7 @@ impl Iterator for LaneTypeIterator {
 ///
 /// A vector type has a lane type which is an instance of `LaneType`,
 /// and a positive number of lanes.
-#[derive(Clone)]
+#[derive(Clone, PartialEq)]
 pub struct VectorType {
     base: LaneType,
     lanes: u64,
@@ -393,7 +393,7 @@ impl fmt::Debug for VectorType {
 }
 
 /// A flat bitvector type. Used for semantics description only.
-#[derive(Clone)]
+#[derive(Clone, PartialEq)]
 pub struct BVType {
     bits: u64,
 }
diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 54e969b3cf..cb6f18ce56 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -795,7 +795,55 @@ fn test_forward_images() {
 }
 
 #[test]
-fn test_singleton() {
+#[should_panic]
+fn test_typeset_singleton_panic_nonsingleton_types() {
+    TypeSetBuilder::new()
+        .ints(8..8)
+        .floats(32..32)
+        .finish()
+        .get_singleton();
+}
+
+#[test]
+#[should_panic]
+fn test_typeset_singleton_panic_nonsingleton_lanes() {
+    TypeSetBuilder::new()
+        .simd_lanes(1..2)
+        .floats(32..32)
+        .finish()
+        .get_singleton();
+}
+
+#[test]
+fn test_typeset_singleton() {
+    use crate::shared::types as shared_types;
+    assert_eq!(
+        TypeSetBuilder::new().ints(16..16).finish().get_singleton(),
+        ValueType::Lane(shared_types::Int::I16.into())
+    );
+    assert_eq!(
+        TypeSetBuilder::new()
+            .floats(64..64)
+            .finish()
+            .get_singleton(),
+        ValueType::Lane(shared_types::Float::F64.into())
+    );
+    assert_eq!(
+        TypeSetBuilder::new().bools(1..1).finish().get_singleton(),
+        ValueType::Lane(shared_types::Bool::B1.into())
+    );
+    assert_eq!(
+        TypeSetBuilder::new()
+            .simd_lanes(4..4)
+            .ints(32..32)
+            .finish()
+            .get_singleton(),
+        LaneType::from(shared_types::Int::I32).by(4)
+    );
+}
+
+#[test]
+fn test_typevar_singleton() {
     use crate::cdsl::types::VectorType;
     use crate::shared::types as shared_types;
 

From 5edee84f678571f067cdf475b4c6fcbcb83c421f Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 29 Mar 2019 16:21:30 +0100
Subject: [PATCH 2369/3084] [meta] Port TypeVar's test_functions to Rust;

Fixes two issues:

- name of derived type variables wasn't correct. (This is used during
code generation.)
- the TypeVar::derived() function wasn't creating the correct type set
(and would instead propagate the parent's one).
---
 cranelift/codegen/meta/src/cdsl/typevar.rs | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index cb6f18ce56..983bb17a16 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -117,7 +117,7 @@ impl TypeVar {
 
     /// Create a type variable that is a function of another.
     fn derived(&self, derived_func: DerivedFunc) -> TypeVar {
-        let ts = self.content.type_set.clone();
+        let ts = self.get_typeset();
 
         // Safety checks to avoid over/underflows.
         assert!(ts.specials.len() == 0, "can't derive from special types");
@@ -169,7 +169,7 @@ impl TypeVar {
 
         return TypeVar {
             content: Rc::new(TypeVarContent {
-                name: "".into(), // XXX Python passes None to these two fields
+                name: format!("{}({})", derived_func.name(), self.name),
                 doc: "".into(),
                 type_set: ts,
                 base: Some(TypeVarParent {
@@ -842,6 +842,23 @@ fn test_typeset_singleton() {
     );
 }
 
+#[test]
+fn test_typevar_functions() {
+    let x = TypeVar::new(
+        "x",
+        "i16 and up",
+        TypeSetBuilder::new().ints(16..64).finish(),
+    );
+    assert_eq!(x.half_width().name, "half_width(x)");
+    assert_eq!(
+        x.half_width().double_width().name,
+        "double_width(half_width(x))"
+    );
+
+    let x = TypeVar::new("x", "up to i32", TypeSetBuilder::new().ints(8..32).finish());
+    assert_eq!(x.double_width().name, "double_width(x)");
+}
+
 #[test]
 fn test_typevar_singleton() {
     use crate::cdsl::types::VectorType;

From 2dd155236999e277d75fc515b3c6e1cefb0c2ed4 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 29 Mar 2019 17:00:02 +0100
Subject: [PATCH 2370/3084] [meta] Make Typevar::type_set private, and add a
 second getter to read it directly;

There might be a silent bug in the Python module which directly reads
from this type_set field; in particular, it does so when reading all
controlling type variables, which all seem to be free variables (i.e. no
parent typevar). So imitate this behavior here, until we're sure there
are no other meta generators that rely on this.
---
 cranelift/codegen/meta/src/cdsl/typevar.rs | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 983bb17a16..d144eab48f 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -25,7 +25,9 @@ pub struct TypeVarContent {
     pub doc: String,
 
     /// Type set associated to the type variable.
-    pub type_set: Rc,
+    /// This field must remain private; use `get_typeset()` or `get_raw_typeset()` to get the
+    /// information you want.
+    type_set: Rc,
 
     pub base: Option,
 }
@@ -84,13 +86,22 @@ impl TypeVar {
         TypeVar::new(name, doc, builder.finish())
     }
 
+    /// Returns this typevar's type set, maybe computing it from the parent.
     fn get_typeset(&self) -> Rc {
+        // TODO Can this be done in a non-lazy way in derived() and we can remove this function and
+        // the one below?
         match &self.content.base {
             Some(base) => Rc::new(base.type_var.get_typeset().image(base.derived_func)),
             None => self.content.type_set.clone(),
         }
     }
 
+    /// Returns this typevar's type set, assuming this type var has no parent.
+    pub fn get_raw_typeset(&self) -> &TypeSet {
+        assert_eq!(self.content.type_set, self.get_typeset());
+        &*self.content.type_set
+    }
+
     /// If the associated typeset has a single type return it. Otherwise return None.
     pub fn singleton_type(&self) -> Option {
         let type_set = self.get_typeset();

From cd8a42e01f1c869190f725ff880fd96364772318 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 12 Mar 2019 12:06:11 +0100
Subject: [PATCH 2371/3084] Fix #686: Allow code shrink in relaxation if the
 shrinking pass hasn't been run;

Also:
- make sure to apply diversions when determining offsets for code
relaxation.
- select the smallest encoding when selecting a relaxed branch
instruction.
---
 cranelift/codegen/src/binemit/relaxation.rs   | 11 ++--
 .../regress/allow-relaxation-shrink.clif      | 53 +++++++++++++++++++
 2 files changed, 58 insertions(+), 6 deletions(-)
 create mode 100644 cranelift/filetests/filetests/regress/allow-relaxation-shrink.clif

diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index 5bfa323646..c354a763cd 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -62,6 +62,7 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> CodegenResult CodegenResult i64 system_v {
+    ss1 = explicit_slot 8
+    sig0 = (i64) -> i64 system_v
+    fn0 = u0:8 sig0
+
+ebb0(v0: i64, v1: i64):
+    v3 = stack_addr.i64 ss1
+    v5 = call fn0(v1)
+    v6 = iconst.i64 0
+    v8 = iconst.i64 0
+    jump ebb3(v6, v1, v8)
+
+ebb3(v39: i64, v40: i64, v42: i64):
+    v9 = load.i64 v3
+    v11 = icmp_imm ugt v9, 1
+    v12 = bint.i8 v11
+    v13 = uextend.i32 v12
+    v14 = icmp_imm eq v13, 0
+    brnz v14, ebb4
+    jump ebb5
+
+ebb4:
+    v18 = icmp_imm.i64 eq v40, 0
+    v19 = bint.i8 v18
+    v20 = uextend.i32 v19
+    brz v20, ebb6
+    trap user0
+
+ebb5:
+    v22 = iconst.i32 1
+    v23 = ishl.i64 v39, v22
+    v25 = iconst.i64 1
+    v26 = band.i64 v42, v25
+    v27 = bor v23, v26
+    v28 = iconst.i32 1
+    v29 = ushr.i64 v42, v28
+    v30 = load.i64 v3
+    v31 = iconst.i32 1
+    v32 = ushr v30, v31
+    store v32, v3
+    jump ebb3(v27, v40, v29)
+
+ebb6:
+    v38 = iconst.i64 0
+    return v38
+}
+

From e6249b541d380c43ace1db27bbf3877cb81f17b2 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 27 Mar 2019 11:31:57 +0100
Subject: [PATCH 2372/3084] Update target-lexicon to 0.4.0;

---
 cranelift/Cargo.toml           | 2 +-
 cranelift/codegen/Cargo.toml   | 2 +-
 cranelift/faerie/Cargo.toml    | 4 ++--
 cranelift/frontend/Cargo.toml  | 2 +-
 cranelift/fuzz/Cargo.toml      | 2 +-
 cranelift/native/Cargo.toml    | 2 +-
 cranelift/reader/Cargo.toml    | 2 +-
 cranelift/simplejit/Cargo.toml | 2 +-
 cranelift/wasm/Cargo.toml      | 2 +-
 9 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 068326a646..5737fa1fa0 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -38,7 +38,7 @@ serde = "1.0.8"
 term = "0.5.1"
 capstone = { version = "0.5.0", optional = true }
 wabt = { version = "0.7.0", optional = true }
-target-lexicon = "0.3.0"
+target-lexicon = "0.4.0"
 pretty_env_logger = "0.3.0"
 file-per-thread-logger = "0.1.2"
 
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index da9b5e5d8f..7a2f2505c8 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -18,7 +18,7 @@ cranelift-bforest = { path = "../cranelift-bforest", version = "0.30.0", default
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
-target-lexicon = { version = "0.3.0", default-features = false }
+target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 0abcdb8de1..71a5179881 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -12,10 +12,10 @@ edition = "2018"
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0" }
 cranelift-module = { path = "../cranelift-module", version = "0.30.0" }
-faerie = "0.9.1"
+faerie = "0.10.0"
 goblin = "0.0.21"
 failure = "0.1.2"
-target-lexicon = "0.3.0"
+target-lexicon = "0.4.0"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 7d08cff58a..b46737cc7c 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -12,7 +12,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
-target-lexicon = { version = "0.3.0", default-features = false }
+target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 
diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
index 0a2acc410f..fb46b96311 100644
--- a/cranelift/fuzz/Cargo.toml
+++ b/cranelift/fuzz/Cargo.toml
@@ -15,7 +15,7 @@ libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" }
 cranelift-codegen = { path = "../cranelift-codegen" }
 cranelift-wasm = { path = "../cranelift-wasm" }
 cranelift-reader = { path = "../cranelift-reader" }
-target-lexicon = "0.3.0"
+target-lexicon = "0.4.0"
 
 # Prevent this from interfering with workspaces
 [workspace]
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index ee692f289e..5c07355bd6 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -11,7 +11,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
-target-lexicon = { version = "0.3.0", default-features = false }
+target-lexicon = { version = "0.4.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
 raw-cpuid = "6.0.0"
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index c1602171b4..d205de95c2 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -11,7 +11,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0" }
-target-lexicon = "0.3.0"
+target-lexicon = "0.4.0"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 646c96ffa9..7e791d3504 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -16,7 +16,7 @@ cranelift-native = { path = "../cranelift-native", version = "0.30.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
-target-lexicon = { version = "0.3.0" }
+target-lexicon = { version = "0.4.0" }
 
 [target.'cfg(target_os = "windows")'.dependencies]
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index c39368eb2c..2309322acb 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -23,7 +23,7 @@ cast = { version = "0.2.2", default-features = false }
 
 [dev-dependencies]
 wabt = "0.7.0"
-target-lexicon = "0.3.0"
+target-lexicon = "0.4.0"
 
 [features]
 default = ["std"]

From 84fede890ccb4763665d0b915e1857dc4eacbe42 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 5 Apr 2019 13:12:49 +0200
Subject: [PATCH 2373/3084] [meta] Optimize the sequence table to use whole
 table suffix prefixing;

---
 cranelift/codegen/meta/src/unique_table.rs | 58 ++++++++++++++++------
 1 file changed, 44 insertions(+), 14 deletions(-)

diff --git a/cranelift/codegen/meta/src/unique_table.rs b/cranelift/codegen/meta/src/unique_table.rs
index d4f4fa00cf..d7a26f67cc 100644
--- a/cranelift/codegen/meta/src/unique_table.rs
+++ b/cranelift/codegen/meta/src/unique_table.rs
@@ -52,9 +52,25 @@ impl UniqueSeqTable {
         if let Some(offset) = find_subsequence(values, &self.table) {
             offset
         } else {
-            let offset = self.table.len();
-            self.table.extend((*values).clone());
-            offset
+            let table_len = self.table.len();
+
+            // Try to put in common the last elements of the table if they're a prefix of the new
+            // sequence.
+            //
+            // We know there wasn't a full match, so the best prefix we can hope to find contains
+            // all the values but the last one.
+            let mut start_from = usize::min(table_len, values.len() - 1);
+            while start_from != 0 {
+                // Loop invariant: start_from <= table_len, so table_len - start_from >= 0.
+                if values[0..start_from] == self.table[table_len - start_from..table_len] {
+                    break;
+                }
+                start_from -= 1;
+            }
+
+            self.table
+                .extend(values[start_from..values.len()].iter().cloned());
+            table_len - start_from
         }
     }
     pub fn len(&self) -> usize {
@@ -74,17 +90,10 @@ fn find_subsequence(sub: &Vec, whole: &Vec) -> Option
     if whole.len() < sub.len() {
         return None;
     }
-    let max = whole.len() + 1 - sub.len();
-    for i in 0..max {
-        let mut found: Option = Some(i);
-        for j in 0..sub.len() {
-            if sub[j] != whole[i + j] {
-                found = None;
-                break;
-            }
-        }
-        if found.is_some() {
-            return found;
+    let max = whole.len() - sub.len();
+    for i in 0..max + 1 {
+        if whole[i..i + sub.len()] == sub[..] {
+            return Some(i);
         }
     }
     return None;
@@ -105,3 +114,24 @@ fn test_find_subsequence() {
         Some(1)
     );
 }
+
+#[test]
+fn test_optimal_add() {
+    let mut seq_table = UniqueSeqTable::new();
+    // [0, 1, 2, 3]
+    assert_eq!(seq_table.add(&vec![0, 1, 2, 3]), 0);
+    assert_eq!(seq_table.add(&vec![0, 1, 2, 3]), 0);
+    assert_eq!(seq_table.add(&vec![1, 2, 3]), 1);
+    assert_eq!(seq_table.add(&vec![2, 3]), 2);
+    assert_eq!(seq_table.len(), 4);
+    // [0, 1, 2, 3, 4]
+    assert_eq!(seq_table.add(&vec![2, 3, 4]), 2);
+    assert_eq!(seq_table.len(), 5);
+    // [0, 1, 2, 3, 4, 6, 5, 7]
+    assert_eq!(seq_table.add(&vec![4, 6, 5, 7]), 4);
+    assert_eq!(seq_table.len(), 8);
+    // [0, 1, 2, 3, 4, 6, 5, 7, 8, 2, 3, 4]
+    assert_eq!(seq_table.add(&vec![8, 2, 3, 4]), 8);
+    assert_eq!(seq_table.add(&vec![8]), 8);
+    assert_eq!(seq_table.len(), 12);
+}

From b74b49f7c6b9355f108754d77b1b32e78a28962d Mon Sep 17 00:00:00 2001
From: Philip Craig 
Date: Fri, 5 Apr 2019 20:40:22 +1000
Subject: [PATCH 2374/3084] Fix x86-64 encoding of uextend.i64.i8

The non-REX encoding of movzbl requires one of the ABCD registers as input.
---
 cranelift/codegen/meta-python/isa/x86/encodings.py |  2 +-
 cranelift/filetests/filetests/isa/x86/shrink.clif  | 12 ++++++++++++
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta-python/isa/x86/encodings.py b/cranelift/codegen/meta-python/isa/x86/encodings.py
index 4ae37c48a2..051b312167 100644
--- a/cranelift/codegen/meta-python/isa/x86/encodings.py
+++ b/cranelift/codegen/meta-python/isa/x86/encodings.py
@@ -631,7 +631,7 @@ X86_64.enc(base.uextend.i32.i16, *r.urm_noflags(0x0f, 0xb7))
 
 # movzbq, encoded as movzbl because it's equivalent and shorter
 X86_64.enc(base.uextend.i64.i8, *r.urm_noflags.rex(0x0f, 0xb6))
-X86_64.enc(base.uextend.i64.i8, *r.urm_noflags(0x0f, 0xb6))
+X86_64.enc(base.uextend.i64.i8, *r.urm_noflags_abcd(0x0f, 0xb6))
 
 # movzwq, encoded as movzwl because it's equivalent and shorter
 X86_64.enc(base.uextend.i64.i16, *r.urm_noflags.rex(0x0f, 0xb7))
diff --git a/cranelift/filetests/filetests/isa/x86/shrink.clif b/cranelift/filetests/filetests/isa/x86/shrink.clif
index 1d1dbfde31..f0d78af226 100644
--- a/cranelift/filetests/filetests/isa/x86/shrink.clif
+++ b/cranelift/filetests/filetests/isa/x86/shrink.clif
@@ -26,3 +26,15 @@ ebb0(v0: i32 [ %r8 ]):
 [-,%r8]    v2 = isub v0, v1      ; bin: 41 29 c8
            return v2
 }
+
+function %test_not_shrinking_i8() {
+ebb0:
+[-,%rsi]   v1 = iconst.i8 1
+           ; asm: movsbl %sil,%esi
+[-,%rsi]   v2 = sextend.i32 v1   ; bin: 40 0f be f6
+           ; asm: movzbl %sil,%esi
+[-,%rsi]   v3 = uextend.i32 v1   ; bin: 40 0f b6 f6
+           ; asm: movzbl %sil,%esi
+[-,%rsi]   v4 = uextend.i64 v1   ; bin: 40 0f b6 f6
+           trap user0
+}

From 72bc035d700f17cce26b6f61086ce362bd7e1791 Mon Sep 17 00:00:00 2001
From: carolinecullen 
Date: Mon, 11 Mar 2019 11:50:24 -0700
Subject: [PATCH 2375/3084] Beginnings of arm32 backend.

---
 cranelift/codegen/src/isa/arm32/abi.rs | 111 +++++++++++++++++++++++--
 cranelift/codegen/src/isa/arm32/mod.rs |   2 +-
 2 files changed, 103 insertions(+), 10 deletions(-)

diff --git a/cranelift/codegen/src/isa/arm32/abi.rs b/cranelift/codegen/src/isa/arm32/abi.rs
index a1890fa329..ec7f1270d9 100644
--- a/cranelift/codegen/src/isa/arm32/abi.rs
+++ b/cranelift/codegen/src/isa/arm32/abi.rs
@@ -1,18 +1,103 @@
 //! ARM ABI implementation.
 
 use super::registers::{D, GPR, Q, S};
-use crate::ir;
+use crate::abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion};
+use crate::ir::{self, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, Type};
 use crate::isa::RegClass;
 use crate::regalloc::RegisterSet;
-use crate::settings as shared_settings;
+use core::i32;
+use target_lexicon::Triple;
+
+struct Args {
+    pointer_bits: u8,
+    pointer_bytes: u8,
+    pointer_type: Type,
+    regs: u32,
+    reg_limit: u32,
+    offset: u32,
+}
+
+impl Args {
+    fn new(bits: u8) -> Self {
+        Self {
+            pointer_bits: bits,
+            pointer_bytes: bits / 8,
+            pointer_type: Type::int(u16::from(bits)).unwrap(),
+            regs: 0,
+            reg_limit: 8,
+            offset: 0,
+        }
+    }
+}
+
+impl ArgAssigner for Args {
+    fn assign(&mut self, arg: &AbiParam) -> ArgAction {
+        fn align(value: u32, to: u32) -> u32 {
+            (value + to - 1) & !(to - 1)
+        }
+
+        let ty = arg.value_type;
+
+        // Check for a legal type.
+        // RISC-V doesn't have SIMD at all, so break all vectors down.
+        if ty.is_vector() {
+            return ValueConversion::VectorSplit.into();
+        }
+
+        // Large integers and booleans are broken down to fit in a register.
+        if !ty.is_float() && ty.bits() > u16::from(self.pointer_bits) {
+            // Align registers and stack to a multiple of two pointers.
+            self.regs = align(self.regs, 2);
+            self.offset = align(self.offset, 2 * u32::from(self.pointer_bytes));
+            return ValueConversion::IntSplit.into();
+        }
+
+        // Small integers are extended to the size of a pointer register.
+        if ty.is_int() && ty.bits() < u16::from(self.pointer_bits) {
+            match arg.extension {
+                ArgumentExtension::None => {}
+                ArgumentExtension::Uext => return ValueConversion::Uext(self.pointer_type).into(),
+                ArgumentExtension::Sext => return ValueConversion::Sext(self.pointer_type).into(),
+            }
+        }
+
+        if self.regs < self.reg_limit {
+            // Assign to a register.
+            let reg = GPR.unit(10 + self.regs as usize);
+            self.regs += 1;
+            ArgumentLoc::Reg(reg).into()
+        } else {
+            // Assign a stack location.
+            let loc = ArgumentLoc::Stack(self.offset as i32);
+            self.offset += u32::from(self.pointer_bytes);
+            debug_assert!(self.offset <= i32::MAX as u32);
+            loc.into()
+        }
+    }
+}
 
 /// Legalize `sig`.
-pub fn legalize_signature(
-    _sig: &mut ir::Signature,
-    _flags: &shared_settings::Flags,
-    _current: bool,
-) {
-    unimplemented!()
+pub fn legalize_signature(sig: &mut ir::Signature, triple: &Triple, current: bool) {
+    let bits = triple.pointer_width().unwrap().bits();
+
+    let mut args = Args::new(bits);
+    legalize_args(&mut sig.params, &mut args);
+
+    let mut rets = Args::new(bits);
+    legalize_args(&mut sig.returns, &mut rets);
+
+    if current {
+        let ptr = Type::int(u16::from(bits)).unwrap();
+
+        // Add the link register as an argument and return value.
+        //
+        // The `jalr` instruction implementing a return can technically accept the return address
+        // in any register, but a micro-architecture with a return address predictor will only
+        // recognize it as a return if the address is in `x1`.
+        let link = AbiParam::special_reg(ptr, ArgumentPurpose::Link, GPR.unit(1));
+        sig.params.push(link);
+        sig.returns.push(link);
+    }
 }
 
 /// Get register class for a type appearing in a legalized signature.
@@ -31,5 +116,13 @@ pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass {
 
 /// Get the set of allocatable registers for `func`.
 pub fn allocatable_registers(_func: &ir::Function) -> RegisterSet {
-    unimplemented!()
+    let mut regs = RegisterSet::new();
+    regs.take(GPR, GPR.unit(0)); // Hard-wired 0.
+                                 // %x1 is the link register which is available for allocation.
+    regs.take(GPR, GPR.unit(2)); // Stack pointer.
+    regs.take(GPR, GPR.unit(3)); // Global pointer.
+    regs.take(GPR, GPR.unit(4)); // Thread pointer.
+                                 // TODO: %x8 is the frame pointer. Reserve it?
+
+    regs
 }
diff --git a/cranelift/codegen/src/isa/arm32/mod.rs b/cranelift/codegen/src/isa/arm32/mod.rs
index 7ceacf036b..7bede88f3c 100644
--- a/cranelift/codegen/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/src/isa/arm32/mod.rs
@@ -102,7 +102,7 @@ impl TargetIsa for Isa {
     }
 
     fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) {
-        abi::legalize_signature(sig, &self.shared_flags, current)
+        abi::legalize_signature(sig, &self.triple, current)
     }
 
     fn regclass_for_abi_type(&self, ty: ir::Type) -> RegClass {

From 8ab7170a0759bfda0740920221bde346095a28c4 Mon Sep 17 00:00:00 2001
From: carolinecullen 
Date: Tue, 12 Mar 2019 08:21:35 -0700
Subject: [PATCH 2376/3084] Updated comments.

---
 cranelift/codegen/src/isa/arm32/abi.rs | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)

diff --git a/cranelift/codegen/src/isa/arm32/abi.rs b/cranelift/codegen/src/isa/arm32/abi.rs
index ec7f1270d9..79323fdbd3 100644
--- a/cranelift/codegen/src/isa/arm32/abi.rs
+++ b/cranelift/codegen/src/isa/arm32/abi.rs
@@ -39,7 +39,7 @@ impl ArgAssigner for Args {
         let ty = arg.value_type;
 
         // Check for a legal type.
-        // RISC-V doesn't have SIMD at all, so break all vectors down.
+        // SIMD instructions are currently no implemented, so break down vectors
         if ty.is_vector() {
             return ValueConversion::VectorSplit.into();
         }
@@ -116,13 +116,5 @@ pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass {
 
 /// Get the set of allocatable registers for `func`.
 pub fn allocatable_registers(_func: &ir::Function) -> RegisterSet {
-    let mut regs = RegisterSet::new();
-    regs.take(GPR, GPR.unit(0)); // Hard-wired 0.
-                                 // %x1 is the link register which is available for allocation.
-    regs.take(GPR, GPR.unit(2)); // Stack pointer.
-    regs.take(GPR, GPR.unit(3)); // Global pointer.
-    regs.take(GPR, GPR.unit(4)); // Thread pointer.
-                                 // TODO: %x8 is the frame pointer. Reserve it?
-
-    regs
+    unimplemented!()
 }

From 0166d6507a678805bfc184510158a68fc0bf137d Mon Sep 17 00:00:00 2001
From: carolinecullen 
Date: Wed, 27 Mar 2019 15:27:20 -0500
Subject: [PATCH 2377/3084] Adding comment about copying RiscV abi file into
 ARM32.

---
 cranelift/codegen/src/isa/arm32/abi.rs | 21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)

diff --git a/cranelift/codegen/src/isa/arm32/abi.rs b/cranelift/codegen/src/isa/arm32/abi.rs
index 79323fdbd3..894c67ecb5 100644
--- a/cranelift/codegen/src/isa/arm32/abi.rs
+++ b/cranelift/codegen/src/isa/arm32/abi.rs
@@ -1,8 +1,9 @@
 //! ARM ABI implementation.
+//! This is from the RISC-V target and will need to be updated for ARM32.
 
 use super::registers::{D, GPR, Q, S};
 use crate::abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion};
-use crate::ir::{self, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, Type};
+use crate::ir::{self, AbiParam, ArgumentExtension, ArgumentLoc, Type};
 use crate::isa::RegClass;
 use crate::regalloc::RegisterSet;
 use core::i32;
@@ -77,27 +78,11 @@ impl ArgAssigner for Args {
 }
 
 /// Legalize `sig`.
-pub fn legalize_signature(sig: &mut ir::Signature, triple: &Triple, current: bool) {
+pub fn legalize_signature(sig: &mut ir::Signature, triple: &Triple, _current: bool) {
     let bits = triple.pointer_width().unwrap().bits();
 
     let mut args = Args::new(bits);
     legalize_args(&mut sig.params, &mut args);
-
-    let mut rets = Args::new(bits);
-    legalize_args(&mut sig.returns, &mut rets);
-
-    if current {
-        let ptr = Type::int(u16::from(bits)).unwrap();
-
-        // Add the link register as an argument and return value.
-        //
-        // The `jalr` instruction implementing a return can technically accept the return address
-        // in any register, but a micro-architecture with a return address predictor will only
-        // recognize it as a return if the address is in `x1`.
-        let link = AbiParam::special_reg(ptr, ArgumentPurpose::Link, GPR.unit(1));
-        sig.params.push(link);
-        sig.returns.push(link);
-    }
 }
 
 /// Get register class for a type appearing in a legalized signature.

From 9062810c346f8d01e0c33759290fdf525471941a Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 8 Apr 2019 16:13:19 +0200
Subject: [PATCH 2378/3084] Fix the publish script to include the remote when
 pushing the tag;

---
 cranelift/publish-all.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 7792f1e9f8..175ccfcb0e 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -36,7 +36,7 @@ cargo update
 echo git commit -a -m "\"Bump version to $version"\"
 echo git tag v$version
 echo git push
-echo git push v$version
+echo git push origin v$version
 for crate in \
     entity bforest codegen/meta codegen frontend native \
     preopt \

From aa926e90979fb38f5323867ba39753d5061a14f7 Mon Sep 17 00:00:00 2001
From: Lars T Hansen 
Date: Tue, 9 Apr 2019 11:40:23 +0200
Subject: [PATCH 2379/3084] Allow readonly nontrapping loads to be hoisted by
 licm (#727)

---
 cranelift/codegen/src/licm.rs                 | 20 ++++++--
 .../filetests/licm/load_readonly_notrap.clif  | 48 ++++++++++++++++++
 .../filetests/licm/reject_load_notrap.clif    | 49 +++++++++++++++++++
 .../filetests/licm/reject_load_readonly.clif  | 49 +++++++++++++++++++
 4 files changed, 163 insertions(+), 3 deletions(-)
 create mode 100644 cranelift/filetests/filetests/licm/load_readonly_notrap.clif
 create mode 100644 cranelift/filetests/filetests/licm/reject_load_notrap.clif
 create mode 100644 cranelift/filetests/filetests/licm/reject_load_readonly.clif

diff --git a/cranelift/codegen/src/licm.rs b/cranelift/codegen/src/licm.rs
index cb3dbd87a9..14c4630dc7 100644
--- a/cranelift/codegen/src/licm.rs
+++ b/cranelift/codegen/src/licm.rs
@@ -5,7 +5,9 @@ use crate::dominator_tree::DominatorTree;
 use crate::entity::{EntityList, ListPool};
 use crate::flowgraph::{BasicBlock, ControlFlowGraph};
 use crate::fx::FxHashSet;
-use crate::ir::{DataFlowGraph, Ebb, Function, Inst, InstBuilder, Layout, Opcode, Type, Value};
+use crate::ir::{
+    DataFlowGraph, Ebb, Function, Inst, InstBuilder, InstructionData, Layout, Opcode, Type, Value,
+};
 use crate::isa::TargetIsa;
 use crate::loop_analysis::{Loop, LoopAnalysis};
 use crate::timing;
@@ -145,8 +147,7 @@ fn change_branch_jump_destination(inst: Inst, new_ebb: Ebb, func: &mut Function)
 
 /// Test whether the given opcode is unsafe to even consider for LICM.
 fn trivially_unsafe_for_licm(opcode: Opcode) -> bool {
-    opcode.can_load()
-        || opcode.can_store()
+    opcode.can_store()
         || opcode.is_call()
         || opcode.is_branch()
         || opcode.is_terminator()
@@ -156,12 +157,25 @@ fn trivially_unsafe_for_licm(opcode: Opcode) -> bool {
         || opcode.writes_cpu_flags()
 }
 
+fn is_unsafe_load(inst_data: &InstructionData) -> bool {
+    match *inst_data {
+        InstructionData::Load { flags, .. } | InstructionData::LoadComplex { flags, .. } => {
+            !flags.readonly() || !flags.notrap()
+        }
+        _ => inst_data.opcode().can_load(),
+    }
+}
+
 /// Test whether the given instruction is loop-invariant.
 fn is_loop_invariant(inst: Inst, dfg: &DataFlowGraph, loop_values: &FxHashSet) -> bool {
     if trivially_unsafe_for_licm(dfg[inst].opcode()) {
         return false;
     }
 
+    if is_unsafe_load(&dfg[inst]) {
+        return false;
+    }
+
     let inst_args = dfg.inst_args(inst);
     for arg in inst_args {
         let arg = dfg.resolve_aliases(*arg);
diff --git a/cranelift/filetests/filetests/licm/load_readonly_notrap.clif b/cranelift/filetests/filetests/licm/load_readonly_notrap.clif
new file mode 100644
index 0000000000..9a9d2dcbfa
--- /dev/null
+++ b/cranelift/filetests/filetests/licm/load_readonly_notrap.clif
@@ -0,0 +1,48 @@
+test licm
+
+target x86_64
+
+;; Nontrapping readonly load from address that is not loop-dependent
+;; should be hoisted out of loop.
+
+function %hoist_load(i32, i64 vmctx) -> i32 {
+    gv0 = vmctx
+    gv1 = load.i64 notrap aligned readonly gv0
+    heap0 = static gv1, min 0x1_0000, bound 0x1_0000_0000, offset_guard 0x8000_0000, index_type i32
+
+ebb0(v0: i32, v1: i64):
+    jump ebb1(v0, v1)
+
+ebb1(v2: i32, v3: i64):
+    v4 = iconst.i32 1
+    v5 = heap_addr.i64 heap0, v4, 1
+    v6 = load.i32 notrap aligned readonly v5
+    v7 = iadd v2, v6
+    brz v2, ebb2(v2)
+    v8 = isub v2, v4
+    jump ebb1(v8, v3)
+
+ebb2(v9: i32):
+    return v9
+}
+
+; sameln: function %hoist_load(i32, i64 vmctx) -> i32 fast {
+; nextln:    gv0 = vmctx
+; nextln:    gv1 = load.i64 notrap aligned readonly gv0
+; nextln:    heap0 = static gv1, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000, index_type i32
+; nextln: 
+; nextln: ebb0(v0: i32, v1: i64):
+; nextln:    v4 = iconst.i32 1
+; nextln:    v5 = heap_addr.i64 heap0, v4, 1
+; nextln:    v6 = load.i32 notrap aligned readonly v5
+; nextln:    jump ebb1(v0, v1)
+; nextln: 
+; nextln: ebb1(v2: i32, v3: i64):
+; nextln:    v7 = iadd v2, v6
+; nextln:    brz v2, ebb2(v2)
+; nextln:    v8 = isub v2, v4
+; nextln:    jump ebb1(v8, v3)
+; nextln: 
+; nextln: ebb2(v9: i32):
+; nextln:    return v9
+; nextln: }
diff --git a/cranelift/filetests/filetests/licm/reject_load_notrap.clif b/cranelift/filetests/filetests/licm/reject_load_notrap.clif
new file mode 100644
index 0000000000..1d26faa71f
--- /dev/null
+++ b/cranelift/filetests/filetests/licm/reject_load_notrap.clif
@@ -0,0 +1,49 @@
+test licm
+
+target x86_64
+
+;; Nontrapping possibly-not-readonly load from address that is not
+;; loop-dependent should *not* be hoisted out of loop, though the
+;; address computation can be.
+
+function %hoist_load(i32, i64 vmctx) -> i32 {
+    gv0 = vmctx
+    gv1 = load.i64 notrap aligned readonly gv0
+    heap0 = static gv1, min 0x1_0000, bound 0x1_0000_0000, offset_guard 0x8000_0000, index_type i32
+
+ebb0(v0: i32, v1: i64):
+    v4 = iconst.i32 1
+    v5 = heap_addr.i64 heap0, v4, 1
+    jump ebb1(v0, v1)
+
+ebb1(v2: i32, v3: i64):
+    v6 = load.i32 notrap aligned v5
+    v7 = iadd v2, v6
+    brz v2, ebb2(v2)
+    v8 = isub v2, v4
+    jump ebb1(v8, v3)
+
+ebb2(v9: i32):
+    return v9
+}
+
+; sameln: function %hoist_load(i32, i64 vmctx) -> i32 fast {
+; nextln:    gv0 = vmctx
+; nextln:    gv1 = load.i64 notrap aligned readonly gv0
+; nextln:    heap0 = static gv1, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000, index_type i32
+; nextln: 
+; nextln: ebb0(v0: i32, v1: i64):
+; nextln:    v4 = iconst.i32 1
+; nextln:    v5 = heap_addr.i64 heap0, v4, 1
+; nextln:    jump ebb1(v0, v1)
+; nextln: 
+; nextln: ebb1(v2: i32, v3: i64):
+; nextln:    v6 = load.i32 notrap aligned v5
+; nextln:    v7 = iadd v2, v6
+; nextln:    brz v2, ebb2(v2)
+; nextln:    v8 = isub v2, v4
+; nextln:    jump ebb1(v8, v3)
+; nextln: 
+; nextln: ebb2(v9: i32):
+; nextln:    return v9
+; nextln: }
diff --git a/cranelift/filetests/filetests/licm/reject_load_readonly.clif b/cranelift/filetests/filetests/licm/reject_load_readonly.clif
new file mode 100644
index 0000000000..5b6a411712
--- /dev/null
+++ b/cranelift/filetests/filetests/licm/reject_load_readonly.clif
@@ -0,0 +1,49 @@
+test licm
+
+target x86_64
+
+;; Maybe-trapping readonly load from address that is not
+;; loop-dependent should *not* be hoisted out of loop, though the
+;; address computation can be hoisted.
+
+function %hoist_load(i32, i64 vmctx) -> i32 {
+    gv0 = vmctx
+    gv1 = load.i64 notrap aligned readonly gv0
+    heap0 = static gv1, min 0x1_0000, bound 0x1_0000_0000, offset_guard 0x8000_0000, index_type i32
+
+ebb0(v0: i32, v1: i64):
+    jump ebb1(v0, v1)
+
+ebb1(v2: i32, v3: i64):
+    v4 = iconst.i32 1
+    v5 = heap_addr.i64 heap0, v4, 1
+    v6 = load.i32 aligned readonly v5
+    v7 = iadd v2, v6
+    brz v2, ebb2(v2)
+    v8 = isub v2, v4
+    jump ebb1(v8, v3)
+
+ebb2(v9: i32):
+    return v9
+}
+
+; sameln: function %hoist_load(i32, i64 vmctx) -> i32 fast {
+; nextln:    gv0 = vmctx
+; nextln:    gv1 = load.i64 notrap aligned readonly gv0
+; nextln:    heap0 = static gv1, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000, index_type i32
+; nextln: 
+; nextln: ebb0(v0: i32, v1: i64):
+; nextln:    v4 = iconst.i32 1
+; nextln:    v5 = heap_addr.i64 heap0, v4, 1
+; nextln:    jump ebb1(v0, v1)
+; nextln: 
+; nextln: ebb1(v2: i32, v3: i64):
+; nextln:    v6 = load.i32 aligned readonly v5
+; nextln:    v7 = iadd v2, v6
+; nextln:    brz v2, ebb2(v2)
+; nextln:    v8 = isub v2, v4
+; nextln:    jump ebb1(v8, v3)
+; nextln: 
+; nextln: ebb2(v9: i32):
+; nextln:    return v9
+; nextln: }

From b5595aadd24f05f77fc3f26560cc282f851d4d5b Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Mar 2019 19:18:48 +0100
Subject: [PATCH 2380/3084] [meta] Generate opcodes.rs/inst_builder.rs with the
 Rust crate;

---
 cranelift/codegen/meta/src/gen_inst.rs | 1092 ++++++++++++++++++++++++
 cranelift/codegen/meta/src/lib.rs      |   12 +
 2 files changed, 1104 insertions(+)
 create mode 100644 cranelift/codegen/meta/src/gen_inst.rs

diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
new file mode 100644
index 0000000000..c975cef961
--- /dev/null
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -0,0 +1,1092 @@
+use crate::cdsl::camel_case;
+use crate::cdsl::formats::{FormatRegistry, InstructionFormat};
+use crate::cdsl::inst::{Instruction, InstructionGroup};
+use crate::cdsl::operands::Operand;
+use crate::cdsl::typevar::{TypeSet, TypeVar};
+use crate::constant_hash;
+use crate::error;
+use crate::srcgen::{Formatter, Match};
+use crate::unique_table::{UniqueSeqTable, UniqueTable};
+
+use std::fmt;
+
+// TypeSet indexes are encoded in 8 bits, with `0xff` reserved.
+const TYPESET_LIMIT: usize = 0xff;
+
+/// Generate an instruction format enumeration.
+fn gen_formats(registry: &FormatRegistry, fmt: &mut Formatter) {
+    fmt.doc_comment(
+        r#"
+        An instruction format
+
+        Every opcode has a corresponding instruction format
+        which is represented by both the `InstructionFormat`
+        and the `InstructionData` enums.
+    "#,
+    );
+    fmt.line("#[derive(Copy, Clone, PartialEq, Eq, Debug)]");
+    fmt.line("pub enum InstructionFormat {");
+    fmt.indent(|fmt| {
+        for format in registry.iter() {
+            fmt.doc_comment(format.to_string());
+            fmtln!(fmt, "{},", format.name);
+        }
+    });
+    fmt.line("}");
+    fmt.empty_line();
+
+    // Emit a From which also serves to verify that
+    // InstructionFormat and InstructionData are in sync.
+    fmt.line("impl<'a> From<&'a InstructionData> for InstructionFormat {");
+    fmt.indent(|fmt| {
+        fmt.line("fn from(inst: &'a InstructionData) -> Self {");
+        fmt.indent(|fmt| {
+            let mut m = Match::new("*inst");
+            for format in registry.iter() {
+                m.arm(
+                    format!("InstructionData::{}", format.name),
+                    vec![".."],
+                    format!("InstructionFormat::{}", format.name),
+                );
+            }
+            fmt.add_match(m);
+        });
+        fmt.line("}");
+    });
+    fmt.line("}");
+    fmt.empty_line();
+}
+
+/// Generate the InstructionData enum.
+///
+/// Every variant must contain an `opcode` field. The size of `InstructionData` should be kept at
+/// 16 bytes on 64-bit architectures. If more space is needed to represent an instruction, use a
+/// `ValueList` to store the additional information out of line.
+fn gen_instruction_data(registry: &FormatRegistry, fmt: &mut Formatter) {
+    fmt.line("#[derive(Clone, Debug)]");
+    fmt.line("#[allow(missing_docs)]");
+    fmt.line("pub enum InstructionData {");
+    fmt.indent(|fmt| {
+        for format in registry.iter() {
+            fmtln!(fmt, "{} {{", format.name);
+            fmt.indent(|fmt| {
+                fmt.line("opcode: Opcode,");
+                if format.typevar_operand.is_some() {
+                    if format.has_value_list {
+                        fmt.line("args: ValueList,");
+                    } else if format.num_value_operands == 1 {
+                        fmt.line("arg: Value,");
+                    } else {
+                        fmtln!(fmt, "args: [Value; {}],", format.num_value_operands);
+                    }
+                }
+                for field in &format.imm_fields {
+                    fmtln!(fmt, "{}: {},", field.member, field.kind.rust_type);
+                }
+            });
+            fmtln!(fmt, "},");
+        }
+    });
+    fmt.line("}");
+}
+
+fn gen_arguments_method(registry: &FormatRegistry, fmt: &mut Formatter, is_mut: bool) {
+    let (method, mut_, rslice, as_slice) = if is_mut {
+        ("arguments_mut", "mut ", "ref_slice_mut", "as_mut_slice")
+    } else {
+        ("arguments", "", "ref_slice", "as_slice")
+    };
+
+    fmtln!(
+        fmt,
+        "pub fn {}<'a>(&'a {}self, pool: &'a {}ir::ValueListPool) -> &{}[Value] {{",
+        method,
+        mut_,
+        mut_,
+        mut_
+    );
+    fmt.indent(|fmt| {
+        let mut m = Match::new("*self");
+        for format in registry.iter() {
+            let name = format!("InstructionData::{}", format.name);
+
+            // Formats with a value list put all of their arguments in the list. We don't split
+            // them up, just return it all as variable arguments. (I expect the distinction to go
+            // away).
+            if format.has_value_list {
+                m.arm(
+                    name,
+                    vec![format!("ref {}args", mut_), "..".to_string()],
+                    format!("args.{}(pool)", as_slice),
+                );
+                continue;
+            }
+
+            // Fixed args.
+            let mut fields = Vec::new();
+            let arg = if format.num_value_operands == 0 {
+                format!("&{}[]", mut_)
+            } else if format.num_value_operands == 1 {
+                fields.push(format!("ref {}arg", mut_));
+                format!("{}(arg)", rslice)
+            } else {
+                let arg = format!("args_arity{}", format.num_value_operands);
+                fields.push(format!("args: ref {}{}", mut_, arg));
+                arg
+            };
+            fields.push("..".into());
+
+            m.arm(name, fields, arg);
+        }
+        fmt.add_match(m);
+    });
+    fmtln!(fmt, "}");
+}
+
+/// Generate the boring parts of the InstructionData implementation.
+///
+/// These methods in `impl InstructionData` can be generated automatically from the instruction
+/// formats:
+///
+/// - `pub fn opcode(&self) -> Opcode`
+/// - `pub fn arguments(&self, &pool) -> &[Value]`
+/// - `pub fn arguments_mut(&mut self, &pool) -> &mut [Value]`
+/// - `pub fn take_value_list(&mut self) -> Option`
+/// - `pub fn put_value_list(&mut self, args: ir::ValueList>`
+/// - `pub fn eq(&self, &other: Self, &pool) -> bool`
+/// - `pub fn hash(&self, state: &mut H, &pool)`
+fn gen_instruction_data_impl(registry: &FormatRegistry, fmt: &mut Formatter) {
+    fmt.line("impl InstructionData {");
+    fmt.indent(|fmt| {
+        fmt.doc_comment("Get the opcode of this instruction.");
+        fmt.line("pub fn opcode(&self) -> Opcode {");
+        fmt.indent(|fmt| {
+            let mut m = Match::new("*self");
+            for format in registry.iter() {
+                m.arm(format!("InstructionData::{}", format.name), vec!["opcode", ".."],
+                      "opcode".to_string());
+            }
+            fmt.add_match(m);
+        });
+        fmt.line("}");
+        fmt.empty_line();
+
+        fmt.doc_comment("Get the controlling type variable operand.");
+        fmt.line("pub fn typevar_operand(&self, pool: &ir::ValueListPool) -> Option {");
+        fmt.indent(|fmt| {
+            let mut m = Match::new("*self");
+            for format in registry.iter() {
+                let name = format!("InstructionData::{}", format.name);
+                if format.typevar_operand.is_none() {
+                    m.arm(name, vec![".."], "None".to_string());
+                } else if format.has_value_list {
+                    // We keep all arguments in a value list.
+                    m.arm(name, vec!["ref args", ".."], format!("args.get({}, pool)", format.typevar_operand.unwrap()));
+                } else if format.num_value_operands == 1 {
+                    m.arm(name, vec!["arg", ".."], "Some(arg)".to_string());
+                } else {
+                    // We have multiple value operands and an array `args`.
+                    // Which `args` index to use?
+                    let args = format!("args_arity{}", format.num_value_operands);
+                    m.arm(name, vec![format!("args: ref {}", args), "..".to_string()],
+                        format!("Some({}[{}])", args, format.typevar_operand.unwrap()));
+                }
+            }
+            fmt.add_match(m);
+        });
+        fmt.line("}");
+        fmt.empty_line();
+
+        fmt.doc_comment("Get the value arguments to this instruction.");
+        gen_arguments_method(registry, fmt, false);
+        fmt.empty_line();
+
+        fmt.doc_comment(r#"Get mutable references to the value arguments to this
+                        instruction."#);
+        gen_arguments_method(registry, fmt, true);
+        fmt.empty_line();
+
+        fmt.doc_comment(r#"
+            Take out the value list with all the value arguments and return
+            it.
+
+            This leaves the value list in the instruction empty. Use
+            `put_value_list` to put the value list back.
+        "#);
+        fmt.line("pub fn take_value_list(&mut self) -> Option {");
+        fmt.indent(|fmt| {
+            let mut m = Match::new("*self");
+
+            for format in registry.iter() {
+                if format.has_value_list {
+                    m.arm(format!("InstructionData::{}", format.name),
+                    vec!["ref mut args", ".."],
+                    "Some(args.take())".to_string());
+                }
+            }
+
+            m.arm_no_fields("_", "None");
+
+            fmt.add_match(m);
+        });
+        fmt.line("}");
+        fmt.empty_line();
+
+        fmt.doc_comment(r#"
+            Put back a value list.
+
+            After removing a value list with `take_value_list()`, use this
+            method to put it back. It is required that this instruction has
+            a format that accepts a value list, and that the existing value
+            list is empty. This avoids leaking list pool memory.
+        "#);
+        fmt.line("pub fn put_value_list(&mut self, vlist: ir::ValueList) {");
+        fmt.indent(|fmt| {
+            fmt.line("let args = match *self {");
+            fmt.indent(|fmt| {
+                for format in registry.iter() {
+                    if format.has_value_list {
+                        fmtln!(fmt, "InstructionData::{} {{ ref mut args, .. }} => args,", format.name);
+                    }
+                }
+                fmt.line("_ => panic!(\"No value list: {:?}\", self),");
+            });
+            fmt.line("};");
+            fmt.line("debug_assert!(args.is_empty(), \"Value list already in use\");");
+            fmt.line("*args = vlist;");
+        });
+        fmt.line("}");
+        fmt.empty_line();
+
+        fmt.doc_comment(r#"
+            Compare two `InstructionData` for equality.
+
+            This operation requires a reference to a `ValueListPool` to
+            determine if the contents of any `ValueLists` are equal.
+        "#);
+        fmt.line("pub fn eq(&self, other: &Self, pool: &ir::ValueListPool) -> bool {");
+        fmt.indent(|fmt| {
+            fmt.line("if ::core::mem::discriminant(self) != ::core::mem::discriminant(other) {");
+            fmt.indent(|fmt| {
+                fmt.line("return false;");
+            });
+            fmt.line("}");
+
+            fmt.line("match (self, other) {");
+            fmt.indent(|fmt| {
+                for format in registry.iter() {
+                    let name = format!("&InstructionData::{}", format.name);
+                    let mut members = vec!["opcode"];
+
+                    let args_eq = if format.typevar_operand.is_none() {
+                        None
+                    } else if format.has_value_list {
+                        members.push("args");
+                        Some("args1.as_slice(pool) == args2.as_slice(pool)")
+                    } else if format.num_value_operands == 1 {
+                        members.push("arg");
+                        Some("arg1 == arg2")
+                    } else {
+                        members.push("args");
+                        Some("args1 == args2")
+                    };
+
+                    for field in &format.imm_fields {
+                        members.push(field.member);
+                    }
+
+                    let pat1 = members.iter().map(|x| format!("{}: ref {}1", x, x)).collect::>().join(", ");
+                    let pat2 = members.iter().map(|x| format!("{}: ref {}2", x, x)).collect::>().join(", ");
+                    fmtln!(fmt, "({} {{ {} }}, {} {{ {} }}) => {{", name, pat1, name, pat2);
+                    fmt.indent(|fmt| {
+                        fmt.line("opcode1 == opcode2");
+                        for field in &format.imm_fields {
+                            fmtln!(fmt, "&& {}1 == {}2", field.member, field.member);
+                        }
+                        if let Some(args_eq) = args_eq {
+                            fmtln!(fmt, "&& {}", args_eq);
+                        }
+                    });
+                    fmtln!(fmt, "}");
+                }
+                fmt.line("_ => unreachable!()");
+            });
+            fmt.line("}");
+        });
+        fmt.line("}");
+        fmt.empty_line();
+
+        fmt.doc_comment(r#"
+            Hash an `InstructionData`.
+
+            This operation requires a reference to a `ValueListPool` to
+            hash the contents of any `ValueLists`.
+        "#);
+        fmt.line("pub fn hash(&self, state: &mut H, pool: &ir::ValueListPool) {");
+        fmt.indent(|fmt| {
+            fmt.line("match *self {");
+            fmt.indent(|fmt| {
+                for format in registry.iter() {
+                    let name = format!("InstructionData::{}", format.name);
+                    let mut members = vec!["opcode"];
+
+                    let args = if format.typevar_operand.is_none() {
+                        "&()"
+                    } else if format.has_value_list {
+                        members.push("ref args");
+                        "args.as_slice(pool)"
+                    } else if format.num_value_operands == 1 {
+                        members.push("ref arg");
+                        "arg"
+                    } else {
+                        members.push("ref args");
+                        "args"
+                    };
+
+                    for field in &format.imm_fields {
+                        members.push(field.member);
+                    }
+                    let members = members.join(", ");
+
+                    fmtln!(fmt, "{}{{{}}} => {{", name, members ); // beware the moustaches
+                    fmt.indent(|fmt| {
+                        fmt.line("::core::hash::Hash::hash( &::core::mem::discriminant(self), state);");
+                        fmt.line("::core::hash::Hash::hash(&opcode, state);");
+                        for field in &format.imm_fields {
+                            fmtln!(fmt, "::core::hash::Hash::hash(&{}, state);", field.member);
+                        }
+                        fmtln!(fmt, "::core::hash::Hash::hash({}, state);", args);
+                    });
+                    fmtln!(fmt, "}");
+                }
+            });
+            fmt.line("}");
+        });
+        fmt.line("}");
+    });
+    fmt.line("}");
+}
+
+fn gen_bool_accessor bool>(
+    instruction_groups: &Vec<&InstructionGroup>,
+    get_attr: T,
+    name: &'static str,
+    doc: &'static str,
+    fmt: &mut Formatter,
+) {
+    fmt.doc_comment(doc);
+    fmtln!(fmt, "pub fn {}(self) -> bool {{", name);
+    fmt.indent(|fmt| {
+        let mut m = Match::new("self");
+        for group in instruction_groups.iter() {
+            for inst in group.iter() {
+                if get_attr(inst) {
+                    m.arm_no_fields(format!("Opcode::{}", inst.camel_name), "true");
+                }
+            }
+        }
+        m.arm_no_fields("_", "false");
+        fmt.add_match(m);
+    });
+    fmtln!(fmt, "}");
+    fmt.empty_line();
+}
+
+fn gen_opcodes<'a>(
+    formats: &FormatRegistry,
+    igroups: &Vec<&'a InstructionGroup>,
+    fmt: &mut Formatter,
+) -> Vec<&'a Instruction> {
+    let mut all_inst = Vec::new();
+    for group in igroups {
+        for inst in group.iter() {
+            all_inst.push(inst);
+        }
+    }
+
+    fmt.doc_comment(
+        r#"
+        An instruction opcode.
+
+        All instructions from all supported ISAs are present.
+    "#,
+    );
+    fmt.line("#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)]");
+
+    // We explicitly set the discriminant of the first variant to 1, which allows us to take
+    // advantage of the NonZero optimization, meaning that wrapping enums can use the 0
+    // discriminant instead of increasing the size of the whole type, and so the size of
+    // Option is the same as Opcode's.
+    fmt.line("pub enum Opcode {");
+    fmt.indent(|fmt| {
+        let mut is_first_opcode = true;
+        for inst in &all_inst {
+            // TODO we might need to set an instruction number here. Probably can do in the
+            // InstructionGroup itself when adding instruction (would need to remember last
+            // instruction number in the SharedDefinitions or somewhere else).
+            let format = formats.get(inst.format);
+            fmt.doc_comment(format!("`{}`. ({})", inst, format.name));
+
+            // Document polymorphism.
+            if let Some(poly) = &inst.polymorphic_info {
+                if poly.use_typevar_operand {
+                    let op_num = inst.value_opnums[format.typevar_operand.unwrap()];
+                    fmt.doc_comment(format!(
+                        "Type inferred from `{}`.",
+                        inst.operands_in[op_num].name
+                    ));
+                }
+            }
+
+            // Enum variant itself.
+            if is_first_opcode {
+                fmtln!(fmt, "{} = 1,", inst.camel_name);
+                is_first_opcode = false;
+            } else {
+                fmtln!(fmt, "{},", inst.camel_name)
+            }
+        }
+    });
+    fmt.line("}");
+    fmt.empty_line();
+
+    fmt.line("impl Opcode {");
+    fmt.indent(|fmt| {
+        gen_bool_accessor(
+            igroups,
+            |inst| inst.is_terminator,
+            "is_terminator",
+            "True for instructions that terminate the EBB",
+            fmt,
+        );
+        gen_bool_accessor(
+            igroups,
+            |inst| inst.is_branch,
+            "is_branch",
+            "True for all branch or jump instructions.",
+            fmt,
+        );
+        gen_bool_accessor(
+            igroups,
+            |inst| inst.is_indirect_branch,
+            "is_indirect_branch",
+            "True for all indirect branch or jump instructions.",
+            fmt,
+        );
+        gen_bool_accessor(
+            igroups,
+            |inst| inst.is_call,
+            "is_call",
+            "Is this a call instruction?",
+            fmt,
+        );
+        gen_bool_accessor(
+            igroups,
+            |inst| inst.is_return,
+            "is_return",
+            "Is this a return instruction?",
+            fmt,
+        );
+        gen_bool_accessor(
+            igroups,
+            |inst| inst.is_ghost,
+            "is_ghost",
+            "Is this a ghost instruction?",
+            fmt,
+        );
+        gen_bool_accessor(
+            igroups,
+            |inst| inst.can_load,
+            "can_load",
+            "Can this instruction read from memory?",
+            fmt,
+        );
+        gen_bool_accessor(
+            igroups,
+            |inst| inst.can_store,
+            "can_store",
+            "Can this instruction write to memory?",
+            fmt,
+        );
+        gen_bool_accessor(
+            igroups,
+            |inst| inst.can_trap,
+            "can_trap",
+            "Can this instruction cause a trap?",
+            fmt,
+        );
+        gen_bool_accessor(
+            igroups,
+            |inst| inst.other_side_effects,
+            "other_side_effects",
+            "Does this instruction have other side effects besides can_* flags?",
+            fmt,
+        );
+        gen_bool_accessor(
+            igroups,
+            |inst| inst.writes_cpu_flags,
+            "writes_cpu_flags",
+            "Does this instruction write to CPU flags?",
+            fmt,
+        );
+    });
+    fmt.line("}");
+    fmt.empty_line();
+
+    // Generate a private opcode_format table.
+    fmtln!(
+        fmt,
+        "const OPCODE_FORMAT: [InstructionFormat; {}] = [",
+        all_inst.len()
+    );
+    fmt.indent(|fmt| {
+        for inst in &all_inst {
+            let format = formats.get(inst.format);
+            fmtln!(fmt, "InstructionFormat::{}, // {}", format.name, inst.name);
+        }
+    });
+    fmtln!(fmt, "];");
+    fmt.empty_line();
+
+    // Generate a private opcode_name function.
+    fmt.line("fn opcode_name(opc: Opcode) -> &\'static str {");
+    fmt.indent(|fmt| {
+        let mut m = Match::new("opc");
+        for inst in &all_inst {
+            m.arm_no_fields(
+                format!("Opcode::{}", inst.camel_name),
+                format!("\"{}\"", inst.name),
+            );
+        }
+        fmt.add_match(m);
+    });
+    fmt.line("}");
+    fmt.empty_line();
+
+    // Generate an opcode hash table for looking up opcodes by name.
+    let hash_table =
+        constant_hash::generate_table(&all_inst, |inst| constant_hash::simple_hash(inst.name));
+    fmtln!(
+        fmt,
+        "const OPCODE_HASH_TABLE: [Option; {}] = [",
+        hash_table.len()
+    );
+    fmt.indent(|fmt| {
+        for i in hash_table {
+            match i {
+                Some(i) => fmtln!(fmt, "Some(Opcode::{}),", i.camel_name),
+                None => fmtln!(fmt, "None,"),
+            }
+        }
+    });
+    fmtln!(fmt, "];");
+    fmt.empty_line();
+
+    all_inst
+}
+
+/// Get the value type constraint for an SSA value operand, where
+/// `ctrl_typevar` is the controlling type variable.
+///
+/// Each operand constraint is represented as a string, one of:
+/// - `Concrete(vt)`, where `vt` is a value type name.
+/// - `Free(idx)` where `idx` is an index into `type_sets`.
+/// - `Same`, `Lane`, `AsBool` for controlling typevar-derived constraints.
+fn get_constraint<'entries, 'table>(
+    operand: &'entries Operand,
+    ctrl_typevar: Option<&TypeVar>,
+    type_sets: &'table mut UniqueTable<'entries, TypeSet>,
+) -> String {
+    assert!(operand.is_value());
+    let type_var = operand.type_var().unwrap();
+
+    if let Some(typ) = type_var.singleton_type() {
+        return format!("Concrete({})", typ.rust_name());
+    }
+
+    if let Some(free_typevar) = type_var.free_typevar() {
+        if ctrl_typevar.is_some() && free_typevar != *ctrl_typevar.unwrap() {
+            assert!(type_var.base.is_none());
+            return format!("Free({})", type_sets.add(&type_var.get_raw_typeset()));
+        }
+    }
+
+    if let Some(base) = &type_var.base {
+        assert!(base.type_var == *ctrl_typevar.unwrap());
+        return camel_case(base.derived_func.name());
+    }
+
+    assert!(type_var == ctrl_typevar.unwrap());
+    return "Same".into();
+}
+
+fn gen_bitset<'a, T: IntoIterator>(
+    iterable: T,
+    name: &'static str,
+    field_size: u8,
+    fmt: &mut Formatter,
+) {
+    let bits = iterable.into_iter().fold(0, |acc, x| {
+        assert!(x.is_power_of_two());
+        assert!(u32::from(*x) < (1 << u32::from(field_size)));
+        acc | x
+    });
+    fmtln!(fmt, "{}: BitSet::({}),", name, field_size, bits);
+}
+
+fn iterable_to_string>(iterable: T) -> String {
+    let elems = iterable
+        .into_iter()
+        .map(|x| x.to_string())
+        .collect::>()
+        .join(", ");
+    format!("{{{}}}", elems)
+}
+
+fn typeset_to_string(ts: &TypeSet) -> String {
+    let mut result = format!("TypeSet(lanes={}", iterable_to_string(&ts.lanes));
+    if ts.ints.len() > 0 {
+        result += &format!(", ints={}", iterable_to_string(&ts.ints));
+    }
+    if ts.floats.len() > 0 {
+        result += &format!(", floats={}", iterable_to_string(&ts.floats));
+    }
+    if ts.bools.len() > 0 {
+        result += &format!(", bools={}", iterable_to_string(&ts.bools));
+    }
+    if ts.bitvecs.len() > 0 {
+        result += &format!(", bitvecs={}", iterable_to_string(&ts.bitvecs));
+    }
+    if ts.specials.len() > 0 {
+        result += &format!(", specials=[{}]", iterable_to_string(&ts.specials));
+    }
+    result += ")";
+    result
+}
+
+/// Generate the table of ValueTypeSets described by type_sets.
+fn gen_typesets_table(type_sets: &UniqueTable, fmt: &mut Formatter) {
+    if type_sets.len() == 0 {
+        return;
+    }
+
+    fmt.comment("Table of value type sets.");
+    assert!(type_sets.len() <= TYPESET_LIMIT, "Too many type sets!");
+    fmtln!(
+        fmt,
+        "const TYPE_SETS: [ir::instructions::ValueTypeSet; {}] = [",
+        type_sets.len()
+    );
+    fmt.indent(|fmt| {
+        for ts in type_sets.iter() {
+            fmt.line("ir::instructions::ValueTypeSet {");
+            fmt.indent(|fmt| {
+                assert!(ts.bitvecs.len() == 0, "Bitvector types are not emittable.");
+                fmt.comment(typeset_to_string(ts));
+                gen_bitset(&ts.lanes, "lanes", 16, fmt);
+                gen_bitset(&ts.ints, "ints", 8, fmt);
+                gen_bitset(&ts.floats, "floats", 8, fmt);
+                gen_bitset(&ts.bools, "bools", 8, fmt);
+            });
+            fmt.line("},");
+        }
+    });
+    fmtln!(fmt, "];");
+}
+
+/// Generate value type constraints for all instructions.
+/// - Emit a compact constant table of ValueTypeSet objects.
+/// - Emit a compact constant table of OperandConstraint objects.
+/// - Emit an opcode-indexed table of instruction constraints.
+fn gen_type_constraints(all_inst: &Vec<&Instruction>, fmt: &mut Formatter) {
+    // Table of TypeSet instances.
+    let mut type_sets = UniqueTable::new();
+
+    // Table of operand constraint sequences (as tuples). Each operand
+    // constraint is represented as a string, one of:
+    // - `Concrete(vt)`, where `vt` is a value type name.
+    // - `Free(idx)` where `idx` is an index into `type_sets`.
+    // - `Same`, `Lane`, `AsBool` for controlling typevar-derived constraints.
+    let mut operand_seqs = UniqueSeqTable::new();
+
+    // Preload table with constraints for typical binops.
+    operand_seqs.add(&vec!["Same".to_string(); 3]);
+
+    fmt.comment("Table of opcode constraints.");
+    fmtln!(
+        fmt,
+        "const OPCODE_CONSTRAINTS: [OpcodeConstraints; {}] = [",
+        all_inst.len()
+    );
+    fmt.indent(|fmt| {
+        for inst in all_inst {
+            let (ctrl_typevar, ctrl_typeset) = if let Some(poly) = &inst.polymorphic_info {
+                let index = type_sets.add(&*poly.ctrl_typevar.get_raw_typeset());
+                (Some(&poly.ctrl_typevar), index)
+            } else {
+                (None, TYPESET_LIMIT)
+            };
+
+            // Collect constraints for the value results, not including `variable_args` results
+            // which are always special cased.
+            let mut constraints = Vec::new();
+            for &index in &inst.value_results {
+                constraints.push(get_constraint(&inst.operands_out[index], ctrl_typevar, &mut type_sets));
+            }
+            for &index in &inst.value_opnums {
+                constraints.push(get_constraint(&inst.operands_in[index], ctrl_typevar, &mut type_sets));
+            }
+
+            let constraint_offset = operand_seqs.add(&constraints);
+
+            let fixed_results = inst.value_results.len();
+            let fixed_values = inst.value_opnums.len();
+
+            // Can the controlling type variable be inferred from the designated operand?
+            let use_typevar_operand = if let Some(poly) = &inst.polymorphic_info {
+                poly.use_typevar_operand
+            } else {
+                false
+            };
+
+            // Can the controlling type variable be inferred from the result?
+            let use_result = fixed_results > 0 && inst.operands_out[inst.value_results[0]].type_var() == ctrl_typevar;
+
+            // Are we required to use the designated operand instead of the result?
+            let requires_typevar_operand = use_typevar_operand && !use_result;
+
+            fmt.comment(
+                format!("{}: fixed_results={}, use_typevar_operand={}, requires_typevar_operand={}, fixed_values={}",
+                inst.camel_name,
+                fixed_results,
+                use_typevar_operand,
+                requires_typevar_operand,
+                fixed_values)
+            );
+            fmt.comment(format!("Constraints=[{}]", constraints
+                .iter()
+                .map(|x| format!("'{}'", x))
+                .collect::>()
+                .join(", ")));
+            if let Some(poly) = &inst.polymorphic_info {
+                fmt.comment(format!("Polymorphic over {}", typeset_to_string(&poly.ctrl_typevar.get_raw_typeset())));
+            }
+
+            // Compute the bit field encoding, c.f. instructions.rs.
+            assert!(fixed_results < 8 && fixed_values < 8, "Bit field encoding too tight");
+            let mut flags = fixed_results; // 3 bits
+            if use_typevar_operand {
+                flags |= 1<<3; // 4th bit
+            }
+            if requires_typevar_operand {
+                flags |= 1<<4; // 5th bit
+            }
+            flags |= fixed_values << 5; // 6th bit and more
+
+            fmt.line("OpcodeConstraints {");
+            fmt.indent(|fmt| {
+                fmtln!(fmt, "flags: {:#04x},", flags);
+                fmtln!(fmt, "typeset_offset: {},", ctrl_typeset);
+                fmtln!(fmt, "constraint_offset: {},", constraint_offset);
+            });
+            fmt.line("},");
+        }
+    });
+    fmtln!(fmt, "];");
+    fmt.empty_line();
+
+    gen_typesets_table(&type_sets, fmt);
+    fmt.empty_line();
+
+    fmt.comment("Table of operand constraint sequences.");
+    fmtln!(
+        fmt,
+        "const OPERAND_CONSTRAINTS: [OperandConstraint; {}] = [",
+        operand_seqs.len()
+    );
+    fmt.indent(|fmt| {
+        for constraint in operand_seqs.iter() {
+            fmtln!(fmt, "OperandConstraint::{},", constraint);
+        }
+    });
+    fmtln!(fmt, "];");
+}
+
+/// Emit member initializers for an instruction format.
+fn gen_member_inits(format: &InstructionFormat, fmt: &mut Formatter) {
+    // Immediate operands.
+    // We have local variables with the same names as the members.
+    for f in &format.imm_fields {
+        fmtln!(fmt, "{},", f.member);
+    }
+
+    // Value operands.
+    if format.has_value_list {
+        fmt.line("args,");
+    } else if format.num_value_operands == 1 {
+        fmt.line("arg: arg0,");
+    } else if format.num_value_operands > 1 {
+        let mut args = Vec::new();
+        for i in 0..format.num_value_operands {
+            args.push(format!("arg{}", i));
+        }
+        fmtln!(fmt, "args: [{}],", args.join(", "));
+    }
+}
+
+/// Emit a method for creating and inserting an instruction format.
+///
+/// All instruction formats take an `opcode` argument and a `ctrl_typevar` argument for deducing
+/// the result types.
+fn gen_format_constructor(format: &InstructionFormat, fmt: &mut Formatter) {
+    // Construct method arguments.
+    let mut args = vec![
+        "self".to_string(),
+        "opcode: Opcode".into(),
+        "ctrl_typevar: Type".into(),
+    ];
+
+    // Normal operand arguments. Start with the immediate operands.
+    for f in &format.imm_fields {
+        args.push(format!("{}: {}", f.member, f.kind.rust_type));
+    }
+
+    // Then the value operands.
+    if format.has_value_list {
+        // Take all value arguments as a finished value list. The value lists
+        // are created by the individual instruction constructors.
+        args.push("args: ir::ValueList".into());
+    } else {
+        // Take a fixed number of value operands.
+        for i in 0..format.num_value_operands {
+            args.push(format!("arg{}: Value", i));
+        }
+    }
+
+    let proto = format!(
+        "{}({}) -> (Inst, &'f mut ir::DataFlowGraph)",
+        format.name,
+        args.join(", ")
+    );
+
+    fmt.doc_comment(format.to_string());
+    fmt.line("#[allow(non_snake_case)]");
+    fmtln!(fmt, "fn {} {{", proto);
+    fmt.indent(|fmt| {
+        // Generate the instruction data.
+        fmtln!(fmt, "let data = ir::InstructionData::{} {{", format.name);
+        fmt.indent(|fmt| {
+            fmt.line("opcode,");
+            gen_member_inits(format, fmt);
+        });
+        fmtln!(fmt, "};");
+        fmt.line("self.build(data, ctrl_typevar)");
+    });
+    fmtln!(fmt, "}");
+}
+
+/// Emit a method for generating the instruction `inst`.
+///
+/// The method will create and insert an instruction, then return the result values, or the
+/// instruction reference itself for instructions that don't have results.
+fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Formatter) {
+    // Construct method arguments.
+    let mut args = vec![if format.has_value_list {
+        "mut self"
+    } else {
+        "self"
+    }
+    .to_string()];
+
+    // The controlling type variable will be inferred from the input values if
+    // possible. Otherwise, it is the first method argument.
+    if let Some(poly) = &inst.polymorphic_info {
+        if !poly.use_typevar_operand {
+            args.push(format!("{}: crate::ir::Type", poly.ctrl_typevar.name));
+        }
+    }
+
+    let mut tmpl_types = Vec::new();
+    let mut into_args = Vec::new();
+    for op in &inst.operands_in {
+        let t = if op.is_pure_immediate() {
+            let t = format!("T{}{}", tmpl_types.len() + 1, op.kind.name);
+            tmpl_types.push(format!("{}: Into<{}>", t, op.kind.rust_type));
+            into_args.push(op.name);
+            t
+        } else {
+            op.kind.rust_type.clone()
+        };
+        args.push(format!("{}: {}", op.name, t));
+    }
+
+    let rtype = match inst.value_results.len() {
+        0 => "Inst".into(),
+        1 => "Value".into(),
+        _ => format!("({})", vec!["Value"; inst.value_results.len()].join(", ")),
+    };
+
+    let tmpl = if tmpl_types.len() > 0 {
+        format!("<{}>", tmpl_types.join(", "))
+    } else {
+        "".into()
+    };
+
+    let proto = format!(
+        "{}{}({}) -> {}",
+        inst.snake_name(),
+        tmpl,
+        args.join(", "),
+        rtype
+    );
+
+    fmt.doc_comment(format!("`{}`\n\n{}", inst, inst.doc_comment_first_line()));
+    fmt.line("#[allow(non_snake_case)]");
+    fmtln!(fmt, "fn {} {{", proto);
+    fmt.indent(|fmt| {
+        // Convert all of the `Into<>` arguments.
+        for arg in &into_args {
+            fmtln!(fmt, "let {} = {}.into();", arg, arg);
+        }
+
+        // Arguments for instruction constructor.
+        let first_arg = format!("Opcode::{}", inst.camel_name);
+        let mut args = vec![first_arg.as_str()];
+        if let Some(poly) = &inst.polymorphic_info {
+            if poly.use_typevar_operand {
+                // Infer the controlling type variable from the input operands.
+                let op_num = inst.value_opnums[format.typevar_operand.unwrap()];
+                fmtln!(
+                    fmt,
+                    "let ctrl_typevar = self.data_flow_graph().value_type({});",
+                    inst.operands_in[op_num].name
+                );
+
+                // The format constructor will resolve the result types from the type var.
+                args.push("ctrl_typevar");
+            } else {
+                // This was an explicit method argument.
+                args.push(&poly.ctrl_typevar.name);
+            }
+        } else {
+            // No controlling type variable needed.
+            args.push("types::INVALID");
+        }
+
+        // Now add all of the immediate operands to the constructor arguments.
+        for &op_num in &inst.imm_opnums {
+            args.push(inst.operands_in[op_num].name);
+        }
+
+        // Finally, the value operands.
+        if format.has_value_list {
+            // We need to build a value list with all the arguments.
+            fmt.line("let mut vlist = ir::ValueList::default();");
+            args.push("vlist");
+            fmt.line("{");
+            fmt.indent(|fmt| {
+                fmt.line("let pool = &mut self.data_flow_graph_mut().value_lists;");
+                for op in &inst.operands_in {
+                    if op.is_value() {
+                        fmtln!(fmt, "vlist.push({}, pool);", op.name);
+                    } else if op.is_varargs() {
+                        fmtln!(fmt, "vlist.extend({}.iter().cloned(), pool);", op.name);
+                    }
+                }
+            });
+            fmt.line("}");
+        } else {
+            // With no value list, we're guaranteed to just have a set of fixed value operands.
+            for &op_num in &inst.value_opnums {
+                args.push(inst.operands_in[op_num].name);
+            }
+        }
+
+        // Call to the format constructor,
+        let fcall = format!("self.{}({})", format.name, args.join(", "));
+
+        if inst.value_results.len() == 0 {
+            fmtln!(fmt, "{}.0", fcall);
+            return;
+        }
+
+        fmtln!(fmt, "let (inst, dfg) = {};", fcall);
+        if inst.value_results.len() == 1 {
+            fmt.line("dfg.first_result(inst)");
+        } else {
+            fmtln!(
+                fmt,
+                "let results = &dfg.inst_results(inst)[0..{}];",
+                inst.value_results.len()
+            );
+            fmtln!(
+                fmt,
+                "({})",
+                inst.value_results
+                    .iter()
+                    .enumerate()
+                    .map(|(i, _)| format!("results[{}]", i))
+                    .collect::>()
+                    .join(", ")
+            );
+        }
+    });
+    fmtln!(fmt, "}")
+}
+
+/// Generate a Builder trait with methods for all instructions.
+fn gen_builder(instructions: &Vec<&Instruction>, formats: &FormatRegistry, fmt: &mut Formatter) {
+    fmt.doc_comment(
+        r#"
+        Convenience methods for building instructions.
+
+        The `InstBuilder` trait has one method per instruction opcode for
+        conveniently constructing the instruction with minimum arguments.
+        Polymorphic instructions infer their result types from the input
+        arguments when possible. In some cases, an explicit `ctrl_typevar`
+        argument is required.
+
+        The opcode methods return the new instruction's result values, or
+        the `Inst` itself for instructions that don't have any results.
+
+        There is also a method per instruction format. These methods all
+        return an `Inst`.
+    "#,
+    );
+    fmt.line("pub trait InstBuilder<'f>: InstBuilderBase<'f> {");
+    fmt.indent(|fmt| {
+        for inst in instructions {
+            gen_inst_builder(inst, formats.get(inst.format), fmt);
+        }
+        for format in formats.iter() {
+            gen_format_constructor(format, fmt);
+        }
+    });
+    fmt.line("}");
+}
+
+pub fn generate(
+    all_inst_groups: Vec<&InstructionGroup>,
+    format_registry: &FormatRegistry,
+    opcode_filename: &str,
+    inst_builder_filename: &str,
+    out_dir: &str,
+) -> Result<(), error::Error> {
+    // Opcodes.
+    let mut fmt = Formatter::new();
+    gen_formats(format_registry, &mut fmt);
+    gen_instruction_data(format_registry, &mut fmt);
+    fmt.empty_line();
+    gen_instruction_data_impl(format_registry, &mut fmt);
+    fmt.empty_line();
+    let all_inst = gen_opcodes(format_registry, &all_inst_groups, &mut fmt);
+    gen_type_constraints(&all_inst, &mut fmt);
+    fmt.update_file(opcode_filename, out_dir)?;
+
+    // Instruction builder.
+    let mut fmt = Formatter::new();
+    gen_builder(&all_inst, format_registry, &mut fmt);
+    fmt.update_file(inst_builder_filename, out_dir)?;
+
+    Ok(())
+}
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index 9b50fe1232..8118922fb0 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -5,6 +5,7 @@ mod srcgen;
 pub mod error;
 pub mod isa;
 
+mod gen_inst;
 mod gen_registers;
 mod gen_settings;
 mod gen_types;
@@ -33,6 +34,17 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
     // Per ISA definitions.
     let isas = isa::define(isas, &mut shared_defs);
 
+    let mut all_inst_groups = vec![&shared_defs.instructions];
+    all_inst_groups.extend(isas.iter().map(|isa| &isa.instructions));
+
+    gen_inst::generate(
+        all_inst_groups,
+        &shared_defs.format_registry,
+        "new_opcodes.rs",
+        "new_inst_builder.rs",
+        &out_dir,
+    )?;
+
     for isa in isas {
         gen_registers::generate(&isa, &format!("registers-{}.rs", isa.name), &out_dir)?;
         gen_settings::generate(

From 9b156fd9bbcbc64968463ac237e7dd8a8e2da2c1 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 11 Mar 2019 19:50:31 +0100
Subject: [PATCH 2381/3084] [meta] Use the Rust crate for opcodes/inst_builder
 code generation;

---
 cranelift/codegen/meta-python/build.py        |  23 +-
 cranelift/codegen/meta-python/gen_instr.py    | 812 ------------------
 .../codegen/meta-python/gen_legalizer.py      |  22 +-
 cranelift/codegen/meta/src/lib.rs             |   4 +-
 4 files changed, 44 insertions(+), 817 deletions(-)
 delete mode 100644 cranelift/codegen/meta-python/gen_instr.py

diff --git a/cranelift/codegen/meta-python/build.py b/cranelift/codegen/meta-python/build.py
index 37c23554b8..28f510c1e9 100644
--- a/cranelift/codegen/meta-python/build.py
+++ b/cranelift/codegen/meta-python/build.py
@@ -5,13 +5,32 @@
 from __future__ import absolute_import
 import argparse
 import isa
-import gen_instr
 import gen_settings
 import gen_build_deps
 import gen_encoding
 import gen_legalizer
 import gen_binemit
 
+try:
+    from typing import List, Set  # noqa
+    from cdsl.isa import TargetISA  # noqa
+    from cdsl.instructions import InstructionGroup  # noqa
+except ImportError:
+    pass
+
+
+def number_all_instructions(isas):
+    # type: (List[TargetISA]) -> None
+    seen = set()  # type: Set[InstructionGroup]
+    num_inst = 1
+    for target_isa in isas:
+        for g in target_isa.instruction_groups:
+            if g not in seen:
+                for i in g.instructions:
+                    i.number = num_inst
+                    num_inst += 1
+                seen.add(g)
+
 
 def main():
     # type: () -> None
@@ -23,8 +42,8 @@ def main():
     out_dir = args.out_dir
 
     isas = isa.all_isas()
+    number_all_instructions(isas)
 
-    gen_instr.generate(isas, out_dir)
     gen_settings.generate(isas, out_dir)
     gen_encoding.generate(isas, out_dir)
     gen_legalizer.generate(isas, out_dir)
diff --git a/cranelift/codegen/meta-python/gen_instr.py b/cranelift/codegen/meta-python/gen_instr.py
deleted file mode 100644
index 94de93b000..0000000000
--- a/cranelift/codegen/meta-python/gen_instr.py
+++ /dev/null
@@ -1,812 +0,0 @@
-"""
-Generate sources with instruction info.
-"""
-from __future__ import absolute_import
-import srcgen
-import constant_hash
-from unique_table import UniqueTable, UniqueSeqTable
-from cdsl import camel_case
-from cdsl.operands import ImmediateKind
-from cdsl.formats import InstructionFormat
-from cdsl.instructions import Instruction
-
-# The typing module is only required by mypy, and we don't use these imports
-# outside type comments.
-try:
-    from typing import List, Sequence, Set, TYPE_CHECKING  # noqa
-    if TYPE_CHECKING:
-        from cdsl.isa import TargetISA  # noqa
-        from cdsl.instructions import InstructionGroup  # noqa
-        from cdsl.operands import Operand  # noqa
-        from cdsl.typevar import TypeVar  # noqa
-
-except ImportError:
-    pass
-
-
-def gen_formats(fmt):
-    # type: (srcgen.Formatter) -> None
-    """Generate an instruction format enumeration"""
-
-    fmt.doc_comment('''
-                    An instruction format
-
-                    Every opcode has a corresponding instruction format
-                    which is represented by both the `InstructionFormat`
-                    and the `InstructionData` enums.
-                    ''')
-    fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug)]')
-    with fmt.indented('pub enum InstructionFormat {', '}'):
-        for f in InstructionFormat.all_formats:
-            fmt.doc_comment(str(f))
-            fmt.line(f.name + ',')
-    fmt.line()
-
-    # Emit a From which also serves to verify that
-    # InstructionFormat and InstructionData are in sync.
-    with fmt.indented(
-            "impl<'a> From<&'a InstructionData> for InstructionFormat {", '}'):
-        with fmt.indented(
-                "fn from(inst: &'a InstructionData) -> Self {",
-                '}'):
-            m = srcgen.Match('*inst')
-            for f in InstructionFormat.all_formats:
-                m.arm('InstructionData::' + f.name, ['..'],
-                      'InstructionFormat::' + f.name)
-            fmt.match(m)
-    fmt.line()
-
-
-def gen_arguments_method(fmt, is_mut):
-    # type: (srcgen.Formatter, bool) -> None
-    method = 'arguments'
-    mut = ''
-    rslice = 'ref_slice'
-    as_slice = 'as_slice'
-    if is_mut:
-        method += '_mut'
-        mut = 'mut '
-        rslice += '_mut'
-        as_slice = 'as_mut_slice'
-
-    with fmt.indented(
-            'pub fn {f}<\'a>(&\'a {m}self, '
-            'pool: &\'a {m}ir::ValueListPool) -> '
-            '&{m}[Value] {{'
-            .format(f=method, m=mut), '}'):
-        m = srcgen.Match('*self')
-        for f in InstructionFormat.all_formats:
-            n = 'InstructionData::' + f.name
-
-            # Formats with a value list put all of their arguments in the
-            # list. We don't split them up, just return it all as variable
-            # arguments. (I expect the distinction to go away).
-            if f.has_value_list:
-                m.arm(n, ['ref {}args'.format(mut), '..'],
-                      'args.{}(pool)'.format(as_slice))
-                continue
-
-            # Fixed args.
-            fields = []
-            if f.num_value_operands == 0:
-                arg = '&{}[]'.format(mut)
-            elif f.num_value_operands == 1:
-                fields.append('ref {}arg'.format(mut))
-                arg = '{}(arg)'.format(rslice)
-            else:
-                args = 'args_arity{}'.format(f.num_value_operands)
-                fields.append('args: ref {}{}'.format(mut, args))
-                arg = args
-            fields.append('..')
-            m.arm(n, fields, arg)
-        fmt.match(m)
-
-
-def gen_instruction_data(fmt):
-    # type: (srcgen.Formatter) -> None
-    """
-    Generate the InstructionData enum.
-
-    Every variant must contain an `opcode` field. The size of `InstructionData`
-    should be kept at 16 bytes on 64-bit architectures. If more space is needed
-    to represent an instruction, use a `Box` to store the additional
-    information out of line.
-    """
-
-    fmt.line('#[derive(Clone, Debug)]')
-    fmt.line('#[allow(missing_docs)]')
-    with fmt.indented('pub enum InstructionData {', '}'):
-        for f in InstructionFormat.all_formats:
-            with fmt.indented('{} {{'.format(f.name), '},'):
-                fmt.line('opcode: Opcode,')
-                if f.typevar_operand is None:
-                    pass
-                elif f.has_value_list:
-                    fmt.line('args: ValueList,')
-                elif f.num_value_operands == 1:
-                    fmt.line('arg: Value,')
-                else:
-                    fmt.line('args: [Value; {}],'.format(f.num_value_operands))
-                for field in f.imm_fields:
-                    fmt.line(
-                            '{}: {},'
-                            .format(field.member, field.kind.rust_type))
-
-
-def gen_instruction_data_impl(fmt):
-    # type: (srcgen.Formatter) -> None
-    """
-    Generate the boring parts of the InstructionData implementation.
-
-    These methods in `impl InstructionData` can be generated automatically from
-    the instruction formats:
-
-    - `pub fn opcode(&self) -> Opcode`
-    - `pub fn arguments(&self, &pool) -> &[Value]`
-    - `pub fn arguments_mut(&mut self, &pool) -> &mut [Value]`
-    - `pub fn take_value_list(&mut self) -> Option`
-    - `pub fn put_value_list(&mut self, args: ir::ValueList>`
-    - `pub fn eq(&self, &other: Self, &pool) -> bool`
-    - `pub fn hash(&self, state: &mut H, &pool)`
-    """
-
-    # The `opcode` method simply reads the `opcode` members. This is really a
-    # workaround for Rust's enum types missing shared members.
-    with fmt.indented('impl InstructionData {', '}'):
-        fmt.doc_comment('Get the opcode of this instruction.')
-        with fmt.indented('pub fn opcode(&self) -> Opcode {', '}'):
-            m = srcgen.Match('*self')
-            for f in InstructionFormat.all_formats:
-                m.arm('InstructionData::' + f.name, ['opcode', '..'],
-                      'opcode')
-            fmt.match(m)
-        fmt.line()
-
-        fmt.doc_comment('Get the controlling type variable operand.')
-        with fmt.indented(
-                'pub fn typevar_operand(&self, pool: &ir::ValueListPool) -> '
-                'Option {', '}'):
-            m = srcgen.Match('*self')
-            for f in InstructionFormat.all_formats:
-                n = 'InstructionData::' + f.name
-                if f.typevar_operand is None:
-                    m.arm(n, ['..'], 'None')
-                elif f.has_value_list:
-                    # We keep all arguments in a value list.
-                    i = f.typevar_operand
-                    m.arm(n, ['ref args', '..'],
-                          'args.get({}, pool)'.format(i))
-                elif f.num_value_operands == 1:
-                    # We have a single value operand called 'arg'.
-                    m.arm(n, ['arg', '..'], 'Some(arg)')
-                else:
-                    # We have multiple value operands and an array `args`.
-                    # Which `args` index to use?
-                    args = 'args_arity{}'.format(f.num_value_operands)
-                    m.arm(n, ['args: ref {}'.format(args), '..'],
-                          'Some({}[{}])'.format(args, f.typevar_operand))
-            fmt.match(m)
-        fmt.line()
-
-        fmt.doc_comment(
-                """
-                Get the value arguments to this instruction.
-                """)
-        gen_arguments_method(fmt, False)
-        fmt.line()
-
-        fmt.doc_comment(
-                """
-                Get mutable references to the value arguments to this
-                instruction.
-                """)
-        gen_arguments_method(fmt, True)
-        fmt.line()
-
-        fmt.doc_comment(
-                """
-                Take out the value list with all the value arguments and return
-                it.
-
-                This leaves the value list in the instruction empty. Use
-                `put_value_list` to put the value list back.
-                """)
-        with fmt.indented(
-                'pub fn take_value_list(&mut self) -> Option {',
-                '}'):
-            m = srcgen.Match('*self')
-            for f in InstructionFormat.all_formats:
-                n = 'InstructionData::' + f.name
-                if f.has_value_list:
-                    m.arm(n, ['ref mut args', '..'], 'Some(args.take())')
-            m.arm('_', [], 'None')
-            fmt.match(m)
-        fmt.line()
-
-        fmt.doc_comment(
-                """
-                Put back a value list.
-
-                After removing a value list with `take_value_list()`, use this
-                method to put it back. It is required that this instruction has
-                a format that accepts a value list, and that the existing value
-                list is empty. This avoids leaking list pool memory.
-                """)
-        with fmt.indented(
-                'pub fn put_value_list(&mut self, vlist: ir::ValueList) {',
-                '}'):
-            with fmt.indented('let args = match *self {', '};'):
-                for f in InstructionFormat.all_formats:
-                    n = 'InstructionData::' + f.name
-                    if f.has_value_list:
-                        fmt.line(n + ' { ref mut args, .. } => args,')
-                fmt.line('_ => panic!("No value list: {:?}", self),')
-            fmt.line(
-                'debug_assert!(args.is_empty(), "Value list already in use");')
-            fmt.line('*args = vlist;')
-        fmt.line()
-
-        fmt.doc_comment(
-                """
-                Compare two `InstructionData` for equality.
-
-                This operation requires a reference to a `ValueListPool` to
-                determine if the contents of any `ValueLists` are equal.
-                """)
-        with fmt.indented(
-                'pub fn eq(&self, other: &Self, pool: &ir::ValueListPool)'
-                ' -> bool {',
-                '}'):
-            with fmt.indented('if ::core::mem::discriminant(self) != '
-                              '::core::mem::discriminant(other) {', '}'):
-                fmt.line('return false;')
-            with fmt.indented('match (self, other) {', '}'):
-                for f in InstructionFormat.all_formats:
-                    n = '&InstructionData::' + f.name
-                    members = ['opcode']
-                    if f.typevar_operand is None:
-                        args_eq = None
-                    elif f.has_value_list:
-                        members.append('args')
-                        args_eq = 'args1.as_slice(pool) == ' \
-                                  'args2.as_slice(pool)'
-                    elif f.num_value_operands == 1:
-                        members.append('arg')
-                        args_eq = 'arg1 == arg2'
-                    else:
-                        members.append('args')
-                        args_eq = 'args1 == args2'
-                    for field in f.imm_fields:
-                        members.append(field.member)
-                    pat1 = ', '.join('{}: ref {}1'.format(x, x)
-                                     for x in members)
-                    pat2 = ', '.join('{}: ref {}2'.format(x, x)
-                                     for x in members)
-                    with fmt.indented('({} {{ {} }}, {} {{ {} }}) => {{'
-                                      .format(n, pat1, n, pat2), '}'):
-                        fmt.line('opcode1 == opcode2')
-                        for field in f.imm_fields:
-                            fmt.line('&& {}1 == {}2'
-                                     .format(field.member, field.member))
-                        if args_eq is not None:
-                            fmt.line('&& {}'.format(args_eq))
-                fmt.line('_ => unreachable!()')
-        fmt.line()
-
-        fmt.doc_comment(
-                """
-                Hash an `InstructionData`.
-
-                This operation requires a reference to a `ValueListPool` to
-                hash the contents of any `ValueLists`.
-                """)
-        with fmt.indented(
-                'pub fn hash'
-                '(&self, state: &mut H, pool: &ir::ValueListPool) {',
-                '}'):
-            with fmt.indented('match *self {', '}'):
-                for f in InstructionFormat.all_formats:
-                    n = 'InstructionData::' + f.name
-                    members = ['opcode']
-                    if f.typevar_operand is None:
-                        args = '&()'
-                    elif f.has_value_list:
-                        members.append('ref args')
-                        args = 'args.as_slice(pool)'
-                    elif f.num_value_operands == 1:
-                        members.append('ref arg')
-                        args = 'arg'
-                    else:
-                        members.append('ref args')
-                        args = 'args'
-                    for field in f.imm_fields:
-                        members.append(field.member)
-                    pat = n + ' { ' + ', '.join(members) + ' }'
-                    with fmt.indented(pat + ' => {', '}'):
-                        fmt.line('::core::hash::Hash::hash( '
-                                 '&::core::mem::discriminant(self), state);')
-                        fmt.line('::core::hash::Hash::hash(&opcode, state);')
-                        for field in f.imm_fields:
-                            fmt.line('::core::hash::Hash::hash(&{}, state);'
-                                     .format(field.member))
-                        fmt.line('::core::hash::Hash::hash({}, state);'
-                                 .format(args))
-
-
-def collect_instr_groups(isas):
-    # type: (Sequence[TargetISA]) -> List[InstructionGroup]
-    seen = set()  # type: Set[InstructionGroup]
-    groups = []
-    for isa in isas:
-        for g in isa.instruction_groups:
-            if g not in seen:
-                groups.append(g)
-                seen.add(g)
-    return groups
-
-
-def gen_opcodes(groups, fmt):
-    # type: (Sequence[InstructionGroup], srcgen.Formatter) -> Sequence[Instruction]  # noqa
-    """
-    Generate opcode enumerations.
-
-    Return a list of all instructions.
-    """
-
-    fmt.doc_comment('''
-                    An instruction opcode.
-
-                    All instructions from all supported ISAs are present.
-                    ''')
-    fmt.line('#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)]')
-    instrs = []
-
-    # We explicitly set the discriminant of the first variant to 1, which
-    # allows us to take advantage of the NonZero optimization, meaning that
-    # wrapping enums can use the 0 discriminant instead of increasing the size
-    # of the whole type, and so SIZEOF(Option>) == SIZEOF(Opcode)
-    is_first_opcode = True
-    with fmt.indented('pub enum Opcode {', '}'):
-        for g in groups:
-            for i in g.instructions:
-                instrs.append(i)
-                i.number = len(instrs)
-                fmt.doc_comment('`{}`. ({})'.format(i, i.format.name))
-                # Document polymorphism.
-                if i.is_polymorphic:
-                    if i.use_typevar_operand:
-                        opnum = i.value_opnums[i.format.typevar_operand]
-                        fmt.doc_comment(
-                                'Type inferred from {}.'
-                                .format(i.ins[opnum]))
-                # Enum variant itself.
-                if is_first_opcode:
-                    fmt.line(i.camel_name + ' = 1,')
-                    is_first_opcode = False
-                else:
-                    fmt.line(i.camel_name + ',')
-    fmt.line()
-
-    with fmt.indented('impl Opcode {', '}'):
-        for attr in sorted(Instruction.ATTRIBS.keys()):
-            fmt.doc_comment(Instruction.ATTRIBS[attr])
-            with fmt.indented('pub fn {}(self) -> bool {{'
-                              .format(attr), '}'):
-                m = srcgen.Match('self')
-                for i in instrs:
-                    if getattr(i, attr):
-                        m.arm('Opcode::' + i.camel_name, [], 'true')
-                m.arm('_', [], 'false')
-                fmt.match(m)
-            fmt.line()
-    fmt.line()
-
-    # Generate a private opcode_format table.
-    with fmt.indented(
-            'const OPCODE_FORMAT: [InstructionFormat; {}] = ['
-            .format(len(instrs)),
-            '];'):
-        for i in instrs:
-            fmt.format(
-                    'InstructionFormat::{}, // {}',
-                    i.format.name, i.name)
-    fmt.line()
-
-    # Generate a private opcode_name function.
-    with fmt.indented('fn opcode_name(opc: Opcode) -> &\'static str {', '}'):
-        m = srcgen.Match('opc')
-        for i in instrs:
-            m.arm('Opcode::' + i.camel_name, [], '"{}"'.format(i.name))
-        fmt.match(m)
-    fmt.line()
-
-    # Generate an opcode hash table for looking up opcodes by name.
-    hash_table = constant_hash.compute_quadratic(
-            instrs,
-            lambda i: constant_hash.simple_hash(i.name))
-    with fmt.indented(
-            'const OPCODE_HASH_TABLE: [Option; {}] = ['
-            .format(len(hash_table)), '];'):
-        for i in hash_table:
-            if i is None:
-                fmt.line('None,')
-            else:
-                fmt.format('Some(Opcode::{}),', i.camel_name)
-    fmt.line()
-    return instrs
-
-
-def get_constraint(op, ctrl_typevar, type_sets):
-    # type: (Operand, TypeVar, UniqueTable) -> str
-    """
-    Get the value type constraint for an SSA value operand, where
-    `ctrl_typevar` is the controlling type variable.
-
-    Each operand constraint is represented as a string, one of:
-
-    - `Concrete(vt)`, where `vt` is a value type name.
-    - `Free(idx)` where `idx` is an index into `type_sets`.
-    - `Same`, `Lane`, `AsBool` for controlling typevar-derived constraints.
-    """
-    assert op.is_value()
-    tv = op.typevar
-
-    # A concrete value type.
-    if tv.singleton_type():
-        return 'Concrete({})'.format(tv.singleton_type().rust_name())
-
-    if tv.free_typevar() is not ctrl_typevar:
-        assert not tv.is_derived
-        return 'Free({})'.format(type_sets.add(tv.type_set))
-
-    if tv.is_derived:
-        assert tv.base is ctrl_typevar, "Not derived from ctrl_typevar"
-        return camel_case(tv.derived_func)
-
-    assert tv is ctrl_typevar
-    return 'Same'
-
-
-# TypeSet indexes are encoded in 8 bits, with `0xff` reserved.
-typeset_limit = 0xff
-
-
-def gen_typesets_table(fmt, type_sets):
-    # type: (srcgen.Formatter, UniqueTable) -> None
-    """
-    Generate the table of ValueTypeSets described by type_sets.
-    """
-    if len(type_sets.table) == 0:
-        return
-    fmt.comment('Table of value type sets.')
-    assert len(type_sets.table) <= typeset_limit, "Too many type sets"
-    with fmt.indented(
-            'const TYPE_SETS: [ir::instructions::ValueTypeSet; {}] = ['
-            .format(len(type_sets.table)), '];'):
-        for ts in type_sets.table:
-            with fmt.indented('ir::instructions::ValueTypeSet {', '},'):
-                ts.emit_fields(fmt)
-
-
-def gen_type_constraints(fmt, instrs):
-    # type: (srcgen.Formatter, Sequence[Instruction]) -> None
-    """
-    Generate value type constraints for all instructions.
-
-    - Emit a compact constant table of ValueTypeSet objects.
-    - Emit a compact constant table of OperandConstraint objects.
-    - Emit an opcode-indexed table of instruction constraints.
-
-    """
-
-    # Table of TypeSet instances.
-    type_sets = UniqueTable()
-
-    # Table of operand constraint sequences (as tuples). Each operand
-    # constraint is represented as a string, one of:
-    # - `Concrete(vt)`, where `vt` is a value type name.
-    # - `Free(idx)` where `idx` is an index into `type_sets`.
-    # - `Same`, `Lane`, `AsBool` for controlling typevar-derived constraints.
-    operand_seqs = UniqueSeqTable()
-
-    # Preload table with constraints for typical binops.
-    operand_seqs.add(['Same'] * 3)
-
-    fmt.comment('Table of opcode constraints.')
-    with fmt.indented(
-            'const OPCODE_CONSTRAINTS: [OpcodeConstraints; {}] = ['
-            .format(len(instrs)), '];'):
-        for i in instrs:
-            # Collect constraints for the value results, not including
-            # `variable_args` results which are always special cased.
-            constraints = list()
-            ctrl_typevar = None
-            ctrl_typeset = typeset_limit
-            if i.is_polymorphic:
-                ctrl_typevar = i.ctrl_typevar
-                ctrl_typeset = type_sets.add(ctrl_typevar.type_set)
-            for idx in i.value_results:
-                constraints.append(
-                        get_constraint(i.outs[idx], ctrl_typevar, type_sets))
-            for opnum in i.value_opnums:
-                constraints.append(
-                        get_constraint(i.ins[opnum], ctrl_typevar, type_sets))
-            offset = operand_seqs.add(constraints)
-            fixed_results = len(i.value_results)
-            fixed_values = len(i.value_opnums)
-            # Can the controlling type variable be inferred from the designated
-            # operand?
-            use_typevar_operand = i.is_polymorphic and i.use_typevar_operand
-            # Can the controlling type variable be inferred from the result?
-            use_result = (fixed_results > 0 and
-                          i.outs[i.value_results[0]].typevar == ctrl_typevar)
-            # Are we required to use the designated operand instead of the
-            # result?
-            requires_typevar_operand = use_typevar_operand and not use_result
-            fmt.comment(
-                    '{}: fixed_results={}, use_typevar_operand={}, '
-                    'requires_typevar_operand={}, fixed_values={}'
-                    .format(i.camel_name, fixed_results, use_typevar_operand,
-                            requires_typevar_operand, fixed_values))
-            fmt.comment('Constraints={}'.format(constraints))
-            if i.is_polymorphic:
-                fmt.comment(
-                        'Polymorphic over {}'.format(ctrl_typevar.type_set))
-            # Compute the bit field encoding, c.f. instructions.rs.
-            assert fixed_results < 8, "Bit field encoding too tight"
-            flags = fixed_results
-            if use_typevar_operand:
-                flags |= 8
-            if requires_typevar_operand:
-                flags |= 0x10
-            assert fixed_values < 8, "Bit field encoding too tight"
-            flags |= fixed_values << 5
-
-            with fmt.indented('OpcodeConstraints {', '},'):
-                fmt.line('flags: {:#04x},'.format(flags))
-                fmt.line('typeset_offset: {},'.format(ctrl_typeset))
-                fmt.line('constraint_offset: {},'.format(offset))
-    fmt.line()
-
-    gen_typesets_table(fmt, type_sets)
-    fmt.line()
-
-    fmt.comment('Table of operand constraint sequences.')
-    with fmt.indented(
-            'const OPERAND_CONSTRAINTS: [OperandConstraint; {}] = ['
-            .format(len(operand_seqs.table)), '];'):
-        for c in operand_seqs.table:
-            fmt.line('OperandConstraint::{},'.format(c))
-
-
-def gen_format_constructor(iform, fmt):
-    # type: (InstructionFormat, srcgen.Formatter) -> None
-    """
-    Emit a method for creating and inserting an `iform` instruction, where
-    `iform` is an instruction format.
-
-    All instruction formats take an `opcode` argument and a `ctrl_typevar`
-    argument for deducing the result types.
-    """
-
-    # Construct method arguments.
-    args = ['self', 'opcode: Opcode', 'ctrl_typevar: Type']
-
-    # Normal operand arguments. Start with the immediate operands.
-    for f in iform.imm_fields:
-        args.append('{}: {}'.format(f.member, f.kind.rust_type))
-    # Then the value operands.
-    if iform.has_value_list:
-        # Take all value arguments as a finished value list. The value lists
-        # are created by the individual instruction constructors.
-        args.append('args: ir::ValueList')
-    else:
-        # Take a fixed number of value operands.
-        for i in range(iform.num_value_operands):
-            args.append('arg{}: Value'.format(i))
-
-    proto = '{}({})'.format(iform.name, ', '.join(args))
-    proto += " -> (Inst, &'f mut ir::DataFlowGraph)"
-
-    fmt.doc_comment(str(iform))
-    fmt.line('#[allow(non_snake_case)]')
-    with fmt.indented('fn {} {{'.format(proto), '}'):
-        # Generate the instruction data.
-        with fmt.indented(
-                'let data = ir::InstructionData::{} {{'.format(iform.name),
-                '};'):
-            fmt.line('opcode,')
-            gen_member_inits(iform, fmt)
-
-        fmt.line('self.build(data, ctrl_typevar)')
-
-
-def gen_member_inits(iform, fmt):
-    # type: (InstructionFormat, srcgen.Formatter) -> None
-    """
-    Emit member initializers for an `iform` instruction.
-    """
-
-    # Immediate operands.
-    # We have local variables with the same names as the members.
-    for f in iform.imm_fields:
-        fmt.line('{},'.format(f.member))
-
-    # Value operands.
-    if iform.has_value_list:
-        fmt.line('args,')
-    elif iform.num_value_operands == 1:
-        fmt.line('arg: arg0,')
-    elif iform.num_value_operands > 1:
-        args = ('arg{}'.format(i) for i in range(iform.num_value_operands))
-        fmt.line('args: [{}],'.format(', '.join(args)))
-
-
-def gen_inst_builder(inst, fmt):
-    # type: (Instruction, srcgen.Formatter) -> None
-    """
-    Emit a method for generating the instruction `inst`.
-
-    The method will create and insert an instruction, then return the result
-    values, or the instruction reference itself for instructions that don't
-    have results.
-    """
-
-    # Construct method arguments.
-    if inst.format.has_value_list:
-        args = ['mut self']
-    else:
-        args = ['self']
-
-    # The controlling type variable will be inferred from the input values if
-    # possible. Otherwise, it is the first method argument.
-    if inst.is_polymorphic and not inst.use_typevar_operand:
-        args.append('{}: crate::ir::Type'.format(inst.ctrl_typevar.name))
-
-    tmpl_types = list()  # type: List[str]
-    into_args = list()  # type: List[str]
-    for op in inst.ins:
-        if isinstance(op.kind, ImmediateKind):
-            t = 'T{}{}'.format(1 + len(tmpl_types), op.kind.name)
-            tmpl_types.append('{}: Into<{}>'.format(t, op.kind.rust_type))
-            into_args.append(op.name)
-        else:
-            t = op.kind.rust_type
-        args.append('{}: {}'.format(op.name, t))
-
-    # Return the inst reference for result-less instructions.
-    if len(inst.value_results) == 0:
-        rtype = 'Inst'
-    elif len(inst.value_results) == 1:
-        rtype = 'Value'
-    else:
-        rvals = ', '.join(len(inst.value_results) * ['Value'])
-        rtype = '({})'.format(rvals)
-
-    if len(tmpl_types) > 0:
-        tmpl = '<{}>'.format(', '.join(tmpl_types))
-    else:
-        tmpl = ''
-    proto = '{}{}({}) -> {}'.format(
-            inst.snake_name(), tmpl,  ', '.join(args), rtype)
-
-    fmt.doc_comment('`{}`\n\n{}'.format(inst, inst.blurb()))
-    fmt.line('#[allow(non_snake_case)]')
-    with fmt.indented('fn {} {{'.format(proto), '}'):
-        # Convert all of the `Into<>` arguments.
-        for arg in into_args:
-            fmt.line('let {} = {}.into();'.format(arg, arg))
-
-        # Arguments for instruction constructor.
-        args = ['Opcode::' + inst.camel_name]
-
-        if inst.is_polymorphic and not inst.use_typevar_operand:
-            # This was an explicit method argument.
-            args.append(inst.ctrl_typevar.name)
-        elif not inst.is_polymorphic:
-            # No controlling type variable needed.
-            args.append('types::INVALID')
-        else:
-            assert inst.is_polymorphic and inst.use_typevar_operand
-            # Infer the controlling type variable from the input operands.
-            opnum = inst.value_opnums[inst.format.typevar_operand]
-            fmt.line(
-                    'let ctrl_typevar = self.data_flow_graph().value_type({});'
-                    .format(inst.ins[opnum].name))
-            # The format constructor will resolve the result types from the
-            # type var.
-            args.append('ctrl_typevar')
-
-        # Now add all of the immediate operands to the constructor arguments.
-        for opnum in inst.imm_opnums:
-            args.append(inst.ins[opnum].name)
-
-        # Finally, the value operands.
-        if inst.format.has_value_list:
-            # We need to build a value list with all the arguments.
-            fmt.line('let mut vlist = ir::ValueList::default();')
-            args.append('vlist')
-            with fmt.indented('{', '}'):
-                fmt.line(
-                        'let pool = '
-                        '&mut self.data_flow_graph_mut().value_lists;')
-                for op in inst.ins:
-                    if op.is_value():
-                        fmt.line('vlist.push({}, pool);'.format(op.name))
-                    elif op.is_varargs():
-                        fmt.line(
-                            'vlist.extend({}.iter().cloned(), pool);'
-                            .format(op.name))
-        else:
-            # With no value list, we're guaranteed to just have a set of fixed
-            # value operands.
-            for opnum in inst.value_opnums:
-                args.append(inst.ins[opnum].name)
-
-        # Call to the format constructor,
-        fcall = 'self.{}({})'.format(inst.format.name, ', '.join(args))
-
-        if len(inst.value_results) == 0:
-            fmt.line(fcall + '.0')
-            return
-
-        fmt.line('let (inst, dfg) = {};'.format(fcall))
-
-        if len(inst.value_results) == 1:
-            fmt.line('dfg.first_result(inst)')
-            return
-
-        fmt.format(
-            'let results = &dfg.inst_results(inst)[0..{}];',
-            len(inst.value_results))
-        fmt.format('({})', ', '.join(
-            'results[{}]'.format(i) for i in range(len(inst.value_results))))
-
-
-def gen_builder(insts, fmt):
-    # type: (Sequence[Instruction], srcgen.Formatter) -> None
-    """
-    Generate a Builder trait with methods for all instructions.
-    """
-    fmt.doc_comment("""
-            Convenience methods for building instructions.
-
-            The `InstBuilder` trait has one method per instruction opcode for
-            conveniently constructing the instruction with minimum arguments.
-            Polymorphic instructions infer their result types from the input
-            arguments when possible. In some cases, an explicit `ctrl_typevar`
-            argument is required.
-
-            The opcode methods return the new instruction's result values, or
-            the `Inst` itself for instructions that don't have any results.
-
-            There is also a method per instruction format. These methods all
-            return an `Inst`.
-            """)
-    with fmt.indented(
-            "pub trait InstBuilder<'f>: InstBuilderBase<'f> {",  '}'):
-        for inst in insts:
-            gen_inst_builder(inst, fmt)
-        for f in InstructionFormat.all_formats:
-            gen_format_constructor(f, fmt)
-
-
-def generate(isas, out_dir):
-    # type: (Sequence[TargetISA], str) -> None
-    groups = collect_instr_groups(isas)
-
-    # opcodes.rs
-    fmt = srcgen.Formatter()
-    gen_formats(fmt)
-    gen_instruction_data(fmt)
-    fmt.line()
-    gen_instruction_data_impl(fmt)
-    fmt.line()
-    instrs = gen_opcodes(groups, fmt)
-    gen_type_constraints(fmt, instrs)
-    fmt.update_file('opcodes.rs', out_dir)
-
-    # inst_builder.rs
-    fmt = srcgen.Formatter()
-    gen_builder(instrs, fmt)
-    fmt.update_file('inst_builder.rs', out_dir)
diff --git a/cranelift/codegen/meta-python/gen_legalizer.py b/cranelift/codegen/meta-python/gen_legalizer.py
index 2f78d086f3..b4dfd0089f 100644
--- a/cranelift/codegen/meta-python/gen_legalizer.py
+++ b/cranelift/codegen/meta-python/gen_legalizer.py
@@ -15,7 +15,6 @@ from cdsl.ast import Var
 from cdsl.ti import ti_rtl, TypeEnv, get_type_env, TypesEqual,\
     InTypeset, WiderOrEq
 from unique_table import UniqueTable
-from gen_instr import gen_typesets_table
 from cdsl.typevar import TypeVar
 
 try:
@@ -29,6 +28,27 @@ except ImportError:
     pass
 
 
+# TypeSet indexes are encoded in 8 bits, with `0xff` reserved.
+typeset_limit = 0xff
+
+
+def gen_typesets_table(fmt, type_sets):
+    # type: (Formatter, UniqueTable) -> None
+    """
+    Generate the table of ValueTypeSets described by type_sets.
+    """
+    if len(type_sets.table) == 0:
+        return
+    fmt.comment('Table of value type sets.')
+    assert len(type_sets.table) <= typeset_limit, "Too many type sets"
+    with fmt.indented(
+            'const TYPE_SETS: [ir::instructions::ValueTypeSet; {}] = ['
+            .format(len(type_sets.table)), '];'):
+        for ts in type_sets.table:
+            with fmt.indented('ir::instructions::ValueTypeSet {', '},'):
+                ts.emit_fields(fmt)
+
+
 def get_runtime_typechecks(xform):
     # type: (XForm) -> List[TypeConstraint]
     """
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index 8118922fb0..fccba327e0 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -40,8 +40,8 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
     gen_inst::generate(
         all_inst_groups,
         &shared_defs.format_registry,
-        "new_opcodes.rs",
-        "new_inst_builder.rs",
+        "opcodes.rs",
+        "inst_builder.rs",
         &out_dir,
     )?;
 

From fc6876c68e91ca1ea1b6614467f1ac110efce866 Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Tue, 18 Dec 2018 12:10:00 -0800
Subject: [PATCH 2382/3084] Fold 'ifcmp_imm' + 'brif' where imm is zero and
 cond is 'eq' or 'ne', into 'brz' or 'brnz'.

---
 cranelift/codegen/src/simple_preopt.rs        | 87 ++++++++++++++++++-
 .../filetests/simple_preopt/branch.clif       | 29 +++++++
 2 files changed, 115 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/filetests/filetests/simple_preopt/branch.clif

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index b71d4a31c4..2890cb262c 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -5,8 +5,9 @@
 use crate::cursor::{Cursor, FuncCursor};
 use crate::divconst_magic_numbers::{magic_s32, magic_s64, magic_u32, magic_u64};
 use crate::divconst_magic_numbers::{MS32, MS64, MU32, MU64};
+use crate::ir::condcodes::IntCC;
 use crate::ir::dfg::ValueDef;
-use crate::ir::instructions::Opcode;
+use crate::ir::instructions::{Opcode, ValueList};
 use crate::ir::types::{I32, I64};
 use crate::ir::Inst;
 use crate::ir::{DataFlowGraph, Function, InstBuilder, InstructionData, Type, Value};
@@ -536,6 +537,88 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
     }
 }
 
+struct BranchOptInfo {
+    br_inst: Inst,
+    cmp_arg: Value,
+    destination: Ebb,
+    args: ValueList,
+    kind: BranchOptKind,
+}
+
+enum BranchOptKind {
+    EqualZero,
+    NotEqualZero,
+}
+
+fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
+    let info = match pos.func.dfg[inst] {
+        InstructionData::BranchInt {
+            opcode: Opcode::Brif,
+            cond: br_cond,
+            destination,
+            ref args,
+        } => {
+            let first_arg = {
+                let args = pos.func.dfg.inst_args(inst);
+                args[0]
+            };
+            if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(first_arg) {
+                if let InstructionData::BinaryImm {
+                    opcode: Opcode::IfcmpImm,
+                    imm: cmp_imm,
+                    arg: cmp_arg,
+                } = pos.func.dfg[iconst_inst]
+                {
+                    let cmp_imm: i64 = cmp_imm.into();
+                    if cmp_imm != 0 {
+                        return;
+                    }
+
+                    match br_cond {
+                        IntCC::NotEqual => BranchOptInfo {
+                            br_inst: inst,
+                            cmp_arg: cmp_arg,
+                            destination: destination,
+                            args: args.clone(),
+                            kind: BranchOptKind::NotEqualZero,
+                        },
+                        IntCC::Equal => BranchOptInfo {
+                            br_inst: inst,
+                            cmp_arg: cmp_arg,
+                            destination: destination,
+                            args: args.clone(),
+                            kind: BranchOptKind::EqualZero,
+                        },
+                        _ => return,
+                    }
+                } else {
+                    return;
+                }
+            } else {
+                return;
+            }
+        }
+        _ => return,
+    };
+
+    match info.kind {
+        BranchOptKind::EqualZero => {
+            let args = info.args.as_slice(&pos.func.dfg.value_lists)[1..].to_vec();
+            pos.func
+                .dfg
+                .replace(info.br_inst)
+                .brz(info.cmp_arg, info.destination, &args);
+        }
+        BranchOptKind::NotEqualZero => {
+            let args = info.args.as_slice(&pos.func.dfg.value_lists)[1..].to_vec();
+            pos.func
+                .dfg
+                .replace(info.br_inst)
+                .brnz(info.cmp_arg, info.destination, &args);
+        }
+    }
+}
+
 /// The main pre-opt pass.
 pub fn do_preopt(func: &mut Function) {
     let _tt = timing::preopt();
@@ -554,6 +637,8 @@ pub fn do_preopt(func: &mut Function) {
             }
 
             //-- END -- division by constants ------------------
+
+            branch_opt(&mut pos, inst);
         }
     }
 }
diff --git a/cranelift/filetests/filetests/simple_preopt/branch.clif b/cranelift/filetests/filetests/simple_preopt/branch.clif
new file mode 100644
index 0000000000..4701052e7f
--- /dev/null
+++ b/cranelift/filetests/filetests/simple_preopt/branch.clif
@@ -0,0 +1,29 @@
+test simple_preopt
+target x86_64
+
+function %brif_to_brz_fold(i32) -> i32 {
+ebb0(v0: i32):
+    v1 = ifcmp_imm v0, 0
+    brif eq v1, ebb1
+    jump ebb2
+ebb1:
+    v2 = iconst.i32 1
+    return v2
+ebb2:
+    v3 = iconst.i32 2
+    return v3
+}
+; sameln: function %brif_to_brz_fold
+; nextln: ebb0(v0: i32):
+; nextln:     v1 = ifcmp_imm v0, 0
+; nextln:     brz v0, ebb1
+; nextln:     jump ebb2
+; nextln: 
+; nextln: ebb1:
+; nextln:     v2 = iconst.i32 1
+; nextln:     return v2
+; nextln: 
+; nextln: ebb2:
+; nextln:     v3 = iconst.i32 2
+; nextln:     return v3
+; nextln: }

From 57f087c9da584c68452d0394ddbfa2f3385fe2ce Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Tue, 18 Dec 2018 12:38:18 -0800
Subject: [PATCH 2383/3084] Add note to simple_preopt differentiating its use
 from the preopt crate.

---
 cranelift/codegen/src/simple_preopt.rs | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index 2890cb262c..640d866919 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -1,4 +1,8 @@
 //! A pre-legalization rewriting pass.
+//!
+//! This module provides early-stage optimizations. The optimizations found
+//! should be useful for already well-optimized code. More general purpose
+//! early-stage optimizations can be found in the preopt crate.
 
 #![allow(non_snake_case)]
 

From 947130be81c6667e6954d14fe3e6c5af308a1db5 Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Thu, 20 Dec 2018 16:36:49 -0800
Subject: [PATCH 2384/3084] Add branch order optimization to simple_preopt to
 encourage fallthroughs.

---
 cranelift/codegen/src/context.rs       |   2 +-
 cranelift/codegen/src/simple_preopt.rs | 193 ++++++++++++++++++++++++-
 2 files changed, 191 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index 67e676581f..0cd1564c06 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -225,7 +225,7 @@ impl Context {
 
     /// Perform pre-legalization rewrites on the function.
     pub fn preopt(&mut self, isa: &TargetIsa) -> CodegenResult<()> {
-        do_preopt(&mut self.func);
+        do_preopt(&mut self.func, &mut self.cfg);
         self.verify_if(isa)?;
         Ok(())
     }
diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index 640d866919..35ee878ce6 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -9,7 +9,7 @@
 use crate::cursor::{Cursor, FuncCursor};
 use crate::divconst_magic_numbers::{magic_s32, magic_s64, magic_u32, magic_u64};
 use crate::divconst_magic_numbers::{MS32, MS64, MU32, MU64};
-use crate::ir::condcodes::IntCC;
+use crate::ir::condcodes::{CondCode, FloatCC, IntCC};
 use crate::ir::dfg::ValueDef;
 use crate::ir::instructions::{Opcode, ValueList};
 use crate::ir::types::{I32, I64};
@@ -623,11 +623,197 @@ fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
     }
 }
 
+struct BranchOrderInfo {
+    term_inst: Inst,
+    term_inst_args: ValueList,
+    term_dest: Ebb,
+    cond_inst: Inst,
+    cond_arg: Value,
+    cond_inst_args: ValueList,
+    cond_dest: Ebb,
+    kind: BranchOrderKind,
+}
+
+enum BranchOrderKind {
+    BrzToBrnz,
+    BrnzToBrz,
+    InvertIntCond(IntCC),
+    InvertFloatCond(FloatCC),
+}
+
+fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst: Inst) {
+    let info = match pos.func.dfg[inst] {
+        InstructionData::Jump {
+            opcode: Opcode::Jump,
+            destination,
+            ref args,
+        } => {
+            if let Some(next_ebb) = pos.func.layout.next_ebb(ebb) {
+                if destination == next_ebb {
+                    return;
+                }
+
+                if let Some(prev_inst) = pos.func.layout.prev_inst(inst) {
+                    let prev_inst_data = &pos.func.dfg[prev_inst];
+                    if !prev_inst_data.opcode().is_branch() {
+                        return;
+                    }
+
+                    if let Some(prev_dest) = prev_inst_data.branch_destination() {
+                        if prev_dest != next_ebb {
+                            return;
+                        }
+
+                        match prev_inst_data {
+                            InstructionData::Branch {
+                                opcode,
+                                args: ref prev_args,
+                                destination: cond_dest,
+                                ..
+                            } => {
+                                let cond_arg = {
+                                    let args = pos.func.dfg.inst_args(prev_inst);
+                                    args[0]
+                                };
+
+                                match opcode {
+                                    Opcode::Brz => BranchOrderInfo {
+                                        term_inst: inst,
+                                        term_inst_args: args.clone(),
+                                        term_dest: destination,
+                                        cond_inst: prev_inst,
+                                        cond_arg: cond_arg,
+                                        cond_inst_args: prev_args.clone(),
+                                        cond_dest: *cond_dest,
+                                        kind: BranchOrderKind::BrzToBrnz,
+                                    },
+                                    Opcode::Brnz => BranchOrderInfo {
+                                        term_inst: inst,
+                                        term_inst_args: args.clone(),
+                                        term_dest: destination,
+                                        cond_inst: prev_inst,
+                                        cond_arg: cond_arg,
+                                        cond_inst_args: prev_args.clone(),
+                                        cond_dest: *cond_dest,
+                                        kind: BranchOrderKind::BrnzToBrz,
+                                    },
+                                    _ => panic!("unexpected opcode"),
+                                }
+                            }
+                            InstructionData::BranchInt {
+                                opcode: Opcode::Brif,
+                                args: ref prev_args,
+                                cond,
+                                destination: cond_dest,
+                                ..
+                            } => {
+                                let cond_arg = {
+                                    let args = pos.func.dfg.inst_args(prev_inst);
+                                    args[0]
+                                };
+                                BranchOrderInfo {
+                                    term_inst: inst,
+                                    term_inst_args: args.clone(),
+                                    term_dest: destination,
+                                    cond_inst: prev_inst,
+                                    cond_arg: cond_arg,
+                                    cond_inst_args: prev_args.clone(),
+                                    cond_dest: *cond_dest,
+                                    kind: BranchOrderKind::InvertIntCond(*cond),
+                                }
+                            }
+                            InstructionData::BranchFloat {
+                                opcode: Opcode::Brff,
+                                args: ref prev_args,
+                                cond,
+                                destination: cond_dest,
+                                ..
+                            } => {
+                                let cond_arg = {
+                                    let args = pos.func.dfg.inst_args(prev_inst);
+                                    args[0]
+                                };
+                                BranchOrderInfo {
+                                    term_inst: inst,
+                                    term_inst_args: args.clone(),
+                                    term_dest: destination,
+                                    cond_inst: prev_inst,
+                                    cond_arg: cond_arg,
+                                    cond_inst_args: prev_args.clone(),
+                                    cond_dest: *cond_dest,
+                                    kind: BranchOrderKind::InvertFloatCond(*cond),
+                                }
+                            }
+                            _ => return,
+                        }
+                    } else {
+                        return;
+                    }
+                } else {
+                    return;
+                }
+            } else {
+                return;
+            }
+        }
+        _ => return,
+    };
+
+    let cond_args = {
+        info.cond_inst_args
+            .as_slice(&pos.func.dfg.value_lists)
+            .to_vec()
+    };
+    let term_args = {
+        info.term_inst_args
+            .as_slice(&pos.func.dfg.value_lists)
+            .to_vec()
+    };
+
+    pos.func
+        .dfg
+        .replace(info.term_inst)
+        .fallthrough(info.cond_dest, &cond_args[1..]);
+
+    match info.kind {
+        BranchOrderKind::BrnzToBrz => {
+            pos.func
+                .dfg
+                .replace(info.cond_inst)
+                .brz(info.cond_arg, info.term_dest, &term_args);
+        }
+        BranchOrderKind::BrzToBrnz => {
+            pos.func
+                .dfg
+                .replace(info.cond_inst)
+                .brnz(info.cond_arg, info.term_dest, &term_args);
+        }
+        BranchOrderKind::InvertIntCond(cond) => {
+            pos.func.dfg.replace(info.cond_inst).brif(
+                cond.inverse(),
+                info.cond_arg,
+                info.term_dest,
+                &term_args,
+            );
+        }
+        BranchOrderKind::InvertFloatCond(cond) => {
+            pos.func.dfg.replace(info.cond_inst).brff(
+                cond.inverse(),
+                info.cond_arg,
+                info.term_dest,
+                &term_args,
+            );
+        }
+    }
+
+    cfg.recompute_ebb(pos.func, ebb);
+}
+
 /// The main pre-opt pass.
-pub fn do_preopt(func: &mut Function) {
+pub fn do_preopt(func: &mut Function, cfg: &mut ControlFlowGraph) {
     let _tt = timing::preopt();
     let mut pos = FuncCursor::new(func);
-    while let Some(_ebb) = pos.next_ebb() {
+    while let Some(ebb) = pos.next_ebb() {
         while let Some(inst) = pos.next_inst() {
             // Apply basic simplifications.
             simplify(&mut pos, inst);
@@ -643,6 +829,7 @@ pub fn do_preopt(func: &mut Function) {
             //-- END -- division by constants ------------------
 
             branch_opt(&mut pos, inst);
+            branch_order(&mut pos, cfg, ebb, inst);
         }
     }
 }

From 89a2dd9414bba9a99f62df222159e8be5f8f4fa3 Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Thu, 20 Dec 2018 16:37:28 -0800
Subject: [PATCH 2385/3084] Add tests for branch order and branch comparison
 folding.

---
 .../filetests/simple_preopt/branch.clif       | 58 ++++++++++++++++++-
 .../filetests/simple_preopt/simplify.clif     |  4 +-
 2 files changed, 58 insertions(+), 4 deletions(-)

diff --git a/cranelift/filetests/filetests/simple_preopt/branch.clif b/cranelift/filetests/filetests/simple_preopt/branch.clif
index 4701052e7f..b0da32807e 100644
--- a/cranelift/filetests/filetests/simple_preopt/branch.clif
+++ b/cranelift/filetests/filetests/simple_preopt/branch.clif
@@ -16,8 +16,8 @@ ebb2:
 ; sameln: function %brif_to_brz_fold
 ; nextln: ebb0(v0: i32):
 ; nextln:     v1 = ifcmp_imm v0, 0
-; nextln:     brz v0, ebb1
-; nextln:     jump ebb2
+; nextln:     brnz v0, ebb2
+; nextln:     fallthrough ebb1
 ; nextln: 
 ; nextln: ebb1:
 ; nextln:     v2 = iconst.i32 1
@@ -27,3 +27,57 @@ ebb2:
 ; nextln:     v3 = iconst.i32 2
 ; nextln:     return v3
 ; nextln: }
+
+function %brif_inversion(i32) -> i32 {
+ebb0(v0: i32):
+    v1 = ifcmp_imm v0, 42
+    brif ugt v1, ebb1
+    jump ebb2
+ebb1:
+    v2 = iconst.i32 1
+    return v2
+ebb2:
+    v3 = iconst.i32 2
+    return v3
+}
+; sameln: function %brif_inversion
+; nextln: ebb0(v0: i32):
+; nextln:     v1 = ifcmp_imm v0, 42
+; nextln:     brif ule v1, ebb2
+; nextln:     fallthrough ebb1
+; nextln: 
+; nextln: ebb1:
+; nextln:     v2 = iconst.i32 1
+; nextln:     return v2
+; nextln: 
+; nextln: ebb2:
+; nextln:     v3 = iconst.i32 2
+; nextln:     return v3
+; nextln: }
+
+function %brff_inversion(f32, f32) -> i32 {
+ebb0(v0: f32, v1: f32):
+    v2 = ffcmp v0, v1
+    brff gt v2, ebb1
+    jump ebb2
+ebb1:
+    v3 = iconst.i32 1
+    return v3
+ebb2:
+    v4 = iconst.i32 2
+    return v4
+}
+; sameln: function %brff_inversion
+; nextln: ebb0(v0: f32, v1: f32):
+; nextln:     v2 = ffcmp v0, v1
+; nextln:     brff ule v2, ebb2
+; nextln:     fallthrough ebb1
+; nextln: 
+; nextln: ebb1:
+; nextln:     v3 = iconst.i32 1
+; nextln:     return v3
+; nextln: 
+; nextln: ebb2:
+; nextln:     v4 = iconst.i32 2
+; nextln:     return v4
+; nextln: }
diff --git a/cranelift/filetests/filetests/simple_preopt/simplify.clif b/cranelift/filetests/filetests/simple_preopt/simplify.clif
index 3df0f3355e..05bbbc59cc 100644
--- a/cranelift/filetests/filetests/simple_preopt/simplify.clif
+++ b/cranelift/filetests/filetests/simple_preopt/simplify.clif
@@ -63,8 +63,8 @@ ebb2:
 ; nextln:    v1 = bint.i32 v3
 ; nextln:    v2 = select v3, v1, v1
 ; nextln:    trapz v3, user0
-; nextln:    brz v3, ebb1
-; nextln:    jump ebb2
+; nextln:    brnz v3, ebb2
+; nextln:    fallthrough ebb1
 
 function %irsub_imm(i32) -> i32 {
 ebb0(v0: i32):

From 861ef3a2e56bb22f6900fcae4130900dd955cba6 Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Thu, 20 Dec 2018 16:38:27 -0800
Subject: [PATCH 2386/3084] Questionable change: Remove domtree generation from
 simple_preopt testing.

---
 cranelift/filetests/src/test_simple_preopt.rs | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/cranelift/filetests/src/test_simple_preopt.rs b/cranelift/filetests/src/test_simple_preopt.rs
index 8b2ea6999c..4d2e7a7414 100644
--- a/cranelift/filetests/src/test_simple_preopt.rs
+++ b/cranelift/filetests/src/test_simple_preopt.rs
@@ -33,11 +33,10 @@ impl SubTest for TestSimplePreopt {
         let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned());
         let isa = context.isa.expect("preopt needs an ISA");
 
-        comp_ctx.flowgraph();
+        comp_ctx.compute_cfg();
         comp_ctx
             .preopt(isa)
             .map_err(|e| pretty_error(&comp_ctx.func, context.isa, Into::into(e)))?;
-
         let text = &comp_ctx.func.display(isa).to_string();
         run_filecheck(&text, context)
     }

From 67cc5aafec542d5236ab2e2d7c75e51ff6358b24 Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Wed, 2 Jan 2019 11:51:58 -0800
Subject: [PATCH 2387/3084] Refactoring of branch ordering and zero-check
 optimization based on PR feedback.

---
 cranelift/codegen/src/simple_preopt.rs | 248 ++++++++++++-------------
 1 file changed, 121 insertions(+), 127 deletions(-)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index 35ee878ce6..eeaa3c8d91 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -566,37 +566,36 @@ fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
                 let args = pos.func.dfg.inst_args(inst);
                 args[0]
             };
-            if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(first_arg) {
-                if let InstructionData::BinaryImm {
-                    opcode: Opcode::IfcmpImm,
-                    imm: cmp_imm,
-                    arg: cmp_arg,
-                } = pos.func.dfg[iconst_inst]
-                {
-                    let cmp_imm: i64 = cmp_imm.into();
-                    if cmp_imm != 0 {
-                        return;
-                    }
-
-                    match br_cond {
-                        IntCC::NotEqual => BranchOptInfo {
-                            br_inst: inst,
-                            cmp_arg: cmp_arg,
-                            destination: destination,
-                            args: args.clone(),
-                            kind: BranchOptKind::NotEqualZero,
-                        },
-                        IntCC::Equal => BranchOptInfo {
-                            br_inst: inst,
-                            cmp_arg: cmp_arg,
-                            destination: destination,
-                            args: args.clone(),
-                            kind: BranchOptKind::EqualZero,
-                        },
-                        _ => return,
-                    }
+            let iconst_inst =
+                if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(first_arg) {
+                    iconst_inst
                 } else {
                     return;
+                };
+
+            if let InstructionData::BinaryImm {
+                opcode: Opcode::IfcmpImm,
+                imm: cmp_imm,
+                arg: cmp_arg,
+            } = pos.func.dfg[iconst_inst]
+            {
+                let cmp_imm: i64 = cmp_imm.into();
+                if cmp_imm != 0 {
+                    return;
+                }
+
+                let kind = match br_cond {
+                    IntCC::NotEqual => BranchOptKind::NotEqualZero,
+                    IntCC::Equal => BranchOptKind::EqualZero,
+                    _ => return,
+                };
+
+                BranchOptInfo {
+                    br_inst: inst,
+                    cmp_arg: cmp_arg,
+                    destination: destination,
+                    args: args.clone(),
+                    kind: kind,
                 }
             } else {
                 return;
@@ -648,114 +647,109 @@ fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst
             destination,
             ref args,
         } => {
-            if let Some(next_ebb) = pos.func.layout.next_ebb(ebb) {
-                if destination == next_ebb {
-                    return;
-                }
+            let next_ebb = if let Some(next_ebb) = pos.func.layout.next_ebb(ebb) {
+                next_ebb
+            } else {
+                return;
+            };
 
-                if let Some(prev_inst) = pos.func.layout.prev_inst(inst) {
-                    let prev_inst_data = &pos.func.dfg[prev_inst];
-                    if !prev_inst_data.opcode().is_branch() {
-                        return;
-                    }
+            if destination == next_ebb {
+                return;
+            }
 
-                    if let Some(prev_dest) = prev_inst_data.branch_destination() {
-                        if prev_dest != next_ebb {
-                            return;
-                        }
+            let prev_inst = if let Some(prev_inst) = pos.func.layout.prev_inst(inst) {
+                prev_inst
+            } else {
+                return;
+            };
 
-                        match prev_inst_data {
-                            InstructionData::Branch {
-                                opcode,
-                                args: ref prev_args,
-                                destination: cond_dest,
-                                ..
-                            } => {
-                                let cond_arg = {
-                                    let args = pos.func.dfg.inst_args(prev_inst);
-                                    args[0]
-                                };
+            let prev_inst_data = &pos.func.dfg[prev_inst];
 
-                                match opcode {
-                                    Opcode::Brz => BranchOrderInfo {
-                                        term_inst: inst,
-                                        term_inst_args: args.clone(),
-                                        term_dest: destination,
-                                        cond_inst: prev_inst,
-                                        cond_arg: cond_arg,
-                                        cond_inst_args: prev_args.clone(),
-                                        cond_dest: *cond_dest,
-                                        kind: BranchOrderKind::BrzToBrnz,
-                                    },
-                                    Opcode::Brnz => BranchOrderInfo {
-                                        term_inst: inst,
-                                        term_inst_args: args.clone(),
-                                        term_dest: destination,
-                                        cond_inst: prev_inst,
-                                        cond_arg: cond_arg,
-                                        cond_inst_args: prev_args.clone(),
-                                        cond_dest: *cond_dest,
-                                        kind: BranchOrderKind::BrnzToBrz,
-                                    },
-                                    _ => panic!("unexpected opcode"),
-                                }
-                            }
-                            InstructionData::BranchInt {
-                                opcode: Opcode::Brif,
-                                args: ref prev_args,
-                                cond,
-                                destination: cond_dest,
-                                ..
-                            } => {
-                                let cond_arg = {
-                                    let args = pos.func.dfg.inst_args(prev_inst);
-                                    args[0]
-                                };
-                                BranchOrderInfo {
-                                    term_inst: inst,
-                                    term_inst_args: args.clone(),
-                                    term_dest: destination,
-                                    cond_inst: prev_inst,
-                                    cond_arg: cond_arg,
-                                    cond_inst_args: prev_args.clone(),
-                                    cond_dest: *cond_dest,
-                                    kind: BranchOrderKind::InvertIntCond(*cond),
-                                }
-                            }
-                            InstructionData::BranchFloat {
-                                opcode: Opcode::Brff,
-                                args: ref prev_args,
-                                cond,
-                                destination: cond_dest,
-                                ..
-                            } => {
-                                let cond_arg = {
-                                    let args = pos.func.dfg.inst_args(prev_inst);
-                                    args[0]
-                                };
-                                BranchOrderInfo {
-                                    term_inst: inst,
-                                    term_inst_args: args.clone(),
-                                    term_dest: destination,
-                                    cond_inst: prev_inst,
-                                    cond_arg: cond_arg,
-                                    cond_inst_args: prev_args.clone(),
-                                    cond_dest: *cond_dest,
-                                    kind: BranchOrderKind::InvertFloatCond(*cond),
-                                }
-                            }
-                            _ => return,
-                        }
-                    } else {
-                        return;
-                    }
-                } else {
+            if let Some(prev_dest) = prev_inst_data.branch_destination() {
+                if prev_dest != next_ebb {
                     return;
                 }
             } else {
                 return;
             }
+
+            match prev_inst_data {
+                InstructionData::Branch {
+                    opcode,
+                    args: ref prev_args,
+                    destination: cond_dest,
+                    ..
+                } => {
+                    let cond_arg = {
+                        let args = pos.func.dfg.inst_args(prev_inst);
+                        args[0]
+                    };
+
+                    let kind = match opcode {
+                        Opcode::Brz => BranchOrderKind::BrzToBrnz,
+                        Opcode::Brnz => BranchOrderKind::BrnzToBrz,
+                        _ => panic!("unexpected opcode"),
+                    };
+
+                    BranchOrderInfo {
+                        term_inst: inst,
+                        term_inst_args: args.clone(),
+                        term_dest: destination,
+                        cond_inst: prev_inst,
+                        cond_arg: cond_arg,
+                        cond_inst_args: prev_args.clone(),
+                        cond_dest: *cond_dest,
+                        kind: kind,
+                    }
+                }
+                InstructionData::BranchInt {
+                    opcode: Opcode::Brif,
+                    args: ref prev_args,
+                    cond,
+                    destination: cond_dest,
+                    ..
+                } => {
+                    let cond_arg = {
+                        let args = pos.func.dfg.inst_args(prev_inst);
+                        args[0]
+                    };
+                    BranchOrderInfo {
+                        term_inst: inst,
+                        term_inst_args: args.clone(),
+                        term_dest: destination,
+                        cond_inst: prev_inst,
+                        cond_arg: cond_arg,
+                        cond_inst_args: prev_args.clone(),
+                        cond_dest: *cond_dest,
+                        kind: BranchOrderKind::InvertIntCond(*cond),
+                    }
+                }
+                InstructionData::BranchFloat {
+                    opcode: Opcode::Brff,
+                    args: ref prev_args,
+                    cond,
+                    destination: cond_dest,
+                    ..
+                } => {
+                    let cond_arg = {
+                        let args = pos.func.dfg.inst_args(prev_inst);
+                        args[0]
+                    };
+                    BranchOrderInfo {
+                        term_inst: inst,
+                        term_inst_args: args.clone(),
+                        term_dest: destination,
+                        cond_inst: prev_inst,
+                        cond_arg: cond_arg,
+                        cond_inst_args: prev_args.clone(),
+                        cond_dest: *cond_dest,
+                        kind: BranchOrderKind::InvertFloatCond(*cond),
+                    }
+                }
+                _ => return,
+            }
         }
+
         _ => return,
     };
 

From 571b87414f4590372ee8fd84637987425ade353e Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Wed, 2 Jan 2019 20:01:52 -0800
Subject: [PATCH 2388/3084] Switch ifcmp+brif folding to be icmp+brz folding,
 which is what frontends actually produce.

---
 cranelift/codegen/src/simple_preopt.rs        | 25 ++++++---
 .../filetests/simple_preopt/branch.clif       | 51 ++++++++++++++-----
 2 files changed, 56 insertions(+), 20 deletions(-)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index eeaa3c8d91..4d6f31c654 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -556,9 +556,8 @@ enum BranchOptKind {
 
 fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
     let info = match pos.func.dfg[inst] {
-        InstructionData::BranchInt {
-            opcode: Opcode::Brif,
-            cond: br_cond,
+        InstructionData::Branch {
+            opcode,
             destination,
             ref args,
         } => {
@@ -566,6 +565,7 @@ fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
                 let args = pos.func.dfg.inst_args(inst);
                 args[0]
             };
+
             let iconst_inst =
                 if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(first_arg) {
                     iconst_inst
@@ -573,10 +573,11 @@ fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
                     return;
                 };
 
-            if let InstructionData::BinaryImm {
-                opcode: Opcode::IfcmpImm,
-                imm: cmp_imm,
+            if let InstructionData::IntCompareImm {
+                opcode: Opcode::IcmpImm,
                 arg: cmp_arg,
+                cond: cmp_cond,
+                imm: cmp_imm,
             } = pos.func.dfg[iconst_inst]
             {
                 let cmp_imm: i64 = cmp_imm.into();
@@ -584,9 +585,17 @@ fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
                     return;
                 }
 
-                let kind = match br_cond {
-                    IntCC::NotEqual => BranchOptKind::NotEqualZero,
+                // icmp_imm returns non-zero when the comparison is true. So, if
+                // we're branching on zero, we need to invert the condition.
+                let cond = match opcode {
+                    Opcode::Brz => cmp_cond.inverse(),
+                    Opcode::Brnz => cmp_cond,
+                    _ => return,
+                };
+
+                let kind = match cond {
                     IntCC::Equal => BranchOptKind::EqualZero,
+                    IntCC::NotEqual => BranchOptKind::NotEqualZero,
                     _ => return,
                 };
 
diff --git a/cranelift/filetests/filetests/simple_preopt/branch.clif b/cranelift/filetests/filetests/simple_preopt/branch.clif
index b0da32807e..6e5f3dcd52 100644
--- a/cranelift/filetests/filetests/simple_preopt/branch.clif
+++ b/cranelift/filetests/filetests/simple_preopt/branch.clif
@@ -1,31 +1,58 @@
 test simple_preopt
 target x86_64
 
-function %brif_to_brz_fold(i32) -> i32 {
+function %icmp_to_brz_fold(i32) -> i32 {
 ebb0(v0: i32):
-    v1 = ifcmp_imm v0, 0
-    brif eq v1, ebb1
+    v1 = icmp_imm eq v0, 0
+    brnz v1, ebb1
     jump ebb2
 ebb1:
-    v2 = iconst.i32 1
-    return v2
-ebb2:
-    v3 = iconst.i32 2
+    v3 = iconst.i32 1
     return v3
+ebb2:
+    v4 = iconst.i32 2
+    return v4
 }
-; sameln: function %brif_to_brz_fold
+; sameln: function %icmp_to_brz_fold
 ; nextln: ebb0(v0: i32):
-; nextln:     v1 = ifcmp_imm v0, 0
+; nextln:     v1 = icmp_imm eq v0, 0
 ; nextln:     brnz v0, ebb2
 ; nextln:     fallthrough ebb1
 ; nextln: 
 ; nextln: ebb1:
-; nextln:     v2 = iconst.i32 1
-; nextln:     return v2
+; nextln:     v3 = iconst.i32 1
+; nextln:     return v3
 ; nextln: 
 ; nextln: ebb2:
-; nextln:     v3 = iconst.i32 2
+; nextln:     v4 = iconst.i32 2
+; nextln:     return v4
+; nextln: }
+
+function %icmp_to_brz_inverted_fold(i32) -> i32 {
+ebb0(v0: i32):
+    v1 = icmp_imm ne v0, 0
+    brz v1, ebb1
+    jump ebb2
+ebb1:
+    v3 = iconst.i32 1
+    return v3
+ebb2:
+    v4 = iconst.i32 2
+    return v4
+}
+; sameln: function %icmp_to_brz_inve
+; nextln: ebb0(v0: i32):
+; nextln:     v1 = icmp_imm ne v0, 0
+; nextln:     brnz v0, ebb2
+; nextln:     fallthrough ebb1
+; nextln: 
+; nextln: ebb1:
+; nextln:     v3 = iconst.i32 1
 ; nextln:     return v3
+; nextln: 
+; nextln: ebb2:
+; nextln:     v4 = iconst.i32 2
+; nextln:     return v4
 ; nextln: }
 
 function %brif_inversion(i32) -> i32 {

From fd6940baaf3ed2d22b1bd912bddf2599a21c609e Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Wed, 2 Jan 2019 20:24:50 -0800
Subject: [PATCH 2389/3084] Switch branch order opt from brif/brff to br_icmp,
 as that's what frontends should produce.

---
 cranelift/codegen/src/simple_preopt.rs        | 94 +++++++------------
 .../filetests/simple_preopt/branch.clif       | 41 ++------
 2 files changed, 40 insertions(+), 95 deletions(-)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index 4d6f31c654..bee1d6e209 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -9,12 +9,13 @@
 use crate::cursor::{Cursor, FuncCursor};
 use crate::divconst_magic_numbers::{magic_s32, magic_s64, magic_u32, magic_u64};
 use crate::divconst_magic_numbers::{MS32, MS64, MU32, MU64};
-use crate::ir::condcodes::{CondCode, FloatCC, IntCC};
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir::condcodes::{CondCode, IntCC};
 use crate::ir::dfg::ValueDef;
 use crate::ir::instructions::{Opcode, ValueList};
 use crate::ir::types::{I32, I64};
 use crate::ir::Inst;
-use crate::ir::{DataFlowGraph, Function, InstBuilder, InstructionData, Type, Value};
+use crate::ir::{DataFlowGraph, Ebb, Function, InstBuilder, InstructionData, Type, Value};
 use crate::timing;
 
 //----------------------------------------------------------------------
@@ -636,17 +637,15 @@ struct BranchOrderInfo {
     term_inst_args: ValueList,
     term_dest: Ebb,
     cond_inst: Inst,
-    cond_arg: Value,
     cond_inst_args: ValueList,
     cond_dest: Ebb,
     kind: BranchOrderKind,
 }
 
 enum BranchOrderKind {
-    BrzToBrnz,
-    BrnzToBrz,
-    InvertIntCond(IntCC),
-    InvertFloatCond(FloatCC),
+    BrzToBrnz(Value),
+    BrnzToBrz(Value),
+    InvertIcmpCond(IntCC, Value, Value),
 }
 
 fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst: Inst) {
@@ -695,8 +694,8 @@ fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst
                     };
 
                     let kind = match opcode {
-                        Opcode::Brz => BranchOrderKind::BrzToBrnz,
-                        Opcode::Brnz => BranchOrderKind::BrnzToBrz,
+                        Opcode::Brz => BranchOrderKind::BrzToBrnz(cond_arg),
+                        Opcode::Brnz => BranchOrderKind::BrnzToBrz(cond_arg),
                         _ => panic!("unexpected opcode"),
                     };
 
@@ -705,54 +704,29 @@ fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst
                         term_inst_args: args.clone(),
                         term_dest: destination,
                         cond_inst: prev_inst,
-                        cond_arg: cond_arg,
                         cond_inst_args: prev_args.clone(),
                         cond_dest: *cond_dest,
                         kind: kind,
                     }
                 }
-                InstructionData::BranchInt {
-                    opcode: Opcode::Brif,
-                    args: ref prev_args,
+                InstructionData::BranchIcmp {
+                    opcode: Opcode::BrIcmp,
                     cond,
                     destination: cond_dest,
-                    ..
+                    args: ref prev_args,
                 } => {
-                    let cond_arg = {
+                    let (x_arg, y_arg) = {
                         let args = pos.func.dfg.inst_args(prev_inst);
-                        args[0]
+                        (args[0], args[1])
                     };
                     BranchOrderInfo {
                         term_inst: inst,
                         term_inst_args: args.clone(),
                         term_dest: destination,
                         cond_inst: prev_inst,
-                        cond_arg: cond_arg,
                         cond_inst_args: prev_args.clone(),
                         cond_dest: *cond_dest,
-                        kind: BranchOrderKind::InvertIntCond(*cond),
-                    }
-                }
-                InstructionData::BranchFloat {
-                    opcode: Opcode::Brff,
-                    args: ref prev_args,
-                    cond,
-                    destination: cond_dest,
-                    ..
-                } => {
-                    let cond_arg = {
-                        let args = pos.func.dfg.inst_args(prev_inst);
-                        args[0]
-                    };
-                    BranchOrderInfo {
-                        term_inst: inst,
-                        term_inst_args: args.clone(),
-                        term_dest: destination,
-                        cond_inst: prev_inst,
-                        cond_arg: cond_arg,
-                        cond_inst_args: prev_args.clone(),
-                        cond_dest: *cond_dest,
-                        kind: BranchOrderKind::InvertFloatCond(*cond),
+                        kind: BranchOrderKind::InvertIcmpCond(*cond, x_arg, y_arg),
                     }
                 }
                 _ => return,
@@ -773,36 +747,36 @@ fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst
             .to_vec()
     };
 
-    pos.func
-        .dfg
-        .replace(info.term_inst)
-        .fallthrough(info.cond_dest, &cond_args[1..]);
-
     match info.kind {
-        BranchOrderKind::BrnzToBrz => {
+        BranchOrderKind::BrnzToBrz(cond_arg) => {
+            pos.func
+                .dfg
+                .replace(info.term_inst)
+                .fallthrough(info.cond_dest, &cond_args[1..]);
             pos.func
                 .dfg
                 .replace(info.cond_inst)
-                .brz(info.cond_arg, info.term_dest, &term_args);
+                .brz(cond_arg, info.term_dest, &term_args);
         }
-        BranchOrderKind::BrzToBrnz => {
+        BranchOrderKind::BrzToBrnz(cond_arg) => {
+            pos.func
+                .dfg
+                .replace(info.term_inst)
+                .fallthrough(info.cond_dest, &cond_args[1..]);
             pos.func
                 .dfg
                 .replace(info.cond_inst)
-                .brnz(info.cond_arg, info.term_dest, &term_args);
+                .brnz(cond_arg, info.term_dest, &term_args);
         }
-        BranchOrderKind::InvertIntCond(cond) => {
-            pos.func.dfg.replace(info.cond_inst).brif(
+        BranchOrderKind::InvertIcmpCond(cond, x_arg, y_arg) => {
+            pos.func
+                .dfg
+                .replace(info.term_inst)
+                .fallthrough(info.cond_dest, &cond_args[2..]);
+            pos.func.dfg.replace(info.cond_inst).br_icmp(
                 cond.inverse(),
-                info.cond_arg,
-                info.term_dest,
-                &term_args,
-            );
-        }
-        BranchOrderKind::InvertFloatCond(cond) => {
-            pos.func.dfg.replace(info.cond_inst).brff(
-                cond.inverse(),
-                info.cond_arg,
+                x_arg,
+                y_arg,
                 info.term_dest,
                 &term_args,
             );
diff --git a/cranelift/filetests/filetests/simple_preopt/branch.clif b/cranelift/filetests/filetests/simple_preopt/branch.clif
index 6e5f3dcd52..be1a99ff8c 100644
--- a/cranelift/filetests/filetests/simple_preopt/branch.clif
+++ b/cranelift/filetests/filetests/simple_preopt/branch.clif
@@ -55,10 +55,9 @@ ebb2:
 ; nextln:     return v4
 ; nextln: }
 
-function %brif_inversion(i32) -> i32 {
-ebb0(v0: i32):
-    v1 = ifcmp_imm v0, 42
-    brif ugt v1, ebb1
+function %br_icmp_inversion(i32, i32) -> i32 {
+ebb0(v0: i32, v1: i32):
+    br_icmp ugt v0, v1, ebb1
     jump ebb2
 ebb1:
     v2 = iconst.i32 1
@@ -67,10 +66,9 @@ ebb2:
     v3 = iconst.i32 2
     return v3
 }
-; sameln: function %brif_inversion
-; nextln: ebb0(v0: i32):
-; nextln:     v1 = ifcmp_imm v0, 42
-; nextln:     brif ule v1, ebb2
+; sameln: function %br_icmp_inversio
+; nextln: ebb0(v0: i32, v1: i32):
+; nextln:     br_icmp ule v0, v1, ebb2
 ; nextln:     fallthrough ebb1
 ; nextln: 
 ; nextln: ebb1:
@@ -81,30 +79,3 @@ ebb2:
 ; nextln:     v3 = iconst.i32 2
 ; nextln:     return v3
 ; nextln: }
-
-function %brff_inversion(f32, f32) -> i32 {
-ebb0(v0: f32, v1: f32):
-    v2 = ffcmp v0, v1
-    brff gt v2, ebb1
-    jump ebb2
-ebb1:
-    v3 = iconst.i32 1
-    return v3
-ebb2:
-    v4 = iconst.i32 2
-    return v4
-}
-; sameln: function %brff_inversion
-; nextln: ebb0(v0: f32, v1: f32):
-; nextln:     v2 = ffcmp v0, v1
-; nextln:     brff ule v2, ebb2
-; nextln:     fallthrough ebb1
-; nextln: 
-; nextln: ebb1:
-; nextln:     v3 = iconst.i32 1
-; nextln:     return v3
-; nextln: 
-; nextln: ebb2:
-; nextln:     v4 = iconst.i32 2
-; nextln:     return v4
-; nextln: }

From 402d0d9c83df14c2c11a6fcd15a494ad6cd6c645 Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Thu, 3 Jan 2019 11:56:36 -0800
Subject: [PATCH 2390/3084] Add comments to branch folding and reordering
 functions in simple_preopt.

---
 cranelift/codegen/src/simple_preopt.rs | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index bee1d6e209..ece05e474b 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -555,6 +555,11 @@ enum BranchOptKind {
     NotEqualZero,
 }
 
+/// Fold comparisons into branch operations when possible.
+///
+/// This matches against operations which compare against zero, then use the
+/// result in a `brz` or `brnz` branch. It folds those two operations into a
+/// single `brz` or `brnz`.
 fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
     let info = match pos.func.dfg[inst] {
         InstructionData::Branch {
@@ -648,6 +653,11 @@ enum BranchOrderKind {
     InvertIcmpCond(IntCC, Value, Value),
 }
 
+/// Reorder branches to encourage fallthroughs.
+///
+/// When an ebb ends with a conditional branch followed by an unconditional
+/// branch, this will reorder them if one of them is branching to the next Ebb
+/// layout-wise. The unconditional jump can then become a fallthrough.
 fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst: Inst) {
     let info = match pos.func.dfg[inst] {
         InstructionData::Jump {

From 5596b5fadc6dcba23575ac35f3f90e62aa19f487 Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Wed, 10 Apr 2019 13:41:49 -0700
Subject: [PATCH 2391/3084] Address code review comments, simplifying some bits
 of branch_opt.

---
 cranelift/codegen/src/simple_preopt.rs        | 153 +++++++++---------
 .../filetests/simple_preopt/branch.clif       |   6 +-
 2 files changed, 79 insertions(+), 80 deletions(-)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index ece05e474b..4ec6086656 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -545,14 +545,8 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
 struct BranchOptInfo {
     br_inst: Inst,
     cmp_arg: Value,
-    destination: Ebb,
     args: ValueList,
-    kind: BranchOptKind,
-}
-
-enum BranchOptKind {
-    EqualZero,
-    NotEqualZero,
+    new_opcode: Opcode,
 }
 
 /// Fold comparisons into branch operations when possible.
@@ -561,80 +555,86 @@ enum BranchOptKind {
 /// result in a `brz` or `brnz` branch. It folds those two operations into a
 /// single `brz` or `brnz`.
 fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
-    let info = match pos.func.dfg[inst] {
-        InstructionData::Branch {
-            opcode,
-            destination,
-            ref args,
-        } => {
-            let first_arg = {
-                let args = pos.func.dfg.inst_args(inst);
-                args[0]
-            };
+    let mut info = if let InstructionData::Branch {
+        opcode, ref args, ..
+    } = pos.func.dfg[inst]
+    {
+        let first_arg = {
+            let args = pos.func.dfg.inst_args(inst);
+            args[0]
+        };
 
-            let iconst_inst =
-                if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(first_arg) {
-                    iconst_inst
-                } else {
-                    return;
-                };
-
-            if let InstructionData::IntCompareImm {
-                opcode: Opcode::IcmpImm,
-                arg: cmp_arg,
-                cond: cmp_cond,
-                imm: cmp_imm,
-            } = pos.func.dfg[iconst_inst]
-            {
-                let cmp_imm: i64 = cmp_imm.into();
-                if cmp_imm != 0 {
-                    return;
-                }
-
-                // icmp_imm returns non-zero when the comparison is true. So, if
-                // we're branching on zero, we need to invert the condition.
-                let cond = match opcode {
-                    Opcode::Brz => cmp_cond.inverse(),
-                    Opcode::Brnz => cmp_cond,
-                    _ => return,
-                };
-
-                let kind = match cond {
-                    IntCC::Equal => BranchOptKind::EqualZero,
-                    IntCC::NotEqual => BranchOptKind::NotEqualZero,
-                    _ => return,
-                };
-
-                BranchOptInfo {
-                    br_inst: inst,
-                    cmp_arg: cmp_arg,
-                    destination: destination,
-                    args: args.clone(),
-                    kind: kind,
-                }
+        let iconst_inst =
+            if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(first_arg) {
+                iconst_inst
             } else {
                 return;
+            };
+
+        if let InstructionData::IntCompareImm {
+            opcode: Opcode::IcmpImm,
+            arg: cmp_arg,
+            cond: cmp_cond,
+            imm: cmp_imm,
+        } = pos.func.dfg[iconst_inst]
+        {
+            let cmp_imm: i64 = cmp_imm.into();
+            if cmp_imm != 0 {
+                return;
             }
+
+            // icmp_imm returns non-zero when the comparison is true. So, if
+            // we're branching on zero, we need to invert the condition.
+            let cond = match opcode {
+                Opcode::Brz => cmp_cond.inverse(),
+                Opcode::Brnz => cmp_cond,
+                _ => return,
+            };
+
+            let new_opcode = match cond {
+                IntCC::Equal => Opcode::Brz,
+                IntCC::NotEqual => Opcode::Brnz,
+                _ => return,
+            };
+
+            BranchOptInfo {
+                br_inst: inst,
+                cmp_arg: cmp_arg,
+                args: args.clone(),
+                new_opcode: new_opcode,
+            }
+        } else {
+            return;
         }
-        _ => return,
+    } else {
+        return;
     };
 
-    match info.kind {
-        BranchOptKind::EqualZero => {
-            let args = info.args.as_slice(&pos.func.dfg.value_lists)[1..].to_vec();
-            pos.func
-                .dfg
-                .replace(info.br_inst)
-                .brz(info.cmp_arg, info.destination, &args);
-        }
-        BranchOptKind::NotEqualZero => {
-            let args = info.args.as_slice(&pos.func.dfg.value_lists)[1..].to_vec();
-            pos.func
-                .dfg
-                .replace(info.br_inst)
-                .brnz(info.cmp_arg, info.destination, &args);
-        }
+    info.args.as_mut_slice(&mut pos.func.dfg.value_lists)[0] = info.cmp_arg;
+    if let InstructionData::Branch { ref mut opcode, .. } = pos.func.dfg[info.br_inst] {
+        *opcode = info.new_opcode;
+    } else {
+        panic!();
     }
+
+    /*
+        match info.kind {
+            BranchOptKind::EqualZero => {
+                let args = info.args.as_slice(&pos.func.dfg.value_lists)[1..].to_vec();
+                pos.func
+                    .dfg
+                    .replace(info.br_inst)
+                    .brz(info.cmp_arg, info.destination, &args);
+            }
+            BranchOptKind::NotEqualZero => {
+                let args = info.args.as_slice(&pos.func.dfg.value_lists)[1..].to_vec();
+                pos.func
+                    .dfg
+                    .replace(info.br_inst)
+                    .brnz(info.cmp_arg, info.destination, &args);
+            }
+        }
+    */
 }
 
 struct BranchOrderInfo {
@@ -696,7 +696,6 @@ fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst
                     opcode,
                     args: ref prev_args,
                     destination: cond_dest,
-                    ..
                 } => {
                     let cond_arg = {
                         let args = pos.func.dfg.inst_args(prev_inst);
@@ -762,7 +761,7 @@ fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst
             pos.func
                 .dfg
                 .replace(info.term_inst)
-                .fallthrough(info.cond_dest, &cond_args[1..]);
+                .jump(info.cond_dest, &cond_args[1..]);
             pos.func
                 .dfg
                 .replace(info.cond_inst)
@@ -772,7 +771,7 @@ fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst
             pos.func
                 .dfg
                 .replace(info.term_inst)
-                .fallthrough(info.cond_dest, &cond_args[1..]);
+                .jump(info.cond_dest, &cond_args[1..]);
             pos.func
                 .dfg
                 .replace(info.cond_inst)
@@ -782,7 +781,7 @@ fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst
             pos.func
                 .dfg
                 .replace(info.term_inst)
-                .fallthrough(info.cond_dest, &cond_args[2..]);
+                .jump(info.cond_dest, &cond_args[2..]);
             pos.func.dfg.replace(info.cond_inst).br_icmp(
                 cond.inverse(),
                 x_arg,
diff --git a/cranelift/filetests/filetests/simple_preopt/branch.clif b/cranelift/filetests/filetests/simple_preopt/branch.clif
index be1a99ff8c..f3535b1e70 100644
--- a/cranelift/filetests/filetests/simple_preopt/branch.clif
+++ b/cranelift/filetests/filetests/simple_preopt/branch.clif
@@ -17,7 +17,7 @@ ebb2:
 ; nextln: ebb0(v0: i32):
 ; nextln:     v1 = icmp_imm eq v0, 0
 ; nextln:     brnz v0, ebb2
-; nextln:     fallthrough ebb1
+; nextln:     jump ebb1
 ; nextln: 
 ; nextln: ebb1:
 ; nextln:     v3 = iconst.i32 1
@@ -44,7 +44,7 @@ ebb2:
 ; nextln: ebb0(v0: i32):
 ; nextln:     v1 = icmp_imm ne v0, 0
 ; nextln:     brnz v0, ebb2
-; nextln:     fallthrough ebb1
+; nextln:     jump ebb1
 ; nextln: 
 ; nextln: ebb1:
 ; nextln:     v3 = iconst.i32 1
@@ -69,7 +69,7 @@ ebb2:
 ; sameln: function %br_icmp_inversio
 ; nextln: ebb0(v0: i32, v1: i32):
 ; nextln:     br_icmp ule v0, v1, ebb2
-; nextln:     fallthrough ebb1
+; nextln:     jump ebb1
 ; nextln: 
 ; nextln: ebb1:
 ; nextln:     v2 = iconst.i32 1

From 1090dc5069764b761ee2a2aad908328328f392a7 Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Wed, 10 Apr 2019 14:17:23 -0700
Subject: [PATCH 2392/3084] Switch from fallthrough back to jump.

---
 cranelift/filetests/filetests/simple_preopt/simplify.clif | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/filetests/filetests/simple_preopt/simplify.clif b/cranelift/filetests/filetests/simple_preopt/simplify.clif
index 05bbbc59cc..1592defe2f 100644
--- a/cranelift/filetests/filetests/simple_preopt/simplify.clif
+++ b/cranelift/filetests/filetests/simple_preopt/simplify.clif
@@ -64,7 +64,7 @@ ebb2:
 ; nextln:    v2 = select v3, v1, v1
 ; nextln:    trapz v3, user0
 ; nextln:    brnz v3, ebb2
-; nextln:    fallthrough ebb1
+; nextln:    jump ebb1
 
 function %irsub_imm(i32) -> i32 {
 ebb0(v0: i32):

From 4d427d7c71122b3550374bb326d17d7f95b6d38a Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Sat, 13 Apr 2019 00:15:52 -0700
Subject: [PATCH 2393/3084] Remove old commented out code

---
 cranelift/codegen/src/simple_preopt.rs | 19 -------------------
 1 file changed, 19 deletions(-)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index 4ec6086656..f144a40175 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -616,25 +616,6 @@ fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
     } else {
         panic!();
     }
-
-    /*
-        match info.kind {
-            BranchOptKind::EqualZero => {
-                let args = info.args.as_slice(&pos.func.dfg.value_lists)[1..].to_vec();
-                pos.func
-                    .dfg
-                    .replace(info.br_inst)
-                    .brz(info.cmp_arg, info.destination, &args);
-            }
-            BranchOptKind::NotEqualZero => {
-                let args = info.args.as_slice(&pos.func.dfg.value_lists)[1..].to_vec();
-                pos.func
-                    .dfg
-                    .replace(info.br_inst)
-                    .brnz(info.cmp_arg, info.destination, &args);
-            }
-        }
-    */
 }
 
 struct BranchOrderInfo {

From 3b1583ebb7f5d089a54e0c429778d0816707603a Mon Sep 17 00:00:00 2001
From: Tyler McMullen 
Date: Tue, 16 Apr 2019 10:57:47 +0900
Subject: [PATCH 2394/3084] Style changes in response to code review.

---
 cranelift/codegen/src/simple_preopt.rs | 231 ++++++++++++-------------
 1 file changed, 108 insertions(+), 123 deletions(-)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index f144a40175..e081a5bd43 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -556,7 +556,9 @@ struct BranchOptInfo {
 /// single `brz` or `brnz`.
 fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
     let mut info = if let InstructionData::Branch {
-        opcode, ref args, ..
+        opcode: br_opcode,
+        args: ref br_args,
+        ..
     } = pos.func.dfg[inst]
     {
         let first_arg = {
@@ -564,19 +566,18 @@ fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
             args[0]
         };
 
-        let iconst_inst =
-            if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(first_arg) {
-                iconst_inst
-            } else {
-                return;
-            };
+        let icmp_inst = if let ValueDef::Result(icmp_inst, _) = pos.func.dfg.value_def(first_arg) {
+            icmp_inst
+        } else {
+            return;
+        };
 
         if let InstructionData::IntCompareImm {
             opcode: Opcode::IcmpImm,
             arg: cmp_arg,
             cond: cmp_cond,
             imm: cmp_imm,
-        } = pos.func.dfg[iconst_inst]
+        } = pos.func.dfg[icmp_inst]
         {
             let cmp_imm: i64 = cmp_imm.into();
             if cmp_imm != 0 {
@@ -585,7 +586,7 @@ fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
 
             // icmp_imm returns non-zero when the comparison is true. So, if
             // we're branching on zero, we need to invert the condition.
-            let cond = match opcode {
+            let cond = match br_opcode {
                 Opcode::Brz => cmp_cond.inverse(),
                 Opcode::Brnz => cmp_cond,
                 _ => return,
@@ -600,7 +601,7 @@ fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
             BranchOptInfo {
                 br_inst: inst,
                 cmp_arg: cmp_arg,
-                args: args.clone(),
+                args: br_args.clone(),
                 new_opcode: new_opcode,
             }
         } else {
@@ -618,16 +619,6 @@ fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
     }
 }
 
-struct BranchOrderInfo {
-    term_inst: Inst,
-    term_inst_args: ValueList,
-    term_dest: Ebb,
-    cond_inst: Inst,
-    cond_inst_args: ValueList,
-    cond_dest: Ebb,
-    kind: BranchOrderKind,
-}
-
 enum BranchOrderKind {
     BrzToBrnz(Value),
     BrnzToBrz(Value),
@@ -640,134 +631,128 @@ enum BranchOrderKind {
 /// branch, this will reorder them if one of them is branching to the next Ebb
 /// layout-wise. The unconditional jump can then become a fallthrough.
 fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst: Inst) {
-    let info = match pos.func.dfg[inst] {
-        InstructionData::Jump {
-            opcode: Opcode::Jump,
-            destination,
-            ref args,
-        } => {
-            let next_ebb = if let Some(next_ebb) = pos.func.layout.next_ebb(ebb) {
-                next_ebb
-            } else {
-                return;
-            };
+    let (term_inst, term_inst_args, term_dest, cond_inst, cond_inst_args, cond_dest, kind) =
+        match pos.func.dfg[inst] {
+            InstructionData::Jump {
+                opcode: Opcode::Jump,
+                destination,
+                ref args,
+            } => {
+                let next_ebb = if let Some(next_ebb) = pos.func.layout.next_ebb(ebb) {
+                    next_ebb
+                } else {
+                    return;
+                };
 
-            if destination == next_ebb {
-                return;
-            }
-
-            let prev_inst = if let Some(prev_inst) = pos.func.layout.prev_inst(inst) {
-                prev_inst
-            } else {
-                return;
-            };
-
-            let prev_inst_data = &pos.func.dfg[prev_inst];
-
-            if let Some(prev_dest) = prev_inst_data.branch_destination() {
-                if prev_dest != next_ebb {
+                if destination == next_ebb {
                     return;
                 }
-            } else {
-                return;
+
+                let prev_inst = if let Some(prev_inst) = pos.func.layout.prev_inst(inst) {
+                    prev_inst
+                } else {
+                    return;
+                };
+
+                let prev_inst_data = &pos.func.dfg[prev_inst];
+
+                if let Some(prev_dest) = prev_inst_data.branch_destination() {
+                    if prev_dest != next_ebb {
+                        return;
+                    }
+                } else {
+                    return;
+                }
+
+                match prev_inst_data {
+                    InstructionData::Branch {
+                        opcode,
+                        args: ref prev_args,
+                        destination: cond_dest,
+                    } => {
+                        let cond_arg = {
+                            let args = pos.func.dfg.inst_args(prev_inst);
+                            args[0]
+                        };
+
+                        let kind = match opcode {
+                            Opcode::Brz => BranchOrderKind::BrzToBrnz(cond_arg),
+                            Opcode::Brnz => BranchOrderKind::BrnzToBrz(cond_arg),
+                            _ => panic!("unexpected opcode"),
+                        };
+
+                        (
+                            inst,
+                            args.clone(),
+                            destination,
+                            prev_inst,
+                            prev_args.clone(),
+                            *cond_dest,
+                            kind,
+                        )
+                    }
+                    InstructionData::BranchIcmp {
+                        opcode: Opcode::BrIcmp,
+                        cond,
+                        destination: cond_dest,
+                        args: ref prev_args,
+                    } => {
+                        let (x_arg, y_arg) = {
+                            let args = pos.func.dfg.inst_args(prev_inst);
+                            (args[0], args[1])
+                        };
+
+                        (
+                            inst,
+                            args.clone(),
+                            destination,
+                            prev_inst,
+                            prev_args.clone(),
+                            *cond_dest,
+                            BranchOrderKind::InvertIcmpCond(*cond, x_arg, y_arg),
+                        )
+                    }
+                    _ => return,
+                }
             }
 
-            match prev_inst_data {
-                InstructionData::Branch {
-                    opcode,
-                    args: ref prev_args,
-                    destination: cond_dest,
-                } => {
-                    let cond_arg = {
-                        let args = pos.func.dfg.inst_args(prev_inst);
-                        args[0]
-                    };
+            _ => return,
+        };
 
-                    let kind = match opcode {
-                        Opcode::Brz => BranchOrderKind::BrzToBrnz(cond_arg),
-                        Opcode::Brnz => BranchOrderKind::BrnzToBrz(cond_arg),
-                        _ => panic!("unexpected opcode"),
-                    };
+    let cond_args = { cond_inst_args.as_slice(&pos.func.dfg.value_lists).to_vec() };
+    let term_args = { term_inst_args.as_slice(&pos.func.dfg.value_lists).to_vec() };
 
-                    BranchOrderInfo {
-                        term_inst: inst,
-                        term_inst_args: args.clone(),
-                        term_dest: destination,
-                        cond_inst: prev_inst,
-                        cond_inst_args: prev_args.clone(),
-                        cond_dest: *cond_dest,
-                        kind: kind,
-                    }
-                }
-                InstructionData::BranchIcmp {
-                    opcode: Opcode::BrIcmp,
-                    cond,
-                    destination: cond_dest,
-                    args: ref prev_args,
-                } => {
-                    let (x_arg, y_arg) = {
-                        let args = pos.func.dfg.inst_args(prev_inst);
-                        (args[0], args[1])
-                    };
-                    BranchOrderInfo {
-                        term_inst: inst,
-                        term_inst_args: args.clone(),
-                        term_dest: destination,
-                        cond_inst: prev_inst,
-                        cond_inst_args: prev_args.clone(),
-                        cond_dest: *cond_dest,
-                        kind: BranchOrderKind::InvertIcmpCond(*cond, x_arg, y_arg),
-                    }
-                }
-                _ => return,
-            }
-        }
-
-        _ => return,
-    };
-
-    let cond_args = {
-        info.cond_inst_args
-            .as_slice(&pos.func.dfg.value_lists)
-            .to_vec()
-    };
-    let term_args = {
-        info.term_inst_args
-            .as_slice(&pos.func.dfg.value_lists)
-            .to_vec()
-    };
-
-    match info.kind {
+    match kind {
         BranchOrderKind::BrnzToBrz(cond_arg) => {
             pos.func
                 .dfg
-                .replace(info.term_inst)
-                .jump(info.cond_dest, &cond_args[1..]);
+                .replace(term_inst)
+                .jump(cond_dest, &cond_args[1..]);
             pos.func
                 .dfg
-                .replace(info.cond_inst)
-                .brz(cond_arg, info.term_dest, &term_args);
+                .replace(cond_inst)
+                .brz(cond_arg, term_dest, &term_args);
         }
         BranchOrderKind::BrzToBrnz(cond_arg) => {
             pos.func
                 .dfg
-                .replace(info.term_inst)
-                .jump(info.cond_dest, &cond_args[1..]);
+                .replace(term_inst)
+                .jump(cond_dest, &cond_args[1..]);
             pos.func
                 .dfg
-                .replace(info.cond_inst)
-                .brnz(cond_arg, info.term_dest, &term_args);
+                .replace(cond_inst)
+                .brnz(cond_arg, term_dest, &term_args);
         }
         BranchOrderKind::InvertIcmpCond(cond, x_arg, y_arg) => {
             pos.func
                 .dfg
-                .replace(info.term_inst)
-                .jump(info.cond_dest, &cond_args[2..]);
-            pos.func.dfg.replace(info.cond_inst).br_icmp(
+                .replace(term_inst)
+                .jump(cond_dest, &cond_args[2..]);
+            pos.func.dfg.replace(cond_inst).br_icmp(
                 cond.inverse(),
                 x_arg,
                 y_arg,
-                info.term_dest,
+                term_dest,
                 &term_args,
             );
         }

From 274415d5ee3a863fbf9da08d526422ada5a3607c Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 16 Apr 2019 17:48:15 +0200
Subject: [PATCH 2395/3084] Fixes #738: Check ebbs used in jump tables in the
 verifier;

---
 cranelift/codegen/src/verifier/mod.rs         | 25 +++++++++++++++----
 .../filetests/verifier/jump_table.clif        | 19 ++++++++++++++
 2 files changed, 39 insertions(+), 5 deletions(-)
 create mode 100644 cranelift/filetests/filetests/verifier/jump_table.clif

diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 0aeb3ad8a9..fd72315735 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -459,6 +459,15 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
+    fn verify_jump_tables(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
+        for (jt, jt_data) in &self.func.jump_tables {
+            for &ebb in jt_data.iter() {
+                self.verify_ebb(jt, ebb, errors)?;
+            }
+        }
+        Ok(())
+    }
+
     fn ebb_integrity(
         &self,
         ebb: Ebb,
@@ -606,8 +615,13 @@ impl<'a> Verifier<'a> {
                 self.verify_ebb(inst, destination, errors)?;
                 self.verify_value_list(inst, args, errors)?;
             }
-            BranchTable { table, .. }
-            | BranchTableBase { table, .. }
+            BranchTable {
+                table, destination, ..
+            } => {
+                self.verify_ebb(inst, destination, errors)?;
+                self.verify_jump_table(inst, table, errors)?;
+            }
+            BranchTableBase { table, .. }
             | BranchTableEntry { table, .. }
             | IndirectJump { table, .. } => {
                 self.verify_jump_table(inst, table, errors)?;
@@ -685,16 +699,16 @@ impl<'a> Verifier<'a> {
 
     fn verify_ebb(
         &self,
-        inst: Inst,
+        loc: impl Into,
         e: Ebb,
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.dfg.ebb_is_valid(e) || !self.func.layout.is_ebb_inserted(e) {
-            return fatal!(errors, inst, "invalid ebb reference {}", e);
+            return fatal!(errors, loc, "invalid ebb reference {}", e);
         }
         if let Some(entry_block) = self.func.layout.entry_block() {
             if e == entry_block {
-                return fatal!(errors, inst, "invalid reference to entry ebb {}", e);
+                return fatal!(errors, loc, "invalid reference to entry ebb {}", e);
             }
         }
         Ok(())
@@ -1679,6 +1693,7 @@ impl<'a> Verifier<'a> {
         self.verify_global_values(errors)?;
         self.verify_heaps(errors)?;
         self.verify_tables(errors)?;
+        self.verify_jump_tables(errors)?;
         self.typecheck_entry_block_params(errors)?;
 
         for ebb in self.func.layout.ebbs() {
diff --git a/cranelift/filetests/filetests/verifier/jump_table.clif b/cranelift/filetests/filetests/verifier/jump_table.clif
new file mode 100644
index 0000000000..bafd751f2e
--- /dev/null
+++ b/cranelift/filetests/filetests/verifier/jump_table.clif
@@ -0,0 +1,19 @@
+test verifier
+
+function %br_invalid_default(i64) {
+    jt0 = jump_table [ebb1, ebb1]
+
+ebb0(v0: i64):
+    br_table.i64 v0, ebb2, jt0 ; error: invalid ebb reference ebb2
+ebb1:
+    return
+}
+
+function %br(i64) {
+    jt0 = jump_table [ebb1, ebb2] ; error: invalid ebb reference ebb2
+
+ebb0(v0: i64):
+    br_table.i64 v0, ebb1, jt0
+ebb1:
+    return
+}

From b3a950b589749f9b0552cd3f62213c6fc422e387 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 11 Apr 2019 19:08:07 +0200
Subject: [PATCH 2396/3084] [meta] Fix condition codes in immediates;

---
 cranelift/codegen/meta/src/shared/immediates.rs | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/immediates.rs b/cranelift/codegen/meta/src/shared/immediates.rs
index 077fe0a511..c31ed11c7f 100644
--- a/cranelift/codegen/meta/src/shared/immediates.rs
+++ b/cranelift/codegen/meta/src/shared/immediates.rs
@@ -71,10 +71,10 @@ pub fn define() -> Vec {
     let mut intcc_values = HashMap::new();
     intcc_values.insert("eq", "Equal");
     intcc_values.insert("ne", "NotEqual");
-    intcc_values.insert("sge", "UnsignedGreaterThanOrEqual");
-    intcc_values.insert("sgt", "UnsignedGreaterThan");
-    intcc_values.insert("sle", "UnsignedLessThanOrEqual");
-    intcc_values.insert("slt", "UnsignedLessThan");
+    intcc_values.insert("sge", "SignedGreaterThanOrEqual");
+    intcc_values.insert("sgt", "SignedGreaterThan");
+    intcc_values.insert("sle", "SignedLessThanOrEqual");
+    intcc_values.insert("slt", "SignedLessThan");
     intcc_values.insert("uge", "UnsignedGreaterThanOrEqual");
     intcc_values.insert("ugt", "UnsignedGreaterThan");
     intcc_values.insert("ule", "UnsignedLessThanOrEqual");

From faa9b2569133ddc86815fe00ddc16ee0eb921732 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 12 Apr 2019 18:27:40 +0200
Subject: [PATCH 2397/3084] [meta-python] Simplify Var ctor since it never is
 given a typevar argument;

---
 cranelift/codegen/meta-python/cdsl/ast.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/meta-python/cdsl/ast.py b/cranelift/codegen/meta-python/cdsl/ast.py
index a524ba7cf6..4a72fc486a 100644
--- a/cranelift/codegen/meta-python/cdsl/ast.py
+++ b/cranelift/codegen/meta-python/cdsl/ast.py
@@ -162,19 +162,19 @@ class Var(Atom):
         Values that are defined only in the destination pattern.
     """
 
-    def __init__(self, name, typevar=None):
-        # type: (str, TypeVar) -> None
+    def __init__(self, name):
+        # type: (str) -> None
         self.name = name
         # The `Def` defining this variable in a source pattern.
         self.src_def = None  # type: Def
         # The `Def` defining this variable in a destination pattern.
         self.dst_def = None  # type: Def
         # TypeVar representing the type of this variable.
-        self.typevar = typevar  # type: TypeVar
+        self.typevar = None  # type: TypeVar
         # The original 'typeof(x)' type variable that was created for this Var.
         # This one doesn't change. `self.typevar` above may be changed to
         # another typevar by type inference.
-        self.original_typevar = self.typevar  # type: TypeVar
+        self.original_typevar = None  # type: TypeVar
 
     def __str__(self):
         # type: () -> str

From d2d2cdcd78ed9e2fcd0641c2faf5957c983167fa Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 12 Apr 2019 18:28:47 +0200
Subject: [PATCH 2398/3084] [meta] Rejigger comments in cdsl/formats;

---
 cranelift/codegen/meta/src/cdsl/formats.rs | 47 +++++++++-------------
 1 file changed, 20 insertions(+), 27 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
index 837440d2c0..244564a765 100644
--- a/cranelift/codegen/meta/src/cdsl/formats.rs
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -10,10 +10,6 @@ use cranelift_entity::{entity_impl, PrimaryMap};
 ///
 /// This corresponds to a single member of a variant of the `InstructionData`
 /// data type.
-///
-/// :param iform: Parent `InstructionFormat`.
-/// :param kind: Immediate Operand kind.
-/// :param member: Member name in `InstructionData` variant.
 #[derive(Debug)]
 pub struct FormatField {
     /// Immediate operand number in parent.
@@ -22,39 +18,36 @@ pub struct FormatField {
     /// Immediate operand kind.
     pub kind: OperandKind,
 
-    /// Member name in InstructionDate variant.
+    /// Member name in InstructionData variant.
     pub member: &'static str,
 }
 
-/// Every instruction opcode has a corresponding instruction format which
-/// determines the number of operands and their kinds. Instruction formats are
-/// identified structurally, i.e., the format of an instruction is derived from
-/// the kinds of operands used in its declaration.
+/// Every instruction opcode has a corresponding instruction format which determines the number of
+/// operands and their kinds. Instruction formats are identified structurally, i.e., the format of
+/// an instruction is derived from the kinds of operands used in its declaration.
 ///
-/// The instruction format stores two separate lists of operands: Immediates
-/// and values. Immediate operands (including entity references) are
-/// represented as explicit members in the `InstructionData` variants. The
-/// value operands are stored differently, depending on how many there are.
-/// Beyond a certain point, instruction formats switch to an external value
-/// list for storing value arguments. Value lists can hold an arbitrary number
-/// of values.
+/// The instruction format stores two separate lists of operands: Immediates and values. Immediate
+/// operands (including entity references) are represented as explicit members in the
+/// `InstructionData` variants. The value operands are stored differently, depending on how many
+/// there are.  Beyond a certain point, instruction formats switch to an external value list for
+/// storing value arguments. Value lists can hold an arbitrary number of values.
 ///
-/// All instruction formats must be predefined in the meta shared/formats module.
-///
-/// :param kinds: List of `OperandKind` objects describing the operands.
-/// :param name: Instruction format name in CamelCase. This is used as a Rust
-///     variant name in both the `InstructionData` and `InstructionFormat`
-///     enums.
-/// :param typevar_operand: Index of the value input operand that is used to
-///     infer the controlling type variable. By default, this is `0`, the first
-///     `value` operand. The index is relative to the values only, ignoring
-///     immediate operands.
+/// All instruction formats must be predefined in the meta shared/formats.rs module.
 #[derive(Debug)]
 pub struct InstructionFormat {
+    /// Instruction format name in CamelCase. This is used as a Rust variant name in both the
+    /// `InstructionData` and `InstructionFormat` enums.
     pub name: &'static str,
+
     pub num_value_operands: usize,
+
     pub has_value_list: bool,
+
     pub imm_fields: Vec,
+
+    /// Index of the value input operand that is used to infer the controlling type variable. By
+    /// default, this is `0`, the first `value` operand. The index is relative to the values only,
+    /// ignoring immediate operands.
     pub typevar_operand: Option,
 }
 
@@ -162,7 +155,7 @@ impl InstructionFormatBuilder {
     }
 }
 
-#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+#[derive(Copy, Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct InstructionFormatIndex(u32);
 entity_impl!(InstructionFormatIndex);
 

From 00429ebe995ccdcdda917a3530e1f7246ac42e6e Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 12 Apr 2019 18:34:20 +0200
Subject: [PATCH 2399/3084] [meta] Fix outdented_line in srcgen;

---
 cranelift/codegen/meta/src/srcgen.rs | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/meta/src/srcgen.rs b/cranelift/codegen/meta/src/srcgen.rs
index 9a6c4af038..ac2b8e8e03 100644
--- a/cranelift/codegen/meta/src/srcgen.rs
+++ b/cranelift/codegen/meta/src/srcgen.rs
@@ -80,9 +80,9 @@ impl Formatter {
     /// Get a string containing whitespace outdented one level. Used for
     /// lines of code that are inside a single indented block.
     fn _get_outdent(&mut self) -> String {
-        self.indent_push();
-        let s = self.get_indent();
         self.indent_pop();
+        let s = self.get_indent();
+        self.indent_push();
         s
     }
 
@@ -99,7 +99,7 @@ impl Formatter {
 
     /// Emit a line outdented one level.
     pub fn _outdented_line(&mut self, s: &str) {
-        let new_line = format!("{}{}", self._get_outdent(), s);
+        let new_line = format!("{}{}\n", self._get_outdent(), s);
         self.lines.push(new_line);
     }
 

From 6acf9be540fdfbc348747ae1934c259f74df7a2a Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 15 Apr 2019 15:46:37 +0200
Subject: [PATCH 2400/3084] Refactor simple-preopt to make it slightly simpler
 to read;

- don't use camel case but snake casing;
- longer variable names;
- more whitespace;
- add/wrap comments;
---
 cranelift/codegen/src/simple_preopt.rs | 180 ++++++++++++-------------
 1 file changed, 90 insertions(+), 90 deletions(-)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index e081a5bd43..fabad6d3b7 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -4,8 +4,6 @@
 //! should be useful for already well-optimized code. More general purpose
 //! early-stage optimizations can be found in the preopt crate.
 
-#![allow(non_snake_case)]
-
 use crate::cursor::{Cursor, FuncCursor};
 use crate::divconst_magic_numbers::{magic_s32, magic_s64, magic_u32, magic_u64};
 use crate::divconst_magic_numbers::{MS32, MS64, MU32, MU64};
@@ -27,7 +25,7 @@ use crate::timing;
 /// if `x` is a power of two, or the negation thereof, return the power along
 /// with a boolean that indicates whether `x` is negative. Else return None.
 #[inline]
-fn isPowerOf2_S32(x: i32) -> Option<(bool, u32)> {
+fn i32_is_power_of_two(x: i32) -> Option<(bool, u32)> {
     // We have to special-case this because abs(x) isn't representable.
     if x == -0x8000_0000 {
         return Some((true, 31));
@@ -39,9 +37,9 @@ fn isPowerOf2_S32(x: i32) -> Option<(bool, u32)> {
     None
 }
 
-/// Same comments as for isPowerOf2_S64 apply.
+/// Same comments as for i32_is_power_of_two apply.
 #[inline]
-fn isPowerOf2_S64(x: i64) -> Option<(bool, u32)> {
+fn i64_is_power_of_two(x: i64) -> Option<(bool, u32)> {
     // We have to special-case this because abs(x) isn't representable.
     if x == -0x8000_0000_0000_0000 {
         return Some((true, 63));
@@ -53,10 +51,12 @@ fn isPowerOf2_S64(x: i64) -> Option<(bool, u32)> {
     None
 }
 
+/// Representation of an instruction that can be replaced by a single division/remainder operation
+/// between a left Value operand and a right immediate operand.
 #[derive(Debug)]
 enum DivRemByConstInfo {
-    DivU32(Value, u32), // In all cases, the arguments are:
-    DivU64(Value, u64), // left operand, right operand
+    DivU32(Value, u32),
+    DivU64(Value, u64),
     DivS32(Value, i32),
     DivS64(Value, i64),
     RemU32(Value, u32),
@@ -65,84 +65,88 @@ enum DivRemByConstInfo {
     RemS64(Value, i64),
 }
 
-/// Possibly create a DivRemByConstInfo from the given components, by
-/// figuring out which, if any, of the 8 cases apply, and also taking care to
-/// sanity-check the immediate.
+/// Possibly create a DivRemByConstInfo from the given components, by figuring out which, if any,
+/// of the 8 cases apply, and also taking care to sanity-check the immediate.
 fn package_up_divrem_info(
-    argL: Value,
-    argL_ty: Type,
-    argRs: i64,
-    isSigned: bool,
-    isRem: bool,
+    value: Value,
+    value_type: Type,
+    imm_i64: i64,
+    is_signed: bool,
+    is_rem: bool,
 ) -> Option {
-    let argRu: u64 = argRs as u64;
-    if !isSigned && argL_ty == I32 && argRu < 0x1_0000_0000 {
-        let con = if isRem {
-            DivRemByConstInfo::RemU32
-        } else {
-            DivRemByConstInfo::DivU32
-        };
-        return Some(con(argL, argRu as u32));
+    let imm_u64 = imm_i64 as u64;
+
+    match (is_signed, value_type) {
+        (false, I32) => {
+            if imm_u64 < 0x1_0000_0000 {
+                if is_rem {
+                    Some(DivRemByConstInfo::RemU32(value, imm_u64 as u32))
+                } else {
+                    Some(DivRemByConstInfo::DivU32(value, imm_u64 as u32))
+                }
+            } else {
+                None
+            }
+        }
+
+        (false, I64) => {
+            // unsigned 64, no range constraint.
+            if is_rem {
+                Some(DivRemByConstInfo::RemU64(value, imm_u64))
+            } else {
+                Some(DivRemByConstInfo::DivU64(value, imm_u64))
+            }
+        }
+
+        (true, I32) => {
+            if imm_u64 <= 0x7fff_ffff || imm_u64 >= 0xffff_ffff_8000_0000 {
+                if is_rem {
+                    Some(DivRemByConstInfo::RemS32(value, imm_u64 as i32))
+                } else {
+                    Some(DivRemByConstInfo::DivS32(value, imm_u64 as i32))
+                }
+            } else {
+                None
+            }
+        }
+
+        (true, I64) => {
+            // signed 64, no range constraint.
+            if is_rem {
+                Some(DivRemByConstInfo::RemS64(value, imm_u64 as i64))
+            } else {
+                Some(DivRemByConstInfo::DivS64(value, imm_u64 as i64))
+            }
+        }
+
+        _ => None,
     }
-    if !isSigned && argL_ty == I64 {
-        // unsigned 64, no range constraint
-        let con = if isRem {
-            DivRemByConstInfo::RemU64
-        } else {
-            DivRemByConstInfo::DivU64
-        };
-        return Some(con(argL, argRu));
-    }
-    if isSigned && argL_ty == I32 && (argRu <= 0x7fff_ffff || argRu >= 0xffff_ffff_8000_0000) {
-        let con = if isRem {
-            DivRemByConstInfo::RemS32
-        } else {
-            DivRemByConstInfo::DivS32
-        };
-        return Some(con(argL, argRu as i32));
-    }
-    if isSigned && argL_ty == I64 {
-        // signed 64, no range constraint
-        let con = if isRem {
-            DivRemByConstInfo::RemS64
-        } else {
-            DivRemByConstInfo::DivS64
-        };
-        return Some(con(argL, argRu as i64));
-    }
-    None
 }
 
-/// Examine `idata` to see if it is a div or rem by a constant, and if so
-/// return the operands, signedness, operation size and div-vs-rem-ness in a
-/// handy bundle.
+/// Examine `inst` to see if it is a div or rem by a constant, and if so return the operands,
+/// signedness, operation size and div-vs-rem-ness in a handy bundle.
 fn get_div_info(inst: Inst, dfg: &DataFlowGraph) -> Option {
-    let idata: &InstructionData = &dfg[inst];
-
-    if let InstructionData::BinaryImm { opcode, arg, imm } = *idata {
-        let (isSigned, isRem) = match opcode {
+    if let InstructionData::BinaryImm { opcode, arg, imm } = dfg[inst] {
+        let (is_signed, is_rem) = match opcode {
             Opcode::UdivImm => (false, false),
             Opcode::UremImm => (false, true),
             Opcode::SdivImm => (true, false),
             Opcode::SremImm => (true, true),
-            _other => return None,
+            _ => return None,
         };
-        // Pull the operation size (type) from the left arg
-        let argL_ty = dfg.value_type(arg);
-        return package_up_divrem_info(arg, argL_ty, imm.into(), isSigned, isRem);
+        return package_up_divrem_info(arg, dfg.value_type(arg), imm.into(), is_signed, is_rem);
     }
 
     None
 }
 
-/// Actually do the transformation given a bundle containing the relevant
-/// information. `divrem_info` describes a div or rem by a constant, that
-/// `pos` currently points at, and `inst` is the associated instruction.
-/// `inst` is replaced by a sequence of other operations that calculate the
-/// same result. Note that there are various `divrem_info` cases where we
-/// cannot do any transformation, in which case `inst` is left unchanged.
+/// Actually do the transformation given a bundle containing the relevant information.
+/// `divrem_info` describes a div or rem by a constant, that `pos` currently points at, and `inst`
+/// is the associated instruction.  `inst` is replaced by a sequence of other operations that
+/// calculate the same result. Note that there are various `divrem_info` cases where we cannot do
+/// any transformation, in which case `inst` is left unchanged.
 fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCursor, inst: Inst) {
-    let isRem = match *divrem_info {
+    let is_rem = match *divrem_info {
         DivRemByConstInfo::DivU32(_, _)
         | DivRemByConstInfo::DivU64(_, _)
         | DivRemByConstInfo::DivS32(_, _)
@@ -162,7 +166,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
         // U32 div by 1: identity
         // U32 rem by 1: zero
         DivRemByConstInfo::DivU32(n1, 1) | DivRemByConstInfo::RemU32(n1, 1) => {
-            if isRem {
+            if is_rem {
                 pos.func.dfg.replace(inst).iconst(I32, 0);
             } else {
                 pos.func.dfg.replace(inst).copy(n1);
@@ -177,7 +181,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             // compute k where d == 2^k
             let k = d.trailing_zeros();
             debug_assert!(k >= 1 && k <= 31);
-            if isRem {
+            if is_rem {
                 let mask = (1u64 << k) - 1;
                 pos.func.dfg.replace(inst).band_imm(n1, mask as i64);
             } else {
@@ -216,7 +220,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             }
             // Now qf holds the final quotient. If necessary calculate the
             // remainder instead.
-            if isRem {
+            if is_rem {
                 let tt = pos.ins().imul_imm(qf, d as i64);
                 pos.func.dfg.replace(inst).isub(n1, tt);
             } else {
@@ -232,7 +236,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
         // U64 div by 1: identity
         // U64 rem by 1: zero
         DivRemByConstInfo::DivU64(n1, 1) | DivRemByConstInfo::RemU64(n1, 1) => {
-            if isRem {
+            if is_rem {
                 pos.func.dfg.replace(inst).iconst(I64, 0);
             } else {
                 pos.func.dfg.replace(inst).copy(n1);
@@ -247,7 +251,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             // compute k where d == 2^k
             let k = d.trailing_zeros();
             debug_assert!(k >= 1 && k <= 63);
-            if isRem {
+            if is_rem {
                 let mask = (1u64 << k) - 1;
                 pos.func.dfg.replace(inst).band_imm(n1, mask as i64);
             } else {
@@ -286,7 +290,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             }
             // Now qf holds the final quotient. If necessary calculate the
             // remainder instead.
-            if isRem {
+            if is_rem {
                 let tt = pos.ins().imul_imm(qf, d as i64);
                 pos.func.dfg.replace(inst).isub(n1, tt);
             } else {
@@ -305,7 +309,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
         // S32 div by 1: identity
         // S32 rem by 1: zero
         DivRemByConstInfo::DivS32(n1, 1) | DivRemByConstInfo::RemS32(n1, 1) => {
-            if isRem {
+            if is_rem {
                 pos.func.dfg.replace(inst).iconst(I32, 0);
             } else {
                 pos.func.dfg.replace(inst).copy(n1);
@@ -313,7 +317,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
         }
 
         DivRemByConstInfo::DivS32(n1, d) | DivRemByConstInfo::RemS32(n1, d) => {
-            if let Some((isNeg, k)) = isPowerOf2_S32(d) {
+            if let Some((is_negative, k)) = i32_is_power_of_two(d) {
                 // k can be 31 only in the case that d is -2^31.
                 debug_assert!(k >= 1 && k <= 31);
                 let t1 = if k - 1 == 0 {
@@ -323,7 +327,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                 };
                 let t2 = pos.ins().ushr_imm(t1, (32 - k) as i64);
                 let t3 = pos.ins().iadd(n1, t2);
-                if isRem {
+                if is_rem {
                     // S32 rem by a power-of-2
                     let t4 = pos.ins().band_imm(t3, i32::wrapping_neg(1 << k) as i64);
                     // Curiously, we don't care here what the sign of d is.
@@ -331,7 +335,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                 } else {
                     // S32 div by a power-of-2
                     let t4 = pos.ins().sshr_imm(t3, k as i64);
-                    if isNeg {
+                    if is_negative {
                         pos.func.dfg.replace(inst).irsub_imm(t4, 0);
                     } else {
                         pos.func.dfg.replace(inst).copy(t4);
@@ -360,7 +364,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                 let qf = pos.ins().iadd(q3, t1);
                 // Now qf holds the final quotient. If necessary calculate
                 // the remainder instead.
-                if isRem {
+                if is_rem {
                     let tt = pos.ins().imul_imm(qf, d as i64);
                     pos.func.dfg.replace(inst).isub(n1, tt);
                 } else {
@@ -380,7 +384,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
         // S64 div by 1: identity
         // S64 rem by 1: zero
         DivRemByConstInfo::DivS64(n1, 1) | DivRemByConstInfo::RemS64(n1, 1) => {
-            if isRem {
+            if is_rem {
                 pos.func.dfg.replace(inst).iconst(I64, 0);
             } else {
                 pos.func.dfg.replace(inst).copy(n1);
@@ -388,7 +392,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
         }
 
         DivRemByConstInfo::DivS64(n1, d) | DivRemByConstInfo::RemS64(n1, d) => {
-            if let Some((isNeg, k)) = isPowerOf2_S64(d) {
+            if let Some((is_negative, k)) = i64_is_power_of_two(d) {
                 // k can be 63 only in the case that d is -2^63.
                 debug_assert!(k >= 1 && k <= 63);
                 let t1 = if k - 1 == 0 {
@@ -398,7 +402,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                 };
                 let t2 = pos.ins().ushr_imm(t1, (64 - k) as i64);
                 let t3 = pos.ins().iadd(n1, t2);
-                if isRem {
+                if is_rem {
                     // S64 rem by a power-of-2
                     let t4 = pos.ins().band_imm(t3, i64::wrapping_neg(1 << k));
                     // Curiously, we don't care here what the sign of d is.
@@ -406,7 +410,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                 } else {
                     // S64 div by a power-of-2
                     let t4 = pos.ins().sshr_imm(t3, k as i64);
-                    if isNeg {
+                    if is_negative {
                         pos.func.dfg.replace(inst).irsub_imm(t4, 0);
                     } else {
                         pos.func.dfg.replace(inst).copy(t4);
@@ -435,7 +439,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                 let qf = pos.ins().iadd(q3, t1);
                 // Now qf holds the final quotient. If necessary calculate
                 // the remainder instead.
-                if isRem {
+                if is_rem {
                     let tt = pos.ins().imul_imm(qf, d);
                     pos.func.dfg.replace(inst).isub(n1, tt);
                 } else {
@@ -770,16 +774,12 @@ pub fn do_preopt(func: &mut Function, cfg: &mut ControlFlowGraph) {
             // Apply basic simplifications.
             simplify(&mut pos, inst);
 
-            //-- BEGIN -- division by constants ----------------
-
-            let mb_dri = get_div_info(inst, &pos.func.dfg);
-            if let Some(divrem_info) = mb_dri {
+            // Try to transform divide-by-constant into simpler operations.
+            if let Some(divrem_info) = get_div_info(inst, &pos.func.dfg) {
                 do_divrem_transformation(&divrem_info, &mut pos, inst);
                 continue;
             }
 
-            //-- END -- division by constants ------------------
-
             branch_opt(&mut pos, inst);
             branch_order(&mut pos, cfg, ebb, inst);
         }

From 45013a1d2bd7a5c7f61683db6ef914a39ac13fa4 Mon Sep 17 00:00:00 2001
From: iximeow 
Date: Thu, 18 Apr 2019 16:51:26 -0700
Subject: [PATCH 2401/3084] Expose function definitions and populate
 FaerieCompiledFunction with function lengths

---
 cranelift/faerie/src/backend.rs | 16 ++++++++++++++--
 cranelift/module/src/lib.rs     |  3 ++-
 cranelift/module/src/module.rs  | 11 ++++++++---
 3 files changed, 24 insertions(+), 6 deletions(-)

diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index f413879ac2..1deb46e073 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -96,7 +96,15 @@ pub struct FaerieBackend {
     libcall_names: Box String>,
 }
 
-pub struct FaerieCompiledFunction {}
+pub struct FaerieCompiledFunction {
+    code_length: u32,
+}
+
+impl FaerieCompiledFunction {
+    pub fn code_length(&self) -> u32 {
+        self.code_length
+    }
+}
 
 pub struct FaerieCompiledData {}
 
@@ -187,10 +195,14 @@ impl Backend for FaerieBackend {
             }
         }
 
+        // because `define` will take ownership of code, this is our last chance
+        let code_length = code.len() as u32;
+
         self.artifact
             .define(name, code)
             .expect("inconsistent declaration");
-        Ok(FaerieCompiledFunction {})
+
+        Ok(FaerieCompiledFunction { code_length })
     }
 
     fn define_data(
diff --git a/cranelift/module/src/lib.rs b/cranelift/module/src/lib.rs
index ce6f00aff0..e30961c3bc 100644
--- a/cranelift/module/src/lib.rs
+++ b/cranelift/module/src/lib.rs
@@ -40,7 +40,8 @@ mod module;
 pub use crate::backend::Backend;
 pub use crate::data_context::{DataContext, DataDescription, Init};
 pub use crate::module::{
-    DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleError, ModuleNamespace, ModuleResult,
+    DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleError, ModuleFunction, ModuleNamespace,
+    ModuleResult,
 };
 
 /// Version number of this crate.
diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 3b0f4673e0..5b11a48e15 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -153,14 +153,14 @@ pub enum ModuleError {
 pub type ModuleResult = Result;
 
 /// A function belonging to a `Module`.
-struct ModuleFunction
+pub struct ModuleFunction
 where
     B: Backend,
 {
     /// The function declaration.
-    decl: FunctionDeclaration,
+    pub decl: FunctionDeclaration,
     /// The compiled artifact, once it's available.
-    compiled: Option,
+    pub compiled: Option,
 }
 
 impl ModuleFunction
@@ -427,6 +427,11 @@ where
         }
     }
 
+    /// An iterator over functions that have been declared in this module.
+    pub fn declared_functions(&self) -> core::slice::Iter<'_, ModuleFunction> {
+        self.contents.functions.values()
+    }
+
     /// Declare a data object in this module.
     pub fn declare_data(
         &mut self,

From d92778a19e117f21ce95db0947188e027be1bb84 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 18 Apr 2019 18:27:33 +0200
Subject: [PATCH 2402/3084] [meta] Add Instruction helpers and change its
 representation to make it easily copiable;

- adds helpers used by other parts of the code
- allows cheap copies by having Instruction be a lightweight ref-cloned
wrapper of the actual instruction's content.
---
 cranelift/codegen/meta/src/cdsl/inst.rs     | 93 +++++++++++++++------
 cranelift/codegen/meta/src/cdsl/operands.rs |  2 +-
 2 files changed, 69 insertions(+), 26 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/inst.rs b/cranelift/codegen/meta/src/cdsl/inst.rs
index 4199d6424d..c3aeed9ac0 100644
--- a/cranelift/codegen/meta/src/cdsl/inst.rs
+++ b/cranelift/codegen/meta/src/cdsl/inst.rs
@@ -1,10 +1,15 @@
 use crate::cdsl::camel_case;
-use crate::cdsl::formats::{FormatRegistry, InstructionFormat, InstructionFormatIndex};
+use crate::cdsl::formats::{
+    FormatField, FormatRegistry, InstructionFormat, InstructionFormatIndex,
+};
 use crate::cdsl::operands::Operand;
 use crate::cdsl::type_inference::Constraint;
+use crate::cdsl::types::ValueType;
 use crate::cdsl::typevar::TypeVar;
 
 use std::fmt;
+use std::ops;
+use std::rc::Rc;
 use std::slice;
 
 /// Every instruction must belong to exactly one instruction group. A given
@@ -32,6 +37,13 @@ impl InstructionGroup {
     pub fn iter(&self) -> slice::Iter {
         self.instructions.iter()
     }
+
+    pub fn by_name(&self, name: &'static str) -> &Instruction {
+        self.instructions
+            .iter()
+            .find(|inst| inst.name == name)
+            .expect(&format!("unexisting instruction with name {}", name))
+    }
 }
 
 pub struct PolymorphicInfo {
@@ -40,7 +52,7 @@ pub struct PolymorphicInfo {
     pub other_typevars: Vec,
 }
 
-pub struct Instruction {
+pub struct InstructionContent {
     /// Instruction mnemonic, also becomes opcode name.
     pub name: &'static str,
     pub camel_name: String,
@@ -53,7 +65,7 @@ pub struct Instruction {
     /// Output operands. The output operands must be SSA values or `variable_args`.
     pub operands_out: Vec,
     /// Instruction-specific TypeConstraints.
-    _constraints: Vec,
+    pub constraints: Vec,
 
     /// Instruction format, automatically derived from the input operands.
     pub format: InstructionFormatIndex,
@@ -90,6 +102,18 @@ pub struct Instruction {
     pub writes_cpu_flags: bool,
 }
 
+#[derive(Clone)]
+pub struct Instruction {
+    content: Rc,
+}
+
+impl ops::Deref for Instruction {
+    type Target = InstructionContent;
+    fn deref(&self) -> &Self::Target {
+        &*self.content
+    }
+}
+
 impl Instruction {
     pub fn snake_name(&self) -> &'static str {
         if self.name == "return" {
@@ -108,6 +132,17 @@ impl Instruction {
         }
         ""
     }
+
+    pub fn all_typevars(&self) -> Vec<&TypeVar> {
+        match &self.polymorphic_info {
+            Some(poly) => {
+                let mut result = vec![&poly.ctrl_typevar];
+                result.extend(&poly.other_typevars);
+                result
+            }
+            None => Vec::new(),
+        }
+    }
 }
 
 impl fmt::Display for Instruction {
@@ -272,32 +307,40 @@ impl InstructionBuilder {
         let writes_cpu_flags = operands_out.iter().any(|op| op.is_cpu_flags());
 
         Instruction {
-            name: self.name,
-            camel_name: camel_case(self.name),
-            doc: self.doc,
-            operands_in,
-            operands_out,
-            _constraints: self.constraints.unwrap_or_else(Vec::new),
-            format: format_index,
-            polymorphic_info,
-            value_opnums,
-            value_results,
-            imm_opnums,
-            is_terminator: self.is_terminator,
-            is_branch: self.is_branch,
-            is_indirect_branch: self.is_indirect_branch,
-            is_call: self.is_call,
-            is_return: self.is_return,
-            is_ghost: self.is_ghost,
-            can_load: self.can_load,
-            can_store: self.can_store,
-            can_trap: self.can_trap,
-            other_side_effects: self.other_side_effects,
-            writes_cpu_flags,
+            content: Rc::new(InstructionContent {
+                name: self.name,
+                camel_name: camel_case(self.name),
+                doc: self.doc,
+                operands_in,
+                operands_out,
+                constraints: self.constraints.unwrap_or_else(Vec::new),
+                format: format_index,
+                polymorphic_info,
+                value_opnums,
+                value_results,
+                imm_opnums,
+                is_terminator: self.is_terminator,
+                is_branch: self.is_branch,
+                is_indirect_branch: self.is_indirect_branch,
+                is_call: self.is_call,
+                is_return: self.is_return,
+                is_ghost: self.is_ghost,
+                can_load: self.can_load,
+                can_store: self.can_store,
+                can_trap: self.can_trap,
+                other_side_effects: self.other_side_effects,
+                writes_cpu_flags,
+            }),
         }
     }
 }
 
+#[derive(Clone)]
+pub struct BoundInstruction {
+    pub inst: Instruction,
+    pub value_types: Vec,
+}
+
 /// Check if this instruction is polymorphic, and verify its use of type variables.
 fn verify_polymorphic(
     operands_in: &Vec,
diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
index f460225bfe..93a92823be 100644
--- a/cranelift/codegen/meta/src/cdsl/operands.rs
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -134,7 +134,7 @@ pub struct OperandKind {
     /// The camel-cased name of an operand kind is also the Rust type used to represent it.
     pub rust_type: String,
 
-    fields: OperandKindFields,
+    pub fields: OperandKindFields,
 }
 
 impl OperandKind {

From e71ae7b02ffd1a3bde295183b55aec07e2caa3f7 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 18 Apr 2019 18:29:20 +0200
Subject: [PATCH 2403/3084] [meta] Add instruction predicates;

---
 cranelift/codegen/meta/src/cdsl/inst.rs | 99 +++++++++++++++++++++++++
 1 file changed, 99 insertions(+)

diff --git a/cranelift/codegen/meta/src/cdsl/inst.rs b/cranelift/codegen/meta/src/cdsl/inst.rs
index c3aeed9ac0..d219419c7e 100644
--- a/cranelift/codegen/meta/src/cdsl/inst.rs
+++ b/cranelift/codegen/meta/src/cdsl/inst.rs
@@ -499,3 +499,102 @@ fn verify_ctrl_typevar(
 
     Ok(other_typevars)
 }
+
+/// A basic node in an instruction predicate: either an atom, or an AND of two conditions.
+pub enum InstructionPredicateNode {
+    /// Is the field member (first member) equal to the actual argument (which name is the second
+    /// field)?
+    IsFieldEqual(String, String),
+
+    /// Is the value argument (at the index designated by the first member) the same type as the
+    /// type name (second member)?
+    TypeVarCheck(usize, String),
+
+    /// Is the controlling type variable the same type as the one designated by the type name
+    /// (only member)?
+    CtrlTypeVarCheck(String),
+
+    /// A combination of two other predicates.
+    And(Vec),
+}
+
+impl InstructionPredicateNode {
+    fn rust_predicate(&self) -> String {
+        match self {
+            InstructionPredicateNode::IsFieldEqual(field_name, arg) => {
+                let new_args = vec![field_name.clone(), arg.clone()];
+                format!("crate::predicates::is_equal({})", new_args.join(", "))
+            }
+            InstructionPredicateNode::TypeVarCheck(index, value_type_name) => format!(
+                "func.dfg.value_type(args[{}]) == {}",
+                index, value_type_name
+            ),
+            InstructionPredicateNode::CtrlTypeVarCheck(value_type_name) => {
+                format!("func.dfg.ctrl_typevar(inst) == {}", value_type_name)
+            }
+            InstructionPredicateNode::And(nodes) => nodes
+                .iter()
+                .map(|x| x.rust_predicate())
+                .collect::>()
+                .join(" &&\n"),
+        }
+    }
+}
+
+pub struct InstructionPredicate {
+    node: Option,
+}
+
+impl InstructionPredicate {
+    pub fn new() -> Self {
+        Self { node: None }
+    }
+
+    pub fn new_typevar_check(
+        inst: &Instruction,
+        type_var: &TypeVar,
+        value_type: &ValueType,
+    ) -> InstructionPredicateNode {
+        let index = inst
+            .value_opnums
+            .iter()
+            .enumerate()
+            .filter(|(_, &op_num)| inst.operands_in[op_num].type_var().unwrap() == type_var)
+            .next()
+            .unwrap()
+            .0;
+        InstructionPredicateNode::TypeVarCheck(index, value_type.rust_name())
+    }
+
+    pub fn new_is_field_equal(
+        format_field: &FormatField,
+        imm_value: String,
+    ) -> InstructionPredicateNode {
+        InstructionPredicateNode::IsFieldEqual(format_field.member.into(), imm_value)
+    }
+
+    pub fn new_ctrl_typevar_check(value_type: &ValueType) -> InstructionPredicateNode {
+        InstructionPredicateNode::CtrlTypeVarCheck(value_type.rust_name())
+    }
+
+    pub fn and(mut self, new_node: InstructionPredicateNode) -> Self {
+        let node = self.node;
+        let mut and_nodes = match node {
+            Some(node) => match node {
+                InstructionPredicateNode::And(nodes) => nodes,
+                _ => vec![node],
+            },
+            _ => Vec::new(),
+        };
+        and_nodes.push(new_node);
+        self.node = Some(InstructionPredicateNode::And(and_nodes));
+        self
+    }
+
+    pub fn rust_predicate(&self) -> String {
+        match &self.node {
+            Some(root) => root.rust_predicate(),
+            None => "true".into(),
+        }
+    }
+}

From b85146e9453f463bc2129210ed145bfb734485a2 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 18 Apr 2019 18:33:33 +0200
Subject: [PATCH 2404/3084] [meta] Add helper methods and tests to TypeSet;

---
 cranelift/codegen/meta/src/cdsl/typevar.rs | 308 ++++++++++++++++++++-
 1 file changed, 305 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index d144eab48f..41d94d67f4 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -1,4 +1,7 @@
-use std::collections::BTreeSet;
+use std::cell::RefCell;
+use std::collections::{BTreeSet, HashSet};
+use std::fmt;
+use std::hash;
 use std::iter::FromIterator;
 use std::ops;
 use std::rc::Rc;
@@ -301,7 +304,7 @@ macro_rules! num_set {
     };
 }
 
-#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+#[derive(Clone, PartialEq, Eq, Hash)]
 pub struct TypeSet {
     pub lanes: NumSet,
     pub ints: NumSet,
@@ -331,7 +334,7 @@ impl TypeSet {
     }
 
     /// Return the number of concrete types represented by this typeset.
-    fn size(&self) -> usize {
+    pub fn size(&self) -> usize {
         self.lanes.len()
             * (self.ints.len() + self.floats.len() + self.bools.len() + self.bitvecs.len())
             + self.specials.len()
@@ -486,6 +489,175 @@ impl TypeSet {
         assert_eq!(types.len(), 1);
         return types.remove(0);
     }
+
+    /// Return the inverse image of self across the derived function func.
+    fn preimage(&self, func: DerivedFunc) -> TypeSet {
+        if self.size() == 0 {
+            // The inverse of the empty set is itself.
+            return self.clone();
+        }
+
+        match func {
+            DerivedFunc::LaneOf => {
+                let mut copy = self.clone();
+                copy.bitvecs = NumSet::new();
+                copy.lanes =
+                    NumSet::from_iter((0..MAX_LANES.trailing_zeros() + 1).map(|i| u16::pow(2, i)));
+                copy
+            }
+            DerivedFunc::AsBool => {
+                let mut copy = self.clone();
+                copy.bitvecs = NumSet::new();
+                if self.bools.contains(&1) {
+                    copy.ints = NumSet::from_iter(vec![8, 16, 32, 64]);
+                    copy.floats = NumSet::from_iter(vec![32, 64]);
+                } else {
+                    copy.ints = &self.bools - &NumSet::from_iter(vec![1]);
+                    copy.floats = &self.bools & &NumSet::from_iter(vec![32, 64]);
+                    // If b1 is not in our typeset, than lanes=1 cannot be in the pre-image, as
+                    // as_bool() of scalars is always b1.
+                    copy.lanes = &self.lanes - &NumSet::from_iter(vec![1]);
+                }
+                copy
+            }
+            DerivedFunc::HalfWidth => self.double_width(),
+            DerivedFunc::DoubleWidth => self.half_width(),
+            DerivedFunc::HalfVector => self.double_vector(),
+            DerivedFunc::DoubleVector => self.half_vector(),
+            DerivedFunc::ToBitVec => {
+                let all_lanes = range_to_set(Some(1..MAX_LANES));
+                let all_ints = range_to_set(Some(8..MAX_BITS));
+                let all_floats = range_to_set(Some(32..64));
+                let all_bools = range_to_set(Some(1..MAX_BITS));
+
+                let mut lanes = range_to_set(Some(1..MAX_LANES));
+                let mut ints = range_to_set(Some(8..MAX_BITS));
+                let mut floats = range_to_set(Some(32..64));
+                let mut bools = range_to_set(Some(1..MAX_BITS));
+
+                for &l in &all_lanes {
+                    for &i in &all_ints {
+                        if self.bitvecs.contains(&(i * l)) {
+                            lanes.insert(l);
+                            ints.insert(i);
+                        }
+                    }
+                    for &f in &all_floats {
+                        if self.bitvecs.contains(&(f * l)) {
+                            lanes.insert(l);
+                            floats.insert(f);
+                        }
+                    }
+                    for &b in &all_bools {
+                        if self.bitvecs.contains(&(b * l)) {
+                            lanes.insert(l);
+                            bools.insert(b);
+                        }
+                    }
+                }
+
+                let bitvecs = NumSet::new();
+                let specials = Vec::new();
+                TypeSet::new(lanes, ints, floats, bools, bitvecs, specials)
+            }
+        }
+    }
+
+    pub fn inplace_intersect_with(&mut self, other: &TypeSet) {
+        self.lanes = &self.lanes & &other.lanes;
+        self.ints = &self.ints & &other.ints;
+        self.floats = &self.floats & &other.floats;
+        self.bools = &self.bools & &other.bools;
+        self.bitvecs = &self.bitvecs & &other.bitvecs;
+
+        let mut new_specials = Vec::new();
+        for spec in &self.specials {
+            if let Some(spec) = other.specials.iter().find(|&other_spec| other_spec == spec) {
+                new_specials.push(*spec);
+            }
+        }
+        self.specials = new_specials;
+    }
+
+    pub fn is_subset(&self, other: &TypeSet) -> bool {
+        self.lanes.is_subset(&other.lanes)
+            && self.ints.is_subset(&other.ints)
+            && self.floats.is_subset(&other.floats)
+            && self.bools.is_subset(&other.bools)
+            && self.bitvecs.is_subset(&other.bitvecs)
+            && {
+                let specials: HashSet = HashSet::from_iter(self.specials.clone());
+                let other_specials = HashSet::from_iter(other.specials.clone());
+                specials.is_subset(&other_specials)
+            }
+    }
+
+    pub fn is_wider_or_equal(&self, other: &TypeSet) -> bool {
+        set_wider_or_equal(&self.ints, &other.ints)
+            && set_wider_or_equal(&self.floats, &other.floats)
+            && set_wider_or_equal(&self.bools, &other.bools)
+    }
+
+    pub fn is_narrower(&self, other: &TypeSet) -> bool {
+        set_narrower(&self.ints, &other.ints)
+            && set_narrower(&self.floats, &other.floats)
+            && set_narrower(&self.bools, &other.bools)
+    }
+}
+
+fn set_wider_or_equal(s1: &NumSet, s2: &NumSet) -> bool {
+    s1.len() > 0 && s2.len() > 0 && s1.iter().min() >= s2.iter().max()
+}
+
+fn set_narrower(s1: &NumSet, s2: &NumSet) -> bool {
+    s1.len() > 0 && s2.len() > 0 && s1.iter().min() < s2.iter().max()
+}
+
+impl fmt::Debug for TypeSet {
+    fn fmt(&self, fmt: &mut fmt::Formatter) -> Result<(), fmt::Error> {
+        write!(fmt, "TypeSet(")?;
+
+        let mut subsets = Vec::new();
+        if !self.lanes.is_empty() {
+            subsets.push(format!(
+                "lanes={{{}}}",
+                Vec::from_iter(self.lanes.iter().map(|x| x.to_string())).join(", ")
+            ));
+        }
+        if !self.ints.is_empty() {
+            subsets.push(format!(
+                "ints={{{}}}",
+                Vec::from_iter(self.ints.iter().map(|x| x.to_string())).join(", ")
+            ));
+        }
+        if !self.floats.is_empty() {
+            subsets.push(format!(
+                "floats={{{}}}",
+                Vec::from_iter(self.floats.iter().map(|x| x.to_string())).join(", ")
+            ));
+        }
+        if !self.bools.is_empty() {
+            subsets.push(format!(
+                "bools={{{}}}",
+                Vec::from_iter(self.bools.iter().map(|x| x.to_string())).join(", ")
+            ));
+        }
+        if !self.bitvecs.is_empty() {
+            subsets.push(format!(
+                "bitvecs={{{}}}",
+                Vec::from_iter(self.bitvecs.iter().map(|x| x.to_string())).join(", ")
+            ));
+        }
+        if !self.specials.is_empty() {
+            subsets.push(format!(
+                "specials={{{}}}",
+                Vec::from_iter(self.specials.iter().map(|x| x.to_string())).join(", ")
+            ));
+        }
+
+        write!(fmt, "{})", subsets.join(", "))?;
+        Ok(())
+    }
 }
 
 pub struct TypeSetBuilder {
@@ -805,6 +977,136 @@ fn test_forward_images() {
     );
 }
 
+#[test]
+fn test_backward_images() {
+    let empty_set = TypeSetBuilder::new().finish();
+
+    // LaneOf.
+    assert_eq!(
+        TypeSetBuilder::new()
+            .simd_lanes(1..1)
+            .ints(8..8)
+            .floats(32..32)
+            .finish()
+            .preimage(DerivedFunc::LaneOf),
+        TypeSetBuilder::new()
+            .simd_lanes(Interval::All)
+            .ints(8..8)
+            .floats(32..32)
+            .finish()
+    );
+    assert_eq!(empty_set.preimage(DerivedFunc::LaneOf), empty_set);
+
+    // AsBool.
+    assert_eq!(
+        TypeSetBuilder::new()
+            .simd_lanes(1..4)
+            .bools(1..64)
+            .finish()
+            .preimage(DerivedFunc::AsBool),
+        TypeSetBuilder::new()
+            .simd_lanes(1..4)
+            .ints(Interval::All)
+            .bools(Interval::All)
+            .floats(Interval::All)
+            .finish()
+    );
+
+    // Double vector.
+    assert_eq!(
+        TypeSetBuilder::new()
+            .simd_lanes(1..1)
+            .ints(8..8)
+            .finish()
+            .preimage(DerivedFunc::DoubleVector)
+            .size(),
+        0
+    );
+    assert_eq!(
+        TypeSetBuilder::new()
+            .simd_lanes(1..16)
+            .ints(8..16)
+            .floats(32..32)
+            .finish()
+            .preimage(DerivedFunc::DoubleVector),
+        TypeSetBuilder::new()
+            .simd_lanes(1..8)
+            .ints(8..16)
+            .floats(32..32)
+            .finish(),
+    );
+
+    // Half vector.
+    assert_eq!(
+        TypeSetBuilder::new()
+            .simd_lanes(256..256)
+            .ints(8..8)
+            .finish()
+            .preimage(DerivedFunc::HalfVector)
+            .size(),
+        0
+    );
+    assert_eq!(
+        TypeSetBuilder::new()
+            .simd_lanes(64..128)
+            .bools(1..32)
+            .finish()
+            .preimage(DerivedFunc::HalfVector),
+        TypeSetBuilder::new()
+            .simd_lanes(128..256)
+            .bools(1..32)
+            .finish(),
+    );
+
+    // Half width.
+    assert_eq!(
+        TypeSetBuilder::new()
+            .ints(64..64)
+            .floats(64..64)
+            .bools(64..64)
+            .finish()
+            .preimage(DerivedFunc::HalfWidth)
+            .size(),
+        0
+    );
+    assert_eq!(
+        TypeSetBuilder::new()
+            .simd_lanes(64..256)
+            .bools(1..64)
+            .finish()
+            .preimage(DerivedFunc::HalfWidth),
+        TypeSetBuilder::new()
+            .simd_lanes(64..256)
+            .bools(16..64)
+            .finish(),
+    );
+
+    // Double width.
+    assert_eq!(
+        TypeSetBuilder::new()
+            .ints(8..8)
+            .floats(32..32)
+            .bools(1..8)
+            .finish()
+            .preimage(DerivedFunc::DoubleWidth)
+            .size(),
+        0
+    );
+    assert_eq!(
+        TypeSetBuilder::new()
+            .simd_lanes(1..16)
+            .ints(8..16)
+            .floats(32..64)
+            .finish()
+            .preimage(DerivedFunc::DoubleWidth),
+        TypeSetBuilder::new()
+            .simd_lanes(1..16)
+            .ints(8..8)
+            .floats(32..32)
+            .finish()
+    );
+}
+
 #[test]
 #[should_panic]
 fn test_typeset_singleton_panic_nonsingleton_types() {

From dfb27c3402ca1dbc8e2e26394e03559f64ebff36 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 18 Apr 2019 18:38:11 +0200
Subject: [PATCH 2405/3084] [meta] Make TypeVar shareable and mutable;

... since its type set can change over time during type inference. Use a
Rc to achieve this.
---
 cranelift/codegen/meta/src/cdsl/typevar.rs | 154 ++++++++++++++++++---
 1 file changed, 131 insertions(+), 23 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 41d94d67f4..3b18d508fc 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -30,25 +30,25 @@ pub struct TypeVarContent {
     /// Type set associated to the type variable.
     /// This field must remain private; use `get_typeset()` or `get_raw_typeset()` to get the
     /// information you want.
-    type_set: Rc,
+    type_set: TypeSet,
 
     pub base: Option,
 }
 
 #[derive(Clone, Debug)]
 pub struct TypeVar {
-    content: Rc,
+    content: Rc>,
 }
 
 impl TypeVar {
     pub fn new(name: impl Into, doc: impl Into, type_set: TypeSet) -> Self {
         Self {
-            content: Rc::new(TypeVarContent {
+            content: Rc::new(RefCell::new(TypeVarContent {
                 name: name.into(),
                 doc: doc.into(),
-                type_set: Rc::new(type_set),
+                type_set,
                 base: None,
-            }),
+            })),
         }
     }
 
@@ -89,20 +89,37 @@ impl TypeVar {
         TypeVar::new(name, doc, builder.finish())
     }
 
-    /// Returns this typevar's type set, maybe computing it from the parent.
-    fn get_typeset(&self) -> Rc {
-        // TODO Can this be done in a non-lazy way in derived() and we can remove this function and
-        // the one below?
-        match &self.content.base {
-            Some(base) => Rc::new(base.type_var.get_typeset().image(base.derived_func)),
-            None => self.content.type_set.clone(),
+    /// Get a fresh copy of self, named after `name`. Can only be called on non-derived typevars.
+    pub fn copy_from(other: &TypeVar, name: String) -> TypeVar {
+        assert!(
+            other.base.is_none(),
+            "copy_from() can only be called on non-derived type variables"
+        );
+        TypeVar {
+            content: Rc::new(RefCell::new(TypeVarContent {
+                name,
+                doc: "".into(),
+                type_set: other.type_set.clone(),
+                base: None,
+            })),
+        }
+    }
+
+    /// Returns the typeset for this TV. If the TV is derived, computes it recursively from the
+    /// derived function and the base's typeset.
+    /// Note this can't be done non-lazily in the constructor, because the TypeSet of the base may
+    /// change over time.
+    pub fn get_typeset(&self) -> TypeSet {
+        match &self.base {
+            Some(base) => base.type_var.get_typeset().image(base.derived_func),
+            None => self.type_set.clone(),
         }
     }
 
     /// Returns this typevar's type set, assuming this type var has no parent.
     pub fn get_raw_typeset(&self) -> &TypeSet {
-        assert_eq!(self.content.type_set, self.get_typeset());
-        &*self.content.type_set
+        assert_eq!(self.type_set, self.get_typeset());
+        &self.type_set
     }
 
     /// If the associated typeset has a single type return it. Otherwise return None.
@@ -117,7 +134,7 @@ impl TypeVar {
 
     /// Get the free type variable controlling this one.
     pub fn free_typevar(&self) -> Option {
-        match &self.content.base {
+        match &self.base {
             Some(base) => base.type_var.free_typevar(),
             None => {
                 match self.singleton_type() {
@@ -130,7 +147,7 @@ impl TypeVar {
     }
 
     /// Create a type variable that is a function of another.
-    fn derived(&self, derived_func: DerivedFunc) -> TypeVar {
+    pub fn derived(&self, derived_func: DerivedFunc) -> TypeVar {
         let ts = self.get_typeset();
 
         // Safety checks to avoid over/underflows.
@@ -182,7 +199,7 @@ impl TypeVar {
         }
 
         return TypeVar {
-            content: Rc::new(TypeVarContent {
+            content: Rc::new(RefCell::new(TypeVarContent {
                 name: format!("{}({})", derived_func.name(), self.name),
                 doc: "".into(),
                 type_set: ts,
@@ -190,7 +207,7 @@ impl TypeVar {
                     type_var: self.clone(),
                     derived_func,
                 }),
-            }),
+            })),
         };
     }
 
@@ -215,6 +232,52 @@ impl TypeVar {
     pub fn to_bitvec(&self) -> TypeVar {
         return self.derived(DerivedFunc::ToBitVec);
     }
+
+    /// Constrain the range of types this variable can assume to a subset of those in the typeset
+    /// ts.
+    /// May mutate itself if it's not derived, or its parent if it is.
+    pub fn constrain_types_by_ts(&self, type_set: TypeSet) {
+        match &self.base {
+            Some(base) => {
+                base.type_var
+                    .constrain_types_by_ts(type_set.preimage(base.derived_func));
+            }
+            None => {
+                self.content
+                    .borrow_mut()
+                    .type_set
+                    .inplace_intersect_with(&type_set);
+            }
+        }
+    }
+
+    /// Constrain the range of types this variable can assume to a subset of those `other` can
+    /// assume.
+    /// May mutate itself if it's not derived, or its parent if it is.
+    pub fn constrain_types(&self, other: TypeVar) {
+        if self == &other {
+            return;
+        }
+        self.constrain_types_by_ts(other.get_typeset());
+    }
+
+    /// Get a Rust expression that computes the type of this type variable.
+    pub fn to_rust_code(&self) -> String {
+        match &self.base {
+            Some(base) => format!(
+                "{}.{}()",
+                base.type_var.to_rust_code(),
+                base.derived_func.name()
+            ),
+            None => {
+                if let Some(singleton) = self.singleton_type() {
+                    singleton.rust_name()
+                } else {
+                    self.name.clone()
+                }
+            }
+        }
+    }
 }
 
 impl Into for &TypeVar {
@@ -228,24 +291,46 @@ impl Into for ValueType {
     }
 }
 
+// Hash TypeVars by pointers.
+// There might be a better way to do this, but since TypeVar's content (namely TypeSet) can be
+// mutated, it makes sense to use pointer equality/hashing here.
+impl hash::Hash for TypeVar {
+    fn hash(&self, h: &mut H) {
+        match &self.base {
+            Some(base) => {
+                base.type_var.hash(h);
+                base.derived_func.hash(h);
+            }
+            None => {
+                (&**self as *const TypeVarContent).hash(h);
+            }
+        }
+    }
+}
+
 impl PartialEq for TypeVar {
     fn eq(&self, other: &TypeVar) -> bool {
-        match (&self.content.base, &other.content.base) {
-            (Some(base1), Some(base2)) => base1.type_var.eq(&base2.type_var),
+        match (&self.base, &other.base) {
+            (Some(base1), Some(base2)) => {
+                base1.type_var.eq(&base2.type_var) && base1.derived_func == base2.derived_func
+            }
             (None, None) => Rc::ptr_eq(&self.content, &other.content),
             _ => false,
         }
     }
 }
 
+// Allow TypeVar as map keys, based on pointer equality (see also above PartialEq impl).
+impl Eq for TypeVar {}
+
 impl ops::Deref for TypeVar {
     type Target = TypeVarContent;
     fn deref(&self) -> &Self::Target {
-        &*self.content
+        unsafe { self.content.as_ptr().as_ref().unwrap() }
     }
 }
 
-#[derive(Clone, Copy, Debug)]
+#[derive(Clone, Copy, Debug, Hash, PartialEq)]
 pub enum DerivedFunc {
     LaneOf,
     AsBool,
@@ -268,9 +353,20 @@ impl DerivedFunc {
             DerivedFunc::ToBitVec => "to_bitvec",
         }
     }
+
+    /// Returns the inverse function of this one, if it is a bijection.
+    pub fn inverse(&self) -> Option {
+        match self {
+            DerivedFunc::HalfWidth => Some(DerivedFunc::DoubleWidth),
+            DerivedFunc::DoubleWidth => Some(DerivedFunc::HalfWidth),
+            DerivedFunc::HalfVector => Some(DerivedFunc::DoubleVector),
+            DerivedFunc::DoubleVector => Some(DerivedFunc::HalfVector),
+            _ => None,
+        }
+    }
 }
 
-#[derive(Debug)]
+#[derive(Debug, Hash)]
 pub struct TypeVarParent {
     pub type_var: TypeVar,
     pub derived_func: DerivedFunc,
@@ -735,6 +831,18 @@ impl TypeSetBuilder {
             self.specials,
         )
     }
+
+    pub fn all() -> TypeSet {
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .floats(Interval::All)
+            .bools(Interval::All)
+            .simd_lanes(Interval::All)
+            .bitvecs(Interval::All)
+            .specials(ValueType::all_special_types().collect())
+            .includes_scalars(true)
+            .finish()
+    }
 }
 
 #[derive(PartialEq)]

From 494f3abf1dd325fd6070b9f1ef8cc42f1c5fcc2e Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 18 Apr 2019 18:16:38 +0200
Subject: [PATCH 2406/3084] [meta] Add type inference, transforms and AST
 helpers for legalization;

---
 cranelift/codegen/meta/src/cdsl/ast.rs        | 653 +++++++++++++++++
 cranelift/codegen/meta/src/cdsl/mod.rs        |   3 +
 .../codegen/meta/src/cdsl/type_inference.rs   | 655 +++++++++++++++++-
 cranelift/codegen/meta/src/cdsl/xform.rs      | 416 +++++++++++
 4 files changed, 1726 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/codegen/meta/src/cdsl/ast.rs
 create mode 100644 cranelift/codegen/meta/src/cdsl/xform.rs

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
new file mode 100644
index 0000000000..1df8ea3050
--- /dev/null
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -0,0 +1,653 @@
+use crate::cdsl::formats::FormatRegistry;
+use crate::cdsl::inst::{BoundInstruction, Instruction, InstructionPredicate};
+use crate::cdsl::operands::{OperandKind, OperandKindFields};
+use crate::cdsl::types::{LaneType, ValueType};
+use crate::cdsl::typevar::{TypeSetBuilder, TypeVar};
+
+use cranelift_entity::{entity_impl, PrimaryMap};
+
+use std::fmt;
+
+pub enum Expr {
+    Var(VarIndex),
+    Literal(Literal),
+    Apply(Apply),
+}
+
+impl Expr {
+    pub fn maybe_literal(&self) -> Option<&Literal> {
+        match &self {
+            Expr::Literal(lit) => Some(lit),
+            _ => None,
+        }
+    }
+
+    pub fn maybe_var(&self) -> Option {
+        if let Expr::Var(var) = &self {
+            Some(*var)
+        } else {
+            None
+        }
+    }
+
+    pub fn unwrap_var(&self) -> VarIndex {
+        self.maybe_var()
+            .expect("tried to unwrap a non-Var content in Expr::unwrap_var")
+    }
+
+    pub fn to_rust_code(&self, var_pool: &VarPool) -> String {
+        match self {
+            Expr::Var(var_index) => var_pool.get(*var_index).to_rust_code(),
+            Expr::Literal(literal) => literal.to_rust_code(),
+            Expr::Apply(a) => a.to_rust_code(var_pool),
+        }
+    }
+}
+
+/// An AST definition associates a set of variables with the values produced by an expression.
+pub struct Def {
+    pub apply: Apply,
+    pub defined_vars: Vec,
+}
+
+impl Def {
+    pub fn to_comment_string(&self, var_pool: &VarPool) -> String {
+        let results = self
+            .defined_vars
+            .iter()
+            .map(|&x| var_pool.get(x).name)
+            .collect::>();
+
+        let results = if results.len() == 1 {
+            results[0].to_string()
+        } else {
+            format!("({})", results.join(", "))
+        };
+
+        format!("{} << {}", results, self.apply.to_comment_string(var_pool))
+    }
+}
+
+pub struct DefPool {
+    pool: PrimaryMap,
+}
+
+impl DefPool {
+    pub fn new() -> Self {
+        Self {
+            pool: PrimaryMap::new(),
+        }
+    }
+    pub fn get(&self, index: DefIndex) -> &Def {
+        self.pool.get(index).unwrap()
+    }
+    pub fn get_mut(&mut self, index: DefIndex) -> &mut Def {
+        self.pool.get_mut(index).unwrap()
+    }
+    pub fn next_index(&self) -> DefIndex {
+        self.pool.next_key()
+    }
+    pub fn create(&mut self, apply: Apply, defined_vars: Vec) -> DefIndex {
+        self.pool.push(Def {
+            apply,
+            defined_vars,
+        })
+    }
+}
+
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct DefIndex(u32);
+entity_impl!(DefIndex);
+
+#[derive(Debug, Clone)]
+enum LiteralValue {
+    /// A value of an enumerated immediate operand.
+    ///
+    /// Some immediate operand kinds like `intcc` and `floatcc` have an enumerated range of values
+    /// corresponding to a Rust enum type. An `Enumerator` object is an AST leaf node representing one
+    /// of the values.
+    Enumerator(&'static str),
+
+    /// A bitwise value of an immediate operand, used for bitwise exact floating point constants.
+    Bits(u64),
+
+    /// A value of an integer immediate operand.
+    Int(i64),
+}
+
+#[derive(Clone)]
+pub struct Literal {
+    kind: OperandKind,
+    value: LiteralValue,
+}
+
+impl fmt::Debug for Literal {
+    fn fmt(&self, fmt: &mut fmt::Formatter) -> Result<(), fmt::Error> {
+        write!(
+            fmt,
+            "Literal(kind={}, value={:?})",
+            self.kind.name, self.value
+        )
+    }
+}
+
+impl Literal {
+    pub fn enumerator_for(kind: &OperandKind, value: &'static str) -> Self {
+        if let OperandKindFields::ImmEnum(values) = &kind.fields {
+            assert!(
+                values.get(value).is_some(),
+                format!(
+                    "nonexistent value '{}' in enumeration '{}'",
+                    value, kind.name
+                )
+            );
+        } else {
+            panic!("enumerator is for enum values");
+        }
+        Self {
+            kind: kind.clone(),
+            value: LiteralValue::Enumerator(value),
+        }
+    }
+
+    pub fn bits(kind: &OperandKind, bits: u64) -> Self {
+        match kind.fields {
+            OperandKindFields::ImmValue => {}
+            _ => panic!("bits_of is for immediate scalar types"),
+        }
+        Self {
+            kind: kind.clone(),
+            value: LiteralValue::Bits(bits),
+        }
+    }
+
+    pub fn constant(kind: &OperandKind, value: i64) -> Self {
+        match kind.fields {
+            OperandKindFields::ImmValue => {}
+            _ => panic!("bits_of is for immediate scalar types"),
+        }
+        Self {
+            kind: kind.clone(),
+            value: LiteralValue::Int(value),
+        }
+    }
+
+    pub fn to_rust_code(&self) -> String {
+        let maybe_values = match &self.kind.fields {
+            OperandKindFields::ImmEnum(values) => Some(values),
+            OperandKindFields::ImmValue => None,
+            _ => panic!("impossible per construction"),
+        };
+
+        match self.value {
+            LiteralValue::Enumerator(value) => {
+                format!("{}::{}", self.kind.rust_type, maybe_values.unwrap()[value])
+            }
+            LiteralValue::Bits(bits) => format!("{}::with_bits({:#x})", self.kind.rust_type, bits),
+            LiteralValue::Int(val) => val.to_string(),
+        }
+    }
+}
+
+#[derive(Clone, Copy, Debug)]
+pub enum PatternPosition {
+    Source,
+    Destination,
+}
+
+/// A free variable.
+///
+/// When variables are used in `XForms` with source and destination patterns, they are classified
+/// as follows:
+///
+/// Input values: Uses in the source pattern with no preceding def. These may appear as inputs in
+/// the destination pattern too, but no new inputs can be introduced.
+///
+/// Output values: Variables that are defined in both the source and destination pattern.  These
+/// values may have uses outside the source pattern, and the destination pattern must compute the
+/// same value.
+///
+/// Intermediate values: Values that are defined in the source pattern, but not in the destination
+/// pattern. These may have uses outside the source pattern, so the defining instruction can't be
+/// deleted immediately.
+///
+/// Temporary values are defined only in the destination pattern.
+pub struct Var {
+    pub name: &'static str,
+
+    /// The `Def` defining this variable in a source pattern.
+    pub src_def: Option,
+
+    /// The `Def` defining this variable in a destination pattern.
+    pub dst_def: Option,
+
+    /// TypeVar representing the type of this variable.
+    type_var: Option,
+
+    /// Is this the original type variable, or has it be redefined with set_typevar?
+    is_original_type_var: bool,
+}
+
+impl Var {
+    fn new(name: &'static str) -> Self {
+        Self {
+            name,
+            src_def: None,
+            dst_def: None,
+            type_var: None,
+            is_original_type_var: false,
+        }
+    }
+
+    /// Is this an input value to the src pattern?
+    pub fn is_input(&self) -> bool {
+        self.src_def.is_none() && self.dst_def.is_none()
+    }
+
+    /// Is this an output value, defined in both src and dst patterns?
+    pub fn is_output(&self) -> bool {
+        self.src_def.is_some() && self.dst_def.is_some()
+    }
+
+    /// Is this an intermediate value, defined only in the src pattern?
+    pub fn is_intermediate(&self) -> bool {
+        self.src_def.is_some() && self.dst_def.is_none()
+    }
+
+    /// Is this a temp value, defined only in the dst pattern?
+    pub fn is_temp(&self) -> bool {
+        self.src_def.is_none() && self.dst_def.is_some()
+    }
+
+    /// Get the def of this variable according to the position.
+    pub fn get_def(&self, position: PatternPosition) -> Option {
+        match position {
+            PatternPosition::Source => self.src_def,
+            PatternPosition::Destination => self.dst_def,
+        }
+    }
+
+    pub fn set_def(&mut self, position: PatternPosition, def: DefIndex) {
+        assert!(
+            self.get_def(position).is_none(),
+            format!("redefinition of variable {}", self.name)
+        );
+        match position {
+            PatternPosition::Source => {
+                self.src_def = Some(def);
+            }
+            PatternPosition::Destination => {
+                self.dst_def = Some(def);
+            }
+        }
+    }
+
+    /// Get the type variable representing the type of this variable.
+    pub fn get_or_create_typevar(&mut self) -> TypeVar {
+        match &self.type_var {
+            Some(tv) => tv.clone(),
+            None => {
+                // Create a new type var in which we allow all types.
+                let tv = TypeVar::new(
+                    format!("typeof_{}", self.name),
+                    format!("Type of the pattern variable {:?}", self),
+                    TypeSetBuilder::all(),
+                );
+                self.type_var = Some(tv.clone());
+                self.is_original_type_var = true;
+                tv
+            }
+        }
+    }
+    pub fn get_typevar(&self) -> Option {
+        self.type_var.clone()
+    }
+    pub fn set_typevar(&mut self, tv: TypeVar) {
+        self.is_original_type_var = if let Some(previous_tv) = &self.type_var {
+            *previous_tv == tv
+        } else {
+            false
+        };
+        self.type_var = Some(tv);
+    }
+
+    /// Check if this variable has a free type variable. If not, the type of this variable is
+    /// computed from the type of another variable.
+    pub fn has_free_typevar(&self) -> bool {
+        match &self.type_var {
+            Some(tv) => tv.base.is_none() && self.is_original_type_var,
+            None => false,
+        }
+    }
+
+    pub fn to_rust_code(&self) -> String {
+        self.name.into()
+    }
+    fn rust_type(&self) -> String {
+        self.type_var.as_ref().unwrap().to_rust_code()
+    }
+}
+
+impl fmt::Debug for Var {
+    fn fmt(&self, fmt: &mut fmt::Formatter) -> Result<(), fmt::Error> {
+        fmt.write_fmt(format_args!(
+            "Var({}{}{})",
+            self.name,
+            if self.src_def.is_some() { ", src" } else { "" },
+            if self.dst_def.is_some() { ", dst" } else { "" }
+        ))
+    }
+}
+
+#[derive(Copy, Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct VarIndex(u32);
+entity_impl!(VarIndex);
+
+pub struct VarPool {
+    pool: PrimaryMap,
+}
+
+impl VarPool {
+    pub fn new() -> Self {
+        Self {
+            pool: PrimaryMap::new(),
+        }
+    }
+    pub fn get(&self, index: VarIndex) -> &Var {
+        self.pool.get(index).unwrap()
+    }
+    pub fn get_mut(&mut self, index: VarIndex) -> &mut Var {
+        self.pool.get_mut(index).unwrap()
+    }
+    pub fn create(&mut self, name: &'static str) -> VarIndex {
+        self.pool.push(Var::new(name))
+    }
+}
+
+pub enum ApplyTarget {
+    Inst(Instruction),
+    Bound(BoundInstruction),
+}
+
+impl ApplyTarget {
+    pub fn inst(&self) -> &Instruction {
+        match &self {
+            ApplyTarget::Inst(inst) => inst,
+            ApplyTarget::Bound(bound_inst) => &bound_inst.inst,
+        }
+    }
+}
+
+impl Into for &Instruction {
+    fn into(self) -> ApplyTarget {
+        ApplyTarget::Inst(self.clone())
+    }
+}
+
+impl Into for BoundInstruction {
+    fn into(self) -> ApplyTarget {
+        ApplyTarget::Bound(self)
+    }
+}
+
+pub fn bind(target: impl Into, lane_type: impl Into) -> BoundInstruction {
+    let value_type = ValueType::from(lane_type.into());
+
+    let (inst, value_types) = match target.into() {
+        ApplyTarget::Inst(inst) => (inst, vec![value_type]),
+        ApplyTarget::Bound(bound_inst) => {
+            let mut new_value_types = bound_inst.value_types;
+            new_value_types.push(value_type);
+            (bound_inst.inst, new_value_types)
+        }
+    };
+
+    match &inst.polymorphic_info {
+        Some(poly) => {
+            assert!(
+                value_types.len() <= 1 + poly.other_typevars.len(),
+                format!("trying to bind too many types for {}", inst.name)
+            );
+        }
+        None => {
+            panic!(format!(
+                "trying to bind a type for {} which is not a polymorphic instruction",
+                inst.name
+            ));
+        }
+    }
+
+    BoundInstruction { inst, value_types }
+}
+
+/// Apply an instruction to arguments.
+///
+/// An `Apply` AST expression is created by using function call syntax on instructions. This
+/// applies to both bound and unbound polymorphic instructions.
+pub struct Apply {
+    pub inst: Instruction,
+    pub args: Vec,
+    pub value_types: Vec,
+}
+
+impl Apply {
+    pub fn new(target: ApplyTarget, args: Vec) -> Self {
+        let (inst, value_types) = match target.into() {
+            ApplyTarget::Inst(inst) => (inst, Vec::new()),
+            ApplyTarget::Bound(bound_inst) => (bound_inst.inst, bound_inst.value_types),
+        };
+
+        // Basic check on number of arguments.
+        assert!(
+            inst.operands_in.len() == args.len(),
+            format!("incorrect number of arguments in instruction {}", inst.name)
+        );
+
+        // Check that the kinds of Literals arguments match the expected operand.
+        for &imm_index in &inst.imm_opnums {
+            let arg = &args[imm_index];
+            if let Some(literal) = arg.maybe_literal() {
+                let op = &inst.operands_in[imm_index];
+                assert!(
+                    op.kind.name == literal.kind.name,
+                    format!(
+                        "Passing literal of kind {} to field of wrong kind {}",
+                        literal.kind.name, op.kind.name
+                    )
+                );
+            }
+        }
+
+        Self {
+            inst,
+            args,
+            value_types,
+        }
+    }
+
+    fn to_comment_string(&self, var_pool: &VarPool) -> String {
+        let args = self
+            .args
+            .iter()
+            .map(|arg| arg.to_rust_code(var_pool))
+            .collect::>()
+            .join(", ");
+
+        let mut inst_and_bound_types = vec![self.inst.name.to_string()];
+        inst_and_bound_types.extend(self.value_types.iter().map(|vt| vt.to_string()));
+        let inst_name = inst_and_bound_types.join(".");
+
+        format!("{}({})", inst_name, args)
+    }
+
+    fn to_rust_code(&self, var_pool: &VarPool) -> String {
+        let args = self
+            .args
+            .iter()
+            .map(|arg| arg.to_rust_code(var_pool))
+            .collect::>()
+            .join(", ");
+        format!("{}({})", self.inst.name, args)
+    }
+
+    fn inst_predicate(
+        &self,
+        format_registry: &FormatRegistry,
+        var_pool: &VarPool,
+    ) -> InstructionPredicate {
+        let iform = format_registry.get(self.inst.format);
+
+        let mut pred = InstructionPredicate::new();
+        for (format_field, &op_num) in iform.imm_fields.iter().zip(self.inst.imm_opnums.iter()) {
+            let arg = &self.args[op_num];
+            if arg.maybe_var().is_some() {
+                // Ignore free variables for now.
+                continue;
+            }
+            pred = pred.and(InstructionPredicate::new_is_field_equal(
+                &format_field,
+                arg.to_rust_code(var_pool),
+            ));
+        }
+
+        // Add checks for any bound secondary type variables.  We can't check the controlling type
+        // variable this way since it may not appear as the type of an operand.
+        if self.value_types.len() > 1 {
+            let poly = self
+                .inst
+                .polymorphic_info
+                .as_ref()
+                .expect("must have polymorphic info if it has bounded types");
+            for (bound_type, type_var) in
+                self.value_types[1..].iter().zip(poly.other_typevars.iter())
+            {
+                pred = pred.and(InstructionPredicate::new_typevar_check(
+                    &self.inst, type_var, bound_type,
+                ));
+            }
+        }
+
+        pred
+    }
+
+    /// Same as `inst_predicate()`, but also check the controlling type variable.
+    pub fn inst_predicate_with_ctrl_typevar(
+        &self,
+        format_registry: &FormatRegistry,
+        var_pool: &VarPool,
+    ) -> InstructionPredicate {
+        let mut pred = self.inst_predicate(format_registry, var_pool);
+
+        if !self.value_types.is_empty() {
+            let bound_type = &self.value_types[0];
+            let poly = self.inst.polymorphic_info.as_ref().unwrap();
+            let type_check = if poly.use_typevar_operand {
+                InstructionPredicate::new_typevar_check(&self.inst, &poly.ctrl_typevar, bound_type)
+            } else {
+                InstructionPredicate::new_ctrl_typevar_check(&bound_type)
+            };
+            pred = pred.and(type_check);
+        }
+
+        pred
+    }
+
+    pub fn rust_builder(&self, defined_vars: &Vec, var_pool: &VarPool) -> String {
+        let mut args = self
+            .args
+            .iter()
+            .map(|expr| expr.to_rust_code(var_pool))
+            .collect::>()
+            .join(", ");
+
+        // Do we need to pass an explicit type argument?
+        if let Some(poly) = &self.inst.polymorphic_info {
+            if !poly.use_typevar_operand {
+                args = format!("{}, {}", var_pool.get(defined_vars[0]).rust_type(), args);
+            }
+        }
+
+        format!("{}({})", self.inst.snake_name(), args)
+    }
+}
+
+// Simple helpers for legalize actions construction.
+
+pub enum DummyExpr {
+    Var(DummyVar),
+    Literal(Literal),
+    Apply(ApplyTarget, Vec),
+}
+
+#[derive(Clone)]
+pub struct DummyVar {
+    pub name: &'static str,
+}
+
+impl Into for DummyVar {
+    fn into(self) -> DummyExpr {
+        DummyExpr::Var(self)
+    }
+}
+impl Into for Literal {
+    fn into(self) -> DummyExpr {
+        DummyExpr::Literal(self)
+    }
+}
+
+pub fn var(name: &'static str) -> DummyVar {
+    DummyVar { name }
+}
+
+pub struct DummyDef {
+    pub expr: DummyExpr,
+    pub defined_vars: Vec,
+}
+
+pub struct ExprBuilder {
+    expr: DummyExpr,
+}
+
+impl ExprBuilder {
+    pub fn apply(inst: ApplyTarget, args: Vec) -> Self {
+        let expr = DummyExpr::Apply(inst, args);
+        Self { expr }
+    }
+
+    pub fn assign_to(self, defined_vars: Vec) -> DummyDef {
+        DummyDef {
+            expr: self.expr,
+            defined_vars,
+        }
+    }
+}
+
+macro_rules! def_rhs {
+    // inst(a, b, c)
+    ($inst:ident($($src:expr),*)) => {
+        ExprBuilder::apply($inst.into(), vec![$($src.clone().into()),*])
+    };
+
+    // inst.type(a, b, c)
+    ($inst:ident.$type:ident($($src:expr),*)) => {
+        ExprBuilder::apply(bind($inst, $type).into(), vec![$($src.clone().into()),*])
+    };
+}
+
+// Helper macro to define legalization recipes.
+macro_rules! def {
+    // x = ...
+    ($dest:ident = $($tt:tt)*) => {
+        def_rhs!($($tt)*).assign_to(vec![$dest.clone()])
+    };
+
+    // (x, y, ...) = ...
+    (($($dest:ident),*) = $($tt:tt)*) => {
+        def_rhs!($($tt)*).assign_to(vec![$($dest.clone()),*])
+    };
+
+    // An instruction with no results.
+    ($($tt:tt)*) => {
+        def_rhs!($($tt)*).assign_to(Vec::new())
+    }
+}
diff --git a/cranelift/codegen/meta/src/cdsl/mod.rs b/cranelift/codegen/meta/src/cdsl/mod.rs
index 540370624a..a6c8be95fe 100644
--- a/cranelift/codegen/meta/src/cdsl/mod.rs
+++ b/cranelift/codegen/meta/src/cdsl/mod.rs
@@ -3,6 +3,8 @@
 //! This module defines the classes that are used to define Cranelift
 //! instructions and other entities.
 
+#[macro_use]
+pub mod ast;
 pub mod formats;
 pub mod inst;
 pub mod isa;
@@ -12,6 +14,7 @@ pub mod settings;
 pub mod type_inference;
 pub mod types;
 pub mod typevar;
+pub mod xform;
 
 /// A macro that converts boolean settings into predicates to look more natural.
 #[macro_export]
diff --git a/cranelift/codegen/meta/src/cdsl/type_inference.rs b/cranelift/codegen/meta/src/cdsl/type_inference.rs
index de104f4b47..e3a1e9bb06 100644
--- a/cranelift/codegen/meta/src/cdsl/type_inference.rs
+++ b/cranelift/codegen/meta/src/cdsl/type_inference.rs
@@ -1,5 +1,658 @@
-use crate::cdsl::typevar::TypeVar;
+use crate::cdsl::ast::{Def, DefIndex, DefPool, Var, VarIndex, VarPool};
+use crate::cdsl::typevar::{DerivedFunc, TypeSet, TypeVar};
 
+use std::collections::{HashMap, HashSet};
+use std::iter::FromIterator;
+
+#[derive(Hash, PartialEq, Eq)]
 pub enum Constraint {
+    /// Constraint specifying that a type var tv1 must be wider than or equal to type var tv2 at
+    /// runtime. This requires that:
+    /// 1) They have the same number of lanes
+    /// 2) In a lane tv1 has at least as many bits as tv2.
     WiderOrEq(TypeVar, TypeVar),
+
+    /// Constraint specifying that two derived type vars must have the same runtime type.
+    Eq(TypeVar, TypeVar),
+
+    /// Constraint specifying that a type var must belong to some typeset.
+    InTypeset(TypeVar, TypeSet),
+}
+
+impl Constraint {
+    fn translate_with TypeVar>(&self, func: F) -> Constraint {
+        match self {
+            Constraint::WiderOrEq(lhs, rhs) => {
+                let lhs = func(&lhs);
+                let rhs = func(&rhs);
+                Constraint::WiderOrEq(lhs, rhs)
+            }
+            Constraint::Eq(lhs, rhs) => {
+                let lhs = func(&lhs);
+                let rhs = func(&rhs);
+                Constraint::Eq(lhs, rhs)
+            }
+            Constraint::InTypeset(tv, ts) => {
+                let tv = func(&tv);
+                Constraint::InTypeset(tv, ts.clone())
+            }
+        }
+    }
+
+    /// Creates a new constraint by replacing type vars by their hashmap equivalent.
+    fn translate_with_map(
+        &self,
+        original_to_own_typevar: &HashMap<&TypeVar, TypeVar>,
+    ) -> Constraint {
+        self.translate_with(|tv| substitute(original_to_own_typevar, tv))
+    }
+
+    /// Creates a new constraint by replacing type vars by their canonical equivalent.
+    fn translate_with_env(&self, type_env: &TypeEnvironment) -> Constraint {
+        self.translate_with(|tv| type_env.get_equivalent(tv))
+    }
+
+    fn is_trivial(&self) -> bool {
+        match self {
+            Constraint::WiderOrEq(lhs, rhs) => {
+                // Trivially true.
+                if lhs == rhs {
+                    return true;
+                }
+
+                let ts1 = lhs.get_typeset();
+                let ts2 = rhs.get_typeset();
+
+                // Trivially true.
+                if ts1.is_wider_or_equal(&ts2) {
+                    return true;
+                }
+
+                // Trivially false.
+                if ts1.is_narrower(&ts2) {
+                    return true;
+                }
+
+                // Trivially false.
+                if (&ts1.lanes & &ts2.lanes).len() == 0 {
+                    return true;
+                }
+
+                self.is_concrete()
+            }
+            Constraint::Eq(lhs, rhs) => lhs == rhs || self.is_concrete(),
+            Constraint::InTypeset(_, _) => {
+                // The way InTypeset are made, they would always be trivial if we were applying the
+                // same logic as the Python code did, so ignore this.
+                self.is_concrete()
+            }
+        }
+    }
+
+    /// Returns true iff all the referenced type vars are singletons.
+    fn is_concrete(&self) -> bool {
+        match self {
+            Constraint::WiderOrEq(lhs, rhs) => {
+                lhs.singleton_type().is_some() && rhs.singleton_type().is_some()
+            }
+            Constraint::Eq(lhs, rhs) => {
+                lhs.singleton_type().is_some() && rhs.singleton_type().is_some()
+            }
+            Constraint::InTypeset(tv, _) => tv.singleton_type().is_some(),
+        }
+    }
+
+    fn typevar_args(&self) -> Vec<&TypeVar> {
+        match self {
+            Constraint::WiderOrEq(lhs, rhs) => vec![lhs, rhs],
+            Constraint::Eq(lhs, rhs) => vec![lhs, rhs],
+            Constraint::InTypeset(tv, _) => vec![tv],
+        }
+    }
+}
+
+#[derive(Clone, Copy)]
+enum TypeEnvRank {
+    Singleton = 5,
+    Input = 4,
+    Intermediate = 3,
+    Output = 2,
+    Temp = 1,
+    Internal = 0,
+}
+
+/// Class encapsulating the necessary bookkeeping for type inference.
+pub struct TypeEnvironment {
+    vars: HashSet,
+    ranks: HashMap,
+    equivalency_map: HashMap,
+    pub constraints: Vec,
+}
+
+impl TypeEnvironment {
+    fn new() -> Self {
+        TypeEnvironment {
+            vars: HashSet::new(),
+            ranks: HashMap::new(),
+            equivalency_map: HashMap::new(),
+            constraints: Vec::new(),
+        }
+    }
+
+    fn register(&mut self, var_index: VarIndex, var: &mut Var) {
+        self.vars.insert(var_index);
+        let rank = if var.is_input() {
+            TypeEnvRank::Input
+        } else if var.is_intermediate() {
+            TypeEnvRank::Intermediate
+        } else if var.is_output() {
+            TypeEnvRank::Output
+        } else {
+            assert!(var.is_temp());
+            TypeEnvRank::Temp
+        };
+        self.ranks.insert(var.get_or_create_typevar(), rank);
+    }
+
+    fn add_constraint(&mut self, constraint: Constraint) {
+        if self
+            .constraints
+            .iter()
+            .find(|&item| item == &constraint)
+            .is_some()
+        {
+            return;
+        }
+
+        // Check extra conditions for InTypeset constraints.
+        if let Constraint::InTypeset(tv, _) = &constraint {
+            assert!(tv.base.is_none());
+            assert!(tv.name.starts_with("typeof_"));
+        }
+
+        self.constraints.push(constraint);
+    }
+
+    /// Returns the canonical representative of the equivalency class of the given argument, or
+    /// duplicates it if it's not there yet.
+    pub fn get_equivalent(&self, tv: &TypeVar) -> TypeVar {
+        let mut tv = tv;
+        while let Some(found) = self.equivalency_map.get(tv) {
+            tv = found;
+        }
+        match &tv.base {
+            Some(parent) => self
+                .get_equivalent(&parent.type_var)
+                .derived(parent.derived_func),
+            None => tv.clone(),
+        }
+    }
+
+    /// Get the rank of tv in the partial order:
+    /// - TVs directly associated with a Var get their rank from the Var (see register()).
+    /// - Internally generated non-derived TVs implicitly get the lowest rank (0).
+    /// - Derived variables get their rank from their free typevar.
+    /// - Singletons have the highest rank.
+    /// - TVs associated with vars in a source pattern have a higher rank than TVs associated with
+    /// temporary vars.
+    fn rank(&self, tv: &TypeVar) -> u8 {
+        let actual_tv = match tv.base {
+            Some(_) => tv.free_typevar(),
+            None => Some(tv.clone()),
+        };
+
+        let rank = match actual_tv {
+            Some(actual_tv) => match self.ranks.get(&actual_tv) {
+                Some(rank) => Some(*rank),
+                None => {
+                    assert!(
+                        !actual_tv.name.starts_with("typeof_"),
+                        format!("variable {} should be explicitly ranked", actual_tv.name)
+                    );
+                    None
+                }
+            },
+            None => None,
+        };
+
+        let rank = match rank {
+            Some(rank) => rank,
+            None => {
+                if tv.singleton_type().is_some() {
+                    TypeEnvRank::Singleton
+                } else {
+                    TypeEnvRank::Internal
+                }
+            }
+        };
+
+        rank as u8
+    }
+
+    /// Record the fact that the free tv1 is part of the same equivalence class as tv2. The
+    /// canonical representative of the merged class is tv2's canonical representative.
+    fn record_equivalent(&mut self, tv1: TypeVar, tv2: TypeVar) {
+        assert!(tv1.base.is_none());
+        assert!(self.get_equivalent(&tv1) == tv1);
+        if let Some(tv2_base) = &tv2.base {
+            // Ensure there are no cycles.
+            assert!(self.get_equivalent(&tv2_base.type_var) != tv1);
+        }
+        self.equivalency_map.insert(tv1, tv2);
+    }
+
+    /// Get the free typevars in the current type environment.
+    pub fn free_typevars(&self, var_pool: &mut VarPool) -> Vec {
+        let mut typevars = Vec::new();
+        typevars.extend(self.equivalency_map.keys().cloned());
+        typevars.extend(
+            self.vars
+                .iter()
+                .map(|&var_index| var_pool.get_mut(var_index).get_or_create_typevar()),
+        );
+
+        let set: HashSet = HashSet::from_iter(
+            typevars
+                .iter()
+                .map(|tv| self.get_equivalent(tv).free_typevar())
+                .filter(|opt_tv| {
+                    // Filter out singleton types.
+                    return opt_tv.is_some();
+                })
+                .map(|tv| tv.unwrap()),
+        );
+        Vec::from_iter(set)
+    }
+
+    /// Normalize by collapsing any roots that don't correspond to a concrete type var AND have a
+    /// single type var derived from them or equivalent to them.
+    ///
+    /// e.g. if we have a root of the tree that looks like:
+    ///
+    ///   typeof_a   typeof_b
+    ///          \\  /
+    ///       typeof_x
+    ///           |
+    ///         half_width(1)
+    ///           |
+    ///           1
+    ///
+    /// we want to collapse the linear path between 1 and typeof_x. The resulting graph is:
+    ///
+    ///   typeof_a   typeof_b
+    ///          \\  /
+    ///       typeof_x
+    fn normalize(&mut self, var_pool: &mut VarPool) {
+        let source_tvs: HashSet = HashSet::from_iter(
+            self.vars
+                .iter()
+                .map(|&var_index| var_pool.get_mut(var_index).get_or_create_typevar()),
+        );
+
+        let mut children: HashMap> = HashMap::new();
+
+        // Insert all the parents found by the derivation relationship.
+        for type_var in self.equivalency_map.values() {
+            if type_var.base.is_none() {
+                continue;
+            }
+
+            let parent_tv = type_var.free_typevar();
+            if parent_tv.is_none() {
+                // Ignore this type variable, it's a singleton.
+                continue;
+            }
+            let parent_tv = parent_tv.unwrap();
+
+            children
+                .entry(parent_tv)
+                .or_insert(HashSet::new())
+                .insert(type_var.clone());
+        }
+
+        // Insert all the explicit equivalency links.
+        for (equivalent_tv, canon_tv) in self.equivalency_map.iter() {
+            children
+                .entry(canon_tv.clone())
+                .or_insert(HashSet::new())
+                .insert(equivalent_tv.clone());
+        }
+
+        // Remove links that are straight paths up to typevar of variables.
+        for free_root in self.free_typevars(var_pool) {
+            let mut root = &free_root;
+            while !source_tvs.contains(&root)
+                && children.contains_key(&root)
+                && children.get(&root).unwrap().len() == 1
+            {
+                let child = children.get(&root).unwrap().iter().next().unwrap();
+                assert_eq!(self.equivalency_map[child], root.clone());
+                self.equivalency_map.remove(child);
+                root = child;
+            }
+        }
+    }
+
+    /// Extract a clean type environment from self, that only mentions type vars associated with
+    /// real variables.
+    fn extract(self, var_pool: &mut VarPool) -> TypeEnvironment {
+        let vars_tv: HashSet = HashSet::from_iter(
+            self.vars
+                .iter()
+                .map(|&var_index| var_pool.get_mut(var_index).get_or_create_typevar()),
+        );
+
+        let mut new_equivalency_map: HashMap = HashMap::new();
+        for tv in &vars_tv {
+            let canon_tv = self.get_equivalent(tv);
+            if *tv != canon_tv {
+                new_equivalency_map.insert(tv.clone(), canon_tv.clone());
+            }
+
+            // Sanity check: the translated type map should only refer to real variables.
+            assert!(vars_tv.contains(tv));
+            let canon_free_tv = canon_tv.free_typevar();
+            assert!(canon_free_tv.is_none() || vars_tv.contains(&canon_free_tv.unwrap()));
+        }
+
+        let mut new_constraints: HashSet = HashSet::new();
+        for constraint in &self.constraints {
+            let constraint = constraint.translate_with_env(&self);
+            if constraint.is_trivial() || new_constraints.contains(&constraint) {
+                continue;
+            }
+
+            // Sanity check: translated constraints should refer only to real variables.
+            for arg in constraint.typevar_args() {
+                assert!(vars_tv.contains(arg));
+                let arg_free_tv = arg.free_typevar();
+                assert!(arg_free_tv.is_none() || vars_tv.contains(&arg_free_tv.unwrap()));
+            }
+
+            new_constraints.insert(constraint);
+        }
+
+        TypeEnvironment {
+            vars: self.vars,
+            ranks: self.ranks,
+            equivalency_map: new_equivalency_map,
+            constraints: Vec::from_iter(new_constraints),
+        }
+    }
+}
+
+/// Replaces an external type variable according to the following rules:
+/// - if a local copy is present in the map, return it.
+/// - or if it's derived, create a local derived one that recursively substitutes the parent.
+/// - or return itself.
+fn substitute(map: &HashMap<&TypeVar, TypeVar>, external_type_var: &TypeVar) -> TypeVar {
+    match map.get(&external_type_var) {
+        Some(own_type_var) => own_type_var.clone(),
+        None => match &external_type_var.base {
+            Some(parent) => {
+                let parent_substitute = substitute(map, &parent.type_var);
+                TypeVar::derived(&parent_substitute, parent.derived_func)
+            }
+            None => external_type_var.clone(),
+        },
+    }
+}
+
+/// Normalize a (potentially derived) typevar using the following rules:
+///
+/// - vector and width derived functions commute
+///     {HALF,DOUBLE}VECTOR({HALF,DOUBLE}WIDTH(base)) ->
+///     {HALF,DOUBLE}WIDTH({HALF,DOUBLE}VECTOR(base))
+///
+/// - half/double pairs collapse
+///     {HALF,DOUBLE}WIDTH({DOUBLE,HALF}WIDTH(base)) -> base
+///     {HALF,DOUBLE}VECTOR({DOUBLE,HALF}VECTOR(base)) -> base
+fn canonicalize_derivations(tv: TypeVar) -> TypeVar {
+    let base = match &tv.base {
+        Some(base) => base,
+        None => return tv,
+    };
+
+    let derived_func = base.derived_func;
+
+    if let Some(base_base) = &base.type_var.base {
+        let base_base_tv = &base_base.type_var;
+        match (derived_func, base_base.derived_func) {
+            (DerivedFunc::HalfWidth, DerivedFunc::DoubleWidth)
+            | (DerivedFunc::DoubleWidth, DerivedFunc::HalfWidth)
+            | (DerivedFunc::HalfVector, DerivedFunc::DoubleVector)
+            | (DerivedFunc::DoubleVector, DerivedFunc::HalfVector) => {
+                // Cancelling bijective transformations. This doesn't hide any overflow issues
+                // since derived type sets are checked upon derivaion, and base typesets are only
+                // allowed to shrink.
+                return canonicalize_derivations(base_base_tv.clone());
+            }
+            (DerivedFunc::HalfWidth, DerivedFunc::HalfVector)
+            | (DerivedFunc::HalfWidth, DerivedFunc::DoubleVector)
+            | (DerivedFunc::DoubleWidth, DerivedFunc::DoubleVector)
+            | (DerivedFunc::DoubleWidth, DerivedFunc::HalfVector) => {
+                // Arbitrarily put WIDTH derivations before VECTOR derivations, since they commute.
+                return canonicalize_derivations(
+                    base_base_tv
+                        .derived(derived_func)
+                        .derived(base_base.derived_func),
+                );
+            }
+            _ => {}
+        };
+    }
+
+    canonicalize_derivations(base.type_var.clone()).derived(derived_func)
+}
+
+/// Given typevars tv1 and tv2 (which could be derived from one another), constrain their typesets
+/// to be the same. When one is derived from the other, repeat the constrain process until
+/// a fixed point is reached.
+fn constrain_fixpoint(tv1: &TypeVar, tv2: &TypeVar) {
+    loop {
+        let old_tv1_ts = tv1.get_typeset().clone();
+        tv2.constrain_types(tv1.clone());
+        if tv1.get_typeset() == old_tv1_ts {
+            break;
+        }
+    }
+
+    let old_tv2_ts = tv2.get_typeset().clone();
+    tv1.constrain_types(tv2.clone());
+    // The above loop should ensure that all reference cycles have been handled.
+    assert!(old_tv2_ts == tv2.get_typeset());
+}
+
+/// Unify tv1 and tv2 in the given type environment. tv1 must have a rank greater or equal to tv2's
+/// one, modulo commutations.
+fn unify(tv1: &TypeVar, tv2: &TypeVar, type_env: &mut TypeEnvironment) -> Result<(), String> {
+    let tv1 = canonicalize_derivations(type_env.get_equivalent(tv1));
+    let tv2 = canonicalize_derivations(type_env.get_equivalent(tv2));
+
+    if tv1 == tv2 {
+        // Already unified.
+        return Ok(());
+    }
+
+    if type_env.rank(&tv2) < type_env.rank(&tv1) {
+        // Make sure tv1 always has the smallest rank, since real variables have the higher rank
+        // and we want them to be the canonical representatives of their equivalency classes.
+        return unify(&tv2, &tv1, type_env);
+    }
+
+    constrain_fixpoint(&tv1, &tv2);
+
+    if tv1.get_typeset().size() == 0 || tv2.get_typeset().size() == 0 {
+        return Err(format!(
+            "Error: empty type created when unifying {} and {}",
+            tv1.name, tv2.name
+        ));
+    }
+
+    let base = match &tv1.base {
+        Some(base) => base,
+        None => {
+            type_env.record_equivalent(tv1, tv2);
+            return Ok(());
+        }
+    };
+
+    if let Some(inverse) = base.derived_func.inverse() {
+        return unify(&base.type_var, &tv2.derived(inverse), type_env);
+    }
+
+    type_env.add_constraint(Constraint::Eq(tv1, tv2));
+    Ok(())
+}
+
+/// Perform type inference on one Def in the current type environment and return an updated type
+/// environment or error.
+///
+/// At a high level this works by creating fresh copies of each formal type var in the Def's
+/// instruction's signature, and unifying the formal typevar with the corresponding actual typevar.
+fn infer_definition(
+    def: &Def,
+    var_pool: &mut VarPool,
+    type_env: TypeEnvironment,
+    last_type_index: &mut usize,
+) -> Result {
+    let apply = &def.apply;
+    let inst = &apply.inst;
+
+    let mut type_env = type_env;
+    let free_formal_tvs = inst.all_typevars();
+
+    let mut original_to_own_typevar: HashMap<&TypeVar, TypeVar> = HashMap::new();
+    for &tv in &free_formal_tvs {
+        assert!(original_to_own_typevar
+            .insert(
+                tv,
+                TypeVar::copy_from(tv, format!("own_{}", last_type_index))
+            )
+            .is_none());
+        *last_type_index += 1;
+    }
+
+    // Update the mapping with any explicity bound type vars:
+    for (i, value_type) in apply.value_types.iter().enumerate() {
+        let singleton = TypeVar::new_singleton(value_type.clone());
+        assert!(original_to_own_typevar
+            .insert(free_formal_tvs[i], singleton)
+            .is_some());
+    }
+
+    // Get fresh copies for each typevar in the signature (both free and derived).
+    let mut formal_tvs = Vec::new();
+    formal_tvs.extend(inst.value_results.iter().map(|&i| {
+        substitute(
+            &original_to_own_typevar,
+            inst.operands_out[i].type_var().unwrap(),
+        )
+    }));
+    formal_tvs.extend(inst.value_opnums.iter().map(|&i| {
+        substitute(
+            &original_to_own_typevar,
+            inst.operands_in[i].type_var().unwrap(),
+        )
+    }));
+
+    // Get the list of actual vars.
+    let mut actual_vars = Vec::new();
+    actual_vars.extend(inst.value_results.iter().map(|&i| def.defined_vars[i]));
+    actual_vars.extend(
+        inst.value_opnums
+            .iter()
+            .map(|&i| apply.args[i].unwrap_var()),
+    );
+
+    // Get the list of the actual TypeVars.
+    let mut actual_tvs = Vec::new();
+    for var_index in actual_vars {
+        let var = var_pool.get_mut(var_index);
+        type_env.register(var_index, var);
+        actual_tvs.push(var.get_or_create_typevar());
+    }
+
+    // Make sure we start unifying with the control type variable first, by putting it at the
+    // front of both vectors.
+    if let Some(poly) = &inst.polymorphic_info {
+        let own_ctrl_tv = &original_to_own_typevar[&poly.ctrl_typevar];
+        let ctrl_index = formal_tvs.iter().position(|tv| tv == own_ctrl_tv).unwrap();
+        if ctrl_index != 0 {
+            formal_tvs.swap(0, ctrl_index);
+            actual_tvs.swap(0, ctrl_index);
+        }
+    }
+
+    // Unify each actual type variable with the corresponding formal type variable.
+    for (actual_tv, formal_tv) in actual_tvs.iter().zip(&formal_tvs) {
+        if let Err(msg) = unify(actual_tv, formal_tv, &mut type_env) {
+            return Err(format!(
+                "fail ti on {} <: {}: {}",
+                actual_tv.name, formal_tv.name, msg
+            ));
+        }
+    }
+
+    // Add any instruction specific constraints.
+    for constraint in &inst.constraints {
+        type_env.add_constraint(constraint.translate_with_map(&original_to_own_typevar));
+    }
+
+    Ok(type_env)
+}
+
+/// Perform type inference on an transformation. Return an updated type environment or error.
+pub fn infer_transform(
+    src: DefIndex,
+    dst: &Vec,
+    def_pool: &DefPool,
+    var_pool: &mut VarPool,
+) -> Result {
+    let mut type_env = TypeEnvironment::new();
+    let mut last_type_index = 0;
+
+    // Execute type inference on the source pattern.
+    type_env = infer_definition(def_pool.get(src), var_pool, type_env, &mut last_type_index)
+        .map_err(|err| format!("In src pattern: {}", err))?;
+
+    // Collect the type sets once after applying the source patterm; we'll compare the typesets
+    // after we've also considered the destination pattern, and will emit supplementary InTypeset
+    // checks if they don't match.
+    let src_typesets = type_env
+        .vars
+        .iter()
+        .map(|&var_index| {
+            let var = var_pool.get_mut(var_index);
+            let tv = type_env.get_equivalent(&var.get_or_create_typevar());
+            (var_index, tv.get_typeset().clone())
+        })
+        .collect::>();
+
+    // Execute type inference on the destination pattern.
+    for (i, &def_index) in dst.iter().enumerate() {
+        let def = def_pool.get(def_index);
+        type_env = infer_definition(def, var_pool, type_env, &mut last_type_index)
+            .map_err(|err| format!("line {}: {}", i, err))?;
+    }
+
+    for (var_index, src_typeset) in src_typesets {
+        let var = var_pool.get(var_index);
+        if !var.has_free_typevar() {
+            continue;
+        }
+        let tv = type_env.get_equivalent(&var.get_typevar().unwrap());
+        let new_typeset = tv.get_typeset();
+        assert!(
+            new_typeset.is_subset(&src_typeset),
+            "type sets can only get narrower"
+        );
+        if new_typeset != src_typeset {
+            type_env.add_constraint(Constraint::InTypeset(tv.clone(), new_typeset.clone()));
+        }
+    }
+
+    type_env.normalize(var_pool);
+
+    Ok(type_env.extract(var_pool))
 }
diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
new file mode 100644
index 0000000000..c748459441
--- /dev/null
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -0,0 +1,416 @@
+use crate::cdsl::ast::{
+    Apply, DefIndex, DefPool, DummyDef, DummyExpr, Expr, PatternPosition, VarIndex, VarPool,
+};
+use crate::cdsl::inst::Instruction;
+use crate::cdsl::type_inference::{infer_transform, TypeEnvironment};
+use crate::cdsl::typevar::TypeVar;
+
+use cranelift_entity::{entity_impl, PrimaryMap};
+
+use std::collections::{HashMap, HashSet};
+use std::iter::FromIterator;
+
+/// An instruction transformation consists of a source and destination pattern.
+///
+/// Patterns are expressed in *register transfer language* as tuples of Def or Expr nodes. A
+/// pattern may optionally have a sequence of TypeConstraints, that additionally limit the set of
+/// cases when it applies.
+///
+/// The source pattern can contain only a single instruction.
+pub struct Transform {
+    pub src: DefIndex,
+    pub dst: Vec,
+    pub var_pool: VarPool,
+    pub def_pool: DefPool,
+    pub type_env: TypeEnvironment,
+}
+
+type SymbolTable = HashMap<&'static str, VarIndex>;
+
+impl Transform {
+    fn new(src: DummyDef, dst: Vec) -> Self {
+        let mut var_pool = VarPool::new();
+        let mut def_pool = DefPool::new();
+
+        let mut input_vars: Vec = Vec::new();
+        let mut defined_vars: Vec = Vec::new();
+
+        // Maps variable names to our own Var copies.
+        let mut symbol_table: SymbolTable = SymbolTable::new();
+
+        // Rewrite variables in src and dst using our own copies.
+        let src = rewrite_def_list(
+            PatternPosition::Source,
+            vec![src],
+            &mut symbol_table,
+            &mut input_vars,
+            &mut defined_vars,
+            &mut var_pool,
+            &mut def_pool,
+        )[0];
+
+        let num_src_inputs = input_vars.len();
+
+        let dst = rewrite_def_list(
+            PatternPosition::Destination,
+            dst,
+            &mut symbol_table,
+            &mut input_vars,
+            &mut defined_vars,
+            &mut var_pool,
+            &mut def_pool,
+        );
+
+        // Sanity checks.
+        for &var_index in &input_vars {
+            assert!(
+                var_pool.get(var_index).is_input(),
+                format!("'{:?}' used as both input and def", var_pool.get(var_index))
+            );
+        }
+        assert!(
+            input_vars.len() == num_src_inputs,
+            format!(
+                "extra input vars in dst pattern: {:?}",
+                input_vars
+                    .iter()
+                    .map(|&i| var_pool.get(i))
+                    .skip(num_src_inputs)
+                    .collect::>()
+            )
+        );
+
+        // Perform type inference and cleanup.
+        let type_env = infer_transform(src, &dst, &def_pool, &mut var_pool).unwrap();
+
+        // Sanity check: the set of inferred free type variables should be a subset of the type
+        // variables corresponding to Vars appearing in the source pattern.
+        {
+            let free_typevars: HashSet =
+                HashSet::from_iter(type_env.free_typevars(&mut var_pool));
+            let src_tvs = HashSet::from_iter(
+                input_vars
+                    .clone()
+                    .iter()
+                    .chain(
+                        defined_vars
+                            .iter()
+                            .filter(|&&var_index| !var_pool.get(var_index).is_temp()),
+                    )
+                    .map(|&var_index| var_pool.get(var_index).get_typevar())
+                    .filter(|maybe_var| maybe_var.is_some())
+                    .map(|var| var.unwrap()),
+            );
+            if !free_typevars.is_subset(&src_tvs) {
+                let missing_tvs = (&free_typevars - &src_tvs)
+                    .iter()
+                    .map(|tv| tv.name.clone())
+                    .collect::>()
+                    .join(", ");
+                panic!("Some free vars don't appear in src: {}", missing_tvs);
+            }
+        }
+
+        for &var_index in input_vars.iter().chain(defined_vars.iter()) {
+            let var = var_pool.get_mut(var_index);
+            let canon_tv = type_env.get_equivalent(&var.get_or_create_typevar());
+            var.set_typevar(canon_tv);
+        }
+
+        Self {
+            src,
+            dst,
+            var_pool,
+            def_pool,
+            type_env,
+        }
+    }
+
+    fn verify_legalize(&self) {
+        let def = self.def_pool.get(self.src);
+        for &var_index in def.defined_vars.iter() {
+            let defined_var = self.var_pool.get(var_index);
+            assert!(
+                defined_var.is_output(),
+                format!("{:?} not defined in the destination pattern", defined_var)
+            );
+        }
+    }
+}
+
+/// Given a list of symbols defined in a Def, rewrite them to local symbols. Yield the new locals.
+fn rewrite_defined_vars(
+    position: PatternPosition,
+    dummy_def: &DummyDef,
+    def_index: DefIndex,
+    symbol_table: &mut SymbolTable,
+    defined_vars: &mut Vec,
+    var_pool: &mut VarPool,
+) -> Vec {
+    let mut new_defined_vars = Vec::new();
+    for var in &dummy_def.defined_vars {
+        let own_var = match symbol_table.get(var.name) {
+            Some(&existing_var) => existing_var,
+            None => {
+                // Materialize the variable.
+                let new_var = var_pool.create(var.name);
+                symbol_table.insert(var.name, new_var);
+                defined_vars.push(new_var);
+                new_var
+            }
+        };
+        var_pool.get_mut(own_var).set_def(position, def_index);
+        new_defined_vars.push(own_var);
+    }
+    new_defined_vars
+}
+
+/// Find all uses of variables in `expr` and replace them with our own local symbols.
+fn rewrite_expr(
+    position: PatternPosition,
+    dummy_expr: DummyExpr,
+    symbol_table: &mut SymbolTable,
+    input_vars: &mut Vec,
+    var_pool: &mut VarPool,
+) -> Apply {
+    let (apply_target, dummy_args) = if let DummyExpr::Apply(apply_target, dummy_args) = dummy_expr
+    {
+        (apply_target, dummy_args)
+    } else {
+        panic!("we only rewrite apply expressions");
+    };
+
+    assert_eq!(
+        apply_target.inst().operands_in.len(),
+        dummy_args.len(),
+        "number of arguments in instruction is incorrect"
+    );
+
+    let mut args = Vec::new();
+    for (i, arg) in dummy_args.into_iter().enumerate() {
+        match arg {
+            DummyExpr::Var(var) => {
+                let own_var = match symbol_table.get(var.name) {
+                    Some(&own_var) => {
+                        let var = var_pool.get(own_var);
+                        assert!(
+                            var.is_input() || var.get_def(position).is_some(),
+                            format!("{:?} used as both input and def", var)
+                        );
+                        own_var
+                    }
+                    None => {
+                        // First time we're using this variable.
+                        let own_var = var_pool.create(var.name);
+                        symbol_table.insert(var.name, own_var);
+                        input_vars.push(own_var);
+                        own_var
+                    }
+                };
+                args.push(Expr::Var(own_var));
+            }
+            DummyExpr::Literal(literal) => {
+                assert!(!apply_target.inst().operands_in[i].is_value());
+                args.push(Expr::Literal(literal));
+            }
+            DummyExpr::Apply(..) => {
+                panic!("Recursive apply is not allowed.");
+            }
+        }
+    }
+
+    Apply::new(apply_target, args)
+}
+
+fn rewrite_def_list(
+    position: PatternPosition,
+    dummy_defs: Vec,
+    symbol_table: &mut SymbolTable,
+    input_vars: &mut Vec,
+    defined_vars: &mut Vec,
+    var_pool: &mut VarPool,
+    def_pool: &mut DefPool,
+) -> Vec {
+    let mut new_defs = Vec::new();
+    for dummy_def in dummy_defs {
+        let def_index = def_pool.next_index();
+
+        let new_defined_vars = rewrite_defined_vars(
+            position,
+            &dummy_def,
+            def_index,
+            symbol_table,
+            defined_vars,
+            var_pool,
+        );
+        let new_apply = rewrite_expr(position, dummy_def.expr, symbol_table, input_vars, var_pool);
+
+        assert!(
+            def_pool.next_index() == def_index,
+            "shouldn't have created new defs in the meanwhile"
+        );
+        assert_eq!(
+            new_apply.inst.value_results.len(),
+            new_defined_vars.len(),
+            "number of Var results in instruction is incorrect"
+        );
+
+        new_defs.push(def_pool.create(new_apply, new_defined_vars));
+    }
+    new_defs
+}
+
+/// A group of related transformations.
+pub struct TransformGroup {
+    pub name: &'static str,
+    pub doc: &'static str,
+    pub chain_with: Option,
+    pub isa_name: Option<&'static str>,
+    pub id: TransformGroupIndex,
+
+    /// Maps Instruction camel_case names to custom legalization functions names.
+    pub custom_legalizes: HashMap,
+    pub transforms: Vec,
+}
+
+impl TransformGroup {
+    pub fn rust_name(&self) -> String {
+        match self.isa_name {
+            Some(_) => {
+                // This is a function in the same module as the LEGALIZE_ACTIONS table referring to
+                // it.
+                self.name.to_string()
+            }
+            None => format!("crate::legalizer::{}", self.name),
+        }
+    }
+}
+
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct TransformGroupIndex(u32);
+entity_impl!(TransformGroupIndex);
+
+pub struct TransformGroupBuilder {
+    name: &'static str,
+    doc: &'static str,
+    chain_with: Option,
+    isa_name: Option<&'static str>,
+    pub custom_legalizes: HashMap,
+    pub transforms: Vec,
+}
+
+impl TransformGroupBuilder {
+    pub fn new(name: &'static str, doc: &'static str) -> Self {
+        Self {
+            name,
+            doc,
+            chain_with: None,
+            isa_name: None,
+            custom_legalizes: HashMap::new(),
+            transforms: Vec::new(),
+        }
+    }
+
+    pub fn chain_with(mut self, next_id: TransformGroupIndex) -> Self {
+        assert!(self.chain_with.is_none());
+        self.chain_with = Some(next_id);
+        self
+    }
+
+    pub fn isa(mut self, isa_name: &'static str) -> Self {
+        assert!(self.isa_name.is_none());
+        self.isa_name = Some(isa_name);
+        self
+    }
+
+    /// Add a custom legalization action for `inst`.
+    ///
+    /// The `func_name` parameter is the fully qualified name of a Rust function which takes the
+    /// same arguments as the `isa::Legalize` actions.
+    ///
+    /// The custom function will be called to legalize `inst` and any return value is ignored.
+    pub fn custom_legalize(&mut self, inst: &Instruction, func_name: &'static str) {
+        assert!(
+            self.custom_legalizes
+                .insert(inst.camel_name.clone(), func_name)
+                .is_none(),
+            format!(
+                "custom legalization action for {} inserted twice",
+                inst.name
+            )
+        );
+    }
+
+    /// Add a legalization pattern to this group.
+    pub fn legalize(&mut self, src: DummyDef, dst: Vec) {
+        let transform = Transform::new(src, dst);
+        transform.verify_legalize();
+        self.transforms.push(transform);
+    }
+
+    pub fn finish_and_add_to(self, owner: &mut TransformGroups) -> TransformGroupIndex {
+        let next_id = owner.next_key();
+        owner.add(TransformGroup {
+            name: self.name,
+            doc: self.doc,
+            isa_name: self.isa_name,
+            id: next_id,
+            chain_with: self.chain_with,
+            custom_legalizes: self.custom_legalizes,
+            transforms: self.transforms,
+        })
+    }
+}
+
+pub struct TransformGroups {
+    groups: PrimaryMap,
+}
+
+impl TransformGroups {
+    pub fn new() -> Self {
+        Self {
+            groups: PrimaryMap::new(),
+        }
+    }
+    pub fn add(&mut self, new_group: TransformGroup) -> TransformGroupIndex {
+        for group in self.groups.values() {
+            assert!(
+                group.name != new_group.name,
+                format!("trying to insert {} for the second time", new_group.name)
+            );
+        }
+        self.groups.push(new_group)
+    }
+    pub fn get(&self, id: TransformGroupIndex) -> &TransformGroup {
+        &self.groups[id]
+    }
+    pub fn get_mut(&mut self, id: TransformGroupIndex) -> &mut TransformGroup {
+        self.groups.get_mut(id).unwrap()
+    }
+    fn next_key(&self) -> TransformGroupIndex {
+        self.groups.next_key()
+    }
+    pub fn by_name(&self, name: &'static str) -> &TransformGroup {
+        for group in self.groups.values() {
+            if group.name == name {
+                return group;
+            }
+        }
+        panic!(format!("transform group with name {} not found", name));
+    }
+}
+
+#[test]
+#[should_panic]
+fn test_double_custom_legalization() {
+    use crate::cdsl::formats::{FormatRegistry, InstructionFormatBuilder};
+    use crate::cdsl::inst::InstructionBuilder;
+
+    let mut format = FormatRegistry::new();
+    format.insert(InstructionFormatBuilder::new("nullary"));
+    let dummy_inst = InstructionBuilder::new("dummy", "doc").finish(&format);
+
+    let mut transform_group = TransformGroupBuilder::new("test", "doc");
+    transform_group.custom_legalize(&dummy_inst, "custom 1");
+    transform_group.custom_legalize(&dummy_inst, "custom 2");
+}

From d00e42ede37ca85b50372245ff3853af846d63b4 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 18 Apr 2019 18:26:01 +0200
Subject: [PATCH 2407/3084] [meta] Port shared and x86 legalizations to the
 Rust crate;

---
 .../codegen/meta/src/isa/x86/legalize.rs      | 293 +++++++
 cranelift/codegen/meta/src/shared/legalize.rs | 785 ++++++++++++++++++
 cranelift/codegen/meta/src/shared/mod.rs      |   9 +-
 3 files changed, 1086 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/codegen/meta/src/isa/x86/legalize.rs
 create mode 100644 cranelift/codegen/meta/src/shared/legalize.rs

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
new file mode 100644
index 0000000000..7a0cd8379c
--- /dev/null
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -0,0 +1,293 @@
+use crate::cdsl::ast::{bind, var, ExprBuilder, Literal};
+use crate::cdsl::inst::InstructionGroup;
+use crate::cdsl::xform::TransformGroupBuilder;
+
+use crate::shared::types::Int::{I32, I64};
+use crate::shared::Definitions as SharedDefinitions;
+
+pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGroup) {
+    let mut group = TransformGroupBuilder::new(
+        "x86_expand",
+        r#"
+    Legalize instructions by expansion.
+
+    Use x86-specific instructions if needed."#,
+    )
+    .isa("x86")
+    .chain_with(shared.transform_groups.by_name("expand_flags").id);
+
+    // List of instructions.
+    let insts = &shared.instructions;
+    let band = insts.by_name("band");
+    let bor = insts.by_name("bor");
+    let clz = insts.by_name("clz");
+    let ctz = insts.by_name("ctz");
+    let fcmp = insts.by_name("fcmp");
+    let fcvt_from_uint = insts.by_name("fcvt_from_uint");
+    let fcvt_to_sint = insts.by_name("fcvt_to_sint");
+    let fcvt_to_uint = insts.by_name("fcvt_to_uint");
+    let fcvt_to_sint_sat = insts.by_name("fcvt_to_sint_sat");
+    let fcvt_to_uint_sat = insts.by_name("fcvt_to_uint_sat");
+    let fmax = insts.by_name("fmax");
+    let fmin = insts.by_name("fmin");
+    let iadd = insts.by_name("iadd");
+    let iconst = insts.by_name("iconst");
+    let imul = insts.by_name("imul");
+    let isub = insts.by_name("isub");
+    let popcnt = insts.by_name("popcnt");
+    let sdiv = insts.by_name("sdiv");
+    let selectif = insts.by_name("selectif");
+    let smulhi = insts.by_name("smulhi");
+    let srem = insts.by_name("srem");
+    let udiv = insts.by_name("udiv");
+    let umulhi = insts.by_name("umulhi");
+    let ushr_imm = insts.by_name("ushr_imm");
+    let urem = insts.by_name("urem");
+
+    let x86_bsf = x86_instructions.by_name("x86_bsf");
+    let x86_bsr = x86_instructions.by_name("x86_bsr");
+    let x86_umulx = x86_instructions.by_name("x86_umulx");
+    let x86_smulx = x86_instructions.by_name("x86_smulx");
+
+    // List of immediates.
+    let floatcc = shared.operand_kinds.by_name("floatcc");
+    let imm64 = shared.operand_kinds.by_name("imm64");
+    let intcc = shared.operand_kinds.by_name("intcc");
+
+    // Division and remainder.
+    //
+    // The srem expansion requires custom code because srem INT_MIN, -1 is not
+    // allowed to trap. The other ops need to check avoid_div_traps.
+    group.custom_legalize(sdiv, "expand_sdivrem");
+    group.custom_legalize(srem, "expand_sdivrem");
+    group.custom_legalize(udiv, "expand_udivrem");
+    group.custom_legalize(urem, "expand_udivrem");
+
+    // Double length (widening) multiplication.
+    let a = var("a");
+    let x = var("x");
+    let y = var("y");
+    let a1 = var("a1");
+    let a2 = var("a2");
+    let res_lo = var("res_lo");
+    let res_hi = var("res_hi");
+
+    group.legalize(
+        def!(res_hi = umulhi(x, y)),
+        vec![def!((res_lo, res_hi) = x86_umulx(x, y))],
+    );
+
+    group.legalize(
+        def!(res_hi = smulhi(x, y)),
+        vec![def!((res_lo, res_hi) = x86_smulx(x, y))],
+    );
+
+    // Floating point condition codes.
+    //
+    // The 8 condition codes in `supported_floatccs` are directly supported by a
+    // `ucomiss` or `ucomisd` instruction. The remaining codes need legalization
+    // patterns.
+
+    let floatcc_eq = Literal::enumerator_for(floatcc, "eq");
+    let floatcc_ord = Literal::enumerator_for(floatcc, "ord");
+    let floatcc_ueq = Literal::enumerator_for(floatcc, "ueq");
+    let floatcc_ne = Literal::enumerator_for(floatcc, "ne");
+    let floatcc_uno = Literal::enumerator_for(floatcc, "uno");
+    let floatcc_one = Literal::enumerator_for(floatcc, "one");
+
+    // Equality needs an explicit `ord` test which checks the parity bit.
+    group.legalize(
+        def!(a = fcmp(floatcc_eq, x, y)),
+        vec![
+            def!(a1 = fcmp(floatcc_ord, x, y)),
+            def!(a2 = fcmp(floatcc_ueq, x, y)),
+            def!(a = band(a1, a2)),
+        ],
+    );
+    group.legalize(
+        def!(a = fcmp(floatcc_ne, x, y)),
+        vec![
+            def!(a1 = fcmp(floatcc_uno, x, y)),
+            def!(a2 = fcmp(floatcc_one, x, y)),
+            def!(a = bor(a1, a2)),
+        ],
+    );
+
+    let floatcc_lt = &Literal::enumerator_for(floatcc, "lt");
+    let floatcc_gt = &Literal::enumerator_for(floatcc, "gt");
+    let floatcc_le = &Literal::enumerator_for(floatcc, "le");
+    let floatcc_ge = &Literal::enumerator_for(floatcc, "ge");
+    let floatcc_ugt = &Literal::enumerator_for(floatcc, "ugt");
+    let floatcc_ult = &Literal::enumerator_for(floatcc, "ult");
+    let floatcc_uge = &Literal::enumerator_for(floatcc, "uge");
+    let floatcc_ule = &Literal::enumerator_for(floatcc, "ule");
+
+    // Inequalities that need to be reversed.
+    for &(cc, rev_cc) in &[
+        (floatcc_lt, floatcc_gt),
+        (floatcc_le, floatcc_ge),
+        (floatcc_ugt, floatcc_ult),
+        (floatcc_uge, floatcc_ule),
+    ] {
+        group.legalize(def!(a = fcmp(cc, x, y)), vec![def!(a = fcmp(rev_cc, y, x))]);
+    }
+
+    // We need to modify the CFG for min/max legalization.
+    group.custom_legalize(fmin, "expand_minmax");
+    group.custom_legalize(fmax, "expand_minmax");
+
+    // Conversions from unsigned need special handling.
+    group.custom_legalize(fcvt_from_uint, "expand_fcvt_from_uint");
+    // Conversions from float to int can trap and modify the control flow graph.
+    group.custom_legalize(fcvt_to_sint, "expand_fcvt_to_sint");
+    group.custom_legalize(fcvt_to_uint, "expand_fcvt_to_uint");
+    group.custom_legalize(fcvt_to_sint_sat, "expand_fcvt_to_sint_sat");
+    group.custom_legalize(fcvt_to_uint_sat, "expand_fcvt_to_uint_sat");
+
+    // Count leading and trailing zeroes, for baseline x86_64
+    let c_minus_one = var("c_minus_one");
+    let c_thirty_one = var("c_thirty_one");
+    let c_thirty_two = var("c_thirty_two");
+    let c_sixty_three = var("c_sixty_three");
+    let c_sixty_four = var("c_sixty_four");
+    let index1 = var("index1");
+    let r2flags = var("r2flags");
+    let index2 = var("index2");
+
+    let intcc_eq = Literal::enumerator_for(intcc, "eq");
+    let imm64_minus_one = Literal::constant(imm64, -1);
+    let imm64_63 = Literal::constant(imm64, 63);
+    group.legalize(
+        def!(a = clz.I64(x)),
+        vec![
+            def!(c_minus_one = iconst(imm64_minus_one)),
+            def!(c_sixty_three = iconst(imm64_63)),
+            def!((index1, r2flags) = x86_bsr(x)),
+            def!(index2 = selectif(intcc_eq, r2flags, c_minus_one, index1)),
+            def!(a = isub(c_sixty_three, index2)),
+        ],
+    );
+
+    let imm64_31 = Literal::constant(imm64, 31);
+    group.legalize(
+        def!(a = clz.I32(x)),
+        vec![
+            def!(c_minus_one = iconst(imm64_minus_one)),
+            def!(c_thirty_one = iconst(imm64_31)),
+            def!((index1, r2flags) = x86_bsr(x)),
+            def!(index2 = selectif(intcc_eq, r2flags, c_minus_one, index1)),
+            def!(a = isub(c_thirty_one, index2)),
+        ],
+    );
+
+    let imm64_64 = Literal::constant(imm64, 64);
+    group.legalize(
+        def!(a = ctz.I64(x)),
+        vec![
+            def!(c_sixty_four = iconst(imm64_64)),
+            def!((index1, r2flags) = x86_bsf(x)),
+            def!(a = selectif(intcc_eq, r2flags, c_sixty_four, index1)),
+        ],
+    );
+
+    let imm64_32 = Literal::constant(imm64, 32);
+    group.legalize(
+        def!(a = ctz.I32(x)),
+        vec![
+            def!(c_thirty_two = iconst(imm64_32)),
+            def!((index1, r2flags) = x86_bsf(x)),
+            def!(a = selectif(intcc_eq, r2flags, c_thirty_two, index1)),
+        ],
+    );
+
+    // Population count for baseline x86_64
+    let qv1 = var("qv1");
+    let qv3 = var("qv3");
+    let qv4 = var("qv4");
+    let qv5 = var("qv5");
+    let qv6 = var("qv6");
+    let qv7 = var("qv7");
+    let qv8 = var("qv8");
+    let qv9 = var("qv9");
+    let qv10 = var("qv10");
+    let qv11 = var("qv11");
+    let qv12 = var("qv12");
+    let qv13 = var("qv13");
+    let qv14 = var("qv14");
+    let qv15 = var("qv15");
+    let qv16 = var("qv16");
+    let qc77 = var("qc77");
+    #[allow(non_snake_case)]
+    let qc0F = var("qc0F");
+    let qc01 = var("qc01");
+
+    let imm64_1 = Literal::constant(imm64, 1);
+    let imm64_4 = Literal::constant(imm64, 4);
+    group.legalize(
+        def!(qv16 = popcnt.I64(qv1)),
+        vec![
+            def!(qv3 = ushr_imm(qv1, imm64_1)),
+            def!(qc77 = iconst(Literal::constant(imm64, 0x7777777777777777))),
+            def!(qv4 = band(qv3, qc77)),
+            def!(qv5 = isub(qv1, qv4)),
+            def!(qv6 = ushr_imm(qv4, imm64_1)),
+            def!(qv7 = band(qv6, qc77)),
+            def!(qv8 = isub(qv5, qv7)),
+            def!(qv9 = ushr_imm(qv7, imm64_1)),
+            def!(qv10 = band(qv9, qc77)),
+            def!(qv11 = isub(qv8, qv10)),
+            def!(qv12 = ushr_imm(qv11, imm64_4)),
+            def!(qv13 = iadd(qv11, qv12)),
+            def!(qc0F = iconst(Literal::constant(imm64, 0x0F0F0F0F0F0F0F0F))),
+            def!(qv14 = band(qv13, qc0F)),
+            def!(qc01 = iconst(Literal::constant(imm64, 0x0101010101010101))),
+            def!(qv15 = imul(qv14, qc01)),
+            def!(qv16 = ushr_imm(qv15, Literal::constant(imm64, 56))),
+        ],
+    );
+
+    let lv1 = var("lv1");
+    let lv3 = var("lv3");
+    let lv4 = var("lv4");
+    let lv5 = var("lv5");
+    let lv6 = var("lv6");
+    let lv7 = var("lv7");
+    let lv8 = var("lv8");
+    let lv9 = var("lv9");
+    let lv10 = var("lv10");
+    let lv11 = var("lv11");
+    let lv12 = var("lv12");
+    let lv13 = var("lv13");
+    let lv14 = var("lv14");
+    let lv15 = var("lv15");
+    let lv16 = var("lv16");
+    let lc77 = var("lc77");
+    #[allow(non_snake_case)]
+    let lc0F = var("lc0F");
+    let lc01 = var("lc01");
+
+    group.legalize(
+        def!(lv16 = popcnt.I32(lv1)),
+        vec![
+            def!(lv3 = ushr_imm(lv1, imm64_1)),
+            def!(lc77 = iconst(Literal::constant(imm64, 0x77777777))),
+            def!(lv4 = band(lv3, lc77)),
+            def!(lv5 = isub(lv1, lv4)),
+            def!(lv6 = ushr_imm(lv4, imm64_1)),
+            def!(lv7 = band(lv6, lc77)),
+            def!(lv8 = isub(lv5, lv7)),
+            def!(lv9 = ushr_imm(lv7, imm64_1)),
+            def!(lv10 = band(lv9, lc77)),
+            def!(lv11 = isub(lv8, lv10)),
+            def!(lv12 = ushr_imm(lv11, imm64_4)),
+            def!(lv13 = iadd(lv11, lv12)),
+            def!(lc0F = iconst(Literal::constant(imm64, 0x0F0F0F0F))),
+            def!(lv14 = band(lv13, lc0F)),
+            def!(lc01 = iconst(Literal::constant(imm64, 0x01010101))),
+            def!(lv15 = imul(lv14, lc01)),
+            def!(lv16 = ushr_imm(lv15, Literal::constant(imm64, 24))),
+        ],
+    );
+
+    group.finish_and_add_to(&mut shared.transform_groups);
+}
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
new file mode 100644
index 0000000000..172d6fd981
--- /dev/null
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -0,0 +1,785 @@
+use crate::cdsl::ast::{bind, var, ExprBuilder, Literal};
+use crate::cdsl::inst::{Instruction, InstructionGroup};
+use crate::cdsl::xform::{TransformGroupBuilder, TransformGroups};
+
+use crate::shared::OperandKinds;
+
+use crate::shared::types::Float::{F32, F64};
+use crate::shared::types::Int::{I16, I32, I64, I8};
+
+pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformGroups {
+    let mut narrow = TransformGroupBuilder::new(
+        "narrow",
+        r#"
+        Legalize instructions by narrowing.
+
+        The transformations in the 'narrow' group work by expressing
+        instructions in terms of smaller types. Operations on vector types are
+        expressed in terms of vector types with fewer lanes, and integer
+        operations are expressed in terms of smaller integer types.
+    "#,
+    );
+
+    let mut widen = TransformGroupBuilder::new(
+        "widen",
+        r#"
+        Legalize instructions by widening.
+
+        The transformations in the 'widen' group work by expressing
+        instructions in terms of larger types.
+    "#,
+    );
+
+    let mut expand = TransformGroupBuilder::new(
+        "expand",
+        r#"
+        Legalize instructions by expansion.
+
+        Rewrite instructions in terms of other instructions, generally
+        operating on the same types as the original instructions.
+    "#,
+    );
+
+    // List of instructions.
+    let band = insts.by_name("band");
+    let band_imm = insts.by_name("band_imm");
+    let band_not = insts.by_name("band_not");
+    let bint = insts.by_name("bint");
+    let bitrev = insts.by_name("bitrev");
+    let bnot = insts.by_name("bnot");
+    let bor = insts.by_name("bor");
+    let bor_imm = insts.by_name("bor_imm");
+    let bor_not = insts.by_name("bor_not");
+    let br_icmp = insts.by_name("br_icmp");
+    let br_table = insts.by_name("br_table");
+    let bxor = insts.by_name("bxor");
+    let bxor_imm = insts.by_name("bxor_imm");
+    let bxor_not = insts.by_name("bxor_not");
+    let cls = insts.by_name("cls");
+    let clz = insts.by_name("clz");
+    let ctz = insts.by_name("ctz");
+    let fabs = insts.by_name("fabs");
+    let f32const = insts.by_name("f32const");
+    let f64const = insts.by_name("f64const");
+    let fcopysign = insts.by_name("fcopysign");
+    let fneg = insts.by_name("fneg");
+    let iadd = insts.by_name("iadd");
+    let iadd_carry = insts.by_name("iadd_carry");
+    let iadd_cin = insts.by_name("iadd_cin");
+    let iadd_cout = insts.by_name("iadd_cout");
+    let iadd_imm = insts.by_name("iadd_imm");
+    let icmp = insts.by_name("icmp");
+    let icmp_imm = insts.by_name("icmp_imm");
+    let iconcat = insts.by_name("iconcat");
+    let iconst = insts.by_name("iconst");
+    let ifcmp = insts.by_name("ifcmp");
+    let ifcmp_imm = insts.by_name("ifcmp_imm");
+    let imul = insts.by_name("imul");
+    let imul_imm = insts.by_name("imul_imm");
+    let ireduce = insts.by_name("ireduce");
+    let irsub_imm = insts.by_name("irsub_imm");
+    let ishl = insts.by_name("ishl");
+    let ishl_imm = insts.by_name("ishl_imm");
+    let isplit = insts.by_name("isplit");
+    let istore8 = insts.by_name("istore8");
+    let istore16 = insts.by_name("istore16");
+    let isub = insts.by_name("isub");
+    let isub_bin = insts.by_name("isub_bin");
+    let isub_borrow = insts.by_name("isub_borrow");
+    let isub_bout = insts.by_name("isub_bout");
+    let load = insts.by_name("load");
+    let popcnt = insts.by_name("popcnt");
+    let rotl = insts.by_name("rotl");
+    let rotl_imm = insts.by_name("rotl_imm");
+    let rotr = insts.by_name("rotr");
+    let rotr_imm = insts.by_name("rotr_imm");
+    let sdiv = insts.by_name("sdiv");
+    let sdiv_imm = insts.by_name("sdiv_imm");
+    let select = insts.by_name("select");
+    let sextend = insts.by_name("sextend");
+    let sshr = insts.by_name("sshr");
+    let sshr_imm = insts.by_name("sshr_imm");
+    let srem = insts.by_name("srem");
+    let srem_imm = insts.by_name("srem_imm");
+    let store = insts.by_name("store");
+    let udiv = insts.by_name("udiv");
+    let udiv_imm = insts.by_name("udiv_imm");
+    let uextend = insts.by_name("uextend");
+    let uload8 = insts.by_name("uload8");
+    let uload16 = insts.by_name("uload16");
+    let ushr = insts.by_name("ushr");
+    let ushr_imm = insts.by_name("ushr_imm");
+    let urem = insts.by_name("urem");
+    let urem_imm = insts.by_name("urem_imm");
+    let trapif = insts.by_name("trapif");
+    let trapnz = insts.by_name("trapnz");
+    let trapz = insts.by_name("trapz");
+
+    // Custom expansions for memory objects.
+    expand.custom_legalize(insts.by_name("global_value"), "expand_global_value");
+    expand.custom_legalize(insts.by_name("heap_addr"), "expand_heap_addr");
+    expand.custom_legalize(insts.by_name("table_addr"), "expand_table_addr");
+
+    // Custom expansions for calls.
+    expand.custom_legalize(insts.by_name("call"), "expand_call");
+
+    // Custom expansions that need to change the CFG.
+    // TODO: Add sufficient XForm syntax that we don't need to hand-code these.
+    expand.custom_legalize(trapz, "expand_cond_trap");
+    expand.custom_legalize(trapnz, "expand_cond_trap");
+    expand.custom_legalize(br_table, "expand_br_table");
+    expand.custom_legalize(select, "expand_select");
+
+    // Custom expansions for floating point constants.
+    // These expansions require bit-casting or creating constant pool entries.
+    expand.custom_legalize(f32const, "expand_fconst");
+    expand.custom_legalize(f64const, "expand_fconst");
+
+    // Custom expansions for stack memory accesses.
+    expand.custom_legalize(insts.by_name("stack_load"), "expand_stack_load");
+    expand.custom_legalize(insts.by_name("stack_store"), "expand_stack_store");
+
+    // List of immediates.
+    let imm64 = immediates.by_name("imm64");
+    let ieee32 = immediates.by_name("ieee32");
+    let ieee64 = immediates.by_name("ieee64");
+    let intcc = immediates.by_name("intcc");
+
+    // List of variables to reuse in patterns.
+    let x = var("x");
+    let y = var("y");
+    let z = var("z");
+    let a = var("a");
+    let a1 = var("a1");
+    let a2 = var("a2");
+    let a3 = var("a3");
+    let a4 = var("a4");
+    let b = var("b");
+    let b1 = var("b1");
+    let b2 = var("b2");
+    let b3 = var("b3");
+    let b4 = var("b4");
+    let b_in = var("b_in");
+    let b_int = var("b_int");
+    let c = var("c");
+    let c1 = var("c1");
+    let c2 = var("c2");
+    let c3 = var("c3");
+    let c4 = var("c4");
+    let c_in = var("c_in");
+    let c_int = var("c_int");
+    let d = var("d");
+    let d1 = var("d1");
+    let d2 = var("d2");
+    let d3 = var("d3");
+    let d4 = var("d4");
+    let e = var("e");
+    let e1 = var("e1");
+    let e2 = var("e2");
+    let e3 = var("e3");
+    let e4 = var("e4");
+    let f = var("f");
+    let f1 = var("f1");
+    let f2 = var("f2");
+    let xl = var("xl");
+    let xh = var("xh");
+    let yl = var("yl");
+    let yh = var("yh");
+    let al = var("al");
+    let ah = var("ah");
+    let cc = var("cc");
+    let ptr = var("ptr");
+    let flags = var("flags");
+    let offset = var("off");
+
+    narrow.legalize(
+        def!(a = iadd(x, y)),
+        vec![
+            def!((xl, xh) = isplit(x)),
+            def!((yl, yh) = isplit(y)),
+            def!((al, c) = iadd_cout(xl, yl)),
+            def!(ah = iadd_cin(xh, yh, c)),
+            def!(a = iconcat(al, ah)),
+        ],
+    );
+
+    narrow.legalize(
+        def!(a = isub(x, y)),
+        vec![
+            def!((xl, xh) = isplit(x)),
+            def!((yl, yh) = isplit(y)),
+            def!((al, b) = isub_bout(xl, yl)),
+            def!(ah = isub_bin(xh, yh, b)),
+            def!(a = iconcat(al, ah)),
+        ],
+    );
+
+    for &bin_op in &[band, bor, bxor] {
+        narrow.legalize(
+            def!(a = bin_op(x, y)),
+            vec![
+                def!((xl, xh) = isplit(x)),
+                def!((yl, yh) = isplit(y)),
+                def!(al = bin_op(xl, yl)),
+                def!(ah = bin_op(xh, yh)),
+                def!(a = iconcat(al, ah)),
+            ],
+        );
+    }
+
+    narrow.legalize(
+        def!(a = select(c, x, y)),
+        vec![
+            def!((xl, xh) = isplit(x)),
+            def!((yl, yh) = isplit(y)),
+            def!(al = select(c, xl, yl)),
+            def!(ah = select(c, xh, yh)),
+            def!(a = iconcat(al, ah)),
+        ],
+    );
+
+    // Widen instructions with one input operand.
+    for &op in &[bnot, popcnt] {
+        for &int_ty in &[I8, I16] {
+            widen.legalize(
+                def!(a = op.int_ty(b)),
+                vec![
+                    def!(x = uextend.I32(b)),
+                    def!(z = op.I32(x)),
+                    def!(a = ireduce.int_ty(z)),
+                ],
+            );
+        }
+    }
+
+    // Widen instructions with two input operands.
+    let mut widen_two_arg = |signed: bool, op: &Instruction| {
+        for &int_ty in &[I8, I16] {
+            let sign_ext_op = if signed { sextend } else { uextend };
+            widen.legalize(
+                def!(a = op.int_ty(b, c)),
+                vec![
+                    def!(x = sign_ext_op.I32(b)),
+                    def!(y = sign_ext_op.I32(c)),
+                    def!(z = op.I32(x, y)),
+                    def!(a = ireduce.int_ty(z)),
+                ],
+            );
+        }
+    };
+
+    for bin_op in &[
+        iadd, isub, imul, udiv, urem, band, bor, bxor, band_not, bor_not, bxor_not,
+    ] {
+        widen_two_arg(false, bin_op);
+    }
+    for bin_op in &[sdiv, srem] {
+        widen_two_arg(true, bin_op);
+    }
+
+    // Widen instructions using immediate operands.
+    let mut widen_imm = |signed: bool, op: &Instruction| {
+        for &int_ty in &[I8, I16] {
+            let sign_ext_op = if signed { sextend } else { uextend };
+            widen.legalize(
+                def!(a = op.int_ty(b, c)),
+                vec![
+                    def!(x = sign_ext_op.I32(b)),
+                    def!(z = op.I32(x, c)),
+                    def!(a = ireduce.int_ty(z)),
+                ],
+            );
+        }
+    };
+
+    for bin_op in &[
+        iadd_imm, imul_imm, udiv_imm, urem_imm, band_imm, bor_imm, bxor_imm, irsub_imm,
+    ] {
+        widen_imm(false, bin_op);
+    }
+    for bin_op in &[sdiv_imm, srem_imm] {
+        widen_imm(true, bin_op);
+    }
+
+    for &(int_ty, num) in &[(I8, 24), (I16, 16)] {
+        let imm = Literal::constant(imm64, -num);
+
+        widen.legalize(
+            def!(a = clz.int_ty(b)),
+            vec![
+                def!(c = uextend.I32(b)),
+                def!(d = clz.I32(c)),
+                def!(e = iadd_imm(d, imm)),
+                def!(a = ireduce.int_ty(e)),
+            ],
+        );
+
+        widen.legalize(
+            def!(a = cls.int_ty(b)),
+            vec![
+                def!(c = sextend.I32(b)),
+                def!(d = cls.I32(c)),
+                def!(e = iadd_imm(d, imm)),
+                def!(a = ireduce.int_ty(e)),
+            ],
+        );
+    }
+
+    for &(int_ty, num) in &[(I8, 1 << 8), (I16, 1 << 16)] {
+        let num = Literal::constant(imm64, num);
+        widen.legalize(
+            def!(a = ctz.int_ty(b)),
+            vec![
+                def!(c = uextend.I32(b)),
+                // When `b` is zero, returns the size of x in bits.
+                def!(d = bor_imm(c, num)),
+                def!(e = ctz.I32(d)),
+                def!(a = ireduce.int_ty(e)),
+            ],
+        );
+    }
+
+    // iconst
+    for &int_ty in &[I8, I16] {
+        widen.legalize(
+            def!(a = iconst.int_ty(b)),
+            vec![def!(c = iconst.I32(b)), def!(a = ireduce.int_ty(c))],
+        );
+    }
+
+    for &extend_op in &[uextend, sextend] {
+        // The sign extension operators have two typevars: the result has one and controls the
+        // instruction, then the input has one.
+        let bound = bind(bind(extend_op, I16), I8);
+        widen.legalize(
+            def!(a = bound(b)),
+            vec![def!(c = extend_op.I32(b)), def!(a = ireduce(c))],
+        );
+    }
+
+    widen.legalize(
+        def!(store.I8(flags, a, ptr, offset)),
+        vec![
+            def!(b = uextend.I32(a)),
+            def!(istore8(flags, b, ptr, offset)),
+        ],
+    );
+
+    widen.legalize(
+        def!(store.I16(flags, a, ptr, offset)),
+        vec![
+            def!(b = uextend.I32(a)),
+            def!(istore16(flags, b, ptr, offset)),
+        ],
+    );
+
+    widen.legalize(
+        def!(a = load.I8(flags, ptr, offset)),
+        vec![
+            def!(b = uload8.I32(flags, ptr, offset)),
+            def!(a = ireduce(b)),
+        ],
+    );
+
+    widen.legalize(
+        def!(a = load.I16(flags, ptr, offset)),
+        vec![
+            def!(b = uload16.I32(flags, ptr, offset)),
+            def!(a = ireduce(b)),
+        ],
+    );
+
+    for &int_ty in &[I8, I16] {
+        widen.legalize(
+            def!(br_table.int_ty(x, y, z)),
+            vec![def!(b = uextend.I32(x)), def!(br_table(b, y, z))],
+        );
+    }
+
+    for &int_ty in &[I8, I16] {
+        widen.legalize(
+            def!(a = bint.int_ty(b)),
+            vec![def!(x = bint.I32(b)), def!(a = ireduce.int_ty(x))],
+        );
+    }
+
+    for &int_ty in &[I8, I16] {
+        for &op in &[ishl, ishl_imm, ushr, ushr_imm] {
+            widen.legalize(
+                def!(a = op.int_ty(b, c)),
+                vec![
+                    def!(x = uextend.I32(b)),
+                    def!(z = op.I32(x, c)),
+                    def!(a = ireduce.int_ty(z)),
+                ],
+            );
+        }
+
+        for &op in &[sshr, sshr_imm] {
+            widen.legalize(
+                def!(a = op.int_ty(b, c)),
+                vec![
+                    def!(x = sextend.I32(b)),
+                    def!(z = op.I32(x, c)),
+                    def!(a = ireduce.int_ty(z)),
+                ],
+            );
+        }
+
+        for cc in &["eq", "ne", "ugt", "ult", "uge", "ule"] {
+            let w_cc = Literal::enumerator_for(intcc, cc);
+            widen.legalize(
+                def!(a = icmp_imm.int_ty(w_cc, b, c)),
+                vec![def!(x = uextend.I32(b)), def!(a = icmp_imm(w_cc, x, c))],
+            );
+            widen.legalize(
+                def!(a = icmp.int_ty(w_cc, b, c)),
+                vec![
+                    def!(x = uextend.I32(b)),
+                    def!(y = uextend.I32(c)),
+                    def!(a = icmp.I32(w_cc, x, y)),
+                ],
+            );
+        }
+
+        for cc in &["sgt", "slt", "sge", "sle"] {
+            let w_cc = Literal::enumerator_for(intcc, cc);
+            widen.legalize(
+                def!(a = icmp_imm.int_ty(w_cc, b, c)),
+                vec![def!(x = sextend.I32(b)), def!(a = icmp_imm(w_cc, x, c))],
+            );
+
+            widen.legalize(
+                def!(a = icmp.int_ty(w_cc, b, c)),
+                vec![
+                    def!(x = sextend.I32(b)),
+                    def!(y = sextend.I32(c)),
+                    def!(a = icmp(w_cc, x, y)),
+                ],
+            );
+        }
+    }
+
+    // Expand integer operations with carry for RISC architectures that don't have
+    // the flags.
+    let intcc_ult = Literal::enumerator_for(intcc, "ult");
+    expand.legalize(
+        def!((a, c) = iadd_cout(x, y)),
+        vec![def!(a = iadd(x, y)), def!(c = icmp(intcc_ult, a, x))],
+    );
+
+    let intcc_ugt = Literal::enumerator_for(intcc, "ugt");
+    expand.legalize(
+        def!((a, b) = isub_bout(x, y)),
+        vec![def!(a = isub(x, y)), def!(b = icmp(intcc_ugt, a, x))],
+    );
+
+    expand.legalize(
+        def!(a = iadd_cin(x, y, c)),
+        vec![
+            def!(a1 = iadd(x, y)),
+            def!(c_int = bint(c)),
+            def!(a = iadd(a1, c_int)),
+        ],
+    );
+
+    expand.legalize(
+        def!(a = isub_bin(x, y, b)),
+        vec![
+            def!(a1 = isub(x, y)),
+            def!(b_int = bint(b)),
+            def!(a = isub(a1, b_int)),
+        ],
+    );
+
+    expand.legalize(
+        def!((a, c) = iadd_carry(x, y, c_in)),
+        vec![
+            def!((a1, c1) = iadd_cout(x, y)),
+            def!(c_int = bint(c_in)),
+            def!((a, c2) = iadd_cout(a1, c_int)),
+            def!(c = bor(c1, c2)),
+        ],
+    );
+
+    expand.legalize(
+        def!((a, b) = isub_borrow(x, y, b_in)),
+        vec![
+            def!((a1, b1) = isub_bout(x, y)),
+            def!(b_int = bint(b_in)),
+            def!((a, b2) = isub_bout(a1, b_int)),
+            def!(b = bor(b1, b2)),
+        ],
+    );
+
+    // Expansions for immediate operands that are out of range.
+    for &(inst_imm, inst) in &[
+        (iadd_imm, iadd),
+        (imul_imm, imul),
+        (sdiv_imm, sdiv),
+        (udiv_imm, udiv),
+        (srem_imm, srem),
+        (urem_imm, urem),
+        (band_imm, band),
+        (bor_imm, bor),
+        (bxor_imm, bxor),
+        (ifcmp_imm, ifcmp),
+    ] {
+        expand.legalize(
+            def!(a = inst_imm(x, y)),
+            vec![def!(a1 = iconst(y)), def!(a = inst(x, a1))],
+        );
+    }
+
+    expand.legalize(
+        def!(a = irsub_imm(y, x)),
+        vec![def!(a1 = iconst(x)), def!(a = isub(a1, y))],
+    );
+
+    // Rotates and shifts.
+    for &(inst_imm, inst) in &[
+        (rotl_imm, rotl),
+        (rotr_imm, rotr),
+        (ishl_imm, ishl),
+        (sshr_imm, sshr),
+        (ushr_imm, ushr),
+    ] {
+        expand.legalize(
+            def!(a = inst_imm(x, y)),
+            vec![def!(a1 = iconst.I32(y)), def!(a = inst(x, a1))],
+        );
+    }
+
+    expand.legalize(
+        def!(a = icmp_imm(cc, x, y)),
+        vec![def!(a1 = iconst(y)), def!(a = icmp(cc, x, a1))],
+    );
+
+    //# Expansions for *_not variants of bitwise ops.
+    for &(inst_not, inst) in &[(band_not, band), (bor_not, bor), (bxor_not, bxor)] {
+        expand.legalize(
+            def!(a = inst_not(x, y)),
+            vec![def!(a1 = bnot(y)), def!(a = inst(x, a1))],
+        );
+    }
+
+    //# Expand bnot using xor.
+    let minus_one = Literal::constant(imm64, -1);
+    expand.legalize(
+        def!(a = bnot(x)),
+        vec![def!(y = iconst(minus_one)), def!(a = bxor(x, y))],
+    );
+
+    //# Expand bitrev
+    //# Adapted from Stack Overflow.
+    //# https://stackoverflow.com/questions/746171/most-efficient-algorithm-for-bit-reversal-from-msb-lsb-to-lsb-msb-in-c
+    let imm64_1 = Literal::constant(imm64, 1);
+    let imm64_2 = Literal::constant(imm64, 2);
+    let imm64_4 = Literal::constant(imm64, 4);
+
+    widen.legalize(
+        def!(a = bitrev.I8(x)),
+        vec![
+            def!(a1 = band_imm(x, Literal::constant(imm64, 0xaa))),
+            def!(a2 = ushr_imm(a1, imm64_1)),
+            def!(a3 = band_imm(x, Literal::constant(imm64, 0x55))),
+            def!(a4 = ishl_imm(a3, imm64_1)),
+            def!(b = bor(a2, a4)),
+            def!(b1 = band_imm(b, Literal::constant(imm64, 0xcc))),
+            def!(b2 = ushr_imm(b1, imm64_2)),
+            def!(b3 = band_imm(b, Literal::constant(imm64, 0x33))),
+            def!(b4 = ishl_imm(b3, imm64_2)),
+            def!(c = bor(b2, b4)),
+            def!(c1 = band_imm(c, Literal::constant(imm64, 0xf0))),
+            def!(c2 = ushr_imm(c1, imm64_4)),
+            def!(c3 = band_imm(c, Literal::constant(imm64, 0x0f))),
+            def!(c4 = ishl_imm(c3, imm64_4)),
+            def!(a = bor(c2, c4)),
+        ],
+    );
+
+    let imm64_8 = Literal::constant(imm64, 8);
+
+    widen.legalize(
+        def!(a = bitrev.I16(x)),
+        vec![
+            def!(a1 = band_imm(x, Literal::constant(imm64, 0xaaaa))),
+            def!(a2 = ushr_imm(a1, imm64_1)),
+            def!(a3 = band_imm(x, Literal::constant(imm64, 0x5555))),
+            def!(a4 = ishl_imm(a3, imm64_1)),
+            def!(b = bor(a2, a4)),
+            def!(b1 = band_imm(b, Literal::constant(imm64, 0xcccc))),
+            def!(b2 = ushr_imm(b1, imm64_2)),
+            def!(b3 = band_imm(b, Literal::constant(imm64, 0x3333))),
+            def!(b4 = ishl_imm(b3, imm64_2)),
+            def!(c = bor(b2, b4)),
+            def!(c1 = band_imm(c, Literal::constant(imm64, 0xf0f0))),
+            def!(c2 = ushr_imm(c1, imm64_4)),
+            def!(c3 = band_imm(c, Literal::constant(imm64, 0x0f0f))),
+            def!(c4 = ishl_imm(c3, imm64_4)),
+            def!(d = bor(c2, c4)),
+            def!(d1 = band_imm(d, Literal::constant(imm64, 0xff00))),
+            def!(d2 = ushr_imm(d1, imm64_8)),
+            def!(d3 = band_imm(d, Literal::constant(imm64, 0x00ff))),
+            def!(d4 = ishl_imm(d3, imm64_8)),
+            def!(a = bor(d2, d4)),
+        ],
+    );
+
+    let imm64_16 = Literal::constant(imm64, 16);
+
+    expand.legalize(
+        def!(a = bitrev.I32(x)),
+        vec![
+            def!(a1 = band_imm(x, Literal::constant(imm64, 0xaaaaaaaa))),
+            def!(a2 = ushr_imm(a1, imm64_1)),
+            def!(a3 = band_imm(x, Literal::constant(imm64, 0x55555555))),
+            def!(a4 = ishl_imm(a3, imm64_1)),
+            def!(b = bor(a2, a4)),
+            def!(b1 = band_imm(b, Literal::constant(imm64, 0xcccccccc))),
+            def!(b2 = ushr_imm(b1, imm64_2)),
+            def!(b3 = band_imm(b, Literal::constant(imm64, 0x33333333))),
+            def!(b4 = ishl_imm(b3, imm64_2)),
+            def!(c = bor(b2, b4)),
+            def!(c1 = band_imm(c, Literal::constant(imm64, 0xf0f0f0f0))),
+            def!(c2 = ushr_imm(c1, imm64_4)),
+            def!(c3 = band_imm(c, Literal::constant(imm64, 0x0f0f0f0f))),
+            def!(c4 = ishl_imm(c3, imm64_4)),
+            def!(d = bor(c2, c4)),
+            def!(d1 = band_imm(d, Literal::constant(imm64, 0xff00ff00))),
+            def!(d2 = ushr_imm(d1, imm64_8)),
+            def!(d3 = band_imm(d, Literal::constant(imm64, 0x00ff00ff))),
+            def!(d4 = ishl_imm(d3, imm64_8)),
+            def!(e = bor(d2, d4)),
+            def!(e1 = ushr_imm(e, imm64_16)),
+            def!(e2 = ishl_imm(e, imm64_16)),
+            def!(a = bor(e1, e2)),
+        ],
+    );
+
+    #[allow(overflowing_literals)]
+    let imm64_0xaaaaaaaaaaaaaaaa = Literal::constant(imm64, 0xaaaaaaaaaaaaaaaa);
+    let imm64_0x5555555555555555 = Literal::constant(imm64, 0x5555555555555555);
+    #[allow(overflowing_literals)]
+    let imm64_0xcccccccccccccccc = Literal::constant(imm64, 0xcccccccccccccccc);
+    let imm64_0x3333333333333333 = Literal::constant(imm64, 0x3333333333333333);
+    #[allow(overflowing_literals)]
+    let imm64_0xf0f0f0f0f0f0f0f0 = Literal::constant(imm64, 0xf0f0f0f0f0f0f0f0);
+    let imm64_0x0f0f0f0f0f0f0f0f = Literal::constant(imm64, 0x0f0f0f0f0f0f0f0f);
+    #[allow(overflowing_literals)]
+    let imm64_0xff00ff00ff00ff00 = Literal::constant(imm64, 0xff00ff00ff00ff00);
+    let imm64_0x00ff00ff00ff00ff = Literal::constant(imm64, 0x00ff00ff00ff00ff);
+    #[allow(overflowing_literals)]
+    let imm64_0xffff0000ffff0000 = Literal::constant(imm64, 0xffff0000ffff0000);
+    let imm64_0x0000ffff0000ffff = Literal::constant(imm64, 0x0000ffff0000ffff);
+    let imm64_32 = Literal::constant(imm64, 32);
+
+    expand.legalize(
+        def!(a = bitrev.I64(x)),
+        vec![
+            def!(a1 = band_imm(x, imm64_0xaaaaaaaaaaaaaaaa)),
+            def!(a2 = ushr_imm(a1, imm64_1)),
+            def!(a3 = band_imm(x, imm64_0x5555555555555555)),
+            def!(a4 = ishl_imm(a3, imm64_1)),
+            def!(b = bor(a2, a4)),
+            def!(b1 = band_imm(b, imm64_0xcccccccccccccccc)),
+            def!(b2 = ushr_imm(b1, imm64_2)),
+            def!(b3 = band_imm(b, imm64_0x3333333333333333)),
+            def!(b4 = ishl_imm(b3, imm64_2)),
+            def!(c = bor(b2, b4)),
+            def!(c1 = band_imm(c, imm64_0xf0f0f0f0f0f0f0f0)),
+            def!(c2 = ushr_imm(c1, imm64_4)),
+            def!(c3 = band_imm(c, imm64_0x0f0f0f0f0f0f0f0f)),
+            def!(c4 = ishl_imm(c3, imm64_4)),
+            def!(d = bor(c2, c4)),
+            def!(d1 = band_imm(d, imm64_0xff00ff00ff00ff00)),
+            def!(d2 = ushr_imm(d1, imm64_8)),
+            def!(d3 = band_imm(d, imm64_0x00ff00ff00ff00ff)),
+            def!(d4 = ishl_imm(d3, imm64_8)),
+            def!(e = bor(d2, d4)),
+            def!(e1 = band_imm(e, imm64_0xffff0000ffff0000)),
+            def!(e2 = ushr_imm(e1, imm64_16)),
+            def!(e3 = band_imm(e, imm64_0x0000ffff0000ffff)),
+            def!(e4 = ishl_imm(e3, imm64_16)),
+            def!(f = bor(e2, e4)),
+            def!(f1 = ushr_imm(f, imm64_32)),
+            def!(f2 = ishl_imm(f, imm64_32)),
+            def!(a = bor(f1, f2)),
+        ],
+    );
+
+    // Floating-point sign manipulations.
+    for &(ty, const_inst, minus_zero) in &[
+        (F32, f32const, &Literal::bits(ieee32, 0x80000000)),
+        (F64, f64const, &Literal::bits(ieee64, 0x8000000000000000)),
+    ] {
+        expand.legalize(
+            def!(a = fabs.ty(x)),
+            vec![def!(b = const_inst(minus_zero)), def!(a = band_not(x, b))],
+        );
+
+        expand.legalize(
+            def!(a = fneg.ty(x)),
+            vec![def!(b = const_inst(minus_zero)), def!(a = bxor(x, b))],
+        );
+
+        expand.legalize(
+            def!(a = fcopysign.ty(x, y)),
+            vec![
+                def!(b = const_inst(minus_zero)),
+                def!(a1 = band_not(x, b)),
+                def!(a2 = band(y, b)),
+                def!(a = bor(a1, a2)),
+            ],
+        );
+    }
+
+    expand.custom_legalize(br_icmp, "expand_br_icmp");
+
+    let mut groups = TransformGroups::new();
+
+    narrow.finish_and_add_to(&mut groups);
+    let expand_id = expand.finish_and_add_to(&mut groups);
+
+    // Expansions using CPU flags.
+    let mut expand_flags = TransformGroupBuilder::new(
+        "expand_flags",
+        r#"
+        Instruction expansions for architectures with flags.
+
+        Expand some instructions using CPU flags, then fall back to the normal
+        expansions. Not all architectures support CPU flags, so these patterns
+        are kept separate.
+    "#,
+    )
+    .chain_with(expand_id);
+
+    let imm64_0 = Literal::constant(imm64, 0);
+    let intcc_ne = Literal::enumerator_for(intcc, "ne");
+    let intcc_eq = Literal::enumerator_for(intcc, "eq");
+
+    expand_flags.legalize(
+        def!(trapnz(x, c)),
+        vec![
+            def!(a = ifcmp_imm(x, imm64_0)),
+            def!(trapif(intcc_ne, a, c)),
+        ],
+    );
+
+    expand_flags.legalize(
+        def!(trapz(x, c)),
+        vec![
+            def!(a = ifcmp_imm(x, imm64_0)),
+            def!(trapif(intcc_eq, a, c)),
+        ],
+    );
+
+    expand_flags.finish_and_add_to(&mut groups);
+
+    // XXX The order of declarations unfortunately matters to be compatible with the Python code.
+    // When it's all migrated, we can put this next to the narrow/expand finish_and_add_to calls
+    // above.
+    widen.finish_and_add_to(&mut groups);
+
+    groups
+}
diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
index 0e6a11f0e1..5237d9ba33 100644
--- a/cranelift/codegen/meta/src/shared/mod.rs
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -4,6 +4,7 @@ pub mod entities;
 pub mod formats;
 pub mod immediates;
 pub mod instructions;
+pub mod legalize;
 pub mod settings;
 pub mod types;
 
@@ -11,12 +12,14 @@ use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::inst::InstructionGroup;
 use crate::cdsl::operands::OperandKind;
 use crate::cdsl::settings::SettingGroup;
+use crate::cdsl::xform::TransformGroups;
 
 pub struct Definitions {
     pub settings: SettingGroup,
     pub instructions: InstructionGroup,
     pub operand_kinds: OperandKinds,
     pub format_registry: FormatRegistry,
+    pub transform_groups: TransformGroups,
 }
 
 pub struct OperandKinds(Vec);
@@ -50,10 +53,14 @@ pub fn define() -> Definitions {
     let immediates = OperandKinds(immediates::define());
     let entities = OperandKinds(entities::define());
     let format_registry = formats::define(&immediates, &entities);
+    let instructions = instructions::define(&format_registry, &immediates, &entities);
+    let transform_groups = legalize::define(&instructions, &immediates);
+
     Definitions {
         settings: settings::define(),
-        instructions: instructions::define(&format_registry, &immediates, &entities),
+        instructions,
         operand_kinds: immediates,
         format_registry,
+        transform_groups,
     }
 }

From 1f21349c4b4196aa9289ce50930c4ab519edea43 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 18 Apr 2019 18:22:43 +0200
Subject: [PATCH 2408/3084] [meta] Add CPU modes to the meta crate;

---
 cranelift/codegen/meta/src/cdsl/cpu_modes.rs | 84 ++++++++++++++++++++
 cranelift/codegen/meta/src/cdsl/isa.rs       | 35 ++++++++
 cranelift/codegen/meta/src/cdsl/mod.rs       |  1 +
 cranelift/codegen/meta/src/isa/arm32/mod.rs  | 15 +++-
 cranelift/codegen/meta/src/isa/arm64/mod.rs  | 12 ++-
 cranelift/codegen/meta/src/isa/riscv/mod.rs  | 27 ++++++-
 cranelift/codegen/meta/src/isa/x86/mod.rs    | 42 +++++++++-
 7 files changed, 210 insertions(+), 6 deletions(-)
 create mode 100644 cranelift/codegen/meta/src/cdsl/cpu_modes.rs

diff --git a/cranelift/codegen/meta/src/cdsl/cpu_modes.rs b/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
new file mode 100644
index 0000000000..5e0e5b4762
--- /dev/null
+++ b/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
@@ -0,0 +1,84 @@
+use crate::cdsl::types::LaneType;
+use crate::cdsl::xform::{TransformGroup, TransformGroupIndex, TransformGroups};
+
+use std::collections::{HashMap, HashSet};
+use std::iter::FromIterator;
+
+pub struct CpuMode {
+    _name: &'static str,
+    default_legalize: Option,
+    monomorphic_legalize: Option,
+    typed_legalize: HashMap,
+}
+
+impl CpuMode {
+    pub fn new(name: &'static str) -> Self {
+        Self {
+            _name: name,
+            default_legalize: None,
+            monomorphic_legalize: None,
+            typed_legalize: HashMap::new(),
+        }
+    }
+    pub fn legalize_monomorphic(&mut self, group: &TransformGroup) {
+        assert!(self.monomorphic_legalize.is_none());
+        self.monomorphic_legalize = Some(group.id);
+    }
+    pub fn legalize_default(&mut self, group: &TransformGroup) {
+        assert!(self.default_legalize.is_none());
+        self.default_legalize = Some(group.id);
+    }
+    pub fn legalize_type(&mut self, lane_type: impl Into, group: &TransformGroup) {
+        assert!(self
+            .typed_legalize
+            .insert(lane_type.into().to_string(), group.id)
+            .is_none());
+    }
+
+    /// Returns a deterministically ordered, deduplicated list of TransformGroupIndex for the
+    /// transitive set of TransformGroup this TargetIsa uses.
+    pub fn transitive_transform_groups(
+        &self,
+        all_groups: &TransformGroups,
+    ) -> Vec {
+        let mut roots = Vec::new();
+        if let Some(i) = &self.default_legalize {
+            roots.push(*i);
+        }
+        if let Some(i) = &self.monomorphic_legalize {
+            roots.push(*i);
+        }
+        roots.extend(self.typed_legalize.values().cloned());
+
+        let mut set = HashSet::new();
+        for root in roots {
+            set.insert(root);
+            let mut base = root;
+            // Follow the chain of chain_with.
+            while let Some(chain_with) = &all_groups.get(base).chain_with {
+                set.insert(*chain_with);
+                base = *chain_with;
+            }
+        }
+
+        let mut ret = Vec::from_iter(set);
+        ret.sort();
+        ret
+    }
+
+    /// Returns a deterministically ordered, deduplicated list of TransformGroupIndex for the directly
+    /// reachable set of TransformGroup this TargetIsa uses.
+    pub fn direct_transform_groups(&self) -> Vec {
+        let mut set = HashSet::new();
+        if let Some(i) = &self.default_legalize {
+            set.insert(*i);
+        }
+        if let Some(i) = &self.monomorphic_legalize {
+            set.insert(*i);
+        }
+        set.extend(self.typed_legalize.values().cloned());
+        let mut ret = Vec::from_iter(set);
+        ret.sort();
+        ret
+    }
+}
diff --git a/cranelift/codegen/meta/src/cdsl/isa.rs b/cranelift/codegen/meta/src/cdsl/isa.rs
index 1091f6db80..b8cfa17465 100644
--- a/cranelift/codegen/meta/src/cdsl/isa.rs
+++ b/cranelift/codegen/meta/src/cdsl/isa.rs
@@ -1,12 +1,18 @@
+use crate::cdsl::cpu_modes::CpuMode;
 use crate::cdsl::inst::InstructionGroup;
 use crate::cdsl::regs::IsaRegs;
 use crate::cdsl::settings::SettingGroup;
+use crate::cdsl::xform::{TransformGroupIndex, TransformGroups};
+
+use std::collections::HashSet;
+use std::iter::FromIterator;
 
 pub struct TargetIsa {
     pub name: &'static str,
     pub instructions: InstructionGroup,
     pub settings: SettingGroup,
     pub regs: IsaRegs,
+    pub cpu_modes: Vec,
 }
 
 impl TargetIsa {
@@ -15,12 +21,41 @@ impl TargetIsa {
         instructions: InstructionGroup,
         settings: SettingGroup,
         regs: IsaRegs,
+        cpu_modes: Vec,
     ) -> Self {
         Self {
             name,
             instructions,
             settings,
             regs,
+            cpu_modes,
         }
     }
+
+    /// Returns a deterministically ordered, deduplicated list of TransformGroupIndex for the
+    /// transitive set of TransformGroup this TargetIsa uses.
+    pub fn transitive_transform_groups(
+        &self,
+        all_groups: &TransformGroups,
+    ) -> Vec {
+        let mut set = HashSet::new();
+        for cpu_mode in &self.cpu_modes {
+            set.extend(cpu_mode.transitive_transform_groups(all_groups));
+        }
+        let mut vec = Vec::from_iter(set);
+        vec.sort();
+        vec
+    }
+
+    /// Returns a deterministically ordered, deduplicated list of TransformGroupIndex for the directly
+    /// reachable set of TransformGroup this TargetIsa uses.
+    pub fn direct_transform_groups(&self) -> Vec {
+        let mut set = HashSet::new();
+        for cpu_mode in &self.cpu_modes {
+            set.extend(cpu_mode.direct_transform_groups());
+        }
+        let mut vec = Vec::from_iter(set);
+        vec.sort();
+        vec
+    }
 }
diff --git a/cranelift/codegen/meta/src/cdsl/mod.rs b/cranelift/codegen/meta/src/cdsl/mod.rs
index a6c8be95fe..516b04d0b2 100644
--- a/cranelift/codegen/meta/src/cdsl/mod.rs
+++ b/cranelift/codegen/meta/src/cdsl/mod.rs
@@ -5,6 +5,7 @@
 
 #[macro_use]
 pub mod ast;
+pub mod cpu_modes;
 pub mod formats;
 pub mod inst;
 pub mod isa;
diff --git a/cranelift/codegen/meta/src/isa/arm32/mod.rs b/cranelift/codegen/meta/src/isa/arm32/mod.rs
index 8c32cdcdcd..15a9190924 100644
--- a/cranelift/codegen/meta/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm32/mod.rs
@@ -1,7 +1,9 @@
+use crate::cdsl::cpu_modes::CpuMode;
 use crate::cdsl::inst::InstructionGroup;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
+
 use crate::shared::Definitions as SharedDefinitions;
 
 fn define_settings(_shared: &SettingGroup) -> SettingGroup {
@@ -52,5 +54,16 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
 
     let inst_group = InstructionGroup::new("arm32", "arm32 specific instruction set");
 
-    TargetIsa::new("arm32", inst_group, settings, regs)
+    // CPU modes for 32-bit ARM and Thumb2.
+    let mut a32 = CpuMode::new("A32");
+    let mut t32 = CpuMode::new("T32");
+
+    // TODO refine these.
+    let narrow = shared_defs.transform_groups.by_name("narrow");
+    a32.legalize_default(narrow);
+    t32.legalize_default(narrow);
+
+    let cpu_modes = vec![a32, t32];
+
+    TargetIsa::new("arm32", inst_group, settings, regs, cpu_modes)
 }
diff --git a/cranelift/codegen/meta/src/isa/arm64/mod.rs b/cranelift/codegen/meta/src/isa/arm64/mod.rs
index 46baac90e6..4aaeb2c336 100644
--- a/cranelift/codegen/meta/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm64/mod.rs
@@ -1,7 +1,9 @@
+use crate::cdsl::cpu_modes::CpuMode;
 use crate::cdsl::inst::InstructionGroup;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
+
 use crate::shared::Definitions as SharedDefinitions;
 
 fn define_settings(_shared: &SettingGroup) -> SettingGroup {
@@ -48,5 +50,13 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
 
     let inst_group = InstructionGroup::new("arm64", "arm64 specific instruction set");
 
-    TargetIsa::new("arm64", inst_group, settings, regs)
+    let mut a64 = CpuMode::new("A64");
+
+    // TODO refine these.
+    let narrow = shared_defs.transform_groups.by_name("narrow");
+    a64.legalize_default(narrow);
+
+    let cpu_modes = vec![a64];
+
+    TargetIsa::new("arm64", inst_group, settings, regs, cpu_modes)
 }
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index 9d5a78a5a0..4e947d3184 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -1,7 +1,11 @@
+use crate::cdsl::cpu_modes::CpuMode;
 use crate::cdsl::inst::InstructionGroup;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
+
+use crate::shared::types::Float::{F32, F64};
+use crate::shared::types::Int::{I32, I64};
 use crate::shared::Definitions as SharedDefinitions;
 
 fn define_settings(shared: &SettingGroup) -> SettingGroup {
@@ -84,5 +88,26 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
 
     let inst_group = InstructionGroup::new("riscv", "riscv specific instruction set");
 
-    TargetIsa::new("riscv", inst_group, settings, regs)
+    // CPU modes for 32-bit and 64-bit operation.
+    let mut rv_32 = CpuMode::new("RV32");
+    let mut rv_64 = CpuMode::new("RV64");
+
+    let expand = shared_defs.transform_groups.by_name("expand");
+    let narrow = shared_defs.transform_groups.by_name("narrow");
+    rv_32.legalize_monomorphic(expand);
+    rv_32.legalize_default(narrow);
+    rv_32.legalize_type(I32, expand);
+    rv_32.legalize_type(F32, expand);
+    rv_32.legalize_type(F64, expand);
+
+    rv_64.legalize_monomorphic(expand);
+    rv_64.legalize_default(narrow);
+    rv_64.legalize_type(I32, expand);
+    rv_64.legalize_type(I64, expand);
+    rv_64.legalize_type(F32, expand);
+    rv_64.legalize_type(F64, expand);
+
+    let cpu_modes = vec![rv_32, rv_64];
+
+    TargetIsa::new("riscv", inst_group, settings, regs, cpu_modes)
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index 6c5fe4d097..8080ea5d43 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -1,11 +1,16 @@
-mod instructions;
-
+use crate::cdsl::cpu_modes::CpuMode;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 
+use crate::shared::types::Bool::B1;
+use crate::shared::types::Float::{F32, F64};
+use crate::shared::types::Int::{I16, I32, I64, I8};
 use crate::shared::Definitions as SharedDefinitions;
 
+mod instructions;
+mod legalize;
+
 fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     let mut settings = SettingGroupBuilder::new("x86");
 
@@ -118,6 +123,37 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let regs = define_registers();
 
     let inst_group = instructions::define(&shared_defs.format_registry);
+    legalize::define(shared_defs, &inst_group);
 
-    TargetIsa::new("x86", inst_group, settings, regs)
+    // CPU modes for 32-bit and 64-bit operations.
+    let mut x86_64 = CpuMode::new("I64");
+    let mut x86_32 = CpuMode::new("I32");
+
+    let expand_flags = shared_defs.transform_groups.by_name("expand_flags");
+    let narrow = shared_defs.transform_groups.by_name("narrow");
+    let widen = shared_defs.transform_groups.by_name("widen");
+    let x86_expand = shared_defs.transform_groups.by_name("x86_expand");
+
+    x86_32.legalize_monomorphic(expand_flags);
+    x86_32.legalize_default(narrow);
+    x86_32.legalize_type(B1, expand_flags);
+    x86_32.legalize_type(I8, widen);
+    x86_32.legalize_type(I16, widen);
+    x86_32.legalize_type(I32, x86_expand);
+    x86_32.legalize_type(F32, x86_expand);
+    x86_32.legalize_type(F64, x86_expand);
+
+    x86_64.legalize_monomorphic(expand_flags);
+    x86_64.legalize_default(narrow);
+    x86_64.legalize_type(B1, expand_flags);
+    x86_64.legalize_type(I8, widen);
+    x86_64.legalize_type(I16, widen);
+    x86_64.legalize_type(I32, x86_expand);
+    x86_64.legalize_type(I64, x86_expand);
+    x86_64.legalize_type(F32, x86_expand);
+    x86_64.legalize_type(F64, x86_expand);
+
+    let cpu_modes = vec![x86_64, x86_32];
+
+    TargetIsa::new("x86", inst_group, settings, regs, cpu_modes)
 }

From e3e66acfb18f1c1f9942f07359a1d0215c24b5c4 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 18 Apr 2019 17:53:10 +0200
Subject: [PATCH 2409/3084] [meta] Generate legalizations in the Rust crate;

---
 cranelift/codegen/meta/src/gen_inst.rs      |   2 +-
 cranelift/codegen/meta/src/gen_legalizer.rs | 578 ++++++++++++++++++++
 cranelift/codegen/meta/src/lib.rs           |   9 +
 cranelift/codegen/meta/src/srcgen.rs        |   6 +-
 cranelift/codegen/meta/src/unique_table.rs  |   3 +
 5 files changed, 594 insertions(+), 4 deletions(-)
 create mode 100644 cranelift/codegen/meta/src/gen_legalizer.rs

diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index c975cef961..a93538ca3f 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -665,7 +665,7 @@ fn typeset_to_string(ts: &TypeSet) -> String {
 }
 
 /// Generate the table of ValueTypeSets described by type_sets.
-fn gen_typesets_table(type_sets: &UniqueTable, fmt: &mut Formatter) {
+pub fn gen_typesets_table(type_sets: &UniqueTable, fmt: &mut Formatter) {
     if type_sets.len() == 0 {
         return;
     }
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
new file mode 100644
index 0000000000..de7a8068b9
--- /dev/null
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -0,0 +1,578 @@
+use crate::cdsl::ast::{Def, DefPool, VarPool};
+use crate::cdsl::formats::FormatRegistry;
+use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::type_inference::Constraint;
+use crate::cdsl::typevar::{TypeSet, TypeVar};
+use crate::cdsl::xform::{Transform, TransformGroup, TransformGroups};
+
+use crate::error;
+use crate::gen_inst::gen_typesets_table;
+use crate::srcgen::Formatter;
+use crate::unique_table::UniqueTable;
+
+use std::collections::{HashMap, HashSet};
+use std::iter::FromIterator;
+
+/// Given a `Def` node, emit code that extracts all the instruction fields from
+/// `pos.func.dfg[iref]`.
+///
+/// Create local variables named after the `Var` instances in `node`.
+///
+/// Also create a local variable named `predicate` with the value of the evaluated instruction
+/// predicate, or `true` if the node has no predicate.
+fn unwrap_inst(
+    transform: &Transform,
+    format_registry: &FormatRegistry,
+    fmt: &mut Formatter,
+) -> bool {
+    let var_pool = &transform.var_pool;
+    let def_pool = &transform.def_pool;
+
+    let def = def_pool.get(transform.src);
+    let apply = &def.apply;
+    let inst = &apply.inst;
+    let iform = format_registry.get(inst.format);
+
+    fmt.comment(format!(
+        "Unwrap {}",
+        def.to_comment_string(&transform.var_pool)
+    ));
+
+    // Extract the Var arguments.
+    let arg_names = apply
+        .args
+        .iter()
+        .map(|arg| match arg.maybe_var() {
+            Some(var_index) => var_pool.get(var_index).name,
+            None => "_",
+        })
+        .collect::>()
+        .join(", ");
+
+    fmtln!(
+        fmt,
+        "let ({}, predicate) = if let crate::ir::InstructionData::{} {{",
+        arg_names,
+        iform.name
+    );
+    fmt.indent(|fmt| {
+        // Fields are encoded directly.
+        for field in &iform.imm_fields {
+            fmtln!(fmt, "{},", field.member);
+        }
+
+        if iform.num_value_operands == 1 {
+            fmt.line("arg,");
+        } else if iform.has_value_list || iform.num_value_operands > 1 {
+            fmt.line("ref args,");
+        }
+
+        fmt.line("..");
+        fmt.outdented_line("} = pos.func.dfg[inst] {");
+        fmt.line("let func = &pos.func;");
+
+        if iform.has_value_list {
+            fmt.line("let args = args.as_slice(&func.dfg.value_lists);");
+        } else if iform.num_value_operands == 1 {
+            fmt.line("let args = [arg];")
+        }
+
+        // Generate the values for the tuple.
+        fmt.line("(");
+        fmt.indent(|fmt| {
+            for (op_num, op) in inst.operands_in.iter().enumerate() {
+                if op.is_immediate() {
+                    let n = inst.imm_opnums.iter().position(|&i| i == op_num).unwrap();
+                    fmtln!(fmt, "{},", iform.imm_fields[n].member);
+                } else if op.is_value() {
+                    let n = inst.value_opnums.iter().position(|&i| i == op_num).unwrap();
+                    fmtln!(fmt, "func.dfg.resolve_aliases(args[{}]),", n);
+                }
+            }
+
+            // Evaluate the instruction predicate if any.
+            fmt.multi_line(
+                &apply
+                    .inst_predicate_with_ctrl_typevar(format_registry, var_pool)
+                    .rust_predicate(),
+            );
+        });
+        fmt.line(")");
+
+        fmt.outdented_line("} else {");
+        fmt.line(r#"unreachable!("bad instruction format")"#);
+    });
+    fmtln!(fmt, "};");
+
+    for &op_num in &inst.value_opnums {
+        let arg = &apply.args[op_num];
+        if let Some(var_index) = arg.maybe_var() {
+            let var = var_pool.get(var_index);
+            if var.has_free_typevar() {
+                fmtln!(
+                    fmt,
+                    "let typeof_{} = pos.func.dfg.value_type({});",
+                    var.name,
+                    var.name
+                );
+            }
+        }
+    }
+
+    // If the definition creates results, detach the values and place them in locals.
+    let mut replace_inst = false;
+    if def.defined_vars.len() > 0 {
+        if def.defined_vars
+            == def_pool
+                .get(var_pool.get(def.defined_vars[0]).dst_def.unwrap())
+                .defined_vars
+        {
+            // Special case: The instruction replacing node defines the exact same values.
+            fmt.comment(format!(
+                "Results handled by {}.",
+                def_pool
+                    .get(var_pool.get(def.defined_vars[0]).dst_def.unwrap())
+                    .to_comment_string(var_pool)
+            ));
+            replace_inst = true;
+        } else {
+            // Boring case: Detach the result values, capture them in locals.
+            for &var_index in &def.defined_vars {
+                fmtln!(fmt, "let {};", var_pool.get(var_index).name);
+            }
+
+            fmt.line("{");
+            fmt.indent(|fmt| {
+                fmt.line("let r = pos.func.dfg.inst_results(inst);");
+                for i in 0..def.defined_vars.len() {
+                    let var = var_pool.get(def.defined_vars[i]);
+                    fmtln!(fmt, "{} = r[{}];", var.name, i);
+                }
+            });
+            fmt.line("}");
+
+            for &var_index in &def.defined_vars {
+                let var = var_pool.get(var_index);
+                if var.has_free_typevar() {
+                    fmtln!(
+                        fmt,
+                        "let typeof_{} = pos.func.dfg.value_type({});",
+                        var.name,
+                        var.name
+                    );
+                }
+            }
+        }
+    }
+    replace_inst
+}
+
+fn build_derived_expr(tv: &TypeVar) -> String {
+    let base = match &tv.base {
+        Some(base) => base,
+        None => {
+            assert!(tv.name.starts_with("typeof_"));
+            return format!("Some({})", tv.name);
+        }
+    };
+    let base_expr = build_derived_expr(&base.type_var);
+    format!(
+        "{}.map(|t: crate::ir::Type| t.{}())",
+        base_expr,
+        base.derived_func.name()
+    )
+}
+
+/// Emit rust code for the given check.
+///
+/// The emitted code is a statement redefining the `predicate` variable like this:
+///     let predicate = predicate && ...
+fn emit_runtime_typecheck<'a, 'b>(
+    constraint: &'a Constraint,
+    type_sets: &mut UniqueTable<'a, TypeSet>,
+    fmt: &mut Formatter,
+) {
+    match constraint {
+        Constraint::InTypeset(tv, ts) => {
+            let ts_index = type_sets.add(&ts);
+            fmt.comment(format!(
+                "{} must belong to {:?}",
+                tv.name,
+                type_sets.get(ts_index)
+            ));
+            fmtln!(
+                fmt,
+                "let predicate = predicate && TYPE_SETS[{}].contains({});",
+                ts_index,
+                tv.name
+            );
+        }
+        Constraint::Eq(tv1, tv2) => {
+            fmtln!(
+                fmt,
+                "let predicate = predicate && match ({}, {}) {{",
+                build_derived_expr(tv1),
+                build_derived_expr(tv2)
+            );
+            fmt.indent(|fmt| {
+                fmt.line("(Some(a), Some(b)) => a == b,");
+                fmt.comment("On overflow, constraint doesn\'t apply");
+                fmt.line("_ => false,");
+            });
+            fmtln!(fmt, "};");
+        }
+        Constraint::WiderOrEq(tv1, tv2) => {
+            fmtln!(
+                fmt,
+                "let predicate = predicate && match ({}, {}) {{",
+                build_derived_expr(tv1),
+                build_derived_expr(tv2)
+            );
+            fmt.indent(|fmt| {
+                fmt.line("(Some(a), Some(b)) => a.wider_or_equal(b),");
+                fmt.comment("On overflow, constraint doesn\'t apply");
+                fmt.line("_ => false,");
+            });
+            fmtln!(fmt, "};");
+        }
+    }
+}
+
+/// Determine if `node` represents one of the value splitting instructions: `isplit` or `vsplit.
+/// These instructions are lowered specially by the `legalize::split` module.
+fn is_value_split(def: &Def) -> bool {
+    let name = def.apply.inst.name;
+    name == "isplit" || name == "vsplit"
+}
+
+fn emit_dst_inst(def: &Def, def_pool: &DefPool, var_pool: &VarPool, fmt: &mut Formatter) {
+    let defined_vars = {
+        let vars = def
+            .defined_vars
+            .iter()
+            .map(|&var_index| var_pool.get(var_index).name)
+            .collect::>();
+        if vars.len() == 1 {
+            vars[0].to_string()
+        } else {
+            format!("({})", vars.join(", "))
+        }
+    };
+
+    if is_value_split(def) {
+        // Split instructions are not emitted with the builder, but by calling special functions in
+        // the `legalizer::split` module. These functions will eliminate concat-split patterns.
+        fmt.line("let curpos = pos.position();");
+        fmt.line("let srcloc = pos.srcloc();");
+        fmtln!(
+            fmt,
+            "let {} = split::{}(pos.func, cfg, curpos, srcloc, {});",
+            defined_vars,
+            def.apply.inst.snake_name(),
+            def.apply.args[0].to_rust_code(var_pool)
+        );
+        return;
+    }
+
+    if def.defined_vars.is_empty() {
+        // This node doesn't define any values, so just insert the new instruction.
+        fmtln!(
+            fmt,
+            "pos.ins().{};",
+            def.apply.rust_builder(&def.defined_vars, var_pool)
+        );
+        return;
+    }
+
+    if let Some(src_def0) = var_pool.get(def.defined_vars[0]).src_def {
+        if def.defined_vars == def_pool.get(src_def0).defined_vars {
+            // The replacement instruction defines the exact same values as the source pattern.
+            // Unwrapping would have left the results intact.  Replace the whole instruction.
+            fmtln!(
+                fmt,
+                "let {} = pos.func.dfg.replace(inst).{};",
+                defined_vars,
+                def.apply.rust_builder(&def.defined_vars, var_pool)
+            );
+
+            // We need to bump the cursor so following instructions are inserted *after* the
+            // replaced instruction.
+            fmt.line("if pos.current_inst() == Some(inst) {");
+            fmt.indent(|fmt| {
+                fmt.line("pos.next_inst();");
+            });
+            fmt.line("}");
+            return;
+        }
+    }
+
+    // Insert a new instruction.
+    let mut builder = format!("let {} = pos.ins()", defined_vars);
+
+    if def.defined_vars.len() == 1 && var_pool.get(def.defined_vars[0]).is_output() {
+        // Reuse the single source result value.
+        builder = format!(
+            "{}.with_result({})",
+            builder,
+            var_pool.get(def.defined_vars[0]).to_rust_code()
+        );
+    } else if def
+        .defined_vars
+        .iter()
+        .any(|&var_index| var_pool.get(var_index).is_output())
+    {
+        // There are more than one output values that can be reused.
+        let array = def
+            .defined_vars
+            .iter()
+            .map(|&var_index| {
+                let var = var_pool.get(var_index);
+                if var.is_output() {
+                    format!("Some({})", var.name)
+                } else {
+                    "None".into()
+                }
+            })
+            .collect::>()
+            .join(", ");
+        builder = format!("{}.with_results([{}])", builder, array);
+    }
+
+    fmtln!(
+        fmt,
+        "{}.{};",
+        builder,
+        def.apply.rust_builder(&def.defined_vars, var_pool)
+    );
+}
+
+/// Emit code for `transform`, assuming that the opcode of transform's root instruction
+/// has already been matched.
+///
+/// `inst: Inst` is the variable to be replaced. It is pointed to by `pos: Cursor`.
+/// `dfg: DataFlowGraph` is available and mutable.
+fn gen_transform<'a>(
+    transform: &'a Transform,
+    format_registry: &FormatRegistry,
+    type_sets: &mut UniqueTable<'a, TypeSet>,
+    fmt: &mut Formatter,
+) {
+    // Unwrap the source instruction, create local variables for the input variables.
+    let replace_inst = unwrap_inst(&transform, format_registry, fmt);
+
+    // Emit any runtime checks; these will rebind `predicate` emitted by unwrap_inst().
+    for constraint in &transform.type_env.constraints {
+        emit_runtime_typecheck(constraint, type_sets, fmt);
+    }
+
+    // Guard the actual expansion by `predicate`.
+    fmt.line("if predicate {");
+    fmt.indent(|fmt| {
+        // If we're going to delete `inst`, we need to detach its results first so they can be
+        // reattached during pattern expansion.
+        if !replace_inst {
+            fmt.line("pos.func.dfg.clear_results(inst);");
+        }
+
+        // Emit the destination pattern.
+        for &def_index in &transform.dst {
+            emit_dst_inst(
+                transform.def_pool.get(def_index),
+                &transform.def_pool,
+                &transform.var_pool,
+                fmt,
+            );
+        }
+
+        // Delete the original instruction if we didn't have an opportunity to replace it.
+        if !replace_inst {
+            fmt.line("let removed = pos.remove_inst();");
+            fmt.line("debug_assert_eq!(removed, inst);");
+        }
+        fmt.line("return true;");
+    });
+    fmt.line("}");
+}
+
+fn gen_transform_group<'a>(
+    group: &'a TransformGroup,
+    format_registry: &FormatRegistry,
+    transform_groups: &TransformGroups,
+    type_sets: &mut UniqueTable<'a, TypeSet>,
+    fmt: &mut Formatter,
+) {
+    fmt.doc_comment(group.doc);
+    fmt.line("#[allow(unused_variables,unused_assignments,non_snake_case)]");
+
+    // Function arguments.
+    fmtln!(fmt, "pub fn {}(", group.name);
+    fmt.indent(|fmt| {
+        fmt.line("inst: crate::ir::Inst,");
+        fmt.line("func: &mut crate::ir::Function,");
+        fmt.line("cfg: &mut crate::flowgraph::ControlFlowGraph,");
+        fmt.line("isa: &crate::isa::TargetIsa,");
+    });
+    fmtln!(fmt, ") -> bool {");
+
+    // Function body.
+    fmt.indent(|fmt| {
+        fmt.line("use crate::ir::InstBuilder;");
+        fmt.line("use crate::cursor::{Cursor, FuncCursor};");
+        fmt.line("let mut pos = FuncCursor::new(func).at_inst(inst);");
+        fmt.line("pos.use_srcloc(inst);");
+
+        // Group the transforms by opcode so we can generate a big switch.
+        // Preserve ordering.
+        let mut inst_to_transforms = HashMap::new();
+        for transform in &group.transforms {
+            let def_index = transform.src;
+            let inst = &transform.def_pool.get(def_index).apply.inst;
+            inst_to_transforms
+                .entry(inst.camel_name.clone())
+                .or_insert(Vec::new())
+                .push(transform);
+        }
+
+        let mut sorted_inst_names = Vec::from_iter(inst_to_transforms.keys());
+        sorted_inst_names.sort();
+
+        fmt.line("{");
+        fmt.indent(|fmt| {
+            fmt.line("match pos.func.dfg[inst].opcode() {");
+            fmt.indent(|fmt| {
+                for camel_name in sorted_inst_names {
+                    fmtln!(fmt, "ir::Opcode::{} => {{", camel_name);
+                    fmt.indent(|fmt| {
+                        for transform in inst_to_transforms.get(camel_name).unwrap() {
+                            gen_transform(transform, format_registry, type_sets, fmt);
+                        }
+                    });
+                    fmtln!(fmt, "}");
+                    fmt.empty_line();
+                }
+
+                // Emit the custom transforms. The Rust compiler will complain about any overlap with
+                // the normal transforms.
+                for (inst_camel_name, func_name) in &group.custom_legalizes {
+                    fmtln!(fmt, "ir::Opcode::{} => {{", inst_camel_name);
+                    fmt.indent(|fmt| {
+                        fmtln!(fmt, "{}(inst, pos.func, cfg, isa);", func_name);
+                        fmt.line("return true;");
+                    });
+                    fmtln!(fmt, "}");
+                    fmt.empty_line();
+                }
+
+                // We'll assume there are uncovered opcodes.
+                fmt.line("_ => {},");
+            });
+            fmt.line("}");
+        });
+        fmt.line("}");
+
+        // If we fall through, nothing was expanded; call the chain if any.
+        match &group.chain_with {
+            Some(group_id) => fmtln!(
+                fmt,
+                "{}(inst, pos.func, cfg, isa)",
+                transform_groups.get(*group_id).rust_name()
+            ),
+            None => fmt.line("false"),
+        };
+    });
+    fmtln!(fmt, "}");
+    fmt.empty_line();
+}
+
+/// Generate legalization functions for `isa` and add any shared `TransformGroup`s
+/// encountered to `shared_groups`.
+///
+/// Generate `TYPE_SETS` and `LEGALIZE_ACTIONS` tables.
+fn gen_isa(
+    isa: &TargetIsa,
+    format_registry: &FormatRegistry,
+    transform_groups: &TransformGroups,
+    shared_group_names: &mut HashSet<&'static str>,
+    fmt: &mut Formatter,
+) {
+    let mut type_sets = UniqueTable::new();
+    for group_index in isa.transitive_transform_groups(transform_groups) {
+        let group = transform_groups.get(group_index);
+        match group.isa_name {
+            Some(isa_name) => {
+                assert!(
+                    isa_name == isa.name,
+                    "ISA-specific legalizations must be used by the same ISA"
+                );
+                gen_transform_group(
+                    group,
+                    format_registry,
+                    transform_groups,
+                    &mut type_sets,
+                    fmt,
+                );
+            }
+            None => {
+                shared_group_names.insert(group.name);
+            }
+        }
+    }
+
+    gen_typesets_table(&type_sets, fmt);
+
+    let direct_groups = isa.direct_transform_groups();
+    fmtln!(
+        fmt,
+        "pub static LEGALIZE_ACTIONS: [isa::Legalize; {}] = [",
+        direct_groups.len()
+    );
+    fmt.indent(|fmt| {
+        for group_index in direct_groups {
+            fmtln!(fmt, "{},", transform_groups.get(group_index).rust_name());
+        }
+    });
+    fmtln!(fmt, "];");
+}
+
+/// Generate the legalizer files.
+pub fn generate(
+    isas: &Vec,
+    format_registry: &FormatRegistry,
+    transform_groups: &TransformGroups,
+    filename_prefix: &str,
+    out_dir: &str,
+) -> Result<(), error::Error> {
+    let mut shared_group_names = HashSet::new();
+
+    for isa in isas {
+        let mut fmt = Formatter::new();
+        gen_isa(
+            isa,
+            format_registry,
+            transform_groups,
+            &mut shared_group_names,
+            &mut fmt,
+        );
+        fmt.update_file(format!("{}-{}.rs", filename_prefix, isa.name), out_dir)?;
+    }
+
+    // Generate shared legalize groups.
+    let mut fmt = Formatter::new();
+    let mut type_sets = UniqueTable::new();
+    let mut sorted_shared_group_names = Vec::from_iter(shared_group_names);
+    sorted_shared_group_names.sort();
+    for group_name in &sorted_shared_group_names {
+        let group = transform_groups.by_name(group_name);
+        gen_transform_group(
+            group,
+            format_registry,
+            transform_groups,
+            &mut type_sets,
+            &mut fmt,
+        );
+    }
+    gen_typesets_table(&type_sets, &mut fmt);
+    fmt.update_file(format!("{}r.rs", filename_prefix), out_dir)?;
+
+    Ok(())
+}
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index fccba327e0..bbf2a031a3 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -6,6 +6,7 @@ pub mod error;
 pub mod isa;
 
 mod gen_inst;
+mod gen_legalizer;
 mod gen_registers;
 mod gen_settings;
 mod gen_types;
@@ -45,6 +46,14 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
         &out_dir,
     )?;
 
+    gen_legalizer::generate(
+        &isas,
+        &shared_defs.format_registry,
+        &shared_defs.transform_groups,
+        "new_legalize",
+        &out_dir,
+    )?;
+
     for isa in isas {
         gen_registers::generate(&isa, &format!("registers-{}.rs", isa.name), &out_dir)?;
         gen_settings::generate(
diff --git a/cranelift/codegen/meta/src/srcgen.rs b/cranelift/codegen/meta/src/srcgen.rs
index ac2b8e8e03..0a1d21c7b9 100644
--- a/cranelift/codegen/meta/src/srcgen.rs
+++ b/cranelift/codegen/meta/src/srcgen.rs
@@ -79,7 +79,7 @@ impl Formatter {
 
     /// Get a string containing whitespace outdented one level. Used for
     /// lines of code that are inside a single indented block.
-    fn _get_outdent(&mut self) -> String {
+    fn get_outdent(&mut self) -> String {
         self.indent_pop();
         let s = self.get_indent();
         self.indent_push();
@@ -98,8 +98,8 @@ impl Formatter {
     }
 
     /// Emit a line outdented one level.
-    pub fn _outdented_line(&mut self, s: &str) {
-        let new_line = format!("{}{}\n", self._get_outdent(), s);
+    pub fn outdented_line(&mut self, s: &str) {
+        let new_line = format!("{}{}\n", self.get_outdent(), s);
         self.lines.push(new_line);
     }
 
diff --git a/cranelift/codegen/meta/src/unique_table.rs b/cranelift/codegen/meta/src/unique_table.rs
index d7a26f67cc..e50a64f437 100644
--- a/cranelift/codegen/meta/src/unique_table.rs
+++ b/cranelift/codegen/meta/src/unique_table.rs
@@ -31,6 +31,9 @@ impl<'entries, T: Eq + Hash> UniqueTable<'entries, T> {
     pub fn len(&self) -> usize {
         self.table.len()
     }
+    pub fn get(&self, index: usize) -> &T {
+        self.table[index]
+    }
     pub fn iter(&self) -> slice::Iter<&'entries T> {
         self.table.iter()
     }

From 95e6fc9efc6f4b422b87b348c92addae1917d9bf Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 24 Apr 2019 16:32:33 +0200
Subject: [PATCH 2410/3084] Avoid inserting checks during div/rem legalization
 when the input is a constant immediate;

---
 cranelift/codegen/src/isa/x86/enc_tables.rs   |  66 ++++++++-
 .../filetests/isa/x86/legalize-div-traps.clif | 125 +++++++++++++++++-
 2 files changed, 182 insertions(+), 9 deletions(-)

diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index bdd3458f14..0ed238ac18 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -91,6 +91,25 @@ fn size_plus_maybe_sib_or_offset_for_in_reg_1(
     sizing.base_size + additional_size_if(1, inst, divert, func, needs_sib_byte_or_offset)
 }
 
+/// If the value's definition is a constant immediate, returns its unpacked value, or None
+/// otherwise.
+fn maybe_iconst_imm(pos: &FuncCursor, value: ir::Value) -> Option {
+    if let ir::ValueDef::Result(inst, _) = &pos.func.dfg.value_def(value) {
+        if let ir::InstructionData::UnaryImm {
+            opcode: ir::Opcode::Iconst,
+            imm,
+        } = &pos.func.dfg[*inst]
+        {
+            let value: i64 = (*imm).into();
+            Some(value)
+        } else {
+            None
+        }
+    } else {
+        None
+    }
+}
+
 /// Expand the `sdiv` and `srem` instructions using `x86_sdivmodx`.
 fn expand_sdivrem(
     inst: ir::Inst,
@@ -109,7 +128,7 @@ fn expand_sdivrem(
         } => (args[0], args[1], true),
         _ => panic!("Need sdiv/srem: {}", func.dfg.display_inst(inst, None)),
     };
-    let avoid_div_traps = isa.flags().avoid_div_traps();
+
     let old_ebb = func.layout.pp_ebb(inst);
     let result = func.dfg.first_result(inst);
     let ty = func.dfg.value_type(result);
@@ -118,6 +137,8 @@ fn expand_sdivrem(
     pos.use_srcloc(inst);
     pos.func.dfg.clear_results(inst);
 
+    let avoid_div_traps = isa.flags().avoid_div_traps();
+
     // If we can tolerate native division traps, sdiv doesn't need branching.
     if !avoid_div_traps && !is_srem {
         let xhi = pos.ins().sshr_imm(x, i64::from(ty.lane_bits()) - 1);
@@ -126,6 +147,32 @@ fn expand_sdivrem(
         return;
     }
 
+    // Try to remove checks if the input value is an immediate other than 0 or -1. For these two
+    // immediates, we'd ideally replace conditional traps by traps, but this requires more
+    // manipulation of the dfg/cfg, which is out of scope here.
+    let (could_be_zero, could_be_minus_one) = if let Some(imm) = maybe_iconst_imm(&pos, y) {
+        (imm == 0, imm == -1)
+    } else {
+        (true, true)
+    };
+
+    // Put in an explicit division-by-zero trap if the environment requires it.
+    if avoid_div_traps && could_be_zero {
+        pos.ins().trapz(y, ir::TrapCode::IntegerDivisionByZero);
+    }
+
+    if !could_be_minus_one {
+        let xhi = pos.ins().sshr_imm(x, i64::from(ty.lane_bits()) - 1);
+        let reuse = if is_srem {
+            [None, Some(result)]
+        } else {
+            [Some(result), None]
+        };
+        pos.ins().with_results(reuse).x86_sdivmodx(x, xhi, y);
+        pos.remove_inst();
+        return;
+    }
+
     // EBB handling the -1 divisor case.
     let minus_one = pos.func.dfg.make_ebb();
 
@@ -139,11 +186,6 @@ fn expand_sdivrem(
     let is_m1 = pos.ins().ifcmp_imm(y, -1);
     pos.ins().brif(IntCC::Equal, is_m1, minus_one, &[]);
 
-    // Put in an explicit division-by-zero trap if the environment requires it.
-    if avoid_div_traps {
-        pos.ins().trapz(y, ir::TrapCode::IntegerDivisionByZero);
-    }
-
     // Now it is safe to execute the `x86_sdivmodx` instruction which will still trap on division
     // by zero.
     let xhi = pos.ins().sshr_imm(x, i64::from(ty.lane_bits()) - 1);
@@ -206,7 +248,17 @@ fn expand_udivrem(
 
     // Put in an explicit division-by-zero trap if the environment requires it.
     if avoid_div_traps {
-        pos.ins().trapz(y, ir::TrapCode::IntegerDivisionByZero);
+        let zero_check = if let Some(imm) = maybe_iconst_imm(&pos, y) {
+            // Ideally, we'd just replace the conditional trap with a trap when the immediate is
+            // zero, but this requires more manipulation of the dfg/cfg, which is out of scope
+            // here.
+            imm == 0
+        } else {
+            true
+        };
+        if zero_check {
+            pos.ins().trapz(y, ir::TrapCode::IntegerDivisionByZero);
+        }
     }
 
     // Now it is safe to execute the `x86_udivmodx` instruction.
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-div-traps.clif b/cranelift/filetests/filetests/isa/x86/legalize-div-traps.clif
index 40c09aee0e..2622ae48f3 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-div-traps.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-div-traps.clif
@@ -19,6 +19,32 @@ ebb0(v0: i64, v1: i64):
     ; nextln: return $d
 }
 
+function %udiv_0(i64) -> i64 {
+ebb0(v0: i64):
+    ; check: ebb0(
+    v1 = iconst.i64 0
+    ; nextln: v1 = iconst.i64 0
+    v2 = udiv v0, v1
+    ; nextln: $(fz=$V) = ifcmp_imm v1, 0
+    ; nextln: trapif eq $fz, int_divz
+    ; nextln: $(hi=$V) = iconst.i64 0
+    ; nextln: $(d=$V), $(r=$V) = x86_udivmodx v0, $hi, v1
+    return v2
+    ; nextln: return $d
+}
+
+function %udiv_minus_1(i64) -> i64 {
+ebb0(v0: i64):
+    ; check: ebb0(
+    v1 = iconst.i64 -1
+    ; nextln: v1 = iconst.i64 -1
+    v2 = udiv v0, v1
+    ; nextln: $(hi=$V) = iconst.i64 0
+    ; nextln: $(d=$V), $(r=$V) = x86_udivmodx v0, $hi, v1
+    return v2
+    ; nextln: return $d
+}
+
 function %urem(i64, i64) -> i64 {
 ebb0(v0: i64, v1: i64):
     ; check: ebb0(
@@ -31,14 +57,74 @@ ebb0(v0: i64, v1: i64):
     ; nextln: return $r
 }
 
+function %urem_0(i64) -> i64 {
+ebb0(v0: i64):
+    ; check: ebb0(
+    v1 = iconst.i64 0
+    ; nextln: v1 = iconst.i64 0
+    v2 = urem v0, v1
+    ; nextln: $(fz=$V) = ifcmp_imm v1, 0
+    ; nextln: trapif eq $fz, int_divz
+    ; nextln: $(hi=$V) = iconst.i64 0
+    ; nextln: $(d=$V), $(r=$V) = x86_udivmodx v0, $hi, v1
+    return v2
+    ; nextln: return $r
+}
+
+function %urem_minus_1(i64) -> i64 {
+ebb0(v0: i64):
+    ; check: ebb0(
+    v1 = iconst.i64 -1
+    ; nextln: v1 = iconst.i64 -1
+    v2 = urem v0, v1
+    ; nextln: $(hi=$V) = iconst.i64 0
+    ; nextln: $(d=$V), $(r=$V) = x86_udivmodx v0, $hi, v1
+    return v2
+    ; nextln: return $r
+}
+
 function %sdiv(i64, i64) -> i64 {
 ebb0(v0: i64, v1: i64):
     ; check: ebb0(
     v2 = sdiv v0, v1
-    ; nextln: $(fm1=$V) = ifcmp_imm v1, -1
-    ; nextln: brif eq $fm1, $(m1=$EBB)
     ; nextln: $(fz=$V) = ifcmp_imm v1, 0
     ; nextln: trapif eq $fz, int_divz
+    ; nextln: $(fm1=$V) = ifcmp_imm v1, -1
+    ; nextln: brif eq $fm1, $(m1=$EBB)
+    ; check: $(hi=$V) = sshr_imm
+    ; nextln: $(q=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1
+    ; nextln: jump $(done=$EBB)($q)
+    ; check: $m1:
+    ; nextln: $(imin=$V) = iconst.i64 0x8000_0000_0000_0000
+    ; nextln: $(fm=$V) = ifcmp.i64 v0, $imin
+    ; nextln: trapif eq $fm, int_ovf
+    ; check: $done(v2: i64):
+    return v2
+    ; nextln: return v2
+}
+
+function %sdiv_0(i64) -> i64 {
+ebb0(v0: i64):
+    ; check: ebb0(
+    v1 = iconst.i64 0
+    ; nextln: v1 = iconst.i64 0
+    v2 = sdiv v0, v1
+    ; nextln: $(fz=$V) = ifcmp_imm v1, 0
+    ; nextln: trapif eq $fz, int_divz
+    ; check: $(hi=$V) = sshr_imm
+    ; nextln: $(q=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1
+    return v2
+    ; nextln: return v2
+}
+
+function %sdiv_minus_1(i64) -> i64 {
+ebb0(v0: i64):
+    ; check: ebb0(
+    v1 = iconst.i64 -1
+    ; nextln: v1 = iconst.i64 -1
+    v2 = sdiv v0, v1
+    ; nextln: $(fm1=$V) = ifcmp_imm v1, -1
+    ; nextln: brif eq $fm1, $(m1=$EBB)
     ; check: $(hi=$V) = sshr_imm
     ; nextln: $(q=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1
     ; nextln: jump $(done=$EBB)($q)
@@ -57,6 +143,41 @@ function %srem(i64, i64) -> i64 {
 ebb0(v0: i64, v1: i64):
     ; check: ebb0(
     v2 = srem v0, v1
+    ; nextln: $(fz=$V) = ifcmp_imm v1, 0
+    ; nextln: trapif eq $fz, int_divz
+    ; nextln: $(fm1=$V) = ifcmp_imm v1, -1
+    ; nextln: brif eq $fm1, $(m1=$EBB)
+    ; check: $(hi=$V) = sshr_imm
+    ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1
+    ; nextln: jump $(done=$EBB)($r)
+    ; check: $m1:
+    ; nextln: $(zero=$V) = iconst.i64 0
+    ; nextln: jump $(done=$EBB)($zero)
+    ; check: $done(v2: i64):
+    return v2
+    ; nextln: return v2
+}
+
+function %srem_0(i64) -> i64 {
+ebb0(v0: i64):
+    ; check: ebb0(
+    v1 = iconst.i64 0
+    ; nextln: v1 = iconst.i64 0
+    v2 = srem v0, v1
+    ; nextln: $(fz=$V) = ifcmp_imm v1, 0
+    ; nextln: trapif eq $fz, int_divz
+    ; check: $(hi=$V) = sshr_imm
+    ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1
+    return v2
+    ; nextln: return v2
+}
+
+function %srem_minus_1(i64) -> i64 {
+ebb0(v0: i64):
+    ; check: ebb0(
+    v1 = iconst.i64 -1
+    ; nextln: v1 = iconst.i64 -1
+    v2 = srem v0, v1
     ; nextln: $(fm1=$V) = ifcmp_imm v1, -1
     ; nextln: brif eq $fm1, $(m1=$EBB)
     ; check: $(hi=$V) = sshr_imm

From 4ee2747c5bae0c21c00dcae712dc4636f6b2658d Mon Sep 17 00:00:00 2001
From: Antoni Boucher 
Date: Sun, 28 Apr 2019 13:28:17 -0400
Subject: [PATCH 2411/3084] Fix cranelift_preopt panic

Fix #611
---
 .../filetests/preopt/constant_fold.clif       | 19 +++++++++++++++++++
 cranelift/preopt/src/constant_folding.rs      |  7 +++++--
 2 files changed, 24 insertions(+), 2 deletions(-)
 create mode 100644 cranelift/filetests/filetests/preopt/constant_fold.clif

diff --git a/cranelift/filetests/filetests/preopt/constant_fold.clif b/cranelift/filetests/filetests/preopt/constant_fold.clif
new file mode 100644
index 0000000000..f0ea9539ba
--- /dev/null
+++ b/cranelift/filetests/filetests/preopt/constant_fold.clif
@@ -0,0 +1,19 @@
+test preopt
+target x86_64
+
+function %constant_fold(f64) -> f64 {
+ebb0(v0: f64):
+    v1 = f64const 0x1.0000000000000p0
+    v2 = f64const 0x1.0000000000000p1
+    v3 = fadd v1, v2
+    v4 = fadd v3, v0
+    return v4
+}
+; sameln: function %constant_fold(f64) -> f64 fast {
+; nextln: ebb0(v0: f64):
+; nextln:     v1 = f64const 0x1.0000000000000p0
+; nextln:     v2 = f64const 0x1.0000000000000p1
+; nextln:     v3 = f64const 0x1.8000000000000p1
+; nextln:     v4 = fadd v3, v0
+; nextln:     return v4
+; nextln: }
diff --git a/cranelift/preopt/src/constant_folding.rs b/cranelift/preopt/src/constant_folding.rs
index 2d7c1c2dbb..018ac04a7e 100644
--- a/cranelift/preopt/src/constant_folding.rs
+++ b/cranelift/preopt/src/constant_folding.rs
@@ -2,7 +2,7 @@
 
 use cranelift_codegen::{
     cursor::{Cursor, FuncCursor},
-    ir::{self, InstBuilder},
+    ir::{self, dfg::ValueDef, InstBuilder},
 };
 // use rustc_apfloat::{
 //     ieee::{Double, Single},
@@ -69,7 +69,10 @@ pub fn fold_constants(func: &mut ir::Function) {
 fn resolve_value_to_imm(dfg: &ir::DataFlowGraph, value: ir::Value) -> Option {
     let original = dfg.resolve_aliases(value);
 
-    let inst = dfg.value_def(original).unwrap_inst();
+    let inst = match dfg.value_def(original) {
+        ValueDef::Result(inst, _) => inst,
+        ValueDef::Param(_, _) => return None,
+    };
 
     use self::ir::{InstructionData::*, Opcode::*};
     match dfg[inst] {

From d8d573208b846873cc45b88bc25de9fd2333390e Mon Sep 17 00:00:00 2001
From: Antoni Boucher 
Date: Sun, 28 Apr 2019 14:06:41 -0400
Subject: [PATCH 2412/3084] Remove unwrap() for branch folding

---
 .../filetests/filetests/preopt/branch.clif    | 25 +++++++++++++++++++
 cranelift/preopt/src/constant_folding.rs      |  5 +++-
 2 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/cranelift/filetests/filetests/preopt/branch.clif b/cranelift/filetests/filetests/preopt/branch.clif
index 2b696fa2ef..8e139952f9 100644
--- a/cranelift/filetests/filetests/preopt/branch.clif
+++ b/cranelift/filetests/filetests/preopt/branch.clif
@@ -52,3 +52,28 @@ ebb2:
 ; nextln:     v2 = iconst.i32 24
 ; nextln:     return v2
 ; nextln: }
+
+function %brz_fold_param(b1) -> i32 {
+ebb0(v0: b1):
+    brz v0, ebb2
+    jump ebb1
+ebb1:
+    v1 = iconst.i32 42
+    return v1
+ebb2:
+    v2 = iconst.i32 24
+    return v2
+}
+; sameln: function %brz_fold_param(b1) -> i32 fast {
+; nextln: ebb0(v0: b1):
+; nextln:     brz v0, ebb2
+; nextln:     jump ebb1
+; nextln: 
+; nextln: ebb1:
+; nextln:     v1 = iconst.i32 42
+; nextln:     return v1
+; nextln: 
+; nextln: ebb2:
+; nextln:     v2 = iconst.i32 24
+; nextln:     return v2
+; nextln: }
diff --git a/cranelift/preopt/src/constant_folding.rs b/cranelift/preopt/src/constant_folding.rs
index 018ac04a7e..cf4dba2815 100644
--- a/cranelift/preopt/src/constant_folding.rs
+++ b/cranelift/preopt/src/constant_folding.rs
@@ -232,7 +232,10 @@ fn fold_branch(pos: &mut FuncCursor, inst: ir::Inst, opcode: ir::Opcode) {
         let values = pos.func.dfg.inst_args(inst);
         let inst_data = &pos.func.dfg[inst];
         (
-            resolve_value_to_imm(&pos.func.dfg, values[0]).unwrap(),
+            match resolve_value_to_imm(&pos.func.dfg, values[0]) {
+                Some(imm) => imm,
+                None => return,
+            },
             inst_data.branch_destination().unwrap(),
             values[1..].to_vec(),
         )

From 79d6978e2933422731194ac16a17e4b1d5405084 Mon Sep 17 00:00:00 2001
From: antoyo 
Date: Mon, 29 Apr 2019 10:58:39 -0400
Subject: [PATCH 2413/3084] cranelift-simplejit: add a translation mechanism
 for LibCalls (#747)

---
 cranelift/faerie/src/backend.rs               | 21 +----
 cranelift/module/src/backend.rs               | 23 ++++++
 cranelift/module/src/lib.rs                   |  2 +-
 .../simplejit/examples/simplejit-minimal.rs   |  5 +-
 cranelift/simplejit/src/backend.rs            | 82 ++++++++++++-------
 cranelift/simplejit/tests/basic.rs            | 59 ++++++++++++-
 6 files changed, 135 insertions(+), 57 deletions(-)

diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index 1deb46e073..0adb564b30 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -46,7 +46,7 @@ impl FaerieBuilder {
     /// The `libcall_names` function provides a way to translate `cranelift_codegen`'s `ir::LibCall`
     /// enum to symbols. LibCalls are inserted in the IR as part of the legalization for certain
     /// floating point instructions, and for stack probes. If you don't know what to use for this
-    /// argument, use `FaerieBuilder::default_libcall_names()`.
+    /// argument, use `cranelift_module::default_libcall_names()`.
     pub fn new(
         isa: Box,
         name: String,
@@ -65,25 +65,6 @@ impl FaerieBuilder {
             libcall_names,
         })
     }
-
-    /// Default names for `ir::LibCall`s. A function by this name is imported into the object as
-    /// part of the translation of a `ir::ExternalName::LibCall` variant.
-    pub fn default_libcall_names() -> Box String> {
-        Box::new(move |libcall| match libcall {
-            ir::LibCall::Probestack => "__cranelift_probestack".to_owned(),
-            ir::LibCall::CeilF32 => "ceilf".to_owned(),
-            ir::LibCall::CeilF64 => "ceil".to_owned(),
-            ir::LibCall::FloorF32 => "floorf".to_owned(),
-            ir::LibCall::FloorF64 => "floor".to_owned(),
-            ir::LibCall::TruncF32 => "truncf".to_owned(),
-            ir::LibCall::TruncF64 => "trunc".to_owned(),
-            ir::LibCall::NearestF32 => "nearbyintf".to_owned(),
-            ir::LibCall::NearestF64 => "nearbyint".to_owned(),
-            ir::LibCall::Memcpy => "memcpy".to_owned(),
-            ir::LibCall::Memset => "memset".to_owned(),
-            ir::LibCall::Memmove => "memmove".to_owned(),
-        })
-    }
 }
 
 /// A `FaerieBackend` implements `Backend` and emits ".o" files using the `faerie` library.
diff --git a/cranelift/module/src/backend.rs b/cranelift/module/src/backend.rs
index 956c206948..3f89e18fde 100644
--- a/cranelift/module/src/backend.rs
+++ b/cranelift/module/src/backend.rs
@@ -9,6 +9,10 @@ use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::Context;
 use cranelift_codegen::{binemit, ir};
 
+use std::borrow::ToOwned;
+use std::boxed::Box;
+use std::string::String;
+
 /// A `Backend` implements the functionality needed to support a `Module`.
 ///
 /// Two notable implementations of this trait are:
@@ -127,3 +131,22 @@ where
     /// provide additional functionality through this result.
     fn finish(self) -> Self::Product;
 }
+
+/// Default names for `ir::LibCall`s. A function by this name is imported into the object as
+/// part of the translation of a `ir::ExternalName::LibCall` variant.
+pub fn default_libcall_names() -> Box String> {
+    Box::new(move |libcall| match libcall {
+        ir::LibCall::Probestack => "__cranelift_probestack".to_owned(),
+        ir::LibCall::CeilF32 => "ceilf".to_owned(),
+        ir::LibCall::CeilF64 => "ceil".to_owned(),
+        ir::LibCall::FloorF32 => "floorf".to_owned(),
+        ir::LibCall::FloorF64 => "floor".to_owned(),
+        ir::LibCall::TruncF32 => "truncf".to_owned(),
+        ir::LibCall::TruncF64 => "trunc".to_owned(),
+        ir::LibCall::NearestF32 => "nearbyintf".to_owned(),
+        ir::LibCall::NearestF64 => "nearbyint".to_owned(),
+        ir::LibCall::Memcpy => "memcpy".to_owned(),
+        ir::LibCall::Memset => "memset".to_owned(),
+        ir::LibCall::Memmove => "memmove".to_owned(),
+    })
+}
diff --git a/cranelift/module/src/lib.rs b/cranelift/module/src/lib.rs
index e30961c3bc..33d12d96c2 100644
--- a/cranelift/module/src/lib.rs
+++ b/cranelift/module/src/lib.rs
@@ -37,7 +37,7 @@ mod backend;
 mod data_context;
 mod module;
 
-pub use crate::backend::Backend;
+pub use crate::backend::{default_libcall_names, Backend};
 pub use crate::data_context::{DataContext, DataDescription, Init};
 pub use crate::module::{
     DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleError, ModuleFunction, ModuleNamespace,
diff --git a/cranelift/simplejit/examples/simplejit-minimal.rs b/cranelift/simplejit/examples/simplejit-minimal.rs
index eb1d3257ac..3b8e147830 100644
--- a/cranelift/simplejit/examples/simplejit-minimal.rs
+++ b/cranelift/simplejit/examples/simplejit-minimal.rs
@@ -1,10 +1,11 @@
 use cranelift::prelude::*;
-use cranelift_module::{Linkage, Module};
+use cranelift_module::{default_libcall_names, Linkage, Module};
 use cranelift_simplejit::{SimpleJITBackend, SimpleJITBuilder};
 use std::mem;
 
 fn main() {
-    let mut module: Module = Module::new(SimpleJITBuilder::new());
+    let mut module: Module =
+        Module::new(SimpleJITBuilder::new(default_libcall_names()));
     let mut ctx = module.make_context();
     let mut func_ctx = FunctionBuilderContext::new();
 
diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index 8dc3ee09d0..7be3b3b2fa 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -21,17 +21,23 @@ use winapi;
 pub struct SimpleJITBuilder {
     isa: Box,
     symbols: HashMap,
+    libcall_names: Box String>,
 }
 
 impl SimpleJITBuilder {
     /// Create a new `SimpleJITBuilder`.
-    pub fn new() -> Self {
+    ///
+    /// The `libcall_names` function provides a way to translate `cranelift_codegen`'s `ir::LibCall`
+    /// enum to symbols. LibCalls are inserted in the IR as part of the legalization for certain
+    /// floating point instructions, and for stack probes. If you don't know what to use for this
+    /// argument, use `cranelift_module::default_libcall_names()`.
+    pub fn new(libcall_names: Box String>) -> Self {
         let flag_builder = settings::builder();
         let isa_builder = cranelift_native::builder().unwrap_or_else(|msg| {
             panic!("host machine is not supported: {}", msg);
         });
         let isa = isa_builder.finish(settings::Flags::new(flag_builder));
-        Self::with_isa(isa)
+        Self::with_isa(isa, libcall_names)
     }
 
     /// Create a new `SimpleJITBuilder` with an arbitrary target. This is mainly
@@ -41,10 +47,19 @@ impl SimpleJITBuilder {
     ///
     /// To create a `SimpleJITBuilder` for native use, use the `new` constructor
     /// instead.
-    pub fn with_isa(isa: Box) -> Self {
+    ///
+    /// The `libcall_names` function provides a way to translate `cranelift_codegen`'s `ir::LibCall`
+    /// enum to symbols. LibCalls are inserted in the IR as part of the legalization for certain
+    /// floating point instructions, and for stack probes. If you don't know what to use for this
+    /// argument, use `cranelift_module::default_libcall_names()`.
+    pub fn with_isa(isa: Box, libcall_names: Box String>) -> Self {
         debug_assert!(!isa.flags().is_pic(), "SimpleJIT requires non-PIC code");
         let symbols = HashMap::new();
-        Self { isa, symbols }
+        Self {
+            isa,
+            symbols,
+            libcall_names,
+        }
     }
 
     /// Define a symbol in the internal symbol table.
@@ -91,6 +106,7 @@ impl SimpleJITBuilder {
 pub struct SimpleJITBackend {
     isa: Box,
     symbols: HashMap,
+    libcall_names: Box String>,
     code_memory: Memory,
     readonly_memory: Memory,
     writable_memory: Memory,
@@ -123,6 +139,35 @@ impl SimpleJITBackend {
             None => lookup_with_dlsym(name),
         }
     }
+
+    fn get_definition(
+        &self,
+        namespace: &ModuleNamespace,
+        name: &ir::ExternalName,
+    ) -> *const u8 {
+        match *name {
+            ir::ExternalName::User { .. } => {
+                if namespace.is_function(name) {
+                    let (def, name_str, _signature) = namespace.get_function_definition(&name);
+                    match def {
+                        Some(compiled) => compiled.code,
+                        None => self.lookup_symbol(name_str),
+                    }
+                } else {
+                    let (def, name_str, _writable) = namespace.get_data_definition(&name);
+                    match def {
+                        Some(compiled) => compiled.storage,
+                        None => self.lookup_symbol(name_str),
+                    }
+                }
+            }
+            ir::ExternalName::LibCall(ref libcall) => {
+                let sym = (self.libcall_names)(*libcall);
+                self.lookup_symbol(&sym)
+            }
+            _ => panic!("invalid ExternalName {}", name),
+        }
+    }
 }
 
 impl<'simple_jit_backend> Backend for SimpleJITBackend {
@@ -149,6 +194,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         Self {
             isa: builder.isa,
             symbols: builder.symbols,
+            libcall_names: builder.libcall_names,
             code_memory: Memory::new(),
             readonly_memory: Memory::new(),
             writable_memory: Memory::new(),
@@ -308,19 +354,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
             let ptr = func.code;
             debug_assert!((offset as usize) < func.size);
             let at = unsafe { ptr.offset(offset as isize) };
-            let base = if namespace.is_function(name) {
-                let (def, name_str, _signature) = namespace.get_function_definition(&name);
-                match def {
-                    Some(compiled) => compiled.code,
-                    None => self.lookup_symbol(name_str),
-                }
-            } else {
-                let (def, name_str, _writable) = namespace.get_data_definition(&name);
-                match def {
-                    Some(compiled) => compiled.storage,
-                    None => self.lookup_symbol(name_str),
-                }
-            };
+            let base = self.get_definition(namespace, name);
             // TODO: Handle overflow.
             let what = unsafe { base.offset(addend as isize) };
             match reloc {
@@ -373,19 +407,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
             let ptr = data.storage;
             debug_assert!((offset as usize) < data.size);
             let at = unsafe { ptr.offset(offset as isize) };
-            let base = if namespace.is_function(name) {
-                let (def, name_str, _signature) = namespace.get_function_definition(&name);
-                match def {
-                    Some(compiled) => compiled.code,
-                    None => self.lookup_symbol(name_str),
-                }
-            } else {
-                let (def, name_str, _writable) = namespace.get_data_definition(&name);
-                match def {
-                    Some(compiled) => compiled.storage,
-                    None => self.lookup_symbol(name_str),
-                }
-            };
+            let base = self.get_definition(namespace, name);
             // TODO: Handle overflow.
             let what = unsafe { base.offset(addend as isize) };
             match reloc {
diff --git a/cranelift/simplejit/tests/basic.rs b/cranelift/simplejit/tests/basic.rs
index 38da80c9a2..a3932b1d6a 100644
--- a/cranelift/simplejit/tests/basic.rs
+++ b/cranelift/simplejit/tests/basic.rs
@@ -1,6 +1,6 @@
 use cranelift_codegen::ir::*;
 use cranelift_codegen::isa::CallConv;
-use cranelift_codegen::Context;
+use cranelift_codegen::{ir::types::I16, Context};
 use cranelift_entity::EntityRef;
 use cranelift_frontend::*;
 use cranelift_module::*;
@@ -8,7 +8,8 @@ use cranelift_simplejit::*;
 
 #[test]
 fn error_on_incompatible_sig_in_declare_function() {
-    let mut module: Module = Module::new(SimpleJITBuilder::new());
+    let mut module: Module =
+        Module::new(SimpleJITBuilder::new(default_libcall_names()));
     let mut sig = Signature {
         params: vec![AbiParam::new(types::I64)],
         returns: vec![],
@@ -52,7 +53,8 @@ fn define_simple_function(module: &mut Module) -> FuncId {
 
 #[test]
 fn double_finalize() {
-    let mut module: Module = Module::new(SimpleJITBuilder::new());
+    let mut module: Module =
+        Module::new(SimpleJITBuilder::new(default_libcall_names()));
 
     define_simple_function(&mut module);
     module.finalize_definitions();
@@ -65,7 +67,8 @@ fn double_finalize() {
 #[test]
 #[should_panic(expected = "Result::unwrap()` on an `Err` value: DuplicateDefinition(\"abc\")")]
 fn panic_on_define_after_finalize() {
-    let mut module: Module = Module::new(SimpleJITBuilder::new());
+    let mut module: Module =
+        Module::new(SimpleJITBuilder::new(default_libcall_names()));
 
     define_simple_function(&mut module);
     module.finalize_definitions();
@@ -144,3 +147,51 @@ fn switch_error() {
         }
     }
 }
+
+#[test]
+fn libcall_function() {
+    let mut module: Module =
+        Module::new(SimpleJITBuilder::new(default_libcall_names()));
+
+    let sig = Signature {
+        params: vec![],
+        returns: vec![],
+        call_conv: CallConv::SystemV,
+    };
+
+    let func_id = module
+        .declare_function("function", Linkage::Local, &sig)
+        .unwrap();
+
+    let mut ctx = Context::new();
+    ctx.func = Function::with_name_signature(ExternalName::user(0, func_id.as_u32()), sig);
+    let mut func_ctx = FunctionBuilderContext::new();
+    {
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
+        let ebb = bcx.create_ebb();
+        bcx.switch_to_block(ebb);
+
+        let int = module.target_config().pointer_type();
+        let zero = bcx.ins().iconst(I16, 0);
+        let size = bcx.ins().iconst(int, 10);
+
+        let mut signature = module.make_signature();
+        signature.params.push(AbiParam::new(int));
+        signature.returns.push(AbiParam::new(int));
+        let callee = module
+            .declare_function("malloc", Linkage::Import, &signature)
+            .expect("declare malloc function");
+        let local_callee = module.declare_func_in_func(callee, &mut bcx.func);
+        let argument_exprs = vec![size];
+        let call = bcx.ins().call(local_callee, &argument_exprs);
+        let buffer = bcx.inst_results(call)[0];
+
+        bcx.call_memset(module.target_config(), buffer, zero, size);
+
+        bcx.ins().return_(&[]);
+    }
+
+    module.define_function(func_id, &mut ctx).unwrap();
+
+    module.finalize_definitions();
+}

From efdb7d86b3ceec27cd6dd4c19e76a9241a821422 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 30 Apr 2019 11:46:54 +0200
Subject: [PATCH 2414/3084] Fixes #607: Use a wrapping multiply when computing
 div/rem magic numbers; (#757)

---
 .../codegen/src/divconst_magic_numbers.rs     | 94 +++++++++++++++----
 1 file changed, 75 insertions(+), 19 deletions(-)

diff --git a/cranelift/codegen/src/divconst_magic_numbers.rs b/cranelift/codegen/src/divconst_magic_numbers.rs
index 30d8287747..0e90fb967f 100644
--- a/cranelift/codegen/src/divconst_magic_numbers.rs
+++ b/cranelift/codegen/src/divconst_magic_numbers.rs
@@ -55,7 +55,7 @@ pub fn magic_u32(d: u32) -> MU32 {
             q1 = u32::wrapping_add(u32::wrapping_mul(2, q1), 1);
             r1 = u32::wrapping_sub(u32::wrapping_mul(2, r1), nc);
         } else {
-            q1 = 2 * q1;
+            q1 = u32::wrapping_mul(2, q1);
             r1 = 2 * r1;
         }
         if r2 + 1 >= d - r2 {
@@ -101,7 +101,7 @@ pub fn magic_u64(d: u64) -> MU64 {
             q1 = u64::wrapping_add(u64::wrapping_mul(2, q1), 1);
             r1 = u64::wrapping_sub(u64::wrapping_mul(2, r1), nc);
         } else {
-            q1 = 2 * q1;
+            q1 = u64::wrapping_mul(2, q1);
             r1 = 2 * r1;
         }
         if r2 + 1 >= d - r2 {
@@ -522,73 +522,129 @@ mod tests {
     #[test]
     fn test_magic_generators_dont_panic() {
         // The point of this is to check that the magic number generators
-        // don't panic with integer wraparounds, especially at boundary
-        // cases for their arguments. The actual results are thrown away.
-        let mut total: u64 = 0;
+        // don't panic with integer wraparounds, especially at boundary cases
+        // for their arguments. The actual results are thrown away, although
+        // we force `total` to be used, so that rustc can't optimise the
+        // entire computation away.
+
         // Testing UP magic_u32
+        let mut total: u64 = 0;
         for x in 2..(200 * 1000u32) {
             let m = magic_u32(x);
             total = total ^ (m.mul_by as u64);
             total = total + (m.shift_by as u64);
-            total = total - (if m.do_add { 123 } else { 456 });
+            total = total + (if m.do_add { 123 } else { 456 });
         }
-        assert_eq!(total, 1747815691);
+        assert_eq!(total, 2481999609);
+
+        total = 0;
+        // Testing MIDPOINT magic_u32
+        for x in 0x8000_0000u32 - 10 * 1000u32..0x8000_0000u32 + 10 * 1000u32 {
+            let m = magic_u32(x);
+            total = total ^ (m.mul_by as u64);
+            total = total + (m.shift_by as u64);
+            total = total + (if m.do_add { 123 } else { 456 });
+        }
+        assert_eq!(total, 2399809723);
+
+        total = 0;
         // Testing DOWN magic_u32
         for x in 0..(200 * 1000u32) {
             let m = magic_u32(0xFFFF_FFFFu32 - x);
             total = total ^ (m.mul_by as u64);
             total = total + (m.shift_by as u64);
-            total = total - (if m.do_add { 123 } else { 456 });
+            total = total + (if m.do_add { 123 } else { 456 });
         }
-        assert_eq!(total, 2210292772);
+        assert_eq!(total, 271138267);
 
         // Testing UP magic_u64
+        total = 0;
         for x in 2..(200 * 1000u64) {
             let m = magic_u64(x);
             total = total ^ m.mul_by;
             total = total + (m.shift_by as u64);
-            total = total - (if m.do_add { 123 } else { 456 });
+            total = total + (if m.do_add { 123 } else { 456 });
         }
-        assert_eq!(total, 7430004084791260605);
+        assert_eq!(total, 7430004086976261161);
+
+        total = 0;
+        // Testing MIDPOINT magic_u64
+        for x in 0x8000_0000_0000_0000u64 - 10 * 1000u64..0x8000_0000_0000_0000u64 + 10 * 1000u64 {
+            let m = magic_u64(x);
+            total = total ^ m.mul_by;
+            total = total + (m.shift_by as u64);
+            total = total + (if m.do_add { 123 } else { 456 });
+        }
+        assert_eq!(total, 10312117246769520603);
+
         // Testing DOWN magic_u64
+        total = 0;
         for x in 0..(200 * 1000u64) {
             let m = magic_u64(0xFFFF_FFFF_FFFF_FFFFu64 - x);
             total = total ^ m.mul_by;
             total = total + (m.shift_by as u64);
-            total = total - (if m.do_add { 123 } else { 456 });
+            total = total + (if m.do_add { 123 } else { 456 });
         }
-        assert_eq!(total, 7547519887519825919);
+        assert_eq!(total, 1126603594357269734);
 
         // Testing UP magic_s32
+        total = 0;
         for x in 0..(200 * 1000i32) {
             let m = magic_s32(-0x8000_0000i32 + x);
             total = total ^ (m.mul_by as u64);
             total = total + (m.shift_by as u64);
         }
-        assert_eq!(total, 10899224186731671235);
+        assert_eq!(total, 18446744069953376812);
+
+        total = 0;
+        // Testing MIDPOINT magic_s32
+        for x in 0..(200 * 1000i32) {
+            let x2 = -100 * 1000i32 + x;
+            if x2 != -1 && x2 != 0 && x2 != 1 {
+                let m = magic_s32(x2);
+                total = total ^ (m.mul_by as u64);
+                total = total + (m.shift_by as u64);
+            }
+        }
+        assert_eq!(total, 351839350);
+
         // Testing DOWN magic_s32
+        total = 0;
         for x in 0..(200 * 1000i32) {
             let m = magic_s32(0x7FFF_FFFFi32 - x);
             total = total ^ (m.mul_by as u64);
             total = total + (m.shift_by as u64);
         }
-        assert_eq!(total, 7547519887517897369);
+        assert_eq!(total, 18446744072916880714);
 
         // Testing UP magic_s64
+        total = 0;
         for x in 0..(200 * 1000i64) {
             let m = magic_s64(-0x8000_0000_0000_0000i64 + x);
             total = total ^ (m.mul_by as u64);
             total = total + (m.shift_by as u64);
         }
-        assert_eq!(total, 8029756891368555163);
+        assert_eq!(total, 17929885647724831014);
+
+        total = 0;
+        // Testing MIDPOINT magic_s64
+        for x in 0..(200 * 1000i64) {
+            let x2 = -100 * 1000i64 + x;
+            if x2 != -1 && x2 != 0 && x2 != 1 {
+                let m = magic_s64(x2);
+                total = total ^ (m.mul_by as u64);
+                total = total + (m.shift_by as u64);
+            }
+        }
+        assert_eq!(total, 18106042338125661964);
+
         // Testing DOWN magic_s64
+        total = 0;
         for x in 0..(200 * 1000i64) {
             let m = magic_s64(0x7FFF_FFFF_FFFF_FFFFi64 - x);
             total = total ^ (m.mul_by as u64);
             total = total + (m.shift_by as u64);
         }
-        // Force `total` -- and hence, the entire computation -- to
-        // be used, so that rustc can't optimise it out.
-        assert_eq!(total, 7547519887532559585u64);
+        assert_eq!(total, 563301797155560970);
     }
 }

From 02e114cf3dcaf158152dca236528277d749d1fdf Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 29 Apr 2019 11:27:13 +0200
Subject: [PATCH 2415/3084] [wasm] Make FuncEnvironment functions fallible
 (fixes #752);

---
 cranelift/wasm/src/code_translator.rs | 62 ++++++++++++++-------------
 cranelift/wasm/src/environ/dummy.rs   | 48 +++++++++++++--------
 cranelift/wasm/src/environ/spec.rs    | 39 +++++++++++------
 cranelift/wasm/src/lib.rs             | 11 ++++-
 cranelift/wasm/src/state.rs           | 62 +++++++++++++++------------
 5 files changed, 131 insertions(+), 91 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index f83b462cef..6c339f3e1f 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -71,7 +71,7 @@ pub fn translate_operator(
          *  `get_global` and `set_global` are handled by the environment.
          ***********************************************************************************/
         Operator::GetGlobal { global_index } => {
-            let val = match state.get_global(builder.func, global_index, environ) {
+            let val = match state.get_global(builder.func, global_index, environ)? {
                 GlobalVariable::Const(val) => val,
                 GlobalVariable::Memory { gv, offset, ty } => {
                     let addr = builder.ins().global_value(environ.pointer_type(), gv);
@@ -82,7 +82,7 @@ pub fn translate_operator(
             state.push1(val);
         }
         Operator::SetGlobal { global_index } => {
-            match state.get_global(builder.func, global_index, environ) {
+            match state.get_global(builder.func, global_index, environ)? {
                 GlobalVariable::Const(_) => panic!("global #{} is a constant", global_index),
                 GlobalVariable::Memory { gv, offset, ty } => {
                     let addr = builder.ins().global_value(environ.pointer_type(), gv);
@@ -137,7 +137,7 @@ pub fn translate_operator(
             builder.ins().jump(loop_body, &[]);
             state.push_loop(loop_body, next, num_return_values(ty));
             builder.switch_to_block(loop_body);
-            environ.translate_loop_header(builder.cursor());
+            environ.translate_loop_header(builder.cursor())?;
         }
         Operator::If { ty } => {
             let val = state.pop1();
@@ -348,7 +348,7 @@ pub fn translate_operator(
          * argument referring to an index in the external functions table of the module.
          ************************************************************************************/
         Operator::Call { function_index } => {
-            let (fref, num_args) = state.get_direct_func(builder.func, function_index, environ);
+            let (fref, num_args) = state.get_direct_func(builder.func, function_index, environ)?;
             let call = environ.translate_call(
                 builder.cursor(),
                 FuncIndex::from_u32(function_index),
@@ -369,8 +369,8 @@ pub fn translate_operator(
         Operator::CallIndirect { index, table_index } => {
             // `index` is the index of the function's signature and `table_index` is the index of
             // the table to search the function in.
-            let (sigref, num_args) = state.get_indirect_sig(builder.func, index, environ);
-            let table = state.get_table(builder.func, table_index, environ);
+            let (sigref, num_args) = state.get_indirect_sig(builder.func, index, environ)?;
+            let table = state.get_table(builder.func, table_index, environ)?;
             let callee = state.pop1();
             let call = environ.translate_call_indirect(
                 builder.cursor(),
@@ -398,13 +398,13 @@ pub fn translate_operator(
             // The WebAssembly MVP only supports one linear memory, but we expect the reserved
             // argument to be a memory index.
             let heap_index = MemoryIndex::from_u32(reserved);
-            let heap = state.get_heap(builder.func, reserved, environ);
+            let heap = state.get_heap(builder.func, reserved, environ)?;
             let val = state.pop1();
             state.push1(environ.translate_memory_grow(builder.cursor(), heap_index, heap, val)?)
         }
         Operator::MemorySize { reserved } => {
             let heap_index = MemoryIndex::from_u32(reserved);
-            let heap = state.get_heap(builder.func, reserved, environ);
+            let heap = state.get_heap(builder.func, reserved, environ)?;
             state.push1(environ.translate_memory_size(builder.cursor(), heap_index, heap)?);
         }
         /******************************* Load instructions ***********************************
@@ -414,72 +414,72 @@ pub fn translate_operator(
         Operator::I32Load8U {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Uload8, I32, builder, state, environ);
+            translate_load(offset, ir::Opcode::Uload8, I32, builder, state, environ)?;
         }
         Operator::I32Load16U {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Uload16, I32, builder, state, environ);
+            translate_load(offset, ir::Opcode::Uload16, I32, builder, state, environ)?;
         }
         Operator::I32Load8S {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Sload8, I32, builder, state, environ);
+            translate_load(offset, ir::Opcode::Sload8, I32, builder, state, environ)?;
         }
         Operator::I32Load16S {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Sload16, I32, builder, state, environ);
+            translate_load(offset, ir::Opcode::Sload16, I32, builder, state, environ)?;
         }
         Operator::I64Load8U {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Uload8, I64, builder, state, environ);
+            translate_load(offset, ir::Opcode::Uload8, I64, builder, state, environ)?;
         }
         Operator::I64Load16U {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Uload16, I64, builder, state, environ);
+            translate_load(offset, ir::Opcode::Uload16, I64, builder, state, environ)?;
         }
         Operator::I64Load8S {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Sload8, I64, builder, state, environ);
+            translate_load(offset, ir::Opcode::Sload8, I64, builder, state, environ)?;
         }
         Operator::I64Load16S {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Sload16, I64, builder, state, environ);
+            translate_load(offset, ir::Opcode::Sload16, I64, builder, state, environ)?;
         }
         Operator::I64Load32S {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Sload32, I64, builder, state, environ);
+            translate_load(offset, ir::Opcode::Sload32, I64, builder, state, environ)?;
         }
         Operator::I64Load32U {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Uload32, I64, builder, state, environ);
+            translate_load(offset, ir::Opcode::Uload32, I64, builder, state, environ)?;
         }
         Operator::I32Load {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Load, I32, builder, state, environ);
+            translate_load(offset, ir::Opcode::Load, I32, builder, state, environ)?;
         }
         Operator::F32Load {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Load, F32, builder, state, environ);
+            translate_load(offset, ir::Opcode::Load, F32, builder, state, environ)?;
         }
         Operator::I64Load {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Load, I64, builder, state, environ);
+            translate_load(offset, ir::Opcode::Load, I64, builder, state, environ)?;
         }
         Operator::F64Load {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Load, F64, builder, state, environ);
+            translate_load(offset, ir::Opcode::Load, F64, builder, state, environ)?;
         }
         /****************************** Store instructions ***********************************
          * Wasm specifies an integer alignment flag but we drop it in Cranelift.
@@ -497,7 +497,7 @@ pub fn translate_operator(
         | Operator::F64Store {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_store(offset, ir::Opcode::Store, builder, state, environ);
+            translate_store(offset, ir::Opcode::Store, builder, state, environ)?;
         }
         Operator::I32Store8 {
             memarg: MemoryImmediate { flags: _, offset },
@@ -505,7 +505,7 @@ pub fn translate_operator(
         | Operator::I64Store8 {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_store(offset, ir::Opcode::Istore8, builder, state, environ);
+            translate_store(offset, ir::Opcode::Istore8, builder, state, environ)?;
         }
         Operator::I32Store16 {
             memarg: MemoryImmediate { flags: _, offset },
@@ -513,12 +513,12 @@ pub fn translate_operator(
         | Operator::I64Store16 {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_store(offset, ir::Opcode::Istore16, builder, state, environ);
+            translate_store(offset, ir::Opcode::Istore16, builder, state, environ)?;
         }
         Operator::I64Store32 {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_store(offset, ir::Opcode::Istore32, builder, state, environ);
+            translate_store(offset, ir::Opcode::Istore32, builder, state, environ)?;
         }
         /****************************** Nullary Operators ************************************/
         Operator::I32Const { value } => state.push1(builder.ins().iconst(I32, i64::from(value))),
@@ -1175,10 +1175,10 @@ fn translate_load(
     builder: &mut FunctionBuilder,
     state: &mut TranslationState,
     environ: &mut FE,
-) {
+) -> WasmResult<()> {
     let addr32 = state.pop1();
     // We don't yet support multiple linear memories.
-    let heap = state.get_heap(builder.func, 0, environ);
+    let heap = state.get_heap(builder.func, 0, environ)?;
     let (base, offset) = get_heap_addr(heap, addr32, offset, environ.pointer_type(), builder);
     // Note that we don't set `is_aligned` here, even if the load instruction's
     // alignment immediate says it's aligned, because WebAssembly's immediate
@@ -1188,6 +1188,7 @@ fn translate_load(
         .ins()
         .Load(opcode, result_ty, flags, offset.into(), base);
     state.push1(dfg.first_result(load));
+    Ok(())
 }
 
 /// Translate a store instruction.
@@ -1197,18 +1198,19 @@ fn translate_store(
     builder: &mut FunctionBuilder,
     state: &mut TranslationState,
     environ: &mut FE,
-) {
+) -> WasmResult<()> {
     let (addr32, val) = state.pop2();
     let val_ty = builder.func.dfg.value_type(val);
 
     // We don't yet support multiple linear memories.
-    let heap = state.get_heap(builder.func, 0, environ);
+    let heap = state.get_heap(builder.func, 0, environ)?;
     let (base, offset) = get_heap_addr(heap, addr32, offset, environ.pointer_type(), builder);
     // See the comments in `translate_load` about the flags.
     let flags = MemFlags::new();
     builder
         .ins()
         .Store(opcode, val_ty, flags, offset.into(), val, base);
+    Ok(())
 }
 
 fn translate_icmp(cc: IntCC, builder: &mut FunctionBuilder, state: &mut TranslationState) {
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index 70b592bbda..e32a3260ab 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -182,18 +182,26 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         self.mod_info.config
     }
 
-    fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalVariable {
+    fn return_mode(&self) -> ReturnMode {
+        self.return_mode
+    }
+
+    fn make_global(
+        &mut self,
+        func: &mut ir::Function,
+        index: GlobalIndex,
+    ) -> WasmResult {
         // Just create a dummy `vmctx` global.
         let offset = cast::i32((index.index() * 8) + 8).unwrap().into();
         let vmctx = func.create_global_value(ir::GlobalValueData::VMContext {});
-        GlobalVariable::Memory {
+        Ok(GlobalVariable::Memory {
             gv: vmctx,
             offset,
             ty: self.mod_info.globals[index].entity.ty,
-        }
+        })
     }
 
-    fn make_heap(&mut self, func: &mut ir::Function, _index: MemoryIndex) -> ir::Heap {
+    fn make_heap(&mut self, func: &mut ir::Function, _index: MemoryIndex) -> WasmResult {
         // Create a static heap whose base address is stored at `vmctx+0`.
         let addr = func.create_global_value(ir::GlobalValueData::VMContext);
         let gv = func.create_global_value(ir::GlobalValueData::Load {
@@ -203,7 +211,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
             readonly: true,
         });
 
-        func.create_heap(ir::HeapData {
+        Ok(func.create_heap(ir::HeapData {
             base: gv,
             min_size: 0.into(),
             offset_guard_size: 0x8000_0000.into(),
@@ -211,10 +219,10 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
                 bound: 0x1_0000_0000.into(),
             },
             index_type: I32,
-        })
+        }))
     }
 
-    fn make_table(&mut self, func: &mut ir::Function, _index: TableIndex) -> ir::Table {
+    fn make_table(&mut self, func: &mut ir::Function, _index: TableIndex) -> WasmResult {
         // Create a table whose base address is stored at `vmctx+0`.
         let vmctx = func.create_global_value(ir::GlobalValueData::VMContext);
         let base_gv = func.create_global_value(ir::GlobalValueData::Load {
@@ -230,32 +238,40 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
             readonly: true,
         });
 
-        func.create_table(ir::TableData {
+        Ok(func.create_table(ir::TableData {
             base_gv,
             min_size: Uimm64::new(0),
             bound_gv,
             element_size: Uimm64::from(u64::from(self.pointer_bytes()) * 2),
             index_type: I32,
-        })
+        }))
     }
 
-    fn make_indirect_sig(&mut self, func: &mut ir::Function, index: SignatureIndex) -> ir::SigRef {
+    fn make_indirect_sig(
+        &mut self,
+        func: &mut ir::Function,
+        index: SignatureIndex,
+    ) -> WasmResult {
         // A real implementation would probably change the calling convention and add `vmctx` and
         // signature index arguments.
-        func.import_signature(self.vmctx_sig(index))
+        Ok(func.import_signature(self.vmctx_sig(index)))
     }
 
-    fn make_direct_func(&mut self, func: &mut ir::Function, index: FuncIndex) -> ir::FuncRef {
+    fn make_direct_func(
+        &mut self,
+        func: &mut ir::Function,
+        index: FuncIndex,
+    ) -> WasmResult {
         let sigidx = self.mod_info.functions[index].entity;
         // A real implementation would probably add a `vmctx` argument.
         // And maybe attempt some signature de-duplication.
         let signature = func.import_signature(self.vmctx_sig(sigidx));
         let name = get_func_name(index);
-        func.import_function(ir::ExtFuncData {
+        Ok(func.import_function(ir::ExtFuncData {
             name,
             signature,
             colocated: false,
-        })
+        }))
     }
 
     fn translate_call_indirect(
@@ -340,10 +356,6 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
     ) -> WasmResult {
         Ok(pos.ins().iconst(I32, -1))
     }
-
-    fn return_mode(&self) -> ReturnMode {
-        self.return_mode
-    }
 }
 
 impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 9b96d42edf..3173a07886 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -110,6 +110,13 @@ pub trait FuncEnvironment {
         self.target_config().pointer_bytes()
     }
 
+    /// Should the code be structured to use a single `fallthrough_return` instruction at the end
+    /// of the function body, rather than `return` instructions as needed? This is used by VMs
+    /// to append custom epilogues.
+    fn return_mode(&self) -> ReturnMode {
+        ReturnMode::NormalReturns
+    }
+
     /// Set up the necessary preamble definitions in `func` to access the global variable
     /// identified by `index`.
     ///
@@ -117,19 +124,23 @@ pub trait FuncEnvironment {
     ///
     /// Return the global variable reference that should be used to access the global and the
     /// WebAssembly type of the global.
-    fn make_global(&mut self, func: &mut ir::Function, index: GlobalIndex) -> GlobalVariable;
+    fn make_global(
+        &mut self,
+        func: &mut ir::Function,
+        index: GlobalIndex,
+    ) -> WasmResult;
 
     /// Set up the necessary preamble definitions in `func` to access the linear memory identified
     /// by `index`.
     ///
     /// The index space covers both imported and locally declared memories.
-    fn make_heap(&mut self, func: &mut ir::Function, index: MemoryIndex) -> ir::Heap;
+    fn make_heap(&mut self, func: &mut ir::Function, index: MemoryIndex) -> WasmResult;
 
     /// Set up the necessary preamble definitions in `func` to access the table identified
     /// by `index`.
     ///
     /// The index space covers both imported and locally declared tables.
-    fn make_table(&mut self, func: &mut ir::Function, index: TableIndex) -> ir::Table;
+    fn make_table(&mut self, func: &mut ir::Function, index: TableIndex) -> WasmResult;
 
     /// Set up a signature definition in the preamble of `func` that can be used for an indirect
     /// call with signature `index`.
@@ -140,7 +151,11 @@ pub trait FuncEnvironment {
     ///
     /// The signature will only be used for indirect calls, even if the module has direct function
     /// calls with the same WebAssembly type.
-    fn make_indirect_sig(&mut self, func: &mut ir::Function, index: SignatureIndex) -> ir::SigRef;
+    fn make_indirect_sig(
+        &mut self,
+        func: &mut ir::Function,
+        index: SignatureIndex,
+    ) -> WasmResult;
 
     /// Set up an external function definition in the preamble of `func` that can be used to
     /// directly call the function `index`.
@@ -153,7 +168,11 @@ pub trait FuncEnvironment {
     ///
     /// The function's signature will only be used for direct calls, even if the module has
     /// indirect calls with the same WebAssembly type.
-    fn make_direct_func(&mut self, func: &mut ir::Function, index: FuncIndex) -> ir::FuncRef;
+    fn make_direct_func(
+        &mut self,
+        func: &mut ir::Function,
+        index: FuncIndex,
+    ) -> WasmResult;
 
     /// Translate a `call_indirect` WebAssembly instruction at `pos`.
     ///
@@ -226,15 +245,9 @@ pub trait FuncEnvironment {
     ///
     /// This can be used to insert explicit interrupt or safepoint checking at
     /// the beginnings of loops.
-    fn translate_loop_header(&mut self, _pos: FuncCursor) {
+    fn translate_loop_header(&mut self, _pos: FuncCursor) -> WasmResult<()> {
         // By default, don't emit anything.
-    }
-
-    /// Should the code be structured to use a single `fallthrough_return` instruction at the end
-    /// of the function body, rather than `return` instructions as needed? This is used by VMs
-    /// to append custom epilogues.
-    fn return_mode(&self) -> ReturnMode {
-        ReturnMode::NormalReturns
+        Ok(())
     }
 }
 
diff --git a/cranelift/wasm/src/lib.rs b/cranelift/wasm/src/lib.rs
index edd5c3215f..c475b342bc 100644
--- a/cranelift/wasm/src/lib.rs
+++ b/cranelift/wasm/src/lib.rs
@@ -38,9 +38,16 @@ extern crate alloc as std;
 extern crate std;
 
 #[cfg(not(feature = "std"))]
-use hashmap_core::{map as hash_map, HashMap};
+use hashmap_core::{
+    hash_map::Entry::{Occupied, Vacant},
+    map as hash_map, HashMap,
+};
 #[cfg(feature = "std")]
-use std::collections::{hash_map, HashMap};
+use std::collections::{
+    hash_map,
+    hash_map::Entry::{Occupied, Vacant},
+    HashMap,
+};
 
 mod code_translator;
 mod environ;
diff --git a/cranelift/wasm/src/state.rs b/cranelift/wasm/src/state.rs
index f392631237..adfa903748 100644
--- a/cranelift/wasm/src/state.rs
+++ b/cranelift/wasm/src/state.rs
@@ -3,8 +3,8 @@
 //! The `TranslationState` struct defined in this module is used to keep track of the WebAssembly
 //! value and control stacks during the translation of a single function.
 
-use super::HashMap;
-use crate::environ::{FuncEnvironment, GlobalVariable};
+use super::{HashMap, Occupied, Vacant};
+use crate::environ::{FuncEnvironment, GlobalVariable, WasmResult};
 use crate::translation_utils::{FuncIndex, GlobalIndex, MemoryIndex, SignatureIndex, TableIndex};
 use cranelift_codegen::ir::{self, Ebb, Inst, Value};
 use std::vec::Vec;
@@ -285,12 +285,12 @@ impl TranslationState {
         func: &mut ir::Function,
         index: u32,
         environ: &mut FE,
-    ) -> GlobalVariable {
+    ) -> WasmResult {
         let index = GlobalIndex::from_u32(index);
-        *self
-            .globals
-            .entry(index)
-            .or_insert_with(|| environ.make_global(func, index))
+        match self.globals.entry(index) {
+            Occupied(entry) => Ok(*entry.get()),
+            Vacant(entry) => Ok(*entry.insert(environ.make_global(func, index)?)),
+        }
     }
 
     /// Get the `Heap` reference that should be used to access linear memory `index`.
@@ -300,12 +300,12 @@ impl TranslationState {
         func: &mut ir::Function,
         index: u32,
         environ: &mut FE,
-    ) -> ir::Heap {
+    ) -> WasmResult {
         let index = MemoryIndex::from_u32(index);
-        *self
-            .heaps
-            .entry(index)
-            .or_insert_with(|| environ.make_heap(func, index))
+        match self.heaps.entry(index) {
+            Occupied(entry) => Ok(*entry.get()),
+            Vacant(entry) => Ok(*entry.insert(environ.make_heap(func, index)?)),
+        }
     }
 
     /// Get the `Table` reference that should be used to access table `index`.
@@ -315,12 +315,12 @@ impl TranslationState {
         func: &mut ir::Function,
         index: u32,
         environ: &mut FE,
-    ) -> ir::Table {
+    ) -> WasmResult {
         let index = TableIndex::from_u32(index);
-        *self
-            .tables
-            .entry(index)
-            .or_insert_with(|| environ.make_table(func, index))
+        match self.tables.entry(index) {
+            Occupied(entry) => Ok(*entry.get()),
+            Vacant(entry) => Ok(*entry.insert(environ.make_table(func, index)?)),
+        }
     }
 
     /// Get the `SigRef` reference that should be used to make an indirect call with signature
@@ -332,12 +332,15 @@ impl TranslationState {
         func: &mut ir::Function,
         index: u32,
         environ: &mut FE,
-    ) -> (ir::SigRef, usize) {
+    ) -> WasmResult<(ir::SigRef, usize)> {
         let index = SignatureIndex::from_u32(index);
-        *self.signatures.entry(index).or_insert_with(|| {
-            let sig = environ.make_indirect_sig(func, index);
-            (sig, normal_args(&func.dfg.signatures[sig]))
-        })
+        match self.signatures.entry(index) {
+            Occupied(entry) => Ok(*entry.get()),
+            Vacant(entry) => {
+                let sig = environ.make_indirect_sig(func, index)?;
+                Ok(*entry.insert((sig, normal_args(&func.dfg.signatures[sig]))))
+            }
+        }
     }
 
     /// Get the `FuncRef` reference that should be used to make a direct call to function
@@ -349,13 +352,16 @@ impl TranslationState {
         func: &mut ir::Function,
         index: u32,
         environ: &mut FE,
-    ) -> (ir::FuncRef, usize) {
+    ) -> WasmResult<(ir::FuncRef, usize)> {
         let index = FuncIndex::from_u32(index);
-        *self.functions.entry(index).or_insert_with(|| {
-            let fref = environ.make_direct_func(func, index);
-            let sig = func.dfg.ext_funcs[fref].signature;
-            (fref, normal_args(&func.dfg.signatures[sig]))
-        })
+        match self.functions.entry(index) {
+            Occupied(entry) => Ok(*entry.get()),
+            Vacant(entry) => {
+                let fref = environ.make_direct_func(func, index)?;
+                let sig = func.dfg.ext_funcs[fref].signature;
+                Ok(*entry.insert((fref, normal_args(&func.dfg.signatures[sig]))))
+            }
+        }
     }
 }
 

From 3ce5d2057dcb3ace77e2585e16234488098e62e7 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 29 Apr 2019 11:34:45 +0200
Subject: [PATCH 2416/3084] [wasm] Add the ability to provide a user-defined
 error;

---
 cranelift/wasm/Cargo.toml          |  2 +-
 cranelift/wasm/src/environ/spec.rs | 14 +++++++++++++-
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 2309322acb..7d9504aa54 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -27,7 +27,7 @@ target-lexicon = "0.4.0"
 
 [features]
 default = ["std"]
-std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std"]
+std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std", "failure/std"]
 core = ["hashmap_core", "cranelift-codegen/core", "cranelift-frontend/core", "wasmparser/core"]
 
 [badges]
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 3173a07886..ac58fd7215 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -39,7 +39,7 @@ pub enum GlobalVariable {
 ///
 /// When a WebAssembly function can't be translated, one of these error codes will be returned
 /// to describe the failure.
-#[derive(Fail, Debug, PartialEq, Eq)]
+#[derive(Fail, Debug)]
 pub enum WasmError {
     /// The input WebAssembly code is invalid.
     ///
@@ -67,6 +67,18 @@ pub enum WasmError {
     /// [limits]: https://cranelift.readthedocs.io/en/latest/ir.html#implementation-limits
     #[fail(display = "Implementation limit exceeded")]
     ImplLimitExceeded,
+
+    /// Any user-defined error. Requires an std build, where failure::Error is defined.
+    #[cfg(feature = "std")]
+    #[fail(display = "User error: {}", _0)]
+    User(failure::Error),
+}
+
+#[cfg(feature = "std")]
+impl From for WasmError {
+    fn from(err: failure::Error) -> Self {
+        WasmError::User(err)
+    }
 }
 
 impl From for WasmError {

From 443e48aee19961b1de50b7c71f90c07b729eb452 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 29 Apr 2019 13:44:01 +0200
Subject: [PATCH 2417/3084] [wasm] Raise an error instead of panicking for
 unhandled function local types;

---
 cranelift/wasm/src/func_translator.rs | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index 6a003f06f8..efa63c15e9 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -5,7 +5,7 @@
 //! WebAssembly module and the runtime environment.
 
 use crate::code_translator::translate_operator;
-use crate::environ::{FuncEnvironment, ReturnMode, WasmResult};
+use crate::environ::{FuncEnvironment, ReturnMode, WasmError, WasmResult};
 use crate::state::TranslationState;
 use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir::{self, Ebb, InstBuilder};
@@ -147,7 +147,7 @@ fn parse_local_decls(
     for _ in 0..local_count {
         builder.set_srcloc(cur_srcloc(reader));
         let (count, ty) = reader.read_local_decl(&mut locals_total)?;
-        declare_locals(builder, count, ty, &mut next_local);
+        declare_locals(builder, count, ty, &mut next_local)?;
     }
 
     Ok(())
@@ -161,7 +161,7 @@ fn declare_locals(
     count: u32,
     wasm_type: wasmparser::Type,
     next_local: &mut usize,
-) {
+) -> WasmResult<()> {
     // All locals are initialized to 0.
     use wasmparser::Type::*;
     let zeroval = match wasm_type {
@@ -169,7 +169,7 @@ fn declare_locals(
         I64 => builder.ins().iconst(ir::types::I64, 0),
         F32 => builder.ins().f32const(ir::immediates::Ieee32::with_bits(0)),
         F64 => builder.ins().f64const(ir::immediates::Ieee64::with_bits(0)),
-        _ => panic!("invalid local type"),
+        _ => return Err(WasmError::Unsupported("unsupported local type")),
     };
 
     let ty = builder.func.dfg.value_type(zeroval);
@@ -179,6 +179,7 @@ fn declare_locals(
         builder.def_var(local, zeroval);
         *next_local += 1;
     }
+    Ok(())
 }
 
 /// Parse the function body in `reader`.

From cb6268118c982c4b8fb0bb3a6c275fb0737c3c8b Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 30 Mar 2019 16:45:58 +0100
Subject: [PATCH 2418/3084] Make it possible to define data alignment

---
 cranelift/faerie/src/backend.rs    | 14 ++++++++------
 cranelift/module/src/backend.rs    |  3 ++-
 cranelift/module/src/module.rs     |  8 ++++++--
 cranelift/simplejit/src/backend.rs | 15 +++++++++++----
 cranelift/simplejit/src/memory.rs  |  7 ++++++-
 5 files changed, 33 insertions(+), 14 deletions(-)

diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index 0adb564b30..372655f1df 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -127,9 +127,9 @@ impl Backend for FaerieBackend {
             .expect("inconsistent declarations");
     }
 
-    fn declare_data(&mut self, name: &str, linkage: Linkage, writable: bool) {
+    fn declare_data(&mut self, name: &str, linkage: Linkage, writable: bool, align: Option) {
         self.artifact
-            .declare(name, translate_data_linkage(linkage, writable))
+            .declare(name, translate_data_linkage(linkage, writable, align))
             .expect("inconsistent declarations");
     }
 
@@ -190,6 +190,7 @@ impl Backend for FaerieBackend {
         &mut self,
         name: &str,
         _writable: bool,
+        _align: Option,
         data_ctx: &DataContext,
         namespace: &ModuleNamespace,
     ) -> ModuleResult {
@@ -334,12 +335,13 @@ fn translate_function_linkage(linkage: Linkage) -> faerie::Decl {
     }
 }
 
-fn translate_data_linkage(linkage: Linkage, writable: bool) -> faerie::Decl {
+fn translate_data_linkage(linkage: Linkage, writable: bool, align: Option) -> faerie::Decl {
+    let align = align.map(|align| usize::from(align));
     match linkage {
         Linkage::Import => faerie::Decl::data_import().into(),
-        Linkage::Local => faerie::Decl::data().with_writable(writable).into(),
-        Linkage::Export => faerie::Decl::data().global().with_writable(writable).into(),
-        Linkage::Preemptible => faerie::Decl::data().weak().with_writable(writable).into(),
+        Linkage::Local => faerie::Decl::data().with_writable(writable).with_align(align).into(),
+        Linkage::Export => faerie::Decl::data().global().with_writable(writable).with_align(align).into(),
+        Linkage::Preemptible => faerie::Decl::data().weak().with_writable(writable).with_align(align).into(),
     }
 }
 
diff --git a/cranelift/module/src/backend.rs b/cranelift/module/src/backend.rs
index 3f89e18fde..fa23d24f01 100644
--- a/cranelift/module/src/backend.rs
+++ b/cranelift/module/src/backend.rs
@@ -59,7 +59,7 @@ where
     fn declare_function(&mut self, name: &str, linkage: Linkage);
 
     /// Declare a data object.
-    fn declare_data(&mut self, name: &str, linkage: Linkage, writable: bool);
+    fn declare_data(&mut self, name: &str, linkage: Linkage, writable: bool, align: Option);
 
     /// Define a function, producing the function body from the given `Context`.
     ///
@@ -79,6 +79,7 @@ where
         &mut self,
         name: &str,
         writable: bool,
+        align: Option,
         data_ctx: &DataContext,
         namespace: &ModuleNamespace,
     ) -> ModuleResult;
diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 5b11a48e15..9eac603a28 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -185,6 +185,7 @@ pub struct DataDeclaration {
     pub name: String,
     pub linkage: Linkage,
     pub writable: bool,
+    pub align: Option,
 }
 
 /// A data object belonging to a `Module`.
@@ -438,6 +439,7 @@ where
         name: &str,
         linkage: Linkage,
         writable: bool,
+        align: Option, // An alignment bigger than 128 is unlikely
     ) -> ModuleResult {
         // TODO: Can we avoid allocating names so often?
         use super::hash_map::Entry::*;
@@ -447,7 +449,7 @@ where
                     let existing = &mut self.contents.data_objects[id];
                     existing.merge(linkage, writable);
                     self.backend
-                        .declare_data(name, existing.decl.linkage, existing.decl.writable);
+                        .declare_data(name, existing.decl.linkage, existing.decl.writable, existing.decl.align);
                     Ok(id)
                 }
 
@@ -461,11 +463,12 @@ where
                         name: name.to_owned(),
                         linkage,
                         writable,
+                        align,
                     },
                     compiled: None,
                 });
                 entry.insert(FuncOrDataId::Data(id));
-                self.backend.declare_data(name, linkage, writable);
+                self.backend.declare_data(name, linkage, writable, align);
                 Ok(id)
             }
         }
@@ -563,6 +566,7 @@ where
             Some(self.backend.define_data(
                 &info.decl.name,
                 info.decl.writable,
+                info.decl.align,
                 data_ctx,
                 &ModuleNamespace:: {
                     contents: &self.contents,
diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index 7be3b3b2fa..1c6b0b1826 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -209,7 +209,13 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         // Nothing to do.
     }
 
-    fn declare_data(&mut self, _name: &str, _linkage: Linkage, _writable: bool) {
+    fn declare_data(
+        &mut self,
+        _name: &str,
+        _linkage: Linkage,
+        _writable: bool,
+        _align: Option,
+    ) {
         // Nothing to do.
     }
 
@@ -223,7 +229,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         let size = code_size as usize;
         let ptr = self
             .code_memory
-            .allocate(size)
+            .allocate(size, 0x10)
             .expect("TODO: handle OOM etc.");
 
         if cfg!(target_os = "linux") && ::std::env::var_os("PERF_BUILDID_DIR").is_some() {
@@ -253,6 +259,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         &mut self,
         _name: &str,
         writable: bool,
+        align: Option,
         data: &DataContext,
         _namespace: &ModuleNamespace,
     ) -> ModuleResult {
@@ -267,11 +274,11 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         let size = init.size();
         let storage = if writable {
             self.writable_memory
-                .allocate(size)
+                .allocate(size, align.unwrap_or(0x8))
                 .expect("TODO: handle OOM etc.")
         } else {
             self.readonly_memory
-                .allocate(size)
+                .allocate(size, align.unwrap_or(1))
                 .expect("TODO: handle OOM etc.")
         };
 
diff --git a/cranelift/simplejit/src/memory.rs b/cranelift/simplejit/src/memory.rs
index 9720e2ae17..6b04193310 100644
--- a/cranelift/simplejit/src/memory.rs
+++ b/cranelift/simplejit/src/memory.rs
@@ -97,7 +97,12 @@ impl Memory {
     }
 
     /// TODO: Use a proper error type.
-    pub fn allocate(&mut self, size: usize) -> Result<*mut u8, String> {
+    pub fn allocate(&mut self, size: usize, align: u8) -> Result<*mut u8, String> {
+        if self.position % align as usize != 0 {
+            self.position += align as usize - self.position % align as usize;
+            debug_assert!(self.position % align as usize == 0);
+        }
+
         if size <= self.current.len - self.position {
             // TODO: Ensure overflow is not possible.
             let ptr = unsafe { self.current.ptr.add(self.position) };

From 556d5d45e9cf193e5b1be3c2c07a108734820f11 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 30 Mar 2019 17:29:28 +0100
Subject: [PATCH 2419/3084] Rustfmt

---
 cranelift/faerie/src/backend.rs | 17 ++++++++++++++---
 cranelift/module/src/module.rs  |  8 ++++++--
 2 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index 372655f1df..047ddeb87d 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -339,9 +339,20 @@ fn translate_data_linkage(linkage: Linkage, writable: bool, align: Option) -
     let align = align.map(|align| usize::from(align));
     match linkage {
         Linkage::Import => faerie::Decl::data_import().into(),
-        Linkage::Local => faerie::Decl::data().with_writable(writable).with_align(align).into(),
-        Linkage::Export => faerie::Decl::data().global().with_writable(writable).with_align(align).into(),
-        Linkage::Preemptible => faerie::Decl::data().weak().with_writable(writable).with_align(align).into(),
+        Linkage::Local => faerie::Decl::data()
+            .with_writable(writable)
+            .with_align(align)
+            .into(),
+        Linkage::Export => faerie::Decl::data()
+            .global()
+            .with_writable(writable)
+            .with_align(align)
+            .into(),
+        Linkage::Preemptible => faerie::Decl::data()
+            .weak()
+            .with_writable(writable)
+            .with_align(align)
+            .into(),
     }
 }
 
diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 9eac603a28..7451e2fda4 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -448,8 +448,12 @@ where
                 FuncOrDataId::Data(id) => {
                     let existing = &mut self.contents.data_objects[id];
                     existing.merge(linkage, writable);
-                    self.backend
-                        .declare_data(name, existing.decl.linkage, existing.decl.writable, existing.decl.align);
+                    self.backend.declare_data(
+                        name,
+                        existing.decl.linkage,
+                        existing.decl.writable,
+                        existing.decl.align,
+                    );
                     Ok(id)
                 }
 

From 8216b83c654be4b3581de592fed57e05756298f6 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 24 Apr 2019 19:53:24 +0200
Subject: [PATCH 2420/3084] Make alignment magic number constants

---
 cranelift/simplejit/src/backend.rs | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index 1c6b0b1826..81b44459ec 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -17,6 +17,10 @@ use target_lexicon::PointerWidth;
 #[cfg(windows)]
 use winapi;
 
+const EXECUTABLE_DATA_ALIGNMENT: u8 = 0x10;
+const WRITABLE_DATA_ALIGNMENT: u8 = 0x8;
+const READONLY_DATA_ALIGNMENT: u8 = 0x1;
+
 /// A builder for `SimpleJITBackend`.
 pub struct SimpleJITBuilder {
     isa: Box,
@@ -229,7 +233,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         let size = code_size as usize;
         let ptr = self
             .code_memory
-            .allocate(size, 0x10)
+            .allocate(size, EXECUTABLE_DATA_ALIGNMENT)
             .expect("TODO: handle OOM etc.");
 
         if cfg!(target_os = "linux") && ::std::env::var_os("PERF_BUILDID_DIR").is_some() {
@@ -274,11 +278,11 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         let size = init.size();
         let storage = if writable {
             self.writable_memory
-                .allocate(size, align.unwrap_or(0x8))
+                .allocate(size, align.unwrap_or(WRITABLE_DATA_ALIGNMENT))
                 .expect("TODO: handle OOM etc.")
         } else {
             self.readonly_memory
-                .allocate(size, align.unwrap_or(1))
+                .allocate(size, align.unwrap_or(READONLY_DATA_ALIGNMENT))
                 .expect("TODO: handle OOM etc.")
         };
 

From abf0048972239a553a757ed74dd4eff8ac2e5efa Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 24 Apr 2019 19:58:16 +0200
Subject: [PATCH 2421/3084] Merge data alignment

---
 cranelift/module/src/module.rs | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 7451e2fda4..3da6c37810 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -203,9 +203,10 @@ impl ModuleData
 where
     B: Backend,
 {
-    fn merge(&mut self, linkage: Linkage, writable: bool) {
+    fn merge(&mut self, linkage: Linkage, writable: bool, align: Option) {
         self.decl.linkage = Linkage::merge(self.decl.linkage, linkage);
         self.decl.writable = self.decl.writable || writable;
+        self.decl.align = self.decl.align.max(align);
     }
 }
 
@@ -447,7 +448,7 @@ where
             Occupied(entry) => match *entry.get() {
                 FuncOrDataId::Data(id) => {
                     let existing = &mut self.contents.data_objects[id];
-                    existing.merge(linkage, writable);
+                    existing.merge(linkage, writable, align);
                     self.backend.declare_data(
                         name,
                         existing.decl.linkage,

From dca168f350567ad129a5ba3f6c2be3051791464c Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 25 Apr 2019 11:15:49 +0200
Subject: [PATCH 2422/3084] [meta] Switch to the Rust crate to generate
 legalizations and remove Python code;

---
 cranelift/codegen/meta-python/build.py        |   2 -
 .../codegen/meta-python/gen_legalizer.py      | 468 ------------------
 .../codegen/meta-python/test_gen_legalizer.py | 196 --------
 cranelift/codegen/meta/src/lib.rs             |   2 +-
 4 files changed, 1 insertion(+), 667 deletions(-)
 delete mode 100644 cranelift/codegen/meta-python/gen_legalizer.py
 delete mode 100644 cranelift/codegen/meta-python/test_gen_legalizer.py

diff --git a/cranelift/codegen/meta-python/build.py b/cranelift/codegen/meta-python/build.py
index 28f510c1e9..7f890a4bb1 100644
--- a/cranelift/codegen/meta-python/build.py
+++ b/cranelift/codegen/meta-python/build.py
@@ -8,7 +8,6 @@ import isa
 import gen_settings
 import gen_build_deps
 import gen_encoding
-import gen_legalizer
 import gen_binemit
 
 try:
@@ -46,7 +45,6 @@ def main():
 
     gen_settings.generate(isas, out_dir)
     gen_encoding.generate(isas, out_dir)
-    gen_legalizer.generate(isas, out_dir)
     gen_binemit.generate(isas, out_dir)
     gen_build_deps.generate()
 
diff --git a/cranelift/codegen/meta-python/gen_legalizer.py b/cranelift/codegen/meta-python/gen_legalizer.py
deleted file mode 100644
index b4dfd0089f..0000000000
--- a/cranelift/codegen/meta-python/gen_legalizer.py
+++ /dev/null
@@ -1,468 +0,0 @@
-"""
-Generate legalizer transformations.
-
-The transformations defined in the `cranelift.legalize` module are all of the
-macro-expansion form where the input pattern is a single instruction. We
-generate a Rust function for each `XFormGroup` which takes a `Cursor` pointing
-at the instruction to be legalized. The expanded destination pattern replaces
-the input instruction.
-"""
-from __future__ import absolute_import
-from srcgen import Formatter
-from collections import defaultdict
-from base import instructions
-from cdsl.ast import Var
-from cdsl.ti import ti_rtl, TypeEnv, get_type_env, TypesEqual,\
-    InTypeset, WiderOrEq
-from unique_table import UniqueTable
-from cdsl.typevar import TypeVar
-
-try:
-    from typing import Sequence, List, Dict, Set, DefaultDict # noqa
-    from cdsl.isa import TargetISA  # noqa
-    from cdsl.ast import Def, VarAtomMap  # noqa
-    from cdsl.xform import XForm, XFormGroup  # noqa
-    from cdsl.typevar import TypeSet # noqa
-    from cdsl.ti import TypeConstraint # noqa
-except ImportError:
-    pass
-
-
-# TypeSet indexes are encoded in 8 bits, with `0xff` reserved.
-typeset_limit = 0xff
-
-
-def gen_typesets_table(fmt, type_sets):
-    # type: (Formatter, UniqueTable) -> None
-    """
-    Generate the table of ValueTypeSets described by type_sets.
-    """
-    if len(type_sets.table) == 0:
-        return
-    fmt.comment('Table of value type sets.')
-    assert len(type_sets.table) <= typeset_limit, "Too many type sets"
-    with fmt.indented(
-            'const TYPE_SETS: [ir::instructions::ValueTypeSet; {}] = ['
-            .format(len(type_sets.table)), '];'):
-        for ts in type_sets.table:
-            with fmt.indented('ir::instructions::ValueTypeSet {', '},'):
-                ts.emit_fields(fmt)
-
-
-def get_runtime_typechecks(xform):
-    # type: (XForm) -> List[TypeConstraint]
-    """
-    Given a XForm build a list of runtime type checks necessary to determine
-    if it applies. We have 2 types of runtime checks:
-        1) typevar tv belongs to typeset T - needed for free tvs whose
-               typeset is constrained by their use in the dst pattern
-
-        2) tv1 == tv2 where tv1 and tv2 are derived TVs - caused by unification
-                of non-bijective functions
-    """
-    check_l = []  # type: List[TypeConstraint]
-
-    # 1) Perform ti only on the source RTL. Accumulate any free tvs that have a
-    #    different inferred type in src, compared to the type inferred for both
-    #    src and dst.
-    symtab = {}  # type: VarAtomMap
-    src_copy = xform.src.copy(symtab)
-    src_typenv = get_type_env(ti_rtl(src_copy, TypeEnv()))
-
-    for v in xform.ti.vars:
-        if not v.has_free_typevar():
-            continue
-
-        # In rust the local variable containing a free TV associated with var v
-        # has name typeof_v. We rely on the python TVs having the same name.
-        assert "typeof_{}".format(v) == xform.ti[v].name
-
-        if v not in symtab:
-            # We can have singleton vars defined only on dst. Ignore them
-            assert v.get_typevar().singleton_type() is not None
-            continue
-
-        inner_v = symtab[v]
-        assert isinstance(inner_v, Var)
-        src_ts = src_typenv[inner_v].get_typeset()
-        xform_ts = xform.ti[v].get_typeset()
-
-        assert xform_ts.issubset(src_ts)
-        if src_ts != xform_ts:
-            check_l.append(InTypeset(xform.ti[v], xform_ts))
-
-    # 2,3) Add any constraints that appear in xform.ti
-    check_l.extend(xform.ti.constraints)
-
-    return check_l
-
-
-def emit_runtime_typecheck(check, fmt, type_sets):
-    # type: (TypeConstraint, Formatter, UniqueTable) -> None
-    """
-    Emit rust code for the given check.
-
-    The emitted code is a statement redefining the `predicate` variable like
-    this:
-
-        let predicate = predicate && ...
-    """
-    def build_derived_expr(tv):
-        # type: (TypeVar) -> str
-        """
-        Build an expression of type Option corresponding to a concrete
-        type transformed by the sequence of derivation functions in tv.
-
-        We are using Option, as some constraints may cause an
-        over/underflow on patterns that do not match them. We want to capture
-        this without panicking at runtime.
-        """
-        if not tv.is_derived:
-            assert tv.name.startswith('typeof_')
-            return "Some({})".format(tv.name)
-
-        base_exp = build_derived_expr(tv.base)
-        if (tv.derived_func == TypeVar.LANEOF):
-            return "{}.map(|t: crate::ir::Type| t.lane_type())"\
-                .format(base_exp)
-        elif (tv.derived_func == TypeVar.ASBOOL):
-            return "{}.map(|t: crate::ir::Type| t.as_bool())".format(base_exp)
-        elif (tv.derived_func == TypeVar.HALFWIDTH):
-            return "{}.and_then(|t: crate::ir::Type| t.half_width())"\
-                .format(base_exp)
-        elif (tv.derived_func == TypeVar.DOUBLEWIDTH):
-            return "{}.and_then(|t: crate::ir::Type| t.double_width())"\
-                .format(base_exp)
-        elif (tv.derived_func == TypeVar.HALFVECTOR):
-            return "{}.and_then(|t: crate::ir::Type| t.half_vector())"\
-                .format(base_exp)
-        elif (tv.derived_func == TypeVar.DOUBLEVECTOR):
-            return "{}.and_then(|t: crate::ir::Type| t.by(2))".format(base_exp)
-        else:
-            assert False, "Unknown derived function {}".format(tv.derived_func)
-
-    if (isinstance(check, InTypeset)):
-        assert not check.tv.is_derived
-        tv = check.tv.name
-        if check.ts not in type_sets.index:
-            type_sets.add(check.ts)
-        ts = type_sets.index[check.ts]
-        fmt.comment("{} must belong to {}".format(tv, check.ts))
-        fmt.format(
-                'let predicate = predicate && TYPE_SETS[{}].contains({});',
-                ts, tv)
-    elif (isinstance(check, TypesEqual)):
-        with fmt.indented(
-            'let predicate = predicate && match ({}, {}) {{'
-            .format(build_derived_expr(check.tv1),
-                    build_derived_expr(check.tv2)), '};'):
-            fmt.line('(Some(a), Some(b)) => a == b,')
-            fmt.comment('On overflow, constraint doesn\'t appply')
-            fmt.line('_ => false,')
-    elif (isinstance(check, WiderOrEq)):
-        with fmt.indented(
-            'let predicate = predicate && match ({}, {}) {{'
-            .format(build_derived_expr(check.tv1),
-                    build_derived_expr(check.tv2)), '};'):
-            fmt.line('(Some(a), Some(b)) => a.wider_or_equal(b),')
-            fmt.comment('On overflow, constraint doesn\'t appply')
-            fmt.line('_ => false,')
-    else:
-        assert False, "Unknown check {}".format(check)
-
-
-def unwrap_inst(iref, node, fmt):
-    # type: (str, Def, Formatter) -> bool
-    """
-    Given a `Def` node, emit code that extracts all the instruction fields from
-    `pos.func.dfg[iref]`.
-
-    Create local variables named after the `Var` instances in `node`.
-
-    Also create a local variable named `predicate` with the value of the
-    evaluated instruction predicate, or `true` if the node has no predicate.
-
-    :param iref: Name of the `Inst` reference to unwrap.
-    :param node: `Def` node providing variable names.
-    :returns: True if the instruction arguments were not detached, expecting a
-              replacement instruction to overwrite the original.
-    """
-    fmt.comment('Unwrap {}'.format(node))
-    expr = node.expr
-    iform = expr.inst.format
-    nvops = iform.num_value_operands
-
-    # The tuple of locals to extract is the `Var` instances in `expr.args`.
-    arg_names = tuple(
-            arg.name if isinstance(arg, Var) else '_' for arg in expr.args)
-    with fmt.indented(
-            'let ({}, predicate) = if let crate::ir::InstructionData::{} {{'
-            .format(', '.join(map(str, arg_names)), iform.name), '};'):
-        # Fields are encoded directly.
-        for f in iform.imm_fields:
-            fmt.line('{},'.format(f.member))
-        if nvops == 1:
-            fmt.line('arg,')
-        elif iform.has_value_list or nvops > 1:
-            fmt.line('ref args,')
-        fmt.line('..')
-        fmt.outdented_line('} = pos.func.dfg[inst] {')
-        fmt.line('let func = &pos.func;')
-        if iform.has_value_list:
-            fmt.line('let args = args.as_slice(&func.dfg.value_lists);')
-        elif nvops == 1:
-            fmt.line('let args = [arg];')
-        # Generate the values for the tuple.
-        with fmt.indented('(', ')'):
-            for opnum, op in enumerate(expr.inst.ins):
-                if op.is_immediate():
-                    n = expr.inst.imm_opnums.index(opnum)
-                    fmt.format('{},', iform.imm_fields[n].member)
-                elif op.is_value():
-                    n = expr.inst.value_opnums.index(opnum)
-                    fmt.format('func.dfg.resolve_aliases(args[{}]),', n)
-            # Evaluate the instruction predicate, if any.
-            instp = expr.inst_predicate_with_ctrl_typevar()
-            fmt.line(instp.rust_predicate(0) if instp else 'true')
-        fmt.outdented_line('} else {')
-        fmt.line('unreachable!("bad instruction format")')
-
-    # Get the types of any variables where it is needed.
-    for opnum in expr.inst.value_opnums:
-        v = expr.args[opnum]
-        if isinstance(v, Var) and v.has_free_typevar():
-            fmt.format('let typeof_{0} = pos.func.dfg.value_type({0});', v)
-
-    # If the node has results, detach the values.
-    # Place the values in locals.
-    replace_inst = False
-    if len(node.defs) > 0:
-        if node.defs == node.defs[0].dst_def.defs:
-            # Special case: The instruction replacing node defines the exact
-            # same values.
-            fmt.comment(
-                    'Results handled by {}.'
-                    .format(node.defs[0].dst_def))
-            replace_inst = True
-        else:
-            # Boring case: Detach the result values, capture them in locals.
-            for d in node.defs:
-                fmt.line('let {};'.format(d))
-            with fmt.indented('{', '}'):
-                fmt.line('let r = pos.func.dfg.inst_results(inst);')
-                for i in range(len(node.defs)):
-                    fmt.line('{} = r[{}];'.format(node.defs[i], i))
-            for d in node.defs:
-                if d.has_free_typevar():
-                    fmt.line(
-                            'let typeof_{0} = pos.func.dfg.value_type({0});'
-                            .format(d))
-
-    return replace_inst
-
-
-def wrap_tup(seq):
-    # type: (Sequence[object]) -> str
-    tup = tuple(map(str, seq))
-    if len(tup) == 1:
-        return tup[0]
-    else:
-        return '({})'.format(', '.join(tup))
-
-
-def is_value_split(node):
-    # type: (Def) -> bool
-    """
-    Determine if `node` represents one of the value splitting instructions:
-    `isplit` or `vsplit. These instructions are lowered specially by the
-    `legalize::split` module.
-    """
-    if len(node.defs) != 2:
-        return False
-    return node.expr.inst in (instructions.isplit, instructions.vsplit)
-
-
-def emit_dst_inst(node, fmt):
-    # type: (Def, Formatter) -> None
-    replaced_inst = None  # type: str
-
-    if is_value_split(node):
-        # Split instructions are not emitted with the builder, but by calling
-        # special functions in the `legalizer::split` module. These functions
-        # will eliminate concat-split patterns.
-        fmt.line('let curpos = pos.position();')
-        fmt.line('let srcloc = pos.srcloc();')
-        fmt.format(
-                'let {} = split::{}(pos.func, cfg, curpos, srcloc, {});',
-                wrap_tup(node.defs),
-                node.expr.inst.snake_name(),
-                node.expr.args[0])
-    else:
-        if len(node.defs) == 0:
-            # This node doesn't define any values, so just insert the new
-            # instruction.
-            builder = 'pos.ins()'
-        else:
-            src_def0 = node.defs[0].src_def
-            if src_def0 and node.defs == src_def0.defs:
-                # The replacement instruction defines the exact same values as
-                # the source pattern. Unwrapping would have left the results
-                # intact.
-                # Replace the whole instruction.
-                builder = 'let {} = pos.func.dfg.replace(inst)'.format(
-                        wrap_tup(node.defs))
-                replaced_inst = 'inst'
-            else:
-                # Insert a new instruction.
-                builder = 'let {} = pos.ins()'.format(wrap_tup(node.defs))
-                # We may want to reuse some of the detached output values.
-                if len(node.defs) == 1 and node.defs[0].is_output():
-                    # Reuse the single source result value.
-                    builder += '.with_result({})'.format(node.defs[0])
-                elif any(d.is_output() for d in node.defs):
-                    # We have some output values to be reused.
-                    array = ', '.join(
-                            ('Some({})'.format(d) if d.is_output()
-                                else 'None')
-                            for d in node.defs)
-                    builder += '.with_results([{}])'.format(array)
-
-        fmt.line('{}.{};'.format(builder, node.expr.rust_builder(node.defs)))
-
-    # If we just replaced an instruction, we need to bump the cursor so
-    # following instructions are inserted *after* the replaced instruction.
-    if replaced_inst:
-        with fmt.indented(
-                'if pos.current_inst() == Some({}) {{'
-                .format(replaced_inst), '}'):
-            fmt.line('pos.next_inst();')
-
-
-def gen_xform(xform, fmt, type_sets):
-    # type: (XForm, Formatter, UniqueTable) -> None
-    """
-    Emit code for `xform`, assuming that the opcode of xform's root instruction
-    has already been matched.
-
-    `inst: Inst` is the variable to be replaced. It is pointed to by `pos:
-    Cursor`.
-    `dfg: DataFlowGraph` is available and mutable.
-    """
-    # Unwrap the source instruction, create local variables for the input
-    # variables.
-    replace_inst = unwrap_inst('inst', xform.src.rtl[0], fmt)
-
-    # Emit any runtime checks.
-    # These will rebind `predicate` emitted by unwrap_inst().
-    for check in get_runtime_typechecks(xform):
-        emit_runtime_typecheck(check, fmt, type_sets)
-
-    # Guard the actual expansion by `predicate`.
-    with fmt.indented('if predicate {', '}'):
-        # If we're going to delete `inst`, we need to detach its results first
-        # so they can be reattached during pattern expansion.
-        if not replace_inst:
-            fmt.line('pos.func.dfg.clear_results(inst);')
-
-        # Emit the destination pattern.
-        for dst in xform.dst.rtl:
-            emit_dst_inst(dst, fmt)
-
-        # Delete the original instruction if we didn't have an opportunity to
-        # replace it.
-        if not replace_inst:
-            fmt.line('let removed = pos.remove_inst();')
-            fmt.line('debug_assert_eq!(removed, inst);')
-        fmt.line('return true;')
-
-
-def gen_xform_group(xgrp, fmt, type_sets):
-    # type: (XFormGroup, Formatter, UniqueTable) -> None
-    fmt.doc_comment("Legalize `inst`.")
-    fmt.line('#[allow(unused_variables,unused_assignments,non_snake_case)]')
-    with fmt.indented('pub fn {}('.format(xgrp.name)):
-        fmt.line('inst: crate::ir::Inst,')
-        fmt.line('func: &mut crate::ir::Function,')
-        fmt.line('cfg: &mut crate::flowgraph::ControlFlowGraph,')
-        fmt.line('isa: &crate::isa::TargetIsa,')
-    with fmt.indented(') -> bool {', '}'):
-        fmt.line('use crate::ir::InstBuilder;')
-        fmt.line('use crate::cursor::{Cursor, FuncCursor};')
-        fmt.line('let mut pos = FuncCursor::new(func).at_inst(inst);')
-        fmt.line('pos.use_srcloc(inst);')
-
-        # Group the xforms by opcode so we can generate a big switch.
-        # Preserve ordering.
-        xforms = defaultdict(list)  # type: DefaultDict[str, List[XForm]]
-        for xform in xgrp.xforms:
-            inst = xform.src.rtl[0].expr.inst
-            xforms[inst.camel_name].append(xform)
-
-        with fmt.indented('{', '}'):
-            with fmt.indented('match pos.func.dfg[inst].opcode() {', '}'):
-                for camel_name in sorted(xforms.keys()):
-                    with fmt.indented(
-                            'ir::Opcode::{} => {{'.format(camel_name), '}'):
-                        for xform in xforms[camel_name]:
-                            gen_xform(xform, fmt, type_sets)
-
-                # Emit the custom transforms. The Rust compiler will complain
-                # about any overlap with the normal xforms.
-                for inst, funcname in xgrp.custom.items():
-                    with fmt.indented(
-                            'ir::Opcode::{} => {{'
-                            .format(inst.camel_name), '}'):
-                        fmt.format('{}(inst, pos.func, cfg, isa);', funcname)
-                        fmt.line('return true;')
-
-                # We'll assume there are uncovered opcodes.
-                fmt.line('_ => {},')
-
-        # If we fall through, nothing was expanded. Call the chain if any.
-        if xgrp.chain:
-            fmt.format('{}(inst, pos.func, cfg, isa)', xgrp.chain.rust_name())
-        else:
-            fmt.line('false')
-
-
-def gen_isa(isa, fmt, shared_groups):
-    # type: (TargetISA, Formatter, Set[XFormGroup]) -> None
-    """
-    Generate legalization functions for `isa` and add any shared `XFormGroup`s
-    encountered to `shared_groups`.
-
-    Generate `TYPE_SETS` and `LEGALIZE_ACTION` tables.
-    """
-    type_sets = UniqueTable()
-    for xgrp in isa.legalize_codes.keys():
-        if xgrp.isa is None:
-            shared_groups.add(xgrp)
-        else:
-            assert xgrp.isa == isa
-            gen_xform_group(xgrp, fmt, type_sets)
-
-    gen_typesets_table(fmt, type_sets)
-
-    with fmt.indented(
-            'pub static LEGALIZE_ACTIONS: [isa::Legalize; {}] = ['
-            .format(len(isa.legalize_codes)), '];'):
-        for xgrp in isa.legalize_codes.keys():
-            fmt.format('{},', xgrp.rust_name())
-
-
-def generate(isas, out_dir):
-    # type: (Sequence[TargetISA], str) -> None
-    shared_groups = set()  # type: Set[XFormGroup]
-
-    for isa in isas:
-        fmt = Formatter()
-        gen_isa(isa, fmt, shared_groups)
-        fmt.update_file('legalize-{}.rs'.format(isa.name), out_dir)
-
-    # Shared xform groups.
-    fmt = Formatter()
-    type_sets = UniqueTable()
-    for xgrp in sorted(shared_groups, key=lambda g: g.name):
-        gen_xform_group(xgrp, fmt, type_sets)
-    gen_typesets_table(fmt, type_sets)
-    fmt.update_file('legalizer.rs', out_dir)
diff --git a/cranelift/codegen/meta-python/test_gen_legalizer.py b/cranelift/codegen/meta-python/test_gen_legalizer.py
deleted file mode 100644
index 2b1dbf4aca..0000000000
--- a/cranelift/codegen/meta-python/test_gen_legalizer.py
+++ /dev/null
@@ -1,196 +0,0 @@
-import doctest
-import gen_legalizer
-from unittest import TestCase
-from srcgen import Formatter
-from gen_legalizer import get_runtime_typechecks, emit_runtime_typecheck
-from base.instructions import vselect, vsplit, isplit, iconcat, vconcat, \
-    iconst, b1, icmp, copy, sextend, uextend, ireduce, fdemote, fpromote # noqa
-from base.legalize import narrow, expand # noqa
-from base.immediates import intcc # noqa
-from cdsl.typevar import TypeVar, TypeSet
-from cdsl.ast import Var, Def # noqa
-from cdsl.xform import Rtl, XForm # noqa
-from cdsl.ti import ti_rtl, subst, TypeEnv, get_type_env # noqa
-from unique_table import UniqueTable
-from functools import reduce
-
-try:
-    from typing import Callable, TYPE_CHECKING, Iterable, Any # noqa
-    if TYPE_CHECKING:
-        CheckProducer = Callable[[UniqueTable], str]
-except ImportError:
-    TYPE_CHECKING = False
-
-
-def load_tests(loader, tests, ignore):
-    # type: (Any, Any, Any) -> Any
-    tests.addTests(doctest.DocTestSuite(gen_legalizer))
-    return tests
-
-
-def format_check(typesets, s, *args):
-    # type: (...) -> str
-    def transform(x):
-        # type: (Any) -> str
-        if isinstance(x, TypeSet):
-            return str(typesets.index[x])
-        elif isinstance(x, TypeVar):
-            assert not x.is_derived
-            return x.name
-        else:
-            return str(x)
-
-    dummy_s = s  # type: str
-    args = tuple(map(lambda x:  transform(x), args))
-    return dummy_s.format(*args)
-
-
-def typeset_check(v, ts):
-    # type: (Var, TypeSet) -> CheckProducer
-    return lambda typesets: format_check(
-        typesets,
-        'let predicate = predicate && TYPE_SETS[{}].contains(typeof_{});\n',
-        ts, v)
-
-
-def equiv_check(tv1, tv2):
-    # type: (str, str) -> CheckProducer
-    return lambda typesets: format_check(
-        typesets,
-        'let predicate = predicate && match ({}, {}) {{\n'
-        '    (Some(a), Some(b)) => a == b,\n'
-        '    _ => false,\n'
-        '}};\n', tv1, tv2)
-
-
-def wider_check(tv1, tv2):
-    # type: (str, str) -> CheckProducer
-    return lambda typesets: format_check(
-        typesets,
-        'let predicate = predicate && match ({}, {}) {{\n'
-        '    (Some(a), Some(b)) => a.wider_or_equal(b),\n'
-        '    _ => false,\n'
-        '}};\n', tv1, tv2)
-
-
-def sequence(*args):
-    # type: (...) -> CheckProducer
-    dummy = args  # type: Iterable[CheckProducer]
-
-    def sequenceF(typesets):
-        # type: (UniqueTable) -> str
-        def strconcat(acc, el):
-            # type: (str, CheckProducer) -> str
-            return acc + el(typesets)
-
-        return reduce(strconcat, dummy, "")
-    return sequenceF
-
-
-class TestRuntimeChecks(TestCase):
-
-    def setUp(self):
-        # type: () -> None
-        self.v0 = Var("v0")
-        self.v1 = Var("v1")
-        self.v2 = Var("v2")
-        self.v3 = Var("v3")
-        self.v4 = Var("v4")
-        self.v5 = Var("v5")
-        self.v6 = Var("v6")
-        self.v7 = Var("v7")
-        self.v8 = Var("v8")
-        self.v9 = Var("v9")
-        self.imm0 = Var("imm0")
-        self.IxN_nonscalar = TypeVar("IxN_nonscalar", "", ints=True,
-                                     scalars=False, simd=True)
-        self.TxN = TypeVar("TxN", "", ints=True, bools=True, floats=True,
-                           scalars=False, simd=True)
-        self.b1 = TypeVar.singleton(b1)
-
-    def check_yo_check(self, xform, expected_f):
-        # type: (XForm, CheckProducer) -> None
-        fmt = Formatter()
-        type_sets = UniqueTable()
-        for check in get_runtime_typechecks(xform):
-            emit_runtime_typecheck(check, fmt, type_sets)
-
-        # Remove comments
-        got = "".join([l for l in fmt.lines if not l.strip().startswith("//")])
-        expected = expected_f(type_sets)
-        self.assertEqual(got, expected)
-
-    def test_width_check(self):
-        # type: () -> None
-        x = XForm(Rtl(self.v0 << copy(self.v1)),
-                  Rtl((self.v2, self.v3) << isplit(self.v1),
-                      self.v0 << iconcat(self.v2, self.v3)))
-
-        WideInt = TypeSet(lanes=(1, 256), ints=(16, 64))
-        self.check_yo_check(x, typeset_check(self.v1, WideInt))
-
-    def test_lanes_check(self):
-        # type: () -> None
-        x = XForm(Rtl(self.v0 << copy(self.v1)),
-                  Rtl((self.v2, self.v3) << vsplit(self.v1),
-                      self.v0 << vconcat(self.v2, self.v3)))
-
-        WideVec = TypeSet(lanes=(2, 256), ints=(8, 64), floats=(32, 64),
-                          bools=(1, 64))
-        self.check_yo_check(x, typeset_check(self.v1, WideVec))
-
-    def test_vselect_imm(self):
-        # type: () -> None
-        ts = TypeSet(lanes=(2, 256), ints=True, floats=True, bools=(8, 64))
-        r = Rtl(
-                self.v0 << iconst(self.imm0),
-                self.v1 << icmp(intcc.eq, self.v2, self.v0),
-                self.v5 << vselect(self.v1, self.v3, self.v4),
-        )
-        x = XForm(r, r)
-        tv2_exp = 'Some({}).map(|t: crate::ir::Type| t.as_bool())'\
-            .format(self.v2.get_typevar().name)
-        tv3_exp = 'Some({}).map(|t: crate::ir::Type| t.as_bool())'\
-            .format(self.v3.get_typevar().name)
-
-        self.check_yo_check(
-            x, sequence(typeset_check(self.v3, ts),
-                        equiv_check(tv2_exp, tv3_exp)))
-
-    def test_reduce_extend(self):
-        # type: () -> None
-        r = Rtl(
-            self.v1 << uextend(self.v0),
-            self.v2 << ireduce(self.v1),
-            self.v3 << sextend(self.v2),
-        )
-        x = XForm(r, r)
-
-        tv0_exp = 'Some({})'.format(self.v0.get_typevar().name)
-        tv1_exp = 'Some({})'.format(self.v1.get_typevar().name)
-        tv2_exp = 'Some({})'.format(self.v2.get_typevar().name)
-        tv3_exp = 'Some({})'.format(self.v3.get_typevar().name)
-
-        self.check_yo_check(
-            x, sequence(wider_check(tv1_exp, tv0_exp),
-                        wider_check(tv1_exp, tv2_exp),
-                        wider_check(tv3_exp, tv2_exp)))
-
-    def test_demote_promote(self):
-        # type: () -> None
-        r = Rtl(
-            self.v1 << fpromote(self.v0),
-            self.v2 << fdemote(self.v1),
-            self.v3 << fpromote(self.v2),
-        )
-        x = XForm(r, r)
-
-        tv0_exp = 'Some({})'.format(self.v0.get_typevar().name)
-        tv1_exp = 'Some({})'.format(self.v1.get_typevar().name)
-        tv2_exp = 'Some({})'.format(self.v2.get_typevar().name)
-        tv3_exp = 'Some({})'.format(self.v3.get_typevar().name)
-
-        self.check_yo_check(
-            x, sequence(wider_check(tv1_exp, tv0_exp),
-                        wider_check(tv1_exp, tv2_exp),
-                        wider_check(tv3_exp, tv2_exp)))
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index bbf2a031a3..4663a29131 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -50,7 +50,7 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
         &isas,
         &shared_defs.format_registry,
         &shared_defs.transform_groups,
-        "new_legalize",
+        "legalize",
         &out_dir,
     )?;
 

From 18a5386c083c1cefed2b486eec754d163e593a45 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 25 Apr 2019 16:54:52 +0200
Subject: [PATCH 2423/3084] Remove and reorganize IntCC/FloatCC imports to
 avoid a build warning;

---
 cranelift/codegen/src/isa/x86/enc_tables.rs | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)

diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 0ed238ac18..dd7bf02c86 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -4,7 +4,7 @@ use super::registers::*;
 use crate::bitset::BitSet;
 use crate::cursor::{Cursor, FuncCursor};
 use crate::flowgraph::ControlFlowGraph;
-use crate::ir::condcodes::IntCC;
+use crate::ir::condcodes::{FloatCC, IntCC};
 use crate::ir::{self, Function, Inst, InstBuilder};
 use crate::isa;
 use crate::isa::constraints::*;
@@ -280,8 +280,6 @@ fn expand_minmax(
     cfg: &mut ControlFlowGraph,
     _isa: &isa::TargetIsa,
 ) {
-    use crate::ir::condcodes::FloatCC;
-
     let (x, y, x86_opc, bitwise_opc) = match func.dfg[inst] {
         ir::InstructionData::Binary {
             opcode: ir::Opcode::Fmin,
@@ -374,8 +372,6 @@ fn expand_fcvt_from_uint(
     cfg: &mut ControlFlowGraph,
     _isa: &isa::TargetIsa,
 ) {
-    use crate::ir::condcodes::IntCC;
-
     let x;
     match func.dfg[inst] {
         ir::InstructionData::Unary {
@@ -447,7 +443,6 @@ fn expand_fcvt_to_sint(
     cfg: &mut ControlFlowGraph,
     _isa: &isa::TargetIsa,
 ) {
-    use crate::ir::condcodes::{FloatCC, IntCC};
     use crate::ir::immediates::{Ieee32, Ieee64};
 
     let x = match func.dfg[inst] {
@@ -543,7 +538,6 @@ fn expand_fcvt_to_sint_sat(
     cfg: &mut ControlFlowGraph,
     _isa: &isa::TargetIsa,
 ) {
-    use crate::ir::condcodes::{FloatCC, IntCC};
     use crate::ir::immediates::{Ieee32, Ieee64};
 
     let x = match func.dfg[inst] {
@@ -663,7 +657,6 @@ fn expand_fcvt_to_uint(
     cfg: &mut ControlFlowGraph,
     _isa: &isa::TargetIsa,
 ) {
-    use crate::ir::condcodes::{FloatCC, IntCC};
     use crate::ir::immediates::{Ieee32, Ieee64};
 
     let x = match func.dfg[inst] {
@@ -745,7 +738,6 @@ fn expand_fcvt_to_uint_sat(
     cfg: &mut ControlFlowGraph,
     _isa: &isa::TargetIsa,
 ) {
-    use crate::ir::condcodes::{FloatCC, IntCC};
     use crate::ir::immediates::{Ieee32, Ieee64};
 
     let x = match func.dfg[inst] {

From 5cd0724fef55447849713208429359f6ed75abd1 Mon Sep 17 00:00:00 2001
From: Lars T Hansen 
Date: Thu, 2 May 2019 14:33:54 +0200
Subject: [PATCH 2424/3084] Clarify that FixedTied constraints are not Tied
 (#756)

* Clarify that FixedTied constraints are not Tied
---
 cranelift/codegen/src/isa/constraints.rs | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/src/isa/constraints.rs b/cranelift/codegen/src/isa/constraints.rs
index d6e8097385..ce1d2a5fe1 100644
--- a/cranelift/codegen/src/isa/constraints.rs
+++ b/cranelift/codegen/src/isa/constraints.rs
@@ -105,13 +105,13 @@ pub struct RecipeConstraints {
     /// constraints must be derived from the calling convention ABI.
     pub outs: &'static [OperandConstraint],
 
-    /// Are any of the input constraints `FixedReg`?
+    /// Are any of the input constraints `FixedReg` or `FixedTied`?
     pub fixed_ins: bool,
 
-    /// Are any of the output constraints `FixedReg`?
+    /// Are any of the output constraints `FixedReg` or `FixedTied`?
     pub fixed_outs: bool,
 
-    /// Are there any tied operands?
+    /// Are any of the input/output constraints `Tied` (but not `FixedTied`)?
     pub tied_ops: bool,
 
     /// Does this instruction clobber the CPU flags?

From 6a2535452046d89ce002084297602fc59a34bc63 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 15 Apr 2019 16:51:15 +0200
Subject: [PATCH 2425/3084] [meta-python] Use named predicates for x86
 encodings instead of anonymous predicates;

And generate those in a deterministic order that the Rust crate can
reproduce easily.
---
 cranelift/codegen/meta-python/cdsl/settings.py | 11 ++++++++++-
 .../codegen/meta-python/isa/x86/encodings.py   | 18 +++++++++---------
 .../codegen/meta-python/isa/x86/settings.py    |  9 ++++++++-
 3 files changed, 27 insertions(+), 11 deletions(-)

diff --git a/cranelift/codegen/meta-python/cdsl/settings.py b/cranelift/codegen/meta-python/cdsl/settings.py
index 28ff0f0b3e..fe79024221 100644
--- a/cranelift/codegen/meta-python/cdsl/settings.py
+++ b/cranelift/codegen/meta-python/cdsl/settings.py
@@ -229,16 +229,25 @@ class SettingGroup(object):
                 .format(self, SettingGroup._current))
         SettingGroup._current = None
         if globs:
+            # Ensure that named predicates are ordered in a deterministic way
+            # that the Rust crate may simply reproduce, by pushing entries into
+            # a vector that we'll sort by name later.
+            named_predicates = []
+
             for name, obj in globs.items():
                 if isinstance(obj, Setting):
                     assert obj.name is None, obj.name
                     obj.name = name
                 if isinstance(obj, Predicate):
-                    self.named_predicates[name] = obj
+                    named_predicates.append((name, obj))
                 if isinstance(obj, Preset):
                     assert obj.name is None, obj.name
                     obj.name = name
 
+            named_predicates.sort(key=lambda x: x[0])
+            for (name, obj) in named_predicates:
+                self.named_predicates[name] = obj
+
         self.layout()
 
     @staticmethod
diff --git a/cranelift/codegen/meta-python/isa/x86/encodings.py b/cranelift/codegen/meta-python/isa/x86/encodings.py
index 051b312167..a032c11898 100644
--- a/cranelift/codegen/meta-python/isa/x86/encodings.py
+++ b/cranelift/codegen/meta-python/isa/x86/encodings.py
@@ -3,7 +3,7 @@ x86 Encodings.
 """
 from __future__ import absolute_import
 from cdsl.predicates import IsZero32BitFloat, IsZero64BitFloat
-from cdsl.predicates import IsUnsignedInt, Not, And
+from cdsl.predicates import IsUnsignedInt
 from base.predicates import IsColocatedFunc, IsColocatedData, LengthEquals
 from base import instructions as base
 from base import types
@@ -15,8 +15,8 @@ from . import settings as cfg
 from . import instructions as x86
 from .legalize import x86_expand
 from base.legalize import narrow, widen, expand_flags
-from base.settings import allones_funcaddrs, is_pic
-from .settings import use_sse41
+from .settings import use_sse41, not_all_ones_funcaddrs_and_not_is_pic, \
+    all_ones_funcaddrs_and_not_is_pic, is_pic, not_is_pic
 
 try:
     from typing import TYPE_CHECKING, Any  # noqa
@@ -407,15 +407,15 @@ enc_both(base.regspill.f64, r.fregspill32, 0xf2, 0x0f, 0x11)
 
 # Non-PIC, all-ones funcaddresses.
 X86_32.enc(base.func_addr.i32, *r.fnaddr4(0xb8),
-           isap=And(Not(allones_funcaddrs), Not(is_pic)))
+           isap=not_all_ones_funcaddrs_and_not_is_pic)
 X86_64.enc(base.func_addr.i64, *r.fnaddr8.rex(0xb8, w=1),
-           isap=And(Not(allones_funcaddrs), Not(is_pic)))
+           isap=not_all_ones_funcaddrs_and_not_is_pic)
 
 # Non-PIC, all-zeros funcaddresses.
 X86_32.enc(base.func_addr.i32, *r.allones_fnaddr4(0xb8),
-           isap=And(allones_funcaddrs, Not(is_pic)))
+           isap=all_ones_funcaddrs_and_not_is_pic)
 X86_64.enc(base.func_addr.i64, *r.allones_fnaddr8.rex(0xb8, w=1),
-           isap=And(allones_funcaddrs, Not(is_pic)))
+           isap=all_ones_funcaddrs_and_not_is_pic)
 
 # 64-bit, colocated, both PIC and non-PIC. Use the lea instruction's
 # pc-relative field.
@@ -432,9 +432,9 @@ X86_64.enc(base.func_addr.i64, *r.got_fnaddr8.rex(0x8b, w=1),
 
 # Non-PIC
 X86_32.enc(base.symbol_value.i32, *r.gvaddr4(0xb8),
-           isap=Not(is_pic))
+           isap=not_is_pic)
 X86_64.enc(base.symbol_value.i64, *r.gvaddr8.rex(0xb8, w=1),
-           isap=Not(is_pic))
+           isap=not_is_pic)
 
 # PIC, colocated
 X86_64.enc(base.symbol_value.i64, *r.pcrel_gvaddr8.rex(0x8d, w=1),
diff --git a/cranelift/codegen/meta-python/isa/x86/settings.py b/cranelift/codegen/meta-python/isa/x86/settings.py
index 76aee65ddd..74633349e2 100644
--- a/cranelift/codegen/meta-python/isa/x86/settings.py
+++ b/cranelift/codegen/meta-python/isa/x86/settings.py
@@ -3,7 +3,7 @@ x86 settings.
 """
 from __future__ import absolute_import
 from cdsl.settings import SettingGroup, BoolSetting, Preset
-from cdsl.predicates import And
+from cdsl.predicates import And, Not
 import base.settings as shared
 from .defs import ISA
 
@@ -35,6 +35,13 @@ use_popcnt = And(has_popcnt, has_sse42)
 use_bmi1 = And(has_bmi1)
 use_lzcnt = And(has_lzcnt)
 
+is_pic = And(shared.is_pic)
+not_is_pic = Not(shared.is_pic)
+all_ones_funcaddrs_and_not_is_pic = And(shared.allones_funcaddrs,
+                                        Not(shared.is_pic))
+not_all_ones_funcaddrs_and_not_is_pic = And(Not(shared.allones_funcaddrs),
+                                            Not(shared.is_pic))
+
 # Presets corresponding to x86 CPUs.
 
 baseline = Preset()

From 390cfb37da677818b8ea2fcd95ea481450c298e1 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 16 Apr 2019 15:24:28 +0200
Subject: [PATCH 2426/3084] [meta] Use named predicates for x86 settings in the
 Rust crate too;

And generate them using the same deterministic order that the Python
code uses.
---
 cranelift/codegen/meta/src/cdsl/mod.rs      |  9 ++++
 cranelift/codegen/meta/src/cdsl/settings.rs | 58 ++++++++++++---------
 cranelift/codegen/meta/src/isa/x86/mod.rs   | 20 ++++++-
 3 files changed, 62 insertions(+), 25 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/mod.rs b/cranelift/codegen/meta/src/cdsl/mod.rs
index 516b04d0b2..2d7f545dd2 100644
--- a/cranelift/codegen/meta/src/cdsl/mod.rs
+++ b/cranelift/codegen/meta/src/cdsl/mod.rs
@@ -23,6 +23,15 @@ macro_rules! predicate {
     ($a:ident && $($b:tt)*) => {
         PredicateNode::And(Box::new($a.into()), Box::new(predicate!($($b)*)))
     };
+    (!$a:ident && $($b:tt)*) => {
+        PredicateNode::And(
+            Box::new(PredicateNode::Not(Box::new($a.into()))),
+            Box::new(predicate!($($b)*))
+        )
+    };
+    (!$a:ident) => {
+        PredicateNode::Not(Box::new($a.into()))
+    };
     ($a:ident) => {
         $a.into()
     };
diff --git a/cranelift/codegen/meta/src/cdsl/settings.rs b/cranelift/codegen/meta/src/cdsl/settings.rs
index 805dfd2a7a..3983946225 100644
--- a/cranelift/codegen/meta/src/cdsl/settings.rs
+++ b/cranelift/codegen/meta/src/cdsl/settings.rs
@@ -153,7 +153,7 @@ impl SettingGroup {
 /// This is the basic information needed to track the specific parts of a setting when building
 /// them.
 pub enum ProtoSpecificSetting {
-    Bool(bool, u8),
+    Bool(bool),
     Enum(Vec<&'static str>),
     Num(u8),
 }
@@ -169,6 +169,7 @@ struct ProtoSetting {
 pub enum PredicateNode {
     OwnedBool(BoolSettingIndex),
     SharedBool(&'static str, &'static str),
+    Not(Box),
     And(Box, Box),
 }
 
@@ -198,10 +199,16 @@ impl PredicateNode {
             PredicateNode::And(ref lhs, ref rhs) => {
                 format!("{} && {}", lhs.render(group), rhs.render(group))
             }
+            PredicateNode::Not(ref node) => format!("!({})", node.render(group)),
         }
     }
 }
 
+struct ProtoPredicate {
+    pub name: &'static str,
+    node: PredicateNode,
+}
+
 pub struct Predicate {
     pub name: &'static str,
     node: PredicateNode,
@@ -218,8 +225,7 @@ pub struct SettingGroupBuilder {
     name: &'static str,
     settings: Vec,
     presets: Vec,
-    predicates: Vec,
-    predicate_number: u8,
+    predicates: Vec,
 }
 
 impl SettingGroupBuilder {
@@ -229,7 +235,6 @@ impl SettingGroupBuilder {
             settings: Vec::new(),
             presets: Vec::new(),
             predicates: Vec::new(),
-            predicate_number: 0,
         }
     }
 
@@ -256,13 +261,7 @@ impl SettingGroupBuilder {
             self.predicates.len() == 0,
             "predicates must be added after the boolean settings"
         );
-        let predicate_number = self.predicate_number;
-        self.predicate_number += 1;
-        self.add_setting(
-            name,
-            comment,
-            ProtoSpecificSetting::Bool(default, predicate_number),
-        );
+        self.add_setting(name, comment, ProtoSpecificSetting::Bool(default));
         BoolSettingIndex(self.settings.len() - 1)
     }
 
@@ -280,9 +279,7 @@ impl SettingGroupBuilder {
     }
 
     pub fn add_predicate(&mut self, name: &'static str, node: PredicateNode) {
-        let number = self.predicate_number;
-        self.predicate_number += 1;
-        self.predicates.push(Predicate { name, node, number });
+        self.predicates.push(ProtoPredicate { name, node });
     }
 
     pub fn add_preset(&mut self, name: &'static str, args: Vec) -> PresetIndex {
@@ -307,8 +304,8 @@ impl SettingGroupBuilder {
     /// 1. Byte-sized settings like `NumSetting` and `EnumSetting`.
     /// 2. `BoolSetting` settings.
     /// 3. Precomputed named predicates.
-    /// 4. Other numbered predicates, including anonymous predicates and parent
-    ///    predicates that need to be accessible by number.
+    /// 4. Other numbered predicates, including parent predicates that need to be accessible by
+    ///    number.
     ///
     /// Set `self.settings_size` to the length of the byte vector prefix that
     /// contains the settings. All bytes after that are computed, not
@@ -318,9 +315,6 @@ impl SettingGroupBuilder {
     /// 2. in the list above.
     ///
     /// Assign `byte_offset` and `bit_offset` fields in all settings.
-    ///
-    /// After calling this method, no more settings can be added, but
-    /// additional predicates can be made accessible with `number_predicate()`.
     pub fn finish(self) -> SettingGroup {
         let mut group = SettingGroup {
             name: self.name,
@@ -353,12 +347,12 @@ impl SettingGroupBuilder {
 
         group.bool_start_byte_offset = byte_offset;
 
+        let mut predicate_number = 0;
+
         // Then the boolean settings.
         for s in &self.settings {
-            let (default, predicate_number) = match s.specific {
-                ProtoSpecificSetting::Bool(default, predicate_number) => {
-                    (default, predicate_number)
-                }
+            let default = match s.specific {
+                ProtoSpecificSetting::Bool(default) => default,
                 ProtoSpecificSetting::Enum(_) | ProtoSpecificSetting::Num(_) => continue,
             };
             group.settings.push(Setting {
@@ -371,6 +365,7 @@ impl SettingGroupBuilder {
                     predicate_number,
                 }),
             });
+            predicate_number += 1;
         }
 
         assert!(
@@ -379,7 +374,22 @@ impl SettingGroupBuilder {
         );
         group.settings_size = group.byte_size();
 
-        group.predicates.extend(self.predicates);
+        // Sort predicates by name to ensure the same order as the Python code.
+        let mut predicates = self.predicates;
+        predicates.sort_by_key(|predicate| predicate.name);
+
+        group
+            .predicates
+            .extend(predicates.into_iter().map(|predicate| {
+                let number = predicate_number;
+                predicate_number += 1;
+                return Predicate {
+                    name: predicate.name,
+                    node: predicate.node,
+                    number,
+                };
+            }));
+
         group.presets.extend(self.presets);
 
         group
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index 8080ea5d43..c0a35a8e1d 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -11,7 +11,7 @@ use crate::shared::Definitions as SharedDefinitions;
 mod instructions;
 mod legalize;
 
-fn define_settings(_shared: &SettingGroup) -> SettingGroup {
+fn define_settings(shared: &SettingGroup) -> SettingGroup {
     let mut settings = SettingGroupBuilder::new("x86");
 
     // CPUID.01H:ECX
@@ -47,6 +47,24 @@ fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     settings.add_predicate("use_bmi1", predicate!(has_bmi1));
     settings.add_predicate("use_lznct", predicate!(has_lzcnt));
 
+    // Some shared boolean values are used in x86 instruction predicates, so we need to group them
+    // in the same TargetIsa, for compabitibity with code generated by meta-python.
+    // TODO Once all the meta generation code has been migrated from Python to Rust, we can put it
+    // back in the shared SettingGroup, and use it in x86 instruction predicates.
+
+    let is_pic = shared.get_bool("is_pic");
+    let allones_funcaddrs = shared.get_bool("allones_funcaddrs");
+    settings.add_predicate("is_pic", predicate!(is_pic));
+    settings.add_predicate("not_is_pic", predicate!(!is_pic));
+    settings.add_predicate(
+        "all_ones_funcaddrs_and_not_is_pic",
+        predicate!(allones_funcaddrs && !is_pic),
+    );
+    settings.add_predicate(
+        "not_all_ones_funcaddrs_and_not_is_pic",
+        predicate!(!allones_funcaddrs && !is_pic),
+    );
+
     settings.add_preset("baseline", preset!());
     let nehalem = settings.add_preset(
         "nehalem",

From f335c5c56c7186cbaa1a9fd52bf9de837e1ef76a Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 16 Apr 2019 15:20:18 +0200
Subject: [PATCH 2427/3084] [meta] Small fixes in the settings generation;

---
 cranelift/codegen/meta/src/gen_settings.rs  | 22 ++++++---------------
 cranelift/codegen/meta/src/isa/riscv/mod.rs |  2 +-
 cranelift/codegen/meta/src/isa/x86/mod.rs   |  2 ++
 cranelift/codegen/meta/src/lib.rs           |  2 +-
 4 files changed, 10 insertions(+), 18 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
index b57e7a4f51..0238ef1ec6 100644
--- a/cranelift/codegen/meta/src/gen_settings.rs
+++ b/cranelift/codegen/meta/src/gen_settings.rs
@@ -288,7 +288,8 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
             });
             fmtln!(fmt, "},");
 
-            descriptor_index_map.insert(SettingOrPreset::Preset(preset), idx);
+            let whole_idx = idx + group.settings.len();
+            descriptor_index_map.insert(SettingOrPreset::Preset(preset), whole_idx);
         }
     });
     fmtln!(fmt, "];");
@@ -304,20 +305,9 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
 
     // Generate hash table.
     let mut hash_entries: Vec = Vec::new();
-    hash_entries.extend(
-        group
-            .settings
-            .iter()
-            .map(|x| SettingOrPreset::Setting(x))
-            .collect::>(),
-    );
-    hash_entries.extend(
-        group
-            .presets
-            .iter()
-            .map(|x| SettingOrPreset::Preset(x))
-            .collect::>(),
-    );
+    hash_entries.extend(group.settings.iter().map(|x| SettingOrPreset::Setting(x)));
+    hash_entries.extend(group.presets.iter().map(|x| SettingOrPreset::Preset(x)));
+
     let hash_table = generate_table(&hash_entries, |entry| simple_hash(entry.name()));
     fmtln!(fmt, "static HASH_TABLE: [u16; {}] = [", hash_table.len());
     fmt.indent(|fmt| {
@@ -341,7 +331,7 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
     fmtln!(
         fmt,
         "static PRESETS: [(u8, u8); {}] = [",
-        group.presets.len()
+        group.presets.len() * (group.settings_size as usize)
     );
     fmt.indent(|fmt| {
         for preset in &group.presets {
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index 4e947d3184..a900f07005 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -41,7 +41,7 @@ fn define_settings(shared: &SettingGroup) -> SettingGroup {
     setting.add_bool(
         "enable_e",
         "Enable the 'RV32E' instruction set with only 16 registers",
-        true,
+        false,
     );
 
     let shared_enable_atomics = shared.get_bool("enable_atomics");
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index c0a35a8e1d..861553bf88 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -65,6 +65,8 @@ fn define_settings(shared: &SettingGroup) -> SettingGroup {
         predicate!(!allones_funcaddrs && !is_pic),
     );
 
+    // Presets corresponding to x86 CPUs.
+
     settings.add_preset("baseline", preset!());
     let nehalem = settings.add_preset(
         "nehalem",
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index 4663a29131..2dff027076 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -59,7 +59,7 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
         gen_settings::generate(
             &isa.settings,
             gen_settings::ParentGroup::Shared,
-            &format!("new_settings-{}", isa.name),
+            &format!("new_settings-{}.rs", isa.name),
             &out_dir,
         )?;
     }

From d6059d4605a36815c93c78c168ca7803b2d8f2dd Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 16 Apr 2019 15:32:06 +0200
Subject: [PATCH 2428/3084] [meta] Use the Rust crate for settings generation;

---
 cranelift/codegen/meta-python/build.py        |   2 -
 cranelift/codegen/meta-python/gen_settings.py | 335 ------------------
 cranelift/codegen/meta/src/lib.rs             |   4 +-
 cranelift/codegen/src/settings.rs             |   4 +-
 4 files changed, 4 insertions(+), 341 deletions(-)
 delete mode 100644 cranelift/codegen/meta-python/gen_settings.py

diff --git a/cranelift/codegen/meta-python/build.py b/cranelift/codegen/meta-python/build.py
index 7f890a4bb1..90a0cf254d 100644
--- a/cranelift/codegen/meta-python/build.py
+++ b/cranelift/codegen/meta-python/build.py
@@ -5,7 +5,6 @@
 from __future__ import absolute_import
 import argparse
 import isa
-import gen_settings
 import gen_build_deps
 import gen_encoding
 import gen_binemit
@@ -43,7 +42,6 @@ def main():
     isas = isa.all_isas()
     number_all_instructions(isas)
 
-    gen_settings.generate(isas, out_dir)
     gen_encoding.generate(isas, out_dir)
     gen_binemit.generate(isas, out_dir)
     gen_build_deps.generate()
diff --git a/cranelift/codegen/meta-python/gen_settings.py b/cranelift/codegen/meta-python/gen_settings.py
deleted file mode 100644
index cb9d3615e3..0000000000
--- a/cranelift/codegen/meta-python/gen_settings.py
+++ /dev/null
@@ -1,335 +0,0 @@
-"""
-Generate sources with settings.
-"""
-from __future__ import absolute_import
-import srcgen
-from unique_table import UniqueSeqTable
-import constant_hash
-from cdsl import camel_case
-from cdsl.settings import BoolSetting, NumSetting, EnumSetting
-from base import settings
-
-try:
-    from typing import Sequence, Set, Tuple, List, Union, TYPE_CHECKING  # noqa
-    if TYPE_CHECKING:
-        from cdsl.isa import TargetISA  # noqa
-        from cdsl.settings import Setting, Preset, SettingGroup  # noqa
-        from cdsl.predicates import Predicate, PredContext  # noqa
-except ImportError:
-    pass
-
-
-def gen_to_and_from_str(ty, values, fmt):
-    # type: (str, Tuple[str, ...], srcgen.Formatter) -> None
-    """
-    Emit Display and FromStr implementations for enum settings.
-    """
-    with fmt.indented('impl fmt::Display for {} {{'.format(ty), '}'):
-        with fmt.indented(
-                'fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {',
-                '}'):
-            with fmt.indented('f.write_str(match *self {', '})'):
-                for v in values:
-                    fmt.line('{}::{} => "{}",'
-                             .format(ty, camel_case(v), v))
-
-    with fmt.indented('impl str::FromStr for {} {{'.format(ty), '}'):
-        fmt.line('type Err = ();')
-        with fmt.indented(
-                'fn from_str(s: &str) -> Result {',
-                '}'):
-            with fmt.indented('match s {', '}'):
-                for v in values:
-                    fmt.line('"{}" => Ok({}::{}),'
-                             .format(v, ty, camel_case(v)))
-                fmt.line('_ => Err(()),')
-
-
-def gen_enum_types(sgrp, fmt):
-    # type: (SettingGroup, srcgen.Formatter) -> None
-    """
-    Emit enum types for any enum settings.
-    """
-    for setting in sgrp.settings:
-        if not isinstance(setting, EnumSetting):
-            continue
-        ty = camel_case(setting.name)
-        fmt.doc_comment('Values for `{}`.'.format(setting))
-        fmt.line('#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]')
-        with fmt.indented('pub enum {} {{'.format(ty), '}'):
-            for v in setting.values:
-                fmt.doc_comment('`{}`.'.format(v))
-                fmt.line(camel_case(v) + ',')
-
-        gen_to_and_from_str(ty, setting.values, fmt)
-
-
-def gen_getter(setting, sgrp, fmt):
-    # type: (Setting, SettingGroup, srcgen.Formatter) -> None
-    """
-    Emit a getter function for `setting`.
-    """
-    fmt.doc_comment(setting.__doc__)
-
-    if isinstance(setting, BoolSetting):
-        proto = 'pub fn {}(&self) -> bool'.format(setting.name)
-        with fmt.indented(proto + ' {', '}'):
-            fmt.line(
-                    'self.numbered_predicate({})'
-                    .format(sgrp.predicate_number[setting]))
-    elif isinstance(setting, NumSetting):
-        proto = 'pub fn {}(&self) -> u8'.format(setting.name)
-        with fmt.indented(proto + ' {', '}'):
-            fmt.line('self.bytes[{}]'.format(setting.byte_offset))
-    elif isinstance(setting, EnumSetting):
-        ty = camel_case(setting.name)
-        proto = 'pub fn {}(&self) -> {}'.format(setting.name, ty)
-        with fmt.indented(proto + ' {', '}'):
-            m = srcgen.Match('self.bytes[{}]'.format(setting.byte_offset))
-            for i, v in enumerate(setting.values):
-                m.arm(str(i), [], '{}::{}'.format(ty, camel_case(v)))
-            m.arm('_', [], 'panic!("Invalid enum value")')
-            fmt.match(m)
-    else:
-        raise AssertionError("Unknown setting kind")
-
-
-def gen_pred_getter(name, pred, sgrp, fmt):
-    # type: (str, Predicate, SettingGroup, srcgen.Formatter) -> None
-    """
-    Emit a getter for a named pre-computed predicate.
-    """
-    fmt.doc_comment('Computed predicate `{}`.'.format(pred.rust_predicate(0)))
-    proto = 'pub fn {}(&self) -> bool'.format(name)
-    with fmt.indented(proto + ' {', '}'):
-        fmt.line(
-                'self.numbered_predicate({})'
-                .format(sgrp.predicate_number[pred]))
-
-
-def gen_getters(sgrp, fmt):
-    # type: (SettingGroup, srcgen.Formatter) -> None
-    """
-    Emit getter functions for all the settings in fmt.
-    """
-    fmt.doc_comment("User-defined settings.")
-    fmt.line('#[allow(dead_code)]')
-    with fmt.indented('impl Flags {', '}'):
-        fmt.doc_comment('Get a view of the boolean predicates.')
-        with fmt.indented(
-                'pub fn predicate_view(&self) -> '
-                'crate::settings::PredicateView {', '}'):
-            fmt.format(
-                    'crate::settings::PredicateView::new(&self.bytes[{}..])',
-                    sgrp.boolean_offset)
-        if sgrp.settings:
-            fmt.doc_comment('Dynamic numbered predicate getter.')
-            with fmt.indented(
-                    'fn numbered_predicate(&self, p: usize) -> bool {', '}'):
-                fmt.line(
-                        'self.bytes[{} + p / 8] & (1 << (p % 8)) != 0'
-                        .format(sgrp.boolean_offset))
-        for setting in sgrp.settings:
-            gen_getter(setting, sgrp, fmt)
-        for name, pred in sgrp.named_predicates.items():
-            gen_pred_getter(name, pred, sgrp, fmt)
-
-
-def gen_descriptors(sgrp, fmt):
-    # type: (SettingGroup, srcgen.Formatter) -> None
-    """
-    Generate the DESCRIPTORS, ENUMERATORS, and PRESETS tables.
-    """
-
-    enums = UniqueSeqTable()
-
-    with fmt.indented(
-            'static DESCRIPTORS: [detail::Descriptor; {}] = ['
-            .format(len(sgrp.settings) + len(sgrp.presets)),
-            '];'):
-        for idx, setting in enumerate(sgrp.settings):
-            setting.descriptor_index = idx
-            with fmt.indented('detail::Descriptor {', '},'):
-                fmt.line('name: "{}",'.format(setting.name))
-                fmt.line('offset: {},'.format(setting.byte_offset))
-                if isinstance(setting, BoolSetting):
-                    fmt.line(
-                            'detail: detail::Detail::Bool {{ bit: {} }},'
-                            .format(setting.bit_offset))
-                elif isinstance(setting, NumSetting):
-                    fmt.line('detail: detail::Detail::Num,')
-                elif isinstance(setting, EnumSetting):
-                    offs = enums.add(setting.values)
-                    fmt.line(
-                            'detail: detail::Detail::Enum ' +
-                            '{{ last: {}, enumerators: {} }},'
-                            .format(len(setting.values)-1, offs))
-                else:
-                    raise AssertionError("Unknown setting kind")
-
-        for idx, preset in enumerate(sgrp.presets):
-            preset.descriptor_index = len(sgrp.settings) + idx
-            with fmt.indented('detail::Descriptor {', '},'):
-                fmt.line('name: "{}",'.format(preset.name))
-                fmt.line('offset: {},'.format(idx * sgrp.settings_size))
-                fmt.line('detail: detail::Detail::Preset,')
-
-    with fmt.indented(
-            'static ENUMERATORS: [&str; {}] = ['
-            .format(len(enums.table)),
-            '];'):
-        for txt in enums.table:
-            fmt.line('"{}",'.format(txt))
-
-    def hash_setting(s):
-        # type: (Union[Setting, Preset]) -> int
-        return constant_hash.simple_hash(s.name)
-
-    hash_elems = []  # type: List[Union[Setting, Preset]]
-    hash_elems.extend(sgrp.settings)
-    hash_elems.extend(sgrp.presets)
-    hash_table = constant_hash.compute_quadratic(hash_elems, hash_setting)
-    with fmt.indented(
-            'static HASH_TABLE: [u16; {}] = ['
-            .format(len(hash_table)),
-            '];'):
-        for h in hash_table:
-            if h is None:
-                fmt.line('0xffff,')
-            else:
-                fmt.line('{},'.format(h.descriptor_index))
-
-    with fmt.indented(
-            'static PRESETS: [(u8, u8); {}] = ['
-            .format(len(sgrp.presets) * sgrp.settings_size),
-            '];'):
-        for preset in sgrp.presets:
-            fmt.comment(preset.name)
-            for mask, value in preset.layout():
-                fmt.format('(0b{:08b}, 0b{:08b}),', mask, value)
-
-
-def gen_template(sgrp, fmt):
-    # type: (SettingGroup, srcgen.Formatter) -> None
-    """
-    Emit a Template constant.
-    """
-    v = [0] * sgrp.settings_size
-    for setting in sgrp.settings:
-        v[setting.byte_offset] |= setting.default_byte()
-
-    with fmt.indented(
-            'static TEMPLATE: detail::Template = detail::Template {', '};'):
-        fmt.line('name: "{}",'.format(sgrp.name))
-        fmt.line('descriptors: &DESCRIPTORS,')
-        fmt.line('enumerators: &ENUMERATORS,')
-        fmt.line('hash_table: &HASH_TABLE,')
-        vs = ', '.join('{:#04x}'.format(x) for x in v)
-        fmt.line('defaults: &[{}],'.format(vs))
-        fmt.line('presets: &PRESETS,')
-
-    fmt.doc_comment(
-            'Create a `settings::Builder` for the {} settings group.'
-            .format(sgrp.name))
-    with fmt.indented('pub fn builder() -> Builder {', '}'):
-        fmt.line('Builder::new(&TEMPLATE)')
-
-
-def gen_display(sgrp, fmt):
-    # type: (SettingGroup, srcgen.Formatter) -> None
-    """
-    Generate the Display impl for Flags.
-    """
-    with fmt.indented('impl fmt::Display for Flags {', '}'):
-        with fmt.indented(
-                'fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {',
-                '}'):
-            fmt.line('writeln!(f, "[{}]")?;'.format(sgrp.name))
-            with fmt.indented('for d in &DESCRIPTORS {', '}'):
-                with fmt.indented('if !d.detail.is_preset() {', '}'):
-                    fmt.line('write!(f, "{} = ", d.name)?;')
-                    fmt.line(
-                            'TEMPLATE.format_toml_value(d.detail, ' +
-                            'self.bytes[d.offset as usize], f)?;')
-                    fmt.line('writeln!(f)?;')
-            fmt.line('Ok(())')
-
-
-def gen_constructor(sgrp, fmt):
-    # type: (SettingGroup, srcgen.Formatter) -> None
-    """
-    Generate a Flags constructor.
-    """
-
-    with fmt.indented('impl Flags {', '}'):
-        args = 'builder: Builder'
-        if sgrp.parent:
-            p = sgrp.parent
-            args = '{}: &{}::Flags, {}'.format(p.name, p.qual_mod, args)
-        fmt.doc_comment('Create flags {} settings group.'.format(sgrp.name))
-        fmt.line('#[allow(unused_variables)]')
-        with fmt.indented(
-                'pub fn new({}) -> Self {{'.format(args), '}'):
-            fmt.line('let bvec = builder.state_for("{}");'.format(sgrp.name))
-            fmt.line(
-                'let mut {} = Self {{ bytes: [0; {}] }};'
-                .format(sgrp.name, sgrp.byte_size()))
-            fmt.line(
-                'debug_assert_eq!(bvec.len(), {});'
-                .format(sgrp.settings_size))
-            fmt.line(
-                '{}.bytes[0..{}].copy_from_slice(&bvec);'
-                .format(sgrp.name, sgrp.settings_size))
-
-            # Now compute the predicates.
-            for pred, number in sgrp.predicate_number.items():
-                # Don't compute our own settings.
-                if number < sgrp.boolean_settings:
-                    continue
-                fmt.comment('Precompute #{}.'.format(number))
-                with fmt.indented(
-                        'if {} {{'.format(pred.rust_predicate(0)),
-                        '}'):
-                    fmt.line(
-                            '{}.bytes[{}] |= 1 << {};'
-                            .format(
-                                sgrp.name,
-                                sgrp.boolean_offset + number // 8,
-                                number % 8))
-
-            fmt.line(sgrp.name)
-
-
-def gen_group(sgrp, fmt):
-    # type: (SettingGroup, srcgen.Formatter) -> None
-    """
-    Generate a Flags struct representing `sgrp`.
-    """
-    fmt.line('#[derive(Clone)]')
-    fmt.doc_comment('Flags group `{}`.'.format(sgrp.name))
-    with fmt.indented('pub struct Flags {', '}'):
-        fmt.line('bytes: [u8; {}],'.format(sgrp.byte_size()))
-
-    gen_constructor(sgrp, fmt)
-    gen_enum_types(sgrp, fmt)
-    gen_getters(sgrp, fmt)
-    gen_descriptors(sgrp, fmt)
-    gen_template(sgrp, fmt)
-    gen_display(sgrp, fmt)
-
-
-def generate(isas, out_dir):
-    # type: (Sequence[TargetISA], str) -> None
-    # Generate shared settings.
-    fmt = srcgen.Formatter()
-    settings.group.qual_mod = 'settings'
-    gen_group(settings.group, fmt)
-    fmt.update_file('settings.rs', out_dir)
-
-    # Generate ISA-specific settings.
-    for isa in isas:
-        isa.settings.qual_mod = 'isa::{}::settings'.format(
-                isa.settings.name)
-        fmt = srcgen.Formatter()
-        gen_group(isa.settings, fmt)
-        fmt.update_file('settings-{}.rs'.format(isa.name), out_dir)
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index 2dff027076..9ac2a54306 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -27,7 +27,7 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
     gen_settings::generate(
         &shared_defs.settings,
         gen_settings::ParentGroup::None,
-        "new_settings.rs",
+        "settings.rs",
         &out_dir,
     )?;
     gen_types::generate("types.rs", &out_dir)?;
@@ -59,7 +59,7 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
         gen_settings::generate(
             &isa.settings,
             gen_settings::ParentGroup::Shared,
-            &format!("new_settings-{}.rs", isa.name),
+            &format!("settings-{}.rs", isa.name),
             &out_dir,
         )?;
     }
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index 404bf58cd5..6da73076e4 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -379,6 +379,8 @@ mod tests {
             f.to_string(),
             "[shared]\n\
              opt_level = \"default\"\n\
+             baldrdash_prologue_words = 0\n\
+             probestack_size_log2 = 12\n\
              enable_verifier = true\n\
              is_pic = false\n\
              colocated_libcalls = false\n\
@@ -387,11 +389,9 @@ mod tests {
              enable_nan_canonicalization = false\n\
              enable_simd = true\n\
              enable_atomics = true\n\
-             baldrdash_prologue_words = 0\n\
              allones_funcaddrs = false\n\
              probestack_enabled = true\n\
              probestack_func_adjusts_sp = false\n\
-             probestack_size_log2 = 12\n\
              jump_tables_enabled = true\n"
         );
         assert_eq!(f.opt_level(), super::OptLevel::Default);

From 8f95c51730c8f7f50abf0e0700869de9202da1d0 Mon Sep 17 00:00:00 2001
From: Yury Delendik 
Date: Wed, 6 Mar 2019 11:20:26 -0600
Subject: [PATCH 2429/3084] Reconstruct locations of the original source
 variable

---
 cranelift/codegen/src/context.rs           |  10 +
 cranelift/codegen/src/ir/dfg.rs            |  18 +-
 cranelift/codegen/src/ir/function.rs       |  51 +++-
 cranelift/codegen/src/ir/mod.rs            |  35 ++-
 cranelift/codegen/src/legalizer/heap.rs    |  10 +
 cranelift/codegen/src/lib.rs               |   2 +
 cranelift/codegen/src/print_errors.rs      |   2 +-
 cranelift/codegen/src/regalloc/context.rs  |   5 +
 cranelift/codegen/src/regalloc/liveness.rs |   5 +
 cranelift/codegen/src/value_label.rs       | 268 +++++++++++++++++++++
 cranelift/codegen/src/write.rs             |  83 ++++++-
 cranelift/frontend/src/frontend.rs         |  24 +-
 cranelift/fuzz/fuzz_translate_module.rs    |   2 +-
 cranelift/src/clif-util.rs                 |   9 +-
 cranelift/src/wasm.rs                      |  17 +-
 cranelift/wasm/src/code_translator.rs      |  11 +-
 cranelift/wasm/src/environ/dummy.rs        |   9 +-
 cranelift/wasm/src/func_translator.rs      |  12 +-
 cranelift/wasm/src/lib.rs                  |   6 +-
 cranelift/wasm/src/translation_utils.rs    |   6 +
 cranelift/wasm/tests/wasm_testsuite.rs     |   2 +-
 21 files changed, 556 insertions(+), 31 deletions(-)
 create mode 100644 cranelift/codegen/src/value_label.rs

diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index 0cd1564c06..493e7a5261 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -29,6 +29,7 @@ use crate::simple_gvn::do_simple_gvn;
 use crate::simple_preopt::do_preopt;
 use crate::timing;
 use crate::unreachable_code::eliminate_unreachable_code;
+use crate::value_label::{build_value_labels_ranges, ComparableSourceLoc, ValueLabelsRanges};
 use crate::verifier::{verify_context, verify_locations, VerifierErrors, VerifierResult};
 use std::vec::Vec;
 
@@ -331,4 +332,13 @@ impl Context {
         self.verify_locations_if(isa)?;
         Ok(code_size)
     }
+
+    /// Builds ranges and location for specified value labels.
+    pub fn build_value_labels_ranges(&self, isa: &TargetIsa) -> CodegenResult {
+        Ok(build_value_labels_ranges::(
+            &self.func,
+            &self.regalloc,
+            isa,
+        ))
+    }
 }
diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index 93ea2bbe8c..362a6cd772 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -6,7 +6,10 @@ use crate::ir::builder::ReplaceBuilder;
 use crate::ir::extfunc::ExtFuncData;
 use crate::ir::instructions::{BranchInfo, CallInfo, InstructionData};
 use crate::ir::types;
-use crate::ir::{Ebb, FuncRef, Inst, SigRef, Signature, Type, Value, ValueList, ValueListPool};
+use crate::ir::{
+    Ebb, FuncRef, Inst, SigRef, Signature, Type, Value, ValueLabelAssignments, ValueList,
+    ValueListPool,
+};
 use crate::isa::TargetIsa;
 use crate::packed_option::ReservedValue;
 use crate::write::write_operands;
@@ -15,6 +18,7 @@ use core::iter;
 use core::mem;
 use core::ops::{Index, IndexMut};
 use core::u16;
+use std::collections::HashMap;
 
 /// A data flow graph defines all instructions and extended basic blocks in a function as well as
 /// the data flow dependencies between them. The DFG also tracks values which can be either
@@ -60,6 +64,9 @@ pub struct DataFlowGraph {
 
     /// External function references. These are functions that can be called directly.
     pub ext_funcs: PrimaryMap,
+
+    /// Saves Value labels.
+    pub values_labels: Option>,
 }
 
 impl DataFlowGraph {
@@ -73,6 +80,7 @@ impl DataFlowGraph {
             values: PrimaryMap::new(),
             signatures: PrimaryMap::new(),
             ext_funcs: PrimaryMap::new(),
+            values_labels: None,
         }
     }
 
@@ -85,6 +93,7 @@ impl DataFlowGraph {
         self.values.clear();
         self.signatures.clear();
         self.ext_funcs.clear();
+        self.values_labels = None;
     }
 
     /// Get the total number of instructions created in this function, whether they are currently
@@ -117,6 +126,13 @@ impl DataFlowGraph {
     pub fn num_values(&self) -> usize {
         self.values.len()
     }
+
+    /// Starts collection of debug information.
+    pub fn collect_debug_info(&mut self) {
+        if self.values_labels.is_none() {
+            self.values_labels = Some(HashMap::new());
+        }
+    }
 }
 
 /// Resolve value aliases.
diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index e0ad323d0a..a07b0318df 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -15,6 +15,7 @@ use crate::ir::{EbbOffsets, InstEncodings, SourceLocs, StackSlots, ValueLocation
 use crate::ir::{JumpTableOffsets, JumpTables};
 use crate::isa::{CallConv, EncInfo, Encoding, Legalize, TargetIsa};
 use crate::regalloc::RegDiversions;
+use crate::value_label::ValueLabelsRanges;
 use crate::write::write_function;
 use core::fmt;
 
@@ -155,7 +156,15 @@ impl Function {
 
     /// Return an object that can display this function with correct ISA-specific annotations.
     pub fn display<'a, I: Into>>(&'a self, isa: I) -> DisplayFunction<'a> {
-        DisplayFunction(self, isa.into())
+        DisplayFunction(self, isa.into().into())
+    }
+
+    /// Return an object that can display this function with correct ISA-specific annotations.
+    pub fn display_with<'a>(
+        &'a self,
+        annotations: DisplayFunctionAnnotations<'a>,
+    ) -> DisplayFunction<'a> {
+        DisplayFunction(self, annotations)
     }
 
     /// Find a presumed unique special-purpose function parameter value.
@@ -202,26 +211,58 @@ impl Function {
     pub fn encode(&self, inst: ir::Inst, isa: &TargetIsa) -> Result {
         isa.encode(&self, &self.dfg[inst], self.dfg.ctrl_typevar(inst))
     }
+
+    /// Starts collection of debug information.
+    pub fn collect_debug_info(&mut self) {
+        self.dfg.collect_debug_info();
+    }
+}
+
+/// Additional annotations for function display.
+pub struct DisplayFunctionAnnotations<'a> {
+    /// Enable ISA annotations.
+    pub isa: Option<&'a TargetIsa>,
+
+    /// Enable value labels annotations.
+    pub value_ranges: Option<&'a ValueLabelsRanges>,
+}
+
+impl<'a> DisplayFunctionAnnotations<'a> {
+    fn default() -> Self {
+        DisplayFunctionAnnotations {
+            isa: None,
+            value_ranges: None,
+        }
+    }
+}
+
+impl<'a> From> for DisplayFunctionAnnotations<'a> {
+    fn from(isa: Option<&'a TargetIsa>) -> DisplayFunctionAnnotations {
+        DisplayFunctionAnnotations {
+            isa,
+            value_ranges: None,
+        }
+    }
 }
 
 /// Wrapper type capable of displaying a `Function` with correct ISA annotations.
-pub struct DisplayFunction<'a>(&'a Function, Option<&'a TargetIsa>);
+pub struct DisplayFunction<'a>(&'a Function, DisplayFunctionAnnotations<'a>);
 
 impl<'a> fmt::Display for DisplayFunction<'a> {
     fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
-        write_function(fmt, self.0, self.1)
+        write_function(fmt, self.0, &self.1)
     }
 }
 
 impl fmt::Display for Function {
     fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
-        write_function(fmt, self, None)
+        write_function(fmt, self, &DisplayFunctionAnnotations::default())
     }
 }
 
 impl fmt::Debug for Function {
     fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
-        write_function(fmt, self, None)
+        write_function(fmt, self, &DisplayFunctionAnnotations::default())
     }
 }
 
diff --git a/cranelift/codegen/src/ir/mod.rs b/cranelift/codegen/src/ir/mod.rs
index 8167d3b607..0873561cf9 100644
--- a/cranelift/codegen/src/ir/mod.rs
+++ b/cranelift/codegen/src/ir/mod.rs
@@ -32,7 +32,7 @@ pub use crate::ir::extfunc::{
     AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData, Signature,
 };
 pub use crate::ir::extname::ExternalName;
-pub use crate::ir::function::Function;
+pub use crate::ir::function::{DisplayFunctionAnnotations, Function};
 pub use crate::ir::globalvalue::GlobalValueData;
 pub use crate::ir::heap::{HeapData, HeapStyle};
 pub use crate::ir::instructions::{
@@ -51,7 +51,7 @@ pub use crate::ir::types::Type;
 pub use crate::ir::valueloc::{ArgumentLoc, ValueLoc};
 
 use crate::binemit;
-use crate::entity::{PrimaryMap, SecondaryMap};
+use crate::entity::{entity_impl, PrimaryMap, SecondaryMap};
 use crate::isa;
 
 /// Map of value locations.
@@ -71,3 +71,34 @@ pub type JumpTableOffsets = SecondaryMap;
 
 /// Source locations for instructions.
 pub type SourceLocs = SecondaryMap;
+
+/// Marked with a label value.
+#[derive(Copy, Clone, PartialEq, Eq, Hash)]
+pub struct ValueLabel(u32);
+entity_impl!(ValueLabel, "val");
+
+/// A label of a Value.
+#[derive(Debug, Clone)]
+pub struct ValueLabelStart {
+    /// Source location when it is in effect
+    pub from: SourceLoc,
+
+    /// The label index.
+    pub label: ValueLabel,
+}
+
+/// Value label assignements: label starts or value aliases.
+#[derive(Debug, Clone)]
+pub enum ValueLabelAssignments {
+    /// Original value labels assigned at transform.
+    Starts(std::vec::Vec),
+
+    /// A value alias to original value.
+    Alias {
+        /// Source location when it is in effect
+        from: SourceLoc,
+
+        /// The label index.
+        value: Value,
+    },
+}
diff --git a/cranelift/codegen/src/legalizer/heap.rs b/cranelift/codegen/src/legalizer/heap.rs
index 8ffd02ec5f..19217a8af9 100644
--- a/cranelift/codegen/src/legalizer/heap.rs
+++ b/cranelift/codegen/src/legalizer/heap.rs
@@ -151,7 +151,17 @@ fn compute_addr(
 
     // Convert `offset` to `addr_ty`.
     if offset_ty != addr_ty {
+        let labels_value = offset;
         offset = pos.ins().uextend(addr_ty, offset);
+        if let Some(values_labels) = pos.func.dfg.values_labels.as_mut() {
+            values_labels.insert(
+                offset,
+                ir::ValueLabelAssignments::Alias {
+                    from: pos.func.srclocs[inst],
+                    value: labels_value,
+                },
+            );
+        }
     }
 
     // Add the heap base address base
diff --git a/cranelift/codegen/src/lib.rs b/cranelift/codegen/src/lib.rs
index 48f5a27902..46ab39d971 100644
--- a/cranelift/codegen/src/lib.rs
+++ b/cranelift/codegen/src/lib.rs
@@ -57,6 +57,7 @@ use std::collections::{hash_map, HashMap, HashSet};
 
 pub use crate::context::Context;
 pub use crate::legalizer::legalize_function;
+pub use crate::value_label::ValueLabelsRanges;
 pub use crate::verifier::verify_function;
 pub use crate::write::write_function;
 
@@ -103,6 +104,7 @@ mod simple_preopt;
 mod stack_layout;
 mod topo_order;
 mod unreachable_code;
+mod value_label;
 
 pub use crate::result::{CodegenError, CodegenResult};
 
diff --git a/cranelift/codegen/src/print_errors.rs b/cranelift/codegen/src/print_errors.rs
index 2361cb714a..2d09845c92 100644
--- a/cranelift/codegen/src/print_errors.rs
+++ b/cranelift/codegen/src/print_errors.rs
@@ -29,7 +29,7 @@ pub fn pretty_verifier_error<'a>(
         &mut PrettyVerifierError(func_w.unwrap_or_else(|| Box::new(PlainWriter)), &mut errors),
         &mut w,
         func,
-        isa,
+        &isa.into(),
     )
     .unwrap();
 
diff --git a/cranelift/codegen/src/regalloc/context.rs b/cranelift/codegen/src/regalloc/context.rs
index 7ad78743db..a12a09dee1 100644
--- a/cranelift/codegen/src/regalloc/context.rs
+++ b/cranelift/codegen/src/regalloc/context.rs
@@ -64,6 +64,11 @@ impl Context {
         self.coloring.clear();
     }
 
+    /// Current values liveness state.
+    pub fn liveness(&self) -> &Liveness {
+        &self.liveness
+    }
+
     /// Allocate registers in `func`.
     ///
     /// After register allocation, all values in `func` have been assigned to a register or stack
diff --git a/cranelift/codegen/src/regalloc/liveness.rs b/cranelift/codegen/src/regalloc/liveness.rs
index 5e21c14c90..0befe50bf3 100644
--- a/cranelift/codegen/src/regalloc/liveness.rs
+++ b/cranelift/codegen/src/regalloc/liveness.rs
@@ -314,6 +314,11 @@ impl Liveness {
         }
     }
 
+    /// Current live ranges.
+    pub fn ranges(&self) -> &LiveRangeSet {
+        &self.ranges
+    }
+
     /// Get a context needed for working with a `LiveRange`.
     pub fn context<'a>(&'a self, layout: &'a Layout) -> LiveRangeContext<'a, Layout> {
         LiveRangeContext::new(layout, &self.forest)
diff --git a/cranelift/codegen/src/value_label.rs b/cranelift/codegen/src/value_label.rs
new file mode 100644
index 0000000000..f8efca7996
--- /dev/null
+++ b/cranelift/codegen/src/value_label.rs
@@ -0,0 +1,268 @@
+use crate::ir::{Function, SourceLoc, Value, ValueLabel, ValueLabelAssignments, ValueLoc};
+use crate::isa::TargetIsa;
+use crate::regalloc::{Context, RegDiversions};
+use std::cmp::Ordering;
+use std::collections::{BTreeMap, HashMap};
+use std::iter::Iterator;
+use std::ops::Bound::*;
+use std::ops::Deref;
+use std::vec::Vec;
+
+/// Value location range.
+#[derive(Debug, Clone, Copy)]
+pub struct ValueLocRange {
+    pub loc: ValueLoc,
+    pub start: u32,
+    pub end: u32,
+}
+
+/// Resulting map of Value labels and their ranges/locations.
+pub type ValueLabelsRanges = HashMap>;
+
+fn build_value_labels_index(func: &Function) -> BTreeMap
+where
+    T: From + Deref + Ord + Copy,
+{
+    if func.dfg.values_labels.is_none() {
+        return BTreeMap::new();
+    }
+    let values_labels = func.dfg.values_labels.as_ref().unwrap();
+
+    // Index values_labels by srcloc/from
+    let mut sorted = BTreeMap::new();
+    for (val, assigns) in values_labels {
+        match assigns {
+            ValueLabelAssignments::Starts(labels) => {
+                for label in labels {
+                    if label.from.is_default() {
+                        continue;
+                    }
+                    let srcloc = T::from(label.from);
+                    let label = label.label;
+                    sorted.insert(srcloc, (*val, label));
+                }
+            }
+            ValueLabelAssignments::Alias { from, value } => {
+                if from.is_default() {
+                    continue;
+                }
+                let mut aliased_value = *value;
+                while let Some(ValueLabelAssignments::Alias { value, .. }) =
+                    values_labels.get(&aliased_value)
+                {
+                    // TODO check/limit recursion?
+                    aliased_value = *value;
+                }
+                let from = T::from(*from);
+                if let Some(ValueLabelAssignments::Starts(labels)) =
+                    values_labels.get(&aliased_value)
+                {
+                    for label in labels {
+                        let srcloc = if label.from.is_default() {
+                            from
+                        } else {
+                            from.max(T::from(label.from))
+                        };
+                        let label = label.label;
+                        sorted.insert(srcloc, (*val, label));
+                    }
+                }
+            }
+        }
+    }
+    sorted
+}
+
+/// Builds ranges and location for specified value labels.
+/// The labels specified at DataFlowGraph's values_labels collection.
+pub fn build_value_labels_ranges(
+    func: &Function,
+    regalloc: &Context,
+    isa: &TargetIsa,
+) -> ValueLabelsRanges
+where
+    T: From + Deref + Ord + Copy,
+{
+    let values_labels = build_value_labels_index::(func);
+
+    let mut ebbs = func.layout.ebbs().collect::>();
+    ebbs.sort_by_key(|ebb| func.offsets[*ebb]); // Ensure inst offsets always increase
+    let encinfo = isa.encoding_info();
+    let values_locations = &func.locations;
+    let liveness_context = regalloc.liveness().context(&func.layout);
+    let liveness_ranges = regalloc.liveness().ranges();
+
+    let mut ranges = HashMap::new();
+    let mut add_range = |label, range: (u32, u32), loc: ValueLoc| {
+        if range.0 >= range.1 || !loc.is_assigned() {
+            return;
+        }
+        if !ranges.contains_key(&label) {
+            ranges.insert(label, Vec::new());
+        }
+        ranges.get_mut(&label).unwrap().push(ValueLocRange {
+            loc,
+            start: range.0,
+            end: range.1,
+        });
+    };
+
+    let mut end_offset = 0;
+    let mut tracked_values: Vec<(Value, ValueLabel, u32, ValueLoc)> = Vec::new();
+    let mut divert = RegDiversions::new();
+    for ebb in ebbs {
+        divert.clear();
+        let mut last_srcloc: Option = None;
+        for (offset, inst, size) in func.inst_offsets(ebb, &encinfo) {
+            divert.apply(&func.dfg[inst]);
+            end_offset = offset + size;
+            // Remove killed values.
+            tracked_values.retain(|(x, label, start_offset, last_loc)| {
+                let range = liveness_ranges.get(*x);
+                if range.expect("value").killed_at(inst, ebb, liveness_context) {
+                    add_range(*label, (*start_offset, end_offset), *last_loc);
+                    return false;
+                }
+                return true;
+            });
+
+            let srcloc = func.srclocs[inst];
+            if srcloc.is_default() {
+                // Don't process instructions without srcloc.
+                continue;
+            }
+            let srcloc = T::from(srcloc);
+
+            // Record and restart ranges if Value location was changed.
+            for (val, label, start_offset, last_loc) in &mut tracked_values {
+                let new_loc = divert.get(*val, values_locations);
+                if new_loc == *last_loc {
+                    continue;
+                }
+                add_range(*label, (*start_offset, end_offset), *last_loc);
+                *start_offset = end_offset;
+                *last_loc = new_loc;
+            }
+
+            // New source locations range started: abandon all tracked values.
+            if last_srcloc.is_some() && last_srcloc.as_ref().unwrap() > &srcloc {
+                for (_, label, start_offset, last_loc) in &tracked_values {
+                    add_range(*label, (*start_offset, end_offset), *last_loc);
+                }
+                tracked_values.clear();
+                last_srcloc = None;
+            }
+
+            // Get non-processed Values based on srcloc
+            let range = (
+                match last_srcloc {
+                    Some(a) => Excluded(a),
+                    None => Unbounded,
+                },
+                Included(srcloc),
+            );
+            let active_values = values_labels.range(range);
+            let active_values = active_values.filter(|(_, (v, _))| {
+                // Ignore dead/inactive Values.
+                let range = liveness_ranges.get(*v);
+                match range {
+                    Some(r) => r.reaches_use(inst, ebb, liveness_context),
+                    None => false,
+                }
+            });
+            // Append new Values to the tracked_values.
+            for (_, (val, label)) in active_values {
+                let loc = divert.get(*val, values_locations);
+                tracked_values.push((*val, *label, end_offset, loc));
+            }
+
+            last_srcloc = Some(srcloc);
+        }
+        // Finish all started ranges.
+        for (_, label, start_offset, last_loc) in &tracked_values {
+            add_range(*label, (*start_offset, end_offset), *last_loc);
+        }
+    }
+
+    // Optimize ranges in-place
+    for (_, label_ranges) in ranges.iter_mut() {
+        assert!(label_ranges.len() > 0);
+        label_ranges.sort_by(|a, b| a.start.cmp(&b.start).then_with(|| a.end.cmp(&b.end)));
+
+        // Merge ranges
+        let mut i = 1;
+        let mut j = 0;
+        while i < label_ranges.len() {
+            assert!(label_ranges[j].start <= label_ranges[i].end);
+            if label_ranges[j].loc != label_ranges[i].loc {
+                // Different location
+                if label_ranges[j].end >= label_ranges[i].end {
+                    // Consumed by previous range, skipping
+                    i += 1;
+                    continue;
+                }
+                j += 1;
+                label_ranges[j] = label_ranges[i];
+                i += 1;
+                continue;
+            }
+            if label_ranges[j].end < label_ranges[i].start {
+                // Gap in the range location
+                j += 1;
+                label_ranges[j] = label_ranges[i];
+                i += 1;
+                continue;
+            }
+            // Merge i-th and j-th ranges
+            if label_ranges[j].end < label_ranges[i].end {
+                label_ranges[j].end = label_ranges[i].end;
+            }
+            i += 1;
+        }
+        label_ranges.truncate(j + 1);
+
+        // Cut/move start position of next range, if two neighbor ranges intersect.
+        for i in 0..j {
+            if label_ranges[i].end > label_ranges[i + 1].start {
+                label_ranges[i + 1].start = label_ranges[i].end;
+                assert!(label_ranges[i + 1].start < label_ranges[i + 1].end);
+            }
+            assert!(label_ranges[i].end <= label_ranges[i + 1].start);
+        }
+    }
+    ranges
+}
+
+#[derive(Eq, Clone, Copy)]
+pub struct ComparableSourceLoc(SourceLoc);
+
+impl From for ComparableSourceLoc {
+    fn from(s: SourceLoc) -> Self {
+        ComparableSourceLoc(s)
+    }
+}
+
+impl Deref for ComparableSourceLoc {
+    type Target = SourceLoc;
+    fn deref(&self) -> &SourceLoc {
+        &self.0
+    }
+}
+
+impl PartialOrd for ComparableSourceLoc {
+    fn partial_cmp(&self, other: &Self) -> Option {
+        Some(self.cmp(other))
+    }
+}
+
+impl Ord for ComparableSourceLoc {
+    fn cmp(&self, other: &Self) -> Ordering {
+        self.0.bits().cmp(&other.0.bits())
+    }
+}
+
+impl PartialEq for ComparableSourceLoc {
+    fn eq(&self, other: &Self) -> bool {
+        self.0 == other.0
+    }
+}
diff --git a/cranelift/codegen/src/write.rs b/cranelift/codegen/src/write.rs
index 7731918c59..f1810524d8 100644
--- a/cranelift/codegen/src/write.rs
+++ b/cranelift/codegen/src/write.rs
@@ -5,10 +5,15 @@
 
 use crate::entity::SecondaryMap;
 use crate::ir::entities::AnyEntity;
-use crate::ir::{DataFlowGraph, Ebb, Function, Inst, SigRef, Type, Value, ValueDef};
+use crate::ir::{
+    DataFlowGraph, DisplayFunctionAnnotations, Ebb, Function, Inst, SigRef, Type, Value, ValueDef,
+    ValueLoc,
+};
 use crate::isa::{RegInfo, TargetIsa};
 use crate::packed_option::ReservedValue;
+use crate::value_label::ValueLabelsRanges;
 use core::fmt::{self, Write};
+use std::collections::HashSet;
 use std::string::String;
 use std::vec::Vec;
 
@@ -154,8 +159,12 @@ impl FuncWriter for PlainWriter {
 
 /// Write `func` to `w` as equivalent text.
 /// Use `isa` to emit ISA-dependent annotations.
-pub fn write_function(w: &mut Write, func: &Function, isa: Option<&TargetIsa>) -> fmt::Result {
-    decorate_function(&mut PlainWriter, w, func, isa)
+pub fn write_function(
+    w: &mut Write,
+    func: &Function,
+    annotations: &DisplayFunctionAnnotations,
+) -> fmt::Result {
+    decorate_function(&mut PlainWriter, w, func, annotations)
 }
 
 /// Create a reverse-alias map from a value to all aliases having that value as a direct target
@@ -177,9 +186,9 @@ pub fn decorate_function(
     func_w: &mut FW,
     w: &mut Write,
     func: &Function,
-    isa: Option<&TargetIsa>,
+    annotations: &DisplayFunctionAnnotations,
 ) -> fmt::Result {
-    let regs = isa.map(TargetIsa::register_info);
+    let regs = annotations.isa.map(TargetIsa::register_info);
     let regs = regs.as_ref();
 
     write!(w, "function ")?;
@@ -191,7 +200,7 @@ pub fn decorate_function(
         if any {
             writeln!(w)?;
         }
-        decorate_ebb(func_w, w, func, &aliases, isa, ebb)?;
+        decorate_ebb(func_w, w, func, &aliases, annotations, ebb)?;
         any = true;
     }
     writeln!(w, "}}")
@@ -254,12 +263,53 @@ pub fn write_ebb_header(
     writeln!(w, "):")
 }
 
+fn write_valueloc(w: &mut Write, loc: &ValueLoc, regs: &RegInfo) -> fmt::Result {
+    match loc {
+        ValueLoc::Reg(r) => write!(w, "{}", regs.display_regunit(*r)),
+        ValueLoc::Stack(ss) => write!(w, "{}", ss),
+        ValueLoc::Unassigned => write!(w, "?"),
+    }
+}
+
+fn write_value_range_markers(
+    w: &mut Write,
+    val_ranges: &ValueLabelsRanges,
+    regs: &RegInfo,
+    offset: u32,
+    indent: usize,
+) -> fmt::Result {
+    let mut result = String::new();
+    let mut shown = HashSet::new();
+    for (val, rng) in val_ranges {
+        for i in (0..rng.len()).rev() {
+            if rng[i].start == offset {
+                write!(&mut result, " {}@", val)?;
+                write_valueloc(&mut result, &rng[i].loc, regs)?;
+                shown.insert(val);
+                break;
+            }
+        }
+    }
+    for (val, rng) in val_ranges {
+        for i in (0..rng.len()).rev() {
+            if rng[i].end == offset && !shown.contains(val) {
+                write!(&mut result, " {}\u{2620}", val)?;
+                break;
+            }
+        }
+    }
+    if result.len() > 0 {
+        writeln!(w, ";{1:0$}; {2}", indent + 24, "", result)?;
+    }
+    Ok(())
+}
+
 fn decorate_ebb(
     func_w: &mut FW,
     w: &mut Write,
     func: &Function,
     aliases: &SecondaryMap>,
-    isa: Option<&TargetIsa>,
+    annotations: &DisplayFunctionAnnotations,
     ebb: Ebb,
 ) -> fmt::Result {
     // Indent all instructions if any encodings are present.
@@ -268,13 +318,28 @@ fn decorate_ebb(
     } else {
         36
     };
+    let isa = annotations.isa;
 
     func_w.write_ebb_header(w, func, isa, ebb, indent)?;
     for a in func.dfg.ebb_params(ebb).iter().cloned() {
         write_value_aliases(w, aliases, a, indent)?;
     }
-    for inst in func.layout.ebb_insts(ebb) {
-        func_w.write_instruction(w, func, aliases, isa, inst, indent)?;
+
+    if isa.is_some() && !func.offsets.is_empty() {
+        let encinfo = isa.unwrap().encoding_info();
+        let regs = &isa.unwrap().register_info();
+        for (offset, inst, size) in func.inst_offsets(ebb, &encinfo) {
+            func_w.write_instruction(w, func, aliases, isa, inst, indent)?;
+            if size > 0 {
+                if let Some(val_ranges) = annotations.value_ranges {
+                    write_value_range_markers(w, val_ranges, regs, offset + size, indent)?;
+                }
+            }
+        }
+    } else {
+        for inst in func.layout.ebb_insts(ebb) {
+            func_w.write_instruction(w, func, aliases, isa, inst, indent)?;
+        }
     }
 
     Ok(())
diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index cefdb5bfe7..c161b85f50 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -9,7 +9,7 @@ use cranelift_codegen::ir::{
     types, AbiParam, DataFlowGraph, Ebb, ExtFuncData, ExternalName, FuncRef, Function, GlobalValue,
     GlobalValueData, Heap, HeapData, Inst, InstBuilder, InstBuilderBase, InstructionData,
     JumpTable, JumpTableData, LibCall, MemFlags, SigRef, Signature, StackSlot, StackSlotData, Type,
-    Value,
+    Value, ValueLabel, ValueLabelAssignments, ValueLabelStart,
 };
 use cranelift_codegen::isa::{TargetFrontendConfig, TargetIsa};
 use cranelift_codegen::packed_option::PackedOption;
@@ -333,6 +333,28 @@ impl<'a> FunctionBuilder<'a> {
             .def_var(var, val, self.position.basic_block.unwrap());
     }
 
+    /// Set label for Value
+    pub fn set_val_label(&mut self, val: Value, label: ValueLabel) {
+        if let Some(values_labels) = self.func.dfg.values_labels.as_mut() {
+            use std::collections::hash_map::Entry;
+
+            let start = ValueLabelStart {
+                from: self.srcloc,
+                label,
+            };
+
+            match values_labels.entry(val) {
+                Entry::Occupied(mut e) => match e.get_mut() {
+                    ValueLabelAssignments::Starts(starts) => starts.push(start),
+                    _ => panic!("Unexpected ValueLabelAssignments at this stage"),
+                },
+                Entry::Vacant(e) => {
+                    e.insert(ValueLabelAssignments::Starts(vec![start]));
+                }
+            }
+        }
+    }
+
     /// Creates a jump table in the function, to be used by `br_table` instructions.
     pub fn create_jump_table(&mut self, data: JumpTableData) -> JumpTable {
         self.func.create_jump_table(data)
diff --git a/cranelift/fuzz/fuzz_translate_module.rs b/cranelift/fuzz/fuzz_translate_module.rs
index abc742a62f..27854f329f 100644
--- a/cranelift/fuzz/fuzz_translate_module.rs
+++ b/cranelift/fuzz/fuzz_translate_module.rs
@@ -20,6 +20,6 @@ fuzz_target!(|data: &[u8]| {
     let flags = settings::Flags::new(settings::builder());
     let triple = triple!("x86_64");
     let isa = isa::lookup(triple).unwrap().finish(flags);
-    let mut dummy_environ = DummyEnvironment::new(isa.frontend_config(), ReturnMode::NormalReturns);
+    let mut dummy_environ = DummyEnvironment::new(isa.frontend_config(), ReturnMode::NormalReturns, false);
     translate_module(&wasm, &mut dummy_environ).unwrap();
 });
diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index d51687b4fe..a2b248b0df 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -184,7 +184,13 @@ fn main() {
                     "Just checks the correctness of Cranelift IR translated from WebAssembly",
                 )),
         )
-        .subcommand(add_wasm_or_compile("wasm"))
+        .subcommand(
+            add_wasm_or_compile("wasm").arg(
+                Arg::with_name("value-ranges")
+                    .long("value-ranges")
+                    .help("Display values ranges and their locations"),
+            ),
+        )
         .subcommand(
             SubCommand::with_name("pass")
                 .about("Run specified pass(s) on an input file.")
@@ -269,6 +275,7 @@ fn main() {
                     target_val,
                     rest_cmd.is_present("print-size"),
                     rest_cmd.is_present("time-passes"),
+                    rest_cmd.is_present("value-ranges"),
                 )
             };
 
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 67d0225f49..61a8d8b1a9 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -13,6 +13,7 @@ use cranelift_codegen::print_errors::{pretty_error, pretty_verifier_error};
 use cranelift_codegen::settings::FlagsOrIsa;
 use cranelift_codegen::timing;
 use cranelift_codegen::Context;
+use cranelift_codegen::ir::DisplayFunctionAnnotations;
 use cranelift_entity::EntityRef;
 use cranelift_wasm::{translate_module, DummyEnvironment, FuncIndex, ReturnMode};
 use std::path::Path;
@@ -47,6 +48,7 @@ pub fn run(
     flag_triple: &str,
     flag_print_size: bool,
     flag_report_times: bool,
+    flag_calc_value_ranges: bool,
 ) -> Result<(), String> {
     let parsed = parse_sets_and_triple(flag_set, flag_triple)?;
 
@@ -61,6 +63,7 @@ pub fn run(
             flag_print_size,
             flag_print_disasm,
             flag_report_times,
+            flag_calc_value_ranges,
             &path.to_path_buf(),
             &name,
             parsed.as_fisa(),
@@ -77,6 +80,7 @@ fn handle_module(
     flag_print_size: bool,
     flag_print_disasm: bool,
     flag_report_times: bool,
+    flag_calc_value_ranges: bool,
     path: &PathBuf,
     name: &str,
     fisa: FlagsOrIsa,
@@ -108,7 +112,8 @@ fn handle_module(
         }
     };
 
-    let mut dummy_environ = DummyEnvironment::new(isa.frontend_config(), ReturnMode::NormalReturns);
+    let debug_info = flag_calc_value_ranges;
+    let mut dummy_environ = DummyEnvironment::new(isa.frontend_config(), ReturnMode::NormalReturns, debug_info);
     translate_module(&module_binary, &mut dummy_environ).map_err(|e| e.to_string())?;
 
     let _ = terminal.fg(term::color::GREEN);
@@ -205,7 +210,15 @@ fn handle_module(
             {
                 println!("; Exported as \"{}\"", export_name);
             }
-            println!("{}", context.func.display(fisa.isa));
+            let value_ranges = if flag_calc_value_ranges {
+                Some(context.build_value_labels_ranges(isa).expect("value location ranges"))
+            } else {
+                None
+            };
+            println!("{}", context.func.display_with(DisplayFunctionAnnotations {
+                isa: fisa.isa,
+                value_ranges: value_ranges.as_ref(),
+            }));
             vprintln!(flag_verbose, "");
         }
 
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 6c339f3e1f..d0b504dd69 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -30,7 +30,7 @@ use crate::translation_utils::{FuncIndex, MemoryIndex, SignatureIndex, TableInde
 use core::{i32, u32};
 use cranelift_codegen::ir::condcodes::{FloatCC, IntCC};
 use cranelift_codegen::ir::types::*;
-use cranelift_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags};
+use cranelift_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags, ValueLabel};
 use cranelift_codegen::packed_option::ReservedValue;
 use cranelift_frontend::{FunctionBuilder, Variable};
 use wasmparser::{MemoryImmediate, Operator};
@@ -57,15 +57,22 @@ pub fn translate_operator(
          *  disappear in the Cranelift Code
          ***********************************************************************************/
         Operator::GetLocal { local_index } => {
-            state.push1(builder.use_var(Variable::with_u32(local_index)))
+            let val = builder.use_var(Variable::with_u32(local_index));
+            state.push1(val);
+            let label = ValueLabel::from_u32(local_index);
+            builder.set_val_label(val, label);
         }
         Operator::SetLocal { local_index } => {
             let val = state.pop1();
             builder.def_var(Variable::with_u32(local_index), val);
+            let label = ValueLabel::from_u32(local_index);
+            builder.set_val_label(val, label);
         }
         Operator::TeeLocal { local_index } => {
             let val = state.peek1();
             builder.def_var(Variable::with_u32(local_index), val);
+            let label = ValueLabel::from_u32(local_index);
+            builder.set_val_label(val, label);
         }
         /********************************** Globals ****************************************
          *  `get_global` and `set_global` are handled by the environment.
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index e32a3260ab..7c6625f069 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -121,16 +121,20 @@ pub struct DummyEnvironment {
 
     /// How to return from functions.
     return_mode: ReturnMode,
+
+    /// Instructs to collect debug data during translation.
+    debug_info: bool,
 }
 
 impl DummyEnvironment {
     /// Creates a new `DummyEnvironment` instance.
-    pub fn new(config: TargetFrontendConfig, return_mode: ReturnMode) -> Self {
+    pub fn new(config: TargetFrontendConfig, return_mode: ReturnMode, debug_info: bool) -> Self {
         Self {
             info: DummyModuleInfo::new(config),
             trans: FuncTranslator::new(),
             func_bytecode_sizes: Vec::new(),
             return_mode,
+            debug_info,
         }
     }
 
@@ -482,6 +486,9 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
             let name = get_func_name(func_index);
             let sig = func_environ.vmctx_sig(self.get_func_type(func_index));
             let mut func = ir::Function::with_name_signature(name, sig);
+            if self.debug_info {
+                func.collect_debug_info();
+            }
             self.trans
                 .translate(body_bytes, body_offset, &mut func, &mut func_environ)?;
             func
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index efa63c15e9..578c12ef3f 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -7,8 +7,9 @@
 use crate::code_translator::translate_operator;
 use crate::environ::{FuncEnvironment, ReturnMode, WasmError, WasmResult};
 use crate::state::TranslationState;
+use crate::translation_utils::get_vmctx_value_label;
 use cranelift_codegen::entity::EntityRef;
-use cranelift_codegen::ir::{self, Ebb, InstBuilder};
+use cranelift_codegen::ir::{self, Ebb, InstBuilder, ValueLabel};
 use cranelift_codegen::timing;
 use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
 use log::info;
@@ -84,6 +85,7 @@ impl FuncTranslator {
 
         // This clears the `FunctionBuilderContext`.
         let mut builder = FunctionBuilder::new(func, &mut self.func_ctx);
+        builder.set_srcloc(cur_srcloc(&reader));
         let entry_block = builder.create_ebb();
         builder.append_ebb_params_for_function_params(entry_block);
         builder.switch_to_block(entry_block); // This also creates values for the arguments.
@@ -127,6 +129,10 @@ fn declare_wasm_parameters(builder: &mut FunctionBuilder, entry_block: Ebb) -> u
             let param_value = builder.ebb_params(entry_block)[i];
             builder.def_var(local, param_value);
         }
+        if param_type.purpose == ir::ArgumentPurpose::VMContext {
+            let param_value = builder.ebb_params(entry_block)[i];
+            builder.set_val_label(param_value, get_vmctx_value_label());
+        }
     }
 
     next_local
@@ -177,6 +183,7 @@ fn declare_locals(
         let local = Variable::new(*next_local);
         builder.declare_var(local, ty);
         builder.def_var(local, zeroval);
+        builder.set_val_label(zeroval, ValueLabel::new(*next_local));
         *next_local += 1;
     }
     Ok(())
@@ -265,6 +272,7 @@ mod tests {
                 pointer_width: PointerWidth::U64,
             },
             ReturnMode::NormalReturns,
+            false,
         );
 
         let mut ctx = Context::new();
@@ -304,6 +312,7 @@ mod tests {
                 pointer_width: PointerWidth::U64,
             },
             ReturnMode::NormalReturns,
+            false,
         );
         let mut ctx = Context::new();
 
@@ -351,6 +360,7 @@ mod tests {
                 pointer_width: PointerWidth::U64,
             },
             ReturnMode::NormalReturns,
+            false,
         );
         let mut ctx = Context::new();
 
diff --git a/cranelift/wasm/src/lib.rs b/cranelift/wasm/src/lib.rs
index c475b342bc..db10a7936c 100644
--- a/cranelift/wasm/src/lib.rs
+++ b/cranelift/wasm/src/lib.rs
@@ -64,9 +64,9 @@ pub use crate::environ::{
 pub use crate::func_translator::FuncTranslator;
 pub use crate::module_translator::translate_module;
 pub use crate::translation_utils::{
-    DefinedFuncIndex, DefinedGlobalIndex, DefinedMemoryIndex, DefinedTableIndex, FuncIndex, Global,
-    GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex, Table, TableElementType,
-    TableIndex,
+    get_vmctx_value_label, DefinedFuncIndex, DefinedGlobalIndex, DefinedMemoryIndex,
+    DefinedTableIndex, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex,
+    SignatureIndex, Table, TableElementType, TableIndex,
 };
 
 /// Version number of this crate.
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index 7ea730c9cb..dc4336c397 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -140,3 +140,9 @@ pub fn num_return_values(ty: wasmparser::Type) -> usize {
         _ => panic!("unsupported return value type"),
     }
 }
+
+/// Special VMContext value label. It is tracked as 0xffff_fffe label.
+pub fn get_vmctx_value_label() -> ir::ValueLabel {
+    const VMCTX_LABEL: u32 = 0xffff_fffe;
+    ir::ValueLabel::from_u32(VMCTX_LABEL)
+}
diff --git a/cranelift/wasm/tests/wasm_testsuite.rs b/cranelift/wasm/tests/wasm_testsuite.rs
index c496470ec0..f521828282 100644
--- a/cranelift/wasm/tests/wasm_testsuite.rs
+++ b/cranelift/wasm/tests/wasm_testsuite.rs
@@ -73,7 +73,7 @@ fn handle_module(path: &Path, flags: &Flags, return_mode: ReturnMode) {
     };
     let triple = triple!("riscv64");
     let isa = isa::lookup(triple).unwrap().finish(flags.clone());
-    let mut dummy_environ = DummyEnvironment::new(isa.frontend_config(), return_mode);
+    let mut dummy_environ = DummyEnvironment::new(isa.frontend_config(), return_mode, false);
 
     translate_module(&data, &mut dummy_environ).unwrap();
 

From a0ddbf403cd52bc6d1e7f9547b87de3099bd35d1 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 3 May 2019 15:38:28 +0200
Subject: [PATCH 2430/3084] [wasm] Have the WasmError::User member be a String;

---
 cranelift/wasm/Cargo.toml          |  2 +-
 cranelift/wasm/src/environ/spec.rs | 12 ++----------
 2 files changed, 3 insertions(+), 11 deletions(-)

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 7d9504aa54..2309322acb 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -27,7 +27,7 @@ target-lexicon = "0.4.0"
 
 [features]
 default = ["std"]
-std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std", "failure/std"]
+std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std"]
 core = ["hashmap_core", "cranelift-codegen/core", "cranelift-frontend/core", "wasmparser/core"]
 
 [badges]
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index ac58fd7215..d307922677 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -68,17 +68,9 @@ pub enum WasmError {
     #[fail(display = "Implementation limit exceeded")]
     ImplLimitExceeded,
 
-    /// Any user-defined error. Requires an std build, where failure::Error is defined.
-    #[cfg(feature = "std")]
+    /// Any user-defined error.
     #[fail(display = "User error: {}", _0)]
-    User(failure::Error),
-}
-
-#[cfg(feature = "std")]
-impl From for WasmError {
-    fn from(err: failure::Error) -> Self {
-        WasmError::User(err)
-    }
+    User(std::string::String),
 }
 
 impl From for WasmError {

From edd2bf12fdd7be176634aa68957a99493a999a48 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Tue, 14 May 2019 15:54:04 +0200
Subject: [PATCH 2431/3084] Export ValueLocRange and
 DisplayFunctionAnnotations::default()

---
 cranelift/codegen/src/ir/function.rs | 3 ++-
 cranelift/codegen/src/lib.rs         | 2 +-
 cranelift/codegen/src/value_label.rs | 3 +++
 3 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index a07b0318df..6a1d4cdd0d 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -228,7 +228,8 @@ pub struct DisplayFunctionAnnotations<'a> {
 }
 
 impl<'a> DisplayFunctionAnnotations<'a> {
-    fn default() -> Self {
+    /// Create a DisplayFunctionAnnotations with all fields set to None.
+    pub fn default() -> Self {
         DisplayFunctionAnnotations {
             isa: None,
             value_ranges: None,
diff --git a/cranelift/codegen/src/lib.rs b/cranelift/codegen/src/lib.rs
index 46ab39d971..5b809460cf 100644
--- a/cranelift/codegen/src/lib.rs
+++ b/cranelift/codegen/src/lib.rs
@@ -57,7 +57,7 @@ use std::collections::{hash_map, HashMap, HashSet};
 
 pub use crate::context::Context;
 pub use crate::legalizer::legalize_function;
-pub use crate::value_label::ValueLabelsRanges;
+pub use crate::value_label::{ValueLabelsRanges, ValueLocRange};
 pub use crate::verifier::verify_function;
 pub use crate::write::write_function;
 
diff --git a/cranelift/codegen/src/value_label.rs b/cranelift/codegen/src/value_label.rs
index f8efca7996..d70d6258fa 100644
--- a/cranelift/codegen/src/value_label.rs
+++ b/cranelift/codegen/src/value_label.rs
@@ -11,8 +11,11 @@ use std::vec::Vec;
 /// Value location range.
 #[derive(Debug, Clone, Copy)]
 pub struct ValueLocRange {
+    /// The ValueLoc containing a ValueLabel during this range.
     pub loc: ValueLoc,
+    /// The start of the range.
     pub start: u32,
+    /// The end of the range.
     pub end: u32,
 }
 

From c27b0a0c3e512b6a1ef07e828573f20d227668bf Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Tue, 14 May 2019 19:25:53 +0200
Subject: [PATCH 2432/3084] Add note about needing to call
 `func.dfg.collect_debug_info` for `set_val_label` to have any effect

---
 cranelift/frontend/src/frontend.rs | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index c161b85f50..407d03fec4 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -334,6 +334,8 @@ impl<'a> FunctionBuilder<'a> {
     }
 
     /// Set label for Value
+    ///
+    /// This will not do anything unless `func.dfg.collect_debug_info` is called first.
     pub fn set_val_label(&mut self, val: Value, label: ValueLabel) {
         if let Some(values_labels) = self.func.dfg.values_labels.as_mut() {
             use std::collections::hash_map::Entry;

From 91ec44acbfdd2523c254d7bc9b81a0a9efbaf741 Mon Sep 17 00:00:00 2001
From: Julian Seward 
Date: Thu, 2 May 2019 10:53:19 +0200
Subject: [PATCH 2433/3084] Add extensive test cases for integer
 division-by-constant magic number generation.

This adds test cases to ensure, to a reasonably high degree of certainty, that
the magic-number generators `magic_u32`, `magic_s32`, `magic_u64` and
`magic_s64` work correctly.  This is done by iterating through a large number
of `(n, d)` pairs, generating the magic numbers for `d`, interpreting the
magic numbers so as to perform the division, and comparing against the result
produced directly by the hardware.  The distribution of numbers is arranged so
that particular emphasis is given to corner cases -- the range ends and
midpoints -- but also so that there is at least some cover for values away
from those areas.  In total 50,148,000 tests are performed.
---
 .../codegen/src/divconst_magic_numbers.rs     | 435 ++++++++++++++++++
 1 file changed, 435 insertions(+)

diff --git a/cranelift/codegen/src/divconst_magic_numbers.rs b/cranelift/codegen/src/divconst_magic_numbers.rs
index 0e90fb967f..b155496ab9 100644
--- a/cranelift/codegen/src/divconst_magic_numbers.rs
+++ b/cranelift/codegen/src/divconst_magic_numbers.rs
@@ -647,4 +647,439 @@ mod tests {
         }
         assert_eq!(total, 563301797155560970);
     }
+
+    #[test]
+    fn test_magic_generators_give_correct_numbers() {
+        // For a variety of values for both `n` and `d`, compute the magic
+        // numbers for `d`, and in effect interpret them so as to compute
+        // `n / d`.  Check that that equals the value of `n / d` computed
+        // directly by the hardware.  This serves to check that the magic
+        // number generates work properly.  In total, 50,148,000 tests are
+        // done.
+
+        // Some constants
+        const MIN_U32: u32 = 0;
+        const MAX_U32: u32 = 0xFFFF_FFFFu32;
+        const MAX_U32_HALF: u32 = 0x8000_0000u32; // more or less
+
+        const MIN_S32: i32 = 0x8000_0000u32 as i32;
+        const MAX_S32: i32 = 0x7FFF_FFFFu32 as i32;
+
+        const MIN_U64: u64 = 0;
+        const MAX_U64: u64 = 0xFFFF_FFFF_FFFF_FFFFu64;
+        const MAX_U64_HALF: u64 = 0x8000_0000_0000_0000u64; // ditto
+
+        const MIN_S64: i64 = 0x8000_0000_0000_0000u64 as i64;
+        const MAX_S64: i64 = 0x7FFF_FFFF_FFFF_FFFFu64 as i64;
+
+        // These generate reference results for signed/unsigned 32/64 bit
+        // division, rounding towards zero.
+        fn div_u32(x: u32, y: u32) -> u32 {
+            return x / y;
+        }
+        fn div_s32(x: i32, y: i32) -> i32 {
+            return x / y;
+        }
+        fn div_u64(x: u64, y: u64) -> u64 {
+            return x / y;
+        }
+        fn div_s64(x: i64, y: i64) -> i64 {
+            return x / y;
+        }
+
+        // Returns the high half of a 32 bit unsigned widening multiply.
+        fn mulhw_u32(x: u32, y: u32) -> u32 {
+            let x64: u64 = x as u64;
+            let y64: u64 = y as u64;
+            let r64: u64 = x64 * y64;
+            (r64 >> 32) as u32
+        }
+
+        // Returns the high half of a 32 bit signed widening multiply.
+        fn mulhw_s32(x: i32, y: i32) -> i32 {
+            let x64: i64 = x as i64;
+            let y64: i64 = y as i64;
+            let r64: i64 = x64 * y64;
+            (r64 >> 32) as i32
+        }
+
+        // Returns the high half of a 64 bit unsigned widening multiply.
+        fn mulhw_u64(x: u64, y: u64) -> u64 {
+            let t0: u64 = x & 0xffffffffu64;
+            let t1: u64 = x >> 32;
+            let t2: u64 = y & 0xffffffffu64;
+            let t3: u64 = y >> 32;
+            let t4: u64 = t0 * t2;
+            let t5: u64 = t1 * t2 + (t4 >> 32);
+            let t6: u64 = t5 & 0xffffffffu64;
+            let t7: u64 = t5 >> 32;
+            let t8: u64 = t0 * t3 + t6;
+            let t9: u64 = t1 * t3 + t7 + (t8 >> 32);
+            t9
+        }
+
+        // Returns the high half of a 64 bit signed widening multiply.
+        fn mulhw_s64(x: i64, y: i64) -> i64 {
+            let t0: u64 = x as u64 & 0xffffffffu64;
+            let t1: i64 = x >> 32;
+            let t2: u64 = y as u64 & 0xffffffffu64;
+            let t3: i64 = y >> 32;
+            let t4: u64 = t0 * t2;
+            let t5: i64 = t1 * t2 as i64 + (t4 >> 32) as i64;
+            let t6: u64 = t5 as u64 & 0xffffffffu64;
+            let t7: i64 = t5 >> 32;
+            let t8: i64 = t0 as i64 * t3 + t6 as i64;
+            let t9: i64 = t1 * t3 + t7 + (t8 >> 32);
+            t9
+        }
+
+        // Compute the magic numbers for `d` and then use them to compute and
+        // check `n / d` for around 1000 values of `n`, using unsigned 32-bit
+        // division.
+        fn test_magic_u32_inner(d: u32, n_tests_done: &mut i32) {
+            // Advance the numerator (the `n` in `n / d`) so as to test
+            // densely near the range ends (and, in the signed variants, near
+            // zero) but not so densely away from those regions.
+            fn advance_n_u32(x: u32) -> u32 {
+                if x < MIN_U32 + 110 {
+                    return x + 1;
+                }
+                if x < MIN_U32 + 1700 {
+                    return x + 23;
+                }
+                if x < MAX_U32 - 1700 {
+                    let xd: f64 = (x as f64) * 1.06415927;
+                    return if xd >= (MAX_U32 - 1700) as f64 {
+                        MAX_U32 - 1700
+                    } else {
+                        xd as u32
+                    };
+                }
+                if x < MAX_U32 - 110 {
+                    return x + 23;
+                }
+                u32::wrapping_add(x, 1)
+            }
+
+            let magic: MU32 = magic_u32(d);
+            let mut n: u32 = MIN_U32;
+            loop {
+                *n_tests_done += 1;
+                // Compute and check `q = n / d` using `magic`.
+                let mut q: u32 = mulhw_u32(n, magic.mul_by);
+                if magic.do_add {
+                    assert!(magic.shift_by >= 1 && magic.shift_by <= 32);
+                    let mut t: u32 = n - q;
+                    t >>= 1;
+                    t = t + q;
+                    q = t >> (magic.shift_by - 1);
+                } else {
+                    assert!(magic.shift_by >= 0 && magic.shift_by <= 31);
+                    q >>= magic.shift_by;
+                }
+
+                assert_eq!(q, div_u32(n, d));
+
+                n = advance_n_u32(n);
+                if n == MIN_U32 {
+                    break;
+                }
+            }
+        }
+
+        // Compute the magic numbers for `d` and then use them to compute and
+        // check `n / d` for around 1000 values of `n`, using signed 32-bit
+        // division.
+        fn test_magic_s32_inner(d: i32, n_tests_done: &mut i32) {
+            // See comment on advance_n_u32 above.
+            fn advance_n_s32(x: i32) -> i32 {
+                if x >= 0 && x <= 29 {
+                    return x + 1;
+                }
+                if x < MIN_S32 + 110 {
+                    return x + 1;
+                }
+                if x < MIN_S32 + 1700 {
+                    return x + 23;
+                }
+                if x < MAX_S32 - 1700 {
+                    let mut xd: f64 = x as f64;
+                    xd = if xd < 0.0 {
+                        xd / 1.06415927
+                    } else {
+                        xd * 1.06415927
+                    };
+                    return if xd >= (MAX_S32 - 1700) as f64 {
+                        MAX_S32 - 1700
+                    } else {
+                        xd as i32
+                    };
+                }
+                if x < MAX_S32 - 110 {
+                    return x + 23;
+                }
+                if x == MAX_S32 {
+                    return MIN_S32;
+                }
+                x + 1
+            }
+
+            let magic: MS32 = magic_s32(d);
+            let mut n: i32 = MIN_S32;
+            loop {
+                *n_tests_done += 1;
+                // Compute and check `q = n / d` using `magic`.
+                let mut q: i32 = mulhw_s32(n, magic.mul_by);
+                if d > 0 && magic.mul_by < 0 {
+                    q = q + n;
+                } else if d < 0 && magic.mul_by > 0 {
+                    q = q - n;
+                }
+                assert!(magic.shift_by >= 0 && magic.shift_by <= 31);
+                q = q >> magic.shift_by;
+                let mut t: u32 = q as u32;
+                t = t >> 31;
+                q = q + (t as i32);
+
+                assert_eq!(q, div_s32(n, d));
+
+                n = advance_n_s32(n);
+                if n == MIN_S32 {
+                    break;
+                }
+            }
+        }
+
+        // Compute the magic numbers for `d` and then use them to compute and
+        // check `n / d` for around 1000 values of `n`, using unsigned 64-bit
+        // division.
+        fn test_magic_u64_inner(d: u64, n_tests_done: &mut i32) {
+            // See comment on advance_n_u32 above.
+            fn advance_n_u64(x: u64) -> u64 {
+                if x < MIN_U64 + 110 {
+                    return x + 1;
+                }
+                if x < MIN_U64 + 1700 {
+                    return x + 23;
+                }
+                if x < MAX_U64 - 1700 {
+                    let xd: f64 = (x as f64) * 1.06415927;
+                    return if xd >= (MAX_U64 - 1700) as f64 {
+                        MAX_U64 - 1700
+                    } else {
+                        xd as u64
+                    };
+                }
+                if x < MAX_U64 - 110 {
+                    return x + 23;
+                }
+                u64::wrapping_add(x, 1)
+            }
+
+            let magic: MU64 = magic_u64(d);
+            let mut n: u64 = MIN_U64;
+            loop {
+                *n_tests_done += 1;
+                // Compute and check `q = n / d` using `magic`.
+                let mut q = mulhw_u64(n, magic.mul_by);
+                if magic.do_add {
+                    assert!(magic.shift_by >= 1 && magic.shift_by <= 64);
+                    let mut t: u64 = n - q;
+                    t >>= 1;
+                    t = t + q;
+                    q = t >> (magic.shift_by - 1);
+                } else {
+                    assert!(magic.shift_by >= 0 && magic.shift_by <= 63);
+                    q >>= magic.shift_by;
+                }
+
+                assert_eq!(q, div_u64(n, d));
+
+                n = advance_n_u64(n);
+                if n == MIN_U64 {
+                    break;
+                }
+            }
+        }
+
+        // Compute the magic numbers for `d` and then use them to compute and
+        // check `n / d` for around 1000 values of `n`, using signed 64-bit
+        // division.
+        fn test_magic_s64_inner(d: i64, n_tests_done: &mut i32) {
+            // See comment on advance_n_u32 above.
+            fn advance_n_s64(x: i64) -> i64 {
+                if x >= 0 && x <= 29 {
+                    return x + 1;
+                }
+                if x < MIN_S64 + 110 {
+                    return x + 1;
+                }
+                if x < MIN_S64 + 1700 {
+                    return x + 23;
+                }
+                if x < MAX_S64 - 1700 {
+                    let mut xd: f64 = x as f64;
+                    xd = if xd < 0.0 {
+                        xd / 1.06415927
+                    } else {
+                        xd * 1.06415927
+                    };
+                    return if xd >= (MAX_S64 - 1700) as f64 {
+                        MAX_S64 - 1700
+                    } else {
+                        xd as i64
+                    };
+                }
+                if x < MAX_S64 - 110 {
+                    return x + 23;
+                }
+                if x == MAX_S64 {
+                    return MIN_S64;
+                }
+                x + 1
+            }
+
+            let magic: MS64 = magic_s64(d);
+            let mut n: i64 = MIN_S64;
+            loop {
+                *n_tests_done += 1;
+                // Compute and check `q = n / d` using `magic`. */
+                let mut q: i64 = mulhw_s64(n, magic.mul_by);
+                if d > 0 && magic.mul_by < 0 {
+                    q = q + n;
+                } else if d < 0 && magic.mul_by > 0 {
+                    q = q - n;
+                }
+                assert!(magic.shift_by >= 0 && magic.shift_by <= 63);
+                q = q >> magic.shift_by;
+                let mut t: u64 = q as u64;
+                t = t >> 63;
+                q = q + (t as i64);
+
+                assert_eq!(q, div_s64(n, d));
+
+                n = advance_n_s64(n);
+                if n == MIN_S64 {
+                    break;
+                }
+            }
+        }
+
+        // Using all the above support machinery, actually run the tests.
+
+        let mut n_tests_done: i32 = 0;
+
+        // u32 division tests
+        {
+            // 2 .. 3k
+            let mut d: u32 = 2;
+            for _ in 0..3 * 1000 {
+                test_magic_u32_inner(d, &mut n_tests_done);
+                d += 1;
+            }
+
+            // across the midpoint: midpoint - 3k .. midpoint + 3k
+            d = MAX_U32_HALF - 3 * 1000;
+            for _ in 0..2 * 3 * 1000 {
+                test_magic_u32_inner(d, &mut n_tests_done);
+                d += 1;
+            }
+
+            // MAX_U32 - 3k .. MAX_U32 (in reverse order)
+            d = MAX_U32;
+            for _ in 0..3 * 1000 {
+                test_magic_u32_inner(d, &mut n_tests_done);
+                d -= 1;
+            }
+        }
+
+        // s32 division tests
+        {
+            // MIN_S32 .. MIN_S32 + 3k
+            let mut d: i32 = MIN_S32;
+            for _ in 0..3 * 1000 {
+                test_magic_s32_inner(d, &mut n_tests_done);
+                d += 1;
+            }
+
+            // -3k .. -2 (in reverse order)
+            d = -2;
+            for _ in 0..3 * 1000 {
+                test_magic_s32_inner(d, &mut n_tests_done);
+                d -= 1;
+            }
+
+            // 2 .. 3k
+            d = 2;
+            for _ in 0..3 * 1000 {
+                test_magic_s32_inner(d, &mut n_tests_done);
+                d += 1;
+            }
+
+            // MAX_S32 - 3k .. MAX_S32 (in reverse order)
+            d = MAX_S32;
+            for _ in 0..3 * 1000 {
+                test_magic_s32_inner(d, &mut n_tests_done);
+                d -= 1;
+            }
+        }
+
+        // u64 division tests
+        {
+            // 2 .. 3k
+            let mut d: u64 = 2;
+            for _ in 0..3 * 1000 {
+                test_magic_u64_inner(d, &mut n_tests_done);
+                d += 1;
+            }
+
+            // across the midpoint: midpoint - 3k .. midpoint + 3k
+            d = MAX_U64_HALF - 3 * 1000;
+            for _ in 0..2 * 3 * 1000 {
+                test_magic_u64_inner(d, &mut n_tests_done);
+                d += 1;
+            }
+
+            // mAX_U64 - 3000 .. mAX_U64 (in reverse order)
+            d = MAX_U64;
+            for _ in 0..3 * 1000 {
+                test_magic_u64_inner(d, &mut n_tests_done);
+                d -= 1;
+            }
+        }
+
+        // s64 division tests
+        {
+            // MIN_S64 .. MIN_S64 + 3k
+            let mut d: i64 = MIN_S64;
+            for _ in 0..3 * 1000 {
+                test_magic_s64_inner(d, &mut n_tests_done);
+                d += 1;
+            }
+
+            // -3k .. -2 (in reverse order)
+            d = -2;
+            for _ in 0..3 * 1000 {
+                test_magic_s64_inner(d, &mut n_tests_done);
+                d -= 1;
+            }
+
+            // 2 .. 3k
+            d = 2;
+            for _ in 0..3 * 1000 {
+                test_magic_s64_inner(d, &mut n_tests_done);
+                d += 1;
+            }
+
+            // MAX_S64 - 3k .. MAX_S64 (in reverse order)
+            d = MAX_S64;
+            for _ in 0..3 * 1000 {
+                test_magic_s64_inner(d, &mut n_tests_done);
+                d -= 1;
+            }
+        }
+
+        assert_eq!(n_tests_done, 50_148_000);
+    }
+
 }

From 71a345e8139e7b68c92c81194c543fa09dc41ee0 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sun, 30 Sep 2018 16:18:23 +0200
Subject: [PATCH 2434/3084] Legalize fcvt_from_u/sint.f32/f64.i8/i16

---
 cranelift/codegen/meta-python/base/legalize.py  | 17 +++++++++++++++++
 .../isa/x86/legalize-fcvt_from_usint-i16.clif   | 14 ++++++++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/legalize-fcvt_from_usint-i16.clif

diff --git a/cranelift/codegen/meta-python/base/legalize.py b/cranelift/codegen/meta-python/base/legalize.py
index 423309dab7..96cb355461 100644
--- a/cranelift/codegen/meta-python/base/legalize.py
+++ b/cranelift/codegen/meta-python/base/legalize.py
@@ -261,6 +261,23 @@ for binop in [iadd_imm, imul_imm, udiv_imm, urem_imm]:
 for binop in [sdiv_imm, srem_imm]:
     widen_imm(True, binop)
 
+# Use expand instead of widen, because widen only gets applied for i8 and i16
+# base type vars, but these take f32 and f64 as base type var.
+for ty in [types.f32, types.f64]:
+    for int_ty in [types.i8, types.i16]:
+        expand.legalize(
+            a << insts.fcvt_from_uint.bind(ty).bind(int_ty)(b),
+            Rtl(
+                x << uextend.i32(b),
+                a << insts.fcvt_from_uint.bind(ty).i32(x),
+            ))
+        expand.legalize(
+            a << insts.fcvt_from_sint.bind(ty).bind(int_ty)(b),
+            Rtl(
+                x << sextend.i32(b),
+                a << insts.fcvt_from_sint.bind(ty).i32(x),
+            ))
+
 widen_imm(False, irsub_imm)
 
 # bit ops
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-fcvt_from_usint-i16.clif b/cranelift/filetests/filetests/isa/x86/legalize-fcvt_from_usint-i16.clif
new file mode 100644
index 0000000000..b7d4b80977
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/legalize-fcvt_from_usint-i16.clif
@@ -0,0 +1,14 @@
+test compile
+target x86_64
+
+function u0:0(i16) -> f64 fast {
+ebb0(v0: i16):
+    v1 = fcvt_from_uint.f64 v0
+    return v1
+}
+
+function u0:1(i16) -> f64 fast {
+ebb0(v0: i16):
+    v1 = fcvt_from_sint.f64 v0
+    return v1
+}

From 97ebaa6f37b6d581a64b2fd122010cbf50cb03f9 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 17 May 2019 17:16:52 +0200
Subject: [PATCH 2435/3084] Add Rust implementation and address review comments
 of #742;

---
 .../codegen/meta-python/base/legalize.py      | 37 ++++++++++---------
 cranelift/codegen/meta/src/cdsl/inst.rs       |  2 +-
 cranelift/codegen/meta/src/shared/legalize.rs | 28 ++++++++++++++
 3 files changed, 49 insertions(+), 18 deletions(-)

diff --git a/cranelift/codegen/meta-python/base/legalize.py b/cranelift/codegen/meta-python/base/legalize.py
index 96cb355461..3e287a7b4e 100644
--- a/cranelift/codegen/meta-python/base/legalize.py
+++ b/cranelift/codegen/meta-python/base/legalize.py
@@ -261,23 +261,6 @@ for binop in [iadd_imm, imul_imm, udiv_imm, urem_imm]:
 for binop in [sdiv_imm, srem_imm]:
     widen_imm(True, binop)
 
-# Use expand instead of widen, because widen only gets applied for i8 and i16
-# base type vars, but these take f32 and f64 as base type var.
-for ty in [types.f32, types.f64]:
-    for int_ty in [types.i8, types.i16]:
-        expand.legalize(
-            a << insts.fcvt_from_uint.bind(ty).bind(int_ty)(b),
-            Rtl(
-                x << uextend.i32(b),
-                a << insts.fcvt_from_uint.bind(ty).i32(x),
-            ))
-        expand.legalize(
-            a << insts.fcvt_from_sint.bind(ty).bind(int_ty)(b),
-            Rtl(
-                x << sextend.i32(b),
-                a << insts.fcvt_from_sint.bind(ty).i32(x),
-            ))
-
 widen_imm(False, irsub_imm)
 
 # bit ops
@@ -492,6 +475,26 @@ expand.legalize(
             b << bor(b1, b2)
         ))
 
+# Expansions for fcvt_from_{u,s}int for smaller integer types.
+# These use expand and not widen because the controlling type variable for
+# these instructions are f32/f64, which are legalized as part of the expand
+# group.
+for dest_ty in [types.f32, types.f64]:
+    for src_ty in [types.i8, types.i16]:
+        expand.legalize(
+            a << insts.fcvt_from_uint.bind(dest_ty).bind(src_ty)(b),
+            Rtl(
+                x << uextend.i32(b),
+                a << insts.fcvt_from_uint.bind(dest_ty).i32(x),
+            ))
+
+        expand.legalize(
+            a << insts.fcvt_from_sint.bind(dest_ty).bind(src_ty)(b),
+            Rtl(
+                x << sextend.i32(b),
+                a << insts.fcvt_from_sint.bind(dest_ty).i32(x),
+            ))
+
 # Expansions for immediate operands that are out of range.
 for inst_imm,      inst in [
         (iadd_imm, iadd),
diff --git a/cranelift/codegen/meta/src/cdsl/inst.rs b/cranelift/codegen/meta/src/cdsl/inst.rs
index d219419c7e..3cd722c260 100644
--- a/cranelift/codegen/meta/src/cdsl/inst.rs
+++ b/cranelift/codegen/meta/src/cdsl/inst.rs
@@ -373,7 +373,7 @@ fn verify_polymorphic(
             let tv = operands_in[op_num].type_var().unwrap();
             let free_typevar = tv.free_typevar();
             if (free_typevar.is_some() && tv == &free_typevar.unwrap())
-                || !tv.singleton_type().is_none()
+                || tv.singleton_type().is_some()
             {
                 match verify_ctrl_typevar(tv, &value_opnums, &operands_in, &operands_out) {
                     Ok(typevars) => {
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 172d6fd981..93819fc430 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -62,6 +62,8 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     let f32const = insts.by_name("f32const");
     let f64const = insts.by_name("f64const");
     let fcopysign = insts.by_name("fcopysign");
+    let fcvt_from_sint = insts.by_name("fcvt_from_sint");
+    let fcvt_from_uint = insts.by_name("fcvt_from_uint");
     let fneg = insts.by_name("fneg");
     let iadd = insts.by_name("iadd");
     let iadd_carry = insts.by_name("iadd_carry");
@@ -512,6 +514,32 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
         ],
     );
 
+    // Expansions for fcvt_from_{u,s}int for smaller integer types.
+    // These use expand and not widen because the controlling type variable for
+    // these instructions are f32/f64, which are legalized as part of the expand
+    // group.
+    for &dest_ty in &[F32, F64] {
+        for &src_ty in &[I8, I16] {
+            let bound_inst = bind(bind(fcvt_from_uint, dest_ty), src_ty);
+            expand.legalize(
+                def!(a = bound_inst(b)),
+                vec![
+                    def!(x = uextend.I32(b)),
+                    def!(a = fcvt_from_uint.dest_ty(x)),
+                ],
+            );
+
+            let bound_inst = bind(bind(fcvt_from_sint, dest_ty), src_ty);
+            expand.legalize(
+                def!(a = bound_inst(b)),
+                vec![
+                    def!(x = sextend.I32(b)),
+                    def!(a = fcvt_from_sint.dest_ty(x)),
+                ],
+            );
+        }
+    }
+
     // Expansions for immediate operands that are out of range.
     for &(inst_imm, inst) in &[
         (iadd_imm, iadd),

From 72b548756327f13d0e49c80e5cece970baf27e4f Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 17 May 2019 18:34:52 +0200
Subject: [PATCH 2436/3084] Add x86 encoding for rotr_imm and rotl_imm.

---
 .../codegen/meta-python/isa/x86/encodings.py  |  2 ++
 .../filetests/filetests/isa/x86/binary32.clif |  6 ++++
 .../filetests/filetests/isa/x86/binary64.clif | 11 ++++++
 .../filetests/isa/x86/legalize-rotate.clif    | 35 +++++++++++++++++++
 4 files changed, 54 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/legalize-rotate.clif

diff --git a/cranelift/codegen/meta-python/isa/x86/encodings.py b/cranelift/codegen/meta-python/isa/x86/encodings.py
index a032c11898..8b5f9cf0bc 100644
--- a/cranelift/codegen/meta-python/isa/x86/encodings.py
+++ b/cranelift/codegen/meta-python/isa/x86/encodings.py
@@ -233,6 +233,8 @@ for inst,           rrr in [
     X86_64.enc(inst.i32.any, *r.rc(0xd3, rrr=rrr))
 
 for inst,           rrr in [
+        (base.rotl_imm, 0),
+        (base.rotr_imm, 1),
         (base.ishl_imm, 4),
         (base.ushr_imm, 5),
         (base.sshr_imm, 7)]:
diff --git a/cranelift/filetests/filetests/isa/x86/binary32.clif b/cranelift/filetests/filetests/isa/x86/binary32.clif
index fec3479e41..e5af412d51 100644
--- a/cranelift/filetests/filetests/isa/x86/binary32.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary32.clif
@@ -432,6 +432,12 @@ ebb0:
     ; asm: shrl $8, %esi
     [-,%rsi]             v515 = ushr_imm v2, 8    ; bin: c1 ee 08
 
+    ; Rotate immediates
+    ; asm: rolq $12, %esi
+    [-,%rsi]             v5101 = rotl_imm v2, 12   ; bin: c1 c6 0c
+    ; asm: rorq $5, %esi
+    [-,%rsi]             v5103 = rotr_imm v2, 5    ; bin: c1 ce 05
+
     ; Load Complex
     [-,%rax]            v521 = iconst.i32 1
     [-,%rbx]            v522 = iconst.i32 1
diff --git a/cranelift/filetests/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
index e876cd59c0..c2a042ac01 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64.clif
@@ -602,6 +602,17 @@ ebb0:
     [-,%r8]              v520 = ushr_imm v4, 63   ; bin: 49 c1 e8 3f
 
 
+    ; Rotate immediates
+    ; asm: rolq $12, %rsi
+    [-,%rsi]             v5101 = rotl_imm v2, 12   ; bin: 48 c1 c6 0c
+    ; asm: rolq $13, %r8
+    [-,%r8]              v5102 = rotl_imm v4, 13   ; bin: 49 c1 c0 0d
+    ; asm: rorq $32, %rsi
+    [-,%rsi]             v5103 = rotr_imm v2, 32   ; bin: 48 c1 ce 20
+    ; asm: rorq $33, %r8
+    [-,%r8]              v5104 = rotr_imm v4, 33   ; bin: 49 c1 c8 21
+
+
     ; Load Complex
     [-,%rax]            v521 = iconst.i64 1
     [-,%rbx]            v522 = iconst.i64 1
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-rotate.clif b/cranelift/filetests/filetests/isa/x86/legalize-rotate.clif
new file mode 100644
index 0000000000..155b6001b4
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/legalize-rotate.clif
@@ -0,0 +1,35 @@
+test compile
+target x86_64
+
+; regex: V=v\d+
+; regex: R=%[a-z0-9]+
+
+function %i32_rotr(i32, i32) -> i32 fast {
+ebb0(v0: i32, v1: i32):
+    ; check: regmove v1, $R -> %rcx
+    ; check: v2 = rotr v0, v1
+    v2 = rotr v0, v1
+    return v2
+}
+
+function %i32_rotr_imm_1(i32) -> i32 fast {
+ebb0(v0: i32):
+    ; check: $V = rotr_imm v0, 1
+    v2 = rotr_imm v0, 1
+    return v2
+}
+
+function %i32_rotl(i32, i32) -> i32 fast {
+ebb0(v0: i32, v1: i32):
+    ; check: regmove v1, $R -> %rcx
+    ; check: v2 = rotl v0, v1
+    v2 = rotl v0, v1
+    return v2
+}
+
+function %i32_rotl_imm_1(i32) -> i32 fast {
+ebb0(v0: i32):
+    ; check: $V = rotl_imm v0, 1
+    v2 = rotl_imm v0, 1
+    return v2
+}

From 92109f664ca2a7c383c8f2c97d569eda03086316 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 21 May 2019 14:09:58 +0200
Subject: [PATCH 2437/3084] [meta] Move x86 settings generation to their own
 file;

---
 cranelift/codegen/meta/src/isa/x86/mod.rs     | 90 +------------------
 .../codegen/meta/src/isa/x86/settings.rs      | 87 ++++++++++++++++++
 2 files changed, 89 insertions(+), 88 deletions(-)
 create mode 100644 cranelift/codegen/meta/src/isa/x86/settings.rs

diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index 861553bf88..bf59b8d9c1 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -1,7 +1,6 @@
 use crate::cdsl::cpu_modes::CpuMode;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
-use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 
 use crate::shared::types::Bool::B1;
 use crate::shared::types::Float::{F32, F64};
@@ -10,92 +9,7 @@ use crate::shared::Definitions as SharedDefinitions;
 
 mod instructions;
 mod legalize;
-
-fn define_settings(shared: &SettingGroup) -> SettingGroup {
-    let mut settings = SettingGroupBuilder::new("x86");
-
-    // CPUID.01H:ECX
-    let has_sse3 = settings.add_bool("has_sse3", "SSE3: CPUID.01H:ECX.SSE3[bit 0]", false);
-    let has_ssse3 = settings.add_bool("has_ssse3", "SSSE3: CPUID.01H:ECX.SSSE3[bit 9]", false);
-    let has_sse41 = settings.add_bool("has_sse41", "SSE4.1: CPUID.01H:ECX.SSE4_1[bit 19]", false);
-    let has_sse42 = settings.add_bool("has_sse42", "SSE4.2: CPUID.01H:ECX.SSE4_2[bit 20]", false);
-    let has_popcnt = settings.add_bool("has_popcnt", "POPCNT: CPUID.01H:ECX.POPCNT[bit 23]", false);
-    settings.add_bool("has_avx", "AVX: CPUID.01H:ECX.AVX[bit 28]", false);
-
-    // CPUID.(EAX=07H, ECX=0H):EBX
-    let has_bmi1 = settings.add_bool(
-        "has_bmi1",
-        "BMI1: CPUID.(EAX=07H, ECX=0H):EBX.BMI1[bit 3]",
-        false,
-    );
-    let has_bmi2 = settings.add_bool(
-        "has_bmi2",
-        "BMI2: CPUID.(EAX=07H, ECX=0H):EBX.BMI2[bit 8]",
-        false,
-    );
-
-    // CPUID.EAX=80000001H:ECX
-    let has_lzcnt = settings.add_bool(
-        "has_lzcnt",
-        "LZCNT: CPUID.EAX=80000001H:ECX.LZCNT[bit 5]",
-        false,
-    );
-
-    settings.add_predicate("use_sse41", predicate!(has_sse41));
-    settings.add_predicate("use_sse42", predicate!(has_sse41 && has_sse42));
-    settings.add_predicate("use_popcnt", predicate!(has_popcnt && has_sse42));
-    settings.add_predicate("use_bmi1", predicate!(has_bmi1));
-    settings.add_predicate("use_lznct", predicate!(has_lzcnt));
-
-    // Some shared boolean values are used in x86 instruction predicates, so we need to group them
-    // in the same TargetIsa, for compabitibity with code generated by meta-python.
-    // TODO Once all the meta generation code has been migrated from Python to Rust, we can put it
-    // back in the shared SettingGroup, and use it in x86 instruction predicates.
-
-    let is_pic = shared.get_bool("is_pic");
-    let allones_funcaddrs = shared.get_bool("allones_funcaddrs");
-    settings.add_predicate("is_pic", predicate!(is_pic));
-    settings.add_predicate("not_is_pic", predicate!(!is_pic));
-    settings.add_predicate(
-        "all_ones_funcaddrs_and_not_is_pic",
-        predicate!(allones_funcaddrs && !is_pic),
-    );
-    settings.add_predicate(
-        "not_all_ones_funcaddrs_and_not_is_pic",
-        predicate!(!allones_funcaddrs && !is_pic),
-    );
-
-    // Presets corresponding to x86 CPUs.
-
-    settings.add_preset("baseline", preset!());
-    let nehalem = settings.add_preset(
-        "nehalem",
-        preset!(has_sse3 && has_ssse3 && has_sse41 && has_sse42 && has_popcnt),
-    );
-    let haswell = settings.add_preset(
-        "haswell",
-        preset!(nehalem && has_bmi1 && has_bmi2 && has_lzcnt),
-    );
-    let broadwell = settings.add_preset("broadwell", preset!(haswell));
-    let skylake = settings.add_preset("skylake", preset!(broadwell));
-    let cannonlake = settings.add_preset("cannonlake", preset!(skylake));
-    settings.add_preset("icelake", preset!(cannonlake));
-    settings.add_preset(
-        "znver1",
-        preset!(
-            has_sse3
-                && has_ssse3
-                && has_sse41
-                && has_sse42
-                && has_popcnt
-                && has_bmi1
-                && has_bmi2
-                && has_lzcnt
-        ),
-    );
-
-    settings.finish()
-}
+mod settings;
 
 fn define_registers() -> IsaRegs {
     let mut regs = IsaRegsBuilder::new();
@@ -139,7 +53,7 @@ fn define_registers() -> IsaRegs {
 }
 
 pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
-    let settings = define_settings(&shared_defs.settings);
+    let settings = settings::define(&shared_defs.settings);
     let regs = define_registers();
 
     let inst_group = instructions::define(&shared_defs.format_registry);
diff --git a/cranelift/codegen/meta/src/isa/x86/settings.rs b/cranelift/codegen/meta/src/isa/x86/settings.rs
new file mode 100644
index 0000000000..f9b1908d11
--- /dev/null
+++ b/cranelift/codegen/meta/src/isa/x86/settings.rs
@@ -0,0 +1,87 @@
+use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
+
+pub fn define(shared: &SettingGroup) -> SettingGroup {
+    let mut settings = SettingGroupBuilder::new("x86");
+
+    // CPUID.01H:ECX
+    let has_sse3 = settings.add_bool("has_sse3", "SSE3: CPUID.01H:ECX.SSE3[bit 0]", false);
+    let has_ssse3 = settings.add_bool("has_ssse3", "SSSE3: CPUID.01H:ECX.SSSE3[bit 9]", false);
+    let has_sse41 = settings.add_bool("has_sse41", "SSE4.1: CPUID.01H:ECX.SSE4_1[bit 19]", false);
+    let has_sse42 = settings.add_bool("has_sse42", "SSE4.2: CPUID.01H:ECX.SSE4_2[bit 20]", false);
+    let has_popcnt = settings.add_bool("has_popcnt", "POPCNT: CPUID.01H:ECX.POPCNT[bit 23]", false);
+    settings.add_bool("has_avx", "AVX: CPUID.01H:ECX.AVX[bit 28]", false);
+
+    // CPUID.(EAX=07H, ECX=0H):EBX
+    let has_bmi1 = settings.add_bool(
+        "has_bmi1",
+        "BMI1: CPUID.(EAX=07H, ECX=0H):EBX.BMI1[bit 3]",
+        false,
+    );
+    let has_bmi2 = settings.add_bool(
+        "has_bmi2",
+        "BMI2: CPUID.(EAX=07H, ECX=0H):EBX.BMI2[bit 8]",
+        false,
+    );
+
+    // CPUID.EAX=80000001H:ECX
+    let has_lzcnt = settings.add_bool(
+        "has_lzcnt",
+        "LZCNT: CPUID.EAX=80000001H:ECX.LZCNT[bit 5]",
+        false,
+    );
+
+    settings.add_predicate("use_sse41", predicate!(has_sse41));
+    settings.add_predicate("use_sse42", predicate!(has_sse41 && has_sse42));
+    settings.add_predicate("use_popcnt", predicate!(has_popcnt && has_sse42));
+    settings.add_predicate("use_bmi1", predicate!(has_bmi1));
+    settings.add_predicate("use_lznct", predicate!(has_lzcnt));
+
+    // Some shared boolean values are used in x86 instruction predicates, so we need to group them
+    // in the same TargetIsa, for compabitibity with code generated by meta-python.
+    // TODO Once all the meta generation code has been migrated from Python to Rust, we can put it
+    // back in the shared SettingGroup, and use it in x86 instruction predicates.
+
+    let is_pic = shared.get_bool("is_pic");
+    let allones_funcaddrs = shared.get_bool("allones_funcaddrs");
+    settings.add_predicate("is_pic", predicate!(is_pic));
+    settings.add_predicate("not_is_pic", predicate!(!is_pic));
+    settings.add_predicate(
+        "all_ones_funcaddrs_and_not_is_pic",
+        predicate!(allones_funcaddrs && !is_pic),
+    );
+    settings.add_predicate(
+        "not_all_ones_funcaddrs_and_not_is_pic",
+        predicate!(!allones_funcaddrs && !is_pic),
+    );
+
+    // Presets corresponding to x86 CPUs.
+
+    settings.add_preset("baseline", preset!());
+    let nehalem = settings.add_preset(
+        "nehalem",
+        preset!(has_sse3 && has_ssse3 && has_sse41 && has_sse42 && has_popcnt),
+    );
+    let haswell = settings.add_preset(
+        "haswell",
+        preset!(nehalem && has_bmi1 && has_bmi2 && has_lzcnt),
+    );
+    let broadwell = settings.add_preset("broadwell", preset!(haswell));
+    let skylake = settings.add_preset("skylake", preset!(broadwell));
+    let cannonlake = settings.add_preset("cannonlake", preset!(skylake));
+    settings.add_preset("icelake", preset!(cannonlake));
+    settings.add_preset(
+        "znver1",
+        preset!(
+            has_sse3
+                && has_ssse3
+                && has_sse41
+                && has_sse42
+                && has_popcnt
+                && has_bmi1
+                && has_bmi2
+                && has_lzcnt
+        ),
+    );
+
+    settings.finish()
+}

From cbbb7a220e1255196ec1451ffa85dede50236504 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 21 May 2019 14:11:48 +0200
Subject: [PATCH 2438/3084] [meta] Move x86 registers generation to their own
 file;

---
 cranelift/codegen/meta/src/isa/x86/mod.rs     | 45 +------------------
 .../codegen/meta/src/isa/x86/registers.rs     | 42 +++++++++++++++++
 2 files changed, 44 insertions(+), 43 deletions(-)
 create mode 100644 cranelift/codegen/meta/src/isa/x86/registers.rs

diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index bf59b8d9c1..dba5737062 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -1,6 +1,5 @@
 use crate::cdsl::cpu_modes::CpuMode;
 use crate::cdsl::isa::TargetIsa;
-use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 
 use crate::shared::types::Bool::B1;
 use crate::shared::types::Float::{F32, F64};
@@ -9,52 +8,12 @@ use crate::shared::Definitions as SharedDefinitions;
 
 mod instructions;
 mod legalize;
+mod registers;
 mod settings;
 
-fn define_registers() -> IsaRegs {
-    let mut regs = IsaRegsBuilder::new();
-
-    let builder = RegBankBuilder::new("IntRegs", "r")
-        .units(16)
-        .names(vec!["rax", "rcx", "rdx", "rbx", "rsp", "rbp", "rsi", "rdi"])
-        .track_pressure(true);
-    let int_regs = regs.add_bank(builder);
-
-    let builder = RegBankBuilder::new("FloatRegs", "xmm")
-        .units(16)
-        .track_pressure(true);
-    let float_regs = regs.add_bank(builder);
-
-    let builder = RegBankBuilder::new("FlagRegs", "")
-        .units(1)
-        .names(vec!["rflags"])
-        .track_pressure(false);
-    let flag_reg = regs.add_bank(builder);
-
-    let builder = RegClassBuilder::new_toplevel("GPR", int_regs);
-    let gpr = regs.add_class(builder);
-
-    let builder = RegClassBuilder::new_toplevel("FPR", float_regs);
-    let fpr = regs.add_class(builder);
-
-    let builder = RegClassBuilder::new_toplevel("FLAG", flag_reg);
-    regs.add_class(builder);
-
-    let builder = RegClassBuilder::subclass_of("GPR8", gpr, 0, 8);
-    let gpr8 = regs.add_class(builder);
-
-    let builder = RegClassBuilder::subclass_of("ABCD", gpr8, 0, 4);
-    regs.add_class(builder);
-
-    let builder = RegClassBuilder::subclass_of("FPR8", fpr, 0, 8);
-    regs.add_class(builder);
-
-    regs.finish()
-}
-
 pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let settings = settings::define(&shared_defs.settings);
-    let regs = define_registers();
+    let regs = registers::define();
 
     let inst_group = instructions::define(&shared_defs.format_registry);
     legalize::define(shared_defs, &inst_group);
diff --git a/cranelift/codegen/meta/src/isa/x86/registers.rs b/cranelift/codegen/meta/src/isa/x86/registers.rs
new file mode 100644
index 0000000000..350a5d904e
--- /dev/null
+++ b/cranelift/codegen/meta/src/isa/x86/registers.rs
@@ -0,0 +1,42 @@
+use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
+
+pub fn define() -> IsaRegs {
+    let mut regs = IsaRegsBuilder::new();
+
+    let builder = RegBankBuilder::new("IntRegs", "r")
+        .units(16)
+        .names(vec!["rax", "rcx", "rdx", "rbx", "rsp", "rbp", "rsi", "rdi"])
+        .track_pressure(true);
+    let int_regs = regs.add_bank(builder);
+
+    let builder = RegBankBuilder::new("FloatRegs", "xmm")
+        .units(16)
+        .track_pressure(true);
+    let float_regs = regs.add_bank(builder);
+
+    let builder = RegBankBuilder::new("FlagRegs", "")
+        .units(1)
+        .names(vec!["rflags"])
+        .track_pressure(false);
+    let flag_reg = regs.add_bank(builder);
+
+    let builder = RegClassBuilder::new_toplevel("GPR", int_regs);
+    let gpr = regs.add_class(builder);
+
+    let builder = RegClassBuilder::new_toplevel("FPR", float_regs);
+    let fpr = regs.add_class(builder);
+
+    let builder = RegClassBuilder::new_toplevel("FLAG", flag_reg);
+    regs.add_class(builder);
+
+    let builder = RegClassBuilder::subclass_of("GPR8", gpr, 0, 8);
+    let gpr8 = regs.add_class(builder);
+
+    let builder = RegClassBuilder::subclass_of("ABCD", gpr8, 0, 4);
+    regs.add_class(builder);
+
+    let builder = RegClassBuilder::subclass_of("FPR8", fpr, 0, 8);
+    regs.add_class(builder);
+
+    regs.finish()
+}

From a46b2d71732bcbed7e695f5af662041cd093a8f9 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 23 May 2019 11:51:19 +0200
Subject: [PATCH 2439/3084] [meta] Move ApplyTarget/bind to cdsl/inst;

---
 cranelift/codegen/meta/src/cdsl/ast.rs        | 60 +------------------
 cranelift/codegen/meta/src/cdsl/inst.rs       | 58 +++++++++++++++++-
 .../codegen/meta/src/isa/x86/legalize.rs      |  4 +-
 cranelift/codegen/meta/src/shared/legalize.rs |  4 +-
 4 files changed, 63 insertions(+), 63 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 1df8ea3050..46214e2abf 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -1,7 +1,7 @@
 use crate::cdsl::formats::FormatRegistry;
-use crate::cdsl::inst::{BoundInstruction, Instruction, InstructionPredicate};
+use crate::cdsl::inst::{ApplyTarget, Instruction, InstructionPredicate};
 use crate::cdsl::operands::{OperandKind, OperandKindFields};
-use crate::cdsl::types::{LaneType, ValueType};
+use crate::cdsl::types::ValueType;
 use crate::cdsl::typevar::{TypeSetBuilder, TypeVar};
 
 use cranelift_entity::{entity_impl, PrimaryMap};
@@ -364,62 +364,6 @@ impl VarPool {
     }
 }
 
-pub enum ApplyTarget {
-    Inst(Instruction),
-    Bound(BoundInstruction),
-}
-
-impl ApplyTarget {
-    pub fn inst(&self) -> &Instruction {
-        match &self {
-            ApplyTarget::Inst(inst) => inst,
-            ApplyTarget::Bound(bound_inst) => &bound_inst.inst,
-        }
-    }
-}
-
-impl Into for &Instruction {
-    fn into(self) -> ApplyTarget {
-        ApplyTarget::Inst(self.clone())
-    }
-}
-
-impl Into for BoundInstruction {
-    fn into(self) -> ApplyTarget {
-        ApplyTarget::Bound(self)
-    }
-}
-
-pub fn bind(target: impl Into, lane_type: impl Into) -> BoundInstruction {
-    let value_type = ValueType::from(lane_type.into());
-
-    let (inst, value_types) = match target.into() {
-        ApplyTarget::Inst(inst) => (inst, vec![value_type]),
-        ApplyTarget::Bound(bound_inst) => {
-            let mut new_value_types = bound_inst.value_types;
-            new_value_types.push(value_type);
-            (bound_inst.inst, new_value_types)
-        }
-    };
-
-    match &inst.polymorphic_info {
-        Some(poly) => {
-            assert!(
-                value_types.len() <= 1 + poly.other_typevars.len(),
-                format!("trying to bind too many types for {}", inst.name)
-            );
-        }
-        None => {
-            panic!(format!(
-                "trying to bind a type for {} which is not a polymorphic instruction",
-                inst.name
-            ));
-        }
-    }
-
-    BoundInstruction { inst, value_types }
-}
-
 /// Apply an instruction to arguments.
 ///
 /// An `Apply` AST expression is created by using function call syntax on instructions. This
diff --git a/cranelift/codegen/meta/src/cdsl/inst.rs b/cranelift/codegen/meta/src/cdsl/inst.rs
index 3cd722c260..6b9bfe60a1 100644
--- a/cranelift/codegen/meta/src/cdsl/inst.rs
+++ b/cranelift/codegen/meta/src/cdsl/inst.rs
@@ -4,7 +4,7 @@ use crate::cdsl::formats::{
 };
 use crate::cdsl::operands::Operand;
 use crate::cdsl::type_inference::Constraint;
-use crate::cdsl::types::ValueType;
+use crate::cdsl::types::{LaneType, ValueType};
 use crate::cdsl::typevar::TypeVar;
 
 use std::fmt;
@@ -598,3 +598,59 @@ impl InstructionPredicate {
         }
     }
 }
+
+pub enum ApplyTarget {
+    Inst(Instruction),
+    Bound(BoundInstruction),
+}
+
+impl ApplyTarget {
+    pub fn inst(&self) -> &Instruction {
+        match &self {
+            ApplyTarget::Inst(inst) => inst,
+            ApplyTarget::Bound(bound_inst) => &bound_inst.inst,
+        }
+    }
+}
+
+impl Into for &Instruction {
+    fn into(self) -> ApplyTarget {
+        ApplyTarget::Inst(self.clone())
+    }
+}
+
+impl Into for BoundInstruction {
+    fn into(self) -> ApplyTarget {
+        ApplyTarget::Bound(self)
+    }
+}
+
+pub fn bind(target: impl Into, lane_type: impl Into) -> BoundInstruction {
+    let value_type = ValueType::from(lane_type.into());
+
+    let (inst, value_types) = match target.into() {
+        ApplyTarget::Inst(inst) => (inst, vec![value_type]),
+        ApplyTarget::Bound(bound_inst) => {
+            let mut new_value_types = bound_inst.value_types;
+            new_value_types.push(value_type);
+            (bound_inst.inst, new_value_types)
+        }
+    };
+
+    match &inst.polymorphic_info {
+        Some(poly) => {
+            assert!(
+                value_types.len() <= 1 + poly.other_typevars.len(),
+                format!("trying to bind too many types for {}", inst.name)
+            );
+        }
+        None => {
+            panic!(format!(
+                "trying to bind a type for {} which is not a polymorphic instruction",
+                inst.name
+            ));
+        }
+    }
+
+    BoundInstruction { inst, value_types }
+}
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 7a0cd8379c..853dff2c17 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -1,5 +1,5 @@
-use crate::cdsl::ast::{bind, var, ExprBuilder, Literal};
-use crate::cdsl::inst::InstructionGroup;
+use crate::cdsl::ast::{var, ExprBuilder, Literal};
+use crate::cdsl::inst::{bind, InstructionGroup};
 use crate::cdsl::xform::TransformGroupBuilder;
 
 use crate::shared::types::Int::{I32, I64};
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 93819fc430..435e28187a 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -1,5 +1,5 @@
-use crate::cdsl::ast::{bind, var, ExprBuilder, Literal};
-use crate::cdsl::inst::{Instruction, InstructionGroup};
+use crate::cdsl::ast::{var, ExprBuilder, Literal};
+use crate::cdsl::inst::{bind, Instruction, InstructionGroup};
 use crate::cdsl::xform::{TransformGroupBuilder, TransformGroups};
 
 use crate::shared::OperandKinds;

From 724d1cd2a1fddd489bc3c5949f44140ee23f5e56 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 23 May 2019 11:53:06 +0200
Subject: [PATCH 2440/3084] [meta] Rename ApplyTarget to InstSpec;

---
 cranelift/codegen/meta/src/cdsl/ast.rs  | 12 +++++------
 cranelift/codegen/meta/src/cdsl/inst.rs | 27 +++++++++++++------------
 2 files changed, 20 insertions(+), 19 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 46214e2abf..37892fc896 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::formats::FormatRegistry;
-use crate::cdsl::inst::{ApplyTarget, Instruction, InstructionPredicate};
+use crate::cdsl::inst::{InstSpec, Instruction, InstructionPredicate};
 use crate::cdsl::operands::{OperandKind, OperandKindFields};
 use crate::cdsl::types::ValueType;
 use crate::cdsl::typevar::{TypeSetBuilder, TypeVar};
@@ -375,10 +375,10 @@ pub struct Apply {
 }
 
 impl Apply {
-    pub fn new(target: ApplyTarget, args: Vec) -> Self {
+    pub fn new(target: InstSpec, args: Vec) -> Self {
         let (inst, value_types) = match target.into() {
-            ApplyTarget::Inst(inst) => (inst, Vec::new()),
-            ApplyTarget::Bound(bound_inst) => (bound_inst.inst, bound_inst.value_types),
+            InstSpec::Inst(inst) => (inst, Vec::new()),
+            InstSpec::Bound(bound_inst) => (bound_inst.inst, bound_inst.value_types),
         };
 
         // Basic check on number of arguments.
@@ -520,7 +520,7 @@ impl Apply {
 pub enum DummyExpr {
     Var(DummyVar),
     Literal(Literal),
-    Apply(ApplyTarget, Vec),
+    Apply(InstSpec, Vec),
 }
 
 #[derive(Clone)]
@@ -553,7 +553,7 @@ pub struct ExprBuilder {
 }
 
 impl ExprBuilder {
-    pub fn apply(inst: ApplyTarget, args: Vec) -> Self {
+    pub fn apply(inst: InstSpec, args: Vec) -> Self {
         let expr = DummyExpr::Apply(inst, args);
         Self { expr }
     }
diff --git a/cranelift/codegen/meta/src/cdsl/inst.rs b/cranelift/codegen/meta/src/cdsl/inst.rs
index 6b9bfe60a1..2dcd3e4d9a 100644
--- a/cranelift/codegen/meta/src/cdsl/inst.rs
+++ b/cranelift/codegen/meta/src/cdsl/inst.rs
@@ -599,38 +599,39 @@ impl InstructionPredicate {
     }
 }
 
-pub enum ApplyTarget {
+/// An instruction specification, containing an instruction that has bound types or not.
+pub enum InstSpec {
     Inst(Instruction),
     Bound(BoundInstruction),
 }
 
-impl ApplyTarget {
+impl InstSpec {
     pub fn inst(&self) -> &Instruction {
         match &self {
-            ApplyTarget::Inst(inst) => inst,
-            ApplyTarget::Bound(bound_inst) => &bound_inst.inst,
+            InstSpec::Inst(inst) => inst,
+            InstSpec::Bound(bound_inst) => &bound_inst.inst,
         }
     }
 }
 
-impl Into for &Instruction {
-    fn into(self) -> ApplyTarget {
-        ApplyTarget::Inst(self.clone())
+impl Into for &Instruction {
+    fn into(self) -> InstSpec {
+        InstSpec::Inst(self.clone())
     }
 }
 
-impl Into for BoundInstruction {
-    fn into(self) -> ApplyTarget {
-        ApplyTarget::Bound(self)
+impl Into for BoundInstruction {
+    fn into(self) -> InstSpec {
+        InstSpec::Bound(self)
     }
 }
 
-pub fn bind(target: impl Into, lane_type: impl Into) -> BoundInstruction {
+pub fn bind(target: impl Into, lane_type: impl Into) -> BoundInstruction {
     let value_type = ValueType::from(lane_type.into());
 
     let (inst, value_types) = match target.into() {
-        ApplyTarget::Inst(inst) => (inst, vec![value_type]),
-        ApplyTarget::Bound(bound_inst) => {
+        InstSpec::Inst(inst) => (inst, vec![value_type]),
+        InstSpec::Bound(bound_inst) => {
             let mut new_value_types = bound_inst.value_types;
             new_value_types.push(value_type);
             (bound_inst.inst, new_value_types)

From 6935033c9e02710664d0971918858d7b0d5d9f0b Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 23 May 2019 12:01:25 +0200
Subject: [PATCH 2441/3084] [meta] Have bind() be a method of
 {Bound,}Instruction instead of a static method;

---
 cranelift/codegen/meta/src/cdsl/ast.rs        |  2 +-
 cranelift/codegen/meta/src/cdsl/inst.rs       | 30 +++++++++++--------
 .../codegen/meta/src/isa/x86/legalize.rs      |  2 +-
 cranelift/codegen/meta/src/shared/legalize.rs |  8 ++---
 4 files changed, 23 insertions(+), 19 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 37892fc896..ccb9adec01 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -574,7 +574,7 @@ macro_rules! def_rhs {
 
     // inst.type(a, b, c)
     ($inst:ident.$type:ident($($src:expr),*)) => {
-        ExprBuilder::apply(bind($inst, $type).into(), vec![$($src.clone().into()),*])
+        ExprBuilder::apply($inst.bind($type).into(), vec![$($src.clone().into()),*])
     };
 }
 
diff --git a/cranelift/codegen/meta/src/cdsl/inst.rs b/cranelift/codegen/meta/src/cdsl/inst.rs
index 2dcd3e4d9a..b469e8b91f 100644
--- a/cranelift/codegen/meta/src/cdsl/inst.rs
+++ b/cranelift/codegen/meta/src/cdsl/inst.rs
@@ -143,6 +143,10 @@ impl Instruction {
             None => Vec::new(),
         }
     }
+
+    pub fn bind(&self, lane_type: impl Into) -> BoundInstruction {
+        bind(self.clone(), lane_type.into(), Vec::new())
+    }
 }
 
 impl fmt::Display for Instruction {
@@ -341,6 +345,12 @@ pub struct BoundInstruction {
     pub value_types: Vec,
 }
 
+impl BoundInstruction {
+    pub fn bind(self, lane_type: impl Into) -> BoundInstruction {
+        bind(self.inst, lane_type.into(), self.value_types)
+    }
+}
+
 /// Check if this instruction is polymorphic, and verify its use of type variables.
 fn verify_polymorphic(
     operands_in: &Vec,
@@ -626,18 +636,13 @@ impl Into for BoundInstruction {
     }
 }
 
-pub fn bind(target: impl Into, lane_type: impl Into) -> BoundInstruction {
-    let value_type = ValueType::from(lane_type.into());
-
-    let (inst, value_types) = match target.into() {
-        InstSpec::Inst(inst) => (inst, vec![value_type]),
-        InstSpec::Bound(bound_inst) => {
-            let mut new_value_types = bound_inst.value_types;
-            new_value_types.push(value_type);
-            (bound_inst.inst, new_value_types)
-        }
-    };
-
+/// Helper bind reused by {Bound,}Instruction::bind.
+fn bind(
+    inst: Instruction,
+    lane_type: LaneType,
+    mut value_types: Vec,
+) -> BoundInstruction {
+    value_types.push(ValueType::from(lane_type));
     match &inst.polymorphic_info {
         Some(poly) => {
             assert!(
@@ -652,6 +657,5 @@ pub fn bind(target: impl Into, lane_type: impl Into) -> Boun
             ));
         }
     }
-
     BoundInstruction { inst, value_types }
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 853dff2c17..b214c26914 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::ast::{var, ExprBuilder, Literal};
-use crate::cdsl::inst::{bind, InstructionGroup};
+use crate::cdsl::inst::InstructionGroup;
 use crate::cdsl::xform::TransformGroupBuilder;
 
 use crate::shared::types::Int::{I32, I64};
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 435e28187a..66ce180c1c 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::ast::{var, ExprBuilder, Literal};
-use crate::cdsl::inst::{bind, Instruction, InstructionGroup};
+use crate::cdsl::inst::{Instruction, InstructionGroup};
 use crate::cdsl::xform::{TransformGroupBuilder, TransformGroups};
 
 use crate::shared::OperandKinds;
@@ -352,7 +352,7 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     for &extend_op in &[uextend, sextend] {
         // The sign extension operators have two typevars: the result has one and controls the
         // instruction, then the input has one.
-        let bound = bind(bind(extend_op, I16), I8);
+        let bound = extend_op.bind(I16).bind(I8);
         widen.legalize(
             def!(a = bound(b)),
             vec![def!(c = extend_op.I32(b)), def!(a = ireduce(c))],
@@ -520,7 +520,7 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     // group.
     for &dest_ty in &[F32, F64] {
         for &src_ty in &[I8, I16] {
-            let bound_inst = bind(bind(fcvt_from_uint, dest_ty), src_ty);
+            let bound_inst = fcvt_from_uint.bind(dest_ty).bind(src_ty);
             expand.legalize(
                 def!(a = bound_inst(b)),
                 vec![
@@ -529,7 +529,7 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
                 ],
             );
 
-            let bound_inst = bind(bind(fcvt_from_sint, dest_ty), src_ty);
+            let bound_inst = fcvt_from_sint.bind(dest_ty).bind(src_ty);
             expand.legalize(
                 def!(a = bound_inst(b)),
                 vec![

From 03368895feebdfb219b61ae51908992dbf6cb045 Mon Sep 17 00:00:00 2001
From: Julian Seward 
Date: Mon, 27 May 2019 11:55:23 +0200
Subject: [PATCH 2442/3084] Cranelift: Redundant stack-slot-to-stack-slot copy
 removal.  PR#773.

This is also https://bugzilla.mozilla.org/show_bug.cgi?id=1552737.

Cranelift currently has a tendency to create redundant copies (self-copies) of
values from a stack slot back to the same stack slot.  This generates a
pointless load and store and an unnecessary register use.  The copies are
created by `visit_inst` in regalloc/reload.rs.  They appear to occur mostly,
but not exclusively, at loop heads.  It's unclear why this happens.

This patch adds a special case to `visit_inst` to find such copies.  They are
converted into a new instruction, `copy_nop`, which takes and produces the
same SSA names, so as not to break any of the SSA invariants, but which has a
zero-length encoding, hence removing the copy at emission time.

`copy_nop`s source and destination operands must be stack slots and of course
the *same* stack slot.  The verifier has been enhanced to check this, since
misuse of `copy_nop` will likely lead to hard-to-find incorrect-code bugs.

Attempts were made to write a standalone .clif test case.  But these failed
because it appears the .clif parser accepts but ignores location hints that
are stack slots.  So it's impossible to write, in clif, the exact form of
`copy` instruction that triggers the transformation.
---
 cranelift/codegen/meta-python/base/formats.py |  2 +
 .../codegen/meta-python/base/instructions.py  | 10 +++++
 .../codegen/meta-python/isa/x86/encodings.py  |  7 +++
 .../codegen/meta-python/isa/x86/recipes.py    |  3 ++
 .../codegen/meta/src/shared/instructions.rs   | 15 +++++++
 cranelift/codegen/src/regalloc/reload.rs      | 38 +++++++++++++++-
 cranelift/codegen/src/verifier/mod.rs         | 44 ++++++++++++++++++-
 7 files changed, 116 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/meta-python/base/formats.py b/cranelift/codegen/meta-python/base/formats.py
index b63863eed0..b45ac26f74 100644
--- a/cranelift/codegen/meta-python/base/formats.py
+++ b/cranelift/codegen/meta-python/base/formats.py
@@ -75,6 +75,8 @@ TableAddr = InstructionFormat(table, VALUE, offset32)
 
 RegMove = InstructionFormat(VALUE, ('src', regunit), ('dst', regunit))
 CopySpecial = InstructionFormat(('src', regunit), ('dst', regunit))
+CopyNop = InstructionFormat(
+        ('src', entities.stack_slot), ('dst', entities.stack_slot))
 RegSpill = InstructionFormat(
         VALUE, ('src', regunit), ('dst', entities.stack_slot))
 RegFill = InstructionFormat(
diff --git a/cranelift/codegen/meta-python/base/instructions.py b/cranelift/codegen/meta-python/base/instructions.py
index ebb7257f65..48adc399ce 100644
--- a/cranelift/codegen/meta-python/base/instructions.py
+++ b/cranelift/codegen/meta-python/base/instructions.py
@@ -764,6 +764,16 @@ copy_special = Instruction(
         ins=(src, dst),
         other_side_effects=True)
 
+copy_nop = Instruction(
+        'copy_nop', r"""
+        Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
+        into a no-op.  This instruction is for use only within Cranelift
+        itself.
+
+        This instruction copies its input, preserving the value type.
+        """,
+        ins=x, outs=a)
+
 delta = Operand('delta', Int)
 adjust_sp_down = Instruction(
     'adjust_sp_down', r"""
diff --git a/cranelift/codegen/meta-python/isa/x86/encodings.py b/cranelift/codegen/meta-python/isa/x86/encodings.py
index 8b5f9cf0bc..cf25d5b2dd 100644
--- a/cranelift/codegen/meta-python/isa/x86/encodings.py
+++ b/cranelift/codegen/meta-python/isa/x86/encodings.py
@@ -341,6 +341,13 @@ enc_x86_64(x86.pop.i64, r.popq, 0x58)
 X86_64.enc(base.copy_special, *r.copysp.rex(0x89, w=1))
 X86_32.enc(base.copy_special, *r.copysp(0x89))
 
+# Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
+# into a no-op.
+X86_64.enc(base.copy_nop.i64, r.stacknull, 0)
+X86_64.enc(base.copy_nop.i32, r.stacknull, 0)
+X86_64.enc(base.copy_nop.f64, r.stacknull, 0)
+X86_64.enc(base.copy_nop.f32, r.stacknull, 0)
+
 # Adjust SP down by a dynamic value (or up, with a negative operand).
 X86_32.enc(base.adjust_sp_down.i32, *r.adjustsp(0x29))
 X86_64.enc(base.adjust_sp_down.i64, *r.adjustsp.rex(0x29, w=1))
diff --git a/cranelift/codegen/meta-python/isa/x86/recipes.py b/cranelift/codegen/meta-python/isa/x86/recipes.py
index 76544b0e28..c596fcd108 100644
--- a/cranelift/codegen/meta-python/isa/x86/recipes.py
+++ b/cranelift/codegen/meta-python/isa/x86/recipes.py
@@ -295,6 +295,9 @@ def valid_scale(iform):
 # copies and no-op conversions.
 null = EncRecipe('null', Unary, base_size=0, ins=GPR, outs=0, emit='')
 
+stacknull = EncRecipe('stacknull', Unary, base_size=0, ins=StackGPR32,
+                      outs=StackGPR32, emit='')
+
 debugtrap = EncRecipe('debugtrap', NullAry, base_size=1, ins=(), outs=(),
                       emit='''
                       sink.put1(0xcc);
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index b0c89b510e..891dadd00a 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1254,6 +1254,21 @@ pub fn define(
         .finish(format_registry),
     );
 
+    ig.push(
+        Inst::new(
+            "copy_nop",
+            r#"
+        Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
+        into a no-op.  This instruction is for use only within Cranelift itself.
+
+        This instruction copies its input, preserving the value type.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .finish(format_registry),
+    );
+
     let delta = &operand("delta", Int);
 
     ig.push(
diff --git a/cranelift/codegen/src/regalloc/reload.rs b/cranelift/codegen/src/regalloc/reload.rs
index efbe2204bd..cc6aae59b8 100644
--- a/cranelift/codegen/src/regalloc/reload.rs
+++ b/cranelift/codegen/src/regalloc/reload.rs
@@ -13,7 +13,7 @@ use crate::cursor::{Cursor, EncCursor};
 use crate::dominator_tree::DominatorTree;
 use crate::entity::{SparseMap, SparseMapValue};
 use crate::ir::{AbiParam, ArgumentLoc, InstBuilder};
-use crate::ir::{Ebb, Function, Inst, InstructionData, Opcode, Value};
+use crate::ir::{Ebb, Function, Inst, InstructionData, Opcode, Value, ValueLoc};
 use crate::isa::RegClass;
 use crate::isa::{ConstraintKind, EncInfo, Encoding, RecipeConstraints, TargetIsa};
 use crate::regalloc::affinity::Affinity;
@@ -211,6 +211,42 @@ impl<'a> Context<'a> {
         debug_assert!(self.candidates.is_empty());
         self.find_candidates(inst, constraints);
 
+        // If we find a copy from a stack slot to the same stack slot, replace
+        // it with a `copy_nop` but otherwise ignore it.  In particular, don't
+        // generate a reload immediately followed by a spill.  The `copy_nop`
+        // has a zero-length encoding, so will disappear at emission time.
+        if let InstructionData::Unary {
+            opcode: Opcode::Copy,
+            arg,
+        } = self.cur.func.dfg[inst]
+        {
+            let dst_vals = self.cur.func.dfg.inst_results(inst);
+            if dst_vals.len() == 1 {
+                let can_transform = match (
+                    self.cur.func.locations[arg],
+                    self.cur.func.locations[dst_vals[0]],
+                ) {
+                    (ValueLoc::Stack(src_slot), ValueLoc::Stack(dst_slot)) => src_slot == dst_slot,
+                    _ => false,
+                };
+                if can_transform {
+                    // Convert the instruction into a `copy_nop`.
+                    self.cur.func.dfg.replace(inst).copy_nop(arg);
+                    let ok = self.cur.func.update_encoding(inst, self.cur.isa).is_ok();
+                    debug_assert!(ok);
+
+                    // And move on to the next insn.
+                    self.reloads.clear();
+                    let _ = tracker.process_inst(inst, &self.cur.func.dfg, self.liveness);
+                    self.cur.next_inst();
+                    self.candidates.clear();
+                    return;
+                }
+            }
+        }
+
+        // Deal with all instructions not special-cased by the immediately
+        // preceding fragment.
         if let InstructionData::Unary {
             opcode: Opcode::Copy,
             ..
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index fd72315735..38a52f0934 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -65,8 +65,8 @@ use crate::ir;
 use crate::ir::entities::AnyEntity;
 use crate::ir::instructions::{BranchInfo, CallInfo, InstructionFormat, ResolvedConstraint};
 use crate::ir::{
-    types, ArgumentLoc, Ebb, FuncRef, Function, GlobalValue, Inst, JumpTable, Opcode, SigRef,
-    StackSlot, StackSlotKind, Type, Value, ValueDef, ValueList, ValueLoc,
+    types, ArgumentLoc, Ebb, FuncRef, Function, GlobalValue, Inst, InstructionData, JumpTable,
+    Opcode, SigRef, StackSlot, StackSlotKind, Type, Value, ValueDef, ValueList, ValueLoc,
 };
 use crate::isa::TargetIsa;
 use crate::iterators::IteratorExtras;
@@ -1079,6 +1079,9 @@ impl<'a> Verifier<'a> {
         self.typecheck_return(inst, errors).is_ok();
         self.typecheck_special(inst, ctrl_type, errors).is_ok();
 
+        // Misuses of copy_nop instructions are fatal
+        self.typecheck_copy_nop(inst, errors)?;
+
         Ok(())
     }
 
@@ -1469,6 +1472,43 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
+    fn typecheck_copy_nop(
+        &self,
+        inst: Inst,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
+        if let InstructionData::Unary {
+            opcode: Opcode::CopyNop,
+            arg,
+        } = self.func.dfg[inst]
+        {
+            let dst_vals = self.func.dfg.inst_results(inst);
+            if dst_vals.len() != 1 {
+                return fatal!(errors, inst, "copy_nop must produce exactly one result");
+            }
+            let dst_val = dst_vals[0];
+            if self.func.dfg.value_type(dst_val) != self.func.dfg.value_type(arg) {
+                return fatal!(errors, inst, "copy_nop src and dst types must be the same");
+            }
+            let src_loc = self.func.locations[arg];
+            let dst_loc = self.func.locations[dst_val];
+            let locs_ok = match (src_loc, dst_loc) {
+                (ValueLoc::Stack(src_slot), ValueLoc::Stack(dst_slot)) => src_slot == dst_slot,
+                _ => false,
+            };
+            if !locs_ok {
+                return fatal!(
+                    errors,
+                    inst,
+                    "copy_nop must refer to identical stack slots, but found {:?} vs {:?}",
+                    src_loc,
+                    dst_loc
+                );
+            }
+        }
+        Ok(())
+    }
+
     fn cfg_integrity(
         &self,
         cfg: &ControlFlowGraph,

From f427a2b923844260ecdabdac4b3baf0ec8ce3870 Mon Sep 17 00:00:00 2001
From: Takanori Ishibashi 
Date: Tue, 28 May 2019 20:03:10 +0900
Subject: [PATCH 2443/3084] Fix url in comments

---
 cranelift/wasm/src/func_translator.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index 578c12ef3f..6842e4f92f 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -45,7 +45,7 @@ impl FuncTranslator {
     ///
     /// See [the WebAssembly specification][wasm].
     ///
-    /// [wasm]: https://webassembly.github.io/spec/binary/modules.html#code-section
+    /// [wasm]: https://webassembly.github.io/spec/core/binary/modules.html#code-section
     ///
     /// The Cranelift IR function `func` should be completely empty except for the `func.signature`
     /// and `func.name` fields. The signature may contain special-purpose arguments which are not

From 6059936113909603bebcd38d7236493cf7c21d2b Mon Sep 17 00:00:00 2001
From: iximeow 
Date: Fri, 17 May 2019 12:28:46 -0700
Subject: [PATCH 2444/3084] remove rex-prefixed recipes for e9 and eb jumps

while not incorrect, the prefix has no additional semantics on these
  instructions other than taking an extra byte for the jump
---
 .../codegen/meta-python/isa/x86/encodings.py  |  6 ++--
 .../filetests/isa/x86/legalize-br-icmp.clif   |  4 +--
 .../filetests/isa/x86/relax_branch.clif       |  2 +-
 .../filetests/regalloc/coloring-227.clif      | 30 +++++++++----------
 4 files changed, 22 insertions(+), 20 deletions(-)

diff --git a/cranelift/codegen/meta-python/isa/x86/encodings.py b/cranelift/codegen/meta-python/isa/x86/encodings.py
index cf25d5b2dd..449e8b8e1b 100644
--- a/cranelift/codegen/meta-python/isa/x86/encodings.py
+++ b/cranelift/codegen/meta-python/isa/x86/encodings.py
@@ -490,8 +490,10 @@ X86_64.enc(base.x_return, *r.ret(0xc3))
 #
 # Branches
 #
-enc_both(base.jump, r.jmpb, 0xeb)
-enc_both(base.jump, r.jmpd, 0xe9)
+X86_32.enc(base.jump, *r.jmpb(0xeb))
+X86_64.enc(base.jump, *r.jmpb(0xeb))
+X86_32.enc(base.jump, *r.jmpd(0xe9))
+X86_64.enc(base.jump, *r.jmpd(0xe9))
 
 enc_both(base.brif, r.brib, 0x70)
 enc_both(base.brif, r.brid, 0x0f, 0x80)
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif
index e4694c36b1..e7abc4a273 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif
@@ -17,7 +17,7 @@ ebb1:
 ; nextln: [RexOp1pu_id#b8]                    v1 = iconst.i64 0
 ; nextln: [RexOp1icscc#8039]                  v2 = icmp eq v0, v1
 ; nextln: [RexOp1t8jccb#75]                   brnz v2, ebb1
-; nextln: [RexOp1jmpb#eb]                     jump ebb1
+; nextln: [Op1jmpb#eb]                        jump ebb1
 ; nextln: 
 ; nextln:                                 ebb1:
 ; nextln: [Op1ret#c3]                         return
@@ -39,7 +39,7 @@ ebb1(v2: i64):
 ; nextln: [RexOp1pu_id#b8]                    v1 = iconst.i64 0
 ; nextln: [RexOp1icscc#8039]                  v3 = icmp eq v0, v1
 ; nextln: [RexOp1t8jccb#75]                   brnz v3, ebb1(v0)
-; nextln: [RexOp1jmpb#eb]                     jump ebb1(v0)
+; nextln: [Op1jmpb#eb]                        jump ebb1(v0)
 ; nextln: 
 ; nextln:                                 ebb1(v2: i64):
 ; nextln: [Op1ret#c3]                         return
diff --git a/cranelift/filetests/filetests/isa/x86/relax_branch.clif b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
index 00a5705643..c735224c0a 100644
--- a/cranelift/filetests/filetests/isa/x86/relax_branch.clif
+++ b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
@@ -103,7 +103,7 @@ function u0:2691(i32 [%rdi], i32 [%rsi], i64 vmctx [%r14]) -> i64 uext [%rax] ba
 [Op1umr#89,%rdx]                    v100 = copy v79
 @00a4 [RexOp1rmov#89]               regmove v100, %rdx -> %rdi
 @00a4 [RexOp1rmov#89]               regmove v99, %rcx -> %rsi
-@00a4 [RexOp1jmpd#e9]               jump ebb3(v100, v99); bin: 40 e9 ffffff2c
+@00a4 [Op1jmpd#e9]                  jump ebb3(v100, v99); bin: e9 ffffff2d
 
                                 ebb9:
 @00a7 [-]                           fallthrough ebb4
diff --git a/cranelift/filetests/filetests/regalloc/coloring-227.clif b/cranelift/filetests/filetests/regalloc/coloring-227.clif
index 41d6817ffd..7f14ca4e25 100644
--- a/cranelift/filetests/filetests/regalloc/coloring-227.clif
+++ b/cranelift/filetests/filetests/regalloc/coloring-227.clif
@@ -9,10 +9,10 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 [RexOp1pu_id#b8]              v5 = iconst.i32 0
 [RexOp1pu_id#b8]              v6 = iconst.i32 0
 [RexOp1tjccb#74]              brz v6, ebb10
-[RexOp1jmpb#eb]               jump ebb3(v5, v5, v5, v5, v5, v5, v0, v1, v2, v3)
+[Op1jmpb#eb]                  jump ebb3(v5, v5, v5, v5, v5, v5, v0, v1, v2, v3)
 
                           ebb3(v15: i32, v17: i32, v25: i32, v31: i32, v40: i32, v47: i32, v54: i32, v61: i32, v68: i32, v75: i32):
-[RexOp1jmpb#eb]               jump ebb6
+[Op1jmpb#eb]                  jump ebb6
 
                           ebb6:
 [RexOp1pu_id#b8]              v8 = iconst.i32 0
@@ -23,7 +23,7 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 [RexOp2urm_noflags#4b6]       v13 = bint.i32 v12
 [RexOp1rr#21]                 v14 = band v9, v13
 [RexOp1tjccb#75]              brnz v14, ebb6
-[RexOp1jmpb#eb]               jump ebb7
+[Op1jmpb#eb]                  jump ebb7
 
                           ebb7:
 [RexOp1tjccb#74]              brz.i32 v17, ebb8
@@ -35,30 +35,30 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 [RexOp1ld#808b]               v81 = load.i64 v80
 [RexOp1rr#8001]               v22 = iadd v81, v79
 [RexMp1st#189]                istore16 v21, v22
-[RexOp1jmpb#eb]               jump ebb9
+[Op1jmpb#eb]                  jump ebb9
 
                           ebb9:
-[RexOp1jmpb#eb]               jump ebb8
+[Op1jmpb#eb]                  jump ebb8
 
                           ebb8:
 [RexOp1pu_id#b8]              v27 = iconst.i32 3
 [RexOp1pu_id#b8]              v28 = iconst.i32 4
 [RexOp1rr#09]                 v35 = bor.i32 v31, v13
 [RexOp1tjccb#75]              brnz v35, ebb15(v27)
-[RexOp1jmpb#eb]               jump ebb15(v28)
+[Op1jmpb#eb]                  jump ebb15(v28)
 
                           ebb15(v36: i32):
-[RexOp1jmpb#eb]               jump ebb3(v25, v36, v25, v31, v40, v47, v54, v61, v68, v75)
+[Op1jmpb#eb]                  jump ebb3(v25, v36, v25, v31, v40, v47, v54, v61, v68, v75)
 
                           ebb5:
-[RexOp1jmpb#eb]               jump ebb4
+[Op1jmpb#eb]                  jump ebb4
 
                           ebb4:
-[RexOp1jmpb#eb]               jump ebb2(v40, v47, v54, v61, v68, v75)
+[Op1jmpb#eb]                  jump ebb2(v40, v47, v54, v61, v68, v75)
 
                           ebb10:
 [RexOp1pu_id#b8]              v43 = iconst.i32 0
-[RexOp1jmpb#eb]               jump ebb2(v43, v5, v0, v1, v2, v3)
+[Op1jmpb#eb]                  jump ebb2(v43, v5, v0, v1, v2, v3)
 
                           ebb2(v7: i32, v45: i32, v52: i32, v59: i32, v66: i32, v73: i32):
 [RexOp1pu_id#b8]              v44 = iconst.i32 0
@@ -76,10 +76,10 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 [RexOp1rr#8001]               v64 = iadd v87, v85
 [RexOp1st#88]                 istore8 v59, v64
 [RexOp1pu_id#b8]              v65 = iconst.i32 0
-[RexOp1jmpb#eb]               jump ebb13(v65)
+[Op1jmpb#eb]                  jump ebb13(v65)
 
                           ebb14:
-[RexOp1jmpb#eb]               jump ebb13(v66)
+[Op1jmpb#eb]                  jump ebb13(v66)
 
                           ebb13(v51: i32):
 [RexOp1umr#89]                v88 = uextend.i64 v45
@@ -87,13 +87,13 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 [RexOp1ld#808b]               v90 = load.i64 v89
 [RexOp1rr#8001]               v71 = iadd v90, v88
 [RexOp1st#89]                 store v51, v71
-[RexOp1jmpb#eb]               jump ebb12
+[Op1jmpb#eb]                  jump ebb12
 
                           ebb12:
-[RexOp1jmpb#eb]               jump ebb11
+[Op1jmpb#eb]                  jump ebb11
 
                           ebb11:
-[RexOp1jmpb#eb]               jump ebb1
+[Op1jmpb#eb]                  jump ebb1
 
                           ebb1:
 [Op1ret#c3]                   return

From 22a6823496c957c4346c18bd101717bc16537d02 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 30 Apr 2019 18:11:45 +0200
Subject: [PATCH 2445/3084] [meta] Rename cdsl/inst to cdsl/instructions;

---
 cranelift/codegen/meta/src/cdsl/ast.rs                       | 2 +-
 cranelift/codegen/meta/src/cdsl/{inst.rs => instructions.rs} | 0
 cranelift/codegen/meta/src/cdsl/isa.rs                       | 2 +-
 cranelift/codegen/meta/src/cdsl/mod.rs                       | 2 +-
 cranelift/codegen/meta/src/cdsl/xform.rs                     | 4 ++--
 cranelift/codegen/meta/src/gen_inst.rs                       | 2 +-
 cranelift/codegen/meta/src/isa/arm32/mod.rs                  | 2 +-
 cranelift/codegen/meta/src/isa/arm64/mod.rs                  | 2 +-
 cranelift/codegen/meta/src/isa/riscv/mod.rs                  | 2 +-
 cranelift/codegen/meta/src/isa/x86/instructions.rs           | 2 +-
 cranelift/codegen/meta/src/isa/x86/legalize.rs               | 2 +-
 cranelift/codegen/meta/src/shared/instructions.rs            | 2 +-
 cranelift/codegen/meta/src/shared/legalize.rs                | 2 +-
 cranelift/codegen/meta/src/shared/mod.rs                     | 2 +-
 14 files changed, 14 insertions(+), 14 deletions(-)
 rename cranelift/codegen/meta/src/cdsl/{inst.rs => instructions.rs} (100%)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index ccb9adec01..f4ebea076e 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::formats::FormatRegistry;
-use crate::cdsl::inst::{InstSpec, Instruction, InstructionPredicate};
+use crate::cdsl::instructions::{InstSpec, Instruction, InstructionPredicate};
 use crate::cdsl::operands::{OperandKind, OperandKindFields};
 use crate::cdsl::types::ValueType;
 use crate::cdsl::typevar::{TypeSetBuilder, TypeVar};
diff --git a/cranelift/codegen/meta/src/cdsl/inst.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
similarity index 100%
rename from cranelift/codegen/meta/src/cdsl/inst.rs
rename to cranelift/codegen/meta/src/cdsl/instructions.rs
diff --git a/cranelift/codegen/meta/src/cdsl/isa.rs b/cranelift/codegen/meta/src/cdsl/isa.rs
index b8cfa17465..dcac2605c1 100644
--- a/cranelift/codegen/meta/src/cdsl/isa.rs
+++ b/cranelift/codegen/meta/src/cdsl/isa.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::cpu_modes::CpuMode;
-use crate::cdsl::inst::InstructionGroup;
+use crate::cdsl::instructions::InstructionGroup;
 use crate::cdsl::regs::IsaRegs;
 use crate::cdsl::settings::SettingGroup;
 use crate::cdsl::xform::{TransformGroupIndex, TransformGroups};
diff --git a/cranelift/codegen/meta/src/cdsl/mod.rs b/cranelift/codegen/meta/src/cdsl/mod.rs
index 2d7f545dd2..d3227e2dd8 100644
--- a/cranelift/codegen/meta/src/cdsl/mod.rs
+++ b/cranelift/codegen/meta/src/cdsl/mod.rs
@@ -7,7 +7,7 @@
 pub mod ast;
 pub mod cpu_modes;
 pub mod formats;
-pub mod inst;
+pub mod instructions;
 pub mod isa;
 pub mod operands;
 pub mod regs;
diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index c748459441..224616f17a 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -1,7 +1,7 @@
 use crate::cdsl::ast::{
     Apply, DefIndex, DefPool, DummyDef, DummyExpr, Expr, PatternPosition, VarIndex, VarPool,
 };
-use crate::cdsl::inst::Instruction;
+use crate::cdsl::instructions::Instruction;
 use crate::cdsl::type_inference::{infer_transform, TypeEnvironment};
 use crate::cdsl::typevar::TypeVar;
 
@@ -404,7 +404,7 @@ impl TransformGroups {
 #[should_panic]
 fn test_double_custom_legalization() {
     use crate::cdsl::formats::{FormatRegistry, InstructionFormatBuilder};
-    use crate::cdsl::inst::InstructionBuilder;
+    use crate::cdsl::instructions::InstructionBuilder;
 
     let mut format = FormatRegistry::new();
     format.insert(InstructionFormatBuilder::new("nullary"));
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index a93538ca3f..a6158258fc 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -1,6 +1,6 @@
 use crate::cdsl::camel_case;
 use crate::cdsl::formats::{FormatRegistry, InstructionFormat};
-use crate::cdsl::inst::{Instruction, InstructionGroup};
+use crate::cdsl::instructions::{Instruction, InstructionGroup};
 use crate::cdsl::operands::Operand;
 use crate::cdsl::typevar::{TypeSet, TypeVar};
 use crate::constant_hash;
diff --git a/cranelift/codegen/meta/src/isa/arm32/mod.rs b/cranelift/codegen/meta/src/isa/arm32/mod.rs
index 15a9190924..454318b8db 100644
--- a/cranelift/codegen/meta/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm32/mod.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::cpu_modes::CpuMode;
-use crate::cdsl::inst::InstructionGroup;
+use crate::cdsl::instructions::InstructionGroup;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
diff --git a/cranelift/codegen/meta/src/isa/arm64/mod.rs b/cranelift/codegen/meta/src/isa/arm64/mod.rs
index 4aaeb2c336..1aac6b26ff 100644
--- a/cranelift/codegen/meta/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm64/mod.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::cpu_modes::CpuMode;
-use crate::cdsl::inst::InstructionGroup;
+use crate::cdsl::instructions::InstructionGroup;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index a900f07005..db64c458b9 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::cpu_modes::CpuMode;
-use crate::cdsl::inst::InstructionGroup;
+use crate::cdsl::instructions::InstructionGroup;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 077860cc46..3255fc99c4 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -1,7 +1,7 @@
 #![allow(non_snake_case)]
 
 use crate::cdsl::formats::FormatRegistry;
-use crate::cdsl::inst::{InstructionBuilder as Inst, InstructionGroup};
+use crate::cdsl::instructions::{InstructionBuilder as Inst, InstructionGroup};
 use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
 use crate::cdsl::types::ValueType;
 use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index b214c26914..df6ea99885 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::ast::{var, ExprBuilder, Literal};
-use crate::cdsl::inst::InstructionGroup;
+use crate::cdsl::instructions::InstructionGroup;
 use crate::cdsl::xform::TransformGroupBuilder;
 
 use crate::shared::types::Int::{I32, I64};
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 891dadd00a..65f8103672 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1,7 +1,7 @@
 #![allow(non_snake_case)]
 
 use crate::cdsl::formats::FormatRegistry;
-use crate::cdsl::inst::{InstructionBuilder as Inst, InstructionGroup};
+use crate::cdsl::instructions::{InstructionBuilder as Inst, InstructionGroup};
 use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
 use crate::cdsl::type_inference::Constraint::WiderOrEq;
 use crate::cdsl::types::{LaneType, ValueType};
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 66ce180c1c..7d452473a2 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::ast::{var, ExprBuilder, Literal};
-use crate::cdsl::inst::{Instruction, InstructionGroup};
+use crate::cdsl::instructions::{Instruction, InstructionGroup};
 use crate::cdsl::xform::{TransformGroupBuilder, TransformGroups};
 
 use crate::shared::OperandKinds;
diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
index 5237d9ba33..f16fb7bb8b 100644
--- a/cranelift/codegen/meta/src/shared/mod.rs
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -9,7 +9,7 @@ pub mod settings;
 pub mod types;
 
 use crate::cdsl::formats::FormatRegistry;
-use crate::cdsl::inst::InstructionGroup;
+use crate::cdsl::instructions::InstructionGroup;
 use crate::cdsl::operands::OperandKind;
 use crate::cdsl::settings::SettingGroup;
 use crate::cdsl::xform::TransformGroups;

From feb90e376a3e8782e8580971f6013cc6f7cb0cf3 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 30 Apr 2019 18:14:40 +0200
Subject: [PATCH 2446/3084] [meta] Make Instruction name/doc Strings so they
 can be automatically generated;

---
 .../codegen/meta/src/cdsl/instructions.rs     | 25 ++++++++++---------
 cranelift/codegen/meta/src/gen_inst.rs        |  2 +-
 cranelift/codegen/meta/src/gen_legalizer.rs   |  2 +-
 3 files changed, 15 insertions(+), 14 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index b469e8b91f..9a4f961462 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -54,11 +54,11 @@ pub struct PolymorphicInfo {
 
 pub struct InstructionContent {
     /// Instruction mnemonic, also becomes opcode name.
-    pub name: &'static str,
+    pub name: String,
     pub camel_name: String,
 
     /// Documentation string.
-    doc: &'static str,
+    doc: String,
 
     /// Input operands. This can be a mix of SSA value operands and other operand kinds.
     pub operands_in: Vec,
@@ -115,15 +115,15 @@ impl ops::Deref for Instruction {
 }
 
 impl Instruction {
-    pub fn snake_name(&self) -> &'static str {
+    pub fn snake_name(&self) -> &str {
         if self.name == "return" {
             "return_"
         } else {
-            self.name
+            &self.name
         }
     }
 
-    pub fn doc_comment_first_line(&self) -> &'static str {
+    pub fn doc_comment_first_line(&self) -> &str {
         for line in self.doc.split("\n") {
             let stripped = line.trim();
             if stripped.len() > 0 {
@@ -162,7 +162,7 @@ impl fmt::Display for Instruction {
             fmt.write_str(" = ")?;
         }
 
-        fmt.write_str(self.name)?;
+        fmt.write_str(&self.name)?;
 
         if self.operands_in.len() > 0 {
             let operands_in = self
@@ -180,8 +180,8 @@ impl fmt::Display for Instruction {
 }
 
 pub struct InstructionBuilder {
-    name: &'static str,
-    doc: &'static str,
+    name: String,
+    doc: String,
     operands_in: Option>,
     operands_out: Option>,
     constraints: Option>,
@@ -200,10 +200,10 @@ pub struct InstructionBuilder {
 }
 
 impl InstructionBuilder {
-    pub fn new(name: &'static str, doc: &'static str) -> Self {
+    pub fn new>(name: S, doc: S) -> Self {
         Self {
-            name,
-            doc,
+            name: name.into(),
+            doc: doc.into(),
             operands_in: None,
             operands_out: None,
             constraints: None,
@@ -310,10 +310,11 @@ impl InstructionBuilder {
         // Infer from output operands whether an instruciton clobbers CPU flags or not.
         let writes_cpu_flags = operands_out.iter().any(|op| op.is_cpu_flags());
 
+        let camel_name = camel_case(&self.name);
         Instruction {
             content: Rc::new(InstructionContent {
                 name: self.name,
-                camel_name: camel_case(self.name),
+                camel_name,
                 doc: self.doc,
                 operands_in,
                 operands_out,
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index a6158258fc..613987421b 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -565,7 +565,7 @@ fn gen_opcodes<'a>(
 
     // Generate an opcode hash table for looking up opcodes by name.
     let hash_table =
-        constant_hash::generate_table(&all_inst, |inst| constant_hash::simple_hash(inst.name));
+        constant_hash::generate_table(&all_inst, |inst| constant_hash::simple_hash(&inst.name));
     fmtln!(
         fmt,
         "const OPCODE_HASH_TABLE: [Option; {}] = [",
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index de7a8068b9..45221b1ffd 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -241,7 +241,7 @@ fn emit_runtime_typecheck<'a, 'b>(
 /// Determine if `node` represents one of the value splitting instructions: `isplit` or `vsplit.
 /// These instructions are lowered specially by the `legalize::split` module.
 fn is_value_split(def: &Def) -> bool {
-    let name = def.apply.inst.name;
+    let name = &def.apply.inst.name;
     name == "isplit" || name == "vsplit"
 }
 

From d9277f249b52f151e3410e4d006f84c5f6711961 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 30 Apr 2019 18:33:01 +0200
Subject: [PATCH 2447/3084] [meta] Introduce the InstructionGroupBuilder;

This follows the rest of the code base data structures, where we have a
mutable data structure builder; once the data structure is constructed,
it's immutable.

This also makes the definition of instructions easier, and it paves the
way for defining immediate variants.
---
 .../codegen/meta/src/cdsl/instructions.rs     |  48 +-
 cranelift/codegen/meta/src/cdsl/xform.rs      |   7 +-
 cranelift/codegen/meta/src/isa/arm32/mod.rs   |   9 +-
 cranelift/codegen/meta/src/isa/arm64/mod.rs   |   9 +-
 cranelift/codegen/meta/src/isa/riscv/mod.rs   |   9 +-
 .../codegen/meta/src/isa/x86/instructions.rs  |  42 +-
 .../codegen/meta/src/shared/instructions.rs   | 509 ++++++------------
 7 files changed, 251 insertions(+), 382 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 9a4f961462..c72395ea73 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -12,6 +12,40 @@ use std::ops;
 use std::rc::Rc;
 use std::slice;
 
+pub struct InstructionGroupBuilder<'format_reg> {
+    _name: &'static str,
+    _doc: &'static str,
+    format_registry: &'format_reg FormatRegistry,
+    instructions: Vec,
+}
+
+impl<'format_reg> InstructionGroupBuilder<'format_reg> {
+    pub fn new(
+        name: &'static str,
+        doc: &'static str,
+        format_registry: &'format_reg FormatRegistry,
+    ) -> Self {
+        Self {
+            _name: name,
+            _doc: doc,
+            format_registry,
+            instructions: Vec::new(),
+        }
+    }
+
+    pub fn push(&mut self, builder: InstructionBuilder) {
+        self.instructions.push(builder.finish(self.format_registry));
+    }
+
+    pub fn finish(self) -> InstructionGroup {
+        InstructionGroup {
+            _name: self._name,
+            _doc: self._doc,
+            instructions: self.instructions,
+        }
+    }
+}
+
 /// Every instruction must belong to exactly one instruction group. A given
 /// target architecture can support instructions from multiple groups, and it
 /// does not necessarily support all instructions in a group.
@@ -22,18 +56,6 @@ pub struct InstructionGroup {
 }
 
 impl InstructionGroup {
-    pub fn new(name: &'static str, doc: &'static str) -> Self {
-        Self {
-            _name: name,
-            _doc: doc,
-            instructions: Vec::new(),
-        }
-    }
-
-    pub fn push(&mut self, inst: Instruction) {
-        self.instructions.push(inst);
-    }
-
     pub fn iter(&self) -> slice::Iter {
         self.instructions.iter()
     }
@@ -278,7 +300,7 @@ impl InstructionBuilder {
         self
     }
 
-    pub fn finish(self, format_registry: &FormatRegistry) -> Instruction {
+    fn finish(self, format_registry: &FormatRegistry) -> Instruction {
         let operands_in = self.operands_in.unwrap_or_else(Vec::new);
         let operands_out = self.operands_out.unwrap_or_else(Vec::new);
 
diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index 224616f17a..811d4517cd 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -404,11 +404,14 @@ impl TransformGroups {
 #[should_panic]
 fn test_double_custom_legalization() {
     use crate::cdsl::formats::{FormatRegistry, InstructionFormatBuilder};
-    use crate::cdsl::instructions::InstructionBuilder;
+    use crate::cdsl::instructions::{InstructionBuilder, InstructionGroupBuilder};
 
     let mut format = FormatRegistry::new();
     format.insert(InstructionFormatBuilder::new("nullary"));
-    let dummy_inst = InstructionBuilder::new("dummy", "doc").finish(&format);
+    let mut inst_group = InstructionGroupBuilder::new("test", "", &format);
+    inst_group.push(InstructionBuilder::new("dummy", "doc"));
+    let inst_group = inst_group.finish();
+    let dummy_inst = inst_group.by_name("dummy");
 
     let mut transform_group = TransformGroupBuilder::new("test", "doc");
     transform_group.custom_legalize(&dummy_inst, "custom 1");
diff --git a/cranelift/codegen/meta/src/isa/arm32/mod.rs b/cranelift/codegen/meta/src/isa/arm32/mod.rs
index 454318b8db..2115616c26 100644
--- a/cranelift/codegen/meta/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm32/mod.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::cpu_modes::CpuMode;
-use crate::cdsl::instructions::InstructionGroup;
+use crate::cdsl::instructions::InstructionGroupBuilder;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
@@ -52,7 +52,12 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let settings = define_settings(&shared_defs.settings);
     let regs = define_regs();
 
-    let inst_group = InstructionGroup::new("arm32", "arm32 specific instruction set");
+    let inst_group = InstructionGroupBuilder::new(
+        "arm32",
+        "arm32 specific instruction set",
+        &shared_defs.format_registry,
+    )
+    .finish();
 
     // CPU modes for 32-bit ARM and Thumb2.
     let mut a32 = CpuMode::new("A32");
diff --git a/cranelift/codegen/meta/src/isa/arm64/mod.rs b/cranelift/codegen/meta/src/isa/arm64/mod.rs
index 1aac6b26ff..07518bf986 100644
--- a/cranelift/codegen/meta/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm64/mod.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::cpu_modes::CpuMode;
-use crate::cdsl::instructions::InstructionGroup;
+use crate::cdsl::instructions::InstructionGroupBuilder;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
@@ -48,7 +48,12 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let settings = define_settings(&shared_defs.settings);
     let regs = define_registers();
 
-    let inst_group = InstructionGroup::new("arm64", "arm64 specific instruction set");
+    let inst_group = InstructionGroupBuilder::new(
+        "arm64",
+        "arm64 specific instruction set",
+        &shared_defs.format_registry,
+    )
+    .finish();
 
     let mut a64 = CpuMode::new("A64");
 
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index db64c458b9..66390c4c43 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::cpu_modes::CpuMode;
-use crate::cdsl::instructions::InstructionGroup;
+use crate::cdsl::instructions::InstructionGroupBuilder;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
@@ -86,7 +86,12 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let settings = define_settings(&shared_defs.settings);
     let regs = define_registers();
 
-    let inst_group = InstructionGroup::new("riscv", "riscv specific instruction set");
+    let inst_group = InstructionGroupBuilder::new(
+        "riscv",
+        "riscv specific instruction set",
+        &shared_defs.format_registry,
+    )
+    .finish();
 
     // CPU modes for 32-bit and 64-bit operation.
     let mut rv_32 = CpuMode::new("RV32");
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 3255fc99c4..e66b7752f7 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -1,14 +1,17 @@
 #![allow(non_snake_case)]
 
 use crate::cdsl::formats::FormatRegistry;
-use crate::cdsl::instructions::{InstructionBuilder as Inst, InstructionGroup};
+use crate::cdsl::instructions::{
+    InstructionBuilder as Inst, InstructionGroup, InstructionGroupBuilder,
+};
 use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
 use crate::cdsl::types::ValueType;
 use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
 use crate::shared::types;
 
 pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
-    let mut ig = InstructionGroup::new("x86", "x86 specific instruction set");
+    let mut ig =
+        InstructionGroupBuilder::new("x86", "x86 specific instruction set", format_registry);
 
     let iflags: &TypeVar = &ValueType::Special(types::Flag::IFlags.into()).into();
 
@@ -39,8 +42,7 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
         )
         .operands_in(vec![nlo, nhi, d])
         .operands_out(vec![q, r])
-        .can_trap(true)
-        .finish(format_registry),
+        .can_trap(true),
     );
 
     ig.push(
@@ -59,8 +61,7 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
         )
         .operands_in(vec![nlo, nhi, d])
         .operands_out(vec![q, r])
-        .can_trap(true)
-        .finish(format_registry),
+        .can_trap(true),
     );
 
     let argL = &operand("argL", iWord);
@@ -79,8 +80,7 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
         "#,
         )
         .operands_in(vec![argL, argR])
-        .operands_out(vec![resLo, resHi])
-        .finish(format_registry),
+        .operands_out(vec![resLo, resHi]),
     );
 
     ig.push(
@@ -94,8 +94,7 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
         "#,
         )
         .operands_in(vec![argL, argR])
-        .operands_out(vec![resLo, resHi])
-        .finish(format_registry),
+        .operands_out(vec![resLo, resHi]),
     );
 
     let Float = &TypeVar::new(
@@ -131,8 +130,7 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let x = &operand("x", Float);
@@ -154,8 +152,7 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -173,8 +170,7 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let x = &operand("x", iWord);
@@ -193,8 +189,7 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
         )
         .operands_in(vec![x])
         .other_side_effects(true)
-        .can_store(true)
-        .finish(format_registry),
+        .can_store(true),
     );
 
     ig.push(
@@ -212,8 +207,7 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
         )
         .operands_out(vec![x])
         .other_side_effects(true)
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     let y = &operand("y", iWord);
@@ -233,8 +227,7 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
     "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![y, rflags])
-        .finish(format_registry),
+        .operands_out(vec![y, rflags]),
     );
 
     ig.push(
@@ -246,9 +239,8 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
     "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![y, rflags])
-        .finish(format_registry),
+        .operands_out(vec![y, rflags]),
     );
 
-    ig
+    ig.finish()
 }
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 65f8103672..5c810b9979 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1,7 +1,9 @@
 #![allow(non_snake_case)]
 
 use crate::cdsl::formats::FormatRegistry;
-use crate::cdsl::instructions::{InstructionBuilder as Inst, InstructionGroup};
+use crate::cdsl::instructions::{
+    InstructionBuilder as Inst, InstructionGroup, InstructionGroupBuilder,
+};
 use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
 use crate::cdsl::type_inference::Constraint::WiderOrEq;
 use crate::cdsl::types::{LaneType, ValueType};
@@ -13,7 +15,8 @@ pub fn define(
     immediates: &OperandKinds,
     entities: &OperandKinds,
 ) -> InstructionGroup {
-    let mut ig = InstructionGroup::new("base", "Shared base instruction set");
+    let mut ig =
+        InstructionGroupBuilder::new("base", "Shared base instruction set", format_registry);
 
     // Operand kind shorthands.
     let intcc = immediates.by_name("intcc");
@@ -151,8 +154,7 @@ pub fn define(
         )
         .operands_in(vec![EBB, args])
         .is_terminator(true)
-        .is_branch(true)
-        .finish(format_registry),
+        .is_branch(true),
     );
 
     ig.push(
@@ -171,8 +173,7 @@ pub fn define(
         )
         .operands_in(vec![EBB, args])
         .is_terminator(true)
-        .is_branch(true)
-        .finish(format_registry),
+        .is_branch(true),
     );
 
     ig.push(
@@ -186,8 +187,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![c, EBB, args])
-        .is_branch(true)
-        .finish(format_registry),
+        .is_branch(true),
     );
 
     ig.push(
@@ -201,8 +201,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![c, EBB, args])
-        .is_branch(true)
-        .finish(format_registry),
+        .is_branch(true),
     );
 
     ig.push(
@@ -227,8 +226,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![Cond, x, y, EBB, args])
-        .is_branch(true)
-        .finish(format_registry),
+        .is_branch(true),
     );
 
     let f = &operand("f", iflags);
@@ -241,8 +239,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![Cond, f, EBB, args])
-        .is_branch(true)
-        .finish(format_registry),
+        .is_branch(true),
     );
 
     let Cond = &operand("Cond", floatcc);
@@ -256,8 +253,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![Cond, f, EBB, args])
-        .is_branch(true)
-        .finish(format_registry),
+        .is_branch(true),
     );
 
     let x = &operand_doc("x", iB, "index into jump table");
@@ -293,8 +289,7 @@ pub fn define(
         )
         .operands_in(vec![x, EBB, JT])
         .is_terminator(true)
-        .is_branch(true)
-        .finish(format_registry),
+        .is_branch(true),
     );
 
     let Size = &operand_doc("Size", uimm8, "Size in bytes");
@@ -314,8 +309,7 @@ pub fn define(
     "#,
         )
         .operands_in(vec![x, addr, Size, JT])
-        .operands_out(vec![entry])
-        .finish(format_registry),
+        .operands_out(vec![entry]),
     );
 
     ig.push(
@@ -331,8 +325,7 @@ pub fn define(
     "#,
         )
         .operands_in(vec![JT])
-        .operands_out(vec![addr])
-        .finish(format_registry),
+        .operands_out(vec![addr]),
     );
 
     ig.push(
@@ -348,8 +341,7 @@ pub fn define(
         .operands_in(vec![addr, JT])
         .is_indirect_branch(true)
         .is_terminator(true)
-        .is_branch(true)
-        .finish(format_registry),
+        .is_branch(true),
     );
 
     ig.push(
@@ -361,8 +353,7 @@ pub fn define(
         )
         .other_side_effects(true)
         .can_load(true)
-        .can_store(true)
-        .finish(format_registry),
+        .can_store(true),
     );
 
     let code = &operand("code", trapcode);
@@ -376,8 +367,7 @@ pub fn define(
         )
         .operands_in(vec![code])
         .can_trap(true)
-        .is_terminator(true)
-        .finish(format_registry),
+        .is_terminator(true),
     );
 
     ig.push(
@@ -390,8 +380,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![c, code])
-        .can_trap(true)
-        .finish(format_registry),
+        .can_trap(true),
     );
 
     ig.push(
@@ -404,8 +393,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![c, code])
-        .can_trap(true)
-        .finish(format_registry),
+        .can_trap(true),
     );
 
     let Cond = &operand("Cond", intcc);
@@ -419,8 +407,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![Cond, f, code])
-        .can_trap(true)
-        .finish(format_registry),
+        .can_trap(true),
     );
 
     let Cond = &operand("Cond", floatcc);
@@ -434,8 +421,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![Cond, f, code])
-        .can_trap(true)
-        .finish(format_registry),
+        .can_trap(true),
     );
 
     let rvals = &operand_doc("rvals", variable_args, "return values");
@@ -453,8 +439,7 @@ pub fn define(
         )
         .operands_in(vec![rvals])
         .is_return(true)
-        .is_terminator(true)
-        .finish(format_registry),
+        .is_terminator(true),
     );
 
     ig.push(
@@ -470,8 +455,7 @@ pub fn define(
         )
         .operands_in(vec![rvals])
         .is_return(true)
-        .is_terminator(true)
-        .finish(format_registry),
+        .is_terminator(true),
     );
 
     let FN = &operand_doc(
@@ -493,8 +477,7 @@ pub fn define(
         )
         .operands_in(vec![FN, args])
         .operands_out(vec![rvals])
-        .is_call(true)
-        .finish(format_registry),
+        .is_call(true),
     );
 
     let SIG = &operand_doc("SIG", sig_ref, "function signature");
@@ -517,8 +500,7 @@ pub fn define(
         )
         .operands_in(vec![SIG, callee, args])
         .operands_out(vec![rvals])
-        .is_call(true)
-        .finish(format_registry),
+        .is_call(true),
     );
 
     ig.push(
@@ -535,8 +517,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![FN])
-        .operands_out(vec![addr])
-        .finish(format_registry),
+        .operands_out(vec![addr]),
     );
 
     let SS = &operand("SS", stack_slot);
@@ -559,8 +540,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -575,8 +555,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -590,8 +569,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![MemFlags, x, p, Offset])
-        .can_store(true)
-        .finish(format_registry),
+        .can_store(true),
     );
 
     ig.push(
@@ -605,8 +583,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![MemFlags, x, args, Offset])
-        .can_store(true)
-        .finish(format_registry),
+        .can_store(true),
     );
 
     let iExt8 = &TypeVar::new(
@@ -628,8 +605,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -643,8 +619,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -658,8 +633,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -673,8 +647,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -687,8 +660,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![MemFlags, x, p, Offset])
-        .can_store(true)
-        .finish(format_registry),
+        .can_store(true),
     );
 
     ig.push(
@@ -701,8 +673,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![MemFlags, x, args, Offset])
-        .can_store(true)
-        .finish(format_registry),
+        .can_store(true),
     );
 
     let iExt16 = &TypeVar::new(
@@ -724,8 +695,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -739,8 +709,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -754,8 +723,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -769,8 +737,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -783,8 +750,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![MemFlags, x, p, Offset])
-        .can_store(true)
-        .finish(format_registry),
+        .can_store(true),
     );
 
     ig.push(
@@ -797,8 +763,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![MemFlags, x, args, Offset])
-        .can_store(true)
-        .finish(format_registry),
+        .can_store(true),
     );
 
     let iExt32 = &TypeVar::new(
@@ -820,8 +785,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -835,8 +799,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -850,8 +813,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -865,8 +827,7 @@ pub fn define(
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -879,8 +840,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![MemFlags, x, p, Offset])
-        .can_store(true)
-        .finish(format_registry),
+        .can_store(true),
     );
 
     ig.push(
@@ -893,8 +853,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![MemFlags, x, args, Offset])
-        .can_store(true)
-        .finish(format_registry),
+        .can_store(true),
     );
 
     let x = &operand_doc("x", Mem, "Value to be stored");
@@ -917,8 +876,7 @@ pub fn define(
         )
         .operands_in(vec![SS, Offset])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     ig.push(
@@ -936,8 +894,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, SS, Offset])
-        .can_store(true)
-        .finish(format_registry),
+        .can_store(true),
     );
 
     ig.push(
@@ -952,8 +909,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![SS, Offset])
-        .operands_out(vec![addr])
-        .finish(format_registry),
+        .operands_out(vec![addr]),
     );
 
     let GV = &operand("GV", global_value);
@@ -966,8 +922,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![GV])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -978,8 +933,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![GV])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let HeapOffset = &TypeVar::new(
@@ -1008,8 +962,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![H, p, Size])
-        .operands_out(vec![addr])
-        .finish(format_registry),
+        .operands_out(vec![addr]),
     );
 
     let TableOffset = &TypeVar::new(
@@ -1039,8 +992,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![T, p, Offset])
-        .operands_out(vec![addr])
-        .finish(format_registry),
+        .operands_out(vec![addr]),
     );
 
     let N = &operand("N", imm64);
@@ -1057,8 +1009,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![N])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let N = &operand("N", ieee32);
@@ -1074,8 +1025,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![N])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let N = &operand("N", ieee64);
@@ -1091,8 +1041,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![N])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let N = &operand("N", boolean);
@@ -1109,21 +1058,17 @@ pub fn define(
         "#,
         )
         .operands_in(vec![N])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
-    ig.push(
-        Inst::new(
-            "nop",
-            r#"
+    ig.push(Inst::new(
+        "nop",
+        r#"
         Just a dummy instruction
 
         Note: this doesn't compile to a machine code nop
         "#,
-        )
-        .finish(format_registry),
-    );
+    ));
 
     let c = &operand_doc("c", Testable, "Controlling value to test");
     let x = &operand_doc("x", Any, "Value to use when `c` is true");
@@ -1141,8 +1086,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![c, x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let cc = &operand_doc("cc", intcc, "Controlling condition code");
@@ -1156,8 +1100,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![cc, flags, x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let x = &operand("x", Any);
@@ -1177,8 +1120,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1193,8 +1135,7 @@ pub fn define(
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
-        .can_store(true)
-        .finish(format_registry),
+        .can_store(true),
     );
 
     ig.push(
@@ -1209,8 +1150,7 @@ pub fn define(
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
-        .can_load(true)
-        .finish(format_registry),
+        .can_load(true),
     );
 
     let src = &operand("src", regunit);
@@ -1233,8 +1173,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, src, dst])
-        .other_side_effects(true)
-        .finish(format_registry),
+        .other_side_effects(true),
     );
 
     ig.push(
@@ -1250,8 +1189,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![src, dst])
-        .other_side_effects(true)
-        .finish(format_registry),
+        .other_side_effects(true),
     );
 
     ig.push(
@@ -1265,8 +1203,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let delta = &operand("delta", Int);
@@ -1281,8 +1218,7 @@ pub fn define(
     "#,
         )
         .operands_in(vec![delta])
-        .other_side_effects(true)
-        .finish(format_registry),
+        .other_side_effects(true),
     );
 
     let Offset = &operand_doc("Offset", imm64, "Offset from current stack pointer");
@@ -1299,8 +1235,7 @@ pub fn define(
     "#,
         )
         .operands_in(vec![Offset])
-        .other_side_effects(true)
-        .finish(format_registry),
+        .other_side_effects(true),
     );
 
     let Offset = &operand_doc("Offset", imm64, "Offset from current stack pointer");
@@ -1318,8 +1253,7 @@ pub fn define(
     "#,
         )
         .operands_in(vec![Offset])
-        .other_side_effects(true)
-        .finish(format_registry),
+        .other_side_effects(true),
     );
 
     let f = &operand("f", iflags);
@@ -1335,8 +1269,7 @@ pub fn define(
     "#,
         )
         .operands_in(vec![addr])
-        .operands_out(vec![f])
-        .finish(format_registry),
+        .operands_out(vec![f]),
     );
 
     ig.push(
@@ -1354,8 +1287,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, src, SS])
-        .other_side_effects(true)
-        .finish(format_registry),
+        .other_side_effects(true),
     );
 
     ig.push(
@@ -1373,8 +1305,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, SS, dst])
-        .other_side_effects(true)
-        .finish(format_registry),
+        .other_side_effects(true),
     );
 
     let x = &operand_doc("x", TxN, "Vector to split");
@@ -1394,8 +1325,7 @@ pub fn define(
         )
         .operands_in(vec![x])
         .operands_out(vec![lo, hi])
-        .is_ghost(true)
-        .finish(format_registry),
+        .is_ghost(true),
     );
 
     let Any128 = &TypeVar::new(
@@ -1430,8 +1360,7 @@ pub fn define(
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a])
-        .is_ghost(true)
-        .finish(format_registry),
+        .is_ghost(true),
     );
 
     let c = &operand_doc("c", &TxN.as_bool(), "Controlling vector");
@@ -1450,8 +1379,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![c, x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let x = &operand("x", &TxN.lane_of());
@@ -1466,8 +1394,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let x = &operand_doc("x", TxN, "SIMD vector to modify");
@@ -1485,8 +1412,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Idx, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let x = &operand("x", TxN);
@@ -1503,8 +1429,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Idx])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let a = &operand("a", &Int.as_bool());
@@ -1537,8 +1462,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![Cond, x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let a = &operand("a", b1);
@@ -1559,8 +1483,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![Cond, x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let f = &operand("f", iflags);
@@ -1578,8 +1501,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![f])
-        .finish(format_registry),
+        .operands_out(vec![f]),
     );
 
     ig.push(
@@ -1593,8 +1515,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![f])
-        .finish(format_registry),
+        .operands_out(vec![f]),
     );
 
     let a = &operand("a", Int);
@@ -1612,8 +1533,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1627,8 +1547,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1645,8 +1564,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1661,8 +1579,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1677,8 +1594,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1692,8 +1608,7 @@ pub fn define(
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a])
-        .can_trap(true)
-        .finish(format_registry),
+        .can_trap(true),
     );
 
     ig.push(
@@ -1710,8 +1625,7 @@ pub fn define(
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a])
-        .can_trap(true)
-        .finish(format_registry),
+        .can_trap(true),
     );
 
     ig.push(
@@ -1725,8 +1639,7 @@ pub fn define(
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a])
-        .can_trap(true)
-        .finish(format_registry),
+        .can_trap(true),
     );
 
     ig.push(
@@ -1740,8 +1653,7 @@ pub fn define(
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a])
-        .can_trap(true)
-        .finish(format_registry),
+        .can_trap(true),
     );
 
     let a = &operand("a", iB);
@@ -1761,8 +1673,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1776,8 +1687,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1790,8 +1700,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1806,8 +1715,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1820,8 +1728,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1834,8 +1741,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1853,8 +1759,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let a = &operand("a", iB);
@@ -1882,8 +1787,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y, c_in])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1904,8 +1808,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a, c_out])
-        .finish(format_registry),
+        .operands_out(vec![a, c_out]),
     );
 
     ig.push(
@@ -1926,8 +1829,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y, c_in])
-        .operands_out(vec![a, c_out])
-        .finish(format_registry),
+        .operands_out(vec![a, c_out]),
     );
 
     ig.push(
@@ -1947,8 +1849,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y, b_in])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -1969,8 +1870,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a, b_out])
-        .finish(format_registry),
+        .operands_out(vec![a, b_out]),
     );
 
     ig.push(
@@ -1991,8 +1891,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y, b_in])
-        .operands_out(vec![a, b_out])
-        .finish(format_registry),
+        .operands_out(vec![a, b_out]),
     );
 
     let bits = &TypeVar::new(
@@ -2018,8 +1917,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2030,8 +1928,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2042,8 +1939,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2054,8 +1950,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2068,8 +1963,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2082,8 +1976,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2096,8 +1989,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let x = &operand("x", iB);
@@ -2117,8 +2009,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2134,8 +2025,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2151,8 +2041,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let x = &operand_doc("x", Int, "Scalar or vector value to shift");
@@ -2170,8 +2059,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2184,8 +2072,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2196,8 +2083,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2208,8 +2094,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2229,8 +2114,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2251,8 +2135,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2267,8 +2150,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2281,8 +2163,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2295,8 +2176,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2309,8 +2189,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, Y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let x = &operand("x", iB);
@@ -2326,8 +2205,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2342,8 +2220,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2358,8 +2235,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2374,8 +2250,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2388,8 +2263,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let Float = &TypeVar::new(
@@ -2471,8 +2345,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![Cond, x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let f = &operand("f", fflags);
@@ -2488,8 +2361,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![f])
-        .finish(format_registry),
+        .operands_out(vec![f]),
     );
 
     let x = &operand("x", Float);
@@ -2505,8 +2377,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2517,8 +2388,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2529,8 +2399,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2545,8 +2414,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2557,8 +2425,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2572,8 +2439,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y, z])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let a = &operand_doc("a", Float, "``x`` with its sign bit inverted");
@@ -2588,8 +2454,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let a = &operand_doc("a", Float, "``x`` with its sign bit cleared");
@@ -2604,8 +2469,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let a = &operand_doc(
@@ -2625,8 +2489,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let a = &operand_doc("a", Float, "The smaller of ``x`` and ``y``");
@@ -2641,8 +2504,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let a = &operand_doc("a", Float, "The larger of ``x`` and ``y``");
@@ -2657,8 +2519,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x, y])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let a = &operand_doc("a", Float, "``x`` rounded to integral value");
@@ -2671,8 +2532,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2683,8 +2543,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2695,8 +2554,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2708,8 +2566,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let Cond = &operand("Cond", intcc);
@@ -2727,8 +2584,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![Cond, f])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let Cond = &operand("Cond", floatcc);
@@ -2745,8 +2601,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![Cond, f])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let x = &operand("x", Mem);
@@ -2764,8 +2619,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let Bool = &TypeVar::new(
@@ -2802,8 +2656,7 @@ pub fn define(
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
-        .constraints(vec![WiderOrEq(Bool.clone(), BoolTo.clone())])
-        .finish(format_registry),
+        .constraints(vec![WiderOrEq(Bool.clone(), BoolTo.clone())]),
     );
 
     let BoolTo = &TypeVar::new(
@@ -2830,8 +2683,7 @@ pub fn define(
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
-        .constraints(vec![WiderOrEq(BoolTo.clone(), Bool.clone())])
-        .finish(format_registry),
+        .constraints(vec![WiderOrEq(BoolTo.clone(), Bool.clone())]),
     );
 
     let IntTo = &TypeVar::new(
@@ -2856,8 +2708,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -2871,8 +2722,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let Int = &TypeVar::new(
@@ -2912,8 +2762,7 @@ pub fn define(
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
-        .constraints(vec![WiderOrEq(Int.clone(), IntTo.clone())])
-        .finish(format_registry),
+        .constraints(vec![WiderOrEq(Int.clone(), IntTo.clone())]),
     );
 
     let IntTo = &TypeVar::new(
@@ -2944,8 +2793,7 @@ pub fn define(
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
-        .constraints(vec![WiderOrEq(IntTo.clone(), Int.clone())])
-        .finish(format_registry),
+        .constraints(vec![WiderOrEq(IntTo.clone(), Int.clone())]),
     );
 
     ig.push(
@@ -2965,8 +2813,7 @@ pub fn define(
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
-        .constraints(vec![WiderOrEq(IntTo.clone(), Int.clone())])
-        .finish(format_registry),
+        .constraints(vec![WiderOrEq(IntTo.clone(), Int.clone())]),
     );
 
     let FloatTo = &TypeVar::new(
@@ -2999,8 +2846,7 @@ pub fn define(
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
-        .constraints(vec![WiderOrEq(FloatTo.clone(), Float.clone())])
-        .finish(format_registry),
+        .constraints(vec![WiderOrEq(FloatTo.clone(), Float.clone())]),
     );
 
     ig.push(
@@ -3022,8 +2868,7 @@ pub fn define(
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
-        .constraints(vec![WiderOrEq(Float.clone(), FloatTo.clone())])
-        .finish(format_registry),
+        .constraints(vec![WiderOrEq(Float.clone(), FloatTo.clone())]),
     );
 
     let x = &operand("x", Float);
@@ -3044,8 +2889,7 @@ pub fn define(
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
-        .can_trap(true)
-        .finish(format_registry),
+        .can_trap(true),
     );
 
     ig.push(
@@ -3058,8 +2902,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -3077,8 +2920,7 @@ pub fn define(
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
-        .can_trap(true)
-        .finish(format_registry),
+        .can_trap(true),
     );
 
     ig.push(
@@ -3090,8 +2932,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let x = &operand("x", Int);
@@ -3110,8 +2951,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     ig.push(
@@ -3127,8 +2967,7 @@ pub fn define(
         "#,
         )
         .operands_in(vec![x])
-        .operands_out(vec![a])
-        .finish(format_registry),
+        .operands_out(vec![a]),
     );
 
     let WideInt = &TypeVar::new(
@@ -3158,8 +2997,7 @@ pub fn define(
         )
         .operands_in(vec![x])
         .operands_out(vec![lo, hi])
-        .is_ghost(true)
-        .finish(format_registry),
+        .is_ghost(true),
     );
 
     let NarrowInt = &TypeVar::new(
@@ -3192,9 +3030,8 @@ pub fn define(
         )
         .operands_in(vec![lo, hi])
         .operands_out(vec![a])
-        .is_ghost(true)
-        .finish(format_registry),
+        .is_ghost(true),
     );
 
-    ig
+    ig.finish()
 }

From 70f79d23bf9f1682c0a1e4fd9f3cad64060d6344 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 28 May 2019 15:01:14 +0200
Subject: [PATCH 2448/3084] [meta] Make Builders build() instead of finish();

---
 cranelift/codegen/meta/src/cdsl/formats.rs    |   4 +-
 .../codegen/meta/src/cdsl/instructions.rs     |   6 +-
 cranelift/codegen/meta/src/cdsl/operands.rs   |  10 +-
 cranelift/codegen/meta/src/cdsl/regs.rs       |   2 +-
 cranelift/codegen/meta/src/cdsl/settings.rs   |   2 +-
 cranelift/codegen/meta/src/cdsl/typevar.rs    | 143 +++++++++---------
 cranelift/codegen/meta/src/cdsl/xform.rs      |   4 +-
 cranelift/codegen/meta/src/isa/arm32/mod.rs   |   6 +-
 cranelift/codegen/meta/src/isa/arm64/mod.rs   |   6 +-
 cranelift/codegen/meta/src/isa/riscv/mod.rs   |   6 +-
 .../codegen/meta/src/isa/x86/instructions.rs  |   8 +-
 .../codegen/meta/src/isa/x86/legalize.rs      |   2 +-
 .../codegen/meta/src/isa/x86/registers.rs     |   2 +-
 .../codegen/meta/src/isa/x86/settings.rs      |   2 +-
 cranelift/codegen/meta/src/shared/entities.rs |  18 +--
 .../codegen/meta/src/shared/immediates.rs     |  24 +--
 .../codegen/meta/src/shared/instructions.rs   |  58 +++----
 cranelift/codegen/meta/src/shared/legalize.rs |  10 +-
 cranelift/codegen/meta/src/shared/settings.rs |   2 +-
 19 files changed, 156 insertions(+), 159 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
index 244564a765..cefcfa1626 100644
--- a/cranelift/codegen/meta/src/cdsl/formats.rs
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -135,7 +135,7 @@ impl InstructionFormatBuilder {
         self
     }
 
-    pub fn finish(self) -> InstructionFormat {
+    pub fn build(self) -> InstructionFormat {
         let typevar_operand = if self.typevar_operand.is_some() {
             self.typevar_operand
         } else if self.has_value_list || self.num_value_operands > 0 {
@@ -213,7 +213,7 @@ impl FormatRegistry {
             );
         }
 
-        let format = inst_format.finish();
+        let format = inst_format.build();
 
         // Compute key.
         let imm_keys = format
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index c72395ea73..d57bcc254a 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -34,10 +34,10 @@ impl<'format_reg> InstructionGroupBuilder<'format_reg> {
     }
 
     pub fn push(&mut self, builder: InstructionBuilder) {
-        self.instructions.push(builder.finish(self.format_registry));
+        self.instructions.push(builder.build(self.format_registry));
     }
 
-    pub fn finish(self) -> InstructionGroup {
+    pub fn build(self) -> InstructionGroup {
         InstructionGroup {
             _name: self._name,
             _doc: self._doc,
@@ -300,7 +300,7 @@ impl InstructionBuilder {
         self
     }
 
-    fn finish(self, format_registry: &FormatRegistry) -> Instruction {
+    fn build(self, format_registry: &FormatRegistry) -> Instruction {
         let operands_in = self.operands_in.unwrap_or_else(Vec::new);
         let operands_out = self.operands_out.unwrap_or_else(Vec::new);
 
diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
index 93a92823be..a219584c9e 100644
--- a/cranelift/codegen/meta/src/cdsl/operands.rs
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -95,7 +95,7 @@ impl OperandBuilder {
         self.doc = Some(doc.into());
         self
     }
-    pub fn finish(self) -> Operand {
+    pub fn build(self) -> Operand {
         let doc = match self.doc {
             Some(doc) => Some(doc),
             None => match &self.kind.fields {
@@ -215,7 +215,7 @@ impl OperandKindBuilder {
         self
     }
 
-    pub fn finish(self) -> OperandKind {
+    pub fn build(self) -> OperandKind {
         let default_member = match self.default_member {
             Some(default_member) => Some(default_member),
             None => match &self.fields {
@@ -261,7 +261,7 @@ impl OperandKindBuilder {
 
 impl Into for &TypeVar {
     fn into(self) -> OperandKind {
-        OperandKindBuilder::new("value", OperandKindFields::TypeVar(self.into())).finish()
+        OperandKindBuilder::new("value", OperandKindFields::TypeVar(self.into())).build()
     }
 }
 impl Into for &OperandKind {
@@ -272,7 +272,7 @@ impl Into for &OperandKind {
 
 /// Helper to create an operand in definitions files.
 pub fn create_operand(name: &'static str, kind: impl Into) -> Operand {
-    OperandBuilder::new(name, kind.into()).finish()
+    OperandBuilder::new(name, kind.into()).build()
 }
 
 /// Helper to create an operand with a documentation in definitions files.
@@ -281,5 +281,5 @@ pub fn create_operand_doc(
     kind: impl Into,
     doc: &'static str,
 ) -> Operand {
-    OperandBuilder::new(name, kind.into()).doc(doc).finish()
+    OperandBuilder::new(name, kind.into()).doc(doc).build()
 }
diff --git a/cranelift/codegen/meta/src/cdsl/regs.rs b/cranelift/codegen/meta/src/cdsl/regs.rs
index 22d7d3a6c1..8a84d03617 100644
--- a/cranelift/codegen/meta/src/cdsl/regs.rs
+++ b/cranelift/codegen/meta/src/cdsl/regs.rs
@@ -273,7 +273,7 @@ impl IsaRegsBuilder {
     /// 2. There are no identical classes under different names.
     /// 3. Classes are sorted topologically such that all subclasses have a
     ///    higher index that the superclass.
-    pub fn finish(self) -> IsaRegs {
+    pub fn build(self) -> IsaRegs {
         for reg_bank in self.banks.values() {
             for i1 in reg_bank.classes.iter() {
                 for i2 in reg_bank.classes.iter() {
diff --git a/cranelift/codegen/meta/src/cdsl/settings.rs b/cranelift/codegen/meta/src/cdsl/settings.rs
index 3983946225..61677a71f3 100644
--- a/cranelift/codegen/meta/src/cdsl/settings.rs
+++ b/cranelift/codegen/meta/src/cdsl/settings.rs
@@ -315,7 +315,7 @@ impl SettingGroupBuilder {
     /// 2. in the list above.
     ///
     /// Assign `byte_offset` and `bit_offset` fields in all settings.
-    pub fn finish(self) -> SettingGroup {
+    pub fn build(self) -> SettingGroup {
         let mut group = SettingGroup {
             name: self.name,
             settings: Vec::new(),
diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 3b18d508fc..22640b6b9c 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -59,10 +59,10 @@ impl TypeVar {
         let (scalar_type, num_lanes) = match value_type {
             ValueType::BV(bitvec_type) => {
                 let bits = bitvec_type.lane_bits() as RangeBound;
-                return TypeVar::new(name, doc, builder.bitvecs(bits..bits).finish());
+                return TypeVar::new(name, doc, builder.bitvecs(bits..bits).build());
             }
             ValueType::Special(special_type) => {
-                return TypeVar::new(name, doc, builder.specials(vec![special_type]).finish());
+                return TypeVar::new(name, doc, builder.specials(vec![special_type]).build());
             }
             ValueType::Lane(lane_type) => (lane_type, 1),
             ValueType::Vector(vec_type) => {
@@ -86,7 +86,7 @@ impl TypeVar {
                 builder.bools(bits..bits)
             }
         };
-        TypeVar::new(name, doc, builder.finish())
+        TypeVar::new(name, doc, builder.build())
     }
 
     /// Get a fresh copy of self, named after `name`. Can only be called on non-derived typevars.
@@ -814,7 +814,7 @@ impl TypeSetBuilder {
         self
     }
 
-    pub fn finish(self) -> TypeSet {
+    pub fn build(self) -> TypeSet {
         let min_lanes = if self.includes_scalars { 1 } else { 2 };
 ;
         let bools = range_to_set(self.bools.to_range(1..MAX_BITS, None))
@@ -841,7 +841,7 @@ impl TypeSetBuilder {
             .bitvecs(Interval::All)
             .specials(ValueType::all_special_types().collect())
             .includes_scalars(true)
-            .finish()
+            .build()
     }
 }
 
@@ -911,7 +911,7 @@ fn range_to_set(range: Option) -> NumSet {
 
 #[test]
 fn test_typevar_builder() {
-    let type_set = TypeSetBuilder::new().ints(Interval::All).finish();
+    let type_set = TypeSetBuilder::new().ints(Interval::All).build();
     assert_eq!(type_set.lanes, num_set![1]);
     assert!(type_set.floats.is_empty());
     assert_eq!(type_set.ints, num_set![8, 16, 32, 64]);
@@ -919,7 +919,7 @@ fn test_typevar_builder() {
     assert!(type_set.bitvecs.is_empty());
     assert!(type_set.specials.is_empty());
 
-    let type_set = TypeSetBuilder::new().bools(Interval::All).finish();
+    let type_set = TypeSetBuilder::new().bools(Interval::All).build();
     assert_eq!(type_set.lanes, num_set![1]);
     assert!(type_set.floats.is_empty());
     assert!(type_set.ints.is_empty());
@@ -927,7 +927,7 @@ fn test_typevar_builder() {
     assert!(type_set.bitvecs.is_empty());
     assert!(type_set.specials.is_empty());
 
-    let type_set = TypeSetBuilder::new().floats(Interval::All).finish();
+    let type_set = TypeSetBuilder::new().floats(Interval::All).build();
     assert_eq!(type_set.lanes, num_set![1]);
     assert_eq!(type_set.floats, num_set![32, 64]);
     assert!(type_set.ints.is_empty());
@@ -939,7 +939,7 @@ fn test_typevar_builder() {
         .floats(Interval::All)
         .simd_lanes(Interval::All)
         .includes_scalars(false)
-        .finish();
+        .build();
     assert_eq!(type_set.lanes, num_set![2, 4, 8, 16, 32, 64, 128, 256]);
     assert_eq!(type_set.floats, num_set![32, 64]);
     assert!(type_set.ints.is_empty());
@@ -951,7 +951,7 @@ fn test_typevar_builder() {
         .floats(Interval::All)
         .simd_lanes(Interval::All)
         .includes_scalars(true)
-        .finish();
+        .build();
     assert_eq!(type_set.lanes, num_set![1, 2, 4, 8, 16, 32, 64, 128, 256]);
     assert_eq!(type_set.floats, num_set![32, 64]);
     assert!(type_set.ints.is_empty());
@@ -959,7 +959,7 @@ fn test_typevar_builder() {
     assert!(type_set.bitvecs.is_empty());
     assert!(type_set.specials.is_empty());
 
-    let type_set = TypeSetBuilder::new().ints(16..64).finish();
+    let type_set = TypeSetBuilder::new().ints(16..64).build();
     assert_eq!(type_set.lanes, num_set![1]);
     assert_eq!(type_set.ints, num_set![16, 32, 64]);
     assert!(type_set.floats.is_empty());
@@ -971,13 +971,13 @@ fn test_typevar_builder() {
 #[test]
 #[should_panic]
 fn test_typevar_builder_too_high_bound_panic() {
-    TypeSetBuilder::new().ints(16..2 * MAX_BITS).finish();
+    TypeSetBuilder::new().ints(16..2 * MAX_BITS).build();
 }
 
 #[test]
 #[should_panic]
 fn test_typevar_builder_inverted_bounds_panic() {
-    TypeSetBuilder::new().ints(32..16).finish();
+    TypeSetBuilder::new().ints(32..16).build();
 }
 
 #[test]
@@ -986,14 +986,14 @@ fn test_as_bool() {
         .simd_lanes(2..8)
         .ints(8..8)
         .floats(32..32)
-        .finish();
+        .build();
     assert_eq!(
         a.lane_of(),
-        TypeSetBuilder::new().ints(8..8).floats(32..32).finish()
+        TypeSetBuilder::new().ints(8..8).floats(32..32).build()
     );
 
     // Test as_bool with disjoint intervals.
-    let mut a_as_bool = TypeSetBuilder::new().simd_lanes(2..8).finish();
+    let mut a_as_bool = TypeSetBuilder::new().simd_lanes(2..8).build();
     a_as_bool.bools = num_set![8, 32];
     assert_eq!(a.as_bool(), a_as_bool);
 
@@ -1001,93 +1001,93 @@ fn test_as_bool() {
         .simd_lanes(1..8)
         .ints(8..8)
         .floats(32..32)
-        .finish();
-    let mut b_as_bool = TypeSetBuilder::new().simd_lanes(1..8).finish();
+        .build();
+    let mut b_as_bool = TypeSetBuilder::new().simd_lanes(1..8).build();
     b_as_bool.bools = num_set![1, 8, 32];
     assert_eq!(b.as_bool(), b_as_bool);
 }
 
 #[test]
 fn test_forward_images() {
-    let empty_set = TypeSetBuilder::new().finish();
+    let empty_set = TypeSetBuilder::new().build();
 
     // Half vector.
     assert_eq!(
         TypeSetBuilder::new()
             .simd_lanes(1..32)
-            .finish()
+            .build()
             .half_vector(),
-        TypeSetBuilder::new().simd_lanes(1..16).finish()
+        TypeSetBuilder::new().simd_lanes(1..16).build()
     );
 
     // Double vector.
     assert_eq!(
         TypeSetBuilder::new()
             .simd_lanes(1..32)
-            .finish()
+            .build()
             .double_vector(),
-        TypeSetBuilder::new().simd_lanes(2..64).finish()
+        TypeSetBuilder::new().simd_lanes(2..64).build()
     );
     assert_eq!(
         TypeSetBuilder::new()
             .simd_lanes(128..256)
-            .finish()
+            .build()
             .double_vector(),
-        TypeSetBuilder::new().simd_lanes(256..256).finish()
+        TypeSetBuilder::new().simd_lanes(256..256).build()
     );
 
     // Half width.
     assert_eq!(
-        TypeSetBuilder::new().ints(8..32).finish().half_width(),
-        TypeSetBuilder::new().ints(8..16).finish()
+        TypeSetBuilder::new().ints(8..32).build().half_width(),
+        TypeSetBuilder::new().ints(8..16).build()
     );
     assert_eq!(
-        TypeSetBuilder::new().floats(32..32).finish().half_width(),
+        TypeSetBuilder::new().floats(32..32).build().half_width(),
         empty_set
     );
     assert_eq!(
-        TypeSetBuilder::new().floats(32..64).finish().half_width(),
-        TypeSetBuilder::new().floats(32..32).finish()
+        TypeSetBuilder::new().floats(32..64).build().half_width(),
+        TypeSetBuilder::new().floats(32..32).build()
     );
     assert_eq!(
-        TypeSetBuilder::new().bools(1..8).finish().half_width(),
+        TypeSetBuilder::new().bools(1..8).build().half_width(),
         empty_set
     );
     assert_eq!(
-        TypeSetBuilder::new().bools(1..32).finish().half_width(),
-        TypeSetBuilder::new().bools(8..16).finish()
+        TypeSetBuilder::new().bools(1..32).build().half_width(),
+        TypeSetBuilder::new().bools(8..16).build()
     );
 
     // Double width.
     assert_eq!(
-        TypeSetBuilder::new().ints(8..32).finish().double_width(),
-        TypeSetBuilder::new().ints(16..64).finish()
+        TypeSetBuilder::new().ints(8..32).build().double_width(),
+        TypeSetBuilder::new().ints(16..64).build()
     );
     assert_eq!(
-        TypeSetBuilder::new().ints(32..64).finish().double_width(),
-        TypeSetBuilder::new().ints(64..64).finish()
+        TypeSetBuilder::new().ints(32..64).build().double_width(),
+        TypeSetBuilder::new().ints(64..64).build()
     );
     assert_eq!(
-        TypeSetBuilder::new().floats(32..32).finish().double_width(),
-        TypeSetBuilder::new().floats(64..64).finish()
+        TypeSetBuilder::new().floats(32..32).build().double_width(),
+        TypeSetBuilder::new().floats(64..64).build()
     );
     assert_eq!(
-        TypeSetBuilder::new().floats(32..64).finish().double_width(),
-        TypeSetBuilder::new().floats(64..64).finish()
+        TypeSetBuilder::new().floats(32..64).build().double_width(),
+        TypeSetBuilder::new().floats(64..64).build()
     );
     assert_eq!(
-        TypeSetBuilder::new().bools(1..16).finish().double_width(),
-        TypeSetBuilder::new().bools(16..32).finish()
+        TypeSetBuilder::new().bools(1..16).build().double_width(),
+        TypeSetBuilder::new().bools(16..32).build()
     );
     assert_eq!(
-        TypeSetBuilder::new().bools(32..64).finish().double_width(),
-        TypeSetBuilder::new().bools(64..64).finish()
+        TypeSetBuilder::new().bools(32..64).build().double_width(),
+        TypeSetBuilder::new().bools(64..64).build()
     );
 }
 
 #[test]
 fn test_backward_images() {
-    let empty_set = TypeSetBuilder::new().finish();
+    let empty_set = TypeSetBuilder::new().build();
 
     // LaneOf.
     assert_eq!(
@@ -1095,13 +1095,13 @@ fn test_backward_images() {
             .simd_lanes(1..1)
             .ints(8..8)
             .floats(32..32)
-            .finish()
+            .build()
             .preimage(DerivedFunc::LaneOf),
         TypeSetBuilder::new()
             .simd_lanes(Interval::All)
             .ints(8..8)
             .floats(32..32)
-            .finish()
+            .build()
     );
     assert_eq!(empty_set.preimage(DerivedFunc::LaneOf), empty_set);
 
@@ -1110,14 +1110,14 @@ fn test_backward_images() {
         TypeSetBuilder::new()
             .simd_lanes(1..4)
             .bools(1..64)
-            .finish()
+            .build()
             .preimage(DerivedFunc::AsBool),
         TypeSetBuilder::new()
             .simd_lanes(1..4)
             .ints(Interval::All)
             .bools(Interval::All)
             .floats(Interval::All)
-            .finish()
+            .build()
     );
 
     // Double vector.
@@ -1125,7 +1125,7 @@ fn test_backward_images() {
         TypeSetBuilder::new()
             .simd_lanes(1..1)
             .ints(8..8)
-            .finish()
+            .build()
             .preimage(DerivedFunc::DoubleVector)
             .size(),
         0
@@ -1135,13 +1135,13 @@ fn test_backward_images() {
             .simd_lanes(1..16)
             .ints(8..16)
             .floats(32..32)
-            .finish()
+            .build()
             .preimage(DerivedFunc::DoubleVector),
         TypeSetBuilder::new()
             .simd_lanes(1..8)
             .ints(8..16)
             .floats(32..32)
-            .finish(),
+            .build(),
     );
 
     // Half vector.
@@ -1149,7 +1149,7 @@ fn test_backward_images() {
         TypeSetBuilder::new()
             .simd_lanes(256..256)
             .ints(8..8)
-            .finish()
+            .build()
             .preimage(DerivedFunc::HalfVector)
             .size(),
         0
@@ -1158,12 +1158,12 @@ fn test_backward_images() {
         TypeSetBuilder::new()
             .simd_lanes(64..128)
             .bools(1..32)
-            .finish()
+            .build()
             .preimage(DerivedFunc::HalfVector),
         TypeSetBuilder::new()
             .simd_lanes(128..256)
             .bools(1..32)
-            .finish(),
+            .build(),
     );
 
     // Half width.
@@ -1172,7 +1172,7 @@ fn test_backward_images() {
             .ints(64..64)
             .floats(64..64)
             .bools(64..64)
-            .finish()
+            .build()
             .preimage(DerivedFunc::HalfWidth)
             .size(),
         0
@@ -1181,12 +1181,12 @@ fn test_backward_images() {
         TypeSetBuilder::new()
             .simd_lanes(64..256)
             .bools(1..64)
-            .finish()
+            .build()
             .preimage(DerivedFunc::HalfWidth),
         TypeSetBuilder::new()
             .simd_lanes(64..256)
             .bools(16..64)
-            .finish(),
+            .build(),
     );
 
     // Double width.
@@ -1195,7 +1195,7 @@ fn test_backward_images() {
             .ints(8..8)
             .floats(32..32)
             .bools(1..8)
-            .finish()
+            .build()
             .preimage(DerivedFunc::DoubleWidth)
             .size(),
         0
@@ -1205,13 +1205,13 @@ fn test_backward_images() {
             .simd_lanes(1..16)
             .ints(8..16)
             .floats(32..64)
-            .finish()
+            .build()
             .preimage(DerivedFunc::DoubleWidth),
         TypeSetBuilder::new()
             .simd_lanes(1..16)
             .ints(8..8)
             .floats(32..32)
-            .finish()
+            .build()
     );
 }
 
@@ -1221,7 +1221,7 @@ fn test_typeset_singleton_panic_nonsingleton_types() {
     TypeSetBuilder::new()
         .ints(8..8)
         .floats(32..32)
-        .finish()
+        .build()
         .get_singleton();
 }
 
@@ -1231,7 +1231,7 @@ fn test_typeset_singleton_panic_nonsingleton_lanes() {
     TypeSetBuilder::new()
         .simd_lanes(1..2)
         .floats(32..32)
-        .finish()
+        .build()
         .get_singleton();
 }
 
@@ -1239,25 +1239,22 @@ fn test_typeset_singleton_panic_nonsingleton_lanes() {
 fn test_typeset_singleton() {
     use crate::shared::types as shared_types;
     assert_eq!(
-        TypeSetBuilder::new().ints(16..16).finish().get_singleton(),
+        TypeSetBuilder::new().ints(16..16).build().get_singleton(),
         ValueType::Lane(shared_types::Int::I16.into())
     );
     assert_eq!(
-        TypeSetBuilder::new()
-            .floats(64..64)
-            .finish()
-            .get_singleton(),
+        TypeSetBuilder::new().floats(64..64).build().get_singleton(),
         ValueType::Lane(shared_types::Float::F64.into())
     );
     assert_eq!(
-        TypeSetBuilder::new().bools(1..1).finish().get_singleton(),
+        TypeSetBuilder::new().bools(1..1).build().get_singleton(),
         ValueType::Lane(shared_types::Bool::B1.into())
     );
     assert_eq!(
         TypeSetBuilder::new()
             .simd_lanes(4..4)
             .ints(32..32)
-            .finish()
+            .build()
             .get_singleton(),
         LaneType::from(shared_types::Int::I32).by(4)
     );
@@ -1268,7 +1265,7 @@ fn test_typevar_functions() {
     let x = TypeVar::new(
         "x",
         "i16 and up",
-        TypeSetBuilder::new().ints(16..64).finish(),
+        TypeSetBuilder::new().ints(16..64).build(),
     );
     assert_eq!(x.half_width().name, "half_width(x)");
     assert_eq!(
@@ -1276,7 +1273,7 @@ fn test_typevar_functions() {
         "double_width(half_width(x))"
     );
 
-    let x = TypeVar::new("x", "up to i32", TypeSetBuilder::new().ints(8..32).finish());
+    let x = TypeVar::new("x", "up to i32", TypeSetBuilder::new().ints(8..32).build());
     assert_eq!(x.double_width().name, "double_width(x)");
 }
 
diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index 811d4517cd..0cc0c6a03c 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -348,7 +348,7 @@ impl TransformGroupBuilder {
         self.transforms.push(transform);
     }
 
-    pub fn finish_and_add_to(self, owner: &mut TransformGroups) -> TransformGroupIndex {
+    pub fn build_and_add_to(self, owner: &mut TransformGroups) -> TransformGroupIndex {
         let next_id = owner.next_key();
         owner.add(TransformGroup {
             name: self.name,
@@ -410,7 +410,7 @@ fn test_double_custom_legalization() {
     format.insert(InstructionFormatBuilder::new("nullary"));
     let mut inst_group = InstructionGroupBuilder::new("test", "", &format);
     inst_group.push(InstructionBuilder::new("dummy", "doc"));
-    let inst_group = inst_group.finish();
+    let inst_group = inst_group.build();
     let dummy_inst = inst_group.by_name("dummy");
 
     let mut transform_group = TransformGroupBuilder::new("test", "doc");
diff --git a/cranelift/codegen/meta/src/isa/arm32/mod.rs b/cranelift/codegen/meta/src/isa/arm32/mod.rs
index 2115616c26..ee238866f7 100644
--- a/cranelift/codegen/meta/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm32/mod.rs
@@ -8,7 +8,7 @@ use crate::shared::Definitions as SharedDefinitions;
 
 fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     let setting = SettingGroupBuilder::new("arm32");
-    setting.finish()
+    setting.build()
 }
 
 fn define_regs() -> IsaRegs {
@@ -45,7 +45,7 @@ fn define_regs() -> IsaRegs {
     let builder = RegClassBuilder::new_toplevel("FLAG", flag_reg);
     regs.add_class(builder);
 
-    regs.finish()
+    regs.build()
 }
 
 pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
@@ -57,7 +57,7 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
         "arm32 specific instruction set",
         &shared_defs.format_registry,
     )
-    .finish();
+    .build();
 
     // CPU modes for 32-bit ARM and Thumb2.
     let mut a32 = CpuMode::new("A32");
diff --git a/cranelift/codegen/meta/src/isa/arm64/mod.rs b/cranelift/codegen/meta/src/isa/arm64/mod.rs
index 07518bf986..5a11e69848 100644
--- a/cranelift/codegen/meta/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm64/mod.rs
@@ -8,7 +8,7 @@ use crate::shared::Definitions as SharedDefinitions;
 
 fn define_settings(_shared: &SettingGroup) -> SettingGroup {
     let setting = SettingGroupBuilder::new("arm64");
-    setting.finish()
+    setting.build()
 }
 
 fn define_registers() -> IsaRegs {
@@ -41,7 +41,7 @@ fn define_registers() -> IsaRegs {
     let builder = RegClassBuilder::new_toplevel("FLAG", flag_reg);
     regs.add_class(builder);
 
-    regs.finish()
+    regs.build()
 }
 
 pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
@@ -53,7 +53,7 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
         "arm64 specific instruction set",
         &shared_defs.format_registry,
     )
-    .finish();
+    .build();
 
     let mut a64 = CpuMode::new("A64");
 
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index 66390c4c43..776224d74f 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -57,7 +57,7 @@ fn define_settings(shared: &SettingGroup) -> SettingGroup {
         predicate!(shared_enable_simd && supports_f && supports_d),
     );
 
-    setting.finish()
+    setting.build()
 }
 
 fn define_registers() -> IsaRegs {
@@ -79,7 +79,7 @@ fn define_registers() -> IsaRegs {
     let builder = RegClassBuilder::new_toplevel("FPR", float_regs);
     regs.add_class(builder);
 
-    regs.finish()
+    regs.build()
 }
 
 pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
@@ -91,7 +91,7 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
         "riscv specific instruction set",
         &shared_defs.format_registry,
     )
-    .finish();
+    .build();
 
     // CPU modes for 32-bit and 64-bit operation.
     let mut rv_32 = CpuMode::new("RV32");
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index e66b7752f7..1fcf37a68f 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -18,7 +18,7 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
     let iWord = &TypeVar::new(
         "iWord",
         "A scalar integer machine word",
-        TypeSetBuilder::new().ints(32..64).finish(),
+        TypeSetBuilder::new().ints(32..64).build(),
     );
     let nlo = &operand_doc("nlo", iWord, "Low part of numerator");
     let nhi = &operand_doc("nhi", iWord, "High part of numerator");
@@ -103,7 +103,7 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
         TypeSetBuilder::new()
             .floats(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
     let IntTo = &TypeVar::new(
         "IntTo",
@@ -111,7 +111,7 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
         TypeSetBuilder::new()
             .ints(32..64)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
     let x = &operand("x", Float);
     let a = &operand("a", IntTo);
@@ -242,5 +242,5 @@ pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
         .operands_out(vec![y, rflags]),
     );
 
-    ig.finish()
+    ig.build()
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index df6ea99885..2815665dac 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -289,5 +289,5 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
         ],
     );
 
-    group.finish_and_add_to(&mut shared.transform_groups);
+    group.build_and_add_to(&mut shared.transform_groups);
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/registers.rs b/cranelift/codegen/meta/src/isa/x86/registers.rs
index 350a5d904e..3039bafba2 100644
--- a/cranelift/codegen/meta/src/isa/x86/registers.rs
+++ b/cranelift/codegen/meta/src/isa/x86/registers.rs
@@ -38,5 +38,5 @@ pub fn define() -> IsaRegs {
     let builder = RegClassBuilder::subclass_of("FPR8", fpr, 0, 8);
     regs.add_class(builder);
 
-    regs.finish()
+    regs.build()
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/settings.rs b/cranelift/codegen/meta/src/isa/x86/settings.rs
index f9b1908d11..52486d7de3 100644
--- a/cranelift/codegen/meta/src/isa/x86/settings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/settings.rs
@@ -83,5 +83,5 @@ pub fn define(shared: &SettingGroup) -> SettingGroup {
         ),
     );
 
-    settings.finish()
+    settings.build()
 }
diff --git a/cranelift/codegen/meta/src/shared/entities.rs b/cranelift/codegen/meta/src/shared/entities.rs
index e3ff8ac705..358ef8f23e 100644
--- a/cranelift/codegen/meta/src/shared/entities.rs
+++ b/cranelift/codegen/meta/src/shared/entities.rs
@@ -12,39 +12,39 @@ pub fn define() -> Vec {
     // This is primarliy used in control flow instructions.
     let ebb = create("ebb", "An extended basic block in the same function.")
         .default_member("destination")
-        .finish();
+        .build();
     kinds.push(ebb);
 
     // A reference to a stack slot declared in the function preamble.
-    let stack_slot = create("stack_slot", "A stack slot").finish();
+    let stack_slot = create("stack_slot", "A stack slot").build();
     kinds.push(stack_slot);
 
     // A reference to a global value.
-    let global_value = create("global_value", "A global value.").finish();
+    let global_value = create("global_value", "A global value.").build();
     kinds.push(global_value);
 
     // A reference to a function signature declared in the function preamble.
     // This is used to provide the call signature in a call_indirect instruction.
-    let sig_ref = create("sig_ref", "A function signature.").finish();
+    let sig_ref = create("sig_ref", "A function signature.").build();
     kinds.push(sig_ref);
 
     // A reference to an external function declared in the function preamble.
     // This is used to provide the callee and signature in a call instruction.
-    let func_ref = create("func_ref", "An external function.").finish();
+    let func_ref = create("func_ref", "An external function.").build();
     kinds.push(func_ref);
 
     // A reference to a jump table declared in the function preamble.
     let jump_table = create("jump_table", "A jump table.")
         .default_member("table")
-        .finish();
+        .build();
     kinds.push(jump_table);
 
     // A reference to a heap declared in the function preamble.
-    let heap = create("heap", "A heap.").finish();
+    let heap = create("heap", "A heap.").build();
     kinds.push(heap);
 
     // A reference to a table declared in the function preamble.
-    let table = create("table", "A table.").finish();
+    let table = create("table", "A table.").build();
     kinds.push(table);
 
     // A variable-sized list of value operands. Use for Ebb and function call arguments.
@@ -58,7 +58,7 @@ pub fn define() -> Vec {
             returned from an instruction.
         "#,
         )
-        .finish();
+        .build();
     kinds.push(varargs);
 
     return kinds;
diff --git a/cranelift/codegen/meta/src/shared/immediates.rs b/cranelift/codegen/meta/src/shared/immediates.rs
index c31ed11c7f..bee762a9e5 100644
--- a/cranelift/codegen/meta/src/shared/immediates.rs
+++ b/cranelift/codegen/meta/src/shared/immediates.rs
@@ -11,7 +11,7 @@ pub fn define() -> Vec {
     // IntType type.
     let imm64 = Builder::new_imm("imm64")
         .doc("A 64-bit immediate integer.")
-        .finish();
+        .build();
     kinds.push(imm64);
 
     // An unsigned 8-bit immediate integer operand.
@@ -20,13 +20,13 @@ pub fn define() -> Vec {
     // immediate bit counts on shift instructions.
     let uimm8 = Builder::new_imm("uimm8")
         .doc("An 8-bit immediate unsigned integer.")
-        .finish();
+        .build();
     kinds.push(uimm8);
 
     // An unsigned 32-bit immediate integer operand.
     let uimm32 = Builder::new_imm("uimm32")
         .doc("A 32-bit immediate unsigned integer.")
-        .finish();
+        .build();
     kinds.push(uimm32);
 
     // A 32-bit immediate signed offset.
@@ -36,7 +36,7 @@ pub fn define() -> Vec {
     let offset32 = Builder::new_imm("offset32")
         .doc("A 32-bit immediate signed offset.")
         .default_member("offset")
-        .finish();
+        .build();
     kinds.push(offset32);
 
     // A 32-bit immediate floating point operand.
@@ -44,7 +44,7 @@ pub fn define() -> Vec {
     // IEEE 754-2008 binary32 interchange format.
     let ieee32 = Builder::new_imm("ieee32")
         .doc("A 32-bit immediate floating point number.")
-        .finish();
+        .build();
     kinds.push(ieee32);
 
     // A 64-bit immediate floating point operand.
@@ -52,7 +52,7 @@ pub fn define() -> Vec {
     // IEEE 754-2008 binary64 interchange format.
     let ieee64 = Builder::new_imm("ieee64")
         .doc("A 64-bit immediate floating point number.")
-        .finish();
+        .build();
     kinds.push(ieee64);
 
     // An immediate boolean operand.
@@ -62,7 +62,7 @@ pub fn define() -> Vec {
     let boolean = Builder::new_imm("boolean")
         .doc("An immediate boolean.")
         .rust_type("bool")
-        .finish();
+        .build();
     kinds.push(boolean);
 
     // A condition code for comparing integer values.
@@ -83,7 +83,7 @@ pub fn define() -> Vec {
         .doc("An integer comparison condition code.")
         .default_member("cond")
         .rust_type("ir::condcodes::IntCC")
-        .finish();
+        .build();
     kinds.push(intcc);
 
     // A condition code for comparing floating point values.  This enumerated operand kind is used
@@ -107,7 +107,7 @@ pub fn define() -> Vec {
         .doc("A floating point comparison condition code")
         .default_member("cond")
         .rust_type("ir::condcodes::FloatCC")
-        .finish();
+        .build();
     kinds.push(floatcc);
 
     // Flags for memory operations like :clif:inst:`load` and :clif:inst:`store`.
@@ -115,14 +115,14 @@ pub fn define() -> Vec {
         .doc("Memory operation flags")
         .default_member("flags")
         .rust_type("ir::MemFlags")
-        .finish();
+        .build();
     kinds.push(memflags);
 
     // A register unit in the current target ISA.
     let regunit = Builder::new_imm("regunit")
         .doc("A register unit in the target ISA")
         .rust_type("isa::RegUnit")
-        .finish();
+        .build();
     kinds.push(regunit);
 
     // A trap code indicating the reason for trapping.
@@ -138,7 +138,7 @@ pub fn define() -> Vec {
         .doc("A trap reason code.")
         .default_member("code")
         .rust_type("ir::TrapCode")
-        .finish();
+        .build();
     kinds.push(trapcode);
 
     return kinds;
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 5c810b9979..284594f6a8 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -56,7 +56,7 @@ pub fn define(
         TypeSetBuilder::new()
             .ints(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
 
     let Bool = &TypeVar::new(
@@ -65,19 +65,19 @@ pub fn define(
         TypeSetBuilder::new()
             .bools(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
 
     let iB = &TypeVar::new(
         "iB",
         "A scalar integer type",
-        TypeSetBuilder::new().ints(Interval::All).finish(),
+        TypeSetBuilder::new().ints(Interval::All).build(),
     );
 
     let iAddr = &TypeVar::new(
         "iAddr",
         "An integer address type",
-        TypeSetBuilder::new().ints(32..64).finish(),
+        TypeSetBuilder::new().ints(32..64).build(),
     );
 
     let Testable = &TypeVar::new(
@@ -86,7 +86,7 @@ pub fn define(
         TypeSetBuilder::new()
             .ints(Interval::All)
             .bools(Interval::All)
-            .finish(),
+            .build(),
     );
 
     let TxN = &TypeVar::new(
@@ -98,7 +98,7 @@ pub fn define(
             .bools(Interval::All)
             .simd_lanes(Interval::All)
             .includes_scalars(false)
-            .finish(),
+            .build(),
     );
 
     let Any = &TypeVar::new(
@@ -110,7 +110,7 @@ pub fn define(
             .bools(Interval::All)
             .simd_lanes(Interval::All)
             .includes_scalars(true)
-            .finish(),
+            .build(),
     );
 
     let Mem = &TypeVar::new(
@@ -120,7 +120,7 @@ pub fn define(
             .ints(Interval::All)
             .floats(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
 
     let MemTo = &TypeVar::new(
@@ -130,7 +130,7 @@ pub fn define(
             .ints(Interval::All)
             .floats(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
 
     let addr = &operand("addr", iAddr);
@@ -261,7 +261,7 @@ pub fn define(
     let Entry = &TypeVar::new(
         "Entry",
         "A scalar integer type",
-        TypeSetBuilder::new().ints(Interval::All).finish(),
+        TypeSetBuilder::new().ints(Interval::All).build(),
     );
 
     let entry = &operand_doc("entry", Entry, "entry of jump table");
@@ -589,7 +589,7 @@ pub fn define(
     let iExt8 = &TypeVar::new(
         "iExt8",
         "An integer type with more than 8 bits",
-        TypeSetBuilder::new().ints(16..64).finish(),
+        TypeSetBuilder::new().ints(16..64).build(),
     );
     let x = &operand("x", iExt8);
     let a = &operand("a", iExt8);
@@ -679,7 +679,7 @@ pub fn define(
     let iExt16 = &TypeVar::new(
         "iExt16",
         "An integer type with more than 16 bits",
-        TypeSetBuilder::new().ints(32..64).finish(),
+        TypeSetBuilder::new().ints(32..64).build(),
     );
     let x = &operand("x", iExt16);
     let a = &operand("a", iExt16);
@@ -769,7 +769,7 @@ pub fn define(
     let iExt32 = &TypeVar::new(
         "iExt32",
         "An integer type with more than 32 bits",
-        TypeSetBuilder::new().ints(64..64).finish(),
+        TypeSetBuilder::new().ints(64..64).build(),
     );
     let x = &operand("x", iExt32);
     let a = &operand("a", iExt32);
@@ -939,7 +939,7 @@ pub fn define(
     let HeapOffset = &TypeVar::new(
         "HeapOffset",
         "An unsigned heap offset",
-        TypeSetBuilder::new().ints(32..64).finish(),
+        TypeSetBuilder::new().ints(32..64).build(),
     );
 
     let H = &operand("H", heap);
@@ -968,7 +968,7 @@ pub fn define(
     let TableOffset = &TypeVar::new(
         "TableOffset",
         "An unsigned table offset",
-        TypeSetBuilder::new().ints(32..64).finish(),
+        TypeSetBuilder::new().ints(32..64).build(),
     );
     let T = &operand("T", table);
     let p = &operand("p", TableOffset);
@@ -1337,7 +1337,7 @@ pub fn define(
             .bools(Interval::All)
             .simd_lanes(1..128)
             .includes_scalars(true)
-            .finish(),
+            .build(),
     );
 
     let x = &operand_doc("x", Any128, "Low-numbered lanes");
@@ -1903,7 +1903,7 @@ pub fn define(
             .bools(Interval::All)
             .simd_lanes(Interval::All)
             .includes_scalars(true)
-            .finish(),
+            .build(),
     );
     let x = &operand("x", bits);
     let y = &operand("y", bits);
@@ -2272,7 +2272,7 @@ pub fn define(
         TypeSetBuilder::new()
             .floats(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
     let Cond = &operand("Cond", floatcc);
     let x = &operand("x", Float);
@@ -2628,7 +2628,7 @@ pub fn define(
         TypeSetBuilder::new()
             .bools(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
 
     let BoolTo = &TypeVar::new(
@@ -2637,7 +2637,7 @@ pub fn define(
         TypeSetBuilder::new()
             .bools(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
 
     let x = &operand("x", Bool);
@@ -2665,7 +2665,7 @@ pub fn define(
         TypeSetBuilder::new()
             .bools(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
     let x = &operand("x", Bool);
     let a = &operand("a", BoolTo);
@@ -2692,7 +2692,7 @@ pub fn define(
         TypeSetBuilder::new()
             .ints(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
     let x = &operand("x", Bool);
     let a = &operand("a", IntTo);
@@ -2731,7 +2731,7 @@ pub fn define(
         TypeSetBuilder::new()
             .ints(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
 
     let IntTo = &TypeVar::new(
@@ -2740,7 +2740,7 @@ pub fn define(
         TypeSetBuilder::new()
             .ints(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
     let x = &operand("x", Int);
     let a = &operand("a", IntTo);
@@ -2771,7 +2771,7 @@ pub fn define(
         TypeSetBuilder::new()
             .ints(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
     let x = &operand("x", Int);
     let a = &operand("a", IntTo);
@@ -2822,7 +2822,7 @@ pub fn define(
         TypeSetBuilder::new()
             .floats(Interval::All)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
     let x = &operand("x", Float);
     let a = &operand("a", FloatTo);
@@ -2976,7 +2976,7 @@ pub fn define(
         TypeSetBuilder::new()
             .ints(16..64)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
     let x = &operand("x", WideInt);
     let lo = &operand_doc("lo", &WideInt.half_width(), "The low bits of `x`");
@@ -3006,7 +3006,7 @@ pub fn define(
         TypeSetBuilder::new()
             .ints(8..32)
             .simd_lanes(Interval::All)
-            .finish(),
+            .build(),
     );
 
     let lo = &operand("lo", NarrowInt);
@@ -3033,5 +3033,5 @@ pub fn define(
         .is_ghost(true),
     );
 
-    ig.finish()
+    ig.build()
 }
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 7d452473a2..d8ec65e3b7 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -766,8 +766,8 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
 
     let mut groups = TransformGroups::new();
 
-    narrow.finish_and_add_to(&mut groups);
-    let expand_id = expand.finish_and_add_to(&mut groups);
+    narrow.build_and_add_to(&mut groups);
+    let expand_id = expand.build_and_add_to(&mut groups);
 
     // Expansions using CPU flags.
     let mut expand_flags = TransformGroupBuilder::new(
@@ -802,12 +802,12 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
         ],
     );
 
-    expand_flags.finish_and_add_to(&mut groups);
+    expand_flags.build_and_add_to(&mut groups);
 
     // XXX The order of declarations unfortunately matters to be compatible with the Python code.
-    // When it's all migrated, we can put this next to the narrow/expand finish_and_add_to calls
+    // When it's all migrated, we can put this next to the narrow/expand build_and_add_to calls
     // above.
-    widen.finish_and_add_to(&mut groups);
+    widen.build_and_add_to(&mut groups);
 
     groups
 }
diff --git a/cranelift/codegen/meta/src/shared/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
index c89efa6b02..6e3af70623 100644
--- a/cranelift/codegen/meta/src/shared/settings.rs
+++ b/cranelift/codegen/meta/src/shared/settings.rs
@@ -160,5 +160,5 @@ pub fn define() -> SettingGroup {
         true,
     );
 
-    settings.finish()
+    settings.build()
 }

From 420850adf0b9558a5fbff443adb9ff2537b6caac Mon Sep 17 00:00:00 2001
From: Lars T Hansen 
Date: Tue, 21 May 2019 15:53:46 +0200
Subject: [PATCH 2449/3084] Record information about sections of emitted
 code+data.

The result of the emitter is a vector of bytes holding machine code,
jump tables, and (in the future) other read-only data.  Some clients,
notably Firefox's Wasm compiler, needs to separate the machine code
from the data in order to insert more code directly after the code
generated by Cranelift.

To make such separation possible, we record more information about the
emitted bytes: the sizes of each of the sections of code, jump tables,
and read-only data, as well as the locations within the code that
reference (PC-relatively) the jump tables and read-only data.
---
 cranelift/codegen/src/binemit/memorysink.rs   | 59 +++++++++++--------
 cranelift/codegen/src/binemit/mod.rs          | 50 +++++++++++++++-
 cranelift/codegen/src/binemit/relaxation.rs   | 21 ++++++-
 cranelift/codegen/src/context.rs              | 34 ++++++-----
 cranelift/codegen/src/isa/x86/binemit.rs      |  1 +
 cranelift/faerie/src/backend.rs               |  6 +-
 .../filetests/filetests/isa/x86/binary64.clif |  4 +-
 cranelift/filetests/src/test_binemit.rs       | 21 ++++---
 cranelift/filetests/src/test_compile.rs       | 13 ++--
 cranelift/module/src/module.rs                | 11 ++--
 cranelift/src/compile.rs                      | 11 +++-
 cranelift/src/wasm.rs                         |  8 +--
 12 files changed, 166 insertions(+), 73 deletions(-)

diff --git a/cranelift/codegen/src/binemit/memorysink.rs b/cranelift/codegen/src/binemit/memorysink.rs
index 18aa578f09..8e07db01f6 100644
--- a/cranelift/codegen/src/binemit/memorysink.rs
+++ b/cranelift/codegen/src/binemit/memorysink.rs
@@ -14,7 +14,7 @@
 //! relocations to a `RelocSink` trait object. Relocations are less frequent than the
 //! `CodeSink::put*` methods, so the performance impact of the virtual callbacks is less severe.
 
-use super::{Addend, CodeOffset, CodeSink, Reloc};
+use super::{Addend, CodeInfo, CodeOffset, CodeSink, Reloc};
 use crate::ir::{ExternalName, JumpTable, SourceLoc, TrapCode};
 use core::ptr::write_unaligned;
 
@@ -30,12 +30,14 @@ use core::ptr::write_unaligned;
 /// Note that `MemoryCodeSink` writes multi-byte values in the native byte order of the host. This
 /// is not the right thing to do for cross compilation.
 pub struct MemoryCodeSink<'a> {
+    /// Pointer to start of sink's preallocated memory.
     data: *mut u8,
+    /// Offset is isize because its major consumer needs it in that form.
     offset: isize,
-    /// Size of the machine code portion of output
-    pub code_size: isize,
     relocs: &'a mut RelocSink,
     traps: &'a mut TrapSink,
+    /// Information about the generated code and read-only data.
+    pub info: CodeInfo,
 }
 
 impl<'a> MemoryCodeSink<'a> {
@@ -47,7 +49,12 @@ impl<'a> MemoryCodeSink<'a> {
         Self {
             data,
             offset: 0,
-            code_size: 0,
+            info: CodeInfo {
+                code_size: 0,
+                jumptables_size: 0,
+                rodata_size: 0,
+                total_size: 0,
+            },
             relocs,
             traps,
         }
@@ -75,40 +82,35 @@ pub trait TrapSink {
     fn trap(&mut self, _: CodeOffset, _: SourceLoc, _: TrapCode);
 }
 
+impl<'a> MemoryCodeSink<'a> {
+    fn write(&mut self, x: T) {
+        unsafe {
+            #[cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
+            write_unaligned(self.data.offset(self.offset) as *mut T, x);
+            self.offset += std::mem::size_of::() as isize;
+        }
+    }
+}
+
 impl<'a> CodeSink for MemoryCodeSink<'a> {
     fn offset(&self) -> CodeOffset {
         self.offset as CodeOffset
     }
 
     fn put1(&mut self, x: u8) {
-        unsafe {
-            write_unaligned(self.data.offset(self.offset), x);
-        }
-        self.offset += 1;
+        self.write(x);
     }
 
     fn put2(&mut self, x: u16) {
-        unsafe {
-            #[cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
-            write_unaligned(self.data.offset(self.offset) as *mut u16, x);
-        }
-        self.offset += 2;
+        self.write(x);
     }
 
     fn put4(&mut self, x: u32) {
-        unsafe {
-            #[cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
-            write_unaligned(self.data.offset(self.offset) as *mut u32, x);
-        }
-        self.offset += 4;
+        self.write(x);
     }
 
     fn put8(&mut self, x: u64) {
-        unsafe {
-            #[cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
-            write_unaligned(self.data.offset(self.offset) as *mut u64, x);
-        }
-        self.offset += 8;
+        self.write(x);
     }
 
     fn reloc_ebb(&mut self, rel: Reloc, ebb_offset: CodeOffset) {
@@ -131,8 +133,17 @@ impl<'a> CodeSink for MemoryCodeSink<'a> {
         self.traps.trap(ofs, srcloc, code);
     }
 
+    fn begin_jumptables(&mut self) {
+        self.info.code_size = self.offset();
+    }
+
     fn begin_rodata(&mut self) {
-        self.code_size = self.offset;
+        self.info.jumptables_size = self.offset() - self.info.code_size;
+    }
+
+    fn end_codegen(&mut self) {
+        self.info.rodata_size = self.offset() - self.info.jumptables_size;
+        self.info.total_size = self.offset();
     }
 }
 
diff --git a/cranelift/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
index 63fa7ef40b..21b07587ed 100644
--- a/cranelift/codegen/src/binemit/mod.rs
+++ b/cranelift/codegen/src/binemit/mod.rs
@@ -33,6 +33,8 @@ pub enum Reloc {
     Abs8,
     /// x86 PC-relative 4-byte
     X86PCRel4,
+    /// x86 PC-relative 4-byte offset to trailing rodata
+    X86PCRelRodata4,
     /// x86 call to PC-relative 4-byte
     X86CallPCRel4,
     /// x86 call to PLT-relative 4-byte
@@ -55,6 +57,7 @@ impl fmt::Display for Reloc {
             Reloc::Abs4 => write!(f, "Abs4"),
             Reloc::Abs8 => write!(f, "Abs8"),
             Reloc::X86PCRel4 => write!(f, "PCRel4"),
+            Reloc::X86PCRelRodata4 => write!(f, "PCRelRodata4"),
             Reloc::X86CallPCRel4 => write!(f, "CallPCRel4"),
             Reloc::X86CallPLTRel4 => write!(f, "CallPLTRel4"),
             Reloc::X86GOTPCRel4 => write!(f, "GOTPCRel4"),
@@ -63,6 +66,38 @@ impl fmt::Display for Reloc {
     }
 }
 
+/// Container for information about a vector of compiled code and its supporting read-only data.
+///
+/// The code starts at offset 0 and is followed optionally by relocatable jump tables and copyable
+/// (raw binary) read-only data.  Any padding between sections is always part of the section that
+/// precedes the boundary between the sections.
+#[derive(PartialEq)]
+pub struct CodeInfo {
+    /// Number of bytes of machine code (the code starts at offset 0).
+    pub code_size: CodeOffset,
+
+    /// Number of bytes of jumptables.
+    pub jumptables_size: CodeOffset,
+
+    /// Number of bytes of rodata.
+    pub rodata_size: CodeOffset,
+
+    /// Number of bytes in total.
+    pub total_size: CodeOffset,
+}
+
+impl CodeInfo {
+    /// Offset of any relocatable jump tables, or equal to rodata if there are no jump tables.
+    pub fn jumptables(&self) -> CodeOffset {
+        self.code_size
+    }
+
+    /// Offset of any copyable read-only data, or equal to total_size if there are no rodata.
+    pub fn rodata(&self) -> CodeOffset {
+        self.code_size + self.jumptables_size
+    }
+}
+
 /// Abstract interface for adding bytes to the code segment.
 ///
 /// A `CodeSink` will receive all of the machine code for a function. It also accepts relocations
@@ -95,8 +130,14 @@ pub trait CodeSink {
     /// Add trap information for the current offset.
     fn trap(&mut self, _: TrapCode, _: SourceLoc);
 
-    /// Code output is complete, read-only data may follow.
+    /// Machine code output is complete, jump table data may follow.
+    fn begin_jumptables(&mut self);
+
+    /// Jump table output is complete, raw read-only data may follow.
     fn begin_rodata(&mut self);
+
+    /// Read-only data output is complete, we're done.
+    fn end_codegen(&mut self);
 }
 
 /// Report a bad encoding error.
@@ -127,7 +168,7 @@ where
         }
     }
 
-    sink.begin_rodata();
+    sink.begin_jumptables();
 
     // output jump tables
     for (jt, jt_data) in func.jump_tables.iter() {
@@ -137,4 +178,9 @@ where
             sink.put4(rel_offset as u32)
         }
     }
+
+    sink.begin_rodata();
+    // TODO: No read-only data (constant pools) at this time.
+
+    sink.end_codegen();
 }
diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index c354a763cd..d1b5815e7c 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -27,7 +27,7 @@
 //! ebb23:
 //! ```
 
-use crate::binemit::CodeOffset;
+use crate::binemit::{CodeInfo, CodeOffset};
 use crate::cursor::{Cursor, FuncCursor};
 use crate::ir::{Function, InstructionData, Opcode};
 use crate::isa::{EncInfo, TargetIsa};
@@ -40,7 +40,7 @@ use log::debug;
 /// Relax branches and compute the final layout of EBB headers in `func`.
 ///
 /// Fill in the `func.offsets` table so the function is ready for binary emission.
-pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> CodegenResult {
+pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> CodegenResult {
     let _tt = timing::relax_branches();
 
     let encinfo = isa.encoding_info();
@@ -109,6 +109,9 @@ pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> CodegenResult CodegenResult,
         relocs: &mut RelocSink,
         traps: &mut TrapSink,
-    ) -> CodegenResult<(CodeOffset, CodeOffset)> {
-        let total_size = self.compile(isa)?;
+    ) -> CodegenResult {
+        let info = self.compile(isa)?;
         let old_len = mem.len();
-        mem.resize(old_len + total_size as usize, 0);
-        let code_size =
+        mem.resize(old_len + info.total_size as usize, 0);
+        let new_info =
             unsafe { self.emit_to_memory(isa, mem.as_mut_ptr().add(old_len), relocs, traps) };
-        Ok((code_size, total_size - code_size))
+        debug_assert!(new_info == info);
+        Ok(info)
     }
 
     /// Compile the function.
@@ -115,8 +116,8 @@ impl Context {
     /// represented by `isa`. This does not include the final step of emitting machine code into a
     /// code sink.
     ///
-    /// Returns the size of the function's code and read-only data.
-    pub fn compile(&mut self, isa: &TargetIsa) -> CodegenResult {
+    /// Returns information about the function's code and read-only data.
+    pub fn compile(&mut self, isa: &TargetIsa) -> CodegenResult {
         let _tt = timing::compile();
         self.verify_if(isa)?;
 
@@ -160,18 +161,18 @@ impl Context {
     /// This function is unsafe since it does not perform bounds checking on the memory buffer,
     /// and it can't guarantee that the `mem` pointer is valid.
     ///
-    /// Returns the size of the function's code.
+    /// Returns information about the emitted code and data.
     pub unsafe fn emit_to_memory(
         &self,
         isa: &TargetIsa,
         mem: *mut u8,
         relocs: &mut RelocSink,
         traps: &mut TrapSink,
-    ) -> CodeOffset {
+    ) -> CodeInfo {
         let _tt = timing::binemit();
         let mut sink = MemoryCodeSink::new(mem, relocs, traps);
         isa.emit_function_to_memory(&self.func, &mut sink);
-        sink.code_size as CodeOffset
+        sink.info
     }
 
     /// Run the verifier on the function.
@@ -325,12 +326,13 @@ impl Context {
         Ok(())
     }
 
-    /// Run the branch relaxation pass and return the final code size.
-    pub fn relax_branches(&mut self, isa: &TargetIsa) -> CodegenResult {
-        let code_size = relax_branches(&mut self.func, isa)?;
+    /// Run the branch relaxation pass and return information about the function's code and
+    /// read-only data.
+    pub fn relax_branches(&mut self, isa: &TargetIsa) -> CodegenResult {
+        let info = relax_branches(&mut self.func, isa)?;
         self.verify_if(isa)?;
         self.verify_locations_if(isa)?;
-        Ok(code_size)
+        Ok(info)
     }
 
     /// Builds ranges and location for specified value labels.
diff --git a/cranelift/codegen/src/isa/x86/binemit.rs b/cranelift/codegen/src/isa/x86/binemit.rs
index 513d6347fa..ef44766d0b 100644
--- a/cranelift/codegen/src/isa/x86/binemit.rs
+++ b/cranelift/codegen/src/isa/x86/binemit.rs
@@ -339,4 +339,5 @@ fn disp4(destination: Ebb, func: &Function, sink: &mut CS
 fn jt_disp4(jt: JumpTable, func: &Function, sink: &mut CS) {
     let delta = func.jt_offsets[jt].wrapping_sub(sink.offset() + 4);
     sink.put4(delta);
+    sink.reloc_jt(Reloc::X86PCRelRodata4, jt);
 }
diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index 047ddeb87d..9313ddaae9 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -138,9 +138,9 @@ impl Backend for FaerieBackend {
         name: &str,
         ctx: &cranelift_codegen::Context,
         namespace: &ModuleNamespace,
-        code_size: u32,
+        total_size: u32,
     ) -> ModuleResult {
-        let mut code: Vec = vec![0; code_size as usize];
+        let mut code: Vec = vec![0; total_size as usize];
 
         // Non-lexical lifetimes would obviate the braces here.
         {
@@ -153,7 +153,7 @@ impl Backend for FaerieBackend {
             };
 
             if let Some(ref mut trap_manifest) = self.trap_manifest {
-                let mut trap_sink = FaerieTrapSink::new(name, code_size);
+                let mut trap_sink = FaerieTrapSink::new(name, total_size);
                 unsafe {
                     ctx.emit_to_memory(
                         &*self.isa,
diff --git a/cranelift/filetests/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
index c2a042ac01..ed1316ad24 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64.clif
@@ -1424,8 +1424,8 @@ function %I64_JT(i64 [%rdi]) {
 ebb0(v0: i64 [%rdi]):
     ; Note: The next two lines will need to change whenever instructions are
     ;        added or removed from this test.
-    [-, %rax]           v1 = jump_table_base.i64 jt0    ; bin: 48 8d 05 00000039
-    [-, %r10]           v2 = jump_table_base.i64 jt0    ; bin: 4c 8d 15 00000032
+    [-, %rax]           v1 = jump_table_base.i64 jt0    ; bin: 48 8d 05 00000039 PCRelRodata4(jt0)
+    [-, %r10]           v2 = jump_table_base.i64 jt0    ; bin: 4c 8d 15 00000032 PCRelRodata4(jt0)
 
     [-, %rbx]           v10 = iconst.i64 1
     [-, %r13]           v11 = iconst.i64 2
diff --git a/cranelift/filetests/src/test_binemit.rs b/cranelift/filetests/src/test_binemit.rs
index 3c959a09bc..a80b3aefe6 100644
--- a/cranelift/filetests/src/test_binemit.rs
+++ b/cranelift/filetests/src/test_binemit.rs
@@ -5,8 +5,7 @@
 
 use crate::match_directive::match_directive;
 use crate::subtest::{Context, SubTest, SubtestResult};
-use cranelift_codegen::binemit;
-use cranelift_codegen::binemit::{CodeSink, RegDiversions};
+use cranelift_codegen::binemit::{self, CodeInfo, CodeSink, RegDiversions};
 use cranelift_codegen::dbg::DisplayList;
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::entities::AnyEntity;
@@ -96,9 +95,12 @@ impl binemit::CodeSink for TextSink {
         write!(self.text, "{} ", code).unwrap();
     }
 
-    fn begin_rodata(&mut self) {
+    fn begin_jumptables(&mut self) {
         self.code_size = self.offset
     }
+
+    fn begin_rodata(&mut self) {}
+    fn end_codegen(&mut self) {}
 }
 
 impl SubTest for TestBinEmit {
@@ -164,7 +166,7 @@ impl SubTest for TestBinEmit {
         }
 
         // Relax branches and compute EBB offsets based on the encodings.
-        let code_size = binemit::relax_branches(&mut func, isa)
+        let CodeInfo { total_size, .. } = binemit::relax_branches(&mut func, isa)
             .map_err(|e| pretty_error(&func, context.isa, e))?;
 
         // Collect all of the 'bin:' directives on instructions.
@@ -288,7 +290,7 @@ impl SubTest for TestBinEmit {
             }
         }
 
-        sink.begin_rodata();
+        sink.begin_jumptables();
 
         for (jt, jt_data) in func.jump_tables.iter() {
             let jt_offset = func.jt_offsets[jt];
@@ -298,10 +300,15 @@ impl SubTest for TestBinEmit {
             }
         }
 
-        if sink.offset != code_size {
+        sink.begin_rodata();
+        // TODO: Read-only (constant pool) data.
+
+        sink.end_codegen();
+
+        if sink.offset != total_size {
             return Err(format!(
                 "Expected code size {}, got {}",
-                code_size, sink.offset
+                total_size, sink.offset
             ));
         }
 
diff --git a/cranelift/filetests/src/test_compile.rs b/cranelift/filetests/src/test_compile.rs
index 7cb9c702a4..fee681d609 100644
--- a/cranelift/filetests/src/test_compile.rs
+++ b/cranelift/filetests/src/test_compile.rs
@@ -4,8 +4,9 @@
 
 use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use cranelift_codegen;
+use cranelift_codegen::binemit::{self, CodeInfo};
+use cranelift_codegen::ir;
 use cranelift_codegen::print_errors::pretty_error;
-use cranelift_codegen::{binemit, ir};
 use cranelift_reader::TestCommand;
 use log::info;
 use std::borrow::Cow;
@@ -38,13 +39,13 @@ impl SubTest for TestCompile {
         let isa = context.isa.expect("compile needs an ISA");
         let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned());
 
-        let code_size = comp_ctx
+        let CodeInfo { total_size, .. } = comp_ctx
             .compile(isa)
             .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?;
 
         info!(
             "Generated {} bytes of code:\n{}",
-            code_size,
+            total_size,
             comp_ctx.func.display(isa)
         );
 
@@ -56,10 +57,10 @@ impl SubTest for TestCompile {
             &mut sink,
         );
 
-        if sink.offset != code_size {
+        if sink.offset != total_size {
             return Err(format!(
                 "Expected code size {}, got {}",
-                code_size, sink.offset
+                total_size, sink.offset
             ));
         }
 
@@ -105,5 +106,7 @@ impl binemit::CodeSink for SizeSink {
     }
     fn reloc_jt(&mut self, _reloc: binemit::Reloc, _jt: ir::JumpTable) {}
     fn trap(&mut self, _code: ir::TrapCode, _srcloc: ir::SourceLoc) {}
+    fn begin_jumptables(&mut self) {}
     fn begin_rodata(&mut self) {}
+    fn end_codegen(&mut self) {}
 }
diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 3da6c37810..5d8f3c5f60 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -8,8 +8,9 @@
 use super::HashMap;
 use crate::data_context::DataContext;
 use crate::Backend;
+use cranelift_codegen::binemit::{self, CodeInfo};
 use cranelift_codegen::entity::{entity_impl, PrimaryMap};
-use cranelift_codegen::{binemit, ir, isa, CodegenError, Context};
+use cranelift_codegen::{ir, isa, CodegenError, Context};
 use failure::Fail;
 use log::info;
 use std::borrow::ToOwned;
@@ -519,7 +520,7 @@ where
 
     /// Define a function, producing the function body from the given `Context`.
     ///
-    /// Returns the size of the function's code.
+    /// Returns the size of the function's code and constant data.
     ///
     /// Note: After calling this function the given `Context` will contain the compiled function.
     pub fn define_function(
@@ -527,7 +528,7 @@ where
         func: FuncId,
         ctx: &mut Context,
     ) -> ModuleResult {
-        let code_size = ctx.compile(self.backend.isa()).map_err(|e| {
+        let CodeInfo { total_size, .. } = ctx.compile(self.backend.isa()).map_err(|e| {
             info!(
                 "defining function {}: {}",
                 func,
@@ -550,12 +551,12 @@ where
             &ModuleNamespace:: {
                 contents: &self.contents,
             },
-            code_size,
+            total_size,
         )?);
 
         self.contents.functions[func].compiled = compiled;
         self.functions_to_finalize.push(func);
-        Ok(code_size)
+        Ok(total_size)
     }
 
     /// Define a function, producing the data contents from the given `DataContext`.
diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs
index f5e34c6417..4c663323b1 100644
--- a/cranelift/src/compile.rs
+++ b/cranelift/src/compile.rs
@@ -65,7 +65,7 @@ fn handle_module(
         let mut mem = vec![];
 
         // Compile and encode the result to machine code.
-        let (code_size, rodata_size) = context
+        let code_info = context
             .compile_and_emit(isa, &mut mem, &mut relocs, &mut traps)
             .map_err(|err| pretty_error(&context.func, Some(isa), err))?;
 
@@ -74,7 +74,14 @@ fn handle_module(
         }
 
         if flag_disasm {
-            print_all(isa, &mem, code_size, rodata_size, &relocs, &traps)?;
+            print_all(
+                isa,
+                &mem,
+                code_info.code_size,
+                code_info.jumptables_size + code_info.rodata_size,
+                &relocs,
+                &traps,
+            )?;
         }
     }
 
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 61a8d8b1a9..dadb13389f 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -176,16 +176,16 @@ fn handle_module(
                 return Err(pretty_verifier_error(&context.func, fisa.isa, None, errors));
             }
         } else {
-            let (code_size, rodata_size) = context
+            let code_info = context
                 .compile_and_emit(isa, &mut mem, &mut relocs, &mut traps)
                 .map_err(|err| pretty_error(&context.func, fisa.isa, err))?;
 
             if flag_print_size {
                 println!(
                     "Function #{} code size: {} bytes",
-                    func_index, code_size + rodata_size
+                    func_index, code_info.total_size,
                 );
-                total_module_code_size += code_size + rodata_size;
+                total_module_code_size += code_info.total_size;
                 println!(
                     "Function #{} bytecode size: {} bytes",
                     func_index,
@@ -194,7 +194,7 @@ fn handle_module(
             }
 
             if flag_print_disasm {
-                saved_sizes = Some((code_size, rodata_size));
+                saved_sizes = Some((code_info.code_size, code_info.jumptables_size + code_info.rodata_size));
             }
         }
 

From da1baf7481a74785eabe170ae5f99dabf96541ba Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 31 May 2019 09:51:11 -0700
Subject: [PATCH 2450/3084] Use `try_from` instead of the cast crate.

Now that `try_from` is in stable Rust, we can use it here.
---
 cranelift/wasm/Cargo.toml                 | 1 -
 cranelift/wasm/src/environ/dummy.rs       | 4 ++--
 cranelift/wasm/src/sections_translator.rs | 3 ++-
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 2309322acb..0130fcaa7a 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -19,7 +19,6 @@ hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
-cast = { version = "0.2.2", default-features = false }
 
 [dev-dependencies]
 wabt = "0.7.0"
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index 7c6625f069..0404ccb522 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -11,7 +11,7 @@ use crate::translation_utils::{
     DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table,
     TableIndex,
 };
-use cast;
+use core::convert::TryFrom;
 use cranelift_codegen::cursor::FuncCursor;
 use cranelift_codegen::ir::immediates::{Offset32, Uimm64};
 use cranelift_codegen::ir::types::*;
@@ -196,7 +196,7 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         index: GlobalIndex,
     ) -> WasmResult {
         // Just create a dummy `vmctx` global.
-        let offset = cast::i32((index.index() * 8) + 8).unwrap().into();
+        let offset = i32::try_from((index.index() * 8) + 8).unwrap().into();
         let vmctx = func.create_global_value(ir::GlobalValueData::VMContext {});
         Ok(GlobalVariable::Memory {
             gv: vmctx,
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index e406ec3eee..929114984a 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -12,6 +12,7 @@ use crate::translation_utils::{
     type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex,
     Table, TableElementType, TableIndex,
 };
+use core::convert::TryFrom;
 use cranelift_codegen::ir::{self, AbiParam, Signature};
 use cranelift_entity::EntityRef;
 use std::vec::Vec;
@@ -270,7 +271,7 @@ pub fn parse_element_section<'data>(
                 ref s => panic!("unsupported init expr in element section: {:?}", s),
             };
             let items_reader = items.get_items_reader()?;
-            let mut elems = Vec::with_capacity(cast::usize(items_reader.get_count()));
+            let mut elems = Vec::with_capacity(usize::try_from(items_reader.get_count()).unwrap());
             for item in items_reader {
                 let x = item?;
                 elems.push(FuncIndex::from_u32(x));

From 23055196fc012633bc11415c8e6f8d4333b92a83 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 31 May 2019 10:14:28 -0700
Subject: [PATCH 2451/3084] Bump the minimum required Rust version to 1.34.

1.34 was [released] on April 11, so it satisfies the stable Rust
[policy].

Notably, this version includes the new `try_from` features.

[released]: https://blog.rust-lang.org/2019/04/11/Rust-1.34.0.html
[policy]: https://github.com/CraneStation/cranelift/blob/master/CONTRIBUTING.md#rustc-version-support
---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index f1460003c7..2465863fe3 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,7 +7,7 @@ language: rust
 rust:
     # The oldest version we currently support. See
     # CONTRIBUTING.md#rustc-version-support for details.
-    - 1.33.0
+    - 1.34.0
     - beta
     - nightly
 matrix:

From b1488decc4be6ca31c1698261707bb51a92d0a6d Mon Sep 17 00:00:00 2001
From: Julian Seward 
Date: Wed, 29 May 2019 17:12:38 +0200
Subject: [PATCH 2452/3084] Only create copy_nop instructions for types for
 which an encoding exists.  Issue #779.

PR #773 detects, at reload time, `copy` instructions that copy a value from stack
slot back to the same stack slot.  It replaces them with `copy_nop` instructions
that have a null encoding (hence producing no code).

For x86_64, `copy_nop` encodings for the types I64, I32, F64 and F32 are
provided.  Unfortunately the code that detects the redundant copy doesn't
check the type of the copied value, hence leaving itself open to the danger of
creating a `copy_nop` instruction cannot be encoded (which is different from
saying it has a null encoding).

This patch:

* Expands the x86_64 set of `copy_nop` encodings to: I64 I32 I16 I8 F64 and F32

* Adds encodings for the same for x86_32, rv64 and rv32.

* In `visit_inst()` in `reload.rs`, checks the type of the copied value accordingly.

* Adds comments explaining the above.
---
 .../meta-python/isa/riscv/encodings.py        |  9 +++++++-
 .../codegen/meta-python/isa/riscv/recipes.py  |  5 ++++
 .../codegen/meta-python/isa/x86/encodings.py  | 22 ++++++++++++++----
 cranelift/codegen/src/regalloc/reload.rs      | 17 +++++++++++---
 .../filetests/regalloc/reload-779.clif        | 23 +++++++++++++++++++
 5 files changed, 67 insertions(+), 9 deletions(-)
 create mode 100644 cranelift/filetests/filetests/regalloc/reload-779.clif

diff --git a/cranelift/codegen/meta-python/isa/riscv/encodings.py b/cranelift/codegen/meta-python/isa/riscv/encodings.py
index 801f5c6932..980a88ba00 100644
--- a/cranelift/codegen/meta-python/isa/riscv/encodings.py
+++ b/cranelift/codegen/meta-python/isa/riscv/encodings.py
@@ -3,12 +3,13 @@ RISC-V Encodings.
 """
 from __future__ import absolute_import
 from base import instructions as base
+from base import types
 from base.immediates import intcc
 from .defs import RV32, RV64
 from .recipes import OPIMM, OPIMM32, OP, OP32, LUI, BRANCH, JALR, JAL
 from .recipes import LOAD, STORE
 from .recipes import R, Rshamt, Ricmp, Ii, Iz, Iicmp, Iret, Icall, Icopy
-from .recipes import U, UJ, UJcall, SB, SBzero, GPsp, GPfi, Irmov
+from .recipes import U, UJ, UJcall, SB, SBzero, GPsp, GPfi, Irmov, stacknull
 from .settings import use_m
 from cdsl.ast import Var
 from base.legalize import narrow, expand
@@ -160,3 +161,9 @@ RV32.enc(base.copy.b1, Icopy, OPIMM(0b000))
 RV64.enc(base.copy.b1, Icopy, OPIMM(0b000))
 RV32.enc(base.regmove.b1, Irmov, OPIMM(0b000))
 RV64.enc(base.regmove.b1, Irmov, OPIMM(0b000))
+
+# Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
+# into a no-op.
+for ty in [types.i64, types.i32, types.i16, types.i8, types.f64, types.f32]:
+    RV64.enc(base.copy_nop.bind(ty), stacknull, 0)
+    RV32.enc(base.copy_nop.bind(ty), stacknull, 0)
diff --git a/cranelift/codegen/meta-python/isa/riscv/recipes.py b/cranelift/codegen/meta-python/isa/riscv/recipes.py
index 9808892b3c..ff27c31f8f 100644
--- a/cranelift/codegen/meta-python/isa/riscv/recipes.py
+++ b/cranelift/codegen/meta-python/isa/riscv/recipes.py
@@ -223,3 +223,8 @@ GPfi = EncRecipe(
         'GPfi', Unary, base_size=4,
         ins=Stack(GPR), outs=GPR,
         emit='unimplemented!();')
+
+# Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
+# into a no-op.
+stacknull = EncRecipe('stacknull', Unary, base_size=0,
+                      ins=Stack(GPR), outs=Stack(GPR), emit='')
diff --git a/cranelift/codegen/meta-python/isa/x86/encodings.py b/cranelift/codegen/meta-python/isa/x86/encodings.py
index 449e8b8e1b..8da4912e77 100644
--- a/cranelift/codegen/meta-python/isa/x86/encodings.py
+++ b/cranelift/codegen/meta-python/isa/x86/encodings.py
@@ -342,11 +342,23 @@ X86_64.enc(base.copy_special, *r.copysp.rex(0x89, w=1))
 X86_32.enc(base.copy_special, *r.copysp(0x89))
 
 # Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
-# into a no-op.
-X86_64.enc(base.copy_nop.i64, r.stacknull, 0)
-X86_64.enc(base.copy_nop.i32, r.stacknull, 0)
-X86_64.enc(base.copy_nop.f64, r.stacknull, 0)
-X86_64.enc(base.copy_nop.f32, r.stacknull, 0)
+# into a no-op.  Ideally we could to make this encoding available for
+# all types, and write `base.copy_nop.any`, but it appears that the
+# controlling type variable must not polymorphic.  So we make do with
+# the following limited set, and guard the generating transformation in
+# regalloc/reload.rs accordingly.
+#
+# The same encoding is generated for both the 64- and 32-bit architectures.
+# Note that we can't use `enc_both` here, because that attempts to create a
+# variant with a REX prefix in the 64-bit-architecture case.  But since
+# there's no actual instruction for the REX prefix to modify the meaning of,
+# it will modify the meaning of whatever instruction happens to follow this
+# one, which is obviously wrong.  Note also that we can and indeed *must*
+# claim that there's a 64-bit encoding for the 32-bit arch case, even though
+# no such single instruction actually exists for the 32-bit arch case.
+for ty in [types.i64, types.i32, types.i16, types.i8, types.f64, types.f32]:
+    X86_64.enc(base.copy_nop.bind(ty), r.stacknull, 0)
+    X86_32.enc(base.copy_nop.bind(ty), r.stacknull, 0)
 
 # Adjust SP down by a dynamic value (or up, with a negative operand).
 X86_32.enc(base.adjust_sp_down.i32, *r.adjustsp(0x29))
diff --git a/cranelift/codegen/src/regalloc/reload.rs b/cranelift/codegen/src/regalloc/reload.rs
index cc6aae59b8..11c1a5513d 100644
--- a/cranelift/codegen/src/regalloc/reload.rs
+++ b/cranelift/codegen/src/regalloc/reload.rs
@@ -222,18 +222,29 @@ impl<'a> Context<'a> {
         {
             let dst_vals = self.cur.func.dfg.inst_results(inst);
             if dst_vals.len() == 1 {
+                let dst_val = dst_vals[0];
                 let can_transform = match (
                     self.cur.func.locations[arg],
-                    self.cur.func.locations[dst_vals[0]],
+                    self.cur.func.locations[dst_val],
                 ) {
-                    (ValueLoc::Stack(src_slot), ValueLoc::Stack(dst_slot)) => src_slot == dst_slot,
+                    (ValueLoc::Stack(src_slot), ValueLoc::Stack(dst_slot)) => {
+                        src_slot == dst_slot && {
+                            let src_ty = self.cur.func.dfg.value_type(arg);
+                            let dst_ty = self.cur.func.dfg.value_type(dst_val);
+                            debug_assert!(src_ty == dst_ty);
+                            // This limits the transformation to copies of the
+                            // types: I64 I32 I16 I8 F64 and F32, since that's
+                            // the set of `copy_nop` encodings available.
+                            src_ty.is_int() || src_ty.is_float()
+                        }
+                    }
                     _ => false,
                 };
                 if can_transform {
                     // Convert the instruction into a `copy_nop`.
                     self.cur.func.dfg.replace(inst).copy_nop(arg);
                     let ok = self.cur.func.update_encoding(inst, self.cur.isa).is_ok();
-                    debug_assert!(ok);
+                    debug_assert!(ok, "copy_nop encoding missing for this type");
 
                     // And move on to the next insn.
                     self.reloads.clear();
diff --git a/cranelift/filetests/filetests/regalloc/reload-779.clif b/cranelift/filetests/filetests/regalloc/reload-779.clif
new file mode 100644
index 0000000000..c02464a11f
--- /dev/null
+++ b/cranelift/filetests/filetests/regalloc/reload-779.clif
@@ -0,0 +1,23 @@
+test compile
+target x86_64
+
+; Filed as https://github.com/CraneStation/cranelift/issues/779
+;
+; The copy_nop optimisation to reload (see Issue 773) was creating
+; copy_nop instructions for types for which there were no encoding.
+
+function u0:0(i64, i64, i64) system_v {
+    sig0 = () system_v
+    sig1 = (i16) system_v
+    fn1 = u0:94 sig0
+    fn2 = u0:95 sig1
+
+ebb0(v0: i64, v1: i64, v2: i64):
+    v3 = iconst.i16 0
+    jump ebb1(v3)
+
+ebb1(v4: i16):
+    call fn1()
+    call fn2(v4)
+    jump ebb1(v4)
+}

From ec5dc3384e8f9f1c3b651caacd733249ec166e78 Mon Sep 17 00:00:00 2001
From: Lars T Hansen 
Date: Fri, 7 Jun 2019 14:37:58 +0200
Subject: [PATCH 2453/3084] Handle Reloc::X86PCRelRodata4 in sundry reloc_jt

---
 cranelift/faerie/src/backend.rs    | 12 ++++++++++--
 cranelift/simplejit/src/backend.rs | 12 ++++++++++--
 2 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index 9313ddaae9..29f3c0480d 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -413,7 +413,15 @@ impl<'a> RelocSink for FaerieRelocSink<'a> {
             .expect("faerie relocation error");
     }
 
-    fn reloc_jt(&mut self, _offset: CodeOffset, _reloc: Reloc, _jt: ir::JumpTable) {
-        unimplemented!();
+    fn reloc_jt(&mut self, _offset: CodeOffset, reloc: Reloc, _jt: ir::JumpTable) {
+        match reloc {
+            Reloc::X86PCRelRodata4 => {
+                // Not necessary to record this unless we are going to split apart code and its
+                // jumptbl/rodata.
+            }
+            _ => {
+                panic!("Unhandled reloc");
+            }
+        }
     }
 }
diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index 81b44459ec..3786e3e964 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -533,7 +533,15 @@ impl RelocSink for SimpleJITRelocSink {
         });
     }
 
-    fn reloc_jt(&mut self, _offset: CodeOffset, _reloc: Reloc, _jt: ir::JumpTable) {
-        unimplemented!();
+    fn reloc_jt(&mut self, _offset: CodeOffset, reloc: Reloc, _jt: ir::JumpTable) {
+        match reloc {
+            Reloc::X86PCRelRodata4 => {
+                // Not necessary to record this unless we are going to split apart code and its
+                // jumptbl/rodata.
+            }
+            _ => {
+                panic!("Unhandled reloc");
+            }
+        }
     }
 }

From 36870c41c8c5b829f23a5ed2fcead1f921691f32 Mon Sep 17 00:00:00 2001
From: Lars T Hansen 
Date: Tue, 18 Jun 2019 07:14:32 -0700
Subject: [PATCH 2454/3084] Fix a calculation error for rodata_size in memsink

---
 cranelift/codegen/src/binemit/memorysink.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/binemit/memorysink.rs b/cranelift/codegen/src/binemit/memorysink.rs
index 8e07db01f6..170e1a95e9 100644
--- a/cranelift/codegen/src/binemit/memorysink.rs
+++ b/cranelift/codegen/src/binemit/memorysink.rs
@@ -142,7 +142,7 @@ impl<'a> CodeSink for MemoryCodeSink<'a> {
     }
 
     fn end_codegen(&mut self) {
-        self.info.rodata_size = self.offset() - self.info.jumptables_size;
+        self.info.rodata_size = self.offset() - (self.info.jumptables_size + self.info.code_size);
         self.info.total_size = self.offset();
     }
 }

From d7d48d5cc6ae06282dad2efa1bb2039a4c676317 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 6 Jun 2019 10:11:41 +0200
Subject: [PATCH 2455/3084] Add the dyn keyword before trait objects;

---
 cranelift/bforest/src/path.rs                 |  2 +-
 cranelift/codegen/meta/src/gen_legalizer.rs   |  2 +-
 cranelift/codegen/src/binemit/memorysink.rs   | 10 ++-
 cranelift/codegen/src/binemit/relaxation.rs   |  4 +-
 cranelift/codegen/src/binemit/shrink.rs       |  2 +-
 cranelift/codegen/src/cfg_printer.rs          |  8 +--
 cranelift/codegen/src/context.rs              | 41 ++++++------
 cranelift/codegen/src/cursor.rs               |  4 +-
 cranelift/codegen/src/ir/dfg.rs               |  4 +-
 cranelift/codegen/src/ir/entities.rs          |  2 +-
 cranelift/codegen/src/ir/function.rs          | 15 +++--
 cranelift/codegen/src/ir/globalvalue.rs       |  2 +-
 cranelift/codegen/src/ir/libcall.rs           | 15 +++--
 cranelift/codegen/src/isa/arm32/mod.rs        |  4 +-
 cranelift/codegen/src/isa/arm64/mod.rs        |  4 +-
 cranelift/codegen/src/isa/mod.rs              |  8 +--
 cranelift/codegen/src/isa/riscv/mod.rs        |  6 +-
 cranelift/codegen/src/isa/x86/abi.rs          | 14 ++---
 cranelift/codegen/src/isa/x86/enc_tables.rs   | 16 ++---
 cranelift/codegen/src/isa/x86/mod.rs          |  4 +-
 cranelift/codegen/src/legalizer/boundary.rs   |  6 +-
 cranelift/codegen/src/legalizer/call.rs       |  2 +-
 .../codegen/src/legalizer/globalvalue.rs      |  6 +-
 cranelift/codegen/src/legalizer/heap.rs       |  2 +-
 cranelift/codegen/src/legalizer/libcall.rs    |  2 +-
 cranelift/codegen/src/legalizer/mod.rs        | 22 +++----
 cranelift/codegen/src/legalizer/table.rs      |  2 +-
 cranelift/codegen/src/licm.rs                 |  4 +-
 cranelift/codegen/src/postopt.rs              |  6 +-
 cranelift/codegen/src/print_errors.rs         | 42 ++++++-------
 cranelift/codegen/src/regalloc/affinity.rs    |  2 +-
 cranelift/codegen/src/regalloc/coalescing.rs  |  4 +-
 cranelift/codegen/src/regalloc/coloring.rs    |  2 +-
 cranelift/codegen/src/regalloc/context.rs     |  2 +-
 cranelift/codegen/src/regalloc/liveness.rs    |  4 +-
 cranelift/codegen/src/regalloc/pressure.rs    |  4 +-
 cranelift/codegen/src/regalloc/reload.rs      |  4 +-
 cranelift/codegen/src/regalloc/solver.rs      |  4 +-
 cranelift/codegen/src/regalloc/spilling.rs    |  2 +-
 cranelift/codegen/src/settings.rs             |  6 +-
 cranelift/codegen/src/value_label.rs          |  2 +-
 cranelift/codegen/src/verifier/flags.rs       |  2 +-
 cranelift/codegen/src/verifier/liveness.rs    |  4 +-
 cranelift/codegen/src/verifier/locations.rs   |  4 +-
 cranelift/codegen/src/verifier/mod.rs         |  2 +-
 cranelift/codegen/src/write.rs                | 63 ++++++++++---------
 cranelift/entity/src/lib.rs                   |  2 +-
 cranelift/faerie/src/backend.rs               | 16 ++---
 cranelift/filetests/src/lib.rs                |  2 +-
 cranelift/filetests/src/runone.rs             |  6 +-
 cranelift/filetests/src/subtest.rs            |  2 +-
 cranelift/filetests/src/test_binemit.rs       |  2 +-
 cranelift/filetests/src/test_cat.rs           |  2 +-
 cranelift/filetests/src/test_compile.rs       |  2 +-
 cranelift/filetests/src/test_dce.rs           |  2 +-
 cranelift/filetests/src/test_domtree.rs       |  2 +-
 cranelift/filetests/src/test_legalizer.rs     |  2 +-
 cranelift/filetests/src/test_licm.rs          |  2 +-
 cranelift/filetests/src/test_postopt.rs       |  2 +-
 cranelift/filetests/src/test_preopt.rs        |  2 +-
 cranelift/filetests/src/test_print_cfg.rs     |  2 +-
 cranelift/filetests/src/test_regalloc.rs      |  2 +-
 cranelift/filetests/src/test_shrink.rs        |  2 +-
 cranelift/filetests/src/test_simple_gvn.rs    |  2 +-
 cranelift/filetests/src/test_simple_preopt.rs |  2 +-
 cranelift/filetests/src/test_verifier.rs      |  2 +-
 cranelift/frontend/src/frontend.rs            |  2 +-
 cranelift/module/src/backend.rs               |  4 +-
 cranelift/module/src/module.rs                |  2 +-
 cranelift/preopt/src/lib.rs                   |  2 +-
 cranelift/reader/src/isaspec.rs               | 10 ++-
 cranelift/reader/src/parser.rs                | 20 +++---
 cranelift/simplejit/src/backend.rs            | 17 ++---
 cranelift/src/disasm.rs                       |  8 +--
 cranelift/src/utils.rs                        |  2 +-
 cranelift/wasm/src/module_translator.rs       |  2 +-
 cranelift/wasm/src/sections_translator.rs     | 22 +++----
 77 files changed, 274 insertions(+), 247 deletions(-)

diff --git a/cranelift/bforest/src/path.rs b/cranelift/bforest/src/path.rs
index 1df66b7ef7..a55de6b2ae 100644
--- a/cranelift/bforest/src/path.rs
+++ b/cranelift/bforest/src/path.rs
@@ -49,7 +49,7 @@ impl Path {
         key: F::Key,
         root: Node,
         pool: &NodePool,
-        comp: &Comparator,
+        comp: &dyn Comparator,
     ) -> Option {
         let mut node = root;
         for level in 0.. {
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index 45221b1ffd..149e29f926 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -410,7 +410,7 @@ fn gen_transform_group<'a>(
         fmt.line("inst: crate::ir::Inst,");
         fmt.line("func: &mut crate::ir::Function,");
         fmt.line("cfg: &mut crate::flowgraph::ControlFlowGraph,");
-        fmt.line("isa: &crate::isa::TargetIsa,");
+        fmt.line("isa: &dyn crate::isa::TargetIsa,");
     });
     fmtln!(fmt, ") -> bool {");
 
diff --git a/cranelift/codegen/src/binemit/memorysink.rs b/cranelift/codegen/src/binemit/memorysink.rs
index 170e1a95e9..b70b4a2c86 100644
--- a/cranelift/codegen/src/binemit/memorysink.rs
+++ b/cranelift/codegen/src/binemit/memorysink.rs
@@ -34,8 +34,8 @@ pub struct MemoryCodeSink<'a> {
     data: *mut u8,
     /// Offset is isize because its major consumer needs it in that form.
     offset: isize,
-    relocs: &'a mut RelocSink,
-    traps: &'a mut TrapSink,
+    relocs: &'a mut dyn RelocSink,
+    traps: &'a mut dyn TrapSink,
     /// Information about the generated code and read-only data.
     pub info: CodeInfo,
 }
@@ -45,7 +45,11 @@ impl<'a> MemoryCodeSink<'a> {
     ///
     /// This function is unsafe since `MemoryCodeSink` does not perform bounds checking on the
     /// memory buffer, and it can't guarantee that the `data` pointer is valid.
-    pub unsafe fn new(data: *mut u8, relocs: &'a mut RelocSink, traps: &'a mut TrapSink) -> Self {
+    pub unsafe fn new(
+        data: *mut u8,
+        relocs: &'a mut dyn RelocSink,
+        traps: &'a mut dyn TrapSink,
+    ) -> Self {
         Self {
             data,
             offset: 0,
diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index d1b5815e7c..1e84f2f4b9 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -40,7 +40,7 @@ use log::debug;
 /// Relax branches and compute the final layout of EBB headers in `func`.
 ///
 /// Fill in the `func.offsets` table so the function is ready for binary emission.
-pub fn relax_branches(func: &mut Function, isa: &TargetIsa) -> CodegenResult {
+pub fn relax_branches(func: &mut Function, isa: &dyn TargetIsa) -> CodegenResult {
     let _tt = timing::relax_branches();
 
     let encinfo = isa.encoding_info();
@@ -174,7 +174,7 @@ fn relax_branch(
     offset: CodeOffset,
     dest_offset: CodeOffset,
     encinfo: &EncInfo,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) -> CodeOffset {
     let inst = cur.current_inst().unwrap();
     debug!(
diff --git a/cranelift/codegen/src/binemit/shrink.rs b/cranelift/codegen/src/binemit/shrink.rs
index b3cfdfa6d1..281a93ae31 100644
--- a/cranelift/codegen/src/binemit/shrink.rs
+++ b/cranelift/codegen/src/binemit/shrink.rs
@@ -13,7 +13,7 @@ use crate::timing;
 use log::debug;
 
 /// Pick the smallest valid encodings for instructions.
-pub fn shrink_instructions(func: &mut Function, isa: &TargetIsa) {
+pub fn shrink_instructions(func: &mut Function, isa: &dyn TargetIsa) {
     let _tt = timing::shrink_instructions();
 
     let encinfo = isa.encoding_info();
diff --git a/cranelift/codegen/src/cfg_printer.rs b/cranelift/codegen/src/cfg_printer.rs
index 6a9932febe..e3ce37e731 100644
--- a/cranelift/codegen/src/cfg_printer.rs
+++ b/cranelift/codegen/src/cfg_printer.rs
@@ -23,14 +23,14 @@ impl<'a> CFGPrinter<'a> {
     }
 
     /// Write the CFG for this function to `w`.
-    pub fn write(&self, w: &mut Write) -> Result {
+    pub fn write(&self, w: &mut dyn Write) -> Result {
         self.header(w)?;
         self.ebb_nodes(w)?;
         self.cfg_connections(w)?;
         writeln!(w, "}}")
     }
 
-    fn header(&self, w: &mut Write) -> Result {
+    fn header(&self, w: &mut dyn Write) -> Result {
         writeln!(w, "digraph \"{}\" {{", self.func.name)?;
         if let Some(entry) = self.func.layout.entry_block() {
             writeln!(w, "    {{rank=min; {}}}", entry)?;
@@ -38,7 +38,7 @@ impl<'a> CFGPrinter<'a> {
         Ok(())
     }
 
-    fn ebb_nodes(&self, w: &mut Write) -> Result {
+    fn ebb_nodes(&self, w: &mut dyn Write) -> Result {
         for ebb in &self.func.layout {
             write!(w, "    {} [shape=record, label=\"{{{}", ebb, ebb)?;
             // Add all outgoing branch instructions to the label.
@@ -62,7 +62,7 @@ impl<'a> CFGPrinter<'a> {
         Ok(())
     }
 
-    fn cfg_connections(&self, w: &mut Write) -> Result {
+    fn cfg_connections(&self, w: &mut dyn Write) -> Result {
         for ebb in &self.func.layout {
             for BasicBlock { ebb: parent, inst } in self.cfg.pred_iter(ebb) {
                 writeln!(w, "    {}:{} -> {}", parent, inst, ebb)?;
diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index 369b956a67..64c382cb23 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -96,10 +96,10 @@ impl Context {
     /// Returns information about the function's code and read-only data.
     pub fn compile_and_emit(
         &mut self,
-        isa: &TargetIsa,
+        isa: &dyn TargetIsa,
         mem: &mut Vec,
-        relocs: &mut RelocSink,
-        traps: &mut TrapSink,
+        relocs: &mut dyn RelocSink,
+        traps: &mut dyn TrapSink,
     ) -> CodegenResult {
         let info = self.compile(isa)?;
         let old_len = mem.len();
@@ -117,7 +117,7 @@ impl Context {
     /// code sink.
     ///
     /// Returns information about the function's code and read-only data.
-    pub fn compile(&mut self, isa: &TargetIsa) -> CodegenResult {
+    pub fn compile(&mut self, isa: &dyn TargetIsa) -> CodegenResult {
         let _tt = timing::compile();
         self.verify_if(isa)?;
 
@@ -164,10 +164,10 @@ impl Context {
     /// Returns information about the emitted code and data.
     pub unsafe fn emit_to_memory(
         &self,
-        isa: &TargetIsa,
+        isa: &dyn TargetIsa,
         mem: *mut u8,
-        relocs: &mut RelocSink,
-        traps: &mut TrapSink,
+        relocs: &mut dyn RelocSink,
+        traps: &mut dyn TrapSink,
     ) -> CodeInfo {
         let _tt = timing::binemit();
         let mut sink = MemoryCodeSink::new(mem, relocs, traps);
@@ -199,7 +199,7 @@ impl Context {
     }
 
     /// Run the locations verifier on the function.
-    pub fn verify_locations(&self, isa: &TargetIsa) -> VerifierResult<()> {
+    pub fn verify_locations(&self, isa: &dyn TargetIsa) -> VerifierResult<()> {
         let mut errors = VerifierErrors::default();
         let _ = verify_locations(isa, &self.func, None, &mut errors);
 
@@ -211,7 +211,7 @@ impl Context {
     }
 
     /// Run the locations verifier only if the `enable_verifier` setting is true.
-    pub fn verify_locations_if(&self, isa: &TargetIsa) -> CodegenResult<()> {
+    pub fn verify_locations_if(&self, isa: &dyn TargetIsa) -> CodegenResult<()> {
         if isa.flags().enable_verifier() {
             self.verify_locations(isa)?;
         }
@@ -226,20 +226,20 @@ impl Context {
     }
 
     /// Perform pre-legalization rewrites on the function.
-    pub fn preopt(&mut self, isa: &TargetIsa) -> CodegenResult<()> {
+    pub fn preopt(&mut self, isa: &dyn TargetIsa) -> CodegenResult<()> {
         do_preopt(&mut self.func, &mut self.cfg);
         self.verify_if(isa)?;
         Ok(())
     }
 
     /// Perform NaN canonicalizing rewrites on the function.
-    pub fn canonicalize_nans(&mut self, isa: &TargetIsa) -> CodegenResult<()> {
+    pub fn canonicalize_nans(&mut self, isa: &dyn TargetIsa) -> CodegenResult<()> {
         do_nan_canonicalization(&mut self.func);
         self.verify_if(isa)
     }
 
     /// Run the legalizer for `isa` on the function.
-    pub fn legalize(&mut self, isa: &TargetIsa) -> CodegenResult<()> {
+    pub fn legalize(&mut self, isa: &dyn TargetIsa) -> CodegenResult<()> {
         // Legalization invalidates the domtree and loop_analysis by mutating the CFG.
         // TODO: Avoid doing this when legalization doesn't actually mutate the CFG.
         self.domtree.clear();
@@ -249,7 +249,7 @@ impl Context {
     }
 
     /// Perform post-legalization rewrites on the function.
-    pub fn postopt(&mut self, isa: &TargetIsa) -> CodegenResult<()> {
+    pub fn postopt(&mut self, isa: &dyn TargetIsa) -> CodegenResult<()> {
         do_postopt(&mut self.func, isa);
         self.verify_if(isa)?;
         Ok(())
@@ -284,7 +284,7 @@ impl Context {
     }
 
     /// Perform LICM on the function.
-    pub fn licm(&mut self, isa: &TargetIsa) -> CodegenResult<()> {
+    pub fn licm(&mut self, isa: &dyn TargetIsa) -> CodegenResult<()> {
         do_licm(
             isa,
             &mut self.func,
@@ -305,13 +305,13 @@ impl Context {
     }
 
     /// Run the register allocator.
-    pub fn regalloc(&mut self, isa: &TargetIsa) -> CodegenResult<()> {
+    pub fn regalloc(&mut self, isa: &dyn TargetIsa) -> CodegenResult<()> {
         self.regalloc
             .run(isa, &mut self.func, &self.cfg, &mut self.domtree)
     }
 
     /// Insert prologue and epilogues after computing the stack frame layout.
-    pub fn prologue_epilogue(&mut self, isa: &TargetIsa) -> CodegenResult<()> {
+    pub fn prologue_epilogue(&mut self, isa: &dyn TargetIsa) -> CodegenResult<()> {
         isa.prologue_epilogue(&mut self.func)?;
         self.verify_if(isa)?;
         self.verify_locations_if(isa)?;
@@ -319,7 +319,7 @@ impl Context {
     }
 
     /// Run the instruction shrinking pass.
-    pub fn shrink_instructions(&mut self, isa: &TargetIsa) -> CodegenResult<()> {
+    pub fn shrink_instructions(&mut self, isa: &dyn TargetIsa) -> CodegenResult<()> {
         shrink_instructions(&mut self.func, isa);
         self.verify_if(isa)?;
         self.verify_locations_if(isa)?;
@@ -328,7 +328,7 @@ impl Context {
 
     /// Run the branch relaxation pass and return information about the function's code and
     /// read-only data.
-    pub fn relax_branches(&mut self, isa: &TargetIsa) -> CodegenResult {
+    pub fn relax_branches(&mut self, isa: &dyn TargetIsa) -> CodegenResult {
         let info = relax_branches(&mut self.func, isa)?;
         self.verify_if(isa)?;
         self.verify_locations_if(isa)?;
@@ -336,7 +336,10 @@ impl Context {
     }
 
     /// Builds ranges and location for specified value labels.
-    pub fn build_value_labels_ranges(&self, isa: &TargetIsa) -> CodegenResult {
+    pub fn build_value_labels_ranges(
+        &self,
+        isa: &dyn TargetIsa,
+    ) -> CodegenResult {
         Ok(build_value_labels_ranges::(
             &self.func,
             &self.regalloc,
diff --git a/cranelift/codegen/src/cursor.rs b/cranelift/codegen/src/cursor.rs
index ba7bc92b13..dbb459ed7a 100644
--- a/cranelift/codegen/src/cursor.rs
+++ b/cranelift/codegen/src/cursor.rs
@@ -657,12 +657,12 @@ pub struct EncCursor<'f> {
     pub func: &'f mut ir::Function,
 
     /// The target ISA that will be used to encode instructions.
-    pub isa: &'f TargetIsa,
+    pub isa: &'f dyn TargetIsa,
 }
 
 impl<'f> EncCursor<'f> {
     /// Create a new `EncCursor` pointing nowhere.
-    pub fn new(func: &'f mut ir::Function, isa: &'f TargetIsa) -> Self {
+    pub fn new(func: &'f mut ir::Function, isa: &'f dyn TargetIsa) -> Self {
         Self {
             pos: CursorPosition::Nowhere,
             srcloc: Default::default(),
diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index 362a6cd772..d26abaa3e7 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -426,7 +426,7 @@ impl DataFlowGraph {
     }
 
     /// Returns an object that displays `inst`.
-    pub fn display_inst<'a, I: Into>>(
+    pub fn display_inst<'a, I: Into>>(
         &'a self,
         inst: Inst,
         isa: I,
@@ -909,7 +909,7 @@ impl EbbData {
 }
 
 /// Object that can display an instruction.
-pub struct DisplayInst<'a>(&'a DataFlowGraph, Option<&'a TargetIsa>, Inst);
+pub struct DisplayInst<'a>(&'a DataFlowGraph, Option<&'a dyn TargetIsa>, Inst);
 
 impl<'a> fmt::Display for DisplayInst<'a> {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
diff --git a/cranelift/codegen/src/ir/entities.rs b/cranelift/codegen/src/ir/entities.rs
index a0b694ad84..2c05fbc84b 100644
--- a/cranelift/codegen/src/ir/entities.rs
+++ b/cranelift/codegen/src/ir/entities.rs
@@ -238,7 +238,7 @@ impl fmt::Display for AnyEntity {
 
 impl fmt::Debug for AnyEntity {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        (self as &fmt::Display).fmt(f)
+        (self as &dyn fmt::Display).fmt(f)
     }
 }
 
diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index 6a1d4cdd0d..f4b62dc59c 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -155,7 +155,10 @@ impl Function {
     }
 
     /// Return an object that can display this function with correct ISA-specific annotations.
-    pub fn display<'a, I: Into>>(&'a self, isa: I) -> DisplayFunction<'a> {
+    pub fn display<'a, I: Into>>(
+        &'a self,
+        isa: I,
+    ) -> DisplayFunction<'a> {
         DisplayFunction(self, isa.into().into())
     }
 
@@ -202,13 +205,13 @@ impl Function {
     }
 
     /// Wrapper around `encode` which assigns `inst` the resulting encoding.
-    pub fn update_encoding(&mut self, inst: ir::Inst, isa: &TargetIsa) -> Result<(), Legalize> {
+    pub fn update_encoding(&mut self, inst: ir::Inst, isa: &dyn TargetIsa) -> Result<(), Legalize> {
         self.encode(inst, isa).map(|e| self.encodings[inst] = e)
     }
 
     /// Wrapper around `TargetIsa::encode` for encoding an existing instruction
     /// in the `Function`.
-    pub fn encode(&self, inst: ir::Inst, isa: &TargetIsa) -> Result {
+    pub fn encode(&self, inst: ir::Inst, isa: &dyn TargetIsa) -> Result {
         isa.encode(&self, &self.dfg[inst], self.dfg.ctrl_typevar(inst))
     }
 
@@ -221,7 +224,7 @@ impl Function {
 /// Additional annotations for function display.
 pub struct DisplayFunctionAnnotations<'a> {
     /// Enable ISA annotations.
-    pub isa: Option<&'a TargetIsa>,
+    pub isa: Option<&'a dyn TargetIsa>,
 
     /// Enable value labels annotations.
     pub value_ranges: Option<&'a ValueLabelsRanges>,
@@ -237,8 +240,8 @@ impl<'a> DisplayFunctionAnnotations<'a> {
     }
 }
 
-impl<'a> From> for DisplayFunctionAnnotations<'a> {
-    fn from(isa: Option<&'a TargetIsa>) -> DisplayFunctionAnnotations {
+impl<'a> From> for DisplayFunctionAnnotations<'a> {
+    fn from(isa: Option<&'a dyn TargetIsa>) -> DisplayFunctionAnnotations {
         DisplayFunctionAnnotations {
             isa,
             value_ranges: None,
diff --git a/cranelift/codegen/src/ir/globalvalue.rs b/cranelift/codegen/src/ir/globalvalue.rs
index b67c26e1a2..6c7b62aab9 100644
--- a/cranelift/codegen/src/ir/globalvalue.rs
+++ b/cranelift/codegen/src/ir/globalvalue.rs
@@ -76,7 +76,7 @@ impl GlobalValueData {
     }
 
     /// Return the type of this global.
-    pub fn global_type(&self, isa: &TargetIsa) -> Type {
+    pub fn global_type(&self, isa: &dyn TargetIsa) -> Type {
         match *self {
             GlobalValueData::VMContext { .. } | GlobalValueData::Symbol { .. } => {
                 isa.pointer_type()
diff --git a/cranelift/codegen/src/ir/libcall.rs b/cranelift/codegen/src/ir/libcall.rs
index 3617f906d6..275e22a91f 100644
--- a/cranelift/codegen/src/ir/libcall.rs
+++ b/cranelift/codegen/src/ir/libcall.rs
@@ -107,7 +107,7 @@ pub fn get_libcall_funcref(
     libcall: LibCall,
     func: &mut Function,
     inst: Inst,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) -> FuncRef {
     find_funcref(libcall, func).unwrap_or_else(|| make_funcref_for_inst(libcall, func, inst, isa))
 }
@@ -119,7 +119,7 @@ pub fn get_probestack_funcref(
     func: &mut Function,
     reg_type: Type,
     arg_reg: RegUnit,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) -> FuncRef {
     find_funcref(LibCall::Probestack, func)
         .unwrap_or_else(|| make_funcref_for_probestack(func, reg_type, arg_reg, isa))
@@ -147,7 +147,7 @@ fn make_funcref_for_probestack(
     func: &mut Function,
     reg_type: Type,
     arg_reg: RegUnit,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) -> FuncRef {
     let mut sig = Signature::new(CallConv::Probestack);
     let rax = AbiParam::special_reg(reg_type, ArgumentPurpose::Normal, arg_reg);
@@ -163,7 +163,7 @@ fn make_funcref_for_inst(
     libcall: LibCall,
     func: &mut Function,
     inst: Inst,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) -> FuncRef {
     let mut sig = Signature::new(isa.default_call_conv());
     for &v in func.dfg.inst_args(inst) {
@@ -177,7 +177,12 @@ fn make_funcref_for_inst(
 }
 
 /// Create a funcref for `libcall`.
-fn make_funcref(libcall: LibCall, func: &mut Function, sig: Signature, isa: &TargetIsa) -> FuncRef {
+fn make_funcref(
+    libcall: LibCall,
+    func: &mut Function,
+    sig: Signature,
+    isa: &dyn TargetIsa,
+) -> FuncRef {
     let sigref = func.import_signature(sig);
 
     func.import_function(ExtFuncData {
diff --git a/cranelift/codegen/src/isa/arm32/mod.rs b/cranelift/codegen/src/isa/arm32/mod.rs
index 7bede88f3c..d4e2f32255 100644
--- a/cranelift/codegen/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/src/isa/arm32/mod.rs
@@ -40,7 +40,7 @@ fn isa_constructor(
     triple: Triple,
     shared_flags: shared_settings::Flags,
     builder: shared_settings::Builder,
-) -> Box {
+) -> Box {
     let level1 = match triple.architecture {
         Architecture::Thumbv6m | Architecture::Thumbv7em | Architecture::Thumbv7m => {
             &enc_tables::LEVEL1_T32[..]
@@ -119,7 +119,7 @@ impl TargetIsa for Isa {
         func: &ir::Function,
         inst: ir::Inst,
         divert: &mut regalloc::RegDiversions,
-        sink: &mut CodeSink,
+        sink: &mut dyn CodeSink,
     ) {
         binemit::emit_inst(func, inst, divert, sink)
     }
diff --git a/cranelift/codegen/src/isa/arm64/mod.rs b/cranelift/codegen/src/isa/arm64/mod.rs
index 84277301c3..c24be19f69 100644
--- a/cranelift/codegen/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/src/isa/arm64/mod.rs
@@ -39,7 +39,7 @@ fn isa_constructor(
     triple: Triple,
     shared_flags: shared_settings::Flags,
     builder: shared_settings::Builder,
-) -> Box {
+) -> Box {
     Box::new(Isa {
         triple,
         isa_flags: settings::Flags::new(&shared_flags, builder),
@@ -106,7 +106,7 @@ impl TargetIsa for Isa {
         func: &ir::Function,
         inst: ir::Inst,
         divert: &mut regalloc::RegDiversions,
-        sink: &mut CodeSink,
+        sink: &mut dyn CodeSink,
     ) {
         binemit::emit_inst(func, inst, divert, sink)
     }
diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index b3e4d81421..eb6b4d2650 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -141,13 +141,13 @@ pub enum LookupError {
 pub struct Builder {
     triple: Triple,
     setup: settings::Builder,
-    constructor: fn(Triple, settings::Flags, settings::Builder) -> Box,
+    constructor: fn(Triple, settings::Flags, settings::Builder) -> Box,
 }
 
 impl Builder {
     /// Combine the ISA-specific settings with the provided ISA-independent settings and allocate a
     /// fully configured `TargetIsa` trait object.
-    pub fn finish(self, shared_flags: settings::Flags) -> Box {
+    pub fn finish(self, shared_flags: settings::Flags) -> Box {
         (self.constructor)(self.triple, shared_flags, self.setup)
     }
 }
@@ -167,7 +167,7 @@ impl settings::Configurable for Builder {
 ///
 /// The `Encodings` iterator returns a legalization function to call.
 pub type Legalize =
-    fn(ir::Inst, &mut ir::Function, &mut flowgraph::ControlFlowGraph, &TargetIsa) -> bool;
+    fn(ir::Inst, &mut ir::Function, &mut flowgraph::ControlFlowGraph, &dyn TargetIsa) -> bool;
 
 /// This struct provides information that a frontend may need to know about a target to
 /// produce Cranelift IR for the target.
@@ -367,7 +367,7 @@ pub trait TargetIsa: fmt::Display + Sync {
         func: &ir::Function,
         inst: ir::Inst,
         divert: &mut regalloc::RegDiversions,
-        sink: &mut binemit::CodeSink,
+        sink: &mut dyn binemit::CodeSink,
     );
 
     /// Emit a whole function into memory.
diff --git a/cranelift/codegen/src/isa/riscv/mod.rs b/cranelift/codegen/src/isa/riscv/mod.rs
index 60154cf162..3096ff69aa 100644
--- a/cranelift/codegen/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/src/isa/riscv/mod.rs
@@ -40,7 +40,7 @@ fn isa_constructor(
     triple: Triple,
     shared_flags: shared_settings::Flags,
     builder: shared_settings::Builder,
-) -> Box {
+) -> Box {
     let level1 = match triple.pointer_width().unwrap() {
         PointerWidth::U16 => panic!("16-bit RISC-V unrecognized"),
         PointerWidth::U32 => &enc_tables::LEVEL1_RV32[..],
@@ -113,7 +113,7 @@ impl TargetIsa for Isa {
         func: &ir::Function,
         inst: ir::Inst,
         divert: &mut regalloc::RegDiversions,
-        sink: &mut CodeSink,
+        sink: &mut dyn CodeSink,
     ) {
         binemit::emit_inst(func, inst, divert, sink)
     }
@@ -133,7 +133,7 @@ mod tests {
     use std::string::{String, ToString};
     use target_lexicon::triple;
 
-    fn encstr(isa: &isa::TargetIsa, enc: Result) -> String {
+    fn encstr(isa: &dyn isa::TargetIsa, enc: Result) -> String {
         match enc {
             Ok(e) => isa.encoding_info().display(e).to_string(),
             Err(_) => "no encoding".to_string(),
diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 37d9ffb257..549ec644c7 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -199,7 +199,7 @@ pub fn allocatable_registers(_func: &ir::Function, triple: &Triple) -> RegisterS
 }
 
 /// Get the set of callee-saved registers.
-fn callee_saved_gprs(isa: &TargetIsa, call_conv: CallConv) -> &'static [RU] {
+fn callee_saved_gprs(isa: &dyn TargetIsa, call_conv: CallConv) -> &'static [RU] {
     match isa.triple().pointer_width().unwrap() {
         PointerWidth::U16 => panic!(),
         PointerWidth::U32 => &[RU::rbx, RU::rsi, RU::rdi],
@@ -227,7 +227,7 @@ fn callee_saved_gprs(isa: &TargetIsa, call_conv: CallConv) -> &'static [RU] {
 }
 
 /// Get the set of callee-saved registers that are used.
-fn callee_saved_gprs_used(isa: &TargetIsa, func: &ir::Function) -> RegisterSet {
+fn callee_saved_gprs_used(isa: &dyn TargetIsa, func: &ir::Function) -> RegisterSet {
     let mut all_callee_saved = RegisterSet::empty();
     for reg in callee_saved_gprs(isa, func.signature.call_conv) {
         all_callee_saved.free(GPR, *reg as RegUnit);
@@ -269,7 +269,7 @@ fn callee_saved_gprs_used(isa: &TargetIsa, func: &ir::Function) -> RegisterSet {
     used
 }
 
-pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> {
+pub fn prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> CodegenResult<()> {
     match func.signature.call_conv {
         // For now, just translate fast and cold as system_v.
         CallConv::Fast | CallConv::Cold | CallConv::SystemV => {
@@ -281,7 +281,7 @@ pub fn prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenRes
     }
 }
 
-fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> {
+fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> CodegenResult<()> {
     debug_assert!(
         !isa.flags().probestack_enabled(),
         "baldrdash does not expect cranelift to emit stack probes"
@@ -302,7 +302,7 @@ fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> Code
 
 /// Implementation of the fastcall-based Win64 calling convention described at [1]
 /// [1] https://msdn.microsoft.com/en-us/library/ms235286.aspx
-fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> {
+fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> CodegenResult<()> {
     if isa.triple().pointer_width().unwrap() != PointerWidth::U64 {
         panic!("TODO: windows-fastcall: x86-32 not implemented yet");
     }
@@ -374,7 +374,7 @@ fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> Codeg
 }
 
 /// Insert a System V-compatible prologue and epilogue.
-fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &TargetIsa) -> CodegenResult<()> {
+fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> CodegenResult<()> {
     // The original 32-bit x86 ELF ABI had a 4-byte aligned stack pointer, but
     // newer versions use a 16-byte aligned stack pointer.
     let stack_align = 16;
@@ -435,7 +435,7 @@ fn insert_common_prologue(
     stack_size: i64,
     reg_type: ir::types::Type,
     csrs: &RegisterSet,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) {
     if stack_size > 0 {
         // Check if there is a special stack limit parameter. If so insert stack check.
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index dd7bf02c86..474ef3c544 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -115,7 +115,7 @@ fn expand_sdivrem(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    isa: &isa::TargetIsa,
+    isa: &dyn isa::TargetIsa,
 ) {
     let (x, y, is_srem) = match func.dfg[inst] {
         ir::InstructionData::Binary {
@@ -225,7 +225,7 @@ fn expand_udivrem(
     inst: ir::Inst,
     func: &mut ir::Function,
     _cfg: &mut ControlFlowGraph,
-    isa: &isa::TargetIsa,
+    isa: &dyn isa::TargetIsa,
 ) {
     let (x, y, is_urem) = match func.dfg[inst] {
         ir::InstructionData::Binary {
@@ -278,7 +278,7 @@ fn expand_minmax(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &isa::TargetIsa,
+    _isa: &dyn isa::TargetIsa,
 ) {
     let (x, y, x86_opc, bitwise_opc) = match func.dfg[inst] {
         ir::InstructionData::Binary {
@@ -370,7 +370,7 @@ fn expand_fcvt_from_uint(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &isa::TargetIsa,
+    _isa: &dyn isa::TargetIsa,
 ) {
     let x;
     match func.dfg[inst] {
@@ -441,7 +441,7 @@ fn expand_fcvt_to_sint(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &isa::TargetIsa,
+    _isa: &dyn isa::TargetIsa,
 ) {
     use crate::ir::immediates::{Ieee32, Ieee64};
 
@@ -536,7 +536,7 @@ fn expand_fcvt_to_sint_sat(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &isa::TargetIsa,
+    _isa: &dyn isa::TargetIsa,
 ) {
     use crate::ir::immediates::{Ieee32, Ieee64};
 
@@ -655,7 +655,7 @@ fn expand_fcvt_to_uint(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &isa::TargetIsa,
+    _isa: &dyn isa::TargetIsa,
 ) {
     use crate::ir::immediates::{Ieee32, Ieee64};
 
@@ -736,7 +736,7 @@ fn expand_fcvt_to_uint_sat(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &isa::TargetIsa,
+    _isa: &dyn isa::TargetIsa,
 ) {
     use crate::ir::immediates::{Ieee32, Ieee64};
 
diff --git a/cranelift/codegen/src/isa/x86/mod.rs b/cranelift/codegen/src/isa/x86/mod.rs
index acefe0e786..c71e97410a 100644
--- a/cranelift/codegen/src/isa/x86/mod.rs
+++ b/cranelift/codegen/src/isa/x86/mod.rs
@@ -42,7 +42,7 @@ fn isa_constructor(
     triple: Triple,
     shared_flags: shared_settings::Flags,
     builder: shared_settings::Builder,
-) -> Box {
+) -> Box {
     let level1 = match triple.pointer_width().unwrap() {
         PointerWidth::U16 => unimplemented!("x86-16"),
         PointerWidth::U32 => &enc_tables::LEVEL1_I32[..],
@@ -123,7 +123,7 @@ impl TargetIsa for Isa {
         func: &ir::Function,
         inst: ir::Inst,
         divert: &mut regalloc::RegDiversions,
-        sink: &mut CodeSink,
+        sink: &mut dyn CodeSink,
     ) {
         binemit::emit_inst(func, inst, divert, sink)
     }
diff --git a/cranelift/codegen/src/legalizer/boundary.rs b/cranelift/codegen/src/legalizer/boundary.rs
index c4bbcc9db6..8912743dd5 100644
--- a/cranelift/codegen/src/legalizer/boundary.rs
+++ b/cranelift/codegen/src/legalizer/boundary.rs
@@ -35,7 +35,7 @@ use std::vec::Vec;
 /// This changes all signatures to be ABI-compliant with full `ArgumentLoc` annotations. It doesn't
 /// change the entry block arguments, calls, or return instructions, so this can leave the function
 /// in a state with type discrepancies.
-pub fn legalize_signatures(func: &mut Function, isa: &TargetIsa) {
+pub fn legalize_signatures(func: &mut Function, isa: &dyn TargetIsa) {
     legalize_signature(&mut func.signature, true, isa);
     for sig_data in func.dfg.signatures.values_mut() {
         legalize_signature(sig_data, false, isa);
@@ -49,14 +49,14 @@ pub fn legalize_signatures(func: &mut Function, isa: &TargetIsa) {
 
 /// Legalize the libcall signature, which we may generate on the fly after
 /// `legalize_signatures` has been called.
-pub fn legalize_libcall_signature(signature: &mut Signature, isa: &TargetIsa) {
+pub fn legalize_libcall_signature(signature: &mut Signature, isa: &dyn TargetIsa) {
     legalize_signature(signature, false, isa);
 }
 
 /// Legalize the given signature.
 ///
 /// `current` is true if this is the signature for the current function.
-fn legalize_signature(signature: &mut Signature, current: bool, isa: &TargetIsa) {
+fn legalize_signature(signature: &mut Signature, current: bool, isa: &dyn TargetIsa) {
     isa.legalize_signature(signature, current);
 }
 
diff --git a/cranelift/codegen/src/legalizer/call.rs b/cranelift/codegen/src/legalizer/call.rs
index 6122937f3c..4321dbb90b 100644
--- a/cranelift/codegen/src/legalizer/call.rs
+++ b/cranelift/codegen/src/legalizer/call.rs
@@ -14,7 +14,7 @@ pub fn expand_call(
     inst: ir::Inst,
     func: &mut ir::Function,
     _cfg: &mut ControlFlowGraph,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) {
     // Unpack the instruction.
     let (func_ref, old_args) = match func.dfg[inst] {
diff --git a/cranelift/codegen/src/legalizer/globalvalue.rs b/cranelift/codegen/src/legalizer/globalvalue.rs
index 344a8b8d0f..97ef58c2bb 100644
--- a/cranelift/codegen/src/legalizer/globalvalue.rs
+++ b/cranelift/codegen/src/legalizer/globalvalue.rs
@@ -13,7 +13,7 @@ pub fn expand_global_value(
     inst: ir::Inst,
     func: &mut ir::Function,
     _cfg: &mut ControlFlowGraph,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) {
     // Unpack the instruction.
     let gv = match func.dfg[inst] {
@@ -90,7 +90,7 @@ fn load_addr(
     offset: ir::immediates::Offset32,
     global_type: ir::Type,
     readonly: bool,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) {
     // We need to load a pointer from the `base` global value, so insert a new `global_value`
     // instruction. This depends on the iterative legalization loop. Note that the IR verifier
@@ -123,7 +123,7 @@ fn load_addr(
 }
 
 /// Expand a `global_value` instruction for a symbolic name global.
-fn symbol(inst: ir::Inst, func: &mut ir::Function, gv: ir::GlobalValue, isa: &TargetIsa) {
+fn symbol(inst: ir::Inst, func: &mut ir::Function, gv: ir::GlobalValue, isa: &dyn TargetIsa) {
     let ptr_ty = isa.pointer_type();
     func.dfg.replace(inst).symbol_value(ptr_ty, gv);
 }
diff --git a/cranelift/codegen/src/legalizer/heap.rs b/cranelift/codegen/src/legalizer/heap.rs
index 19217a8af9..33f37155ee 100644
--- a/cranelift/codegen/src/legalizer/heap.rs
+++ b/cranelift/codegen/src/legalizer/heap.rs
@@ -14,7 +14,7 @@ pub fn expand_heap_addr(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &TargetIsa,
+    _isa: &dyn TargetIsa,
 ) {
     // Unpack the instruction.
     let (heap, offset, access_size) = match func.dfg[inst] {
diff --git a/cranelift/codegen/src/legalizer/libcall.rs b/cranelift/codegen/src/legalizer/libcall.rs
index c4cbac30ee..01ff630f01 100644
--- a/cranelift/codegen/src/legalizer/libcall.rs
+++ b/cranelift/codegen/src/legalizer/libcall.rs
@@ -7,7 +7,7 @@ use crate::legalizer::boundary::legalize_libcall_signature;
 use std::vec::Vec;
 
 /// Try to expand `inst` as a library call, returning true is successful.
-pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function, isa: &TargetIsa) -> bool {
+pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function, isa: &dyn TargetIsa) -> bool {
     // Does the opcode/ctrl_type combo even have a well-known runtime library name.
     let libcall = match ir::LibCall::for_inst(func.dfg[inst].opcode(), func.dfg.ctrl_typevar(inst))
     {
diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 2e1db93d42..0a59f0d1b5 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -41,7 +41,7 @@ fn legalize_inst(
     inst: ir::Inst,
     pos: &mut FuncCursor,
     cfg: &mut ControlFlowGraph,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) -> bool {
     let opcode = pos.func.dfg[inst].opcode();
 
@@ -83,7 +83,7 @@ fn legalize_inst(
 /// - Transform any instructions that don't have a legal representation in `isa`.
 /// - Fill out `func.encodings`.
 ///
-pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, isa: &TargetIsa) {
+pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, isa: &dyn TargetIsa) {
     let _tt = timing::legalize();
     debug_assert!(cfg.is_valid());
 
@@ -129,7 +129,7 @@ fn expand_cond_trap(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &TargetIsa,
+    _isa: &dyn TargetIsa,
 ) {
     // Parse the instruction.
     let trapz;
@@ -179,7 +179,7 @@ fn expand_br_table(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) {
     if isa.flags().jump_tables_enabled() {
         expand_br_table_jt(inst, func, cfg, isa);
@@ -193,7 +193,7 @@ fn expand_br_table_jt(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) {
     use crate::ir::condcodes::IntCC;
 
@@ -239,7 +239,7 @@ fn expand_br_table_conds(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &TargetIsa,
+    _isa: &dyn TargetIsa,
 ) {
     use crate::ir::condcodes::IntCC;
 
@@ -280,7 +280,7 @@ fn expand_select(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &TargetIsa,
+    _isa: &dyn TargetIsa,
 ) {
     let (ctrl, tval, fval) = match func.dfg[inst] {
         ir::InstructionData::Ternary {
@@ -315,7 +315,7 @@ fn expand_br_icmp(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &TargetIsa,
+    _isa: &dyn TargetIsa,
 ) {
     let (cond, a, b, destination, ebb_args) = match func.dfg[inst] {
         ir::InstructionData::BranchIcmp {
@@ -350,7 +350,7 @@ fn expand_fconst(
     inst: ir::Inst,
     func: &mut ir::Function,
     _cfg: &mut ControlFlowGraph,
-    _isa: &TargetIsa,
+    _isa: &dyn TargetIsa,
 ) {
     let ty = func.dfg.value_type(func.dfg.first_result(inst));
     debug_assert!(!ty.is_vector(), "Only scalar fconst supported: {}", ty);
@@ -378,7 +378,7 @@ fn expand_stack_load(
     inst: ir::Inst,
     func: &mut ir::Function,
     _cfg: &mut ControlFlowGraph,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) {
     let ty = func.dfg.value_type(func.dfg.first_result(inst));
     let addr_ty = isa.pointer_type();
@@ -410,7 +410,7 @@ fn expand_stack_store(
     inst: ir::Inst,
     func: &mut ir::Function,
     _cfg: &mut ControlFlowGraph,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) {
     let addr_ty = isa.pointer_type();
 
diff --git a/cranelift/codegen/src/legalizer/table.rs b/cranelift/codegen/src/legalizer/table.rs
index 14da52612f..0c4385e96b 100644
--- a/cranelift/codegen/src/legalizer/table.rs
+++ b/cranelift/codegen/src/legalizer/table.rs
@@ -15,7 +15,7 @@ pub fn expand_table_addr(
     inst: ir::Inst,
     func: &mut ir::Function,
     _cfg: &mut ControlFlowGraph,
-    _isa: &TargetIsa,
+    _isa: &dyn TargetIsa,
 ) {
     // Unpack the instruction.
     let (table, index, element_offset) = match func.dfg[inst] {
diff --git a/cranelift/codegen/src/licm.rs b/cranelift/codegen/src/licm.rs
index 14c4630dc7..2af4ef3244 100644
--- a/cranelift/codegen/src/licm.rs
+++ b/cranelift/codegen/src/licm.rs
@@ -17,7 +17,7 @@ use std::vec::Vec;
 /// loop-invariant instructions out of them.
 /// Changes the CFG and domtree in-place during the operation.
 pub fn do_licm(
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
     func: &mut Function,
     cfg: &mut ControlFlowGraph,
     domtree: &mut DominatorTree,
@@ -64,7 +64,7 @@ pub fn do_licm(
 // Insert a pre-header before the header, modifying the function layout and CFG to reflect it.
 // A jump instruction to the header is placed at the end of the pre-header.
 fn create_pre_header(
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
     header: Ebb,
     func: &mut Function,
     cfg: &mut ControlFlowGraph,
diff --git a/cranelift/codegen/src/postopt.rs b/cranelift/codegen/src/postopt.rs
index a82a3e59d3..ef027f6f25 100644
--- a/cranelift/codegen/src/postopt.rs
+++ b/cranelift/codegen/src/postopt.rs
@@ -45,7 +45,7 @@ fn optimize_cpu_flags(
     pos: &mut EncCursor,
     inst: Inst,
     last_flags_clobber: Option,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) {
     // Look for compare and branch patterns.
     // This code could be considerably simplified with non-lexical lifetimes.
@@ -179,7 +179,7 @@ struct MemOpInfo {
     offset: Offset32,
 }
 
-fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa) {
+fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &dyn TargetIsa) {
     // Look for simple loads and stores we can optimize.
     let info = match pos.func.dfg[inst] {
         InstructionData::Load {
@@ -357,7 +357,7 @@ fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &TargetIsa)
 //
 // The main post-opt pass.
 
-pub fn do_postopt(func: &mut Function, isa: &TargetIsa) {
+pub fn do_postopt(func: &mut Function, isa: &dyn TargetIsa) {
     let _tt = timing::postopt();
     let mut pos = EncCursor::new(func, isa);
     while let Some(_ebb) = pos.next_ebb() {
diff --git a/cranelift/codegen/src/print_errors.rs b/cranelift/codegen/src/print_errors.rs
index 2d09845c92..a18b5d6955 100644
--- a/cranelift/codegen/src/print_errors.rs
+++ b/cranelift/codegen/src/print_errors.rs
@@ -17,8 +17,8 @@ use std::vec::Vec;
 /// Pretty-print a verifier error.
 pub fn pretty_verifier_error<'a>(
     func: &ir::Function,
-    isa: Option<&TargetIsa>,
-    func_w: Option>,
+    isa: Option<&dyn TargetIsa>,
+    func_w: Option>,
     errors: VerifierErrors,
 ) -> String {
     let mut errors = errors.0;
@@ -44,14 +44,14 @@ pub fn pretty_verifier_error<'a>(
     w
 }
 
-struct PrettyVerifierError<'a>(Box, &'a mut Vec);
+struct PrettyVerifierError<'a>(Box, &'a mut Vec);
 
 impl<'a> FuncWriter for PrettyVerifierError<'a> {
     fn write_ebb_header(
         &mut self,
-        w: &mut Write,
+        w: &mut dyn Write,
         func: &Function,
-        isa: Option<&TargetIsa>,
+        isa: Option<&dyn TargetIsa>,
         ebb: Ebb,
         indent: usize,
     ) -> fmt::Result {
@@ -60,10 +60,10 @@ impl<'a> FuncWriter for PrettyVerifierError<'a> {
 
     fn write_instruction(
         &mut self,
-        w: &mut Write,
+        w: &mut dyn Write,
         func: &Function,
         aliases: &SecondaryMap>,
-        isa: Option<&TargetIsa>,
+        isa: Option<&dyn TargetIsa>,
         inst: Inst,
         indent: usize,
     ) -> fmt::Result {
@@ -72,10 +72,10 @@ impl<'a> FuncWriter for PrettyVerifierError<'a> {
 
     fn write_entity_definition(
         &mut self,
-        w: &mut Write,
+        w: &mut dyn Write,
         func: &Function,
         entity: AnyEntity,
-        value: &fmt::Display,
+        value: &dyn fmt::Display,
     ) -> fmt::Result {
         pretty_preamble_error(w, func, entity, value, &mut *self.0, self.1)
     }
@@ -83,12 +83,12 @@ impl<'a> FuncWriter for PrettyVerifierError<'a> {
 
 /// Pretty-print a function verifier error for a given EBB.
 fn pretty_ebb_header_error(
-    w: &mut Write,
+    w: &mut dyn Write,
     func: &Function,
-    isa: Option<&TargetIsa>,
+    isa: Option<&dyn TargetIsa>,
     cur_ebb: Ebb,
     indent: usize,
-    func_w: &mut FuncWriter,
+    func_w: &mut dyn FuncWriter,
     errors: &mut Vec,
 ) -> fmt::Result {
     let mut s = String::new();
@@ -121,13 +121,13 @@ fn pretty_ebb_header_error(
 
 /// Pretty-print a function verifier error for a given instruction.
 fn pretty_instruction_error(
-    w: &mut Write,
+    w: &mut dyn Write,
     func: &Function,
     aliases: &SecondaryMap>,
-    isa: Option<&TargetIsa>,
+    isa: Option<&dyn TargetIsa>,
     cur_inst: Inst,
     indent: usize,
-    func_w: &mut FuncWriter,
+    func_w: &mut dyn FuncWriter,
     errors: &mut Vec,
 ) -> fmt::Result {
     let mut s = String::new();
@@ -159,11 +159,11 @@ fn pretty_instruction_error(
 }
 
 fn pretty_preamble_error(
-    w: &mut Write,
+    w: &mut dyn Write,
     func: &Function,
     entity: AnyEntity,
-    value: &fmt::Display,
-    func_w: &mut FuncWriter,
+    value: &dyn fmt::Display,
+    func_w: &mut dyn FuncWriter,
     errors: &mut Vec,
 ) -> fmt::Result {
     let mut s = String::new();
@@ -195,7 +195,7 @@ fn pretty_preamble_error(
 
 /// Prints:
 ///    ;   ^~~~~~
-fn print_arrow(w: &mut Write, entity: &str) -> fmt::Result {
+fn print_arrow(w: &mut dyn Write, entity: &str) -> fmt::Result {
     write!(w, ";")?;
 
     let indent = entity.len() - entity.trim_start().len();
@@ -212,13 +212,13 @@ fn print_arrow(w: &mut Write, entity: &str) -> fmt::Result {
 
 /// Prints:
 ///    ; error: [ERROR BODY]
-fn print_error(w: &mut Write, err: VerifierError) -> fmt::Result {
+fn print_error(w: &mut dyn Write, err: VerifierError) -> fmt::Result {
     writeln!(w, "; error: {}", err.to_string())?;
     Ok(())
 }
 
 /// Pretty-print a Cranelift error.
-pub fn pretty_error(func: &ir::Function, isa: Option<&TargetIsa>, err: CodegenError) -> String {
+pub fn pretty_error(func: &ir::Function, isa: Option<&dyn TargetIsa>, err: CodegenError) -> String {
     if let CodegenError::Verifier(e) = err {
         pretty_verifier_error(func, isa, None, e)
     } else {
diff --git a/cranelift/codegen/src/regalloc/affinity.rs b/cranelift/codegen/src/regalloc/affinity.rs
index bbd29f8f2e..7eea98ab8a 100644
--- a/cranelift/codegen/src/regalloc/affinity.rs
+++ b/cranelift/codegen/src/regalloc/affinity.rs
@@ -48,7 +48,7 @@ impl Affinity {
     }
 
     /// Create an affinity that matches an ABI argument for `isa`.
-    pub fn abi(arg: &AbiParam, isa: &TargetIsa) -> Self {
+    pub fn abi(arg: &AbiParam, isa: &dyn TargetIsa) -> Self {
         match arg.location {
             ArgumentLoc::Unassigned => Affinity::Unassigned,
             ArgumentLoc::Reg(_) => Affinity::Reg(isa.regclass_for_abi_type(arg.value_type).into()),
diff --git a/cranelift/codegen/src/regalloc/coalescing.rs b/cranelift/codegen/src/regalloc/coalescing.rs
index 401b795eea..4ba19ebbf4 100644
--- a/cranelift/codegen/src/regalloc/coalescing.rs
+++ b/cranelift/codegen/src/regalloc/coalescing.rs
@@ -66,7 +66,7 @@ pub struct Coalescing {
 
 /// One-shot context created once per invocation.
 struct Context<'a> {
-    isa: &'a TargetIsa,
+    isa: &'a dyn TargetIsa,
     encinfo: EncInfo,
 
     func: &'a mut Function,
@@ -108,7 +108,7 @@ impl Coalescing {
     /// Convert `func` to Conventional SSA form and build virtual registers in the process.
     pub fn conventional_ssa(
         &mut self,
-        isa: &TargetIsa,
+        isa: &dyn TargetIsa,
         func: &mut Function,
         cfg: &ControlFlowGraph,
         domtree: &DominatorTree,
diff --git a/cranelift/codegen/src/regalloc/coloring.rs b/cranelift/codegen/src/regalloc/coloring.rs
index 12ef018329..4e584b5a64 100644
--- a/cranelift/codegen/src/regalloc/coloring.rs
+++ b/cranelift/codegen/src/regalloc/coloring.rs
@@ -118,7 +118,7 @@ impl Coloring {
     /// Run the coloring algorithm over `func`.
     pub fn run(
         &mut self,
-        isa: &TargetIsa,
+        isa: &dyn TargetIsa,
         func: &mut Function,
         domtree: &DominatorTree,
         liveness: &mut Liveness,
diff --git a/cranelift/codegen/src/regalloc/context.rs b/cranelift/codegen/src/regalloc/context.rs
index a12a09dee1..cb4b4245f4 100644
--- a/cranelift/codegen/src/regalloc/context.rs
+++ b/cranelift/codegen/src/regalloc/context.rs
@@ -75,7 +75,7 @@ impl Context {
     /// location that is consistent with instruction encoding constraints.
     pub fn run(
         &mut self,
-        isa: &TargetIsa,
+        isa: &dyn TargetIsa,
         func: &mut Function,
         cfg: &ControlFlowGraph,
         domtree: &mut DominatorTree,
diff --git a/cranelift/codegen/src/regalloc/liveness.rs b/cranelift/codegen/src/regalloc/liveness.rs
index 0befe50bf3..7b31aedd54 100644
--- a/cranelift/codegen/src/regalloc/liveness.rs
+++ b/cranelift/codegen/src/regalloc/liveness.rs
@@ -195,7 +195,7 @@ type LiveRangeSet = SparseMap;
 fn get_or_create<'a>(
     lrset: &'a mut LiveRangeSet,
     value: Value,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
     func: &Function,
     encinfo: &EncInfo,
 ) -> &'a mut LiveRange {
@@ -389,7 +389,7 @@ impl Liveness {
 
     /// Compute the live ranges of all SSA values used in `func`.
     /// This clears out any existing analysis stored in this data structure.
-    pub fn compute(&mut self, isa: &TargetIsa, func: &mut Function, cfg: &ControlFlowGraph) {
+    pub fn compute(&mut self, isa: &dyn TargetIsa, func: &mut Function, cfg: &ControlFlowGraph) {
         let _tt = timing::ra_liveness();
         self.ranges.clear();
 
diff --git a/cranelift/codegen/src/regalloc/pressure.rs b/cranelift/codegen/src/regalloc/pressure.rs
index 2db3ec3d03..3b15a1c6e2 100644
--- a/cranelift/codegen/src/regalloc/pressure.rs
+++ b/cranelift/codegen/src/regalloc/pressure.rs
@@ -281,7 +281,7 @@ mod tests {
     use target_lexicon::triple;
 
     // Make an arm32 `TargetIsa`, if possible.
-    fn arm32() -> Option> {
+    fn arm32() -> Option> {
         use crate::isa;
         use crate::settings;
 
@@ -294,7 +294,7 @@ mod tests {
     }
 
     // Get a register class by name.
-    fn rc_by_name(isa: &TargetIsa, name: &str) -> RegClass {
+    fn rc_by_name(isa: &dyn TargetIsa, name: &str) -> RegClass {
         isa.register_info()
             .classes
             .iter()
diff --git a/cranelift/codegen/src/regalloc/reload.rs b/cranelift/codegen/src/regalloc/reload.rs
index 11c1a5513d..fb6b61ec6f 100644
--- a/cranelift/codegen/src/regalloc/reload.rs
+++ b/cranelift/codegen/src/regalloc/reload.rs
@@ -65,7 +65,7 @@ impl Reload {
     /// Run the reload algorithm over `func`.
     pub fn run(
         &mut self,
-        isa: &TargetIsa,
+        isa: &dyn TargetIsa,
         func: &mut Function,
         domtree: &DominatorTree,
         liveness: &mut Liveness,
@@ -466,7 +466,7 @@ fn handle_abi_args(
     abi_types: &[AbiParam],
     var_args: &[Value],
     offset: usize,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
     liveness: &Liveness,
 ) {
     debug_assert_eq!(abi_types.len(), var_args.len());
diff --git a/cranelift/codegen/src/regalloc/solver.rs b/cranelift/codegen/src/regalloc/solver.rs
index 96ccdd2841..0d6a816dcb 100644
--- a/cranelift/codegen/src/regalloc/solver.rs
+++ b/cranelift/codegen/src/regalloc/solver.rs
@@ -406,7 +406,7 @@ impl fmt::Display for Move {
 
 impl fmt::Debug for Move {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        let as_display: &fmt::Display = self;
+        let as_display: &dyn fmt::Display = self;
         as_display.fmt(f)
     }
 }
@@ -1139,7 +1139,7 @@ mod tests {
     use target_lexicon::triple;
 
     // Make an arm32 `TargetIsa`, if possible.
-    fn arm32() -> Option> {
+    fn arm32() -> Option> {
         use crate::isa;
         use crate::settings;
 
diff --git a/cranelift/codegen/src/regalloc/spilling.rs b/cranelift/codegen/src/regalloc/spilling.rs
index 9065566003..f45dd41c94 100644
--- a/cranelift/codegen/src/regalloc/spilling.rs
+++ b/cranelift/codegen/src/regalloc/spilling.rs
@@ -91,7 +91,7 @@ impl Spilling {
     /// Run the spilling algorithm over `func`.
     pub fn run(
         &mut self,
-        isa: &TargetIsa,
+        isa: &dyn TargetIsa,
         func: &mut Function,
         domtree: &DominatorTree,
         liveness: &mut Liveness,
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index 6da73076e4..748484b10c 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -346,7 +346,7 @@ pub struct FlagsOrIsa<'a> {
     pub flags: &'a Flags,
 
     /// The ISA may not be present.
-    pub isa: Option<&'a TargetIsa>,
+    pub isa: Option<&'a dyn TargetIsa>,
 }
 
 impl<'a> From<&'a Flags> for FlagsOrIsa<'a> {
@@ -355,8 +355,8 @@ impl<'a> From<&'a Flags> for FlagsOrIsa<'a> {
     }
 }
 
-impl<'a> From<&'a TargetIsa> for FlagsOrIsa<'a> {
-    fn from(isa: &'a TargetIsa) -> FlagsOrIsa {
+impl<'a> From<&'a dyn TargetIsa> for FlagsOrIsa<'a> {
+    fn from(isa: &'a dyn TargetIsa) -> FlagsOrIsa {
         FlagsOrIsa {
             flags: isa.flags(),
             isa: Some(isa),
diff --git a/cranelift/codegen/src/value_label.rs b/cranelift/codegen/src/value_label.rs
index d70d6258fa..8dab92b892 100644
--- a/cranelift/codegen/src/value_label.rs
+++ b/cranelift/codegen/src/value_label.rs
@@ -81,7 +81,7 @@ where
 pub fn build_value_labels_ranges(
     func: &Function,
     regalloc: &Context,
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
 ) -> ValueLabelsRanges
 where
     T: From + Deref + Ord + Copy,
diff --git a/cranelift/codegen/src/verifier/flags.rs b/cranelift/codegen/src/verifier/flags.rs
index 9be8d48280..43a235def5 100644
--- a/cranelift/codegen/src/verifier/flags.rs
+++ b/cranelift/codegen/src/verifier/flags.rs
@@ -24,7 +24,7 @@ use crate::verifier::{VerifierErrors, VerifierStepResult};
 pub fn verify_flags(
     func: &ir::Function,
     cfg: &ControlFlowGraph,
-    isa: Option<&isa::TargetIsa>,
+    isa: Option<&dyn isa::TargetIsa>,
     errors: &mut VerifierErrors,
 ) -> VerifierStepResult<()> {
     let _tt = timing::verify_flags();
diff --git a/cranelift/codegen/src/verifier/liveness.rs b/cranelift/codegen/src/verifier/liveness.rs
index 9556ae3703..c396a46803 100644
--- a/cranelift/codegen/src/verifier/liveness.rs
+++ b/cranelift/codegen/src/verifier/liveness.rs
@@ -23,7 +23,7 @@ use core::cmp::Ordering;
 /// We don't verify that live ranges are minimal. This would require recomputing live ranges for
 /// all values.
 pub fn verify_liveness(
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
     func: &Function,
     cfg: &ControlFlowGraph,
     liveness: &Liveness,
@@ -42,7 +42,7 @@ pub fn verify_liveness(
 }
 
 struct LivenessVerifier<'a> {
-    isa: &'a TargetIsa,
+    isa: &'a dyn TargetIsa,
     func: &'a Function,
     cfg: &'a ControlFlowGraph,
     liveness: &'a Liveness,
diff --git a/cranelift/codegen/src/verifier/locations.rs b/cranelift/codegen/src/verifier/locations.rs
index 26e5edfaf1..bcf006e7f5 100644
--- a/cranelift/codegen/src/verifier/locations.rs
+++ b/cranelift/codegen/src/verifier/locations.rs
@@ -19,7 +19,7 @@ use crate::verifier::{VerifierErrors, VerifierStepResult};
 /// If a liveness analysis is provided, it is used to verify that there are no active register
 /// diversions across control flow edges.
 pub fn verify_locations(
-    isa: &isa::TargetIsa,
+    isa: &dyn isa::TargetIsa,
     func: &ir::Function,
     liveness: Option<&Liveness>,
     errors: &mut VerifierErrors,
@@ -37,7 +37,7 @@ pub fn verify_locations(
 }
 
 struct LocationVerifier<'a> {
-    isa: &'a isa::TargetIsa,
+    isa: &'a dyn isa::TargetIsa,
     func: &'a ir::Function,
     reginfo: isa::RegInfo,
     encinfo: isa::EncInfo,
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 38a52f0934..fc1a38ed99 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -266,7 +266,7 @@ struct Verifier<'a> {
     func: &'a Function,
     expected_cfg: ControlFlowGraph,
     expected_domtree: DominatorTree,
-    isa: Option<&'a TargetIsa>,
+    isa: Option<&'a dyn TargetIsa>,
 }
 
 impl<'a> Verifier<'a> {
diff --git a/cranelift/codegen/src/write.rs b/cranelift/codegen/src/write.rs
index f1810524d8..cf44fa8aca 100644
--- a/cranelift/codegen/src/write.rs
+++ b/cranelift/codegen/src/write.rs
@@ -22,9 +22,9 @@ pub trait FuncWriter {
     /// Write the extended basic block header for the current function.
     fn write_ebb_header(
         &mut self,
-        w: &mut Write,
+        w: &mut dyn Write,
         func: &Function,
-        isa: Option<&TargetIsa>,
+        isa: Option<&dyn TargetIsa>,
         ebb: Ebb,
         indent: usize,
     ) -> fmt::Result;
@@ -32,10 +32,10 @@ pub trait FuncWriter {
     /// Write the given `inst` to `w`.
     fn write_instruction(
         &mut self,
-        w: &mut Write,
+        w: &mut dyn Write,
         func: &Function,
         aliases: &SecondaryMap>,
-        isa: Option<&TargetIsa>,
+        isa: Option<&dyn TargetIsa>,
         inst: Inst,
         indent: usize,
     ) -> fmt::Result;
@@ -43,7 +43,7 @@ pub trait FuncWriter {
     /// Write the preamble to `w`. By default, this uses `write_entity_definition`.
     fn write_preamble(
         &mut self,
-        w: &mut Write,
+        w: &mut dyn Write,
         func: &Function,
         regs: Option<&RegInfo>,
     ) -> Result {
@@ -53,7 +53,7 @@ pub trait FuncWriter {
     /// Default impl of `write_preamble`
     fn super_preamble(
         &mut self,
-        w: &mut Write,
+        w: &mut dyn Write,
         func: &Function,
         regs: Option<&RegInfo>,
     ) -> Result {
@@ -108,10 +108,10 @@ pub trait FuncWriter {
     /// Write an entity definition defined in the preamble to `w`.
     fn write_entity_definition(
         &mut self,
-        w: &mut Write,
+        w: &mut dyn Write,
         func: &Function,
         entity: AnyEntity,
-        value: &fmt::Display,
+        value: &dyn fmt::Display,
     ) -> fmt::Result {
         self.super_entity_definition(w, func, entity, value)
     }
@@ -120,10 +120,10 @@ pub trait FuncWriter {
     #[allow(unused_variables)]
     fn super_entity_definition(
         &mut self,
-        w: &mut Write,
+        w: &mut dyn Write,
         func: &Function,
         entity: AnyEntity,
-        value: &fmt::Display,
+        value: &dyn fmt::Display,
     ) -> fmt::Result {
         writeln!(w, "    {} = {}", entity, value)
     }
@@ -135,10 +135,10 @@ pub struct PlainWriter;
 impl FuncWriter for PlainWriter {
     fn write_instruction(
         &mut self,
-        w: &mut Write,
+        w: &mut dyn Write,
         func: &Function,
         aliases: &SecondaryMap>,
-        isa: Option<&TargetIsa>,
+        isa: Option<&dyn TargetIsa>,
         inst: Inst,
         indent: usize,
     ) -> fmt::Result {
@@ -147,9 +147,9 @@ impl FuncWriter for PlainWriter {
 
     fn write_ebb_header(
         &mut self,
-        w: &mut Write,
+        w: &mut dyn Write,
         func: &Function,
-        isa: Option<&TargetIsa>,
+        isa: Option<&dyn TargetIsa>,
         ebb: Ebb,
         indent: usize,
     ) -> fmt::Result {
@@ -160,7 +160,7 @@ impl FuncWriter for PlainWriter {
 /// Write `func` to `w` as equivalent text.
 /// Use `isa` to emit ISA-dependent annotations.
 pub fn write_function(
-    w: &mut Write,
+    w: &mut dyn Write,
     func: &Function,
     annotations: &DisplayFunctionAnnotations,
 ) -> fmt::Result {
@@ -184,7 +184,7 @@ fn alias_map(func: &Function) -> SecondaryMap> {
 /// pretty_function_error is passed as 'closure' to add error decoration.
 pub fn decorate_function(
     func_w: &mut FW,
-    w: &mut Write,
+    w: &mut dyn Write,
     func: &Function,
     annotations: &DisplayFunctionAnnotations,
 ) -> fmt::Result {
@@ -210,7 +210,7 @@ pub fn decorate_function(
 //
 // Function spec.
 
-fn write_spec(w: &mut Write, func: &Function, regs: Option<&RegInfo>) -> fmt::Result {
+fn write_spec(w: &mut dyn Write, func: &Function, regs: Option<&RegInfo>) -> fmt::Result {
     write!(w, "{}{}", func.name, func.signature.display(regs))
 }
 
@@ -218,7 +218,12 @@ fn write_spec(w: &mut Write, func: &Function, regs: Option<&RegInfo>) -> fmt::Re
 //
 // Basic blocks
 
-fn write_arg(w: &mut Write, func: &Function, regs: Option<&RegInfo>, arg: Value) -> fmt::Result {
+fn write_arg(
+    w: &mut dyn Write,
+    func: &Function,
+    regs: Option<&RegInfo>,
+    arg: Value,
+) -> fmt::Result {
     write!(w, "{}: {}", arg, func.dfg.value_type(arg))?;
     let loc = func.locations[arg];
     if loc.is_assigned() {
@@ -235,9 +240,9 @@ fn write_arg(w: &mut Write, func: &Function, regs: Option<&RegInfo>, arg: Value)
 ///    ebb10(v4: f64, v5: b1):
 ///
 pub fn write_ebb_header(
-    w: &mut Write,
+    w: &mut dyn Write,
     func: &Function,
-    isa: Option<&TargetIsa>,
+    isa: Option<&dyn TargetIsa>,
     ebb: Ebb,
     indent: usize,
 ) -> fmt::Result {
@@ -263,7 +268,7 @@ pub fn write_ebb_header(
     writeln!(w, "):")
 }
 
-fn write_valueloc(w: &mut Write, loc: &ValueLoc, regs: &RegInfo) -> fmt::Result {
+fn write_valueloc(w: &mut dyn Write, loc: &ValueLoc, regs: &RegInfo) -> fmt::Result {
     match loc {
         ValueLoc::Reg(r) => write!(w, "{}", regs.display_regunit(*r)),
         ValueLoc::Stack(ss) => write!(w, "{}", ss),
@@ -272,7 +277,7 @@ fn write_valueloc(w: &mut Write, loc: &ValueLoc, regs: &RegInfo) -> fmt::Result
 }
 
 fn write_value_range_markers(
-    w: &mut Write,
+    w: &mut dyn Write,
     val_ranges: &ValueLabelsRanges,
     regs: &RegInfo,
     offset: u32,
@@ -306,7 +311,7 @@ fn write_value_range_markers(
 
 fn decorate_ebb(
     func_w: &mut FW,
-    w: &mut Write,
+    w: &mut dyn Write,
     func: &Function,
     aliases: &SecondaryMap>,
     annotations: &DisplayFunctionAnnotations,
@@ -385,7 +390,7 @@ fn type_suffix(func: &Function, inst: Inst) -> Option {
 
 /// Write out any aliases to the given target, including indirect aliases
 fn write_value_aliases(
-    w: &mut Write,
+    w: &mut dyn Write,
     aliases: &SecondaryMap>,
     target: Value,
     indent: usize,
@@ -402,10 +407,10 @@ fn write_value_aliases(
 }
 
 fn write_instruction(
-    w: &mut Write,
+    w: &mut dyn Write,
     func: &Function,
     aliases: &SecondaryMap>,
-    isa: Option<&TargetIsa>,
+    isa: Option<&dyn TargetIsa>,
     inst: Inst,
     indent: usize,
 ) -> fmt::Result {
@@ -472,9 +477,9 @@ fn write_instruction(
 
 /// Write the operands of `inst` to `w` with a prepended space.
 pub fn write_operands(
-    w: &mut Write,
+    w: &mut dyn Write,
     dfg: &DataFlowGraph,
-    isa: Option<&TargetIsa>,
+    isa: Option<&dyn TargetIsa>,
     inst: Inst,
 ) -> fmt::Result {
     let pool = &dfg.value_lists;
@@ -687,7 +692,7 @@ pub fn write_operands(
 }
 
 /// Write EBB args using optional parantheses.
-fn write_ebb_args(w: &mut Write, args: &[Value]) -> fmt::Result {
+fn write_ebb_args(w: &mut dyn Write, args: &[Value]) -> fmt::Result {
     if args.is_empty() {
         Ok(())
     } else {
diff --git a/cranelift/entity/src/lib.rs b/cranelift/entity/src/lib.rs
index 8d5579000b..57287d6c79 100644
--- a/cranelift/entity/src/lib.rs
+++ b/cranelift/entity/src/lib.rs
@@ -126,7 +126,7 @@ macro_rules! entity_impl {
 
         impl $crate::__core::fmt::Debug for $entity {
             fn fmt(&self, f: &mut $crate::__core::fmt::Formatter) -> $crate::__core::fmt::Result {
-                (self as &$crate::__core::fmt::Display).fmt(f)
+                (self as &dyn $crate::__core::fmt::Display).fmt(f)
             }
         }
     };
diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index 29f3c0480d..57dc21376a 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -27,10 +27,10 @@ pub enum FaerieTrapCollection {
 
 /// A builder for `FaerieBackend`.
 pub struct FaerieBuilder {
-    isa: Box,
+    isa: Box,
     name: String,
     collect_traps: FaerieTrapCollection,
-    libcall_names: Box String>,
+    libcall_names: Box String>,
 }
 
 impl FaerieBuilder {
@@ -48,10 +48,10 @@ impl FaerieBuilder {
     /// floating point instructions, and for stack probes. If you don't know what to use for this
     /// argument, use `cranelift_module::default_libcall_names()`.
     pub fn new(
-        isa: Box,
+        isa: Box,
         name: String,
         collect_traps: FaerieTrapCollection,
-        libcall_names: Box String>,
+        libcall_names: Box String>,
     ) -> ModuleResult {
         if !isa.flags().is_pic() {
             return Err(ModuleError::Backend(
@@ -71,10 +71,10 @@ impl FaerieBuilder {
 ///
 /// See the `FaerieBuilder` for a convenient way to construct `FaerieBackend` instances.
 pub struct FaerieBackend {
-    isa: Box,
+    isa: Box,
     artifact: faerie::Artifact,
     trap_manifest: Option,
-    libcall_names: Box String>,
+    libcall_names: Box String>,
 }
 
 pub struct FaerieCompiledFunction {
@@ -117,7 +117,7 @@ impl Backend for FaerieBackend {
         }
     }
 
-    fn isa(&self) -> &TargetIsa {
+    fn isa(&self) -> &dyn TargetIsa {
         &*self.isa
     }
 
@@ -361,7 +361,7 @@ struct FaerieRelocSink<'a> {
     artifact: &'a mut faerie::Artifact,
     name: &'a str,
     namespace: &'a ModuleNamespace<'a, FaerieBackend>,
-    libcall_names: &'a Fn(ir::LibCall) -> String,
+    libcall_names: &'a dyn Fn(ir::LibCall) -> String,
 }
 
 impl<'a> RelocSink for FaerieRelocSink<'a> {
diff --git a/cranelift/filetests/src/lib.rs b/cranelift/filetests/src/lib.rs
index 6284e945fa..fbd069bcb5 100644
--- a/cranelift/filetests/src/lib.rs
+++ b/cranelift/filetests/src/lib.rs
@@ -110,7 +110,7 @@ pub fn run_passes(
 ///
 /// This function knows how to create all of the possible `test ` commands that can appear in
 /// a `.clif` test file.
-fn new_subtest(parsed: &TestCommand) -> subtest::SubtestResult> {
+fn new_subtest(parsed: &TestCommand) -> subtest::SubtestResult> {
     match parsed.command {
         "binemit" => test_binemit::subtest(parsed),
         "cat" => test_cat::subtest(parsed),
diff --git a/cranelift/filetests/src/runone.rs b/cranelift/filetests/src/runone.rs
index 323442d4b5..3a69c37c4e 100644
--- a/cranelift/filetests/src/runone.rs
+++ b/cranelift/filetests/src/runone.rs
@@ -102,10 +102,10 @@ pub fn run(path: &Path, passes: Option<&[String]>, target: Option<&str>) -> Test
 
 // Given a slice of tests, generate a vector of (test, flags, isa) tuples.
 fn test_tuples<'a>(
-    tests: &'a [Box],
+    tests: &'a [Box],
     isa_spec: &'a IsaSpec,
     no_isa_flags: &'a Flags,
-) -> SubtestResult)>> {
+) -> SubtestResult)>> {
     let mut out = Vec::new();
     for test in tests {
         if test.needs_isa() {
@@ -131,7 +131,7 @@ fn test_tuples<'a>(
 }
 
 fn run_one_test<'a>(
-    tuple: (&'a SubTest, &'a Flags, Option<&'a TargetIsa>),
+    tuple: (&'a dyn SubTest, &'a Flags, Option<&'a dyn TargetIsa>),
     func: Cow,
     context: &mut Context<'a>,
 ) -> SubtestResult<()> {
diff --git a/cranelift/filetests/src/subtest.rs b/cranelift/filetests/src/subtest.rs
index 4400561cc8..0264169e4c 100644
--- a/cranelift/filetests/src/subtest.rs
+++ b/cranelift/filetests/src/subtest.rs
@@ -25,7 +25,7 @@ pub struct Context<'a> {
 
     /// Target ISA to test against. Only guaranteed to be present for sub-tests whose `needs_isa`
     /// method returned `true`. For other sub-tests, this is set if the test file has a unique ISA.
-    pub isa: Option<&'a TargetIsa>,
+    pub isa: Option<&'a dyn TargetIsa>,
 }
 
 impl<'a> Context<'a> {
diff --git a/cranelift/filetests/src/test_binemit.rs b/cranelift/filetests/src/test_binemit.rs
index a80b3aefe6..0b1e838795 100644
--- a/cranelift/filetests/src/test_binemit.rs
+++ b/cranelift/filetests/src/test_binemit.rs
@@ -18,7 +18,7 @@ use std::fmt::Write;
 
 struct TestBinEmit;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "binemit");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_cat.rs b/cranelift/filetests/src/test_cat.rs
index 88ebadf11e..756b5e5700 100644
--- a/cranelift/filetests/src/test_cat.rs
+++ b/cranelift/filetests/src/test_cat.rs
@@ -13,7 +13,7 @@ use std::borrow::Cow;
 /// The result is verified by filecheck.
 struct TestCat;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "cat");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_compile.rs b/cranelift/filetests/src/test_compile.rs
index fee681d609..e9150a73ee 100644
--- a/cranelift/filetests/src/test_compile.rs
+++ b/cranelift/filetests/src/test_compile.rs
@@ -13,7 +13,7 @@ use std::borrow::Cow;
 
 struct TestCompile;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "compile");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_dce.rs b/cranelift/filetests/src/test_dce.rs
index 2b07654f64..b7b72d2a77 100644
--- a/cranelift/filetests/src/test_dce.rs
+++ b/cranelift/filetests/src/test_dce.rs
@@ -14,7 +14,7 @@ use std::borrow::Cow;
 
 struct TestDCE;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "dce");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_domtree.rs b/cranelift/filetests/src/test_domtree.rs
index e8aeca9bfa..05c7dc31e5 100644
--- a/cranelift/filetests/src/test_domtree.rs
+++ b/cranelift/filetests/src/test_domtree.rs
@@ -25,7 +25,7 @@ use std::fmt::{self, Write};
 
 struct TestDomtree;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "domtree");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_legalizer.rs b/cranelift/filetests/src/test_legalizer.rs
index cb9ab8858c..ec182f4092 100644
--- a/cranelift/filetests/src/test_legalizer.rs
+++ b/cranelift/filetests/src/test_legalizer.rs
@@ -12,7 +12,7 @@ use std::borrow::Cow;
 
 struct TestLegalizer;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "legalizer");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_licm.rs b/cranelift/filetests/src/test_licm.rs
index 32360cadc7..31385f8081 100644
--- a/cranelift/filetests/src/test_licm.rs
+++ b/cranelift/filetests/src/test_licm.rs
@@ -14,7 +14,7 @@ use std::borrow::Cow;
 
 struct TestLICM;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "licm");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_postopt.rs b/cranelift/filetests/src/test_postopt.rs
index e7919b13ca..5d5486e912 100644
--- a/cranelift/filetests/src/test_postopt.rs
+++ b/cranelift/filetests/src/test_postopt.rs
@@ -11,7 +11,7 @@ use std::borrow::Cow;
 
 struct TestPostopt;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "postopt");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_preopt.rs b/cranelift/filetests/src/test_preopt.rs
index 2da3005df6..7e0e23475a 100644
--- a/cranelift/filetests/src/test_preopt.rs
+++ b/cranelift/filetests/src/test_preopt.rs
@@ -15,7 +15,7 @@ use std::borrow::Cow;
 
 struct TestPreopt;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "preopt");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_print_cfg.rs b/cranelift/filetests/src/test_print_cfg.rs
index 7053234dc3..4483c23939 100644
--- a/cranelift/filetests/src/test_print_cfg.rs
+++ b/cranelift/filetests/src/test_print_cfg.rs
@@ -13,7 +13,7 @@ use cranelift_reader::TestCommand;
 /// Object implementing the `test print-cfg` sub-test.
 struct TestPrintCfg;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "print-cfg");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_regalloc.rs b/cranelift/filetests/src/test_regalloc.rs
index a813b43e2f..5a316c022f 100644
--- a/cranelift/filetests/src/test_regalloc.rs
+++ b/cranelift/filetests/src/test_regalloc.rs
@@ -14,7 +14,7 @@ use std::borrow::Cow;
 
 struct TestRegalloc;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "regalloc");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_shrink.rs b/cranelift/filetests/src/test_shrink.rs
index 2cbd2b614f..d32da3c384 100644
--- a/cranelift/filetests/src/test_shrink.rs
+++ b/cranelift/filetests/src/test_shrink.rs
@@ -14,7 +14,7 @@ use std::borrow::Cow;
 
 struct TestShrink;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "shrink");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_simple_gvn.rs b/cranelift/filetests/src/test_simple_gvn.rs
index 3752129a92..831a632951 100644
--- a/cranelift/filetests/src/test_simple_gvn.rs
+++ b/cranelift/filetests/src/test_simple_gvn.rs
@@ -14,7 +14,7 @@ use std::borrow::Cow;
 
 struct TestSimpleGVN;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "simple-gvn");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_simple_preopt.rs b/cranelift/filetests/src/test_simple_preopt.rs
index 4d2e7a7414..286a86ba23 100644
--- a/cranelift/filetests/src/test_simple_preopt.rs
+++ b/cranelift/filetests/src/test_simple_preopt.rs
@@ -11,7 +11,7 @@ use std::borrow::Cow;
 
 struct TestSimplePreopt;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "simple_preopt");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/filetests/src/test_verifier.rs b/cranelift/filetests/src/test_verifier.rs
index 7e057248fb..e8d3ffb3ef 100644
--- a/cranelift/filetests/src/test_verifier.rs
+++ b/cranelift/filetests/src/test_verifier.rs
@@ -19,7 +19,7 @@ use std::fmt::Write;
 
 struct TestVerifier;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "verifier");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 407d03fec4..82a3a09f1a 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -571,7 +571,7 @@ impl<'a> FunctionBuilder<'a> {
     /// Useful for debug purposes. Use it with `None` for standard printing.
     // Clippy thinks the lifetime that follows is needless, but rustc needs it
     #[cfg_attr(feature = "cargo-clippy", allow(clippy::needless_lifetimes))]
-    pub fn display<'b, I: Into>>(&'b self, isa: I) -> DisplayFunction {
+    pub fn display<'b, I: Into>>(&'b self, isa: I) -> DisplayFunction {
         self.func.display(isa)
     }
 }
diff --git a/cranelift/module/src/backend.rs b/cranelift/module/src/backend.rs
index fa23d24f01..316c0a3e2b 100644
--- a/cranelift/module/src/backend.rs
+++ b/cranelift/module/src/backend.rs
@@ -53,7 +53,7 @@ where
     fn new(_: Self::Builder) -> Self;
 
     /// Return the `TargetIsa` to compile for.
-    fn isa(&self) -> &TargetIsa;
+    fn isa(&self) -> &dyn TargetIsa;
 
     /// Declare a function.
     fn declare_function(&mut self, name: &str, linkage: Linkage);
@@ -135,7 +135,7 @@ where
 
 /// Default names for `ir::LibCall`s. A function by this name is imported into the object as
 /// part of the translation of a `ir::ExternalName::LibCall` variant.
-pub fn default_libcall_names() -> Box String> {
+pub fn default_libcall_names() -> Box String> {
     Box::new(move |libcall| match libcall {
         ir::LibCall::Probestack => "__cranelift_probestack".to_owned(),
         ir::LibCall::CeilF32 => "ceilf".to_owned(),
diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 5d8f3c5f60..33a13b7f9a 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -690,7 +690,7 @@ where
     }
 
     /// Return the target isa
-    pub fn isa(&self) -> &isa::TargetIsa {
+    pub fn isa(&self) -> &dyn isa::TargetIsa {
         self.backend.isa()
     }
 
diff --git a/cranelift/preopt/src/lib.rs b/cranelift/preopt/src/lib.rs
index 310d1ba614..e8cf7be321 100644
--- a/cranelift/preopt/src/lib.rs
+++ b/cranelift/preopt/src/lib.rs
@@ -37,7 +37,7 @@ use cranelift_codegen::{isa::TargetIsa, settings::FlagsOrIsa, CodegenResult, Con
 /// Since this can be resource intensive (and code-size inflating),
 /// it is separated from `Context::compile` to allow DCE to remove it
 /// if it's not used.
-pub fn optimize(ctx: &mut Context, isa: &TargetIsa) -> CodegenResult<()> {
+pub fn optimize(ctx: &mut Context, isa: &dyn TargetIsa) -> CodegenResult<()> {
     ctx.verify_if(isa)?;
     fold_constants(ctx, isa)?;
 
diff --git a/cranelift/reader/src/isaspec.rs b/cranelift/reader/src/isaspec.rs
index 6aefce354a..140eb8324b 100644
--- a/cranelift/reader/src/isaspec.rs
+++ b/cranelift/reader/src/isaspec.rs
@@ -19,12 +19,12 @@ pub enum IsaSpec {
 
     /// The parsed file does contain `isa` commands.
     /// Each `isa` command is used to configure a `TargetIsa` trait object.
-    Some(Vec>),
+    Some(Vec>),
 }
 
 impl IsaSpec {
     /// If the `IsaSpec` contains exactly 1 `TargetIsa` we return a reference to it
-    pub fn unique_isa(&self) -> Option<&TargetIsa> {
+    pub fn unique_isa(&self) -> Option<&dyn TargetIsa> {
         if let IsaSpec::Some(ref isa_vec) = *self {
             if isa_vec.len() == 1 {
                 return Some(&*isa_vec[0]);
@@ -35,7 +35,11 @@ impl IsaSpec {
 }
 
 /// Parse an iterator of command line options and apply them to `config`.
-pub fn parse_options<'a, I>(iter: I, config: &mut Configurable, loc: Location) -> ParseResult<()>
+pub fn parse_options<'a, I>(
+    iter: I,
+    config: &mut dyn Configurable,
+    loc: Location,
+) -> ParseResult<()>
 where
     I: Iterator,
 {
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 42dfd55f01..037ed2acce 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -113,11 +113,11 @@ struct Context<'a> {
     /// information. This is only `Some` if exactly one set of `isa` directives were found in the
     /// prologue (it is valid to have directives for multiple different targets, but in that case
     /// we couldn't know which target the provided encodings are intended for)
-    unique_isa: Option<&'a TargetIsa>,
+    unique_isa: Option<&'a dyn TargetIsa>,
 }
 
 impl<'a> Context<'a> {
-    fn new(f: Function, unique_isa: Option<&'a TargetIsa>) -> Self {
+    fn new(f: Function, unique_isa: Option<&'a dyn TargetIsa>) -> Self {
         Self {
             function: f,
             map: SourceMap::new(),
@@ -720,7 +720,7 @@ impl<'a> Parser<'a> {
     }
 
     // Match and consume a register unit either by number `%15` or by name `%rax`.
-    fn match_regunit(&mut self, isa: Option<&TargetIsa>) -> ParseResult {
+    fn match_regunit(&mut self, isa: Option<&dyn TargetIsa>) -> ParseResult {
         if let Some(Token::Name(name)) = self.token() {
             self.consume();
             match isa {
@@ -891,7 +891,7 @@ impl<'a> Parser<'a> {
     /// This is the top-level parse function matching the whole contents of a file.
     pub fn parse_function_list(
         &mut self,
-        unique_isa: Option<&TargetIsa>,
+        unique_isa: Option<&dyn TargetIsa>,
     ) -> ParseResult)>> {
         let mut list = Vec::new();
         while self.token().is_some() {
@@ -911,7 +911,7 @@ impl<'a> Parser<'a> {
     //
     fn parse_function(
         &mut self,
-        unique_isa: Option<&TargetIsa>,
+        unique_isa: Option<&dyn TargetIsa>,
     ) -> ParseResult<(Function, Details<'a>)> {
         // Begin gathering comments.
         // Make sure we don't include any comments before the `function` keyword.
@@ -999,7 +999,7 @@ impl<'a> Parser<'a> {
     //
     // signature ::=  * "(" [paramlist] ")" ["->" retlist] [callconv]
     //
-    fn parse_signature(&mut self, unique_isa: Option<&TargetIsa>) -> ParseResult {
+    fn parse_signature(&mut self, unique_isa: Option<&dyn TargetIsa>) -> ParseResult {
         // Calling convention defaults to `fast`, but can be changed.
         let mut sig = Signature::new(CallConv::Fast);
 
@@ -1033,7 +1033,7 @@ impl<'a> Parser<'a> {
     //
     fn parse_abi_param_list(
         &mut self,
-        unique_isa: Option<&TargetIsa>,
+        unique_isa: Option<&dyn TargetIsa>,
     ) -> ParseResult> {
         let mut list = Vec::new();
 
@@ -1050,7 +1050,7 @@ impl<'a> Parser<'a> {
     }
 
     // Parse a single argument type with flags.
-    fn parse_abi_param(&mut self, unique_isa: Option<&TargetIsa>) -> ParseResult {
+    fn parse_abi_param(&mut self, unique_isa: Option<&dyn TargetIsa>) -> ParseResult {
         // abi-param ::= * type { flag } [ argumentloc ]
         let mut arg = AbiParam::new(self.match_type("expected parameter type")?);
 
@@ -1079,7 +1079,7 @@ impl<'a> Parser<'a> {
     // Parse an argument location specifier; either a register or a byte offset into the stack.
     fn parse_argument_location(
         &mut self,
-        unique_isa: Option<&TargetIsa>,
+        unique_isa: Option<&dyn TargetIsa>,
     ) -> ParseResult {
         // argumentloc ::= '[' regname | uimm32 ']'
         if self.optional(Token::LBracket) {
@@ -1426,7 +1426,7 @@ impl<'a> Parser<'a> {
     //
     fn parse_signature_decl(
         &mut self,
-        unique_isa: Option<&TargetIsa>,
+        unique_isa: Option<&dyn TargetIsa>,
     ) -> ParseResult<(SigRef, Signature)> {
         let sig = self.match_sig("expected signature number: sig«n»")?;
         self.match_token(Token::Equal, "expected '=' in signature decl")?;
diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index 3786e3e964..c182ada674 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -23,9 +23,9 @@ const READONLY_DATA_ALIGNMENT: u8 = 0x1;
 
 /// A builder for `SimpleJITBackend`.
 pub struct SimpleJITBuilder {
-    isa: Box,
+    isa: Box,
     symbols: HashMap,
-    libcall_names: Box String>,
+    libcall_names: Box String>,
 }
 
 impl SimpleJITBuilder {
@@ -35,7 +35,7 @@ impl SimpleJITBuilder {
     /// enum to symbols. LibCalls are inserted in the IR as part of the legalization for certain
     /// floating point instructions, and for stack probes. If you don't know what to use for this
     /// argument, use `cranelift_module::default_libcall_names()`.
-    pub fn new(libcall_names: Box String>) -> Self {
+    pub fn new(libcall_names: Box String>) -> Self {
         let flag_builder = settings::builder();
         let isa_builder = cranelift_native::builder().unwrap_or_else(|msg| {
             panic!("host machine is not supported: {}", msg);
@@ -56,7 +56,10 @@ impl SimpleJITBuilder {
     /// enum to symbols. LibCalls are inserted in the IR as part of the legalization for certain
     /// floating point instructions, and for stack probes. If you don't know what to use for this
     /// argument, use `cranelift_module::default_libcall_names()`.
-    pub fn with_isa(isa: Box, libcall_names: Box String>) -> Self {
+    pub fn with_isa(
+        isa: Box,
+        libcall_names: Box String>,
+    ) -> Self {
         debug_assert!(!isa.flags().is_pic(), "SimpleJIT requires non-PIC code");
         let symbols = HashMap::new();
         Self {
@@ -108,9 +111,9 @@ impl SimpleJITBuilder {
 ///
 /// See the `SimpleJITBuilder` for a convenient way to construct `SimpleJITBackend` instances.
 pub struct SimpleJITBackend {
-    isa: Box,
+    isa: Box,
     symbols: HashMap,
-    libcall_names: Box String>,
+    libcall_names: Box String>,
     code_memory: Memory,
     readonly_memory: Memory,
     writable_memory: Memory,
@@ -205,7 +208,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         }
     }
 
-    fn isa(&self) -> &TargetIsa {
+    fn isa(&self) -> &dyn TargetIsa {
         &*self.isa
     }
 
diff --git a/cranelift/src/disasm.rs b/cranelift/src/disasm.rs
index 65fb33ec06..d7ba0dca6b 100644
--- a/cranelift/src/disasm.rs
+++ b/cranelift/src/disasm.rs
@@ -85,7 +85,7 @@ cfg_if! {
         use capstone::prelude::*;
         use target_lexicon::Architecture;
 
-        fn get_disassembler(isa: &TargetIsa) -> Result {
+        fn get_disassembler(isa: &dyn TargetIsa) -> Result {
             let cs = match isa.triple().architecture {
                 Architecture::Riscv32 | Architecture::Riscv64 => {
                     return Err(String::from("No disassembler for RiscV"))
@@ -117,7 +117,7 @@ cfg_if! {
             cs.map_err(|err| err.to_string())
         }
 
-        pub fn print_disassembly(isa: &TargetIsa, mem: &[u8]) -> Result<(), String> {
+        pub fn print_disassembly(isa: &dyn TargetIsa, mem: &[u8]) -> Result<(), String> {
             let mut cs = get_disassembler(isa)?;
 
             println!("\nDisassembly of {} bytes:", mem.len());
@@ -156,7 +156,7 @@ cfg_if! {
             Ok(())
         }
     } else {
-        pub fn print_disassembly(_: &TargetIsa, _: &[u8]) -> Result<(), String> {
+        pub fn print_disassembly(_: &dyn TargetIsa, _: &[u8]) -> Result<(), String> {
             println!("\nNo disassembly available.");
             Ok(())
         }
@@ -164,7 +164,7 @@ cfg_if! {
 }
 
 pub fn print_all(
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
     mem: &[u8],
     code_size: u32,
     rodata_size: u32,
diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs
index a05c0ac3b0..908b310007 100644
--- a/cranelift/src/utils.rs
+++ b/cranelift/src/utils.rs
@@ -42,7 +42,7 @@ pub fn read_to_end>(path: P) -> io::Result> {
 /// Like `FlagsOrIsa`, but holds ownership.
 pub enum OwnedFlagsOrIsa {
     Flags(settings::Flags),
-    Isa(Box),
+    Isa(Box),
 }
 
 impl OwnedFlagsOrIsa {
diff --git a/cranelift/wasm/src/module_translator.rs b/cranelift/wasm/src/module_translator.rs
index 1cf0f4cd2d..72f257acea 100644
--- a/cranelift/wasm/src/module_translator.rs
+++ b/cranelift/wasm/src/module_translator.rs
@@ -13,7 +13,7 @@ use wasmparser::{ModuleReader, SectionCode};
 /// [`Function`](../codegen/ir/function/struct.Function.html).
 pub fn translate_module<'data>(
     data: &'data [u8],
-    environ: &mut ModuleEnvironment<'data>,
+    environ: &mut dyn ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
     let _tt = timing::wasm_translate_module();
     let mut reader = ModuleReader::new(data)?;
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index 929114984a..e3dacc1b38 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -27,7 +27,7 @@ use wasmparser::{
 /// Parses the Type section of the wasm module.
 pub fn parse_type_section(
     types: TypeSectionReader,
-    environ: &mut ModuleEnvironment,
+    environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
     environ.reserve_signatures(types.get_count());
 
@@ -60,7 +60,7 @@ pub fn parse_type_section(
 /// Parses the Import section of the wasm module.
 pub fn parse_import_section<'data>(
     imports: ImportSectionReader<'data>,
-    environ: &mut ModuleEnvironment<'data>,
+    environ: &mut dyn ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
     environ.reserve_imports(imports.get_count());
 
@@ -122,7 +122,7 @@ pub fn parse_import_section<'data>(
 /// Parses the Function section of the wasm module.
 pub fn parse_function_section(
     functions: FunctionSectionReader,
-    environ: &mut ModuleEnvironment,
+    environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
     environ.reserve_func_types(functions.get_count());
 
@@ -137,7 +137,7 @@ pub fn parse_function_section(
 /// Parses the Table section of the wasm module.
 pub fn parse_table_section(
     tables: TableSectionReader,
-    environ: &mut ModuleEnvironment,
+    environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
     environ.reserve_tables(tables.get_count());
 
@@ -159,7 +159,7 @@ pub fn parse_table_section(
 /// Parses the Memory section of the wasm module.
 pub fn parse_memory_section(
     memories: MemorySectionReader,
-    environ: &mut ModuleEnvironment,
+    environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
     environ.reserve_memories(memories.get_count());
 
@@ -178,7 +178,7 @@ pub fn parse_memory_section(
 /// Parses the Global section of the wasm module.
 pub fn parse_global_section(
     globals: GlobalSectionReader,
-    environ: &mut ModuleEnvironment,
+    environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
     environ.reserve_globals(globals.get_count());
 
@@ -215,7 +215,7 @@ pub fn parse_global_section(
 /// Parses the Export section of the wasm module.
 pub fn parse_export_section<'data>(
     exports: ExportSectionReader<'data>,
-    environ: &mut ModuleEnvironment<'data>,
+    environ: &mut dyn ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
     environ.reserve_exports(exports.get_count());
 
@@ -243,7 +243,7 @@ pub fn parse_export_section<'data>(
 }
 
 /// Parses the Start section of the wasm module.
-pub fn parse_start_section(index: u32, environ: &mut ModuleEnvironment) -> WasmResult<()> {
+pub fn parse_start_section(index: u32, environ: &mut dyn ModuleEnvironment) -> WasmResult<()> {
     environ.declare_start_func(FuncIndex::from_u32(index));
     Ok(())
 }
@@ -251,7 +251,7 @@ pub fn parse_start_section(index: u32, environ: &mut ModuleEnvironment) -> WasmR
 /// Parses the Element section of the wasm module.
 pub fn parse_element_section<'data>(
     elements: ElementSectionReader<'data>,
-    environ: &mut ModuleEnvironment,
+    environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
     environ.reserve_table_elements(elements.get_count());
 
@@ -292,7 +292,7 @@ pub fn parse_element_section<'data>(
 /// Parses the Code section of the wasm module.
 pub fn parse_code_section<'data>(
     code: CodeSectionReader<'data>,
-    environ: &mut ModuleEnvironment<'data>,
+    environ: &mut dyn ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
     for body in code {
         let mut reader = body?.get_binary_reader();
@@ -306,7 +306,7 @@ pub fn parse_code_section<'data>(
 /// Parses the Data section of the wasm module.
 pub fn parse_data_section<'data>(
     data: DataSectionReader<'data>,
-    environ: &mut ModuleEnvironment<'data>,
+    environ: &mut dyn ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
     environ.reserve_data_initializers(data.get_count());
 

From 079ccf1f6e1628e2b373a6cac89fcf5284f15194 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 6 Jun 2019 10:14:17 +0200
Subject: [PATCH 2456/3084] Remove unused results warnings;

---
 cranelift/codegen/src/verifier/mod.rs | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index fc1a38ed99..7e8d7909cf 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -580,7 +580,7 @@ impl<'a> Verifier<'a> {
         }
 
         for &res in self.func.dfg.inst_results(inst) {
-            self.verify_inst_result(inst, res, errors).is_ok();
+            self.verify_inst_result(inst, res, errors)?;
         }
 
         match self.func.dfg[inst] {
@@ -1073,11 +1073,11 @@ impl<'a> Verifier<'a> {
         };
 
         // Typechecking instructions is never fatal
-        self.typecheck_results(inst, ctrl_type, errors).is_ok();
-        self.typecheck_fixed_args(inst, ctrl_type, errors).is_ok();
-        self.typecheck_variable_args(inst, errors).is_ok();
-        self.typecheck_return(inst, errors).is_ok();
-        self.typecheck_special(inst, ctrl_type, errors).is_ok();
+        let _ = self.typecheck_results(inst, ctrl_type, errors);
+        let _ = self.typecheck_fixed_args(inst, ctrl_type, errors);
+        let _ = self.typecheck_variable_args(inst, errors);
+        let _ = self.typecheck_return(inst, errors);
+        let _ = self.typecheck_special(inst, ctrl_type, errors);
 
         // Misuses of copy_nop instructions are fatal
         self.typecheck_copy_nop(inst, errors)?;

From 83336290c23aa9b801c51cff337bd7a9d0412a69 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 26 Jun 2019 17:10:20 +0200
Subject: [PATCH 2457/3084] Mark the jump_table_entry Instruction as loading;

---
 .../codegen/meta-python/base/instructions.py  |  2 +-
 .../codegen/meta/src/shared/instructions.rs   |  3 +-
 .../filetests/licm/jump-table-entry.clif      | 29 +++++++++++++++++++
 3 files changed, 32 insertions(+), 2 deletions(-)
 create mode 100644 cranelift/filetests/filetests/licm/jump-table-entry.clif

diff --git a/cranelift/codegen/meta-python/base/instructions.py b/cranelift/codegen/meta-python/base/instructions.py
index 48adc399ce..1c0ebc164f 100644
--- a/cranelift/codegen/meta-python/base/instructions.py
+++ b/cranelift/codegen/meta-python/base/instructions.py
@@ -164,7 +164,7 @@ jump_table_entry = Instruction(
     Currently, the only type supported is entries which are relative to the
     base of the jump table.
     """,
-    ins=(x, addr, Size, JT), outs=entry)
+    ins=(x, addr, Size, JT), outs=entry, can_load=True)
 
 jump_table_base = Instruction(
     'jump_table_base', r"""
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 284594f6a8..d9a45f0a3b 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -309,7 +309,8 @@ pub fn define(
     "#,
         )
         .operands_in(vec![x, addr, Size, JT])
-        .operands_out(vec![entry]),
+        .operands_out(vec![entry])
+        .can_load(true),
     );
 
     ig.push(
diff --git a/cranelift/filetests/filetests/licm/jump-table-entry.clif b/cranelift/filetests/filetests/licm/jump-table-entry.clif
new file mode 100644
index 0000000000..00a6b3fd14
--- /dev/null
+++ b/cranelift/filetests/filetests/licm/jump-table-entry.clif
@@ -0,0 +1,29 @@
+test licm
+
+target x86_64
+
+function %dont_hoist_jump_table_entry_during_licm() {
+    jt0 = jump_table [ebb1, ebb1]
+
+ebb0:
+    fallthrough ebb1
+
+ebb1: ; the loop!
+    v2 = iconst.i32 42
+    v3 = ifcmp_imm v2, 0
+    brif uge v3, ebb1
+    fallthrough ebb2
+
+ebb2:
+    v1 = iconst.i32 -14
+    v8 = ifcmp_imm v1, 2
+    brif uge v8, ebb1
+    v5 = jump_table_base.i64 jt0
+    v6 = jump_table_entry.i64 v1, v5, 4, jt0
+    v7 = iadd v5, v6
+    indirect_jump_table_br v7, jt0
+; check: ebb2:
+; nextln: v8 = ifcmp_imm.i32 v1, 2
+; nextln: brif uge v8, ebb1
+; nextln: jump_table_entry.i64
+}

From a08444c4c67904dd8a9ee7086baead5f45681112 Mon Sep 17 00:00:00 2001
From: data-pup <16364986+data-pup@users.noreply.github.com>
Date: Thu, 27 Jun 2019 19:44:41 +0000
Subject: [PATCH 2458/3084] fix sections translator doc comment

---
 cranelift/wasm/src/sections_translator.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index e3dacc1b38..1236e943dc 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -1,8 +1,8 @@
 //! Helper functions to gather information for each of the non-function sections of a
 //! WebAssembly module.
 //!
-//! The code of theses helper function is straightforward since it is only about reading metadata
-//! about linear memories, tables, globals, etc. and storing them for later use.
+//! The code of these helper functions is straightforward since they only read metadata
+//! about linear memories, tables, globals, etc. and store them for later use.
 //!
 //! The special case of the initialize expressions for table elements offsets or global variables
 //! is handled, according to the semantics of WebAssembly, to only specific expressions that are

From 83715d638b3c402d4360ab89a6eb07b44634c14f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 27 Jun 2019 16:55:28 -0700
Subject: [PATCH 2459/3084] Update the Rust version badge to match what's
 tested in CI.

---
 cranelift/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index c68452f2c0..bd05f8d67a 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -10,7 +10,7 @@ into executable machine code.
 [![Travis Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
 [![Appveyor Status](https://ci.appveyor.com/api/projects/status/oub7wrrb59utuv8x?svg=true)](https://ci.appveyor.com/project/CraneStation/cranelift)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
-![Minimum rustc 1.33](https://img.shields.io/badge/rustc-1.33+-green.svg)
+![Minimum rustc 1.34](https://img.shields.io/badge/rustc-1.34+-green.svg)
 
 For more information, see [the
 documentation](https://cranelift.readthedocs.io/en/latest/?badge=latest).

From ccd77c1d0b19dd9459900119d83b6804c7baa08c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 27 Jun 2019 16:20:14 -0700
Subject: [PATCH 2460/3084] Update to wasmparser 0.31.0 and goblin 0.0.22.

---
 cranelift/faerie/Cargo.toml             |  2 +-
 cranelift/wasm/Cargo.toml               |  2 +-
 cranelift/wasm/src/code_translator.rs   | 10 ++++++----
 cranelift/wasm/src/translation_utils.rs | 25 ++++++++++++++++++-------
 4 files changed, 26 insertions(+), 13 deletions(-)

diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 71a5179881..1154d422d6 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -13,7 +13,7 @@ edition = "2018"
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0" }
 cranelift-module = { path = "../cranelift-module", version = "0.30.0" }
 faerie = "0.10.0"
-goblin = "0.0.21"
+goblin = "0.0.22"
 failure = "0.1.2"
 target-lexicon = "0.4.0"
 
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 0130fcaa7a..7ad1c55d23 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -11,7 +11,7 @@ keywords = ["webassembly", "wasm"]
 edition = "2018"
 
 [dependencies]
-wasmparser = { version = "0.29.2", default-features = false }
+wasmparser = { version = "0.31.0", default-features = false }
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.30.0", default-features = false }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.30.0", default-features = false }
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index d0b504dd69..14b27b0275 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -25,7 +25,9 @@
 use super::{hash_map, HashMap};
 use crate::environ::{FuncEnvironment, GlobalVariable, ReturnMode, WasmError, WasmResult};
 use crate::state::{ControlStackFrame, TranslationState};
-use crate::translation_utils::{f32_translation, f64_translation, num_return_values, type_to_type};
+use crate::translation_utils::{
+    blocktype_to_type, f32_translation, f64_translation, num_return_values,
+};
 use crate::translation_utils::{FuncIndex, MemoryIndex, SignatureIndex, TableIndex};
 use core::{i32, u32};
 use cranelift_codegen::ir::condcodes::{FloatCC, IntCC};
@@ -130,7 +132,7 @@ pub fn translate_operator(
          ***********************************************************************************/
         Operator::Block { ty } => {
             let next = builder.create_ebb();
-            if let Ok(ty_cre) = type_to_type(ty) {
+            if let Ok(ty_cre) = blocktype_to_type(ty) {
                 builder.append_ebb_param(next, ty_cre);
             }
             state.push_block(next, num_return_values(ty));
@@ -138,7 +140,7 @@ pub fn translate_operator(
         Operator::Loop { ty } => {
             let loop_body = builder.create_ebb();
             let next = builder.create_ebb();
-            if let Ok(ty_cre) = type_to_type(ty) {
+            if let Ok(ty_cre) = blocktype_to_type(ty) {
                 builder.append_ebb_param(next, ty_cre);
             }
             builder.ins().jump(loop_body, &[]);
@@ -156,7 +158,7 @@ pub fn translate_operator(
             //   and we add nothing;
             // - either the If have an Else clause, in that case the destination of this jump
             //   instruction will be changed later when we translate the Else operator.
-            if let Ok(ty_cre) = type_to_type(ty) {
+            if let Ok(ty_cre) = blocktype_to_type(ty) {
                 builder.append_ebb_param(if_not, ty_cre);
             }
             state.push_if(jump_inst, if_not, num_return_values(ty));
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index dc4336c397..18d69c7641 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -119,6 +119,14 @@ pub fn type_to_type(ty: wasmparser::Type) -> Result {
     })
 }
 
+/// Helper function translating wasmparser block signatures to Cranelift types when possible.
+pub fn blocktype_to_type(ty: wasmparser::TypeOrFuncType) -> Result {
+    match ty {
+        wasmparser::TypeOrFuncType::Type(ty) => type_to_type(ty),
+        wasmparser::TypeOrFuncType::FuncType(_) => unimplemented!("multi-value block signatures"),
+    }
+}
+
 /// Turns a `wasmparser` `f32` into a `Cranelift` one.
 pub fn f32_translation(x: wasmparser::Ieee32) -> ir::immediates::Ieee32 {
     ir::immediates::Ieee32::with_bits(x.bits())
@@ -130,14 +138,17 @@ pub fn f64_translation(x: wasmparser::Ieee64) -> ir::immediates::Ieee64 {
 }
 
 /// Translate a `wasmparser` type into its `Cranelift` equivalent, when possible
-pub fn num_return_values(ty: wasmparser::Type) -> usize {
+pub fn num_return_values(ty: wasmparser::TypeOrFuncType) -> usize {
     match ty {
-        wasmparser::Type::EmptyBlockType => 0,
-        wasmparser::Type::I32
-        | wasmparser::Type::F32
-        | wasmparser::Type::I64
-        | wasmparser::Type::F64 => 1,
-        _ => panic!("unsupported return value type"),
+        wasmparser::TypeOrFuncType::Type(ty) => match ty {
+            wasmparser::Type::EmptyBlockType => 0,
+            wasmparser::Type::I32
+            | wasmparser::Type::F32
+            | wasmparser::Type::I64
+            | wasmparser::Type::F64 => 1,
+            _ => panic!("unsupported return value type"),
+        },
+        wasmparser::TypeOrFuncType::FuncType(_) => unimplemented!("multi-value block signatures"),
     }
 }
 

From 4b924d34b64ff9e2dfe9255fc724f621d160dafd Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 28 Jun 2019 11:37:39 -0700
Subject: [PATCH 2461/3084] Bump version to 0.31.0

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    |  8 ++++----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 5737fa1fa0..a25beb4185 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.30.0"
+version = "0.31.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.30.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.30.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.30.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.30.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.30.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.30.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.30.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.30.0" }
-cranelift-module = { path = "cranelift-module", version = "0.30.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.30.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.30.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.30.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.30.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.31.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.31.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.31.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.31.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.31.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.31.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.31.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.31.0" }
+cranelift-module = { path = "cranelift-module", version = "0.31.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.31.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.31.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.31.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.31.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index d5bd3565e4..4c7b5b4cf7 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.30.0"
+version = "0.31.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.30.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.31.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 7a2f2505c8..61aee20daa 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.30.0"
+version = "0.31.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.30.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.30.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.31.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.31.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -26,7 +26,7 @@ log = { version = "0.4.6", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.30.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.31.0", default-features = false }
 
 [features]
 default = ["std", "x86", "arm32", "arm64", "riscv"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index a5f734494d..d8a211e616 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.30.0"
+version = "0.31.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.30.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.31.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index f050953a94..ccc8ca9bd2 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.30.0"
+version = "0.31.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 1154d422d6..41774e1853 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.30.0"
+version = "0.31.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.30.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.31.0" }
 faerie = "0.10.0"
 goblin = "0.0.22"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index dfd63ac0e6..2cf4bb3053 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.30.0"
+version = "0.31.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../cranelift-reader", version = "0.30.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.30.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../cranelift-reader", version = "0.31.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.31.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index b46737cc7c..f1634bdd95 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.30.0"
+version = "0.31.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 9ac859d901..7d67638b24 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.30.0"
+version = "0.31.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.30.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.31.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 5c07355bd6..c5adcead1f 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.30.0"
+version = "0.31.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 9cf1ab4e57..942c94f05d 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.30.0"
+version = "0.31.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.30.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.31.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 175ccfcb0e..ffe2fe4213 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.30.0"
+version="0.31.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index d205de95c2..74b63aff4e 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.30.0"
+version = "0.31.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0" }
 target-lexicon = "0.4.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index ab035ae3fe..7c77ff513d 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.30.0"
+version = "0.31.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.30.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.31.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 7e791d3504..2ced838fb3 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.30.0"
+version = "0.31.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.30.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.30.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.31.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.31.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.4.0" }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.30.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.30.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.30.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.31.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.31.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.31.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 0528694a8f..82c30d6bb0 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.30.0"
+version = "0.31.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.30.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.31.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 7ad1c55d23..6009803751 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.30.0"
+version = "0.31.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.31.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.30.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.30.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.30.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.31.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.31.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From f163050c9a208dc80bc564f3aa662ed317a1d4f2 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 2 Jul 2019 11:54:31 -0700
Subject: [PATCH 2462/3084] Update to wasmparser 0.32.1.

---
 cranelift/wasm/Cargo.toml             | 2 +-
 cranelift/wasm/src/code_translator.rs | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 6009803751..7903cdd300 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -11,7 +11,7 @@ keywords = ["webassembly", "wasm"]
 edition = "2018"
 
 [dependencies]
-wasmparser = { version = "0.31.0", default-features = false }
+wasmparser = { version = "0.32.1", default-features = false }
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.31.0", default-features = false }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.31.0", default-features = false }
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 14b27b0275..839e178f7e 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1047,7 +1047,9 @@ pub fn translate_operator(
         | Operator::F32x4ConvertSI32x4
         | Operator::F32x4ConvertUI32x4
         | Operator::F64x2ConvertSI64x2
-        | Operator::F64x2ConvertUI64x2 => {
+        | Operator::F64x2ConvertUI64x2
+        | Operator::V8x16Shuffle1
+        | Operator::V8x16Shuffle2Imm { .. } => {
             return Err(WasmError::Unsupported("proposed SIMD operators"));
         }
     };

From b35227b4174f9b4ad3b0e3787a4741aed475c2cc Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 2 Jul 2019 12:27:39 -0700
Subject: [PATCH 2463/3084] Bump version to 0.32.0

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    |  8 ++++----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index a25beb4185..b0a1bbbc04 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.31.0"
+version = "0.32.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.31.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.31.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.31.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.31.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.31.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.31.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.31.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.31.0" }
-cranelift-module = { path = "cranelift-module", version = "0.31.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.31.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.31.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.31.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.31.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.32.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.32.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.32.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.32.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.32.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.32.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.32.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.32.0" }
+cranelift-module = { path = "cranelift-module", version = "0.32.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.32.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.32.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.32.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.32.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 4c7b5b4cf7..3672a97827 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.31.0"
+version = "0.32.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.31.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.32.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 61aee20daa..1fa3e22219 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.31.0"
+version = "0.32.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.31.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.31.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.32.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.32.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -26,7 +26,7 @@ log = { version = "0.4.6", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.31.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.32.0", default-features = false }
 
 [features]
 default = ["std", "x86", "arm32", "arm64", "riscv"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index d8a211e616..336dfd9921 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.31.0"
+version = "0.32.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.31.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.32.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index ccc8ca9bd2..d7119634f4 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.31.0"
+version = "0.32.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 41774e1853..cc8311ebe5 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.31.0"
+version = "0.32.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.31.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.32.0" }
 faerie = "0.10.0"
 goblin = "0.0.22"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 2cf4bb3053..fa0a282c5d 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.31.0"
+version = "0.32.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../cranelift-reader", version = "0.31.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.31.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../cranelift-reader", version = "0.32.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.32.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index f1634bdd95..2c33167aa3 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.31.0"
+version = "0.32.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 7d67638b24..4c9121503b 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.31.0"
+version = "0.32.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.31.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.32.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index c5adcead1f..4ceaec7d6b 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.31.0"
+version = "0.32.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 942c94f05d..d8fd89974a 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.31.0"
+version = "0.32.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.31.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.32.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index ffe2fe4213..e9bf6ba1c8 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.31.0"
+version="0.32.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 74b63aff4e..d0ec95bb15 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.31.0"
+version = "0.32.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0" }
 target-lexicon = "0.4.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 7c77ff513d..6208b1d599 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.31.0"
+version = "0.32.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.31.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.32.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 2ced838fb3..6d6e8c8340 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.31.0"
+version = "0.32.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.31.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.31.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.32.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.32.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.4.0" }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.31.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.31.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.31.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.32.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.32.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.32.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 82c30d6bb0..c725fc8cc6 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.31.0"
+version = "0.32.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.31.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.32.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 7903cdd300..a61b52be1d 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.31.0"
+version = "0.32.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.32.1", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.31.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.31.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.31.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.32.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.32.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 460fdaa34d3de41c06715e1fb9337b679c36520d Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Wed, 26 Jun 2019 14:34:27 +0200
Subject: [PATCH 2464/3084] Add verification pass to migrate from EBB to BB.

---
 cranelift/codegen/src/verifier/mod.rs | 51 +++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)

diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 7e8d7909cf..694e5e46de 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -267,6 +267,8 @@ struct Verifier<'a> {
     expected_cfg: ControlFlowGraph,
     expected_domtree: DominatorTree,
     isa: Option<&'a dyn TargetIsa>,
+    // To be removed when #796 is completed.
+    verify_encodable_as_bb: bool,
 }
 
 impl<'a> Verifier<'a> {
@@ -278,6 +280,7 @@ impl<'a> Verifier<'a> {
             expected_cfg,
             expected_domtree,
             isa: fisa.isa,
+            verify_encodable_as_bb: std::env::var("CRANELIFT_BB").is_ok(),
         }
     }
 
@@ -468,6 +471,53 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
+    /// Check that the given EBB can be encoded as a BB, by checking that only
+    /// branching instructions are ending the EBB.
+    fn encodable_as_bb(&self, ebb: Ebb, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
+        // Skip this verification if the environment variable is not set.
+        if !self.verify_encodable_as_bb {
+            return Ok(());
+        };
+
+        let dfg = &self.func.dfg;
+        let inst_iter = self.func.layout.ebb_insts(ebb);
+        // Skip non-branching instructions.
+        let mut inst_iter = inst_iter.skip_while(|&inst| !dfg[inst].opcode().is_branch());
+
+        let branch = match inst_iter.next() {
+            // There is no branch in the current EBB.
+            None => return Ok(()),
+            Some(br) => br,
+        };
+
+        let after_branch = match inst_iter.next() {
+            // The branch is also the terminator.
+            None => return Ok(()),
+            Some(inst) => inst,
+        };
+
+        let after_branch_opcode = dfg[after_branch].opcode();
+        if !after_branch_opcode.is_terminator() {
+            return fatal!(
+                errors,
+                branch,
+                "branch followed by a non-terminator instruction."
+            );
+        };
+
+        // Allow only one conditional branch and a fallthrough implemented with
+        // a jump or fallthrough instruction. Any other, which returns or check
+        // a different condition would have to be moved to a different EBB.
+        match after_branch_opcode {
+            Opcode::Fallthrough | Opcode::Jump => Ok(()),
+            _ => fatal!(
+                errors,
+                after_branch,
+                "terminator instruction not fallthrough or jump"
+            ),
+        }
+    }
+
     fn ebb_integrity(
         &self,
         ebb: Ebb,
@@ -1744,6 +1794,7 @@ impl<'a> Verifier<'a> {
                 self.verify_encoding(inst, errors)?;
                 self.immediate_constraints(inst, errors)?;
             }
+            self.encodable_as_bb(ebb, errors)?;
         }
 
         verify_flags(self.func, &self.expected_cfg, self.isa, errors)?;

From f6ac165ff67b594adab7c9d6c226a6dc82ca22b3 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 1 Jul 2019 12:23:14 +0200
Subject: [PATCH 2465/3084] [wasm] Don't panic when seeing unexpected types but
 properly fail instead;

---
 cranelift/wasm/src/code_translator.rs     |  6 ++--
 cranelift/wasm/src/sections_translator.rs | 20 ++++++-------
 cranelift/wasm/src/translation_utils.rs   | 36 +++++++++++++++++------
 3 files changed, 40 insertions(+), 22 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 839e178f7e..6365ed7e68 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -135,7 +135,7 @@ pub fn translate_operator(
             if let Ok(ty_cre) = blocktype_to_type(ty) {
                 builder.append_ebb_param(next, ty_cre);
             }
-            state.push_block(next, num_return_values(ty));
+            state.push_block(next, num_return_values(ty)?);
         }
         Operator::Loop { ty } => {
             let loop_body = builder.create_ebb();
@@ -144,7 +144,7 @@ pub fn translate_operator(
                 builder.append_ebb_param(next, ty_cre);
             }
             builder.ins().jump(loop_body, &[]);
-            state.push_loop(loop_body, next, num_return_values(ty));
+            state.push_loop(loop_body, next, num_return_values(ty)?);
             builder.switch_to_block(loop_body);
             environ.translate_loop_header(builder.cursor())?;
         }
@@ -161,7 +161,7 @@ pub fn translate_operator(
             if let Ok(ty_cre) = blocktype_to_type(ty) {
                 builder.append_ebb_param(if_not, ty_cre);
             }
-            state.push_if(jump_inst, if_not, num_return_values(ty));
+            state.push_if(jump_inst, if_not, num_return_values(ty)?);
         }
         Operator::Else => {
             // We take the control frame pushed by the if, use its ebb as the else body
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index 1236e943dc..f0975d9643 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -7,10 +7,10 @@
 //! The special case of the initialize expressions for table elements offsets or global variables
 //! is handled, according to the semantics of WebAssembly, to only specific expressions that are
 //! interpreted on the fly.
-use crate::environ::{ModuleEnvironment, WasmResult};
+use crate::environ::{ModuleEnvironment, WasmError, WasmResult};
 use crate::translation_utils::{
-    type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex, SignatureIndex,
-    Table, TableElementType, TableIndex,
+    tabletype_to_type, type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory,
+    MemoryIndex, SignatureIndex, Table, TableElementType, TableIndex,
 };
 use core::convert::TryFrom;
 use cranelift_codegen::ir::{self, AbiParam, Signature};
@@ -51,7 +51,7 @@ pub fn parse_type_section(
                 }));
                 environ.declare_signature(sig);
             }
-            ref s => panic!("unsupported type: {:?}", s),
+            _ => return Err(WasmError::Unsupported("unsupported type in type section")),
         }
     }
     Ok(())
@@ -101,9 +101,9 @@ pub fn parse_import_section<'data>(
             ImportSectionEntryType::Table(ref tab) => {
                 environ.declare_table_import(
                     Table {
-                        ty: match type_to_type(tab.element_type) {
-                            Ok(t) => TableElementType::Val(t),
-                            Err(()) => TableElementType::Func,
+                        ty: match tabletype_to_type(tab.element_type)? {
+                            Some(t) => TableElementType::Val(t),
+                            None => TableElementType::Func,
                         },
                         minimum: tab.limits.initial,
                         maximum: tab.limits.maximum,
@@ -144,9 +144,9 @@ pub fn parse_table_section(
     for entry in tables {
         let table = entry?;
         environ.declare_table(Table {
-            ty: match type_to_type(table.element_type) {
-                Ok(t) => TableElementType::Val(t),
-                Err(()) => TableElementType::Func,
+            ty: match tabletype_to_type(table.element_type)? {
+                Some(t) => TableElementType::Val(t),
+                None => TableElementType::Func,
             },
             minimum: table.limits.initial,
             maximum: table.limits.maximum,
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index 18d69c7641..c221f751ff 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -1,4 +1,5 @@
 //! Helper functions and structures for the translation.
+use crate::environ::{WasmError, WasmResult};
 use core::u32;
 use cranelift_codegen::entity::entity_impl;
 use cranelift_codegen::ir;
@@ -109,21 +110,36 @@ pub struct Memory {
 }
 
 /// Helper function translating wasmparser types to Cranelift types when possible.
-pub fn type_to_type(ty: wasmparser::Type) -> Result {
+pub fn type_to_type(ty: wasmparser::Type) -> WasmResult {
     Ok(match ty {
         wasmparser::Type::I32 => ir::types::I32,
         wasmparser::Type::I64 => ir::types::I64,
         wasmparser::Type::F32 => ir::types::F32,
         wasmparser::Type::F64 => ir::types::F64,
-        _ => return Err(()),
+        _ => return Err(WasmError::Unsupported("unsupported wasm type")),
+    })
+}
+
+/// Helper function translating wasmparser possible table types to Cranelift types when possible,
+/// or None for Func tables.
+pub fn tabletype_to_type(ty: wasmparser::Type) -> WasmResult> {
+    Ok(match ty {
+        wasmparser::Type::I32 => Some(ir::types::I32),
+        wasmparser::Type::I64 => Some(ir::types::I64),
+        wasmparser::Type::F32 => Some(ir::types::F32),
+        wasmparser::Type::F64 => Some(ir::types::F64),
+        wasmparser::Type::AnyFunc => None,
+        _ => return Err(WasmError::Unsupported("unsupported table wasm type")),
     })
 }
 
 /// Helper function translating wasmparser block signatures to Cranelift types when possible.
-pub fn blocktype_to_type(ty: wasmparser::TypeOrFuncType) -> Result {
+pub fn blocktype_to_type(ty: wasmparser::TypeOrFuncType) -> WasmResult {
     match ty {
         wasmparser::TypeOrFuncType::Type(ty) => type_to_type(ty),
-        wasmparser::TypeOrFuncType::FuncType(_) => unimplemented!("multi-value block signatures"),
+        wasmparser::TypeOrFuncType::FuncType(_) => {
+            Err(WasmError::Unsupported("multi-value block signatures"))
+        }
     }
 }
 
@@ -138,17 +154,19 @@ pub fn f64_translation(x: wasmparser::Ieee64) -> ir::immediates::Ieee64 {
 }
 
 /// Translate a `wasmparser` type into its `Cranelift` equivalent, when possible
-pub fn num_return_values(ty: wasmparser::TypeOrFuncType) -> usize {
+pub fn num_return_values(ty: wasmparser::TypeOrFuncType) -> WasmResult {
     match ty {
         wasmparser::TypeOrFuncType::Type(ty) => match ty {
-            wasmparser::Type::EmptyBlockType => 0,
+            wasmparser::Type::EmptyBlockType => Ok(0),
             wasmparser::Type::I32
             | wasmparser::Type::F32
             | wasmparser::Type::I64
-            | wasmparser::Type::F64 => 1,
-            _ => panic!("unsupported return value type"),
+            | wasmparser::Type::F64 => Ok(1),
+            _ => Err(WasmError::Unsupported("unsupported return value type")),
         },
-        wasmparser::TypeOrFuncType::FuncType(_) => unimplemented!("multi-value block signatures"),
+        wasmparser::TypeOrFuncType::FuncType(_) => {
+            Err(WasmError::Unsupported("multi-value block signatures"))
+        }
     }
 }
 

From 102dbbb343aca151086527754fa79d7eb72f65d4 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 2 Jul 2019 15:43:41 -0700
Subject: [PATCH 2466/3084] Bump the required rust version to 1.35.

Rust 1.34 gets errors like this:

```
$ cargo +1.34.0 build --release
   Compiling cranelift-wasm v0.32.0 (cranelift/cranelift-wasm)
error[E0259]: the name `std` is defined multiple times
  |
  = note: `std` must be defined only once in the type namespace of this module

error: aborting due to previous error

For more information about this error, try `rustc --explain E0259`.
error: Could not compile `cranelift-wasm`.

To learn more, run the command again with --verbose.
```

Updating to Rust 1.35 fixes this.
---
 .travis.yml         | 2 +-
 cranelift/README.md | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 2465863fe3..212f86aff6 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,7 +7,7 @@ language: rust
 rust:
     # The oldest version we currently support. See
     # CONTRIBUTING.md#rustc-version-support for details.
-    - 1.34.0
+    - 1.35.0
     - beta
     - nightly
 matrix:
diff --git a/cranelift/README.md b/cranelift/README.md
index bd05f8d67a..fe25d2a842 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -10,7 +10,7 @@ into executable machine code.
 [![Travis Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
 [![Appveyor Status](https://ci.appveyor.com/api/projects/status/oub7wrrb59utuv8x?svg=true)](https://ci.appveyor.com/project/CraneStation/cranelift)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
-![Minimum rustc 1.34](https://img.shields.io/badge/rustc-1.34+-green.svg)
+![Minimum rustc 1.35](https://img.shields.io/badge/rustc-1.35+-green.svg)
 
 For more information, see [the
 documentation](https://cranelift.readthedocs.io/en/latest/?badge=latest).
@@ -53,7 +53,7 @@ needed before it would be ready for a production use case.
 
 Cranelift's APIs are not yet stable.
 
-Cranelift currently requires Rust 1.32 or later, and Python 2.7 or 3
+Cranelift currently requires Rust 1.35 or later, and Python 2.7 or 3
 to build.
 
 Planned uses

From f1d1d1e960496befd5ea9dd0f727800a431f2ad7 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 6 Jun 2019 16:33:18 +0200
Subject: [PATCH 2467/3084] [meta] Uniquely number every instruction in the
 Rust crate;

---
 .../codegen/meta/src/cdsl/instructions.rs     | 40 +++++---
 cranelift/codegen/meta/src/cdsl/xform.rs      |  5 +-
 cranelift/codegen/meta/src/constant_hash.rs   | 16 +++-
 cranelift/codegen/meta/src/gen_inst.rs        | 94 +++++++++----------
 cranelift/codegen/meta/src/gen_settings.rs    |  4 +-
 cranelift/codegen/meta/src/isa/arm32/mod.rs   |  1 +
 cranelift/codegen/meta/src/isa/arm64/mod.rs   |  1 +
 cranelift/codegen/meta/src/isa/riscv/mod.rs   |  1 +
 .../codegen/meta/src/isa/x86/instructions.rs  | 15 ++-
 cranelift/codegen/meta/src/isa/x86/mod.rs     |  5 +-
 cranelift/codegen/meta/src/lib.rs             | 11 +--
 .../codegen/meta/src/shared/instructions.rs   | 11 ++-
 cranelift/codegen/meta/src/shared/mod.rs      | 13 ++-
 13 files changed, 127 insertions(+), 90 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index d57bcc254a..4bcd6ea402 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -1,3 +1,10 @@
+use std::fmt;
+use std::ops;
+use std::rc::Rc;
+use std::slice;
+
+use cranelift_entity::{entity_impl, PrimaryMap};
+
 use crate::cdsl::camel_case;
 use crate::cdsl::formats::{
     FormatField, FormatRegistry, InstructionFormat, InstructionFormatIndex,
@@ -7,41 +14,49 @@ use crate::cdsl::type_inference::Constraint;
 use crate::cdsl::types::{LaneType, ValueType};
 use crate::cdsl::typevar::TypeVar;
 
-use std::fmt;
-use std::ops;
-use std::rc::Rc;
-use std::slice;
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct OpcodeNumber(u32);
+entity_impl!(OpcodeNumber);
 
-pub struct InstructionGroupBuilder<'format_reg> {
+pub type AllInstructions = PrimaryMap;
+
+pub struct InstructionGroupBuilder<'format_reg, 'all_inst> {
     _name: &'static str,
     _doc: &'static str,
     format_registry: &'format_reg FormatRegistry,
-    instructions: Vec,
+    all_instructions: &'all_inst mut AllInstructions,
+    own_instructions: Vec,
 }
 
-impl<'format_reg> InstructionGroupBuilder<'format_reg> {
+impl<'format_reg, 'all_inst> InstructionGroupBuilder<'format_reg, 'all_inst> {
     pub fn new(
         name: &'static str,
         doc: &'static str,
+        all_instructions: &'all_inst mut AllInstructions,
         format_registry: &'format_reg FormatRegistry,
     ) -> Self {
         Self {
             _name: name,
             _doc: doc,
             format_registry,
-            instructions: Vec::new(),
+            all_instructions,
+            own_instructions: Vec::new(),
         }
     }
 
     pub fn push(&mut self, builder: InstructionBuilder) {
-        self.instructions.push(builder.build(self.format_registry));
+        let opcode_number = OpcodeNumber(self.all_instructions.next_key().as_u32());
+        let inst = builder.build(self.format_registry, opcode_number);
+        // Note this clone is cheap, since Instruction is a Rc<> wrapper for InstructionContent.
+        self.own_instructions.push(inst.clone());
+        self.all_instructions.push(inst);
     }
 
     pub fn build(self) -> InstructionGroup {
         InstructionGroup {
             _name: self._name,
             _doc: self._doc,
-            instructions: self.instructions,
+            instructions: self.own_instructions,
         }
     }
 }
@@ -78,6 +93,7 @@ pub struct InstructionContent {
     /// Instruction mnemonic, also becomes opcode name.
     pub name: String,
     pub camel_name: String,
+    pub opcode_number: OpcodeNumber,
 
     /// Documentation string.
     doc: String,
@@ -300,7 +316,7 @@ impl InstructionBuilder {
         self
     }
 
-    fn build(self, format_registry: &FormatRegistry) -> Instruction {
+    fn build(self, format_registry: &FormatRegistry, opcode_number: OpcodeNumber) -> Instruction {
         let operands_in = self.operands_in.unwrap_or_else(Vec::new);
         let operands_out = self.operands_out.unwrap_or_else(Vec::new);
 
@@ -333,10 +349,12 @@ impl InstructionBuilder {
         let writes_cpu_flags = operands_out.iter().any(|op| op.is_cpu_flags());
 
         let camel_name = camel_case(&self.name);
+
         Instruction {
             content: Rc::new(InstructionContent {
                 name: self.name,
                 camel_name,
+                opcode_number,
                 doc: self.doc,
                 operands_in,
                 operands_out,
diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index 0cc0c6a03c..e988bf12ac 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -404,11 +404,12 @@ impl TransformGroups {
 #[should_panic]
 fn test_double_custom_legalization() {
     use crate::cdsl::formats::{FormatRegistry, InstructionFormatBuilder};
-    use crate::cdsl::instructions::{InstructionBuilder, InstructionGroupBuilder};
+    use crate::cdsl::instructions::{AllInstructions, InstructionBuilder, InstructionGroupBuilder};
 
+    let mut dummy_all = AllInstructions::new();
     let mut format = FormatRegistry::new();
     format.insert(InstructionFormatBuilder::new("nullary"));
-    let mut inst_group = InstructionGroupBuilder::new("test", "", &format);
+    let mut inst_group = InstructionGroupBuilder::new("test", "", &mut dummy_all, &format);
     inst_group.push(InstructionBuilder::new("dummy", "doc"));
     let inst_group = inst_group.build();
     let dummy_inst = inst_group.by_name("dummy");
diff --git a/cranelift/codegen/meta/src/constant_hash.rs b/cranelift/codegen/meta/src/constant_hash.rs
index 7f629d0d8b..4090e4d77d 100644
--- a/cranelift/codegen/meta/src/constant_hash.rs
+++ b/cranelift/codegen/meta/src/constant_hash.rs
@@ -1,3 +1,5 @@
+use std::iter;
+
 pub fn simple_hash(s: &str) -> usize {
     let mut h: u32 = 5381;
     for c in s.chars() {
@@ -9,8 +11,12 @@ pub fn simple_hash(s: &str) -> usize {
 /// Compute an open addressed, quadratically probed hash table containing
 /// `items`. The returned table is a list containing the elements of the
 /// iterable `items` and `None` in unused slots.
-pub fn generate_table usize>(items: &Vec, hash_function: H) -> Vec> {
-    let size = (1.20 * items.len() as f64) as usize;
+pub fn generate_table<'cont, T, I: iter::Iterator, H: Fn(&T) -> usize>(
+    items: I,
+    num_items: usize,
+    hash_function: H,
+) -> Vec> {
+    let size = (1.20 * num_items as f64) as usize;
     // TODO do we really need the multiply by two here?
     let size = if size.is_power_of_two() {
         size * 2
@@ -18,10 +24,10 @@ pub fn generate_table usize>(items: &Vec, hash_function: H) -
         size.next_power_of_two()
     };
 
-    let mut table: Vec> = vec![None; size];
+    let mut table = vec![None; size];
 
     for i in items {
-        let mut h = hash_function(i) % size;
+        let mut h = hash_function(&i) % size;
         let mut s = 0;
         while table[h].is_some() {
             s += 1;
@@ -36,7 +42,7 @@ pub fn generate_table usize>(items: &Vec, hash_function: H) -
 #[test]
 fn test_generate_table() {
     let v = vec!["Hello".to_string(), "world".to_string()];
-    let table = generate_table(&v, |s| simple_hash(&s));
+    let table = generate_table(v.iter(), v.len(), |s| simple_hash(&s));
     assert_eq!(
         table,
         vec![
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index 613987421b..8e499cb866 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -1,15 +1,20 @@
+use std::fmt;
+
+use cranelift_entity::EntityRef;
+
 use crate::cdsl::camel_case;
 use crate::cdsl::formats::{FormatRegistry, InstructionFormat};
-use crate::cdsl::instructions::{Instruction, InstructionGroup};
+use crate::cdsl::instructions::{AllInstructions, Instruction};
 use crate::cdsl::operands::Operand;
 use crate::cdsl::typevar::{TypeSet, TypeVar};
+
+use crate::shared::Definitions as SharedDefinitions;
+
 use crate::constant_hash;
 use crate::error;
 use crate::srcgen::{Formatter, Match};
 use crate::unique_table::{UniqueSeqTable, UniqueTable};
 
-use std::fmt;
-
 // TypeSet indexes are encoded in 8 bits, with `0xff` reserved.
 const TYPESET_LIMIT: usize = 0xff;
 
@@ -368,7 +373,7 @@ fn gen_instruction_data_impl(registry: &FormatRegistry, fmt: &mut Formatter) {
 }
 
 fn gen_bool_accessor bool>(
-    instruction_groups: &Vec<&InstructionGroup>,
+    all_inst: &AllInstructions,
     get_attr: T,
     name: &'static str,
     doc: &'static str,
@@ -378,11 +383,9 @@ fn gen_bool_accessor bool>(
     fmtln!(fmt, "pub fn {}(self) -> bool {{", name);
     fmt.indent(|fmt| {
         let mut m = Match::new("self");
-        for group in instruction_groups.iter() {
-            for inst in group.iter() {
-                if get_attr(inst) {
-                    m.arm_no_fields(format!("Opcode::{}", inst.camel_name), "true");
-                }
+        for inst in all_inst.values() {
+            if get_attr(inst) {
+                m.arm_no_fields(format!("Opcode::{}", inst.camel_name), "true");
             }
         }
         m.arm_no_fields("_", "false");
@@ -392,18 +395,7 @@ fn gen_bool_accessor bool>(
     fmt.empty_line();
 }
 
-fn gen_opcodes<'a>(
-    formats: &FormatRegistry,
-    igroups: &Vec<&'a InstructionGroup>,
-    fmt: &mut Formatter,
-) -> Vec<&'a Instruction> {
-    let mut all_inst = Vec::new();
-    for group in igroups {
-        for inst in group.iter() {
-            all_inst.push(inst);
-        }
-    }
-
+fn gen_opcodes<'a>(all_inst: &AllInstructions, formats: &FormatRegistry, fmt: &mut Formatter) {
     fmt.doc_comment(
         r#"
         An instruction opcode.
@@ -420,10 +412,7 @@ fn gen_opcodes<'a>(
     fmt.line("pub enum Opcode {");
     fmt.indent(|fmt| {
         let mut is_first_opcode = true;
-        for inst in &all_inst {
-            // TODO we might need to set an instruction number here. Probably can do in the
-            // InstructionGroup itself when adding instruction (would need to remember last
-            // instruction number in the SharedDefinitions or somewhere else).
+        for inst in all_inst.values() {
             let format = formats.get(inst.format);
             fmt.doc_comment(format!("`{}`. ({})", inst, format.name));
 
@@ -440,6 +429,8 @@ fn gen_opcodes<'a>(
 
             // Enum variant itself.
             if is_first_opcode {
+                assert!(inst.opcode_number.index() == 0);
+                // TODO the python crate requires opcode numbers to start from one.
                 fmtln!(fmt, "{} = 1,", inst.camel_name);
                 is_first_opcode = false;
             } else {
@@ -453,77 +444,77 @@ fn gen_opcodes<'a>(
     fmt.line("impl Opcode {");
     fmt.indent(|fmt| {
         gen_bool_accessor(
-            igroups,
+            all_inst,
             |inst| inst.is_terminator,
             "is_terminator",
             "True for instructions that terminate the EBB",
             fmt,
         );
         gen_bool_accessor(
-            igroups,
+            all_inst,
             |inst| inst.is_branch,
             "is_branch",
             "True for all branch or jump instructions.",
             fmt,
         );
         gen_bool_accessor(
-            igroups,
+            all_inst,
             |inst| inst.is_indirect_branch,
             "is_indirect_branch",
             "True for all indirect branch or jump instructions.",
             fmt,
         );
         gen_bool_accessor(
-            igroups,
+            all_inst,
             |inst| inst.is_call,
             "is_call",
             "Is this a call instruction?",
             fmt,
         );
         gen_bool_accessor(
-            igroups,
+            all_inst,
             |inst| inst.is_return,
             "is_return",
             "Is this a return instruction?",
             fmt,
         );
         gen_bool_accessor(
-            igroups,
+            all_inst,
             |inst| inst.is_ghost,
             "is_ghost",
             "Is this a ghost instruction?",
             fmt,
         );
         gen_bool_accessor(
-            igroups,
+            all_inst,
             |inst| inst.can_load,
             "can_load",
             "Can this instruction read from memory?",
             fmt,
         );
         gen_bool_accessor(
-            igroups,
+            all_inst,
             |inst| inst.can_store,
             "can_store",
             "Can this instruction write to memory?",
             fmt,
         );
         gen_bool_accessor(
-            igroups,
+            all_inst,
             |inst| inst.can_trap,
             "can_trap",
             "Can this instruction cause a trap?",
             fmt,
         );
         gen_bool_accessor(
-            igroups,
+            all_inst,
             |inst| inst.other_side_effects,
             "other_side_effects",
             "Does this instruction have other side effects besides can_* flags?",
             fmt,
         );
         gen_bool_accessor(
-            igroups,
+            all_inst,
             |inst| inst.writes_cpu_flags,
             "writes_cpu_flags",
             "Does this instruction write to CPU flags?",
@@ -540,7 +531,7 @@ fn gen_opcodes<'a>(
         all_inst.len()
     );
     fmt.indent(|fmt| {
-        for inst in &all_inst {
+        for inst in all_inst.values() {
             let format = formats.get(inst.format);
             fmtln!(fmt, "InstructionFormat::{}, // {}", format.name, inst.name);
         }
@@ -552,7 +543,7 @@ fn gen_opcodes<'a>(
     fmt.line("fn opcode_name(opc: Opcode) -> &\'static str {");
     fmt.indent(|fmt| {
         let mut m = Match::new("opc");
-        for inst in &all_inst {
+        for inst in all_inst.values() {
             m.arm_no_fields(
                 format!("Opcode::{}", inst.camel_name),
                 format!("\"{}\"", inst.name),
@@ -564,8 +555,9 @@ fn gen_opcodes<'a>(
     fmt.empty_line();
 
     // Generate an opcode hash table for looking up opcodes by name.
-    let hash_table =
-        constant_hash::generate_table(&all_inst, |inst| constant_hash::simple_hash(&inst.name));
+    let hash_table = constant_hash::generate_table(all_inst.values(), all_inst.len(), |inst| {
+        constant_hash::simple_hash(&inst.name)
+    });
     fmtln!(
         fmt,
         "const OPCODE_HASH_TABLE: [Option; {}] = [",
@@ -581,8 +573,6 @@ fn gen_opcodes<'a>(
     });
     fmtln!(fmt, "];");
     fmt.empty_line();
-
-    all_inst
 }
 
 /// Get the value type constraint for an SSA value operand, where
@@ -698,7 +688,7 @@ pub fn gen_typesets_table(type_sets: &UniqueTable, fmt: &mut Formatter)
 /// - Emit a compact constant table of ValueTypeSet objects.
 /// - Emit a compact constant table of OperandConstraint objects.
 /// - Emit an opcode-indexed table of instruction constraints.
-fn gen_type_constraints(all_inst: &Vec<&Instruction>, fmt: &mut Formatter) {
+fn gen_type_constraints(all_inst: &AllInstructions, fmt: &mut Formatter) {
     // Table of TypeSet instances.
     let mut type_sets = UniqueTable::new();
 
@@ -719,7 +709,7 @@ fn gen_type_constraints(all_inst: &Vec<&Instruction>, fmt: &mut Formatter) {
         all_inst.len()
     );
     fmt.indent(|fmt| {
-        for inst in all_inst {
+        for inst in all_inst.values() {
             let (ctrl_typevar, ctrl_typeset) = if let Some(poly) = &inst.polymorphic_info {
                 let index = type_sets.add(&*poly.ctrl_typevar.get_raw_typeset());
                 (Some(&poly.ctrl_typevar), index)
@@ -1035,7 +1025,7 @@ fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Fo
 }
 
 /// Generate a Builder trait with methods for all instructions.
-fn gen_builder(instructions: &Vec<&Instruction>, formats: &FormatRegistry, fmt: &mut Formatter) {
+fn gen_builder(instructions: &AllInstructions, formats: &FormatRegistry, fmt: &mut Formatter) {
     fmt.doc_comment(
         r#"
         Convenience methods for building instructions.
@@ -1055,7 +1045,7 @@ fn gen_builder(instructions: &Vec<&Instruction>, formats: &FormatRegistry, fmt:
     );
     fmt.line("pub trait InstBuilder<'f>: InstBuilderBase<'f> {");
     fmt.indent(|fmt| {
-        for inst in instructions {
+        for inst in instructions.values() {
             gen_inst_builder(inst, formats.get(inst.format), fmt);
         }
         for format in formats.iter() {
@@ -1066,12 +1056,14 @@ fn gen_builder(instructions: &Vec<&Instruction>, formats: &FormatRegistry, fmt:
 }
 
 pub fn generate(
-    all_inst_groups: Vec<&InstructionGroup>,
-    format_registry: &FormatRegistry,
+    shared_defs: &SharedDefinitions,
     opcode_filename: &str,
     inst_builder_filename: &str,
     out_dir: &str,
 ) -> Result<(), error::Error> {
+    let format_registry = &shared_defs.format_registry;
+    let all_inst = &shared_defs.all_instructions;
+
     // Opcodes.
     let mut fmt = Formatter::new();
     gen_formats(format_registry, &mut fmt);
@@ -1079,13 +1071,13 @@ pub fn generate(
     fmt.empty_line();
     gen_instruction_data_impl(format_registry, &mut fmt);
     fmt.empty_line();
-    let all_inst = gen_opcodes(format_registry, &all_inst_groups, &mut fmt);
-    gen_type_constraints(&all_inst, &mut fmt);
+    gen_opcodes(all_inst, format_registry, &mut fmt);
+    gen_type_constraints(all_inst, &mut fmt);
     fmt.update_file(opcode_filename, out_dir)?;
 
     // Instruction builder.
     let mut fmt = Formatter::new();
-    gen_builder(&all_inst, format_registry, &mut fmt);
+    gen_builder(all_inst, format_registry, &mut fmt);
     fmt.update_file(inst_builder_filename, out_dir)?;
 
     Ok(())
diff --git a/cranelift/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
index 0238ef1ec6..cdd07d78bd 100644
--- a/cranelift/codegen/meta/src/gen_settings.rs
+++ b/cranelift/codegen/meta/src/gen_settings.rs
@@ -308,7 +308,9 @@ fn gen_descriptors(group: &SettingGroup, fmt: &mut Formatter) {
     hash_entries.extend(group.settings.iter().map(|x| SettingOrPreset::Setting(x)));
     hash_entries.extend(group.presets.iter().map(|x| SettingOrPreset::Preset(x)));
 
-    let hash_table = generate_table(&hash_entries, |entry| simple_hash(entry.name()));
+    let hash_table = generate_table(hash_entries.iter(), hash_entries.len(), |entry| {
+        simple_hash(entry.name())
+    });
     fmtln!(fmt, "static HASH_TABLE: [u16; {}] = [", hash_table.len());
     fmt.indent(|fmt| {
         for h in &hash_table {
diff --git a/cranelift/codegen/meta/src/isa/arm32/mod.rs b/cranelift/codegen/meta/src/isa/arm32/mod.rs
index ee238866f7..c1b6e05fda 100644
--- a/cranelift/codegen/meta/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm32/mod.rs
@@ -55,6 +55,7 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let inst_group = InstructionGroupBuilder::new(
         "arm32",
         "arm32 specific instruction set",
+        &mut shared_defs.all_instructions,
         &shared_defs.format_registry,
     )
     .build();
diff --git a/cranelift/codegen/meta/src/isa/arm64/mod.rs b/cranelift/codegen/meta/src/isa/arm64/mod.rs
index 5a11e69848..c1276f5924 100644
--- a/cranelift/codegen/meta/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm64/mod.rs
@@ -51,6 +51,7 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let inst_group = InstructionGroupBuilder::new(
         "arm64",
         "arm64 specific instruction set",
+        &mut shared_defs.all_instructions,
         &shared_defs.format_registry,
     )
     .build();
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index 776224d74f..6c00b98f52 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -89,6 +89,7 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let inst_group = InstructionGroupBuilder::new(
         "riscv",
         "riscv specific instruction set",
+        &mut shared_defs.all_instructions,
         &shared_defs.format_registry,
     )
     .build();
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 1fcf37a68f..2f365165b8 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -2,16 +2,23 @@
 
 use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::instructions::{
-    InstructionBuilder as Inst, InstructionGroup, InstructionGroupBuilder,
+    AllInstructions, InstructionBuilder as Inst, InstructionGroup, InstructionGroupBuilder,
 };
 use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
 use crate::cdsl::types::ValueType;
 use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
 use crate::shared::types;
 
-pub fn define(format_registry: &FormatRegistry) -> InstructionGroup {
-    let mut ig =
-        InstructionGroupBuilder::new("x86", "x86 specific instruction set", format_registry);
+pub fn define(
+    mut all_instructions: &mut AllInstructions,
+    format_registry: &FormatRegistry,
+) -> InstructionGroup {
+    let mut ig = InstructionGroupBuilder::new(
+        "x86",
+        "x86 specific instruction set",
+        &mut all_instructions,
+        format_registry,
+    );
 
     let iflags: &TypeVar = &ValueType::Special(types::Flag::IFlags.into()).into();
 
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index dba5737062..993a4d9a3b 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -15,7 +15,10 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let settings = settings::define(&shared_defs.settings);
     let regs = registers::define();
 
-    let inst_group = instructions::define(&shared_defs.format_registry);
+    let inst_group = instructions::define(
+        &mut shared_defs.all_instructions,
+        &shared_defs.format_registry,
+    );
     legalize::define(shared_defs, &inst_group);
 
     // CPU modes for 32-bit and 64-bit operations.
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index 9ac2a54306..af8e51bb9d 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -35,16 +35,7 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
     // Per ISA definitions.
     let isas = isa::define(isas, &mut shared_defs);
 
-    let mut all_inst_groups = vec![&shared_defs.instructions];
-    all_inst_groups.extend(isas.iter().map(|isa| &isa.instructions));
-
-    gen_inst::generate(
-        all_inst_groups,
-        &shared_defs.format_registry,
-        "opcodes.rs",
-        "inst_builder.rs",
-        &out_dir,
-    )?;
+    gen_inst::generate(&shared_defs, "opcodes.rs", "inst_builder.rs", &out_dir)?;
 
     gen_legalizer::generate(
         &isas,
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index d9a45f0a3b..b89fa019b9 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -2,7 +2,7 @@
 
 use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::instructions::{
-    InstructionBuilder as Inst, InstructionGroup, InstructionGroupBuilder,
+    AllInstructions, InstructionBuilder as Inst, InstructionGroup, InstructionGroupBuilder,
 };
 use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
 use crate::cdsl::type_inference::Constraint::WiderOrEq;
@@ -11,12 +11,17 @@ use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
 use crate::shared::{types, OperandKinds};
 
 pub fn define(
+    all_instructions: &mut AllInstructions,
     format_registry: &FormatRegistry,
     immediates: &OperandKinds,
     entities: &OperandKinds,
 ) -> InstructionGroup {
-    let mut ig =
-        InstructionGroupBuilder::new("base", "Shared base instruction set", format_registry);
+    let mut ig = InstructionGroupBuilder::new(
+        "base",
+        "Shared base instruction set",
+        all_instructions,
+        format_registry,
+    );
 
     // Operand kind shorthands.
     let intcc = immediates.by_name("intcc");
diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
index f16fb7bb8b..817f67c340 100644
--- a/cranelift/codegen/meta/src/shared/mod.rs
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -9,13 +9,14 @@ pub mod settings;
 pub mod types;
 
 use crate::cdsl::formats::FormatRegistry;
-use crate::cdsl::instructions::InstructionGroup;
+use crate::cdsl::instructions::{AllInstructions, InstructionGroup};
 use crate::cdsl::operands::OperandKind;
 use crate::cdsl::settings::SettingGroup;
 use crate::cdsl::xform::TransformGroups;
 
 pub struct Definitions {
     pub settings: SettingGroup,
+    pub all_instructions: AllInstructions,
     pub instructions: InstructionGroup,
     pub operand_kinds: OperandKinds,
     pub format_registry: FormatRegistry,
@@ -50,14 +51,22 @@ impl OperandKinds {
 }
 
 pub fn define() -> Definitions {
+    let mut all_instructions = AllInstructions::new();
+
     let immediates = OperandKinds(immediates::define());
     let entities = OperandKinds(entities::define());
     let format_registry = formats::define(&immediates, &entities);
-    let instructions = instructions::define(&format_registry, &immediates, &entities);
+    let instructions = instructions::define(
+        &mut all_instructions,
+        &format_registry,
+        &immediates,
+        &entities,
+    );
     let transform_groups = legalize::define(&instructions, &immediates);
 
     Definitions {
         settings: settings::define(),
+        all_instructions,
         instructions,
         operand_kinds: immediates,
         format_registry,

From 18fb87cd9c97d15eec26d42f646094f603e24975 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 29 May 2019 17:20:34 +0200
Subject: [PATCH 2468/3084] [meta] Refactor instruction predicates to
 distinguish format and type checks;

Also add the instruction format name in format predicates, since they're
going to be used when generating encodings.
---
 cranelift/codegen/meta/src/cdsl/ast.rs        |  1 +
 .../codegen/meta/src/cdsl/instructions.rs     | 87 +++++++++++++++----
 2 files changed, 69 insertions(+), 19 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index f4ebea076e..6604a5e870 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -449,6 +449,7 @@ impl Apply {
                 continue;
             }
             pred = pred.and(InstructionPredicate::new_is_field_equal(
+                iform.name,
                 &format_field,
                 arg.to_rust_code(var_pool),
             ));
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 4bcd6ea402..e1ab128d2c 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -551,12 +551,39 @@ fn verify_ctrl_typevar(
     Ok(other_typevars)
 }
 
-/// A basic node in an instruction predicate: either an atom, or an AND of two conditions.
-pub enum InstructionPredicateNode {
-    /// Is the field member (first member) equal to the actual argument (which name is the second
-    /// field)?
-    IsFieldEqual(String, String),
+pub enum FormatPredicateKind {
+    /// Is the field member equal to the expected value (stored here)?
+    IsEqual(String),
+}
 
+pub struct FormatPredicateNode {
+    _format_name: &'static str,
+    field_name: &'static str,
+    kind: FormatPredicateKind,
+}
+
+impl FormatPredicateNode {
+    fn new(
+        _format_name: &'static str,
+        field_name: &'static str,
+        kind: FormatPredicateKind,
+    ) -> Self {
+        Self {
+            _format_name,
+            field_name,
+            kind,
+        }
+    }
+    fn rust_predicate(&self) -> String {
+        match &self.kind {
+            FormatPredicateKind::IsEqual(arg) => {
+                format!("crate::predicates::is_equal({}, {})", self.field_name, arg)
+            }
+        }
+    }
+}
+
+pub enum TypePredicateNode {
     /// Is the value argument (at the index designated by the first member) the same type as the
     /// type name (second member)?
     TypeVarCheck(usize, String),
@@ -564,6 +591,27 @@ pub enum InstructionPredicateNode {
     /// Is the controlling type variable the same type as the one designated by the type name
     /// (only member)?
     CtrlTypeVarCheck(String),
+}
+
+impl TypePredicateNode {
+    fn rust_predicate(&self) -> String {
+        match self {
+            TypePredicateNode::TypeVarCheck(index, value_type_name) => format!(
+                "func.dfg.value_type(args[{}]) == {}",
+                index, value_type_name
+            ),
+            TypePredicateNode::CtrlTypeVarCheck(value_type_name) => {
+                format!("func.dfg.ctrl_typevar(inst) == {}", value_type_name)
+            }
+        }
+    }
+}
+
+/// A basic node in an instruction predicate: either an atom, or an AND of two conditions.
+pub enum InstructionPredicateNode {
+    FormatPredicate(FormatPredicateNode),
+
+    TypePredicate(TypePredicateNode),
 
     /// A combination of two other predicates.
     And(Vec),
@@ -572,17 +620,8 @@ pub enum InstructionPredicateNode {
 impl InstructionPredicateNode {
     fn rust_predicate(&self) -> String {
         match self {
-            InstructionPredicateNode::IsFieldEqual(field_name, arg) => {
-                let new_args = vec![field_name.clone(), arg.clone()];
-                format!("crate::predicates::is_equal({})", new_args.join(", "))
-            }
-            InstructionPredicateNode::TypeVarCheck(index, value_type_name) => format!(
-                "func.dfg.value_type(args[{}]) == {}",
-                index, value_type_name
-            ),
-            InstructionPredicateNode::CtrlTypeVarCheck(value_type_name) => {
-                format!("func.dfg.ctrl_typevar(inst) == {}", value_type_name)
-            }
+            InstructionPredicateNode::FormatPredicate(node) => node.rust_predicate(),
+            InstructionPredicateNode::TypePredicate(node) => node.rust_predicate(),
             InstructionPredicateNode::And(nodes) => nodes
                 .iter()
                 .map(|x| x.rust_predicate())
@@ -614,18 +653,28 @@ impl InstructionPredicate {
             .next()
             .unwrap()
             .0;
-        InstructionPredicateNode::TypeVarCheck(index, value_type.rust_name())
+        InstructionPredicateNode::TypePredicate(TypePredicateNode::TypeVarCheck(
+            index,
+            value_type.rust_name(),
+        ))
     }
 
     pub fn new_is_field_equal(
+        format_name: &'static str,
         format_field: &FormatField,
         imm_value: String,
     ) -> InstructionPredicateNode {
-        InstructionPredicateNode::IsFieldEqual(format_field.member.into(), imm_value)
+        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
+            format_name,
+            format_field.member,
+            FormatPredicateKind::IsEqual(imm_value),
+        ))
     }
 
     pub fn new_ctrl_typevar_check(value_type: &ValueType) -> InstructionPredicateNode {
-        InstructionPredicateNode::CtrlTypeVarCheck(value_type.rust_name())
+        InstructionPredicateNode::TypePredicate(TypePredicateNode::CtrlTypeVarCheck(
+            value_type.rust_name(),
+        ))
     }
 
     pub fn and(mut self, new_node: InstructionPredicateNode) -> Self {

From 9dcc185264e7b15f042925c51fbf4841216f2d31 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 21 Jun 2019 12:08:43 +0200
Subject: [PATCH 2469/3084] [meta] Add a MapWithDefault trait;

This traits augments HashMap so they have a `get_or_default` method that
can be used to avoid boilerplate around usage of `entry.or_default`.
---
 cranelift/codegen/meta/src/default_map.rs | 19 +++++++++++++++++++
 cranelift/codegen/meta/src/lib.rs         |  1 +
 2 files changed, 20 insertions(+)
 create mode 100644 cranelift/codegen/meta/src/default_map.rs

diff --git a/cranelift/codegen/meta/src/default_map.rs b/cranelift/codegen/meta/src/default_map.rs
new file mode 100644
index 0000000000..da6d471439
--- /dev/null
+++ b/cranelift/codegen/meta/src/default_map.rs
@@ -0,0 +1,19 @@
+use std::collections::HashMap;
+use std::hash::Hash;
+
+pub trait MapWithDefault {
+    fn get_or_default(&mut self, k: K) -> &mut V;
+}
+
+impl MapWithDefault for HashMap {
+    fn get_or_default(&mut self, k: K) -> &mut V {
+        self.entry(k).or_insert_with(|| V::default())
+    }
+}
+
+#[test]
+fn test_default() {
+    let mut hash_map = HashMap::new();
+    hash_map.insert(42, "hello");
+    assert_eq!(*hash_map.get_or_default(43), "");
+}
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index af8e51bb9d..901f640a51 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -12,6 +12,7 @@ mod gen_settings;
 mod gen_types;
 
 mod constant_hash;
+mod default_map;
 mod shared;
 mod unique_table;
 

From ec5678ab7a41694457b7261e3859cbd924d5dd68 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 21 Jun 2019 12:15:09 +0200
Subject: [PATCH 2470/3084] [meta] Add the ability to bind any type to an
 instruction;

---
 cranelift/codegen/meta/src/cdsl/ast.rs        |  8 ++-
 .../codegen/meta/src/cdsl/instructions.rs     | 49 ++++++++++++++++---
 2 files changed, 50 insertions(+), 7 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 6604a5e870..68d99a0218 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -376,11 +376,17 @@ pub struct Apply {
 
 impl Apply {
     pub fn new(target: InstSpec, args: Vec) -> Self {
-        let (inst, value_types) = match target.into() {
+        let (inst, value_types) = match target {
             InstSpec::Inst(inst) => (inst, Vec::new()),
             InstSpec::Bound(bound_inst) => (bound_inst.inst, bound_inst.value_types),
         };
 
+        // Apply should only operate on concrete value types, not "any".
+        let value_types = value_types
+            .into_iter()
+            .map(|vt| vt.expect("shouldn't be Any"))
+            .collect();
+
         // Basic check on number of arguments.
         assert!(
             inst.operands_in.len() == args.len(),
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index e1ab128d2c..13bac67c8f 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -183,7 +183,10 @@ impl Instruction {
     }
 
     pub fn bind(&self, lane_type: impl Into) -> BoundInstruction {
-        bind(self.clone(), lane_type.into(), Vec::new())
+        bind(self.clone(), Some(lane_type.into()), Vec::new())
+    }
+    pub fn bind_any(&self) -> BoundInstruction {
+        bind(self.clone(), None, Vec::new())
     }
 }
 
@@ -380,15 +383,34 @@ impl InstructionBuilder {
     }
 }
 
+/// A thin wrapper like Option, but with more precise semantics.
+#[derive(Clone)]
+pub enum ValueTypeOrAny {
+    ValueType(ValueType),
+    Any,
+}
+
+impl ValueTypeOrAny {
+    pub fn expect(self, msg: &str) -> ValueType {
+        match self {
+            ValueTypeOrAny::ValueType(vt) => vt,
+            ValueTypeOrAny::Any => panic!(format!("Unexpected Any: {}", msg)),
+        }
+    }
+}
+
 #[derive(Clone)]
 pub struct BoundInstruction {
     pub inst: Instruction,
-    pub value_types: Vec,
+    pub value_types: Vec,
 }
 
 impl BoundInstruction {
     pub fn bind(self, lane_type: impl Into) -> BoundInstruction {
-        bind(self.inst, lane_type.into(), self.value_types)
+        bind(self.inst, Some(lane_type.into()), self.value_types)
+    }
+    pub fn bind_any(self) -> BoundInstruction {
+        bind(self.inst, None, self.value_types)
     }
 }
 
@@ -712,6 +734,12 @@ impl InstSpec {
             InstSpec::Bound(bound_inst) => &bound_inst.inst,
         }
     }
+    pub fn bind(&self, lane_type: impl Into) -> BoundInstruction {
+        match self {
+            InstSpec::Inst(inst) => inst.bind(lane_type),
+            InstSpec::Bound(inst) => inst.clone().bind(lane_type),
+        }
+    }
 }
 
 impl Into for &Instruction {
@@ -729,10 +757,18 @@ impl Into for BoundInstruction {
 /// Helper bind reused by {Bound,}Instruction::bind.
 fn bind(
     inst: Instruction,
-    lane_type: LaneType,
-    mut value_types: Vec,
+    lane_type: Option,
+    mut value_types: Vec,
 ) -> BoundInstruction {
-    value_types.push(ValueType::from(lane_type));
+    match lane_type {
+        Some(lane_type) => {
+            value_types.push(ValueTypeOrAny::ValueType(lane_type.into()));
+        }
+        None => {
+            value_types.push(ValueTypeOrAny::Any);
+        }
+    }
+
     match &inst.polymorphic_info {
         Some(poly) => {
             assert!(
@@ -747,5 +783,6 @@ fn bind(
             ));
         }
     }
+
     BoundInstruction { inst, value_types }
 }

From e34a4759cd3df5fd7472512eb5879de3f989785e Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 21 Jun 2019 12:17:05 +0200
Subject: [PATCH 2471/3084] [meta] Fix typo in x86 setting name use_lzcnt;

---
 cranelift/codegen/meta/src/isa/x86/settings.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/settings.rs b/cranelift/codegen/meta/src/isa/x86/settings.rs
index 52486d7de3..15f17621ce 100644
--- a/cranelift/codegen/meta/src/isa/x86/settings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/settings.rs
@@ -34,7 +34,7 @@ pub fn define(shared: &SettingGroup) -> SettingGroup {
     settings.add_predicate("use_sse42", predicate!(has_sse41 && has_sse42));
     settings.add_predicate("use_popcnt", predicate!(has_popcnt && has_sse42));
     settings.add_predicate("use_bmi1", predicate!(has_bmi1));
-    settings.add_predicate("use_lznct", predicate!(has_lzcnt));
+    settings.add_predicate("use_lzcnt", predicate!(has_lzcnt));
 
     // Some shared boolean values are used in x86 instruction predicates, so we need to group them
     // in the same TargetIsa, for compabitibity with code generated by meta-python.

From a3e459269e9c55d3f3837e149a610d1f0901281e Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 3 Jul 2019 10:40:58 -0700
Subject: [PATCH 2472/3084] Bump version to 0.33.0

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    |  8 ++++----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index b0a1bbbc04..6076d978cd 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.32.0"
+version = "0.33.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.32.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.32.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.32.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.32.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.32.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.32.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.32.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.32.0" }
-cranelift-module = { path = "cranelift-module", version = "0.32.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.32.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.32.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.32.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.32.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.33.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.33.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.33.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.33.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.33.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.33.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.33.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.33.0" }
+cranelift-module = { path = "cranelift-module", version = "0.33.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.33.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.33.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.33.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.33.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 3672a97827..9cc6c7a18a 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.32.0"
+version = "0.33.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.32.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.33.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 1fa3e22219..5fcb8f96db 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.32.0"
+version = "0.33.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.32.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.32.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.33.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.33.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -26,7 +26,7 @@ log = { version = "0.4.6", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.32.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.33.0", default-features = false }
 
 [features]
 default = ["std", "x86", "arm32", "arm64", "riscv"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 336dfd9921..56f5190830 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.32.0"
+version = "0.33.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.32.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.33.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index d7119634f4..bb40aaa17b 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.32.0"
+version = "0.33.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index cc8311ebe5..fd205a61f7 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.32.0"
+version = "0.33.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.32.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.33.0" }
 faerie = "0.10.0"
 goblin = "0.0.22"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index fa0a282c5d..5e28a630e9 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.32.0"
+version = "0.33.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../cranelift-reader", version = "0.32.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.32.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../cranelift-reader", version = "0.33.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.33.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 2c33167aa3..ec7424c84a 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.32.0"
+version = "0.33.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 4c9121503b..e3d4081ba7 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.32.0"
+version = "0.33.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.32.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.33.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 4ceaec7d6b..4a0904ebc1 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.32.0"
+version = "0.33.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index d8fd89974a..faec5baa05 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.32.0"
+version = "0.33.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.32.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.33.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index e9bf6ba1c8..aed801e351 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.32.0"
+version="0.33.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index d0ec95bb15..1c911f9612 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.32.0"
+version = "0.33.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0" }
 target-lexicon = "0.4.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 6208b1d599..c34d09d92a 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.32.0"
+version = "0.33.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.32.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.33.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 6d6e8c8340..6811538631 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.32.0"
+version = "0.33.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.32.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.32.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.33.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.33.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.4.0" }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.32.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.32.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.32.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.33.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.33.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.33.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index c725fc8cc6..c4ddf0db2f 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.32.0"
+version = "0.33.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.32.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.33.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index a61b52be1d..3a9d9ebc98 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.32.0"
+version = "0.33.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.32.1", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.32.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.32.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.32.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.33.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.33.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 1e42aac41a08d9fc67db8f18af4411dff91830eb Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 24 Jun 2019 16:56:00 +0200
Subject: [PATCH 2473/3084] [meta] Add new instruction predicates and the
 InstructionPredicateMap;

The latter helps deduplicating predicates during encodings and recipes
construction.
---
 cranelift/codegen/meta/src/cdsl/ast.rs        |   6 +-
 cranelift/codegen/meta/src/cdsl/formats.rs    |  22 ++
 .../codegen/meta/src/cdsl/instructions.rs     | 341 +++++++++++++++++-
 3 files changed, 349 insertions(+), 20 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 68d99a0218..5e1c192f0f 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -440,7 +440,7 @@ impl Apply {
         format!("{}({})", self.inst.name, args)
     }
 
-    fn inst_predicate(
+    pub fn inst_predicate(
         &self,
         format_registry: &FormatRegistry,
         var_pool: &VarPool,
@@ -454,8 +454,8 @@ impl Apply {
                 // Ignore free variables for now.
                 continue;
             }
-            pred = pred.and(InstructionPredicate::new_is_field_equal(
-                iform.name,
+            pred = pred.and(InstructionPredicate::new_is_field_equal_ast(
+                iform,
                 &format_field,
                 arg.to_rust_code(var_pool),
             ));
diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
index cefcfa1626..b716a8a976 100644
--- a/cranelift/codegen/meta/src/cdsl/formats.rs
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -67,6 +67,20 @@ impl fmt::Display for InstructionFormat {
     }
 }
 
+impl InstructionFormat {
+    pub fn imm_by_name(&self, name: &'static str) -> &FormatField {
+        self.imm_fields
+            .iter()
+            .find(|&field| field.member == name)
+            .unwrap_or_else(|| {
+                panic!(
+                    "unexpected immediate field named {} in instruction format {}",
+                    name, self.name
+                )
+            })
+    }
+}
+
 pub struct InstructionFormatBuilder {
     name: &'static str,
     num_value_operands: usize,
@@ -200,6 +214,14 @@ impl FormatRegistry {
             .expect("unknown InstructionFormat; please define it in shared/formats.rs first")
     }
 
+    pub fn by_name(&self, name: &str) -> InstructionFormatIndex {
+        self.map
+            .iter()
+            .find(|(_key, value)| value.name == name)
+            .unwrap_or_else(|| panic!("format with name '{}' doesn't exist", name))
+            .0
+    }
+
     pub fn get(&self, index: InstructionFormatIndex) -> &InstructionFormat {
         self.map.get(index).unwrap()
     }
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 13bac67c8f..9b6a354ca0 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -1,10 +1,11 @@
+use cranelift_entity::{entity_impl, PrimaryMap};
+
+use std::collections::HashMap;
 use std::fmt;
 use std::ops;
 use std::rc::Rc;
 use std::slice;
 
-use cranelift_entity::{entity_impl, PrimaryMap};
-
 use crate::cdsl::camel_case;
 use crate::cdsl::formats::{
     FormatField, FormatRegistry, InstructionFormat, InstructionFormatIndex,
@@ -573,38 +574,121 @@ fn verify_ctrl_typevar(
     Ok(other_typevars)
 }
 
+#[derive(Clone, Hash, PartialEq, Eq)]
 pub enum FormatPredicateKind {
     /// Is the field member equal to the expected value (stored here)?
     IsEqual(String),
+
+    /// Is the immediate instruction format field representable as an n-bit two's complement
+    /// integer? (with width: first member, scale: second member).
+    /// The predicate is true if the field is in the range: `-2^(width-1) -- 2^(width-1)-1` and a
+    /// multiple of `2^scale`.
+    IsSignedInt(usize, usize),
+
+    /// Is the immediate instruction format field representable as an n-bit unsigned integer? (with
+    /// width: first member, scale: second member).
+    /// The predicate is true if the field is in the range: `0 -- 2^width - 1` and a multiple of
+    /// `2^scale`.
+    IsUnsignedInt(usize, usize),
+
+    /// Is the immediate format field member equal to zero? (float32 version)
+    IsZero32BitFloat,
+
+    /// Is the immediate format field member equal to zero? (float64 version)
+    IsZero64BitFloat,
+
+    /// Has the value list (in member_name) the size specified in parameter?
+    LengthEquals(usize),
+
+    /// Is the referenced function colocated?
+    IsColocatedFunc,
+
+    /// Is the referenced data object colocated?
+    IsColocatedData,
 }
 
+#[derive(Clone, Hash, PartialEq, Eq)]
 pub struct FormatPredicateNode {
-    _format_name: &'static str,
-    field_name: &'static str,
+    format_name: &'static str,
+    member_name: &'static str,
     kind: FormatPredicateKind,
 }
 
 impl FormatPredicateNode {
     fn new(
-        _format_name: &'static str,
+        format: &InstructionFormat,
         field_name: &'static str,
         kind: FormatPredicateKind,
     ) -> Self {
+        let member_name = format.imm_by_name(field_name).member;
         Self {
-            _format_name,
-            field_name,
+            format_name: format.name,
+            member_name,
             kind,
         }
     }
+
+    fn new_raw(
+        format: &InstructionFormat,
+        member_name: &'static str,
+        kind: FormatPredicateKind,
+    ) -> Self {
+        Self {
+            format_name: format.name,
+            member_name,
+            kind,
+        }
+    }
+
+    fn destructuring_member_name(&self) -> &'static str {
+        match &self.kind {
+            FormatPredicateKind::LengthEquals(_) => {
+                // Length operates on the argument value list.
+                assert!(self.member_name == "args");
+                "ref args"
+            }
+            _ => self.member_name,
+        }
+    }
+
     fn rust_predicate(&self) -> String {
         match &self.kind {
             FormatPredicateKind::IsEqual(arg) => {
-                format!("crate::predicates::is_equal({}, {})", self.field_name, arg)
+                format!("crate::predicates::is_equal({}, {})", self.member_name, arg)
             }
+            FormatPredicateKind::IsSignedInt(width, scale) => format!(
+                "crate::predicates::is_signed_int({}, {}, {})",
+                self.member_name, width, scale
+            ),
+            FormatPredicateKind::IsUnsignedInt(width, scale) => format!(
+                "crate::predicates::is_unsigned_int({}, {}, {})",
+                self.member_name, width, scale
+            ),
+            FormatPredicateKind::IsZero32BitFloat => format!(
+                "crate::predicates::is_zero_32_bit_float({})",
+                self.member_name
+            ),
+            FormatPredicateKind::IsZero64BitFloat => format!(
+                "crate::predicates::is_zero_64_bit_float({})",
+                self.member_name
+            ),
+            FormatPredicateKind::LengthEquals(num) => format!(
+                "crate::predicates::has_length_of({}, {}, func)",
+                self.member_name, num
+            ),
+            FormatPredicateKind::IsColocatedFunc => format!(
+                "crate::predicates::is_colocated_func({}, func)",
+                self.member_name,
+            ),
+            FormatPredicateKind::IsColocatedData => format!(
+                "crate::predicates::is_colocated_data({}, func)",
+                self.member_name
+            ),
         }
     }
 }
 
+#[derive(Clone, Hash, PartialEq, Eq)]
 pub enum TypePredicateNode {
     /// Is the value argument (at the index designated by the first member) the same type as the
     /// type name (second member)?
@@ -630,13 +714,17 @@ impl TypePredicateNode {
 }
 
 /// A basic node in an instruction predicate: either an atom, or an AND of two conditions.
+#[derive(Clone, Hash, PartialEq, Eq)]
 pub enum InstructionPredicateNode {
     FormatPredicate(FormatPredicateNode),
 
     TypePredicate(TypePredicateNode),
 
-    /// A combination of two other predicates.
+    /// An AND-combination of two or more other predicates.
     And(Vec),
+
+    /// An OR-combination of two or more other predicates.
+    Or(Vec),
 }
 
 impl InstructionPredicateNode {
@@ -648,20 +736,74 @@ impl InstructionPredicateNode {
                 .iter()
                 .map(|x| x.rust_predicate())
                 .collect::>()
-                .join(" &&\n"),
+                .join(" && "),
+            InstructionPredicateNode::Or(nodes) => nodes
+                .iter()
+                .map(|x| x.rust_predicate())
+                .collect::>()
+                .join(" || "),
         }
     }
+
+    pub fn format_destructuring_member_name(&self) -> &str {
+        match self {
+            InstructionPredicateNode::FormatPredicate(format_pred) => {
+                format_pred.destructuring_member_name()
+            }
+            _ => panic!("Only for leaf format predicates"),
+        }
+    }
+
+    pub fn format_name(&self) -> &str {
+        match self {
+            InstructionPredicateNode::FormatPredicate(format_pred) => format_pred.format_name,
+            _ => panic!("Only for leaf format predicates"),
+        }
+    }
+
+    pub fn is_type_predicate(&self) -> bool {
+        match self {
+            InstructionPredicateNode::FormatPredicate(_)
+            | InstructionPredicateNode::And(_)
+            | InstructionPredicateNode::Or(_) => false,
+            InstructionPredicateNode::TypePredicate(_) => true,
+        }
+    }
+
+    fn collect_leaves(&self) -> Vec<&InstructionPredicateNode> {
+        let mut ret = Vec::new();
+        match self {
+            InstructionPredicateNode::And(nodes) | InstructionPredicateNode::Or(nodes) => {
+                for node in nodes {
+                    ret.extend(node.collect_leaves());
+                }
+            }
+            _ => ret.push(&self),
+        }
+        ret
+    }
 }
 
+#[derive(Clone, Hash, PartialEq, Eq)]
 pub struct InstructionPredicate {
     node: Option,
 }
 
+impl Into for InstructionPredicateNode {
+    fn into(self) -> InstructionPredicate {
+        InstructionPredicate { node: Some(self) }
+    }
+}
+
 impl InstructionPredicate {
     pub fn new() -> Self {
         Self { node: None }
     }
 
+    pub fn unwrap(self) -> InstructionPredicateNode {
+        self.node.unwrap()
+    }
+
     pub fn new_typevar_check(
         inst: &Instruction,
         type_var: &TypeVar,
@@ -681,21 +823,114 @@ impl InstructionPredicate {
         ))
     }
 
+    pub fn new_ctrl_typevar_check(value_type: &ValueType) -> InstructionPredicateNode {
+        InstructionPredicateNode::TypePredicate(TypePredicateNode::CtrlTypeVarCheck(
+            value_type.rust_name(),
+        ))
+    }
+
     pub fn new_is_field_equal(
-        format_name: &'static str,
-        format_field: &FormatField,
+        format: &InstructionFormat,
+        field_name: &'static str,
         imm_value: String,
     ) -> InstructionPredicateNode {
         InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
-            format_name,
-            format_field.member,
+            format,
+            field_name,
             FormatPredicateKind::IsEqual(imm_value),
         ))
     }
 
-    pub fn new_ctrl_typevar_check(value_type: &ValueType) -> InstructionPredicateNode {
-        InstructionPredicateNode::TypePredicate(TypePredicateNode::CtrlTypeVarCheck(
-            value_type.rust_name(),
+    /// Used only for the AST module, which directly passes in the format field.
+    pub fn new_is_field_equal_ast(
+        format: &InstructionFormat,
+        field: &FormatField,
+        imm_value: String,
+    ) -> InstructionPredicateNode {
+        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new_raw(
+            format,
+            field.member,
+            FormatPredicateKind::IsEqual(imm_value),
+        ))
+    }
+
+    pub fn new_is_signed_int(
+        format: &InstructionFormat,
+        field_name: &'static str,
+        width: usize,
+        scale: usize,
+    ) -> InstructionPredicateNode {
+        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
+            format,
+            field_name,
+            FormatPredicateKind::IsSignedInt(width, scale),
+        ))
+    }
+
+    pub fn new_is_unsigned_int(
+        format: &InstructionFormat,
+        field_name: &'static str,
+        width: usize,
+        scale: usize,
+    ) -> InstructionPredicateNode {
+        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
+            format,
+            field_name,
+            FormatPredicateKind::IsUnsignedInt(width, scale),
+        ))
+    }
+
+    pub fn new_is_zero_32bit_float(
+        format: &InstructionFormat,
+        field_name: &'static str,
+    ) -> InstructionPredicateNode {
+        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
+            format,
+            field_name,
+            FormatPredicateKind::IsZero32BitFloat,
+        ))
+    }
+
+    pub fn new_is_zero_64bit_float(
+        format: &InstructionFormat,
+        field_name: &'static str,
+    ) -> InstructionPredicateNode {
+        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
+            format,
+            field_name,
+            FormatPredicateKind::IsZero64BitFloat,
+        ))
+    }
+
+    pub fn new_length_equals(format: &InstructionFormat, size: usize) -> InstructionPredicateNode {
+        assert!(
+            format.has_value_list,
+            "the format must be variadic in number of arguments"
+        );
+        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new_raw(
+            format,
+            "args",
+            FormatPredicateKind::LengthEquals(size),
+        ))
+    }
+
+    pub fn new_is_colocated_func(
+        format: &InstructionFormat,
+        field_name: &'static str,
+    ) -> InstructionPredicateNode {
+        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
+            format,
+            field_name,
+            FormatPredicateKind::IsColocatedFunc,
+        ))
+    }
+
+    pub fn new_is_colocated_data(format_registry: &FormatRegistry) -> InstructionPredicateNode {
+        let format = format_registry.get(format_registry.by_name("UnaryGlobalValue"));
+        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
+            format,
+            "global_value",
+            FormatPredicateKind::IsColocatedData,
         ))
     }
 
@@ -704,6 +939,9 @@ impl InstructionPredicate {
         let mut and_nodes = match node {
             Some(node) => match node {
                 InstructionPredicateNode::And(nodes) => nodes,
+                InstructionPredicateNode::Or(_) => {
+                    panic!("Can't mix and/or without implementing operator precedence!")
+                }
                 _ => vec![node],
             },
             _ => Vec::new(),
@@ -713,12 +951,81 @@ impl InstructionPredicate {
         self
     }
 
+    pub fn or(mut self, new_node: InstructionPredicateNode) -> Self {
+        let node = self.node;
+        let mut or_nodes = match node {
+            Some(node) => match node {
+                InstructionPredicateNode::Or(nodes) => nodes,
+                InstructionPredicateNode::And(_) => {
+                    panic!("Can't mix and/or without implementing operator precedence!")
+                }
+                _ => vec![node],
+            },
+            _ => Vec::new(),
+        };
+        or_nodes.push(new_node);
+        self.node = Some(InstructionPredicateNode::Or(or_nodes));
+        self
+    }
+
     pub fn rust_predicate(&self) -> String {
         match &self.node {
             Some(root) => root.rust_predicate(),
             None => "true".into(),
         }
     }
+
+    /// Returns true if the predicate only depends on type parameters (and not on an instruction
+    /// format).
+    pub fn is_type_predicate(&self) -> bool {
+        self.node.as_ref().unwrap().is_type_predicate()
+    }
+
+    /// Returns references to all the nodes that are leaves in the condition (i.e. by flattening
+    /// AND/OR).
+    pub fn collect_leaves(&self) -> Vec<&InstructionPredicateNode> {
+        self.node.as_ref().unwrap().collect_leaves()
+    }
+}
+
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct InstructionPredicateNumber(u32);
+entity_impl!(InstructionPredicateNumber);
+
+pub type InstructionPredicateMap = PrimaryMap;
+
+/// A registry of predicates to help deduplicating them, during Encodings construction. When the
+/// construction process is over, it needs to be extracted with `extract` and associated to the
+/// TargetIsa.
+pub struct InstructionPredicateRegistry {
+    /// Maps a predicate number to its actual predicate.
+    map: InstructionPredicateMap,
+
+    /// Inverse map: maps a predicate to its predicate number. This is used before inserting a
+    /// predicate, to check whether it already exists.
+    inverted_map: HashMap,
+}
+
+impl InstructionPredicateRegistry {
+    pub fn new() -> Self {
+        Self {
+            map: PrimaryMap::new(),
+            inverted_map: HashMap::new(),
+        }
+    }
+    pub fn insert(&mut self, predicate: InstructionPredicate) -> InstructionPredicateNumber {
+        match self.inverted_map.get(&predicate) {
+            Some(&found) => found,
+            None => {
+                let key = self.map.push(predicate.clone());
+                self.inverted_map.insert(predicate, key);
+                key
+            }
+        }
+    }
+    pub fn extract(self) -> InstructionPredicateMap {
+        self.map
+    }
 }
 
 /// An instruction specification, containing an instruction that has bound types or not.

From 4a6b88193e881e6e76c7e5201cec0e58e1fa57ac Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 24 Jun 2019 17:20:05 +0200
Subject: [PATCH 2474/3084] [meta] Map global TransformGroup to local
 TransformGroup indices;

---
 cranelift/codegen/meta/src/cdsl/cpu_modes.rs | 31 ------------
 cranelift/codegen/meta/src/cdsl/isa.rs       | 51 ++++++++++++++++----
 cranelift/codegen/meta/src/cdsl/settings.rs  | 12 ++++-
 cranelift/codegen/meta/src/gen_legalizer.rs  |  2 +-
 4 files changed, 53 insertions(+), 43 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/cpu_modes.rs b/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
index 5e0e5b4762..30a7028cf9 100644
--- a/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
+++ b/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
@@ -35,37 +35,6 @@ impl CpuMode {
             .is_none());
     }
 
-    /// Returns a deterministically ordered, deduplicated list of TransformGroupIndex for the
-    /// transitive set of TransformGroup this TargetIsa uses.
-    pub fn transitive_transform_groups(
-        &self,
-        all_groups: &TransformGroups,
-    ) -> Vec {
-        let mut roots = Vec::new();
-        if let Some(i) = &self.default_legalize {
-            roots.push(*i);
-        }
-        if let Some(i) = &self.monomorphic_legalize {
-            roots.push(*i);
-        }
-        roots.extend(self.typed_legalize.values().cloned());
-
-        let mut set = HashSet::new();
-        for root in roots {
-            set.insert(root);
-            let mut base = root;
-            // Follow the chain of chain_with.
-            while let Some(chain_with) = &all_groups.get(base).chain_with {
-                set.insert(*chain_with);
-                base = *chain_with;
-            }
-        }
-
-        let mut ret = Vec::from_iter(set);
-        ret.sort();
-        ret
-    }
-
     /// Returns a deterministically ordered, deduplicated list of TransformGroupIndex for the directly
     /// reachable set of TransformGroup this TargetIsa uses.
     pub fn direct_transform_groups(&self) -> Vec {
diff --git a/cranelift/codegen/meta/src/cdsl/isa.rs b/cranelift/codegen/meta/src/cdsl/isa.rs
index dcac2605c1..07467229c7 100644
--- a/cranelift/codegen/meta/src/cdsl/isa.rs
+++ b/cranelift/codegen/meta/src/cdsl/isa.rs
@@ -13,6 +13,11 @@ pub struct TargetIsa {
     pub settings: SettingGroup,
     pub regs: IsaRegs,
     pub cpu_modes: Vec,
+
+    /// TransformGroupIndex are global to all the ISAs, while we want to have indices into the
+    /// local array of transform groups that are directly used. We use this map to get this
+    /// information.
+    pub local_transform_groups: Vec,
 }
 
 impl TargetIsa {
@@ -23,12 +28,29 @@ impl TargetIsa {
         regs: IsaRegs,
         cpu_modes: Vec,
     ) -> Self {
+        // Compute the local TransformGroup index.
+        let mut local_transform_groups = Vec::new();
+        for cpu_mode in &cpu_modes {
+            let transform_groups = cpu_mode.direct_transform_groups();
+            for group_index in transform_groups {
+                // find() is fine here: the number of transform group is < 5 as of June 2019.
+                if local_transform_groups
+                    .iter()
+                    .find(|&val| group_index == *val)
+                    .is_none()
+                {
+                    local_transform_groups.push(group_index);
+                }
+            }
+        }
+
         Self {
             name,
             instructions,
             settings,
             regs,
             cpu_modes,
+            local_transform_groups,
         }
     }
 
@@ -39,9 +61,17 @@ impl TargetIsa {
         all_groups: &TransformGroups,
     ) -> Vec {
         let mut set = HashSet::new();
-        for cpu_mode in &self.cpu_modes {
-            set.extend(cpu_mode.transitive_transform_groups(all_groups));
+
+        for &root in self.local_transform_groups.iter() {
+            set.insert(root);
+            let mut base = root;
+            // Follow the chain of chain_with.
+            while let Some(chain_with) = &all_groups.get(base).chain_with {
+                set.insert(*chain_with);
+                base = *chain_with;
+            }
         }
+
         let mut vec = Vec::from_iter(set);
         vec.sort();
         vec
@@ -49,13 +79,14 @@ impl TargetIsa {
 
     /// Returns a deterministically ordered, deduplicated list of TransformGroupIndex for the directly
     /// reachable set of TransformGroup this TargetIsa uses.
-    pub fn direct_transform_groups(&self) -> Vec {
-        let mut set = HashSet::new();
-        for cpu_mode in &self.cpu_modes {
-            set.extend(cpu_mode.direct_transform_groups());
-        }
-        let mut vec = Vec::from_iter(set);
-        vec.sort();
-        vec
+    pub fn direct_transform_groups(&self) -> &Vec {
+        &self.local_transform_groups
+    }
+
+    pub fn translate_group_index(&self, group_index: TransformGroupIndex) -> usize {
+        self.local_transform_groups
+            .iter()
+            .position(|&val| val == group_index)
+            .expect("TransformGroup unused by this TargetIsa!")
     }
 }
diff --git a/cranelift/codegen/meta/src/cdsl/settings.rs b/cranelift/codegen/meta/src/cdsl/settings.rs
index 61677a71f3..5a45d9fb9b 100644
--- a/cranelift/codegen/meta/src/cdsl/settings.rs
+++ b/cranelift/codegen/meta/src/cdsl/settings.rs
@@ -148,6 +148,14 @@ impl SettingGroup {
         }
         panic!("Should have found bool setting by name.");
     }
+
+    pub fn predicate_by_name(&self, name: &'static str) -> SettingPredicateNumber {
+        self.predicates
+            .iter()
+            .find(|pred| pred.name == name)
+            .unwrap_or_else(|| panic!("unknown predicate {}", name))
+            .number
+    }
 }
 
 /// This is the basic information needed to track the specific parts of a setting when building
@@ -209,10 +217,12 @@ struct ProtoPredicate {
     node: PredicateNode,
 }
 
+pub type SettingPredicateNumber = u8;
+
 pub struct Predicate {
     pub name: &'static str,
     node: PredicateNode,
-    pub number: u8,
+    pub number: SettingPredicateNumber,
 }
 
 impl Predicate {
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index 149e29f926..545928db1e 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -527,7 +527,7 @@ fn gen_isa(
         direct_groups.len()
     );
     fmt.indent(|fmt| {
-        for group_index in direct_groups {
+        for &group_index in direct_groups {
             fmtln!(fmt, "{},", transform_groups.get(group_index).rust_name());
         }
     });

From 21aaf0c89f7609d60feaaaa174c9af68ebd01d7d Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 24 Jun 2019 16:48:31 +0200
Subject: [PATCH 2475/3084] [meta] Add cdsl facilities for encodings and
 recipes;

Co-authored-by: Benjamin Bouvier 
Co-authored-by: bjorn3 
---
 cranelift/codegen/meta/src/cdsl/cpu_modes.rs  |  45 ++-
 cranelift/codegen/meta/src/cdsl/encodings.rs  | 160 ++++++++++
 cranelift/codegen/meta/src/cdsl/isa.rs        |  15 +-
 cranelift/codegen/meta/src/cdsl/mod.rs        |   2 +
 cranelift/codegen/meta/src/cdsl/recipes.rs    | 297 ++++++++++++++++++
 cranelift/codegen/meta/src/cdsl/regs.rs       |  30 +-
 cranelift/codegen/meta/src/cdsl/types.rs      |   8 +-
 cranelift/codegen/meta/src/isa/arm32/mod.rs   |  19 +-
 cranelift/codegen/meta/src/isa/arm64/mod.rs   |  19 +-
 cranelift/codegen/meta/src/isa/riscv/mod.rs   |  17 +-
 cranelift/codegen/meta/src/isa/x86/mod.rs     |  16 +-
 cranelift/codegen/meta/src/shared/legalize.rs |   2 +-
 cranelift/codegen/meta/src/shared/types.rs    |   6 +-
 13 files changed, 608 insertions(+), 28 deletions(-)
 create mode 100644 cranelift/codegen/meta/src/cdsl/encodings.rs
 create mode 100644 cranelift/codegen/meta/src/cdsl/recipes.rs

diff --git a/cranelift/codegen/meta/src/cdsl/cpu_modes.rs b/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
index 30a7028cf9..4b4406d2d3 100644
--- a/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
+++ b/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
@@ -1,25 +1,34 @@
-use crate::cdsl::types::LaneType;
-use crate::cdsl::xform::{TransformGroup, TransformGroupIndex, TransformGroups};
-
-use std::collections::{HashMap, HashSet};
+use std::collections::{hash_map, HashMap, HashSet};
 use std::iter::FromIterator;
 
+use crate::cdsl::encodings::Encoding;
+use crate::cdsl::types::{LaneType, ValueType};
+use crate::cdsl::xform::{TransformGroup, TransformGroupIndex};
+
 pub struct CpuMode {
-    _name: &'static str,
+    pub name: &'static str,
     default_legalize: Option,
     monomorphic_legalize: Option,
-    typed_legalize: HashMap,
+    typed_legalize: HashMap,
+    pub encodings: Vec,
 }
 
 impl CpuMode {
     pub fn new(name: &'static str) -> Self {
         Self {
-            _name: name,
+            name,
             default_legalize: None,
             monomorphic_legalize: None,
             typed_legalize: HashMap::new(),
+            encodings: Vec::new(),
         }
     }
+
+    pub fn set_encodings(&mut self, encodings: Vec) {
+        assert!(self.encodings.is_empty(), "clobbering encodings");
+        self.encodings = encodings;
+    }
+
     pub fn legalize_monomorphic(&mut self, group: &TransformGroup) {
         assert!(self.monomorphic_legalize.is_none());
         self.monomorphic_legalize = Some(group.id);
@@ -31,10 +40,30 @@ impl CpuMode {
     pub fn legalize_type(&mut self, lane_type: impl Into, group: &TransformGroup) {
         assert!(self
             .typed_legalize
-            .insert(lane_type.into().to_string(), group.id)
+            .insert(lane_type.into().into(), group.id)
             .is_none());
     }
 
+    pub fn get_default_legalize_code(&self) -> TransformGroupIndex {
+        self.default_legalize
+            .expect("a finished CpuMode must have a default legalize code")
+    }
+    pub fn get_legalize_code_for(&self, typ: &Option) -> TransformGroupIndex {
+        match typ {
+            Some(typ) => self
+                .typed_legalize
+                .get(typ)
+                .map(|x| *x)
+                .unwrap_or_else(|| self.get_default_legalize_code()),
+            None => self
+                .monomorphic_legalize
+                .unwrap_or_else(|| self.get_default_legalize_code()),
+        }
+    }
+    pub fn get_legalized_types(&self) -> hash_map::Keys {
+        self.typed_legalize.keys()
+    }
+
     /// Returns a deterministically ordered, deduplicated list of TransformGroupIndex for the directly
     /// reachable set of TransformGroup this TargetIsa uses.
     pub fn direct_transform_groups(&self) -> Vec {
diff --git a/cranelift/codegen/meta/src/cdsl/encodings.rs b/cranelift/codegen/meta/src/cdsl/encodings.rs
new file mode 100644
index 0000000000..cd05eb9b38
--- /dev/null
+++ b/cranelift/codegen/meta/src/cdsl/encodings.rs
@@ -0,0 +1,160 @@
+use std::rc::Rc;
+
+use crate::cdsl::instructions::{
+    InstSpec, Instruction, InstructionPredicate, InstructionPredicateNode,
+    InstructionPredicateNumber, InstructionPredicateRegistry, ValueTypeOrAny,
+};
+use crate::cdsl::recipes::{EncodingRecipeNumber, Recipes};
+use crate::cdsl::settings::SettingPredicateNumber;
+use crate::cdsl::types::ValueType;
+
+/// Encoding for a concrete instruction.
+///
+/// An `Encoding` object ties an instruction opcode with concrete type variables together with an
+/// encoding recipe and encoding encbits.
+///
+/// The concrete instruction can be in three different forms:
+///
+/// 1. A naked opcode: `trap` for non-polymorphic instructions.
+/// 2. With bound type variables: `iadd.i32` for polymorphic instructions.
+/// 3. With operands providing constraints: `icmp.i32(intcc.eq, x, y)`.
+///
+/// If the instruction is polymorphic, all type variables must be provided.
+pub struct EncodingContent {
+    /// The `Instruction` or `BoundInstruction` being encoded.
+    inst: InstSpec,
+
+    /// The `EncodingRecipe` to use.
+    pub recipe: EncodingRecipeNumber,
+
+    /// Additional encoding bits to be interpreted by `recipe`.
+    pub encbits: u16,
+
+    /// An instruction predicate that must be true to allow selecting this encoding.
+    pub inst_predicate: Option,
+
+    /// An ISA predicate that must be true to allow selecting this encoding.
+    pub isa_predicate: Option,
+
+    /// The value type this encoding has been bound to, for encodings of polymorphic instructions.
+    pub bound_type: Option,
+}
+
+impl EncodingContent {
+    pub fn inst(&self) -> &Instruction {
+        self.inst.inst()
+    }
+    pub fn to_rust_comment(&self, recipes: &Recipes) -> String {
+        format!("[{}#{:02x}]", recipes[self.recipe].name, self.encbits)
+    }
+}
+
+pub type Encoding = Rc;
+
+pub struct EncodingBuilder {
+    inst: InstSpec,
+    recipe: EncodingRecipeNumber,
+    encbits: u16,
+    inst_predicate: Option,
+    isa_predicate: Option,
+    bound_type: Option,
+}
+
+impl EncodingBuilder {
+    pub fn new(inst: InstSpec, recipe: EncodingRecipeNumber, encbits: u16) -> Self {
+        let (inst_predicate, bound_type) = match &inst {
+            InstSpec::Bound(inst) => {
+                let other_typevars = &inst.inst.polymorphic_info.as_ref().unwrap().other_typevars;
+
+                assert!(
+                    inst.value_types.len() == other_typevars.len() + 1,
+                    "partially bound polymorphic instruction"
+                );
+
+                // Add secondary type variables to the instruction predicate.
+                let value_types = &inst.value_types;
+                let mut inst_predicate = None;
+                for (typevar, value_type) in other_typevars.iter().zip(value_types.iter().skip(1)) {
+                    let value_type = match value_type {
+                        ValueTypeOrAny::Any => continue,
+                        ValueTypeOrAny::ValueType(vt) => vt,
+                    };
+                    let type_predicate =
+                        InstructionPredicate::new_typevar_check(&inst.inst, typevar, value_type);
+                    inst_predicate = Some(type_predicate.into());
+                }
+
+                let ctrl_type = value_types[0]
+                    .clone()
+                    .expect("Controlling type shouldn't be Any");
+                (inst_predicate, Some(ctrl_type))
+            }
+
+            InstSpec::Inst(inst) => {
+                assert!(
+                    inst.polymorphic_info.is_none(),
+                    "unbound polymorphic instruction"
+                );
+                (None, None)
+            }
+        };
+
+        Self {
+            inst,
+            recipe,
+            encbits,
+            inst_predicate,
+            isa_predicate: None,
+            bound_type,
+        }
+    }
+
+    pub fn inst_predicate(mut self, inst_predicate: InstructionPredicateNode) -> Self {
+        let inst_predicate = Some(match self.inst_predicate {
+            Some(node) => node.and(inst_predicate),
+            None => inst_predicate.into(),
+        });
+        self.inst_predicate = inst_predicate;
+        self
+    }
+
+    pub fn isa_predicate(mut self, isa_predicate: SettingPredicateNumber) -> Self {
+        assert!(self.isa_predicate.is_none());
+        self.isa_predicate = Some(isa_predicate);
+        self
+    }
+
+    pub fn build(
+        self,
+        recipes: &Recipes,
+        inst_pred_reg: &mut InstructionPredicateRegistry,
+    ) -> Encoding {
+        let inst_predicate = self.inst_predicate.map(|pred| inst_pred_reg.insert(pred));
+
+        let inst = self.inst.inst();
+        assert!(
+            inst.format == recipes[self.recipe].format,
+            format!(
+                "Inst {} and recipe {} must have the same format!",
+                inst.name, recipes[self.recipe].name
+            )
+        );
+
+        assert_eq!(
+            inst.is_branch && !inst.is_indirect_branch,
+            recipes[self.recipe].branch_range.is_some(),
+            "Inst {}'s is_branch contradicts recipe {} branch_range!",
+            inst.name,
+            recipes[self.recipe].name
+        );
+
+        Rc::new(EncodingContent {
+            inst: self.inst,
+            recipe: self.recipe,
+            encbits: self.encbits,
+            inst_predicate,
+            isa_predicate: self.isa_predicate,
+            bound_type: self.bound_type,
+        })
+    }
+}
diff --git a/cranelift/codegen/meta/src/cdsl/isa.rs b/cranelift/codegen/meta/src/cdsl/isa.rs
index 07467229c7..e561ce2e9c 100644
--- a/cranelift/codegen/meta/src/cdsl/isa.rs
+++ b/cranelift/codegen/meta/src/cdsl/isa.rs
@@ -1,18 +1,21 @@
+use std::collections::HashSet;
+use std::iter::FromIterator;
+
 use crate::cdsl::cpu_modes::CpuMode;
-use crate::cdsl::instructions::InstructionGroup;
+use crate::cdsl::instructions::{InstructionGroup, InstructionPredicateMap};
+use crate::cdsl::recipes::Recipes;
 use crate::cdsl::regs::IsaRegs;
 use crate::cdsl::settings::SettingGroup;
 use crate::cdsl::xform::{TransformGroupIndex, TransformGroups};
 
-use std::collections::HashSet;
-use std::iter::FromIterator;
-
 pub struct TargetIsa {
     pub name: &'static str,
     pub instructions: InstructionGroup,
     pub settings: SettingGroup,
     pub regs: IsaRegs,
+    pub recipes: Recipes,
     pub cpu_modes: Vec,
+    pub encodings_predicates: InstructionPredicateMap,
 
     /// TransformGroupIndex are global to all the ISAs, while we want to have indices into the
     /// local array of transform groups that are directly used. We use this map to get this
@@ -26,7 +29,9 @@ impl TargetIsa {
         instructions: InstructionGroup,
         settings: SettingGroup,
         regs: IsaRegs,
+        recipes: Recipes,
         cpu_modes: Vec,
+        encodings_predicates: InstructionPredicateMap,
     ) -> Self {
         // Compute the local TransformGroup index.
         let mut local_transform_groups = Vec::new();
@@ -49,7 +54,9 @@ impl TargetIsa {
             instructions,
             settings,
             regs,
+            recipes,
             cpu_modes,
+            encodings_predicates,
             local_transform_groups,
         }
     }
diff --git a/cranelift/codegen/meta/src/cdsl/mod.rs b/cranelift/codegen/meta/src/cdsl/mod.rs
index d3227e2dd8..370b442ae4 100644
--- a/cranelift/codegen/meta/src/cdsl/mod.rs
+++ b/cranelift/codegen/meta/src/cdsl/mod.rs
@@ -6,10 +6,12 @@
 #[macro_use]
 pub mod ast;
 pub mod cpu_modes;
+pub mod encodings;
 pub mod formats;
 pub mod instructions;
 pub mod isa;
 pub mod operands;
+pub mod recipes;
 pub mod regs;
 pub mod settings;
 pub mod type_inference;
diff --git a/cranelift/codegen/meta/src/cdsl/recipes.rs b/cranelift/codegen/meta/src/cdsl/recipes.rs
new file mode 100644
index 0000000000..84510cf0d1
--- /dev/null
+++ b/cranelift/codegen/meta/src/cdsl/recipes.rs
@@ -0,0 +1,297 @@
+use cranelift_entity::{entity_impl, PrimaryMap};
+
+use crate::cdsl::formats::{FormatRegistry, InstructionFormatIndex};
+use crate::cdsl::instructions::InstructionPredicate;
+use crate::cdsl::regs::RegClassIndex;
+use crate::cdsl::settings::SettingPredicateNumber;
+
+/// A specific register in a register class.
+///
+/// A register is identified by the top-level register class it belongs to and
+/// its first register unit.
+///
+/// Specific registers are used to describe constraints on instructions where
+/// some operands must use a fixed register.
+///
+/// Register instances can be created with the constructor, or accessed as
+/// attributes on the register class: `GPR.rcx`.
+#[derive(Copy, Clone, Hash, PartialEq, Eq)]
+pub struct Register {
+    pub regclass: RegClassIndex,
+    pub unit: u8,
+}
+
+impl Register {
+    pub fn new(regclass: RegClassIndex, unit: u8) -> Self {
+        Self { regclass, unit }
+    }
+}
+
+/// An operand that must be in a stack slot.
+///
+/// A `Stack` object can be used to indicate an operand constraint for a value
+/// operand that must live in a stack slot.
+#[derive(Copy, Clone, Hash, PartialEq)]
+pub struct Stack {
+    pub regclass: RegClassIndex,
+}
+
+impl Stack {
+    pub fn new(regclass: RegClassIndex) -> Self {
+        Self { regclass }
+    }
+    pub fn stack_base_mask(&self) -> &'static str {
+        // TODO: Make this configurable instead of just using the SP.
+        "StackBaseMask(1)"
+    }
+}
+
+#[derive(Clone, Hash, PartialEq)]
+pub struct BranchRange {
+    pub inst_size: u64,
+    pub range: u64,
+}
+
+#[derive(Copy, Clone, Hash, PartialEq)]
+pub enum OperandConstraint {
+    RegClass(RegClassIndex),
+    FixedReg(Register),
+    TiedInput(usize),
+    Stack(Stack),
+}
+
+impl Into for RegClassIndex {
+    fn into(self) -> OperandConstraint {
+        OperandConstraint::RegClass(self)
+    }
+}
+
+impl Into for Register {
+    fn into(self) -> OperandConstraint {
+        OperandConstraint::FixedReg(self)
+    }
+}
+
+impl Into for usize {
+    fn into(self) -> OperandConstraint {
+        OperandConstraint::TiedInput(self)
+    }
+}
+
+impl Into for Stack {
+    fn into(self) -> OperandConstraint {
+        OperandConstraint::Stack(self)
+    }
+}
+
+/// A recipe for encoding instructions with a given format.
+///
+/// Many different instructions can be encoded by the same recipe, but they
+/// must all have the same instruction format.
+///
+/// The `operands_in` and `operands_out` arguments are tuples specifying the register
+/// allocation constraints for the value operands and results respectively. The
+/// possible constraints for an operand are:
+///
+/// - A `RegClass` specifying the set of allowed registers.
+/// - A `Register` specifying a fixed-register operand.
+/// - An integer indicating that this result is tied to a value operand, so
+///   they must use the same register.
+/// - A `Stack` specifying a value in a stack slot.
+///
+/// The `branch_range` argument must be provided for recipes that can encode
+/// branch instructions. It is an `(origin, bits)` tuple describing the exact
+/// range that can be encoded in a branch instruction.
+#[derive(Clone, Hash)]
+pub struct EncodingRecipe {
+    /// Short mnemonic name for this recipe.
+    pub name: String,
+
+    /// Associated instruction format.
+    pub format: InstructionFormatIndex,
+
+    /// Base number of bytes in the binary encoded instruction.
+    pub base_size: u64,
+
+    /// Tuple of register constraints for value operands.
+    pub operands_in: Vec,
+
+    /// Tuple of register constraints for results.
+    pub operands_out: Vec,
+
+    /// Function name to use when computing actual size.
+    pub compute_size: &'static str,
+
+    /// `(origin, bits)` range for branches.
+    pub branch_range: Option,
+
+    /// This instruction clobbers `iflags` and `fflags`; true by default.
+    pub clobbers_flags: bool,
+
+    /// Instruction predicate.
+    pub inst_predicate: Option,
+
+    /// ISA predicate.
+    pub isa_predicate: Option,
+
+    /// Rust code for binary emission.
+    pub emit: Option,
+}
+
+// Implement PartialEq ourselves: take all the fields into account but the name.
+impl PartialEq for EncodingRecipe {
+    fn eq(&self, other: &Self) -> bool {
+        self.format == other.format
+            && self.base_size == other.base_size
+            && self.operands_in == other.operands_in
+            && self.operands_out == other.operands_out
+            && self.compute_size == other.compute_size
+            && self.branch_range == other.branch_range
+            && self.clobbers_flags == other.clobbers_flags
+            && self.inst_predicate == other.inst_predicate
+            && self.isa_predicate == other.isa_predicate
+            && self.emit == other.emit
+    }
+}
+
+// To allow using it in a hashmap.
+impl Eq for EncodingRecipe {}
+
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct EncodingRecipeNumber(u32);
+entity_impl!(EncodingRecipeNumber);
+
+pub type Recipes = PrimaryMap;
+
+#[derive(Clone)]
+pub struct EncodingRecipeBuilder {
+    pub name: String,
+    format: InstructionFormatIndex,
+    pub base_size: u64,
+    pub operands_in: Option>,
+    pub operands_out: Option>,
+    compute_size: Option<&'static str>,
+    pub branch_range: Option,
+    pub emit: Option,
+    clobbers_flags: Option,
+    inst_predicate: Option,
+    isa_predicate: Option,
+}
+
+impl EncodingRecipeBuilder {
+    pub fn new(name: impl Into, format: InstructionFormatIndex, base_size: u64) -> Self {
+        Self {
+            name: name.into(),
+            format,
+            base_size,
+            operands_in: None,
+            operands_out: None,
+            compute_size: None,
+            branch_range: None,
+            emit: None,
+            clobbers_flags: None,
+            inst_predicate: None,
+            isa_predicate: None,
+        }
+    }
+
+    // Setters.
+    pub fn operands_in(mut self, constraints: Vec>) -> Self {
+        assert!(self.operands_in.is_none());
+        self.operands_in = Some(
+            constraints
+                .into_iter()
+                .map(|constr| constr.into())
+                .collect(),
+        );
+        self
+    }
+    pub fn operands_out(mut self, constraints: Vec>) -> Self {
+        assert!(self.operands_out.is_none());
+        self.operands_out = Some(
+            constraints
+                .into_iter()
+                .map(|constr| constr.into())
+                .collect(),
+        );
+        self
+    }
+    pub fn clobbers_flags(mut self, flag: bool) -> Self {
+        assert!(self.clobbers_flags.is_none());
+        self.clobbers_flags = Some(flag);
+        self
+    }
+    pub fn emit(mut self, code: impl Into) -> Self {
+        assert!(self.emit.is_none());
+        self.emit = Some(code.into());
+        self
+    }
+    pub fn branch_range(mut self, range: (u64, u64)) -> Self {
+        assert!(self.branch_range.is_none());
+        self.branch_range = Some(BranchRange {
+            inst_size: range.0,
+            range: range.1,
+        });
+        self
+    }
+    pub fn isa_predicate(mut self, pred: SettingPredicateNumber) -> Self {
+        assert!(self.isa_predicate.is_none());
+        self.isa_predicate = Some(pred);
+        self
+    }
+    pub fn inst_predicate(mut self, inst_predicate: impl Into) -> Self {
+        assert!(self.inst_predicate.is_none());
+        self.inst_predicate = Some(inst_predicate.into());
+        self
+    }
+    pub fn compute_size(mut self, compute_size: &'static str) -> Self {
+        assert!(self.compute_size.is_none());
+        self.compute_size = Some(compute_size);
+        self
+    }
+
+    pub fn build(self, formats: &FormatRegistry) -> EncodingRecipe {
+        let operands_in = self.operands_in.unwrap_or(Vec::new());
+        let operands_out = self.operands_out.unwrap_or(Vec::new());
+
+        // The number of input constraints must match the number of format input operands.
+        if !formats.get(self.format).has_value_list {
+            let format = formats.get(self.format);
+            assert!(
+                operands_in.len() == format.num_value_operands,
+                format!(
+                    "missing operand constraints for recipe {} (format {})",
+                    self.name, format.name
+                )
+            );
+        }
+
+        // Ensure tied inputs actually refer to existing inputs.
+        for constraint in operands_in.iter().chain(operands_out.iter()) {
+            if let OperandConstraint::TiedInput(n) = *constraint {
+                assert!(n < operands_in.len());
+            }
+        }
+
+        let compute_size = match self.compute_size {
+            Some(compute_size) => compute_size,
+            None => "base_size",
+        };
+
+        let clobbers_flags = self.clobbers_flags.unwrap_or(true);
+
+        EncodingRecipe {
+            name: self.name.into(),
+            format: self.format,
+            base_size: self.base_size,
+            operands_in,
+            operands_out,
+            compute_size,
+            branch_range: self.branch_range,
+            clobbers_flags,
+            inst_predicate: self.inst_predicate,
+            isa_predicate: self.isa_predicate,
+            emit: self.emit,
+        }
+    }
+}
diff --git a/cranelift/codegen/meta/src/cdsl/regs.rs b/cranelift/codegen/meta/src/cdsl/regs.rs
index 8a84d03617..60bf8f5818 100644
--- a/cranelift/codegen/meta/src/cdsl/regs.rs
+++ b/cranelift/codegen/meta/src/cdsl/regs.rs
@@ -35,9 +35,24 @@ impl RegBank {
             classes: Vec::new(),
         }
     }
+
+    fn unit_by_name(&self, name: &'static str) -> u8 {
+        let unit = if let Some(found) = self.names.iter().position(|®_name| reg_name == name) {
+            found
+        } else {
+            // Try to match without the bank prefix.
+            assert!(name.starts_with(self.prefix));
+            let name_without_prefix = &name[self.prefix.len()..];
+            self.names
+                .iter()
+                .position(|®_name| reg_name == name_without_prefix)
+                .expect(&format!("invalid register name {}", name))
+        };
+        self.first_unit + (unit as u8)
+    }
 }
 
-#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
 pub struct RegClassIndex(u32);
 entity_impl!(RegClassIndex);
 
@@ -352,4 +367,17 @@ impl IsaRegs {
     ) -> Self {
         Self { banks, classes }
     }
+
+    pub fn class_by_name(&self, name: &str) -> RegClassIndex {
+        self.classes
+            .values()
+            .find(|&class| class.name == name)
+            .expect(&format!("register class {} not found", name))
+            .index
+    }
+
+    pub fn regunit_by_name(&self, class_index: RegClassIndex, name: &'static str) -> u8 {
+        let bank_index = self.classes.get(class_index).unwrap().bank;
+        self.banks.get(bank_index).unwrap().unit_by_name(name)
+    }
 }
diff --git a/cranelift/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
index a8aa4020d8..21bf0161c4 100644
--- a/cranelift/codegen/meta/src/cdsl/types.rs
+++ b/cranelift/codegen/meta/src/cdsl/types.rs
@@ -27,7 +27,7 @@ static _RUST_NAME_PREFIX: &'static str = "ir::types::";
 ///
 /// All SSA values have a type that is described by an instance of `ValueType`
 /// or one of its subclasses.
-#[derive(Clone, Debug, PartialEq)]
+#[derive(Clone, Debug, PartialEq, Eq, Hash)]
 pub enum ValueType {
     BV(BVType),
     Lane(LaneType),
@@ -147,7 +147,7 @@ impl From for ValueType {
 }
 
 /// A concrete scalar type that can appear as a vector lane too.
-#[derive(Clone, Copy, PartialEq)]
+#[derive(Clone, Copy, PartialEq, Eq, Hash)]
 pub enum LaneType {
     BoolType(shared_types::Bool),
     FloatType(shared_types::Float),
@@ -327,7 +327,7 @@ impl Iterator for LaneTypeIterator {
 ///
 /// A vector type has a lane type which is an instance of `LaneType`,
 /// and a positive number of lanes.
-#[derive(Clone, PartialEq)]
+#[derive(Clone, PartialEq, Eq, Hash)]
 pub struct VectorType {
     base: LaneType,
     lanes: u64,
@@ -393,7 +393,7 @@ impl fmt::Debug for VectorType {
 }
 
 /// A flat bitvector type. Used for semantics description only.
-#[derive(Clone, PartialEq)]
+#[derive(Clone, PartialEq, Eq, Hash)]
 pub struct BVType {
     bits: u64,
 }
diff --git a/cranelift/codegen/meta/src/isa/arm32/mod.rs b/cranelift/codegen/meta/src/isa/arm32/mod.rs
index c1b6e05fda..d143a15163 100644
--- a/cranelift/codegen/meta/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm32/mod.rs
@@ -1,6 +1,7 @@
 use crate::cdsl::cpu_modes::CpuMode;
-use crate::cdsl::instructions::InstructionGroupBuilder;
+use crate::cdsl::instructions::{InstructionGroupBuilder, InstructionPredicateMap};
 use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::recipes::Recipes;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
 
@@ -71,5 +72,19 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
 
     let cpu_modes = vec![a32, t32];
 
-    TargetIsa::new("arm32", inst_group, settings, regs, cpu_modes)
+    // TODO implement arm32 recipes.
+    let recipes = Recipes::new();
+
+    // TODO implement arm32 encodings and predicates.
+    let encodings_predicates = InstructionPredicateMap::new();
+
+    TargetIsa::new(
+        "arm32",
+        inst_group,
+        settings,
+        regs,
+        recipes,
+        cpu_modes,
+        encodings_predicates,
+    )
 }
diff --git a/cranelift/codegen/meta/src/isa/arm64/mod.rs b/cranelift/codegen/meta/src/isa/arm64/mod.rs
index c1276f5924..195fc80ae6 100644
--- a/cranelift/codegen/meta/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm64/mod.rs
@@ -1,6 +1,7 @@
 use crate::cdsl::cpu_modes::CpuMode;
-use crate::cdsl::instructions::InstructionGroupBuilder;
+use crate::cdsl::instructions::{InstructionGroupBuilder, InstructionPredicateMap};
 use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::recipes::Recipes;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
 
@@ -64,5 +65,19 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
 
     let cpu_modes = vec![a64];
 
-    TargetIsa::new("arm64", inst_group, settings, regs, cpu_modes)
+    // TODO implement arm64 recipes.
+    let recipes = Recipes::new();
+
+    // TODO implement arm64 encodings and predicates.
+    let encodings_predicates = InstructionPredicateMap::new();
+
+    TargetIsa::new(
+        "arm64",
+        inst_group,
+        settings,
+        regs,
+        recipes,
+        cpu_modes,
+        encodings_predicates,
+    )
 }
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index 6c00b98f52..5bf9c05390 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -1,6 +1,7 @@
 use crate::cdsl::cpu_modes::CpuMode;
-use crate::cdsl::instructions::InstructionGroupBuilder;
+use crate::cdsl::instructions::{InstructionGroupBuilder, InstructionPredicateMap};
 use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::recipes::Recipes;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 
@@ -115,5 +116,17 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
 
     let cpu_modes = vec![rv_32, rv_64];
 
-    TargetIsa::new("riscv", inst_group, settings, regs, cpu_modes)
+    let recipes = Recipes::new();
+
+    let encodings_predicates = InstructionPredicateMap::new();
+
+    TargetIsa::new(
+        "riscv",
+        inst_group,
+        settings,
+        regs,
+        recipes,
+        cpu_modes,
+        encodings_predicates,
+    )
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index 993a4d9a3b..5407ec2e87 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -1,5 +1,7 @@
 use crate::cdsl::cpu_modes::CpuMode;
+use crate::cdsl::instructions::InstructionPredicateMap;
 use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::recipes::Recipes;
 
 use crate::shared::types::Bool::B1;
 use crate::shared::types::Float::{F32, F64};
@@ -51,5 +53,17 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
 
     let cpu_modes = vec![x86_64, x86_32];
 
-    TargetIsa::new("x86", inst_group, settings, regs, cpu_modes)
+    let recipes = Recipes::new();
+
+    let encodings_predicates = InstructionPredicateMap::new();
+
+    TargetIsa::new(
+        "x86",
+        inst_group,
+        settings,
+        regs,
+        recipes,
+        cpu_modes,
+        encodings_predicates,
+    )
 }
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index d8ec65e3b7..004bdff95b 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -804,7 +804,7 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
 
     expand_flags.build_and_add_to(&mut groups);
 
-    // XXX The order of declarations unfortunately matters to be compatible with the Python code.
+    // TODO The order of declarations unfortunately matters to be compatible with the Python code.
     // When it's all migrated, we can put this next to the narrow/expand build_and_add_to calls
     // above.
     widen.build_and_add_to(&mut groups);
diff --git a/cranelift/codegen/meta/src/shared/types.rs b/cranelift/codegen/meta/src/shared/types.rs
index aff837252b..b327da67db 100644
--- a/cranelift/codegen/meta/src/shared/types.rs
+++ b/cranelift/codegen/meta/src/shared/types.rs
@@ -1,6 +1,6 @@
 //! This module predefines all the Cranelift scalar types.
 
-#[derive(Debug, Clone, Copy, Eq, PartialEq)]
+#[derive(Debug, Clone, Copy, Eq, PartialEq, Hash)]
 pub enum Bool {
     /// 1-bit bool.
     B1 = 1,
@@ -41,7 +41,7 @@ impl Iterator for BoolIterator {
     }
 }
 
-#[derive(Debug, Clone, Copy, Eq, PartialEq)]
+#[derive(Debug, Clone, Copy, Eq, PartialEq, Hash)]
 pub enum Int {
     /// 8-bit int.
     I8 = 8,
@@ -79,7 +79,7 @@ impl Iterator for IntIterator {
     }
 }
 
-#[derive(Debug, Clone, Copy, Eq, PartialEq)]
+#[derive(Debug, Clone, Copy, Eq, PartialEq, Hash)]
 pub enum Float {
     F32 = 32,
     F64 = 64,

From ca277422bb68090f8ac9fc21aa9d00f3ae9177dd Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 24 Jun 2019 16:46:59 +0200
Subject: [PATCH 2476/3084] [meta] Recipes and encodings descriptions for
 RiscV;

---
 .../codegen/meta/src/isa/riscv/encodings.rs   | 383 ++++++++++++++++++
 cranelift/codegen/meta/src/isa/riscv/mod.rs   |  19 +-
 .../codegen/meta/src/isa/riscv/recipes.rs     | 267 ++++++++++++
 3 files changed, 663 insertions(+), 6 deletions(-)
 create mode 100644 cranelift/codegen/meta/src/isa/riscv/encodings.rs
 create mode 100644 cranelift/codegen/meta/src/isa/riscv/recipes.rs

diff --git a/cranelift/codegen/meta/src/isa/riscv/encodings.rs b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
new file mode 100644
index 0000000000..3f1c199bc0
--- /dev/null
+++ b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
@@ -0,0 +1,383 @@
+use crate::cdsl::ast::{Apply, Expr, Literal, VarPool};
+use crate::cdsl::encodings::{Encoding, EncodingBuilder};
+use crate::cdsl::instructions::{
+    BoundInstruction, InstSpec, InstructionPredicateNode, InstructionPredicateRegistry,
+};
+use crate::cdsl::recipes::{EncodingRecipeNumber, Recipes};
+use crate::cdsl::settings::SettingGroup;
+
+use crate::shared::types::Bool::B1;
+use crate::shared::types::Int::{I32, I64};
+use crate::shared::Definitions as SharedDefinitions;
+
+use super::recipes::RecipeGroup;
+
+fn enc(inst: impl Into, recipe: EncodingRecipeNumber, bits: u16) -> EncodingBuilder {
+    EncodingBuilder::new(inst.into(), recipe, bits)
+}
+
+pub struct PerCpuModeEncodings<'defs> {
+    pub inst_pred_reg: InstructionPredicateRegistry,
+    pub enc32: Vec,
+    pub enc64: Vec,
+    recipes: &'defs Recipes,
+}
+
+impl<'defs> PerCpuModeEncodings<'defs> {
+    fn new(recipes: &'defs Recipes) -> Self {
+        Self {
+            inst_pred_reg: InstructionPredicateRegistry::new(),
+            enc32: Vec::new(),
+            enc64: Vec::new(),
+            recipes,
+        }
+    }
+    fn add32(&mut self, encoding: EncodingBuilder) {
+        self.enc32
+            .push(encoding.build(self.recipes, &mut self.inst_pred_reg));
+    }
+    fn add64(&mut self, encoding: EncodingBuilder) {
+        self.enc64
+            .push(encoding.build(self.recipes, &mut self.inst_pred_reg));
+    }
+}
+
+// The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit instructions have 11 as
+// the two low bits, with bits 6:2 determining the base opcode.
+//
+// Encbits for the 32-bit recipes are opcode[6:2] | (funct3 << 5) | ...
+// The functions below encode the encbits.
+
+fn load_bits(funct3: u16) -> u16 {
+    assert!(funct3 <= 0b111);
+    0b00000 | (funct3 << 5)
+}
+
+fn store_bits(funct3: u16) -> u16 {
+    assert!(funct3 <= 0b111);
+    0b01000 | (funct3 << 5)
+}
+
+fn branch_bits(funct3: u16) -> u16 {
+    assert!(funct3 <= 0b111);
+    0b11000 | (funct3 << 5)
+}
+
+fn jalr_bits() -> u16 {
+    // This was previously accepting an argument funct3 of 3 bits and used the following formula:
+    //0b11001 | (funct3 << 5)
+    0b11001
+}
+
+fn jal_bits() -> u16 {
+    0b11011
+}
+
+fn opimm_bits(funct3: u16, funct7: u16) -> u16 {
+    assert!(funct3 <= 0b111);
+    0b00100 | (funct3 << 5) | (funct7 << 8)
+}
+
+fn opimm32_bits(funct3: u16, funct7: u16) -> u16 {
+    assert!(funct3 <= 0b111);
+    0b00110 | (funct3 << 5) | (funct7 << 8)
+}
+
+fn op_bits(funct3: u16, funct7: u16) -> u16 {
+    assert!(funct3 <= 0b111);
+    assert!(funct7 <= 0b1111111);
+    0b01100 | (funct3 << 5) | (funct7 << 8)
+}
+
+fn op32_bits(funct3: u16, funct7: u16) -> u16 {
+    assert!(funct3 <= 0b111);
+    assert!(funct7 <= 0b1111111);
+    0b01110 | (funct3 << 5) | (funct7 << 8)
+}
+
+fn lui_bits() -> u16 {
+    0b01101
+}
+
+pub fn define<'defs>(
+    shared_defs: &'defs SharedDefinitions,
+    isa_settings: &SettingGroup,
+    recipes: &'defs RecipeGroup,
+) -> PerCpuModeEncodings<'defs> {
+    // Instructions shorthands.
+    let shared = &shared_defs.instructions;
+
+    let band = shared.by_name("band");
+    let band_imm = shared.by_name("band_imm");
+    let bor = shared.by_name("bor");
+    let bor_imm = shared.by_name("bor_imm");
+    let br_icmp = shared.by_name("br_icmp");
+    let brz = shared.by_name("brz");
+    let brnz = shared.by_name("brnz");
+    let bxor = shared.by_name("bxor");
+    let bxor_imm = shared.by_name("bxor_imm");
+    let call = shared.by_name("call");
+    let call_indirect = shared.by_name("call_indirect");
+    let copy = shared.by_name("copy");
+    let fill = shared.by_name("fill");
+    let iadd = shared.by_name("iadd");
+    let iadd_imm = shared.by_name("iadd_imm");
+    let iconst = shared.by_name("iconst");
+    let icmp = shared.by_name("icmp");
+    let icmp_imm = shared.by_name("icmp_imm");
+    let imul = shared.by_name("imul");
+    let ishl = shared.by_name("ishl");
+    let ishl_imm = shared.by_name("ishl_imm");
+    let isub = shared.by_name("isub");
+    let jump = shared.by_name("jump");
+    let regmove = shared.by_name("regmove");
+    let spill = shared.by_name("spill");
+    let sshr = shared.by_name("sshr");
+    let sshr_imm = shared.by_name("sshr_imm");
+    let ushr = shared.by_name("ushr");
+    let ushr_imm = shared.by_name("ushr_imm");
+    let return_ = shared.by_name("return");
+
+    // Recipes shorthands, prefixed with r_.
+    let r_icall = recipes.by_name("Icall");
+    let r_icopy = recipes.by_name("Icopy");
+    let r_ii = recipes.by_name("Ii");
+    let r_iicmp = recipes.by_name("Iicmp");
+    let r_iret = recipes.by_name("Iret");
+    let r_irmov = recipes.by_name("Irmov");
+    let r_iz = recipes.by_name("Iz");
+    let r_gp_sp = recipes.by_name("GPsp");
+    let r_gp_fi = recipes.by_name("GPfi");
+    let r_r = recipes.by_name("R");
+    let r_ricmp = recipes.by_name("Ricmp");
+    let r_rshamt = recipes.by_name("Rshamt");
+    let r_sb = recipes.by_name("SB");
+    let r_sb_zero = recipes.by_name("SBzero");
+    let r_u = recipes.by_name("U");
+    let r_uj = recipes.by_name("UJ");
+    let r_uj_call = recipes.by_name("UJcall");
+
+    // Predicates shorthands.
+    let use_m = isa_settings.predicate_by_name("use_m");
+
+    // Definitions.
+    let mut e = PerCpuModeEncodings::new(&recipes.recipes);
+
+    // Basic arithmetic binary instructions are encoded in an R-type instruction.
+    for &(inst, inst_imm, f3, f7) in &[
+        (iadd, Some(iadd_imm), 0b000, 0b0000000),
+        (isub, None, 0b000, 0b0100000),
+        (bxor, Some(bxor_imm), 0b100, 0b0000000),
+        (bor, Some(bor_imm), 0b110, 0b0000000),
+        (band, Some(band_imm), 0b111, 0b0000000),
+    ] {
+        e.add32(enc(inst.bind(I32), r_r, op_bits(f3, f7)));
+        e.add64(enc(inst.bind(I64), r_r, op_bits(f3, f7)));
+
+        // Immediate versions for add/xor/or/and.
+        if let Some(inst_imm) = inst_imm {
+            e.add32(enc(inst_imm.bind(I32), r_ii, opimm_bits(f3, 0)));
+            e.add64(enc(inst_imm.bind(I64), r_ii, opimm_bits(f3, 0)));
+        }
+    }
+
+    // 32-bit ops in RV64.
+    e.add64(enc(iadd.bind(I32), r_r, op32_bits(0b000, 0b0000000)));
+    e.add64(enc(isub.bind(I32), r_r, op32_bits(0b000, 0b0100000)));
+    // There are no andiw/oriw/xoriw variations.
+    e.add64(enc(iadd_imm.bind(I32), r_ii, opimm32_bits(0b000, 0)));
+
+    // Use iadd_imm with %x0 to materialize constants.
+    e.add32(enc(iconst.bind(I32), r_iz, opimm_bits(0b0, 0)));
+    e.add64(enc(iconst.bind(I32), r_iz, opimm_bits(0b0, 0)));
+    e.add64(enc(iconst.bind(I64), r_iz, opimm_bits(0b0, 0)));
+
+    // Dynamic shifts have the same masking semantics as the clif base instructions.
+    for &(inst, inst_imm, f3, f7) in &[
+        (ishl, ishl_imm, 0b1, 0b0),
+        (ushr, ushr_imm, 0b101, 0b0),
+        (sshr, sshr_imm, 0b101, 0b100000),
+    ] {
+        e.add32(enc(inst.bind(I32).bind(I32), r_r, op_bits(f3, f7)));
+        e.add64(enc(inst.bind(I64).bind(I64), r_r, op_bits(f3, f7)));
+        e.add64(enc(inst.bind(I32).bind(I32), r_r, op32_bits(f3, f7)));
+        // Allow i32 shift amounts in 64-bit shifts.
+        e.add64(enc(inst.bind(I64).bind(I32), r_r, op_bits(f3, f7)));
+        e.add64(enc(inst.bind(I32).bind(I64), r_r, op32_bits(f3, f7)));
+
+        // Immediate shifts.
+        e.add32(enc(inst_imm.bind(I32), r_rshamt, opimm_bits(f3, f7)));
+        e.add64(enc(inst_imm.bind(I64), r_rshamt, opimm_bits(f3, f7)));
+        e.add64(enc(inst_imm.bind(I32), r_rshamt, opimm32_bits(f3, f7)));
+    }
+
+    // Signed and unsigned integer 'less than'. There are no 'w' variants for comparing 32-bit
+    // numbers in RV64.
+    {
+        let mut var_pool = VarPool::new();
+
+        // Helper that creates an instruction predicate for an instruction in the icmp family.
+        let mut icmp_instp = |bound_inst: &BoundInstruction,
+                              intcc_field: &'static str|
+         -> InstructionPredicateNode {
+            let x = var_pool.create("x");
+            let y = var_pool.create("y");
+            let cc =
+                Literal::enumerator_for(shared_defs.operand_kinds.by_name("intcc"), intcc_field);
+            Apply::new(
+                bound_inst.clone().into(),
+                vec![Expr::Literal(cc), Expr::Var(x), Expr::Var(y)],
+            )
+            .inst_predicate(&shared_defs.format_registry, &var_pool)
+            .unwrap()
+        };
+
+        let icmp_i32 = icmp.bind(I32);
+        let icmp_i64 = icmp.bind(I64);
+        e.add32(
+            enc(icmp_i32.clone(), r_ricmp, op_bits(0b010, 0b0000000))
+                .inst_predicate(icmp_instp(&icmp_i32, "slt")),
+        );
+        e.add64(
+            enc(icmp_i64.clone(), r_ricmp, op_bits(0b010, 0b0000000))
+                .inst_predicate(icmp_instp(&icmp_i64, "slt")),
+        );
+
+        e.add32(
+            enc(icmp_i32.clone(), r_ricmp, op_bits(0b011, 0b0000000))
+                .inst_predicate(icmp_instp(&icmp_i32, "ult")),
+        );
+        e.add64(
+            enc(icmp_i64.clone(), r_ricmp, op_bits(0b011, 0b0000000))
+                .inst_predicate(icmp_instp(&icmp_i64, "ult")),
+        );
+
+        // Immediate variants.
+        let icmp_i32 = icmp_imm.bind(I32);
+        let icmp_i64 = icmp_imm.bind(I64);
+        e.add32(
+            enc(icmp_i32.clone(), r_iicmp, opimm_bits(0b010, 0))
+                .inst_predicate(icmp_instp(&icmp_i32, "slt")),
+        );
+        e.add64(
+            enc(icmp_i64.clone(), r_iicmp, opimm_bits(0b010, 0))
+                .inst_predicate(icmp_instp(&icmp_i64, "slt")),
+        );
+
+        e.add32(
+            enc(icmp_i32.clone(), r_iicmp, opimm_bits(0b011, 0))
+                .inst_predicate(icmp_instp(&icmp_i32, "ult")),
+        );
+        e.add64(
+            enc(icmp_i64.clone(), r_iicmp, opimm_bits(0b011, 0))
+                .inst_predicate(icmp_instp(&icmp_i64, "ult")),
+        );
+    }
+
+    // Integer constants with the low 12 bits clear are materialized by lui.
+    e.add32(enc(iconst.bind(I32), r_u, lui_bits()));
+    e.add64(enc(iconst.bind(I32), r_u, lui_bits()));
+    e.add64(enc(iconst.bind(I64), r_u, lui_bits()));
+
+    // "M" Standard Extension for Integer Multiplication and Division.
+    // Gated by the `use_m` flag.
+    e.add32(enc(imul.bind(I32), r_r, op_bits(0b000, 0b00000001)).isa_predicate(use_m));
+    e.add64(enc(imul.bind(I64), r_r, op_bits(0b000, 0b00000001)).isa_predicate(use_m));
+    e.add64(enc(imul.bind(I32), r_r, op32_bits(0b000, 0b00000001)).isa_predicate(use_m));
+
+    // Control flow.
+
+    // Unconditional branches.
+    e.add32(enc(jump, r_uj, jal_bits()));
+    e.add64(enc(jump, r_uj, jal_bits()));
+    e.add32(enc(call, r_uj_call, jal_bits()));
+    e.add64(enc(call, r_uj_call, jal_bits()));
+
+    // Conditional branches.
+    {
+        let mut var_pool = VarPool::new();
+
+        // Helper that creates an instruction predicate for an instruction in the icmp family.
+        let mut br_icmp_instp = |bound_inst: &BoundInstruction,
+                                 intcc_field: &'static str|
+         -> InstructionPredicateNode {
+            let x = var_pool.create("x");
+            let y = var_pool.create("y");
+            let dest = var_pool.create("dest");
+            let args = var_pool.create("args");
+            let cc =
+                Literal::enumerator_for(shared_defs.operand_kinds.by_name("intcc"), intcc_field);
+            Apply::new(
+                bound_inst.clone().into(),
+                vec![
+                    Expr::Literal(cc),
+                    Expr::Var(x),
+                    Expr::Var(y),
+                    Expr::Var(dest),
+                    Expr::Var(args),
+                ],
+            )
+            .inst_predicate(&shared_defs.format_registry, &var_pool)
+            .unwrap()
+        };
+
+        let br_icmp_i32 = br_icmp.bind(I32);
+        let br_icmp_i64 = br_icmp.bind(I64);
+        for &(cond, f3) in &[
+            ("eq", 0b000),
+            ("ne", 0b001),
+            ("slt", 0b100),
+            ("sge", 0b101),
+            ("ult", 0b110),
+            ("uge", 0b111),
+        ] {
+            e.add32(
+                enc(br_icmp_i32.clone(), r_sb, branch_bits(f3))
+                    .inst_predicate(br_icmp_instp(&br_icmp_i32, cond)),
+            );
+            e.add64(
+                enc(br_icmp_i64.clone(), r_sb, branch_bits(f3))
+                    .inst_predicate(br_icmp_instp(&br_icmp_i64, cond)),
+            );
+        }
+    }
+
+    for &(inst, f3) in &[(brz, 0b000), (brnz, 0b001)] {
+        e.add32(enc(inst.bind(I32), r_sb_zero, branch_bits(f3)));
+        e.add64(enc(inst.bind(I64), r_sb_zero, branch_bits(f3)));
+        e.add32(enc(inst.bind(B1), r_sb_zero, branch_bits(f3)));
+        e.add64(enc(inst.bind(B1), r_sb_zero, branch_bits(f3)));
+    }
+
+    // Returns are a special case of jalr_bits using %x1 to hold the return address.
+    // The return address is provided by a special-purpose `link` return value that
+    // is added by legalize_signature().
+    e.add32(enc(return_, r_iret, jalr_bits()));
+    e.add64(enc(return_, r_iret, jalr_bits()));
+    e.add32(enc(call_indirect.bind(I32), r_icall, jalr_bits()));
+    e.add64(enc(call_indirect.bind(I64), r_icall, jalr_bits()));
+
+    // Spill and fill.
+    e.add32(enc(spill.bind(I32), r_gp_sp, store_bits(0b010)));
+    e.add64(enc(spill.bind(I32), r_gp_sp, store_bits(0b010)));
+    e.add64(enc(spill.bind(I64), r_gp_sp, store_bits(0b011)));
+    e.add32(enc(fill.bind(I32), r_gp_fi, load_bits(0b010)));
+    e.add64(enc(fill.bind(I32), r_gp_fi, load_bits(0b010)));
+    e.add64(enc(fill.bind(I64), r_gp_fi, load_bits(0b011)));
+
+    // Register copies.
+    e.add32(enc(copy.bind(I32), r_icopy, opimm_bits(0b000, 0)));
+    e.add64(enc(copy.bind(I64), r_icopy, opimm_bits(0b000, 0)));
+    e.add64(enc(copy.bind(I32), r_icopy, opimm32_bits(0b000, 0)));
+
+    e.add32(enc(regmove.bind(I32), r_irmov, opimm_bits(0b000, 0)));
+    e.add64(enc(regmove.bind(I64), r_irmov, opimm_bits(0b000, 0)));
+    e.add64(enc(regmove.bind(I32), r_irmov, opimm32_bits(0b000, 0)));
+
+    e.add32(enc(copy.bind(B1), r_icopy, opimm_bits(0b000, 0)));
+    e.add64(enc(copy.bind(B1), r_icopy, opimm_bits(0b000, 0)));
+    e.add32(enc(regmove.bind(B1), r_irmov, opimm_bits(0b000, 0)));
+    e.add64(enc(regmove.bind(B1), r_irmov, opimm_bits(0b000, 0)));
+
+    e
+}
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index 5bf9c05390..435b38eb34 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -1,7 +1,6 @@
 use crate::cdsl::cpu_modes::CpuMode;
-use crate::cdsl::instructions::{InstructionGroupBuilder, InstructionPredicateMap};
+use crate::cdsl::instructions::InstructionGroupBuilder;
 use crate::cdsl::isa::TargetIsa;
-use crate::cdsl::recipes::Recipes;
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 
@@ -9,6 +8,9 @@ use crate::shared::types::Float::{F32, F64};
 use crate::shared::types::Int::{I32, I64};
 use crate::shared::Definitions as SharedDefinitions;
 
+mod encodings;
+mod recipes;
+
 fn define_settings(shared: &SettingGroup) -> SettingGroup {
     let mut setting = SettingGroupBuilder::new("riscv");
 
@@ -114,12 +116,17 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     rv_64.legalize_type(F32, expand);
     rv_64.legalize_type(F64, expand);
 
+    let recipes = recipes::define(shared_defs, ®s);
+
+    let encodings = encodings::define(shared_defs, &settings, &recipes);
+    rv_32.set_encodings(encodings.enc32);
+    rv_64.set_encodings(encodings.enc64);
+    let encodings_predicates = encodings.inst_pred_reg.extract();
+
+    let recipes = recipes.collect();
+
     let cpu_modes = vec![rv_32, rv_64];
 
-    let recipes = Recipes::new();
-
-    let encodings_predicates = InstructionPredicateMap::new();
-
     TargetIsa::new(
         "riscv",
         inst_group,
diff --git a/cranelift/codegen/meta/src/isa/riscv/recipes.rs b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
new file mode 100644
index 0000000000..8309802b92
--- /dev/null
+++ b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
@@ -0,0 +1,267 @@
+use std::collections::HashMap;
+
+use crate::cdsl::formats::FormatRegistry;
+use crate::cdsl::instructions::InstructionPredicate;
+use crate::cdsl::recipes::{EncodingRecipeBuilder, EncodingRecipeNumber, Recipes, Stack};
+use crate::cdsl::regs::IsaRegs;
+use crate::shared::Definitions as SharedDefinitions;
+
+/// An helper to create recipes and use them when defining the RISCV encodings.
+pub struct RecipeGroup<'formats> {
+    /// Memoized format registry, to pass it to the builders.
+    formats: &'formats FormatRegistry,
+
+    /// The actualy list of recipes explicitly created in this file.
+    pub recipes: Recipes,
+
+    /// Provides fast lookup from a name to an encoding recipe.
+    name_to_recipe: HashMap,
+}
+
+impl<'formats> RecipeGroup<'formats> {
+    fn new(formats: &'formats FormatRegistry) -> Self {
+        Self {
+            formats,
+            recipes: Recipes::new(),
+            name_to_recipe: HashMap::new(),
+        }
+    }
+
+    fn push(&mut self, builder: EncodingRecipeBuilder) {
+        assert!(
+            self.name_to_recipe.get(&builder.name).is_none(),
+            format!("riscv recipe '{}' created twice", builder.name)
+        );
+        let name = builder.name.clone();
+        let number = self.recipes.push(builder.build(self.formats));
+        self.name_to_recipe.insert(name, number);
+    }
+
+    pub fn by_name(&self, name: &str) -> EncodingRecipeNumber {
+        let number = *self
+            .name_to_recipe
+            .get(name)
+            .expect(&format!("unknown riscv recipe name {}", name));
+        number
+    }
+
+    pub fn collect(self) -> Recipes {
+        self.recipes
+    }
+}
+
+pub fn define<'formats>(
+    shared_defs: &'formats SharedDefinitions,
+    regs: &IsaRegs,
+) -> RecipeGroup<'formats> {
+    let formats = &shared_defs.format_registry;
+
+    // Format shorthands.
+    let f_binary = formats.by_name("Binary");
+    let f_binary_imm = formats.by_name("BinaryImm");
+    let f_branch = formats.by_name("Branch");
+    let f_branch_icmp = formats.by_name("BranchIcmp");
+    let f_call = formats.by_name("Call");
+    let f_call_indirect = formats.by_name("CallIndirect");
+    let f_int_compare = formats.by_name("IntCompare");
+    let f_int_compare_imm = formats.by_name("IntCompareImm");
+    let f_jump = formats.by_name("Jump");
+    let f_multiary = formats.by_name("MultiAry");
+    let f_regmove = formats.by_name("RegMove");
+    let f_unary = formats.by_name("Unary");
+    let f_unary_imm = formats.by_name("UnaryImm");
+
+    // Register classes shorthands.
+    let gpr = regs.class_by_name("GPR");
+
+    // Definitions.
+    let mut recipes = RecipeGroup::new(&shared_defs.format_registry);
+
+    // R-type 32-bit instructions: These are mostly binary arithmetic instructions.
+    // The encbits are `opcode[6:2] | (funct3 << 5) | (funct7 << 8)
+    recipes.push(
+        EncodingRecipeBuilder::new("R", f_binary, 4)
+            .operands_in(vec![gpr, gpr])
+            .operands_out(vec![gpr])
+            .emit("put_r(bits, in_reg0, in_reg1, out_reg0, sink);"),
+    );
+
+    // R-type with an immediate shift amount instead of rs2.
+    recipes.push(
+        EncodingRecipeBuilder::new("Rshamt", f_binary_imm, 4)
+            .operands_in(vec![gpr])
+            .operands_out(vec![gpr])
+            .emit("put_rshamt(bits, in_reg0, imm.into(), out_reg0, sink);"),
+    );
+
+    // R-type encoding of an integer comparison.
+    recipes.push(
+        EncodingRecipeBuilder::new("Ricmp", f_int_compare, 4)
+            .operands_in(vec![gpr, gpr])
+            .operands_out(vec![gpr])
+            .emit("put_r(bits, in_reg0, in_reg1, out_reg0, sink);"),
+    );
+
+    let format = formats.get(f_binary_imm);
+    recipes.push(
+        EncodingRecipeBuilder::new("Ii", f_binary_imm, 4)
+            .operands_in(vec![gpr])
+            .operands_out(vec![gpr])
+            .inst_predicate(InstructionPredicate::new_is_signed_int(
+                format, "imm", 12, 0,
+            ))
+            .emit("put_i(bits, in_reg0, imm.into(), out_reg0, sink);"),
+    );
+
+    // I-type instruction with a hardcoded %x0 rs1.
+    let format = formats.get(f_unary_imm);
+    recipes.push(
+        EncodingRecipeBuilder::new("Iz", f_unary_imm, 4)
+            .operands_out(vec![gpr])
+            .inst_predicate(InstructionPredicate::new_is_signed_int(
+                format, "imm", 12, 0,
+            ))
+            .emit("put_i(bits, 0, imm.into(), out_reg0, sink);"),
+    );
+
+    // I-type encoding of an integer comparison.
+    let format = formats.get(f_int_compare_imm);
+    recipes.push(
+        EncodingRecipeBuilder::new("Iicmp", f_int_compare_imm, 4)
+            .operands_in(vec![gpr])
+            .operands_out(vec![gpr])
+            .inst_predicate(InstructionPredicate::new_is_signed_int(
+                format, "imm", 12, 0,
+            ))
+            .emit("put_i(bits, in_reg0, imm.into(), out_reg0, sink);"),
+    );
+
+    // I-type encoding for `jalr` as a return instruction. We won't use the immediate offset.  The
+    // variable return values are not encoded.
+    recipes.push(EncodingRecipeBuilder::new("Iret", f_multiary, 4).emit(
+        r#"
+                    // Return instructions are always a jalr to %x1.
+                    // The return address is provided as a special-purpose link argument.
+                    put_i(
+                        bits,
+                        1, // rs1 = %x1
+                        0, // no offset.
+                        0, // rd = %x0: no address written.
+                        sink,
+                    );
+                "#,
+    ));
+
+    // I-type encoding for `jalr` as a call_indirect.
+    recipes.push(
+        EncodingRecipeBuilder::new("Icall", f_call_indirect, 4)
+            .operands_in(vec![gpr])
+            .emit(
+                r#"
+                    // call_indirect instructions are jalr with rd=%x1.
+                    put_i(
+                        bits,
+                        in_reg0,
+                        0, // no offset.
+                        1, // rd = %x1: link register.
+                        sink,
+                    );
+                "#,
+            ),
+    );
+
+    // Copy of a GPR is implemented as addi x, 0.
+    recipes.push(
+        EncodingRecipeBuilder::new("Icopy", f_unary, 4)
+            .operands_in(vec![gpr])
+            .operands_out(vec![gpr])
+            .emit("put_i(bits, in_reg0, 0, out_reg0, sink);"),
+    );
+
+    // Same for a GPR regmove.
+    recipes.push(
+        EncodingRecipeBuilder::new("Irmov", f_regmove, 4)
+            .operands_in(vec![gpr])
+            .emit("put_i(bits, src, 0, dst, sink);"),
+    );
+
+    // U-type instructions have a 20-bit immediate that targets bits 12-31.
+    let format = formats.get(f_unary_imm);
+    recipes.push(
+        EncodingRecipeBuilder::new("U", f_unary_imm, 4)
+            .operands_out(vec![gpr])
+            .inst_predicate(InstructionPredicate::new_is_signed_int(
+                format, "imm", 32, 12,
+            ))
+            .emit("put_u(bits, imm.into(), out_reg0, sink);"),
+    );
+
+    // UJ-type unconditional branch instructions.
+    recipes.push(
+        EncodingRecipeBuilder::new("UJ", f_jump, 4)
+            .branch_range((0, 21))
+            .emit(
+                r#"
+                    let dest = i64::from(func.offsets[destination]);
+                    let disp = dest - i64::from(sink.offset());
+                    put_uj(bits, disp, 0, sink);
+                "#,
+            ),
+    );
+
+    recipes.push(EncodingRecipeBuilder::new("UJcall", f_call, 4).emit(
+        r#"
+                    sink.reloc_external(Reloc::RiscvCall,
+                                        &func.dfg.ext_funcs[func_ref].name,
+                                        0);
+                    // rd=%x1 is the standard link register.
+                    put_uj(bits, 0, 1, sink);
+                "#,
+    ));
+
+    // SB-type branch instructions.
+    recipes.push(
+        EncodingRecipeBuilder::new("SB", f_branch_icmp, 4)
+            .operands_in(vec![gpr, gpr])
+            .branch_range((0, 13))
+            .emit(
+                r#"
+                    let dest = i64::from(func.offsets[destination]);
+                    let disp = dest - i64::from(sink.offset());
+                    put_sb(bits, disp, in_reg0, in_reg1, sink);
+                "#,
+            ),
+    );
+
+    // SB-type branch instruction with rs2 fixed to zero.
+    recipes.push(
+        EncodingRecipeBuilder::new("SBzero", f_branch, 4)
+            .operands_in(vec![gpr])
+            .branch_range((0, 13))
+            .emit(
+                r#"
+                    let dest = i64::from(func.offsets[destination]);
+                    let disp = dest - i64::from(sink.offset());
+                    put_sb(bits, disp, in_reg0, 0, sink);
+                "#,
+            ),
+    );
+
+    // Spill of a GPR.
+    recipes.push(
+        EncodingRecipeBuilder::new("GPsp", f_unary, 4)
+            .operands_in(vec![gpr])
+            .operands_out(vec![Stack::new(gpr)])
+            .emit("unimplemented!();"),
+    );
+
+    // Fill of a GPR.
+    recipes.push(
+        EncodingRecipeBuilder::new("GPfi", f_unary, 4)
+            .operands_in(vec![Stack::new(gpr)])
+            .operands_out(vec![gpr])
+            .emit("unimplemented!();"),
+    );
+
+    recipes
+}

From fd036772924379e64f099843bf28deb227e7543c Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 24 Jun 2019 16:47:18 +0200
Subject: [PATCH 2477/3084] [meta] Recipes and encodings descriptions for x86;

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 1569 +++++++++
 cranelift/codegen/meta/src/isa/x86/mod.rs     |   17 +-
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 2805 +++++++++++++++++
 3 files changed, 4385 insertions(+), 6 deletions(-)
 create mode 100644 cranelift/codegen/meta/src/isa/x86/encodings.rs
 create mode 100644 cranelift/codegen/meta/src/isa/x86/recipes.rs

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
new file mode 100644
index 0000000000..99bfecafc2
--- /dev/null
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -0,0 +1,1569 @@
+#![allow(non_snake_case)]
+
+use std::collections::HashMap;
+
+use crate::cdsl::encodings::{Encoding, EncodingBuilder};
+use crate::cdsl::instructions::{
+    BoundInstruction, InstSpec, Instruction, InstructionGroup, InstructionPredicate,
+    InstructionPredicateNode, InstructionPredicateRegistry,
+};
+use crate::cdsl::recipes::{EncodingRecipe, EncodingRecipeNumber, Recipes};
+use crate::cdsl::settings::{SettingGroup, SettingPredicateNumber};
+
+use crate::shared::types::Bool::B1;
+use crate::shared::types::Float::{F32, F64};
+use crate::shared::types::Int::{I16, I32, I64, I8};
+use crate::shared::Definitions as SharedDefinitions;
+
+use super::recipes::{RecipeGroup, Template};
+
+pub struct PerCpuModeEncodings {
+    pub enc32: Vec,
+    pub enc64: Vec,
+    pub recipes: Recipes,
+    recipes_inverse: HashMap,
+    pub inst_pred_reg: InstructionPredicateRegistry,
+}
+
+impl PerCpuModeEncodings {
+    fn new() -> Self {
+        Self {
+            enc32: Vec::new(),
+            enc64: Vec::new(),
+            recipes: Recipes::new(),
+            recipes_inverse: HashMap::new(),
+            inst_pred_reg: InstructionPredicateRegistry::new(),
+        }
+    }
+
+    fn add_recipe(&mut self, recipe: EncodingRecipe) -> EncodingRecipeNumber {
+        if let Some(found_index) = self.recipes_inverse.get(&recipe) {
+            assert!(
+                self.recipes[*found_index].name == recipe.name,
+                format!(
+                    "trying to insert different recipes with a same name ({})",
+                    recipe.name
+                )
+            );
+            *found_index
+        } else {
+            let index = self.recipes.push(recipe.clone());
+            self.recipes_inverse.insert(recipe, index);
+            index
+        }
+    }
+
+    fn make_encoding(
+        &mut self,
+        inst: InstSpec,
+        template: Template,
+        builder_closure: T,
+    ) -> Encoding
+    where
+        T: FnOnce(EncodingBuilder) -> EncodingBuilder,
+    {
+        let (recipe, bits) = template.build();
+        let recipe_number = self.add_recipe(recipe);
+        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits);
+        builder_closure(builder).build(&self.recipes, &mut self.inst_pred_reg)
+    }
+
+    fn enc32_func(&mut self, inst: impl Into, template: Template, builder_closure: T)
+    where
+        T: FnOnce(EncodingBuilder) -> EncodingBuilder,
+    {
+        let encoding = self.make_encoding(inst.into(), template, builder_closure);
+        self.enc32.push(encoding);
+    }
+    fn enc32(&mut self, inst: impl Into, template: Template) {
+        self.enc32_func(inst, template, |x| x);
+    }
+    fn enc32_isap(
+        &mut self,
+        inst: impl Into,
+        template: Template,
+        isap: SettingPredicateNumber,
+    ) {
+        self.enc32_func(inst, template, |encoding| encoding.isa_predicate(isap));
+    }
+    fn enc32_instp(
+        &mut self,
+        inst: impl Into,
+        template: Template,
+        instp: InstructionPredicateNode,
+    ) {
+        self.enc32_func(inst, template, |encoding| encoding.inst_predicate(instp));
+    }
+    fn enc32_rec(&mut self, inst: impl Into, recipe: &EncodingRecipe, bits: u16) {
+        let recipe_number = self.add_recipe(recipe.clone());
+        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits);
+        let encoding = builder.build(&self.recipes, &mut self.inst_pred_reg);
+        self.enc32.push(encoding);
+    }
+
+    fn enc64_func(&mut self, inst: impl Into, template: Template, builder_closure: T)
+    where
+        T: FnOnce(EncodingBuilder) -> EncodingBuilder,
+    {
+        let encoding = self.make_encoding(inst.into(), template, builder_closure);
+        self.enc64.push(encoding);
+    }
+    fn enc64(&mut self, inst: impl Into, template: Template) {
+        self.enc64_func(inst, template, |x| x);
+    }
+    fn enc64_isap(
+        &mut self,
+        inst: impl Into,
+        template: Template,
+        isap: SettingPredicateNumber,
+    ) {
+        self.enc64_func(inst, template, |encoding| encoding.isa_predicate(isap));
+    }
+    fn enc64_instp(
+        &mut self,
+        inst: impl Into,
+        template: Template,
+        instp: InstructionPredicateNode,
+    ) {
+        self.enc64_func(inst, template, |encoding| encoding.inst_predicate(instp));
+    }
+    fn enc64_rec(&mut self, inst: impl Into, recipe: &EncodingRecipe, bits: u16) {
+        let recipe_number = self.add_recipe(recipe.clone());
+        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits);
+        let encoding = builder.build(&self.recipes, &mut self.inst_pred_reg);
+        self.enc64.push(encoding);
+    }
+
+    /// Add encodings for `inst.i32` to X86_32.
+    /// Add encodings for `inst.i32` to X86_64 with and without REX.
+    /// Add encodings for `inst.i64` to X86_64 with a REX.W prefix.
+    fn enc_i32_i64(&mut self, inst: impl Into, template: Template) {
+        let inst: InstSpec = inst.into();
+        self.enc32(inst.bind(I32), template.nonrex());
+
+        // REX-less encoding must come after REX encoding so we don't use it by default. Otherwise
+        // reg-alloc would never use r8 and up.
+        self.enc64(inst.bind(I32), template.rex());
+        self.enc64(inst.bind(I32), template.nonrex());
+        self.enc64(inst.bind(I64), template.rex().w());
+    }
+
+    /// Add encodings for `inst.i32` to X86_32.
+    /// Add encodings for `inst.i32` to X86_64 with and without REX.
+    /// Add encodings for `inst.i64` to X86_64 with a REX.W prefix.
+    fn enc_i32_i64_instp(
+        &mut self,
+        inst: &Instruction,
+        template: Template,
+        instp: InstructionPredicateNode,
+    ) {
+        self.enc32_func(inst.bind(I32), template.nonrex(), |builder| {
+            builder.inst_predicate(instp.clone())
+        });
+
+        // REX-less encoding must come after REX encoding so we don't use it by default. Otherwise
+        // reg-alloc would never use r8 and up.
+        self.enc64_func(inst.bind(I32), template.rex(), |builder| {
+            builder.inst_predicate(instp.clone())
+        });
+        self.enc64_func(inst.bind(I32), template.nonrex(), |builder| {
+            builder.inst_predicate(instp.clone())
+        });
+        self.enc64_func(inst.bind(I64), template.rex().w(), |builder| {
+            builder.inst_predicate(instp)
+        });
+    }
+
+    /// Add encodings for `inst` to X86_64 with and without a REX prefix.
+    fn enc_x86_64(&mut self, inst: impl Into + Clone, template: Template) {
+        // See above comment about the ordering of rex vs non-rex encodings.
+        self.enc64(inst.clone(), template.rex());
+        self.enc64(inst, template);
+    }
+
+    /// Add encodings for `inst` to X86_64 with and without a REX prefix.
+    fn enc_x86_64_instp(
+        &mut self,
+        inst: impl Clone + Into,
+        template: Template,
+        instp: InstructionPredicateNode,
+    ) {
+        // See above comment about the ordering of rex vs non-rex encodings.
+        self.enc64_func(inst.clone(), template.rex(), |builder| {
+            builder.inst_predicate(instp.clone())
+        });
+        self.enc64_func(inst, template, |builder| builder.inst_predicate(instp));
+    }
+    fn enc_x86_64_isap(
+        &mut self,
+        inst: impl Clone + Into,
+        template: Template,
+        isap: SettingPredicateNumber,
+    ) {
+        // See above comment about the ordering of rex vs non-rex encodings.
+        self.enc64_isap(inst.clone(), template.rex(), isap);
+        self.enc64_isap(inst, template, isap);
+    }
+
+    /// Add all three encodings for `inst`:
+    /// - X86_32
+    /// - X86_64 with and without the REX prefix.
+    fn enc_both(&mut self, inst: impl Clone + Into, template: Template) {
+        self.enc32(inst.clone(), template.clone());
+        self.enc_x86_64(inst, template);
+    }
+    fn enc_both_isap(
+        &mut self,
+        inst: BoundInstruction,
+        template: Template,
+        isap: SettingPredicateNumber,
+    ) {
+        self.enc32_isap(inst.clone(), template.clone(), isap);
+        self.enc_x86_64_isap(inst, template, isap);
+    }
+    fn enc_both_instp(
+        &mut self,
+        inst: BoundInstruction,
+        template: Template,
+        instp: InstructionPredicateNode,
+    ) {
+        self.enc32_instp(inst.clone(), template.clone(), instp.clone());
+        self.enc_x86_64_instp(inst, template, instp);
+    }
+
+    /// Add encodings for `inst.i32` to X86_32.
+    /// Add encodings for `inst.i32` to X86_64 with and without REX.
+    /// Add encodings for `inst.i64` to X86_64 with a REX prefix, using the `w_bit`
+    /// argument to determine whether or not to set the REX.W bit.
+    fn enc_i32_i64_ld_st(&mut self, inst: &Instruction, w_bit: bool, template: Template) {
+        self.enc32(inst.clone().bind(I32).bind_any(), template.clone());
+
+        // REX-less encoding must come after REX encoding so we don't use it by
+        // default. Otherwise reg-alloc would never use r8 and up.
+        self.enc64(inst.clone().bind(I32).bind_any(), template.clone().rex());
+        self.enc64(inst.clone().bind(I32).bind_any(), template.clone());
+
+        if w_bit {
+            self.enc64(inst.clone().bind(I64).bind_any(), template.rex().w());
+        } else {
+            self.enc64(inst.clone().bind(I64).bind_any(), template.clone().rex());
+            self.enc64(inst.clone().bind(I64).bind_any(), template);
+        }
+    }
+}
+
+// Definitions.
+
+pub fn define(
+    shared_defs: &SharedDefinitions,
+    settings: &SettingGroup,
+    x86: &InstructionGroup,
+    r: &RecipeGroup,
+) -> PerCpuModeEncodings {
+    let shared = &shared_defs.instructions;
+    let formats = &shared_defs.format_registry;
+
+    // Shorthands for instructions.
+    let adjust_sp_down = shared.by_name("adjust_sp_down");
+    let adjust_sp_down_imm = shared.by_name("adjust_sp_down_imm");
+    let adjust_sp_up_imm = shared.by_name("adjust_sp_up_imm");
+    let band = shared.by_name("band");
+    let band_imm = shared.by_name("band_imm");
+    let band_not = shared.by_name("band_not");
+    let bconst = shared.by_name("bconst");
+    let bint = shared.by_name("bint");
+    let bitcast = shared.by_name("bitcast");
+    let bnot = shared.by_name("bnot");
+    let bor = shared.by_name("bor");
+    let bor_imm = shared.by_name("bor_imm");
+    let brff = shared.by_name("brff");
+    let brif = shared.by_name("brif");
+    let brnz = shared.by_name("brnz");
+    let brz = shared.by_name("brz");
+    let bxor = shared.by_name("bxor");
+    let bxor_imm = shared.by_name("bxor_imm");
+    let call = shared.by_name("call");
+    let call_indirect = shared.by_name("call_indirect");
+    let ceil = shared.by_name("ceil");
+    let clz = shared.by_name("clz");
+    let copy = shared.by_name("copy");
+    let copy_nop = shared.by_name("copy_nop");
+    let copy_special = shared.by_name("copy_special");
+    let ctz = shared.by_name("ctz");
+    let debugtrap = shared.by_name("debugtrap");
+    let f32const = shared.by_name("f32const");
+    let f64const = shared.by_name("f64const");
+    let fadd = shared.by_name("fadd");
+    let fcmp = shared.by_name("fcmp");
+    let fcvt_from_sint = shared.by_name("fcvt_from_sint");
+    let fdemote = shared.by_name("fdemote");
+    let fdiv = shared.by_name("fdiv");
+    let ffcmp = shared.by_name("ffcmp");
+    let fill = shared.by_name("fill");
+    let floor = shared.by_name("floor");
+    let fmul = shared.by_name("fmul");
+    let fpromote = shared.by_name("fpromote");
+    let fsub = shared.by_name("fsub");
+    let func_addr = shared.by_name("func_addr");
+    let iadd = shared.by_name("iadd");
+    let iadd_imm = shared.by_name("iadd_imm");
+    let icmp = shared.by_name("icmp");
+    let icmp_imm = shared.by_name("icmp_imm");
+    let iconst = shared.by_name("iconst");
+    let ifcmp = shared.by_name("ifcmp");
+    let ifcmp_imm = shared.by_name("ifcmp_imm");
+    let ifcmp_sp = shared.by_name("ifcmp_sp");
+    let imul = shared.by_name("imul");
+    let indirect_jump_table_br = shared.by_name("indirect_jump_table_br");
+    let ireduce = shared.by_name("ireduce");
+    let ishl = shared.by_name("ishl");
+    let ishl_imm = shared.by_name("ishl_imm");
+    let istore16 = shared.by_name("istore16");
+    let istore16_complex = shared.by_name("istore16_complex");
+    let istore32 = shared.by_name("istore32");
+    let istore32_complex = shared.by_name("istore32_complex");
+    let istore8 = shared.by_name("istore8");
+    let istore8_complex = shared.by_name("istore8_complex");
+    let isub = shared.by_name("isub");
+    let jump = shared.by_name("jump");
+    let jump_table_base = shared.by_name("jump_table_base");
+    let jump_table_entry = shared.by_name("jump_table_entry");
+    let load = shared.by_name("load");
+    let load_complex = shared.by_name("load_complex");
+    let nearest = shared.by_name("nearest");
+    let popcnt = shared.by_name("popcnt");
+    let regfill = shared.by_name("regfill");
+    let regmove = shared.by_name("regmove");
+    let regspill = shared.by_name("regspill");
+    let return_ = shared.by_name("return");
+    let rotl = shared.by_name("rotl");
+    let rotl_imm = shared.by_name("rotl_imm");
+    let rotr = shared.by_name("rotr");
+    let rotr_imm = shared.by_name("rotr_imm");
+    let selectif = shared.by_name("selectif");
+    let sextend = shared.by_name("sextend");
+    let sload16 = shared.by_name("sload16");
+    let sload16_complex = shared.by_name("sload16_complex");
+    let sload32 = shared.by_name("sload32");
+    let sload32_complex = shared.by_name("sload32_complex");
+    let sload8 = shared.by_name("sload8");
+    let sload8_complex = shared.by_name("sload8_complex");
+    let spill = shared.by_name("spill");
+    let sqrt = shared.by_name("sqrt");
+    let sshr = shared.by_name("sshr");
+    let sshr_imm = shared.by_name("sshr_imm");
+    let stack_addr = shared.by_name("stack_addr");
+    let store = shared.by_name("store");
+    let store_complex = shared.by_name("store_complex");
+    let symbol_value = shared.by_name("symbol_value");
+    let trap = shared.by_name("trap");
+    let trapff = shared.by_name("trapff");
+    let trapif = shared.by_name("trapif");
+    let trueff = shared.by_name("trueff");
+    let trueif = shared.by_name("trueif");
+    let trunc = shared.by_name("trunc");
+    let uextend = shared.by_name("uextend");
+    let uload16 = shared.by_name("uload16");
+    let uload16_complex = shared.by_name("uload16_complex");
+    let uload32 = shared.by_name("uload32");
+    let uload32_complex = shared.by_name("uload32_complex");
+    let uload8 = shared.by_name("uload8");
+    let uload8_complex = shared.by_name("uload8_complex");
+    let ushr = shared.by_name("ushr");
+    let ushr_imm = shared.by_name("ushr_imm");
+    let x86_bsf = x86.by_name("x86_bsf");
+    let x86_bsr = x86.by_name("x86_bsr");
+    let x86_cvtt2si = x86.by_name("x86_cvtt2si");
+    let x86_fmax = x86.by_name("x86_fmax");
+    let x86_fmin = x86.by_name("x86_fmin");
+    let x86_pop = x86.by_name("x86_pop");
+    let x86_push = x86.by_name("x86_push");
+    let x86_sdivmodx = x86.by_name("x86_sdivmodx");
+    let x86_smulx = x86.by_name("x86_smulx");
+    let x86_udivmodx = x86.by_name("x86_udivmodx");
+    let x86_umulx = x86.by_name("x86_umulx");
+
+    // Shorthands for recipes.
+    let rec_adjustsp = r.template("adjustsp");
+    let rec_adjustsp_ib = r.template("adjustsp_ib");
+    let rec_adjustsp_id = r.template("adjustsp_id");
+    let rec_allones_fnaddr4 = r.template("allones_fnaddr4");
+    let rec_allones_fnaddr8 = r.template("allones_fnaddr8");
+    let rec_brfb = r.template("brfb");
+    let rec_brfd = r.template("brfd");
+    let rec_brib = r.template("brib");
+    let rec_brid = r.template("brid");
+    let rec_bsf_and_bsr = r.template("bsf_and_bsr");
+    let rec_call_id = r.template("call_id");
+    let rec_call_plt_id = r.template("call_plt_id");
+    let rec_call_r = r.template("call_r");
+    let rec_cmov = r.template("cmov");
+    let rec_copysp = r.template("copysp");
+    let rec_div = r.template("div");
+    let rec_debugtrap = r.recipe("debugtrap");
+    let rec_f32imm_z = r.template("f32imm_z");
+    let rec_f64imm_z = r.template("f64imm_z");
+    let rec_fa = r.template("fa");
+    let rec_fax = r.template("fax");
+    let rec_fcmp = r.template("fcmp");
+    let rec_fcscc = r.template("fcscc");
+    let rec_ffillSib32 = r.template("ffillSib32");
+    let rec_fillSib32 = r.template("fillSib32");
+    let rec_fld = r.template("fld");
+    let rec_fldDisp32 = r.template("fldDisp32");
+    let rec_fldDisp8 = r.template("fldDisp8");
+    let rec_fldWithIndex = r.template("fldWithIndex");
+    let rec_fldWithIndexDisp32 = r.template("fldWithIndexDisp32");
+    let rec_fldWithIndexDisp8 = r.template("fldWithIndexDisp8");
+    let rec_fnaddr4 = r.template("fnaddr4");
+    let rec_fnaddr8 = r.template("fnaddr8");
+    let rec_fregfill32 = r.template("fregfill32");
+    let rec_fregspill32 = r.template("fregspill32");
+    let rec_frmov = r.template("frmov");
+    let rec_frurm = r.template("frurm");
+    let rec_fspillSib32 = r.template("fspillSib32");
+    let rec_fst = r.template("fst");
+    let rec_fstDisp32 = r.template("fstDisp32");
+    let rec_fstDisp8 = r.template("fstDisp8");
+    let rec_fstWithIndex = r.template("fstWithIndex");
+    let rec_fstWithIndexDisp32 = r.template("fstWithIndexDisp32");
+    let rec_fstWithIndexDisp8 = r.template("fstWithIndexDisp8");
+    let rec_furm = r.template("furm");
+    let rec_furmi_rnd = r.template("furmi_rnd");
+    let rec_got_fnaddr8 = r.template("got_fnaddr8");
+    let rec_got_gvaddr8 = r.template("got_gvaddr8");
+    let rec_gvaddr4 = r.template("gvaddr4");
+    let rec_gvaddr8 = r.template("gvaddr8");
+    let rec_icscc = r.template("icscc");
+    let rec_icscc_ib = r.template("icscc_ib");
+    let rec_icscc_id = r.template("icscc_id");
+    let rec_indirect_jmp = r.template("indirect_jmp");
+    let rec_jmpb = r.template("jmpb");
+    let rec_jmpd = r.template("jmpd");
+    let rec_jt_base = r.template("jt_base");
+    let rec_jt_entry = r.template("jt_entry");
+    let rec_ld = r.template("ld");
+    let rec_ldDisp32 = r.template("ldDisp32");
+    let rec_ldDisp8 = r.template("ldDisp8");
+    let rec_ldWithIndex = r.template("ldWithIndex");
+    let rec_ldWithIndexDisp32 = r.template("ldWithIndexDisp32");
+    let rec_ldWithIndexDisp8 = r.template("ldWithIndexDisp8");
+    let rec_mulx = r.template("mulx");
+    let rec_null = r.recipe("null");
+    let rec_pcrel_fnaddr8 = r.template("pcrel_fnaddr8");
+    let rec_pcrel_gvaddr8 = r.template("pcrel_gvaddr8");
+    let rec_popq = r.template("popq");
+    let rec_pu_id = r.template("pu_id");
+    let rec_pu_id_bool = r.template("pu_id_bool");
+    let rec_pu_iq = r.template("pu_iq");
+    let rec_pushq = r.template("pushq");
+    let rec_ret = r.template("ret");
+    let rec_r_ib = r.template("r_ib");
+    let rec_r_id = r.template("r_id");
+    let rec_rcmp = r.template("rcmp");
+    let rec_rcmp_ib = r.template("rcmp_ib");
+    let rec_rcmp_id = r.template("rcmp_id");
+    let rec_rcmp_sp = r.template("rcmp_sp");
+    let rec_regfill32 = r.template("regfill32");
+    let rec_regspill32 = r.template("regspill32");
+    let rec_rc = r.template("rc");
+    let rec_rfumr = r.template("rfumr");
+    let rec_rfurm = r.template("rfurm");
+    let rec_rmov = r.template("rmov");
+    let rec_rr = r.template("rr");
+    let rec_rrx = r.template("rrx");
+    let rec_setf_abcd = r.template("setf_abcd");
+    let rec_seti_abcd = r.template("seti_abcd");
+    let rec_spaddr4_id = r.template("spaddr4_id");
+    let rec_spaddr8_id = r.template("spaddr8_id");
+    let rec_spillSib32 = r.template("spillSib32");
+    let rec_st = r.template("st");
+    let rec_stacknull = r.recipe("stacknull");
+    let rec_stDisp32 = r.template("stDisp32");
+    let rec_stDisp32_abcd = r.template("stDisp32_abcd");
+    let rec_stDisp8 = r.template("stDisp8");
+    let rec_stDisp8_abcd = r.template("stDisp8_abcd");
+    let rec_stWithIndex = r.template("stWithIndex");
+    let rec_stWithIndexDisp32 = r.template("stWithIndexDisp32");
+    let rec_stWithIndexDisp32_abcd = r.template("stWithIndexDisp32_abcd");
+    let rec_stWithIndexDisp8 = r.template("stWithIndexDisp8");
+    let rec_stWithIndexDisp8_abcd = r.template("stWithIndexDisp8_abcd");
+    let rec_stWithIndex_abcd = r.template("stWithIndex_abcd");
+    let rec_st_abcd = r.template("st_abcd");
+    let rec_t8jccb_abcd = r.template("t8jccb_abcd");
+    let rec_t8jccd_abcd = r.template("t8jccd_abcd");
+    let rec_t8jccd_long = r.template("t8jccd_long");
+    let rec_tjccb = r.template("tjccb");
+    let rec_tjccd = r.template("tjccd");
+    let rec_trap = r.template("trap");
+    let rec_trapif = r.recipe("trapif");
+    let rec_trapff = r.recipe("trapff");
+    let rec_u_id = r.template("u_id");
+    let rec_umr = r.template("umr");
+    let rec_ur = r.template("ur");
+    let rec_urm = r.template("urm");
+    let rec_urm_noflags = r.template("urm_noflags");
+    let rec_urm_noflags_abcd = r.template("urm_noflags_abcd");
+
+    // Predicates shorthands.
+    let all_ones_funcaddrs_and_not_is_pic =
+        settings.predicate_by_name("all_ones_funcaddrs_and_not_is_pic");
+    let is_pic = settings.predicate_by_name("is_pic");
+    let not_all_ones_funcaddrs_and_not_is_pic =
+        settings.predicate_by_name("not_all_ones_funcaddrs_and_not_is_pic");
+    let not_is_pic = settings.predicate_by_name("not_is_pic");
+    let use_popcnt = settings.predicate_by_name("use_popcnt");
+    let use_lzcnt = settings.predicate_by_name("use_lzcnt");
+    let use_bmi1 = settings.predicate_by_name("use_bmi1");
+    let use_sse41 = settings.predicate_by_name("use_sse41");
+
+    // Definitions.
+    let mut e = PerCpuModeEncodings::new();
+
+    e.enc_i32_i64(iadd, rec_rr.opcodes(vec![0x01]));
+    e.enc_i32_i64(isub, rec_rr.opcodes(vec![0x29]));
+    e.enc_i32_i64(band, rec_rr.opcodes(vec![0x21]));
+    e.enc_i32_i64(bor, rec_rr.opcodes(vec![0x09]));
+    e.enc_i32_i64(bxor, rec_rr.opcodes(vec![0x31]));
+
+    // x86 has a bitwise not instruction NOT.
+    e.enc_i32_i64(bnot, rec_ur.opcodes(vec![0xf7]).rrr(2));
+
+    // Also add a `b1` encodings for the logic instructions.
+    // TODO: Should this be done with 8-bit instructions? It would improve partial register
+    // dependencies.
+    e.enc_both(band.bind(B1), rec_rr.opcodes(vec![0x21]));
+    e.enc_both(bor.bind(B1), rec_rr.opcodes(vec![0x09]));
+    e.enc_both(bxor.bind(B1), rec_rr.opcodes(vec![0x31]));
+
+    e.enc_i32_i64(imul, rec_rrx.opcodes(vec![0x0f, 0xaf]));
+    e.enc_i32_i64(x86_sdivmodx, rec_div.opcodes(vec![0xf7]).rrr(7));
+    e.enc_i32_i64(x86_udivmodx, rec_div.opcodes(vec![0xf7]).rrr(6));
+
+    e.enc_i32_i64(x86_smulx, rec_mulx.opcodes(vec![0xf7]).rrr(5));
+    e.enc_i32_i64(x86_umulx, rec_mulx.opcodes(vec![0xf7]).rrr(4));
+
+    e.enc_i32_i64(copy, rec_umr.opcodes(vec![0x89]));
+    e.enc_both(copy.bind(B1), rec_umr.opcodes(vec![0x89]));
+    e.enc_both(copy.bind(I8), rec_umr.opcodes(vec![0x89]));
+    e.enc_both(copy.bind(I16), rec_umr.opcodes(vec![0x89]));
+
+    // TODO For x86-64, only define REX forms for now, since we can't describe the
+    // special regunit immediate operands with the current constraint language.
+    for &ty in &[I8, I16, I32] {
+        e.enc32(regmove.bind(ty), rec_rmov.opcodes(vec![0x89]));
+        e.enc64(regmove.bind(ty), rec_rmov.opcodes(vec![0x89]).rex());
+    }
+    e.enc64(regmove.bind(I64), rec_rmov.opcodes(vec![0x89]).rex().w());
+    e.enc_both(regmove.bind(B1), rec_rmov.opcodes(vec![0x89]));
+    e.enc_both(regmove.bind(I8), rec_rmov.opcodes(vec![0x89]));
+
+    e.enc_i32_i64(iadd_imm, rec_r_ib.opcodes(vec![0x83]).rrr(0));
+    e.enc_i32_i64(iadd_imm, rec_r_id.opcodes(vec![0x81]).rrr(0));
+
+    e.enc_i32_i64(band_imm, rec_r_ib.opcodes(vec![0x83]).rrr(4));
+    e.enc_i32_i64(band_imm, rec_r_id.opcodes(vec![0x81]).rrr(4));
+
+    e.enc_i32_i64(bor_imm, rec_r_ib.opcodes(vec![0x83]).rrr(1));
+    e.enc_i32_i64(bor_imm, rec_r_id.opcodes(vec![0x81]).rrr(1));
+
+    e.enc_i32_i64(bxor_imm, rec_r_ib.opcodes(vec![0x83]).rrr(6));
+    e.enc_i32_i64(bxor_imm, rec_r_id.opcodes(vec![0x81]).rrr(6));
+
+    // TODO: band_imm.i64 with an unsigned 32-bit immediate can be encoded as band_imm.i32. Can
+    // even use the single-byte immediate for 0xffff_ffXX masks.
+
+    // Immediate constants.
+    e.enc32(iconst.bind(I32), rec_pu_id.opcodes(vec![0xb8]));
+
+    e.enc64(iconst.bind(I32), rec_pu_id.rex().opcodes(vec![0xb8]));
+    e.enc64(iconst.bind(I32), rec_pu_id.opcodes(vec![0xb8]));
+
+    // The 32-bit immediate movl also zero-extends to 64 bits.
+    let f_unary_imm = formats.get(formats.by_name("UnaryImm"));
+    let is_unsigned_int32 = InstructionPredicate::new_is_unsigned_int(f_unary_imm, "imm", 32, 0);
+
+    e.enc64_func(
+        iconst.bind(I64),
+        rec_pu_id.opcodes(vec![0xb8]).rex(),
+        |encoding| encoding.inst_predicate(is_unsigned_int32.clone()),
+    );
+    e.enc64_func(
+        iconst.bind(I64),
+        rec_pu_id.opcodes(vec![0xb8]),
+        |encoding| encoding.inst_predicate(is_unsigned_int32),
+    );
+
+    // Sign-extended 32-bit immediate.
+    e.enc64(
+        iconst.bind(I64),
+        rec_u_id.rex().opcodes(vec![0xc7]).rrr(0).w(),
+    );
+
+    // Finally, the 0xb8 opcode takes an 8-byte immediate with a REX.W prefix.
+    e.enc64(iconst.bind(I64), rec_pu_iq.opcodes(vec![0xb8]).rex().w());
+
+    // Bool constants.
+    e.enc_both(bconst.bind(B1), rec_pu_id_bool.opcodes(vec![0xb8]));
+
+    // Shifts and rotates.
+    // Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit
+    // and 16-bit shifts would need explicit masking.
+
+    for &(inst, rrr) in &[(rotl, 0), (rotr, 1), (ishl, 4), (ushr, 5), (sshr, 7)] {
+        // Cannot use enc_i32_i64 for this pattern because instructions require
+        // to bind any.
+        e.enc32(
+            inst.bind(I32).bind_any(),
+            rec_rc.opcodes(vec![0xd3]).rrr(rrr),
+        );
+        e.enc64(
+            inst.bind(I64).bind_any(),
+            rec_rc.opcodes(vec![0xd3]).rrr(rrr).rex().w(),
+        );
+        e.enc64(
+            inst.bind(I32).bind_any(),
+            rec_rc.opcodes(vec![0xd3]).rrr(rrr).rex(),
+        );
+        e.enc64(
+            inst.bind(I32).bind_any(),
+            rec_rc.opcodes(vec![0xd3]).rrr(rrr),
+        );
+    }
+
+    for &(inst, rrr) in &[
+        (rotl_imm, 0),
+        (rotr_imm, 1),
+        (ishl_imm, 4),
+        (ushr_imm, 5),
+        (sshr_imm, 7),
+    ] {
+        e.enc_i32_i64(inst, rec_r_ib.opcodes(vec![0xc1]).rrr(rrr));
+    }
+
+    // Population count.
+    e.enc32_isap(
+        popcnt.bind(I32),
+        rec_urm.opcodes(vec![0xf3, 0x0f, 0xb8]),
+        use_popcnt,
+    );
+    e.enc64_isap(
+        popcnt.bind(I64),
+        rec_urm.opcodes(vec![0xf3, 0x0f, 0xb8]).rex().w(),
+        use_popcnt,
+    );
+    e.enc64_isap(
+        popcnt.bind(I32),
+        rec_urm.opcodes(vec![0xf3, 0x0f, 0xb8]).rex(),
+        use_popcnt,
+    );
+    e.enc64_isap(
+        popcnt.bind(I32),
+        rec_urm.opcodes(vec![0xf3, 0x0f, 0xb8]),
+        use_popcnt,
+    );
+
+    // Count leading zero bits.
+    e.enc32_isap(
+        clz.bind(I32),
+        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbd]),
+        use_lzcnt,
+    );
+    e.enc64_isap(
+        clz.bind(I64),
+        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbd]).rex().w(),
+        use_lzcnt,
+    );
+    e.enc64_isap(
+        clz.bind(I32),
+        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbd]).rex(),
+        use_lzcnt,
+    );
+    e.enc64_isap(
+        clz.bind(I32),
+        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbd]),
+        use_lzcnt,
+    );
+
+    // Count trailing zero bits.
+    e.enc32_isap(
+        ctz.bind(I32),
+        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbc]),
+        use_bmi1,
+    );
+    e.enc64_isap(
+        ctz.bind(I64),
+        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbc]).rex().w(),
+        use_bmi1,
+    );
+    e.enc64_isap(
+        ctz.bind(I32),
+        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbc]).rex(),
+        use_bmi1,
+    );
+    e.enc64_isap(
+        ctz.bind(I32),
+        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbc]),
+        use_bmi1,
+    );
+
+    // Loads and stores.
+    let f_load_complex = formats.get(formats.by_name("LoadComplex"));
+    let is_load_complex_length_two = InstructionPredicate::new_length_equals(f_load_complex, 2);
+
+    for recipe in &[rec_ldWithIndex, rec_ldWithIndexDisp8, rec_ldWithIndexDisp32] {
+        e.enc_i32_i64_instp(
+            load_complex,
+            recipe.opcodes(vec![0x8b]),
+            is_load_complex_length_two.clone(),
+        );
+        e.enc_x86_64_instp(
+            uload32_complex,
+            recipe.opcodes(vec![0x8b]),
+            is_load_complex_length_two.clone(),
+        );
+
+        e.enc64_instp(
+            sload32_complex,
+            recipe.opcodes(vec![0x63]).rex().w(),
+            is_load_complex_length_two.clone(),
+        );
+
+        e.enc_i32_i64_instp(
+            uload16_complex,
+            recipe.opcodes(vec![0x0f, 0xb7]),
+            is_load_complex_length_two.clone(),
+        );
+        e.enc_i32_i64_instp(
+            sload16_complex,
+            recipe.opcodes(vec![0x0f, 0xbf]),
+            is_load_complex_length_two.clone(),
+        );
+
+        e.enc_i32_i64_instp(
+            uload8_complex,
+            recipe.opcodes(vec![0x0f, 0xb6]),
+            is_load_complex_length_two.clone(),
+        );
+
+        e.enc_i32_i64_instp(
+            sload8_complex,
+            recipe.opcodes(vec![0x0f, 0xbe]),
+            is_load_complex_length_two.clone(),
+        );
+    }
+
+    let f_store_complex = formats.get(formats.by_name("StoreComplex"));
+    let is_store_complex_length_three = InstructionPredicate::new_length_equals(f_store_complex, 3);
+
+    for recipe in &[rec_stWithIndex, rec_stWithIndexDisp8, rec_stWithIndexDisp32] {
+        e.enc_i32_i64_instp(
+            store_complex,
+            recipe.opcodes(vec![0x89]),
+            is_store_complex_length_three.clone(),
+        );
+        e.enc_x86_64_instp(
+            istore32_complex,
+            recipe.opcodes(vec![0x89]),
+            is_store_complex_length_three.clone(),
+        );
+        e.enc_both_instp(
+            istore16_complex.bind(I32),
+            recipe.opcodes(vec![0x66, 0x89]),
+            is_store_complex_length_three.clone(),
+        );
+        e.enc_x86_64_instp(
+            istore16_complex.bind(I64),
+            recipe.opcodes(vec![0x66, 0x89]),
+            is_store_complex_length_three.clone(),
+        );
+    }
+
+    for recipe in &[
+        rec_stWithIndex_abcd,
+        rec_stWithIndexDisp8_abcd,
+        rec_stWithIndexDisp32_abcd,
+    ] {
+        e.enc_both_instp(
+            istore8_complex.bind(I32),
+            recipe.opcodes(vec![0x88]),
+            is_store_complex_length_three.clone(),
+        );
+        e.enc_x86_64_instp(
+            istore8_complex.bind(I64),
+            recipe.opcodes(vec![0x88]),
+            is_store_complex_length_three.clone(),
+        );
+    }
+
+    for recipe in &[rec_st, rec_stDisp8, rec_stDisp32] {
+        e.enc_i32_i64_ld_st(store, true, recipe.opcodes(vec![0x89]));
+        e.enc_x86_64(istore32.bind(I64).bind_any(), recipe.opcodes(vec![0x89]));
+        e.enc_i32_i64_ld_st(istore16, false, recipe.opcodes(vec![0x66, 0x89]));
+    }
+
+    // Byte stores are more complicated because the registers they can address
+    // depends of the presence of a REX prefix. The st*_abcd recipes fall back to
+    // the corresponding st* recipes when a REX prefix is applied.
+
+    for recipe in &[rec_st_abcd, rec_stDisp8_abcd, rec_stDisp32_abcd] {
+        e.enc_both(istore8.bind(I32).bind_any(), recipe.opcodes(vec![0x88]));
+        e.enc_x86_64(istore8.bind(I64).bind_any(), recipe.opcodes(vec![0x88]));
+    }
+
+    e.enc_i32_i64(spill, rec_spillSib32.opcodes(vec![0x89]));
+    e.enc_i32_i64(regspill, rec_regspill32.opcodes(vec![0x89]));
+
+    // Use a 32-bit write for spilling `b1`, `i8` and `i16` to avoid
+    // constraining the permitted registers.
+    // See MIN_SPILL_SLOT_SIZE which makes this safe.
+
+    e.enc_both(spill.bind(B1), rec_spillSib32.opcodes(vec![0x89]));
+    e.enc_both(regspill.bind(B1), rec_regspill32.opcodes(vec![0x89]));
+    for &ty in &[I8, I16] {
+        e.enc_both(spill.bind(ty), rec_spillSib32.opcodes(vec![0x89]));
+        e.enc_both(regspill.bind(ty), rec_regspill32.opcodes(vec![0x89]));
+    }
+
+    for recipe in &[rec_ld, rec_ldDisp8, rec_ldDisp32] {
+        e.enc_i32_i64_ld_st(load, true, recipe.opcodes(vec![0x8b]));
+        e.enc_x86_64(uload32.bind(I64), recipe.opcodes(vec![0x8b]));
+        e.enc64(sload32.bind(I64), recipe.opcodes(vec![0x63]).rex().w());
+        e.enc_i32_i64_ld_st(uload16, true, recipe.opcodes(vec![0x0f, 0xb7]));
+        e.enc_i32_i64_ld_st(sload16, true, recipe.opcodes(vec![0x0f, 0xbf]));
+        e.enc_i32_i64_ld_st(uload8, true, recipe.opcodes(vec![0x0f, 0xb6]));
+        e.enc_i32_i64_ld_st(sload8, true, recipe.opcodes(vec![0x0f, 0xbe]));
+    }
+
+    e.enc_i32_i64(fill, rec_fillSib32.opcodes(vec![0x8b]));
+    e.enc_i32_i64(regfill, rec_regfill32.opcodes(vec![0x8b]));
+
+    // Load 32 bits from `b1`, `i8` and `i16` spill slots. See `spill.b1` above.
+
+    e.enc_both(fill.bind(B1), rec_fillSib32.opcodes(vec![0x8b]));
+    e.enc_both(regfill.bind(B1), rec_regfill32.opcodes(vec![0x8b]));
+    for &ty in &[I8, I16] {
+        e.enc_both(fill.bind(ty), rec_fillSib32.opcodes(vec![0x8b]));
+        e.enc_both(regfill.bind(ty), rec_regfill32.opcodes(vec![0x8b]));
+    }
+
+    // Push and Pop.
+    e.enc32(x86_push.bind(I32), rec_pushq.opcodes(vec![0x50]));
+    e.enc_x86_64(x86_push.bind(I64), rec_pushq.opcodes(vec![0x50]));
+
+    e.enc32(x86_pop.bind(I32), rec_popq.opcodes(vec![0x58]));
+    e.enc_x86_64(x86_pop.bind(I64), rec_popq.opcodes(vec![0x58]));
+
+    // Copy Special
+    // For x86-64, only define REX forms for now, since we can't describe the
+    // special regunit immediate operands with the current constraint language.
+    e.enc64(copy_special, rec_copysp.opcodes(vec![0x89]).rex().w());
+    e.enc32(copy_special, rec_copysp.opcodes(vec![0x89]));
+
+    // Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
+    // into a no-op.
+    // The same encoding is generated for both the 64- and 32-bit architectures.
+    for &ty in &[I64, I32, I16, I8] {
+        e.enc64_rec(copy_nop.bind(ty), rec_stacknull, 0);
+        e.enc32_rec(copy_nop.bind(ty), rec_stacknull, 0);
+    }
+    for &ty in &[F64, F32] {
+        e.enc64_rec(copy_nop.bind(ty), rec_stacknull, 0);
+        e.enc32_rec(copy_nop.bind(ty), rec_stacknull, 0);
+    }
+
+    // Adjust SP down by a dynamic value (or up, with a negative operand).
+    e.enc32(adjust_sp_down.bind(I32), rec_adjustsp.opcodes(vec![0x29]));
+    e.enc64(
+        adjust_sp_down.bind(I64),
+        rec_adjustsp.opcodes(vec![0x29]).rex().w(),
+    );
+
+    // Adjust SP up by an immediate (or down, with a negative immediate).
+    e.enc32(adjust_sp_up_imm, rec_adjustsp_ib.opcodes(vec![0x83]));
+    e.enc32(adjust_sp_up_imm, rec_adjustsp_id.opcodes(vec![0x81]));
+    e.enc64(
+        adjust_sp_up_imm,
+        rec_adjustsp_ib.opcodes(vec![0x83]).rex().w(),
+    );
+    e.enc64(
+        adjust_sp_up_imm,
+        rec_adjustsp_id.opcodes(vec![0x81]).rex().w(),
+    );
+
+    // Adjust SP down by an immediate (or up, with a negative immediate).
+    e.enc32(
+        adjust_sp_down_imm,
+        rec_adjustsp_ib.opcodes(vec![0x83]).rrr(5),
+    );
+    e.enc32(
+        adjust_sp_down_imm,
+        rec_adjustsp_id.opcodes(vec![0x81]).rrr(5),
+    );
+    e.enc64(
+        adjust_sp_down_imm,
+        rec_adjustsp_ib.opcodes(vec![0x83]).rrr(5).rex().w(),
+    );
+    e.enc64(
+        adjust_sp_down_imm,
+        rec_adjustsp_id.opcodes(vec![0x81]).rrr(5).rex().w(),
+    );
+
+    // Float loads and stores.
+    e.enc_both(
+        load.bind(F32).bind_any(),
+        rec_fld.opcodes(vec![0xf3, 0x0f, 0x10]),
+    );
+    e.enc_both(
+        load.bind(F32).bind_any(),
+        rec_fldDisp8.opcodes(vec![0xf3, 0x0f, 0x10]),
+    );
+    e.enc_both(
+        load.bind(F32).bind_any(),
+        rec_fldDisp32.opcodes(vec![0xf3, 0x0f, 0x10]),
+    );
+
+    e.enc_both(
+        load_complex.bind(F32),
+        rec_fldWithIndex.opcodes(vec![0xf3, 0x0f, 0x10]),
+    );
+    e.enc_both(
+        load_complex.bind(F32),
+        rec_fldWithIndexDisp8.opcodes(vec![0xf3, 0x0f, 0x10]),
+    );
+    e.enc_both(
+        load_complex.bind(F32),
+        rec_fldWithIndexDisp32.opcodes(vec![0xf3, 0x0f, 0x10]),
+    );
+
+    e.enc_both(
+        load.bind(F64).bind_any(),
+        rec_fld.opcodes(vec![0xf2, 0x0f, 0x10]),
+    );
+    e.enc_both(
+        load.bind(F64).bind_any(),
+        rec_fldDisp8.opcodes(vec![0xf2, 0x0f, 0x10]),
+    );
+    e.enc_both(
+        load.bind(F64).bind_any(),
+        rec_fldDisp32.opcodes(vec![0xf2, 0x0f, 0x10]),
+    );
+
+    e.enc_both(
+        load_complex.bind(F64),
+        rec_fldWithIndex.opcodes(vec![0xf2, 0x0f, 0x10]),
+    );
+    e.enc_both(
+        load_complex.bind(F64),
+        rec_fldWithIndexDisp8.opcodes(vec![0xf2, 0x0f, 0x10]),
+    );
+    e.enc_both(
+        load_complex.bind(F64),
+        rec_fldWithIndexDisp32.opcodes(vec![0xf2, 0x0f, 0x10]),
+    );
+
+    e.enc_both(
+        store.bind(F32).bind_any(),
+        rec_fst.opcodes(vec![0xf3, 0x0f, 0x11]),
+    );
+    e.enc_both(
+        store.bind(F32).bind_any(),
+        rec_fstDisp8.opcodes(vec![0xf3, 0x0f, 0x11]),
+    );
+    e.enc_both(
+        store.bind(F32).bind_any(),
+        rec_fstDisp32.opcodes(vec![0xf3, 0x0f, 0x11]),
+    );
+
+    e.enc_both(
+        store_complex.bind(F32),
+        rec_fstWithIndex.opcodes(vec![0xf3, 0x0f, 0x11]),
+    );
+    e.enc_both(
+        store_complex.bind(F32),
+        rec_fstWithIndexDisp8.opcodes(vec![0xf3, 0x0f, 0x11]),
+    );
+    e.enc_both(
+        store_complex.bind(F32),
+        rec_fstWithIndexDisp32.opcodes(vec![0xf3, 0x0f, 0x11]),
+    );
+
+    e.enc_both(
+        store.bind(F64).bind_any(),
+        rec_fst.opcodes(vec![0xf2, 0x0f, 0x11]),
+    );
+    e.enc_both(
+        store.bind(F64).bind_any(),
+        rec_fstDisp8.opcodes(vec![0xf2, 0x0f, 0x11]),
+    );
+    e.enc_both(
+        store.bind(F64).bind_any(),
+        rec_fstDisp32.opcodes(vec![0xf2, 0x0f, 0x11]),
+    );
+
+    e.enc_both(
+        store_complex.bind(F64),
+        rec_fstWithIndex.opcodes(vec![0xf2, 0x0f, 0x11]),
+    );
+    e.enc_both(
+        store_complex.bind(F64),
+        rec_fstWithIndexDisp8.opcodes(vec![0xf2, 0x0f, 0x11]),
+    );
+    e.enc_both(
+        store_complex.bind(F64),
+        rec_fstWithIndexDisp32.opcodes(vec![0xf2, 0x0f, 0x11]),
+    );
+
+    e.enc_both(
+        fill.bind(F32),
+        rec_ffillSib32.opcodes(vec![0xf3, 0x0f, 0x10]),
+    );
+    e.enc_both(
+        regfill.bind(F32),
+        rec_fregfill32.opcodes(vec![0xf3, 0x0f, 0x10]),
+    );
+    e.enc_both(
+        fill.bind(F64),
+        rec_ffillSib32.opcodes(vec![0xf2, 0x0f, 0x10]),
+    );
+    e.enc_both(
+        regfill.bind(F64),
+        rec_fregfill32.opcodes(vec![0xf2, 0x0f, 0x10]),
+    );
+
+    e.enc_both(
+        spill.bind(F32),
+        rec_fspillSib32.opcodes(vec![0xf3, 0x0f, 0x11]),
+    );
+    e.enc_both(
+        regspill.bind(F32),
+        rec_fregspill32.opcodes(vec![0xf3, 0x0f, 0x11]),
+    );
+    e.enc_both(
+        spill.bind(F64),
+        rec_fspillSib32.opcodes(vec![0xf2, 0x0f, 0x11]),
+    );
+    e.enc_both(
+        regspill.bind(F64),
+        rec_fregspill32.opcodes(vec![0xf2, 0x0f, 0x11]),
+    );
+
+    // Function addresses.
+
+    // Non-PIC, all-ones funcaddresses.
+    e.enc32_isap(
+        func_addr.bind(I32),
+        rec_fnaddr4.opcodes(vec![0xb8]),
+        not_all_ones_funcaddrs_and_not_is_pic,
+    );
+    e.enc64_isap(
+        func_addr.bind(I64),
+        rec_fnaddr8.opcodes(vec![0xb8]).rex().w(),
+        not_all_ones_funcaddrs_and_not_is_pic,
+    );
+
+    // Non-PIC, all-zeros funcaddresses.
+    e.enc32_isap(
+        func_addr.bind(I32),
+        rec_allones_fnaddr4.opcodes(vec![0xb8]),
+        all_ones_funcaddrs_and_not_is_pic,
+    );
+    e.enc64_isap(
+        func_addr.bind(I64),
+        rec_allones_fnaddr8.opcodes(vec![0xb8]).rex().w(),
+        all_ones_funcaddrs_and_not_is_pic,
+    );
+
+    // 64-bit, colocated, both PIC and non-PIC. Use the lea instruction's pc-relative field.
+    let f_func_addr = formats.get(formats.by_name("FuncAddr"));
+    let is_colocated_func = InstructionPredicate::new_is_colocated_func(f_func_addr, "func_ref");
+    e.enc64_instp(
+        func_addr.bind(I64),
+        rec_pcrel_fnaddr8.opcodes(vec![0x8d]).rex().w(),
+        is_colocated_func,
+    );
+
+    // 64-bit, non-colocated, PIC.
+    e.enc64_isap(
+        func_addr.bind(I64),
+        rec_got_fnaddr8.opcodes(vec![0x8b]).rex().w(),
+        is_pic,
+    );
+
+    // Global addresses.
+
+    // Non-PIC.
+    e.enc32_isap(
+        symbol_value.bind(I32),
+        rec_gvaddr4.opcodes(vec![0xb8]),
+        not_is_pic,
+    );
+    e.enc64_isap(
+        symbol_value.bind(I64),
+        rec_gvaddr8.opcodes(vec![0xb8]).rex().w(),
+        not_is_pic,
+    );
+
+    // PIC, colocated.
+    e.enc64_func(
+        symbol_value.bind(I64),
+        rec_pcrel_gvaddr8.opcodes(vec![0x8d]).rex().w(),
+        |encoding| {
+            encoding
+                .isa_predicate(is_pic)
+                .inst_predicate(InstructionPredicate::new_is_colocated_data(formats))
+        },
+    );
+
+    // PIC, non-colocated.
+    e.enc64_isap(
+        symbol_value.bind(I64),
+        rec_got_gvaddr8.opcodes(vec![0x8b]).rex().w(),
+        is_pic,
+    );
+
+    // Stack addresses.
+    //
+    // TODO: Add encoding rules for stack_load and stack_store, so that they
+    // don't get legalized to stack_addr + load/store.
+    e.enc32(stack_addr.bind(I32), rec_spaddr4_id.opcodes(vec![0x8d]));
+    e.enc64(
+        stack_addr.bind(I64),
+        rec_spaddr8_id.opcodes(vec![0x8d]).rex().w(),
+    );
+
+    // Call/return
+
+    // 32-bit, both PIC and non-PIC.
+    e.enc32(call, rec_call_id.opcodes(vec![0xe8]));
+
+    // 64-bit, colocated, both PIC and non-PIC. Use the call instruction's pc-relative field.
+    let f_call = formats.get(formats.by_name("Call"));
+    let is_colocated_func = InstructionPredicate::new_is_colocated_func(f_call, "func_ref");
+    e.enc64_instp(call, rec_call_id.opcodes(vec![0xe8]), is_colocated_func);
+
+    // 64-bit, non-colocated, PIC. There is no 64-bit non-colocated non-PIC version, since non-PIC
+    // is currently using the large model, which requires calls be lowered to
+    // func_addr+call_indirect.
+    e.enc64_isap(call, rec_call_plt_id.opcodes(vec![0xe8]), is_pic);
+
+    e.enc32(
+        call_indirect.bind(I32),
+        rec_call_r.opcodes(vec![0xff]).rrr(2),
+    );
+    e.enc64(
+        call_indirect.bind(I64),
+        rec_call_r.opcodes(vec![0xff]).rrr(2).rex(),
+    );
+    e.enc64(
+        call_indirect.bind(I64),
+        rec_call_r.opcodes(vec![0xff]).rrr(2),
+    );
+
+    e.enc32(return_, rec_ret.opcodes(vec![0xc3]));
+    e.enc64(return_, rec_ret.opcodes(vec![0xc3]));
+
+    // Branches.
+    e.enc32(jump, rec_jmpb.opcodes(vec![0xeb]));
+    e.enc64(jump, rec_jmpb.opcodes(vec![0xeb]));
+    e.enc32(jump, rec_jmpd.opcodes(vec![0xe9]));
+    e.enc64(jump, rec_jmpd.opcodes(vec![0xe9]));
+
+    e.enc_both(brif, rec_brib.opcodes(vec![0x70]));
+    e.enc_both(brif, rec_brid.opcodes(vec![0x0f, 0x80]));
+
+    // Not all float condition codes are legal, see `supported_floatccs`.
+    e.enc_both(brff, rec_brfb.opcodes(vec![0x70]));
+    e.enc_both(brff, rec_brfd.opcodes(vec![0x0f, 0x80]));
+
+    // Note that the tjccd opcode will be prefixed with 0x0f.
+    e.enc_i32_i64(brz, rec_tjccb.opcodes(vec![0x74]));
+    e.enc_i32_i64(brz, rec_tjccd.opcodes(vec![0x84]));
+    e.enc_i32_i64(brnz, rec_tjccb.opcodes(vec![0x75]));
+    e.enc_i32_i64(brnz, rec_tjccd.opcodes(vec![0x85]));
+
+    // Branch on a b1 value in a register only looks at the low 8 bits. See also
+    // bint encodings below.
+    //
+    // Start with the worst-case encoding for X86_32 only. The register allocator
+    // can't handle a branch with an ABCD-constrained operand.
+    e.enc32(brz.bind(B1), rec_t8jccd_long.opcodes(vec![0x84]));
+    e.enc32(brnz.bind(B1), rec_t8jccd_long.opcodes(vec![0x85]));
+
+    e.enc_both(brz.bind(B1), rec_t8jccb_abcd.opcodes(vec![0x74]));
+    e.enc_both(brz.bind(B1), rec_t8jccd_abcd.opcodes(vec![0x84]));
+    e.enc_both(brnz.bind(B1), rec_t8jccb_abcd.opcodes(vec![0x75]));
+    e.enc_both(brnz.bind(B1), rec_t8jccd_abcd.opcodes(vec![0x85]));
+
+    // Jump tables.
+    e.enc64(
+        jump_table_entry.bind(I64).bind_any().bind_any(),
+        rec_jt_entry.opcodes(vec![0x63]).rex().w(),
+    );
+    e.enc32(
+        jump_table_entry.bind(I32).bind_any().bind_any(),
+        rec_jt_entry.opcodes(vec![0x8b]),
+    );
+
+    e.enc64(
+        jump_table_base.bind(I64),
+        rec_jt_base.opcodes(vec![0x8d]).rex().w(),
+    );
+    e.enc32(jump_table_base.bind(I32), rec_jt_base.opcodes(vec![0x8d]));
+
+    e.enc_x86_64(
+        indirect_jump_table_br.bind(I64),
+        rec_indirect_jmp.opcodes(vec![0xff]).rrr(4),
+    );
+    e.enc32(
+        indirect_jump_table_br.bind(I32),
+        rec_indirect_jmp.opcodes(vec![0xff]).rrr(4),
+    );
+
+    // Trap as ud2
+    e.enc32(trap, rec_trap.opcodes(vec![0x0f, 0x0b]));
+    e.enc64(trap, rec_trap.opcodes(vec![0x0f, 0x0b]));
+
+    // Debug trap as int3
+    e.enc32_rec(debugtrap, rec_debugtrap, 0);
+    e.enc64_rec(debugtrap, rec_debugtrap, 0);
+
+    e.enc32_rec(trapif, rec_trapif, 0);
+    e.enc64_rec(trapif, rec_trapif, 0);
+    e.enc32_rec(trapff, rec_trapff, 0);
+    e.enc64_rec(trapff, rec_trapff, 0);
+
+    // Comparisons
+    e.enc_i32_i64(icmp, rec_icscc.opcodes(vec![0x39]));
+    e.enc_i32_i64(icmp_imm, rec_icscc_ib.opcodes(vec![0x83]).rrr(7));
+    e.enc_i32_i64(icmp_imm, rec_icscc_id.opcodes(vec![0x81]).rrr(7));
+    e.enc_i32_i64(ifcmp, rec_rcmp.opcodes(vec![0x39]));
+    e.enc_i32_i64(ifcmp_imm, rec_rcmp_ib.opcodes(vec![0x83]).rrr(7));
+    e.enc_i32_i64(ifcmp_imm, rec_rcmp_id.opcodes(vec![0x81]).rrr(7));
+    // TODO: We could special-case ifcmp_imm(x, 0) to TEST(x, x).
+
+    e.enc32(ifcmp_sp.bind(I32), rec_rcmp_sp.opcodes(vec![0x39]));
+    e.enc64(
+        ifcmp_sp.bind(I64),
+        rec_rcmp_sp.opcodes(vec![0x39]).rex().w(),
+    );
+
+    // Convert flags to bool.
+    // This encodes `b1` as an 8-bit low register with the value 0 or 1.
+    e.enc_both(trueif, rec_seti_abcd.opcodes(vec![0x0f, 0x90]));
+    e.enc_both(trueff, rec_setf_abcd.opcodes(vec![0x0f, 0x90]));
+
+    // Conditional move (a.k.a integer select).
+    e.enc_i32_i64(selectif, rec_cmov.opcodes(vec![0x0f, 0x40]));
+
+    // Bit scan forwards and reverse
+    e.enc_i32_i64(x86_bsf, rec_bsf_and_bsr.opcodes(vec![0x0f, 0xbc]));
+    e.enc_i32_i64(x86_bsr, rec_bsf_and_bsr.opcodes(vec![0x0f, 0xbd]));
+
+    // Convert bool to int.
+    //
+    // This assumes that b1 is represented as an 8-bit low register with the value 0
+    // or 1.
+    //
+    // Encode movzbq as movzbl, because it's equivalent and shorter.
+    e.enc32(
+        bint.bind(I32).bind(B1),
+        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xb6]),
+    );
+
+    e.enc64(
+        bint.bind(I64).bind(B1),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xb6]).rex(),
+    );
+    e.enc64(
+        bint.bind(I64).bind(B1),
+        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xb6]),
+    );
+    e.enc64(
+        bint.bind(I32).bind(B1),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xb6]).rex(),
+    );
+    e.enc64(
+        bint.bind(I32).bind(B1),
+        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xb6]),
+    );
+
+    // Numerical conversions.
+
+    // Reducing an integer is a no-op.
+    e.enc32_rec(ireduce.bind(I8).bind(I16), rec_null, 0);
+    e.enc32_rec(ireduce.bind(I8).bind(I32), rec_null, 0);
+    e.enc32_rec(ireduce.bind(I16).bind(I32), rec_null, 0);
+
+    e.enc64_rec(ireduce.bind(I8).bind(I16), rec_null, 0);
+    e.enc64_rec(ireduce.bind(I8).bind(I32), rec_null, 0);
+    e.enc64_rec(ireduce.bind(I16).bind(I32), rec_null, 0);
+    e.enc64_rec(ireduce.bind(I8).bind(I64), rec_null, 0);
+    e.enc64_rec(ireduce.bind(I16).bind(I64), rec_null, 0);
+    e.enc64_rec(ireduce.bind(I32).bind(I64), rec_null, 0);
+
+    // TODO: Add encodings for cbw, cwde, cdqe, which are sign-extending
+    // instructions for %al/%ax/%eax to %ax/%eax/%rax.
+
+    // movsbl
+    e.enc32(
+        sextend.bind(I32).bind(I8),
+        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xbe]),
+    );
+    e.enc64(
+        sextend.bind(I32).bind(I8),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xbe]).rex(),
+    );
+    e.enc64(
+        sextend.bind(I32).bind(I8),
+        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xbe]),
+    );
+
+    // movswl
+    e.enc32(
+        sextend.bind(I32).bind(I16),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xbf]),
+    );
+    e.enc64(
+        sextend.bind(I32).bind(I16),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xbf]).rex(),
+    );
+    e.enc64(
+        sextend.bind(I32).bind(I16),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xbf]),
+    );
+
+    // movsbq
+    e.enc64(
+        sextend.bind(I64).bind(I8),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xbe]).rex().w(),
+    );
+
+    // movswq
+    e.enc64(
+        sextend.bind(I64).bind(I16),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xbf]).rex().w(),
+    );
+
+    // movslq
+    e.enc64(
+        sextend.bind(I64).bind(I32),
+        rec_urm_noflags.opcodes(vec![0x63]).rex().w(),
+    );
+
+    // movzbl
+    e.enc32(
+        uextend.bind(I32).bind(I8),
+        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xb6]),
+    );
+    e.enc64(
+        uextend.bind(I32).bind(I8),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xb6]).rex(),
+    );
+    e.enc64(
+        uextend.bind(I32).bind(I8),
+        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xb6]),
+    );
+
+    // movzwl
+    e.enc32(
+        uextend.bind(I32).bind(I16),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xb7]),
+    );
+    e.enc64(
+        uextend.bind(I32).bind(I16),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xb7]).rex(),
+    );
+    e.enc64(
+        uextend.bind(I32).bind(I16),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xb7]),
+    );
+
+    // movzbq, encoded as movzbl because it's equivalent and shorter.
+    e.enc64(
+        uextend.bind(I64).bind(I8),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xb6]).rex(),
+    );
+    e.enc64(
+        uextend.bind(I64).bind(I8),
+        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xb6]),
+    );
+
+    // movzwq, encoded as movzwl because it's equivalent and shorter
+    e.enc64(
+        uextend.bind(I64).bind(I16),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xb7]).rex(),
+    );
+    e.enc64(
+        uextend.bind(I64).bind(I16),
+        rec_urm_noflags.opcodes(vec![0x0f, 0xb7]),
+    );
+
+    // A 32-bit register copy clears the high 32 bits.
+    e.enc64(
+        uextend.bind(I64).bind(I32),
+        rec_umr.opcodes(vec![0x89]).rex(),
+    );
+    e.enc64(uextend.bind(I64).bind(I32), rec_umr.opcodes(vec![0x89]));
+
+    // Floating point
+
+    // Floating-point constants equal to 0.0 can be encoded using either `xorps` or `xorpd`, for
+    // 32-bit and 64-bit floats respectively.
+    let f_unary_ieee32 = formats.get(formats.by_name("UnaryIeee32"));
+    let is_zero_32_bit_float = InstructionPredicate::new_is_zero_32bit_float(f_unary_ieee32, "imm");
+    e.enc32_instp(
+        f32const,
+        rec_f32imm_z.opcodes(vec![0x0f, 0x57]),
+        is_zero_32_bit_float.clone(),
+    );
+
+    let f_unary_ieee64 = formats.get(formats.by_name("UnaryIeee64"));
+    let is_zero_64_bit_float = InstructionPredicate::new_is_zero_64bit_float(f_unary_ieee64, "imm");
+    e.enc32_instp(
+        f64const,
+        rec_f64imm_z.opcodes(vec![0x66, 0x0f, 0x57]),
+        is_zero_64_bit_float.clone(),
+    );
+
+    e.enc_x86_64_instp(
+        f32const,
+        rec_f32imm_z.opcodes(vec![0x0f, 0x57]),
+        is_zero_32_bit_float,
+    );
+    e.enc_x86_64_instp(
+        f64const,
+        rec_f64imm_z.opcodes(vec![0x66, 0x0f, 0x57]),
+        is_zero_64_bit_float,
+    );
+
+    // movd
+    e.enc_both(
+        bitcast.bind(F32).bind(I32),
+        rec_frurm.opcodes(vec![0x66, 0x0f, 0x6e]),
+    );
+    e.enc_both(
+        bitcast.bind(I32).bind(F32),
+        rec_rfumr.opcodes(vec![0x66, 0x0f, 0x7e]),
+    );
+
+    // movq
+    e.enc64(
+        bitcast.bind(F64).bind(I64),
+        rec_frurm.opcodes(vec![0x66, 0x0f, 0x6e]).rex().w(),
+    );
+    e.enc64(
+        bitcast.bind(I64).bind(F64),
+        rec_rfumr.opcodes(vec![0x66, 0x0f, 0x7e]).rex().w(),
+    );
+
+    // movaps
+    e.enc_both(copy.bind(F32), rec_furm.opcodes(vec![0x0f, 0x28]));
+    e.enc_both(copy.bind(F64), rec_furm.opcodes(vec![0x0f, 0x28]));
+
+    // TODO For x86-64, only define REX forms for now, since we can't describe the special regunit
+    // immediate operands with the current constraint language.
+    e.enc32(regmove.bind(F32), rec_frmov.opcodes(vec![0x0f, 0x28]));
+    e.enc64(regmove.bind(F32), rec_frmov.opcodes(vec![0x0f, 0x28]).rex());
+
+    // TODO For x86-64, only define REX forms for now, since we can't describe the special regunit
+    // immediate operands with the current constraint language.
+    e.enc32(regmove.bind(F64), rec_frmov.opcodes(vec![0x0f, 0x28]));
+    e.enc64(regmove.bind(F64), rec_frmov.opcodes(vec![0x0f, 0x28]).rex());
+
+    // cvtsi2ss
+    e.enc_i32_i64(
+        fcvt_from_sint.bind(F32),
+        rec_frurm.opcodes(vec![0xf3, 0x0f, 0x2a]),
+    );
+
+    // cvtsi2sd
+    e.enc_i32_i64(
+        fcvt_from_sint.bind(F64),
+        rec_frurm.opcodes(vec![0xf2, 0x0f, 0x2a]),
+    );
+
+    // cvtss2sd
+    e.enc_both(
+        fpromote.bind(F64).bind(F32),
+        rec_furm.opcodes(vec![0xf3, 0x0f, 0x5a]),
+    );
+
+    // cvtsd2ss
+    e.enc_both(
+        fdemote.bind(F32).bind(F64),
+        rec_furm.opcodes(vec![0xf2, 0x0f, 0x5a]),
+    );
+
+    // cvttss2si
+    e.enc_both(
+        x86_cvtt2si.bind(I32).bind(F32),
+        rec_rfurm.opcodes(vec![0xf3, 0x0f, 0x2c]),
+    );
+    e.enc64(
+        x86_cvtt2si.bind(I64).bind(F32),
+        rec_rfurm.opcodes(vec![0xf3, 0x0f, 0x2c]).rex().w(),
+    );
+
+    // cvttsd2si
+    e.enc_both(
+        x86_cvtt2si.bind(I32).bind(F64),
+        rec_rfurm.opcodes(vec![0xf2, 0x0f, 0x2c]),
+    );
+    e.enc64(
+        x86_cvtt2si.bind(I64).bind(F64),
+        rec_rfurm.opcodes(vec![0xf2, 0x0f, 0x2c]).rex().w(),
+    );
+
+    // Exact square roots.
+    e.enc_both(sqrt.bind(F32), rec_furm.opcodes(vec![0xf3, 0x0f, 0x51]));
+    e.enc_both(sqrt.bind(F64), rec_furm.opcodes(vec![0xf2, 0x0f, 0x51]));
+
+    // Rounding. The recipe looks at the opcode to pick an immediate.
+    for inst in &[nearest, floor, ceil, trunc] {
+        e.enc_both_isap(
+            inst.bind(F32),
+            rec_furmi_rnd.opcodes(vec![0x66, 0x0f, 0x3a, 0x0a]),
+            use_sse41,
+        );
+        e.enc_both_isap(
+            inst.bind(F64),
+            rec_furmi_rnd.opcodes(vec![0x66, 0x0f, 0x3a, 0x0b]),
+            use_sse41,
+        );
+    }
+
+    // Binary arithmetic ops.
+    for &(inst, opc) in &[
+        (fadd, 0x58),
+        (fsub, 0x5c),
+        (fmul, 0x59),
+        (fdiv, 0x5e),
+        (x86_fmin, 0x5d),
+        (x86_fmax, 0x5f),
+    ] {
+        e.enc_both(inst.bind(F32), rec_fa.opcodes(vec![0xf3, 0x0f, opc]));
+        e.enc_both(inst.bind(F64), rec_fa.opcodes(vec![0xf2, 0x0f, opc]));
+    }
+
+    // Binary bitwise ops.
+    for &(inst, opc) in &[(band, 0x54), (bor, 0x56), (bxor, 0x57)] {
+        e.enc_both(inst.bind(F32), rec_fa.opcodes(vec![0x0f, opc]));
+        e.enc_both(inst.bind(F64), rec_fa.opcodes(vec![0x0f, opc]));
+    }
+
+    // The `andnps(x,y)` instruction computes `~x&y`, while band_not(x,y)` is `x&~y.
+    e.enc_both(band_not.bind(F32), rec_fax.opcodes(vec![0x0f, 0x55]));
+    e.enc_both(band_not.bind(F64), rec_fax.opcodes(vec![0x0f, 0x55]));
+
+    // Comparisons.
+    //
+    // This only covers the condition codes in `supported_floatccs`, the rest are
+    // handled by legalization patterns.
+    e.enc_both(fcmp.bind(F32), rec_fcscc.opcodes(vec![0x0f, 0x2e]));
+    e.enc_both(fcmp.bind(F64), rec_fcscc.opcodes(vec![0x66, 0x0f, 0x2e]));
+    e.enc_both(ffcmp.bind(F32), rec_fcmp.opcodes(vec![0x0f, 0x2e]));
+    e.enc_both(ffcmp.bind(F64), rec_fcmp.opcodes(vec![0x66, 0x0f, 0x2e]));
+
+    e
+}
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index 5407ec2e87..a8002890f7 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -1,15 +1,15 @@
 use crate::cdsl::cpu_modes::CpuMode;
-use crate::cdsl::instructions::InstructionPredicateMap;
 use crate::cdsl::isa::TargetIsa;
-use crate::cdsl::recipes::Recipes;
 
 use crate::shared::types::Bool::B1;
 use crate::shared::types::Float::{F32, F64};
 use crate::shared::types::Int::{I16, I32, I64, I8};
 use crate::shared::Definitions as SharedDefinitions;
 
+mod encodings;
 mod instructions;
 mod legalize;
+mod recipes;
 mod registers;
 mod settings;
 
@@ -51,12 +51,17 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     x86_64.legalize_type(F32, x86_expand);
     x86_64.legalize_type(F64, x86_expand);
 
+    let recipes = recipes::define(shared_defs, &settings, ®s);
+
+    let encodings = encodings::define(shared_defs, &settings, &inst_group, &recipes);
+    x86_32.set_encodings(encodings.enc32);
+    x86_64.set_encodings(encodings.enc64);
+    let encodings_predicates = encodings.inst_pred_reg.extract();
+
+    let recipes = encodings.recipes;
+
     let cpu_modes = vec![x86_64, x86_32];
 
-    let recipes = Recipes::new();
-
-    let encodings_predicates = InstructionPredicateMap::new();
-
     TargetIsa::new(
         "x86",
         inst_group,
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
new file mode 100644
index 0000000000..5c1d28ef20
--- /dev/null
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -0,0 +1,2805 @@
+use std::rc::Rc;
+
+use crate::cdsl::ast::Literal;
+use crate::cdsl::formats::{FormatRegistry, InstructionFormat};
+use crate::cdsl::instructions::InstructionPredicate;
+use crate::cdsl::recipes::{
+    EncodingRecipe, EncodingRecipeBuilder, OperandConstraint, Register, Stack,
+};
+use crate::cdsl::regs::IsaRegs;
+use crate::cdsl::settings::SettingGroup;
+use crate::shared::Definitions as SharedDefinitions;
+
+/// Helper data structure to create recipes and template recipes.
+/// It contains all the recipes and recipe templates that might be used in the encodings crate of
+/// this same directory.
+pub struct RecipeGroup<'builder> {
+    /// Memoized format pointer, to pass it to builders later.
+    formats: &'builder FormatRegistry,
+
+    /// Memoized registers description, to pass it to builders later.
+    regs: &'builder IsaRegs,
+
+    /// All the recipes explicitly created in this file. This is different from the final set of
+    /// recipes, which is definitive only once encodings have generated new recipes on the fly.
+    recipes: Vec,
+
+    /// All the recipe templates created in this file.
+    templates: Vec>>,
+}
+
+impl<'builder> RecipeGroup<'builder> {
+    fn new(formats: &'builder FormatRegistry, regs: &'builder IsaRegs) -> Self {
+        Self {
+            formats,
+            regs,
+            recipes: Vec::new(),
+            templates: Vec::new(),
+        }
+    }
+    fn add_recipe(&mut self, recipe: EncodingRecipeBuilder) {
+        self.recipes.push(recipe.build(self.formats));
+    }
+    fn add_template_recipe(&mut self, recipe: EncodingRecipeBuilder) -> Rc> {
+        let template = Rc::new(Template::new(recipe, self.formats, self.regs));
+        self.templates.push(template.clone());
+        template
+    }
+    fn add_template(&mut self, template: Template<'builder>) -> Rc> {
+        let template = Rc::new(template);
+        self.templates.push(template.clone());
+        template
+    }
+    pub fn recipe(&self, name: &str) -> &EncodingRecipe {
+        self.recipes
+            .iter()
+            .find(|recipe| recipe.name == name)
+            .expect(&format!("unknown recipe name: {}. Try template?", name))
+    }
+    pub fn template(&self, name: &str) -> &Template {
+        self.templates
+            .iter()
+            .find(|recipe| recipe.name() == name)
+            .expect(&format!("unknown tail recipe name: {}. Try recipe?", name))
+    }
+}
+
+// Opcode representation.
+//
+// Cranelift requires each recipe to have a single encoding size in bytes, and x86 opcodes are
+// variable length, so we use separate recipes for different styles of opcodes and prefixes. The
+// opcode format is indicated by the recipe name prefix.
+//
+// The match case below does not include the REX prefix which goes after the mandatory prefix.
+// VEX/XOP and EVEX prefixes are not yet supported. Encodings using any of these prefixes are
+// represented by separate recipes.
+//
+// The encoding bits are:
+//
+// 0-7:   The opcode byte .
+// 8-9:   pp, mandatory prefix:
+//        00 none (Op*)
+//        01 66   (Mp*)
+//        10 F3   (Mp*)
+//        11 F2   (Mp*)
+// 10-11: mm, opcode map:
+//        00         (Op1/Mp1)
+//        01 0F      (Op2/Mp2)
+//        10 0F 38   (Op3/Mp3)
+//        11 0F 3A   (Op3/Mp3)
+// 12-14  rrr, opcode bits for the ModR/M byte for certain opcodes.
+// 15:    REX.W bit (or VEX.W/E)
+//
+// There is some redundancy between bits 8-11 and the recipe names, but we have enough bits, and
+// the pp+mm format is ready for supporting VEX prefixes.
+//
+// TODO Cranelift doesn't actually require recipe to have different encoding sizes anymore, so this
+// could be simplified.
+
+/// Given a sequence of opcode bytes, compute the recipe name prefix and encoding bits.
+fn decode_opcodes(op_bytes: &[u8], rrr: u16, w: u16) -> (&'static str, u16) {
+    assert!(op_bytes.len() >= 1, "at least one opcode byte");
+
+    let prefix_bytes = &op_bytes[..op_bytes.len() - 1];
+    let (name, mmpp) = match prefix_bytes {
+        [] => ("Op1", 0b000),
+        [0x66] => ("Mp1", 0b0001),
+        [0xf3] => ("Mp1", 0b0010),
+        [0xf2] => ("Mp1", 0b0011),
+        [0x0f] => ("Op2", 0b0100),
+        [0x66, 0x0f] => ("Mp2", 0b0101),
+        [0xf3, 0x0f] => ("Mp2", 0b0110),
+        [0xf2, 0x0f] => ("Mp2", 0b0111),
+        [0x0f, 0x38] => ("Op3", 0b1000),
+        [0x66, 0x0f, 0x38] => ("Mp3", 0b1001),
+        [0xf3, 0x0f, 0x38] => ("Mp3", 0b1010),
+        [0xf2, 0x0f, 0x38] => ("Mp3", 0b1011),
+        [0x0f, 0x3a] => ("Op3", 0b1100),
+        [0x66, 0x0f, 0x3a] => ("Mp3", 0b1101),
+        [0xf3, 0x0f, 0x3a] => ("Mp3", 0b1110),
+        [0xf2, 0x0f, 0x3a] => ("Mp3", 0b1111),
+        _ => {
+            panic!("unexpected opcode sequence: {:?}", op_bytes);
+        }
+    };
+
+    let opcode_byte = op_bytes[op_bytes.len() - 1] as u16;
+    (name, opcode_byte | (mmpp << 8) | (rrr << 12) | w << 15)
+}
+
+/// Given a snippet of Rust code (or None), replace the `PUT_OP` macro with the
+/// corresponding `put_*` function from the `binemit.rs` module.
+fn replace_put_op(code: Option, prefix: &str) -> Option {
+    code.map(|code| code.replace("{{PUT_OP}}", &format!("put_{}", prefix.to_lowercase())))
+}
+
+/// Replaces constraints to a REX-prefixed register class by the equivalent non-REX register class.
+fn replace_nonrex_constraints(
+    regs: &IsaRegs,
+    constraints: Vec,
+) -> Vec {
+    constraints
+        .into_iter()
+        .map(|constraint| match constraint {
+            OperandConstraint::RegClass(rc_index) => {
+                let new_rc_index = if rc_index == regs.class_by_name("GPR") {
+                    regs.class_by_name("GPR8")
+                } else if rc_index == regs.class_by_name("FPR") {
+                    regs.class_by_name("FPR8")
+                } else {
+                    rc_index
+                };
+                OperandConstraint::RegClass(new_rc_index)
+            }
+            _ => constraint,
+        })
+        .collect()
+}
+
+/// Previously called a TailRecipe in the Python meta language, this allows to create multiple
+/// variants of a single base EncodingRecipe (rex prefix, specialized w/rrr bits, different
+/// opcodes). It serves as a prototype of an EncodingRecipe, which is then used when actually creating
+/// Encodings, in encodings.rs. This is an idiosyncrasy of the x86 meta-language, and could be
+/// reconsidered later.
+#[derive(Clone)]
+pub struct Template<'builder> {
+    /// Mapping of format indexes to format data, used in the build() method.
+    formats: &'builder FormatRegistry,
+
+    /// Description of registers, used in the build() method.
+    regs: &'builder IsaRegs,
+
+    /// The recipe template, which is to be specialized (by copy).
+    recipe: EncodingRecipeBuilder,
+
+    /// Does this recipe requires a REX prefix?
+    requires_prefix: bool,
+
+    /// Other recipe to use when REX-prefixed.
+    when_prefixed: Option>>,
+
+    // Specialized parameters.
+    /// Should we include the REX prefix?
+    rex: bool,
+    /// Value of the W bit (0 or 1).
+    w_bit: u16,
+    /// Value of the RRR bits (between 0 and 0b111).
+    rrr_bits: u16,
+    /// Opcode bytes.
+    op_bytes: Vec,
+}
+
+impl<'builder> Template<'builder> {
+    fn new(
+        recipe: EncodingRecipeBuilder,
+        formats: &'builder FormatRegistry,
+        regs: &'builder IsaRegs,
+    ) -> Self {
+        Self {
+            formats,
+            regs,
+            recipe,
+            requires_prefix: false,
+            when_prefixed: None,
+            rex: false,
+            w_bit: 0,
+            rrr_bits: 0,
+            op_bytes: Vec::new(),
+        }
+    }
+
+    fn name(&self) -> &str {
+        &self.recipe.name
+    }
+    fn requires_prefix(self, value: bool) -> Self {
+        Self {
+            requires_prefix: value,
+            ..self
+        }
+    }
+    fn when_prefixed(self, template: Rc>) -> Self {
+        assert!(self.when_prefixed.is_none());
+        Self {
+            when_prefixed: Some(template),
+            ..self
+        }
+    }
+
+    // Copy setters.
+    pub fn opcodes(&self, op_bytes: Vec) -> Self {
+        assert!(!op_bytes.is_empty());
+        let mut copy = self.clone();
+        copy.op_bytes = op_bytes;
+        copy
+    }
+    pub fn w(&self) -> Self {
+        let mut copy = self.clone();
+        copy.w_bit = 1;
+        copy
+    }
+    pub fn rrr(&self, value: u16) -> Self {
+        assert!(value <= 0b111);
+        let mut copy = self.clone();
+        copy.rrr_bits = value;
+        copy
+    }
+    pub fn nonrex(&self) -> Self {
+        assert!(!self.requires_prefix, "Tail recipe requires REX prefix.");
+        let mut copy = self.clone();
+        copy.rex = false;
+        copy
+    }
+    pub fn rex(&self) -> Self {
+        if let Some(prefixed) = &self.when_prefixed {
+            let mut ret = prefixed.rex();
+            // Forward specialized parameters.
+            ret.op_bytes = self.op_bytes.clone();
+            ret.w_bit = self.w_bit;
+            ret.rrr_bits = self.rrr_bits;
+            return ret;
+        }
+        let mut copy = self.clone();
+        copy.rex = true;
+        copy
+    }
+
+    pub fn build(mut self) -> (EncodingRecipe, u16) {
+        let (name, bits) = decode_opcodes(&self.op_bytes, self.rrr_bits, self.w_bit);
+
+        let (name, rex_prefix_size) = if self.rex {
+            ("Rex".to_string() + name, 1)
+        } else {
+            (name.into(), 0)
+        };
+
+        let size_addendum = self.op_bytes.len() as u64 + rex_prefix_size;
+        self.recipe.base_size += size_addendum;
+
+        // Branch ranges are relative to the end of the instruction.
+        self.recipe
+            .branch_range
+            .as_mut()
+            .map(|range| range.inst_size += size_addendum);
+
+        self.recipe.emit = replace_put_op(self.recipe.emit, &name);
+        self.recipe.name = name + &self.recipe.name;
+
+        if !self.rex {
+            let operands_in = self.recipe.operands_in.unwrap_or(Vec::new());
+            self.recipe.operands_in = Some(replace_nonrex_constraints(self.regs, operands_in));
+            let operands_out = self.recipe.operands_out.unwrap_or(Vec::new());
+            self.recipe.operands_out = Some(replace_nonrex_constraints(self.regs, operands_out));
+        }
+
+        (self.recipe.build(self.formats), bits)
+    }
+}
+
+/// Returns a predicate checking that the "cond" field of the instruction contains one of the
+/// directly supported floating point condition codes.
+fn supported_floatccs_predicate(
+    supported_cc: &[Literal],
+    format: &InstructionFormat,
+) -> InstructionPredicate {
+    supported_cc
+        .iter()
+        .fold(InstructionPredicate::new(), |pred, literal| {
+            pred.or(InstructionPredicate::new_is_field_equal(
+                format,
+                "cond",
+                literal.to_rust_code(),
+            ))
+        })
+}
+
+/// Return an instruction predicate that checks if `iform.imm` is a valid `scale` for a SIB byte.
+fn valid_scale(format: &InstructionFormat) -> InstructionPredicate {
+    ["1", "2", "4", "8"]
+        .iter()
+        .fold(InstructionPredicate::new(), |pred, &literal| {
+            pred.or(InstructionPredicate::new_is_field_equal(
+                format,
+                "imm",
+                literal.into(),
+            ))
+        })
+}
+
+pub fn define<'shared>(
+    shared_defs: &'shared SharedDefinitions,
+    settings: &'shared SettingGroup,
+    regs: &'shared IsaRegs,
+) -> RecipeGroup<'shared> {
+    // The set of floating point condition codes that are directly supported.
+    // Other condition codes need to be reversed or expressed as two tests.
+    let floatcc = shared_defs.operand_kinds.by_name("floatcc");
+    let supported_floatccs: Vec = ["ord", "uno", "one", "ueq", "gt", "ge", "ult", "ule"]
+        .iter()
+        .map(|name| Literal::enumerator_for(floatcc, name))
+        .collect();
+
+    let formats = &shared_defs.format_registry;
+
+    // Register classes shorthands.
+    let abcd = regs.class_by_name("ABCD");
+    let gpr = regs.class_by_name("GPR");
+    let fpr = regs.class_by_name("FPR");
+    let flag = regs.class_by_name("FLAG");
+
+    // Operand constraints shorthands.
+    let reg_rflags = Register::new(flag, regs.regunit_by_name(flag, "rflags"));
+    let reg_rax = Register::new(gpr, regs.regunit_by_name(gpr, "rax"));
+    let reg_rcx = Register::new(gpr, regs.regunit_by_name(gpr, "rcx"));
+    let reg_rdx = Register::new(gpr, regs.regunit_by_name(gpr, "rdx"));
+
+    // Stack operand with a 32-bit signed displacement from either RBP or RSP.
+    let stack_gpr32 = Stack::new(gpr);
+    let stack_fpr32 = Stack::new(fpr);
+
+    // Format shorthands, prefixed with f_.
+    let f_binary = formats.by_name("Binary");
+    let f_binary_imm = formats.by_name("BinaryImm");
+    let f_branch = formats.by_name("Branch");
+    let f_branch_float = formats.by_name("BranchFloat");
+    let f_branch_int = formats.by_name("BranchInt");
+    let f_branch_table_entry = formats.by_name("BranchTableEntry");
+    let f_branch_table_base = formats.by_name("BranchTableBase");
+    let f_call = formats.by_name("Call");
+    let f_call_indirect = formats.by_name("CallIndirect");
+    let f_copy_special = formats.by_name("CopySpecial");
+    let f_float_compare = formats.by_name("FloatCompare");
+    let f_float_cond = formats.by_name("FloatCond");
+    let f_float_cond_trap = formats.by_name("FloatCondTrap");
+    let f_func_addr = formats.by_name("FuncAddr");
+    let f_indirect_jump = formats.by_name("IndirectJump");
+    let f_int_compare = formats.by_name("IntCompare");
+    let f_int_compare_imm = formats.by_name("IntCompareImm");
+    let f_int_cond = formats.by_name("IntCond");
+    let f_int_cond_trap = formats.by_name("IntCondTrap");
+    let f_int_select = formats.by_name("IntSelect");
+    let f_jump = formats.by_name("Jump");
+    let f_load = formats.by_name("Load");
+    let f_load_complex = formats.by_name("LoadComplex");
+    let f_multiary = formats.by_name("MultiAry");
+    let f_nullary = formats.by_name("NullAry");
+    let f_reg_fill = formats.by_name("RegFill");
+    let f_reg_move = formats.by_name("RegMove");
+    let f_reg_spill = formats.by_name("RegSpill");
+    let f_stack_load = formats.by_name("StackLoad");
+    let f_store = formats.by_name("Store");
+    let f_store_complex = formats.by_name("StoreComplex");
+    let f_ternary = formats.by_name("Ternary");
+    let f_trap = formats.by_name("Trap");
+    let f_unary = formats.by_name("Unary");
+    let f_unary_bool = formats.by_name("UnaryBool");
+    let f_unary_global_value = formats.by_name("UnaryGlobalValue");
+    let f_unary_ieee32 = formats.by_name("UnaryIeee32");
+    let f_unary_ieee64 = formats.by_name("UnaryIeee64");
+    let f_unary_imm = formats.by_name("UnaryImm");
+
+    // Predicates shorthands.
+    let use_sse41 = settings.predicate_by_name("use_sse41");
+
+    // Definitions.
+    let mut recipes = RecipeGroup::new(formats, regs);
+
+    // A null unary instruction that takes a GPR register. Can be used for identity copies and
+    // no-op conversions.
+    recipes.add_recipe(
+        EncodingRecipeBuilder::new("null", f_unary, 0)
+            .operands_in(vec![gpr])
+            .operands_out(vec![0])
+            .emit(""),
+    );
+    recipes.add_recipe(
+        EncodingRecipeBuilder::new("stacknull", f_unary, 0)
+            .operands_in(vec![stack_gpr32])
+            .operands_out(vec![stack_gpr32])
+            .emit(""),
+    );
+
+    recipes
+        .add_recipe(EncodingRecipeBuilder::new("debugtrap", f_nullary, 1).emit("sink.put1(0xcc);"));
+
+    // XX opcode, no ModR/M.
+    recipes.add_template_recipe(EncodingRecipeBuilder::new("trap", f_trap, 0).emit(
+        r#"
+            sink.trap(code, func.srclocs[inst]);
+            {{PUT_OP}}(bits, BASE_REX, sink);
+        "#,
+    ));
+
+    // Macro: conditional jump over a ud2.
+    recipes.add_recipe(
+        EncodingRecipeBuilder::new("trapif", f_int_cond_trap, 4)
+            .operands_in(vec![reg_rflags])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    // Jump over a 2-byte ud2.
+                    sink.put1(0x70 | (icc2opc(cond.inverse()) as u8));
+                    sink.put1(2);
+                    // ud2.
+                    sink.trap(code, func.srclocs[inst]);
+                    sink.put1(0x0f);
+                    sink.put1(0x0b);
+                "#,
+            ),
+    );
+
+    recipes.add_recipe(
+        EncodingRecipeBuilder::new("trapff", f_float_cond_trap, 4)
+            .operands_in(vec![reg_rflags])
+            .clobbers_flags(false)
+            .inst_predicate(supported_floatccs_predicate(
+                &supported_floatccs,
+                formats.get(f_float_cond_trap),
+            ))
+            .emit(
+                r#"
+                    // Jump over a 2-byte ud2.
+                    sink.put1(0x70 | (fcc2opc(cond.inverse()) as u8));
+                    sink.put1(2);
+                    // ud2.
+                    sink.trap(code, func.srclocs[inst]);
+                    sink.put1(0x0f);
+                    sink.put1(0x0b);
+                "#,
+            ),
+    );
+
+    // XX /r
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("rr", f_binary, 1)
+            .operands_in(vec![gpr, gpr])
+            .operands_out(vec![0])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
+                    modrm_rr(in_reg0, in_reg1, sink);
+                "#,
+            ),
+    );
+
+    // XX /r with operands swapped. (RM form).
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("rrx", f_binary, 1)
+            .operands_in(vec![gpr, gpr])
+            .operands_out(vec![0])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                    modrm_rr(in_reg1, in_reg0, sink);
+                "#,
+            ),
+    );
+
+    // XX /r with FPR ins and outs. A form.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("fa", f_binary, 1)
+            .operands_in(vec![fpr, fpr])
+            .operands_out(vec![0])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                    modrm_rr(in_reg1, in_reg0, sink);
+                "#,
+            ),
+    );
+
+    // XX /r with FPR ins and outs. A form with input operands swapped.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("fax", f_binary, 1)
+            .operands_in(vec![fpr, fpr])
+            .operands_out(vec![1])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
+                    modrm_rr(in_reg0, in_reg1, sink);
+                "#,
+            ),
+    );
+
+    // XX /n for a unary operation with extension bits.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("ur", f_unary, 1)
+            .operands_in(vec![gpr])
+            .operands_out(vec![0])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                    modrm_r_bits(in_reg0, bits, sink);
+                "#,
+            ),
+    );
+
+    // XX /r, but for a unary operator with separate input/output register, like
+    // copies. MR form, preserving flags.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("umr", f_unary, 1)
+            .operands_in(vec![gpr])
+            .operands_out(vec![gpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(out_reg0, in_reg0), sink);
+                    modrm_rr(out_reg0, in_reg0, sink);
+                "#,
+            ),
+    );
+
+    // Same as umr, but with FPR -> GPR registers.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("rfumr", f_unary, 1)
+            .operands_in(vec![fpr])
+            .operands_out(vec![gpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(out_reg0, in_reg0), sink);
+                    modrm_rr(out_reg0, in_reg0, sink);
+                "#,
+            ),
+    );
+
+    // XX /r, but for a unary operator with separate input/output register.
+    // RM form. Clobbers FLAGS.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("urm", f_unary, 1)
+            .operands_in(vec![gpr])
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                    modrm_rr(in_reg0, out_reg0, sink);
+                "#,
+            ),
+    );
+
+    // XX /r. Same as urm, but doesn't clobber FLAGS.
+    let urm_noflags = recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("urm_noflags", f_unary, 1)
+            .operands_in(vec![gpr])
+            .operands_out(vec![gpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                    modrm_rr(in_reg0, out_reg0, sink);
+                "#,
+            ),
+    );
+
+    // XX /r. Same as urm_noflags, but input limited to ABCD.
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("urm_noflags_abcd", f_unary, 1)
+                .operands_in(vec![abcd])
+                .operands_out(vec![gpr])
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                    modrm_rr(in_reg0, out_reg0, sink);
+                "#,
+                ),
+            formats,
+            regs,
+        )
+        .when_prefixed(urm_noflags),
+    );
+
+    // XX /r, RM form, FPR -> FPR.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("furm", f_unary, 1)
+            .operands_in(vec![fpr])
+            .operands_out(vec![fpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                    modrm_rr(in_reg0, out_reg0, sink);
+                "#,
+            ),
+    );
+
+    // XX /r, RM form, GPR -> FPR.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("frurm", f_unary, 1)
+            .operands_in(vec![gpr])
+            .operands_out(vec![fpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                    modrm_rr(in_reg0, out_reg0, sink);
+                "#,
+            ),
+    );
+
+    // XX /r, RM form, FPR -> GPR.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("rfurm", f_unary, 1)
+            .operands_in(vec![fpr])
+            .operands_out(vec![gpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                    modrm_rr(in_reg0, out_reg0, sink);
+                "#,
+            ),
+    );
+
+    // XX /r, RMI form for one of the roundXX SSE 4.1 instructions.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("furmi_rnd", f_unary, 2)
+            .operands_in(vec![fpr])
+            .operands_out(vec![fpr])
+            .isa_predicate(use_sse41)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                    modrm_rr(in_reg0, out_reg0, sink);
+                    sink.put1(match opcode {
+                        Opcode::Nearest => 0b00,
+                        Opcode::Floor => 0b01,
+                        Opcode::Ceil => 0b10,
+                        Opcode::Trunc => 0b11,
+                        x => panic!("{} unexpected for furmi_rnd", opcode),
+                    });
+                "#,
+            ),
+    );
+
+    // XX /r, for regmove instructions.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("rmov", f_reg_move, 1)
+            .operands_in(vec![gpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(dst, src), sink);
+                    modrm_rr(dst, src, sink);
+                "#,
+            ),
+    );
+
+    // XX /r, for regmove instructions (FPR version, RM encoded).
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("frmov", f_reg_move, 1)
+            .operands_in(vec![fpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(src, dst), sink);
+                    modrm_rr(src, dst, sink);
+                "#,
+            ),
+    );
+
+    // XX /n with one arg in %rcx, for shifts.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("rc", f_binary, 1)
+            .operands_in(vec![
+                OperandConstraint::RegClass(gpr),
+                OperandConstraint::FixedReg(reg_rcx),
+            ])
+            .operands_out(vec![0])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                    modrm_r_bits(in_reg0, bits, sink);
+                "#,
+            ),
+    );
+
+    // XX /n for division: inputs in %rax, %rdx, r. Outputs in %rax, %rdx.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("div", f_ternary, 1)
+            .operands_in(vec![
+                OperandConstraint::FixedReg(reg_rax),
+                OperandConstraint::FixedReg(reg_rdx),
+                OperandConstraint::RegClass(gpr),
+            ])
+            .operands_out(vec![reg_rax, reg_rdx])
+            .emit(
+                r#"
+                    sink.trap(TrapCode::IntegerDivisionByZero, func.srclocs[inst]);
+                    {{PUT_OP}}(bits, rex1(in_reg2), sink);
+                    modrm_r_bits(in_reg2, bits, sink);
+                "#,
+            ),
+    );
+
+    // XX /n for {s,u}mulx: inputs in %rax, r. Outputs in %rdx(hi):%rax(lo)
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("mulx", f_binary, 1)
+            .operands_in(vec![
+                OperandConstraint::FixedReg(reg_rax),
+                OperandConstraint::RegClass(gpr),
+            ])
+            .operands_out(vec![
+                OperandConstraint::FixedReg(reg_rax),
+                OperandConstraint::FixedReg(reg_rdx),
+            ])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex1(in_reg1), sink);
+                    modrm_r_bits(in_reg1, bits, sink);
+                "#,
+            ),
+    );
+
+    // XX /n ib with 8-bit immediate sign-extended.
+    {
+        let format = formats.get(f_binary_imm);
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("r_ib", f_binary_imm, 2)
+                .operands_in(vec![gpr])
+                .operands_out(vec![0])
+                .inst_predicate(InstructionPredicate::new_is_signed_int(format, "imm", 8, 0))
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                        modrm_r_bits(in_reg0, bits, sink);
+                        let imm: i64 = imm.into();
+                        sink.put1(imm as u8);
+                    "#,
+                ),
+        );
+
+        // XX /n id with 32-bit immediate sign-extended.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("r_id", f_binary_imm, 5)
+                .operands_in(vec![gpr])
+                .operands_out(vec![0])
+                .inst_predicate(InstructionPredicate::new_is_signed_int(
+                    format, "imm", 32, 0,
+                ))
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                        modrm_r_bits(in_reg0, bits, sink);
+                        let imm: i64 = imm.into();
+                        sink.put4(imm as u32);
+                    "#,
+                ),
+        );
+    }
+
+    {
+        // XX /n id with 32-bit immediate sign-extended. UnaryImm version.
+        let format = formats.get(f_unary_imm);
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("u_id", f_unary_imm, 5)
+                .operands_out(vec![gpr])
+                .inst_predicate(InstructionPredicate::new_is_signed_int(
+                    format, "imm", 32, 0,
+                ))
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex1(out_reg0), sink);
+                        modrm_r_bits(out_reg0, bits, sink);
+                        let imm: i64 = imm.into();
+                        sink.put4(imm as u32);
+                    "#,
+                ),
+        );
+    }
+
+    // XX+rd id unary with 32-bit immediate. Note no recipe predicate.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("pu_id", f_unary_imm, 4)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    // The destination register is encoded in the low bits of the opcode.
+                    // No ModR/M.
+                    {{PUT_OP}}(bits | (out_reg0 & 7), rex1(out_reg0), sink);
+                    let imm: i64 = imm.into();
+                    sink.put4(imm as u32);
+                "#,
+            ),
+    );
+
+    // XX+rd id unary with bool immediate. Note no recipe predicate.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("pu_id_bool", f_unary_bool, 4)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    // The destination register is encoded in the low bits of the opcode.
+                    // No ModR/M.
+                    {{PUT_OP}}(bits | (out_reg0 & 7), rex1(out_reg0), sink);
+                    let imm: u32 = if imm { 1 } else { 0 };
+                    sink.put4(imm);
+                "#,
+            ),
+    );
+
+    // XX+rd iq unary with 64-bit immediate.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("pu_iq", f_unary_imm, 8)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits | (out_reg0 & 7), rex1(out_reg0), sink);
+                    let imm: i64 = imm.into();
+                    sink.put8(imm as u64);
+                "#,
+            ),
+    );
+
+    // XX /n Unary with floating point 32-bit immediate equal to zero.
+    {
+        let format = formats.get(f_unary_ieee32);
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("f32imm_z", f_unary_ieee32, 1)
+                .operands_out(vec![fpr])
+                .inst_predicate(InstructionPredicate::new_is_zero_32bit_float(format, "imm"))
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex2(out_reg0, out_reg0), sink);
+                        modrm_rr(out_reg0, out_reg0, sink);
+                    "#,
+                ),
+        );
+    }
+
+    // XX /n Unary with floating point 64-bit immediate equal to zero.
+    {
+        let format = formats.get(f_unary_ieee64);
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("f64imm_z", f_unary_ieee64, 1)
+                .operands_out(vec![fpr])
+                .inst_predicate(InstructionPredicate::new_is_zero_64bit_float(format, "imm"))
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex2(out_reg0, out_reg0), sink);
+                        modrm_rr(out_reg0, out_reg0, sink);
+                    "#,
+                ),
+        );
+    }
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("pushq", f_unary, 0)
+            .operands_in(vec![gpr])
+            .emit(
+                r#"
+                    sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
+                    {{PUT_OP}}(bits | (in_reg0 & 7), rex1(in_reg0), sink);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("popq", f_nullary, 0)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits | (out_reg0 & 7), rex1(out_reg0), sink);
+                "#,
+            ),
+    );
+
+    // XX /r, for regmove instructions.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("copysp", f_copy_special, 1)
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(dst, src), sink);
+                    modrm_rr(dst, src, sink);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("adjustsp", f_unary, 1)
+            .operands_in(vec![gpr])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(RU::rsp.into(), in_reg0), sink);
+                    modrm_rr(RU::rsp.into(), in_reg0, sink);
+                "#,
+            ),
+    );
+
+    {
+        let format = formats.get(f_unary_imm);
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("adjustsp_ib", f_unary_imm, 2)
+                .inst_predicate(InstructionPredicate::new_is_signed_int(format, "imm", 8, 0))
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex1(RU::rsp.into()), sink);
+                        modrm_r_bits(RU::rsp.into(), bits, sink);
+                        let imm: i64 = imm.into();
+                        sink.put1(imm as u8);
+                    "#,
+                ),
+        );
+
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("adjustsp_id", f_unary_imm, 5)
+                .inst_predicate(InstructionPredicate::new_is_signed_int(
+                    format, "imm", 32, 0,
+                ))
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex1(RU::rsp.into()), sink);
+                        modrm_r_bits(RU::rsp.into(), bits, sink);
+                        let imm: i64 = imm.into();
+                        sink.put4(imm as u32);
+                    "#,
+                ),
+        );
+    }
+
+    // XX+rd id with Abs4 function relocation.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("fnaddr4", f_func_addr, 4)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits | (out_reg0 & 7), rex1(out_reg0), sink);
+                    sink.reloc_external(Reloc::Abs4,
+                                        &func.dfg.ext_funcs[func_ref].name,
+                                        0);
+                    sink.put4(0);
+                "#,
+            ),
+    );
+
+    // XX+rd iq with Abs8 function relocation.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("fnaddr8", f_func_addr, 8)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits | (out_reg0 & 7), rex1(out_reg0), sink);
+                    sink.reloc_external(Reloc::Abs8,
+                                        &func.dfg.ext_funcs[func_ref].name,
+                                        0);
+                    sink.put8(0);
+                "#,
+            ),
+    );
+
+    // Similar to fnaddr4, but writes !0 (this is used by BaldrMonkey).
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("allones_fnaddr4", f_func_addr, 4)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits | (out_reg0 & 7), rex1(out_reg0), sink);
+                    sink.reloc_external(Reloc::Abs4,
+                                        &func.dfg.ext_funcs[func_ref].name,
+                                        0);
+                    // Write the immediate as `!0` for the benefit of BaldrMonkey.
+                    sink.put4(!0);
+                "#,
+            ),
+    );
+
+    // Similar to fnaddr8, but writes !0 (this is used by BaldrMonkey).
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("allones_fnaddr8", f_func_addr, 8)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits | (out_reg0 & 7), rex1(out_reg0), sink);
+                    sink.reloc_external(Reloc::Abs8,
+                                        &func.dfg.ext_funcs[func_ref].name,
+                                        0);
+                    // Write the immediate as `!0` for the benefit of BaldrMonkey.
+                    sink.put8(!0);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("pcrel_fnaddr8", f_func_addr, 5)
+            .operands_out(vec![gpr])
+            // rex2 gets passed 0 for r/m register because the upper bit of
+            // r/m doesn't get decoded when in rip-relative addressing mode.
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(0, out_reg0), sink);
+                    modrm_riprel(out_reg0, sink);
+                    // The addend adjusts for the difference between the end of the
+                    // instruction and the beginning of the immediate field.
+                    sink.reloc_external(Reloc::X86PCRel4,
+                                        &func.dfg.ext_funcs[func_ref].name,
+                                        -4);
+                    sink.put4(0);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("got_fnaddr8", f_func_addr, 5)
+            .operands_out(vec![gpr])
+            // rex2 gets passed 0 for r/m register because the upper bit of
+            // r/m doesn't get decoded when in rip-relative addressing mode.
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(0, out_reg0), sink);
+                    modrm_riprel(out_reg0, sink);
+                    // The addend adjusts for the difference between the end of the
+                    // instruction and the beginning of the immediate field.
+                    sink.reloc_external(Reloc::X86GOTPCRel4,
+                                        &func.dfg.ext_funcs[func_ref].name,
+                                        -4);
+                    sink.put4(0);
+                "#,
+            ),
+    );
+
+    // XX+rd id with Abs4 globalsym relocation.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("gvaddr4", f_unary_global_value, 4)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits | (out_reg0 & 7), rex1(out_reg0), sink);
+                    sink.reloc_external(Reloc::Abs4,
+                                        &func.global_values[global_value].symbol_name(),
+                                        0);
+                    sink.put4(0);
+                "#,
+            ),
+    );
+
+    // XX+rd iq with Abs8 globalsym relocation.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("gvaddr8", f_unary_global_value, 8)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits | (out_reg0 & 7), rex1(out_reg0), sink);
+                    sink.reloc_external(Reloc::Abs8,
+                                        &func.global_values[global_value].symbol_name(),
+                                        0);
+                    sink.put8(0);
+                "#,
+            ),
+    );
+
+    // XX+rd iq with PCRel4 globalsym relocation.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("pcrel_gvaddr8", f_unary_global_value, 5)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(0, out_reg0), sink);
+                    modrm_rm(5, out_reg0, sink);
+                    // The addend adjusts for the difference between the end of the
+                    // instruction and the beginning of the immediate field.
+                    sink.reloc_external(Reloc::X86PCRel4,
+                                        &func.global_values[global_value].symbol_name(),
+                                        -4);
+                    sink.put4(0);
+                "#,
+            ),
+    );
+
+    // XX+rd iq with Abs8 globalsym relocation.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("got_gvaddr8", f_unary_global_value, 5)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(0, out_reg0), sink);
+                    modrm_rm(5, out_reg0, sink);
+                    // The addend adjusts for the difference between the end of the
+                    // instruction and the beginning of the immediate field.
+                    sink.reloc_external(Reloc::X86GOTPCRel4,
+                                        &func.global_values[global_value].symbol_name(),
+                                        -4);
+                    sink.put4(0);
+                "#,
+            ),
+    );
+
+    // Stack addresses.
+    //
+    // TODO Alternative forms for 8-bit immediates, when applicable.
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("spaddr4_id", f_stack_load, 6)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    let sp = StackRef::sp(stack_slot, &func.stack_slots);
+                    let base = stk_base(sp.base);
+                    {{PUT_OP}}(bits, rex2(out_reg0, base), sink);
+                    modrm_sib_disp8(out_reg0, sink);
+                    sib_noindex(base, sink);
+                    let imm : i32 = offset.into();
+                    sink.put4(sp.offset.checked_add(imm).unwrap() as u32);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("spaddr8_id", f_stack_load, 6)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    let sp = StackRef::sp(stack_slot, &func.stack_slots);
+                    let base = stk_base(sp.base);
+                    {{PUT_OP}}(bits, rex2(base, out_reg0), sink);
+                    modrm_sib_disp32(out_reg0, sink);
+                    sib_noindex(base, sink);
+                    let imm : i32 = offset.into();
+                    sink.put4(sp.offset.checked_add(imm).unwrap() as u32);
+                "#,
+            ),
+    );
+
+    // Store recipes.
+
+    {
+        // Simple stores.
+        let format = formats.get(f_store);
+
+        // A predicate asking if the offset is zero.
+        let has_no_offset = InstructionPredicate::new_is_field_equal(format, "offset", "0".into());
+
+        // XX /r register-indirect store with no offset.
+        let st = recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("st", f_store, 1)
+                .operands_in(vec![gpr, gpr])
+                .inst_predicate(has_no_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_sib_or_offset_for_in_reg_1")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                        if needs_sib_byte(in_reg1) {
+                            modrm_sib(in_reg0, sink);
+                            sib_noindex(in_reg1, sink);
+                        } else if needs_offset(in_reg1) {
+                            modrm_disp8(in_reg1, in_reg0, sink);
+                            sink.put1(0);
+                        } else {
+                            modrm_rm(in_reg1, in_reg0, sink);
+                        }
+                    "#,
+                ),
+        );
+
+        // XX /r register-indirect store with no offset.
+        // Only ABCD allowed for stored value. This is for byte stores with no REX.
+        recipes.add_template(
+            Template::new(
+                EncodingRecipeBuilder::new("st_abcd", f_store, 1)
+                    .operands_in(vec![abcd, gpr])
+                    .inst_predicate(has_no_offset.clone())
+                    .clobbers_flags(false)
+                    .compute_size("size_plus_maybe_sib_or_offset_for_in_reg_1")
+                    .emit(
+                        r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                        if needs_sib_byte(in_reg1) {
+                            modrm_sib(in_reg0, sink);
+                            sib_noindex(in_reg1, sink);
+                        } else if needs_offset(in_reg1) {
+                            modrm_disp8(in_reg1, in_reg0, sink);
+                            sink.put1(0);
+                        } else {
+                            modrm_rm(in_reg1, in_reg0, sink);
+                        }
+                    "#,
+                    ),
+                formats,
+                regs,
+            )
+            .when_prefixed(st),
+        );
+
+        // XX /r register-indirect store of FPR with no offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("fst", f_store, 1)
+                .operands_in(vec![fpr, gpr])
+                .inst_predicate(has_no_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_sib_or_offset_for_in_reg_1")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                        if needs_sib_byte(in_reg1) {
+                            modrm_sib(in_reg0, sink);
+                            sib_noindex(in_reg1, sink);
+                        } else if needs_offset(in_reg1) {
+                            modrm_disp8(in_reg1, in_reg0, sink);
+                            sink.put1(0);
+                        } else {
+                            modrm_rm(in_reg1, in_reg0, sink);
+                        }
+                    "#,
+                ),
+        );
+
+        let has_small_offset = InstructionPredicate::new_is_signed_int(format, "offset", 8, 0);
+
+        // XX /r register-indirect store with 8-bit offset.
+        let st_disp8 = recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("stDisp8", f_store, 2)
+                .operands_in(vec![gpr, gpr])
+                .inst_predicate(has_small_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_sib_for_in_reg_1")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                        if needs_sib_byte(in_reg1) {
+                            modrm_sib_disp8(in_reg0, sink);
+                            sib_noindex(in_reg1, sink);
+                        } else {
+                            modrm_disp8(in_reg1, in_reg0, sink);
+                        }
+                        let offset: i32 = offset.into();
+                        sink.put1(offset as u8);
+                    "#,
+                ),
+        );
+
+        // XX /r register-indirect store with 8-bit offset.
+        // Only ABCD allowed for stored value. This is for byte stores with no REX.
+        recipes.add_template(
+            Template::new(
+                EncodingRecipeBuilder::new("stDisp8_abcd", f_store, 2)
+                    .operands_in(vec![abcd, gpr])
+                    .inst_predicate(has_small_offset.clone())
+                    .clobbers_flags(false)
+                    .compute_size("size_plus_maybe_sib_for_in_reg_1")
+                    .emit(
+                        r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                        if needs_sib_byte(in_reg1) {
+                            modrm_sib_disp8(in_reg0, sink);
+                            sib_noindex(in_reg1, sink);
+                        } else {
+                            modrm_disp8(in_reg1, in_reg0, sink);
+                        }
+                        let offset: i32 = offset.into();
+                        sink.put1(offset as u8);
+                    "#,
+                    ),
+                formats,
+                regs,
+            )
+            .when_prefixed(st_disp8),
+        );
+
+        // XX /r register-indirect store with 8-bit offset of FPR.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("fstDisp8", f_store, 2)
+                .operands_in(vec![fpr, gpr])
+                .inst_predicate(has_small_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_sib_for_in_reg_1")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                        if needs_sib_byte(in_reg1) {
+                            modrm_sib_disp8(in_reg0, sink);
+                            sib_noindex(in_reg1, sink);
+                        } else {
+                            modrm_disp8(in_reg1, in_reg0, sink);
+                        }
+                        let offset: i32 = offset.into();
+                        sink.put1(offset as u8);
+                    "#,
+                ),
+        );
+
+        // XX /r register-indirect store with 32-bit offset.
+        let st_disp32 = recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("stDisp32", f_store, 5)
+                .operands_in(vec![gpr, gpr])
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_sib_for_in_reg_1")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                        if needs_sib_byte(in_reg1) {
+                            modrm_sib_disp32(in_reg0, sink);
+                            sib_noindex(in_reg1, sink);
+                        } else {
+                            modrm_disp32(in_reg1, in_reg0, sink);
+                        }
+                        let offset: i32 = offset.into();
+                        sink.put4(offset as u32);
+                    "#,
+                ),
+        );
+
+        // XX /r register-indirect store with 32-bit offset.
+        // Only ABCD allowed for stored value. This is for byte stores with no REX.
+        recipes.add_template(
+            Template::new(
+                EncodingRecipeBuilder::new("stDisp32_abcd", f_store, 5)
+                    .operands_in(vec![abcd, gpr])
+                    .clobbers_flags(false)
+                    .compute_size("size_plus_maybe_sib_for_in_reg_1")
+                    .emit(
+                        r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                        if needs_sib_byte(in_reg1) {
+                            modrm_sib_disp32(in_reg0, sink);
+                            sib_noindex(in_reg1, sink);
+                        } else {
+                            modrm_disp32(in_reg1, in_reg0, sink);
+                        }
+                        let offset: i32 = offset.into();
+                        sink.put4(offset as u32);
+                    "#,
+                    ),
+                formats,
+                regs,
+            )
+            .when_prefixed(st_disp32),
+        );
+
+        // XX /r register-indirect store with 32-bit offset of FPR.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("fstDisp32", f_store, 5)
+                .operands_in(vec![fpr, gpr])
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_sib_for_in_reg_1")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                        if needs_sib_byte(in_reg1) {
+                            modrm_sib_disp32(in_reg0, sink);
+                            sib_noindex(in_reg1, sink);
+                        } else {
+                            modrm_disp32(in_reg1, in_reg0, sink);
+                        }
+                        let offset: i32 = offset.into();
+                        sink.put4(offset as u32);
+                    "#,
+                ),
+        );
+    }
+
+    {
+        // Complex stores.
+        let format = formats.get(f_store_complex);
+
+        // A predicate asking if the offset is zero.
+        let has_no_offset = InstructionPredicate::new_is_field_equal(format, "offset", "0".into());
+
+        // XX /r register-indirect store with index and no offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("stWithIndex", f_store_complex, 2)
+                .operands_in(vec![gpr, gpr, gpr])
+                .inst_predicate(has_no_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_offset_for_in_reg_1")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
+                        // The else branch always inserts an SIB byte.
+                        if needs_offset(in_reg1) {
+                            modrm_sib_disp8(in_reg0, sink);
+                            sib(0, in_reg2, in_reg1, sink);
+                            sink.put1(0);
+                        } else {
+                            modrm_sib(in_reg0, sink);
+                            sib(0, in_reg2, in_reg1, sink);
+                        }
+                    "#,
+                ),
+        );
+
+        // XX /r register-indirect store with index and no offset.
+        // Only ABCD allowed for stored value. This is for byte stores with no REX.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("stWithIndex_abcd", f_store_complex, 2)
+                .operands_in(vec![abcd, gpr, gpr])
+                .inst_predicate(has_no_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_offset_for_in_reg_1")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
+                        // The else branch always inserts an SIB byte.
+                        if needs_offset(in_reg1) {
+                            modrm_sib_disp8(in_reg0, sink);
+                            sib(0, in_reg2, in_reg1, sink);
+                            sink.put1(0);
+                        } else {
+                            modrm_sib(in_reg0, sink);
+                            sib(0, in_reg2, in_reg1, sink);
+                        }
+                    "#,
+                ),
+        );
+
+        // XX /r register-indirect store with index and no offset of FPR.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("fstWithIndex", f_store_complex, 2)
+                .operands_in(vec![fpr, gpr, gpr])
+                .inst_predicate(has_no_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_offset_for_in_reg_1")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
+                        // The else branch always inserts an SIB byte.
+                        if needs_offset(in_reg1) {
+                            modrm_sib_disp8(in_reg0, sink);
+                            sib(0, in_reg2, in_reg1, sink);
+                            sink.put1(0);
+                        } else {
+                            modrm_sib(in_reg0, sink);
+                            sib(0, in_reg2, in_reg1, sink);
+                        }
+                    "#,
+                ),
+        );
+
+        let has_small_offset = InstructionPredicate::new_is_signed_int(format, "offset", 8, 0);
+
+        // XX /r register-indirect store with index and 8-bit offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("stWithIndexDisp8", f_store_complex, 3)
+                .operands_in(vec![gpr, gpr, gpr])
+                .inst_predicate(has_small_offset.clone())
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
+                        modrm_sib_disp8(in_reg0, sink);
+                        sib(0, in_reg2, in_reg1, sink);
+                        let offset: i32 = offset.into();
+                        sink.put1(offset as u8);
+                    "#,
+                ),
+        );
+
+        // XX /r register-indirect store with index and 8-bit offset.
+        // Only ABCD allowed for stored value. This is for byte stores with no REX.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("stWithIndexDisp8_abcd", f_store_complex, 3)
+                .operands_in(vec![abcd, gpr, gpr])
+                .inst_predicate(has_small_offset.clone())
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
+                        modrm_sib_disp8(in_reg0, sink);
+                        sib(0, in_reg2, in_reg1, sink);
+                        let offset: i32 = offset.into();
+                        sink.put1(offset as u8);
+                    "#,
+                ),
+        );
+
+        // XX /r register-indirect store with index and 8-bit offset of FPR.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("fstWithIndexDisp8", f_store_complex, 3)
+                .operands_in(vec![fpr, gpr, gpr])
+                .inst_predicate(has_small_offset.clone())
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
+                        modrm_sib_disp8(in_reg0, sink);
+                        sib(0, in_reg2, in_reg1, sink);
+                        let offset: i32 = offset.into();
+                        sink.put1(offset as u8);
+                    "#,
+                ),
+        );
+
+        let has_big_offset = InstructionPredicate::new_is_signed_int(format, "offset", 32, 0);
+
+        // XX /r register-indirect store with index and 32-bit offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("stWithIndexDisp32", f_store_complex, 6)
+                .operands_in(vec![gpr, gpr, gpr])
+                .inst_predicate(has_big_offset.clone())
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
+                        modrm_sib_disp32(in_reg0, sink);
+                        sib(0, in_reg2, in_reg1, sink);
+                        let offset: i32 = offset.into();
+                        sink.put4(offset as u32);
+                    "#,
+                ),
+        );
+
+        // XX /r register-indirect store with index and 32-bit offset.
+        // Only ABCD allowed for stored value. This is for byte stores with no REX.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("stWithIndexDisp32_abcd", f_store_complex, 6)
+                .operands_in(vec![abcd, gpr, gpr])
+                .inst_predicate(has_big_offset.clone())
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
+                        modrm_sib_disp32(in_reg0, sink);
+                        sib(0, in_reg2, in_reg1, sink);
+                        let offset: i32 = offset.into();
+                        sink.put4(offset as u32);
+                    "#,
+                ),
+        );
+
+        // XX /r register-indirect store with index and 32-bit offset of FPR.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("fstWithIndexDisp32", f_store_complex, 6)
+                .operands_in(vec![fpr, gpr, gpr])
+                .inst_predicate(has_big_offset.clone())
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
+                        modrm_sib_disp32(in_reg0, sink);
+                        sib(0, in_reg2, in_reg1, sink);
+                        let offset: i32 = offset.into();
+                        sink.put4(offset as u32);
+                    "#,
+                ),
+        );
+    }
+
+    // Unary spill with SIB and 32-bit displacement.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("spillSib32", f_unary, 6)
+            .operands_in(vec![gpr])
+            .operands_out(vec![stack_gpr32])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
+                    let base = stk_base(out_stk0.base);
+                    {{PUT_OP}}(bits, rex2(base, in_reg0), sink);
+                    modrm_sib_disp32(in_reg0, sink);
+                    sib_noindex(base, sink);
+                    sink.put4(out_stk0.offset as u32);
+                "#,
+            ),
+    );
+
+    // Like spillSib32, but targeting an FPR rather than a GPR.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("fspillSib32", f_unary, 6)
+            .operands_in(vec![fpr])
+            .operands_out(vec![stack_fpr32])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
+                    let base = stk_base(out_stk0.base);
+                    {{PUT_OP}}(bits, rex2(base, in_reg0), sink);
+                    modrm_sib_disp32(in_reg0, sink);
+                    sib_noindex(base, sink);
+                    sink.put4(out_stk0.offset as u32);
+                "#,
+            ),
+    );
+
+    // Regspill using RSP-relative addressing.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("regspill32", f_reg_spill, 6)
+            .operands_in(vec![gpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
+                    let dst = StackRef::sp(dst, &func.stack_slots);
+                    let base = stk_base(dst.base);
+                    {{PUT_OP}}(bits, rex2(base, src), sink);
+                    modrm_sib_disp32(src, sink);
+                    sib_noindex(base, sink);
+                    sink.put4(dst.offset as u32);
+                "#,
+            ),
+    );
+
+    // Like regspill32, but targeting an FPR rather than a GPR.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("fregspill32", f_reg_spill, 6)
+            .operands_in(vec![fpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
+                    let dst = StackRef::sp(dst, &func.stack_slots);
+                    let base = stk_base(dst.base);
+                    {{PUT_OP}}(bits, rex2(base, src), sink);
+                    modrm_sib_disp32(src, sink);
+                    sib_noindex(base, sink);
+                    sink.put4(dst.offset as u32);
+                "#,
+            ),
+    );
+
+    // Load recipes.
+
+    {
+        // Simple loads.
+        let format = formats.get(f_load);
+
+        // A predicate asking if the offset is zero.
+        let has_no_offset = InstructionPredicate::new_is_field_equal(format, "offset", "0".into());
+
+        // XX /r load with no offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("ld", f_load, 1)
+                .operands_in(vec![gpr])
+                .operands_out(vec![gpr])
+                .inst_predicate(has_no_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_sib_or_offset_for_in_reg_0")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                        if needs_sib_byte(in_reg0) {
+                            modrm_sib(out_reg0, sink);
+                            sib_noindex(in_reg0, sink);
+                        } else if needs_offset(in_reg0) {
+                            modrm_disp8(in_reg0, out_reg0, sink);
+                            sink.put1(0);
+                        } else {
+                            modrm_rm(in_reg0, out_reg0, sink);
+                        }
+                    "#,
+                ),
+        );
+
+        // XX /r float load with no offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("fld", f_load, 1)
+                .operands_in(vec![gpr])
+                .operands_out(vec![fpr])
+                .inst_predicate(has_no_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_sib_or_offset_for_in_reg_0")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                        if needs_sib_byte(in_reg0) {
+                            modrm_sib(out_reg0, sink);
+                            sib_noindex(in_reg0, sink);
+                        } else if needs_offset(in_reg0) {
+                            modrm_disp8(in_reg0, out_reg0, sink);
+                            sink.put1(0);
+                        } else {
+                            modrm_rm(in_reg0, out_reg0, sink);
+                        }
+                    "#,
+                ),
+        );
+
+        let has_small_offset = InstructionPredicate::new_is_signed_int(format, "offset", 8, 0);
+
+        // XX /r load with 8-bit offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("ldDisp8", f_load, 2)
+                .operands_in(vec![gpr])
+                .operands_out(vec![gpr])
+                .inst_predicate(has_small_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_sib_for_in_reg_0")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                        if needs_sib_byte(in_reg0) {
+                            modrm_sib_disp8(out_reg0, sink);
+                            sib_noindex(in_reg0, sink);
+                        } else {
+                            modrm_disp8(in_reg0, out_reg0, sink);
+                        }
+                        let offset: i32 = offset.into();
+                        sink.put1(offset as u8);
+                    "#,
+                ),
+        );
+
+        // XX /r float load with 8-bit offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("fldDisp8", f_load, 2)
+                .operands_in(vec![gpr])
+                .operands_out(vec![fpr])
+                .inst_predicate(has_small_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_sib_for_in_reg_0")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                        if needs_sib_byte(in_reg0) {
+                            modrm_sib_disp8(out_reg0, sink);
+                            sib_noindex(in_reg0, sink);
+                        } else {
+                            modrm_disp8(in_reg0, out_reg0, sink);
+                        }
+                        let offset: i32 = offset.into();
+                        sink.put1(offset as u8);
+                    "#,
+                ),
+        );
+
+        let has_big_offset = InstructionPredicate::new_is_signed_int(format, "offset", 32, 0);
+
+        // XX /r load with 32-bit offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("ldDisp32", f_load, 5)
+                .operands_in(vec![gpr])
+                .operands_out(vec![gpr])
+                .inst_predicate(has_big_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_sib_for_in_reg_0")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                        if needs_sib_byte(in_reg0) {
+                            modrm_sib_disp32(out_reg0, sink);
+                            sib_noindex(in_reg0, sink);
+                        } else {
+                            modrm_disp32(in_reg0, out_reg0, sink);
+                        }
+                        let offset: i32 = offset.into();
+                        sink.put4(offset as u32);
+                    "#,
+                ),
+        );
+
+        // XX /r float load with 32-bit offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("fldDisp32", f_load, 5)
+                .operands_in(vec![gpr])
+                .operands_out(vec![fpr])
+                .inst_predicate(has_big_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_sib_for_in_reg_0")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                        if needs_sib_byte(in_reg0) {
+                            modrm_sib_disp32(out_reg0, sink);
+                            sib_noindex(in_reg0, sink);
+                        } else {
+                            modrm_disp32(in_reg0, out_reg0, sink);
+                        }
+                        let offset: i32 = offset.into();
+                        sink.put4(offset as u32);
+                    "#,
+                ),
+        );
+    }
+
+    {
+        // Complex loads.
+        let format = formats.get(f_load_complex);
+
+        // A predicate asking if the offset is zero.
+        let has_no_offset = InstructionPredicate::new_is_field_equal(format, "offset", "0".into());
+
+        // XX /r load with index and no offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("ldWithIndex", f_load_complex, 2)
+                .operands_in(vec![gpr, gpr])
+                .operands_out(vec![gpr])
+                .inst_predicate(has_no_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_offset_for_in_reg_0")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
+                        // The else branch always inserts an SIB byte.
+                        if needs_offset(in_reg0) {
+                            modrm_sib_disp8(out_reg0, sink);
+                            sib(0, in_reg1, in_reg0, sink);
+                            sink.put1(0);
+                        } else {
+                            modrm_sib(out_reg0, sink);
+                            sib(0, in_reg1, in_reg0, sink);
+                        }
+                    "#,
+                ),
+        );
+
+        // XX /r float load with index and no offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("fldWithIndex", f_load_complex, 2)
+                .operands_in(vec![gpr, gpr])
+                .operands_out(vec![fpr])
+                .inst_predicate(has_no_offset.clone())
+                .clobbers_flags(false)
+                .compute_size("size_plus_maybe_offset_for_in_reg_0")
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
+                        // The else branch always inserts an SIB byte.
+                        if needs_offset(in_reg0) {
+                            modrm_sib_disp8(out_reg0, sink);
+                            sib(0, in_reg1, in_reg0, sink);
+                            sink.put1(0);
+                        } else {
+                            modrm_sib(out_reg0, sink);
+                            sib(0, in_reg1, in_reg0, sink);
+                        }
+                    "#,
+                ),
+        );
+
+        let has_small_offset = InstructionPredicate::new_is_signed_int(format, "offset", 8, 0);
+
+        // XX /r load with index and 8-bit offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("ldWithIndexDisp8", f_load_complex, 3)
+                .operands_in(vec![gpr, gpr])
+                .operands_out(vec![gpr])
+                .inst_predicate(has_small_offset.clone())
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
+                        modrm_sib_disp8(out_reg0, sink);
+                        sib(0, in_reg1, in_reg0, sink);
+                        let offset: i32 = offset.into();
+                        sink.put1(offset as u8);
+                    "#,
+                ),
+        );
+
+        // XX /r float load with 8-bit offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("fldWithIndexDisp8", f_load_complex, 3)
+                .operands_in(vec![gpr, gpr])
+                .operands_out(vec![fpr])
+                .inst_predicate(has_small_offset.clone())
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
+                        modrm_sib_disp8(out_reg0, sink);
+                        sib(0, in_reg1, in_reg0, sink);
+                        let offset: i32 = offset.into();
+                        sink.put1(offset as u8);
+                    "#,
+                ),
+        );
+
+        let has_big_offset = InstructionPredicate::new_is_signed_int(format, "offset", 32, 0);
+
+        // XX /r load with index and 32-bit offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("ldWithIndexDisp32", f_load_complex, 6)
+                .operands_in(vec![gpr, gpr])
+                .operands_out(vec![gpr])
+                .inst_predicate(has_big_offset.clone())
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
+                        modrm_sib_disp32(out_reg0, sink);
+                        sib(0, in_reg1, in_reg0, sink);
+                        let offset: i32 = offset.into();
+                        sink.put4(offset as u32);
+                    "#,
+                ),
+        );
+
+        // XX /r float load with index and 32-bit offset.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("fldWithIndexDisp32", f_load_complex, 6)
+                .operands_in(vec![gpr, gpr])
+                .operands_out(vec![fpr])
+                .inst_predicate(has_big_offset.clone())
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                        if !flags.notrap() {
+                            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
+                        }
+                        {{PUT_OP}}(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
+                        modrm_sib_disp32(out_reg0, sink);
+                        sib(0, in_reg1, in_reg0, sink);
+                        let offset: i32 = offset.into();
+                        sink.put4(offset as u32);
+                    "#,
+                ),
+        );
+    }
+
+    // Unary fill with SIB and 32-bit displacement.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("fillSib32", f_unary, 6)
+            .operands_in(vec![stack_gpr32])
+            .operands_out(vec![gpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    let base = stk_base(in_stk0.base);
+                    {{PUT_OP}}(bits, rex2(base, out_reg0), sink);
+                    modrm_sib_disp32(out_reg0, sink);
+                    sib_noindex(base, sink);
+                    sink.put4(in_stk0.offset as u32);
+                "#,
+            ),
+    );
+
+    // Like fillSib32, but targeting an FPR rather than a GPR.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("ffillSib32", f_unary, 6)
+            .operands_in(vec![stack_fpr32])
+            .operands_out(vec![fpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    let base = stk_base(in_stk0.base);
+                    {{PUT_OP}}(bits, rex2(base, out_reg0), sink);
+                    modrm_sib_disp32(out_reg0, sink);
+                    sib_noindex(base, sink);
+                    sink.put4(in_stk0.offset as u32);
+                "#,
+            ),
+    );
+
+    // Regfill with RSP-relative 32-bit displacement.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("regfill32", f_reg_fill, 6)
+            .operands_in(vec![stack_gpr32])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    let src = StackRef::sp(src, &func.stack_slots);
+                    let base = stk_base(src.base);
+                    {{PUT_OP}}(bits, rex2(base, dst), sink);
+                    modrm_sib_disp32(dst, sink);
+                    sib_noindex(base, sink);
+                    sink.put4(src.offset as u32);
+                "#,
+            ),
+    );
+
+    // Like regfill32, but targeting an FPR rather than a GPR.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("fregfill32", f_reg_fill, 6)
+            .operands_in(vec![stack_fpr32])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    let src = StackRef::sp(src, &func.stack_slots);
+                    let base = stk_base(src.base);
+                    {{PUT_OP}}(bits, rex2(base, dst), sink);
+                    modrm_sib_disp32(dst, sink);
+                    sib_noindex(base, sink);
+                    sink.put4(src.offset as u32);
+                "#,
+            ),
+    );
+
+    // Call/return.
+
+    recipes.add_template_recipe(EncodingRecipeBuilder::new("call_id", f_call, 4).emit(
+        r#"
+            sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
+            {{PUT_OP}}(bits, BASE_REX, sink);
+            // The addend adjusts for the difference between the end of the
+            // instruction and the beginning of the immediate field.
+            sink.reloc_external(Reloc::X86CallPCRel4,
+                                &func.dfg.ext_funcs[func_ref].name,
+                                -4);
+            sink.put4(0);
+        "#,
+    ));
+
+    recipes.add_template_recipe(EncodingRecipeBuilder::new("call_plt_id", f_call, 4).emit(
+        r#"
+            sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
+            {{PUT_OP}}(bits, BASE_REX, sink);
+            sink.reloc_external(Reloc::X86CallPLTRel4,
+                                &func.dfg.ext_funcs[func_ref].name,
+                                -4);
+            sink.put4(0);
+        "#,
+    ));
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("call_r", f_call_indirect, 1)
+            .operands_in(vec![gpr])
+            .emit(
+                r#"
+                    sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
+                    {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                    modrm_r_bits(in_reg0, bits, sink);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("ret", f_multiary, 0).emit("{{PUT_OP}}(bits, BASE_REX, sink);"),
+    );
+
+    // Branches.
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("jmpb", f_jump, 1)
+            .branch_range((1, 8))
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, BASE_REX, sink);
+                    disp1(destination, func, sink);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("jmpd", f_jump, 4)
+            .branch_range((4, 32))
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, BASE_REX, sink);
+                    disp4(destination, func, sink);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("brib", f_branch_int, 1)
+            .operands_in(vec![reg_rflags])
+            .branch_range((1, 8))
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits | icc2opc(cond), BASE_REX, sink);
+                    disp1(destination, func, sink);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("brid", f_branch_int, 4)
+            .operands_in(vec![reg_rflags])
+            .branch_range((4, 32))
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits | icc2opc(cond), BASE_REX, sink);
+                    disp4(destination, func, sink);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("brfb", f_branch_float, 1)
+            .operands_in(vec![reg_rflags])
+            .branch_range((1, 8))
+            .clobbers_flags(false)
+            .inst_predicate(supported_floatccs_predicate(
+                &supported_floatccs,
+                formats.get(f_branch_float),
+            ))
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits | fcc2opc(cond), BASE_REX, sink);
+                    disp1(destination, func, sink);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("brfd", f_branch_float, 4)
+            .operands_in(vec![reg_rflags])
+            .branch_range((4, 32))
+            .clobbers_flags(false)
+            .inst_predicate(supported_floatccs_predicate(
+                &supported_floatccs,
+                formats.get(f_branch_float),
+            ))
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits | fcc2opc(cond), BASE_REX, sink);
+                    disp4(destination, func, sink);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("indirect_jmp", f_indirect_jump, 1)
+            .operands_in(vec![gpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                    modrm_r_bits(in_reg0, bits, sink);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("jt_entry", f_branch_table_entry, 2)
+            .operands_in(vec![gpr, gpr])
+            .operands_out(vec![gpr])
+            .clobbers_flags(false)
+            .inst_predicate(valid_scale(formats.get(f_branch_table_entry)))
+            .compute_size("size_plus_maybe_offset_for_in_reg_1")
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex3(in_reg1, out_reg0, in_reg0), sink);
+                    if needs_offset(in_reg1) {
+                        modrm_sib_disp8(out_reg0, sink);
+                        sib(imm.trailing_zeros() as u8, in_reg0, in_reg1, sink);
+                        sink.put1(0);
+                    } else {
+                        modrm_sib(out_reg0, sink);
+                        sib(imm.trailing_zeros() as u8, in_reg0, in_reg1, sink);
+                    }
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("jt_base", f_branch_table_base, 5)
+            .operands_out(vec![gpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(0, out_reg0), sink);
+                    modrm_riprel(out_reg0, sink);
+
+                    // No reloc is needed here as the jump table is emitted directly after
+                    // the function body.
+                    jt_disp4(table, func, sink);
+                "#,
+            ),
+    );
+
+    // Test flags and set a register.
+    //
+    // These setCC instructions only set the low 8 bits, and they can only write ABCD registers
+    // without a REX prefix.
+    //
+    // Other instruction encodings accepting `b1` inputs have the same constraints and only look at
+    // the low 8 bits of the input register.
+
+    let seti = recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("seti", f_int_cond, 1)
+                .operands_in(vec![reg_rflags])
+                .operands_out(vec![gpr])
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                    {{PUT_OP}}(bits | icc2opc(cond), rex1(out_reg0), sink);
+                    modrm_r_bits(out_reg0, bits, sink);
+                "#,
+                ),
+            formats,
+            regs,
+        )
+        .requires_prefix(true),
+    );
+
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("seti_abcd", f_int_cond, 1)
+                .operands_in(vec![reg_rflags])
+                .operands_out(vec![abcd])
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                    {{PUT_OP}}(bits | icc2opc(cond), rex1(out_reg0), sink);
+                    modrm_r_bits(out_reg0, bits, sink);
+                "#,
+                ),
+            formats,
+            regs,
+        )
+        .when_prefixed(seti),
+    );
+
+    let setf = recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("setf", f_float_cond, 1)
+                .operands_in(vec![reg_rflags])
+                .operands_out(vec![gpr])
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                    {{PUT_OP}}(bits | fcc2opc(cond), rex1(out_reg0), sink);
+                    modrm_r_bits(out_reg0, bits, sink);
+                "#,
+                ),
+            formats,
+            regs,
+        )
+        .requires_prefix(true),
+    );
+
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("setf_abcd", f_float_cond, 1)
+                .operands_in(vec![reg_rflags])
+                .operands_out(vec![abcd])
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                    {{PUT_OP}}(bits | fcc2opc(cond), rex1(out_reg0), sink);
+                    modrm_r_bits(out_reg0, bits, sink);
+                "#,
+                ),
+            formats,
+            regs,
+        )
+        .when_prefixed(setf),
+    );
+
+    // Conditional move (a.k.a integer select)
+    // (maybe-REX.W) 0F 4x modrm(r,r)
+    // 1 byte, modrm(r,r), is after the opcode
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("cmov", f_int_select, 1)
+            .operands_in(vec![
+                OperandConstraint::FixedReg(reg_rflags),
+                OperandConstraint::RegClass(gpr),
+                OperandConstraint::RegClass(gpr),
+            ])
+            .operands_out(vec![2])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits | icc2opc(cond), rex2(in_reg1, in_reg2), sink);
+                    modrm_rr(in_reg1, in_reg2, sink);
+                "#,
+            ),
+    );
+
+    // Bit scan forwards and reverse
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("bsf_and_bsr", f_unary, 1)
+            .operands_in(vec![gpr])
+            .operands_out(vec![
+                OperandConstraint::RegClass(gpr),
+                OperandConstraint::FixedReg(reg_rflags),
+            ])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                    modrm_rr(in_reg0, out_reg0, sink);
+                "#,
+            ),
+    );
+
+    // Compare and set flags.
+
+    // XX /r, MR form. Compare two GPR registers and set flags.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("rcmp", f_binary, 1)
+            .operands_in(vec![gpr, gpr])
+            .operands_out(vec![reg_rflags])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
+                    modrm_rr(in_reg0, in_reg1, sink);
+                "#,
+            ),
+    );
+
+    // Same as rcmp, but second operand is the stack pointer.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("rcmp_sp", f_unary, 1)
+            .operands_in(vec![gpr])
+            .operands_out(vec![reg_rflags])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, RU::rsp.into()), sink);
+                    modrm_rr(in_reg0, RU::rsp.into(), sink);
+                "#,
+            ),
+    );
+
+    // XX /r, RM form. Compare two FPR registers and set flags.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("fcmp", f_binary, 1)
+            .operands_in(vec![fpr, fpr])
+            .operands_out(vec![reg_rflags])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                    modrm_rr(in_reg1, in_reg0, sink);
+                "#,
+            ),
+    );
+
+    {
+        let format = formats.get(f_binary_imm);
+
+        let has_small_offset = InstructionPredicate::new_is_signed_int(format, "imm", 8, 0);
+
+        // XX /n, MI form with imm8.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("rcmp_ib", f_binary_imm, 2)
+                .operands_in(vec![gpr])
+                .operands_out(vec![reg_rflags])
+                .inst_predicate(has_small_offset)
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                        modrm_r_bits(in_reg0, bits, sink);
+                        let imm: i64 = imm.into();
+                        sink.put1(imm as u8);
+                    "#,
+                ),
+        );
+
+        let has_big_offset = InstructionPredicate::new_is_signed_int(format, "imm", 32, 0);
+
+        // XX /n, MI form with imm32.
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("rcmp_id", f_binary_imm, 5)
+                .operands_in(vec![gpr])
+                .operands_out(vec![reg_rflags])
+                .inst_predicate(has_big_offset)
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                        modrm_r_bits(in_reg0, bits, sink);
+                        let imm: i64 = imm.into();
+                        sink.put4(imm as u32);
+                    "#,
+                ),
+        );
+    }
+
+    // Test-and-branch.
+    //
+    // This recipe represents the macro fusion of a test and a conditional branch.
+    // This serves two purposes:
+    //
+    // 1. Guarantee that the test and branch get scheduled next to each other so
+    //    macro fusion is guaranteed to be possible.
+    // 2. Hide the status flags from Cranelift which doesn't currently model flags.
+    //
+    // The encoding bits affect both the test and the branch instruction:
+    //
+    // Bits 0-7 are the Jcc opcode.
+    // Bits 8-15 control the test instruction which always has opcode byte 0x85.
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("tjccb", f_branch, 1 + 2)
+            .operands_in(vec![gpr])
+            .branch_range((3, 8))
+            .emit(
+                r#"
+                    // test r, r.
+                    {{PUT_OP}}((bits & 0xff00) | 0x85, rex2(in_reg0, in_reg0), sink);
+                    modrm_rr(in_reg0, in_reg0, sink);
+                    // Jcc instruction.
+                    sink.put1(bits as u8);
+                    disp1(destination, func, sink);
+                "#,
+            ),
+    );
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("tjccd", f_branch, 1 + 6)
+            .operands_in(vec![gpr])
+            .branch_range((7, 32))
+            .emit(
+                r#"
+                    // test r, r.
+                    {{PUT_OP}}((bits & 0xff00) | 0x85, rex2(in_reg0, in_reg0), sink);
+                    modrm_rr(in_reg0, in_reg0, sink);
+                    // Jcc instruction.
+                    sink.put1(0x0f);
+                    sink.put1(bits as u8);
+                    disp4(destination, func, sink);
+                "#,
+            ),
+    );
+
+    // 8-bit test-and-branch.
+
+    let t8jccb = recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("t8jccb", f_branch, 1 + 2)
+                .operands_in(vec![gpr])
+                .branch_range((3, 8))
+                .emit(
+                    r#"
+                    // test8 r, r.
+                    {{PUT_OP}}((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink);
+                    modrm_rr(in_reg0, in_reg0, sink);
+                    // Jcc instruction.
+                    sink.put1(bits as u8);
+                    disp1(destination, func, sink);
+                "#,
+                ),
+            formats,
+            regs,
+        )
+        .requires_prefix(true),
+    );
+
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("t8jccb_abcd", f_branch, 1 + 2)
+                .operands_in(vec![abcd])
+                .branch_range((3, 8))
+                .emit(
+                    r#"
+                    // test8 r, r.
+                    {{PUT_OP}}((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink);
+                    modrm_rr(in_reg0, in_reg0, sink);
+                    // Jcc instruction.
+                    sink.put1(bits as u8);
+                    disp1(destination, func, sink);
+                "#,
+                ),
+            formats,
+            regs,
+        )
+        .when_prefixed(t8jccb),
+    );
+
+    let t8jccd = recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("t8jccd", f_branch, 1 + 6)
+                .operands_in(vec![gpr])
+                .branch_range((7, 32))
+                .emit(
+                    r#"
+                    // test8 r, r.
+                    {{PUT_OP}}((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink);
+                    modrm_rr(in_reg0, in_reg0, sink);
+                    // Jcc instruction.
+                    sink.put1(0x0f);
+                    sink.put1(bits as u8);
+                    disp4(destination, func, sink);
+                "#,
+                ),
+            formats,
+            regs,
+        )
+        .requires_prefix(true),
+    );
+
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("t8jccd_abcd", f_branch, 1 + 6)
+                .operands_in(vec![abcd])
+                .branch_range((7, 32))
+                .emit(
+                    r#"
+                    // test8 r, r.
+                    {{PUT_OP}}((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink);
+                    modrm_rr(in_reg0, in_reg0, sink);
+                    // Jcc instruction.
+                    sink.put1(0x0f);
+                    sink.put1(bits as u8);
+                    disp4(destination, func, sink);
+                "#,
+                ),
+            formats,
+            regs,
+        )
+        .when_prefixed(t8jccd),
+    );
+
+    // Worst case test-and-branch recipe for brz.b1 and brnz.b1 in 32-bit mode.
+    // The register allocator can't handle a branch instruction with constrained
+    // operands like the t8jccd_abcd above. This variant can accept the b1 opernd in
+    // any register, but is is larger because it uses a 32-bit test instruction with
+    // a 0xff immediate.
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("t8jccd_long", f_branch, 5 + 6)
+            .operands_in(vec![gpr])
+            .branch_range((11, 32))
+            .emit(
+                r#"
+                    // test32 r, 0xff.
+                    {{PUT_OP}}((bits & 0xff00) | 0xf7, rex1(in_reg0), sink);
+                    modrm_r_bits(in_reg0, bits, sink);
+                    sink.put4(0xff);
+                    // Jcc instruction.
+                    sink.put1(0x0f);
+                    sink.put1(bits as u8);
+                    disp4(destination, func, sink);
+                "#,
+            ),
+    );
+
+    // Comparison that produces a `b1` result in a GPR.
+    //
+    // This is a macro of a `cmp` instruction followed by a `setCC` instruction.
+    //
+    // TODO This is not a great solution because:
+    //
+    // - The cmp+setcc combination is not recognized by CPU's macro fusion.
+    // - The 64-bit encoding has issues with REX prefixes. The `cmp` and `setCC`
+    //   instructions may need a REX independently.
+    // - Modeling CPU flags in the type system would be better.
+    //
+    // Since the `setCC` instructions only write an 8-bit register, we use that as
+    // our `b1` representation: A `b1` value is represented as a GPR where the low 8
+    // bits are known to be 0 or 1. The high bits are undefined.
+    //
+    // This bandaid macro doesn't support a REX prefix for the final `setCC`
+    // instruction, so it is limited to the `ABCD` register class for booleans.
+    // The omission of a `when_prefixed` alternative is deliberate here.
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("icscc", f_int_compare, 1 + 3)
+            .operands_in(vec![gpr, gpr])
+            .operands_out(vec![abcd])
+            .emit(
+                r#"
+                    // Comparison instruction.
+                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
+                    modrm_rr(in_reg0, in_reg1, sink);
+                    // `setCC` instruction, no REX.
+                    use crate::ir::condcodes::IntCC::*;
+                    let setcc = match cond {
+                        Equal => 0x94,
+                        NotEqual => 0x95,
+                        SignedLessThan => 0x9c,
+                        SignedGreaterThanOrEqual => 0x9d,
+                        SignedGreaterThan => 0x9f,
+                        SignedLessThanOrEqual => 0x9e,
+                        UnsignedLessThan => 0x92,
+                        UnsignedGreaterThanOrEqual => 0x93,
+                        UnsignedGreaterThan => 0x97,
+                        UnsignedLessThanOrEqual => 0x96,
+                    };
+                    sink.put1(0x0f);
+                    sink.put1(setcc);
+                    modrm_rr(out_reg0, 0, sink);
+                "#,
+            ),
+    );
+
+    {
+        let format = formats.get(f_int_compare_imm);
+
+        let is_small_imm = InstructionPredicate::new_is_signed_int(format, "imm", 8, 0);
+
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("icscc_ib", f_int_compare_imm, 2 + 3)
+                .operands_in(vec![gpr])
+                .operands_out(vec![abcd])
+                .inst_predicate(is_small_imm)
+                .emit(
+                    r#"
+                        // Comparison instruction.
+                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                        modrm_r_bits(in_reg0, bits, sink);
+                        let imm: i64 = imm.into();
+                        sink.put1(imm as u8);
+                        // `setCC` instruction, no REX.
+                        use crate::ir::condcodes::IntCC::*;
+                        let setcc = match cond {
+                            Equal => 0x94,
+                            NotEqual => 0x95,
+                            SignedLessThan => 0x9c,
+                            SignedGreaterThanOrEqual => 0x9d,
+                            SignedGreaterThan => 0x9f,
+                            SignedLessThanOrEqual => 0x9e,
+                            UnsignedLessThan => 0x92,
+                            UnsignedGreaterThanOrEqual => 0x93,
+                            UnsignedGreaterThan => 0x97,
+                            UnsignedLessThanOrEqual => 0x96,
+                        };
+                        sink.put1(0x0f);
+                        sink.put1(setcc);
+                        modrm_rr(out_reg0, 0, sink);
+                    "#,
+                ),
+        );
+
+        let is_big_imm = InstructionPredicate::new_is_signed_int(format, "imm", 32, 0);
+
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("icscc_id", f_int_compare_imm, 5 + 3)
+                .operands_in(vec![gpr])
+                .operands_out(vec![abcd])
+                .inst_predicate(is_big_imm)
+                .emit(
+                    r#"
+                        // Comparison instruction.
+                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                        modrm_r_bits(in_reg0, bits, sink);
+                        let imm: i64 = imm.into();
+                        sink.put4(imm as u32);
+                        // `setCC` instruction, no REX.
+                        use crate::ir::condcodes::IntCC::*;
+                        let setcc = match cond {
+                            Equal => 0x94,
+                            NotEqual => 0x95,
+                            SignedLessThan => 0x9c,
+                            SignedGreaterThanOrEqual => 0x9d,
+                            SignedGreaterThan => 0x9f,
+                            SignedLessThanOrEqual => 0x9e,
+                            UnsignedLessThan => 0x92,
+                            UnsignedGreaterThanOrEqual => 0x93,
+                            UnsignedGreaterThan => 0x97,
+                            UnsignedLessThanOrEqual => 0x96,
+                        };
+                        sink.put1(0x0f);
+                        sink.put1(setcc);
+                        modrm_rr(out_reg0, 0, sink);
+                    "#,
+                ),
+        );
+    }
+
+    // Make a FloatCompare instruction predicate with the supported condition codes.
+    //
+    // Same thing for floating point.
+    //
+    // The ucomiss/ucomisd instructions set the FLAGS bits CF/PF/CF like this:
+    //
+    //    ZPC OSA
+    // UN 111 000
+    // GT 000 000
+    // LT 001 000
+    // EQ 100 000
+    //
+    // Not all floating point condition codes are supported.
+    // The omission of a `when_prefixed` alternative is deliberate here.
+
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("fcscc", f_float_compare, 1 + 3)
+            .operands_in(vec![fpr, fpr])
+            .operands_out(vec![abcd])
+            .inst_predicate(supported_floatccs_predicate(
+                &supported_floatccs,
+                formats.get(f_float_compare),
+            ))
+            .emit(
+                r#"
+                    // Comparison instruction.
+                    {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                    modrm_rr(in_reg1, in_reg0, sink);
+                    // `setCC` instruction, no REX.
+                    use crate::ir::condcodes::FloatCC::*;
+                    let setcc = match cond {
+                        Ordered                    => 0x9b, // EQ|LT|GT => setnp (P=0)
+                        Unordered                  => 0x9a, // UN       => setp  (P=1)
+                        OrderedNotEqual            => 0x95, // LT|GT    => setne (Z=0),
+                        UnorderedOrEqual           => 0x94, // UN|EQ    => sete  (Z=1)
+                        GreaterThan                => 0x97, // GT       => seta  (C=0&Z=0)
+                        GreaterThanOrEqual         => 0x93, // GT|EQ    => setae (C=0)
+                        UnorderedOrLessThan        => 0x92, // UN|LT    => setb  (C=1)
+                        UnorderedOrLessThanOrEqual => 0x96, // UN|LT|EQ => setbe (Z=1|C=1)
+                        Equal |                       // EQ
+                        NotEqual |                    // UN|LT|GT
+                        LessThan |                    // LT
+                        LessThanOrEqual |             // LT|EQ
+                        UnorderedOrGreaterThan |      // UN|GT
+                        UnorderedOrGreaterThanOrEqual // UN|GT|EQ
+                        => panic!("{} not supported by fcscc", cond),
+                    };
+                    sink.put1(0x0f);
+                    sink.put1(setcc);
+                    modrm_rr(out_reg0, 0, sink);
+                "#,
+            ),
+    );
+
+    recipes
+}

From f574ab870309f6bc2de0681dfb4cf6f69c5a06e6 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 24 Jun 2019 16:45:43 +0200
Subject: [PATCH 2478/3084] [meta] Generate the binemits files;

Co-authored-by: Benjamin Bouvier 
Co-authored-by: bjorn3 
---
 cranelift/codegen/meta/src/gen_binemit.rs | 224 ++++++++++++++++++++++
 cranelift/codegen/meta/src/lib.rs         |  10 +
 2 files changed, 234 insertions(+)
 create mode 100644 cranelift/codegen/meta/src/gen_binemit.rs

diff --git a/cranelift/codegen/meta/src/gen_binemit.rs b/cranelift/codegen/meta/src/gen_binemit.rs
new file mode 100644
index 0000000000..1950f59e9e
--- /dev/null
+++ b/cranelift/codegen/meta/src/gen_binemit.rs
@@ -0,0 +1,224 @@
+//! Generate binary emission code for each ISA.
+
+use cranelift_entity::EntityRef;
+
+use crate::error;
+use crate::srcgen::Formatter;
+
+use crate::cdsl::formats::FormatRegistry;
+use crate::cdsl::recipes::{EncodingRecipe, OperandConstraint, Recipes};
+
+/// Generate code to handle a single recipe.
+///
+/// - Unpack the instruction data, knowing the format.
+/// - Determine register locations for operands with register constraints.
+/// - Determine stack slot locations for operands with stack constraints.
+/// - Call hand-written code for the actual emission.
+fn gen_recipe(formats: &FormatRegistry, recipe: &EncodingRecipe, fmt: &mut Formatter) {
+    let inst_format = formats.get(recipe.format);
+    let num_value_ops = inst_format.num_value_operands;
+
+    let want_args = recipe.operands_in.iter().any(|c| match c {
+        OperandConstraint::RegClass(_) | OperandConstraint::Stack(_) => true,
+        OperandConstraint::FixedReg(_) | OperandConstraint::TiedInput(_) => false,
+    });
+    assert!(!want_args || num_value_ops > 0 || inst_format.has_value_list);
+
+    let want_outs = recipe.operands_out.iter().any(|c| match c {
+        OperandConstraint::RegClass(_) | OperandConstraint::Stack(_) => true,
+        OperandConstraint::FixedReg(_) | OperandConstraint::TiedInput(_) => false,
+    });
+
+    let is_regmove = ["RegMove", "RegSpill", "RegFill"].contains(&inst_format.name);
+
+    // Unpack the instruction data.
+    fmtln!(fmt, "if let InstructionData::{} {{", inst_format.name);
+    fmt.indent(|fmt| {
+        fmt.line("opcode,");
+        for f in &inst_format.imm_fields {
+            fmtln!(fmt, "{},", f.member);
+        }
+        if want_args {
+            if inst_format.has_value_list || num_value_ops > 1 {
+                fmt.line("ref args,");
+            } else {
+                fmt.line("arg,");
+            }
+        }
+        fmt.line("..");
+
+        fmt.outdented_line("} = func.dfg[inst] {");
+
+        // Pass recipe arguments in this order: inputs, imm_fields, outputs.
+        let mut args = String::new();
+
+        if want_args && !is_regmove {
+            if inst_format.has_value_list {
+                fmt.line("let args = args.as_slice(&func.dfg.value_lists);");
+            } else if num_value_ops == 1 {
+                fmt.line("let args = [arg];");
+            }
+            args += &unwrap_values(&recipe.operands_in, "in", "args", fmt);
+        }
+
+        for f in &inst_format.imm_fields {
+            args += &format!(", {}", f.member);
+        }
+
+        // Unwrap interesting output arguments.
+        if want_outs {
+            if recipe.operands_out.len() == 1 {
+                fmt.line("let results = [func.dfg.first_result(inst)];")
+            } else {
+                fmt.line("let results = func.dfg.inst_results(inst);");
+            }
+            args += &unwrap_values(&recipe.operands_out, "out", "results", fmt);
+        }
+
+        // Special handling for regmove instructions. Update the register
+        // diversion tracker
+        match &*inst_format.name {
+            "RegMove" => fmt.line("divert.regmove(arg, src, dst);"),
+            "RegSpill" => fmt.line("divert.regspill(arg, src, dst);"),
+            "RegFill" => fmt.line("divert.regfill(arg, src, dst);"),
+            _ => {}
+        }
+
+        match &recipe.emit {
+            Some(emit) => {
+                fmt.multi_line(emit);
+                fmt.line("return;");
+            }
+            None => {
+                fmtln!(
+                    fmt,
+                    "return recipe_{}(func, inst, sink, bits{});",
+                    recipe.name.to_lowercase(),
+                    args
+                );
+            }
+        }
+    });
+    fmt.line("}");
+}
+
+/// Emit code that unwraps values living in registers or stack slots.
+///
+/// :param args: Input or output constraints.
+/// :param prefix: Prefix to be used for the generated local variables.
+/// :param values: Name of slice containing the values to be unwrapped.
+/// :returns: Comma separated list of the generated variables
+fn unwrap_values(
+    args: &[OperandConstraint],
+    prefix: &str,
+    values_slice: &str,
+    fmt: &mut Formatter,
+) -> String {
+    let mut varlist = String::new();
+    for (i, cst) in args.iter().enumerate() {
+        match cst {
+            OperandConstraint::RegClass(_reg_class) => {
+                let v = format!("{}_reg{}", prefix, i);
+                varlist += &format!(", {}", v);
+                fmtln!(
+                    fmt,
+                    "let {} = divert.reg({}[{}], &func.locations);",
+                    v,
+                    values_slice,
+                    i
+                );
+            }
+            OperandConstraint::Stack(stack) => {
+                let v = format!("{}_stk{}", prefix, i);
+                varlist += &format!(", {}", v);
+                fmtln!(fmt, "let {} = StackRef::masked(", v);
+                fmt.indent(|fmt| {
+                    fmtln!(
+                        fmt,
+                        "divert.stack({}[{}], &func.locations),",
+                        values_slice,
+                        i
+                    );
+                    fmt.line(format!("{}, ", stack.stack_base_mask()));
+                    fmt.line("&func.stack_slots,");
+                });
+                fmt.line(").unwrap();");
+            }
+            _ => {}
+        }
+    }
+    varlist
+}
+
+fn gen_isa(formats: &FormatRegistry, isa_name: &str, recipes: &Recipes, fmt: &mut Formatter) {
+    fmt.doc_comment(format!(
+        "Emit binary machine code for `inst` for the {} ISA.",
+        isa_name
+    ));
+
+    if recipes.is_empty() {
+        fmt.line("pub fn emit_inst(");
+        fmt.indent(|fmt| {
+            fmt.line("func: &Function,");
+            fmt.line("inst: Inst,");
+            fmt.line("_divert: &mut RegDiversions,");
+            fmt.line("_sink: &mut CS,");
+        });
+        fmt.line(") {");
+        fmt.indent(|fmt| {
+            // No encoding recipes: Emit a stub.
+            fmt.line("bad_encoding(func, inst)");
+        });
+        fmt.line("}");
+        return;
+    }
+
+    fmt.line("#[allow(unused_variables, unreachable_code)]");
+    fmt.line("pub fn emit_inst(");
+    fmt.indent(|fmt| {
+        fmt.line("func: &Function,");
+        fmt.line("inst: Inst,");
+        fmt.line("divert: &mut RegDiversions,");
+        fmt.line("sink: &mut CS,");
+    });
+
+    fmt.line(") {");
+    fmt.indent(|fmt| {
+        fmt.line("let encoding = func.encodings[inst];");
+        fmt.line("let bits = encoding.bits();");
+        fmt.line("match func.encodings[inst].recipe() {");
+        fmt.indent(|fmt| {
+            for (i, recipe) in recipes.iter() {
+                fmt.comment(format!("Recipe {}", recipe.name));
+                fmtln!(fmt, "{} => {{", i.index());
+                fmt.indent(|fmt| {
+                    gen_recipe(formats, recipe, fmt);
+                });
+                fmt.line("}");
+            }
+            fmt.line("_ => {},");
+        });
+        fmt.line("}");
+
+        // Allow for unencoded ghost instructions. The verifier will check details.
+        fmt.line("if encoding.is_legal() {");
+        fmt.indent(|fmt| {
+            fmt.line("bad_encoding(func, inst);");
+        });
+        fmt.line("}");
+    });
+    fmt.line("}");
+}
+
+pub fn generate(
+    formats: &FormatRegistry,
+    isa_name: &str,
+    recipes: &Recipes,
+    binemit_filename: &str,
+    out_dir: &str,
+) -> Result<(), error::Error> {
+    let mut fmt = Formatter::new();
+    gen_isa(formats, isa_name, recipes, &mut fmt);
+    fmt.update_file(binemit_filename, out_dir)?;
+    Ok(())
+}
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index 901f640a51..bc43d26af5 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -5,6 +5,7 @@ mod srcgen;
 pub mod error;
 pub mod isa;
 
+mod gen_binemit;
 mod gen_inst;
 mod gen_legalizer;
 mod gen_registers;
@@ -48,12 +49,21 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
 
     for isa in isas {
         gen_registers::generate(&isa, &format!("registers-{}.rs", isa.name), &out_dir)?;
+
         gen_settings::generate(
             &isa.settings,
             gen_settings::ParentGroup::Shared,
             &format!("settings-{}.rs", isa.name),
             &out_dir,
         )?;
+
+        gen_binemit::generate(
+            &shared_defs.format_registry,
+            &isa.name,
+            &isa.recipes,
+            &format!("binemit-{}.rs", isa.name),
+            &out_dir,
+        )?;
     }
 
     Ok(())

From 88307f693a122249cdead70e9af2e67ded105d54 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 24 Jun 2019 16:44:59 +0200
Subject: [PATCH 2479/3084] [meta] Generate the encodings files;

---
 cranelift/codegen/meta/src/gen_encodings.rs   | 1071 +++++++++++++++++
 cranelift/codegen/meta/src/lib.rs             |    8 +
 cranelift/codegen/src/isa/arm32/enc_tables.rs |    1 +
 cranelift/codegen/src/isa/arm64/enc_tables.rs |    1 +
 4 files changed, 1081 insertions(+)
 create mode 100644 cranelift/codegen/meta/src/gen_encodings.rs

diff --git a/cranelift/codegen/meta/src/gen_encodings.rs b/cranelift/codegen/meta/src/gen_encodings.rs
new file mode 100644
index 0000000000..aff6d32d5d
--- /dev/null
+++ b/cranelift/codegen/meta/src/gen_encodings.rs
@@ -0,0 +1,1071 @@
+//! Generate sources for instruction encoding.
+//!
+//! The tables and functions generated here support the `TargetISA::encode()` function which
+//! determines if a given instruction is legal, and if so, its `Encoding` data which consists of a
+//! *recipe* and some *encoding* bits.
+//!
+//! The `encode` function doesn't actually generate the binary machine bits. Each recipe has a
+//! corresponding hand-written function to do that after registers are allocated.
+//!
+//! This is the information available to us:
+//!
+//! - The instruction to be encoded as an `InstructionData` reference.
+//! - The controlling type variable.
+//! - The data-flow graph giving us access to the types of all values involved. This is needed for
+//! testing any secondary type variables.
+//! - A `PredicateView` reference for the ISA-specific settings for evaluating ISA predicates.
+//! - The currently active CPU mode is determined by the ISA.
+//!
+//! ## Level 1 table lookup
+//!
+//! The CPU mode provides the first table. The key is the instruction's controlling type variable.
+//! If the instruction is not polymorphic, use `INVALID` for the type variable. The table values
+//! are level 2 tables.
+//!
+//! ## Level 2 table lookup
+//!
+//! The level 2 table is keyed by the instruction's opcode. The table values are *encoding lists*.
+//!
+//! The two-level table lookup allows the level 2 tables to be much smaller with good locality.
+//! Code in any given function usually only uses a few different types, so many of the level 2
+//! tables will be cold.
+//!
+//! ## Encoding lists
+//!
+//! An encoding list is a non-empty sequence of list entries. Each entry has one of these forms:
+//!
+//! 1. Recipe + bits. Use this encoding if the recipe predicate is satisfied.
+//! 2. Recipe + bits, final entry. Use this encoding if the recipe predicate is satisfied.
+//!    Otherwise, stop with the default legalization code.
+//! 3. Stop with legalization code.
+//! 4. Predicate + skip count. Test predicate and skip N entries if it is false.
+//! 5. Predicate + stop. Test predicate and stop with the default legalization code if it is false.
+//!
+//! The instruction predicate is also used to distinguish between polymorphic instructions with
+//! different types for secondary type variables.
+
+use std::collections::btree_map;
+use std::collections::{BTreeMap, HashMap, HashSet};
+use std::iter::FromIterator;
+
+use cranelift_entity::EntityRef;
+
+use crate::error;
+use crate::srcgen::Formatter;
+
+use crate::cdsl::cpu_modes::CpuMode;
+use crate::cdsl::encodings::Encoding;
+use crate::cdsl::instructions::{Instruction, InstructionPredicate, InstructionPredicateNumber};
+use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::recipes::{EncodingRecipe, OperandConstraint, Recipes, Register};
+use crate::cdsl::regs::IsaRegs;
+use crate::cdsl::settings::SettingPredicateNumber;
+use crate::cdsl::types::ValueType;
+use crate::cdsl::xform::TransformGroupIndex;
+
+use crate::shared::Definitions as SharedDefinitions;
+
+use crate::constant_hash::generate_table;
+use crate::default_map::MapWithDefault;
+use crate::unique_table::UniqueSeqTable;
+
+/// Emit code for matching an instruction predicate against an `InstructionData` reference called
+/// `inst`.
+///
+/// The generated code is an `if let` pattern match that falls through if the instruction has an
+/// unexpected format. This should lead to a panic.
+fn emit_instp(instp: &InstructionPredicate, has_func: bool, fmt: &mut Formatter) {
+    if instp.is_type_predicate() {
+        fmt.line("let args = inst.arguments(&func.dfg.value_lists);");
+        fmt.line(instp.rust_predicate());
+        return;
+    }
+
+    let leaves = instp.collect_leaves();
+
+    let mut has_type_check = false;
+    let mut format_name = None;
+    let mut field_names = HashSet::new();
+
+    for leaf in leaves {
+        if leaf.is_type_predicate() {
+            has_type_check = true;
+        } else {
+            field_names.insert(leaf.format_destructuring_member_name());
+            let leaf_format_name = leaf.format_name();
+            match format_name {
+                None => format_name = Some(leaf_format_name),
+                Some(previous_format_name) => {
+                    assert!(
+                        previous_format_name == leaf_format_name,
+                        format!("Format predicate can only operate on a single InstructionFormat; trying to use both {} and {}", previous_format_name, leaf_format_name
+                    ));
+                }
+            }
+        }
+    }
+
+    let mut fields = Vec::from_iter(field_names);
+    fields.sort();
+    let fields = fields.join(", ");
+
+    let format_name = format_name.expect("There should be a format name!");
+
+    fmtln!(
+        fmt,
+        "if let crate::ir::InstructionData::{} {{ {}, .. }} = *inst {{",
+        format_name,
+        fields
+    );
+    fmt.indent(|fmt| {
+        if has_type_check {
+            // We could implement this.
+            assert!(has_func, "recipe predicates can't check type variables.");
+            fmt.line("let args = inst.arguments(&func.dfg.value_lists);");
+        } else if has_func {
+            // Silence dead argument.
+            fmt.line("let _ = func;");
+        }
+        fmtln!(fmt, "return {};", instp.rust_predicate());
+    });
+    fmtln!(fmt, "}");
+
+    fmt.line("unreachable!();");
+}
+
+/// Emit private functions for checking recipe predicates as well as a static `RECIPE_PREDICATES`
+/// array indexed by recipe number.
+///
+/// A recipe predicate is a combination of an ISA predicate and an instruction predicate. Many
+/// recipes have identical predicates.
+fn emit_recipe_predicates(isa: &TargetIsa, fmt: &mut Formatter) {
+    let mut predicate_names = HashMap::new();
+
+    for recipe in isa.recipes.values() {
+        let (isap, instp) = match (&recipe.isa_predicate, &recipe.inst_predicate) {
+            (None, None) => continue,
+            (isap, instp) if predicate_names.contains_key(&(isap, instp)) => continue,
+            (isap, instp) => (isap, instp),
+        };
+
+        let func_name = format!("recipe_predicate_{}", recipe.name.to_lowercase());
+        predicate_names.insert((isap, instp), func_name.clone());
+
+        // Generate the predicate function.
+        fmtln!(
+            fmt,
+            "fn {}({}: crate::settings::PredicateView, {}: &ir::InstructionData) -> bool {{",
+            func_name,
+            if let Some(_) = isap { "isap" } else { "_" },
+            if let Some(_) = instp { "inst" } else { "_" }
+        );
+        fmt.indent(|fmt| {
+            match (isap, instp) {
+                (Some(isap), None) => {
+                    fmtln!(fmt, "isap.test({})", isap);
+                }
+                (None, Some(instp)) => {
+                    emit_instp(instp, /* has func */ false, fmt);
+                }
+                (Some(isap), Some(instp)) => {
+                    fmtln!(fmt, "isap.test({}) &&", isap);
+                    emit_instp(instp, /* has func */ false, fmt);
+                }
+                _ => panic!("skipped above"),
+            }
+        });
+        fmtln!(fmt, "}");
+    }
+
+    // Generate the static table.
+    fmtln!(
+        fmt,
+        "pub static RECIPE_PREDICATES: [RecipePredicate; {}] = [",
+        isa.recipes.len()
+    );
+    fmt.indent(|fmt| {
+        for recipe in isa.recipes.values() {
+            match (&recipe.isa_predicate, &recipe.inst_predicate) {
+                (None, None) => fmt.line("None,"),
+                key => fmtln!(fmt, "Some({}),", predicate_names.get(&key).unwrap()),
+            }
+        }
+    });
+    fmtln!(fmt, "];");
+}
+
+/// Emit private functions for matching instruction predicates as well as a static
+/// `INST_PREDICATES` array indexed by predicate number.
+fn emit_inst_predicates(isa: &TargetIsa, fmt: &mut Formatter) {
+    for (id, instp) in isa.encodings_predicates.iter() {
+        fmtln!(fmt, "fn inst_predicate_{}(func: &crate::ir::Function, inst: &crate::ir::InstructionData) -> bool {{", id.index());
+        fmt.indent(|fmt| {
+            emit_instp(instp, /* has func */ true, fmt);
+        });
+        fmtln!(fmt, "}");
+    }
+
+    // Generate the static table.
+    fmtln!(
+        fmt,
+        "pub static INST_PREDICATES: [InstPredicate; {}] = [",
+        isa.encodings_predicates.len()
+    );
+    fmt.indent(|fmt| {
+        for id in isa.encodings_predicates.keys() {
+            fmtln!(fmt, "inst_predicate_{},", id.index());
+        }
+    });
+    fmtln!(fmt, "];");
+}
+
+/// Emit a table of encoding recipe names keyed by recipe number.
+///
+/// This is used for pretty-printing encodings.
+fn emit_recipe_names(isa: &TargetIsa, fmt: &mut Formatter) {
+    fmtln!(
+        fmt,
+        "static RECIPE_NAMES: [&str; {}] = [",
+        isa.recipes.len()
+    );
+    fmt.indent(|fmt| {
+        for recipe in isa.recipes.values() {
+            fmtln!(fmt, r#""{}","#, recipe.name);
+        }
+    });
+    fmtln!(fmt, "];");
+}
+
+/// Returns a set of all the registers involved in fixed register constraints.
+fn get_fixed_registers(operands_in: &Vec) -> HashSet {
+    HashSet::from_iter(
+        operands_in
+            .iter()
+            .map(|constraint| {
+                if let OperandConstraint::FixedReg(reg) = &constraint {
+                    Some(reg.clone())
+                } else {
+                    None
+                }
+            })
+            .filter(|opt| opt.is_some())
+            .map(|opt| opt.unwrap()),
+    )
+}
+
+/// Emit a struct field initializer for an array of operand constraints.
+///
+/// Note "fixed_registers" must refer to the other kind of operands (i.e. if we're operating on
+/// inputs, fixed_registers must contain the fixed output registers).
+fn emit_operand_constraints(
+    registers: &IsaRegs,
+    recipe: &EncodingRecipe,
+    constraints: &Vec,
+    field_name: &'static str,
+    tied_operands: &HashMap,
+    fixed_registers: &HashSet,
+    fmt: &mut Formatter,
+) {
+    if constraints.len() == 0 {
+        fmtln!(fmt, "{}: &[],", field_name);
+        return;
+    }
+
+    fmtln!(fmt, "{}: &[", field_name);
+    fmt.indent(|fmt| {
+        for (n, constraint) in constraints.iter().enumerate() {
+            fmt.line("OperandConstraint {");
+            fmt.indent(|fmt| {
+                match constraint {
+                    OperandConstraint::RegClass(reg_class) => {
+                        if let Some(tied_input) = tied_operands.get(&n) {
+                            fmtln!(fmt, "kind: ConstraintKind::Tied({}),", tied_input);
+                        } else {
+                            fmt.line("kind: ConstraintKind::Reg,");
+                        }
+                        fmtln!(
+                            fmt,
+                            "regclass: &{}_DATA,",
+                            registers.classes[*reg_class].name
+                        );
+                    }
+                    OperandConstraint::FixedReg(reg) => {
+                        assert!(!tied_operands.contains_key(&n), "can't tie fixed registers");
+                        let constraint_kind = if fixed_registers.contains(®) {
+                            "FixedTied"
+                        } else {
+                            "FixedReg"
+                        };
+                        fmtln!(
+                            fmt,
+                            "kind: ConstraintKind::{}({}),",
+                            constraint_kind,
+                            reg.unit
+                        );
+                        fmtln!(
+                            fmt,
+                            "regclass: &{}_DATA,",
+                            registers.classes[reg.regclass].name
+                        );
+                    }
+                    OperandConstraint::TiedInput(tied_input) => {
+                        // This is a tied output constraint. It should never happen
+                        // for input constraints.
+                        assert!(
+                            tied_input == tied_operands.get(&n).unwrap(),
+                            "invalid tied constraint"
+                        );
+                        fmtln!(fmt, "kind: ConstraintKind::Tied({}),", tied_input);
+
+                        let tied_class = if let OperandConstraint::RegClass(tied_class) =
+                            recipe.operands_in[*tied_input]
+                        {
+                            tied_class
+                        } else {
+                            panic!("tied constraints relate only to register inputs");
+                        };
+
+                        fmtln!(
+                            fmt,
+                            "regclass: &{}_DATA,",
+                            registers.classes[tied_class].name
+                        );
+                    }
+                    OperandConstraint::Stack(stack) => {
+                        assert!(!tied_operands.contains_key(&n), "can't tie stack operand");
+                        fmt.line("kind: ConstraintKind::Stack,");
+                        fmtln!(
+                            fmt,
+                            "regclass: &{}_DATA,",
+                            registers.classes[stack.regclass].name
+                        );
+                    }
+                }
+            });
+            fmt.line("},");
+        }
+    });
+    fmtln!(fmt, "],");
+}
+
+/// Emit a table of encoding recipe operand constraints keyed by recipe number.
+///
+/// These are used by the register allocator to pick registers that can be properly encoded.
+fn emit_recipe_constraints(isa: &TargetIsa, fmt: &mut Formatter) {
+    fmtln!(
+        fmt,
+        "static RECIPE_CONSTRAINTS: [RecipeConstraints; {}] = [",
+        isa.recipes.len()
+    );
+    fmt.indent(|fmt| {
+        for recipe in isa.recipes.values() {
+            // Compute a mapping of tied operands in both directions (input tied to outputs and
+            // conversely).
+            let mut tied_in_to_out = HashMap::new();
+            let mut tied_out_to_in = HashMap::new();
+            for (out_index, constraint) in recipe.operands_out.iter().enumerate() {
+                if let OperandConstraint::TiedInput(in_index) = &constraint {
+                    tied_in_to_out.insert(*in_index, out_index);
+                    tied_out_to_in.insert(out_index, *in_index);
+                }
+            }
+
+            // Find the sets of registers involved in fixed register constraints.
+            let fixed_inputs = get_fixed_registers(&recipe.operands_in);
+            let fixed_outputs = get_fixed_registers(&recipe.operands_out);
+
+            fmt.comment(format!("Constraints for recipe {}:", recipe.name));
+            fmt.line("RecipeConstraints {");
+            fmt.indent(|fmt| {
+                emit_operand_constraints(
+                    &isa.regs,
+                    recipe,
+                    &recipe.operands_in,
+                    "ins",
+                    &tied_in_to_out,
+                    &fixed_outputs,
+                    fmt,
+                );
+                emit_operand_constraints(
+                    &isa.regs,
+                    recipe,
+                    &recipe.operands_out,
+                    "outs",
+                    &tied_out_to_in,
+                    &fixed_inputs,
+                    fmt,
+                );
+                fmtln!(
+                    fmt,
+                    "fixed_ins: {},",
+                    if !fixed_inputs.is_empty() {
+                        "true"
+                    } else {
+                        "false"
+                    }
+                );
+                fmtln!(
+                    fmt,
+                    "fixed_outs: {},",
+                    if !fixed_outputs.is_empty() {
+                        "true"
+                    } else {
+                        "false"
+                    }
+                );
+                fmtln!(
+                    fmt,
+                    "tied_ops: {},",
+                    if !tied_in_to_out.is_empty() {
+                        "true"
+                    } else {
+                        "false"
+                    }
+                );
+                fmtln!(
+                    fmt,
+                    "clobbers_flags: {},",
+                    if recipe.clobbers_flags {
+                        "true"
+                    } else {
+                        "false"
+                    }
+                );
+            });
+            fmt.line("},");
+        }
+    });
+    fmtln!(fmt, "];");
+}
+
+/// Emit a table of encoding recipe code size information.
+fn emit_recipe_sizing(isa: &TargetIsa, fmt: &mut Formatter) {
+    fmtln!(
+        fmt,
+        "static RECIPE_SIZING: [RecipeSizing; {}] = [",
+        isa.recipes.len()
+    );
+    fmt.indent(|fmt| {
+        for recipe in isa.recipes.values() {
+            fmt.comment(format!("Code size information for recipe {}:", recipe.name));
+            fmt.line("RecipeSizing {");
+            fmt.indent(|fmt| {
+                fmtln!(fmt, "base_size: {},", recipe.base_size);
+                fmtln!(fmt, "compute_size: {},", recipe.compute_size);
+                if let Some(range) = &recipe.branch_range {
+                    fmtln!(
+                        fmt,
+                        "branch_range: Some(BranchRange {{ origin: {}, bits: {} }}),",
+                        range.inst_size,
+                        range.range
+                    );
+                } else {
+                    fmt.line("branch_range: None,");
+                }
+            });
+            fmt.line("},");
+        }
+    });
+    fmtln!(fmt, "];");
+}
+
+/// Level 1 table mapping types to `Level2` objects.
+struct Level1Table<'cpu_mode> {
+    cpu_mode: &'cpu_mode CpuMode,
+    legalize_code: TransformGroupIndex,
+
+    table_map: HashMap, usize>,
+    table_vec: Vec,
+}
+
+impl<'cpu_mode> Level1Table<'cpu_mode> {
+    fn new(cpu_mode: &'cpu_mode CpuMode) -> Self {
+        Self {
+            cpu_mode,
+            legalize_code: cpu_mode.get_default_legalize_code(),
+            table_map: HashMap::new(),
+            table_vec: Vec::new(),
+        }
+    }
+
+    /// Returns the level2 table for the given type; None means monomorphic, in this context.
+    fn l2table_for(&mut self, typ: Option) -> &mut Level2Table {
+        let cpu_mode = &self.cpu_mode;
+        let index = match self.table_map.get(&typ) {
+            Some(&index) => index,
+            None => {
+                let legalize_code = cpu_mode.get_legalize_code_for(&typ);
+                let table = Level2Table::new(typ.clone(), legalize_code);
+                let index = self.table_vec.len();
+                self.table_map.insert(typ, index);
+                self.table_vec.push(table);
+                index
+            }
+        };
+        self.table_vec.get_mut(index).unwrap()
+    }
+
+    fn l2tables(&mut self) -> Vec<&mut Level2Table> {
+        self.table_vec
+            .iter_mut()
+            .filter(|table| !table.is_empty())
+            .collect::>()
+    }
+}
+
+struct Level2HashTableEntry {
+    inst_name: String,
+    offset: usize,
+}
+
+/// Level 2 table mapping instruction opcodes to `EncList` objects.
+///
+/// A level 2 table can be completely empty if it only holds a custom legalization action for `ty`.
+struct Level2Table {
+    typ: Option,
+    legalize_code: TransformGroupIndex,
+    inst_to_encodings: BTreeMap,
+    hash_table_offset: Option,
+    hash_table_len: Option,
+}
+
+impl Level2Table {
+    fn new(typ: Option, legalize_code: TransformGroupIndex) -> Self {
+        Self {
+            typ,
+            legalize_code,
+            inst_to_encodings: BTreeMap::new(),
+            hash_table_offset: None,
+            hash_table_len: None,
+        }
+    }
+
+    fn enclist_for(&mut self, inst: &Instruction) -> &mut EncodingList {
+        let copied_typ = self.typ.clone();
+        self.inst_to_encodings
+            .entry(inst.name.clone())
+            .or_insert_with(|| EncodingList::new(inst, copied_typ))
+    }
+
+    fn enclists(&mut self) -> btree_map::ValuesMut<'_, String, EncodingList> {
+        self.inst_to_encodings.values_mut()
+    }
+
+    fn is_empty(&self) -> bool {
+        self.inst_to_encodings.is_empty()
+    }
+
+    fn layout_hashtable(
+        &mut self,
+        level2_hashtables: &mut Vec>,
+        level2_doc: &mut HashMap>,
+    ) {
+        let hash_table = generate_table(
+            self.inst_to_encodings.values(),
+            self.inst_to_encodings.len(),
+            // TODO the Python code wanted opcode numbers to start from 1.
+            |enc_list| enc_list.inst.opcode_number.index() + 1,
+        );
+
+        let hash_table_offset = level2_hashtables.len();
+        let hash_table_len = hash_table.len();
+
+        assert!(self.hash_table_offset.is_none());
+        assert!(self.hash_table_len.is_none());
+        self.hash_table_offset = Some(hash_table_offset);
+        self.hash_table_len = Some(hash_table_len);
+
+        level2_hashtables.extend(hash_table.iter().map(|opt_enc_list| {
+            opt_enc_list.map(|enc_list| Level2HashTableEntry {
+                inst_name: enc_list.inst.camel_name.clone(),
+                offset: enc_list.offset.unwrap(),
+            })
+        }));
+
+        let typ_comment = match &self.typ {
+            Some(ty) => ty.to_string(),
+            None => "typeless".into(),
+        };
+
+        level2_doc.get_or_default(hash_table_offset).push(format!(
+            "{:06x}: {}, {} entries",
+            hash_table_offset, typ_comment, hash_table_len
+        ));
+    }
+}
+
+/// The u16 values in an encoding list entry are interpreted as follows:
+///
+/// NR = len(all_recipes)
+///
+/// entry < 2*NR
+///     Try Encoding(entry/2, next_entry) if the recipe predicate is satisfied.
+///     If bit 0 is set, stop with the default legalization code.
+///     If bit 0 is clear, keep going down the list.
+/// entry < PRED_START
+///     Stop with legalization code `entry - 2*NR`.
+///
+/// Remaining entries are interpreted as (skip, pred) pairs, where:
+///
+/// skip = (entry - PRED_START) >> PRED_BITS
+/// pred = (entry - PRED_START) & PRED_MASK
+///
+/// If the predicate is satisfied, keep going. Otherwise skip over the next
+/// `skip` entries. If skip == 0, stop with the default legalization code.
+///
+/// The `pred` predicate number is interpreted as an instruction predicate if it
+/// is in range, otherwise an ISA predicate.
+
+/// Encoding lists are represented as u16 arrays.
+const CODE_BITS: usize = 16;
+
+/// Beginning of the predicate code words.
+const PRED_START: u16 = 0x1000;
+
+/// Number of bits used to hold a predicate number (instruction + ISA predicates).
+const PRED_BITS: usize = 12;
+
+/// Mask for extracting the predicate number.
+const PRED_MASK: usize = (1 << PRED_BITS) - 1;
+
+/// Encoder for the list format above.
+struct Encoder {
+    num_instruction_predicates: usize,
+
+    /// u16 encoding list words.
+    words: Vec,
+
+    /// Documentation comments: Index into `words` + comment.
+    docs: Vec<(usize, String)>,
+}
+
+impl Encoder {
+    fn new(num_instruction_predicates: usize) -> Self {
+        Self {
+            num_instruction_predicates,
+            words: Vec::new(),
+            docs: Vec::new(),
+        }
+    }
+
+    /// Add a recipe+bits entry to the list.
+    fn recipe(&mut self, recipes: &Recipes, enc: &Encoding, is_final: bool) {
+        let code = (2 * enc.recipe.index() + if is_final { 1 } else { 0 }) as u16;
+        assert!(code < PRED_START);
+
+        let doc = format!(
+            "--> {}{}",
+            enc.to_rust_comment(recipes),
+            if is_final { " and stop" } else { "" }
+        );
+        self.docs.push((self.words.len(), doc));
+
+        self.words.push(code);
+        self.words.push(enc.encbits);
+    }
+
+    /// Add a predicate entry.
+    fn pred(&mut self, pred_comment: String, skip: usize, n: usize) {
+        assert!(n <= PRED_MASK);
+        let entry = (PRED_START as usize) + (n | (skip << PRED_BITS));
+        assert!(entry < (1 << CODE_BITS));
+        let entry = entry as u16;
+
+        let doc = if skip == 0 {
+            "stop".to_string()
+        } else {
+            format!("skip {}", skip)
+        };
+        let doc = format!("{} unless {}", doc, pred_comment);
+
+        self.docs.push((self.words.len(), doc));
+        self.words.push(entry);
+    }
+
+    /// Add an instruction predicate entry.
+    fn inst_predicate(&mut self, pred: InstructionPredicateNumber, skip: usize) {
+        let number = pred.index();
+        let pred_comment = format!("inst_predicate_{}", number);
+        self.pred(pred_comment, skip, number);
+    }
+
+    /// Add an ISA predicate entry.
+    fn isa_predicate(&mut self, pred: SettingPredicateNumber, skip: usize) {
+        // ISA predicates follow the instruction predicates.
+        let n = self.num_instruction_predicates + (pred as usize);
+        let pred_comment = format!("PredicateView({})", pred);
+        self.pred(pred_comment, skip, n);
+    }
+}
+
+/// List of instructions for encoding a given type + opcode pair.
+///
+/// An encoding list contains a sequence of predicates and encoding recipes, all encoded as u16
+/// values.
+struct EncodingList {
+    inst: Instruction,
+    typ: Option,
+    encodings: Vec,
+    offset: Option,
+}
+
+impl EncodingList {
+    fn new(inst: &Instruction, typ: Option) -> Self {
+        Self {
+            inst: inst.clone(),
+            typ,
+            encodings: Default::default(),
+            offset: None,
+        }
+    }
+
+    /// Encode this list as a sequence of u16 numbers.
+    ///
+    /// Adds the sequence to `enc_lists` and records the returned offset as
+    /// `self.offset`.
+    ///
+    /// Adds comment lines to `enc_lists_doc` keyed by enc_lists offsets.
+    fn encode(
+        &mut self,
+        isa: &TargetIsa,
+        cpu_mode: &CpuMode,
+        enc_lists: &mut UniqueSeqTable,
+        enc_lists_doc: &mut HashMap>,
+    ) {
+        assert!(!self.encodings.is_empty());
+
+        let mut encoder = Encoder::new(isa.encodings_predicates.len());
+
+        let mut index = 0;
+        while index < self.encodings.len() {
+            let encoding = &self.encodings[index];
+
+            // Try to see how many encodings are following and have the same ISA predicate and
+            // instruction predicate, so as to reduce the number of tests carried out by the
+            // encoding list interpreter..
+            //
+            // Encodings with similar tests are hereby called a group. The group includes the
+            // current encoding we're looking at.
+            let (isa_predicate, inst_predicate) =
+                (&encoding.isa_predicate, &encoding.inst_predicate);
+
+            let group_size = {
+                let mut group_size = 1;
+                while index + group_size < self.encodings.len() {
+                    let next_encoding = &self.encodings[index + group_size];
+                    if &next_encoding.inst_predicate != inst_predicate
+                        || &next_encoding.isa_predicate != isa_predicate
+                    {
+                        break;
+                    }
+                    group_size += 1;
+                }
+                group_size
+            };
+
+            let is_last_group = index + group_size == self.encodings.len();
+
+            // The number of entries to skip when a predicate isn't satisfied is the size of both
+            // predicates + the size of the group, minus one (for this predicate). Each recipe
+            // entry has a size of two u16 (recipe index + bits).
+            let mut skip = if is_last_group {
+                0
+            } else {
+                let isap_size = match isa_predicate {
+                    Some(_) => 1,
+                    None => 0,
+                };
+                let instp_size = match inst_predicate {
+                    Some(_) => 1,
+                    None => 0,
+                };
+                isap_size + instp_size + group_size * 2 - 1
+            };
+
+            if let Some(pred) = isa_predicate {
+                encoder.isa_predicate(*pred, skip);
+                if !is_last_group {
+                    skip -= 1;
+                }
+            }
+
+            if let Some(pred) = inst_predicate {
+                encoder.inst_predicate(*pred, skip);
+                // No need to update skip, it's dead after this point.
+            }
+
+            for i in 0..group_size {
+                let encoding = &self.encodings[index + i];
+                let is_last_encoding = index + i == self.encodings.len() - 1;
+                encoder.recipe(&isa.recipes, encoding, is_last_encoding);
+            }
+
+            index += group_size;
+        }
+
+        assert!(self.offset.is_none());
+        let offset = enc_lists.add(&encoder.words);
+        self.offset = Some(offset);
+
+        // Doc comments.
+        let recipe_typ_mode_name = format!(
+            "{}{} ({})",
+            self.inst.name,
+            if let Some(typ) = &self.typ {
+                format!(".{}", typ.to_string())
+            } else {
+                "".into()
+            },
+            cpu_mode.name
+        );
+
+        enc_lists_doc
+            .get_or_default(offset)
+            .push(format!("{:06x}: {}", offset, recipe_typ_mode_name));
+        for (pos, doc) in encoder.docs {
+            enc_lists_doc.get_or_default(offset + pos).push(doc);
+        }
+        enc_lists_doc
+            .get_or_default(offset + encoder.words.len())
+            .insert(0, format!("end of {}", recipe_typ_mode_name));
+    }
+}
+
+fn make_tables(cpu_mode: &CpuMode) -> Level1Table {
+    let mut table = Level1Table::new(cpu_mode);
+
+    for encoding in &cpu_mode.encodings {
+        table
+            .l2table_for(encoding.bound_type.clone())
+            .enclist_for(encoding.inst())
+            .encodings
+            .push(encoding.clone());
+    }
+
+    // Ensure there are level 1 table entries for all types with a custom legalize action.
+    for value_type in cpu_mode.get_legalized_types() {
+        table.l2table_for(Some(value_type.clone()));
+    }
+    // ... and also for monomorphic instructions.
+    table.l2table_for(None);
+
+    table
+}
+
+/// Compute encodings and doc comments for encoding lists in `level1`.
+fn encode_enclists(
+    isa: &TargetIsa,
+    cpu_mode: &CpuMode,
+    level1: &mut Level1Table,
+    enc_lists: &mut UniqueSeqTable,
+    enc_lists_doc: &mut HashMap>,
+) {
+    for level2 in level1.l2tables() {
+        for enclist in level2.enclists() {
+            enclist.encode(isa, cpu_mode, enc_lists, enc_lists_doc);
+        }
+    }
+}
+
+fn encode_level2_hashtables<'a>(
+    level1: &'a mut Level1Table,
+    level2_hashtables: &mut Vec>,
+    level2_doc: &mut HashMap>,
+) {
+    for level2 in level1.l2tables() {
+        level2.layout_hashtable(level2_hashtables, level2_doc);
+    }
+}
+
+fn emit_tables(defs: &SharedDefinitions, isa: &TargetIsa, fmt: &mut Formatter) {
+    // Level 1 tables, one per CPU mode.
+    let mut level1_tables: HashMap<&'static str, Level1Table> = HashMap::new();
+
+    // Single table containing all the level2 hash tables.
+    let mut level2_hashtables = Vec::new();
+    let mut level2_doc: HashMap> = HashMap::new();
+
+    // Tables for encoding lists with comments.
+    let mut enc_lists = UniqueSeqTable::new();
+    let mut enc_lists_doc = HashMap::new();
+
+    for cpu_mode in &isa.cpu_modes {
+        level2_doc
+            .get_or_default(level2_hashtables.len())
+            .push(cpu_mode.name.into());
+
+        let mut level1 = make_tables(cpu_mode);
+
+        encode_enclists(
+            isa,
+            cpu_mode,
+            &mut level1,
+            &mut enc_lists,
+            &mut enc_lists_doc,
+        );
+        encode_level2_hashtables(&mut level1, &mut level2_hashtables, &mut level2_doc);
+
+        level1_tables.insert(cpu_mode.name, level1);
+    }
+
+    // Compute an appropriate Rust integer type to use for offsets into a table of the given length.
+    let offset_type = |length: usize| {
+        if length <= 0x10000 {
+            "u16"
+        } else {
+            assert!(length <= 0x100000000, "table too big!");
+            "u32"
+        }
+    };
+
+    let level1_offset_type = offset_type(level2_hashtables.len());
+    let level2_offset_type = offset_type(enc_lists.len());
+
+    // Emit encoding lists.
+    fmtln!(fmt, "pub static ENCLISTS: [u16; {}] = [", enc_lists.len());
+    fmt.indent(|fmt| {
+        let mut line = Vec::new();
+        for (index, entry) in enc_lists.iter().enumerate() {
+            if let Some(comments) = enc_lists_doc.get(&index) {
+                if !line.is_empty() {
+                    fmtln!(fmt, "{},", line.join(", "));
+                    line.clear();
+                }
+                for comment in comments {
+                    fmt.comment(comment);
+                }
+            }
+            line.push(format!("{:#06x}", entry));
+        }
+        if !line.is_empty() {
+            fmtln!(fmt, "{},", line.join(", "));
+        }
+    });
+    fmtln!(fmt, "];");
+
+    // Emit the full concatenation of level 2 hash tables.
+    fmtln!(
+        fmt,
+        "pub static LEVEL2: [Level2Entry<{}>; {}] = [",
+        level2_offset_type,
+        level2_hashtables.len()
+    );
+    fmt.indent(|fmt| {
+        for (offset, entry) in level2_hashtables.iter().enumerate() {
+            if let Some(comments) = level2_doc.get(&offset) {
+                for comment in comments {
+                    fmt.comment(comment);
+                }
+            }
+            if let Some(entry) = entry {
+                fmtln!(
+                    fmt,
+                    "Level2Entry {{ opcode: Some(crate::ir::Opcode::{}), offset: {:#08x} }},",
+                    entry.inst_name,
+                    entry.offset
+                );
+            } else {
+                fmt.line("Level2Entry { opcode: None, offset: 0 },");
+            }
+        }
+    });
+    fmtln!(fmt, "];");
+
+    // Emit a level 1 hash table for each CPU mode.
+    for cpu_mode in &isa.cpu_modes {
+        let level1 = &level1_tables.get(cpu_mode.name).unwrap();
+        let hash_table = generate_table(
+            level1.table_vec.iter(),
+            level1.table_vec.len(),
+            |level2_table| {
+                if let Some(typ) = &level2_table.typ {
+                    typ.number().expect("type without a number") as usize
+                } else {
+                    0
+                }
+            },
+        );
+
+        fmtln!(
+            fmt,
+            "pub static LEVEL1_{}: [Level1Entry<{}>; {}] = [",
+            cpu_mode.name.to_uppercase(),
+            level1_offset_type,
+            hash_table.len()
+        );
+        fmt.indent(|fmt| {
+            for opt_level2 in hash_table {
+                let level2 = match opt_level2 {
+                    None => {
+                        // Empty hash table entry. Include the default legalization action.
+                        fmtln!(fmt, "Level1Entry {{ ty: ir::types::INVALID, log2len: !0, offset: 0, legalize: {} }},",
+                               isa.translate_group_index(level1.legalize_code));
+                        continue;
+                    }
+                    Some(level2) => level2,
+                };
+
+                let legalize_comment = defs.transform_groups.get(level2.legalize_code).name;
+                let legalize_code = isa.translate_group_index(level2.legalize_code);
+
+                let typ_name = if let Some(typ) = &level2.typ {
+                    typ.rust_name()
+                } else {
+                    "ir::types::INVALID".into()
+                };
+
+                if level2.is_empty() {
+                    // Empty level 2 table: Only a specialized legalization action, no actual
+                    // table.
+                    // Set an offset that is out of bounds, but make sure it doesn't overflow its
+                    // type when adding `1< 0, "Level2 hash table was too small.");
+                fmtln!(fmt, "Level1Entry {{ ty: {}, log2len: {}, offset: {:#08x}, legalize: {} }}, // {}",
+                       typ_name, l2l, level2.hash_table_offset.unwrap(), legalize_code, legalize_comment);
+            }
+        });
+        fmtln!(fmt, "];");
+    }
+}
+
+fn gen_isa(defs: &SharedDefinitions, isa: &TargetIsa, fmt: &mut Formatter) {
+    // Make the `RECIPE_PREDICATES` table.
+    emit_recipe_predicates(isa, fmt);
+
+    // Make the `INST_PREDICATES` table.
+    emit_inst_predicates(isa, fmt);
+
+    emit_tables(defs, isa, fmt);
+
+    emit_recipe_names(isa, fmt);
+    emit_recipe_constraints(isa, fmt);
+    emit_recipe_sizing(isa, fmt);
+
+    // Finally, tie it all together in an `EncInfo`.
+    fmt.line("pub static INFO: isa::EncInfo = isa::EncInfo {");
+    fmt.indent(|fmt| {
+        fmt.line("constraints: &RECIPE_CONSTRAINTS,");
+        fmt.line("sizing: &RECIPE_SIZING,");
+        fmt.line("names: &RECIPE_NAMES,");
+    });
+    fmt.line("};");
+}
+
+pub fn generate(
+    defs: &SharedDefinitions,
+    isa: &TargetIsa,
+    filename: &str,
+    out_dir: &str,
+) -> Result<(), error::Error> {
+    let mut fmt = Formatter::new();
+    gen_isa(defs, isa, &mut fmt);
+    fmt.update_file(filename, out_dir)?;
+    Ok(())
+}
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index bc43d26af5..ef450e7875 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -6,6 +6,7 @@ pub mod error;
 pub mod isa;
 
 mod gen_binemit;
+mod gen_encodings;
 mod gen_inst;
 mod gen_legalizer;
 mod gen_registers;
@@ -57,6 +58,13 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
             &out_dir,
         )?;
 
+        gen_encodings::generate(
+            &shared_defs,
+            &isa,
+            &format!("encoding-{}.rs", isa.name),
+            &out_dir,
+        )?;
+
         gen_binemit::generate(
             &shared_defs.format_registry,
             &isa.name,
diff --git a/cranelift/codegen/src/isa/arm32/enc_tables.rs b/cranelift/codegen/src/isa/arm32/enc_tables.rs
index e3833eecf7..5cb4e5c4c0 100644
--- a/cranelift/codegen/src/isa/arm32/enc_tables.rs
+++ b/cranelift/codegen/src/isa/arm32/enc_tables.rs
@@ -1,5 +1,6 @@
 //! Encoding tables for ARM32 ISA.
 
+use crate::ir;
 use crate::isa;
 use crate::isa::constraints::*;
 use crate::isa::enc_tables::*;
diff --git a/cranelift/codegen/src/isa/arm64/enc_tables.rs b/cranelift/codegen/src/isa/arm64/enc_tables.rs
index 6e1b73e0e9..6040a9b866 100644
--- a/cranelift/codegen/src/isa/arm64/enc_tables.rs
+++ b/cranelift/codegen/src/isa/arm64/enc_tables.rs
@@ -1,5 +1,6 @@
 //! Encoding tables for ARM64 ISA.
 
+use crate::ir;
 use crate::isa;
 use crate::isa::constraints::*;
 use crate::isa::enc_tables::*;

From f29a26de146a90724de25d28a69196b749923775 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 20 Jun 2019 17:04:55 +0200
Subject: [PATCH 2480/3084] [meta] Remove the Python DSL

KILL THE SNAKE WITH FIRE.
---
 cranelift/CONTRIBUTING.md                     |    8 -
 cranelift/README.md                           |    3 +-
 cranelift/codegen/build.rs                    |   39 -
 .../codegen/meta-python/base/__init__.py      |    1 -
 .../codegen/meta-python/base/entities.py      |   38 -
 cranelift/codegen/meta-python/base/formats.py |   91 -
 .../codegen/meta-python/base/immediates.py    |  123 -
 .../codegen/meta-python/base/instructions.py  | 2054 ----------------
 .../codegen/meta-python/base/legalize.py      |  713 ------
 .../codegen/meta-python/base/predicates.py    |   42 -
 .../codegen/meta-python/base/semantics.py     |  218 --
 .../codegen/meta-python/base/settings.py      |  144 --
 cranelift/codegen/meta-python/base/types.py   |   49 -
 cranelift/codegen/meta-python/build.py        |   51 -
 .../codegen/meta-python/cdsl/__init__.py      |   59 -
 cranelift/codegen/meta-python/cdsl/ast.py     |  581 -----
 cranelift/codegen/meta-python/cdsl/formats.py |  268 ---
 .../codegen/meta-python/cdsl/instructions.py  |  446 ----
 cranelift/codegen/meta-python/cdsl/isa.py     |  455 ----
 .../codegen/meta-python/cdsl/operands.py      |  251 --
 .../codegen/meta-python/cdsl/predicates.py    |  448 ----
 .../codegen/meta-python/cdsl/registers.py     |  413 ----
 .../codegen/meta-python/cdsl/settings.py      |  416 ----
 .../codegen/meta-python/cdsl/test_ast.py      |   28 -
 .../codegen/meta-python/cdsl/test_package.py  |    8 -
 cranelift/codegen/meta-python/cdsl/test_ti.py |  605 -----
 .../codegen/meta-python/cdsl/test_typevar.py  |  266 ---
 .../codegen/meta-python/cdsl/test_xform.py    |  131 --
 cranelift/codegen/meta-python/cdsl/ti.py      |  894 -------
 cranelift/codegen/meta-python/cdsl/types.py   |  348 ---
 cranelift/codegen/meta-python/cdsl/typevar.py |  906 --------
 cranelift/codegen/meta-python/cdsl/xform.py   |  423 ----
 cranelift/codegen/meta-python/check.sh        |   32 -
 .../codegen/meta-python/constant_hash.py      |   63 -
 cranelift/codegen/meta-python/gen_binemit.py  |  170 --
 .../codegen/meta-python/gen_build_deps.py     |   32 -
 cranelift/codegen/meta-python/gen_encoding.py |  902 --------
 cranelift/codegen/meta-python/isa/__init__.py |   24 -
 .../codegen/meta-python/isa/arm32/__init__.py |   15 -
 .../codegen/meta-python/isa/arm32/defs.py     |   19 -
 .../meta-python/isa/arm32/registers.py        |   45 -
 .../codegen/meta-python/isa/arm32/settings.py |   11 -
 .../codegen/meta-python/isa/arm64/__init__.py |   14 -
 .../codegen/meta-python/isa/arm64/defs.py     |   15 -
 .../meta-python/isa/arm64/registers.py        |   32 -
 .../codegen/meta-python/isa/arm64/settings.py |   11 -
 .../codegen/meta-python/isa/riscv/__init__.py |   33 -
 .../codegen/meta-python/isa/riscv/defs.py     |   14 -
 .../meta-python/isa/riscv/encodings.py        |  169 --
 .../codegen/meta-python/isa/riscv/recipes.py  |  230 --
 .../meta-python/isa/riscv/registers.py        |   23 -
 .../codegen/meta-python/isa/riscv/settings.py |   31 -
 .../codegen/meta-python/isa/x86/__init__.py   |   22 -
 cranelift/codegen/meta-python/isa/x86/defs.py |   28 -
 .../codegen/meta-python/isa/x86/encodings.py  |  771 ------
 .../meta-python/isa/x86/instructions.py       |  173 --
 .../codegen/meta-python/isa/x86/legalize.py   |  229 --
 .../codegen/meta-python/isa/x86/recipes.py    | 2059 -----------------
 .../codegen/meta-python/isa/x86/registers.py  |   61 -
 .../codegen/meta-python/isa/x86/settings.py   |   61 -
 cranelift/codegen/meta-python/mypy.ini        |    5 -
 .../codegen/meta-python/semantics/__init__.py |   77 -
 .../meta-python/semantics/elaborate.py        |  146 --
 .../codegen/meta-python/semantics/macros.py   |   45 -
 .../meta-python/semantics/primitives.py       |  133 --
 .../codegen/meta-python/semantics/smtlib.py   |  241 --
 .../meta-python/semantics/test_elaborate.py   |  392 ----
 cranelift/codegen/meta-python/srcgen.py       |  277 ---
 .../codegen/meta-python/stubs/z3/__init__.pyi |  151 --
 .../codegen/meta-python/stubs/z3/z3core.pyi   |    3 -
 .../codegen/meta-python/stubs/z3/z3types.pyi  |   12 -
 .../codegen/meta-python/test_constant_hash.py |    8 -
 cranelift/codegen/meta-python/test_srcgen.py  |    8 -
 cranelift/codegen/meta-python/unique_table.py |   80 -
 cranelift/test-all.sh                         |   19 -
 75 files changed, 1 insertion(+), 17405 deletions(-)
 delete mode 100644 cranelift/codegen/meta-python/base/__init__.py
 delete mode 100644 cranelift/codegen/meta-python/base/entities.py
 delete mode 100644 cranelift/codegen/meta-python/base/formats.py
 delete mode 100644 cranelift/codegen/meta-python/base/immediates.py
 delete mode 100644 cranelift/codegen/meta-python/base/instructions.py
 delete mode 100644 cranelift/codegen/meta-python/base/legalize.py
 delete mode 100644 cranelift/codegen/meta-python/base/predicates.py
 delete mode 100644 cranelift/codegen/meta-python/base/semantics.py
 delete mode 100644 cranelift/codegen/meta-python/base/settings.py
 delete mode 100644 cranelift/codegen/meta-python/base/types.py
 delete mode 100644 cranelift/codegen/meta-python/build.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/__init__.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/ast.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/formats.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/instructions.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/isa.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/operands.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/predicates.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/registers.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/settings.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/test_ast.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/test_package.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/test_ti.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/test_typevar.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/test_xform.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/ti.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/types.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/typevar.py
 delete mode 100644 cranelift/codegen/meta-python/cdsl/xform.py
 delete mode 100755 cranelift/codegen/meta-python/check.sh
 delete mode 100644 cranelift/codegen/meta-python/constant_hash.py
 delete mode 100644 cranelift/codegen/meta-python/gen_binemit.py
 delete mode 100644 cranelift/codegen/meta-python/gen_build_deps.py
 delete mode 100644 cranelift/codegen/meta-python/gen_encoding.py
 delete mode 100644 cranelift/codegen/meta-python/isa/__init__.py
 delete mode 100644 cranelift/codegen/meta-python/isa/arm32/__init__.py
 delete mode 100644 cranelift/codegen/meta-python/isa/arm32/defs.py
 delete mode 100644 cranelift/codegen/meta-python/isa/arm32/registers.py
 delete mode 100644 cranelift/codegen/meta-python/isa/arm32/settings.py
 delete mode 100644 cranelift/codegen/meta-python/isa/arm64/__init__.py
 delete mode 100644 cranelift/codegen/meta-python/isa/arm64/defs.py
 delete mode 100644 cranelift/codegen/meta-python/isa/arm64/registers.py
 delete mode 100644 cranelift/codegen/meta-python/isa/arm64/settings.py
 delete mode 100644 cranelift/codegen/meta-python/isa/riscv/__init__.py
 delete mode 100644 cranelift/codegen/meta-python/isa/riscv/defs.py
 delete mode 100644 cranelift/codegen/meta-python/isa/riscv/encodings.py
 delete mode 100644 cranelift/codegen/meta-python/isa/riscv/recipes.py
 delete mode 100644 cranelift/codegen/meta-python/isa/riscv/registers.py
 delete mode 100644 cranelift/codegen/meta-python/isa/riscv/settings.py
 delete mode 100644 cranelift/codegen/meta-python/isa/x86/__init__.py
 delete mode 100644 cranelift/codegen/meta-python/isa/x86/defs.py
 delete mode 100644 cranelift/codegen/meta-python/isa/x86/encodings.py
 delete mode 100644 cranelift/codegen/meta-python/isa/x86/instructions.py
 delete mode 100644 cranelift/codegen/meta-python/isa/x86/legalize.py
 delete mode 100644 cranelift/codegen/meta-python/isa/x86/recipes.py
 delete mode 100644 cranelift/codegen/meta-python/isa/x86/registers.py
 delete mode 100644 cranelift/codegen/meta-python/isa/x86/settings.py
 delete mode 100644 cranelift/codegen/meta-python/mypy.ini
 delete mode 100644 cranelift/codegen/meta-python/semantics/__init__.py
 delete mode 100644 cranelift/codegen/meta-python/semantics/elaborate.py
 delete mode 100644 cranelift/codegen/meta-python/semantics/macros.py
 delete mode 100644 cranelift/codegen/meta-python/semantics/primitives.py
 delete mode 100644 cranelift/codegen/meta-python/semantics/smtlib.py
 delete mode 100644 cranelift/codegen/meta-python/semantics/test_elaborate.py
 delete mode 100644 cranelift/codegen/meta-python/srcgen.py
 delete mode 100644 cranelift/codegen/meta-python/stubs/z3/__init__.pyi
 delete mode 100644 cranelift/codegen/meta-python/stubs/z3/z3core.pyi
 delete mode 100644 cranelift/codegen/meta-python/stubs/z3/z3types.pyi
 delete mode 100644 cranelift/codegen/meta-python/test_constant_hash.py
 delete mode 100644 cranelift/codegen/meta-python/test_srcgen.py
 delete mode 100644 cranelift/codegen/meta-python/unique_table.py

diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
index 403e912e60..4d97970a6c 100644
--- a/cranelift/CONTRIBUTING.md
+++ b/cranelift/CONTRIBUTING.md
@@ -84,14 +84,6 @@ build.
 
 [Rust Update Policy for Firefox]: https://wiki.mozilla.org/Rust_Update_Policy_for_Firefox#Schedule
 
-### Python
-
-Our Python code is checked with [mypy](http://mypy-lang.org/) and
-[flake8](http://flake8.pycqa.org/en/latest/); see the
-[check.sh](https://github.com/CraneStation/cranelift/blob/master/cranelift-codegen/meta-python/check.sh)
-file for details. The versions available in common package repositories such
-as Ubuntu or Homebrew typically work fine.
-
 ## Development Process
 
 We use [issues] for asking questions and tracking bugs and unimplemented
diff --git a/cranelift/README.md b/cranelift/README.md
index fe25d2a842..da1b3adb4b 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -53,8 +53,7 @@ needed before it would be ready for a production use case.
 
 Cranelift's APIs are not yet stable.
 
-Cranelift currently requires Rust 1.35 or later, and Python 2.7 or 3
-to build.
+Cranelift currently requires Rust 1.35 or later to build.
 
 Planned uses
 ------------
diff --git a/cranelift/codegen/build.rs b/cranelift/codegen/build.rs
index 733a1c782f..d9d6f13fc5 100644
--- a/cranelift/codegen/build.rs
+++ b/cranelift/codegen/build.rs
@@ -57,32 +57,6 @@ fn main() {
         crate_dir.join("build.rs").to_str().unwrap()
     );
 
-    // Scripts are in `$crate_dir/meta-python`.
-    let meta_dir = crate_dir.join("meta-python");
-    let build_script = meta_dir.join("build.py");
-
-    // Launch build script with Python. We'll just find python in the path.
-    // Use -B to disable .pyc files, because they cause trouble for vendoring
-    // scripts, and this is a build step that isn't run very often anyway.
-    let python = identify_python();
-    let status = process::Command::new(python)
-        .current_dir(crate_dir)
-        .arg("-B")
-        .arg(build_script)
-        .arg("--out-dir")
-        .arg(out_dir.clone())
-        .status()
-        .expect("Failed to launch second-level build script; is python installed?");
-    if !status.success() {
-        process::exit(status.code().unwrap());
-    }
-
-    // DEVELOPMENT:
-    // ------------------------------------------------------------------------
-    // Now that the Python build process is complete, generate files that are
-    // emitted by the `meta` crate.
-    // ------------------------------------------------------------------------
-
     if let Err(err) = meta::generate(&isas, &out_dir) {
         eprintln!("Error: {}", err);
         process::exit(1);
@@ -99,16 +73,3 @@ fn main() {
         println!("cargo:warning=Generated files are in {}", out_dir);
     }
 }
-
-fn identify_python() -> &'static str {
-    for python in &["python", "python3", "python2.7"] {
-        if process::Command::new(python)
-            .arg("--version")
-            .status()
-            .is_ok()
-        {
-            return python;
-        }
-    }
-    panic!("The Cranelift build requires Python (version 2.7 or version 3)");
-}
diff --git a/cranelift/codegen/meta-python/base/__init__.py b/cranelift/codegen/meta-python/base/__init__.py
deleted file mode 100644
index 79f6ccbf46..0000000000
--- a/cranelift/codegen/meta-python/base/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-"""Definitions for the base Cranelift language."""
diff --git a/cranelift/codegen/meta-python/base/entities.py b/cranelift/codegen/meta-python/base/entities.py
deleted file mode 100644
index 0226b18ca4..0000000000
--- a/cranelift/codegen/meta-python/base/entities.py
+++ /dev/null
@@ -1,38 +0,0 @@
-"""
-The `cranelift.entities` module predefines all the Cranelift entity reference
-operand types. There are corresponding definitions in the `cranelift.entities`
-Rust module.
-"""
-from __future__ import absolute_import
-from cdsl.operands import EntityRefKind
-
-
-#: A reference to an extended basic block in the same function.
-#: This is primarliy used in control flow instructions.
-ebb = EntityRefKind(
-        'ebb', 'An extended basic block in the same function.',
-        default_member='destination')
-
-#: A reference to a stack slot declared in the function preamble.
-stack_slot = EntityRefKind('stack_slot', 'A stack slot.')
-
-#: A reference to a global value.
-global_value = EntityRefKind('global_value', 'A global value.')
-
-#: A reference to a function signature declared in the function preamble.
-#: This is used to provide the call signature in a call_indirect instruction.
-sig_ref = EntityRefKind('sig_ref', 'A function signature.')
-
-#: A reference to an external function declared in the function preamble.
-#: This is used to provide the callee and signature in a call instruction.
-func_ref = EntityRefKind('func_ref', 'An external function.')
-
-#: A reference to a jump table declared in the function preamble.
-jump_table = EntityRefKind(
-        'jump_table', 'A jump table.', default_member='table')
-
-#: A reference to a heap declared in the function preamble.
-heap = EntityRefKind('heap', 'A heap.')
-
-#: A reference to a table declared in the function preamble.
-table = EntityRefKind('table', 'A table.')
diff --git a/cranelift/codegen/meta-python/base/formats.py b/cranelift/codegen/meta-python/base/formats.py
deleted file mode 100644
index b45ac26f74..0000000000
--- a/cranelift/codegen/meta-python/base/formats.py
+++ /dev/null
@@ -1,91 +0,0 @@
-"""
-The cranelift.formats defines all instruction formats.
-
-Every instruction format has a corresponding `InstructionData` variant in the
-Rust representation of Cranelift IR, so all instruction formats must be defined
-in this module.
-"""
-from __future__ import absolute_import
-from cdsl.formats import InstructionFormat
-from cdsl.operands import VALUE, VARIABLE_ARGS
-from .immediates import imm64, uimm8, uimm32, ieee32, ieee64, offset32
-from .immediates import boolean, intcc, floatcc, memflags, regunit, trapcode
-from . import entities
-from .entities import ebb, sig_ref, func_ref, stack_slot, heap, table
-
-Unary = InstructionFormat(VALUE)
-UnaryImm = InstructionFormat(imm64)
-UnaryIeee32 = InstructionFormat(ieee32)
-UnaryIeee64 = InstructionFormat(ieee64)
-UnaryBool = InstructionFormat(boolean)
-UnaryGlobalValue = InstructionFormat(entities.global_value)
-
-Binary = InstructionFormat(VALUE, VALUE)
-BinaryImm = InstructionFormat(VALUE, imm64)
-
-# The select instructions are controlled by the second VALUE operand.
-# The first VALUE operand is the controlling flag which has a derived type.
-# The fma instruction has the same constraint on all inputs.
-Ternary = InstructionFormat(VALUE, VALUE, VALUE, typevar_operand=1)
-
-# Catch-all for instructions with many outputs and inputs and no immediate
-# operands.
-MultiAry = InstructionFormat(VARIABLE_ARGS)
-
-NullAry = InstructionFormat()
-
-InsertLane = InstructionFormat(VALUE, ('lane', uimm8), VALUE)
-ExtractLane = InstructionFormat(VALUE, ('lane', uimm8))
-
-IntCompare = InstructionFormat(intcc, VALUE, VALUE)
-IntCompareImm = InstructionFormat(intcc, VALUE, imm64)
-IntCond = InstructionFormat(intcc, VALUE)
-FloatCompare = InstructionFormat(floatcc, VALUE, VALUE)
-FloatCond = InstructionFormat(floatcc, VALUE)
-
-IntSelect = InstructionFormat(intcc, VALUE, VALUE, VALUE)
-
-Jump = InstructionFormat(ebb, VARIABLE_ARGS)
-Branch = InstructionFormat(VALUE, ebb, VARIABLE_ARGS)
-BranchInt = InstructionFormat(intcc, VALUE, ebb, VARIABLE_ARGS)
-BranchFloat = InstructionFormat(floatcc, VALUE, ebb, VARIABLE_ARGS)
-BranchIcmp = InstructionFormat(intcc, VALUE, VALUE, ebb, VARIABLE_ARGS)
-BranchTable = InstructionFormat(VALUE, ebb, entities.jump_table)
-BranchTableEntry = InstructionFormat(VALUE, VALUE, uimm8, entities.jump_table)
-BranchTableBase = InstructionFormat(entities.jump_table)
-IndirectJump = InstructionFormat(VALUE, entities.jump_table)
-
-Call = InstructionFormat(func_ref, VARIABLE_ARGS)
-CallIndirect = InstructionFormat(sig_ref, VALUE, VARIABLE_ARGS)
-FuncAddr = InstructionFormat(func_ref)
-
-Load = InstructionFormat(memflags, VALUE, offset32)
-LoadComplex = InstructionFormat(memflags, VARIABLE_ARGS, offset32)
-Store = InstructionFormat(memflags, VALUE, VALUE, offset32)
-StoreComplex = InstructionFormat(memflags, VALUE, VARIABLE_ARGS, offset32)
-
-StackLoad = InstructionFormat(stack_slot, offset32)
-StackStore = InstructionFormat(VALUE, stack_slot, offset32)
-
-# Accessing a WebAssembly heap.
-HeapAddr = InstructionFormat(heap, VALUE, uimm32)
-
-# Accessing a WebAssembly table.
-TableAddr = InstructionFormat(table, VALUE, offset32)
-
-RegMove = InstructionFormat(VALUE, ('src', regunit), ('dst', regunit))
-CopySpecial = InstructionFormat(('src', regunit), ('dst', regunit))
-CopyNop = InstructionFormat(
-        ('src', entities.stack_slot), ('dst', entities.stack_slot))
-RegSpill = InstructionFormat(
-        VALUE, ('src', regunit), ('dst', entities.stack_slot))
-RegFill = InstructionFormat(
-        VALUE, ('src', entities.stack_slot), ('dst', regunit))
-
-Trap = InstructionFormat(trapcode)
-CondTrap = InstructionFormat(VALUE, trapcode)
-IntCondTrap = InstructionFormat(intcc, VALUE, trapcode)
-FloatCondTrap = InstructionFormat(floatcc, VALUE, trapcode)
-
-# Finally extract the names of global values in this module.
-InstructionFormat.extract_names(globals())
diff --git a/cranelift/codegen/meta-python/base/immediates.py b/cranelift/codegen/meta-python/base/immediates.py
deleted file mode 100644
index e8a6fccc78..0000000000
--- a/cranelift/codegen/meta-python/base/immediates.py
+++ /dev/null
@@ -1,123 +0,0 @@
-"""
-The `cranelift.immediates` module predefines all the Cranelift immediate
-operand types.
-"""
-from __future__ import absolute_import
-from cdsl.operands import ImmediateKind
-
-#: A 64-bit immediate integer operand.
-#:
-#: This type of immediate integer can interact with SSA values with any
-#: :py:class:`cranelift.IntType` type.
-imm64 = ImmediateKind('imm64', 'A 64-bit immediate integer.')
-
-#: An unsigned 8-bit immediate integer operand.
-#:
-#: This small operand is used to indicate lane indexes in SIMD vectors and
-#: immediate bit counts on shift instructions.
-uimm8 = ImmediateKind('uimm8', 'An 8-bit immediate unsigned integer.')
-
-#: An unsigned 32-bit immediate integer operand.
-uimm32 = ImmediateKind('uimm32', 'A 32-bit immediate unsigned integer.')
-
-#: A 32-bit immediate signed offset.
-#:
-#: This is used to represent an immediate address offset in load/store
-#: instructions.
-offset32 = ImmediateKind(
-        'offset32',
-        'A 32-bit immediate signed offset.',
-        default_member='offset')
-
-#: A 32-bit immediate floating point operand.
-#:
-#: IEEE 754-2008 binary32 interchange format.
-ieee32 = ImmediateKind('ieee32', 'A 32-bit immediate floating point number.')
-
-#: A 64-bit immediate floating point operand.
-#:
-#: IEEE 754-2008 binary64 interchange format.
-ieee64 = ImmediateKind('ieee64', 'A 64-bit immediate floating point number.')
-
-#: An immediate boolean operand.
-#:
-#: This type of immediate boolean can interact with SSA values with any
-#: :py:class:`cranelift.BoolType` type.
-boolean = ImmediateKind('bool', 'An immediate boolean.',
-                        rust_type='bool')
-
-#: A condition code for comparing integer values.
-#:
-#: This enumerated operand kind is used for the :clif:inst:`icmp` instruction
-#: and corresponds to the `condcodes::IntCC` Rust type.
-intcc = ImmediateKind(
-        'intcc',
-        'An integer comparison condition code.',
-        default_member='cond',
-        rust_type='ir::condcodes::IntCC',
-        values={
-            'eq':  'Equal',
-            'ne':  'NotEqual',
-            'sge': 'SignedGreaterThanOrEqual',
-            'sgt': 'SignedGreaterThan',
-            'sle': 'SignedLessThanOrEqual',
-            'slt': 'SignedLessThan',
-            'uge': 'UnsignedGreaterThanOrEqual',
-            'ugt': 'UnsignedGreaterThan',
-            'ule': 'UnsignedLessThanOrEqual',
-            'ult': 'UnsignedLessThan',
-        })
-
-#: A condition code for comparing floating point values.
-#:
-#: This enumerated operand kind is used for the :clif:inst:`fcmp` instruction
-#: and corresponds to the `condcodes::FloatCC` Rust type.
-floatcc = ImmediateKind(
-        'floatcc',
-        'A floating point comparison condition code.',
-        default_member='cond',
-        rust_type='ir::condcodes::FloatCC',
-        values={
-            'ord': 'Ordered',
-            'uno': 'Unordered',
-            'eq':  'Equal',
-            'ne':  'NotEqual',
-            'one': 'OrderedNotEqual',
-            'ueq': 'UnorderedOrEqual',
-            'lt':  'LessThan',
-            'le':  'LessThanOrEqual',
-            'gt':  'GreaterThan',
-            'ge':  'GreaterThanOrEqual',
-            'ult': 'UnorderedOrLessThan',
-            'ule': 'UnorderedOrLessThanOrEqual',
-            'ugt': 'UnorderedOrGreaterThan',
-            'uge': 'UnorderedOrGreaterThanOrEqual',
-        })
-
-#: Flags for memory operations like :clif:inst:`load` and :clif:inst:`store`.
-memflags = ImmediateKind(
-        'memflags',
-        'Memory operation flags',
-        default_member='flags', rust_type='ir::MemFlags')
-
-#: A register unit in the current target ISA.
-regunit = ImmediateKind(
-        'regunit',
-        'A register unit in the target ISA',
-        rust_type='isa::RegUnit')
-
-#: A trap code indicating the reason for trapping.
-#:
-#: The Rust enum type also has a `User(u16)` variant for user-provided trap
-#: codes.
-trapcode = ImmediateKind(
-        'trapcode',
-        'A trap reason code.',
-        default_member='code',
-        rust_type='ir::TrapCode',
-        values={
-            "stk_ovf": 'StackOverflow',
-            "heap_oob": 'HeapOutOfBounds',
-            "int_ovf": 'IntegerOverflow',
-            "int_divz": 'IntegerDivisionByZero',
-        })
diff --git a/cranelift/codegen/meta-python/base/instructions.py b/cranelift/codegen/meta-python/base/instructions.py
deleted file mode 100644
index 1c0ebc164f..0000000000
--- a/cranelift/codegen/meta-python/base/instructions.py
+++ /dev/null
@@ -1,2054 +0,0 @@
-"""
-Cranelift base instruction set.
-
-This module defines the basic Cranelift instruction set that all targets
-support.
-"""
-from __future__ import absolute_import
-from cdsl.operands import Operand, VARIABLE_ARGS
-from cdsl.typevar import TypeVar
-from cdsl.instructions import Instruction, InstructionGroup
-from base.types import f32, f64, b1, iflags, fflags
-from base.immediates import imm64, uimm8, uimm32, ieee32, ieee64, offset32
-from base.immediates import boolean, intcc, floatcc, memflags, regunit
-from base.immediates import trapcode
-from base import entities
-from cdsl.ti import WiderOrEq
-import base.formats  # noqa
-
-GROUP = InstructionGroup("base", "Shared base instruction set")
-
-Int = TypeVar('Int', 'A scalar or vector integer type', ints=True, simd=True)
-Bool = TypeVar('Bool', 'A scalar or vector boolean type',
-               bools=True, simd=True)
-iB = TypeVar('iB', 'A scalar integer type', ints=True)
-iAddr = TypeVar('iAddr', 'An integer address type', ints=(32, 64))
-Testable = TypeVar(
-        'Testable', 'A scalar boolean or integer type',
-        ints=True, bools=True)
-TxN = TypeVar(
-        'TxN', 'A SIMD vector type',
-        ints=True, floats=True, bools=True, scalars=False, simd=True)
-Any = TypeVar(
-        'Any', 'Any integer, float, or boolean scalar or vector type',
-        ints=True, floats=True, bools=True, scalars=True, simd=True)
-Mem = TypeVar(
-        'Mem', 'Any type that can be stored in memory',
-        ints=True, floats=True, simd=True)
-MemTo = TypeVar(
-        'MemTo', 'Any type that can be stored in memory',
-        ints=True, floats=True, simd=True)
-
-addr = Operand('addr', iAddr)
-
-#
-# Control flow
-#
-c = Operand('c', Testable, doc='Controlling value to test')
-Cond = Operand('Cond', intcc)
-x = Operand('x', iB)
-y = Operand('y', iB)
-EBB = Operand('EBB', entities.ebb, doc='Destination extended basic block')
-args = Operand('args', VARIABLE_ARGS, doc='EBB arguments')
-
-jump = Instruction(
-        'jump', r"""
-        Jump.
-
-        Unconditionally jump to an extended basic block, passing the specified
-        EBB arguments. The number and types of arguments must match the
-        destination EBB.
-        """,
-        ins=(EBB, args), is_branch=True, is_terminator=True)
-
-fallthrough = Instruction(
-        'fallthrough', r"""
-        Fall through to the next EBB.
-
-        This is the same as :inst:`jump`, except the destination EBB must be
-        the next one in the layout.
-
-        Jumps are turned into fall-through instructions by the branch
-        relaxation pass. There is no reason to use this instruction outside
-        that pass.
-        """,
-        ins=(EBB, args), is_branch=True, is_terminator=True)
-
-brz = Instruction(
-        'brz', r"""
-        Branch when zero.
-
-        If ``c`` is a :type:`b1` value, take the branch when ``c`` is false. If
-        ``c`` is an integer value, take the branch when ``c = 0``.
-        """,
-        ins=(c, EBB, args), is_branch=True)
-
-brnz = Instruction(
-        'brnz', r"""
-        Branch when non-zero.
-
-        If ``c`` is a :type:`b1` value, take the branch when ``c`` is true. If
-        ``c`` is an integer value, take the branch when ``c != 0``.
-        """,
-        ins=(c, EBB, args), is_branch=True)
-
-br_icmp = Instruction(
-        'br_icmp', r"""
-        Compare scalar integers and branch.
-
-        Compare ``x`` and ``y`` in the same way as the :inst:`icmp` instruction
-        and take the branch if the condition is true::
-
-            br_icmp ugt v1, v2, ebb4(v5, v6)
-
-        is semantically equivalent to::
-
-            v10 = icmp ugt, v1, v2
-            brnz v10, ebb4(v5, v6)
-
-        Some RISC architectures like MIPS and RISC-V provide instructions that
-        implement all or some of the condition codes. The instruction can also
-        be used to represent *macro-op fusion* on architectures like Intel's.
-        """,
-        ins=(Cond, x, y, EBB, args), is_branch=True)
-
-f = Operand('f', iflags)
-
-brif = Instruction(
-        'brif', r"""
-        Branch when condition is true in integer CPU flags.
-        """,
-        ins=(Cond, f, EBB, args), is_branch=True)
-
-Cond = Operand('Cond', floatcc)
-f = Operand('f', fflags)
-
-brff = Instruction(
-        'brff', r"""
-        Branch when condition is true in floating point CPU flags.
-        """,
-        ins=(Cond, f, EBB, args), is_branch=True)
-
-x = Operand('x', iB, doc='index into jump table')
-Entry = TypeVar('Entry', 'A scalar integer type', ints=True)
-entry = Operand('entry', Entry, doc='entry of jump table')
-JT = Operand('JT', entities.jump_table)
-br_table = Instruction(
-        'br_table', r"""
-        Indirect branch via jump table.
-
-        Use ``x`` as an unsigned index into the jump table ``JT``. If a jump
-        table entry is found, branch to the corresponding EBB. If no entry was
-        found or the index is out-of-bounds, branch to the given default EBB.
-
-        Note that this branch instruction can't pass arguments to the targeted
-        blocks. Split critical edges as needed to work around this.
-
-        Do not confuse this with "tables" in WebAssembly. ``br_table`` is for
-        jump tables with destinations within the current function only -- think
-        of a ``match`` in Rust or a ``switch`` in C.  If you want to call a
-        function in a dynamic library, that will typically use
-        ``call_indirect``.
-        """,
-        ins=(x, EBB, JT), is_branch=True, is_terminator=True)
-
-Size = Operand('Size', uimm8, 'Size in bytes')
-jump_table_entry = Instruction(
-    'jump_table_entry', r"""
-    Get an entry from a jump table.
-
-    Load a serialized ``entry`` from a jump table ``JT`` at a given index
-    ``addr`` with a specific ``Size``. The retrieved entry may need to be
-    decoded after loading, depending upon the jump table type used.
-
-    Currently, the only type supported is entries which are relative to the
-    base of the jump table.
-    """,
-    ins=(x, addr, Size, JT), outs=entry, can_load=True)
-
-jump_table_base = Instruction(
-    'jump_table_base', r"""
-    Get the absolute base address of a jump table.
-
-    This is used for jump tables wherein the entries are stored relative to
-    the base of jump table. In order to use these, generated code should first
-    load an entry using ``jump_table_entry``, then use this instruction to add
-    the relative base back to it.
-    """,
-    ins=JT, outs=addr)
-
-indirect_jump_table_br = Instruction(
-    'indirect_jump_table_br', r"""
-    Branch indirectly via a jump table entry.
-
-    Unconditionally jump via a jump table entry that was previously loaded
-    with the ``jump_table_entry`` instruction.
-    """,
-    ins=(addr, JT),
-    is_branch=True, is_indirect_branch=True, is_terminator=True)
-
-debugtrap = Instruction('debugtrap', r"""
-    Encodes an assembly debug trap.
-    """, can_load=True, can_store=True, other_side_effects=True)
-
-code = Operand('code', trapcode)
-trap = Instruction(
-        'trap', r"""
-        Terminate execution unconditionally.
-        """,
-        ins=code, is_terminator=True, can_trap=True)
-
-trapz = Instruction(
-        'trapz', r"""
-        Trap when zero.
-
-        if ``c`` is non-zero, execution continues at the following instruction.
-        """,
-        ins=(c, code), can_trap=True)
-
-trapnz = Instruction(
-        'trapnz', r"""
-        Trap when non-zero.
-
-        if ``c`` is zero, execution continues at the following instruction.
-        """,
-        ins=(c, code), can_trap=True)
-
-Cond = Operand('Cond', intcc)
-f = Operand('f', iflags)
-
-trapif = Instruction(
-        'trapif', r"""
-        Trap when condition is true in integer CPU flags.
-        """,
-        ins=(Cond, f, code), can_trap=True)
-
-Cond = Operand('Cond', floatcc)
-f = Operand('f', fflags)
-
-trapff = Instruction(
-        'trapff', r"""
-        Trap when condition is true in floating point CPU flags.
-        """,
-        ins=(Cond, f, code), can_trap=True)
-
-rvals = Operand('rvals', VARIABLE_ARGS, doc='return values')
-
-x_return = Instruction(
-        'return', r"""
-        Return from the function.
-
-        Unconditionally transfer control to the calling function, passing the
-        provided return values. The list of return values must match the
-        function signature's return types.
-        """,
-        ins=rvals, is_return=True, is_terminator=True)
-
-fallthrough_return = Instruction(
-        'fallthrough_return', r"""
-        Return from the function by fallthrough.
-
-        This is a specialized instruction for use where one wants to append
-        a custom epilogue, which will then perform the real return. This
-        instruction has no encoding.
-        """,
-        ins=rvals, is_return=True, is_terminator=True)
-
-FN = Operand(
-        'FN',
-        entities.func_ref,
-        doc='function to call, declared by :inst:`function`')
-args = Operand('args', VARIABLE_ARGS, doc='call arguments')
-
-call = Instruction(
-        'call', r"""
-        Direct function call.
-
-        Call a function which has been declared in the preamble. The argument
-        types must match the function's signature.
-        """,
-        ins=(FN, args), outs=rvals, is_call=True)
-
-SIG = Operand('SIG', entities.sig_ref, doc='function signature')
-callee = Operand('callee', iAddr, doc='address of function to call')
-
-call_indirect = Instruction(
-        'call_indirect', r"""
-        Indirect function call.
-
-        Call the function pointed to by `callee` with the given arguments. The
-        called function must match the specified signature.
-
-        Note that this is different from WebAssembly's ``call_indirect``; the
-        callee is a native address, rather than a table index. For WebAssembly,
-        :inst:`table_addr` and :inst:`load` are used to obtain a native address
-        from a table.
-        """,
-        ins=(SIG, callee, args), outs=rvals, is_call=True)
-
-func_addr = Instruction(
-        'func_addr', r"""
-        Get the address of a function.
-
-        Compute the absolute address of a function declared in the preamble.
-        The returned address can be used as a ``callee`` argument to
-        :inst:`call_indirect`. This is also a method for calling functions that
-        are too far away to be addressable by a direct :inst:`call`
-        instruction.
-        """,
-        ins=FN, outs=addr)
-
-#
-# Memory operations
-#
-
-SS = Operand('SS', entities.stack_slot)
-Offset = Operand('Offset', offset32, 'Byte offset from base address')
-x = Operand('x', Mem, doc='Value to be stored')
-a = Operand('a', Mem, doc='Value loaded')
-p = Operand('p', iAddr)
-MemFlags = Operand('MemFlags', memflags)
-args = Operand('args', VARIABLE_ARGS, doc='Address arguments')
-
-load = Instruction(
-        'load', r"""
-        Load from memory at ``p + Offset``.
-
-        This is a polymorphic instruction that can load any value type which
-        has a memory representation.
-        """,
-        ins=(MemFlags, p, Offset), outs=a, can_load=True)
-
-load_complex = Instruction(
-        'load_complex', r"""
-        Load from memory at ``sum(args) + Offset``.
-
-        This is a polymorphic instruction that can load any value type which
-        has a memory representation.
-        """,
-        ins=(MemFlags, args, Offset), outs=a, can_load=True)
-
-store = Instruction(
-        'store', r"""
-        Store ``x`` to memory at ``p + Offset``.
-
-        This is a polymorphic instruction that can store any value type with a
-        memory representation.
-        """,
-        ins=(MemFlags, x, p, Offset), can_store=True)
-
-store_complex = Instruction(
-        'store_complex', r"""
-        Store ``x`` to memory at ``sum(args) + Offset``.
-
-        This is a polymorphic instruction that can store any value type with a
-        memory representation.
-        """,
-        ins=(MemFlags, x, args, Offset), can_store=True)
-
-
-iExt8 = TypeVar(
-        'iExt8', 'An integer type with more than 8 bits',
-        ints=(16, 64))
-x = Operand('x', iExt8)
-a = Operand('a', iExt8)
-
-uload8 = Instruction(
-        'uload8', r"""
-        Load 8 bits from memory at ``p + Offset`` and zero-extend.
-
-        This is equivalent to ``load.i8`` followed by ``uextend``.
-        """,
-        ins=(MemFlags, p, Offset), outs=a, can_load=True)
-
-uload8_complex = Instruction(
-        'uload8_complex', r"""
-        Load 8 bits from memory at ``sum(args) + Offset`` and zero-extend.
-
-        This is equivalent to ``load.i8`` followed by ``uextend``.
-        """,
-        ins=(MemFlags, args, Offset), outs=a, can_load=True)
-
-sload8 = Instruction(
-        'sload8', r"""
-        Load 8 bits from memory at ``p + Offset`` and sign-extend.
-
-        This is equivalent to ``load.i8`` followed by ``sextend``.
-        """,
-        ins=(MemFlags, p, Offset), outs=a, can_load=True)
-
-sload8_complex = Instruction(
-        'sload8_complex', r"""
-        Load 8 bits from memory at ``sum(args) + Offset`` and sign-extend.
-
-        This is equivalent to ``load.i8`` followed by ``sextend``.
-        """,
-        ins=(MemFlags, args, Offset), outs=a, can_load=True)
-
-istore8 = Instruction(
-        'istore8', r"""
-        Store the low 8 bits of ``x`` to memory at ``p + Offset``.
-
-        This is equivalent to ``ireduce.i8`` followed by ``store.i8``.
-        """,
-        ins=(MemFlags, x, p, Offset), can_store=True)
-
-istore8_complex = Instruction(
-        'istore8_complex', r"""
-        Store the low 8 bits of ``x`` to memory at ``sum(args) + Offset``.
-
-        This is equivalent to ``ireduce.i8`` followed by ``store.i8``.
-        """,
-        ins=(MemFlags, x, args, Offset), can_store=True)
-
-iExt16 = TypeVar(
-        'iExt16', 'An integer type with more than 16 bits',
-        ints=(32, 64))
-x = Operand('x', iExt16)
-a = Operand('a', iExt16)
-
-uload16 = Instruction(
-        'uload16', r"""
-        Load 16 bits from memory at ``p + Offset`` and zero-extend.
-
-        This is equivalent to ``load.i16`` followed by ``uextend``.
-        """,
-        ins=(MemFlags, p, Offset), outs=a, can_load=True)
-
-uload16_complex = Instruction(
-        'uload16_complex', r"""
-        Load 16 bits from memory at ``sum(args) + Offset`` and zero-extend.
-
-        This is equivalent to ``load.i16`` followed by ``uextend``.
-        """,
-        ins=(MemFlags, args, Offset), outs=a, can_load=True)
-
-sload16 = Instruction(
-        'sload16', r"""
-        Load 16 bits from memory at ``p + Offset`` and sign-extend.
-
-        This is equivalent to ``load.i16`` followed by ``sextend``.
-        """,
-        ins=(MemFlags, p, Offset), outs=a, can_load=True)
-
-sload16_complex = Instruction(
-        'sload16_complex', r"""
-        Load 16 bits from memory at ``sum(args) + Offset`` and sign-extend.
-
-        This is equivalent to ``load.i16`` followed by ``sextend``.
-        """,
-        ins=(MemFlags, args, Offset), outs=a, can_load=True)
-
-istore16 = Instruction(
-        'istore16', r"""
-        Store the low 16 bits of ``x`` to memory at ``p + Offset``.
-
-        This is equivalent to ``ireduce.i16`` followed by ``store.i16``.
-        """,
-        ins=(MemFlags, x, p, Offset), can_store=True)
-
-istore16_complex = Instruction(
-        'istore16_complex', r"""
-        Store the low 16 bits of ``x`` to memory at ``sum(args) + Offset``.
-
-        This is equivalent to ``ireduce.i16`` followed by ``store.i16``.
-        """,
-        ins=(MemFlags, x, args, Offset), can_store=True)
-
-iExt32 = TypeVar(
-        'iExt32', 'An integer type with more than 32 bits',
-        ints=(64, 64))
-x = Operand('x', iExt32)
-a = Operand('a', iExt32)
-
-uload32 = Instruction(
-        'uload32', r"""
-        Load 32 bits from memory at ``p + Offset`` and zero-extend.
-
-        This is equivalent to ``load.i32`` followed by ``uextend``.
-        """,
-        ins=(MemFlags, p, Offset), outs=a, can_load=True)
-
-uload32_complex = Instruction(
-        'uload32_complex', r"""
-        Load 32 bits from memory at ``sum(args) + Offset`` and zero-extend.
-
-        This is equivalent to ``load.i32`` followed by ``uextend``.
-        """,
-        ins=(MemFlags, args, Offset), outs=a, can_load=True)
-
-sload32 = Instruction(
-        'sload32', r"""
-        Load 32 bits from memory at ``p + Offset`` and sign-extend.
-
-        This is equivalent to ``load.i32`` followed by ``sextend``.
-        """,
-        ins=(MemFlags, p, Offset), outs=a, can_load=True)
-
-sload32_complex = Instruction(
-        'sload32_complex', r"""
-        Load 32 bits from memory at ``sum(args) + Offset`` and sign-extend.
-
-        This is equivalent to ``load.i32`` followed by ``sextend``.
-        """,
-        ins=(MemFlags, args, Offset), outs=a, can_load=True)
-
-istore32 = Instruction(
-        'istore32', r"""
-        Store the low 32 bits of ``x`` to memory at ``p + Offset``.
-
-        This is equivalent to ``ireduce.i32`` followed by ``store.i32``.
-        """,
-        ins=(MemFlags, x, p, Offset), can_store=True)
-
-istore32_complex = Instruction(
-        'istore32_complex', r"""
-        Store the low 32 bits of ``x`` to memory at ``sum(args) + Offset``.
-
-        This is equivalent to ``ireduce.i32`` followed by ``store.i32``.
-        """,
-        ins=(MemFlags, x, args, Offset), can_store=True)
-
-x = Operand('x', Mem, doc='Value to be stored')
-a = Operand('a', Mem, doc='Value loaded')
-Offset = Operand('Offset', offset32, 'In-bounds offset into stack slot')
-
-stack_load = Instruction(
-        'stack_load', r"""
-        Load a value from a stack slot at the constant offset.
-
-        This is a polymorphic instruction that can load any value type which
-        has a memory representation.
-
-        The offset is an immediate constant, not an SSA value. The memory
-        access cannot go out of bounds, i.e.
-        :math:`sizeof(a) + Offset <= sizeof(SS)`.
-        """,
-        ins=(SS, Offset), outs=a, can_load=True)
-
-stack_store = Instruction(
-        'stack_store', r"""
-        Store a value to a stack slot at a constant offset.
-
-        This is a polymorphic instruction that can store any value type with a
-        memory representation.
-
-        The offset is an immediate constant, not an SSA value. The memory
-        access cannot go out of bounds, i.e.
-        :math:`sizeof(a) + Offset <= sizeof(SS)`.
-        """,
-        ins=(x, SS, Offset), can_store=True)
-
-stack_addr = Instruction(
-        'stack_addr', r"""
-        Get the address of a stack slot.
-
-        Compute the absolute address of a byte in a stack slot. The offset must
-        refer to a byte inside the stack slot:
-        :math:`0 <= Offset < sizeof(SS)`.
-        """,
-        ins=(SS, Offset), outs=addr)
-
-#
-# Global values.
-#
-
-GV = Operand('GV', entities.global_value)
-
-global_value = Instruction(
-        'global_value', r"""
-        Compute the value of global GV.
-        """,
-        ins=GV, outs=a)
-
-# A specialized form of global_value instructions that only handles
-# symbolic names.
-symbol_value = Instruction(
-        'symbol_value', r"""
-        Compute the value of global GV, which is a symbolic value.
-        """,
-        ins=GV, outs=a)
-
-#
-# WebAssembly bounds-checked heap accesses.
-#
-
-HeapOffset = TypeVar('HeapOffset', 'An unsigned heap offset', ints=(32, 64))
-
-H = Operand('H', entities.heap)
-p = Operand('p', HeapOffset)
-Size = Operand('Size', uimm32, 'Size in bytes')
-
-heap_addr = Instruction(
-        'heap_addr', r"""
-        Bounds check and compute absolute address of heap memory.
-
-        Verify that the offset range ``p .. p + Size - 1`` is in bounds for the
-        heap H, and generate an absolute address that is safe to dereference.
-
-        1. If ``p + Size`` is not greater than the heap bound, return an
-           absolute address corresponding to a byte offset of ``p`` from the
-           heap's base address.
-        2. If ``p + Size`` is greater than the heap bound, generate a trap.
-        """,
-        ins=(H, p, Size), outs=addr)
-
-#
-# WebAssembly bounds-checked table accesses.
-#
-
-TableOffset = TypeVar('TableOffset', 'An unsigned table offset', ints=(32, 64))
-
-T = Operand('T', entities.table)
-p = Operand('p', TableOffset)
-Offset = Operand('Offset', offset32, 'Byte offset from element address')
-
-table_addr = Instruction(
-        'table_addr', r"""
-        Bounds check and compute absolute address of a table entry.
-
-        Verify that the offset ``p`` is in bounds for the table T, and generate
-        an absolute address that is safe to dereference.
-
-        ``Offset`` must be less than the size of a table element.
-
-        1. If ``p`` is not greater than the table bound, return an absolute
-           address corresponding to a byte offset of ``p`` from the table's
-           base address.
-        2. If ``p`` is greater than the table bound, generate a trap.
-        """,
-        ins=(T, p, Offset), outs=addr)
-
-
-#
-# Materializing constants.
-#
-
-N = Operand('N', imm64)
-a = Operand('a', Int, doc='A constant integer scalar or vector value')
-iconst = Instruction(
-        'iconst', r"""
-        Integer constant.
-
-        Create a scalar integer SSA value with an immediate constant value, or
-        an integer vector where all the lanes have the same value.
-        """,
-        ins=N, outs=a)
-
-N = Operand('N', ieee32)
-a = Operand('a', f32, doc='A constant f32 scalar value')
-f32const = Instruction(
-        'f32const', r"""
-        Floating point constant.
-
-        Create a :type:`f32` SSA value with an immediate constant value.
-        """,
-        ins=N, outs=a)
-
-N = Operand('N', ieee64)
-a = Operand('a', f64, doc='A constant f64 scalar value')
-f64const = Instruction(
-        'f64const', r"""
-        Floating point constant.
-
-        Create a :type:`f64` SSA value with an immediate constant value.
-        """,
-        ins=N, outs=a)
-
-N = Operand('N', boolean)
-a = Operand('a', Bool, doc='A constant boolean scalar or vector value')
-bconst = Instruction(
-        'bconst', r"""
-        Boolean constant.
-
-        Create a scalar boolean SSA value with an immediate constant value, or
-        a boolean vector where all the lanes have the same value.
-        """,
-        ins=N, outs=a)
-
-#
-# Generics.
-#
-
-nop = Instruction(
-        'nop', r"""
-        Just a dummy instruction
-
-        Note: this doesn't compile to a machine code nop
-        """)
-
-c = Operand('c', Testable, doc='Controlling value to test')
-x = Operand('x', Any, doc='Value to use when `c` is true')
-y = Operand('y', Any, doc='Value to use when `c` is false')
-a = Operand('a', Any)
-
-select = Instruction(
-        'select', r"""
-        Conditional select.
-
-        This instruction selects whole values. Use :inst:`vselect` for
-        lane-wise selection.
-        """,
-        ins=(c, x, y), outs=a)
-
-cc = Operand('cc', intcc, doc='Controlling condition code')
-flags = Operand('flags', iflags, doc='The machine\'s flag register')
-
-selectif = Instruction(
-        'selectif', r"""
-        Conditional select, dependent on integer condition codes.
-        """,
-        ins=(cc, flags, x, y), outs=a)
-
-x = Operand('x', Any)
-
-copy = Instruction(
-        'copy', r"""
-        Register-register copy.
-
-        This instruction copies its input, preserving the value type.
-
-        A pure SSA-form program does not need to copy values, but this
-        instruction is useful for representing intermediate stages during
-        instruction transformations, and the register allocator needs a way of
-        representing register copies.
-        """,
-        ins=x, outs=a)
-
-spill = Instruction(
-        'spill', r"""
-        Spill a register value to a stack slot.
-
-        This instruction behaves exactly like :inst:`copy`, but the result
-        value is assigned to a spill slot.
-        """,
-        ins=x, outs=a, can_store=True)
-
-fill = Instruction(
-        'fill', r"""
-        Load a register value from a stack slot.
-
-        This instruction behaves exactly like :inst:`copy`, but creates a new
-        SSA value for the spilled input value.
-        """,
-        ins=x, outs=a, can_load=True)
-
-src = Operand('src', regunit)
-dst = Operand('dst', regunit)
-
-regmove = Instruction(
-        'regmove', r"""
-        Temporarily divert ``x`` from ``src`` to ``dst``.
-
-        This instruction moves the location of a value from one register to
-        another without creating a new SSA value. It is used by the register
-        allocator to temporarily rearrange register assignments in order to
-        satisfy instruction constraints.
-
-        The register diversions created by this instruction must be undone
-        before the value leaves the EBB. At the entry to a new EBB, all live
-        values must be in their originally assigned registers.
-        """,
-        ins=(x, src, dst),
-        other_side_effects=True)
-
-copy_special = Instruction(
-        'copy_special', r"""
-        Copies the contents of ''src'' register to ''dst'' register.
-
-        This instructions copies the contents of one register to another
-        register without involving any SSA values. This is used for copying
-        special registers, e.g. copying the stack register to the frame
-        register in a function prologue.
-        """,
-        ins=(src, dst),
-        other_side_effects=True)
-
-copy_nop = Instruction(
-        'copy_nop', r"""
-        Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
-        into a no-op.  This instruction is for use only within Cranelift
-        itself.
-
-        This instruction copies its input, preserving the value type.
-        """,
-        ins=x, outs=a)
-
-delta = Operand('delta', Int)
-adjust_sp_down = Instruction(
-    'adjust_sp_down', r"""
-    Subtracts ``delta`` offset value from the stack pointer register.
-
-    This instruction is used to adjust the stack pointer by a dynamic amount.
-    """,
-    ins=(delta,),
-    other_side_effects=True)
-
-StackOffset = Operand('Offset', imm64, 'Offset from current stack pointer')
-adjust_sp_up_imm = Instruction(
-    'adjust_sp_up_imm', r"""
-    Adds ``Offset`` immediate offset value to the stack pointer register.
-
-    This instruction is used to adjust the stack pointer, primarily in function
-    prologues and epilogues. ``Offset`` is constrained to the size of a signed
-    32-bit integer.
-    """,
-    ins=(StackOffset,),
-    other_side_effects=True)
-
-StackOffset = Operand('Offset', imm64, 'Offset from current stack pointer')
-adjust_sp_down_imm = Instruction(
-    'adjust_sp_down_imm', r"""
-    Subtracts ``Offset`` immediate offset value from the stack pointer
-    register.
-
-    This instruction is used to adjust the stack pointer, primarily in function
-    prologues and epilogues. ``Offset`` is constrained to the size of a signed
-    32-bit integer.
-    """,
-    ins=(StackOffset,),
-    other_side_effects=True)
-
-f = Operand('f', iflags)
-
-ifcmp_sp = Instruction(
-    'ifcmp_sp', r"""
-    Compare ``addr`` with the stack pointer and set the CPU flags.
-
-    This is like :inst:`ifcmp` where ``addr`` is the LHS operand and the stack
-    pointer is the RHS.
-    """,
-    ins=addr, outs=f)
-
-regspill = Instruction(
-        'regspill', r"""
-        Temporarily divert ``x`` from ``src`` to ``SS``.
-
-        This instruction moves the location of a value from a register to a
-        stack slot without creating a new SSA value. It is used by the register
-        allocator to temporarily rearrange register assignments in order to
-        satisfy instruction constraints.
-
-        See also :inst:`regmove`.
-        """,
-        ins=(x, src, SS),
-        other_side_effects=True)
-
-
-regfill = Instruction(
-        'regfill', r"""
-        Temporarily divert ``x`` from ``SS`` to ``dst``.
-
-        This instruction moves the location of a value from a stack slot to a
-        register without creating a new SSA value. It is used by the register
-        allocator to temporarily rearrange register assignments in order to
-        satisfy instruction constraints.
-
-        See also :inst:`regmove`.
-        """,
-        ins=(x, SS, dst),
-        other_side_effects=True)
-#
-# Vector operations
-#
-
-x = Operand('x', TxN, doc='Vector to split')
-lo = Operand('lo', TxN.half_vector(), doc='Low-numbered lanes of `x`')
-hi = Operand('hi', TxN.half_vector(), doc='High-numbered lanes of `x`')
-
-vsplit = Instruction(
-        'vsplit', r"""
-        Split a vector into two halves.
-
-        Split the vector `x` into two separate values, each containing half of
-        the lanes from ``x``. The result may be two scalars if ``x`` only had
-        two lanes.
-        """,
-        ins=x, outs=(lo, hi), is_ghost=True)
-
-Any128 = TypeVar(
-        'Any128', 'Any scalar or vector type with as most 128 lanes',
-        ints=True, floats=True, bools=True, scalars=True, simd=(1, 128))
-x = Operand('x', Any128, doc='Low-numbered lanes')
-y = Operand('y', Any128, doc='High-numbered lanes')
-a = Operand('a', Any128.double_vector(), doc='Concatenation of `x` and `y`')
-
-vconcat = Instruction(
-        'vconcat', r"""
-        Vector concatenation.
-
-        Return a vector formed by concatenating ``x`` and ``y``. The resulting
-        vector type has twice as many lanes as each of the inputs. The lanes of
-        ``x`` appear as the low-numbered lanes, and the lanes of ``y`` become
-        the high-numbered lanes of ``a``.
-
-        It is possible to form a vector by concatenating two scalars.
-        """,
-        ins=(x, y), outs=a, is_ghost=True)
-
-c = Operand('c', TxN.as_bool(), doc='Controlling vector')
-x = Operand('x', TxN, doc='Value to use where `c` is true')
-y = Operand('y', TxN, doc='Value to use where `c` is false')
-a = Operand('a', TxN)
-
-vselect = Instruction(
-        'vselect', r"""
-        Vector lane select.
-
-        Select lanes from ``x`` or ``y`` controlled by the lanes of the boolean
-        vector ``c``.
-        """,
-        ins=(c, x, y), outs=a)
-
-x = Operand('x', TxN.lane_of())
-
-splat = Instruction(
-        'splat', r"""
-        Vector splat.
-
-        Return a vector whose lanes are all ``x``.
-        """,
-        ins=x, outs=a)
-
-x = Operand('x', TxN, doc='SIMD vector to modify')
-y = Operand('y', TxN.lane_of(), doc='New lane value')
-Idx = Operand('Idx', uimm8, doc='Lane index')
-
-insertlane = Instruction(
-        'insertlane', r"""
-        Insert ``y`` as lane ``Idx`` in x.
-
-        The lane index, ``Idx``, is an immediate value, not an SSA value. It
-        must indicate a valid lane index for the type of ``x``.
-        """,
-        ins=(x, Idx, y), outs=a)
-
-x = Operand('x', TxN)
-a = Operand('a', TxN.lane_of())
-
-extractlane = Instruction(
-        'extractlane', r"""
-        Extract lane ``Idx`` from ``x``.
-
-        The lane index, ``Idx``, is an immediate value, not an SSA value. It
-        must indicate a valid lane index for the type of ``x``.
-        """,
-        ins=(x, Idx), outs=a)
-
-#
-# Integer arithmetic
-#
-
-a = Operand('a', Int.as_bool())
-Cond = Operand('Cond', intcc)
-x = Operand('x', Int)
-y = Operand('y', Int)
-
-icmp = Instruction(
-        'icmp', r"""
-        Integer comparison.
-
-        The condition code determines if the operands are interpreted as signed
-        or unsigned integers.
-
-        ====== ======== =========
-        Signed Unsigned Condition
-        ====== ======== =========
-        eq     eq       Equal
-        ne     ne       Not equal
-        slt    ult      Less than
-        sge    uge      Greater than or equal
-        sgt    ugt      Greater than
-        sle    ule      Less than or equal
-        ====== ======== =========
-
-        When this instruction compares integer vectors, it returns a boolean
-        vector of lane-wise comparisons.
-        """,
-        ins=(Cond, x, y), outs=a)
-
-a = Operand('a', b1)
-x = Operand('x', iB)
-Y = Operand('Y', imm64)
-
-icmp_imm = Instruction(
-        'icmp_imm', r"""
-        Compare scalar integer to a constant.
-
-        This is the same as the :inst:`icmp` instruction, except one operand is
-        an immediate constant.
-
-        This instruction can only compare scalars. Use :inst:`icmp` for
-        lane-wise vector comparisons.
-        """,
-        ins=(Cond, x, Y), outs=a)
-
-f = Operand('f', iflags)
-x = Operand('x', iB)
-y = Operand('y', iB)
-
-ifcmp = Instruction(
-        'ifcmp', r"""
-        Compare scalar integers and return flags.
-
-        Compare two scalar integer values and return integer CPU flags
-        representing the result.
-        """,
-        ins=(x, y), outs=f)
-
-ifcmp_imm = Instruction(
-        'ifcmp_imm', r"""
-        Compare scalar integer to a constant and return flags.
-
-        Like :inst:`icmp_imm`, but returns integer CPU flags instead of testing
-        a specific condition code.
-        """,
-        ins=(x, Y), outs=f)
-
-a = Operand('a', Int)
-x = Operand('x', Int)
-y = Operand('y', Int)
-
-iadd = Instruction(
-        'iadd', r"""
-        Wrapping integer addition: :math:`a := x + y \pmod{2^B}`.
-
-        This instruction does not depend on the signed/unsigned interpretation
-        of the operands.
-        """,
-        ins=(x, y), outs=a)
-
-isub = Instruction(
-        'isub', r"""
-        Wrapping integer subtraction: :math:`a := x - y \pmod{2^B}`.
-
-        This instruction does not depend on the signed/unsigned interpretation
-        of the operands.
-        """,
-        ins=(x, y), outs=a)
-
-imul = Instruction(
-        'imul', r"""
-        Wrapping integer multiplication: :math:`a := x y \pmod{2^B}`.
-
-        This instruction does not depend on the signed/unsigned interpretation
-        of the
-        operands.
-
-        Polymorphic over all integer types (vector and scalar).
-        """,
-        ins=(x, y), outs=a)
-
-umulhi = Instruction(
-        'umulhi', r"""
-        Unsigned integer multiplication, producing the high half of a
-        double-length result.
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, y), outs=a)
-
-smulhi = Instruction(
-        'smulhi', """
-        Signed integer multiplication, producing the high half of a
-        double-length result.
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, y), outs=a)
-
-udiv = Instruction(
-        'udiv', r"""
-        Unsigned integer division: :math:`a := \lfloor {x \over y} \rfloor`.
-
-        This operation traps if the divisor is zero.
-        """,
-        ins=(x, y), outs=a, can_trap=True)
-
-sdiv = Instruction(
-        'sdiv', r"""
-        Signed integer division rounded toward zero: :math:`a := sign(xy)
-        \lfloor {|x| \over |y|}\rfloor`.
-
-        This operation traps if the divisor is zero, or if the result is not
-        representable in :math:`B` bits two's complement. This only happens
-        when :math:`x = -2^{B-1}, y = -1`.
-        """,
-        ins=(x, y), outs=a, can_trap=True)
-
-urem = Instruction(
-        'urem', """
-        Unsigned integer remainder.
-
-        This operation traps if the divisor is zero.
-        """,
-        ins=(x, y), outs=a, can_trap=True)
-
-srem = Instruction(
-        'srem', """
-        Signed integer remainder. The result has the sign of the dividend.
-
-        This operation traps if the divisor is zero.
-        """,
-        ins=(x, y), outs=a, can_trap=True)
-
-a = Operand('a', iB)
-x = Operand('x', iB)
-Y = Operand('Y', imm64)
-
-iadd_imm = Instruction(
-        'iadd_imm', """
-        Add immediate integer.
-
-        Same as :inst:`iadd`, but one operand is an immediate constant.
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, Y), outs=a)
-
-imul_imm = Instruction(
-        'imul_imm', """
-        Integer multiplication by immediate constant.
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, Y), outs=a)
-
-udiv_imm = Instruction(
-        'udiv_imm', """
-        Unsigned integer division by an immediate constant.
-
-        This operation traps if the divisor is zero.
-        """,
-        ins=(x, Y), outs=a)
-
-sdiv_imm = Instruction(
-        'sdiv_imm', """
-        Signed integer division by an immediate constant.
-
-        This operation traps if the divisor is zero, or if the result is not
-        representable in :math:`B` bits two's complement. This only happens
-        when :math:`x = -2^{B-1}, Y = -1`.
-        """,
-        ins=(x, Y), outs=a)
-
-urem_imm = Instruction(
-        'urem_imm', """
-        Unsigned integer remainder with immediate divisor.
-
-        This operation traps if the divisor is zero.
-        """,
-        ins=(x, Y), outs=a)
-
-srem_imm = Instruction(
-        'srem_imm', """
-        Signed integer remainder with immediate divisor.
-
-        This operation traps if the divisor is zero.
-        """,
-        ins=(x, Y), outs=a)
-
-irsub_imm = Instruction(
-        'irsub_imm', """
-        Immediate reverse wrapping subtraction: :math:`a := Y - x \\pmod{2^B}`.
-
-        Also works as integer negation when :math:`Y = 0`. Use :inst:`iadd_imm`
-        with a negative immediate operand for the reverse immediate
-        subtraction.
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, Y), outs=a)
-
-#
-# Integer arithmetic with carry and/or borrow.
-#
-a = Operand('a', iB)
-x = Operand('x', iB)
-y = Operand('y', iB)
-c_in = Operand('c_in', b1, doc="Input carry flag")
-c_out = Operand('c_out', b1, doc="Output carry flag")
-b_in = Operand('b_in', b1, doc="Input borrow flag")
-b_out = Operand('b_out', b1, doc="Output borrow flag")
-
-iadd_cin = Instruction(
-        'iadd_cin', r"""
-        Add integers with carry in.
-
-        Same as :inst:`iadd` with an additional carry input. Computes:
-
-        .. math::
-
-            a = x + y + c_{in} \pmod 2^B
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, y, c_in), outs=a)
-
-iadd_cout = Instruction(
-        'iadd_cout', r"""
-        Add integers with carry out.
-
-        Same as :inst:`iadd` with an additional carry output.
-
-        .. math::
-
-            a &= x + y \pmod 2^B \\
-            c_{out} &= x+y >= 2^B
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, y), outs=(a, c_out))
-
-iadd_carry = Instruction(
-        'iadd_carry', r"""
-        Add integers with carry in and out.
-
-        Same as :inst:`iadd` with an additional carry input and output.
-
-        .. math::
-
-            a &= x + y + c_{in} \pmod 2^B \\
-            c_{out} &= x + y + c_{in} >= 2^B
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, y, c_in), outs=(a, c_out))
-
-isub_bin = Instruction(
-        'isub_bin', r"""
-        Subtract integers with borrow in.
-
-        Same as :inst:`isub` with an additional borrow flag input. Computes:
-
-        .. math::
-
-            a = x - (y + b_{in}) \pmod 2^B
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, y, b_in), outs=a)
-
-isub_bout = Instruction(
-        'isub_bout', r"""
-        Subtract integers with borrow out.
-
-        Same as :inst:`isub` with an additional borrow flag output.
-
-        .. math::
-
-            a &= x - y \pmod 2^B \\
-            b_{out} &= x < y
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, y), outs=(a, b_out))
-
-isub_borrow = Instruction(
-        'isub_borrow', r"""
-        Subtract integers with borrow in and out.
-
-        Same as :inst:`isub` with an additional borrow flag input and output.
-
-        .. math::
-
-            a &= x - (y + b_{in}) \pmod 2^B \\
-            b_{out} &= x < y + b_{in}
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, y, b_in), outs=(a, b_out))
-
-#
-# Bitwise operations.
-#
-
-# TODO: Which types should permit boolean operations? Any reason to restrict?
-bits = TypeVar(
-        'bits', 'Any integer, float, or boolean scalar or vector type',
-        ints=True, floats=True, bools=True, scalars=True, simd=True)
-
-x = Operand('x', bits)
-y = Operand('y', bits)
-a = Operand('a', bits)
-
-band = Instruction(
-        'band', """
-        Bitwise and.
-        """,
-        ins=(x, y), outs=a)
-
-bor = Instruction(
-        'bor', """
-        Bitwise or.
-        """,
-        ins=(x, y), outs=a)
-
-bxor = Instruction(
-        'bxor', """
-        Bitwise xor.
-        """,
-        ins=(x, y), outs=a)
-
-bnot = Instruction(
-        'bnot', """
-        Bitwise not.
-        """,
-        ins=x, outs=a)
-
-band_not = Instruction(
-        'band_not', """
-        Bitwise and not.
-
-        Computes `x & ~y`.
-        """,
-        ins=(x, y), outs=a)
-
-bor_not = Instruction(
-        'bor_not', """
-        Bitwise or not.
-
-        Computes `x | ~y`.
-        """,
-        ins=(x, y), outs=a)
-
-bxor_not = Instruction(
-        'bxor_not', """
-        Bitwise xor not.
-
-        Computes `x ^ ~y`.
-        """,
-        ins=(x, y), outs=a)
-
-# Bitwise binary ops with immediate arg.
-x = Operand('x', iB)
-Y = Operand('Y', imm64)
-a = Operand('a', iB)
-
-band_imm = Instruction(
-        'band_imm', """
-        Bitwise and with immediate.
-
-        Same as :inst:`band`, but one operand is an immediate constant.
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, Y), outs=a)
-
-bor_imm = Instruction(
-        'bor_imm', """
-        Bitwise or with immediate.
-
-        Same as :inst:`bor`, but one operand is an immediate constant.
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, Y), outs=a)
-
-bxor_imm = Instruction(
-        'bxor_imm', """
-        Bitwise xor with immediate.
-
-        Same as :inst:`bxor`, but one operand is an immediate constant.
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(x, Y), outs=a)
-
-# Shift/rotate.
-x = Operand('x', Int, doc='Scalar or vector value to shift')
-y = Operand('y', iB, doc='Number of bits to shift')
-Y = Operand('Y', imm64)
-
-a = Operand('a', Int)
-
-rotl = Instruction(
-        'rotl', r"""
-        Rotate left.
-
-        Rotate the bits in ``x`` by ``y`` places.
-        """,
-        ins=(x, y), outs=a)
-
-rotr = Instruction(
-        'rotr', r"""
-        Rotate right.
-
-        Rotate the bits in ``x`` by ``y`` places.
-        """,
-        ins=(x, y), outs=a)
-
-rotl_imm = Instruction(
-        'rotl_imm', r"""
-        Rotate left by immediate.
-        """,
-        ins=(x, Y), outs=a)
-
-rotr_imm = Instruction(
-        'rotr_imm', r"""
-        Rotate right by immediate.
-        """,
-        ins=(x, Y), outs=a)
-
-ishl = Instruction(
-        'ishl', r"""
-        Integer shift left. Shift the bits in ``x`` towards the MSB by ``y``
-        places. Shift in zero bits to the LSB.
-
-        The shift amount is masked to the size of ``x``.
-
-        When shifting a B-bits integer type, this instruction computes:
-
-        .. math::
-            s &:= y \pmod B,                \\
-            a &:= x \cdot 2^s \pmod{2^B}.
-        """,
-        ins=(x, y), outs=a)
-
-ushr = Instruction(
-        'ushr', r"""
-        Unsigned shift right. Shift bits in ``x`` towards the LSB by ``y``
-        places, shifting in zero bits to the MSB. Also called a *logical
-        shift*.
-
-        The shift amount is masked to the size of the register.
-
-        When shifting a B-bits integer type, this instruction computes:
-
-        .. math::
-            s &:= y \pmod B,                \\
-            a &:= \lfloor x \cdot 2^{-s} \rfloor.
-        """,
-        ins=(x, y), outs=a)
-
-sshr = Instruction(
-        'sshr', r"""
-        Signed shift right. Shift bits in ``x`` towards the LSB by ``y``
-        places, shifting in sign bits to the MSB. Also called an *arithmetic
-        shift*.
-
-        The shift amount is masked to the size of the register.
-        """,
-        ins=(x, y), outs=a)
-
-ishl_imm = Instruction(
-        'ishl_imm', r"""
-        Integer shift left by immediate.
-
-        The shift amount is masked to the size of ``x``.
-        """,
-        ins=(x, Y), outs=a)
-
-ushr_imm = Instruction(
-        'ushr_imm', r"""
-        Unsigned shift right by immediate.
-
-        The shift amount is masked to the size of the register.
-        """,
-        ins=(x, Y), outs=a)
-
-sshr_imm = Instruction(
-        'sshr_imm', r"""
-        Signed shift right by immediate.
-
-        The shift amount is masked to the size of the register.
-        """,
-        ins=(x, Y), outs=a)
-
-#
-# Bit counting.
-#
-
-x = Operand('x', iB)
-a = Operand('a', iB)
-
-bitrev = Instruction(
-        'bitrev', r"""
-        Reverse the bits of a integer.
-
-        Reverses the bits in ``x``.
-        """,
-        ins=x, outs=a)
-
-clz = Instruction(
-        'clz', r"""
-        Count leading zero bits.
-
-        Starting from the MSB in ``x``, count the number of zero bits before
-        reaching the first one bit. When ``x`` is zero, returns the size of x
-        in bits.
-        """,
-        ins=x, outs=a)
-
-cls = Instruction(
-        'cls', r"""
-        Count leading sign bits.
-
-        Starting from the MSB after the sign bit in ``x``, count the number of
-        consecutive bits identical to the sign bit. When ``x`` is 0 or -1,
-        returns one less than the size of x in bits.
-        """,
-        ins=x, outs=a)
-
-ctz = Instruction(
-        'ctz', r"""
-        Count trailing zeros.
-
-        Starting from the LSB in ``x``, count the number of zero bits before
-        reaching the first one bit. When ``x`` is zero, returns the size of x
-        in bits.
-        """,
-        ins=x, outs=a)
-
-popcnt = Instruction(
-        'popcnt', r"""
-        Population count
-
-        Count the number of one bits in ``x``.
-        """,
-        ins=x, outs=a)
-
-#
-# Floating point.
-#
-
-Float = TypeVar(
-        'Float', 'A scalar or vector floating point number',
-        floats=True, simd=True)
-fB = TypeVar('fB', 'A scalar floating point number', floats=True)
-
-Cond = Operand('Cond', floatcc)
-x = Operand('x', Float)
-y = Operand('y', Float)
-a = Operand('a', Float.as_bool())
-
-fcmp = Instruction(
-        'fcmp', r"""
-        Floating point comparison.
-
-        Two IEEE 754-2008 floating point numbers, `x` and `y`, relate to each
-        other in exactly one of four ways:
-
-        == ==========================================
-        UN Unordered when one or both numbers is NaN.
-        EQ When :math:`x = y`. (And :math:`0.0 = -0.0`).
-        LT When :math:`x < y`.
-        GT When :math:`x > y`.
-        == ==========================================
-
-        The 14 :type:`floatcc` condition codes each correspond to a subset of
-        the four relations, except for the empty set which would always be
-        false, and the full set which would always be true.
-
-        The condition codes are divided into 7 'ordered' conditions which don't
-        include UN, and 7 unordered conditions which all include UN.
-
-        +-------+------------+---------+------------+-------------------------+
-        |Ordered             |Unordered             |Condition                |
-        +=======+============+=========+============+=========================+
-        |ord    |EQ | LT | GT|uno      |UN          |NaNs absent / present.   |
-        +-------+------------+---------+------------+-------------------------+
-        |eq     |EQ          |ueq      |UN | EQ     |Equal                    |
-        +-------+------------+---------+------------+-------------------------+
-        |one    |LT | GT     |ne       |UN | LT | GT|Not equal                |
-        +-------+------------+---------+------------+-------------------------+
-        |lt     |LT          |ult      |UN | LT     |Less than                |
-        +-------+------------+---------+------------+-------------------------+
-        |le     |LT | EQ     |ule      |UN | LT | EQ|Less than or equal       |
-        +-------+------------+---------+------------+-------------------------+
-        |gt     |GT          |ugt      |UN | GT     |Greater than             |
-        +-------+------------+---------+------------+-------------------------+
-        |ge     |GT | EQ     |uge      |UN | GT | EQ|Greater than or equal    |
-        +-------+------------+---------+------------+-------------------------+
-
-        The standard C comparison operators, `<, <=, >, >=`, are all ordered,
-        so they are false if either operand is NaN. The C equality operator,
-        `==`, is ordered, and since inequality is defined as the logical
-        inverse it is *unordered*. They map to the :type:`floatcc` condition
-        codes as follows:
-
-        ==== ====== ============
-        C    `Cond` Subset
-        ==== ====== ============
-        `==` eq     EQ
-        `!=` ne     UN | LT | GT
-        `<`  lt     LT
-        `<=` le     LT | EQ
-        `>`  gt     GT
-        `>=` ge     GT | EQ
-        ==== ====== ============
-
-        This subset of condition codes also corresponds to the WebAssembly
-        floating point comparisons of the same name.
-
-        When this instruction compares floating point vectors, it returns a
-        boolean vector with the results of lane-wise comparisons.
-        """,
-        ins=(Cond, x, y), outs=a)
-
-f = Operand('f', fflags)
-
-ffcmp = Instruction(
-        'ffcmp', r"""
-        Floating point comparison returning flags.
-
-        Compares two numbers like :inst:`fcmp`, but returns floating point CPU
-        flags instead of testing a specific condition.
-        """,
-        ins=(x, y), outs=f)
-
-x = Operand('x', Float)
-y = Operand('y', Float)
-z = Operand('z', Float)
-a = Operand('a', Float, 'Result of applying operator to each lane')
-
-fadd = Instruction(
-        'fadd', r"""
-        Floating point addition.
-        """,
-        ins=(x, y), outs=a)
-
-fsub = Instruction(
-        'fsub', r"""
-        Floating point subtraction.
-        """,
-        ins=(x, y), outs=a)
-
-fmul = Instruction(
-        'fmul', r"""
-        Floating point multiplication.
-        """,
-        ins=(x, y), outs=a)
-
-fdiv = Instruction(
-        'fdiv', r"""
-        Floating point division.
-
-        Unlike the integer division instructions :clif:inst:`sdiv` and
-        :clif:inst:`udiv`, this can't trap. Division by zero is infinity or
-        NaN, depending on the dividend.
-        """,
-        ins=(x, y), outs=a)
-
-sqrt = Instruction(
-        'sqrt', r"""
-        Floating point square root.
-        """,
-        ins=x, outs=a)
-
-fma = Instruction(
-        'fma', r"""
-        Floating point fused multiply-and-add.
-
-        Computes :math:`a := xy+z` without any intermediate rounding of the
-        product.
-        """,
-        ins=(x, y, z), outs=a)
-
-a = Operand('a', Float, '``x`` with its sign bit inverted')
-fneg = Instruction(
-        'fneg', r"""
-        Floating point negation.
-
-        Note that this is a pure bitwise operation.
-        """,
-        ins=x, outs=a)
-
-a = Operand('a', Float, '``x`` with its sign bit cleared')
-fabs = Instruction(
-        'fabs', r"""
-        Floating point absolute value.
-
-        Note that this is a pure bitwise operation.
-        """,
-        ins=x, outs=a)
-
-a = Operand('a', Float, '``x`` with its sign bit changed to that of ``y``')
-fcopysign = Instruction(
-        'fcopysign', r"""
-        Floating point copy sign.
-
-        Note that this is a pure bitwise operation. The sign bit from ``y`` is
-        copied to the sign bit of ``x``.
-        """,
-        ins=(x, y), outs=a)
-
-a = Operand('a', Float, 'The smaller of ``x`` and ``y``')
-
-fmin = Instruction(
-        'fmin', r"""
-        Floating point minimum, propagating NaNs.
-
-        If either operand is NaN, this returns a NaN.
-        """,
-        ins=(x, y), outs=a)
-
-a = Operand('a', Float, 'The larger of ``x`` and ``y``')
-
-fmax = Instruction(
-        'fmax', r"""
-        Floating point maximum, propagating NaNs.
-
-        If either operand is NaN, this returns a NaN.
-        """,
-        ins=(x, y), outs=a)
-
-a = Operand('a', Float, '``x`` rounded to integral value')
-
-ceil = Instruction(
-        'ceil', r"""
-        Round floating point round to integral, towards positive infinity.
-        """,
-        ins=x, outs=a)
-
-floor = Instruction(
-        'floor', r"""
-        Round floating point round to integral, towards negative infinity.
-        """,
-        ins=x, outs=a)
-
-trunc = Instruction(
-        'trunc', r"""
-        Round floating point round to integral, towards zero.
-        """,
-        ins=x, outs=a)
-
-nearest = Instruction(
-        'nearest', r"""
-        Round floating point round to integral, towards nearest with ties to
-        even.
-        """,
-        ins=x, outs=a)
-
-#
-# CPU flag operations
-#
-
-
-Cond = Operand('Cond', intcc)
-f = Operand('f', iflags)
-a = Operand('a', b1)
-
-trueif = Instruction(
-        'trueif', r"""
-        Test integer CPU flags for a specific condition.
-
-        Check the CPU flags in ``f`` against the ``Cond`` condition code and
-        return true when the condition code is satisfied.
-        """,
-        ins=(Cond, f), outs=a)
-
-Cond = Operand('Cond', floatcc)
-f = Operand('f', fflags)
-
-trueff = Instruction(
-        'trueff', r"""
-        Test floating point CPU flags for a specific condition.
-
-        Check the CPU flags in ``f`` against the ``Cond`` condition code and
-        return true when the condition code is satisfied.
-        """,
-        ins=(Cond, f), outs=a)
-
-#
-# Conversions
-#
-
-x = Operand('x', Mem)
-a = Operand('a', MemTo, 'Bits of `x` reinterpreted')
-
-bitcast = Instruction(
-        'bitcast', r"""
-        Reinterpret the bits in `x` as a different type.
-
-        The input and output types must be storable to memory and of the same
-        size. A bitcast is equivalent to storing one type and loading the other
-        type from the same address.
-        """,
-        ins=x, outs=a)
-
-Bool = TypeVar(
-        'Bool',
-        'A scalar or vector boolean type',
-        bools=True, simd=True)
-BoolTo = TypeVar(
-        'BoolTo',
-        'A smaller boolean type with the same number of lanes',
-        bools=True, simd=True)
-
-x = Operand('x', Bool)
-a = Operand('a', BoolTo)
-
-breduce = Instruction(
-        'breduce', r"""
-        Convert `x` to a smaller boolean type in the platform-defined way.
-
-        The result type must have the same number of vector lanes as the input,
-        and each lane must not have more bits that the input lanes. If the
-        input and output types are the same, this is a no-op.
-        """, ins=x, outs=a, constraints=WiderOrEq(Bool, BoolTo))
-
-BoolTo = TypeVar(
-        'BoolTo',
-        'A larger boolean type with the same number of lanes',
-        bools=True, simd=True)
-
-x = Operand('x', Bool)
-a = Operand('a', BoolTo)
-
-bextend = Instruction(
-        'bextend', r"""
-        Convert `x` to a larger boolean type in the platform-defined way.
-
-        The result type must have the same number of vector lanes as the input,
-        and each lane must not have fewer bits that the input lanes. If the
-        input and output types are the same, this is a no-op.
-        """, ins=x, outs=a, constraints=WiderOrEq(BoolTo, Bool))
-
-IntTo = TypeVar(
-        'IntTo', 'An integer type with the same number of lanes',
-        ints=True, simd=True)
-
-x = Operand('x', Bool)
-a = Operand('a', IntTo)
-
-bint = Instruction(
-        'bint', r"""
-        Convert `x` to an integer.
-
-        True maps to 1 and false maps to 0. The result type must have the same
-        number of vector lanes as the input.
-        """, ins=x, outs=a)
-
-bmask = Instruction(
-        'bmask', r"""
-        Convert `x` to an integer mask.
-
-        True maps to all 1s and false maps to all 0s. The result type must have
-        the same number of vector lanes as the input.
-        """, ins=x, outs=a)
-
-Int = TypeVar('Int', 'A scalar or vector integer type', ints=True, simd=True)
-IntTo = TypeVar(
-        'IntTo', 'A smaller integer type with the same number of lanes',
-        ints=True, simd=True)
-
-x = Operand('x', Int)
-a = Operand('a', IntTo)
-
-ireduce = Instruction(
-        'ireduce', r"""
-        Convert `x` to a smaller integer type by dropping high bits.
-
-        Each lane in `x` is converted to a smaller integer type by discarding
-        the most significant bits. This is the same as reducing modulo
-        :math:`2^n`.
-
-        The result type must have the same number of vector lanes as the input,
-        and each lane must not have more bits that the input lanes. If the
-        input and output types are the same, this is a no-op.
-        """,
-        ins=x, outs=a, constraints=WiderOrEq(Int, IntTo))
-
-
-IntTo = TypeVar(
-        'IntTo', 'A larger integer type with the same number of lanes',
-        ints=True, simd=True)
-
-x = Operand('x', Int)
-a = Operand('a', IntTo)
-
-uextend = Instruction(
-        'uextend', r"""
-        Convert `x` to a larger integer type by zero-extending.
-
-        Each lane in `x` is converted to a larger integer type by adding
-        zeroes. The result has the same numerical value as `x` when both are
-        interpreted as unsigned integers.
-
-        The result type must have the same number of vector lanes as the input,
-        and each lane must not have fewer bits that the input lanes. If the
-        input and output types are the same, this is a no-op.
-        """,
-        ins=x, outs=a, constraints=WiderOrEq(IntTo, Int))
-
-sextend = Instruction(
-        'sextend', r"""
-        Convert `x` to a larger integer type by sign-extending.
-
-        Each lane in `x` is converted to a larger integer type by replicating
-        the sign bit. The result has the same numerical value as `x` when both
-        are interpreted as signed integers.
-
-        The result type must have the same number of vector lanes as the input,
-        and each lane must not have fewer bits that the input lanes. If the
-        input and output types are the same, this is a no-op.
-        """,
-        ins=x, outs=a, constraints=WiderOrEq(IntTo, Int))
-
-FloatTo = TypeVar(
-        'FloatTo', 'A scalar or vector floating point number',
-        floats=True, simd=True)
-
-x = Operand('x', Float)
-a = Operand('a', FloatTo)
-
-fpromote = Instruction(
-        'fpromote', r"""
-        Convert `x` to a larger floating point format.
-
-        Each lane in `x` is converted to the destination floating point format.
-        This is an exact operation.
-
-        Cranelift currently only supports two floating point formats
-        - :type:`f32` and :type:`f64`. This may change in the future.
-
-        The result type must have the same number of vector lanes as the input,
-        and the result lanes must not have fewer bits than the input lanes. If
-        the input and output types are the same, this is a no-op.
-        """,
-        ins=x, outs=a, constraints=WiderOrEq(FloatTo, Float))
-
-fdemote = Instruction(
-        'fdemote', r"""
-        Convert `x` to a smaller floating point format.
-
-        Each lane in `x` is converted to the destination floating point format
-        by rounding to nearest, ties to even.
-
-        Cranelift currently only supports two floating point formats
-        - :type:`f32` and :type:`f64`. This may change in the future.
-
-        The result type must have the same number of vector lanes as the input,
-        and the result lanes must not have more bits than the input lanes. If
-        the input and output types are the same, this is a no-op.
-        """,
-        ins=x, outs=a, constraints=WiderOrEq(Float, FloatTo))
-
-x = Operand('x', Float)
-a = Operand('a', IntTo)
-
-fcvt_to_uint = Instruction(
-        'fcvt_to_uint', r"""
-        Convert floating point to unsigned integer.
-
-        Each lane in `x` is converted to an unsigned integer by rounding
-        towards zero. If `x` is NaN or if the unsigned integral value cannot be
-        represented in the result type, this instruction traps.
-
-        The result type must have the same number of vector lanes as the input.
-        """,
-        ins=x, outs=a, can_trap=True)
-
-fcvt_to_uint_sat = Instruction(
-        'fcvt_to_uint_sat', r"""
-        Convert floating point to unsigned integer as fcvt_to_uint does, but
-        saturates the input instead of trapping. NaN and negative values are
-        converted to 0.
-        """,
-        ins=x, outs=a)
-
-fcvt_to_sint = Instruction(
-        'fcvt_to_sint', r"""
-        Convert floating point to signed integer.
-
-        Each lane in `x` is converted to a signed integer by rounding towards
-        zero. If `x` is NaN or if the signed integral value cannot be
-        represented in the result type, this instruction traps.
-
-        The result type must have the same number of vector lanes as the input.
-        """,
-        ins=x, outs=a, can_trap=True)
-
-fcvt_to_sint_sat = Instruction(
-        'fcvt_to_sint_sat', r"""
-        Convert floating point to signed integer as fcvt_to_sint does, but
-        saturates the input instead of trapping. NaN values are converted to 0.
-        """,
-        ins=x, outs=a)
-
-x = Operand('x', Int)
-a = Operand('a', FloatTo)
-
-fcvt_from_uint = Instruction(
-        'fcvt_from_uint', r"""
-        Convert unsigned integer to floating point.
-
-        Each lane in `x` is interpreted as an unsigned integer and converted to
-        floating point using round to nearest, ties to even.
-
-        The result type must have the same number of vector lanes as the input.
-        """,
-        ins=x, outs=a)
-
-fcvt_from_sint = Instruction(
-        'fcvt_from_sint', r"""
-        Convert signed integer to floating point.
-
-        Each lane in `x` is interpreted as a signed integer and converted to
-        floating point using round to nearest, ties to even.
-
-        The result type must have the same number of vector lanes as the input.
-        """,
-        ins=x, outs=a)
-
-#
-# Legalization helper instructions.
-#
-
-WideInt = TypeVar(
-        'WideInt', 'An integer type with lanes from `i16` upwards',
-        ints=(16, 64), simd=True)
-x = Operand('x', WideInt)
-lo = Operand(
-        'lo', WideInt.half_width(), 'The low bits of `x`')
-hi = Operand(
-        'hi', WideInt.half_width(), 'The high bits of `x`')
-
-isplit = Instruction(
-        'isplit', r"""
-        Split an integer into low and high parts.
-
-        Vectors of integers are split lane-wise, so the results have the same
-        number of lanes as the input, but the lanes are half the size.
-
-        Returns the low half of `x` and the high half of `x` as two independent
-        values.
-        """,
-        ins=x, outs=(lo, hi), is_ghost=True)
-
-
-NarrowInt = TypeVar(
-        'NarrowInt', 'An integer type with lanes type to `i32`',
-        ints=(8, 32), simd=True)
-lo = Operand('lo', NarrowInt)
-hi = Operand('hi', NarrowInt)
-a = Operand(
-        'a', NarrowInt.double_width(),
-        doc='The concatenation of `lo` and `hi`')
-
-iconcat = Instruction(
-        'iconcat', r"""
-        Concatenate low and high bits to form a larger integer type.
-
-        Vectors of integers are concatenated lane-wise such that the result has
-        the same number of lanes as the inputs, but the lanes are twice the
-        size.
-        """,
-        ins=(lo, hi), outs=a, is_ghost=True)
-
-GROUP.close()
diff --git a/cranelift/codegen/meta-python/base/legalize.py b/cranelift/codegen/meta-python/base/legalize.py
deleted file mode 100644
index 3e287a7b4e..0000000000
--- a/cranelift/codegen/meta-python/base/legalize.py
+++ /dev/null
@@ -1,713 +0,0 @@
-"""
-Patterns for legalizing the `base` instruction set.
-
-The base Cranelift instruction set is 'fat', and many instructions don't have
-legal representations in a given target ISA. This module defines legalization
-patterns that describe how base instructions can be transformed to other base
-instructions that are legal.
-"""
-from __future__ import absolute_import
-from .immediates import intcc, imm64, ieee32, ieee64
-from . import instructions as insts
-from . import types
-from .instructions import uextend, sextend, ireduce
-from .instructions import iadd, iadd_cout, iadd_cin, iadd_carry, iadd_imm
-from .instructions import isub, isub_bin, isub_bout, isub_borrow, irsub_imm
-from .instructions import imul, imul_imm
-from .instructions import sdiv, sdiv_imm, udiv, udiv_imm
-from .instructions import srem, srem_imm, urem, urem_imm
-from .instructions import band, bor, bxor, isplit, iconcat
-from .instructions import bnot, band_not, bor_not, bxor_not
-from .instructions import band_imm, bor_imm, bxor_imm
-from .instructions import icmp, icmp_imm, ifcmp, ifcmp_imm
-from .instructions import iconst, bint, select
-from .instructions import ishl, ishl_imm, sshr, sshr_imm, ushr, ushr_imm
-from .instructions import rotl, rotl_imm, rotr, rotr_imm
-from .instructions import f32const, f64const
-from .instructions import store, load
-from .instructions import br_table
-from .instructions import bitrev
-from cdsl.ast import Var
-from cdsl.xform import Rtl, XFormGroup
-
-try:
-    from typing import TYPE_CHECKING # noqa
-    if TYPE_CHECKING:
-        from cdsl.instructions import Instruction # noqa
-except ImportError:
-    TYPE_CHECKING = False
-
-
-narrow = XFormGroup('narrow', """
-        Legalize instructions by narrowing.
-
-        The transformations in the 'narrow' group work by expressing
-        instructions in terms of smaller types. Operations on vector types are
-        expressed in terms of vector types with fewer lanes, and integer
-        operations are expressed in terms of smaller integer types.
-        """)
-
-widen = XFormGroup('widen', """
-        Legalize instructions by widening.
-
-        The transformations in the 'widen' group work by expressing
-        instructions in terms of larger types.
-        """)
-
-expand = XFormGroup('expand', """
-        Legalize instructions by expansion.
-
-        Rewrite instructions in terms of other instructions, generally
-        operating on the same types as the original instructions.
-        """)
-
-expand_flags = XFormGroup('expand_flags', """
-        Instruction expansions for architectures with flags.
-
-        Expand some instructions using CPU flags, then fall back to the normal
-        expansions. Not all architectures support CPU flags, so these patterns
-        are kept separate.
-        """, chain=expand)
-
-
-# Custom expansions for memory objects.
-expand.custom_legalize(insts.global_value, 'expand_global_value')
-expand.custom_legalize(insts.heap_addr, 'expand_heap_addr')
-expand.custom_legalize(insts.table_addr, 'expand_table_addr')
-
-# Custom expansions for calls.
-expand.custom_legalize(insts.call, 'expand_call')
-
-# Custom expansions that need to change the CFG.
-# TODO: Add sufficient XForm syntax that we don't need to hand-code these.
-expand.custom_legalize(insts.trapz, 'expand_cond_trap')
-expand.custom_legalize(insts.trapnz, 'expand_cond_trap')
-expand.custom_legalize(insts.br_table, 'expand_br_table')
-expand.custom_legalize(insts.select, 'expand_select')
-
-# Custom expansions for floating point constants.
-# These expansions require bit-casting or creating constant pool entries.
-expand.custom_legalize(insts.f32const, 'expand_fconst')
-expand.custom_legalize(insts.f64const, 'expand_fconst')
-
-# Custom expansions for stack memory accesses.
-expand.custom_legalize(insts.stack_load, 'expand_stack_load')
-expand.custom_legalize(insts.stack_store, 'expand_stack_store')
-
-x = Var('x')
-y = Var('y')
-z = Var('z')
-a = Var('a')
-a1 = Var('a1')
-a2 = Var('a2')
-a3 = Var('a3')
-a4 = Var('a4')
-b = Var('b')
-b1 = Var('b1')
-b2 = Var('b2')
-b3 = Var('b3')
-b4 = Var('b4')
-b_in = Var('b_in')
-b_int = Var('b_int')
-c = Var('c')
-c1 = Var('c1')
-c2 = Var('c2')
-c3 = Var('c3')
-c4 = Var('c4')
-c_in = Var('c_in')
-c_int = Var('c_int')
-d = Var('d')
-d1 = Var('d1')
-d2 = Var('d2')
-d3 = Var('d3')
-d4 = Var('d4')
-e = Var('e')
-e1 = Var('e1')
-e2 = Var('e2')
-e3 = Var('e3')
-e4 = Var('e4')
-f = Var('f')
-f1 = Var('f1')
-f2 = Var('f2')
-xl = Var('xl')
-xh = Var('xh')
-yl = Var('yl')
-yh = Var('yh')
-al = Var('al')
-ah = Var('ah')
-cc = Var('cc')
-ptr = Var('ptr')
-flags = Var('flags')
-offset = Var('off')
-ss = Var('ss')
-
-narrow.legalize(
-        a << iadd(x, y),
-        Rtl(
-            (xl, xh) << isplit(x),
-            (yl, yh) << isplit(y),
-            (al, c) << iadd_cout(xl, yl),
-            ah << iadd_cin(xh, yh, c),
-            a << iconcat(al, ah)
-        ))
-
-narrow.legalize(
-        a << isub(x, y),
-        Rtl(
-            (xl, xh) << isplit(x),
-            (yl, yh) << isplit(y),
-            (al, b) << isub_bout(xl, yl),
-            ah << isub_bin(xh, yh, b),
-            a << iconcat(al, ah)
-        ))
-
-for bitop in [band, bor, bxor]:
-    narrow.legalize(
-            a << bitop(x, y),
-            Rtl(
-                (xl, xh) << isplit(x),
-                (yl, yh) << isplit(y),
-                al << bitop(xl, yl),
-                ah << bitop(xh, yh),
-                a << iconcat(al, ah)
-            ))
-
-narrow.legalize(
-        a << select(c, x, y),
-        Rtl(
-            (xl, xh) << isplit(x),
-            (yl, yh) << isplit(y),
-            al << select(c, xl, yl),
-            ah << select(c, xh, yh),
-            a << iconcat(al, ah)
-        ))
-
-
-def widen_one_arg(signed, op):
-    # type: (bool, Instruction) -> None
-    for int_ty in [types.i8, types.i16]:
-        if signed:
-            widen.legalize(
-                a << op.bind(int_ty)(b),
-                Rtl(
-                    x << sextend.i32(b),
-                    z << op.i32(x),
-                    a << ireduce.bind(int_ty)(z)
-                ))
-        else:
-            widen.legalize(
-                a << op.bind(int_ty)(b),
-                Rtl(
-                    x << uextend.i32(b),
-                    z << op.i32(x),
-                    a << ireduce.bind(int_ty)(z)
-                ))
-
-
-def widen_two_arg(signed, op):
-    # type: (bool, Instruction) -> None
-    for int_ty in [types.i8, types.i16]:
-        if signed:
-            widen.legalize(
-                a << op.bind(int_ty)(b, c),
-                Rtl(
-                    x << sextend.i32(b),
-                    y << sextend.i32(c),
-                    z << op.i32(x, y),
-                    a << ireduce.bind(int_ty)(z)
-                ))
-        else:
-            widen.legalize(
-                a << op.bind(int_ty)(b, c),
-                Rtl(
-                    x << uextend.i32(b),
-                    y << uextend.i32(c),
-                    z << op.i32(x, y),
-                    a << ireduce.bind(int_ty)(z)
-                ))
-
-
-def widen_imm(signed, op):
-    # type: (bool, Instruction) -> None
-    for int_ty in [types.i8, types.i16]:
-        if signed:
-            widen.legalize(
-                a << op.bind(int_ty)(b, c),
-                Rtl(
-                    x << sextend.i32(b),
-                    z << op.i32(x, c),
-                    a << ireduce.bind(int_ty)(z)
-                ))
-        else:
-            widen.legalize(
-                a << op.bind(int_ty)(b, c),
-                Rtl(
-                    x << uextend.i32(b),
-                    z << op.i32(x, c),
-                    a << ireduce.bind(int_ty)(z)
-                ))
-
-
-# int ops
-for binop in [iadd, isub, imul, udiv, urem]:
-    widen_two_arg(False, binop)
-
-for binop in [sdiv, srem]:
-    widen_two_arg(True, binop)
-
-for binop in [iadd_imm, imul_imm, udiv_imm, urem_imm]:
-    widen_imm(False, binop)
-
-for binop in [sdiv_imm, srem_imm]:
-    widen_imm(True, binop)
-
-widen_imm(False, irsub_imm)
-
-# bit ops
-widen_one_arg(False, bnot)
-
-for binop in [band, bor, bxor, band_not, bor_not, bxor_not]:
-    widen_two_arg(False, binop)
-
-for binop in [band_imm, bor_imm, bxor_imm]:
-    widen_imm(False, binop)
-
-widen_one_arg(False, insts.popcnt)
-
-for (int_ty, num) in [(types.i8, 24), (types.i16, 16)]:
-    widen.legalize(
-        a << insts.clz.bind(int_ty)(b),
-        Rtl(
-            c << uextend.i32(b),
-            d << insts.clz.i32(c),
-            e << iadd_imm(d, imm64(-num)),
-            a << ireduce.bind(int_ty)(e)
-        ))
-
-    widen.legalize(
-        a << insts.cls.bind(int_ty)(b),
-        Rtl(
-            c << sextend.i32(b),
-            d << insts.cls.i32(c),
-            e << iadd_imm(d, imm64(-num)),
-            a << ireduce.bind(int_ty)(e)
-        ))
-
-for (int_ty, num) in [(types.i8, 1 << 8), (types.i16, 1 << 16)]:
-    widen.legalize(
-        a << insts.ctz.bind(int_ty)(b),
-        Rtl(
-            c << uextend.i32(b),
-            # When `b` is zero, returns the size of x in bits.
-            d << bor_imm(c, imm64(num)),
-            e << insts.ctz.i32(d),
-            a << ireduce.bind(int_ty)(e)
-        ))
-
-# iconst
-for int_ty in [types.i8, types.i16]:
-    widen.legalize(
-        a << iconst.bind(int_ty)(b),
-        Rtl(
-            c << iconst.i32(b),
-            a << ireduce.bind(int_ty)(c)
-        ))
-
-widen.legalize(
-    a << uextend.i16.i8(b),
-    Rtl(
-        c << uextend.i32(b),
-        a << ireduce(c)
-    ))
-
-widen.legalize(
-    a << sextend.i16.i8(b),
-    Rtl(
-        c << sextend.i32(b),
-        a << ireduce(c)
-    ))
-
-
-widen.legalize(
-    store.i8(flags, a, ptr, offset),
-    Rtl(
-        b << uextend.i32(a),
-        insts.istore8(flags, b, ptr, offset)
-    ))
-
-widen.legalize(
-    store.i16(flags, a, ptr, offset),
-    Rtl(
-        b << uextend.i32(a),
-        insts.istore16(flags, b, ptr, offset)
-    ))
-
-widen.legalize(
-    a << load.i8(flags, ptr, offset),
-    Rtl(
-        b << insts.uload8.i32(flags, ptr, offset),
-        a << ireduce(b)
-    ))
-
-widen.legalize(
-    a << load.i16(flags, ptr, offset),
-    Rtl(
-        b << insts.uload16.i32(flags, ptr, offset),
-        a << ireduce(b)
-    ))
-
-for int_ty in [types.i8, types.i16]:
-    widen.legalize(
-        br_table.bind(int_ty)(x, y, z),
-        Rtl(
-            b << uextend.i32(x),
-            br_table(b, y, z),
-        )
-    )
-
-for int_ty in [types.i8, types.i16]:
-    widen.legalize(
-        a << insts.bint.bind(int_ty)(b),
-        Rtl(
-            x << insts.bint.i32(b),
-            a << ireduce.bind(int_ty)(x)
-        )
-    )
-
-for int_ty in [types.i8, types.i16]:
-    for op in [ishl, ishl_imm, ushr, ushr_imm]:
-        widen.legalize(
-            a << op.bind(int_ty)(b, c),
-            Rtl(
-                x << uextend.i32(b),
-                z << op.i32(x, c),
-                a << ireduce.bind(int_ty)(z)
-            ))
-
-    for op in [sshr, sshr_imm]:
-        widen.legalize(
-            a << op.bind(int_ty)(b, c),
-            Rtl(
-                x << sextend.i32(b),
-                z << op.i32(x, c),
-                a << ireduce.bind(int_ty)(z)
-            ))
-
-    for w_cc in [
-        intcc.eq, intcc.ne, intcc.ugt, intcc.ult, intcc.uge, intcc.ule
-    ]:
-        widen.legalize(
-            a << insts.icmp_imm.bind(int_ty)(w_cc, b, c),
-            Rtl(
-                x << uextend.i32(b),
-                a << insts.icmp_imm(w_cc, x, c)
-            ))
-        widen.legalize(
-            a << insts.icmp.bind(int_ty)(w_cc, b, c),
-            Rtl(
-                x << uextend.i32(b),
-                y << uextend.i32(c),
-                a << insts.icmp.i32(w_cc, x, y)
-            ))
-    for w_cc in [intcc.sgt, intcc.slt, intcc.sge, intcc.sle]:
-        widen.legalize(
-            a << insts.icmp_imm.bind(int_ty)(w_cc, b, c),
-            Rtl(
-                x << sextend.i32(b),
-                a << insts.icmp_imm(w_cc, x, c)
-            ))
-        widen.legalize(
-            a << insts.icmp.bind(int_ty)(w_cc, b, c),
-            Rtl(
-                x << sextend.i32(b),
-                y << sextend.i32(c),
-                a << insts.icmp(w_cc, x, y)
-            )
-        )
-
-# Expand integer operations with carry for RISC architectures that don't have
-# the flags.
-expand.legalize(
-        (a, c) << iadd_cout(x, y),
-        Rtl(
-            a << iadd(x, y),
-            c << icmp(intcc.ult, a, x)
-        ))
-
-expand.legalize(
-        (a, b) << isub_bout(x, y),
-        Rtl(
-            a << isub(x, y),
-            b << icmp(intcc.ugt, a, x)
-        ))
-
-expand.legalize(
-        a << iadd_cin(x, y, c),
-        Rtl(
-            a1 << iadd(x, y),
-            c_int << bint(c),
-            a << iadd(a1, c_int)
-        ))
-
-expand.legalize(
-        a << isub_bin(x, y, b),
-        Rtl(
-            a1 << isub(x, y),
-            b_int << bint(b),
-            a << isub(a1, b_int)
-        ))
-
-expand.legalize(
-        (a, c) << iadd_carry(x, y, c_in),
-        Rtl(
-            (a1, c1) << iadd_cout(x, y),
-            c_int << bint(c_in),
-            (a, c2) << iadd_cout(a1, c_int),
-            c << bor(c1, c2)
-        ))
-
-expand.legalize(
-        (a, b) << isub_borrow(x, y, b_in),
-        Rtl(
-            (a1, b1) << isub_bout(x, y),
-            b_int << bint(b_in),
-            (a, b2) << isub_bout(a1, b_int),
-            b << bor(b1, b2)
-        ))
-
-# Expansions for fcvt_from_{u,s}int for smaller integer types.
-# These use expand and not widen because the controlling type variable for
-# these instructions are f32/f64, which are legalized as part of the expand
-# group.
-for dest_ty in [types.f32, types.f64]:
-    for src_ty in [types.i8, types.i16]:
-        expand.legalize(
-            a << insts.fcvt_from_uint.bind(dest_ty).bind(src_ty)(b),
-            Rtl(
-                x << uextend.i32(b),
-                a << insts.fcvt_from_uint.bind(dest_ty).i32(x),
-            ))
-
-        expand.legalize(
-            a << insts.fcvt_from_sint.bind(dest_ty).bind(src_ty)(b),
-            Rtl(
-                x << sextend.i32(b),
-                a << insts.fcvt_from_sint.bind(dest_ty).i32(x),
-            ))
-
-# Expansions for immediate operands that are out of range.
-for inst_imm,      inst in [
-        (iadd_imm, iadd),
-        (imul_imm, imul),
-        (sdiv_imm, sdiv),
-        (udiv_imm, udiv),
-        (srem_imm, srem),
-        (urem_imm, urem),
-        (band_imm, band),
-        (bor_imm, bor),
-        (bxor_imm, bxor),
-        (ifcmp_imm, ifcmp)]:
-    expand.legalize(
-            a << inst_imm(x, y),
-            Rtl(
-                a1 << iconst(y),
-                a << inst(x, a1)
-            ))
-expand.legalize(
-    a << irsub_imm(y, x),
-    Rtl(
-        a1 << iconst(x),
-        a << isub(a1, y)
-    ))
-
-# Rotates and shifts.
-for inst_imm,      inst in [
-        (rotl_imm, rotl),
-        (rotr_imm, rotr),
-        (ishl_imm, ishl),
-        (sshr_imm, sshr),
-        (ushr_imm, ushr)]:
-    expand.legalize(
-            a << inst_imm(x, y),
-            Rtl(
-                a1 << iconst.i32(y),
-                a << inst(x, a1)
-            ))
-
-expand.legalize(
-        a << icmp_imm(cc, x, y),
-        Rtl(
-            a1 << iconst(y),
-            a << icmp(cc, x, a1)
-        ))
-
-# Expansions for *_not variants of bitwise ops.
-for inst_not,      inst in [
-        (band_not, band),
-        (bor_not,  bor),
-        (bxor_not, bxor)]:
-    expand.legalize(
-            a << inst_not(x, y),
-            Rtl(
-                a1 << bnot(y),
-                a << inst(x, a1)
-            ))
-
-# Expand bnot using xor.
-expand.legalize(
-        a << bnot(x),
-        Rtl(
-            y << iconst(imm64(-1)),
-            a << bxor(x, y)
-        ))
-
-# Expand bitrev
-# Adapted from Stack Overflow.
-# https://stackoverflow.com/questions/746171/most-efficient-algorithm-for-bit-reversal-from-msb-lsb-to-lsb-msb-in-c
-widen.legalize(
-        a << bitrev.i8(x),
-        Rtl(
-            a1 << band_imm(x, imm64(0xaa)),
-            a2 << ushr_imm(a1, imm64(1)),
-            a3 << band_imm(x, imm64(0x55)),
-            a4 << ishl_imm(a3, imm64(1)),
-            b << bor(a2, a4),
-            b1 << band_imm(b, imm64(0xcc)),
-            b2 << ushr_imm(b1, imm64(2)),
-            b3 << band_imm(b, imm64(0x33)),
-            b4 << ishl_imm(b3, imm64(2)),
-            c << bor(b2, b4),
-            c1 << band_imm(c, imm64(0xf0)),
-            c2 << ushr_imm(c1, imm64(4)),
-            c3 << band_imm(c, imm64(0x0f)),
-            c4 << ishl_imm(c3, imm64(4)),
-            a << bor(c2, c4),
-        ))
-
-widen.legalize(
-        a << bitrev.i16(x),
-        Rtl(
-            a1 << band_imm(x, imm64(0xaaaa)),
-            a2 << ushr_imm(a1, imm64(1)),
-            a3 << band_imm(x, imm64(0x5555)),
-            a4 << ishl_imm(a3, imm64(1)),
-            b << bor(a2, a4),
-            b1 << band_imm(b, imm64(0xcccc)),
-            b2 << ushr_imm(b1, imm64(2)),
-            b3 << band_imm(b, imm64(0x3333)),
-            b4 << ishl_imm(b3, imm64(2)),
-            c << bor(b2, b4),
-            c1 << band_imm(c, imm64(0xf0f0)),
-            c2 << ushr_imm(c1, imm64(4)),
-            c3 << band_imm(c, imm64(0x0f0f)),
-            c4 << ishl_imm(c3, imm64(4)),
-            d << bor(c2, c4),
-            d1 << band_imm(d, imm64(0xff00)),
-            d2 << ushr_imm(d1, imm64(8)),
-            d3 << band_imm(d, imm64(0x00ff)),
-            d4 << ishl_imm(d3, imm64(8)),
-            a << bor(d2, d4),
-        ))
-
-expand.legalize(
-        a << bitrev.i32(x),
-        Rtl(
-            a1 << band_imm(x, imm64(0xaaaaaaaa)),
-            a2 << ushr_imm(a1, imm64(1)),
-            a3 << band_imm(x, imm64(0x55555555)),
-            a4 << ishl_imm(a3, imm64(1)),
-            b << bor(a2, a4),
-            b1 << band_imm(b, imm64(0xcccccccc)),
-            b2 << ushr_imm(b1, imm64(2)),
-            b3 << band_imm(b, imm64(0x33333333)),
-            b4 << ishl_imm(b3, imm64(2)),
-            c << bor(b2, b4),
-            c1 << band_imm(c, imm64(0xf0f0f0f0)),
-            c2 << ushr_imm(c1, imm64(4)),
-            c3 << band_imm(c, imm64(0x0f0f0f0f)),
-            c4 << ishl_imm(c3, imm64(4)),
-            d << bor(c2, c4),
-            d1 << band_imm(d, imm64(0xff00ff00)),
-            d2 << ushr_imm(d1, imm64(8)),
-            d3 << band_imm(d, imm64(0x00ff00ff)),
-            d4 << ishl_imm(d3, imm64(8)),
-            e << bor(d2, d4),
-            e1 << ushr_imm(e, imm64(16)),
-            e2 << ishl_imm(e, imm64(16)),
-            a << bor(e1, e2),
-        ))
-
-expand.legalize(
-        a << bitrev.i64(x),
-        Rtl(
-            a1 << band_imm(x, imm64(0xaaaaaaaaaaaaaaaa)),
-            a2 << ushr_imm(a1, imm64(1)),
-            a3 << band_imm(x, imm64(0x5555555555555555)),
-            a4 << ishl_imm(a3, imm64(1)),
-            b << bor(a2, a4),
-            b1 << band_imm(b, imm64(0xcccccccccccccccc)),
-            b2 << ushr_imm(b1, imm64(2)),
-            b3 << band_imm(b, imm64(0x3333333333333333)),
-            b4 << ishl_imm(b3, imm64(2)),
-            c << bor(b2, b4),
-            c1 << band_imm(c, imm64(0xf0f0f0f0f0f0f0f0)),
-            c2 << ushr_imm(c1, imm64(4)),
-            c3 << band_imm(c, imm64(0x0f0f0f0f0f0f0f0f)),
-            c4 << ishl_imm(c3, imm64(4)),
-            d << bor(c2, c4),
-            d1 << band_imm(d, imm64(0xff00ff00ff00ff00)),
-            d2 << ushr_imm(d1, imm64(8)),
-            d3 << band_imm(d, imm64(0x00ff00ff00ff00ff)),
-            d4 << ishl_imm(d3, imm64(8)),
-            e << bor(d2, d4),
-            e1 << band_imm(e, imm64(0xffff0000ffff0000)),
-            e2 << ushr_imm(e1, imm64(16)),
-            e3 << band_imm(e, imm64(0x0000ffff0000ffff)),
-            e4 << ishl_imm(e3, imm64(16)),
-            f << bor(e2, e4),
-            f1 << ushr_imm(f, imm64(32)),
-            f2 << ishl_imm(f, imm64(32)),
-            a << bor(f1, f2),
-        ))
-
-# Floating-point sign manipulations.
-for ty,             minus_zero in [
-        (types.f32, f32const(ieee32.bits(0x80000000))),
-        (types.f64, f64const(ieee64.bits(0x8000000000000000)))]:
-    expand.legalize(
-            a << insts.fabs.bind(ty)(x),
-            Rtl(
-                b << minus_zero,
-                a << band_not(x, b),
-            ))
-    expand.legalize(
-            a << insts.fneg.bind(ty)(x),
-            Rtl(
-                b << minus_zero,
-                a << bxor(x, b),
-            ))
-    expand.legalize(
-            a << insts.fcopysign.bind(ty)(x, y),
-            Rtl(
-                b << minus_zero,
-                a1 << band_not(x, b),
-                a2 << band(y, b),
-                a << bor(a1, a2)
-            ))
-
-expand.custom_legalize(insts.br_icmp, 'expand_br_icmp')
-
-# Expansions using CPU flags.
-
-expand_flags.legalize(
-    insts.trapnz(x, c),
-    Rtl(
-        a << insts.ifcmp_imm(x, imm64(0)),
-        insts.trapif(intcc.ne, a, c)
-    ))
-expand_flags.legalize(
-    insts.trapz(x, c),
-    Rtl(
-        a << insts.ifcmp_imm(x, imm64(0)),
-        insts.trapif(intcc.eq, a, c)
-    ))
diff --git a/cranelift/codegen/meta-python/base/predicates.py b/cranelift/codegen/meta-python/base/predicates.py
deleted file mode 100644
index 2a521f7ccb..0000000000
--- a/cranelift/codegen/meta-python/base/predicates.py
+++ /dev/null
@@ -1,42 +0,0 @@
-"""
-Cranelift predicates that consider `Function` fields.
-"""
-from cdsl.predicates import FieldPredicate
-from .formats import UnaryGlobalValue, InstructionFormat
-
-try:
-    from typing import TYPE_CHECKING
-    if TYPE_CHECKING:
-        from cdsl.formats import InstructionFormat, FormatField  # noqa
-except ImportError:
-    pass
-
-
-class IsColocatedFunc(FieldPredicate):
-    """
-    An instruction predicate that checks the referenced function is colocated.
-    """
-
-    def __init__(self, field):
-        # type: (FormatField) -> None
-        super(IsColocatedFunc, self).__init__(
-            field, 'is_colocated_func', ('func',))
-
-
-class IsColocatedData(FieldPredicate):
-    """
-    An instruction predicate that checks the referenced data object is
-    colocated.
-    """
-
-    def __init__(self):
-        # type: () -> None
-        super(IsColocatedData, self).__init__(
-            UnaryGlobalValue.global_value, 'is_colocated_data', ('func',))
-
-
-class LengthEquals(FieldPredicate):
-    def __init__(self, iform, num):
-        # type: (InstructionFormat, int) -> None
-        super(LengthEquals, self).__init__(
-            iform.args(), 'has_length_of', (num, 'func'))
diff --git a/cranelift/codegen/meta-python/base/semantics.py b/cranelift/codegen/meta-python/base/semantics.py
deleted file mode 100644
index ec1852133e..0000000000
--- a/cranelift/codegen/meta-python/base/semantics.py
+++ /dev/null
@@ -1,218 +0,0 @@
-from __future__ import absolute_import
-from semantics.primitives import prim_to_bv, prim_from_bv, bvsplit, bvconcat,\
-    bvadd, bvzeroext, bvsignext
-from semantics.primitives import bveq, bvne, bvsge, bvsgt, bvsle, bvslt,\
-        bvuge, bvugt, bvule, bvult
-from semantics.macros import bool2bv
-from .instructions import vsplit, vconcat, iadd, iadd_cout, icmp, bextend, \
-    isplit, iconcat, iadd_cin, iadd_carry
-from .immediates import intcc
-from cdsl.xform import Rtl, XForm
-from cdsl.ast import Var
-from cdsl.typevar import TypeSet
-from cdsl.ti import InTypeset
-
-try:
-    from typing import TYPE_CHECKING # noqa
-    if TYPE_CHECKING:
-        from cdsl.ast import Enumerator # noqa
-        from cdsl.instructions import Instruction # noqa
-except ImportError:
-    TYPE_CHECKING = False
-
-x = Var('x')
-y = Var('y')
-a = Var('a')
-b = Var('b')
-c_out = Var('c_out')
-c_in = Var('c_in')
-CC = Var('CC')
-bc_out = Var('bc_out')
-bvc_out = Var('bvc_out')
-bvc_in = Var('bvc_in')
-xhi = Var('xhi')
-yhi = Var('yhi')
-ahi = Var('ahi')
-bhi = Var('bhi')
-xlo = Var('xlo')
-ylo = Var('ylo')
-alo = Var('alo')
-blo = Var('blo')
-lo = Var('lo')
-hi = Var('hi')
-bvx = Var('bvx')
-bvy = Var('bvy')
-bva = Var('bva')
-bvt = Var('bvt')
-bvs = Var('bvs')
-bva_wide = Var('bva_wide')
-bvlo = Var('bvlo')
-bvhi = Var('bvhi')
-
-ScalarTS = TypeSet(lanes=(1, 1), ints=True, floats=True, bools=True)
-
-vsplit.set_semantics(
-    (lo, hi) << vsplit(x),
-    Rtl(
-        bvx << prim_to_bv(x),
-        (bvlo, bvhi) << bvsplit(bvx),
-        lo << prim_from_bv(bvlo),
-        hi << prim_from_bv(bvhi)
-    ))
-
-vconcat.set_semantics(
-    x << vconcat(lo, hi),
-    Rtl(
-        bvlo << prim_to_bv(lo),
-        bvhi << prim_to_bv(hi),
-        bvx << bvconcat(bvlo, bvhi),
-        x << prim_from_bv(bvx)
-    ))
-
-iadd.set_semantics(
-    a << iadd(x, y),
-    (Rtl(
-        bvx << prim_to_bv(x),
-        bvy << prim_to_bv(y),
-        bva << bvadd(bvx, bvy),
-        a << prim_from_bv(bva)
-    ), [InTypeset(x.get_typevar(), ScalarTS)]),
-    Rtl(
-        (xlo, xhi) << vsplit(x),
-        (ylo, yhi) << vsplit(y),
-        alo << iadd(xlo, ylo),
-        ahi << iadd(xhi, yhi),
-        a << vconcat(alo, ahi)
-    ))
-
-#
-# Integer arithmetic with carry and/or borrow.
-#
-iadd_cin.set_semantics(
-    a << iadd_cin(x, y, c_in),
-    Rtl(
-        bvx << prim_to_bv(x),
-        bvy << prim_to_bv(y),
-        bvc_in << prim_to_bv(c_in),
-        bvs << bvzeroext(bvc_in),
-        bvt << bvadd(bvx, bvy),
-        bva << bvadd(bvt, bvs),
-        a << prim_from_bv(bva)
-    ))
-
-iadd_cout.set_semantics(
-    (a, c_out) << iadd_cout(x, y),
-    Rtl(
-        bvx << prim_to_bv(x),
-        bvy << prim_to_bv(y),
-        bva << bvadd(bvx, bvy),
-        bc_out << bvult(bva, bvx),
-        bvc_out << bool2bv(bc_out),
-        a << prim_from_bv(bva),
-        c_out << prim_from_bv(bvc_out)
-    ))
-
-iadd_carry.set_semantics(
-    (a, c_out) << iadd_carry(x, y, c_in),
-    Rtl(
-        bvx << prim_to_bv(x),
-        bvy << prim_to_bv(y),
-        bvc_in << prim_to_bv(c_in),
-        bvs << bvzeroext(bvc_in),
-        bvt << bvadd(bvx, bvy),
-        bva << bvadd(bvt, bvs),
-        bc_out << bvult(bva, bvx),
-        bvc_out << bool2bv(bc_out),
-        a << prim_from_bv(bva),
-        c_out << prim_from_bv(bvc_out)
-    ))
-
-bextend.set_semantics(
-    a << bextend(x),
-    (Rtl(
-        bvx << prim_to_bv(x),
-        bvy << bvsignext(bvx),
-        a << prim_from_bv(bvy)
-    ), [InTypeset(x.get_typevar(), ScalarTS)]),
-    Rtl(
-        (xlo, xhi) << vsplit(x),
-        alo << bextend(xlo),
-        ahi << bextend(xhi),
-        a << vconcat(alo, ahi)
-    ))
-
-
-def create_comp_xform(cc, bvcmp_func):
-    # type: (Enumerator, Instruction) -> XForm
-    ba = Var('ba')
-    return XForm(
-               Rtl(
-                   a << icmp(cc, x, y)
-               ),
-               Rtl(
-                   bvx << prim_to_bv(x),
-                   bvy << prim_to_bv(y),
-                   ba << bvcmp_func(bvx, bvy),
-                   bva << bool2bv(ba),
-                   bva_wide << bvzeroext(bva),
-                   a << prim_from_bv(bva_wide),
-               ),
-               constraints=InTypeset(x.get_typevar(), ScalarTS))
-
-
-icmp.set_semantics(
-    a << icmp(CC, x, y),
-    Rtl(
-        (xlo, xhi) << vsplit(x),
-        (ylo, yhi) << vsplit(y),
-        alo << icmp(CC, xlo, ylo),
-        ahi << icmp(CC, xhi, yhi),
-        b << vconcat(alo, ahi),
-        a << bextend(b)
-    ),
-    create_comp_xform(intcc.eq, bveq),
-    create_comp_xform(intcc.ne, bvne),
-    create_comp_xform(intcc.sge, bvsge),
-    create_comp_xform(intcc.sgt, bvsgt),
-    create_comp_xform(intcc.sle, bvsle),
-    create_comp_xform(intcc.slt, bvslt),
-    create_comp_xform(intcc.uge, bvuge),
-    create_comp_xform(intcc.ugt, bvugt),
-    create_comp_xform(intcc.ule, bvule),
-    create_comp_xform(intcc.ult, bvult))
-
-#
-# Legalization helper instructions.
-#
-
-isplit.set_semantics(
-    (xlo, xhi) << isplit(x),
-    (Rtl(
-        bvx << prim_to_bv(x),
-        (bvlo, bvhi) << bvsplit(bvx),
-        xlo << prim_from_bv(bvlo),
-        xhi << prim_from_bv(bvhi)
-    ), [InTypeset(x.get_typevar(), ScalarTS)]),
-    Rtl(
-        (a, b) << vsplit(x),
-        (alo, ahi) << isplit(a),
-        (blo, bhi) << isplit(b),
-        xlo << vconcat(alo, blo),
-        xhi << vconcat(bhi, bhi)
-    ))
-
-iconcat.set_semantics(
-    x << iconcat(xlo, xhi),
-    (Rtl(
-        bvlo << prim_to_bv(xlo),
-        bvhi << prim_to_bv(xhi),
-        bvx << bvconcat(bvlo, bvhi),
-        x << prim_from_bv(bvx)
-    ), [InTypeset(x.get_typevar(), ScalarTS)]),
-    Rtl(
-        (alo, ahi) << vsplit(xlo),
-        (blo, bhi) << vsplit(xhi),
-        a << iconcat(alo, blo),
-        b << iconcat(ahi, bhi),
-        x << vconcat(a, b),
-    ))
diff --git a/cranelift/codegen/meta-python/base/settings.py b/cranelift/codegen/meta-python/base/settings.py
deleted file mode 100644
index 534ef28830..0000000000
--- a/cranelift/codegen/meta-python/base/settings.py
+++ /dev/null
@@ -1,144 +0,0 @@
-"""
-Cranelift shared settings.
-
-This module defines settings relevant for all code generators.
-"""
-from __future__ import absolute_import
-from cdsl.settings import SettingGroup, BoolSetting, EnumSetting, NumSetting
-
-group = SettingGroup('shared')
-
-opt_level = EnumSetting(
-        """
-        Optimization level:
-
-        - default: Very profitable optimizations enabled, none slow.
-        - best: Enable all optimizations
-        - fastest: Optimize for compile time by disabling most optimizations.
-        """,
-        'default', 'best', 'fastest')
-
-enable_verifier = BoolSetting(
-        """
-        Run the Cranelift IR verifier at strategic times during compilation.
-
-        This makes compilation slower but catches many bugs. The verifier is
-        disabled by default, except when reading Cranelift IR from a text file.
-        """,
-        default=True)
-
-# Note that Cranelift doesn't currently need an is_pie flag, because PIE is
-# just PIC where symbols can't be pre-empted, which can be expressed with the
-# `colocated` flag on external functions and global values.
-is_pic = BoolSetting("Enable Position-Independent Code generation")
-
-colocated_libcalls = BoolSetting(
-        """
-        Use colocated libcalls.
-
-        Generate code that assumes that libcalls can be declared "colocated",
-        meaning they will be defined along with the current function, such that
-        they can use more efficient addressing.
-        """)
-
-avoid_div_traps = BoolSetting(
-        """
-        Generate explicit checks around native division instructions to avoid
-        their trapping.
-
-        This is primarily used by SpiderMonkey which doesn't install a signal
-        handler for SIGFPE, but expects a SIGILL trap for division by zero.
-
-        On ISAs like ARM where the native division instructions don't trap,
-        this setting has no effect - explicit checks are always inserted.
-        """)
-
-enable_float = BoolSetting(
-        """
-        Enable the use of floating-point instructions
-
-        Disabling use of floating-point instructions is not yet implemented.
-        """,
-        default=True)
-
-enable_nan_canonicalization = BoolSetting(
-        """
-        Enable NaN canonicalization
-
-        This replaces NaNs with a single canonical value, for users requiring
-        entirely deterministic WebAssembly computation. This is not required
-        by the WebAssembly spec, so it is not enabled by default.
-        """,
-        default=False)
-
-enable_simd = BoolSetting(
-        """Enable the use of SIMD instructions.""",
-        default=True)
-
-enable_atomics = BoolSetting(
-        """Enable the use of atomic instructions""",
-        default=True)
-
-#
-# Settings specific to the `baldrdash` calling convention.
-#
-baldrdash_prologue_words = NumSetting(
-        """
-        Number of pointer-sized words pushed by the baldrdash prologue.
-
-        Functions with the `baldrdash` calling convention don't generate their
-        own prologue and epilogue. They depend on externally generated code
-        that pushes a fixed number of words in the prologue and restores them
-        in the epilogue.
-
-        This setting configures the number of pointer-sized words pushed on the
-        stack when the Cranelift-generated code is entered. This includes the
-        pushed return address on x86.
-        """)
-
-#
-# BaldrMonkey requires that not-yet-relocated function addresses be encoded
-# as all-ones bitpatterns.
-#
-allones_funcaddrs = BoolSetting(
-        """
-        Emit not-yet-relocated function addresses as all-ones bit patterns.
-        """)
-
-#
-# Stack probing options.
-#
-probestack_enabled = BoolSetting(
-        """
-        Enable the use of stack probes, for calling conventions which support
-        this functionality.
-        """,
-        default=True)
-
-probestack_func_adjusts_sp = BoolSetting(
-        """
-        Set this to true of the stack probe function modifies the stack pointer
-        itself.
-        """)
-
-probestack_size_log2 = NumSetting(
-        """
-        The log2 of the size of the stack guard region.
-
-        Stack frames larger than this size will have stack overflow checked
-        by calling the probestack function.
-
-        The default is 12, which translates to a size of 4096.
-        """,
-        default=12)
-
-#
-# Jump table options.
-#
-jump_tables_enabled = BoolSetting(
-        """
-        Enable the use of jump tables in generated machine code.
-        """,
-        default=True)
-
-group.close(globals())
diff --git a/cranelift/codegen/meta-python/base/types.py b/cranelift/codegen/meta-python/base/types.py
deleted file mode 100644
index 9141f4fbce..0000000000
--- a/cranelift/codegen/meta-python/base/types.py
+++ /dev/null
@@ -1,49 +0,0 @@
-"""
-The base.types module predefines all the Cranelift scalar types.
-"""
-from __future__ import absolute_import
-from cdsl.types import IntType, FloatType, BoolType, FlagsType
-
-#: Abstract boolean (can't be stored in memory, use bint to convert to 0 or 1).
-b1 = BoolType(1)    #: 1-bit bool.
-
-#: Booleans used as SIMD elements (can be stored in memory, true is all-ones).
-b8 = BoolType(8)    #: 8-bit bool.
-b16 = BoolType(16)  #: 16-bit bool.
-b32 = BoolType(32)  #: 32-bit bool.
-b64 = BoolType(64)  #: 64-bit bool.
-
-# Integers.
-i8 = IntType(8)     #: 8-bit int.
-i16 = IntType(16)   #: 16-bit int.
-i32 = IntType(32)   #: 32-bit int.
-i64 = IntType(64)   #: 64-bit int.
-
-#: IEEE single precision.
-f32 = FloatType(
-        32, """
-        A 32-bit floating point type represented in the IEEE 754-2008
-        *binary32* interchange format. This corresponds to the :c:type:`float`
-        type in most C implementations.
-        """)
-
-#: IEEE double precision.
-f64 = FloatType(
-        64, """
-        A 64-bit floating point type represented in the IEEE 754-2008
-        *binary64* interchange format. This corresponds to the :c:type:`double`
-        type in most C implementations.
-        """)
-#: CPU flags from an integer comparison.
-iflags = FlagsType(
-        'iflags', """
-        CPU flags representing the result of an integer comparison. These flags
-        can be tested with an :type:`intcc` condition code.
-        """)
-
-#: CPU flags from a floating point comparison.
-fflags = FlagsType(
-        'fflags', """
-        CPU flags representing the result of a floating point comparison. These
-        flags can be tested with a :type:`floatcc` condition code.
-        """)
diff --git a/cranelift/codegen/meta-python/build.py b/cranelift/codegen/meta-python/build.py
deleted file mode 100644
index 90a0cf254d..0000000000
--- a/cranelift/codegen/meta-python/build.py
+++ /dev/null
@@ -1,51 +0,0 @@
-# Second-level build script.
-#
-# This script is run from cranelift-codegen/build.rs to generate Rust files.
-
-from __future__ import absolute_import
-import argparse
-import isa
-import gen_build_deps
-import gen_encoding
-import gen_binemit
-
-try:
-    from typing import List, Set  # noqa
-    from cdsl.isa import TargetISA  # noqa
-    from cdsl.instructions import InstructionGroup  # noqa
-except ImportError:
-    pass
-
-
-def number_all_instructions(isas):
-    # type: (List[TargetISA]) -> None
-    seen = set()  # type: Set[InstructionGroup]
-    num_inst = 1
-    for target_isa in isas:
-        for g in target_isa.instruction_groups:
-            if g not in seen:
-                for i in g.instructions:
-                    i.number = num_inst
-                    num_inst += 1
-                seen.add(g)
-
-
-def main():
-    # type: () -> None
-    parser = argparse.ArgumentParser(
-            description='Generate sources for Cranelift.')
-    parser.add_argument('--out-dir', help='set output directory')
-
-    args = parser.parse_args()
-    out_dir = args.out_dir
-
-    isas = isa.all_isas()
-    number_all_instructions(isas)
-
-    gen_encoding.generate(isas, out_dir)
-    gen_binemit.generate(isas, out_dir)
-    gen_build_deps.generate()
-
-
-if __name__ == "__main__":
-    main()
diff --git a/cranelift/codegen/meta-python/cdsl/__init__.py b/cranelift/codegen/meta-python/cdsl/__init__.py
deleted file mode 100644
index a0b5d4c13b..0000000000
--- a/cranelift/codegen/meta-python/cdsl/__init__.py
+++ /dev/null
@@ -1,59 +0,0 @@
-"""
-Cranelift DSL classes.
-
-This module defines the classes that are used to define Cranelift instructions
-and other entities.
-"""
-from __future__ import absolute_import
-import re
-
-
-camel_re = re.compile('(^|_)([a-z])')
-
-
-def camel_case(s):
-    # type: (str) -> str
-    """Convert the string s to CamelCase:
-        >>> camel_case('x')
-        'X'
-        >>> camel_case('camel_case')
-        'CamelCase'
-    """
-    return camel_re.sub(lambda m: m.group(2).upper(), s)
-
-
-def is_power_of_two(x):
-    # type: (int) -> bool
-    """Check if `x` is a power of two:
-        >>> is_power_of_two(0)
-        False
-        >>> is_power_of_two(1)
-        True
-        >>> is_power_of_two(2)
-        True
-        >>> is_power_of_two(3)
-        False
-    """
-    return x > 0 and x & (x-1) == 0
-
-
-def next_power_of_two(x):
-    # type: (int) -> int
-    """
-    Compute the next power of two that is greater than `x`:
-        >>> next_power_of_two(0)
-        1
-        >>> next_power_of_two(1)
-        2
-        >>> next_power_of_two(2)
-        4
-        >>> next_power_of_two(3)
-        4
-        >>> next_power_of_two(4)
-        8
-    """
-    s = 1
-    while x & (x + 1) != 0:
-        x |= x >> s
-        s *= 2
-    return x + 1
diff --git a/cranelift/codegen/meta-python/cdsl/ast.py b/cranelift/codegen/meta-python/cdsl/ast.py
deleted file mode 100644
index 4a72fc486a..0000000000
--- a/cranelift/codegen/meta-python/cdsl/ast.py
+++ /dev/null
@@ -1,581 +0,0 @@
-"""
-Abstract syntax trees.
-
-This module defines classes that can be used to create abstract syntax trees
-for pattern matching an rewriting of cranelift instructions.
-"""
-from __future__ import absolute_import
-from . import instructions
-from .typevar import TypeVar
-from .predicates import IsEqual, And, TypePredicate, CtrlTypePredicate
-
-try:
-    from typing import Union, Tuple, Sequence, TYPE_CHECKING, Dict, List  # noqa
-    from typing import Optional, Set, Any # noqa
-    if TYPE_CHECKING:
-        from .operands import ImmediateKind  # noqa
-        from .predicates import PredNode  # noqa
-        VarAtomMap = Dict["Var", "Atom"]
-except ImportError:
-    pass
-
-
-def replace_var(arg, m):
-    # type: (Expr, VarAtomMap) -> Expr
-    """
-    Given a var v return either m[v] or a new variable v' (and remember
-    m[v]=v'). Otherwise return the argument unchanged
-    """
-    if isinstance(arg, Var):
-        new_arg = m.get(arg, Var(arg.name))  # type: Atom
-        m[arg] = new_arg
-        return new_arg
-    return arg
-
-
-class Def(object):
-    """
-    An AST definition associates a set of variables with the values produced by
-    an expression.
-
-    Example:
-
-    >>> from base.instructions import iadd_cout, iconst
-    >>> x = Var('x')
-    >>> y = Var('y')
-    >>> x << iconst(4)
-    (Var(x),) << Apply(iconst, (4,))
-    >>> (x, y) << iadd_cout(4, 5)
-    (Var(x), Var(y)) << Apply(iadd_cout, (4, 5))
-
-    The `<<` operator is used to create variable definitions.
-
-    :param defs: Single variable or tuple of variables to be defined.
-    :param expr: Expression generating the values.
-    """
-
-    def __init__(self, defs, expr):
-        # type: (Union[Var, Tuple[Var, ...]], Apply) -> None
-        if not isinstance(defs, tuple):
-            self.defs = (defs,)  # type: Tuple[Var, ...]
-        else:
-            self.defs = defs
-        assert isinstance(expr, Apply)
-        self.expr = expr
-
-    def __repr__(self):
-        # type: () -> str
-        return "{} << {!r}".format(self.defs, self.expr)
-
-    def __str__(self):
-        # type: () -> str
-        if len(self.defs) == 1:
-            return "{!s} << {!s}".format(self.defs[0], self.expr)
-        else:
-            return "({}) << {!s}".format(
-                    ', '.join(map(str, self.defs)), self.expr)
-
-    def copy(self, m):
-        # type: (VarAtomMap) -> Def
-        """
-        Return a copy of this Def with vars replaced with fresh variables,
-        in accordance with the map m. Update m as necessary.
-        """
-        new_expr = self.expr.copy(m)
-        new_defs = []  # type: List[Var]
-        for v in self.defs:
-            new_v = replace_var(v, m)
-            assert(isinstance(new_v, Var))
-            new_defs.append(new_v)
-
-        return Def(tuple(new_defs), new_expr)
-
-    def definitions(self):
-        # type: () -> Set[Var]
-        """ Return the set of all Vars that are defined by self"""
-        return set(self.defs)
-
-    def uses(self):
-        # type: () -> Set[Var]
-        """ Return the set of all Vars that are used(read) by self"""
-        return set(self.expr.vars())
-
-    def vars(self):
-        # type: () -> Set[Var]
-        """Return the set of all Vars in self that correspond to SSA values"""
-        return self.definitions().union(self.uses())
-
-    def substitution(self, other, s):
-        # type: (Def, VarAtomMap) -> Optional[VarAtomMap]
-        """
-        If the Defs self and other agree structurally, return a variable
-        substitution to transform self to other. Otherwise return None. Two
-        Defs agree structurally if there exists a Var substitution, that can
-        transform one into the other. See Apply.substitution() for more
-        details.
-        """
-        s = self.expr.substitution(other.expr, s)
-
-        if (s is None):
-            return s
-
-        assert len(self.defs) == len(other.defs)
-        for (self_d, other_d) in zip(self.defs, other.defs):
-            assert self_d not in s  # Guaranteed by SSA form
-            s[self_d] = other_d
-
-        return s
-
-
-class Expr(object):
-    """
-    An AST expression.
-    """
-
-
-class Atom(Expr):
-    """
-    An Atom in the DSL is either a literal or a Var
-    """
-
-
-class Var(Atom):
-    """
-    A free variable.
-
-    When variables are used in `XForms` with source and destination patterns,
-    they are classified as follows:
-
-    Input values
-        Uses in the source pattern with no preceding def. These may appear as
-        inputs in the destination pattern too, but no new inputs can be
-        introduced.
-    Output values
-        Variables that are defined in both the source and destination pattern.
-        These values may have uses outside the source pattern, and the
-        destination pattern must compute the same value.
-    Intermediate values
-        Values that are defined in the source pattern, but not in the
-        destination pattern. These may have uses outside the source pattern, so
-        the defining instruction can't be deleted immediately.
-    Temporary values
-        Values that are defined only in the destination pattern.
-    """
-
-    def __init__(self, name):
-        # type: (str) -> None
-        self.name = name
-        # The `Def` defining this variable in a source pattern.
-        self.src_def = None  # type: Def
-        # The `Def` defining this variable in a destination pattern.
-        self.dst_def = None  # type: Def
-        # TypeVar representing the type of this variable.
-        self.typevar = None  # type: TypeVar
-        # The original 'typeof(x)' type variable that was created for this Var.
-        # This one doesn't change. `self.typevar` above may be changed to
-        # another typevar by type inference.
-        self.original_typevar = None  # type: TypeVar
-
-    def __str__(self):
-        # type: () -> str
-        return self.name
-
-    def __repr__(self):
-        # type: () -> str
-        s = self.name
-        if self.src_def:
-            s += ", src"
-        if self.dst_def:
-            s += ", dst"
-        return "Var({})".format(s)
-
-    # Context bits for `set_def` indicating which pattern has defines of this
-    # var.
-    SRCCTX = 1
-    DSTCTX = 2
-
-    def set_def(self, context, d):
-        # type: (int, Def) -> None
-        """
-        Set the `Def` that defines this variable in the given context.
-
-        The `context` must be one of `SRCCTX` or `DSTCTX`
-        """
-        if context == self.SRCCTX:
-            self.src_def = d
-        else:
-            self.dst_def = d
-
-    def get_def(self, context):
-        # type: (int) -> Def
-        """
-        Get the def of this variable in context.
-
-        The `context` must be one of `SRCCTX` or `DSTCTX`
-        """
-        if context == self.SRCCTX:
-            return self.src_def
-        else:
-            return self.dst_def
-
-    def is_input(self):
-        # type: () -> bool
-        """Is this an input value to the src pattern?"""
-        return self.src_def is None and self.dst_def is None
-
-    def is_output(self):
-        # type: () -> bool
-        """Is this an output value, defined in both src and dst patterns?"""
-        return self.src_def is not None and self.dst_def is not None
-
-    def is_intermediate(self):
-        # type: () -> bool
-        """Is this an intermediate value, defined only in the src pattern?"""
-        return self.src_def is not None and self.dst_def is None
-
-    def is_temp(self):
-        # type: () -> bool
-        """Is this a temp value, defined only in the dst pattern?"""
-        return self.src_def is None and self.dst_def is not None
-
-    def get_typevar(self):
-        # type: () -> TypeVar
-        """Get the type variable representing the type of this variable."""
-        if not self.typevar:
-            # Create a TypeVar allowing all types.
-            tv = TypeVar(
-                    'typeof_{}'.format(self),
-                    'Type of the pattern variable `{}`'.format(self),
-                    ints=True, floats=True, bools=True,
-                    scalars=True, simd=True, bitvecs=True,
-                    specials=True)
-            self.original_typevar = tv
-            self.typevar = tv
-        return self.typevar
-
-    def set_typevar(self, tv):
-        # type: (TypeVar) -> None
-        self.typevar = tv
-
-    def has_free_typevar(self):
-        # type: () -> bool
-        """
-        Check if this variable has a free type variable.
-
-        If not, the type of this variable is computed from the type of another
-        variable.
-        """
-        if not self.typevar or self.typevar.is_derived:
-            return False
-        return self.typevar is self.original_typevar
-
-    def rust_type(self):
-        # type: () -> str
-        """
-        Get a Rust expression that computes the type of this variable.
-
-        It is assumed that local variables exist corresponding to the free type
-        variables.
-        """
-        return self.typevar.rust_expr()
-
-
-class Apply(Expr):
-    """
-    Apply an instruction to arguments.
-
-    An `Apply` AST expression is created by using function call syntax on
-    instructions. This applies to both bound and unbound polymorphic
-    instructions:
-
-    >>> from base.instructions import jump, iadd
-    >>> jump('next', ())
-    Apply(jump, ('next', ()))
-    >>> iadd.i32('x', 'y')
-    Apply(iadd.i32, ('x', 'y'))
-
-    :param inst: The instruction being applied, an `Instruction` or
-                 `BoundInstruction` instance.
-    :param args: Tuple of arguments.
-    """
-
-    def __init__(self, inst, args):
-        # type: (instructions.MaybeBoundInst, Tuple[Expr, ...]) -> None  # noqa
-        if isinstance(inst, instructions.BoundInstruction):
-            self.inst = inst.inst
-            self.typevars = inst.typevars
-        else:
-            assert isinstance(inst, instructions.Instruction)
-            self.inst = inst
-            self.typevars = ()
-        self.args = args
-        assert len(self.inst.ins) == len(args)
-
-        # Check that the kinds of Literals arguments match the expected Operand
-        for op_idx in self.inst.imm_opnums:
-            arg = self.args[op_idx]
-            op = self.inst.ins[op_idx]
-
-            if isinstance(arg, Literal):
-                assert arg.kind == op.kind, \
-                    "Passing literal {} to field of wrong kind {}."\
-                    .format(arg, op.kind)
-
-    def __rlshift__(self, other):
-        # type: (Union[Var, Tuple[Var, ...]]) -> Def
-        """
-        Define variables using `var << expr` or `(v1, v2) << expr`.
-        """
-        return Def(other, self)
-
-    def instname(self):
-        # type: () -> str
-        i = self.inst.name
-        for t in self.typevars:
-            i += '.{}'.format(t)
-        return i
-
-    def __repr__(self):
-        # type: () -> str
-        return "Apply({}, {})".format(self.instname(), self.args)
-
-    def __str__(self):
-        # type: () -> str
-        args = ', '.join(map(str, self.args))
-        return '{}({})'.format(self.instname(), args)
-
-    def rust_builder(self, defs=None):
-        # type: (Sequence[Var]) -> str
-        """
-        Return a Rust Builder method call for instantiating this instruction
-        application.
-
-        The `defs` argument should be a list of variables defined by this
-        instruction. It is used to construct a result type if necessary.
-        """
-        args = ', '.join(map(str, self.args))
-        # Do we need to pass an explicit type argument?
-        if self.inst.is_polymorphic and not self.inst.use_typevar_operand:
-            args = defs[0].rust_type() + ', ' + args
-        method = self.inst.snake_name()
-        return '{}({})'.format(method, args)
-
-    def inst_predicate(self):
-        # type: () -> PredNode
-        """
-        Construct an instruction predicate that verifies the immediate operands
-        on this instruction.
-
-        Immediate operands in a source pattern can be either free variables or
-        constants like `ConstantInt` and `Enumerator`. We don't currently
-        support constraints on free variables, but we may in the future.
-        """
-        pred = None  # type: PredNode
-        iform = self.inst.format
-
-        # Examine all of the immediate operands.
-        for ffield, opnum in zip(iform.imm_fields, self.inst.imm_opnums):
-            arg = self.args[opnum]
-
-            # Ignore free variables for now. We may add variable predicates
-            # later.
-            if isinstance(arg, Var):
-                continue
-
-            pred = And.combine(pred, IsEqual(ffield, arg))
-
-        # Add checks for any bound secondary type variables.
-        # We can't check the controlling type variable this way since it may
-        # not appear as the type of an operand.
-        if len(self.typevars) > 1:
-            for bound_ty, tv in zip(self.typevars[1:],
-                                    self.inst.other_typevars):
-                if bound_ty is None:
-                    continue
-                type_chk = TypePredicate.typevar_check(self.inst, tv, bound_ty)
-                pred = And.combine(pred, type_chk)
-
-        return pred
-
-    def inst_predicate_with_ctrl_typevar(self):
-        # type: () -> PredNode
-        """
-        Same as `inst_predicate()`, but also check the controlling type
-        variable.
-        """
-        pred = self.inst_predicate()
-
-        if len(self.typevars) > 0:
-            bound_ty = self.typevars[0]
-            type_chk = None  # type: PredNode
-            if bound_ty is not None:
-                # Prefer to look at the types of input operands.
-                if self.inst.use_typevar_operand:
-                    type_chk = TypePredicate.typevar_check(
-                            self.inst, self.inst.ctrl_typevar, bound_ty)
-                else:
-                    type_chk = CtrlTypePredicate(bound_ty)
-                pred = And.combine(pred, type_chk)
-
-        return pred
-
-    def copy(self, m):
-        # type: (VarAtomMap) -> Apply
-        """
-        Return a copy of this Expr with vars replaced with fresh variables,
-        in accordance with the map m. Update m as necessary.
-        """
-        return Apply(self.inst, tuple(map(lambda e: replace_var(e, m),
-                                          self.args)))
-
-    def vars(self):
-        # type: () -> Set[Var]
-        """Return the set of all Vars in self that correspond to SSA values"""
-        res = set()
-        for i in self.inst.value_opnums:
-            arg = self.args[i]
-            assert isinstance(arg, Var)
-            res.add(arg)
-        return res
-
-    def substitution(self, other, s):
-        # type: (Apply, VarAtomMap) -> Optional[VarAtomMap]
-        """
-        If there is a substitution from Var->Atom that converts self to other,
-        return it, otherwise return None. Note that this is strictly weaker
-        than unification (see TestXForm.test_subst_enum_bad_var_const for
-        example).
-        """
-        if self.inst != other.inst:
-            return None
-
-        # Guaranteed by self.inst == other.inst
-        assert (len(self.args) == len(other.args))
-
-        for (self_a, other_a) in zip(self.args, other.args):
-            assert isinstance(self_a, Atom) and isinstance(other_a, Atom)
-
-            if (isinstance(self_a, Var)):
-                if (self_a not in s):
-                    s[self_a] = other_a
-                else:
-                    if (s[self_a] != other_a):
-                        return None
-            elif isinstance(other_a, Var):
-                assert isinstance(self_a, Literal)
-                if (other_a not in s):
-                    s[other_a] = self_a
-                else:
-                    if s[other_a] != self_a:
-                        return None
-            else:
-                assert (isinstance(self_a, Literal) and
-                        isinstance(other_a, Literal))
-                # Guaranteed by self.inst == other.inst
-                assert self_a.kind == other_a.kind
-                if (self_a.value != other_a.value):
-                    return None
-
-        return s
-
-
-class Literal(Atom):
-    """
-    Base Class for all literal expressions in the DSL.
-    """
-    def __init__(self, kind, value):
-        # type: (ImmediateKind, Any) -> None
-        self.kind = kind
-        self.value = value
-
-    def __eq__(self, other):
-        # type: (Any) -> bool
-        if not isinstance(other, Literal):
-            return False
-
-        if self.kind != other.kind:
-            return False
-
-        # Can't just compare value here, as comparison Any <> Any returns Any
-        return repr(self) == repr(other)
-
-    def __ne__(self, other):
-        # type: (Any) -> bool
-        return not self.__eq__(other)
-
-    def __repr__(self):
-        # type: () -> str
-        return '{}.{}'.format(self.kind, self.value)
-
-
-class ConstantInt(Literal):
-    """
-    A value of an integer immediate operand.
-
-    Immediate operands like `imm64` or `offset32` can be specified in AST
-    expressions using the call syntax: `imm64(5)` which creates a `ConstantInt`
-    node.
-    """
-
-    def __init__(self, kind, value):
-        # type: (ImmediateKind, int) -> None
-        super(ConstantInt, self).__init__(kind, value)
-
-    def __str__(self):
-        # type: () -> str
-        # If the value is in the signed imm64 range, print it as-is.
-        if self.value >= -(2**63) and self.value < (2**63):
-            return str(self.value)
-        # Otherwise if the value is in the unsigned imm64 range, print its
-        # bitwise counterpart in the signed imm64 range.
-        if self.value >= (2**63) and self.value < (2**64):
-            return str(self.value - (2**64))
-        assert False, "immediate value not in signed or unsigned imm64 range"
-
-
-class ConstantBits(Literal):
-    """
-    A bitwise value of an immediate operand.
-
-    This is used to create bitwise exact floating point constants using
-    `ieee32.bits(0x80000000)`.
-    """
-
-    def __init__(self, kind, bits):
-        # type: (ImmediateKind, int) -> None
-        v = '{}::with_bits({:#x})'.format(kind.rust_type, bits)
-        super(ConstantBits, self).__init__(kind, v)
-
-    def __str__(self):
-        # type: () -> str
-        """
-        Get the Rust expression form of this constant.
-        """
-        return str(self.value)
-
-
-class Enumerator(Literal):
-    """
-    A value of an enumerated immediate operand.
-
-    Some immediate operand kinds like `intcc` and `floatcc` have an enumerated
-    range of values corresponding to a Rust enum type. An `Enumerator` object
-    is an AST leaf node representing one of the values.
-
-    :param kind: The enumerated `ImmediateKind` containing the value.
-    :param value: The textual IR representation of the value.
-
-    `Enumerator` nodes are not usually created directly. They are created by
-    using the dot syntax on immediate kinds: `intcc.ult`.
-    """
-
-    def __init__(self, kind, value):
-        # type: (ImmediateKind, str) -> None
-        super(Enumerator, self).__init__(kind, value)
-
-    def __str__(self):
-        # type: () -> str
-        """
-        Get the Rust expression form of this enumerator.
-        """
-        return self.kind.rust_enumerator(self.value)
diff --git a/cranelift/codegen/meta-python/cdsl/formats.py b/cranelift/codegen/meta-python/cdsl/formats.py
deleted file mode 100644
index 3eee94d248..0000000000
--- a/cranelift/codegen/meta-python/cdsl/formats.py
+++ /dev/null
@@ -1,268 +0,0 @@
-"""Classes for describing instruction formats."""
-from __future__ import absolute_import
-from .operands import OperandKind, VALUE, VARIABLE_ARGS
-from .operands import Operand  # noqa
-
-# The typing module is only required by mypy, and we don't use these imports
-# outside type comments.
-try:
-    from typing import Dict, List, Tuple, Union, Any, Sequence, Iterable  # noqa
-except ImportError:
-    pass
-
-
-class InstructionContext(object):
-    """
-    Most instruction predicates refer to immediate fields of a specific
-    instruction format, so their `predicate_context()` method returns the
-    specific instruction format.
-
-    Predicates that only care about the types of SSA values are independent of
-    the instruction format. They can be evaluated in the context of any
-    instruction.
-
-    The singleton `InstructionContext` class serves as the predicate context
-    for these predicates.
-    """
-
-    def __init__(self):
-        # type: () -> None
-        self.name = 'inst'
-
-
-# Singleton instance.
-instruction_context = InstructionContext()
-
-
-class InstructionFormat(object):
-    """
-    Every instruction opcode has a corresponding instruction format which
-    determines the number of operands and their kinds. Instruction formats are
-    identified structurally, i.e., the format of an instruction is derived from
-    the kinds of operands used in its declaration.
-
-    The instruction format stores two separate lists of operands: Immediates
-    and values. Immediate operands (including entity references) are
-    represented as explicit members in the `InstructionData` variants. The
-    value operands are stored differently, depending on how many there are.
-    Beyond a certain point, instruction formats switch to an external value
-    list for storing value arguments. Value lists can hold an arbitrary number
-    of values.
-
-    All instruction formats must be predefined in the
-    :py:mod:`cranelift.formats` module.
-
-    :param kinds: List of `OperandKind` objects describing the operands.
-    :param name: Instruction format name in CamelCase. This is used as a Rust
-        variant name in both the `InstructionData` and `InstructionFormat`
-        enums.
-    :param typevar_operand: Index of the value input operand that is used to
-        infer the controlling type variable. By default, this is `0`, the first
-        `value` operand. The index is relative to the values only, ignoring
-        immediate operands.
-    """
-
-    # Map (imm_kinds, num_value_operands) -> format
-    _registry = dict()  # type: Dict[Tuple[Tuple[OperandKind, ...], int, bool], InstructionFormat]  # noqa
-
-    # All existing formats.
-    all_formats = list()  # type: List[InstructionFormat]
-
-    def __init__(self, *kinds, **kwargs):
-        # type: (*Union[OperandKind, Tuple[str, OperandKind]], **Any) -> None # noqa
-        self.name = kwargs.get('name', None)  # type: str
-        self.parent = instruction_context
-
-        # The number of value operands stored in the format, or `None` when
-        # `has_value_list` is set.
-        self.num_value_operands = 0
-        # Does this format use a value list for storing value operands?
-        self.has_value_list = False
-        # Operand fields for the immediate operands. All other instruction
-        # operands are values or variable argument lists. They are all handled
-        # specially.
-        self.imm_fields = tuple(self._process_member_names(kinds))
-
-        # The typevar_operand argument must point to a 'value' operand.
-        self.typevar_operand = kwargs.get('typevar_operand', None)  # type: int
-        if self.typevar_operand is not None:
-            if not self.has_value_list:
-                assert self.typevar_operand < self.num_value_operands, \
-                        "typevar_operand must indicate a 'value' operand"
-        elif self.has_value_list or self.num_value_operands > 0:
-            # Default to the first 'value' operand, if there is one.
-            self.typevar_operand = 0
-
-        # Compute a signature for the global registry.
-        imm_kinds = tuple(f.kind for f in self.imm_fields)
-        sig = (imm_kinds, self.num_value_operands, self.has_value_list)
-        if sig in InstructionFormat._registry:
-            raise RuntimeError(
-                "Format '{}' has the same signature as existing format '{}'"
-                .format(self.name, InstructionFormat._registry[sig]))
-        InstructionFormat._registry[sig] = self
-        InstructionFormat.all_formats.append(self)
-
-    def args(self):
-        # type: () -> FormatField
-        """
-        Provides a ValueListField, which is derived from FormatField,
-        corresponding to the full ValueList of the instruction format. This
-        is useful for creating predicates for instructions which use variadic
-        arguments.
-        """
-
-        if self.has_value_list:
-            return ValueListField(self)
-        return None
-
-    def _process_member_names(self, kinds):
-        # type: (Sequence[Union[OperandKind, Tuple[str, OperandKind]]]) -> Iterable[FormatField]  # noqa
-        """
-        Extract names of all the immediate operands in the kinds tuple.
-
-        Each entry is either an `OperandKind` instance, or a `(member, kind)`
-        pair. The member names correspond to members in the Rust
-        `InstructionData` data structure.
-
-        Updates the fields `self.num_value_operands` and `self.has_value_list`.
-
-        Yields the immediate operand fields.
-        """
-        inum = 0
-        for arg in kinds:
-            if isinstance(arg, OperandKind):
-                member = arg.default_member
-                k = arg
-            else:
-                member, k = arg
-
-            # We define 'immediate' as not a value or variable arguments.
-            if k is VALUE:
-                self.num_value_operands += 1
-            elif k is VARIABLE_ARGS:
-                self.has_value_list = True
-            else:
-                yield FormatField(self, inum, k, member)
-                inum += 1
-
-    def __str__(self):
-        # type: () -> str
-        args = ', '.join(
-                '{}: {}'.format(f.member, f.kind) for f in self.imm_fields)
-        return '{}(imms=({}), vals={})'.format(
-                self.name, args, self.num_value_operands)
-
-    def __getattr__(self, attr):
-        # type: (str) -> FormatField
-        """
-        Make immediate instruction format members available as attributes.
-
-        Each non-value format member becomes a corresponding `FormatField`
-        attribute.
-        """
-        for f in self.imm_fields:
-            if f.member == attr:
-                # Cache this field attribute so we won't have to search again.
-                setattr(self, attr, f)
-                return f
-
-        raise AttributeError(
-                '{} is neither a {} member or a '
-                .format(attr, self.name) +
-                'normal InstructionFormat attribute')
-
-    @staticmethod
-    def lookup(ins, outs):
-        # type: (Sequence[Operand], Sequence[Operand]) -> InstructionFormat
-        """
-        Find an existing instruction format that matches the given lists of
-        instruction inputs and outputs.
-
-        The `ins` and `outs` arguments correspond to the
-        :py:class:`Instruction` arguments of the same name, except they must be
-        tuples of :py:`Operand` objects.
-        """
-        # Construct a signature.
-        imm_kinds = tuple(op.kind for op in ins if op.is_immediate())
-        num_values = sum(1 for op in ins if op.is_value())
-        has_varargs = (VARIABLE_ARGS in tuple(op.kind for op in ins))
-
-        sig = (imm_kinds, num_values, has_varargs)
-        if sig in InstructionFormat._registry:
-            return InstructionFormat._registry[sig]
-
-        # Try another value list format as an alternative.
-        sig = (imm_kinds, 0, True)
-        if sig in InstructionFormat._registry:
-            return InstructionFormat._registry[sig]
-
-        raise RuntimeError(
-                'No instruction format matches '
-                'imms={}, vals={}, varargs={}'.format(
-                    imm_kinds, num_values, has_varargs))
-
-    @staticmethod
-    def extract_names(globs):
-        # type: (Dict[str, Any]) -> None
-        """
-        Given a dict mapping name -> object as returned by `globals()`, find
-        all the InstructionFormat objects and set their name from the dict key.
-        This is used to name a bunch of global values in a module.
-        """
-        for name, obj in globs.items():
-            if isinstance(obj, InstructionFormat):
-                assert obj.name is None
-                obj.name = name
-
-
-class FormatField(object):
-    """
-    An immediate field in an instruction format.
-
-    This corresponds to a single member of a variant of the `InstructionData`
-    data type.
-
-    :param iform: Parent `InstructionFormat`.
-    :param immnum: Immediate operand number in parent.
-    :param kind: Immediate Operand kind.
-    :param member: Member name in `InstructionData` variant.
-    """
-
-    def __init__(self, iform, immnum, kind, member):
-        # type: (InstructionFormat, int, OperandKind, str) -> None
-        self.format = iform
-        self.immnum = immnum
-        self.kind = kind
-        self.member = member
-
-    def __str__(self):
-        # type: () -> str
-        return '{}.{}'.format(self.format.name, self.member)
-
-    def rust_destructuring_name(self):
-        # type: () -> str
-        return self.member
-
-    def rust_name(self):
-        # type: () -> str
-        return self.member
-
-
-class ValueListField(FormatField):
-    """
-    The full value list field of an instruction format.
-
-    This corresponds to all Value-type members of a variant of the
-    `InstructionData` format, which contains a ValueList.
-
-    :param iform: Parent `InstructionFormat`.
-    """
-    def __init__(self, iform):
-        # type: (InstructionFormat) -> None
-        self.format = iform
-        self.member = "args"
-
-    def rust_destructuring_name(self):
-        # type: () -> str
-        return 'ref {}'.format(self.member)
diff --git a/cranelift/codegen/meta-python/cdsl/instructions.py b/cranelift/codegen/meta-python/cdsl/instructions.py
deleted file mode 100644
index 34741991da..0000000000
--- a/cranelift/codegen/meta-python/cdsl/instructions.py
+++ /dev/null
@@ -1,446 +0,0 @@
-"""Classes for defining instructions."""
-from __future__ import absolute_import
-from . import camel_case
-from .types import ValueType
-from .operands import Operand
-from .formats import InstructionFormat
-
-try:
-    from typing import Union, Sequence, List, Tuple, Any, TYPE_CHECKING  # noqa
-    from typing import Dict # noqa
-    if TYPE_CHECKING:
-        from .ast import Expr, Apply, Var, Def, VarAtomMap  # noqa
-        from .typevar import TypeVar  # noqa
-        from .ti import TypeConstraint  # noqa
-        from .xform import XForm, Rtl
-        # List of operands for ins/outs:
-        OpList = Union[Sequence[Operand], Operand]
-        ConstrList = Union[Sequence[TypeConstraint], TypeConstraint]
-        MaybeBoundInst = Union['Instruction', 'BoundInstruction']
-        InstructionSemantics = Sequence[XForm]
-        SemDefCase = Union[Rtl, Tuple[Rtl, Sequence[TypeConstraint]], XForm]
-except ImportError:
-    pass
-
-
-class InstructionGroup(object):
-    """
-    Every instruction must belong to exactly one instruction group. A given
-    target architecture can support instructions from multiple groups, and it
-    does not necessarily support all instructions in a group.
-
-    New instructions are automatically added to the currently open instruction
-    group.
-    """
-
-    # The currently open instruction group.
-    _current = None  # type: InstructionGroup
-
-    def open(self):
-        # type: () -> None
-        """
-        Open this instruction group such that future new instructions are
-        added to this group.
-        """
-        assert InstructionGroup._current is None, (
-                "Can't open {} since {} is already open"
-                .format(self, InstructionGroup._current))
-        InstructionGroup._current = self
-
-    def close(self):
-        # type: () -> None
-        """
-        Close this instruction group. This function should be called before
-        opening another instruction group.
-        """
-        assert InstructionGroup._current is self, (
-                "Can't close {}, the open instuction group is {}"
-                .format(self, InstructionGroup._current))
-        InstructionGroup._current = None
-
-    def __init__(self, name, doc):
-        # type: (str, str) -> None
-        self.name = name
-        self.__doc__ = doc
-        self.instructions = []  # type: List[Instruction]
-        self.open()
-
-    @staticmethod
-    def append(inst):
-        # type: (Instruction) -> None
-        assert InstructionGroup._current, \
-                "Open an instruction group before defining instructions."
-        InstructionGroup._current.instructions.append(inst)
-
-
-class Instruction(object):
-    """
-    The operands to the instruction are specified as two tuples: ``ins`` and
-    ``outs``. Since the Python singleton tuple syntax is a bit awkward, it is
-    allowed to specify a singleton as just the operand itself, i.e., `ins=x`
-    and `ins=(x,)` are both allowed and mean the same thing.
-
-    :param name: Instruction mnemonic, also becomes opcode name.
-    :param doc: Documentation string.
-    :param ins: Tuple of input operands. This can be a mix of SSA value
-                operands and other operand kinds.
-    :param outs: Tuple of output operands. The output operands must be SSA
-                values or `variable_args`.
-    :param constraints: Tuple of instruction-specific TypeConstraints.
-    :param is_terminator: This is a terminator instruction.
-    :param is_branch: This is a branch instruction.
-    :param is_indirect_branch: This is an indirect branch instruction.
-    :param is_call: This is a call instruction.
-    :param is_return: This is a return instruction.
-    :param is_ghost: This is a ghost instruction, which has no encoding and no
-                     other register allocation constraints.
-    :param can_trap: This instruction can trap.
-    :param can_load: This instruction can load from memory.
-    :param can_store: This instruction can store to memory.
-    :param other_side_effects: Instruction has other side effects.
-    """
-
-    # Boolean instruction attributes that can be passed as keyword arguments to
-    # the constructor. Map attribute name to doc comment for generated Rust
-    # code.
-    ATTRIBS = {
-            'is_terminator': 'True for instructions that terminate the EBB.',
-            'is_branch': 'True for all branch or jump instructions.',
-            'is_indirect_branch':
-            'True for all indirect branch or jump instructions.',
-            'is_call': 'Is this a call instruction?',
-            'is_return': 'Is this a return instruction?',
-            'is_ghost': 'Is this a ghost instruction?',
-            'can_load': 'Can this instruction read from memory?',
-            'can_store': 'Can this instruction write to memory?',
-            'can_trap': 'Can this instruction cause a trap?',
-            'other_side_effects':
-            'Does this instruction have other side effects besides can_*',
-            'writes_cpu_flags': 'Does this instruction write to CPU flags?',
-            }
-
-    def __init__(self, name, doc, ins=(), outs=(), constraints=(), **kwargs):
-        # type: (str, str, OpList, OpList, ConstrList, **Any) -> None
-        self.name = name
-        self.camel_name = camel_case(name)
-        self.__doc__ = doc
-        self.ins = self._to_operand_tuple(ins)
-        self.outs = self._to_operand_tuple(outs)
-        self.constraints = self._to_constraint_tuple(constraints)
-        self.format = InstructionFormat.lookup(self.ins, self.outs)
-        self.semantics = None  # type: InstructionSemantics
-
-        # Opcode number, assigned by gen_instr.py.
-        self.number = None  # type: int
-
-        # Indexes into `self.outs` for value results.
-        # Other results are `variable_args`.
-        self.value_results = tuple(
-                i for i, o in enumerate(self.outs) if o.is_value())
-        # Indexes into `self.ins` for value operands.
-        self.value_opnums = tuple(
-                i for i, o in enumerate(self.ins) if o.is_value())
-        # Indexes into `self.ins` for non-value operands.
-        self.imm_opnums = tuple(
-                i for i, o in enumerate(self.ins) if o.is_immediate())
-
-        self._verify_polymorphic()
-        for attr in kwargs:
-            if attr not in Instruction.ATTRIBS:
-                raise AssertionError(
-                        "unknown instruction attribute '" + attr + "'")
-        for attr in Instruction.ATTRIBS:
-            setattr(self, attr, not not kwargs.get(attr, False))
-
-        # Infer the 'writes_cpu_flags' field value.
-        if 'writes_cpu_flags' not in kwargs:
-            self.writes_cpu_flags = any(
-                out.is_cpu_flags() for out in self.outs)
-
-        InstructionGroup.append(self)
-
-    def __str__(self):
-        # type: () -> str
-        prefix = ', '.join(o.name for o in self.outs)
-        if prefix:
-            prefix = prefix + ' = '
-        suffix = ', '.join(o.name for o in self.ins)
-        return '{}{} {}'.format(prefix, self.name, suffix)
-
-    def snake_name(self):
-        # type: () -> str
-        """
-        Get the snake_case name of this instruction.
-
-        Keywords in Rust and Python are altered by appending a '_'
-        """
-        if self.name == 'return':
-            return 'return_'
-        else:
-            return self.name
-
-    def blurb(self):
-        # type: () -> str
-        """Get the first line of the doc comment"""
-        for line in self.__doc__.split('\n'):
-            line = line.strip()
-            if line:
-                return line
-        return ""
-
-    def _verify_polymorphic(self):
-        # type: () -> None
-        """
-        Check if this instruction is polymorphic, and verify its use of type
-        variables.
-        """
-        poly_ins = [
-                i for i in self.value_opnums
-                if self.ins[i].typevar.free_typevar()]
-        poly_outs = [
-                i for i, o in enumerate(self.outs)
-                if o.is_value() and o.typevar.free_typevar()]
-        self.is_polymorphic = len(poly_ins) > 0 or len(poly_outs) > 0
-        if not self.is_polymorphic:
-            return
-
-        # Prefer to use the typevar_operand to infer the controlling typevar.
-        self.use_typevar_operand = False
-        typevar_error = None
-        tv_op = self.format.typevar_operand
-        if tv_op is not None and tv_op < len(self.value_opnums):
-            try:
-                opnum = self.value_opnums[tv_op]
-                tv = self.ins[opnum].typevar
-                if tv is tv.free_typevar() or tv.singleton_type() is not None:
-                    self.other_typevars = self._verify_ctrl_typevar(tv)
-                    self.ctrl_typevar = tv
-                    self.use_typevar_operand = True
-            except RuntimeError as e:
-                typevar_error = e
-
-        if not self.use_typevar_operand:
-            # The typevar_operand argument doesn't work. Can we infer from the
-            # first result instead?
-            if len(self.outs) == 0:
-                if typevar_error:
-                    raise typevar_error
-                else:
-                    raise RuntimeError(
-                            "typevar_operand must be a free type variable")
-            tv = self.outs[0].typevar
-            if tv is not tv.free_typevar():
-                raise RuntimeError("first result must be a free type variable")
-            self.other_typevars = self._verify_ctrl_typevar(tv)
-            self.ctrl_typevar = tv
-
-    def _verify_ctrl_typevar(self, ctrl_typevar):
-        # type: (TypeVar) -> List[TypeVar]
-        """
-        Verify that the use of TypeVars is consistent with `ctrl_typevar` as
-        the controlling type variable.
-
-        All polymorhic inputs must either be derived from `ctrl_typevar` or be
-        independent free type variables only used once.
-
-        All polymorphic results must be derived from `ctrl_typevar`.
-
-        Return list of other type variables used, or raise an error.
-        """
-        other_tvs = []  # type: List[TypeVar]
-        # Check value inputs.
-        for opnum in self.value_opnums:
-            typ = self.ins[opnum].typevar
-            tv = typ.free_typevar()
-            # Non-polymorphic or derived from ctrl_typevar is OK.
-            if tv is None or tv is ctrl_typevar:
-                continue
-            # No other derived typevars allowed.
-            if typ is not tv:
-                raise RuntimeError(
-                        "{}: type variable {} must be derived from {}"
-                        .format(self.ins[opnum], typ.name, ctrl_typevar))
-            # Other free type variables can only be used once each.
-            if tv in other_tvs:
-                raise RuntimeError(
-                        "type variable {} can't be used more than once"
-                        .format(tv.name))
-            other_tvs.append(tv)
-
-        # Check outputs.
-        for result in self.outs:
-            if not result.is_value():
-                continue
-            typ = result.typevar
-            tv = typ.free_typevar()
-            # Non-polymorphic or derived from ctrl_typevar is OK.
-            if tv is None or tv is ctrl_typevar:
-                continue
-            raise RuntimeError(
-                    "type variable in output not derived from ctrl_typevar")
-
-        return other_tvs
-
-    def all_typevars(self):
-        # type: () -> List[TypeVar]
-        """
-        Get a list of all type variables in the instruction.
-        """
-        if self.is_polymorphic:
-            return [self.ctrl_typevar] + self.other_typevars
-        else:
-            return []
-
-    @staticmethod
-    def _to_operand_tuple(x):
-        # type: (Union[Sequence[Operand], Operand]) -> Tuple[Operand, ...]
-        # Allow a single Operand instance instead of the awkward singleton
-        # tuple syntax.
-        if isinstance(x, Operand):
-            y = (x,)  # type: Tuple[Operand, ...]
-        else:
-            y = tuple(x)
-        for op in y:
-            assert isinstance(op, Operand)
-        return y
-
-    @staticmethod
-    def _to_constraint_tuple(x):
-        # type: (ConstrList) -> Tuple[TypeConstraint, ...]
-        """
-        Allow a single TypeConstraint instance instead of the awkward singleton
-        tuple syntax.
-        """
-        # import placed here to avoid circular dependency
-        from .ti import TypeConstraint  # noqa
-        if isinstance(x, TypeConstraint):
-            y = (x,)  # type: Tuple[TypeConstraint, ...]
-        else:
-            y = tuple(x)
-        for op in y:
-            assert isinstance(op, TypeConstraint)
-        return y
-
-    def bind(self, *args):
-        # type: (*ValueType) -> BoundInstruction
-        """
-        Bind a polymorphic instruction to a concrete list of type variable
-        values.
-        """
-        assert self.is_polymorphic
-        return BoundInstruction(self, args)
-
-    def __getattr__(self, name):
-        # type: (str) -> BoundInstruction
-        """
-        Bind a polymorphic instruction to a single type variable with dot
-        syntax:
-
-        >>> iadd.i32
-        """
-        assert name != 'any', 'Wildcard not allowed for ctrl_typevar'
-        return self.bind(ValueType.by_name(name))
-
-    def fully_bound(self):
-        # type: () -> Tuple[Instruction, Tuple[ValueType, ...]]
-        """
-        Verify that all typevars have been bound, and return a
-        `(inst, typevars)` pair.
-
-        This version in `Instruction` itself allows non-polymorphic
-        instructions to duck-type as `BoundInstruction`\\s.
-        """
-        assert not self.is_polymorphic, self
-        return (self, ())
-
-    def __call__(self, *args):
-        # type: (*Expr) -> Apply
-        """
-        Create an `ast.Apply` AST node representing the application of this
-        instruction to the arguments.
-        """
-        from .ast import Apply  # noqa
-        return Apply(self, args)
-
-    def set_semantics(self, src, *dsts):
-        # type: (Union[Def, Apply], *SemDefCase) -> None
-        """Set our semantics."""
-        from semantics import verify_semantics
-        from .xform import XForm, Rtl
-
-        sem = []  # type: List[XForm]
-        for dst in dsts:
-            if isinstance(dst, Rtl):
-                sem.append(XForm(Rtl(src).copy({}), dst))
-            elif isinstance(dst, XForm):
-                sem.append(XForm(
-                    dst.src.copy({}),
-                    dst.dst.copy({}),
-                    dst.constraints))
-            else:
-                assert isinstance(dst, tuple)
-                sem.append(XForm(Rtl(src).copy({}), dst[0],
-                                 constraints=dst[1]))
-
-        verify_semantics(self, Rtl(src), sem)
-
-        self.semantics = sem
-
-
-class BoundInstruction(object):
-    """
-    A polymorphic `Instruction` bound to concrete type variables.
-    """
-
-    def __init__(self, inst, typevars):
-        # type: (Instruction, Tuple[ValueType, ...]) -> None
-        self.inst = inst
-        self.typevars = typevars
-        assert len(typevars) <= 1 + len(inst.other_typevars)
-
-    def __str__(self):
-        # type: () -> str
-        return '.'.join([self.inst.name, ] + list(map(str, self.typevars)))
-
-    def bind(self, *args):
-        # type: (*ValueType) -> BoundInstruction
-        """
-        Bind additional typevars.
-        """
-        return BoundInstruction(self.inst, self.typevars + args)
-
-    def __getattr__(self, name):
-        # type: (str) -> BoundInstruction
-        """
-        Bind an additional typevar dot syntax:
-
-        >>> uext.i32.i8
-        """
-        if name == 'any':
-            # This is a wild card bind represented as a None type variable.
-            return self.bind(None)
-
-        return self.bind(ValueType.by_name(name))
-
-    def fully_bound(self):
-        # type: () -> Tuple[Instruction, Tuple[ValueType, ...]]
-        """
-        Verify that all typevars have been bound, and return a
-        `(inst, typevars)` pair.
-        """
-        if len(self.typevars) < 1 + len(self.inst.other_typevars):
-            unb = ', '.join(
-                    str(tv) for tv in
-                    self.inst.other_typevars[len(self.typevars) - 1:])
-            raise AssertionError("Unbound typevar {} in {}".format(unb, self))
-        assert len(self.typevars) == 1 + len(self.inst.other_typevars)
-        return (self.inst, self.typevars)
-
-    def __call__(self, *args):
-        # type: (*Expr) -> Apply
-        """
-        Create an `ast.Apply` AST node representing the application of this
-        instruction to the arguments.
-        """
-        from .ast import Apply  # noqa
-        return Apply(self, args)
diff --git a/cranelift/codegen/meta-python/cdsl/isa.py b/cranelift/codegen/meta-python/cdsl/isa.py
deleted file mode 100644
index 3e3ac0b4fe..0000000000
--- a/cranelift/codegen/meta-python/cdsl/isa.py
+++ /dev/null
@@ -1,455 +0,0 @@
-"""Defining instruction set architectures."""
-from __future__ import absolute_import
-from collections import OrderedDict
-from .predicates import And, TypePredicate
-from .registers import RegClass, Register, Stack
-from .ast import Apply
-from .types import ValueType
-from .instructions import InstructionGroup
-
-# The typing module is only required by mypy, and we don't use these imports
-# outside type comments.
-try:
-    from typing import Tuple, Union, Any, Iterable, Sequence, List, Set, Dict, TYPE_CHECKING  # noqa
-    if TYPE_CHECKING:
-        from .instructions import MaybeBoundInst, InstructionFormat  # noqa
-        from .predicates import PredNode, PredKey  # noqa
-        from .settings import SettingGroup  # noqa
-        from .registers import RegBank  # noqa
-        from .xform import XFormGroup  # noqa
-        OperandConstraint = Union[RegClass, Register, int, Stack]
-        ConstraintSeq = Union[OperandConstraint, Tuple[OperandConstraint, ...]]
-        # Instruction specification for encodings. Allows for predicated
-        # instructions.
-        InstSpec = Union[MaybeBoundInst, Apply]
-        BranchRange = Sequence[int]
-        # A recipe predicate consisting of an ISA predicate and an instruction
-        # predicate.
-        RecipePred = Tuple[PredNode, PredNode]
-except ImportError:
-    pass
-
-
-class TargetISA(object):
-    """
-    A target instruction set architecture.
-
-    The `TargetISA` class collects everything known about a target ISA.
-
-    :param name: Short mnemonic name for the ISA.
-    :param instruction_groups: List of `InstructionGroup` instances that are
-        relevant for this ISA.
-    """
-
-    def __init__(self, name, instruction_groups):
-        # type: (str, Sequence[InstructionGroup]) -> None
-        self.name = name
-        self.settings = None  # type: SettingGroup
-        self.instruction_groups = instruction_groups
-        self.cpumodes = list()  # type: List[CPUMode]
-        self.regbanks = list()  # type: List[RegBank]
-        self.legalize_codes = OrderedDict()  # type: OrderedDict[XFormGroup, int]  # noqa
-        # Unique copies of all predicates.
-        self._predicates = dict()  # type: Dict[PredKey, PredNode]
-
-        assert InstructionGroup._current is None,\
-            "InstructionGroup {} is still open"\
-            .format(InstructionGroup._current.name)
-
-    def __str__(self):
-        # type: () -> str
-        return self.name
-
-    def finish(self):
-        # type: () -> TargetISA
-        """
-        Finish the definition of a target ISA after adding all CPU modes and
-        settings.
-
-        This computes some derived properties that are used in multiple
-        places.
-
-        :returns self:
-        """
-        self._collect_encoding_recipes()
-        self._collect_predicates()
-        self._collect_legalize_codes()
-        return self
-
-    def _collect_encoding_recipes(self):
-        # type: () -> None
-        """
-        Collect and number all encoding recipes in use.
-        """
-        self.all_recipes = list()  # type: List[EncRecipe]
-        rcps = set()  # type: Set[EncRecipe]
-        for cpumode in self.cpumodes:
-            for enc in cpumode.encodings:
-                recipe = enc.recipe
-                if recipe not in rcps:
-                    assert recipe.number is None
-                    recipe.number = len(rcps)
-                    rcps.add(recipe)
-                    self.all_recipes.append(recipe)
-                    # Make sure ISA predicates are registered.
-                    if recipe.isap:
-                        recipe.isap = self.unique_pred(recipe.isap)
-                        self.settings.number_predicate(recipe.isap)
-                    recipe.instp = self.unique_pred(recipe.instp)
-
-    def _collect_predicates(self):
-        # type: () -> None
-        """
-        Collect and number all predicates in use.
-
-        Ensures that all ISA predicates have an assigned bit number in
-        `self.settings`.
-        """
-        self.instp_number = OrderedDict()  # type: OrderedDict[PredNode, int]
-        for cpumode in self.cpumodes:
-            for enc in cpumode.encodings:
-                instp = enc.instp
-                if instp and instp not in self.instp_number:
-                    # assign predicate number starting from 0.
-                    n = len(self.instp_number)
-                    self.instp_number[instp] = n
-
-                # All referenced ISA predicates must have a number in
-                # `self.settings`. This may cause some parent predicates to be
-                # replicated here, which is OK.
-                if enc.isap:
-                    self.settings.number_predicate(enc.isap)
-
-    def _collect_legalize_codes(self):
-        # type: () -> None
-        """
-        Make sure all legalization transforms have been assigned a code.
-        """
-        for cpumode in self.cpumodes:
-            self.legalize_code(cpumode.default_legalize)
-            for x in cpumode.type_legalize.values():
-                self.legalize_code(x)
-
-    def legalize_code(self, xgrp):
-        # type: (XFormGroup) -> int
-        """
-        Get the legalization code for the transform group `xgrp`. Assign one if
-        necessary.
-
-        Each target ISA has its own list of legalization actions with
-        associated legalize codes that appear in the encoding tables.
-
-        This method is used to maintain the registry of legalization actions
-        and their table codes.
-        """
-        if xgrp in self.legalize_codes:
-            code = self.legalize_codes[xgrp]
-        else:
-            code = len(self.legalize_codes)
-            self.legalize_codes[xgrp] = code
-        return code
-
-    def unique_pred(self, pred):
-        # type: (PredNode) -> PredNode
-        """
-        Get a unique predicate that is equivalent to `pred`.
-        """
-        if pred is None:
-            return pred
-        # TODO: We could actually perform some algebraic simplifications. It's
-        # not clear if it is worthwhile.
-        k = pred.predicate_key()
-        if k in self._predicates:
-            return self._predicates[k]
-        self._predicates[k] = pred
-        return pred
-
-
-class CPUMode(object):
-    """
-    A CPU mode determines which instruction encodings are active.
-
-    All instruction encodings are associated with exactly one `CPUMode`, and
-    all CPU modes are associated with exactly one `TargetISA`.
-
-    :param name: Short mnemonic name for the CPU mode.
-    :param target: Associated `TargetISA`.
-    """
-
-    def __init__(self, name, isa):
-        # type: (str, TargetISA) -> None
-        self.name = name
-        self.isa = isa
-        self.encodings = []  # type: List[Encoding]
-        isa.cpumodes.append(self)
-
-        # Tables for configuring legalization actions when no valid encoding
-        # exists for an instruction.
-        self.default_legalize = None  # type: XFormGroup
-        self.type_legalize = OrderedDict()  # type: OrderedDict[ValueType, XFormGroup]  # noqa
-
-    def __str__(self):
-        # type: () -> str
-        return self.name
-
-    def enc(self, *args, **kwargs):
-        # type: (*Any, **Any) -> None
-        """
-        Add a new encoding to this CPU mode.
-
-        Arguments are the `Encoding constructor arguments, except for the first
-        `CPUMode argument which is implied.
-        """
-        self.encodings.append(Encoding(self, *args, **kwargs))
-
-    def legalize_type(self, default=None, **kwargs):
-        # type: (XFormGroup, **XFormGroup) -> None
-        """
-        Configure the legalization action per controlling type variable.
-
-        Instructions that have a controlling type variable mentioned in one of
-        the arguments will be legalized according to the action specified here
-        instead of  using the `legalize_default` action.
-
-        The keyword arguments are value type names:
-
-            mode.legalize_type(i8=widen, i16=widen, i32=expand)
-
-        The `default` argument specifies the action to take for controlling
-        type variables that don't have an explicitly configured action.
-        """
-        if default is not None:
-            self.default_legalize = default
-
-        for name, xgrp in kwargs.items():
-            ty = ValueType.by_name(name)
-            self.type_legalize[ty] = xgrp
-
-    def legalize_monomorphic(self, xgrp):
-        # type: (XFormGroup) -> None
-        """
-        Configure the legalization action to take for monomorphic instructions
-        which don't have a controlling type variable.
-
-        See also `legalize_type()` for polymorphic instructions.
-        """
-        self.type_legalize[None] = xgrp
-
-    def get_legalize_action(self, ty):
-        # type: (ValueType) -> XFormGroup
-        """
-        Get the legalization action to use for `ty`.
-        """
-        return self.type_legalize.get(ty, self.default_legalize)
-
-
-class EncRecipe(object):
-    """
-    A recipe for encoding instructions with a given format.
-
-    Many different instructions can be encoded by the same recipe, but they
-    must all have the same instruction format.
-
-    The `ins` and `outs` arguments are tuples specifying the register
-    allocation constraints for the value operands and results respectively. The
-    possible constraints for an operand are:
-
-    - A `RegClass` specifying the set of allowed registers.
-    - A `Register` specifying a fixed-register operand.
-    - An integer indicating that this result is tied to a value operand, so
-      they must use the same register.
-    - A `Stack` specifying a value in a stack slot.
-
-    The `branch_range` argument must be provided for recipes that can encode
-    branch instructions. It is an `(origin, bits)` tuple describing the exact
-    range that can be encoded in a branch instruction.
-
-    For ISAs that use CPU flags in `iflags` and `fflags` value types, the
-    `clobbers_flags` is used to indicate instruction encodings that clobbers
-    the CPU flags, so they can't be used where a flag value is live.
-
-    :param name: Short mnemonic name for this recipe.
-    :param format: All encoded instructions must have this
-            :py:class:`InstructionFormat`.
-    :param base_size: Base number of bytes in the binary encoded instruction.
-    :param compute_size: Function name to use when computing actual size.
-    :param ins: Tuple of register constraints for value operands.
-    :param outs: Tuple of register constraints for results.
-    :param branch_range: `(origin, bits)` range for branches.
-    :param clobbers_flags: This instruction clobbers `iflags` and `fflags`.
-    :param instp: Instruction predicate.
-    :param isap: ISA predicate.
-    :param emit: Rust code for binary emission.
-    """
-
-    def __init__(
-            self,
-            name,                     # type: str
-            format,                   # type: InstructionFormat
-            base_size,                # type: int
-            ins,                      # type: ConstraintSeq
-            outs,                     # type: ConstraintSeq
-            compute_size=None,        # type: str
-            branch_range=None,        # type: BranchRange
-            clobbers_flags=True,      # type: bool
-            instp=None,               # type: PredNode
-            isap=None,                # type: PredNode
-            emit=None                 # type: str
-            ):
-        # type: (...) -> None
-        self.name = name
-        self.format = format
-        assert base_size >= 0
-        self.base_size = base_size
-        self.compute_size = compute_size if compute_size is not None \
-            else 'base_size'
-        self.branch_range = branch_range
-        self.clobbers_flags = clobbers_flags
-        self.instp = instp
-        self.isap = isap
-        self.emit = emit
-        if instp:
-            assert instp.predicate_context() == format
-        self.number = None  # type: int
-
-        self.ins = self._verify_constraints(ins)
-        if not format.has_value_list:
-            assert len(self.ins) == format.num_value_operands
-        self.outs = self._verify_constraints(outs)
-
-    def __str__(self):
-        # type: () -> str
-        return self.name
-
-    def _verify_constraints(self, seq):
-        # type: (ConstraintSeq) -> Sequence[OperandConstraint]
-        if not isinstance(seq, tuple):
-            seq = (seq,)
-        for c in seq:
-            if isinstance(c, int):
-                # An integer constraint is bound to a value operand.
-                # Check that it is in range.
-                assert c >= 0 and c < len(self.ins)
-            else:
-                assert (isinstance(c, RegClass)
-                        or isinstance(c, Register)
-                        or isinstance(c, Stack))
-        return seq
-
-    def ties(self):
-        # type: () -> Tuple[Dict[int, int], Dict[int, int]]
-        """
-        Return two dictionaries representing the tied operands.
-
-        The first maps input number to tied output number, the second maps
-        output number to tied input number.
-        """
-        i2o = dict()  # type: Dict[int, int]
-        o2i = dict()  # type: Dict[int, int]
-        for o, i in enumerate(self.outs):
-            if isinstance(i, int):
-                i2o[i] = o
-                o2i[o] = i
-        return (i2o, o2i)
-
-    def fixed_ops(self):
-        # type: () -> Tuple[Set[Register], Set[Register]]
-        """
-        Return two sets of registers representing the fixed input and output
-        operands.
-        """
-        i = set(r for r in self.ins if isinstance(r, Register))
-        o = set(r for r in self.outs if isinstance(r, Register))
-        return (i, o)
-
-    def recipe_pred(self):
-        # type: () -> RecipePred
-        """
-        Get the combined recipe predicate which includes both the ISA predicate
-        and the instruction predicate.
-
-        Return `None` if this recipe has neither predicate.
-        """
-        if self.isap is None and self.instp is None:
-            return None
-        else:
-            return (self.isap, self.instp)
-
-
-class Encoding(object):
-    """
-    Encoding for a concrete instruction.
-
-    An `Encoding` object ties an instruction opcode with concrete type
-    variables together with and encoding recipe and encoding bits.
-
-    The concrete instruction can be in three different forms:
-
-    1. A naked opcode: `trap` for non-polymorphic instructions.
-    2. With bound type variables: `iadd.i32` for polymorphic instructions.
-    3. With operands providing constraints: `icmp.i32(intcc.eq, x, y)`.
-
-    If the instruction is polymorphic, all type variables must be provided.
-
-    :param cpumode: The CPU mode where the encoding is active.
-    :param inst: The :py:class:`Instruction` or :py:class:`BoundInstruction`
-                 being encoded.
-    :param recipe: The :py:class:`EncRecipe` to use.
-    :param encbits: Additional encoding bits to be interpreted by `recipe`.
-    :param instp: Instruction predicate, or `None`.
-    :param isap: ISA predicate, or `None`.
-    """
-
-    def __init__(self, cpumode, inst, recipe, encbits, instp=None, isap=None):
-        # type: (CPUMode, InstSpec, EncRecipe, int, PredNode, PredNode) -> None # noqa
-        assert isinstance(cpumode, CPUMode)
-        assert isinstance(recipe, EncRecipe)
-
-        # Check for possible instruction predicates in `inst`.
-        if isinstance(inst, Apply):
-            instp = And.combine(instp, inst.inst_predicate())
-            self.inst = inst.inst
-            self.typevars = inst.typevars
-        else:
-            self.inst, self.typevars = inst.fully_bound()
-
-            # Add secondary type variables to the instruction predicate.
-            # This is already included by Apply.inst_predicate() above.
-            if len(self.typevars) > 1:
-                for tv, vt in zip(self.inst.other_typevars, self.typevars[1:]):
-                    # A None tv is an 'any' wild card: `ishl.i32.any`.
-                    if vt is None:
-                        continue
-                    typred = TypePredicate.typevar_check(self.inst, tv, vt)
-                    instp = And.combine(instp, typred)
-
-        self.cpumode = cpumode
-        assert self.inst.format == recipe.format, (
-                "Format {} must match recipe: {}".format(
-                    self.inst.format, recipe.format))
-
-        if self.inst.is_branch and not self.inst.is_indirect_branch:
-            assert recipe.branch_range, (
-                    'Recipe {} for {} must have a branch_range'
-                    .format(recipe, self.inst.name))
-
-        self.recipe = recipe
-        self.encbits = encbits
-
-        # Record specific predicates. Note that the recipe also has predicates.
-        self.instp = self.cpumode.isa.unique_pred(instp)
-        self.isap = self.cpumode.isa.unique_pred(isap)
-
-    def __str__(self):
-        # type: () -> str
-        return '[{}#{:02x}]'.format(self.recipe, self.encbits)
-
-    def ctrl_typevar(self):
-        # type: () -> ValueType
-        """
-        Get the controlling type variable for this encoding or `None`.
-        """
-        if self.typevars:
-            return self.typevars[0]
-        else:
-            return None
diff --git a/cranelift/codegen/meta-python/cdsl/operands.py b/cranelift/codegen/meta-python/cdsl/operands.py
deleted file mode 100644
index cf99645df9..0000000000
--- a/cranelift/codegen/meta-python/cdsl/operands.py
+++ /dev/null
@@ -1,251 +0,0 @@
-"""Classes for describing instruction operands."""
-from __future__ import absolute_import
-from . import camel_case
-from .types import ValueType
-from .typevar import TypeVar
-
-try:
-    from typing import Union, Dict, TYPE_CHECKING, Iterable  # noqa
-    OperandSpec = Union['OperandKind', ValueType, TypeVar]
-    if TYPE_CHECKING:
-        from .ast import Enumerator, ConstantInt, ConstantBits, Literal  # noqa
-except ImportError:
-    pass
-
-
-# Kinds of operands.
-#
-# Each instruction has an opcode and a number of operands. The opcode
-# determines the instruction format, and the format determines the number of
-# operands and the kind of each operand.
-class OperandKind(object):
-    """
-    An instance of the `OperandKind` class corresponds to a kind of operand.
-    Each operand kind has a corresponding type in the Rust representation of an
-    instruction.
-    """
-
-    def __init__(self, name, doc, default_member=None, rust_type=None):
-        # type: (str, str, str, str) -> None
-        self.name = name
-        self.__doc__ = doc
-        self.default_member = default_member
-        # The camel-cased name of an operand kind is also the Rust type used to
-        # represent it.
-        self.rust_type = rust_type or ('ir::' + camel_case(name))
-
-    def __str__(self):
-        # type: () -> str
-        return self.name
-
-    def __repr__(self):
-        # type: () -> str
-        return 'OperandKind({})'.format(self.name)
-
-
-#: An SSA value operand. This is a value defined by another instruction.
-VALUE = OperandKind(
-        'value', """
-        An SSA value defined by another instruction.
-
-        This kind of operand can represent any SSA value type, but the
-        instruction format may restrict the valid value types for a given
-        operand.
-        """)
-
-#: A variable-sized list of value operands. Use for Ebb and function call
-#: arguments.
-VARIABLE_ARGS = OperandKind(
-        'variable_args', """
-        A variable size list of `value` operands.
-
-        Use this to represent arguments passed to a function call, arguments
-        passed to an extended basic block, or a variable number of results
-        returned from an instruction.
-        """,
-        rust_type='&[Value]')
-
-
-# Instances of immediate operand types are provided in the
-# `cranelift.immediates` module.
-class ImmediateKind(OperandKind):
-    """
-    The kind of an immediate instruction operand.
-
-    :param default_member: The default member name of this kind the
-                           `InstructionData` data structure.
-    """
-
-    def __init__(
-            self, name, doc,
-            default_member='imm',
-            rust_type=None,
-            values=None):
-        # type: (str, str, str, str, Dict[str, str]) -> None
-        if rust_type is None:
-            rust_type = 'ir::immediates::' + camel_case(name)
-        super(ImmediateKind, self).__init__(
-                name, doc, default_member, rust_type)
-        self.values = values
-
-    def __repr__(self):
-        # type: () -> str
-        return 'ImmediateKind({})'.format(self.name)
-
-    def __getattr__(self, value):
-        # type: (str) -> Enumerator
-        """
-        Enumerated immediate kinds allow the use of dot syntax to produce
-        `Enumerator` AST nodes: `icmp.i32(intcc.ult, a, b)`.
-        """
-        from .ast import Enumerator  # noqa
-        if not self.values:
-            raise AssertionError(
-                    '{n} is not an enumerated operand kind: {n}.{a}'.format(
-                        n=self.name, a=value))
-        if value not in self.values:
-            raise AssertionError(
-                    'No such {n} enumerator: {n}.{a}'.format(
-                        n=self.name, a=value))
-        return Enumerator(self, value)
-
-    def __call__(self, value):
-        # type: (int) -> ConstantInt
-        """
-        Create an AST node representing a constant integer:
-
-            iconst(imm64(0))
-        """
-        from .ast import ConstantInt  # noqa
-        if self.values:
-            raise AssertionError(
-                    "{}({}): Can't make a constant numeric value for an enum"
-                    .format(self.name, value))
-        return ConstantInt(self, value)
-
-    def bits(self, bits):
-        # type: (int) -> ConstantBits
-        """
-        Create an AST literal node for the given bitwise representation of this
-        immediate operand kind.
-        """
-        from .ast import ConstantBits  # noqa
-        return ConstantBits(self, bits)
-
-    def rust_enumerator(self, value):
-        # type: (str) -> str
-        """
-        Get the qualified Rust name of the enumerator value `value`.
-        """
-        return '{}::{}'.format(self.rust_type, self.values[value])
-
-    def is_enumerable(self):
-        # type: () -> bool
-        return self.values is not None
-
-    def possible_values(self):
-        # type: () -> Iterable[Literal]
-        from cdsl.ast import Enumerator # noqa
-        assert self.is_enumerable()
-        for v in self.values.keys():
-            yield Enumerator(self, v)
-
-
-# Instances of entity reference operand types are provided in the
-# `cranelift.entities` module.
-class EntityRefKind(OperandKind):
-    """
-    The kind of an entity reference instruction operand.
-    """
-
-    def __init__(self, name, doc, default_member=None, rust_type=None):
-        # type: (str, str, str, str) -> None
-        super(EntityRefKind, self).__init__(
-                name, doc, default_member or name, rust_type)
-
-    def __repr__(self):
-        # type: () -> str
-        return 'EntityRefKind({})'.format(self.name)
-
-
-class Operand(object):
-    """
-    An instruction operand can be an *immediate*, an *SSA value*, or an *entity
-    reference*. The type of the operand is one of:
-
-    1. A :py:class:`ValueType` instance indicates an SSA value operand with a
-       concrete type.
-
-    2. A :py:class:`TypeVar` instance indicates an SSA value operand, and the
-       instruction is polymorphic over the possible concrete types that the
-       type variable can assume.
-
-    3. An :py:class:`ImmediateKind` instance indicates an immediate operand
-       whose value is encoded in the instruction itself rather than being
-       passed as an SSA value.
-
-    4. An :py:class:`EntityRefKind` instance indicates an operand that
-       references another entity in the function, typically something declared
-       in the function preamble.
-
-    """
-    def __init__(self, name, typ, doc=''):
-        # type: (str, OperandSpec, str) -> None
-        self.name = name
-        self.__doc__ = doc
-
-        # Decode the operand spec and set self.kind.
-        # Only VALUE operands have a typevar member.
-        if isinstance(typ, ValueType):
-            self.kind = VALUE
-            self.typevar = TypeVar.singleton(typ)
-        elif isinstance(typ, TypeVar):
-            self.kind = VALUE
-            self.typevar = typ
-        else:
-            assert isinstance(typ, OperandKind)
-            self.kind = typ
-
-    def get_doc(self):
-        # type: () -> str
-        if self.__doc__:
-            return self.__doc__
-        if self.kind is VALUE:
-            return self.typevar.__doc__
-        return self.kind.__doc__
-
-    def __str__(self):
-        # type: () -> str
-        return "`{}`".format(self.name)
-
-    def is_value(self):
-        # type: () -> bool
-        """
-        Is this an SSA value operand?
-        """
-        return self.kind is VALUE
-
-    def is_varargs(self):
-        # type: () -> bool
-        """
-        Is this a VARIABLE_ARGS operand?
-        """
-        return self.kind is VARIABLE_ARGS
-
-    def is_immediate(self):
-        # type: () -> bool
-        """
-        Is this an immediate operand?
-
-        Note that this includes both `ImmediateKind` operands *and* entity
-        references. It is any operand that doesn't represent a value
-        dependency.
-        """
-        return self.kind is not VALUE and self.kind is not VARIABLE_ARGS
-
-    def is_cpu_flags(self):
-        # type: () -> bool
-        """
-        Is this a CPU flags operand?
-        """
-        return self.kind is VALUE and self.typevar.name in ['iflags', 'fflags']
diff --git a/cranelift/codegen/meta-python/cdsl/predicates.py b/cranelift/codegen/meta-python/cdsl/predicates.py
deleted file mode 100644
index 0177c09fff..0000000000
--- a/cranelift/codegen/meta-python/cdsl/predicates.py
+++ /dev/null
@@ -1,448 +0,0 @@
-"""
-Cranelift predicates.
-
-A *predicate* is a function that computes a boolean result. The inputs to the
-function determine the kind of predicate:
-
-- An *ISA predicate* is evaluated on the current ISA settings together with the
-  shared settings defined in the :py:mod:`settings` module. Once a target ISA
-  has been configured, the value of all ISA predicates is known.
-
-- An *Instruction predicate* is evaluated on an instruction instance, so it can
-  inspect all the immediate fields and type variables of the instruction.
-  Instruction predicates can be evaluated before register allocation, so they
-  can not depend on specific register assignments to the value operands or
-  outputs.
-
-Predicates can also be computed from other predicates using the `And`, `Or`,
-and `Not` combinators defined in this module.
-
-All predicates have a *context* which determines where they can be evaluated.
-For an ISA predicate, the context is the ISA settings group. For an instruction
-predicate, the context is the instruction format.
-"""
-from __future__ import absolute_import
-from functools import reduce
-from .formats import instruction_context
-
-try:
-    from typing import Sequence, Tuple, Set, Any, Union, TYPE_CHECKING  # noqa
-    if TYPE_CHECKING:
-        from .formats import InstructionFormat, InstructionContext, FormatField  # noqa
-        from .instructions import Instruction  # noqa
-        from .settings import BoolSetting, SettingGroup  # noqa
-        from .types import ValueType  # noqa
-        from .typevar import TypeVar  # noqa
-        PredContext = Union[SettingGroup, InstructionFormat,
-                            InstructionContext]
-        PredLeaf = Union[BoolSetting, 'FieldPredicate', 'TypePredicate',
-                         'CtrlTypePredicate']
-        PredNode = Union[PredLeaf, 'Predicate']
-        # A predicate key is a (recursive) tuple of primitive types that
-        # uniquely describes a predicate. It is used for interning.
-        PredKey = Tuple[Any, ...]
-except ImportError:
-    pass
-
-
-def _is_parent(a, b):
-    # type: (PredContext, PredContext) -> bool
-    """
-    Return true if a is a parent of b, or equal to it.
-    """
-    while b and a is not b:
-        b = getattr(b, 'parent', None)
-    return a is b
-
-
-def _descendant(a, b):
-    # type: (PredContext, PredContext) -> PredContext
-    """
-    If a is a parent of b or b is a parent of a, return the descendant of the
-    two.
-
-    If neither is a parent of the other, return None.
-    """
-    if _is_parent(a, b):
-        return b
-    if _is_parent(b, a):
-        return a
-    return None
-
-
-class Predicate(object):
-    """
-    Superclass for all computed predicates.
-
-    Leaf predicates can have other types, such as `Setting`.
-
-    :param parts: Tuple of components in the predicate expression.
-    """
-
-    def __init__(self, parts):
-        # type: (Sequence[PredNode]) -> None
-        self.parts = parts
-        self.context = reduce(
-                _descendant,
-                (p.predicate_context() for p in parts))
-        assert self.context, "Incompatible predicate parts"
-        self.predkey = None  # type: PredKey
-
-    def __str__(self):
-        # type: () -> str
-        return '{}({})'.format(type(self).__name__,
-                               ', '.join(map(str, self.parts)))
-
-    def predicate_context(self):
-        # type: () -> PredContext
-        return self.context
-
-    def predicate_leafs(self, leafs):
-        # type: (Set[PredLeaf]) -> None
-        """
-        Collect all leaf predicates into the `leafs` set.
-        """
-        for part in self.parts:
-            part.predicate_leafs(leafs)
-
-    def rust_predicate(self, prec):
-        # type: (int) -> str
-        raise NotImplementedError("rust_predicate is an abstract method")
-
-    def predicate_key(self):
-        # type: () -> PredKey
-        """Tuple uniquely identifying a predicate."""
-        if not self.predkey:
-            p = tuple(p.predicate_key() for p in self.parts)  # type: PredKey
-            self.predkey = (type(self).__name__,) + p
-        return self.predkey
-
-
-class And(Predicate):
-    """
-    Computed predicate that is true if all parts are true.
-    """
-
-    precedence = 2
-
-    def __init__(self, *args):
-        # type: (*PredNode) -> None
-        super(And, self).__init__(args)
-
-    def rust_predicate(self, prec):
-        # type: (int) -> str
-        """
-        Return a Rust expression computing the value of this predicate.
-
-        The surrounding precedence determines whether parentheses are needed:
-
-        0. An `if` statement.
-        1. An `||` expression.
-        2. An `&&` expression.
-        3. A `!` expression.
-        """
-        s = ' && '.join(p.rust_predicate(And.precedence) for p in self.parts)
-        if prec > And.precedence:
-            s = '({})'.format(s)
-        return s
-
-    @staticmethod
-    def combine(*args):
-        # type: (*PredNode) -> PredNode
-        """
-        Combine a sequence of predicates, allowing for `None` members.
-
-        Return a predicate that is true when all non-`None` arguments are true,
-        or `None` if all of the arguments are `None`.
-        """
-        args = tuple(p for p in args if p)
-        if args == ():
-            return None
-        if len(args) == 1:
-            return args[0]
-        # We have multiple predicate args. Combine with `And`.
-        return And(*args)
-
-
-class Or(Predicate):
-    """
-    Computed predicate that is true if any parts are true.
-    """
-
-    precedence = 1
-
-    def __init__(self, *args):
-        # type: (*PredNode) -> None
-        super(Or, self).__init__(args)
-
-    def rust_predicate(self, prec):
-        # type: (int) -> str
-        s = ' || '.join(p.rust_predicate(Or.precedence) for p in self.parts)
-        if prec > Or.precedence:
-            s = '({})'.format(s)
-        return s
-
-
-class Not(Predicate):
-    """
-    Computed predicate that is true if its single part is false.
-    """
-
-    precedence = 3
-
-    def __init__(self, part):
-        # type: (PredNode) -> None
-        super(Not, self).__init__((part,))
-
-    def rust_predicate(self, prec):
-        # type: (int) -> str
-        return '!' + self.parts[0].rust_predicate(Not.precedence)
-
-
-class FieldPredicate(object):
-    """
-    An instruction predicate that performs a test on a single `FormatField`.
-
-    :param field: The `FormatField` to be tested.
-    :param function: Boolean predicate function to call.
-    :param args: Additional arguments for the predicate function.
-    """
-
-    def __init__(self, field, function, args):
-        # type: (FormatField, str, Sequence[Any]) -> None
-        self.field = field
-        self.function = function
-        self.args = args
-
-    def __str__(self):
-        # type: () -> str
-        args = (self.field.rust_name(),) + tuple(map(str, self.args))
-        return '{}({})'.format(self.function, ', '.join(args))
-
-    def predicate_context(self):
-        # type: () -> PredContext
-        """
-        This predicate can be evaluated in the context of an instruction
-        format.
-        """
-        iform = self.field.format  # type: InstructionFormat
-        return iform
-
-    def predicate_key(self):
-        # type: () -> PredKey
-        a = tuple(map(str, self.args))
-        return (self.function, str(self.field)) + a
-
-    def predicate_leafs(self, leafs):
-        # type: (Set[PredLeaf]) -> None
-        leafs.add(self)
-
-    def rust_predicate(self, prec):
-        # type: (int) -> str
-        """
-        Return a string of Rust code that evaluates this predicate.
-        """
-        # Prepend `field` to the predicate function arguments.
-        args = (self.field.rust_name(),) + tuple(map(str, self.args))
-        return 'crate::predicates::{}({})'\
-            .format(self.function, ', '.join(args))
-
-
-class IsEqual(FieldPredicate):
-    """
-    Instruction predicate that checks if an immediate instruction format field
-    is equal to a constant value.
-
-    :param field: `FormatField` to be checked.
-    :param value: The constant value to compare against.
-    """
-
-    def __init__(self, field, value):
-        # type: (FormatField, Any) -> None
-        super(IsEqual, self).__init__(field, 'is_equal', (value,))
-        self.value = value
-
-
-class IsZero32BitFloat(FieldPredicate):
-    """
-    Instruction predicate that checks if an immediate instruction format field
-    is equal to zero.
-
-    :param field: `FormatField` to be checked.
-    :param value: The constant value to check.
-    """
-
-    def __init__(self, field):
-        # type: (FormatField) -> None
-        super(IsZero32BitFloat, self).__init__(field,
-                                               'is_zero_32_bit_float',
-                                               ())
-
-
-class IsZero64BitFloat(FieldPredicate):
-    """
-    Instruction predicate that checks if an immediate instruction format field
-    is equal to zero.
-
-    :param field: `FormatField` to be checked.
-    :param value: The constant value to check.
-    """
-
-    def __init__(self, field):
-        # type: (FormatField) -> None
-        super(IsZero64BitFloat, self).__init__(field,
-                                               'is_zero_64_bit_float',
-                                               ())
-
-
-class IsSignedInt(FieldPredicate):
-    """
-    Instruction predicate that checks if an immediate instruction format field
-    is representable as an n-bit two's complement integer.
-
-    :param field: `FormatField` to be checked.
-    :param width: Number of bits in the allowed range.
-    :param scale: Number of low bits that must be 0.
-
-    The predicate is true if the field is in the range:
-    `-2^(width-1) -- 2^(width-1)-1`
-    and a multiple of `2^scale`.
-    """
-
-    def __init__(self, field, width, scale=0):
-        # type: (FormatField, int, int) -> None
-        super(IsSignedInt, self).__init__(
-                field, 'is_signed_int', (width, scale))
-        self.width = width
-        self.scale = scale
-        assert width >= 0 and width <= 64
-        assert scale >= 0 and scale < width
-
-
-class IsUnsignedInt(FieldPredicate):
-    """
-    Instruction predicate that checks if an immediate instruction format field
-    is representable as an n-bit unsigned complement integer.
-
-    :param field: `FormatField` to be checked.
-    :param width: Number of bits in the allowed range.
-    :param scale: Number of low bits that must be 0.
-
-    The predicate is true if the field is in the range:
-    `0 -- 2^width - 1` and a multiple of `2^scale`.
-    """
-
-    def __init__(self, field, width, scale=0):
-        # type: (FormatField, int, int) -> None
-        super(IsUnsignedInt, self).__init__(
-                field, 'is_unsigned_int', (width, scale))
-        self.width = width
-        self.scale = scale
-        assert width >= 0 and width <= 64
-        assert scale >= 0 and scale < width
-
-
-class TypePredicate(object):
-    """
-    An instruction predicate that checks the type of an SSA argument value.
-
-    Type predicates are used to implement encodings for instructions with
-    multiple type variables. The encoding tables are keyed by the controlling
-    type variable, type predicates check any secondary type variables.
-
-    A type predicate is not bound to any specific instruction format.
-
-    :param value_arg: Index of the value argument to type check.
-    :param value_type: The required value type.
-    """
-
-    def __init__(self, value_arg, value_type):
-        # type: (int, ValueType) -> None
-        assert value_arg >= 0
-        assert value_type is not None
-        self.value_arg = value_arg
-        self.value_type = value_type
-
-    def __str__(self):
-        # type: () -> str
-        return 'args[{}]:{}'.format(self.value_arg, self.value_type)
-
-    def predicate_context(self):
-        # type: () -> PredContext
-        return instruction_context
-
-    def predicate_key(self):
-        # type: () -> PredKey
-        return ('typecheck', self.value_arg, self.value_type.name)
-
-    def predicate_leafs(self, leafs):
-        # type: (Set[PredLeaf]) -> None
-        leafs.add(self)
-
-    @staticmethod
-    def typevar_check(inst, typevar, value_type):
-        # type: (Instruction, TypeVar, ValueType) -> TypePredicate
-        """
-        Return a type check predicate for the given type variable in `inst`.
-
-        The type variable must appear directly as the type of one of the
-        operands to `inst`, so this is only guaranteed to work for secondary
-        type variables.
-
-        Find an `inst` value operand whose type is determined by `typevar` and
-        create a `TypePredicate` that checks that the type variable has the
-        value `value_type`.
-        """
-        # Find the first value operand whose type is `typevar`.
-        value_arg = next(i for i, opnum in enumerate(inst.value_opnums)
-                         if inst.ins[opnum].typevar == typevar)
-        return TypePredicate(value_arg, value_type)
-
-    def rust_predicate(self, prec):
-        # type: (int) -> str
-        """
-        Return Rust code for evaluating this predicate.
-
-        It is assumed that the context has `func` and `args` variables.
-        """
-        return 'func.dfg.value_type(args[{}]) == {}'.format(
-                self.value_arg, self.value_type.rust_name())
-
-
-class CtrlTypePredicate(object):
-    """
-    An instruction predicate that checks the controlling type variable
-
-    :param value_type: The required value type.
-    """
-
-    def __init__(self, value_type):
-        # type: (ValueType) -> None
-        assert value_type is not None
-        self.value_type = value_type
-
-    def __str__(self):
-        # type: () -> str
-        return 'ctrl_typevar:{}'.format(self.value_type)
-
-    def predicate_context(self):
-        # type: () -> PredContext
-        return instruction_context
-
-    def predicate_key(self):
-        # type: () -> PredKey
-        return ('ctrltypecheck', self.value_type.name)
-
-    def predicate_leafs(self, leafs):
-        # type: (Set[PredLeaf]) -> None
-        leafs.add(self)
-
-    def rust_predicate(self, prec):
-        # type: (int) -> str
-        """
-        Return Rust code for evaluating this predicate.
-
-        It is assumed that the context has `func` and `inst` variables.
-        """
-        return 'func.dfg.ctrl_typevar(inst) == {}'.format(
-                self.value_type.rust_name())
diff --git a/cranelift/codegen/meta-python/cdsl/registers.py b/cranelift/codegen/meta-python/cdsl/registers.py
deleted file mode 100644
index 1e4ffe75b1..0000000000
--- a/cranelift/codegen/meta-python/cdsl/registers.py
+++ /dev/null
@@ -1,413 +0,0 @@
-"""
-Register set definitions
-------------------------
-
-Each ISA defines a separate register set that is used by the register allocator
-and the final binary encoding of machine code.
-
-The CPU registers are first divided into disjoint register banks, represented
-by a `RegBank` instance. Registers in different register banks never interfere
-with each other. A typical CPU will have a general purpose and a floating point
-register bank.
-
-A register bank consists of a number of *register units* which are the smallest
-indivisible units of allocation and interference. A register unit doesn't
-necessarily correspond to a particular number of bits in a register, it is more
-like a placeholder that can be used to determine of a register is taken or not.
-
-The register allocator works with *register classes* which can allocate one or
-more register units at a time. A register class allocates more than one
-register unit at a time when its registers are composed of smaller allocatable
-units. For example, the ARM double precision floating point registers are
-composed of two single precision registers.
-"""
-from __future__ import absolute_import
-from . import is_power_of_two, next_power_of_two
-
-
-try:
-    from typing import Sequence, Tuple, List, Dict, Any, Optional, TYPE_CHECKING  # noqa
-    if TYPE_CHECKING:
-        from .isa import TargetISA  # noqa
-        # A tuple uniquely identifying a register class inside a register bank.
-        # (width, bitmask)
-        RCTup = Tuple[int, int]
-except ImportError:
-    pass
-
-
-# The number of 32-bit elements in a register unit mask
-MASK_LEN = 3
-
-# The maximum total number of register units allowed.
-# This limit can be raised by also adjusting the RegUnitMask type in
-# src/isa/registers.rs.
-MAX_UNITS = MASK_LEN * 32
-
-
-class RegBank(object):
-    """
-    A register bank belonging to an ISA.
-
-    A register bank controls a set of *register units* disjoint from all the
-    other register banks in the ISA. The register units are numbered uniquely
-    within the target ISA, and the units in a register bank form a contiguous
-    sequence starting from a sufficiently aligned point that their low bits can
-    be used directly when encoding machine code instructions.
-
-    Register units can be given generated names like `r0`, `r1`, ..., or a
-    tuple of special register unit names can be provided.
-
-    :param name: Name of this register bank.
-    :param doc: Documentation string.
-    :param units: Number of register units.
-    :param pressure_tracking: Enable tracking of register pressure.
-    :param prefix: Prefix for generated unit names.
-    :param names: Special names for the first units. May be shorter than
-                  `units`, the remaining units are named using `prefix`.
-    """
-
-    def __init__(
-            self,
-            name,                       # type: str
-            isa,                        # type: TargetISA
-            doc,                        # type: str
-            units,                      # type: int
-            pressure_tracking=True,     # type: bool
-            prefix='r',                 # type: str
-            names=()                    # type: Sequence[str]
-            ):
-        # type: (...) -> None
-        self.name = name
-        self.isa = isa
-        self.first_unit = 0
-        self.units = units
-        self.pressure_tracking = pressure_tracking
-        self.prefix = prefix
-        self.names = names
-        self.classes = list()  # type: List[RegClass]
-        self.toprcs = list()  # type: List[RegClass]
-
-        assert len(names) <= units
-
-        if isa.regbanks:
-            # Get the next free unit number.
-            last = isa.regbanks[-1]
-            u = last.first_unit + last.units
-            align = units
-            if not is_power_of_two(align):
-                align = next_power_of_two(align)
-            self.first_unit = (u + align - 1) & -align
-
-        self.index = len(isa.regbanks)
-        isa.regbanks.append(self)
-
-    def __repr__(self):
-        # type: () -> str
-        return ('RegBank({}, units={}, first_unit={})'
-                .format(self.name, self.units, self.first_unit))
-
-    def finish_regclasses(self):
-        # type: () -> None
-        """
-        Compute subclasses and the top-level register class.
-
-        Verify that the set of register classes satisfies:
-
-        1. Closed under intersection: The intersection of any two register
-           classes in the set is either empty or identical to a member of the
-           set.
-        2. There are no identical classes under different names.
-        3. Classes are sorted topologically such that all subclasses have a
-           higher index that the superclass.
-
-        We could reorder classes topologically here instead of just enforcing
-        the order, but the ordering tends to fall out naturally anyway.
-        """
-        cmap = dict()  # type: Dict[RCTup, RegClass]
-
-        for rc in self.classes:
-            # All register classes must be given a name.
-            assert rc.name, "Anonymous register class found"
-
-            # Check for duplicates.
-            tup = rc.rctup()
-            if tup in cmap:
-                raise AssertionError(
-                        '{} and {} are identical register classes'
-                        .format(rc, cmap[tup]))
-            cmap[tup] = rc
-
-        # Check intersections and topological order.
-        for idx, rc1 in enumerate(self.classes):
-            rc1.toprc = rc1
-            for rc2 in self.classes[0:idx]:
-                itup = rc1.intersect(rc2)
-                if itup is None:
-                    continue
-                if itup not in cmap:
-                    raise AssertionError(
-                        'intersection of {} and {} missing'
-                        .format(rc1, rc2))
-                irc = cmap[itup]
-                # rc1 > rc2, so rc2 can't be the sub-class.
-                if irc is rc2:
-                    raise AssertionError(
-                            'Bad topological order: {}/{}'
-                            .format(rc1, rc2))
-                if irc is rc1:
-                    # The intersection of rc1 and rc2 is rc1, so it must be a
-                    # sub-class.
-                    rc2.subclasses.append(rc1)
-                    rc1.toprc = rc2.toprc
-
-            if rc1.is_toprc():
-                self.toprcs.append(rc1)
-
-    def unit_by_name(self, name):
-        # type: (str) -> int
-        """
-        Get a register unit in this bank by name.
-        """
-        if name in self.names:
-            r = self.names.index(name)
-        elif name.startswith(self.prefix):
-            r = int(name[len(self.prefix):])
-        assert r < self.units, 'Invalid register name: ' + name
-        return self.first_unit + r
-
-
-class RegClass(object):
-    """
-    A register class is a subset of register units in a RegBank along with a
-    strategy for allocating registers.
-
-    The *width* parameter determines how many register units are allocated at a
-    time. Usually it that is one, but for example the ARM D registers are
-    allocated two units at a time. When multiple units are allocated, it is
-    always a contiguous set of unit numbers.
-
-    :param bank: The register bank we're allocating from.
-    :param count: The maximum number of allocations in this register class. By
-                  default, the whole register bank can be allocated.
-    :param width: How many units to allocate at a time.
-    :param start: The first unit to allocate, relative to `bank.first.unit`.
-    """
-
-    def __init__(self, bank, count=0, width=1, start=0, bitmask=None):
-        # type: (RegBank, int, int, int, Optional[int]) -> None
-        self.name = None  # type: str
-        self.index = None  # type: int
-        self.bank = bank
-        self.width = width
-        self.bitmask = 0
-
-        # This is computed later in `finish_regclasses()`.
-        self.subclasses = list()  # type: List[RegClass]
-        self.toprc = None  # type: RegClass
-
-        assert width > 0
-
-        if bitmask:
-            self.bitmask = bitmask
-        else:
-            assert start >= 0 and start < bank.units
-            if count == 0:
-                count = bank.units // width
-            for a in range(count):
-                u = start + a * self.width
-                self.bitmask |= 1 << u
-
-        bank.classes.append(self)
-
-    def __str__(self):
-        # type: () -> str
-        return self.name
-
-    def is_toprc(self):
-        # type: () -> bool
-        """
-        Is this a top-level register class?
-
-        A top-level register class has no sub-classes. This can only be
-        answered aster running `finish_regclasses()`.
-        """
-        return self.toprc is self
-
-    def rctup(self):
-        # type: () -> RCTup
-        """
-        Get a tuple that uniquely identifies the registers in this class.
-
-        The tuple can be used as a dictionary key to ensure that there are no
-        duplicate register classes.
-        """
-        return (self.width, self.bitmask)
-
-    def intersect(self, other):
-        # type: (RegClass) -> RCTup
-        """
-        Get a tuple representing the intersection of two register classes.
-
-        Returns `None` if the two classes are disjoint.
-        """
-        if self.width != other.width:
-            return None
-        intersection = self.bitmask & other.bitmask
-        if intersection == 0:
-            return None
-
-        return (self.width, intersection)
-
-    def __getitem__(self, sliced):
-        # type: (slice) -> RegClass
-        """
-        Create a sub-class of a register class using slice notation. The slice
-        indexes refer to allocations in the parent register class, not register
-        units.
-        """
-        assert isinstance(sliced, slice), "RegClass slicing can't be 1 reg"
-        # We could add strided sub-classes if needed.
-        assert sliced.step is None, 'Subclass striding not supported'
-        # Can't slice a non-contiguous class
-        assert self.is_contiguous(), 'Cannot slice non-contiguous RegClass'
-
-        w = self.width
-        s = self.start() + sliced.start * w
-        c = sliced.stop - sliced.start
-        assert c > 1, "Can't have single-register classes"
-
-        return RegClass(self.bank, count=c, width=w, start=s)
-
-    def without(self, *registers):
-        # type: (*Register) -> RegClass
-        """
-        Create a sub-class of a register class excluding a specific set of
-        registers.
-
-        For example: GPR.without(GPR.r9)
-        """
-        bm = self.bitmask
-        w = self.width
-        fmask = (1 << self.width) - 1
-        for reg in registers:
-            bm &= ~(fmask << (reg.unit * w))
-
-        return RegClass(self.bank, bitmask=bm)
-
-    def is_contiguous(self):
-        # type: () -> bool
-        """
-        Returns boolean indicating whether a register class is a contiguous set
-        of register units.
-        """
-        x = self.bitmask | (self.bitmask-1)
-        return self.bitmask != 0 and ((x+1) & x) == 0
-
-    def start(self):
-        # type: () -> int
-        """
-        Returns the first valid register unit in this class.
-        """
-        start = 0
-        bm = self.bitmask
-        fmask = (1 << self.width) - 1
-        while True:
-            if bm & fmask > 0:
-                break
-            start += 1
-            bm >>= self.width
-
-        return start
-
-    def __getattr__(self, attr):
-        # type: (str) -> Register
-        """
-        Get a specific register in the class by name.
-
-        For example: `GPR.r5`.
-        """
-        reg = Register(self, self.bank.unit_by_name(attr))
-        # Save this register so we won't have to create it again.
-        setattr(self, attr, reg)
-        return reg
-
-    def mask(self):
-        # type: () -> List[int]
-        """
-        Compute a bit-mask of the register units allocated by this register
-        class.
-
-        Return as a list of 32-bit integers.
-        """
-        out_mask = []
-        mask32 = (1 << 32) - 1
-        bitmask = self.bitmask << self.bank.first_unit
-        for i in range(MASK_LEN):
-            out_mask.append((bitmask >> (i * 32)) & mask32)
-
-        return out_mask
-
-    def subclass_mask(self):
-        # type: () -> int
-        """
-        Compute a bit-mask of subclasses, including self.
-        """
-        m = 1 << self.index
-        for rc in self.subclasses:
-            m |= 1 << rc.index
-        return m
-
-    @staticmethod
-    def extract_names(globs):
-        # type: (Dict[str, Any]) -> None
-        """
-        Given a dict mapping name -> object as returned by `globals()`, find
-        all the RegClass objects and set their name from the dict key.
-        This is used to name a bunch of global values in a module.
-        """
-        for name, obj in globs.items():
-            if isinstance(obj, RegClass):
-                assert obj.name is None
-                obj.name = name
-
-
-class Register(object):
-    """
-    A specific register in a register class.
-
-    A register is identified by the top-level register class it belongs to and
-    its first register unit.
-
-    Specific registers are used to describe constraints on instructions where
-    some operands must use a fixed register.
-
-    Register instances can be created with the constructor, or accessed as
-    attributes on the register class: `GPR.rcx`.
-    """
-    def __init__(self, rc, unit):
-        # type: (RegClass, int) -> None
-        self.regclass = rc
-        self.unit = unit
-
-
-class Stack(object):
-    """
-    An operand that must be in a stack slot.
-
-    A `Stack` object can be used to indicate an operand constraint for a value
-    operand that must live in a stack slot.
-    """
-    def __init__(self, rc):
-        # type: (RegClass) -> None
-        self.regclass = rc
-
-    def stack_base_mask(self):
-        # type: () -> str
-        """
-        Get the StackBaseMask to use for this operand.
-
-        This is a mask of base registers that can be supported by this operand.
-        """
-        # TODO: Make this configurable instead of just using the SP.
-        return 'StackBaseMask(1)'
diff --git a/cranelift/codegen/meta-python/cdsl/settings.py b/cranelift/codegen/meta-python/cdsl/settings.py
deleted file mode 100644
index fe79024221..0000000000
--- a/cranelift/codegen/meta-python/cdsl/settings.py
+++ /dev/null
@@ -1,416 +0,0 @@
-"""Classes for describing settings and groups of settings."""
-from __future__ import absolute_import
-from collections import OrderedDict
-from .predicates import Predicate
-
-try:
-    from typing import Tuple, Set, List, Dict, Any, Union, TYPE_CHECKING  # noqa
-    BoolOrPresetOrDict = Union['BoolSetting', 'Preset', Dict['Setting', Any]]
-    if TYPE_CHECKING:
-        from .predicates import PredLeaf, PredNode, PredKey  # noqa
-except ImportError:
-    pass
-
-
-class Setting(object):
-    """
-    A named setting variable that can be configured externally to Cranelift.
-
-    Settings are normally not named when they are created. They get their name
-    from the `extract_names` method.
-    """
-
-    def __init__(self, doc):
-        # type: (str) -> None
-        self.name = None  # type: str  # Assigned later by `extract_names()`.
-        self.__doc__ = doc
-        # Offset of byte in settings vector containing this setting.
-        self.byte_offset = None  # type: int
-        # Index into the generated DESCRIPTORS table.
-        self.descriptor_index = None  # type: int
-
-        self.group = SettingGroup.append(self)
-
-    def __str__(self):
-        # type: () -> str
-        return '{}.{}'.format(self.group.name, self.name)
-
-    def default_byte(self):
-        # type: () -> int
-        raise NotImplementedError("default_byte is an abstract method")
-
-    def byte_for_value(self, value):
-        # type: (Any) -> int
-        """Get the setting byte value that corresponds to `value`"""
-        raise NotImplementedError("byte_for_value is an abstract method")
-
-    def byte_mask(self):
-        # type: () -> int
-        """Get a mask of bits in our byte that are relevant to this setting."""
-        # Only BoolSetting has a different mask.
-        return 0xff
-
-
-class BoolSetting(Setting):
-    """
-    A named setting with a boolean on/off value.
-
-    :param doc: Documentation string.
-    :param default: The default value of this setting.
-    """
-
-    def __init__(self, doc, default=False):
-        # type: (str, bool) -> None
-        super(BoolSetting, self).__init__(doc)
-        self.default = default
-        self.bit_offset = None  # type: int
-
-    def default_byte(self):
-        # type: () -> int
-        """
-        Get the default value of this setting, as a byte that can be bitwise
-        or'ed with the other booleans sharing the same byte.
-        """
-        if self.default:
-            return 1 << self.bit_offset
-        else:
-            return 0
-
-    def byte_for_value(self, value):
-        # type: (Any) -> int
-        if value:
-            return 1 << self.bit_offset
-        else:
-            return 0
-
-    def byte_mask(self):
-        # type: () -> int
-        return 1 << self.bit_offset
-
-    def predicate_context(self):
-        # type: () -> SettingGroup
-        """
-        Return the context where this setting can be evaluated as a (leaf)
-        predicate.
-        """
-        return self.group
-
-    def predicate_key(self):
-        # type: () -> PredKey
-        assert self.name, "Can't compute key before setting is named"
-        return ('setting', self.group.name, self.name)
-
-    def predicate_leafs(self, leafs):
-        # type: (Set[PredLeaf]) -> None
-        leafs.add(self)
-
-    def rust_predicate(self, prec):
-        # type: (int) -> str
-        """
-        Return the Rust code to compute the value of this setting.
-
-        The emitted code assumes that the setting group exists as a local
-        variable.
-        """
-        return '{}.{}()'.format(self.group.name, self.name)
-
-
-class NumSetting(Setting):
-    """
-    A named setting with an integral value in the range 0--255.
-
-    :param doc: Documentation string.
-    :param default: The default value of this setting.
-    """
-
-    def __init__(self, doc, default=0):
-        # type: (str, int) -> None
-        super(NumSetting, self).__init__(doc)
-        assert default == int(default)
-        assert default >= 0 and default <= 255
-        self.default = default
-
-    def default_byte(self):
-        # type: () -> int
-        return self.default
-
-    def byte_for_value(self, value):
-        # type: (Any) -> int
-        assert isinstance(value, int), "NumSetting must be set to an int"
-        assert value >= 0 and value <= 255
-        return value
-
-
-class EnumSetting(Setting):
-    """
-    A named setting with an enumerated set of possible values.
-
-    The default value is always the first enumerator.
-
-    :param doc: Documentation string.
-    :param args: Tuple of unique strings representing the possible values.
-    """
-
-    def __init__(self, doc, *args):
-        # type: (str, *str) -> None
-        super(EnumSetting, self).__init__(doc)
-        assert len(args) > 0, "EnumSetting must have at least one value"
-        self.values = tuple(str(x) for x in args)
-        self.default = self.values[0]
-
-    def default_byte(self):
-        # type: () -> int
-        return 0
-
-    def byte_for_value(self, value):
-        # type: (Any) -> int
-        return self.values.index(value)
-
-
-class SettingGroup(object):
-    """
-    A group of settings.
-
-    Whenever a :class:`Setting` object is created, it is added to the currently
-    open group. A setting group must be closed explicitly before another can be
-    opened.
-
-    :param name: Short mnemonic name for setting group.
-    :param parent: Parent settings group.
-    """
-
-    # The currently open setting group.
-    _current = None  # type: SettingGroup
-
-    def __init__(self, name, parent=None):
-        # type: (str, SettingGroup) -> None
-        self.name = name
-        self.parent = parent
-        self.settings = []  # type: List[Setting]
-        # Named predicates computed from settings in this group or its
-        # parents.
-        self.named_predicates = OrderedDict()  # type: OrderedDict[str, Predicate]  # noqa
-        # All boolean predicates that can be accessed by number. This includes:
-        # - All boolean settings in this group.
-        # - All named predicates.
-        # - Added anonymous predicates, see `number_predicate()`.
-        # - Added parent predicates that are replicated in this group.
-        # Maps predicate -> number.
-        self.predicate_number = OrderedDict()  # type: OrderedDict[PredNode, int]  # noqa
-        self.presets = []  # type: List[Preset]
-
-        # Fully qualified Rust module name. See gen_settings.py.
-        self.qual_mod = None  # type: str
-
-        self.open()
-
-    def open(self):
-        # type: () -> None
-        """
-        Open this setting group such that future new settings are added to this
-        group.
-        """
-        assert SettingGroup._current is None, (
-                "Can't open {} since {} is already open"
-                .format(self, SettingGroup._current))
-        SettingGroup._current = self
-
-    def close(self, globs=None):
-        # type: (Dict[str, Any]) -> None
-        """
-        Close this setting group. This function must be called before opening
-        another setting group.
-
-        :param globs: Pass in `globals()` to run `extract_names` on all
-            settings defined in the module.
-        """
-        assert SettingGroup._current is self, (
-                "Can't close {}, the open setting group is {}"
-                .format(self, SettingGroup._current))
-        SettingGroup._current = None
-        if globs:
-            # Ensure that named predicates are ordered in a deterministic way
-            # that the Rust crate may simply reproduce, by pushing entries into
-            # a vector that we'll sort by name later.
-            named_predicates = []
-
-            for name, obj in globs.items():
-                if isinstance(obj, Setting):
-                    assert obj.name is None, obj.name
-                    obj.name = name
-                if isinstance(obj, Predicate):
-                    named_predicates.append((name, obj))
-                if isinstance(obj, Preset):
-                    assert obj.name is None, obj.name
-                    obj.name = name
-
-            named_predicates.sort(key=lambda x: x[0])
-            for (name, obj) in named_predicates:
-                self.named_predicates[name] = obj
-
-        self.layout()
-
-    @staticmethod
-    def append(setting):
-        # type: (Setting) -> SettingGroup
-        g = SettingGroup._current
-        assert g, "Open a setting group before defining settings."
-        g.settings.append(setting)
-        return g
-
-    @staticmethod
-    def append_preset(preset):
-        # type: (Preset) -> SettingGroup
-        g = SettingGroup._current
-        assert g, "Open a setting group before defining presets."
-        g.presets.append(preset)
-        return g
-
-    def number_predicate(self, pred):
-        # type: (PredNode) -> int
-        """
-        Make sure that `pred` has an assigned number, and will be included in
-        this group's bit vector.
-
-        The numbered predicates include:
-        - `BoolSetting` settings that belong to this group.
-        - `Predicate` instances in `named_predicates`.
-        - `Predicate` instances without a name.
-        - Settings or computed predicates that belong to the parent group, but
-          need to be accessible by number in this group.
-
-        The numbered predicates are referenced by the encoding tables as ISA
-        predicates. See the `isap` field on `Encoding`.
-
-        :returns: The assigned predicate number in this group.
-        """
-        if pred in self.predicate_number:
-            return self.predicate_number[pred]
-        else:
-            number = len(self.predicate_number)
-            self.predicate_number[pred] = number
-            return number
-
-    def layout(self):
-        # type: () -> None
-        """
-        Compute the layout of the byte vector used to represent this settings
-        group.
-
-        The byte vector contains the following entries in order:
-
-        1. Byte-sized settings like `NumSetting` and `EnumSetting`.
-        2. `BoolSetting` settings.
-        3. Precomputed named predicates.
-        4. Other numbered predicates, including anonymous predicates and parent
-           predicates that need to be accessible by number.
-
-        Set `self.settings_size` to the length of the byte vector prefix that
-        contains the settings. All bytes after that are computed, not
-        configured.
-
-        Set `self.boolean_offset` to the beginning of the numbered predicates,
-        2. in the list above.
-
-        Assign `byte_offset` and `bit_offset` fields in all settings.
-
-        After calling this method, no more settings can be added, but
-        additional predicates can be made accessible with `number_predicate()`.
-        """
-        assert len(self.predicate_number) == 0, "Too late for layout"
-
-        # Assign the non-boolean settings.
-        byte_offset = 0
-        for s in self.settings:
-            if not isinstance(s, BoolSetting):
-                s.byte_offset = byte_offset
-                byte_offset += 1
-
-        # Then the boolean settings.
-        self.boolean_offset = byte_offset
-        for s in self.settings:
-            if isinstance(s, BoolSetting):
-                number = self.number_predicate(s)
-                s.byte_offset = byte_offset + number // 8
-                s.bit_offset = number % 8
-
-        # This is the end of the settings. Round up to a whole number of bytes.
-        self.boolean_settings = len(self.predicate_number)
-        self.settings_size = self.byte_size()
-
-        # Now assign numbers to all our named predicates.
-        for name, pred in self.named_predicates.items():
-            self.number_predicate(pred)
-
-    def byte_size(self):
-        # type: () -> int
-        """
-        Compute the number of bytes required to hold all settings and
-        precomputed predicates.
-
-        This is the size of the byte-sized settings plus all the numbered
-        predicate bits rounded up to a whole number of bytes.
-        """
-        return self.boolean_offset + (len(self.predicate_number) + 7) // 8
-
-
-class Preset(object):
-    """
-    A collection of setting values that are applied at once.
-
-    A `Preset` represents a shorthand notation for applying a number of
-    settings at once. Example:
-
-        nehalem = Preset(has_sse41, has_cmov, has_avx=0)
-
-    Enabling the `nehalem` setting is equivalent to enabling `has_sse41` and
-    `has_cmov` while disabling the `has_avx` setting.
-    """
-
-    def __init__(self, *args):
-        # type: (*BoolOrPresetOrDict) -> None
-        self.name = None  # type: str  # Assigned later by `SettingGroup`.
-        # Each tuple provides the value for a setting.
-        self.values = list()  # type: List[Tuple[Setting, Any]]
-
-        for arg in args:
-            if isinstance(arg, Preset):
-                # Any presets in args are immediately expanded.
-                self.values.extend(arg.values)
-            elif isinstance(arg, dict):
-                # A dictionary of key: value pairs.
-                self.values.extend(arg.items())
-            else:
-                # A BoolSetting to enable.
-                assert isinstance(arg, BoolSetting)
-                self.values.append((arg, True))
-
-        self.group = SettingGroup.append_preset(self)
-        # Index into the generated DESCRIPTORS table.
-        self.descriptor_index = None  # type: int
-
-    def layout(self):
-        # type: () -> List[Tuple[int, int]]
-        """
-        Compute a list of (mask, byte) pairs that incorporate all values in
-        this preset.
-
-        The list will have an entry for each setting byte in the settings
-        group.
-        """
-        lst = [(0, 0)] * self.group.settings_size
-
-        # Apply setting values in order.
-        for s, v in self.values:
-            ofs = s.byte_offset
-            s_mask = s.byte_mask()
-            s_val = s.byte_for_value(v)
-            assert (s_val & ~s_mask) == 0
-            l_mask, l_val = lst[ofs]
-            # Accumulated mask of modified bits.
-            l_mask |= s_mask
-            # Overwrite the relevant bits with the new value.
-            l_val = (l_val & ~s_mask) | s_val
-            lst[ofs] = (l_mask, l_val)
-
-        return lst
diff --git a/cranelift/codegen/meta-python/cdsl/test_ast.py b/cranelift/codegen/meta-python/cdsl/test_ast.py
deleted file mode 100644
index 750142af0a..0000000000
--- a/cranelift/codegen/meta-python/cdsl/test_ast.py
+++ /dev/null
@@ -1,28 +0,0 @@
-from __future__ import absolute_import
-from unittest import TestCase
-from doctest import DocTestSuite
-from . import ast
-from base.instructions import jump, iadd
-
-
-def load_tests(loader, tests, ignore):
-    tests.addTests(DocTestSuite(ast))
-    return tests
-
-
-x = 'x'
-y = 'y'
-a = 'a'
-
-
-class TestPatterns(TestCase):
-    def test_apply(self):
-        i = jump(x, y)
-        self.assertEqual(repr(i), "Apply(jump, ('x', 'y'))")
-
-        i = iadd.i32(x, y)
-        self.assertEqual(repr(i), "Apply(iadd.i32, ('x', 'y'))")
-
-    def test_single_ins(self):
-        pat = a << iadd.i32(x, y)
-        self.assertEqual(repr(pat), "('a',) << Apply(iadd.i32, ('x', 'y'))")
diff --git a/cranelift/codegen/meta-python/cdsl/test_package.py b/cranelift/codegen/meta-python/cdsl/test_package.py
deleted file mode 100644
index b66d60d694..0000000000
--- a/cranelift/codegen/meta-python/cdsl/test_package.py
+++ /dev/null
@@ -1,8 +0,0 @@
-from __future__ import absolute_import
-import doctest
-import cdsl
-
-
-def load_tests(loader, tests, ignore):
-    tests.addTests(doctest.DocTestSuite(cdsl))
-    return tests
diff --git a/cranelift/codegen/meta-python/cdsl/test_ti.py b/cranelift/codegen/meta-python/cdsl/test_ti.py
deleted file mode 100644
index b88113a20f..0000000000
--- a/cranelift/codegen/meta-python/cdsl/test_ti.py
+++ /dev/null
@@ -1,605 +0,0 @@
-from __future__ import absolute_import
-from base.instructions import vselect, vsplit, vconcat, iconst, iadd, bint,\
-    b1, icmp, iadd_cout, iadd_cin, uextend, sextend, ireduce, fpromote, \
-    fdemote
-from base.legalize import narrow, expand
-from base.immediates import intcc
-from base.types import i32, i8
-from .typevar import TypeVar
-from .ast import Var, Def
-from .xform import Rtl, XForm
-from .ti import ti_rtl, subst, TypeEnv, get_type_env, TypesEqual, WiderOrEq
-from unittest import TestCase
-from functools import reduce
-
-try:
-    from .ti import TypeMap, ConstraintList, VarTyping, TypingOrError # noqa
-    from typing import List, Dict, Tuple, TYPE_CHECKING, cast # noqa
-except ImportError:
-    TYPE_CHECKING = False
-
-
-def agree(me, other):
-    # type: (TypeEnv, TypeEnv) -> bool
-    """
-    Given TypeEnvs me and other, check if they agree. As part of that build
-    a map m from TVs in me to their corresponding TVs in other.
-    Specifically:
-
-        1. Check that all TVs that are keys in me.type_map are also defined
-           in other.type_map
-
-        2. For any tv in me.type_map check that:
-            me[tv].get_typeset() == other[tv].get_typeset()
-
-        3. Set m[me[tv]] = other[tv] in the substitution m
-
-        4. If we find another tv1 such that me[tv1] == me[tv], assert that
-           other[tv1] == m[me[tv1]] == m[me[tv]] = other[tv]
-
-        5. Check that me and other have the same constraints under the
-           substitution m
-    """
-    m = {}  # type: TypeMap
-    # Check that our type map and other's agree and built substitution m
-    for tv in me.type_map:
-        if (me[tv] not in m):
-            m[me[tv]] = other[tv]
-            if me[tv].get_typeset() != other[tv].get_typeset():
-                return False
-        else:
-            if m[me[tv]] != other[tv]:
-                return False
-
-    # Translate our constraints using m, and sort
-    me_equiv_constr = sorted([constr.translate(m)
-                              for constr in me.constraints], key=repr)
-    # Sort other's constraints
-    other_equiv_constr = sorted([constr.translate(other)
-                                 for constr in other.constraints], key=repr)
-    return me_equiv_constr == other_equiv_constr
-
-
-def check_typing(got_or_err, expected, symtab=None):
-    # type: (TypingOrError, Tuple[VarTyping, ConstraintList], Dict[str, Var]) -> None # noqa
-    """
-    Check that a the typing we received (got_or_err) complies with the
-    expected typing (expected). If symtab is specified, substitute the Vars in
-    expected using symtab first (used when checking type inference on XForms)
-    """
-    (m, c) = expected
-    got = get_type_env(got_or_err)
-
-    if (symtab is not None):
-        # For xforms we first need to re-write our TVs in terms of the tvs
-        # stored internally in the XForm. Use the symtab passed
-        subst_m = {k.get_typevar(): symtab[str(k)].get_typevar()
-                   for k in m.keys()}
-        # Convert m from a Var->TypeVar map to TypeVar->TypeVar map where
-        # the key TypeVar is re-written to its XForm internal version
-        tv_m = {subst(k.get_typevar(), subst_m): v for (k, v) in m.items()}
-        # Rewrite the TVs in the input constraints to their XForm internal
-        # versions
-        c = [constr.translate(subst_m) for constr in c]
-    else:
-        # If no symtab, just convert m from Var->TypeVar map to a
-        # TypeVar->TypeVar map
-        tv_m = {k.get_typevar(): v for (k, v) in m.items()}
-
-    expected_typ = TypeEnv((tv_m, c))
-    assert agree(expected_typ, got), \
-        "typings disagree:\n {} \n {}".format(got.dot(),
-                                              expected_typ.dot())
-
-
-def check_concrete_typing_rtl(var_types, rtl):
-    # type: (VarTyping, Rtl) -> None
-    """
-    Check that a concrete type assignment var_types (Dict[Var, TypeVar]) is
-    valid for an Rtl rtl. Specifically check that:
-
-    1) For each Var v \\in rtl, v is defined in var_types
-
-    2) For all v, var_types[v] is a singleton type
-
-    3) For each v, and each location u, where v is used with expected type
-       tv_u, var_types[v].get_typeset() is a subset of
-       subst(tv_u, m).get_typeset() where m is the substitution of
-       formals->actuals we are building so far.
-
-    4) If tv_u is non-derived and not in m, set m[tv_u]= var_types[v]
-    """
-    for d in rtl.rtl:
-        assert isinstance(d, Def)
-        inst = d.expr.inst
-        # Accumulate all actual TVs for value defs/opnums in actual_tvs
-        actual_tvs = [var_types[d.defs[i]] for i in inst.value_results]
-        for v in [d.expr.args[i] for i in inst.value_opnums]:
-            assert isinstance(v, Var)
-            actual_tvs.append(var_types[v])
-
-        # Accumulate all formal TVs for value defs/opnums in actual_tvs
-        formal_tvs = [inst.outs[i].typevar for i in inst.value_results] +\
-                     [inst.ins[i].typevar for i in inst.value_opnums]
-        m = {}  # type: TypeMap
-
-        # For each actual/formal pair check that they agree
-        for (actual_tv, formal_tv) in zip(actual_tvs, formal_tvs):
-            # actual should be a singleton
-            assert actual_tv.singleton_type() is not None
-            formal_tv = subst(formal_tv, m)
-            # actual should agree with the concretized formal
-            assert actual_tv.get_typeset().issubset(formal_tv.get_typeset())
-
-            if formal_tv not in m and not formal_tv.is_derived:
-                m[formal_tv] = actual_tv
-
-
-def check_concrete_typing_xform(var_types, xform):
-    # type: (VarTyping, XForm) -> None
-    """
-    Check a concrete type assignment var_types for an XForm xform
-    """
-    check_concrete_typing_rtl(var_types, xform.src)
-    check_concrete_typing_rtl(var_types, xform.dst)
-
-
-class TypeCheckingBaseTest(TestCase):
-    def setUp(self):
-        # type: () -> None
-        self.v0 = Var("v0")
-        self.v1 = Var("v1")
-        self.v2 = Var("v2")
-        self.v3 = Var("v3")
-        self.v4 = Var("v4")
-        self.v5 = Var("v5")
-        self.v6 = Var("v6")
-        self.v7 = Var("v7")
-        self.v8 = Var("v8")
-        self.v9 = Var("v9")
-        self.imm0 = Var("imm0")
-        self.IxN_nonscalar = TypeVar("IxN", "", ints=True, scalars=False,
-                                     simd=True)
-        self.TxN = TypeVar("TxN", "", ints=True, bools=True, floats=True,
-                           scalars=False, simd=True)
-        self.b1 = TypeVar.singleton(b1)
-
-
-class TestRTL(TypeCheckingBaseTest):
-    def test_bad_rtl1(self):
-        # type: () -> None
-        r = Rtl(
-                (self.v0, self.v1) << vsplit(self.v2),
-                self.v3 << vconcat(self.v0, self.v2),
-        )
-        ti = TypeEnv()
-        self.assertEqual(ti_rtl(r, ti),
-                         "On line 1: fail ti on `typeof_v2` <: `1`: " +
-                         "Error: empty type created when unifying " +
-                         "`typeof_v2` and `half_vector(typeof_v2)`")
-
-    def test_vselect(self):
-        # type: () -> None
-        r = Rtl(
-                self.v0 << vselect(self.v1, self.v2, self.v3),
-        )
-        ti = TypeEnv()
-        typing = ti_rtl(r, ti)
-        txn = self.TxN.get_fresh_copy("TxN1")
-        check_typing(typing, ({
-            self.v0: txn,
-            self.v1: txn.as_bool(),
-            self.v2: txn,
-            self.v3: txn
-        }, []))
-
-    def test_vselect_icmpimm(self):
-        # type: () -> None
-        r = Rtl(
-                self.v0 << iconst(self.imm0),
-                self.v1 << icmp(intcc.eq, self.v2, self.v0),
-                self.v5 << vselect(self.v1, self.v3, self.v4),
-        )
-        ti = TypeEnv()
-        typing = ti_rtl(r, ti)
-        ixn = self.IxN_nonscalar.get_fresh_copy("IxN1")
-        txn = self.TxN.get_fresh_copy("TxN1")
-        check_typing(typing, ({
-            self.v0: ixn,
-            self.v1: ixn.as_bool(),
-            self.v2: ixn,
-            self.v3: txn,
-            self.v4: txn,
-            self.v5: txn,
-        }, [TypesEqual(ixn.as_bool(), txn.as_bool())]))
-
-    def test_vselect_vsplits(self):
-        # type: () -> None
-        r = Rtl(
-                self.v3 << vselect(self.v0, self.v1, self.v2),
-                (self.v4, self.v5) << vsplit(self.v3),
-                (self.v6, self.v7) << vsplit(self.v4),
-        )
-        ti = TypeEnv()
-        typing = ti_rtl(r, ti)
-        t = TypeVar("t", "", ints=True, bools=True, floats=True,
-                    simd=(4, 256))
-        check_typing(typing, ({
-            self.v0: t.as_bool(),
-            self.v1: t,
-            self.v2: t,
-            self.v3: t,
-            self.v4: t.half_vector(),
-            self.v5: t.half_vector(),
-            self.v6: t.half_vector().half_vector(),
-            self.v7: t.half_vector().half_vector(),
-        }, []))
-
-    def test_vselect_vconcats(self):
-        # type: () -> None
-        r = Rtl(
-                self.v3 << vselect(self.v0, self.v1, self.v2),
-                self.v8 << vconcat(self.v3, self.v3),
-                self.v9 << vconcat(self.v8, self.v8),
-        )
-        ti = TypeEnv()
-        typing = ti_rtl(r, ti)
-        t = TypeVar("t", "", ints=True, bools=True, floats=True,
-                    simd=(2, 64))
-        check_typing(typing, ({
-            self.v0: t.as_bool(),
-            self.v1: t,
-            self.v2: t,
-            self.v3: t,
-            self.v8: t.double_vector(),
-            self.v9: t.double_vector().double_vector(),
-        }, []))
-
-    def test_vselect_vsplits_vconcats(self):
-        # type: () -> None
-        r = Rtl(
-                self.v3 << vselect(self.v0, self.v1, self.v2),
-                (self.v4, self.v5) << vsplit(self.v3),
-                (self.v6, self.v7) << vsplit(self.v4),
-                self.v8 << vconcat(self.v3, self.v3),
-                self.v9 << vconcat(self.v8, self.v8),
-        )
-        ti = TypeEnv()
-        typing = ti_rtl(r, ti)
-        t = TypeVar("t", "", ints=True, bools=True, floats=True,
-                    simd=(4, 64))
-        check_typing(typing, ({
-            self.v0: t.as_bool(),
-            self.v1: t,
-            self.v2: t,
-            self.v3: t,
-            self.v4: t.half_vector(),
-            self.v5: t.half_vector(),
-            self.v6: t.half_vector().half_vector(),
-            self.v7: t.half_vector().half_vector(),
-            self.v8: t.double_vector(),
-            self.v9: t.double_vector().double_vector(),
-        }, []))
-
-    def test_bint(self):
-        # type: () -> None
-        r = Rtl(
-            self.v4 << iadd(self.v1, self.v2),
-            self.v5 << bint(self.v3),
-            self.v0 << iadd(self.v4, self.v5)
-        )
-        ti = TypeEnv()
-        typing = ti_rtl(r, ti)
-        itype = TypeVar("t", "", ints=True, simd=(1, 256))
-        btype = TypeVar("b", "", bools=True, simd=True)
-
-        # Check that self.v5 gets the same integer type as
-        # the rest of them
-        # TODO: Add constraint nlanes(v3) == nlanes(v1) when we
-        # add that type constraint to bint
-        check_typing(typing, ({
-            self.v1:    itype,
-            self.v2:    itype,
-            self.v4:    itype,
-            self.v5:    itype,
-            self.v3:    btype,
-            self.v0:    itype,
-        }, []))
-
-    def test_fully_bound_inst_inference_bad(self):
-        # Incompatible bound instructions fail accordingly
-        r = Rtl(
-                self.v3 << uextend.i32(self.v1),
-                self.v4 << uextend.i16(self.v2),
-                self.v5 << iadd(self.v3, self.v4),
-            )
-        ti = TypeEnv()
-        typing = ti_rtl(r, ti)
-
-        self.assertEqual(typing,
-                         "On line 2: fail ti on `typeof_v4` <: `4`: " +
-                         "Error: empty type created when unifying " +
-                         "`i16` and `i32`")
-
-    def test_extend_reduce(self):
-        # type: () -> None
-        r = Rtl(
-            self.v1 << uextend(self.v0),
-            self.v2 << ireduce(self.v1),
-            self.v3 << sextend(self.v2),
-        )
-        ti = TypeEnv()
-        typing = ti_rtl(r, ti)
-        typing = typing.extract()
-
-        itype0 = TypeVar("t", "", ints=True, simd=(1, 256))
-        itype1 = TypeVar("t1", "", ints=True, simd=(1, 256))
-        itype2 = TypeVar("t2", "", ints=True, simd=(1, 256))
-        itype3 = TypeVar("t3", "", ints=True, simd=(1, 256))
-
-        check_typing(typing, ({
-            self.v0:    itype0,
-            self.v1:    itype1,
-            self.v2:    itype2,
-            self.v3:    itype3,
-        }, [WiderOrEq(itype1, itype0),
-            WiderOrEq(itype1, itype2),
-            WiderOrEq(itype3, itype2)]))
-
-    def test_extend_reduce_enumeration(self):
-        # type: () -> None
-        for op in (uextend, sextend, ireduce):
-            r = Rtl(
-                self.v1 << op(self.v0),
-            )
-            ti = TypeEnv()
-            typing = ti_rtl(r, ti).extract()
-
-            # The number of possible typings is 9 * (3+ 2*2 + 3) = 90
-            lst = [(t[self.v0], t[self.v1]) for t in typing.concrete_typings()]
-            assert (len(lst) == len(set(lst)) and len(lst) == 90)
-            for (tv0, tv1) in lst:
-                typ0, typ1 = (tv0.singleton_type(), tv1.singleton_type())
-                if (op == ireduce):
-                    assert typ0.wider_or_equal(typ1)
-                else:
-                    assert typ1.wider_or_equal(typ0)
-
-    def test_fpromote_fdemote(self):
-        # type: () -> None
-        r = Rtl(
-            self.v1 << fpromote(self.v0),
-            self.v2 << fdemote(self.v1),
-        )
-        ti = TypeEnv()
-        typing = ti_rtl(r, ti)
-        typing = typing.extract()
-
-        ftype0 = TypeVar("t", "", floats=True, simd=(1, 256))
-        ftype1 = TypeVar("t1", "", floats=True, simd=(1, 256))
-        ftype2 = TypeVar("t2", "", floats=True, simd=(1, 256))
-
-        check_typing(typing, ({
-            self.v0:    ftype0,
-            self.v1:    ftype1,
-            self.v2:    ftype2,
-        }, [WiderOrEq(ftype1, ftype0),
-            WiderOrEq(ftype1, ftype2)]))
-
-    def test_fpromote_fdemote_enumeration(self):
-        # type: () -> None
-        for op in (fpromote, fdemote):
-            r = Rtl(
-                self.v1 << op(self.v0),
-            )
-            ti = TypeEnv()
-            typing = ti_rtl(r, ti).extract()
-
-            # The number of possible typings is 9*(2 + 1) = 27
-            lst = [(t[self.v0], t[self.v1]) for t in typing.concrete_typings()]
-            assert (len(lst) == len(set(lst)) and len(lst) == 27)
-            for (tv0, tv1) in lst:
-                (typ0, typ1) = (tv0.singleton_type(), tv1.singleton_type())
-                if (op == fdemote):
-                    assert typ0.wider_or_equal(typ1)
-                else:
-                    assert typ1.wider_or_equal(typ0)
-
-
-class TestXForm(TypeCheckingBaseTest):
-    def test_iadd_cout(self):
-        # type: () -> None
-        x = XForm(Rtl((self.v0, self.v1) << iadd_cout(self.v2, self.v3),),
-                  Rtl(
-                      self.v0 << iadd(self.v2, self.v3),
-                      self.v1 << icmp(intcc.ult, self.v0, self.v2)
-                  ))
-        itype = TypeVar("t", "", ints=True, simd=(1, 1))
-
-        check_typing(x.ti, ({
-            self.v0:    itype,
-            self.v2:    itype,
-            self.v3:    itype,
-            self.v1:    itype.as_bool(),
-        }, []), x.symtab)
-
-    def test_iadd_cin(self):
-        # type: () -> None
-        x = XForm(Rtl(self.v0 << iadd_cin(self.v1, self.v2, self.v3)),
-                  Rtl(
-                      self.v4 << iadd(self.v1, self.v2),
-                      self.v5 << bint(self.v3),
-                      self.v0 << iadd(self.v4, self.v5)
-                  ))
-        itype = TypeVar("t", "", ints=True, simd=(1, 1))
-
-        check_typing(x.ti, ({
-            self.v0:    itype,
-            self.v1:    itype,
-            self.v2:    itype,
-            self.v3:    self.b1,
-            self.v4:    itype,
-            self.v5:    itype,
-        }, []), x.symtab)
-
-    def test_enumeration_with_constraints(self):
-        # type: () -> None
-        xform = XForm(
-            Rtl(
-                self.v0 << iconst(self.imm0),
-                self.v1 << icmp(intcc.eq, self.v2, self.v0),
-                self.v5 << vselect(self.v1, self.v3, self.v4)
-            ),
-            Rtl(
-                self.v0 << iconst(self.imm0),
-                self.v1 << icmp(intcc.eq, self.v2, self.v0),
-                self.v5 << vselect(self.v1, self.v3, self.v4)
-            ))
-
-        # Check all var assigns are correct
-        assert len(xform.ti.constraints) > 0
-        concrete_var_assigns = list(xform.ti.concrete_typings())
-
-        v0 = xform.symtab[str(self.v0)]
-        v1 = xform.symtab[str(self.v1)]
-        v2 = xform.symtab[str(self.v2)]
-        v3 = xform.symtab[str(self.v3)]
-        v4 = xform.symtab[str(self.v4)]
-        v5 = xform.symtab[str(self.v5)]
-
-        for var_m in concrete_var_assigns:
-            assert var_m[v0] == var_m[v2] and \
-                   var_m[v3] == var_m[v4] and\
-                   var_m[v5] == var_m[v3] and\
-                   var_m[v1] == var_m[v2].as_bool() and\
-                   var_m[v1].get_typeset() == var_m[v3].as_bool().get_typeset()
-            check_concrete_typing_xform(var_m, xform)
-
-        # The number of possible typings here is:
-        # 8 cases for v0 = i8xN times 2 options for v3 - i8, b8 = 16
-        # 8 cases for v0 = i16xN times 2 options for v3 - i16, b16 = 16
-        # 8 cases for v0 = i32xN times 3 options for v3 - i32, b32, f32 = 24
-        # 8 cases for v0 = i64xN times 3 options for v3 - i64, b64, f64 = 24
-        #
-        # (Note we have 8 cases for lanes since vselect prevents scalars)
-        # Total: 2*16 + 2*24 = 80
-        assert len(concrete_var_assigns) == 80
-
-    def test_base_legalizations_enumeration(self):
-        # type: () -> None
-        for xform in narrow.xforms + expand.xforms:
-            # Any legalization patterns we defined should have at least 1
-            # concrete typing
-            concrete_typings_list = list(xform.ti.concrete_typings())
-            assert len(concrete_typings_list) > 0
-
-            # If there are no free_typevars, this is a non-polymorphic pattern.
-            # There should be only one possible concrete typing.
-            if (len(xform.ti.free_typevars()) == 0):
-                assert len(concrete_typings_list) == 1
-                continue
-
-            # For any patterns where the type env includes constraints, at
-            # least one of the "theoretically possible" concrete typings must
-            # be prevented by the constraints. (i.e. we are not emitting
-            # unnecessary constraints).
-            # We check that by asserting that the number of concrete typings is
-            # less than the number of all possible free typevar assignments
-            if (len(xform.ti.constraints) > 0):
-                theoretical_num_typings =\
-                    reduce(lambda x, y:    x*y,
-                           [tv.get_typeset().size()
-                            for tv in xform.ti.free_typevars()], 1)
-                assert len(concrete_typings_list) < theoretical_num_typings
-
-            # Check the validity of each individual concrete typing against the
-            # xform
-            for concrete_typing in concrete_typings_list:
-                check_concrete_typing_xform(concrete_typing, xform)
-
-    def test_bound_inst_inference(self):
-        # First example from issue #26
-        x = XForm(
-            Rtl(
-                self.v0 << iadd(self.v1, self.v2),
-            ),
-            Rtl(
-                self.v3 << uextend.i32(self.v1),
-                self.v4 << uextend.i32(self.v2),
-                self.v5 << iadd(self.v3, self.v4),
-                self.v0 << ireduce(self.v5)
-            ))
-        itype = TypeVar("t", "", ints=True, simd=True)
-        i32t = TypeVar.singleton(i32)
-
-        check_typing(x.ti, ({
-            self.v0:    itype,
-            self.v1:    itype,
-            self.v2:    itype,
-            self.v3:    i32t,
-            self.v4:    i32t,
-            self.v5:    i32t,
-        }, [WiderOrEq(i32t, itype)]), x.symtab)
-
-    def test_bound_inst_inference1(self):
-        # Second example taken from issue #26
-        x = XForm(
-            Rtl(
-                self.v0 << iadd(self.v1, self.v2),
-            ),
-            Rtl(
-                self.v3 << uextend(self.v1),
-                self.v4 << uextend(self.v2),
-                self.v5 << iadd.i32(self.v3, self.v4),
-                self.v0 << ireduce(self.v5)
-            ))
-        itype = TypeVar("t", "", ints=True, simd=True)
-        i32t = TypeVar.singleton(i32)
-
-        check_typing(x.ti, ({
-            self.v0:    itype,
-            self.v1:    itype,
-            self.v2:    itype,
-            self.v3:    i32t,
-            self.v4:    i32t,
-            self.v5:    i32t,
-        }, [WiderOrEq(i32t, itype)]), x.symtab)
-
-    def test_fully_bound_inst_inference(self):
-        # Second example taken from issue #26 with complete bounds
-        x = XForm(
-            Rtl(
-                self.v0 << iadd(self.v1, self.v2),
-            ),
-            Rtl(
-                self.v3 << uextend.i32.i8(self.v1),
-                self.v4 << uextend.i32.i8(self.v2),
-                self.v5 << iadd(self.v3, self.v4),
-                self.v0 << ireduce(self.v5)
-            ))
-        i8t = TypeVar.singleton(i8)
-        i32t = TypeVar.singleton(i32)
-
-        # Note no constraints here since they are all trivial
-        check_typing(x.ti, ({
-            self.v0:    i8t,
-            self.v1:    i8t,
-            self.v2:    i8t,
-            self.v3:    i32t,
-            self.v4:    i32t,
-            self.v5:    i32t,
-        }, []), x.symtab)
-
-    def test_fully_bound_inst_inference_bad(self):
-        # Can't force a mistyped XForm using bound instructions
-        with self.assertRaises(AssertionError):
-            XForm(
-                Rtl(
-                    self.v0 << iadd(self.v1, self.v2),
-                ),
-                Rtl(
-                    self.v3 << uextend.i32.i8(self.v1),
-                    self.v4 << uextend.i32.i16(self.v2),
-                    self.v5 << iadd(self.v3, self.v4),
-                    self.v0 << ireduce(self.v5)
-                ))
diff --git a/cranelift/codegen/meta-python/cdsl/test_typevar.py b/cranelift/codegen/meta-python/cdsl/test_typevar.py
deleted file mode 100644
index 48806cc4aa..0000000000
--- a/cranelift/codegen/meta-python/cdsl/test_typevar.py
+++ /dev/null
@@ -1,266 +0,0 @@
-from __future__ import absolute_import
-from unittest import TestCase
-from doctest import DocTestSuite
-from . import typevar
-from .typevar import TypeSet, TypeVar
-from base.types import i32, i16, b1, f64
-from itertools import product
-from functools import reduce
-
-
-def load_tests(loader, tests, ignore):
-    tests.addTests(DocTestSuite(typevar))
-    return tests
-
-
-class TestTypeSet(TestCase):
-    def test_invalid(self):
-        with self.assertRaises(AssertionError):
-            TypeSet(lanes=(2, 1))
-        with self.assertRaises(AssertionError):
-            TypeSet(ints=(32, 16))
-        with self.assertRaises(AssertionError):
-            TypeSet(floats=(32, 16))
-        with self.assertRaises(AssertionError):
-            TypeSet(bools=(32, 16))
-        with self.assertRaises(AssertionError):
-            TypeSet(ints=(32, 33))
-
-    def test_hash(self):
-        a = TypeSet(lanes=True, ints=True, floats=True)
-        b = TypeSet(lanes=True, ints=True, floats=True)
-        c = TypeSet(lanes=True, ints=(8, 16), floats=True)
-        self.assertEqual(a, b)
-        self.assertNotEqual(a, c)
-        s = set()
-        s.add(a)
-        self.assertTrue(a in s)
-        self.assertTrue(b in s)
-        self.assertFalse(c in s)
-
-    def test_hash_modified(self):
-        a = TypeSet(lanes=True, ints=True, floats=True)
-        s = set()
-        s.add(a)
-        a.ints.remove(64)
-        # Can't rehash after modification.
-        with self.assertRaises(AssertionError):
-            a in s
-
-    def test_forward_images(self):
-        a = TypeSet(lanes=(2, 8), ints=(8, 8), floats=(32, 32))
-        b = TypeSet(lanes=(1, 8), ints=(8, 8), floats=(32, 32))
-        self.assertEqual(a.lane_of(), TypeSet(ints=(8, 8), floats=(32, 32)))
-
-        c = TypeSet(lanes=(2, 8))
-        c.bools = set([8, 32])
-
-        # Test case with disjoint intervals
-        self.assertEqual(a.as_bool(), c)
-
-        # For as_bool check b1 is present when 1 \in lanes
-        d = TypeSet(lanes=(1, 8))
-        d.bools = set([1, 8, 32])
-        self.assertEqual(b.as_bool(), d)
-
-        self.assertEqual(TypeSet(lanes=(1, 32)).half_vector(),
-                         TypeSet(lanes=(1, 16)))
-
-        self.assertEqual(TypeSet(lanes=(1, 32)).double_vector(),
-                         TypeSet(lanes=(2, 64)))
-
-        self.assertEqual(TypeSet(lanes=(128, 256)).double_vector(),
-                         TypeSet(lanes=(256, 256)))
-
-        self.assertEqual(TypeSet(ints=(8, 32)).half_width(),
-                         TypeSet(ints=(8, 16)))
-
-        self.assertEqual(TypeSet(ints=(8, 32)).double_width(),
-                         TypeSet(ints=(16, 64)))
-
-        self.assertEqual(TypeSet(ints=(32, 64)).double_width(),
-                         TypeSet(ints=(64, 64)))
-
-        # Should produce an empty ts
-        self.assertEqual(TypeSet(floats=(32, 32)).half_width(),
-                         TypeSet())
-
-        self.assertEqual(TypeSet(floats=(32, 64)).half_width(),
-                         TypeSet(floats=(32, 32)))
-
-        self.assertEqual(TypeSet(floats=(32, 32)).double_width(),
-                         TypeSet(floats=(64, 64)))
-
-        self.assertEqual(TypeSet(floats=(32, 64)).double_width(),
-                         TypeSet(floats=(64, 64)))
-
-        # Bools have trickier behavior around b1 (since b2, b4 don't exist)
-        self.assertEqual(TypeSet(bools=(1, 8)).half_width(),
-                         TypeSet())
-
-        t = TypeSet()
-        t.bools = set([8, 16])
-        self.assertEqual(TypeSet(bools=(1, 32)).half_width(), t)
-
-        # double_width() of bools={1, 8, 16} must not include 2 or 8
-        t.bools = set([16, 32])
-        self.assertEqual(TypeSet(bools=(1, 16)).double_width(), t)
-
-        self.assertEqual(TypeSet(bools=(32, 64)).double_width(),
-                         TypeSet(bools=(64, 64)))
-
-    def test_get_singleton(self):
-        # Raise error when calling get_singleton() on non-singleton TS
-        t = TypeSet(lanes=(1, 1), ints=(8, 8), floats=(32, 32))
-        with self.assertRaises(AssertionError):
-            t.get_singleton()
-        t = TypeSet(lanes=(1, 2), floats=(32, 32))
-
-        with self.assertRaises(AssertionError):
-            t.get_singleton()
-
-        self.assertEqual(TypeSet(ints=(16, 16)).get_singleton(), i16)
-        self.assertEqual(TypeSet(floats=(64, 64)).get_singleton(), f64)
-        self.assertEqual(TypeSet(bools=(1, 1)).get_singleton(), b1)
-        self.assertEqual(TypeSet(lanes=(4, 4), ints=(32, 32)).get_singleton(),
-                         i32.by(4))
-
-    def test_preimage(self):
-        t = TypeSet(lanes=(1, 1), ints=(8, 8), floats=(32, 32))
-
-        # LANEOF
-        self.assertEqual(TypeSet(lanes=True, ints=(8, 8), floats=(32, 32)),
-                         t.preimage(TypeVar.LANEOF))
-        # Inverse of empty set is still empty across LANEOF
-        self.assertEqual(TypeSet(),
-                         TypeSet().preimage(TypeVar.LANEOF))
-
-        # ASBOOL
-        t = TypeSet(lanes=(1, 4), bools=(1, 64))
-        self.assertEqual(t.preimage(TypeVar.ASBOOL),
-                         TypeSet(lanes=(1, 4), ints=True, bools=True,
-                                 floats=True))
-
-        # Half/Double Vector
-        t = TypeSet(lanes=(1, 1), ints=(8, 8))
-        t1 = TypeSet(lanes=(256, 256), ints=(8, 8))
-        self.assertEqual(t.preimage(TypeVar.DOUBLEVECTOR).size(), 0)
-        self.assertEqual(t1.preimage(TypeVar.HALFVECTOR).size(), 0)
-
-        t = TypeSet(lanes=(1, 16), ints=(8, 16), floats=(32, 32))
-        t1 = TypeSet(lanes=(64, 256), bools=(1, 32))
-
-        self.assertEqual(t.preimage(TypeVar.DOUBLEVECTOR),
-                         TypeSet(lanes=(1, 8), ints=(8, 16), floats=(32, 32)))
-        self.assertEqual(t1.preimage(TypeVar.HALFVECTOR),
-                         TypeSet(lanes=(128, 256), bools=(1, 32)))
-
-        # Half/Double Width
-        t = TypeSet(ints=(8, 8), floats=(32, 32), bools=(1, 8))
-        t1 = TypeSet(ints=(64, 64), floats=(64, 64), bools=(64, 64))
-        self.assertEqual(t.preimage(TypeVar.DOUBLEWIDTH).size(), 0)
-        self.assertEqual(t1.preimage(TypeVar.HALFWIDTH).size(), 0)
-
-        t = TypeSet(lanes=(1, 16), ints=(8, 16), floats=(32, 64))
-        t1 = TypeSet(lanes=(64, 256), bools=(1, 64))
-
-        self.assertEqual(t.preimage(TypeVar.DOUBLEWIDTH),
-                         TypeSet(lanes=(1, 16), ints=(8, 8), floats=(32, 32)))
-        self.assertEqual(t1.preimage(TypeVar.HALFWIDTH),
-                         TypeSet(lanes=(64, 256), bools=(16, 64)))
-
-
-def has_non_bijective_derived_f(iterable):
-    return any(not TypeVar.is_bijection(x) for x in iterable)
-
-
-class TestTypeVar(TestCase):
-    def test_functions(self):
-        x = TypeVar('x', 'all ints', ints=True)
-        with self.assertRaises(AssertionError):
-            x.double_width()
-        with self.assertRaises(AssertionError):
-            x.half_width()
-
-        x2 = TypeVar('x2', 'i16 and up', ints=(16, 64))
-        with self.assertRaises(AssertionError):
-            x2.double_width()
-        self.assertEqual(str(x2.half_width()), '`half_width(x2)`')
-        self.assertEqual(x2.half_width().rust_expr(), 'x2.half_width()')
-        self.assertEqual(
-                x2.half_width().double_width().rust_expr(),
-                'x2.half_width().double_width()')
-
-        x3 = TypeVar('x3', 'up to i32', ints=(8, 32))
-        self.assertEqual(str(x3.double_width()), '`double_width(x3)`')
-        with self.assertRaises(AssertionError):
-            x3.half_width()
-
-    def test_singleton(self):
-        x = TypeVar.singleton(i32)
-        self.assertEqual(str(x), '`i32`')
-        self.assertEqual(min(x.type_set.ints), 32)
-        self.assertEqual(max(x.type_set.ints), 32)
-        self.assertEqual(min(x.type_set.lanes), 1)
-        self.assertEqual(max(x.type_set.lanes), 1)
-        self.assertEqual(len(x.type_set.floats), 0)
-        self.assertEqual(len(x.type_set.bools), 0)
-
-        x = TypeVar.singleton(i32.by(4))
-        self.assertEqual(str(x), '`i32x4`')
-        self.assertEqual(min(x.type_set.ints), 32)
-        self.assertEqual(max(x.type_set.ints), 32)
-        self.assertEqual(min(x.type_set.lanes), 4)
-        self.assertEqual(max(x.type_set.lanes), 4)
-        self.assertEqual(len(x.type_set.floats), 0)
-        self.assertEqual(len(x.type_set.bools), 0)
-
-    def test_stress_constrain_types(self):
-        # Get all 43 possible derived vars of length up to 2
-        funcs = [TypeVar.LANEOF,
-                 TypeVar.ASBOOL, TypeVar.HALFVECTOR, TypeVar.DOUBLEVECTOR,
-                 TypeVar.HALFWIDTH, TypeVar.DOUBLEWIDTH]
-        v = [()] + [(x,) for x in funcs] + list(product(*[funcs, funcs]))
-
-        # For each pair of derived variables
-        for (i1, i2) in product(v, v):
-            # Compute the derived sets for each  starting with a full typeset
-            full_ts = TypeSet(lanes=True, floats=True, ints=True, bools=True)
-            ts1 = reduce(lambda ts, func:   ts.image(func), i1, full_ts)
-            ts2 = reduce(lambda ts, func:   ts.image(func), i2, full_ts)
-
-            # Compute intersection
-            intersect = ts1.copy()
-            intersect &= ts2
-
-            # Propagate intersections backward
-            ts1_src = reduce(lambda ts, func:   ts.preimage(func),
-                             reversed(i1),
-                             intersect)
-            ts2_src = reduce(lambda ts, func:   ts.preimage(func),
-                             reversed(i2),
-                             intersect)
-
-            # If the intersection or its propagated forms are empty, then these
-            # two variables can never overlap. For example x.double_vector and
-            # x.lane_of.
-            if (intersect.size() == 0 or ts1_src.size() == 0 or
-                    ts2_src.size() == 0):
-                continue
-
-            # Should be safe to create derived tvs from ts1_src and ts2_src
-            tv1 = reduce(lambda tv, func:   TypeVar.derived(tv, func),
-                         i1,
-                         TypeVar.from_typeset(ts1_src))
-
-            tv2 = reduce(lambda tv, func:   TypeVar.derived(tv, func),
-                         i2,
-                         TypeVar.from_typeset(ts2_src))
-
-            # In the absence of AS_BOOL image(preimage(f)) == f so the
-            # typesets of tv1 and tv2 should be exactly intersection
-            assert tv1.get_typeset() == intersect or\
-                has_non_bijective_derived_f(i1)
-
-            assert tv2.get_typeset() == intersect or\
-                has_non_bijective_derived_f(i2)
diff --git a/cranelift/codegen/meta-python/cdsl/test_xform.py b/cranelift/codegen/meta-python/cdsl/test_xform.py
deleted file mode 100644
index 424a7c824d..0000000000
--- a/cranelift/codegen/meta-python/cdsl/test_xform.py
+++ /dev/null
@@ -1,131 +0,0 @@
-from __future__ import absolute_import
-from unittest import TestCase
-from doctest import DocTestSuite
-from base.instructions import iadd, iadd_imm, iconst, icmp
-from base.immediates import intcc
-from . import xform
-from .ast import Var
-from .xform import Rtl, XForm
-
-
-def load_tests(loader, tests, ignore):
-    tests.addTests(DocTestSuite(xform))
-    return tests
-
-
-x = Var('x')
-y = Var('y')
-z = Var('z')
-u = Var('u')
-a = Var('a')
-b = Var('b')
-c = Var('c')
-
-CC1 = Var('CC1')
-CC2 = Var('CC2')
-
-
-class TestXForm(TestCase):
-    def test_macro_pattern(self):
-        src = Rtl(a << iadd_imm(x, y))
-        dst = Rtl(
-                c << iconst(y),
-                a << iadd(x, c))
-        XForm(src, dst)
-
-    def test_def_input(self):
-        # Src pattern has a def which is an input in dst.
-        src = Rtl(a << iadd_imm(x, 1))
-        dst = Rtl(y << iadd_imm(a, 1))
-        with self.assertRaisesRegexp(
-                AssertionError,
-                "'a' used as both input and def"):
-            XForm(src, dst)
-
-    def test_input_def(self):
-        # Converse of the above.
-        src = Rtl(y << iadd_imm(a, 1))
-        dst = Rtl(a << iadd_imm(x, 1))
-        with self.assertRaisesRegexp(
-                AssertionError,
-                "'a' used as both input and def"):
-            XForm(src, dst)
-
-    def test_extra_input(self):
-        src = Rtl(a << iadd_imm(x, 1))
-        dst = Rtl(a << iadd(x, y))
-        with self.assertRaisesRegexp(AssertionError, "extra inputs in dst"):
-            XForm(src, dst)
-
-    def test_double_def(self):
-        src = Rtl(
-                a << iadd_imm(x, 1),
-                a << iadd(x, y))
-        dst = Rtl(a << iadd(x, y))
-        with self.assertRaisesRegexp(AssertionError, "'a' multiply defined"):
-            XForm(src, dst)
-
-    def test_subst_imm(self):
-        src = Rtl(a << iconst(x))
-        dst = Rtl(c << iconst(y))
-        assert src.substitution(dst, {}) == {a: c, x: y}
-
-    def test_subst_enum_var(self):
-        src = Rtl(a << icmp(CC1, x, y))
-        dst = Rtl(b << icmp(CC2, z, u))
-        assert src.substitution(dst, {}) == {a: b, CC1: CC2, x: z, y: u}
-
-    def test_subst_enum_const(self):
-        src = Rtl(a << icmp(intcc.eq, x, y))
-        dst = Rtl(b << icmp(intcc.eq, z, u))
-        assert src.substitution(dst, {}) == {a: b, x: z, y: u}
-
-    def test_subst_enum_var_const(self):
-        src = Rtl(a << icmp(CC1, x, y))
-        dst = Rtl(b << icmp(intcc.eq, z, u))
-        assert src.substitution(dst, {}) == {CC1: intcc.eq, x: z, y: u, a: b},\
-            "{} != {}".format(src.substitution(dst, {}),
-                              {CC1: intcc.eq, x: z, y: u, a: b})
-
-        src = Rtl(a << icmp(intcc.eq, x, y))
-        dst = Rtl(b << icmp(CC1, z, u))
-        assert src.substitution(dst, {}) == {CC1: intcc.eq, x: z, y: u, a: b}
-
-    def test_subst_enum_bad(self):
-        src = Rtl(a << icmp(intcc.eq, x, y))
-        dst = Rtl(b << icmp(intcc.sge, z, u))
-        assert src.substitution(dst, {}) is None
-
-    def test_subst_enum_bad_var_const(self):
-        a1 = Var('a1')
-        x1 = Var('x1')
-        y1 = Var('y1')
-
-        b1 = Var('b1')
-        z1 = Var('z1')
-        u1 = Var('u1')
-
-        # Var mapping to 2 different constants
-        src = Rtl(a << icmp(CC1, x, y),
-                  a1 << icmp(CC1, x1, y1))
-        dst = Rtl(b << icmp(intcc.eq, z, u),
-                  b1 << icmp(intcc.sge, z1, u1))
-
-        assert src.substitution(dst, {}) is None
-
-        # 2 different constants mapping to the same var
-        src = Rtl(a << icmp(intcc.eq, x, y),
-                  a1 << icmp(intcc.sge, x1, y1))
-        dst = Rtl(b << icmp(CC1, z, u),
-                  b1 << icmp(CC1, z1, u1))
-
-        assert src.substitution(dst, {}) is None
-
-        # Var mapping to var and constant - note that full unification would
-        # have allowed this.
-        src = Rtl(a << icmp(CC1, x, y),
-                  a1 << icmp(CC1, x1, y1))
-        dst = Rtl(b << icmp(CC2, z, u),
-                  b1 << icmp(intcc.sge, z1, u1))
-
-        assert src.substitution(dst, {}) is None
diff --git a/cranelift/codegen/meta-python/cdsl/ti.py b/cranelift/codegen/meta-python/cdsl/ti.py
deleted file mode 100644
index 26f01f9e6b..0000000000
--- a/cranelift/codegen/meta-python/cdsl/ti.py
+++ /dev/null
@@ -1,894 +0,0 @@
-"""
-Type Inference
-"""
-from .typevar import TypeVar
-from .ast import Def, Var
-from copy import copy
-from itertools import product
-
-try:
-    from typing import Dict, TYPE_CHECKING, Union, Tuple, Optional, Set # noqa
-    from typing import Iterable, List, Any, TypeVar as MTypeVar # noqa
-    from typing import cast
-    from .xform import Rtl, XForm # noqa
-    from .ast import Expr # noqa
-    from .typevar import TypeSet # noqa
-    if TYPE_CHECKING:
-        T = MTypeVar('T')
-        TypeMap = Dict[TypeVar, TypeVar]
-        VarTyping = Dict[Var, TypeVar]
-except ImportError:
-    TYPE_CHECKING = False
-    pass
-
-
-class TypeConstraint(object):
-    """
-    Base class for all runtime-emittable type constraints.
-    """
-
-    def __init__(self, tv, tc):
-        # type: (TypeVar, Union[TypeVar, TypeSet]) -> None
-        """
-        Abstract "constructor" for linters
-        """
-        assert False, "Abstract"
-
-    def translate(self, m):
-        # type: (Union[TypeEnv, TypeMap]) -> TypeConstraint
-        """
-        Translate any TypeVars in the constraint according to the map or
-        TypeEnv m
-        """
-        def translate_one(a):
-            # type: (Any) -> Any
-            if (isinstance(a, TypeVar)):
-                return m[a] if isinstance(m, TypeEnv) else subst(a, m)
-            return a
-
-        res = None  # type: TypeConstraint
-        res = self.__class__(*tuple(map(translate_one, self._args())))
-        return res
-
-    def __eq__(self, other):
-        # type: (object) -> bool
-        if (not isinstance(other, self.__class__)):
-            return False
-
-        assert isinstance(other, TypeConstraint)  # help MyPy figure out other
-        return self._args() == other._args()
-
-    def is_concrete(self):
-        # type: () -> bool
-        """
-        Return true iff all typevars in the constraint are singletons.
-        """
-        return [] == list(filter(lambda x:  x.singleton_type() is None,
-                                 self.tvs()))
-
-    def __hash__(self):
-        # type: () -> int
-        return hash(self._args())
-
-    def _args(self):
-        # type: () -> Tuple[Any,...]
-        """
-        Return a tuple with the exact arguments passed to __init__ to create
-        this object.
-        """
-        assert False, "Abstract"
-
-    def tvs(self):
-        # type: () -> Iterable[TypeVar]
-        """
-        Return the typevars contained in this constraint.
-        """
-        return list(filter(lambda x: isinstance(x, TypeVar), self._args()))
-
-    def is_trivial(self):
-        # type: () -> bool
-        """
-        Return true if this constrain is statically decidable.
-        """
-        assert False, "Abstract"
-
-    def eval(self):
-        # type: () -> bool
-        """
-        Evaluate this constraint. Should only be called when the constraint has
-        been translated to concrete types.
-        """
-        assert False, "Abstract"
-
-    def __repr__(self):
-        # type: () -> str
-        return (self.__class__.__name__ + '(' +
-                ', '.join(map(str, self._args())) + ')')
-
-
-class TypesEqual(TypeConstraint):
-    """
-    Constraint specifying that two derived type vars must have the same runtime
-    type.
-    """
-    def __init__(self, tv1, tv2):
-        # type: (TypeVar, TypeVar) -> None
-        (self.tv1, self.tv2) = sorted([tv1, tv2], key=repr)
-
-    def _args(self):
-        # type: () -> Tuple[Any,...]
-        """ See TypeConstraint._args() """
-        return (self.tv1, self.tv2)
-
-    def is_trivial(self):
-        # type: () -> bool
-        """ See TypeConstraint.is_trivial() """
-        return self.tv1 == self.tv2 or self.is_concrete()
-
-    def eval(self):
-        # type: () -> bool
-        """ See TypeConstraint.eval() """
-        assert self.is_concrete()
-        return self.tv1.singleton_type() == self.tv2.singleton_type()
-
-
-class InTypeset(TypeConstraint):
-    """
-    Constraint specifying that a type var must belong to some typeset.
-    """
-    def __init__(self, tv, ts):
-        # type: (TypeVar, TypeSet) -> None
-        assert not tv.is_derived and tv.name.startswith("typeof_")
-        self.tv = tv
-        self.ts = ts
-
-    def _args(self):
-        # type: () -> Tuple[Any,...]
-        """ See TypeConstraint._args() """
-        return (self.tv, self.ts)
-
-    def is_trivial(self):
-        # type: () -> bool
-        """ See TypeConstraint.is_trivial() """
-        tv_ts = self.tv.get_typeset().copy()
-
-        # Trivially True
-        if (tv_ts.issubset(self.ts)):
-            return True
-
-        # Trivially false
-        tv_ts &= self.ts
-        if (tv_ts.size() == 0):
-            return True
-
-        return self.is_concrete()
-
-    def eval(self):
-        # type: () -> bool
-        """ See TypeConstraint.eval() """
-        assert self.is_concrete()
-        return self.tv.get_typeset().issubset(self.ts)
-
-
-class WiderOrEq(TypeConstraint):
-    """
-    Constraint specifying that a type var tv1 must be wider than or equal to
-    type var tv2 at runtime. This requires that:
-        1) They have the same number of lanes
-        2) In a lane tv1 has at least as many bits as tv2.
-    """
-    def __init__(self, tv1, tv2):
-        # type: (TypeVar, TypeVar) -> None
-        self.tv1 = tv1
-        self.tv2 = tv2
-
-    def _args(self):
-        # type: () -> Tuple[Any,...]
-        """ See TypeConstraint._args() """
-        return (self.tv1, self.tv2)
-
-    def is_trivial(self):
-        # type: () -> bool
-        """ See TypeConstraint.is_trivial() """
-        # Trivially true
-        if (self.tv1 == self.tv2):
-            return True
-
-        ts1 = self.tv1.get_typeset()
-        ts2 = self.tv2.get_typeset()
-
-        def set_wider_or_equal(s1, s2):
-            # type: (Set[int], Set[int]) -> bool
-            return len(s1) > 0 and len(s2) > 0 and min(s1) >= max(s2)
-
-        # Trivially True
-        if set_wider_or_equal(ts1.ints, ts2.ints) and\
-           set_wider_or_equal(ts1.floats, ts2.floats) and\
-           set_wider_or_equal(ts1.bools, ts2.bools):
-            return True
-
-        def set_narrower(s1, s2):
-            # type: (Set[int], Set[int]) -> bool
-            return len(s1) > 0 and len(s2) > 0 and min(s1) < max(s2)
-
-        # Trivially False
-        if set_narrower(ts1.ints, ts2.ints) and\
-           set_narrower(ts1.floats, ts2.floats) and\
-           set_narrower(ts1.bools, ts2.bools):
-            return True
-
-        # Trivially False
-        if len(ts1.lanes.intersection(ts2.lanes)) == 0:
-            return True
-
-        return self.is_concrete()
-
-    def eval(self):
-        # type: () -> bool
-        """ See TypeConstraint.eval() """
-        assert self.is_concrete()
-        typ1 = self.tv1.singleton_type()
-        typ2 = self.tv2.singleton_type()
-
-        return typ1.wider_or_equal(typ2)
-
-
-class SameWidth(TypeConstraint):
-    """
-    Constraint specifying that two types have the same width. E.g. i32x2 has
-    the same width as i64x1, i16x4, f32x2, f64, b1x64 etc.
-    """
-    def __init__(self, tv1, tv2):
-        # type: (TypeVar, TypeVar) -> None
-        self.tv1 = tv1
-        self.tv2 = tv2
-
-    def _args(self):
-        # type: () -> Tuple[Any,...]
-        """ See TypeConstraint._args() """
-        return (self.tv1, self.tv2)
-
-    def is_trivial(self):
-        # type: () -> bool
-        """ See TypeConstraint.is_trivial() """
-        # Trivially true
-        if (self.tv1 == self.tv2):
-            return True
-
-        ts1 = self.tv1.get_typeset()
-        ts2 = self.tv2.get_typeset()
-
-        # Trivially False
-        if len(ts1.widths().intersection(ts2.widths())) == 0:
-            return True
-
-        return self.is_concrete()
-
-    def eval(self):
-        # type: () -> bool
-        """ See TypeConstraint.eval() """
-        assert self.is_concrete()
-        typ1 = self.tv1.singleton_type()
-        typ2 = self.tv2.singleton_type()
-
-        return (typ1.width() == typ2.width())
-
-
-class TypeEnv(object):
-    """
-    Class encapsulating the necessary book keeping for type inference.
-        :attribute type_map: dict holding the equivalence relations between tvs
-        :attribute constraints: a list of accumulated constraints - tuples
-                            (tv1, tv2)) where tv1 and tv2 are equal
-        :attribute ranks: dictionary recording the (optional) ranks for tvs.
-                          'rank' is a partial ordering on TVs based on their
-                          origin. See comments in rank() and register().
-        :attribute vars: a set containing all known Vars
-        :attribute idx: counter used to get fresh ids
-    """
-
-    RANK_SINGLETON = 5
-    RANK_INPUT = 4
-    RANK_INTERMEDIATE = 3
-    RANK_OUTPUT = 2
-    RANK_TEMP = 1
-    RANK_INTERNAL = 0
-
-    def __init__(self, arg=None):
-        # type: (Optional[Tuple[TypeMap, List[TypeConstraint]]]) -> None
-        self.ranks = {}  # type: Dict[TypeVar, int]
-        self.vars = set()  # type: Set[Var]
-
-        if arg is None:
-            self.type_map = {}  # type: TypeMap
-            self.constraints = []  # type: List[TypeConstraint]
-        else:
-            self.type_map, self.constraints = arg
-
-        self.idx = 0
-
-    def __getitem__(self, arg):
-        # type: (Union[TypeVar, Var]) -> TypeVar
-        """
-        Lookup the canonical representative for a Var/TypeVar.
-        """
-        if (isinstance(arg, Var)):
-            assert arg in self.vars
-            tv = arg.get_typevar()
-        else:
-            assert (isinstance(arg, TypeVar))
-            tv = arg
-
-        while tv in self.type_map:
-            tv = self.type_map[tv]
-
-        if tv.is_derived:
-            tv = TypeVar.derived(self[tv.base], tv.derived_func)
-        return tv
-
-    def equivalent(self, tv1, tv2):
-        # type: (TypeVar, TypeVar) -> None
-        """
-        Record a that the free tv1 is part of the same equivalence class as
-        tv2. The canonical representative of the merged class is tv2's
-        canonical representative.
-        """
-        assert not tv1.is_derived
-        assert self[tv1] == tv1
-
-        # Make sure we don't create cycles
-        if tv2.is_derived:
-            assert self[tv2.base] != tv1
-
-        self.type_map[tv1] = tv2
-
-    def add_constraint(self, constr):
-        # type: (TypeConstraint) -> None
-        """
-        Add a new constraint
-        """
-        if (constr in self.constraints):
-            return
-
-        # InTypeset constraints can be expressed by constraining the typeset of
-        # a variable. No need to add them to self.constraints
-        if (isinstance(constr, InTypeset)):
-            self[constr.tv].constrain_types_by_ts(constr.ts)
-            return
-
-        self.constraints.append(constr)
-
-    def get_uid(self):
-        # type: () -> str
-        r = str(self.idx)
-        self.idx += 1
-        return r
-
-    def __repr__(self):
-        # type: () -> str
-        return self.dot()
-
-    def rank(self, tv):
-        # type: (TypeVar) -> int
-        """
-        Get the rank of tv in the partial order. TVs directly associated with a
-        Var get their rank from the Var (see register()). Internally generated
-        non-derived TVs implicitly get the lowest rank (0). Derived variables
-        get their rank from their free typevar. Singletons have the highest
-        rank. TVs associated with vars in a source pattern have a higher rank
-        than TVs associated with temporary vars.
-        """
-        default_rank = TypeEnv.RANK_INTERNAL if tv.singleton_type() is None \
-            else TypeEnv.RANK_SINGLETON
-
-        if tv.is_derived:
-            tv = tv.free_typevar()
-
-        return self.ranks.get(tv, default_rank)
-
-    def register(self, v):
-        # type: (Var) -> None
-        """
-        Register a new Var v. This computes a rank for the associated TypeVar
-        for v, which is used to impose a partial order on type variables.
-        """
-        self.vars.add(v)
-
-        if v.is_input():
-            r = TypeEnv.RANK_INPUT
-        elif v.is_intermediate():
-            r = TypeEnv.RANK_INTERMEDIATE
-        elif v.is_output():
-            r = TypeEnv.RANK_OUTPUT
-        else:
-            assert(v.is_temp())
-            r = TypeEnv.RANK_TEMP
-
-        self.ranks[v.get_typevar()] = r
-
-    def free_typevars(self):
-        # type: () -> List[TypeVar]
-        """
-        Get the free typevars in the current type env.
-        """
-        tvs = set([self[tv].free_typevar() for tv in self.type_map.keys()])
-        tvs = tvs.union(set([self[v].free_typevar() for v in self.vars]))
-        # Filter out None here due to singleton type vars
-        return sorted(filter(lambda x: x is not None, tvs),
-                      key=lambda x:   x.name)
-
-    def normalize(self):
-        # type: () -> None
-        """
-        Normalize by:
-            - collapsing any roots that don't correspond to a concrete TV AND
-              have a single TV derived from them or equivalent to them
-
-        E.g. if we have a root of the tree that looks like:
-
-          typeof_a   typeof_b
-                 \\  /
-              typeof_x
-                  |
-                half_width(1)
-                  |
-                  1
-
-        we want to collapse the linear path between 1 and typeof_x. The
-        resulting graph is:
-
-          typeof_a   typeof_b
-                 \\  /
-              typeof_x
-        """
-        source_tvs = set([v.get_typevar() for v in self.vars])
-        children = {}  # type: Dict[TypeVar, Set[TypeVar]]
-        for v in self.type_map.values():
-            if not v.is_derived:
-                continue
-
-            t = v.free_typevar()
-            s = children.get(t, set())
-            s.add(v)
-            children[t] = s
-
-        for (a, b) in self.type_map.items():
-            s = children.get(b, set())
-            s.add(a)
-            children[b] = s
-
-        for r in self.free_typevars():
-            while (r not in source_tvs and r in children and
-                   len(children[r]) == 1):
-                child = list(children[r])[0]
-                if child in self.type_map:
-                    assert self.type_map[child] == r
-                    del self.type_map[child]
-
-                r = child
-
-    def extract(self):
-        # type: () -> TypeEnv
-        """
-        Extract a clean type environment from self, that only mentions
-        TVs associated with real variables
-        """
-        vars_tvs = set([v.get_typevar() for v in self.vars])
-        new_type_map = {tv: self[tv] for tv in vars_tvs if tv != self[tv]}
-
-        new_constraints = []  # type: List[TypeConstraint]
-        for constr in self.constraints:
-            constr = constr.translate(self)
-
-            if constr.is_trivial() or constr in new_constraints:
-                continue
-
-            # Sanity: translated constraints should refer to only real vars
-            for arg in constr._args():
-                if (not isinstance(arg, TypeVar)):
-                    continue
-
-                arg_free_tv = arg.free_typevar()
-                assert arg_free_tv is None or arg_free_tv in vars_tvs
-
-            new_constraints.append(constr)
-
-        # Sanity: translated typemap should refer to only real vars
-        for (k, v) in new_type_map.items():
-            assert k in vars_tvs
-            assert v.free_typevar() is None or v.free_typevar() in vars_tvs
-
-        t = TypeEnv()
-        t.type_map = new_type_map
-        t.constraints = new_constraints
-        # ranks and vars contain only TVs associated with real vars
-        t.ranks = copy(self.ranks)
-        t.vars = copy(self.vars)
-        return t
-
-    def concrete_typings(self):
-        # type: () -> Iterable[VarTyping]
-        """
-        Return an iterable over all possible concrete typings permitted by this
-        TypeEnv.
-        """
-        free_tvs = self.free_typevars()
-        free_tv_iters = [tv.get_typeset().concrete_types() for tv in free_tvs]
-        for concrete_types in product(*free_tv_iters):
-            # Build type substitutions for all free vars
-            m = {tv: TypeVar.singleton(typ)
-                 for (tv, typ) in zip(free_tvs, concrete_types)}
-
-            concrete_var_map = {v: subst(self[v.get_typevar()], m)
-                                for v in self.vars}
-
-            # Check if constraints are satisfied for this typing
-            failed = None
-            for constr in self.constraints:
-                concrete_constr = constr.translate(m)
-                if not concrete_constr.eval():
-                    failed = concrete_constr
-                    break
-
-            if (failed is not None):
-                continue
-
-            yield concrete_var_map
-
-    def permits(self, concrete_typing):
-        # type: (VarTyping) -> bool
-        """
-        Return true iff this TypeEnv permits the (possibly partial) concrete
-        variable type mapping concrete_typing.
-        """
-        # Each variable has a concrete type, that is a subset of its inferred
-        # typeset.
-        for (v, typ) in concrete_typing.items():
-            assert typ.singleton_type() is not None
-            if not typ.get_typeset().issubset(self[v].get_typeset()):
-                return False
-
-        m = {self[v]: typ for (v, typ) in concrete_typing.items()}
-
-        # Constraints involving vars in concrete_typing are satisfied
-        for constr in self.constraints:
-            try:
-                # If the constraint includes only vars in concrete_typing, we
-                # can translate it using m. Otherwise we encounter a KeyError
-                # and ignore it
-                constr = constr.translate(m)
-                if not constr.eval():
-                    return False
-            except KeyError:
-                pass
-
-        return True
-
-    def dot(self):
-        # type: () -> str
-        """
-        Return a representation of self as a graph in dot format.
-            Nodes correspond to TypeVariables.
-            Dotted edges correspond to equivalences between TVS
-            Solid edges correspond to derivation relations between TVs.
-            Dashed edges correspond to equivalence constraints.
-        """
-        def label(s):
-            # type: (TypeVar) -> str
-            return "\"" + str(s) + "\""
-
-        # Add all registered TVs (as some of them may be singleton nodes not
-        # appearing in the graph
-        nodes = set()  # type: Set[TypeVar]
-        edges = set()  # type: Set[Tuple[TypeVar, TypeVar, str, str, Optional[str]]] # noqa
-
-        def add_nodes(*args):
-            # type: (*TypeVar) -> None
-            for tv in args:
-                nodes.add(tv)
-                while (tv.is_derived):
-                    nodes.add(tv.base)
-                    edges.add((tv, tv.base, "solid", "forward",
-                               tv.derived_func))
-                    tv = tv.base
-
-        for v in self.vars:
-            add_nodes(v.get_typevar())
-
-        for (tv1, tv2) in self.type_map.items():
-            # Add all intermediate TVs appearing in edges
-            add_nodes(tv1, tv2)
-            edges.add((tv1, tv2, "dotted", "forward", None))
-
-        for constr in self.constraints:
-            if isinstance(constr, TypesEqual):
-                add_nodes(constr.tv1, constr.tv2)
-                edges.add((constr.tv1, constr.tv2, "dashed", "none", "equal"))
-            elif isinstance(constr, WiderOrEq):
-                add_nodes(constr.tv1, constr.tv2)
-                edges.add((constr.tv1, constr.tv2, "dashed", "forward", ">="))
-            elif isinstance(constr, SameWidth):
-                add_nodes(constr.tv1, constr.tv2)
-                edges.add((constr.tv1, constr.tv2, "dashed", "none",
-                           "same_width"))
-            else:
-                assert False, "Can't display constraint {}".format(constr)
-
-        root_nodes = set([x for x in nodes
-                          if x not in self.type_map and not x.is_derived])
-
-        r = "digraph {\n"
-        for n in nodes:
-            r += label(n)
-            if n in root_nodes:
-                r += "[xlabel=\"{}\"]".format(self[n].get_typeset())
-            r += ";\n"
-
-        for (n1, n2, style, direction, elabel) in edges:
-            e = label(n1) + "->" + label(n2)
-            e += "[style={},dir={}".format(style, direction)
-
-            if elabel is not None:
-                e += ",label=\"{}\"".format(elabel)
-            e += "];\n"
-
-            r += e
-        r += "}"
-
-        return r
-
-
-if TYPE_CHECKING:
-    TypingError = str
-    TypingOrError = Union[TypeEnv, TypingError]
-
-
-def get_error(typing_or_err):
-    # type: (TypingOrError) -> Optional[TypingError]
-    """
-    Helper function to appease mypy when checking the result of typing.
-    """
-    if isinstance(typing_or_err, str):
-        if (TYPE_CHECKING):
-            return cast(TypingError, typing_or_err)
-        else:
-            return typing_or_err
-    else:
-        return None
-
-
-def get_type_env(typing_or_err):
-    # type: (TypingOrError) -> TypeEnv
-    """
-    Helper function to appease mypy when checking the result of typing.
-    """
-    assert isinstance(typing_or_err, TypeEnv), \
-        "Unexpected error: {}".format(typing_or_err)
-
-    if (TYPE_CHECKING):
-        return cast(TypeEnv, typing_or_err)
-    else:
-        return typing_or_err
-
-
-def subst(tv, tv_map):
-    # type: (TypeVar, TypeMap) -> TypeVar
-    """
-    Perform substition on the input tv using the TypeMap tv_map.
-    """
-    if tv in tv_map:
-        return tv_map[tv]
-
-    if tv.is_derived:
-        return TypeVar.derived(subst(tv.base, tv_map), tv.derived_func)
-
-    return tv
-
-
-def normalize_tv(tv):
-    # type: (TypeVar) -> TypeVar
-    """
-    Normalize a (potentially derived) TV using the following rules:
-        - vector and width derived functions commute
-        {HALF,DOUBLE}VECTOR({HALF,DOUBLE}WIDTH(base)) ->
-            {HALF,DOUBLE}WIDTH({HALF,DOUBLE}VECTOR(base))
-
-        - half/double pairs collapse
-        {HALF,DOUBLE}WIDTH({DOUBLE,HALF}WIDTH(base)) -> base
-        {HALF,DOUBLE}VECTOR({DOUBLE,HALF}VECTOR(base)) -> base
-    """
-    vector_derives = [TypeVar.HALFVECTOR, TypeVar.DOUBLEVECTOR]
-    width_derives = [TypeVar.HALFWIDTH, TypeVar.DOUBLEWIDTH]
-
-    if not tv.is_derived:
-        return tv
-
-    df = tv.derived_func
-
-    if (tv.base.is_derived):
-        base_df = tv.base.derived_func
-
-        # Reordering: {HALFWIDTH, DOUBLEWIDTH} commute with {HALFVECTOR,
-        # DOUBLEVECTOR}. Arbitrarily pick WIDTH < VECTOR
-        if df in vector_derives and base_df in width_derives:
-            return normalize_tv(
-                    TypeVar.derived(
-                        TypeVar.derived(tv.base.base, df), base_df))
-
-        # Cancelling: HALFWIDTH, DOUBLEWIDTH and HALFVECTOR, DOUBLEVECTOR
-        # cancel each other. Note: This doesn't hide any over/underflows,
-        # since we 1) assert the safety of each TV in the chain upon its
-        # creation, and 2) the base typeset is only allowed to shrink.
-
-        if (df, base_df) in \
-                [(TypeVar.HALFVECTOR, TypeVar.DOUBLEVECTOR),
-                 (TypeVar.DOUBLEVECTOR, TypeVar.HALFVECTOR),
-                 (TypeVar.HALFWIDTH, TypeVar.DOUBLEWIDTH),
-                 (TypeVar.DOUBLEWIDTH, TypeVar.HALFWIDTH)]:
-            return normalize_tv(tv.base.base)
-
-    return TypeVar.derived(normalize_tv(tv.base), df)
-
-
-def constrain_fixpoint(tv1, tv2):
-    # type: (TypeVar, TypeVar) -> None
-    """
-    Given typevars tv1 and tv2 (which could be derived from one another)
-    constrain their typesets to be the same. When one is derived from the
-    other, repeat the constrain process until fixpoint.
-    """
-    # Constrain tv2's typeset as long as tv1's typeset is changing.
-    while True:
-        old_tv1_ts = tv1.get_typeset().copy()
-        tv2.constrain_types(tv1)
-        if tv1.get_typeset() == old_tv1_ts:
-            break
-
-    old_tv2_ts = tv2.get_typeset().copy()
-    tv1.constrain_types(tv2)
-    assert old_tv2_ts == tv2.get_typeset()
-
-
-def unify(tv1, tv2, typ):
-    # type: (TypeVar, TypeVar, TypeEnv) -> TypingOrError
-    """
-    Unify tv1 and tv2 in the current type environment typ, and return an
-    updated type environment or error.
-    """
-    tv1 = normalize_tv(typ[tv1])
-    tv2 = normalize_tv(typ[tv2])
-
-    # Already unified
-    if tv1 == tv2:
-        return typ
-
-    if typ.rank(tv2) < typ.rank(tv1):
-        return unify(tv2, tv1, typ)
-
-    constrain_fixpoint(tv1, tv2)
-
-    if (tv1.get_typeset().size() == 0 or tv2.get_typeset().size() == 0):
-        return "Error: empty type created when unifying {} and {}"\
-               .format(tv1, tv2)
-
-    # Free -> Derived(Free)
-    if not tv1.is_derived:
-        typ.equivalent(tv1, tv2)
-        return typ
-
-    if (tv1.is_derived and TypeVar.is_bijection(tv1.derived_func)):
-        inv_f = TypeVar.inverse_func(tv1.derived_func)
-        return unify(tv1.base, normalize_tv(TypeVar.derived(tv2, inv_f)), typ)
-
-    typ.add_constraint(TypesEqual(tv1, tv2))
-    return typ
-
-
-def move_first(l, i):
-    # type: (List[T], int) -> List[T]
-    return [l[i]] + l[:i] + l[i+1:]
-
-
-def ti_def(definition, typ):
-    # type: (Def, TypeEnv) -> TypingOrError
-    """
-    Perform type inference on one Def in the current type environment typ and
-    return an updated type environment or error.
-
-    At a high level this works by creating fresh copies of each formal type var
-    in the Def's instruction's signature, and unifying the formal tv with the
-    corresponding actual tv.
-    """
-    expr = definition.expr
-    inst = expr.inst
-
-    # Create a dict m mapping each free typevar in the signature of definition
-    # to a fresh copy of itself.
-    free_formal_tvs = inst.all_typevars()
-    m = {tv: tv.get_fresh_copy(str(typ.get_uid())) for tv in free_formal_tvs}
-
-    # Update m with any explicitly bound type vars
-    for (idx, bound_typ) in enumerate(expr.typevars):
-        m[free_formal_tvs[idx]] = TypeVar.singleton(bound_typ)
-
-    # Get fresh copies for each typevar in the signature (both free and
-    # derived)
-    fresh_formal_tvs = \
-        [subst(inst.outs[i].typevar, m) for i in inst.value_results] +\
-        [subst(inst.ins[i].typevar, m) for i in inst.value_opnums]
-
-    # Get the list of actual Vars
-    actual_vars = []  # type: List[Expr]
-    actual_vars += [definition.defs[i] for i in inst.value_results]
-    actual_vars += [expr.args[i] for i in inst.value_opnums]
-
-    # Get the list of the actual TypeVars
-    actual_tvs = []
-    for v in actual_vars:
-        assert(isinstance(v, Var))
-        # Register with TypeEnv that this typevar corresponds ot variable v,
-        # and thus has a given rank
-        typ.register(v)
-        actual_tvs.append(v.get_typevar())
-
-    # Make sure we unify the control typevar first.
-    if inst.is_polymorphic:
-        idx = fresh_formal_tvs.index(m[inst.ctrl_typevar])
-        fresh_formal_tvs = move_first(fresh_formal_tvs, idx)
-        actual_tvs = move_first(actual_tvs, idx)
-
-    # Unify each actual typevar with the corresponding fresh formal tv
-    for (actual_tv, formal_tv) in zip(actual_tvs, fresh_formal_tvs):
-        typ_or_err = unify(actual_tv, formal_tv, typ)
-        err = get_error(typ_or_err)
-        if (err):
-            return "fail ti on {} <: {}: ".format(actual_tv, formal_tv) + err
-
-        typ = get_type_env(typ_or_err)
-
-    # Add any instruction specific constraints
-    for constr in inst.constraints:
-        typ.add_constraint(constr.translate(m))
-
-    return typ
-
-
-def ti_rtl(rtl, typ):
-    # type: (Rtl, TypeEnv) -> TypingOrError
-    """
-    Perform type inference on an Rtl in a starting type env typ. Return an
-    updated type environment or error.
-    """
-    for (i, d) in enumerate(rtl.rtl):
-        assert (isinstance(d, Def))
-        typ_or_err = ti_def(d, typ)
-        err = get_error(typ_or_err)  # type: Optional[TypingError]
-        if (err):
-            return "On line {}: ".format(i) + err
-
-        typ = get_type_env(typ_or_err)
-
-    return typ
-
-
-def ti_xform(xform, typ):
-    # type: (XForm, TypeEnv) -> TypingOrError
-    """
-    Perform type inference on an Rtl in a starting type env typ. Return an
-    updated type environment or error.
-    """
-    typ_or_err = ti_rtl(xform.src, typ)
-    err = get_error(typ_or_err)  # type: Optional[TypingError]
-    if (err):
-        return "In src pattern: " + err
-
-    typ = get_type_env(typ_or_err)
-
-    typ_or_err = ti_rtl(xform.dst, typ)
-    err = get_error(typ_or_err)
-    if (err):
-        return "In dst pattern: " + err
-
-    typ = get_type_env(typ_or_err)
-
-    return get_type_env(typ_or_err)
diff --git a/cranelift/codegen/meta-python/cdsl/types.py b/cranelift/codegen/meta-python/cdsl/types.py
deleted file mode 100644
index 26777152da..0000000000
--- a/cranelift/codegen/meta-python/cdsl/types.py
+++ /dev/null
@@ -1,348 +0,0 @@
-"""Cranelift ValueType hierarchy"""
-from __future__ import absolute_import
-import math
-
-try:
-    from typing import Dict, List, cast, TYPE_CHECKING # noqa
-except ImportError:
-    TYPE_CHECKING = False
-    pass
-
-
-# Numbering scheme for value types:
-#
-# 0: Void
-# 0x01-0x6f: Special types
-# 0x70-0x7f: Lane types
-# 0x80-0xff: Vector types
-#
-# Vector types are encoded with the lane type in the low 4 bits and log2(lanes)
-# in the high 4 bits, giving a range of 2-256 lanes.
-LANE_BASE = 0x70
-
-
-# ValueType instances (i8, i32, ...) are provided in the `base.types` module.
-class ValueType(object):
-    """
-    A concrete SSA value type.
-
-    All SSA values have a type that is described by an instance of `ValueType`
-    or one of its subclasses.
-    """
-
-    # Map name -> ValueType.
-    _registry = dict()  # type: Dict[str, ValueType]
-
-    # List of all the lane types.
-    all_lane_types = list()  # type: List[LaneType]
-
-    # List of all the special types (neither lanes nor vectors).
-    all_special_types = list()  # type: List[SpecialType]
-
-    def __init__(self, name, membytes, doc):
-        # type: (str, int, str) -> None
-        self.name = name
-        self.number = None  # type: int
-        self.membytes = membytes
-        self.__doc__ = doc
-        assert name not in ValueType._registry
-        ValueType._registry[name] = self
-
-    def __str__(self):
-        # type: () -> str
-        return self.name
-
-    def rust_name(self):
-        # type: () -> str
-        return 'ir::types::' + self.name.upper()
-
-    @staticmethod
-    def by_name(name):
-        # type: (str) -> ValueType
-        if name in ValueType._registry:
-            return ValueType._registry[name]
-        else:
-            raise AttributeError("No type named '{}'".format(name))
-
-    def lane_bits(self):
-        # type: () -> int
-        """Return the number of bits in a lane."""
-        assert False, "Abstract"
-
-    def lane_count(self):
-        # type: () -> int
-        """Return the number of lanes."""
-        assert False, "Abstract"
-
-    def width(self):
-        # type: () -> int
-        """Return the total number of bits of an instance of this type."""
-        return self.lane_count() * self.lane_bits()
-
-    def wider_or_equal(self, other):
-        # type: (ValueType) -> bool
-        """
-        Return true iff:
-            1. self and other have equal number of lanes
-            2. each lane in self has at least as many bits as a lane in other
-        """
-        return (self.lane_count() == other.lane_count() and
-                self.lane_bits() >= other.lane_bits())
-
-
-class LaneType(ValueType):
-    """
-    A concrete scalar type that can appear as a vector lane too.
-
-    Also tracks a unique set of :py:class:`VectorType` instances with this type
-    as the lane type.
-    """
-
-    def __init__(self, name, membytes, doc):
-        # type: (str, int, str) -> None
-        super(LaneType, self).__init__(name, membytes, doc)
-        self._vectors = dict()  # type: Dict[int, VectorType]
-        # Assign numbers starting from LANE_BASE.
-        n = len(ValueType.all_lane_types)
-        ValueType.all_lane_types.append(self)
-        assert n < 16, 'Too many lane types'
-        self.number = LANE_BASE + n
-
-    def __repr__(self):
-        # type: () -> str
-        return 'LaneType({})'.format(self.name)
-
-    def by(self, lanes):
-        # type: (int) -> VectorType
-        """
-        Get a vector type with this type as the lane type.
-
-        For example, ``i32.by(4)`` returns the :obj:`i32x4` type.
-        """
-        if lanes in self._vectors:
-            return self._vectors[lanes]
-        else:
-            v = VectorType(self, lanes)
-            self._vectors[lanes] = v
-            return v
-
-    def lane_count(self):
-        # type: () -> int
-        """Return the number of lanes."""
-        return 1
-
-
-class VectorType(ValueType):
-    """
-    A concrete SIMD vector type.
-
-    A vector type has a lane type which is an instance of :class:`LaneType`,
-    and a positive number of lanes.
-    """
-
-    def __init__(self, base, lanes):
-        # type: (LaneType, int) -> None
-        super(VectorType, self).__init__(
-                name='{}x{}'.format(base.name, lanes),
-                membytes=lanes*base.membytes,
-                doc="""
-                A SIMD vector with {} lanes containing a `{}` each.
-                """.format(lanes, base.name))
-        assert lanes <= 256, "Too many lanes"
-        self.base = base
-        self.lanes = lanes
-        self.number = 16*int(math.log(lanes, 2)) + base.number
-
-    def __repr__(self):
-        # type: () -> str
-        return ('VectorType(base={}, lanes={})'
-                .format(self.base.name, self.lanes))
-
-    def lane_count(self):
-        # type: () -> int
-        """Return the number of lanes."""
-        return self.lanes
-
-    def lane_bits(self):
-        # type: () -> int
-        """Return the number of bits in a lane."""
-        return self.base.lane_bits()
-
-
-class SpecialType(ValueType):
-    """
-    A concrete scalar type that is neither a vector nor a lane type.
-
-    Special types cannot be used to form vectors.
-    """
-
-    def __init__(self, name, membytes, doc):
-        # type: (str, int, str) -> None
-        super(SpecialType, self).__init__(name, membytes, doc)
-        # Assign numbers starting from 1. (0 is INVALID)
-        ValueType.all_special_types.append(self)
-        self.number = len(ValueType.all_special_types)
-        assert self.number < LANE_BASE, 'Too many special types'
-
-    def __repr__(self):
-        # type: () -> str
-        return 'SpecialType({})'.format(self.name)
-
-    def lane_count(self):
-        # type: () -> int
-        """Return the number of lanes."""
-        return 1
-
-
-class IntType(LaneType):
-    """A concrete scalar integer type."""
-
-    def __init__(self, bits):
-        # type: (int) -> None
-        assert bits > 0, 'IntType must have positive number of bits'
-        warning = ""
-        if bits < 32:
-            warning += "\nWARNING: "
-            warning += "arithmetic on {}bit integers is incomplete".format(
-                bits)
-        super(IntType, self).__init__(
-                name='i{:d}'.format(bits),
-                membytes=bits // 8,
-                doc="An integer type with {} bits.{}".format(bits, warning))
-        self.bits = bits
-
-    def __repr__(self):
-        # type: () -> str
-        return 'IntType(bits={})'.format(self.bits)
-
-    @staticmethod
-    def with_bits(bits):
-        # type: (int) -> IntType
-        typ = ValueType.by_name('i{:d}'.format(bits))
-        if TYPE_CHECKING:
-            return cast(IntType, typ)
-        else:
-            return typ
-
-    def lane_bits(self):
-        # type: () -> int
-        """Return the number of bits in a lane."""
-        return self.bits
-
-
-class FloatType(LaneType):
-    """A concrete scalar floating point type."""
-
-    def __init__(self, bits, doc):
-        # type: (int, str) -> None
-        assert bits > 0, 'FloatType must have positive number of bits'
-        super(FloatType, self).__init__(
-                name='f{:d}'.format(bits),
-                membytes=bits // 8,
-                doc=doc)
-        self.bits = bits
-
-    def __repr__(self):
-        # type: () -> str
-        return 'FloatType(bits={})'.format(self.bits)
-
-    @staticmethod
-    def with_bits(bits):
-        # type: (int) -> FloatType
-        typ = ValueType.by_name('f{:d}'.format(bits))
-        if TYPE_CHECKING:
-            return cast(FloatType, typ)
-        else:
-            return typ
-
-    def lane_bits(self):
-        # type: () -> int
-        """Return the number of bits in a lane."""
-        return self.bits
-
-
-class BoolType(LaneType):
-    """A concrete scalar boolean type."""
-
-    def __init__(self, bits):
-        # type: (int) -> None
-        assert bits > 0, 'BoolType must have positive number of bits'
-        super(BoolType, self).__init__(
-                name='b{:d}'.format(bits),
-                membytes=bits // 8,
-                doc="A boolean type with {} bits.".format(bits))
-        self.bits = bits
-
-    def __repr__(self):
-        # type: () -> str
-        return 'BoolType(bits={})'.format(self.bits)
-
-    @staticmethod
-    def with_bits(bits):
-        # type: (int) -> BoolType
-        typ = ValueType.by_name('b{:d}'.format(bits))
-        if TYPE_CHECKING:
-            return cast(BoolType, typ)
-        else:
-            return typ
-
-    def lane_bits(self):
-        # type: () -> int
-        """Return the number of bits in a lane."""
-        return self.bits
-
-
-class FlagsType(SpecialType):
-    """
-    A type representing CPU flags.
-
-    Flags can't be stored in memory.
-    """
-
-    def __init__(self, name, doc):
-        # type: (str, str) -> None
-        super(FlagsType, self).__init__(name, 0, doc)
-
-    def __repr__(self):
-        # type: () -> str
-        return 'FlagsType({})'.format(self.name)
-
-
-class BVType(ValueType):
-    """A flat bitvector type. Used for semantics description only."""
-
-    def __init__(self, bits):
-        # type: (int) -> None
-        assert bits > 0, 'Must have positive number of bits'
-        super(BVType, self).__init__(
-                name='bv{:d}'.format(bits),
-                membytes=bits // 8,
-                doc="A bitvector type with {} bits.".format(bits))
-        self.bits = bits
-
-    def __repr__(self):
-        # type: () -> str
-        return 'BVType(bits={})'.format(self.bits)
-
-    @staticmethod
-    def with_bits(bits):
-        # type: (int) -> BVType
-        name = 'bv{:d}'.format(bits)
-        if name not in ValueType._registry:
-            return BVType(bits)
-
-        typ = ValueType.by_name(name)
-        if TYPE_CHECKING:
-            return cast(BVType, typ)
-        else:
-            return typ
-
-    def lane_bits(self):
-        # type: () -> int
-        """Return the number of bits in a lane."""
-        return self.bits
-
-    def lane_count(self):
-        # type: () -> int
-        """Return the number of lane. For BVtypes always 1."""
-        return 1
diff --git a/cranelift/codegen/meta-python/cdsl/typevar.py b/cranelift/codegen/meta-python/cdsl/typevar.py
deleted file mode 100644
index 9d2dace044..0000000000
--- a/cranelift/codegen/meta-python/cdsl/typevar.py
+++ /dev/null
@@ -1,906 +0,0 @@
-"""
-Type variables for Parametric polymorphism.
-
-Cranelift instructions and instruction transformations can be specified to be
-polymorphic by using type variables.
-"""
-from __future__ import absolute_import
-import math
-from . import types, is_power_of_two
-from copy import copy
-
-try:
-    from typing import Tuple, Union, Iterable, Any, Set, TYPE_CHECKING # noqa
-    if TYPE_CHECKING:
-        from srcgen import Formatter  # noqa
-        Interval = Tuple[int, int]
-        # An Interval where `True` means 'everything'
-        BoolInterval = Union[bool, Interval]
-        # Set of special types: None, False, True, or iterable.
-        SpecialSpec = Union[bool, Iterable[types.SpecialType]]
-except ImportError:
-    pass
-
-MAX_LANES = 256
-MAX_BITS = 64
-MAX_BITVEC = MAX_BITS * MAX_LANES
-
-
-def int_log2(x):
-    # type: (int) -> int
-    return int(math.log(x, 2))
-
-
-def intersect(a, b):
-    # type: (Interval, Interval) -> Interval
-    """
-    Given two `(min, max)` inclusive intervals, compute their intersection.
-
-    Use `(None, None)` to represent the empty interval on input and output.
-    """
-    if a[0] is None or b[0] is None:
-        return (None, None)
-    lo = max(a[0], b[0])
-    assert lo is not None
-    hi = min(a[1], b[1])
-    assert hi is not None
-    if lo <= hi:
-        return (lo, hi)
-    else:
-        return (None, None)
-
-
-def is_empty(intv):
-    # type: (Interval) -> bool
-    return intv is None or intv is False or intv == (None, None)
-
-
-def encode_bitset(vals, size):
-    # type: (Iterable[int], int) -> int
-    """
-    Encode a set of values (each between 0 and size) as a bitset of width size.
-    """
-    res = 0
-    assert is_power_of_two(size) and size <= 64
-    for v in vals:
-        assert 0 <= v and v < size
-        res |= 1 << v
-    return res
-
-
-def pp_set(s):
-    # type: (Iterable[Any]) -> str
-    """
-    Return a consistent string representation of a set (ordering is fixed)
-    """
-    return '{' + ', '.join([repr(x) for x in sorted(s)]) + '}'
-
-
-def decode_interval(intv, full_range, default=None):
-    # type: (BoolInterval, Interval, int) -> Interval
-    """
-    Decode an interval specification which can take the following values:
-
-    True
-        Use the `full_range`.
-    `False` or `None`
-        An empty interval
-    (lo, hi)
-        An explicit interval
-    """
-    if isinstance(intv, tuple):
-        # mypy bug here: 'builtins.None' object is not iterable
-        lo, hi = intv
-        assert is_power_of_two(lo)
-        assert is_power_of_two(hi)
-        assert lo <= hi
-        assert lo >= full_range[0]
-        assert hi <= full_range[1]
-        return intv
-
-    if intv:
-        return full_range
-    else:
-        return (default, default)
-
-
-def interval_to_set(intv):
-    # type: (Interval) -> Set
-    if is_empty(intv):
-        return set()
-
-    (lo, hi) = intv
-    assert is_power_of_two(lo)
-    assert is_power_of_two(hi)
-    assert lo <= hi
-    return set([2**i for i in range(int_log2(lo), int_log2(hi)+1)])
-
-
-def legal_bool(bits):
-    # type: (int) -> bool
-    """
-    True iff bits is a legal bit width for a bool type.
-    bits == 1 || bits \\in { 8, 16, .. MAX_BITS }
-    """
-    return bits == 1 or \
-        (bits >= 8 and bits <= MAX_BITS and is_power_of_two(bits))
-
-
-class TypeSet(object):
-    """
-    A set of types.
-
-    We don't allow arbitrary subsets of types, but use a parametrized approach
-    instead.
-
-    Objects of this class can be used as dictionary keys.
-
-    Parametrized type sets are specified in terms of ranges:
-
-    - The permitted range of vector lanes, where 1 indicates a scalar type.
-    - The permitted range of integer types.
-    - The permitted range of floating point types, and
-    - The permitted range of boolean types.
-
-    The ranges are inclusive from smallest bit-width to largest bit-width.
-
-    A typeset representing scalar integer types `i8` through `i32`:
-
-    >>> TypeSet(ints=(8, 32))
-    TypeSet(lanes={1}, ints={8, 16, 32})
-
-    Passing `True` instead of a range selects all available scalar types:
-
-    >>> TypeSet(ints=True)
-    TypeSet(lanes={1}, ints={8, 16, 32, 64})
-    >>> TypeSet(floats=True)
-    TypeSet(lanes={1}, floats={32, 64})
-    >>> TypeSet(bools=True)
-    TypeSet(lanes={1}, bools={1, 8, 16, 32, 64})
-
-    Similarly, passing `True` for the lanes selects all possible scalar and
-    vector types:
-
-    >>> TypeSet(lanes=True, ints=True)
-    TypeSet(lanes={1, 2, 4, 8, 16, 32, 64, 128, 256}, ints={8, 16, 32, 64})
-
-    Finally, a type set can contain special types (derived from `SpecialType`)
-    which can't appear as lane types.
-
-    :param lanes: `(min, max)` inclusive range of permitted vector lane counts.
-    :param ints: `(min, max)` inclusive range of permitted scalar integer
-                 widths.
-    :param floats: `(min, max)` inclusive range of permitted scalar floating
-                   point widths.
-    :param bools: `(min, max)` inclusive range of permitted scalar boolean
-                  widths.
-    :param bitvecs : `(min, max)` inclusive range of permitted bitvector
-                  widths.
-    :param specials: Sequence of special types to appear in the set.
-    """
-
-    def __init__(
-            self,
-            lanes=None,     # type: BoolInterval
-            ints=None,      # type: BoolInterval
-            floats=None,    # type: BoolInterval
-            bools=None,     # type: BoolInterval
-            bitvecs=None,   # type: BoolInterval
-            specials=None   # type: SpecialSpec
-            ):
-        # type: (...) -> None
-        self.lanes = interval_to_set(decode_interval(lanes, (1, MAX_LANES), 1))
-        self.ints = interval_to_set(decode_interval(ints, (8, MAX_BITS)))
-        self.floats = interval_to_set(decode_interval(floats, (32, 64)))
-        self.bools = interval_to_set(decode_interval(bools, (1, MAX_BITS)))
-        self.bools = set(filter(legal_bool, self.bools))
-        self.bitvecs = interval_to_set(decode_interval(bitvecs,
-                                                       (1, MAX_BITVEC)))
-        # Allow specials=None, specials=True, specials=(...)
-        self.specials = set()  # type: Set[types.SpecialType]
-        if isinstance(specials, bool):
-            if specials:
-                self.specials = set(types.ValueType.all_special_types)
-        elif specials:
-            self.specials = set(specials)
-
-    def copy(self):
-        # type: (TypeSet) -> TypeSet
-        """
-        Return a copy of our self.
-        """
-        n = TypeSet()
-        n.lanes = copy(self.lanes)
-        n.ints = copy(self.ints)
-        n.floats = copy(self.floats)
-        n.bools = copy(self.bools)
-        n.bitvecs = copy(self.bitvecs)
-        n.specials = copy(self.specials)
-        return n
-
-    def typeset_key(self):
-        # type: () -> Tuple[Tuple, Tuple, Tuple, Tuple, Tuple, Tuple]
-        """Key tuple used for hashing and equality."""
-        return (tuple(sorted(list(self.lanes))),
-                tuple(sorted(list(self.ints))),
-                tuple(sorted(list(self.floats))),
-                tuple(sorted(list(self.bools))),
-                tuple(sorted(list(self.bitvecs))),
-                tuple(sorted(s.name for s in self.specials)))
-
-    def __hash__(self):
-        # type: () -> int
-        h = hash(self.typeset_key())
-        assert h == getattr(self, 'prev_hash', h), "TypeSet changed"
-        self.prev_hash = h
-        return h
-
-    def __eq__(self, other):
-        # type: (object) -> bool
-        if isinstance(other, TypeSet):
-            return self.typeset_key() == other.typeset_key()
-        else:
-            return False
-
-    def __ne__(self, other):
-        # type: (object) -> bool
-        return not self.__eq__(other)
-
-    def __repr__(self):
-        # type: () -> str
-        s = 'TypeSet(lanes={}'.format(pp_set(self.lanes))
-        if len(self.ints) > 0:
-            s += ', ints={}'.format(pp_set(self.ints))
-        if len(self.floats) > 0:
-            s += ', floats={}'.format(pp_set(self.floats))
-        if len(self.bools) > 0:
-            s += ', bools={}'.format(pp_set(self.bools))
-        if len(self.bitvecs) > 0:
-            s += ', bitvecs={}'.format(pp_set(self.bitvecs))
-        if len(self.specials) > 0:
-            s += ', specials=[{}]'.format(pp_set(self.specials))
-        return s + ')'
-
-    def emit_fields(self, fmt):
-        # type: (Formatter) -> None
-        """Emit field initializers for this typeset."""
-        assert len(self.bitvecs) == 0, "Bitvector types are not emitable."
-        fmt.comment(repr(self))
-
-        fields = (('lanes', 16),
-                  ('ints', 8),
-                  ('floats', 8),
-                  ('bools', 8))
-
-        for (field, bits) in fields:
-            vals = [int_log2(x) for x in getattr(self, field)]
-            fmt.line('{}: BitSet::({}),'
-                     .format(field, bits, encode_bitset(vals, bits)))
-
-    def __iand__(self, other):
-        # type: (TypeSet) -> TypeSet
-        """
-        Intersect self with other type set.
-
-        >>> a = TypeSet(lanes=True, ints=(16, 32))
-        >>> a
-        TypeSet(lanes={1, 2, 4, 8, 16, 32, 64, 128, 256}, ints={16, 32})
-        >>> b = TypeSet(lanes=(4, 16), ints=True)
-        >>> a &= b
-        >>> a
-        TypeSet(lanes={4, 8, 16}, ints={16, 32})
-
-        >>> a = TypeSet(lanes=True, bools=(1, 8))
-        >>> b = TypeSet(lanes=True, bools=(16, 32))
-        >>> a &= b
-        >>> a
-        TypeSet(lanes={1, 2, 4, 8, 16, 32, 64, 128, 256})
-        """
-        self.lanes.intersection_update(other.lanes)
-        self.ints.intersection_update(other.ints)
-        self.floats.intersection_update(other.floats)
-        self.bools.intersection_update(other.bools)
-        self.bitvecs.intersection_update(other.bitvecs)
-        self.specials.intersection_update(other.specials)
-
-        return self
-
-    def issubset(self, other):
-        # type: (TypeSet) -> bool
-        """
-        Return true iff self is a subset of other
-        """
-        return self.lanes.issubset(other.lanes) and \
-            self.ints.issubset(other.ints) and \
-            self.floats.issubset(other.floats) and \
-            self.bools.issubset(other.bools) and \
-            self.bitvecs.issubset(other.bitvecs) and \
-            self.specials.issubset(other.specials)
-
-    def lane_of(self):
-        # type: () -> TypeSet
-        """
-        Return a TypeSet describing the image of self across lane_of
-        """
-        new = self.copy()
-        new.lanes = set([1])
-        new.bitvecs = set()
-        return new
-
-    def as_bool(self):
-        # type: () -> TypeSet
-        """
-        Return a TypeSet describing the image of self across as_bool
-        """
-        new = self.copy()
-        new.ints = set()
-        new.floats = set()
-        new.bitvecs = set()
-
-        if len(self.lanes.difference(set([1]))) > 0:
-            new.bools = self.ints.union(self.floats).union(self.bools)
-
-        if 1 in self.lanes:
-            new.bools.add(1)
-        return new
-
-    def half_width(self):
-        # type: () -> TypeSet
-        """
-        Return a TypeSet describing the image of self across halfwidth
-        """
-        new = self.copy()
-        new.ints = set([x//2 for x in self.ints if x > 8])
-        new.floats = set([x//2 for x in self.floats if x > 32])
-        new.bools = set([x//2 for x in self.bools if x > 8])
-        new.bitvecs = set([x//2 for x in self.bitvecs if x > 1])
-        new.specials = set()
-
-        return new
-
-    def double_width(self):
-        # type: () -> TypeSet
-        """
-        Return a TypeSet describing the image of self across doublewidth
-        """
-        new = self.copy()
-        new.ints = set([x*2 for x in self.ints if x < MAX_BITS])
-        new.floats = set([x*2 for x in self.floats if x < MAX_BITS])
-        new.bools = set(filter(legal_bool,
-                               set([x*2 for x in self.bools if x < MAX_BITS])))
-        new.bitvecs = set([x*2 for x in self.bitvecs if x < MAX_BITVEC])
-        new.specials = set()
-
-        return new
-
-    def half_vector(self):
-        # type: () -> TypeSet
-        """
-        Return a TypeSet describing the image of self across halfvector
-        """
-        new = self.copy()
-        new.bitvecs = set()
-        new.lanes = set([x//2 for x in self.lanes if x > 1])
-        new.specials = set()
-
-        return new
-
-    def double_vector(self):
-        # type: () -> TypeSet
-        """
-        Return a TypeSet describing the image of self across doublevector
-        """
-        new = self.copy()
-        new.bitvecs = set()
-        new.lanes = set([x*2 for x in self.lanes if x < MAX_LANES])
-        new.specials = set()
-
-        return new
-
-    def to_bitvec(self):
-        # type: () -> TypeSet
-        """
-        Return a TypeSet describing the image of self across to_bitvec
-        """
-        assert len(self.bitvecs) == 0
-        all_scalars = self.ints.union(self.floats.union(self.bools))
-
-        new = self.copy()
-        new.lanes = set([1])
-        new.ints = set()
-        new.bools = set()
-        new.floats = set()
-        new.bitvecs = set([lane_w * nlanes for lane_w in all_scalars
-                           for nlanes in self.lanes])
-        new.specials = set()
-
-        return new
-
-    def image(self, func):
-        # type: (str) -> TypeSet
-        """
-        Return the image of self across the derived function func
-        """
-        if (func == TypeVar.LANEOF):
-            return self.lane_of()
-        elif (func == TypeVar.ASBOOL):
-            return self.as_bool()
-        elif (func == TypeVar.HALFWIDTH):
-            return self.half_width()
-        elif (func == TypeVar.DOUBLEWIDTH):
-            return self.double_width()
-        elif (func == TypeVar.HALFVECTOR):
-            return self.half_vector()
-        elif (func == TypeVar.DOUBLEVECTOR):
-            return self.double_vector()
-        elif (func == TypeVar.TOBITVEC):
-            return self.to_bitvec()
-        else:
-            assert False, "Unknown derived function: " + func
-
-    def preimage(self, func):
-        # type: (str) -> TypeSet
-        """
-        Return the inverse image of self across the derived function func
-        """
-        # The inverse of the empty set is always empty
-        if (self.size() == 0):
-            return self
-
-        if (func == TypeVar.LANEOF):
-            new = self.copy()
-            new.bitvecs = set()
-            new.lanes = set([2**i for i in range(0, int_log2(MAX_LANES)+1)])
-            return new
-        elif (func == TypeVar.ASBOOL):
-            new = self.copy()
-            new.bitvecs = set()
-
-            if 1 not in self.bools:
-                new.ints = self.bools.difference(set([1]))
-                new.floats = self.bools.intersection(set([32, 64]))
-                # If b1 is not in our typeset, than lanes=1 cannot be in the
-                # pre-image, as as_bool() of scalars is always b1.
-                new.lanes = self.lanes.difference(set([1]))
-            else:
-                new.ints = set([2**x for x in range(3, 7)])
-                new.floats = set([32, 64])
-
-            return new
-        elif (func == TypeVar.HALFWIDTH):
-            return self.double_width()
-        elif (func == TypeVar.DOUBLEWIDTH):
-            return self.half_width()
-        elif (func == TypeVar.HALFVECTOR):
-            return self.double_vector()
-        elif (func == TypeVar.DOUBLEVECTOR):
-            return self.half_vector()
-        elif (func == TypeVar.TOBITVEC):
-            new = TypeSet()
-
-            # Start with all possible lanes/ints/floats/bools
-            lanes = interval_to_set(decode_interval(True, (1, MAX_LANES), 1))
-            ints = interval_to_set(decode_interval(True, (8, MAX_BITS)))
-            floats = interval_to_set(decode_interval(True, (32, 64)))
-            bools = interval_to_set(decode_interval(True, (1, MAX_BITS)))
-
-            # See which combinations have a size that appears in self.bitvecs
-            has_t = set()  # type: Set[Tuple[str, int, int]]
-            for l in lanes:
-                for i in ints:
-                    if i * l in self.bitvecs:
-                        has_t.add(('i', i, l))
-                for i in bools:
-                    if i * l in self.bitvecs:
-                        has_t.add(('b', i, l))
-                for i in floats:
-                    if i * l in self.bitvecs:
-                        has_t.add(('f', i, l))
-
-            for (t, width, lane) in has_t:
-                new.lanes.add(lane)
-                if (t == 'i'):
-                    new.ints.add(width)
-                elif (t == 'b'):
-                    new.bools.add(width)
-                else:
-                    assert t == 'f'
-                    new.floats.add(width)
-
-            return new
-        else:
-            assert False, "Unknown derived function: " + func
-
-    def size(self):
-        # type: () -> int
-        """
-        Return the number of concrete types represented by this typeset
-        """
-        return (len(self.lanes) * (len(self.ints) + len(self.floats) +
-                                   len(self.bools) + len(self.bitvecs)) +
-                len(self.specials))
-
-    def concrete_types(self):
-        # type: () -> Iterable[types.ValueType]
-        def by(scalar, lanes):
-            # type: (types.LaneType, int) -> types.ValueType
-            if (lanes == 1):
-                return scalar
-            else:
-                return scalar.by(lanes)
-
-        for nlanes in self.lanes:
-            for bits in self.ints:
-                yield by(types.IntType.with_bits(bits), nlanes)
-            for bits in self.floats:
-                yield by(types.FloatType.with_bits(bits), nlanes)
-            for bits in self.bools:
-                yield by(types.BoolType.with_bits(bits), nlanes)
-            for bits in self.bitvecs:
-                assert nlanes == 1
-                yield types.BVType.with_bits(bits)
-
-        for spec in self.specials:
-            yield spec
-
-    def get_singleton(self):
-        # type: () -> types.ValueType
-        """
-        Return the singleton type represented by self. Can only call on
-        typesets containing 1 type.
-        """
-        types = list(self.concrete_types())
-        assert len(types) == 1
-        return types[0]
-
-    def widths(self):
-        # type: () -> Set[int]
-        """ Return a set of the widths of all possible types in self"""
-        scalar_w = self.ints.union(self.floats.union(self.bools))
-        scalar_w = scalar_w.union(self.bitvecs)
-        return set(w * l for l in self.lanes for w in scalar_w)
-
-
-class TypeVar(object):
-    """
-    Type variables can be used in place of concrete types when defining
-    instructions. This makes the instructions *polymorphic*.
-
-    A type variable is restricted to vary over a subset of the value types.
-    This subset is specified by a set of flags that control the permitted base
-    types and whether the type variable can assume scalar or vector types, or
-    both.
-
-    :param name: Short name of type variable used in instruction descriptions.
-    :param doc: Documentation string.
-    :param ints: Allow all integer base types, or `(min, max)` bit-range.
-    :param floats: Allow all floating point base types, or `(min, max)`
-                   bit-range.
-    :param bools: Allow all boolean base types, or `(min, max)` bit-range.
-    :param scalars: Allow type variable to assume scalar types.
-    :param simd: Allow type variable to assume vector types, or `(min, max)`
-                 lane count range.
-    :param bitvecs: Allow all BitVec base types, or `(min, max)` bit-range.
-    """
-
-    def __init__(
-            self,
-            name,                   # type: str
-            doc,                    # type: str
-            ints=False,             # type: BoolInterval
-            floats=False,           # type: BoolInterval
-            bools=False,            # type: BoolInterval
-            scalars=True,           # type: bool
-            simd=False,             # type: BoolInterval
-            bitvecs=False,          # type: BoolInterval
-            base=None,              # type: TypeVar
-            derived_func=None,      # type: str
-            specials=None           # type: SpecialSpec
-            ):
-        # type: (...) -> None
-        self.name = name
-        self.__doc__ = doc
-        self.is_derived = isinstance(base, TypeVar)
-        if base:
-            assert self.is_derived
-            assert derived_func
-            self.base = base
-            self.derived_func = derived_func
-            self.name = '{}({})'.format(derived_func, base.name)
-        else:
-            min_lanes = 1 if scalars else 2
-            lanes = decode_interval(simd, (min_lanes, MAX_LANES), 1)
-            self.type_set = TypeSet(
-                    lanes=lanes,
-                    ints=ints,
-                    floats=floats,
-                    bools=bools,
-                    bitvecs=bitvecs,
-                    specials=specials)
-
-    @staticmethod
-    def singleton(typ):
-        # type: (types.ValueType) -> TypeVar
-        """Create a type variable that can only assume a single type."""
-        scalar = None  # type: types.ValueType
-        if isinstance(typ, types.VectorType):
-            scalar = typ.base
-            lanes = (typ.lanes, typ.lanes)
-        elif isinstance(typ, types.LaneType):
-            scalar = typ
-            lanes = (1, 1)
-        elif isinstance(typ, types.SpecialType):
-            return TypeVar(typ.name, typ.__doc__, specials=[typ])
-        else:
-            assert isinstance(typ, types.BVType)
-            scalar = typ
-            lanes = (1, 1)
-
-        ints = None
-        floats = None
-        bools = None
-        bitvecs = None
-
-        if isinstance(scalar, types.IntType):
-            ints = (scalar.bits, scalar.bits)
-        elif isinstance(scalar, types.FloatType):
-            floats = (scalar.bits, scalar.bits)
-        elif isinstance(scalar, types.BoolType):
-            bools = (scalar.bits, scalar.bits)
-        elif isinstance(scalar, types.BVType):
-            bitvecs = (scalar.bits, scalar.bits)
-
-        tv = TypeVar(
-                typ.name, typ.__doc__,
-                ints=ints, floats=floats, bools=bools,
-                bitvecs=bitvecs, simd=lanes)
-        return tv
-
-    def __str__(self):
-        # type: () -> str
-        return "`{}`".format(self.name)
-
-    def __repr__(self):
-        # type: () -> str
-        if self.is_derived:
-            return (
-                    'TypeVar({}, base={}, derived_func={})'
-                    .format(self.name, self.base, self.derived_func))
-        else:
-            return (
-                    'TypeVar({}, {})'
-                    .format(self.name, self.type_set))
-
-    def __hash__(self):
-        # type: () -> int
-        if (not self.is_derived):
-            return object.__hash__(self)
-
-        return hash((self.derived_func, self.base))
-
-    def __eq__(self, other):
-        # type: (object) -> bool
-        if not isinstance(other, TypeVar):
-            return False
-        if self.is_derived and other.is_derived:
-            return (
-                    self.derived_func == other.derived_func and
-                    self.base == other.base)
-        else:
-            return self is other
-
-    def __ne__(self, other):
-        # type: (object) -> bool
-        return not self.__eq__(other)
-
-    # Supported functions for derived type variables.
-    # The names here must match the method names on `ir::types::Type`.
-    # The camel_case of the names must match `enum OperandConstraint` in
-    # `instructions.rs`.
-    LANEOF = 'lane_of'
-    ASBOOL = 'as_bool'
-    HALFWIDTH = 'half_width'
-    DOUBLEWIDTH = 'double_width'
-    HALFVECTOR = 'half_vector'
-    DOUBLEVECTOR = 'double_vector'
-    TOBITVEC = 'to_bitvec'
-
-    @staticmethod
-    def is_bijection(func):
-        # type: (str) -> bool
-        return func in [
-            TypeVar.HALFWIDTH,
-            TypeVar.DOUBLEWIDTH,
-            TypeVar.HALFVECTOR,
-            TypeVar.DOUBLEVECTOR]
-
-    @staticmethod
-    def inverse_func(func):
-        # type: (str) -> str
-        return {
-            TypeVar.HALFWIDTH: TypeVar.DOUBLEWIDTH,
-            TypeVar.DOUBLEWIDTH: TypeVar.HALFWIDTH,
-            TypeVar.HALFVECTOR: TypeVar.DOUBLEVECTOR,
-            TypeVar.DOUBLEVECTOR: TypeVar.HALFVECTOR
-        }[func]
-
-    @staticmethod
-    def derived(base, derived_func):
-        # type: (TypeVar, str) -> TypeVar
-        """Create a type variable that is a function of another."""
-
-        # Safety checks to avoid over/underflows.
-        ts = base.get_typeset()
-
-        assert len(ts.specials) == 0, "Can't derive from special types"
-
-        if derived_func == TypeVar.HALFWIDTH:
-            if len(ts.ints) > 0:
-                assert min(ts.ints) > 8, "Can't halve all integer types"
-            if len(ts.floats) > 0:
-                assert min(ts.floats) > 32, "Can't halve all float types"
-            if len(ts.bools) > 0:
-                assert min(ts.bools) > 8, "Can't halve all boolean types"
-        elif derived_func == TypeVar.DOUBLEWIDTH:
-            if len(ts.ints) > 0:
-                assert max(ts.ints) < MAX_BITS,\
-                    "Can't double all integer types."
-            if len(ts.floats) > 0:
-                assert max(ts.floats) < MAX_BITS,\
-                    "Can't double all float types."
-            if len(ts.bools) > 0:
-                assert max(ts.bools) < MAX_BITS, "Can't double all bool types."
-        elif derived_func == TypeVar.HALFVECTOR:
-            assert min(ts.lanes) > 1, "Can't halve a scalar type"
-        elif derived_func == TypeVar.DOUBLEVECTOR:
-            assert max(ts.lanes) < MAX_LANES, "Can't double 256 lanes."
-
-        return TypeVar(None, None, base=base, derived_func=derived_func)
-
-    @staticmethod
-    def from_typeset(ts):
-        # type: (TypeSet) -> TypeVar
-        """ Create a type variable from a type set."""
-        tv = TypeVar(None, None)
-        tv.type_set = ts
-        return tv
-
-    def lane_of(self):
-        # type: () -> TypeVar
-        """
-        Return a derived type variable that is the scalar lane type of this
-        type variable.
-
-        When this type variable assumes a scalar type, the derived type will be
-        the same scalar type.
-        """
-        return TypeVar.derived(self, self.LANEOF)
-
-    def as_bool(self):
-        # type: () -> TypeVar
-        """
-        Return a derived type variable that has the same vector geometry as
-        this type variable, but with boolean lanes. Scalar types map to `b1`.
-        """
-        return TypeVar.derived(self, self.ASBOOL)
-
-    def half_width(self):
-        # type: () -> TypeVar
-        """
-        Return a derived type variable that has the same number of vector lanes
-        as this one, but the lanes are half the width.
-        """
-        return TypeVar.derived(self, self.HALFWIDTH)
-
-    def double_width(self):
-        # type: () -> TypeVar
-        """
-        Return a derived type variable that has the same number of vector lanes
-        as this one, but the lanes are double the width.
-        """
-        return TypeVar.derived(self, self.DOUBLEWIDTH)
-
-    def half_vector(self):
-        # type: () -> TypeVar
-        """
-        Return a derived type variable that has half the number of vector lanes
-        as this one, with the same lane type.
-        """
-        return TypeVar.derived(self, self.HALFVECTOR)
-
-    def double_vector(self):
-        # type: () -> TypeVar
-        """
-        Return a derived type variable that has twice the number of vector
-        lanes as this one, with the same lane type.
-        """
-        return TypeVar.derived(self, self.DOUBLEVECTOR)
-
-    def to_bitvec(self):
-        # type: () -> TypeVar
-        """
-        Return a derived type variable that represent a flat bitvector with
-        the same size as self
-        """
-        return TypeVar.derived(self, self.TOBITVEC)
-
-    def singleton_type(self):
-        # type: () -> types.ValueType
-        """
-        If the associated typeset has a single type return it. Otherwise return
-        None
-        """
-        ts = self.get_typeset()
-        if ts.size() != 1:
-            return None
-
-        return ts.get_singleton()
-
-    def free_typevar(self):
-        # type: () -> TypeVar
-        """
-        Get the free type variable controlling this one.
-        """
-        if self.is_derived:
-            return self.base.free_typevar()
-        elif self.singleton_type() is not None:
-            # A singleton type variable is not a proper free variable.
-            return None
-        else:
-            return self
-
-    def rust_expr(self):
-        # type: () -> str
-        """
-        Get a Rust expression that computes the type of this type variable.
-        """
-        if self.is_derived:
-            return '{}.{}()'.format(
-                    self.base.rust_expr(), self.derived_func)
-        elif self.singleton_type():
-            return self.singleton_type().rust_name()
-        else:
-            return self.name
-
-    def constrain_types_by_ts(self, ts):
-        # type: (TypeSet) -> None
-        """
-        Constrain the range of types this variable can assume to a subset of
-        those in the typeset ts.
-        """
-        if not self.is_derived:
-            self.type_set &= ts
-        else:
-            self.base.constrain_types_by_ts(ts.preimage(self.derived_func))
-
-    def constrain_types(self, other):
-        # type: (TypeVar) -> None
-        """
-        Constrain the range of types this variable can assume to a subset of
-        those `other` can assume.
-        """
-        if self is other:
-            return
-
-        self.constrain_types_by_ts(other.get_typeset())
-
-    def get_typeset(self):
-        # type: () -> TypeSet
-        """
-        Returns the typeset for this TV. If the TV is derived, computes it
-        recursively from the derived function and the base's typeset.
-        """
-        if not self.is_derived:
-            return self.type_set
-        else:
-            return self.base.get_typeset().image(self.derived_func)
-
-    def get_fresh_copy(self, name):
-        # type: (str) -> TypeVar
-        """
-        Get a fresh copy of self. Can only be called on free typevars.
-        """
-        assert not self.is_derived
-        tv = TypeVar.from_typeset(self.type_set.copy())
-        tv.name = name
-        return tv
diff --git a/cranelift/codegen/meta-python/cdsl/xform.py b/cranelift/codegen/meta-python/cdsl/xform.py
deleted file mode 100644
index 27aa515f94..0000000000
--- a/cranelift/codegen/meta-python/cdsl/xform.py
+++ /dev/null
@@ -1,423 +0,0 @@
-"""
-Instruction transformations.
-"""
-from __future__ import absolute_import
-from .ast import Def, Var, Apply
-from .ti import ti_xform, TypeEnv, get_type_env, TypeConstraint
-from collections import OrderedDict
-from functools import reduce
-
-try:
-    from typing import Union, Iterator, Sequence, Iterable, List, Dict  # noqa
-    from typing import Optional, Set # noqa
-    from .ast import Expr, VarAtomMap  # noqa
-    from .isa import TargetISA  # noqa
-    from .typevar import TypeVar  # noqa
-    from .instructions import ConstrList, Instruction # noqa
-    DefApply = Union[Def, Apply]
-except ImportError:
-    pass
-
-
-def canonicalize_defapply(node):
-    # type: (DefApply) -> Def
-    """
-    Canonicalize a `Def` or `Apply` node into a `Def`.
-
-    An `Apply` becomes a `Def` with an empty list of defs.
-    """
-    if isinstance(node, Apply):
-        return Def((), node)
-    else:
-        return node
-
-
-class Rtl(object):
-    """
-    Register Transfer Language list.
-
-    An RTL object contains a list of register assignments in the form of `Def`
-    objects.
-
-    An RTL list can represent both a source pattern to be matched, or a
-    destination pattern to be inserted.
-    """
-
-    def __init__(self, *args):
-        # type: (*DefApply) -> None
-        self.rtl = tuple(map(canonicalize_defapply, args))
-
-    def copy(self, m):
-        # type: (VarAtomMap) -> Rtl
-        """
-        Return a copy of this rtl with all Vars substituted with copies or
-        according to m. Update m as necessary.
-        """
-        return Rtl(*[d.copy(m) for d in self.rtl])
-
-    def vars(self):
-        # type: () -> Set[Var]
-        """Return the set of all Vars in self that correspond to SSA values"""
-        return reduce(lambda x, y:  x.union(y),
-                      [d.vars() for d in self.rtl],
-                      set([]))
-
-    def definitions(self):
-        # type: () -> Set[Var]
-        """ Return the set of all Vars defined in self"""
-        return reduce(lambda x, y:  x.union(y),
-                      [d.definitions() for d in self.rtl],
-                      set([]))
-
-    def free_vars(self):
-        # type: () -> Set[Var]
-        """Return the set of free Vars corresp. to SSA vals used in self"""
-        def flow_f(s, d):
-            # type: (Set[Var], Def) -> Set[Var]
-            """Compute the change in the set of free vars across a Def"""
-            s = s.difference(set(d.defs))
-            uses = set(d.expr.args[i] for i in d.expr.inst.value_opnums)
-            for v in uses:
-                assert isinstance(v, Var)
-                s.add(v)
-
-            return s
-
-        return reduce(flow_f, reversed(self.rtl), set([]))
-
-    def substitution(self, other, s):
-        # type: (Rtl, VarAtomMap) -> Optional[VarAtomMap]
-        """
-        If the Rtl self agrees structurally with the Rtl other, return a
-        substitution to transform self to other. Two Rtls agree structurally if
-        they have the same sequence of Defs, that agree structurally.
-        """
-        if len(self.rtl) != len(other.rtl):
-            return None
-
-        for i in range(len(self.rtl)):
-            s = self.rtl[i].substitution(other.rtl[i], s)
-
-            if s is None:
-                return None
-
-        return s
-
-    def is_concrete(self):
-        # type: (Rtl) -> bool
-        """Return True iff every Var in the self has a singleton type."""
-        return all(v.get_typevar().singleton_type() is not None
-                   for v in self.vars())
-
-    def cleanup_concrete_rtl(self):
-        # type: (Rtl) -> None
-        """
-        Given that there is only 1 possible concrete typing T for self, assign
-        a singleton TV with type t=T[v] for each Var v \\in self. Its an error
-        to call this on an Rtl with more than 1 possible typing. This modifies
-        the Rtl in-place.
-        """
-        from .ti import ti_rtl, TypeEnv
-        # 1) Infer the types of all vars in res
-        typenv = get_type_env(ti_rtl(self, TypeEnv()))
-        typenv.normalize()
-        typenv = typenv.extract()
-
-        # 2) Make sure there is only one possible type assignment
-        typings = list(typenv.concrete_typings())
-        assert len(typings) == 1
-        typing = typings[0]
-
-        # 3) Assign the only possible type to each variable.
-        for v in typenv.vars:
-            assert typing[v].singleton_type() is not None
-            v.set_typevar(typing[v])
-
-    def __str__(self):
-        # type: () -> str
-        return "\n".join(map(str, self.rtl))
-
-
-class XForm(object):
-    """
-    An instruction transformation consists of a source and destination pattern.
-
-    Patterns are expressed in *register transfer language* as tuples of
-    `ast.Def` or `ast.Expr` nodes. A pattern may optionally have a sequence of
-    TypeConstraints, that additionally limit the set of cases when it applies.
-
-    A legalization pattern must have a source pattern containing only a single
-    instruction.
-
-    >>> from base.instructions import iconst, iadd, iadd_imm
-    >>> a = Var('a')
-    >>> c = Var('c')
-    >>> v = Var('v')
-    >>> x = Var('x')
-    >>> XForm(
-    ...     Rtl(c << iconst(v),
-    ...         a << iadd(x, c)),
-    ...     Rtl(a << iadd_imm(x, v)))
-    XForm(inputs=[Var(v), Var(x)], defs=[Var(c, src), Var(a, src, dst)],
-      c << iconst(v)
-      a << iadd(x, c)
-    =>
-      a << iadd_imm(x, v)
-    )
-    """
-
-    def __init__(self, src, dst, constraints=None):
-        # type: (Rtl, Rtl, Optional[ConstrList]) -> None
-        self.src = src
-        self.dst = dst
-        # Variables that are inputs to the source pattern.
-        self.inputs = list()  # type: List[Var]
-        # Variables defined in either src or dst.
-        self.defs = list()  # type: List[Var]
-
-        # Rewrite variables in src and dst RTL lists to our own copies.
-        # Map name -> private Var.
-        symtab = dict()  # type: Dict[str, Var]
-        self._rewrite_rtl(src, symtab, Var.SRCCTX)
-        num_src_inputs = len(self.inputs)
-        self._rewrite_rtl(dst, symtab, Var.DSTCTX)
-        # Needed for testing type inference on XForms
-        self.symtab = symtab
-
-        # Check for inconsistently used inputs.
-        for i in self.inputs:
-            if not i.is_input():
-                raise AssertionError(
-                        "'{}' used as both input and def".format(i))
-
-        # Check for spurious inputs in dst.
-        if len(self.inputs) > num_src_inputs:
-            raise AssertionError(
-                    "extra inputs in dst RTL: {}".format(
-                        self.inputs[num_src_inputs:]))
-
-        # Perform type inference and cleanup
-        raw_ti = get_type_env(ti_xform(self, TypeEnv()))
-        raw_ti.normalize()
-        self.ti = raw_ti.extract()
-
-        def interp_tv(tv):
-            # type: (TypeVar) -> TypeVar
-            """ Convert typevars according to symtab """
-            if not tv.name.startswith("typeof_"):
-                return tv
-            return symtab[tv.name[len("typeof_"):]].get_typevar()
-
-        self.constraints = []  # type: List[TypeConstraint]
-        if constraints is not None:
-            if isinstance(constraints, TypeConstraint):
-                constr_list = [constraints]  # type: Sequence[TypeConstraint]
-            else:
-                constr_list = constraints
-
-            for c in constr_list:
-                type_m = {tv: interp_tv(tv) for tv in c.tvs()}
-                inner_c = c.translate(type_m)
-                self.constraints.append(inner_c)
-                self.ti.add_constraint(inner_c)
-
-        # Sanity: The set of inferred free typevars should be a subset of the
-        # TVs corresponding to Vars appearing in src
-        free_typevars = set(self.ti.free_typevars())
-        src_vars = set(self.inputs).union(
-            [x for x in self.defs if not x.is_temp()])
-        src_tvs = set([v.get_typevar() for v in src_vars])
-        if (not free_typevars.issubset(src_tvs)):
-            raise AssertionError(
-                "Some free vars don't appear in src - {}"
-                .format(free_typevars.difference(src_tvs)))
-
-        # Update the type vars for each Var to their inferred values
-        for v in self.inputs + self.defs:
-            v.set_typevar(self.ti[v.get_typevar()])
-
-    def __repr__(self):
-        # type: () -> str
-        s = "XForm(inputs={}, defs={},\n  ".format(self.inputs, self.defs)
-        s += '\n  '.join(str(n) for n in self.src.rtl)
-        s += '\n=>\n  '
-        s += '\n  '.join(str(n) for n in self.dst.rtl)
-        s += '\n)'
-        return s
-
-    def _rewrite_rtl(self, rtl, symtab, context):
-        # type: (Rtl, Dict[str, Var], int) -> None
-        for line in rtl.rtl:
-            if isinstance(line, Def):
-                line.defs = tuple(
-                        self._rewrite_defs(line, symtab, context))
-                expr = line.expr
-            else:
-                expr = line
-            self._rewrite_expr(expr, symtab, context)
-
-    def _rewrite_expr(self, expr, symtab, context):
-        # type: (Apply, Dict[str, Var], int) -> None
-        """
-        Find all uses of variables in `expr` and replace them with our own
-        local symbols.
-        """
-
-        # Accept a whole expression tree.
-        stack = [expr]
-        while len(stack) > 0:
-            expr = stack.pop()
-            expr.args = tuple(
-                    self._rewrite_uses(expr, stack, symtab, context))
-
-    def _rewrite_defs(self, line, symtab, context):
-        # type: (Def, Dict[str, Var], int) -> Iterable[Var]
-        """
-        Given a tuple of symbols defined in a Def, rewrite them to local
-        symbols. Yield the new locals.
-        """
-        for sym in line.defs:
-            name = str(sym)
-            if name in symtab:
-                var = symtab[name]
-                if var.get_def(context):
-                    raise AssertionError("'{}' multiply defined".format(name))
-            else:
-                var = Var(name)
-                symtab[name] = var
-                self.defs.append(var)
-            var.set_def(context, line)
-            yield var
-
-    def _rewrite_uses(self, expr, stack, symtab, context):
-        # type: (Apply, List[Apply], Dict[str, Var], int) -> Iterable[Expr]
-        """
-        Given an `Apply` expr, rewrite all uses in its arguments to local
-        variables. Yield a sequence of new arguments.
-
-        Append any `Apply` arguments to `stack`.
-        """
-        for arg, operand in zip(expr.args, expr.inst.ins):
-            # Nested instructions are allowed. Visit recursively.
-            if isinstance(arg, Apply):
-                stack.append(arg)
-                yield arg
-                continue
-            if not isinstance(arg, Var):
-                assert not operand.is_value(), "Value arg must be `Var`"
-                yield arg
-                continue
-            # This is supposed to be a symbolic value reference.
-            name = str(arg)
-            if name in symtab:
-                var = symtab[name]
-                # The variable must be used consistently as a def or input.
-                if not var.is_input() and not var.get_def(context):
-                    raise AssertionError(
-                            "'{}' used as both input and def"
-                            .format(name))
-            else:
-                # First time use of variable.
-                var = Var(name)
-                symtab[name] = var
-                self.inputs.append(var)
-            yield var
-
-    def verify_legalize(self):
-        # type: () -> None
-        """
-        Verify that this is a valid legalization XForm.
-
-        - The source pattern must describe a single instruction.
-        - All values defined in the output pattern must be defined in the
-          destination pattern.
-        """
-        assert len(self.src.rtl) == 1, "Legalize needs single instruction."
-        for d in self.src.rtl[0].defs:
-            if not d.is_output():
-                raise AssertionError(
-                        '{} not defined in dest pattern'.format(d))
-
-    def apply(self, r, suffix=None):
-        # type: (Rtl, str) -> Rtl
-        """
-        Given a concrete Rtl r s.t. r matches self.src, return the
-        corresponding concrete self.dst. If suffix is provided, any temporary
-        defs are renamed with '.suffix' appended to their old name.
-        """
-        assert r.is_concrete()
-        s = self.src.substitution(r, {})  # type: VarAtomMap
-        assert s is not None
-
-        if (suffix is not None):
-            for v in self.dst.vars():
-                if v.is_temp():
-                    assert v not in s
-                    s[v] = Var(v.name + '.' + suffix)
-
-        dst = self.dst.copy(s)
-        dst.cleanup_concrete_rtl()
-        return dst
-
-
-class XFormGroup(object):
-    """
-    A group of related transformations.
-
-    :param isa: A target ISA whose instructions are allowed.
-    :param chain: A next level group to try if this one doesn't match.
-    """
-
-    def __init__(self, name, doc, isa=None, chain=None):
-        # type: (str, str, TargetISA, XFormGroup) -> None
-        self.xforms = list()  # type: List[XForm]
-        self.custom = OrderedDict()  # type: OrderedDict[Instruction, str]
-        self.name = name
-        self.__doc__ = doc
-        self.isa = isa
-        self.chain = chain
-
-    def __str__(self):
-        # type: () -> str
-        if self.isa:
-            return '{}.{}'.format(self.isa.name, self.name)
-        else:
-            return self.name
-
-    def rust_name(self):
-        # type: () -> str
-        """
-        Get the Rust name of this function implementing this transform.
-        """
-        if self.isa:
-            # This is a function in the same module as the LEGALIZE_ACTION
-            # table referring to it.
-            return self.name
-        else:
-            return 'crate::legalizer::{}'.format(self.name)
-
-    def legalize(self, src, dst):
-        # type: (Union[Def, Apply], Rtl) -> None
-        """
-        Add a legalization pattern to this group.
-
-        :param src: Single `Def` or `Apply` to be legalized.
-        :param dst: `Rtl` list of replacement instructions.
-        """
-        xform = XForm(Rtl(src), dst)
-        xform.verify_legalize()
-        self.xforms.append(xform)
-
-    def custom_legalize(self, inst, funcname):
-        # type: (Instruction, str) -> None
-        """
-        Add a custom legalization action for `inst`.
-
-        The `funcname` parameter is the fully qualified name of a Rust function
-        which takes the same arguments as the `isa::Legalize` actions.
-
-        The custom function will be called to legalize `inst` and any return
-        value is ignored.
-        """
-        assert inst not in self.custom, "Duplicate custom_legalize"
-        self.custom[inst] = funcname
diff --git a/cranelift/codegen/meta-python/check.sh b/cranelift/codegen/meta-python/check.sh
deleted file mode 100755
index fec63798df..0000000000
--- a/cranelift/codegen/meta-python/check.sh
+++ /dev/null
@@ -1,32 +0,0 @@
-#!/bin/bash
-set -euo pipefail
-topdir=$(dirname "$0")
-cd "$topdir"
-
-function runif {
-    if type "$1" > /dev/null 2>&1; then
-        version=$("$1" --version 2>&1)
-        echo "   === $1: $version ==="
-        "$@"
-    else
-        echo "$1 not found"
-    fi
-}
-
-# Style linting.
-runif flake8 .
-
-# Type checking.
-# TODO: Re-enable mypy on Travis osx. Pip currently installs mypy into a
-# directory which is not in the PATH.
-if [ "${TRAVIS_OS_NAME:-other}" != "osx" ]; then
-    runif mypy --py2 build.py
-fi
-
-# Python unit tests.
-runif python2.7 -m unittest discover
-
-# Then run the unit tests again with Python 3.
-# We get deprecation warnings about assertRaisesRegexp which was renamed in
-# Python 3, but there doesn't seem to be an easy workaround.
-runif python3 -Wignore:Deprecation -m unittest discover
diff --git a/cranelift/codegen/meta-python/constant_hash.py b/cranelift/codegen/meta-python/constant_hash.py
deleted file mode 100644
index e282936723..0000000000
--- a/cranelift/codegen/meta-python/constant_hash.py
+++ /dev/null
@@ -1,63 +0,0 @@
-"""
-Generate constant hash tables.
-
-The `constant_hash` module can generate constant pre-populated hash tables. We
-don't attempt perfect hashing, but simply generate an open addressed
-quadratically probed hash table.
-"""
-from __future__ import absolute_import
-from cdsl import next_power_of_two
-
-try:
-    from typing import Any, List, Iterable, Callable  # noqa
-except ImportError:
-    pass
-
-
-def simple_hash(s):
-    # type: (str) -> int
-    """
-    Compute a primitive hash of a string.
-
-    Example:
-        >>> "0x%x" % simple_hash("Hello")
-        '0x2fa70c01'
-        >>> "0x%x" % simple_hash("world")
-        '0x5b0c31d5'
-    """
-    h = 5381
-    for c in s:
-        h = ((h ^ ord(c)) + ((h >> 6) + (h << 26))) & 0xffffffff
-    return h
-
-
-def compute_quadratic(items, hash_function):
-    # type: (Iterable[Any], Callable[[Any], int]) -> List[Any]
-    """
-    Compute an open addressed, quadratically probed hash table containing
-    `items`. The returned table is a list containing the elements of the
-    iterable `items` and `None` in unused slots.
-
-    :param items: Iterable set of items to place in hash table.
-    :param hash_function: Hash function which takes an item and returns a
-            number.
-
-    Simple example (see hash values above, they collide on slot 1):
-        >>> compute_quadratic(['Hello', 'world'], simple_hash)
-        [None, 'Hello', 'world', None]
-    """
-
-    items = list(items)
-    # Table size must be a power of two. Aim for >20% unused slots.
-    size = next_power_of_two(int(1.20*len(items)))
-    table = [None] * size  # type: List[Any]
-
-    for i in items:
-        h = hash_function(i) % size
-        s = 0
-        while table[h] is not None:
-            s += 1
-            h = (h + s) % size
-        table[h] = i
-
-    return table
diff --git a/cranelift/codegen/meta-python/gen_binemit.py b/cranelift/codegen/meta-python/gen_binemit.py
deleted file mode 100644
index c813d12977..0000000000
--- a/cranelift/codegen/meta-python/gen_binemit.py
+++ /dev/null
@@ -1,170 +0,0 @@
-"""
-Generate binary emission code for each ISA.
-"""
-
-from __future__ import absolute_import
-from cdsl.registers import RegClass, Stack
-import srcgen
-
-try:
-    from typing import Sequence, List  # noqa
-    from cdsl.isa import TargetISA, EncRecipe, OperandConstraint  # noqa
-except ImportError:
-    pass
-
-
-def gen_recipe(recipe, fmt):
-    # type: (EncRecipe, srcgen.Formatter) -> None
-    """
-    Generate code to handle a single recipe.
-
-    - Unpack the instruction data, knowing the format.
-    - Determine register locations for operands with register constraints.
-    - Determine stack slot locations for operands with stack constraints.
-    - Call hand-written code for the actual emission.
-    """
-    iform = recipe.format
-    nvops = iform.num_value_operands
-    want_args = any(isinstance(i, RegClass) or isinstance(i, Stack)
-                    for i in recipe.ins)
-    assert not want_args or nvops > 0 or iform.has_value_list
-    want_outs = any(isinstance(o, RegClass) or isinstance(o, Stack)
-                    for o in recipe.outs)
-
-    # Regmove instructions get special treatment.
-    is_regmove = (recipe.format.name in ('RegMove', 'RegSpill', 'RegFill'))
-
-    # First unpack the instruction.
-    with fmt.indented(
-            'if let InstructionData::{} {{'.format(iform.name),
-            '}'):
-        fmt.line('opcode,')
-        for f in iform.imm_fields:
-            fmt.line('{},'.format(f.member))
-        if want_args:
-            if iform.has_value_list or nvops > 1:
-                fmt.line('ref args,')
-            else:
-                fmt.line('arg,')
-        fmt.line('..')
-        fmt.outdented_line('} = func.dfg[inst] {')
-
-        # Pass recipe arguments in this order: inputs, imm_fields, outputs.
-        args = ''
-
-        # Normalize to an `args` array.
-        if want_args and not is_regmove:
-            if iform.has_value_list:
-                fmt.line('let args = args.as_slice(&func.dfg.value_lists);')
-            elif nvops == 1:
-                fmt.line('let args = [arg];')
-            args += unwrap_values(recipe.ins, 'in', 'args', fmt)
-
-        for f in iform.imm_fields:
-            args += ', ' + f.member
-
-        # Unwrap interesting output arguments.
-        if want_outs:
-            if len(recipe.outs) == 1:
-                fmt.line('let results = [func.dfg.first_result(inst)];')
-            else:
-                fmt.line('let results = func.dfg.inst_results(inst);')
-            args += unwrap_values(recipe.outs, 'out', 'results', fmt)
-
-        # Special handling for regmove instructions. Update the register
-        # diversion tracker.
-        if recipe.format.name == 'RegMove':
-            fmt.line('divert.regmove(arg, src, dst);')
-        elif recipe.format.name == 'RegSpill':
-            fmt.line('divert.regspill(arg, src, dst);')
-        elif recipe.format.name == 'RegFill':
-            fmt.line('divert.regfill(arg, src, dst);')
-
-        # Call hand-written code. If the recipe contains a code snippet, use
-        # that. Otherwise cal a recipe function in the target ISA's binemit
-        # module.
-        if recipe.emit is None:
-            fmt.format(
-                    'return recipe_{}(func, inst, sink, bits{});',
-                    recipe.name.lower(), args)
-        else:
-            fmt.multi_line(recipe.emit)
-            fmt.line('return;')
-
-
-def unwrap_values(args, prefix, values, fmt):
-    # type: (Sequence[OperandConstraint], str, str, srcgen.Formatter) -> str  # noqa
-    """
-    Emit code that unwraps values living in registers or stack slots.
-
-    :param args: Input or output constraints.
-    :param prefix: Prefix to be used for the generated local variables.
-    :param values: Name of slice containing the values to be unwrapped.
-    :returns: Comma separated list of the generated variables
-    """
-    varlist = ''
-    for i, cst in enumerate(args):
-        if isinstance(cst, RegClass):
-            v = '{}_reg{}'.format(prefix, i)
-            varlist += ', ' + v
-            fmt.format(
-                    'let {} = divert.reg({}[{}], &func.locations);',
-                    v, values, i)
-        elif isinstance(cst, Stack):
-            v = '{}_stk{}'.format(prefix, i)
-            varlist += ', ' + v
-            with fmt.indented(
-                    'let {} = StackRef::masked('.format(v),
-                    ').unwrap();'):
-                fmt.format('divert.stack({}[{}], &func.locations),', values, i)
-                fmt.format('{},', cst.stack_base_mask())
-                fmt.line('&func.stack_slots,')
-    return varlist
-
-
-def gen_isa(isa, fmt):
-    # type: (TargetISA, srcgen.Formatter) -> None
-    """
-    Generate Binary emission code for `isa`.
-    """
-    fmt.doc_comment(
-            '''
-            Emit binary machine code for `inst` for the {} ISA.
-            '''.format(isa.name))
-    if len(isa.all_recipes) == 0:
-        # No encoding recipes: Emit a stub.
-        with fmt.indented('pub fn emit_inst('):
-            fmt.line('func: &Function,')
-            fmt.line('inst: Inst,')
-            fmt.line('_divert: &mut RegDiversions,')
-            fmt.line('_sink: &mut CS,')
-        with fmt.indented(') {', '}'):
-            fmt.line('bad_encoding(func, inst)')
-    else:
-        fmt.line('#[allow(unused_variables, unreachable_code)]')
-        with fmt.indented('pub fn emit_inst('):
-            fmt.line('func: &Function,')
-            fmt.line('inst: Inst,')
-            fmt.line('divert: &mut RegDiversions,')
-            fmt.line('sink: &mut CS,')
-        with fmt.indented(') {', '}'):
-            fmt.line('let encoding = func.encodings[inst];')
-            fmt.line('let bits = encoding.bits();')
-            with fmt.indented('match func.encodings[inst].recipe() {', '}'):
-                for i, recipe in enumerate(isa.all_recipes):
-                    fmt.comment('Recipe {}'.format(recipe.name))
-                    with fmt.indented('{} => {{'.format(i), '}'):
-                        gen_recipe(recipe, fmt)
-                fmt.line('_ => {},')
-            # Allow for un-encoded ghost instructions.
-            # Verifier checks the details.
-            with fmt.indented('if encoding.is_legal() {', '}'):
-                fmt.line('bad_encoding(func, inst);')
-
-
-def generate(isas, out_dir):
-    # type: (Sequence[TargetISA], str) -> None
-    for isa in isas:
-        fmt = srcgen.Formatter()
-        gen_isa(isa, fmt)
-        fmt.update_file('binemit-{}.rs'.format(isa.name), out_dir)
diff --git a/cranelift/codegen/meta-python/gen_build_deps.py b/cranelift/codegen/meta-python/gen_build_deps.py
deleted file mode 100644
index 637865a55c..0000000000
--- a/cranelift/codegen/meta-python/gen_build_deps.py
+++ /dev/null
@@ -1,32 +0,0 @@
-"""
-Generate build dependencies for Cargo.
-
-The `build.py` script is invoked by cargo when building cranelift-codegen to
-generate Rust code from the instruction descriptions. Cargo needs to know when
-it is necessary to rerun the build script.
-
-If the build script outputs lines of the form:
-
-    cargo:rerun-if-changed=/path/to/file
-
-cargo will rerun the build script when those files have changed since the last
-build.
-"""
-from __future__ import absolute_import, print_function
-import os
-from os.path import dirname, abspath, join
-
-try:
-    from typing import Iterable  # noqa
-except ImportError:
-    pass
-
-
-def generate():
-    # type: () -> None
-    print("Dependencies from meta language directory:")
-    meta = dirname(abspath(__file__))
-    for (dirpath, _, filenames) in os.walk(meta):
-        for f in filenames:
-            if f.endswith('.py'):
-                print("cargo:rerun-if-changed=" + join(dirpath, f))
diff --git a/cranelift/codegen/meta-python/gen_encoding.py b/cranelift/codegen/meta-python/gen_encoding.py
deleted file mode 100644
index 5f70c97eca..0000000000
--- a/cranelift/codegen/meta-python/gen_encoding.py
+++ /dev/null
@@ -1,902 +0,0 @@
-"""
-Generate sources for instruction encoding.
-
-The tables and functions generated here support the `TargetISA::encode()`
-function which determines if a given instruction is legal, and if so, it's
-`Encoding` data which consists of a *recipe* and some *encoding* bits.
-
-The `encode` function doesn't actually generate the binary machine bits. Each
-recipe has a corresponding hand-written function to do that after registers
-are allocated.
-
-This is the information available to us:
-
-- The instruction to be encoded as an `InstructionData` reference.
-- The controlling type variable.
-- The data-flow graph giving us access to the types of all values involved.
-  This is needed for testing any secondary type variables.
-- A `PredicateView` reference for the ISA-specific settings for evaluating ISA
-  predicates.
-- The currently active CPU mode is determined by the ISA.
-
-## Level 1 table lookup
-
-The CPU mode provides the first table. The key is the instruction's controlling
-type variable. If the instruction is not polymorphic, use `INVALID` for the
-type variable. The table values are level 2 tables.
-
-## Level 2 table lookup
-
-The level 2 table is keyed by the instruction's opcode. The table values are
-*encoding lists*.
-
-The two-level table lookup allows the level 2 tables to be much smaller with
-good locality. Code in any given function usually only uses a few different
-types, so many of the level 2 tables will be cold.
-
-## Encoding lists
-
-An encoding list is a non-empty sequence of list entries. Each entry has
-one of these forms:
-
-1. Recipe + bits. Use this encoding if the recipe predicate is satisfied.
-2. Recipe + bits, final entry. Use this encoding if the recipe predicate is
-   satisfied. Otherwise, stop with the default legalization code.
-3. Stop with legalization code.
-4. Predicate + skip count. Test predicate and skip N entries if it is false.
-4. Predicate + stop. Test predicate and stop with the default legalization code
-   if it is false.
-
-The instruction predicate is also used to distinguish between polymorphic
-instructions with different types for secondary type variables.
-"""
-from __future__ import absolute_import
-import srcgen
-from constant_hash import compute_quadratic
-from unique_table import UniqueSeqTable
-from collections import OrderedDict, defaultdict
-import math
-from itertools import groupby
-from cdsl.registers import RegClass, Register, Stack
-from cdsl.predicates import FieldPredicate, TypePredicate
-from cdsl.settings import SettingGroup
-from cdsl.formats import instruction_context, InstructionFormat
-
-try:
-    from typing import Sequence, Set, Tuple, List, Dict, Iterable, DefaultDict, TYPE_CHECKING  # noqa
-    if TYPE_CHECKING:
-        from cdsl.isa import TargetISA, OperandConstraint, Encoding, CPUMode, EncRecipe, RecipePred  # noqa
-        from cdsl.predicates import PredNode, PredLeaf  # noqa
-        from cdsl.types import ValueType  # noqa
-        from cdsl.instructions import Instruction  # noqa
-        from cdsl.xform import XFormGroup  # noqa
-except ImportError:
-    pass
-
-
-def emit_instp(instp, fmt, has_func=False):
-    # type: (PredNode, srcgen.Formatter, bool) -> None
-    """
-    Emit code for matching an instruction predicate against an
-    `InstructionData` reference called `inst`.
-
-    The generated code is an `if let` pattern match that falls through if the
-    instruction has an unexpected format. This should lead to a panic.
-    """
-    iform = instp.predicate_context()
-
-    # Deal with pure type check predicates which apply to any instruction.
-    if iform == instruction_context:
-        fmt.line('let args = inst.arguments(&func.dfg.value_lists);')
-        fmt.line(instp.rust_predicate(0))
-        return
-
-    assert isinstance(iform, InstructionFormat)
-
-    # Which fields do we need in the InstructionData pattern match?
-    has_type_check = False
-    # Collect the leaf predicates.
-    leafs = set()  # type: Set[PredLeaf]
-    instp.predicate_leafs(leafs)
-    # All the leafs are FieldPredicate or TypePredicate instances. Here we just
-    # care about the field names.
-    fnames = set()  # type: Set[str]
-    for p in leafs:
-        if isinstance(p, FieldPredicate):
-            fnames.add(p.field.rust_destructuring_name())
-        else:
-            assert isinstance(p, TypePredicate)
-            has_type_check = True
-    fields = ', '.join(sorted(fnames))
-
-    with fmt.indented(
-            'if let crate::ir::InstructionData::{} {{ {}, .. }} = *inst {{'
-            .format(iform.name, fields), '}'):
-        if has_type_check:
-            # We could implement this if we need to.
-            assert has_func, "Recipe predicates can't check type variables."
-            fmt.line('let args = inst.arguments(&func.dfg.value_lists);')
-        elif has_func:
-            # Silence dead argument warning.
-            fmt.line('let _ = func;')
-        fmt.format('return {};', instp.rust_predicate(0))
-    fmt.line('unreachable!();')
-
-
-def emit_inst_predicates(instps, fmt):
-    # type: (OrderedDict[PredNode, int], srcgen.Formatter) -> None
-    """
-    Emit private functions for matching instruction predicates as well as a
-    static `INST_PREDICATES` array indexed by predicate number.
-    """
-    for instp, number in instps.items():
-        name = 'inst_predicate_{}'.format(number)
-        with fmt.indented(
-                'fn {}(func: &crate::ir::Function, '
-                'inst: &crate::ir::InstructionData)'
-                '-> bool {{'.format(name), '}'):
-            emit_instp(instp, fmt, has_func=True)
-
-    # Generate the static table.
-    with fmt.indented(
-            'pub static INST_PREDICATES: [InstPredicate; {}] = ['
-            .format(len(instps)), '];'):
-        for instp, number in instps.items():
-            fmt.format('inst_predicate_{},', number)
-
-
-def emit_recipe_predicates(isa, fmt):
-    # type: (TargetISA, srcgen.Formatter) -> None
-    """
-    Emit private functions for checking recipe predicates as well as a static
-    `RECIPE_PREDICATES` array indexed by recipe number.
-
-    A recipe predicate is a combination of an ISA predicate and an instruction
-    predicates. Many recipes have identical predicates.
-    """
-    # Table for uniquing recipe predicates. Maps predicate to generated
-    # function name.
-    pname = dict()  # type: Dict[RecipePred, str]
-
-    # Generate unique recipe predicates.
-    for rcp in isa.all_recipes:
-        p = rcp.recipe_pred()
-        if p is None or p in pname:
-            continue
-        name = 'recipe_predicate_{}'.format(rcp.name.lower())
-        pname[p] = name
-        isap, instp = p
-
-        # Generate the predicate function.
-        with fmt.indented(
-                'fn {}({}: crate::settings::PredicateView, '
-                '{}: &ir::InstructionData) -> bool {{'
-                .format(
-                    name,
-                    'isap' if isap else '_',
-                    'inst' if instp else '_'), '}'):
-            if isap:
-                n = isa.settings.predicate_number[isap]
-                with fmt.indented('if !isap.test({}) {{'.format(n), '}'):
-                    fmt.line('return false;')
-            if instp:
-                emit_instp(instp, fmt)
-            else:
-                fmt.line('true')
-
-    # Generate the static table.
-    with fmt.indented(
-            'pub static RECIPE_PREDICATES: [RecipePredicate; {}] = ['
-            .format(len(isa.all_recipes)), '];'):
-        for rcp in isa.all_recipes:
-            p = rcp.recipe_pred()
-            if p is None:
-                fmt.line('None,')
-            else:
-                fmt.format('Some({}),', pname[p])
-
-
-# The u16 values in an encoding list entry are interpreted as follows:
-#
-# NR = len(all_recipes)
-#
-# entry < 2*NR
-#     Try Encoding(entry/2, next_entry) if the recipe predicate is satisfied.
-#     If bit 0 is set, stop with the default legalization code.
-#     If bit 0 is clear, keep going down the list.
-# entry < PRED_START
-#     Stop with legalization code `entry - 2*NR`.
-#
-# Remaining entries are interpreted as (skip, pred) pairs, where:
-#
-#     skip = (entry - PRED_START) >> PRED_BITS
-#     pred = (entry - PRED_START) & PRED_MASK
-#
-# If the predicate is satisfied, keep going. Otherwise skip over the next
-# `skip` entries. If skip == 0, stop with the default legalization code.
-#
-# The `pred` predicate number is interpreted as an instruction predicate if it
-# is in range, otherwise an ISA predicate.
-
-
-class Encoder:
-    """
-    Encoder for the list format above.
-
-    Two parameters are needed:
-
-    :param NR: Number of recipes.
-    :param NI: Number of instruction predicates.
-    """
-
-    def __init__(self, isa):
-        # type: (TargetISA) -> None
-        self.isa = isa
-        self.NR = len(isa.all_recipes)
-        self.NI = len(isa.instp_number)
-        # u16 encoding list words.
-        self.words = list()  # type: List[int]
-        # Documentation comments: Index into `words` + comment.
-        self.docs = list()  # type: List[Tuple[int, str]]
-
-    # Encoding lists are represented as u16 arrays.
-    CODE_BITS = 16
-
-    # Beginning of the predicate code words.
-    PRED_START = 0x1000
-
-    # Number of bits used to hold a predicate number (instruction + ISA
-    # predicates.
-    PRED_BITS = 12
-
-    # Mask for extracting the predicate number.
-    PRED_MASK = (1 << PRED_BITS) - 1
-
-    def max_skip(self):
-        # type: () -> int
-        """The maximum number of entries that a predicate can skip."""
-        return (1 << (self.CODE_BITS - self.PRED_BITS)) - 1
-
-    def recipe(self, enc, final):
-        # type: (Encoding, bool) -> None
-        """Add a recipe+bits entry to the list."""
-        offset = len(self.words)
-        code = 2 * enc.recipe.number
-        doc = '--> {}'.format(enc)
-        if final:
-            code += 1
-            doc += ' and stop'
-
-        assert(code < self.PRED_START)
-        self.words.extend((code, enc.encbits))
-        self.docs.append((offset, doc))
-
-    def _pred(self, pred, skip, n):
-        # type: (PredNode, int, int) -> None
-        """Add a predicate entry."""
-        assert n <= self.PRED_MASK
-        code = n | (skip << self.PRED_BITS)
-        code += self.PRED_START
-        assert code < (1 << self.CODE_BITS)
-
-        if skip == 0:
-            doc = 'stop'
-        else:
-            doc = 'skip ' + str(skip)
-        doc = '{} unless {}'.format(doc, pred)
-
-        self.docs.append((len(self.words), doc))
-        self.words.append(code)
-
-    def instp(self, pred, skip):
-        # type: (PredNode, int) -> None
-        """Add an instruction predicate entry."""
-        number = self.isa.instp_number[pred]
-        self._pred(pred, skip, number)
-
-    def isap(self, pred, skip):
-        # type: (PredNode, int) -> None
-        """Add an ISA predicate entry."""
-        n = self.isa.settings.predicate_number[pred]
-        # ISA predicates follow the instruction predicates.
-        self._pred(pred, skip, self.NI + n)
-
-
-class EncNode(object):
-    """
-    An abstract node in the encoder tree for an instruction.
-
-    This tree is used to simplify the predicates guarding recipe+bits entries.
-    """
-
-    def size(self):
-        # type: () -> int
-        """Get the number of list entries needed to encode this tree."""
-        raise NotImplementedError('EncNode.size() is abstract')
-
-    def encode(self, encoder, final):
-        # type: (Encoder, bool) -> None
-        """Encode this tree."""
-        raise NotImplementedError('EncNode.encode() is abstract')
-
-    def optimize(self):
-        # type: () -> EncNode
-        """Transform this encoder tree into something simpler."""
-        return self
-
-    def predicate(self):
-        # type: () -> PredNode
-        """Get the predicate guarding this tree, or `None` for always"""
-        return None
-
-
-class EncPred(EncNode):
-    """
-    An encoder tree node which asserts a predicate on its child nodes.
-
-    A `None` predicate is always satisfied.
-    """
-
-    def __init__(self, pred, children):
-        # type: (PredNode, List[EncNode]) -> None
-        self.pred = pred
-        self.children = children
-
-    def size(self):
-        # type: () -> int
-        s = 1 if self.pred else 0
-        s += sum(c.size() for c in self.children)
-        return s
-
-    def encode(self, encoder, final):
-        # type: (Encoder, bool) -> None
-        if self.pred:
-            skip = 0 if final else self.size() - 1
-            ctx = self.pred.predicate_context()
-            if isinstance(ctx, SettingGroup):
-                encoder.isap(self.pred, skip)
-            else:
-                encoder.instp(self.pred, skip)
-
-        final_idx = len(self.children) - 1 if final else -1
-        for idx, node in enumerate(self.children):
-            node.encode(encoder, idx == final_idx)
-
-    def predicate(self):
-        # type: () -> PredNode
-        return self.pred
-
-    def optimize(self):
-        # type: () -> EncNode
-        """
-        Optimize a predicate node in the tree by combining child nodes that
-        have identical predicates.
-        """
-        cnodes = list()  # type: List[EncNode]
-        for pred, niter in groupby(
-                map(lambda c: c.optimize(), self.children),
-                key=lambda c: c.predicate()):
-            nodes = list(niter)
-            if pred is None or len(nodes) <= 1:
-                cnodes.extend(nodes)
-                continue
-
-            # We have multiple children with identical predicates.
-            # Group them all into `n0`.
-            n0 = nodes[0]
-            assert isinstance(n0, EncPred)
-            for n in nodes[1:]:
-                assert isinstance(n, EncPred)
-                n0.children.extend(n.children)
-
-            cnodes.append(n0)
-
-        # Finally strip a redundant grouping node.
-        if self.pred is None and len(cnodes) == 1:
-            return cnodes[0]
-        else:
-            self.children = cnodes
-            return self
-
-
-class EncLeaf(EncNode):
-    """
-    A leaf in the encoder tree.
-
-    This represents a single `Encoding`, without its predicates (they are
-    represented in the tree by parent nodes.
-    """
-
-    def __init__(self, encoding):
-        # type: (Encoding) -> None
-        self.encoding = encoding
-
-    def size(self):
-        # type: () -> int
-        # recipe + bits.
-        return 2
-
-    def encode(self, encoder, final):
-        # type: (Encoder, bool) -> None
-        encoder.recipe(self.encoding, final)
-
-
-class EncList(object):
-    """
-    List of instructions for encoding a given type + opcode pair.
-
-    An encoding list contains a sequence of predicates and encoding recipes,
-    all encoded as u16 values.
-
-    :param inst: The instruction opcode being encoded.
-    :param ty: Value of the controlling type variable, or `None`.
-    """
-
-    def __init__(self, inst, ty):
-        # type: (Instruction, ValueType) -> None
-        self.inst = inst
-        self.ty = ty
-        # List of applicable Encoding instances.
-        # These will have different predicates.
-        self.encodings = []  # type: List[Encoding]
-
-    def name(self):
-        # type: () -> str
-        name = self.inst.name
-        if self.ty:
-            name = '{}.{}'.format(name, self.ty.name)
-        if self.encodings:
-            name += ' ({})'.format(self.encodings[0].cpumode)
-        return name
-
-    def encoder_tree(self):
-        # type: () -> EncNode
-        """
-        Generate an optimized encoder tree for this list. The tree represents
-        all of the encodings with parent nodes for the predicates that need
-        checking.
-        """
-        forest = list()  # type: List[EncNode]
-        for enc in self.encodings:
-            n = EncLeaf(enc)  # type: EncNode
-            if enc.instp:
-                n = EncPred(enc.instp, [n])
-            if enc.isap:
-                n = EncPred(enc.isap, [n])
-            forest.append(n)
-
-        return EncPred(None, forest).optimize()
-
-    def encode(self, seq_table, doc_table, isa):
-        # type: (UniqueSeqTable, DefaultDict[int, List[str]], TargetISA) -> None  # noqa
-        """
-        Encode this list as a sequence of u16 numbers.
-
-        Adds the sequence to `seq_table` and records the returned offset as
-        `self.offset`.
-
-        Adds comment lines to `doc_table` keyed by seq_table offsets.
-        """
-        # Use an encoder object to hold the parameters.
-        encoder = Encoder(isa)
-        tree = self.encoder_tree()
-        tree.encode(encoder, True)
-
-        self.offset = seq_table.add(encoder.words)
-
-        # Add doc comments.
-        doc_table[self.offset].append(
-                '{:06x}: {}'.format(self.offset, self.name()))
-        for pos, doc in encoder.docs:
-            doc_table[self.offset + pos].append(doc)
-        doc_table[self.offset + len(encoder.words)].insert(
-                0, 'end of: {}'.format(self.name()))
-
-
-class Level2Table(object):
-    """
-    Level 2 table mapping instruction opcodes to `EncList` objects.
-
-    A level 2 table can be completely empty if it only holds a custom
-    legalization action for `ty`.
-
-    :param ty: Controlling type variable of all entries, or `None`.
-    :param legalize: Default legalize action for `ty`.
-    """
-
-    def __init__(self, ty, legalize):
-        # type: (ValueType, XFormGroup) -> None
-        self.ty = ty
-        self.legalize = legalize
-        # Maps inst -> EncList
-        self.lists = OrderedDict()  # type: OrderedDict[Instruction, EncList]
-
-    def __getitem__(self, inst):
-        # type: (Instruction) -> EncList
-        ls = self.lists.get(inst)
-        if not ls:
-            ls = EncList(inst, self.ty)
-            self.lists[inst] = ls
-        return ls
-
-    def is_empty(self):
-        # type: () -> bool
-        """
-        Check if this level 2 table is completely empty.
-
-        This can happen if the associated type simply has an overridden
-        legalize action.
-        """
-        return len(self.lists) == 0
-
-    def enclists(self):
-        # type: () -> Iterable[EncList]
-        return iter(self.lists.values())
-
-    def layout_hashtable(self, level2_hashtables, level2_doc):
-        # type: (List[EncList], DefaultDict[int, List[str]]) -> None
-        """
-        Compute the hash table mapping opcode -> enclist.
-
-        Append the hash table to `level2_hashtables` and record the offset.
-        """
-        def hash_func(enclist):
-            # type: (EncList) -> int
-            return enclist.inst.number
-        hash_table = compute_quadratic(self.lists.values(), hash_func)
-
-        self.hash_table_offset = len(level2_hashtables)
-        self.hash_table_len = len(hash_table)
-
-        level2_doc[self.hash_table_offset].append(
-                '{:06x}: {}, {} entries'.format(
-                    self.hash_table_offset,
-                    self.ty,
-                    self.hash_table_len))
-        level2_hashtables.extend(hash_table)
-
-
-class Level1Table(object):
-    """
-    Level 1 table mapping types to `Level2` objects.
-    """
-
-    def __init__(self, cpumode):
-        # type: (CPUMode) -> None
-        self.cpumode = cpumode
-        self.tables = OrderedDict()  # type: OrderedDict[ValueType, Level2Table]  # noqa
-
-        if cpumode.default_legalize is None:
-            raise AssertionError(
-                    'CPU mode {}.{} needs a default legalize action'
-                    .format(cpumode.isa, cpumode))
-        self.legalize_code = cpumode.isa.legalize_code(
-                cpumode.default_legalize)
-
-    def __getitem__(self, ty):
-        # type: (ValueType) -> Level2Table
-        tbl = self.tables.get(ty)
-        if not tbl:
-            legalize = self.cpumode.get_legalize_action(ty)
-            # Allocate a legalization code in a predictable order.
-            self.cpumode.isa.legalize_code(legalize)
-            tbl = Level2Table(ty, legalize)
-            self.tables[ty] = tbl
-        return tbl
-
-    def l2tables(self):
-        # type: () -> Iterable[Level2Table]
-        return (l2 for l2 in self.tables.values() if not l2.is_empty())
-
-
-def make_tables(cpumode):
-    # type: (CPUMode) -> Level1Table
-    """
-    Generate tables for `cpumode` as described above.
-    """
-    table = Level1Table(cpumode)
-    for enc in cpumode.encodings:
-        ty = enc.ctrl_typevar()
-        inst = enc.inst
-        table[ty][inst].encodings.append(enc)
-
-    # Ensure there are level 1 table entries for all types with a custom
-    # legalize action.
-    for ty in cpumode.type_legalize.keys():
-        table[ty]
-
-    return table
-
-
-def encode_enclists(level1, seq_table, doc_table, isa):
-    # type: (Level1Table, UniqueSeqTable, DefaultDict[int, List[str]], TargetISA) -> None  # noqa
-    """
-    Compute encodings and doc comments for encoding lists in `level1`.
-    """
-    for level2 in level1.l2tables():
-        for enclist in level2.enclists():
-            enclist.encode(seq_table, doc_table, isa)
-
-
-def emit_enclists(seq_table, doc_table, fmt):
-    # type: (UniqueSeqTable, DefaultDict[int, List[str]], srcgen.Formatter) -> None  # noqa
-    with fmt.indented(
-            'pub static ENCLISTS: [u16; {}] = ['.format(len(seq_table.table)),
-            '];'):
-        line = ''
-        for idx, entry in enumerate(seq_table.table):
-            if idx in doc_table:
-                if line:
-                    fmt.line(line)
-                    line = ''
-                for doc in doc_table[idx]:
-                    fmt.comment(doc)
-            line += '{:#06x}, '.format(entry)
-        if line:
-            fmt.line(line)
-
-
-def encode_level2_hashtables(level1, level2_hashtables, level2_doc):
-    # type: (Level1Table, List[EncList], DefaultDict[int, List[str]]) -> None
-    for level2 in level1.l2tables():
-        level2.layout_hashtable(level2_hashtables, level2_doc)
-
-
-def emit_level2_hashtables(level2_hashtables, offt, level2_doc, fmt):
-    # type: (List[EncList], str, DefaultDict[int, List[str]], srcgen.Formatter) -> None  # noqa
-    """
-    Emit the big concatenation of level 2 hash tables.
-    """
-    with fmt.indented(
-            'pub static LEVEL2: [Level2Entry<{}>; {}] = ['
-            .format(offt, len(level2_hashtables)),
-            '];'):
-        for offset, entry in enumerate(level2_hashtables):
-            if offset in level2_doc:
-                for doc in level2_doc[offset]:
-                    fmt.comment(doc)
-            if entry:
-                fmt.line(
-                        'Level2Entry ' +
-                        '{{ opcode: Some(crate::ir::Opcode::{}), '
-                        'offset: {:#08x} }},'
-                        .format(entry.inst.camel_name, entry.offset))
-            else:
-                fmt.line(
-                        'Level2Entry ' +
-                        '{ opcode: None, offset: 0 },')
-
-
-def emit_level1_hashtable(cpumode, level1, offt, fmt):
-    # type: (CPUMode, Level1Table, str, srcgen.Formatter) -> None  # noqa
-    """
-    Emit a level 1 hash table for `cpumode`.
-    """
-    def hash_func(level2):
-        # type: (Level2Table) -> int
-        return level2.ty.number if level2.ty is not None else 0
-    hash_table = compute_quadratic(level1.tables.values(), hash_func)
-
-    with fmt.indented(
-            'pub static LEVEL1_{}: [Level1Entry<{}>; {}] = ['
-            .format(cpumode.name.upper(), offt, len(hash_table)), '];'):
-        for level2 in hash_table:
-            # Empty hash table entry. Include the default legalization action.
-            if not level2:
-                fmt.format(
-                        'Level1Entry {{ ty: crate::ir::types::INVALID, '
-                        'log2len: !0, offset: 0, legalize: {} }},',
-                        level1.legalize_code)
-                continue
-
-            if level2.ty is not None:
-                tyname = level2.ty.rust_name()
-            else:
-                tyname = 'crate::ir::types::INVALID'
-
-            lcode = cpumode.isa.legalize_code(level2.legalize)
-
-            # Empty level 2 table: Only a specialized legalization action, no
-            # actual table.
-            # Set an offset that is out of bounds, but make sure it doesn't
-            # overflow its type when adding `1< 0, "Level2 hash table too small"
-            fmt.format(
-                    'Level1Entry {{ '
-                    'ty: {}, log2len: {}, offset: {:#08x}, '
-                    'legalize: {} }}, // {}',
-                    tyname, l2l, level2.hash_table_offset,
-                    lcode, level2.legalize)
-
-
-def offset_type(length):
-    # type: (int) -> str
-    """
-    Compute an appropriate Rust integer type to use for offsets into a table of
-    the given length.
-    """
-    if length <= 0x10000:
-        return 'u16'
-    else:
-        assert length <= 0x100000000, "Table too big"
-        return 'u32'
-
-
-def emit_recipe_names(isa, fmt):
-    # type: (TargetISA, srcgen.Formatter) -> None
-    """
-    Emit a table of encoding recipe names keyed by recipe number.
-
-    This is used for pretty-printing encodings.
-    """
-    with fmt.indented(
-            'static RECIPE_NAMES: [&str; {}] = ['
-            .format(len(isa.all_recipes)), '];'):
-        for r in isa.all_recipes:
-            fmt.line('"{}",'.format(r.name))
-
-
-def emit_recipe_constraints(isa, fmt):
-    # type: (TargetISA, srcgen.Formatter) -> None
-    """
-    Emit a table of encoding recipe operand constraints keyed by recipe number.
-
-    These are used by the register allocator to pick registers that can be
-    properly encoded.
-    """
-    with fmt.indented(
-            'static RECIPE_CONSTRAINTS: [RecipeConstraints; {}] = ['
-            .format(len(isa.all_recipes)), '];'):
-        for r in isa.all_recipes:
-            fmt.comment('Constraints for recipe {}:'.format(r.name))
-            tied_i2o, tied_o2i = r.ties()
-            fixed_ins, fixed_outs = r.fixed_ops()
-            with fmt.indented('RecipeConstraints {', '},'):
-                emit_operand_constraints(
-                    r, r.ins, 'ins', tied_i2o, fixed_outs, fmt)
-                emit_operand_constraints(
-                    r, r.outs, 'outs', tied_o2i, fixed_ins, fmt)
-                fmt.format('fixed_ins: {},', str(bool(fixed_ins)).lower())
-                fmt.format('fixed_outs: {},', str(bool(fixed_outs)).lower())
-                fmt.format('tied_ops: {},', str(bool(tied_i2o)).lower())
-                fmt.format(
-                        'clobbers_flags: {},',
-                        str(bool(r.clobbers_flags)).lower())
-
-
-def emit_operand_constraints(
-        recipe,  # type: EncRecipe
-        seq,     # type: Sequence[OperandConstraint]
-        field,   # type: str
-        tied,    # type: Dict[int, int]
-        fixops,  # type: Set[Register]
-        fmt      # type: srcgen.Formatter
-        ):
-    # type: (...) -> None
-    """
-    Emit a struct field initializer for an array of operand constraints.
-
-    :param field: The name of the struct field to emit.
-    :param tied: Map of tied opnums to counterparts.
-    :param fix_ops: Set of fixed operands on the other side of the inst.
-    """
-    if len(seq) == 0:
-        fmt.line('{}: &[],'.format(field))
-        return
-    with fmt.indented('{}: &['.format(field), '],'):
-        for n, cons in enumerate(seq):
-            with fmt.indented('OperandConstraint {', '},'):
-                if isinstance(cons, RegClass):
-                    if n in tied:
-                        fmt.format('kind: ConstraintKind::Tied({}),', tied[n])
-                    else:
-                        fmt.line('kind: ConstraintKind::Reg,')
-                    fmt.format('regclass: &{}_DATA,', cons)
-                elif isinstance(cons, Register):
-                    assert n not in tied, "Can't tie fixed register operand"
-                    # See if this fixed register is also on the other side.
-                    t = 'FixedTied' if cons in fixops else 'FixedReg'
-                    fmt.format('kind: ConstraintKind::{}({}),', t, cons.unit)
-                    fmt.format('regclass: &{}_DATA,', cons.regclass)
-                elif isinstance(cons, int):
-                    # This is a tied output constraint. It should never happen
-                    # for input constraints.
-                    assert cons == tied[n], "Invalid tied constraint"
-                    fmt.format('kind: ConstraintKind::Tied({}),', cons)
-                    fmt.format('regclass: &{}_DATA,', recipe.ins[cons])
-                elif isinstance(cons, Stack):
-                    assert n not in tied, "Can't tie stack operand"
-                    fmt.line('kind: ConstraintKind::Stack,')
-                    fmt.format('regclass: &{}_DATA,', cons.regclass)
-                else:
-                    raise AssertionError(
-                            'Unsupported constraint {}'.format(cons))
-
-
-def emit_recipe_sizing(isa, fmt):
-    # type: (TargetISA, srcgen.Formatter) -> None
-    """
-    Emit a table of encoding recipe code size information.
-    """
-    with fmt.indented(
-            'static RECIPE_SIZING: [RecipeSizing; {}] = ['
-            .format(len(isa.all_recipes)), '];'):
-        for r in isa.all_recipes:
-            fmt.comment('Code size information for recipe {}:'.format(r.name))
-            with fmt.indented('RecipeSizing {', '},'):
-                fmt.format('base_size: {},', r.base_size)
-                fmt.format('compute_size: {},', r.compute_size)
-                if r.branch_range:
-                    fmt.format(
-                        'branch_range: '
-                        'Some(BranchRange {{ origin: {}, bits: {} }}),',
-                        *r.branch_range)
-                else:
-                    fmt.line('branch_range: None,')
-
-
-def gen_isa(isa, fmt):
-    # type: (TargetISA, srcgen.Formatter) -> None
-
-    # Make the `RECIPE_PREDICATES` table.
-    emit_recipe_predicates(isa, fmt)
-
-    # Make the `INST_PREDICATES` table.
-    emit_inst_predicates(isa.instp_number, fmt)
-
-    # Level1 tables, one per CPU mode
-    level1_tables = dict()
-
-    # Tables for enclists with comments.
-    seq_table = UniqueSeqTable()
-    doc_table = defaultdict(list)  # type: DefaultDict[int, List[str]]
-
-    # Single table containing all the level2 hash tables.
-    level2_hashtables = list()  # type: List[EncList]
-    level2_doc = defaultdict(list)  # type: DefaultDict[int, List[str]]
-
-    for cpumode in isa.cpumodes:
-        level2_doc[len(level2_hashtables)].append(cpumode.name)
-        level1 = make_tables(cpumode)
-        level1_tables[cpumode] = level1
-        encode_enclists(level1, seq_table, doc_table, isa)
-        encode_level2_hashtables(level1, level2_hashtables, level2_doc)
-
-    # Level 1 table encodes offsets into the level 2 table.
-    level1_offt = offset_type(len(level2_hashtables))
-    # Level 2 tables encodes offsets into seq_table.
-    level2_offt = offset_type(len(seq_table.table))
-
-    emit_enclists(seq_table, doc_table, fmt)
-    emit_level2_hashtables(level2_hashtables, level2_offt, level2_doc, fmt)
-    for cpumode in isa.cpumodes:
-        emit_level1_hashtable(
-                cpumode, level1_tables[cpumode], level1_offt, fmt)
-
-    emit_recipe_names(isa, fmt)
-    emit_recipe_constraints(isa, fmt)
-    emit_recipe_sizing(isa, fmt)
-
-    # Finally, tie it all together in an `EncInfo`.
-    with fmt.indented('pub static INFO: isa::EncInfo = isa::EncInfo {', '};'):
-        fmt.line('constraints: &RECIPE_CONSTRAINTS,')
-        fmt.line('sizing: &RECIPE_SIZING,')
-        fmt.line('names: &RECIPE_NAMES,')
-
-
-def generate(isas, out_dir):
-    # type: (Sequence[TargetISA], str) -> None
-    for isa in isas:
-        fmt = srcgen.Formatter()
-        gen_isa(isa, fmt)
-        fmt.update_file('encoding-{}.rs'.format(isa.name), out_dir)
diff --git a/cranelift/codegen/meta-python/isa/__init__.py b/cranelift/codegen/meta-python/isa/__init__.py
deleted file mode 100644
index d3cff62c91..0000000000
--- a/cranelift/codegen/meta-python/isa/__init__.py
+++ /dev/null
@@ -1,24 +0,0 @@
-"""
-Cranelift target ISA definitions
---------------------------------
-
-The :py:mod:`isa` package contains sub-packages for each target instruction set
-architecture supported by Cranelift.
-"""
-from __future__ import absolute_import
-from cdsl.isa import TargetISA  # noqa
-from . import riscv, x86, arm32, arm64
-
-try:
-    from typing import List  # noqa
-except ImportError:
-    pass
-
-
-def all_isas():
-    # type: () -> List[TargetISA]
-    """
-    Get a list of all the supported target ISAs. Each target ISA is represented
-    as a :py:class:`cranelift.TargetISA` instance.
-    """
-    return [riscv.ISA, x86.ISA, arm32.ISA, arm64.ISA]
diff --git a/cranelift/codegen/meta-python/isa/arm32/__init__.py b/cranelift/codegen/meta-python/isa/arm32/__init__.py
deleted file mode 100644
index 06773f8754..0000000000
--- a/cranelift/codegen/meta-python/isa/arm32/__init__.py
+++ /dev/null
@@ -1,15 +0,0 @@
-"""
-ARM 32-bit Architecture
------------------------
-
-This target ISA generates code for ARMv7 and ARMv8 CPUs in 32-bit mode
-(AArch32). We support both ARM and Thumb2 instruction encodings.
-"""
-
-from __future__ import absolute_import
-from . import defs
-from . import settings, registers  # noqa
-from cdsl.isa import TargetISA  # noqa
-
-# Re-export the primary target ISA definition.
-ISA = defs.ISA.finish()  # type: TargetISA
diff --git a/cranelift/codegen/meta-python/isa/arm32/defs.py b/cranelift/codegen/meta-python/isa/arm32/defs.py
deleted file mode 100644
index 88b8c53db4..0000000000
--- a/cranelift/codegen/meta-python/isa/arm32/defs.py
+++ /dev/null
@@ -1,19 +0,0 @@
-"""
-ARM 32-bit definitions.
-
-Commonly used definitions.
-"""
-from __future__ import absolute_import
-from cdsl.isa import TargetISA, CPUMode
-import base.instructions
-from base.legalize import narrow
-
-ISA = TargetISA('arm32', [base.instructions.GROUP])  # type: TargetISA
-
-# CPU modes for 32-bit ARM and Thumb2.
-A32 = CPUMode('A32', ISA)
-T32 = CPUMode('T32', ISA)
-
-# TODO: Refine these.
-A32.legalize_type(narrow)
-T32.legalize_type(narrow)
diff --git a/cranelift/codegen/meta-python/isa/arm32/registers.py b/cranelift/codegen/meta-python/isa/arm32/registers.py
deleted file mode 100644
index 054e95fa0e..0000000000
--- a/cranelift/codegen/meta-python/isa/arm32/registers.py
+++ /dev/null
@@ -1,45 +0,0 @@
-"""
-ARM32 register banks.
-"""
-from __future__ import absolute_import
-from cdsl.registers import RegBank, RegClass
-from .defs import ISA
-
-
-# Define the larger float bank first to avoid the alignment gap.
-FloatRegs = RegBank(
-        'FloatRegs', ISA, r"""
-        Floating point registers.
-
-        The floating point register units correspond to the S-registers, but
-        extended as if there were 64 registers.
-
-        - S registers are one unit each.
-        - D registers are two units each, even D16 and above.
-        - Q registers are 4 units each.
-        """,
-        units=64, prefix='s')
-
-# Special register units:
-# - r15 is the program counter.
-# - r14 is the link register.
-# - r13 is usually the stack pointer.
-IntRegs = RegBank(
-        'IntRegs', ISA,
-        'General purpose registers',
-        units=16, prefix='r')
-
-FlagRegs = RegBank(
-        'FlagRegs', ISA,
-        'Flag registers',
-        units=1,
-        pressure_tracking=False,
-        names=['nzcv'])
-
-GPR = RegClass(IntRegs)
-S = RegClass(FloatRegs, count=32)
-D = RegClass(FloatRegs, width=2)
-Q = RegClass(FloatRegs, width=4)
-FLAG = RegClass(FlagRegs)
-
-RegClass.extract_names(globals())
diff --git a/cranelift/codegen/meta-python/isa/arm32/settings.py b/cranelift/codegen/meta-python/isa/arm32/settings.py
deleted file mode 100644
index 5cc948cf2d..0000000000
--- a/cranelift/codegen/meta-python/isa/arm32/settings.py
+++ /dev/null
@@ -1,11 +0,0 @@
-"""
-ARM32 settings.
-"""
-from __future__ import absolute_import
-from cdsl.settings import SettingGroup
-import base.settings as shared
-from .defs import ISA
-
-ISA.settings = SettingGroup('arm32', parent=shared.group)
-
-ISA.settings.close(globals())
diff --git a/cranelift/codegen/meta-python/isa/arm64/__init__.py b/cranelift/codegen/meta-python/isa/arm64/__init__.py
deleted file mode 100644
index fb9005c037..0000000000
--- a/cranelift/codegen/meta-python/isa/arm64/__init__.py
+++ /dev/null
@@ -1,14 +0,0 @@
-"""
-ARM 64-bit Architecture
------------------------
-
-ARMv8 CPUs running the Aarch64 architecture.
-"""
-
-from __future__ import absolute_import
-from . import defs
-from . import settings, registers  # noqa
-from cdsl.isa import TargetISA  # noqa
-
-# Re-export the primary target ISA definition.
-ISA = defs.ISA.finish()  # type: TargetISA
diff --git a/cranelift/codegen/meta-python/isa/arm64/defs.py b/cranelift/codegen/meta-python/isa/arm64/defs.py
deleted file mode 100644
index 0350908f9b..0000000000
--- a/cranelift/codegen/meta-python/isa/arm64/defs.py
+++ /dev/null
@@ -1,15 +0,0 @@
-"""
-ARM64 definitions.
-
-Commonly used definitions.
-"""
-from __future__ import absolute_import
-from cdsl.isa import TargetISA, CPUMode
-import base.instructions
-from base.legalize import narrow
-
-ISA = TargetISA('arm64', [base.instructions.GROUP])  # type: TargetISA
-A64 = CPUMode('A64', ISA)
-
-# TODO: Refine these
-A64.legalize_type(narrow)
diff --git a/cranelift/codegen/meta-python/isa/arm64/registers.py b/cranelift/codegen/meta-python/isa/arm64/registers.py
deleted file mode 100644
index df680b1a14..0000000000
--- a/cranelift/codegen/meta-python/isa/arm64/registers.py
+++ /dev/null
@@ -1,32 +0,0 @@
-"""
-Aarch64 register banks.
-"""
-from __future__ import absolute_import
-from cdsl.registers import RegBank, RegClass
-from .defs import ISA
-
-
-# The `x31` regunit serves as the stack pointer / zero register depending on
-# context. We reserve it and don't model the difference.
-IntRegs = RegBank(
-        'IntRegs', ISA,
-        'General purpose registers',
-        units=32, prefix='x')
-
-FloatRegs = RegBank(
-        'FloatRegs', ISA,
-        'Floating point registers',
-        units=32, prefix='v')
-
-FlagRegs = RegBank(
-        'FlagRegs', ISA,
-        'Flag registers',
-        units=1,
-        pressure_tracking=False,
-        names=['nzcv'])
-
-GPR = RegClass(IntRegs)
-FPR = RegClass(FloatRegs)
-FLAG = RegClass(FlagRegs)
-
-RegClass.extract_names(globals())
diff --git a/cranelift/codegen/meta-python/isa/arm64/settings.py b/cranelift/codegen/meta-python/isa/arm64/settings.py
deleted file mode 100644
index 9a2fc13dc7..0000000000
--- a/cranelift/codegen/meta-python/isa/arm64/settings.py
+++ /dev/null
@@ -1,11 +0,0 @@
-"""
-ARM64 settings.
-"""
-from __future__ import absolute_import
-from cdsl.settings import SettingGroup
-import base.settings as shared
-from .defs import ISA
-
-ISA.settings = SettingGroup('arm64', parent=shared.group)
-
-ISA.settings.close(globals())
diff --git a/cranelift/codegen/meta-python/isa/riscv/__init__.py b/cranelift/codegen/meta-python/isa/riscv/__init__.py
deleted file mode 100644
index b58dd68ae2..0000000000
--- a/cranelift/codegen/meta-python/isa/riscv/__init__.py
+++ /dev/null
@@ -1,33 +0,0 @@
-"""
-RISC-V Target
--------------
-
-`RISC-V `_ is an open instruction set architecture
-originally developed at UC Berkeley. It is a RISC-style ISA with either a
-32-bit (RV32I) or 64-bit (RV32I) base instruction set and a number of optional
-extensions:
-
-RV32M / RV64M
-    Integer multiplication and division.
-
-RV32A / RV64A
-    Atomics.
-
-RV32F / RV64F
-    Single-precision IEEE floating point.
-
-RV32D / RV64D
-    Double-precision IEEE floating point.
-
-RV32G / RV64G
-    General purpose instruction sets. This represents the union of the I, M, A,
-    F, and D instruction sets listed above.
-
-"""
-from __future__ import absolute_import
-from . import defs
-from . import encodings, settings, registers  # noqa
-from cdsl.isa import TargetISA  # noqa
-
-# Re-export the primary target ISA definition.
-ISA = defs.ISA.finish()  # type: TargetISA
diff --git a/cranelift/codegen/meta-python/isa/riscv/defs.py b/cranelift/codegen/meta-python/isa/riscv/defs.py
deleted file mode 100644
index 404895c502..0000000000
--- a/cranelift/codegen/meta-python/isa/riscv/defs.py
+++ /dev/null
@@ -1,14 +0,0 @@
-"""
-RISC-V definitions.
-
-Commonly used definitions.
-"""
-from __future__ import absolute_import
-from cdsl.isa import TargetISA, CPUMode
-import base.instructions
-
-ISA = TargetISA('riscv', [base.instructions.GROUP])  # type: TargetISA
-
-# CPU modes for 32-bit and 64-bit operation.
-RV32 = CPUMode('RV32', ISA)
-RV64 = CPUMode('RV64', ISA)
diff --git a/cranelift/codegen/meta-python/isa/riscv/encodings.py b/cranelift/codegen/meta-python/isa/riscv/encodings.py
deleted file mode 100644
index 980a88ba00..0000000000
--- a/cranelift/codegen/meta-python/isa/riscv/encodings.py
+++ /dev/null
@@ -1,169 +0,0 @@
-"""
-RISC-V Encodings.
-"""
-from __future__ import absolute_import
-from base import instructions as base
-from base import types
-from base.immediates import intcc
-from .defs import RV32, RV64
-from .recipes import OPIMM, OPIMM32, OP, OP32, LUI, BRANCH, JALR, JAL
-from .recipes import LOAD, STORE
-from .recipes import R, Rshamt, Ricmp, Ii, Iz, Iicmp, Iret, Icall, Icopy
-from .recipes import U, UJ, UJcall, SB, SBzero, GPsp, GPfi, Irmov, stacknull
-from .settings import use_m
-from cdsl.ast import Var
-from base.legalize import narrow, expand
-
-RV32.legalize_monomorphic(expand)
-RV32.legalize_type(
-        default=narrow,
-        i32=expand,
-        f32=expand,
-        f64=expand)
-
-RV64.legalize_monomorphic(expand)
-RV64.legalize_type(
-        default=narrow,
-        i32=expand,
-        i64=expand,
-        f32=expand,
-        f64=expand)
-
-# Dummies for instruction predicates.
-x = Var('x')
-y = Var('y')
-dest = Var('dest')
-args = Var('args')
-
-# Basic arithmetic binary instructions are encoded in an R-type instruction.
-for inst,           inst_imm,      f3,    f7 in [
-        (base.iadd, base.iadd_imm, 0b000, 0b0000000),
-        (base.isub, None,          0b000, 0b0100000),
-        (base.bxor, base.bxor_imm, 0b100, 0b0000000),
-        (base.bor,  base.bor_imm,  0b110, 0b0000000),
-        (base.band, base.band_imm, 0b111, 0b0000000)
-        ]:
-    RV32.enc(inst.i32, R, OP(f3, f7))
-    RV64.enc(inst.i64, R, OP(f3, f7))
-
-    # Immediate versions for add/xor/or/and.
-    if inst_imm:
-        RV32.enc(inst_imm.i32, Ii, OPIMM(f3))
-        RV64.enc(inst_imm.i64, Ii, OPIMM(f3))
-
-# 32-bit ops in RV64.
-RV64.enc(base.iadd.i32, R, OP32(0b000, 0b0000000))
-RV64.enc(base.isub.i32, R, OP32(0b000, 0b0100000))
-# There are no andiw/oriw/xoriw variations.
-RV64.enc(base.iadd_imm.i32, Ii, OPIMM32(0b000))
-
-# Use iadd_imm with %x0 to materialize constants.
-RV32.enc(base.iconst.i32, Iz, OPIMM(0b000))
-RV64.enc(base.iconst.i32, Iz, OPIMM(0b000))
-RV64.enc(base.iconst.i64, Iz, OPIMM(0b000))
-
-# Dynamic shifts have the same masking semantics as the clif base instructions.
-for inst,           inst_imm,      f3,    f7 in [
-        (base.ishl, base.ishl_imm, 0b001, 0b0000000),
-        (base.ushr, base.ushr_imm, 0b101, 0b0000000),
-        (base.sshr, base.sshr_imm, 0b101, 0b0100000),
-        ]:
-    RV32.enc(inst.i32.i32, R, OP(f3, f7))
-    RV64.enc(inst.i64.i64, R, OP(f3, f7))
-    RV64.enc(inst.i32.i32, R, OP32(f3, f7))
-    # Allow i32 shift amounts in 64-bit shifts.
-    RV64.enc(inst.i64.i32, R, OP(f3, f7))
-    RV64.enc(inst.i32.i64, R, OP32(f3, f7))
-
-    # Immediate shifts.
-    RV32.enc(inst_imm.i32, Rshamt, OPIMM(f3, f7))
-    RV64.enc(inst_imm.i64, Rshamt, OPIMM(f3, f7))
-    RV64.enc(inst_imm.i32, Rshamt, OPIMM32(f3, f7))
-
-# Signed and unsigned integer 'less than'. There are no 'w' variants for
-# comparing 32-bit numbers in RV64.
-RV32.enc(base.icmp.i32(intcc.slt, x, y), Ricmp, OP(0b010, 0b0000000))
-RV64.enc(base.icmp.i64(intcc.slt, x, y), Ricmp, OP(0b010, 0b0000000))
-RV32.enc(base.icmp.i32(intcc.ult, x, y), Ricmp, OP(0b011, 0b0000000))
-RV64.enc(base.icmp.i64(intcc.ult, x, y), Ricmp, OP(0b011, 0b0000000))
-
-RV32.enc(base.icmp_imm.i32(intcc.slt, x, y), Iicmp, OPIMM(0b010))
-RV64.enc(base.icmp_imm.i64(intcc.slt, x, y), Iicmp, OPIMM(0b010))
-RV32.enc(base.icmp_imm.i32(intcc.ult, x, y), Iicmp, OPIMM(0b011))
-RV64.enc(base.icmp_imm.i64(intcc.ult, x, y), Iicmp, OPIMM(0b011))
-
-# Integer constants with the low 12 bits clear are materialized by lui.
-RV32.enc(base.iconst.i32, U, LUI())
-RV64.enc(base.iconst.i32, U, LUI())
-RV64.enc(base.iconst.i64, U, LUI())
-
-# "M" Standard Extension for Integer Multiplication and Division.
-# Gated by the `use_m` flag.
-RV32.enc(base.imul.i32, R, OP(0b000, 0b0000001), isap=use_m)
-RV64.enc(base.imul.i64, R, OP(0b000, 0b0000001), isap=use_m)
-RV64.enc(base.imul.i32, R, OP32(0b000, 0b0000001), isap=use_m)
-
-# Control flow.
-
-# Unconditional branches.
-RV32.enc(base.jump, UJ, JAL())
-RV64.enc(base.jump, UJ, JAL())
-RV32.enc(base.call, UJcall, JAL())
-RV64.enc(base.call, UJcall, JAL())
-
-# Conditional branches.
-for cond,           f3 in [
-        (intcc.eq,  0b000),
-        (intcc.ne,  0b001),
-        (intcc.slt, 0b100),
-        (intcc.sge, 0b101),
-        (intcc.ult, 0b110),
-        (intcc.uge, 0b111)
-        ]:
-    RV32.enc(base.br_icmp.i32(cond, x, y, dest, args), SB, BRANCH(f3))
-    RV64.enc(base.br_icmp.i64(cond, x, y, dest, args), SB, BRANCH(f3))
-
-for inst,           f3 in [
-        (base.brz,  0b000),
-        (base.brnz, 0b001)
-        ]:
-    RV32.enc(inst.i32, SBzero, BRANCH(f3))
-    RV64.enc(inst.i64, SBzero, BRANCH(f3))
-    RV32.enc(inst.b1, SBzero, BRANCH(f3))
-    RV64.enc(inst.b1, SBzero, BRANCH(f3))
-
-# Returns are a special case of JALR using %x1 to hold the return address.
-# The return address is provided by a special-purpose `link` return value that
-# is added by legalize_signature().
-RV32.enc(base.x_return, Iret, JALR())
-RV64.enc(base.x_return, Iret, JALR())
-RV32.enc(base.call_indirect.i32, Icall, JALR())
-RV64.enc(base.call_indirect.i64, Icall, JALR())
-
-# Spill and fill.
-RV32.enc(base.spill.i32, GPsp, STORE(0b010))
-RV64.enc(base.spill.i32, GPsp, STORE(0b010))
-RV64.enc(base.spill.i64, GPsp, STORE(0b011))
-RV32.enc(base.fill.i32, GPfi, LOAD(0b010))
-RV64.enc(base.fill.i32, GPfi, LOAD(0b010))
-RV64.enc(base.fill.i64, GPfi, LOAD(0b011))
-
-# Register copies.
-RV32.enc(base.copy.i32, Icopy, OPIMM(0b000))
-RV64.enc(base.copy.i64, Icopy, OPIMM(0b000))
-RV64.enc(base.copy.i32, Icopy, OPIMM32(0b000))
-
-RV32.enc(base.regmove.i32, Irmov, OPIMM(0b000))
-RV64.enc(base.regmove.i64, Irmov, OPIMM(0b000))
-RV64.enc(base.regmove.i32, Irmov, OPIMM32(0b000))
-
-RV32.enc(base.copy.b1, Icopy, OPIMM(0b000))
-RV64.enc(base.copy.b1, Icopy, OPIMM(0b000))
-RV32.enc(base.regmove.b1, Irmov, OPIMM(0b000))
-RV64.enc(base.regmove.b1, Irmov, OPIMM(0b000))
-
-# Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
-# into a no-op.
-for ty in [types.i64, types.i32, types.i16, types.i8, types.f64, types.f32]:
-    RV64.enc(base.copy_nop.bind(ty), stacknull, 0)
-    RV32.enc(base.copy_nop.bind(ty), stacknull, 0)
diff --git a/cranelift/codegen/meta-python/isa/riscv/recipes.py b/cranelift/codegen/meta-python/isa/riscv/recipes.py
deleted file mode 100644
index ff27c31f8f..0000000000
--- a/cranelift/codegen/meta-python/isa/riscv/recipes.py
+++ /dev/null
@@ -1,230 +0,0 @@
-"""
-RISC-V Encoding recipes.
-
-The encoding recipes defined here more or less correspond to the RISC-V native
-instruction formats described in the reference:
-
-    The RISC-V Instruction Set Manual
-    Volume I: User-Level ISA
-    Version 2.1
-"""
-from __future__ import absolute_import
-from cdsl.isa import EncRecipe
-from cdsl.predicates import IsSignedInt
-from cdsl.registers import Stack
-from base.formats import Binary, BinaryImm, MultiAry, IntCompare, IntCompareImm
-from base.formats import Unary, UnaryImm, BranchIcmp, Branch, Jump
-from base.formats import Call, CallIndirect, RegMove
-from .registers import GPR
-
-# The low 7 bits of a RISC-V instruction is the base opcode. All 32-bit
-# instructions have 11 as the two low bits, with bits 6:2 determining the base
-# opcode.
-#
-# Encbits for the 32-bit recipes are opcode[6:2] | (funct3 << 5) | ...
-# The functions below encode the encbits.
-
-
-def LOAD(funct3):
-    # type: (int) -> int
-    assert funct3 <= 0b111
-    return 0b00000 | (funct3 << 5)
-
-
-def STORE(funct3):
-    # type: (int) -> int
-    assert funct3 <= 0b111
-    return 0b01000 | (funct3 << 5)
-
-
-def BRANCH(funct3):
-    # type: (int) -> int
-    assert funct3 <= 0b111
-    return 0b11000 | (funct3 << 5)
-
-
-def JALR(funct3=0):
-    # type: (int) -> int
-    assert funct3 <= 0b111
-    return 0b11001 | (funct3 << 5)
-
-
-def JAL():
-    # type: () -> int
-    return 0b11011
-
-
-def OPIMM(funct3, funct7=0):
-    # type: (int, int) -> int
-    assert funct3 <= 0b111
-    return 0b00100 | (funct3 << 5) | (funct7 << 8)
-
-
-def OPIMM32(funct3, funct7=0):
-    # type: (int, int) -> int
-    assert funct3 <= 0b111
-    return 0b00110 | (funct3 << 5) | (funct7 << 8)
-
-
-def OP(funct3, funct7):
-    # type: (int, int) -> int
-    assert funct3 <= 0b111
-    assert funct7 <= 0b1111111
-    return 0b01100 | (funct3 << 5) | (funct7 << 8)
-
-
-def OP32(funct3, funct7):
-    # type: (int, int) -> int
-    assert funct3 <= 0b111
-    assert funct7 <= 0b1111111
-    return 0b01110 | (funct3 << 5) | (funct7 << 8)
-
-
-def AIUPC():
-    # type: () -> int
-    return 0b00101
-
-
-def LUI():
-    # type: () -> int
-    return 0b01101
-
-
-# R-type 32-bit instructions: These are mostly binary arithmetic instructions.
-# The encbits are `opcode[6:2] | (funct3 << 5) | (funct7 << 8)
-R = EncRecipe(
-        'R', Binary, base_size=4, ins=(GPR, GPR), outs=GPR,
-        emit='put_r(bits, in_reg0, in_reg1, out_reg0, sink);')
-
-# R-type with an immediate shift amount instead of rs2.
-Rshamt = EncRecipe(
-        'Rshamt', BinaryImm, base_size=4, ins=GPR, outs=GPR,
-        emit='put_rshamt(bits, in_reg0, imm.into(), out_reg0, sink);')
-
-# R-type encoding of an integer comparison.
-Ricmp = EncRecipe(
-        'Ricmp', IntCompare, base_size=4, ins=(GPR, GPR), outs=GPR,
-        emit='put_r(bits, in_reg0, in_reg1, out_reg0, sink);')
-
-Ii = EncRecipe(
-        'Ii', BinaryImm, base_size=4, ins=GPR, outs=GPR,
-        instp=IsSignedInt(BinaryImm.imm, 12),
-        emit='put_i(bits, in_reg0, imm.into(), out_reg0, sink);')
-
-# I-type instruction with a hardcoded %x0 rs1.
-Iz = EncRecipe(
-        'Iz', UnaryImm, base_size=4, ins=(), outs=GPR,
-        instp=IsSignedInt(UnaryImm.imm, 12),
-        emit='put_i(bits, 0, imm.into(), out_reg0, sink);')
-
-# I-type encoding of an integer comparison.
-Iicmp = EncRecipe(
-        'Iicmp', IntCompareImm, base_size=4, ins=GPR, outs=GPR,
-        instp=IsSignedInt(IntCompareImm.imm, 12),
-        emit='put_i(bits, in_reg0, imm.into(), out_reg0, sink);')
-
-# I-type encoding for `jalr` as a return instruction. We won't use the
-# immediate offset.
-# The variable return values are not encoded.
-Iret = EncRecipe(
-        'Iret', MultiAry, base_size=4, ins=(), outs=(),
-        emit='''
-        // Return instructions are always a jalr to %x1.
-        // The return address is provided as a special-purpose link argument.
-        put_i(
-            bits,
-            1, // rs1 = %x1
-            0, // no offset.
-            0, // rd = %x0: no address written.
-            sink,
-        );
-        ''')
-
-# I-type encoding for `jalr` as a call_indirect.
-Icall = EncRecipe(
-        'Icall', CallIndirect, base_size=4, ins=GPR, outs=(),
-        emit='''
-        // call_indirect instructions are jalr with rd=%x1.
-        put_i(
-            bits,
-            in_reg0,
-            0, // no offset.
-            1, // rd = %x1: link register.
-            sink,
-        );
-        ''')
-
-
-# Copy of a GPR is implemented as addi x, 0.
-Icopy = EncRecipe(
-        'Icopy', Unary, base_size=4, ins=GPR, outs=GPR,
-        emit='put_i(bits, in_reg0, 0, out_reg0, sink);')
-
-# Same for a GPR regmove.
-Irmov = EncRecipe(
-        'Irmov', RegMove, base_size=4, ins=GPR, outs=(),
-        emit='put_i(bits, src, 0, dst, sink);')
-
-# U-type instructions have a 20-bit immediate that targets bits 12-31.
-U = EncRecipe(
-        'U', UnaryImm, base_size=4, ins=(), outs=GPR,
-        instp=IsSignedInt(UnaryImm.imm, 32, 12),
-        emit='put_u(bits, imm.into(), out_reg0, sink);')
-
-# UJ-type unconditional branch instructions.
-UJ = EncRecipe(
-        'UJ', Jump, base_size=4, ins=(), outs=(), branch_range=(0, 21),
-        emit='''
-        let dest = i64::from(func.offsets[destination]);
-        let disp = dest - i64::from(sink.offset());
-        put_uj(bits, disp, 0, sink);
-        ''')
-
-UJcall = EncRecipe(
-        'UJcall', Call, base_size=4, ins=(), outs=(),
-        emit='''
-        sink.reloc_external(Reloc::RiscvCall,
-                            &func.dfg.ext_funcs[func_ref].name,
-                            0);
-        // rd=%x1 is the standard link register.
-        put_uj(bits, 0, 1, sink);
-        ''')
-
-# SB-type branch instructions.
-SB = EncRecipe(
-        'SB', BranchIcmp, base_size=4,
-        ins=(GPR, GPR), outs=(),
-        branch_range=(0, 13),
-        emit='''
-        let dest = i64::from(func.offsets[destination]);
-        let disp = dest - i64::from(sink.offset());
-        put_sb(bits, disp, in_reg0, in_reg1, sink);
-        ''')
-
-# SB-type branch instruction with rs2 fixed to zero.
-SBzero = EncRecipe(
-        'SBzero', Branch, base_size=4,
-        ins=(GPR), outs=(),
-        branch_range=(0, 13),
-        emit='''
-        let dest = i64::from(func.offsets[destination]);
-        let disp = dest - i64::from(sink.offset());
-        put_sb(bits, disp, in_reg0, 0, sink);
-        ''')
-
-# Spill of a GPR.
-GPsp = EncRecipe(
-        'GPsp', Unary, base_size=4,
-        ins=GPR, outs=Stack(GPR),
-        emit='unimplemented!();')
-
-# Fill of a GPR.
-GPfi = EncRecipe(
-        'GPfi', Unary, base_size=4,
-        ins=Stack(GPR), outs=GPR,
-        emit='unimplemented!();')
-
-# Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
-# into a no-op.
-stacknull = EncRecipe('stacknull', Unary, base_size=0,
-                      ins=Stack(GPR), outs=Stack(GPR), emit='')
diff --git a/cranelift/codegen/meta-python/isa/riscv/registers.py b/cranelift/codegen/meta-python/isa/riscv/registers.py
deleted file mode 100644
index d9d43f0432..0000000000
--- a/cranelift/codegen/meta-python/isa/riscv/registers.py
+++ /dev/null
@@ -1,23 +0,0 @@
-"""
-RISC-V register banks.
-"""
-from __future__ import absolute_import
-from cdsl.registers import RegBank, RegClass
-from .defs import ISA
-
-
-# We include `x0`, a.k.a `zero` in the register bank. It will be reserved.
-IntRegs = RegBank(
-        'IntRegs', ISA,
-        'General purpose registers',
-        units=32, prefix='x')
-
-FloatRegs = RegBank(
-        'FloatRegs', ISA,
-        'Floating point registers',
-        units=32, prefix='f')
-
-GPR = RegClass(IntRegs)
-FPR = RegClass(FloatRegs)
-
-RegClass.extract_names(globals())
diff --git a/cranelift/codegen/meta-python/isa/riscv/settings.py b/cranelift/codegen/meta-python/isa/riscv/settings.py
deleted file mode 100644
index c8b88db55e..0000000000
--- a/cranelift/codegen/meta-python/isa/riscv/settings.py
+++ /dev/null
@@ -1,31 +0,0 @@
-"""
-RISC-V settings.
-"""
-from __future__ import absolute_import
-from cdsl.settings import SettingGroup, BoolSetting
-from cdsl.predicates import And
-import base.settings as shared
-from .defs import ISA
-
-ISA.settings = SettingGroup('riscv', parent=shared.group)
-
-supports_m = BoolSetting("CPU supports the 'M' extension (mul/div)")
-supports_a = BoolSetting("CPU supports the 'A' extension (atomics)")
-supports_f = BoolSetting("CPU supports the 'F' extension (float)")
-supports_d = BoolSetting("CPU supports the 'D' extension (double)")
-
-enable_m = BoolSetting(
-        "Enable the use of 'M' instructions if available",
-        default=True)
-
-enable_e = BoolSetting(
-        "Enable the 'RV32E' instruction set with only 16 registers")
-
-use_m = And(supports_m, enable_m)
-use_a = And(supports_a, shared.enable_atomics)
-use_f = And(supports_f, shared.enable_float)
-use_d = And(supports_d, shared.enable_float)
-
-full_float = And(shared.enable_simd, supports_f, supports_d)
-
-ISA.settings.close(globals())
diff --git a/cranelift/codegen/meta-python/isa/x86/__init__.py b/cranelift/codegen/meta-python/isa/x86/__init__.py
deleted file mode 100644
index 9691ce6470..0000000000
--- a/cranelift/codegen/meta-python/isa/x86/__init__.py
+++ /dev/null
@@ -1,22 +0,0 @@
-"""
-x86 Target Architecture
------------------------
-
-This target ISA generates code for x86 CPUs with two separate CPU modes:
-
-`I32`
-    32-bit x86 architecture, also known as 'IA-32', also sometimes referred
-    to as 'i386', however note that Cranelift depends on instructions not
-    in the original `i386`, such as SSE2, CMOVcc, and UD2.
-
-`I64`
-    x86-64 architecture, also known as 'AMD64`, `Intel 64`, and 'x64'.
-"""
-
-from __future__ import absolute_import
-from . import defs
-from . import encodings, settings, registers  # noqa
-from cdsl.isa import TargetISA  # noqa
-
-# Re-export the primary target ISA definition.
-ISA = defs.ISA.finish()  # type: TargetISA
diff --git a/cranelift/codegen/meta-python/isa/x86/defs.py b/cranelift/codegen/meta-python/isa/x86/defs.py
deleted file mode 100644
index 00ac2bbbf1..0000000000
--- a/cranelift/codegen/meta-python/isa/x86/defs.py
+++ /dev/null
@@ -1,28 +0,0 @@
-"""
-x86 definitions.
-
-Commonly used definitions.
-"""
-from __future__ import absolute_import
-from cdsl.isa import TargetISA, CPUMode
-import base.instructions
-from . import instructions as x86
-from base.immediates import floatcc
-
-ISA = TargetISA('x86', [base.instructions.GROUP, x86.GROUP])  # type: TargetISA
-
-# CPU modes for 32-bit and 64-bit operation.
-X86_64 = CPUMode('I64', ISA)
-X86_32 = CPUMode('I32', ISA)
-
-# The set of floating point condition codes that are directly supported.
-# Other condition codes need to be reversed or expressed as two tests.
-supported_floatccs = [
-        floatcc.ord,
-        floatcc.uno,
-        floatcc.one,
-        floatcc.ueq,
-        floatcc.gt,
-        floatcc.ge,
-        floatcc.ult,
-        floatcc.ule]
diff --git a/cranelift/codegen/meta-python/isa/x86/encodings.py b/cranelift/codegen/meta-python/isa/x86/encodings.py
deleted file mode 100644
index 8da4912e77..0000000000
--- a/cranelift/codegen/meta-python/isa/x86/encodings.py
+++ /dev/null
@@ -1,771 +0,0 @@
-"""
-x86 Encodings.
-"""
-from __future__ import absolute_import
-from cdsl.predicates import IsZero32BitFloat, IsZero64BitFloat
-from cdsl.predicates import IsUnsignedInt
-from base.predicates import IsColocatedFunc, IsColocatedData, LengthEquals
-from base import instructions as base
-from base import types
-from base.formats import UnaryIeee32, UnaryIeee64, UnaryImm
-from base.formats import FuncAddr, Call, LoadComplex, StoreComplex
-from .defs import X86_64, X86_32
-from . import recipes as r
-from . import settings as cfg
-from . import instructions as x86
-from .legalize import x86_expand
-from base.legalize import narrow, widen, expand_flags
-from .settings import use_sse41, not_all_ones_funcaddrs_and_not_is_pic, \
-    all_ones_funcaddrs_and_not_is_pic, is_pic, not_is_pic
-
-try:
-    from typing import TYPE_CHECKING, Any  # noqa
-    if TYPE_CHECKING:
-        from cdsl.instructions import MaybeBoundInst  # noqa
-        from cdsl.predicates import FieldPredicate # noqa
-except ImportError:
-    pass
-
-
-X86_32.legalize_monomorphic(expand_flags)
-X86_32.legalize_type(
-    default=narrow,
-    b1=expand_flags,
-    i8=widen,
-    i16=widen,
-    i32=x86_expand,
-    f32=x86_expand,
-    f64=x86_expand)
-
-X86_64.legalize_monomorphic(expand_flags)
-X86_64.legalize_type(
-    default=narrow,
-    b1=expand_flags,
-    i8=widen,
-    i16=widen,
-    i32=x86_expand,
-    i64=x86_expand,
-    f32=x86_expand,
-    f64=x86_expand)
-
-
-#
-# Helper functions for generating encodings.
-#
-
-def enc_x86_64(inst, recipe, *args, **kwargs):
-    # type: (MaybeBoundInst, r.TailRecipe, *int, **int) -> None
-    """
-    Add encodings for `inst` to X86_64 with and without a REX prefix.
-    """
-    X86_64.enc(inst, *recipe.rex(*args, **kwargs))
-    X86_64.enc(inst, *recipe(*args, **kwargs))
-
-
-def enc_x86_64_instp(inst, recipe, instp, *args, **kwargs):
-    # type: (MaybeBoundInst, r.TailRecipe, FieldPredicate, *int, **int) -> None
-    """
-    Add encodings for `inst` to X86_64 with and without a REX prefix.
-    """
-    X86_64.enc(inst, *recipe.rex(*args, **kwargs), instp=instp)
-    X86_64.enc(inst, *recipe(*args, **kwargs), instp=instp)
-
-
-def enc_both(inst, recipe, *args, **kwargs):
-    # type: (MaybeBoundInst, r.TailRecipe, *int, **Any) -> None
-    """
-    Add encodings for `inst` to both X86_32 and X86_64.
-    """
-    X86_32.enc(inst, *recipe(*args, **kwargs))
-    enc_x86_64(inst, recipe, *args, **kwargs)
-
-
-def enc_both_instp(inst, recipe, instp, *args, **kwargs):
-    # type: (MaybeBoundInst, r.TailRecipe, FieldPredicate, *int, **Any) -> None
-    """
-    Add encodings for `inst` to both X86_32 and X86_64.
-    """
-    X86_32.enc(inst, *recipe(*args, **kwargs), instp=instp)
-    enc_x86_64_instp(inst, recipe, instp, *args, **kwargs)
-
-
-def enc_i32_i64(inst, recipe, *args, **kwargs):
-    # type: (MaybeBoundInst, r.TailRecipe, *int, **int) -> None
-    """
-    Add encodings for `inst.i32` to X86_32.
-    Add encodings for `inst.i32` to X86_64 with and without REX.
-    Add encodings for `inst.i64` to X86_64 with a REX.W prefix.
-    """
-    X86_32.enc(inst.i32, *recipe(*args, **kwargs))
-
-    # REX-less encoding must come after REX encoding so we don't use it by
-    # default. Otherwise reg-alloc would never use r8 and up.
-    X86_64.enc(inst.i32, *recipe.rex(*args, **kwargs))
-    X86_64.enc(inst.i32, *recipe(*args, **kwargs))
-
-    X86_64.enc(inst.i64, *recipe.rex(*args, w=1, **kwargs))
-
-
-def enc_i32_i64_instp(inst, recipe, instp, *args, **kwargs):
-    # type: (MaybeBoundInst, r.TailRecipe, FieldPredicate, *int, **int) -> None
-    """
-    Add encodings for `inst.i32` to X86_32.
-    Add encodings for `inst.i32` to X86_64 with and without REX.
-    Add encodings for `inst.i64` to X86_64 with a REX.W prefix.
-
-    Similar to `enc_i32_i64` but applies `instp` to each encoding.
-    """
-    X86_32.enc(inst.i32, *recipe(*args, **kwargs), instp=instp)
-
-    # REX-less encoding must come after REX encoding so we don't use it by
-    # default. Otherwise reg-alloc would never use r8 and up.
-    X86_64.enc(inst.i32, *recipe.rex(*args, **kwargs), instp=instp)
-    X86_64.enc(inst.i32, *recipe(*args, **kwargs), instp=instp)
-
-    X86_64.enc(inst.i64, *recipe.rex(*args, w=1, **kwargs), instp=instp)
-
-
-def enc_i32_i64_ld_st(inst, w_bit, recipe, *args, **kwargs):
-    # type: (MaybeBoundInst, bool, r.TailRecipe, *int, **int) -> None
-    """
-    Add encodings for `inst.i32` to X86_32.
-    Add encodings for `inst.i32` to X86_64 with and without REX.
-    Add encodings for `inst.i64` to X86_64 with a REX prefix, using the `w_bit`
-    argument to determine whether or not to set the REX.W bit.
-    """
-    X86_32.enc(inst.i32.any, *recipe(*args, **kwargs))
-
-    # REX-less encoding must come after REX encoding so we don't use it by
-    # default. Otherwise reg-alloc would never use r8 and up.
-    X86_64.enc(inst.i32.any, *recipe.rex(*args, **kwargs))
-    X86_64.enc(inst.i32.any, *recipe(*args, **kwargs))
-
-    if w_bit:
-        X86_64.enc(inst.i64.any, *recipe.rex(*args, w=1, **kwargs))
-    else:
-        X86_64.enc(inst.i64.any, *recipe.rex(*args, **kwargs))
-        X86_64.enc(inst.i64.any, *recipe(*args, **kwargs))
-
-
-for inst,           opc in [
-        (base.iadd, 0x01),
-        (base.isub, 0x29),
-        (base.band, 0x21),
-        (base.bor,  0x09),
-        (base.bxor, 0x31)]:
-    enc_i32_i64(inst, r.rr, opc)
-
-# x86 has a bitwise not instruction NOT.
-enc_i32_i64(base.bnot, r.ur, 0xf7, rrr=2)
-
-# Also add a `b1` encodings for the logic instructions.
-# TODO: Should this be done with 8-bit instructions? It would improve
-# partial register dependencies.
-enc_both(base.band.b1, r.rr, 0x21)
-enc_both(base.bor.b1,  r.rr, 0x09)
-enc_both(base.bxor.b1, r.rr, 0x31)
-
-enc_i32_i64(base.imul, r.rrx, 0x0f, 0xaf)
-enc_i32_i64(x86.sdivmodx, r.div, 0xf7, rrr=7)
-enc_i32_i64(x86.udivmodx, r.div, 0xf7, rrr=6)
-
-enc_i32_i64(x86.smulx, r.mulx, 0xf7, rrr=5)
-enc_i32_i64(x86.umulx, r.mulx, 0xf7, rrr=4)
-
-enc_i32_i64(base.copy, r.umr, 0x89)
-for ty in [types.b1, types.i8, types.i16]:
-    enc_both(base.copy.bind(ty), r.umr, 0x89)
-
-# For x86-64, only define REX forms for now, since we can't describe the
-# special regunit immediate operands with the current constraint language.
-for ty in [types.i8, types.i16, types.i32]:
-    X86_32.enc(base.regmove.bind(ty), *r.rmov(0x89))
-    X86_64.enc(base.regmove.bind(ty), *r.rmov.rex(0x89))
-X86_64.enc(base.regmove.i64, *r.rmov.rex(0x89, w=1))
-
-enc_both(base.regmove.b1, r.rmov, 0x89)
-enc_both(base.regmove.i8, r.rmov, 0x89)
-
-# Immediate instructions with sign-extended 8-bit and 32-bit immediate.
-for inst,               rrr in [
-        (base.iadd_imm, 0),
-        (base.band_imm, 4),
-        (base.bor_imm,  1),
-        (base.bxor_imm, 6)]:
-    enc_i32_i64(inst, r.r_ib, 0x83, rrr=rrr)
-    enc_i32_i64(inst, r.r_id, 0x81, rrr=rrr)
-
-# TODO: band_imm.i64 with an unsigned 32-bit immediate can be encoded as
-# band_imm.i32. Can even use the single-byte immediate for 0xffff_ffXX masks.
-
-# Immediate constants.
-X86_32.enc(base.iconst.i32, *r.pu_id(0xb8))
-
-X86_64.enc(base.iconst.i32, *r.pu_id.rex(0xb8))
-X86_64.enc(base.iconst.i32, *r.pu_id(0xb8))
-# The 32-bit immediate movl also zero-extends to 64 bits.
-X86_64.enc(base.iconst.i64, *r.pu_id.rex(0xb8),
-           instp=IsUnsignedInt(UnaryImm.imm, 32))
-X86_64.enc(base.iconst.i64, *r.pu_id(0xb8),
-           instp=IsUnsignedInt(UnaryImm.imm, 32))
-# Sign-extended 32-bit immediate.
-X86_64.enc(base.iconst.i64, *r.u_id.rex(0xc7, rrr=0, w=1))
-# Finally, the 0xb8 opcode takes an 8-byte immediate with a REX.W prefix.
-X86_64.enc(base.iconst.i64, *r.pu_iq.rex(0xb8, w=1))
-
-# bool constants.
-enc_both(base.bconst.b1, r.pu_id_bool, 0xb8)
-
-# Shifts and rotates.
-# Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit
-# and 16-bit shifts would need explicit masking.
-for inst,           rrr in [
-        (base.rotl, 0),
-        (base.rotr, 1),
-        (base.ishl, 4),
-        (base.ushr, 5),
-        (base.sshr, 7)]:
-    # Cannot use enc_i32_i64 for this pattern because instructions require
-    # .any suffix.
-    X86_32.enc(inst.i32.any, *r.rc(0xd3, rrr=rrr))
-    X86_64.enc(inst.i64.any, *r.rc.rex(0xd3, rrr=rrr, w=1))
-    X86_64.enc(inst.i32.any, *r.rc.rex(0xd3, rrr=rrr))
-    X86_64.enc(inst.i32.any, *r.rc(0xd3, rrr=rrr))
-
-for inst,           rrr in [
-        (base.rotl_imm, 0),
-        (base.rotr_imm, 1),
-        (base.ishl_imm, 4),
-        (base.ushr_imm, 5),
-        (base.sshr_imm, 7)]:
-    enc_i32_i64(inst, r.r_ib, 0xc1, rrr=rrr)
-
-# Population count.
-X86_32.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt)
-X86_64.enc(base.popcnt.i64, *r.urm.rex(0xf3, 0x0f, 0xb8, w=1),
-           isap=cfg.use_popcnt)
-X86_64.enc(base.popcnt.i32, *r.urm.rex(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt)
-X86_64.enc(base.popcnt.i32, *r.urm(0xf3, 0x0f, 0xb8), isap=cfg.use_popcnt)
-
-# Count leading zero bits.
-X86_32.enc(base.clz.i32, *r.urm(0xf3, 0x0f, 0xbd), isap=cfg.use_lzcnt)
-X86_64.enc(base.clz.i64, *r.urm.rex(0xf3, 0x0f, 0xbd, w=1),
-           isap=cfg.use_lzcnt)
-X86_64.enc(base.clz.i32, *r.urm.rex(0xf3, 0x0f, 0xbd), isap=cfg.use_lzcnt)
-X86_64.enc(base.clz.i32, *r.urm(0xf3, 0x0f, 0xbd), isap=cfg.use_lzcnt)
-
-# Count trailing zero bits.
-X86_32.enc(base.ctz.i32, *r.urm(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1)
-X86_64.enc(base.ctz.i64, *r.urm.rex(0xf3, 0x0f, 0xbc, w=1),
-           isap=cfg.use_bmi1)
-X86_64.enc(base.ctz.i32, *r.urm.rex(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1)
-X86_64.enc(base.ctz.i32, *r.urm(0xf3, 0x0f, 0xbc), isap=cfg.use_bmi1)
-
-#
-# Loads and stores.
-#
-
-ldcomplexp = LengthEquals(LoadComplex, 2)
-for recipe in [r.ldWithIndex, r.ldWithIndexDisp8, r.ldWithIndexDisp32]:
-    enc_i32_i64_instp(base.load_complex, recipe, ldcomplexp, 0x8b)
-    enc_x86_64_instp(base.uload32_complex, recipe, ldcomplexp, 0x8b)
-    X86_64.enc(base.sload32_complex, *recipe.rex(0x63, w=1),
-               instp=ldcomplexp)
-    enc_i32_i64_instp(base.uload16_complex, recipe, ldcomplexp, 0x0f, 0xb7)
-    enc_i32_i64_instp(base.sload16_complex, recipe, ldcomplexp, 0x0f, 0xbf)
-    enc_i32_i64_instp(base.uload8_complex, recipe, ldcomplexp, 0x0f, 0xb6)
-    enc_i32_i64_instp(base.sload8_complex, recipe, ldcomplexp, 0x0f, 0xbe)
-
-stcomplexp = LengthEquals(StoreComplex, 3)
-for recipe in [r.stWithIndex, r.stWithIndexDisp8, r.stWithIndexDisp32]:
-    enc_i32_i64_instp(base.store_complex, recipe, stcomplexp, 0x89)
-    enc_x86_64_instp(base.istore32_complex, recipe, stcomplexp, 0x89)
-    enc_both_instp(base.istore16_complex.i32, recipe, stcomplexp, 0x66, 0x89)
-    enc_x86_64_instp(base.istore16_complex.i64, recipe, stcomplexp, 0x66, 0x89)
-
-for recipe in [r.stWithIndex_abcd,
-               r.stWithIndexDisp8_abcd,
-               r.stWithIndexDisp32_abcd]:
-    enc_both_instp(base.istore8_complex.i32, recipe, stcomplexp, 0x88)
-    enc_x86_64_instp(base.istore8_complex.i64, recipe, stcomplexp, 0x88)
-
-for recipe in [r.st, r.stDisp8, r.stDisp32]:
-    enc_i32_i64_ld_st(base.store, True, recipe, 0x89)
-    enc_x86_64(base.istore32.i64.any, recipe, 0x89)
-    enc_i32_i64_ld_st(base.istore16, False, recipe, 0x66, 0x89)
-
-# Byte stores are more complicated because the registers they can address
-# depends of the presence of a REX prefix. The st*_abcd recipes fall back to
-# the corresponding st* recipes when a REX prefix is applied.
-for recipe in [r.st_abcd, r.stDisp8_abcd, r.stDisp32_abcd]:
-    enc_both(base.istore8.i32.any, recipe, 0x88)
-    enc_x86_64(base.istore8.i64.any, recipe, 0x88)
-
-enc_i32_i64(base.spill, r.spillSib32, 0x89)
-enc_i32_i64(base.regspill, r.regspill32, 0x89)
-
-# Use a 32-bit write for spilling `b1`, `i8` and `i16` to avoid
-# constraining the permitted registers.
-# See MIN_SPILL_SLOT_SIZE which makes this safe.
-for ty in [types.b1, types.i8, types.i16]:
-    enc_both(base.spill.bind(ty), r.spillSib32, 0x89)
-    enc_both(base.regspill.bind(ty), r.regspill32, 0x89)
-
-for recipe in [r.ld, r.ldDisp8, r.ldDisp32]:
-    enc_i32_i64_ld_st(base.load, True, recipe, 0x8b)
-    enc_x86_64(base.uload32.i64, recipe, 0x8b)
-    X86_64.enc(base.sload32.i64, *recipe.rex(0x63, w=1))
-    enc_i32_i64_ld_st(base.uload16, True, recipe, 0x0f, 0xb7)
-    enc_i32_i64_ld_st(base.sload16, True, recipe, 0x0f, 0xbf)
-    enc_i32_i64_ld_st(base.uload8, True, recipe, 0x0f, 0xb6)
-    enc_i32_i64_ld_st(base.sload8, True, recipe, 0x0f, 0xbe)
-
-enc_i32_i64(base.fill, r.fillSib32, 0x8b)
-enc_i32_i64(base.regfill, r.regfill32, 0x8b)
-
-# Load 32 bits from `b1`, `i8` and `i16` spill slots. See `spill.b1` above.
-for ty in [types.b1, types.i8, types.i16]:
-    enc_both(base.fill.bind(ty), r.fillSib32, 0x8b)
-    enc_both(base.regfill.bind(ty), r.regfill32, 0x8b)
-
-# Push and Pop
-X86_32.enc(x86.push.i32, *r.pushq(0x50))
-enc_x86_64(x86.push.i64, r.pushq, 0x50)
-
-X86_32.enc(x86.pop.i32, *r.popq(0x58))
-enc_x86_64(x86.pop.i64, r.popq, 0x58)
-
-# Copy Special
-# For x86-64, only define REX forms for now, since we can't describe the
-# special regunit immediate operands with the current constraint language.
-X86_64.enc(base.copy_special, *r.copysp.rex(0x89, w=1))
-X86_32.enc(base.copy_special, *r.copysp(0x89))
-
-# Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
-# into a no-op.  Ideally we could to make this encoding available for
-# all types, and write `base.copy_nop.any`, but it appears that the
-# controlling type variable must not polymorphic.  So we make do with
-# the following limited set, and guard the generating transformation in
-# regalloc/reload.rs accordingly.
-#
-# The same encoding is generated for both the 64- and 32-bit architectures.
-# Note that we can't use `enc_both` here, because that attempts to create a
-# variant with a REX prefix in the 64-bit-architecture case.  But since
-# there's no actual instruction for the REX prefix to modify the meaning of,
-# it will modify the meaning of whatever instruction happens to follow this
-# one, which is obviously wrong.  Note also that we can and indeed *must*
-# claim that there's a 64-bit encoding for the 32-bit arch case, even though
-# no such single instruction actually exists for the 32-bit arch case.
-for ty in [types.i64, types.i32, types.i16, types.i8, types.f64, types.f32]:
-    X86_64.enc(base.copy_nop.bind(ty), r.stacknull, 0)
-    X86_32.enc(base.copy_nop.bind(ty), r.stacknull, 0)
-
-# Adjust SP down by a dynamic value (or up, with a negative operand).
-X86_32.enc(base.adjust_sp_down.i32, *r.adjustsp(0x29))
-X86_64.enc(base.adjust_sp_down.i64, *r.adjustsp.rex(0x29, w=1))
-
-# Adjust SP up by an immediate (or down, with a negative immediate)
-X86_32.enc(base.adjust_sp_up_imm, *r.adjustsp_ib(0x83))
-X86_32.enc(base.adjust_sp_up_imm, *r.adjustsp_id(0x81))
-X86_64.enc(base.adjust_sp_up_imm, *r.adjustsp_ib.rex(0x83, w=1))
-X86_64.enc(base.adjust_sp_up_imm, *r.adjustsp_id.rex(0x81, w=1))
-
-# Adjust SP down by an immediate (or up, with a negative immediate)
-X86_32.enc(base.adjust_sp_down_imm, *r.adjustsp_ib(0x83, rrr=5))
-X86_32.enc(base.adjust_sp_down_imm, *r.adjustsp_id(0x81, rrr=5))
-X86_64.enc(base.adjust_sp_down_imm, *r.adjustsp_ib.rex(0x83, rrr=5, w=1))
-X86_64.enc(base.adjust_sp_down_imm, *r.adjustsp_id.rex(0x81, rrr=5, w=1))
-
-#
-# Float loads and stores.
-#
-
-enc_both(base.load.f32.any, r.fld, 0xf3, 0x0f, 0x10)
-enc_both(base.load.f32.any, r.fldDisp8, 0xf3, 0x0f, 0x10)
-enc_both(base.load.f32.any, r.fldDisp32, 0xf3, 0x0f, 0x10)
-
-enc_both(base.load_complex.f32, r.fldWithIndex, 0xf3, 0x0f, 0x10)
-enc_both(base.load_complex.f32, r.fldWithIndexDisp8, 0xf3, 0x0f, 0x10)
-enc_both(base.load_complex.f32, r.fldWithIndexDisp32, 0xf3, 0x0f, 0x10)
-
-enc_both(base.load.f64.any, r.fld, 0xf2, 0x0f, 0x10)
-enc_both(base.load.f64.any, r.fldDisp8, 0xf2, 0x0f, 0x10)
-enc_both(base.load.f64.any, r.fldDisp32, 0xf2, 0x0f, 0x10)
-
-enc_both(base.load_complex.f64, r.fldWithIndex, 0xf2, 0x0f, 0x10)
-enc_both(base.load_complex.f64, r.fldWithIndexDisp8, 0xf2, 0x0f, 0x10)
-enc_both(base.load_complex.f64, r.fldWithIndexDisp32, 0xf2, 0x0f, 0x10)
-
-enc_both(base.store.f32.any, r.fst, 0xf3, 0x0f, 0x11)
-enc_both(base.store.f32.any, r.fstDisp8, 0xf3, 0x0f, 0x11)
-enc_both(base.store.f32.any, r.fstDisp32, 0xf3, 0x0f, 0x11)
-
-enc_both(base.store_complex.f32, r.fstWithIndex, 0xf3, 0x0f, 0x11)
-enc_both(base.store_complex.f32, r.fstWithIndexDisp8, 0xf3, 0x0f, 0x11)
-enc_both(base.store_complex.f32, r.fstWithIndexDisp32, 0xf3, 0x0f, 0x11)
-
-enc_both(base.store.f64.any, r.fst, 0xf2, 0x0f, 0x11)
-enc_both(base.store.f64.any, r.fstDisp8, 0xf2, 0x0f, 0x11)
-enc_both(base.store.f64.any, r.fstDisp32, 0xf2, 0x0f, 0x11)
-
-enc_both(base.store_complex.f64, r.fstWithIndex, 0xf2, 0x0f, 0x11)
-enc_both(base.store_complex.f64, r.fstWithIndexDisp8, 0xf2, 0x0f, 0x11)
-enc_both(base.store_complex.f64, r.fstWithIndexDisp32, 0xf2, 0x0f, 0x11)
-
-enc_both(base.fill.f32, r.ffillSib32, 0xf3, 0x0f, 0x10)
-enc_both(base.regfill.f32, r.fregfill32, 0xf3, 0x0f, 0x10)
-enc_both(base.fill.f64, r.ffillSib32, 0xf2, 0x0f, 0x10)
-enc_both(base.regfill.f64, r.fregfill32, 0xf2, 0x0f, 0x10)
-
-enc_both(base.spill.f32, r.fspillSib32, 0xf3, 0x0f, 0x11)
-enc_both(base.regspill.f32, r.fregspill32, 0xf3, 0x0f, 0x11)
-enc_both(base.spill.f64, r.fspillSib32, 0xf2, 0x0f, 0x11)
-enc_both(base.regspill.f64, r.fregspill32, 0xf2, 0x0f, 0x11)
-
-#
-# Function addresses.
-#
-
-# Non-PIC, all-ones funcaddresses.
-X86_32.enc(base.func_addr.i32, *r.fnaddr4(0xb8),
-           isap=not_all_ones_funcaddrs_and_not_is_pic)
-X86_64.enc(base.func_addr.i64, *r.fnaddr8.rex(0xb8, w=1),
-           isap=not_all_ones_funcaddrs_and_not_is_pic)
-
-# Non-PIC, all-zeros funcaddresses.
-X86_32.enc(base.func_addr.i32, *r.allones_fnaddr4(0xb8),
-           isap=all_ones_funcaddrs_and_not_is_pic)
-X86_64.enc(base.func_addr.i64, *r.allones_fnaddr8.rex(0xb8, w=1),
-           isap=all_ones_funcaddrs_and_not_is_pic)
-
-# 64-bit, colocated, both PIC and non-PIC. Use the lea instruction's
-# pc-relative field.
-X86_64.enc(base.func_addr.i64, *r.pcrel_fnaddr8.rex(0x8d, w=1),
-           instp=IsColocatedFunc(FuncAddr.func_ref))
-
-# 64-bit, non-colocated, PIC.
-X86_64.enc(base.func_addr.i64, *r.got_fnaddr8.rex(0x8b, w=1),
-           isap=is_pic)
-
-#
-# Global addresses.
-#
-
-# Non-PIC
-X86_32.enc(base.symbol_value.i32, *r.gvaddr4(0xb8),
-           isap=not_is_pic)
-X86_64.enc(base.symbol_value.i64, *r.gvaddr8.rex(0xb8, w=1),
-           isap=not_is_pic)
-
-# PIC, colocated
-X86_64.enc(base.symbol_value.i64, *r.pcrel_gvaddr8.rex(0x8d, w=1),
-           isap=is_pic,
-           instp=IsColocatedData())
-
-# PIC, non-colocated
-X86_64.enc(base.symbol_value.i64, *r.got_gvaddr8.rex(0x8b, w=1),
-           isap=is_pic)
-
-#
-# Stack addresses.
-#
-# TODO: Add encoding rules for stack_load and stack_store, so that they
-# don't get legalized to stack_addr + load/store.
-#
-X86_32.enc(base.stack_addr.i32, *r.spaddr4_id(0x8d))
-X86_64.enc(base.stack_addr.i64, *r.spaddr8_id.rex(0x8d, w=1))
-
-#
-# Call/return
-#
-
-# 32-bit, both PIC and non-PIC.
-X86_32.enc(base.call, *r.call_id(0xe8))
-
-# 64-bit, colocated, both PIC and non-PIC. Use the call instruction's
-# pc-relative field.
-X86_64.enc(base.call, *r.call_id(0xe8),
-           instp=IsColocatedFunc(Call.func_ref))
-
-# 64-bit, non-colocated, PIC. There is no 64-bit non-colocated non-PIC version,
-# since non-PIC is currently using the large model, which requires calls be
-# lowered to func_addr+call_indirect.
-X86_64.enc(base.call, *r.call_plt_id(0xe8), isap=is_pic)
-
-X86_32.enc(base.call_indirect.i32, *r.call_r(0xff, rrr=2))
-X86_64.enc(base.call_indirect.i64, *r.call_r.rex(0xff, rrr=2))
-X86_64.enc(base.call_indirect.i64, *r.call_r(0xff, rrr=2))
-
-X86_32.enc(base.x_return, *r.ret(0xc3))
-X86_64.enc(base.x_return, *r.ret(0xc3))
-
-#
-# Branches
-#
-X86_32.enc(base.jump, *r.jmpb(0xeb))
-X86_64.enc(base.jump, *r.jmpb(0xeb))
-X86_32.enc(base.jump, *r.jmpd(0xe9))
-X86_64.enc(base.jump, *r.jmpd(0xe9))
-
-enc_both(base.brif, r.brib, 0x70)
-enc_both(base.brif, r.brid, 0x0f, 0x80)
-
-# Not all float condition codes are legal, see `supported_floatccs`.
-enc_both(base.brff, r.brfb, 0x70)
-enc_both(base.brff, r.brfd, 0x0f, 0x80)
-
-# Note that the tjccd opcode will be prefixed with 0x0f.
-enc_i32_i64(base.brz, r.tjccb, 0x74)
-enc_i32_i64(base.brz, r.tjccd, 0x84)
-enc_i32_i64(base.brnz, r.tjccb, 0x75)
-enc_i32_i64(base.brnz, r.tjccd, 0x85)
-
-# Branch on a b1 value in a register only looks at the low 8 bits. See also
-# bint encodings below.
-#
-# Start with the worst-case encoding for X86_32 only. The register allocator
-# can't handle a branch with an ABCD-constrained operand.
-X86_32.enc(base.brz.b1, *r.t8jccd_long(0x84))
-X86_32.enc(base.brnz.b1, *r.t8jccd_long(0x85))
-
-enc_both(base.brz.b1, r.t8jccb_abcd, 0x74)
-enc_both(base.brz.b1, r.t8jccd_abcd, 0x84)
-enc_both(base.brnz.b1, r.t8jccb_abcd, 0x75)
-enc_both(base.brnz.b1, r.t8jccd_abcd, 0x85)
-
-#
-# Jump tables
-#
-X86_64.enc(base.jump_table_entry.i64.any.any, *r.jt_entry.rex(0x63, w=1))
-X86_32.enc(base.jump_table_entry.i32.any.any, *r.jt_entry(0x8b))
-
-X86_64.enc(base.jump_table_base.i64, *r.jt_base.rex(0x8d, w=1))
-X86_32.enc(base.jump_table_base.i32, *r.jt_base(0x8d))
-
-enc_x86_64(base.indirect_jump_table_br.i64, r.indirect_jmp, 0xff, rrr=4)
-X86_32.enc(base.indirect_jump_table_br.i32, *r.indirect_jmp(0xff, rrr=4))
-
-#
-# Trap as ud2
-#
-X86_32.enc(base.trap, *r.trap(0x0f, 0x0b))
-X86_64.enc(base.trap, *r.trap(0x0f, 0x0b))
-
-# Debug trap as int3
-X86_32.enc(base.debugtrap, r.debugtrap, 0)
-X86_64.enc(base.debugtrap, r.debugtrap, 0)
-
-# Using a standard EncRecipe, not the TailRecipe.
-X86_32.enc(base.trapif, r.trapif, 0)
-X86_64.enc(base.trapif, r.trapif, 0)
-X86_32.enc(base.trapff, r.trapff, 0)
-X86_64.enc(base.trapff, r.trapff, 0)
-
-#
-# Comparisons
-#
-enc_i32_i64(base.icmp, r.icscc, 0x39)
-enc_i32_i64(base.icmp_imm, r.icscc_ib, 0x83, rrr=7)
-enc_i32_i64(base.icmp_imm, r.icscc_id, 0x81, rrr=7)
-enc_i32_i64(base.ifcmp, r.rcmp, 0x39)
-enc_i32_i64(base.ifcmp_imm, r.rcmp_ib, 0x83, rrr=7)
-enc_i32_i64(base.ifcmp_imm, r.rcmp_id, 0x81, rrr=7)
-# TODO: We could special-case ifcmp_imm(x, 0) to TEST(x, x).
-
-X86_32.enc(base.ifcmp_sp.i32, *r.rcmp_sp(0x39))
-X86_64.enc(base.ifcmp_sp.i64, *r.rcmp_sp.rex(0x39, w=1))
-
-#
-# Convert flags to bool.
-#
-# This encodes `b1` as an 8-bit low register with the value 0 or 1.
-enc_both(base.trueif, r.seti_abcd, 0x0f, 0x90)
-enc_both(base.trueff, r.setf_abcd, 0x0f, 0x90)
-
-#
-# Conditional move (a.k.a integer select)
-#
-enc_i32_i64(base.selectif, r.cmov, 0x0F, 0x40)
-
-#
-# Bit scan forwards and reverse
-#
-enc_i32_i64(x86.bsf, r.bsf_and_bsr, 0x0F, 0xBC)
-enc_i32_i64(x86.bsr, r.bsf_and_bsr, 0x0F, 0xBD)
-
-#
-# Convert bool to int.
-#
-# This assumes that b1 is represented as an 8-bit low register with the value 0
-# or 1.
-#
-# Encode movzbq as movzbl, because it's equivalent and shorter.
-X86_32.enc(base.bint.i32.b1, *r.urm_noflags_abcd(0x0f, 0xb6))
-X86_64.enc(base.bint.i64.b1, *r.urm_noflags.rex(0x0f, 0xb6))
-X86_64.enc(base.bint.i64.b1, *r.urm_noflags_abcd(0x0f, 0xb6))
-X86_64.enc(base.bint.i32.b1, *r.urm_noflags.rex(0x0f, 0xb6))
-X86_64.enc(base.bint.i32.b1, *r.urm_noflags_abcd(0x0f, 0xb6))
-
-# Numerical conversions.
-
-# Reducing an integer is a no-op.
-X86_32.enc(base.ireduce.i8.i16, r.null, 0)
-X86_32.enc(base.ireduce.i8.i32, r.null, 0)
-X86_32.enc(base.ireduce.i16.i32, r.null, 0)
-
-X86_64.enc(base.ireduce.i8.i16, r.null, 0)
-X86_64.enc(base.ireduce.i8.i32, r.null, 0)
-X86_64.enc(base.ireduce.i16.i32, r.null, 0)
-X86_64.enc(base.ireduce.i8.i64, r.null, 0)
-X86_64.enc(base.ireduce.i16.i64, r.null, 0)
-X86_64.enc(base.ireduce.i32.i64, r.null, 0)
-
-# TODO: Add encodings for cbw, cwde, cdqe, which are sign-extending
-# instructions for %al/%ax/%eax to %ax/%eax/%rax.
-
-# movsbl
-X86_32.enc(base.sextend.i32.i8, *r.urm_noflags_abcd(0x0f, 0xbe))
-X86_64.enc(base.sextend.i32.i8, *r.urm_noflags.rex(0x0f, 0xbe))
-X86_64.enc(base.sextend.i32.i8, *r.urm_noflags_abcd(0x0f, 0xbe))
-
-# movswl
-X86_32.enc(base.sextend.i32.i16, *r.urm_noflags(0x0f, 0xbf))
-X86_64.enc(base.sextend.i32.i16, *r.urm_noflags.rex(0x0f, 0xbf))
-X86_64.enc(base.sextend.i32.i16, *r.urm_noflags(0x0f, 0xbf))
-
-# movsbq
-X86_64.enc(base.sextend.i64.i8, *r.urm_noflags.rex(0x0f, 0xbe, w=1))
-
-# movswq
-X86_64.enc(base.sextend.i64.i16, *r.urm_noflags.rex(0x0f, 0xbf, w=1))
-
-# movslq
-X86_64.enc(base.sextend.i64.i32, *r.urm_noflags.rex(0x63, w=1))
-
-# movzbl
-X86_32.enc(base.uextend.i32.i8, *r.urm_noflags_abcd(0x0f, 0xb6))
-X86_64.enc(base.uextend.i32.i8, *r.urm_noflags.rex(0x0f, 0xb6))
-X86_64.enc(base.uextend.i32.i8, *r.urm_noflags_abcd(0x0f, 0xb6))
-
-# movzwl
-X86_32.enc(base.uextend.i32.i16, *r.urm_noflags(0x0f, 0xb7))
-X86_64.enc(base.uextend.i32.i16, *r.urm_noflags.rex(0x0f, 0xb7))
-X86_64.enc(base.uextend.i32.i16, *r.urm_noflags(0x0f, 0xb7))
-
-# movzbq, encoded as movzbl because it's equivalent and shorter
-X86_64.enc(base.uextend.i64.i8, *r.urm_noflags.rex(0x0f, 0xb6))
-X86_64.enc(base.uextend.i64.i8, *r.urm_noflags_abcd(0x0f, 0xb6))
-
-# movzwq, encoded as movzwl because it's equivalent and shorter
-X86_64.enc(base.uextend.i64.i16, *r.urm_noflags.rex(0x0f, 0xb7))
-X86_64.enc(base.uextend.i64.i16, *r.urm_noflags(0x0f, 0xb7))
-
-# A 32-bit register copy clears the high 32 bits.
-X86_64.enc(base.uextend.i64.i32, *r.umr.rex(0x89))
-X86_64.enc(base.uextend.i64.i32, *r.umr(0x89))
-
-
-#
-# Floating point
-#
-
-# floating-point constants equal to 0.0 can be encoded using either
-# `xorps` or `xorpd`, for 32-bit and 64-bit floats respectively.
-X86_32.enc(base.f32const, *r.f32imm_z(0x0f, 0x57),
-           instp=IsZero32BitFloat(UnaryIeee32.imm))
-X86_32.enc(base.f64const, *r.f64imm_z(0x66, 0x0f, 0x57),
-           instp=IsZero64BitFloat(UnaryIeee64.imm))
-
-enc_x86_64_instp(base.f32const, r.f32imm_z,
-                 IsZero32BitFloat(UnaryIeee32.imm), 0x0f, 0x57)
-enc_x86_64_instp(base.f64const, r.f64imm_z,
-                 IsZero64BitFloat(UnaryIeee64.imm), 0x66, 0x0f, 0x57)
-
-# movd
-enc_both(base.bitcast.f32.i32, r.frurm, 0x66, 0x0f, 0x6e)
-enc_both(base.bitcast.i32.f32, r.rfumr, 0x66, 0x0f, 0x7e)
-
-# movq
-X86_64.enc(base.bitcast.f64.i64, *r.frurm.rex(0x66, 0x0f, 0x6e, w=1))
-X86_64.enc(base.bitcast.i64.f64, *r.rfumr.rex(0x66, 0x0f, 0x7e, w=1))
-
-# movaps
-enc_both(base.copy.f32, r.furm, 0x0f, 0x28)
-enc_both(base.copy.f64, r.furm, 0x0f, 0x28)
-
-# For x86-64, only define REX forms for now, since we can't describe the
-# special regunit immediate operands with the current constraint language.
-X86_32.enc(base.regmove.f32, *r.frmov(0x0f, 0x28))
-X86_64.enc(base.regmove.f32, *r.frmov.rex(0x0f, 0x28))
-
-# For x86-64, only define REX forms for now, since we can't describe the
-# special regunit immediate operands with the current constraint language.
-X86_32.enc(base.regmove.f64, *r.frmov(0x0f, 0x28))
-X86_64.enc(base.regmove.f64, *r.frmov.rex(0x0f, 0x28))
-
-# cvtsi2ss
-enc_i32_i64(base.fcvt_from_sint.f32, r.frurm, 0xf3, 0x0f, 0x2a)
-
-# cvtsi2sd
-enc_i32_i64(base.fcvt_from_sint.f64, r.frurm, 0xf2, 0x0f, 0x2a)
-
-# cvtss2sd
-enc_both(base.fpromote.f64.f32, r.furm, 0xf3, 0x0f, 0x5a)
-
-# cvtsd2ss
-enc_both(base.fdemote.f32.f64, r.furm, 0xf2, 0x0f, 0x5a)
-
-# cvttss2si
-enc_both(x86.cvtt2si.i32.f32, r.rfurm, 0xf3, 0x0f, 0x2c)
-X86_64.enc(x86.cvtt2si.i64.f32, *r.rfurm.rex(0xf3, 0x0f, 0x2c, w=1))
-
-# cvttsd2si
-enc_both(x86.cvtt2si.i32.f64, r.rfurm, 0xf2, 0x0f, 0x2c)
-X86_64.enc(x86.cvtt2si.i64.f64, *r.rfurm.rex(0xf2, 0x0f, 0x2c, w=1))
-
-# Exact square roots.
-enc_both(base.sqrt.f32, r.furm, 0xf3, 0x0f, 0x51)
-enc_both(base.sqrt.f64, r.furm, 0xf2, 0x0f, 0x51)
-
-# Rounding. The recipe looks at the opcode to pick an immediate.
-for inst in [
-        base.nearest,
-        base.floor,
-        base.ceil,
-        base.trunc]:
-    enc_both(inst.f32, r.furmi_rnd, 0x66, 0x0f, 0x3a, 0x0a, isap=use_sse41)
-    enc_both(inst.f64, r.furmi_rnd, 0x66, 0x0f, 0x3a, 0x0b, isap=use_sse41)
-
-
-# Binary arithmetic ops.
-for inst,           opc in [
-        (base.fadd, 0x58),
-        (base.fsub, 0x5c),
-        (base.fmul, 0x59),
-        (base.fdiv, 0x5e),
-        (x86.fmin,  0x5d),
-        (x86.fmax,  0x5f)]:
-    enc_both(inst.f32, r.fa, 0xf3, 0x0f, opc)
-    enc_both(inst.f64, r.fa, 0xf2, 0x0f, opc)
-
-# Binary bitwise ops.
-for inst,               opc in [
-        (base.band,     0x54),
-        (base.bor,      0x56),
-        (base.bxor,     0x57)]:
-    enc_both(inst.f32, r.fa, 0x0f, opc)
-    enc_both(inst.f64, r.fa, 0x0f, opc)
-
-# The `andnps(x,y)` instruction computes `~x&y`, while band_not(x,y)` is `x&~y.
-enc_both(base.band_not.f32, r.fax, 0x0f, 0x55)
-enc_both(base.band_not.f64, r.fax, 0x0f, 0x55)
-
-# Comparisons.
-#
-# This only covers the condition codes in `supported_floatccs`, the rest are
-# handled by legalization patterns.
-enc_both(base.fcmp.f32, r.fcscc, 0x0f, 0x2e)
-enc_both(base.fcmp.f64, r.fcscc, 0x66, 0x0f, 0x2e)
-
-enc_both(base.ffcmp.f32, r.fcmp, 0x0f, 0x2e)
-enc_both(base.ffcmp.f64, r.fcmp, 0x66, 0x0f, 0x2e)
diff --git a/cranelift/codegen/meta-python/isa/x86/instructions.py b/cranelift/codegen/meta-python/isa/x86/instructions.py
deleted file mode 100644
index 6adc2ad689..0000000000
--- a/cranelift/codegen/meta-python/isa/x86/instructions.py
+++ /dev/null
@@ -1,173 +0,0 @@
-"""
-Supplementary instruction definitions for x86.
-
-This module defines additional instructions that are useful only to the x86
-target ISA.
-"""
-
-from base.types import iflags
-from cdsl.operands import Operand
-from cdsl.typevar import TypeVar
-from cdsl.instructions import Instruction, InstructionGroup
-
-
-GROUP = InstructionGroup("x86", "x86-specific instruction set")
-
-iWord = TypeVar('iWord', 'A scalar integer machine word', ints=(32, 64))
-
-nlo = Operand('nlo', iWord, doc='Low part of numerator')
-nhi = Operand('nhi', iWord, doc='High part of numerator')
-d = Operand('d', iWord, doc='Denominator')
-q = Operand('q', iWord, doc='Quotient')
-r = Operand('r', iWord, doc='Remainder')
-
-udivmodx = Instruction(
-        'x86_udivmodx', r"""
-        Extended unsigned division.
-
-        Concatenate the bits in `nhi` and `nlo` to form the numerator.
-        Interpret the bits as an unsigned number and divide by the unsigned
-        denominator `d`. Trap when `d` is zero or if the quotient is larger
-        than the range of the output.
-
-        Return both quotient and remainder.
-        """,
-        ins=(nlo, nhi, d), outs=(q, r), can_trap=True)
-
-sdivmodx = Instruction(
-        'x86_sdivmodx', r"""
-        Extended signed division.
-
-        Concatenate the bits in `nhi` and `nlo` to form the numerator.
-        Interpret the bits as a signed number and divide by the signed
-        denominator `d`. Trap when `d` is zero or if the quotient is outside
-        the range of the output.
-
-        Return both quotient and remainder.
-        """,
-        ins=(nlo, nhi, d), outs=(q, r), can_trap=True)
-
-argL = Operand('argL', iWord)
-argR = Operand('argR', iWord)
-resLo = Operand('resLo', iWord)
-resHi = Operand('resHi', iWord)
-
-umulx = Instruction(
-        'x86_umulx', r"""
-        Unsigned integer multiplication, producing a double-length result.
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(argL, argR), outs=(resLo, resHi))
-
-smulx = Instruction(
-        'x86_smulx', r"""
-        Signed integer multiplication, producing a double-length result.
-
-        Polymorphic over all scalar integer types, but does not support vector
-        types.
-        """,
-        ins=(argL, argR), outs=(resLo, resHi))
-
-Float = TypeVar(
-        'Float', 'A scalar or vector floating point number',
-        floats=True, simd=True)
-IntTo = TypeVar(
-        'IntTo', 'An integer type with the same number of lanes',
-        ints=(32, 64), simd=True)
-
-x = Operand('x', Float)
-a = Operand('a', IntTo)
-
-cvtt2si = Instruction(
-        'x86_cvtt2si', r"""
-        Convert with truncation floating point to signed integer.
-
-        The source floating point operand is converted to a signed integer by
-        rounding towards zero. If the result can't be represented in the output
-        type, returns the smallest signed value the output type can represent.
-
-        This instruction does not trap.
-        """,
-        ins=x, outs=a)
-
-x = Operand('x', Float)
-a = Operand('a', Float)
-y = Operand('y', Float)
-
-fmin = Instruction(
-        'x86_fmin', r"""
-        Floating point minimum with x86 semantics.
-
-        This is equivalent to the C ternary operator `x < y ? x : y` which
-        differs from :inst:`fmin` when either operand is NaN or when comparing
-        +0.0 to -0.0.
-
-        When the two operands don't compare as LT, `y` is returned unchanged,
-        even if it is a signalling NaN.
-        """,
-        ins=(x, y), outs=a)
-
-fmax = Instruction(
-        'x86_fmax', r"""
-        Floating point maximum with x86 semantics.
-
-        This is equivalent to the C ternary operator `x > y ? x : y` which
-        differs from :inst:`fmax` when either operand is NaN or when comparing
-        +0.0 to -0.0.
-
-        When the two operands don't compare as GT, `y` is returned unchanged,
-        even if it is a signalling NaN.
-        """,
-        ins=(x, y), outs=a)
-
-
-x = Operand('x', iWord)
-
-push = Instruction(
-    'x86_push', r"""
-    Pushes a value onto the stack.
-
-    Decrements the stack pointer and stores the specified value on to the top.
-
-    This is polymorphic in i32 and i64. However, it is only implemented for i64
-    in 64-bit mode, and only for i32 in 32-bit mode.
-    """,
-    ins=x, can_store=True, other_side_effects=True)
-
-pop = Instruction(
-    'x86_pop', r"""
-    Pops a value from the stack.
-
-    Loads a value from the top of the stack and then increments the stack
-    pointer.
-
-    This is polymorphic in i32 and i64. However, it is only implemented for i64
-    in 64-bit mode, and only for i32 in 32-bit mode.
-    """,
-    outs=x, can_load=True, other_side_effects=True)
-
-y = Operand('y', iWord)
-rflags = Operand('rflags', iflags)
-
-bsr = Instruction(
-    'x86_bsr', r"""
-    Bit Scan Reverse -- returns the bit-index of the most significant 1
-    in the word. Result is undefined if the argument is zero. However, it
-    sets the Z flag depending on the argument, so it is at least easy to
-    detect and handle that case.
-
-    This is polymorphic in i32 and i64. It is implemented for both i64 and
-    i32 in 64-bit mode, and only for i32 in 32-bit mode.
-    """,
-    ins=x, outs=(y, rflags))
-
-bsf = Instruction(
-    'x86_bsf', r"""
-    Bit Scan Forwards -- returns the bit-index of the least significant 1
-    in the word. Is otherwise identical to 'bsr', just above.
-    """,
-    ins=x, outs=(y, rflags))
-
-GROUP.close()
diff --git a/cranelift/codegen/meta-python/isa/x86/legalize.py b/cranelift/codegen/meta-python/isa/x86/legalize.py
deleted file mode 100644
index 15f08a07b9..0000000000
--- a/cranelift/codegen/meta-python/isa/x86/legalize.py
+++ /dev/null
@@ -1,229 +0,0 @@
-"""
-Custom legalization patterns for x86.
-"""
-from __future__ import absolute_import
-from cdsl.ast import Var
-from cdsl.xform import Rtl, XFormGroup
-from base.immediates import imm64, intcc, floatcc
-from base import legalize as shared
-from base import instructions as insts
-from . import instructions as x86
-from .defs import ISA
-
-x86_expand = XFormGroup(
-        'x86_expand',
-        """
-        Legalize instructions by expansion.
-
-        Use x86-specific instructions if needed.
-        """,
-        isa=ISA, chain=shared.expand_flags)
-
-a = Var('a')
-dead = Var('dead')
-x = Var('x')
-xhi = Var('xhi')
-y = Var('y')
-a1 = Var('a1')
-a2 = Var('a2')
-
-#
-# Division and remainder.
-#
-# The srem expansion requires custom code because srem INT_MIN, -1 is not
-# allowed to trap. The other ops need to check avoid_div_traps.
-x86_expand.custom_legalize(insts.sdiv, 'expand_sdivrem')
-x86_expand.custom_legalize(insts.srem, 'expand_sdivrem')
-x86_expand.custom_legalize(insts.udiv, 'expand_udivrem')
-x86_expand.custom_legalize(insts.urem, 'expand_udivrem')
-
-#
-# Double length (widening) multiplication
-#
-resLo = Var('resLo')
-resHi = Var('resHi')
-x86_expand.legalize(
-        resHi << insts.umulhi(x, y),
-        Rtl(
-            (resLo, resHi) << x86.umulx(x, y)
-        ))
-
-x86_expand.legalize(
-        resHi << insts.smulhi(x, y),
-        Rtl(
-            (resLo, resHi) << x86.smulx(x, y)
-        ))
-
-# Floating point condition codes.
-#
-# The 8 condition codes in `supported_floatccs` are directly supported by a
-# `ucomiss` or `ucomisd` instruction. The remaining codes need legalization
-# patterns.
-
-# Equality needs an explicit `ord` test which checks the parity bit.
-x86_expand.legalize(
-        a << insts.fcmp(floatcc.eq, x, y),
-        Rtl(
-            a1 << insts.fcmp(floatcc.ord, x, y),
-            a2 << insts.fcmp(floatcc.ueq, x, y),
-            a << insts.band(a1, a2)
-        ))
-x86_expand.legalize(
-        a << insts.fcmp(floatcc.ne, x, y),
-        Rtl(
-            a1 << insts.fcmp(floatcc.uno, x, y),
-            a2 << insts.fcmp(floatcc.one, x, y),
-            a << insts.bor(a1, a2)
-        ))
-
-# Inequalities that need to be reversed.
-for cc,               rev_cc in [
-        (floatcc.lt,  floatcc.gt),
-        (floatcc.le,  floatcc.ge),
-        (floatcc.ugt, floatcc.ult),
-        (floatcc.uge, floatcc.ule)]:
-    x86_expand.legalize(
-            a << insts.fcmp(cc, x, y),
-            Rtl(
-                a << insts.fcmp(rev_cc, y, x)
-            ))
-
-# We need to modify the CFG for min/max legalization.
-x86_expand.custom_legalize(insts.fmin, 'expand_minmax')
-x86_expand.custom_legalize(insts.fmax, 'expand_minmax')
-
-# Conversions from unsigned need special handling.
-x86_expand.custom_legalize(insts.fcvt_from_uint, 'expand_fcvt_from_uint')
-# Conversions from float to int can trap and modify the control flow graph.
-x86_expand.custom_legalize(insts.fcvt_to_sint, 'expand_fcvt_to_sint')
-x86_expand.custom_legalize(insts.fcvt_to_uint, 'expand_fcvt_to_uint')
-x86_expand.custom_legalize(insts.fcvt_to_sint_sat, 'expand_fcvt_to_sint_sat')
-x86_expand.custom_legalize(insts.fcvt_to_uint_sat, 'expand_fcvt_to_uint_sat')
-
-# Count leading and trailing zeroes, for baseline x86_64
-c_minus_one = Var('c_minus_one')
-c_thirty_one = Var('c_thirty_one')
-c_thirty_two = Var('c_thirty_two')
-c_sixty_three = Var('c_sixty_three')
-c_sixty_four = Var('c_sixty_four')
-index1 = Var('index1')
-r2flags = Var('r2flags')
-index2 = Var('index2')
-
-x86_expand.legalize(
-    a << insts.clz.i64(x),
-    Rtl(
-        c_minus_one << insts.iconst(imm64(-1)),
-        c_sixty_three << insts.iconst(imm64(63)),
-        (index1, r2flags) << x86.bsr(x),
-        index2 << insts.selectif(intcc.eq, r2flags, c_minus_one, index1),
-        a << insts.isub(c_sixty_three, index2),
-    ))
-
-x86_expand.legalize(
-    a << insts.clz.i32(x),
-    Rtl(
-        c_minus_one << insts.iconst(imm64(-1)),
-        c_thirty_one << insts.iconst(imm64(31)),
-        (index1, r2flags) << x86.bsr(x),
-        index2 << insts.selectif(intcc.eq, r2flags, c_minus_one, index1),
-        a << insts.isub(c_thirty_one, index2),
-    ))
-
-x86_expand.legalize(
-    a << insts.ctz.i64(x),
-    Rtl(
-        c_sixty_four << insts.iconst(imm64(64)),
-        (index1, r2flags) << x86.bsf(x),
-        a << insts.selectif(intcc.eq, r2flags, c_sixty_four, index1),
-    ))
-
-x86_expand.legalize(
-    a << insts.ctz.i32(x),
-    Rtl(
-        c_thirty_two << insts.iconst(imm64(32)),
-        (index1, r2flags) << x86.bsf(x),
-        a << insts.selectif(intcc.eq, r2flags, c_thirty_two, index1),
-    ))
-
-
-# Population count for baseline x86_64
-qv1 = Var('qv1')
-qv3 = Var('qv3')
-qv4 = Var('qv4')
-qv5 = Var('qv5')
-qv6 = Var('qv6')
-qv7 = Var('qv7')
-qv8 = Var('qv8')
-qv9 = Var('qv9')
-qv10 = Var('qv10')
-qv11 = Var('qv11')
-qv12 = Var('qv12')
-qv13 = Var('qv13')
-qv14 = Var('qv14')
-qv15 = Var('qv15')
-qv16 = Var('qv16')
-qc77 = Var('qc77')
-qc0F = Var('qc0F')
-qc01 = Var('qc01')
-x86_expand.legalize(
-    qv16 << insts.popcnt.i64(qv1),
-    Rtl(
-        qv3 << insts.ushr_imm(qv1, imm64(1)),
-        qc77 << insts.iconst(imm64(0x7777777777777777)),
-        qv4 << insts.band(qv3, qc77),
-        qv5 << insts.isub(qv1, qv4),
-        qv6 << insts.ushr_imm(qv4, imm64(1)),
-        qv7 << insts.band(qv6, qc77),
-        qv8 << insts.isub(qv5, qv7),
-        qv9 << insts.ushr_imm(qv7, imm64(1)),
-        qv10 << insts.band(qv9, qc77),
-        qv11 << insts.isub(qv8, qv10),
-        qv12 << insts.ushr_imm(qv11, imm64(4)),
-        qv13 << insts.iadd(qv11, qv12),
-        qc0F << insts.iconst(imm64(0x0F0F0F0F0F0F0F0F)),
-        qv14 << insts.band(qv13, qc0F),
-        qc01 << insts.iconst(imm64(0x0101010101010101)),
-        qv15 << insts.imul(qv14, qc01),
-        qv16 << insts.ushr_imm(qv15, imm64(56))
-    ))
-
-lv1 = Var('lv1')
-lv3 = Var('lv3')
-lv4 = Var('lv4')
-lv5 = Var('lv5')
-lv6 = Var('lv6')
-lv7 = Var('lv7')
-lv8 = Var('lv8')
-lv9 = Var('lv9')
-lv10 = Var('lv10')
-lv11 = Var('lv11')
-lv12 = Var('lv12')
-lv13 = Var('lv13')
-lv14 = Var('lv14')
-lv15 = Var('lv15')
-lv16 = Var('lv16')
-lc77 = Var('lc77')
-lc0F = Var('lc0F')
-lc01 = Var('lc01')
-x86_expand.legalize(
-    lv16 << insts.popcnt.i32(lv1),
-    Rtl(
-        lv3 << insts.ushr_imm(lv1, imm64(1)),
-        lc77 << insts.iconst(imm64(0x77777777)),
-        lv4 << insts.band(lv3, lc77),
-        lv5 << insts.isub(lv1, lv4),
-        lv6 << insts.ushr_imm(lv4, imm64(1)),
-        lv7 << insts.band(lv6, lc77),
-        lv8 << insts.isub(lv5, lv7),
-        lv9 << insts.ushr_imm(lv7, imm64(1)),
-        lv10 << insts.band(lv9, lc77),
-        lv11 << insts.isub(lv8, lv10),
-        lv12 << insts.ushr_imm(lv11, imm64(4)),
-        lv13 << insts.iadd(lv11, lv12),
-        lc0F << insts.iconst(imm64(0x0F0F0F0F)),
-        lv14 << insts.band(lv13, lc0F),
-        lc01 << insts.iconst(imm64(0x01010101)),
-        lv15 << insts.imul(lv14, lc01),
-        lv16 << insts.ushr_imm(lv15, imm64(24))
-    ))
diff --git a/cranelift/codegen/meta-python/isa/x86/recipes.py b/cranelift/codegen/meta-python/isa/x86/recipes.py
deleted file mode 100644
index c596fcd108..0000000000
--- a/cranelift/codegen/meta-python/isa/x86/recipes.py
+++ /dev/null
@@ -1,2059 +0,0 @@
-"""
-x86 Encoding recipes.
-"""
-from __future__ import absolute_import
-from cdsl.isa import EncRecipe
-from cdsl.predicates import IsSignedInt, IsEqual, Or
-from cdsl.predicates import IsZero32BitFloat, IsZero64BitFloat
-from cdsl.registers import RegClass
-from base.formats import Unary, UnaryIeee32, UnaryIeee64, UnaryImm, UnaryBool
-from base.formats import Binary, BinaryImm
-from base.formats import MultiAry, NullAry
-from base.formats import Trap, Call, CallIndirect, Store, Load
-from base.formats import IntCompare, IntCompareImm, FloatCompare
-from base.formats import IntCond, FloatCond
-from base.formats import IntSelect, IntCondTrap, FloatCondTrap
-from base.formats import Jump, Branch, BranchInt, BranchFloat
-from base.formats import BranchTableEntry, BranchTableBase, IndirectJump
-from base.formats import Ternary, FuncAddr, UnaryGlobalValue
-from base.formats import RegMove, RegSpill, RegFill, CopySpecial
-from base.formats import LoadComplex, StoreComplex
-from base.formats import StackLoad
-from .registers import GPR, ABCD, FPR
-from .registers import GPR8, FPR8, FLAG
-from .registers import StackGPR32, StackFPR32
-from .defs import supported_floatccs
-from .settings import use_sse41
-
-try:
-    from typing import Tuple, Dict, Sequence, Any  # noqa
-    from cdsl.instructions import InstructionFormat  # noqa
-    from cdsl.isa import ConstraintSeq, BranchRange, PredNode, OperandConstraint  # noqa
-except ImportError:
-    pass
-
-
-# Opcode representation.
-#
-# Cranelift requires each recipe to have a single encoding size in bytes, and
-# x86 opcodes are variable length, so we use separate recipes for different
-# styles of opcodes and prefixes. The opcode format is indicated by the recipe
-# name prefix:
-
-OPCODE_PREFIX = {
-        # Prefix bytes       Name     mmpp
-        ():                 ('Op1', 0b0000),
-        (0x66,):            ('Mp1', 0b0001),
-        (0xf3,):            ('Mp1', 0b0010),
-        (0xf2,):            ('Mp1', 0b0011),
-        (0x0f,):            ('Op2', 0b0100),
-        (0x66, 0x0f):       ('Mp2', 0b0101),
-        (0xf3, 0x0f):       ('Mp2', 0b0110),
-        (0xf2, 0x0f):       ('Mp2', 0b0111),
-        (0x0f, 0x38):       ('Op3', 0b1000),
-        (0x66, 0x0f, 0x38): ('Mp3', 0b1001),
-        (0xf3, 0x0f, 0x38): ('Mp3', 0b1010),
-        (0xf2, 0x0f, 0x38): ('Mp3', 0b1011),
-        (0x0f, 0x3a):       ('Op3', 0b1100),
-        (0x66, 0x0f, 0x3a): ('Mp3', 0b1101),
-        (0xf3, 0x0f, 0x3a): ('Mp3', 0b1110),
-        (0xf2, 0x0f, 0x3a): ('Mp3', 0b1111)
-        }
-
-# The table above does not include the REX prefix which goes after the
-# mandatory prefix. VEX/XOP and EVEX prefixes are not yet supported. Encodings
-# using any of these prefixes are represented by separate recipes.
-#
-# The encoding bits are:
-#
-# 0-7:   The opcode byte .
-# 8-9:   pp, mandatory prefix:
-#        00 none (Op*)
-#        01 66   (Mp*)
-#        10 F3   (Mp*)
-#        11 F2   (Mp*)
-# 10-11: mm, opcode map:
-#        00         (Op1/Mp1)
-#        01 0F      (Op2/Mp2)
-#        10 0F 38   (Op3/Mp3)
-#        11 0F 3A   (Op3/Mp3)
-# 12-14  rrr, opcode bits for the ModR/M byte for certain opcodes.
-# 15:    REX.W bit (or VEX.W/E)
-#
-# There is some redundancy between bits 8-11 and the recipe names, but we have
-# enough bits, and the pp+mm format is ready for supporting VEX prefixes.
-
-
-def decode_ops(ops, rrr=0, w=0):
-    # type: (Tuple[int, ...], int, int) -> Tuple[str, int]
-    """
-    Given a sequence of opcode bytes, compute the recipe name prefix and
-    encoding bits.
-    """
-    assert rrr <= 0b111
-    assert w <= 1
-    name, mmpp = OPCODE_PREFIX[ops[:-1]]
-    op = ops[-1]
-    assert op <= 256
-    return (name, op | (mmpp << 8) | (rrr << 12) | (w << 15))
-
-
-def replace_put_op(emit, prefix):
-    # type: (str, str) -> str
-    """
-    Given a snippet of Rust code (or None), replace the `PUT_OP` macro with the
-    corresponding `put_*` function from the `binemit.rs` module.
-    """
-    if emit is None:
-        return None
-    else:
-        return emit.replace('PUT_OP', 'put_' + prefix.lower())
-
-
-# Register class mapping for no-REX instructions.
-NOREX_MAP = {
-        GPR: GPR8,
-        FPR: FPR8
-    }
-
-
-def map_regs_norex(regs):
-    # type: (Sequence[OperandConstraint]) -> Sequence[OperandConstraint]
-    return tuple(NOREX_MAP.get(rc, rc) if isinstance(rc, RegClass) else rc
-                 for rc in regs)
-
-
-class TailRecipe:
-    """
-    Generate encoding recipes on demand.
-
-    x86 encodings are somewhat orthogonal with the opcode representation on
-    one side and the ModR/M, SIB and immediate fields on the other side.
-
-    A `TailRecipe` represents the part of an encoding that follow the opcode.
-    It is used to generate full encoding recipes on demand when combined with
-    an opcode.
-
-    The arguments are the same as for an `EncRecipe`, except for `size` which
-    does not include the size of the opcode.
-
-    The `when_prefixed` parameter specifies a recipe that should be substituted
-    for this one when a REX (or VEX) prefix is present. This is relevant for
-    recipes that can only access the ABCD registers without a REX prefix, but
-    are able to access all registers with a prefix.
-
-    The `requires_prefix` parameter indicates that the recipe can't be used
-    without a REX prefix.
-
-    The `emit` parameter contains Rust code to actually emit an encoding, like
-    `EncRecipe` does it. Additionally, the text `PUT_OP` is substituted with
-    the proper `put_*` function from the `x86/binemit.rs` module.
-    """
-
-    def __init__(
-            self,
-            name,                   # type: str
-            format,                 # type: InstructionFormat
-            base_size,              # type: int
-            ins,                    # type: ConstraintSeq
-            outs,                   # type: ConstraintSeq
-            branch_range=None,      # type: int
-            clobbers_flags=True,    # type: bool
-            instp=None,             # type: PredNode
-            isap=None,              # type: PredNode
-            when_prefixed=None,     # type: TailRecipe
-            requires_prefix=False,  # type: bool
-            emit=None,              # type: str
-            compute_size=None       # type: str
-            ):
-        # type: (...) -> None
-        self.name = name
-        self.format = format
-        self.base_size = base_size
-        self.ins = ins
-        self.outs = outs
-        self.branch_range = branch_range
-        self.clobbers_flags = clobbers_flags
-        self.instp = instp
-        self.isap = isap
-        self.when_prefixed = when_prefixed
-        self.requires_prefix = requires_prefix
-        self.emit = emit
-        self.compute_size = compute_size
-
-        # Cached recipes, keyed by name prefix.
-        self.recipes = dict()  # type: Dict[str, EncRecipe]
-
-    def __call__(self, *ops, **kwargs):
-        # type: (*int, **int) -> Tuple[EncRecipe, int]
-        """
-        Create an encoding recipe and encoding bits for the opcode bytes in
-        `ops`.
-        """
-        assert not self.requires_prefix, "Tail recipe requires REX prefix."
-        rrr = kwargs.get('rrr', 0)
-        w = kwargs.get('w', 0)
-        name, bits = decode_ops(ops, rrr, w)
-        base_size = len(ops) + self.base_size
-
-        # All branch ranges are relative to the end of the instruction.
-        branch_range = None  # type BranchRange
-        if self.branch_range is not None:
-            branch_range = (base_size, self.branch_range)
-
-        if name not in self.recipes:
-            recipe = EncRecipe(
-                name + self.name,
-                self.format,
-                base_size,
-                ins=self.ins,
-                outs=self.outs,
-                branch_range=branch_range,
-                clobbers_flags=self.clobbers_flags,
-                instp=self.instp,
-                isap=self.isap,
-                emit=replace_put_op(self.emit, name),
-                compute_size=self.compute_size)
-
-            recipe.ins = map_regs_norex(recipe.ins)
-            recipe.outs = map_regs_norex(recipe.outs)
-            self.recipes[name] = recipe
-        return (self.recipes[name], bits)
-
-    def rex(self, *ops, **kwargs):
-        # type: (*int, **int) -> Tuple[EncRecipe, int]
-        """
-        Create a REX encoding recipe and encoding bits for the opcode bytes in
-        `ops`.
-
-        The recipe will always generate a REX prefix, whether it is required or
-        not. For instructions that don't require a REX prefix, two encodings
-        should be added: One with REX and one without.
-        """
-        # Use the prefixed alternative recipe when applicable.
-        if self.when_prefixed:
-            return self.when_prefixed.rex(*ops, **kwargs)
-
-        rrr = kwargs.get('rrr', 0)
-        w = kwargs.get('w', 0)
-        name, bits = decode_ops(ops, rrr, w)
-        name = 'Rex' + name
-        base_size = 1 + len(ops) + self.base_size
-
-        # All branch ranges are relative to the end of the instruction.
-        branch_range = None  # type BranchRange
-        if self.branch_range is not None:
-            branch_range = (base_size, self.branch_range)
-
-        if name not in self.recipes:
-            recipe = EncRecipe(
-                name + self.name,
-                self.format,
-                base_size,
-                ins=self.ins,
-                outs=self.outs,
-                branch_range=branch_range,
-                clobbers_flags=self.clobbers_flags,
-                instp=self.instp,
-                isap=self.isap,
-                emit=replace_put_op(self.emit, name),
-                compute_size=self.compute_size)
-            self.recipes[name] = recipe
-
-        return (self.recipes[name], bits)
-
-    @staticmethod
-    def check_names(globs):
-        # type: (Dict[str, Any]) -> None
-        for name, obj in globs.items():
-            if isinstance(obj, TailRecipe):
-                assert name == obj.name, "Mismatched TailRecipe name: " + name
-
-
-def floatccs(iform):
-    # type: (InstructionFormat) -> PredNode
-    """
-    Return an instruction predicate that checks in `iform.cond` is one of the
-    directly supported floating point condition codes.
-    """
-    return Or(*(IsEqual(iform.cond, cc) for cc in supported_floatccs))
-
-
-def valid_scale(iform):
-    # type: (InstructionFormat) -> PredNode
-    """
-    Return an instruction predicate that checks if `iform.imm` is a valid
-    `scale` for a SIB byte.
-    """
-    return Or(IsEqual(iform.imm, 1),
-              IsEqual(iform.imm, 2),
-              IsEqual(iform.imm, 4),
-              IsEqual(iform.imm, 8))
-
-
-# A null unary instruction that takes a GPR register. Can be used for identity
-# copies and no-op conversions.
-null = EncRecipe('null', Unary, base_size=0, ins=GPR, outs=0, emit='')
-
-stacknull = EncRecipe('stacknull', Unary, base_size=0, ins=StackGPR32,
-                      outs=StackGPR32, emit='')
-
-debugtrap = EncRecipe('debugtrap', NullAry, base_size=1, ins=(), outs=(),
-                      emit='''
-                      sink.put1(0xcc);
-                      ''')
-
-# XX opcode, no ModR/M.
-trap = TailRecipe(
-        'trap', Trap, base_size=0, ins=(), outs=(),
-        emit='''
-        sink.trap(code, func.srclocs[inst]);
-        PUT_OP(bits, BASE_REX, sink);
-        ''')
-
-# Macro: conditional jump over a ud2.
-trapif = EncRecipe(
-        'trapif', IntCondTrap, base_size=4, ins=FLAG.rflags, outs=(),
-        clobbers_flags=False,
-        emit='''
-        // Jump over a 2-byte ud2.
-        sink.put1(0x70 | (icc2opc(cond.inverse()) as u8));
-        sink.put1(2);
-        // ud2.
-        sink.trap(code, func.srclocs[inst]);
-        sink.put1(0x0f);
-        sink.put1(0x0b);
-        ''')
-
-trapff = EncRecipe(
-        'trapff', FloatCondTrap, base_size=4, ins=FLAG.rflags, outs=(),
-        clobbers_flags=False,
-        instp=floatccs(FloatCondTrap),
-        emit='''
-        // Jump over a 2-byte ud2.
-        sink.put1(0x70 | (fcc2opc(cond.inverse()) as u8));
-        sink.put1(2);
-        // ud2.
-        sink.trap(code, func.srclocs[inst]);
-        sink.put1(0x0f);
-        sink.put1(0x0b);
-        ''')
-
-
-# XX /r
-rr = TailRecipe(
-        'rr', Binary, base_size=1, ins=(GPR, GPR), outs=0,
-        emit='''
-        PUT_OP(bits, rex2(in_reg0, in_reg1), sink);
-        modrm_rr(in_reg0, in_reg1, sink);
-        ''')
-
-# XX /r with operands swapped. (RM form).
-rrx = TailRecipe(
-        'rrx', Binary, base_size=1, ins=(GPR, GPR), outs=0,
-        emit='''
-        PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        modrm_rr(in_reg1, in_reg0, sink);
-        ''')
-
-# XX /r with FPR ins and outs. A form.
-fa = TailRecipe(
-        'fa', Binary, base_size=1, ins=(FPR, FPR), outs=0,
-        emit='''
-        PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        modrm_rr(in_reg1, in_reg0, sink);
-        ''')
-
-# XX /r with FPR ins and outs. A form with input operands swapped.
-fax = TailRecipe(
-        'fax', Binary, base_size=1, ins=(FPR, FPR), outs=1,
-        emit='''
-        PUT_OP(bits, rex2(in_reg0, in_reg1), sink);
-        modrm_rr(in_reg0, in_reg1, sink);
-        ''')
-
-# XX /n for a unary operation with extension bits.
-ur = TailRecipe(
-        'ur', Unary, base_size=1, ins=GPR, outs=0,
-        emit='''
-        PUT_OP(bits, rex1(in_reg0), sink);
-        modrm_r_bits(in_reg0, bits, sink);
-        ''')
-
-# XX /r, but for a unary operator with separate input/output register, like
-# copies. MR form, preserving flags.
-umr = TailRecipe(
-        'umr', Unary, base_size=1, ins=GPR, outs=GPR,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, rex2(out_reg0, in_reg0), sink);
-        modrm_rr(out_reg0, in_reg0, sink);
-        ''')
-
-# Same as umr, but with FPR -> GPR registers.
-rfumr = TailRecipe(
-        'rfumr', Unary, base_size=1, ins=FPR, outs=GPR,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, rex2(out_reg0, in_reg0), sink);
-        modrm_rr(out_reg0, in_reg0, sink);
-        ''')
-
-# XX /r, but for a unary operator with separate input/output register.
-# RM form. Clobbers FLAGS.
-urm = TailRecipe(
-        'urm', Unary, base_size=1, ins=GPR, outs=GPR,
-        emit='''
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_rr(in_reg0, out_reg0, sink);
-        ''')
-
-# XX /r. Same as urm, but doesn't clobber FLAGS.
-urm_noflags = TailRecipe(
-        'urm_noflags', Unary, base_size=1, ins=GPR, outs=GPR,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_rr(in_reg0, out_reg0, sink);
-        ''')
-
-# XX /r. Same as urm_noflags, but input limited to ABCD.
-urm_noflags_abcd = TailRecipe(
-        'urm_noflags_abcd', Unary, base_size=1, ins=ABCD, outs=GPR,
-        when_prefixed=urm_noflags,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_rr(in_reg0, out_reg0, sink);
-        ''')
-
-# XX /r, RM form, FPR -> FPR.
-furm = TailRecipe(
-        'furm', Unary, base_size=1, ins=FPR, outs=FPR,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_rr(in_reg0, out_reg0, sink);
-        ''')
-
-# XX /r, RM form, GPR -> FPR.
-frurm = TailRecipe(
-        'frurm', Unary, base_size=1, ins=GPR, outs=FPR,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_rr(in_reg0, out_reg0, sink);
-        ''')
-
-# XX /r, RM form, FPR -> GPR.
-rfurm = TailRecipe(
-        'rfurm', Unary, base_size=1, ins=FPR, outs=GPR,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_rr(in_reg0, out_reg0, sink);
-        ''')
-
-# XX /r, RMI form for one of the roundXX SSE 4.1 instructions.
-furmi_rnd = TailRecipe(
-        'furmi_rnd', Unary, base_size=2, ins=FPR, outs=FPR,
-        isap=use_sse41,
-        emit='''
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_rr(in_reg0, out_reg0, sink);
-        sink.put1(match opcode {
-            Opcode::Nearest => 0b00,
-            Opcode::Floor => 0b01,
-            Opcode::Ceil => 0b10,
-            Opcode::Trunc => 0b11,
-            x => panic!("{} unexpected for furmi_rnd", opcode),
-        });
-        ''')
-
-# XX /r, for regmove instructions.
-rmov = TailRecipe(
-        'rmov', RegMove, base_size=1, ins=GPR, outs=(),
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, rex2(dst, src), sink);
-        modrm_rr(dst, src, sink);
-        ''')
-
-# XX /r, for regmove instructions (FPR version, RM encoded).
-frmov = TailRecipe(
-        'frmov', RegMove, base_size=1, ins=FPR, outs=(),
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, rex2(src, dst), sink);
-        modrm_rr(src, dst, sink);
-        ''')
-
-# XX /n with one arg in %rcx, for shifts.
-rc = TailRecipe(
-        'rc', Binary, base_size=1, ins=(GPR, GPR.rcx), outs=0,
-        emit='''
-        PUT_OP(bits, rex1(in_reg0), sink);
-        modrm_r_bits(in_reg0, bits, sink);
-        ''')
-
-# XX /n for division: inputs in %rax, %rdx, r. Outputs in %rax, %rdx.
-div = TailRecipe(
-        'div', Ternary, base_size=1,
-        ins=(GPR.rax, GPR.rdx, GPR), outs=(GPR.rax, GPR.rdx),
-        emit='''
-        sink.trap(TrapCode::IntegerDivisionByZero, func.srclocs[inst]);
-        PUT_OP(bits, rex1(in_reg2), sink);
-        modrm_r_bits(in_reg2, bits, sink);
-        ''')
-
-# XX /n for {s,u}mulx: inputs in %rax, r. Outputs in %rdx(hi):%rax(lo)
-mulx = TailRecipe(
-        'mulx', Binary, base_size=1,
-        ins=(GPR.rax, GPR), outs=(GPR.rax, GPR.rdx),
-        emit='''
-        PUT_OP(bits, rex1(in_reg1), sink);
-        modrm_r_bits(in_reg1, bits, sink);
-        ''')
-
-# XX /n ib with 8-bit immediate sign-extended.
-r_ib = TailRecipe(
-        'r_ib', BinaryImm, base_size=2, ins=GPR, outs=0,
-        instp=IsSignedInt(BinaryImm.imm, 8),
-        emit='''
-        PUT_OP(bits, rex1(in_reg0), sink);
-        modrm_r_bits(in_reg0, bits, sink);
-        let imm: i64 = imm.into();
-        sink.put1(imm as u8);
-        ''')
-
-# XX /n id with 32-bit immediate sign-extended.
-r_id = TailRecipe(
-        'r_id', BinaryImm, base_size=5, ins=GPR, outs=0,
-        instp=IsSignedInt(BinaryImm.imm, 32),
-        emit='''
-        PUT_OP(bits, rex1(in_reg0), sink);
-        modrm_r_bits(in_reg0, bits, sink);
-        let imm: i64 = imm.into();
-        sink.put4(imm as u32);
-        ''')
-
-# XX /n id with 32-bit immediate sign-extended. UnaryImm version.
-u_id = TailRecipe(
-        'u_id', UnaryImm, base_size=5, ins=(), outs=GPR,
-        instp=IsSignedInt(UnaryImm.imm, 32),
-        emit='''
-        PUT_OP(bits, rex1(out_reg0), sink);
-        modrm_r_bits(out_reg0, bits, sink);
-        let imm: i64 = imm.into();
-        sink.put4(imm as u32);
-        ''')
-
-# XX+rd id unary with 32-bit immediate. Note no recipe predicate.
-pu_id = TailRecipe(
-        'pu_id', UnaryImm, base_size=4, ins=(), outs=GPR,
-        emit='''
-        // The destination register is encoded in the low bits of the opcode.
-        // No ModR/M.
-        PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
-        let imm: i64 = imm.into();
-        sink.put4(imm as u32);
-        ''')
-
-# XX+rd id unary with bool immediate. Note no recipe predicate.
-pu_id_bool = TailRecipe(
-        'pu_id_bool', UnaryBool, base_size=4, ins=(), outs=GPR,
-        emit='''
-        // The destination register is encoded in the low bits of the opcode.
-        // No ModR/M.
-        PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
-        let imm: u32 = if imm { 1 } else { 0 };
-        sink.put4(imm);
-        ''')
-
-# XX+rd iq unary with 64-bit immediate.
-pu_iq = TailRecipe(
-        'pu_iq', UnaryImm, base_size=8, ins=(), outs=GPR,
-        emit='''
-        PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
-        let imm: i64 = imm.into();
-        sink.put8(imm as u64);
-        ''')
-
-# XX /n Unary with floating point 32-bit immediate equal to zero.
-f32imm_z = TailRecipe(
-    'f32imm_z', UnaryIeee32, base_size=1, ins=(), outs=FPR,
-    instp=IsZero32BitFloat(UnaryIeee32.imm),
-    emit='''
-        PUT_OP(bits, rex2(out_reg0, out_reg0), sink);
-        modrm_rr(out_reg0, out_reg0, sink);
-    ''')
-
-# XX /n Unary with floating point 64-bit immediate equal to zero.
-f64imm_z = TailRecipe(
-    'f64imm_z', UnaryIeee64, base_size=1, ins=(), outs=FPR,
-    instp=IsZero64BitFloat(UnaryIeee64.imm),
-    emit='''
-        PUT_OP(bits, rex2(out_reg0, out_reg0), sink);
-        modrm_rr(out_reg0, out_reg0, sink);
-    ''')
-
-pushq = TailRecipe(
-    'pushq', Unary, base_size=0, ins=GPR, outs=(),
-    emit='''
-    sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
-    PUT_OP(bits | (in_reg0 & 7), rex1(in_reg0), sink);
-    ''')
-
-popq = TailRecipe(
-    'popq', NullAry, base_size=0, ins=(), outs=GPR,
-    emit='''
-    PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
-    ''')
-
-# XX /r, for regmove instructions.
-copysp = TailRecipe(
-        'copysp', CopySpecial, base_size=1, ins=(), outs=(),
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, rex2(dst, src), sink);
-        modrm_rr(dst, src, sink);
-        ''')
-
-adjustsp = TailRecipe(
-    'adjustsp', Unary, base_size=1, ins=(GPR), outs=(),
-    emit='''
-    PUT_OP(bits, rex2(RU::rsp.into(), in_reg0), sink);
-    modrm_rr(RU::rsp.into(), in_reg0, sink);
-    ''')
-
-adjustsp_ib = TailRecipe(
-    'adjustsp_ib', UnaryImm, base_size=2, ins=(), outs=(),
-    instp=IsSignedInt(UnaryImm.imm, 8),
-    emit='''
-    PUT_OP(bits, rex1(RU::rsp.into()), sink);
-    modrm_r_bits(RU::rsp.into(), bits, sink);
-    let imm: i64 = imm.into();
-    sink.put1(imm as u8);
-    ''')
-
-adjustsp_id = TailRecipe(
-    'adjustsp_id', UnaryImm, base_size=5, ins=(), outs=(),
-    instp=IsSignedInt(UnaryImm.imm, 32),
-    emit='''
-    PUT_OP(bits, rex1(RU::rsp.into()), sink);
-    modrm_r_bits(RU::rsp.into(), bits, sink);
-    let imm: i64 = imm.into();
-    sink.put4(imm as u32);
-    ''')
-
-
-# XX+rd id with Abs4 function relocation.
-fnaddr4 = TailRecipe(
-        'fnaddr4', FuncAddr, base_size=4, ins=(), outs=GPR,
-        emit='''
-        PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
-        sink.reloc_external(Reloc::Abs4,
-                            &func.dfg.ext_funcs[func_ref].name,
-                            0);
-        sink.put4(0);
-        ''')
-
-# XX+rd iq with Abs8 function relocation.
-fnaddr8 = TailRecipe(
-        'fnaddr8', FuncAddr, base_size=8, ins=(), outs=GPR,
-        emit='''
-        PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
-        sink.reloc_external(Reloc::Abs8,
-                            &func.dfg.ext_funcs[func_ref].name,
-                            0);
-        sink.put8(0);
-        ''')
-
-# Similar to fnaddr4, but writes !0 (this is used by BaldrMonkey).
-allones_fnaddr4 = TailRecipe(
-        'allones_fnaddr4', FuncAddr, base_size=4, ins=(), outs=GPR,
-        emit='''
-        PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
-        sink.reloc_external(Reloc::Abs4,
-                            &func.dfg.ext_funcs[func_ref].name,
-                            0);
-        // Write the immediate as `!0` for the benefit of BaldrMonkey.
-        sink.put4(!0);
-        ''')
-
-# Similar to fnaddr8, but writes !0 (this is used by BaldrMonkey).
-allones_fnaddr8 = TailRecipe(
-        'allones_fnaddr8', FuncAddr, base_size=8, ins=(), outs=GPR,
-        emit='''
-        PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
-        sink.reloc_external(Reloc::Abs8,
-                            &func.dfg.ext_funcs[func_ref].name,
-                            0);
-        // Write the immediate as `!0` for the benefit of BaldrMonkey.
-        sink.put8(!0);
-        ''')
-
-pcrel_fnaddr8 = TailRecipe(
-        'pcrel_fnaddr8', FuncAddr, base_size=5, ins=(), outs=GPR,
-        # rex2 gets passed 0 for r/m register because the upper bit of
-        # r/m doesnt get decoded when in rip-relative addressing mode.
-        emit='''
-        PUT_OP(bits, rex2(0, out_reg0), sink);
-        modrm_riprel(out_reg0, sink);
-        // The addend adjusts for the difference between the end of the
-        // instruction and the beginning of the immediate field.
-        sink.reloc_external(Reloc::X86PCRel4,
-                            &func.dfg.ext_funcs[func_ref].name,
-                            -4);
-        sink.put4(0);
-        ''')
-
-got_fnaddr8 = TailRecipe(
-        'got_fnaddr8', FuncAddr, base_size=5, ins=(), outs=GPR,
-        # rex2 gets passed 0 for r/m register because the upper bit of
-        # r/m doesnt get decoded when in rip-relative addressing mode.
-        emit='''
-        PUT_OP(bits, rex2(0, out_reg0), sink);
-        modrm_riprel(out_reg0, sink);
-        // The addend adjusts for the difference between the end of the
-        // instruction and the beginning of the immediate field.
-        sink.reloc_external(Reloc::X86GOTPCRel4,
-                            &func.dfg.ext_funcs[func_ref].name,
-                            -4);
-        sink.put4(0);
-        ''')
-
-
-# XX+rd id with Abs4 globalsym relocation.
-gvaddr4 = TailRecipe(
-        'gvaddr4', UnaryGlobalValue, base_size=4, ins=(), outs=GPR,
-        emit='''
-        PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
-        sink.reloc_external(Reloc::Abs4,
-                            &func.global_values[global_value].symbol_name(),
-                            0);
-        sink.put4(0);
-        ''')
-
-# XX+rd iq with Abs8 globalsym relocation.
-gvaddr8 = TailRecipe(
-        'gvaddr8', UnaryGlobalValue, base_size=8, ins=(), outs=GPR,
-        emit='''
-        PUT_OP(bits | (out_reg0 & 7), rex1(out_reg0), sink);
-        sink.reloc_external(Reloc::Abs8,
-                            &func.global_values[global_value].symbol_name(),
-                            0);
-        sink.put8(0);
-        ''')
-
-# XX+rd iq with PCRel4 globalsym relocation.
-pcrel_gvaddr8 = TailRecipe(
-        'pcrel_gvaddr8', UnaryGlobalValue, base_size=5, ins=(), outs=GPR,
-        emit='''
-        PUT_OP(bits, rex2(0, out_reg0), sink);
-        modrm_rm(5, out_reg0, sink);
-        // The addend adjusts for the difference between the end of the
-        // instruction and the beginning of the immediate field.
-        sink.reloc_external(Reloc::X86PCRel4,
-                            &func.global_values[global_value].symbol_name(),
-                            -4);
-        sink.put4(0);
-        ''')
-
-# XX+rd iq with Abs8 globalsym relocation.
-got_gvaddr8 = TailRecipe(
-        'got_gvaddr8', UnaryGlobalValue, base_size=5, ins=(), outs=GPR,
-        emit='''
-        PUT_OP(bits, rex2(0, out_reg0), sink);
-        modrm_rm(5, out_reg0, sink);
-        // The addend adjusts for the difference between the end of the
-        // instruction and the beginning of the immediate field.
-        sink.reloc_external(Reloc::X86GOTPCRel4,
-                            &func.global_values[global_value].symbol_name(),
-                            -4);
-        sink.put4(0);
-        ''')
-
-#
-# Stack addresses.
-#
-# TODO: Alternative forms for 8-bit immediates, when applicable.
-#
-
-spaddr4_id = TailRecipe(
-        'spaddr4_id', StackLoad, base_size=6, ins=(), outs=GPR,
-        emit='''
-        let sp = StackRef::sp(stack_slot, &func.stack_slots);
-        let base = stk_base(sp.base);
-        PUT_OP(bits, rex2(out_reg0, base), sink);
-        modrm_sib_disp8(out_reg0, sink);
-        sib_noindex(base, sink);
-        let imm : i32 = offset.into();
-        sink.put4(sp.offset.checked_add(imm).unwrap() as u32);
-        ''')
-
-spaddr8_id = TailRecipe(
-        'spaddr8_id', StackLoad, base_size=6, ins=(), outs=GPR,
-        emit='''
-        let sp = StackRef::sp(stack_slot, &func.stack_slots);
-        let base = stk_base(sp.base);
-        PUT_OP(bits, rex2(base, out_reg0), sink);
-        modrm_sib_disp32(out_reg0, sink);
-        sib_noindex(base, sink);
-        let imm : i32 = offset.into();
-        sink.put4(sp.offset.checked_add(imm).unwrap() as u32);
-        ''')
-
-
-#
-# Store recipes.
-#
-
-# XX /r register-indirect store with no offset.
-st = TailRecipe(
-        'st', Store, base_size=1, ins=(GPR, GPR), outs=(),
-        instp=IsEqual(Store.offset, 0),
-        clobbers_flags=False,
-        compute_size="size_plus_maybe_sib_or_offset_for_in_reg_1",
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        if needs_sib_byte(in_reg1) {
-            modrm_sib(in_reg0, sink);
-            sib_noindex(in_reg1, sink);
-        } else if needs_offset(in_reg1) {
-            modrm_disp8(in_reg1, in_reg0, sink);
-            sink.put1(0);
-        } else {
-            modrm_rm(in_reg1, in_reg0, sink);
-        }
-        ''')
-
-# XX /r register-indirect store with index and no offset.
-stWithIndex = TailRecipe(
-    'stWithIndex', StoreComplex, base_size=2,
-    ins=(GPR, GPR, GPR),
-    outs=(),
-    instp=IsEqual(StoreComplex.offset, 0),
-    clobbers_flags=False,
-    compute_size="size_plus_maybe_offset_for_in_reg_1",
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
-    // The else branch always inserts an SIB byte.
-    if needs_offset(in_reg1) {
-        modrm_sib_disp8(in_reg0, sink);
-        sib(0, in_reg2, in_reg1, sink);
-        sink.put1(0);
-    } else {
-        modrm_sib(in_reg0, sink);
-        sib(0, in_reg2, in_reg1, sink);
-    }
-    ''')
-
-# XX /r register-indirect store with no offset.
-# Only ABCD allowed for stored value. This is for byte stores with no REX.
-st_abcd = TailRecipe(
-        'st_abcd', Store, base_size=1, ins=(ABCD, GPR), outs=(),
-        instp=IsEqual(Store.offset, 0),
-        when_prefixed=st,
-        clobbers_flags=False,
-        compute_size="size_plus_maybe_sib_or_offset_for_in_reg_1",
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        if needs_sib_byte(in_reg1) {
-            modrm_sib(in_reg0, sink);
-            sib_noindex(in_reg1, sink);
-        } else if needs_offset(in_reg1) {
-            modrm_disp8(in_reg1, in_reg0, sink);
-            sink.put1(0);
-        } else {
-            modrm_rm(in_reg1, in_reg0, sink);
-        }
-        ''')
-
-# XX /r register-indirect store with index and no offset.
-# Only ABCD allowed for stored value. This is for byte stores with no REX.
-stWithIndex_abcd = TailRecipe(
-    'stWithIndex_abcd', StoreComplex, base_size=2,
-    ins=(ABCD, GPR, GPR),
-    outs=(),
-    instp=IsEqual(StoreComplex.offset, 0),
-    clobbers_flags=False,
-    compute_size="size_plus_maybe_offset_for_in_reg_1",
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
-    // The else branch always inserts an SIB byte.
-    if needs_offset(in_reg1) {
-        modrm_sib_disp8(in_reg0, sink);
-        sib(0, in_reg2, in_reg1, sink);
-        sink.put1(0);
-    } else {
-        modrm_sib(in_reg0, sink);
-        sib(0, in_reg2, in_reg1, sink);
-    }
-    ''')
-
-# XX /r register-indirect store of FPR with no offset.
-fst = TailRecipe(
-        'fst', Store, base_size=1, ins=(FPR, GPR), outs=(),
-        instp=IsEqual(Store.offset, 0),
-        clobbers_flags=False,
-        compute_size="size_plus_maybe_sib_or_offset_for_in_reg_1",
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        if needs_sib_byte(in_reg1) {
-            modrm_sib(in_reg0, sink);
-            sib_noindex(in_reg1, sink);
-        } else if needs_offset(in_reg1) {
-            modrm_disp8(in_reg1, in_reg0, sink);
-            sink.put1(0);
-        } else {
-            modrm_rm(in_reg1, in_reg0, sink);
-        }
-        ''')
-# XX /r register-indirect store with index and no offset of FPR.
-fstWithIndex = TailRecipe(
-        'fstWithIndex', StoreComplex, base_size=2,
-        ins=(FPR, GPR, GPR), outs=(),
-        instp=IsEqual(StoreComplex.offset, 0),
-        clobbers_flags=False,
-        compute_size="size_plus_maybe_offset_for_in_reg_1",
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
-        // The else branch always inserts an SIB byte.
-        if needs_offset(in_reg1) {
-            modrm_sib_disp8(in_reg0, sink);
-            sib(0, in_reg2, in_reg1, sink);
-            sink.put1(0);
-        } else {
-            modrm_sib(in_reg0, sink);
-            sib(0, in_reg2, in_reg1, sink);
-        }
-        ''')
-
-# XX /r register-indirect store with 8-bit offset.
-stDisp8 = TailRecipe(
-        'stDisp8', Store, base_size=2, ins=(GPR, GPR), outs=(),
-        instp=IsSignedInt(Store.offset, 8),
-        clobbers_flags=False,
-        compute_size="size_plus_maybe_sib_for_in_reg_1",
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        if needs_sib_byte(in_reg1) {
-            modrm_sib_disp8(in_reg0, sink);
-            sib_noindex(in_reg1, sink);
-        } else {
-            modrm_disp8(in_reg1, in_reg0, sink);
-        }
-        let offset: i32 = offset.into();
-        sink.put1(offset as u8);
-        ''')
-
-# XX /r register-indirect store with index and 8-bit offset.
-stWithIndexDisp8 = TailRecipe(
-    'stWithIndexDisp8', StoreComplex, base_size=3,
-    ins=(GPR, GPR, GPR),
-    outs=(),
-    instp=IsSignedInt(StoreComplex.offset, 8),
-    clobbers_flags=False,
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
-    modrm_sib_disp8(in_reg0, sink);
-    sib(0, in_reg2, in_reg1, sink);
-    let offset: i32 = offset.into();
-    sink.put1(offset as u8);
-    ''')
-
-# XX /r register-indirect store with 8-bit offset.
-# Only ABCD allowed for stored value. This is for byte stores with no REX.
-stDisp8_abcd = TailRecipe(
-        'stDisp8_abcd', Store, base_size=2, ins=(ABCD, GPR), outs=(),
-        instp=IsSignedInt(Store.offset, 8),
-        when_prefixed=stDisp8,
-        clobbers_flags=False,
-        compute_size="size_plus_maybe_sib_for_in_reg_1",
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        if needs_sib_byte(in_reg1) {
-            modrm_sib_disp8(in_reg0, sink);
-            sib_noindex(in_reg1, sink);
-        } else {
-            modrm_disp8(in_reg1, in_reg0, sink);
-        }
-        let offset: i32 = offset.into();
-        sink.put1(offset as u8);
-        ''')
-
-# XX /r register-indirect store with index and 8-bit offset.
-# Only ABCD allowed for stored value. This is for byte stores with no REX.
-stWithIndexDisp8_abcd = TailRecipe(
-    'stWithIndexDisp8_abcd', StoreComplex, base_size=3,
-    ins=(ABCD, GPR, GPR),
-    outs=(),
-    instp=IsSignedInt(StoreComplex.offset, 8),
-    clobbers_flags=False,
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
-    modrm_sib_disp8(in_reg0, sink);
-    sib(0, in_reg2, in_reg1, sink);
-    let offset: i32 = offset.into();
-    sink.put1(offset as u8);
-    ''')
-
-# XX /r register-indirect store with 8-bit offset of FPR.
-fstDisp8 = TailRecipe(
-        'fstDisp8', Store, base_size=2, ins=(FPR, GPR), outs=(),
-        instp=IsSignedInt(Store.offset, 8),
-        clobbers_flags=False,
-        compute_size='size_plus_maybe_sib_for_in_reg_1',
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        if needs_sib_byte(in_reg1) {
-            modrm_sib_disp8(in_reg0, sink);
-            sib_noindex(in_reg1, sink);
-        } else {
-            modrm_disp8(in_reg1, in_reg0, sink);
-        }
-        let offset: i32 = offset.into();
-        sink.put1(offset as u8);
-        ''')
-
-# XX /r register-indirect store with index and 8-bit offset of FPR.
-fstWithIndexDisp8 = TailRecipe(
-    'fstWithIndexDisp8', StoreComplex, base_size=3,
-    ins=(FPR, GPR, GPR),
-    outs=(),
-    instp=IsSignedInt(StoreComplex.offset, 8),
-    clobbers_flags=False,
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
-    modrm_sib_disp8(in_reg0, sink);
-    sib(0, in_reg2, in_reg1, sink);
-    let offset: i32 = offset.into();
-    sink.put1(offset as u8);
-    ''')
-
-# XX /r register-indirect store with 32-bit offset.
-stDisp32 = TailRecipe(
-        'stDisp32', Store, base_size=5, ins=(GPR, GPR), outs=(),
-        clobbers_flags=False,
-        compute_size='size_plus_maybe_sib_for_in_reg_1',
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        if needs_sib_byte(in_reg1) {
-            modrm_sib_disp32(in_reg0, sink);
-            sib_noindex(in_reg1, sink);
-        } else {
-            modrm_disp32(in_reg1, in_reg0, sink);
-        }
-        let offset: i32 = offset.into();
-        sink.put4(offset as u32);
-        ''')
-
-# XX /r register-indirect store with index and 32-bit offset.
-stWithIndexDisp32 = TailRecipe(
-    'stWithIndexDisp32', StoreComplex, base_size=6,
-    ins=(GPR, GPR, GPR),
-    outs=(),
-    instp=IsSignedInt(StoreComplex.offset, 32),
-    clobbers_flags=False,
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
-    modrm_sib_disp32(in_reg0, sink);
-    sib(0, in_reg2, in_reg1, sink);
-    let offset: i32 = offset.into();
-    sink.put4(offset as u32);
-    ''')
-
-# XX /r register-indirect store with 32-bit offset.
-# Only ABCD allowed for stored value. This is for byte stores with no REX.
-stDisp32_abcd = TailRecipe(
-        'stDisp32_abcd', Store, base_size=5, ins=(ABCD, GPR), outs=(),
-        when_prefixed=stDisp32,
-        clobbers_flags=False,
-        compute_size="size_plus_maybe_sib_for_in_reg_1",
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        if needs_sib_byte(in_reg1) {
-            modrm_sib_disp32(in_reg0, sink);
-            sib_noindex(in_reg1, sink);
-        } else {
-            modrm_disp32(in_reg1, in_reg0, sink);
-        }
-        let offset: i32 = offset.into();
-        sink.put4(offset as u32);
-        ''')
-
-# XX /r register-indirect store with index and 32-bit offset.
-# Only ABCD allowed for stored value. This is for byte stores with no REX.
-stWithIndexDisp32_abcd = TailRecipe(
-    'stWithIndexDisp32_abcd', StoreComplex, base_size=6,
-    ins=(ABCD, GPR, GPR),
-    outs=(),
-    instp=IsSignedInt(StoreComplex.offset, 32),
-    clobbers_flags=False,
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
-    modrm_sib_disp32(in_reg0, sink);
-    sib(0, in_reg2, in_reg1, sink);
-    let offset: i32 = offset.into();
-    sink.put4(offset as u32);
-    ''')
-
-# XX /r register-indirect store with 32-bit offset of FPR.
-fstDisp32 = TailRecipe(
-        'fstDisp32', Store, base_size=5, ins=(FPR, GPR), outs=(),
-        clobbers_flags=False,
-        compute_size='size_plus_maybe_sib_for_in_reg_1',
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        if needs_sib_byte(in_reg1) {
-            modrm_sib_disp32(in_reg0, sink);
-            sib_noindex(in_reg1, sink);
-        } else {
-            modrm_disp32(in_reg1, in_reg0, sink);
-        }
-        let offset: i32 = offset.into();
-        sink.put4(offset as u32);
-        ''')
-
-# XX /r register-indirect store with index and 32-bit offset of FPR.
-fstWithIndexDisp32 = TailRecipe(
-    'fstWithIndexDisp32', StoreComplex, base_size=6,
-    ins=(FPR, GPR, GPR),
-    outs=(),
-    instp=IsSignedInt(StoreComplex.offset, 32),
-    clobbers_flags=False,
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg1, in_reg0, in_reg2), sink);
-    modrm_sib_disp32(in_reg0, sink);
-    sib(0, in_reg2, in_reg1, sink);
-    let offset: i32 = offset.into();
-    sink.put4(offset as u32);
-    ''')
-
-# Unary spill with SIB and 32-bit displacement.
-spillSib32 = TailRecipe(
-        'spillSib32', Unary, base_size=6, ins=GPR, outs=StackGPR32,
-        clobbers_flags=False,
-        emit='''
-        sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
-        let base = stk_base(out_stk0.base);
-        PUT_OP(bits, rex2(base, in_reg0), sink);
-        modrm_sib_disp32(in_reg0, sink);
-        sib_noindex(base, sink);
-        sink.put4(out_stk0.offset as u32);
-        ''')
-
-# Like spillSib32, but targeting an FPR rather than a GPR.
-fspillSib32 = TailRecipe(
-        'fspillSib32', Unary, base_size=6, ins=FPR, outs=StackFPR32,
-        clobbers_flags=False,
-        emit='''
-        sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
-        let base = stk_base(out_stk0.base);
-        PUT_OP(bits, rex2(base, in_reg0), sink);
-        modrm_sib_disp32(in_reg0, sink);
-        sib_noindex(base, sink);
-        sink.put4(out_stk0.offset as u32);
-        ''')
-
-# Regspill using RSP-relative addressing.
-regspill32 = TailRecipe(
-        'regspill32', RegSpill, base_size=6, ins=GPR, outs=(),
-        clobbers_flags=False,
-        emit='''
-        sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
-        let dst = StackRef::sp(dst, &func.stack_slots);
-        let base = stk_base(dst.base);
-        PUT_OP(bits, rex2(base, src), sink);
-        modrm_sib_disp32(src, sink);
-        sib_noindex(base, sink);
-        sink.put4(dst.offset as u32);
-        ''')
-
-# Like regspill32, but targeting an FPR rather than a GPR.
-fregspill32 = TailRecipe(
-        'fregspill32', RegSpill, base_size=6, ins=FPR, outs=(),
-        clobbers_flags=False,
-        emit='''
-        sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
-        let dst = StackRef::sp(dst, &func.stack_slots);
-        let base = stk_base(dst.base);
-        PUT_OP(bits, rex2(base, src), sink);
-        modrm_sib_disp32(src, sink);
-        sib_noindex(base, sink);
-        sink.put4(dst.offset as u32);
-        ''')
-
-#
-# Load recipes
-#
-
-# XX /r load with no offset.
-ld = TailRecipe(
-        'ld', Load, base_size=1, ins=(GPR), outs=(GPR),
-        instp=IsEqual(Load.offset, 0),
-        clobbers_flags=False,
-        compute_size="size_plus_maybe_sib_or_offset_for_in_reg_0",
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        if needs_sib_byte(in_reg0) {
-            modrm_sib(out_reg0, sink);
-            sib_noindex(in_reg0, sink);
-        } else if needs_offset(in_reg0) {
-            modrm_disp8(in_reg0, out_reg0, sink);
-            sink.put1(0);
-        } else {
-            modrm_rm(in_reg0, out_reg0, sink);
-        }
-        ''')
-
-# XX /r load with index and no offset.
-ldWithIndex = TailRecipe(
-    'ldWithIndex', LoadComplex, base_size=2,
-    ins=(GPR, GPR),
-    outs=(GPR),
-    instp=IsEqual(LoadComplex.offset, 0),
-    clobbers_flags=False,
-    compute_size="size_plus_maybe_offset_for_in_reg_0",
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
-    // The else branch always inserts an SIB byte.
-    if needs_offset(in_reg0) {
-        modrm_sib_disp8(out_reg0, sink);
-        sib(0, in_reg1, in_reg0, sink);
-        sink.put1(0);
-    } else {
-        modrm_sib(out_reg0, sink);
-        sib(0, in_reg1, in_reg0, sink);
-    }
-    ''')
-
-# XX /r float load with no offset.
-fld = TailRecipe(
-        'fld', Load, base_size=1, ins=(GPR), outs=(FPR),
-        instp=IsEqual(Load.offset, 0),
-        clobbers_flags=False,
-        compute_size="size_plus_maybe_sib_or_offset_for_in_reg_0",
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        if needs_sib_byte(in_reg0) {
-            modrm_sib(out_reg0, sink);
-            sib_noindex(in_reg0, sink);
-        } else if needs_offset(in_reg0) {
-            modrm_disp8(in_reg0, out_reg0, sink);
-            sink.put1(0);
-        } else {
-            modrm_rm(in_reg0, out_reg0, sink);
-        }
-        ''')
-
-# XX /r float load with index and no offset.
-fldWithIndex = TailRecipe(
-    'fldWithIndex', LoadComplex, base_size=2,
-    ins=(GPR, GPR),
-    outs=(FPR),
-    instp=IsEqual(LoadComplex.offset, 0),
-    clobbers_flags=False,
-    compute_size="size_plus_maybe_offset_for_in_reg_0",
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
-    // The else branch always inserts an SIB byte.
-    if needs_offset(in_reg0) {
-        modrm_sib_disp8(out_reg0, sink);
-        sib(0, in_reg1, in_reg0, sink);
-        sink.put1(0);
-    } else {
-        modrm_sib(out_reg0, sink);
-        sib(0, in_reg1, in_reg0, sink);
-    }
-    ''')
-
-# XX /r load with 8-bit offset.
-ldDisp8 = TailRecipe(
-        'ldDisp8', Load, base_size=2, ins=(GPR), outs=(GPR),
-        instp=IsSignedInt(Load.offset, 8),
-        clobbers_flags=False,
-        compute_size="size_plus_maybe_sib_for_in_reg_0",
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        if needs_sib_byte(in_reg0) {
-            modrm_sib_disp8(out_reg0, sink);
-            sib_noindex(in_reg0, sink);
-        } else {
-            modrm_disp8(in_reg0, out_reg0, sink);
-        }
-        let offset: i32 = offset.into();
-        sink.put1(offset as u8);
-        ''')
-
-# XX /r load with index and 8-bit offset.
-ldWithIndexDisp8 = TailRecipe(
-    'ldWithIndexDisp8', LoadComplex, base_size=3,
-    ins=(GPR, GPR),
-    outs=(GPR),
-    instp=IsSignedInt(LoadComplex.offset, 8),
-    clobbers_flags=False,
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
-    modrm_sib_disp8(out_reg0, sink);
-    sib(0, in_reg1, in_reg0, sink);
-    let offset: i32 = offset.into();
-    sink.put1(offset as u8);
-    ''')
-
-# XX /r float load with 8-bit offset.
-fldDisp8 = TailRecipe(
-        'fldDisp8', Load, base_size=2, ins=(GPR), outs=(FPR),
-        instp=IsSignedInt(Load.offset, 8),
-        clobbers_flags=False,
-        compute_size="size_plus_maybe_sib_for_in_reg_0",
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        if needs_sib_byte(in_reg0) {
-            modrm_sib_disp8(out_reg0, sink);
-            sib_noindex(in_reg0, sink);
-        } else {
-            modrm_disp8(in_reg0, out_reg0, sink);
-        }
-        let offset: i32 = offset.into();
-        sink.put1(offset as u8);
-        ''')
-
-# XX /r float load with 8-bit offset.
-fldWithIndexDisp8 = TailRecipe(
-    'fldWithIndexDisp8', LoadComplex, base_size=3,
-    ins=(GPR, GPR),
-    outs=(FPR),
-    instp=IsSignedInt(LoadComplex.offset, 8),
-    clobbers_flags=False,
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
-    modrm_sib_disp8(out_reg0, sink);
-    sib(0, in_reg1, in_reg0, sink);
-    let offset: i32 = offset.into();
-    sink.put1(offset as u8);
-    ''')
-
-# XX /r load with 32-bit offset.
-ldDisp32 = TailRecipe(
-        'ldDisp32', Load, base_size=5, ins=(GPR), outs=(GPR),
-        instp=IsSignedInt(Load.offset, 32),
-        clobbers_flags=False,
-        compute_size='size_plus_maybe_sib_for_in_reg_0',
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        if needs_sib_byte(in_reg0) {
-            modrm_sib_disp32(out_reg0, sink);
-            sib_noindex(in_reg0, sink);
-        } else {
-            modrm_disp32(in_reg0, out_reg0, sink);
-        }
-        let offset: i32 = offset.into();
-        sink.put4(offset as u32);
-        ''')
-
-# XX /r load with index and 32-bit offset.
-ldWithIndexDisp32 = TailRecipe(
-    'ldWithIndexDisp32', LoadComplex, base_size=6,
-    ins=(GPR, GPR),
-    outs=(GPR),
-    instp=IsSignedInt(LoadComplex.offset, 32),
-    clobbers_flags=False,
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
-    modrm_sib_disp32(out_reg0, sink);
-    sib(0, in_reg1, in_reg0, sink);
-    let offset: i32 = offset.into();
-    sink.put4(offset as u32);
-    ''')
-
-# XX /r float load with 32-bit offset.
-fldDisp32 = TailRecipe(
-        'fldDisp32', Load, base_size=5, ins=(GPR), outs=(FPR),
-        instp=IsSignedInt(Load.offset, 32),
-        clobbers_flags=False,
-        compute_size="size_plus_maybe_sib_for_in_reg_0",
-        emit='''
-        if !flags.notrap() {
-            sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-        }
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        if needs_sib_byte(in_reg0) {
-            modrm_sib_disp32(out_reg0, sink);
-            sib_noindex(in_reg0, sink);
-        } else {
-            modrm_disp32(in_reg0, out_reg0, sink);
-        }
-        let offset: i32 = offset.into();
-        sink.put4(offset as u32);
-        ''')
-
-# XX /r float load with index and 32-bit offset.
-fldWithIndexDisp32 = TailRecipe(
-    'fldWithIndexDisp32', LoadComplex, base_size=6,
-    ins=(GPR, GPR),
-    outs=(FPR),
-    instp=IsSignedInt(LoadComplex.offset, 32),
-    clobbers_flags=False,
-    emit='''
-    if !flags.notrap() {
-        sink.trap(TrapCode::HeapOutOfBounds, func.srclocs[inst]);
-    }
-    PUT_OP(bits, rex3(in_reg0, out_reg0, in_reg1), sink);
-    modrm_sib_disp32(out_reg0, sink);
-    sib(0, in_reg1, in_reg0, sink);
-    let offset: i32 = offset.into();
-    sink.put4(offset as u32);
-    ''')
-
-# Unary fill with SIB and 32-bit displacement.
-fillSib32 = TailRecipe(
-        'fillSib32', Unary, base_size=6, ins=StackGPR32, outs=GPR,
-        clobbers_flags=False,
-        emit='''
-        let base = stk_base(in_stk0.base);
-        PUT_OP(bits, rex2(base, out_reg0), sink);
-        modrm_sib_disp32(out_reg0, sink);
-        sib_noindex(base, sink);
-        sink.put4(in_stk0.offset as u32);
-        ''')
-
-# Like fillSib32, but targeting an FPR rather than a GPR.
-ffillSib32 = TailRecipe(
-        'ffillSib32', Unary, base_size=6, ins=StackFPR32, outs=FPR,
-        clobbers_flags=False,
-        emit='''
-        let base = stk_base(in_stk0.base);
-        PUT_OP(bits, rex2(base, out_reg0), sink);
-        modrm_sib_disp32(out_reg0, sink);
-        sib_noindex(base, sink);
-        sink.put4(in_stk0.offset as u32);
-        ''')
-
-# Regfill with RSP-relative 32-bit displacement.
-regfill32 = TailRecipe(
-        'regfill32', RegFill, base_size=6, ins=StackGPR32, outs=(),
-        clobbers_flags=False,
-        emit='''
-        let src = StackRef::sp(src, &func.stack_slots);
-        let base = stk_base(src.base);
-        PUT_OP(bits, rex2(base, dst), sink);
-        modrm_sib_disp32(dst, sink);
-        sib_noindex(base, sink);
-        sink.put4(src.offset as u32);
-        ''')
-
-# Like regfill32, but targeting an FPR rather than a GPR.
-fregfill32 = TailRecipe(
-        'fregfill32', RegFill, base_size=6, ins=StackFPR32, outs=(),
-        clobbers_flags=False,
-        emit='''
-        let src = StackRef::sp(src, &func.stack_slots);
-        let base = stk_base(src.base);
-        PUT_OP(bits, rex2(base, dst), sink);
-        modrm_sib_disp32(dst, sink);
-        sib_noindex(base, sink);
-        sink.put4(src.offset as u32);
-        ''')
-
-#
-# Call/return
-#
-call_id = TailRecipe(
-        'call_id', Call, base_size=4, ins=(), outs=(),
-        emit='''
-        sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
-        PUT_OP(bits, BASE_REX, sink);
-        // The addend adjusts for the difference between the end of the
-        // instruction and the beginning of the immediate field.
-        sink.reloc_external(Reloc::X86CallPCRel4,
-                            &func.dfg.ext_funcs[func_ref].name,
-                            -4);
-        sink.put4(0);
-        ''')
-
-call_plt_id = TailRecipe(
-        'call_plt_id', Call, base_size=4, ins=(), outs=(),
-        emit='''
-        sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
-        PUT_OP(bits, BASE_REX, sink);
-        sink.reloc_external(Reloc::X86CallPLTRel4,
-                            &func.dfg.ext_funcs[func_ref].name,
-                            -4);
-        sink.put4(0);
-        ''')
-
-call_r = TailRecipe(
-        'call_r', CallIndirect, base_size=1, ins=GPR, outs=(),
-        emit='''
-        sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
-        PUT_OP(bits, rex1(in_reg0), sink);
-        modrm_r_bits(in_reg0, bits, sink);
-        ''')
-
-ret = TailRecipe(
-        'ret', MultiAry, base_size=0, ins=(), outs=(),
-        emit='''
-        PUT_OP(bits, BASE_REX, sink);
-        ''')
-
-#
-# Branches
-#
-jmpb = TailRecipe(
-        'jmpb', Jump, base_size=1, ins=(), outs=(),
-        branch_range=8,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, BASE_REX, sink);
-        disp1(destination, func, sink);
-        ''')
-
-jmpd = TailRecipe(
-        'jmpd', Jump, base_size=4, ins=(), outs=(),
-        branch_range=32,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, BASE_REX, sink);
-        disp4(destination, func, sink);
-        ''')
-
-brib = TailRecipe(
-        'brib', BranchInt, base_size=1, ins=FLAG.rflags, outs=(),
-        branch_range=8,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits | icc2opc(cond), BASE_REX, sink);
-        disp1(destination, func, sink);
-        ''')
-
-brid = TailRecipe(
-        'brid', BranchInt, base_size=4, ins=FLAG.rflags, outs=(),
-        branch_range=32,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits | icc2opc(cond), BASE_REX, sink);
-        disp4(destination, func, sink);
-        ''')
-
-brfb = TailRecipe(
-        'brfb', BranchFloat, base_size=1, ins=FLAG.rflags, outs=(),
-        branch_range=8,
-        clobbers_flags=False,
-        instp=floatccs(BranchFloat),
-        emit='''
-        PUT_OP(bits | fcc2opc(cond), BASE_REX, sink);
-        disp1(destination, func, sink);
-        ''')
-
-brfd = TailRecipe(
-        'brfd', BranchFloat, base_size=4, ins=FLAG.rflags, outs=(),
-        branch_range=32,
-        clobbers_flags=False,
-        instp=floatccs(BranchFloat),
-        emit='''
-        PUT_OP(bits | fcc2opc(cond), BASE_REX, sink);
-        disp4(destination, func, sink);
-        ''')
-
-indirect_jmp = TailRecipe(
-        'indirect_jmp', IndirectJump, base_size=1, ins=GPR, outs=(),
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, rex1(in_reg0), sink);
-        modrm_r_bits(in_reg0, bits, sink);
-        ''')
-
-jt_entry = TailRecipe(
-        'jt_entry', BranchTableEntry, base_size=2,
-        ins=(GPR, GPR),
-        outs=(GPR),
-        clobbers_flags=False,
-        instp=valid_scale(BranchTableEntry),
-        compute_size="size_plus_maybe_offset_for_in_reg_1",
-        emit='''
-        PUT_OP(bits, rex3(in_reg1, out_reg0, in_reg0), sink);
-        if needs_offset(in_reg1) {
-            modrm_sib_disp8(out_reg0, sink);
-            sib(imm.trailing_zeros() as u8, in_reg0, in_reg1, sink);
-            sink.put1(0);
-        } else {
-            modrm_sib(out_reg0, sink);
-            sib(imm.trailing_zeros() as u8, in_reg0, in_reg1, sink);
-        }
-        ''')
-
-jt_base = TailRecipe(
-        'jt_base', BranchTableBase, base_size=5, ins=(), outs=(GPR),
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits, rex2(0, out_reg0), sink);
-        modrm_riprel(out_reg0, sink);
-
-        // No reloc is needed here as the jump table is emitted directly after
-        // the function body.
-        jt_disp4(table, func, sink);
-        ''')
-
-#
-# Test flags and set a register.
-#
-# These setCC instructions only set the low 8 bits, and they can only write
-# ABCD registers without a REX prefix.
-#
-# Other instruction encodings accepting `b1` inputs have the same constraints
-# and only look at the low 8 bits of the input register.
-#
-
-seti = TailRecipe(
-        'seti', IntCond, base_size=1, ins=FLAG.rflags, outs=GPR,
-        requires_prefix=True,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits | icc2opc(cond), rex1(out_reg0), sink);
-        modrm_r_bits(out_reg0, bits, sink);
-        ''')
-seti_abcd = TailRecipe(
-        'seti_abcd', IntCond, base_size=1, ins=FLAG.rflags, outs=ABCD,
-        when_prefixed=seti,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits | icc2opc(cond), rex1(out_reg0), sink);
-        modrm_r_bits(out_reg0, bits, sink);
-        ''')
-
-setf = TailRecipe(
-        'setf', FloatCond, base_size=1, ins=FLAG.rflags, outs=GPR,
-        requires_prefix=True,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits | fcc2opc(cond), rex1(out_reg0), sink);
-        modrm_r_bits(out_reg0, bits, sink);
-        ''')
-setf_abcd = TailRecipe(
-        'setf_abcd', FloatCond, base_size=1, ins=FLAG.rflags, outs=ABCD,
-        when_prefixed=setf,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits | fcc2opc(cond), rex1(out_reg0), sink);
-        modrm_r_bits(out_reg0, bits, sink);
-        ''')
-
-#
-# Conditional move (a.k.a integer select)
-# (maybe-REX.W) 0F 4x modrm(r,r)
-# 1 byte, modrm(r,r), is after the opcode
-#
-cmov = TailRecipe(
-        'cmov', IntSelect, base_size=1, ins=(FLAG.rflags, GPR, GPR), outs=2,
-        requires_prefix=False,
-        clobbers_flags=False,
-        emit='''
-        PUT_OP(bits | icc2opc(cond), rex2(in_reg1, in_reg2), sink);
-        modrm_rr(in_reg1, in_reg2, sink);
-        ''')
-
-#
-# Bit scan forwards and reverse
-#
-bsf_and_bsr = TailRecipe(
-        'bsf_and_bsr', Unary, base_size=1, ins=GPR, outs=(GPR, FLAG.rflags),
-        requires_prefix=False,
-        clobbers_flags=True,
-        emit='''
-        PUT_OP(bits, rex2(in_reg0, out_reg0), sink);
-        modrm_rr(in_reg0, out_reg0, sink);
-        ''')
-
-#
-# Compare and set flags.
-#
-
-# XX /r, MR form. Compare two GPR registers and set flags.
-rcmp = TailRecipe(
-        'rcmp', Binary, base_size=1, ins=(GPR, GPR), outs=FLAG.rflags,
-        emit='''
-        PUT_OP(bits, rex2(in_reg0, in_reg1), sink);
-        modrm_rr(in_reg0, in_reg1, sink);
-        ''')
-
-# XX /r, RM form. Compare two FPR registers and set flags.
-fcmp = TailRecipe(
-        'fcmp', Binary, base_size=1, ins=(FPR, FPR), outs=FLAG.rflags,
-        emit='''
-        PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        modrm_rr(in_reg1, in_reg0, sink);
-        ''')
-
-# XX /n, MI form with imm8.
-rcmp_ib = TailRecipe(
-        'rcmp_ib', BinaryImm, base_size=2, ins=GPR, outs=FLAG.rflags,
-        instp=IsSignedInt(BinaryImm.imm, 8),
-        emit='''
-        PUT_OP(bits, rex1(in_reg0), sink);
-        modrm_r_bits(in_reg0, bits, sink);
-        let imm: i64 = imm.into();
-        sink.put1(imm as u8);
-        ''')
-
-# XX /n, MI form with imm32.
-rcmp_id = TailRecipe(
-        'rcmp_id', BinaryImm, base_size=5, ins=GPR, outs=FLAG.rflags,
-        instp=IsSignedInt(BinaryImm.imm, 32),
-        emit='''
-        PUT_OP(bits, rex1(in_reg0), sink);
-        modrm_r_bits(in_reg0, bits, sink);
-        let imm: i64 = imm.into();
-        sink.put4(imm as u32);
-        ''')
-
-# Same as rcmp, but second operand is the stack pointer.
-rcmp_sp = TailRecipe(
-        'rcmp_sp', Unary, base_size=1, ins=GPR, outs=FLAG.rflags,
-        emit='''
-        PUT_OP(bits, rex2(in_reg0, RU::rsp.into()), sink);
-        modrm_rr(in_reg0, RU::rsp.into(), sink);
-        ''')
-
-# Test-and-branch.
-#
-# This recipe represents the macro fusion of a test and a conditional branch.
-# This serves two purposes:
-#
-# 1. Guarantee that the test and branch get scheduled next to each other so
-#    macro fusion is guaranteed to be possible.
-# 2. Hide the status flags from Cranelift which doesn't currently model flags.
-#
-# The encoding bits affect both the test and the branch instruction:
-#
-# Bits 0-7 are the Jcc opcode.
-# Bits 8-15 control the test instruction which always has opcode byte 0x85.
-tjccb = TailRecipe(
-        'tjccb', Branch, base_size=1 + 2, ins=GPR, outs=(),
-        branch_range=8,
-        emit='''
-        // test r, r.
-        PUT_OP((bits & 0xff00) | 0x85, rex2(in_reg0, in_reg0), sink);
-        modrm_rr(in_reg0, in_reg0, sink);
-        // Jcc instruction.
-        sink.put1(bits as u8);
-        disp1(destination, func, sink);
-        ''')
-
-tjccd = TailRecipe(
-        'tjccd', Branch, base_size=1 + 6, ins=GPR, outs=(),
-        branch_range=32,
-        emit='''
-        // test r, r.
-        PUT_OP((bits & 0xff00) | 0x85, rex2(in_reg0, in_reg0), sink);
-        modrm_rr(in_reg0, in_reg0, sink);
-        // Jcc instruction.
-        sink.put1(0x0f);
-        sink.put1(bits as u8);
-        disp4(destination, func, sink);
-        ''')
-
-# 8-bit test-and-branch.
-#
-# Same as tjccb, but only looks at the low 8 bits of the register, for b1
-# types.
-t8jccb = TailRecipe(
-        't8jccb', Branch, base_size=1 + 2, ins=GPR, outs=(),
-        branch_range=8,
-        requires_prefix=True,
-        emit='''
-        // test8 r, r.
-        PUT_OP((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink);
-        modrm_rr(in_reg0, in_reg0, sink);
-        // Jcc instruction.
-        sink.put1(bits as u8);
-        disp1(destination, func, sink);
-        ''')
-t8jccb_abcd = TailRecipe(
-        't8jccb_abcd', Branch, base_size=1 + 2, ins=ABCD, outs=(),
-        branch_range=8,
-        when_prefixed=t8jccb,
-        emit='''
-        // test8 r, r.
-        PUT_OP((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink);
-        modrm_rr(in_reg0, in_reg0, sink);
-        // Jcc instruction.
-        sink.put1(bits as u8);
-        disp1(destination, func, sink);
-        ''')
-
-t8jccd = TailRecipe(
-        't8jccd', Branch, base_size=1 + 6, ins=GPR, outs=(),
-        branch_range=32,
-        requires_prefix=True,
-        emit='''
-        // test8 r, r.
-        PUT_OP((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink);
-        modrm_rr(in_reg0, in_reg0, sink);
-        // Jcc instruction.
-        sink.put1(0x0f);
-        sink.put1(bits as u8);
-        disp4(destination, func, sink);
-        ''')
-t8jccd_abcd = TailRecipe(
-        't8jccd_abcd', Branch, base_size=1 + 6, ins=ABCD, outs=(),
-        branch_range=32,
-        when_prefixed=t8jccd,
-        emit='''
-        // test8 r, r.
-        PUT_OP((bits & 0xff00) | 0x84, rex2(in_reg0, in_reg0), sink);
-        modrm_rr(in_reg0, in_reg0, sink);
-        // Jcc instruction.
-        sink.put1(0x0f);
-        sink.put1(bits as u8);
-        disp4(destination, func, sink);
-        ''')
-
-# Worst case test-and-branch recipe for brz.b1 and brnz.b1 in 32-bit mode.
-# The register allocator can't handle a branch instruction with constrained
-# operands like the t8jccd_abcd above. This variant can accept the b1 opernd in
-# any register, but is is larger because it uses a 32-bit test instruction with
-# a 0xff immediate.
-t8jccd_long = TailRecipe(
-        't8jccd_long', Branch, base_size=5 + 6, ins=GPR, outs=(),
-        branch_range=32,
-        emit='''
-        // test32 r, 0xff.
-        PUT_OP((bits & 0xff00) | 0xf7, rex1(in_reg0), sink);
-        modrm_r_bits(in_reg0, bits, sink);
-        sink.put4(0xff);
-        // Jcc instruction.
-        sink.put1(0x0f);
-        sink.put1(bits as u8);
-        disp4(destination, func, sink);
-        ''')
-
-# Comparison that produces a `b1` result in a GPR.
-#
-# This is a macro of a `cmp` instruction followed by a `setCC` instruction.
-# This is not a great solution because:
-#
-# - The cmp+setcc combination is not recognized by CPU's macro fusion.
-# - The 64-bit encoding has issues with REX prefixes. The `cmp` and `setCC`
-#   instructions may need a REX independently.
-# - Modeling CPU flags in the type system would be better.
-#
-# Since the `setCC` instructions only write an 8-bit register, we use that as
-# our `b1` representation: A `b1` value is represented as a GPR where the low 8
-# bits are known to be 0 or 1. The high bits are undefined.
-#
-# This bandaid macro doesn't support a REX prefix for the final `setCC`
-# instruction, so it is limited to the `ABCD` register class for booleans.
-# The omission of a `when_prefixed` alternative is deliberate here.
-icscc = TailRecipe(
-        'icscc', IntCompare, base_size=1 + 3, ins=(GPR, GPR), outs=ABCD,
-        emit='''
-        // Comparison instruction.
-        PUT_OP(bits, rex2(in_reg0, in_reg1), sink);
-        modrm_rr(in_reg0, in_reg1, sink);
-        // `setCC` instruction, no REX.
-        use crate::ir::condcodes::IntCC::*;
-        let setcc = match cond {
-            Equal => 0x94,
-            NotEqual => 0x95,
-            SignedLessThan => 0x9c,
-            SignedGreaterThanOrEqual => 0x9d,
-            SignedGreaterThan => 0x9f,
-            SignedLessThanOrEqual => 0x9e,
-            UnsignedLessThan => 0x92,
-            UnsignedGreaterThanOrEqual => 0x93,
-            UnsignedGreaterThan => 0x97,
-            UnsignedLessThanOrEqual => 0x96,
-        };
-        sink.put1(0x0f);
-        sink.put1(setcc);
-        modrm_rr(out_reg0, 0, sink);
-        ''')
-
-icscc_ib = TailRecipe(
-        'icscc_ib', IntCompareImm, base_size=2 + 3, ins=GPR, outs=ABCD,
-        instp=IsSignedInt(IntCompareImm.imm, 8),
-        emit='''
-        // Comparison instruction.
-        PUT_OP(bits, rex1(in_reg0), sink);
-        modrm_r_bits(in_reg0, bits, sink);
-        let imm: i64 = imm.into();
-        sink.put1(imm as u8);
-        // `setCC` instruction, no REX.
-        use crate::ir::condcodes::IntCC::*;
-        let setcc = match cond {
-            Equal => 0x94,
-            NotEqual => 0x95,
-            SignedLessThan => 0x9c,
-            SignedGreaterThanOrEqual => 0x9d,
-            SignedGreaterThan => 0x9f,
-            SignedLessThanOrEqual => 0x9e,
-            UnsignedLessThan => 0x92,
-            UnsignedGreaterThanOrEqual => 0x93,
-            UnsignedGreaterThan => 0x97,
-            UnsignedLessThanOrEqual => 0x96,
-        };
-        sink.put1(0x0f);
-        sink.put1(setcc);
-        modrm_rr(out_reg0, 0, sink);
-        ''')
-
-icscc_id = TailRecipe(
-        'icscc_id', IntCompareImm, base_size=5 + 3, ins=GPR, outs=ABCD,
-        instp=IsSignedInt(IntCompareImm.imm, 32),
-        emit='''
-        // Comparison instruction.
-        PUT_OP(bits, rex1(in_reg0), sink);
-        modrm_r_bits(in_reg0, bits, sink);
-        let imm: i64 = imm.into();
-        sink.put4(imm as u32);
-        // `setCC` instruction, no REX.
-        use crate::ir::condcodes::IntCC::*;
-        let setcc = match cond {
-            Equal => 0x94,
-            NotEqual => 0x95,
-            SignedLessThan => 0x9c,
-            SignedGreaterThanOrEqual => 0x9d,
-            SignedGreaterThan => 0x9f,
-            SignedLessThanOrEqual => 0x9e,
-            UnsignedLessThan => 0x92,
-            UnsignedGreaterThanOrEqual => 0x93,
-            UnsignedGreaterThan => 0x97,
-            UnsignedLessThanOrEqual => 0x96,
-        };
-        sink.put1(0x0f);
-        sink.put1(setcc);
-        modrm_rr(out_reg0, 0, sink);
-        ''')
-
-# Make a FloatCompare instruction predicate with the supported condition codes.
-
-# Same thing for floating point.
-#
-# The ucomiss/ucomisd instructions set the FLAGS bits CF/PF/CF like this:
-#
-#    ZPC OSA
-# UN 111 000
-# GT 000 000
-# LT 001 000
-# EQ 100 000
-#
-# Not all floating point condition codes are supported.
-# The omission of a `when_prefixed` alternative is deliberate here.
-fcscc = TailRecipe(
-        'fcscc', FloatCompare, base_size=1 + 3, ins=(FPR, FPR), outs=ABCD,
-        instp=floatccs(FloatCompare),
-        emit='''
-        // Comparison instruction.
-        PUT_OP(bits, rex2(in_reg1, in_reg0), sink);
-        modrm_rr(in_reg1, in_reg0, sink);
-        // `setCC` instruction, no REX.
-        use crate::ir::condcodes::FloatCC::*;
-        let setcc = match cond {
-            Ordered                    => 0x9b, // EQ|LT|GT => setnp (P=0)
-            Unordered                  => 0x9a, // UN       => setp  (P=1)
-            OrderedNotEqual            => 0x95, // LT|GT    => setne (Z=0),
-            UnorderedOrEqual           => 0x94, // UN|EQ    => sete  (Z=1)
-            GreaterThan                => 0x97, // GT       => seta  (C=0&Z=0)
-            GreaterThanOrEqual         => 0x93, // GT|EQ    => setae (C=0)
-            UnorderedOrLessThan        => 0x92, // UN|LT    => setb  (C=1)
-            UnorderedOrLessThanOrEqual => 0x96, // UN|LT|EQ => setbe (Z=1|C=1)
-            Equal |                       // EQ
-            NotEqual |                    // UN|LT|GT
-            LessThan |                    // LT
-            LessThanOrEqual |             // LT|EQ
-            UnorderedOrGreaterThan |      // UN|GT
-            UnorderedOrGreaterThanOrEqual // UN|GT|EQ
-            => panic!("{} not supported by fcscc", cond),
-        };
-        sink.put1(0x0f);
-        sink.put1(setcc);
-        modrm_rr(out_reg0, 0, sink);
-        ''')
-
-TailRecipe.check_names(globals())
diff --git a/cranelift/codegen/meta-python/isa/x86/registers.py b/cranelift/codegen/meta-python/isa/x86/registers.py
deleted file mode 100644
index 3cca0bc377..0000000000
--- a/cranelift/codegen/meta-python/isa/x86/registers.py
+++ /dev/null
@@ -1,61 +0,0 @@
-"""
-x86 register banks.
-
-While the floating-point registers are straight-forward, the general purpose
-register bank has a few quirks on x86. We have these encodings of the 8-bit
-registers:
-
-         I32 I64  |  16b 32b  64b
-    000  AL  AL   |  AX  EAX  RAX
-    001  CL  CL   |  CX  ECX  RCX
-    010  DL  DL   |  DX  EDX  RDX
-    011  BL  BL   |  BX  EBX  RBX
-    100  AH  SPL  |  SP  ESP  RSP
-    101  CH  BPL  |  BP  EBP  RBP
-    110  DH  SIL  |  SI  ESI  RSI
-    111  BH  DIL  |  DI  EDI  RDI
-
-Here, the I64 column refers to the registers you get with a REX prefix. Without
-the REX prefix, you get the I32 registers.
-
-The 8-bit registers are not that useful since WebAssembly only has i32 and i64
-data types, and the H-registers even less so. Rather than trying to model the
-H-registers accurately, we'll avoid using them in both I32 and I64 modes.
-"""
-from __future__ import absolute_import
-from cdsl.registers import RegBank, RegClass, Stack
-from .defs import ISA
-
-
-IntRegs = RegBank(
-        'IntRegs', ISA,
-        'General purpose registers',
-        units=16, prefix='r',
-        names='rax rcx rdx rbx rsp rbp rsi rdi'.split())
-
-FloatRegs = RegBank(
-        'FloatRegs', ISA,
-        'SSE floating point registers',
-        units=16, prefix='xmm')
-
-FlagRegs = RegBank(
-        'FlagRegs', ISA,
-        'Flag registers',
-        units=1,
-        pressure_tracking=False,
-        names=['rflags'])
-
-GPR = RegClass(IntRegs)
-GPR8 = GPR[0:8]
-ABCD = GPR[0:4]
-FPR = RegClass(FloatRegs)
-FPR8 = FPR[0:8]
-FLAG = RegClass(FlagRegs)
-
-# Constraints for stack operands.
-
-# Stack operand with a 32-bit signed displacement from either RBP or RSP.
-StackGPR32 = Stack(GPR)
-StackFPR32 = Stack(FPR)
-
-RegClass.extract_names(globals())
diff --git a/cranelift/codegen/meta-python/isa/x86/settings.py b/cranelift/codegen/meta-python/isa/x86/settings.py
deleted file mode 100644
index 74633349e2..0000000000
--- a/cranelift/codegen/meta-python/isa/x86/settings.py
+++ /dev/null
@@ -1,61 +0,0 @@
-"""
-x86 settings.
-"""
-from __future__ import absolute_import
-from cdsl.settings import SettingGroup, BoolSetting, Preset
-from cdsl.predicates import And, Not
-import base.settings as shared
-from .defs import ISA
-
-ISA.settings = SettingGroup('x86', parent=shared.group)
-
-# The has_* settings here correspond to CPUID bits.
-
-# CPUID.01H:ECX
-has_sse3 = BoolSetting("SSE3: CPUID.01H:ECX.SSE3[bit 0]")
-has_ssse3 = BoolSetting("SSSE3: CPUID.01H:ECX.SSSE3[bit 9]")
-has_sse41 = BoolSetting("SSE4.1: CPUID.01H:ECX.SSE4_1[bit 19]")
-has_sse42 = BoolSetting("SSE4.2: CPUID.01H:ECX.SSE4_2[bit 20]")
-has_popcnt = BoolSetting("POPCNT: CPUID.01H:ECX.POPCNT[bit 23]")
-has_avx = BoolSetting("AVX: CPUID.01H:ECX.AVX[bit 28]")
-
-# CPUID.(EAX=07H, ECX=0H):EBX
-has_bmi1 = BoolSetting("BMI1: CPUID.(EAX=07H, ECX=0H):EBX.BMI1[bit 3]")
-has_bmi2 = BoolSetting("BMI2: CPUID.(EAX=07H, ECX=0H):EBX.BMI2[bit 8]")
-
-# CPUID.EAX=80000001H:ECX
-has_lzcnt = BoolSetting("LZCNT: CPUID.EAX=80000001H:ECX.LZCNT[bit 5]")
-
-
-# The use_* settings here are used to determine if a feature can be used.
-
-use_sse41 = And(has_sse41)
-use_sse42 = And(has_sse42, use_sse41)
-use_popcnt = And(has_popcnt, has_sse42)
-use_bmi1 = And(has_bmi1)
-use_lzcnt = And(has_lzcnt)
-
-is_pic = And(shared.is_pic)
-not_is_pic = Not(shared.is_pic)
-all_ones_funcaddrs_and_not_is_pic = And(shared.allones_funcaddrs,
-                                        Not(shared.is_pic))
-not_all_ones_funcaddrs_and_not_is_pic = And(Not(shared.allones_funcaddrs),
-                                            Not(shared.is_pic))
-
-# Presets corresponding to x86 CPUs.
-
-baseline = Preset()
-
-nehalem = Preset(
-        has_sse3, has_ssse3, has_sse41, has_sse42, has_popcnt)
-haswell = Preset(nehalem, has_bmi1, has_bmi2, has_lzcnt)
-broadwell = Preset(haswell)
-skylake = Preset(broadwell)
-cannonlake = Preset(skylake)
-icelake = Preset(cannonlake)
-
-znver1 = Preset(
-        has_sse3, has_ssse3, has_sse41, has_sse42, has_popcnt,
-        has_bmi1, has_bmi2, has_lzcnt)
-
-ISA.settings.close(globals())
diff --git a/cranelift/codegen/meta-python/mypy.ini b/cranelift/codegen/meta-python/mypy.ini
deleted file mode 100644
index 877e4c9ff5..0000000000
--- a/cranelift/codegen/meta-python/mypy.ini
+++ /dev/null
@@ -1,5 +0,0 @@
-[mypy]
-disallow_untyped_defs = True
-warn_unused_ignores = True
-warn_return_any = True
-strict_optional = False
diff --git a/cranelift/codegen/meta-python/semantics/__init__.py b/cranelift/codegen/meta-python/semantics/__init__.py
deleted file mode 100644
index 1ce6b46712..0000000000
--- a/cranelift/codegen/meta-python/semantics/__init__.py
+++ /dev/null
@@ -1,77 +0,0 @@
-"""Definitions for the semantics segment of the Cranelift language."""
-from cdsl.ti import TypeEnv, ti_rtl, get_type_env
-from cdsl.operands import ImmediateKind
-from cdsl.ast import Var
-
-try:
-    from typing import List, Dict, Tuple # noqa
-    from cdsl.ast import VarAtomMap  # noqa
-    from cdsl.xform import XForm, Rtl # noqa
-    from cdsl.ti import VarTyping # noqa
-    from cdsl.instructions import Instruction, InstructionSemantics # noqa
-except ImportError:
-    pass
-
-
-def verify_semantics(inst, src, xforms):
-    # type: (Instruction, Rtl, InstructionSemantics) -> None
-    """
-    Verify that the semantics transforms in xforms correctly describe the
-    instruction described by the src Rtl. This involves checking that:
-        0) src is a single instance of inst
-        1) For all x \\in xforms x.src is a single instance of inst
-        2) For any concrete values V of Literals in inst:
-            For all concrete typing T of inst:
-                Exists single x \\in xforms that applies to src conretazied to
-                V and T
-    """
-    # 0) The source rtl is always a single instance of inst
-    assert len(src.rtl) == 1 and src.rtl[0].expr.inst == inst
-
-    # 1) For all XForms x, x.src is a single instance of inst
-    for x in xforms:
-        assert len(x.src.rtl) == 1 and x.src.rtl[0].expr.inst == inst
-
-    variants = [src]  # type: List[Rtl]
-
-    # 2) For all enumerated immediates, compute all the possible
-    #    versions of src with the concrete value filled in.
-    for i in inst.imm_opnums:
-        op = inst.ins[i]
-        if not (isinstance(op.kind, ImmediateKind) and
-                op.kind.is_enumerable()):
-            continue
-
-        new_variants = []  # type: List[Rtl]
-        for rtl_var in variants:
-            s = {v: v for v in rtl_var.vars()}  # type: VarAtomMap
-            arg = rtl_var.rtl[0].expr.args[i]
-            assert isinstance(arg, Var)
-            for val in op.kind.possible_values():
-                s[arg] = val
-                new_variants.append(rtl_var.copy(s))
-        variants = new_variants
-
-    # For any possible version of the src with concrete enumerated immediates
-    for src in variants:
-        # 2) Any possible typing should be covered by exactly ONE semantic
-        # XForm
-        src = src.copy({})
-        typenv = get_type_env(ti_rtl(src, TypeEnv()))
-        typenv.normalize()
-        typenv = typenv.extract()
-
-        for t in typenv.concrete_typings():
-            matching_xforms = []  # type: List[XForm]
-            for x in xforms:
-                if src.substitution(x.src, {}) is None:
-                    continue
-
-                # Translate t using x.symtab
-                t = {x.symtab[str(v)]:  tv for (v, tv) in t.items()}
-                if (x.ti.permits(t)):
-                    matching_xforms.append(x)
-
-            assert len(matching_xforms) == 1,\
-                ("Possible typing {} of {} not matched by exactly one case " +
-                 ": {}").format(t, src.rtl[0], matching_xforms)
diff --git a/cranelift/codegen/meta-python/semantics/elaborate.py b/cranelift/codegen/meta-python/semantics/elaborate.py
deleted file mode 100644
index 1525b1deef..0000000000
--- a/cranelift/codegen/meta-python/semantics/elaborate.py
+++ /dev/null
@@ -1,146 +0,0 @@
-"""
-Tools to elaborate a given Rtl with concrete types into its semantically
-equivalent primitive version. Its elaborated primitive version contains only
-primitive cranelift instructions, which map well to SMTLIB functions.
-"""
-from .primitives import GROUP as PRIMITIVES, prim_to_bv, prim_from_bv
-from cdsl.xform import Rtl
-from cdsl.ast import Var
-
-try:
-    from typing import TYPE_CHECKING, Dict, Union, List, Set, Tuple # noqa
-    from cdsl.xform import XForm # noqa
-    from cdsl.ast import Def, VarAtomMap # noqa
-    from cdsl.ti import VarTyping # noqa
-except ImportError:
-    TYPE_CHECKING = False
-
-
-def find_matching_xform(d):
-    # type: (Def) -> XForm
-    """
-    Given a concrete Def d, find the unique semantic XForm x in
-    d.expr.inst.semantics that applies to it.
-    """
-    res = []  # type: List[XForm]
-    typing = {v:   v.get_typevar() for v in d.vars()}  # type: VarTyping
-
-    for x in d.expr.inst.semantics:
-        subst = d.substitution(x.src.rtl[0], {})
-
-        # There may not be a substitution if there are concrete Enumerator
-        # values in the src pattern. (e.g. specifying the semantics of icmp.eq,
-        # icmp.ge... as separate transforms)
-        if (subst is None):
-            continue
-
-        inner_typing = {}  # type: VarTyping
-        for (v, tv) in typing.items():
-            inner_v = subst[v]
-            assert isinstance(inner_v, Var)
-            inner_typing[inner_v] = tv
-
-        if x.ti.permits(inner_typing):
-            res.append(x)
-
-    assert len(res) == 1, "Couldn't find semantic transform for {}".format(d)
-    return res[0]
-
-
-def cleanup_semantics(r, outputs):
-    # type: (Rtl, Set[Var]) -> Rtl
-    """
-    The elaboration process creates a lot of redundant prim_to_bv conversions.
-    Cleanup the following cases:
-
-    1) prim_to_bv/prim_from_bv pair:
-        a.0 << prim_from_bv(bva.0)
-        ...
-        bva.1 << prim_to_bv(a.0)  <-- redundant, replace by bva.0
-        ...
-
-    2) prim_to_bv/prim_to-bv pair:
-        bva.0 << prim_to_bv(a)
-        ...
-        bva.1 << prim_to_bv(a) <-- redundant, replace by bva.0
-        ...
-    """
-    new_defs = []  # type: List[Def]
-    subst_m = {v: v for v in r.vars()}  # type: VarAtomMap
-    definition = {}  # type: Dict[Var, Def]
-    prim_to_bv_map = {}  # type: Dict[Var, Def]
-
-    # Pass 1: Remove redundant prim_to_bv
-    for d in r.rtl:
-        inst = d.expr.inst
-
-        if (inst == prim_to_bv):
-            arg = d.expr.args[0]
-            df = d.defs[0]
-            assert isinstance(arg, Var)
-
-            if arg in definition:
-                def_loc = definition[arg]
-                if def_loc.expr.inst == prim_from_bv:
-                    assert isinstance(def_loc.expr.args[0], Var)
-                    subst_m[df] = def_loc.expr.args[0]
-                    continue
-
-            if arg in prim_to_bv_map:
-                subst_m[df] = prim_to_bv_map[arg].defs[0]
-                continue
-
-            prim_to_bv_map[arg] = d
-
-        new_def = d.copy(subst_m)
-
-        for v in new_def.defs:
-            assert v not in definition  # Guaranteed by SSA
-            definition[v] = new_def
-
-        new_defs.append(new_def)
-
-    # Pass 2: Remove dead prim_from_bv
-    live = set(outputs)  # type: Set[Var]
-    for d in new_defs:
-        live = live.union(d.uses())
-
-    new_defs = [d for d in new_defs if not (d.expr.inst == prim_from_bv and
-                                            d.defs[0] not in live)]
-
-    return Rtl(*new_defs)
-
-
-def elaborate(r):
-    # type: (Rtl) -> Rtl
-    """
-    Given a concrete Rtl r, return a semantically equivalent Rtl r1 containing
-    only primitive instructions.
-    """
-    fp = False
-    primitives = set(PRIMITIVES.instructions)
-    idx = 0
-
-    res = Rtl(*r.rtl)
-    outputs = res.definitions()
-
-    while not fp:
-        assert res.is_concrete()
-        new_defs = []  # type: List[Def]
-        fp = True
-
-        for d in res.rtl:
-            inst = d.expr.inst
-
-            if (inst not in primitives):
-                t = find_matching_xform(d)
-                transformed = t.apply(Rtl(d), str(idx))
-                idx += 1
-                new_defs.extend(transformed.rtl)
-                fp = False
-            else:
-                new_defs.append(d)
-
-        res.rtl = tuple(new_defs)
-
-    return cleanup_semantics(res, outputs)
diff --git a/cranelift/codegen/meta-python/semantics/macros.py b/cranelift/codegen/meta-python/semantics/macros.py
deleted file mode 100644
index 566bf92eae..0000000000
--- a/cranelift/codegen/meta-python/semantics/macros.py
+++ /dev/null
@@ -1,45 +0,0 @@
-"""
-Useful semantics "macro" instructions built on top of
-the primitives.
-"""
-from __future__ import absolute_import
-from cdsl.operands import Operand
-from cdsl.typevar import TypeVar
-from cdsl.instructions import Instruction, InstructionGroup
-from base.types import b1
-from base.immediates import imm64
-from cdsl.ast import Var
-from cdsl.xform import Rtl
-from semantics.primitives import bv_from_imm64, bvite
-import base.formats # noqa
-
-GROUP = InstructionGroup("primitive_macros", "Semantic macros instruction set")
-AnyBV = TypeVar('AnyBV', bitvecs=True, doc="")
-x = Var('x')
-y = Var('y')
-imm = Var('imm')
-a = Var('a')
-
-#
-# Bool-to-bv1
-#
-BV1 = TypeVar("BV1", bitvecs=(1, 1), doc="")
-bv1_op = Operand('bv1_op', BV1, doc="")
-cond_op = Operand("cond", b1, doc="")
-bool2bv = Instruction(
-        'bool2bv', r"""Convert a b1 value to a 1-bit BV""",
-        ins=cond_op, outs=bv1_op)
-
-v1 = Var('v1')
-v2 = Var('v2')
-bvone = Var('bvone')
-bvzero = Var('bvzero')
-bool2bv.set_semantics(
-        v1 << bool2bv(v2),
-        Rtl(
-            bvone << bv_from_imm64(imm64(1)),
-            bvzero << bv_from_imm64(imm64(0)),
-            v1 << bvite(v2, bvone, bvzero)
-        ))
-
-GROUP.close()
diff --git a/cranelift/codegen/meta-python/semantics/primitives.py b/cranelift/codegen/meta-python/semantics/primitives.py
deleted file mode 100644
index a3e498f4e6..0000000000
--- a/cranelift/codegen/meta-python/semantics/primitives.py
+++ /dev/null
@@ -1,133 +0,0 @@
-"""
-Cranelift primitive instruction set.
-
-This module defines a primitive instruction set, in terms of which the base set
-is described. Most instructions in this set correspond 1-1 with an SMTLIB
-bitvector function.
-"""
-from __future__ import absolute_import
-from cdsl.operands import Operand
-from cdsl.typevar import TypeVar
-from cdsl.instructions import Instruction, InstructionGroup
-from cdsl.ti import WiderOrEq
-from base.types import b1
-from base.immediates import imm64
-import base.formats # noqa
-
-GROUP = InstructionGroup("primitive", "Primitive instruction set")
-
-BV = TypeVar('BV', 'A bitvector type.', bitvecs=True)
-BV1 = TypeVar('BV1', 'A single bit bitvector.', bitvecs=(1, 1))
-Real = TypeVar('Real', 'Any real type.', ints=True, floats=True,
-               bools=True, simd=True)
-
-x = Operand('x', BV, doc="A semantic value X")
-y = Operand('x', BV, doc="A semantic value Y (same width as X)")
-a = Operand('a', BV, doc="A semantic value A (same width as X)")
-cond = Operand('b', TypeVar.singleton(b1), doc='A b1 value')
-
-real = Operand('real', Real, doc="A real cranelift value")
-fromReal = Operand('fromReal', Real.to_bitvec(),
-                   doc="A real cranelift value converted to a BV")
-
-#
-# BV Conversion/Materialization
-#
-prim_to_bv = Instruction(
-        'prim_to_bv', r"""
-        Convert an SSA Value to a flat bitvector
-        """,
-        ins=(real), outs=(fromReal))
-
-prim_from_bv = Instruction(
-        'prim_from_bv', r"""
-        Convert a flat bitvector to a real SSA Value.
-        """,
-        ins=(fromReal), outs=(real))
-
-N = Operand('N', imm64)
-bv_from_imm64 = Instruction(
-        'bv_from_imm64', r"""Materialize an imm64 as a bitvector.""",
-        ins=(N), outs=a)
-
-#
-# Generics
-#
-bvite = Instruction(
-        'bvite', r"""Bitvector ternary operator""",
-        ins=(cond, x, y), outs=a)
-
-
-xh = Operand('xh', BV.half_width(),
-             doc="A semantic value representing the upper half of X")
-xl = Operand('xl', BV.half_width(),
-             doc="A semantic value representing the lower half of X")
-bvsplit = Instruction(
-        'bvsplit', r"""
-        """,
-        ins=(x), outs=(xh, xl))
-
-xy = Operand('xy', BV.double_width(),
-             doc="A semantic value representing the concatenation of X and Y")
-bvconcat = Instruction(
-        'bvconcat', r"""
-        """,
-        ins=(x, y), outs=xy)
-
-bvadd = Instruction(
-        'bvadd', r"""
-        Standard 2's complement addition. Equivalent to wrapping integer
-        addition: :math:`a := x + y \pmod{2^B}`.
-
-        This instruction does not depend on the signed/unsigned interpretation
-        of the operands.
-        """,
-        ins=(x, y), outs=a)
-#
-# Bitvector comparisons
-#
-
-bveq = Instruction(
-        'bveq', r"""Unsigned bitvector equality""",
-        ins=(x, y), outs=cond)
-bvne = Instruction(
-        'bveq', r"""Unsigned bitvector inequality""",
-        ins=(x, y), outs=cond)
-bvsge = Instruction(
-        'bvsge', r"""Signed bitvector greater or equal""",
-        ins=(x, y), outs=cond)
-bvsgt = Instruction(
-        'bvsgt', r"""Signed bitvector greater than""",
-        ins=(x, y), outs=cond)
-bvsle = Instruction(
-        'bvsle', r"""Signed bitvector less than or equal""",
-        ins=(x, y), outs=cond)
-bvslt = Instruction(
-        'bvslt', r"""Signed bitvector less than""",
-        ins=(x, y), outs=cond)
-bvuge = Instruction(
-        'bvuge', r"""Unsigned bitvector greater or equal""",
-        ins=(x, y), outs=cond)
-bvugt = Instruction(
-        'bvugt', r"""Unsigned bitvector greater than""",
-        ins=(x, y), outs=cond)
-bvule = Instruction(
-        'bvule', r"""Unsigned bitvector less than or equal""",
-        ins=(x, y), outs=cond)
-bvult = Instruction(
-        'bvult', r"""Unsigned bitvector less than""",
-        ins=(x, y), outs=cond)
-
-# Extensions
-ToBV = TypeVar('ToBV', 'A bitvector type.', bitvecs=True)
-x1 = Operand('x1', ToBV, doc="")
-
-bvzeroext = Instruction(
-        'bvzeroext', r"""Unsigned bitvector extension""",
-        ins=x, outs=x1, constraints=WiderOrEq(ToBV, BV))
-
-bvsignext = Instruction(
-        'bvsignext', r"""Signed bitvector extension""",
-        ins=x, outs=x1, constraints=WiderOrEq(ToBV, BV))
-
-GROUP.close()
diff --git a/cranelift/codegen/meta-python/semantics/smtlib.py b/cranelift/codegen/meta-python/semantics/smtlib.py
deleted file mode 100644
index 9ae9fbfaa7..0000000000
--- a/cranelift/codegen/meta-python/semantics/smtlib.py
+++ /dev/null
@@ -1,241 +0,0 @@
-"""
-Tools to emit SMTLIB bitvector queries encoding concrete RTLs containing only
-primitive instructions.
-"""
-from .primitives import GROUP as PRIMITIVES, prim_from_bv, prim_to_bv, bvadd,\
-    bvult, bvzeroext, bvsplit, bvconcat, bvsignext
-from cdsl.ast import Var
-from cdsl.types import BVType
-from .elaborate import elaborate
-from z3 import BitVec, ZeroExt, SignExt, And, Extract, Concat, Not, Solver,\
-    unsat, BoolRef, BitVecVal, If
-from z3.z3core import Z3_mk_eq
-
-try:
-    from typing import TYPE_CHECKING, Tuple, Dict, List # noqa
-    from cdsl.xform import Rtl, XForm # noqa
-    from cdsl.ast import VarAtomMap, Atom # noqa
-    from cdsl.ti import VarTyping # noqa
-    if TYPE_CHECKING:
-        from z3 import ExprRef, BitVecRef # noqa
-        Z3VarMap = Dict[Var, BitVecRef]
-except ImportError:
-    TYPE_CHECKING = False
-
-
-# Use this for constructing a == b instead of == since MyPy doesn't
-# accept overloading of __eq__ that doesn't return bool
-def mk_eq(e1, e2):
-    # type: (ExprRef, ExprRef) -> ExprRef
-    """Return a z3 expression equivalent to e1 == e2"""
-    return BoolRef(Z3_mk_eq(e1.ctx_ref(), e1.as_ast(), e2.as_ast()), e1.ctx)
-
-
-def to_smt(r):
-    # type: (Rtl) -> Tuple[List[ExprRef], Z3VarMap]
-    """
-    Encode a concrete primitive Rtl r sa z3 query.
-    Returns a tuple (query, var_m) where:
-        - query is a list of z3 expressions
-        - var_m is a map from Vars v with non-BVType to their correspodning z3
-          bitvector variable.
-    """
-    assert r.is_concrete()
-    # Should contain only primitives
-    primitives = set(PRIMITIVES.instructions)
-    assert set(d.expr.inst for d in r.rtl).issubset(primitives)
-
-    q = []  # type: List[ExprRef]
-    m = {}  # type: Z3VarMap
-
-    # Build declarations for any bitvector Vars
-    var_to_bv = {}  # type: Z3VarMap
-    for v in r.vars():
-        typ = v.get_typevar().singleton_type()
-        if not isinstance(typ, BVType):
-            continue
-
-        var_to_bv[v] = BitVec(v.name, typ.bits)
-
-    # Encode each instruction as a equality assertion
-    for d in r.rtl:
-        inst = d.expr.inst
-
-        exp = None  # type: ExprRef
-        # For prim_to_bv/prim_from_bv just update var_m. No assertion needed
-        if inst == prim_to_bv:
-            assert isinstance(d.expr.args[0], Var)
-            m[d.expr.args[0]] = var_to_bv[d.defs[0]]
-            continue
-
-        if inst == prim_from_bv:
-            assert isinstance(d.expr.args[0], Var)
-            m[d.defs[0]] = var_to_bv[d.expr.args[0]]
-            continue
-
-        if inst in [bvadd, bvult]:  # Binary instructions
-            assert len(d.expr.args) == 2 and len(d.defs) == 1
-            lhs = d.expr.args[0]
-            rhs = d.expr.args[1]
-            df = d.defs[0]
-            assert isinstance(lhs, Var) and isinstance(rhs, Var)
-
-            if inst == bvadd:  # Normal binary - output type same as args
-                exp = (var_to_bv[lhs] + var_to_bv[rhs])
-            else:
-                assert inst == bvult
-                exp = (var_to_bv[lhs] < var_to_bv[rhs])
-                # Comparison binary - need to convert bool to BitVec 1
-                exp = If(exp, BitVecVal(1, 1), BitVecVal(0, 1))
-
-            exp = mk_eq(var_to_bv[df], exp)
-        elif inst == bvzeroext:
-            arg = d.expr.args[0]
-            df = d.defs[0]
-            assert isinstance(arg, Var)
-            fromW = arg.get_typevar().singleton_type().width()
-            toW = df.get_typevar().singleton_type().width()
-
-            exp = mk_eq(var_to_bv[df], ZeroExt(toW-fromW, var_to_bv[arg]))
-        elif inst == bvsignext:
-            arg = d.expr.args[0]
-            df = d.defs[0]
-            assert isinstance(arg, Var)
-            fromW = arg.get_typevar().singleton_type().width()
-            toW = df.get_typevar().singleton_type().width()
-
-            exp = mk_eq(var_to_bv[df], SignExt(toW-fromW, var_to_bv[arg]))
-        elif inst == bvsplit:
-            arg = d.expr.args[0]
-            assert isinstance(arg, Var)
-            arg_typ = arg.get_typevar().singleton_type()
-            width = arg_typ.width()
-            assert (width % 2 == 0)
-
-            lo = d.defs[0]
-            hi = d.defs[1]
-
-            exp = And(mk_eq(var_to_bv[lo],
-                      Extract(width//2-1, 0, var_to_bv[arg])),
-                      mk_eq(var_to_bv[hi],
-                      Extract(width-1, width//2, var_to_bv[arg])))
-        elif inst == bvconcat:
-            assert isinstance(d.expr.args[0], Var) and \
-                isinstance(d.expr.args[1], Var)
-            lo = d.expr.args[0]
-            hi = d.expr.args[1]
-            df = d.defs[0]
-
-            # Z3 Concat expects hi bits first, then lo bits
-            exp = mk_eq(var_to_bv[df], Concat(var_to_bv[hi], var_to_bv[lo]))
-        else:
-            assert False, "Unknown primitive instruction {}".format(inst)
-
-        q.append(exp)
-
-    return (q, m)
-
-
-def equivalent(r1, r2, inp_m, out_m):
-    # type: (Rtl, Rtl, VarAtomMap, VarAtomMap) -> List[ExprRef]
-    """
-    Given:
-        - concrete source Rtl r1
-        - concrete dest Rtl r2
-        - VarAtomMap inp_m mapping r1's non-bitvector inputs to r2
-        - VarAtomMap out_m mapping r1's non-bitvector outputs to r2
-
-    Build a query checking whether r1 and r2 are semantically equivalent.
-    If the returned query is unsatisfiable, then r1 and r2 are equivalent.
-    Otherwise, the satisfying example for the query gives us values
-    for which the two Rtls disagree.
-    """
-    # Sanity - inp_m is a bijection from the set of inputs of r1 to the set of
-    # inputs of r2
-    assert set(r1.free_vars()) == set(inp_m.keys())
-    assert set(r2.free_vars()) == set(inp_m.values())
-
-    # Note that the same rule is not expected to hold for out_m due to
-    # temporaries/intermediates. out_m specified which values are enough for
-    # equivalence.
-
-    # Rename the vars in r1 and r2 with unique suffixes to avoid conflicts
-    src_m = {v: Var(v.name + ".a", v.get_typevar()) for v in r1.vars()}  # type: VarAtomMap # noqa
-    dst_m = {v: Var(v.name + ".b", v.get_typevar()) for v in r2.vars()}  # type: VarAtomMap # noqa
-    r1 = r1.copy(src_m)
-    r2 = r2.copy(dst_m)
-
-    def _translate(m, k_m, v_m):
-        # type: (VarAtomMap, VarAtomMap, VarAtomMap) -> VarAtomMap
-        """Obtain a new map from m, by mapping m's keys with k_m and m's values
-        with v_m"""
-        res = {}  # type: VarAtomMap
-        for (k, v) in m1.items():
-            new_k = k_m[k]
-            new_v = v_m[v]
-            assert isinstance(new_k, Var)
-            res[new_k] = new_v
-
-        return res
-
-    # Convert inp_m, out_m in terms of variables with the .a/.b suffixes
-    inp_m = _translate(inp_m, src_m, dst_m)
-    out_m = _translate(out_m, src_m, dst_m)
-
-    # Encode r1 and r2 as SMT queries
-    (q1, m1) = to_smt(r1)
-    (q2, m2) = to_smt(r2)
-
-    # Build an expression for the equality of real Cranelift inputs of
-    # r1 and r2
-    args_eq_exp = []  # type: List[ExprRef]
-
-    for (v1, v2) in inp_m.items():
-        assert isinstance(v2, Var)
-        args_eq_exp.append(mk_eq(m1[v1], m2[v2]))
-
-    # Build an expression for the equality of real Cranelift outputs of
-    # r1 and r2
-    results_eq_exp = []  # type: List[ExprRef]
-    for (v1, v2) in out_m.items():
-        assert isinstance(v2, Var)
-        results_eq_exp.append(mk_eq(m1[v1], m2[v2]))
-
-    # Put the whole query together
-    return q1 + q2 + args_eq_exp + [Not(And(*results_eq_exp))]
-
-
-def xform_correct(x, typing):
-    # type: (XForm, VarTyping) -> bool
-    """
-    Given an XForm x and a concrete variable typing for x check whether x is
-    semantically preserving for the concrete typing.
-    """
-    assert x.ti.permits(typing)
-
-    # Create copies of the x.src and x.dst with their concrete types
-    src_m = {v: Var(v.name, typing[v]) for v in x.src.vars()}  # type: VarAtomMap # noqa
-    src = x.src.copy(src_m)
-    dst = x.apply(src)
-    dst_m = x.dst.substitution(dst, {})
-
-    # Build maps for the inputs/outputs for src->dst
-    inp_m = {}  # type: VarAtomMap
-    out_m = {}  # type: VarAtomMap
-
-    for v in x.src.vars():
-        src_v = src_m[v]
-        assert isinstance(src_v, Var)
-        if v.is_input():
-            inp_m[src_v] = dst_m[v]
-        elif v.is_output():
-            out_m[src_v] = dst_m[v]
-
-    # Get the primitive semantic Rtls for src and dst
-    prim_src = elaborate(src)
-    prim_dst = elaborate(dst)
-    asserts = equivalent(prim_src, prim_dst, inp_m, out_m)
-
-    s = Solver()
-    s.add(*asserts)
-    return s.check() == unsat
diff --git a/cranelift/codegen/meta-python/semantics/test_elaborate.py b/cranelift/codegen/meta-python/semantics/test_elaborate.py
deleted file mode 100644
index 9ca938bc1f..0000000000
--- a/cranelift/codegen/meta-python/semantics/test_elaborate.py
+++ /dev/null
@@ -1,392 +0,0 @@
-from __future__ import absolute_import
-from base.instructions import vselect, vsplit, vconcat, iconst, iadd, bint
-from base.instructions import b1, icmp, ireduce, iadd_cout
-from base.immediates import intcc, imm64
-from base.types import i64, i8, b32, i32, i16, f32
-from cdsl.typevar import TypeVar
-from cdsl.ast import Var
-from cdsl.xform import Rtl
-from unittest import TestCase
-from .elaborate import elaborate
-from .primitives import prim_to_bv, bvsplit, prim_from_bv, bvconcat, bvadd, \
-    bvult, bv_from_imm64, bvite
-import base.semantics  # noqa
-
-
-def concrete_rtls_eq(r1, r2):
-    # type: (Rtl, Rtl) -> bool
-    """
-    Check whether 2 concrete Rtls are equivalent. That is:
-        1) They are structurally the same (i.e. there is a substitution between
-        them)
-        2) Corresponding Vars between them have the same singleton type.
-    """
-    assert r1.is_concrete()
-    assert r2.is_concrete()
-
-    s = r1.substitution(r2, {})
-
-    if s is None:
-        return False
-
-    for (v, v1) in s.items():
-        if v.get_typevar().singleton_type() !=\
-           v1.get_typevar().singleton_type():
-            return False
-
-    return True
-
-
-class TestCleanupConcreteRtl(TestCase):
-    """
-    Test cleanup_concrete_rtl(). cleanup_concrete_rtl() should take Rtls for
-    which we can infer a single concrete typing, and update the TypeVars
-    in-place to singleton TVs.
-    """
-    def test_cleanup_concrete_rtl(self):
-        # type: () -> None
-        typ = i64.by(4)
-        x = Var('x')
-        lo = Var('lo')
-        hi = Var('hi')
-
-        r = Rtl(
-                (lo, hi) << vsplit(x),
-        )
-        r1 = r.copy({})
-        s = r.substitution(r1, {})
-
-        s[x].set_typevar(TypeVar.singleton(typ))
-        r1.cleanup_concrete_rtl()
-        assert s is not None
-        assert s[x].get_typevar().singleton_type() == typ
-        assert s[lo].get_typevar().singleton_type() == i64.by(2)
-        assert s[hi].get_typevar().singleton_type() == i64.by(2)
-
-    def test_cleanup_concrete_rtl_fail(self):
-        # type: () -> None
-        x = Var('x')
-        lo = Var('lo')
-        hi = Var('hi')
-        r = Rtl(
-                (lo, hi) << vsplit(x),
-        )
-
-        with self.assertRaises(AssertionError):
-            r.cleanup_concrete_rtl()
-
-    def test_cleanup_concrete_rtl_ireduce(self):
-        # type: () -> None
-        x = Var('x')
-        y = Var('y')
-        r = Rtl(
-                y << ireduce(x),
-        )
-        r1 = r.copy({})
-        s = r.substitution(r1, {})
-        s[x].set_typevar(TypeVar.singleton(i8.by(2)))
-        r1.cleanup_concrete_rtl()
-
-        assert s is not None
-        assert s[x].get_typevar().singleton_type() == i8.by(2)
-        assert s[y].get_typevar().singleton_type() == i8.by(2)
-
-    def test_cleanup_concrete_rtl_ireduce_bad(self):
-        # type: () -> None
-        x = Var('x')
-        y = Var('y')
-        x.set_typevar(TypeVar.singleton(i16.by(1)))
-        r = Rtl(
-                y << ireduce(x),
-        )
-
-        with self.assertRaises(AssertionError):
-            r.cleanup_concrete_rtl()
-
-    def test_vselect_icmpimm(self):
-        # type: () -> None
-        x = Var('x')
-        y = Var('y')
-        z = Var('z')
-        w = Var('w')
-        v = Var('v')
-        zeroes = Var('zeroes')
-        imm0 = Var("imm0")
-
-        r = Rtl(
-                zeroes << iconst(imm0),
-                y << icmp(intcc.eq, x, zeroes),
-                v << vselect(y, z, w),
-        )
-        r1 = r.copy({})
-
-        s = r.substitution(r1, {})
-        s[zeroes].set_typevar(TypeVar.singleton(i32.by(4)))
-        s[z].set_typevar(TypeVar.singleton(f32.by(4)))
-
-        r1.cleanup_concrete_rtl()
-
-        assert s is not None
-        assert s[zeroes].get_typevar().singleton_type() == i32.by(4)
-        assert s[x].get_typevar().singleton_type() == i32.by(4)
-        assert s[y].get_typevar().singleton_type() == b32.by(4)
-        assert s[z].get_typevar().singleton_type() == f32.by(4)
-        assert s[w].get_typevar().singleton_type() == f32.by(4)
-        assert s[v].get_typevar().singleton_type() == f32.by(4)
-
-    def test_bint(self):
-        # type: () -> None
-        x = Var('x')
-        y = Var('y')
-        z = Var('z')
-        w = Var('w')
-        v = Var('v')
-        u = Var('u')
-
-        r = Rtl(
-            z << iadd(x, y),
-            w << bint(v),
-            u << iadd(z, w)
-        )
-        r1 = r.copy({})
-        s = r.substitution(r1, {})
-
-        s[x].set_typevar(TypeVar.singleton(i32.by(8)))
-        s[z].set_typevar(TypeVar.singleton(i32.by(8)))
-        # TODO: Relax this to simd=True
-        s[v].set_typevar(TypeVar('v', '', bools=(1, 1), simd=(8, 8)))
-        r1.cleanup_concrete_rtl()
-
-        assert s is not None
-        assert s[x].get_typevar().singleton_type() == i32.by(8)
-        assert s[y].get_typevar().singleton_type() == i32.by(8)
-        assert s[z].get_typevar().singleton_type() == i32.by(8)
-        assert s[w].get_typevar().singleton_type() == i32.by(8)
-        assert s[u].get_typevar().singleton_type() == i32.by(8)
-        assert s[v].get_typevar().singleton_type() == b1.by(8)
-
-
-class TestElaborate(TestCase):
-    """
-    Test semantics elaboration.
-    """
-    def setUp(self):
-        # type: () -> None
-        self.v0 = Var("v0")
-        self.v1 = Var("v1")
-        self.v2 = Var("v2")
-        self.v3 = Var("v3")
-        self.v4 = Var("v4")
-        self.v5 = Var("v5")
-        self.v6 = Var("v6")
-        self.v7 = Var("v7")
-        self.v8 = Var("v8")
-        self.v9 = Var("v9")
-        self.imm0 = Var("imm0")
-        self.IxN_nonscalar = TypeVar("IxN_nonscalar", "", ints=True,
-                                     scalars=False, simd=True)
-        self.TxN = TypeVar("TxN", "", ints=True, bools=True, floats=True,
-                           scalars=False, simd=True)
-        self.b1 = TypeVar.singleton(b1)
-
-    def test_elaborate_vsplit(self):
-        # type: () -> None
-        i32.by(4)  # Make sure i32x4 exists.
-        i32.by(2)  # Make sure i32x2 exists.
-        r = Rtl(
-                (self.v0, self.v1) << vsplit.i32x4(self.v2),
-        )
-        r.cleanup_concrete_rtl()
-        sem = elaborate(r)
-        bvx = Var('bvx')
-        bvlo = Var('bvlo')
-        bvhi = Var('bvhi')
-        x = Var('x')
-        lo = Var('lo')
-        hi = Var('hi')
-
-        exp = Rtl(
-            bvx << prim_to_bv.i32x4(x),
-            (bvlo, bvhi) << bvsplit.bv128(bvx),
-            lo << prim_from_bv.i32x2(bvlo),
-            hi << prim_from_bv.i32x2(bvhi)
-        )
-        exp.cleanup_concrete_rtl()
-
-        assert concrete_rtls_eq(sem, exp)
-
-    def test_elaborate_vconcat(self):
-        # type: () -> None
-        i32.by(4)  # Make sure i32x4 exists.
-        i32.by(2)  # Make sure i32x2 exists.
-        r = Rtl(
-                self.v0 << vconcat.i32x2(self.v1, self.v2),
-        )
-        r.cleanup_concrete_rtl()
-        sem = elaborate(r)
-        bvx = Var('bvx')
-        bvlo = Var('bvlo')
-        bvhi = Var('bvhi')
-        x = Var('x')
-        lo = Var('lo')
-        hi = Var('hi')
-
-        exp = Rtl(
-            bvlo << prim_to_bv.i32x2(lo),
-            bvhi << prim_to_bv.i32x2(hi),
-            bvx << bvconcat.bv64(bvlo, bvhi),
-            x << prim_from_bv.i32x4(bvx)
-        )
-        exp.cleanup_concrete_rtl()
-
-        assert concrete_rtls_eq(sem, exp)
-
-    def test_elaborate_iadd_simple(self):
-        # type: () -> None
-        i32.by(2)  # Make sure i32x2 exists.
-        x = Var('x')
-        y = Var('y')
-        a = Var('a')
-        bvx = Var('bvx')
-        bvy = Var('bvy')
-        bva = Var('bva')
-        r = Rtl(
-                a << iadd.i32(x, y),
-        )
-        r.cleanup_concrete_rtl()
-        sem = elaborate(r)
-        exp = Rtl(
-            bvx << prim_to_bv.i32(x),
-            bvy << prim_to_bv.i32(y),
-            bva << bvadd.bv32(bvx, bvy),
-            a << prim_from_bv.i32(bva)
-        )
-        exp.cleanup_concrete_rtl()
-
-        assert concrete_rtls_eq(sem, exp)
-
-    def test_elaborate_iadd_elaborate_1(self):
-        # type: () -> None
-        i32.by(2)  # Make sure i32x2 exists.
-        r = Rtl(
-                self.v0 << iadd.i32x2(self.v1, self.v2),
-        )
-        r.cleanup_concrete_rtl()
-        sem = elaborate(r)
-        x = Var('x')
-        y = Var('y')
-        a = Var('a')
-        bvx_1 = Var('bvx_1')
-        bvx_2 = Var('bvx_2')
-        bvx_5 = Var('bvx_5')
-        bvlo_1 = Var('bvlo_1')
-        bvlo_2 = Var('bvlo_2')
-        bvhi_1 = Var('bvhi_1')
-        bvhi_2 = Var('bvhi_2')
-
-        bva_3 = Var('bva_3')
-        bva_4 = Var('bva_4')
-
-        exp = Rtl(
-            bvx_1 << prim_to_bv.i32x2(x),
-            (bvlo_1, bvhi_1) << bvsplit.bv64(bvx_1),
-            bvx_2 << prim_to_bv.i32x2(y),
-            (bvlo_2, bvhi_2) << bvsplit.bv64(bvx_2),
-            bva_3 << bvadd.bv32(bvlo_1, bvlo_2),
-            bva_4 << bvadd.bv32(bvhi_1, bvhi_2),
-            bvx_5 << bvconcat.bv32(bva_3, bva_4),
-            a << prim_from_bv.i32x2(bvx_5)
-        )
-        exp.cleanup_concrete_rtl()
-
-        assert concrete_rtls_eq(sem, exp)
-
-    def test_elaborate_iadd_elaborate_2(self):
-        # type: () -> None
-        i8.by(4)  # Make sure i32x2 exists.
-        r = Rtl(
-                self.v0 << iadd.i8x4(self.v1, self.v2),
-        )
-        r.cleanup_concrete_rtl()
-
-        sem = elaborate(r)
-        x = Var('x')
-        y = Var('y')
-        a = Var('a')
-        bvx_1 = Var('bvx_1')
-        bvx_2 = Var('bvx_2')
-        bvx_5 = Var('bvx_5')
-        bvx_10 = Var('bvx_10')
-        bvx_15 = Var('bvx_15')
-
-        bvlo_1 = Var('bvlo_1')
-        bvlo_2 = Var('bvlo_2')
-        bvlo_6 = Var('bvlo_6')
-        bvlo_7 = Var('bvlo_7')
-        bvlo_11 = Var('bvlo_11')
-        bvlo_12 = Var('bvlo_12')
-
-        bvhi_1 = Var('bvhi_1')
-        bvhi_2 = Var('bvhi_2')
-        bvhi_6 = Var('bvhi_6')
-        bvhi_7 = Var('bvhi_7')
-        bvhi_11 = Var('bvhi_11')
-        bvhi_12 = Var('bvhi_12')
-
-        bva_8 = Var('bva_8')
-        bva_9 = Var('bva_9')
-        bva_13 = Var('bva_13')
-        bva_14 = Var('bva_14')
-
-        exp = Rtl(
-            bvx_1 << prim_to_bv.i8x4(x),
-            (bvlo_1, bvhi_1) << bvsplit.bv32(bvx_1),
-            bvx_2 << prim_to_bv.i8x4(y),
-            (bvlo_2, bvhi_2) << bvsplit.bv32(bvx_2),
-            (bvlo_6, bvhi_6) << bvsplit.bv16(bvlo_1),
-            (bvlo_7, bvhi_7) << bvsplit.bv16(bvlo_2),
-            bva_8 << bvadd.bv8(bvlo_6, bvlo_7),
-            bva_9 << bvadd.bv8(bvhi_6, bvhi_7),
-            bvx_10 << bvconcat.bv8(bva_8, bva_9),
-            (bvlo_11, bvhi_11) << bvsplit.bv16(bvhi_1),
-            (bvlo_12, bvhi_12) << bvsplit.bv16(bvhi_2),
-            bva_13 << bvadd.bv8(bvlo_11, bvlo_12),
-            bva_14 << bvadd.bv8(bvhi_11, bvhi_12),
-            bvx_15 << bvconcat.bv8(bva_13, bva_14),
-            bvx_5 << bvconcat.bv16(bvx_10, bvx_15),
-            a << prim_from_bv.i8x4(bvx_5)
-        )
-        exp.cleanup_concrete_rtl()
-        assert concrete_rtls_eq(sem, exp)
-
-    def test_elaborate_iadd_cout_simple(self):
-        # type: () -> None
-        x = Var('x')
-        y = Var('y')
-        a = Var('a')
-        c_out = Var('c_out')
-        bvc_out = Var('bvc_out')
-        bc_out = Var('bc_out')
-        bvx = Var('bvx')
-        bvy = Var('bvy')
-        bva = Var('bva')
-        bvone = Var('bvone')
-        bvzero = Var('bvzero')
-        r = Rtl(
-                (a, c_out) << iadd_cout.i32(x, y),
-        )
-        r.cleanup_concrete_rtl()
-        sem = elaborate(r)
-        exp = Rtl(
-            bvx << prim_to_bv.i32(x),
-            bvy << prim_to_bv.i32(y),
-            bva << bvadd.bv32(bvx, bvy),
-            bc_out << bvult.bv32(bva, bvx),
-            bvone << bv_from_imm64(imm64(1)),
-            bvzero << bv_from_imm64(imm64(0)),
-            bvc_out << bvite(bc_out, bvone, bvzero),
-            a << prim_from_bv.i32(bva),
-            c_out << prim_from_bv.b1(bvc_out)
-        )
-        exp.cleanup_concrete_rtl()
-        assert concrete_rtls_eq(sem, exp)
diff --git a/cranelift/codegen/meta-python/srcgen.py b/cranelift/codegen/meta-python/srcgen.py
deleted file mode 100644
index df5794cedb..0000000000
--- a/cranelift/codegen/meta-python/srcgen.py
+++ /dev/null
@@ -1,277 +0,0 @@
-"""
-Source code generator.
-
-The `srcgen` module contains generic helper routines and classes for generating
-source code.
-
-"""
-from __future__ import absolute_import
-import sys
-import os
-from collections import OrderedDict
-
-try:
-    from typing import Any, List, Set, Tuple  # noqa
-except ImportError:
-    pass
-
-
-class Formatter(object):
-    """
-    Source code formatter class.
-
-    - Collect source code to be written to a file.
-    - Keep track of indentation.
-
-    Indentation example:
-
-        >>> f = Formatter()
-        >>> f.line('Hello line 1')
-        >>> f.writelines()
-        Hello line 1
-        >>> f.indent_push()
-        >>> f.comment('Nested comment')
-        >>> f.indent_pop()
-        >>> f.format('Back {} again', 'home')
-        >>> f.writelines()
-        Hello line 1
-            // Nested comment
-        Back home again
-
-    """
-
-    shiftwidth = 4
-
-    def __init__(self):
-        # type: () -> None
-        self.indent = ''
-        self.lines = []  # type: List[str]
-
-    def indent_push(self):
-        # type: () -> None
-        """Increase current indentation level by one."""
-        self.indent += ' ' * self.shiftwidth
-
-    def indent_pop(self):
-        # type: () -> None
-        """Decrease indentation by one level."""
-        assert self.indent != '', 'Already at top level indentation'
-        self.indent = self.indent[0:-self.shiftwidth]
-
-    def line(self, s=None):
-        # type: (str) -> None
-        """Add an indented line."""
-        if s:
-            self.lines.append('{}{}\n'.format(self.indent, s))
-        else:
-            self.lines.append('\n')
-
-    def outdented_line(self, s):
-        # type: (str) -> None
-        """
-        Emit a line outdented one level.
-
-        This is used for '} else {' and similar things inside a single indented
-        block.
-        """
-        self.lines.append('{}{}\n'.format(self.indent[0:-self.shiftwidth], s))
-
-    def writelines(self, f=None):
-        # type: (Any) -> None
-        """Write all lines to `f`."""
-        if not f:
-            f = sys.stdout
-        f.writelines(self.lines)
-
-    def update_file(self, filename, directory):
-        # type: (str, str) -> None
-        if directory is not None:
-            filename = os.path.join(directory, filename)
-        with open(filename, 'w') as f:
-            self.writelines(f)
-
-    class _IndentedScope(object):
-        def __init__(self, fmt, after):
-            # type: (Formatter, str) -> None
-            self.fmt = fmt
-            self.after = after
-
-        def __enter__(self):
-            # type: () -> None
-            self.fmt.indent_push()
-
-        def __exit__(self, t, v, tb):
-            # type: (object, object, object) -> None
-            self.fmt.indent_pop()
-            if self.after:
-                self.fmt.line(self.after)
-
-    def indented(self, before=None, after=None):
-        # type: (str, str) -> Formatter._IndentedScope
-        """
-        Return a scope object for use with a `with` statement:
-
-            >>> f = Formatter()
-            >>> with f.indented('prefix {', '} suffix'):
-            ...     f.line('hello')
-            >>> f.writelines()
-            prefix {
-                hello
-            } suffix
-
-        The optional `before` and `after` parameters are surrounding lines
-        which are *not* indented.
-        """
-        if before:
-            self.line(before)
-        return Formatter._IndentedScope(self, after)
-
-    def format(self, fmt, *args):
-        # type: (str, *Any) -> None
-        self.line(fmt.format(*args))
-
-    def multi_line(self, s):
-        # type: (str) -> None
-        """Add one or more lines after stripping common indentation."""
-        for l in parse_multiline(s):
-            self.line(l)
-
-    def comment(self, s):
-        # type: (str) -> None
-        """Add a comment line."""
-        self.line('// ' + s)
-
-    def doc_comment(self, s):
-        # type: (str) -> None
-        """Add a (multi-line) documentation comment."""
-        for l in parse_multiline(s):
-            self.line('/// ' + l if l else '///')
-
-    def match(self, m):
-        # type: (Match) -> None
-        """
-        Add a match expression.
-
-        Example:
-
-            >>> f = Formatter()
-            >>> m = Match('x')
-            >>> m.arm('Orange', ['a', 'b'], 'some body')
-            >>> m.arm('Yellow', ['a', 'b'], 'some body')
-            >>> m.arm('Green', ['a', 'b'], 'different body')
-            >>> m.arm('Blue', ['x', 'y'], 'some body')
-            >>> f.match(m)
-            >>> f.writelines()
-            match x {
-                Orange { a, b } |
-                Yellow { a, b } => {
-                    some body
-                }
-                Green { a, b } => {
-                    different body
-                }
-                Blue { x, y } => {
-                    some body
-                }
-            }
-
-        """
-        with self.indented('match {} {{'.format(m.expr), '}'):
-            for (fields, body), names in m.arms.items():
-                with self.indented('', '}'):
-                    names_left = len(names)
-                    for name in names.keys():
-                        fields_str = ', '.join(fields)
-                        if len(fields) != 0:
-                            fields_str = '{{ {} }} '.format(fields_str)
-                        names_left -= 1
-                        if names_left > 0:
-                            suffix = '|'
-                        else:
-                            suffix = '=> {'
-                        self.outdented_line(name + ' ' + fields_str + suffix)
-                        if names_left == 0:
-                            self.multi_line(body)
-
-
-def _indent(s):
-    # type: (str) -> int
-    """
-    Compute the indentation of s, or None of an empty line.
-
-    Example:
-        >>> _indent("foo")
-        0
-        >>> _indent("    bar")
-        4
-        >>> _indent("   ")
-        >>> _indent("")
-    """
-    t = s.lstrip()
-    return len(s) - len(t) if t else None
-
-
-def parse_multiline(s):
-    # type: (str) -> List[str]
-    """
-    Given a multi-line string, split it into a sequence of lines after
-    stripping a common indentation, as described in the "trim" function
-    from PEP 257. This is useful for strings defined with doc strings:
-        >>> parse_multiline('\\n    hello\\n    world\\n')
-        ['hello', 'world']
-    """
-    if not s:
-        return []
-    # Convert tabs to spaces (following the normal Python rules)
-    # and split into a list of lines:
-    lines = s.expandtabs().splitlines()
-    # Determine minimum indentation (first line doesn't count):
-    indent = sys.maxsize
-    for line in lines[1:]:
-        stripped = line.lstrip()
-        if stripped:
-            indent = min(indent, len(line) - len(stripped))
-    # Remove indentation (first line is special):
-    trimmed = [lines[0].strip()]
-    if indent < sys.maxsize:
-        for line in lines[1:]:
-            trimmed.append(line[indent:].rstrip())
-    # Strip off trailing and leading blank lines:
-    while trimmed and not trimmed[-1]:
-        trimmed.pop()
-    while trimmed and not trimmed[0]:
-        trimmed.pop(0)
-    return trimmed
-
-
-class Match(object):
-    """
-    Match formatting class.
-
-    Match objects collect all the information needed to emit a Rust `match`
-    expression, automatically deduplicating overlapping identical arms.
-
-    Example:
-
-        >>> m = Match('x')
-        >>> m.arm('Orange', ['a', 'b'], 'some body')
-        >>> m.arm('Yellow', ['a', 'b'], 'some body')
-        >>> m.arm('Green', ['a', 'b'], 'different body')
-        >>> m.arm('Blue', ['x', 'y'], 'some body')
-        >>> assert(len(m.arms) == 3)
-
-    Note that this class is ignorant of Rust types, and considers two fields
-    with the same name to be equivalent.
-    """
-
-    def __init__(self, expr):
-        # type: (str) -> None
-        self.expr = expr
-        self.arms = OrderedDict()  # type: OrderedDict[Tuple[Tuple[str, ...], str], OrderedDict[str, None]]  # noqa
-
-    def arm(self, name, fields, body):
-        # type: (str, List[str], str) -> None
-        key = (tuple(fields), body)
-        if key not in self.arms:
-            self.arms[key] = OrderedDict()
-        self.arms[key][name] = None
diff --git a/cranelift/codegen/meta-python/stubs/z3/__init__.pyi b/cranelift/codegen/meta-python/stubs/z3/__init__.pyi
deleted file mode 100644
index 2fd6c8341f..0000000000
--- a/cranelift/codegen/meta-python/stubs/z3/__init__.pyi
+++ /dev/null
@@ -1,151 +0,0 @@
-from typing import overload, Tuple, Any, List, Iterable, Union, TypeVar
-from .z3types import Ast, ContextObj
-
-TExprRef = TypeVar("TExprRef", bound="ExprRef")
-
-class Context:
-  ...
-
-class Z3PPObject:
-  ...
-
-class AstRef(Z3PPObject):
-  @overload
-  def __init__(self, ast: Ast, ctx: Context) -> None:
-    self.ast: Ast = ...
-    self.ctx: Context= ...
-
-  @overload
-  def __init__(self, ast: Ast) -> None:
-    self.ast: Ast = ...
-    self.ctx: Context= ...
-  def ctx_ref(self) -> ContextObj:  ...
-  def as_ast(self) -> Ast:  ...
-  def children(self) -> List[AstRef]: ...
-
-class SortRef(AstRef):
-  ...
-
-class FuncDeclRef(AstRef):
-  def arity(self) -> int: ...
-  def name(self) -> str:  ...
-
-class ExprRef(AstRef):
-  def eq(self, other: ExprRef) -> ExprRef:  ...
-  def sort(self) -> SortRef:  ...
-  def decl(self) -> FuncDeclRef:  ...
-
-class BoolSortRef(SortRef):
-  ...
-
-class BoolRef(ExprRef):
-  ...
-
-
-def is_true(a: BoolRef) -> bool:  ...
-def is_false(a: BoolRef) -> bool:  ...
-def is_int_value(a: AstRef) -> bool:  ...
-def substitute(a: AstRef, *m: Tuple[AstRef, AstRef]) -> AstRef: ...
-
-
-class ArithSortRef(SortRef):
-  ...
-
-class ArithRef(ExprRef):
-  def __neg__(self) -> ExprRef: ...
-  def __le__(self, other: ArithRef) -> ArithRef:  ...
-  def __lt__(self, other: ArithRef) -> ArithRef:  ...
-  def __ge__(self, other: ArithRef) -> ArithRef:  ...
-  def __gt__(self, other: ArithRef) -> ArithRef:  ...
-  def __add__(self, other: ArithRef) -> ArithRef:  ...
-  def __sub__(self, other: ArithRef) -> ArithRef:  ...
-  def __mul__(self, other: ArithRef) -> ArithRef:  ...
-  def __div__(self, other: ArithRef) -> ArithRef:  ...
-  def __mod__(self, other: ArithRef) -> ArithRef:  ...
-
-class IntNumRef(ArithRef):
-  def as_long(self) -> int: ...
-
-class BitVecRef(ExprRef):
-  def __neg__(self) -> ExprRef: ...
-  def __le__(self, other: BitVecRef) -> ExprRef:  ...
-  def __lt__(self, other: BitVecRef) -> ExprRef:  ...
-  def __ge__(self, other: BitVecRef) -> ExprRef:  ...
-  def __gt__(self, other: BitVecRef) -> ExprRef:  ...
-  def __add__(self, other: BitVecRef) -> BitVecRef:  ...
-  def __sub__(self, other: BitVecRef) -> BitVecRef:  ...
-  def __mul__(self, other: BitVecRef) -> BitVecRef:  ...
-  def __div__(self, other: BitVecRef) -> BitVecRef:  ...
-  def __mod__(self, other: BitVecRef) -> BitVecRef:  ...
-
-class BitVecNumRef(BitVecRef):
-  def as_long(self) -> int: ...
-
-class CheckSatResult: ...
-
-class ModelRef(Z3PPObject):
-  def __getitem__(self, k:  FuncDeclRef) -> IntNumRef:  ...
-  def decls(self) ->  Iterable[FuncDeclRef]:  ...
-
-class Solver(Z3PPObject):
-  @overload
-  def __init__(self) -> None:
-    self.ctx: Context = ...
-  @overload
-  def __init__(self, ctx:Context) -> None:
-    self.ctx: Context = ...
-
-  def add(self, e:ExprRef) -> None: ...
-  def to_smt2(self) -> str: ...
-  def check(self) -> CheckSatResult: ...
-  def push(self) -> None:  ...
-  def pop(self) -> None:  ...
-  def model(self) -> ModelRef:  ...
-
-sat: CheckSatResult = ...
-unsat: CheckSatResult = ...
-
-@overload
-def Int(name: str) -> ArithRef: ...
-@overload
-def Int(name: str, ctx: Context) -> ArithRef: ...
-
-@overload
-def Bool(name: str) -> BoolRef: ...
-@overload
-def Bool(name: str, ctx: Context) -> BoolRef: ...
-
-def BitVec(name: str, width: int) -> BitVecRef: ...
-
-@overload
-def parse_smt2_string(s: str) -> ExprRef: ...
-@overload
-def parse_smt2_string(s: str, ctx: Context) -> ExprRef: ...
-
-# Can't give more precise types here since func signature is
-# a vararg list of ExprRef optionally followed by a Context
-def Or(*args: Union[ExprRef, Context]) -> ExprRef: ...
-def And(*args: Union[ExprRef, Context]) -> ExprRef: ...
-@overload
-def Not(p: ExprRef) -> ExprRef: ...
-@overload
-def Not(p: ExprRef, ctx: Context) -> ExprRef: ...
-def Implies(a: ExprRef, b: ExprRef, ctx:Context) -> ExprRef: ...
-def If(a: ExprRef, b:TExprRef, c:TExprRef) -> TExprRef:  ...
-
-def ZeroExt(width: int, expr: BitVecRef) -> BitVecRef:  ...
-def SignExt(width: int, expr: BitVecRef) -> BitVecRef:  ...
-def Extract(hi: int, lo: int, expr: BitVecRef) -> BitVecRef:  ...
-def Concat(expr1: BitVecRef, expr2: BitVecRef) -> BitVecRef:  ...
-
-def Function(name: str, *sig: Tuple[SortRef,...]) -> FuncDeclRef:  ...
-
-def IntVal(val: int, ctx: Context) -> IntNumRef:  ...
-@overload
-def BoolVal(val: bool, ctx: Context) -> BoolRef:  ...
-@overload
-def BoolVal(val: bool) -> BoolRef:  ...
-@overload
-def BitVecVal(val: int, bits: int, ctx: Context) -> BitVecNumRef:  ...
-@overload
-def BitVecVal(val: int, bits: int) -> BitVecNumRef:  ...
diff --git a/cranelift/codegen/meta-python/stubs/z3/z3core.pyi b/cranelift/codegen/meta-python/stubs/z3/z3core.pyi
deleted file mode 100644
index 36f1f88792..0000000000
--- a/cranelift/codegen/meta-python/stubs/z3/z3core.pyi
+++ /dev/null
@@ -1,3 +0,0 @@
-from .z3types import Ast, ContextObj
-def Z3_mk_eq(ctx: ContextObj, a: Ast, b: Ast) -> Ast: ...
-def Z3_mk_div(ctx: ContextObj, a: Ast, b: Ast) -> Ast: ...
diff --git a/cranelift/codegen/meta-python/stubs/z3/z3types.pyi b/cranelift/codegen/meta-python/stubs/z3/z3types.pyi
deleted file mode 100644
index fa8fc446d1..0000000000
--- a/cranelift/codegen/meta-python/stubs/z3/z3types.pyi
+++ /dev/null
@@ -1,12 +0,0 @@
-from typing import Any
-
-class Z3Exception(Exception):
-  def __init__(self, a: Any) -> None:
-    self.value = a
-    ...
-
-class ContextObj:
-  ...
-
-class Ast:
-  ...
diff --git a/cranelift/codegen/meta-python/test_constant_hash.py b/cranelift/codegen/meta-python/test_constant_hash.py
deleted file mode 100644
index e76f09aed9..0000000000
--- a/cranelift/codegen/meta-python/test_constant_hash.py
+++ /dev/null
@@ -1,8 +0,0 @@
-from __future__ import absolute_import
-import doctest
-import constant_hash
-
-
-def load_tests(loader, tests, ignore):
-    tests.addTests(doctest.DocTestSuite(constant_hash))
-    return tests
diff --git a/cranelift/codegen/meta-python/test_srcgen.py b/cranelift/codegen/meta-python/test_srcgen.py
deleted file mode 100644
index 2fb5e0fb61..0000000000
--- a/cranelift/codegen/meta-python/test_srcgen.py
+++ /dev/null
@@ -1,8 +0,0 @@
-from __future__ import absolute_import
-import doctest
-import srcgen
-
-
-def load_tests(loader, tests, ignore):
-    tests.addTests(doctest.DocTestSuite(srcgen))
-    return tests
diff --git a/cranelift/codegen/meta-python/unique_table.py b/cranelift/codegen/meta-python/unique_table.py
deleted file mode 100644
index d8482b3c7d..0000000000
--- a/cranelift/codegen/meta-python/unique_table.py
+++ /dev/null
@@ -1,80 +0,0 @@
-"""
-Generate a table of unique items.
-
-The `UniqueTable` class collects items into an array, removing duplicates. Each
-item is mapped to its offset in the final array.
-
-This is a compression technique for compile-time generated tables.
-"""
-
-try:
-    from typing import Any, List, Dict, Tuple, Sequence  # noqa
-except ImportError:
-    pass
-
-
-class UniqueTable:
-    """
-    Collect items into the `table` list, removing duplicates.
-    """
-    def __init__(self):
-        # type: () -> None
-        # List of items added in order.
-        self.table = list()  # type: List[Any]
-        # Map item -> index.
-        self.index = dict()  # type: Dict[Any, int]
-
-    def add(self, item):
-        # type: (Any) -> int
-        """
-        Add a single item to the table if it isn't already there.
-
-        Return the offset into `self.table` of the item.
-        """
-        if item in self.index:
-            return self.index[item]
-
-        idx = len(self.table)
-        self.index[item] = idx
-        self.table.append(item)
-        return idx
-
-
-class UniqueSeqTable:
-    """
-    Collect sequences into the `table` list, removing duplicates.
-
-    Sequences don't have to be of the same length.
-    """
-    def __init__(self):
-        # type: () -> None
-        self.table = list()  # type: List[Any]
-        # Map seq -> index.
-        self.index = dict()  # type: Dict[Tuple[Any, ...], int]
-
-    def add(self, seq):
-        # type: (Sequence[Any]) -> int
-        """
-        Add a sequence of items to the table. If the table already contains the
-        items in `seq` in the same order, use those instead.
-
-        Return the offset into `self.table` of the beginning of `seq`.
-        """
-        if len(seq) == 0:
-            return 0
-        tseq = tuple(seq)
-        if tseq in self.index:
-            return self.index[tseq]
-
-        idx = len(self.table)
-        self.table.extend(tseq)
-
-        # Add seq and all sub-sequences to `index`.
-        index = self.index  # type: Dict[Tuple[Any, ...], int]
-        assert index is not None
-        for length in range(1, len(tseq) + 1):
-            for offset in range(len(tseq) - length + 1):
-                key = tseq[offset:offset+length]
-                index[key] = idx + offset
-
-        return idx
diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index 0ee57a0c82..64f4343bf5 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -4,7 +4,6 @@ set -euo pipefail
 # This is the top-level test script:
 #
 # - Check code formatting.
-# - Perform checks on Python code.
 # - Make a debug build.
 # - Make a release build.
 # - Run unit tests for all Rust crates (including the filetests)
@@ -13,10 +12,6 @@ set -euo pipefail
 #
 # All tests run by this script should be passing at all times.
 
-# Disable generation of .pyc files because they cause trouble for vendoring
-# scripts, and this is a build step that isn't run very often anyway.
-export PYTHONDONTWRITEBYTECODE=1
-
 # Repository top-level directory.
 topdir=$(dirname "$0")
 cd "$topdir"
@@ -40,20 +35,6 @@ else
     echo "https://github.com/rust-lang-nursery/rustfmt for more information."
 fi
 
-# Check if any Python files have changed since we last checked them.
-tsfile="$topdir/target/meta-checked"
-meta_python="$topdir/cranelift-codegen/meta-python"
-if [ -f "$tsfile" ]; then
-    needcheck=$(find "$meta_python" -name '*.py' -newer "$tsfile")
-else
-    needcheck=yes
-fi
-if [ -n "$needcheck" ]; then
-    banner "Checking python source files"
-    "$meta_python/check.sh"
-    touch "$tsfile" || echo no target directory
-fi
-
 # Make sure the code builds in release mode.
 banner "Rust release build"
 cargo build --release

From 563525b090d2f8d5935374fab83bf0ecba7013c8 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 20 Jun 2019 18:42:23 +0200
Subject: [PATCH 2481/3084] [meta] Remove mentions to Python in comments of the
 non-meta crate;

---
 cranelift/codegen/build.rs                     | 3 +--
 cranelift/codegen/src/constant_hash.rs         | 6 +++---
 cranelift/codegen/src/ir/builder.rs            | 2 +-
 cranelift/codegen/src/ir/immediates.rs         | 4 ++--
 cranelift/codegen/src/ir/instructions.rs       | 2 +-
 cranelift/codegen/src/ir/types.rs              | 2 +-
 cranelift/codegen/src/isa/arm32/settings.rs    | 6 +++---
 cranelift/codegen/src/isa/arm64/settings.rs    | 6 +++---
 cranelift/codegen/src/isa/enc_tables.rs        | 6 +++---
 cranelift/codegen/src/isa/registers.rs         | 8 ++++----
 cranelift/codegen/src/isa/riscv/settings.rs    | 6 +++---
 cranelift/codegen/src/isa/stack.rs             | 2 +-
 cranelift/codegen/src/isa/x86/settings.rs      | 6 +++---
 cranelift/codegen/src/legalizer/mod.rs         | 6 +++---
 cranelift/codegen/src/predicates.rs            | 2 +-
 cranelift/codegen/src/regalloc/register_set.rs | 6 +++---
 cranelift/codegen/src/settings.rs              | 6 +++---
 17 files changed, 39 insertions(+), 40 deletions(-)

diff --git a/cranelift/codegen/build.rs b/cranelift/codegen/build.rs
index d9d6f13fc5..ef89d48ffe 100644
--- a/cranelift/codegen/build.rs
+++ b/cranelift/codegen/build.rs
@@ -50,8 +50,7 @@ fn main() {
     let crate_dir = cur_dir.as_path();
 
     // Make sure we rebuild if this build script changes (will not happen with
-    // if the path to this file contains non-UTF8 bytes). The `build.py` script
-    // prints out its own dependencies.
+    // if the path to this file contains non-UTF-8 bytes).
     println!(
         "cargo:rerun-if-changed={}",
         crate_dir.join("build.rs").to_str().unwrap()
diff --git a/cranelift/codegen/src/constant_hash.rs b/cranelift/codegen/src/constant_hash.rs
index d6afc5eed0..5785a89825 100644
--- a/cranelift/codegen/src/constant_hash.rs
+++ b/cranelift/codegen/src/constant_hash.rs
@@ -1,6 +1,6 @@
 //! Runtime support for precomputed constant hash tables.
 //!
-//! The `cranelift-codegen/meta-python/constant_hash.py` Python module can generate constant hash tables
+//! The `cranelift-codegen/meta/src/constant_hash.rs` Rust crate can generate constant hash tables
 //! using open addressing and quadratic probing. The hash tables are arrays that are guaranteed to:
 //!
 //! - Have a power-of-two size.
@@ -56,7 +56,7 @@ pub fn probe + ?Sized>(
 }
 
 /// A primitive hash function for matching opcodes.
-/// Must match `cranelift-codegen/meta-python/constant_hash.py` and `cranelift-codegen/meta/constant_hash.rs`.
+/// Must match `cranelift-codegen/meta/src/constant_hash.rs`.
 pub fn simple_hash(s: &str) -> usize {
     let mut h: u32 = 5381;
     for c in s.chars() {
@@ -71,7 +71,7 @@ mod tests {
 
     #[test]
     fn basic() {
-        // c.f. `meta-python/constant_hash.py` tests.
+        // c.f. `meta/src/constant_hash.rs` tests.
         assert_eq!(simple_hash("Hello"), 0x2fa70c01);
         assert_eq!(simple_hash("world"), 0x5b0c31d5);
     }
diff --git a/cranelift/codegen/src/ir/builder.rs b/cranelift/codegen/src/ir/builder.rs
index 43578d5378..13202870f3 100644
--- a/cranelift/codegen/src/ir/builder.rs
+++ b/cranelift/codegen/src/ir/builder.rs
@@ -32,7 +32,7 @@ pub trait InstBuilderBase<'f>: Sized {
     fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'f mut DataFlowGraph);
 }
 
-// Include trait code generated by `cranelift-codegen/meta-python/gen_instr.py`.
+// Include trait code generated by `cranelift-codegen/meta/src/gen_inst.rs`.
 //
 // This file defines the `InstBuilder` trait as an extension of `InstBuilderBase` with methods per
 // instruction format and per opcode.
diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index 36cc470aef..73d1d8cc21 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -1,8 +1,8 @@
 //! Immediate operands for Cranelift instructions
 //!
 //! This module defines the types of immediate operands that can appear on Cranelift instructions.
-//! Each type here should have a corresponding definition in the `cranelift.immediates` Python
-//! module in the meta language.
+//! Each type here should have a corresponding definition in the
+//! `cranelift-codegen/meta/src/shared/immediates` crate in the meta language.
 
 use core::fmt::{self, Display, Formatter};
 use core::mem;
diff --git a/cranelift/codegen/src/ir/instructions.rs b/cranelift/codegen/src/ir/instructions.rs
index ae227dc5f7..8278888d7e 100644
--- a/cranelift/codegen/src/ir/instructions.rs
+++ b/cranelift/codegen/src/ir/instructions.rs
@@ -28,7 +28,7 @@ pub type ValueList = entity::EntityList;
 /// Memory pool for holding value lists. See `ValueList`.
 pub type ValueListPool = entity::ListPool;
 
-// Include code generated by `cranelift-codegen/meta-python/gen_instr.py`. This file contains:
+// Include code generated by `cranelift-codegen/meta/src/gen_inst.rs`. This file contains:
 //
 // - The `pub enum InstructionFormat` enum with all the instruction formats.
 // - The `pub enum InstructionData` enum with all the instruction data fields.
diff --git a/cranelift/codegen/src/ir/types.rs b/cranelift/codegen/src/ir/types.rs
index eee39c83e4..20bed0c4ce 100644
--- a/cranelift/codegen/src/ir/types.rs
+++ b/cranelift/codegen/src/ir/types.rs
@@ -25,7 +25,7 @@ pub struct Type(u8);
 /// Not a valid type. Can't be loaded or stored. Can't be part of a SIMD vector.
 pub const INVALID: Type = Type(0);
 
-/// Start of the lane types. See also `meta-python/cdsl/types.py`.
+/// Start of the lane types. See also `meta/src/cdsl/types.rs`.
 const LANE_BASE: u8 = 0x70;
 
 /// Start of the 2-lane vector types.
diff --git a/cranelift/codegen/src/isa/arm32/settings.rs b/cranelift/codegen/src/isa/arm32/settings.rs
index 084c142399..bef631b2bb 100644
--- a/cranelift/codegen/src/isa/arm32/settings.rs
+++ b/cranelift/codegen/src/isa/arm32/settings.rs
@@ -3,7 +3,7 @@
 use crate::settings::{self, detail, Builder};
 use core::fmt;
 
-// Include code generated by `cranelift-codegen/meta-python/gen_settings.py`. This file contains a public
-// `Flags` struct with an impl for all of the settings defined in
-// `cranelift-codegen/meta-python/isa/arm32/settings.py`.
+// Include code generated by `cranelift-codegen/meta/src/gen_settings.rs`. This file contains a
+// public `Flags` struct with an impl for all of the settings defined in
+// `cranelift-codegen/meta/src/isa/arm32/mod.rs`.
 include!(concat!(env!("OUT_DIR"), "/settings-arm32.rs"));
diff --git a/cranelift/codegen/src/isa/arm64/settings.rs b/cranelift/codegen/src/isa/arm64/settings.rs
index b7be97e368..56d0f4ee0b 100644
--- a/cranelift/codegen/src/isa/arm64/settings.rs
+++ b/cranelift/codegen/src/isa/arm64/settings.rs
@@ -3,7 +3,7 @@
 use crate::settings::{self, detail, Builder};
 use core::fmt;
 
-// Include code generated by `cranelift-codegen/meta-python/gen_settings.py`. This file contains a public
-// `Flags` struct with an impl for all of the settings defined in
-// `cranelift-codegen/meta-python/isa/arm64/settings.py`.
+// Include code generated by `cranelift-codegen/meta/src/gen_settings.rs`. This file contains a
+// public `Flags` struct with an impl for all of the settings defined in
+// `cranelift-codegen/meta/src/isa/arm64/mod.rs`.
 include!(concat!(env!("OUT_DIR"), "/settings-arm64.rs"));
diff --git a/cranelift/codegen/src/isa/enc_tables.rs b/cranelift/codegen/src/isa/enc_tables.rs
index 45ad86441b..e21557497e 100644
--- a/cranelift/codegen/src/isa/enc_tables.rs
+++ b/cranelift/codegen/src/isa/enc_tables.rs
@@ -1,7 +1,7 @@
 //! Support types for generated encoding tables.
 //!
 //! This module contains types and functions for working with the encoding tables generated by
-//! `cranelift-codegen/meta-python/gen_encoding.py`.
+//! `cranelift-codegen/meta/src/gen_encodings.rs`.
 
 use crate::constant_hash::{probe, Table};
 use crate::ir::{Function, InstructionData, Opcode, Type};
@@ -164,10 +164,10 @@ where
 /// Encoding lists are represented as sequences of u16 words.
 pub type EncListEntry = u16;
 
-/// Number of bits used to represent a predicate. c.f. `meta-python/gen_encoding.py`.
+/// Number of bits used to represent a predicate. c.f. `meta/src/gen_encodings.rs`.
 const PRED_BITS: u8 = 12;
 const PRED_MASK: usize = (1 << PRED_BITS) - 1;
-/// First code word representing a predicate check. c.f. `meta-python/gen_encoding.py`.
+/// First code word representing a predicate check. c.f. `meta/src/gen_encodings.rs`.
 const PRED_START: usize = 0x1000;
 
 /// An iterator over legal encodings for the instruction.
diff --git a/cranelift/codegen/src/isa/registers.rs b/cranelift/codegen/src/isa/registers.rs
index 6baebd7237..3c1b4262c1 100644
--- a/cranelift/codegen/src/isa/registers.rs
+++ b/cranelift/codegen/src/isa/registers.rs
@@ -17,19 +17,19 @@ pub type RegUnit = u16;
 /// The size of this type is determined by the target ISA that has the most register units defined.
 /// Currently that is arm32 which has 64+16 units.
 ///
-/// This type should be coordinated with meta-python/cdsl/registers.py.
+/// This type should be coordinated with meta/src/cdsl/regs.rs.
 pub type RegUnitMask = [u32; 3];
 
 /// A bit mask indexed by register classes.
 ///
 /// The size of this type is determined by the ISA with the most register classes.
 ///
-/// This type should be coordinated with meta-python/cdsl/isa.py.
+/// This type should be coordinated with meta/src/cdsl/regs.rs.
 pub type RegClassMask = u32;
 
 /// Guaranteed maximum number of top-level register classes with pressure tracking in any ISA.
 ///
-/// This can be increased, but should be coordinated with meta-python/cdsl/isa.py.
+/// This can be increased, but should be coordinated with meta/src/cdsl/regs.rs.
 pub const MAX_TRACKED_TOPRCS: usize = 4;
 
 /// The register units in a target ISA are divided into disjoint register banks. Each bank covers a
@@ -37,7 +37,7 @@ pub const MAX_TRACKED_TOPRCS: usize = 4;
 ///
 /// The `RegBank` struct provides a static description of a register bank.
 pub struct RegBank {
-    /// The name of this register bank as defined in the ISA's `registers.py` file.
+    /// The name of this register bank as defined in the ISA's DSL definition.
     pub name: &'static str,
 
     /// The first register unit in this bank.
diff --git a/cranelift/codegen/src/isa/riscv/settings.rs b/cranelift/codegen/src/isa/riscv/settings.rs
index 26999b87ee..3599a93495 100644
--- a/cranelift/codegen/src/isa/riscv/settings.rs
+++ b/cranelift/codegen/src/isa/riscv/settings.rs
@@ -3,9 +3,9 @@
 use crate::settings::{self, detail, Builder};
 use core::fmt;
 
-// Include code generated by `cranelift-codegen/meta-python/gen_settings.py`. This file contains a public
-// `Flags` struct with an impl for all of the settings defined in
-// `cranelift-codegen/meta-python/isa/riscv/settings.py`.
+// Include code generated by `cranelift-codegen/meta/src/gen_settings.rs`. This file contains a
+// public `Flags` struct with an impl for all of the settings defined in
+// `cranelift-codegen/meta/src/isa/riscv/mod.rs`.
 include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs"));
 
 #[cfg(test)]
diff --git a/cranelift/codegen/src/isa/stack.rs b/cranelift/codegen/src/isa/stack.rs
index cc7b578039..852aedddd8 100644
--- a/cranelift/codegen/src/isa/stack.rs
+++ b/cranelift/codegen/src/isa/stack.rs
@@ -82,7 +82,7 @@ pub enum StackBase {
 /// This behaves like a set of `StackBase` variants.
 ///
 /// The internal representation as a `u8` is public because stack base masks are used in constant
-/// tables generated from the Python encoding definitions.
+/// tables generated from the meta-language encoding definitions.
 #[derive(Clone, Copy, Debug, PartialEq, Eq)]
 pub struct StackBaseMask(pub u8);
 
diff --git a/cranelift/codegen/src/isa/x86/settings.rs b/cranelift/codegen/src/isa/x86/settings.rs
index ab5e080662..2d3a3f6698 100644
--- a/cranelift/codegen/src/isa/x86/settings.rs
+++ b/cranelift/codegen/src/isa/x86/settings.rs
@@ -3,9 +3,9 @@
 use crate::settings::{self, detail, Builder};
 use core::fmt;
 
-// Include code generated by `cranelift-codegen/meta-python/gen_settings.py`. This file contains a public
-// `Flags` struct with an impl for all of the settings defined in
-// `cranelift-codegen/meta-python/isa/x86/settings.py`.
+// Include code generated by `cranelift-codegen/meta/src/gen_settings.rs:`. This file contains a
+// public `Flags` struct with an impl for all of the settings defined in
+// `cranelift-codegen/meta/src/isa/x86/settings.rs`.
 include!(concat!(env!("OUT_DIR"), "/settings-x86.rs"));
 
 #[cfg(test)]
diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 0a59f0d1b5..b7aa287835 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -117,14 +117,14 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
     }
 }
 
-// Include legalization patterns that were generated by `gen_legalizer.py` from the `XForms` in
-// `cranelift-codegen/meta-python/base/legalize.py`.
+// Include legalization patterns that were generated by `gen_legalizer.rs` from the
+// `TransformGroup` in `cranelift-codegen/meta/shared/legalize.rs`.
 //
 // Concretely, this defines private functions `narrow()`, and `expand()`.
 include!(concat!(env!("OUT_DIR"), "/legalizer.rs"));
 
 /// Custom expansion for conditional trap instructions.
-/// TODO: Add CFG support to the Python patterns so we won't have to do this.
+/// TODO: Add CFG support to the Rust DSL patterns so we won't have to do this.
 fn expand_cond_trap(
     inst: ir::Inst,
     func: &mut ir::Function,
diff --git a/cranelift/codegen/src/predicates.rs b/cranelift/codegen/src/predicates.rs
index da01ad6b2b..978e733c66 100644
--- a/cranelift/codegen/src/predicates.rs
+++ b/cranelift/codegen/src/predicates.rs
@@ -1,7 +1,7 @@
 //! Predicate functions for testing instruction fields.
 //!
 //! This module defines functions that are used by the instruction predicates defined by
-//! `cranelift-codegen/meta-python/cdsl/predicates.py` classes.
+//! `cranelift-codegen/meta/src/cdsl/instructions.rs` classes.
 //!
 //! The predicates the operate on integer fields use `Into` as a shared trait bound. This
 //! bound is implemented by all the native integer types as well as `Imm64`.
diff --git a/cranelift/codegen/src/regalloc/register_set.rs b/cranelift/codegen/src/regalloc/register_set.rs
index 29b6df7625..c9b419a517 100644
--- a/cranelift/codegen/src/regalloc/register_set.rs
+++ b/cranelift/codegen/src/regalloc/register_set.rs
@@ -89,9 +89,9 @@ impl RegisterSet {
 
         // Mask out the unavailable units.
         for idx in 0..self.avail.len() {
-            // If a single unit in a register is unavailable, the whole register can't be used.
-            // If a register straddles a word boundary, it will be marked as unavailable.
-            // There's an assertion in `cdsl/registers.py` to check for that.
+            // If a single unit in a register is unavailable, the whole register can't be used.  If
+            // a register straddles a word boundary, it will be marked as unavailable.  There's an
+            // assertion in `cranelift-codegen/meta/src/cdsl/regs.rs` to check for that.
             for i in 0..rc.width {
                 rsi.regs[idx] &= self.avail[idx] >> i;
             }
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index 748484b10c..836b47a5f6 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -331,9 +331,9 @@ pub mod detail {
     }
 }
 
-// Include code generated by `meta-python/gen_settings.py`. This file contains a public `Flags`
-// struct with an impl for all of the settings defined in
-// `cranelift-codegen/meta-python/base/settings.py`.
+// Include code generated by `meta/gen_settings.rs`. This file contains a public `Flags` struct
+// with an implementation for all of the settings defined in
+// `cranelift-codegen/meta/src/shared/settings.rs`.
 include!(concat!(env!("OUT_DIR"), "/settings.rs"));
 
 /// Wrapper containing flags and optionally a `TargetIsa` trait object.

From bc75eee0cd4fb10c18ad8a4f73573dd40fd3d365 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Thu, 27 Jun 2019 15:46:31 +0200
Subject: [PATCH 2482/3084] Use BB-like EBB in docs/*.clif

---
 cranelift/docs/callex.clif  |  7 +++++--
 cranelift/docs/example.clif | 18 ++++++++++++------
 2 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/cranelift/docs/callex.clif b/cranelift/docs/callex.clif
index 853f5c4091..1d93239199 100644
--- a/cranelift/docs/callex.clif
+++ b/cranelift/docs/callex.clif
@@ -4,10 +4,13 @@ function %gcd(i32 uext, i32 uext) -> i32 uext system_v {
     fn0 = %divmod(i32 uext, i32 uext) -> i32 uext, i32 uext
 
 ebb1(v0: i32, v1: i32):
-    brz v1, ebb2
+    brz v1, ebb3
+    jump ebb2
+
+ebb2:
     v2, v3 = call fn0(v0, v1)
     return v2
 
-ebb2:
+ebb3:
     return v0
 }
diff --git a/cranelift/docs/example.clif b/cranelift/docs/example.clif
index 2bc5c9cc4c..b848f7026a 100644
--- a/cranelift/docs/example.clif
+++ b/cranelift/docs/example.clif
@@ -6,11 +6,14 @@ function %average(i32, i32) -> f32 system_v {
 ebb1(v0: i32, v1: i32):
     v2 = f64const 0x0.0
     stack_store v2, ss0
-    brz v1, ebb3                  ; Handle count == 0.
-    v3 = iconst.i32 0
-    jump ebb2(v3)
+    brz v1, ebb5                  ; Handle count == 0.
+    jump ebb2
 
-ebb2(v4: i32):
+ebb2:
+    v3 = iconst.i32 0
+    jump ebb3(v3)
+
+ebb3(v4: i32):
     v5 = imul_imm v4, 4
     v6 = iadd v0, v5
     v7 = load.f32 v6              ; array[i]
@@ -20,14 +23,17 @@ ebb2(v4: i32):
     stack_store v10, ss0
     v11 = iadd_imm v4, 1
     v12 = icmp ult v11, v1
-    brnz v12, ebb2(v11)           ; Loop backedge.
+    brnz v12, ebb3(v11)           ; Loop backedge.
+    jump ebb4
+
+ebb4:
     v13 = stack_load.f64 ss0
     v14 = fcvt_from_uint.f64 v1
     v15 = fdiv v13, v14
     v16 = fdemote.f32 v15
     return v16
 
-ebb3:
+ebb5:
     v100 = f32const +NaN
     return v100
 }

From 44abcbec1e93e8f87df4fd8d3da61b86b960e0e9 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Thu, 27 Jun 2019 15:51:51 +0200
Subject: [PATCH 2483/3084] Use BB-like EBB in filetests/licm/*.clif

---
 cranelift/filetests/filetests/licm/basic.clif | 16 +++++---
 .../filetests/filetests/licm/complex.clif     | 14 +++++--
 .../filetests/licm/critical-edge.clif         | 40 ++++++++++++-------
 .../filetests/filetests/licm/encoding.clif    | 16 +++++---
 .../filetests/licm/load_readonly_notrap.clif  | 16 +++++---
 .../filetests/licm/multiple-blocks.clif       | 32 ++++++++++-----
 .../filetests/licm/nested_loops.clif          | 24 ++++++-----
 .../filetests/filetests/licm/reject.clif      | 28 +++++++++----
 .../filetests/licm/reject_load_notrap.clif    | 16 +++++---
 .../filetests/licm/reject_load_readonly.clif  | 16 +++++---
 10 files changed, 148 insertions(+), 70 deletions(-)

diff --git a/cranelift/filetests/filetests/licm/basic.clif b/cranelift/filetests/filetests/licm/basic.clif
index f409523c70..3f5dfbbe14 100644
--- a/cranelift/filetests/filetests/licm/basic.clif
+++ b/cranelift/filetests/filetests/licm/basic.clif
@@ -10,11 +10,14 @@ ebb1(v1: i32):
     v2 = iconst.i32 1
     v3 = iconst.i32 2
     v4 = iadd v2, v3
-    brz v1, ebb2(v1)
+    brz v1, ebb3(v1)
+    jump ebb2
+
+ebb2:
     v5 = isub v1, v2
     jump ebb1(v5)
 
-ebb2(v6: i32):
+ebb3(v6: i32):
     return v6
 
 }
@@ -26,10 +29,13 @@ ebb2(v6: i32):
 ; nextln:     jump ebb1(v0)
 ; nextln: 
 ; nextln: ebb1(v1: i32):
-; nextln:     brz v1, ebb2(v1)
-; nextln:     v5 = isub v1, v2
+; nextln:     brz v1, ebb3(v1)
+; nextln:     jump ebb2
+; nextln: 
+; nextln: ebb2:
+; nextln:     v5 = isub.i32 v1, v2
 ; nextln:     jump ebb1(v5)
 ; nextln: 
-; nextln: ebb2(v6: i32):
+; nextln: ebb3(v6: i32):
 ; nextln:     return v6
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/complex.clif b/cranelift/filetests/filetests/licm/complex.clif
index c7d26f9a33..e2bfffc3d6 100644
--- a/cranelift/filetests/filetests/licm/complex.clif
+++ b/cranelift/filetests/filetests/licm/complex.clif
@@ -39,6 +39,9 @@ ebb0(v0: i32):
             v19 = iadd v18, v2
             v20 = iadd.i32 v2, v3
 [SBzero#18] brz.i32 v1, ebb1(v20)
+            fallthrough ebb7
+
+         ebb7:
 [Iret#19]   return v19
 }
 
@@ -53,10 +56,10 @@ ebb0(v0: i32):
 ; nextln: 
 ; nextln: ebb1(v1: i32):
 ; nextln:     v4 = iadd.i32 v2, v1
-; nextln:     brz v1, ebb7(v2)
-; nextln:     jump ebb8(v4)
+; nextln:     brz v1, ebb8(v2)
+; nextln:     jump ebb9(v4)
 ; nextln: 
-; nextln: ebb7(v21: i32):
+; nextln: ebb8(v21: i32):
 ; nextln:     v8 = iadd.i32 v6, v1
 ; nextln:     v11 = iadd.i32 v1, v4
 ; nextln:     jump ebb2(v21)
@@ -70,7 +73,7 @@ ebb0(v0: i32):
 ; nextln:     brz.i32 v1, ebb2(v9)
 ; nextln:     jump ebb6(v10)
 ; nextln: 
-; nextln: ebb8(v22: i32):
+; nextln: ebb9(v22: i32):
 ; nextln:     v15 = iadd.i32 v4, v13
 ; nextln:     jump ebb4(v22)
 ; nextln: 
@@ -86,5 +89,8 @@ ebb0(v0: i32):
 ; nextln: ebb6(v18: i32):
 ; nextln:     v19 = iadd v18, v2
 ; nextln:     brz.i32 v1, ebb1(v20)
+; nextln:     fallthrough ebb7
+; nextln: 
+; nextln: ebb7:
 ; nextln:     return v19
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/critical-edge.clif b/cranelift/filetests/filetests/licm/critical-edge.clif
index 1c69f6364e..b3636c10e0 100644
--- a/cranelift/filetests/filetests/licm/critical-edge.clif
+++ b/cranelift/filetests/filetests/licm/critical-edge.clif
@@ -6,37 +6,49 @@ target riscv32
 function %critical_edge(i32, i32) -> i32 {
 
             ebb0(v0: i32, v7: i32):
-[SBzero#38]   brnz v7, ebb1(v0)
+[SBzero#38]   brnz v7, ebb2(v0)
+[-]           fallthrough ebb1
+
+            ebb1:
 [Iret#19]     return v0
 
-            ebb1(v1: i32):
+            ebb2(v1: i32):
               v2 = iconst.i32 1
               v3 = iconst.i32 2
               v4 = iadd v2, v3
-[SBzero#18]   brz v1, ebb2(v1)
-              v5 = isub v1, v2
-[UJ#1b]       jump ebb1(v5)
+[SBzero#18]   brz v1, ebb4(v1)
+[UJ#1b]       jump ebb3
 
-            ebb2(v6: i32):
+            ebb3:
+              v5 = isub v1, v2
+[UJ#1b]       jump ebb2(v5)
+
+            ebb4(v6: i32):
 [Iret#19]     return v6
 
 }
 ; sameln: function %critical_edge
 ; nextln: ebb0(v0: i32, v7: i32):
-; nextln:     brnz v7, ebb3(v0)
+; nextln:     brnz v7, ebb5(v0)
+; nextln:     fallthrough ebb1
+; nextln: 
+; nextln: ebb1:
 ; nextln:     return v0
 ; nextln: 
-; nextln: ebb3(v8: i32):
+; nextln: ebb5(v8: i32):
 ; nextln:     v2 = iconst.i32 1
 ; nextln:     v3 = iconst.i32 2
 ; nextln:     v4 = iadd v2, v3
-; nextln:     jump ebb1(v8)
+; nextln:     jump ebb2(v8)
 ; nextln: 
-; nextln: ebb1(v1: i32):
-; nextln:     brz v1, ebb2(v1)
-; nextln:     v5 = isub v1, v2
-; nextln:     jump ebb1(v5)
+; nextln: ebb2(v1: i32):
+; nextln:     brz v1, ebb4(v1)
+; nextln:     jump ebb3
 ; nextln: 
-; nextln: ebb2(v6: i32):
+; nextln: ebb3:
+; nextln:     v5 = isub.i32 v1, v2
+; nextln:     jump ebb2(v5)
+; nextln: 
+; nextln: ebb4(v6: i32):
 ; nextln:     return v6
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/encoding.clif b/cranelift/filetests/filetests/licm/encoding.clif
index 27a965f55b..b029a51c18 100644
--- a/cranelift/filetests/filetests/licm/encoding.clif
+++ b/cranelift/filetests/filetests/licm/encoding.clif
@@ -11,11 +11,14 @@ function %simple_loop(i32) -> i32 {
 [Iz#04,%x0]   v2 = iconst.i32 1
 [Iz#04,%x1]   v3 = iconst.i32 2
 [R#0c,%x2]    v4 = iadd v2, v3
-[SBzero#18]   brz v1, ebb2(v1)
+[SBzero#18]   brz v1, ebb3(v1)
+[UJ#1b]       jump ebb2
+
+            ebb2:
 [R#200c,%x5]  v5 = isub v1, v2
 [UJ#1b]       jump ebb1(v5)
 
-            ebb2(v6: i32):
+            ebb3(v6: i32):
 [Iret#19]     return v6
 }
 
@@ -27,10 +30,13 @@ function %simple_loop(i32) -> i32 {
 ; nextln: [UJ#1b]                             jump ebb1(v0)
 ; nextln: 
 ; nextln:                                 ebb1(v1: i32):
-; nextln: [SBzero#18]                         brz v1, ebb2(v1)
-; nextln: [R#200c,%x5]                        v5 = isub v1, v2
+; nextln: [SBzero#18]                         brz v1, ebb3(v1)
+; nextln: [UJ#1b]                             jump ebb2
+; nextln: 
+; nextln:                                 ebb2:
+; nextln: [R#200c,%x5]                        v5 = isub.i32 v1, v2
 ; nextln: [UJ#1b]                             jump ebb1(v5)
 ; nextln: 
-; nextln:                                 ebb2(v6: i32):
+; nextln:                                 ebb3(v6: i32):
 ; nextln: [Iret#19]                           return v6
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/load_readonly_notrap.clif b/cranelift/filetests/filetests/licm/load_readonly_notrap.clif
index 9a9d2dcbfa..4731bd664e 100644
--- a/cranelift/filetests/filetests/licm/load_readonly_notrap.clif
+++ b/cranelift/filetests/filetests/licm/load_readonly_notrap.clif
@@ -18,11 +18,14 @@ ebb1(v2: i32, v3: i64):
     v5 = heap_addr.i64 heap0, v4, 1
     v6 = load.i32 notrap aligned readonly v5
     v7 = iadd v2, v6
-    brz v2, ebb2(v2)
+    brz v2, ebb3(v2)
+    jump ebb2
+
+ebb2:
     v8 = isub v2, v4
     jump ebb1(v8, v3)
 
-ebb2(v9: i32):
+ebb3(v9: i32):
     return v9
 }
 
@@ -39,10 +42,13 @@ ebb2(v9: i32):
 ; nextln: 
 ; nextln: ebb1(v2: i32, v3: i64):
 ; nextln:    v7 = iadd v2, v6
-; nextln:    brz v2, ebb2(v2)
-; nextln:    v8 = isub v2, v4
+; nextln:    brz v2, ebb3(v2)
+; nextln:     jump ebb2
+; nextln: 
+; nextln: ebb2:
+; nextln:    v8 = isub.i32 v2, v4
 ; nextln:    jump ebb1(v8, v3)
 ; nextln: 
-; nextln: ebb2(v9: i32):
+; nextln: ebb3(v9: i32):
 ; nextln:    return v9
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/multiple-blocks.clif b/cranelift/filetests/filetests/licm/multiple-blocks.clif
index de5af884e8..ea23505ef5 100644
--- a/cranelift/filetests/filetests/licm/multiple-blocks.clif
+++ b/cranelift/filetests/filetests/licm/multiple-blocks.clif
@@ -10,16 +10,22 @@ ebb1(v10: i32):
     v11 = iconst.i32 1
     v12 = iconst.i32 2
     v13 = iadd v11, v12
-    brz v10, ebb2(v10)
+    brz v10, ebb4(v10)
+    jump ebb2
+
+ebb2:
     v15 = isub v10, v11
-    brz v15, ebb3(v15)
+    brz v15, ebb5(v15)
+    jump ebb3
+
+ebb3:
     v14 = isub v10, v11
     jump ebb1(v14)
 
-ebb2(v20: i32):
+ebb4(v20: i32):
     return v20
 
-ebb3(v30: i32):
+ebb5(v30: i32):
     v31 = iadd v11, v13
     jump ebb1(v30)
 
@@ -33,15 +39,21 @@ ebb3(v30: i32):
 ; nextln:     jump ebb1(v0)
 ; nextln: 
 ; nextln: ebb1(v10: i32):
-; nextln:     brz v10, ebb2(v10)
-; nextln:     v15 = isub v10, v11
-; nextln:     brz v15, ebb3(v15)
-; nextln:     v14 = isub v10, v11
+; nextln:     brz v10, ebb4(v10)
+; nextln:     jump ebb2
+; nextln: 
+; nextln: ebb2:
+; nextln:     v15 = isub.i32 v10, v11
+; nextln:     brz v15, ebb5(v15)
+; nextln:     jump ebb3
+; nextln: 
+; nextln: ebb3:
+; nextln:     v14 = isub.i32 v10, v11
 ; nextln:     jump ebb1(v14)
 ; nextln: 
-; nextln: ebb2(v20: i32):
+; nextln: ebb4(v20: i32):
 ; nextln:     return v20
 ; nextln: 
-; nextln: ebb3(v30: i32):
+; nextln: ebb5(v30: i32):
 ; nextln:     jump ebb1(v30)
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/nested_loops.clif b/cranelift/filetests/filetests/licm/nested_loops.clif
index f9ade38cf2..423b24d33f 100644
--- a/cranelift/filetests/filetests/licm/nested_loops.clif
+++ b/cranelift/filetests/filetests/licm/nested_loops.clif
@@ -14,17 +14,20 @@ ebb1(v1: i32):
     jump ebb2(v5, v5)
 
 ebb2(v10: i32, v11: i32):
-    brz v11, ebb3(v10)
+    brz v11, ebb4(v10)
+    jump ebb3
+
+ebb3:
     v12 = iconst.i32 1
     v15 = iadd v12, v5
     v13 = isub v11, v12
     jump ebb2(v10,v13)
 
-ebb3(v20: i32):
-    brz v20, ebb4(v20)
+ebb4(v20: i32):
+    brz v20, ebb5(v20)
     jump ebb1(v20)
 
-ebb4(v30: i32):
+ebb5(v30: i32):
     return v30
 
 }
@@ -43,14 +46,17 @@ ebb4(v30: i32):
 ; nextln:     jump ebb2(v5, v5)
 ; nextln: 
 ; nextln: ebb2(v10: i32, v11: i32):
-; nextln:     brz v11, ebb3(v10)
-; nextln:     v13 = isub v11, v12
+; nextln:     brz v11, ebb4(v10)
+; nextln:     jump ebb3
+; nextln: 
+; nextln: ebb3:
+; nextln:     v13 = isub.i32 v11, v12
 ; nextln:     jump ebb2(v10, v13)
 ; nextln: 
-; nextln: ebb3(v20: i32):
-; nextln:     brz v20, ebb4(v20)
+; nextln: ebb4(v20: i32):
+; nextln:     brz v20, ebb5(v20)
 ; nextln:     jump ebb1(v20)
 ; nextln: 
-; nextln: ebb4(v30: i32):
+; nextln: ebb5(v30: i32):
 ; nextln:     return v30
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/reject.clif b/cranelift/filetests/filetests/licm/reject.clif
index a23decb1eb..43823c1295 100644
--- a/cranelift/filetests/filetests/licm/reject.clif
+++ b/cranelift/filetests/filetests/licm/reject.clif
@@ -11,11 +11,14 @@ ebb1(v1: i32):
 ; check: ebb1(v1: i32):
 ; check: regmove.i32 v0, %x10 -> %x20
     v2 = iconst.i32 1
-    brz v1, ebb2(v1)
+    brz v1, ebb3(v1)
+    jump ebb2
+
+ebb2:
     v5 = isub v1, v2
     jump ebb1(v5)
 
-ebb2(v6: i32):
+ebb3(v6: i32):
     return v6
 
 }
@@ -31,12 +34,15 @@ ebb1(v2: i32, v3: i32):
 ; check: ifcmp.i32 v0, v1
 ; check: v5 = selectif.i32 eq v4, v2, v3
     v8 = iconst.i32 1
-    brz v1, ebb2(v1)
+    brz v1, ebb3(v1)
+    jump ebb2
+
+ebb2:
     v9 = isub v1, v8
     v10 = iadd v1, v8
     jump ebb1(v9, v10)
 
-ebb2(v6: i32):
+ebb3(v6: i32):
     return v6
 }
 
@@ -53,11 +59,14 @@ ebb1(v3: i32, v4: i32):
 ; check: v5 = spill.i32 v1
 ; check: v6 = fill.i32 v2
 ; check: v7 = fill v5
-    brz v1, ebb2(v1)
+    brz v1, ebb3(v1)
+    jump ebb2
+
+ebb2:
     v9 = isub v1, v4
     jump ebb1(v9, v3)
 
-ebb2(v10: i32):
+ebb3(v10: i32):
     return v10
 }
 
@@ -72,11 +81,14 @@ ebb1(v1: i32):
     v2 = iadd v8, v9
 ; check: ebb1(v1: i32):
 ; check: v2 = iadd v8, v9
-    brz v1, ebb2(v1)
+    brz v1, ebb3(v1)
+    jump ebb2
+
+ebb2:
     v5 = isub v1, v2
     jump ebb1(v5)
 
-ebb2(v6: i32):
+ebb3(v6: i32):
     return v6
 
 }
diff --git a/cranelift/filetests/filetests/licm/reject_load_notrap.clif b/cranelift/filetests/filetests/licm/reject_load_notrap.clif
index 1d26faa71f..71385807e3 100644
--- a/cranelift/filetests/filetests/licm/reject_load_notrap.clif
+++ b/cranelift/filetests/filetests/licm/reject_load_notrap.clif
@@ -19,11 +19,14 @@ ebb0(v0: i32, v1: i64):
 ebb1(v2: i32, v3: i64):
     v6 = load.i32 notrap aligned v5
     v7 = iadd v2, v6
-    brz v2, ebb2(v2)
+    brz v2, ebb3(v2)
+    jump ebb2
+
+ebb2:
     v8 = isub v2, v4
     jump ebb1(v8, v3)
 
-ebb2(v9: i32):
+ebb3(v9: i32):
     return v9
 }
 
@@ -40,10 +43,13 @@ ebb2(v9: i32):
 ; nextln: ebb1(v2: i32, v3: i64):
 ; nextln:    v6 = load.i32 notrap aligned v5
 ; nextln:    v7 = iadd v2, v6
-; nextln:    brz v2, ebb2(v2)
-; nextln:    v8 = isub v2, v4
+; nextln:    brz v2, ebb3(v2)
+; nextln:    jump ebb2
+; nextln: 
+; nextln: ebb2:
+; nextln:    v8 = isub.i32 v2, v4
 ; nextln:    jump ebb1(v8, v3)
 ; nextln: 
-; nextln: ebb2(v9: i32):
+; nextln: ebb3(v9: i32):
 ; nextln:    return v9
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/reject_load_readonly.clif b/cranelift/filetests/filetests/licm/reject_load_readonly.clif
index 5b6a411712..ea7b72469e 100644
--- a/cranelift/filetests/filetests/licm/reject_load_readonly.clif
+++ b/cranelift/filetests/filetests/licm/reject_load_readonly.clif
@@ -19,11 +19,14 @@ ebb1(v2: i32, v3: i64):
     v5 = heap_addr.i64 heap0, v4, 1
     v6 = load.i32 aligned readonly v5
     v7 = iadd v2, v6
-    brz v2, ebb2(v2)
+    brz v2, ebb3(v2)
+    jump ebb2
+
+ebb2:
     v8 = isub v2, v4
     jump ebb1(v8, v3)
 
-ebb2(v9: i32):
+ebb3(v9: i32):
     return v9
 }
 
@@ -40,10 +43,13 @@ ebb2(v9: i32):
 ; nextln: ebb1(v2: i32, v3: i64):
 ; nextln:    v6 = load.i32 aligned readonly v5
 ; nextln:    v7 = iadd v2, v6
-; nextln:    brz v2, ebb2(v2)
-; nextln:    v8 = isub v2, v4
+; nextln:    brz v2, ebb3(v2)
+; nextln:    jump ebb2
+; nextln: 
+; nextln: ebb2:
+; nextln:    v8 = isub.i32 v2, v4
 ; nextln:    jump ebb1(v8, v3)
 ; nextln: 
-; nextln: ebb2(v9: i32):
+; nextln: ebb3(v9: i32):
 ; nextln:    return v9
 ; nextln: }

From 40eede927b7a6cf2787a9a0a0c065989bda65616 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 28 Jun 2019 13:01:36 +0200
Subject: [PATCH 2484/3084] Use BB-like EBB in filetests/domtree/*.clif

---
 .../filetests/filetests/domtree/loops.clif    | 22 ++++++++++++++----
 .../filetests/filetests/domtree/loops2.clif   | 23 +++++++++++++++----
 .../filetests/domtree/tall-tree.clif          | 10 ++++++--
 .../filetests/domtree/wide-tree.clif          | 21 ++++++++++++++---
 4 files changed, 62 insertions(+), 14 deletions(-)

diff --git a/cranelift/filetests/filetests/domtree/loops.clif b/cranelift/filetests/filetests/domtree/loops.clif
index fda2984e94..3f3fafb01d 100644
--- a/cranelift/filetests/filetests/domtree/loops.clif
+++ b/cranelift/filetests/filetests/domtree/loops.clif
@@ -16,6 +16,8 @@ function %test(i32) {
         jump ebb5
     ebb5:
         brz v0, ebb4
+        jump ebb6     ; dominates: ebb6
+    ebb6:
         return
 }
 ; Fall-through-first, prune-at-source DFT:
@@ -28,7 +30,9 @@ function %test(i32) {
 ;                 ebb2:brz v3, ebb1 -
 ;                 ebb2:brz v4, ebb4 {
 ;                     ebb2: jump ebb5 {
-;                         ebb5 {}
+;                         ebb5: jump ebb6 {
+;                             ebb6 {}
+;                         }
 ;                     }
 ;                     ebb4 {}
 ;                 }
@@ -43,6 +47,7 @@ function %test(i32) {
 ; } ebb0
 ;
 ; check: cfg_postorder:
+; sameln: ebb6
 ; sameln: ebb5
 ; sameln: ebb3
 ; sameln: ebb4
@@ -56,7 +61,8 @@ function %test(i32) {
 ; nextln: ebb2:
 ; nextln: ebb4:
 ; nextln: ebb3:
-; nextln: ebb5:
+; nextln: ebb5: ebb6
+; nextln: ebb6:
 ; nextln: }
 
 function %loop2(i32) system_v {
@@ -72,10 +78,14 @@ function %loop2(i32) system_v {
         jump ebb4
     ebb4:
         brz v0, ebb3
+        jump ebb8       ; dominates: ebb8
+    ebb8:
         brnz v0, ebb5
         jump ebb6       ; dominates: ebb6
     ebb5:
         brz v0, ebb4
+        jump ebb9       ; dominates: ebb9
+    ebb9:
         trap user0
     ebb6:
         jump ebb7       ; dominates: ebb7
@@ -83,9 +93,11 @@ function %loop2(i32) system_v {
         return
 }
 ; check: cfg_postorder:
+; sameln: ebb9
 ; sameln: ebb5
 ; sameln: ebb7
 ; sameln: ebb6
+; sameln: ebb8
 ; sameln: ebb3
 ; sameln: ebb4
 ; sameln: ebb2
@@ -96,9 +108,11 @@ function %loop2(i32) system_v {
 ; nextln: ebb0: ebb1 ebb2 ebb4 ebb3 ebb5
 ; nextln: ebb1:
 ; nextln: ebb2:
-; nextln: ebb4: ebb6
+; nextln: ebb4: ebb8
+; nextln: ebb8: ebb6
 ; nextln: ebb6: ebb7
 ; nextln: ebb7:
 ; nextln: ebb3:
-; nextln: ebb5:
+; nextln: ebb5: ebb9
+; nextln: ebb9:
 ; nextln: }
diff --git a/cranelift/filetests/filetests/domtree/loops2.clif b/cranelift/filetests/filetests/domtree/loops2.clif
index 80e00ca278..84712c112f 100644
--- a/cranelift/filetests/filetests/domtree/loops2.clif
+++ b/cranelift/filetests/filetests/domtree/loops2.clif
@@ -3,6 +3,8 @@ test domtree
 function %loop1(i32) {
     ebb0(v0: i32):
         brz v0, ebb1    ; dominates: ebb1 ebb6
+        jump ebb10      ; dominates: ebb10
+    ebb10:
         brnz v0, ebb2   ; dominates: ebb2 ebb9
         jump ebb3       ; dominates: ebb3
     ebb1:
@@ -14,10 +16,14 @@ function %loop1(i32) {
         jump ebb9
     ebb4:
         brz v0, ebb4
+        jump ebb11      ; dominates: ebb11
+    ebb11:
         brnz v0, ebb6
         jump ebb7
     ebb5:
         brz v0, ebb7
+        jump ebb12      ; dominates: ebb12
+    ebb12:
         brnz v0, ebb8
         jump ebb9
     ebb6:
@@ -31,16 +37,19 @@ function %loop1(i32) {
 }
 
 ; check: domtree_preorder {
-; nextln: ebb0: ebb1 ebb2 ebb6 ebb3 ebb9
+; nextln: ebb0: ebb1 ebb10 ebb6
 ; nextln: ebb1:
+; nextln: ebb10: ebb2 ebb3 ebb9
 ; nextln: ebb2: ebb4 ebb5 ebb7 ebb8
-; nextln: ebb4:
-; nextln: ebb5:
+; nextln: ebb4: ebb11
+; nextln: ebb11:
+; nextln: ebb5: ebb12
+; nextln: ebb12:
 ; nextln: ebb7:
 ; nextln: ebb8:
-; nextln: ebb6:
 ; nextln: ebb3:
 ; nextln: ebb9:
+; nextln: ebb6:
 ; nextln: }
 
 function %loop2(i32) system_v {
@@ -59,9 +68,12 @@ function %loop2(i32) system_v {
         jump ebb5
     ebb5:
         brz v0, ebb4
+        jump ebb6       ; dominates: ebb6
+    ebb6:
         return
 }
 ; check: cfg_postorder:
+; sameln: ebb6
 ; sameln: ebb5
 ; sameln: ebb3
 ; sameln: ebb4
@@ -75,5 +87,6 @@ function %loop2(i32) system_v {
 ; nextln: ebb2:
 ; nextln: ebb4:
 ; nextln: ebb3:
-; nextln: ebb5:
+; nextln: ebb5: ebb6
+; nextln: ebb6:
 ; nextln: }
diff --git a/cranelift/filetests/filetests/domtree/tall-tree.clif b/cranelift/filetests/filetests/domtree/tall-tree.clif
index cad763fc36..6f93c023e8 100644
--- a/cranelift/filetests/filetests/domtree/tall-tree.clif
+++ b/cranelift/filetests/filetests/domtree/tall-tree.clif
@@ -3,6 +3,8 @@ test domtree
 function %test(i32) {
     ebb0(v0: i32):
         brz v0, ebb1    ; dominates: ebb1
+        jump ebb12      ; dominates: ebb12
+    ebb12:
         brnz v0, ebb2   ; dominates: ebb2 ebb5
         jump ebb3       ; dominates: ebb3
     ebb1:
@@ -18,6 +20,8 @@ function %test(i32) {
         return
     ebb6:
         brz v0, ebb8    ; dominates: ebb11 ebb8
+        jump ebb13      ; dominates: ebb13
+    ebb13:
         brnz v0, ebb9   ; dominates: ebb9
         jump ebb10
     ebb7:
@@ -33,15 +37,17 @@ function %test(i32) {
 }
 
 ; check: domtree_preorder {
-; nextln: ebb0: ebb1 ebb2 ebb3 ebb5
+; nextln: ebb0: ebb1 ebb12
 ; nextln: ebb1: ebb4
 ; nextln: ebb4: ebb6 ebb7 ebb10
-; nextln: ebb6: ebb8 ebb9 ebb11
+; nextln: ebb6: ebb8 ebb13 ebb11
 ; nextln: ebb8:
+; nextln: ebb13: ebb9
 ; nextln: ebb9:
 ; nextln: ebb11:
 ; nextln: ebb7:
 ; nextln: ebb10:
+; nextln: ebb12: ebb2 ebb3 ebb5
 ; nextln: ebb2:
 ; nextln: ebb3:
 ; nextln: ebb5:
diff --git a/cranelift/filetests/filetests/domtree/wide-tree.clif b/cranelift/filetests/filetests/domtree/wide-tree.clif
index ae943dd7c2..fdfdc169a1 100644
--- a/cranelift/filetests/filetests/domtree/wide-tree.clif
+++ b/cranelift/filetests/filetests/domtree/wide-tree.clif
@@ -6,8 +6,14 @@ function %test(i32) {
         jump ebb1       ; dominates: ebb1
     ebb1:
         brz v0, ebb2    ; dominates: ebb2 ebb7
+        jump ebb20      ; dominates: ebb20
+    ebb20:
         brnz v0, ebb3   ; dominates: ebb3
+        jump ebb21      ; dominates: ebb21
+    ebb21:
         brz v0, ebb4    ; dominates: ebb4
+        jump ebb22      ; dominates: ebb22
+    ebb22:
         brnz v0, ebb5   ; dominates: ebb5
         jump ebb6       ; dominates: ebb6
     ebb2:
@@ -22,7 +28,11 @@ function %test(i32) {
         jump ebb7
     ebb7:
         brnz v0, ebb8   ; dominates: ebb8 ebb12
+        jump ebb23      ; dominates: ebb23
+    ebb23:
         brz v0, ebb9    ; dominates: ebb9
+        jump ebb24      ; dominates: ebb24
+    ebb24:
         brnz v0, ebb10  ; dominates: ebb10
         jump ebb11      ; dominates: ebb11
     ebb8:
@@ -43,16 +53,21 @@ function %test(i32) {
 ; check: domtree_preorder {
 ; nextln: ebb0: ebb13 ebb1
 ; nextln: ebb13:
-; nextln: ebb1: ebb2 ebb3 ebb4 ebb5 ebb6 ebb7
+; nextln: ebb1: ebb2 ebb20 ebb7
 ; nextln: ebb2:
+; nextln: ebb20: ebb3 ebb21
 ; nextln: ebb3:
+; nextln: ebb21: ebb4 ebb22
 ; nextln: ebb4:
+; nextln: ebb22: ebb5 ebb6
 ; nextln: ebb5:
 ; nextln: ebb6:
-; nextln: ebb7: ebb8 ebb9 ebb10 ebb12 ebb11
+; nextln: ebb7: ebb8 ebb23 ebb12
 ; nextln: ebb8:
+; nextln: ebb23: ebb9 ebb24
 ; nextln: ebb9:
+; nextln: ebb24: ebb10 ebb11
 ; nextln: ebb10:
-; nextln: ebb12:
 ; nextln: ebb11:
+; nextln: ebb12:
 ; nextln: }

From 77eb34e20fb3c32efcacb783a0d0885b1d60f394 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 28 Jun 2019 14:29:30 +0200
Subject: [PATCH 2485/3084] Use BB-like EBB in filetests/isa/riscv/*.clif

---
 .../filetests/isa/riscv/binary32.clif         | 46 ++++++++++++++++++-
 1 file changed, 45 insertions(+), 1 deletion(-)

diff --git a/cranelift/filetests/filetests/isa/riscv/binary32.clif b/cranelift/filetests/filetests/isa/riscv/binary32.clif
index 4d63084f9a..b76bfcce69 100644
--- a/cranelift/filetests/filetests/isa/riscv/binary32.clif
+++ b/cranelift/filetests/filetests/isa/riscv/binary32.clif
@@ -95,36 +95,77 @@ ebb0(v9999: i32):
     call_indirect sig0, v2()                    ; bin: 000a80e7
 
     brz v1, ebb3
-    brnz v1, ebb1
+    fallthrough ebb4
 
+ebb4:
+    brnz v1, ebb1
+    fallthrough ebb5
+
+ebb5:
     ; jalr %x0, %x1, 0
     return v9999                        ; bin: 00008067
 
 ebb1:
     ; beq 0x000
     br_icmp eq v1, v2, ebb1             ; bin: 01550063
+    fallthrough ebb100
+
+ebb100:
     ; bne 0xffc
     br_icmp ne v1, v2, ebb1             ; bin: ff551ee3
+    fallthrough ebb101
+
+ebb101:
     ; blt 0xff8
     br_icmp slt v1, v2, ebb1            ; bin: ff554ce3
+    fallthrough ebb102
+
+ebb102:
     ; bge 0xff4
     br_icmp sge v1, v2, ebb1            ; bin: ff555ae3
+    fallthrough ebb103
+
+ebb103:
     ; bltu 0xff0
     br_icmp ult v1, v2, ebb1            ; bin: ff5568e3
+    fallthrough ebb104
+
+ebb104:
     ; bgeu 0xfec
     br_icmp uge v1, v2, ebb1            ; bin: ff5576e3
+    fallthrough ebb105
+
+ebb105:
 
     ; Forward branches.
+    fallthrough ebb106
+
+ebb106:
     ; beq 0x018
     br_icmp eq v2, v1, ebb2             ; bin: 00aa8c63
+    fallthrough ebb107
+
+ebb107:
     ; bne 0x014
     br_icmp ne v2, v1, ebb2             ; bin: 00aa9a63
+    fallthrough ebb108
+
+ebb108:
     ; blt 0x010
     br_icmp slt v2, v1, ebb2            ; bin: 00aac863
+    fallthrough ebb109
+
+ebb109:
     ; bge 0x00c
     br_icmp sge v2, v1, ebb2            ; bin: 00aad663
+    fallthrough ebb110
+
+ebb110:
     ; bltu 0x008
     br_icmp ult v2, v1, ebb2            ; bin: 00aae463
+    fallthrough ebb111
+
+ebb111:
     ; bgeu 0x004
     br_icmp uge v2, v1, ebb2            ; bin: 00aaf263
 
@@ -137,6 +178,9 @@ ebb2:
 ebb3:
     ; beq x, %x0
     brz v1, ebb3                        ; bin: 00050063
+    fallthrough ebb6
+
+ebb6:
     ; bne x, %x0
     brnz v1, ebb3                       ; bin: fe051ee3
 

From 7b36cb58a02a57285ccff61f91db9af7aaa06a32 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 28 Jun 2019 17:05:45 +0200
Subject: [PATCH 2486/3084] Use BB-like EBB in filetests/isa/x86/*.clif

---
 .../filetests/isa/x86/binary32-float.clif     | 23 +++++++
 .../filetests/filetests/isa/x86/binary32.clif | 50 ++++++++++++++++
 .../filetests/isa/x86/binary64-float.clif     | 23 +++++++
 .../filetests/filetests/isa/x86/binary64.clif | 60 +++++++++++++++++++
 .../filetests/isa/x86/prologue-epilogue.clif  | 15 +++--
 .../filetests/isa/x86/relax_branch.clif       |  9 +++
 .../isa/x86/shrink-multiple-uses.clif         |  3 +
 7 files changed, 177 insertions(+), 6 deletions(-)

diff --git a/cranelift/filetests/filetests/isa/x86/binary32-float.clif b/cranelift/filetests/filetests/isa/x86/binary32-float.clif
index 0b171388ae..8a4ae5fe7e 100644
--- a/cranelift/filetests/filetests/isa/x86/binary32-float.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary32-float.clif
@@ -481,21 +481,44 @@ ebb0(v0: f32 [%xmm0]):
 ebb1:
     ; asm: jnp ebb1
     brff ord v1, ebb1                                           ; bin: 7b fe
+    jump ebb2
+
+ebb2:
     ; asm: jp ebb1
     brff uno v1, ebb1                                           ; bin: 7a fc
+    jump ebb3
+
+ebb3:
     ; asm: jne ebb1
     brff one v1, ebb1                                           ; bin: 75 fa
+    jump ebb4
+
+ebb4:
     ; asm: je ebb1
     brff ueq v1, ebb1                                           ; bin: 74 f8
+    jump ebb5
+
+ebb5:
     ; asm: ja ebb1
     brff gt v1, ebb1                                            ; bin: 77 f6
+    jump ebb6
+
+ebb6:
     ; asm: jae ebb1
     brff ge v1, ebb1                                            ; bin: 73 f4
+    jump ebb7
+
+ebb7:
     ; asm: jb ebb1
     brff ult v1, ebb1                                           ; bin: 72 f2
+    jump ebb8
+
+ebb8:
     ; asm: jbe ebb1
     brff ule v1, ebb1                                           ; bin: 76 f0
+    jump ebb9
 
+ebb9:
     ; asm: jp .+4; ud2
     trapff ord v1, user0                                        ; bin: 7a 02 user0 0f 0b
     ; asm: jnp .+4; ud2
diff --git a/cranelift/filetests/filetests/isa/x86/binary32.clif b/cranelift/filetests/filetests/isa/x86/binary32.clif
index e5af412d51..602b2d4f31 100644
--- a/cranelift/filetests/filetests/isa/x86/binary32.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary32.clif
@@ -472,12 +472,21 @@ ebb0:
     ; asm: testl %ecx, %ecx
     ; asm: je ebb1
     brz v1, ebb1                                ; bin: 85 c9 74 0e
+    fallthrough ebb3
+
+ebb3:
     ; asm: testl %esi, %esi
     ; asm: je ebb1
     brz v2, ebb1                                ; bin: 85 f6 74 0a
+    fallthrough ebb4
+
+ebb4:
     ; asm: testl %ecx, %ecx
     ; asm: jne ebb1
     brnz v1, ebb1                               ; bin: 85 c9 75 06
+    fallthrough ebb5
+
+ebb5:
     ; asm: testl %esi, %esi
     ; asm: jne ebb1
     brnz v2, ebb1                               ; bin: 85 f6 75 02
@@ -506,16 +515,27 @@ ebb0:
     ; asm: testl $0xff, %edi
     ; asm: je ebb1
     brz v3, ebb1                                ; bin: f7 c7 000000ff 0f 84 00000015
+    fallthrough ebb2
+
+ebb2:
     ; asm: testb %bl, %bl
     ; asm: je ebb1
     brz v4, ebb1                                ; bin: 84 db 74 11
+    fallthrough ebb3
+
+ebb3:
     ; asm: testl $0xff, %edi
     ; asm: jne ebb1
     brnz v3, ebb1                               ; bin: f7 c7 000000ff 0f 85 00000005
+    fallthrough ebb4
+
+ebb4:
     ; asm: testb %bl, %bl
     ; asm: jne ebb1
     brnz v4, ebb1                               ; bin: 84 db 75 01
+    fallthrough ebb5
 
+ebb5:
     return
 
 ebb1:
@@ -537,24 +557,54 @@ ebb1:
 
     ; asm: je ebb1
     brif eq v11, ebb1                           ; bin: 74 fa
+    jump ebb2
+
+ebb2:
     ; asm: jne ebb1
     brif ne v11, ebb1                           ; bin: 75 f8
+    jump ebb3
+
+ebb3:
     ; asm: jl ebb1
     brif slt v11, ebb1                          ; bin: 7c f6
+    jump ebb4
+
+ebb4:
     ; asm: jge ebb1
     brif sge v11, ebb1                          ; bin: 7d f4
+    jump ebb5
+
+ebb5:
     ; asm: jg ebb1
     brif sgt v11, ebb1                          ; bin: 7f f2
+    jump ebb6
+
+ebb6:
     ; asm: jle ebb1
     brif sle v11, ebb1                          ; bin: 7e f0
+    jump ebb7
+
+ebb7:
     ; asm: jb ebb1
     brif ult v11, ebb1                          ; bin: 72 ee
+    jump ebb8
+
+ebb8:
     ; asm: jae ebb1
     brif uge v11, ebb1                          ; bin: 73 ec
+    jump ebb9
+
+ebb9:
     ; asm: ja ebb1
     brif ugt v11, ebb1                          ; bin: 77 ea
+    jump ebb10
+
+ebb10:
     ; asm: jbe ebb1
     brif ule v11, ebb1                          ; bin: 76 e8
+    jump ebb11
+
+ebb11:
 
     ; asm: sete %bl
     [-,%rbx]            v20 = trueif eq v11                           ; bin: 0f 94 c3
diff --git a/cranelift/filetests/filetests/isa/x86/binary64-float.clif b/cranelift/filetests/filetests/isa/x86/binary64-float.clif
index 582f4e9c45..59e024a042 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64-float.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64-float.clif
@@ -542,21 +542,44 @@ ebb0(v0: f32 [%xmm0]):
 ebb1:
     ; asm: jnp ebb1
     brff ord v1, ebb1                                           ; bin: 7b fe
+    jump ebb2
+
+ebb2:
     ; asm: jp ebb1
     brff uno v1, ebb1                                           ; bin: 7a fc
+    jump ebb3
+
+ebb3:
     ; asm: jne ebb1
     brff one v1, ebb1                                           ; bin: 75 fa
+    jump ebb4
+
+ebb4:
     ; asm: je ebb1
     brff ueq v1, ebb1                                           ; bin: 74 f8
+    jump ebb5
+
+ebb5:
     ; asm: ja ebb1
     brff gt v1, ebb1                                            ; bin: 77 f6
+    jump ebb6
+
+ebb6:
     ; asm: jae ebb1
     brff ge v1, ebb1                                            ; bin: 73 f4
+    jump ebb7
+
+ebb7:
     ; asm: jb ebb1
     brff ult v1, ebb1                                           ; bin: 72 f2
+    jump ebb8
+
+ebb8:
     ; asm: jbe ebb1
     brff ule v1, ebb1                                           ; bin: 76 f0
+    jump ebb9
 
+ebb9:
     ; asm: jp .+4; ud2
     trapff ord v1, user0                                        ; bin: 7a 02 user0 0f 0b
     ; asm: jnp .+4; ud2
diff --git a/cranelift/filetests/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
index ed1316ad24..a65b3d3d1d 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64.clif
@@ -690,18 +690,33 @@ ebb0:
     ; asm: testq %rcx, %rcx
     ; asm: je ebb1
     brz v1, ebb1                                ; bin: 48 85 c9 74 1b
+    fallthrough ebb3
+
+ebb3:
     ; asm: testq %rsi, %rsi
     ; asm: je ebb1
     brz v2, ebb1                                ; bin: 48 85 f6 74 16
+    fallthrough ebb4
+
+ebb4:
     ; asm: testq %r10, %r10
     ; asm: je ebb1
     brz v3, ebb1                                ; bin: 4d 85 d2 74 11
+    fallthrough ebb5
+
+ebb5:
     ; asm: testq %rcx, %rcx
     ; asm: jne ebb1
     brnz v1, ebb1                               ; bin: 48 85 c9 75 0c
+    fallthrough ebb6
+
+ebb6:
     ; asm: testq %rsi, %rsi
     ; asm: jne ebb1
     brnz v2, ebb1                               ; bin: 48 85 f6 75 07
+    fallthrough ebb7
+
+ebb7:
     ; asm: testq %r10, %r10
     ; asm: jne ebb1
     brnz v3, ebb1                               ; bin: 4d 85 d2 75 02
@@ -733,24 +748,54 @@ ebb1:
 
     ; asm: je ebb1
     brif eq v11, ebb1                           ; bin: 74 f8
+    jump ebb2
+
+ebb2:
     ; asm: jne ebb1
     brif ne v11, ebb1                           ; bin: 75 f6
+    jump ebb3
+
+ebb3:
     ; asm: jl ebb1
     brif slt v11, ebb1                          ; bin: 7c f4
+    jump ebb4
+
+ebb4:
     ; asm: jge ebb1
     brif sge v11, ebb1                          ; bin: 7d f2
+    jump ebb5
+
+ebb5:
     ; asm: jg ebb1
     brif sgt v11, ebb1                          ; bin: 7f f0
+    jump ebb6
+
+ebb6:
     ; asm: jle ebb1
     brif sle v11, ebb1                          ; bin: 7e ee
+    jump ebb7
+
+ebb7:
     ; asm: jb ebb1
     brif ult v11, ebb1                          ; bin: 72 ec
+    jump ebb8
+
+ebb8:
     ; asm: jae ebb1
     brif uge v11, ebb1                          ; bin: 73 ea
+    jump ebb9
+
+ebb9:
     ; asm: ja ebb1
     brif ugt v11, ebb1                          ; bin: 77 e8
+    jump ebb10
+
+ebb10:
     ; asm: jbe ebb1
     brif ule v11, ebb1                          ; bin: 76 e6
+    jump ebb11
+
+ebb11:
 
     ; asm: sete %bl
     [-,%rbx]            v20 = trueif eq v11                           ; bin: 0f 94 c3
@@ -1248,18 +1293,33 @@ ebb0:
     ; asm: testl %ecx, %ecx
     ; asm: je ebb1x
     brz v1, ebb1                                ; bin: 85 c9 74 18
+    fallthrough ebb3
+
+ebb3:
     ; asm: testl %esi, %esi
     ; asm: je ebb1x
     brz v2, ebb1                                ; bin: 85 f6 74 14
+    fallthrough ebb4
+
+ebb4:
     ; asm: testl %r10d, %r10d
     ; asm: je ebb1x
     brz v3, ebb1                                ; bin: 45 85 d2 74 0f
+    fallthrough ebb5
+
+ebb5:
     ; asm: testl %ecx, %ecx
     ; asm: jne ebb1x
     brnz v1, ebb1                               ; bin: 85 c9 75 0b
+    fallthrough ebb6
+
+ebb6:
     ; asm: testl %esi, %esi
     ; asm: jne ebb1x
     brnz v2, ebb1                               ; bin: 85 f6 75 07
+    fallthrough ebb7
+
+ebb7:
     ; asm: testl %r10d, %r10d
     ; asm: jne ebb1x
     brnz v3, ebb1                               ; bin: 45 85 d2 75 02
diff --git a/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif b/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
index f408d1a193..1c715f2a21 100644
--- a/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
+++ b/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
@@ -213,20 +213,23 @@ function %divert(i32) -> i32 system_v {
 ebb0(v0: i32):
     v2 = iconst.i32 0
     v3 = iconst.i32 1
-    jump ebb3(v0, v3, v2)
+    jump ebb1(v0, v3, v2)
 
-ebb3(v4: i32, v5: i32, v6: i32):
-    brz v4, ebb4
+ebb1(v4: i32, v5: i32, v6: i32):
+    brz v4, ebb3
+    fallthrough ebb2
+
+ebb2:
     v7 = iadd v5, v6
     v8 = iadd_imm v4, -1
-    jump ebb3(v8, v7, v5)
+    jump ebb1(v8, v7, v5)
 
-ebb4:
+ebb3:
     return v5
 }
 
 ; check: function %divert
-; check: regmove v5, %rcx -> %rbx
+; check: regmove.i32 v5, %rcx -> %rbx
 ; check: [Op1popq#58,%rbx]                   v15 = x86_pop.i64
 
 ; Stack limit checking
diff --git a/cranelift/filetests/filetests/isa/x86/relax_branch.clif b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
index c735224c0a..373e2dda6d 100644
--- a/cranelift/filetests/filetests/isa/x86/relax_branch.clif
+++ b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
@@ -55,6 +55,9 @@ function u0:2691(i32 [%rdi], i32 [%rsi], i64 vmctx [%r14]) -> i64 uext [%rax] ba
 @004c [Op1r_id#4081,%rcx]           v37 = band_imm v35, 255
 [Op1rcmp_ib#7083,%rflags]           v97 = ifcmp_imm v37, 26
 @0050 [Op1brib#70]                  brif sge v97, ebb6
+@0050 [-]                           fallthrough ebb10
+
+                                ebb10:
 [Op1umr#89,%rcx]                    v101 = copy v18
 @0054 [Op1jmpb#eb]                  jump ebb5(v18, v101)
 
@@ -82,6 +85,9 @@ function u0:2691(i32 [%rdi], i32 [%rsi], i64 vmctx [%r14]) -> i64 uext [%rax] ba
 [Op1rcmp_ib#7083,%rflags]           v98 = ifcmp_imm v63, 26
 @0084 [RexOp1rmov#89]               regmove v47, %rdi -> %rbx
 @0084 [Op1brib#70]                  brif sge v98, ebb8
+@0084 [-]                           fallthrough ebb11
+
+                                ebb11:
 [RexOp1umr#89,%rdx]                 v103 = copy.i32 v29
 @0088 [Op1jmpb#eb]                  jump ebb7(v29, v10, v21, v103)
 
@@ -99,6 +105,9 @@ function u0:2691(i32 [%rdi], i32 [%rsi], i64 vmctx [%r14]) -> i64 uext [%rax] ba
 @0098 [Op2urm_noflags_abcd#4b6,%rbx] v77 = bint.i32 v76
 @0099 [RexOp1rr#21,%r10]            v78 = band.i32 v50, v77
 @009a [RexOp1tjccb#74]              brz v78, ebb9
+@009a [-]                           fallthrough ebb12
+
+                                ebb12:
 [RexOp1umr#89,%rcx]                 v99 = copy v81
 [Op1umr#89,%rdx]                    v100 = copy v79
 @00a4 [RexOp1rmov#89]               regmove v100, %rdx -> %rdi
diff --git a/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif b/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
index d20cc78066..deec4ff505 100644
--- a/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
+++ b/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
@@ -8,6 +8,9 @@ ebb0(v0: i32 [%rdi]):
 [Op2seti_abcd#490,%rax]             v1 = trueif eq v3
 [RexOp2urm_noflags#4b6,%rax]        v2 = bint.i32 v1
 [Op1brib#70]                        brif eq v3, ebb1
+[-]                                 fallthrough ebb2
+
+ebb2:
 [Op1ret#c3]                         return v2
 
 ebb1:

From 4316917cb928386ec525e112d501ecb143489166 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 28 Jun 2019 17:23:12 +0200
Subject: [PATCH 2487/3084] Use BB-like EBB in filetests/parser/*.clif

---
 .../filetests/filetests/parser/flags.clif     | 20 ++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/cranelift/filetests/filetests/parser/flags.clif b/cranelift/filetests/filetests/parser/flags.clif
index d3c096f1c5..aac8017e85 100644
--- a/cranelift/filetests/filetests/parser/flags.clif
+++ b/cranelift/filetests/filetests/parser/flags.clif
@@ -5,11 +5,20 @@ function %iflags(i32) {
 ebb200(v0: i32):
     v1 = ifcmp_imm v0, 17
     brif eq v1, ebb201
+    jump ebb400
+
+ebb400:
     brif ugt v1, ebb202
+    jump ebb401
+
+ebb401:
     v2 = iconst.i32 34
     v3 = ifcmp v0, v2
     v4 = trueif eq v3
     brnz v4, ebb202
+    jump ebb402
+
+ebb402:
     return
 
 ebb201:
@@ -21,7 +30,7 @@ ebb202:
 ; check: v1 = ifcmp_imm v0, 17
 ; check: brif eq v1, ebb201
 ; check: brif ugt v1, ebb202
-; check: v3 = ifcmp v0, v2
+; check: v3 = ifcmp.i32 v0, v2
 ; check: v4 = trueif eq v3
 
 function %fflags(f32) {
@@ -29,9 +38,18 @@ ebb200(v0: f32):
     v1 = f32const 0x34.0p0
     v2 = ffcmp v0, v1
     brff eq v2, ebb201
+    jump ebb400
+
+ebb400:
     brff ord v2, ebb202
+    jump ebb401
+
+ebb401:
     v3 = trueff gt v2
     brnz v3, ebb202
+    jump ebb402
+
+ebb402:
     return
 
 ebb201:

From 3970fa2752aa89fed7155a05990df3244505ba30 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 28 Jun 2019 17:31:08 +0200
Subject: [PATCH 2488/3084] Use BB-like EBB in filetests/postopt/*.clif

---
 .../filetests/filetests/postopt/basic.clif    | 24 +++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/cranelift/filetests/filetests/postopt/basic.clif b/cranelift/filetests/filetests/postopt/basic.clif
index 5b65407004..962434bfac 100644
--- a/cranelift/filetests/filetests/postopt/basic.clif
+++ b/cranelift/filetests/filetests/postopt/basic.clif
@@ -7,6 +7,9 @@ function %br_icmp(i32, i32) -> i32 {
 ebb0(v0: i32, v1: i32):
 [Op1icscc#39,%rdx]  v2 = icmp slt v0, v1
 [Op1t8jccd_long#85] brnz v2, ebb1
+[-]                 fallthrough ebb2
+
+ebb2:
 [Op1ret#c3]         return v1
 
 ebb1:
@@ -18,6 +21,9 @@ ebb1:
 ; nextln:    v9 = ifcmp v0, v1
 ; nextln:    v2 = trueif slt v9
 ; nextln:    brif slt v9, ebb1
+; nextln:    fallthrough ebb2
+; nextln: 
+; nextln: ebb2:
 ; nextln:    return v1
 ; nextln: 
 ; nextln: ebb1:
@@ -31,6 +37,9 @@ function %br_icmp_inverse(i32, i32) -> i32 {
 ebb0(v0: i32, v1: i32):
 [Op1icscc#39,%rdx]  v2 = icmp slt v0, v1
 [Op1t8jccd_long#84] brz v2, ebb1
+[-]                 fallthrough ebb2
+
+ebb2:
 [Op1ret#c3]         return v1
 
 ebb1:
@@ -42,6 +51,9 @@ ebb1:
 ; nextln:    v9 = ifcmp v0, v1
 ; nextln:    v2 = trueif slt v9
 ; nextln:    brif sge v9, ebb1
+; nextln:    fallthrough ebb2
+; nextln: 
+; nextln: ebb2:
 ; nextln:    return v1
 ; nextln: 
 ; nextln: ebb1:
@@ -55,6 +67,9 @@ function %br_icmp_imm(i32, i32) -> i32 {
 ebb0(v0: i32, v1: i32):
 [Op1icscc_ib#7083]  v2 = icmp_imm slt v0, 2
 [Op1t8jccd_long#84] brz v2, ebb1
+[-]                 fallthrough ebb2
+
+ebb2:
 [Op1ret#c3]         return v1
 
 ebb1:
@@ -66,6 +81,9 @@ ebb1:
 ; nextln:    v9 = ifcmp_imm v0, 2
 ; nextln:    v2 = trueif slt v9
 ; nextln:    brif sge v9, ebb1
+; nextln:    fallthrough ebb2
+; nextln: 
+; nextln: ebb2:
 ; nextln:    return v1
 ; nextln: 
 ; nextln: ebb1:
@@ -79,6 +97,9 @@ function %br_fcmp(f32, f32) -> f32 {
 ebb0(v0: f32, v1: f32):
 [Op2fcscc#42e,%rdx] v2 = fcmp gt v0, v1
 [Op1t8jccd_long#84] brz v2, ebb1
+[-]                 fallthrough ebb2
+
+ebb2:
 [Op1ret#c3]         return v1
 
 ebb1:
@@ -91,6 +112,9 @@ ebb1:
 ; nextln:    v19 = ffcmp v0, v1
 ; nextln:    v2 = trueff gt v19
 ; nextln:    brff ule v19, ebb1
+; nextln:    fallthrough ebb2
+; nextln: 
+; nextln: ebb2:
 ; nextln:    return v1
 ; nextln: 
 ; nextln: ebb1:

From 560619f7498eb030a21804006a7148d1fe88a5ef Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 28 Jun 2019 17:39:59 +0200
Subject: [PATCH 2489/3084] Use BB-like EBB in filetests/cfg/*.clif

---
 cranelift/filetests/filetests/cfg/loop.clif | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/cranelift/filetests/filetests/cfg/loop.clif b/cranelift/filetests/filetests/cfg/loop.clif
index 728240940f..42e9cb483d 100644
--- a/cranelift/filetests/filetests/cfg/loop.clif
+++ b/cranelift/filetests/filetests/cfg/loop.clif
@@ -5,13 +5,16 @@ test verifier
 function %nonsense(i32, i32) -> f32 {
 ; check: digraph "%nonsense" {
 ; regex: I=\binst\d+\b
-; check: label="{ebb0 | <$(BRZ=$I)>brz ebb2 | <$(JUMP=$I)>jump ebb1}"]
+; check: label="{ebb0 | <$(BRZ=$I)>brz ebb2 | <$(JUMP=$I)>jump ebb3}"]
 
 ebb0(v1: i32, v2: i32):
     v3 = f64const 0x0.0
     brz v2, ebb2            ; unordered: ebb0:$BRZ -> ebb2
+    jump ebb3               ; unordered: ebb0:$JUMP -> ebb3
+
+ebb3:
     v4 = iconst.i32 0
-    jump ebb1(v4)           ; unordered: ebb0:$JUMP -> ebb1
+    jump ebb1(v4)           ; unordered: ebb3:inst4 -> ebb1
 
 ebb1(v5: i32):
     v6 = imul_imm v5, 4
@@ -22,7 +25,10 @@ ebb1(v5: i32):
     v11 = fadd v9, v10
     v12 = iadd_imm v5, 1
     v13 = icmp ult v12, v2
-    brnz v13, ebb1(v12)     ; unordered: ebb1:inst12 -> ebb1
+    brnz v13, ebb1(v12)     ; unordered: ebb1:inst13 -> ebb1
+    jump ebb4               ; unordered: ebb1:inst14 -> ebb4
+
+ebb4:
     v14 = f64const 0.0
     v15 = f64const 0.0
     v16 = fdiv v14, v15

From 3c6ca9049e090641fc321409e72a8967ffa5a5d3 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 28 Jun 2019 17:47:56 +0200
Subject: [PATCH 2490/3084] Use BB-like EBB in filetests/simple_gvn/*.clif

---
 cranelift/filetests/filetests/simple_gvn/basic.clif  | 3 +++
 cranelift/filetests/filetests/simple_gvn/scopes.clif | 6 ++++++
 2 files changed, 9 insertions(+)

diff --git a/cranelift/filetests/filetests/simple_gvn/basic.clif b/cranelift/filetests/filetests/simple_gvn/basic.clif
index df8fd495c9..72155a324c 100644
--- a/cranelift/filetests/filetests/simple_gvn/basic.clif
+++ b/cranelift/filetests/filetests/simple_gvn/basic.clif
@@ -24,6 +24,9 @@ function %redundancies_on_some_paths(i32, i32, i32) -> i32 {
 ebb0(v0: i32, v1: i32, v2: i32):
     v3 = iadd v0, v1
     brz v3, ebb1
+    jump ebb3
+
+ebb3:
     v4 = iadd v0, v1
     jump ebb2(v4)
 ; check: jump ebb2(v3)
diff --git a/cranelift/filetests/filetests/simple_gvn/scopes.clif b/cranelift/filetests/filetests/simple_gvn/scopes.clif
index 85ea5583c2..bf4e7fac94 100644
--- a/cranelift/filetests/filetests/simple_gvn/scopes.clif
+++ b/cranelift/filetests/filetests/simple_gvn/scopes.clif
@@ -5,6 +5,9 @@ ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32):
     v5 = iconst.i32 16
     ; check: v5 = iconst.i32 16
     brz v0, ebb1
+    jump ebb5
+
+ebb5:
     v6 = iconst.i32 17
     ; check: v6 = iconst.i32 17
     v7 = iconst.i32 16
@@ -30,6 +33,9 @@ ebb2:
     v14 = iconst.i32 16
     ; not: v14 = iconst.i32 16
     brz v1, ebb3
+    jump ebb6
+
+ebb6:
     v15 = iconst.i32 20
     ; check: v15 = iconst.i32 20
     v16 = iconst.i32 19

From 8c65ec0dded33431a6162076e3cb4a3ca9fbf92b Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 28 Jun 2019 18:12:48 +0200
Subject: [PATCH 2491/3084] Use BB-like EBB in filetests/regress/*.clif

---
 .../filetests/filetests/regress/allow-relaxation-shrink.clif   | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/cranelift/filetests/filetests/regress/allow-relaxation-shrink.clif b/cranelift/filetests/filetests/regress/allow-relaxation-shrink.clif
index 21c0f847b5..3dd8250656 100644
--- a/cranelift/filetests/filetests/regress/allow-relaxation-shrink.clif
+++ b/cranelift/filetests/filetests/regress/allow-relaxation-shrink.clif
@@ -30,6 +30,9 @@ ebb4:
     v19 = bint.i8 v18
     v20 = uextend.i32 v19
     brz v20, ebb6
+    jump ebb7
+
+ebb7:
     trap user0
 
 ebb5:

From a1a4b9bfb1c548de93b8878984227d6c46349f9a Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 28 Jun 2019 18:13:09 +0200
Subject: [PATCH 2492/3084] Use BB-like EBB in filetests/verifier/*.clif

---
 .../filetests/filetests/verifier/bad_layout.clif     | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/cranelift/filetests/filetests/verifier/bad_layout.clif b/cranelift/filetests/filetests/verifier/bad_layout.clif
index fd597359be..034dd7843f 100644
--- a/cranelift/filetests/filetests/verifier/bad_layout.clif
+++ b/cranelift/filetests/filetests/verifier/bad_layout.clif
@@ -1,11 +1,13 @@
 test verifier
 
-function %test(i32) {
+function %test_1(i32) {
     ebb0(v0: i32):
-        jump ebb1       ; error: terminator
+        return          ; error: terminator
         return
-    ebb1:
-        jump ebb2
+}
+function %test_2(i32) {
+    ebb0(v0: i32):
+        jump ebb2       ; error: a terminator instruction was encountered before the end of ebb0
         brz v0, ebb3
     ebb2:
         jump ebb3
@@ -13,7 +15,7 @@ function %test(i32) {
         return
 }
 
-function %test(i32) {    ; Ok
+function %test_3(i32) { ; Ok
     ebb0(v0: i32):
         return
 }

From 8bfdfbe68b3a1a50ebc0c0da52414f016f5cf7eb Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 28 Jun 2019 19:09:41 +0200
Subject: [PATCH 2493/3084] Use BB-like EBB in filetests/regalloc/*.clif

---
 .../filetests/regalloc/coalesce.clif          |  21 ++
 .../filetests/regalloc/coalescing-207.clif    | 264 ++++++++++++++++++
 .../filetests/regalloc/coalescing-216.clif    |  15 +
 .../filetests/regalloc/coloring-227.clif      |  15 +
 .../filetests/regalloc/ghost-param.clif       |   3 +
 .../regalloc/global-constraints.clif          |   3 +
 .../filetests/filetests/regalloc/iterate.clif |  24 ++
 .../filetests/regalloc/reload-208.clif        |  12 +
 .../filetests/filetests/regalloc/spill.clif   |   3 +
 9 files changed, 360 insertions(+)

diff --git a/cranelift/filetests/filetests/regalloc/coalesce.clif b/cranelift/filetests/filetests/regalloc/coalesce.clif
index 0085ce0fd8..f667d7440e 100644
--- a/cranelift/filetests/filetests/regalloc/coalesce.clif
+++ b/cranelift/filetests/filetests/regalloc/coalesce.clif
@@ -12,6 +12,9 @@ ebb0(v0: i32):
     ; not: copy
     ; v0 is used by the branch and passed as an arg - that's no conflict.
     brnz v0, ebb1(v0)
+    fallthrough ebb2
+
+ebb2:
     ; v0 is live across the branch above. That's no conflict.
     v1 = iadd_imm v0, 7
     jump ebb1(v1)
@@ -26,6 +29,9 @@ ebb0(v0: i32):
     ; check: $(cp1=$V) = copy v0
     ; nextln: brnz v0, ebb1($cp1)
     brnz v0, ebb1(v0)
+    fallthrough ebb2
+
+ebb2:
     ; not: copy
     v1 = iadd_imm v0, 7
     jump ebb1(v1)
@@ -42,6 +48,9 @@ ebb0(v0: i32):
     ; check: $(cp1=$V) = copy v0
     ; nextln: brnz v0, ebb1($cp1, v0)
     brnz v0, ebb1(v0, v0)
+    fallthrough ebb2
+
+ebb2:
     v1 = iadd_imm v0, 7
     v2 = iadd_imm v1, 56
     jump ebb1(v1, v2)
@@ -59,6 +68,9 @@ ebb0(v0: i32):
     ; not: copy
     ; check: brnz v0, ebb1($cp0)
     brnz v0, ebb1(v0)
+    fallthrough ebb2
+
+ebb2:
     v1 = iadd_imm v0, 7
     ; v1 and v0 interfere here:
     v2 = iadd_imm v0, 8
@@ -89,6 +101,9 @@ ebb1(v10: i32, v11: i32):
     ; not: copy
     ; check: brnz v13, ebb1($nv11b, v12)
     brnz v13, ebb1(v11, v12)
+    fallthrough ebb2
+
+ebb2:
     return v12
 }
 
@@ -116,7 +131,13 @@ ebb0(v0: i32):
 
 ebb2(v3: i32, v4: i32):
     brnz v3, ebb2(v3, v4)
+    fallthrough ebb3
+
+ebb3:
     v5 = iconst.i32 1
     brnz v3, ebb2(v2, v5)
+    fallthrough ebb4
+
+ebb4:
     return
 }
diff --git a/cranelift/filetests/filetests/regalloc/coalescing-207.clif b/cranelift/filetests/filetests/regalloc/coalescing-207.clif
index 30b33ee44a..3f01f6fcc5 100644
--- a/cranelift/filetests/filetests/regalloc/coalescing-207.clif
+++ b/cranelift/filetests/filetests/regalloc/coalescing-207.clif
@@ -22,6 +22,9 @@ ebb0(v0: i64, v1: i32, v2: i32):
     v6 = iconst.i32 0x4ffe
     v7 = icmp uge v5, v6
     brz v7, ebb1
+    fallthrough ebb100
+
+ebb100:
     trap heap_oob
 
 ebb1:
@@ -35,6 +38,9 @@ ebb1:
     v15 = iconst.i32 0x4ffe
     v16 = icmp.i32 uge v4, v15
     brz v16, ebb2
+    fallthrough ebb101
+
+ebb101:
     trap heap_oob
 
 ebb2:
@@ -46,6 +52,9 @@ ebb2:
     v21 = iconst.i32 0x4ffe
     v22 = icmp.i32 uge v2, v21
     brz v22, ebb3
+    fallthrough ebb102
+
+ebb102:
     trap heap_oob
 
 ebb3:
@@ -60,15 +69,24 @@ ebb3:
     v31 = icmp eq v29, v30
     v32 = bint.i32 v31
     brnz v32, ebb90(v14, v1)
+    fallthrough ebb103
+
+ebb103:
     v33 = call fn0(v0, v1, v27)
     v34 = iconst.i32 0
     v35 = iconst.i32 0
     v36 = icmp eq v33, v35
     v37 = bint.i32 v36
     brnz v37, ebb90(v14, v34)
+    fallthrough ebb104
+
+ebb104:
     v38 = iconst.i32 0x4ffe
     v39 = icmp.i32 uge v2, v38
     brz v39, ebb4
+    fallthrough ebb105
+
+ebb105:
     trap heap_oob
 
 ebb4:
@@ -81,9 +99,15 @@ ebb4:
     v46 = icmp eq v44, v45
     v47 = bint.i32 v46
     brnz v47, ebb56(v33, v14)
+    fallthrough ebb106
+
+ebb106:
     v48 = iconst.i32 0x4ffe
     v49 = icmp.i32 uge v33, v48
     brz v49, ebb5
+    fallthrough ebb107
+
+ebb107:
     trap heap_oob
 
 ebb5:
@@ -96,9 +120,15 @@ ebb5:
     v56 = icmp eq v54, v55
     v57 = bint.i32 v56
     brnz v57, ebb90(v14, v34)
+    fallthrough ebb108
+
+ebb108:
     v58 = iconst.i32 0x4ffe
     v59 = icmp.i32 uge v2, v58
     brz v59, ebb6
+    fallthrough ebb109
+
+ebb109:
     trap heap_oob
 
 ebb6:
@@ -111,9 +141,15 @@ ebb6:
     v66 = icmp eq v64, v65
     v67 = bint.i32 v66
     brnz v67, ebb42
+    fallthrough ebb110
+
+ebb110:
     v68 = iconst.i32 0x4ffe
     v69 = icmp.i32 uge v33, v68
     brz v69, ebb7
+    fallthrough ebb111
+
+ebb111:
     trap heap_oob
 
 ebb7:
@@ -126,9 +162,15 @@ ebb7:
     v76 = icmp eq v74, v75
     v77 = bint.i32 v76
     brnz v77, ebb90(v14, v34)
+    fallthrough ebb112
+
+ebb112:
     v78 = iconst.i32 0x4ffe
     v79 = icmp.i32 uge v2, v78
     brz v79, ebb8
+    fallthrough ebb113
+
+ebb113:
     trap heap_oob
 
 ebb8:
@@ -141,9 +183,15 @@ ebb8:
     v86 = icmp eq v84, v85
     v87 = bint.i32 v86
     brnz v87, ebb46
+    fallthrough ebb114
+
+ebb114:
     v88 = iconst.i32 0x4ffe
     v89 = icmp.i32 uge v33, v88
     brz v89, ebb9
+    fallthrough ebb115
+
+ebb115:
     trap heap_oob
 
 ebb9:
@@ -156,9 +204,15 @@ ebb9:
     v96 = icmp eq v94, v95
     v97 = bint.i32 v96
     brnz v97, ebb90(v14, v34)
+    fallthrough ebb116
+
+ebb116:
     v98 = iconst.i32 0x4ffe
     v99 = icmp.i32 uge v2, v98
     brz v99, ebb10
+    fallthrough ebb117
+
+ebb117:
     trap heap_oob
 
 ebb10:
@@ -171,6 +225,9 @@ ebb10:
     v106 = icmp eq v104, v105
     v107 = bint.i32 v106
     brnz v107, ebb54
+    fallthrough ebb118
+
+ebb118:
     v108 = iconst.i32 1
     v109 = iadd.i32 v2, v108
     v110 = iconst.i32 1048
@@ -179,6 +236,9 @@ ebb10:
     v113 = iconst.i32 0x4ffe
     v114 = icmp uge v111, v113
     brz v114, ebb11
+    fallthrough ebb119
+
+ebb119:
     trap heap_oob
 
 ebb11:
@@ -193,6 +253,9 @@ ebb11:
     v122 = iconst.i32 0x4ffe
     v123 = icmp uge v120, v122
     brz v123, ebb12
+    fallthrough ebb120
+
+ebb120:
     trap heap_oob
 
 ebb12:
@@ -205,6 +268,9 @@ ebb12:
     v129 = iconst.i32 0x4ffe
     v130 = icmp.i32 uge v14, v129
     brz v130, ebb13
+    fallthrough ebb121
+
+ebb121:
     trap heap_oob
 
 ebb13:
@@ -217,6 +283,9 @@ ebb13:
     v136 = iconst.i32 0x4ffe
     v137 = icmp.i32 uge v14, v136
     brz v137, ebb14
+    fallthrough ebb122
+
+ebb122:
     trap heap_oob
 
 ebb14:
@@ -235,6 +304,9 @@ ebb15(v143: i32, v144: i32):
     v148 = iconst.i32 0x4ffe
     v149 = icmp uge v147, v148
     brz v149, ebb16
+    fallthrough ebb123
+
+ebb123:
     trap heap_oob
 
 ebb16:
@@ -247,6 +319,9 @@ ebb16:
     v156 = icmp eq v154, v155
     v157 = bint.i32 v156
     brnz v157, ebb89(v14)
+    fallthrough ebb124
+
+ebb124:
     v158 = iconst.i32 255
     v159 = band.i32 v144, v158
     v160 = iconst.i32 2
@@ -257,6 +332,9 @@ ebb16:
     v165 = iconst.i32 0x4ffe
     v166 = icmp uge v162, v165
     brz v166, ebb17
+    fallthrough ebb125
+
+ebb125:
     trap heap_oob
 
 ebb17:
@@ -275,6 +353,9 @@ ebb17:
     v178 = iconst.i32 0x4ffe
     v179 = icmp uge v177, v178
     brz v179, ebb18
+    fallthrough ebb126
+
+ebb126:
     trap heap_oob
 
 ebb18:
@@ -291,6 +372,9 @@ ebb18:
     v190 = iconst.i32 0x4ffe
     v191 = icmp.i32 uge v177, v190
     brz v191, ebb19
+    fallthrough ebb127
+
+ebb127:
     trap heap_oob
 
 ebb19:
@@ -307,6 +391,9 @@ ebb19:
     v201 = iconst.i32 0x4ffe
     v202 = icmp uge v200, v201
     brz v202, ebb20
+    fallthrough ebb128
+
+ebb128:
     trap heap_oob
 
 ebb20:
@@ -329,6 +416,9 @@ ebb21:
     v215 = icmp ult v213, v214
     v216 = bint.i32 v215
     brnz v216, ebb38(v2, v211, v209, v210, v208, v198, v213, v33, v14)
+    fallthrough ebb129
+
+ebb129:
     v217 = iconst.i32 -1
     v218 = iconst.i32 0
     v219 = iconst.i32 1
@@ -344,6 +434,9 @@ ebb22(v223: i32, v224: i32, v225: i32, v226: i32, v227: i32, v228: i32, v229: i3
     v237 = iconst.i32 0x4ffe
     v238 = icmp uge v236, v237
     brz v238, ebb23
+    fallthrough ebb130
+
+ebb130:
     trap heap_oob
 
 ebb23:
@@ -357,9 +450,15 @@ ebb23:
     v246 = icmp ne v243, v245
     v247 = bint.i32 v246
     brnz v247, ebb24
+    fallthrough ebb131
+
+ebb131:
     v248 = icmp.i32 ne v224, v226
     v249 = bint.i32 v248
     brnz v249, ebb25
+    fallthrough ebb132
+
+ebb132:
     v250 = iadd.i32 v227, v226
     v251 = iconst.i32 1
     jump ebb27(v251, v250, v223, v226)
@@ -368,6 +467,9 @@ ebb24:
     v252 = icmp.i32 ule v243, v245
     v253 = bint.i32 v252
     brnz v253, ebb26
+    fallthrough ebb133
+
+ebb133:
     v254 = isub.i32 v234, v223
     v255 = iconst.i32 1
     jump ebb27(v255, v234, v223, v254)
@@ -391,10 +493,16 @@ ebb27(v264: i32, v265: i32, v266: i32, v267: i32):
     v269 = icmp uge v268, v229
     v270 = bint.i32 v269
     brnz v270, ebb29
+    fallthrough ebb134
+
+ebb134:
     v271 = iadd.i32 v2, v268
     v272 = iconst.i32 0x4ffe
     v273 = icmp uge v271, v272
     brz v273, ebb28
+    fallthrough ebb135
+
+ebb135:
     trap heap_oob
 
 ebb28:
@@ -424,6 +532,9 @@ ebb31(v285: i32, v286: i32, v287: i32, v288: i32, v289: i32, v290: i32, v291: i3
     v300 = iconst.i32 0x4ffe
     v301 = icmp uge v299, v300
     brz v301, ebb32
+    fallthrough ebb136
+
+ebb136:
     trap heap_oob
 
 ebb32:
@@ -437,9 +548,15 @@ ebb32:
     v309 = icmp ne v306, v308
     v310 = bint.i32 v309
     brnz v310, ebb33
+    fallthrough ebb137
+
+ebb137:
     v311 = icmp.i32 ne v286, v288
     v312 = bint.i32 v311
     brnz v312, ebb34
+    fallthrough ebb138
+
+ebb138:
     v313 = iadd.i32 v289, v288
     v314 = iconst.i32 1
     jump ebb36(v314, v313, v285, v288)
@@ -448,6 +565,9 @@ ebb33:
     v315 = icmp.i32 uge v306, v308
     v316 = bint.i32 v315
     brnz v316, ebb35
+    fallthrough ebb139
+
+ebb139:
     v317 = isub.i32 v297, v285
     v318 = iconst.i32 1
     jump ebb36(v318, v297, v285, v317)
@@ -471,10 +591,16 @@ ebb36(v327: i32, v328: i32, v329: i32, v330: i32):
     v332 = icmp uge v331, v291
     v333 = bint.i32 v332
     brnz v333, ebb38(v2, v330, v292, v329, v293, v294, v291, v295, v296)
+    fallthrough ebb140
+
+ebb140:
     v334 = iadd.i32 v2, v331
     v335 = iconst.i32 0x4ffe
     v336 = icmp uge v334, v335
     brz v336, ebb37
+    fallthrough ebb141
+
+ebb141:
     trap heap_oob
 
 ebb37:
@@ -494,12 +620,18 @@ ebb38(v343: i32, v344: i32, v345: i32, v346: i32, v347: i32, v348: i32, v349: i3
     v356 = icmp ugt v353, v355
     v357 = bint.i32 v356
     brnz v357, ebb39(v344)
+    fallthrough ebb142
+
+ebb142:
     v358 = copy v345
     jump ebb39(v358)
 
 ebb39(v359: i32):
     v360 = iadd.i32 v343, v359
     brnz.i32 v357, ebb40(v346)
+    fallthrough ebb143
+
+ebb143:
     v361 = copy.i32 v347
     jump ebb40(v361)
 
@@ -511,6 +643,9 @@ ebb40(v362: i32):
     v367 = icmp eq v365, v366
     v368 = bint.i32 v367
     brnz v368, ebb63
+    fallthrough ebb144
+
+ebb144:
     v369 = iconst.i32 1
     v370 = iadd v362, v369
     v371 = isub.i32 v348, v370
@@ -520,6 +655,9 @@ ebb40(v362: i32):
     v375 = bint.i32 v374
     v376 = copy v362
     brnz v375, ebb41(v376)
+    fallthrough ebb145
+
+ebb145:
     v377 = copy v373
     jump ebb41(v377)
 
@@ -536,6 +674,9 @@ ebb42:
     v385 = iconst.i32 0x4ffe
     v386 = icmp.i32 uge v33, v385
     brz v386, ebb43
+    fallthrough ebb146
+
+ebb146:
     trap heap_oob
 
 ebb43:
@@ -557,6 +698,9 @@ ebb44(v392: i32, v393: i32, v394: i32):
     v402 = icmp eq v401, v384
     v403 = bint.i32 v402
     brnz v403, ebb56(v394, v14)
+    fallthrough ebb147
+
+ebb147:
     v404 = iconst.i32 2
     v405 = iadd v394, v404
     v406 = iconst.i32 1
@@ -564,6 +708,9 @@ ebb44(v392: i32, v393: i32, v394: i32):
     v408 = iconst.i32 0x4ffe
     v409 = icmp uge v405, v408
     brz v409, ebb45
+    fallthrough ebb148
+
+ebb148:
     trap heap_oob
 
 ebb45:
@@ -584,6 +731,9 @@ ebb46:
     v420 = iconst.i32 0x4ffe
     v421 = icmp.i32 uge v33, v420
     brz v421, ebb47
+    fallthrough ebb149
+
+ebb149:
     trap heap_oob
 
 ebb47:
@@ -616,6 +766,9 @@ ebb48(v440: i32, v441: i32):
     v446 = iconst.i32 0x4ffe
     v447 = icmp uge v445, v446
     brz v447, ebb49
+    fallthrough ebb150
+
+ebb150:
     trap heap_oob
 
 ebb49:
@@ -628,6 +781,9 @@ ebb49:
     v454 = icmp eq v452, v453
     v455 = bint.i32 v454
     brnz v455, ebb51(v14)
+    fallthrough ebb151
+
+ebb151:
     v456 = bor.i32 v441, v452
     v457 = iconst.i32 8
     v458 = ishl v456, v457
@@ -647,6 +803,9 @@ ebb51(v462: i32):
     v466 = iconst.i32 0x4ffe
     v467 = icmp uge v463, v466
     brz v467, ebb52
+    fallthrough ebb152
+
+ebb152:
     trap heap_oob
 
 ebb52:
@@ -657,6 +816,9 @@ ebb52:
     store.i32 v465, v471+4
     v472 = iconst.i32 0
     brnz.i32 v452, ebb53(v443)
+    fallthrough ebb153
+
+ebb153:
     v473 = copy v472
     jump ebb53(v473)
 
@@ -673,6 +835,9 @@ ebb54:
     v481 = iconst.i32 0x4ffe
     v482 = icmp.i32 uge v33, v481
     brz v482, ebb55
+    fallthrough ebb154
+
+ebb154:
     trap heap_oob
 
 ebb55:
@@ -713,6 +878,9 @@ ebb58(v505: i32, v506: i32):
     v511 = iconst.i32 0x4ffe
     v512 = icmp uge v508, v511
     brz v512, ebb59
+    fallthrough ebb155
+
+ebb155:
     trap heap_oob
 
 ebb59:
@@ -725,6 +893,9 @@ ebb59:
     v519 = icmp eq v517, v518
     v520 = bint.i32 v519
     brnz v520, ebb61(v14)
+    fallthrough ebb156
+
+ebb156:
     v521 = iconst.i32 8
     v522 = ishl.i32 v506, v521
     v523 = bor v522, v517
@@ -739,6 +910,9 @@ ebb60:
 ebb61(v526: i32):
     v527 = iconst.i32 0
     brnz.i32 v517, ebb62(v510)
+    fallthrough ebb157
+
+ebb157:
     v528 = copy v527
     jump ebb62(v528)
 
@@ -772,9 +946,15 @@ ebb65(v547: i32, v548: i32, v549: i32, v550: i32, v551: i32, v552: i32, v553: i3
     v564 = icmp uge v563, v549
     v565 = bint.i32 v564
     brnz v565, ebb67(v547)
+    fallthrough ebb158
+
+ebb158:
     v566 = iconst.i32 0
     v567 = call fn2(v0, v547, v566, v550)
     brnz v567, ebb66
+    fallthrough ebb159
+
+ebb159:
     v568 = iadd v547, v550
     jump ebb67(v568)
 
@@ -783,6 +963,9 @@ ebb66:
     v570 = icmp ult v569, v549
     v571 = bint.i32 v570
     brnz v571, ebb89(v552)
+    fallthrough ebb160
+
+ebb160:
     v572 = copy.i32 v567
     jump ebb67(v572)
 
@@ -792,6 +975,9 @@ ebb67(v573: i32):
     v576 = iconst.i32 0x4ffe
     v577 = icmp uge v575, v576
     brz v577, ebb68
+    fallthrough ebb161
+
+ebb161:
     trap heap_oob
 
 ebb68:
@@ -813,6 +999,9 @@ ebb68:
     v593 = iconst.i32 0x4ffe
     v594 = icmp uge v592, v593
     brz v594, ebb69
+    fallthrough ebb162
+
+ebb162:
     trap heap_oob
 
 ebb69:
@@ -826,12 +1015,18 @@ ebb69:
     v602 = icmp eq v600, v601
     v603 = bint.i32 v602
     brnz v603, ebb74
+    fallthrough ebb163
+
+ebb163:
     v604 = iconst.i32 2
     v605 = ishl.i32 v582, v604
     v606 = iadd.i32 v552, v605
     v607 = iconst.i32 0x4ffe
     v608 = icmp uge v606, v607
     brz v608, ebb70
+    fallthrough ebb164
+
+ebb164:
     trap heap_oob
 
 ebb70:
@@ -845,12 +1040,18 @@ ebb70:
     v616 = icmp eq v614, v615
     v617 = bint.i32 v616
     brnz v617, ebb75
+    fallthrough ebb165
+
+ebb165:
     v618 = iconst.i32 1
     v619 = iadd v614, v618
     v620 = icmp ult v619, v554
     v621 = bint.i32 v620
     v622 = copy.i32 v553
     brnz v621, ebb71(v622)
+    fallthrough ebb166
+
+ebb166:
     v623 = copy v619
     jump ebb71(v623)
 
@@ -879,6 +1080,9 @@ ebb75:
     v634 = bint.i32 v633
     v635 = copy.i32 v558
     brnz v634, ebb76(v635)
+    fallthrough ebb167
+
+ebb167:
     v636 = copy.i32 v555
     jump ebb76(v636)
 
@@ -887,6 +1091,9 @@ ebb76(v637: i32):
     v639 = iconst.i32 0x4ffe
     v640 = icmp uge v638, v639
     brz v640, ebb77
+    fallthrough ebb168
+
+ebb168:
     trap heap_oob
 
 ebb77:
@@ -899,6 +1106,9 @@ ebb77:
     v647 = icmp eq v645, v646
     v648 = bint.i32 v647
     brnz v648, ebb82(v548, v549, v551, v552)
+    fallthrough ebb169
+
+ebb169:
     v649 = iadd.i32 v548, v637
     v650 = iadd.i32 v559, v637
     v651 = iadd.i32 v560, v637
@@ -910,6 +1120,9 @@ ebb78(v652: i32, v653: i32, v654: i32, v655: i32):
     v658 = iconst.i32 0x4ffe
     v659 = icmp uge v653, v658
     brz v659, ebb79
+    fallthrough ebb170
+
+ebb170:
     trap heap_oob
 
 ebb79:
@@ -923,6 +1136,9 @@ ebb79:
     v667 = copy.i32 v554
     v668 = copy.i32 v562
     brnz v666, ebb87(v548, v654, v573, v549, v550, v551, v552, v553, v667, v668, v557, v558, v559, v560, v561)
+    fallthrough ebb171
+
+ebb171:
     v669 = iconst.i32 1
     v670 = iadd.i32 v653, v669
     v671 = iconst.i32 1
@@ -930,6 +1146,9 @@ ebb79:
     v673 = iconst.i32 0x4ffe
     v674 = icmp.i32 uge v655, v673
     brz v674, ebb80
+    fallthrough ebb172
+
+ebb172:
     trap heap_oob
 
 ebb80:
@@ -950,6 +1169,9 @@ ebb82(v682: i32, v683: i32, v684: i32, v685: i32):
     v686 = icmp.i32 ule v558, v555
     v687 = bint.i32 v686
     brnz v687, ebb90(v685, v682)
+    fallthrough ebb173
+
+ebb173:
     v688 = copy.i32 v561
     jump ebb83(v688)
 
@@ -958,6 +1180,9 @@ ebb83(v689: i32):
     v691 = iconst.i32 0x4ffe
     v692 = icmp uge v690, v691
     brz v692, ebb84
+    fallthrough ebb174
+
+ebb174:
     trap heap_oob
 
 ebb84:
@@ -970,6 +1195,9 @@ ebb84:
     v699 = iconst.i32 0x4ffe
     v700 = icmp uge v698, v699
     brz v700, ebb85
+    fallthrough ebb175
+
+ebb175:
     trap heap_oob
 
 ebb85:
@@ -981,6 +1209,9 @@ ebb85:
     v706 = icmp.i32 ne v697, v705
     v707 = bint.i32 v706
     brnz v707, ebb86
+    fallthrough ebb176
+
+ebb176:
     v708 = icmp.i32 ule v689, v555
     v709 = bint.i32 v708
     v710 = iconst.i32 -1
@@ -1019,6 +1250,9 @@ ebb90(v756: i32, v757: i32):
     v761 = iconst.i32 0x4ffe
     v762 = icmp uge v758, v761
     brz v762, ebb91
+    fallthrough ebb177
+
+ebb177:
     trap heap_oob
 
 ebb91:
@@ -1073,6 +1307,9 @@ ebb0(v0: f64, v1: i64):
     v24 = icmp ult v22, v23
     v25 = bint.i32 v24
     brnz v25, ebb10
+    fallthrough ebb178
+
+ebb178:
     v26 = iconst.i64 0x7fff_ffff_ffff_ffff
     v27 = band v14, v26
     v28 = iconst.i64 0x7ff0_0000_0000_0000
@@ -1086,10 +1323,16 @@ ebb10:
     v32 = icmp.i32 ult v22, v31
     v33 = bint.i32 v32
     brnz v33, ebb8
+    fallthrough ebb179
+
+ebb179:
     v34 = iconst.i32 0x3ff0_a2b2
     v35 = icmp.i32 uge v22, v34
     v36 = bint.i32 v35
     brnz v36, ebb6
+    fallthrough ebb180
+
+ebb180:
     v37 = iconst.i32 1
     v38 = bxor.i32 v17, v37
     v39 = isub v38, v17
@@ -1106,6 +1349,9 @@ ebb9:
     v44 = bint.i32 v43
     v45 = bor v42, v44
     brnz v45, ebb7
+    fallthrough ebb181
+
+ebb181:
     v141 = iconst.i64 0x7fe0_0000_0000_0000
     v46 = bitcast.f64 v141
     v47 = fmul.f64 v0, v46
@@ -1116,6 +1362,9 @@ ebb8:
     v49 = icmp.i32 ule v22, v48
     v50 = bint.i32 v49
     brnz v50, ebb3
+    fallthrough ebb182
+
+ebb182:
     v51 = iconst.i32 0
     v142 = iconst.i64 0
     v52 = bitcast.f64 v142
@@ -1129,6 +1378,9 @@ ebb7:
     v55 = bint.i32 v54
     v56 = bor v55, v44
     brnz v56, ebb6
+    fallthrough ebb183
+
+ebb183:
     v144 = iconst.i64 0xb6a0_0000_0000_0000
     v57 = bitcast.f64 v144
     v58 = fdiv v57, v0
@@ -1165,8 +1417,14 @@ ebb6:
     v158 = iconst.i32 0x8000_0000
     v154 = icmp ne v76, v158
     brnz v154, ebb11
+    fallthrough ebb184
+
+ebb184:
     v155 = fcmp uno v75, v75
     brz v155, ebb12
+    fallthrough ebb185
+
+ebb185:
     trap bad_toint
 
 ebb12:
@@ -1174,6 +1432,9 @@ ebb12:
     v156 = bitcast.f64 v159
     v157 = fcmp ge v156, v75
     brz v157, ebb13
+    fallthrough ebb186
+
+ebb186:
     trap int_ovf
 
 ebb13:
@@ -1230,6 +1491,9 @@ ebb4(v86: f64, v87: f64, v108: f64, v113: i32):
     v114 = icmp eq v113, v169
     v115 = bint.i32 v114
     brnz v115, ebb2(v12, v112)
+    fallthrough ebb187
+
+ebb187:
     v116 = call fn0(v112, v113, v1)
     jump ebb2(v12, v116)
 
diff --git a/cranelift/filetests/filetests/regalloc/coalescing-216.clif b/cranelift/filetests/filetests/regalloc/coalescing-216.clif
index b4d6e6393b..4276893fd2 100644
--- a/cranelift/filetests/filetests/regalloc/coalescing-216.clif
+++ b/cranelift/filetests/filetests/regalloc/coalescing-216.clif
@@ -14,15 +14,27 @@ ebb0(v0: i32, v1: i64):
 ebb4(v11: i64, v29: i64):
     v6 = iconst.i32 0
     brz v6, ebb14
+    fallthrough ebb15
+
+ebb15:
     v9 = iconst.i32 -17
     v12 = iconst.i32 0xffff_ffff_ffff_8000
     jump ebb9(v12)
 
 ebb9(v10: i32):
     brnz v10, ebb8(v9, v11, v11)
+    fallthrough ebb16
+
+ebb16:
     brz.i32 v9, ebb13
+    fallthrough ebb17
+
+ebb17:
     v13 = iconst.i32 0
     brnz v13, ebb6(v11, v11)
+    fallthrough ebb18
+
+ebb18:
     v14 = iconst.i32 0
     brz v14, ebb12
     jump ebb11
@@ -40,6 +52,9 @@ ebb13:
 ebb10(v21: i64):
     v16 = iconst.i32 0
     brnz v16, ebb6(v21, v11)
+    fallthrough ebb19
+
+ebb19:
     v17 = iconst.i32 0xffff_ffff_ffff_9f35
     jump ebb8(v17, v21, v11)
 
diff --git a/cranelift/filetests/filetests/regalloc/coloring-227.clif b/cranelift/filetests/filetests/regalloc/coloring-227.clif
index 7f14ca4e25..511134ddaa 100644
--- a/cranelift/filetests/filetests/regalloc/coloring-227.clif
+++ b/cranelift/filetests/filetests/regalloc/coloring-227.clif
@@ -17,6 +17,9 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
                           ebb6:
 [RexOp1pu_id#b8]              v8 = iconst.i32 0
 [RexOp1tjccb#75]              brnz v8, ebb5
+[-]                           fallthrough ebb20
+
+                          ebb20:
 [RexOp1pu_id#b8]              v9 = iconst.i32 0
 [RexOp1pu_id#b8]              v11 = iconst.i32 0
 [RexOp1icscc#39]              v12 = icmp.i32 eq v15, v11
@@ -27,8 +30,14 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 
                           ebb7:
 [RexOp1tjccb#74]              brz.i32 v17, ebb8
+[-]                           fallthrough ebb17
+
+                          ebb17:
 [RexOp1pu_id#b8]              v18 = iconst.i32 0
 [RexOp1tjccb#74]              brz v18, ebb9
+[-]                           fallthrough ebb16
+
+                          ebb16:
 [RexOp1pu_id#b8]              v21 = iconst.i32 0
 [RexOp1umr#89]                v79 = uextend.i64 v5
 [RexOp1r_ib#8083]             v80 = iadd_imm.i64 v4, 0
@@ -63,8 +72,14 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
                           ebb2(v7: i32, v45: i32, v52: i32, v59: i32, v66: i32, v73: i32):
 [RexOp1pu_id#b8]              v44 = iconst.i32 0
 [RexOp1tjccb#74]              brz v44, ebb12
+[-]                           fallthrough ebb18
+
+                          ebb18:
 [RexOp1pu_id#b8]              v50 = iconst.i32 11
 [RexOp1tjccb#74]              brz v50, ebb14
+[-]                           fallthrough ebb19
+
+                          ebb19:
 [RexOp1umr#89]                v82 = uextend.i64 v52
 [RexOp1r_ib#8083]             v83 = iadd_imm.i64 v4, 0
 [RexOp1ld#808b]               v84 = load.i64 v83
diff --git a/cranelift/filetests/filetests/regalloc/ghost-param.clif b/cranelift/filetests/filetests/regalloc/ghost-param.clif
index 8a022b4af5..af007179dc 100644
--- a/cranelift/filetests/filetests/regalloc/ghost-param.clif
+++ b/cranelift/filetests/filetests/regalloc/ghost-param.clif
@@ -20,6 +20,9 @@ ebb5(v9: f64):
     v6 = iconst.i32 0
     v7 = iconst.i32 1
     brnz v7, ebb4(v6)
+    fallthrough ebb8
+
+ebb8:
     v8 = iconst.i32 0
     jump ebb7(v8)
 
diff --git a/cranelift/filetests/filetests/regalloc/global-constraints.clif b/cranelift/filetests/filetests/regalloc/global-constraints.clif
index e3e59dd470..4ea87eaca9 100644
--- a/cranelift/filetests/filetests/regalloc/global-constraints.clif
+++ b/cranelift/filetests/filetests/regalloc/global-constraints.clif
@@ -14,6 +14,9 @@ ebb0(v0: i32):
     v4 = icmp_imm ne v0, 4
     v5 = icmp_imm sge v0, 5
     brnz v5, ebb1
+    fallthrough ebb2
+
+ebb2:
     return
 
 ebb1:
diff --git a/cranelift/filetests/filetests/regalloc/iterate.clif b/cranelift/filetests/filetests/regalloc/iterate.clif
index 424ecfa740..fe8c9cbaef 100644
--- a/cranelift/filetests/filetests/regalloc/iterate.clif
+++ b/cranelift/filetests/filetests/regalloc/iterate.clif
@@ -20,6 +20,9 @@ ebb0(v0: i64, v1: f32, v2: f64, v3: i32, v4: i32, v5: i64):
     v44 = iconst.i64 0
     v37 = icmp slt v0, v44
     brnz v37, ebb2
+    fallthrough ebb11
+
+ebb11:
     v38 = fcvt_from_sint.f64 v0
     jump ebb3(v38)
 
@@ -41,6 +44,9 @@ ebb3(v15: f64):
     v54 = iconst.i64 0
     v47 = icmp.i64 slt v13, v54
     brnz v47, ebb4
+    fallthrough ebb12
+
+ebb12:
     v48 = fcvt_from_sint.f64 v13
     jump ebb5(v48)
 
@@ -57,6 +63,9 @@ ebb5(v20: f64):
     v63 = iconst.i64 0
     v56 = icmp.i64 slt v7, v63
     brnz v56, ebb6
+    fallthrough ebb13
+
+ebb13:
     v57 = fcvt_from_sint.f64 v7
     jump ebb7(v57)
 
@@ -82,8 +91,14 @@ ebb7(v21: f64):
     v69 = iconst.i64 0x8000_0000_0000_0000
     v65 = icmp ne v30, v69
     brnz v65, ebb8
+    fallthrough ebb15
+
+ebb15:
     v66 = fcmp uno v29, v29
     brz v66, ebb9
+    fallthrough ebb16
+
+ebb16:
     trap bad_toint
 
 ebb9:
@@ -91,6 +106,9 @@ ebb9:
     v67 = bitcast.f64 v70
     v68 = fcmp gt v67, v29
     brz v68, ebb10
+    fallthrough ebb17
+
+ebb17:
     trap int_ovf
 
 ebb10:
@@ -118,6 +136,9 @@ ebb0(v0: i64):
     v7 = icmp uge v3, v6
     ; If we're unlucky, there are no ABCD registers available for v7 at this branch.
     brz v7, ebb2
+    fallthrough ebb4
+
+ebb4:
     trap oob
 
 ebb2:
@@ -128,6 +149,9 @@ ebb2:
     v11 = iadd v8, v10
     v12 = load.i64 v11
     brnz v12, ebb3
+    fallthrough ebb5
+
+ebb5:
     trap icall_null
 
 ebb3:
diff --git a/cranelift/filetests/filetests/regalloc/reload-208.clif b/cranelift/filetests/filetests/regalloc/reload-208.clif
index 116e5b719f..e89c6371ba 100644
--- a/cranelift/filetests/filetests/regalloc/reload-208.clif
+++ b/cranelift/filetests/filetests/regalloc/reload-208.clif
@@ -25,6 +25,9 @@ ebb0(v0: i64):
     v20 = iconst.i32 0x4ffe
     v16 = icmp uge v2, v20
     brz v16, ebb5
+    fallthrough ebb9
+
+ebb9:
     trap heap_oob
 
 ebb5:
@@ -44,6 +47,9 @@ ebb3(v7: i32):
     v26 = iconst.i32 0x4ffe
     v22 = icmp uge v7, v26
     brz v22, ebb6
+    fallthrough ebb10
+
+ebb10:
     trap heap_oob
 
 ebb6:
@@ -65,6 +71,9 @@ ebb2:
     v31 = iconst.i32 0x4ffe
     v27 = icmp uge v10, v31
     brz v27, ebb7
+    fallthrough ebb11
+
+ebb11:
     trap heap_oob
 
 ebb7:
@@ -78,6 +87,9 @@ ebb7:
     v36 = iconst.i32 0x4ffe
     v32 = icmp uge v13, v36
     brz v32, ebb8
+    fallthrough ebb12
+
+ebb12:
     trap heap_oob
 
 ebb8:
diff --git a/cranelift/filetests/filetests/regalloc/spill.clif b/cranelift/filetests/filetests/regalloc/spill.clif
index 21731f321a..8cf221d5af 100644
--- a/cranelift/filetests/filetests/regalloc/spill.clif
+++ b/cranelift/filetests/filetests/regalloc/spill.clif
@@ -151,6 +151,9 @@ ebb0(v1: i32):
     ; check: v1 = spill
     v2 = iconst.i32 1
     brnz v1, ebb1(v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2)
+    fallthrough ebb2
+
+ebb2:
     return v1
 
 ebb1(v10: i32, v11: i32, v12: i32, v13: i32, v14: i32, v15: i32, v16: i32, v17: i32, v18: i32, v19: i32, v20: i32, v21: i32):

From f431465802718ba33b274fc44b050d301219d34e Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Thu, 4 Jul 2019 19:38:45 +0200
Subject: [PATCH 2494/3084] Replace newly inserted fallthrough by jump.

---
 .../filetests/isa/x86/prologue-epilogue.clif  |   2 +-
 .../isa/x86/shrink-multiple-uses.clif         |   2 +-
 .../filetests/filetests/licm/complex.clif     |   4 +-
 .../filetests/licm/critical-edge.clif         |   4 +-
 .../filetests/filetests/postopt/basic.clif    |  16 +-
 .../filetests/regalloc/coalesce.clif          |  14 +-
 .../filetests/regalloc/coalescing-207.clif    | 176 +++++++++---------
 .../filetests/regalloc/coalescing-216.clif    |  10 +-
 .../filetests/regalloc/coloring-227.clif      |  10 +-
 .../filetests/regalloc/ghost-param.clif       |   2 +-
 .../regalloc/global-constraints.clif          |   2 +-
 .../filetests/filetests/regalloc/iterate.clif |  16 +-
 .../filetests/regalloc/reload-208.clif        |   8 +-
 .../filetests/filetests/regalloc/spill.clif   |   2 +-
 14 files changed, 134 insertions(+), 134 deletions(-)

diff --git a/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif b/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
index 1c715f2a21..ec67358804 100644
--- a/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
+++ b/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
@@ -217,7 +217,7 @@ ebb0(v0: i32):
 
 ebb1(v4: i32, v5: i32, v6: i32):
     brz v4, ebb3
-    fallthrough ebb2
+    jump ebb2
 
 ebb2:
     v7 = iadd v5, v6
diff --git a/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif b/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
index deec4ff505..c0a34f2ecc 100644
--- a/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
+++ b/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
@@ -8,7 +8,7 @@ ebb0(v0: i32 [%rdi]):
 [Op2seti_abcd#490,%rax]             v1 = trueif eq v3
 [RexOp2urm_noflags#4b6,%rax]        v2 = bint.i32 v1
 [Op1brib#70]                        brif eq v3, ebb1
-[-]                                 fallthrough ebb2
+[Op1jmpb#eb]                        jump ebb2
 
 ebb2:
 [Op1ret#c3]                         return v2
diff --git a/cranelift/filetests/filetests/licm/complex.clif b/cranelift/filetests/filetests/licm/complex.clif
index e2bfffc3d6..2774cde5b3 100644
--- a/cranelift/filetests/filetests/licm/complex.clif
+++ b/cranelift/filetests/filetests/licm/complex.clif
@@ -39,7 +39,7 @@ ebb0(v0: i32):
             v19 = iadd v18, v2
             v20 = iadd.i32 v2, v3
 [SBzero#18] brz.i32 v1, ebb1(v20)
-            fallthrough ebb7
+[UJ#1b]     jump ebb7
 
          ebb7:
 [Iret#19]   return v19
@@ -89,7 +89,7 @@ ebb0(v0: i32):
 ; nextln: ebb6(v18: i32):
 ; nextln:     v19 = iadd v18, v2
 ; nextln:     brz.i32 v1, ebb1(v20)
-; nextln:     fallthrough ebb7
+; nextln:     jump ebb7
 ; nextln: 
 ; nextln: ebb7:
 ; nextln:     return v19
diff --git a/cranelift/filetests/filetests/licm/critical-edge.clif b/cranelift/filetests/filetests/licm/critical-edge.clif
index b3636c10e0..89beb387cc 100644
--- a/cranelift/filetests/filetests/licm/critical-edge.clif
+++ b/cranelift/filetests/filetests/licm/critical-edge.clif
@@ -7,7 +7,7 @@ function %critical_edge(i32, i32) -> i32 {
 
             ebb0(v0: i32, v7: i32):
 [SBzero#38]   brnz v7, ebb2(v0)
-[-]           fallthrough ebb1
+[UJ#1b]       jump ebb1
 
             ebb1:
 [Iret#19]     return v0
@@ -30,7 +30,7 @@ function %critical_edge(i32, i32) -> i32 {
 ; sameln: function %critical_edge
 ; nextln: ebb0(v0: i32, v7: i32):
 ; nextln:     brnz v7, ebb5(v0)
-; nextln:     fallthrough ebb1
+; nextln:     jump ebb1
 ; nextln: 
 ; nextln: ebb1:
 ; nextln:     return v0
diff --git a/cranelift/filetests/filetests/postopt/basic.clif b/cranelift/filetests/filetests/postopt/basic.clif
index 962434bfac..442d47de89 100644
--- a/cranelift/filetests/filetests/postopt/basic.clif
+++ b/cranelift/filetests/filetests/postopt/basic.clif
@@ -7,7 +7,7 @@ function %br_icmp(i32, i32) -> i32 {
 ebb0(v0: i32, v1: i32):
 [Op1icscc#39,%rdx]  v2 = icmp slt v0, v1
 [Op1t8jccd_long#85] brnz v2, ebb1
-[-]                 fallthrough ebb2
+[Op1jmpb#eb]        jump ebb2
 
 ebb2:
 [Op1ret#c3]         return v1
@@ -21,7 +21,7 @@ ebb1:
 ; nextln:    v9 = ifcmp v0, v1
 ; nextln:    v2 = trueif slt v9
 ; nextln:    brif slt v9, ebb1
-; nextln:    fallthrough ebb2
+; nextln:    jump ebb2
 ; nextln: 
 ; nextln: ebb2:
 ; nextln:    return v1
@@ -37,7 +37,7 @@ function %br_icmp_inverse(i32, i32) -> i32 {
 ebb0(v0: i32, v1: i32):
 [Op1icscc#39,%rdx]  v2 = icmp slt v0, v1
 [Op1t8jccd_long#84] brz v2, ebb1
-[-]                 fallthrough ebb2
+[Op1jmpb#eb]        jump ebb2
 
 ebb2:
 [Op1ret#c3]         return v1
@@ -51,7 +51,7 @@ ebb1:
 ; nextln:    v9 = ifcmp v0, v1
 ; nextln:    v2 = trueif slt v9
 ; nextln:    brif sge v9, ebb1
-; nextln:    fallthrough ebb2
+; nextln:    jump ebb2
 ; nextln: 
 ; nextln: ebb2:
 ; nextln:    return v1
@@ -67,7 +67,7 @@ function %br_icmp_imm(i32, i32) -> i32 {
 ebb0(v0: i32, v1: i32):
 [Op1icscc_ib#7083]  v2 = icmp_imm slt v0, 2
 [Op1t8jccd_long#84] brz v2, ebb1
-[-]                 fallthrough ebb2
+[Op1jmpb#eb]        jump ebb2
 
 ebb2:
 [Op1ret#c3]         return v1
@@ -81,7 +81,7 @@ ebb1:
 ; nextln:    v9 = ifcmp_imm v0, 2
 ; nextln:    v2 = trueif slt v9
 ; nextln:    brif sge v9, ebb1
-; nextln:    fallthrough ebb2
+; nextln:    jump ebb2
 ; nextln: 
 ; nextln: ebb2:
 ; nextln:    return v1
@@ -97,7 +97,7 @@ function %br_fcmp(f32, f32) -> f32 {
 ebb0(v0: f32, v1: f32):
 [Op2fcscc#42e,%rdx] v2 = fcmp gt v0, v1
 [Op1t8jccd_long#84] brz v2, ebb1
-[-]                 fallthrough ebb2
+[Op1jmpb#eb]        jump ebb2
 
 ebb2:
 [Op1ret#c3]         return v1
@@ -112,7 +112,7 @@ ebb1:
 ; nextln:    v19 = ffcmp v0, v1
 ; nextln:    v2 = trueff gt v19
 ; nextln:    brff ule v19, ebb1
-; nextln:    fallthrough ebb2
+; nextln:    jump ebb2
 ; nextln: 
 ; nextln: ebb2:
 ; nextln:    return v1
diff --git a/cranelift/filetests/filetests/regalloc/coalesce.clif b/cranelift/filetests/filetests/regalloc/coalesce.clif
index f667d7440e..fa9f64ba4d 100644
--- a/cranelift/filetests/filetests/regalloc/coalesce.clif
+++ b/cranelift/filetests/filetests/regalloc/coalesce.clif
@@ -12,7 +12,7 @@ ebb0(v0: i32):
     ; not: copy
     ; v0 is used by the branch and passed as an arg - that's no conflict.
     brnz v0, ebb1(v0)
-    fallthrough ebb2
+    jump ebb2
 
 ebb2:
     ; v0 is live across the branch above. That's no conflict.
@@ -29,7 +29,7 @@ ebb0(v0: i32):
     ; check: $(cp1=$V) = copy v0
     ; nextln: brnz v0, ebb1($cp1)
     brnz v0, ebb1(v0)
-    fallthrough ebb2
+    jump ebb2
 
 ebb2:
     ; not: copy
@@ -48,7 +48,7 @@ ebb0(v0: i32):
     ; check: $(cp1=$V) = copy v0
     ; nextln: brnz v0, ebb1($cp1, v0)
     brnz v0, ebb1(v0, v0)
-    fallthrough ebb2
+    jump ebb2
 
 ebb2:
     v1 = iadd_imm v0, 7
@@ -68,7 +68,7 @@ ebb0(v0: i32):
     ; not: copy
     ; check: brnz v0, ebb1($cp0)
     brnz v0, ebb1(v0)
-    fallthrough ebb2
+    jump ebb2
 
 ebb2:
     v1 = iadd_imm v0, 7
@@ -101,7 +101,7 @@ ebb1(v10: i32, v11: i32):
     ; not: copy
     ; check: brnz v13, ebb1($nv11b, v12)
     brnz v13, ebb1(v11, v12)
-    fallthrough ebb2
+    jump ebb2
 
 ebb2:
     return v12
@@ -131,12 +131,12 @@ ebb0(v0: i32):
 
 ebb2(v3: i32, v4: i32):
     brnz v3, ebb2(v3, v4)
-    fallthrough ebb3
+    jump ebb3
 
 ebb3:
     v5 = iconst.i32 1
     brnz v3, ebb2(v2, v5)
-    fallthrough ebb4
+    jump ebb4
 
 ebb4:
     return
diff --git a/cranelift/filetests/filetests/regalloc/coalescing-207.clif b/cranelift/filetests/filetests/regalloc/coalescing-207.clif
index 3f01f6fcc5..17d33d33b3 100644
--- a/cranelift/filetests/filetests/regalloc/coalescing-207.clif
+++ b/cranelift/filetests/filetests/regalloc/coalescing-207.clif
@@ -22,7 +22,7 @@ ebb0(v0: i64, v1: i32, v2: i32):
     v6 = iconst.i32 0x4ffe
     v7 = icmp uge v5, v6
     brz v7, ebb1
-    fallthrough ebb100
+    jump ebb100
 
 ebb100:
     trap heap_oob
@@ -38,7 +38,7 @@ ebb1:
     v15 = iconst.i32 0x4ffe
     v16 = icmp.i32 uge v4, v15
     brz v16, ebb2
-    fallthrough ebb101
+    jump ebb101
 
 ebb101:
     trap heap_oob
@@ -52,7 +52,7 @@ ebb2:
     v21 = iconst.i32 0x4ffe
     v22 = icmp.i32 uge v2, v21
     brz v22, ebb3
-    fallthrough ebb102
+    jump ebb102
 
 ebb102:
     trap heap_oob
@@ -69,7 +69,7 @@ ebb3:
     v31 = icmp eq v29, v30
     v32 = bint.i32 v31
     brnz v32, ebb90(v14, v1)
-    fallthrough ebb103
+    jump ebb103
 
 ebb103:
     v33 = call fn0(v0, v1, v27)
@@ -78,13 +78,13 @@ ebb103:
     v36 = icmp eq v33, v35
     v37 = bint.i32 v36
     brnz v37, ebb90(v14, v34)
-    fallthrough ebb104
+    jump ebb104
 
 ebb104:
     v38 = iconst.i32 0x4ffe
     v39 = icmp.i32 uge v2, v38
     brz v39, ebb4
-    fallthrough ebb105
+    jump ebb105
 
 ebb105:
     trap heap_oob
@@ -99,13 +99,13 @@ ebb4:
     v46 = icmp eq v44, v45
     v47 = bint.i32 v46
     brnz v47, ebb56(v33, v14)
-    fallthrough ebb106
+    jump ebb106
 
 ebb106:
     v48 = iconst.i32 0x4ffe
     v49 = icmp.i32 uge v33, v48
     brz v49, ebb5
-    fallthrough ebb107
+    jump ebb107
 
 ebb107:
     trap heap_oob
@@ -120,13 +120,13 @@ ebb5:
     v56 = icmp eq v54, v55
     v57 = bint.i32 v56
     brnz v57, ebb90(v14, v34)
-    fallthrough ebb108
+    jump ebb108
 
 ebb108:
     v58 = iconst.i32 0x4ffe
     v59 = icmp.i32 uge v2, v58
     brz v59, ebb6
-    fallthrough ebb109
+    jump ebb109
 
 ebb109:
     trap heap_oob
@@ -141,13 +141,13 @@ ebb6:
     v66 = icmp eq v64, v65
     v67 = bint.i32 v66
     brnz v67, ebb42
-    fallthrough ebb110
+    jump ebb110
 
 ebb110:
     v68 = iconst.i32 0x4ffe
     v69 = icmp.i32 uge v33, v68
     brz v69, ebb7
-    fallthrough ebb111
+    jump ebb111
 
 ebb111:
     trap heap_oob
@@ -162,13 +162,13 @@ ebb7:
     v76 = icmp eq v74, v75
     v77 = bint.i32 v76
     brnz v77, ebb90(v14, v34)
-    fallthrough ebb112
+    jump ebb112
 
 ebb112:
     v78 = iconst.i32 0x4ffe
     v79 = icmp.i32 uge v2, v78
     brz v79, ebb8
-    fallthrough ebb113
+    jump ebb113
 
 ebb113:
     trap heap_oob
@@ -183,13 +183,13 @@ ebb8:
     v86 = icmp eq v84, v85
     v87 = bint.i32 v86
     brnz v87, ebb46
-    fallthrough ebb114
+    jump ebb114
 
 ebb114:
     v88 = iconst.i32 0x4ffe
     v89 = icmp.i32 uge v33, v88
     brz v89, ebb9
-    fallthrough ebb115
+    jump ebb115
 
 ebb115:
     trap heap_oob
@@ -204,13 +204,13 @@ ebb9:
     v96 = icmp eq v94, v95
     v97 = bint.i32 v96
     brnz v97, ebb90(v14, v34)
-    fallthrough ebb116
+    jump ebb116
 
 ebb116:
     v98 = iconst.i32 0x4ffe
     v99 = icmp.i32 uge v2, v98
     brz v99, ebb10
-    fallthrough ebb117
+    jump ebb117
 
 ebb117:
     trap heap_oob
@@ -225,7 +225,7 @@ ebb10:
     v106 = icmp eq v104, v105
     v107 = bint.i32 v106
     brnz v107, ebb54
-    fallthrough ebb118
+    jump ebb118
 
 ebb118:
     v108 = iconst.i32 1
@@ -236,7 +236,7 @@ ebb118:
     v113 = iconst.i32 0x4ffe
     v114 = icmp uge v111, v113
     brz v114, ebb11
-    fallthrough ebb119
+    jump ebb119
 
 ebb119:
     trap heap_oob
@@ -253,7 +253,7 @@ ebb11:
     v122 = iconst.i32 0x4ffe
     v123 = icmp uge v120, v122
     brz v123, ebb12
-    fallthrough ebb120
+    jump ebb120
 
 ebb120:
     trap heap_oob
@@ -268,7 +268,7 @@ ebb12:
     v129 = iconst.i32 0x4ffe
     v130 = icmp.i32 uge v14, v129
     brz v130, ebb13
-    fallthrough ebb121
+    jump ebb121
 
 ebb121:
     trap heap_oob
@@ -283,7 +283,7 @@ ebb13:
     v136 = iconst.i32 0x4ffe
     v137 = icmp.i32 uge v14, v136
     brz v137, ebb14
-    fallthrough ebb122
+    jump ebb122
 
 ebb122:
     trap heap_oob
@@ -304,7 +304,7 @@ ebb15(v143: i32, v144: i32):
     v148 = iconst.i32 0x4ffe
     v149 = icmp uge v147, v148
     brz v149, ebb16
-    fallthrough ebb123
+    jump ebb123
 
 ebb123:
     trap heap_oob
@@ -319,7 +319,7 @@ ebb16:
     v156 = icmp eq v154, v155
     v157 = bint.i32 v156
     brnz v157, ebb89(v14)
-    fallthrough ebb124
+    jump ebb124
 
 ebb124:
     v158 = iconst.i32 255
@@ -332,7 +332,7 @@ ebb124:
     v165 = iconst.i32 0x4ffe
     v166 = icmp uge v162, v165
     brz v166, ebb17
-    fallthrough ebb125
+    jump ebb125
 
 ebb125:
     trap heap_oob
@@ -353,7 +353,7 @@ ebb17:
     v178 = iconst.i32 0x4ffe
     v179 = icmp uge v177, v178
     brz v179, ebb18
-    fallthrough ebb126
+    jump ebb126
 
 ebb126:
     trap heap_oob
@@ -372,7 +372,7 @@ ebb18:
     v190 = iconst.i32 0x4ffe
     v191 = icmp.i32 uge v177, v190
     brz v191, ebb19
-    fallthrough ebb127
+    jump ebb127
 
 ebb127:
     trap heap_oob
@@ -391,7 +391,7 @@ ebb19:
     v201 = iconst.i32 0x4ffe
     v202 = icmp uge v200, v201
     brz v202, ebb20
-    fallthrough ebb128
+    jump ebb128
 
 ebb128:
     trap heap_oob
@@ -416,7 +416,7 @@ ebb21:
     v215 = icmp ult v213, v214
     v216 = bint.i32 v215
     brnz v216, ebb38(v2, v211, v209, v210, v208, v198, v213, v33, v14)
-    fallthrough ebb129
+    jump ebb129
 
 ebb129:
     v217 = iconst.i32 -1
@@ -434,7 +434,7 @@ ebb22(v223: i32, v224: i32, v225: i32, v226: i32, v227: i32, v228: i32, v229: i3
     v237 = iconst.i32 0x4ffe
     v238 = icmp uge v236, v237
     brz v238, ebb23
-    fallthrough ebb130
+    jump ebb130
 
 ebb130:
     trap heap_oob
@@ -450,13 +450,13 @@ ebb23:
     v246 = icmp ne v243, v245
     v247 = bint.i32 v246
     brnz v247, ebb24
-    fallthrough ebb131
+    jump ebb131
 
 ebb131:
     v248 = icmp.i32 ne v224, v226
     v249 = bint.i32 v248
     brnz v249, ebb25
-    fallthrough ebb132
+    jump ebb132
 
 ebb132:
     v250 = iadd.i32 v227, v226
@@ -467,7 +467,7 @@ ebb24:
     v252 = icmp.i32 ule v243, v245
     v253 = bint.i32 v252
     brnz v253, ebb26
-    fallthrough ebb133
+    jump ebb133
 
 ebb133:
     v254 = isub.i32 v234, v223
@@ -493,14 +493,14 @@ ebb27(v264: i32, v265: i32, v266: i32, v267: i32):
     v269 = icmp uge v268, v229
     v270 = bint.i32 v269
     brnz v270, ebb29
-    fallthrough ebb134
+    jump ebb134
 
 ebb134:
     v271 = iadd.i32 v2, v268
     v272 = iconst.i32 0x4ffe
     v273 = icmp uge v271, v272
     brz v273, ebb28
-    fallthrough ebb135
+    jump ebb135
 
 ebb135:
     trap heap_oob
@@ -532,7 +532,7 @@ ebb31(v285: i32, v286: i32, v287: i32, v288: i32, v289: i32, v290: i32, v291: i3
     v300 = iconst.i32 0x4ffe
     v301 = icmp uge v299, v300
     brz v301, ebb32
-    fallthrough ebb136
+    jump ebb136
 
 ebb136:
     trap heap_oob
@@ -548,13 +548,13 @@ ebb32:
     v309 = icmp ne v306, v308
     v310 = bint.i32 v309
     brnz v310, ebb33
-    fallthrough ebb137
+    jump ebb137
 
 ebb137:
     v311 = icmp.i32 ne v286, v288
     v312 = bint.i32 v311
     brnz v312, ebb34
-    fallthrough ebb138
+    jump ebb138
 
 ebb138:
     v313 = iadd.i32 v289, v288
@@ -565,7 +565,7 @@ ebb33:
     v315 = icmp.i32 uge v306, v308
     v316 = bint.i32 v315
     brnz v316, ebb35
-    fallthrough ebb139
+    jump ebb139
 
 ebb139:
     v317 = isub.i32 v297, v285
@@ -591,14 +591,14 @@ ebb36(v327: i32, v328: i32, v329: i32, v330: i32):
     v332 = icmp uge v331, v291
     v333 = bint.i32 v332
     brnz v333, ebb38(v2, v330, v292, v329, v293, v294, v291, v295, v296)
-    fallthrough ebb140
+    jump ebb140
 
 ebb140:
     v334 = iadd.i32 v2, v331
     v335 = iconst.i32 0x4ffe
     v336 = icmp uge v334, v335
     brz v336, ebb37
-    fallthrough ebb141
+    jump ebb141
 
 ebb141:
     trap heap_oob
@@ -620,7 +620,7 @@ ebb38(v343: i32, v344: i32, v345: i32, v346: i32, v347: i32, v348: i32, v349: i3
     v356 = icmp ugt v353, v355
     v357 = bint.i32 v356
     brnz v357, ebb39(v344)
-    fallthrough ebb142
+    jump ebb142
 
 ebb142:
     v358 = copy v345
@@ -629,7 +629,7 @@ ebb142:
 ebb39(v359: i32):
     v360 = iadd.i32 v343, v359
     brnz.i32 v357, ebb40(v346)
-    fallthrough ebb143
+    jump ebb143
 
 ebb143:
     v361 = copy.i32 v347
@@ -643,7 +643,7 @@ ebb40(v362: i32):
     v367 = icmp eq v365, v366
     v368 = bint.i32 v367
     brnz v368, ebb63
-    fallthrough ebb144
+    jump ebb144
 
 ebb144:
     v369 = iconst.i32 1
@@ -655,7 +655,7 @@ ebb144:
     v375 = bint.i32 v374
     v376 = copy v362
     brnz v375, ebb41(v376)
-    fallthrough ebb145
+    jump ebb145
 
 ebb145:
     v377 = copy v373
@@ -674,7 +674,7 @@ ebb42:
     v385 = iconst.i32 0x4ffe
     v386 = icmp.i32 uge v33, v385
     brz v386, ebb43
-    fallthrough ebb146
+    jump ebb146
 
 ebb146:
     trap heap_oob
@@ -698,7 +698,7 @@ ebb44(v392: i32, v393: i32, v394: i32):
     v402 = icmp eq v401, v384
     v403 = bint.i32 v402
     brnz v403, ebb56(v394, v14)
-    fallthrough ebb147
+    jump ebb147
 
 ebb147:
     v404 = iconst.i32 2
@@ -708,7 +708,7 @@ ebb147:
     v408 = iconst.i32 0x4ffe
     v409 = icmp uge v405, v408
     brz v409, ebb45
-    fallthrough ebb148
+    jump ebb148
 
 ebb148:
     trap heap_oob
@@ -731,7 +731,7 @@ ebb46:
     v420 = iconst.i32 0x4ffe
     v421 = icmp.i32 uge v33, v420
     brz v421, ebb47
-    fallthrough ebb149
+    jump ebb149
 
 ebb149:
     trap heap_oob
@@ -766,7 +766,7 @@ ebb48(v440: i32, v441: i32):
     v446 = iconst.i32 0x4ffe
     v447 = icmp uge v445, v446
     brz v447, ebb49
-    fallthrough ebb150
+    jump ebb150
 
 ebb150:
     trap heap_oob
@@ -781,7 +781,7 @@ ebb49:
     v454 = icmp eq v452, v453
     v455 = bint.i32 v454
     brnz v455, ebb51(v14)
-    fallthrough ebb151
+    jump ebb151
 
 ebb151:
     v456 = bor.i32 v441, v452
@@ -803,7 +803,7 @@ ebb51(v462: i32):
     v466 = iconst.i32 0x4ffe
     v467 = icmp uge v463, v466
     brz v467, ebb52
-    fallthrough ebb152
+    jump ebb152
 
 ebb152:
     trap heap_oob
@@ -816,7 +816,7 @@ ebb52:
     store.i32 v465, v471+4
     v472 = iconst.i32 0
     brnz.i32 v452, ebb53(v443)
-    fallthrough ebb153
+    jump ebb153
 
 ebb153:
     v473 = copy v472
@@ -835,7 +835,7 @@ ebb54:
     v481 = iconst.i32 0x4ffe
     v482 = icmp.i32 uge v33, v481
     brz v482, ebb55
-    fallthrough ebb154
+    jump ebb154
 
 ebb154:
     trap heap_oob
@@ -878,7 +878,7 @@ ebb58(v505: i32, v506: i32):
     v511 = iconst.i32 0x4ffe
     v512 = icmp uge v508, v511
     brz v512, ebb59
-    fallthrough ebb155
+    jump ebb155
 
 ebb155:
     trap heap_oob
@@ -893,7 +893,7 @@ ebb59:
     v519 = icmp eq v517, v518
     v520 = bint.i32 v519
     brnz v520, ebb61(v14)
-    fallthrough ebb156
+    jump ebb156
 
 ebb156:
     v521 = iconst.i32 8
@@ -910,7 +910,7 @@ ebb60:
 ebb61(v526: i32):
     v527 = iconst.i32 0
     brnz.i32 v517, ebb62(v510)
-    fallthrough ebb157
+    jump ebb157
 
 ebb157:
     v528 = copy v527
@@ -946,13 +946,13 @@ ebb65(v547: i32, v548: i32, v549: i32, v550: i32, v551: i32, v552: i32, v553: i3
     v564 = icmp uge v563, v549
     v565 = bint.i32 v564
     brnz v565, ebb67(v547)
-    fallthrough ebb158
+    jump ebb158
 
 ebb158:
     v566 = iconst.i32 0
     v567 = call fn2(v0, v547, v566, v550)
     brnz v567, ebb66
-    fallthrough ebb159
+    jump ebb159
 
 ebb159:
     v568 = iadd v547, v550
@@ -963,7 +963,7 @@ ebb66:
     v570 = icmp ult v569, v549
     v571 = bint.i32 v570
     brnz v571, ebb89(v552)
-    fallthrough ebb160
+    jump ebb160
 
 ebb160:
     v572 = copy.i32 v567
@@ -975,7 +975,7 @@ ebb67(v573: i32):
     v576 = iconst.i32 0x4ffe
     v577 = icmp uge v575, v576
     brz v577, ebb68
-    fallthrough ebb161
+    jump ebb161
 
 ebb161:
     trap heap_oob
@@ -999,7 +999,7 @@ ebb68:
     v593 = iconst.i32 0x4ffe
     v594 = icmp uge v592, v593
     brz v594, ebb69
-    fallthrough ebb162
+    jump ebb162
 
 ebb162:
     trap heap_oob
@@ -1015,7 +1015,7 @@ ebb69:
     v602 = icmp eq v600, v601
     v603 = bint.i32 v602
     brnz v603, ebb74
-    fallthrough ebb163
+    jump ebb163
 
 ebb163:
     v604 = iconst.i32 2
@@ -1024,7 +1024,7 @@ ebb163:
     v607 = iconst.i32 0x4ffe
     v608 = icmp uge v606, v607
     brz v608, ebb70
-    fallthrough ebb164
+    jump ebb164
 
 ebb164:
     trap heap_oob
@@ -1040,7 +1040,7 @@ ebb70:
     v616 = icmp eq v614, v615
     v617 = bint.i32 v616
     brnz v617, ebb75
-    fallthrough ebb165
+    jump ebb165
 
 ebb165:
     v618 = iconst.i32 1
@@ -1049,7 +1049,7 @@ ebb165:
     v621 = bint.i32 v620
     v622 = copy.i32 v553
     brnz v621, ebb71(v622)
-    fallthrough ebb166
+    jump ebb166
 
 ebb166:
     v623 = copy v619
@@ -1080,7 +1080,7 @@ ebb75:
     v634 = bint.i32 v633
     v635 = copy.i32 v558
     brnz v634, ebb76(v635)
-    fallthrough ebb167
+    jump ebb167
 
 ebb167:
     v636 = copy.i32 v555
@@ -1091,7 +1091,7 @@ ebb76(v637: i32):
     v639 = iconst.i32 0x4ffe
     v640 = icmp uge v638, v639
     brz v640, ebb77
-    fallthrough ebb168
+    jump ebb168
 
 ebb168:
     trap heap_oob
@@ -1106,7 +1106,7 @@ ebb77:
     v647 = icmp eq v645, v646
     v648 = bint.i32 v647
     brnz v648, ebb82(v548, v549, v551, v552)
-    fallthrough ebb169
+    jump ebb169
 
 ebb169:
     v649 = iadd.i32 v548, v637
@@ -1120,7 +1120,7 @@ ebb78(v652: i32, v653: i32, v654: i32, v655: i32):
     v658 = iconst.i32 0x4ffe
     v659 = icmp uge v653, v658
     brz v659, ebb79
-    fallthrough ebb170
+    jump ebb170
 
 ebb170:
     trap heap_oob
@@ -1136,7 +1136,7 @@ ebb79:
     v667 = copy.i32 v554
     v668 = copy.i32 v562
     brnz v666, ebb87(v548, v654, v573, v549, v550, v551, v552, v553, v667, v668, v557, v558, v559, v560, v561)
-    fallthrough ebb171
+    jump ebb171
 
 ebb171:
     v669 = iconst.i32 1
@@ -1146,7 +1146,7 @@ ebb171:
     v673 = iconst.i32 0x4ffe
     v674 = icmp.i32 uge v655, v673
     brz v674, ebb80
-    fallthrough ebb172
+    jump ebb172
 
 ebb172:
     trap heap_oob
@@ -1169,7 +1169,7 @@ ebb82(v682: i32, v683: i32, v684: i32, v685: i32):
     v686 = icmp.i32 ule v558, v555
     v687 = bint.i32 v686
     brnz v687, ebb90(v685, v682)
-    fallthrough ebb173
+    jump ebb173
 
 ebb173:
     v688 = copy.i32 v561
@@ -1180,7 +1180,7 @@ ebb83(v689: i32):
     v691 = iconst.i32 0x4ffe
     v692 = icmp uge v690, v691
     brz v692, ebb84
-    fallthrough ebb174
+    jump ebb174
 
 ebb174:
     trap heap_oob
@@ -1195,7 +1195,7 @@ ebb84:
     v699 = iconst.i32 0x4ffe
     v700 = icmp uge v698, v699
     brz v700, ebb85
-    fallthrough ebb175
+    jump ebb175
 
 ebb175:
     trap heap_oob
@@ -1209,7 +1209,7 @@ ebb85:
     v706 = icmp.i32 ne v697, v705
     v707 = bint.i32 v706
     brnz v707, ebb86
-    fallthrough ebb176
+    jump ebb176
 
 ebb176:
     v708 = icmp.i32 ule v689, v555
@@ -1250,7 +1250,7 @@ ebb90(v756: i32, v757: i32):
     v761 = iconst.i32 0x4ffe
     v762 = icmp uge v758, v761
     brz v762, ebb91
-    fallthrough ebb177
+    jump ebb177
 
 ebb177:
     trap heap_oob
@@ -1307,7 +1307,7 @@ ebb0(v0: f64, v1: i64):
     v24 = icmp ult v22, v23
     v25 = bint.i32 v24
     brnz v25, ebb10
-    fallthrough ebb178
+    jump ebb178
 
 ebb178:
     v26 = iconst.i64 0x7fff_ffff_ffff_ffff
@@ -1323,14 +1323,14 @@ ebb10:
     v32 = icmp.i32 ult v22, v31
     v33 = bint.i32 v32
     brnz v33, ebb8
-    fallthrough ebb179
+    jump ebb179
 
 ebb179:
     v34 = iconst.i32 0x3ff0_a2b2
     v35 = icmp.i32 uge v22, v34
     v36 = bint.i32 v35
     brnz v36, ebb6
-    fallthrough ebb180
+    jump ebb180
 
 ebb180:
     v37 = iconst.i32 1
@@ -1349,7 +1349,7 @@ ebb9:
     v44 = bint.i32 v43
     v45 = bor v42, v44
     brnz v45, ebb7
-    fallthrough ebb181
+    jump ebb181
 
 ebb181:
     v141 = iconst.i64 0x7fe0_0000_0000_0000
@@ -1362,7 +1362,7 @@ ebb8:
     v49 = icmp.i32 ule v22, v48
     v50 = bint.i32 v49
     brnz v50, ebb3
-    fallthrough ebb182
+    jump ebb182
 
 ebb182:
     v51 = iconst.i32 0
@@ -1378,7 +1378,7 @@ ebb7:
     v55 = bint.i32 v54
     v56 = bor v55, v44
     brnz v56, ebb6
-    fallthrough ebb183
+    jump ebb183
 
 ebb183:
     v144 = iconst.i64 0xb6a0_0000_0000_0000
@@ -1417,12 +1417,12 @@ ebb6:
     v158 = iconst.i32 0x8000_0000
     v154 = icmp ne v76, v158
     brnz v154, ebb11
-    fallthrough ebb184
+    jump ebb184
 
 ebb184:
     v155 = fcmp uno v75, v75
     brz v155, ebb12
-    fallthrough ebb185
+    jump ebb185
 
 ebb185:
     trap bad_toint
@@ -1432,7 +1432,7 @@ ebb12:
     v156 = bitcast.f64 v159
     v157 = fcmp ge v156, v75
     brz v157, ebb13
-    fallthrough ebb186
+    jump ebb186
 
 ebb186:
     trap int_ovf
@@ -1491,7 +1491,7 @@ ebb4(v86: f64, v87: f64, v108: f64, v113: i32):
     v114 = icmp eq v113, v169
     v115 = bint.i32 v114
     brnz v115, ebb2(v12, v112)
-    fallthrough ebb187
+    jump ebb187
 
 ebb187:
     v116 = call fn0(v112, v113, v1)
diff --git a/cranelift/filetests/filetests/regalloc/coalescing-216.clif b/cranelift/filetests/filetests/regalloc/coalescing-216.clif
index 4276893fd2..a6a73cd2e5 100644
--- a/cranelift/filetests/filetests/regalloc/coalescing-216.clif
+++ b/cranelift/filetests/filetests/regalloc/coalescing-216.clif
@@ -14,7 +14,7 @@ ebb0(v0: i32, v1: i64):
 ebb4(v11: i64, v29: i64):
     v6 = iconst.i32 0
     brz v6, ebb14
-    fallthrough ebb15
+    jump ebb15
 
 ebb15:
     v9 = iconst.i32 -17
@@ -23,16 +23,16 @@ ebb15:
 
 ebb9(v10: i32):
     brnz v10, ebb8(v9, v11, v11)
-    fallthrough ebb16
+    jump ebb16
 
 ebb16:
     brz.i32 v9, ebb13
-    fallthrough ebb17
+    jump ebb17
 
 ebb17:
     v13 = iconst.i32 0
     brnz v13, ebb6(v11, v11)
-    fallthrough ebb18
+    jump ebb18
 
 ebb18:
     v14 = iconst.i32 0
@@ -52,7 +52,7 @@ ebb13:
 ebb10(v21: i64):
     v16 = iconst.i32 0
     brnz v16, ebb6(v21, v11)
-    fallthrough ebb19
+    jump ebb19
 
 ebb19:
     v17 = iconst.i32 0xffff_ffff_ffff_9f35
diff --git a/cranelift/filetests/filetests/regalloc/coloring-227.clif b/cranelift/filetests/filetests/regalloc/coloring-227.clif
index 511134ddaa..a469230b51 100644
--- a/cranelift/filetests/filetests/regalloc/coloring-227.clif
+++ b/cranelift/filetests/filetests/regalloc/coloring-227.clif
@@ -17,7 +17,7 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
                           ebb6:
 [RexOp1pu_id#b8]              v8 = iconst.i32 0
 [RexOp1tjccb#75]              brnz v8, ebb5
-[-]                           fallthrough ebb20
+[Op1jmpb#eb]                  jump ebb20
 
                           ebb20:
 [RexOp1pu_id#b8]              v9 = iconst.i32 0
@@ -30,12 +30,12 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 
                           ebb7:
 [RexOp1tjccb#74]              brz.i32 v17, ebb8
-[-]                           fallthrough ebb17
+[Op1jmpb#eb]                  jump ebb17
 
                           ebb17:
 [RexOp1pu_id#b8]              v18 = iconst.i32 0
 [RexOp1tjccb#74]              brz v18, ebb9
-[-]                           fallthrough ebb16
+[Op1jmpb#eb]                  jump ebb16
 
                           ebb16:
 [RexOp1pu_id#b8]              v21 = iconst.i32 0
@@ -72,12 +72,12 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
                           ebb2(v7: i32, v45: i32, v52: i32, v59: i32, v66: i32, v73: i32):
 [RexOp1pu_id#b8]              v44 = iconst.i32 0
 [RexOp1tjccb#74]              brz v44, ebb12
-[-]                           fallthrough ebb18
+[Op1jmpb#eb]                  jump ebb18
 
                           ebb18:
 [RexOp1pu_id#b8]              v50 = iconst.i32 11
 [RexOp1tjccb#74]              brz v50, ebb14
-[-]                           fallthrough ebb19
+[Op1jmpb#eb]                  jump ebb19
 
                           ebb19:
 [RexOp1umr#89]                v82 = uextend.i64 v52
diff --git a/cranelift/filetests/filetests/regalloc/ghost-param.clif b/cranelift/filetests/filetests/regalloc/ghost-param.clif
index af007179dc..f2a1883a0d 100644
--- a/cranelift/filetests/filetests/regalloc/ghost-param.clif
+++ b/cranelift/filetests/filetests/regalloc/ghost-param.clif
@@ -20,7 +20,7 @@ ebb5(v9: f64):
     v6 = iconst.i32 0
     v7 = iconst.i32 1
     brnz v7, ebb4(v6)
-    fallthrough ebb8
+    jump ebb8
 
 ebb8:
     v8 = iconst.i32 0
diff --git a/cranelift/filetests/filetests/regalloc/global-constraints.clif b/cranelift/filetests/filetests/regalloc/global-constraints.clif
index 4ea87eaca9..11a3dbef2c 100644
--- a/cranelift/filetests/filetests/regalloc/global-constraints.clif
+++ b/cranelift/filetests/filetests/regalloc/global-constraints.clif
@@ -14,7 +14,7 @@ ebb0(v0: i32):
     v4 = icmp_imm ne v0, 4
     v5 = icmp_imm sge v0, 5
     brnz v5, ebb1
-    fallthrough ebb2
+    jump ebb2
 
 ebb2:
     return
diff --git a/cranelift/filetests/filetests/regalloc/iterate.clif b/cranelift/filetests/filetests/regalloc/iterate.clif
index fe8c9cbaef..a18364e0b4 100644
--- a/cranelift/filetests/filetests/regalloc/iterate.clif
+++ b/cranelift/filetests/filetests/regalloc/iterate.clif
@@ -20,7 +20,7 @@ ebb0(v0: i64, v1: f32, v2: f64, v3: i32, v4: i32, v5: i64):
     v44 = iconst.i64 0
     v37 = icmp slt v0, v44
     brnz v37, ebb2
-    fallthrough ebb11
+    jump ebb11
 
 ebb11:
     v38 = fcvt_from_sint.f64 v0
@@ -44,7 +44,7 @@ ebb3(v15: f64):
     v54 = iconst.i64 0
     v47 = icmp.i64 slt v13, v54
     brnz v47, ebb4
-    fallthrough ebb12
+    jump ebb12
 
 ebb12:
     v48 = fcvt_from_sint.f64 v13
@@ -63,7 +63,7 @@ ebb5(v20: f64):
     v63 = iconst.i64 0
     v56 = icmp.i64 slt v7, v63
     brnz v56, ebb6
-    fallthrough ebb13
+    jump ebb13
 
 ebb13:
     v57 = fcvt_from_sint.f64 v7
@@ -91,12 +91,12 @@ ebb7(v21: f64):
     v69 = iconst.i64 0x8000_0000_0000_0000
     v65 = icmp ne v30, v69
     brnz v65, ebb8
-    fallthrough ebb15
+    jump ebb15
 
 ebb15:
     v66 = fcmp uno v29, v29
     brz v66, ebb9
-    fallthrough ebb16
+    jump ebb16
 
 ebb16:
     trap bad_toint
@@ -106,7 +106,7 @@ ebb9:
     v67 = bitcast.f64 v70
     v68 = fcmp gt v67, v29
     brz v68, ebb10
-    fallthrough ebb17
+    jump ebb17
 
 ebb17:
     trap int_ovf
@@ -136,7 +136,7 @@ ebb0(v0: i64):
     v7 = icmp uge v3, v6
     ; If we're unlucky, there are no ABCD registers available for v7 at this branch.
     brz v7, ebb2
-    fallthrough ebb4
+    jump ebb4
 
 ebb4:
     trap oob
@@ -149,7 +149,7 @@ ebb2:
     v11 = iadd v8, v10
     v12 = load.i64 v11
     brnz v12, ebb3
-    fallthrough ebb5
+    jump ebb5
 
 ebb5:
     trap icall_null
diff --git a/cranelift/filetests/filetests/regalloc/reload-208.clif b/cranelift/filetests/filetests/regalloc/reload-208.clif
index e89c6371ba..0c65ffb1e8 100644
--- a/cranelift/filetests/filetests/regalloc/reload-208.clif
+++ b/cranelift/filetests/filetests/regalloc/reload-208.clif
@@ -25,7 +25,7 @@ ebb0(v0: i64):
     v20 = iconst.i32 0x4ffe
     v16 = icmp uge v2, v20
     brz v16, ebb5
-    fallthrough ebb9
+    jump ebb9
 
 ebb9:
     trap heap_oob
@@ -47,7 +47,7 @@ ebb3(v7: i32):
     v26 = iconst.i32 0x4ffe
     v22 = icmp uge v7, v26
     brz v22, ebb6
-    fallthrough ebb10
+    jump ebb10
 
 ebb10:
     trap heap_oob
@@ -71,7 +71,7 @@ ebb2:
     v31 = iconst.i32 0x4ffe
     v27 = icmp uge v10, v31
     brz v27, ebb7
-    fallthrough ebb11
+    jump ebb11
 
 ebb11:
     trap heap_oob
@@ -87,7 +87,7 @@ ebb7:
     v36 = iconst.i32 0x4ffe
     v32 = icmp uge v13, v36
     brz v32, ebb8
-    fallthrough ebb12
+    jump ebb12
 
 ebb12:
     trap heap_oob
diff --git a/cranelift/filetests/filetests/regalloc/spill.clif b/cranelift/filetests/filetests/regalloc/spill.clif
index 8cf221d5af..525921a374 100644
--- a/cranelift/filetests/filetests/regalloc/spill.clif
+++ b/cranelift/filetests/filetests/regalloc/spill.clif
@@ -151,7 +151,7 @@ ebb0(v1: i32):
     ; check: v1 = spill
     v2 = iconst.i32 1
     brnz v1, ebb1(v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2)
-    fallthrough ebb2
+    jump ebb2
 
 ebb2:
     return v1

From 5672cd651ccbcae3755f37b567c42a97483296da Mon Sep 17 00:00:00 2001
From: superriva 
Date: Sun, 7 Jul 2019 14:52:27 +0300
Subject: [PATCH 2495/3084] Fix mem::uninitialized depricated since 1.38

Fix for `https://github.com/CraneStation/cranelift/issues/826`.
This fix will require 1.36 minimal version for all Cranelift. Right?

Update cranelift-simplejit/src/memory.rs

Co-Authored-By: bjorn3 

Update memory.rs
---
 cranelift/simplejit/src/memory.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/simplejit/src/memory.rs b/cranelift/simplejit/src/memory.rs
index 6b04193310..0730e086ef 100644
--- a/cranelift/simplejit/src/memory.rs
+++ b/cranelift/simplejit/src/memory.rs
@@ -28,10 +28,10 @@ impl PtrLen {
     /// suitably sized and aligned for memory protection.
     #[cfg(not(target_os = "windows"))]
     fn with_size(size: usize) -> Result {
+        let mut ptr = ptr::null_mut();
         let page_size = region::page::size();
         let alloc_size = round_up_to_page_size(size, page_size);
         unsafe {
-            let mut ptr: *mut libc::c_void = mem::uninitialized();
             let err = libc::posix_memalign(&mut ptr, page_size, alloc_size);
             if err == 0 {
                 Ok(Self {

From 4fef03f5f8e1d24528c4f22931f2f0f2219878ec Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 2 Jul 2019 18:24:50 +0200
Subject: [PATCH 2496/3084] [meta] Legalization: remove spurious assert;

This assert was added when porting legalization from Python to Rust and
doesn't hold when we have derived type variables.
---
 cranelift/codegen/meta/src/cdsl/type_inference.rs | 1 -
 1 file changed, 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/cdsl/type_inference.rs b/cranelift/codegen/meta/src/cdsl/type_inference.rs
index e3a1e9bb06..101cfa4104 100644
--- a/cranelift/codegen/meta/src/cdsl/type_inference.rs
+++ b/cranelift/codegen/meta/src/cdsl/type_inference.rs
@@ -364,7 +364,6 @@ impl TypeEnvironment {
 
             // Sanity check: translated constraints should refer only to real variables.
             for arg in constraint.typevar_args() {
-                assert!(vars_tv.contains(arg));
                 let arg_free_tv = arg.free_typevar();
                 assert!(arg_free_tv.is_none() || vars_tv.contains(&arg_free_tv.unwrap()));
             }

From 3545363006249b8748ae6177d9f200372d04325f Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 2 Jul 2019 18:29:41 +0200
Subject: [PATCH 2497/3084] Add ir::Types::lane_of as an alias of lane_type to
 be used in typevar constraints;

---
 cranelift/codegen/src/ir/instructions.rs | 4 ++--
 cranelift/codegen/src/ir/types.rs        | 9 ++++++++-
 2 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/src/ir/instructions.rs b/cranelift/codegen/src/ir/instructions.rs
index 8278888d7e..a94b0f3a60 100644
--- a/cranelift/codegen/src/ir/instructions.rs
+++ b/cranelift/codegen/src/ir/instructions.rs
@@ -498,7 +498,7 @@ enum OperandConstraint {
     /// This operand is the same type as the controlling type variable.
     Same,
 
-    /// This operand is `ctrlType.lane_type()`.
+    /// This operand is `ctrlType.lane_of()`.
     LaneOf,
 
     /// This operand is `ctrlType.as_bool()`.
@@ -527,7 +527,7 @@ impl OperandConstraint {
             Concrete(t) => Bound(t),
             Free(vts) => ResolvedConstraint::Free(TYPE_SETS[vts as usize]),
             Same => Bound(ctrl_type),
-            LaneOf => Bound(ctrl_type.lane_type()),
+            LaneOf => Bound(ctrl_type.lane_of()),
             AsBool => Bound(ctrl_type.as_bool()),
             HalfWidth => Bound(ctrl_type.half_width().expect("invalid type for half_width")),
             DoubleWidth => Bound(
diff --git a/cranelift/codegen/src/ir/types.rs b/cranelift/codegen/src/ir/types.rs
index 20bed0c4ce..9da5b971b5 100644
--- a/cranelift/codegen/src/ir/types.rs
+++ b/cranelift/codegen/src/ir/types.rs
@@ -44,10 +44,17 @@ impl Type {
         if self.0 < VECTOR_BASE {
             self
         } else {
-            Type(LANE_BASE | (self.0 & 0x0f))
+            Self(LANE_BASE | (self.0 & 0x0f))
         }
     }
 
+    /// The type transformation that returns the lane type of a type variable; it is just a
+    /// renaming of lane_type() to be used in context where we think in terms of type variable
+    /// transformations.
+    pub fn lane_of(self) -> Self {
+        self.lane_type()
+    }
+
     /// Get log_2 of the number of bits in a lane.
     pub fn log2_lane_bits(self) -> u8 {
         match self.lane_type() {

From f1222dce10bd605b092a23d0a913a2e83e8e76ec Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 2 Jul 2019 18:30:04 +0200
Subject: [PATCH 2498/3084] [meta] Legalization: emit typeof type variables for
 results in all the cases;

---
 cranelift/codegen/meta/src/gen_legalizer.rs | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index 545928db1e..f9cf8ba1ae 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -134,6 +134,21 @@ fn unwrap_inst(
                     .get(var_pool.get(def.defined_vars[0]).dst_def.unwrap())
                     .to_comment_string(var_pool)
             ));
+
+            fmt.line("let results = pos.func.dfg.inst_results(inst);");
+            for (i, &var_index) in def.defined_vars.iter().enumerate() {
+                let var = var_pool.get(var_index);
+                fmtln!(fmt, "let {} = &results[{}];", var.name, i);
+                if var.has_free_typevar() {
+                    fmtln!(
+                        fmt,
+                        "let typeof_{} = pos.func.dfg.value_type(*{});",
+                        var.name,
+                        var.name
+                    );
+                }
+            }
+
             replace_inst = true;
         } else {
             // Boring case: Detach the result values, capture them in locals.

From 15d8b95e7295549ebfd6ee9ef49c354d6ccbc238 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 4 Jul 2019 17:36:49 +0200
Subject: [PATCH 2499/3084] [meta] Legalization: remove spurious Option
 wrapping for type variables;

---
 cranelift/codegen/meta/src/gen_legalizer.rs | 23 ++++-----------------
 1 file changed, 4 insertions(+), 19 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index f9cf8ba1ae..f56da5fe53 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -187,15 +187,11 @@ fn build_derived_expr(tv: &TypeVar) -> String {
         Some(base) => base,
         None => {
             assert!(tv.name.starts_with("typeof_"));
-            return format!("Some({})", tv.name);
+            return format!("{}", tv.name);
         }
     };
     let base_expr = build_derived_expr(&base.type_var);
-    format!(
-        "{}.map(|t: crate::ir::Type| t.{}())",
-        base_expr,
-        base.derived_func.name()
-    )
+    format!("{}.{}()", base_expr, base.derived_func.name())
 }
 
 /// Emit rust code for the given check.
@@ -225,29 +221,18 @@ fn emit_runtime_typecheck<'a, 'b>(
         Constraint::Eq(tv1, tv2) => {
             fmtln!(
                 fmt,
-                "let predicate = predicate && match ({}, {}) {{",
+                "let predicate = predicate && {} == {};",
                 build_derived_expr(tv1),
                 build_derived_expr(tv2)
             );
-            fmt.indent(|fmt| {
-                fmt.line("(Some(a), Some(b)) => a == b,");
-                fmt.comment("On overflow, constraint doesn\'t apply");
-                fmt.line("_ => false,");
-            });
-            fmtln!(fmt, "};");
         }
         Constraint::WiderOrEq(tv1, tv2) => {
             fmtln!(
                 fmt,
-                "let predicate = predicate && match ({}, {}) {{",
+                "let predicate = predicate && {}.wider_or_equal({});",
                 build_derived_expr(tv1),
                 build_derived_expr(tv2)
             );
-            fmt.indent(|fmt| {
-                fmt.line("(Some(a), Some(b)) => a.wider_or_equal(b),");
-                fmt.comment("On overflow, constraint doesn\'t apply");
-                fmt.line("_ => false,");
-            });
             fmtln!(fmt, "};");
         }
     }

From cd4c28ad97c0998fd50fce35f3c2a4845c731ec7 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 4 Jul 2019 17:45:20 +0200
Subject: [PATCH 2500/3084] [meta] Legalization: Unprefix some module paths to
 make code neater;

---
 .../codegen/meta/src/cdsl/instructions.rs     | 36 +++++++++----------
 cranelift/codegen/meta/src/gen_legalizer.rs   | 13 ++++---
 cranelift/codegen/src/isa/x86/enc_tables.rs   | 19 +++++-----
 cranelift/codegen/src/legalizer/mod.rs        |  1 +
 4 files changed, 33 insertions(+), 36 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 9b6a354ca0..aa3262c66f 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -654,36 +654,32 @@ impl FormatPredicateNode {
     fn rust_predicate(&self) -> String {
         match &self.kind {
             FormatPredicateKind::IsEqual(arg) => {
-                format!("crate::predicates::is_equal({}, {})", self.member_name, arg)
+                format!("predicates::is_equal({}, {})", self.member_name, arg)
             }
             FormatPredicateKind::IsSignedInt(width, scale) => format!(
-                "crate::predicates::is_signed_int({}, {}, {})",
+                "predicates::is_signed_int({}, {}, {})",
                 self.member_name, width, scale
             ),
             FormatPredicateKind::IsUnsignedInt(width, scale) => format!(
-                "crate::predicates::is_unsigned_int({}, {}, {})",
+                "predicates::is_unsigned_int({}, {}, {})",
                 self.member_name, width, scale
             ),
-            FormatPredicateKind::IsZero32BitFloat => format!(
-                "crate::predicates::is_zero_32_bit_float({})",
-                self.member_name
-            ),
-            FormatPredicateKind::IsZero64BitFloat => format!(
-                "crate::predicates::is_zero_64_bit_float({})",
-                self.member_name
-            ),
+            FormatPredicateKind::IsZero32BitFloat => {
+                format!("predicates::is_zero_32_bit_float({})", self.member_name)
+            }
+            FormatPredicateKind::IsZero64BitFloat => {
+                format!("predicates::is_zero_64_bit_float({})", self.member_name)
+            }
             FormatPredicateKind::LengthEquals(num) => format!(
-                "crate::predicates::has_length_of({}, {}, func)",
+                "predicates::has_length_of({}, {}, func)",
                 self.member_name, num
             ),
-            FormatPredicateKind::IsColocatedFunc => format!(
-                "crate::predicates::is_colocated_func({}, func)",
-                self.member_name,
-            ),
-            FormatPredicateKind::IsColocatedData => format!(
-                "crate::predicates::is_colocated_data({}, func)",
-                self.member_name
-            ),
+            FormatPredicateKind::IsColocatedFunc => {
+                format!("predicates::is_colocated_func({}, func)", self.member_name,)
+            }
+            FormatPredicateKind::IsColocatedData => {
+                format!("predicates::is_colocated_data({}, func)", self.member_name)
+            }
         }
     }
 }
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index f56da5fe53..554ae13e22 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -51,7 +51,7 @@ fn unwrap_inst(
 
     fmtln!(
         fmt,
-        "let ({}, predicate) = if let crate::ir::InstructionData::{} {{",
+        "let ({}, predicate) = if let ir::InstructionData::{} {{",
         arg_names,
         iform.name
     );
@@ -407,17 +407,16 @@ fn gen_transform_group<'a>(
     // Function arguments.
     fmtln!(fmt, "pub fn {}(", group.name);
     fmt.indent(|fmt| {
-        fmt.line("inst: crate::ir::Inst,");
-        fmt.line("func: &mut crate::ir::Function,");
-        fmt.line("cfg: &mut crate::flowgraph::ControlFlowGraph,");
-        fmt.line("isa: &dyn crate::isa::TargetIsa,");
+        fmt.line("inst: ir::Inst,");
+        fmt.line("func: &mut ir::Function,");
+        fmt.line("cfg: &mut ControlFlowGraph,");
+        fmt.line("isa: &dyn TargetIsa,");
     });
     fmtln!(fmt, ") -> bool {");
 
     // Function body.
     fmt.indent(|fmt| {
-        fmt.line("use crate::ir::InstBuilder;");
-        fmt.line("use crate::cursor::{Cursor, FuncCursor};");
+        fmt.line("use ir::InstBuilder;");
         fmt.line("let mut pos = FuncCursor::new(func).at_inst(inst);");
         fmt.line("pos.use_srcloc(inst);");
 
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 474ef3c544..832058c1e0 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -6,12 +6,13 @@ use crate::cursor::{Cursor, FuncCursor};
 use crate::flowgraph::ControlFlowGraph;
 use crate::ir::condcodes::{FloatCC, IntCC};
 use crate::ir::{self, Function, Inst, InstBuilder};
-use crate::isa;
 use crate::isa::constraints::*;
 use crate::isa::enc_tables::*;
 use crate::isa::encoding::base_size;
 use crate::isa::encoding::RecipeSizing;
 use crate::isa::RegUnit;
+use crate::isa::{self, TargetIsa};
+use crate::predicates;
 use crate::regalloc::RegDiversions;
 
 include!(concat!(env!("OUT_DIR"), "/encoding-x86.rs"));
@@ -115,7 +116,7 @@ fn expand_sdivrem(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    isa: &dyn isa::TargetIsa,
+    isa: &dyn TargetIsa,
 ) {
     let (x, y, is_srem) = match func.dfg[inst] {
         ir::InstructionData::Binary {
@@ -225,7 +226,7 @@ fn expand_udivrem(
     inst: ir::Inst,
     func: &mut ir::Function,
     _cfg: &mut ControlFlowGraph,
-    isa: &dyn isa::TargetIsa,
+    isa: &dyn TargetIsa,
 ) {
     let (x, y, is_urem) = match func.dfg[inst] {
         ir::InstructionData::Binary {
@@ -278,7 +279,7 @@ fn expand_minmax(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &dyn isa::TargetIsa,
+    _isa: &dyn TargetIsa,
 ) {
     let (x, y, x86_opc, bitwise_opc) = match func.dfg[inst] {
         ir::InstructionData::Binary {
@@ -370,7 +371,7 @@ fn expand_fcvt_from_uint(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &dyn isa::TargetIsa,
+    _isa: &dyn TargetIsa,
 ) {
     let x;
     match func.dfg[inst] {
@@ -441,7 +442,7 @@ fn expand_fcvt_to_sint(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &dyn isa::TargetIsa,
+    _isa: &dyn TargetIsa,
 ) {
     use crate::ir::immediates::{Ieee32, Ieee64};
 
@@ -536,7 +537,7 @@ fn expand_fcvt_to_sint_sat(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &dyn isa::TargetIsa,
+    _isa: &dyn TargetIsa,
 ) {
     use crate::ir::immediates::{Ieee32, Ieee64};
 
@@ -655,7 +656,7 @@ fn expand_fcvt_to_uint(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &dyn isa::TargetIsa,
+    _isa: &dyn TargetIsa,
 ) {
     use crate::ir::immediates::{Ieee32, Ieee64};
 
@@ -736,7 +737,7 @@ fn expand_fcvt_to_uint_sat(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &dyn isa::TargetIsa,
+    _isa: &dyn TargetIsa,
 ) {
     use crate::ir::immediates::{Ieee32, Ieee64};
 
diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index b7aa287835..97211e2243 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -19,6 +19,7 @@ use crate::flowgraph::ControlFlowGraph;
 use crate::ir::types::I32;
 use crate::ir::{self, InstBuilder, MemFlags};
 use crate::isa::TargetIsa;
+use crate::predicates;
 use crate::timing;
 
 mod boundary;

From 56f6908020ae5ceba3ff5ec4e4d4cd887606bdba Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 4 Jul 2019 17:48:55 +0200
Subject: [PATCH 2501/3084] [meta] Legalization: Don't generate a variable for
 replaced instructions;

(since they're unused)
---
 cranelift/codegen/meta/src/gen_legalizer.rs | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index 554ae13e22..b09a06a333 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -290,8 +290,7 @@ fn emit_dst_inst(def: &Def, def_pool: &DefPool, var_pool: &VarPool, fmt: &mut Fo
             // Unwrapping would have left the results intact.  Replace the whole instruction.
             fmtln!(
                 fmt,
-                "let {} = pos.func.dfg.replace(inst).{};",
-                defined_vars,
+                "pos.func.dfg.replace(inst).{};",
                 def.apply.rust_builder(&def.defined_vars, var_pool)
             );
 

From f11fc34066ec5c17f32c19ec84ed212d7aab4cee Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 9 Jul 2019 11:29:48 +0200
Subject: [PATCH 2502/3084] Build fix: add crates::predicates to the Riscv
 enc_tables file;

---
 cranelift/codegen/src/isa/riscv/enc_tables.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cranelift/codegen/src/isa/riscv/enc_tables.rs b/cranelift/codegen/src/isa/riscv/enc_tables.rs
index 19488003a3..76184ad727 100644
--- a/cranelift/codegen/src/isa/riscv/enc_tables.rs
+++ b/cranelift/codegen/src/isa/riscv/enc_tables.rs
@@ -6,6 +6,7 @@ use crate::isa;
 use crate::isa::constraints::*;
 use crate::isa::enc_tables::*;
 use crate::isa::encoding::{base_size, RecipeSizing};
+use crate::predicates;
 
 // Include the generated encoding tables:
 // - `LEVEL1_RV32`

From 1963c223b115b490499c72b7137f8ae4b4ec8098 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Tue, 9 Jul 2019 16:02:49 +0200
Subject: [PATCH 2503/3084] Legalize trapz/trapnz to a BB-like format.

---
 cranelift/codegen/src/legalizer/mod.rs        | 32 ++++++++---
 .../filetests/isa/x86/legalize-custom.clif    |  4 ++
 .../filetests/isa/x86/legalize-heaps.clif     | 57 ++++++++++++-------
 .../filetests/isa/x86/legalize-memory.clif    |  2 +
 .../filetests/isa/x86/legalize-tables.clif    | 33 +++++++----
 5 files changed, 87 insertions(+), 41 deletions(-)

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 97211e2243..be0c5c5da8 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -150,29 +150,45 @@ fn expand_cond_trap(
     // Split the EBB after `inst`:
     //
     //     trapnz arg
+    //     ..
     //
     // Becomes:
     //
-    //     brz arg, new_ebb
-    //     trap
-    //   new_ebb:
+    //     brz arg, new_ebb_resume
+    //     jump new_ebb_trap
     //
+    //   new_ebb_trap:
+    //     trap
+    //
+    //   new_ebb_resume:
+    //     ..
     let old_ebb = func.layout.pp_ebb(inst);
-    let new_ebb = func.dfg.make_ebb();
+    let new_ebb_trap = func.dfg.make_ebb();
+    let new_ebb_resume = func.dfg.make_ebb();
+
+    // Replace trap instruction by the inverted condition.
     if trapz {
-        func.dfg.replace(inst).brnz(arg, new_ebb, &[]);
+        func.dfg.replace(inst).brnz(arg, new_ebb_resume, &[]);
     } else {
-        func.dfg.replace(inst).brz(arg, new_ebb, &[]);
+        func.dfg.replace(inst).brz(arg, new_ebb_resume, &[]);
     }
 
+    // Add jump instruction after the inverted branch.
     let mut pos = FuncCursor::new(func).after_inst(inst);
     pos.use_srcloc(inst);
+    pos.ins().jump(new_ebb_trap, &[]);
+
+    // Insert the new label and the unconditional trap terminator.
+    pos.insert_ebb(new_ebb_trap);
     pos.ins().trap(code);
-    pos.insert_ebb(new_ebb);
+
+    // Insert the new label and resume the execution when the trap fails.
+    pos.insert_ebb(new_ebb_resume);
 
     // Finally update the CFG.
     cfg.recompute_ebb(pos.func, old_ebb);
-    cfg.recompute_ebb(pos.func, new_ebb);
+    cfg.recompute_ebb(pos.func, new_ebb_resume);
+    cfg.recompute_ebb(pos.func, new_ebb_trap);
 }
 
 /// Jump tables.
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-custom.clif b/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
index 667c08804c..51e7ca487d 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
@@ -33,6 +33,8 @@ ebb0(v1: i32):
     return
     ; check: ebb0(v1: i32
     ; check: brnz v2, $(new=$EBB)
+    ; check: jump $(trap=$EBB)
+    ; check: $trap:
     ; nextln: trap user7
     ; check: $new:
     ; nextln: return
@@ -45,6 +47,8 @@ ebb0(v1: i32):
     return
     ; check: ebb0(v1: i32
     ; check: brz v2, $(new=$EBB)
+    ; check: jump $(trap=$EBB)
+    ; check: $trap:
     ; nextln: trap user9
     ; check: $new:
     ; nextln: return
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-heaps.clif b/cranelift/filetests/filetests/isa/x86/legalize-heaps.clif
index 5f60c7b43f..c2f1ccb2d9 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-heaps.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-heaps.clif
@@ -2,6 +2,7 @@ test legalizer
 target x86_64
 
 ; Test legalization for various forms of heap addresses.
+; regex: EBB=ebb\d+
 
 function %heap_addrs(i32, i64, i64 vmctx) {
     gv4 = vmctx
@@ -37,9 +38,11 @@ ebb0(v0: i32, v1: i64, v3: i64):
 
     v5 = heap_addr.i64 heap1, v0, 0
     ; check:         v14 = icmp_imm ugt v0, 0x0001_0000
-    ; check:         brz v14, ebb1
-    ; check:         trap heap_oob
-    ; check:     ebb1:
+    ; check:         brz v14, $(resume_1=$EBB)
+    ; nextln:        jump $(trap_1=$EBB)
+    ; check:     $trap_1:
+    ; nextln:        trap heap_oob
+    ; check:     $resume_1:
     ; check:         v15 = uextend.i64 v0
     ; check:         v16 = iadd_imm.i64 v3, 64
     ; check:         v5 = iadd v16, v15
@@ -47,17 +50,21 @@ ebb0(v0: i32, v1: i64, v3: i64):
     v6 = heap_addr.i64 heap2, v1, 0
     ; check:         v19 = iconst.i64 0x0001_0000_0000
     ; check:         v17 = icmp.i64 ugt v1, v19
-    ; check:         brz v17, ebb2
-    ; check:         trap heap_oob
-    ; check:     ebb2:
+    ; check:         brz v17, $(resume_2=$EBB)
+    ; nextln:        jump $(trap_2=$EBB)
+    ; check:     $trap_2:
+    ; nextln:        trap heap_oob
+    ; check:     $resume_2:
     ; check:         v18 = iadd_imm.i64 v3, 64
     ; check:         v6 = iadd v18, v1
 
     v7 = heap_addr.i64 heap3, v1, 0
     ; check:         v20 = icmp_imm.i64 ugt v1, 0x0001_0000
-    ; check:         brz v20, ebb3
-    ; check:         trap heap_oob
-    ; check:     ebb3:
+    ; check:         brz v20, $(resume_3=$EBB)
+    ; nextln:        jump $(trap_3=$EBB)
+    ; check:     $trap_3:
+    ; nextln:        trap heap_oob
+    ; check:     $resume_3:
     ; check:         v21 = iadd_imm.i64 v3, 64
     ; check:         v7 = iadd v21, v1
 
@@ -65,9 +72,11 @@ ebb0(v0: i32, v1: i64, v3: i64):
     ; check:         v22 = load.i32 notrap aligned v3+88
     ; check:         v23 = iadd_imm v22, 0
     ; check:         v24 = icmp.i32 ugt v0, v23
-    ; check:         brz v24, ebb4
-    ; check:         trap heap_oob
-    ; check:     ebb4:
+    ; check:         brz v24, $(resume_4=$EBB)
+    ; nextln:        jump $(trap_4=$EBB)
+    ; check:     $trap_4:
+    ; nextln:        trap heap_oob
+    ; check:     $resume_4:
     ; check:         v25 = uextend.i64 v0
     ; check:         v26 = iadd_imm.i64 v3, 72
     ; check:         v8 = iadd v26, v25
@@ -76,9 +85,11 @@ ebb0(v0: i32, v1: i64, v3: i64):
     ; check:         v27 = load.i32 notrap aligned v3+88
     ; check:         v28 = iadd_imm v27, 0
     ; check:         v29 = icmp.i32 ugt v0, v28
-    ; check:         brz v29, ebb5
-    ; check:         trap heap_oob
-    ; check:     ebb5:
+    ; check:         brz v29, $(resume_5=$EBB)
+    ; nextln:        jump $(trap_5=$EBB)
+    ; check:     $trap_5:
+    ; nextln:        trap heap_oob
+    ; check:     $resume_5:
     ; check:         v30 = uextend.i64 v0
     ; check:         v31 = iadd_imm.i64 v3, 72
     ; check:         v9 = iadd v31, v30
@@ -87,9 +98,11 @@ ebb0(v0: i32, v1: i64, v3: i64):
     ; check:         v32 = iadd_imm.i64 v3, 80
     ; check:         v33 = iadd_imm v32, 0
     ; check:         v34 = icmp.i64 ugt v1, v33
-    ; check:         brz v34, ebb6
-    ; check:         trap heap_oob
-    ; check:     ebb6:
+    ; check:         brz v34, $(resume_6=$EBB)
+    ; nextln:        jump $(trap_6=$EBB)
+    ; check:     $trap_6:
+    ; nextln:        trap heap_oob
+    ; check:     $resume_6:
     ; check:         v35 = iadd_imm.i64 v3, 72
     ; check:         v10 = iadd v35, v1
 
@@ -97,9 +110,11 @@ ebb0(v0: i32, v1: i64, v3: i64):
     ; check:         v36 = iadd_imm.i64 v3, 80
     ; check:         v37 = iadd_imm v36, 0
     ; check:         v38 = icmp.i64 ugt v1, v37
-    ; check:         brz v38, ebb7
-    ; check:         trap heap_oob
-    ; check:     ebb7:
+    ; check:         brz v38, $(resume_7=$EBB)
+    ; nextln:        jump $(trap_7=$EBB)
+    ; check:     $trap_7:
+    ; nextln:        trap heap_oob
+    ; check:     $resume_7:
     ; check:         v39 = iadd_imm.i64 v3, 72
     ; check:         v11 = iadd v39, v1
 
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-memory.clif b/cranelift/filetests/filetests/isa/x86/legalize-memory.clif
index eb24523b14..7b506d3876 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-memory.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-memory.clif
@@ -100,6 +100,8 @@ ebb0(v0: i32, v999: i64):
     ; Boundscheck code
     ; check: $(oob=$V) = icmp
     ; nextln: brz $oob, $(ok=$EBB)
+    ; nextln: jump $(trap_oob=$EBB)
+    ; check: $trap_oob:
     ; nextln: trap heap_oob
     ; check: $ok:
     ; Checks here are assuming that no pipehole opts fold the load offsets.
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-tables.clif b/cranelift/filetests/filetests/isa/x86/legalize-tables.clif
index 5995c230e0..762f8a1038 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-tables.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-tables.clif
@@ -2,6 +2,7 @@ test legalizer
 target x86_64
 
 ; Test legalization for various forms of table addresses.
+; regex: EBB=ebb\d+
 
 function %table_addrs(i32, i64, i64 vmctx) {
     gv4 = vmctx
@@ -23,9 +24,11 @@ ebb0(v0: i32, v1: i64, v3: i64):
     v4 = table_addr.i64 table0, v0, +0
     ; check:         v8 = load.i32 notrap aligned v3+88
     ; check:         v9 = icmp uge v0, v8
-    ; check:         brz v9, ebb1
-    ; check:         trap table_oob
-    ; check:     ebb1:
+    ; check:         brz v9, $(resume_1=$EBB)
+    ; nextln:        jump $(trap_1=$EBB)
+    ; check:     $trap_1:
+    ; nextln:        trap table_oob
+    ; check:     $resume_1:
     ; check:         v10 = uextend.i64 v0
     ; check:         v11 = iadd_imm.i64 v3, 72
     ; check:         v4 = iadd v11, v10
@@ -33,9 +36,11 @@ ebb0(v0: i32, v1: i64, v3: i64):
     v5 = table_addr.i64 table1, v0, +0
     ; check:         v12 = load.i32 notrap aligned v3+88
     ; check:         v13 = icmp.i32 uge v0, v12
-    ; check:         brz v13, ebb2
-    ; check:         trap table_oob
-    ; check:     ebb2:
+    ; check:         brz v13, $(resume_2=$EBB)
+    ; nextln:        jump $(trap_2=$EBB)
+    ; check:     $trap_2:
+    ; nextln:        trap table_oob
+    ; check:     $resume_2:
     ; check:         v14 = uextend.i64 v0
     ; check:         v15 = iadd_imm.i64 v3, 72
     ; check:         v16 = ishl_imm v14, 4
@@ -44,18 +49,22 @@ ebb0(v0: i32, v1: i64, v3: i64):
     v6 = table_addr.i64 table2, v1, +0
     ; check:         v17 = iadd_imm.i64 v3, 80
     ; check:         v18 = icmp.i64 uge v1, v17
-    ; check:         brz v18, ebb3
-    ; check:         trap table_oob
-    ; check:     ebb3:
+    ; check:         brz v18, $(resume_3=$EBB)
+    ; nextln:        jump $(trap_3=$EBB)
+    ; check:     $trap_3:
+    ; nextln:        trap table_oob
+    ; check:     $resume_3:
     ; check:         v19 = iadd_imm.i64 v3, 72
     ; check:         v6 = iadd v19, v1
 
     v7 = table_addr.i64 table3, v1, +0
     ; check:         v20 = iadd_imm.i64 v3, 80
     ; check:         v21 = icmp.i64 uge v1, v20
-    ; check:         brz v21, ebb4
-    ; check:         trap table_oob
-    ; check:     ebb4:
+    ; check:         brz v21, $(resume_4=$EBB)
+    ; nextln:        jump $(trap_4=$EBB)
+    ; check:     $trap_4:
+    ; nextln:        trap table_oob
+    ; check:     $resume_4:
     ; check:         v22 = iadd_imm.i64 v3, 72
     ; check:         v23 = ishl_imm.i64 v1, 4
     ; check:         v7 = iadd v22, v23

From 3ac7466cabce87ccc6b7b4163ba6f0f541ddbd88 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Tue, 9 Jul 2019 16:48:54 +0200
Subject: [PATCH 2504/3084] Legalize br_table to a BB-like format.

---
 cranelift/codegen/src/legalizer/mod.rs        | 39 ++++++++++++++++++-
 .../filetests/isa/x86/legalize-br-table.clif  | 10 +++++
 .../filetests/legalizer/br_table_cond.clif    | 24 ++++++++++++
 3 files changed, 71 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index be0c5c5da8..0905fb05c5 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -224,10 +224,29 @@ fn expand_br_table_jt(
         _ => panic!("Expected br_table: {}", func.dfg.display_inst(inst, None)),
     };
 
+    // Rewrite:
+    //
+    //     br_table $idx, default_ebb, $jt
+    //
+    // To:
+    //
+    //     $oob = ifcmp_imm $idx, len($jt)
+    //     brif uge $oob, default_ebb
+    //     jump fallthrough_ebb
+    //
+    //   fallthrough_ebb:
+    //     $base = jump_table_base.i64 $jt
+    //     $rel_addr = jump_table_entry.i64 $idx, $base, 4, $jt
+    //     $addr = iadd $base, $rel_addr
+    //     indirect_jump_table_br $addr, $jt
+
     let table_size = func.jump_tables[table].len();
     let addr_ty = isa.pointer_type();
     let entry_ty = I32;
 
+    let ebb = func.layout.pp_ebb(inst);
+    let jump_table_ebb = func.dfg.make_ebb();
+
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
 
@@ -237,6 +256,8 @@ fn expand_br_table_jt(
         .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, arg, table_size as i64);
 
     pos.ins().brnz(oob, default_ebb, &[]);
+    pos.ins().jump(jump_table_ebb, &[]);
+    pos.insert_ebb(jump_table_ebb);
 
     let base_addr = pos.ins().jump_table_base(addr_ty, table);
     let entry = pos
@@ -246,9 +267,9 @@ fn expand_br_table_jt(
     let addr = pos.ins().iadd(base_addr, entry);
     pos.ins().indirect_jump_table_br(addr, table);
 
-    let ebb = pos.current_ebb().unwrap();
     pos.remove_inst();
     cfg.recompute_ebb(pos.func, ebb);
+    cfg.recompute_ebb(pos.func, jump_table_ebb);
 }
 
 /// Expand br_table to series of conditionals.
@@ -270,8 +291,15 @@ fn expand_br_table_conds(
         _ => panic!("Expected br_table: {}", func.dfg.display_inst(inst, None)),
     };
 
+    let ebb = func.layout.pp_ebb(inst);
+
     // This is a poor man's jump table using just a sequence of conditional branches.
     let table_size = func.jump_tables[table].len();
+    let mut cond_failed_ebb = std::vec::Vec::with_capacity(table_size - 1);
+    for _ in 0..table_size - 1 {
+        cond_failed_ebb.push(func.dfg.make_ebb());
+    }
+
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
 
@@ -279,14 +307,21 @@ fn expand_br_table_conds(
         let dest = pos.func.jump_tables[table].as_slice()[i];
         let t = pos.ins().icmp_imm(IntCC::Equal, arg, i as i64);
         pos.ins().brnz(t, dest, &[]);
+        // Jump to the next case.
+        if i < table_size - 1 {
+            pos.ins().jump(cond_failed_ebb[i], &[]);
+            pos.insert_ebb(cond_failed_ebb[i]);
+        }
     }
 
     // `br_table` jumps to the default destination if nothing matches
     pos.ins().jump(default_ebb, &[]);
 
-    let ebb = pos.current_ebb().unwrap();
     pos.remove_inst();
     cfg.recompute_ebb(pos.func, ebb);
+    for failed_ebb in cond_failed_ebb.into_iter() {
+        cfg.recompute_ebb(pos.func, failed_ebb);
+    }
 }
 
 /// Expand the select instruction.
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
index 959d3028b2..fb0aec0606 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
@@ -1,6 +1,8 @@
 test compile
 
 target x86_64
+; regex: V=v\d+
+; regex: EBB=ebb\d+
 
 function u0:0(i64) system_v {
     ss0 = explicit_slot 1
@@ -10,6 +12,14 @@ ebb0(v0: i64):
     v1 = stack_addr.i64 ss0
     v2 = load.i8 v1
     br_table v2, ebb2, jt0
+; check:     $(oob=$V) = ifcmp_imm $(idx=$V), 1
+; nextln:    brif uge $oob, ebb2
+; nextln:    fallthrough $(inb=$EBB)
+; check:   $inb:
+; nextln:    $(base=$V) = jump_table_base.i64 jt0
+; nextln:    $(rel_addr=$V) = jump_table_entry.i64 $idx, $base, 4, jt0
+; nextln:    $(addr=$V) = iadd $base, $rel_addr
+; nextln:    indirect_jump_table_br $addr, jt0
 
 ebb2:
     jump ebb1
diff --git a/cranelift/filetests/filetests/legalizer/br_table_cond.clif b/cranelift/filetests/filetests/legalizer/br_table_cond.clif
index 5b7beb3253..c88becab08 100644
--- a/cranelift/filetests/filetests/legalizer/br_table_cond.clif
+++ b/cranelift/filetests/filetests/legalizer/br_table_cond.clif
@@ -4,6 +4,8 @@ set jump_tables_enabled=false
 target x86_64
 
 ; Test that when jump_tables_enables is false, all jump tables are eliminated.
+; regex: V=v\d+
+; regex: EBB=ebb\d+
 
 function u0:0(i64 vmctx) baldrdash {
     gv0 = vmctx
@@ -20,10 +22,32 @@ ebb5:
     trapnz v2, interrupt
     v3 = iconst.i32 0
     br_table v3, ebb3, jt0
+; check:  ebb5:
+; check:    $(val0=$V) = iconst.i32 0
+; nextln:   $(cmp0=$V) = icmp_imm eq $val0, 0
+; nextln:   brnz $cmp0, ebb2
+; nextln:   jump $(fail0=$EBB)
+; check:  $fail0:
+; nextln:   $(cmp1=$V) = icmp_imm.i32 eq $val0, 1
+; nextln:   brnz $cmp1, ebb2
+; nextln:   jump $(fail1=$EBB)
+; check:  $fail1:
+; nextln:   $(cmp2=$V) = icmp_imm.i32 eq $val0, 2
+; nextln:   brnz $cmp2, ebb7
+; nextln:   jump ebb3
 
 ebb7:
     v4 = iconst.i32 0
     br_table v4, ebb3, jt1
+; check:  ebb7:
+; check:    $(val1=$V) = iconst.i32 0
+; nextln:   $(cmp3=$V) = icmp_imm eq $val1, 0
+; nextln:   brnz $cmp3, ebb8
+; nextln:   jump $(fail3=$EBB)
+; check:  $fail3:
+; nextln:   $(cmp4=$V) = icmp_imm.i32 eq $val1, 1
+; nextln:   brnz $cmp4, ebb8
+; nextln:   jump ebb3
 
 ebb8:
     jump ebb5

From 237d48477ab8b0e666dace3081e16b335aa00219 Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Tue, 9 Jul 2019 10:01:29 -0600
Subject: [PATCH 2505/3084] Fix an outdated comment referring to
 `FunctionLayout` instead of `Layout`

---
 cranelift/codegen/src/ir/dfg.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index d26abaa3e7..1467ab23b3 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -25,7 +25,7 @@ use std::collections::HashMap;
 /// instruction results or EBB parameters.
 ///
 /// The layout of EBBs in the function and of instructions in each EBB is recorded by the
-/// `FunctionLayout` data structure which form the other half of the function representation.
+/// `Layout` data structure which forms the other half of the function representation.
 ///
 #[derive(Clone)]
 pub struct DataFlowGraph {

From 41a3d88b37b727b1a78e684576934a1d61ceaf94 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 9 Jul 2019 17:59:36 +0200
Subject: [PATCH 2506/3084] Fixes #837: Use an u64 comparison instead of a
 usize comparison in meta;

---
 cranelift/codegen/meta/src/gen_encodings.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/gen_encodings.rs b/cranelift/codegen/meta/src/gen_encodings.rs
index aff6d32d5d..14ea392671 100644
--- a/cranelift/codegen/meta/src/gen_encodings.rs
+++ b/cranelift/codegen/meta/src/gen_encodings.rs
@@ -46,6 +46,7 @@
 
 use std::collections::btree_map;
 use std::collections::{BTreeMap, HashMap, HashSet};
+use std::convert::TryFrom;
 use std::iter::FromIterator;
 
 use cranelift_entity::EntityRef;
@@ -913,7 +914,7 @@ fn emit_tables(defs: &SharedDefinitions, isa: &TargetIsa, fmt: &mut Formatter) {
         if length <= 0x10000 {
             "u16"
         } else {
-            assert!(length <= 0x100000000, "table too big!");
+            assert!(u32::try_from(length).is_ok(), "table too big!");
             "u32"
         }
     };

From 59f6c81e4f7c5283fef877fc70ce47aea52d0070 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 9 Jul 2019 12:31:58 -0700
Subject: [PATCH 2507/3084] Bump version to 0.34.0

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    |  8 ++++----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 6076d978cd..6fe8e44cda 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.33.0"
+version = "0.34.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.33.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.33.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.33.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.33.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.33.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.33.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.33.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.33.0" }
-cranelift-module = { path = "cranelift-module", version = "0.33.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.33.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.33.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.33.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.33.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.34.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.34.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.34.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.34.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.34.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.34.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.34.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.34.0" }
+cranelift-module = { path = "cranelift-module", version = "0.34.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.34.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.34.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.34.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.34.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 9cc6c7a18a..2308f5e827 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.33.0"
+version = "0.34.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.33.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.34.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 5fcb8f96db..5cde9d47df 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.33.0"
+version = "0.34.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.33.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.33.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.34.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.34.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -26,7 +26,7 @@ log = { version = "0.4.6", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.33.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.34.0", default-features = false }
 
 [features]
 default = ["std", "x86", "arm32", "arm64", "riscv"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 56f5190830..574e40db24 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.33.0"
+version = "0.34.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.33.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.34.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index bb40aaa17b..d2dc9ab91d 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.33.0"
+version = "0.34.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index fd205a61f7..cdc79f2397 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.33.0"
+version = "0.34.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.33.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.34.0" }
 faerie = "0.10.0"
 goblin = "0.0.22"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 5e28a630e9..b16271017c 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.33.0"
+version = "0.34.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../cranelift-reader", version = "0.33.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.33.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../cranelift-reader", version = "0.34.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.34.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index ec7424c84a..97a706122b 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.33.0"
+version = "0.34.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index e3d4081ba7..b62c947fb0 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.33.0"
+version = "0.34.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.33.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.34.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 4a0904ebc1..8e01226676 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.33.0"
+version = "0.34.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index faec5baa05..5a8c53259c 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.33.0"
+version = "0.34.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.33.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.34.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index aed801e351..3a1d78e715 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.33.0"
+version="0.34.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 1c911f9612..ea226ec0fd 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.33.0"
+version = "0.34.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0" }
 target-lexicon = "0.4.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index c34d09d92a..8516812a16 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.33.0"
+version = "0.34.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.33.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.34.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 6811538631..a8edcd3328 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.33.0"
+version = "0.34.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.33.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.33.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.34.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.34.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -22,9 +22,9 @@ target-lexicon = { version = "0.4.0" }
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.33.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.33.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.33.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.34.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.34.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.34.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index c4ddf0db2f..fc3089268f 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.33.0"
+version = "0.34.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.33.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.34.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 3a9d9ebc98..53a64e7896 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.33.0"
+version = "0.34.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.32.1", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.33.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.33.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.33.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.34.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.34.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 84a6795873fdb4187ff48a777309f6eb7ee711cd Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 8 Jul 2019 13:00:02 +0200
Subject: [PATCH 2508/3084] [meta] Riscv: add back stacknull encodings for
 copy_nop;

---
 .../codegen/meta/src/isa/riscv/encodings.rs     | 17 ++++++++++++++++-
 cranelift/codegen/meta/src/isa/riscv/recipes.rs |  8 ++++++++
 2 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/isa/riscv/encodings.rs b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
index 3f1c199bc0..88a62ac74e 100644
--- a/cranelift/codegen/meta/src/isa/riscv/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
@@ -7,7 +7,8 @@ use crate::cdsl::recipes::{EncodingRecipeNumber, Recipes};
 use crate::cdsl::settings::SettingGroup;
 
 use crate::shared::types::Bool::B1;
-use crate::shared::types::Int::{I32, I64};
+use crate::shared::types::Float::{F32, F64};
+use crate::shared::types::Int::{I16, I32, I64, I8};
 use crate::shared::Definitions as SharedDefinitions;
 
 use super::recipes::RecipeGroup;
@@ -119,6 +120,7 @@ pub fn define<'defs>(
     let call = shared.by_name("call");
     let call_indirect = shared.by_name("call_indirect");
     let copy = shared.by_name("copy");
+    let copy_nop = shared.by_name("copy_nop");
     let fill = shared.by_name("fill");
     let iadd = shared.by_name("iadd");
     let iadd_imm = shared.by_name("iadd_imm");
@@ -153,6 +155,7 @@ pub fn define<'defs>(
     let r_rshamt = recipes.by_name("Rshamt");
     let r_sb = recipes.by_name("SB");
     let r_sb_zero = recipes.by_name("SBzero");
+    let r_stacknull = recipes.by_name("stacknull");
     let r_u = recipes.by_name("U");
     let r_uj = recipes.by_name("UJ");
     let r_uj_call = recipes.by_name("UJcall");
@@ -379,5 +382,17 @@ pub fn define<'defs>(
     e.add32(enc(regmove.bind(B1), r_irmov, opimm_bits(0b000, 0)));
     e.add64(enc(regmove.bind(B1), r_irmov, opimm_bits(0b000, 0)));
 
+    // Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
+    // into a no-op.
+    // The same encoding is generated for both the 64- and 32-bit architectures.
+    for &ty in &[I64, I32, I16, I8] {
+        e.add32(enc(copy_nop.bind(ty), r_stacknull, 0));
+        e.add64(enc(copy_nop.bind(ty), r_stacknull, 0));
+    }
+    for &ty in &[F64, F32] {
+        e.add32(enc(copy_nop.bind(ty), r_stacknull, 0));
+        e.add64(enc(copy_nop.bind(ty), r_stacknull, 0));
+    }
+
     e
 }
diff --git a/cranelift/codegen/meta/src/isa/riscv/recipes.rs b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
index 8309802b92..67146ce17d 100644
--- a/cranelift/codegen/meta/src/isa/riscv/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
@@ -263,5 +263,13 @@ pub fn define<'formats>(
             .emit("unimplemented!();"),
     );
 
+    // Stack-slot to same stack-slot copy, which is guaranteed to turn into a no-op.
+    recipes.push(
+        EncodingRecipeBuilder::new("stacknull", f_unary, 0)
+            .operands_in(vec![Stack::new(gpr)])
+            .operands_out(vec![Stack::new(gpr)])
+            .emit(""),
+    );
+
     recipes
 }

From d5b80b2803a8df6fd8e4174598b080e0f1fbeea2 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 10 Jul 2019 12:01:37 +0200
Subject: [PATCH 2509/3084] [meta] Generate full documentation for instructions
 in InstBuilder;

---
 cranelift/codegen/meta/src/cdsl/instructions.rs | 12 +-----------
 cranelift/codegen/meta/src/gen_inst.rs          |  2 +-
 2 files changed, 2 insertions(+), 12 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index aa3262c66f..11d7114b52 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -97,7 +97,7 @@ pub struct InstructionContent {
     pub opcode_number: OpcodeNumber,
 
     /// Documentation string.
-    doc: String,
+    pub doc: String,
 
     /// Input operands. This can be a mix of SSA value operands and other operand kinds.
     pub operands_in: Vec,
@@ -162,16 +162,6 @@ impl Instruction {
         }
     }
 
-    pub fn doc_comment_first_line(&self) -> &str {
-        for line in self.doc.split("\n") {
-            let stripped = line.trim();
-            if stripped.len() > 0 {
-                return stripped;
-            }
-        }
-        ""
-    }
-
     pub fn all_typevars(&self) -> Vec<&TypeVar> {
         match &self.polymorphic_info {
             Some(poly) => {
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index 8e499cb866..a3c2a9fccd 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -930,7 +930,7 @@ fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Fo
         rtype
     );
 
-    fmt.doc_comment(format!("`{}`\n\n{}", inst, inst.doc_comment_first_line()));
+    fmt.doc_comment(&inst.doc);
     fmt.line("#[allow(non_snake_case)]");
     fmtln!(fmt, "fn {} {{", proto);
     fmt.indent(|fmt| {

From 55f36ce81a4f501a8f237b7e000d390558885215 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 10 Jul 2019 12:38:54 +0200
Subject: [PATCH 2510/3084] [docs] Update docs to point to the docs.rs website;

---
 cranelift/docs/conf.py   |  2 +-
 cranelift/docs/index.rst | 10 +++++++++-
 cranelift/docs/ir.rst    |  7 +++++++
 cranelift/docs/meta.rst  | 11 +++++++++--
 4 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py
index 09fd164537..81e035eb9e 100644
--- a/cranelift/docs/conf.py
+++ b/cranelift/docs/conf.py
@@ -59,7 +59,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'cranelift'
-copyright = u'2018, Cranelift Developers'
+copyright = u'2019, Cranelift Developers'
 author = u'Cranelift Developers'
 
 # The version info for the project you're documenting, acts as replacement for
diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst
index f37c83e34a..9951f6d9e9 100644
--- a/cranelift/docs/index.rst
+++ b/cranelift/docs/index.rst
@@ -15,11 +15,19 @@ Contents:
 Rust Crate Documentation
 ========================
 
-`cranelift `_
+`cranelift `_
+    This is an umbrella crate that re-exports the codegen and frontend crates,
+    to make them easier to use.
+
+`cranelift-codegen `_
     This is the core code generator crate. It takes Cranelift IR as input
     and emits encoded machine instructions, along with symbolic relocations,
     as output.
 
+`cranelift-codegen-meta `_
+    This crate contains the meta-language utilities and descriptions used by the
+    code generator.
+
 `cranelift-wasm `_
     This crate translates WebAssembly code into Cranelift IR.
 
diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index 550d42a81c..573dddc5b0 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -5,6 +5,13 @@ Cranelift IR Reference
 .. default-domain:: clif
 .. highlight:: clif
 
+.. todo:: Update the IR reference
+
+   This document is likely to be outdated and missing some important
+   information. It is recommended to look at the list of instructions as
+   documented in the `InstBuilder` documentation:
+   https://docs.rs/cranelift-codegen/latest/cranelift_codegen/ir/trait.InstBuilder.html
+
 The Cranelift intermediate representation (:term:`IR`) has two primary forms:
 an *in-memory data structure* that the code generator library is using, and a
 *text format* which is used for test cases and debug output.
diff --git a/cranelift/docs/meta.rst b/cranelift/docs/meta.rst
index 67457835b5..84145c1114 100644
--- a/cranelift/docs/meta.rst
+++ b/cranelift/docs/meta.rst
@@ -7,8 +7,15 @@ Cranelift Meta Language Reference
 .. module:: cdsl
 
 The Cranelift meta language is used to define instructions for Cranelift. It is a
-domain specific language embedded in Python. This document describes the Python
-modules that form the embedded DSL.
+domain specific language embedded in Rust.
+
+.. todo:: Point to the Rust documentation of the meta crate here.
+
+   This document is very out-of-date. Instead, you can have a look at the
+   work-in-progress documentation of the `meta` crate there:
+   https://docs.rs/cranelift-codegen-meta/0.34.0/cranelift_codegen_meta/.
+
+This document describes the Python modules that form the embedded DSL.
 
 The meta language descriptions are Python modules under the
 :file:`cranelift-codegen/meta-python` directory. The descriptions are processed in two

From abc3397017434bee3e09d660dd9df9f6f7470ffa Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 10 Jul 2019 12:39:10 +0200
Subject: [PATCH 2511/3084] [docs] Remove special handling of CDSL python
 modules;

---
 cranelift/docs/clif_domain.py | 385 ---------------------------
 cranelift/docs/conf.py        |   5 -
 cranelift/docs/ir.rst         | 474 ++++++++--------------------------
 cranelift/docs/meta.rst       | 138 ++--------
 4 files changed, 126 insertions(+), 876 deletions(-)
 delete mode 100644 cranelift/docs/clif_domain.py

diff --git a/cranelift/docs/clif_domain.py b/cranelift/docs/clif_domain.py
deleted file mode 100644
index eb9de01575..0000000000
--- a/cranelift/docs/clif_domain.py
+++ /dev/null
@@ -1,385 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Sphinx domain for documenting compiler intermediate representations.
-#
-# This defines a 'clif' Sphinx domain with the following directives and roles:
-#
-# .. clif::type:: type
-#     Document an IR type.
-# .. clif:inst:: v0, v1 = inst op0, op1
-#     Document an IR instruction.
-#
-from __future__ import absolute_import
-
-import re
-
-from docutils import nodes
-from docutils.parsers.rst import directives
-
-from sphinx import addnodes
-from sphinx.directives import ObjectDescription
-from sphinx.domains import Domain, ObjType
-from sphinx.locale import l_
-from sphinx.roles import XRefRole
-from sphinx.util.docfields import Field, GroupedField, TypedField
-from sphinx.util.nodes import make_refnode
-
-import sphinx.ext.autodoc
-
-
-class ClifObject(ObjectDescription):
-    """
-    Any kind of Cranelift IR object.
-
-    This is a shared base class for the different kinds of indexable objects
-    in the Cranelift IR reference.
-    """
-    option_spec = {
-        'noindex': directives.flag,
-        'module': directives.unchanged,
-        'annotation': directives.unchanged,
-    }
-
-    def add_target_and_index(self, name, sig, signode):
-        """
-        Add ``name`` to the index.
-
-        :param name: The object name returned by :func:`handle_signature`.
-        :param sig: The signature text.
-        :param signode: The output node.
-        """
-        targetname = self.objtype + '-' + name
-        if targetname not in self.state.document.ids:
-            signode['names'].append(targetname)
-            signode['ids'].append(targetname)
-            signode['first'] = (not self.names)
-            self.state.document.note_explicit_target(signode)
-            inv = self.env.domaindata['clif']['objects']
-            if name in inv:
-                self.state_machine.reporter.warning(
-                    'duplicate Cranelift object description of %s, ' % name +
-                    'other instance in ' + self.env.doc2path(inv[name][0]),
-                    line=self.lineno)
-            inv[name] = (self.env.docname, self.objtype)
-
-        indextext = self.get_index_text(name)
-        if indextext:
-            self.indexnode['entries'].append(('single', indextext,
-                                              targetname, '', None))
-
-
-# Type variables are indicated as %T.
-typevar = re.compile('(\%[A-Z])')
-
-
-def parse_type(name, signode):
-    """
-    Parse a type with embedded type vars and append to signode.
-
-    Return a string that can be compiled into a regular expression matching
-    the type.
-    """
-
-    re_str = ''
-
-    for part in typevar.split(name):
-        if part == '':
-            continue
-        if len(part) == 2 and part[0] == '%':
-            # This is a type parameter. Don't display the %, use emphasis
-            # instead.
-            part = part[1]
-            signode += nodes.emphasis(part, part)
-            re_str += r'\w+'
-        else:
-            signode += addnodes.desc_name(part, part)
-            re_str += re.escape(part)
-    return re_str
-
-
-class ClifType(ClifObject):
-    """A Cranelift IR type description."""
-
-    def handle_signature(self, sig, signode):
-        """
-        Parse type signature in ``sig`` and append description to signode.
-
-        Return a global object name for ``add_target_and_index``.
-        """
-
-        name = sig.strip()
-        parse_type(name, signode)
-        return name
-
-    def get_index_text(self, name):
-        return name + ' (IR type)'
-
-
-sep_equal = re.compile('\s*=\s*')
-sep_comma = re.compile('\s*,\s*')
-
-
-def parse_params(s, signode):
-    for i, p in enumerate(sep_comma.split(s)):
-        if i != 0:
-            signode += nodes.Text(', ')
-        signode += nodes.emphasis(p, p)
-
-
-class ClifInst(ClifObject):
-    """A Cranelift IR instruction."""
-
-    doc_field_types = [
-        TypedField('argument', label=l_('Arguments'),
-                   names=('in', 'arg'),
-                   typerolename='type', typenames=('type',)),
-        TypedField('result', label=l_('Results'),
-                   names=('out', 'result'),
-                   typerolename='type', typenames=('type',)),
-        GroupedField(
-            'typevar', names=('typevar',), label=l_('Type Variables')),
-        GroupedField('flag', names=('flag',), label=l_('Flags')),
-        Field('resulttype', label=l_('Result type'), has_arg=False,
-              names=('rtype',)),
-    ]
-
-    def handle_signature(self, sig, signode):
-        # Look for signatures like
-        #
-        #   v0, v1 = foo op0, op1
-        #   v0 = foo
-        #   foo op0
-
-        parts = re.split(sep_equal, sig, 1)
-        if len(parts) == 2:
-            # Outgoing parameters.
-            parse_params(parts[0], signode)
-            signode += nodes.Text(' = ')
-            name = parts[1]
-        else:
-            name = parts[0]
-
-        # Parse 'name arg, arg'
-        parts = name.split(None, 1)
-        name = parts[0]
-        signode += addnodes.desc_name(name, name)
-
-        if len(parts) == 2:
-            # Incoming parameters.
-            signode += nodes.Text(' ')
-            parse_params(parts[1], signode)
-
-        return name
-
-    def get_index_text(self, name):
-        return name
-
-
-class ClifInstGroup(ClifObject):
-    """A Cranelift IR instruction group."""
-
-
-class CraneliftDomain(Domain):
-    """Cranelift domain for IR objects."""
-    name = 'clif'
-    label = 'Cranelift'
-
-    object_types = {
-        'type': ObjType(l_('type'), 'type'),
-        'inst': ObjType(l_('instruction'), 'inst')
-    }
-
-    directives = {
-        'type': ClifType,
-        'inst': ClifInst,
-        'instgroup': ClifInstGroup,
-    }
-
-    roles = {
-        'type': XRefRole(),
-        'inst': XRefRole(),
-        'instgroup': XRefRole(),
-    }
-
-    initial_data = {
-        'objects': {},  # fullname -> docname, objtype
-    }
-
-    def clear_doc(self, docname):
-        for fullname, (fn, _l) in list(self.data['objects'].items()):
-            if fn == docname:
-                del self.data['objects'][fullname]
-
-    def merge_domaindata(self, docnames, otherdata):
-        for fullname, (fn, objtype) in otherdata['objects'].items():
-            if fn in docnames:
-                self.data['objects'][fullname] = (fn, objtype)
-
-    def resolve_xref(self, env, fromdocname, builder, typ, target, node,
-                     contnode):
-        objects = self.data['objects']
-        if target not in objects:
-            return None
-        obj = objects[target]
-        return make_refnode(builder, fromdocname, obj[0],
-                            obj[1] + '-' + target, contnode, target)
-
-    def resolve_any_xref(self, env, fromdocname, builder, target,
-                         node, contnode):
-        objects = self.data['objects']
-        if target not in objects:
-            return []
-        obj = objects[target]
-        return [('clif:' + self.role_for_objtype(obj[1]),
-                 make_refnode(builder, fromdocname, obj[0],
-                              obj[1] + '-' + target, contnode, target))]
-
-
-class TypeDocumenter(sphinx.ext.autodoc.Documenter):
-    # Invoke with .. autocliftype::
-    objtype = 'cliftype'
-    # Convert into clif:type directives
-    domain = 'clif'
-    directivetype = 'type'
-
-    @classmethod
-    def can_document_member(cls, member, membername, isattr, parent):
-        return False
-
-    def resolve_name(self, modname, parents, path, base):
-        return 'base.types', [base]
-
-    def add_content(self, more_content, no_docstring=False):
-        super(TypeDocumenter, self).add_content(more_content, no_docstring)
-        sourcename = self.get_sourcename()
-        membytes = self.object.membytes
-        if membytes:
-            self.add_line(u':bytes: {}'.format(membytes), sourcename)
-        else:
-            self.add_line(u':bytes: Can\'t be stored in memory', sourcename)
-
-
-class InstDocumenter(sphinx.ext.autodoc.Documenter):
-    # Invoke with .. autoinst::
-    objtype = 'inst'
-    # Convert into clif:inst directives
-    domain = 'clif'
-    directivetype = 'inst'
-
-    @classmethod
-    def can_document_member(cls, member, membername, isattr, parent):
-        return False
-
-    def resolve_name(self, modname, parents, path, base):
-        if path:
-            return path.rstrip('.'), [base]
-        else:
-            return 'base.instructions', [base]
-
-    def format_signature(self):
-        inst = self.object
-        sig = inst.name
-        if len(inst.outs) > 0:
-            sig = ', '.join([op.name for op in inst.outs]) + ' = ' + sig
-        if len(inst.ins) > 0:
-            op = inst.ins[0]
-            sig += ' ' + op.name
-            # If the first input is variable-args, this is 'return'. No parens.
-            if op.kind.name == 'variable_args':
-                sig += '...'.format(op.name)
-            for op in inst.ins[1:]:
-                # This is a call or branch with args in (...).
-                if op.kind.name == 'variable_args':
-                    sig += '({}...)'.format(op.name)
-                else:
-                    sig += ', ' + op.name
-        return sig
-
-    def add_directive_header(self, sig):
-        """Add the directive header and options to the generated content."""
-        domain = getattr(self, 'domain', 'clif')
-        directive = getattr(self, 'directivetype', self.objtype)
-        sourcename = self.get_sourcename()
-        self.add_line(u'.. %s:%s:: %s' % (domain, directive, sig), sourcename)
-        if self.options.noindex:
-            self.add_line(u'   :noindex:', sourcename)
-
-    def add_content(self, more_content, no_docstring=False):
-        super(InstDocumenter, self).add_content(more_content, no_docstring)
-        sourcename = self.get_sourcename()
-        inst = self.object
-
-        # Add inputs and outputs.
-        for op in inst.ins:
-            if op.is_value():
-                typ = op.typevar
-            else:
-                typ = op.kind
-            self.add_line(u':in {} {}: {}'.format(
-                typ, op.name, op.get_doc()), sourcename)
-        for op in inst.outs:
-            if op.is_value():
-                typ = op.typevar
-            else:
-                typ = op.kind
-            self.add_line(u':out {} {}: {}'.format(
-                typ, op.name, op.get_doc()), sourcename)
-
-        # Document type inference for polymorphic instructions.
-        if inst.is_polymorphic:
-            if inst.ctrl_typevar is not None:
-                if inst.use_typevar_operand:
-                    tvopnum = inst.value_opnums[inst.format.typevar_operand]
-                    self.add_line(
-                            u':typevar {}: inferred from {}'
-                            .format(
-                                inst.ctrl_typevar.name,
-                                inst.ins[tvopnum]),
-                            sourcename)
-                else:
-                    self.add_line(
-                            u':typevar {}: explicitly provided'
-                            .format(inst.ctrl_typevar.name),
-                            sourcename)
-            for tv in inst.other_typevars:
-                self.add_line(
-                        u':typevar {}: from input operand'.format(tv.name),
-                        sourcename)
-
-
-class InstGroupDocumenter(sphinx.ext.autodoc.ModuleLevelDocumenter):
-    # Invoke with .. autoinstgroup::
-    objtype = 'instgroup'
-    # Convert into clif:instgroup directives
-    domain = 'clif'
-    directivetype = 'instgroup'
-
-    @classmethod
-    def can_document_member(cls, member, membername, isattr, parent):
-        return False
-
-    def format_name(self):
-        return "{}.{}".format(self.modname, ".".join(self.objpath))
-
-    def add_content(self, more_content, no_docstring=False):
-        super(InstGroupDocumenter, self).add_content(
-                more_content, no_docstring)
-        sourcename = self.get_sourcename()
-        indexed = self.env.domaindata['clif']['objects']
-
-        names = [inst.name for inst in self.object.instructions]
-        names.sort()
-        for name in names:
-            if name in indexed:
-                self.add_line(u':clif:inst:`{}`'.format(name), sourcename)
-            else:
-                self.add_line(u'``{}``'.format(name), sourcename)
-
-
-def setup(app):
-    app.add_domain(CraneliftDomain)
-    app.add_autodocumenter(TypeDocumenter)
-    app.add_autodocumenter(InstDocumenter)
-    app.add_autodocumenter(InstGroupDocumenter)
-
-    return {'version': '0.1'}
diff --git a/cranelift/docs/conf.py b/cranelift/docs/conf.py
index 81e035eb9e..05be11238a 100644
--- a/cranelift/docs/conf.py
+++ b/cranelift/docs/conf.py
@@ -21,10 +21,6 @@ import os
 import sys
 sys.path.insert(0, os.path.abspath('.'))
 
-# Also add the meta-python directory to sys.path so autodoc can find the Cranelift meta
-# language definitions.
-sys.path.insert(0, os.path.abspath('../cranelift-codegen/meta-python'))
-
 # -- General configuration ------------------------------------------------
 
 # We don't support Sphinx versions before 1.4 since the format of index
@@ -41,7 +37,6 @@ extensions = [
     'sphinx.ext.ifconfig',
     'sphinx.ext.graphviz',
     'sphinx.ext.inheritance_diagram',
-    'clif_domain',
     'clif_lexer',
 ]
 
diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index 573dddc5b0..7d159905d2 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -86,10 +86,9 @@ containing multiple assignments to the same variables into SSA form for
 Cranelift :term:`IR`.
 
 Such variables can also be presented to Cranelift as :term:`stack slot`\s.
-Stack slots are accessed with the :inst:`stack_store` and :inst:`stack_load`
-instructions, and can have their address taken with :inst:`stack_addr`, which
-supports C-like programming languages where local variables can have their
-address taken.
+Stack slots are accessed with the `stack_store` and `stack_load` instructions,
+and can have their address taken with `stack_addr`, which supports C-like
+programming languages where local variables can have their address taken.
 
 .. _value-types:
 
@@ -105,20 +104,20 @@ Boolean types
 
 Boolean values are either true or false.
 
-The :type:`b1` type represents an abstract boolean value. It can only exist as
+The `b1` type represents an abstract boolean value. It can only exist as
 an SSA value, and can't be directly stored in memory. It can, however, be
-converted into an integer with value 0 or 1 by the :inst:`bint` instruction (and
-converted back with :inst:`icmp_imm` with 0).
+converted into an integer with value 0 or 1 by the `bint` instruction (and
+converted back with `icmp_imm` with 0).
 
 Several larger boolean types are also defined, primarily to be used as SIMD
 element types. They can be stored in memory, and are represented as either all
 zero bits or all one bits.
 
-.. autocliftype:: b1
-.. autocliftype:: b8
-.. autocliftype:: b16
-.. autocliftype:: b32
-.. autocliftype:: b64
+- b1
+- b8
+- b16
+- b32
+- b64
 
 Integer types
 -------------
@@ -129,10 +128,10 @@ number, others don't care.
 
 The support for i8 and i16 arithmetic is incomplete and use could lead to bugs.
 
-.. autocliftype:: i8
-.. autocliftype:: i16
-.. autocliftype:: i32
-.. autocliftype:: i64
+- i8
+- i16
+- i32
+- i64
 
 Floating point types
 --------------------
@@ -160,8 +159,8 @@ instructions are encoded as follows:
   and all bits of the trailing significand other than the MSB set to
   nondeterministic values.
 
-.. autocliftype:: f32
-.. autocliftype:: f64
+- f32
+- f64
 
 CPU flags types
 ---------------
@@ -172,15 +171,15 @@ compared.
 
 Since some ISAs don't have CPU flags, these value types should not be used
 until the legalization phase of compilation where the code is adapted to fit
-the target ISA. Use instructions like :inst:`icmp` instead.
+the target ISA. Use instructions like `icmp` instead.
 
 The CPU flags types are also restricted such that two flags values can not be
 live at the same time. After legalization, some instruction encodings will
 clobber the flags, and flags values are not allowed to be live across such
 instructions either. The verifier enforces these rules.
 
-.. autocliftype:: iflags
-.. autocliftype:: fflags
+- iflags
+- fflags
 
 SIMD vector types
 -----------------
@@ -189,42 +188,38 @@ A SIMD vector type represents a vector of values from one of the scalar types
 (boolean, integer, and floating point). Each scalar value in a SIMD type is
 called a *lane*. The number of lanes must be a power of two in the range 2-256.
 
-.. type:: i%Bx%N
+i%Bx%N
+    A SIMD vector of integers. The lane type `iB` is one of the integer
+    types `i8` ... `i64`.
 
-    A SIMD vector of integers. The lane type :type:`iB` is one of the integer
-    types :type:`i8` ... :type:`i64`.
-
-    Some concrete integer vector types are :type:`i32x4`, :type:`i64x8`, and
-    :type:`i16x4`.
+    Some concrete integer vector types are `i32x4`, `i64x8`, and
+    `i16x4`.
 
     The size of a SIMD integer vector in memory is :math:`N B\over 8` bytes.
 
-.. type:: f32x%N
-
+f32x%N
     A SIMD vector of single precision floating point numbers.
 
-    Some concrete :type:`f32` vector types are: :type:`f32x2`, :type:`f32x4`,
-    and :type:`f32x8`.
+    Some concrete `f32` vector types are: `f32x2`, `f32x4`,
+    and `f32x8`.
 
-    The size of a :type:`f32` vector in memory is :math:`4N` bytes.
-
-.. type:: f64x%N
+    The size of a `f32` vector in memory is :math:`4N` bytes.
 
+f64x%N
     A SIMD vector of double precision floating point numbers.
 
-    Some concrete :type:`f64` vector types are: :type:`f64x2`, :type:`f64x4`,
-    and :type:`f64x8`.
+    Some concrete `f64` vector types are: `f64x2`, `f64x4`,
+    and `f64x8`.
 
-    The size of a :type:`f64` vector in memory is :math:`8N` bytes.
-
-.. type:: b1x%N
+    The size of a `f64` vector in memory is :math:`8N` bytes.
 
+b1x%N
     A boolean SIMD vector.
 
     Boolean vectors are used when comparing SIMD vectors. For example,
-    comparing two :type:`i32x4` values would produce a :type:`b1x4` result.
+    comparing two `i32x4` values would produce a `b1x4` result.
 
-    Like the :type:`b1` type, a boolean vector cannot be stored in memory.
+    Like the `b1` type, a boolean vector cannot be stored in memory.
 
 Pseudo-types and type classes
 -----------------------------
@@ -232,40 +227,32 @@ Pseudo-types and type classes
 These are not concrete types, but convenient names used to refer to real types
 in this reference.
 
-.. type:: iAddr
-
+iAddr
     A Pointer-sized integer representing an address.
 
-    This is either :type:`i32`, or :type:`i64`, depending on whether the target
+    This is either `i32`, or `i64`, depending on whether the target
     platform has 32-bit or 64-bit pointers.
 
-.. type:: iB
+iB
+    Any of the scalar integer types `i8` -- `i64`.
 
-    Any of the scalar integer types :type:`i8` -- :type:`i64`.
+Int
+    Any scalar *or vector* integer type: `iB` or `iBxN`.
 
-.. type:: Int
+fB
+    Either of the floating point scalar types: `f32` or `f64`.
 
-    Any scalar *or vector* integer type: :type:`iB` or :type:`iBxN`.
-
-.. type:: fB
-
-    Either of the floating point scalar types: :type:`f32` or :type:`f64`.
-
-.. type:: Float
-
-    Any scalar *or vector* floating point type: :type:`fB` or :type:`fBxN`.
-
-.. type:: %Tx%N
+Float
+    Any scalar *or vector* floating point type: `fB` or `fBxN`.
 
+%Tx%N
     Any SIMD vector type.
 
-.. type:: Mem
+Mem
+    Any type that can be stored in memory: `Int` or `Float`.
 
-    Any type that can be stored in memory: :type:`Int` or :type:`Float`.
-
-.. type:: Testable
-
-    Either :type:`b1` or :type:`iN`.
+Testable
+    Either `b1` or `iN`.
 
 Immediate operand types
 -----------------------
@@ -273,48 +260,41 @@ Immediate operand types
 These types are not part of the normal SSA type system. They are used to
 indicate the different kinds of immediate operands on an instruction.
 
-.. type:: imm64
-
+imm64
     A 64-bit immediate integer. The value of this operand is interpreted as a
     signed two's complement integer. Instruction encodings may limit the valid
     range.
 
-    In the textual format, :type:`imm64` immediates appear as decimal or
+    In the textual format, `imm64` immediates appear as decimal or
     hexadecimal literals using the same syntax as C.
 
-.. type:: offset32
-
+offset32
     A signed 32-bit immediate address offset.
 
-    In the textual format, :type:`offset32` immediates always have an explicit
+    In the textual format, `offset32` immediates always have an explicit
     sign, and a 0 offset may be omitted.
 
-.. type:: ieee32
-
+ieee32
     A 32-bit immediate floating point number in the IEEE 754-2008 binary32
     interchange format. All bit patterns are allowed.
 
-.. type:: ieee64
-
+ieee64
     A 64-bit immediate floating point number in the IEEE 754-2008 binary64
     interchange format. All bit patterns are allowed.
 
-.. type:: bool
-
+bool
     A boolean immediate value, either false or true.
 
-    In the textual format, :type:`bool` immediates appear as 'false'
+    In the textual format, `bool` immediates appear as 'false'
     and 'true'.
 
-.. type:: intcc
+intcc
+    An integer condition code. See the `icmp` instruction for details.
 
-    An integer condition code. See the :inst:`icmp` instruction for details.
+floatcc
+    A floating point condition code. See the `fcmp` instruction for details.
 
-.. type:: floatcc
-
-    A floating point condition code. See the :inst:`fcmp` instruction for details.
-
-The two IEEE floating point immediate types :type:`ieee32` and :type:`ieee64`
+The two IEEE floating point immediate types `ieee32` and `ieee64`
 are displayed as hexadecimal floating point literals in the textual :term:`IR`
 format. Decimal floating point literals are not allowed because some computer
 systems can round differently when converting to binary. The hexadecimal
@@ -324,9 +304,9 @@ to represent all NaN bit patterns:
 Normal numbers
     Compatible with C99: ``-0x1.Tpe`` where ``T`` are the trailing
     significand bits encoded as hexadecimal, and ``e`` is the unbiased exponent
-    as a decimal number. :type:`ieee32` has 23 trailing significand bits. They
+    as a decimal number. `ieee32` has 23 trailing significand bits. They
     are padded with an extra LSB to produce 6 hexadecimal digits. This is not
-    necessary for :type:`ieee64` which has 52 trailing significand bits
+    necessary for `ieee64` which has 52 trailing significand bits
     forming 13 hexadecimal digits with no padding.
 
 Zeros
@@ -358,17 +338,10 @@ arguments, if it has any. Conditional branches only take the branch if their
 condition is satisfied, otherwise execution continues at the following
 instruction in the EBB.
 
-.. autoinst:: jump
-.. autoinst:: brz
-.. autoinst:: brnz
-.. autoinst:: br_icmp
-.. autoinst:: br_table
-
-.. inst:: JT = jump_table [EBB0, EBB1, ..., EBBn]
-
+JT = jump_table [EBB0, EBB1, ..., EBBn]
     Declare a jump table in the :term:`function preamble`.
 
-    This declares a jump table for use by the :inst:`br_table` indirect branch
+    This declares a jump table for use by the `br_table` indirect branch
     instruction. Entries in the table are EBB names.
 
     The EBBs listed must belong to the current function, and they can't have
@@ -382,13 +355,9 @@ instruction in the EBB.
 Traps stop the program because something went wrong. The exact behavior depends
 on the target instruction set architecture and operating system. There are
 explicit trap instructions defined below, but some instructions may also cause
-traps for certain input value. For example, :inst:`udiv` traps when the divisor
+traps for certain input value. For example, `udiv` traps when the divisor
 is zero.
 
-.. autoinst:: trap
-.. autoinst:: trapz
-.. autoinst:: trapnz
-
 
 Function calls
 ==============
@@ -448,8 +417,7 @@ compilers.
 Functions that are called directly must be declared in the :term:`function
 preamble`:
 
-.. inst:: FN = [colocated] NAME signature
-
+FN = [colocated] NAME signature
     Declare a function so it can be called directly.
 
     If the colocated keyword is present, the symbol's definition will be
@@ -458,11 +426,7 @@ preamble`:
 
     :arg NAME: Name of the function, passed to the linker for resolution.
     :arg signature: Function signature. See below.
-    :result FN: A function identifier that can be used with :inst:`call`.
-
-.. autoinst:: call
-.. autoinst:: x_return
-.. autoinst:: fallthrough_return
+    :result FN: A function identifier that can be used with `call`.
 
 This simple example illustrates direct function calls and signatures:
 
@@ -472,46 +436,39 @@ This simple example illustrates direct function calls and signatures:
 
 Indirect function calls use a signature declared in the preamble.
 
-.. autoinst:: call_indirect
-.. autoinst:: func_addr
-
 .. _memory:
 
 Memory
 ======
 
-Cranelift provides fully general :inst:`load` and :inst:`store` instructions for
-accessing memory, as well as :ref:`extending loads and truncating stores
+Cranelift provides fully general `load` and `store` instructions for accessing
+memory, as well as :ref:`extending loads and truncating stores
 `.
 
 If the memory at the given address is not :term:`addressable`, the behavior of
 these instructions is undefined. If it is addressable but not
 :term:`accessible`, they :term:`trap`.
 
-.. autoinst:: load
-.. autoinst:: store
-
 There are also more restricted operations for accessing specific types of memory
 objects.
 
 Additionally, instructions are provided for handling multi-register addressing.
 
-.. autoinst:: load_complex
-.. autoinst:: store_complex
-
 Memory operation flags
 ----------------------
 
 Loads and stores can have flags that loosen their semantics in order to enable
 optimizations.
 
-=======  ===========================================
+======== ===========================================
 Flag     Description
-=======  ===========================================
+======== ===========================================
 notrap   Memory is assumed to be :term:`accessible`.
 aligned  Trapping allowed for misaligned accesses.
-readonly The data at the specified address will not modified between when this function is called and exited.
-=======  ===========================================
+readonly The data at the specified address will not
+         modified between when this function is
+         called and exited.
+======== ===========================================
 
 When the ``accessible`` flag is set, the behavior is undefined if the memory
 is not :term:`accessible`.
@@ -530,8 +487,7 @@ allocated in the :term:`function preamble`. Stack slots are not typed, they
 simply represent a contiguous sequence of :term:`accessible` bytes in the stack
 frame.
 
-.. inst:: SS = explicit_slot Bytes, Flags...
-
+SS = explicit_slot Bytes, Flags...
     Allocate a stack slot in the preamble.
 
     If no alignment is specified, Cranelift will pick an appropriate alignment
@@ -541,9 +497,6 @@ frame.
     :flag align(N): Request at least N bytes alignment.
     :result SS: Stack slot index.
 
-.. autoinst:: stack_load
-.. autoinst:: stack_store
-
 The dedicated stack access instructions are easy for the compiler to reason
 about because stack slots and offsets are fixed at compile time. For example,
 the alignment of these stack memory accesses can be inferred from the offsets
@@ -552,9 +505,7 @@ and stack slot alignments.
 It's also possible to obtain the address of a stack slot, which can be used
 in :ref:`unrestricted loads and stores `.
 
-.. autoinst:: stack_addr
-
-The :inst:`stack_addr` instruction can be used to macro-expand the stack access
+The `stack_addr` instruction can be used to macro-expand the stack access
 instructions before instruction selection::
 
     v0 = stack_load.f64 ss3, 16
@@ -569,7 +520,7 @@ Global values
 -------------
 
 A *global value* is an object whose value is not known at compile time. The
-value is computed at runtime by :inst:`global_value`, possibly using
+value is computed at runtime by `global_value`, possibly using
 information provided by the linker via relocations. There are multiple
 kinds of global values using different methods for determining their value.
 Cranelift does not track the type of a global value, for they are just
@@ -584,8 +535,7 @@ Cranelift functions.
 Chains of global value expressions are possible, but cycles are not allowed.
 They will be caught by the IR verifier.
 
-.. inst:: GV = vmctx
-
+GV = vmctx
     Declare a global value of the address of the VM context struct.
 
     This declares a global value which is the VM context pointer which may
@@ -599,8 +549,7 @@ A global value can also be derived by treating another global variable as a
 struct pointer and loading from one of its fields. This makes it possible to
 chase pointers into VM runtime data structures.
 
-.. inst:: GV = load.Type BaseGV [Offset]
-
+GV = load.Type BaseGV [Offset]
     Declare a global value pointed to by BaseGV plus Offset, with type Type.
 
     It is assumed the BaseGV plus Offset resides in accessible memory with the
@@ -610,15 +559,13 @@ chase pointers into VM runtime data structures.
     :arg Offset: Offset added to the base before loading.
     :result GV: Global value.
 
-.. inst:: GV = iadd_imm BaseGV, Offset
-
+GV = iadd_imm BaseGV, Offset
     Declare a global value which has the value of BaseGV offset by Offset.
 
     :arg BaseGV: Global value providing the base value.
     :arg Offset: Offset added to the base value.
 
-.. inst:: GV = [colocated] symbol Name
-
+GV = [colocated] symbol Name
     Declare a symbolic address global value.
 
     The value of GV is symbolic and will be assigned a relocation, so that
@@ -631,10 +578,6 @@ chase pointers into VM runtime data structures.
     :arg Name: External name.
     :result GV: Global value.
 
-.. autoinst:: global_value
-.. autoinst:: symbol_value
-
-
 Heaps
 -----
 
@@ -644,9 +587,9 @@ in, and all accesses are bounds checked. Cranelift models this through the
 concept of *heaps*.
 
 A heap is declared in the function preamble and can be accessed with the
-:inst:`heap_addr` instruction that :term:`traps` on out-of-bounds accesses or
+`heap_addr` instruction that :term:`traps` on out-of-bounds accesses or
 returns a pointer that is guaranteed to trap. Heap addresses can be smaller than
-the native pointer size, for example unsigned :type:`i32` offsets on a 64-bit
+the native pointer size, for example unsigned `i32` offsets on a 64-bit
 architecture.
 
 .. digraph:: static
@@ -674,12 +617,10 @@ A heap appears as three consecutive ranges of address space:
    not :term:`accessible`.
 
 The *heap bound* is the total size of the mapped and unmapped pages. This is
-the bound that :inst:`heap_addr` checks against. Memory accesses inside the
+the bound that `heap_addr` checks against. Memory accesses inside the
 heap bounds can trap if they hit an unmapped page (which is not
 :term:`accessible`).
 
-.. autoinst:: heap_addr
-
 Two styles of heaps are supported, *static* and *dynamic*. They behave
 differently when resized.
 
@@ -693,8 +634,7 @@ unmapped pages where the heap can grow up to its maximum size. After the
 unmapped pages follow the offset-guard pages which are also guaranteed to
 generate a trap when accessed.
 
-.. inst:: H = static Base, min MinBytes, bound BoundBytes, offset_guard OffsetGuardBytes
-
+H = static Base, min MinBytes, bound BoundBytes, offset_guard OffsetGuardBytes
     Declare a static heap in the preamble.
 
     :arg Base: Global value holding the heap's base address.
@@ -712,8 +652,7 @@ A *dynamic heap* can be relocated to a different base address when it is
 resized, and its bound can move dynamically. The offset-guard pages move when
 the heap is resized. The bound of a dynamic heap is stored in a global value.
 
-.. inst:: H = dynamic Base, min MinBytes, bound BoundGV, offset_guard OffsetGuardBytes
-
+H = dynamic Base, min MinBytes, bound BoundGV, offset_guard OffsetGuardBytes
     Declare a dynamic heap in the preamble.
 
     :arg Base: Global value holding the heap's base address.
@@ -762,25 +701,22 @@ linear memory. WebAssembly uses *tables* to allow programs to refer to opaque
 values through integer indices.
 
 A table is declared in the function preamble and can be accessed with the
-:inst:`table_addr` instruction that :term:`traps` on out-of-bounds accesses.
+`table_addr` instruction that :term:`traps` on out-of-bounds accesses.
 Table addresses can be smaller than the native pointer size, for example
-unsigned :type:`i32` offsets on a 64-bit architecture.
+unsigned `i32` offsets on a 64-bit architecture.
 
 A table appears as a consecutive range of address space, conceptually
 divided into elements of fixed sizes, which are identified by their index.
 The memory is :term:`accessible`.
 
 The *table bound* is the number of elements currently in the table. This is
-the bound that :inst:`table_addr` checks against.
-
-.. autoinst:: table_addr
+the bound that `table_addr` checks against.
 
 A table can be relocated to a different base address when it is resized, and
 its bound can move dynamically. The bound of a table is stored in a global
 value.
 
-.. inst:: T = dynamic Base, min MinElements, bound BoundGV, element_size ElementSize
-
+T = dynamic Base, min MinElements, bound BoundGV, element_size ElementSize
     Declare a table in the preamble.
 
     :arg Base: Global value holding the table's base address.
@@ -788,85 +724,12 @@ value.
     :arg BoundGV: Global value containing the current heap bound in elements.
     :arg ElementSize: Size of each element.
 
-Operations
-==========
-
-.. autoinst:: select
-.. autoinst:: selectif
-
 Constant materialization
 ------------------------
 
-A few instructions have variants that take immediate operands (e.g.,
-:inst:`band` / :inst:`band_imm`), but in general an instruction is required to
-load a constant into an SSA value.
-
-.. autoinst:: iconst
-.. autoinst:: f32const
-.. autoinst:: f64const
-.. autoinst:: bconst
-
-Vector operations
------------------
-
-.. autoinst:: vsplit
-.. autoinst:: vconcat
-.. autoinst:: vselect
-.. autoinst:: splat
-.. autoinst:: insertlane
-.. autoinst:: extractlane
-
-Integer operations
-------------------
-
-.. autoinst:: icmp
-.. autoinst:: icmp_imm
-.. autoinst:: iadd
-.. autoinst:: iadd_imm
-.. autoinst:: iadd_cin
-.. autoinst:: iadd_cout
-.. autoinst:: iadd_carry
-.. autoinst:: isub
-.. autoinst:: irsub_imm
-.. autoinst:: isub_bin
-.. autoinst:: isub_bout
-.. autoinst:: isub_borrow
-
-.. todo:: Add and subtract with signed overflow.
-
-    For example, see
-    `llvm.sadd.with.overflow.*` and `llvm.ssub.with.overflow.*` in
-    `LLVM `_.
-
-.. autoinst:: imul
-.. autoinst:: imul_imm
-
-.. todo:: Larger multiplication results.
-
-    For example, ``smulx`` which multiplies :type:`i32` operands to produce a
-    :type:`i64` result. Alternatively, ``smulhi`` and ``smullo`` pairs.
-
-.. autoinst:: udiv
-.. autoinst:: udiv_imm
-.. autoinst:: sdiv
-.. autoinst:: sdiv_imm
-.. autoinst:: urem
-.. autoinst:: urem_imm
-.. autoinst:: srem
-.. autoinst:: srem_imm
-
-.. todo:: Integer minimum / maximum.
-
-    NEON has ``smin``, ``smax``, ``umin``, and ``umax`` instructions. We should
-    replicate those for both scalar and vector integer types. Even if the
-    target ISA doesn't have scalar operations, these are good pattern matching
-    targets.
-
-.. todo:: Saturating arithmetic.
-
-    Mostly for SIMD use, but again these are good patterns for contraction.
-    Something like ``usatadd``, ``usatsub``, ``ssatadd``, and ``ssatsub`` is a
-    good start.
+A few instructions have variants that take immediate operands, but in general
+an instruction is required to load a constant into an SSA value: `iconst`,
+`f32const`, `f64const` and `bconst` serve this purpose.
 
 Bitwise operations
 ------------------
@@ -876,17 +739,6 @@ numbers, and booleans. When operating on integer or floating point types, the
 bitwise operations are working on the binary representation of the values. When
 operating on boolean values, the bitwise operations work as logical operators.
 
-.. autoinst:: band
-.. autoinst:: band_imm
-.. autoinst:: bor
-.. autoinst:: bor_imm
-.. autoinst:: bxor
-.. autoinst:: bxor_imm
-.. autoinst:: bnot
-.. autoinst:: band_not
-.. autoinst:: bor_not
-.. autoinst:: bxor_not
-
 The shift and rotate operations only work on integer types (scalar and vector).
 The shift amount does not have to be the same type as the value being shifted.
 Only the low `B` bits of the shift amount is significant.
@@ -895,37 +747,13 @@ When operating on an integer vector type, the shift amount is still a scalar
 type, and all the lanes are shifted the same amount. The shift amount is masked
 to the number of bits in a *lane*, not the full size of the vector type.
 
-.. autoinst:: rotl
-.. autoinst:: rotl_imm
-.. autoinst:: rotr
-.. autoinst:: rotr_imm
-.. autoinst:: ishl
-.. autoinst:: ishl_imm
-.. autoinst:: ushr
-.. autoinst:: ushr_imm
-.. autoinst:: sshr
-.. autoinst:: sshr_imm
-
-The bit-counting instructions below are scalar only.
-
-.. autoinst:: clz
-.. autoinst:: cls
-.. autoinst:: ctz
-.. autoinst:: popcnt
+The bit-counting instructions are scalar only.
 
 Floating point operations
 -------------------------
 
 These operations generally follow IEEE 754-2008 semantics.
 
-.. autoinst:: fcmp
-.. autoinst:: fadd
-.. autoinst:: fsub
-.. autoinst:: fmul
-.. autoinst:: fdiv
-.. autoinst:: sqrt
-.. autoinst:: fma
-
 Sign bit manipulations
 ~~~~~~~~~~~~~~~~~~~~~~
 
@@ -933,10 +761,6 @@ The sign manipulating instructions work as bitwise operations, so they don't
 have special behavior for signaling NaN operands. The exponent and trailing
 significand bits are always preserved.
 
-.. autoinst:: fneg
-.. autoinst:: fabs
-.. autoinst:: fcopysign
-
 Minimum and maximum
 ~~~~~~~~~~~~~~~~~~~
 
@@ -946,40 +770,15 @@ return NaN when either input is NaN.
 
 When comparing zeroes, these instructions behave as if :math:`-0.0 < 0.0`.
 
-.. autoinst:: fmin
-.. autoinst:: fmax
-
 Rounding
 ~~~~~~~~
 
 These instructions round their argument to a nearby integral value, still
 represented as a floating point number.
 
-.. autoinst:: ceil
-.. autoinst:: floor
-.. autoinst:: trunc
-.. autoinst:: nearest
-
 Conversion operations
 ---------------------
 
-.. autoinst:: bitcast
-.. autoinst:: breduce
-.. autoinst:: bextend
-.. autoinst:: bint
-.. autoinst:: bmask
-.. autoinst:: ireduce
-.. autoinst:: uextend
-.. autoinst:: sextend
-.. autoinst:: fpromote
-.. autoinst:: fdemote
-.. autoinst:: fcvt_to_uint
-.. autoinst:: fcvt_to_sint
-.. autoinst:: fcvt_to_uint_sat
-.. autoinst:: fcvt_to_sint_sat
-.. autoinst:: fcvt_from_uint
-.. autoinst:: fcvt_from_sint
-
 .. _extload-truncstore:
 
 Extending loads and truncating stores
@@ -989,23 +788,13 @@ Most ISAs provide instructions that load an integer value smaller than a registe
 and extends it to the width of the register. Similarly, store instructions that
 only write the low bits of an integer register are common.
 
-In addition to the normal :inst:`load` and :inst:`store` instructions, Cranelift
+In addition to the normal `load` and `store` instructions, Cranelift
 provides extending loads and truncation stores for 8, 16, and 32-bit memory
 accesses.
 
 These instructions succeed, trap, or have undefined behavior, under the same
 conditions as :ref:`normal loads and stores `.
 
-.. autoinst:: uload8
-.. autoinst:: sload8
-.. autoinst:: istore8
-.. autoinst:: uload16
-.. autoinst:: sload16
-.. autoinst:: istore16
-.. autoinst:: uload32
-.. autoinst:: sload32
-.. autoinst:: istore32
-
 ISA-specific instructions
 =========================
 
@@ -1017,16 +806,6 @@ x86
 
 Instructions that can only be used by the x86 target ISA.
 
-.. autoinst:: isa.x86.instructions.sdivmodx
-.. autoinst:: isa.x86.instructions.udivmodx
-.. autoinst:: isa.x86.instructions.cvtt2si
-.. autoinst:: isa.x86.instructions.fmin
-.. autoinst:: isa.x86.instructions.fmax
-.. autoinst:: isa.x86.instructions.bsf
-.. autoinst:: isa.x86.instructions.bsr
-.. autoinst:: isa.x86.instructions.push
-.. autoinst:: isa.x86.instructions.pop
-
 Codegen implementation instructions
 ===================================
 
@@ -1039,42 +818,18 @@ Legalization operations
 These instructions are used as helpers when legalizing types and operations for
 the target ISA.
 
-.. autoinst:: isplit
-.. autoinst:: iconcat
-
 Special register operations
 ---------------------------
 
 The prologue and epilogue of a function needs to manipulate special registers like the stack
 pointer and the frame pointer. These instructions should not be used in regular code.
 
-.. autoinst:: adjust_sp_down
-.. autoinst:: adjust_sp_up_imm
-.. autoinst:: adjust_sp_down_imm
-.. autoinst:: ifcmp_sp
-.. autoinst:: copy_special
-
-Low-level control flow operations
----------------------------------
-
-.. autoinst:: fallthrough
-
 CPU flag operations
 -------------------
 
 These operations are for working with the "flags" registers of some CPU
 architectures.
 
-.. autoinst:: ifcmp
-.. autoinst:: ifcmp_imm
-.. autoinst:: ffcmp
-.. autoinst:: trueif
-.. autoinst:: trueff
-.. autoinst:: trapif
-.. autoinst:: trapff
-.. autoinst:: brif
-.. autoinst:: brff
-
 Live range splitting
 --------------------
 
@@ -1084,37 +839,24 @@ value can be quite large, it is sometimes beneficial to split the live range
 into smaller parts.
 
 A live range is split by creating new SSA values that are copies or the
-original value or each other. The copies are created by inserting :inst:`copy`,
-:inst:`spill`, or :inst:`fill` instructions, depending on whether the values
+original value or each other. The copies are created by inserting `copy`,
+`spill`, or `fill` instructions, depending on whether the values
 are assigned to registers or stack slots.
 
 This approach permits SSA form to be preserved throughout the register
 allocation pass and beyond.
 
-.. autoinst:: copy
-.. autoinst:: spill
-.. autoinst:: fill
-
 Register values can be temporarily diverted to other registers by the
-:inst:`regmove` instruction, and to and from stack slots by :inst:`regspill`
-and :inst:`regfill`.
-
-.. autoinst:: regmove
-.. autoinst:: regspill
-.. autoinst:: regfill
-
+`regmove` instruction, and to and from stack slots by `regspill`
+and `regfill`.
 
 Instruction groups
 ==================
 
-All of the shared instructions are part of the :instgroup:`base` instruction
+All of the shared instructions are part of the `base` instruction
 group.
 
-.. autoinstgroup:: base.instructions.GROUP
-
-Target ISAs may define further instructions in their own instruction groups:
-
-.. autoinstgroup:: isa.x86.instructions.GROUP
+Target ISAs may define further instructions in their own instruction groups.
 
 Implementation limits
 =====================
@@ -1272,8 +1014,8 @@ Glossary
         execution somewhere else. Execution never continues at the instruction
         following a terminator instruction.
 
-        The basic terminator instructions are :inst:`br`, :inst:`return`, and
-        :inst:`trap`. Conditional branches and instructions that trap
+        The basic terminator instructions are `br`, `return`, and
+        `trap`. Conditional branches and instructions that trap
         conditionally are not terminator instructions.
 
     trap
diff --git a/cranelift/docs/meta.rst b/cranelift/docs/meta.rst
index 84145c1114..5bc000a800 100644
--- a/cranelift/docs/meta.rst
+++ b/cranelift/docs/meta.rst
@@ -4,7 +4,6 @@ Cranelift Meta Language Reference
 
 .. default-domain:: py
 .. highlight:: python
-.. module:: cdsl
 
 The Cranelift meta language is used to define instructions for Cranelift. It is a
 domain specific language embedded in Rust.
@@ -18,50 +17,36 @@ domain specific language embedded in Rust.
 This document describes the Python modules that form the embedded DSL.
 
 The meta language descriptions are Python modules under the
-:file:`cranelift-codegen/meta-python` directory. The descriptions are processed in two
+`cranelift-codegen/meta-python` directory. The descriptions are processed in two
 steps:
 
 1. The Python modules are imported. This has the effect of building static data
    structures in global values in the modules. These static data structures
-   in the :mod:`base` and :mod:`isa` packages use the classes in the
-   :mod:`cdsl` package to describe instruction sets and other properties.
+   in the `base` and `isa` packages use the classes in the
+   `cdsl` package to describe instruction sets and other properties.
 
 2. The static data structures are processed to produce Rust source code and
    constant tables.
 
 The main driver for this source code generation process is the
-:file:`cranelift-codegen/meta-python/build.py` script which is invoked as part of the build
-process if anything in the :file:`cranelift-codegen/meta-python` directory has changed
+`cranelift-codegen/meta-python/build.py` script which is invoked as part of the build
+process if anything in the `cranelift-codegen/meta-python` directory has changed
 since the last build.
 
-
-.. module:: cdsl.settings
-
 Settings
 ========
 
 Settings are used by the environment embedding Cranelift to control the details
 of code generation. Each setting is defined in the meta language so a compact
 and consistent Rust representation can be generated. Shared settings are defined
-in the :mod:`base.settings` module. Some settings are specific to a target ISA,
-and defined in a :file:`settings.py` module under the appropriate
-:file:`cranelift-codegen/meta-python/isa/*` directory.
+in the `base.settings` module. Some settings are specific to a target ISA,
+and defined in a `settings.py` module under the appropriate
+`cranelift-codegen/meta-python/isa/*` directory.
 
 Settings can take boolean on/off values, small numbers, or explicitly enumerated
-symbolic values. Each type is represented by a sub-class of :class:`Setting`:
+symbolic values.
 
-.. inheritance-diagram:: Setting BoolSetting NumSetting EnumSetting
-    :parts: 1
-
-.. autoclass:: Setting
-.. autoclass:: BoolSetting
-.. autoclass:: NumSetting
-.. autoclass:: EnumSetting
-
-All settings must belong to a *group*, represented by a :class:`SettingGroup`
-object.
-
-.. autoclass:: SettingGroup
+All settings must belong to a *group*, represented by a :class:`SettingGroup` object.
 
 Normally, a setting group corresponds to all settings defined in a module. Such
 a module looks like this::
@@ -74,9 +59,6 @@ a module looks like this::
 
     group.close(globals())
 
-
-.. module:: cdsl.instructions
-
 Instruction descriptions
 ========================
 
@@ -84,27 +66,16 @@ New instructions are defined as instances of the :class:`Instruction`
 class. As instruction instances are created, they are added to the currently
 open :class:`InstructionGroup`.
 
-.. autoclass:: InstructionGroup
-    :members:
-
 The basic Cranelift instruction set described in :doc:`ir` is defined by the
-Python module :mod:`base.instructions`. This module has a global value
-:data:`base.instructions.GROUP` which is an :class:`InstructionGroup` instance
+Python module `base.instructions`. This module has a global value
+`base.instructions.GROUP` which is an :class:`InstructionGroup` instance
 containing all the base instructions.
 
-.. autoclass:: Instruction
-
-.. currentmodule:: cdsl.operands
-
 An instruction is defined with a set of distinct input and output operands which
 must be instances of the :class:`Operand` class.
 
-.. autoclass:: Operand
-
 Cranelift uses two separate type systems for operand kinds and SSA values.
 
-.. module:: cdsl.typevar
-
 Type variables
 --------------
 
@@ -113,9 +84,6 @@ Instruction descriptions can be made polymorphic by using
 instead of a concrete value type. Polymorphism only works for SSA value
 operands. Other operands have a fixed operand kind.
 
-.. autoclass:: TypeVar
-    :members:
-
 If multiple operands refer to the same type variable they will be required to
 have the same concrete type. For example, this defines an integer addition
 instruction::
@@ -138,61 +106,27 @@ There are some practical restrictions on the use of type variables, see
 Immediate operands
 ------------------
 
-.. currentmodule:: cdsl.operands
-
 Immediate instruction operands don't correspond to SSA values, but have values
 that are encoded directly in the instruction. Immediate operands don't
 have types from the :class:`cdsl.types.ValueType` type system; they often have
 enumerated values of a specific type. The type of an immediate operand is
 indicated with an instance of :class:`ImmediateKind`.
 
-.. autoclass:: ImmediateKind
-
-.. automodule:: base.immediates
-    :members:
-
 Entity references
 -----------------
 
-.. currentmodule:: cdsl.operands
-
 Instruction operands can also refer to other entities in the same function. This
 can be extended basic blocks, or entities declared in the function preamble.
 
-.. autoclass:: EntityRefKind
-
-.. automodule:: base.entities
-    :members:
-
 Value types
 -----------
 
-.. currentmodule:: cdsl.types
-
 Concrete value types are represented as instances of :class:`ValueType`. There
 are subclasses to represent scalar and vector types.
 
-.. autoclass:: ValueType
-.. inheritance-diagram:: ValueType LaneType VectorType IntType FloatType BoolType SpecialType FlagsType
-    :parts: 1
-.. autoclass:: LaneType
-    :members:
-.. autoclass:: VectorType
-.. autoclass:: SpecialType
-.. autoclass:: IntType
-.. autoclass:: FloatType
-.. autoclass:: BoolType
-.. autoclass:: FlagsType
-
-.. automodule:: base.types
-    :members:
-
 There are no predefined vector types, but they can be created as needed with
 the :func:`LaneType.by` function.
 
-
-.. module:: cdsl.operands
-
 Instruction representation
 ==========================
 
@@ -202,24 +136,12 @@ written as Rust code in the ``cranelift.instructions`` module. The instruction
 representation depends on the input operand kinds and whether the instruction
 can produce multiple results.
 
-.. autoclass:: OperandKind
-.. inheritance-diagram:: OperandKind ImmediateKind EntityRefKind
-
 Since all SSA value operands are represented as a `Value` in Rust code, value
-types don't affect the representation. Two special operand kinds are used to
-represent SSA values:
-
-.. autodata:: VALUE
-.. autodata:: VARIABLE_ARGS
-
-.. module:: cdsl.formats
+types don't affect the representation.
 
 When an instruction description is created, it is automatically assigned a
 predefined instruction format which is an instance of
-:class:`InstructionFormat`:
-
-.. autoclass:: InstructionFormat
-
+:class:`InstructionFormat`.
 
 .. _restricted-polymorphism:
 
@@ -264,8 +186,6 @@ controlling type variable, or it can vary independently of the other operands.
 Encodings
 =========
 
-.. currentmodule:: cdsl.isa
-
 Encodings describe how Cranelift instructions are mapped to binary machine code
 for the target architecture. After the legalization pass, all remaining
 instructions are expected to map 1-1 to native instruction encodings. Cranelift
@@ -277,7 +197,7 @@ incompatible encodings. For example, a modern ARMv8 CPU might support three
 different CPU modes: *A64* where instructions are encoded in 32 bits, *A32*
 where all instructions are 32 bits, and *T32* which has a mix of 16-bit and
 32-bit instruction encodings. These are incompatible encoding spaces, and while
-an :clif:inst:`iadd` instruction can be encoded in 32 bits in each of them, it's
+an `iadd` instruction can be encoded in 32 bits in each of them, it's
 not the same 32 bits. It's a judgement call if CPU modes should be modelled as
 separate targets, or as sub-modes of the same target. In the ARMv8 case, the
 different register banks means that it makes sense to model A64 as a separate
@@ -288,8 +208,6 @@ instruction. Both RISC-V and ARMv8's T32 mode have 32-bit encodings of all
 instructions with 16-bit encodings available for some opcodes if certain
 constraints are satisfied.
 
-.. autoclass:: CPUMode
-
 Encodings are guarded by :term:`sub-target predicate`\s. For example, the RISC-V
 "C" extension which specifies the compressed encodings may not be supported, and
 a predicate would be used to disable all of the 16-bit encodings in that case.
@@ -327,7 +245,7 @@ An :py:class:`Encoding` instance specifies the encoding of a concrete
 instruction. The following properties are used to select instructions to be
 encoded:
 
-- An opcode, i.e. :clif:inst:`iadd_imm`, that must match the instruction's
+- An opcode, i.e. `iadd_imm`, that must match the instruction's
   opcode.
 - Values for any type variables if the opcode represents a polymorphic
   instruction.
@@ -350,8 +268,6 @@ The additional predicates in the :py:class:`EncRecipe` are merged with the
 per-encoding predicates when generating the encoding matcher code. Often
 encodings only need the recipe predicates.
 
-.. autoclass:: EncRecipe
-
 Register constraints
 ====================
 
@@ -371,9 +287,6 @@ Each encoding recipe specifies separate constraints for its value operands and
 result. These constraints are separate from the instruction predicate which can
 only evaluate the instruction's immediate operands.
 
-.. module:: cdsl.registers
-.. autoclass:: RegBank
-
 Register class constraints
 --------------------------
 
@@ -388,8 +301,6 @@ register class::
 This defines an encoding recipe for the ``Binary`` instruction format where
 both input operands must be allocated from the ``GPR`` register class.
 
-.. autoclass:: RegClass
-
 Tied register operands
 ----------------------
 
@@ -420,7 +331,7 @@ Stack operands
 --------------
 
 Cranelift's register allocator can assign an SSA value to a stack slot if there
-isn't enough registers. It will insert :clif:inst:`spill` and :clif:inst:`fill`
+isn't enough registers. It will insert `spill` and `fill`
 instructions as needed to satisfy instruction operand constraints, but it is
 also possible to have instructions that can access stack slots directly::
 
@@ -429,27 +340,14 @@ also possible to have instructions that can access stack slots directly::
 An output stack value implies a store to the stack, an input value implies a
 load.
 
-.. module:: cdsl.isa
-
 Targets
 =======
 
 Cranelift can be compiled with support for multiple target instruction set
 architectures. Each ISA is represented by a :py:class:`cdsl.isa.TargetISA` instance.
 
-.. autoclass:: TargetISA
-
 The definitions for each supported target live in a package under
-:file:`cranelift-codegen/meta-python/isa`.
-
-.. automodule:: isa
-    :members:
-
-.. automodule:: isa.riscv
-.. automodule:: isa.x86
-.. automodule:: isa.arm32
-.. automodule:: isa.arm64
-
+`cranelift-codegen/meta-python/isa`.
 
 Glossary
 ========

From 062ed8f6ea330775fd9c66fbf1154f0de48b587d Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 10 Jul 2019 17:03:55 +0200
Subject: [PATCH 2512/3084] [docs] Remove rst annotations in instructions doc
 comments;

---
 .../codegen/meta/src/isa/x86/instructions.rs  |   4 +-
 .../codegen/meta/src/shared/instructions.rs   | 114 +++++++++---------
 2 files changed, 57 insertions(+), 61 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 2f365165b8..9aa7363e1a 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -151,7 +151,7 @@ pub fn define(
         Floating point minimum with x86 semantics.
 
         This is equivalent to the C ternary operator `x < y ? x : y` which
-        differs from :inst:`fmin` when either operand is NaN or when comparing
+        differs from `fmin` when either operand is NaN or when comparing
         +0.0 to -0.0.
 
         When the two operands don't compare as LT, `y` is returned unchanged,
@@ -169,7 +169,7 @@ pub fn define(
         Floating point maximum with x86 semantics.
 
         This is equivalent to the C ternary operator `x > y ? x : y` which
-        differs from :inst:`fmax` when either operand is NaN or when comparing
+        differs from `fmax` when either operand is NaN or when comparing
         +0.0 to -0.0.
 
         When the two operands don't compare as GT, `y` is returned unchanged,
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index b89fa019b9..ed8f0b957c 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -168,7 +168,7 @@ pub fn define(
             r#"
         Fall through to the next EBB.
 
-        This is the same as :inst:`jump`, except the destination EBB must be
+        This is the same as `jump`, except the destination EBB must be
         the next one in the layout.
 
         Jumps are turned into fall-through instructions by the branch
@@ -187,7 +187,7 @@ pub fn define(
             r#"
         Branch when zero.
 
-        If ``c`` is a :type:`b1` value, take the branch when ``c`` is false. If
+        If ``c`` is a `b1` value, take the branch when ``c`` is false. If
         ``c`` is an integer value, take the branch when ``c = 0``.
         "#,
         )
@@ -201,7 +201,7 @@ pub fn define(
             r#"
         Branch when non-zero.
 
-        If ``c`` is a :type:`b1` value, take the branch when ``c`` is true. If
+        If ``c`` is a `b1` value, take the branch when ``c`` is true. If
         ``c`` is an integer value, take the branch when ``c != 0``.
         "#,
         )
@@ -215,7 +215,7 @@ pub fn define(
             r#"
         Compare scalar integers and branch.
 
-        Compare ``x`` and ``y`` in the same way as the :inst:`icmp` instruction
+        Compare ``x`` and ``y`` in the same way as the `icmp` instruction
         and take the branch if the condition is true::
 
             br_icmp ugt v1, v2, ebb4(v5, v6)
@@ -464,11 +464,7 @@ pub fn define(
         .is_terminator(true),
     );
 
-    let FN = &operand_doc(
-        "FN",
-        func_ref,
-        "function to call, declared by :inst:`function`",
-    );
+    let FN = &operand_doc("FN", func_ref, "function to call, declared by `function`");
     let args = &operand_doc("args", variable_args, "call arguments");
 
     ig.push(
@@ -500,7 +496,7 @@ pub fn define(
 
         Note that this is different from WebAssembly's ``call_indirect``; the
         callee is a native address, rather than a table index. For WebAssembly,
-        :inst:`table_addr` and :inst:`load` are used to obtain a native address
+        `table_addr` and `load` are used to obtain a native address
         from a table.
         "#,
         )
@@ -517,8 +513,8 @@ pub fn define(
 
         Compute the absolute address of a function declared in the preamble.
         The returned address can be used as a ``callee`` argument to
-        :inst:`call_indirect`. This is also a method for calling functions that
-        are too far away to be addressable by a direct :inst:`call`
+        `call_indirect`. This is also a method for calling functions that
+        are too far away to be addressable by a direct `call`
         instruction.
         "#,
         )
@@ -877,7 +873,7 @@ pub fn define(
 
         The offset is an immediate constant, not an SSA value. The memory
         access cannot go out of bounds, i.e.
-        :math:`sizeof(a) + Offset <= sizeof(SS)`.
+        `sizeof(a) + Offset <= sizeof(SS)`.
         "#,
         )
         .operands_in(vec![SS, Offset])
@@ -896,7 +892,7 @@ pub fn define(
 
         The offset is an immediate constant, not an SSA value. The memory
         access cannot go out of bounds, i.e.
-        :math:`sizeof(a) + Offset <= sizeof(SS)`.
+        `sizeof(a) + Offset <= sizeof(SS)`.
         "#,
         )
         .operands_in(vec![x, SS, Offset])
@@ -911,7 +907,7 @@ pub fn define(
 
         Compute the absolute address of a byte in a stack slot. The offset must
         refer to a byte inside the stack slot:
-        :math:`0 <= Offset < sizeof(SS)`.
+        `0 <= Offset < sizeof(SS)`.
         "#,
         )
         .operands_in(vec![SS, Offset])
@@ -1027,7 +1023,7 @@ pub fn define(
             r#"
         Floating point constant.
 
-        Create a :type:`f32` SSA value with an immediate constant value.
+        Create a `f32` SSA value with an immediate constant value.
         "#,
         )
         .operands_in(vec![N])
@@ -1043,7 +1039,7 @@ pub fn define(
             r#"
         Floating point constant.
 
-        Create a :type:`f64` SSA value with an immediate constant value.
+        Create a `f64` SSA value with an immediate constant value.
         "#,
         )
         .operands_in(vec![N])
@@ -1087,7 +1083,7 @@ pub fn define(
             r#"
         Conditional select.
 
-        This instruction selects whole values. Use :inst:`vselect` for
+        This instruction selects whole values. Use `vselect` for
         lane-wise selection.
         "#,
         )
@@ -1135,7 +1131,7 @@ pub fn define(
             r#"
         Spill a register value to a stack slot.
 
-        This instruction behaves exactly like :inst:`copy`, but the result
+        This instruction behaves exactly like `copy`, but the result
         value is assigned to a spill slot.
         "#,
         )
@@ -1150,7 +1146,7 @@ pub fn define(
             r#"
         Load a register value from a stack slot.
 
-        This instruction behaves exactly like :inst:`copy`, but creates a new
+        This instruction behaves exactly like `copy`, but creates a new
         SSA value for the spilled input value.
         "#,
         )
@@ -1270,7 +1266,7 @@ pub fn define(
             r#"
     Compare ``addr`` with the stack pointer and set the CPU flags.
 
-    This is like :inst:`ifcmp` where ``addr`` is the LHS operand and the stack
+    This is like `ifcmp` where ``addr`` is the LHS operand and the stack
     pointer is the RHS.
     "#,
         )
@@ -1289,7 +1285,7 @@ pub fn define(
         allocator to temporarily rearrange register assignments in order to
         satisfy instruction constraints.
 
-        See also :inst:`regmove`.
+        See also `regmove`.
         "#,
         )
         .operands_in(vec![x, src, SS])
@@ -1307,7 +1303,7 @@ pub fn define(
         allocator to temporarily rearrange register assignments in order to
         satisfy instruction constraints.
 
-        See also :inst:`regmove`.
+        See also `regmove`.
         "#,
         )
         .operands_in(vec![x, SS, dst])
@@ -1481,10 +1477,10 @@ pub fn define(
             r#"
         Compare scalar integer to a constant.
 
-        This is the same as the :inst:`icmp` instruction, except one operand is
+        This is the same as the `icmp` instruction, except one operand is
         an immediate constant.
 
-        This instruction can only compare scalars. Use :inst:`icmp` for
+        This instruction can only compare scalars. Use `icmp` for
         lane-wise vector comparisons.
         "#,
         )
@@ -1516,7 +1512,7 @@ pub fn define(
             r#"
         Compare scalar integer to a constant and return flags.
 
-        Like :inst:`icmp_imm`, but returns integer CPU flags instead of testing
+        Like `icmp_imm`, but returns integer CPU flags instead of testing
         a specific condition code.
         "#,
         )
@@ -1532,7 +1528,7 @@ pub fn define(
         Inst::new(
             "iadd",
             r#"
-        Wrapping integer addition: :math:`a := x + y \pmod{2^B}`.
+        Wrapping integer addition: `a := x + y \pmod{2^B}`.
 
         This instruction does not depend on the signed/unsigned interpretation
         of the operands.
@@ -1546,7 +1542,7 @@ pub fn define(
         Inst::new(
             "isub",
             r#"
-        Wrapping integer subtraction: :math:`a := x - y \pmod{2^B}`.
+        Wrapping integer subtraction: `a := x - y \pmod{2^B}`.
 
         This instruction does not depend on the signed/unsigned interpretation
         of the operands.
@@ -1560,7 +1556,7 @@ pub fn define(
         Inst::new(
             "imul",
             r#"
-        Wrapping integer multiplication: :math:`a := x y \pmod{2^B}`.
+        Wrapping integer multiplication: `a := x y \pmod{2^B}`.
 
         This instruction does not depend on the signed/unsigned interpretation
         of the
@@ -1607,7 +1603,7 @@ pub fn define(
         Inst::new(
             "udiv",
             r#"
-        Unsigned integer division: :math:`a := \lfloor {x \over y} \rfloor`.
+        Unsigned integer division: `a := \lfloor {x \over y} \rfloor`.
 
         This operation traps if the divisor is zero.
         "#,
@@ -1621,12 +1617,12 @@ pub fn define(
         Inst::new(
             "sdiv",
             r#"
-        Signed integer division rounded toward zero: :math:`a := sign(xy)
+        Signed integer division rounded toward zero: `a := sign(xy)
         \lfloor {|x| \over |y|}\rfloor`.
 
         This operation traps if the divisor is zero, or if the result is not
-        representable in :math:`B` bits two's complement. This only happens
-        when :math:`x = -2^{B-1}, y = -1`.
+        representable in `B` bits two's complement. This only happens
+        when `x = -2^{B-1}, y = -1`.
         "#,
         )
         .operands_in(vec![x, y])
@@ -1672,7 +1668,7 @@ pub fn define(
             r#"
         Add immediate integer.
 
-        Same as :inst:`iadd`, but one operand is an immediate constant.
+        Same as `iadd`, but one operand is an immediate constant.
 
         Polymorphic over all scalar integer types, but does not support vector
         types.
@@ -1716,8 +1712,8 @@ pub fn define(
         Signed integer division by an immediate constant.
 
         This operation traps if the divisor is zero, or if the result is not
-        representable in :math:`B` bits two's complement. This only happens
-        when :math:`x = -2^{B-1}, Y = -1`.
+        representable in `B` bits two's complement. This only happens
+        when `x = -2^{B-1}, Y = -1`.
         "#,
         )
         .operands_in(vec![x, Y])
@@ -1754,9 +1750,9 @@ pub fn define(
         Inst::new(
             "irsub_imm",
             r#"
-        Immediate reverse wrapping subtraction: :math:`a := Y - x \pmod{2^B}`.
+        Immediate reverse wrapping subtraction: `a := Y - x \pmod{2^B}`.
 
-        Also works as integer negation when :math:`Y = 0`. Use :inst:`iadd_imm`
+        Also works as integer negation when `Y = 0`. Use `iadd_imm`
         with a negative immediate operand for the reverse immediate
         subtraction.
 
@@ -1782,7 +1778,7 @@ pub fn define(
             r#"
         Add integers with carry in.
 
-        Same as :inst:`iadd` with an additional carry input. Computes:
+        Same as `iadd` with an additional carry input. Computes:
 
         .. math::
 
@@ -1802,7 +1798,7 @@ pub fn define(
             r#"
         Add integers with carry out.
 
-        Same as :inst:`iadd` with an additional carry output.
+        Same as `iadd` with an additional carry output.
 
         .. math::
 
@@ -1823,7 +1819,7 @@ pub fn define(
             r#"
         Add integers with carry in and out.
 
-        Same as :inst:`iadd` with an additional carry input and output.
+        Same as `iadd` with an additional carry input and output.
 
         .. math::
 
@@ -1844,7 +1840,7 @@ pub fn define(
             r#"
         Subtract integers with borrow in.
 
-        Same as :inst:`isub` with an additional borrow flag input. Computes:
+        Same as `isub` with an additional borrow flag input. Computes:
 
         .. math::
 
@@ -1864,7 +1860,7 @@ pub fn define(
             r#"
         Subtract integers with borrow out.
 
-        Same as :inst:`isub` with an additional borrow flag output.
+        Same as `isub` with an additional borrow flag output.
 
         .. math::
 
@@ -1885,7 +1881,7 @@ pub fn define(
             r#"
         Subtract integers with borrow in and out.
 
-        Same as :inst:`isub` with an additional borrow flag input and output.
+        Same as `isub` with an additional borrow flag input and output.
 
         .. math::
 
@@ -2008,7 +2004,7 @@ pub fn define(
             r#"
         Bitwise and with immediate.
 
-        Same as :inst:`band`, but one operand is an immediate constant.
+        Same as `band`, but one operand is an immediate constant.
 
         Polymorphic over all scalar integer types, but does not support vector
         types.
@@ -2024,7 +2020,7 @@ pub fn define(
             r#"
         Bitwise or with immediate.
 
-        Same as :inst:`bor`, but one operand is an immediate constant.
+        Same as `bor`, but one operand is an immediate constant.
 
         Polymorphic over all scalar integer types, but does not support vector
         types.
@@ -2040,7 +2036,7 @@ pub fn define(
             r#"
         Bitwise xor with immediate.
 
-        Same as :inst:`bxor`, but one operand is an immediate constant.
+        Same as `bxor`, but one operand is an immediate constant.
 
         Polymorphic over all scalar integer types, but does not support vector
         types.
@@ -2296,12 +2292,12 @@ pub fn define(
 
         == ==========================================
         UN Unordered when one or both numbers is NaN.
-        EQ When :math:`x = y`. (And :math:`0.0 = -0.0`).
-        LT When :math:`x < y`.
-        GT When :math:`x > y`.
+        EQ When `x = y`. (And `0.0 = -0.0`).
+        LT When `x < y`.
+        GT When `x > y`.
         == ==========================================
 
-        The 14 :type:`floatcc` condition codes each correspond to a subset of
+        The 14 `floatcc` condition codes each correspond to a subset of
         the four relations, except for the empty set which would always be
         false, and the full set which would always be true.
 
@@ -2329,7 +2325,7 @@ pub fn define(
         The standard C comparison operators, `<, <=, >, >=`, are all ordered,
         so they are false if either operand is NaN. The C equality operator,
         `==`, is ordered, and since inequality is defined as the logical
-        inverse it is *unordered*. They map to the :type:`floatcc` condition
+        inverse it is *unordered*. They map to the `floatcc` condition
         codes as follows:
 
         ==== ====== ============
@@ -2362,7 +2358,7 @@ pub fn define(
             r#"
         Floating point comparison returning flags.
 
-        Compares two numbers like :inst:`fcmp`, but returns floating point CPU
+        Compares two numbers like `fcmp`, but returns floating point CPU
         flags instead of testing a specific condition.
         "#,
         )
@@ -2414,8 +2410,8 @@ pub fn define(
             r#"
         Floating point division.
 
-        Unlike the integer division instructions :clif:inst:`sdiv` and
-        :clif:inst:`udiv`, this can't trap. Division by zero is infinity or
+        Unlike the integer division instructions ` and
+        `udiv`, this can't trap. Division by zero is infinity or
         NaN, depending on the dividend.
         "#,
         )
@@ -2440,7 +2436,7 @@ pub fn define(
             r#"
         Floating point fused multiply-and-add.
 
-        Computes :math:`a := xy+z` without any intermediate rounding of the
+        Computes `a := xy+z` without any intermediate rounding of the
         product.
         "#,
         )
@@ -2759,7 +2755,7 @@ pub fn define(
 
         Each lane in `x` is converted to a smaller integer type by discarding
         the most significant bits. This is the same as reducing modulo
-        :math:`2^n`.
+        `2^n`.
 
         The result type must have the same number of vector lanes as the input,
         and each lane must not have more bits that the input lanes. If the
@@ -2843,7 +2839,7 @@ pub fn define(
         This is an exact operation.
 
         Cranelift currently only supports two floating point formats
-        - :type:`f32` and :type:`f64`. This may change in the future.
+        - `f32` and `f64`. This may change in the future.
 
         The result type must have the same number of vector lanes as the input,
         and the result lanes must not have fewer bits than the input lanes. If
@@ -2865,7 +2861,7 @@ pub fn define(
         by rounding to nearest, ties to even.
 
         Cranelift currently only supports two floating point formats
-        - :type:`f32` and :type:`f64`. This may change in the future.
+        - `f32` and `f64`. This may change in the future.
 
         The result type must have the same number of vector lanes as the input,
         and the result lanes must not have more bits than the input lanes. If

From bfc146868841b3aae8c5c737fda19d61b31c6230 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 10 Jul 2019 17:19:57 +0200
Subject: [PATCH 2513/3084] [docs] Don't accidentally generate doc tests;

---
 .../codegen/meta/src/shared/instructions.rs   | 42 +++++++++++--------
 1 file changed, 24 insertions(+), 18 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index ed8f0b957c..8aaa71947c 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -216,14 +216,18 @@ pub fn define(
         Compare scalar integers and branch.
 
         Compare ``x`` and ``y`` in the same way as the `icmp` instruction
-        and take the branch if the condition is true::
+        and take the branch if the condition is true:
 
+        ```text
             br_icmp ugt v1, v2, ebb4(v5, v6)
+        ```
 
-        is semantically equivalent to::
+        is semantically equivalent to:
 
+        ```text
             v10 = icmp ugt, v1, v2
             brnz v10, ebb4(v5, v6)
+        ```
 
         Some RISC architectures like MIPS and RISC-V provide instructions that
         implement all or some of the condition codes. The instruction can also
@@ -1780,9 +1784,9 @@ pub fn define(
 
         Same as `iadd` with an additional carry input. Computes:
 
-        .. math::
-
+        ```text
             a = x + y + c_{in} \pmod 2^B
+        ```
 
         Polymorphic over all scalar integer types, but does not support vector
         types.
@@ -1800,10 +1804,10 @@ pub fn define(
 
         Same as `iadd` with an additional carry output.
 
-        .. math::
-
+        ```text
             a &= x + y \pmod 2^B \\
             c_{out} &= x+y >= 2^B
+        ```
 
         Polymorphic over all scalar integer types, but does not support vector
         types.
@@ -1821,10 +1825,10 @@ pub fn define(
 
         Same as `iadd` with an additional carry input and output.
 
-        .. math::
-
+        ```text
             a &= x + y + c_{in} \pmod 2^B \\
             c_{out} &= x + y + c_{in} >= 2^B
+        ```
 
         Polymorphic over all scalar integer types, but does not support vector
         types.
@@ -1842,9 +1846,9 @@ pub fn define(
 
         Same as `isub` with an additional borrow flag input. Computes:
 
-        .. math::
-
+        ```text
             a = x - (y + b_{in}) \pmod 2^B
+        ```
 
         Polymorphic over all scalar integer types, but does not support vector
         types.
@@ -1862,10 +1866,10 @@ pub fn define(
 
         Same as `isub` with an additional borrow flag output.
 
-        .. math::
-
+        ```text
             a &= x - y \pmod 2^B \\
             b_{out} &= x < y
+        ```
 
         Polymorphic over all scalar integer types, but does not support vector
         types.
@@ -1883,10 +1887,10 @@ pub fn define(
 
         Same as `isub` with an additional borrow flag input and output.
 
-        .. math::
-
+        ```text
             a &= x - (y + b_{in}) \pmod 2^B \\
             b_{out} &= x < y + b_{in}
+        ```
 
         Polymorphic over all scalar integer types, but does not support vector
         types.
@@ -2110,9 +2114,10 @@ pub fn define(
 
         When shifting a B-bits integer type, this instruction computes:
 
-        .. math::
-            s &:= y \pmod B,                \\
+        ```text
+            s &:= y \pmod B,
             a &:= x \cdot 2^s \pmod{2^B}.
+        ```
         "#,
         )
         .operands_in(vec![x, y])
@@ -2131,9 +2136,10 @@ pub fn define(
 
         When shifting a B-bits integer type, this instruction computes:
 
-        .. math::
-            s &:= y \pmod B,                \\
+        ```text
+            s &:= y \pmod B,
             a &:= \lfloor x \cdot 2^{-s} \rfloor.
+        ```
         "#,
         )
         .operands_in(vec![x, y])

From 6a19866da23f7dd05d9dc019bbace638d837af20 Mon Sep 17 00:00:00 2001
From: Benson Chau 
Date: Thu, 23 May 2019 16:14:45 -0400
Subject: [PATCH 2514/3084] issue #772: added an memmap replacement to support
 selinux

---
 cranelift/simplejit/Cargo.toml    |  5 ++
 cranelift/simplejit/src/memory.rs | 86 +++++++++++++++++++++++++++----
 2 files changed, 80 insertions(+), 11 deletions(-)

diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index a8edcd3328..b584cc490f 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -17,10 +17,15 @@ region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
 target-lexicon = { version = "0.4.0" }
+memmap = { version = "0.7.0", optional = true } 
 
 [target.'cfg(target_os = "windows")'.dependencies]
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }
 
+[features]
+selinux-fix = ['memmap']
+default = []
+
 [dev-dependencies]
 cranelift = { path = "../cranelift-umbrella", version = "0.34.0" }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.34.0" }
diff --git a/cranelift/simplejit/src/memory.rs b/cranelift/simplejit/src/memory.rs
index 0730e086ef..fde0f88871 100644
--- a/cranelift/simplejit/src/memory.rs
+++ b/cranelift/simplejit/src/memory.rs
@@ -1,5 +1,12 @@
+#[cfg(not(feature = "selinux-fix"))]
 use errno;
+
+#[cfg(not(feature = "selinux-fix"))]
 use libc;
+
+#[cfg(feature = "selinux-fix")]
+use memmap::MmapMut;
+
 use region;
 use std::mem;
 use std::ptr;
@@ -11,6 +18,9 @@ fn round_up_to_page_size(size: usize, page_size: usize) -> usize {
 
 /// A simple struct consisting of a pointer and length.
 struct PtrLen {
+    #[cfg(feature = "selinux-fix")]
+    map: Option,
+
     ptr: *mut u8,
     len: usize,
 }
@@ -19,6 +29,9 @@ impl PtrLen {
     /// Create a new empty `PtrLen`.
     fn new() -> Self {
         Self {
+            #[cfg(feature = "selinux-fix")]
+            map: None,
+
             ptr: ptr::null_mut(),
             len: 0,
         }
@@ -26,13 +39,34 @@ impl PtrLen {
 
     /// Create a new `PtrLen` pointing to at least `size` bytes of memory,
     /// suitably sized and aligned for memory protection.
-    #[cfg(not(target_os = "windows"))]
+    #[cfg(all(not(target_os = "windows"), feature = "selinux-fix"))]
+    fn with_size(size: usize) -> Result {
+        let page_size = region::page::size();
+        let alloc_size = round_up_to_page_size(size, page_size);
+        let map = MmapMut::map_anon(alloc_size);
+
+        match map {
+            Ok(mut map) => {
+                // The order here is important; we assign the pointer first to get
+                // around compile time borrow errors.
+                Ok(Self {
+                    ptr: map.as_mut_ptr(),
+                    map: Some(map),
+                    len: alloc_size,
+                })
+            }
+            Err(e) => Err(e.to_string()),
+        }
+    }
+
+    #[cfg(all(not(target_os = "windows"), not(feature = "selinux-fix")))]
     fn with_size(size: usize) -> Result {
         let mut ptr = ptr::null_mut();
         let page_size = region::page::size();
         let alloc_size = round_up_to_page_size(size, page_size);
         unsafe {
             let err = libc::posix_memalign(&mut ptr, page_size, alloc_size);
+
             if err == 0 {
                 Ok(Self {
                     ptr: ptr as *mut u8,
@@ -122,11 +156,26 @@ impl Memory {
     pub fn set_readable_and_executable(&mut self) {
         self.finish_current();
 
-        for &PtrLen { ptr, len } in &self.allocations[self.executable..] {
-            if len != 0 {
-                unsafe {
-                    region::protect(ptr, len, region::Protection::ReadExecute)
-                        .expect("unable to make memory readable+executable");
+        #[cfg(feature = "selinux-fix")]
+        {
+            for &PtrLen { ref map, ptr, len } in &self.allocations[self.executable..] {
+                if len != 0 && map.is_some() {
+                    unsafe {
+                        region::protect(ptr, len, region::Protection::ReadExecute)
+                            .expect("unable to make memory readable+executable");
+                    }
+                }
+            }
+        }
+
+        #[cfg(not(feature = "selinux-fix"))]
+        {
+            for &PtrLen { ptr, len } in &self.allocations[self.executable..] {
+                if len != 0 {
+                    unsafe {
+                        region::protect(ptr, len, region::Protection::ReadExecute)
+                            .expect("unable to make memory readable+executable");
+                    }
                 }
             }
         }
@@ -136,11 +185,26 @@ impl Memory {
     pub fn set_readonly(&mut self) {
         self.finish_current();
 
-        for &PtrLen { ptr, len } in &self.allocations[self.executable..] {
-            if len != 0 {
-                unsafe {
-                    region::protect(ptr, len, region::Protection::Read)
-                        .expect("unable to make memory readonly");
+        #[cfg(feature = "selinux-fix")]
+        {
+            for &PtrLen { ref map, ptr, len } in &self.allocations[self.executable..] {
+                if len != 0 && map.is_some() {
+                    unsafe {
+                        region::protect(ptr, len, region::Protection::Read)
+                            .expect("unable to make memory readonly");
+                    }
+                }
+            }
+        }
+
+        #[cfg(not(feature = "selinux-fix"))]
+        {
+            for &PtrLen { ptr, len } in &self.allocations[self.executable..] {
+                if len != 0 {
+                    unsafe {
+                        region::protect(ptr, len, region::Protection::Read)
+                            .expect("unable to make memory readonly");
+                    }
                 }
             }
         }

From f856b124fd0d98ddf81ae6feb9bbed25da072bd1 Mon Sep 17 00:00:00 2001
From: Mark McCaskey 
Date: Fri, 12 Jul 2019 02:21:00 -0700
Subject: [PATCH 2515/3084] Use Default trait for Position and
 DisplayFunctionAnnotations (#843)

---
 cranelift/codegen/src/ir/function.rs | 11 +----------
 cranelift/frontend/src/frontend.rs   |  8 +-------
 2 files changed, 2 insertions(+), 17 deletions(-)

diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index f4b62dc59c..c856f70e54 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -222,6 +222,7 @@ impl Function {
 }
 
 /// Additional annotations for function display.
+#[derive(Default)]
 pub struct DisplayFunctionAnnotations<'a> {
     /// Enable ISA annotations.
     pub isa: Option<&'a dyn TargetIsa>,
@@ -230,16 +231,6 @@ pub struct DisplayFunctionAnnotations<'a> {
     pub value_ranges: Option<&'a ValueLabelsRanges>,
 }
 
-impl<'a> DisplayFunctionAnnotations<'a> {
-    /// Create a DisplayFunctionAnnotations with all fields set to None.
-    pub fn default() -> Self {
-        DisplayFunctionAnnotations {
-            isa: None,
-            value_ranges: None,
-        }
-    }
-}
-
 impl<'a> From> for DisplayFunctionAnnotations<'a> {
     fn from(isa: Option<&'a dyn TargetIsa>) -> DisplayFunctionAnnotations {
         DisplayFunctionAnnotations {
diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 82a3a09f1a..355b460067 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -46,6 +46,7 @@ struct EbbData {
     user_param_count: usize,
 }
 
+#[derive(Default)]
 struct Position {
     ebb: PackedOption,
     basic_block: PackedOption,
@@ -59,13 +60,6 @@ impl Position {
         }
     }
 
-    fn default() -> Self {
-        Self {
-            ebb: PackedOption::default(),
-            basic_block: PackedOption::default(),
-        }
-    }
-
     fn is_default(&self) -> bool {
         self.ebb.is_none() && self.basic_block.is_none()
     }

From 8edc40cb490b0360ff4fe94352a7b9fbfac05b28 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 12 Jul 2019 14:20:26 +0200
Subject: [PATCH 2516/3084] BB-like manual legalization for x86 ISA

---
 cranelift/codegen/src/isa/x86/enc_tables.rs   | 74 ++++++++++++++++++-
 .../filetests/isa/x86/legalize-custom.clif    |  2 +-
 2 files changed, 74 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 832058c1e0..e0fc05178d 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -174,6 +174,9 @@ fn expand_sdivrem(
         return;
     }
 
+    // EBB handling the nominal case.
+    let nominal = pos.func.dfg.make_ebb();
+
     // EBB handling the -1 divisor case.
     let minus_one = pos.func.dfg.make_ebb();
 
@@ -186,9 +189,11 @@ fn expand_sdivrem(
     // Start by checking for a -1 divisor which needs to be handled specially.
     let is_m1 = pos.ins().ifcmp_imm(y, -1);
     pos.ins().brif(IntCC::Equal, is_m1, minus_one, &[]);
+    pos.ins().jump(nominal, &[]);
 
     // Now it is safe to execute the `x86_sdivmodx` instruction which will still trap on division
     // by zero.
+    pos.insert_ebb(nominal);
     let xhi = pos.ins().sshr_imm(x, i64::from(ty.lane_bits()) - 1);
     let (quot, rem) = pos.ins().x86_sdivmodx(x, xhi, y);
     let divres = if is_srem { rem } else { quot };
@@ -217,6 +222,7 @@ fn expand_sdivrem(
     pos.insert_ebb(done);
 
     cfg.recompute_ebb(pos.func, old_ebb);
+    cfg.recompute_ebb(pos.func, nominal);
     cfg.recompute_ebb(pos.func, minus_one);
     cfg.recompute_ebb(pos.func, done);
 }
@@ -301,12 +307,18 @@ fn expand_minmax(
     //    fmin(0.0, -0.0) -> -0.0 and fmax(0.0, -0.0) -> 0.0.
     // 3. UN: We need to produce a quiet NaN that is canonical if the inputs are canonical.
 
+    // EBB handling case 1) where operands are ordered but not equal.
+    let one_ebb = func.dfg.make_ebb();
+
     // EBB handling case 3) where one operand is NaN.
     let uno_ebb = func.dfg.make_ebb();
 
     // EBB that handles the unordered or equal cases 2) and 3).
     let ueq_ebb = func.dfg.make_ebb();
 
+    // EBB handling case 2) where operands are ordered and equal.
+    let eq_ebb = func.dfg.make_ebb();
+
     // Final EBB with one argument representing the final result value.
     let done = func.dfg.make_ebb();
 
@@ -327,8 +339,10 @@ fn expand_minmax(
     pos.use_srcloc(inst);
     let cmp_ueq = pos.ins().fcmp(FloatCC::UnorderedOrEqual, x, y);
     pos.ins().brnz(cmp_ueq, ueq_ebb, &[]);
+    pos.ins().jump(one_ebb, &[]);
 
     // Handle the common ordered, not equal (LT|GT) case.
+    pos.insert_ebb(one_ebb);
     let one_inst = pos.ins().Binary(x86_opc, ty, x, y).0;
     let one_result = pos.func.dfg.first_result(one_inst);
     pos.ins().jump(done, &[one_result]);
@@ -346,9 +360,11 @@ fn expand_minmax(
     // TODO: When we get support for flag values, we can reuse the above comparison.
     let cmp_uno = pos.ins().fcmp(FloatCC::Unordered, x, y);
     pos.ins().brnz(cmp_uno, uno_ebb, &[]);
+    pos.ins().jump(eq_ebb, &[]);
 
     // We are now in case 2) where x and y compare EQ.
     // We need a bitwise operation to get the sign right.
+    pos.insert_ebb(eq_ebb);
     let bw_inst = pos.ins().Binary(bitwise_opc, ty, x, y).0;
     let bw_result = pos.func.dfg.first_result(bw_inst);
     // This should become a fall-through for this second most common case.
@@ -360,8 +376,10 @@ fn expand_minmax(
     pos.insert_ebb(done);
 
     cfg.recompute_ebb(pos.func, old_ebb);
-    cfg.recompute_ebb(pos.func, ueq_ebb);
+    cfg.recompute_ebb(pos.func, one_ebb);
     cfg.recompute_ebb(pos.func, uno_ebb);
+    cfg.recompute_ebb(pos.func, ueq_ebb);
+    cfg.recompute_ebb(pos.func, eq_ebb);
     cfg.recompute_ebb(pos.func, done);
 }
 
@@ -397,6 +415,9 @@ fn expand_fcvt_from_uint(
 
     let old_ebb = pos.func.layout.pp_ebb(inst);
 
+    // EBB handling the case where x >= 0.
+    let poszero_ebb = pos.func.dfg.make_ebb();
+
     // EBB handling the case where x < 0.
     let neg_ebb = pos.func.dfg.make_ebb();
 
@@ -410,8 +431,10 @@ fn expand_fcvt_from_uint(
     // If x as a signed int is not negative, we can use the existing `fcvt_from_sint` instruction.
     let is_neg = pos.ins().icmp_imm(IntCC::SignedLessThan, x, 0);
     pos.ins().brnz(is_neg, neg_ebb, &[]);
+    pos.ins().jump(poszero_ebb, &[]);
 
     // Easy case: just use a signed conversion.
+    pos.insert_ebb(poszero_ebb);
     let posres = pos.ins().fcvt_from_sint(ty, x);
     pos.ins().jump(done, &[posres]);
 
@@ -434,6 +457,7 @@ fn expand_fcvt_from_uint(
     pos.insert_ebb(done);
 
     cfg.recompute_ebb(pos.func, old_ebb);
+    cfg.recompute_ebb(pos.func, poszero_ebb);
     cfg.recompute_ebb(pos.func, neg_ebb);
     cfg.recompute_ebb(pos.func, done);
 }
@@ -461,6 +485,9 @@ fn expand_fcvt_to_sint(
     // Final EBB after the bad value checks.
     let done = func.dfg.make_ebb();
 
+    // EBB for checking failure cases.
+    let maybe_trap_ebb = func.dfg.make_ebb();
+
     // The `x86_cvtt2si` performs the desired conversion, but it doesn't trap on NaN or overflow.
     // It produces an INT_MIN result instead.
     func.dfg.replace(inst).x86_cvtt2si(ty, x);
@@ -472,6 +499,7 @@ fn expand_fcvt_to_sint(
         .ins()
         .icmp_imm(IntCC::NotEqual, result, 1 << (ty.lane_bits() - 1));
     pos.ins().brnz(is_done, done, &[]);
+    pos.ins().jump(maybe_trap_ebb, &[]);
 
     // We now have the following possibilities:
     //
@@ -479,6 +507,7 @@ fn expand_fcvt_to_sint(
     // 2. The input was NaN -> trap bad_toint
     // 3. The input was out of range -> trap int_ovf
     //
+    pos.insert_ebb(maybe_trap_ebb);
 
     // Check for NaN.
     let is_nan = pos.ins().fcmp(FloatCC::Unordered, x, x);
@@ -530,6 +559,7 @@ fn expand_fcvt_to_sint(
     pos.insert_ebb(done);
 
     cfg.recompute_ebb(pos.func, old_ebb);
+    cfg.recompute_ebb(pos.func, maybe_trap_ebb);
     cfg.recompute_ebb(pos.func, done);
 }
 
@@ -559,6 +589,9 @@ fn expand_fcvt_to_sint_sat(
 
     // Final EBB after the bad value checks.
     let done_ebb = func.dfg.make_ebb();
+    let intmin_ebb = func.dfg.make_ebb();
+    let minsat_ebb = func.dfg.make_ebb();
+    let maxsat_ebb = func.dfg.make_ebb();
     func.dfg.clear_results(inst);
     func.dfg.attach_ebb_param(done_ebb, result);
 
@@ -573,20 +606,24 @@ fn expand_fcvt_to_sint_sat(
         .ins()
         .icmp_imm(IntCC::NotEqual, cvtt2si, 1 << (ty.lane_bits() - 1));
     pos.ins().brnz(is_done, done_ebb, &[cvtt2si]);
+    pos.ins().jump(intmin_ebb, &[]);
 
     // We now have the following possibilities:
     //
     // 1. INT_MIN was actually the correct conversion result.
     // 2. The input was NaN -> replace the result value with 0.
     // 3. The input was out of range -> saturate the result to the min/max value.
+    pos.insert_ebb(intmin_ebb);
 
     // Check for NaN, which is truncated to 0.
     let zero = pos.ins().iconst(ty, 0);
     let is_nan = pos.ins().fcmp(FloatCC::Unordered, x, x);
     pos.ins().brnz(is_nan, done_ebb, &[zero]);
+    pos.ins().jump(minsat_ebb, &[]);
 
     // Check for case 1: INT_MIN is the correct result.
     // Determine the smallest floating point number that would convert to INT_MIN.
+    pos.insert_ebb(minsat_ebb);
     let mut overflow_cc = FloatCC::LessThan;
     let output_bits = ty.lane_bits();
     let flimit = match xty {
@@ -623,8 +660,10 @@ fn expand_fcvt_to_sint_sat(
     };
     let min_value = pos.ins().iconst(ty, min_imm);
     pos.ins().brnz(overflow, done_ebb, &[min_value]);
+    pos.ins().jump(maxsat_ebb, &[]);
 
     // Finally, we could have a positive value that is too large.
+    pos.insert_ebb(maxsat_ebb);
     let fzero = match xty {
         ir::types::F32 => pos.ins().f32const(Ieee32::with_bits(0)),
         ir::types::F64 => pos.ins().f64const(Ieee64::with_bits(0)),
@@ -649,6 +688,9 @@ fn expand_fcvt_to_sint_sat(
     pos.insert_ebb(done_ebb);
 
     cfg.recompute_ebb(pos.func, old_ebb);
+    cfg.recompute_ebb(pos.func, intmin_ebb);
+    cfg.recompute_ebb(pos.func, minsat_ebb);
+    cfg.recompute_ebb(pos.func, maxsat_ebb);
     cfg.recompute_ebb(pos.func, done_ebb);
 }
 
@@ -673,6 +715,12 @@ fn expand_fcvt_to_uint(
     let result = func.dfg.first_result(inst);
     let ty = func.dfg.value_type(result);
 
+    // EBB handle numbers < 2^(N-1).
+    let below_uint_max_ebb = func.dfg.make_ebb();
+
+    // EBB handle numbers < 0.
+    let below_zero_ebb = func.dfg.make_ebb();
+
     // EBB handling numbers >= 2^(N-1).
     let large = func.dfg.make_ebb();
 
@@ -696,9 +744,11 @@ fn expand_fcvt_to_uint(
     let is_large = pos.ins().ffcmp(x, pow2nm1);
     pos.ins()
         .brff(FloatCC::GreaterThanOrEqual, is_large, large, &[]);
+    pos.ins().jump(below_uint_max_ebb, &[]);
 
     // We need to generate a specific trap code when `x` is NaN, so reuse the flags from the
     // previous comparison.
+    pos.insert_ebb(below_uint_max_ebb);
     pos.ins().trapff(
         FloatCC::Unordered,
         is_large,
@@ -710,6 +760,9 @@ fn expand_fcvt_to_uint(
     let is_neg = pos.ins().ifcmp_imm(sres, 0);
     pos.ins()
         .brif(IntCC::SignedGreaterThanOrEqual, is_neg, done, &[sres]);
+    pos.ins().jump(below_zero_ebb, &[]);
+
+    pos.insert_ebb(below_zero_ebb);
     pos.ins().trap(ir::TrapCode::IntegerOverflow);
 
     // Handle the case where x >= 2^(N-1) and not NaN.
@@ -729,6 +782,8 @@ fn expand_fcvt_to_uint(
     pos.insert_ebb(done);
 
     cfg.recompute_ebb(pos.func, old_ebb);
+    cfg.recompute_ebb(pos.func, below_uint_max_ebb);
+    cfg.recompute_ebb(pos.func, below_zero_ebb);
     cfg.recompute_ebb(pos.func, large);
     cfg.recompute_ebb(pos.func, done);
 }
@@ -757,9 +812,16 @@ fn expand_fcvt_to_uint_sat(
     let result = func.dfg.first_result(inst);
     let ty = func.dfg.value_type(result);
 
+    // EBB handle numbers < 2^(N-1).
+    let below_pow2nm1_or_nan_ebb = func.dfg.make_ebb();
+    let below_pow2nm1_ebb = func.dfg.make_ebb();
+
     // EBB handling numbers >= 2^(N-1).
     let large = func.dfg.make_ebb();
 
+    // EBB handling numbers < 2^N.
+    let uint_large_ebb = func.dfg.make_ebb();
+
     // Final EBB after the bad value checks.
     let done = func.dfg.make_ebb();
 
@@ -781,12 +843,16 @@ fn expand_fcvt_to_uint_sat(
     let is_large = pos.ins().ffcmp(x, pow2nm1);
     pos.ins()
         .brff(FloatCC::GreaterThanOrEqual, is_large, large, &[]);
+    pos.ins().jump(below_pow2nm1_or_nan_ebb, &[]);
 
     // We need to generate zero when `x` is NaN, so reuse the flags from the previous comparison.
+    pos.insert_ebb(below_pow2nm1_or_nan_ebb);
     pos.ins().brff(FloatCC::Unordered, is_large, done, &[zero]);
+    pos.ins().jump(below_pow2nm1_ebb, &[]);
 
     // Now we know that x < 2^(N-1) and not NaN. If the result of the cvtt2si is positive, we're
     // done; otherwise saturate to the minimum unsigned value, that is 0.
+    pos.insert_ebb(below_pow2nm1_ebb);
     let sres = pos.ins().x86_cvtt2si(ty, x);
     let is_neg = pos.ins().ifcmp_imm(sres, 0);
     pos.ins()
@@ -808,6 +874,9 @@ fn expand_fcvt_to_uint_sat(
     let is_neg = pos.ins().ifcmp_imm(lres, 0);
     pos.ins()
         .brif(IntCC::SignedLessThan, is_neg, done, &[max_value]);
+    pos.ins().jump(uint_large_ebb, &[]);
+
+    pos.insert_ebb(uint_large_ebb);
     let lfinal = pos.ins().iadd_imm(lres, 1 << (ty.lane_bits() - 1));
 
     // Recycle the original instruction as a jump.
@@ -818,6 +887,9 @@ fn expand_fcvt_to_uint_sat(
     pos.insert_ebb(done);
 
     cfg.recompute_ebb(pos.func, old_ebb);
+    cfg.recompute_ebb(pos.func, below_pow2nm1_or_nan_ebb);
+    cfg.recompute_ebb(pos.func, below_pow2nm1_ebb);
     cfg.recompute_ebb(pos.func, large);
+    cfg.recompute_ebb(pos.func, uint_large_ebb);
     cfg.recompute_ebb(pos.func, done);
 }
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-custom.clif b/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
index 51e7ca487d..747f4e819b 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
@@ -84,7 +84,7 @@ function %f32_min(f32, f32) -> f32 {
 ebb0(v0: f32, v1: f32):
     v2 = fmin v0, v1
     return v2
-    ; check: $(vnat=$V) = x86_fmin v0, v1
+    ; check: $(vnat=$V) = x86_fmin.f32 v0, v1
     ; nextln: jump $(done=$EBB)($vnat)
 
     ; check: $(uno=$EBB):

From 9e884b4433d142718b78b79660fdc1267d682d77 Mon Sep 17 00:00:00 2001
From: Artur Jamro 
Date: Fri, 12 Jul 2019 15:30:50 -0700
Subject: [PATCH 2517/3084] Add support for some serde serialization (#847)

* Add support for some serde serialization
---
 cranelift/codegen/Cargo.toml            |  4 ++++
 cranelift/codegen/src/binemit/mod.rs    |  3 +++
 cranelift/codegen/src/ir/entities.rs    |  3 +++
 cranelift/codegen/src/ir/libcall.rs     |  3 +++
 cranelift/codegen/src/ir/sourceloc.rs   |  3 +++
 cranelift/entity/src/map.rs             | 12 ++++++++++++
 cranelift/wasm/Cargo.toml               |  2 ++
 cranelift/wasm/src/translation_utils.rs |  3 +++
 8 files changed, 33 insertions(+)

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 5cde9d47df..e088e6f84a 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -20,6 +20,7 @@ failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
+serde = { version = "1.0.94", features = ["derive"], optional = true }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
 # machine code. Integration tests that need external dependencies can be
@@ -58,6 +59,9 @@ arm32 = []
 arm64 = []
 riscv = []
 
+# For dependent crates that want to serialize some parts of cranelift
+enable-serde = ["serde"]
+
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/cranelift/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
index 21b07587ed..b0408e97e7 100644
--- a/cranelift/codegen/src/binemit/mod.rs
+++ b/cranelift/codegen/src/binemit/mod.rs
@@ -14,6 +14,8 @@ pub use crate::regalloc::RegDiversions;
 
 use crate::ir::{ExternalName, Function, Inst, JumpTable, SourceLoc, TrapCode};
 use core::fmt;
+#[cfg(feature = "enable-serde")]
+use serde::{Deserialize, Serialize};
 
 /// Offset in bytes from the beginning of the function.
 ///
@@ -26,6 +28,7 @@ pub type Addend = i64;
 
 /// Relocation kinds for every ISA
 #[derive(Copy, Clone, Debug)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub enum Reloc {
     /// absolute 4-byte
     Abs4,
diff --git a/cranelift/codegen/src/ir/entities.rs b/cranelift/codegen/src/ir/entities.rs
index 2c05fbc84b..a2a405bc46 100644
--- a/cranelift/codegen/src/ir/entities.rs
+++ b/cranelift/codegen/src/ir/entities.rs
@@ -22,6 +22,8 @@
 use crate::entity::entity_impl;
 use core::fmt;
 use core::u32;
+#[cfg(feature = "enable-serde")]
+use serde::{Deserialize, Serialize};
 
 /// An opaque reference to an extended basic block in a function.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
@@ -103,6 +105,7 @@ impl GlobalValue {
 
 /// An opaque reference to a jump table.
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct JumpTable(u32);
 entity_impl!(JumpTable, "jt");
 
diff --git a/cranelift/codegen/src/ir/libcall.rs b/cranelift/codegen/src/ir/libcall.rs
index 275e22a91f..f544dd5732 100644
--- a/cranelift/codegen/src/ir/libcall.rs
+++ b/cranelift/codegen/src/ir/libcall.rs
@@ -7,6 +7,8 @@ use crate::ir::{
 use crate::isa::{CallConv, RegUnit, TargetIsa};
 use core::fmt;
 use core::str::FromStr;
+#[cfg(feature = "enable-serde")]
+use serde::{Deserialize, Serialize};
 
 /// The name of a runtime library routine.
 ///
@@ -17,6 +19,7 @@ use core::str::FromStr;
 ///
 /// This list is likely to grow over time.
 #[derive(Copy, Clone, Debug, PartialEq, Eq)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub enum LibCall {
     /// probe for stack overflow. These are emitted for functions which need
     /// when the `probestack_enabled` setting is true.
diff --git a/cranelift/codegen/src/ir/sourceloc.rs b/cranelift/codegen/src/ir/sourceloc.rs
index ab5722a57d..a0d051d3aa 100644
--- a/cranelift/codegen/src/ir/sourceloc.rs
+++ b/cranelift/codegen/src/ir/sourceloc.rs
@@ -4,6 +4,8 @@
 //! location when instructions are transformed.
 
 use core::fmt;
+#[cfg(feature = "enable-serde")]
+use serde::{Deserialize, Serialize};
 
 /// A source location.
 ///
@@ -13,6 +15,7 @@ use core::fmt;
 /// The default source location uses the all-ones bit pattern `!0`. It is used for instructions
 /// that can't be given a real source location.
 #[derive(Clone, Copy, Debug, PartialEq, Eq)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct SourceLoc(u32);
 
 impl SourceLoc {
diff --git a/cranelift/entity/src/map.rs b/cranelift/entity/src/map.rs
index 9410cc2f90..a0d31a4309 100644
--- a/cranelift/entity/src/map.rs
+++ b/cranelift/entity/src/map.rs
@@ -56,11 +56,23 @@ where
         }
     }
 
+    /// Returns the number of elements in the underlying vector.
+    ///
+    /// The number is not necessarily the same as the length of the corresponding PrimaryMap.
+    pub fn len(&self) -> usize {
+        self.elems.len()
+    }
+
     /// Get the element at `k` if it exists.
     pub fn get(&self, k: K) -> Option<&V> {
         self.elems.get(k.index())
     }
 
+    /// Get the default value.
+    pub fn get_default(&self) -> &V {
+        &self.default
+    }
+
     /// Is this map completely empty?
     pub fn is_empty(&self) -> bool {
         self.elems.is_empty()
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 53a64e7896..5f9bfc4737 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -19,6 +19,7 @@ hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
+serde = { version = "1.0.94", features = ["derive"], optional = true }
 
 [dev-dependencies]
 wabt = "0.7.0"
@@ -28,6 +29,7 @@ target-lexicon = "0.4.0"
 default = ["std"]
 std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std"]
 core = ["hashmap_core", "cranelift-codegen/core", "cranelift-frontend/core", "wasmparser/core"]
+enable-serde = ["serde"]
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index c221f751ff..759e4ab4ff 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -3,10 +3,13 @@ use crate::environ::{WasmError, WasmResult};
 use core::u32;
 use cranelift_codegen::entity::entity_impl;
 use cranelift_codegen::ir;
+#[cfg(feature = "enable-serde")]
+use serde::{Deserialize, Serialize};
 use wasmparser;
 
 /// Index type of a function (imported or defined) inside the WebAssembly module.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct FuncIndex(u32);
 entity_impl!(FuncIndex);
 

From a64ada7e303dfc6bf8460d3f040df86b450616ba Mon Sep 17 00:00:00 2001
From: Artur Jamro 
Date: Fri, 12 Jul 2019 15:42:00 -0700
Subject: [PATCH 2518/3084] Do not import libc on windows (#848)

* Do not import libc on windows
---
 cranelift/simplejit/src/backend.rs | 1 +
 cranelift/simplejit/src/memory.rs  | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index c182ada674..16612d0a39 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -8,6 +8,7 @@ use cranelift_module::{
     Backend, DataContext, DataDescription, Init, Linkage, ModuleNamespace, ModuleResult,
 };
 use cranelift_native;
+#[cfg(not(windows))]
 use libc;
 use std::collections::HashMap;
 use std::ffi::CString;
diff --git a/cranelift/simplejit/src/memory.rs b/cranelift/simplejit/src/memory.rs
index fde0f88871..c50bb480dc 100644
--- a/cranelift/simplejit/src/memory.rs
+++ b/cranelift/simplejit/src/memory.rs
@@ -1,7 +1,7 @@
 #[cfg(not(feature = "selinux-fix"))]
 use errno;
 
-#[cfg(not(feature = "selinux-fix"))]
+#[cfg(not(any(feature = "selinux-fix", windows)))]
 use libc;
 
 #[cfg(feature = "selinux-fix")]

From 67dd0b501564daa10eda670333a64bac6c1d628f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 12 Jul 2019 15:48:19 -0700
Subject: [PATCH 2519/3084] Bump version to 0.35.0

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    |  8 ++++----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 6fe8e44cda..7bb3bf55d4 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.34.0"
+version = "0.35.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.34.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.34.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.34.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.34.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.34.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.34.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.34.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.34.0" }
-cranelift-module = { path = "cranelift-module", version = "0.34.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.34.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.34.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.34.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.34.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.35.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.35.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.35.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.35.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.35.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.35.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.35.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.35.0" }
+cranelift-module = { path = "cranelift-module", version = "0.35.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.35.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.35.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.35.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.35.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 2308f5e827..bd4c741199 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.34.0"
+version = "0.35.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.34.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.35.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index e088e6f84a..aa579c4cde 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.34.0"
+version = "0.35.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.34.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.34.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.35.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.35.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -27,7 +27,7 @@ serde = { version = "1.0.94", features = ["derive"], optional = true }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.34.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.35.0", default-features = false }
 
 [features]
 default = ["std", "x86", "arm32", "arm64", "riscv"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 574e40db24..a20b7cd198 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.34.0"
+version = "0.35.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.34.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.35.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index d2dc9ab91d..0e390b9b7b 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.34.0"
+version = "0.35.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index cdc79f2397..7319bf3f7f 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.34.0"
+version = "0.35.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.34.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.35.0" }
 faerie = "0.10.0"
 goblin = "0.0.22"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index b16271017c..f0b544cee2 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.34.0"
+version = "0.35.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../cranelift-reader", version = "0.34.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.34.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../cranelift-reader", version = "0.35.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.35.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 97a706122b..fc6ca6eb31 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.34.0"
+version = "0.35.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index b62c947fb0..011c0830c9 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.34.0"
+version = "0.35.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.34.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.35.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 8e01226676..10a52c60ef 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.34.0"
+version = "0.35.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 5a8c53259c..fe2707d739 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.34.0"
+version = "0.35.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.34.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.35.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 3a1d78e715..99beb98348 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.34.0"
+version="0.35.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index ea226ec0fd..e2c48e859d 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.34.0"
+version = "0.35.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0" }
 target-lexicon = "0.4.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 8516812a16..5abfb104bb 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.34.0"
+version = "0.35.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.34.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.35.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index b584cc490f..1fcfc58860 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.34.0"
+version = "0.35.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.34.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.34.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.35.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.35.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.34.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.34.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.34.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.35.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.35.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.35.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index fc3089268f..a7cb8dc153 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.34.0"
+version = "0.35.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.34.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.35.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 5f9bfc4737..a1e28e8b03 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.34.0"
+version = "0.35.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.32.1", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.34.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.34.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.34.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.35.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.35.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 8378297f33d4ae6d261bb696f1c12313b1bb2d86 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 11 Jul 2019 10:09:58 -0700
Subject: [PATCH 2520/3084] Prepare legalizer codegen for SIMD features

Contains fixes from @bnjbvr to codegen as a part of https://github.com/bnjbvr/cranelift/pull/2; necessary for SIMD features to work
---
 cranelift/codegen/meta/src/gen_legalizer.rs | 57 +++++++++++++--------
 1 file changed, 36 insertions(+), 21 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index b09a06a333..b823bb5597 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -51,7 +51,7 @@ fn unwrap_inst(
 
     fmtln!(
         fmt,
-        "let ({}, predicate) = if let ir::InstructionData::{} {{",
+        "let ({}, predicate) = if let crate::ir::InstructionData::{} {{",
         arg_names,
         iform.name
     );
@@ -135,18 +135,16 @@ fn unwrap_inst(
                     .to_comment_string(var_pool)
             ));
 
-            fmt.line("let results = pos.func.dfg.inst_results(inst);");
+            fmt.line("let r = pos.func.dfg.inst_results(inst);");
             for (i, &var_index) in def.defined_vars.iter().enumerate() {
                 let var = var_pool.get(var_index);
-                fmtln!(fmt, "let {} = &results[{}];", var.name, i);
-                if var.has_free_typevar() {
-                    fmtln!(
-                        fmt,
-                        "let typeof_{} = pos.func.dfg.value_type(*{});",
-                        var.name,
-                        var.name
-                    );
-                }
+                fmtln!(fmt, "let {} = &r[{}];", var.name, i);
+                fmtln!(
+                    fmt,
+                    "let typeof_{} = pos.func.dfg.value_type(*{});",
+                    var.name,
+                    var.name
+                );
             }
 
             replace_inst = true;
@@ -187,11 +185,15 @@ fn build_derived_expr(tv: &TypeVar) -> String {
         Some(base) => base,
         None => {
             assert!(tv.name.starts_with("typeof_"));
-            return format!("{}", tv.name);
+            return format!("Some({})", tv.name);
         }
     };
     let base_expr = build_derived_expr(&base.type_var);
-    format!("{}.{}()", base_expr, base.derived_func.name())
+    format!(
+        "{}.map(|t: crate::ir::Type| t.{}())",
+        base_expr,
+        base.derived_func.name()
+    )
 }
 
 /// Emit rust code for the given check.
@@ -221,18 +223,29 @@ fn emit_runtime_typecheck<'a, 'b>(
         Constraint::Eq(tv1, tv2) => {
             fmtln!(
                 fmt,
-                "let predicate = predicate && {} == {};",
+                "let predicate = predicate && match ({}, {}) {{",
                 build_derived_expr(tv1),
                 build_derived_expr(tv2)
             );
+            fmt.indent(|fmt| {
+                fmt.line("(Some(a), Some(b)) => a == b,");
+                fmt.comment("On overflow, constraint doesn\'t apply");
+                fmt.line("_ => false,");
+            });
+            fmtln!(fmt, "};");
         }
         Constraint::WiderOrEq(tv1, tv2) => {
             fmtln!(
                 fmt,
-                "let predicate = predicate && {}.wider_or_equal({});",
+                "let predicate = predicate && match ({}, {}) {{",
                 build_derived_expr(tv1),
                 build_derived_expr(tv2)
             );
+            fmt.indent(|fmt| {
+                fmt.line("(Some(a), Some(b)) => a.wider_or_equal(b),");
+                fmt.comment("On overflow, constraint doesn\'t apply");
+                fmt.line("_ => false,");
+            });
             fmtln!(fmt, "};");
         }
     }
@@ -290,7 +303,8 @@ fn emit_dst_inst(def: &Def, def_pool: &DefPool, var_pool: &VarPool, fmt: &mut Fo
             // Unwrapping would have left the results intact.  Replace the whole instruction.
             fmtln!(
                 fmt,
-                "pos.func.dfg.replace(inst).{};",
+                "let {} = pos.func.dfg.replace(inst).{};",
+                defined_vars,
                 def.apply.rust_builder(&def.defined_vars, var_pool)
             );
 
@@ -406,16 +420,17 @@ fn gen_transform_group<'a>(
     // Function arguments.
     fmtln!(fmt, "pub fn {}(", group.name);
     fmt.indent(|fmt| {
-        fmt.line("inst: ir::Inst,");
-        fmt.line("func: &mut ir::Function,");
-        fmt.line("cfg: &mut ControlFlowGraph,");
-        fmt.line("isa: &dyn TargetIsa,");
+        fmt.line("inst: crate::ir::Inst,");
+        fmt.line("func: &mut crate::ir::Function,");
+        fmt.line("cfg: &mut crate::flowgraph::ControlFlowGraph,");
+        fmt.line("isa: &dyn crate::isa::TargetIsa,");
     });
     fmtln!(fmt, ") -> bool {");
 
     // Function body.
     fmt.indent(|fmt| {
-        fmt.line("use ir::InstBuilder;");
+        fmt.line("use crate::ir::InstBuilder;");
+        fmt.line("use crate::cursor::{Cursor, FuncCursor};");
         fmt.line("let mut pos = FuncCursor::new(func).at_inst(inst);");
         fmt.line("pos.use_srcloc(inst);");
 

From f2c48009e877f6a7ea524f95697e2f8ff5fa214b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 11 Jul 2019 10:17:14 -0700
Subject: [PATCH 2521/3084] Disable SIMD features by default

---
 cranelift/codegen/meta/src/shared/settings.rs             | 2 +-
 cranelift/codegen/src/isa/riscv/settings.rs               | 4 +++-
 cranelift/codegen/src/settings.rs                         | 4 ++--
 cranelift/filetests/filetests/isa/x86/binary32-float.clif | 1 +
 cranelift/filetests/filetests/isa/x86/binary64-float.clif | 1 +
 5 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
index 6e3af70623..6b8b7a017f 100644
--- a/cranelift/codegen/meta/src/shared/settings.rs
+++ b/cranelift/codegen/meta/src/shared/settings.rs
@@ -84,7 +84,7 @@ pub fn define() -> SettingGroup {
         false,
     );
 
-    settings.add_bool("enable_simd", "Enable the use of SIMD instructions.", true);
+    settings.add_bool("enable_simd", "Enable the use of SIMD instructions.", false);
 
     settings.add_bool(
         "enable_atomics",
diff --git a/cranelift/codegen/src/isa/riscv/settings.rs b/cranelift/codegen/src/isa/riscv/settings.rs
index 3599a93495..24c0e2af11 100644
--- a/cranelift/codegen/src/isa/riscv/settings.rs
+++ b/cranelift/codegen/src/isa/riscv/settings.rs
@@ -35,7 +35,9 @@ mod tests {
 
     #[test]
     fn predicates() {
-        let shared = settings::Flags::new(settings::builder());
+        let mut sb = settings::builder();
+        sb.set("enable_simd", "true").unwrap();
+        let shared = settings::Flags::new(sb);
         let mut b = builder();
         b.enable("supports_f").unwrap();
         b.enable("supports_d").unwrap();
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index 836b47a5f6..b93e3d7149 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -387,7 +387,7 @@ mod tests {
              avoid_div_traps = false\n\
              enable_float = true\n\
              enable_nan_canonicalization = false\n\
-             enable_simd = true\n\
+             enable_simd = false\n\
              enable_atomics = true\n\
              allones_funcaddrs = false\n\
              probestack_enabled = true\n\
@@ -395,7 +395,7 @@ mod tests {
              jump_tables_enabled = true\n"
         );
         assert_eq!(f.opt_level(), super::OptLevel::Default);
-        assert_eq!(f.enable_simd(), true);
+        assert_eq!(f.enable_simd(), false);
         assert_eq!(f.baldrdash_prologue_words(), 0);
     }
 
diff --git a/cranelift/filetests/filetests/isa/x86/binary32-float.clif b/cranelift/filetests/filetests/isa/x86/binary32-float.clif
index 8a4ae5fe7e..c6092e2e22 100644
--- a/cranelift/filetests/filetests/isa/x86/binary32-float.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary32-float.clif
@@ -1,5 +1,6 @@
 ; Binary emission of 32-bit floating point code.
 test binemit
+set enable_simd
 target i686 haswell
 
 ; The binary encodings can be verified with the command:
diff --git a/cranelift/filetests/filetests/isa/x86/binary64-float.clif b/cranelift/filetests/filetests/isa/x86/binary64-float.clif
index 59e024a042..8c34299705 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64-float.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64-float.clif
@@ -1,6 +1,7 @@
 ; Binary emission of 64-bit floating point code.
 test binemit
 set opt_level=best
+set enable_simd
 target x86_64 haswell
 
 ; The binary encodings can be verified with the command:

From 659725b46500f2111aeb8ae1956e4f5e3b4112af Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 11 Jul 2019 10:21:12 -0700
Subject: [PATCH 2522/3084] Add x86-specific SIMD settings, e.g. SSE2

Also, ties SIMD ISA predicates to the shared enable_simd setting
---
 .../codegen/meta/src/isa/x86/settings.rs      | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/settings.rs b/cranelift/codegen/meta/src/isa/x86/settings.rs
index 15f17621ce..bc8c81f484 100644
--- a/cranelift/codegen/meta/src/isa/x86/settings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/settings.rs
@@ -3,6 +3,9 @@ use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 pub fn define(shared: &SettingGroup) -> SettingGroup {
     let mut settings = SettingGroupBuilder::new("x86");
 
+    // CPUID.01H:EDX
+    let has_sse2 = settings.add_bool("has_sse2", "SSE2: CPUID.01H:EDX.SSE2[bit 26]", false);
+
     // CPUID.01H:ECX
     let has_sse3 = settings.add_bool("has_sse3", "SSE3: CPUID.01H:ECX.SSE3[bit 0]", false);
     let has_ssse3 = settings.add_bool("has_ssse3", "SSSE3: CPUID.01H:ECX.SSSE3[bit 9]", false);
@@ -30,8 +33,15 @@ pub fn define(shared: &SettingGroup) -> SettingGroup {
         false,
     );
 
-    settings.add_predicate("use_sse41", predicate!(has_sse41));
-    settings.add_predicate("use_sse42", predicate!(has_sse41 && has_sse42));
+    let shared_enable_simd = shared.get_bool("enable_simd");
+
+    settings.add_predicate("use_sse2", predicate!(shared_enable_simd && has_sse2));
+    settings.add_predicate("use_ssse3", predicate!(shared_enable_simd && has_ssse3));
+    settings.add_predicate("use_sse41", predicate!(shared_enable_simd && has_sse41));
+    settings.add_predicate(
+        "use_sse42",
+        predicate!(shared_enable_simd && has_sse41 && has_sse42),
+    );
     settings.add_predicate("use_popcnt", predicate!(has_popcnt && has_sse42));
     settings.add_predicate("use_bmi1", predicate!(has_bmi1));
     settings.add_predicate("use_lzcnt", predicate!(has_lzcnt));
@@ -59,7 +69,7 @@ pub fn define(shared: &SettingGroup) -> SettingGroup {
     settings.add_preset("baseline", preset!());
     let nehalem = settings.add_preset(
         "nehalem",
-        preset!(has_sse3 && has_ssse3 && has_sse41 && has_sse42 && has_popcnt),
+        preset!(has_sse2 && has_sse3 && has_ssse3 && has_sse41 && has_sse42 && has_popcnt),
     );
     let haswell = settings.add_preset(
         "haswell",
@@ -72,7 +82,8 @@ pub fn define(shared: &SettingGroup) -> SettingGroup {
     settings.add_preset(
         "znver1",
         preset!(
-            has_sse3
+            has_sse2
+                && has_sse3
                 && has_ssse3
                 && has_sse41
                 && has_sse42

From c39a9b4e3f3c3192c70ab5f3434ed85c3df06c2e Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 11 Jul 2019 10:26:08 -0700
Subject: [PATCH 2523/3084] Assign vector arguments to FPR registers

---
 cranelift/codegen/src/isa/x86/abi.rs | 64 ++++++++++++++++++++++++----
 cranelift/codegen/src/isa/x86/mod.rs |  8 +++-
 2 files changed, 62 insertions(+), 10 deletions(-)

diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 549ec644c7..85e9c18ffd 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -1,6 +1,8 @@
 //! x86 ABI implementation.
 
+use super::super::settings as shared_settings;
 use super::registers::{FPR, GPR, RU};
+use super::settings as isa_settings;
 use crate::abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion};
 use crate::cursor::{Cursor, CursorPosition, EncCursor};
 use crate::ir;
@@ -39,10 +41,20 @@ struct Args {
     fpr_used: usize,
     offset: u32,
     call_conv: CallConv,
+    shared_flags: shared_settings::Flags,
+    #[allow(dead_code)]
+    isa_flags: isa_settings::Flags,
 }
 
 impl Args {
-    fn new(bits: u8, gpr: &'static [RU], fpr_limit: usize, call_conv: CallConv) -> Self {
+    fn new(
+        bits: u8,
+        gpr: &'static [RU],
+        fpr_limit: usize,
+        call_conv: CallConv,
+        shared_flags: &shared_settings::Flags,
+        isa_flags: &isa_settings::Flags,
+    ) -> Self {
         let offset = if let CallConv::WindowsFastcall = call_conv {
             // [1] "The caller is responsible for allocating space for parameters to the callee,
             // and must always allocate sufficient space to store four register parameters"
@@ -61,6 +73,8 @@ impl Args {
             fpr_used: 0,
             offset,
             call_conv,
+            shared_flags: shared_flags.clone(),
+            isa_flags: isa_flags.clone(),
         }
     }
 }
@@ -69,10 +83,15 @@ impl ArgAssigner for Args {
     fn assign(&mut self, arg: &AbiParam) -> ArgAction {
         let ty = arg.value_type;
 
-        // Check for a legal type.
-        // We don't support SIMD yet, so break all vectors down.
+        // Vectors should stay in vector registers unless SIMD is not enabled--then they are split
         if ty.is_vector() {
-            return ValueConversion::VectorSplit.into();
+            if self.shared_flags.enable_simd() {
+                let reg = FPR.unit(self.fpr_used);
+                self.fpr_used += 1;
+                return ArgumentLoc::Reg(reg).into();
+            } else {
+                return ValueConversion::VectorSplit.into();
+            }
         }
 
         // Large integers and booleans are broken down to fit in a register.
@@ -139,7 +158,13 @@ impl ArgAssigner for Args {
 }
 
 /// Legalize `sig`.
-pub fn legalize_signature(sig: &mut ir::Signature, triple: &Triple, _current: bool) {
+pub fn legalize_signature(
+    sig: &mut ir::Signature,
+    triple: &Triple,
+    _current: bool,
+    shared_flags: &shared_settings::Flags,
+    isa_flags: &isa_settings::Flags,
+) {
     let bits;
     let mut args;
 
@@ -147,14 +172,28 @@ pub fn legalize_signature(sig: &mut ir::Signature, triple: &Triple, _current: bo
         PointerWidth::U16 => panic!(),
         PointerWidth::U32 => {
             bits = 32;
-            args = Args::new(bits, &[], 0, sig.call_conv);
+            args = Args::new(bits, &[], 0, sig.call_conv, shared_flags, isa_flags);
         }
         PointerWidth::U64 => {
             bits = 64;
             args = if sig.call_conv == CallConv::WindowsFastcall {
-                Args::new(bits, &ARG_GPRS_WIN_FASTCALL_X64[..], 4, sig.call_conv)
+                Args::new(
+                    bits,
+                    &ARG_GPRS_WIN_FASTCALL_X64[..],
+                    4,
+                    sig.call_conv,
+                    shared_flags,
+                    isa_flags,
+                )
             } else {
-                Args::new(bits, &ARG_GPRS[..], 8, sig.call_conv)
+                Args::new(
+                    bits,
+                    &ARG_GPRS[..],
+                    8,
+                    sig.call_conv,
+                    shared_flags,
+                    isa_flags,
+                )
             };
         }
     }
@@ -168,7 +207,14 @@ pub fn legalize_signature(sig: &mut ir::Signature, triple: &Triple, _current: bo
         (&RET_GPRS[..], 2)
     };
 
-    let mut rets = Args::new(bits, regs, fpr_limit, sig.call_conv);
+    let mut rets = Args::new(
+        bits,
+        regs,
+        fpr_limit,
+        sig.call_conv,
+        shared_flags,
+        isa_flags,
+    );
     legalize_args(&mut sig.returns, &mut rets);
 }
 
diff --git a/cranelift/codegen/src/isa/x86/mod.rs b/cranelift/codegen/src/isa/x86/mod.rs
index c71e97410a..dfac5cacc3 100644
--- a/cranelift/codegen/src/isa/x86/mod.rs
+++ b/cranelift/codegen/src/isa/x86/mod.rs
@@ -106,7 +106,13 @@ impl TargetIsa for Isa {
     }
 
     fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) {
-        abi::legalize_signature(sig, &self.triple, current)
+        abi::legalize_signature(
+            sig,
+            &self.triple,
+            current,
+            &self.shared_flags,
+            &self.isa_flags,
+        )
     }
 
     fn regclass_for_abi_type(&self, ty: ir::Type) -> RegClass {

From 356e6dafe22e622dcd5145e9ee86dd568444d7f3 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 11 Jul 2019 10:29:39 -0700
Subject: [PATCH 2524/3084] Allow CDSL instructions to bind to vector types

---
 .../codegen/meta/src/cdsl/instructions.rs     | 34 +++++++++++++++++--
 1 file changed, 31 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 11d7114b52..00a166fb64 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -12,7 +12,7 @@ use crate::cdsl::formats::{
 };
 use crate::cdsl::operands::Operand;
 use crate::cdsl::type_inference::Constraint;
-use crate::cdsl::types::{LaneType, ValueType};
+use crate::cdsl::types::{LaneType, ValueType, VectorType};
 use crate::cdsl::typevar::TypeVar;
 
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
@@ -176,6 +176,11 @@ impl Instruction {
     pub fn bind(&self, lane_type: impl Into) -> BoundInstruction {
         bind(self.clone(), Some(lane_type.into()), Vec::new())
     }
+
+    pub fn bind_vector(&self, lane_type: impl Into, num_lanes: u64) -> BoundInstruction {
+        bind_vector(self.clone(), lane_type.into(), num_lanes, Vec::new())
+    }
+
     pub fn bind_any(&self) -> BoundInstruction {
         bind(self.clone(), None, Vec::new())
     }
@@ -400,6 +405,11 @@ impl BoundInstruction {
     pub fn bind(self, lane_type: impl Into) -> BoundInstruction {
         bind(self.inst, Some(lane_type.into()), self.value_types)
     }
+
+    pub fn bind_vector(self, lane_type: impl Into, num_lanes: u64) -> BoundInstruction {
+        bind_vector(self.inst, lane_type.into(), num_lanes, self.value_types)
+    }
+
     pub fn bind_any(self) -> BoundInstruction {
         bind(self.inst, None, self.value_types)
     }
@@ -1062,6 +1072,26 @@ fn bind(
         }
     }
 
+    verify_polymorphic_binding(&inst, &value_types);
+
+    BoundInstruction { inst, value_types }
+}
+
+/// Helper bind for vector types reused by {Bound,}Instruction::bind.
+fn bind_vector(
+    inst: Instruction,
+    lane_type: LaneType,
+    num_lanes: u64,
+    mut value_types: Vec,
+) -> BoundInstruction {
+    let vector_type = ValueType::Vector(VectorType::new(lane_type, num_lanes));
+    value_types.push(ValueTypeOrAny::ValueType(vector_type));
+    verify_polymorphic_binding(&inst, &value_types);
+    BoundInstruction { inst, value_types }
+}
+
+/// Helper to verify that binding types to the instruction does not violate polymorphic rules
+fn verify_polymorphic_binding(inst: &Instruction, value_types: &Vec) {
     match &inst.polymorphic_info {
         Some(poly) => {
             assert!(
@@ -1076,6 +1106,4 @@ fn bind(
             ));
         }
     }
-
-    BoundInstruction { inst, value_types }
 }

From 5f0e5567c171ded2d16491b9f6b89aa72134b0e6 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 11 Jul 2019 10:50:06 -0700
Subject: [PATCH 2525/3084] Add scalar_to_vector instruction

Moves scalar values in a GPR register to an FPR register
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 29 +++++++++++++++--
 .../codegen/meta/src/shared/instructions.rs   | 26 +++++++++++++++
 .../filetests/isa/x86/scalar_to_vector.clif   | 32 +++++++++++++++++++
 3 files changed, 84 insertions(+), 3 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/scalar_to_vector.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 99bfecafc2..506abdc53c 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -10,7 +10,8 @@ use crate::cdsl::instructions::{
 use crate::cdsl::recipes::{EncodingRecipe, EncodingRecipeNumber, Recipes};
 use crate::cdsl::settings::{SettingGroup, SettingPredicateNumber};
 
-use crate::shared::types::Bool::B1;
+use crate::cdsl::types::ValueType;
+use crate::shared::types::Bool::{B1, B16, B32, B64, B8};
 use crate::shared::types::Float::{F32, F64};
 use crate::shared::types::Int::{I16, I32, I64, I8};
 use crate::shared::Definitions as SharedDefinitions;
@@ -340,6 +341,7 @@ pub fn define(
     let rotl_imm = shared.by_name("rotl_imm");
     let rotr = shared.by_name("rotr");
     let rotr_imm = shared.by_name("rotr_imm");
+    let scalar_to_vector = shared.by_name("scalar_to_vector");
     let selectif = shared.by_name("selectif");
     let sextend = shared.by_name("sextend");
     let sload16 = shared.by_name("sload16");
@@ -515,6 +517,7 @@ pub fn define(
     let use_popcnt = settings.predicate_by_name("use_popcnt");
     let use_lzcnt = settings.predicate_by_name("use_lzcnt");
     let use_bmi1 = settings.predicate_by_name("use_bmi1");
+    let use_sse2 = settings.predicate_by_name("use_sse2");
     let use_sse41 = settings.predicate_by_name("use_sse41");
 
     // Definitions.
@@ -603,8 +606,14 @@ pub fn define(
     // Finally, the 0xb8 opcode takes an 8-byte immediate with a REX.W prefix.
     e.enc64(iconst.bind(I64), rec_pu_iq.opcodes(vec![0xb8]).rex().w());
 
-    // Bool constants.
-    e.enc_both(bconst.bind(B1), rec_pu_id_bool.opcodes(vec![0xb8]));
+    // Bool constants (uses MOV)
+    for &ty in &[B1, B8, B16, B32] {
+        e.enc_both(bconst.bind(ty), rec_pu_id_bool.opcodes(vec![0xb8]));
+    }
+    e.enc64(
+        bconst.bind(B64),
+        rec_pu_id_bool.opcodes(vec![0xb8]).rex().w(),
+    );
 
     // Shifts and rotates.
     // Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit
@@ -1565,5 +1574,19 @@ pub fn define(
     e.enc_both(ffcmp.bind(F32), rec_fcmp.opcodes(vec![0x0f, 0x2e]));
     e.enc_both(ffcmp.bind(F64), rec_fcmp.opcodes(vec![0x66, 0x0f, 0x2e]));
 
+    // SIMD scalar_to_vector; this uses MOV to copy the scalar value to an XMM register; according
+    // to the Intel manual: "When the destination operand is an XMM register, the source operand is
+    // written to the low doubleword of the register and the regiser is zero-extended to 128 bits."
+    for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() >= 8) {
+        let number_of_lanes = 128 / ty.lane_bits();
+        let instruction = scalar_to_vector.bind_vector(ty, number_of_lanes).bind(ty);
+        let template = rec_frurm.opcodes(vec![0x66, 0x0f, 0x6e]); // MOVD/MOVQ
+        if ty.lane_bits() < 64 {
+            // no 32-bit encodings for 64-bit widths
+            e.enc32_isap(instruction.clone(), template.clone(), use_sse2);
+        }
+        e.enc_x86_64_isap(instruction, template, use_sse2);
+    }
+
     e
 }
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 8aaa71947c..7af098f920 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -106,6 +106,16 @@ pub fn define(
             .build(),
     );
 
+    let Scalar = &TypeVar::new(
+        "scalar",
+        "Any scalar value that can be used as a lane in a vector",
+        TypeSetBuilder::new()
+            .bools(Interval::All)
+            .ints(Interval::All)
+            .floats(Interval::All)
+            .build(),
+    );
+
     let Any = &TypeVar::new(
         "Any",
         "Any integer, float, or boolean scalar or vector type",
@@ -2630,6 +2640,22 @@ pub fn define(
         .operands_out(vec![a]),
     );
 
+    let s = &operand_doc("s", Scalar, "A scalar value");
+    let a = &operand_doc("a", TxN, "A vector value (i.e. held in an XMM register)");
+
+    ig.push(
+        Inst::new(
+            "scalar_to_vector",
+            r#"
+    Scalar To Vector -- move a value out of a scalar register and into a vector
+    register; the scalar will be moved to the lowest-order bits of the vector
+    register and any higher bits will be zeroed.
+    "#,
+        )
+        .operands_in(vec![s])
+        .operands_out(vec![a]),
+    );
+
     let Bool = &TypeVar::new(
         "Bool",
         "A scalar or vector boolean type",
diff --git a/cranelift/filetests/filetests/isa/x86/scalar_to_vector.clif b/cranelift/filetests/filetests/isa/x86/scalar_to_vector.clif
new file mode 100644
index 0000000000..6c77dfafdb
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/scalar_to_vector.clif
@@ -0,0 +1,32 @@
+test binemit
+set opt_level=best
+set enable_simd
+target x86_64 has_sse2=true
+
+function %test_scalar_to_vector_b8() {
+ebb0:
+[-, %rax]   v0 = bconst.b8 true
+[-, %xmm0]  v1 = scalar_to_vector.b8x16 v0    ; bin: 66 0f 6e c0
+            return
+}
+
+function %test_scalar_to_vector_i16() {
+ebb0:
+[-, %rbx]   v0 = iconst.i16 42
+[-, %xmm2]  v1 = scalar_to_vector.i16x8 v0    ; bin: 66 0f 6e d3
+            return
+}
+
+function %test_scalar_to_vector_f32() {
+ebb0:
+[-, %rcx]   v0 = f32const 0x0.42
+[-, %xmm3]  v1 = scalar_to_vector.f32x4 v0    ; bin: 66 0f 6e d9
+            return
+}
+
+function %test_scalar_to_vector_i64() {
+ebb0:
+[-, %rdx]   v0 = iconst.i64 42
+[-, %xmm7]  v1 = scalar_to_vector.i64x2 v0    ; bin: 66 0f 6e fa
+            return
+}

From 61772e9775e9fb428004ee1a42e3a7f09f3e10af Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 11 Jul 2019 11:07:26 -0700
Subject: [PATCH 2526/3084] Add raw_bitcast instruction

Casts bits as a different type of the same width with no change to the data (unlike bitcast)
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 15 +++++++-
 cranelift/codegen/meta/src/isa/x86/recipes.rs |  6 ++++
 .../codegen/meta/src/shared/instructions.rs   | 34 ++++++++++++++-----
 cranelift/codegen/meta/src/shared/mod.rs      |  6 ++++
 .../filetests/isa/x86/raw_bitcast.clif        | 10 ++++++
 5 files changed, 61 insertions(+), 10 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/raw_bitcast.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 506abdc53c..c34c13d089 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -9,7 +9,6 @@ use crate::cdsl::instructions::{
 };
 use crate::cdsl::recipes::{EncodingRecipe, EncodingRecipeNumber, Recipes};
 use crate::cdsl::settings::{SettingGroup, SettingPredicateNumber};
-
 use crate::cdsl::types::ValueType;
 use crate::shared::types::Bool::{B1, B16, B32, B64, B8};
 use crate::shared::types::Float::{F32, F64};
@@ -333,6 +332,7 @@ pub fn define(
     let load_complex = shared.by_name("load_complex");
     let nearest = shared.by_name("nearest");
     let popcnt = shared.by_name("popcnt");
+    let raw_bitcast = shared.by_name("raw_bitcast");
     let regfill = shared.by_name("regfill");
     let regmove = shared.by_name("regmove");
     let regspill = shared.by_name("regspill");
@@ -452,6 +452,7 @@ pub fn define(
     let rec_ldWithIndexDisp8 = r.template("ldWithIndexDisp8");
     let rec_mulx = r.template("mulx");
     let rec_null = r.recipe("null");
+    let rec_null_fpr = r.recipe("null_fpr");
     let rec_pcrel_fnaddr8 = r.template("pcrel_fnaddr8");
     let rec_pcrel_gvaddr8 = r.template("pcrel_gvaddr8");
     let rec_popq = r.template("popq");
@@ -1588,5 +1589,17 @@ pub fn define(
         e.enc_x86_64_isap(instruction, template, use_sse2);
     }
 
+    // SIMD bitcast all 128-bit vectors to each other (for legalizing splat.x16x8)
+    for from_type in ValueType::all_lane_types().filter(|t| t.lane_bits() >= 8) {
+        for to_type in ValueType::all_lane_types().filter(|t| t.lane_bits() >= 8 && *t != from_type)
+        {
+            let instruction = raw_bitcast
+                .bind_vector(to_type, 128 / to_type.lane_bits())
+                .bind_vector(from_type, 128 / from_type.lane_bits());
+            e.enc32_rec(instruction.clone(), rec_null_fpr, 0);
+            e.enc64_rec(instruction, rec_null_fpr, 0);
+        }
+    }
+
     e
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 5c1d28ef20..45441cf67a 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -411,6 +411,12 @@ pub fn define<'shared>(
             .operands_out(vec![0])
             .emit(""),
     );
+    recipes.add_recipe(
+        EncodingRecipeBuilder::new("null_fpr", f_unary, 0)
+            .operands_in(vec![fpr])
+            .operands_out(vec![0])
+            .emit(""),
+    );
     recipes.add_recipe(
         EncodingRecipeBuilder::new("stacknull", f_unary, 0)
             .operands_in(vec![stack_gpr32])
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 7af098f920..78d017f1a8 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -128,6 +128,8 @@ pub fn define(
             .build(),
     );
 
+    let AnyTo = &TypeVar::copy_from(Any, "AnyTo".to_string());
+
     let Mem = &TypeVar::new(
         "Mem",
         "Any type that can be stored in memory",
@@ -138,15 +140,7 @@ pub fn define(
             .build(),
     );
 
-    let MemTo = &TypeVar::new(
-        "MemTo",
-        "Any type that can be stored in memory",
-        TypeSetBuilder::new()
-            .ints(Interval::All)
-            .floats(Interval::All)
-            .simd_lanes(Interval::All)
-            .build(),
-    );
+    let MemTo = &TypeVar::copy_from(Mem, "MemTo".to_string());
 
     let addr = &operand("addr", iAddr);
     let c = &operand_doc("c", Testable, "Controlling value to test");
@@ -2640,6 +2634,28 @@ pub fn define(
         .operands_out(vec![a]),
     );
 
+    let x = &operand("x", Any);
+    let a = &operand_doc("a", AnyTo, "Bits of `x` reinterpreted");
+
+    ig.push(
+        Inst::new(
+            "raw_bitcast",
+            r#"
+        Cast the bits in `x` as a different type of the same bit width.
+
+        This instruction does not change the data's representation but allows
+        data in registers to be used as different types, e.g. an i32x4 as a 
+        b8x16. The only constraint on the result `a` is that it can be 
+        `raw_bitcast` back to the original type. Also, in a raw_bitcast between
+        vector types with the same number of lanes, the value of each result 
+        lane is a raw_bitcast of the corresponding operand lane. TODO there is 
+        currently no mechanism for enforcing the bit width constraint.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a]),
+    );
+
     let s = &operand_doc("s", Scalar, "A scalar value");
     let a = &operand_doc("a", TxN, "A vector value (i.e. held in an XMM register)");
 
diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
index 817f67c340..ba1425b844 100644
--- a/cranelift/codegen/meta/src/shared/mod.rs
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -50,6 +50,12 @@ impl OperandKinds {
     }
 }
 
+impl From> for OperandKinds {
+    fn from(kinds: Vec) -> Self {
+        OperandKinds(kinds)
+    }
+}
+
 pub fn define() -> Definitions {
     let mut all_instructions = AllInstructions::new();
 
diff --git a/cranelift/filetests/filetests/isa/x86/raw_bitcast.clif b/cranelift/filetests/filetests/isa/x86/raw_bitcast.clif
new file mode 100644
index 0000000000..5c1c2ea322
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/raw_bitcast.clif
@@ -0,0 +1,10 @@
+test binemit
+target x86_64
+
+function %test_raw_bitcast_i16x8_to_b32x4() {
+ebb0:
+[-, %rbx]   v0 = bconst.b16 true
+[-, %xmm2]  v1 = scalar_to_vector.b16x8 v0
+[-, %xmm2]  v2 = raw_bitcast.i32x4 v1       ; bin:
+            return
+}

From 683e7c75a3380d933f963c86ba2193fddb6c9855 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 11 Jul 2019 13:12:25 -0700
Subject: [PATCH 2527/3084] Add x86-specific shuffle instructions

This includes both PSHUFD and PSHUFB; these are necessary to legalize future SIMD instructions.
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 37 ++++++++++++++++
 .../codegen/meta/src/isa/x86/instructions.rs  | 43 ++++++++++++++++++-
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 22 ++++++++++
 .../filetests/filetests/isa/x86/pshufb.clif   | 13 ++++++
 .../filetests/filetests/isa/x86/pshufd.clif   | 11 +++++
 5 files changed, 125 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/pshufb.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/pshufd.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index c34c13d089..32c1e2f957 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -250,6 +250,17 @@ impl PerCpuModeEncodings {
             self.enc64(inst.clone().bind(I64).bind_any(), template);
         }
     }
+
+    /// Add the same encoding to both X86_32 and X86_64; assumes configuration (e.g. REX, operand binding) has already happened
+    fn enc_32_64_isap(
+        &mut self,
+        inst: BoundInstruction,
+        template: Template,
+        isap: SettingPredicateNumber,
+    ) {
+        self.enc32_isap(inst.clone(), template.clone(), isap);
+        self.enc64_isap(inst, template, isap);
+    }
 }
 
 // Definitions.
@@ -379,6 +390,8 @@ pub fn define(
     let x86_fmax = x86.by_name("x86_fmax");
     let x86_fmin = x86.by_name("x86_fmin");
     let x86_pop = x86.by_name("x86_pop");
+    let x86_pshufd = x86.by_name("x86_pshufd");
+    let x86_pshufb = x86.by_name("x86_pshufb");
     let x86_push = x86.by_name("x86_push");
     let x86_sdivmodx = x86.by_name("x86_sdivmodx");
     let x86_smulx = x86.by_name("x86_smulx");
@@ -462,6 +475,7 @@ pub fn define(
     let rec_pushq = r.template("pushq");
     let rec_ret = r.template("ret");
     let rec_r_ib = r.template("r_ib");
+    let rec_r_ib_unsigned = r.template("r_ib_unsigned");
     let rec_r_id = r.template("r_id");
     let rec_rcmp = r.template("rcmp");
     let rec_rcmp_ib = r.template("rcmp_ib");
@@ -519,6 +533,7 @@ pub fn define(
     let use_lzcnt = settings.predicate_by_name("use_lzcnt");
     let use_bmi1 = settings.predicate_by_name("use_bmi1");
     let use_sse2 = settings.predicate_by_name("use_sse2");
+    let use_ssse3 = settings.predicate_by_name("use_ssse3");
     let use_sse41 = settings.predicate_by_name("use_sse41");
 
     // Definitions.
@@ -1575,6 +1590,28 @@ pub fn define(
     e.enc_both(ffcmp.bind(F32), rec_fcmp.opcodes(vec![0x0f, 0x2e]));
     e.enc_both(ffcmp.bind(F64), rec_fcmp.opcodes(vec![0x66, 0x0f, 0x2e]));
 
+    // SIMD splat: before x86 can use vector data, it must be moved to XMM registers; see
+    // legalize.rs for how this is done; once there, x86_pshuf* (below) is used for broadcasting the
+    // value across the register
+
+    // PSHUFB, 8-bit shuffle using two XMM registers
+    for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
+        let number_of_lanes = 128 / ty.lane_bits();
+        let instruction = x86_pshufb.bind_vector(ty, number_of_lanes);
+        let template = rec_fa.nonrex().opcodes(vec![0x66, 0x0f, 0x38, 0x00]);
+        e.enc32_isap(instruction.clone(), template.clone(), use_ssse3);
+        e.enc64_isap(instruction, template, use_ssse3);
+    }
+
+    // PSHUFD, 32-bit shuffle using one XMM register and a u8 immediate
+    for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 32) {
+        let number_of_lanes = 128 / ty.lane_bits();
+        let instruction = x86_pshufd.bind_vector(ty, number_of_lanes);
+        let template = rec_r_ib_unsigned.nonrex().opcodes(vec![0x66, 0x0f, 0x70]);
+        e.enc32_isap(instruction.clone(), template.clone(), use_sse2);
+        e.enc64_isap(instruction, template, use_sse2);
+    }
+
     // SIMD scalar_to_vector; this uses MOV to copy the scalar value to an XMM register; according
     // to the Intel manual: "When the destination operand is an XMM register, the source operand is
     // written to the low doubleword of the register and the regiser is zero-extended to 128 bits."
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 9aa7363e1a..6464261938 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -7,7 +7,7 @@ use crate::cdsl::instructions::{
 use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
 use crate::cdsl::types::ValueType;
 use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
-use crate::shared::types;
+use crate::shared::{immediates, types, OperandKinds};
 
 pub fn define(
     mut all_instructions: &mut AllInstructions,
@@ -249,5 +249,46 @@ pub fn define(
         .operands_out(vec![y, rflags]),
     );
 
+    let immediates = OperandKinds::from(immediates::define());
+    let uimm8 = immediates.by_name("uimm8");
+    let TxN = &TypeVar::new(
+        "TxN",
+        "A SIMD vector type",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .floats(Interval::All)
+            .bools(Interval::All)
+            .simd_lanes(Interval::All)
+            .includes_scalars(false)
+            .build(),
+    );
+    let a = &operand_doc("a", TxN, "A vector value (i.e. held in an XMM register)");
+    let b = &operand_doc("b", TxN, "A vector value (i.e. held in an XMM register)");
+    let i = &operand_doc("i", uimm8, "An ordering operand controlling the copying of data from the source to the destination; see PSHUFD in Intel manual for details");
+
+    ig.push(
+        Inst::new(
+            "x86_pshufd",
+            r#"
+    Packed Shuffle Doublewords -- copies data from either memory or lanes in an extended
+    register and re-orders the data according to the passed immediate byte.
+    "#,
+        )
+        .operands_in(vec![a, i]) // TODO allow copying from memory here (need more permissive type than TxN)
+        .operands_out(vec![a]),
+    );
+
+    ig.push(
+        Inst::new(
+            "x86_pshufb",
+            r#"
+    Packed Shuffle Bytes -- re-orders data in an extended register using a shuffle
+    mask from either memory or another extended register
+    "#,
+        )
+        .operands_in(vec![a, b]) // TODO allow re-ordering from memory here (need more permissive type than TxN)
+        .operands_out(vec![a]),
+    );
+
     ig.build()
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 45441cf67a..f2353d3a66 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -367,6 +367,7 @@ pub fn define<'shared>(
     let f_call = formats.by_name("Call");
     let f_call_indirect = formats.by_name("CallIndirect");
     let f_copy_special = formats.by_name("CopySpecial");
+    let f_extract_lane = formats.by_name("ExtractLane"); // TODO this would preferably retrieve a BinaryImm8 format but because formats are compared structurally and ExtractLane has the same structure this is impossible--if we rename ExtractLane, it may even impact parsing
     let f_float_compare = formats.by_name("FloatCompare");
     let f_float_cond = formats.by_name("FloatCond");
     let f_float_cond_trap = formats.by_name("FloatCondTrap");
@@ -794,6 +795,27 @@ pub fn define<'shared>(
         );
     }
 
+    // XX /r ib with 8-bit unsigned immediate (e.g. for pshufd)
+    {
+        let format = formats.get(f_extract_lane);
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("r_ib_unsigned", f_extract_lane, 2)
+                .operands_in(vec![fpr])
+                .operands_out(vec![fpr])
+                .inst_predicate(InstructionPredicate::new_is_unsigned_int(
+                    format, "lane", 8, 0,
+                )) // TODO if the format name is changed then "lane" should be renamed to something more appropriate--ordering mask? broadcast immediate?
+                .emit(
+                    r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                    modrm_rr(in_reg0, out_reg0, sink);
+                    let imm:i64 = lane.into();
+                    sink.put1(imm as u8);
+                "#,
+                ),
+        );
+    }
+
     {
         // XX /n id with 32-bit immediate sign-extended. UnaryImm version.
         let format = formats.get(f_unary_imm);
diff --git a/cranelift/filetests/filetests/isa/x86/pshufb.clif b/cranelift/filetests/filetests/isa/x86/pshufb.clif
new file mode 100644
index 0000000000..7c23c5ab61
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/pshufb.clif
@@ -0,0 +1,13 @@
+test binemit
+set enable_simd
+target x86_64 has_sse2=true has_ssse3=true
+
+function %test_pshufb() {
+ebb0:
+[-, %rax]   v0 = iconst.i8 42
+[-, %xmm0]  v1 = scalar_to_vector.i8x16 v0   ; bin: 66 40 0f 6e c0
+[-, %rbx]   v2 = iconst.i8 43
+[-, %xmm4]  v3 = scalar_to_vector.i8x16 v2   ; bin: 66 40 0f 6e e3
+[-, %xmm0]  v4 = x86_pshufb v1, v3               ; bin: 66 0f 38 00 c4
+            return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/pshufd.clif b/cranelift/filetests/filetests/isa/x86/pshufd.clif
new file mode 100644
index 0000000000..183af4fc0e
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/pshufd.clif
@@ -0,0 +1,11 @@
+test binemit
+set enable_simd
+target x86_64 has_sse2=true
+
+function %test_pshuf() {
+ebb0:
+[-, %rax]   v0 = iconst.i32 42
+[-, %xmm0]  v1 = scalar_to_vector.i32x4 v0  ; bin: 66 40 0f 6e c0
+[-, %xmm0]  v2 = x86_pshufd v1, 0                ; bin: 66 0f 70 c0 00
+            return
+}

From 3b36a1d1d880333fc8436710455563e3c7c6fc95 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 11 Jul 2019 13:18:55 -0700
Subject: [PATCH 2528/3084] Add x86 implementation of insertlane instruction

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 30 ++++++++++++++
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 22 +++++++++++
 .../filetests/isa/x86/insertlane.clif         | 39 +++++++++++++++++++
 3 files changed, 91 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/insertlane.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 32c1e2f957..788c0c3ec0 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -326,6 +326,7 @@ pub fn define(
     let ifcmp_sp = shared.by_name("ifcmp_sp");
     let imul = shared.by_name("imul");
     let indirect_jump_table_br = shared.by_name("indirect_jump_table_br");
+    let insertlane = shared.by_name("insertlane");
     let ireduce = shared.by_name("ireduce");
     let ishl = shared.by_name("ishl");
     let ishl_imm = shared.by_name("ishl_imm");
@@ -476,6 +477,7 @@ pub fn define(
     let rec_ret = r.template("ret");
     let rec_r_ib = r.template("r_ib");
     let rec_r_ib_unsigned = r.template("r_ib_unsigned");
+    let rec_r_ib_unsigned_r = r.template("r_ib_unsigned_r");
     let rec_r_id = r.template("r_id");
     let rec_rcmp = r.template("rcmp");
     let rec_rcmp_ib = r.template("rcmp_ib");
@@ -1626,6 +1628,34 @@ pub fn define(
         e.enc_x86_64_isap(instruction, template, use_sse2);
     }
 
+    // SIMD insertlane
+    let mut insertlane_mapping: HashMap, SettingPredicateNumber)> = HashMap::new();
+    insertlane_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x20], use_sse41)); // PINSRB
+    insertlane_mapping.insert(16, (vec![0x66, 0x0f, 0xc4], use_sse2)); // PINSRW
+    insertlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x22], use_sse41)); // PINSRD
+    insertlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x22], use_sse41)); // PINSRQ, only x86_64
+
+    for ty in ValueType::all_lane_types() {
+        if let Some((opcode, isap)) = insertlane_mapping.get(&ty.lane_bits()) {
+            let number_of_lanes = 128 / ty.lane_bits();
+            let instruction = insertlane.bind_vector(ty, number_of_lanes);
+            let template = rec_r_ib_unsigned_r.opcodes(opcode.clone());
+            if ty.lane_bits() < 64 {
+                e.enc_32_64_isap(instruction, template.nonrex(), isap.clone());
+            } else {
+                // turns out the 64-bit widths have REX/W encodings and only are available on x86_64
+                e.enc64_isap(instruction, template.rex().w(), isap.clone());
+            }
+        }
+    }
+
+    // SIMD bitcast f64 to all 8-bit-lane vectors (for legalizing splat.x8x16); assumes that f64 is stored in an XMM register
+    for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
+        let instruction = bitcast.bind_vector(ty, 16).bind(F64);
+        e.enc32_rec(instruction.clone(), rec_null_fpr, 0);
+        e.enc64_rec(instruction, rec_null_fpr, 0);
+    }
+
     // SIMD bitcast all 128-bit vectors to each other (for legalizing splat.x16x8)
     for from_type in ValueType::all_lane_types().filter(|t| t.lane_bits() >= 8) {
         for to_type in ValueType::all_lane_types().filter(|t| t.lane_bits() >= 8 && *t != from_type)
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index f2353d3a66..11063e39b9 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -373,6 +373,7 @@ pub fn define<'shared>(
     let f_float_cond_trap = formats.by_name("FloatCondTrap");
     let f_func_addr = formats.by_name("FuncAddr");
     let f_indirect_jump = formats.by_name("IndirectJump");
+    let f_insert_lane = formats.by_name("InsertLane");
     let f_int_compare = formats.by_name("IntCompare");
     let f_int_compare_imm = formats.by_name("IntCompareImm");
     let f_int_cond = formats.by_name("IntCond");
@@ -816,6 +817,27 @@ pub fn define<'shared>(
         );
     }
 
+    // XX /r ib with 8-bit unsigned immediate (e.g. for insertlane)
+    {
+        let format = formats.get(f_insert_lane);
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("r_ib_unsigned_r", f_insert_lane, 2)
+                .operands_in(vec![fpr, gpr])
+                .operands_out(vec![0])
+                .inst_predicate(InstructionPredicate::new_is_unsigned_int(
+                    format, "lane", 8, 0,
+                ))
+                .emit(
+                    r#"
+                    {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                    modrm_rr(in_reg1, in_reg0, sink);
+                    let imm:i64 = lane.into();
+                    sink.put1(imm as u8);
+                "#,
+                ),
+        );
+    }
+
     {
         // XX /n id with 32-bit immediate sign-extended. UnaryImm version.
         let format = formats.get(f_unary_imm);
diff --git a/cranelift/filetests/filetests/isa/x86/insertlane.clif b/cranelift/filetests/filetests/isa/x86/insertlane.clif
new file mode 100644
index 0000000000..c55dc40333
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/insertlane.clif
@@ -0,0 +1,39 @@
+test binemit
+set enable_simd
+target x86_64 haswell
+
+function %test_insertlane_b8() {
+ebb0:
+[-, %rax]   v0 = bconst.b8 true
+[-, %rbx]   v1 = bconst.b8 false
+[-, %xmm0]  v2 = splat.b8x16 v0
+[-, %xmm0]  v3 = insertlane v2, 10, v1    ; bin: 66 0f 3a 20 c3 0a
+            return
+}
+
+function %test_insertlane_i16() {
+ebb0:
+[-, %rax]   v0 = iconst.i16 4
+[-, %rbx]   v1 = iconst.i16 5
+[-, %xmm1]  v2 = splat.i16x8 v0
+[-, %xmm1]  v3 = insertlane v2, 4, v1    ; bin: 66 0f c4 cb 04
+            return
+}
+
+function %test_insertlane_i32() {
+ebb0:
+[-, %rax]   v0 = iconst.i32 42
+[-, %rbx]   v1 = iconst.i32 99
+[-, %xmm4]  v2 = splat.i32x4 v0
+[-, %xmm4]  v3 = insertlane v2, 2, v1    ; bin: 66 0f 3a 22 e3 02
+            return
+}
+
+function %test_insertlane_f64() {
+ebb0:
+[-, %rax]   v0 = f64const 0x0.0
+[-, %rbx]   v1 = f64const 0x4.2
+[-, %xmm2]  v2 = splat.f64x2 v0
+[-, %xmm2]  v3 = insertlane v2, 1, v1    ; bin: 66 48 0f 3a 22 d3 01
+            return
+}

From 084e279def1787f083c37895236120ad8a886d2a Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 11 Jul 2019 13:21:36 -0700
Subject: [PATCH 2529/3084] Add x86 implementation of splat instruction

---
 .../codegen/meta/src/isa/x86/legalize.rs      | 93 ++++++++++++++++++-
 cranelift/codegen/meta/src/isa/x86/mod.rs     |  3 +-
 cranelift/codegen/src/ir/immediates.rs        |  6 ++
 .../filetests/isa/x86/legalize-splat.clif     | 73 +++++++++++++++
 4 files changed, 173 insertions(+), 2 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/legalize-splat.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 2815665dac..8423b10990 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -1,7 +1,8 @@
 use crate::cdsl::ast::{var, ExprBuilder, Literal};
 use crate::cdsl::instructions::InstructionGroup;
+use crate::cdsl::types::ValueType;
 use crate::cdsl::xform::TransformGroupBuilder;
-
+use crate::shared::types::Float::F64;
 use crate::shared::types::Int::{I32, I64};
 use crate::shared::Definitions as SharedDefinitions;
 
@@ -19,9 +20,11 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
     // List of instructions.
     let insts = &shared.instructions;
     let band = insts.by_name("band");
+    let bitcast = insts.by_name("bitcast");
     let bor = insts.by_name("bor");
     let clz = insts.by_name("clz");
     let ctz = insts.by_name("ctz");
+    let f64const = insts.by_name("f64const");
     let fcmp = insts.by_name("fcmp");
     let fcvt_from_uint = insts.by_name("fcvt_from_uint");
     let fcvt_to_sint = insts.by_name("fcvt_to_sint");
@@ -33,11 +36,15 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
     let iadd = insts.by_name("iadd");
     let iconst = insts.by_name("iconst");
     let imul = insts.by_name("imul");
+    let insertlane = insts.by_name("insertlane");
     let isub = insts.by_name("isub");
     let popcnt = insts.by_name("popcnt");
+    let raw_bitcast = insts.by_name("raw_bitcast");
+    let scalar_to_vector = insts.by_name("scalar_to_vector");
     let sdiv = insts.by_name("sdiv");
     let selectif = insts.by_name("selectif");
     let smulhi = insts.by_name("smulhi");
+    let splat = insts.by_name("splat");
     let srem = insts.by_name("srem");
     let udiv = insts.by_name("udiv");
     let umulhi = insts.by_name("umulhi");
@@ -46,6 +53,8 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
 
     let x86_bsf = x86_instructions.by_name("x86_bsf");
     let x86_bsr = x86_instructions.by_name("x86_bsr");
+    let x86_pshufb = x86_instructions.by_name("x86_pshufb");
+    let x86_pshufd = x86_instructions.by_name("x86_pshufd");
     let x86_umulx = x86_instructions.by_name("x86_umulx");
     let x86_smulx = x86_instructions.by_name("x86_smulx");
 
@@ -53,6 +62,8 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
     let floatcc = shared.operand_kinds.by_name("floatcc");
     let imm64 = shared.operand_kinds.by_name("imm64");
     let intcc = shared.operand_kinds.by_name("intcc");
+    let uimm8 = shared.operand_kinds.by_name("uimm8");
+    let ieee64 = shared.operand_kinds.by_name("ieee64");
 
     // Division and remainder.
     //
@@ -290,4 +301,84 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
     );
 
     group.build_and_add_to(&mut shared.transform_groups);
+
+    let mut narrow = TransformGroupBuilder::new(
+        "x86_narrow",
+        r#"
+    Legalize instructions by narrowing.
+
+    Use x86-specific instructions if needed."#,
+    )
+    .isa("x86")
+    .chain_with(shared.transform_groups.by_name("narrow").id);
+
+    // SIMD
+    let uimm8_zero = Literal::constant(uimm8, 0x00);
+    let uimm8_one = Literal::constant(uimm8, 0x01);
+    let ieee64_zero = Literal::constant(ieee64, 0x00);
+    let b = var("b");
+    let c = var("c");
+    let d = var("d");
+
+    // SIMD splat: 8-bits
+    for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
+        let splat_x8x16 = splat.bind_vector(ty, 128 / ty.lane_bits());
+        let bitcast_f64_to_any8x16 = bitcast.bind_vector(ty, 128 / ty.lane_bits()).bind(F64);
+        narrow.legalize(
+            def!(y = splat_x8x16(x)),
+            vec![
+                def!(a = scalar_to_vector(x)), // move into the lowest 8 bits of an XMM register
+                def!(b = f64const(ieee64_zero)), // zero out a different XMM register; the shuffle mask for moving the lowest byte to all other byte lanes is 0x0
+                def!(c = bitcast_f64_to_any8x16(b)), // no instruction emitted; informs the SSA that the 0 in b can be used as a vector of this type
+                def!(y = x86_pshufb(a, c)), // PSHUFB takes two XMM operands, one of which is a shuffle mask (i.e. b)
+            ],
+        );
+    }
+
+    // SIMD splat: 16-bits
+    for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 16) {
+        let splat_x16x8 = splat.bind_vector(ty, 128 / ty.lane_bits());
+        let raw_bitcast_any16x8_to_i32x4 = raw_bitcast
+            .bind_vector(I32, 4)
+            .bind_vector(ty, 128 / ty.lane_bits());
+        let raw_bitcast_i32x4_to_any16x8 = raw_bitcast
+            .bind_vector(ty, 128 / ty.lane_bits())
+            .bind_vector(I32, 4);
+        narrow.legalize(
+            def!(y = splat_x16x8(x)),
+            vec![
+                def!(a = scalar_to_vector(x)), // move into the lowest 16 bits of an XMM register
+                def!(b = insertlane(a, uimm8_one, x)), // insert the value again but in the next lowest 16 bits
+                def!(c = raw_bitcast_any16x8_to_i32x4(b)), // no instruction emitted; pretend this is an I32x4 so we can use PSHUFD
+                def!(d = x86_pshufd(c, uimm8_zero)), // broadcast the bytes in the XMM register with PSHUFD
+                def!(y = raw_bitcast_i32x4_to_any16x8(d)), // no instruction emitted; pretend this is an X16x8 again
+            ],
+        );
+    }
+
+    // SIMD splat: 32-bits
+    for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 32) {
+        let splat_any32x4 = splat.bind_vector(ty, 128 / ty.lane_bits());
+        narrow.legalize(
+            def!(y = splat_any32x4(x)),
+            vec![
+                def!(a = scalar_to_vector(x)), // translate to an x86 MOV to get the value in an XMM register
+                def!(y = x86_pshufd(a, uimm8_zero)), // broadcast the bytes in the XMM register with PSHUF
+            ],
+        );
+    }
+
+    // SIMD splat: 64-bits
+    for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 64) {
+        let splat_any64x2 = splat.bind_vector(ty, 128 / ty.lane_bits());
+        narrow.legalize(
+            def!(y = splat_any64x2(x)),
+            vec![
+                def!(a = scalar_to_vector(x)), // move into the lowest 64 bits of an XMM register
+                def!(y = insertlane(a, uimm8_one, x)), // move into the highest 64 bits of the same XMM register
+            ],
+        );
+    }
+
+    narrow.build_and_add_to(&mut shared.transform_groups);
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index a8002890f7..9f4a4a334a 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -30,6 +30,7 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let expand_flags = shared_defs.transform_groups.by_name("expand_flags");
     let narrow = shared_defs.transform_groups.by_name("narrow");
     let widen = shared_defs.transform_groups.by_name("widen");
+    let x86_narrow = shared_defs.transform_groups.by_name("x86_narrow");
     let x86_expand = shared_defs.transform_groups.by_name("x86_expand");
 
     x86_32.legalize_monomorphic(expand_flags);
@@ -42,7 +43,7 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     x86_32.legalize_type(F64, x86_expand);
 
     x86_64.legalize_monomorphic(expand_flags);
-    x86_64.legalize_default(narrow);
+    x86_64.legalize_default(x86_narrow);
     x86_64.legalize_type(B1, expand_flags);
     x86_64.legalize_type(I8, widen);
     x86_64.legalize_type(I16, widen);
diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index 73d1d8cc21..4a8d5e1fbc 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -738,6 +738,12 @@ impl From for Ieee64 {
     }
 }
 
+impl From for Ieee64 {
+    fn from(x: u64) -> Self {
+        Ieee64::with_float(f64::from_bits(x))
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-splat.clif b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
new file mode 100644
index 0000000000..c50f5a2f81
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
@@ -0,0 +1,73 @@
+test compile
+set enable_simd=true
+set probestack_enabled=false
+target x86_64 haswell
+
+; use baldrdash calling convention here for simplicity (avoids prologue, epilogue)
+function %test_splat_i32() -> i32x4 baldrdash {
+ebb0:
+    v0 = iconst.i32 42
+    v1 = splat.i32x4 v0
+    return v1
+}
+
+; sameln: function %test_splat_i32() -> i32x4 [%xmm0] baldrdash {
+; nextln:   ss0 = incoming_arg 0, offset 0
+; nextln: 
+; nextln:   ebb0:
+; nextln:     v0 = iconst.i32 42
+; nextln:     v2 = scalar_to_vector.i32x4 v0
+; nextln:     v1 = x86_pshufd v2, 0
+; nextln:     return v1
+; nextln: }
+
+
+
+function %test_splat_i64() -> i64x2 baldrdash {
+ebb0:
+    v0 = iconst.i64 42
+    v1 = splat.i64x2 v0
+    return v1
+}
+
+; check:   ebb0:
+; nextln:     v0 = iconst.i64 42
+; nextln:     v2 = scalar_to_vector.i64x2 v0
+; nextln:     v1 = insertlane v2, 1, v0
+; nextln:     return v1
+
+
+
+function %test_splat_b16() -> b16x8 baldrdash {
+ebb0:
+    v0 = bconst.b16 true
+    v1 = splat.b16x8 v0
+    return v1
+}
+
+; check:   ebb0:
+; nextln:     v0 = bconst.b16 true
+; nextln:     v2 = scalar_to_vector.b16x8 v0
+; nextln:     v3 = insertlane v2, 1, v0
+; nextln:     v4 = raw_bitcast.i32x4 v3
+; nextln:     v5 = x86_pshufd v4, 0
+; nextln:     v1 = raw_bitcast.b16x8 v5
+; nextln:     return v1
+
+
+
+function %test_splat_i8() -> i8x16 baldrdash {
+ebb0:
+    v0 = iconst.i8 42
+    v1 = splat.i8x16 v0
+    return v1
+}
+
+; check:   ebb0:
+; nextln:     v2 = iconst.i32 42
+; nextln:     v0 = ireduce.i8 v2
+; nextln:     v3 = scalar_to_vector.i8x16 v0
+; nextln:     v4 = f64const 0.0
+; nextln:     v5 = bitcast.i8x16 v4
+; nextln:     v1 = x86_pshufb v3, v5
+; nextln:     return v1

From 9b97ddf29a10ec9c73d5e2afe5013eaff9e4b5ba Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Thu, 18 Jul 2019 09:59:28 -0600
Subject: [PATCH 2530/3084] Enable basic block checks through a feature. (#856)

This allows prefixing BB-specific code with "#[cfg(feature = "basic-blocks")]",
which avoids having to reference an environment variable across the codebase.

The easiest way to enable the feature locally is to add the arguments
'features = ["basic-blocks"]' to the workspace Cargo.toml, where it defines
the cranelift-codegen dependency.
---
 cranelift/codegen/Cargo.toml          |  3 +++
 cranelift/codegen/src/verifier/mod.rs | 11 +++--------
 2 files changed, 6 insertions(+), 8 deletions(-)

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index aa579c4cde..a9834dac07 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -62,6 +62,9 @@ riscv = []
 # For dependent crates that want to serialize some parts of cranelift
 enable-serde = ["serde"]
 
+# Temporary feature that enforces basic block semantics.
+basic-blocks = []
+
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 694e5e46de..276e43adf9 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -267,8 +267,6 @@ struct Verifier<'a> {
     expected_cfg: ControlFlowGraph,
     expected_domtree: DominatorTree,
     isa: Option<&'a dyn TargetIsa>,
-    // To be removed when #796 is completed.
-    verify_encodable_as_bb: bool,
 }
 
 impl<'a> Verifier<'a> {
@@ -280,7 +278,6 @@ impl<'a> Verifier<'a> {
             expected_cfg,
             expected_domtree,
             isa: fisa.isa,
-            verify_encodable_as_bb: std::env::var("CRANELIFT_BB").is_ok(),
         }
     }
 
@@ -473,12 +470,8 @@ impl<'a> Verifier<'a> {
 
     /// Check that the given EBB can be encoded as a BB, by checking that only
     /// branching instructions are ending the EBB.
+    #[cfg(feature = "basic-blocks")]
     fn encodable_as_bb(&self, ebb: Ebb, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
-        // Skip this verification if the environment variable is not set.
-        if !self.verify_encodable_as_bb {
-            return Ok(());
-        };
-
         let dfg = &self.func.dfg;
         let inst_iter = self.func.layout.ebb_insts(ebb);
         // Skip non-branching instructions.
@@ -1794,6 +1787,8 @@ impl<'a> Verifier<'a> {
                 self.verify_encoding(inst, errors)?;
                 self.immediate_constraints(inst, errors)?;
             }
+
+            #[cfg(feature = "basic-blocks")]
             self.encodable_as_bb(ebb, errors)?;
         }
 

From 8efaeec502678898210b519411afe7bf4fdc328c Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Fri, 19 Jul 2019 09:52:05 -0600
Subject: [PATCH 2531/3084] Verify that FunctionBuilder blocks are basic blocks
 in debug mode (#857)

To use, enable the "basic-blocks" feature on cranelift-frontend.
---
 cranelift/codegen/src/ir/function.rs  | 28 +++++++++++++++++++
 cranelift/codegen/src/verifier/mod.rs | 39 +++------------------------
 cranelift/frontend/Cargo.toml         |  3 +++
 cranelift/frontend/src/frontend.rs    | 11 ++++++++
 4 files changed, 45 insertions(+), 36 deletions(-)

diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index c856f70e54..adbb244278 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -19,6 +19,9 @@ use crate::value_label::ValueLabelsRanges;
 use crate::write::write_function;
 use core::fmt;
 
+#[cfg(feature = "basic-blocks")]
+use crate::ir::{Inst, Opcode};
+
 /// A function.
 ///
 /// Functions can be cloned, but it is not a very fast operation.
@@ -219,6 +222,31 @@ impl Function {
     pub fn collect_debug_info(&mut self) {
         self.dfg.collect_debug_info();
     }
+
+    /// Checks that the specified EBB can be encoded as a basic block.
+    ///
+    /// On error, returns the first invalid instruction and an error message.
+    #[cfg(feature = "basic-blocks")]
+    pub fn is_ebb_basic(&self, ebb: Ebb) -> Result<(), (Inst, &'static str)> {
+        let dfg = &self.dfg;
+        let inst_iter = self.layout.ebb_insts(ebb);
+
+        // Ignore all instructions prior to the first branch.
+        let mut inst_iter = inst_iter.skip_while(|&inst| !dfg[inst].opcode().is_branch());
+
+        // A conditional branch is permitted in a basic block only when followed
+        // by a terminal jump or fallthrough instruction.
+        if let Some(_branch) = inst_iter.next() {
+            if let Some(next) = inst_iter.next() {
+                match dfg[next].opcode() {
+                    Opcode::Fallthrough | Opcode::Jump => (),
+                    _ => return Err((next, "post-branch instruction not fallthrough or jump")),
+                }
+            }
+        }
+
+        Ok(())
+    }
 }
 
 /// Additional annotations for function display.
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 276e43adf9..a09a696fff 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -472,42 +472,9 @@ impl<'a> Verifier<'a> {
     /// branching instructions are ending the EBB.
     #[cfg(feature = "basic-blocks")]
     fn encodable_as_bb(&self, ebb: Ebb, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
-        let dfg = &self.func.dfg;
-        let inst_iter = self.func.layout.ebb_insts(ebb);
-        // Skip non-branching instructions.
-        let mut inst_iter = inst_iter.skip_while(|&inst| !dfg[inst].opcode().is_branch());
-
-        let branch = match inst_iter.next() {
-            // There is no branch in the current EBB.
-            None => return Ok(()),
-            Some(br) => br,
-        };
-
-        let after_branch = match inst_iter.next() {
-            // The branch is also the terminator.
-            None => return Ok(()),
-            Some(inst) => inst,
-        };
-
-        let after_branch_opcode = dfg[after_branch].opcode();
-        if !after_branch_opcode.is_terminator() {
-            return fatal!(
-                errors,
-                branch,
-                "branch followed by a non-terminator instruction."
-            );
-        };
-
-        // Allow only one conditional branch and a fallthrough implemented with
-        // a jump or fallthrough instruction. Any other, which returns or check
-        // a different condition would have to be moved to a different EBB.
-        match after_branch_opcode {
-            Opcode::Fallthrough | Opcode::Jump => Ok(()),
-            _ => fatal!(
-                errors,
-                after_branch,
-                "terminator instruction not fallthrough or jump"
-            ),
+        match self.func.is_ebb_basic(ebb) {
+            Ok(()) => Ok(()),
+            Err((inst, message)) => fatal!(errors, inst, message),
         }
     }
 
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index fc6ca6eb31..835bd3734a 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -21,6 +21,9 @@ default = ["std"]
 std = ["cranelift-codegen/std"]
 core = ["hashmap_core", "cranelift-codegen/core"]
 
+# Temporary feature that enforces basic block semantics.
+basic-blocks = ["cranelift-codegen/basic-blocks"]
+
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 355b460067..29dd5ca1df 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -470,6 +470,16 @@ impl<'a> FunctionBuilder<'a> {
             "all blocks should be filled before dropping a FunctionBuilder"
         );
 
+        // Check that all blocks are valid basic blocks.
+        #[cfg(feature = "basic-blocks")]
+        debug_assert!(
+            self.func_ctx
+                .ebbs
+                .keys()
+                .all(|ebb| self.func.is_ebb_basic(ebb).is_ok()),
+            "all blocks should be encodable as basic blocks"
+        );
+
         // Clear the state (but preserve the allocated buffers) in preparation
         // for translation another function.
         self.func_ctx.clear();
@@ -840,6 +850,7 @@ impl<'a> FunctionBuilder<'a> {
         );
     }
 
+    /// An Ebb is 'filled' when a terminator instruction is present.
     fn fill_current_block(&mut self) {
         self.func_ctx.ebbs[self.position.ebb.unwrap()].filled = true;
     }

From 7a72ffefdd521c761b88b165d85881428d0393da Mon Sep 17 00:00:00 2001
From: Artur Jamro 
Date: Fri, 19 Jul 2019 15:53:12 -0700
Subject: [PATCH 2532/3084] Add serde derive to PrimaryMap

---
 cranelift/entity/Cargo.toml     | 4 ++++
 cranelift/entity/src/primary.rs | 3 +++
 2 files changed, 7 insertions(+)

diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 0e390b9b7b..d25353da92 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -11,10 +11,14 @@ readme = "README.md"
 keywords = ["entity", "set", "map"]
 edition = "2018"
 
+[dependencies]
+serde = { version = "1.0.94", features = ["derive"], optional = true }
+
 [features]
 default = ["std"]
 std = []
 core = []
+enable-serde = ["serde"]
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/src/primary.rs b/cranelift/entity/src/primary.rs
index d3fc7470c0..257cb240f4 100644
--- a/cranelift/entity/src/primary.rs
+++ b/cranelift/entity/src/primary.rs
@@ -7,6 +7,8 @@ use core::iter::FromIterator;
 use core::marker::PhantomData;
 use core::ops::{Index, IndexMut};
 use core::slice;
+#[cfg(feature = "enable-serde")]
+use serde::{Deserialize, Serialize};
 use std::boxed::Box;
 use std::vec::Vec;
 
@@ -26,6 +28,7 @@ use std::vec::Vec;
 /// plain slice would make it easier to use incorrectly. To make a slice of a `PrimaryMap`, use
 /// `into_boxed_slice`.
 #[derive(Debug, Clone)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct PrimaryMap
 where
     K: EntityRef,

From 4641fdd302ebf0f4e38f49f581d41aac4f1f5eba Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 19 Jul 2019 16:28:40 -0700
Subject: [PATCH 2533/3084] Bump version to 0.36.0

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    |  8 ++++----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 7bb3bf55d4..d20e1948bd 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.35.0"
+version = "0.36.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.35.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.35.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.35.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.35.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.35.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.35.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.35.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.35.0" }
-cranelift-module = { path = "cranelift-module", version = "0.35.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.35.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.35.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.35.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.35.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.36.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.36.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.36.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.36.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.36.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.36.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.36.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.36.0" }
+cranelift-module = { path = "cranelift-module", version = "0.36.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.36.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.36.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.36.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.36.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index bd4c741199..46731b014e 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.35.0"
+version = "0.36.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.35.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.36.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index a9834dac07..dd5e8eebfd 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.35.0"
+version = "0.36.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.35.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.35.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.36.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.36.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -27,7 +27,7 @@ serde = { version = "1.0.94", features = ["derive"], optional = true }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.35.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.36.0", default-features = false }
 
 [features]
 default = ["std", "x86", "arm32", "arm64", "riscv"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index a20b7cd198..977d5d546e 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.35.0"
+version = "0.36.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.35.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.36.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index d25353da92..62853082a6 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.35.0"
+version = "0.36.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 7319bf3f7f..1c83d9bb10 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.35.0"
+version = "0.36.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.35.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.36.0" }
 faerie = "0.10.0"
 goblin = "0.0.22"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index f0b544cee2..f67c5c1490 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.35.0"
+version = "0.36.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../cranelift-reader", version = "0.35.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.35.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../cranelift-reader", version = "0.36.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.36.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 835bd3734a..12d9181218 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.35.0"
+version = "0.36.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 011c0830c9..412b70e1aa 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.35.0"
+version = "0.36.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.35.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.36.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 10a52c60ef..d36bccbf9d 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.35.0"
+version = "0.36.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index fe2707d739..71d9a7106b 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.35.0"
+version = "0.36.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.35.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.36.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 99beb98348..f176b80dc5 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.35.0"
+version="0.36.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index e2c48e859d..73df0da9eb 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.35.0"
+version = "0.36.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0" }
 target-lexicon = "0.4.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 5abfb104bb..156bbfb176 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.35.0"
+version = "0.36.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.35.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.36.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 1fcfc58860..fe273d8883 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.35.0"
+version = "0.36.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.35.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.35.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.36.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.36.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.35.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.35.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.35.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.36.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.36.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.36.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index a7cb8dc153..aca01b55d1 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.35.0"
+version = "0.36.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.35.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.36.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index a1e28e8b03..2f1555c315 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.35.0"
+version = "0.36.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.32.1", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.35.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.35.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.35.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.36.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.36.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 5d1deecbb4b7f4c22d4ad304a1bc65db62cf11be Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Fri, 19 Jul 2019 10:38:56 -0600
Subject: [PATCH 2534/3084] Give cranelift-wasm a "basic-blocks" feature.

---
 cranelift/wasm/Cargo.toml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 2f1555c315..5e9c144339 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -31,6 +31,9 @@ std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std"]
 core = ["hashmap_core", "cranelift-codegen/core", "cranelift-frontend/core", "wasmparser/core"]
 enable-serde = ["serde"]
 
+# Temporary feature that enforces basic block semantics.
+basic-blocks = ["cranelift-codegen/basic-blocks", "cranelift-frontend/basic-blocks"]
+
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }

From ffa9d315e6efb957901a4ae120d36a47a5cf90ee Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Mon, 22 Jul 2019 12:50:22 -0600
Subject: [PATCH 2535/3084] Add some comments to the frontend code.

---
 cranelift/frontend/src/frontend.rs    | 11 ++++++++++-
 cranelift/wasm/src/func_translator.rs |  3 ++-
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 29dd5ca1df..2ea5242d2a 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -41,8 +41,17 @@ pub struct FunctionBuilder<'a> {
 
 #[derive(Clone, Default)]
 struct EbbData {
-    filled: bool,
+    /// An Ebb is "pristine" iff no instructions have been added since the last
+    /// call to `switch_to_block()`.
     pristine: bool,
+
+    /// An Ebb is "filled" iff a terminator instruction has been inserted since
+    /// the last call to `switch_to_block()`.
+    ///
+    /// A filled block cannot be pristine.
+    filled: bool,
+
+    /// Count of parameters not supplied implicitly by the SSABuilder.
     user_param_count: usize,
 }
 
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index 6842e4f92f..665f7f8cb4 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -89,7 +89,8 @@ impl FuncTranslator {
         let entry_block = builder.create_ebb();
         builder.append_ebb_params_for_function_params(entry_block);
         builder.switch_to_block(entry_block); // This also creates values for the arguments.
-        builder.seal_block(entry_block);
+        builder.seal_block(entry_block); // Declare all predecessors known.
+
         // Make sure the entry block is inserted in the layout before we make any callbacks to
         // `environ`. The callback functions may need to insert things in the entry block.
         builder.ensure_inserted_ebb();

From 926e4c8bf82d14ac6485e3084b3f705bd326dd58 Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Mon, 22 Jul 2019 14:49:39 -0600
Subject: [PATCH 2536/3084] Generate basic blocks for Wasm `br_if`.

---
 cranelift/wasm/src/code_translator.rs | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 6365ed7e68..2fe9a7843b 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1244,6 +1244,14 @@ fn translate_br_if(
     let val = state.pop1();
     let (br_destination, inputs) = translate_br_if_args(relative_depth, state);
     builder.ins().brnz(val, br_destination, inputs);
+
+    #[cfg(feature = "basic-blocks")]
+    {
+        let next_ebb = builder.create_ebb();
+        builder.ins().jump(next_ebb, &[]);
+        builder.seal_block(next_ebb); // The only predecessor is the current block.
+        builder.switch_to_block(next_ebb);
+    }
 }
 
 fn translate_br_if_args(

From 70c91f913d453c40aaa3dd5bde33a03c2e7df277 Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Mon, 22 Jul 2019 15:02:40 -0600
Subject: [PATCH 2537/3084] Produce more helpful basic block errors in
 cranelift-frontend.

Previously, the error just notified that there was a failure.
The new-style error says specifically in which ebb, on which instruction.
---
 cranelift/frontend/src/frontend.rs | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 2ea5242d2a..a68f131458 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -479,15 +479,18 @@ impl<'a> FunctionBuilder<'a> {
             "all blocks should be filled before dropping a FunctionBuilder"
         );
 
-        // Check that all blocks are valid basic blocks.
+        // In debug mode, check that all blocks are valid basic blocks.
         #[cfg(feature = "basic-blocks")]
-        debug_assert!(
-            self.func_ctx
-                .ebbs
-                .keys()
-                .all(|ebb| self.func.is_ebb_basic(ebb).is_ok()),
-            "all blocks should be encodable as basic blocks"
-        );
+        #[cfg(debug_assertions)]
+        {
+            // Iterate manually to provide more helpful error messages.
+            for ebb in self.func_ctx.ebbs.keys() {
+                if let Err((inst, _msg)) = self.func.is_ebb_basic(ebb) {
+                    let inst_str = self.func.dfg.display_inst(inst, None);
+                    panic!("{} failed basic block invariants on {}", ebb, inst_str);
+                }
+            }
+        }
 
         // Clear the state (but preserve the allocated buffers) in preparation
         // for translation another function.

From 6792c062bf7d1b8964070888b66e7f18576cc46f Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Wed, 24 Jul 2019 18:28:59 +0200
Subject: [PATCH 2538/3084] Add basic-blocks feature flag at the top-level
 Cargo.toml.

---
 cranelift/Cargo.toml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index d20e1948bd..75cf4c32e3 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -46,6 +46,7 @@ file-per-thread-logger = "0.1.2"
 default = ["disas", "wasm"]
 disas = ["capstone"]
 wasm = ["wabt", "cranelift-wasm"]
+basic-blocks = ["cranelift-codegen/basic-blocks", "cranelift-frontend/basic-blocks", "cranelift-wasm/basic-blocks"]
 
 # We want debug symbols on release binaries by default since it allows profiling
 # tools to give more accurate information. We can always strip them out later if

From 42ebd2a5f4bdea0a53dacbfb1c2a34cba4fd7ac3 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Wed, 24 Jul 2019 18:27:57 +0200
Subject: [PATCH 2539/3084] Fix licm/jump-table-entry.clif test case to work
 with basic blocks.

---
 cranelift/filetests/filetests/licm/jump-table-entry.clif | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/cranelift/filetests/filetests/licm/jump-table-entry.clif b/cranelift/filetests/filetests/licm/jump-table-entry.clif
index 00a6b3fd14..1d41af25db 100644
--- a/cranelift/filetests/filetests/licm/jump-table-entry.clif
+++ b/cranelift/filetests/filetests/licm/jump-table-entry.clif
@@ -1,5 +1,4 @@
 test licm
-
 target x86_64
 
 function %dont_hoist_jump_table_entry_during_licm() {
@@ -18,6 +17,9 @@ ebb2:
     v1 = iconst.i32 -14
     v8 = ifcmp_imm v1, 2
     brif uge v8, ebb1
+    jump ebb3
+
+ebb3:
     v5 = jump_table_base.i64 jt0
     v6 = jump_table_entry.i64 v1, v5, 4, jt0
     v7 = iadd v5, v6
@@ -25,5 +27,7 @@ ebb2:
 ; check: ebb2:
 ; nextln: v8 = ifcmp_imm.i32 v1, 2
 ; nextln: brif uge v8, ebb1
+; nextln: jump ebb3
+; check: ebb3:
 ; nextln: jump_table_entry.i64
 }

From feecd239671350d98dae5f4244cc7a5fa5e469e9 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Thu, 25 Jul 2019 14:33:25 +0200
Subject: [PATCH 2540/3084] Assert if newly added instructions break the Basic
 Block invariant.

---
 cranelift/codegen/src/cursor.rs | 45 +++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)

diff --git a/cranelift/codegen/src/cursor.rs b/cranelift/codegen/src/cursor.rs
index dbb459ed7a..e688a90b3c 100644
--- a/cranelift/codegen/src/cursor.rs
+++ b/cranelift/codegen/src/cursor.rs
@@ -635,6 +635,28 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut FuncCursor<'f> {
     }
 
     fn insert_built_inst(self, inst: ir::Inst, _: ir::Type) -> &'c mut ir::DataFlowGraph {
+        // TODO: Remove this assertion once #796 is fixed.
+        #[cfg(feature = "basic-blocks")]
+        #[cfg(debug_assertions)]
+        {
+            if let CursorPosition::At(_) = self.position() {
+                if let Some(curr) = self.current_inst() {
+                    if let Some(prev) = self.layout().prev_inst(curr) {
+                        let prev_op = self.data_flow_graph()[prev].opcode();
+                        let inst_op = self.data_flow_graph()[inst].opcode();
+                        let curr_op = self.data_flow_graph()[curr].opcode();
+                        if prev_op.is_branch() && !prev_op.is_terminator() {
+                            if !inst_op.is_terminator() {
+                                panic!(
+                                    "Inserting instruction {} after {}, and before {}",
+                                    inst_op, prev_op, curr_op
+                                )
+                            };
+                        }
+                    };
+                };
+            };
+        }
         self.insert_inst(inst);
         if !self.srcloc.is_default() {
             self.func.srclocs[inst] = self.srcloc;
@@ -742,6 +764,29 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut EncCursor<'f> {
         inst: ir::Inst,
         ctrl_typevar: ir::Type,
     ) -> &'c mut ir::DataFlowGraph {
+        // TODO: Remove this assertion once #796 is fixed.
+        #[cfg(feature = "basic-blocks")]
+        #[cfg(debug_assertions)]
+        {
+            if let CursorPosition::At(_) = self.position() {
+                if let Some(curr) = self.current_inst() {
+                    if let Some(prev) = self.layout().prev_inst(curr) {
+                        let prev_op = self.data_flow_graph()[prev].opcode();
+                        let inst_op = self.data_flow_graph()[inst].opcode();
+                        if prev_op.is_branch() && !prev_op.is_terminator() {
+                            if !inst_op.is_terminator() {
+                                panic!(
+                                    "Inserting instruction {} after {} and before {}",
+                                    self.display_inst(inst),
+                                    self.display_inst(prev),
+                                    self.display_inst(curr)
+                                )
+                            };
+                        }
+                    };
+                };
+            };
+        }
         // Insert the instruction and remember the reference.
         self.insert_inst(inst);
         self.built_inst = Some(inst);

From b7a9d654585ad52212a867c1b1a6e9120a91f993 Mon Sep 17 00:00:00 2001
From: Andy Wortman 
Date: Thu, 25 Jul 2019 09:36:17 -0700
Subject: [PATCH 2541/3084] cranelift-wasm hooks to instrument wasm operators
 (#861)

* cranelift-wasm hooks to instrument wasm operators
---
 cranelift/wasm/src/code_translator.rs | 100 +++++++++++++-------------
 cranelift/wasm/src/environ/spec.rs    |  22 ++++++
 cranelift/wasm/src/func_translator.rs |   4 +-
 cranelift/wasm/src/state.rs           |  12 +++-
 4 files changed, 84 insertions(+), 54 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 2fe9a7843b..4717cf4904 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -42,7 +42,7 @@ use wasmparser::{MemoryImmediate, Operator};
 /// Translates wasm operators into Cranelift IR instructions. Returns `true` if it inserted
 /// a return.
 pub fn translate_operator(
-    op: Operator,
+    op: &Operator,
     builder: &mut FunctionBuilder,
     state: &mut TranslationState,
     environ: &mut FE,
@@ -59,28 +59,28 @@ pub fn translate_operator(
          *  disappear in the Cranelift Code
          ***********************************************************************************/
         Operator::GetLocal { local_index } => {
-            let val = builder.use_var(Variable::with_u32(local_index));
+            let val = builder.use_var(Variable::with_u32(*local_index));
             state.push1(val);
-            let label = ValueLabel::from_u32(local_index);
+            let label = ValueLabel::from_u32(*local_index);
             builder.set_val_label(val, label);
         }
         Operator::SetLocal { local_index } => {
             let val = state.pop1();
-            builder.def_var(Variable::with_u32(local_index), val);
-            let label = ValueLabel::from_u32(local_index);
+            builder.def_var(Variable::with_u32(*local_index), val);
+            let label = ValueLabel::from_u32(*local_index);
             builder.set_val_label(val, label);
         }
         Operator::TeeLocal { local_index } => {
             let val = state.peek1();
-            builder.def_var(Variable::with_u32(local_index), val);
-            let label = ValueLabel::from_u32(local_index);
+            builder.def_var(Variable::with_u32(*local_index), val);
+            let label = ValueLabel::from_u32(*local_index);
             builder.set_val_label(val, label);
         }
         /********************************** Globals ****************************************
          *  `get_global` and `set_global` are handled by the environment.
          ***********************************************************************************/
         Operator::GetGlobal { global_index } => {
-            let val = match state.get_global(builder.func, global_index, environ)? {
+            let val = match state.get_global(builder.func, *global_index, environ)? {
                 GlobalVariable::Const(val) => val,
                 GlobalVariable::Memory { gv, offset, ty } => {
                     let addr = builder.ins().global_value(environ.pointer_type(), gv);
@@ -91,8 +91,8 @@ pub fn translate_operator(
             state.push1(val);
         }
         Operator::SetGlobal { global_index } => {
-            match state.get_global(builder.func, global_index, environ)? {
-                GlobalVariable::Const(_) => panic!("global #{} is a constant", global_index),
+            match state.get_global(builder.func, *global_index, environ)? {
+                GlobalVariable::Const(_) => panic!("global #{} is a constant", *global_index),
                 GlobalVariable::Memory { gv, offset, ty } => {
                     let addr = builder.ins().global_value(environ.pointer_type(), gv);
                     let flags = ir::MemFlags::trusted();
@@ -132,19 +132,19 @@ pub fn translate_operator(
          ***********************************************************************************/
         Operator::Block { ty } => {
             let next = builder.create_ebb();
-            if let Ok(ty_cre) = blocktype_to_type(ty) {
+            if let Ok(ty_cre) = blocktype_to_type(*ty) {
                 builder.append_ebb_param(next, ty_cre);
             }
-            state.push_block(next, num_return_values(ty)?);
+            state.push_block(next, num_return_values(*ty)?);
         }
         Operator::Loop { ty } => {
             let loop_body = builder.create_ebb();
             let next = builder.create_ebb();
-            if let Ok(ty_cre) = blocktype_to_type(ty) {
+            if let Ok(ty_cre) = blocktype_to_type(*ty) {
                 builder.append_ebb_param(next, ty_cre);
             }
             builder.ins().jump(loop_body, &[]);
-            state.push_loop(loop_body, next, num_return_values(ty)?);
+            state.push_loop(loop_body, next, num_return_values(*ty)?);
             builder.switch_to_block(loop_body);
             environ.translate_loop_header(builder.cursor())?;
         }
@@ -158,10 +158,10 @@ pub fn translate_operator(
             //   and we add nothing;
             // - either the If have an Else clause, in that case the destination of this jump
             //   instruction will be changed later when we translate the Else operator.
-            if let Ok(ty_cre) = blocktype_to_type(ty) {
+            if let Ok(ty_cre) = blocktype_to_type(*ty) {
                 builder.append_ebb_param(if_not, ty_cre);
             }
-            state.push_if(jump_inst, if_not, num_return_values(ty)?);
+            state.push_if(jump_inst, if_not, num_return_values(*ty)?);
         }
         Operator::Else => {
             // We take the control frame pushed by the if, use its ebb as the else body
@@ -235,7 +235,7 @@ pub fn translate_operator(
          * `br_table`.
          ***********************************************************************************/
         Operator::Br { relative_depth } => {
-            let i = state.control_stack.len() - 1 - (relative_depth as usize);
+            let i = state.control_stack.len() - 1 - (*relative_depth as usize);
             let (return_count, br_destination) = {
                 let frame = &mut state.control_stack[i];
                 // We signal that all the code that follows until the next End is unreachable
@@ -253,7 +253,7 @@ pub fn translate_operator(
             state.popn(return_count);
             state.reachable = false;
         }
-        Operator::BrIf { relative_depth } => translate_br_if(relative_depth, builder, state),
+        Operator::BrIf { relative_depth } => translate_br_if(*relative_depth, builder, state),
         Operator::BrTable { table } => {
             let (depths, default) = table.read_table()?;
             let mut min_depth = default;
@@ -357,10 +357,10 @@ pub fn translate_operator(
          * argument referring to an index in the external functions table of the module.
          ************************************************************************************/
         Operator::Call { function_index } => {
-            let (fref, num_args) = state.get_direct_func(builder.func, function_index, environ)?;
+            let (fref, num_args) = state.get_direct_func(builder.func, *function_index, environ)?;
             let call = environ.translate_call(
                 builder.cursor(),
-                FuncIndex::from_u32(function_index),
+                FuncIndex::from_u32(*function_index),
                 fref,
                 state.peekn(num_args),
             )?;
@@ -378,14 +378,14 @@ pub fn translate_operator(
         Operator::CallIndirect { index, table_index } => {
             // `index` is the index of the function's signature and `table_index` is the index of
             // the table to search the function in.
-            let (sigref, num_args) = state.get_indirect_sig(builder.func, index, environ)?;
-            let table = state.get_table(builder.func, table_index, environ)?;
+            let (sigref, num_args) = state.get_indirect_sig(builder.func, *index, environ)?;
+            let table = state.get_table(builder.func, *table_index, environ)?;
             let callee = state.pop1();
             let call = environ.translate_call_indirect(
                 builder.cursor(),
-                TableIndex::from_u32(table_index),
+                TableIndex::from_u32(*table_index),
                 table,
-                SignatureIndex::from_u32(index),
+                SignatureIndex::from_u32(*index),
                 sigref,
                 callee,
                 state.peekn(num_args),
@@ -406,14 +406,14 @@ pub fn translate_operator(
         Operator::MemoryGrow { reserved } => {
             // The WebAssembly MVP only supports one linear memory, but we expect the reserved
             // argument to be a memory index.
-            let heap_index = MemoryIndex::from_u32(reserved);
-            let heap = state.get_heap(builder.func, reserved, environ)?;
+            let heap_index = MemoryIndex::from_u32(*reserved);
+            let heap = state.get_heap(builder.func, *reserved, environ)?;
             let val = state.pop1();
             state.push1(environ.translate_memory_grow(builder.cursor(), heap_index, heap, val)?)
         }
         Operator::MemorySize { reserved } => {
-            let heap_index = MemoryIndex::from_u32(reserved);
-            let heap = state.get_heap(builder.func, reserved, environ)?;
+            let heap_index = MemoryIndex::from_u32(*reserved);
+            let heap = state.get_heap(builder.func, *reserved, environ)?;
             state.push1(environ.translate_memory_size(builder.cursor(), heap_index, heap)?);
         }
         /******************************* Load instructions ***********************************
@@ -423,72 +423,72 @@ pub fn translate_operator(
         Operator::I32Load8U {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Uload8, I32, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Uload8, I32, builder, state, environ)?;
         }
         Operator::I32Load16U {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Uload16, I32, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Uload16, I32, builder, state, environ)?;
         }
         Operator::I32Load8S {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Sload8, I32, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Sload8, I32, builder, state, environ)?;
         }
         Operator::I32Load16S {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Sload16, I32, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Sload16, I32, builder, state, environ)?;
         }
         Operator::I64Load8U {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Uload8, I64, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Uload8, I64, builder, state, environ)?;
         }
         Operator::I64Load16U {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Uload16, I64, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Uload16, I64, builder, state, environ)?;
         }
         Operator::I64Load8S {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Sload8, I64, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Sload8, I64, builder, state, environ)?;
         }
         Operator::I64Load16S {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Sload16, I64, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Sload16, I64, builder, state, environ)?;
         }
         Operator::I64Load32S {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Sload32, I64, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Sload32, I64, builder, state, environ)?;
         }
         Operator::I64Load32U {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Uload32, I64, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Uload32, I64, builder, state, environ)?;
         }
         Operator::I32Load {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Load, I32, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Load, I32, builder, state, environ)?;
         }
         Operator::F32Load {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Load, F32, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Load, F32, builder, state, environ)?;
         }
         Operator::I64Load {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Load, I64, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Load, I64, builder, state, environ)?;
         }
         Operator::F64Load {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_load(offset, ir::Opcode::Load, F64, builder, state, environ)?;
+            translate_load(*offset, ir::Opcode::Load, F64, builder, state, environ)?;
         }
         /****************************** Store instructions ***********************************
          * Wasm specifies an integer alignment flag but we drop it in Cranelift.
@@ -506,7 +506,7 @@ pub fn translate_operator(
         | Operator::F64Store {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_store(offset, ir::Opcode::Store, builder, state, environ)?;
+            translate_store(*offset, ir::Opcode::Store, builder, state, environ)?;
         }
         Operator::I32Store8 {
             memarg: MemoryImmediate { flags: _, offset },
@@ -514,7 +514,7 @@ pub fn translate_operator(
         | Operator::I64Store8 {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_store(offset, ir::Opcode::Istore8, builder, state, environ)?;
+            translate_store(*offset, ir::Opcode::Istore8, builder, state, environ)?;
         }
         Operator::I32Store16 {
             memarg: MemoryImmediate { flags: _, offset },
@@ -522,21 +522,21 @@ pub fn translate_operator(
         | Operator::I64Store16 {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_store(offset, ir::Opcode::Istore16, builder, state, environ)?;
+            translate_store(*offset, ir::Opcode::Istore16, builder, state, environ)?;
         }
         Operator::I64Store32 {
             memarg: MemoryImmediate { flags: _, offset },
         } => {
-            translate_store(offset, ir::Opcode::Istore32, builder, state, environ)?;
+            translate_store(*offset, ir::Opcode::Istore32, builder, state, environ)?;
         }
         /****************************** Nullary Operators ************************************/
-        Operator::I32Const { value } => state.push1(builder.ins().iconst(I32, i64::from(value))),
-        Operator::I64Const { value } => state.push1(builder.ins().iconst(I64, value)),
+        Operator::I32Const { value } => state.push1(builder.ins().iconst(I32, i64::from(*value))),
+        Operator::I64Const { value } => state.push1(builder.ins().iconst(I64, *value)),
         Operator::F32Const { value } => {
-            state.push1(builder.ins().f32const(f32_translation(value)));
+            state.push1(builder.ins().f32const(f32_translation(*value)));
         }
         Operator::F64Const { value } => {
-            state.push1(builder.ins().f64const(f64_translation(value)));
+            state.push1(builder.ins().f64const(f64_translation(*value)));
         }
         /******************************* Unary Operators *************************************/
         Operator::I32Clz | Operator::I64Clz => {
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index d307922677..6837e30187 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -14,9 +14,11 @@ use cranelift_codegen::cursor::FuncCursor;
 use cranelift_codegen::ir::immediates::Offset32;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::isa::TargetFrontendConfig;
+use cranelift_frontend::FunctionBuilder;
 use failure_derive::Fail;
 use std::boxed::Box;
 use wasmparser::BinaryReaderError;
+use wasmparser::Operator;
 
 /// The value of a WebAssembly global variable.
 #[derive(Clone, Copy)]
@@ -253,6 +255,26 @@ pub trait FuncEnvironment {
         // By default, don't emit anything.
         Ok(())
     }
+
+    /// Optional callback for the `FunctionEnvironment` performing this translation to maintain
+    /// internal state or prepare custom state for the operator to translate
+    fn before_translate_operator(
+        &mut self,
+        _op: &Operator,
+        _builder: &mut FunctionBuilder,
+    ) -> WasmResult<()> {
+        Ok(())
+    }
+
+    /// Optional callback for the `FunctionEnvironment` performing this translation to maintain
+    /// internal state or finalize custom state for the operator that was translated
+    fn after_translate_operator(
+        &mut self,
+        _op: &Operator,
+        _builder: &mut FunctionBuilder,
+    ) -> WasmResult<()> {
+        Ok(())
+    }
 }
 
 /// An object satisfying the `ModuleEnvironment` trait can be passed as argument to the
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index 665f7f8cb4..9d9e923db5 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -207,7 +207,9 @@ fn parse_function_body(
     while !state.control_stack.is_empty() {
         builder.set_srcloc(cur_srcloc(&reader));
         let op = reader.read_operator()?;
-        translate_operator(op, builder, state, environ)?;
+        environ.before_translate_operator(&op, builder)?;
+        translate_operator(&op, builder, state, environ)?;
+        environ.after_translate_operator(&op, builder)?;
     }
 
     // The final `End` operator left us in the exit block where we need to manually add a return
diff --git a/cranelift/wasm/src/state.rs b/cranelift/wasm/src/state.rs
index adfa903748..f791c98032 100644
--- a/cranelift/wasm/src/state.rs
+++ b/cranelift/wasm/src/state.rs
@@ -130,8 +130,13 @@ impl ControlStackFrame {
 /// - The depth of the two unreachable control blocks stacks, that are manipulated when translating
 ///   unreachable code;
 pub struct TranslationState {
+    /// A stack of values corresponding to the active values in the input wasm function at this
+    /// point.
     pub stack: Vec,
+    /// A stack of active control flow operations at this point in the input wasm function.
     pub control_stack: Vec,
+    /// Is the current translation state still reachable? This is false when translating operators
+    /// like End, Return, or Unreachable.
     pub reachable: bool,
 
     // Map of global variables that have already been created by `FuncEnvironment::make_global`.
@@ -155,6 +160,7 @@ pub struct TranslationState {
 }
 
 impl TranslationState {
+    /// Construct a new, empty, `TranslationState`
     pub fn new() -> Self {
         Self {
             stack: Vec::new(),
@@ -242,7 +248,7 @@ impl TranslationState {
         &self.stack[self.stack.len() - n..]
     }
 
-    // Push a block on the control stack.
+    /// Push a block on the control stack.
     pub fn push_block(&mut self, following_code: Ebb, num_result_types: usize) {
         self.control_stack.push(ControlStackFrame::Block {
             destination: following_code,
@@ -252,7 +258,7 @@ impl TranslationState {
         });
     }
 
-    // Push a loop on the control stack.
+    /// Push a loop on the control stack.
     pub fn push_loop(&mut self, header: Ebb, following_code: Ebb, num_result_types: usize) {
         self.control_stack.push(ControlStackFrame::Loop {
             header,
@@ -262,7 +268,7 @@ impl TranslationState {
         });
     }
 
-    // Push an if on the control stack.
+    /// Push an if on the control stack.
     pub fn push_if(&mut self, branch_inst: Inst, following_code: Ebb, num_result_types: usize) {
         self.control_stack.push(ControlStackFrame::If {
             branch_inst,

From be36cc6538fee9b0f652f14406a0296b33e07ba5 Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Wed, 24 Jul 2019 13:46:24 -0600
Subject: [PATCH 2542/3084] Generate basic blocks for wasm if..then..else.

---
 cranelift/wasm/src/code_translator.rs | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 4717cf4904..43d08c6713 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -152,6 +152,15 @@ pub fn translate_operator(
             let val = state.pop1();
             let if_not = builder.create_ebb();
             let jump_inst = builder.ins().brz(val, if_not, &[]);
+
+            #[cfg(feature = "basic-blocks")]
+            {
+                let next_ebb = builder.create_ebb();
+                builder.ins().jump(next_ebb, &[]);
+                builder.seal_block(next_ebb); // Only predecessor is the current block.
+                builder.switch_to_block(next_ebb);
+            }
+
             // Here we append an argument to an Ebb targeted by an argumentless jump instruction
             // But in fact there are two cases:
             // - either the If does not have a Else clause, in that case ty = EmptyBlock

From 09ec0d4149391b6b73513a612ddab8e42d8a9253 Mon Sep 17 00:00:00 2001
From: Artur Jamro 
Date: Wed, 24 Jul 2019 17:18:50 -0700
Subject: [PATCH 2543/3084] Derive Hash for some types

---
 cranelift/entity/src/primary.rs         |  2 +-
 cranelift/wasm/src/translation_utils.rs | 10 +++++-----
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/cranelift/entity/src/primary.rs b/cranelift/entity/src/primary.rs
index 257cb240f4..6734a1f5e5 100644
--- a/cranelift/entity/src/primary.rs
+++ b/cranelift/entity/src/primary.rs
@@ -27,7 +27,7 @@ use std::vec::Vec;
 /// that it only allows indexing with the distinct `EntityRef` key type, so converting to a
 /// plain slice would make it easier to use incorrectly. To make a slice of a `PrimaryMap`, use
 /// `into_boxed_slice`.
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, Hash)]
 #[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct PrimaryMap
 where
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index 759e4ab4ff..a41c70c4d9 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -54,7 +54,7 @@ pub struct SignatureIndex(u32);
 entity_impl!(SignatureIndex);
 
 /// WebAssembly global.
-#[derive(Debug, Clone, Copy)]
+#[derive(Debug, Clone, Copy, Hash)]
 pub struct Global {
     /// The type of the value stored in the global.
     pub ty: ir::Type,
@@ -65,7 +65,7 @@ pub struct Global {
 }
 
 /// Globals are initialized via the four `const` operators or by referring to another import.
-#[derive(Debug, Clone, Copy)]
+#[derive(Debug, Clone, Copy, Hash)]
 pub enum GlobalInit {
     /// An `i32.const`.
     I32Const(i32),
@@ -82,7 +82,7 @@ pub enum GlobalInit {
 }
 
 /// WebAssembly table.
-#[derive(Debug, Clone, Copy)]
+#[derive(Debug, Clone, Copy, Hash)]
 pub struct Table {
     /// The type of data stored in elements of the table.
     pub ty: TableElementType,
@@ -93,7 +93,7 @@ pub struct Table {
 }
 
 /// WebAssembly table element. Can be a function or a scalar type.
-#[derive(Debug, Clone, Copy)]
+#[derive(Debug, Clone, Copy, Hash)]
 pub enum TableElementType {
     /// A scalar type.
     Val(ir::Type),
@@ -102,7 +102,7 @@ pub enum TableElementType {
 }
 
 /// WebAssembly linear memory.
-#[derive(Debug, Clone, Copy)]
+#[derive(Debug, Clone, Copy, Hash)]
 pub struct Memory {
     /// The minimum number of pages in the memory.
     pub minimum: u32,

From 16e16c49a7b60fe11641b30ee8d0b6ad057ee26e Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Sat, 27 Jul 2019 04:50:19 -0700
Subject: [PATCH 2544/3084] Bump version to 0.37.0

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    |  8 ++++----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 75cf4c32e3..4c7ada1eae 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.36.0"
+version = "0.37.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.36.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.36.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.36.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.36.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.36.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.36.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.36.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.36.0" }
-cranelift-module = { path = "cranelift-module", version = "0.36.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.36.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.36.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.36.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.36.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.37.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.37.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.37.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.37.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.37.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.37.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.37.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.37.0" }
+cranelift-module = { path = "cranelift-module", version = "0.37.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.37.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.37.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.37.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.37.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 46731b014e..cfb3bac2a3 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.36.0"
+version = "0.37.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.36.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.37.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index dd5e8eebfd..33dfa3b88a 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.36.0"
+version = "0.37.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.36.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.36.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.37.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.37.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -27,7 +27,7 @@ serde = { version = "1.0.94", features = ["derive"], optional = true }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.36.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.37.0", default-features = false }
 
 [features]
 default = ["std", "x86", "arm32", "arm64", "riscv"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 977d5d546e..80f2364b38 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.36.0"
+version = "0.37.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.36.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.37.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 62853082a6..1c34b3f359 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.36.0"
+version = "0.37.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 1c83d9bb10..d853b0a4b3 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.36.0"
+version = "0.37.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.36.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.37.0" }
 faerie = "0.10.0"
 goblin = "0.0.22"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index f67c5c1490..cc2f374ff9 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.36.0"
+version = "0.37.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../cranelift-reader", version = "0.36.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.36.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../cranelift-reader", version = "0.37.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.37.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 12d9181218..09937ee9b2 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.36.0"
+version = "0.37.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 412b70e1aa..67f5456de7 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.36.0"
+version = "0.37.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.36.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.37.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index d36bccbf9d..8a407d521b 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.36.0"
+version = "0.37.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 71d9a7106b..0bd5376c21 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.36.0"
+version = "0.37.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.36.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.37.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index f176b80dc5..dc017d69ee 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.36.0"
+version="0.37.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 73df0da9eb..7c5c5b6204 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.36.0"
+version = "0.37.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0" }
 target-lexicon = "0.4.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 156bbfb176..ade3cb9ace 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.36.0"
+version = "0.37.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.36.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.37.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index fe273d8883..5a87f4bc8e 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.36.0"
+version = "0.37.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.36.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.36.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.37.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.37.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.36.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.36.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.36.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.37.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.37.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.37.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index aca01b55d1..08283067e0 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.36.0"
+version = "0.37.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.36.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.37.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 5e9c144339..426a202157 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.36.0"
+version = "0.37.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.32.1", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.36.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.36.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.36.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.37.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.37.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 4074ce2f5d265c3b7ae7e288b09c820373366bd7 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 25 Jul 2019 10:10:30 -0700
Subject: [PATCH 2545/3084] Add a unit test for ir::Type::lane_type() for a
 vector type

Closes #834
---
 cranelift/codegen/src/ir/types.rs | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/cranelift/codegen/src/ir/types.rs b/cranelift/codegen/src/ir/types.rs
index 9da5b971b5..fd28b5bd7c 100644
--- a/cranelift/codegen/src/ir/types.rs
+++ b/cranelift/codegen/src/ir/types.rs
@@ -363,6 +363,9 @@ mod tests {
         assert_eq!(I64, I64.lane_type());
         assert_eq!(F32, F32.lane_type());
         assert_eq!(F64, F64.lane_type());
+        assert_eq!(B1, B1.by(8).unwrap().lane_type());
+        assert_eq!(I32, I32X4.lane_type());
+        assert_eq!(F64, F64X2.lane_type());
 
         assert_eq!(INVALID.lane_bits(), 0);
         assert_eq!(IFLAGS.lane_bits(), 0);

From 5ca13a70a581d07e2410324b12dc104670002a14 Mon Sep 17 00:00:00 2001
From: laizy 
Date: Tue, 16 Jul 2019 13:19:39 +0800
Subject: [PATCH 2546/3084] check no other sections after data section

---
 cranelift/wasm/src/module_translator.rs | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/cranelift/wasm/src/module_translator.rs b/cranelift/wasm/src/module_translator.rs
index 72f257acea..669f9154e8 100644
--- a/cranelift/wasm/src/module_translator.rs
+++ b/cranelift/wasm/src/module_translator.rs
@@ -1,6 +1,6 @@
 //! Translation skeleton that traverses the whole WebAssembly module and call helper functions
 //! to deal with each part of it.
-use crate::environ::{ModuleEnvironment, WasmResult};
+use crate::environ::{ModuleEnvironment, WasmError, WasmResult};
 use crate::sections_translator::{
     parse_code_section, parse_data_section, parse_element_section, parse_export_section,
     parse_function_section, parse_global_section, parse_import_section, parse_memory_section,
@@ -139,5 +139,13 @@ pub fn translate_module<'data>(
         parse_data_section(data, environ)?;
     }
 
+    reader.skip_custom_sections()?;
+    if !reader.eof() {
+        return Err(WasmError::InvalidWebAssembly {
+            message: "sections must occur at most once and in the prescribed order",
+            offset: reader.current_position(),
+        });
+    }
+
     Ok(())
 }

From b5cb8556f62378b7608e2269baf063a83b25b26d Mon Sep 17 00:00:00 2001
From: Andy Wortman 
Date: Tue, 30 Jul 2019 07:32:50 -0700
Subject: [PATCH 2547/3084] [wasm] Pass translation state along for
 before/after_translate_operator callbacks (#879)

---
 cranelift/wasm/src/environ/spec.rs    | 3 +++
 cranelift/wasm/src/func_translator.rs | 4 ++--
 cranelift/wasm/src/lib.rs             | 1 +
 3 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 6837e30187..71c11e0b23 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -6,6 +6,7 @@
 //!
 //! [Wasmtime]: https://github.com/CraneStation/wasmtime
 
+use crate::state::TranslationState;
 use crate::translation_utils::{
     FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
 };
@@ -262,6 +263,7 @@ pub trait FuncEnvironment {
         &mut self,
         _op: &Operator,
         _builder: &mut FunctionBuilder,
+        _state: &mut TranslationState,
     ) -> WasmResult<()> {
         Ok(())
     }
@@ -272,6 +274,7 @@ pub trait FuncEnvironment {
         &mut self,
         _op: &Operator,
         _builder: &mut FunctionBuilder,
+        _state: &mut TranslationState,
     ) -> WasmResult<()> {
         Ok(())
     }
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index 9d9e923db5..84c76297aa 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -207,9 +207,9 @@ fn parse_function_body(
     while !state.control_stack.is_empty() {
         builder.set_srcloc(cur_srcloc(&reader));
         let op = reader.read_operator()?;
-        environ.before_translate_operator(&op, builder)?;
+        environ.before_translate_operator(&op, builder, state)?;
         translate_operator(&op, builder, state, environ)?;
-        environ.after_translate_operator(&op, builder)?;
+        environ.after_translate_operator(&op, builder, state)?;
     }
 
     // The final `End` operator left us in the exit block where we need to manually add a return
diff --git a/cranelift/wasm/src/lib.rs b/cranelift/wasm/src/lib.rs
index db10a7936c..04ccb90451 100644
--- a/cranelift/wasm/src/lib.rs
+++ b/cranelift/wasm/src/lib.rs
@@ -63,6 +63,7 @@ pub use crate::environ::{
 };
 pub use crate::func_translator::FuncTranslator;
 pub use crate::module_translator::translate_module;
+pub use crate::state::TranslationState;
 pub use crate::translation_utils::{
     get_vmctx_value_label, DefinedFuncIndex, DefinedGlobalIndex, DefinedMemoryIndex,
     DefinedTableIndex, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex,

From d5342bfdfa3d232f941024b07a60aaf680517ae7 Mon Sep 17 00:00:00 2001
From: Luca Barbato 
Date: Tue, 30 Jul 2019 16:12:27 -0400
Subject: [PATCH 2548/3084] Fix the label filter

---
 cranelift/rustc.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/rustc.md b/cranelift/rustc.md
index 90e9fb03cc..1541940817 100644
--- a/cranelift/rustc.md
+++ b/cranelift/rustc.md
@@ -22,7 +22,7 @@ There's plenty of work to do to achieve these goals, and if we achieve
 them, we'll have enabled a Rust compiler written entirely in Rust, and
 enabled faster Rust compile times for important use cases.
 
-See [issues tagged "rustc"](https://github.com/CraneStation/cranelift/labels/FA-rustc)
+See [issues tagged "rustc"](https://github.com/CraneStation/cranelift/labels/goal%3Arustc)
 for a list of some of the things that will be needed.
 
 With all that said, there is a potential goal beyond that, which is to

From 3d42753535bd7202e3b4d7b429f62d6544bb8257 Mon Sep 17 00:00:00 2001
From: iximeow 
Date: Tue, 30 Jul 2019 09:48:06 -0700
Subject: [PATCH 2549/3084] add VisibleTranslationState for a public-friendly
 interface

---
 cranelift/wasm/src/environ/spec.rs    |  6 +++---
 cranelift/wasm/src/func_translator.rs |  6 +++---
 cranelift/wasm/src/lib.rs             |  2 +-
 cranelift/wasm/src/state.rs           | 22 ++++++++++++++++++++++
 4 files changed, 29 insertions(+), 7 deletions(-)

diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 71c11e0b23..c1aa9e8dd7 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -6,7 +6,7 @@
 //!
 //! [Wasmtime]: https://github.com/CraneStation/wasmtime
 
-use crate::state::TranslationState;
+use crate::state::VisibleTranslationState;
 use crate::translation_utils::{
     FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
 };
@@ -263,7 +263,7 @@ pub trait FuncEnvironment {
         &mut self,
         _op: &Operator,
         _builder: &mut FunctionBuilder,
-        _state: &mut TranslationState,
+        _state: &VisibleTranslationState,
     ) -> WasmResult<()> {
         Ok(())
     }
@@ -274,7 +274,7 @@ pub trait FuncEnvironment {
         &mut self,
         _op: &Operator,
         _builder: &mut FunctionBuilder,
-        _state: &mut TranslationState,
+        _state: &VisibleTranslationState,
     ) -> WasmResult<()> {
         Ok(())
     }
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index 84c76297aa..a2ecdd06b3 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -6,7 +6,7 @@
 
 use crate::code_translator::translate_operator;
 use crate::environ::{FuncEnvironment, ReturnMode, WasmError, WasmResult};
-use crate::state::TranslationState;
+use crate::state::{TranslationState, VisibleTranslationState};
 use crate::translation_utils::get_vmctx_value_label;
 use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir::{self, Ebb, InstBuilder, ValueLabel};
@@ -207,9 +207,9 @@ fn parse_function_body(
     while !state.control_stack.is_empty() {
         builder.set_srcloc(cur_srcloc(&reader));
         let op = reader.read_operator()?;
-        environ.before_translate_operator(&op, builder, state)?;
+        environ.before_translate_operator(&op, builder, &VisibleTranslationState::new(state))?;
         translate_operator(&op, builder, state, environ)?;
-        environ.after_translate_operator(&op, builder, state)?;
+        environ.after_translate_operator(&op, builder, &VisibleTranslationState::new(state))?;
     }
 
     // The final `End` operator left us in the exit block where we need to manually add a return
diff --git a/cranelift/wasm/src/lib.rs b/cranelift/wasm/src/lib.rs
index 04ccb90451..c4770a68ea 100644
--- a/cranelift/wasm/src/lib.rs
+++ b/cranelift/wasm/src/lib.rs
@@ -63,7 +63,7 @@ pub use crate::environ::{
 };
 pub use crate::func_translator::FuncTranslator;
 pub use crate::module_translator::translate_module;
-pub use crate::state::TranslationState;
+pub use crate::state::VisibleTranslationState;
 pub use crate::translation_utils::{
     get_vmctx_value_label, DefinedFuncIndex, DefinedGlobalIndex, DefinedMemoryIndex,
     DefinedTableIndex, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex,
diff --git a/cranelift/wasm/src/state.rs b/cranelift/wasm/src/state.rs
index f791c98032..7016035b72 100644
--- a/cranelift/wasm/src/state.rs
+++ b/cranelift/wasm/src/state.rs
@@ -124,6 +124,28 @@ impl ControlStackFrame {
     }
 }
 
+/// VisibleTranslationState wraps a TranslationState with an interface appropriate for users
+/// outside this `cranelift-wasm`.
+///
+/// VisibleTranslationState is currently very minimal (only exposing reachability information), but
+/// is anticipated to grow in the future, with functions to inspect or modify the wasm operand
+/// stack for example.
+pub struct VisibleTranslationState<'a> {
+    state: &'a TranslationState,
+}
+
+impl<'a> VisibleTranslationState<'a> {
+    /// Build a VisibleTranslationState from an existing TranslationState
+    pub fn new(state: &'a TranslationState) -> Self {
+        VisibleTranslationState { state }
+    }
+
+    /// True if the current translation state expresses reachable code, false if it is unreachable
+    pub fn reachable(&self) -> bool {
+        self.state.reachable
+    }
+}
+
 /// Contains information passed along during the translation and that records:
 ///
 /// - The current value and control stacks.

From 6042ee6f2db677bd0b72624e0c5ea5f7219c6e57 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sun, 17 Mar 2019 11:58:40 +0100
Subject: [PATCH 2550/3084] Improve graphviz rendering

---
 cranelift/codegen/src/cfg_printer.rs | 30 ++++++++++++++--------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/cranelift/codegen/src/cfg_printer.rs b/cranelift/codegen/src/cfg_printer.rs
index e3ce37e731..e3c62820cd 100644
--- a/cranelift/codegen/src/cfg_printer.rs
+++ b/cranelift/codegen/src/cfg_printer.rs
@@ -1,10 +1,12 @@
 //! The `CFGPrinter` utility.
 
 use core::fmt::{Display, Formatter, Result, Write};
+use std::vec::Vec;
 
+use crate::entity::SecondaryMap;
 use crate::flowgraph::{BasicBlock, ControlFlowGraph};
-use crate::ir::instructions::BranchInfo;
 use crate::ir::Function;
+use crate::write::{FuncWriter, PlainWriter};
 
 /// A utility for pretty-printing the CFG of a `Function`.
 pub struct CFGPrinter<'a> {
@@ -39,23 +41,21 @@ impl<'a> CFGPrinter<'a> {
     }
 
     fn ebb_nodes(&self, w: &mut dyn Write) -> Result {
+        let mut aliases = SecondaryMap::<_, Vec<_>>::new();
+        for v in self.func.dfg.values() {
+            // VADFS returns the immediate target of an alias
+            if let Some(k) = self.func.dfg.value_alias_dest_for_serialization(v) {
+                aliases[k].push(v);
+            }
+        }
+
         for ebb in &self.func.layout {
-            write!(w, "    {} [shape=record, label=\"{{{}", ebb, ebb)?;
+            write!(w, "    {} [shape=record, label=\"{{", ebb)?;
+            crate::write::write_ebb_header(w, self.func, None, ebb, 4)?;
             // Add all outgoing branch instructions to the label.
             for inst in self.func.layout.ebb_insts(ebb) {
-                let idata = &self.func.dfg[inst];
-                match idata.analyze_branch(&self.func.dfg.value_lists) {
-                    BranchInfo::SingleDest(dest, _) => {
-                        write!(w, " | <{}>{} {}", inst, idata.opcode(), dest)?
-                    }
-                    BranchInfo::Table(table, dest) => {
-                        write!(w, " | <{}>{} {}", inst, idata.opcode(), table)?;
-                        if let Some(dest) = dest {
-                            write!(w, " {}", dest)?
-                        }
-                    }
-                    BranchInfo::NotABranch => {}
-                }
+                write!(w, " | <{}>", inst)?;
+                PlainWriter.write_instruction(w, self.func, &aliases, None, inst, 0)?;
             }
             writeln!(w, "}}\"]")?
         }

From 8e33ca3055b2f33a6728302907081a60b710715a Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 27 Mar 2019 17:29:54 +0100
Subject: [PATCH 2551/3084] Fix tests

---
 cranelift/filetests/filetests/cfg/loop.clif   | 29 ++++++++++++++-----
 .../filetests/filetests/cfg/unused_node.clif  | 14 ++++++---
 2 files changed, 32 insertions(+), 11 deletions(-)

diff --git a/cranelift/filetests/filetests/cfg/loop.clif b/cranelift/filetests/filetests/cfg/loop.clif
index 42e9cb483d..a52ae09986 100644
--- a/cranelift/filetests/filetests/cfg/loop.clif
+++ b/cranelift/filetests/filetests/cfg/loop.clif
@@ -3,18 +3,33 @@ test print-cfg
 test verifier
 
 function %nonsense(i32, i32) -> f32 {
-; check: digraph "%nonsense" {
 ; regex: I=\binst\d+\b
-; check: label="{ebb0 | <$(BRZ=$I)>brz ebb2 | <$(JUMP=$I)>jump ebb3}"]
-
+; check: digraph "%nonsense" {
+; check:     ebb0 [shape=record, label="{ebb0(v1: i32, v2: i32):
+; check: | <$(BRZ=$I)>brz v2, ebb2
+; nextln: | <$(JUMP0=$I)>jump ebb3
+; nextln: }"]
+; nextln:     ebb3 [shape=record, label="{ebb3:
+; check: | <$(JUMP3=$I)>jump ebb1(v4)
+; nextln: }"]
+; nextln:     ebb1 [shape=record, label="{ebb1(v5: i32):
+; check:  | <$(BRNZ1=$I)>brnz v13, ebb1(v12)
+; nextln: | <$(JUMP1=$I)>jump ebb4
+; nextln: }"]
+; nextln:    ebb4 [shape=record, label="{ebb4:
+; check:  | <$I>return v17
+; nextln: }"]
+; nextln:    ebb2 [shape=record, label="{ebb2:
+; check:  | <$I>return v100
+; check:}"]
 ebb0(v1: i32, v2: i32):
     v3 = f64const 0x0.0
     brz v2, ebb2            ; unordered: ebb0:$BRZ -> ebb2
-    jump ebb3               ; unordered: ebb0:$JUMP -> ebb3
+    jump ebb3               ; unordered: ebb0:$JUMP0 -> ebb3
 
 ebb3:
     v4 = iconst.i32 0
-    jump ebb1(v4)           ; unordered: ebb3:inst4 -> ebb1
+    jump ebb1(v4)           ; unordered: ebb3:$JUMP3 -> ebb1
 
 ebb1(v5: i32):
     v6 = imul_imm v5, 4
@@ -25,8 +40,8 @@ ebb1(v5: i32):
     v11 = fadd v9, v10
     v12 = iadd_imm v5, 1
     v13 = icmp ult v12, v2
-    brnz v13, ebb1(v12)     ; unordered: ebb1:inst13 -> ebb1
-    jump ebb4               ; unordered: ebb1:inst14 -> ebb4
+    brnz v13, ebb1(v12)     ; unordered: ebb1:$BRNZ1 -> ebb1
+    jump ebb4               ; unordered: ebb1:$JUMP1 -> ebb4
 
 ebb4:
     v14 = f64const 0.0
diff --git a/cranelift/filetests/filetests/cfg/unused_node.clif b/cranelift/filetests/filetests/cfg/unused_node.clif
index 80f2402c07..1a2dd9fb1d 100644
--- a/cranelift/filetests/filetests/cfg/unused_node.clif
+++ b/cranelift/filetests/filetests/cfg/unused_node.clif
@@ -3,10 +3,16 @@ test print-cfg
 
 function %not_reached(i32) -> i32 {
 ; check: digraph "%not_reached" {
-; check:     ebb0 [shape=record, label="{ebb0 | brnz ebb2}"]
-; check:     ebb1 [shape=record, label="{ebb1 | jump ebb0}"]
-; check:     ebb2 [shape=record, label="{ebb2}"]
-
+; check:     ebb0 [shape=record, label="{ebb0(v0: i32):
+; check:  | brnz v0, ebb2
+; check:  | trap user0
+; check: }"]
+; check:     ebb1 [shape=record, label="{ebb1:
+; check:  | jump ebb0(v2)
+; check: }"]
+; check:     ebb2 [shape=record, label="{ebb2:
+; check:  | return v0
+; check: }"]
 ebb0(v0: i32):
     brnz v0, ebb2       ; unordered: ebb0:inst0 -> ebb2
     trap user0

From dc58a5fc5c7725a4c08023b74e95f6d99b0dadea Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 1 Jul 2019 17:25:11 +0200
Subject: [PATCH 2552/3084] Simple preopt: use the immediate form for
 adjust_sp_down/ifcmp whenever possible;

---
 cranelift/codegen/src/simple_preopt.rs | 138 ++++++++++++++-----------
 1 file changed, 75 insertions(+), 63 deletions(-)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index fabad6d3b7..d5e21d6368 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -8,12 +8,14 @@ use crate::cursor::{Cursor, FuncCursor};
 use crate::divconst_magic_numbers::{magic_s32, magic_s64, magic_u32, magic_u64};
 use crate::divconst_magic_numbers::{MS32, MS64, MU32, MU64};
 use crate::flowgraph::ControlFlowGraph;
-use crate::ir::condcodes::{CondCode, IntCC};
-use crate::ir::dfg::ValueDef;
-use crate::ir::instructions::{Opcode, ValueList};
-use crate::ir::types::{I32, I64};
-use crate::ir::Inst;
-use crate::ir::{DataFlowGraph, Ebb, Function, InstBuilder, InstructionData, Type, Value};
+use crate::ir::{
+    condcodes::{CondCode, IntCC},
+    dfg::ValueDef,
+    immediates,
+    instructions::{Opcode, ValueList},
+    types::{I32, I64},
+    DataFlowGraph, Ebb, Function, Inst, InstBuilder, InstructionData, Type, Value,
+};
 use crate::timing;
 
 //----------------------------------------------------------------------
@@ -450,6 +452,20 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
     }
 }
 
+#[inline]
+fn resolve_imm64_value(dfg: &DataFlowGraph, value: Value) -> Option {
+    if let ValueDef::Result(candidate_inst, _) = dfg.value_def(value) {
+        if let InstructionData::UnaryImm {
+            opcode: Opcode::Iconst,
+            imm,
+        } = dfg[candidate_inst]
+        {
+            return Some(imm);
+        }
+    }
+    None
+}
+
 /// Apply basic simplifications.
 ///
 /// This folds constants with arithmetic to form `_imm` instructions, and other
@@ -457,69 +473,64 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
 fn simplify(pos: &mut FuncCursor, inst: Inst) {
     match pos.func.dfg[inst] {
         InstructionData::Binary { opcode, args } => {
-            if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(args[1]) {
-                if let InstructionData::UnaryImm {
-                    opcode: Opcode::Iconst,
-                    mut imm,
-                } = pos.func.dfg[iconst_inst]
-                {
-                    let new_opcode = match opcode {
-                        Opcode::Iadd => Opcode::IaddImm,
-                        Opcode::Imul => Opcode::ImulImm,
-                        Opcode::Sdiv => Opcode::SdivImm,
-                        Opcode::Udiv => Opcode::UdivImm,
-                        Opcode::Srem => Opcode::SremImm,
-                        Opcode::Urem => Opcode::UremImm,
-                        Opcode::Band => Opcode::BandImm,
-                        Opcode::Bor => Opcode::BorImm,
-                        Opcode::Bxor => Opcode::BxorImm,
-                        Opcode::Rotl => Opcode::RotlImm,
-                        Opcode::Rotr => Opcode::RotrImm,
-                        Opcode::Ishl => Opcode::IshlImm,
-                        Opcode::Ushr => Opcode::UshrImm,
-                        Opcode::Sshr => Opcode::SshrImm,
-                        Opcode::Isub => {
-                            imm = imm.wrapping_neg();
-                            Opcode::IaddImm
-                        }
-                        _ => return,
-                    };
-                    let ty = pos.func.dfg.ctrl_typevar(inst);
-                    pos.func
-                        .dfg
-                        .replace(inst)
-                        .BinaryImm(new_opcode, ty, imm, args[0]);
-                }
-            } else if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(args[0]) {
-                if let InstructionData::UnaryImm {
-                    opcode: Opcode::Iconst,
-                    imm,
-                } = pos.func.dfg[iconst_inst]
-                {
-                    let new_opcode = match opcode {
-                        Opcode::Isub => Opcode::IrsubImm,
-                        _ => return,
-                    };
-                    let ty = pos.func.dfg.ctrl_typevar(inst);
-                    pos.func
-                        .dfg
-                        .replace(inst)
-                        .BinaryImm(new_opcode, ty, imm, args[1]);
-                }
+            if let Some(mut imm) = resolve_imm64_value(&pos.func.dfg, args[1]) {
+                let new_opcode = match opcode {
+                    Opcode::Iadd => Opcode::IaddImm,
+                    Opcode::Imul => Opcode::ImulImm,
+                    Opcode::Sdiv => Opcode::SdivImm,
+                    Opcode::Udiv => Opcode::UdivImm,
+                    Opcode::Srem => Opcode::SremImm,
+                    Opcode::Urem => Opcode::UremImm,
+                    Opcode::Band => Opcode::BandImm,
+                    Opcode::Bor => Opcode::BorImm,
+                    Opcode::Bxor => Opcode::BxorImm,
+                    Opcode::Rotl => Opcode::RotlImm,
+                    Opcode::Rotr => Opcode::RotrImm,
+                    Opcode::Ishl => Opcode::IshlImm,
+                    Opcode::Ushr => Opcode::UshrImm,
+                    Opcode::Sshr => Opcode::SshrImm,
+                    Opcode::Isub => {
+                        imm = imm.wrapping_neg();
+                        Opcode::IaddImm
+                    }
+                    Opcode::Ifcmp => Opcode::IfcmpImm,
+                    _ => return,
+                };
+                let ty = pos.func.dfg.ctrl_typevar(inst);
+                pos.func
+                    .dfg
+                    .replace(inst)
+                    .BinaryImm(new_opcode, ty, imm, args[0]);
+            } else if let Some(imm) = resolve_imm64_value(&pos.func.dfg, args[0]) {
+                let new_opcode = match opcode {
+                    Opcode::Isub => Opcode::IrsubImm,
+                    _ => return,
+                };
+                let ty = pos.func.dfg.ctrl_typevar(inst);
+                pos.func
+                    .dfg
+                    .replace(inst)
+                    .BinaryImm(new_opcode, ty, imm, args[1]);
             }
         }
+
+        InstructionData::Unary { opcode, arg } => match opcode {
+            Opcode::AdjustSpDown => {
+                if let Some(imm) = resolve_imm64_value(&pos.func.dfg, arg) {
+                    // Note this works for both positive and negative immediate values.
+                    pos.func.dfg.replace(inst).adjust_sp_down_imm(imm);
+                }
+            }
+            _ => {}
+        },
+
         InstructionData::IntCompare { opcode, cond, args } => {
             debug_assert_eq!(opcode, Opcode::Icmp);
-            if let ValueDef::Result(iconst_inst, _) = pos.func.dfg.value_def(args[1]) {
-                if let InstructionData::UnaryImm {
-                    opcode: Opcode::Iconst,
-                    imm,
-                } = pos.func.dfg[iconst_inst]
-                {
-                    pos.func.dfg.replace(inst).icmp_imm(cond, args[0], imm);
-                }
+            if let Some(imm) = resolve_imm64_value(&pos.func.dfg, args[1]) {
+                pos.func.dfg.replace(inst).icmp_imm(cond, args[0], imm);
             }
         }
+
         InstructionData::CondTrap { .. }
         | InstructionData::Branch { .. }
         | InstructionData::Ternary {
@@ -542,6 +553,7 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
                 }
             }
         }
+
         _ => {}
     }
 }

From 141b45e0e1ed7cc03785a26bd3c55b17358eb309 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 5 Jul 2019 12:24:35 +0200
Subject: [PATCH 2553/3084] Simple preopt: fold binary opcodes chains together;

---
 cranelift/codegen/src/simple_preopt.rs | 44 ++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index d5e21d6368..de31036477 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -501,6 +501,9 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
                     .dfg
                     .replace(inst)
                     .BinaryImm(new_opcode, ty, imm, args[0]);
+
+                // Repeat for BinaryImm simplification.
+                simplify(pos, inst);
             } else if let Some(imm) = resolve_imm64_value(&pos.func.dfg, args[0]) {
                 let new_opcode = match opcode {
                     Opcode::Isub => Opcode::IrsubImm,
@@ -524,6 +527,47 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
             _ => {}
         },
 
+        InstructionData::BinaryImm { opcode, arg, imm } => match opcode {
+            Opcode::IaddImm
+            | Opcode::ImulImm
+            | Opcode::BorImm
+            | Opcode::BandImm
+            | Opcode::BxorImm => {
+                // Fold binary_op(C2, binary_op(C1, x)) into binary_op(binary_op(C1 | C2), x)
+                if let ValueDef::Result(arg_inst, _) = pos.func.dfg.value_def(arg) {
+                    if let InstructionData::BinaryImm {
+                        opcode: prev_opcode,
+                        arg: prev_arg,
+                        imm: prev_imm,
+                    } = &pos.func.dfg[arg_inst]
+                    {
+                        if opcode == *prev_opcode {
+                            let ty = pos.func.dfg.ctrl_typevar(inst);
+                            if ty == pos.func.dfg.ctrl_typevar(arg_inst) {
+                                let lhs: i64 = imm.into();
+                                let rhs: i64 = (*prev_imm).into();
+                                let new_imm = match opcode {
+                                    Opcode::BorImm => lhs | rhs,
+                                    Opcode::BandImm => lhs & rhs,
+                                    Opcode::BxorImm => lhs ^ rhs,
+                                    Opcode::IaddImm => lhs.wrapping_add(rhs),
+                                    Opcode::ImulImm => lhs.wrapping_mul(rhs),
+                                    _ => panic!("can't happen"),
+                                };
+                                let new_imm = immediates::Imm64::from(new_imm);
+                                let arg = *prev_arg;
+                                pos.func
+                                    .dfg
+                                    .replace(inst)
+                                    .BinaryImm(opcode, ty, new_imm, arg);
+                            }
+                        }
+                    }
+                }
+            }
+            _ => {}
+        },
+
         InstructionData::IntCompare { opcode, cond, args } => {
             debug_assert_eq!(opcode, Opcode::Icmp);
             if let Some(imm) = resolve_imm64_value(&pos.func.dfg, args[1]) {

From 2fef2eef67881ef2c921e4f2387ed1d854d387b2 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 5 Jul 2019 17:21:17 +0200
Subject: [PATCH 2554/3084] Simple preopt: try to fold right-shift of
 left-shift into an extended move;

---
 cranelift/codegen/src/simple_preopt.rs        |  58 ++++-
 .../filetests/simple_preopt/simplify.clif     | 200 ++++++++++++++++++
 2 files changed, 257 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index de31036477..fc1592099a 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -13,7 +13,7 @@ use crate::ir::{
     dfg::ValueDef,
     immediates,
     instructions::{Opcode, ValueList},
-    types::{I32, I64},
+    types::{I16, I32, I64, I8},
     DataFlowGraph, Ebb, Function, Inst, InstBuilder, InstructionData, Type, Value,
 };
 use crate::timing;
@@ -466,6 +466,56 @@ fn resolve_imm64_value(dfg: &DataFlowGraph, value: Value) -> Option> N] into a (un)signed-extending move.
+/// Returns true if the final instruction has been converted to such a move.
+fn try_fold_extended_move(
+    pos: &mut FuncCursor,
+    inst: Inst,
+    opcode: Opcode,
+    arg: Value,
+    imm: immediates::Imm64,
+) -> bool {
+    if let ValueDef::Result(arg_inst, _) = pos.func.dfg.value_def(arg) {
+        if let InstructionData::BinaryImm {
+            opcode: Opcode::IshlImm,
+            arg: prev_arg,
+            imm: prev_imm,
+        } = &pos.func.dfg[arg_inst]
+        {
+            if imm != *prev_imm {
+                return false;
+            }
+
+            let dest_ty = pos.func.dfg.ctrl_typevar(inst);
+            if dest_ty != pos.func.dfg.ctrl_typevar(arg_inst) || !dest_ty.is_int() {
+                return false;
+            }
+
+            let imm_bits: i64 = imm.into();
+            let ireduce_ty = match dest_ty.lane_bits() as i64 - imm_bits {
+                8 => I8,
+                16 => I16,
+                32 => I32,
+                _ => return false,
+            };
+            let ireduce_ty = ireduce_ty.by(dest_ty.lane_count()).unwrap();
+
+            // This becomes a no-op, since ireduce_ty has a smaller lane width than
+            // the argument type (also the destination type).
+            let arg = *prev_arg;
+            let narrower_arg = pos.ins().ireduce(ireduce_ty, arg);
+
+            if opcode == Opcode::UshrImm {
+                pos.func.dfg.replace(inst).uextend(dest_ty, narrower_arg);
+            } else {
+                pos.func.dfg.replace(inst).sextend(dest_ty, narrower_arg);
+            }
+            return true;
+        }
+    }
+    false
+}
+
 /// Apply basic simplifications.
 ///
 /// This folds constants with arithmetic to form `_imm` instructions, and other
@@ -565,6 +615,12 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
                     }
                 }
             }
+
+            Opcode::UshrImm | Opcode::SshrImm => {
+                if try_fold_extended_move(pos, inst, opcode, arg, imm) {
+                    return;
+                }
+            }
             _ => {}
         },
 
diff --git a/cranelift/filetests/filetests/simple_preopt/simplify.clif b/cranelift/filetests/filetests/simple_preopt/simplify.clif
index 1592defe2f..587f64617d 100644
--- a/cranelift/filetests/filetests/simple_preopt/simplify.clif
+++ b/cranelift/filetests/filetests/simple_preopt/simplify.clif
@@ -78,3 +78,203 @@ ebb0(v0: i32):
 ; nextln:     v2 = irsub_imm v0, 2
 ; nextln:     return v2
 ; nextln: }
+
+;; Sign-extensions.
+
+;; 8 -> 16
+function %uextend_8_16() -> i16 {
+ebb0:
+    v0 = iconst.i16 37
+    v1 = ishl_imm v0, 8
+    v2 = ushr_imm v1, 8
+    return v2
+}
+; sameln: function %uextend_8_16
+; nextln: ebb0:
+; nextln:     v0 = iconst.i16 37
+; nextln:     v1 = ishl_imm v0, 8
+; nextln:     v3 = ireduce.i8 v0
+; nextln:     v2 = uextend.i16 v3
+; nextln:     return v2
+; nextln: }
+
+function %sextend_8_16() -> i16 {
+ebb0:
+    v0 = iconst.i16 37
+    v1 = ishl_imm v0, 8
+    v2 = sshr_imm v1, 8
+    return v2
+}
+; sameln: function %sextend_8_16
+; nextln: ebb0:
+; nextln:     v0 = iconst.i16 37
+; nextln:     v1 = ishl_imm v0, 8
+; nextln:     v3 = ireduce.i8 v0
+; nextln:     v2 = sextend.i16 v3
+; nextln:     return v2
+; nextln: }
+
+;; 8 -> 32
+function %uextend_8_32() -> i32 {
+ebb0:
+    v0 = iconst.i32 37
+    v1 = ishl_imm v0, 24
+    v2 = ushr_imm v1, 24
+    return v2
+}
+; sameln: function %uextend_8_32
+; nextln: ebb0:
+; nextln:     v0 = iconst.i32 37
+; nextln:     v1 = ishl_imm v0, 24
+; nextln:     v3 = ireduce.i8 v0
+; nextln:     v2 = uextend.i32 v3
+; nextln:     return v2
+; nextln: }
+
+function %sextend_8_32() -> i32 {
+ebb0:
+    v0 = iconst.i32 37
+    v1 = ishl_imm v0, 24
+    v2 = sshr_imm v1, 24
+    return v2
+}
+; sameln: function %sextend_8_32
+; nextln: ebb0:
+; nextln:     v0 = iconst.i32 37
+; nextln:     v1 = ishl_imm v0, 24
+; nextln:     v3 = ireduce.i8 v0
+; nextln:     v2 = sextend.i32 v3
+; nextln:     return v2
+; nextln: }
+
+;; 16 -> 32
+function %uextend_16_32() -> i32 {
+ebb0:
+    v0 = iconst.i32 37
+    v1 = ishl_imm v0, 16
+    v2 = ushr_imm v1, 16
+    return v2
+}
+; sameln: function %uextend_16_32
+; nextln: ebb0:
+; nextln:     v0 = iconst.i32 37
+; nextln:     v1 = ishl_imm v0, 16
+; nextln:     v3 = ireduce.i16 v0
+; nextln:     v2 = uextend.i32 v3
+; nextln:     return v2
+; nextln: }
+
+function %sextend_16_32() -> i32 {
+ebb0:
+    v0 = iconst.i32 37
+    v1 = ishl_imm v0, 16
+    v2 = sshr_imm v1, 16
+    return v2
+}
+; sameln: function %sextend_16_32
+; nextln: ebb0:
+; nextln:     v0 = iconst.i32 37
+; nextln:     v1 = ishl_imm v0, 16
+; nextln:     v3 = ireduce.i16 v0
+; nextln:     v2 = sextend.i32 v3
+; nextln:     return v2
+; nextln: }
+
+;; 8 -> 64
+function %uextend_8_64() -> i64 {
+ebb0:
+    v0 = iconst.i64 37
+    v1 = ishl_imm v0, 56
+    v2 = ushr_imm v1, 56
+    return v2
+}
+; sameln: function %uextend_8_64
+; nextln: ebb0:
+; nextln:     v0 = iconst.i64 37
+; nextln:     v1 = ishl_imm v0, 56
+; nextln:     v3 = ireduce.i8 v0
+; nextln:     v2 = uextend.i64 v3
+; nextln:     return v2
+; nextln: }
+
+function %sextend_8_64() -> i64 {
+ebb0:
+    v0 = iconst.i64 37
+    v1 = ishl_imm v0, 56
+    v2 = sshr_imm v1, 56
+    return v2
+}
+; sameln: function %sextend_8_64
+; nextln: ebb0:
+; nextln:     v0 = iconst.i64 37
+; nextln:     v1 = ishl_imm v0, 56
+; nextln:     v3 = ireduce.i8 v0
+; nextln:     v2 = sextend.i64 v3
+; nextln:     return v2
+; nextln: }
+
+;; 16 -> 64
+function %uextend_16_64() -> i64 {
+ebb0:
+    v0 = iconst.i64 37
+    v1 = ishl_imm v0, 48
+    v2 = ushr_imm v1, 48
+    return v2
+}
+; sameln: function %uextend_16_64
+; nextln: ebb0:
+; nextln:     v0 = iconst.i64 37
+; nextln:     v1 = ishl_imm v0, 48
+; nextln:     v3 = ireduce.i16 v0
+; nextln:     v2 = uextend.i64 v3
+; nextln:     return v2
+; nextln: }
+
+function %sextend_16_64() -> i64 {
+ebb0:
+    v0 = iconst.i64 37
+    v1 = ishl_imm v0, 48
+    v2 = sshr_imm v1, 48
+    return v2
+}
+; sameln: function %sextend_16_64
+; nextln: ebb0:
+; nextln:     v0 = iconst.i64 37
+; nextln:     v1 = ishl_imm v0, 48
+; nextln:     v3 = ireduce.i16 v0
+; nextln:     v2 = sextend.i64 v3
+; nextln:     return v2
+; nextln: }
+
+;; 32 -> 64
+function %uextend_32_64() -> i64 {
+ebb0:
+    v0 = iconst.i64 37
+    v1 = ishl_imm v0, 32
+    v2 = ushr_imm v1, 32
+    return v2
+}
+; sameln: function %uextend_32_64
+; nextln: ebb0:
+; nextln:     v0 = iconst.i64 37
+; nextln:     v1 = ishl_imm v0, 32
+; nextln:     v3 = ireduce.i32 v0
+; nextln:     v2 = uextend.i64 v3
+; nextln:     return v2
+; nextln: }
+
+function %sextend_32_64() -> i64 {
+ebb0:
+    v0 = iconst.i64 37
+    v1 = ishl_imm v0, 32
+    v2 = sshr_imm v1, 32
+    return v2
+}
+; sameln: function %sextend_32_64
+; nextln: ebb0:
+; nextln:     v0 = iconst.i64 37
+; nextln:     v1 = ishl_imm v0, 32
+; nextln:     v3 = ireduce.i32 v0
+; nextln:     v2 = sextend.i64 v3
+; nextln:     return v2
+; nextln: }

From 23ac723d4c3acef65f3055b0d995034688162906 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 10 Jul 2019 19:50:41 +0200
Subject: [PATCH 2555/3084] Simple preopt: fold instructions using simple
 algebraic identities;

---
 cranelift/codegen/src/simple_preopt.rs        | 119 ++++++++++++------
 .../div_by_const_power_of_2.clif              |  16 +--
 2 files changed, 89 insertions(+), 46 deletions(-)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index fc1592099a..b164ca21f6 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -577,52 +577,95 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
             _ => {}
         },
 
-        InstructionData::BinaryImm { opcode, arg, imm } => match opcode {
-            Opcode::IaddImm
-            | Opcode::ImulImm
-            | Opcode::BorImm
-            | Opcode::BandImm
-            | Opcode::BxorImm => {
-                // Fold binary_op(C2, binary_op(C1, x)) into binary_op(binary_op(C1 | C2), x)
-                if let ValueDef::Result(arg_inst, _) = pos.func.dfg.value_def(arg) {
-                    if let InstructionData::BinaryImm {
-                        opcode: prev_opcode,
-                        arg: prev_arg,
-                        imm: prev_imm,
-                    } = &pos.func.dfg[arg_inst]
-                    {
-                        if opcode == *prev_opcode {
-                            let ty = pos.func.dfg.ctrl_typevar(inst);
-                            if ty == pos.func.dfg.ctrl_typevar(arg_inst) {
-                                let lhs: i64 = imm.into();
-                                let rhs: i64 = (*prev_imm).into();
-                                let new_imm = match opcode {
-                                    Opcode::BorImm => lhs | rhs,
-                                    Opcode::BandImm => lhs & rhs,
-                                    Opcode::BxorImm => lhs ^ rhs,
-                                    Opcode::IaddImm => lhs.wrapping_add(rhs),
-                                    Opcode::ImulImm => lhs.wrapping_mul(rhs),
-                                    _ => panic!("can't happen"),
-                                };
-                                let new_imm = immediates::Imm64::from(new_imm);
-                                let arg = *prev_arg;
-                                pos.func
-                                    .dfg
-                                    .replace(inst)
-                                    .BinaryImm(opcode, ty, new_imm, arg);
+        InstructionData::BinaryImm { opcode, arg, imm } => {
+            let ty = pos.func.dfg.ctrl_typevar(inst);
+
+            let mut imm = imm;
+            match opcode {
+                Opcode::IaddImm
+                | Opcode::ImulImm
+                | Opcode::BorImm
+                | Opcode::BandImm
+                | Opcode::BxorImm => {
+                    // Fold binary_op(C2, binary_op(C1, x)) into binary_op(binary_op(C1, C2), x)
+                    if let ValueDef::Result(arg_inst, _) = pos.func.dfg.value_def(arg) {
+                        if let InstructionData::BinaryImm {
+                            opcode: prev_opcode,
+                            arg: prev_arg,
+                            imm: prev_imm,
+                        } = &pos.func.dfg[arg_inst]
+                        {
+                            if opcode == *prev_opcode {
+                                if ty == pos.func.dfg.ctrl_typevar(arg_inst) {
+                                    let lhs: i64 = imm.into();
+                                    let rhs: i64 = (*prev_imm).into();
+                                    let new_imm = match opcode {
+                                        Opcode::BorImm => lhs | rhs,
+                                        Opcode::BandImm => lhs & rhs,
+                                        Opcode::BxorImm => lhs ^ rhs,
+                                        Opcode::IaddImm => lhs.wrapping_add(rhs),
+                                        Opcode::ImulImm => lhs.wrapping_mul(rhs),
+                                        _ => panic!("can't happen"),
+                                    };
+                                    let new_imm = immediates::Imm64::from(new_imm);
+                                    let arg = *prev_arg;
+                                    pos.func
+                                        .dfg
+                                        .replace(inst)
+                                        .BinaryImm(opcode, ty, new_imm, arg);
+                                    imm = new_imm;
+                                }
                             }
                         }
                     }
                 }
-            }
 
-            Opcode::UshrImm | Opcode::SshrImm => {
-                if try_fold_extended_move(pos, inst, opcode, arg, imm) {
+                Opcode::UshrImm | Opcode::SshrImm => {
+                    if try_fold_extended_move(pos, inst, opcode, arg, imm) {
+                        return;
+                    }
+                }
+
+                _ => {}
+            };
+
+            // Replace operations that are no-ops.
+            match (opcode, imm.into()) {
+                (Opcode::IaddImm, 0)
+                | (Opcode::ImulImm, 1)
+                | (Opcode::SdivImm, 1)
+                | (Opcode::UdivImm, 1)
+                | (Opcode::BorImm, 0)
+                | (Opcode::BandImm, -1)
+                | (Opcode::BxorImm, 0)
+                | (Opcode::RotlImm, 0)
+                | (Opcode::RotrImm, 0)
+                | (Opcode::IshlImm, 0)
+                | (Opcode::UshrImm, 0)
+                | (Opcode::SshrImm, 0) => {
+                    // Alias the result value with the original argument.
+                    let results = pos.func.dfg.detach_results(inst);
+                    debug_assert!(results.len(&pos.func.dfg.value_lists) == 1);
+                    let first_result = results.get(0, &pos.func.dfg.value_lists).unwrap();
+                    pos.func.dfg.change_to_alias(first_result, arg);
+
+                    // Replace instruction by a nop.
+                    pos.func.dfg.replace(inst).nop();
                     return;
                 }
+                (Opcode::ImulImm, 0) | (Opcode::BandImm, 0) => {
+                    // Replace by zero.
+                    pos.func.dfg.replace(inst).iconst(ty, 0);
+                    return;
+                }
+                (Opcode::BorImm, -1) => {
+                    // Replace by minus one.
+                    pos.func.dfg.replace(inst).iconst(ty, -1);
+                    return;
+                }
+                _ => {}
             }
-            _ => {}
-        },
+        }
 
         InstructionData::IntCompare { opcode, cond, args } => {
             debug_assert_eq!(opcode, Opcode::Icmp);
diff --git a/cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif b/cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif
index 5a959750b9..7e6bf7fd14 100644
--- a/cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif
+++ b/cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif
@@ -11,11 +11,11 @@ ebb0(v0: i32):
     return v1
 }
 
-; converted to a copy
+; converted to a nop
 function %t_udiv32_p1(i32) -> i32 {
 ebb0(v0: i32):
     v1 = udiv_imm v0, 1
-    ; check: copy v0
+    ; check: nop
     return v1
 }
 
@@ -46,11 +46,11 @@ ebb0(v0: i64):
     return v1
 }
 
-; converted to a copy
+; converted to a nop
 function %t_udiv64_p1(i64) -> i64 {
 ebb0(v0: i64):
     v1 = udiv_imm v0, 1
-    ; check: copy v0
+    ; check: nop
     return v1
 }
 
@@ -81,11 +81,11 @@ ebb0(v0: i32):
     return v1
 }
 
-; converted to a copy
+; converted to a nop
 function %t_sdiv32_p1(i32) -> i32 {
 ebb0(v0: i32):
     v1 = sdiv_imm v0, 1
-    ; check: copy v0
+    ; check: nop
     return v1
 }
 
@@ -192,11 +192,11 @@ ebb0(v0: i64):
     return v1
 }
 
-; converted to a copy
+; converted to a nop
 function %t_sdiv64_p1(i64) -> i64 {
 ebb0(v0: i64):
     v1 = sdiv_imm v0, 1
-    ; check: copy v0
+    ; check: nop
     return v1
 }
 

From 057d4f6e3cfa4b1bfbca69da024d96c62486647c Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 31 Jul 2019 12:43:01 +0200
Subject: [PATCH 2556/3084] Simple preopt: Fold (imm OP x) into (OP_IMM x imm)
 whenever possible;

---
 cranelift/codegen/src/simple_preopt.rs | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index b164ca21f6..9da90d2e4e 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -556,6 +556,11 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
                 simplify(pos, inst);
             } else if let Some(imm) = resolve_imm64_value(&pos.func.dfg, args[0]) {
                 let new_opcode = match opcode {
+                    Opcode::Iadd => Opcode::IaddImm,
+                    Opcode::Imul => Opcode::ImulImm,
+                    Opcode::Band => Opcode::BandImm,
+                    Opcode::Bor => Opcode::BorImm,
+                    Opcode::Bxor => Opcode::BxorImm,
                     Opcode::Isub => Opcode::IrsubImm,
                     _ => return,
                 };

From 6e57e3f8f364fc2aefbbe3aa5cfb0a5336503416 Mon Sep 17 00:00:00 2001
From: iximeow 
Date: Wed, 31 Jul 2019 19:06:31 -0700
Subject: [PATCH 2557/3084] preopt: use replaced arg after having replaced
 BinaryImm

when replacing BinaryImm, we use the prior arg, but later use the arg
that was replaced when writing an alias if we can determine the new op
is actually equivalent to a simple copy
---
 cranelift/codegen/src/simple_preopt.rs              |  6 ++++--
 .../filetests/filetests/simple_preopt/simplify.clif | 13 +++++++++++++
 2 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index 9da90d2e4e..73c6a56148 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -585,6 +585,7 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
         InstructionData::BinaryImm { opcode, arg, imm } => {
             let ty = pos.func.dfg.ctrl_typevar(inst);
 
+            let mut arg = arg;
             let mut imm = imm;
             match opcode {
                 Opcode::IaddImm
@@ -613,12 +614,13 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
                                         _ => panic!("can't happen"),
                                     };
                                     let new_imm = immediates::Imm64::from(new_imm);
-                                    let arg = *prev_arg;
+                                    let new_arg = *prev_arg;
                                     pos.func
                                         .dfg
                                         .replace(inst)
-                                        .BinaryImm(opcode, ty, new_imm, arg);
+                                        .BinaryImm(opcode, ty, new_imm, new_arg);
                                     imm = new_imm;
+                                    arg = new_arg;
                                 }
                             }
                         }
diff --git a/cranelift/filetests/filetests/simple_preopt/simplify.clif b/cranelift/filetests/filetests/simple_preopt/simplify.clif
index 587f64617d..e74be4738a 100644
--- a/cranelift/filetests/filetests/simple_preopt/simplify.clif
+++ b/cranelift/filetests/filetests/simple_preopt/simplify.clif
@@ -278,3 +278,16 @@ ebb0:
 ; nextln:     v2 = sextend.i64 v3
 ; nextln:     return v2
 ; nextln: }
+
+function %add_imm_fold(i32) -> i32 {
+ebb0(v0: i32):
+  v1 = iadd_imm v0, 42
+  v2 = iadd_imm v1, -42
+  return v2
+}
+; sameln: function %add_imm_fold(i32)
+; nextln: ebb0(v0: i32):
+; nextln:    v2 -> v0
+; nextln:    v1 = iadd_imm v0, 42
+; nextln:    nop
+; nextln:    return v2

From f0d7438728fd176de0ebf72c4e793334d627e10c Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Mon, 29 Jul 2019 14:11:40 +0200
Subject: [PATCH 2558/3084] Properly legalize with empty jump tables.

---
 cranelift/codegen/src/legalizer/mod.rs          |  9 ++++++---
 .../filetests/legalizer/empty_br_table.clif     | 17 +++++++++++++++++
 2 files changed, 23 insertions(+), 3 deletions(-)
 create mode 100644 cranelift/filetests/filetests/legalizer/empty_br_table.clif

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 0905fb05c5..5a87492964 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -295,9 +295,12 @@ fn expand_br_table_conds(
 
     // This is a poor man's jump table using just a sequence of conditional branches.
     let table_size = func.jump_tables[table].len();
-    let mut cond_failed_ebb = std::vec::Vec::with_capacity(table_size - 1);
-    for _ in 0..table_size - 1 {
-        cond_failed_ebb.push(func.dfg.make_ebb());
+    let mut cond_failed_ebb = vec![];
+    if table_size >= 1 {
+        cond_failed_ebb = std::vec::Vec::with_capacity(table_size - 1);
+        for _ in 0..table_size - 1 {
+            cond_failed_ebb.push(func.dfg.make_ebb());
+        }
     }
 
     let mut pos = FuncCursor::new(func).at_inst(inst);
diff --git a/cranelift/filetests/filetests/legalizer/empty_br_table.clif b/cranelift/filetests/filetests/legalizer/empty_br_table.clif
new file mode 100644
index 0000000000..6dfceb5a6b
--- /dev/null
+++ b/cranelift/filetests/filetests/legalizer/empty_br_table.clif
@@ -0,0 +1,17 @@
+test legalizer
+set probestack_enabled=false
+set jump_tables_enabled=false
+target x86_64
+
+function u0:0(i64) {
+    jt0 = jump_table []
+
+ebb0(v0: i64):
+    br_table v0, ebb1, jt0
+; check:  ebb0(v0: i64):
+; nextln:   jump ebb1
+
+ebb1:
+    return
+}
+; not: jump_table

From 627ba24b59c0ea5059fffd8566833a8f79857811 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 1 Aug 2019 17:05:38 +0200
Subject: [PATCH 2559/3084] Simplify jump table instructions and add missing
 conversion;

This makes non-legalized jump table instructions operate on operands with
pointer-sized types. This means we need to extend smaller types into the
pointer-sized operand, when the two don't match.
---
 .../codegen/meta/src/isa/x86/encodings.rs     |  7 ++-----
 .../codegen/meta/src/shared/instructions.rs   | 21 ++++++++-----------
 cranelift/codegen/src/legalizer/mod.rs        | 19 ++++++++++-------
 .../filetests/isa/x86/legalize-br-table.clif  |  3 ++-
 .../filetests/licm/jump-table-entry.clif      |  4 ++--
 5 files changed, 27 insertions(+), 27 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 788c0c3ec0..5a049eeaab 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1224,13 +1224,10 @@ pub fn define(
 
     // Jump tables.
     e.enc64(
-        jump_table_entry.bind(I64).bind_any().bind_any(),
+        jump_table_entry.bind(I64),
         rec_jt_entry.opcodes(vec![0x63]).rex().w(),
     );
-    e.enc32(
-        jump_table_entry.bind(I32).bind_any().bind_any(),
-        rec_jt_entry.opcodes(vec![0x8b]),
-    );
+    e.enc32(jump_table_entry.bind(I32), rec_jt_entry.opcodes(vec![0x8b]));
 
     e.enc64(
         jump_table_base.bind(I64),
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 78d017f1a8..36a1a89fb5 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -269,15 +269,9 @@ pub fn define(
         .is_branch(true),
     );
 
+    // The index into the br_table can be any type; legalizer will convert it to the right type.
     let x = &operand_doc("x", iB, "index into jump table");
-
-    let Entry = &TypeVar::new(
-        "Entry",
-        "A scalar integer type",
-        TypeSetBuilder::new().ints(Interval::All).build(),
-    );
-
-    let entry = &operand_doc("entry", Entry, "entry of jump table");
+    let entry = &operand_doc("entry", iAddr, "entry of jump table");
     let JT = &operand("JT", jump_table);
 
     ig.push(
@@ -305,6 +299,9 @@ pub fn define(
         .is_branch(true),
     );
 
+    // These are the instructions which br_table legalizes to: they perform address computations,
+    // using pointer-sized integers, so their type variables are more constrained.
+    let x = &operand_doc("x", iAddr, "index into jump table");
     let Size = &operand_doc("Size", uimm8, "Size in bytes");
 
     ig.push(
@@ -2644,11 +2641,11 @@ pub fn define(
         Cast the bits in `x` as a different type of the same bit width.
 
         This instruction does not change the data's representation but allows
-        data in registers to be used as different types, e.g. an i32x4 as a 
-        b8x16. The only constraint on the result `a` is that it can be 
+        data in registers to be used as different types, e.g. an i32x4 as a
+        b8x16. The only constraint on the result `a` is that it can be
         `raw_bitcast` back to the original type. Also, in a raw_bitcast between
-        vector types with the same number of lanes, the value of each result 
-        lane is a raw_bitcast of the corresponding operand lane. TODO there is 
+        vector types with the same number of lanes, the value of each result
+        lane is a raw_bitcast of the corresponding operand lane. TODO there is
         currently no mechanism for enforcing the bit width constraint.
         "#,
         )
diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 5a87492964..e6f7bcb00d 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -240,29 +240,34 @@ fn expand_br_table_jt(
     //     $addr = iadd $base, $rel_addr
     //     indirect_jump_table_br $addr, $jt
 
-    let table_size = func.jump_tables[table].len();
-    let addr_ty = isa.pointer_type();
-    let entry_ty = I32;
-
     let ebb = func.layout.pp_ebb(inst);
     let jump_table_ebb = func.dfg.make_ebb();
 
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
 
-    // Bounds check
+    // Bounds check.
+    let table_size = pos.func.jump_tables[table].len() as i64;
     let oob = pos
         .ins()
-        .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, arg, table_size as i64);
+        .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, arg, table_size);
 
     pos.ins().brnz(oob, default_ebb, &[]);
     pos.ins().jump(jump_table_ebb, &[]);
     pos.insert_ebb(jump_table_ebb);
 
+    let addr_ty = isa.pointer_type();
+
+    let arg = if pos.func.dfg.value_type(arg) == addr_ty {
+        arg
+    } else {
+        pos.ins().uextend(addr_ty, arg)
+    };
+
     let base_addr = pos.ins().jump_table_base(addr_ty, table);
     let entry = pos
         .ins()
-        .jump_table_entry(addr_ty, arg, base_addr, entry_ty.bytes() as u8, table);
+        .jump_table_entry(arg, base_addr, I32.bytes() as u8, table);
 
     let addr = pos.ins().iadd(base_addr, entry);
     pos.ins().indirect_jump_table_br(addr, table);
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
index fb0aec0606..6d49a6e6d0 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
@@ -16,8 +16,9 @@ ebb0(v0: i64):
 ; nextln:    brif uge $oob, ebb2
 ; nextln:    fallthrough $(inb=$EBB)
 ; check:   $inb:
+; nextln:    $(final_idx=$V) = uextend.i64 $idx
 ; nextln:    $(base=$V) = jump_table_base.i64 jt0
-; nextln:    $(rel_addr=$V) = jump_table_entry.i64 $idx, $base, 4, jt0
+; nextln:    $(rel_addr=$V) = jump_table_entry $final_idx, $base, 4, jt0
 ; nextln:    $(addr=$V) = iadd $base, $rel_addr
 ; nextln:    indirect_jump_table_br $addr, jt0
 
diff --git a/cranelift/filetests/filetests/licm/jump-table-entry.clif b/cranelift/filetests/filetests/licm/jump-table-entry.clif
index 1d41af25db..cbf51cd080 100644
--- a/cranelift/filetests/filetests/licm/jump-table-entry.clif
+++ b/cranelift/filetests/filetests/licm/jump-table-entry.clif
@@ -14,7 +14,7 @@ ebb1: ; the loop!
     fallthrough ebb2
 
 ebb2:
-    v1 = iconst.i32 -14
+    v1 = iconst.i64 -14
     v8 = ifcmp_imm v1, 2
     brif uge v8, ebb1
     jump ebb3
@@ -25,7 +25,7 @@ ebb3:
     v7 = iadd v5, v6
     indirect_jump_table_br v7, jt0
 ; check: ebb2:
-; nextln: v8 = ifcmp_imm.i32 v1, 2
+; nextln: v8 = ifcmp_imm.i64 v1, 2
 ; nextln: brif uge v8, ebb1
 ; nextln: jump ebb3
 ; check: ebb3:

From ad7171530457e6daa87b6489980e487417390b27 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Mon, 29 Jul 2019 17:00:17 +0200
Subject: [PATCH 2560/3084] Factor out func.dfg[inst] accesses in binemit.

---
 cranelift/codegen/meta/src/gen_binemit.rs | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_binemit.rs b/cranelift/codegen/meta/src/gen_binemit.rs
index 1950f59e9e..bb75ce5409 100644
--- a/cranelift/codegen/meta/src/gen_binemit.rs
+++ b/cranelift/codegen/meta/src/gen_binemit.rs
@@ -32,7 +32,7 @@ fn gen_recipe(formats: &FormatRegistry, recipe: &EncodingRecipe, fmt: &mut Forma
     let is_regmove = ["RegMove", "RegSpill", "RegFill"].contains(&inst_format.name);
 
     // Unpack the instruction data.
-    fmtln!(fmt, "if let InstructionData::{} {{", inst_format.name);
+    fmtln!(fmt, "if let &InstructionData::{} {{", inst_format.name);
     fmt.indent(|fmt| {
         fmt.line("opcode,");
         for f in &inst_format.imm_fields {
@@ -47,7 +47,7 @@ fn gen_recipe(formats: &FormatRegistry, recipe: &EncodingRecipe, fmt: &mut Forma
         }
         fmt.line("..");
 
-        fmt.outdented_line("} = func.dfg[inst] {");
+        fmt.outdented_line("} = data {");
 
         // Pass recipe arguments in this order: inputs, imm_fields, outputs.
         let mut args = String::new();
@@ -186,7 +186,8 @@ fn gen_isa(formats: &FormatRegistry, isa_name: &str, recipes: &Recipes, fmt: &mu
     fmt.indent(|fmt| {
         fmt.line("let encoding = func.encodings[inst];");
         fmt.line("let bits = encoding.bits();");
-        fmt.line("match func.encodings[inst].recipe() {");
+        fmt.line("let data = &func.dfg[inst];");
+        fmt.line("match encoding.recipe() {");
         fmt.indent(|fmt| {
             for (i, recipe) in recipes.iter() {
                 fmt.comment(format!("Recipe {}", recipe.name));

From 3585ee34b095e9817f5d8e315d99deafe82ecdba Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Mon, 29 Jul 2019 17:04:48 +0200
Subject: [PATCH 2561/3084] Uses divert.apply for all instruction within
 binemit.

---
 cranelift/codegen/meta/src/gen_binemit.rs | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_binemit.rs b/cranelift/codegen/meta/src/gen_binemit.rs
index bb75ce5409..c1bd752bc7 100644
--- a/cranelift/codegen/meta/src/gen_binemit.rs
+++ b/cranelift/codegen/meta/src/gen_binemit.rs
@@ -75,13 +75,9 @@ fn gen_recipe(formats: &FormatRegistry, recipe: &EncodingRecipe, fmt: &mut Forma
             args += &unwrap_values(&recipe.operands_out, "out", "results", fmt);
         }
 
-        // Special handling for regmove instructions. Update the register
-        // diversion tracker
-        match &*inst_format.name {
-            "RegMove" => fmt.line("divert.regmove(arg, src, dst);"),
-            "RegSpill" => fmt.line("divert.regspill(arg, src, dst);"),
-            "RegFill" => fmt.line("divert.regfill(arg, src, dst);"),
-            _ => {}
+        // Optimization: Only update the register diversion tracker for regmove instructions.
+        if is_regmove {
+            fmt.line("divert.apply(data);")
         }
 
         match &recipe.emit {

From c903735ea8ad9a13bf2b91eb0e5bf2e34eb48d96 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Thu, 1 Aug 2019 15:08:06 +0200
Subject: [PATCH 2562/3084] Shrink: Factor accesses of instruction data

---
 cranelift/codegen/src/binemit/shrink.rs | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/src/binemit/shrink.rs b/cranelift/codegen/src/binemit/shrink.rs
index 281a93ae31..0f9838c8ab 100644
--- a/cranelift/codegen/src/binemit/shrink.rs
+++ b/cranelift/codegen/src/binemit/shrink.rs
@@ -33,11 +33,12 @@ pub fn shrink_instructions(func: &mut Function, isa: &dyn TargetIsa) {
                 //
                 // TODO: Eventually, we want the register allocator to avoid leaving these special
                 // instructions behind, but for now, just temporarily avoid trying to shrink them.
-                match func.dfg[inst] {
+                let inst_data = &func.dfg[inst];
+                match inst_data {
                     InstructionData::RegMove { .. }
                     | InstructionData::RegFill { .. }
                     | InstructionData::RegSpill { .. } => {
-                        divert.apply(&func.dfg[inst]);
+                        divert.apply(inst_data);
                         continue;
                     }
                     _ => (),

From bc1b56f739ed5906f0c597edfdf8e0fd01b72e18 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 2 Aug 2019 17:52:39 +0200
Subject: [PATCH 2563/3084] Rename data to inst_data in binemit generated code.

---
 cranelift/codegen/meta/src/gen_binemit.rs | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_binemit.rs b/cranelift/codegen/meta/src/gen_binemit.rs
index c1bd752bc7..71684ffb55 100644
--- a/cranelift/codegen/meta/src/gen_binemit.rs
+++ b/cranelift/codegen/meta/src/gen_binemit.rs
@@ -47,7 +47,7 @@ fn gen_recipe(formats: &FormatRegistry, recipe: &EncodingRecipe, fmt: &mut Forma
         }
         fmt.line("..");
 
-        fmt.outdented_line("} = data {");
+        fmt.outdented_line("} = inst_data {");
 
         // Pass recipe arguments in this order: inputs, imm_fields, outputs.
         let mut args = String::new();
@@ -77,7 +77,7 @@ fn gen_recipe(formats: &FormatRegistry, recipe: &EncodingRecipe, fmt: &mut Forma
 
         // Optimization: Only update the register diversion tracker for regmove instructions.
         if is_regmove {
-            fmt.line("divert.apply(data);")
+            fmt.line("divert.apply(inst_data);")
         }
 
         match &recipe.emit {
@@ -182,7 +182,7 @@ fn gen_isa(formats: &FormatRegistry, isa_name: &str, recipes: &Recipes, fmt: &mu
     fmt.indent(|fmt| {
         fmt.line("let encoding = func.encodings[inst];");
         fmt.line("let bits = encoding.bits();");
-        fmt.line("let data = &func.dfg[inst];");
+        fmt.line("let inst_data = &func.dfg[inst];");
         fmt.line("match encoding.recipe() {");
         fmt.indent(|fmt| {
             for (i, recipe) in recipes.iter() {

From 383ce584ae5a5c76838161a6298a49f333543762 Mon Sep 17 00:00:00 2001
From: David Lattimore 
Date: Sat, 3 Aug 2019 21:50:31 +1000
Subject: [PATCH 2564/3084] Fix an assertion that wasn't doing what it said

---
 cranelift/codegen/meta/src/cdsl/recipes.rs      |  2 +-
 cranelift/codegen/meta/src/isa/x86/encodings.rs | 13 +++++++------
 2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/recipes.rs b/cranelift/codegen/meta/src/cdsl/recipes.rs
index 84510cf0d1..8157e08864 100644
--- a/cranelift/codegen/meta/src/cdsl/recipes.rs
+++ b/cranelift/codegen/meta/src/cdsl/recipes.rs
@@ -102,7 +102,7 @@ impl Into for Stack {
 /// The `branch_range` argument must be provided for recipes that can encode
 /// branch instructions. It is an `(origin, bits)` tuple describing the exact
 /// range that can be encoded in a branch instruction.
-#[derive(Clone, Hash)]
+#[derive(Clone)]
 pub struct EncodingRecipe {
     /// Short mnemonic name for this recipe.
     pub name: String,
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 5a049eeaab..7ddde02bec 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -21,7 +21,7 @@ pub struct PerCpuModeEncodings {
     pub enc32: Vec,
     pub enc64: Vec,
     pub recipes: Recipes,
-    recipes_inverse: HashMap,
+    recipes_by_name: HashMap,
     pub inst_pred_reg: InstructionPredicateRegistry,
 }
 
@@ -31,15 +31,15 @@ impl PerCpuModeEncodings {
             enc32: Vec::new(),
             enc64: Vec::new(),
             recipes: Recipes::new(),
-            recipes_inverse: HashMap::new(),
+            recipes_by_name: HashMap::new(),
             inst_pred_reg: InstructionPredicateRegistry::new(),
         }
     }
 
     fn add_recipe(&mut self, recipe: EncodingRecipe) -> EncodingRecipeNumber {
-        if let Some(found_index) = self.recipes_inverse.get(&recipe) {
+        if let Some(found_index) = self.recipes_by_name.get(&recipe.name) {
             assert!(
-                self.recipes[*found_index].name == recipe.name,
+                self.recipes[*found_index] == recipe,
                 format!(
                     "trying to insert different recipes with a same name ({})",
                     recipe.name
@@ -47,8 +47,9 @@ impl PerCpuModeEncodings {
             );
             *found_index
         } else {
-            let index = self.recipes.push(recipe.clone());
-            self.recipes_inverse.insert(recipe, index);
+            let recipe_name = recipe.name.clone();
+            let index = self.recipes.push(recipe);
+            self.recipes_by_name.insert(recipe_name, index);
             index
         }
     }

From 29b32b30627edb3d7a543cbc4cdf00e11af9fbea Mon Sep 17 00:00:00 2001
From: Yury Delendik 
Date: Mon, 5 Aug 2019 08:31:00 -0500
Subject: [PATCH 2565/3084] Serialize ValueLabel and StackSlots (#888)

---
 cranelift/codegen/src/ir/entities.rs  | 1 +
 cranelift/codegen/src/ir/mod.rs       | 4 ++++
 cranelift/codegen/src/ir/stackslot.rs | 6 ++++++
 cranelift/codegen/src/ir/valueloc.rs  | 4 ++++
 cranelift/codegen/src/value_label.rs  | 4 ++++
 5 files changed, 19 insertions(+)

diff --git a/cranelift/codegen/src/ir/entities.rs b/cranelift/codegen/src/ir/entities.rs
index a2a405bc46..67fd9157e9 100644
--- a/cranelift/codegen/src/ir/entities.rs
+++ b/cranelift/codegen/src/ir/entities.rs
@@ -69,6 +69,7 @@ entity_impl!(Inst, "inst");
 
 /// An opaque reference to a stack slot.
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct StackSlot(u32);
 entity_impl!(StackSlot, "ss");
 
diff --git a/cranelift/codegen/src/ir/mod.rs b/cranelift/codegen/src/ir/mod.rs
index 0873561cf9..caa1da831a 100644
--- a/cranelift/codegen/src/ir/mod.rs
+++ b/cranelift/codegen/src/ir/mod.rs
@@ -23,6 +23,9 @@ mod trapcode;
 pub mod types;
 mod valueloc;
 
+#[cfg(feature = "enable-serde")]
+use serde::{Deserialize, Serialize};
+
 pub use crate::ir::builder::{InsertBuilder, InstBuilder, InstBuilderBase, InstInserterBase};
 pub use crate::ir::dfg::{DataFlowGraph, ValueDef};
 pub use crate::ir::entities::{
@@ -74,6 +77,7 @@ pub type SourceLocs = SecondaryMap;
 
 /// Marked with a label value.
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct ValueLabel(u32);
 entity_impl!(ValueLabel, "val");
 
diff --git a/cranelift/codegen/src/ir/stackslot.rs b/cranelift/codegen/src/ir/stackslot.rs
index 4c94a6b012..0101b54f2e 100644
--- a/cranelift/codegen/src/ir/stackslot.rs
+++ b/cranelift/codegen/src/ir/stackslot.rs
@@ -13,6 +13,9 @@ use core::slice;
 use core::str::FromStr;
 use std::vec::Vec;
 
+#[cfg(feature = "enable-serde")]
+use serde::{Deserialize, Serialize};
+
 /// The size of an object on the stack, or the size of a stack frame.
 ///
 /// We don't use `usize` to represent object sizes on the target platform because Cranelift supports
@@ -38,6 +41,7 @@ fn spill_size(ty: Type) -> StackSize {
 
 /// The kind of a stack slot.
 #[derive(Clone, Copy, Debug, PartialEq, Eq)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub enum StackSlotKind {
     /// A spill slot. This is a stack slot created by the register allocator.
     SpillSlot,
@@ -98,6 +102,7 @@ impl fmt::Display for StackSlotKind {
 
 /// Contents of a stack slot.
 #[derive(Clone, Debug)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct StackSlotData {
     /// The kind of stack slot.
     pub kind: StackSlotKind,
@@ -150,6 +155,7 @@ impl fmt::Display for StackSlotData {
 ///
 /// Keep track of all the stack slots used by a function.
 #[derive(Clone, Debug)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct StackSlots {
     /// All allocated stack slots.
     slots: PrimaryMap,
diff --git a/cranelift/codegen/src/ir/valueloc.rs b/cranelift/codegen/src/ir/valueloc.rs
index 0cbcd5d2c3..f3ba44896e 100644
--- a/cranelift/codegen/src/ir/valueloc.rs
+++ b/cranelift/codegen/src/ir/valueloc.rs
@@ -7,8 +7,12 @@ use crate::ir::StackSlot;
 use crate::isa::{RegInfo, RegUnit};
 use core::fmt;
 
+#[cfg(feature = "enable-serde")]
+use serde::{Deserialize, Serialize};
+
 /// Value location.
 #[derive(Copy, Clone, Debug, PartialEq, Eq)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub enum ValueLoc {
     /// This value has not been assigned to a location yet.
     Unassigned,
diff --git a/cranelift/codegen/src/value_label.rs b/cranelift/codegen/src/value_label.rs
index 8dab92b892..c31067cc24 100644
--- a/cranelift/codegen/src/value_label.rs
+++ b/cranelift/codegen/src/value_label.rs
@@ -8,8 +8,12 @@ use std::ops::Bound::*;
 use std::ops::Deref;
 use std::vec::Vec;
 
+#[cfg(feature = "enable-serde")]
+use serde::{Deserialize, Serialize};
+
 /// Value location range.
 #[derive(Debug, Clone, Copy)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct ValueLocRange {
     /// The ValueLoc containing a ValueLabel during this range.
     pub loc: ValueLoc,

From 00b8d019c94d2fedb3a9e1bb8ef3544c36a0d468 Mon Sep 17 00:00:00 2001
From: Yury Delendik 
Date: Tue, 6 Aug 2019 14:41:13 -0500
Subject: [PATCH 2566/3084] Bump version to 0.38.0 (#894)

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    |  8 ++++----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 4c7ada1eae..f5e066734c 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.37.0"
+version = "0.38.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.37.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.37.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.37.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.37.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.37.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.37.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.37.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.37.0" }
-cranelift-module = { path = "cranelift-module", version = "0.37.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.37.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.37.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.37.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.37.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.38.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.38.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.38.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.38.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.38.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.38.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.38.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.38.0" }
+cranelift-module = { path = "cranelift-module", version = "0.38.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.38.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.38.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.38.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.38.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index cfb3bac2a3..f98b9e3563 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.37.0"
+version = "0.38.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.37.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.38.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 33dfa3b88a..e723a23309 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.37.0"
+version = "0.38.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.37.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.37.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.38.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.38.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -27,7 +27,7 @@ serde = { version = "1.0.94", features = ["derive"], optional = true }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.37.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.38.0", default-features = false }
 
 [features]
 default = ["std", "x86", "arm32", "arm64", "riscv"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 80f2364b38..9c90a691ad 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.37.0"
+version = "0.38.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.37.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.38.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 1c34b3f359..dc10dd8631 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.37.0"
+version = "0.38.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index d853b0a4b3..184808f7a1 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.37.0"
+version = "0.38.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.37.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.38.0" }
 faerie = "0.10.0"
 goblin = "0.0.22"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index cc2f374ff9..24a14e584d 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.37.0"
+version = "0.38.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../cranelift-reader", version = "0.37.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.37.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../cranelift-reader", version = "0.38.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.38.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 09937ee9b2..92d166fe51 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.37.0"
+version = "0.38.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 67f5456de7..e4c0429a13 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.37.0"
+version = "0.38.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.37.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.38.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 8a407d521b..560868a05a 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.37.0"
+version = "0.38.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 0bd5376c21..43d87d5055 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.37.0"
+version = "0.38.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.37.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.38.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index dc017d69ee..edd609b509 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.37.0"
+version="0.38.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 7c5c5b6204..255a4a0d50 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.37.0"
+version = "0.38.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0" }
 target-lexicon = "0.4.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index ade3cb9ace..de6ee0eb8c 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.37.0"
+version = "0.38.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.37.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.38.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 5a87f4bc8e..3564132909 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.37.0"
+version = "0.38.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.37.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.37.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.38.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.38.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.37.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.37.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.37.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.38.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.38.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.38.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 08283067e0..f0b0f46ac1 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.37.0"
+version = "0.38.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.37.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.38.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 426a202157..410f815775 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.37.0"
+version = "0.38.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.32.1", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.37.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.37.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.37.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.38.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.38.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 73670aab43ee05fad0036674b9604b8646885f2f Mon Sep 17 00:00:00 2001
From: "Adam C. Foltzer" 
Date: Wed, 7 Aug 2019 13:23:32 -0700
Subject: [PATCH 2567/3084] Return a `WasmResult` from `ModuleEnvironment`
 methods (#886)

* [wasm] return a WasmResult from `declare_table_elements`

This method in particular needs to accommodate failure because any table index other than zero is
currently invalid.

* [wasm] additional failure handling improvements

- Adds `WasmResult<()>` as the return type for most of the `ModuleEnvironment` methods that
previously returned nothing.

- Replaces some panics with `WasmError::Unsupported` now that the methods can return a result.

- Adds a `wasm_unsupported!()` macro for early returns with a formatted unsupported message.
---
 cranelift/wasm/src/code_translator.rs     |  11 ++-
 cranelift/wasm/src/environ/dummy.rs       |  75 +++++++++++----
 cranelift/wasm/src/environ/mod.rs         |   1 +
 cranelift/wasm/src/environ/spec.rs        | 109 ++++++++++++++++------
 cranelift/wasm/src/func_translator.rs     |   5 +-
 cranelift/wasm/src/sections_translator.rs |  81 +++++++++-------
 cranelift/wasm/src/translation_utils.rs   |  13 +--
 7 files changed, 204 insertions(+), 91 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 43d08c6713..05c5de4f55 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -23,12 +23,13 @@
 //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as
 //! argument.
 use super::{hash_map, HashMap};
-use crate::environ::{FuncEnvironment, GlobalVariable, ReturnMode, WasmError, WasmResult};
+use crate::environ::{FuncEnvironment, GlobalVariable, ReturnMode, WasmResult};
 use crate::state::{ControlStackFrame, TranslationState};
 use crate::translation_utils::{
     blocktype_to_type, f32_translation, f64_translation, num_return_values,
 };
 use crate::translation_utils::{FuncIndex, MemoryIndex, SignatureIndex, TableIndex};
+use crate::wasm_unsupported;
 use core::{i32, u32};
 use cranelift_codegen::ir::condcodes::{FloatCC, IntCC};
 use cranelift_codegen::ir::types::*;
@@ -899,10 +900,10 @@ pub fn translate_operator(
         | Operator::I64AtomicRmw8UCmpxchg { .. }
         | Operator::I64AtomicRmw16UCmpxchg { .. }
         | Operator::I64AtomicRmw32UCmpxchg { .. } => {
-            return Err(WasmError::Unsupported("proposed thread operators"));
+            wasm_unsupported!("proposed thread operator {:?}", op);
         }
         Operator::RefNull | Operator::RefIsNull { .. } => {
-            return Err(WasmError::Unsupported("proposed reference-type operators"));
+            wasm_unsupported!("proposed reference-type operator {:?}", op);
         }
         Operator::MemoryInit { .. }
         | Operator::DataDrop { .. }
@@ -915,7 +916,7 @@ pub fn translate_operator(
         | Operator::TableSet { .. }
         | Operator::TableGrow { .. }
         | Operator::TableSize { .. } => {
-            return Err(WasmError::Unsupported("proposed bulk memory operators"));
+            wasm_unsupported!("proposed bulk memory operator {:?}", op);
         }
         Operator::V128Load { .. }
         | Operator::V128Store { .. }
@@ -1059,7 +1060,7 @@ pub fn translate_operator(
         | Operator::F64x2ConvertUI64x2
         | Operator::V8x16Shuffle1
         | Operator::V8x16Shuffle2Imm { .. } => {
-            return Err(WasmError::Unsupported("proposed SIMD operators"));
+            wasm_unsupported!("proposed SIMD operator {:?}", op);
         }
     };
     Ok(())
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index 0404ccb522..fcec4e648c 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -367,8 +367,9 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         self.info.config
     }
 
-    fn declare_signature(&mut self, sig: ir::Signature) {
+    fn declare_signature(&mut self, sig: ir::Signature) -> WasmResult<()> {
         self.info.signatures.push(sig);
+        Ok(())
     }
 
     fn declare_func_import(
@@ -376,7 +377,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         sig_index: SignatureIndex,
         module: &'data str,
         field: &'data str,
-    ) {
+    ) -> WasmResult<()> {
         assert_eq!(
             self.info.functions.len(),
             self.info.imported_funcs.len(),
@@ -386,32 +387,48 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         self.info
             .imported_funcs
             .push((String::from(module), String::from(field)));
+        Ok(())
     }
 
-    fn declare_func_type(&mut self, sig_index: SignatureIndex) {
+    fn declare_func_type(&mut self, sig_index: SignatureIndex) -> WasmResult<()> {
         self.info.functions.push(Exportable::new(sig_index));
+        Ok(())
     }
 
-    fn declare_global(&mut self, global: Global) {
+    fn declare_global(&mut self, global: Global) -> WasmResult<()> {
         self.info.globals.push(Exportable::new(global));
+        Ok(())
     }
 
-    fn declare_global_import(&mut self, global: Global, module: &'data str, field: &'data str) {
+    fn declare_global_import(
+        &mut self,
+        global: Global,
+        module: &'data str,
+        field: &'data str,
+    ) -> WasmResult<()> {
         self.info.globals.push(Exportable::new(global));
         self.info
             .imported_globals
             .push((String::from(module), String::from(field)));
+        Ok(())
     }
 
-    fn declare_table(&mut self, table: Table) {
+    fn declare_table(&mut self, table: Table) -> WasmResult<()> {
         self.info.tables.push(Exportable::new(table));
+        Ok(())
     }
 
-    fn declare_table_import(&mut self, table: Table, module: &'data str, field: &'data str) {
+    fn declare_table_import(
+        &mut self,
+        table: Table,
+        module: &'data str,
+        field: &'data str,
+    ) -> WasmResult<()> {
         self.info.tables.push(Exportable::new(table));
         self.info
             .imported_tables
             .push((String::from(module), String::from(field)));
+        Ok(())
     }
 
     fn declare_table_elements(
@@ -420,19 +437,27 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         _base: Option,
         _offset: usize,
         _elements: Box<[FuncIndex]>,
-    ) {
+    ) -> WasmResult<()> {
         // We do nothing
+        Ok(())
     }
 
-    fn declare_memory(&mut self, memory: Memory) {
+    fn declare_memory(&mut self, memory: Memory) -> WasmResult<()> {
         self.info.memories.push(Exportable::new(memory));
+        Ok(())
     }
 
-    fn declare_memory_import(&mut self, memory: Memory, module: &'data str, field: &'data str) {
+    fn declare_memory_import(
+        &mut self,
+        memory: Memory,
+        module: &'data str,
+        field: &'data str,
+    ) -> WasmResult<()> {
         self.info.memories.push(Exportable::new(memory));
         self.info
             .imported_memories
             .push((String::from(module), String::from(field)));
+        Ok(())
     }
 
     fn declare_data_initialization(
@@ -441,37 +466,55 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         _base: Option,
         _offset: usize,
         _data: &'data [u8],
-    ) {
+    ) -> WasmResult<()> {
         // We do nothing
+        Ok(())
     }
 
-    fn declare_func_export(&mut self, func_index: FuncIndex, name: &'data str) {
+    fn declare_func_export(&mut self, func_index: FuncIndex, name: &'data str) -> WasmResult<()> {
         self.info.functions[func_index]
             .export_names
             .push(String::from(name));
+        Ok(())
     }
 
-    fn declare_table_export(&mut self, table_index: TableIndex, name: &'data str) {
+    fn declare_table_export(
+        &mut self,
+        table_index: TableIndex,
+        name: &'data str,
+    ) -> WasmResult<()> {
         self.info.tables[table_index]
             .export_names
             .push(String::from(name));
+        Ok(())
     }
 
-    fn declare_memory_export(&mut self, memory_index: MemoryIndex, name: &'data str) {
+    fn declare_memory_export(
+        &mut self,
+        memory_index: MemoryIndex,
+        name: &'data str,
+    ) -> WasmResult<()> {
         self.info.memories[memory_index]
             .export_names
             .push(String::from(name));
+        Ok(())
     }
 
-    fn declare_global_export(&mut self, global_index: GlobalIndex, name: &'data str) {
+    fn declare_global_export(
+        &mut self,
+        global_index: GlobalIndex,
+        name: &'data str,
+    ) -> WasmResult<()> {
         self.info.globals[global_index]
             .export_names
             .push(String::from(name));
+        Ok(())
     }
 
-    fn declare_start_func(&mut self, func_index: FuncIndex) {
+    fn declare_start_func(&mut self, func_index: FuncIndex) -> WasmResult<()> {
         debug_assert!(self.info.start_func.is_none());
         self.info.start_func = Some(func_index);
+        Ok(())
     }
 
     fn define_function_body(
diff --git a/cranelift/wasm/src/environ/mod.rs b/cranelift/wasm/src/environ/mod.rs
index 831fa0ef5e..4b7405ea7b 100644
--- a/cranelift/wasm/src/environ/mod.rs
+++ b/cranelift/wasm/src/environ/mod.rs
@@ -1,6 +1,7 @@
 //! Support for configurable wasm translation.
 
 mod dummy;
+#[macro_use]
 mod spec;
 
 pub use crate::environ::dummy::DummyEnvironment;
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index c1aa9e8dd7..c94131fa9f 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -60,7 +60,7 @@ pub enum WasmError {
     ///
     /// Embedding environments may have their own limitations and feature restrictions.
     #[fail(display = "Unsupported feature: {}", _0)]
-    Unsupported(&'static str),
+    Unsupported(std::string::String),
 
     /// An implementation limit was exceeded.
     ///
@@ -76,6 +76,13 @@ pub enum WasmError {
     User(std::string::String),
 }
 
+/// Return an `Err(WasmError::Unsupported(msg))` where `msg` the string built by calling `format!`
+/// on the arguments to this macro.
+#[macro_export]
+macro_rules! wasm_unsupported {
+    ($($arg:tt)*) => { return Err($crate::environ::WasmError::Unsupported(format!($($arg)*))) }
+}
+
 impl From for WasmError {
     /// Convert from a `BinaryReaderError` to a `WasmError`.
     fn from(e: BinaryReaderError) -> Self {
@@ -289,14 +296,18 @@ pub trait ModuleEnvironment<'data> {
 
     /// Provides the number of signatures up front. By default this does nothing, but
     /// implementations can use this to preallocate memory if desired.
-    fn reserve_signatures(&mut self, _num: u32) {}
+    fn reserve_signatures(&mut self, _num: u32) -> WasmResult<()> {
+        Ok(())
+    }
 
     /// Declares a function signature to the environment.
-    fn declare_signature(&mut self, sig: ir::Signature);
+    fn declare_signature(&mut self, sig: ir::Signature) -> WasmResult<()>;
 
     /// Provides the number of imports up front. By default this does nothing, but
     /// implementations can use this to preallocate memory if desired.
-    fn reserve_imports(&mut self, _num: u32) {}
+    fn reserve_imports(&mut self, _num: u32) -> WasmResult<()> {
+        Ok(())
+    }
 
     /// Declares a function import to the environment.
     fn declare_func_import(
@@ -304,73 +315,113 @@ pub trait ModuleEnvironment<'data> {
         sig_index: SignatureIndex,
         module: &'data str,
         field: &'data str,
-    );
+    ) -> WasmResult<()>;
 
     /// Declares a table import to the environment.
-    fn declare_table_import(&mut self, table: Table, module: &'data str, field: &'data str);
+    fn declare_table_import(
+        &mut self,
+        table: Table,
+        module: &'data str,
+        field: &'data str,
+    ) -> WasmResult<()>;
 
     /// Declares a memory import to the environment.
-    fn declare_memory_import(&mut self, memory: Memory, module: &'data str, field: &'data str);
+    fn declare_memory_import(
+        &mut self,
+        memory: Memory,
+        module: &'data str,
+        field: &'data str,
+    ) -> WasmResult<()>;
 
     /// Declares a global import to the environment.
-    fn declare_global_import(&mut self, global: Global, module: &'data str, field: &'data str);
+    fn declare_global_import(
+        &mut self,
+        global: Global,
+        module: &'data str,
+        field: &'data str,
+    ) -> WasmResult<()>;
 
     /// Notifies the implementation that all imports have been declared.
-    fn finish_imports(&mut self) {}
+    fn finish_imports(&mut self) -> WasmResult<()> {
+        Ok(())
+    }
 
     /// Provides the number of defined functions up front. By default this does nothing, but
     /// implementations can use this to preallocate memory if desired.
-    fn reserve_func_types(&mut self, _num: u32) {}
+    fn reserve_func_types(&mut self, _num: u32) -> WasmResult<()> {
+        Ok(())
+    }
 
     /// Declares the type (signature) of a local function in the module.
-    fn declare_func_type(&mut self, sig_index: SignatureIndex);
+    fn declare_func_type(&mut self, sig_index: SignatureIndex) -> WasmResult<()>;
 
     /// Provides the number of defined tables up front. By default this does nothing, but
     /// implementations can use this to preallocate memory if desired.
-    fn reserve_tables(&mut self, _num: u32) {}
+    fn reserve_tables(&mut self, _num: u32) -> WasmResult<()> {
+        Ok(())
+    }
 
     /// Declares a table to the environment.
-    fn declare_table(&mut self, table: Table);
+    fn declare_table(&mut self, table: Table) -> WasmResult<()>;
 
     /// Provides the number of defined memories up front. By default this does nothing, but
     /// implementations can use this to preallocate memory if desired.
-    fn reserve_memories(&mut self, _num: u32) {}
+    fn reserve_memories(&mut self, _num: u32) -> WasmResult<()> {
+        Ok(())
+    }
 
     /// Declares a memory to the environment
-    fn declare_memory(&mut self, memory: Memory);
+    fn declare_memory(&mut self, memory: Memory) -> WasmResult<()>;
 
     /// Provides the number of defined globals up front. By default this does nothing, but
     /// implementations can use this to preallocate memory if desired.
-    fn reserve_globals(&mut self, _num: u32) {}
+    fn reserve_globals(&mut self, _num: u32) -> WasmResult<()> {
+        Ok(())
+    }
 
     /// Declares a global to the environment.
-    fn declare_global(&mut self, global: Global);
+    fn declare_global(&mut self, global: Global) -> WasmResult<()>;
 
     /// Provides the number of exports up front. By default this does nothing, but
     /// implementations can use this to preallocate memory if desired.
-    fn reserve_exports(&mut self, _num: u32) {}
+    fn reserve_exports(&mut self, _num: u32) -> WasmResult<()> {
+        Ok(())
+    }
 
     /// Declares a function export to the environment.
-    fn declare_func_export(&mut self, func_index: FuncIndex, name: &'data str);
+    fn declare_func_export(&mut self, func_index: FuncIndex, name: &'data str) -> WasmResult<()>;
 
     /// Declares a table export to the environment.
-    fn declare_table_export(&mut self, table_index: TableIndex, name: &'data str);
+    fn declare_table_export(&mut self, table_index: TableIndex, name: &'data str)
+        -> WasmResult<()>;
 
     /// Declares a memory export to the environment.
-    fn declare_memory_export(&mut self, memory_index: MemoryIndex, name: &'data str);
+    fn declare_memory_export(
+        &mut self,
+        memory_index: MemoryIndex,
+        name: &'data str,
+    ) -> WasmResult<()>;
 
     /// Declares a global export to the environment.
-    fn declare_global_export(&mut self, global_index: GlobalIndex, name: &'data str);
+    fn declare_global_export(
+        &mut self,
+        global_index: GlobalIndex,
+        name: &'data str,
+    ) -> WasmResult<()>;
 
     /// Notifies the implementation that all exports have been declared.
-    fn finish_exports(&mut self) {}
+    fn finish_exports(&mut self) -> WasmResult<()> {
+        Ok(())
+    }
 
     /// Declares the optional start function.
-    fn declare_start_func(&mut self, index: FuncIndex);
+    fn declare_start_func(&mut self, index: FuncIndex) -> WasmResult<()>;
 
     /// Provides the number of element initializers up front. By default this does nothing, but
     /// implementations can use this to preallocate memory if desired.
-    fn reserve_table_elements(&mut self, _num: u32) {}
+    fn reserve_table_elements(&mut self, _num: u32) -> WasmResult<()> {
+        Ok(())
+    }
 
     /// Fills a declared table with references to functions in the module.
     fn declare_table_elements(
@@ -379,7 +430,7 @@ pub trait ModuleEnvironment<'data> {
         base: Option,
         offset: usize,
         elements: Box<[FuncIndex]>,
-    );
+    ) -> WasmResult<()>;
 
     /// Provides the contents of a function body.
     ///
@@ -393,7 +444,9 @@ pub trait ModuleEnvironment<'data> {
 
     /// Provides the number of data initializers up front. By default this does nothing, but
     /// implementations can use this to preallocate memory if desired.
-    fn reserve_data_initializers(&mut self, _num: u32) {}
+    fn reserve_data_initializers(&mut self, _num: u32) -> WasmResult<()> {
+        Ok(())
+    }
 
     /// Fills a declared memory with bytes at module instantiation.
     fn declare_data_initialization(
@@ -402,5 +455,5 @@ pub trait ModuleEnvironment<'data> {
         base: Option,
         offset: usize,
         data: &'data [u8],
-    );
+    ) -> WasmResult<()>;
 }
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index a2ecdd06b3..7f62f5cfe0 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -5,9 +5,10 @@
 //! WebAssembly module and the runtime environment.
 
 use crate::code_translator::translate_operator;
-use crate::environ::{FuncEnvironment, ReturnMode, WasmError, WasmResult};
+use crate::environ::{FuncEnvironment, ReturnMode, WasmResult};
 use crate::state::{TranslationState, VisibleTranslationState};
 use crate::translation_utils::get_vmctx_value_label;
+use crate::wasm_unsupported;
 use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir::{self, Ebb, InstBuilder, ValueLabel};
 use cranelift_codegen::timing;
@@ -176,7 +177,7 @@ fn declare_locals(
         I64 => builder.ins().iconst(ir::types::I64, 0),
         F32 => builder.ins().f32const(ir::immediates::Ieee32::with_bits(0)),
         F64 => builder.ins().f64const(ir::immediates::Ieee64::with_bits(0)),
-        _ => return Err(WasmError::Unsupported("unsupported local type")),
+        ty => wasm_unsupported!("unsupported local type {:?}", ty),
     };
 
     let ty = builder.func.dfg.value_type(zeroval);
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index f0975d9643..4ca8a66b87 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -7,11 +7,12 @@
 //! The special case of the initialize expressions for table elements offsets or global variables
 //! is handled, according to the semantics of WebAssembly, to only specific expressions that are
 //! interpreted on the fly.
-use crate::environ::{ModuleEnvironment, WasmError, WasmResult};
+use crate::environ::{ModuleEnvironment, WasmResult};
 use crate::translation_utils::{
     tabletype_to_type, type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory,
     MemoryIndex, SignatureIndex, Table, TableElementType, TableIndex,
 };
+use crate::wasm_unsupported;
 use core::convert::TryFrom;
 use cranelift_codegen::ir::{self, AbiParam, Signature};
 use cranelift_entity::EntityRef;
@@ -29,7 +30,7 @@ pub fn parse_type_section(
     types: TypeSectionReader,
     environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
-    environ.reserve_signatures(types.get_count());
+    environ.reserve_signatures(types.get_count())?;
 
     for entry in types {
         match entry? {
@@ -49,9 +50,9 @@ pub fn parse_type_section(
                         .expect("only numeric types are supported in function signatures");
                     AbiParam::new(cret_arg)
                 }));
-                environ.declare_signature(sig);
+                environ.declare_signature(sig)?;
             }
-            _ => return Err(WasmError::Unsupported("unsupported type in type section")),
+            ty => wasm_unsupported!("unsupported type in type section: {:?}", ty),
         }
     }
     Ok(())
@@ -62,7 +63,7 @@ pub fn parse_import_section<'data>(
     imports: ImportSectionReader<'data>,
     environ: &mut dyn ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
-    environ.reserve_imports(imports.get_count());
+    environ.reserve_imports(imports.get_count())?;
 
     for entry in imports {
         let import = entry?;
@@ -71,7 +72,11 @@ pub fn parse_import_section<'data>(
 
         match import.ty {
             ImportSectionEntryType::Function(sig) => {
-                environ.declare_func_import(SignatureIndex::from_u32(sig), module_name, field_name);
+                environ.declare_func_import(
+                    SignatureIndex::from_u32(sig),
+                    module_name,
+                    field_name,
+                )?;
             }
             ImportSectionEntryType::Memory(MemoryType {
                 limits: ref memlimits,
@@ -85,7 +90,7 @@ pub fn parse_import_section<'data>(
                     },
                     module_name,
                     field_name,
-                );
+                )?;
             }
             ImportSectionEntryType::Global(ref ty) => {
                 environ.declare_global_import(
@@ -96,7 +101,7 @@ pub fn parse_import_section<'data>(
                     },
                     module_name,
                     field_name,
-                );
+                )?;
             }
             ImportSectionEntryType::Table(ref tab) => {
                 environ.declare_table_import(
@@ -110,12 +115,12 @@ pub fn parse_import_section<'data>(
                     },
                     module_name,
                     field_name,
-                );
+                )?;
             }
         }
     }
 
-    environ.finish_imports();
+    environ.finish_imports()?;
     Ok(())
 }
 
@@ -124,11 +129,11 @@ pub fn parse_function_section(
     functions: FunctionSectionReader,
     environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
-    environ.reserve_func_types(functions.get_count());
+    environ.reserve_func_types(functions.get_count())?;
 
     for entry in functions {
         let sigindex = entry?;
-        environ.declare_func_type(SignatureIndex::from_u32(sigindex));
+        environ.declare_func_type(SignatureIndex::from_u32(sigindex))?;
     }
 
     Ok(())
@@ -139,7 +144,7 @@ pub fn parse_table_section(
     tables: TableSectionReader,
     environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
-    environ.reserve_tables(tables.get_count());
+    environ.reserve_tables(tables.get_count())?;
 
     for entry in tables {
         let table = entry?;
@@ -150,7 +155,7 @@ pub fn parse_table_section(
             },
             minimum: table.limits.initial,
             maximum: table.limits.maximum,
-        });
+        })?;
     }
 
     Ok(())
@@ -161,7 +166,7 @@ pub fn parse_memory_section(
     memories: MemorySectionReader,
     environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
-    environ.reserve_memories(memories.get_count());
+    environ.reserve_memories(memories.get_count())?;
 
     for entry in memories {
         let memory = entry?;
@@ -169,7 +174,7 @@ pub fn parse_memory_section(
             minimum: memory.limits.initial,
             maximum: memory.limits.maximum,
             shared: memory.shared,
-        });
+        })?;
     }
 
     Ok(())
@@ -180,7 +185,7 @@ pub fn parse_global_section(
     globals: GlobalSectionReader,
     environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
-    environ.reserve_globals(globals.get_count());
+    environ.reserve_globals(globals.get_count())?;
 
     for entry in globals {
         let wasmparser::Global {
@@ -199,14 +204,16 @@ pub fn parse_global_section(
             Operator::GetGlobal { global_index } => {
                 GlobalInit::GetGlobal(GlobalIndex::from_u32(global_index))
             }
-            ref s => panic!("unsupported init expr in global section: {:?}", s),
+            ref s => {
+                wasm_unsupported!("unsupported init expr in global section: {:?}", s);
+            }
         };
         let global = Global {
             ty: type_to_type(content_type).unwrap(),
             mutability: mutable,
             initializer,
         };
-        environ.declare_global(global);
+        environ.declare_global(global)?;
     }
 
     Ok(())
@@ -217,7 +224,7 @@ pub fn parse_export_section<'data>(
     exports: ExportSectionReader<'data>,
     environ: &mut dyn ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
-    environ.reserve_exports(exports.get_count());
+    environ.reserve_exports(exports.get_count())?;
 
     for entry in exports {
         let Export {
@@ -231,20 +238,24 @@ pub fn parse_export_section<'data>(
         // becomes a concern here.
         let index = index as usize;
         match *kind {
-            ExternalKind::Function => environ.declare_func_export(FuncIndex::new(index), field),
-            ExternalKind::Table => environ.declare_table_export(TableIndex::new(index), field),
-            ExternalKind::Memory => environ.declare_memory_export(MemoryIndex::new(index), field),
-            ExternalKind::Global => environ.declare_global_export(GlobalIndex::new(index), field),
+            ExternalKind::Function => environ.declare_func_export(FuncIndex::new(index), field)?,
+            ExternalKind::Table => environ.declare_table_export(TableIndex::new(index), field)?,
+            ExternalKind::Memory => {
+                environ.declare_memory_export(MemoryIndex::new(index), field)?
+            }
+            ExternalKind::Global => {
+                environ.declare_global_export(GlobalIndex::new(index), field)?
+            }
         }
     }
 
-    environ.finish_exports();
+    environ.finish_exports()?;
     Ok(())
 }
 
 /// Parses the Start section of the wasm module.
 pub fn parse_start_section(index: u32, environ: &mut dyn ModuleEnvironment) -> WasmResult<()> {
-    environ.declare_start_func(FuncIndex::from_u32(index));
+    environ.declare_start_func(FuncIndex::from_u32(index))?;
     Ok(())
 }
 
@@ -253,7 +264,7 @@ pub fn parse_element_section<'data>(
     elements: ElementSectionReader<'data>,
     environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
-    environ.reserve_table_elements(elements.get_count());
+    environ.reserve_table_elements(elements.get_count())?;
 
     for entry in elements {
         let Element { kind, items } = entry?;
@@ -268,7 +279,9 @@ pub fn parse_element_section<'data>(
                 Operator::GetGlobal { global_index } => {
                     (Some(GlobalIndex::from_u32(global_index)), 0)
                 }
-                ref s => panic!("unsupported init expr in element section: {:?}", s),
+                ref s => {
+                    wasm_unsupported!("unsupported init expr in element section: {:?}", s);
+                }
             };
             let items_reader = items.get_items_reader()?;
             let mut elems = Vec::with_capacity(usize::try_from(items_reader.get_count()).unwrap());
@@ -281,9 +294,9 @@ pub fn parse_element_section<'data>(
                 base,
                 offset,
                 elems.into_boxed_slice(),
-            )
+            )?
         } else {
-            panic!("unsupported passive elements section");
+            wasm_unsupported!("unsupported passive elements section: {:?}", kind);
         }
     }
     Ok(())
@@ -308,7 +321,7 @@ pub fn parse_data_section<'data>(
     data: DataSectionReader<'data>,
     environ: &mut dyn ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
-    environ.reserve_data_initializers(data.get_count());
+    environ.reserve_data_initializers(data.get_count())?;
 
     for entry in data {
         let Data { kind, data } = entry?;
@@ -323,16 +336,16 @@ pub fn parse_data_section<'data>(
                 Operator::GetGlobal { global_index } => {
                     (Some(GlobalIndex::from_u32(global_index)), 0)
                 }
-                ref s => panic!("unsupported init expr in data section: {:?}", s),
+                ref s => wasm_unsupported!("unsupported init expr in data section: {:?}", s),
             };
             environ.declare_data_initialization(
                 MemoryIndex::from_u32(memory_index),
                 base,
                 offset,
                 data,
-            );
+            )?;
         } else {
-            panic!("unsupported passive data section");
+            wasm_unsupported!("unsupported passive data section: {:?}", kind);
         }
     }
 
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index a41c70c4d9..90967d4102 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -1,5 +1,6 @@
 //! Helper functions and structures for the translation.
-use crate::environ::{WasmError, WasmResult};
+use crate::environ::WasmResult;
+use crate::wasm_unsupported;
 use core::u32;
 use cranelift_codegen::entity::entity_impl;
 use cranelift_codegen::ir;
@@ -119,7 +120,7 @@ pub fn type_to_type(ty: wasmparser::Type) -> WasmResult {
         wasmparser::Type::I64 => ir::types::I64,
         wasmparser::Type::F32 => ir::types::F32,
         wasmparser::Type::F64 => ir::types::F64,
-        _ => return Err(WasmError::Unsupported("unsupported wasm type")),
+        ty => wasm_unsupported!("unsupported wasm type {:?}", ty),
     })
 }
 
@@ -132,7 +133,7 @@ pub fn tabletype_to_type(ty: wasmparser::Type) -> WasmResult> {
         wasmparser::Type::F32 => Some(ir::types::F32),
         wasmparser::Type::F64 => Some(ir::types::F64),
         wasmparser::Type::AnyFunc => None,
-        _ => return Err(WasmError::Unsupported("unsupported table wasm type")),
+        ty => wasm_unsupported!("unsupported table wasm type {:?}", ty),
     })
 }
 
@@ -141,7 +142,7 @@ pub fn blocktype_to_type(ty: wasmparser::TypeOrFuncType) -> WasmResult
     match ty {
         wasmparser::TypeOrFuncType::Type(ty) => type_to_type(ty),
         wasmparser::TypeOrFuncType::FuncType(_) => {
-            Err(WasmError::Unsupported("multi-value block signatures"))
+            wasm_unsupported!("multi-value block signature {:?}", ty);
         }
     }
 }
@@ -165,10 +166,10 @@ pub fn num_return_values(ty: wasmparser::TypeOrFuncType) -> WasmResult {
             | wasmparser::Type::F32
             | wasmparser::Type::I64
             | wasmparser::Type::F64 => Ok(1),
-            _ => Err(WasmError::Unsupported("unsupported return value type")),
+            ty => wasm_unsupported!("unsupported return value type {:?}", ty),
         },
         wasmparser::TypeOrFuncType::FuncType(_) => {
-            Err(WasmError::Unsupported("multi-value block signatures"))
+            wasm_unsupported!("multi-value block signature {:?}", ty);
         }
     }
 }

From e6e274a3aa2564b18438c82a645a50fbe3e570d6 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Thu, 8 Aug 2019 11:40:41 +0200
Subject: [PATCH 2568/3084] Add bugpoint command

Add the command `cargo run --release bugpoint crash.clif 2>/dev/null` to reduce test cases.
---
 cranelift/Cargo.toml             |    1 +
 cranelift/src/bugpoint.rs        |  850 ++++++++++++++
 cranelift/src/bugpoint_test.clif | 1784 ++++++++++++++++++++++++++++++
 cranelift/src/clif-util.rs       |   22 +
 4 files changed, 2657 insertions(+)
 create mode 100644 cranelift/src/bugpoint.rs
 create mode 100644 cranelift/src/bugpoint_test.clif

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index f5e066734c..490fc8cdc2 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -41,6 +41,7 @@ wabt = { version = "0.7.0", optional = true }
 target-lexicon = "0.4.0"
 pretty_env_logger = "0.3.0"
 file-per-thread-logger = "0.1.2"
+indicatif = "0.11.0"
 
 [features]
 default = ["disas", "wasm"]
diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
new file mode 100644
index 0000000000..fcac964aa3
--- /dev/null
+++ b/cranelift/src/bugpoint.rs
@@ -0,0 +1,850 @@
+//! CLI tool to reduce Cranelift IR files crashing during compilation.
+
+use crate::disasm::{PrintRelocs, PrintTraps};
+use crate::utils::{parse_sets_and_triple, read_to_string};
+use cranelift_codegen::ir::{
+    Ebb, FuncRef, Function, GlobalValueData, Inst, InstBuilder, InstructionData, StackSlots,
+    TrapCode,
+};
+use cranelift_codegen::isa::TargetIsa;
+use cranelift_codegen::Context;
+use cranelift_entity::PrimaryMap;
+use cranelift_reader::parse_test;
+use std::collections::HashMap;
+use std::path::Path;
+
+use indicatif::{ProgressBar, ProgressDrawTarget, ProgressStyle};
+
+pub fn run(
+    filename: &str,
+    flag_set: &[String],
+    flag_isa: &str,
+    verbose: bool,
+) -> Result<(), String> {
+    let parsed = parse_sets_and_triple(flag_set, flag_isa)?;
+    let fisa = parsed.as_fisa();
+
+    let path = Path::new(&filename).to_path_buf();
+
+    let buffer = read_to_string(&path).map_err(|e| format!("{}: {}", filename, e))?;
+    let test_file = parse_test(&buffer, None, None).map_err(|e| format!("{}: {}", filename, e))?;
+
+    // If we have an isa from the command-line, use that. Otherwise if the
+    // file contains a unique isa, use that.
+    let isa = if let Some(isa) = fisa.isa {
+        isa
+    } else if let Some(isa) = test_file.isa_spec.unique_isa() {
+        isa
+    } else {
+        return Err(String::from("compilation requires a target isa"));
+    };
+
+    std::env::set_var("RUST_BACKTRACE", "0"); // Disable backtraces to reduce verbosity
+
+    for (func, _) in test_file.functions {
+        let (orig_ebb_count, orig_inst_count) = (ebb_count(&func), inst_count(&func));
+
+        match reduce(isa, func, verbose) {
+            Ok((func, crash_msg)) => {
+                println!("Crash message: {}", crash_msg);
+
+                println!("\n{}", func);
+
+                println!(
+                    "{} ebbs {} insts -> {} ebbs {} insts",
+                    orig_ebb_count,
+                    orig_inst_count,
+                    ebb_count(&func),
+                    inst_count(&func)
+                );
+            }
+            Err(err) => println!("Warning: {}", err),
+        }
+    }
+
+    Ok(())
+}
+
+enum MutationKind {
+    /// The mutation reduced the amount of instructions or ebbs.
+    Shrinked,
+    /// The mutation only changed an instruction. Performing another round of mutations may only
+    /// reduce the test case if another mutation shrank the test case.
+    Changed,
+}
+
+trait Mutator {
+    fn name(&self) -> &'static str;
+
+    fn mutation_count(&self, func: &Function) -> Option;
+
+    fn mutate(&mut self, func: Function) -> Option<(Function, String, MutationKind)>;
+
+    fn reduce(
+        &mut self,
+        ccc: &mut CrashCheckContext,
+        mut func: Function,
+        progress_bar_prefix: String,
+        verbose: bool,
+        should_keep_reducing: &mut bool,
+    ) -> Function {
+        let progress = ProgressBar::with_draw_target(
+            self.mutation_count(&func).unwrap_or(0) as u64,
+            ProgressDrawTarget::stdout(),
+        );
+        progress.set_style(
+            ProgressStyle::default_bar().template("{bar:60} {prefix:40} {pos:>4}/{len:>4} {msg}"),
+        );
+
+        progress.set_prefix(&(progress_bar_prefix + &format!(" phase {}", self.name())));
+
+        for _ in 0..10000 {
+            progress.inc(1);
+
+            let (mutated_func, msg, mutation_kind) = match self.mutate(func.clone()) {
+                Some(res) => res,
+                None => {
+                    break;
+                }
+            };
+
+            progress.set_message(&msg);
+
+            match ccc.check_for_crash(&mutated_func) {
+                CheckResult::Succeed => {
+                    // Shrinking didn't hit the problem anymore, discard changes.
+                    continue;
+                }
+                CheckResult::Crash(_) => {
+                    // Panic remained while shrinking, make changes definitive.
+                    func = mutated_func;
+                    match mutation_kind {
+                        MutationKind::Shrinked => {
+                            *should_keep_reducing = true;
+                            if verbose {
+                                progress.println(format!("{}: shrink", msg));
+                            }
+                        }
+                        MutationKind::Changed => {
+                            if verbose {
+                                progress.println(format!("{}: changed", msg));
+                            }
+                        }
+                    }
+                }
+            }
+        }
+
+        progress.set_message("done");
+        progress.finish();
+
+        func
+    }
+}
+
+/// Try to remove instructions.
+struct RemoveInst {
+    ebb: Ebb,
+    inst: Inst,
+}
+
+impl RemoveInst {
+    fn new(func: &Function) -> Self {
+        let first_ebb = func.layout.entry_block().unwrap();
+        let first_inst = func.layout.first_inst(first_ebb).unwrap();
+        Self {
+            ebb: first_ebb,
+            inst: first_inst,
+        }
+    }
+}
+
+impl Mutator for RemoveInst {
+    fn name(&self) -> &'static str {
+        "remove inst"
+    }
+
+    fn mutation_count(&self, func: &Function) -> Option {
+        Some(inst_count(func))
+    }
+
+    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
+        if let Some((prev_ebb, prev_inst)) =
+            next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst)
+        {
+            func.layout.remove_inst(prev_inst);
+            if func.layout.ebb_insts(prev_ebb).next().is_none() {
+                // Make sure empty ebbs are removed, as `next_inst_ret_prev` depends on non empty ebbs
+                func.layout.remove_ebb(prev_ebb);
+                Some((
+                    func,
+                    format!("Remove inst {} and empty ebb {}", prev_inst, prev_ebb),
+                    MutationKind::Shrinked,
+                ))
+            } else {
+                Some((
+                    func,
+                    format!("Remove inst {}", prev_inst),
+                    MutationKind::Shrinked,
+                ))
+            }
+        } else {
+            None
+        }
+    }
+}
+
+/// Try to replace instructions with `iconst`.
+struct ReplaceInstWithIconst {
+    ebb: Ebb,
+    inst: Inst,
+}
+
+impl ReplaceInstWithIconst {
+    fn new(func: &Function) -> Self {
+        let first_ebb = func.layout.entry_block().unwrap();
+        let first_inst = func.layout.first_inst(first_ebb).unwrap();
+        Self {
+            ebb: first_ebb,
+            inst: first_inst,
+        }
+    }
+}
+
+impl Mutator for ReplaceInstWithIconst {
+    fn name(&self) -> &'static str {
+        "replace inst with iconst"
+    }
+
+    fn mutation_count(&self, func: &Function) -> Option {
+        Some(inst_count(func))
+    }
+
+    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
+        if let Some((_prev_ebb, prev_inst)) =
+            next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst)
+        {
+            let results = func.dfg.inst_results(prev_inst);
+            if results.len() == 1 {
+                let ty = func.dfg.value_type(results[0]);
+                func.dfg.replace(prev_inst).iconst(ty, 0);
+                Some((
+                    func,
+                    format!("Replace inst {} with iconst.{}", prev_inst, ty),
+                    MutationKind::Changed,
+                ))
+            } else {
+                Some((func, format!(""), MutationKind::Changed))
+            }
+        } else {
+            None
+        }
+    }
+}
+
+/// Try to replace instructions with `trap`.
+struct ReplaceInstWithTrap {
+    ebb: Ebb,
+    inst: Inst,
+}
+
+impl ReplaceInstWithTrap {
+    fn new(func: &Function) -> Self {
+        let first_ebb = func.layout.entry_block().unwrap();
+        let first_inst = func.layout.first_inst(first_ebb).unwrap();
+        Self {
+            ebb: first_ebb,
+            inst: first_inst,
+        }
+    }
+}
+
+impl Mutator for ReplaceInstWithTrap {
+    fn name(&self) -> &'static str {
+        "replace inst with trap"
+    }
+
+    fn mutation_count(&self, func: &Function) -> Option {
+        Some(inst_count(func))
+    }
+
+    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
+        if let Some((_prev_ebb, prev_inst)) =
+            next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst)
+        {
+            func.dfg.replace(prev_inst).trap(TrapCode::User(0));
+            Some((
+                func,
+                format!("Replace inst {} with trap", prev_inst),
+                MutationKind::Changed,
+            ))
+        } else {
+            None
+        }
+    }
+}
+
+/// Try to remove an ebb.
+struct RemoveEbb {
+    ebb: Ebb,
+}
+
+impl RemoveEbb {
+    fn new(func: &Function) -> Self {
+        Self {
+            ebb: func.layout.entry_block().unwrap(),
+        }
+    }
+}
+
+impl Mutator for RemoveEbb {
+    fn name(&self) -> &'static str {
+        "remove ebb"
+    }
+
+    fn mutation_count(&self, func: &Function) -> Option {
+        Some(ebb_count(func))
+    }
+
+    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
+        if let Some(next_ebb) = func.layout.next_ebb(self.ebb) {
+            self.ebb = next_ebb;
+            while let Some(inst) = func.layout.last_inst(self.ebb) {
+                func.layout.remove_inst(inst);
+            }
+            func.layout.remove_ebb(self.ebb);
+            Some((
+                func,
+                format!("Remove ebb {}", next_ebb),
+                MutationKind::Shrinked,
+            ))
+        } else {
+            None
+        }
+    }
+}
+
+/// Try to remove unused entities.
+struct RemoveUnusedEntities {
+    kind: u32,
+}
+
+impl RemoveUnusedEntities {
+    fn new() -> Self {
+        Self { kind: 0 }
+    }
+}
+
+impl Mutator for RemoveUnusedEntities {
+    fn name(&self) -> &'static str {
+        "remove unused entities"
+    }
+
+    fn mutation_count(&self, _func: &Function) -> Option {
+        Some(4)
+    }
+
+    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
+        let name = match self.kind {
+            0 => {
+                let mut ext_func_usage_map = HashMap::new();
+                for ebb in func.layout.ebbs() {
+                    for inst in func.layout.ebb_insts(ebb) {
+                        match func.dfg[inst] {
+                            // Add new cases when there are new instruction formats taking a `FuncRef`.
+                            InstructionData::Call { func_ref, .. }
+                            | InstructionData::FuncAddr { func_ref, .. } => {
+                                ext_func_usage_map
+                                    .entry(func_ref)
+                                    .or_insert_with(Vec::new)
+                                    .push(inst);
+                            }
+                            _ => {}
+                        }
+                    }
+                }
+
+                let mut ext_funcs = PrimaryMap::new();
+
+                for (func_ref, ext_func_data) in func.dfg.ext_funcs.clone().into_iter() {
+                    if let Some(func_ref_usage) = ext_func_usage_map.get(&func_ref) {
+                        let new_func_ref = ext_funcs.push(ext_func_data.clone());
+                        for &inst in func_ref_usage {
+                            match func.dfg[inst] {
+                                // Keep in sync with the above match.
+                                InstructionData::Call {
+                                    ref mut func_ref, ..
+                                }
+                                | InstructionData::FuncAddr {
+                                    ref mut func_ref, ..
+                                } => {
+                                    *func_ref = new_func_ref;
+                                }
+                                _ => unreachable!(),
+                            }
+                        }
+                    }
+                }
+
+                func.dfg.ext_funcs = ext_funcs;
+
+                "Remove unused ext funcs"
+            }
+            1 => {
+                #[derive(Copy, Clone)]
+                enum SigRefUser {
+                    Instruction(Inst),
+                    ExtFunc(FuncRef),
+                }
+
+                let mut signatures_usage_map = HashMap::new();
+                for ebb in func.layout.ebbs() {
+                    for inst in func.layout.ebb_insts(ebb) {
+                        match func.dfg[inst] {
+                            // Add new cases when there are new instruction formats taking a `SigRef`.
+                            InstructionData::CallIndirect { sig_ref, .. } => {
+                                signatures_usage_map
+                                    .entry(sig_ref)
+                                    .or_insert_with(Vec::new)
+                                    .push(SigRefUser::Instruction(inst));
+                            }
+                            _ => {}
+                        }
+                    }
+                }
+                for (func_ref, ext_func_data) in func.dfg.ext_funcs.iter() {
+                    signatures_usage_map
+                        .entry(ext_func_data.signature)
+                        .or_insert_with(Vec::new)
+                        .push(SigRefUser::ExtFunc(func_ref));
+                }
+
+                let mut signatures = PrimaryMap::new();
+
+                for (sig_ref, sig_data) in func.dfg.signatures.clone().into_iter() {
+                    if let Some(sig_ref_usage) = signatures_usage_map.get(&sig_ref) {
+                        let new_sig_ref = signatures.push(sig_data.clone());
+                        for &sig_ref_user in sig_ref_usage {
+                            match sig_ref_user {
+                                SigRefUser::Instruction(inst) => match func.dfg[inst] {
+                                    // Keep in sync with the above match.
+                                    InstructionData::CallIndirect {
+                                        ref mut sig_ref, ..
+                                    } => {
+                                        *sig_ref = new_sig_ref;
+                                    }
+                                    _ => unreachable!(),
+                                },
+                                SigRefUser::ExtFunc(func_ref) => {
+                                    func.dfg.ext_funcs[func_ref].signature = new_sig_ref;
+                                }
+                            }
+                        }
+                    }
+                }
+
+                func.dfg.signatures = signatures;
+
+                "Remove unused signatures"
+            }
+            2 => {
+                let mut stack_slot_usage_map = HashMap::new();
+                for ebb in func.layout.ebbs() {
+                    for inst in func.layout.ebb_insts(ebb) {
+                        match func.dfg[inst] {
+                            // Add new cases when there are new instruction formats taking a `StackSlot`.
+                            InstructionData::StackLoad { stack_slot, .. }
+                            | InstructionData::StackStore { stack_slot, .. } => {
+                                stack_slot_usage_map
+                                    .entry(stack_slot)
+                                    .or_insert_with(Vec::new)
+                                    .push(inst);
+                            }
+
+                            InstructionData::RegSpill { dst, .. } => {
+                                stack_slot_usage_map
+                                    .entry(dst)
+                                    .or_insert_with(Vec::new)
+                                    .push(inst);
+                            }
+                            InstructionData::RegFill { src, .. } => {
+                                stack_slot_usage_map
+                                    .entry(src)
+                                    .or_insert_with(Vec::new)
+                                    .push(inst);
+                            }
+                            _ => {}
+                        }
+                    }
+                }
+
+                let mut stack_slots = StackSlots::new();
+
+                for (stack_slot, stack_slot_data) in func.stack_slots.clone().iter() {
+                    if let Some(stack_slot_usage) = stack_slot_usage_map.get(&stack_slot) {
+                        let new_stack_slot = stack_slots.push(stack_slot_data.clone());
+                        for &inst in stack_slot_usage {
+                            match &mut func.dfg[inst] {
+                                // Keep in sync with the above match.
+                                InstructionData::StackLoad { stack_slot, .. }
+                                | InstructionData::StackStore { stack_slot, .. } => {
+                                    *stack_slot = new_stack_slot;
+                                }
+                                InstructionData::RegSpill { dst, .. } => {
+                                    *dst = new_stack_slot;
+                                }
+                                InstructionData::RegFill { src, .. } => {
+                                    *src = new_stack_slot;
+                                }
+                                _ => unreachable!(),
+                            }
+                        }
+                    }
+                }
+
+                func.stack_slots = stack_slots;
+
+                "Remove unused stack slots"
+            }
+            3 => {
+                let mut global_value_usage_map = HashMap::new();
+                for ebb in func.layout.ebbs() {
+                    for inst in func.layout.ebb_insts(ebb) {
+                        match func.dfg[inst] {
+                            // Add new cases when there are new instruction formats taking a `GlobalValue`.
+                            InstructionData::UnaryGlobalValue { global_value, .. } => {
+                                global_value_usage_map
+                                    .entry(global_value)
+                                    .or_insert_with(Vec::new)
+                                    .push(inst);
+                            }
+                            _ => {}
+                        }
+                    }
+                }
+
+                for (_global_value, global_value_data) in func.global_values.iter() {
+                    match *global_value_data {
+                        GlobalValueData::VMContext | GlobalValueData::Symbol { .. } => {}
+                        // These can create cyclic references, which cause complications. Just skip
+                        // the global value removal for now.
+                        // FIXME Handle them in a better way.
+                        GlobalValueData::Load { base: _, .. }
+                        | GlobalValueData::IAddImm { base: _, .. } => return None,
+                    }
+                }
+
+                let mut global_values = PrimaryMap::new();
+
+                for (global_value, global_value_data) in func.global_values.clone().into_iter() {
+                    if let Some(global_value_usage) = global_value_usage_map.get(&global_value) {
+                        let new_global_value = global_values.push(global_value_data.clone());
+                        for &inst in global_value_usage {
+                            match &mut func.dfg[inst] {
+                                // Keep in sync with the above match.
+                                InstructionData::UnaryGlobalValue { global_value, .. } => {
+                                    *global_value = new_global_value;
+                                }
+                                _ => unreachable!(),
+                            }
+                        }
+                    }
+                }
+
+                func.global_values = global_values;
+
+                "Remove unused global values"
+            }
+            _ => return None,
+        };
+        self.kind += 1;
+        Some((func, name.to_owned(), MutationKind::Changed))
+    }
+}
+
+fn next_inst_ret_prev(func: &Function, ebb: &mut Ebb, inst: &mut Inst) -> Option<(Ebb, Inst)> {
+    let prev = (*ebb, *inst);
+    if let Some(next_inst) = func.layout.next_inst(*inst) {
+        *inst = next_inst;
+        return Some(prev);
+    } else if let Some(next_ebb) = func.layout.next_ebb(*ebb) {
+        *ebb = next_ebb;
+        *inst = func.layout.first_inst(*ebb).expect("no inst");
+        return Some(prev);
+    } else {
+        return None;
+    }
+}
+
+fn ebb_count(func: &Function) -> usize {
+    func.layout.ebbs().count()
+}
+
+fn inst_count(func: &Function) -> usize {
+    func.layout
+        .ebbs()
+        .map(|ebb| func.layout.ebb_insts(ebb).count())
+        .sum()
+}
+
+fn resolve_aliases(func: &mut Function) {
+    for ebb in func.layout.ebbs() {
+        for inst in func.layout.ebb_insts(ebb) {
+            func.dfg.resolve_aliases_in_arguments(inst);
+        }
+    }
+}
+
+fn reduce(
+    isa: &TargetIsa,
+    mut func: Function,
+    verbose: bool,
+) -> Result<(Function, String), String> {
+    let mut ccc = CrashCheckContext::new(isa);
+
+    match ccc.check_for_crash(&func) {
+        CheckResult::Succeed => {
+            return Err(format!(
+                "Given function compiled successfully or gave an verifier error."
+            ));
+        }
+        CheckResult::Crash(_) => {}
+    }
+
+    resolve_aliases(&mut func);
+
+    for pass_idx in 0..100 {
+        let mut should_keep_reducing = false;
+        let mut phase = 0;
+
+        loop {
+            let mut mutator = match phase {
+                0 => Box::new(RemoveInst::new(&func)) as Box,
+                1 => Box::new(ReplaceInstWithIconst::new(&func)) as Box,
+                2 => Box::new(ReplaceInstWithTrap::new(&func)) as Box,
+                3 => Box::new(RemoveEbb::new(&func)) as Box,
+                4 => Box::new(RemoveUnusedEntities::new()) as Box,
+                _ => break,
+            };
+
+            func = mutator.reduce(
+                &mut ccc,
+                func,
+                format!("pass {}", pass_idx),
+                verbose,
+                &mut should_keep_reducing,
+            );
+
+            phase += 1;
+        }
+
+        if !should_keep_reducing {
+            // No new shrinking opportunities have been found this pass. This means none will ever
+            // be found. Skip the rest of the passes over the function.
+            break;
+        }
+    }
+
+    let crash_msg = match ccc.check_for_crash(&func) {
+        CheckResult::Succeed => unreachable!("Used to crash, but doesn't anymore???"),
+        CheckResult::Crash(crash_msg) => crash_msg,
+    };
+
+    Ok((func, crash_msg))
+}
+
+struct CrashCheckContext<'a> {
+    /// Cached `Context`, to prevent repeated allocation.
+    context: Context,
+
+    /// The target isa to compile for.
+    isa: &'a TargetIsa,
+}
+
+fn get_panic_string(panic: Box) -> String {
+    let panic = match panic.downcast::<&'static str>() {
+        Ok(panic_msg) => panic_msg.to_owned(),
+        Err(panic) => panic,
+    };
+    match panic.downcast::() {
+        Ok(panic_msg) => *panic_msg,
+        Err(_) => "Box".to_owned(),
+    }
+}
+
+enum CheckResult {
+    /// The function compiled fine, or the verifier noticed an error.
+    Succeed,
+
+    /// The compilation of the function panicked.
+    Crash(String),
+}
+
+impl<'a> CrashCheckContext<'a> {
+    fn new(isa: &'a TargetIsa) -> Self {
+        CrashCheckContext {
+            context: Context::new(),
+            isa,
+        }
+    }
+
+    #[cfg_attr(test, allow(unreachable_code))]
+    fn check_for_crash(&mut self, func: &Function) -> CheckResult {
+        self.context.clear();
+        self.context.func = func.clone();
+
+        use std::io::Write;
+        std::io::stdout().flush().unwrap(); // Flush stdout to sync with panic messages on stderr
+
+        match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
+            cranelift_codegen::verifier::verify_function(&func, self.isa).err()
+        })) {
+            Ok(Some(_)) => return CheckResult::Succeed,
+            Ok(None) => {}
+            // The verifier panicked. compiling it will probably give the same panic.
+            // We treat it as succeeding to make it possible to reduce for the actual error.
+            // FIXME prevent verifier panic on removing ebb1
+            Err(_) => return CheckResult::Succeed,
+        }
+
+        #[cfg(test)]
+        {
+            // For testing purposes we emulate a panic caused by the existence of
+            // a `call` instruction.
+            let contains_call = func.layout.ebbs().any(|ebb| {
+                func.layout.ebb_insts(ebb).any(|inst| match func.dfg[inst] {
+                    InstructionData::Call { .. } => true,
+                    _ => false,
+                })
+            });
+            if contains_call {
+                return CheckResult::Crash("test crash".to_string());
+            } else {
+                return CheckResult::Succeed;
+            }
+        }
+
+        let old_panic_hook = std::panic::take_hook();
+        std::panic::set_hook(Box::new(|_| {})); // silence panics
+
+        let res = match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
+            let mut relocs = PrintRelocs::new(false);
+            let mut traps = PrintTraps::new(false);
+            let mut mem = vec![];
+
+            let _ = self
+                .context
+                .compile_and_emit(self.isa, &mut mem, &mut relocs, &mut traps);
+        })) {
+            Ok(()) => CheckResult::Succeed,
+            Err(err) => CheckResult::Crash(get_panic_string(err)),
+        };
+
+        std::panic::set_hook(old_panic_hook);
+
+        res
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_reduce() {
+        const TEST: &'static str = include_str!("./bugpoint_test.clif");
+
+        let test_file = parse_test(TEST, None, None).unwrap();
+
+        // If we have an isa from the command-line, use that. Otherwise if the
+        // file contains a unique isa, use that.
+        let isa = test_file.isa_spec.unique_isa().expect("Unknown isa");
+
+        for (func, _) in test_file.functions {
+            let (func, crash_msg) = reduce(isa, func, false).expect("Couldn't reduce test case");
+
+            assert_eq!(crash_msg, "test crash");
+
+            assert_eq!(
+                format!("{}", func),
+                "function u0:0(i64, i64, i64) system_v {
+    sig0 = (i64, i64, i16, i64, i64, i64, i64, i64) system_v
+    fn0 = u0:95 sig0
+
+ebb0(v0: i64, v1: i64, v2: i64):
+    v113 -> v1
+    v124 -> v1
+    v136 -> v1
+    v148 -> v1
+    v160 -> v1
+    v185 -> v1
+    v222 -> v1
+    v237 -> v1
+    v241 -> v1
+    v256 -> v1
+    v262 -> v1
+    v105 = iconst.i64 0
+    trap user0
+
+ebb99(v804: i64, v1035: i64, v1037: i64, v1039: i64, v1044: i64, v1052: i16, v1057: i64):
+    v817 -> v1035
+    v830 -> v1037
+    v844 -> v1039
+    v857 -> v1039
+    v939 -> v1039
+    v1042 -> v1039
+    v1050 -> v1039
+    v908 -> v1044
+    v917 -> v1044
+    v921 -> v1044
+    v1043 -> v1044
+    v960 -> v1052
+    v990 -> v1052
+    v1051 -> v1052
+    v1055 -> v1052
+    v963 -> v1057
+    v1056 -> v1057
+    v1060 -> v1057
+    trap user0
+
+ebb101:
+    v829 = iconst.i64 0
+    v935 -> v829
+    v962 -> v829
+    v992 -> v829
+    v1036 -> v829
+    v1049 -> v829
+    trap user0
+
+ebb102:
+    v842 = iconst.i64 0
+    v976 -> v842
+    v989 -> v842
+    v1038 -> v842
+    v1061 -> v842
+    trap user0
+
+ebb105:
+    v883 = iconst.i64 0
+    v934 -> v883
+    v961 -> v883
+    v991 -> v883
+    v1005 -> v883
+    v1048 -> v883
+    trap user0
+
+ebb114:
+    v951 = iconst.i64 0
+    v988 -> v951
+    trap user0
+
+ebb117:
+    v987 = iconst.i64 0
+    call fn0(v0, v105, v1052, v883, v829, v987, v951, v842)
+    trap user0
+}
+"
+            );
+        }
+    }
+}
diff --git a/cranelift/src/bugpoint_test.clif b/cranelift/src/bugpoint_test.clif
new file mode 100644
index 0000000000..20c34e2351
--- /dev/null
+++ b/cranelift/src/bugpoint_test.clif
@@ -0,0 +1,1784 @@
+test compile
+set is_pic
+target x86_64-unknown-linux-gnu
+
+function u0:0(i64, i64, i64) system_v {
+
+
+    ss0 = explicit_slot 16
+    ss1 = explicit_slot 1     ss2 = explicit_slot 16     ss3 = explicit_slot 1     ss4 = explicit_slot 16     ss5 = explicit_slot 8     ss6 = explicit_slot 16     ss7 = explicit_slot 16     ss8 = explicit_slot 16     ss9 = explicit_slot 16     ss10 = explicit_slot 16     ss11 = explicit_slot 16     ss12 = explicit_slot 16     ss13 = explicit_slot 16     ss14 = explicit_slot 16     ss15 = explicit_slot 16     ss16 = explicit_slot 16     ss17 = explicit_slot 16     ss18 = explicit_slot 24     ss19 = explicit_slot 4     ss20 = explicit_slot 4     ss21 = explicit_slot 4     ss22 = explicit_slot 4     ss23 = explicit_slot 16     ss24 = explicit_slot 16     ss25 = explicit_slot 16     ss26 = explicit_slot 16     ss27 = explicit_slot 48     ss28 = explicit_slot 16     ss29 = explicit_slot 16     ss30 = explicit_slot 32     ss31 = explicit_slot 16     ss32 = explicit_slot 8     ss33 = explicit_slot 8     ss34 = explicit_slot 16     ss35 = explicit_slot 16     ss36 = explicit_slot 16     ss37 = explicit_slot 48     ss38 = explicit_slot 16     ss39 = explicit_slot 16     ss40 = explicit_slot 32     ss41 = explicit_slot 16     ss42 = explicit_slot 8     ss43 = explicit_slot 8     ss44 = explicit_slot 16     ss45 = explicit_slot 16     ss46 = explicit_slot 16     ss47 = explicit_slot 16     ss48 = explicit_slot 16     ss49 = explicit_slot 16     ss50 = explicit_slot 16     ss51 = explicit_slot 8     ss52 = explicit_slot 4     ss53 = explicit_slot 4     ss54 = explicit_slot 16     ss55 = explicit_slot 16     ss56 = explicit_slot 16     ss57 = explicit_slot 2     ss58 = explicit_slot 4     ss59 = explicit_slot 2     ss60 = explicit_slot 16     ss61 = explicit_slot 16     ss62 = explicit_slot 16     ss63 = explicit_slot 16     ss64 = explicit_slot 16     ss65 = explicit_slot 16     ss66 = explicit_slot 16     ss67 = explicit_slot 16     ss68 = explicit_slot 8     ss69 = explicit_slot 16     ss70 = explicit_slot 16     ss71 = explicit_slot 48     ss72 = explicit_slot 16     ss73 = explicit_slot 16     ss74 = explicit_slot 32     ss75 = explicit_slot 16     ss76 = explicit_slot 8     ss77 = explicit_slot 8     ss78 = explicit_slot 16     ss79 = explicit_slot 16     ss80 = explicit_slot 16     ss81 = explicit_slot 48     ss82 = explicit_slot 16     ss83 = explicit_slot 16     ss84 = explicit_slot 32     ss85 = explicit_slot 16     ss86 = explicit_slot 8     ss87 = explicit_slot 8     ss88 = explicit_slot 16     ss89 = explicit_slot 16     ss90 = explicit_slot 4     ss91 = explicit_slot 16     ss92 = explicit_slot 16     ss93 = explicit_slot 16     ss94 = explicit_slot 16     ss95 = explicit_slot 16     ss96 = explicit_slot 16     ss97 = explicit_slot 2     ss98 = explicit_slot 16     ss99 = explicit_slot 16     ss100 = explicit_slot 16     ss101 = explicit_slot 16     ss102 = explicit_slot 16     ss103 = explicit_slot 16     ss104 = explicit_slot 8     ss105 = explicit_slot 16     ss106 = explicit_slot 16     ss107 = explicit_slot 4     ss108 = explicit_slot 16
+    ss109 = explicit_slot 16
+    ss110 = explicit_slot 16
+    ss111 = explicit_slot 16
+    ss112 = explicit_slot 4
+    ss113 = explicit_slot 4
+    ss114 = explicit_slot 4
+    ss115 = explicit_slot 4
+    ss116 = explicit_slot 16
+    ss117 = explicit_slot 16
+    ss118 = explicit_slot 16
+    ss119 = explicit_slot 16
+    ss120 = explicit_slot 16
+    ss121 = explicit_slot 4
+    ss122 = explicit_slot 4
+    ss123 = explicit_slot 16
+    ss124 = explicit_slot 16
+    ss125 = explicit_slot 16
+    ss126 = explicit_slot 2
+    ss127 = explicit_slot 16
+    ss128 = explicit_slot 16
+    ss129 = explicit_slot 16
+    ss130 = explicit_slot 16
+    ss131 = explicit_slot 16
+    ss132 = explicit_slot 4
+    ss133 = explicit_slot 16
+    ss134 = explicit_slot 16
+    ss135 = explicit_slot 16
+    ss136 = explicit_slot 16
+    ss137 = explicit_slot 16
+    ss138 = explicit_slot 16
+    ss139 = explicit_slot 2
+    ss140 = explicit_slot 16
+    ss141 = explicit_slot 16
+    ss142 = explicit_slot 16
+    ss143 = explicit_slot 16
+    ss144 = explicit_slot 4
+    gv0 = symbol colocated u1:22
+    gv1 = symbol colocated u1:23
+    gv2 = symbol colocated u1:24
+    gv3 = symbol colocated u1:23
+    gv4 = symbol colocated u1:25
+    gv5 = symbol colocated u1:23
+    gv6 = symbol colocated u1:26
+    gv7 = symbol colocated u1:23
+    gv8 = symbol colocated u1:27
+    gv9 = symbol colocated u1:23
+    gv10 = symbol colocated u1:28
+    gv11 = symbol colocated u1:23
+    gv12 = symbol colocated u1:29
+    gv13 = symbol colocated u1:30
+    gv14 = symbol colocated u1:31
+    gv15 = symbol colocated u1:23
+    gv16 = symbol colocated u1:29
+    gv17 = symbol colocated u1:32
+    gv18 = symbol colocated u1:32
+    gv19 = symbol colocated u1:32
+    gv20 = symbol colocated u1:32
+    gv21 = symbol colocated u1:32
+    gv22 = symbol colocated u1:33
+    gv23 = symbol colocated u1:34
+    gv24 = symbol colocated u1:23
+    gv25 = symbol colocated u1:35
+    gv26 = symbol colocated u1:36
+    gv27 = symbol colocated u1:23
+    gv28 = symbol colocated u1:29
+    gv29 = symbol colocated u1:32
+    gv30 = symbol colocated u1:37
+    gv31 = symbol colocated u1:38
+    gv32 = symbol colocated u1:30
+    gv33 = symbol colocated u1:32
+    gv34 = symbol colocated u1:32
+    gv35 = symbol colocated u1:29
+    gv36 = symbol colocated u1:32
+    gv37 = symbol colocated u1:30
+    gv38 = symbol colocated u1:32
+    gv39 = symbol colocated u1:39
+    gv40 = symbol colocated u1:40
+    gv41 = symbol colocated u1:41
+    gv42 = symbol colocated u1:23
+    gv43 = symbol colocated u1:29
+    gv44 = symbol colocated u1:42
+    gv45 = symbol colocated u1:29
+    gv46 = symbol colocated u1:30
+    gv47 = symbol colocated u1:29
+    gv48 = symbol colocated u1:30
+    gv49 = symbol colocated u1:32
+    gv50 = symbol colocated u1:43
+    gv51 = symbol colocated u1:44
+    gv52 = symbol colocated u1:45
+    gv53 = symbol colocated u1:23
+    gv54 = symbol colocated u1:46
+    gv55 = symbol colocated u1:47
+    gv56 = symbol colocated u1:48
+    gv57 = symbol colocated u1:23
+    gv58 = symbol colocated u1:32
+    gv59 = symbol colocated u1:39
+    gv60 = symbol colocated u1:49
+    gv61 = symbol colocated u1:49
+    gv62 = symbol colocated u1:49
+    gv63 = symbol colocated u1:38
+    gv64 = symbol colocated u1:30
+    gv65 = symbol colocated u1:32
+    gv66 = symbol colocated u1:50
+    gv67 = symbol colocated u1:23
+    gv68 = symbol colocated u1:29
+    gv69 = symbol colocated u1:51
+    gv70 = symbol colocated u1:29
+    gv71 = symbol colocated u1:30
+    gv72 = symbol colocated u1:32
+    gv73 = symbol colocated u1:49
+    gv74 = symbol colocated u1:32
+    sig0 = (i64) system_v
+    sig1 = (i64) system_v
+    sig2 = (i64) system_v
+    sig3 = (i64) system_v
+    sig4 = (i64) system_v
+    sig5 = (i64) system_v
+    sig6 = (i64, i64, i64) system_v
+    sig7 = (i64) -> i8 system_v
+    sig8 = (i64) system_v
+    sig9 = (i64) system_v
+    sig10 = (i64, i64, i64) system_v
+    sig11 = (i64) -> i8 system_v
+    sig12 = (i64) system_v
+    sig13 = (i64) system_v
+    sig14 = (i64) -> i64 system_v
+    sig15 = (i64) system_v
+    sig16 = (i64) system_v
+    sig17 = (i64) system_v
+    sig18 = (i64) system_v
+    sig19 = (i64) system_v
+    sig20 = (i64) system_v
+    sig21 = (i64) system_v
+    sig22 = (i64, i64) system_v
+    sig23 = (i64) system_v
+    sig24 = (i64, i64, i16) system_v
+    sig25 = (i64, i64, i16) system_v
+    sig26 = (i64) system_v
+    sig27 = (i64) system_v
+    sig28 = (i64) system_v
+    sig29 = (i64) system_v
+    sig30 = (i64, i16, i16) system_v
+    sig31 = (i64, i64, i64) system_v
+    sig32 = (i64, i64, i64) system_v
+    sig33 = (i64, i64, i64) system_v
+    sig34 = (i64, i64) -> i8 system_v
+    sig35 = (i64, i64, i64) system_v
+    sig36 = (i64, i64) -> i8 system_v
+    sig37 = (i64, i64, i64) system_v
+    sig38 = (i64, i64, i64) system_v
+    sig39 = (i64, i64) system_v
+    sig40 = (i64) system_v
+    sig41 = (i64, i64) -> i8 system_v
+    sig42 = (i64, i64, i64) system_v
+    sig43 = (i64, i64) -> i8 system_v
+    sig44 = (i64, i64, i64) system_v
+    sig45 = (i64, i64, i64) system_v
+    sig46 = (i64, i64) system_v
+    sig47 = (i64) system_v
+    sig48 = (i64) system_v
+    sig49 = (i64) system_v
+    sig50 = (i64) system_v
+    sig51 = (i64) system_v
+    sig52 = (i64) system_v
+    sig53 = (i64) system_v
+    sig54 = (i64, i32) system_v
+    sig55 = (i64) system_v
+    sig56 = (i64) system_v
+    sig57 = (i64) system_v
+    sig58 = (i64) system_v
+    sig59 = (i64) system_v
+    sig60 = (i64) system_v
+    sig61 = (i64) system_v
+    sig62 = (i64) system_v
+    sig63 = (i64) system_v
+    sig64 = (i64) system_v
+    sig65 = (i64) system_v
+    sig66 = (i64) system_v
+    sig67 = (i64) system_v
+    sig68 = (i64) system_v
+    sig69 = (i64) system_v
+    sig70 = (i64, i64, i64) system_v
+    sig71 = (i64) system_v
+    sig72 = (i64, i64, i16, i64, i64, i64, i64, i64) system_v
+    sig73 = (i64, i64) -> i8 system_v
+    sig74 = (i64, i64, i64) system_v
+    sig75 = (i64, i64) -> i8 system_v
+    sig76 = (i64, i64, i64) system_v
+    sig77 = (i64, i64, i64) system_v
+    sig78 = (i64, i64) system_v
+    sig79 = (i64) system_v
+    sig80 = (i64, i64) -> i8 system_v
+    sig81 = (i64, i64, i64) system_v
+    sig82 = (i64, i64) -> i8 system_v
+    sig83 = (i64, i64, i64) system_v
+    sig84 = (i64, i64, i64) system_v
+    sig85 = (i64, i64) system_v
+    sig86 = (i64) system_v
+    sig87 = (i64) system_v
+    sig88 = (i64) system_v
+    sig89 = (i64) system_v
+    sig90 = (i64) system_v
+    sig91 = (i64) system_v
+    sig92 = (i64) system_v
+    sig93 = (i64) system_v
+    sig94 = (i64) system_v
+    sig95 = (i64) system_v
+    sig96 = (i64) system_v
+    sig97 = (i64) system_v
+    sig98 = (i64) system_v
+    sig99 = (i64) system_v
+    sig100 = (i64) system_v
+    sig101 = (i64, i64, i64) system_v
+    sig102 = (i64) system_v
+    sig103 = (i64) system_v
+    sig104 = (i64, i64, i16, i64, i64, i64, i64, i64) system_v
+    sig105 = (i64) system_v
+    fn0 = u0:83 sig0
+    fn1 = u0:13 sig1
+    fn2 = u0:83 sig2
+    fn3 = u0:13 sig3
+    fn4 = u0:83 sig4
+    fn5 = u0:13 sig5
+    fn6 = u0:84 sig6
+    fn7 = u0:85 sig7
+    fn8 = u0:83 sig8
+    fn9 = u0:13 sig9
+    fn10 = u0:86 sig10     fn11 = u0:85 sig11     fn12 = u0:83 sig12     fn13 = u0:13 sig13
+    fn14 = u0:16 sig14     fn15 = u0:83 sig15     fn16 = u0:13 sig16
+    fn17 = u0:13 sig17
+    fn18 = u0:13 sig18
+    fn19 = u0:83 sig19     fn20 = u0:13 sig20
+    fn21 = u0:13 sig21
+    fn22 = u0:87 sig22     fn23 = u0:13 sig23
+    fn24 = u0:88 sig24     fn25 = u0:88 sig25     fn26 = u0:13 sig26
+    fn27 = u0:13 sig27
+    fn28 = u0:13 sig28
+    fn29 = u0:13 sig29
+    fn30 = u0:89 sig30     fn31 = u0:90 sig31     fn32 = u0:90 sig32     fn33 = u0:90 sig33     fn34 = u0:91 sig34     fn35 = u0:92 sig35     fn36 = u0:91 sig36     fn37 = u0:92 sig37     fn38 = u0:11 sig38     fn39 = u0:12 sig39     fn40 = u0:13 sig40
+    fn41 = u0:91 sig41     fn42 = u0:92 sig42     fn43 = u0:91 sig43     fn44 = u0:92 sig44     fn45 = u0:11 sig45     fn46 = u0:12 sig46     fn47 = u0:13 sig47
+    fn48 = u0:13 sig48
+    fn49 = u0:13 sig49
+    fn50 = u0:13 sig50
+    fn51 = u0:13 sig51
+    fn52 = u0:13 sig52
+    fn53 = u0:13 sig53
+    fn54 = u0:93 sig54     fn55 = u0:13 sig55
+    fn56 = u0:13 sig56
+    fn57 = u0:13 sig57
+    fn58 = u0:13 sig58
+    fn59 = u0:13 sig59
+    fn60 = u0:13 sig60
+    fn61 = u0:13 sig61
+    fn62 = u0:83 sig62     fn63 = u0:13 sig63
+    fn64 = u0:13 sig64
+    fn65 = u0:13 sig65
+    fn66 = u0:13 sig66
+    fn67 = u0:13 sig67
+    fn68 = u0:13 sig68
+    fn69 = u0:13 sig69
+    fn70 = u0:94 sig70     fn71 = u0:13 sig71
+    fn72 = u0:95 sig72     fn73 = u0:96 sig73     fn74 = u0:97 sig74     fn75 = u0:96 sig75     fn76 = u0:97 sig76     fn77 = u0:11 sig77     fn78 = u0:12 sig78     fn79 = u0:13 sig79
+    fn80 = u0:91 sig80     fn81 = u0:92 sig81     fn82 = u0:91 sig82     fn83 = u0:92 sig83     fn84 = u0:11 sig84     fn85 = u0:12 sig85     fn86 = u0:13 sig86
+    fn87 = u0:13 sig87
+    fn88 = u0:13 sig88
+    fn89 = u0:13 sig89
+    fn90 = u0:13 sig90
+    fn91 = u0:13 sig91
+    fn92 = u0:13 sig92
+    fn93 = u0:13 sig93
+    fn94 = u0:13 sig94
+    fn95 = u0:83 sig95     fn96 = u0:13 sig96
+    fn97 = u0:13 sig97
+    fn98 = u0:13 sig98
+    fn99 = u0:13 sig99
+    fn100 = u0:13 sig100
+    fn101 = u0:94 sig101
+    fn102 = u0:13 sig102
+    fn103 = u0:13 sig103
+    fn104 = u0:95 sig104
+
+ebb0(v0: i64, v1: i64, v2: i64):
+    v113 -> v1
+    v124 -> v1
+    v136 -> v1
+    v148 -> v1
+    v160 -> v1
+    v185 -> v1
+    v222 -> v1
+    v237 -> v1
+    v241 -> v1
+    v256 -> v1
+    v262 -> v1
+    v3 = stack_addr.i64 ss0
+    v4 = load.i64 aligned v2
+    store aligned v4, v3
+    v5 = load.i64 aligned v2+8
+    store aligned v5, v3+8
+    v6 = stack_addr.i64 ss1
+    v7 = stack_addr.i64 ss2
+    v8 = stack_addr.i64 ss3
+    v9 = stack_addr.i64 ss4
+    v10 = stack_addr.i64 ss5
+    v11 = stack_addr.i64 ss6
+    v12 = stack_addr.i64 ss7
+    v13 = stack_addr.i64 ss8
+    v14 = stack_addr.i64 ss9
+    v15 = stack_addr.i64 ss10
+    v16 = stack_addr.i64 ss11
+    v17 = stack_addr.i64 ss12
+    v18 = stack_addr.i64 ss13
+    v19 = stack_addr.i64 ss14
+    v20 = stack_addr.i64 ss15
+    v21 = stack_addr.i64 ss16
+    v22 = stack_addr.i64 ss17
+    v23 = stack_addr.i64 ss18
+    v24 = stack_addr.i64 ss19
+    v25 = stack_addr.i64 ss20
+    v26 = stack_addr.i64 ss21
+    v27 = stack_addr.i64 ss22
+    v28 = stack_addr.i64 ss23
+    v29 = stack_addr.i64 ss24
+    v30 = stack_addr.i64 ss25
+    v31 = stack_addr.i64 ss26
+    v32 = stack_addr.i64 ss27
+    v33 = stack_addr.i64 ss28
+    v34 = stack_addr.i64 ss29
+    v35 = stack_addr.i64 ss30
+    v36 = stack_addr.i64 ss31
+    v37 = stack_addr.i64 ss32
+    v38 = stack_addr.i64 ss33
+    v39 = stack_addr.i64 ss34
+    v40 = stack_addr.i64 ss35
+    v41 = stack_addr.i64 ss36
+    v42 = stack_addr.i64 ss37
+    v43 = stack_addr.i64 ss38
+    v44 = stack_addr.i64 ss39
+    v45 = stack_addr.i64 ss40
+    v46 = stack_addr.i64 ss41
+    v47 = stack_addr.i64 ss42
+    v48 = stack_addr.i64 ss43
+    v49 = stack_addr.i64 ss44
+    v50 = stack_addr.i64 ss45
+    v51 = stack_addr.i64 ss46
+    v52 = stack_addr.i64 ss47
+    v53 = stack_addr.i64 ss48
+    v54 = stack_addr.i64 ss49
+    v55 = stack_addr.i64 ss50
+    v56 = stack_addr.i64 ss51
+    v57 = stack_addr.i64 ss52
+    v58 = stack_addr.i64 ss53
+    v59 = stack_addr.i64 ss54
+    v60 = stack_addr.i64 ss55
+    v61 = stack_addr.i64 ss56
+    v62 = stack_addr.i64 ss57
+    v63 = stack_addr.i64 ss58
+    v64 = stack_addr.i64 ss59
+    v65 = stack_addr.i64 ss60
+    v66 = stack_addr.i64 ss61
+    v67 = stack_addr.i64 ss62
+    v68 = stack_addr.i64 ss63
+    v69 = stack_addr.i64 ss64
+    v70 = stack_addr.i64 ss65
+    v71 = stack_addr.i64 ss66
+    v72 = stack_addr.i64 ss67
+    v73 = stack_addr.i64 ss68
+    v74 = stack_addr.i64 ss69
+    v75 = stack_addr.i64 ss70
+    v76 = stack_addr.i64 ss71
+    v77 = stack_addr.i64 ss72
+    v78 = stack_addr.i64 ss73
+    v79 = stack_addr.i64 ss74
+    v80 = stack_addr.i64 ss75
+    v81 = stack_addr.i64 ss76
+    v82 = stack_addr.i64 ss77
+    v83 = stack_addr.i64 ss78
+    v84 = stack_addr.i64 ss79
+    v85 = stack_addr.i64 ss80
+    v86 = stack_addr.i64 ss81
+    v87 = stack_addr.i64 ss82
+    v88 = stack_addr.i64 ss83
+    v89 = stack_addr.i64 ss84
+    v90 = stack_addr.i64 ss85
+    v91 = stack_addr.i64 ss86
+    v92 = stack_addr.i64 ss87
+    v93 = stack_addr.i64 ss88
+    v94 = stack_addr.i64 ss89
+    v95 = stack_addr.i64 ss90
+    v96 = stack_addr.i64 ss91
+    v97 = stack_addr.i64 ss92
+    v98 = stack_addr.i64 ss93
+    v99 = stack_addr.i64 ss94
+    v100 = stack_addr.i64 ss95
+    v101 = stack_addr.i64 ss96
+    v102 = stack_addr.i64 ss97
+    v103 = stack_addr.i64 ss98
+    v104 = stack_addr.i64 ss99
+    v105 = stack_addr.i64 ss100
+    v106 = stack_addr.i64 ss101
+    v107 = stack_addr.i64 ss102
+    v108 = stack_addr.i64 ss103
+    v109 = stack_addr.i64 ss104
+    v110 = stack_addr.i64 ss105
+    v111 = stack_addr.i64 ss106
+    v112 = stack_addr.i64 ss107
+    jump ebb1
+
+ebb1:
+    v114 = load.i64 v113
+    v115 = iconst.i64 0
+    v116 = icmp ugt v114, v115
+    v117 = bint.i8 v116
+    v118 = uextend.i32 v117
+    v119 = icmp_imm eq v118, 0
+    v120 = bint.i8 v119
+    v121 = uextend.i32 v120
+    brz v121, ebb3
+    jump ebb2
+
+ebb2:
+    v122 = global_value.i64 gv0
+    v123 = global_value.i64 gv1
+    trap user65535
+
+ebb3:
+    v125 = iadd_imm.i64 v124, 8
+    v126 = load.i64 v125
+    v127 = iconst.i64 0
+    v128 = icmp ugt v126, v127
+    v129 = bint.i8 v128
+    v130 = uextend.i32 v129
+    v131 = icmp_imm eq v130, 0
+    v132 = bint.i8 v131
+    v133 = uextend.i32 v132
+    brz v133, ebb5
+    jump ebb4
+
+ebb4:
+    v134 = global_value.i64 gv2
+    v135 = global_value.i64 gv3
+    trap user65535
+
+ebb5:
+    v137 = iadd_imm.i64 v136, 16
+    v138 = load.i64 v137+42
+    v139 = iconst.i64 0
+    v140 = icmp ugt v138, v139
+    v141 = bint.i8 v140
+    v142 = uextend.i32 v141
+    v143 = icmp_imm eq v142, 0
+    v144 = bint.i8 v143
+    v145 = uextend.i32 v144
+    brz v145, ebb7
+    jump ebb6
+
+ebb6:
+    v146 = global_value.i64 gv4
+    v147 = global_value.i64 gv5
+    trap user65535
+
+ebb7:
+    v149 = load.i64 v148
+    v150 = iadd_imm.i64 v148, 16
+    v151 = load.i64 v150
+    call fn6(v7, v149, v151)
+    jump ebb8
+
+ebb8:
+    v152 = call fn7(v7)
+    jump ebb9
+
+ebb9:
+    v153 = load.i8 v6
+    v154 = uextend.i32 v153
+    v155 = icmp_imm eq v154, 0
+    v156 = bint.i8 v155
+    v157 = uextend.i32 v156
+    brz v157, ebb11
+    jump ebb10
+
+ebb10:
+    v158 = global_value.i64 gv6
+    v159 = global_value.i64 gv7
+    trap user65535
+
+ebb11:
+    v161 = load.i64 v160
+    v162 = iadd_imm.i64 v160, 8
+    v163 = load.i64 v162
+    call fn10(v9, v161, v163)
+    jump ebb12
+
+ebb12:
+    v164 = call fn11(v9)
+    jump ebb13
+
+ebb13:
+    v165 = load.i8 v8
+    v166 = uextend.i32 v165
+    v167 = icmp_imm eq v166, 0
+    v168 = bint.i8 v167
+    v169 = uextend.i32 v168
+    brz v169, ebb15
+    jump ebb14
+
+ebb14:
+    v170 = global_value.i64 gv8
+    v171 = global_value.i64 gv9
+    trap user65535
+
+ebb15:
+    v172 = load.i64 aligned v3
+    v173 = load.i64 aligned v3+8
+    v174 = call fn14(v11)
+    jump ebb16
+
+ebb16:
+    v175 = iconst.i64 17
+    v176 = load.i64 v10
+    v177 = icmp uge v176, v175
+    v178 = bint.i8 v177
+    v179 = uextend.i32 v178
+    v180 = icmp_imm eq v179, 0
+    v181 = bint.i8 v180
+    v182 = uextend.i32 v181
+    brz v182, ebb18
+    jump ebb17
+
+ebb17:
+    v183 = global_value.i64 gv10
+    v184 = global_value.i64 gv11
+    trap user65535
+
+ebb18:
+    v186 = load.i64 v185
+    v187 = iadd_imm.i64 v185, 16
+    v188 = load.i64 v187
+    v189 = iadd v186, v188
+    v190 = iconst.i8 0
+    v191 = stack_addr.i64 ss108
+    v192 = stack_addr.i64 ss108
+    v193 = load.i64 aligned v192
+    v194 = load.i64 aligned v192+8
+    v195 = iadd_imm.i64 v12, 8
+    v196 = load.i8 v195
+    v197 = uextend.i32 v196
+    brz v197, ebb19
+    v198 = global_value.i64 gv12
+    trap user0
+
+ebb19:
+    v199 = load.i64 v12
+        v213 -> v199
+    v200 = iconst.i64 1
+    v201 = iconst.i32 61
+    v202 = ishl v200, v201
+    v203 = iconst.i8 0
+    v204 = stack_addr.i64 ss109
+    v205 = stack_addr.i64 ss109
+    v206 = load.i64 aligned v205
+    v207 = load.i64 aligned v205+8
+    v208 = iadd_imm.i64 v13, 8
+    v209 = load.i8 v208
+    v210 = uextend.i32 v209
+    brz v210, ebb20
+    v211 = global_value.i64 gv13
+    trap user0
+
+ebb20:
+    v212 = load.i64 v13
+    v214 = icmp.i64 ult v213, v212
+    v215 = bint.i8 v214
+    v216 = uextend.i32 v215
+    v217 = icmp_imm eq v216, 0
+    v218 = bint.i8 v217
+    v219 = uextend.i32 v218
+    brz v219, ebb22
+    jump ebb21
+
+ebb21:
+    v220 = global_value.i64 gv14
+    v221 = global_value.i64 gv15
+    trap user65535
+
+ebb22:
+    v223 = load.i64 v222
+    v224 = iadd_imm.i64 v222, 16
+    v225 = load.i64 v224
+    v226 = iadd v223, v225
+    v227 = iconst.i8 0
+    v228 = stack_addr.i64 ss110
+    v229 = stack_addr.i64 ss110
+    v230 = load.i64 aligned v229
+    v231 = load.i64 aligned v229+8
+    v232 = iadd_imm.i64 v16, 8
+    v233 = load.i8 v232
+    v234 = uextend.i32 v233
+    brz v234, ebb23
+    v235 = global_value.i64 gv16
+    trap user0
+
+ebb23:
+    v236 = load.i64 v16
+    v238 = iadd_imm.i64 v237, 24
+    v239 = load.i16 v238
+    v240 = iadd_imm.i64 v15, 8
+    call fn22(v14, v15)
+    jump ebb24
+
+ebb24:
+    v242 = load.i64 v241
+    v243 = iadd_imm.i64 v241, 8
+    v244 = load.i64 v243
+    v245 = isub v242, v244
+    v246 = iconst.i8 0
+    v247 = stack_addr.i64 ss111
+    v248 = stack_addr.i64 ss111
+    v249 = load.i64 aligned v248
+    v250 = load.i64 aligned v248+8
+    v251 = iadd_imm.i64 v19, 8
+    v252 = load.i8 v251
+    v253 = uextend.i32 v252
+    brz v253, ebb25
+    v254 = global_value.i64 gv17
+    trap user0
+
+ebb25:
+    v255 = load.i64 v19
+    v257 = iadd_imm.i64 v256, 24
+    v258 = load.i16 v257
+    v259 = iadd_imm.i64 v18, 8
+    v260 = iadd_imm.i64 v14, 8
+    v261 = load.i16 v260
+    call fn24(v17, v18, v261)
+    jump ebb26
+
+ebb26:
+    v263 = load.i64 v262
+    v264 = iadd_imm.i64 v262, 24
+    v265 = load.i16 v264
+    v266 = iadd_imm.i64 v21, 8
+    v267 = iadd_imm.i64 v14, 8
+    v268 = load.i16 v267
+    call fn25(v20, v21, v268)
+    jump ebb27
+
+ebb27:
+    v269 = iadd_imm.i64 v14, 8
+    v270 = load.i16 v269
+    v271 = iconst.i16 -60
+    v272 = isub v271, v270
+    v273 = iconst.i8 0
+    v274 = stack_addr.i64 ss112
+    v275 = stack_addr.i64 ss112
+    v276 = load.i32 aligned v275
+    v277 = iadd_imm.i64 v24, 2
+    v278 = load.i8 v277
+    v279 = uextend.i32 v278
+    brz v279, ebb28
+    v280 = global_value.i64 gv18
+    trap user0
+
+ebb28:
+    v281 = load.i16 v24
+    v282 = iconst.i16 64
+    v283 = isub v281, v282
+    v284 = iconst.i8 0
+    v285 = stack_addr.i64 ss113
+    v286 = stack_addr.i64 ss113
+    v287 = load.i32 aligned v286
+    v288 = iadd_imm.i64 v25, 2
+    v289 = load.i8 v288
+    v290 = uextend.i32 v289
+    brz v290, ebb29
+    v291 = global_value.i64 gv19
+    trap user0
+
+ebb29:
+    v292 = load.i16 v25
+        v317 -> v292
+    v293 = iadd_imm.i64 v14, 8
+    v294 = load.i16 v293
+    v295 = iconst.i16 -32
+    v296 = isub v295, v294
+    v297 = iconst.i8 0
+    v298 = stack_addr.i64 ss114
+    v299 = stack_addr.i64 ss114
+    v300 = load.i32 aligned v299
+    v301 = iadd_imm.i64 v26, 2
+    v302 = load.i8 v301
+    v303 = uextend.i32 v302
+    brz v303, ebb30
+    v304 = global_value.i64 gv20
+    trap user0
+
+ebb30:
+    v305 = load.i16 v26
+    v306 = iconst.i16 64
+    v307 = isub v305, v306
+    v308 = iconst.i8 0
+    v309 = stack_addr.i64 ss115
+    v310 = stack_addr.i64 ss115
+    v311 = load.i32 aligned v310
+    v312 = iadd_imm.i64 v27, 2
+    v313 = load.i8 v312
+    v314 = uextend.i32 v313
+    brz v314, ebb31
+    v315 = global_value.i64 gv21
+    trap user0
+
+ebb31:
+    v316 = load.i16 v27
+    call fn30(v23, v317, v316)
+    jump ebb32
+
+ebb32:
+    v318 = load.i16 v23
+        v1007 -> v318
+    v319 = iadd_imm.i64 v23, 8
+    v320 = load.i64 aligned v319
+    v321 = load.i64 aligned v319+8
+    call fn31(v28, v14, v22)
+    jump ebb33
+
+ebb33:
+    call fn32(v29, v17, v22)
+    jump ebb34
+
+ebb34:
+    call fn33(v30, v20, v22)
+    jump ebb35
+
+ebb35:
+    v322 = iconst.i8 1
+    v323 = uextend.i32 v322
+    brz v323, ebb42
+    jump ebb36
+
+ebb36:
+    v324 = iadd_imm.i64 v28, 8
+    v325 = iadd_imm.i64 v29, 8
+    v326 = iadd_imm.i64 v31, 8
+    v327 = load.i64 v31
+        v340 -> v327
+    v328 = iadd_imm.i64 v31, 8
+    v329 = load.i64 v328
+        v341 -> v329
+    v330 = load.i16 v327
+    v331 = load.i16 v329
+    v332 = icmp eq v330, v331
+    v333 = bint.i8 v332
+    v334 = uextend.i32 v333
+    v335 = icmp_imm eq v334, 0
+    v336 = bint.i8 v335
+    v337 = uextend.i32 v336
+    brz v337, ebb38
+    jump ebb37
+
+ebb37:
+    v338 = global_value.i64 gv22
+    v339 = iconst.i64 3
+    v342 = iadd_imm.i64 v36, 8
+    v343 = load.i64 v36
+    v344 = iadd_imm.i64 v36, 8
+    v345 = load.i64 v344
+        v347 -> v345
+    v346 = func_addr.i64 fn34
+    call fn35(v39, v343, v346)
+    jump ebb39
+
+ebb38:
+    jump ebb42
+
+ebb39:
+    v348 = func_addr.i64 fn36
+    call fn37(v40, v347, v348)
+    jump ebb40
+
+ebb40:
+    v349 = iconst.i64 0
+    v350 = imul_imm v349, 16
+    v351 = iadd.i64 v35, v350
+    v352 = load.i64 aligned v39
+    v353 = load.i64 aligned v39+8
+    v354 = iconst.i64 1
+    v355 = imul_imm v354, 16
+    v356 = iadd.i64 v35, v355
+    v357 = load.i64 aligned v40
+    v358 = load.i64 aligned v40+8
+    v359 = iconst.i64 2
+    call fn38(v32, v33, v34)
+    jump ebb41
+
+ebb41:
+    v360 = global_value.i64 gv23
+    call fn39(v32, v360)
+    v361 = global_value.i64 gv24
+    trap user65535
+
+ebb42:
+    v362 = iconst.i8 1
+    v363 = uextend.i32 v362
+    brz v363, ebb49(v1007)
+    jump ebb43
+
+ebb43:
+    v364 = iadd_imm.i64 v28, 8
+    v365 = iadd_imm.i64 v30, 8
+    v366 = iadd_imm.i64 v41, 8
+    v367 = load.i64 v41
+        v380 -> v367
+    v368 = iadd_imm.i64 v41, 8
+    v369 = load.i64 v368
+        v381 -> v369
+    v370 = load.i16 v367
+    v371 = load.i16 v369
+    v372 = icmp eq v370, v371
+    v373 = bint.i8 v372
+    v374 = uextend.i32 v373
+    v375 = icmp_imm eq v374, 0
+    v376 = bint.i8 v375
+    v377 = uextend.i32 v376
+    brz v377, ebb45
+    jump ebb44
+
+ebb44:
+    v378 = global_value.i64 gv25
+    v379 = iconst.i64 3
+    v382 = iadd_imm.i64 v46, 8
+    v383 = load.i64 v46
+    v384 = iadd_imm.i64 v46, 8
+    v385 = load.i64 v384
+        v387 -> v385
+    v386 = func_addr.i64 fn41
+    call fn42(v49, v383, v386)
+    jump ebb46
+
+ebb45:
+    jump ebb49(v1007)
+
+ebb46:
+    v388 = func_addr.i64 fn43
+    call fn44(v50, v387, v388)
+    jump ebb47
+
+ebb47:
+    v389 = iconst.i64 0
+    v390 = imul_imm v389, 16
+    v391 = iadd.i64 v45, v390
+    v392 = load.i64 aligned v49
+    v393 = load.i64 aligned v49+8
+    v394 = iconst.i64 1
+    v395 = imul_imm v394, 16
+    v396 = iadd.i64 v45, v395
+    v397 = load.i64 aligned v50
+    v398 = load.i64 aligned v50+8
+    v399 = iconst.i64 2
+    call fn45(v42, v43, v44)
+    jump ebb48
+
+ebb48:
+    v400 = global_value.i64 gv26
+    call fn46(v42, v400)
+    v401 = global_value.i64 gv27
+    trap user65535
+
+ebb49(v1006: i16):
+        v486 -> v1006
+    v402 = load.i64 v28
+    v403 = iconst.i64 1
+    v404 = iadd v402, v403
+    v405 = iconst.i8 0
+    v406 = stack_addr.i64 ss116
+    v407 = stack_addr.i64 ss116
+    v408 = load.i64 aligned v407
+    v409 = load.i64 aligned v407+8
+    v410 = iadd_imm.i64 v51, 8
+    v411 = load.i8 v410
+    v412 = uextend.i32 v411
+    brz v412, ebb50
+    v413 = global_value.i64 gv28
+    trap user0
+
+ebb50:
+    v414 = load.i64 v51
+        v439 -> v414
+        v452 -> v414
+        v478 -> v414
+        v508 -> v414
+    v415 = load.i64 v29
+    v416 = iconst.i64 1
+    v417 = isub v415, v416
+    v418 = iconst.i8 0
+    v419 = stack_addr.i64 ss117
+    v420 = stack_addr.i64 ss117
+    v421 = load.i64 aligned v420
+    v422 = load.i64 aligned v420+8
+    v423 = iadd_imm.i64 v52, 8
+    v424 = load.i8 v423
+    v425 = uextend.i32 v424
+    brz v425, ebb51
+    v426 = global_value.i64 gv29
+    trap user0
+
+ebb51:
+    v427 = load.i64 v52
+        v509 -> v427
+    v428 = iadd_imm.i64 v28, 8
+    v429 = load.i16 v428
+        v435 -> v429
+    v430 = iconst.i16 0xffff_ffff_ffff_8000
+    v431 = icmp eq v429, v430
+    v432 = bint.i8 v431
+    v433 = uextend.i32 v432
+    brz v433, ebb52
+    v434 = global_value.i64 gv30
+    trap user0
+
+ebb52:
+    v436 = iconst.i16 0
+    v437 = isub v436, v435
+    v438 = sextend.i64 v437
+        v453 -> v438
+        v521 -> v438
+    v440 = ushr.i64 v439, v438
+    v441 = iconst.i8 0
+    v442 = stack_addr.i64 ss118
+    v443 = stack_addr.i64 ss118
+    v444 = load.i64 aligned v443
+    v445 = load.i64 aligned v443+8
+    v446 = iadd_imm.i64 v53, 8
+    v447 = load.i8 v446
+    v448 = uextend.i32 v447
+    brz v448, ebb53
+    v449 = global_value.i64 gv31
+    trap user0
+
+ebb53:
+    v450 = load.i64 v53
+    v451 = ireduce.i32 v450
+        v480 -> v451
+        v551 -> v451
+    v454 = iconst.i64 1
+    v455 = ishl v454, v453
+    v456 = iconst.i8 0
+    v457 = stack_addr.i64 ss119
+    v458 = stack_addr.i64 ss119
+    v459 = load.i64 aligned v458
+    v460 = load.i64 aligned v458+8
+    v461 = iadd_imm.i64 v54, 8
+    v462 = load.i8 v461
+    v463 = uextend.i32 v462
+    brz v463, ebb54
+    v464 = global_value.i64 gv32
+    trap user0
+
+ebb54:
+    v465 = load.i64 v54
+    v466 = iconst.i64 1
+    v467 = isub v465, v466
+    v468 = iconst.i8 0
+    v469 = stack_addr.i64 ss120
+    v470 = stack_addr.i64 ss120
+    v471 = load.i64 aligned v470
+    v472 = load.i64 aligned v470+8
+    v473 = iadd_imm.i64 v55, 8
+    v474 = load.i8 v473
+    v475 = uextend.i32 v474
+    brz v475, ebb55
+    v476 = global_value.i64 gv33
+    trap user0
+
+ebb55:
+    v477 = load.i64 v55
+    v479 = band.i64 v478, v477
+    call fn54(v56, v480)
+    jump ebb56
+
+ebb56:
+    v481 = load.i8 v56
+        v548 -> v481
+    v482 = iadd_imm.i64 v56, 4
+    v483 = load.i32 v482
+        v550 -> v483
+    v484 = iconst.i64 0
+    v485 = uextend.i16 v481
+    v487 = isub v485, v486
+    v488 = iconst.i8 0
+    v489 = stack_addr.i64 ss121
+    v490 = stack_addr.i64 ss121
+    v491 = load.i32 aligned v490
+    v492 = iadd_imm.i64 v57, 2
+    v493 = load.i8 v492
+    v494 = uextend.i32 v493
+    brz v494, ebb57
+    v495 = global_value.i64 gv34
+    trap user0
+
+ebb57:
+    v496 = load.i16 v57
+    v497 = iconst.i16 1
+    v498 = iadd v496, v497
+    v499 = iconst.i8 0
+    v500 = stack_addr.i64 ss122
+    v501 = stack_addr.i64 ss122
+    v502 = load.i32 aligned v501
+    v503 = iadd_imm.i64 v58, 2
+    v504 = load.i8 v503
+    v505 = uextend.i32 v504
+    brz v505, ebb58
+    v506 = global_value.i64 gv35
+    trap user0
+
+ebb58:
+    v507 = load.i16 v58
+    v510 = isub.i64 v508, v509
+    v511 = iconst.i8 0
+    v512 = stack_addr.i64 ss123
+    v513 = stack_addr.i64 ss123
+    v514 = load.i64 aligned v513
+    v515 = load.i64 aligned v513+8
+    v516 = iadd_imm.i64 v59, 8
+    v517 = load.i8 v516
+    v518 = uextend.i32 v517
+    brz v518, ebb59
+    v519 = global_value.i64 gv36
+    trap user0
+
+ebb59:
+    v520 = load.i64 v59
+        v546 -> v520
+    v522 = iconst.i64 1
+    v523 = ishl v522, v521
+    v524 = iconst.i8 0
+    v525 = stack_addr.i64 ss124
+    v526 = stack_addr.i64 ss124
+    v527 = load.i64 aligned v526
+    v528 = load.i64 aligned v526+8
+    v529 = iadd_imm.i64 v60, 8
+    v530 = load.i8 v529
+    v531 = uextend.i32 v530
+    brz v531, ebb60
+    v532 = global_value.i64 gv37
+    trap user0
+
+ebb60:
+    v533 = load.i64 v60
+    v534 = iconst.i64 1
+    v535 = isub v533, v534
+    v536 = iconst.i8 0
+    v537 = stack_addr.i64 ss125
+    v538 = stack_addr.i64 ss125
+    v539 = load.i64 aligned v538
+    v540 = load.i64 aligned v538+8
+    v541 = iadd_imm.i64 v61, 8
+    v542 = load.i8 v541
+    v543 = uextend.i32 v542
+    brz v543, ebb61
+    v544 = global_value.i64 gv38
+    trap user0
+
+ebb61:
+    v545 = load.i64 v61
+    v547 = band.i64 v546, v545
+    v549 = uextend.i16 v548
+    jump ebb62(v551, v484, v521, v479, v520, v507, v508, v548, v547)
+
+ebb62(v552: i32, v1009: i64, v1013: i64, v1016: i64, v1019: i64, v1022: i16, v1025: i64, v1028: i8, v1033: i64):
+        v559 -> v552
+        v562 -> v552
+        v569 -> v552
+        v596 -> v1009
+        v605 -> v1009
+        v609 -> v1009
+        v1008 -> v1009
+        v624 -> v1013
+        v654 -> v1013
+        v1012 -> v1013
+        v1014 -> v1013
+        v1041 -> v1013
+        v636 -> v1016
+        v1015 -> v1016
+        v1017 -> v1016
+        v1030 -> v1016
+        v648 -> v1019
+        v676 -> v1019
+        v693 -> v1019
+        v1018 -> v1019
+        v1020 -> v1019
+        v674 -> v1022
+        v691 -> v1022
+        v1021 -> v1022
+        v1023 -> v1022
+        v1054 -> v1022
+        v677 -> v1025
+        v1024 -> v1025
+        v1026 -> v1025
+        v1059 -> v1025
+        v696 -> v1028
+        v1027 -> v1028
+        v1029 -> v1028
+        v1031 -> v1033
+        v1032 -> v1033
+        v1034 -> v1033
+    v553 = load.i32 v63
+        v560 -> v553
+    v554 = iconst.i32 0
+    v555 = icmp eq v553, v554
+    v556 = bint.i8 v555
+    v557 = uextend.i32 v556
+    brz v557, ebb63
+    v558 = global_value.i64 gv39
+    trap user0
+
+ebb63:
+    v561 = udiv.i32 v559, v560
+        v574 -> v561
+    v563 = load.i32 v63
+        v570 -> v563
+    v564 = iconst.i32 0
+    v565 = icmp eq v563, v564
+    v566 = bint.i8 v565
+    v567 = uextend.i32 v566
+    brz v567, ebb64
+    v568 = global_value.i64 gv40
+    trap user0
+
+ebb64:
+    v571 = urem.i32 v569, v570
+        v622 -> v571
+        v803 -> v571
+        v1011 -> v571
+    v572 = iconst.i8 1
+    v573 = uextend.i32 v572
+    brz v573, ebb68(v561)
+    jump ebb65
+
+ebb65:
+    v575 = iconst.i32 10
+    v576 = icmp.i32 ult v574, v575
+    v577 = bint.i8 v576
+    v578 = uextend.i32 v577
+    v579 = icmp_imm eq v578, 0
+    v580 = bint.i8 v579
+    v581 = uextend.i32 v580
+    brz v581, ebb67
+    jump ebb66
+
+ebb66:
+    v582 = global_value.i64 gv41
+    v583 = global_value.i64 gv42
+    trap user65535
+
+ebb67:
+    jump ebb68(v574)
+
+ebb68(v584: i32):
+    v585 = ireduce.i8 v584
+    v586 = iconst.i8 48
+    v587 = iadd v586, v585
+    v588 = iconst.i8 0
+    v589 = stack_addr.i64 ss126
+    v590 = stack_addr.i64 ss126
+    v591 = load.i16 aligned v590
+    v592 = iadd_imm.i64 v64, 1
+    v593 = load.i8 v592
+    v594 = uextend.i32 v593
+    brz v594, ebb69
+    v595 = global_value.i64 gv43
+    trap user0
+
+ebb69:
+    v597 = load.i64 v3
+    v598 = load.i64 v3+8
+    v599 = icmp.i64 ult v596, v598
+    v600 = bint.i8 v599
+    v601 = uextend.i32 v600
+    brnz v601, ebb70
+    v602 = global_value.i64 gv44
+    trap user0
+
+ebb70:
+    v603 = load.i64 v3
+    v604 = load.i64 v3+8
+    v606 = imul_imm.i64 v605, 1
+    v607 = iadd v603, v606
+    v608 = load.i8 aligned v64
+    v610 = iconst.i64 1
+    v611 = iadd.i64 v609, v610
+    v612 = iconst.i8 0
+    v613 = stack_addr.i64 ss127
+    v614 = stack_addr.i64 ss127
+    v615 = load.i64 aligned v614
+    v616 = load.i64 aligned v614+8
+    v617 = iadd_imm.i64 v65, 8
+    v618 = load.i8 v617
+    v619 = uextend.i32 v618
+    brz v619, ebb71
+    v620 = global_value.i64 gv45
+    trap user0
+
+ebb71:
+    v621 = load.i64 v65
+        v668 -> v621
+        v695 -> v621
+        v1010 -> v621
+        v1046 -> v621
+    v623 = uextend.i64 v622
+    v625 = ishl v623, v624
+    v626 = iconst.i8 0
+    v627 = stack_addr.i64 ss128
+    v628 = stack_addr.i64 ss128
+    v629 = load.i64 aligned v628
+    v630 = load.i64 aligned v628+8
+    v631 = iadd_imm.i64 v66, 8
+    v632 = load.i8 v631
+    v633 = uextend.i32 v632
+    brz v633, ebb72
+    v634 = global_value.i64 gv46
+    trap user0
+
+ebb72:
+    v635 = load.i64 v66
+    v637 = iadd v635, v636
+    v638 = iconst.i8 0
+    v639 = stack_addr.i64 ss129
+    v640 = stack_addr.i64 ss129
+    v641 = load.i64 aligned v640
+    v642 = load.i64 aligned v640+8
+    v643 = iadd_imm.i64 v67, 8
+    v644 = load.i8 v643
+    v645 = uextend.i32 v644
+    brz v645, ebb73
+    v646 = global_value.i64 gv47
+    trap user0
+
+ebb73:
+    v647 = load.i64 v67
+        v675 -> v647
+        v692 -> v647
+    v649 = icmp ult v647, v648
+    v650 = bint.i8 v649
+    v651 = uextend.i32 v650
+    brz v651, ebb80
+    jump ebb74
+
+ebb74:
+    v652 = load.i32 v63
+    v653 = uextend.i64 v652
+    v655 = ishl v653, v654
+    v656 = iconst.i8 0
+    v657 = stack_addr.i64 ss130
+    v658 = stack_addr.i64 ss130
+    v659 = load.i64 aligned v658
+    v660 = load.i64 aligned v658+8
+    v661 = iadd_imm.i64 v68, 8
+    v662 = load.i8 v661
+    v663 = uextend.i32 v662
+    brz v663, ebb75
+    v664 = global_value.i64 gv48
+    trap user0
+
+ebb75:
+    v665 = load.i64 v68
+        v690 -> v665
+    v666 = load.i64 aligned v3
+    v667 = load.i64 aligned v3+8
+    v669 = load.i64 v73
+    call fn70(v71, v72, v669)
+    jump ebb76
+
+ebb76:
+    v670 = load.i64 aligned v71
+    v671 = load.i64 aligned v71+8
+    v672 = load.i64 aligned v70
+    v673 = load.i64 aligned v70+8
+    v678 = load.i64 v30
+    v679 = isub.i64 v677, v678
+    v680 = iconst.i8 0
+    v681 = stack_addr.i64 ss131
+    v682 = stack_addr.i64 ss131
+    v683 = load.i64 aligned v682
+    v684 = load.i64 aligned v682+8
+    v685 = iadd_imm.i64 v74, 8
+    v686 = load.i8 v685
+    v687 = uextend.i32 v686
+    brz v687, ebb77
+    v688 = global_value.i64 gv49
+    trap user0
+
+ebb77:
+    v689 = load.i64 v74
+    v694 = iconst.i64 1
+    call fn72(v0, v69, v691, v692, v693, v689, v690, v694)
+    jump ebb78
+
+ebb78:
+    jump ebb79
+
+ebb79:
+    return
+
+ebb80:
+    v697 = uextend.i64 v696
+    v698 = icmp.i64 ugt v695, v697
+    v699 = bint.i8 v698
+    v700 = uextend.i32 v699
+    brz v700, ebb96
+    jump ebb81
+
+ebb81:
+    v701 = iconst.i8 1
+    v702 = uextend.i32 v701
+    brz v702, ebb88
+    jump ebb82
+
+ebb82:
+    v703 = global_value.i64 gv50
+    v704 = iadd_imm.i64 v75, 8
+    v705 = load.i64 v75
+        v718 -> v705
+    v706 = iadd_imm.i64 v75, 8
+    v707 = load.i64 v706
+        v719 -> v707
+    v708 = load.i32 v705
+    v709 = load.i32 v707
+    v710 = icmp eq v708, v709
+    v711 = bint.i8 v710
+    v712 = uextend.i32 v711
+    v713 = icmp_imm eq v712, 0
+    v714 = bint.i8 v713
+    v715 = uextend.i32 v714
+    brz v715, ebb84
+    jump ebb83
+
+ebb83:
+    v716 = global_value.i64 gv51
+    v717 = iconst.i64 3
+    v720 = iadd_imm.i64 v80, 8
+    v721 = load.i64 v80
+    v722 = iadd_imm.i64 v80, 8
+    v723 = load.i64 v722
+        v725 -> v723
+    v724 = func_addr.i64 fn73
+    call fn74(v83, v721, v724)
+    jump ebb85
+
+ebb84:
+    jump ebb88
+
+ebb85:
+    v726 = func_addr.i64 fn75
+    call fn76(v84, v725, v726)
+    jump ebb86
+
+ebb86:
+    v727 = iconst.i64 0
+    v728 = imul_imm v727, 16
+    v729 = iadd.i64 v79, v728
+    v730 = load.i64 aligned v83
+    v731 = load.i64 aligned v83+8
+    v732 = iconst.i64 1
+    v733 = imul_imm v732, 16
+    v734 = iadd.i64 v79, v733
+    v735 = load.i64 aligned v84
+    v736 = load.i64 aligned v84+8
+    v737 = iconst.i64 2
+    call fn77(v76, v77, v78)
+    jump ebb87
+
+ebb87:
+    v738 = global_value.i64 gv52
+    call fn78(v76, v738)
+    v739 = global_value.i64 gv53
+    trap user65535
+
+ebb88:
+    v740 = iconst.i8 1
+    v741 = uextend.i32 v740
+    brz v741, ebb95(v1030, v1031, v1041, v1046, v1054, v1059)
+    jump ebb89
+
+ebb89:
+    v742 = global_value.i64 gv54
+    v743 = iadd_imm.i64 v85, 8
+    v744 = load.i64 v85
+        v757 -> v744
+    v745 = iadd_imm.i64 v85, 8
+    v746 = load.i64 v745
+        v758 -> v746
+    v747 = load.i16 v744
+    v748 = load.i16 v746
+    v749 = icmp eq v747, v748
+    v750 = bint.i8 v749
+    v751 = uextend.i32 v750
+    v752 = icmp_imm eq v751, 0
+    v753 = bint.i8 v752
+    v754 = uextend.i32 v753
+    brz v754, ebb91
+    jump ebb90
+
+ebb90:
+    v755 = global_value.i64 gv55
+    v756 = iconst.i64 3
+    v759 = iadd_imm.i64 v90, 8
+    v760 = load.i64 v90
+    v761 = iadd_imm.i64 v90, 8
+    v762 = load.i64 v761
+        v764 -> v762
+    v763 = func_addr.i64 fn80
+    call fn81(v93, v760, v763)
+    jump ebb92
+
+ebb91:
+    jump ebb95(v1030, v1031, v1041, v1046, v1054, v1059)
+
+ebb92:
+    v765 = func_addr.i64 fn82
+    call fn83(v94, v764, v765)
+    jump ebb93
+
+ebb93:
+    v766 = iconst.i64 0
+    v767 = imul_imm v766, 16
+    v768 = iadd.i64 v89, v767
+    v769 = load.i64 aligned v93
+    v770 = load.i64 aligned v93+8
+    v771 = iconst.i64 1
+    v772 = imul_imm v771, 16
+    v773 = iadd.i64 v89, v772
+    v774 = load.i64 aligned v94
+    v775 = load.i64 aligned v94+8
+    v776 = iconst.i64 2
+    call fn84(v86, v87, v88)
+    jump ebb94
+
+ebb94:
+    v777 = global_value.i64 gv56
+    call fn85(v86, v777)
+    v778 = global_value.i64 gv57
+    trap user65535
+
+ebb95(v779: i64, v780: i64, v1040: i64, v1045: i64, v1053: i16, v1058: i64):
+    v781 = iconst.i64 1
+    jump ebb99(v779, v780, v781, v1040, v1045, v1053, v1058)
+
+ebb96:
+    v782 = iconst.i16 1
+    v783 = load.i16 v62
+    v784 = isub v783, v782
+    v785 = iconst.i8 0
+    v786 = stack_addr.i64 ss132
+    v787 = stack_addr.i64 ss132
+    v788 = load.i32 aligned v787
+    v789 = iadd_imm.i64 v95, 2
+    v790 = load.i8 v789
+    v791 = uextend.i32 v790
+    brz v791, ebb97
+    v792 = global_value.i64 gv58
+    trap user0
+
+ebb97:
+    v793 = load.i16 aligned v95
+    v794 = iconst.i32 10
+    v795 = iconst.i32 0
+    v796 = icmp eq v794, v795
+    v797 = bint.i8 v796
+    v798 = uextend.i32 v797
+    brz v798, ebb98
+    v799 = global_value.i64 gv59
+    trap user0
+
+ebb98:
+    v800 = iconst.i32 10
+    v801 = load.i32 v63
+    v802 = udiv v801, v800
+    jump ebb62(v803, v1010, v1014, v1017, v1020, v1023, v1026, v1029, v1034)
+
+ebb99(v804: i64, v1035: i64, v1037: i64, v1039: i64, v1044: i64, v1052: i16, v1057: i64):
+        v817 -> v1035
+        v830 -> v1037
+        v844 -> v1039
+        v857 -> v1039
+        v939 -> v1039
+        v1042 -> v1039
+        v1050 -> v1039
+        v908 -> v1044
+        v917 -> v1044
+        v921 -> v1044
+        v1043 -> v1044
+        v960 -> v1052
+        v990 -> v1052
+        v1051 -> v1052
+        v1055 -> v1052
+        v963 -> v1057
+        v1056 -> v1057
+        v1060 -> v1057
+    v805 = iconst.i64 10
+    v806 = imul v804, v805
+    v807 = iconst.i8 0
+    v808 = stack_addr.i64 ss133
+    v809 = stack_addr.i64 ss133
+    v810 = load.i64 aligned v809
+    v811 = load.i64 aligned v809+8
+    v812 = iadd_imm.i64 v96, 8
+    v813 = load.i8 v812
+    v814 = uextend.i32 v813
+    brz v814, ebb100
+    v815 = global_value.i64 gv60
+    trap user0
+
+ebb100:
+    v816 = load.i64 v96
+        v843 -> v816
+        v856 -> v816
+        v882 -> v816
+    v818 = iconst.i64 10
+    v819 = imul.i64 v817, v818
+    v820 = iconst.i8 0
+    v821 = stack_addr.i64 ss134
+    v822 = stack_addr.i64 ss134
+    v823 = load.i64 aligned v822
+    v824 = load.i64 aligned v822+8
+    v825 = iadd_imm.i64 v97, 8
+    v826 = load.i8 v825
+    v827 = uextend.i32 v826
+    brz v827, ebb101
+    v828 = global_value.i64 gv61
+    trap user0
+
+ebb101:
+    v829 = load.i64 v97
+        v935 -> v829
+        v962 -> v829
+        v992 -> v829
+        v1036 -> v829
+        v1049 -> v829
+    v831 = iconst.i64 10
+    v832 = imul.i64 v830, v831
+    v833 = iconst.i8 0
+    v834 = stack_addr.i64 ss135
+    v835 = stack_addr.i64 ss135
+    v836 = load.i64 aligned v835
+    v837 = load.i64 aligned v835+8
+    v838 = iadd_imm.i64 v98, 8
+    v839 = load.i8 v838
+    v840 = uextend.i32 v839
+    brz v840, ebb102
+    v841 = global_value.i64 gv62
+    trap user0
+
+ebb102:
+    v842 = load.i64 v98
+        v976 -> v842
+        v989 -> v842
+        v1038 -> v842
+        v1061 -> v842
+    v845 = ushr.i64 v843, v844
+    v846 = iconst.i8 0
+    v847 = stack_addr.i64 ss136
+    v848 = stack_addr.i64 ss136
+    v849 = load.i64 aligned v848
+    v850 = load.i64 aligned v848+8
+    v851 = iadd_imm.i64 v99, 8
+    v852 = load.i8 v851
+    v853 = uextend.i32 v852
+    brz v853, ebb103
+    v854 = global_value.i64 gv63
+    trap user0
+
+ebb103:
+    v855 = load.i64 v99
+        v886 -> v855
+    v858 = iconst.i64 1
+    v859 = ishl v858, v857
+    v860 = iconst.i8 0
+    v861 = stack_addr.i64 ss137
+    v862 = stack_addr.i64 ss137
+    v863 = load.i64 aligned v862
+    v864 = load.i64 aligned v862+8
+    v865 = iadd_imm.i64 v100, 8
+    v866 = load.i8 v865
+    v867 = uextend.i32 v866
+    brz v867, ebb104
+    v868 = global_value.i64 gv64
+    trap user0
+
+ebb104:
+    v869 = load.i64 v100
+    v870 = iconst.i64 1
+    v871 = isub v869, v870
+    v872 = iconst.i8 0
+    v873 = stack_addr.i64 ss138
+    v874 = stack_addr.i64 ss138
+    v875 = load.i64 aligned v874
+    v876 = load.i64 aligned v874+8
+    v877 = iadd_imm.i64 v101, 8
+    v878 = load.i8 v877
+    v879 = uextend.i32 v878
+    brz v879, ebb105
+    v880 = global_value.i64 gv65
+    trap user0
+
+ebb105:
+    v881 = load.i64 v101
+    v883 = band.i64 v882, v881
+        v934 -> v883
+        v961 -> v883
+        v991 -> v883
+        v1005 -> v883
+        v1048 -> v883
+    v884 = iconst.i8 1
+    v885 = uextend.i32 v884
+    brz v885, ebb109(v855)
+    jump ebb106
+
+ebb106:
+    v887 = iconst.i64 10
+    v888 = icmp.i64 ult v886, v887
+    v889 = bint.i8 v888
+    v890 = uextend.i32 v889
+    v891 = icmp_imm eq v890, 0
+    v892 = bint.i8 v891
+    v893 = uextend.i32 v892
+    brz v893, ebb108
+    jump ebb107
+
+ebb107:
+    v894 = global_value.i64 gv66
+    v895 = global_value.i64 gv67
+    trap user65535
+
+ebb108:
+    jump ebb109(v886)
+
+ebb109(v896: i64):
+    v897 = ireduce.i8 v896
+    v898 = iconst.i8 48
+    v899 = iadd v898, v897
+    v900 = iconst.i8 0
+    v901 = stack_addr.i64 ss139
+    v902 = stack_addr.i64 ss139
+    v903 = load.i16 aligned v902
+    v904 = iadd_imm.i64 v102, 1
+    v905 = load.i8 v904
+    v906 = uextend.i32 v905
+    brz v906, ebb110
+    v907 = global_value.i64 gv68
+    trap user0
+
+ebb110:
+    v909 = load.i64 v3
+    v910 = load.i64 v3+8
+    v911 = icmp.i64 ult v908, v910
+    v912 = bint.i8 v911
+    v913 = uextend.i32 v912
+    brnz v913, ebb111
+    v914 = global_value.i64 gv69
+    trap user0
+
+ebb111:
+    v915 = load.i64 v3
+    v916 = load.i64 v3+8
+    v918 = imul_imm.i64 v917, 1
+    v919 = iadd v915, v918
+    v920 = load.i8 aligned v102
+    v922 = iconst.i64 1
+    v923 = iadd.i64 v921, v922
+    v924 = iconst.i8 0
+    v925 = stack_addr.i64 ss140
+    v926 = stack_addr.i64 ss140
+    v927 = load.i64 aligned v926
+    v928 = load.i64 aligned v926+8
+    v929 = iadd_imm.i64 v103, 8
+    v930 = load.i8 v929
+    v931 = uextend.i32 v930
+    brz v931, ebb112
+    v932 = global_value.i64 gv70
+    trap user0
+
+ebb112:
+    v933 = load.i64 v103
+        v954 -> v933
+        v1047 -> v933
+    v936 = icmp.i64 ult v934, v935
+    v937 = bint.i8 v936
+    v938 = uextend.i32 v937
+    brz v938, ebb119
+    jump ebb113
+
+ebb113:
+    v940 = iconst.i64 1
+    v941 = ishl v940, v939
+    v942 = iconst.i8 0
+    v943 = stack_addr.i64 ss141
+    v944 = stack_addr.i64 ss141
+    v945 = load.i64 aligned v944
+    v946 = load.i64 aligned v944+8
+    v947 = iadd_imm.i64 v104, 8
+    v948 = load.i8 v947
+    v949 = uextend.i32 v948
+    brz v949, ebb114
+    v950 = global_value.i64 gv71
+    trap user0
+
+ebb114:
+    v951 = load.i64 v104
+        v988 -> v951
+    v952 = load.i64 aligned v3
+    v953 = load.i64 aligned v3+8
+    v955 = load.i64 v109
+    call fn101(v107, v108, v955)
+    jump ebb115
+
+ebb115:
+    v956 = load.i64 aligned v107
+    v957 = load.i64 aligned v107+8
+    v958 = load.i64 aligned v106
+    v959 = load.i64 aligned v106+8
+    v964 = load.i64 v30
+    v965 = isub.i64 v963, v964
+    v966 = iconst.i8 0
+    v967 = stack_addr.i64 ss142
+    v968 = stack_addr.i64 ss142
+    v969 = load.i64 aligned v968
+    v970 = load.i64 aligned v968+8
+    v971 = iadd_imm.i64 v110, 8
+    v972 = load.i8 v971
+    v973 = uextend.i32 v972
+    brz v973, ebb116
+    v974 = global_value.i64 gv72
+    trap user0
+
+ebb116:
+    v975 = load.i64 v110
+    v977 = imul v975, v976
+    v978 = iconst.i8 0
+    v979 = stack_addr.i64 ss143
+    v980 = stack_addr.i64 ss143
+    v981 = load.i64 aligned v980
+    v982 = load.i64 aligned v980+8
+    v983 = iadd_imm.i64 v111, 8
+    v984 = load.i8 v983
+    v985 = uextend.i32 v984
+    brz v985, ebb117
+    v986 = global_value.i64 gv73
+    trap user0
+
+ebb117:
+    v987 = load.i64 v111
+    call fn104(v0, v105, v990, v991, v992, v987, v988, v989)
+    jump ebb118
+
+ebb118:
+    jump ebb79
+
+ebb119:
+    v993 = iconst.i16 1
+    v994 = load.i16 v62
+    v995 = isub v994, v993
+    v996 = iconst.i8 0
+    v997 = stack_addr.i64 ss144
+    v998 = stack_addr.i64 ss144
+    v999 = load.i32 aligned v998
+    v1000 = iadd_imm.i64 v112, 2
+    v1001 = load.i8 v1000
+    v1002 = uextend.i32 v1001
+    brz v1002, ebb120
+    v1003 = global_value.i64 gv74
+    trap user0
+
+ebb120:
+    v1004 = load.i16 aligned v112
+    jump ebb99(v1005, v1036, v1038, v1042, v1047, v1055, v1060)
+}
diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index a2b248b0df..a182889059 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -28,6 +28,7 @@ use std::io::{self, Write};
 use std::option::Option;
 use std::process;
 
+mod bugpoint;
 mod cat;
 mod compile;
 mod disasm;
@@ -199,6 +200,14 @@ fn main() {
                 .arg(add_pass_arg())
                 .arg(add_debug_flag())
                 .arg(add_time_flag()),
+        )
+        .subcommand(
+            SubCommand::with_name("bugpoint")
+                .about("Reduce size of clif file causing panic during compilation.")
+                .arg(add_single_input_file_arg())
+                .arg(add_set_flag())
+                .arg(add_target_flag())
+                .arg(add_verbose_flag()),
         );
 
     let res_util = match app_cmds.get_matches().subcommand() {
@@ -284,6 +293,19 @@ fn main() {
 
             result
         }
+        ("bugpoint", Some(rest_cmd)) => {
+            let mut target_val: &str = "";
+            if let Some(clap_target) = rest_cmd.value_of("target") {
+                target_val = clap_target;
+            }
+
+            bugpoint::run(
+                rest_cmd.value_of("single-file").unwrap(),
+                &get_vec(rest_cmd.values_of("set")),
+                target_val,
+                rest_cmd.is_present("verbose"),
+            )
+        }
         _ => Err("Invalid subcommand.".to_owned()),
     };
 

From c7b4b98caca006b6e89798f9c730ea2733827106 Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Fri, 9 Aug 2019 15:30:11 -0600
Subject: [PATCH 2569/3084] Add a fold_redundant_jumps() pass to the branch
 relaxation phase. (#887)

---
 cranelift/codegen/src/binemit/relaxation.rs   | 142 +++++++++++++++++-
 cranelift/codegen/src/context.rs              |   2 +-
 cranelift/codegen/src/flowgraph.rs            |   2 +-
 cranelift/codegen/src/ir/function.rs          |  13 +-
 cranelift/codegen/src/licm.rs                 |  11 +-
 .../filetests/filetests/isa/x86/binary64.clif |  33 ++--
 .../filetests/isa/x86/legalize-br-table.clif  |   7 +-
 cranelift/filetests/src/test_binemit.rs       |   9 +-
 8 files changed, 178 insertions(+), 41 deletions(-)

diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index 1e84f2f4b9..62f692e557 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -29,7 +29,9 @@
 
 use crate::binemit::{CodeInfo, CodeOffset};
 use crate::cursor::{Cursor, FuncCursor};
-use crate::ir::{Function, InstructionData, Opcode};
+use crate::dominator_tree::DominatorTree;
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir::{Ebb, Function, Inst, InstructionData, Opcode, Value, ValueList};
 use crate::isa::{EncInfo, TargetIsa};
 use crate::iterators::IteratorExtras;
 use crate::regalloc::RegDiversions;
@@ -40,7 +42,12 @@ use log::debug;
 /// Relax branches and compute the final layout of EBB headers in `func`.
 ///
 /// Fill in the `func.offsets` table so the function is ready for binary emission.
-pub fn relax_branches(func: &mut Function, isa: &dyn TargetIsa) -> CodegenResult {
+pub fn relax_branches(
+    func: &mut Function,
+    cfg: &mut ControlFlowGraph,
+    domtree: &mut DominatorTree,
+    isa: &dyn TargetIsa,
+) -> CodegenResult {
     let _tt = timing::relax_branches();
 
     let encinfo = isa.encoding_info();
@@ -49,7 +56,10 @@ pub fn relax_branches(func: &mut Function, isa: &dyn TargetIsa) -> CodegenResult
     func.offsets.clear();
     func.offsets.resize(func.dfg.num_ebbs());
 
-    // Start by inserting fall through instructions.
+    // Start by removing redundant jumps.
+    fold_redundant_jumps(func, cfg, domtree);
+
+    // Convert jumps to fallthrough instructions where possible.
     fallthroughs(func);
 
     let mut offset = 0;
@@ -79,7 +89,6 @@ pub fn relax_branches(func: &mut Function, isa: &dyn TargetIsa) -> CodegenResult
         let mut cur = FuncCursor::new(func);
         while let Some(ebb) = cur.next_ebb() {
             divert.clear();
-
             // Record the offset for `ebb` and make sure we iterate until offsets are stable.
             if cur.func.offsets[ebb] != offset {
                 cur.func.offsets[ebb] = offset;
@@ -134,6 +143,131 @@ pub fn relax_branches(func: &mut Function, isa: &dyn TargetIsa) -> CodegenResult
     })
 }
 
+/// Folds an instruction if it is a redundant jump.
+/// Returns whether folding was performed (which invalidates the CFG).
+fn try_fold_redundant_jump(
+    func: &mut Function,
+    cfg: &mut ControlFlowGraph,
+    ebb: Ebb,
+    first_inst: Inst,
+) -> bool {
+    let first_dest = match func.dfg[first_inst].branch_destination() {
+        Some(ebb) => ebb, // The instruction was a single-target branch.
+        None => {
+            return false; // The instruction was either multi-target or not a branch.
+        }
+    };
+
+    // Look at the first instruction of the first branch's destination.
+    // If it is an unconditional branch, maybe the second jump can be bypassed.
+    let second_inst = func.layout.first_inst(first_dest).expect("Instructions");
+    if func.dfg[second_inst].opcode() != Opcode::Jump {
+        return false;
+    }
+
+    // Now we need to fix up first_inst's ebb parameters to match second_inst's,
+    // without changing the branch-specific arguments.
+    //
+    // The intermediary block is allowed to reference any SSA value that dominates it,
+    // but that SSA value may not necessarily also dominate the instruction that's
+    // being patched.
+
+    // Get the arguments and parameters passed by the first branch.
+    let num_fixed = func.dfg[first_inst]
+        .opcode()
+        .constraints()
+        .num_fixed_value_arguments();
+    let (first_args, first_params) = func.dfg[first_inst]
+        .arguments(&func.dfg.value_lists)
+        .split_at(num_fixed);
+
+    // Get the parameters passed by the second jump.
+    let num_fixed = func.dfg[second_inst]
+        .opcode()
+        .constraints()
+        .num_fixed_value_arguments();
+    let (_, second_params) = func.dfg[second_inst]
+        .arguments(&func.dfg.value_lists)
+        .split_at(num_fixed);
+    let mut second_params = second_params.to_vec(); // Clone for rewriting below.
+
+    // For each parameter passed by the second jump, if any of those parameters
+    // was a block parameter, rewrite it to refer to the value that the first jump
+    // passed in its parameters. Otherwise, make sure it dominates first_inst.
+    //
+    // For example: if we `ebb0: jump ebb1(v1)` to `ebb1(v2): jump ebb2(v2)`,
+    // we want to rewrite the original jump to `jump ebb2(v1)`.
+    let ebb_params: &[Value] = func.dfg.ebb_params(first_dest);
+    debug_assert!(ebb_params.len() == first_params.len());
+
+    for value in second_params.iter_mut() {
+        if let Some((n, _)) = ebb_params.iter().enumerate().find(|(_, &p)| p == *value) {
+            // This value was the Nth parameter passed to the second_inst's ebb.
+            // Rewrite it as the Nth parameter passed by first_inst.
+            *value = first_params[n];
+        }
+    }
+
+    // Build a value list of first_args (unchanged) followed by second_params (rewritten).
+    let arguments_vec: std::vec::Vec<_> = first_args
+        .iter()
+        .chain(second_params.iter())
+        .map(|x| *x)
+        .collect();
+    let value_list = ValueList::from_slice(&arguments_vec, &mut func.dfg.value_lists);
+
+    func.dfg[first_inst].take_value_list(); // Drop the current list.
+    func.dfg[first_inst].put_value_list(value_list); // Put the new list.
+
+    // Bypass the second jump.
+    // This can disconnect the Ebb containing `second_inst`, to be cleaned up later.
+    let second_dest = func.dfg[second_inst].branch_destination().expect("Dest");
+    func.change_branch_destination(first_inst, second_dest);
+    cfg.recompute_ebb(func, ebb);
+
+    // The previously-intermediary Ebb may now be unreachable. Update CFG.
+    if cfg.pred_iter(first_dest).count() == 0 {
+        // Remove all instructions from that ebb.
+        while let Some(inst) = func.layout.first_inst(first_dest) {
+            func.layout.remove_inst(inst);
+        }
+
+        // Remove the block...
+        cfg.recompute_ebb(func, first_dest); // ...from predecessor lists.
+        func.layout.remove_ebb(first_dest); // ...from the layout.
+    }
+
+    return true;
+}
+
+/// Redirects `jump` instructions that point to other `jump` instructions to the final destination.
+/// This transformation may orphan some blocks.
+fn fold_redundant_jumps(
+    func: &mut Function,
+    cfg: &mut ControlFlowGraph,
+    domtree: &mut DominatorTree,
+) {
+    let mut folded = false;
+
+    // Postorder iteration guarantees that a chain of jumps is visited from
+    // the end of the chain to the start of the chain.
+    for &ebb in domtree.cfg_postorder() {
+        // Only proceed if the first terminator instruction is a single-target branch.
+        let first_inst = func.layout.last_inst(ebb).expect("Ebb has no terminator");
+        folded |= try_fold_redundant_jump(func, cfg, ebb, first_inst);
+
+        // Also try the previous instruction.
+        if let Some(prev_inst) = func.layout.prev_inst(first_inst) {
+            folded |= try_fold_redundant_jump(func, cfg, ebb, prev_inst);
+        }
+    }
+
+    // Folding jumps invalidates the dominator tree.
+    if folded {
+        domtree.compute(func, cfg);
+    }
+}
+
 /// Convert `jump` instructions to `fallthrough` instructions where possible and verify that any
 /// existing `fallthrough` instructions are correct.
 fn fallthroughs(func: &mut Function) {
diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index 64c382cb23..a80d3c41e4 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -329,7 +329,7 @@ impl Context {
     /// Run the branch relaxation pass and return information about the function's code and
     /// read-only data.
     pub fn relax_branches(&mut self, isa: &dyn TargetIsa) -> CodegenResult {
-        let info = relax_branches(&mut self.func, isa)?;
+        let info = relax_branches(&mut self.func, &mut self.cfg, &mut self.domtree, isa)?;
         self.verify_if(isa)?;
         self.verify_locations_if(isa)?;
         Ok(info)
diff --git a/cranelift/codegen/src/flowgraph.rs b/cranelift/codegen/src/flowgraph.rs
index 335ca15ce4..645b7d4fff 100644
--- a/cranelift/codegen/src/flowgraph.rs
+++ b/cranelift/codegen/src/flowgraph.rs
@@ -31,7 +31,7 @@ use crate::timing;
 use core::mem;
 
 /// A basic block denoted by its enclosing Ebb and last instruction.
-#[derive(PartialEq, Eq)]
+#[derive(Debug, PartialEq, Eq)]
 pub struct BasicBlock {
     /// Enclosing Ebb key.
     pub ebb: Ebb,
diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index adbb244278..8eea542564 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -8,7 +8,7 @@ use crate::entity::{PrimaryMap, SecondaryMap};
 use crate::ir;
 use crate::ir::{DataFlowGraph, ExternalName, Layout, Signature};
 use crate::ir::{
-    Ebb, ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, JumpTable,
+    Ebb, ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, Inst, JumpTable,
     JumpTableData, SigRef, StackSlot, StackSlotData, Table, TableData,
 };
 use crate::ir::{EbbOffsets, InstEncodings, SourceLocs, StackSlots, ValueLocations};
@@ -20,7 +20,7 @@ use crate::write::write_function;
 use core::fmt;
 
 #[cfg(feature = "basic-blocks")]
-use crate::ir::{Inst, Opcode};
+use crate::ir::Opcode;
 
 /// A function.
 ///
@@ -223,6 +223,15 @@ impl Function {
         self.dfg.collect_debug_info();
     }
 
+    /// Changes the destination of a jump or branch instruction.
+    /// Does nothing if called with a non-jump or non-branch instruction.
+    pub fn change_branch_destination(&mut self, inst: Inst, new_dest: Ebb) {
+        match self.dfg[inst].branch_destination_mut() {
+            None => (),
+            Some(inst_dest) => *inst_dest = new_dest,
+        }
+    }
+
     /// Checks that the specified EBB can be encoded as a basic block.
     ///
     /// On error, returns the first invalid instruction and an error message.
diff --git a/cranelift/codegen/src/licm.rs b/cranelift/codegen/src/licm.rs
index 2af4ef3244..2866da87d7 100644
--- a/cranelift/codegen/src/licm.rs
+++ b/cranelift/codegen/src/licm.rs
@@ -88,7 +88,7 @@ fn create_pre_header(
     {
         // We only follow normal edges (not the back edges)
         if !domtree.dominates(header, last_inst, &func.layout) {
-            change_branch_jump_destination(last_inst, pre_header, func);
+            func.change_branch_destination(last_inst, pre_header);
         }
     }
     {
@@ -136,15 +136,6 @@ fn has_pre_header(
     result
 }
 
-// Change the destination of a jump or branch instruction. Does nothing if called with a non-jump
-// or non-branch instruction.
-fn change_branch_jump_destination(inst: Inst, new_ebb: Ebb, func: &mut Function) {
-    match func.dfg[inst].branch_destination_mut() {
-        None => (),
-        Some(instruction_dest) => *instruction_dest = new_ebb,
-    }
-}
-
 /// Test whether the given opcode is unsafe to even consider for LICM.
 fn trivially_unsafe_for_licm(opcode: Opcode) -> bool {
     opcode.can_store()
diff --git a/cranelift/filetests/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
index a65b3d3d1d..aa732e67f5 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64.clif
@@ -689,48 +689,48 @@ ebb0:
 
     ; asm: testq %rcx, %rcx
     ; asm: je ebb1
-    brz v1, ebb1                                ; bin: 48 85 c9 74 1b
+    brz v1, ebb1                                ; bin: 48 85 c9 74 19
     fallthrough ebb3
 
 ebb3:
     ; asm: testq %rsi, %rsi
     ; asm: je ebb1
-    brz v2, ebb1                                ; bin: 48 85 f6 74 16
+    brz v2, ebb1                                ; bin: 48 85 f6 74 14
     fallthrough ebb4
 
 ebb4:
     ; asm: testq %r10, %r10
     ; asm: je ebb1
-    brz v3, ebb1                                ; bin: 4d 85 d2 74 11
+    brz v3, ebb1                                ; bin: 4d 85 d2 74 0f
     fallthrough ebb5
 
 ebb5:
     ; asm: testq %rcx, %rcx
     ; asm: jne ebb1
-    brnz v1, ebb1                               ; bin: 48 85 c9 75 0c
+    brnz v1, ebb1                               ; bin: 48 85 c9 75 0a
     fallthrough ebb6
 
 ebb6:
     ; asm: testq %rsi, %rsi
     ; asm: jne ebb1
-    brnz v2, ebb1                               ; bin: 48 85 f6 75 07
+    brnz v2, ebb1                               ; bin: 48 85 f6 75 05
     fallthrough ebb7
 
 ebb7:
     ; asm: testq %r10, %r10
     ; asm: jne ebb1
-    brnz v3, ebb1                               ; bin: 4d 85 d2 75 02
+    brnz v3, ebb1                               ; bin: 4d 85 d2 75 00
 
     ; asm: jmp ebb2
-    jump ebb2                                   ; bin: eb 01
+    jump ebb2
 
     ; asm: ebb1:
 ebb1:
-    return                                       ; bin: c3
+    return
 
     ; asm: ebb2:
 ebb2:
-    jump ebb1                                   ; bin: eb fd
+    jump ebb1
 }
 
 ; CPU flag instructions.
@@ -1292,40 +1292,41 @@ ebb0:
 
     ; asm: testl %ecx, %ecx
     ; asm: je ebb1x
-    brz v1, ebb1                                ; bin: 85 c9 74 18
+    brz v1, ebb1                                ; bin: 85 c9 74 16
     fallthrough ebb3
 
 ebb3:
     ; asm: testl %esi, %esi
     ; asm: je ebb1x
-    brz v2, ebb1                                ; bin: 85 f6 74 14
+    brz v2, ebb1                                ; bin: 85 f6 74 12
     fallthrough ebb4
 
 ebb4:
     ; asm: testl %r10d, %r10d
     ; asm: je ebb1x
-    brz v3, ebb1                                ; bin: 45 85 d2 74 0f
+    brz v3, ebb1                                ; bin: 45 85 d2 74 0d
     fallthrough ebb5
 
 ebb5:
     ; asm: testl %ecx, %ecx
     ; asm: jne ebb1x
-    brnz v1, ebb1                               ; bin: 85 c9 75 0b
+    brnz v1, ebb1                               ; bin: 85 c9 75 09
     fallthrough ebb6
 
 ebb6:
     ; asm: testl %esi, %esi
     ; asm: jne ebb1x
-    brnz v2, ebb1                               ; bin: 85 f6 75 07
+    brnz v2, ebb1                               ; bin: 85 f6 75 05
     fallthrough ebb7
 
 ebb7:
     ; asm: testl %r10d, %r10d
     ; asm: jne ebb1x
-    brnz v3, ebb1                               ; bin: 45 85 d2 75 02
+    brnz v3, ebb1                               ; bin: 45 85 d2 75 00
 
     ; asm: jmp ebb2x
-    jump ebb2                                   ; bin: eb 01
+    ; branch relaxation translates this into `fallthrough ebb1`
+    jump ebb2
 
     ; asm: ebb1x:
 ebb1:
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
index 6d49a6e6d0..5656549cf7 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
@@ -11,9 +11,9 @@ function u0:0(i64) system_v {
 ebb0(v0: i64):
     v1 = stack_addr.i64 ss0
     v2 = load.i8 v1
-    br_table v2, ebb2, jt0
+    br_table v2, ebb1, jt0
 ; check:     $(oob=$V) = ifcmp_imm $(idx=$V), 1
-; nextln:    brif uge $oob, ebb2
+; nextln:    brif uge $oob, ebb1
 ; nextln:    fallthrough $(inb=$EBB)
 ; check:   $inb:
 ; nextln:    $(final_idx=$V) = uextend.i64 $idx
@@ -22,9 +22,6 @@ ebb0(v0: i64):
 ; nextln:    $(addr=$V) = iadd $base, $rel_addr
 ; nextln:    indirect_jump_table_br $addr, jt0
 
-ebb2:
-    jump ebb1
-
 ebb1:
     return
 }
diff --git a/cranelift/filetests/src/test_binemit.rs b/cranelift/filetests/src/test_binemit.rs
index 0b1e838795..3925acb8eb 100644
--- a/cranelift/filetests/src/test_binemit.rs
+++ b/cranelift/filetests/src/test_binemit.rs
@@ -7,6 +7,8 @@ use crate::match_directive::match_directive;
 use crate::subtest::{Context, SubTest, SubtestResult};
 use cranelift_codegen::binemit::{self, CodeInfo, CodeSink, RegDiversions};
 use cranelift_codegen::dbg::DisplayList;
+use cranelift_codegen::dominator_tree::DominatorTree;
+use cranelift_codegen::flowgraph::ControlFlowGraph;
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::entities::AnyEntity;
 use cranelift_codegen::print_errors::pretty_error;
@@ -166,8 +168,11 @@ impl SubTest for TestBinEmit {
         }
 
         // Relax branches and compute EBB offsets based on the encodings.
-        let CodeInfo { total_size, .. } = binemit::relax_branches(&mut func, isa)
-            .map_err(|e| pretty_error(&func, context.isa, e))?;
+        let mut cfg = ControlFlowGraph::with_function(&func);
+        let mut domtree = DominatorTree::with_function(&func, &cfg);
+        let CodeInfo { total_size, .. } =
+            binemit::relax_branches(&mut func, &mut cfg, &mut domtree, isa)
+                .map_err(|e| pretty_error(&func, context.isa, e))?;
 
         // Collect all of the 'bin:' directives on instructions.
         let mut bins = HashMap::new();

From d8d3602257e5bef2a5e5178ed26b86846a797bbb Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 5 Aug 2019 12:23:39 +0200
Subject: [PATCH 2570/3084] Adds the libcall_call_conv setting and use it for
 libcall calls expansion;

---
 cranelift/codegen/meta/src/shared/settings.rs | 23 +++++++++++++++++++
 cranelift/codegen/src/ir/libcall.rs           | 15 ++++++++++--
 cranelift/codegen/src/isa/call_conv.rs        | 15 ++++++++++++
 cranelift/codegen/src/legalizer/libcall.rs    | 13 +++++++++--
 cranelift/codegen/src/settings.rs             |  1 +
 5 files changed, 63 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
index 6b8b7a017f..3b6980daa1 100644
--- a/cranelift/codegen/meta/src/shared/settings.rs
+++ b/cranelift/codegen/meta/src/shared/settings.rs
@@ -94,6 +94,29 @@ pub fn define() -> SettingGroup {
 
     // Settings specific to the `baldrdash` calling convention.
 
+    settings.add_enum(
+        "libcall_call_conv",
+        r#"
+            Defines the calling convention to use for LibCalls call expansion,
+            since it may be different from the ISA default calling convention.
+
+            The default value is to use the same calling convention as the ISA
+            default calling convention.
+
+            This list should be kept in sync with the list of calling
+            conventions available in isa/call_conv.rs.
+        "#,
+        vec![
+            "isa_default",
+            "fast",
+            "cold",
+            "system_v",
+            "windows_fastcall",
+            "baldrdash",
+            "probestack",
+        ],
+    );
+
     settings.add_num(
         "baldrdash_prologue_words",
         r#"
diff --git a/cranelift/codegen/src/ir/libcall.rs b/cranelift/codegen/src/ir/libcall.rs
index f544dd5732..2398e043a0 100644
--- a/cranelift/codegen/src/ir/libcall.rs
+++ b/cranelift/codegen/src/ir/libcall.rs
@@ -108,11 +108,13 @@ impl LibCall {
 /// If there is an existing reference, use it, otherwise make a new one.
 pub fn get_libcall_funcref(
     libcall: LibCall,
+    call_conv: CallConv,
     func: &mut Function,
     inst: Inst,
     isa: &dyn TargetIsa,
 ) -> FuncRef {
-    find_funcref(libcall, func).unwrap_or_else(|| make_funcref_for_inst(libcall, func, inst, isa))
+    find_funcref(libcall, func)
+        .unwrap_or_else(|| make_funcref_for_inst(libcall, call_conv, func, inst, isa))
 }
 
 /// Get a function reference for the probestack function in `func`.
@@ -164,11 +166,12 @@ fn make_funcref_for_probestack(
 /// Create a funcref for `libcall` with a signature matching `inst`.
 fn make_funcref_for_inst(
     libcall: LibCall,
+    call_conv: CallConv,
     func: &mut Function,
     inst: Inst,
     isa: &dyn TargetIsa,
 ) -> FuncRef {
-    let mut sig = Signature::new(isa.default_call_conv());
+    let mut sig = Signature::new(call_conv);
     for &v in func.dfg.inst_args(inst) {
         sig.params.push(AbiParam::new(func.dfg.value_type(v)));
     }
@@ -176,6 +179,14 @@ fn make_funcref_for_inst(
         sig.returns.push(AbiParam::new(func.dfg.value_type(v)));
     }
 
+    if call_conv == CallConv::Baldrdash {
+        // Adds the special VMContext parameter to the signature.
+        sig.params.push(AbiParam::special(
+            isa.pointer_type(),
+            ArgumentPurpose::VMContext,
+        ));
+    }
+
     make_funcref(libcall, func, sig, isa)
 }
 
diff --git a/cranelift/codegen/src/isa/call_conv.rs b/cranelift/codegen/src/isa/call_conv.rs
index d4b4d3999b..dcebde35e5 100644
--- a/cranelift/codegen/src/isa/call_conv.rs
+++ b/cranelift/codegen/src/isa/call_conv.rs
@@ -1,3 +1,5 @@
+use crate::isa::TargetIsa;
+use crate::settings::LibcallCallConv;
 use core::fmt;
 use core::str;
 use target_lexicon::{CallingConvention, Triple};
@@ -29,6 +31,19 @@ impl CallConv {
             Ok(CallingConvention::WindowsFastcall) => CallConv::WindowsFastcall,
         }
     }
+
+    /// Returns the calling convention used for libcalls for the given ISA.
+    pub fn for_libcall(isa: &dyn TargetIsa) -> Self {
+        match isa.flags().libcall_call_conv() {
+            LibcallCallConv::IsaDefault => isa.default_call_conv(),
+            LibcallCallConv::Fast => CallConv::Fast,
+            LibcallCallConv::Cold => CallConv::Cold,
+            LibcallCallConv::SystemV => CallConv::SystemV,
+            LibcallCallConv::WindowsFastcall => CallConv::WindowsFastcall,
+            LibcallCallConv::Baldrdash => CallConv::Baldrdash,
+            LibcallCallConv::Probestack => CallConv::Probestack,
+        }
+    }
 }
 
 impl fmt::Display for CallConv {
diff --git a/cranelift/codegen/src/legalizer/libcall.rs b/cranelift/codegen/src/legalizer/libcall.rs
index 01ff630f01..a5368948e9 100644
--- a/cranelift/codegen/src/legalizer/libcall.rs
+++ b/cranelift/codegen/src/legalizer/libcall.rs
@@ -2,7 +2,7 @@
 
 use crate::ir;
 use crate::ir::{get_libcall_funcref, InstBuilder};
-use crate::isa::TargetIsa;
+use crate::isa::{CallConv, TargetIsa};
 use crate::legalizer::boundary::legalize_libcall_signature;
 use std::vec::Vec;
 
@@ -18,8 +18,17 @@ pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function, isa: &dyn Targ
     // Now we convert `inst` to a call. First save the arguments.
     let mut args = Vec::new();
     args.extend_from_slice(func.dfg.inst_args(inst));
+
+    let call_conv = CallConv::for_libcall(isa);
+    if call_conv == CallConv::Baldrdash {
+        let vmctx = func
+            .special_param(ir::ArgumentPurpose::VMContext)
+            .expect("Missing vmctx parameter for baldrdash libcall");
+        args.push(vmctx);
+    }
+
     // The replace builder will preserve the instruction result values.
-    let funcref = get_libcall_funcref(libcall, func, inst, isa);
+    let funcref = get_libcall_funcref(libcall, call_conv, func, inst, isa);
     func.dfg.replace(inst).call(funcref, &args);
 
     // Ask the ISA to legalize the signature.
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index b93e3d7149..75462869bc 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -379,6 +379,7 @@ mod tests {
             f.to_string(),
             "[shared]\n\
              opt_level = \"default\"\n\
+             libcall_call_conv = \"isa_default\"\n\
              baldrdash_prologue_words = 0\n\
              probestack_size_log2 = 12\n\
              enable_verifier = true\n\

From 2ee35b7ea175393dd433ceba2ed01d0d88b3b63e Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 14 Aug 2019 10:39:18 +0200
Subject: [PATCH 2571/3084] Implement a Windows Baldrdash calling convention;

---
 cranelift/codegen/meta/src/shared/settings.rs |  3 +-
 cranelift/codegen/src/ir/extfunc.rs           | 20 +++++++-----
 cranelift/codegen/src/ir/libcall.rs           |  2 +-
 cranelift/codegen/src/isa/call_conv.rs        | 31 ++++++++++++++++---
 cranelift/codegen/src/isa/mod.rs              |  2 +-
 cranelift/codegen/src/isa/x86/abi.rs          | 26 +++++++++++-----
 cranelift/codegen/src/legalizer/libcall.rs    |  2 +-
 cranelift/docs/heapex-dyn.clif                |  2 +-
 cranelift/docs/heapex-sm32.clif               |  2 +-
 cranelift/docs/heapex-sm64.clif               |  2 +-
 cranelift/docs/ir.rst                         |  5 +--
 .../filetests/filetests/isa/x86/abi64.clif    |  4 +--
 .../filetests/isa/x86/legalize-memory.clif    |  6 ++--
 .../filetests/isa/x86/legalize-splat.clif     | 12 +++----
 .../filetests/isa/x86/relax_branch.clif       |  2 +-
 .../filetests/legalizer/br_table_cond.clif    |  2 +-
 .../filetests/filetests/parser/call.clif      |  8 ++---
 .../filetests/filetests/regalloc/aliases.clif |  2 +-
 .../filetests/filetests/regalloc/iterate.clif |  6 ++--
 .../filetests/regalloc/unreachable_code.clif  |  2 +-
 cranelift/reader/src/parser.rs                |  6 ++--
 21 files changed, 92 insertions(+), 55 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
index 3b6980daa1..14ace5de14 100644
--- a/cranelift/codegen/meta/src/shared/settings.rs
+++ b/cranelift/codegen/meta/src/shared/settings.rs
@@ -112,7 +112,8 @@ pub fn define() -> SettingGroup {
             "cold",
             "system_v",
             "windows_fastcall",
-            "baldrdash",
+            "baldrdash_system_v",
+            "baldrdash_windows",
             "probestack",
         ],
     );
diff --git a/cranelift/codegen/src/ir/extfunc.rs b/cranelift/codegen/src/ir/extfunc.rs
index ada7bab8ca..0b74fd3d24 100644
--- a/cranelift/codegen/src/ir/extfunc.rs
+++ b/cranelift/codegen/src/ir/extfunc.rs
@@ -373,7 +373,8 @@ mod tests {
             CallConv::Cold,
             CallConv::SystemV,
             CallConv::WindowsFastcall,
-            CallConv::Baldrdash,
+            CallConv::BaldrdashSystemV,
+            CallConv::BaldrdashWindows,
         ] {
             assert_eq!(Ok(cc), cc.to_string().parse())
         }
@@ -381,16 +382,19 @@ mod tests {
 
     #[test]
     fn signatures() {
-        let mut sig = Signature::new(CallConv::Baldrdash);
-        assert_eq!(sig.to_string(), "() baldrdash");
+        let mut sig = Signature::new(CallConv::BaldrdashSystemV);
+        assert_eq!(sig.to_string(), "() baldrdash_system_v");
         sig.params.push(AbiParam::new(I32));
-        assert_eq!(sig.to_string(), "(i32) baldrdash");
+        assert_eq!(sig.to_string(), "(i32) baldrdash_system_v");
         sig.returns.push(AbiParam::new(F32));
-        assert_eq!(sig.to_string(), "(i32) -> f32 baldrdash");
+        assert_eq!(sig.to_string(), "(i32) -> f32 baldrdash_system_v");
         sig.params.push(AbiParam::new(I32.by(4).unwrap()));
-        assert_eq!(sig.to_string(), "(i32, i32x4) -> f32 baldrdash");
+        assert_eq!(sig.to_string(), "(i32, i32x4) -> f32 baldrdash_system_v");
         sig.returns.push(AbiParam::new(B8));
-        assert_eq!(sig.to_string(), "(i32, i32x4) -> f32, b8 baldrdash");
+        assert_eq!(
+            sig.to_string(),
+            "(i32, i32x4) -> f32, b8 baldrdash_system_v"
+        );
 
         // Order does not matter.
         sig.params[0].location = ArgumentLoc::Stack(24);
@@ -399,7 +403,7 @@ mod tests {
         // Writing ABI-annotated signatures.
         assert_eq!(
             sig.to_string(),
-            "(i32 [24], i32x4 [8]) -> f32, b8 baldrdash"
+            "(i32 [24], i32x4 [8]) -> f32, b8 baldrdash_system_v"
         );
     }
 }
diff --git a/cranelift/codegen/src/ir/libcall.rs b/cranelift/codegen/src/ir/libcall.rs
index 2398e043a0..59fb951a32 100644
--- a/cranelift/codegen/src/ir/libcall.rs
+++ b/cranelift/codegen/src/ir/libcall.rs
@@ -179,7 +179,7 @@ fn make_funcref_for_inst(
         sig.returns.push(AbiParam::new(func.dfg.value_type(v)));
     }
 
-    if call_conv == CallConv::Baldrdash {
+    if call_conv.extends_baldrdash() {
         // Adds the special VMContext parameter to the signature.
         sig.params.push(AbiParam::special(
             isa.pointer_type(),
diff --git a/cranelift/codegen/src/isa/call_conv.rs b/cranelift/codegen/src/isa/call_conv.rs
index dcebde35e5..dddd4bbd8a 100644
--- a/cranelift/codegen/src/isa/call_conv.rs
+++ b/cranelift/codegen/src/isa/call_conv.rs
@@ -15,8 +15,10 @@ pub enum CallConv {
     SystemV,
     /// Windows "fastcall" convention, also used for x64 and ARM
     WindowsFastcall,
-    /// SpiderMonkey WebAssembly convention
-    Baldrdash,
+    /// SpiderMonkey WebAssembly convention on systems using natively SystemV
+    BaldrdashSystemV,
+    /// SpiderMonkey WebAssembly convention on Windows
+    BaldrdashWindows,
     /// Specialized convention for the probestack function
     Probestack,
 }
@@ -40,10 +42,27 @@ impl CallConv {
             LibcallCallConv::Cold => CallConv::Cold,
             LibcallCallConv::SystemV => CallConv::SystemV,
             LibcallCallConv::WindowsFastcall => CallConv::WindowsFastcall,
-            LibcallCallConv::Baldrdash => CallConv::Baldrdash,
+            LibcallCallConv::BaldrdashSystemV => CallConv::BaldrdashSystemV,
+            LibcallCallConv::BaldrdashWindows => CallConv::BaldrdashWindows,
             LibcallCallConv::Probestack => CallConv::Probestack,
         }
     }
+
+    /// Is the calling convention extending the Windows Fastcall ABI?
+    pub fn extends_windows_fastcall(&self) -> bool {
+        match self {
+            CallConv::WindowsFastcall | CallConv::BaldrdashWindows => true,
+            _ => false,
+        }
+    }
+
+    /// Is the calling convention extending the Baldrdash ABI?
+    pub fn extends_baldrdash(&self) -> bool {
+        match self {
+            CallConv::BaldrdashSystemV | CallConv::BaldrdashWindows => true,
+            _ => false,
+        }
+    }
 }
 
 impl fmt::Display for CallConv {
@@ -53,7 +72,8 @@ impl fmt::Display for CallConv {
             CallConv::Cold => "cold",
             CallConv::SystemV => "system_v",
             CallConv::WindowsFastcall => "windows_fastcall",
-            CallConv::Baldrdash => "baldrdash",
+            CallConv::BaldrdashSystemV => "baldrdash_system_v",
+            CallConv::BaldrdashWindows => "baldrdash_windows",
             CallConv::Probestack => "probestack",
         })
     }
@@ -67,7 +87,8 @@ impl str::FromStr for CallConv {
             "cold" => Ok(CallConv::Cold),
             "system_v" => Ok(CallConv::SystemV),
             "windows_fastcall" => Ok(CallConv::WindowsFastcall),
-            "baldrdash" => Ok(CallConv::Baldrdash),
+            "baldrdash_system_v" => Ok(CallConv::BaldrdashSystemV),
+            "baldrdash_windows" => Ok(CallConv::BaldrdashWindows),
             "probestack" => Ok(CallConv::Probestack),
             _ => Err(()),
         }
diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index eb6b4d2650..407afc5f39 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -343,7 +343,7 @@ pub trait TargetIsa: fmt::Display + Sync {
         let word_size = StackSize::from(self.pointer_bytes());
 
         // Account for the SpiderMonkey standard prologue pushes.
-        if func.signature.call_conv == CallConv::Baldrdash {
+        if func.signature.call_conv.extends_baldrdash() {
             let bytes = StackSize::from(self.flags().baldrdash_prologue_words()) * word_size;
             let mut ss = ir::StackSlotData::new(ir::StackSlotKind::IncomingArg, bytes);
             ss.offset = Some(-(bytes as StackOffset));
diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 85e9c18ffd..0072935229 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -55,7 +55,7 @@ impl Args {
         shared_flags: &shared_settings::Flags,
         isa_flags: &isa_settings::Flags,
     ) -> Self {
-        let offset = if let CallConv::WindowsFastcall = call_conv {
+        let offset = if call_conv.extends_windows_fastcall() {
             // [1] "The caller is responsible for allocating space for parameters to the callee,
             // and must always allocate sufficient space to store four register parameters"
             32
@@ -109,7 +109,7 @@ impl ArgAssigner for Args {
         }
 
         // Handle special-purpose arguments.
-        if ty.is_int() && self.call_conv == CallConv::Baldrdash {
+        if ty.is_int() && self.call_conv.extends_baldrdash() {
             match arg.purpose {
                 // This is SpiderMonkey's `WasmTlsReg`.
                 ArgumentPurpose::VMContext => {
@@ -134,7 +134,7 @@ impl ArgAssigner for Args {
         }
 
         // Try to use an FPR.
-        let fpr_offset = if self.call_conv == CallConv::WindowsFastcall {
+        let fpr_offset = if self.call_conv.extends_windows_fastcall() {
             // Float and general registers on windows share the same parameter index.
             // The used register depends entirely on the parameter index: Even if XMM0
             // is not used for the first parameter, it cannot be used for the second parameter.
@@ -143,6 +143,7 @@ impl ArgAssigner for Args {
         } else {
             &mut self.fpr_used
         };
+
         if ty.is_float() && *fpr_offset < self.fpr_limit {
             let reg = FPR.unit(*fpr_offset);
             *fpr_offset += 1;
@@ -176,7 +177,7 @@ pub fn legalize_signature(
         }
         PointerWidth::U64 => {
             bits = 64;
-            args = if sig.call_conv == CallConv::WindowsFastcall {
+            args = if sig.call_conv.extends_windows_fastcall() {
                 Args::new(
                     bits,
                     &ARG_GPRS_WIN_FASTCALL_X64[..],
@@ -200,7 +201,7 @@ pub fn legalize_signature(
 
     legalize_args(&mut sig.params, &mut args);
 
-    let (regs, fpr_limit) = if sig.call_conv == CallConv::WindowsFastcall {
+    let (regs, fpr_limit) = if sig.call_conv.extends_windows_fastcall() {
         // windows-x64 calling convention only uses XMM0 or RAX for return values
         (&RET_GPRS_WIN_FASTCALL_X64[..], 1)
     } else {
@@ -250,7 +251,7 @@ fn callee_saved_gprs(isa: &dyn TargetIsa, call_conv: CallConv) -> &'static [RU]
         PointerWidth::U16 => panic!(),
         PointerWidth::U32 => &[RU::rbx, RU::rsi, RU::rdi],
         PointerWidth::U64 => {
-            if call_conv == CallConv::WindowsFastcall {
+            if call_conv.extends_windows_fastcall() {
                 // "registers RBX, RBP, RDI, RSI, RSP, R12, R13, R14, R15 are considered nonvolatile
                 //  and must be saved and restored by a function that uses them."
                 // as per https://msdn.microsoft.com/en-us/library/6t169e9c.aspx
@@ -322,7 +323,9 @@ pub fn prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> Codege
             system_v_prologue_epilogue(func, isa)
         }
         CallConv::WindowsFastcall => fastcall_prologue_epilogue(func, isa),
-        CallConv::Baldrdash => baldrdash_prologue_epilogue(func, isa),
+        CallConv::BaldrdashSystemV | CallConv::BaldrdashWindows => {
+            baldrdash_prologue_epilogue(func, isa)
+        }
         CallConv::Probestack => unimplemented!("probestack calling convention"),
     }
 }
@@ -336,7 +339,14 @@ fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) ->
     // Baldrdash on 32-bit x86 always aligns its stack pointer to 16 bytes.
     let stack_align = 16;
     let word_size = StackSize::from(isa.pointer_bytes());
-    let bytes = StackSize::from(isa.flags().baldrdash_prologue_words()) * word_size;
+    let shadow_store_size = if func.signature.call_conv.extends_windows_fastcall() {
+        32
+    } else {
+        0
+    };
+
+    let bytes =
+        StackSize::from(isa.flags().baldrdash_prologue_words()) * word_size + shadow_store_size;
 
     let mut ss = ir::StackSlotData::new(ir::StackSlotKind::IncomingArg, bytes);
     ss.offset = Some(-(bytes as StackOffset));
diff --git a/cranelift/codegen/src/legalizer/libcall.rs b/cranelift/codegen/src/legalizer/libcall.rs
index a5368948e9..1b11ae7593 100644
--- a/cranelift/codegen/src/legalizer/libcall.rs
+++ b/cranelift/codegen/src/legalizer/libcall.rs
@@ -20,7 +20,7 @@ pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function, isa: &dyn Targ
     args.extend_from_slice(func.dfg.inst_args(inst));
 
     let call_conv = CallConv::for_libcall(isa);
-    if call_conv == CallConv::Baldrdash {
+    if call_conv.extends_baldrdash() {
         let vmctx = func
             .special_param(ir::ArgumentPurpose::VMContext)
             .expect("Missing vmctx parameter for baldrdash libcall");
diff --git a/cranelift/docs/heapex-dyn.clif b/cranelift/docs/heapex-dyn.clif
index 24fc254d98..93c40bd29c 100644
--- a/cranelift/docs/heapex-dyn.clif
+++ b/cranelift/docs/heapex-dyn.clif
@@ -1,6 +1,6 @@
 test verifier
 
-function %add_members(i32, i64 vmctx) -> f32 baldrdash {
+function %add_members(i32, i64 vmctx) -> f32 baldrdash_system_v {
     gv0 = vmctx
     gv1 = load.i64 notrap aligned gv0+64
     gv2 = load.i32 notrap aligned gv0+72
diff --git a/cranelift/docs/heapex-sm32.clif b/cranelift/docs/heapex-sm32.clif
index 9c9c35e8f9..acd38a6564 100644
--- a/cranelift/docs/heapex-sm32.clif
+++ b/cranelift/docs/heapex-sm32.clif
@@ -1,6 +1,6 @@
 test verifier
 
-function %add_members(i32, i32 vmctx) -> f32 baldrdash {
+function %add_members(i32, i32 vmctx) -> f32 baldrdash_system_v {
     gv0 = vmctx
     gv1 = load.i32 notrap aligned gv0+64
     heap0 = static gv1, min 0x1000, bound 0x10_0000, offset_guard 0x1000
diff --git a/cranelift/docs/heapex-sm64.clif b/cranelift/docs/heapex-sm64.clif
index 22752eb401..20934ecba5 100644
--- a/cranelift/docs/heapex-sm64.clif
+++ b/cranelift/docs/heapex-sm64.clif
@@ -1,6 +1,6 @@
 test verifier
 
-function %add_members(i32, i64 vmctx) -> f32 baldrdash {
+function %add_members(i32, i64 vmctx) -> f32 baldrdash_system_v {
     gv0 = vmctx
     gv1 = load.i64 notrap aligned gv0+64
     heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, offset_guard 0x8000_0000
diff --git a/cranelift/docs/ir.rst b/cranelift/docs/ir.rst
index 7d159905d2..dca2399991 100644
--- a/cranelift/docs/ir.rst
+++ b/cranelift/docs/ir.rst
@@ -375,7 +375,7 @@ convention:
     param        : type [paramext] [paramspecial]
     paramext     : "uext" | "sext"
     paramspecial : "sret" | "link" | "fp" | "csr" | "vmctx" | "sigid" | "stack_limit"
-    callconv     : "fast" | "cold" | "system_v" | "fastcall" | "baldrdash"
+    callconv     : "fast" | "cold" | "system_v" | "fastcall" | "baldrdash_system_v" | "baldrdash_windows"
 
 A function's calling convention determines exactly how arguments and return
 values are passed, and how stack frames are managed. Since all of these details
@@ -402,7 +402,8 @@ fast       not-ABI-stable convention for best performance
 cold       not-ABI-stable convention for infrequently executed code
 system_v   System V-style convention used on many platforms
 fastcall   Windows "fastcall" convention, also used for x64 and ARM
-baldrdash  SpiderMonkey WebAssembly convention
+baldrdash_system_v  SpiderMonkey WebAssembly convention on platforms natively using SystemV.
+baldrdash_windows  SpiderMonkey WebAssembly convention on platforms natively using Windows.
 ========== ===========================================
 
 The "not-ABI-stable" conventions do not follow an external specification and
diff --git a/cranelift/filetests/filetests/isa/x86/abi64.clif b/cranelift/filetests/filetests/isa/x86/abi64.clif
index 209b036ee2..ccea6304e6 100644
--- a/cranelift/filetests/filetests/isa/x86/abi64.clif
+++ b/cranelift/filetests/filetests/isa/x86/abi64.clif
@@ -18,8 +18,8 @@ ebb0:
     return
 }
 
-function %pass_stack_int64(i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) baldrdash {
-    sig0 = (i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) baldrdash
+function %pass_stack_int64(i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) baldrdash_system_v {
+    sig0 = (i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) baldrdash_system_v
     fn0 = u0:0 sig0
 
 ebb0(v0: i64, v1: i64, v2: i64, v3: i64, v4: i64, v5: i64, v6: i64, v7: i64, v8: i64, v9: i64, v10: i64, v11: i64, v12: i64, v13: i64, v14: i64, v15: i64, v16: i64, v17: i64, v18: i64, v19: i64, v20: i64):
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-memory.clif b/cranelift/filetests/filetests/isa/x86/legalize-memory.clif
index 7b506d3876..348d763716 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-memory.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-memory.clif
@@ -44,7 +44,7 @@ ebb1:
 
 ; SpiderMonkey VM-style static 4+2 GB heap.
 ; This eliminates bounds checks completely for offsets < 2GB.
-function %staticheap_sm64(i32, i64 vmctx) -> f32 baldrdash {
+function %staticheap_sm64(i32, i64 vmctx) -> f32 baldrdash_system_v {
     gv0 = vmctx
     gv1 = iadd_imm.i64 gv0, 64
     heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, offset_guard 0x8000_0000
@@ -65,7 +65,7 @@ ebb0(v0: i32, v999: i64):
     return v4
 }
 
-function %staticheap_static_oob_sm64(i32, i64 vmctx) -> f32 baldrdash {
+function %staticheap_static_oob_sm64(i32, i64 vmctx) -> f32 baldrdash_system_v {
     gv0 = vmctx
     gv1 = iadd_imm.i64 gv0, 64
     heap0 = static gv1, min 0x1000, bound 0x1000_0000, offset_guard 0x8000_0000
@@ -89,7 +89,7 @@ ebb0(v0: i32, v999: i64):
 
 ; SpiderMonkey VM-style static 4+2 GB heap.
 ; Offsets >= 2 GB do require a boundscheck.
-function %staticheap_sm64(i32, i64 vmctx) -> f32 baldrdash {
+function %staticheap_sm64(i32, i64 vmctx) -> f32 baldrdash_system_v {
     gv0 = vmctx
     gv1 = iadd_imm.i64 gv0, 64
     heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, offset_guard 0x8000_0000
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-splat.clif b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
index c50f5a2f81..fa07f80c11 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
@@ -3,15 +3,15 @@ set enable_simd=true
 set probestack_enabled=false
 target x86_64 haswell
 
-; use baldrdash calling convention here for simplicity (avoids prologue, epilogue)
-function %test_splat_i32() -> i32x4 baldrdash {
+; use baldrdash_system_v calling convention here for simplicity (avoids prologue, epilogue)
+function %test_splat_i32() -> i32x4 baldrdash_system_v {
 ebb0:
     v0 = iconst.i32 42
     v1 = splat.i32x4 v0
     return v1
 }
 
-; sameln: function %test_splat_i32() -> i32x4 [%xmm0] baldrdash {
+; sameln: function %test_splat_i32() -> i32x4 [%xmm0] baldrdash_system_v {
 ; nextln:   ss0 = incoming_arg 0, offset 0
 ; nextln: 
 ; nextln:   ebb0:
@@ -23,7 +23,7 @@ ebb0:
 
 
 
-function %test_splat_i64() -> i64x2 baldrdash {
+function %test_splat_i64() -> i64x2 baldrdash_system_v {
 ebb0:
     v0 = iconst.i64 42
     v1 = splat.i64x2 v0
@@ -38,7 +38,7 @@ ebb0:
 
 
 
-function %test_splat_b16() -> b16x8 baldrdash {
+function %test_splat_b16() -> b16x8 baldrdash_system_v {
 ebb0:
     v0 = bconst.b16 true
     v1 = splat.b16x8 v0
@@ -56,7 +56,7 @@ ebb0:
 
 
 
-function %test_splat_i8() -> i8x16 baldrdash {
+function %test_splat_i8() -> i8x16 baldrdash_system_v {
 ebb0:
     v0 = iconst.i8 42
     v1 = splat.i8x16 v0
diff --git a/cranelift/filetests/filetests/isa/x86/relax_branch.clif b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
index 373e2dda6d..7b86aef0e0 100644
--- a/cranelift/filetests/filetests/isa/x86/relax_branch.clif
+++ b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
@@ -10,7 +10,7 @@ target x86_64 haswell
 ; particular, the first block has to be non-empty but its encoding size must be
 ; zero (i.e. not generate any code). See also issue #666 for more details.
 
-function u0:2691(i32 [%rdi], i32 [%rsi], i64 vmctx [%r14]) -> i64 uext [%rax] baldrdash {
+function u0:2691(i32 [%rdi], i32 [%rsi], i64 vmctx [%r14]) -> i64 uext [%rax] baldrdash_system_v {
     ss0 = incoming_arg 24, offset -24
     gv0 = vmctx
     gv1 = iadd_imm.i64 gv0, 48
diff --git a/cranelift/filetests/filetests/legalizer/br_table_cond.clif b/cranelift/filetests/filetests/legalizer/br_table_cond.clif
index c88becab08..bd823d3cad 100644
--- a/cranelift/filetests/filetests/legalizer/br_table_cond.clif
+++ b/cranelift/filetests/filetests/legalizer/br_table_cond.clif
@@ -7,7 +7,7 @@ target x86_64
 ; regex: V=v\d+
 ; regex: EBB=ebb\d+
 
-function u0:0(i64 vmctx) baldrdash {
+function u0:0(i64 vmctx) baldrdash_system_v {
     gv0 = vmctx
     gv1 = iadd_imm.i64 gv0, 48
     jt0 = jump_table [ebb2, ebb2, ebb7]
diff --git a/cranelift/filetests/filetests/parser/call.clif b/cranelift/filetests/filetests/parser/call.clif
index c75fa02393..28e3011f33 100644
--- a/cranelift/filetests/filetests/parser/call.clif
+++ b/cranelift/filetests/filetests/parser/call.clif
@@ -10,13 +10,13 @@ ebb1:
 ; nextln:     return
 ; nextln: }
 
-function %r1() -> i32, f32 baldrdash {
+function %r1() -> i32, f32 baldrdash_system_v {
 ebb1:
     v1 = iconst.i32 3
     v2 = f32const 0.0
     return v1, v2
 }
-; sameln: function %r1() -> i32, f32 baldrdash {
+; sameln: function %r1() -> i32, f32 baldrdash_system_v {
 ; nextln: ebb1:
 ; nextln:     v1 = iconst.i32 3
 ; nextln:     v2 = f32const 0.0
@@ -25,13 +25,13 @@ ebb1:
 
 function %signatures() {
     sig10 = ()
-    sig11 = (i32, f64) -> i32, b1 baldrdash
+    sig11 = (i32, f64) -> i32, b1 baldrdash_system_v
     fn5 = %foo sig11
     fn8 = %bar(i32) -> b1
 }
 ; sameln: function %signatures() fast {
 ; check:      sig10 = () fast
-; check:      sig11 = (i32, f64) -> i32, b1 baldrdash
+; check:      sig11 = (i32, f64) -> i32, b1 baldrdash_system_v
 ; check:      sig12 = (i32) -> b1 fast
 ; not:        fn0
 ; check:      fn5 = %foo sig11
diff --git a/cranelift/filetests/filetests/regalloc/aliases.clif b/cranelift/filetests/filetests/regalloc/aliases.clif
index 2a94192415..7e6d5c6028 100644
--- a/cranelift/filetests/filetests/regalloc/aliases.clif
+++ b/cranelift/filetests/filetests/regalloc/aliases.clif
@@ -1,7 +1,7 @@
 test regalloc
 target x86_64 haswell
 
-function %value_aliases(i32, f32, i64 vmctx) baldrdash {
+function %value_aliases(i32, f32, i64 vmctx) baldrdash_system_v {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
diff --git a/cranelift/filetests/filetests/regalloc/iterate.clif b/cranelift/filetests/filetests/regalloc/iterate.clif
index a18364e0b4..347dbc5f29 100644
--- a/cranelift/filetests/filetests/regalloc/iterate.clif
+++ b/cranelift/filetests/filetests/regalloc/iterate.clif
@@ -1,7 +1,7 @@
 test regalloc
 target x86_64 haswell
 
-function u0:9(i64 [%rdi], f32 [%xmm0], f64 [%xmm1], i32 [%rsi], i32 [%rdx], i64 vmctx [%r14]) -> i64 [%rax] baldrdash {
+function u0:9(i64 [%rdi], f32 [%xmm0], f64 [%xmm1], i32 [%rsi], i32 [%rdx], i64 vmctx [%r14]) -> i64 [%rax] baldrdash_system_v {
 ebb0(v0: i64, v1: f32, v2: f64, v3: i32, v4: i32, v5: i64):
     v32 = iconst.i32 0
     v6 = bitcast.f32 v32
@@ -121,10 +121,10 @@ ebb1(v31: i64):
     return v31
 }
 
-function u0:26(i64 vmctx [%r14]) -> i64 [%rax] baldrdash {
+function u0:26(i64 vmctx [%r14]) -> i64 [%rax] baldrdash_system_v {
     gv1 = vmctx
     gv0 = iadd_imm.i64 gv1, 48
-    sig0 = (i32 [%rdi], i64 [%rsi], i64 vmctx [%r14], i64 sigid [%rbx]) -> i64 [%rax] baldrdash
+    sig0 = (i32 [%rdi], i64 [%rsi], i64 vmctx [%r14], i64 sigid [%rbx]) -> i64 [%rax] baldrdash_system_v
 
 ebb0(v0: i64):
     v1 = iconst.i32 32
diff --git a/cranelift/filetests/filetests/regalloc/unreachable_code.clif b/cranelift/filetests/filetests/regalloc/unreachable_code.clif
index 9d11ded01a..79e95ab9b0 100644
--- a/cranelift/filetests/filetests/regalloc/unreachable_code.clif
+++ b/cranelift/filetests/filetests/regalloc/unreachable_code.clif
@@ -6,7 +6,7 @@ target x86_64 haswell
 
 ; This function contains unreachable blocks which trip up the register
 ; allocator if they don't get cleared out.
-function %unreachable_blocks(i64 vmctx) -> i32 baldrdash {
+function %unreachable_blocks(i64 vmctx) -> i32 baldrdash_system_v {
 ebb0(v0: i64):
     v1 = iconst.i32 0
     v2 = iconst.i32 0
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 037ed2acce..3a5906d5be 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -2630,14 +2630,14 @@ mod tests {
         assert_eq!(sig.returns.len(), 0);
         assert_eq!(sig.call_conv, CallConv::SystemV);
 
-        let sig2 = Parser::new("(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 baldrdash")
+        let sig2 = Parser::new("(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 baldrdash_system_v")
             .parse_signature(None)
             .unwrap();
         assert_eq!(
             sig2.to_string(),
-            "(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 baldrdash"
+            "(i8 uext, f32, f64, i32 sret) -> i32 sext, f64 baldrdash_system_v"
         );
-        assert_eq!(sig2.call_conv, CallConv::Baldrdash);
+        assert_eq!(sig2.call_conv, CallConv::BaldrdashSystemV);
 
         // Old-style signature without a calling convention.
         assert_eq!(

From b659262d2a6e6c04e149f427ba009ed0786b4e34 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 16 Aug 2019 14:55:34 +0200
Subject: [PATCH 2572/3084] Use aliasing instead of copying in simple_preopt;

---
 cranelift/codegen/src/simple_preopt.rs        | 42 ++++++----
 .../simple_preopt/div_by_const_indirect.clif  | 16 ++--
 .../div_by_const_non_power_of_2.clif          | 80 +++++++++----------
 .../div_by_const_power_of_2.clif              | 22 ++---
 4 files changed, 84 insertions(+), 76 deletions(-)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index 73c6a56148..92cbb4c44f 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -18,6 +18,20 @@ use crate::ir::{
 };
 use crate::timing;
 
+#[inline]
+/// Replaces the result_index nth result of instruction inst to an alias of the given value, and
+/// replaces the instruction with a nop.
+fn replace_with_alias(dfg: &mut DataFlowGraph, inst: Inst, result_index: usize, value: Value) {
+    // Replace the result value by an alias.
+    let results = dfg.detach_results(inst);
+    debug_assert!(results.len(&dfg.value_lists) > result_index);
+    let result = results.get(result_index, &dfg.value_lists).unwrap();
+    dfg.change_to_alias(result, value);
+
+    // Replace instruction by a nop.
+    dfg.replace(inst).nop();
+}
+
 //----------------------------------------------------------------------
 //
 // Pattern-match helpers and transformation for div and rem by constants.
@@ -171,7 +185,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             if is_rem {
                 pos.func.dfg.replace(inst).iconst(I32, 0);
             } else {
-                pos.func.dfg.replace(inst).copy(n1);
+                replace_with_alias(&mut pos.func.dfg, inst, 0, n1);
             }
         }
 
@@ -226,7 +240,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                 let tt = pos.ins().imul_imm(qf, d as i64);
                 pos.func.dfg.replace(inst).isub(n1, tt);
             } else {
-                pos.func.dfg.replace(inst).copy(qf);
+                replace_with_alias(&mut pos.func.dfg, inst, 0, qf);
             }
         }
 
@@ -241,7 +255,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             if is_rem {
                 pos.func.dfg.replace(inst).iconst(I64, 0);
             } else {
-                pos.func.dfg.replace(inst).copy(n1);
+                replace_with_alias(&mut pos.func.dfg, inst, 0, n1);
             }
         }
 
@@ -296,7 +310,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                 let tt = pos.ins().imul_imm(qf, d as i64);
                 pos.func.dfg.replace(inst).isub(n1, tt);
             } else {
-                pos.func.dfg.replace(inst).copy(qf);
+                replace_with_alias(&mut pos.func.dfg, inst, 0, qf);
             }
         }
 
@@ -314,7 +328,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             if is_rem {
                 pos.func.dfg.replace(inst).iconst(I32, 0);
             } else {
-                pos.func.dfg.replace(inst).copy(n1);
+                replace_with_alias(&mut pos.func.dfg, inst, 0, n1);
             }
         }
 
@@ -340,7 +354,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                     if is_negative {
                         pos.func.dfg.replace(inst).irsub_imm(t4, 0);
                     } else {
-                        pos.func.dfg.replace(inst).copy(t4);
+                        replace_with_alias(&mut pos.func.dfg, inst, 0, t4);
                     }
                 }
             } else {
@@ -370,7 +384,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                     let tt = pos.ins().imul_imm(qf, d as i64);
                     pos.func.dfg.replace(inst).isub(n1, tt);
                 } else {
-                    pos.func.dfg.replace(inst).copy(qf);
+                    replace_with_alias(&mut pos.func.dfg, inst, 0, qf);
                 }
             }
         }
@@ -389,7 +403,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             if is_rem {
                 pos.func.dfg.replace(inst).iconst(I64, 0);
             } else {
-                pos.func.dfg.replace(inst).copy(n1);
+                replace_with_alias(&mut pos.func.dfg, inst, 0, n1);
             }
         }
 
@@ -415,7 +429,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                     if is_negative {
                         pos.func.dfg.replace(inst).irsub_imm(t4, 0);
                     } else {
-                        pos.func.dfg.replace(inst).copy(t4);
+                        replace_with_alias(&mut pos.func.dfg, inst, 0, t4);
                     }
                 }
             } else {
@@ -445,7 +459,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                     let tt = pos.ins().imul_imm(qf, d);
                     pos.func.dfg.replace(inst).isub(n1, tt);
                 } else {
-                    pos.func.dfg.replace(inst).copy(qf);
+                    replace_with_alias(&mut pos.func.dfg, inst, 0, qf);
                 }
             }
         }
@@ -651,13 +665,7 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
                 | (Opcode::UshrImm, 0)
                 | (Opcode::SshrImm, 0) => {
                     // Alias the result value with the original argument.
-                    let results = pos.func.dfg.detach_results(inst);
-                    debug_assert!(results.len(&pos.func.dfg.value_lists) == 1);
-                    let first_result = results.get(0, &pos.func.dfg.value_lists).unwrap();
-                    pos.func.dfg.change_to_alias(first_result, arg);
-
-                    // Replace instruction by a nop.
-                    pos.func.dfg.replace(inst).nop();
+                    replace_with_alias(&mut pos.func.dfg, inst, 0, arg);
                     return;
                 }
                 (Opcode::ImulImm, 0) | (Opcode::BandImm, 0) => {
diff --git a/cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif b/cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif
index fa66337fc2..5833b0f371 100644
--- a/cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif
+++ b/cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif
@@ -13,8 +13,8 @@ ebb0(v0: i32):
     ; check: isub v0, v4
     ; check: ushr_imm v5, 1
     ; check: iadd v6, v4
-    ; check: ushr_imm v7, 2
-    ; check: copy v8
+    ; check: v8 = ushr_imm v7, 2
+    ; check: v2 -> v8
     return v2
 }
 
@@ -27,8 +27,8 @@ ebb0(v0: i32):
     ; check: smulhi v0, v3
     ; check: sshr_imm v4, 3
     ; check: ushr_imm v5, 31
-    ; check: iadd v5, v6
-    ; check: copy v7
+    ; check: v7 = iadd v5, v6
+    ; check: v2 -> v7
     return v2
 }
 
@@ -39,8 +39,8 @@ ebb0(v0: i64):
     ; check: iconst.i64 1337
     ; check: iconst.i64 0xc411_9d95_2866_a139
     ; check: umulhi v0, v3
-    ; check: ushr_imm v4, 10
-    ; check: copy v5
+    ; check: v5 = ushr_imm v4, 10
+    ; check: v2 -> v5
     return v2
 }
 
@@ -53,7 +53,7 @@ ebb0(v0: i64):
     ; check: smulhi v0, v3
     ; check: sshr_imm v4, 14
     ; check: ushr_imm v5, 63
-    ; check: iadd v5, v6
-    ; check: copy v7
+    ; check: v7 = iadd v5, v6
+    ; check: v2 -> v7
     return v2
 }
diff --git a/cranelift/filetests/filetests/simple_preopt/div_by_const_non_power_of_2.clif b/cranelift/filetests/filetests/simple_preopt/div_by_const_non_power_of_2.clif
index fa0ac41bff..2a16699aae 100644
--- a/cranelift/filetests/filetests/simple_preopt/div_by_const_non_power_of_2.clif
+++ b/cranelift/filetests/filetests/simple_preopt/div_by_const_non_power_of_2.clif
@@ -12,8 +12,8 @@ ebb0(v0: i32):
     ; check: isub v0, v3
     ; check: ushr_imm v4, 1
     ; check: iadd v5, v3
-    ; check: ushr_imm v6, 2
-    ; check: copy v7
+    ; check: v7 = ushr_imm v6, 2
+    ; check: v1 -> v7
     return v1
 }
 
@@ -23,8 +23,8 @@ ebb0(v0: i32):
     v1 = udiv_imm v0, 125
     ; check: iconst.i32 0x1062_4dd3
     ; check: umulhi v0, v2
-    ; check: ushr_imm v3, 3
-    ; check: copy v4
+    ; check: v4 = ushr_imm v3, 3
+    ; check: v1 -> v4
     return v1
 }
 
@@ -33,8 +33,8 @@ function %t_udiv32_p641(i32) -> i32 {
 ebb0(v0: i32):
     v1 = udiv_imm v0, 641
     ; check: iconst.i32 0x0066_3d81
-    ; check: umulhi v0, v2
-    ; check: copy v3
+    ; check: v3 = umulhi v0, v2
+    ; check: v1 -> v3
     return v1
 }
 
@@ -48,8 +48,8 @@ ebb0(v0: i32):
     ; check: iconst.i32 0xffff_ffff_d555_5555
     ; check: smulhi v0, v2
     ; check: ushr_imm v3, 31
-    ; check: iadd v3, v4
-    ; check: copy v5
+    ; check: v5 = iadd v3, v4
+    ; check: v1 -> v5
     return v1
 }
 
@@ -61,8 +61,8 @@ ebb0(v0: i32):
     ; check: smulhi v0, v2
     ; check: sshr_imm v3, 1
     ; check: ushr_imm v4, 31
-    ; check: iadd v4, v5
-    ; check: copy v6
+    ; check: v6 = iadd v4, v5
+    ; check: v1 -> v6
     return v1
 }
 
@@ -75,8 +75,8 @@ ebb0(v0: i32):
     ; check: isub v3, v0
     ; check: sshr_imm v4, 1
     ; check: ushr_imm v5, 31
-    ; check: iadd v5, v6
-    ; check: copy v7
+    ; check: v7 = iadd v5, v6
+    ; check: v1 -> v7
     return v1
 }
 
@@ -87,8 +87,8 @@ ebb0(v0: i32):
     ; check: iconst.i32 0x2aaa_aaab
     ; check: smulhi v0, v2
     ; check: ushr_imm v3, 31
-    ; check: iadd v3, v4
-    ; check: copy v5
+    ; check: v5 = iadd v3, v4
+    ; check: v1 -> v5
     return v1
 }
 
@@ -101,8 +101,8 @@ ebb0(v0: i32):
     ; check: iadd v3, v0
     ; check: sshr_imm v4, 2
     ; check: ushr_imm v5, 31
-    ; check: iadd v5, v6
-    ; check: copy v7
+    ; check: v7 = iadd v5, v6
+    ; check: v1 -> v7
     return v1
 }
 
@@ -114,8 +114,8 @@ ebb0(v0: i32):
     ; check: smulhi v0, v2
     ; check: sshr_imm v3, 8
     ; check: ushr_imm v4, 31
-    ; check: iadd v4, v5
-    ; check: copy v6
+    ; check: v6 = iadd v4, v5
+    ; check: v1 -> v6
     return v1
 }
 
@@ -131,8 +131,8 @@ ebb0(v0: i64):
     ; check: isub v0, v3
     ; check: ushr_imm v4, 1
     ; check: iadd v5, v3
-    ; check: ushr_imm v6, 2
-    ; check: copy v7
+    ; check: v7 = ushr_imm v6, 2
+    ; check: v1 -> v7
     return v1
 }
 
@@ -142,8 +142,8 @@ ebb0(v0: i64):
     v1 = udiv_imm v0, 9
     ; check: iconst.i64 0xe38e_38e3_8e38_e38f
     ; check: umulhi v0, v2
-    ; check: ushr_imm v3, 3
-    ; check: copy v4
+    ; check: v4 = ushr_imm v3, 3
+    ; check: v1 -> v4
     return v1
 }
 
@@ -156,8 +156,8 @@ ebb0(v0: i64):
     ; check: isub v0, v3
     ; check: ushr_imm v4, 1
     ; check: iadd v5, v3
-    ; check: ushr_imm v6, 6
-    ; check: copy v7
+    ; check: v7 = ushr_imm v6, 6
+    ; check: v1 -> v7
     return v1
 }
 
@@ -166,8 +166,8 @@ function %t_udiv64_p274177(i64) -> i64 {
 ebb0(v0: i64):
     v1 = udiv_imm v0, 274177
     ; check: iconst.i64 0x3d30_f19c_d101
-    ; check: umulhi v0, v2
-    ; check: copy v3
+    ; check: v3 = umulhi v0, v2
+    ; check: v1 -> v3
     return v1
 }
 
@@ -182,8 +182,8 @@ ebb0(v0: i64):
     ; check: smulhi v0, v2
     ; check: sshr_imm v3, 7
     ; check: ushr_imm v4, 63
-    ; check: iadd v4, v5
-    ; check: copy v6
+    ; check: v6 = iadd v4, v5
+    ; check: v1 -> v6
     return v1
 }
 
@@ -194,8 +194,8 @@ ebb0(v0: i64):
     ; check: iconst.i64 0xd555_5555_5555_5555
     ; check: smulhi v0, v2
     ; check: ushr_imm v3, 63
-    ; check: iadd v3, v4
-    ; check: copy v5
+    ; check: v5 = iadd v3, v4
+    ; check: v1 -> v5
     return v1
 }
 
@@ -207,8 +207,8 @@ ebb0(v0: i64):
     ; check: smulhi v0, v2
     ; check: sshr_imm v3, 1
     ; check: ushr_imm v4, 63
-    ; check: iadd v4, v5
-    ; check: copy v6
+    ; check: v6 = iadd v4, v5
+    ; check: v1 -> v6
     return v1
 }
 
@@ -221,8 +221,8 @@ ebb0(v0: i64):
     ; check: isub v3, v0
     ; check: sshr_imm v4, 1
     ; check: ushr_imm v5, 63
-    ; check: iadd v5, v6
-    ; check: copy v7
+    ; check: v7 = iadd v5, v6
+    ; check: v1 -> v7
     return v1
 }
 
@@ -233,8 +233,8 @@ ebb0(v0: i64):
     ; check: iconst.i64 0x2aaa_aaaa_aaaa_aaab
     ; check: smulhi v0, v2
     ; check: ushr_imm v3, 63
-    ; check: iadd v3, v4
-    ; check: copy v5
+    ; check: v5 = iadd v3, v4
+    ; check: v1 -> v5
     return v1
 }
 
@@ -247,8 +247,8 @@ ebb0(v0: i64):
     ; check: iadd v3, v0
     ; check: sshr_imm v4, 3
     ; check: ushr_imm v5, 63
-    ; check: iadd v5, v6
-    ; check: copy v7
+    ; check: v7 = iadd v5, v6
+    ; check: v1 -> v7
     return v1
 }
 
@@ -260,7 +260,7 @@ ebb0(v0: i64):
     ; check: smulhi v0, v2
     ; check: sshr_imm v3, 7
     ; check: ushr_imm v4, 63
-    ; check: iadd v4, v5
-    ; check: copy v6
+    ; check: v6 = iadd v4, v5
+    ; check: v1 -> v6
     return v1
 }
diff --git a/cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif b/cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif
index 7e6bf7fd14..fb9c1744f5 100644
--- a/cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif
+++ b/cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif
@@ -104,7 +104,7 @@ ebb0(v0: i32):
     ; check: ushr_imm v0, 31
     ; check: iadd v0, v2
     ; check: sshr_imm v3, 1
-    ; check: copy v4
+    ; check: v1 -> v4
     return v1
 }
 
@@ -126,8 +126,8 @@ ebb0(v0: i32):
     ; check: v2 = sshr_imm v0, 1
     ; check: ushr_imm v2, 30
     ; check: iadd v0, v3
-    ; check: sshr_imm v4, 2
-    ; check: copy v5
+    ; check: v5 = sshr_imm v4, 2
+    ; check: v1 -> v5
 
     return v1
 }
@@ -151,8 +151,8 @@ ebb0(v0: i32):
     ; check: sshr_imm v0, 29
     ; check: ushr_imm v2, 2
     ; check: iadd v0, v3
-    ; check: sshr_imm v4, 30
-    ; check: copy v5
+    ; check: v5 = sshr_imm v4, 30
+    ; check: v1 -> v5
     return v1
 }
 
@@ -214,8 +214,8 @@ ebb0(v0: i64):
     v1 = sdiv_imm v0, 2
     ; check: ushr_imm v0, 63
     ; check: iadd v0, v2
-    ; check: sshr_imm v3, 1
-    ; check: copy v4
+    ; check: v4 = sshr_imm v3, 1
+    ; check: v1 -> v4
     return v1
 }
 
@@ -237,8 +237,8 @@ ebb0(v0: i64):
     ; check: sshr_imm v0, 1
     ; check: ushr_imm v2, 62
     ; check: iadd v0, v3
-    ; check: sshr_imm v4, 2
-    ; check: copy v5
+    ; check: v5 = sshr_imm v4, 2
+    ; check: v1 -> v5
     return v1
 }
 
@@ -261,8 +261,8 @@ ebb0(v0: i64):
     ; check: sshr_imm v0, 61
     ; check: ushr_imm v2, 2
     ; check: iadd v0, v3
-    ; check: sshr_imm v4, 62
-    ; check: copy v5
+    ; check: v5 = sshr_imm v4, 62
+    ; check: v1 -> v5
     return v1
 }
 

From 19257f80c1d7812a6790cb74be8edf33631a4e9f Mon Sep 17 00:00:00 2001
From: Carmen Kwan 
Date: Tue, 23 Jul 2019 16:28:54 -0700
Subject: [PATCH 2573/3084] Add reference types R32 and R64

-Add resumable_trap, safepoint, isnull, and null instructions
-Add Stackmap struct and StackmapSink trait

Co-authored-by: Mir Ahmed 
Co-authored-by: Dan Gohman 
---
 .../codegen/meta/src/cdsl/instructions.rs     |  37 +++++-
 cranelift/codegen/meta/src/cdsl/types.rs      | 100 +++++++++++++-
 cranelift/codegen/meta/src/cdsl/typevar.rs    |  42 +++++-
 cranelift/codegen/meta/src/gen_binemit.rs     |  12 +-
 cranelift/codegen/meta/src/gen_inst.rs        |   4 +
 cranelift/codegen/meta/src/gen_types.rs       |   5 +
 .../codegen/meta/src/isa/x86/encodings.rs     |  50 +++++++
 cranelift/codegen/meta/src/isa/x86/recipes.rs |  37 ++++++
 .../codegen/meta/src/shared/instructions.rs   |  71 +++++++++-
 cranelift/codegen/meta/src/shared/settings.rs |  12 ++
 cranelift/codegen/meta/src/shared/types.rs    |  40 ++++++
 cranelift/codegen/src/binemit/memorysink.rs   |  28 +++-
 cranelift/codegen/src/binemit/mod.rs          |  20 ++-
 cranelift/codegen/src/binemit/stackmap.rs     | 122 ++++++++++++++++++
 cranelift/codegen/src/context.rs              |  12 +-
 cranelift/codegen/src/ir/instructions.rs      |  13 ++
 cranelift/codegen/src/ir/types.rs             |  29 ++++-
 cranelift/codegen/src/isa/arm32/binemit.rs    |   1 +
 cranelift/codegen/src/isa/arm32/mod.rs        |   4 +-
 cranelift/codegen/src/isa/arm64/binemit.rs    |   1 +
 cranelift/codegen/src/isa/arm64/mod.rs        |   4 +-
 cranelift/codegen/src/isa/riscv/binemit.rs    |   2 +-
 cranelift/codegen/src/isa/riscv/mod.rs        |   4 +-
 cranelift/codegen/src/isa/x86/binemit.rs      |   2 +-
 cranelift/codegen/src/isa/x86/mod.rs          |   4 +-
 cranelift/codegen/src/regalloc/context.rs     |  15 +++
 cranelift/codegen/src/regalloc/mod.rs         |   2 +
 cranelift/codegen/src/regalloc/safepoint.rs   |  72 +++++++++++
 cranelift/codegen/src/settings.rs             |   1 +
 cranelift/codegen/src/verifier/mod.rs         |  28 +++-
 cranelift/faerie/src/backend.rs               |  21 ++-
 .../filetests/filetests/safepoint/basic.clif  |  55 ++++++++
 .../filetests/filetests/safepoint/call.clif   |  52 ++++++++
 cranelift/filetests/src/lib.rs                |   2 +
 cranelift/filetests/src/test_binemit.rs       |   9 +-
 cranelift/filetests/src/test_compile.rs       |  11 +-
 cranelift/filetests/src/test_safepoint.rs     |  39 ++++++
 cranelift/frontend/src/ssa.rs                 |   2 +
 cranelift/reader/src/lexer.rs                 |   2 +
 cranelift/simplejit/src/backend.rs            |  40 +++++-
 cranelift/src/bugpoint.rs                     |  13 +-
 cranelift/src/compile.rs                      |   6 +-
 cranelift/src/disasm.rs                       |  25 +++-
 cranelift/src/wasm.rs                         |   7 +-
 cranelift/wasm/src/code_translator.rs         |   9 +-
 cranelift/wasm/src/environ/spec.rs            |  11 ++
 cranelift/wasm/src/func_translator.rs         |  11 +-
 47 files changed, 1027 insertions(+), 62 deletions(-)
 create mode 100644 cranelift/codegen/src/binemit/stackmap.rs
 create mode 100644 cranelift/codegen/src/regalloc/safepoint.rs
 create mode 100644 cranelift/filetests/filetests/safepoint/basic.clif
 create mode 100644 cranelift/filetests/filetests/safepoint/call.clif
 create mode 100644 cranelift/filetests/src/test_safepoint.rs

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 00a166fb64..43a879066c 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -12,7 +12,7 @@ use crate::cdsl::formats::{
 };
 use crate::cdsl::operands::Operand;
 use crate::cdsl::type_inference::Constraint;
-use crate::cdsl::types::{LaneType, ValueType, VectorType};
+use crate::cdsl::types::{LaneType, ReferenceType, ValueType, VectorType};
 use crate::cdsl::typevar::TypeVar;
 
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
@@ -177,6 +177,10 @@ impl Instruction {
         bind(self.clone(), Some(lane_type.into()), Vec::new())
     }
 
+    pub fn bind_ref(&self, reference_type: impl Into) -> BoundInstruction {
+        bind_ref(self.clone(), Some(reference_type.into()), Vec::new())
+    }
+
     pub fn bind_vector(&self, lane_type: impl Into, num_lanes: u64) -> BoundInstruction {
         bind_vector(self.clone(), lane_type.into(), num_lanes, Vec::new())
     }
@@ -406,6 +410,10 @@ impl BoundInstruction {
         bind(self.inst, Some(lane_type.into()), self.value_types)
     }
 
+    pub fn bind_ref(self, reference_type: impl Into) -> BoundInstruction {
+        bind_ref(self.inst, Some(reference_type.into()), self.value_types)
+    }
+
     pub fn bind_vector(self, lane_type: impl Into, num_lanes: u64) -> BoundInstruction {
         bind_vector(self.inst, lane_type.into(), num_lanes, self.value_types)
     }
@@ -1043,6 +1051,13 @@ impl InstSpec {
             InstSpec::Bound(inst) => inst.clone().bind(lane_type),
         }
     }
+
+    pub fn bind_ref(&self, reference_type: impl Into) -> BoundInstruction {
+        match self {
+            InstSpec::Inst(inst) => inst.bind_ref(reference_type),
+            InstSpec::Bound(inst) => inst.clone().bind_ref(reference_type),
+        }
+    }
 }
 
 impl Into for &Instruction {
@@ -1077,6 +1092,26 @@ fn bind(
     BoundInstruction { inst, value_types }
 }
 
+/// Helper bind for reference types reused by {Bound,}Instruction::bind_ref.
+fn bind_ref(
+    inst: Instruction,
+    reference_type: Option,
+    mut value_types: Vec,
+) -> BoundInstruction {
+    match reference_type {
+        Some(reference_type) => {
+            value_types.push(ValueTypeOrAny::ValueType(reference_type.into()));
+        }
+        None => {
+            value_types.push(ValueTypeOrAny::Any);
+        }
+    }
+
+    verify_polymorphic_binding(&inst, &value_types);
+
+    BoundInstruction { inst, value_types }
+}
+
 /// Helper bind for vector types reused by {Bound,}Instruction::bind.
 fn bind_vector(
     inst: Instruction,
diff --git a/cranelift/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
index 21bf0161c4..eba239d1d7 100644
--- a/cranelift/codegen/meta/src/cdsl/types.rs
+++ b/cranelift/codegen/meta/src/cdsl/types.rs
@@ -11,12 +11,14 @@ use crate::shared::types as shared_types;
 //
 // 0: Void
 // 0x01-0x6f: Special types
-// 0x70-0x7f: Lane types
+// 0x70-0x7d: Lane types
+// 0x7e-0x7f: Reference types
 // 0x80-0xff: Vector types
 //
 // Vector types are encoded with the lane type in the low 4 bits and log2(lanes)
 // in the high 4 bits, giving a range of 2-256 lanes.
 static LANE_BASE: u8 = 0x70;
+static REFERENCE_BASE: u8 = 0x7E;
 
 // Rust name prefix used for the `rust_name` method.
 static _RUST_NAME_PREFIX: &'static str = "ir::types::";
@@ -31,6 +33,7 @@ static _RUST_NAME_PREFIX: &'static str = "ir::types::";
 pub enum ValueType {
     BV(BVType),
     Lane(LaneType),
+    Reference(ReferenceType),
     Special(SpecialType),
     Vector(VectorType),
 }
@@ -46,11 +49,16 @@ impl ValueType {
         SpecialTypeIterator::new()
     }
 
+    pub fn all_reference_types() -> ReferenceTypeIterator {
+        ReferenceTypeIterator::new()
+    }
+
     /// Return a string containing the documentation comment for this type.
     pub fn doc(&self) -> String {
         match *self {
             ValueType::BV(ref b) => b.doc(),
             ValueType::Lane(l) => l.doc(),
+            ValueType::Reference(r) => r.doc(),
             ValueType::Special(s) => s.doc(),
             ValueType::Vector(ref v) => v.doc(),
         }
@@ -61,6 +69,7 @@ impl ValueType {
         match *self {
             ValueType::BV(ref b) => b.lane_bits(),
             ValueType::Lane(l) => l.lane_bits(),
+            ValueType::Reference(r) => r.lane_bits(),
             ValueType::Special(s) => s.lane_bits(),
             ValueType::Vector(ref v) => v.lane_bits(),
         }
@@ -84,6 +93,7 @@ impl ValueType {
         match *self {
             ValueType::BV(_) => None,
             ValueType::Lane(l) => Some(l.number()),
+            ValueType::Reference(r) => Some(r.number()),
             ValueType::Special(s) => Some(s.number()),
             ValueType::Vector(ref v) => Some(v.number()),
         }
@@ -112,6 +122,7 @@ impl fmt::Display for ValueType {
         match *self {
             ValueType::BV(ref b) => b.fmt(f),
             ValueType::Lane(l) => l.fmt(f),
+            ValueType::Reference(r) => r.fmt(f),
             ValueType::Special(s) => s.fmt(f),
             ValueType::Vector(ref v) => v.fmt(f),
         }
@@ -132,6 +143,13 @@ impl From for ValueType {
     }
 }
 
+/// Create a ValueType from a given reference type.
+impl From for ValueType {
+    fn from(reference: ReferenceType) -> Self {
+        ValueType::Reference(reference)
+    }
+}
+
 /// Create a ValueType from a given special type.
 impl From for ValueType {
     fn from(spec: SpecialType) -> Self {
@@ -515,3 +533,83 @@ impl Iterator for SpecialTypeIterator {
         }
     }
 }
+
+/// Reference type is scalar type, but not lane type.
+#[derive(Clone, Copy, PartialEq, Eq, Hash)]
+pub struct ReferenceType(pub shared_types::Reference);
+
+impl ReferenceType {
+    /// Return a string containing the documentation comment for this reference type.
+    pub fn doc(self) -> String {
+        format!("An opaque reference type with {} bits.", self.lane_bits())
+    }
+
+    /// Return the number of bits in a lane.
+    pub fn lane_bits(self) -> u64 {
+        match self.0 {
+            shared_types::Reference::R32 => 32,
+            shared_types::Reference::R64 => 64,
+        }
+    }
+
+    /// Find the unique number associated with this reference type.
+    pub fn number(self) -> u8 {
+        REFERENCE_BASE
+            + match self {
+                ReferenceType(shared_types::Reference::R32) => 0,
+                ReferenceType(shared_types::Reference::R64) => 1,
+            }
+    }
+
+    pub fn ref_from_bits(num_bits: u16) -> ReferenceType {
+        ReferenceType(match num_bits {
+            32 => shared_types::Reference::R32,
+            64 => shared_types::Reference::R64,
+            _ => unreachable!("unexpected number of bits for a reference type"),
+        })
+    }
+}
+
+impl fmt::Display for ReferenceType {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "r{}", self.lane_bits())
+    }
+}
+
+impl fmt::Debug for ReferenceType {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "ReferenceType(bits={})", self.lane_bits())
+    }
+}
+
+/// Create a ReferenceType from a given reference variant.
+impl From for ReferenceType {
+    fn from(r: shared_types::Reference) -> Self {
+        ReferenceType(r)
+    }
+}
+
+/// An iterator for different reference types.
+pub struct ReferenceTypeIterator {
+    reference_iter: shared_types::ReferenceIterator,
+}
+
+impl ReferenceTypeIterator {
+    /// Create a new reference type iterator.
+    fn new() -> Self {
+        Self {
+            reference_iter: shared_types::ReferenceIterator::new(),
+        }
+    }
+}
+
+impl Iterator for ReferenceTypeIterator {
+    type Item = ReferenceType;
+    fn next(&mut self) -> Option {
+        if let Some(r) = self.reference_iter.next() {
+            Some(ReferenceType::from(r))
+        } else {
+            None
+        }
+    }
+}
diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 22640b6b9c..9ae4c33fd8 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -6,7 +6,7 @@ use std::iter::FromIterator;
 use std::ops;
 use std::rc::Rc;
 
-use crate::cdsl::types::{BVType, LaneType, SpecialType, ValueType};
+use crate::cdsl::types::{BVType, LaneType, ReferenceType, SpecialType, ValueType};
 
 const MAX_LANES: u16 = 256;
 const MAX_BITS: u16 = 64;
@@ -64,6 +64,10 @@ impl TypeVar {
             ValueType::Special(special_type) => {
                 return TypeVar::new(name, doc, builder.specials(vec![special_type]).build());
             }
+            ValueType::Reference(ReferenceType(reference_type)) => {
+                let bits = reference_type as RangeBound;
+                return TypeVar::new(name, doc, builder.refs(bits..bits).build());
+            }
             ValueType::Lane(lane_type) => (lane_type, 1),
             ValueType::Vector(vec_type) => {
                 (vec_type.lane_type(), vec_type.lane_count() as RangeBound)
@@ -406,6 +410,7 @@ pub struct TypeSet {
     pub ints: NumSet,
     pub floats: NumSet,
     pub bools: NumSet,
+    pub refs: NumSet,
     pub bitvecs: NumSet,
     pub specials: Vec,
 }
@@ -416,6 +421,7 @@ impl TypeSet {
         ints: NumSet,
         floats: NumSet,
         bools: NumSet,
+        refs: NumSet,
         bitvecs: NumSet,
         specials: Vec,
     ) -> Self {
@@ -424,6 +430,7 @@ impl TypeSet {
             ints,
             floats,
             bools,
+            refs,
             bitvecs,
             specials,
         }
@@ -432,7 +439,11 @@ impl TypeSet {
     /// Return the number of concrete types represented by this typeset.
     pub fn size(&self) -> usize {
         self.lanes.len()
-            * (self.ints.len() + self.floats.len() + self.bools.len() + self.bitvecs.len())
+            * (self.ints.len()
+                + self.floats.len()
+                + self.bools.len()
+                + self.refs.len()
+                + self.bitvecs.len())
             + self.specials.len()
     }
 
@@ -462,6 +473,7 @@ impl TypeSet {
         let mut copy = self.clone();
         copy.ints = NumSet::new();
         copy.floats = NumSet::new();
+        copy.refs = NumSet::new();
         copy.bitvecs = NumSet::new();
         if (&self.lanes - &num_set![1]).len() > 0 {
             copy.bools = &self.ints | &self.floats;
@@ -544,6 +556,7 @@ impl TypeSet {
         copy.ints = NumSet::new();
         copy.bools = NumSet::new();
         copy.floats = NumSet::new();
+        copy.refs = NumSet::new();
         copy.bitvecs = self
             .lanes
             .iter()
@@ -568,6 +581,9 @@ impl TypeSet {
             for &bits in &self.bools {
                 ret.push(LaneType::bool_from_bits(bits).by(num_lanes));
             }
+            for &bits in &self.refs {
+                ret.push(ReferenceType::ref_from_bits(bits).into());
+            }
             for &bits in &self.bitvecs {
                 assert_eq!(num_lanes, 1);
                 ret.push(BVType::new(bits).into());
@@ -630,6 +646,7 @@ impl TypeSet {
                 let mut ints = range_to_set(Some(8..MAX_BITS));
                 let mut floats = range_to_set(Some(32..64));
                 let mut bools = range_to_set(Some(1..MAX_BITS));
+                let refs = range_to_set(Some(32..64));
 
                 for &l in &all_lanes {
                     for &i in &all_ints {
@@ -654,7 +671,7 @@ impl TypeSet {
 
                 let bitvecs = NumSet::new();
                 let specials = Vec::new();
-                TypeSet::new(lanes, ints, floats, bools, bitvecs, specials)
+                TypeSet::new(lanes, ints, floats, bools, refs, bitvecs, specials)
             }
         }
     }
@@ -664,6 +681,7 @@ impl TypeSet {
         self.ints = &self.ints & &other.ints;
         self.floats = &self.floats & &other.floats;
         self.bools = &self.bools & &other.bools;
+        self.refs = &self.refs & &other.refs;
         self.bitvecs = &self.bitvecs & &other.bitvecs;
 
         let mut new_specials = Vec::new();
@@ -680,6 +698,7 @@ impl TypeSet {
             && self.ints.is_subset(&other.ints)
             && self.floats.is_subset(&other.floats)
             && self.bools.is_subset(&other.bools)
+            && self.refs.is_subset(&other.refs)
             && self.bitvecs.is_subset(&other.bitvecs)
             && {
                 let specials: HashSet = HashSet::from_iter(self.specials.clone());
@@ -692,12 +711,14 @@ impl TypeSet {
         set_wider_or_equal(&self.ints, &other.ints)
             && set_wider_or_equal(&self.floats, &other.floats)
             && set_wider_or_equal(&self.bools, &other.bools)
+            && set_wider_or_equal(&self.refs, &other.refs)
     }
 
     pub fn is_narrower(&self, other: &TypeSet) -> bool {
         set_narrower(&self.ints, &other.ints)
             && set_narrower(&self.floats, &other.floats)
             && set_narrower(&self.bools, &other.bools)
+            && set_narrower(&self.refs, &other.refs)
     }
 }
 
@@ -738,6 +759,12 @@ impl fmt::Debug for TypeSet {
                 Vec::from_iter(self.bools.iter().map(|x| x.to_string())).join(", ")
             ));
         }
+        if !self.refs.is_empty() {
+            subsets.push(format!(
+                "refs={{{}}}",
+                Vec::from_iter(self.refs.iter().map(|x| x.to_string())).join(", ")
+            ));
+        }
         if !self.bitvecs.is_empty() {
             subsets.push(format!(
                 "bitvecs={{{}}}",
@@ -760,6 +787,7 @@ pub struct TypeSetBuilder {
     ints: Interval,
     floats: Interval,
     bools: Interval,
+    refs: Interval,
     bitvecs: Interval,
     includes_scalars: bool,
     simd_lanes: Interval,
@@ -772,6 +800,7 @@ impl TypeSetBuilder {
             ints: Interval::None,
             floats: Interval::None,
             bools: Interval::None,
+            refs: Interval::None,
             bitvecs: Interval::None,
             includes_scalars: true,
             simd_lanes: Interval::None,
@@ -794,6 +823,11 @@ impl TypeSetBuilder {
         self.bools = interval.into();
         self
     }
+    pub fn refs(mut self, interval: impl Into) -> Self {
+        assert!(self.refs == Interval::None);
+        self.refs = interval.into();
+        self
+    }
     pub fn includes_scalars(mut self, includes_scalars: bool) -> Self {
         self.includes_scalars = includes_scalars;
         self
@@ -827,6 +861,7 @@ impl TypeSetBuilder {
             range_to_set(self.ints.to_range(8..MAX_BITS, None)),
             range_to_set(self.floats.to_range(32..64, None)),
             bools,
+            range_to_set(self.refs.to_range(32..64, None)),
             range_to_set(self.bitvecs.to_range(1..MAX_BITVEC, None)),
             self.specials,
         )
@@ -837,6 +872,7 @@ impl TypeSetBuilder {
             .ints(Interval::All)
             .floats(Interval::All)
             .bools(Interval::All)
+            .refs(Interval::All)
             .simd_lanes(Interval::All)
             .bitvecs(Interval::All)
             .specials(ValueType::all_special_types().collect())
diff --git a/cranelift/codegen/meta/src/gen_binemit.rs b/cranelift/codegen/meta/src/gen_binemit.rs
index 71684ffb55..1bf95a83e0 100644
--- a/cranelift/codegen/meta/src/gen_binemit.rs
+++ b/cranelift/codegen/meta/src/gen_binemit.rs
@@ -18,10 +18,12 @@ fn gen_recipe(formats: &FormatRegistry, recipe: &EncodingRecipe, fmt: &mut Forma
     let inst_format = formats.get(recipe.format);
     let num_value_ops = inst_format.num_value_operands;
 
-    let want_args = recipe.operands_in.iter().any(|c| match c {
-        OperandConstraint::RegClass(_) | OperandConstraint::Stack(_) => true,
-        OperandConstraint::FixedReg(_) | OperandConstraint::TiedInput(_) => false,
-    });
+    // TODO: Set want_args to true for only MultiAry instructions instead of all formats with value list.
+    let want_args = inst_format.has_value_list
+        || recipe.operands_in.iter().any(|c| match c {
+            OperandConstraint::RegClass(_) | OperandConstraint::Stack(_) => true,
+            OperandConstraint::FixedReg(_) | OperandConstraint::TiedInput(_) => false,
+        });
     assert!(!want_args || num_value_ops > 0 || inst_format.has_value_list);
 
     let want_outs = recipe.operands_out.iter().any(|c| match c {
@@ -159,6 +161,7 @@ fn gen_isa(formats: &FormatRegistry, isa_name: &str, recipes: &Recipes, fmt: &mu
             fmt.line("inst: Inst,");
             fmt.line("_divert: &mut RegDiversions,");
             fmt.line("_sink: &mut CS,");
+            fmt.line("_isa: &dyn TargetIsa,");
         });
         fmt.line(") {");
         fmt.indent(|fmt| {
@@ -176,6 +179,7 @@ fn gen_isa(formats: &FormatRegistry, isa_name: &str, recipes: &Recipes, fmt: &mu
         fmt.line("inst: Inst,");
         fmt.line("divert: &mut RegDiversions,");
         fmt.line("sink: &mut CS,");
+        fmt.line("isa: &dyn TargetIsa,")
     });
 
     fmt.line(") {");
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index a3c2a9fccd..5dbb9dc095 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -650,6 +650,9 @@ fn typeset_to_string(ts: &TypeSet) -> String {
     if ts.specials.len() > 0 {
         result += &format!(", specials=[{}]", iterable_to_string(&ts.specials));
     }
+    if ts.refs.len() > 0 {
+        result += &format!(", refs={}", iterable_to_string(&ts.refs));
+    }
     result += ")";
     result
 }
@@ -677,6 +680,7 @@ pub fn gen_typesets_table(type_sets: &UniqueTable, fmt: &mut Formatter)
                 gen_bitset(&ts.ints, "ints", 8, fmt);
                 gen_bitset(&ts.floats, "floats", 8, fmt);
                 gen_bitset(&ts.bools, "bools", 8, fmt);
+                gen_bitset(&ts.refs, "refs", 8, fmt);
             });
             fmt.line("},");
         }
diff --git a/cranelift/codegen/meta/src/gen_types.rs b/cranelift/codegen/meta/src/gen_types.rs
index 0a52eb371e..d4b4c60d65 100644
--- a/cranelift/codegen/meta/src/gen_types.rs
+++ b/cranelift/codegen/meta/src/gen_types.rs
@@ -54,6 +54,11 @@ fn emit_types(fmt: &mut srcgen::Formatter) -> Result<(), error::Error> {
         emit_type(&ty, fmt)?;
     }
 
+    // Emit all reference types.
+    for ty in cdsl_types::ValueType::all_reference_types().map(cdsl_types::ValueType::from) {
+        emit_type(&ty, fmt)?;
+    }
+
     // Emit vector definitions for common SIMD sizes.
     for vec_size in &[64_u64, 128, 256, 512] {
         emit_vectors(*vec_size, fmt)?;
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 7ddde02bec..eaa5614bd0 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -13,6 +13,7 @@ use crate::cdsl::types::ValueType;
 use crate::shared::types::Bool::{B1, B16, B32, B64, B8};
 use crate::shared::types::Float::{F32, F64};
 use crate::shared::types::Int::{I16, I32, I64, I8};
+use crate::shared::types::Reference::{R32, R64};
 use crate::shared::Definitions as SharedDefinitions;
 
 use super::recipes::{RecipeGroup, Template};
@@ -175,6 +176,20 @@ impl PerCpuModeEncodings {
         });
     }
 
+    /// Add encodings for `inst.r32` to X86_32.
+    /// Add encodings for `inst.r32` to X86_64 with and without REX.
+    /// Add encodings for `inst.r64` to X86_64 with a REX.W prefix.
+    fn enc_r32_r64(&mut self, inst: impl Into, template: Template) {
+        let inst: InstSpec = inst.into();
+        self.enc32(inst.bind_ref(R32), template.nonrex());
+
+        // REX-less encoding must come after REX encoding so we don't use it by default. Otherwise
+        // reg-alloc would never use r8 and up.
+        self.enc64(inst.bind_ref(R32), template.rex());
+        self.enc64(inst.bind_ref(R32), template.nonrex());
+        self.enc64(inst.bind_ref(R64), template.rex().w());
+    }
+
     /// Add encodings for `inst` to X86_64 with and without a REX prefix.
     fn enc_x86_64(&mut self, inst: impl Into + Clone, template: Template) {
         // See above comment about the ordering of rex vs non-rex encodings.
@@ -331,6 +346,7 @@ pub fn define(
     let ireduce = shared.by_name("ireduce");
     let ishl = shared.by_name("ishl");
     let ishl_imm = shared.by_name("ishl_imm");
+    let is_null = shared.by_name("is_null");
     let istore16 = shared.by_name("istore16");
     let istore16_complex = shared.by_name("istore16_complex");
     let istore32 = shared.by_name("istore32");
@@ -344,6 +360,7 @@ pub fn define(
     let load = shared.by_name("load");
     let load_complex = shared.by_name("load_complex");
     let nearest = shared.by_name("nearest");
+    let null = shared.by_name("null");
     let popcnt = shared.by_name("popcnt");
     let raw_bitcast = shared.by_name("raw_bitcast");
     let regfill = shared.by_name("regfill");
@@ -354,6 +371,7 @@ pub fn define(
     let rotl_imm = shared.by_name("rotl_imm");
     let rotr = shared.by_name("rotr");
     let rotr_imm = shared.by_name("rotr_imm");
+    let safepoint = shared.by_name("safepoint");
     let scalar_to_vector = shared.by_name("scalar_to_vector");
     let selectif = shared.by_name("selectif");
     let sextend = shared.by_name("sextend");
@@ -374,6 +392,7 @@ pub fn define(
     let trap = shared.by_name("trap");
     let trapff = shared.by_name("trapff");
     let trapif = shared.by_name("trapif");
+    let resumable_trap = shared.by_name("resumable_trap");
     let trueff = shared.by_name("trueff");
     let trueif = shared.by_name("trueif");
     let trunc = shared.by_name("trunc");
@@ -455,6 +474,7 @@ pub fn define(
     let rec_icscc_ib = r.template("icscc_ib");
     let rec_icscc_id = r.template("icscc_id");
     let rec_indirect_jmp = r.template("indirect_jmp");
+    let rec_is_zero = r.template("is_zero");
     let rec_jmpb = r.template("jmpb");
     let rec_jmpd = r.template("jmpd");
     let rec_jt_base = r.template("jt_base");
@@ -473,6 +493,7 @@ pub fn define(
     let rec_popq = r.template("popq");
     let rec_pu_id = r.template("pu_id");
     let rec_pu_id_bool = r.template("pu_id_bool");
+    let rec_pu_id_ref = r.template("pu_id_ref");
     let rec_pu_iq = r.template("pu_iq");
     let rec_pushq = r.template("pushq");
     let rec_ret = r.template("ret");
@@ -492,6 +513,7 @@ pub fn define(
     let rec_rmov = r.template("rmov");
     let rec_rr = r.template("rr");
     let rec_rrx = r.template("rrx");
+    let rec_safepoint = r.recipe("safepoint");
     let rec_setf_abcd = r.template("setf_abcd");
     let rec_seti_abcd = r.template("seti_abcd");
     let rec_spaddr4_id = r.template("spaddr4_id");
@@ -566,6 +588,7 @@ pub fn define(
     e.enc_i32_i64(x86_umulx, rec_mulx.opcodes(vec![0xf7]).rrr(4));
 
     e.enc_i32_i64(copy, rec_umr.opcodes(vec![0x89]));
+    e.enc_r32_r64(copy, rec_umr.opcodes(vec![0x89]));
     e.enc_both(copy.bind(B1), rec_umr.opcodes(vec![0x89]));
     e.enc_both(copy.bind(I8), rec_umr.opcodes(vec![0x89]));
     e.enc_both(copy.bind(I16), rec_umr.opcodes(vec![0x89]));
@@ -579,6 +602,12 @@ pub fn define(
     e.enc64(regmove.bind(I64), rec_rmov.opcodes(vec![0x89]).rex().w());
     e.enc_both(regmove.bind(B1), rec_rmov.opcodes(vec![0x89]));
     e.enc_both(regmove.bind(I8), rec_rmov.opcodes(vec![0x89]));
+    e.enc32(regmove.bind_ref(R32), rec_rmov.opcodes(vec![0x89]));
+    e.enc64(regmove.bind_ref(R32), rec_rmov.opcodes(vec![0x89]).rex());
+    e.enc64(
+        regmove.bind_ref(R64),
+        rec_rmov.opcodes(vec![0x89]).rex().w(),
+    );
 
     e.enc_i32_i64(iadd_imm, rec_r_ib.opcodes(vec![0x83]).rrr(0));
     e.enc_i32_i64(iadd_imm, rec_r_id.opcodes(vec![0x81]).rrr(0));
@@ -841,6 +870,8 @@ pub fn define(
 
     e.enc_i32_i64(spill, rec_spillSib32.opcodes(vec![0x89]));
     e.enc_i32_i64(regspill, rec_regspill32.opcodes(vec![0x89]));
+    e.enc_r32_r64(spill, rec_spillSib32.opcodes(vec![0x89]));
+    e.enc_r32_r64(regspill, rec_regspill32.opcodes(vec![0x89]));
 
     // Use a 32-bit write for spilling `b1`, `i8` and `i16` to avoid
     // constraining the permitted registers.
@@ -865,6 +896,8 @@ pub fn define(
 
     e.enc_i32_i64(fill, rec_fillSib32.opcodes(vec![0x8b]));
     e.enc_i32_i64(regfill, rec_regfill32.opcodes(vec![0x8b]));
+    e.enc_r32_r64(fill, rec_fillSib32.opcodes(vec![0x8b]));
+    e.enc_r32_r64(regfill, rec_regfill32.opcodes(vec![0x8b]));
 
     // Load 32 bits from `b1`, `i8` and `i16` spill slots. See `spill.b1` above.
 
@@ -1248,6 +1281,8 @@ pub fn define(
     // Trap as ud2
     e.enc32(trap, rec_trap.opcodes(vec![0x0f, 0x0b]));
     e.enc64(trap, rec_trap.opcodes(vec![0x0f, 0x0b]));
+    e.enc32(resumable_trap, rec_trap.opcodes(vec![0x0f, 0x0b]));
+    e.enc64(resumable_trap, rec_trap.opcodes(vec![0x0f, 0x0b]));
 
     // Debug trap as int3
     e.enc32_rec(debugtrap, rec_debugtrap, 0);
@@ -1666,5 +1701,20 @@ pub fn define(
         }
     }
 
+    // Reference type instructions
+
+    // Null references implemented as iconst 0.
+    e.enc32(null.bind_ref(R32), rec_pu_id_ref.opcodes(vec![0xb8]));
+
+    e.enc64(null.bind_ref(R64), rec_pu_id_ref.rex().opcodes(vec![0xb8]));
+    e.enc64(null.bind_ref(R64), rec_pu_id_ref.opcodes(vec![0xb8]));
+
+    // is_null, implemented by testing whether the value is 0.
+    e.enc_r32_r64(is_null, rec_is_zero.opcodes(vec![0x85]));
+
+    // safepoint instruction calls sink, no actual encoding.
+    e.enc32_rec(safepoint, rec_safepoint, 0);
+    e.enc64_rec(safepoint, rec_safepoint, 0);
+
     e
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 11063e39b9..b948c0c2e5 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -888,6 +888,20 @@ pub fn define<'shared>(
             ),
     );
 
+    // XX+rd id nullary with 0 as 32-bit immediate. Note no recipe predicate.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("pu_id_ref", f_nullary, 4)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    // The destination register is encoded in the low bits of the opcode.
+                    // No ModR/M.
+                    {{PUT_OP}}(bits | (out_reg0 & 7), rex1(out_reg0), sink);
+                    sink.put4(0);
+                "#,
+            ),
+    );
+
     // XX+rd iq unary with 64-bit immediate.
     recipes.add_template_recipe(
         EncodingRecipeBuilder::new("pu_iq", f_unary_imm, 8)
@@ -2851,5 +2865,28 @@ pub fn define<'shared>(
             ),
     );
 
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("is_zero", f_unary, 2 + 2)
+            .operands_in(vec![gpr])
+            .operands_out(vec![abcd])
+            .emit(
+                r#"
+                    // Test instruction.
+                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg0), sink);
+                    modrm_rr(in_reg0, in_reg0, sink);
+                    // Check ZF = 1 flag to see if register holds 0.
+                    sink.put1(0x0f);
+                    sink.put1(0x94);
+                    modrm_rr(out_reg0, 0, sink);
+                "#,
+            ),
+    );
+
+    recipes.add_recipe(EncodingRecipeBuilder::new("safepoint", f_multiary, 0).emit(
+        r#"
+            sink.add_stackmap(args, func, isa);
+        "#,
+    ));
+
     recipes
 }
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 36a1a89fb5..dfeeec4906 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -85,6 +85,12 @@ pub fn define(
         TypeSetBuilder::new().ints(32..64).build(),
     );
 
+    let Ref = &TypeVar::new(
+        "Ref",
+        "A scalar reference type",
+        TypeSetBuilder::new().refs(Interval::All).build(),
+    );
+
     let Testable = &TypeVar::new(
         "Testable",
         "A scalar boolean or integer type",
@@ -118,11 +124,12 @@ pub fn define(
 
     let Any = &TypeVar::new(
         "Any",
-        "Any integer, float, or boolean scalar or vector type",
+        "Any integer, float, boolean, or reference scalar or vector type",
         TypeSetBuilder::new()
             .ints(Interval::All)
             .floats(Interval::All)
             .bools(Interval::All)
+            .refs(Interval::All)
             .simd_lanes(Interval::All)
             .includes_scalars(true)
             .build(),
@@ -394,6 +401,19 @@ pub fn define(
         .can_trap(true),
     );
 
+    ig.push(
+        Inst::new(
+            "resumable_trap",
+            r#"
+        A resumable trap.
+        
+        This instruction allows non-conditional traps to be used as non-terminal instructions.
+        "#,
+        )
+        .operands_in(vec![code])
+        .can_trap(true),
+    );
+
     ig.push(
         Inst::new(
             "trapnz",
@@ -1068,6 +1088,20 @@ pub fn define(
         .operands_out(vec![a]),
     );
 
+    let a = &operand_doc("a", Ref, "A constant reference null value");
+
+    ig.push(
+        Inst::new(
+            "null",
+            r#"
+        Null constant value for reference types.
+
+        Create a scalar reference SSA value with a constant null value.
+        "#,
+        )
+        .operands_out(vec![a]),
+    );
+
     ig.push(Inst::new(
         "nop",
         r#"
@@ -1315,6 +1349,24 @@ pub fn define(
         .other_side_effects(true),
     );
 
+    let N = &operand_doc(
+        "args",
+        variable_args,
+        "Variable number of args for Stackmap",
+    );
+
+    ig.push(
+        Inst::new(
+            "safepoint",
+            r#"
+        This instruction will provide live reference values at a point in
+        the function. It can only be used by the compiler.
+        "#,
+        )
+        .operands_in(vec![N])
+        .other_side_effects(true),
+    );
+
     let x = &operand_doc("x", TxN, "Vector to split");
     let lo = &operand_doc("lo", &TxN.half_vector(), "Low-numbered lanes of `x`");
     let hi = &operand_doc("hi", &TxN.half_vector(), "High-numbered lanes of `x`");
@@ -2578,6 +2630,23 @@ pub fn define(
         .operands_out(vec![a]),
     );
 
+    let a = &operand("a", b1);
+    let x = &operand("x", Ref);
+
+    ig.push(
+        Inst::new(
+            "is_null",
+            r#"
+        Reference verification.
+
+        The condition code determines if the reference type in question is
+        null or not.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a]),
+    );
+
     let Cond = &operand("Cond", intcc);
     let f = &operand("f", iflags);
     let a = &operand("a", b1);
diff --git a/cranelift/codegen/meta/src/shared/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
index 14ace5de14..6561ee9856 100644
--- a/cranelift/codegen/meta/src/shared/settings.rs
+++ b/cranelift/codegen/meta/src/shared/settings.rs
@@ -92,6 +92,18 @@ pub fn define() -> SettingGroup {
         true,
     );
 
+    settings.add_bool(
+        "enable_safepoints",
+        r#"
+            Enable safepoint instruction insertions.
+
+            This will allow the emit_stackmaps() function to insert the safepoint
+            instruction on top of calls and interrupt traps in order to display the
+            live reference values at that point in the program.
+            "#,
+        false,
+    );
+
     // Settings specific to the `baldrdash` calling convention.
 
     settings.add_enum(
diff --git a/cranelift/codegen/meta/src/shared/types.rs b/cranelift/codegen/meta/src/shared/types.rs
index b327da67db..266c30b3d8 100644
--- a/cranelift/codegen/meta/src/shared/types.rs
+++ b/cranelift/codegen/meta/src/shared/types.rs
@@ -145,6 +145,38 @@ impl Iterator for FlagIterator {
     }
 }
 
+#[derive(Debug, Clone, Copy, Eq, PartialEq, Hash)]
+pub enum Reference {
+    /// 32-bit reference.
+    R32 = 32,
+    /// 64-bit reference.
+    R64 = 64,
+}
+
+/// This provides an iterator through all of the supported reference variants.
+pub struct ReferenceIterator {
+    index: u8,
+}
+
+impl ReferenceIterator {
+    pub fn new() -> Self {
+        Self { index: 0 }
+    }
+}
+
+impl Iterator for ReferenceIterator {
+    type Item = Reference;
+    fn next(&mut self) -> Option {
+        let res = match self.index {
+            0 => Some(Reference::R32),
+            1 => Some(Reference::R64),
+            _ => return None,
+        };
+        self.index += 1;
+        res
+    }
+}
+
 #[cfg(test)]
 mod iter_tests {
     use super::*;
@@ -185,4 +217,12 @@ mod iter_tests {
         assert_eq!(flag_iter.next(), Some(Flag::FFlags));
         assert_eq!(flag_iter.next(), None);
     }
+
+    #[test]
+    fn reference_iter_works() {
+        let mut reference_iter = ReferenceIterator::new();
+        assert_eq!(reference_iter.next(), Some(Reference::R32));
+        assert_eq!(reference_iter.next(), Some(Reference::R64));
+        assert_eq!(reference_iter.next(), None);
+    }
 }
diff --git a/cranelift/codegen/src/binemit/memorysink.rs b/cranelift/codegen/src/binemit/memorysink.rs
index b70b4a2c86..cde6cd9c9a 100644
--- a/cranelift/codegen/src/binemit/memorysink.rs
+++ b/cranelift/codegen/src/binemit/memorysink.rs
@@ -13,9 +13,11 @@
 //! that a `MemoryCodeSink` will always write binary machine code to raw memory. It forwards any
 //! relocations to a `RelocSink` trait object. Relocations are less frequent than the
 //! `CodeSink::put*` methods, so the performance impact of the virtual callbacks is less severe.
-
 use super::{Addend, CodeInfo, CodeOffset, CodeSink, Reloc};
-use crate::ir::{ExternalName, JumpTable, SourceLoc, TrapCode};
+use crate::binemit::stackmap::Stackmap;
+use crate::ir::entities::Value;
+use crate::ir::{ExternalName, Function, JumpTable, SourceLoc, TrapCode};
+use crate::isa::TargetIsa;
 use core::ptr::write_unaligned;
 
 /// A `CodeSink` that writes binary machine code directly into memory.
@@ -36,6 +38,7 @@ pub struct MemoryCodeSink<'a> {
     offset: isize,
     relocs: &'a mut dyn RelocSink,
     traps: &'a mut dyn TrapSink,
+    stackmaps: &'a mut dyn StackmapSink,
     /// Information about the generated code and read-only data.
     pub info: CodeInfo,
 }
@@ -49,6 +52,7 @@ impl<'a> MemoryCodeSink<'a> {
         data: *mut u8,
         relocs: &'a mut dyn RelocSink,
         traps: &'a mut dyn TrapSink,
+        stackmaps: &'a mut dyn StackmapSink,
     ) -> Self {
         Self {
             data,
@@ -61,6 +65,7 @@ impl<'a> MemoryCodeSink<'a> {
             },
             relocs,
             traps,
+            stackmaps,
         }
     }
 }
@@ -149,6 +154,12 @@ impl<'a> CodeSink for MemoryCodeSink<'a> {
         self.info.rodata_size = self.offset() - (self.info.jumptables_size + self.info.code_size);
         self.info.total_size = self.offset();
     }
+
+    fn add_stackmap(&mut self, val_list: &[Value], func: &Function, isa: &dyn TargetIsa) {
+        let ofs = self.offset();
+        let stackmap = Stackmap::from_values(&val_list, func, isa);
+        self.stackmaps.add_stackmap(ofs, stackmap);
+    }
 }
 
 /// A `TrapSink` implementation that does nothing, which is convenient when
@@ -158,3 +169,16 @@ pub struct NullTrapSink {}
 impl TrapSink for NullTrapSink {
     fn trap(&mut self, _offset: CodeOffset, _srcloc: SourceLoc, _code: TrapCode) {}
 }
+
+/// A trait for emitting stackmaps.
+pub trait StackmapSink {
+    /// Output a bitmap of the stack representing the live reference variables at this code offset.
+    fn add_stackmap(&mut self, _: CodeOffset, _: Stackmap);
+}
+
+/// Placeholder StackmapSink that does nothing.
+pub struct NullStackmapSink {}
+
+impl StackmapSink for NullStackmapSink {
+    fn add_stackmap(&mut self, _: CodeOffset, _: Stackmap) {}
+}
diff --git a/cranelift/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
index b0408e97e7..d6b72553bb 100644
--- a/cranelift/codegen/src/binemit/mod.rs
+++ b/cranelift/codegen/src/binemit/mod.rs
@@ -6,13 +6,18 @@
 mod memorysink;
 mod relaxation;
 mod shrink;
+mod stackmap;
 
-pub use self::memorysink::{MemoryCodeSink, NullTrapSink, RelocSink, TrapSink};
+pub use self::memorysink::{
+    MemoryCodeSink, NullStackmapSink, NullTrapSink, RelocSink, StackmapSink, TrapSink,
+};
 pub use self::relaxation::relax_branches;
 pub use self::shrink::shrink_instructions;
-pub use crate::regalloc::RegDiversions;
-
+pub use self::stackmap::Stackmap;
+use crate::ir::entities::Value;
 use crate::ir::{ExternalName, Function, Inst, JumpTable, SourceLoc, TrapCode};
+use crate::isa::TargetIsa;
+pub use crate::regalloc::RegDiversions;
 use core::fmt;
 #[cfg(feature = "enable-serde")]
 use serde::{Deserialize, Serialize};
@@ -141,6 +146,9 @@ pub trait CodeSink {
 
     /// Read-only data output is complete, we're done.
     fn end_codegen(&mut self);
+
+    /// Add a stackmap at the current code offset.
+    fn add_stackmap(&mut self, _: &[Value], _: &Function, _: &dyn TargetIsa);
 }
 
 /// Report a bad encoding error.
@@ -157,17 +165,17 @@ pub fn bad_encoding(func: &Function, inst: Inst) -> ! {
 ///
 /// This function is called from the `TargetIsa::emit_function()` implementations with the
 /// appropriate instruction emitter.
-pub fn emit_function(func: &Function, emit_inst: EI, sink: &mut CS)
+pub fn emit_function(func: &Function, emit_inst: EI, sink: &mut CS, isa: &dyn TargetIsa)
 where
     CS: CodeSink,
-    EI: Fn(&Function, Inst, &mut RegDiversions, &mut CS),
+    EI: Fn(&Function, Inst, &mut RegDiversions, &mut CS, &dyn TargetIsa),
 {
     let mut divert = RegDiversions::new();
     for ebb in func.layout.ebbs() {
         divert.clear();
         debug_assert_eq!(func.offsets[ebb], sink.offset());
         for inst in func.layout.ebb_insts(ebb) {
-            emit_inst(func, inst, &mut divert, sink);
+            emit_inst(func, inst, &mut divert, sink, isa);
         }
     }
 
diff --git a/cranelift/codegen/src/binemit/stackmap.rs b/cranelift/codegen/src/binemit/stackmap.rs
new file mode 100644
index 0000000000..8d0d2a3204
--- /dev/null
+++ b/cranelift/codegen/src/binemit/stackmap.rs
@@ -0,0 +1,122 @@
+use crate::bitset::BitSet;
+use crate::ir;
+use crate::isa::TargetIsa;
+use std::vec::Vec;
+
+/// Wrapper class for longer bit vectors that cannot be represented by a single BitSet.
+#[derive(Clone, Debug)]
+pub struct Stackmap {
+    bitmap: Vec>,
+}
+
+impl Stackmap {
+    /// Create a stackmap based on where references are located on a function's stack.
+    pub fn from_values(
+        args: &[ir::entities::Value],
+        func: &ir::Function,
+        isa: &dyn TargetIsa,
+    ) -> Self {
+        let loc = &func.locations;
+        let mut live_ref_in_stack_slot = std::collections::HashSet::new();
+        // References can be in registers, and live registers values are pushed onto the stack before calls and traps.
+        // TODO: Implement register maps. If a register containing a reference is spilled and reused after a safepoint,
+        // it could contain a stale reference value if the garbage collector relocated the value.
+        for val in args {
+            if let Some(value_loc) = loc.get(*val) {
+                match *value_loc {
+                    ir::ValueLoc::Stack(stack_slot) => live_ref_in_stack_slot.insert(stack_slot),
+                    _ => false,
+                };
+            }
+        }
+
+        // SpiderMonkey stackmap structure:
+        //  +  +  + 
+        // Bit vector goes from lower addresses to higher addresses.
+
+        // TODO: Get trap register layout from Spidermonkey and prepend to bitvector below.
+        let stack = &func.stack_slots;
+        let frame_size = stack.frame_size.unwrap();
+        let word_size = ir::stackslot::StackSize::from(isa.pointer_bytes());
+        let num_words = (frame_size / word_size) as usize;
+        let mut vec = std::vec::Vec::with_capacity(num_words);
+
+        vec.resize(num_words, false);
+
+        // Frame (includes spills and inbound args).
+        for (ss, ssd) in stack.iter() {
+            if live_ref_in_stack_slot.contains(&ss) {
+                // Assumption: greater magnitude of offset imply higher address.
+                let index = (((ssd.offset.unwrap().abs() as u32) - ssd.size) / word_size) as usize;
+                vec[index] = true;
+            }
+        }
+
+        Stackmap::from_vec(&vec)
+    }
+
+    /// Create a vec of Bitsets from a vec of bools.
+    pub fn from_vec(vec: &Vec) -> Self {
+        let mut rem = vec.len();
+        let num_word = ((rem as f32) / 32.0).ceil() as usize;
+        let mut bitmap = Vec::with_capacity(num_word);
+
+        for i in 0..num_word {
+            let mut curr_word = 0;
+            let count = if rem > 32 { 32 } else { rem };
+            for j in 0..count {
+                if vec[i * 32 + j] {
+                    curr_word |= 1 << j;
+                }
+            }
+            bitmap.push(BitSet::(curr_word));
+            rem -= count;
+        }
+        Self { bitmap }
+    }
+
+    /// Returns a specified bit.
+    pub fn get_bit(&self, bit_index: usize) -> bool {
+        assert!(bit_index < 32 * self.bitmap.len());
+        let word_index = bit_index / 32;
+        let word_offset = (bit_index % 32) as u8;
+        self.bitmap[word_index].contains(word_offset)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn stackmaps() {
+        let vec: Vec = Vec::new();
+        assert!(Stackmap::from_vec(&vec).bitmap.is_empty());
+
+        let mut vec: [bool; 32] = Default::default();
+        let set_true_idx = [5, 7, 24, 31];
+
+        for idx in set_true_idx.iter() {
+            vec[*idx] = true;
+        }
+
+        let mut vec = vec.to_vec();
+        assert_eq!(
+            vec![BitSet::(2164261024)],
+            Stackmap::from_vec(&vec).bitmap
+        );
+
+        vec.push(false);
+        vec.push(true);
+        let res = Stackmap::from_vec(&vec);
+        assert_eq!(
+            vec![BitSet::(2164261024), BitSet::(2)],
+            res.bitmap
+        );
+
+        assert!(res.get_bit(5));
+        assert!(res.get_bit(31));
+        assert!(res.get_bit(33));
+        assert!(!res.get_bit(1));
+    }
+}
diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index a80d3c41e4..cbf412aa20 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -10,7 +10,8 @@
 //! single ISA instance.
 
 use crate::binemit::{
-    relax_branches, shrink_instructions, CodeInfo, MemoryCodeSink, RelocSink, TrapSink,
+    relax_branches, shrink_instructions, CodeInfo, MemoryCodeSink, RelocSink, StackmapSink,
+    TrapSink,
 };
 use crate::dce::do_dce;
 use crate::dominator_tree::DominatorTree;
@@ -100,12 +101,14 @@ impl Context {
         mem: &mut Vec,
         relocs: &mut dyn RelocSink,
         traps: &mut dyn TrapSink,
+        stackmaps: &mut dyn StackmapSink,
     ) -> CodegenResult {
         let info = self.compile(isa)?;
         let old_len = mem.len();
         mem.resize(old_len + info.total_size as usize, 0);
-        let new_info =
-            unsafe { self.emit_to_memory(isa, mem.as_mut_ptr().add(old_len), relocs, traps) };
+        let new_info = unsafe {
+            self.emit_to_memory(isa, mem.as_mut_ptr().add(old_len), relocs, traps, stackmaps)
+        };
         debug_assert!(new_info == info);
         Ok(info)
     }
@@ -168,9 +171,10 @@ impl Context {
         mem: *mut u8,
         relocs: &mut dyn RelocSink,
         traps: &mut dyn TrapSink,
+        stackmaps: &mut dyn StackmapSink,
     ) -> CodeInfo {
         let _tt = timing::binemit();
-        let mut sink = MemoryCodeSink::new(mem, relocs, traps);
+        let mut sink = MemoryCodeSink::new(mem, relocs, traps, stackmaps);
         isa.emit_function_to_memory(&self.func, &mut sink);
         sink.info
     }
diff --git a/cranelift/codegen/src/ir/instructions.rs b/cranelift/codegen/src/ir/instructions.rs
index a94b0f3a60..30e8d14689 100644
--- a/cranelift/codegen/src/ir/instructions.rs
+++ b/cranelift/codegen/src/ir/instructions.rs
@@ -444,6 +444,8 @@ pub struct ValueTypeSet {
     pub floats: BitSet8,
     /// Allowed bool widths
     pub bools: BitSet8,
+    /// Allowed ref widths
+    pub refs: BitSet8,
 }
 
 impl ValueTypeSet {
@@ -458,6 +460,8 @@ impl ValueTypeSet {
             self.floats.contains(l2b)
         } else if scalar.is_bool() {
             self.bools.contains(l2b)
+        } else if scalar.is_ref() {
+            self.refs.contains(l2b)
         } else {
             false
         }
@@ -652,6 +656,7 @@ mod tests {
             ints: BitSet8::from_range(4, 7),
             floats: BitSet8::from_range(0, 0),
             bools: BitSet8::from_range(3, 7),
+            refs: BitSet8::from_range(5, 7),
         };
         assert!(!vts.contains(I8));
         assert!(vts.contains(I32));
@@ -661,6 +666,8 @@ mod tests {
         assert!(!vts.contains(B1));
         assert!(vts.contains(B8));
         assert!(vts.contains(B64));
+        assert!(vts.contains(R32));
+        assert!(vts.contains(R64));
         assert_eq!(vts.example().to_string(), "i32");
 
         let vts = ValueTypeSet {
@@ -668,6 +675,7 @@ mod tests {
             ints: BitSet8::from_range(0, 0),
             floats: BitSet8::from_range(5, 7),
             bools: BitSet8::from_range(3, 7),
+            refs: BitSet8::from_range(0, 0),
         };
         assert_eq!(vts.example().to_string(), "f32");
 
@@ -676,6 +684,7 @@ mod tests {
             ints: BitSet8::from_range(0, 0),
             floats: BitSet8::from_range(5, 7),
             bools: BitSet8::from_range(3, 7),
+            refs: BitSet8::from_range(0, 0),
         };
         assert_eq!(vts.example().to_string(), "f32x2");
 
@@ -684,6 +693,7 @@ mod tests {
             ints: BitSet8::from_range(0, 0),
             floats: BitSet8::from_range(0, 0),
             bools: BitSet8::from_range(3, 7),
+            refs: BitSet8::from_range(0, 0),
         };
         assert!(!vts.contains(B32X2));
         assert!(vts.contains(B32X4));
@@ -695,8 +705,11 @@ mod tests {
             ints: BitSet8::from_range(3, 7),
             floats: BitSet8::from_range(0, 0),
             bools: BitSet8::from_range(0, 0),
+            refs: BitSet8::from_range(0, 0),
         };
         assert!(vts.contains(I32));
         assert!(vts.contains(I32X4));
+        assert!(!vts.contains(R32));
+        assert!(!vts.contains(R64));
     }
 }
diff --git a/cranelift/codegen/src/ir/types.rs b/cranelift/codegen/src/ir/types.rs
index fd28b5bd7c..4eb72f3fc0 100644
--- a/cranelift/codegen/src/ir/types.rs
+++ b/cranelift/codegen/src/ir/types.rs
@@ -61,8 +61,8 @@ impl Type {
             B1 => 0,
             B8 | I8 => 3,
             B16 | I16 => 4,
-            B32 | I32 | F32 => 5,
-            B64 | I64 | F64 => 6,
+            B32 | I32 | F32 | R32 => 5,
+            B64 | I64 | F64 | R64 => 6,
             _ => 0,
         }
     }
@@ -73,8 +73,8 @@ impl Type {
             B1 => 1,
             B8 | I8 => 8,
             B16 | I16 => 16,
-            B32 | I32 | F32 => 32,
-            B64 | I64 | F64 => 64,
+            B32 | I32 | F32 | R32 => 32,
+            B64 | I64 | F64 | R64 => 64,
             _ => 0,
         }
     }
@@ -99,7 +99,7 @@ impl Type {
     /// Get a type with the same number of lanes as this type, but with the lanes replaced by
     /// booleans of the same size.
     ///
-    /// Scalar types are treated as vectors with one lane, so they are converted to the multi-bit
+    /// Lane types are treated as vectors with one lane, so they are converted to the multi-bit
     /// boolean types.
     pub fn as_bool_pedantic(self) -> Self {
         // Replace the low 4 bits with the boolean version, preserve the high 4 bits.
@@ -108,6 +108,7 @@ impl Type {
             B16 | I16 => B16,
             B32 | I32 | F32 => B32,
             B64 | I64 | F64 => B64,
+            R32 | R64 => panic!("Reference types should not convert to bool"),
             _ => B1,
         })
     }
@@ -210,6 +211,14 @@ impl Type {
         }
     }
 
+    /// Is this a ref type?
+    pub fn is_ref(self) -> bool {
+        match self {
+            R32 | R64 => true,
+            _ => false,
+        }
+    }
+
     /// Get log_2 of the number of lanes in this SIMD vector type.
     ///
     /// All SIMD types have a lane count that is a power of two and no larger than 256, so this
@@ -301,6 +310,8 @@ impl Display for Type {
             write!(f, "f{}", self.lane_bits())
         } else if self.is_vector() {
             write!(f, "{}x{}", self.lane_type(), self.lane_count())
+        } else if self.is_ref() {
+            write!(f, "r{}", self.lane_bits())
         } else {
             f.write_str(match *self {
                 IFLAGS => "iflags",
@@ -322,6 +333,8 @@ impl Debug for Type {
             write!(f, "types::F{}", self.lane_bits())
         } else if self.is_vector() {
             write!(f, "{:?}X{}", self.lane_type(), self.lane_count())
+        } else if self.is_ref() {
+            write!(f, "types::R{}", self.lane_bits())
         } else {
             match *self {
                 INVALID => write!(f, "types::INVALID"),
@@ -366,6 +379,8 @@ mod tests {
         assert_eq!(B1, B1.by(8).unwrap().lane_type());
         assert_eq!(I32, I32X4.lane_type());
         assert_eq!(F64, F64X2.lane_type());
+        assert_eq!(R32, R32.lane_type());
+        assert_eq!(R64, R64.lane_type());
 
         assert_eq!(INVALID.lane_bits(), 0);
         assert_eq!(IFLAGS.lane_bits(), 0);
@@ -381,6 +396,8 @@ mod tests {
         assert_eq!(I64.lane_bits(), 64);
         assert_eq!(F32.lane_bits(), 32);
         assert_eq!(F64.lane_bits(), 64);
+        assert_eq!(R32.lane_bits(), 32);
+        assert_eq!(R64.lane_bits(), 64);
     }
 
     #[test]
@@ -450,6 +467,8 @@ mod tests {
         assert_eq!(I64.to_string(), "i64");
         assert_eq!(F32.to_string(), "f32");
         assert_eq!(F64.to_string(), "f64");
+        assert_eq!(R32.to_string(), "r32");
+        assert_eq!(R64.to_string(), "r64");
     }
 
     #[test]
diff --git a/cranelift/codegen/src/isa/arm32/binemit.rs b/cranelift/codegen/src/isa/arm32/binemit.rs
index 2ca78b4d6f..d74ee0911a 100644
--- a/cranelift/codegen/src/isa/arm32/binemit.rs
+++ b/cranelift/codegen/src/isa/arm32/binemit.rs
@@ -2,6 +2,7 @@
 
 use crate::binemit::{bad_encoding, CodeSink};
 use crate::ir::{Function, Inst};
+use crate::isa::TargetIsa;
 use crate::regalloc::RegDiversions;
 
 include!(concat!(env!("OUT_DIR"), "/binemit-arm32.rs"));
diff --git a/cranelift/codegen/src/isa/arm32/mod.rs b/cranelift/codegen/src/isa/arm32/mod.rs
index d4e2f32255..fde3339700 100644
--- a/cranelift/codegen/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/src/isa/arm32/mod.rs
@@ -121,11 +121,11 @@ impl TargetIsa for Isa {
         divert: &mut regalloc::RegDiversions,
         sink: &mut dyn CodeSink,
     ) {
-        binemit::emit_inst(func, inst, divert, sink)
+        binemit::emit_inst(func, inst, divert, sink, self)
     }
 
     fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut MemoryCodeSink) {
-        emit_function(func, binemit::emit_inst, sink)
+        emit_function(func, binemit::emit_inst, sink, self)
     }
 }
 
diff --git a/cranelift/codegen/src/isa/arm64/binemit.rs b/cranelift/codegen/src/isa/arm64/binemit.rs
index 05df67e5bb..4401b6d6f5 100644
--- a/cranelift/codegen/src/isa/arm64/binemit.rs
+++ b/cranelift/codegen/src/isa/arm64/binemit.rs
@@ -2,6 +2,7 @@
 
 use crate::binemit::{bad_encoding, CodeSink};
 use crate::ir::{Function, Inst};
+use crate::isa::TargetIsa;
 use crate::regalloc::RegDiversions;
 
 include!(concat!(env!("OUT_DIR"), "/binemit-arm64.rs"));
diff --git a/cranelift/codegen/src/isa/arm64/mod.rs b/cranelift/codegen/src/isa/arm64/mod.rs
index c24be19f69..d787524a6a 100644
--- a/cranelift/codegen/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/src/isa/arm64/mod.rs
@@ -108,11 +108,11 @@ impl TargetIsa for Isa {
         divert: &mut regalloc::RegDiversions,
         sink: &mut dyn CodeSink,
     ) {
-        binemit::emit_inst(func, inst, divert, sink)
+        binemit::emit_inst(func, inst, divert, sink, self)
     }
 
     fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut MemoryCodeSink) {
-        emit_function(func, binemit::emit_inst, sink)
+        emit_function(func, binemit::emit_inst, sink, self)
     }
 }
 
diff --git a/cranelift/codegen/src/isa/riscv/binemit.rs b/cranelift/codegen/src/isa/riscv/binemit.rs
index 64f3c00298..a1d2b82e12 100644
--- a/cranelift/codegen/src/isa/riscv/binemit.rs
+++ b/cranelift/codegen/src/isa/riscv/binemit.rs
@@ -2,7 +2,7 @@
 
 use crate::binemit::{bad_encoding, CodeSink, Reloc};
 use crate::ir::{Function, Inst, InstructionData};
-use crate::isa::{RegUnit, StackBaseMask, StackRef};
+use crate::isa::{RegUnit, StackBaseMask, StackRef, TargetIsa};
 use crate::predicates::is_signed_int;
 use crate::regalloc::RegDiversions;
 use core::u32;
diff --git a/cranelift/codegen/src/isa/riscv/mod.rs b/cranelift/codegen/src/isa/riscv/mod.rs
index 3096ff69aa..233e92a3bb 100644
--- a/cranelift/codegen/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/src/isa/riscv/mod.rs
@@ -115,11 +115,11 @@ impl TargetIsa for Isa {
         divert: &mut regalloc::RegDiversions,
         sink: &mut dyn CodeSink,
     ) {
-        binemit::emit_inst(func, inst, divert, sink)
+        binemit::emit_inst(func, inst, divert, sink, self)
     }
 
     fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut MemoryCodeSink) {
-        emit_function(func, binemit::emit_inst, sink)
+        emit_function(func, binemit::emit_inst, sink, self)
     }
 }
 
diff --git a/cranelift/codegen/src/isa/x86/binemit.rs b/cranelift/codegen/src/isa/x86/binemit.rs
index ef44766d0b..8aa51c07b6 100644
--- a/cranelift/codegen/src/isa/x86/binemit.rs
+++ b/cranelift/codegen/src/isa/x86/binemit.rs
@@ -5,7 +5,7 @@ use super::registers::RU;
 use crate::binemit::{bad_encoding, CodeSink, Reloc};
 use crate::ir::condcodes::{CondCode, FloatCC, IntCC};
 use crate::ir::{Ebb, Function, Inst, InstructionData, JumpTable, Opcode, TrapCode};
-use crate::isa::{RegUnit, StackBase, StackBaseMask, StackRef};
+use crate::isa::{RegUnit, StackBase, StackBaseMask, StackRef, TargetIsa};
 use crate::regalloc::RegDiversions;
 
 include!(concat!(env!("OUT_DIR"), "/binemit-x86.rs"));
diff --git a/cranelift/codegen/src/isa/x86/mod.rs b/cranelift/codegen/src/isa/x86/mod.rs
index dfac5cacc3..9244791ea8 100644
--- a/cranelift/codegen/src/isa/x86/mod.rs
+++ b/cranelift/codegen/src/isa/x86/mod.rs
@@ -131,11 +131,11 @@ impl TargetIsa for Isa {
         divert: &mut regalloc::RegDiversions,
         sink: &mut dyn CodeSink,
     ) {
-        binemit::emit_inst(func, inst, divert, sink)
+        binemit::emit_inst(func, inst, divert, sink, self)
     }
 
     fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut MemoryCodeSink) {
-        emit_function(func, binemit::emit_inst, sink)
+        emit_function(func, binemit::emit_inst, sink, self)
     }
 
     fn prologue_epilogue(&self, func: &mut ir::Function) -> CodegenResult<()> {
diff --git a/cranelift/codegen/src/regalloc/context.rs b/cranelift/codegen/src/regalloc/context.rs
index cb4b4245f4..fe2e702647 100644
--- a/cranelift/codegen/src/regalloc/context.rs
+++ b/cranelift/codegen/src/regalloc/context.rs
@@ -13,6 +13,7 @@ use crate::regalloc::coloring::Coloring;
 use crate::regalloc::live_value_tracker::LiveValueTracker;
 use crate::regalloc::liveness::Liveness;
 use crate::regalloc::reload::Reload;
+use crate::regalloc::safepoint::emit_stackmaps;
 use crate::regalloc::spilling::Spilling;
 use crate::regalloc::virtregs::VirtRegs;
 use crate::result::CodegenResult;
@@ -192,6 +193,20 @@ impl Context {
         self.coloring
             .run(isa, func, domtree, &mut self.liveness, &mut self.tracker);
 
+        // This function runs after register allocation has taken
+        // place, meaning values have locations assigned already.
+        if isa.flags().enable_safepoints() {
+            emit_stackmaps(func, domtree, &self.liveness, &mut self.tracker, isa);
+        } else {
+            // Make sure no references are used.
+            for val in func.dfg.values() {
+                let ty = func.dfg.value_type(val);
+                if ty.lane_type().is_ref() {
+                    panic!("reference types were found but safepoints were not enabled.");
+                }
+            }
+        }
+
         if isa.flags().enable_verifier() {
             let ok = verify_context(func, cfg, domtree, isa, &mut errors).is_ok()
                 && verify_liveness(isa, func, cfg, &self.liveness, &mut errors).is_ok()
diff --git a/cranelift/codegen/src/regalloc/mod.rs b/cranelift/codegen/src/regalloc/mod.rs
index dbec691602..cfa3ca7323 100644
--- a/cranelift/codegen/src/regalloc/mod.rs
+++ b/cranelift/codegen/src/regalloc/mod.rs
@@ -15,9 +15,11 @@ mod context;
 mod diversion;
 mod pressure;
 mod reload;
+mod safepoint;
 mod solver;
 mod spilling;
 
 pub use self::context::Context;
 pub use self::diversion::RegDiversions;
 pub use self::register_set::RegisterSet;
+pub use self::safepoint::emit_stackmaps;
diff --git a/cranelift/codegen/src/regalloc/safepoint.rs b/cranelift/codegen/src/regalloc/safepoint.rs
new file mode 100644
index 0000000000..9b27b6227d
--- /dev/null
+++ b/cranelift/codegen/src/regalloc/safepoint.rs
@@ -0,0 +1,72 @@
+use crate::cursor::{Cursor, FuncCursor};
+use crate::dominator_tree::DominatorTree;
+use crate::ir::{Function, InstBuilder, InstructionData, Opcode, TrapCode};
+use crate::isa::TargetIsa;
+use crate::regalloc::live_value_tracker::LiveValueTracker;
+use crate::regalloc::liveness::Liveness;
+use std::vec::Vec;
+
+fn insert_and_encode_safepoint<'f>(
+    pos: &mut FuncCursor<'f>,
+    tracker: &LiveValueTracker,
+    isa: &dyn TargetIsa,
+) {
+    // Iterate through all live values, collect only the references.
+    let live_ref_values = tracker
+        .live()
+        .iter()
+        .filter(|live_value| pos.func.dfg.value_type(live_value.value).is_ref())
+        .map(|live_val| live_val.value)
+        .collect::>();
+
+    if !live_ref_values.is_empty() {
+        pos.ins().safepoint(&live_ref_values);
+        // Move cursor to the new safepoint instruction to encode it.
+        if let Some(inst) = pos.prev_inst() {
+            let ok = pos.func.update_encoding(inst, isa).is_ok();
+            debug_assert!(ok);
+        }
+        // Restore cursor position.
+        pos.next_inst();
+    }
+}
+
+// The emit_stackmaps() function analyzes each instruction to retrieve the liveness of
+// the defs and operands by traversing a function's ebbs in layout order.
+pub fn emit_stackmaps(
+    func: &mut Function,
+    domtree: &DominatorTree,
+    liveness: &Liveness,
+    tracker: &mut LiveValueTracker,
+    isa: &dyn TargetIsa,
+) {
+    let mut curr = func.layout.entry_block();
+
+    while let Some(ebb) = curr {
+        tracker.ebb_top(ebb, &func.dfg, liveness, &func.layout, domtree);
+        tracker.drop_dead_params();
+        let mut pos = FuncCursor::new(func);
+
+        // From the top of the ebb, step through the instructions.
+        pos.goto_top(ebb);
+
+        while let Some(inst) = pos.next_inst() {
+            if let InstructionData::Trap {
+                code: TrapCode::Interrupt,
+                ..
+            } = &pos.func.dfg[inst]
+            {
+                insert_and_encode_safepoint(&mut pos, tracker, isa);
+            } else if pos.func.dfg[inst].opcode().is_call() {
+                insert_and_encode_safepoint(&mut pos, tracker, isa);
+            } else if pos.func.dfg[inst].opcode() == Opcode::Safepoint {
+                panic!("safepoint instruction can only be used by the compiler!");
+            }
+
+            // Process the instruction and get rid of dead values.
+            tracker.process_inst(inst, &pos.func.dfg, liveness);
+            tracker.drop_dead(inst);
+        }
+        curr = func.layout.next_ebb(ebb);
+    }
+}
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index 75462869bc..f9bc9f6d1a 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -390,6 +390,7 @@ mod tests {
              enable_nan_canonicalization = false\n\
              enable_simd = false\n\
              enable_atomics = true\n\
+             enable_safepoints = false\n\
              allones_funcaddrs = false\n\
              probestack_enabled = true\n\
              probestack_func_adjusts_sp = false\n\
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index a09a696fff..39055b1232 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -1672,9 +1672,12 @@ impl<'a> Verifier<'a> {
         // Instructions with side effects are not allowed to be ghost instructions.
         let opcode = self.func.dfg[inst].opcode();
 
-        // The `fallthrough` and `fallthrough_return` instructions are marked as terminators and
-        // branches, but they are not required to have an encoding.
-        if opcode == Opcode::Fallthrough || opcode == Opcode::FallthroughReturn {
+        // The `fallthrough`, `fallthrough_return`, and `safepoint` instructions are not required
+        // to have an encoding.
+        if opcode == Opcode::Fallthrough
+            || opcode == Opcode::FallthroughReturn
+            || opcode == Opcode::Safepoint
+        {
             return Ok(());
         }
 
@@ -1739,6 +1742,24 @@ impl<'a> Verifier<'a> {
         }
     }
 
+    fn verify_safepoint_unused(
+        &self,
+        inst: Inst,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
+        if let Some(isa) = self.isa {
+            if !isa.flags().enable_safepoints() && self.func.dfg[inst].opcode() == Opcode::Safepoint
+            {
+                return fatal!(
+                    errors,
+                    inst,
+                    "safepoint instruction cannot be used when it is not enabled."
+                );
+            }
+        }
+        Ok(())
+    }
+
     pub fn run(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         self.verify_global_values(errors)?;
         self.verify_heaps(errors)?;
@@ -1750,6 +1771,7 @@ impl<'a> Verifier<'a> {
             for inst in self.func.layout.ebb_insts(ebb) {
                 self.ebb_integrity(ebb, inst, errors)?;
                 self.instruction_integrity(inst, errors)?;
+                self.verify_safepoint_unused(inst, errors)?;
                 self.typecheck(inst, errors)?;
                 self.verify_encoding(inst, errors)?;
                 self.immediate_constraints(inst, errors)?;
diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index 57dc21376a..bed56122f0 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -2,7 +2,9 @@
 
 use crate::container;
 use crate::traps::{FaerieTrapManifest, FaerieTrapSink};
-use cranelift_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSink};
+use cranelift_codegen::binemit::{
+    Addend, CodeOffset, NullStackmapSink, NullTrapSink, Reloc, RelocSink, Stackmap, StackmapSink,
+};
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::{self, binemit, ir};
 use cranelift_module::{
@@ -141,6 +143,8 @@ impl Backend for FaerieBackend {
         total_size: u32,
     ) -> ModuleResult {
         let mut code: Vec = vec![0; total_size as usize];
+        // TODO: Replace this with FaerieStackmapSink once it is implemented.
+        let mut stackmap_sink = NullStackmapSink {};
 
         // Non-lexical lifetimes would obviate the braces here.
         {
@@ -160,6 +164,7 @@ impl Backend for FaerieBackend {
                         code.as_mut_ptr(),
                         &mut reloc_sink,
                         &mut trap_sink,
+                        &mut stackmap_sink,
                     )
                 };
                 trap_manifest.add_sink(trap_sink);
@@ -171,6 +176,7 @@ impl Backend for FaerieBackend {
                         code.as_mut_ptr(),
                         &mut reloc_sink,
                         &mut trap_sink,
+                        &mut stackmap_sink,
                     )
                 };
             }
@@ -425,3 +431,16 @@ impl<'a> RelocSink for FaerieRelocSink<'a> {
         }
     }
 }
+
+#[allow(dead_code)]
+struct FaerieStackmapSink<'a> {
+    artifact: &'a mut faerie::Artifact,
+    namespace: &'a ModuleNamespace<'a, FaerieBackend>,
+}
+
+/// Faerie is currently not used in SpiderMonkey. Methods are unimplemented.
+impl<'a> StackmapSink for FaerieStackmapSink<'a> {
+    fn add_stackmap(&mut self, _: CodeOffset, _: Stackmap) {
+        unimplemented!("faerie support for stackmaps");
+    }
+}
diff --git a/cranelift/filetests/filetests/safepoint/basic.clif b/cranelift/filetests/filetests/safepoint/basic.clif
new file mode 100644
index 0000000000..b47a5bf393
--- /dev/null
+++ b/cranelift/filetests/filetests/safepoint/basic.clif
@@ -0,0 +1,55 @@
+test safepoint
+
+set enable_safepoints=true
+target x86_64
+
+function %test(i32, r64, r64) -> r64 {
+    ebb0(v0: i32, v1:r64, v2:r64):
+        jump ebb1(v0)
+    ebb1(v3: i32):
+        v4 = irsub_imm v3, 1
+        jump ebb2(v4)
+    ebb2(v5: i32):
+        resumable_trap interrupt
+        brz v5, ebb1(v5)
+        v6 = null.r64
+        v7 = is_null v6
+        brnz v7, ebb2(v0)
+        brnz v0, ebb3
+        jump ebb4
+    ebb3:
+        return v1
+    ebb4:
+        return v2
+}
+
+; sameln: function %test(i32 [%rdi], r64 [%rsi], r64 [%rdx]) -> r64 [%rax] fast {
+; nextln: ebb0(v0: i32 [%rdi], v1: r64 [%rsi], v2: r64 [%rdx]):
+; nextln:   v10 = copy v0
+; nextln:   jump ebb1(v10)
+; nextln: 
+; nextln: ebb1(v3: i32 [%rax]):
+; nextln:   v8 = iconst.i32 1
+; nextln:   v4 = isub v8, v3
+; nextln:   jump ebb2(v4)
+; nextln: 
+; nextln: ebb2(v5: i32 [%rcx]):
+; nextln:   safepoint v1, v2
+; nextln:   resumable_trap interrupt
+; nextln:   regmove v5, %rcx -> %rax
+; nextln:   brz v5, ebb1(v5)
+; nextln:   v6 = null.r64 
+; nextln:   v7 = is_null v6
+; nextln:   v9 = copy.i32 v0
+; nextln:   brnz v7, ebb2(v9)
+; nextln:   brnz.i32 v0, ebb3
+; nextln:   jump ebb4
+; nextln: 
+; nextln: ebb3:
+; nextln:   regmove.r64 v1, %rsi -> %rax
+; nextln:   return v1
+; nextln: 
+; nextln: ebb4:
+; nextln:   regmove.r64 v2, %rdx -> %rax
+; nextln:   return v2
+; nextln: }
diff --git a/cranelift/filetests/filetests/safepoint/call.clif b/cranelift/filetests/filetests/safepoint/call.clif
new file mode 100644
index 0000000000..4b9de997e0
--- /dev/null
+++ b/cranelift/filetests/filetests/safepoint/call.clif
@@ -0,0 +1,52 @@
+test safepoint
+
+set enable_safepoints=true
+target x86_64
+
+function %direct() -> r64 {
+    fn0 = %none()
+    fn1 = %one() -> r64
+    fn2 = %two() -> i32, r64
+
+ebb0:
+    call fn0()
+    v1 = call fn1()
+    v2, v3 = call fn2()
+    brz v2, ebb1 
+    return v1
+ebb1:
+    v4 = call fn1()
+    return v3
+}
+
+; sameln: function %direct() -> r64 [%rax] fast {
+; nextln: ss0 = spill_slot 8
+; nextln: ss1 = spill_slot 8
+; nextln: sig0 = () fast
+; nextln: sig1 = () -> r64 [%rax] fast
+; nextln: sig2 = () -> i32 [%rax], r64 [%rdx] fast
+; nextln: fn0 = %none sig0
+; nextln: fn1 = %one sig1
+; nextln: fn2 = %two sig2
+; nextln: 
+; nextln: ebb0:
+; nextln:   v5 = func_addr.i64 fn0
+; nextln:   call_indirect sig0, v5()
+; nextln:   v6 = func_addr.i64 fn1
+; nextln:   v9 = call_indirect sig1, v6()
+; nextln:   v1 = spill v9
+; nextln:   v7 = func_addr.i64 fn2
+; nextln:   safepoint v1
+; nextln:   v2, v10 = call_indirect sig2, v7()
+; nextln:   v3 = spill v10
+; nextln:   brz v2, ebb1
+; nextln:   v11 = fill v1
+; nextln:   return v11
+; nextln: 
+; nextln: ebb1:
+; nextln:   v8 = func_addr.i64 fn1
+; nextln:   safepoint v3
+; nextln:   v4 = call_indirect sig1, v8()
+; nextln:   v12 = fill.r64 v3
+; nextln:   return v12
+; nextln: }
diff --git a/cranelift/filetests/src/lib.rs b/cranelift/filetests/src/lib.rs
index fbd069bcb5..6099ce3726 100644
--- a/cranelift/filetests/src/lib.rs
+++ b/cranelift/filetests/src/lib.rs
@@ -46,6 +46,7 @@ mod test_postopt;
 mod test_preopt;
 mod test_print_cfg;
 mod test_regalloc;
+mod test_safepoint;
 mod test_shrink;
 mod test_simple_gvn;
 mod test_simple_preopt;
@@ -127,6 +128,7 @@ fn new_subtest(parsed: &TestCommand) -> subtest::SubtestResult test_simple_gvn::subtest(parsed),
         "verifier" => test_verifier::subtest(parsed),
         "preopt" => test_preopt::subtest(parsed),
+        "safepoint" => test_safepoint::subtest(parsed),
         _ => Err(format!("unknown test command '{}'", parsed.command)),
     }
 }
diff --git a/cranelift/filetests/src/test_binemit.rs b/cranelift/filetests/src/test_binemit.rs
index 3925acb8eb..f134806663 100644
--- a/cranelift/filetests/src/test_binemit.rs
+++ b/cranelift/filetests/src/test_binemit.rs
@@ -11,6 +11,7 @@ use cranelift_codegen::dominator_tree::DominatorTree;
 use cranelift_codegen::flowgraph::ControlFlowGraph;
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::entities::AnyEntity;
+use cranelift_codegen::isa;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_codegen::settings::OptLevel;
 use cranelift_reader::TestCommand;
@@ -100,9 +101,15 @@ impl binemit::CodeSink for TextSink {
     fn begin_jumptables(&mut self) {
         self.code_size = self.offset
     }
-
     fn begin_rodata(&mut self) {}
     fn end_codegen(&mut self) {}
+    fn add_stackmap(
+        &mut self,
+        _: &[ir::entities::Value],
+        _: &ir::Function,
+        _: &dyn isa::TargetIsa,
+    ) {
+    }
 }
 
 impl SubTest for TestBinEmit {
diff --git a/cranelift/filetests/src/test_compile.rs b/cranelift/filetests/src/test_compile.rs
index e9150a73ee..706aa00331 100644
--- a/cranelift/filetests/src/test_compile.rs
+++ b/cranelift/filetests/src/test_compile.rs
@@ -6,6 +6,7 @@ use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use cranelift_codegen;
 use cranelift_codegen::binemit::{self, CodeInfo};
 use cranelift_codegen::ir;
+use cranelift_codegen::isa;
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_reader::TestCommand;
 use log::info;
@@ -53,8 +54,9 @@ impl SubTest for TestCompile {
         let mut sink = SizeSink { offset: 0 };
         binemit::emit_function(
             &comp_ctx.func,
-            |func, inst, div, sink| isa.emit_inst(func, inst, div, sink),
+            |func, inst, div, sink, isa| isa.emit_inst(func, inst, div, sink),
             &mut sink,
+            isa,
         );
 
         if sink.offset != total_size {
@@ -109,4 +111,11 @@ impl binemit::CodeSink for SizeSink {
     fn begin_jumptables(&mut self) {}
     fn begin_rodata(&mut self) {}
     fn end_codegen(&mut self) {}
+    fn add_stackmap(
+        &mut self,
+        _: &[ir::entities::Value],
+        _: &ir::Function,
+        _: &dyn isa::TargetIsa,
+    ) {
+    }
 }
diff --git a/cranelift/filetests/src/test_safepoint.rs b/cranelift/filetests/src/test_safepoint.rs
new file mode 100644
index 0000000000..cec7368ae8
--- /dev/null
+++ b/cranelift/filetests/src/test_safepoint.rs
@@ -0,0 +1,39 @@
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
+use cranelift_codegen::ir::Function;
+use cranelift_codegen::print_errors::pretty_error;
+use cranelift_reader::TestCommand;
+use std::borrow::Cow;
+
+struct TestSafepoint;
+
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+    assert_eq!(parsed.command, "safepoint");
+    if !parsed.options.is_empty() {
+        Err(format!("No options allowed on {}", parsed))
+    } else {
+        Ok(Box::new(TestSafepoint))
+    }
+}
+
+impl SubTest for TestSafepoint {
+    fn name(&self) -> &'static str {
+        "safepoint"
+    }
+
+    fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> {
+        let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned());
+
+        let isa = context.isa.expect("register allocator needs an ISA");
+        comp_ctx.compute_cfg();
+        comp_ctx
+            .legalize(isa)
+            .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?;
+        comp_ctx.compute_domtree();
+        comp_ctx
+            .regalloc(isa)
+            .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?;
+
+        let text = comp_ctx.func.display(context.isa).to_string();
+        run_filecheck(&text, context)
+    }
+}
diff --git a/cranelift/frontend/src/ssa.rs b/cranelift/frontend/src/ssa.rs
index 0d48c314af..aa735f7747 100644
--- a/cranelift/frontend/src/ssa.rs
+++ b/cranelift/frontend/src/ssa.rs
@@ -221,6 +221,8 @@ fn emit_zero(ty: Type, mut cur: FuncCursor) -> Value {
         cur.ins().f32const(Ieee32::with_bits(0))
     } else if ty == F64 {
         cur.ins().f64const(Ieee64::with_bits(0))
+    } else if ty.is_ref() {
+        cur.ins().null(ty)
     } else if ty.is_vector() {
         let scalar_ty = ty.lane_type();
         if scalar_ty.is_int() {
diff --git a/cranelift/reader/src/lexer.rs b/cranelift/reader/src/lexer.rs
index 8673b2b922..76327a7a8d 100644
--- a/cranelift/reader/src/lexer.rs
+++ b/cranelift/reader/src/lexer.rs
@@ -370,6 +370,8 @@ impl<'a> Lexer<'a> {
             "b16" => types::B16,
             "b32" => types::B32,
             "b64" => types::B64,
+            "r32" => types::R32,
+            "r64" => types::R64,
             _ => return None,
         };
         if is_vector {
diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index 16612d0a39..0a97df8f46 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -1,7 +1,9 @@
 //! Defines `SimpleJITBackend`.
 
 use crate::memory::Memory;
-use cranelift_codegen::binemit::{Addend, CodeOffset, NullTrapSink, Reloc, RelocSink};
+use cranelift_codegen::binemit::{
+    Addend, CodeOffset, NullTrapSink, Reloc, RelocSink, Stackmap, StackmapSink,
+};
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::{self, ir, settings};
 use cranelift_module::{
@@ -128,6 +130,13 @@ struct RelocRecord {
     addend: Addend,
 }
 
+struct StackmapRecord {
+    #[allow(dead_code)]
+    offset: CodeOffset,
+    #[allow(dead_code)]
+    stackmap: Stackmap,
+}
+
 pub struct SimpleJITCompiledFunction {
     code: *mut u8,
     size: usize,
@@ -254,7 +263,16 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         // Ignore traps for now. For now, frontends should just avoid generating code
         // that traps.
         let mut trap_sink = NullTrapSink {};
-        unsafe { ctx.emit_to_memory(&*self.isa, ptr, &mut reloc_sink, &mut trap_sink) };
+        let mut stackmap_sink = SimpleJITStackmapSink::new();
+        unsafe {
+            ctx.emit_to_memory(
+                &*self.isa,
+                ptr,
+                &mut reloc_sink,
+                &mut trap_sink,
+                &mut stackmap_sink,
+            )
+        };
 
         Ok(Self::CompiledFunction {
             code: ptr,
@@ -549,3 +567,21 @@ impl RelocSink for SimpleJITRelocSink {
         }
     }
 }
+
+struct SimpleJITStackmapSink {
+    pub stackmaps: Vec,
+}
+
+impl SimpleJITStackmapSink {
+    pub fn new() -> Self {
+        Self {
+            stackmaps: Vec::new(),
+        }
+    }
+}
+
+impl StackmapSink for SimpleJITStackmapSink {
+    fn add_stackmap(&mut self, offset: CodeOffset, stackmap: Stackmap) {
+        self.stackmaps.push(StackmapRecord { offset, stackmap });
+    }
+}
diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index fcac964aa3..643ed414bc 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -1,6 +1,6 @@
 //! CLI tool to reduce Cranelift IR files crashing during compilation.
 
-use crate::disasm::{PrintRelocs, PrintTraps};
+use crate::disasm::{PrintRelocs, PrintStackmaps, PrintTraps};
 use crate::utils::{parse_sets_and_triple, read_to_string};
 use cranelift_codegen::ir::{
     Ebb, FuncRef, Function, GlobalValueData, Inst, InstBuilder, InstructionData, StackSlots,
@@ -730,11 +730,16 @@ impl<'a> CrashCheckContext<'a> {
         let res = match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
             let mut relocs = PrintRelocs::new(false);
             let mut traps = PrintTraps::new(false);
+            let mut stackmaps = PrintStackmaps::new(false);
             let mut mem = vec![];
 
-            let _ = self
-                .context
-                .compile_and_emit(self.isa, &mut mem, &mut relocs, &mut traps);
+            let _ = self.context.compile_and_emit(
+                self.isa,
+                &mut mem,
+                &mut relocs,
+                &mut traps,
+                &mut stackmaps,
+            );
         })) {
             Ok(()) => CheckResult::Succeed,
             Err(err) => CheckResult::Crash(get_panic_string(err)),
diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs
index 4c663323b1..934a955480 100644
--- a/cranelift/src/compile.rs
+++ b/cranelift/src/compile.rs
@@ -1,6 +1,6 @@
 //! CLI tool to read Cranelift IR files and compile them into native code.
 
-use crate::disasm::{print_all, PrintRelocs, PrintTraps};
+use crate::disasm::{print_all, PrintRelocs, PrintStackmaps, PrintTraps};
 use crate::utils::{parse_sets_and_triple, read_to_string};
 use cranelift_codegen::print_errors::pretty_error;
 use cranelift_codegen::settings::FlagsOrIsa;
@@ -62,11 +62,12 @@ fn handle_module(
 
         let mut relocs = PrintRelocs::new(flag_print);
         let mut traps = PrintTraps::new(flag_print);
+        let mut stackmaps = PrintStackmaps::new(flag_print);
         let mut mem = vec![];
 
         // Compile and encode the result to machine code.
         let code_info = context
-            .compile_and_emit(isa, &mut mem, &mut relocs, &mut traps)
+            .compile_and_emit(isa, &mut mem, &mut relocs, &mut traps, &mut stackmaps)
             .map_err(|err| pretty_error(&context.func, Some(isa), err))?;
 
         if flag_print {
@@ -81,6 +82,7 @@ fn handle_module(
                 code_info.jumptables_size + code_info.rodata_size,
                 &relocs,
                 &traps,
+                &stackmaps,
             )?;
         }
     }
diff --git a/cranelift/src/disasm.rs b/cranelift/src/disasm.rs
index d7ba0dca6b..a24371206a 100644
--- a/cranelift/src/disasm.rs
+++ b/cranelift/src/disasm.rs
@@ -80,6 +80,28 @@ impl binemit::TrapSink for PrintTraps {
     }
 }
 
+pub struct PrintStackmaps {
+    pub flag_print: bool,
+    pub text: String,
+}
+
+impl PrintStackmaps {
+    pub fn new(flag_print: bool) -> PrintStackmaps {
+        Self {
+            flag_print,
+            text: String::new(),
+        }
+    }
+}
+
+impl binemit::StackmapSink for PrintStackmaps {
+    fn add_stackmap(&mut self, offset: binemit::CodeOffset, _: binemit::Stackmap) {
+        if self.flag_print {
+            write!(&mut self.text, "add_stackmap at {}\n", offset).unwrap();
+        }
+    }
+}
+
 cfg_if! {
     if #[cfg(feature = "disas")] {
         use capstone::prelude::*;
@@ -170,11 +192,12 @@ pub fn print_all(
     rodata_size: u32,
     relocs: &PrintRelocs,
     traps: &PrintTraps,
+    stackmaps: &PrintStackmaps,
 ) -> Result<(), String> {
     print_bytes(&mem);
     print_disassembly(isa, &mem[0..code_size as usize])?;
     print_readonly_data(&mem[code_size as usize..(code_size + rodata_size) as usize]);
-    println!("\n{}\n{}", &relocs.text, &traps.text);
+    println!("\n{}\n{}\n{}", &relocs.text, &traps.text, &stackmaps.text);
     Ok(())
 }
 
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index dadb13389f..f58c5ae863 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -7,7 +7,7 @@
     allow(clippy::too_many_arguments, clippy::cyclomatic_complexity)
 )]
 
-use crate::disasm::{print_all, PrintRelocs, PrintTraps};
+use crate::disasm::{print_all, PrintRelocs, PrintTraps, PrintStackmaps};
 use crate::utils::{parse_sets_and_triple, read_to_end};
 use cranelift_codegen::print_errors::{pretty_error, pretty_verifier_error};
 use cranelift_codegen::settings::FlagsOrIsa;
@@ -171,13 +171,14 @@ fn handle_module(
         let mut mem = vec![];
         let mut relocs = PrintRelocs::new(flag_print);
         let mut traps = PrintTraps::new(flag_print);
+        let mut stackmaps = PrintStackmaps::new(flag_print);
         if flag_check_translation {
             if let Err(errors) = context.verify(fisa) {
                 return Err(pretty_verifier_error(&context.func, fisa.isa, None, errors));
             }
         } else {
             let code_info = context
-                .compile_and_emit(isa, &mut mem, &mut relocs, &mut traps)
+                .compile_and_emit(isa, &mut mem, &mut relocs, &mut traps, &mut stackmaps)
                 .map_err(|err| pretty_error(&context.func, fisa.isa, err))?;
 
             if flag_print_size {
@@ -223,7 +224,7 @@ fn handle_module(
         }
 
         if let Some((code_size, rodata_size)) = saved_sizes {
-            print_all(isa, &mem, code_size, rodata_size, &relocs, &traps)?;
+            print_all(isa, &mem, code_size, rodata_size, &relocs, &traps, &stackmaps)?;
         }
 
         context.clear();
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 05c5de4f55..f3accb825b 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -834,6 +834,12 @@ pub fn translate_operator(
         Operator::F32Le | Operator::F64Le => {
             translate_fcmp(FloatCC::LessThanOrEqual, builder, state)
         }
+        Operator::RefNull => state.push1(builder.ins().null(environ.reference_type())),
+        Operator::RefIsNull => {
+            let arg = state.pop1();
+            let val = builder.ins().is_null(arg);
+            state.push1(val);
+        }
         Operator::Wake { .. }
         | Operator::I32Wait { .. }
         | Operator::I64Wait { .. }
@@ -902,9 +908,6 @@ pub fn translate_operator(
         | Operator::I64AtomicRmw32UCmpxchg { .. } => {
             wasm_unsupported!("proposed thread operator {:?}", op);
         }
-        Operator::RefNull | Operator::RefIsNull { .. } => {
-            wasm_unsupported!("proposed reference-type operator {:?}", op);
-        }
         Operator::MemoryInit { .. }
         | Operator::DataDrop { .. }
         | Operator::MemoryCopy
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index c94131fa9f..59208ea562 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -131,6 +131,17 @@ pub trait FuncEnvironment {
         ReturnMode::NormalReturns
     }
 
+    /// Get the Cranelift reference type to use for native references.
+    ///
+    /// This returns `R64` for 64-bit architectures and `R32` for 32-bit architectures.
+    fn reference_type(&self) -> ir::Type {
+        match self.pointer_type() {
+            ir::types::I32 => ir::types::R32,
+            ir::types::I64 => ir::types::R64,
+            _ => panic!("unsupported pointer type"),
+        }
+    }
+
     /// Set up the necessary preamble definitions in `func` to access the global variable
     /// identified by `index`.
     ///
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index 7f62f5cfe0..b49cffd474 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -104,7 +104,7 @@ impl FuncTranslator {
         builder.append_ebb_params_for_function_returns(exit_block);
         self.state.initialize(&builder.func.signature, exit_block);
 
-        parse_local_decls(&mut reader, &mut builder, num_params)?;
+        parse_local_decls(&mut reader, &mut builder, num_params, environ)?;
         parse_function_body(reader, &mut builder, &mut self.state, environ)?;
 
         builder.finalize();
@@ -143,10 +143,11 @@ fn declare_wasm_parameters(builder: &mut FunctionBuilder, entry_block: Ebb) -> u
 /// Parse the local variable declarations that precede the function body.
 ///
 /// Declare local variables, starting from `num_params`.
-fn parse_local_decls(
+fn parse_local_decls(
     reader: &mut BinaryReader,
     builder: &mut FunctionBuilder,
     num_params: usize,
+    environ: &mut FE,
 ) -> WasmResult<()> {
     let mut next_local = num_params;
     let local_count = reader.read_local_count()?;
@@ -155,7 +156,7 @@ fn parse_local_decls(
     for _ in 0..local_count {
         builder.set_srcloc(cur_srcloc(reader));
         let (count, ty) = reader.read_local_decl(&mut locals_total)?;
-        declare_locals(builder, count, ty, &mut next_local)?;
+        declare_locals(builder, count, ty, &mut next_local, environ)?;
     }
 
     Ok(())
@@ -164,11 +165,12 @@ fn parse_local_decls(
 /// Declare `count` local variables of the same type, starting from `next_local`.
 ///
 /// Fail of too many locals are declared in the function, or if the type is not valid for a local.
-fn declare_locals(
+fn declare_locals(
     builder: &mut FunctionBuilder,
     count: u32,
     wasm_type: wasmparser::Type,
     next_local: &mut usize,
+    environ: &mut FE,
 ) -> WasmResult<()> {
     // All locals are initialized to 0.
     use wasmparser::Type::*;
@@ -177,6 +179,7 @@ fn declare_locals(
         I64 => builder.ins().iconst(ir::types::I64, 0),
         F32 => builder.ins().f32const(ir::immediates::Ieee32::with_bits(0)),
         F64 => builder.ins().f64const(ir::immediates::Ieee64::with_bits(0)),
+        AnyRef => builder.ins().null(environ.reference_type()),
         ty => wasm_unsupported!("unsupported local type {:?}", ty),
     };
 

From bf77985e25003d0571c5ff70b68923e14b865b87 Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Fri, 16 Aug 2019 10:38:37 -0400
Subject: [PATCH 2574/3084] Fix broken links using rustdoc nightly

Uses cross-crate documentation links so that rustdoc does the hard work
of making relative links for us.

Requires nightly version of rustdoc in order to generate links based on
path names, see
https://github.com/rust-lang/rfcs/blob/master/text/1946-intra-rustdoc-links.md
for details.
---
 cranelift/codegen/src/binemit/memorysink.rs | 2 +-
 cranelift/faerie/src/backend.rs             | 2 +-
 cranelift/frontend/src/frontend.rs          | 6 +++---
 cranelift/test-all.sh                       | 6 +++++-
 cranelift/wasm/src/module_translator.rs     | 2 +-
 5 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/cranelift/codegen/src/binemit/memorysink.rs b/cranelift/codegen/src/binemit/memorysink.rs
index cde6cd9c9a..b543d3e89c 100644
--- a/cranelift/codegen/src/binemit/memorysink.rs
+++ b/cranelift/codegen/src/binemit/memorysink.rs
@@ -85,7 +85,7 @@ pub trait RelocSink {
 /// A trait for receiving trap codes and offsets.
 ///
 /// If you don't need information about possible traps, you can use the
-/// [`NullTrapSink`](binemit/trait.TrapSink.html) implementation.
+/// [`NullTrapSink`](NullTrapSink) implementation.
 pub trait TrapSink {
     /// Add trap information for a specific offset.
     fn trap(&mut self, _: CodeOffset, _: SourceLoc, _: TrapCode);
diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index bed56122f0..a33955a212 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -38,7 +38,7 @@ pub struct FaerieBuilder {
 impl FaerieBuilder {
     /// Create a new `FaerieBuilder` using the given Cranelift target, that
     /// can be passed to
-    /// [`Module::new`](cranelift_module/struct.Module.html#method.new].
+    /// [`Module::new`](cranelift_module::Module::new)
     ///
     /// Faerie output requires that TargetIsa have PIC (Position Independent Code) enabled.
     ///
diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index a68f131458..0961314559 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -96,7 +96,7 @@ impl FunctionBuilderContext {
     }
 }
 
-/// Implementation of the [`InstBuilder`](../codegen/ir/builder/trait.InstBuilder.html) that has
+/// Implementation of the [`InstBuilder`](cranelift_codegen::ir::InstBuilder) that has
 /// one convenience method per Cranelift IR instruction.
 pub struct FuncInstBuilder<'short, 'long: 'short> {
     builder: &'short mut FunctionBuilder<'long>,
@@ -207,7 +207,7 @@ impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
 /// modifies with the information stored in the mutable borrowed
 /// [`FunctionBuilderContext`](struct.FunctionBuilderContext.html). The function passed in
 /// argument should be newly created with
-/// [`Function::with_name_signature()`](../function/struct.Function.html), whereas the
+/// [`Function::with_name_signature()`](Function::with_name_signature), whereas the
 /// `FunctionBuilderContext` can be kept as is between two function translations.
 ///
 /// # Errors
@@ -391,7 +391,7 @@ impl<'a> FunctionBuilder<'a> {
         self.func.create_heap(data)
     }
 
-    /// Returns an object with the [`InstBuilder`](../codegen/ir/builder/trait.InstBuilder.html)
+    /// Returns an object with the [`InstBuilder`](cranelift_codegen::ir::InstBuilder)
     /// trait that allows to conveniently append an instruction to the current `Ebb` being built.
     pub fn ins<'short>(&'short mut self) -> FuncInstBuilder<'short, 'a> {
         let ebb = self
diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index 64f4343bf5..2509430414 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -49,7 +49,11 @@ RUST_BACKTRACE=1 cargo test --all
 
 # Make sure the documentation builds.
 banner "Rust documentation: $topdir/target/doc/cranelift/index.html"
-cargo doc
+cargo +nightly doc
+
+# Make sure the documentation doesn't have broken links.
+banner "Rust documentation link test"
+find ./target/doc -maxdepth 1 -type d -name "cranelift*" | xargs -I{} cargo deadlinks --dir {}
 
 # Ensure fuzzer works by running it with a single input
 # Note LSAN is disabled due to https://github.com/google/sanitizers/issues/764
diff --git a/cranelift/wasm/src/module_translator.rs b/cranelift/wasm/src/module_translator.rs
index 669f9154e8..b35d1d4e38 100644
--- a/cranelift/wasm/src/module_translator.rs
+++ b/cranelift/wasm/src/module_translator.rs
@@ -10,7 +10,7 @@ use cranelift_codegen::timing;
 use wasmparser::{ModuleReader, SectionCode};
 
 /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cranelift IR
-/// [`Function`](../codegen/ir/function/struct.Function.html).
+/// [`Function`](cranelift_codegen::ir::Function).
 pub fn translate_module<'data>(
     data: &'data [u8],
     environ: &mut dyn ModuleEnvironment<'data>,

From 3b0695bef407aec58d851100d025cd453dbe2b3d Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Fri, 16 Aug 2019 10:40:32 -0400
Subject: [PATCH 2575/3084] Installed deadlinks if not already installed

If nightly isn't installed, it also gives a warning that some links will
be broken
---
 cranelift/test-all.sh | 41 ++++++++++++++++++++++++++++++-----------
 1 file changed, 30 insertions(+), 11 deletions(-)

diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index 2509430414..48e53116ea 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -47,25 +47,44 @@ cargo build
 banner "Rust unit tests"
 RUST_BACKTRACE=1 cargo test --all
 
+has_toolchain() {
+    rustup toolchain list | grep -q $1
+}
+
+ensure_installed() {
+    program="$1"
+    toolchain="${2:-stable}"
+    if has_toolchain $toolchain; then
+        if cargo +$toolchain install --list | grep -q $program; then
+            echo "$program found"
+        else
+            echo "installing $program"
+            cargo +$toolchain install $program
+        fi
+    else
+        return 1
+    fi
+}
+
 # Make sure the documentation builds.
 banner "Rust documentation: $topdir/target/doc/cranelift/index.html"
-cargo +nightly doc
+if has_toolchain nightly; then
+    cargo +nightly doc
 
-# Make sure the documentation doesn't have broken links.
-banner "Rust documentation link test"
-find ./target/doc -maxdepth 1 -type d -name "cranelift*" | xargs -I{} cargo deadlinks --dir {}
+    # Make sure the documentation doesn't have broken links.
+    banner "Rust documentation link test"
+    ensure_installed cargo-deadlinks
+    find ./target/doc -maxdepth 1 -type d -name "cranelift*" | xargs -I{} cargo deadlinks --dir {}
+else
+    cargo doc
+    echo "nightly toolchain not found, some documentation links will not work"
+fi
 
 # Ensure fuzzer works by running it with a single input
 # Note LSAN is disabled due to https://github.com/google/sanitizers/issues/764
 banner "cargo fuzz check"
-if rustup toolchain list | grep -q nightly; then
-    if cargo install --list | grep -q cargo-fuzz; then
-        echo "cargo-fuzz found"
-    else
-        echo "installing cargo-fuzz"
-        cargo +nightly install cargo-fuzz
-    fi
 
+if ensure_installed cargo-fuzz nightly; then
     fuzz_module="ffaefab69523eb11935a9b420d58826c8ea65c4c"
     ASAN_OPTIONS=detect_leaks=0 \
     cargo +nightly fuzz run fuzz_translate_module \

From 036b9aea946ce956c33d0ddc9ad55e86fd407fd0 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 19 Aug 2019 14:59:19 +0200
Subject: [PATCH 2576/3084] [codegen] Rename and explicit usage of
 replace_with_alias;

It can actually only replace one result; don't try to make it generic
yet, since there's no point in doing so right now, and make it do the
dumb thing so it's not surprising to use.
---
 cranelift/codegen/src/simple_preopt.rs | 33 +++++++++++++-------------
 1 file changed, 17 insertions(+), 16 deletions(-)

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index 92cbb4c44f..8219f6627c 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -19,13 +19,14 @@ use crate::ir::{
 use crate::timing;
 
 #[inline]
-/// Replaces the result_index nth result of instruction inst to an alias of the given value, and
-/// replaces the instruction with a nop.
-fn replace_with_alias(dfg: &mut DataFlowGraph, inst: Inst, result_index: usize, value: Value) {
+/// Replaces the unique result of the instruction inst to an alias of the given value, and
+/// replaces the instruction with a nop. Can be used only on instructions producing one unique
+/// result, otherwise will assert.
+fn replace_single_result_with_alias(dfg: &mut DataFlowGraph, inst: Inst, value: Value) {
     // Replace the result value by an alias.
     let results = dfg.detach_results(inst);
-    debug_assert!(results.len(&dfg.value_lists) > result_index);
-    let result = results.get(result_index, &dfg.value_lists).unwrap();
+    debug_assert!(results.len(&dfg.value_lists) == 1);
+    let result = results.get(0, &dfg.value_lists).unwrap();
     dfg.change_to_alias(result, value);
 
     // Replace instruction by a nop.
@@ -185,7 +186,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             if is_rem {
                 pos.func.dfg.replace(inst).iconst(I32, 0);
             } else {
-                replace_with_alias(&mut pos.func.dfg, inst, 0, n1);
+                replace_single_result_with_alias(&mut pos.func.dfg, inst, n1);
             }
         }
 
@@ -240,7 +241,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                 let tt = pos.ins().imul_imm(qf, d as i64);
                 pos.func.dfg.replace(inst).isub(n1, tt);
             } else {
-                replace_with_alias(&mut pos.func.dfg, inst, 0, qf);
+                replace_single_result_with_alias(&mut pos.func.dfg, inst, qf);
             }
         }
 
@@ -255,7 +256,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             if is_rem {
                 pos.func.dfg.replace(inst).iconst(I64, 0);
             } else {
-                replace_with_alias(&mut pos.func.dfg, inst, 0, n1);
+                replace_single_result_with_alias(&mut pos.func.dfg, inst, n1);
             }
         }
 
@@ -310,7 +311,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                 let tt = pos.ins().imul_imm(qf, d as i64);
                 pos.func.dfg.replace(inst).isub(n1, tt);
             } else {
-                replace_with_alias(&mut pos.func.dfg, inst, 0, qf);
+                replace_single_result_with_alias(&mut pos.func.dfg, inst, qf);
             }
         }
 
@@ -328,7 +329,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             if is_rem {
                 pos.func.dfg.replace(inst).iconst(I32, 0);
             } else {
-                replace_with_alias(&mut pos.func.dfg, inst, 0, n1);
+                replace_single_result_with_alias(&mut pos.func.dfg, inst, n1);
             }
         }
 
@@ -354,7 +355,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                     if is_negative {
                         pos.func.dfg.replace(inst).irsub_imm(t4, 0);
                     } else {
-                        replace_with_alias(&mut pos.func.dfg, inst, 0, t4);
+                        replace_single_result_with_alias(&mut pos.func.dfg, inst, t4);
                     }
                 }
             } else {
@@ -384,7 +385,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                     let tt = pos.ins().imul_imm(qf, d as i64);
                     pos.func.dfg.replace(inst).isub(n1, tt);
                 } else {
-                    replace_with_alias(&mut pos.func.dfg, inst, 0, qf);
+                    replace_single_result_with_alias(&mut pos.func.dfg, inst, qf);
                 }
             }
         }
@@ -403,7 +404,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
             if is_rem {
                 pos.func.dfg.replace(inst).iconst(I64, 0);
             } else {
-                replace_with_alias(&mut pos.func.dfg, inst, 0, n1);
+                replace_single_result_with_alias(&mut pos.func.dfg, inst, n1);
             }
         }
 
@@ -429,7 +430,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                     if is_negative {
                         pos.func.dfg.replace(inst).irsub_imm(t4, 0);
                     } else {
-                        replace_with_alias(&mut pos.func.dfg, inst, 0, t4);
+                        replace_single_result_with_alias(&mut pos.func.dfg, inst, t4);
                     }
                 }
             } else {
@@ -459,7 +460,7 @@ fn do_divrem_transformation(divrem_info: &DivRemByConstInfo, pos: &mut FuncCurso
                     let tt = pos.ins().imul_imm(qf, d);
                     pos.func.dfg.replace(inst).isub(n1, tt);
                 } else {
-                    replace_with_alias(&mut pos.func.dfg, inst, 0, qf);
+                    replace_single_result_with_alias(&mut pos.func.dfg, inst, qf);
                 }
             }
         }
@@ -665,7 +666,7 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
                 | (Opcode::UshrImm, 0)
                 | (Opcode::SshrImm, 0) => {
                     // Alias the result value with the original argument.
-                    replace_with_alias(&mut pos.func.dfg, inst, 0, arg);
+                    replace_single_result_with_alias(&mut pos.func.dfg, inst, arg);
                     return;
                 }
                 (Opcode::ImulImm, 0) | (Opcode::BandImm, 0) => {

From 3fdc78174ff3fe30d807961821ca870513acf3e2 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 11 Jul 2019 15:49:28 -0700
Subject: [PATCH 2577/3084] Add x86 implementation of extractlane instruction

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 29 +++++++++++++--
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 23 +++++++++++-
 .../filetests/isa/x86/extractlane.clif        | 35 +++++++++++++++++++
 3 files changed, 84 insertions(+), 3 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/extractlane.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index eaa5614bd0..ed4cf18d94 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -318,6 +318,7 @@ pub fn define(
     let copy_special = shared.by_name("copy_special");
     let ctz = shared.by_name("ctz");
     let debugtrap = shared.by_name("debugtrap");
+    let extractlane = shared.by_name("extractlane");
     let f32const = shared.by_name("f32const");
     let f64const = shared.by_name("f64const");
     let fadd = shared.by_name("fadd");
@@ -498,7 +499,8 @@ pub fn define(
     let rec_pushq = r.template("pushq");
     let rec_ret = r.template("ret");
     let rec_r_ib = r.template("r_ib");
-    let rec_r_ib_unsigned = r.template("r_ib_unsigned");
+    let rec_r_ib_unsigned_gpr = r.template("r_ib_unsigned_gpr");
+    let rec_r_ib_unsigned_fpr = r.template("r_ib_unsigned_fpr");
     let rec_r_ib_unsigned_r = r.template("r_ib_unsigned_r");
     let rec_r_id = r.template("r_id");
     let rec_rcmp = r.template("rcmp");
@@ -1642,7 +1644,9 @@ pub fn define(
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 32) {
         let number_of_lanes = 128 / ty.lane_bits();
         let instruction = x86_pshufd.bind_vector(ty, number_of_lanes);
-        let template = rec_r_ib_unsigned.nonrex().opcodes(vec![0x66, 0x0f, 0x70]);
+        let template = rec_r_ib_unsigned_fpr
+            .nonrex()
+            .opcodes(vec![0x66, 0x0f, 0x70]);
         e.enc32_isap(instruction.clone(), template.clone(), use_sse2);
         e.enc64_isap(instruction, template, use_sse2);
     }
@@ -1682,6 +1686,27 @@ pub fn define(
         }
     }
 
+    // SIMD extractlane
+    let mut extractlane_mapping: HashMap, SettingPredicateNumber)> = HashMap::new();
+    extractlane_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x14], use_sse41)); // PEXTRB
+    extractlane_mapping.insert(16, (vec![0x66, 0x0f, 0xc5], use_sse2)); // PEXTRW, SSE4.1 has a PEXTRW that can move to reg/m16 but the opcode is four bytes
+    extractlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x16], use_sse41)); // PEXTRD
+    extractlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x16], use_sse41)); // PEXTRQ, only x86_64
+
+    for ty in ValueType::all_lane_types() {
+        if let Some((opcode, isap)) = extractlane_mapping.get(&ty.lane_bits()) {
+            let number_of_lanes = 128 / ty.lane_bits();
+            let instruction = extractlane.bind_vector(ty, number_of_lanes);
+            let template = rec_r_ib_unsigned_gpr.opcodes(opcode.clone());
+            if ty.lane_bits() < 64 {
+                e.enc_32_64_isap(instruction, template.nonrex(), isap.clone());
+            } else {
+                // turns out the 64-bit widths have REX/W encodings and only are available on x86_64
+                e.enc64_isap(instruction, template.rex().w(), isap.clone());
+            }
+        }
+    }
+
     // SIMD bitcast f64 to all 8-bit-lane vectors (for legalizing splat.x8x16); assumes that f64 is stored in an XMM register
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
         let instruction = bitcast.bind_vector(ty, 16).bind(F64);
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index b948c0c2e5..623689cea9 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -800,7 +800,7 @@ pub fn define<'shared>(
     {
         let format = formats.get(f_extract_lane);
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("r_ib_unsigned", f_extract_lane, 2)
+            EncodingRecipeBuilder::new("r_ib_unsigned_fpr", f_extract_lane, 2)
                 .operands_in(vec![fpr])
                 .operands_out(vec![fpr])
                 .inst_predicate(InstructionPredicate::new_is_unsigned_int(
@@ -817,6 +817,27 @@ pub fn define<'shared>(
         );
     }
 
+    // XX /r ib with 8-bit unsigned immediate (e.g. for extractlane)
+    {
+        let format = formats.get(f_extract_lane);
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("r_ib_unsigned_gpr", f_extract_lane, 2)
+                .operands_in(vec![fpr])
+                .operands_out(vec![gpr])
+                .inst_predicate(InstructionPredicate::new_is_unsigned_int(
+                    format, "lane", 8, 0,
+                ))
+                .emit(
+                    r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                    modrm_rr(out_reg0, in_reg0, sink); // note the flipped register in the ModR/M byte
+                    let imm:i64 = lane.into();
+                    sink.put1(imm as u8);
+                "#,
+                ),
+        );
+    }
+
     // XX /r ib with 8-bit unsigned immediate (e.g. for insertlane)
     {
         let format = formats.get(f_insert_lane);
diff --git a/cranelift/filetests/filetests/isa/x86/extractlane.clif b/cranelift/filetests/filetests/isa/x86/extractlane.clif
new file mode 100644
index 0000000000..e7a1ea898e
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/extractlane.clif
@@ -0,0 +1,35 @@
+test binemit
+set enable_simd
+target x86_64 haswell
+
+function %test_extractlane_b8() {
+ebb0:
+[-, %rax]   v0 = bconst.b8 true
+[-, %xmm0]  v1 = splat.b8x16 v0
+[-, %rax]   v2 = extractlane v1, 10    ; bin: 66 0f 3a 14 c0 0a
+            return
+}
+
+function %test_extractlane_i16() {
+ebb0:
+[-, %rax]   v0 = iconst.i16 4
+[-, %xmm1]  v1 = splat.i16x8 v0
+[-, %rax]   v2 = extractlane v1, 4    ; bin: 66 0f c5 c8 04
+            return
+}
+
+function %test_extractlane_i32() {
+ebb0:
+[-, %rax]   v0 = iconst.i32 42
+[-, %xmm4]  v1 = splat.i32x4 v0
+[-, %rcx]   v2 = extractlane v1, 2    ; bin: 66 0f 3a 16 e1 02
+            return
+}
+
+function %test_extractlane_f64() {
+ebb0:
+[-, %rax]   v0 = f64const 0x0.0
+[-, %xmm2]  v1 = splat.f64x2 v0
+[-, %rbx]   v2 = extractlane v1, 1    ; bin: 66 48 0f 3a 16 d3 01
+            return
+}

From d492cf7e0e57e24e37b870fd11142ec630fff848 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 11 Jul 2019 15:59:39 -0700
Subject: [PATCH 2578/3084] Avoid unnecessary lane calculations in codegen code

This refactor moves the calculation of the number of lanes to code closer to where the Instruction/BoundInstruction is bound.
---
 .../codegen/meta/src/cdsl/instructions.rs     | 29 +++++++++++++++----
 .../codegen/meta/src/isa/x86/encodings.rs     | 28 +++++++++---------
 .../codegen/meta/src/isa/x86/legalize.rs      | 23 ++++++++-------
 3 files changed, 51 insertions(+), 29 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 43a879066c..2a40d5128d 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -181,8 +181,17 @@ impl Instruction {
         bind_ref(self.clone(), Some(reference_type.into()), Vec::new())
     }
 
-    pub fn bind_vector(&self, lane_type: impl Into, num_lanes: u64) -> BoundInstruction {
-        bind_vector(self.clone(), lane_type.into(), num_lanes, Vec::new())
+    pub fn bind_vector_from_lane(
+        &self,
+        lane_type: impl Into,
+        vector_size_in_bits: u64,
+    ) -> BoundInstruction {
+        bind_vector(
+            self.clone(),
+            lane_type.into(),
+            vector_size_in_bits,
+            Vec::new(),
+        )
     }
 
     pub fn bind_any(&self) -> BoundInstruction {
@@ -414,8 +423,17 @@ impl BoundInstruction {
         bind_ref(self.inst, Some(reference_type.into()), self.value_types)
     }
 
-    pub fn bind_vector(self, lane_type: impl Into, num_lanes: u64) -> BoundInstruction {
-        bind_vector(self.inst, lane_type.into(), num_lanes, self.value_types)
+    pub fn bind_vector_from_lane(
+        self,
+        lane_type: impl Into,
+        vector_size_in_bits: u64,
+    ) -> BoundInstruction {
+        bind_vector(
+            self.inst,
+            lane_type.into(),
+            vector_size_in_bits,
+            self.value_types,
+        )
     }
 
     pub fn bind_any(self) -> BoundInstruction {
@@ -1116,9 +1134,10 @@ fn bind_ref(
 fn bind_vector(
     inst: Instruction,
     lane_type: LaneType,
-    num_lanes: u64,
+    vector_size_in_bits: u64,
     mut value_types: Vec,
 ) -> BoundInstruction {
+    let num_lanes = vector_size_in_bits / lane_type.lane_bits();
     let vector_type = ValueType::Vector(VectorType::new(lane_type, num_lanes));
     value_types.push(ValueTypeOrAny::ValueType(vector_type));
     verify_polymorphic_binding(&inst, &value_types);
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index ed4cf18d94..3b4be51a66 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1627,23 +1627,24 @@ pub fn define(
     e.enc_both(ffcmp.bind(F32), rec_fcmp.opcodes(vec![0x0f, 0x2e]));
     e.enc_both(ffcmp.bind(F64), rec_fcmp.opcodes(vec![0x66, 0x0f, 0x2e]));
 
+    // SIMD vector size: eventually multiple vector sizes may be supported but for now only SSE-sized vectors are available
+    let sse_vector_size: u64 = 128;
+
     // SIMD splat: before x86 can use vector data, it must be moved to XMM registers; see
     // legalize.rs for how this is done; once there, x86_pshuf* (below) is used for broadcasting the
     // value across the register
 
     // PSHUFB, 8-bit shuffle using two XMM registers
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
-        let number_of_lanes = 128 / ty.lane_bits();
-        let instruction = x86_pshufb.bind_vector(ty, number_of_lanes);
-        let template = rec_fa.nonrex().opcodes(vec![0x66, 0x0f, 0x38, 0x00]);
+        let instruction = x86_pshufb.bind_vector_from_lane(ty, sse_vector_size);
+        let template = rec_fa.nonrex().opcodes(vec![0x66, 0x0f, 0x38, 00]);
         e.enc32_isap(instruction.clone(), template.clone(), use_ssse3);
         e.enc64_isap(instruction, template, use_ssse3);
     }
 
     // PSHUFD, 32-bit shuffle using one XMM register and a u8 immediate
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 32) {
-        let number_of_lanes = 128 / ty.lane_bits();
-        let instruction = x86_pshufd.bind_vector(ty, number_of_lanes);
+        let instruction = x86_pshufd.bind_vector_from_lane(ty, sse_vector_size);
         let template = rec_r_ib_unsigned_fpr
             .nonrex()
             .opcodes(vec![0x66, 0x0f, 0x70]);
@@ -1655,8 +1656,9 @@ pub fn define(
     // to the Intel manual: "When the destination operand is an XMM register, the source operand is
     // written to the low doubleword of the register and the regiser is zero-extended to 128 bits."
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() >= 8) {
-        let number_of_lanes = 128 / ty.lane_bits();
-        let instruction = scalar_to_vector.bind_vector(ty, number_of_lanes).bind(ty);
+        let instruction = scalar_to_vector
+            .bind_vector_from_lane(ty, sse_vector_size)
+            .bind(ty);
         let template = rec_frurm.opcodes(vec![0x66, 0x0f, 0x6e]); // MOVD/MOVQ
         if ty.lane_bits() < 64 {
             // no 32-bit encodings for 64-bit widths
@@ -1674,8 +1676,7 @@ pub fn define(
 
     for ty in ValueType::all_lane_types() {
         if let Some((opcode, isap)) = insertlane_mapping.get(&ty.lane_bits()) {
-            let number_of_lanes = 128 / ty.lane_bits();
-            let instruction = insertlane.bind_vector(ty, number_of_lanes);
+            let instruction = insertlane.bind_vector_from_lane(ty, sse_vector_size);
             let template = rec_r_ib_unsigned_r.opcodes(opcode.clone());
             if ty.lane_bits() < 64 {
                 e.enc_32_64_isap(instruction, template.nonrex(), isap.clone());
@@ -1695,8 +1696,7 @@ pub fn define(
 
     for ty in ValueType::all_lane_types() {
         if let Some((opcode, isap)) = extractlane_mapping.get(&ty.lane_bits()) {
-            let number_of_lanes = 128 / ty.lane_bits();
-            let instruction = extractlane.bind_vector(ty, number_of_lanes);
+            let instruction = extractlane.bind_vector_from_lane(ty, sse_vector_size);
             let template = rec_r_ib_unsigned_gpr.opcodes(opcode.clone());
             if ty.lane_bits() < 64 {
                 e.enc_32_64_isap(instruction, template.nonrex(), isap.clone());
@@ -1709,7 +1709,7 @@ pub fn define(
 
     // SIMD bitcast f64 to all 8-bit-lane vectors (for legalizing splat.x8x16); assumes that f64 is stored in an XMM register
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
-        let instruction = bitcast.bind_vector(ty, 16).bind(F64);
+        let instruction = bitcast.bind_vector_from_lane(ty, sse_vector_size).bind(F64);
         e.enc32_rec(instruction.clone(), rec_null_fpr, 0);
         e.enc64_rec(instruction, rec_null_fpr, 0);
     }
@@ -1719,8 +1719,8 @@ pub fn define(
         for to_type in ValueType::all_lane_types().filter(|t| t.lane_bits() >= 8 && *t != from_type)
         {
             let instruction = raw_bitcast
-                .bind_vector(to_type, 128 / to_type.lane_bits())
-                .bind_vector(from_type, 128 / from_type.lane_bits());
+                .bind_vector_from_lane(to_type, sse_vector_size)
+                .bind_vector_from_lane(from_type, sse_vector_size);
             e.enc32_rec(instruction.clone(), rec_null_fpr, 0);
             e.enc64_rec(instruction, rec_null_fpr, 0);
         }
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 8423b10990..a64f631168 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -320,12 +320,15 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
     let c = var("c");
     let d = var("d");
 
+    // SIMD vector size: eventually multiple vector sizes may be supported but for now only SSE-sized vectors are available
+    let sse_vector_size: u64 = 128;
+
     // SIMD splat: 8-bits
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
-        let splat_x8x16 = splat.bind_vector(ty, 128 / ty.lane_bits());
-        let bitcast_f64_to_any8x16 = bitcast.bind_vector(ty, 128 / ty.lane_bits()).bind(F64);
+        let splat_any8x16 = splat.bind_vector_from_lane(ty, sse_vector_size);
+        let bitcast_f64_to_any8x16 = bitcast.bind_vector_from_lane(ty, sse_vector_size).bind(F64);
         narrow.legalize(
-            def!(y = splat_x8x16(x)),
+            def!(y = splat_any8x16(x)),
             vec![
                 def!(a = scalar_to_vector(x)), // move into the lowest 8 bits of an XMM register
                 def!(b = f64const(ieee64_zero)), // zero out a different XMM register; the shuffle mask for moving the lowest byte to all other byte lanes is 0x0
@@ -337,13 +340,13 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
 
     // SIMD splat: 16-bits
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 16) {
-        let splat_x16x8 = splat.bind_vector(ty, 128 / ty.lane_bits());
+        let splat_x16x8 = splat.bind_vector_from_lane(ty, sse_vector_size);
         let raw_bitcast_any16x8_to_i32x4 = raw_bitcast
-            .bind_vector(I32, 4)
-            .bind_vector(ty, 128 / ty.lane_bits());
+            .bind_vector_from_lane(I32, sse_vector_size)
+            .bind_vector_from_lane(ty, sse_vector_size);
         let raw_bitcast_i32x4_to_any16x8 = raw_bitcast
-            .bind_vector(ty, 128 / ty.lane_bits())
-            .bind_vector(I32, 4);
+            .bind_vector_from_lane(ty, sse_vector_size)
+            .bind_vector_from_lane(I32, sse_vector_size);
         narrow.legalize(
             def!(y = splat_x16x8(x)),
             vec![
@@ -358,7 +361,7 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
 
     // SIMD splat: 32-bits
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 32) {
-        let splat_any32x4 = splat.bind_vector(ty, 128 / ty.lane_bits());
+        let splat_any32x4 = splat.bind_vector_from_lane(ty, sse_vector_size);
         narrow.legalize(
             def!(y = splat_any32x4(x)),
             vec![
@@ -370,7 +373,7 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
 
     // SIMD splat: 64-bits
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 64) {
-        let splat_any64x2 = splat.bind_vector(ty, 128 / ty.lane_bits());
+        let splat_any64x2 = splat.bind_vector_from_lane(ty, sse_vector_size);
         narrow.legalize(
             def!(y = splat_any64x2(x)),
             vec![

From 6605f308b38fd72ef4f1b78efda4edc0811ba26b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 11 Jul 2019 16:10:42 -0700
Subject: [PATCH 2579/3084] Fix static analysis warnings

---
 cranelift/codegen/meta/src/cdsl/instructions.rs | 6 +++---
 cranelift/codegen/meta/src/isa/x86/recipes.rs   | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 2a40d5128d..f31e530771 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -79,7 +79,7 @@ impl InstructionGroup {
     pub fn by_name(&self, name: &'static str) -> &Instruction {
         self.instructions
             .iter()
-            .find(|inst| inst.name == name)
+            .find(|inst| &inst.name == name)
             .expect(&format!("unexisting instruction with name {}", name))
     }
 }
@@ -155,7 +155,7 @@ impl ops::Deref for Instruction {
 
 impl Instruction {
     pub fn snake_name(&self) -> &str {
-        if self.name == "return" {
+        if &self.name == "return" {
             "return_"
         } else {
             &self.name
@@ -800,7 +800,7 @@ impl InstructionPredicateNode {
                     ret.extend(node.collect_leaves());
                 }
             }
-            _ => ret.push(&self),
+            _ => ret.push(self),
         }
         ret
     }
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 623689cea9..3e773bc34c 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -53,7 +53,7 @@ impl<'builder> RecipeGroup<'builder> {
     pub fn recipe(&self, name: &str) -> &EncodingRecipe {
         self.recipes
             .iter()
-            .find(|recipe| recipe.name == name)
+            .find(|recipe| &recipe.name == name)
             .expect(&format!("unknown recipe name: {}. Try template?", name))
     }
     pub fn template(&self, name: &str) -> &Template {

From b4ef90cfcdae28122902717ce248cb5c34b4ab88 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 17 Jul 2019 09:45:58 -0700
Subject: [PATCH 2580/3084] Remove SSE2 setting for x86

In talking to @sunfishcode, he preferred to avoid the confusion of more ISA predicates by eliminating SSE2. SSE2 was released with the Pentium 4 in 2000 so it is unlikely that current CPUs would have SIMD enabled and not have this feature. I tried to note the SSE2-specific instructions with comments in the code.
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 71 +++++++++++++------
 .../codegen/meta/src/isa/x86/settings.rs      |  9 +--
 .../filetests/filetests/isa/x86/pshufb.clif   |  2 +-
 .../filetests/filetests/isa/x86/pshufd.clif   |  2 +-
 .../filetests/isa/x86/scalar_to_vector.clif   |  2 +-
 5 files changed, 53 insertions(+), 33 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 3b4be51a66..c5bbfe1a83 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -268,14 +268,38 @@ impl PerCpuModeEncodings {
     }
 
     /// Add the same encoding to both X86_32 and X86_64; assumes configuration (e.g. REX, operand binding) has already happened
-    fn enc_32_64_isap(
+    fn enc_32_64_maybe_isap(
         &mut self,
         inst: BoundInstruction,
         template: Template,
-        isap: SettingPredicateNumber,
+        isap: Option,
     ) {
-        self.enc32_isap(inst.clone(), template.clone(), isap);
-        self.enc64_isap(inst, template, isap);
+        self.enc32_maybe_isap(inst.clone(), template.clone(), isap);
+        self.enc64_maybe_isap(inst, template, isap);
+    }
+
+    fn enc32_maybe_isap(
+        &mut self,
+        inst: BoundInstruction,
+        template: Template,
+        isap: Option,
+    ) {
+        match isap {
+            None => self.enc32(inst, template),
+            Some(isap) => self.enc32_isap(inst, template, isap),
+        }
+    }
+
+    fn enc64_maybe_isap(
+        &mut self,
+        inst: BoundInstruction,
+        template: Template,
+        isap: Option,
+    ) {
+        match isap {
+            None => self.enc64(inst, template),
+            Some(isap) => self.enc64_isap(inst, template, isap),
+        }
     }
 }
 
@@ -559,7 +583,6 @@ pub fn define(
     let use_popcnt = settings.predicate_by_name("use_popcnt");
     let use_lzcnt = settings.predicate_by_name("use_lzcnt");
     let use_bmi1 = settings.predicate_by_name("use_bmi1");
-    let use_sse2 = settings.predicate_by_name("use_sse2");
     let use_ssse3 = settings.predicate_by_name("use_ssse3");
     let use_sse41 = settings.predicate_by_name("use_sse41");
 
@@ -1648,8 +1671,8 @@ pub fn define(
         let template = rec_r_ib_unsigned_fpr
             .nonrex()
             .opcodes(vec![0x66, 0x0f, 0x70]);
-        e.enc32_isap(instruction.clone(), template.clone(), use_sse2);
-        e.enc64_isap(instruction, template, use_sse2);
+        e.enc32(instruction.clone(), template.clone());
+        e.enc64(instruction, template);
     }
 
     // SIMD scalar_to_vector; this uses MOV to copy the scalar value to an XMM register; according
@@ -1662,47 +1685,49 @@ pub fn define(
         let template = rec_frurm.opcodes(vec![0x66, 0x0f, 0x6e]); // MOVD/MOVQ
         if ty.lane_bits() < 64 {
             // no 32-bit encodings for 64-bit widths
-            e.enc32_isap(instruction.clone(), template.clone(), use_sse2);
+            e.enc32(instruction.clone(), template.clone());
         }
-        e.enc_x86_64_isap(instruction, template, use_sse2);
+        e.enc_x86_64(instruction, template);
     }
 
     // SIMD insertlane
-    let mut insertlane_mapping: HashMap, SettingPredicateNumber)> = HashMap::new();
-    insertlane_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x20], use_sse41)); // PINSRB
-    insertlane_mapping.insert(16, (vec![0x66, 0x0f, 0xc4], use_sse2)); // PINSRW
-    insertlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x22], use_sse41)); // PINSRD
-    insertlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x22], use_sse41)); // PINSRQ, only x86_64
+    let mut insertlane_mapping: HashMap, Option)> =
+        HashMap::new();
+    insertlane_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x20], Some(use_sse41))); // PINSRB
+    insertlane_mapping.insert(16, (vec![0x66, 0x0f, 0xc4], None)); // PINSRW from SSE2
+    insertlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41))); // PINSRD
+    insertlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41))); // PINSRQ, only x86_64
 
     for ty in ValueType::all_lane_types() {
         if let Some((opcode, isap)) = insertlane_mapping.get(&ty.lane_bits()) {
             let instruction = insertlane.bind_vector_from_lane(ty, sse_vector_size);
             let template = rec_r_ib_unsigned_r.opcodes(opcode.clone());
             if ty.lane_bits() < 64 {
-                e.enc_32_64_isap(instruction, template.nonrex(), isap.clone());
+                e.enc_32_64_maybe_isap(instruction, template.nonrex(), isap.clone());
             } else {
                 // turns out the 64-bit widths have REX/W encodings and only are available on x86_64
-                e.enc64_isap(instruction, template.rex().w(), isap.clone());
+                e.enc64_maybe_isap(instruction, template.rex().w(), isap.clone());
             }
         }
     }
 
     // SIMD extractlane
-    let mut extractlane_mapping: HashMap, SettingPredicateNumber)> = HashMap::new();
-    extractlane_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x14], use_sse41)); // PEXTRB
-    extractlane_mapping.insert(16, (vec![0x66, 0x0f, 0xc5], use_sse2)); // PEXTRW, SSE4.1 has a PEXTRW that can move to reg/m16 but the opcode is four bytes
-    extractlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x16], use_sse41)); // PEXTRD
-    extractlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x16], use_sse41)); // PEXTRQ, only x86_64
+    let mut extractlane_mapping: HashMap, Option)> =
+        HashMap::new();
+    extractlane_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x14], Some(use_sse41))); // PEXTRB
+    extractlane_mapping.insert(16, (vec![0x66, 0x0f, 0xc5], None)); // PEXTRW from zSSE2, SSE4.1 has a PEXTRW that can move to reg/m16 but the opcode is four bytes
+    extractlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41))); // PEXTRD
+    extractlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41))); // PEXTRQ, only x86_64
 
     for ty in ValueType::all_lane_types() {
         if let Some((opcode, isap)) = extractlane_mapping.get(&ty.lane_bits()) {
             let instruction = extractlane.bind_vector_from_lane(ty, sse_vector_size);
             let template = rec_r_ib_unsigned_gpr.opcodes(opcode.clone());
             if ty.lane_bits() < 64 {
-                e.enc_32_64_isap(instruction, template.nonrex(), isap.clone());
+                e.enc_32_64_maybe_isap(instruction, template.nonrex(), isap.clone());
             } else {
                 // turns out the 64-bit widths have REX/W encodings and only are available on x86_64
-                e.enc64_isap(instruction, template.rex().w(), isap.clone());
+                e.enc64_maybe_isap(instruction, template.rex().w(), isap.clone());
             }
         }
     }
diff --git a/cranelift/codegen/meta/src/isa/x86/settings.rs b/cranelift/codegen/meta/src/isa/x86/settings.rs
index bc8c81f484..3a42553386 100644
--- a/cranelift/codegen/meta/src/isa/x86/settings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/settings.rs
@@ -3,9 +3,6 @@ use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 pub fn define(shared: &SettingGroup) -> SettingGroup {
     let mut settings = SettingGroupBuilder::new("x86");
 
-    // CPUID.01H:EDX
-    let has_sse2 = settings.add_bool("has_sse2", "SSE2: CPUID.01H:EDX.SSE2[bit 26]", false);
-
     // CPUID.01H:ECX
     let has_sse3 = settings.add_bool("has_sse3", "SSE3: CPUID.01H:ECX.SSE3[bit 0]", false);
     let has_ssse3 = settings.add_bool("has_ssse3", "SSSE3: CPUID.01H:ECX.SSSE3[bit 9]", false);
@@ -35,7 +32,6 @@ pub fn define(shared: &SettingGroup) -> SettingGroup {
 
     let shared_enable_simd = shared.get_bool("enable_simd");
 
-    settings.add_predicate("use_sse2", predicate!(shared_enable_simd && has_sse2));
     settings.add_predicate("use_ssse3", predicate!(shared_enable_simd && has_ssse3));
     settings.add_predicate("use_sse41", predicate!(shared_enable_simd && has_sse41));
     settings.add_predicate(
@@ -69,7 +65,7 @@ pub fn define(shared: &SettingGroup) -> SettingGroup {
     settings.add_preset("baseline", preset!());
     let nehalem = settings.add_preset(
         "nehalem",
-        preset!(has_sse2 && has_sse3 && has_ssse3 && has_sse41 && has_sse42 && has_popcnt),
+        preset!(has_sse3 && has_ssse3 && has_sse41 && has_sse42 && has_popcnt),
     );
     let haswell = settings.add_preset(
         "haswell",
@@ -82,8 +78,7 @@ pub fn define(shared: &SettingGroup) -> SettingGroup {
     settings.add_preset(
         "znver1",
         preset!(
-            has_sse2
-                && has_sse3
+            has_sse3
                 && has_ssse3
                 && has_sse41
                 && has_sse42
diff --git a/cranelift/filetests/filetests/isa/x86/pshufb.clif b/cranelift/filetests/filetests/isa/x86/pshufb.clif
index 7c23c5ab61..6fb31b198c 100644
--- a/cranelift/filetests/filetests/isa/x86/pshufb.clif
+++ b/cranelift/filetests/filetests/isa/x86/pshufb.clif
@@ -1,6 +1,6 @@
 test binemit
 set enable_simd
-target x86_64 has_sse2=true has_ssse3=true
+target x86_64 has_ssse3=true
 
 function %test_pshufb() {
 ebb0:
diff --git a/cranelift/filetests/filetests/isa/x86/pshufd.clif b/cranelift/filetests/filetests/isa/x86/pshufd.clif
index 183af4fc0e..6f4896d0d9 100644
--- a/cranelift/filetests/filetests/isa/x86/pshufd.clif
+++ b/cranelift/filetests/filetests/isa/x86/pshufd.clif
@@ -1,6 +1,6 @@
 test binemit
 set enable_simd
-target x86_64 has_sse2=true
+target x86_64
 
 function %test_pshuf() {
 ebb0:
diff --git a/cranelift/filetests/filetests/isa/x86/scalar_to_vector.clif b/cranelift/filetests/filetests/isa/x86/scalar_to_vector.clif
index 6c77dfafdb..51ddea3e7e 100644
--- a/cranelift/filetests/filetests/isa/x86/scalar_to_vector.clif
+++ b/cranelift/filetests/filetests/isa/x86/scalar_to_vector.clif
@@ -1,7 +1,7 @@
 test binemit
 set opt_level=best
 set enable_simd
-target x86_64 has_sse2=true
+target x86_64
 
 function %test_scalar_to_vector_b8() {
 ebb0:

From 0c2c59785255d87df32f637faa0c09e81adc4c9c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 20 Aug 2019 14:12:09 -0700
Subject: [PATCH 2581/3084] Update to latest versions of term, capstone, wabt,
 goblin, wasmparser.

---
 cranelift/Cargo.toml                      |  6 +++---
 cranelift/faerie/Cargo.toml               |  2 +-
 cranelift/filetests/src/test_safepoint.rs |  2 +-
 cranelift/src/bugpoint.rs                 |  8 ++++----
 cranelift/src/disasm.rs                   |  2 +-
 cranelift/wasm/Cargo.toml                 |  4 ++--
 cranelift/wasm/src/code_translator.rs     | 12 ++++++++----
 7 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 490fc8cdc2..c0aaed6d40 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -35,9 +35,9 @@ cranelift = { path = "cranelift-umbrella", version = "0.38.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
-term = "0.5.1"
-capstone = { version = "0.5.0", optional = true }
-wabt = { version = "0.7.0", optional = true }
+term = "0.6.1"
+capstone = { version = "0.6.0", optional = true }
+wabt = { version = "0.9.1", optional = true }
 target-lexicon = "0.4.0"
 pretty_env_logger = "0.3.0"
 file-per-thread-logger = "0.1.2"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 184808f7a1..7423d2ec17 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -13,7 +13,7 @@ edition = "2018"
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0" }
 cranelift-module = { path = "../cranelift-module", version = "0.38.0" }
 faerie = "0.10.0"
-goblin = "0.0.22"
+goblin = "0.0.24"
 failure = "0.1.2"
 target-lexicon = "0.4.0"
 
diff --git a/cranelift/filetests/src/test_safepoint.rs b/cranelift/filetests/src/test_safepoint.rs
index cec7368ae8..b213fb274d 100644
--- a/cranelift/filetests/src/test_safepoint.rs
+++ b/cranelift/filetests/src/test_safepoint.rs
@@ -6,7 +6,7 @@ use std::borrow::Cow;
 
 struct TestSafepoint;
 
-pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
     assert_eq!(parsed.command, "safepoint");
     if !parsed.options.is_empty() {
         Err(format!("No options allowed on {}", parsed))
diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index 643ed414bc..11110f0b00 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -596,7 +596,7 @@ fn resolve_aliases(func: &mut Function) {
 }
 
 fn reduce(
-    isa: &TargetIsa,
+    isa: &dyn TargetIsa,
     mut func: Function,
     verbose: bool,
 ) -> Result<(Function, String), String> {
@@ -658,10 +658,10 @@ struct CrashCheckContext<'a> {
     context: Context,
 
     /// The target isa to compile for.
-    isa: &'a TargetIsa,
+    isa: &'a dyn TargetIsa,
 }
 
-fn get_panic_string(panic: Box) -> String {
+fn get_panic_string(panic: Box) -> String {
     let panic = match panic.downcast::<&'static str>() {
         Ok(panic_msg) => panic_msg.to_owned(),
         Err(panic) => panic,
@@ -681,7 +681,7 @@ enum CheckResult {
 }
 
 impl<'a> CrashCheckContext<'a> {
-    fn new(isa: &'a TargetIsa) -> Self {
+    fn new(isa: &'a dyn TargetIsa) -> Self {
         CrashCheckContext {
             context: Context::new(),
             isa,
diff --git a/cranelift/src/disasm.rs b/cranelift/src/disasm.rs
index a24371206a..5bfa14e313 100644
--- a/cranelift/src/disasm.rs
+++ b/cranelift/src/disasm.rs
@@ -140,7 +140,7 @@ cfg_if! {
         }
 
         pub fn print_disassembly(isa: &dyn TargetIsa, mem: &[u8]) -> Result<(), String> {
-            let mut cs = get_disassembler(isa)?;
+            let cs = get_disassembler(isa)?;
 
             println!("\nDisassembly of {} bytes:", mem.len());
             let insns = cs.disasm_all(&mem, 0x0).unwrap();
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 410f815775..bfe3d06241 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -11,7 +11,7 @@ keywords = ["webassembly", "wasm"]
 edition = "2018"
 
 [dependencies]
-wasmparser = { version = "0.32.1", default-features = false }
+wasmparser = { version = "0.36.0", default-features = false }
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.38.0", default-features = false }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.38.0", default-features = false }
@@ -22,7 +22,7 @@ log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
 
 [dev-dependencies]
-wabt = "0.7.0"
+wabt = "0.9.1"
 target-lexicon = "0.4.0"
 
 [features]
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index f3accb825b..ad8bbda438 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -905,7 +905,8 @@ pub fn translate_operator(
         | Operator::I32AtomicRmw16UCmpxchg { .. }
         | Operator::I64AtomicRmw8UCmpxchg { .. }
         | Operator::I64AtomicRmw16UCmpxchg { .. }
-        | Operator::I64AtomicRmw32UCmpxchg { .. } => {
+        | Operator::I64AtomicRmw32UCmpxchg { .. }
+        | Operator::Fence { .. } => {
             wasm_unsupported!("proposed thread operator {:?}", op);
         }
         Operator::MemoryInit { .. }
@@ -1060,9 +1061,12 @@ pub fn translate_operator(
         | Operator::F32x4ConvertSI32x4
         | Operator::F32x4ConvertUI32x4
         | Operator::F64x2ConvertSI64x2
-        | Operator::F64x2ConvertUI64x2
-        | Operator::V8x16Shuffle1
-        | Operator::V8x16Shuffle2Imm { .. } => {
+        | Operator::F64x2ConvertUI64x2 { .. }
+        | Operator::V8x16Swizzle
+        | Operator::I8x16LoadSplat { .. }
+        | Operator::I16x8LoadSplat { .. }
+        | Operator::I32x4LoadSplat { .. }
+        | Operator::I64x2LoadSplat { .. } => {
             wasm_unsupported!("proposed SIMD operator {:?}", op);
         }
     };

From 0d54517d356c15c63fc586ac35ea9a6e3f87558c Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 20 Aug 2019 14:26:16 -0700
Subject: [PATCH 2582/3084] Bump version to 0.39.0

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    |  8 ++++----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 60 insertions(+), 60 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index c0aaed6d40..64471c4df5 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.38.0"
+version = "0.39.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.38.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.38.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.38.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.38.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.38.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.38.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.38.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.38.0" }
-cranelift-module = { path = "cranelift-module", version = "0.38.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.38.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.38.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.38.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.38.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.39.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.39.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.39.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.39.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.39.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.39.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.39.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.39.0" }
+cranelift-module = { path = "cranelift-module", version = "0.39.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.39.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.39.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.39.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.39.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index f98b9e3563..35e579892a 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.38.0"
+version = "0.39.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.38.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.39.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index e723a23309..9a7de70ef0 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.38.0"
+version = "0.39.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.38.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.38.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.39.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.39.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -27,7 +27,7 @@ serde = { version = "1.0.94", features = ["derive"], optional = true }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.38.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.39.0", default-features = false }
 
 [features]
 default = ["std", "x86", "arm32", "arm64", "riscv"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 9c90a691ad..3f4ff42f4c 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.38.0"
+version = "0.39.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.38.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.39.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index dc10dd8631..135726df36 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.38.0"
+version = "0.39.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 7423d2ec17..a6ed284c63 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.38.0"
+version = "0.39.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.38.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.39.0" }
 faerie = "0.10.0"
 goblin = "0.0.24"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 24a14e584d..449e97080a 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.38.0"
+version = "0.39.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../cranelift-reader", version = "0.38.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.38.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../cranelift-reader", version = "0.39.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.39.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 92d166fe51..921a0dd04f 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.38.0"
+version = "0.39.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index e4c0429a13..a849fe45b6 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.38.0"
+version = "0.39.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.38.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.39.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 560868a05a..d98466c666 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.38.0"
+version = "0.39.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 43d87d5055..476f27df91 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.38.0"
+version = "0.39.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.38.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.39.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index edd609b509..7fcabe49ea 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.38.0"
+version="0.39.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 255a4a0d50..6cf4e345a2 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.38.0"
+version = "0.39.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0" }
 target-lexicon = "0.4.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index de6ee0eb8c..f1477d641e 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.38.0"
+version = "0.39.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.38.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.39.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 3564132909..51f308bc1d 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.38.0"
+version = "0.39.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.38.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.38.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.39.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.39.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.38.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.38.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.38.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.39.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.39.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.39.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index f0b0f46ac1..ecca29146d 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.38.0"
+version = "0.39.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.38.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.39.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index bfe3d06241..b4f3b05c8a 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.38.0"
+version = "0.39.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.36.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.38.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.38.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.38.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.39.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.39.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 291afaf4ad0bfd750467f0d2c6339bf865106748 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 20 Aug 2019 15:12:27 -0700
Subject: [PATCH 2583/3084] Temporarily disable `fold_redundant_jumps`.

See #916 for details.
---
 cranelift/codegen/src/binemit/relaxation.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index 62f692e557..e5f7772210 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -57,7 +57,8 @@ pub fn relax_branches(
     func.offsets.resize(func.dfg.num_ebbs());
 
     // Start by removing redundant jumps.
-    fold_redundant_jumps(func, cfg, domtree);
+    // FIXME: Temporarily disabled due to #916.
+    /* fold_redundant_jumps(func, cfg, domtree); */
 
     // Convert jumps to fallthrough instructions where possible.
     fallthroughs(func);

From fbfeaaa32b18e11c70e99401d922af688a9b52ea Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 20 Aug 2019 15:25:39 -0700
Subject: [PATCH 2584/3084] Revert the test changes too.

In 1169dc520815fa4a6b1285914748408729a28caa, I forgot to git add the
test file changes.
---
 .../filetests/filetests/isa/x86/binary64.clif | 33 +++++++++----------
 .../filetests/isa/x86/legalize-br-table.clif  |  7 ++--
 2 files changed, 21 insertions(+), 19 deletions(-)

diff --git a/cranelift/filetests/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
index aa732e67f5..a65b3d3d1d 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64.clif
@@ -689,48 +689,48 @@ ebb0:
 
     ; asm: testq %rcx, %rcx
     ; asm: je ebb1
-    brz v1, ebb1                                ; bin: 48 85 c9 74 19
+    brz v1, ebb1                                ; bin: 48 85 c9 74 1b
     fallthrough ebb3
 
 ebb3:
     ; asm: testq %rsi, %rsi
     ; asm: je ebb1
-    brz v2, ebb1                                ; bin: 48 85 f6 74 14
+    brz v2, ebb1                                ; bin: 48 85 f6 74 16
     fallthrough ebb4
 
 ebb4:
     ; asm: testq %r10, %r10
     ; asm: je ebb1
-    brz v3, ebb1                                ; bin: 4d 85 d2 74 0f
+    brz v3, ebb1                                ; bin: 4d 85 d2 74 11
     fallthrough ebb5
 
 ebb5:
     ; asm: testq %rcx, %rcx
     ; asm: jne ebb1
-    brnz v1, ebb1                               ; bin: 48 85 c9 75 0a
+    brnz v1, ebb1                               ; bin: 48 85 c9 75 0c
     fallthrough ebb6
 
 ebb6:
     ; asm: testq %rsi, %rsi
     ; asm: jne ebb1
-    brnz v2, ebb1                               ; bin: 48 85 f6 75 05
+    brnz v2, ebb1                               ; bin: 48 85 f6 75 07
     fallthrough ebb7
 
 ebb7:
     ; asm: testq %r10, %r10
     ; asm: jne ebb1
-    brnz v3, ebb1                               ; bin: 4d 85 d2 75 00
+    brnz v3, ebb1                               ; bin: 4d 85 d2 75 02
 
     ; asm: jmp ebb2
-    jump ebb2
+    jump ebb2                                   ; bin: eb 01
 
     ; asm: ebb1:
 ebb1:
-    return
+    return                                       ; bin: c3
 
     ; asm: ebb2:
 ebb2:
-    jump ebb1
+    jump ebb1                                   ; bin: eb fd
 }
 
 ; CPU flag instructions.
@@ -1292,41 +1292,40 @@ ebb0:
 
     ; asm: testl %ecx, %ecx
     ; asm: je ebb1x
-    brz v1, ebb1                                ; bin: 85 c9 74 16
+    brz v1, ebb1                                ; bin: 85 c9 74 18
     fallthrough ebb3
 
 ebb3:
     ; asm: testl %esi, %esi
     ; asm: je ebb1x
-    brz v2, ebb1                                ; bin: 85 f6 74 12
+    brz v2, ebb1                                ; bin: 85 f6 74 14
     fallthrough ebb4
 
 ebb4:
     ; asm: testl %r10d, %r10d
     ; asm: je ebb1x
-    brz v3, ebb1                                ; bin: 45 85 d2 74 0d
+    brz v3, ebb1                                ; bin: 45 85 d2 74 0f
     fallthrough ebb5
 
 ebb5:
     ; asm: testl %ecx, %ecx
     ; asm: jne ebb1x
-    brnz v1, ebb1                               ; bin: 85 c9 75 09
+    brnz v1, ebb1                               ; bin: 85 c9 75 0b
     fallthrough ebb6
 
 ebb6:
     ; asm: testl %esi, %esi
     ; asm: jne ebb1x
-    brnz v2, ebb1                               ; bin: 85 f6 75 05
+    brnz v2, ebb1                               ; bin: 85 f6 75 07
     fallthrough ebb7
 
 ebb7:
     ; asm: testl %r10d, %r10d
     ; asm: jne ebb1x
-    brnz v3, ebb1                               ; bin: 45 85 d2 75 00
+    brnz v3, ebb1                               ; bin: 45 85 d2 75 02
 
     ; asm: jmp ebb2x
-    ; branch relaxation translates this into `fallthrough ebb1`
-    jump ebb2
+    jump ebb2                                   ; bin: eb 01
 
     ; asm: ebb1x:
 ebb1:
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
index 5656549cf7..6d49a6e6d0 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
@@ -11,9 +11,9 @@ function u0:0(i64) system_v {
 ebb0(v0: i64):
     v1 = stack_addr.i64 ss0
     v2 = load.i8 v1
-    br_table v2, ebb1, jt0
+    br_table v2, ebb2, jt0
 ; check:     $(oob=$V) = ifcmp_imm $(idx=$V), 1
-; nextln:    brif uge $oob, ebb1
+; nextln:    brif uge $oob, ebb2
 ; nextln:    fallthrough $(inb=$EBB)
 ; check:   $inb:
 ; nextln:    $(final_idx=$V) = uextend.i64 $idx
@@ -22,6 +22,9 @@ ebb0(v0: i64):
 ; nextln:    $(addr=$V) = iadd $base, $rel_addr
 ; nextln:    indirect_jump_table_br $addr, jt0
 
+ebb2:
+    jump ebb1
+
 ebb1:
     return
 }

From 13f83d8291dd58c8766c518fa2544ade1a5456ce Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 20 Aug 2019 15:32:19 -0700
Subject: [PATCH 2585/3084] Bump version to 0.40.0

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    |  8 ++++----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  4 ++--
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 61 insertions(+), 61 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 64471c4df5..a8c2addb05 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.39.0"
+version = "0.40.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.39.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.39.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.39.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.39.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.39.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.39.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.39.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.39.0" }
-cranelift-module = { path = "cranelift-module", version = "0.39.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.39.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.39.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.39.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.39.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.40.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.40.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.40.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.40.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.40.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.40.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.40.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.40.0" }
+cranelift-module = { path = "cranelift-module", version = "0.40.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.40.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.40.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.40.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.40.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 35e579892a..bd7cf33701 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.39.0"
+version = "0.40.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.39.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.40.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 9a7de70ef0..d432d5313b 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.39.0"
+version = "0.40.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.39.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.39.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.40.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.40.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -27,7 +27,7 @@ serde = { version = "1.0.94", features = ["derive"], optional = true }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.39.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.40.0", default-features = false }
 
 [features]
 default = ["std", "x86", "arm32", "arm64", "riscv"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 3f4ff42f4c..1d1fd894f9 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.39.0"
+version = "0.40.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.39.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.40.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 135726df36..538d1299df 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.39.0"
+version = "0.40.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index a6ed284c63..c93840f705 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.39.0"
+version = "0.40.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.39.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.40.0" }
 faerie = "0.10.0"
 goblin = "0.0.24"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 449e97080a..fd8b309bb5 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.39.0"
+version = "0.40.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,9 +10,9 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", features = ["testing_hooks"] }
-cranelift-reader = { path = "../cranelift-reader", version = "0.39.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.39.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", features = ["testing_hooks"] }
+cranelift-reader = { path = "../cranelift-reader", version = "0.40.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.40.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 num_cpus = "1.8.0"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 921a0dd04f..fec5f408e4 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.39.0"
+version = "0.40.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index a849fe45b6..9ed7dbe6f8 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.39.0"
+version = "0.40.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.39.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.40.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index d98466c666..58a2094174 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.39.0"
+version = "0.40.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 476f27df91..6372ff2d24 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.39.0"
+version = "0.40.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.39.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.40.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 7fcabe49ea..c178a6e060 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.39.0"
+version="0.40.0"
 
 # Update all of the Cargo.toml files.
 #
@@ -47,5 +47,5 @@ do
 
     # Sleep for a few seconds to allow the server to update the index.
     # https://internals.rust-lang.org/t/changes-to-how-crates-io-handles-index-updates/9608
-    echo sleep 3
+    echo sleep 10
 done
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 6cf4e345a2..433dc1f539 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.39.0"
+version = "0.40.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0" }
 target-lexicon = "0.4.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index f1477d641e..7b63d2be90 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.39.0"
+version = "0.40.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.39.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.40.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 51f308bc1d..7dd767f497 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.39.0"
+version = "0.40.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.39.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.39.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.40.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.40.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.39.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.39.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.39.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.40.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.40.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.40.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index ecca29146d..78cc0cde91 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.39.0"
+version = "0.40.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.39.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.40.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index b4f3b05c8a..3a90de6ccc 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.39.0"
+version = "0.40.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.36.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.39.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.39.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.39.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.40.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.40.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From d93673508c59509086edc8aad732833e93009300 Mon Sep 17 00:00:00 2001
From: Mark Bestavros 
Date: Tue, 16 Jul 2019 11:32:00 -0400
Subject: [PATCH 2586/3084] cranelift-codegen: Remove all default architecture
 support

---
 cranelift/Cargo.toml         |  2 +-
 cranelift/codegen/Cargo.toml | 11 ++++++++++-
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index a8c2addb05..eb25cb2a65 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -44,7 +44,7 @@ file-per-thread-logger = "0.1.2"
 indicatif = "0.11.0"
 
 [features]
-default = ["disas", "wasm"]
+default = ["disas", "wasm", "cranelift-codegen/all-arch"]
 disas = ["capstone"]
 wasm = ["wabt", "cranelift-wasm"]
 basic-blocks = ["cranelift-codegen/basic-blocks", "cranelift-frontend/basic-blocks", "cranelift-wasm/basic-blocks"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index d432d5313b..868db6ca3b 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -30,7 +30,7 @@ serde = { version = "1.0.94", features = ["derive"], optional = true }
 cranelift-codegen-meta = { path = "meta", version = "0.40.0", default-features = false }
 
 [features]
-default = ["std", "x86", "arm32", "arm64", "riscv"]
+default = ["std"]
 
 # The "std" feature enables use of libstd. The "core" feature enables use
 # of some minimal std-like replacement libraries. At least one of these two
@@ -54,11 +54,20 @@ core = [
 testing_hooks = []
 
 # ISA targets for which we should build.
+# If no ISA targets are explicitly enabled, the ISA target for the host machine is enabled.
 x86 = []
 arm32 = []
 arm64 = []
 riscv = []
 
+# Option to enable all architectures.
+all-arch = [
+    "x86",
+    "arm32",
+    "arm64",
+    "riscv"
+]
+
 # For dependent crates that want to serialize some parts of cranelift
 enable-serde = ["serde"]
 

From 276bb5e26db1311cf32bd284cb57635f6c0baa06 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 16 Aug 2019 11:01:19 +0200
Subject: [PATCH 2587/3084] Fixes #877: Remove appveyor configuration and
 badge;

See #877: there's an attempt to switch to Azure Pipelines.
---
 appveyor.yml        | 28 ----------------------------
 cranelift/README.md |  1 -
 2 files changed, 29 deletions(-)
 delete mode 100644 appveyor.yml

diff --git a/appveyor.yml b/appveyor.yml
deleted file mode 100644
index 5b95b731d1..0000000000
--- a/appveyor.yml
+++ /dev/null
@@ -1,28 +0,0 @@
-environment:
-  matrix:
-  - TARGET: x86_64-pc-windows-gnu
-    BITS: 64
-    MSYS2: 1
-  - TARGET: x86_64-pc-windows-msvc
-    BITS: 64
-  - TARGET: i686-pc-windows-gnu
-    BITS: 32
-    MSYS2: 1
-  - TARGET: i686-pc-windows-msvc
-    BITS: 32
-install:
-  - curl -sSf -o rustup-init.exe https://win.rustup.rs/
-  - rustup-init.exe -y --default-host %TARGET%
-  - set PATH=%PATH%;C:\Users\appveyor\.cargo\bin
-  - if defined MSYS2 set PATH=C:\msys64\mingw%BITS%\bin;%PATH%
-  - rustc -V
-  - cargo -V
-build: false
-test_script:
-  # TODO: Remove --no-default-features here. wabt and disass currently
-  # don't build on appveyor due to cmake issues.
-  - cargo build --verbose --all --no-default-features
-  - cargo test --verbose --all --no-default-features
-branches:
-  only:
-    - master
diff --git a/cranelift/README.md b/cranelift/README.md
index da1b3adb4b..419fc95a6d 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -8,7 +8,6 @@ into executable machine code.
 
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
 [![Travis Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
-[![Appveyor Status](https://ci.appveyor.com/api/projects/status/oub7wrrb59utuv8x?svg=true)](https://ci.appveyor.com/project/CraneStation/cranelift)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
 ![Minimum rustc 1.35](https://img.shields.io/badge/rustc-1.35+-green.svg)
 

From ff3c44385cc521db704ded259bf07b92c33299c3 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 21 Aug 2019 09:03:09 -0700
Subject: [PATCH 2588/3084] Add `test run` to cranelift-filetests to allow
 executing CLIF (#890)

* Add ability to run CLIF IR using `clif-util run [-v] {file}` and add `test run` to cranelift-filetests to allow executing CLIF

This re-factors the compile/execute parts to a FunctionRunner that is shared between cranelift-filetests and clif-util. CLIF can be now be run using `clif-util run` as well as during `clif-util test` for files with a `test run` header. As before, only functions suffixed with a `run` comment are executed. The `run: fn(...) == ...` expression syntax is left for a subsequent change.
---
 cranelift/Cargo.toml                          |   1 +
 cranelift/codegen/src/binemit/memorysink.rs   |  10 ++
 cranelift/codegen/src/binemit/mod.rs          |   3 +-
 cranelift/filetests/Cargo.toml                |   5 +-
 .../filetests/isa/x86/run-const.clif          |  11 ++
 cranelift/filetests/src/function_runner.rs    | 118 ++++++++++++++++++
 cranelift/filetests/src/lib.rs                |   4 +
 cranelift/filetests/src/test_run.rs           |  46 +++++++
 cranelift/src/clif-util.rs                    |  16 +++
 cranelift/src/run.rs                          | 118 ++++++++++++++++++
 10 files changed, 330 insertions(+), 2 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/run-const.clif
 create mode 100644 cranelift/filetests/src/function_runner.rs
 create mode 100644 cranelift/filetests/src/test_run.rs
 create mode 100644 cranelift/src/run.rs

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index eb25cb2a65..a8f85c30b7 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -42,6 +42,7 @@ target-lexicon = "0.4.0"
 pretty_env_logger = "0.3.0"
 file-per-thread-logger = "0.1.2"
 indicatif = "0.11.0"
+walkdir = "2.2"
 
 [features]
 default = ["disas", "wasm", "cranelift-codegen/all-arch"]
diff --git a/cranelift/codegen/src/binemit/memorysink.rs b/cranelift/codegen/src/binemit/memorysink.rs
index b543d3e89c..49c519f6b9 100644
--- a/cranelift/codegen/src/binemit/memorysink.rs
+++ b/cranelift/codegen/src/binemit/memorysink.rs
@@ -162,6 +162,16 @@ impl<'a> CodeSink for MemoryCodeSink<'a> {
     }
 }
 
+/// A `RelocSink` implementation that does nothing, which is convenient when
+/// compiling code that does not relocate anything.
+pub struct NullRelocSink {}
+
+impl RelocSink for NullRelocSink {
+    fn reloc_ebb(&mut self, _: u32, _: Reloc, _: u32) {}
+    fn reloc_external(&mut self, _: u32, _: Reloc, _: &ExternalName, _: i64) {}
+    fn reloc_jt(&mut self, _: u32, _: Reloc, _: JumpTable) {}
+}
+
 /// A `TrapSink` implementation that does nothing, which is convenient when
 /// compiling code that does not rely on trapping semantics.
 pub struct NullTrapSink {}
diff --git a/cranelift/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
index d6b72553bb..996f2e4494 100644
--- a/cranelift/codegen/src/binemit/mod.rs
+++ b/cranelift/codegen/src/binemit/mod.rs
@@ -9,7 +9,8 @@ mod shrink;
 mod stackmap;
 
 pub use self::memorysink::{
-    MemoryCodeSink, NullStackmapSink, NullTrapSink, RelocSink, StackmapSink, TrapSink,
+    MemoryCodeSink, NullRelocSink, NullStackmapSink, NullTrapSink, RelocSink, StackmapSink,
+    TrapSink,
 };
 pub use self::relaxation::relax_branches;
 pub use self::shrink::shrink_instructions;
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index fd8b309bb5..f5d3220ed9 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -11,9 +11,12 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.40.0" }
 cranelift-reader = { path = "../cranelift-reader", version = "0.40.0" }
 cranelift-preopt = { path = "../cranelift-preopt", version = "0.40.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
-num_cpus = "1.8.0"
 log = "0.4.6"
+mmap = "0.1.1"
+num_cpus = "1.8.0"
+region = "2.1.2"
diff --git a/cranelift/filetests/filetests/isa/x86/run-const.clif b/cranelift/filetests/filetests/isa/x86/run-const.clif
new file mode 100644
index 0000000000..1ac5062e49
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/run-const.clif
@@ -0,0 +1,11 @@
+test run
+
+function %test_compare_i32() -> b1 {
+ebb0:
+    v0 = iconst.i32 42
+    v1 = iconst.i32 42
+    v2 = icmp eq v0, v1
+    return v2
+}
+
+; run
diff --git a/cranelift/filetests/src/function_runner.rs b/cranelift/filetests/src/function_runner.rs
new file mode 100644
index 0000000000..dee7320384
--- /dev/null
+++ b/cranelift/filetests/src/function_runner.rs
@@ -0,0 +1,118 @@
+use core::mem;
+use cranelift_codegen::binemit::{NullRelocSink, NullStackmapSink, NullTrapSink};
+use cranelift_codegen::ir::Function;
+use cranelift_codegen::isa::{CallConv, TargetIsa};
+use cranelift_codegen::{settings, Context};
+use cranelift_native::builder as host_isa_builder;
+use mmap::{MapOption, MemoryMap};
+use region;
+use region::Protection;
+
+/// Run a function on a host
+pub struct FunctionRunner {
+    function: Function,
+    isa: Box,
+}
+
+impl FunctionRunner {
+    /// Build a function runner from a function and the ISA to run on (must be the host machine's ISA)
+    pub fn new(function: Function, isa: Box) -> Self {
+        FunctionRunner { function, isa }
+    }
+
+    /// Build a function runner using the host machine's ISA and the passed flags
+    pub fn with_host_isa(function: Function, flags: settings::Flags) -> Self {
+        let builder = host_isa_builder().expect("Unable to build a TargetIsa for the current host");
+        let isa = builder.finish(flags);
+        FunctionRunner::new(function, isa)
+    }
+
+    /// Build a function runner using the host machine's ISA and the default flags for this ISA
+    pub fn with_default_host_isa(function: Function) -> Self {
+        let flags = settings::Flags::new(settings::builder());
+        FunctionRunner::with_host_isa(function, flags)
+    }
+
+    /// Compile and execute a single function, expecting a boolean to be returned; a 'true' value is
+    /// interpreted as a successful test execution and mapped to Ok whereas a 'false' value is
+    /// interpreted as a failed test and mapped to Err.
+    pub fn run(&self) -> Result<(), String> {
+        let func = self.function.clone();
+        if !(func.signature.params.is_empty()
+            && func.signature.returns.len() == 1
+            && func.signature.returns.first().unwrap().value_type.is_bool())
+        {
+            return Err(String::from(
+                "Functions must have a signature like: () -> boolean",
+            ));
+        }
+
+        if func.signature.call_conv != self.isa.default_call_conv()
+            && func.signature.call_conv != CallConv::Fast
+        {
+            // ideally we wouldn't have to also check for Fast here but currently there is no way to inform the filetest parser that we would like to use a default other than Fast
+            return Err(String::from(
+                "Functions only run on the host's default calling convention; remove the specified calling convention in the function signature to use the host's default.",
+            ));
+        }
+
+        // set up the context
+        let mut context = Context::new();
+        context.func = func;
+
+        // compile and encode the result to machine code
+        let relocs = &mut NullRelocSink {};
+        let traps = &mut NullTrapSink {};
+        let stackmaps = &mut NullStackmapSink {};
+        let code_info = context
+            .compile(self.isa.as_ref())
+            .map_err(|e| e.to_string())?;
+        let code_page = MemoryMap::new(code_info.total_size as usize, &[MapOption::MapWritable])
+            .map_err(|e| e.to_string())?;
+        let callable_fn: fn() -> bool = unsafe {
+            context.emit_to_memory(
+                self.isa.as_ref(),
+                code_page.data(),
+                relocs,
+                traps,
+                stackmaps,
+            );
+            region::protect(code_page.data(), code_page.len(), Protection::ReadExecute)
+                .map_err(|e| e.to_string())?;
+            mem::transmute(code_page.data())
+        };
+
+        // execute
+        match callable_fn() {
+            true => Ok(()),
+            false => Err(format!("Failed: {}", context.func.name.to_string())),
+        }
+    }
+}
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use cranelift_reader::parse_test;
+
+    #[test]
+    fn nop() {
+        let code = String::from(
+            "function %test() -> b8 system_v {
+            ebb0:
+                nop
+                v1 = bconst.b8 true
+                return v1
+            }",
+        );
+
+        // extract function
+        let test_file = parse_test(code.as_str(), None, None).unwrap();
+        assert_eq!(1, test_file.functions.len());
+        let function = test_file.functions[0].0.clone();
+
+        // execute function
+        let runner = FunctionRunner::with_default_host_isa(function);
+        runner.run().unwrap() // will panic if execution fails
+    }
+}
diff --git a/cranelift/filetests/src/lib.rs b/cranelift/filetests/src/lib.rs
index 6099ce3726..08f02afbf5 100644
--- a/cranelift/filetests/src/lib.rs
+++ b/cranelift/filetests/src/lib.rs
@@ -23,6 +23,7 @@
     )
 )]
 
+pub use crate::function_runner::FunctionRunner;
 use crate::runner::TestRunner;
 use cranelift_codegen::timing;
 use cranelift_reader::TestCommand;
@@ -30,6 +31,7 @@ use std::path::Path;
 use std::time;
 
 mod concurrent;
+mod function_runner;
 mod match_directive;
 mod runner;
 mod runone;
@@ -46,6 +48,7 @@ mod test_postopt;
 mod test_preopt;
 mod test_print_cfg;
 mod test_regalloc;
+mod test_run;
 mod test_safepoint;
 mod test_shrink;
 mod test_simple_gvn;
@@ -124,6 +127,7 @@ fn new_subtest(parsed: &TestCommand) -> subtest::SubtestResult test_simple_preopt::subtest(parsed),
         "print-cfg" => test_print_cfg::subtest(parsed),
         "regalloc" => test_regalloc::subtest(parsed),
+        "run" => test_run::subtest(parsed),
         "shrink" => test_shrink::subtest(parsed),
         "simple-gvn" => test_simple_gvn::subtest(parsed),
         "verifier" => test_verifier::subtest(parsed),
diff --git a/cranelift/filetests/src/test_run.rs b/cranelift/filetests/src/test_run.rs
new file mode 100644
index 0000000000..6e34bfebfa
--- /dev/null
+++ b/cranelift/filetests/src/test_run.rs
@@ -0,0 +1,46 @@
+//! Test command for running CLIF files and verifying their results
+//!
+//! The `run` test command compiles each function on the host machine and executes it
+
+use crate::function_runner::FunctionRunner;
+use crate::subtest::{Context, SubTest, SubtestResult};
+use cranelift_codegen;
+use cranelift_codegen::ir;
+use cranelift_reader::TestCommand;
+use std::borrow::Cow;
+
+struct TestRun;
+
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+    assert_eq!(parsed.command, "run");
+    if !parsed.options.is_empty() {
+        Err(format!("No options allowed on {}", parsed))
+    } else {
+        Ok(Box::new(TestRun))
+    }
+}
+
+impl SubTest for TestRun {
+    fn name(&self) -> &'static str {
+        "run"
+    }
+
+    fn is_mutating(&self) -> bool {
+        false
+    }
+
+    fn needs_isa(&self) -> bool {
+        false
+    }
+
+    fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> {
+        for comment in context.details.comments.iter() {
+            if comment.text.contains("run") {
+                let runner =
+                    FunctionRunner::with_host_isa(func.clone().into_owned(), context.flags.clone());
+                runner.run()?
+            }
+        }
+        Ok(())
+    }
+}
diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index a182889059..51bac2a533 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -33,6 +33,7 @@ mod cat;
 mod compile;
 mod disasm;
 mod print_cfg;
+mod run;
 mod utils;
 
 /// A command either succeeds or fails with an error message.
@@ -162,6 +163,13 @@ fn main() {
                 .arg(add_input_file_arg())
                 .arg(add_debug_flag()),
         )
+        .subcommand(
+            SubCommand::with_name("run")
+                .about("Execute CLIF code and verify with test expressions")
+                .arg(add_verbose_flag())
+                .arg(add_input_file_arg())
+                .arg(add_debug_flag()),
+        )
         .subcommand(
             SubCommand::with_name("cat")
                 .about("Outputs .clif file")
@@ -224,6 +232,14 @@ fn main() {
             )
             .map(|_time| ())
         }
+        ("run", Some(rest_cmd)) => {
+            handle_debug_flag(rest_cmd.is_present("debug"));
+            run::run(
+                get_vec(rest_cmd.values_of("file")),
+                rest_cmd.is_present("verbose"),
+            )
+            .map(|_time| ())
+        }
         ("pass", Some(rest_cmd)) => {
             handle_debug_flag(rest_cmd.is_present("debug"));
 
diff --git a/cranelift/src/run.rs b/cranelift/src/run.rs
new file mode 100644
index 0000000000..a6c460083f
--- /dev/null
+++ b/cranelift/src/run.rs
@@ -0,0 +1,118 @@
+//! CLI tool to compile Cranelift IR files to native code in memory and execute them.
+
+use crate::utils::read_to_string;
+use cranelift_codegen::isa::TargetIsa;
+use cranelift_filetests::FunctionRunner;
+use cranelift_native::builder as host_isa_builder;
+use cranelift_reader::{parse_test, Details, IsaSpec};
+use std::path::PathBuf;
+use walkdir::WalkDir;
+
+pub fn run(files: Vec, flag_print: bool) -> Result<(), String> {
+    let mut total = 0;
+    let mut errors = 0;
+    for file in iterate_files(files) {
+        total += 1;
+        match run_single_file(&file) {
+            Ok(_) => {
+                if flag_print {
+                    println!("{}", file.to_string_lossy());
+                }
+            }
+            Err(e) => {
+                if flag_print {
+                    println!("{}: {}", file.to_string_lossy(), e);
+                }
+                errors += 1;
+            }
+        }
+    }
+
+    if flag_print {
+        match total {
+            0 => println!("0 files"),
+            1 => println!("1 file"),
+            n => println!("{} files", n),
+        }
+    }
+
+    match errors {
+        0 => Ok(()),
+        1 => Err(String::from("1 failure")),
+        n => Err(format!("{} failures", n)),
+    }
+}
+
+/// Iterate over all of the files passed as arguments, recursively iterating through directories
+fn iterate_files(files: Vec) -> impl Iterator {
+    files
+        .into_iter()
+        .flat_map(WalkDir::new)
+        .filter(|f| match f {
+            Ok(d) => {
+                // filter out hidden files (starting with .)
+                !d.file_name().to_str().map_or(false, |s| s.starts_with("."))
+                    // filter out directories
+                    && !d.file_type().is_dir()
+            }
+            Err(e) => {
+                println!("Unable to read file: {}", e);
+                false
+            }
+        })
+        .map(|f| {
+            f.expect("This should not happen: we have already filtered out the errors")
+                .into_path()
+        })
+}
+
+/// Run all functions in a file that are succeeded by "run:" comments
+fn run_single_file(path: &PathBuf) -> Result<(), String> {
+    let file_contents = read_to_string(&path).map_err(|e| e.to_string())?;
+    run_file_contents(file_contents)
+}
+
+/// Main body of `run_single_file` separated for testing
+fn run_file_contents(file_contents: String) -> Result<(), String> {
+    let test_file = parse_test(&file_contents, None, None).map_err(|e| e.to_string())?;
+    for (func, Details { comments, .. }) in test_file.functions {
+        if comments.iter().any(|c| c.text.contains("run")) {
+            let isa = create_target_isa(&test_file.isa_spec)?;
+            FunctionRunner::new(func, isa).run()?
+        }
+    }
+    Ok(())
+}
+
+/// Build an ISA based on the current machine running this code (the host)
+fn create_target_isa(isa_spec: &IsaSpec) -> Result, String> {
+    if let IsaSpec::None(flags) = isa_spec {
+        // build an ISA for the current machine
+        let builder = host_isa_builder()?;
+        Ok(builder.finish(flags.clone()))
+    } else {
+        Err(String::from("A target ISA was specified in the file but should not have been--only the host ISA can be used for running CLIF files"))?
+    }
+}
+
+#[cfg(test)]
+mod test {
+    use super::*;
+
+    #[test]
+    fn nop() {
+        let code = String::from(
+            "
+            function %test() -> b8 system_v {
+            ebb0:
+                nop
+                v1 = bconst.b8 true
+                return v1
+            }
+            
+            ; run
+            ",
+        );
+        run_file_contents(code).unwrap()
+    }
+}

From 97996d79cece7a879c0591380eae3490a7ba87d7 Mon Sep 17 00:00:00 2001
From: Mike Hommey 
Date: Thu, 22 Aug 2019 17:32:39 +0900
Subject: [PATCH 2589/3084] Sort custom_legalizes for more deterministic
 generated code

For better caching.
---
 cranelift/codegen/meta/src/gen_legalizer.rs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index b823bb5597..7b59844e60 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -466,7 +466,9 @@ fn gen_transform_group<'a>(
 
                 // Emit the custom transforms. The Rust compiler will complain about any overlap with
                 // the normal transforms.
-                for (inst_camel_name, func_name) in &group.custom_legalizes {
+                let mut sorted_custom_legalizes = Vec::from_iter(&group.custom_legalizes);
+                sorted_custom_legalizes.sort();
+                for (inst_camel_name, func_name) in sorted_custom_legalizes {
                     fmtln!(fmt, "ir::Opcode::{} => {{", inst_camel_name);
                     fmt.indent(|fmt| {
                         fmtln!(fmt, "{}(inst, pos.func, cfg, isa);", func_name);

From e736367b8cbe5e2c060aa20a5d7e72d604659ff4 Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Tue, 20 Aug 2019 13:18:54 -0600
Subject: [PATCH 2590/3084] Make fold_redundant_jumps() feature-gated on
 basic-blocks

---
 cranelift/codegen/src/binemit/relaxation.rs | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index e5f7772210..262ef986d0 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -31,7 +31,7 @@ use crate::binemit::{CodeInfo, CodeOffset};
 use crate::cursor::{Cursor, FuncCursor};
 use crate::dominator_tree::DominatorTree;
 use crate::flowgraph::ControlFlowGraph;
-use crate::ir::{Ebb, Function, Inst, InstructionData, Opcode, Value, ValueList};
+use crate::ir::{Function, InstructionData, Opcode};
 use crate::isa::{EncInfo, TargetIsa};
 use crate::iterators::IteratorExtras;
 use crate::regalloc::RegDiversions;
@@ -39,13 +39,16 @@ use crate::timing;
 use crate::CodegenResult;
 use log::debug;
 
+#[cfg(feature = "basic-blocks")]
+use crate::ir::{Ebb, Inst, Value, ValueList};
+
 /// Relax branches and compute the final layout of EBB headers in `func`.
 ///
 /// Fill in the `func.offsets` table so the function is ready for binary emission.
 pub fn relax_branches(
     func: &mut Function,
-    cfg: &mut ControlFlowGraph,
-    domtree: &mut DominatorTree,
+    _cfg: &mut ControlFlowGraph,
+    _domtree: &mut DominatorTree,
     isa: &dyn TargetIsa,
 ) -> CodegenResult {
     let _tt = timing::relax_branches();
@@ -57,8 +60,8 @@ pub fn relax_branches(
     func.offsets.resize(func.dfg.num_ebbs());
 
     // Start by removing redundant jumps.
-    // FIXME: Temporarily disabled due to #916.
-    /* fold_redundant_jumps(func, cfg, domtree); */
+    #[cfg(feature = "basic-blocks")]
+    fold_redundant_jumps(func, _cfg, _domtree);
 
     // Convert jumps to fallthrough instructions where possible.
     fallthroughs(func);
@@ -146,6 +149,7 @@ pub fn relax_branches(
 
 /// Folds an instruction if it is a redundant jump.
 /// Returns whether folding was performed (which invalidates the CFG).
+#[cfg(feature = "basic-blocks")]
 fn try_fold_redundant_jump(
     func: &mut Function,
     cfg: &mut ControlFlowGraph,
@@ -243,6 +247,7 @@ fn try_fold_redundant_jump(
 
 /// Redirects `jump` instructions that point to other `jump` instructions to the final destination.
 /// This transformation may orphan some blocks.
+#[cfg(feature = "basic-blocks")]
 fn fold_redundant_jumps(
     func: &mut Function,
     cfg: &mut ControlFlowGraph,

From d3815a0399a83a9c649a843e85485d0f979b6305 Mon Sep 17 00:00:00 2001
From: Artur Jamro 
Date: Thu, 22 Aug 2019 10:11:41 -0700
Subject: [PATCH 2591/3084] Implement serde and equality traits for
 SecondaryMap

---
 cranelift/codegen/src/binemit/mod.rs  |   2 +-
 cranelift/codegen/src/ir/stackslot.rs |   4 +-
 cranelift/codegen/src/value_label.rs  |   2 +-
 cranelift/entity/src/map.rs           | 119 +++++++++++++++++++++++---
 cranelift/entity/src/primary.rs       |   2 +-
 5 files changed, 112 insertions(+), 17 deletions(-)

diff --git a/cranelift/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
index 996f2e4494..ce9321ec19 100644
--- a/cranelift/codegen/src/binemit/mod.rs
+++ b/cranelift/codegen/src/binemit/mod.rs
@@ -33,7 +33,7 @@ pub type CodeOffset = u32;
 pub type Addend = i64;
 
 /// Relocation kinds for every ISA
-#[derive(Copy, Clone, Debug)]
+#[derive(Copy, Clone, Debug, PartialEq, Eq)]
 #[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub enum Reloc {
     /// absolute 4-byte
diff --git a/cranelift/codegen/src/ir/stackslot.rs b/cranelift/codegen/src/ir/stackslot.rs
index 0101b54f2e..0e1ec4777a 100644
--- a/cranelift/codegen/src/ir/stackslot.rs
+++ b/cranelift/codegen/src/ir/stackslot.rs
@@ -101,7 +101,7 @@ impl fmt::Display for StackSlotKind {
 }
 
 /// Contents of a stack slot.
-#[derive(Clone, Debug)]
+#[derive(Clone, Debug, PartialEq, Eq)]
 #[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct StackSlotData {
     /// The kind of stack slot.
@@ -154,7 +154,7 @@ impl fmt::Display for StackSlotData {
 /// Stack frame manager.
 ///
 /// Keep track of all the stack slots used by a function.
-#[derive(Clone, Debug)]
+#[derive(Clone, Debug, PartialEq, Eq)]
 #[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct StackSlots {
     /// All allocated stack slots.
diff --git a/cranelift/codegen/src/value_label.rs b/cranelift/codegen/src/value_label.rs
index c31067cc24..60ea7b342a 100644
--- a/cranelift/codegen/src/value_label.rs
+++ b/cranelift/codegen/src/value_label.rs
@@ -12,7 +12,7 @@ use std::vec::Vec;
 use serde::{Deserialize, Serialize};
 
 /// Value location range.
-#[derive(Debug, Clone, Copy)]
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
 #[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct ValueLocRange {
     /// The ValueLoc containing a ValueLabel during this range.
diff --git a/cranelift/entity/src/map.rs b/cranelift/entity/src/map.rs
index a0d31a4309..bb1b94aeca 100644
--- a/cranelift/entity/src/map.rs
+++ b/cranelift/entity/src/map.rs
@@ -6,6 +6,13 @@ use crate::EntityRef;
 use core::marker::PhantomData;
 use core::ops::{Index, IndexMut};
 use core::slice;
+#[cfg(feature = "enable-serde")]
+use serde::{
+    de::{Deserializer, SeqAccess, Visitor},
+    ser::{SerializeSeq, Serializer},
+    Deserialize, Serialize,
+};
+use std::cmp::min;
 use std::vec::Vec;
 
 /// A mapping `K -> V` for densely indexed entity references.
@@ -56,23 +63,11 @@ where
         }
     }
 
-    /// Returns the number of elements in the underlying vector.
-    ///
-    /// The number is not necessarily the same as the length of the corresponding PrimaryMap.
-    pub fn len(&self) -> usize {
-        self.elems.len()
-    }
-
     /// Get the element at `k` if it exists.
     pub fn get(&self, k: K) -> Option<&V> {
         self.elems.get(k.index())
     }
 
-    /// Get the default value.
-    pub fn get_default(&self) -> &V {
-        &self.default
-    }
-
     /// Is this map completely empty?
     pub fn is_empty(&self) -> bool {
         self.elems.is_empty()
@@ -148,6 +143,106 @@ where
     }
 }
 
+impl PartialEq for SecondaryMap
+where
+    K: EntityRef,
+    V: Clone + PartialEq,
+{
+    fn eq(&self, other: &Self) -> bool {
+        let min_size = min(self.elems.len(), other.elems.len());
+        self.default == other.default
+            && self.elems[..min_size] == other.elems[..min_size]
+            && self.elems[min_size..].iter().all(|e| *e == self.default)
+            && other.elems[min_size..].iter().all(|e| *e == other.default)
+    }
+}
+
+impl Eq for SecondaryMap
+where
+    K: EntityRef,
+    V: Clone + PartialEq + Eq,
+{
+}
+
+#[cfg(feature = "enable-serde")]
+impl Serialize for SecondaryMap
+where
+    K: EntityRef,
+    V: Clone + PartialEq + Serialize,
+{
+    fn serialize(&self, serializer: S) -> Result
+    where
+        S: Serializer,
+    {
+        // TODO: bincode encodes option as "byte for Some/None" and then optionally the content
+        // TODO: we can actually optimize it by encoding manually bitmask, then elements
+        let mut elems_cnt = self.elems.len();
+        while elems_cnt > 0 && self.elems[elems_cnt - 1] == self.default {
+            elems_cnt -= 1;
+        }
+        let mut seq = serializer.serialize_seq(Some(1 + elems_cnt))?;
+        seq.serialize_element(&Some(self.default.clone()))?;
+        for e in self.elems.iter().take(elems_cnt) {
+            let some_e = Some(e);
+            seq.serialize_element(if *e == self.default { &None } else { &some_e })?;
+        }
+        seq.end()
+    }
+}
+
+#[cfg(feature = "enable-serde")]
+impl<'de, K, V> Deserialize<'de> for SecondaryMap
+where
+    K: EntityRef,
+    V: Clone + Deserialize<'de>,
+{
+    fn deserialize(deserializer: D) -> Result
+    where
+        D: Deserializer<'de>,
+    {
+        use std::fmt;
+        struct SecondaryMapVisitor {
+            unused: PhantomData V>,
+        }
+
+        impl<'de, K, V> Visitor<'de> for SecondaryMapVisitor
+        where
+            K: EntityRef,
+            V: Clone + Deserialize<'de>,
+        {
+            type Value = SecondaryMap;
+
+            fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
+                formatter.write_str("struct SecondaryMap")
+            }
+
+            fn visit_seq(self, mut seq: A) -> Result
+            where
+                A: SeqAccess<'de>,
+            {
+                match seq.next_element()? {
+                    Some(Some(default_val)) => {
+                        let default_val: V = default_val; // compiler can't infer the type
+                        let mut m = SecondaryMap::with_default(default_val.clone());
+                        let mut idx = 0;
+                        while let Some(val) = seq.next_element()? {
+                            let val: Option<_> = val; // compiler can't infer the type
+                            m[K::new(idx)] = val.unwrap_or_else(|| default_val.clone());
+                            idx += 1;
+                        }
+                        Ok(m)
+                    }
+                    _ => Err(serde::de::Error::custom("Default value required")),
+                }
+            }
+        }
+
+        deserializer.deserialize_seq(SecondaryMapVisitor {
+            unused: PhantomData {},
+        })
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
diff --git a/cranelift/entity/src/primary.rs b/cranelift/entity/src/primary.rs
index 6734a1f5e5..9cde5e779d 100644
--- a/cranelift/entity/src/primary.rs
+++ b/cranelift/entity/src/primary.rs
@@ -27,7 +27,7 @@ use std::vec::Vec;
 /// that it only allows indexing with the distinct `EntityRef` key type, so converting to a
 /// plain slice would make it easier to use incorrectly. To make a slice of a `PrimaryMap`, use
 /// `into_boxed_slice`.
-#[derive(Debug, Clone, Hash)]
+#[derive(Debug, Clone, Hash, PartialEq, Eq)]
 #[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct PrimaryMap
 where

From 18ecf60ad55356667efd7b40fded10f2823fd2e5 Mon Sep 17 00:00:00 2001
From: Yaron Wittenstein 
Date: Thu, 22 Aug 2019 15:32:01 +0300
Subject: [PATCH 2592/3084] cranelift-wasm: bump `wasmparser` version to
 `0.37.0`

---
 cranelift/wasm/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 3a90de6ccc..5beb48fdbf 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -11,7 +11,7 @@ keywords = ["webassembly", "wasm"]
 edition = "2018"
 
 [dependencies]
-wasmparser = { version = "0.36.0", default-features = false }
+wasmparser = { version = "0.37.0", default-features = false }
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.40.0", default-features = false }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.40.0", default-features = false }

From 1eb6cd93b20f62664d3e593a2c5f32be2c3856f9 Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Fri, 23 Aug 2019 09:00:04 -0400
Subject: [PATCH 2593/3084] Fix documentation typo

function -> data object
---
 cranelift/module/src/module.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 33a13b7f9a..d9cb1e00c5 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -559,7 +559,7 @@ where
         Ok(total_size)
     }
 
-    /// Define a function, producing the data contents from the given `DataContext`.
+    /// Define a data object, producing the data contents from the given `DataContext`.
     pub fn define_data(&mut self, data: DataId, data_ctx: &DataContext) -> ModuleResult<()> {
         let compiled = {
             let info = &self.contents.data_objects[data];

From cc57e84cbd4b76936b1daac52a28d5eccfb3bb2d Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 23 Aug 2019 09:04:34 -0700
Subject: [PATCH 2594/3084] Fix segfault due to b64 encoding (#919)

* Fix segfault due to b64 encoding

Prior to this patch, bconst.b64 encoded its instruction with a 32-bit immediate that caused improper decoding of the MOV instruction; instead, use a REX prefix and rely on zero-extension of the immediate. Fixes #911.
---
 cranelift/codegen/meta/src/isa/x86/encodings.rs        |  5 +----
 .../filetests/filetests/isa/x86/binary64-run.clif      | 10 ++++++++++
 cranelift/filetests/filetests/isa/x86/binary64.clif    |  7 +++++++
 3 files changed, 18 insertions(+), 4 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/binary64-run.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index c5bbfe1a83..a71b2cc68f 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -683,10 +683,7 @@ pub fn define(
     for &ty in &[B1, B8, B16, B32] {
         e.enc_both(bconst.bind(ty), rec_pu_id_bool.opcodes(vec![0xb8]));
     }
-    e.enc64(
-        bconst.bind(B64),
-        rec_pu_id_bool.opcodes(vec![0xb8]).rex().w(),
-    );
+    e.enc64(bconst.bind(B64), rec_pu_id_bool.opcodes(vec![0xb8]).rex());
 
     // Shifts and rotates.
     // Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit
diff --git a/cranelift/filetests/filetests/isa/x86/binary64-run.clif b/cranelift/filetests/filetests/isa/x86/binary64-run.clif
new file mode 100644
index 0000000000..b255770c1e
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/binary64-run.clif
@@ -0,0 +1,10 @@
+test run
+target x86_64
+
+; this verifies that returning b64 immediates does not result in a segmentation fault, see https://github.com/CraneStation/cranelift/issues/911
+function %test_b64() -> b64 {
+ebb0:
+[-, %r10]   v0 = bconst.b64 true
+            return v0
+}
+; run
diff --git a/cranelift/filetests/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
index a65b3d3d1d..7742a24ee3 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64.clif
@@ -1642,3 +1642,10 @@ ebb0:
 
     return
 }
+
+function %B64() {
+ebb0:
+    [-, %rax]   v1 = bconst.b64 true    ; bin: 40 b8 00000001
+    [-, %r10]   v0 = bconst.b64 true    ; bin: 41 ba 00000001
+    return
+}

From b8fb52446c85f73cff0afa5c1bcc7e650af724b9 Mon Sep 17 00:00:00 2001
From: julian-seward1 
Date: Sun, 25 Aug 2019 19:37:34 +0200
Subject: [PATCH 2595/3084] Cranelift: implement redundant fill removal on
 tree-shaped CFG regions. Mozilla bug 1570584. (#906)

---
 .../codegen/meta/src/isa/riscv/encodings.rs   |  42 +
 .../codegen/meta/src/isa/riscv/recipes.rs     |  18 +
 .../codegen/meta/src/isa/x86/encodings.rs     |  36 +
 cranelift/codegen/meta/src/isa/x86/recipes.rs |  45 +
 cranelift/codegen/meta/src/shared/formats.rs  |   1 +
 .../codegen/meta/src/shared/instructions.rs   |  33 +
 cranelift/codegen/src/context.rs              |  15 +
 cranelift/codegen/src/ir/stackslot.rs         |   5 -
 cranelift/codegen/src/lib.rs                  |   1 +
 .../codegen/src/redundant_reload_remover.rs   | 904 ++++++++++++++++++
 cranelift/codegen/src/regalloc/coloring.rs    |  29 +-
 .../codegen/src/regalloc/register_set.rs      |  65 ++
 cranelift/codegen/src/regalloc/solver.rs      |  51 +-
 cranelift/codegen/src/verifier/mod.rs         |   1 +
 cranelift/codegen/src/write.rs                |   8 +
 cranelift/entity/src/set.rs                   |  18 +
 .../filetests/filetests/safepoint/call.clif   |   2 +
 cranelift/reader/src/parser.rs                |   4 +
 cranelift/serde/src/serde_clif_json.rs        |   8 +
 19 files changed, 1262 insertions(+), 24 deletions(-)
 create mode 100644 cranelift/codegen/src/redundant_reload_remover.rs

diff --git a/cranelift/codegen/meta/src/isa/riscv/encodings.rs b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
index 88a62ac74e..a5af1214aa 100644
--- a/cranelift/codegen/meta/src/isa/riscv/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
@@ -9,6 +9,7 @@ use crate::cdsl::settings::SettingGroup;
 use crate::shared::types::Bool::B1;
 use crate::shared::types::Float::{F32, F64};
 use crate::shared::types::Int::{I16, I32, I64, I8};
+use crate::shared::types::Reference::{R32, R64};
 use crate::shared::Definitions as SharedDefinitions;
 
 use super::recipes::RecipeGroup;
@@ -121,7 +122,9 @@ pub fn define<'defs>(
     let call_indirect = shared.by_name("call_indirect");
     let copy = shared.by_name("copy");
     let copy_nop = shared.by_name("copy_nop");
+    let copy_to_ssa = shared.by_name("copy_to_ssa");
     let fill = shared.by_name("fill");
+    let fill_nop = shared.by_name("fill_nop");
     let iadd = shared.by_name("iadd");
     let iadd_imm = shared.by_name("iadd_imm");
     let iconst = shared.by_name("iconst");
@@ -141,6 +144,8 @@ pub fn define<'defs>(
     let return_ = shared.by_name("return");
 
     // Recipes shorthands, prefixed with r_.
+    let r_copytossa = recipes.by_name("copytossa");
+    let r_fillnull = recipes.by_name("fillnull");
     let r_icall = recipes.by_name("Icall");
     let r_icopy = recipes.by_name("Icopy");
     let r_ii = recipes.by_name("Ii");
@@ -368,6 +373,14 @@ pub fn define<'defs>(
     e.add64(enc(fill.bind(I32), r_gp_fi, load_bits(0b010)));
     e.add64(enc(fill.bind(I64), r_gp_fi, load_bits(0b011)));
 
+    // No-op fills, created by late-stage redundant-fill removal.
+    for &ty in &[I64, I32] {
+        e.add64(enc(fill_nop.bind(ty), r_fillnull, 0));
+        e.add32(enc(fill_nop.bind(ty), r_fillnull, 0));
+    }
+    e.add64(enc(fill_nop.bind(B1), r_fillnull, 0));
+    e.add32(enc(fill_nop.bind(B1), r_fillnull, 0));
+
     // Register copies.
     e.add32(enc(copy.bind(I32), r_icopy, opimm_bits(0b000, 0)));
     e.add64(enc(copy.bind(I64), r_icopy, opimm_bits(0b000, 0)));
@@ -394,5 +407,34 @@ pub fn define<'defs>(
         e.add64(enc(copy_nop.bind(ty), r_stacknull, 0));
     }
 
+    // Copy-to-SSA
+    e.add32(enc(
+        copy_to_ssa.bind(I32),
+        r_copytossa,
+        opimm_bits(0b000, 0),
+    ));
+    e.add64(enc(
+        copy_to_ssa.bind(I64),
+        r_copytossa,
+        opimm_bits(0b000, 0),
+    ));
+    e.add64(enc(
+        copy_to_ssa.bind(I32),
+        r_copytossa,
+        opimm32_bits(0b000, 0),
+    ));
+    e.add32(enc(copy_to_ssa.bind(B1), r_copytossa, opimm_bits(0b000, 0)));
+    e.add64(enc(copy_to_ssa.bind(B1), r_copytossa, opimm_bits(0b000, 0)));
+    e.add32(enc(
+        copy_to_ssa.bind_ref(R32),
+        r_copytossa,
+        opimm_bits(0b000, 0),
+    ));
+    e.add64(enc(
+        copy_to_ssa.bind_ref(R64),
+        r_copytossa,
+        opimm_bits(0b000, 0),
+    ));
+
     e
 }
diff --git a/cranelift/codegen/meta/src/isa/riscv/recipes.rs b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
index 67146ce17d..40396b4ab3 100644
--- a/cranelift/codegen/meta/src/isa/riscv/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
@@ -63,6 +63,7 @@ pub fn define<'formats>(
     let f_branch_icmp = formats.by_name("BranchIcmp");
     let f_call = formats.by_name("Call");
     let f_call_indirect = formats.by_name("CallIndirect");
+    let f_copy_to_ssa = formats.by_name("CopyToSsa");
     let f_int_compare = formats.by_name("IntCompare");
     let f_int_compare_imm = formats.by_name("IntCompareImm");
     let f_jump = formats.by_name("Jump");
@@ -185,6 +186,14 @@ pub fn define<'formats>(
             .emit("put_i(bits, src, 0, dst, sink);"),
     );
 
+    // Same for copy-to-SSA -- GPR regmove.
+    recipes.push(
+        EncodingRecipeBuilder::new("copytossa", f_copy_to_ssa, 4)
+            // No operands_in to mention, because a source register is specified directly.
+            .operands_out(vec![gpr])
+            .emit("put_i(bits, src, 0, out_reg0, sink);"),
+    );
+
     // U-type instructions have a 20-bit immediate that targets bits 12-31.
     let format = formats.get(f_unary_imm);
     recipes.push(
@@ -271,5 +280,14 @@ pub fn define<'formats>(
             .emit(""),
     );
 
+    // No-op fills, created by late-stage redundant-fill removal.
+    recipes.push(
+        EncodingRecipeBuilder::new("fillnull", f_unary, 0)
+            .operands_in(vec![Stack::new(gpr)])
+            .operands_out(vec![gpr])
+            .clobbers_flags(false)
+            .emit(""),
+    );
+
     recipes
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index a71b2cc68f..818e07bd0b 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -340,6 +340,7 @@ pub fn define(
     let copy = shared.by_name("copy");
     let copy_nop = shared.by_name("copy_nop");
     let copy_special = shared.by_name("copy_special");
+    let copy_to_ssa = shared.by_name("copy_to_ssa");
     let ctz = shared.by_name("ctz");
     let debugtrap = shared.by_name("debugtrap");
     let extractlane = shared.by_name("extractlane");
@@ -352,6 +353,7 @@ pub fn define(
     let fdiv = shared.by_name("fdiv");
     let ffcmp = shared.by_name("ffcmp");
     let fill = shared.by_name("fill");
+    let fill_nop = shared.by_name("fill_nop");
     let floor = shared.by_name("floor");
     let fmul = shared.by_name("fmul");
     let fpromote = shared.by_name("fpromote");
@@ -468,7 +470,9 @@ pub fn define(
     let rec_fax = r.template("fax");
     let rec_fcmp = r.template("fcmp");
     let rec_fcscc = r.template("fcscc");
+    let rec_ffillnull = r.recipe("ffillnull");
     let rec_ffillSib32 = r.template("ffillSib32");
+    let rec_fillnull = r.recipe("fillnull");
     let rec_fillSib32 = r.template("fillSib32");
     let rec_fld = r.template("fld");
     let rec_fldDisp32 = r.template("fldDisp32");
@@ -490,6 +494,7 @@ pub fn define(
     let rec_fstWithIndexDisp32 = r.template("fstWithIndexDisp32");
     let rec_fstWithIndexDisp8 = r.template("fstWithIndexDisp8");
     let rec_furm = r.template("furm");
+    let rec_furm_reg_to_ssa = r.template("furm_reg_to_ssa");
     let rec_furmi_rnd = r.template("furmi_rnd");
     let rec_got_fnaddr8 = r.template("got_fnaddr8");
     let rec_got_gvaddr8 = r.template("got_gvaddr8");
@@ -568,6 +573,7 @@ pub fn define(
     let rec_trapff = r.recipe("trapff");
     let rec_u_id = r.template("u_id");
     let rec_umr = r.template("umr");
+    let rec_umr_reg_to_ssa = r.template("umr_reg_to_ssa");
     let rec_ur = r.template("ur");
     let rec_urm = r.template("urm");
     let rec_urm_noflags = r.template("urm_noflags");
@@ -921,6 +927,18 @@ pub fn define(
     e.enc_r32_r64(fill, rec_fillSib32.opcodes(vec![0x8b]));
     e.enc_r32_r64(regfill, rec_regfill32.opcodes(vec![0x8b]));
 
+    // No-op fills, created by late-stage redundant-fill removal.
+    for &ty in &[I64, I32, I16, I8] {
+        e.enc64_rec(fill_nop.bind(ty), rec_fillnull, 0);
+        e.enc32_rec(fill_nop.bind(ty), rec_fillnull, 0);
+    }
+    e.enc64_rec(fill_nop.bind(B1), rec_fillnull, 0);
+    e.enc32_rec(fill_nop.bind(B1), rec_fillnull, 0);
+    for &ty in &[F64, F32] {
+        e.enc64_rec(fill_nop.bind(ty), rec_ffillnull, 0);
+        e.enc32_rec(fill_nop.bind(ty), rec_ffillnull, 0);
+    }
+
     // Load 32 bits from `b1`, `i8` and `i16` spill slots. See `spill.b1` above.
 
     e.enc_both(fill.bind(B1), rec_fillSib32.opcodes(vec![0x8b]));
@@ -943,6 +961,24 @@ pub fn define(
     e.enc64(copy_special, rec_copysp.opcodes(vec![0x89]).rex().w());
     e.enc32(copy_special, rec_copysp.opcodes(vec![0x89]));
 
+    // Copy to SSA
+    e.enc_i32_i64(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(vec![0x89]));
+    e.enc_r32_r64(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(vec![0x89]));
+    e.enc_both(copy_to_ssa.bind(B1), rec_umr_reg_to_ssa.opcodes(vec![0x89]));
+    e.enc_both(copy_to_ssa.bind(I8), rec_umr_reg_to_ssa.opcodes(vec![0x89]));
+    e.enc_both(
+        copy_to_ssa.bind(I16),
+        rec_umr_reg_to_ssa.opcodes(vec![0x89]),
+    );
+    e.enc_both(
+        copy_to_ssa.bind(F64),
+        rec_furm_reg_to_ssa.opcodes(vec![0xf2, 0x0f, 0x10]),
+    );
+    e.enc_both(
+        copy_to_ssa.bind(F32),
+        rec_furm_reg_to_ssa.opcodes(vec![0xf3, 0x0f, 0x10]),
+    );
+
     // Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
     // into a no-op.
     // The same encoding is generated for both the 64- and 32-bit architectures.
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 3e773bc34c..50ac9fca7f 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -367,6 +367,7 @@ pub fn define<'shared>(
     let f_call = formats.by_name("Call");
     let f_call_indirect = formats.by_name("CallIndirect");
     let f_copy_special = formats.by_name("CopySpecial");
+    let f_copy_to_ssa = formats.by_name("CopyToSsa");
     let f_extract_lane = formats.by_name("ExtractLane"); // TODO this would preferably retrieve a BinaryImm8 format but because formats are compared structurally and ExtractLane has the same structure this is impossible--if we rename ExtractLane, it may even impact parsing
     let f_float_compare = formats.by_name("FloatCompare");
     let f_float_cond = formats.by_name("FloatCond");
@@ -426,6 +427,22 @@ pub fn define<'shared>(
             .emit(""),
     );
 
+    // No-op fills, created by late-stage redundant-fill removal.
+    recipes.add_recipe(
+        EncodingRecipeBuilder::new("fillnull", f_unary, 0)
+            .operands_in(vec![stack_gpr32])
+            .operands_out(vec![gpr])
+            .clobbers_flags(false)
+            .emit(""),
+    );
+    recipes.add_recipe(
+        EncodingRecipeBuilder::new("ffillnull", f_unary, 0)
+            .operands_in(vec![stack_gpr32])
+            .operands_out(vec![fpr])
+            .clobbers_flags(false)
+            .emit(""),
+    );
+
     recipes
         .add_recipe(EncodingRecipeBuilder::new("debugtrap", f_nullary, 1).emit("sink.put1(0xcc);"));
 
@@ -570,6 +587,20 @@ pub fn define<'shared>(
             ),
     );
 
+    // Same as umr, but with the source register specified directly.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("umr_reg_to_ssa", f_copy_to_ssa, 1)
+            // No operands_in to mention, because a source register is specified directly.
+            .operands_out(vec![gpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(out_reg0, src), sink);
+                    modrm_rr(out_reg0, src, sink);
+                "#,
+            ),
+    );
+
     // XX /r, but for a unary operator with separate input/output register.
     // RM form. Clobbers FLAGS.
     recipes.add_template_recipe(
@@ -631,6 +662,20 @@ pub fn define<'shared>(
             ),
     );
 
+    // Same as furm, but with the source register specified directly.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("furm_reg_to_ssa", f_copy_to_ssa, 1)
+            // No operands_in to mention, because a source register is specified directly.
+            .operands_out(vec![fpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(src, out_reg0), sink);
+                    modrm_rr(src, out_reg0, sink);
+                "#,
+            ),
+    );
+
     // XX /r, RM form, GPR -> FPR.
     recipes.add_template_recipe(
         EncodingRecipeBuilder::new("frurm", f_unary, 1)
diff --git a/cranelift/codegen/meta/src/shared/formats.rs b/cranelift/codegen/meta/src/shared/formats.rs
index 0af3c264ae..e6e7c92df7 100644
--- a/cranelift/codegen/meta/src/shared/formats.rs
+++ b/cranelift/codegen/meta/src/shared/formats.rs
@@ -157,6 +157,7 @@ pub fn define(immediates: &OperandKinds, entities: &OperandKinds) -> FormatRegis
             .imm(("src", regunit))
             .imm(("dst", regunit)),
     );
+    registry.insert(Builder::new("CopyToSsa").imm(("src", regunit)));
     registry.insert(
         Builder::new("RegSpill")
             .value()
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index dfeeec4906..3d3fda49b9 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1194,6 +1194,22 @@ pub fn define(
         .can_load(true),
     );
 
+    ig.push(
+        Inst::new(
+            "fill_nop",
+            r#"
+        This is identical to `fill`, except it has no encoding, since it is a no-op.
+
+        This instruction is created only during late-stage redundant-reload removal, after all
+        registers and stack slots have been assigned.  It is used to replace `fill`s that have
+        been identified as redundant.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a])
+        .can_load(true),
+    );
+
     let src = &operand("src", regunit);
     let dst = &operand("dst", regunit);
 
@@ -1233,6 +1249,23 @@ pub fn define(
         .other_side_effects(true),
     );
 
+    ig.push(
+        Inst::new(
+            "copy_to_ssa",
+            r#"
+        Copies the contents of ''src'' register to ''a'' SSA name.
+
+        This instruction copies the contents of one register, regardless of its SSA name, to
+        another register, creating a new SSA name.  In that sense it is a one-sided version
+        of ''copy_special''.  This instruction is internal and should not be created by
+        Cranelift users.
+        "#,
+        )
+        .operands_in(vec![src])
+        .operands_out(vec![a])
+        .other_side_effects(true),
+    );
+
     ig.push(
         Inst::new(
             "copy_nop",
diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index cbf412aa20..3953f7e4d4 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -23,6 +23,7 @@ use crate::licm::do_licm;
 use crate::loop_analysis::LoopAnalysis;
 use crate::nan_canonicalization::do_nan_canonicalization;
 use crate::postopt::do_postopt;
+use crate::redundant_reload_remover::RedundantReloadRemover;
 use crate::regalloc;
 use crate::result::CodegenResult;
 use crate::settings::{FlagsOrIsa, OptLevel};
@@ -50,6 +51,9 @@ pub struct Context {
 
     /// Loop analysis of `func`.
     pub loop_analysis: LoopAnalysis,
+
+    /// Redundant-reload remover context.
+    pub redundant_reload_remover: RedundantReloadRemover,
 }
 
 impl Context {
@@ -72,6 +76,7 @@ impl Context {
             domtree: DominatorTree::new(),
             regalloc: regalloc::Context::new(),
             loop_analysis: LoopAnalysis::new(),
+            redundant_reload_remover: RedundantReloadRemover::new(),
         }
     }
 
@@ -82,6 +87,7 @@ impl Context {
         self.domtree.clear();
         self.regalloc.clear();
         self.loop_analysis.clear();
+        self.redundant_reload_remover.clear();
     }
 
     /// Compile the function, and emit machine code into a `Vec`.
@@ -149,6 +155,7 @@ impl Context {
         self.regalloc(isa)?;
         self.prologue_epilogue(isa)?;
         if isa.flags().opt_level() == OptLevel::Best {
+            self.redundant_reload_remover(isa)?;
             self.shrink_instructions(isa)?;
         }
         self.relax_branches(isa)
@@ -322,6 +329,14 @@ impl Context {
         Ok(())
     }
 
+    /// Do redundant-reload removal after allocation of both registers and stack slots.
+    pub fn redundant_reload_remover(&mut self, isa: &dyn TargetIsa) -> CodegenResult<()> {
+        self.redundant_reload_remover
+            .run(isa, &mut self.func, &self.cfg);
+        self.verify_if(isa)?;
+        Ok(())
+    }
+
     /// Run the instruction shrinking pass.
     pub fn shrink_instructions(&mut self, isa: &dyn TargetIsa) -> CodegenResult<()> {
         shrink_instructions(&mut self.func, isa);
diff --git a/cranelift/codegen/src/ir/stackslot.rs b/cranelift/codegen/src/ir/stackslot.rs
index 0e1ec4777a..0b488582d4 100644
--- a/cranelift/codegen/src/ir/stackslot.rs
+++ b/cranelift/codegen/src/ir/stackslot.rs
@@ -209,11 +209,6 @@ impl StackSlots {
         self.slots.is_valid(ss)
     }
 
-    /// Set the offset of a stack slot.
-    pub fn set_offset(&mut self, ss: StackSlot, offset: StackOffset) {
-        self.slots[ss].offset = Some(offset);
-    }
-
     /// Get an iterator over all the stack slot keys.
     pub fn iter(&self) -> Iter {
         self.slots.iter()
diff --git a/cranelift/codegen/src/lib.rs b/cranelift/codegen/src/lib.rs
index 5b809460cf..b682ea3867 100644
--- a/cranelift/codegen/src/lib.rs
+++ b/cranelift/codegen/src/lib.rs
@@ -95,6 +95,7 @@ mod nan_canonicalization;
 mod partition_slice;
 mod postopt;
 mod predicates;
+mod redundant_reload_remover;
 mod ref_slice;
 mod regalloc;
 mod result;
diff --git a/cranelift/codegen/src/redundant_reload_remover.rs b/cranelift/codegen/src/redundant_reload_remover.rs
new file mode 100644
index 0000000000..9bcc3fbc9d
--- /dev/null
+++ b/cranelift/codegen/src/redundant_reload_remover.rs
@@ -0,0 +1,904 @@
+//! This module implements a late-stage redundant-reload remover, which runs after registers have
+//! been allocated and stack slots have been given specific offsets.
+
+use crate::cursor::{Cursor, CursorPosition, EncCursor, FuncCursor};
+use crate::entity::EntitySet;
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir::dfg::DataFlowGraph;
+use crate::ir::instructions::BranchInfo;
+use crate::ir::stackslot::{StackSlotKind, StackSlots};
+use crate::ir::{
+    Ebb, Function, Inst, InstBuilder, InstructionData, Opcode, StackSlotData, Type, Value, ValueLoc,
+};
+use crate::isa::{RegInfo, RegUnit, TargetIsa};
+use crate::regalloc::RegDiversions;
+use core::convert::TryInto;
+use cranelift_entity::{PrimaryMap, SecondaryMap};
+use std::vec::Vec;
+
+// =============================================================================================
+// A description of the redundant-fill-removal algorithm
+//
+//
+// The algorithm works forwards through each Ebb.  It carries along and updates a table,
+// AvailEnv, with which it tracks registers that are known to have the same value as some stack
+// slot.  The actions on encountering an instruction depend on the instruction, as follows:
+//
+// ss1 = spill r0: update the AvailEnv so as to note that slot `ss1` and register `r0`
+//                 have the same value.
+//
+// r1 = fill ss0: look in the AvailEnv.  If it tells us that register `r1` and slot `ss0`
+//                have the same value, then delete the instruction by converting it to a
+//                `fill_nop`.
+//
+//                If it tells us that some other register `r2` has the same value as
+//                slot `ss0`, convert the instruction into a copy from `r2` to `r1`.
+//
+// any other insn: remove from the AvailEnv, any bindings associated with registers
+//                 written by this instruction, since they will be invalidated by it.
+//
+// Tracking the effects of `copy` instructions in AvailEnv for the case when both source and
+// destination are registers does not cause any more fills to be removed or converted to copies.
+// It's not clear why.
+//
+// There are various other instruction-handling cases in `visit_inst`, which are documented
+// in-line, and do not change the core algorithm, so are not described here.
+//
+// The registers tracked by AvailEnv are the post-diversion registers that are really used by the
+// code; they are not the pre-diversion names associated with each SSA `Value`.  The second
+// `fill` case above opportunistically copies values from registers that may have been diversion
+// targets in some predecessor block, and so are no longer associated with any specific SSA-level
+// name at the point the copy is made.  Hence those copies (from `r2` to `r1`) cannot be done
+// with an ordinary `copy` instruction.  Instead they have to be done using a new `copy_to_ssa`
+// instruction, which copies from an arbitrary register to a register-resident `Value` (that is,
+// "back to" SSA-world).
+//
+// That completes the description of the core algorithm.
+//
+// In the case where a block `A` jumps to `B` and `A` is the only predecessor of `B`, the
+// AvailEnv at the end of `A` will still be valid at the entry to `B`.  In such a case, we can
+// profitably transform `B` using the AvailEnv "inherited" from `A`.  In order to take full
+// advantage of this, this module partitions the function's CFG into tree-shaped groups of
+// blocks, and processes each tree as described above.  So the AvailEnv is only initialised to
+// empty at the start of blocks that form the root of each tree; that is, for blocks which have
+// two or more predecessors.
+
+// =============================================================================================
+// Top level algorithm structure
+//
+// The overall algorithm, for a function, starts like this:
+//
+// * (once per function): finds Ebbs that have two or more predecessors, since they will be the
+//   roots of Ebb trees.  Also, the entry node for the function is considered to be a root.
+//
+// It then continues with a loop that first finds a tree of Ebbs ("discovery") and then removes
+// redundant fills as described above ("processing"):
+//
+// * (discovery; once per tree): for each root, performs a depth first search to find all the Ebbs
+//   in the tree, guided by RedundantReloadRemover::discovery_stack.
+//
+// * (processing; once per tree): the just-discovered tree is then processed as described above,
+//   guided by RedundantReloadRemover::processing_stack.
+//
+// In this way, all Ebbs reachable from the function's entry point are eventually processed.  Note
+// that each tree is processed as soon as it has been discovered, so the algorithm never creates a
+// list of trees for the function.
+//
+// The running state is stored in `RedundantReloadRemover`.  This is allocated once and can be
+// reused for multiple functions so as to minimise heap turnover.  The fields are, roughly:
+//
+//   num_regunits -- constant for the whole function; used by the tree processing phase
+//   num_preds_per_ebb -- constant for the whole function; used by the tree discovery process
+//
+//   discovery_stack -- used to guide the tree discovery process
+//   nodes_in_tree -- the discovered nodes are recorded here
+//
+//   processing_stack -- used to guide the tree processing process
+//   nodes_already_visited -- used to ensure the tree processing logic terminates in the case
+//                            where a tree has a branch back to its root node.
+//
+// There is further documentation in line below, as appropriate.
+
+// =============================================================================================
+// A side note on register choice heuristics
+
+// The core algorithm opportunistically replaces fill instructions when it knows of a register
+// that already holds the required value.  How effective this is largely depends on how long
+// reloaded values happen to stay alive before the relevant register is overwritten.  And that
+// depends on the register allocator's register choice heuristics.  The worst case is, when the
+// register allocator reuses registers as soon as possible after they become free.  Unfortunately
+// that was indeed the selection scheme, prior to development of this pass.
+//
+// As part of this work, the register selection scheme has been changed as follows: for registers
+// written by any instruction other than a fill, use the lowest numbered available register.  But
+// for registers written by a fill instruction, use the highest numbered available register.  The
+// aim is to try and keep reload- and non-reload registers disjoint to the extent possible.
+// Several other schemes were tried, but this one is simple and can be worth an extra 2% of
+// performance in some cases.
+//
+// The relevant change is more or less a one-line change in the solver.
+
+// =============================================================================================
+// Data structures used for discovery of trees
+
+// `ZeroOneOrMany` is used to record the number of predecessors an Ebb block has.  The `Zero` case
+// is included so as to cleanly handle the case where the incoming graph has unreachable Ebbs.
+
+#[derive(Clone, PartialEq)]
+enum ZeroOneOrMany {
+    Zero,
+    One,
+    Many,
+}
+
+// =============================================================================================
+// Data structures used for processing of trees
+
+// `SlotInfo` describes a spill slot in the obvious way.  Note that it doesn't indicate which
+// register(s) are currently associated with the slot.  That job is done by `AvailEnv` instead.
+//
+// In the CL framework, stack slots are partitioned into disjoint sets, one for each
+// `StackSlotKind`.  The offset and size only give a unique identity within any particular
+// `StackSlotKind`.  So, to uniquely identify a stack slot, all three fields are necessary.
+
+#[derive(Clone, Copy)]
+struct SlotInfo {
+    kind: StackSlotKind,
+    offset: i32,
+    size: u32,
+}
+
+// `AvailEnv` maps each possible register to a stack slot that holds the same value.  The index
+// space of `AvailEnv::map` is exactly the set of registers available on the current target.  If
+// (as is mostly the case) a register is not known to have the same value as a stack slot, then
+// its entry is `None` rather than `Some(..)`.
+//
+// Invariants for AvailEnv:
+//
+// AvailEnv may have multiple different registers bound to the same stack slot -- that is, `(kind,
+// offset, size)` triple.  That's OK, and reflects the reality that those two registers contain
+// the same value.  This could happen, for example, in the case
+//
+//   ss1 = spill r0
+//   ..
+//   r2 = fill ss1
+//
+// Then both `r0` and `r2` will have the same value as `ss1`, provided that ".." doesn't write to
+// `r1`.
+//
+// To say that two different registers may be bound to the same stack slot is the same as saying
+// that it is allowed to have two different entries in AvailEnv with the same `(kind, offset,
+// size)` triple.  What is *not* allowed is to have partial overlaps.  That is, if two SlotInfos
+// have the same `kind` field and have `offset` and `size` fields that overlap, then their
+// `offset` and `size` fields must be identical.  This is so as to make the algorithm safe against
+// situations where, for example, a 64 bit register is spilled, but then only the bottom 32 bits
+// are reloaded from the slot.
+//
+// Although in such a case it seems likely that the Cranelift IR would be ill-typed, and so this
+// case could probably not occur in practice.
+
+#[derive(Clone)]
+struct AvailEnv {
+    map: Vec>,
+}
+
+// `ProcessingStackElem` combines AvailEnv with contextual information needed to "navigate" within
+// an Ebb.
+//
+// A ProcessingStackElem conceptually has the lifetime of exactly one Ebb: once the current Ebb is
+// completed, the ProcessingStackElem will be abandoned.  In practice the top level state,
+// RedundantReloadRemover, caches them, so as to avoid heap turnover.
+//
+// Note that ProcessingStackElem must contain a CursorPosition.  The CursorPosition, which
+// indicates where we are in the current Ebb, cannot be implicitly maintained by looping over all
+// the instructions in an Ebb in turn, because we may choose to suspend processing the current Ebb
+// at a side exit, continue by processing the subtree reached via the side exit, and only later
+// resume the current Ebb.
+
+struct ProcessingStackElem {
+    /// Indicates the AvailEnv at the current point in the Ebb.
+    avail_env: AvailEnv,
+
+    /// Shows where we currently are inside the Ebb.
+    cursor: CursorPosition,
+
+    /// Indicates the currently active register diversions at the current point.
+    diversions: RegDiversions,
+}
+
+// =============================================================================================
+// The top level data structure
+
+// `RedundantReloadRemover` contains data structures for the two passes: discovery of tree shaped
+// regions, and processing of them.  These are allocated once and stay alive for the entire
+// function, even though they are cleared out for each new tree shaped region.  It also caches
+// `num_regunits` and `num_preds_per_ebb`, which are computed at the start of each function and
+// then remain constant.
+
+/// The redundant reload remover's state.
+pub struct RedundantReloadRemover {
+    /// The total number of RegUnits available on this architecture.  This is unknown when the
+    /// RedundantReloadRemover is created.  It becomes known at the beginning of processing of a
+    /// function.
+    num_regunits: Option,
+
+    /// This stores, for each Ebb, a characterisation of the number of predecessors it has.
+    num_preds_per_ebb: PrimaryMap,
+
+    /// The stack used for the first phase (discovery).  There is one element on the discovery
+    /// stack for each currently unexplored Ebb in the tree being searched.
+    discovery_stack: Vec,
+
+    /// The nodes in the discovered tree are inserted here.
+    nodes_in_tree: EntitySet,
+
+    /// The stack used during the second phase (transformation).  There is one element on the
+    /// processing stack for each currently-open node in the tree being transformed.
+    processing_stack: Vec,
+
+    /// Used in the second phase to avoid visiting nodes more than once.
+    nodes_already_visited: EntitySet,
+}
+
+// =============================================================================================
+// Miscellaneous small helper functions
+
+// Is this a kind of stack slot that is safe to track in AvailEnv?  This is probably overly
+// conservative, but tracking only the SpillSlot and IncomingArgument kinds catches almost all
+// available redundancy in practice.
+fn is_slot_kind_tracked(kind: StackSlotKind) -> bool {
+    match kind {
+        StackSlotKind::SpillSlot | StackSlotKind::IncomingArg => true,
+        _ => false,
+    }
+}
+
+// Find out if the range `[offset, +size)` overlaps with the range in `si`.
+fn overlaps(si: &SlotInfo, offset: i32, size: u32) -> bool {
+    let a_offset = si.offset as i64;
+    let a_size = si.size as i64;
+    let b_offset = offset as i64;
+    let b_size = size as i64;
+    let no_overlap = a_offset + a_size <= b_offset || b_offset + b_size <= a_offset;
+    !no_overlap
+}
+
+// Find, in `reginfo`, the register bank that `reg` lives in, and return the lower limit and size
+// of the bank.  This is so the caller can conveniently iterate over all RegUnits in the bank that
+// `reg` lives in.
+fn find_bank_limits(reginfo: &RegInfo, reg: RegUnit) -> (RegUnit, u16) {
+    if let Some(bank) = reginfo.bank_containing_regunit(reg) {
+        return (bank.first_unit, bank.units);
+    }
+    // We should never get here, since `reg` must come from *some* RegBank.
+    panic!("find_regclass_limits: reg not found");
+}
+
+// Returns the register that `v` is allocated to.  Assumes that `v` actually resides in a
+// register.
+fn reg_of_value(locations: &SecondaryMap, v: Value) -> RegUnit {
+    match locations[v] {
+        ValueLoc::Reg(ru) => ru,
+        _ => panic!("reg_of_value: value isn't in a reg"),
+    }
+}
+
+// Returns the stack slot that `v` is allocated to.  Assumes that `v` actually resides in a stack
+// slot.
+fn slot_of_value<'s>(
+    locations: &SecondaryMap,
+    stack_slots: &'s StackSlots,
+    v: Value,
+) -> &'s StackSlotData {
+    match locations[v] {
+        ValueLoc::Stack(slot) => &stack_slots[slot],
+        _ => panic!("slot_of_value: value isn't in a stack slot"),
+    }
+}
+
+// =============================================================================================
+// Top level: discovery of tree shaped regions
+
+impl RedundantReloadRemover {
+    // A helper for `add_nodes_to_tree` below.
+    fn discovery_stack_push_successors_of(&mut self, cfg: &ControlFlowGraph, node: Ebb) {
+        for successor in cfg.succ_iter(node) {
+            self.discovery_stack.push(successor);
+        }
+    }
+
+    // Visit the tree of Ebbs rooted at `starting_point` and add them to `self.nodes_in_tree`.
+    // `self.num_preds_per_ebb` guides the process, ensuring we don't leave the tree-ish region
+    // and indirectly ensuring that the process will terminate in the presence of cycles in the
+    // graph.  `self.discovery_stack` holds the search state in this function.
+    fn add_nodes_to_tree(&mut self, cfg: &ControlFlowGraph, starting_point: Ebb) {
+        // One might well ask why this doesn't loop forever when it encounters cycles in the
+        // control flow graph.  The reason is that any cycle in the graph that is reachable from
+        // anywhere outside the cycle -- in particular, that is reachable from the function's
+        // entry node -- must have at least one node that has two or more predecessors.  So the
+        // logic below won't follow into it, because it regards any such node as the root of some
+        // other tree.
+        debug_assert!(self.discovery_stack.is_empty());
+        debug_assert!(self.nodes_in_tree.is_empty());
+
+        self.nodes_in_tree.insert(starting_point);
+        self.discovery_stack_push_successors_of(cfg, starting_point);
+
+        while let Some(node) = self.discovery_stack.pop() {
+            match self.num_preds_per_ebb[node] {
+                // We arrived at a node with multiple predecessors, so it's a new root.  Ignore it.
+                ZeroOneOrMany::Many => {}
+                // This node has just one predecessor, so we should incorporate it in the tree and
+                // immediately transition into searching from it instead.
+                ZeroOneOrMany::One => {
+                    self.nodes_in_tree.insert(node);
+                    self.discovery_stack_push_successors_of(cfg, node);
+                }
+                // This is meaningless.  We arrived at a node that doesn't point back at where we
+                // came from.
+                ZeroOneOrMany::Zero => panic!("add_nodes_to_tree: inconsistent graph"),
+            }
+        }
+    }
+}
+
+// =============================================================================================
+// Operations relating to `AvailEnv`
+
+impl AvailEnv {
+    // Create a new one.
+    fn new(size: usize) -> Self {
+        let mut env = AvailEnv {
+            map: Vec::>::new(),
+        };
+        env.map.resize(size, None);
+        env
+    }
+
+    // Debug only: checks (some of) the required AvailEnv invariants.
+    #[cfg(debug_assertions)]
+    fn check_invariants(&self) -> bool {
+        // Check that any overlapping entries overlap exactly.  This is super lame (quadratic),
+        // but it's only used in debug builds.
+        for i in 0..self.map.len() {
+            if let Some(si) = self.map[i] {
+                for j in i + 1..self.map.len() {
+                    if let Some(sj) = self.map[j] {
+                        // "si and sj overlap, but not exactly"
+                        if si.kind == sj.kind
+                            && overlaps(&si, sj.offset, sj.size)
+                            && !(si.offset == sj.offset && si.size == sj.size)
+                        {
+                            return false;
+                        }
+                    }
+                }
+            }
+        }
+        true
+    }
+
+    // Invalidates the binding associated with `reg`.  Note that by construction of AvailEnv,
+    // `reg` can only be associated with one binding at once.
+    fn invalidate_by_reg(&mut self, reg: RegUnit) {
+        self.map[reg as usize] = None;
+    }
+
+    // Invalidates any binding that has any overlap with `(kind, offset, size)`.
+    fn invalidate_by_offset(&mut self, kind: StackSlotKind, offset: i32, size: u32) {
+        debug_assert!(is_slot_kind_tracked(kind));
+        for i in 0..self.map.len() {
+            if let Some(si) = &self.map[i] {
+                if si.kind == kind && overlaps(&si, offset, size) {
+                    self.map[i] = None;
+                }
+            }
+        }
+    }
+
+    // Invalidates all bindings.
+    fn invalidate_all(&mut self) {
+        for i in 0..self.map.len() {
+            self.map[i] = None;
+        }
+    }
+
+    // Updates AvailEnv to track the effect of a `regmove` instruction.
+    fn copy_reg(&mut self, src: RegUnit, dst: RegUnit) {
+        self.map[dst as usize] = self.map[src as usize];
+    }
+
+    // Does `env` have the exact binding characterised by `(reg, kind, offset, size)` ?
+    fn has_exact_binding(&self, reg: RegUnit, kind: StackSlotKind, offset: i32, size: u32) -> bool {
+        debug_assert!(is_slot_kind_tracked(kind));
+        if let Some(si) = &self.map[reg as usize] {
+            return si.kind == kind && si.offset == offset && si.size == size;
+        }
+        // No such binding.
+        false
+    }
+
+    // Does `env` have a binding characterised by `(kind, offset, size)` but to a register, let's
+    // call it `other_reg`, that isn't `reg`?  If so, return `other_reg`.  Note that `other_reg`
+    // will have the same bank as `reg`.  It is a checked error to call this function with a
+    // binding matching all four of `(reg, kind, offset, size)`.
+    fn has_inexact_binding(
+        &self,
+        reginfo: &RegInfo,
+        reg: RegUnit,
+        kind: StackSlotKind,
+        offset: i32,
+        size: u32,
+    ) -> Option {
+        debug_assert!(is_slot_kind_tracked(kind));
+        // Find the range of RegUnit numbers for the bank that contains `reg`, and use that as our
+        // search space.  This is so as to guarantee that any match is restricted to the same bank
+        // as `reg`.
+        let (first_unit, num_units) = find_bank_limits(reginfo, reg);
+        for other_reg in first_unit..first_unit + num_units {
+            if let Some(si) = &self.map[other_reg as usize] {
+                if si.kind == kind && si.offset == offset && si.size == size {
+                    if other_reg == reg {
+                        panic!("has_inexact_binding: binding *is* exact!");
+                    }
+                    return Some(other_reg);
+                }
+            }
+        }
+        // No such binding.
+        None
+    }
+
+    // Create the binding `(reg, kind, offset, size)` in `env`, and throw away any previous
+    // binding associated with either `reg` or the `(kind, offset, size)` triple.
+    fn bind(&mut self, reg: RegUnit, kind: StackSlotKind, offset: i32, size: u32) {
+        debug_assert!(is_slot_kind_tracked(kind));
+        self.invalidate_by_offset(kind, offset, size);
+        self.map[reg as usize] = Some(SlotInfo { kind, offset, size });
+    }
+}
+
+// Invalidates in `avail_env`, any binding associated with a regunit that is written by `inst`.
+fn invalidate_regs_written_by_inst(
+    locations: &SecondaryMap,
+    diversions: &RegDiversions,
+    dfg: &DataFlowGraph,
+    avail_env: &mut AvailEnv,
+    inst: Inst,
+) {
+    for v in dfg.inst_results(inst).iter() {
+        if let ValueLoc::Reg(ru) = locations[*v] {
+            // This must be true.  It would be meaningless for an SSA value to be diverted before
+            // the point where it is defined.
+            debug_assert!(diversions.reg(*v, locations) == ru);
+            avail_env.invalidate_by_reg(ru);
+        }
+    }
+}
+
+// =============================================================================================
+// Processing of individual instructions
+
+impl RedundantReloadRemover {
+    // Process `inst`, possibly changing it into a different instruction, and possibly changing
+    // `self.avail_env` and `func.dfg`.
+    fn visit_inst(
+        &mut self,
+        func: &mut Function,
+        reginfo: &RegInfo,
+        isa: &dyn TargetIsa,
+        inst: Inst,
+    ) {
+        // Get hold of the top-of-stack work item.  This is the state that we will mutate during
+        // processing of this instruction.
+        debug_assert!(!self.processing_stack.is_empty());
+        let ProcessingStackElem {
+            avail_env,
+            cursor: _,
+            diversions,
+        } = &mut self.processing_stack.last_mut().unwrap();
+
+        #[cfg(debug_assertions)]
+        debug_assert!(
+            avail_env.check_invariants(),
+            "visit_inst: env invariants not ok"
+        );
+
+        let dfg = &mut func.dfg;
+        let locations = &func.locations;
+        let stack_slots = &func.stack_slots;
+
+        // To avoid difficulties with the borrow checker, do this in two stages.  First, examine
+        // the instruction to see if it can be deleted or modified, and park the relevant
+        // information in `transform`.  Update `self.avail_env` too.  Later, use `transform` to
+        // actually do the transformation if necessary.
+        enum Transform {
+            NoChange,
+            ChangeToNopFill(Value),           // delete this insn entirely
+            ChangeToCopyToSSA(Type, RegUnit), // change it into a copy from the specified reg
+        }
+        let mut transform = Transform::NoChange;
+
+        // In this match { .. } statement, either we must treat the instruction specially, or we
+        // must call `invalidate_regs_written_by_inst` on it.
+        match &dfg[inst] {
+            InstructionData::Unary {
+                opcode: Opcode::Spill,
+                arg: src_value,
+            } => {
+                // Extract: (src_reg, kind, offset, size)
+                // Invalidate: (kind, offset, size)
+                // Add new binding: {src_reg -> (kind, offset, size)}
+                // Don't forget that src_value might be diverted, so we have to deref it.
+                let slot = slot_of_value(locations, stack_slots, dfg.inst_results(inst)[0]);
+                let src_reg = diversions.reg(*src_value, locations);
+                let kind = slot.kind;
+                if is_slot_kind_tracked(kind) {
+                    let offset = slot.offset.expect("visit_inst: spill with no offset");
+                    let size = slot.size;
+                    avail_env.bind(src_reg, kind, offset, size);
+                } else {
+                    // We don't expect this insn to write any regs.  But to be consistent with the
+                    // rule above, do this anyway.
+                    invalidate_regs_written_by_inst(locations, diversions, dfg, avail_env, inst);
+                }
+            }
+            InstructionData::Unary {
+                opcode: Opcode::Fill,
+                arg: src_value,
+            } => {
+                // Extract: (dst_reg, kind, offset, size)
+                // Invalidate: (kind, offset, size)
+                // Add new: {dst_reg -> (dst_value, kind, offset, size)}
+                let slot = slot_of_value(locations, stack_slots, *src_value);
+                let dst_value = dfg.inst_results(inst)[0];
+                let dst_reg = reg_of_value(locations, dst_value);
+                // This must be true.  It would be meaningless for an SSA value to be diverted
+                // before it was defined.
+                debug_assert!(dst_reg == diversions.reg(dst_value, locations));
+                let kind = slot.kind;
+                if is_slot_kind_tracked(kind) {
+                    let offset = slot.offset.expect("visit_inst: fill with no offset");
+                    let size = slot.size;
+                    if avail_env.has_exact_binding(dst_reg, kind, offset, size) {
+                        // This instruction is an exact copy of a fill we saw earlier, and the
+                        // loaded value is still valid.  So we'll schedule this instruction for
+                        // deletion (below).  No need to make any changes to `avail_env`.
+                        transform = Transform::ChangeToNopFill(*src_value);
+                    } else if let Some(other_reg) =
+                        avail_env.has_inexact_binding(reginfo, dst_reg, kind, offset, size)
+                    {
+                        // This fill is from the required slot, but into a different register
+                        // `other_reg`.  So replace it with a copy from `other_reg` to `dst_reg`
+                        // and update `dst_reg`s binding to make it the same as `other_reg`'s, so
+                        // as to maximise the chances of future matches after this instruction.
+                        debug_assert!(other_reg != dst_reg);
+                        transform =
+                            Transform::ChangeToCopyToSSA(dfg.value_type(dst_value), other_reg);
+                        avail_env.copy_reg(other_reg, dst_reg);
+                    } else {
+                        // This fill creates some new binding we don't know about.  Update
+                        // `avail_env` to track it.
+                        avail_env.bind(dst_reg, kind, offset, size);
+                    }
+                } else {
+                    // Else it's "just another instruction that writes a reg", so we'd better
+                    // treat it as such, just as we do below for instructions that we don't handle
+                    // specially.
+                    invalidate_regs_written_by_inst(locations, diversions, dfg, avail_env, inst);
+                }
+            }
+            InstructionData::RegMove {
+                opcode: _,
+                arg: _,
+                src,
+                dst,
+            } => {
+                // These happen relatively rarely, but just frequently enough that it's worth
+                // tracking the copy (at the machine level, it's really a copy) in `avail_env`.
+                avail_env.copy_reg(*src, *dst);
+            }
+            InstructionData::RegSpill { .. }
+            | InstructionData::RegFill { .. }
+            | InstructionData::Call { .. }
+            | InstructionData::CallIndirect { .. }
+            | InstructionData::StackLoad { .. }
+            | InstructionData::StackStore { .. }
+            | InstructionData::Unary {
+                opcode: Opcode::AdjustSpDown,
+                ..
+            }
+            | InstructionData::UnaryImm {
+                opcode: Opcode::AdjustSpUpImm,
+                ..
+            }
+            | InstructionData::UnaryImm {
+                opcode: Opcode::AdjustSpDownImm,
+                ..
+            } => {
+                // All of these change, or might change, the memory-register bindings tracked in
+                // `avail_env` in some way we don't know about, or at least, we might be able to
+                // track, but for which the effort-to-benefit ratio seems too low to bother.  So
+                // play safe: forget everything we know.
+                //
+                // For Call/CallIndirect, we could do better when compiling for calling
+                // conventions that have callee-saved registers, since bindings for them would
+                // remain valid across the call.
+                avail_env.invalidate_all();
+            }
+            _ => {
+                // Invalidate: any `avail_env` entry associated with a reg written by `inst`.
+                invalidate_regs_written_by_inst(locations, diversions, dfg, avail_env, inst);
+            }
+        }
+
+        // Actually do the transformation.
+        match transform {
+            Transform::NoChange => {}
+            Transform::ChangeToNopFill(arg) => {
+                // Load is completely redundant.  Convert it to a no-op.
+                dfg.replace(inst).fill_nop(arg);
+                let ok = func.update_encoding(inst, isa).is_ok();
+                debug_assert!(ok, "fill_nop encoding missing for this type");
+            }
+            Transform::ChangeToCopyToSSA(ty, reg) => {
+                // We already have the relevant value in some other register.  Convert the
+                // load into a reg-reg copy.
+                dfg.replace(inst).copy_to_ssa(ty, reg);
+                let ok = func.update_encoding(inst, isa).is_ok();
+                debug_assert!(ok, "copy_to_ssa encoding missing for type {}", ty);
+            }
+        }
+    }
+}
+
+// =============================================================================================
+// Top level: processing of tree shaped regions
+
+impl RedundantReloadRemover {
+    // Push a clone of the top-of-stack ProcessingStackElem.  This will be used to process exactly
+    // one Ebb.  The diversions are created new, rather than cloned, to reflect the fact
+    // that diversions are local to each Ebb.
+    fn processing_stack_push(&mut self, cursor: CursorPosition) {
+        let avail_env = if let Some(stack_top) = self.processing_stack.last() {
+            stack_top.avail_env.clone()
+        } else {
+            AvailEnv::new(
+                self.num_regunits
+                    .expect("processing_stack_push: num_regunits unknown!")
+                    as usize,
+            )
+        };
+        self.processing_stack.push(ProcessingStackElem {
+            avail_env,
+            cursor,
+            diversions: RegDiversions::new(),
+        });
+    }
+
+    // This pushes the node `dst` onto the processing stack, and sets up the new
+    // ProcessingStackElem accordingly.  But it does all that only if `dst` is part of the current
+    // tree *and* we haven't yet visited it.
+    fn processing_stack_maybe_push(&mut self, dst: Ebb) {
+        if self.nodes_in_tree.contains(dst) && !self.nodes_already_visited.contains(dst) {
+            if !self.processing_stack.is_empty() {
+                // If this isn't the outermost node in the tree (that is, the root), then it must
+                // have exactly one predecessor.  Nodes with no predecessors are dead and not
+                // incorporated in any tree.  Nodes with two or more predecessors are the root of
+                // some other tree, and visiting them as if they were part of the current tree
+                // would be a serious error.
+                debug_assert!(self.num_preds_per_ebb[dst] == ZeroOneOrMany::One);
+            }
+            self.processing_stack_push(CursorPosition::Before(dst));
+            self.nodes_already_visited.insert(dst);
+        }
+    }
+
+    // Perform redundant-reload removal on the tree shaped region of graph defined by `root` and
+    // `self.nodes_in_tree`.  The following state is modified: `self.processing_stack`,
+    // `self.nodes_already_visited`, and `func.dfg`.
+    fn process_tree(
+        &mut self,
+        func: &mut Function,
+        reginfo: &RegInfo,
+        isa: &dyn TargetIsa,
+        root: Ebb,
+    ) {
+        debug_assert!(self.nodes_in_tree.contains(root));
+        debug_assert!(self.processing_stack.is_empty());
+        debug_assert!(self.nodes_already_visited.is_empty());
+
+        // Create the initial work item
+        self.processing_stack_maybe_push(root);
+
+        while !self.processing_stack.is_empty() {
+            // It seems somewhat ridiculous to construct a whole new FuncCursor just so we can do
+            // next_inst() on it once, and then copy the resulting position back out.  But use of
+            // a function-global FuncCursor, or of the EncCursor in struct Context, leads to
+            // borrow checker problems, as does including FuncCursor directly in
+            // ProcessingStackElem.  In any case this is not as bad as it looks, since profiling
+            // shows that the build-insert-step-extract work is reduced to just 8 machine
+            // instructions in an optimised x86_64 build, presumably because rustc can inline and
+            // then optimise out almost all the work.
+            let tos = self.processing_stack.len() - 1;
+            let mut pos = FuncCursor::new(func).at_position(self.processing_stack[tos].cursor);
+            let maybe_inst = pos.next_inst();
+            self.processing_stack[tos].cursor = pos.position();
+
+            if let Some(inst) = maybe_inst {
+                // Deal with this insn, possibly changing it, possibly updating the top item of
+                // `self.processing_stack`.
+                self.visit_inst(func, reginfo, isa, inst);
+
+                // Update diversions after the insn.
+                self.processing_stack[tos].diversions.apply(&func.dfg[inst]);
+
+                // If the insn can branch outside this Ebb, push work items on the stack for all
+                // target Ebbs that are part of the same tree and that we haven't yet visited.
+                // The next iteration of this instruction-processing loop will immediately start
+                // work on the most recently pushed Ebb, and will eventually continue in this Ebb
+                // when those new items have been removed from the stack.
+                match func.dfg.analyze_branch(inst) {
+                    BranchInfo::NotABranch => (),
+                    BranchInfo::SingleDest(dst, _) => {
+                        self.processing_stack_maybe_push(dst);
+                    }
+                    BranchInfo::Table(jt, default) => {
+                        func.jump_tables[jt]
+                            .iter()
+                            .for_each(|dst| self.processing_stack_maybe_push(*dst));
+                        if let Some(dst) = default {
+                            self.processing_stack_maybe_push(dst);
+                        }
+                    }
+                }
+            } else {
+                // We've come to the end of the current work-item (Ebb).  We'll already have
+                // processed the fallthrough/continuation/whatever for it using the logic above.
+                // Pop it off the stack and resume work on its parent.
+                self.processing_stack.pop();
+            }
+        }
+    }
+}
+
+// =============================================================================================
+// Top level: perform redundant fill removal for a complete function
+
+impl RedundantReloadRemover {
+    /// Create a new remover state.
+    pub fn new() -> Self {
+        Self {
+            num_regunits: None,
+            num_preds_per_ebb: PrimaryMap::::with_capacity(8),
+            discovery_stack: Vec::::with_capacity(16),
+            nodes_in_tree: EntitySet::::new(),
+            processing_stack: Vec::::with_capacity(8),
+            nodes_already_visited: EntitySet::::new(),
+        }
+    }
+
+    /// Clear the state of the remover.
+    pub fn clear(&mut self) {
+        self.clear_for_new_function();
+    }
+
+    fn clear_for_new_function(&mut self) {
+        self.num_preds_per_ebb.clear();
+        self.clear_for_new_tree();
+    }
+
+    fn clear_for_new_tree(&mut self) {
+        self.discovery_stack.clear();
+        self.nodes_in_tree.clear();
+        self.processing_stack.clear();
+        self.nodes_already_visited.clear();
+    }
+
+    #[inline(never)]
+    fn do_redundant_fill_removal_on_function(
+        &mut self,
+        func: &mut Function,
+        reginfo: &RegInfo,
+        isa: &dyn TargetIsa,
+        cfg: &ControlFlowGraph,
+    ) {
+        // Fail in an obvious way if there are more than (2^32)-1 Ebbs in this function.
+        let num_ebbs: u32 = func.dfg.num_ebbs().try_into().unwrap();
+
+        // Clear out per-tree state.
+        self.clear_for_new_function();
+
+        // Create a PrimaryMap that summarises the number of predecessors for each block, as 0, 1
+        // or "many", and that also claims the entry block as having "many" predecessors.
+        self.num_preds_per_ebb.clear();
+        self.num_preds_per_ebb.reserve(num_ebbs as usize);
+
+        for i in 0..num_ebbs {
+            let mut pi = cfg.pred_iter(Ebb::from_u32(i));
+            let mut n_pi = ZeroOneOrMany::Zero;
+            if let Some(_) = pi.next() {
+                n_pi = ZeroOneOrMany::One;
+                if let Some(_) = pi.next() {
+                    n_pi = ZeroOneOrMany::Many;
+                    // We don't care if there are more than two preds, so stop counting now.
+                }
+            }
+            self.num_preds_per_ebb.push(n_pi);
+        }
+        debug_assert!(self.num_preds_per_ebb.len() == num_ebbs as usize);
+
+        // The entry block must be the root of some tree, so set up the state to reflect that.
+        let entry_ebb = func
+            .layout
+            .entry_block()
+            .expect("do_redundant_fill_removal_on_function: entry ebb unknown");
+        debug_assert!(self.num_preds_per_ebb[entry_ebb] == ZeroOneOrMany::Zero);
+        self.num_preds_per_ebb[entry_ebb] = ZeroOneOrMany::Many;
+
+        // Now build and process trees.
+        for root_ix in 0..self.num_preds_per_ebb.len() {
+            let root = Ebb::from_u32(root_ix as u32);
+
+            // Build a tree for each node that has two or more preds, and ignore all other nodes.
+            if self.num_preds_per_ebb[root] != ZeroOneOrMany::Many {
+                continue;
+            }
+
+            // Clear out per-tree state.
+            self.clear_for_new_tree();
+
+            // Discovery phase: build the tree, as `root` and `self.nodes_in_tree`.
+            self.add_nodes_to_tree(cfg, root);
+            debug_assert!(self.nodes_in_tree.cardinality() > 0);
+            debug_assert!(self.num_preds_per_ebb[root] == ZeroOneOrMany::Many);
+
+            // Processing phase: do redundant-reload-removal.
+            self.process_tree(func, reginfo, isa, root);
+            debug_assert!(
+                self.nodes_in_tree.cardinality() == self.nodes_already_visited.cardinality()
+            );
+        }
+    }
+}
+
+// =============================================================================================
+// Top level: the external interface
+
+struct Context<'a> {
+    // Current instruction as well as reference to function and ISA.
+    cur: EncCursor<'a>,
+
+    // Cached ISA information.  We save it here to avoid frequent virtual function calls on the
+    // `TargetIsa` trait object.
+    reginfo: RegInfo,
+
+    // References to contextual data structures we need.
+    cfg: &'a ControlFlowGraph,
+
+    // The running state.
+    state: &'a mut RedundantReloadRemover,
+}
+
+impl RedundantReloadRemover {
+    /// Run the remover.
+    pub fn run(&mut self, isa: &dyn TargetIsa, func: &mut Function, cfg: &ControlFlowGraph) {
+        let ctx = Context {
+            cur: EncCursor::new(func, isa),
+            reginfo: isa.register_info(),
+            cfg: cfg,
+            state: &mut RedundantReloadRemover::new(),
+        };
+        let mut total_regunits = 0;
+        for rb in isa.register_info().banks {
+            total_regunits += rb.units;
+        }
+        ctx.state.num_regunits = Some(total_regunits);
+        ctx.state.do_redundant_fill_removal_on_function(
+            ctx.cur.func,
+            &ctx.reginfo,
+            ctx.cur.isa,
+            &ctx.cfg,
+        );
+    }
+}
diff --git a/cranelift/codegen/src/regalloc/coloring.rs b/cranelift/codegen/src/regalloc/coloring.rs
index 4e584b5a64..fa158863a4 100644
--- a/cranelift/codegen/src/regalloc/coloring.rs
+++ b/cranelift/codegen/src/regalloc/coloring.rs
@@ -45,7 +45,7 @@
 use crate::cursor::{Cursor, EncCursor};
 use crate::dominator_tree::DominatorTree;
 use crate::ir::{AbiParam, ArgumentLoc, InstBuilder, ValueDef};
-use crate::ir::{Ebb, Function, Inst, Layout, SigRef, Value, ValueLoc};
+use crate::ir::{Ebb, Function, Inst, InstructionData, Layout, Opcode, SigRef, Value, ValueLoc};
 use crate::isa::{regs_overlap, RegClass, RegInfo, RegUnit};
 use crate::isa::{ConstraintKind, EncInfo, OperandConstraint, RecipeConstraints, TargetIsa};
 use crate::packed_option::PackedOption;
@@ -428,10 +428,26 @@ impl<'a> Context<'a> {
 
         // Finally, we've fully programmed the constraint solver.
         // We expect a quick solution in most cases.
-        let output_regs = self.solver.quick_solve(®s.global).unwrap_or_else(|_| {
-            debug!("quick_solve failed for {}", self.solver);
-            self.iterate_solution(throughs, ®s.global, &mut replace_global_defines)
-        });
+        let is_reload = match &self.cur.func.dfg[inst] {
+            InstructionData::Unary {
+                opcode: Opcode::Fill,
+                arg: _,
+            } => true,
+            _ => false,
+        };
+
+        let output_regs = self
+            .solver
+            .quick_solve(®s.global, is_reload)
+            .unwrap_or_else(|_| {
+                debug!("quick_solve failed for {}", self.solver);
+                self.iterate_solution(
+                    throughs,
+                    ®s.global,
+                    &mut replace_global_defines,
+                    is_reload,
+                )
+            });
 
         // The solution and/or fixed input constraints may require us to shuffle the set of live
         // registers around.
@@ -847,12 +863,13 @@ impl<'a> Context<'a> {
         throughs: &[LiveValue],
         global_regs: &RegisterSet,
         replace_global_defines: &mut bool,
+        is_reload: bool,
     ) -> RegisterSet {
         // Make sure `try_add_var()` below doesn't create a variable with too loose constraints.
         self.program_complete_input_constraints();
 
         loop {
-            match self.solver.real_solve(global_regs) {
+            match self.solver.real_solve(global_regs, is_reload) {
                 Ok(regs) => return regs,
                 Err(SolverError::Divert(rc)) => {
                     // Do we have any live-through `rc` registers that are not already variables?
diff --git a/cranelift/codegen/src/regalloc/register_set.rs b/cranelift/codegen/src/regalloc/register_set.rs
index c9b419a517..fb7f208c59 100644
--- a/cranelift/codegen/src/regalloc/register_set.rs
+++ b/cranelift/codegen/src/regalloc/register_set.rs
@@ -126,6 +126,7 @@ impl RegisterSet {
 }
 
 /// Iterator over available registers in a register class.
+#[derive(Clone)]
 pub struct RegSetIter {
     regs: RegUnitMask,
 }
@@ -161,6 +162,31 @@ impl Iterator for RegSetIter {
     }
 }
 
+impl RegSetIter {
+    pub fn rnext(&mut self) -> Option {
+        let num_words = self.regs.len();
+        let bits_per_word = 8 * size_of_val(&self.regs[0]);
+
+        // Find the last set bit in `self.regs`.
+        for i in 0..num_words {
+            let word_ix = num_words - 1 - i;
+
+            let word = &mut self.regs[word_ix];
+            if *word != 0 {
+                let lzeroes = word.leading_zeros() as usize;
+
+                // Clear that highest bit so we won't find it again.
+                *word &= !(1 << (bits_per_word - 1 - lzeroes));
+
+                return Some((word_ix * bits_per_word + bits_per_word - 1 - lzeroes) as RegUnit);
+            }
+        }
+
+        // All of `self.regs` is 0.
+        None
+    }
+}
+
 impl ExactSizeIterator for RegSetIter {}
 
 /// Displaying an `RegisterSet` correctly requires the associated `RegInfo` from the target ISA.
@@ -261,6 +287,45 @@ mod tests {
         classes: &[],
     };
 
+    const RSI_1: RegSetIter = RegSetIter {
+        regs: [0x31415927, 0x27182818, 0x14141356],
+    };
+
+    const RSI_2: RegSetIter = RegSetIter {
+        regs: [0x00000000, 0x00000000, 0x00000000],
+    };
+
+    const RSI_3: RegSetIter = RegSetIter {
+        regs: [0xffffffff, 0xffffffff, 0xffffffff],
+    };
+
+    fn reverse_regset_iteration_work(rsi: &RegSetIter) {
+        // Check the reverse iterator by comparing its output with the forward iterator.
+        let rsi_f = (*rsi).clone();
+        let results_f = rsi_f.collect::>();
+
+        let mut rsi_r = (*rsi).clone();
+        let mut results_r = Vec::::new();
+        while let Some(r) = rsi_r.rnext() {
+            results_r.push(r);
+        }
+
+        let len_f = results_f.len();
+        let len_r = results_r.len();
+        assert_eq!(len_f, len_r);
+
+        for i in 0..len_f {
+            assert_eq!(results_f[i], results_r[len_f - 1 - i]);
+        }
+    }
+
+    #[test]
+    fn reverse_regset_iteration() {
+        reverse_regset_iteration_work(&RSI_1);
+        reverse_regset_iteration_work(&RSI_2);
+        reverse_regset_iteration_work(&RSI_3);
+    }
+
     #[test]
     fn put_and_take() {
         let mut regs = RegisterSet::new();
diff --git a/cranelift/codegen/src/regalloc/solver.rs b/cranelift/codegen/src/regalloc/solver.rs
index 0d6a816dcb..ec517028cc 100644
--- a/cranelift/codegen/src/regalloc/solver.rs
+++ b/cranelift/codegen/src/regalloc/solver.rs
@@ -852,8 +852,12 @@ impl Solver {
     /// always trivial.
     ///
     /// Returns `Ok(regs)` if a solution was found.
-    pub fn quick_solve(&mut self, global_regs: &RegisterSet) -> Result {
-        self.find_solution(global_regs)
+    pub fn quick_solve(
+        &mut self,
+        global_regs: &RegisterSet,
+        is_reload: bool,
+    ) -> Result {
+        self.find_solution(global_regs, is_reload)
     }
 
     /// Try harder to find a solution.
@@ -863,7 +867,11 @@ impl Solver {
     /// This may return an error with a register class that has run out of registers. If registers
     /// can be freed up in the starving class, this method can be called again after adding
     /// variables for the freed registers.
-    pub fn real_solve(&mut self, global_regs: &RegisterSet) -> Result {
+    pub fn real_solve(
+        &mut self,
+        global_regs: &RegisterSet,
+        is_reload: bool,
+    ) -> Result {
         // Compute domain sizes for all the variables given the current register sets.
         for v in &mut self.vars {
             let d = v.iter(&self.regs_in, &self.regs_out, global_regs).len();
@@ -901,7 +909,7 @@ impl Solver {
         });
 
         debug!("real_solve for {}", self);
-        self.find_solution(global_regs)
+        self.find_solution(global_regs, is_reload)
     }
 
     /// Search for a solution with the current list of variables.
@@ -909,7 +917,11 @@ impl Solver {
     /// If a solution was found, returns `Ok(regs)` with the set of available registers on the
     /// output side after the solution. If no solution could be found, returns `Err(rc)` with the
     /// constraint register class that needs more available registers.
-    fn find_solution(&mut self, global_regs: &RegisterSet) -> Result {
+    fn find_solution(
+        &mut self,
+        global_regs: &RegisterSet,
+        is_reload: bool,
+    ) -> Result {
         // Available registers on the input and output sides respectively.
         let mut iregs = self.regs_in.clone();
         let mut oregs = self.regs_out.clone();
@@ -917,7 +929,20 @@ impl Solver {
 
         for v in &mut self.vars {
             let rc = v.constraint;
-            let reg = match v.iter(&iregs, &oregs, &gregs).next() {
+
+            // Decide which register to assign.  In order to try and keep registers holding
+            // reloaded values separate from all other registers to the extent possible, we choose
+            // the first available register in the normal case, but the last available one in the
+            // case of a reload.  See "A side note on register choice heuristics" in
+            // src/redundant_reload_remover.rs for further details.
+            let mut reg_set_iter = v.iter(&iregs, &oregs, &gregs);
+            let maybe_reg = if is_reload {
+                reg_set_iter.rnext()
+            } else {
+                reg_set_iter.next()
+            };
+
+            let reg = match maybe_reg {
                 Some(reg) => reg,
                 None => {
                     // If `v` must avoid global interference, there is not point in requesting
@@ -1207,7 +1232,7 @@ mod tests {
         solver.reset(®s);
         solver.reassign_in(v10, gpr, r1, r0);
         solver.inputs_done();
-        assert!(solver.quick_solve(&gregs).is_ok());
+        assert!(solver.quick_solve(&gregs, false).is_ok());
         assert_eq!(solver.schedule_moves(®s), 0);
         assert_eq!(solver.moves(), &[mov(v10, gpr, r1, r0)]);
 
@@ -1217,7 +1242,7 @@ mod tests {
         solver.reassign_in(v10, gpr, r0, r1);
         solver.reassign_in(v11, gpr, r1, r2);
         solver.inputs_done();
-        assert!(solver.quick_solve(&gregs).is_ok());
+        assert!(solver.quick_solve(&gregs, false).is_ok());
         assert_eq!(solver.schedule_moves(®s), 0);
         assert_eq!(
             solver.moves(),
@@ -1229,7 +1254,7 @@ mod tests {
         solver.reassign_in(v10, gpr, r0, r1);
         solver.reassign_in(v11, gpr, r1, r0);
         solver.inputs_done();
-        assert!(solver.quick_solve(&gregs).is_ok());
+        assert!(solver.quick_solve(&gregs, false).is_ok());
         assert_eq!(solver.schedule_moves(®s), 0);
         assert_eq!(
             solver.moves(),
@@ -1269,7 +1294,7 @@ mod tests {
         solver.reassign_in(v11, s, s2, s0);
         solver.reassign_in(v12, s, s3, s1);
         solver.inputs_done();
-        assert!(solver.quick_solve(&gregs).is_ok());
+        assert!(solver.quick_solve(&gregs, false).is_ok());
         assert_eq!(solver.schedule_moves(®s), 0);
         assert_eq!(
             solver.moves(),
@@ -1290,7 +1315,7 @@ mod tests {
         solver.reassign_in(v12, s, s1, s3);
         solver.reassign_in(v10, d, d1, d0);
         solver.inputs_done();
-        assert!(solver.quick_solve(&gregs).is_ok());
+        assert!(solver.quick_solve(&gregs, false).is_ok());
         assert_eq!(solver.schedule_moves(®s), 0);
         assert_eq!(
             solver.moves(),
@@ -1335,7 +1360,7 @@ mod tests {
         solver.reassign_in(v11, gpr, r1, r2);
         solver.reassign_in(v12, gpr, r2, r0);
         solver.inputs_done();
-        assert!(solver.quick_solve(&gregs).is_ok());
+        assert!(solver.quick_solve(&gregs, false).is_ok());
         assert_eq!(solver.schedule_moves(®s), 1);
         assert_eq!(
             solver.moves(),
@@ -1359,7 +1384,7 @@ mod tests {
         solver.reassign_in(v15, gpr, r5, r3);
 
         solver.inputs_done();
-        assert!(solver.quick_solve(&gregs).is_ok());
+        assert!(solver.quick_solve(&gregs, false).is_ok());
         // We resolve two cycles with one spill.
         assert_eq!(solver.schedule_moves(®s), 1);
         assert_eq!(
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 39055b1232..84acd0bd09 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -697,6 +697,7 @@ impl<'a> Verifier<'a> {
             | Store { .. }
             | RegMove { .. }
             | CopySpecial { .. }
+            | CopyToSsa { .. }
             | Trap { .. }
             | CondTrap { .. }
             | IntCondTrap { .. }
diff --git a/cranelift/codegen/src/write.rs b/cranelift/codegen/src/write.rs
index cf44fa8aca..4fe2b67f3c 100644
--- a/cranelift/codegen/src/write.rs
+++ b/cranelift/codegen/src/write.rs
@@ -664,6 +664,14 @@ pub fn write_operands(
                 write!(w, " %{} -> %{}", src, dst)
             }
         }
+        CopyToSsa { src, .. } => {
+            if let Some(isa) = isa {
+                let regs = isa.register_info();
+                write!(w, " {}", regs.display_regunit(src))
+            } else {
+                write!(w, " %{}", src)
+            }
+        }
         RegSpill { arg, src, dst, .. } => {
             if let Some(isa) = isa {
                 let regs = isa.register_info();
diff --git a/cranelift/entity/src/set.rs b/cranelift/entity/src/set.rs
index f89e96538a..a4759a1712 100644
--- a/cranelift/entity/src/set.rs
+++ b/cranelift/entity/src/set.rs
@@ -45,9 +45,27 @@ where
 
     /// Is this set completely empty?
     pub fn is_empty(&self) -> bool {
+        // Note that this implementation will become incorrect should it ever become possible
+        // to remove elements from an `EntitySet`.
         self.len == 0
     }
 
+    /// Returns the cardinality of the set.  More precisely, it returns the number of calls to
+    /// `insert` with different key values, that have happened since the the set was most recently
+    /// `clear`ed or created with `new`.
+    pub fn cardinality(&self) -> usize {
+        let mut n: usize = 0;
+        for byte_ix in 0..self.len / 8 {
+            n += self.elems[byte_ix].count_ones() as usize;
+        }
+        for bit_ix in (self.len / 8) * 8..self.len {
+            if (self.elems[bit_ix / 8] & (1 << (bit_ix % 8))) != 0 {
+                n += 1;
+            }
+        }
+        n
+    }
+
     /// Remove all entries from this set.
     pub fn clear(&mut self) {
         self.len = 0;
diff --git a/cranelift/filetests/filetests/safepoint/call.clif b/cranelift/filetests/filetests/safepoint/call.clif
index 4b9de997e0..489ebd0438 100644
--- a/cranelift/filetests/filetests/safepoint/call.clif
+++ b/cranelift/filetests/filetests/safepoint/call.clif
@@ -41,6 +41,7 @@ ebb1:
 ; nextln:   v3 = spill v10
 ; nextln:   brz v2, ebb1
 ; nextln:   v11 = fill v1
+; nextln:   regmove v11, %r15 -> %rax
 ; nextln:   return v11
 ; nextln: 
 ; nextln: ebb1:
@@ -48,5 +49,6 @@ ebb1:
 ; nextln:   safepoint v3
 ; nextln:   v4 = call_indirect sig1, v8()
 ; nextln:   v12 = fill.r64 v3
+; nextln:   regmove v12, %r15 -> %rax
 ; nextln:   return v12
 ; nextln: }
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 3a5906d5be..5bcc87de0c 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -2498,6 +2498,10 @@ impl<'a> Parser<'a> {
                 let dst = self.match_regunit(ctx.unique_isa)?;
                 InstructionData::CopySpecial { opcode, src, dst }
             }
+            InstructionFormat::CopyToSsa => InstructionData::CopyToSsa {
+                opcode,
+                src: self.match_regunit(ctx.unique_isa)?,
+            },
             InstructionFormat::RegSpill => {
                 let arg = self.match_value("expected SSA value operand")?;
                 self.match_token(Token::Comma, "expected ',' between operands")?;
diff --git a/cranelift/serde/src/serde_clif_json.rs b/cranelift/serde/src/serde_clif_json.rs
index 50655b732d..0a66cd710c 100644
--- a/cranelift/serde/src/serde_clif_json.rs
+++ b/cranelift/serde/src/serde_clif_json.rs
@@ -210,6 +210,10 @@ pub enum SerInstData {
         src: String,
         dst: String,
     },
+    CopyToSsa {
+        opcode: String,
+        src: String,
+    },
     RegSpill {
         opcode: String,
         arg: String,
@@ -651,6 +655,10 @@ pub fn get_inst_data(inst_index: Inst, func: &Function) -> SerInstData {
             src: src.to_string(),
             dst: dst.to_string(),
         },
+        InstructionData::CopyToSsa { opcode, src } => SerInstData::CopyToSsa {
+            opcode: opcode.to_string(),
+            src: src.to_string(),
+        },
         InstructionData::RegSpill {
             opcode,
             arg,

From ec8f72bf2022240b93700a84c494ed46bf0a66e0 Mon Sep 17 00:00:00 2001
From: Ujjwal Sharma 
Date: Mon, 26 Aug 2019 17:07:27 +0530
Subject: [PATCH 2596/3084] Use roundss/roundsd when available for
 Ceil/Floor/Trunc/Nearest (#931)

Don't tie the preexisting SIMD ISA predicates to the shared
enable_simd setting but make new ones instead.

Fixes: https://github.com/CraneStation/cranelift/issues/908
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 19 ++++++++++---------
 .../codegen/meta/src/isa/x86/settings.rs      | 17 ++++++++++++++---
 .../filetests/isa/x86/binary32-float.clif     |  1 -
 .../filetests/isa/x86/binary64-float.clif     |  1 -
 4 files changed, 24 insertions(+), 14 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 818e07bd0b..9ce33817c7 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -589,8 +589,9 @@ pub fn define(
     let use_popcnt = settings.predicate_by_name("use_popcnt");
     let use_lzcnt = settings.predicate_by_name("use_lzcnt");
     let use_bmi1 = settings.predicate_by_name("use_bmi1");
-    let use_ssse3 = settings.predicate_by_name("use_ssse3");
     let use_sse41 = settings.predicate_by_name("use_sse41");
+    let use_ssse3_simd = settings.predicate_by_name("use_ssse3_simd");
+    let use_sse41_simd = settings.predicate_by_name("use_sse41_simd");
 
     // Definitions.
     let mut e = PerCpuModeEncodings::new();
@@ -1694,8 +1695,8 @@ pub fn define(
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
         let instruction = x86_pshufb.bind_vector_from_lane(ty, sse_vector_size);
         let template = rec_fa.nonrex().opcodes(vec![0x66, 0x0f, 0x38, 00]);
-        e.enc32_isap(instruction.clone(), template.clone(), use_ssse3);
-        e.enc64_isap(instruction, template, use_ssse3);
+        e.enc32_isap(instruction.clone(), template.clone(), use_ssse3_simd);
+        e.enc64_isap(instruction, template, use_ssse3_simd);
     }
 
     // PSHUFD, 32-bit shuffle using one XMM register and a u8 immediate
@@ -1726,10 +1727,10 @@ pub fn define(
     // SIMD insertlane
     let mut insertlane_mapping: HashMap, Option)> =
         HashMap::new();
-    insertlane_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x20], Some(use_sse41))); // PINSRB
+    insertlane_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x20], Some(use_sse41_simd))); // PINSRB
     insertlane_mapping.insert(16, (vec![0x66, 0x0f, 0xc4], None)); // PINSRW from SSE2
-    insertlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41))); // PINSRD
-    insertlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41))); // PINSRQ, only x86_64
+    insertlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41_simd))); // PINSRD
+    insertlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41_simd))); // PINSRQ, only x86_64
 
     for ty in ValueType::all_lane_types() {
         if let Some((opcode, isap)) = insertlane_mapping.get(&ty.lane_bits()) {
@@ -1747,10 +1748,10 @@ pub fn define(
     // SIMD extractlane
     let mut extractlane_mapping: HashMap, Option)> =
         HashMap::new();
-    extractlane_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x14], Some(use_sse41))); // PEXTRB
+    extractlane_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x14], Some(use_sse41_simd))); // PEXTRB
     extractlane_mapping.insert(16, (vec![0x66, 0x0f, 0xc5], None)); // PEXTRW from zSSE2, SSE4.1 has a PEXTRW that can move to reg/m16 but the opcode is four bytes
-    extractlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41))); // PEXTRD
-    extractlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41))); // PEXTRQ, only x86_64
+    extractlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41_simd))); // PEXTRD
+    extractlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41_simd))); // PEXTRQ, only x86_64
 
     for ty in ValueType::all_lane_types() {
         if let Some((opcode, isap)) = extractlane_mapping.get(&ty.lane_bits()) {
diff --git a/cranelift/codegen/meta/src/isa/x86/settings.rs b/cranelift/codegen/meta/src/isa/x86/settings.rs
index 3a42553386..10cb516207 100644
--- a/cranelift/codegen/meta/src/isa/x86/settings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/settings.rs
@@ -32,12 +32,23 @@ pub fn define(shared: &SettingGroup) -> SettingGroup {
 
     let shared_enable_simd = shared.get_bool("enable_simd");
 
-    settings.add_predicate("use_ssse3", predicate!(shared_enable_simd && has_ssse3));
-    settings.add_predicate("use_sse41", predicate!(shared_enable_simd && has_sse41));
+    settings.add_predicate("use_ssse3", predicate!(has_ssse3));
+    settings.add_predicate("use_sse41", predicate!(has_sse41));
+    settings.add_predicate("use_sse42", predicate!(has_sse41 && has_sse42));
+
     settings.add_predicate(
-        "use_sse42",
+        "use_ssse3_simd",
+        predicate!(shared_enable_simd && has_ssse3),
+    );
+    settings.add_predicate(
+        "use_sse41_simd",
+        predicate!(shared_enable_simd && has_sse41),
+    );
+    settings.add_predicate(
+        "use_sse42_simd",
         predicate!(shared_enable_simd && has_sse41 && has_sse42),
     );
+
     settings.add_predicate("use_popcnt", predicate!(has_popcnt && has_sse42));
     settings.add_predicate("use_bmi1", predicate!(has_bmi1));
     settings.add_predicate("use_lzcnt", predicate!(has_lzcnt));
diff --git a/cranelift/filetests/filetests/isa/x86/binary32-float.clif b/cranelift/filetests/filetests/isa/x86/binary32-float.clif
index c6092e2e22..8a4ae5fe7e 100644
--- a/cranelift/filetests/filetests/isa/x86/binary32-float.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary32-float.clif
@@ -1,6 +1,5 @@
 ; Binary emission of 32-bit floating point code.
 test binemit
-set enable_simd
 target i686 haswell
 
 ; The binary encodings can be verified with the command:
diff --git a/cranelift/filetests/filetests/isa/x86/binary64-float.clif b/cranelift/filetests/filetests/isa/x86/binary64-float.clif
index 8c34299705..59e024a042 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64-float.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64-float.clif
@@ -1,7 +1,6 @@
 ; Binary emission of 64-bit floating point code.
 test binemit
 set opt_level=best
-set enable_simd
 target x86_64 haswell
 
 ; The binary encodings can be verified with the command:

From 9edbfed65f5580520df74a87139bd2d61dde4d4f Mon Sep 17 00:00:00 2001
From: Till Schneidereit 
Date: Mon, 26 Aug 2019 13:12:55 +0200
Subject: [PATCH 2597/3084] Switch from mmap to memmap to support Windows

---
 cranelift/filetests/Cargo.toml             |  2 +-
 cranelift/filetests/src/function_runner.rs | 18 +++++++++++-------
 2 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index f5d3220ed9..2b92833070 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -17,6 +17,6 @@ cranelift-preopt = { path = "../cranelift-preopt", version = "0.40.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
-mmap = "0.1.1"
+memmap = "0.7.0"
 num_cpus = "1.8.0"
 region = "2.1.2"
diff --git a/cranelift/filetests/src/function_runner.rs b/cranelift/filetests/src/function_runner.rs
index dee7320384..954306f5d0 100644
--- a/cranelift/filetests/src/function_runner.rs
+++ b/cranelift/filetests/src/function_runner.rs
@@ -4,7 +4,7 @@ use cranelift_codegen::ir::Function;
 use cranelift_codegen::isa::{CallConv, TargetIsa};
 use cranelift_codegen::{settings, Context};
 use cranelift_native::builder as host_isa_builder;
-use mmap::{MapOption, MemoryMap};
+use memmap::MmapMut;
 use region;
 use region::Protection;
 
@@ -67,19 +67,23 @@ impl FunctionRunner {
         let code_info = context
             .compile(self.isa.as_ref())
             .map_err(|e| e.to_string())?;
-        let code_page = MemoryMap::new(code_info.total_size as usize, &[MapOption::MapWritable])
-            .map_err(|e| e.to_string())?;
+        let mut code_page =
+            MmapMut::map_anon(code_info.total_size as usize).map_err(|e| e.to_string())?;
         let callable_fn: fn() -> bool = unsafe {
             context.emit_to_memory(
                 self.isa.as_ref(),
-                code_page.data(),
+                code_page.as_mut_ptr(),
                 relocs,
                 traps,
                 stackmaps,
             );
-            region::protect(code_page.data(), code_page.len(), Protection::ReadExecute)
-                .map_err(|e| e.to_string())?;
-            mem::transmute(code_page.data())
+            region::protect(
+                code_page.as_mut_ptr(),
+                code_page.len(),
+                Protection::ReadExecute,
+            )
+            .map_err(|e| e.to_string())?;
+            mem::transmute(code_page.as_mut_ptr())
         };
 
         // execute

From 67a995f10789c1ba624cfbb8ae4e9a98e4dd4bf4 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 23 Aug 2019 12:39:33 +0200
Subject: [PATCH 2598/3084] [clif-util] Use a simple cfg guard instead of
 cfg_if for the wasm module;

---
 cranelift/src/clif-util.rs | 11 +++--------
 cranelift/src/wasm.rs      | 39 ++++++++++++++++++++++++++++----------
 2 files changed, 32 insertions(+), 18 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 51bac2a533..8077cf9b23 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -13,14 +13,6 @@
     )
 )]
 
-use cfg_if::cfg_if;
-
-cfg_if! {
-    if #[cfg(feature = "wasm")] {
-        mod wasm;
-    }
-}
-
 use clap::{App, Arg, SubCommand};
 use cranelift_codegen::dbg::LOG_FILENAME_PREFIX;
 use cranelift_codegen::VERSION;
@@ -36,6 +28,9 @@ mod print_cfg;
 mod run;
 mod utils;
 
+#[cfg(feature = "wasm")]
+mod wasm;
+
 /// A command either succeeds or fails with an error message.
 pub type CommandResult = Result<(), String>;
 
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index f58c5ae863..f6b2036d57 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -7,13 +7,13 @@
     allow(clippy::too_many_arguments, clippy::cyclomatic_complexity)
 )]
 
-use crate::disasm::{print_all, PrintRelocs, PrintTraps, PrintStackmaps};
+use crate::disasm::{print_all, PrintRelocs, PrintStackmaps, PrintTraps};
 use crate::utils::{parse_sets_and_triple, read_to_end};
+use cranelift_codegen::ir::DisplayFunctionAnnotations;
 use cranelift_codegen::print_errors::{pretty_error, pretty_verifier_error};
 use cranelift_codegen::settings::FlagsOrIsa;
 use cranelift_codegen::timing;
 use cranelift_codegen::Context;
-use cranelift_codegen::ir::DisplayFunctionAnnotations;
 use cranelift_entity::EntityRef;
 use cranelift_wasm::{translate_module, DummyEnvironment, FuncIndex, ReturnMode};
 use std::path::Path;
@@ -113,7 +113,8 @@ fn handle_module(
     };
 
     let debug_info = flag_calc_value_ranges;
-    let mut dummy_environ = DummyEnvironment::new(isa.frontend_config(), ReturnMode::NormalReturns, debug_info);
+    let mut dummy_environ =
+        DummyEnvironment::new(isa.frontend_config(), ReturnMode::NormalReturns, debug_info);
     translate_module(&module_binary, &mut dummy_environ).map_err(|e| e.to_string())?;
 
     let _ = terminal.fg(term::color::GREEN);
@@ -195,7 +196,10 @@ fn handle_module(
             }
 
             if flag_print_disasm {
-                saved_sizes = Some((code_info.code_size, code_info.jumptables_size + code_info.rodata_size));
+                saved_sizes = Some((
+                    code_info.code_size,
+                    code_info.jumptables_size + code_info.rodata_size,
+                ));
             }
         }
 
@@ -212,19 +216,34 @@ fn handle_module(
                 println!("; Exported as \"{}\"", export_name);
             }
             let value_ranges = if flag_calc_value_ranges {
-                Some(context.build_value_labels_ranges(isa).expect("value location ranges"))
+                Some(
+                    context
+                        .build_value_labels_ranges(isa)
+                        .expect("value location ranges"),
+                )
             } else {
                 None
             };
-            println!("{}", context.func.display_with(DisplayFunctionAnnotations {
-                isa: fisa.isa,
-                value_ranges: value_ranges.as_ref(),
-            }));
+            println!(
+                "{}",
+                context.func.display_with(DisplayFunctionAnnotations {
+                    isa: fisa.isa,
+                    value_ranges: value_ranges.as_ref(),
+                })
+            );
             vprintln!(flag_verbose, "");
         }
 
         if let Some((code_size, rodata_size)) = saved_sizes {
-            print_all(isa, &mem, code_size, rodata_size, &relocs, &traps, &stackmaps)?;
+            print_all(
+                isa,
+                &mem,
+                code_size,
+                rodata_size,
+                &relocs,
+                &traps,
+                &stackmaps,
+            )?;
         }
 
         context.clear();

From ea9ee202bb9f54dfb09d35ce67559708e348a91a Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 23 Jul 2019 10:18:50 -0700
Subject: [PATCH 2599/3084] Clear jump tables when function data is cleared

---
 cranelift/codegen/src/ir/function.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index 8eea542564..1dcdeee91c 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -113,6 +113,7 @@ impl Function {
         self.encodings.clear();
         self.locations.clear();
         self.offsets.clear();
+        self.jt_offsets.clear();
         self.srclocs.clear();
     }
 

From c20b13d5a9dd12fef3fa0976bca8f359700f8ffb Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 23 Jul 2019 10:38:29 -0700
Subject: [PATCH 2600/3084] Add ConstantPool

---
 cranelift/codegen/src/ir/constant.rs | 218 +++++++++++++++++++++++++++
 cranelift/codegen/src/ir/dfg.rs      |   7 +-
 cranelift/codegen/src/ir/entities.rs |  26 +++-
 cranelift/codegen/src/ir/mod.rs      |   4 +-
 4 files changed, 252 insertions(+), 3 deletions(-)
 create mode 100644 cranelift/codegen/src/ir/constant.rs

diff --git a/cranelift/codegen/src/ir/constant.rs b/cranelift/codegen/src/ir/constant.rs
new file mode 100644
index 0000000000..c44dcd3379
--- /dev/null
+++ b/cranelift/codegen/src/ir/constant.rs
@@ -0,0 +1,218 @@
+//! Constants
+//!
+//! The constant pool defined here allows cranelift to avoid emitting the same constant multiple
+//! times. As constants are inserted in the pool, a handle is returned; the handle is a cranelift
+//! Entity. Inserting the same data multiple times will always return the same handle. Future work
+//! could include: ensuring alignment of constants within the pool, bucketing constants by size.
+
+use crate::ir::Constant;
+use cranelift_entity::EntityRef;
+use std::collections::{BTreeMap, HashMap};
+use std::vec::Vec;
+
+/// This type describes the actual constant data.
+pub type ConstantData = Vec;
+
+/// This type describes an offset in bytes within a constant pool.
+pub type ConstantOffset = u32;
+
+/// Inner type for storing data and offset together in the constant pool. The offset is optional
+/// because it must be set relative to the function code size (i.e. constants are emitted after the
+/// function body); because the function is not yet compiled when constants are inserted,
+/// [set_offset](ir::ConstantPool::set_offset) must be called once a constant's offset from the
+/// beginning of the function is known (see [relaxation.rs](binemit::relaxation)).
+#[derive(Clone)]
+pub struct ConstantPoolEntry {
+    data: ConstantData,
+    offset: Option,
+}
+
+impl ConstantPoolEntry {
+    fn new(data: ConstantData) -> Self {
+        ConstantPoolEntry { data, offset: None }
+    }
+
+    /// Return the size of the constant at this entry.
+    pub fn len(&self) -> usize {
+        self.data.len()
+    }
+
+    /// Assign a new offset to the constant at this entry.
+    pub fn set_offset(&mut self, offset: ConstantOffset) {
+        self.offset = Some(offset)
+    }
+}
+
+/// Maintains the mapping between a constant handle (i.e. [Constant](ir::entities::Constant)) and
+/// its constant data (i.e. [ConstantData](ir::constant::ConstantData)).
+#[derive(Clone)]
+pub struct ConstantPool {
+    /// This mapping maintains the insertion order as long as Constants are created with sequentially increasing integers.
+    handles_to_values: BTreeMap,
+    /// This mapping is unordered (no need for lexicographic ordering) but allows us to map constant data back to handles.
+    values_to_handles: HashMap,
+}
+
+impl ConstantPool {
+    /// Create a new constant pool instance.
+    pub fn new() -> Self {
+        ConstantPool {
+            handles_to_values: BTreeMap::new(),
+            values_to_handles: HashMap::new(),
+        }
+    }
+
+    /// Empty the constant pool of all data.
+    pub fn clear(&mut self) {
+        self.handles_to_values.clear();
+        self.values_to_handles.clear();
+    }
+
+    /// Insert constant data into the pool, returning a handle for later referencing; when constant
+    /// data is inserted that is a duplicate of previous constant data, the existing handle will be
+    /// returned.
+    pub fn insert(&mut self, constant_value: ConstantData) -> Constant {
+        if self.values_to_handles.contains_key(&constant_value) {
+            self.values_to_handles
+                .get(&constant_value)
+                .expect("A constant handle must have a corresponding constant value; this is an implementation error in ConstantPool")
+                .clone()
+        } else {
+            let constant_handle = Constant::new(self.len());
+            self.values_to_handles
+                .insert(constant_value.clone(), constant_handle.clone());
+            self.handles_to_values.insert(
+                constant_handle.clone(),
+                ConstantPoolEntry::new(constant_value),
+            );
+            constant_handle
+        }
+    }
+
+    /// Retrieve the constant data given a handle.
+    pub fn get(&self, constant_handle: Constant) -> &ConstantData {
+        assert!(self.handles_to_values.contains_key(&constant_handle));
+        &self.handles_to_values
+            .get(&constant_handle)
+            .expect("A constant handle must have a corresponding constant value; was a constant handle created outside of the pool?")
+            .data
+    }
+
+    /// Assign an offset to a given constant, where the offset is the number of bytes from the
+    /// beginning of the function to the beginning of the constant data inside the pool.
+    pub fn set_offset(&mut self, constant_handle: Constant, constant_offset: ConstantOffset) {
+        assert!(self.handles_to_values.contains_key(&constant_handle), "A constant handle must have already been inserted into the pool; perhaps a constant pool was created outside of the pool?");
+        self.handles_to_values
+            .entry(constant_handle)
+            .and_modify(|e| e.offset = Some(constant_offset));
+    }
+
+    /// Retrieve the offset of a given constant, where the offset is the number of bytes from the
+    /// beginning of the function to the beginning of the constant data inside the pool.
+    pub fn get_offset(&self, constant_handle: Constant) -> ConstantOffset {
+        self.handles_to_values.get(&constant_handle)
+            .expect("A constant handle must have a corresponding constant value; was a constant handle created outside of the pool?")
+            .offset
+            .expect("A constant offset has not yet been set; verify that `set_offset` has been called before this point")
+    }
+
+    /// Iterate over the constants in insertion order.
+    pub fn iter(&self) -> impl Iterator {
+        self.handles_to_values.iter().map(|(h, e)| (h, &e.data))
+    }
+
+    /// Iterate over mutable entries in the constant pool in insertion order.
+    pub fn entries_mut(&mut self) -> impl Iterator {
+        self.handles_to_values.values_mut()
+    }
+
+    /// Return the number of constants in the pool.
+    pub fn len(&self) -> usize {
+        self.handles_to_values.len()
+    }
+
+    /// Return the combined size of all of the constant values in the pool.
+    pub fn byte_size(&self) -> usize {
+        self.values_to_handles.keys().map(|c| c.len()).sum()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn empty() {
+        let sut = ConstantPool::new();
+        assert_eq!(sut.len(), 0);
+    }
+
+    #[test]
+    fn insert() {
+        let mut sut = ConstantPool::new();
+        sut.insert(vec![1, 2, 3]);
+        sut.insert(vec![4, 5, 6]);
+        assert_eq!(sut.len(), 2);
+    }
+
+    #[test]
+    fn insert_duplicate() {
+        let mut sut = ConstantPool::new();
+        let a = sut.insert(vec![1, 2, 3]);
+        sut.insert(vec![4, 5, 6]);
+        let b = sut.insert(vec![1, 2, 3]);
+        assert_eq!(a, b);
+    }
+
+    #[test]
+    fn clear() {
+        let mut sut = ConstantPool::new();
+        sut.insert(vec![1, 2, 3]);
+        assert_eq!(sut.len(), 1);
+
+        sut.clear();
+        assert_eq!(sut.len(), 0);
+    }
+
+    #[test]
+    fn iteration_order() {
+        let mut sut = ConstantPool::new();
+        sut.insert(vec![1, 2, 3]);
+        sut.insert(vec![4, 5, 6]);
+        sut.insert(vec![1, 2, 3]);
+        let data = sut.iter().map(|(_, v)| v).collect::>();
+        assert_eq!(data, vec![&vec![1, 2, 3], &vec![4, 5, 6]]);
+    }
+
+    #[test]
+    fn get() {
+        let mut sut = ConstantPool::new();
+        let data = vec![1, 2, 3];
+        let handle = sut.insert(data.clone());
+        assert_eq!(sut.get(handle), &data);
+    }
+
+    #[test]
+    #[should_panic]
+    fn get_nonexistent_constant() {
+        let sut = ConstantPool::new();
+        let a = Constant::with_number(42).unwrap();
+        sut.get(a); // panics, only use constants returned by ConstantPool
+    }
+
+    #[test]
+    fn get_offset() {
+        let mut sut = ConstantPool::new();
+        let a = sut.insert(vec![1]);
+        sut.set_offset(a, 42);
+        assert_eq!(sut.get_offset(a), 42)
+    }
+
+    #[test]
+    #[should_panic]
+    fn get_nonexistent_offset() {
+        let mut sut = ConstantPool::new();
+        let a = sut.insert(vec![1]);
+        sut.get_offset(a); // panics, set_offset should have been called
+    }
+}
diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index 1467ab23b3..7392e893e7 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -5,7 +5,7 @@ use crate::ir;
 use crate::ir::builder::ReplaceBuilder;
 use crate::ir::extfunc::ExtFuncData;
 use crate::ir::instructions::{BranchInfo, CallInfo, InstructionData};
-use crate::ir::types;
+use crate::ir::{types, ConstantPool};
 use crate::ir::{
     Ebb, FuncRef, Inst, SigRef, Signature, Type, Value, ValueLabelAssignments, ValueList,
     ValueListPool,
@@ -67,6 +67,9 @@ pub struct DataFlowGraph {
 
     /// Saves Value labels.
     pub values_labels: Option>,
+
+    /// Constants used within the function
+    pub constants: ConstantPool,
 }
 
 impl DataFlowGraph {
@@ -81,6 +84,7 @@ impl DataFlowGraph {
             signatures: PrimaryMap::new(),
             ext_funcs: PrimaryMap::new(),
             values_labels: None,
+            constants: ConstantPool::new(),
         }
     }
 
@@ -94,6 +98,7 @@ impl DataFlowGraph {
         self.signatures.clear();
         self.ext_funcs.clear();
         self.values_labels = None;
+        self.constants.clear()
     }
 
     /// Get the total number of instructions created in this function, whether they are currently
diff --git a/cranelift/codegen/src/ir/entities.rs b/cranelift/codegen/src/ir/entities.rs
index 67fd9157e9..ab78e0f3a8 100644
--- a/cranelift/codegen/src/ir/entities.rs
+++ b/cranelift/codegen/src/ir/entities.rs
@@ -104,6 +104,24 @@ impl GlobalValue {
     }
 }
 
+/// An opaque reference to a constant
+#[derive(Copy, Clone, PartialEq, Eq, Hash, Ord, PartialOrd)]
+pub struct Constant(u32);
+entity_impl!(Constant, "const");
+
+impl Constant {
+    /// Create a const reference from its number.
+    ///
+    /// This method is for use by the parser.
+    pub fn with_number(n: u32) -> Option {
+        if n < u32::MAX {
+            Some(Constant(n))
+        } else {
+            None
+        }
+    }
+}
+
 /// An opaque reference to a jump table.
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
 #[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
@@ -195,7 +213,7 @@ impl Table {
     }
 }
 
-/// A reference to any of the entities defined in this module.
+/// A reference to any of the entities defined in this module that can appear in CLIF IR.
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
 pub enum AnyEntity {
     /// The whole function.
@@ -331,4 +349,10 @@ mod tests {
             mem::size_of::>()
         );
     }
+
+    #[test]
+    fn constant_with_number() {
+        assert_eq!(Constant::with_number(0).unwrap().to_string(), "const0");
+        assert_eq!(Constant::with_number(1).unwrap().to_string(), "const1");
+    }
 }
diff --git a/cranelift/codegen/src/ir/mod.rs b/cranelift/codegen/src/ir/mod.rs
index caa1da831a..930f5d496d 100644
--- a/cranelift/codegen/src/ir/mod.rs
+++ b/cranelift/codegen/src/ir/mod.rs
@@ -2,6 +2,7 @@
 
 mod builder;
 pub mod condcodes;
+pub mod constant;
 pub mod dfg;
 pub mod entities;
 mod extfunc;
@@ -27,9 +28,10 @@ mod valueloc;
 use serde::{Deserialize, Serialize};
 
 pub use crate::ir::builder::{InsertBuilder, InstBuilder, InstBuilderBase, InstInserterBase};
+pub use crate::ir::constant::{ConstantData, ConstantOffset, ConstantPool};
 pub use crate::ir::dfg::{DataFlowGraph, ValueDef};
 pub use crate::ir::entities::{
-    Ebb, FuncRef, GlobalValue, Heap, Inst, JumpTable, SigRef, StackSlot, Table, Value,
+    Constant, Ebb, FuncRef, GlobalValue, Heap, Inst, JumpTable, SigRef, StackSlot, Table, Value,
 };
 pub use crate::ir::extfunc::{
     AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData, Signature,

From 7b2d055f786ba4bd62bcfef1b9eb94b44f722255 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 23 Jul 2019 10:47:03 -0700
Subject: [PATCH 2601/3084] Add ability to relocate constants using RelocSink

---
 cranelift/codegen/src/binemit/memorysink.rs | 11 ++++++++++-
 cranelift/codegen/src/binemit/mod.rs        | 13 +++++++++++--
 cranelift/codegen/src/binemit/relaxation.rs |  7 ++++++-
 cranelift/faerie/src/backend.rs             | 12 ++++++++++++
 cranelift/filetests/src/test_binemit.rs     | 12 +++++++++++-
 cranelift/filetests/src/test_compile.rs     |  1 +
 cranelift/simplejit/src/backend.rs          | 12 ++++++++++++
 cranelift/src/disasm.rs                     | 16 ++++++++++++++++
 8 files changed, 79 insertions(+), 5 deletions(-)

diff --git a/cranelift/codegen/src/binemit/memorysink.rs b/cranelift/codegen/src/binemit/memorysink.rs
index 49c519f6b9..ffb8b44b49 100644
--- a/cranelift/codegen/src/binemit/memorysink.rs
+++ b/cranelift/codegen/src/binemit/memorysink.rs
@@ -16,7 +16,7 @@
 use super::{Addend, CodeInfo, CodeOffset, CodeSink, Reloc};
 use crate::binemit::stackmap::Stackmap;
 use crate::ir::entities::Value;
-use crate::ir::{ExternalName, Function, JumpTable, SourceLoc, TrapCode};
+use crate::ir::{ConstantOffset, ExternalName, Function, JumpTable, SourceLoc, TrapCode};
 use crate::isa::TargetIsa;
 use core::ptr::write_unaligned;
 
@@ -78,6 +78,9 @@ pub trait RelocSink {
     /// Add a relocation referencing an external symbol at the current offset.
     fn reloc_external(&mut self, _: CodeOffset, _: Reloc, _: &ExternalName, _: Addend);
 
+    /// Add a relocation referencing a constant.
+    fn reloc_constant(&mut self, _: CodeOffset, _: Reloc, _: ConstantOffset);
+
     /// Add a relocation referencing a jump table.
     fn reloc_jt(&mut self, _: CodeOffset, _: Reloc, _: JumpTable);
 }
@@ -132,6 +135,11 @@ impl<'a> CodeSink for MemoryCodeSink<'a> {
         self.relocs.reloc_external(ofs, rel, name, addend);
     }
 
+    fn reloc_constant(&mut self, rel: Reloc, constant_offset: ConstantOffset) {
+        let ofs = self.offset();
+        self.relocs.reloc_constant(ofs, rel, constant_offset);
+    }
+
     fn reloc_jt(&mut self, rel: Reloc, jt: JumpTable) {
         let ofs = self.offset();
         self.relocs.reloc_jt(ofs, rel, jt);
@@ -169,6 +177,7 @@ pub struct NullRelocSink {}
 impl RelocSink for NullRelocSink {
     fn reloc_ebb(&mut self, _: u32, _: Reloc, _: u32) {}
     fn reloc_external(&mut self, _: u32, _: Reloc, _: &ExternalName, _: i64) {}
+    fn reloc_constant(&mut self, _: CodeOffset, _: Reloc, _: ConstantOffset) {}
     fn reloc_jt(&mut self, _: u32, _: Reloc, _: JumpTable) {}
 }
 
diff --git a/cranelift/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
index ce9321ec19..e5d0cbbec5 100644
--- a/cranelift/codegen/src/binemit/mod.rs
+++ b/cranelift/codegen/src/binemit/mod.rs
@@ -16,7 +16,7 @@ pub use self::relaxation::relax_branches;
 pub use self::shrink::shrink_instructions;
 pub use self::stackmap::Stackmap;
 use crate::ir::entities::Value;
-use crate::ir::{ExternalName, Function, Inst, JumpTable, SourceLoc, TrapCode};
+use crate::ir::{ConstantOffset, ExternalName, Function, Inst, JumpTable, SourceLoc, TrapCode};
 use crate::isa::TargetIsa;
 pub use crate::regalloc::RegDiversions;
 use core::fmt;
@@ -133,6 +133,9 @@ pub trait CodeSink {
     /// Add a relocation referencing an external symbol plus the addend at the current offset.
     fn reloc_external(&mut self, _: Reloc, _: &ExternalName, _: Addend);
 
+    /// Add a relocation referencing a jump table.
+    fn reloc_constant(&mut self, _: Reloc, _: ConstantOffset);
+
     /// Add a relocation referencing a jump table.
     fn reloc_jt(&mut self, _: Reloc, _: JumpTable);
 
@@ -192,7 +195,13 @@ where
     }
 
     sink.begin_rodata();
-    // TODO: No read-only data (constant pools) at this time.
+
+    // output constants
+    for (_, constant_data) in func.dfg.constants.iter() {
+        for byte in constant_data.iter() {
+            sink.put1(*byte)
+        }
+    }
 
     sink.end_codegen();
 }
diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index 262ef986d0..1fdf51ea0e 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -37,6 +37,7 @@ use crate::iterators::IteratorExtras;
 use crate::regalloc::RegDiversions;
 use crate::timing;
 use crate::CodegenResult;
+use core::convert::TryFrom;
 use log::debug;
 
 #[cfg(feature = "basic-blocks")]
@@ -135,7 +136,11 @@ pub fn relax_branches(
     let jumptables_size = offset - jumptables;
     let rodata = offset;
 
-    // TODO: Once we have constant pools we'll do some processing here to update offset.
+    for constant in func.dfg.constants.entries_mut() {
+        constant.set_offset(offset);
+        offset +=
+            u32::try_from(constant.len()).expect("Constants must have a length that fits in a u32")
+    }
 
     let rodata_size = offset - rodata;
 
diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index a33955a212..d605757c4e 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -430,6 +430,18 @@ impl<'a> RelocSink for FaerieRelocSink<'a> {
             }
         }
     }
+
+    fn reloc_constant(&mut self, _offset: CodeOffset, reloc: Reloc, _jt: ir::ConstantOffset) {
+        match reloc {
+            Reloc::X86PCRelRodata4 => {
+                // Not necessary to record this unless we are going to split apart code and its
+                // jumptbl/rodata.
+            }
+            _ => {
+                panic!("Unhandled reloc");
+            }
+        }
+    }
 }
 
 #[allow(dead_code)]
diff --git a/cranelift/filetests/src/test_binemit.rs b/cranelift/filetests/src/test_binemit.rs
index f134806663..75bd86008f 100644
--- a/cranelift/filetests/src/test_binemit.rs
+++ b/cranelift/filetests/src/test_binemit.rs
@@ -90,6 +90,10 @@ impl binemit::CodeSink for TextSink {
         write!(self.text, ") ").unwrap();
     }
 
+    fn reloc_constant(&mut self, reloc: binemit::Reloc, constant: ir::ConstantOffset) {
+        write!(self.text, "{}({}) ", reloc, constant).unwrap();
+    }
+
     fn reloc_jt(&mut self, reloc: binemit::Reloc, jt: ir::JumpTable) {
         write!(self.text, "{}({}) ", reloc, jt).unwrap();
     }
@@ -313,7 +317,13 @@ impl SubTest for TestBinEmit {
         }
 
         sink.begin_rodata();
-        // TODO: Read-only (constant pool) data.
+
+        // output constants
+        for (_, constant_data) in func.dfg.constants.iter() {
+            for byte in constant_data.iter() {
+                sink.put1(*byte)
+            }
+        }
 
         sink.end_codegen();
 
diff --git a/cranelift/filetests/src/test_compile.rs b/cranelift/filetests/src/test_compile.rs
index 706aa00331..10e07440ed 100644
--- a/cranelift/filetests/src/test_compile.rs
+++ b/cranelift/filetests/src/test_compile.rs
@@ -106,6 +106,7 @@ impl binemit::CodeSink for SizeSink {
         _addend: binemit::Addend,
     ) {
     }
+    fn reloc_constant(&mut self, _: binemit::Reloc, _: ir::ConstantOffset) {}
     fn reloc_jt(&mut self, _reloc: binemit::Reloc, _jt: ir::JumpTable) {}
     fn trap(&mut self, _code: ir::TrapCode, _srcloc: ir::SourceLoc) {}
     fn begin_jumptables(&mut self) {}
diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index 0a97df8f46..372f112e8d 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -566,6 +566,18 @@ impl RelocSink for SimpleJITRelocSink {
             }
         }
     }
+
+    fn reloc_constant(&mut self, _offset: CodeOffset, reloc: Reloc, _constant: ir::ConstantOffset) {
+        match reloc {
+            Reloc::X86PCRelRodata4 => {
+                // Not necessary to record this unless we are going to split apart code and its
+                // jumptbl/rodata.
+            }
+            _ => {
+                panic!("Unhandled reloc");
+            }
+        }
+    }
 }
 
 struct SimpleJITStackmapSink {
diff --git a/cranelift/src/disasm.rs b/cranelift/src/disasm.rs
index 5bfa14e313..7a98e02b0b 100644
--- a/cranelift/src/disasm.rs
+++ b/cranelift/src/disasm.rs
@@ -56,6 +56,22 @@ impl binemit::RelocSink for PrintRelocs {
             write!(&mut self.text, "reloc_jt: {} {} at {}\n", r, jt, where_).unwrap();
         }
     }
+
+    fn reloc_constant(
+        &mut self,
+        code_offset: binemit::CodeOffset,
+        reloc: binemit::Reloc,
+        constant: ir::ConstantOffset,
+    ) {
+        if self.flag_print {
+            write!(
+                &mut self.text,
+                "reloc_constant: {} {} at {}\n",
+                reloc, constant, code_offset
+            )
+            .unwrap();
+        }
+    }
 }
 
 pub struct PrintTraps {

From 5ded38ce3e2c0a059521364a68670e9ca3dd3a8e Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 23 Jul 2019 10:52:38 -0700
Subject: [PATCH 2602/3084] Add unsigned 128-bit immediate

---
 cranelift/codegen/src/ir/immediates.rs | 100 +++++++++++++++++++++++++
 1 file changed, 100 insertions(+)

diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index 4a8d5e1fbc..5c36f362ab 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -264,6 +264,74 @@ impl FromStr for Uimm32 {
     }
 }
 
+/// A 128-bit unsigned integer immediate operand.
+///
+/// This is used as an immediate value in SIMD instructions
+#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)]
+pub struct Uimm128(pub [u8; 16]);
+
+impl Display for Uimm128 {
+    // print a 128-bit vector in hexadecimal, e.g. 0x000102030405060708090a0b0c0d0e0f
+    fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+        write!(f, "0x")?;
+        let mut anything_written = false;
+        for &b in self.0.iter() {
+            if b == 0 && !anything_written {
+                continue;
+            } else {
+                anything_written = true;
+                write!(f, "{:02x}", b)?;
+            }
+        }
+        if !anything_written {
+            write!(f, "00")?;
+        }
+        Ok(())
+    }
+}
+
+impl From for Uimm128 {
+    fn from(x: u64) -> Self {
+        let mut buffer: [u8; 16] = [0; 16]; // zero-fill
+        (0..8).for_each(|byte| buffer[15 - byte] = (x >> (byte as u64 * 8) & 0xff) as u8); // insert each byte from the u64 into v after byte position 8
+        Uimm128(buffer)
+    }
+}
+
+impl From<&[u8]> for Uimm128 {
+    fn from(slice: &[u8]) -> Self {
+        assert_eq!(slice.len(), 16);
+        let mut buffer = [0; 16];
+        buffer.copy_from_slice(slice);
+        Uimm128(buffer)
+    }
+}
+
+impl FromStr for Uimm128 {
+    type Err = &'static str;
+
+    // parse a 128-bit vector from a hexadecimal string, formatted as above
+    fn from_str(s: &str) -> Result {
+        if s.len() <= 2 || &s[0..2] != "0x" {
+            Err("Expected a hexadecimal string, e.g. 0x1234")
+        } else if s.len() % 2 != 0 {
+            Err("Hexadecimal string must have an even number of digits")
+        } else if s.len() > 32 {
+            Err("Hexadecimal string has too many digits to fit in a 128-bit vector")
+        } else {
+            let mut buffer = [0; 16]; // zero-fill
+            let start_at = 16 - s.len() / 2;
+            for i in (2..s.len()).step_by(2) {
+                let byte = u8::from_str_radix(&s[i..i + 2], 16)
+                    .or_else(|_| Err("Unable to parse as hexadecimal"))?;
+                let position = start_at + (i / 2);
+                buffer[position] = byte;
+            }
+            Ok(Uimm128(buffer))
+        }
+    }
+}
+
 /// 32-bit signed immediate offset.
 ///
 /// This is used to encode an immediate offset for load/store instructions. All supported ISAs have
@@ -884,6 +952,38 @@ mod tests {
         parse_err::("0x0_0000_0000_0000_0000", "Too many hexadecimal digits");
     }
 
+    #[test]
+    fn format_uimm128() {
+        assert_eq!(Uimm128::from(0).to_string(), "0x00");
+        assert_eq!(Uimm128::from(42).to_string(), "0x2a");
+        assert_eq!(Uimm128::from(3735928559).to_string(), "0xdeadbeef");
+        assert_eq!(
+            Uimm128::from(0x0102030405060708).to_string(),
+            "0x0102030405060708"
+        );
+    }
+
+    #[test]
+    fn parse_uimm128() {
+        parse_ok::("0x00", "0x00");
+        parse_ok::("0x00000042", "0x42");
+        parse_ok::(
+            "0x0102030405060708090a0b0c0d0e0f",
+            "0x0102030405060708090a0b0c0d0e0f",
+        );
+
+        parse_err::("", "Expected a hexadecimal string, e.g. 0x1234");
+        parse_err::("0x", "Expected a hexadecimal string, e.g. 0x1234");
+        parse_err::(
+            "0x042",
+            "Hexadecimal string must have an even number of digits",
+        );
+        parse_err::(
+            "0x00000000000000000000000000000000000000000000000000",
+            "Hexadecimal string has too many digits to fit in a 128-bit vector",
+        );
+    }
+
     #[test]
     fn format_offset32() {
         assert_eq!(Offset32(0).to_string(), "");

From 407d24c0132dcb9d050990bb225ee3cf39c7a1b1 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 23 Jul 2019 11:02:52 -0700
Subject: [PATCH 2603/3084] Add operand kind and format for unsigned 128-bit
 immediates

---
 cranelift/codegen/meta/src/shared/formats.rs  |  2 ++
 .../codegen/meta/src/shared/immediates.rs     | 10 +++++++
 .../codegen/meta/src/shared/instructions.rs   | 17 ++++++++++++
 cranelift/codegen/src/verifier/mod.rs         |  1 +
 cranelift/codegen/src/write.rs                |  6 +++++
 cranelift/reader/src/parser.rs                | 27 ++++++++++++++++++-
 cranelift/serde/src/serde_clif_json.rs        |  9 +++++++
 7 files changed, 71 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/shared/formats.rs b/cranelift/codegen/meta/src/shared/formats.rs
index e6e7c92df7..394eb6d58f 100644
--- a/cranelift/codegen/meta/src/shared/formats.rs
+++ b/cranelift/codegen/meta/src/shared/formats.rs
@@ -5,6 +5,7 @@ pub fn define(immediates: &OperandKinds, entities: &OperandKinds) -> FormatRegis
     // Shorthands for immediates.
     let uimm8 = immediates.by_name("uimm8");
     let uimm32 = immediates.by_name("uimm32");
+    let uimm128 = immediates.by_name("uimm128");
     let imm64 = immediates.by_name("imm64");
     let ieee32 = immediates.by_name("ieee32");
     let ieee64 = immediates.by_name("ieee64");
@@ -30,6 +31,7 @@ pub fn define(immediates: &OperandKinds, entities: &OperandKinds) -> FormatRegis
 
     registry.insert(Builder::new("Unary").value());
     registry.insert(Builder::new("UnaryImm").imm(imm64));
+    registry.insert(Builder::new("UnaryImm128").imm(uimm128));
     registry.insert(Builder::new("UnaryIeee32").imm(ieee32));
     registry.insert(Builder::new("UnaryIeee64").imm(ieee64));
     registry.insert(Builder::new("UnaryBool").imm(boolean));
diff --git a/cranelift/codegen/meta/src/shared/immediates.rs b/cranelift/codegen/meta/src/shared/immediates.rs
index bee762a9e5..5b8baca898 100644
--- a/cranelift/codegen/meta/src/shared/immediates.rs
+++ b/cranelift/codegen/meta/src/shared/immediates.rs
@@ -29,6 +29,16 @@ pub fn define() -> Vec {
         .build();
     kinds.push(uimm32);
 
+    // An unsigned 128-bit immediate integer operand.
+    //
+    // This operand is used to pass entire 128-bit vectors as immediates to
+    // instructions like const.
+    let uimm128 = Builder::new_imm("uimm128")
+        .doc("A 128-bit immediate unsigned integer.")
+        .rust_type("ir::Constant")
+        .build();
+    kinds.push(uimm128);
+
     // A 32-bit immediate signed offset.
     //
     // This is used to represent an immediate address offset in load/store
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 3d3fda49b9..c11016c64d 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -30,6 +30,7 @@ pub fn define(
     let uimm8 = immediates.by_name("uimm8");
     let uimm32 = immediates.by_name("uimm32");
     let imm64 = immediates.by_name("imm64");
+    let uimm128 = immediates.by_name("uimm128");
     let offset32 = immediates.by_name("offset32");
     let memflags = immediates.by_name("memflags");
     let ieee32 = immediates.by_name("ieee32");
@@ -1088,6 +1089,22 @@ pub fn define(
         .operands_out(vec![a]),
     );
 
+    let N = &operand_doc("N", uimm128, "The 16 immediate bytes of a 128-bit vector");
+    let a = &operand_doc("a", TxN, "A constant vector value");
+
+    ig.push(
+        Inst::new(
+            "vconst",
+            r#"
+        SIMD vector constant.
+
+        Construct a vector with the given immediate bytes.
+        "#,
+        )
+        .operands_in(vec![N])
+        .operands_out(vec![a]),
+    );
+
     let a = &operand_doc("a", Ref, "A constant reference null value");
 
     ig.push(
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 84acd0bd09..72fa1f7e17 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -679,6 +679,7 @@ impl<'a> Verifier<'a> {
             // Exhaustive list so we can't forget to add new formats
             Unary { .. }
             | UnaryImm { .. }
+            | UnaryImm128 { .. }
             | UnaryIeee32 { .. }
             | UnaryIeee64 { .. }
             | UnaryBool { .. }
diff --git a/cranelift/codegen/src/write.rs b/cranelift/codegen/src/write.rs
index 4fe2b67f3c..4b616cab2e 100644
--- a/cranelift/codegen/src/write.rs
+++ b/cranelift/codegen/src/write.rs
@@ -5,6 +5,7 @@
 
 use crate::entity::SecondaryMap;
 use crate::ir::entities::AnyEntity;
+use crate::ir::immediates::Uimm128;
 use crate::ir::{
     DataFlowGraph, DisplayFunctionAnnotations, Ebb, Function, Inst, SigRef, Type, Value, ValueDef,
     ValueLoc,
@@ -487,6 +488,11 @@ pub fn write_operands(
     match dfg[inst] {
         Unary { arg, .. } => write!(w, " {}", arg),
         UnaryImm { imm, .. } => write!(w, " {}", imm),
+        UnaryImm128 { imm, .. } => {
+            let data = dfg.constants.get(imm);
+            let uimm128 = Uimm128::from(&data[..]);
+            write!(w, " {}", uimm128)
+        }
         UnaryIeee32 { imm, .. } => write!(w, " {}", imm),
         UnaryIeee64 { imm, .. } => write!(w, " {}", imm),
         UnaryBool { imm, .. } => write!(w, " {}", imm),
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 5bcc87de0c..d503151701 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -9,7 +9,7 @@ use crate::testfile::{Comment, Details, TestFile};
 use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::entities::AnyEntity;
-use cranelift_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32, Uimm64};
+use cranelift_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm128, Uimm32, Uimm64};
 use cranelift_codegen::ir::instructions::{InstructionData, InstructionFormat, VariableArgs};
 use cranelift_codegen::ir::types::INVALID;
 use cranelift_codegen::ir::{
@@ -546,6 +546,23 @@ impl<'a> Parser<'a> {
         }
     }
 
+    // Match and consume a Uimm128 immediate; due to size restrictions on InstructionData, Uimm128 is boxed in cranelift-codegen/meta/src/shared/immediates.rs
+    fn match_uimm128(&mut self, err_msg: &str) -> ParseResult {
+        if let Some(Token::Integer(text)) = self.token() {
+            self.consume();
+            // Lexer just gives us raw text that looks like hex code.
+            // Parse it as an Uimm128 to check for overflow and other issues.
+            text.parse().map_err(|e| {
+                self.error(&format!(
+                    "expected u128 hexadecimal immediate, failed to parse: {}",
+                    e
+                ))
+            })
+        } else {
+            err!(self.loc, err_msg)
+        }
+    }
+
     // Match and consume a Uimm64 immediate.
     fn match_uimm64(&mut self, err_msg: &str) -> ParseResult {
         if let Some(Token::Integer(text)) = self.token() {
@@ -2109,6 +2126,14 @@ impl<'a> Parser<'a> {
                 opcode,
                 imm: self.match_imm64("expected immediate integer operand")?,
             },
+            InstructionFormat::UnaryImm128 => {
+                let uimm128 = self.match_uimm128("expected immediate hexadecimal operand")?;
+                let constant_handle = ctx.function.dfg.constants.insert(uimm128.0.to_vec());
+                InstructionData::UnaryImm128 {
+                    opcode,
+                    imm: constant_handle,
+                }
+            }
             InstructionFormat::UnaryIeee32 => InstructionData::UnaryIeee32 {
                 opcode,
                 imm: self.match_ieee32("expected immediate 32-bit float operand")?,
diff --git a/cranelift/serde/src/serde_clif_json.rs b/cranelift/serde/src/serde_clif_json.rs
index 0a66cd710c..90935e9234 100644
--- a/cranelift/serde/src/serde_clif_json.rs
+++ b/cranelift/serde/src/serde_clif_json.rs
@@ -1,3 +1,4 @@
+use cranelift_codegen::ir::immediates::Uimm128;
 use cranelift_codegen::ir::{Ebb, Function, Inst, InstructionData, Signature};
 use serde_derive::{Deserialize, Serialize};
 
@@ -261,6 +262,14 @@ pub fn get_inst_data(inst_index: Inst, func: &Function) -> SerInstData {
             opcode: opcode.to_string(),
             imm: imm.to_string(),
         },
+        InstructionData::UnaryImm128 { opcode, imm } => {
+            let data = func.dfg.constants.get(imm);
+            let uimm128 = Uimm128::from(&data[..]);
+            SerInstData::UnaryImm {
+                opcode: opcode.to_string(),
+                imm: uimm128.to_string(),
+            }
+        }
         InstructionData::UnaryIeee32 { opcode, imm } => SerInstData::UnaryIeee32 {
             opcode: opcode.to_string(),
             imm: imm.to_string(),

From 684721ca2902c93daa1ff7d80a48f823ba16c719 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 23 Jul 2019 11:04:11 -0700
Subject: [PATCH 2604/3084] Add x86 recipe for vconst

---
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 14 ++++++++++++++
 cranelift/codegen/src/isa/x86/binemit.rs      | 10 +++++++++-
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 50ac9fca7f..d126580804 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -399,6 +399,7 @@ pub fn define<'shared>(
     let f_unary_ieee32 = formats.by_name("UnaryIeee32");
     let f_unary_ieee64 = formats.by_name("UnaryIeee64");
     let f_unary_imm = formats.by_name("UnaryImm");
+    let f_unary_imm128 = formats.by_name("UnaryImm128");
 
     // Predicates shorthands.
     let use_sse41 = settings.predicate_by_name("use_sse41");
@@ -2382,6 +2383,19 @@ pub fn define<'shared>(
             ),
     );
 
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("vconst", f_unary_imm128, 5)
+            .operands_out(vec![fpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(0, out_reg0), sink);
+                    modrm_riprel(out_reg0, sink);
+                    const_disp4(imm, func, sink);
+                "#,
+            ),
+    );
+
     recipes.add_template_recipe(
         EncodingRecipeBuilder::new("jt_base", f_branch_table_base, 5)
             .operands_out(vec![gpr])
diff --git a/cranelift/codegen/src/isa/x86/binemit.rs b/cranelift/codegen/src/isa/x86/binemit.rs
index 8aa51c07b6..afe2c2611f 100644
--- a/cranelift/codegen/src/isa/x86/binemit.rs
+++ b/cranelift/codegen/src/isa/x86/binemit.rs
@@ -4,7 +4,7 @@ use super::enc_tables::{needs_offset, needs_sib_byte};
 use super::registers::RU;
 use crate::binemit::{bad_encoding, CodeSink, Reloc};
 use crate::ir::condcodes::{CondCode, FloatCC, IntCC};
-use crate::ir::{Ebb, Function, Inst, InstructionData, JumpTable, Opcode, TrapCode};
+use crate::ir::{Constant, Ebb, Function, Inst, InstructionData, JumpTable, Opcode, TrapCode};
 use crate::isa::{RegUnit, StackBase, StackBaseMask, StackRef, TargetIsa};
 use crate::regalloc::RegDiversions;
 
@@ -341,3 +341,11 @@ fn jt_disp4(jt: JumpTable, func: &Function, sink: &mut CS
     sink.put4(delta);
     sink.reloc_jt(Reloc::X86PCRelRodata4, jt);
 }
+
+/// Emit a four-byte displacement to `constant`
+fn const_disp4(constant: Constant, func: &Function, sink: &mut CS) {
+    let offset = func.dfg.constants.get_offset(constant);
+    let delta = offset.wrapping_sub(sink.offset() + 4);
+    sink.put4(delta);
+    sink.reloc_constant(Reloc::X86PCRelRodata4, offset);
+}

From 8d812b24cc654b6450fc3e16a4699afbb3501c68 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 23 Jul 2019 11:04:54 -0700
Subject: [PATCH 2605/3084] Add x86 encoding for vconst

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 20 ++++++++++++++++---
 .../filetests/isa/x86/compile-vconst.clif     | 16 +++++++++++++++
 .../filetests/filetests/isa/x86/vconst.clif   | 11 ++++++++++
 3 files changed, 44 insertions(+), 3 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/compile-vconst.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/vconst.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 9ce33817c7..a5df9298e8 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -270,7 +270,7 @@ impl PerCpuModeEncodings {
     /// Add the same encoding to both X86_32 and X86_64; assumes configuration (e.g. REX, operand binding) has already happened
     fn enc_32_64_maybe_isap(
         &mut self,
-        inst: BoundInstruction,
+        inst: impl Clone + Into,
         template: Template,
         isap: Option,
     ) {
@@ -280,7 +280,7 @@ impl PerCpuModeEncodings {
 
     fn enc32_maybe_isap(
         &mut self,
-        inst: BoundInstruction,
+        inst: impl Into,
         template: Template,
         isap: Option,
     ) {
@@ -292,7 +292,7 @@ impl PerCpuModeEncodings {
 
     fn enc64_maybe_isap(
         &mut self,
-        inst: BoundInstruction,
+        inst: impl Into,
         template: Template,
         isap: Option,
     ) {
@@ -432,6 +432,7 @@ pub fn define(
     let uload8_complex = shared.by_name("uload8_complex");
     let ushr = shared.by_name("ushr");
     let ushr_imm = shared.by_name("ushr_imm");
+    let vconst = shared.by_name("vconst");
     let x86_bsf = x86.by_name("x86_bsf");
     let x86_bsr = x86.by_name("x86_bsr");
     let x86_cvtt2si = x86.by_name("x86_cvtt2si");
@@ -578,6 +579,7 @@ pub fn define(
     let rec_urm = r.template("urm");
     let rec_urm_noflags = r.template("urm_noflags");
     let rec_urm_noflags_abcd = r.template("urm_noflags_abcd");
+    let rec_vconst = r.template("vconst");
 
     // Predicates shorthands.
     let all_ones_funcaddrs_and_not_is_pic =
@@ -1785,6 +1787,18 @@ pub fn define(
         }
     }
 
+    // SIMD vconst using MOVUPS
+    // TODO it would be ideal if eventually this became the more efficient MOVAPS but we would have
+    // to guarantee that the constants are aligned when emitted and there is currently no mechanism
+    // for that; alternately, constants could be loaded into XMM registers using a sequence like:
+    // MOVQ + MOVHPD + MOVQ + MOVLPD (this allows the constants to be immediates instead of stored
+    // in memory) but some performance measurements are needed.
+    for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() >= 8) {
+        let instruction = vconst.bind_vector_from_lane(ty, sse_vector_size);
+        let template = rec_vconst.nonrex().opcodes(vec![0x0f, 0x10]);
+        e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
+    }
+
     // Reference type instructions
 
     // Null references implemented as iconst 0.
diff --git a/cranelift/filetests/filetests/isa/x86/compile-vconst.clif b/cranelift/filetests/filetests/isa/x86/compile-vconst.clif
new file mode 100644
index 0000000000..c64c9fc503
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/compile-vconst.clif
@@ -0,0 +1,16 @@
+test compile
+set enable_simd=true
+set probestack_enabled=false
+target x86_64 haswell
+
+; use baldrdash calling convention here for simplicity (avoids prologue, epilogue)
+function %test_vconst_i32() -> i32x4 baldrdash_system_v {
+ebb0:
+    v0 = vconst.i32x4 0x1234
+    return v0
+}
+
+; check:   ebb0:
+; nextln:     v0 = vconst.i32x4 0x1234
+; nextln:     return v0
+; nextln: }
diff --git a/cranelift/filetests/filetests/isa/x86/vconst.clif b/cranelift/filetests/filetests/isa/x86/vconst.clif
new file mode 100644
index 0000000000..f7b9ce4627
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/vconst.clif
@@ -0,0 +1,11 @@
+test binemit
+set opt_level=best
+set enable_simd
+target x86_64
+
+function %test_vconst_b8() {
+ebb0:
+[-, %xmm2]  v0 = vconst.b8x16 0x00  ; bin: 0f 10 15 00000008 PCRelRodata4(15)
+[-, %xmm3]  v1 = vconst.b8x16 0x01  ; bin: 0f 10 1d 00000011 PCRelRodata4(31)
+            return
+}

From c3cc225de97bf04882c563eb584a7c1eaf92b19b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 23 Jul 2019 11:06:08 -0700
Subject: [PATCH 2606/3084] Add filetest for verifying emitted rodata (i.e.
 `test rodata`)

---
 .../filetests/isa/x86/rodata-vconst.clif      |  13 ++
 cranelift/filetests/src/lib.rs                |   2 +
 cranelift/filetests/src/test_rodata.rs        | 123 ++++++++++++++++++
 3 files changed, 138 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/rodata-vconst.clif
 create mode 100644 cranelift/filetests/src/test_rodata.rs

diff --git a/cranelift/filetests/filetests/isa/x86/rodata-vconst.clif b/cranelift/filetests/filetests/isa/x86/rodata-vconst.clif
new file mode 100644
index 0000000000..772cf3b03c
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/rodata-vconst.clif
@@ -0,0 +1,13 @@
+test rodata
+set enable_simd=true
+set probestack_enabled=false
+target x86_64 haswell
+
+; use baldrdash calling convention here for simplicity (avoids prologue, epilogue)
+function %test_vconst_i32() -> i32x4 baldrdash_system_v {
+ebb0:
+    v0 = vconst.i32x4 0x1234
+    return v0
+}
+
+; sameln: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 12, 34]
diff --git a/cranelift/filetests/src/lib.rs b/cranelift/filetests/src/lib.rs
index 08f02afbf5..073fc24492 100644
--- a/cranelift/filetests/src/lib.rs
+++ b/cranelift/filetests/src/lib.rs
@@ -48,6 +48,7 @@ mod test_postopt;
 mod test_preopt;
 mod test_print_cfg;
 mod test_regalloc;
+mod test_rodata;
 mod test_run;
 mod test_safepoint;
 mod test_shrink;
@@ -119,6 +120,7 @@ fn new_subtest(parsed: &TestCommand) -> subtest::SubtestResult test_binemit::subtest(parsed),
         "cat" => test_cat::subtest(parsed),
         "compile" => test_compile::subtest(parsed),
+        "rodata" => test_rodata::subtest(parsed),
         "dce" => test_dce::subtest(parsed),
         "domtree" => test_domtree::subtest(parsed),
         "legalizer" => test_legalizer::subtest(parsed),
diff --git a/cranelift/filetests/src/test_rodata.rs b/cranelift/filetests/src/test_rodata.rs
new file mode 100644
index 0000000000..412fa55ff0
--- /dev/null
+++ b/cranelift/filetests/src/test_rodata.rs
@@ -0,0 +1,123 @@
+//! Test command for verifying the rodata emitted after each function
+//!
+//! The `rodata` test command runs each function through the full code generator pipeline
+
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
+use cranelift_codegen;
+use cranelift_codegen::binemit::{self, CodeInfo};
+use cranelift_codegen::ir;
+use cranelift_codegen::ir::{Function, Value};
+use cranelift_codegen::isa::TargetIsa;
+use cranelift_codegen::print_errors::pretty_error;
+use cranelift_reader::TestCommand;
+use log::info;
+use std::borrow::Cow;
+
+struct TestRodata;
+
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+    assert_eq!(parsed.command, "rodata");
+    if !parsed.options.is_empty() {
+        Err(format!("No options allowed on {}", parsed))
+    } else {
+        Ok(Box::new(TestRodata))
+    }
+}
+
+impl SubTest for TestRodata {
+    fn name(&self) -> &'static str {
+        "rodata"
+    }
+
+    fn is_mutating(&self) -> bool {
+        true
+    }
+
+    fn needs_isa(&self) -> bool {
+        true
+    }
+
+    fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> {
+        let isa = context.isa.expect("rodata needs an ISA");
+        let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned());
+
+        let CodeInfo { total_size, .. } = comp_ctx
+            .compile(isa)
+            .map_err(|e| pretty_error(&comp_ctx.func, context.isa, e))?;
+
+        info!(
+            "Generated {} bytes of code:\n{}",
+            total_size,
+            comp_ctx.func.display(isa)
+        );
+
+        // Verify that the returned code size matches the emitted bytes.
+        let mut sink = RodataSink {
+            rodata: Vec::new(),
+            in_rodata: false,
+        };
+        binemit::emit_function(
+            &comp_ctx.func,
+            |func, inst, div, sink, isa| isa.emit_inst(func, inst, div, sink),
+            &mut sink,
+            isa,
+        );
+
+        // Run final code through filecheck.
+        let text = format!("{:X?}", sink.rodata);
+        info!("Found rodata: {}", text);
+        run_filecheck(&text, context)
+    }
+}
+
+/// Code sink that only captures emitted rodata
+struct RodataSink {
+    in_rodata: bool,
+    rodata: Vec,
+}
+
+impl binemit::CodeSink for RodataSink {
+    fn offset(&self) -> binemit::CodeOffset {
+        0
+    }
+
+    fn put1(&mut self, byte: u8) {
+        if self.in_rodata {
+            self.rodata.push(byte);
+        }
+    }
+
+    fn put2(&mut self, bytes: u16) {
+        if self.in_rodata {
+            self.rodata.extend_from_slice(&bytes.to_be_bytes());
+        }
+    }
+
+    fn put4(&mut self, bytes: u32) {
+        if self.in_rodata {
+            self.rodata.extend_from_slice(&bytes.to_be_bytes());
+        }
+    }
+
+    fn put8(&mut self, bytes: u64) {
+        if self.in_rodata {
+            self.rodata.extend_from_slice(&bytes.to_be_bytes());
+        }
+    }
+
+    fn reloc_ebb(&mut self, _reloc: binemit::Reloc, _ebb_offset: binemit::CodeOffset) {}
+    fn reloc_external(&mut self, _: binemit::Reloc, _: &ir::ExternalName, _: binemit::Addend) {}
+    fn reloc_constant(&mut self, _: binemit::Reloc, _: ir::ConstantOffset) {}
+    fn reloc_jt(&mut self, _reloc: binemit::Reloc, _jt: ir::JumpTable) {}
+    fn trap(&mut self, _code: ir::TrapCode, _srcloc: ir::SourceLoc) {}
+    fn begin_jumptables(&mut self) {
+        assert!(!self.in_rodata);
+    }
+    fn begin_rodata(&mut self) {
+        self.in_rodata = true;
+    }
+    fn end_codegen(&mut self) {
+        assert!(self.in_rodata);
+    }
+    fn add_stackmap(&mut self, _: &[Value], _: &Function, _: &dyn TargetIsa) {}
+}

From cb041407c1bda6d24b055b03ba26e5b3fc5b156c Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 29 Jul 2019 14:56:38 -0700
Subject: [PATCH 2607/3084] Translate existing WASM SIMD operations to CLIF

---
 cranelift/wasm/src/code_translator.rs | 209 +++++++++++++++++++++++---
 1 file changed, 191 insertions(+), 18 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index ad8bbda438..9c50847212 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -922,30 +922,46 @@ pub fn translate_operator(
         | Operator::TableSize { .. } => {
             wasm_unsupported!("proposed bulk memory operator {:?}", op);
         }
+        Operator::V128Const { value } => {
+            let constant = builder.func.dfg.constants.insert(value.bytes().to_vec());
+            state.push1(builder.ins().vconst(I32X4, constant)) // TODO need to decide what to do about I32X4 here; the WASM spec has V128
+        }
+        Operator::I8x16Splat
+        | Operator::I16x8Splat
+        | Operator::I32x4Splat
+        | Operator::I64x2Splat
+        | Operator::F32x4Splat
+        | Operator::F64x2Splat => {
+            let value_to_splat = state.pop1();
+            let ty = type_of(op);
+            state.push1(builder.ins().splat(ty, value_to_splat))
+        }
+        Operator::I32x4ExtractLane { lane }
+        | Operator::I64x2ExtractLane { lane }
+        | Operator::F32x4ExtractLane { lane }
+        | Operator::F64x2ExtractLane { lane } => {
+            let vector = state.pop1();
+            state.push1(builder.ins().extractlane(vector, lane.clone()))
+        }
+        Operator::I8x16ReplaceLane { lane }
+        | Operator::I16x8ReplaceLane { lane }
+        | Operator::I32x4ReplaceLane { lane }
+        | Operator::I64x2ReplaceLane { lane }
+        | Operator::F32x4ReplaceLane { lane }
+        | Operator::F64x2ReplaceLane { lane } => {
+            let (vector, replacement_value) = state.pop2();
+            let replaced_vector = builder
+                .ins()
+                .insertlane(vector, lane.clone(), replacement_value);
+            state.push1(replaced_vector)
+        }
         Operator::V128Load { .. }
         | Operator::V128Store { .. }
-        | Operator::V128Const { .. }
-        | Operator::V8x16Shuffle { .. }
-        | Operator::I8x16Splat
         | Operator::I8x16ExtractLaneS { .. }
         | Operator::I8x16ExtractLaneU { .. }
-        | Operator::I8x16ReplaceLane { .. }
-        | Operator::I16x8Splat
         | Operator::I16x8ExtractLaneS { .. }
         | Operator::I16x8ExtractLaneU { .. }
-        | Operator::I16x8ReplaceLane { .. }
-        | Operator::I32x4Splat
-        | Operator::I32x4ExtractLane { .. }
-        | Operator::I32x4ReplaceLane { .. }
-        | Operator::I64x2Splat
-        | Operator::I64x2ExtractLane { .. }
-        | Operator::I64x2ReplaceLane { .. }
-        | Operator::F32x4Splat
-        | Operator::F32x4ExtractLane { .. }
-        | Operator::F32x4ReplaceLane { .. }
-        | Operator::F64x2Splat
-        | Operator::F64x2ExtractLane { .. }
-        | Operator::F64x2ReplaceLane { .. }
+        | Operator::V8x16Shuffle { .. }
         | Operator::I8x16Eq
         | Operator::I8x16Ne
         | Operator::I8x16LtS
@@ -1291,3 +1307,160 @@ fn translate_br_if_args(
     let inputs = state.peekn(return_count);
     (br_destination, inputs)
 }
+
+/// Determine the returned value type of a WebAssembly operator
+fn type_of(operator: &Operator) -> Type {
+    match operator {
+        Operator::V128Load { .. }
+        | Operator::V128Store { .. }
+        | Operator::V128Const { .. }
+        | Operator::V128Not
+        | Operator::V128And
+        | Operator::V128Or
+        | Operator::V128Xor
+        | Operator::V128Bitselect => I32X4, // TODO this is used as a default until we determine what to do about the spec's V128
+
+        Operator::V8x16Shuffle { .. }
+        | Operator::I8x16Splat
+        | Operator::I8x16ExtractLaneS { .. }
+        | Operator::I8x16ExtractLaneU { .. }
+        | Operator::I8x16ReplaceLane { .. }
+        | Operator::I8x16Eq
+        | Operator::I8x16Ne
+        | Operator::I8x16LtS
+        | Operator::I8x16LtU
+        | Operator::I8x16GtS
+        | Operator::I8x16GtU
+        | Operator::I8x16LeS
+        | Operator::I8x16LeU
+        | Operator::I8x16GeS
+        | Operator::I8x16GeU
+        | Operator::I8x16Neg
+        | Operator::I8x16AnyTrue
+        | Operator::I8x16AllTrue
+        | Operator::I8x16Shl
+        | Operator::I8x16ShrS
+        | Operator::I8x16ShrU
+        | Operator::I8x16Add
+        | Operator::I8x16AddSaturateS
+        | Operator::I8x16AddSaturateU
+        | Operator::I8x16Sub
+        | Operator::I8x16SubSaturateS
+        | Operator::I8x16SubSaturateU
+        | Operator::I8x16Mul => I8X16,
+
+        Operator::I16x8Splat
+        | Operator::I16x8ExtractLaneS { .. }
+        | Operator::I16x8ExtractLaneU { .. }
+        | Operator::I16x8ReplaceLane { .. }
+        | Operator::I16x8Eq
+        | Operator::I16x8Ne
+        | Operator::I16x8LtS
+        | Operator::I16x8LtU
+        | Operator::I16x8GtS
+        | Operator::I16x8GtU
+        | Operator::I16x8LeS
+        | Operator::I16x8LeU
+        | Operator::I16x8GeS
+        | Operator::I16x8GeU
+        | Operator::I16x8Neg
+        | Operator::I16x8AnyTrue
+        | Operator::I16x8AllTrue
+        | Operator::I16x8Shl
+        | Operator::I16x8ShrS
+        | Operator::I16x8ShrU
+        | Operator::I16x8Add
+        | Operator::I16x8AddSaturateS
+        | Operator::I16x8AddSaturateU
+        | Operator::I16x8Sub
+        | Operator::I16x8SubSaturateS
+        | Operator::I16x8SubSaturateU
+        | Operator::I16x8Mul => I16X8,
+
+        Operator::I32x4Splat
+        | Operator::I32x4ExtractLane { .. }
+        | Operator::I32x4ReplaceLane { .. }
+        | Operator::I32x4Eq
+        | Operator::I32x4Ne
+        | Operator::I32x4LtS
+        | Operator::I32x4LtU
+        | Operator::I32x4GtS
+        | Operator::I32x4GtU
+        | Operator::I32x4LeS
+        | Operator::I32x4LeU
+        | Operator::I32x4GeS
+        | Operator::I32x4GeU
+        | Operator::I32x4Neg
+        | Operator::I32x4AnyTrue
+        | Operator::I32x4AllTrue
+        | Operator::I32x4Shl
+        | Operator::I32x4ShrS
+        | Operator::I32x4ShrU
+        | Operator::I32x4Add
+        | Operator::I32x4Sub
+        | Operator::I32x4Mul
+        | Operator::F32x4ConvertSI32x4
+        | Operator::F32x4ConvertUI32x4 => I32X4,
+
+        Operator::I64x2Splat
+        | Operator::I64x2ExtractLane { .. }
+        | Operator::I64x2ReplaceLane { .. }
+        | Operator::I64x2Neg
+        | Operator::I64x2AnyTrue
+        | Operator::I64x2AllTrue
+        | Operator::I64x2Shl
+        | Operator::I64x2ShrS
+        | Operator::I64x2ShrU
+        | Operator::I64x2Add
+        | Operator::I64x2Sub
+        | Operator::F64x2ConvertSI64x2
+        | Operator::F64x2ConvertUI64x2 => I64X2,
+
+        Operator::F32x4Splat
+        | Operator::F32x4ExtractLane { .. }
+        | Operator::F32x4ReplaceLane { .. }
+        | Operator::F32x4Eq
+        | Operator::F32x4Ne
+        | Operator::F32x4Lt
+        | Operator::F32x4Gt
+        | Operator::F32x4Le
+        | Operator::F32x4Ge
+        | Operator::F32x4Abs
+        | Operator::F32x4Neg
+        | Operator::F32x4Sqrt
+        | Operator::F32x4Add
+        | Operator::F32x4Sub
+        | Operator::F32x4Mul
+        | Operator::F32x4Div
+        | Operator::F32x4Min
+        | Operator::F32x4Max
+        | Operator::I32x4TruncSF32x4Sat
+        | Operator::I32x4TruncUF32x4Sat => F32X4,
+
+        Operator::F64x2Splat
+        | Operator::F64x2ExtractLane { .. }
+        | Operator::F64x2ReplaceLane { .. }
+        | Operator::F64x2Eq
+        | Operator::F64x2Ne
+        | Operator::F64x2Lt
+        | Operator::F64x2Gt
+        | Operator::F64x2Le
+        | Operator::F64x2Ge
+        | Operator::F64x2Abs
+        | Operator::F64x2Neg
+        | Operator::F64x2Sqrt
+        | Operator::F64x2Add
+        | Operator::F64x2Sub
+        | Operator::F64x2Mul
+        | Operator::F64x2Div
+        | Operator::F64x2Min
+        | Operator::F64x2Max
+        | Operator::I64x2TruncSF64x2Sat
+        | Operator::I64x2TruncUF64x2Sat => F64X2,
+
+        _ => unimplemented!(
+            "Currently only the SIMD instructions are translated to their return type: {:?}",
+            operator
+        ),
+    }
+}

From 020e5987d30be6bbb15d07c18db185e0b12ee1ec Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 2 Aug 2019 13:27:13 -0700
Subject: [PATCH 2608/3084] Use little-endian ordering for CLIF vconst
 immediate

Examining wasm-objdump revealed that it stores SIMD constants in little-endian order, e.g.:

000071 func[2] :
 000072: fd 02 01 00 00 00 02 00 00 | v128.const 0x00000001 0x00000002 0x00000003 0x00000004
 00007b: 00 03 00 00 00 04 00 00 00 |
 000084: fd 0d 03                   | i32x4.extract_lane 3
 000087: 0b                         | end

This change avoids confusion by making the CLIF representation use little-endian order as well.
---
 cranelift/codegen/src/ir/immediates.rs        | 26 +++++++++++++++----
 .../filetests/isa/x86/rodata-vconst.clif      |  2 +-
 2 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index 5c36f362ab..8917d96d4b 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -275,7 +275,7 @@ impl Display for Uimm128 {
     fn fmt(&self, f: &mut Formatter) -> fmt::Result {
         write!(f, "0x")?;
         let mut anything_written = false;
-        for &b in self.0.iter() {
+        for &b in self.0.iter().rev() {
             if b == 0 && !anything_written {
                 continue;
             } else {
@@ -293,7 +293,7 @@ impl Display for Uimm128 {
 impl From for Uimm128 {
     fn from(x: u64) -> Self {
         let mut buffer: [u8; 16] = [0; 16]; // zero-fill
-        (0..8).for_each(|byte| buffer[15 - byte] = (x >> (byte as u64 * 8) & 0xff) as u8); // insert each byte from the u64 into v after byte position 8
+        (0..8).for_each(|byte| buffer[byte] = (x >> (byte as u64 * 8) & 0xff) as u8); // insert each byte from the u64 into v in little-endian order
         Uimm128(buffer)
     }
 }
@@ -316,15 +316,15 @@ impl FromStr for Uimm128 {
             Err("Expected a hexadecimal string, e.g. 0x1234")
         } else if s.len() % 2 != 0 {
             Err("Hexadecimal string must have an even number of digits")
-        } else if s.len() > 32 {
+        } else if s.len() > 34 {
             Err("Hexadecimal string has too many digits to fit in a 128-bit vector")
         } else {
             let mut buffer = [0; 16]; // zero-fill
-            let start_at = 16 - s.len() / 2;
+            let start_at = s.len() / 2 - 1;
             for i in (2..s.len()).step_by(2) {
                 let byte = u8::from_str_radix(&s[i..i + 2], 16)
                     .or_else(|_| Err("Unable to parse as hexadecimal"))?;
-                let position = start_at + (i / 2);
+                let position = start_at - (i / 2);
                 buffer[position] = byte;
             }
             Ok(Uimm128(buffer))
@@ -984,6 +984,22 @@ mod tests {
         );
     }
 
+    #[test]
+    fn uimm128_endianness() {
+        assert_eq!(
+            "0x42".parse::().unwrap().0,
+            [0x42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+        );
+        assert_eq!(
+            "0x00".parse::().unwrap().0,
+            [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+        );
+        assert_eq!(
+            "0x12345678".parse::().unwrap().0,
+            [0x78, 0x56, 0x34, 0x12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+        )
+    }
+
     #[test]
     fn format_offset32() {
         assert_eq!(Offset32(0).to_string(), "");
diff --git a/cranelift/filetests/filetests/isa/x86/rodata-vconst.clif b/cranelift/filetests/filetests/isa/x86/rodata-vconst.clif
index 772cf3b03c..fd029b678d 100644
--- a/cranelift/filetests/filetests/isa/x86/rodata-vconst.clif
+++ b/cranelift/filetests/filetests/isa/x86/rodata-vconst.clif
@@ -10,4 +10,4 @@ ebb0:
     return v0
 }
 
-; sameln: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 12, 34]
+; sameln: [34, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

From c595acfd0d64273a81e35f951b79b75c3fd3df4f Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 8 Aug 2019 11:31:35 -0700
Subject: [PATCH 2609/3084] Convert constants added by v128.const to the
 appropriate type before use

By default, constants added by SIMD's v128.const will be typed as I8x16 in CLIF. This type must be changed to the appropriate vector type before use to satisfy cranelift's type checking. To do this, we track what SSA values are created by v128.const and convert them with a raw_bitcast immediately before use in the currently implemented SIMD instructions.
---
 cranelift/wasm/src/code_translator.rs | 37 ++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 7 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 9c50847212..b87844bfd4 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -33,7 +33,7 @@ use crate::wasm_unsupported;
 use core::{i32, u32};
 use cranelift_codegen::ir::condcodes::{FloatCC, IntCC};
 use cranelift_codegen::ir::types::*;
-use cranelift_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags, ValueLabel};
+use cranelift_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags, Value, ValueLabel};
 use cranelift_codegen::packed_option::ReservedValue;
 use cranelift_frontend::{FunctionBuilder, Variable};
 use wasmparser::{MemoryImmediate, Operator};
@@ -923,8 +923,10 @@ pub fn translate_operator(
             wasm_unsupported!("proposed bulk memory operator {:?}", op);
         }
         Operator::V128Const { value } => {
-            let constant = builder.func.dfg.constants.insert(value.bytes().to_vec());
-            state.push1(builder.ins().vconst(I32X4, constant)) // TODO need to decide what to do about I32X4 here; the WASM spec has V128
+            let handle = builder.func.dfg.constants.insert(value.bytes().to_vec());
+            let value = builder.ins().vconst(I8X16, handle);
+            // the v128.const is typed in CLIF as a I8x16 but raw_bitcast to a different type before use
+            state.push1(value)
         }
         Operator::I8x16Splat
         | Operator::I16x8Splat
@@ -934,13 +936,14 @@ pub fn translate_operator(
         | Operator::F64x2Splat => {
             let value_to_splat = state.pop1();
             let ty = type_of(op);
-            state.push1(builder.ins().splat(ty, value_to_splat))
+            let splatted = builder.ins().splat(ty, value_to_splat);
+            state.push1(splatted)
         }
         Operator::I32x4ExtractLane { lane }
         | Operator::I64x2ExtractLane { lane }
         | Operator::F32x4ExtractLane { lane }
         | Operator::F64x2ExtractLane { lane } => {
-            let vector = state.pop1();
+            let vector = optionally_bitcast_vector(state.pop1(), type_of(op), builder);
             state.push1(builder.ins().extractlane(vector, lane.clone()))
         }
         Operator::I8x16ReplaceLane { lane }
@@ -950,10 +953,16 @@ pub fn translate_operator(
         | Operator::F32x4ReplaceLane { lane }
         | Operator::F64x2ReplaceLane { lane } => {
             let (vector, replacement_value) = state.pop2();
+            let original_vector_type = builder.func.dfg.value_type(vector);
+            let vector = optionally_bitcast_vector(vector, type_of(op), builder);
             let replaced_vector = builder
                 .ins()
                 .insertlane(vector, lane.clone(), replacement_value);
-            state.push1(replaced_vector)
+            state.push1(optionally_bitcast_vector(
+                replaced_vector,
+                original_vector_type,
+                builder,
+            ))
         }
         Operator::V128Load { .. }
         | Operator::V128Store { .. }
@@ -1318,7 +1327,7 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::V128And
         | Operator::V128Or
         | Operator::V128Xor
-        | Operator::V128Bitselect => I32X4, // TODO this is used as a default until we determine what to do about the spec's V128
+        | Operator::V128Bitselect => I8X16, // default type representing V128
 
         Operator::V8x16Shuffle { .. }
         | Operator::I8x16Splat
@@ -1464,3 +1473,17 @@ fn type_of(operator: &Operator) -> Type {
         ),
     }
 }
+
+/// Some SIMD operations only operate on I8X16 in CLIF; this will convert them to that type by
+/// adding a raw_bitcast if necessary
+fn optionally_bitcast_vector(
+    value: Value,
+    needed_type: Type,
+    builder: &mut FunctionBuilder,
+) -> Value {
+    if builder.func.dfg.value_type(value) != needed_type {
+        builder.ins().raw_bitcast(needed_type, value)
+    } else {
+        value
+    }
+}

From ca6449626fce3db8ee403b86c84d4d34e47cde79 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 8 Aug 2019 11:33:05 -0700
Subject: [PATCH 2610/3084] Verify that cranelift-wasm can translate SIMD
 instructions

To do so we must use a new version of wabt-rs that allows us to enable features (e.g. SIMD) when translating the wasmtests
---
 cranelift/wasm/tests/wasm_testsuite.rs |  6 ++++--
 cranelift/wasmtests/simd.wat           | 23 +++++++++++++++++++++++
 2 files changed, 27 insertions(+), 2 deletions(-)
 create mode 100644 cranelift/wasmtests/simd.wat

diff --git a/cranelift/wasm/tests/wasm_testsuite.rs b/cranelift/wasm/tests/wasm_testsuite.rs
index f521828282..6b16f69791 100644
--- a/cranelift/wasm/tests/wasm_testsuite.rs
+++ b/cranelift/wasm/tests/wasm_testsuite.rs
@@ -10,7 +10,7 @@ use std::io::prelude::*;
 use std::path::Path;
 use std::str::FromStr;
 use target_lexicon::triple;
-use wabt::wat2wasm;
+use wabt::{wat2wasm_with_features, Features};
 
 #[test]
 fn testsuite() {
@@ -61,7 +61,9 @@ fn handle_module(path: &Path, flags: &Flags, return_mode: ReturnMode) {
             Some("wasm") => read_file(path).expect("error reading wasm file"),
             Some("wat") => {
                 let wat = read_file(path).expect("error reading wat file");
-                match wat2wasm(&wat) {
+                let mut features = Features::new();
+                features.enable_all();
+                match wat2wasm_with_features(&wat, features) {
                     Ok(wasm) => wasm,
                     Err(e) => {
                         panic!("error converting wat to wasm: {:?}", e);
diff --git a/cranelift/wasmtests/simd.wat b/cranelift/wasmtests/simd.wat
new file mode 100644
index 0000000000..99b7d5c10d
--- /dev/null
+++ b/cranelift/wasmtests/simd.wat
@@ -0,0 +1,23 @@
+(module
+  (func $test_splat (result i32)
+    i32.const 42
+    i32x4.splat
+    i32x4.extract_lane 0
+  )
+
+  (func $test_insert_lane (result i32)
+      v128.const i64x2 0 0
+      i32.const 99
+      i32x4.replace_lane 1
+      i32x4.extract_lane 1
+  )
+
+  (func $test_const (result i32)
+    v128.const i32x4 1 2 3 4
+    i32x4.extract_lane 3
+  )
+
+  (export "test_splat" (func $test_splat))
+  (export "test_insert_lane" (func $test_insert_lane))
+  (export "test_const" (func $test_const))
+)

From 2b49b513065fbf8837147ce183dc8f689e48ea4b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 27 Aug 2019 01:52:06 -0700
Subject: [PATCH 2611/3084] Add flags to allow wasm SIMD instructions (#910)

Add `--enable-simd` flag to `clif-util wasm`
---
 cranelift/src/clif-util.rs             | 10 +++++++++-
 cranelift/src/wasm.rs                  | 11 +++++++++--
 cranelift/wasm/tests/wasm_testsuite.rs |  4 ++--
 3 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 8077cf9b23..19402d4bff 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -104,7 +104,13 @@ fn add_print_flag<'a>() -> clap::Arg<'a, 'a> {
 fn add_debug_flag<'a>() -> clap::Arg<'a, 'a> {
     Arg::with_name("debug")
         .short("d")
-        .help("enable debug output on stderr/stdout")
+        .help("Enable debug output on stderr/stdout")
+}
+
+fn add_enable_simd_flag<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("enable-simd")
+        .long("enable-simd")
+        .help("Enable WASM's SIMD operations")
 }
 
 /// Returns a vector of clap value options and changes these options into a vector of strings
@@ -137,6 +143,7 @@ fn add_wasm_or_compile<'a>(cmd: &str) -> clap::App<'a, 'a> {
         .arg(add_target_flag())
         .arg(add_input_file_arg())
         .arg(add_debug_flag())
+        .arg(add_enable_simd_flag())
 }
 
 fn handle_debug_flag(debug: bool) {
@@ -296,6 +303,7 @@ fn main() {
                     rest_cmd.is_present("print-size"),
                     rest_cmd.is_present("time-passes"),
                     rest_cmd.is_present("value-ranges"),
+                    rest_cmd.is_present("enable-simd"),
                 )
             };
 
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index f6b2036d57..cc55238822 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -19,7 +19,7 @@ use cranelift_wasm::{translate_module, DummyEnvironment, FuncIndex, ReturnMode};
 use std::path::Path;
 use std::path::PathBuf;
 use term;
-use wabt::wat2wasm;
+use wabt::{wat2wasm_with_features, Features};
 
 macro_rules! vprintln {
     ($x: expr, $($tts:tt)*) => {
@@ -49,6 +49,7 @@ pub fn run(
     flag_print_size: bool,
     flag_report_times: bool,
     flag_calc_value_ranges: bool,
+    flag_enable_simd: bool,
 ) -> Result<(), String> {
     let parsed = parse_sets_and_triple(flag_set, flag_triple)?;
 
@@ -64,6 +65,7 @@ pub fn run(
             flag_print_disasm,
             flag_report_times,
             flag_calc_value_ranges,
+            flag_enable_simd,
             &path.to_path_buf(),
             &name,
             parsed.as_fisa(),
@@ -81,6 +83,7 @@ fn handle_module(
     flag_print_disasm: bool,
     flag_report_times: bool,
     flag_calc_value_ranges: bool,
+    flag_enable_simd: bool,
     path: &PathBuf,
     name: &str,
     fisa: FlagsOrIsa,
@@ -97,7 +100,11 @@ fn handle_module(
     let mut module_binary = read_to_end(path.clone()).map_err(|err| err.to_string())?;
 
     if !module_binary.starts_with(&[b'\0', b'a', b's', b'm']) {
-        module_binary = match wat2wasm(&module_binary) {
+        let mut features = Features::new();
+        if flag_enable_simd {
+            features.enable_simd();
+        }
+        module_binary = match wat2wasm_with_features(&module_binary, features) {
             Ok(data) => data,
             Err(e) => return Err(e.to_string()),
         };
diff --git a/cranelift/wasm/tests/wasm_testsuite.rs b/cranelift/wasm/tests/wasm_testsuite.rs
index 6b16f69791..fd46458d53 100644
--- a/cranelift/wasm/tests/wasm_testsuite.rs
+++ b/cranelift/wasm/tests/wasm_testsuite.rs
@@ -53,6 +53,8 @@ fn read_file(path: &Path) -> io::Result> {
 }
 
 fn handle_module(path: &Path, flags: &Flags, return_mode: ReturnMode) {
+    let mut features = Features::new();
+    features.enable_all();
     let data = match path.extension() {
         None => {
             panic!("the file extension is not wasm or wat");
@@ -61,8 +63,6 @@ fn handle_module(path: &Path, flags: &Flags, return_mode: ReturnMode) {
             Some("wasm") => read_file(path).expect("error reading wasm file"),
             Some("wat") => {
                 let wat = read_file(path).expect("error reading wat file");
-                let mut features = Features::new();
-                features.enable_all();
                 match wat2wasm_with_features(&wat, features) {
                     Ok(wasm) => wasm,
                     Err(e) => {

From 173cfb02e647b0d3d66e222494e85ece5966920b Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 22 Aug 2019 12:41:57 +0200
Subject: [PATCH 2612/3084] Fixes #841: Bump the required version of rustc to
 1.37;

---
 .travis.yml         | 2 +-
 cranelift/README.md | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 212f86aff6..85de06c5c7 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,7 +7,7 @@ language: rust
 rust:
     # The oldest version we currently support. See
     # CONTRIBUTING.md#rustc-version-support for details.
-    - 1.35.0
+    - 1.37.0
     - beta
     - nightly
 matrix:
diff --git a/cranelift/README.md b/cranelift/README.md
index 419fc95a6d..3bfa2dc6cd 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -9,7 +9,7 @@ into executable machine code.
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
 [![Travis Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
-![Minimum rustc 1.35](https://img.shields.io/badge/rustc-1.35+-green.svg)
+![Minimum rustc 1.37](https://img.shields.io/badge/rustc-1.37+-green.svg)
 
 For more information, see [the
 documentation](https://cranelift.readthedocs.io/en/latest/?badge=latest).
@@ -52,7 +52,7 @@ needed before it would be ready for a production use case.
 
 Cranelift's APIs are not yet stable.
 
-Cranelift currently requires Rust 1.35 or later to build.
+Cranelift currently requires Rust 1.37 or later to build.
 
 Planned uses
 ------------

From 69f90b390e7c9453d73e95e4be5b5aaaf4744f51 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 22 Aug 2019 12:42:24 +0200
Subject: [PATCH 2613/3084] Build: remove pip packages from the install step;

Pip packages were installed because of the meta build in Python, but
they're now unused, so it's a waste of build time and cache space.
---
 .travis.yml | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 85de06c5c7..18d3aa37e9 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -21,12 +21,6 @@ matrix:
     fast_finish: true
 dist: xenial
 sudo: false
-addons:
-    apt:
-        packages:
-            - python3-pip
-install:
-    - pip3 install --verbose --user --upgrade mypy flake8
 before_script:
     # If an old version of rustfmt from cargo is already installed, uninstall
     # it, since it can prevent the installation of the new version from rustup.
@@ -47,5 +41,3 @@ before_script:
 script: ./test-all.sh
 cache:
     cargo: true
-    directories:
-        - $HOME/.cache/pip

From e4702d695e71899ddf6480f05e9bf3f74cf66fcf Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 21 Aug 2019 16:35:45 +0200
Subject: [PATCH 2614/3084] [meta] Generate doc comments for the encodings
 tables;

---
 cranelift/codegen/meta/src/gen_binemit.rs   |  2 +-
 cranelift/codegen/meta/src/gen_encodings.rs | 71 ++++++++++++++++++++-
 2 files changed, 70 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_binemit.rs b/cranelift/codegen/meta/src/gen_binemit.rs
index 1bf95a83e0..53929fe8c5 100644
--- a/cranelift/codegen/meta/src/gen_binemit.rs
+++ b/cranelift/codegen/meta/src/gen_binemit.rs
@@ -137,7 +137,7 @@ fn unwrap_values(
                         values_slice,
                         i
                     );
-                    fmt.line(format!("{}, ", stack.stack_base_mask()));
+                    fmt.line(format!("{},", stack.stack_base_mask()));
                     fmt.line("&func.stack_slots,");
                 });
                 fmt.line(").unwrap();");
diff --git a/cranelift/codegen/meta/src/gen_encodings.rs b/cranelift/codegen/meta/src/gen_encodings.rs
index 14ea392671..71fcf17002 100644
--- a/cranelift/codegen/meta/src/gen_encodings.rs
+++ b/cranelift/codegen/meta/src/gen_encodings.rs
@@ -142,6 +142,7 @@ fn emit_instp(instp: &InstructionPredicate, has_func: bool, fmt: &mut Formatter)
 fn emit_recipe_predicates(isa: &TargetIsa, fmt: &mut Formatter) {
     let mut predicate_names = HashMap::new();
 
+    fmt.comment(format!("{} recipe predicates.", isa.name));
     for recipe in isa.recipes.values() {
         let (isap, instp) = match (&recipe.isa_predicate, &recipe.inst_predicate) {
             (None, None) => continue,
@@ -177,8 +178,15 @@ fn emit_recipe_predicates(isa: &TargetIsa, fmt: &mut Formatter) {
         });
         fmtln!(fmt, "}");
     }
+    fmt.empty_line();
 
     // Generate the static table.
+    fmt.doc_comment(format!(
+        r#"{} recipe predicate table.
+
+        One entry per recipe, set to Some only when the recipe is guarded by a predicate."#,
+        isa.name
+    ));
     fmtln!(
         fmt,
         "pub static RECIPE_PREDICATES: [RecipePredicate; {}] = [",
@@ -193,11 +201,13 @@ fn emit_recipe_predicates(isa: &TargetIsa, fmt: &mut Formatter) {
         }
     });
     fmtln!(fmt, "];");
+    fmt.empty_line();
 }
 
 /// Emit private functions for matching instruction predicates as well as a static
 /// `INST_PREDICATES` array indexed by predicate number.
 fn emit_inst_predicates(isa: &TargetIsa, fmt: &mut Formatter) {
+    fmt.comment(format!("{} instruction predicates.", isa.name));
     for (id, instp) in isa.encodings_predicates.iter() {
         fmtln!(fmt, "fn inst_predicate_{}(func: &crate::ir::Function, inst: &crate::ir::InstructionData) -> bool {{", id.index());
         fmt.indent(|fmt| {
@@ -205,8 +215,16 @@ fn emit_inst_predicates(isa: &TargetIsa, fmt: &mut Formatter) {
         });
         fmtln!(fmt, "}");
     }
+    fmt.empty_line();
 
     // Generate the static table.
+    fmt.doc_comment(format!(
+        r#"{} instruction predicate table.
+
+        One entry per instruction predicate, so the encoding bytecode can embed indexes into this
+        table."#,
+        isa.name
+    ));
     fmtln!(
         fmt,
         "pub static INST_PREDICATES: [InstPredicate; {}] = [",
@@ -218,12 +236,18 @@ fn emit_inst_predicates(isa: &TargetIsa, fmt: &mut Formatter) {
         }
     });
     fmtln!(fmt, "];");
+    fmt.empty_line();
 }
 
 /// Emit a table of encoding recipe names keyed by recipe number.
 ///
 /// This is used for pretty-printing encodings.
 fn emit_recipe_names(isa: &TargetIsa, fmt: &mut Formatter) {
+    fmt.doc_comment(format!(
+        r#"{} recipe names, using the same recipe index spaces as the one specified by the
+        corresponding binemit file."#,
+        isa.name
+    ));
     fmtln!(
         fmt,
         "static RECIPE_NAMES: [&str; {}] = [",
@@ -235,6 +259,7 @@ fn emit_recipe_names(isa: &TargetIsa, fmt: &mut Formatter) {
         }
     });
     fmtln!(fmt, "];");
+    fmt.empty_line();
 }
 
 /// Returns a set of all the registers involved in fixed register constraints.
@@ -353,6 +378,12 @@ fn emit_operand_constraints(
 ///
 /// These are used by the register allocator to pick registers that can be properly encoded.
 fn emit_recipe_constraints(isa: &TargetIsa, fmt: &mut Formatter) {
+    fmt.doc_comment(format!(
+        r#"{} recipe constraints list, using the same recipe index spaces as the one
+        specified by the corresponding binemit file. These constraints are used by register
+        allocation to select the right location to use for input and output values."#,
+        isa.name
+    ));
     fmtln!(
         fmt,
         "static RECIPE_CONSTRAINTS: [RecipeConstraints; {}] = [",
@@ -437,10 +468,17 @@ fn emit_recipe_constraints(isa: &TargetIsa, fmt: &mut Formatter) {
         }
     });
     fmtln!(fmt, "];");
+    fmt.empty_line();
 }
 
 /// Emit a table of encoding recipe code size information.
 fn emit_recipe_sizing(isa: &TargetIsa, fmt: &mut Formatter) {
+    fmt.doc_comment(format!(
+        r#"{} recipe sizing descriptors, using the same recipe index spaces as the one
+        specified by the corresponding binemit file. These are used to compute the final size of an
+        instruction, as well as to compute the range of branches."#,
+        isa.name
+    ));
     fmtln!(
         fmt,
         "static RECIPE_SIZING: [RecipeSizing; {}] = [",
@@ -468,6 +506,7 @@ fn emit_recipe_sizing(isa: &TargetIsa, fmt: &mut Formatter) {
         }
     });
     fmtln!(fmt, "];");
+    fmt.empty_line();
 }
 
 /// Level 1 table mapping types to `Level2` objects.
@@ -878,7 +917,7 @@ fn encode_level2_hashtables<'a>(
     }
 }
 
-fn emit_tables(defs: &SharedDefinitions, isa: &TargetIsa, fmt: &mut Formatter) {
+fn emit_encoding_tables(defs: &SharedDefinitions, isa: &TargetIsa, fmt: &mut Formatter) {
     // Level 1 tables, one per CPU mode.
     let mut level1_tables: HashMap<&'static str, Level1Table> = HashMap::new();
 
@@ -923,6 +962,13 @@ fn emit_tables(defs: &SharedDefinitions, isa: &TargetIsa, fmt: &mut Formatter) {
     let level2_offset_type = offset_type(enc_lists.len());
 
     // Emit encoding lists.
+    fmt.doc_comment(
+        format!(r#"{} encoding lists.
+
+        This contains the entire encodings bytecode for every single instruction; the encodings
+        interpreter knows where to start from thanks to the initial lookup in the level 1 and level 2
+        table entries below."#, isa.name)
+    );
     fmtln!(fmt, "pub static ENCLISTS: [u16; {}] = [", enc_lists.len());
     fmt.indent(|fmt| {
         let mut line = Vec::new();
@@ -943,8 +989,17 @@ fn emit_tables(defs: &SharedDefinitions, isa: &TargetIsa, fmt: &mut Formatter) {
         }
     });
     fmtln!(fmt, "];");
+    fmt.empty_line();
 
     // Emit the full concatenation of level 2 hash tables.
+    fmt.doc_comment(format!(
+        r#"{} level 2 hash tables.
+
+        This hash table, keyed by instruction opcode, contains all the starting offsets for the
+        encodings interpreter, for all the CPU modes. It is jumped to after a lookup on the
+        instruction's controlling type in the level 1 hash table."#,
+        isa.name
+    ));
     fmtln!(
         fmt,
         "pub static LEVEL2: [Level2Entry<{}>; {}] = [",
@@ -971,6 +1026,7 @@ fn emit_tables(defs: &SharedDefinitions, isa: &TargetIsa, fmt: &mut Formatter) {
         }
     });
     fmtln!(fmt, "];");
+    fmt.empty_line();
 
     // Emit a level 1 hash table for each CPU mode.
     for cpu_mode in &isa.cpu_modes {
@@ -987,6 +1043,16 @@ fn emit_tables(defs: &SharedDefinitions, isa: &TargetIsa, fmt: &mut Formatter) {
             },
         );
 
+        fmt.doc_comment(format!(
+            r#"{} level 1 hash table for the CPU mode {}.
+
+            This hash table, keyed by instruction controlling type, contains all the level 2
+            hash-tables offsets for the given CPU mode, as well as a legalization identifier indicating
+            which legalization scheme to apply when the instruction doesn't have any valid encoding for
+            this CPU mode.
+        "#,
+            isa.name, cpu_mode.name
+        ));
         fmtln!(
             fmt,
             "pub static LEVEL1_{}: [Level1Entry<{}>; {}] = [",
@@ -1033,6 +1099,7 @@ fn emit_tables(defs: &SharedDefinitions, isa: &TargetIsa, fmt: &mut Formatter) {
             }
         });
         fmtln!(fmt, "];");
+        fmt.empty_line();
     }
 }
 
@@ -1043,7 +1110,7 @@ fn gen_isa(defs: &SharedDefinitions, isa: &TargetIsa, fmt: &mut Formatter) {
     // Make the `INST_PREDICATES` table.
     emit_inst_predicates(isa, fmt);
 
-    emit_tables(defs, isa, fmt);
+    emit_encoding_tables(defs, isa, fmt);
 
     emit_recipe_names(isa, fmt);
     emit_recipe_constraints(isa, fmt);

From 6fdc69ff2ee940c326944d1c0c8069e1efc57e92 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 27 Aug 2019 11:31:08 -0700
Subject: [PATCH 2615/3084] Add options for parsing test files (#942)

* Add options for parsing test files

This change allows adding parsing parameters more easily; e.g. a parameter is needed for setting the default calling convention for functions parsed as a part of the `run` test feature.

* Set default calling convention that of the host for `test run` file tests

Previously `test run` used the parser's hard-coded CallConv::Fast as the default calling convention but with this change any test being `run` will use the default calling convention of the machine running the test. `test run` will now throw an error if the calling convention of the function does not match the host's.
---
 cranelift/filetests/src/function_runner.rs |  11 +--
 cranelift/filetests/src/runone.rs          |   9 +-
 cranelift/reader/src/lib.rs                |   2 +-
 cranelift/reader/src/parser.rs             | 101 ++++++++++++++++++---
 cranelift/reader/src/sourcemap.rs          |   5 +-
 cranelift/src/bugpoint.rs                  |   8 +-
 cranelift/src/compile.rs                   |   5 +-
 cranelift/src/run.rs                       |  11 ++-
 8 files changed, 116 insertions(+), 36 deletions(-)

diff --git a/cranelift/filetests/src/function_runner.rs b/cranelift/filetests/src/function_runner.rs
index 954306f5d0..1c1272cce7 100644
--- a/cranelift/filetests/src/function_runner.rs
+++ b/cranelift/filetests/src/function_runner.rs
@@ -1,7 +1,7 @@
 use core::mem;
 use cranelift_codegen::binemit::{NullRelocSink, NullStackmapSink, NullTrapSink};
 use cranelift_codegen::ir::Function;
-use cranelift_codegen::isa::{CallConv, TargetIsa};
+use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::{settings, Context};
 use cranelift_native::builder as host_isa_builder;
 use memmap::MmapMut;
@@ -47,10 +47,7 @@ impl FunctionRunner {
             ));
         }
 
-        if func.signature.call_conv != self.isa.default_call_conv()
-            && func.signature.call_conv != CallConv::Fast
-        {
-            // ideally we wouldn't have to also check for Fast here but currently there is no way to inform the filetest parser that we would like to use a default other than Fast
+        if func.signature.call_conv != self.isa.default_call_conv() {
             return Err(String::from(
                 "Functions only run on the host's default calling convention; remove the specified calling convention in the function signature to use the host's default.",
             ));
@@ -97,7 +94,7 @@ impl FunctionRunner {
 #[cfg(test)]
 mod test {
     use super::*;
-    use cranelift_reader::parse_test;
+    use cranelift_reader::{parse_test, ParseOptions};
 
     #[test]
     fn nop() {
@@ -111,7 +108,7 @@ mod test {
         );
 
         // extract function
-        let test_file = parse_test(code.as_str(), None, None).unwrap();
+        let test_file = parse_test(code.as_str(), ParseOptions::default()).unwrap();
         assert_eq!(1, test_file.functions.len());
         let function = test_file.functions[0].0.clone();
 
diff --git a/cranelift/filetests/src/runone.rs b/cranelift/filetests/src/runone.rs
index 3a69c37c4e..4a2071b2db 100644
--- a/cranelift/filetests/src/runone.rs
+++ b/cranelift/filetests/src/runone.rs
@@ -8,8 +8,8 @@ use cranelift_codegen::print_errors::pretty_verifier_error;
 use cranelift_codegen::settings::Flags;
 use cranelift_codegen::timing;
 use cranelift_codegen::verify_function;
-use cranelift_reader::parse_test;
 use cranelift_reader::IsaSpec;
+use cranelift_reader::{parse_test, ParseOptions};
 use log::info;
 use std::borrow::Cow;
 use std::fs;
@@ -33,8 +33,13 @@ pub fn run(path: &Path, passes: Option<&[String]>, target: Option<&str>) -> Test
     info!("---\nFile: {}", path.to_string_lossy());
     let started = time::Instant::now();
     let buffer = read_to_string(path).map_err(|e| e.to_string())?;
+    let options = ParseOptions {
+        target,
+        passes,
+        ..ParseOptions::default()
+    };
 
-    let testfile = match parse_test(&buffer, passes, target) {
+    let testfile = match parse_test(&buffer, options) {
         Ok(testfile) => testfile,
         Err(e) => {
             if e.is_warning {
diff --git a/cranelift/reader/src/lib.rs b/cranelift/reader/src/lib.rs
index 803c17c62d..871bd80b2c 100644
--- a/cranelift/reader/src/lib.rs
+++ b/cranelift/reader/src/lib.rs
@@ -28,7 +28,7 @@
 
 pub use crate::error::{Location, ParseError, ParseResult};
 pub use crate::isaspec::{parse_options, IsaSpec};
-pub use crate::parser::{parse_functions, parse_test};
+pub use crate::parser::{parse_functions, parse_test, ParseOptions};
 pub use crate::sourcemap::SourceMap;
 pub use crate::testcommand::{TestCommand, TestOption};
 pub use crate::testfile::{Comment, Details, TestFile};
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index d503151701..a0fa0f127c 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -31,20 +31,37 @@ use target_lexicon::Triple;
 /// Any test commands or target declarations are ignored.
 pub fn parse_functions(text: &str) -> ParseResult> {
     let _tt = timing::parse_text();
-    parse_test(text, None, None)
+    parse_test(text, ParseOptions::default())
         .map(|file| file.functions.into_iter().map(|(func, _)| func).collect())
 }
 
+/// Options for configuring the parsing of filetests.
+pub struct ParseOptions<'a> {
+    /// Compiler passes to run on the parsed functions.
+    pub passes: Option<&'a [String]>,
+    /// Target ISA for compiling the parsed functions, e.g. "x86_64 skylake".
+    pub target: Option<&'a str>,
+    /// Default calling convention used when none is specified for a parsed function.
+    pub default_calling_convention: CallConv,
+}
+
+impl Default for ParseOptions<'_> {
+    fn default() -> Self {
+        Self {
+            passes: None,
+            target: None,
+            default_calling_convention: CallConv::Fast,
+        }
+    }
+}
+
 /// Parse the entire `text` as a test case file.
 ///
 /// The returned `TestFile` contains direct references to substrings of `text`.
-pub fn parse_test<'a>(
-    text: &'a str,
-    passes: Option<&'a [String]>,
-    target: Option<&str>,
-) -> ParseResult> {
+pub fn parse_test<'a>(text: &'a str, options: ParseOptions<'a>) -> ParseResult> {
     let _tt = timing::parse_text();
     let mut parser = Parser::new(text);
+
     // Gather the preamble comments.
     parser.start_gathering_comments();
 
@@ -53,12 +70,12 @@ pub fn parse_test<'a>(
 
     // Check for specified passes and target, if present throw out test commands/targets specified
     // in file.
-    match passes {
+    match options.passes {
         Some(pass_vec) => {
             parser.parse_test_commands();
             commands = parser.parse_cmdline_passes(pass_vec);
             parser.parse_target_specs()?;
-            isa_spec = parser.parse_cmdline_target(target)?;
+            isa_spec = parser.parse_cmdline_target(options.target)?;
         }
         None => {
             commands = parser.parse_test_commands();
@@ -66,6 +83,16 @@ pub fn parse_test<'a>(
         }
     };
 
+    // Decide between using the calling convention passed in the options or using the
+    // host's calling convention--if any tests are to be run on the host we should default to the
+    // host's calling convention.
+    parser = if commands.iter().any(|tc| tc.command == "run") {
+        let host_default_calling_convention = CallConv::triple_default(&Triple::host());
+        parser.with_default_calling_convention(host_default_calling_convention)
+    } else {
+        parser.with_default_calling_convention(options.default_calling_convention)
+    };
+
     parser.token();
     parser.claim_gathered_comments(AnyEntity::Function);
 
@@ -99,6 +126,9 @@ pub struct Parser<'a> {
 
     /// Comments collected so far.
     comments: Vec>,
+
+    /// Default calling conventions; used when none is specified.
+    default_calling_convention: CallConv,
 }
 
 /// Context for resolving references when parsing a single function.
@@ -235,11 +265,16 @@ impl<'a> Context<'a> {
     }
 
     // Allocate a new signature.
-    fn add_sig(&mut self, sig: SigRef, data: Signature, loc: Location) -> ParseResult<()> {
+    fn add_sig(
+        &mut self,
+        sig: SigRef,
+        data: Signature,
+        loc: Location,
+        defaultcc: CallConv,
+    ) -> ParseResult<()> {
         self.map.def_sig(sig, loc)?;
         while self.function.dfg.signatures.next_key().index() <= sig.index() {
-            self.function
-                .import_signature(Signature::new(CallConv::Fast));
+            self.function.import_signature(Signature::new(defaultcc));
         }
         self.function.dfg.signatures[sig] = data;
         Ok(())
@@ -318,6 +353,16 @@ impl<'a> Parser<'a> {
             gathering_comments: false,
             gathered_comments: Vec::new(),
             comments: Vec::new(),
+            default_calling_convention: CallConv::Fast,
+        }
+    }
+
+    /// Modify the default calling convention; returns a new parser with the changed calling
+    /// convention.
+    pub fn with_default_calling_convention(self, default_calling_convention: CallConv) -> Self {
+        Self {
+            default_calling_convention,
+            ..self
         }
     }
 
@@ -1018,7 +1063,7 @@ impl<'a> Parser<'a> {
     //
     fn parse_signature(&mut self, unique_isa: Option<&dyn TargetIsa>) -> ParseResult {
         // Calling convention defaults to `fast`, but can be changed.
-        let mut sig = Signature::new(CallConv::Fast);
+        let mut sig = Signature::new(self.default_calling_convention);
 
         self.match_token(Token::LPar, "expected function signature: ( args... )")?;
         // signature ::=  "(" * [abi-param-list] ")" ["->" retlist] [callconv]
@@ -1170,7 +1215,9 @@ impl<'a> Parser<'a> {
                 Some(Token::SigRef(..)) => {
                     self.start_gathering_comments();
                     self.parse_signature_decl(ctx.unique_isa)
-                        .and_then(|(sig, dat)| ctx.add_sig(sig, dat, self.loc))
+                        .and_then(|(sig, dat)| {
+                            ctx.add_sig(sig, dat, self.loc, self.default_calling_convention)
+                        })
                 }
                 Some(Token::FuncRef(..)) => {
                     self.start_gathering_comments();
@@ -2951,8 +2998,7 @@ mod tests {
                              set enable_float=false
                              ; still preamble
                              function %comment() system_v {}",
-            None,
-            None,
+            ParseOptions::default(),
         )
         .unwrap();
         assert_eq!(tf.commands.len(), 2);
@@ -2978,6 +3024,7 @@ mod tests {
         assert!(parse_test(
             "target
                             function %foo() system_v {}",
+            ParseOptions::default()
         )
         .is_err());
 
@@ -2985,6 +3032,7 @@ mod tests {
             "target riscv32
                             set enable_float=false
                             function %foo() system_v {}",
+            ParseOptions::default()
         )
         .is_err());
 
@@ -2992,6 +3040,7 @@ mod tests {
             "set enable_float=false
                           isa riscv
                           function %foo() system_v {}",
+            ParseOptions::default(),
         )
         .unwrap()
         .isa_spec
@@ -3052,4 +3101,26 @@ mod tests {
         );
         assert!(parser.parse_function(None).is_err());
     }
+
+    #[test]
+    fn change_default_calling_convention() {
+        let code = "function %test() {
+        ebb0:
+            return
+        }";
+
+        // By default the parser will use the fast calling convention if none is specified.
+        let mut parser = Parser::new(code);
+        assert_eq!(
+            parser.parse_function(None).unwrap().0.signature.call_conv,
+            CallConv::Fast
+        );
+
+        // However, we can specify a different calling convention to be the default.
+        let mut parser = Parser::new(code).with_default_calling_convention(CallConv::Cold);
+        assert_eq!(
+            parser.parse_function(None).unwrap().0.signature.call_conv,
+            CallConv::Cold
+        );
+    }
 }
diff --git a/cranelift/reader/src/sourcemap.rs b/cranelift/reader/src/sourcemap.rs
index a398dd5849..6291d0cd76 100644
--- a/cranelift/reader/src/sourcemap.rs
+++ b/cranelift/reader/src/sourcemap.rs
@@ -211,7 +211,7 @@ impl SourceMap {
 
 #[cfg(test)]
 mod tests {
-    use crate::parse_test;
+    use crate::{parse_test, ParseOptions};
 
     #[test]
     fn details() {
@@ -222,8 +222,7 @@ mod tests {
                              ebb0(v4: i32, v7: i32):
                                v10 = iadd v4, v7
                              }",
-            None,
-            None,
+            ParseOptions::default(),
         )
         .unwrap();
         let map = &tf.functions[0].1.map;
diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index 11110f0b00..90d475199f 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -9,7 +9,7 @@ use cranelift_codegen::ir::{
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::Context;
 use cranelift_entity::PrimaryMap;
-use cranelift_reader::parse_test;
+use cranelift_reader::{parse_test, ParseOptions};
 use std::collections::HashMap;
 use std::path::Path;
 
@@ -27,7 +27,8 @@ pub fn run(
     let path = Path::new(&filename).to_path_buf();
 
     let buffer = read_to_string(&path).map_err(|e| format!("{}: {}", filename, e))?;
-    let test_file = parse_test(&buffer, None, None).map_err(|e| format!("{}: {}", filename, e))?;
+    let test_file =
+        parse_test(&buffer, ParseOptions::default()).map_err(|e| format!("{}: {}", filename, e))?;
 
     // If we have an isa from the command-line, use that. Otherwise if the
     // file contains a unique isa, use that.
@@ -754,12 +755,13 @@ impl<'a> CrashCheckContext<'a> {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use cranelift_reader::ParseOptions;
 
     #[test]
     fn test_reduce() {
         const TEST: &'static str = include_str!("./bugpoint_test.clif");
 
-        let test_file = parse_test(TEST, None, None).unwrap();
+        let test_file = parse_test(TEST, ParseOptions::default()).unwrap();
 
         // If we have an isa from the command-line, use that. Otherwise if the
         // file contains a unique isa, use that.
diff --git a/cranelift/src/compile.rs b/cranelift/src/compile.rs
index 934a955480..7d888f3113 100644
--- a/cranelift/src/compile.rs
+++ b/cranelift/src/compile.rs
@@ -6,7 +6,7 @@ use cranelift_codegen::print_errors::pretty_error;
 use cranelift_codegen::settings::FlagsOrIsa;
 use cranelift_codegen::timing;
 use cranelift_codegen::Context;
-use cranelift_reader::parse_test;
+use cranelift_reader::{parse_test, ParseOptions};
 use std::path::Path;
 use std::path::PathBuf;
 
@@ -44,7 +44,8 @@ fn handle_module(
     fisa: FlagsOrIsa,
 ) -> Result<(), String> {
     let buffer = read_to_string(&path).map_err(|e| format!("{}: {}", name, e))?;
-    let test_file = parse_test(&buffer, None, None).map_err(|e| format!("{}: {}", name, e))?;
+    let test_file =
+        parse_test(&buffer, ParseOptions::default()).map_err(|e| format!("{}: {}", name, e))?;
 
     // If we have an isa from the command-line, use that. Otherwise if the
     // file contains a unique isa, use that.
diff --git a/cranelift/src/run.rs b/cranelift/src/run.rs
index a6c460083f..103c2a885f 100644
--- a/cranelift/src/run.rs
+++ b/cranelift/src/run.rs
@@ -1,11 +1,12 @@
 //! CLI tool to compile Cranelift IR files to native code in memory and execute them.
 
 use crate::utils::read_to_string;
-use cranelift_codegen::isa::TargetIsa;
+use cranelift_codegen::isa::{CallConv, TargetIsa};
 use cranelift_filetests::FunctionRunner;
 use cranelift_native::builder as host_isa_builder;
-use cranelift_reader::{parse_test, Details, IsaSpec};
+use cranelift_reader::{parse_test, Details, IsaSpec, ParseOptions};
 use std::path::PathBuf;
+use target_lexicon::Triple;
 use walkdir::WalkDir;
 
 pub fn run(files: Vec, flag_print: bool) -> Result<(), String> {
@@ -74,7 +75,11 @@ fn run_single_file(path: &PathBuf) -> Result<(), String> {
 
 /// Main body of `run_single_file` separated for testing
 fn run_file_contents(file_contents: String) -> Result<(), String> {
-    let test_file = parse_test(&file_contents, None, None).map_err(|e| e.to_string())?;
+    let options = ParseOptions {
+        default_calling_convention: CallConv::triple_default(&Triple::host()), // use the host's default calling convention
+        ..ParseOptions::default()
+    };
+    let test_file = parse_test(&file_contents, options).map_err(|e| e.to_string())?;
     for (func, Details { comments, .. }) in test_file.functions {
         if comments.iter().any(|c| c.text.contains("run")) {
             let isa = create_target_isa(&test_file.isa_spec)?;

From 9612c71fbd3c3f182e6fdec274e89e3ece923369 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 27 Aug 2019 13:23:55 -0700
Subject: [PATCH 2616/3084] Remove calling convention on FunctionRunner test

For this test to run on Windows, the explicit calling convention
must be removed so that the parser is free to use the host's
calling convention.
---
 cranelift/filetests/src/function_runner.rs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/cranelift/filetests/src/function_runner.rs b/cranelift/filetests/src/function_runner.rs
index 1c1272cce7..8837596ed9 100644
--- a/cranelift/filetests/src/function_runner.rs
+++ b/cranelift/filetests/src/function_runner.rs
@@ -99,7 +99,9 @@ mod test {
     #[test]
     fn nop() {
         let code = String::from(
-            "function %test() -> b8 system_v {
+            "
+            test run
+            function %test() -> b8 {
             ebb0:
                 nop
                 v1 = bconst.b8 true

From 26efc696c61190e41b95ae2ee6f7bdbc94c3d9d1 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 28 Aug 2019 16:05:54 +0200
Subject: [PATCH 2617/3084] Remove specific calling convention in clif-util's
 run test.

---
 cranelift/src/run.rs | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/cranelift/src/run.rs b/cranelift/src/run.rs
index 103c2a885f..a3cd94b25d 100644
--- a/cranelift/src/run.rs
+++ b/cranelift/src/run.rs
@@ -108,13 +108,12 @@ mod test {
     fn nop() {
         let code = String::from(
             "
-            function %test() -> b8 system_v {
+            function %test() -> b8 {
             ebb0:
                 nop
                 v1 = bconst.b8 true
                 return v1
             }
-            
             ; run
             ",
         );

From 04b10b3fde529518477369a71393110599e2184d Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Tue, 27 Aug 2019 14:25:21 +0200
Subject: [PATCH 2618/3084] Add feature flags to test files.

Cranelift can be compiled with feature flags which can change its output. To
accomodate changes of output related to feature flags, test file can now include
`feature "..."` and `feature ! "..."` directives in the preamble of the test
file.

The test runner would skip the test if the flag does not match the expectation
of the test case.
---
 cranelift/Cargo.toml              |  3 +-
 cranelift/filetests/Cargo.toml    |  3 ++
 cranelift/filetests/src/runone.rs | 28 +++++++++++++++--
 cranelift/reader/src/lexer.rs     | 52 +++++++++++++++++++++++++++++++
 cranelift/reader/src/lib.rs       |  2 +-
 cranelift/reader/src/parser.rs    | 33 ++++++++++++++++++--
 cranelift/reader/src/testfile.rs  | 16 ++++++++++
 7 files changed, 130 insertions(+), 7 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index a8f85c30b7..674db7cb92 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -48,7 +48,8 @@ walkdir = "2.2"
 default = ["disas", "wasm", "cranelift-codegen/all-arch"]
 disas = ["capstone"]
 wasm = ["wabt", "cranelift-wasm"]
-basic-blocks = ["cranelift-codegen/basic-blocks", "cranelift-frontend/basic-blocks", "cranelift-wasm/basic-blocks"]
+basic-blocks = ["cranelift-codegen/basic-blocks", "cranelift-frontend/basic-blocks",
+"cranelift-wasm/basic-blocks", "cranelift-filetests/basic-blocks"]
 
 # We want debug symbols on release binaries by default since it allows profiling
 # tools to give more accurate information. We can always strip them out later if
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 2b92833070..8913a8bfab 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -20,3 +20,6 @@ log = "0.4.6"
 memmap = "0.7.0"
 num_cpus = "1.8.0"
 region = "2.1.2"
+
+[features]
+basic-blocks = []
diff --git a/cranelift/filetests/src/runone.rs b/cranelift/filetests/src/runone.rs
index 4a2071b2db..1bb7d1c7f2 100644
--- a/cranelift/filetests/src/runone.rs
+++ b/cranelift/filetests/src/runone.rs
@@ -8,8 +8,7 @@ use cranelift_codegen::print_errors::pretty_verifier_error;
 use cranelift_codegen::settings::Flags;
 use cranelift_codegen::timing;
 use cranelift_codegen::verify_function;
-use cranelift_reader::IsaSpec;
-use cranelift_reader::{parse_test, ParseOptions};
+use cranelift_reader::{parse_test, Feature, IsaSpec, ParseOptions};
 use log::info;
 use std::borrow::Cow;
 use std::fs;
@@ -53,6 +52,31 @@ pub fn run(path: &Path, passes: Option<&[String]>, target: Option<&str>) -> Test
         }
     };
 
+    for feature in testfile.features.iter() {
+        let (flag, test_expect) = match feature {
+            Feature::With(name) => (name, true),
+            Feature::Without(name) => (name, false),
+        };
+        let cranelift_has = match flag {
+            // Add any cranelift feature flag here, and make sure that it is forwarded to the
+            // cranelift-filetest crate in the top-level Cargo.toml.
+            &"basic-blocks" => cfg!(feature = "basic-blocks"),
+            _ => {
+                return Err(format!(
+                    r#"{:?}: Unknown feature flag named "{}""#,
+                    path, flag
+                ))
+            }
+        };
+        if cranelift_has != test_expect {
+            println!(
+                r#"skipping test {:?}: non-matching feature flag "{}""#,
+                path, flag
+            );
+            return Ok(started.elapsed());
+        }
+    }
+
     if testfile.functions.is_empty() {
         return Err("no functions found".to_string());
     }
diff --git a/cranelift/reader/src/lexer.rs b/cranelift/reader/src/lexer.rs
index 76327a7a8d..465e79e97f 100644
--- a/cranelift/reader/src/lexer.rs
+++ b/cranelift/reader/src/lexer.rs
@@ -27,6 +27,7 @@ pub enum Token<'a> {
     Dot,                  // '.'
     Colon,                // ':'
     Equal,                // '='
+    Not,                  // '!'
     Arrow,                // '->'
     Float(&'a str),       // Floating point immediate
     Integer(&'a str),     // Integer immediate
@@ -42,6 +43,7 @@ pub enum Token<'a> {
     SigRef(u32),          // sig2
     UserRef(u32),         // u345
     Name(&'a str),        // %9arbitrary_alphanum, %x3, %0, %function ...
+    String(&'a str),      // "abritrary quoted string with no escape" ...
     HexSequence(&'a str), // #89AF
     Identifier(&'a str),  // Unrecognized identifier (opcode, enumerator, ...)
     SourceLoc(&'a str),   // @00c7
@@ -401,6 +403,27 @@ impl<'a> Lexer<'a> {
         token(Token::Name(&self.source[begin..end]), loc)
     }
 
+    /// Scan for a multi-line quoted string with no escape character.
+    fn scan_string(&mut self) -> Result, LocatedError> {
+        let loc = self.loc();
+        let begin = self.pos + 1;
+
+        assert_eq!(self.lookahead, Some('"'));
+
+        while let Some(c) = self.next_ch() {
+            if c == '"' {
+                break;
+            }
+        }
+
+        let end = self.pos;
+        if self.lookahead != Some('"') {
+            return error(LexError::InvalidChar, self.loc());
+        }
+        self.next_ch();
+        token(Token::String(&self.source[begin..end]), loc)
+    }
+
     fn scan_hex_sequence(&mut self) -> Result, LocatedError> {
         let loc = self.loc();
         let begin = self.pos + 1;
@@ -452,6 +475,7 @@ impl<'a> Lexer<'a> {
                 Some('.') => Some(self.scan_char(Token::Dot)),
                 Some(':') => Some(self.scan_char(Token::Colon)),
                 Some('=') => Some(self.scan_char(Token::Equal)),
+                Some('!') => Some(self.scan_char(Token::Not)),
                 Some('+') => Some(self.scan_number()),
                 Some('-') => {
                     if self.looking_at("->") {
@@ -463,6 +487,7 @@ impl<'a> Lexer<'a> {
                 Some(ch) if ch.is_digit(10) => Some(self.scan_number()),
                 Some(ch) if ch.is_alphabetic() => Some(self.scan_word()),
                 Some('%') => Some(self.scan_name()),
+                Some('"') => Some(self.scan_string()),
                 Some('#') => Some(self.scan_hex_sequence()),
                 Some('@') => Some(self.scan_srcloc()),
                 Some(ch) if ch.is_whitespace() => {
@@ -633,6 +658,33 @@ mod tests {
         assert_eq!(lex.next(), token(Token::Name("_"), 1));
     }
 
+    #[test]
+    fn lex_strings() {
+        let mut lex = Lexer::new(
+            r#"""  "0" "x3""function" "123 abc" "\" "start
+                    and end on
+                    different lines" "#,
+        );
+
+        assert_eq!(lex.next(), token(Token::String(""), 1));
+        assert_eq!(lex.next(), token(Token::String("0"), 1));
+        assert_eq!(lex.next(), token(Token::String("x3"), 1));
+        assert_eq!(lex.next(), token(Token::String("function"), 1));
+        assert_eq!(lex.next(), token(Token::String("123 abc"), 1));
+        assert_eq!(lex.next(), token(Token::String(r#"\"#), 1));
+        assert_eq!(
+            lex.next(),
+            token(
+                Token::String(
+                    r#"start
+                    and end on
+                    different lines"#
+                ),
+                1
+            )
+        );
+    }
+
     #[test]
     fn lex_userrefs() {
         let mut lex = Lexer::new("u0 u1 u234567890 u9:8765");
diff --git a/cranelift/reader/src/lib.rs b/cranelift/reader/src/lib.rs
index 871bd80b2c..f0922bf884 100644
--- a/cranelift/reader/src/lib.rs
+++ b/cranelift/reader/src/lib.rs
@@ -31,7 +31,7 @@ pub use crate::isaspec::{parse_options, IsaSpec};
 pub use crate::parser::{parse_functions, parse_test, ParseOptions};
 pub use crate::sourcemap::SourceMap;
 pub use crate::testcommand::{TestCommand, TestOption};
-pub use crate::testfile::{Comment, Details, TestFile};
+pub use crate::testfile::{Comment, Details, Feature, TestFile};
 
 mod error;
 mod isaspec;
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index a0fa0f127c..df0fc18a0e 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -5,7 +5,7 @@ use crate::isaspec;
 use crate::lexer::{LexError, Lexer, LocatedError, LocatedToken, Token};
 use crate::sourcemap::SourceMap;
 use crate::testcommand::TestCommand;
-use crate::testfile::{Comment, Details, TestFile};
+use crate::testfile::{Comment, Details, Feature, TestFile};
 use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::entities::AnyEntity;
@@ -82,6 +82,7 @@ pub fn parse_test<'a>(text: &'a str, options: ParseOptions<'a>) -> ParseResult(text: &'a str, options: ParseOptions<'a>) -> ParseResult Parser<'a> {
         }
     }
 
+    /// Parse a list of expected features that Cranelift should be compiled with, or without.
+    pub fn parse_cranelift_features(&mut self) -> ParseResult>> {
+        let mut list = Vec::new();
+        while self.token() == Some(Token::Identifier("feature")) {
+            self.consume();
+            let has = !self.optional(Token::Not);
+            match (self.token(), has) {
+                (Some(Token::String(flag)), true) => list.push(Feature::With(flag)),
+                (Some(Token::String(flag)), false) => list.push(Feature::Without(flag)),
+                (tok, _) => {
+                    return err!(
+                        self.loc,
+                        format!("Expected feature flag string, got {:?}", tok)
+                    )
+                }
+            }
+            self.consume();
+        }
+        Ok(list)
+    }
+
     /// Parse a list of function definitions.
     ///
     /// This is the top-level parse function matching the whole contents of a file.
@@ -2992,12 +3015,14 @@ mod tests {
     #[test]
     fn test_file() {
         let tf = parse_test(
-            "; before
+            r#"; before
                              test cfg option=5
                              test verify
                              set enable_float=false
+                             feature "foo"
+                             feature !"bar"
                              ; still preamble
-                             function %comment() system_v {}",
+                             function %comment() system_v {}"#,
             ParseOptions::default(),
         )
         .unwrap();
@@ -3011,6 +3036,8 @@ mod tests {
             }
             _ => panic!("unexpected ISAs"),
         }
+        assert_eq!(tf.features[0], Feature::With(&"foo"));
+        assert_eq!(tf.features[1], Feature::Without(&"bar"));
         assert_eq!(tf.preamble_comments.len(), 2);
         assert_eq!(tf.preamble_comments[0].text, "; before");
         assert_eq!(tf.preamble_comments[1].text, "; still preamble");
diff --git a/cranelift/reader/src/testfile.rs b/cranelift/reader/src/testfile.rs
index 506694586f..68c7d30a93 100644
--- a/cranelift/reader/src/testfile.rs
+++ b/cranelift/reader/src/testfile.rs
@@ -20,6 +20,8 @@ pub struct TestFile<'a> {
     pub commands: Vec>,
     /// `isa bar ...` lines.
     pub isa_spec: IsaSpec,
+    /// `feature ...` lines
+    pub features: Vec>,
     /// Comments appearing before the first function.
     /// These are all tagged as 'Function' scope for lack of a better entity.
     pub preamble_comments: Vec>,
@@ -55,3 +57,17 @@ pub struct Comment<'a> {
     /// Text of the comment, including the leading `;`.
     pub text: &'a str,
 }
+
+/// A cranelift feature in a test file preamble.
+///
+/// This represents the expectation of the test case. Before running any of the
+/// functions of the test file, the feature set should be compared with the
+/// feature set used to compile Cranelift. If there is any differences, then the
+/// test file should be skipped.
+#[derive(PartialEq, Eq, Debug)]
+pub enum Feature<'a> {
+    /// `feature "..."` lines
+    With(&'a str),
+    /// `feature ! "..."` lines.
+    Without(&'a str),
+}

From 4e7226ddcace71fcd0f4ecb1658aaeac95f38fa1 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 28 Aug 2019 09:18:43 -0700
Subject: [PATCH 2619/3084] Fix documentation warnings in ConstantPool

---
 cranelift/codegen/src/ir/constant.rs | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/src/ir/constant.rs b/cranelift/codegen/src/ir/constant.rs
index c44dcd3379..50e93df1ab 100644
--- a/cranelift/codegen/src/ir/constant.rs
+++ b/cranelift/codegen/src/ir/constant.rs
@@ -19,8 +19,9 @@ pub type ConstantOffset = u32;
 /// Inner type for storing data and offset together in the constant pool. The offset is optional
 /// because it must be set relative to the function code size (i.e. constants are emitted after the
 /// function body); because the function is not yet compiled when constants are inserted,
-/// [set_offset](ir::ConstantPool::set_offset) must be called once a constant's offset from the
-/// beginning of the function is known (see [relaxation.rs](binemit::relaxation)).
+/// [`set_offset`](crate::ir::ConstantPool::set_offset) must be called once a constant's
+/// offset from the beginning of the function is known (see
+/// [`relaxation.rs`](crate::binemit::relaxation)).
 #[derive(Clone)]
 pub struct ConstantPoolEntry {
     data: ConstantData,
@@ -43,8 +44,9 @@ impl ConstantPoolEntry {
     }
 }
 
-/// Maintains the mapping between a constant handle (i.e. [Constant](ir::entities::Constant)) and
-/// its constant data (i.e. [ConstantData](ir::constant::ConstantData)).
+/// Maintains the mapping between a constant handle (i.e.
+/// [`Constant`](crate::ir::Constant)) and its constant data (i.e.
+/// [`ConstantData`](crate::ir::ConstantData)).
 #[derive(Clone)]
 pub struct ConstantPool {
     /// This mapping maintains the insertion order as long as Constants are created with sequentially increasing integers.

From 9eb1847d806d89cead9530d07017980374476016 Mon Sep 17 00:00:00 2001
From: Till Schneidereit 
Date: Thu, 29 Aug 2019 11:31:38 +0200
Subject: [PATCH 2620/3084] Add Azure Pipelines CI setup (#948)

Add Azure Pipelines CI setup
---
 .azure-pipelines.yml                 | 156 +++++++++++++++++++++++++++
 cranelift/ci/azure-build-release.yml |  66 ++++++++++++
 cranelift/ci/azure-install-rust.yml  |  33 ++++++
 3 files changed, 255 insertions(+)
 create mode 100644 .azure-pipelines.yml
 create mode 100644 cranelift/ci/azure-build-release.yml
 create mode 100644 cranelift/ci/azure-install-rust.yml

diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml
new file mode 100644
index 0000000000..a18a45652e
--- /dev/null
+++ b/.azure-pipelines.yml
@@ -0,0 +1,156 @@
+name: $(Build.SourceBranch)-$(date:yyyy-MM-dd)$(rev:.r)
+trigger:
+  branches:
+    include:
+    - 'master'
+  tags:
+    include:
+    - '*'
+    exclude:
+      - 'dev'
+
+jobs:
+- job: rustfmt
+  pool:
+    vmImage: 'macos-10.14'
+  steps:
+  - checkout: self
+    submodules: true
+  - template: ci/azure-install-rust.yml
+  - script: rustup component add rustfmt
+    displayName: Add rustfmt
+  - script: cargo fmt --all -- --check
+    displayName: Check formatting
+  variables:
+    toolchain: stable
+
+# Smoke test to build docs on one builder, using OSX for now since it's the
+# fastest
+- job: docs
+  pool:
+    vmImage: 'macos-10.14'
+  steps:
+  - checkout: self
+    submodules: true
+  - template: ci/azure-install-rust.yml
+  - script: cargo doc
+    displayName: Build documentation
+  - script: cargo install cargo-deadlinks
+    displayName: Install cargo-deadlinks
+  - bash: |
+      find ./target/doc -maxdepth 1 -type d -name "cranelift*" | xargs -I{} cargo deadlinks --dir {}
+    displayName: Run cargo-deadlinks
+  variables:
+    toolchain: nightly
+
+- job: Test
+  strategy:
+    matrix:
+      windows-earliest:
+        imageName: 'vs2017-win2016'
+        toolchain: '1.37.0'
+      linux-earliest:
+        imageName: 'ubuntu-16.04'
+        toolchain: '1.37.0'
+      mac-earliest:
+        imageName: 'macos-10.14'
+        toolchain: '1.37.0'
+      mac-stable:
+        imageName: 'macos-10.14'
+        toolchain: stable
+      mac-beta:
+        imageName: 'macos-10.14'
+        toolchain: beta
+      mac-nightly:
+        imageName: 'macos-10.14'
+        toolchain: nightly
+
+  pool:
+    vmImage: $(imageName)
+
+  steps:
+    - checkout: self
+      submodules: true
+    - template: ci/azure-install-rust.yml
+
+    - script: cargo fetch
+      displayName: Fetch cargo dependencies
+
+    - script: cargo build
+      displayName: Cargo build
+
+    - bash: cargo test --all
+      displayName: Cargo test
+      env:
+        RUST_BACKTRACE: 1
+
+    # Ensure fuzzer works by running it with a single input
+    - bash: cargo install cargo-fuzz
+      displayName: Install cargo-fuzz
+      condition: and(succeeded(), eq(variables['toolchain'], 'nightly'))
+    - bash: |
+        fuzz_module="ffaefab69523eb11935a9b420d58826c8ea65c4c"
+        cargo fuzz run fuzz_translate_module \
+        "fuzz/corpus/fuzz_translate_module/$fuzz_module"
+      displayName: Run cargo-fuzz
+      env:
+        RUST_BACKTRACE: 1
+      condition: and(succeeded(), eq(variables['toolchain'], 'nightly'))
+
+- job: Build
+  strategy:
+    matrix:
+      windows:
+        imageName: 'vs2017-win2016'
+        # Statically link against msvcrt to produce slightly more portable
+        # binaries on Windows by reducing our binary compatibility requirements.
+        RUSTFLAGS: -Ctarget-feature=+crt-static
+      mac:
+        imageName: 'macos-10.14'
+        # Lower the deployment target from our build image in an attempt to
+        # build more portable binaries that run on older releases. Note that
+        # 10.9 here is arbitrarily chosen and just happens to be the lowest that
+        # works at this time. Raising this is probably fine.
+        MACOSX_DEPLOYMENT_TARGET: 10.9
+  variables:
+    toolchain: '1.37.0'
+  pool:
+    vmImage: $(imageName)
+  # We try to be compatible with beta and nightly, but they occasionally
+  # fail, so we don't allow them to hold up people using stable.
+  continueOnError: $[ne(variables['toolchain'], 'stable')]
+  steps:
+  - template: ci/azure-build-release.yml
+
+# Build the Linux release binary in an older Linux container (in this case
+# Centos 6)
+- job: Build_linux
+  variables:
+    toolchain: '1.37.0'
+  container:
+    image: centos:6
+    options: "--name ci-container -v /usr/bin/docker:/tmp/docker:ro"
+  steps:
+  # We're executing in the container as non-root but `yum` requires root. We
+  # need to install `sudo` but to do that we need `sudo`. Do a bit of a weird
+  # hack where we use the host `docker` executable to re-execute in our own
+  # container with the root user to install `sudo`
+  - bash: /tmp/docker exec -t -u 0 ci-container sh -c "yum install -y sudo"
+    displayName: Configure sudo
+
+  # See https://edwards.sdsu.edu/research/c11-on-centos-6/ for where these
+  # various commands came from.
+  - bash: |
+      set -e
+      sudo yum install -y centos-release-scl cmake xz
+      sudo yum install -y devtoolset-8-gcc devtoolset-8-binutils devtoolset-8-gcc-c++
+      echo "##vso[task.prependpath]/opt/rh/devtoolset-8/root/usr/bin"
+    displayName: Install system dependencies
+
+  # Delete `libstdc++.so` to force gcc to link against `libstdc++.a` instead.
+  # This is a hack and not the right way to do this, but it ends up doing the
+  # right thing for now.
+  - bash: sudo rm -f /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/libstdc++.so
+    displayName: Force a static libstdc++
+
+  - template: ci/azure-build-release.yml
diff --git a/cranelift/ci/azure-build-release.yml b/cranelift/ci/azure-build-release.yml
new file mode 100644
index 0000000000..6400eed0eb
--- /dev/null
+++ b/cranelift/ci/azure-build-release.yml
@@ -0,0 +1,66 @@
+steps:
+- checkout: self
+  submodules: true
+
+- template: azure-install-rust.yml
+
+- bash: echo "##vso[task.setvariable variable=RUSTC_VERSION;]`rustc --version`"
+  displayName: Set rustc version string for caching
+
+- bash: cargo build --release
+  displayName: Cargo build
+
+# Test what we're about to release in release mode itself.
+- bash: cargo test --release --all
+  displayName: Cargo test
+  env:
+    RUST_BACKTRACE: 1
+
+- bash: |
+    echo "##vso[task.setvariable variable=tagName;]`echo $BUILD_SOURCEBRANCH | sed -e 's|refs/tags/||'`"
+  displayName: Set tag name
+  condition: startsWith(variables['Build.SourceBranch'], 'refs/tags/')
+- bash: |
+    echo "##vso[task.setvariable variable=tagName;]dev"
+  displayName: Set tag name to "dev"
+  condition: not(startsWith(variables['Build.SourceBranch'], 'refs/tags/'))
+
+- bash: echo "##vso[task.setvariable variable=basename;]cranelift-$(tagName)-x86_64-windows"
+  displayName: Configure basename var
+  condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
+- bash: echo "##vso[task.setvariable variable=basename;]cranelift-$(tagName)-x86_64-macos"
+  displayName: Configure basename var
+  condition: and(succeeded(), eq(variables['Agent.OS'], 'Darwin'))
+- bash: echo "##vso[task.setvariable variable=basename;]cranelift-$(tagName)-x86_64-linux"
+  displayName: Configure basename var
+  condition: and(succeeded(), eq( variables['Agent.OS'], 'Linux' ))
+
+- bash: |
+    set -e
+    mkdir -p $BUILD_BINARIESDIRECTORY/$BASENAME
+    if [ "$AGENT_OS" = "Windows_NT" ]; then
+      ext=.exe
+    fi
+    cp LICENSE README.md target/release/clif-util$ext $BUILD_BINARIESDIRECTORY/$BASENAME
+  displayName: Copy binaries
+
+- task: ArchiveFiles@2
+  inputs:
+    rootFolderOrFile: $(Build.BinariesDirectory)/$(basename)
+    archiveType: 'zip'
+    archiveFile: '$(Build.ArtifactStagingDirectory)/$(basename).zip'
+  displayName: Archive files (Win)
+  condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
+- task: ArchiveFiles@2
+  inputs:
+    rootFolderOrFile: $(Build.BinariesDirectory)/$(basename)
+    archiveType: 'tar'
+    tarCompression: 'xz'
+    archiveFile: '$(Build.ArtifactStagingDirectory)/$(basename).tar.xz'
+  displayName: Archive files (Unix)
+  condition: and(succeeded(), ne(variables['Agent.OS'], 'Windows_NT'))
+- task: PublishPipelineArtifact@1
+  inputs:
+    path: $(Build.ArtifactStagingDirectory)/
+    artifactName: 'bundle-$(Agent.OS)'
+
diff --git a/cranelift/ci/azure-install-rust.yml b/cranelift/ci/azure-install-rust.yml
new file mode 100644
index 0000000000..64e502b775
--- /dev/null
+++ b/cranelift/ci/azure-install-rust.yml
@@ -0,0 +1,33 @@
+steps:
+  # Rustup is currently installed on Windows and Linux, but not macOS.
+  # It is installed in /usr/local/cargo/bin/ or C:\Program Files\Rust\.cargo\bin\
+  # This steps ensures that rustup is installed, mainly for macOS, or if the
+  # azure image changes in the future.
+  - bash: |
+      set -ex
+      if [ -x "`command -v rustup`" ]; then
+        echo `command -v rustup` `rustup -V` already installed
+        rustup self update
+      else
+        if [ "$AGENT_OS" = "Windows_NT" ]; then
+          curl -sSf -o rustup-init.exe https://win.rustup.rs
+          ./rustup-init.exe -y --default-toolchain $TOOLCHAIN
+          echo "##vso[task.prependpath]$USERPROFILE/.cargo/bin"
+        else
+          curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain $TOOLCHAIN
+          echo "##vso[task.prependpath]$HOME/.cargo/bin"
+        fi
+      fi
+    displayName: Install rustup
+
+  - bash: |
+      set -ex
+      rustup update $TOOLCHAIN
+      rustup default $TOOLCHAIN
+    displayName: Install rust
+
+  - bash: |
+      set -ex
+      rustc -Vv
+      cargo -V
+    displayName: Query rust and cargo versions

From bb87f1a54ade651f179c423c76ec0ba36ff8ce6f Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Thu, 1 Aug 2019 15:06:49 +0200
Subject: [PATCH 2621/3084] Add EntryRegDiversions to record diversions for
 each block entry.

---
 cranelift/codegen/src/binemit/mod.rs        |   2 +-
 cranelift/codegen/src/binemit/relaxation.rs |   5 +-
 cranelift/codegen/src/binemit/shrink.rs     |   4 +-
 cranelift/codegen/src/ir/function.rs        |  14 ++-
 cranelift/codegen/src/regalloc/coloring.rs  |   1 +
 cranelift/codegen/src/regalloc/diversion.rs | 107 +++++++++++++++++++-
 cranelift/codegen/src/regalloc/mod.rs       |   2 +-
 cranelift/codegen/src/value_label.rs        |   2 +-
 cranelift/codegen/src/verifier/locations.rs |   5 +-
 9 files changed, 126 insertions(+), 16 deletions(-)

diff --git a/cranelift/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
index e5d0cbbec5..1287dcd439 100644
--- a/cranelift/codegen/src/binemit/mod.rs
+++ b/cranelift/codegen/src/binemit/mod.rs
@@ -176,7 +176,7 @@ where
 {
     let mut divert = RegDiversions::new();
     for ebb in func.layout.ebbs() {
-        divert.clear();
+        divert.at_ebb(&func.entry_diversions, ebb);
         debug_assert_eq!(func.offsets[ebb], sink.offset());
         for inst in func.layout.ebb_insts(ebb) {
             emit_inst(func, inst, &mut divert, sink, isa);
diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index 1fdf51ea0e..6648369cf3 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -74,7 +74,7 @@ pub fn relax_branches(
     {
         let mut cur = FuncCursor::new(func);
         while let Some(ebb) = cur.next_ebb() {
-            divert.clear();
+            divert.at_ebb(&cur.func.entry_diversions, ebb);
             cur.func.offsets[ebb] = offset;
             while let Some(inst) = cur.next_inst() {
                 divert.apply(&cur.func.dfg[inst]);
@@ -93,7 +93,8 @@ pub fn relax_branches(
         // Visit all instructions in layout order.
         let mut cur = FuncCursor::new(func);
         while let Some(ebb) = cur.next_ebb() {
-            divert.clear();
+            divert.at_ebb(&cur.func.entry_diversions, ebb);
+
             // Record the offset for `ebb` and make sure we iterate until offsets are stable.
             if cur.func.offsets[ebb] != offset {
                 cur.func.offsets[ebb] = offset;
diff --git a/cranelift/codegen/src/binemit/shrink.rs b/cranelift/codegen/src/binemit/shrink.rs
index 0f9838c8ab..084ed2bc3d 100644
--- a/cranelift/codegen/src/binemit/shrink.rs
+++ b/cranelift/codegen/src/binemit/shrink.rs
@@ -20,7 +20,9 @@ pub fn shrink_instructions(func: &mut Function, isa: &dyn TargetIsa) {
     let mut divert = RegDiversions::new();
 
     for ebb in func.layout.ebbs() {
-        divert.clear();
+        // Load diversions from predecessors.
+        divert.at_ebb(&func.entry_diversions, ebb);
+
         for inst in func.layout.ebb_insts(ebb) {
             let enc = func.encodings[inst];
             if enc.is_legal() {
diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index 1dcdeee91c..00827240d9 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -14,7 +14,7 @@ use crate::ir::{
 use crate::ir::{EbbOffsets, InstEncodings, SourceLocs, StackSlots, ValueLocations};
 use crate::ir::{JumpTableOffsets, JumpTables};
 use crate::isa::{CallConv, EncInfo, Encoding, Legalize, TargetIsa};
-use crate::regalloc::RegDiversions;
+use crate::regalloc::{EntryRegDiversions, RegDiversions};
 use crate::value_label::ValueLabelsRanges;
 use crate::write::write_function;
 use core::fmt;
@@ -62,6 +62,12 @@ pub struct Function {
     /// Location assigned to every value.
     pub locations: ValueLocations,
 
+    /// Non-default locations assigned to value at the entry of basic blocks.
+    ///
+    /// At the entry of each basic block, we might have values which are not in their default
+    /// ValueLocation. This field records these register-to-register moves as Diversions.
+    pub entry_diversions: EntryRegDiversions,
+
     /// Code offsets of the EBB headers.
     ///
     /// This information is only transiently available after the `binemit::relax_branches` function
@@ -94,6 +100,7 @@ impl Function {
             layout: Layout::new(),
             encodings: SecondaryMap::new(),
             locations: SecondaryMap::new(),
+            entry_diversions: EntryRegDiversions::new(),
             offsets: SecondaryMap::new(),
             jt_offsets: SecondaryMap::new(),
             srclocs: SecondaryMap::new(),
@@ -112,6 +119,7 @@ impl Function {
         self.layout.clear();
         self.encodings.clear();
         self.locations.clear();
+        self.entry_diversions.clear();
         self.offsets.clear();
         self.jt_offsets.clear();
         self.srclocs.clear();
@@ -198,10 +206,12 @@ impl Function {
             !self.offsets.is_empty(),
             "Code layout must be computed first"
         );
+        let mut divert = RegDiversions::new();
+        divert.at_ebb(&self.entry_diversions, ebb);
         InstOffsetIter {
             encinfo: encinfo.clone(),
             func: self,
-            divert: RegDiversions::new(),
+            divert,
             encodings: &self.encodings,
             offset: self.offsets[ebb],
             iter: self.layout.ebb_insts(ebb),
diff --git a/cranelift/codegen/src/regalloc/coloring.rs b/cranelift/codegen/src/regalloc/coloring.rs
index fa158863a4..0df5795c11 100644
--- a/cranelift/codegen/src/regalloc/coloring.rs
+++ b/cranelift/codegen/src/regalloc/coloring.rs
@@ -199,6 +199,7 @@ impl<'a> Context<'a> {
             self.domtree,
         );
 
+        self.divert.at_ebb(&self.cur.func.entry_diversions, ebb);
         if self.cur.func.layout.entry_block() == Some(ebb) {
             // Parameters on the entry block have ABI constraints.
             self.color_entry_params(tracker.live())
diff --git a/cranelift/codegen/src/regalloc/diversion.rs b/cranelift/codegen/src/regalloc/diversion.rs
index 6e9b1f23cd..12461c5cea 100644
--- a/cranelift/codegen/src/regalloc/diversion.rs
+++ b/cranelift/codegen/src/regalloc/diversion.rs
@@ -9,10 +9,11 @@
 
 use crate::fx::FxHashMap;
 use crate::hash_map::{Entry, Iter};
+use crate::ir::{Ebb, StackSlot, Value, ValueLoc, ValueLocations};
 use crate::ir::{InstructionData, Opcode};
-use crate::ir::{StackSlot, Value, ValueLoc, ValueLocations};
 use crate::isa::{RegInfo, RegUnit};
 use core::fmt;
+use cranelift_entity::{SparseMap, SparseMapValue};
 
 /// A diversion of a value from its original location to a new register or stack location.
 ///
@@ -38,10 +39,23 @@ impl Diversion {
 }
 
 /// Keep track of diversions in an EBB.
+#[derive(Clone)]
 pub struct RegDiversions {
     current: FxHashMap,
 }
 
+/// Keep track of diversions at the entry of EBB.
+#[derive(Clone)]
+struct EntryRegDiversionsValue {
+    key: Ebb,
+    divert: RegDiversions,
+}
+
+/// Map EBB to their matching RegDiversions at basic blocks entry.
+pub struct EntryRegDiversions {
+    map: SparseMap,
+}
+
 impl RegDiversions {
     /// Create a new empty diversion tracker.
     pub fn new() -> Self {
@@ -50,7 +64,7 @@ impl RegDiversions {
         }
     }
 
-    /// Clear the tracker, preparing for a new EBB.
+    /// Clear the content of the diversions, to reset the state of the compiler.
     pub fn clear(&mut self) {
         self.current.clear()
     }
@@ -92,7 +106,7 @@ impl RegDiversions {
     /// Record any kind of move.
     ///
     /// The `from` location must match an existing `to` location, if any.
-    pub fn divert(&mut self, value: Value, from: ValueLoc, to: ValueLoc) {
+    fn divert(&mut self, value: Value, from: ValueLoc, to: ValueLoc) {
         debug_assert!(from.is_assigned() && to.is_assigned());
         match self.current.entry(value) {
             Entry::Occupied(mut e) => {
@@ -163,9 +177,92 @@ impl RegDiversions {
         self.current.remove(&value).map(|d| d.to)
     }
 
+    /// Resets the state of the current diversions to the recorded diversions at the entry of the
+    /// given `ebb`. The recoded diversions is available after coloring on `func.entry_diversions`
+    /// field.
+    pub fn at_ebb(&mut self, entry_diversions: &EntryRegDiversions, ebb: Ebb) {
+        self.clear();
+        if let Some(entry_divert) = entry_diversions.map.get(ebb) {
+            let iter = entry_divert.divert.current.iter();
+            self.current.extend(iter);
+        }
+    }
+
+    /// Copy the current state of the diversions, and save it for the entry of the `ebb` given as
+    /// argument.
+    ///
+    /// Note: This function can only be called once on an `ebb` with a given `entry_diversions`
+    /// argument, otherwise it would panic.
+    pub fn save_for_ebb(&mut self, entry_diversions: &mut EntryRegDiversions, target: Ebb) {
+        // No need to save anything if there is no diversions to be recorded.
+        if self.is_empty() {
+            return;
+        }
+        debug_assert!(!entry_diversions.map.contains_key(target));
+        let iter = self.current.iter();
+        let mut entry_divert = RegDiversions::new();
+        entry_divert.current.extend(iter);
+        entry_diversions.map.insert(EntryRegDiversionsValue {
+            key: target,
+            divert: entry_divert,
+        });
+    }
+
+    /// Check that the recorded entry for a given `ebb` matches what is recorded in the
+    /// `entry_diversions`.
+    pub fn check_ebb_entry(&self, entry_diversions: &EntryRegDiversions, target: Ebb) -> bool {
+        let entry_divert = match entry_diversions.map.get(target) {
+            Some(entry_divert) => entry_divert,
+            None => return self.is_empty(),
+        };
+
+        if entry_divert.divert.current.len() != self.current.len() {
+            return false;
+        }
+
+        for (val, _) in entry_divert.divert.current.iter() {
+            if !self.current.contains_key(val) {
+                return false;
+            }
+        }
+        return true;
+    }
+
     /// Return an object that can display the diversions.
     pub fn display<'a, R: Into>>(&'a self, regs: R) -> DisplayDiversions<'a> {
-        DisplayDiversions(self, regs.into())
+        DisplayDiversions(&self, regs.into())
+    }
+}
+
+impl EntryRegDiversions {
+    /// Create a new empty entry diversion, to associate diversions to each EBB entry.
+    pub fn new() -> Self {
+        EntryRegDiversions {
+            map: SparseMap::new(),
+        }
+    }
+
+    pub fn clear(&mut self) {
+        self.map.clear();
+    }
+}
+
+impl Clone for EntryRegDiversions {
+    /// The Clone trait is required by `ir::Function`.
+    fn clone(&self) -> Self {
+        let mut tmp = Self::new();
+        for v in self.map.values() {
+            tmp.map.insert(v.clone());
+        }
+        tmp
+    }
+}
+
+/// Implement `SparseMapValue`, as required to make use of a `SparseMap` for mapping the entry
+/// diversions for each EBB.
+impl SparseMapValue for EntryRegDiversionsValue {
+    fn key(&self) -> Ebb {
+        self.key
     }
 }
 
@@ -175,7 +272,7 @@ pub struct DisplayDiversions<'a>(&'a RegDiversions, Option<&'a RegInfo>);
 impl<'a> fmt::Display for DisplayDiversions<'a> {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         write!(f, "{{")?;
-        for (value, div) in self.0.iter() {
+        for (value, div) in self.0.current.iter() {
             write!(
                 f,
                 " {}: {} -> {}",
diff --git a/cranelift/codegen/src/regalloc/mod.rs b/cranelift/codegen/src/regalloc/mod.rs
index cfa3ca7323..4df7cc9cef 100644
--- a/cranelift/codegen/src/regalloc/mod.rs
+++ b/cranelift/codegen/src/regalloc/mod.rs
@@ -20,6 +20,6 @@ mod solver;
 mod spilling;
 
 pub use self::context::Context;
-pub use self::diversion::RegDiversions;
+pub use self::diversion::{EntryRegDiversions, RegDiversions};
 pub use self::register_set::RegisterSet;
 pub use self::safepoint::emit_stackmaps;
diff --git a/cranelift/codegen/src/value_label.rs b/cranelift/codegen/src/value_label.rs
index 60ea7b342a..2bd3bdc13d 100644
--- a/cranelift/codegen/src/value_label.rs
+++ b/cranelift/codegen/src/value_label.rs
@@ -118,7 +118,7 @@ where
     let mut tracked_values: Vec<(Value, ValueLabel, u32, ValueLoc)> = Vec::new();
     let mut divert = RegDiversions::new();
     for ebb in ebbs {
-        divert.clear();
+        divert.at_ebb(&func.entry_diversions, ebb);
         let mut last_srcloc: Option = None;
         for (offset, inst, size) in func.inst_offsets(ebb, &encinfo) {
             divert.apply(&func.dfg[inst]);
diff --git a/cranelift/codegen/src/verifier/locations.rs b/cranelift/codegen/src/verifier/locations.rs
index bcf006e7f5..8c290bb281 100644
--- a/cranelift/codegen/src/verifier/locations.rs
+++ b/cranelift/codegen/src/verifier/locations.rs
@@ -51,9 +51,8 @@ impl<'a> LocationVerifier<'a> {
         let mut divert = RegDiversions::new();
 
         for ebb in self.func.layout.ebbs() {
-            // Diversions are reset at the top of each EBB. No diversions can exist across control
-            // flow edges.
-            divert.clear();
+            divert.at_ebb(&self.func.entry_diversions, ebb);
+
             for inst in self.func.layout.ebb_insts(ebb) {
                 let enc = self.func.encodings[inst];
 

From 381578311ca70ae5dc7cc6b900ec08c1c7aedc8a Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 30 Aug 2019 18:44:35 +0200
Subject: [PATCH 2622/3084] Split edges to have a block to add regmove & copy
 instructions.

When using basic block instructions cannot be added in-between jump instructions which are ending basic blocks. These changes create extra basic blocks such that extra space is available for the spilling and moving registers where they are expected.
---
 cranelift/codegen/src/context.rs              |   2 +-
 .../codegen/src/regalloc/branch_splitting.rs  | 197 ++++++++++++++++++
 cranelift/codegen/src/regalloc/context.rs     |  10 +-
 cranelift/codegen/src/regalloc/mod.rs         |   1 +
 .../filetests/regalloc/coalesce-bb.clif       | 158 ++++++++++++++
 .../filetests/regalloc/coalesce.clif          |   1 +
 .../filetests/regalloc/reload-208-bb.clif     | 111 ++++++++++
 .../filetests/regalloc/reload-208.clif        |   1 +
 .../filetests/regalloc/x86-regres-bb.clif     |  49 +++++
 .../filetests/regalloc/x86-regres.clif        |   2 +-
 10 files changed, 529 insertions(+), 3 deletions(-)
 create mode 100644 cranelift/codegen/src/regalloc/branch_splitting.rs
 create mode 100644 cranelift/filetests/filetests/regalloc/coalesce-bb.clif
 create mode 100644 cranelift/filetests/filetests/regalloc/reload-208-bb.clif
 create mode 100644 cranelift/filetests/filetests/regalloc/x86-regres-bb.clif

diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index 3953f7e4d4..aeec864d79 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -318,7 +318,7 @@ impl Context {
     /// Run the register allocator.
     pub fn regalloc(&mut self, isa: &dyn TargetIsa) -> CodegenResult<()> {
         self.regalloc
-            .run(isa, &mut self.func, &self.cfg, &mut self.domtree)
+            .run(isa, &mut self.func, &mut self.cfg, &mut self.domtree)
     }
 
     /// Insert prologue and epilogues after computing the stack frame layout.
diff --git a/cranelift/codegen/src/regalloc/branch_splitting.rs b/cranelift/codegen/src/regalloc/branch_splitting.rs
new file mode 100644
index 0000000000..e44f452296
--- /dev/null
+++ b/cranelift/codegen/src/regalloc/branch_splitting.rs
@@ -0,0 +1,197 @@
+//! Split the outgoing edges of conditional branches that pass parameters.
+//!
+//! One of the reason for splitting edges is to be able to insert `copy` and `regmove` instructions
+//! between a conditional branch and the following terminator.
+#![cfg(feature = "basic-blocks")]
+
+use std::vec::Vec;
+
+use crate::cursor::{Cursor, EncCursor};
+use crate::dominator_tree::DominatorTree;
+use crate::flowgraph::ControlFlowGraph;
+use crate::ir::{Ebb, Function, Inst, InstBuilder, InstructionData, Opcode, ValueList};
+use crate::isa::TargetIsa;
+use crate::topo_order::TopoOrder;
+
+pub fn run(
+    isa: &dyn TargetIsa,
+    func: &mut Function,
+    cfg: &mut ControlFlowGraph,
+    domtree: &mut DominatorTree,
+    topo: &mut TopoOrder,
+) {
+    let mut ctx = Context {
+        has_new_blocks: false,
+        has_fallthrough_return: None,
+        cur: EncCursor::new(func, isa),
+        domtree,
+        topo,
+        cfg,
+    };
+    ctx.run()
+}
+
+struct Context<'a> {
+    /// True if new blocks were inserted.
+    has_new_blocks: bool,
+
+    /// Record whether newly inserted empty blocks should be inserted last, or before the last, to
+    /// avoid disturbing the expected control flow of `fallthroug_return` statements.
+    ///
+    /// This value is computed when needed. The Option wraps the computed value if any.
+    has_fallthrough_return: Option,
+
+    /// Current instruction as well as reference to function and ISA.
+    cur: EncCursor<'a>,
+
+    /// References to contextual data structures we need.
+    domtree: &'a mut DominatorTree,
+    topo: &'a mut TopoOrder,
+    cfg: &'a mut ControlFlowGraph,
+}
+
+impl<'a> Context<'a> {
+    fn run(&mut self) {
+        // Any ebb order will do.
+        self.topo.reset(self.cur.func.layout.ebbs());
+        while let Some(ebb) = self.topo.next(&self.cur.func.layout, self.domtree) {
+            // Branches can only be at the last or second to last position in an extended basic
+            // block.
+            self.cur.goto_last_inst(ebb);
+            let terminator_inst = self.cur.current_inst().expect("terminator");
+            if let Some(inst) = self.cur.prev_inst() {
+                let opcode = self.cur.func.dfg[inst].opcode();
+                if opcode.is_branch() {
+                    self.visit_conditional_branch(inst, opcode);
+                    self.cur.goto_inst(terminator_inst);
+                    self.visit_terminator_branch(terminator_inst);
+                }
+            }
+        }
+
+        // If blocks were added the cfg and domtree are inconsistent and must be recomputed.
+        if self.has_new_blocks {
+            self.cfg.compute(&self.cur.func);
+            self.domtree.compute(&self.cur.func, self.cfg);
+        }
+    }
+
+    fn visit_conditional_branch(&mut self, branch: Inst, opcode: Opcode) {
+        // TODO: target = dfg[branch].branch_destination().expect("conditional branch");
+        let target = match self.cur.func.dfg[branch] {
+            InstructionData::Branch { destination, .. }
+            | InstructionData::BranchIcmp { destination, .. }
+            | InstructionData::BranchInt { destination, .. }
+            | InstructionData::BranchFloat { destination, .. } => destination,
+            _ => panic!("Unexpected instruction in visit_conditional_branch"),
+        };
+
+        // If there are any parameters, split the edge.
+        if self.should_split_edge(target) {
+            // Create the block the branch will jump to.
+            let new_ebb = self.make_empty_ebb();
+
+            // Extract the arguments of the branch instruction, split the Ebb parameters and the
+            // branch arguments
+            let num_fixed = opcode.constraints().num_fixed_value_arguments();
+            let dfg = &mut self.cur.func.dfg;
+            let old_args: Vec<_> = {
+                let args = dfg[branch].take_value_list().expect("ebb parameters");
+                args.as_slice(&dfg.value_lists).iter().map(|x| *x).collect()
+            };
+            let (branch_args, ebb_params) = old_args.split_at(num_fixed);
+
+            // Replace the branch destination by the new Ebb created with no parameters, and restore
+            // the branch arguments, without the original Ebb parameters.
+            {
+                let branch_args = ValueList::from_slice(branch_args, &mut dfg.value_lists);
+                let data = &mut dfg[branch];
+                *data.branch_destination_mut().expect("branch") = new_ebb;
+                data.put_value_list(branch_args);
+            }
+            let ok = self.cur.func.update_encoding(branch, self.cur.isa).is_ok();
+            debug_assert!(ok);
+
+            // Insert a jump to the original target with its arguments into the new block.
+            self.cur.goto_first_insertion_point(new_ebb);
+            self.cur.ins().jump(target, ebb_params);
+
+            // Reset the cursor to point to the branch.
+            self.cur.goto_inst(branch);
+        }
+    }
+
+    fn visit_terminator_branch(&mut self, inst: Inst) {
+        let inst_data = &self.cur.func.dfg[inst];
+        let opcode = inst_data.opcode();
+        if opcode != Opcode::Jump && opcode != Opcode::Fallthrough {
+            // This opcode is ignored as it does not have any EBB parameters.
+            if opcode != Opcode::IndirectJumpTableBr {
+                debug_assert!(!opcode.is_branch())
+            }
+            return;
+        }
+
+        let target = match inst_data {
+            InstructionData::Jump { destination, .. } => destination,
+            _ => panic!(
+                "Unexpected instruction {} in visit_terminator_branch",
+                self.cur.display_inst(inst)
+            ),
+        };
+        debug_assert!(self.cur.func.dfg[inst].opcode().is_terminator());
+
+        // If there are any parameters, split the edge.
+        if self.should_split_edge(*target) {
+            // Create the block the branch will jump to.
+            let new_ebb = self.cur.func.dfg.make_ebb();
+            self.has_new_blocks = true;
+
+            // Split the current block before its terminator, and insert a new jump instruction to
+            // jump to it.
+            let jump = self.cur.ins().jump(new_ebb, &[]);
+            self.cur.insert_ebb(new_ebb);
+
+            // Reset the cursor to point to new terminator of the old ebb.
+            self.cur.goto_inst(jump);
+        }
+    }
+
+    // A new ebb must be inserted before the last ebb because the last ebb may have a
+    // fallthrough_return and can't have anything after it.
+    fn make_empty_ebb(&mut self) -> Ebb {
+        let last_ebb = self.cur.layout().last_ebb().unwrap();
+        if self.has_fallthrough_return == None {
+            let last_inst = self.cur.layout().last_inst(last_ebb).unwrap();
+            self.has_fallthrough_return =
+                Some(self.cur.func.dfg[last_inst].opcode() == Opcode::FallthroughReturn);
+        }
+        let new_ebb = self.cur.func.dfg.make_ebb();
+        if self.has_fallthrough_return == Some(true) {
+            // Insert before the last block which has a fallthrough_return
+            // instruction.
+            self.cur.layout_mut().insert_ebb(new_ebb, last_ebb);
+        } else {
+            // Insert after the last block.
+            self.cur.layout_mut().insert_ebb_after(new_ebb, last_ebb);
+        }
+        self.has_new_blocks = true;
+        new_ebb
+    }
+
+    /// Returns whether we should introduce a new branch.
+    fn should_split_edge(&self, target: Ebb) -> bool {
+        // We should split the edge if the target has any parameters.
+        if self.cur.func.dfg.ebb_params(target).len() > 0 {
+            return true;
+        };
+
+        // Or, if the target has more than one block reaching it.
+        debug_assert!(self.cfg.pred_iter(target).next() != None);
+        if let Some(_) = self.cfg.pred_iter(target).skip(1).next() {
+            return true;
+        };
+
+        false
+    }
+}
diff --git a/cranelift/codegen/src/regalloc/context.rs b/cranelift/codegen/src/regalloc/context.rs
index fe2e702647..84fe139d87 100644
--- a/cranelift/codegen/src/regalloc/context.rs
+++ b/cranelift/codegen/src/regalloc/context.rs
@@ -8,6 +8,8 @@ use crate::dominator_tree::DominatorTree;
 use crate::flowgraph::ControlFlowGraph;
 use crate::ir::Function;
 use crate::isa::TargetIsa;
+#[cfg(feature = "basic-blocks")]
+use crate::regalloc::branch_splitting;
 use crate::regalloc::coalescing::Coalescing;
 use crate::regalloc::coloring::Coloring;
 use crate::regalloc::live_value_tracker::LiveValueTracker;
@@ -78,7 +80,7 @@ impl Context {
         &mut self,
         isa: &dyn TargetIsa,
         func: &mut Function,
-        cfg: &ControlFlowGraph,
+        cfg: &mut ControlFlowGraph,
         domtree: &mut DominatorTree,
     ) -> CodegenResult<()> {
         let _tt = timing::regalloc();
@@ -93,6 +95,12 @@ impl Context {
         // phases.
         self.tracker.clear();
 
+        // Pass: Split branches, add space where to add copy & regmove instructions.
+        #[cfg(feature = "basic-blocks")]
+        {
+            branch_splitting::run(isa, func, cfg, domtree, &mut self.topo);
+        }
+
         // Pass: Liveness analysis.
         self.liveness.compute(isa, func, cfg);
 
diff --git a/cranelift/codegen/src/regalloc/mod.rs b/cranelift/codegen/src/regalloc/mod.rs
index 4df7cc9cef..37fcccb3b0 100644
--- a/cranelift/codegen/src/regalloc/mod.rs
+++ b/cranelift/codegen/src/regalloc/mod.rs
@@ -10,6 +10,7 @@ pub mod register_set;
 pub mod virtregs;
 
 mod affinity;
+mod branch_splitting;
 mod coalescing;
 mod context;
 mod diversion;
diff --git a/cranelift/filetests/filetests/regalloc/coalesce-bb.clif b/cranelift/filetests/filetests/regalloc/coalesce-bb.clif
new file mode 100644
index 0000000000..384ef2ca4c
--- /dev/null
+++ b/cranelift/filetests/filetests/regalloc/coalesce-bb.clif
@@ -0,0 +1,158 @@
+test regalloc
+target riscv32
+feature "basic-blocks"
+
+; Test the coalescer.
+; regex: V=v\d+
+; regex: WS=\s+
+; regex: LOC=%\w+
+; regex: EBB=ebb\d+
+
+; This function is already CSSA, so no copies should be inserted.
+function %cssa(i32) -> i32 {
+ebb0(v0: i32):
+    ; not: copy
+    ; v0 is used by the branch and passed as an arg - that's no conflict.
+    brnz v0, ebb1(v0)
+    jump ebb2
+
+ebb2:
+    ; v0 is live across the branch above. That's no conflict.
+    v1 = iadd_imm v0, 7
+    jump ebb1(v1)
+
+ebb1(v10: i32):
+    v11 = iadd_imm v10, 7
+    return v11
+}
+
+function %trivial(i32) -> i32 {
+ebb0(v0: i32):
+    ; check:    brnz v0, $(splitEdge=$EBB)
+    brnz v0, ebb1(v0)
+    jump ebb2
+
+ebb2:
+    ; not: copy
+    v1 = iadd_imm v0, 7
+    jump ebb1(v1)
+
+    ; check:  $splitEdge:
+    ; nextln:   $(cp1=$V) = copy.i32 v0
+    ; nextln:   jump ebb1($cp1)
+
+ebb1(v10: i32):
+    ; Use v0 in the destination EBB causes a conflict.
+    v11 = iadd v10, v0
+    return v11
+}
+
+; A value is used as an SSA argument twice in the same branch.
+function %dualuse(i32) -> i32 {
+ebb0(v0: i32):
+    ; check:  brnz v0, $(splitEdge=$EBB)
+    brnz v0, ebb1(v0, v0)
+    jump ebb2
+
+ebb2:
+    v1 = iadd_imm v0, 7
+    v2 = iadd_imm v1, 56
+    jump ebb1(v1, v2)
+
+    ; check:  $splitEdge:
+    ; check:    $(cp1=$V) = copy.i32 v0
+    ; nextln:   jump ebb1($cp1, v0)
+
+ebb1(v10: i32, v11: i32):
+    v12 = iadd v10, v11
+    return v12
+}
+
+; Interference away from the branch
+; The interference can be broken with a copy at either branch.
+function %interference(i32) -> i32 {
+ebb0(v0: i32):
+    ; not:    copy
+    ; check:  brnz v0, $(splitEdge=$EBB)
+    ; not:    copy
+    brnz v0, ebb1(v0)
+    jump ebb2
+
+ebb2:
+    v1 = iadd_imm v0, 7
+    ; v1 and v0 interfere here:
+    v2 = iadd_imm v0, 8
+    ; check: $(cp0=$V) = copy v1
+    ; check: jump ebb1($cp0)
+    jump ebb1(v1)
+
+    ; check:  $splitEdge:
+    ; not:      copy
+    ; nextln:   jump ebb1(v0)
+
+ebb1(v10: i32):
+    ; not: copy
+    v11 = iadd_imm v10, 7
+    return v11
+}
+
+; A loop where one induction variable is used as a backedge argument.
+function %fibonacci(i32) -> i32 {
+ebb0(v0: i32):
+    v1 = iconst.i32 1
+    v2 = iconst.i32 2
+    jump ebb1(v1, v2)
+
+ebb1(v10: i32, v11: i32):
+    ; v11 needs to be isolated because it interferes with v10.
+    ; check: ebb1(v10: i32 [$LOC], $(nv11a=$V): i32 [$LOC])
+    ; check: v11 = copy $nv11a
+    v12 = iadd v10, v11
+    v13 = icmp ult v12, v0
+    ; check:  brnz v13, $(splitEdge=$EBB)
+    brnz v13, ebb1(v11, v12)
+    jump ebb2
+
+    ; check:  $splitEdge:
+    ; check:    $(nv11b=$V) = copy.i32 v11
+    ; not:      copy
+    ; check:   jump ebb1($nv11b, v12)
+
+ebb2:
+    return v12
+}
+
+; Function arguments passed on the stack aren't allowed to be part of a virtual
+; register, at least for now. This is because the other values in the virtual
+; register would need to be spilled to the incoming_arg stack slot which we treat
+; as belonging to the caller.
+function %stackarg(i32, i32, i32, i32, i32, i32, i32, i32, i32) -> i32 {
+; check: ss0 = incoming_arg 4
+; not: incoming_arg
+ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32, v7: i32, v8: i32):
+    ; check: fill v8
+    ; not: v8
+    jump ebb1(v8)
+
+ebb1(v10: i32):
+    v11 = iadd_imm v10, 1
+    return v11
+}
+
+function %gvn_unremovable_phi(i32) system_v {
+ebb0(v0: i32):
+    v2 = iconst.i32 0
+    jump ebb2(v2, v0)
+
+ebb2(v3: i32, v4: i32):
+    brnz v3, ebb2(v3, v4)
+    jump ebb3
+
+ebb3:
+    v5 = iconst.i32 1
+    brnz v3, ebb2(v2, v5)
+    jump ebb4
+
+ebb4:
+    return
+}
diff --git a/cranelift/filetests/filetests/regalloc/coalesce.clif b/cranelift/filetests/filetests/regalloc/coalesce.clif
index fa9f64ba4d..c78219ea53 100644
--- a/cranelift/filetests/filetests/regalloc/coalesce.clif
+++ b/cranelift/filetests/filetests/regalloc/coalesce.clif
@@ -1,5 +1,6 @@
 test regalloc
 target riscv32
+feature !"basic-blocks"
 
 ; Test the coalescer.
 ; regex: V=v\d+
diff --git a/cranelift/filetests/filetests/regalloc/reload-208-bb.clif b/cranelift/filetests/filetests/regalloc/reload-208-bb.clif
new file mode 100644
index 0000000000..47438d7d61
--- /dev/null
+++ b/cranelift/filetests/filetests/regalloc/reload-208-bb.clif
@@ -0,0 +1,111 @@
+test regalloc
+target x86_64 haswell
+feature "basic-blocks"
+
+; regex: V=v\d+
+; regex: EBB=ebb\d+
+
+; Filed as https://github.com/CraneStation/cranelift/issues/208
+;
+; The verifier complains about a branch argument that is not in the same virtual register as the
+; corresponding EBB argument.
+;
+; The problem was the reload pass rewriting EBB arguments on "brnz v9, ebb3(v9)"
+
+function %pr208(i64 vmctx [%rdi]) system_v {
+    gv1 = vmctx
+    gv0 = iadd_imm.i64 gv1, -8
+    heap0 = static gv0, min 0, bound 0x5000, offset_guard 0x0040_0000
+    sig0 = (i64 vmctx [%rdi]) -> i32 [%rax] system_v
+    sig1 = (i64 vmctx [%rdi], i32 [%rsi]) system_v
+    fn0 = u0:1 sig0
+    fn1 = u0:3 sig1
+
+ebb0(v0: i64):
+    v1 = iconst.i32 0
+    v2 = call fn0(v0)
+    v20 = iconst.i32 0x4ffe
+    v16 = icmp uge v2, v20
+    brz v16, ebb5
+    jump ebb9
+
+ebb9:
+    trap heap_oob
+
+ebb5:
+    v17 = uextend.i64 v2
+    v18 = iadd_imm.i64 v0, -8
+    v19 = load.i64 v18
+    v3 = iadd v19, v17
+    v4 = load.i32 v3
+    v21 = iconst.i32 0
+    v5 = icmp eq v4, v21
+    v6 = bint.i32 v5
+    brnz v6, ebb2
+    jump ebb3(v4)
+
+ebb3(v7: i32):
+    call fn1(v0, v7)
+    v26 = iconst.i32 0x4ffe
+    v22 = icmp uge v7, v26
+    brz v22, ebb6
+    jump ebb10
+
+ebb10:
+    trap heap_oob
+
+ebb6:
+    v23 = uextend.i64 v7
+    v24 = iadd_imm.i64 v0, -8
+    v25 = load.i64 v24
+    v8 = iadd v25, v23
+    v9 = load.i32 v8+56
+    ; check: v9 = spill
+    ; check: brnz $V, $(splitEdge=$EBB)
+    brnz v9, ebb3(v9)
+    jump ebb4
+
+    ; check: $splitEdge:
+    ; nextln:  jump ebb3(v9)
+
+ebb4:
+    jump ebb2
+
+ebb2:
+    v10 = iconst.i32 0
+    v31 = iconst.i32 0x4ffe
+    v27 = icmp uge v10, v31
+    brz v27, ebb7
+    jump ebb11
+
+ebb11:
+    trap heap_oob
+
+ebb7:
+    v28 = uextend.i64 v10
+    v29 = iadd_imm.i64 v0, -8
+    v30 = load.i64 v29
+    v11 = iadd v30, v28
+    v12 = load.i32 v11+12
+    call fn1(v0, v12)
+    v13 = iconst.i32 0
+    v36 = iconst.i32 0x4ffe
+    v32 = icmp uge v13, v36
+    brz v32, ebb8
+    jump ebb12
+
+ebb12:
+    trap heap_oob
+
+ebb8:
+    v33 = uextend.i64 v13
+    v34 = iadd_imm.i64 v0, -8
+    v35 = load.i64 v34
+    v14 = iadd v35, v33
+    v15 = load.i32 v14+12
+    call fn1(v0, v15)
+    jump ebb1
+
+ebb1:
+    return
+}
diff --git a/cranelift/filetests/filetests/regalloc/reload-208.clif b/cranelift/filetests/filetests/regalloc/reload-208.clif
index 0c65ffb1e8..c767670252 100644
--- a/cranelift/filetests/filetests/regalloc/reload-208.clif
+++ b/cranelift/filetests/filetests/regalloc/reload-208.clif
@@ -1,5 +1,6 @@
 test regalloc
 target x86_64 haswell
+feature !"basic-blocks"
 
 ; regex: V=v\d+
 
diff --git a/cranelift/filetests/filetests/regalloc/x86-regres-bb.clif b/cranelift/filetests/filetests/regalloc/x86-regres-bb.clif
new file mode 100644
index 0000000000..28eba22a64
--- /dev/null
+++ b/cranelift/filetests/filetests/regalloc/x86-regres-bb.clif
@@ -0,0 +1,49 @@
+test regalloc
+target i686
+feature "basic-blocks"
+
+; regex: V=v\d+
+; regex: EBB=ebb\d+
+
+; The value v9 appears both as the branch control and one of the EBB arguments
+; in the brnz instruction in ebb2. It also happens that v7 and v9 are assigned
+; to the same register, so v9 doesn't need to be moved before the brnz.
+;
+; This ended up confusong the constraint solver which had not made a record of
+; the fixed register assignment for v9 since it was already in the correct
+; register.
+function %pr147(i32) -> i32 system_v {
+ebb0(v0: i32):
+    v1 = iconst.i32 0
+    v2 = iconst.i32 1
+    v3 = iconst.i32 0
+    jump ebb2(v3, v2, v0)
+
+ebb2(v4: i32, v5: i32, v7: i32):
+    ; check: ebb2
+    v6 = iadd v4, v5
+    v8 = iconst.i32 -1
+    ; v7 is killed here and v9 gets the same register.
+    v9 = iadd v7, v8
+    ; check:   v9 = iadd v7, v8
+    ; Here v9 the brnz control appears to interfere with v9 the EBB argument,
+    ; so divert_fixed_input_conflicts() calls add_var(v9), which is ok. The
+    ; add_var sanity checks got confused when no fixed assignment could be
+    ; found for v9.
+    ;
+    ; We should be able to handle this situation without making copies of v9.
+    brnz v9, ebb2(v5, v6, v9)
+    ; check:   brnz v9, $(splitEdge=$EBB)
+    jump ebb3
+
+    ; check: $splitEdge:
+    ; check:   jump ebb2($V, $V, v9)
+ebb3:
+    return v5
+}
+
+function %select_i64(i64, i64, i32) -> i64 {
+ebb0(v0: i64, v1: i64, v2: i32):
+    v3 = select v2, v0, v1
+    return v3
+}
diff --git a/cranelift/filetests/filetests/regalloc/x86-regres.clif b/cranelift/filetests/filetests/regalloc/x86-regres.clif
index 66c3d45c2e..ac6e82d66f 100644
--- a/cranelift/filetests/filetests/regalloc/x86-regres.clif
+++ b/cranelift/filetests/filetests/regalloc/x86-regres.clif
@@ -1,6 +1,6 @@
 test regalloc
-
 target i686
+feature !"basic-blocks"
 
 ; regex: V=v\d+
 

From 8becc9a64cb1cdc86f95f874c5f7f1ea28066330 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 30 Aug 2019 23:22:48 -0700
Subject: [PATCH 2623/3084] Bump version to 0.41.0

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    | 10 +++++-----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 61 insertions(+), 61 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 674db7cb92..b59837f27f 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.40.0"
+version = "0.41.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.40.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.40.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.40.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.40.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.40.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.40.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.40.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.40.0" }
-cranelift-module = { path = "cranelift-module", version = "0.40.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.40.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.40.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.40.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.40.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.41.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.41.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.41.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.41.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.41.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.41.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.41.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.41.0" }
+cranelift-module = { path = "cranelift-module", version = "0.41.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.41.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.41.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.41.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.41.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index bd7cf33701..3775c66c79 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.40.0"
+version = "0.41.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.40.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.41.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 868db6ca3b..19351b363c 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.40.0"
+version = "0.41.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.40.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.40.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.41.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.41.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -27,7 +27,7 @@ serde = { version = "1.0.94", features = ["derive"], optional = true }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.40.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.41.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 1d1fd894f9..4e5834d0d0 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.40.0"
+version = "0.41.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.40.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.41.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 538d1299df..3aecf29d9b 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.40.0"
+version = "0.41.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index c93840f705..ef90f9d45a 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.40.0"
+version = "0.41.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.40.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.41.0" }
 faerie = "0.10.0"
 goblin = "0.0.24"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 8913a8bfab..6873cc17df 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.40.0"
+version = "0.41.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.40.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.40.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.40.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.41.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.41.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.41.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index fec5f408e4..5f9d8dceca 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.40.0"
+version = "0.41.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 9ed7dbe6f8..0ff8046fe8 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.40.0"
+version = "0.41.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.40.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.41.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 58a2094174..0e7028b451 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.40.0"
+version = "0.41.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
 target-lexicon = { version = "0.4.0", default-features = false }
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 6372ff2d24..fc87ad6997 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.40.0"
+version = "0.41.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.40.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.41.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index c178a6e060..a8f5425e35 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.40.0"
+version="0.41.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 433dc1f539..0a86a8af35 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.40.0"
+version = "0.41.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0" }
 target-lexicon = "0.4.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 7b63d2be90..01d6de4900 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.40.0"
+version = "0.41.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.40.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.41.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 7dd767f497..ecea5d58fb 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.40.0"
+version = "0.41.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.40.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.40.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.41.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.41.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.40.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.40.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.40.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.41.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.41.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.41.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 78cc0cde91..3004135f4c 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.40.0"
+version = "0.41.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.40.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.41.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 5beb48fdbf..8f2346d6a7 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.40.0"
+version = "0.41.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.37.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.40.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.40.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.40.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.41.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.41.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From cd1b2c0af029f65742927b75e9ef9769f2e8d121 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 28 Aug 2019 17:02:10 +0200
Subject: [PATCH 2624/3084] [meta] Try to use {prefix+number} when looking up a
 register by name;

This makes it possible to look up registers like r15 on x86.
---
 cranelift/codegen/meta/src/cdsl/regs.rs | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/regs.rs b/cranelift/codegen/meta/src/cdsl/regs.rs
index 60bf8f5818..e7a3464d10 100644
--- a/cranelift/codegen/meta/src/cdsl/regs.rs
+++ b/cranelift/codegen/meta/src/cdsl/regs.rs
@@ -43,10 +43,26 @@ impl RegBank {
             // Try to match without the bank prefix.
             assert!(name.starts_with(self.prefix));
             let name_without_prefix = &name[self.prefix.len()..];
-            self.names
+            if let Some(found) = self
+                .names
                 .iter()
                 .position(|®_name| reg_name == name_without_prefix)
-                .expect(&format!("invalid register name {}", name))
+            {
+                found
+            } else {
+                // Ultimate try: try to parse a number and use this in the array, eg r15 on x86.
+                if let Ok(as_num) = name_without_prefix.parse::() {
+                    assert!(
+                        (as_num - self.first_unit) < self.units,
+                        "trying to get {}, but bank only has {} registers!",
+                        name,
+                        self.units
+                    );
+                    (as_num - self.first_unit) as usize
+                } else {
+                    panic!("invalid register name {}", name);
+                }
+            }
         };
         self.first_unit + (unit as u8)
     }

From 47e5d6c83eff94a97035612b719b38d196978b9a Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 29 Aug 2019 16:10:04 +0200
Subject: [PATCH 2625/3084] [regalloc] Transform the program_input_abi function
 into a Context method;

It was implemented this way before to avoid borrow-checking issues,
where self would be both mutably borrowed (because of the solver) and
immutably borrowed (because of the ABI parameters list). This is worked
around by adding a local AbiParams struct which contains a summary of
the information that's needed by program_input_abi, allowing to retrieve
the ABI params within the method's body itself.
---
 cranelift/codegen/src/regalloc/coloring.rs | 87 ++++++++++------------
 1 file changed, 39 insertions(+), 48 deletions(-)

diff --git a/cranelift/codegen/src/regalloc/coloring.rs b/cranelift/codegen/src/regalloc/coloring.rs
index 0df5795c11..507d5e3ff7 100644
--- a/cranelift/codegen/src/regalloc/coloring.rs
+++ b/cranelift/codegen/src/regalloc/coloring.rs
@@ -44,7 +44,7 @@
 
 use crate::cursor::{Cursor, EncCursor};
 use crate::dominator_tree::DominatorTree;
-use crate::ir::{AbiParam, ArgumentLoc, InstBuilder, ValueDef};
+use crate::ir::{ArgumentLoc, InstBuilder, ValueDef};
 use crate::ir::{Ebb, Function, Inst, InstructionData, Layout, Opcode, SigRef, Value, ValueLoc};
 use crate::isa::{regs_overlap, RegClass, RegInfo, RegUnit};
 use crate::isa::{ConstraintKind, EncInfo, OperandConstraint, RecipeConstraints, TargetIsa};
@@ -68,6 +68,12 @@ pub struct Coloring {
     solver: Solver,
 }
 
+/// Kinds of ABI parameters.
+enum AbiParams {
+    Parameters(SigRef),
+    Returns,
+}
+
 /// Bundle of references that the coloring algorithm needs.
 ///
 /// Some of the needed mutable references are passed around as explicit function arguments so we
@@ -285,6 +291,36 @@ impl<'a> Context<'a> {
         regs
     }
 
+    /// Program the input-side ABI constraints for `inst` into the constraint solver.
+    ///
+    /// ABI constraints are the fixed register assignments useds for calls and returns.
+    fn program_input_abi(&mut self, inst: Inst, abi_params: AbiParams) {
+        let abi_types = match abi_params {
+            AbiParams::Parameters(sig) => &self.cur.func.dfg.signatures[sig].params,
+            AbiParams::Returns => &self.cur.func.signature.returns,
+        };
+
+        for (abi, &value) in abi_types
+            .iter()
+            .zip(self.cur.func.dfg.inst_variable_args(inst))
+        {
+            if let ArgumentLoc::Reg(reg) = abi.location {
+                if let Affinity::Reg(rci) = self
+                    .liveness
+                    .get(value)
+                    .expect("ABI register must have live range")
+                    .affinity
+                {
+                    let rc = self.reginfo.rc(rci);
+                    let cur_reg = self.divert.reg(value, &self.cur.func.locations);
+                    self.solver.reassign_in(value, rc, cur_reg, reg);
+                } else {
+                    panic!("ABI argument {} should be in a register", value);
+                }
+            }
+        }
+    }
+
     /// Color the values defined by `inst` and insert any necessary shuffle code to satisfy
     /// instruction constraints.
     ///
@@ -316,25 +352,9 @@ impl<'a> Context<'a> {
         }
         let call_sig = self.cur.func.dfg.call_signature(inst);
         if let Some(sig) = call_sig {
-            program_input_abi(
-                &mut self.solver,
-                inst,
-                &self.cur.func.dfg.signatures[sig].params,
-                &self.cur.func,
-                &self.liveness,
-                &self.reginfo,
-                &self.divert,
-            );
+            self.program_input_abi(inst, AbiParams::Parameters(sig));
         } else if self.cur.func.dfg[inst].opcode().is_return() {
-            program_input_abi(
-                &mut self.solver,
-                inst,
-                &self.cur.func.signature.returns,
-                &self.cur.func,
-                &self.liveness,
-                &self.reginfo,
-                &self.divert,
-            );
+            self.program_input_abi(inst, AbiParams::Returns);
         } else if self.cur.func.dfg[inst].opcode().is_branch() {
             // This is a branch, so we need to make sure that globally live values are in their
             // global registers. For EBBs that take arguments, we also need to place the argument
@@ -1104,35 +1124,6 @@ impl<'a> Context<'a> {
     }
 }
 
-/// Program the input-side ABI constraints for `inst` into the constraint solver.
-///
-/// ABI constraints are the fixed register assignments used for calls and returns.
-fn program_input_abi(
-    solver: &mut Solver,
-    inst: Inst,
-    abi_types: &[AbiParam],
-    func: &Function,
-    liveness: &Liveness,
-    reginfo: &RegInfo,
-    divert: &RegDiversions,
-) {
-    for (abi, &value) in abi_types.iter().zip(func.dfg.inst_variable_args(inst)) {
-        if let ArgumentLoc::Reg(reg) = abi.location {
-            if let Affinity::Reg(rci) = liveness
-                .get(value)
-                .expect("ABI register must have live range")
-                .affinity
-            {
-                let rc = reginfo.rc(rci);
-                let cur_reg = divert.reg(value, &func.locations);
-                solver.reassign_in(value, rc, cur_reg, reg);
-            } else {
-                panic!("ABI argument {} should be in a register", value);
-            }
-        }
-    }
-}
-
 /// Keep track of the set of available registers in two interference domains: all registers
 /// considering diversions and global registers not considering diversions.
 struct AvailableRegs {

From 44942a26a229eded88a99c7571673a990c0a88be Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 27 Aug 2019 17:04:04 +0200
Subject: [PATCH 2626/3084] Tweak comments;

---
 cranelift/codegen/src/binemit/mod.rs     | 2 +-
 cranelift/codegen/src/isa/x86/binemit.rs | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
index 1287dcd439..d6ca75d2d9 100644
--- a/cranelift/codegen/src/binemit/mod.rs
+++ b/cranelift/codegen/src/binemit/mod.rs
@@ -133,7 +133,7 @@ pub trait CodeSink {
     /// Add a relocation referencing an external symbol plus the addend at the current offset.
     fn reloc_external(&mut self, _: Reloc, _: &ExternalName, _: Addend);
 
-    /// Add a relocation referencing a jump table.
+    /// Add a relocation referencing a constant.
     fn reloc_constant(&mut self, _: Reloc, _: ConstantOffset);
 
     /// Add a relocation referencing a jump table.
diff --git a/cranelift/codegen/src/isa/x86/binemit.rs b/cranelift/codegen/src/isa/x86/binemit.rs
index afe2c2611f..a5924ed030 100644
--- a/cranelift/codegen/src/isa/x86/binemit.rs
+++ b/cranelift/codegen/src/isa/x86/binemit.rs
@@ -342,7 +342,7 @@ fn jt_disp4(jt: JumpTable, func: &Function, sink: &mut CS
     sink.reloc_jt(Reloc::X86PCRelRodata4, jt);
 }
 
-/// Emit a four-byte displacement to `constant`
+/// Emit a four-byte displacement to `constant`.
 fn const_disp4(constant: Constant, func: &Function, sink: &mut CS) {
     let offset = func.dfg.constants.get_offset(constant);
     let delta = offset.wrapping_sub(sink.offset() + 4);

From d09fc93fef6c26dfa2d1581d5dabfb3b550021a7 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 3 Sep 2019 15:20:38 +0200
Subject: [PATCH 2627/3084] Remove Travis CI; (#967)

---
 .travis.yml | 43 -------------------------------------------
 1 file changed, 43 deletions(-)
 delete mode 100644 .travis.yml

diff --git a/.travis.yml b/.travis.yml
deleted file mode 100644
index 18d3aa37e9..0000000000
--- a/.travis.yml
+++ /dev/null
@@ -1,43 +0,0 @@
-# Travis CI script. See https://travis-ci.org/ for more info.
-
-os:
-    - linux
-    - osx
-language: rust
-rust:
-    # The oldest version we currently support. See
-    # CONTRIBUTING.md#rustc-version-support for details.
-    - 1.37.0
-    - beta
-    - nightly
-matrix:
-    allow_failures:
-        # We try to be compatible with beta and nightly, but they occasionally
-        # fail, so we don't allow them to hold up people using stable.
-        - rust: beta
-        - rust: nightly
-    # Similarly, we don't need to hold up people using stable while we wait
-    # for the results which may fail.
-    fast_finish: true
-dist: xenial
-sudo: false
-before_script:
-    # If an old version of rustfmt from cargo is already installed, uninstall
-    # it, since it can prevent the installation of the new version from rustup.
-    - cargo uninstall rustfmt || true
-    - cargo install --list
-    # If we're testing beta or nightly, we still need to install the stable
-    # toolchain so that we can run the stable version of rustfmt.
-    - rustup toolchain install stable
-    # Install the stable version of rustfmt.
-    - rustup component add --toolchain=stable rustfmt-preview
-    - rustup component list --toolchain=stable
-    - rustup show
-    - rustfmt +stable --version || echo fail
-    # Sometimes the component isn't actually ready after being installed, and
-    # rustup update makes it ready.
-    - rustup update
-    - rustfmt +stable --version
-script: ./test-all.sh
-cache:
-    cargo: true

From 98056aa05df22a9d518e3ec8741c509580179c70 Mon Sep 17 00:00:00 2001
From: Julian Seward 
Date: Tue, 3 Sep 2019 05:33:18 +0200
Subject: [PATCH 2628/3084] Don't incorrectly omit a REX prefix for some
 encodings of `copy_to_ssa`.  Mozilla bug #1576969.

Also, as a ridealong fix, removes R32 encodings for x86_64 in `enc_r32_r64`,
since the type `rXX` by definition only exists for targets with word size `XX`
bits.
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 56 ++++++++++++-------
 .../filetests/isa/x86/binary64-float.clif     | 20 +++++++
 .../filetests/filetests/isa/x86/binary64.clif | 20 +++++++
 3 files changed, 75 insertions(+), 21 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index a5df9298e8..ac8e5e725f 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -150,6 +150,16 @@ impl PerCpuModeEncodings {
         self.enc64(inst.bind(I64), template.rex().w());
     }
 
+    /// Add encodings for `inst.i32` to X86_32.
+    /// Add encodings for `inst.i32` to X86_64 with a REX prefix.
+    /// Add encodings for `inst.i64` to X86_64 with a REX.W prefix.
+    fn enc_i32_i64_rex_only(&mut self, inst: impl Into, template: Template) {
+        let inst: InstSpec = inst.into();
+        self.enc32(inst.bind(I32), template.nonrex());
+        self.enc64(inst.bind(I32), template.rex());
+        self.enc64(inst.bind(I64), template.rex().w());
+    }
+
     /// Add encodings for `inst.i32` to X86_32.
     /// Add encodings for `inst.i32` to X86_64 with and without REX.
     /// Add encodings for `inst.i64` to X86_64 with a REX.W prefix.
@@ -177,16 +187,10 @@ impl PerCpuModeEncodings {
     }
 
     /// Add encodings for `inst.r32` to X86_32.
-    /// Add encodings for `inst.r32` to X86_64 with and without REX.
     /// Add encodings for `inst.r64` to X86_64 with a REX.W prefix.
-    fn enc_r32_r64(&mut self, inst: impl Into, template: Template) {
+    fn enc_r32_r64_rex_only(&mut self, inst: impl Into, template: Template) {
         let inst: InstSpec = inst.into();
         self.enc32(inst.bind_ref(R32), template.nonrex());
-
-        // REX-less encoding must come after REX encoding so we don't use it by default. Otherwise
-        // reg-alloc would never use r8 and up.
-        self.enc64(inst.bind_ref(R32), template.rex());
-        self.enc64(inst.bind_ref(R32), template.nonrex());
         self.enc64(inst.bind_ref(R64), template.rex().w());
     }
 
@@ -247,6 +251,14 @@ impl PerCpuModeEncodings {
         self.enc_x86_64_instp(inst, template, instp);
     }
 
+    /// Add two encodings for `inst`:
+    /// - X86_32
+    /// - X86_64 with the REX prefix.
+    fn enc_both_rex_only(&mut self, inst: impl Clone + Into, template: Template) {
+        self.enc32(inst.clone(), template.clone());
+        self.enc64(inst, template.rex());
+    }
+
     /// Add encodings for `inst.i32` to X86_32.
     /// Add encodings for `inst.i32` to X86_64 with and without REX.
     /// Add encodings for `inst.i64` to X86_64 with a REX prefix, using the `w_bit`
@@ -622,7 +634,7 @@ pub fn define(
     e.enc_i32_i64(x86_umulx, rec_mulx.opcodes(vec![0xf7]).rrr(4));
 
     e.enc_i32_i64(copy, rec_umr.opcodes(vec![0x89]));
-    e.enc_r32_r64(copy, rec_umr.opcodes(vec![0x89]));
+    e.enc_r32_r64_rex_only(copy, rec_umr.opcodes(vec![0x89]));
     e.enc_both(copy.bind(B1), rec_umr.opcodes(vec![0x89]));
     e.enc_both(copy.bind(I8), rec_umr.opcodes(vec![0x89]));
     e.enc_both(copy.bind(I16), rec_umr.opcodes(vec![0x89]));
@@ -901,8 +913,8 @@ pub fn define(
 
     e.enc_i32_i64(spill, rec_spillSib32.opcodes(vec![0x89]));
     e.enc_i32_i64(regspill, rec_regspill32.opcodes(vec![0x89]));
-    e.enc_r32_r64(spill, rec_spillSib32.opcodes(vec![0x89]));
-    e.enc_r32_r64(regspill, rec_regspill32.opcodes(vec![0x89]));
+    e.enc_r32_r64_rex_only(spill, rec_spillSib32.opcodes(vec![0x89]));
+    e.enc_r32_r64_rex_only(regspill, rec_regspill32.opcodes(vec![0x89]));
 
     // Use a 32-bit write for spilling `b1`, `i8` and `i16` to avoid
     // constraining the permitted registers.
@@ -927,8 +939,8 @@ pub fn define(
 
     e.enc_i32_i64(fill, rec_fillSib32.opcodes(vec![0x8b]));
     e.enc_i32_i64(regfill, rec_regfill32.opcodes(vec![0x8b]));
-    e.enc_r32_r64(fill, rec_fillSib32.opcodes(vec![0x8b]));
-    e.enc_r32_r64(regfill, rec_regfill32.opcodes(vec![0x8b]));
+    e.enc_r32_r64_rex_only(fill, rec_fillSib32.opcodes(vec![0x8b]));
+    e.enc_r32_r64_rex_only(regfill, rec_regfill32.opcodes(vec![0x8b]));
 
     // No-op fills, created by late-stage redundant-fill removal.
     for &ty in &[I64, I32, I16, I8] {
@@ -964,20 +976,22 @@ pub fn define(
     e.enc64(copy_special, rec_copysp.opcodes(vec![0x89]).rex().w());
     e.enc32(copy_special, rec_copysp.opcodes(vec![0x89]));
 
-    // Copy to SSA
-    e.enc_i32_i64(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(vec![0x89]));
-    e.enc_r32_r64(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(vec![0x89]));
-    e.enc_both(copy_to_ssa.bind(B1), rec_umr_reg_to_ssa.opcodes(vec![0x89]));
-    e.enc_both(copy_to_ssa.bind(I8), rec_umr_reg_to_ssa.opcodes(vec![0x89]));
-    e.enc_both(
+    // Copy to SSA.  These have to be done with special _rex_only encoders, because the standard
+    // machinery for deciding whether a REX.{RXB} prefix is needed doesn't take into account
+    // the source register, which is specified directly in the instruction.
+    e.enc_i32_i64_rex_only(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(vec![0x89]));
+    e.enc_r32_r64_rex_only(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(vec![0x89]));
+    e.enc_both_rex_only(copy_to_ssa.bind(B1), rec_umr_reg_to_ssa.opcodes(vec![0x89]));
+    e.enc_both_rex_only(copy_to_ssa.bind(I8), rec_umr_reg_to_ssa.opcodes(vec![0x89]));
+    e.enc_both_rex_only(
         copy_to_ssa.bind(I16),
         rec_umr_reg_to_ssa.opcodes(vec![0x89]),
     );
-    e.enc_both(
+    e.enc_both_rex_only(
         copy_to_ssa.bind(F64),
         rec_furm_reg_to_ssa.opcodes(vec![0xf2, 0x0f, 0x10]),
     );
-    e.enc_both(
+    e.enc_both_rex_only(
         copy_to_ssa.bind(F32),
         rec_furm_reg_to_ssa.opcodes(vec![0xf3, 0x0f, 0x10]),
     );
@@ -1808,7 +1822,7 @@ pub fn define(
     e.enc64(null.bind_ref(R64), rec_pu_id_ref.opcodes(vec![0xb8]));
 
     // is_null, implemented by testing whether the value is 0.
-    e.enc_r32_r64(is_null, rec_is_zero.opcodes(vec![0x85]));
+    e.enc_r32_r64_rex_only(is_null, rec_is_zero.opcodes(vec![0x85]));
 
     // safepoint instruction calls sink, no actual encoding.
     e.enc32_rec(safepoint, rec_safepoint, 0);
diff --git a/cranelift/filetests/filetests/isa/x86/binary64-float.clif b/cranelift/filetests/filetests/isa/x86/binary64-float.clif
index 59e024a042..831079070d 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64-float.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64-float.clif
@@ -96,6 +96,26 @@ ebb0:
     ; asm: movaps %xmm5, %xmm10
     [-,%xmm10]          v39 = copy v10                          ; bin: 44 0f 28 d5
 
+    ; Copy to SSA
+
+    ; asm: movsd %xmm0, %xmm15
+    [-,%xmm15]          v400 = copy_to_ssa.f64 %xmm0            ; bin: f2 44 0f 10 f8
+    ; asm: movsd %xmm15, %xmm0
+    [-,%xmm0]           v401 = copy_to_ssa.f64 %xmm15           ; bin: f2 41 0f 10 c7
+    ; asm: movsd %xmm7, %xmm6.  Unfortunately we get a redundant REX prefix.
+    [-,%xmm6]           v402 = copy_to_ssa.f64 %xmm7            ; bin: f2 40 0f 10 f7
+    ; asm: movsd %xmm11, %xmm14
+    [-,%xmm14]          v403 = copy_to_ssa.f64 %xmm11           ; bin: f2 45 0f 10 f3
+
+    ; asm: movss %xmm0, %xmm15
+    [-,%xmm15]          v404 = copy_to_ssa.f32 %xmm0            ; bin: f3 44 0f 10 f8
+    ; asm: movss %xmm15, %xmm0
+    [-,%xmm0]           v405 = copy_to_ssa.f32 %xmm15           ; bin: f3 41 0f 10 c7
+    ; asm: movss %xmm7, %xmm6.  Unfortunately we get a redundant REX prefix.
+    [-,%xmm6]           v406 = copy_to_ssa.f32 %xmm7            ; bin: f3 40 0f 10 f7
+    ; asm: movss %xmm11, %xmm14
+    [-,%xmm14]          v407 = copy_to_ssa.f32 %xmm11           ; bin: f3 45 0f 10 f3
+
     ; Convert float to int.
 
     ; asm: cvttss2si %xmm5, %ecx
diff --git a/cranelift/filetests/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
index 7742a24ee3..d1b862c85a 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64.clif
@@ -179,6 +179,26 @@ ebb0:
     ; asm: movq %r10, %rsp
     copy_special %r10 -> %rsp                   ; bin: 4c 89 d4
 
+    ; Copy to SSA
+
+    ; asm: movq %rax, %r15
+    [-,%r15]      v700 = copy_to_ssa.i64 %rax    ; bin: 49 89 c7
+    ; asm: movq %r15, %rax
+    [-,%rax]      v701 = copy_to_ssa.i64 %r15    ; bin: 4c 89 f8
+    ; asm: movq %rdi, %rsi
+    [-,%rsi]      v702 = copy_to_ssa.i64 %rdi    ; bin: 48 89 fe
+    ; asm: movq %r11, %r14
+    [-,%r14]      v703 = copy_to_ssa.i64 %r11    ; bin: 4d 89 de
+
+    ; asm: movl %eax, %r15d
+    [-,%r15]      v704 = copy_to_ssa.i32 %rax    ; bin: 41 89 c7
+    ; asm: movl %r15d, %eax
+    [-,%rax]      v705 = copy_to_ssa.i32 %r15    ; bin: 44 89 f8
+    ; asm: movl %edi, %esi.  Unfortunately we get a redundant REX prefix.
+    [-,%rsi]      v706 = copy_to_ssa.i32 %rdi    ; bin: 40 89 fe
+    ; asm: movl %r11, %r14
+    [-,%r14]      v707 = copy_to_ssa.i32 %r11    ; bin: 45 89 de
+
     ; Load/Store instructions.
 
     ; Register indirect addressing with no displacement.

From d64e4540047cbee408b4d45fd9996345b7e84b25 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 28 Aug 2019 10:01:19 -0700
Subject: [PATCH 2629/3084] Improve uimm128 parsing

This commit changes 128-bit constant parsing in two ways:
 - it adds the ability to use underscores to separate digits when writing a 128-bit constant in hexadecimal; e.g. `0x00010203...` can now be written as `0x0001_0203_...`
 - it adds a new mechanism for parsing 128-bit constants using integer/float/boolean literals; e.g. `vconst.i32x4 [1 2 3 4]`. Note that currently the controlling type of the instruction dictates how many literals to parse inside the brackets.
---
 cranelift/codegen/src/ir/immediates.rs        | 208 ++++++++++++++++--
 .../filetests/isa/x86/rodata-vconst.clif      |  13 --
 .../x86/{vconst.clif => vconst-binemit.clif}  |   0
 .../filetests/isa/x86/vconst-rodata.clif      |  20 ++
 .../filetests/isa/x86/vconst-run.clif         |  21 ++
 cranelift/reader/src/parser.rs                | 102 ++++++++-
 6 files changed, 328 insertions(+), 36 deletions(-)
 delete mode 100644 cranelift/filetests/filetests/isa/x86/rodata-vconst.clif
 rename cranelift/filetests/filetests/isa/x86/{vconst.clif => vconst-binemit.clif} (100%)
 create mode 100644 cranelift/filetests/filetests/isa/x86/vconst-rodata.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/vconst-run.clif

diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index 8917d96d4b..d69aed4390 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -5,9 +5,29 @@
 //! `cranelift-codegen/meta/src/shared/immediates` crate in the meta language.
 
 use core::fmt::{self, Display, Formatter};
+use core::iter::FromIterator;
 use core::mem;
-use core::str::FromStr;
+use core::str::{from_utf8, FromStr};
 use core::{i32, u32};
+use std::vec::Vec;
+
+/// Convert a type into a vector of bytes; all implementors in this file must use little-endian
+/// orderings of bytes to match WebAssembly's little-endianness.
+trait IntoBytes {
+    fn into_bytes(self) -> Vec;
+}
+
+impl IntoBytes for u8 {
+    fn into_bytes(self) -> Vec {
+        vec![self]
+    }
+}
+
+impl IntoBytes for i32 {
+    fn into_bytes(self) -> Vec {
+        self.to_le_bytes().to_vec()
+    }
+}
 
 /// 64-bit immediate signed integer operand.
 ///
@@ -34,6 +54,12 @@ impl Into for Imm64 {
     }
 }
 
+impl IntoBytes for Imm64 {
+    fn into_bytes(self) -> Vec {
+        self.0.to_le_bytes().to_vec()
+    }
+}
+
 impl From for Imm64 {
     fn from(x: i64) -> Self {
         Imm64(x)
@@ -270,6 +296,23 @@ impl FromStr for Uimm32 {
 #[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)]
 pub struct Uimm128(pub [u8; 16]);
 
+impl Uimm128 {
+    /// Iterate over the bytes in the constant
+    pub fn bytes(&self) -> impl Iterator {
+        self.0.iter()
+    }
+
+    /// Convert the immediate into a vector
+    pub fn to_vec(self) -> Vec {
+        self.0.to_vec()
+    }
+
+    /// Convert the immediate into a slice
+    pub fn as_slice(&self) -> &[u8] {
+        &self.0[..]
+    }
+}
+
 impl Display for Uimm128 {
     // print a 128-bit vector in hexadecimal, e.g. 0x000102030405060708090a0b0c0d0e0f
     fn fmt(&self, f: &mut Formatter) -> fmt::Result {
@@ -314,24 +357,84 @@ impl FromStr for Uimm128 {
     fn from_str(s: &str) -> Result {
         if s.len() <= 2 || &s[0..2] != "0x" {
             Err("Expected a hexadecimal string, e.g. 0x1234")
-        } else if s.len() % 2 != 0 {
-            Err("Hexadecimal string must have an even number of digits")
-        } else if s.len() > 34 {
-            Err("Hexadecimal string has too many digits to fit in a 128-bit vector")
         } else {
-            let mut buffer = [0; 16]; // zero-fill
-            let start_at = s.len() / 2 - 1;
-            for i in (2..s.len()).step_by(2) {
-                let byte = u8::from_str_radix(&s[i..i + 2], 16)
-                    .or_else(|_| Err("Unable to parse as hexadecimal"))?;
-                let position = start_at - (i / 2);
-                buffer[position] = byte;
+            // clean and check the string
+            let cleaned: Vec = s[2..]
+                .as_bytes()
+                .iter()
+                .filter(|&&b| b as char != '_')
+                .cloned()
+                .collect(); // remove 0x prefix and any intervening _ characters
+
+            if cleaned.len() == 0 {
+                Err("Hexadecimal string must have some digits")
+            } else if cleaned.len() % 2 != 0 {
+                Err("Hexadecimal string must have an even number of digits")
+            } else if cleaned.len() > 32 {
+                Err("Hexadecimal string has too many digits to fit in a 128-bit vector")
+            } else {
+                let mut buffer = [0; 16]; // zero-fill the buffer
+                let mut position = cleaned.len() / 2 - 1; // since Uimm128 is little-endian but the string is not, we write from back to front but must start at the highest position required by the string
+                for i in (0..cleaned.len()).step_by(2) {
+                    let pair = from_utf8(&cleaned[i..i + 2])
+                        .or_else(|_| Err("Unable to parse hexadecimal pair as UTF-8"))?;
+                    let byte = u8::from_str_radix(pair, 16)
+                        .or_else(|_| Err("Unable to parse as hexadecimal"))?;
+                    buffer[position] = byte;
+                    position = position.wrapping_sub(1); // should only wrap on the last iteration
+                }
+
+                Ok(Uimm128(buffer))
             }
-            Ok(Uimm128(buffer))
         }
     }
 }
 
+/// Implement a way to convert an iterator of immediates to a Uimm128:
+///  - this expects the items in reverse order (e.g. last lane first) which is the natural output of pushing items into a vector
+///  - this may not fully consume the iterator or may fail if it cannot take the expected number of items
+///  - this requires the input type (i.e. $ty) to implement ToBytes
+macro_rules! construct_uimm128_from_iterator_of {
+    ( $ty:ident, $lanes:expr ) => {
+        impl FromIterator<$ty> for Uimm128 {
+            fn from_iter>(iter: T) -> Self {
+                let mut buffer: [u8; 16] = [0; 16];
+                iter.into_iter()
+                    .take($lanes)
+                    .map(|f| f.into_bytes())
+                    .flat_map(|b| b)
+                    .enumerate()
+                    .for_each(|(i, b)| buffer[i] = b);
+                Uimm128(buffer)
+            }
+        }
+    };
+}
+
+/// Special case for booleans since we have to decide the bit-width based on the number of items
+impl FromIterator for Uimm128 {
+    fn from_iter>(iter: T) -> Self {
+        let bools = Vec::from_iter(iter);
+        let count = bools.len();
+        assert!(count > 0 && count <= 16); // ensure we don't have too many booleans
+        assert_eq!(count & (count - 1), 0); // ensure count is a power of two, see https://stackoverflow.com/questions/600293
+        let mut buffer: [u8; 16] = [0; 16];
+        let step = 16 / count;
+        bools
+            .iter()
+            .enumerate()
+            .map(|(i, &b)| (i * step, if b { 1 } else { 0 }))
+            .for_each(|(i, b)| buffer[i] = b);
+        Uimm128(buffer)
+    }
+}
+
+construct_uimm128_from_iterator_of!(u8, 16);
+construct_uimm128_from_iterator_of!(i32, 4);
+construct_uimm128_from_iterator_of!(Ieee32, 4);
+construct_uimm128_from_iterator_of!(Imm64, 2);
+construct_uimm128_from_iterator_of!(Ieee64, 2);
+
 /// 32-bit signed immediate offset.
 ///
 /// This is used to encode an immediate offset for load/store instructions. All supported ISAs have
@@ -739,6 +842,12 @@ impl From for Ieee32 {
     }
 }
 
+impl IntoBytes for Ieee32 {
+    fn into_bytes(self) -> Vec {
+        self.0.to_le_bytes().to_vec()
+    }
+}
+
 impl Ieee64 {
     /// Create a new `Ieee64` containing the bits of `x`.
     pub fn with_bits(x: u64) -> Self {
@@ -812,6 +921,12 @@ impl From for Ieee64 {
     }
 }
 
+impl IntoBytes for Ieee64 {
+    fn into_bytes(self) -> Vec {
+        self.0.to_le_bytes().to_vec()
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -968,9 +1083,10 @@ mod tests {
         parse_ok::("0x00", "0x00");
         parse_ok::("0x00000042", "0x42");
         parse_ok::(
-            "0x0102030405060708090a0b0c0d0e0f",
-            "0x0102030405060708090a0b0c0d0e0f",
+            "0x0102030405060708090a0b0c0d0e0f00",
+            "0x0102030405060708090a0b0c0d0e0f00",
         );
+        parse_ok::("0x_0000_0043_21", "0x4321");
 
         parse_err::("", "Expected a hexadecimal string, e.g. 0x1234");
         parse_err::("0x", "Expected a hexadecimal string, e.g. 0x1234");
@@ -982,6 +1098,24 @@ mod tests {
             "0x00000000000000000000000000000000000000000000000000",
             "Hexadecimal string has too many digits to fit in a 128-bit vector",
         );
+        parse_err::("0xrstu", "Unable to parse as hexadecimal");
+        parse_err::("0x__", "Hexadecimal string must have some digits");
+    }
+
+    #[test]
+    fn uimm128_equivalence() {
+        assert_eq!(
+            "0x01".parse::().unwrap().0,
+            [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+        );
+        assert_eq!(
+            Uimm128::from_iter(vec![1, 0, 0, 0]).0,
+            [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+        );
+        assert_eq!(
+            Uimm128::from(1).0,
+            [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+        );
     }
 
     #[test]
@@ -997,6 +1131,50 @@ mod tests {
         assert_eq!(
             "0x12345678".parse::().unwrap().0,
             [0x78, 0x56, 0x34, 0x12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+        );
+        assert_eq!(
+            "0x1234_5678".parse::().unwrap().0,
+            [0x78, 0x56, 0x34, 0x12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+        );
+    }
+
+    #[test]
+    fn uimm128_from_iter() {
+        assert_eq!(
+            Uimm128::from_iter(vec![4, 3, 2, 1]).0,
+            [4, 0, 0, 0, 3, 0, 0, 0, 2, 0, 0, 0, 1, 0, 0, 0]
+        );
+
+        assert_eq!(
+            Uimm128::from_iter(vec![false, true]).0,
+            [/* false */ 0, 0, 0, 0, 0, 0, 0, 0, /* true */ 1, 0, 0, 0, 0, 0, 0, 0]
+        );
+
+        assert_eq!(
+            Uimm128::from_iter(vec![false, true, false, true, false, true, false, true]).0,
+            [0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0]
+        );
+
+        #[allow(trivial_numeric_casts)]
+        let u8s = vec![
+            1 as u8, 2, 3, 4, 5, 6, 7, 8, 9, 0xa, 0xb, 0xc, 0xd, 0xe, 0xf, 0,
+        ];
+        assert_eq!(
+            Uimm128::from_iter(u8s).0,
+            [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 0]
+        );
+
+        #[allow(trivial_numeric_casts)]
+        let ieee32s: Vec = vec![32.4 as f32, 0.0, 1.0, 6.6666]
+            .iter()
+            .map(|&f| Ieee32::from(f))
+            .collect();
+        assert_eq!(
+            Uimm128::from_iter(ieee32s).0,
+            [
+                /* 32.4 == */ 0x9a, 0x99, 0x01, 0x42, /* 0 == */ 0, 0, 0, 0,
+                /* 1 == */ 0, 0, 0x80, 0x3f, /* 6.6666 == */ 0xca, 0x54, 0xd5, 0x40,
+            ]
         )
     }
 
diff --git a/cranelift/filetests/filetests/isa/x86/rodata-vconst.clif b/cranelift/filetests/filetests/isa/x86/rodata-vconst.clif
deleted file mode 100644
index fd029b678d..0000000000
--- a/cranelift/filetests/filetests/isa/x86/rodata-vconst.clif
+++ /dev/null
@@ -1,13 +0,0 @@
-test rodata
-set enable_simd=true
-set probestack_enabled=false
-target x86_64 haswell
-
-; use baldrdash calling convention here for simplicity (avoids prologue, epilogue)
-function %test_vconst_i32() -> i32x4 baldrdash_system_v {
-ebb0:
-    v0 = vconst.i32x4 0x1234
-    return v0
-}
-
-; sameln: [34, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
diff --git a/cranelift/filetests/filetests/isa/x86/vconst.clif b/cranelift/filetests/filetests/isa/x86/vconst-binemit.clif
similarity index 100%
rename from cranelift/filetests/filetests/isa/x86/vconst.clif
rename to cranelift/filetests/filetests/isa/x86/vconst-binemit.clif
diff --git a/cranelift/filetests/filetests/isa/x86/vconst-rodata.clif b/cranelift/filetests/filetests/isa/x86/vconst-rodata.clif
new file mode 100644
index 0000000000..99e8455ed4
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/vconst-rodata.clif
@@ -0,0 +1,20 @@
+test rodata
+set enable_simd=true
+set probestack_enabled=false
+target x86_64 haswell
+
+function %test_vconst_i32() -> i32x4 {
+ebb0:
+    v0 = vconst.i32x4 0x1234
+    return v0
+}
+
+; sameln: [34, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+
+function %test_vconst_b16() -> b16x8 {
+ebb0:
+    v0 = vconst.b16x8 [true false true false true false true true]
+    return v0
+}
+
+; sameln: [1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0]
diff --git a/cranelift/filetests/filetests/isa/x86/vconst-run.clif b/cranelift/filetests/filetests/isa/x86/vconst-run.clif
new file mode 100644
index 0000000000..9ec160c2e5
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/vconst-run.clif
@@ -0,0 +1,21 @@
+test run
+set enable_simd
+
+function %test_vconst_syntax() -> b1 {
+ebb0:
+    v0 = vconst.i32x4 0x00000004_00000003_00000002_00000001     ; build constant using hexadecimal syntax
+    v1 = vconst.i32x4 [1 2 3 4]                                 ; build constant using literal list syntax
+
+    ; verify lane 1 matches
+    v2 = extractlane v0, 1
+    v3 = extractlane v1, 1
+    v4 = icmp eq v3, v2
+
+    ; verify lane 1 has the correct value
+    v5 = icmp_imm eq v3, 2
+
+    v6 = band v4, v5
+    return v6
+}
+
+; run
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index df0fc18a0e..6971ca663d 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -12,6 +12,7 @@ use cranelift_codegen::ir::entities::AnyEntity;
 use cranelift_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm128, Uimm32, Uimm64};
 use cranelift_codegen::ir::instructions::{InstructionData, InstructionFormat, VariableArgs};
 use cranelift_codegen::ir::types::INVALID;
+use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{
     AbiParam, ArgumentExtension, ArgumentLoc, Ebb, ExtFuncData, ExternalName, FuncRef, Function,
     GlobalValue, GlobalValueData, Heap, HeapData, HeapStyle, JumpTable, JumpTableData, MemFlags,
@@ -21,6 +22,7 @@ use cranelift_codegen::ir::{
 use cranelift_codegen::isa::{self, CallConv, Encoding, RegUnit, TargetIsa};
 use cranelift_codegen::packed_option::ReservedValue;
 use cranelift_codegen::{settings, timing};
+use std::iter::FromIterator;
 use std::mem;
 use std::str::FromStr;
 use std::{u16, u32};
@@ -610,6 +612,19 @@ impl<'a> Parser<'a> {
         }
     }
 
+    // Match and consume either a hexadecimal Uimm128 immediate (e.g. 0x000102...) or its literal list form (e.g. [0 1 2...])
+    fn match_uimm128_or_literals(&mut self, controlling_type: Type) -> ParseResult {
+        if self.optional(Token::LBracket) {
+            // parse using a list of values, e.g. vconst.i32x4 [0 1 2 3]
+            let uimm128 = self.parse_literals_to_uimm128(controlling_type)?;
+            self.match_token(Token::RBracket, "expected a terminating right bracket")?;
+            Ok(uimm128)
+        } else {
+            // parse using a hexadecimal value
+            self.match_uimm128("expected an immediate hexadecimal operand")
+        }
+    }
+
     // Match and consume a Uimm64 immediate.
     fn match_uimm64(&mut self, err_msg: &str) -> ParseResult {
         if let Some(Token::Integer(text)) = self.token() {
@@ -821,6 +836,36 @@ impl<'a> Parser<'a> {
         }
     }
 
+    /// Parse a list of literals (i.e. integers, floats, booleans); e.g.
+    fn parse_literals_to_uimm128(&mut self, ty: Type) -> ParseResult {
+        macro_rules! consume {
+            ( $ty:ident, $match_fn:expr ) => {{
+                assert!($ty.is_vector());
+                let mut v = Vec::with_capacity($ty.lane_count() as usize);
+                for _ in 0..$ty.lane_count() {
+                    v.push($match_fn?);
+                }
+                Uimm128::from_iter(v)
+            }};
+        }
+
+        if !ty.is_vector() {
+            err!(self.loc, "Expected a controlling vector type, not {}", ty)
+        } else {
+            let uimm128 = match ty.lane_type() {
+                I8 => consume!(ty, self.match_uimm8("Expected an 8-bit unsigned integer")),
+                I16 => unimplemented!(), // TODO no 16-bit match yet
+                I32 => consume!(ty, self.match_imm32("Expected a 32-bit integer")),
+                I64 => consume!(ty, self.match_imm64("Expected a 64-bit integer")),
+                F32 => consume!(ty, self.match_ieee32("Expected a 32-bit float...")),
+                F64 => consume!(ty, self.match_ieee64("Expected a 64-bit float")),
+                b if b.is_bool() => consume!(ty, self.match_bool("Expected a boolean")),
+                _ => return err!(self.loc, "Expected a type of: float, int, bool"),
+            };
+            Ok(uimm128)
+        }
+    }
+
     /// Parse a list of test command passes specified in command line.
     pub fn parse_cmdline_passes(&mut self, passes: &'a [String]) -> Vec> {
         let mut list = Vec::new();
@@ -1977,7 +2022,7 @@ impl<'a> Parser<'a> {
         };
 
         // instruction ::=  [inst-results "="] Opcode(opc) ["." Type] * ...
-        let inst_data = self.parse_inst_operands(ctx, opcode)?;
+        let inst_data = self.parse_inst_operands(ctx, opcode, explicit_ctrl_type)?;
 
         // We're done parsing the instruction now.
         //
@@ -2186,6 +2231,7 @@ impl<'a> Parser<'a> {
         &mut self,
         ctx: &mut Context,
         opcode: Opcode,
+        explicit_control_type: Option,
     ) -> ParseResult {
         let idata = match opcode.format() {
             InstructionFormat::Unary => InstructionData::Unary {
@@ -2196,14 +2242,23 @@ impl<'a> Parser<'a> {
                 opcode,
                 imm: self.match_imm64("expected immediate integer operand")?,
             },
-            InstructionFormat::UnaryImm128 => {
-                let uimm128 = self.match_uimm128("expected immediate hexadecimal operand")?;
-                let constant_handle = ctx.function.dfg.constants.insert(uimm128.0.to_vec());
-                InstructionData::UnaryImm128 {
-                    opcode,
-                    imm: constant_handle,
+            InstructionFormat::UnaryImm128 => match explicit_control_type {
+                None => {
+                    return err!(
+                        self.loc,
+                        "Expected {:?} to have a controlling type variable, e.g. inst.i32x4",
+                        opcode
+                    )
                 }
-            }
+                Some(ty) => {
+                    let uimm128 = self.match_uimm128_or_literals(ty)?;
+                    let constant_handle = ctx.function.dfg.constants.insert(uimm128.0.to_vec());
+                    InstructionData::UnaryImm128 {
+                        opcode,
+                        imm: constant_handle,
+                    }
+                }
+            },
             InstructionFormat::UnaryIeee32 => InstructionData::UnaryIeee32 {
                 opcode,
                 imm: self.match_ieee32("expected immediate 32-bit float operand")?,
@@ -3150,4 +3205,35 @@ mod tests {
             CallConv::Cold
         );
     }
+
+    #[test]
+    fn uimm128() {
+        macro_rules! parse_as_uimm128 {
+            ($text:expr, $type:expr) => {{
+                Parser::new($text).parse_literals_to_uimm128($type)
+            }};
+        }
+        macro_rules! can_parse_as_uimm128 {
+            ($text:expr, $type:expr) => {{
+                assert!(parse_as_uimm128!($text, $type).is_ok())
+            }};
+        }
+        macro_rules! cannot_parse_as_uimm128 {
+            ($text:expr, $type:expr) => {{
+                assert!(parse_as_uimm128!($text, $type).is_err())
+            }};
+        }
+
+        can_parse_as_uimm128!("1 2 3 4", I32X4);
+        can_parse_as_uimm128!("1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16", I8X16);
+        can_parse_as_uimm128!("0x1.1 0x2.2 0x3.3 0x4.4", F32X4);
+        can_parse_as_uimm128!("true false true false true false true false", B16X8);
+        can_parse_as_uimm128!("0 -1", I64X2);
+        can_parse_as_uimm128!("true false", B64X2);
+        can_parse_as_uimm128!("true true true true true", B32X4); // note that parse_literals_to_uimm128 will leave extra tokens unconsumed
+
+        cannot_parse_as_uimm128!("0x0 0x1 0x2 0x3", I32X4);
+        cannot_parse_as_uimm128!("1 2 3", I32X4);
+        cannot_parse_as_uimm128!(" ", F32X4);
+    }
 }

From 49a37e48fb4b8163a17385d4254fcfd1b88074f2 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 23 Aug 2019 11:56:00 +0200
Subject: [PATCH 2630/3084] [codegen] Make scalar_to_vector's output type a
 lane of its input type;

---
 cranelift/codegen/meta/src/isa/x86/encodings.rs  |  4 +---
 .../codegen/meta/src/shared/instructions.rs      | 16 +++-------------
 .../filetests/verifier/scalar-to-vector.clif     | 10 ++++++++++
 3 files changed, 14 insertions(+), 16 deletions(-)
 create mode 100644 cranelift/filetests/filetests/verifier/scalar-to-vector.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index ac8e5e725f..02186983b3 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1729,9 +1729,7 @@ pub fn define(
     // to the Intel manual: "When the destination operand is an XMM register, the source operand is
     // written to the low doubleword of the register and the regiser is zero-extended to 128 bits."
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() >= 8) {
-        let instruction = scalar_to_vector
-            .bind_vector_from_lane(ty, sse_vector_size)
-            .bind(ty);
+        let instruction = scalar_to_vector.bind_vector_from_lane(ty, sse_vector_size);
         let template = rec_frurm.opcodes(vec![0x66, 0x0f, 0x6e]); // MOVD/MOVQ
         if ty.lane_bits() < 64 {
             // no 32-bit encodings for 64-bit widths
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index c11016c64d..a40a6dc247 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -113,16 +113,6 @@ pub fn define(
             .build(),
     );
 
-    let Scalar = &TypeVar::new(
-        "scalar",
-        "Any scalar value that can be used as a lane in a vector",
-        TypeSetBuilder::new()
-            .bools(Interval::All)
-            .ints(Interval::All)
-            .floats(Interval::All)
-            .build(),
-    );
-
     let Any = &TypeVar::new(
         "Any",
         "Any integer, float, boolean, or reference scalar or vector type",
@@ -407,7 +397,7 @@ pub fn define(
             "resumable_trap",
             r#"
         A resumable trap.
-        
+
         This instruction allows non-conditional traps to be used as non-terminal instructions.
         "#,
         )
@@ -2772,8 +2762,8 @@ pub fn define(
         .operands_out(vec![a]),
     );
 
-    let s = &operand_doc("s", Scalar, "A scalar value");
-    let a = &operand_doc("a", TxN, "A vector value (i.e. held in an XMM register)");
+    let a = &operand_doc("a", TxN, "A vector value");
+    let s = &operand_doc("s", &TxN.lane_of(), "A scalar value");
 
     ig.push(
         Inst::new(
diff --git a/cranelift/filetests/filetests/verifier/scalar-to-vector.clif b/cranelift/filetests/filetests/verifier/scalar-to-vector.clif
new file mode 100644
index 0000000000..927abdafc2
--- /dev/null
+++ b/cranelift/filetests/filetests/verifier/scalar-to-vector.clif
@@ -0,0 +1,10 @@
+test verifier
+set enable_simd=true
+target x86_64
+
+function %scalar_to_vector() {
+ebb0:
+    v0 = iconst.i32 42
+    v1 = scalar_to_vector.f32x4 v0 ; error: arg 0 (v0) has type i32, expected f32
+    return
+}

From 89d741f8aece93feed7ed169c9358eb1e59c23be Mon Sep 17 00:00:00 2001
From: Pat Hickey 
Date: Wed, 4 Sep 2019 12:25:58 -0700
Subject: [PATCH 2631/3084] upgrade to target-lexicon 0.8.0

* the target-lexicon crate no longer has or needs the std feature
  in cargo, so we can delete all default-features=false, any mentions
  of its std feature, and the nostd configs in many lib.rs files
* the representation of arm architectures has changed, so some case
  statements needed refactoring
---
 cranelift/Cargo.toml                   |  2 +-
 cranelift/bforest/src/lib.rs           |  1 -
 cranelift/codegen/Cargo.toml           |  3 +--
 cranelift/codegen/src/isa/arm32/mod.rs | 13 ++++++-------
 cranelift/codegen/src/isa/call_conv.rs |  1 +
 cranelift/codegen/src/isa/mod.rs       | 11 ++---------
 cranelift/codegen/src/lib.rs           |  1 -
 cranelift/entity/src/lib.rs            |  1 -
 cranelift/faerie/Cargo.toml            |  2 +-
 cranelift/frontend/Cargo.toml          |  2 +-
 cranelift/frontend/src/lib.rs          |  1 -
 cranelift/fuzz/Cargo.toml              |  2 +-
 cranelift/module/src/lib.rs            |  1 -
 cranelift/native/Cargo.toml            |  4 ++--
 cranelift/preopt/src/lib.rs            |  1 -
 cranelift/reader/Cargo.toml            |  2 +-
 cranelift/simplejit/Cargo.toml         |  2 +-
 cranelift/src/disasm.rs                | 24 ++++++++++++++----------
 cranelift/wasm/Cargo.toml              |  2 +-
 cranelift/wasm/src/lib.rs              |  1 -
 20 files changed, 33 insertions(+), 44 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index b59837f27f..d714b67ad1 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -38,7 +38,7 @@ serde = "1.0.8"
 term = "0.6.1"
 capstone = { version = "0.6.0", optional = true }
 wabt = { version = "0.9.1", optional = true }
-target-lexicon = "0.4.0"
+target-lexicon = "0.8.0"
 pretty_env_logger = "0.3.0"
 file-per-thread-logger = "0.1.2"
 indicatif = "0.11.0"
diff --git a/cranelift/bforest/src/lib.rs b/cranelift/bforest/src/lib.rs
index 7b6beb2599..5d33068b7a 100644
--- a/cranelift/bforest/src/lib.rs
+++ b/cranelift/bforest/src/lib.rs
@@ -32,7 +32,6 @@
     )
 )]
 #![no_std]
-#![cfg_attr(not(feature = "std"), feature(alloc))]
 
 #[cfg(test)]
 #[cfg(not(feature = "std"))]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 19351b363c..bd25b25ce3 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -18,7 +18,7 @@ cranelift-bforest = { path = "../cranelift-bforest", version = "0.41.0", default
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
-target-lexicon = { version = "0.4.0", default-features = false }
+target-lexicon = "0.8.0"
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
@@ -38,7 +38,6 @@ default = ["std"]
 std = [
     "cranelift-entity/std",
     "cranelift-bforest/std",
-    "target-lexicon/std",
     "cranelift-codegen-meta/std"
 ]
 
diff --git a/cranelift/codegen/src/isa/arm32/mod.rs b/cranelift/codegen/src/isa/arm32/mod.rs
index fde3339700..ced1b88767 100644
--- a/cranelift/codegen/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/src/isa/arm32/mod.rs
@@ -42,14 +42,13 @@ fn isa_constructor(
     builder: shared_settings::Builder,
 ) -> Box {
     let level1 = match triple.architecture {
-        Architecture::Thumbv6m | Architecture::Thumbv7em | Architecture::Thumbv7m => {
-            &enc_tables::LEVEL1_T32[..]
+        Architecture::Arm(arm) => {
+            if arm.is_thumb() {
+                &enc_tables::LEVEL1_T32[..]
+            } else {
+                &enc_tables::LEVEL1_A32[..]
+            }
         }
-        Architecture::Arm
-        | Architecture::Armv4t
-        | Architecture::Armv5te
-        | Architecture::Armv7
-        | Architecture::Armv7s => &enc_tables::LEVEL1_A32[..],
         _ => panic!(),
     };
     Box::new(Isa {
diff --git a/cranelift/codegen/src/isa/call_conv.rs b/cranelift/codegen/src/isa/call_conv.rs
index dddd4bbd8a..e834ad2515 100644
--- a/cranelift/codegen/src/isa/call_conv.rs
+++ b/cranelift/codegen/src/isa/call_conv.rs
@@ -31,6 +31,7 @@ impl CallConv {
             // uses System V.
             Ok(CallingConvention::SystemV) | Err(()) => CallConv::SystemV,
             Ok(CallingConvention::WindowsFastcall) => CallConv::WindowsFastcall,
+            Ok(unimp) => unimplemented!("calling convention: {:?}", unimp),
         }
     }
 
diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index 407afc5f39..816e185f81 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -111,15 +111,8 @@ pub fn lookup(triple: Triple) -> Result {
         Architecture::I386 | Architecture::I586 | Architecture::I686 | Architecture::X86_64 => {
             isa_builder!(x86, "x86")(triple)
         }
-        Architecture::Thumbv6m
-        | Architecture::Thumbv7em
-        | Architecture::Thumbv7m
-        | Architecture::Arm
-        | Architecture::Armv4t
-        | Architecture::Armv5te
-        | Architecture::Armv7
-        | Architecture::Armv7s => isa_builder!(arm32, "arm32")(triple),
-        Architecture::Aarch64 => isa_builder!(arm64, "arm64")(triple),
+        Architecture::Arm { .. } => isa_builder!(arm32, "arm32")(triple),
+        Architecture::Aarch64 { .. } => isa_builder!(arm64, "arm64")(triple),
         _ => Err(LookupError::Unsupported),
     }
 }
diff --git a/cranelift/codegen/src/lib.rs b/cranelift/codegen/src/lib.rs
index b682ea3867..1e97dfd6e0 100644
--- a/cranelift/codegen/src/lib.rs
+++ b/cranelift/codegen/src/lib.rs
@@ -41,7 +41,6 @@
     )
 )]
 #![no_std]
-#![cfg_attr(not(feature = "std"), feature(alloc))]
 
 #[cfg(not(feature = "std"))]
 #[macro_use]
diff --git a/cranelift/entity/src/lib.rs b/cranelift/entity/src/lib.rs
index 57287d6c79..aa10264ab8 100644
--- a/cranelift/entity/src/lib.rs
+++ b/cranelift/entity/src/lib.rs
@@ -51,7 +51,6 @@
     )
 )]
 #![no_std]
-#![cfg_attr(not(feature = "std"), feature(alloc))]
 
 #[cfg(not(feature = "std"))]
 #[macro_use]
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index ef90f9d45a..15493d7345 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -15,7 +15,7 @@ cranelift-module = { path = "../cranelift-module", version = "0.41.0" }
 faerie = "0.10.0"
 goblin = "0.0.24"
 failure = "0.1.2"
-target-lexicon = "0.4.0"
+target-lexicon = "0.8.0"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 5f9d8dceca..32348e6311 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -12,7 +12,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
-target-lexicon = { version = "0.4.0", default-features = false }
+target-lexicon = "0.8.0"
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 
diff --git a/cranelift/frontend/src/lib.rs b/cranelift/frontend/src/lib.rs
index 4cfb49939a..5b426d608f 100644
--- a/cranelift/frontend/src/lib.rs
+++ b/cranelift/frontend/src/lib.rs
@@ -175,7 +175,6 @@
     )
 )]
 #![no_std]
-#![cfg_attr(not(feature = "std"), feature(alloc))]
 
 #[cfg(not(feature = "std"))]
 #[macro_use]
diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
index fb46b96311..adde8dce3a 100644
--- a/cranelift/fuzz/Cargo.toml
+++ b/cranelift/fuzz/Cargo.toml
@@ -15,7 +15,7 @@ libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" }
 cranelift-codegen = { path = "../cranelift-codegen" }
 cranelift-wasm = { path = "../cranelift-wasm" }
 cranelift-reader = { path = "../cranelift-reader" }
-target-lexicon = "0.4.0"
+target-lexicon = "0.8.0"
 
 # Prevent this from interfering with workspaces
 [workspace]
diff --git a/cranelift/module/src/lib.rs b/cranelift/module/src/lib.rs
index 33d12d96c2..7cb9c10f22 100644
--- a/cranelift/module/src/lib.rs
+++ b/cranelift/module/src/lib.rs
@@ -19,7 +19,6 @@
     )
 )]
 #![no_std]
-#![cfg_attr(not(feature = "std"), feature(alloc))]
 
 #[cfg(not(feature = "std"))]
 #[macro_use]
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 0e7028b451..5023efaf5d 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -11,14 +11,14 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
-target-lexicon = { version = "0.4.0", default-features = false }
+target-lexicon = "0.8.0"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
 raw-cpuid = "6.0.0"
 
 [features]
 default = ["std"]
-std = ["cranelift-codegen/std", "target-lexicon/std"]
+std = ["cranelift-codegen/std"]
 # when compiling with the "core" feature, nightly must be enabled
 # enabling the "nightly" feature for raw-cpuid allows avoiding
 # linking in a c-library.
diff --git a/cranelift/preopt/src/lib.rs b/cranelift/preopt/src/lib.rs
index e8cf7be321..c35eea14a2 100644
--- a/cranelift/preopt/src/lib.rs
+++ b/cranelift/preopt/src/lib.rs
@@ -19,7 +19,6 @@
     )
 )]
 #![no_std]
-#![cfg_attr(not(feature = "std"), feature(alloc))]
 
 #[cfg(not(feature = "std"))]
 #[macro_use]
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 0a86a8af35..2678a91395 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -11,7 +11,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0" }
-target-lexicon = "0.4.0"
+target-lexicon = "0.8.0"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index ecea5d58fb..3210717c7d 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -16,7 +16,7 @@ cranelift-native = { path = "../cranelift-native", version = "0.41.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
-target-lexicon = { version = "0.4.0" }
+target-lexicon = "0.8.0"
 memmap = { version = "0.7.0", optional = true } 
 
 [target.'cfg(target_os = "windows")'.dependencies]
diff --git a/cranelift/src/disasm.rs b/cranelift/src/disasm.rs
index 7a98e02b0b..9cd6851de7 100644
--- a/cranelift/src/disasm.rs
+++ b/cranelift/src/disasm.rs
@@ -136,16 +136,20 @@ cfg_if! {
                     .x86()
                     .mode(arch::x86::ArchMode::Mode64)
                     .build(),
-                Architecture::Arm
-                | Architecture::Armv4t
-                | Architecture::Armv5te
-                | Architecture::Armv7
-                | Architecture::Armv7s => Capstone::new().arm().mode(arch::arm::ArchMode::Arm).build(),
-                Architecture::Thumbv6m | Architecture::Thumbv7em | Architecture::Thumbv7m => Capstone::new(
-                ).arm()
-                    .mode(arch::arm::ArchMode::Thumb)
-                    .build(),
-                Architecture::Aarch64 => Capstone::new()
+                Architecture::Arm(arm) => {
+                    if arm.is_thumb() {
+                        Capstone::new()
+                            .arm()
+                            .mode(arch::arm::ArchMode::Thumb)
+                            .build()
+                    } else {
+                        Capstone::new()
+                            .arm()
+                            .mode(arch::arm::ArchMode::Arm)
+                            .build()
+                    }
+                }
+                Architecture::Aarch64 {..} => Capstone::new()
                     .arm64()
                     .mode(arch::arm64::ArchMode::Arm)
                     .build(),
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 8f2346d6a7..18f9b6f4a6 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -23,7 +23,7 @@ serde = { version = "1.0.94", features = ["derive"], optional = true }
 
 [dev-dependencies]
 wabt = "0.9.1"
-target-lexicon = "0.4.0"
+target-lexicon = "0.8.0"
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/src/lib.rs b/cranelift/wasm/src/lib.rs
index c4770a68ea..3c1065228b 100644
--- a/cranelift/wasm/src/lib.rs
+++ b/cranelift/wasm/src/lib.rs
@@ -28,7 +28,6 @@
     )
 )]
 #![no_std]
-#![cfg_attr(not(feature = "std"), feature(alloc))]
 
 #[cfg(not(feature = "std"))]
 #[macro_use]

From 0f4101a509c4ad1f67df07cb061be4e40d09a6d4 Mon Sep 17 00:00:00 2001
From: Pat Hickey 
Date: Wed, 4 Sep 2019 12:26:28 -0700
Subject: [PATCH 2632/3084] upgrade to faerie 0.11.0

which fixes a use-after-free bug under the hood
---
 cranelift/faerie/Cargo.toml     | 2 +-
 cranelift/faerie/src/backend.rs | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 15493d7345..26f497d3bd 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -12,7 +12,7 @@ edition = "2018"
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0" }
 cranelift-module = { path = "../cranelift-module", version = "0.41.0" }
-faerie = "0.10.0"
+faerie = "0.11.0"
 goblin = "0.0.24"
 failure = "0.1.2"
 target-lexicon = "0.8.0"
diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index d605757c4e..4a4d189612 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -342,7 +342,7 @@ fn translate_function_linkage(linkage: Linkage) -> faerie::Decl {
 }
 
 fn translate_data_linkage(linkage: Linkage, writable: bool, align: Option) -> faerie::Decl {
-    let align = align.map(|align| usize::from(align));
+    let align = align.map(|align| u64::from(align));
     match linkage {
         Linkage::Import => faerie::Decl::data_import().into(),
         Linkage::Local => faerie::Decl::data()

From 7e398af9992307cc3e38c7d06949b474825a1251 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Thu, 5 Sep 2019 14:55:35 +0200
Subject: [PATCH 2633/3084] Basic-block: Insert regmove instructions in new
 blocks dedicated to hold the diversions.

---
 cranelift/codegen/src/context.rs              |   2 +-
 cranelift/codegen/src/regalloc/coloring.rs    |  96 ++++++++++++-
 cranelift/codegen/src/regalloc/context.rs     |  12 +-
 cranelift/codegen/src/verifier/locations.rs   |  42 ++++--
 .../filetests/isa/x86/legalize-br-table.clif  |   1 +
 .../filetests/filetests/safepoint/basic.clif  |   2 +-
 .../filetests/filetests/safepoint/call.clif   |   2 +-
 cranelift/src/bugpoint_test.clif              | 129 ++++++++++++++++++
 8 files changed, 264 insertions(+), 22 deletions(-)

diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index aeec864d79..704d891776 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -212,7 +212,7 @@ impl Context {
     /// Run the locations verifier on the function.
     pub fn verify_locations(&self, isa: &dyn TargetIsa) -> VerifierResult<()> {
         let mut errors = VerifierErrors::default();
-        let _ = verify_locations(isa, &self.func, None, &mut errors);
+        let _ = verify_locations(isa, &self.func, &self.cfg, None, &mut errors);
 
         if errors.is_empty() {
             Ok(())
diff --git a/cranelift/codegen/src/regalloc/coloring.rs b/cranelift/codegen/src/regalloc/coloring.rs
index 507d5e3ff7..f23c738583 100644
--- a/cranelift/codegen/src/regalloc/coloring.rs
+++ b/cranelift/codegen/src/regalloc/coloring.rs
@@ -44,18 +44,19 @@
 
 use crate::cursor::{Cursor, EncCursor};
 use crate::dominator_tree::DominatorTree;
+use crate::flowgraph::ControlFlowGraph;
 use crate::ir::{ArgumentLoc, InstBuilder, ValueDef};
 use crate::ir::{Ebb, Function, Inst, InstructionData, Layout, Opcode, SigRef, Value, ValueLoc};
 use crate::isa::{regs_overlap, RegClass, RegInfo, RegUnit};
 use crate::isa::{ConstraintKind, EncInfo, OperandConstraint, RecipeConstraints, TargetIsa};
 use crate::packed_option::PackedOption;
 use crate::regalloc::affinity::Affinity;
+use crate::regalloc::diversion::RegDiversions;
 use crate::regalloc::live_value_tracker::{LiveValue, LiveValueTracker};
 use crate::regalloc::liveness::Liveness;
 use crate::regalloc::liverange::{LiveRange, LiveRangeContext};
 use crate::regalloc::register_set::RegisterSet;
 use crate::regalloc::solver::{Solver, SolverError};
-use crate::regalloc::RegDiversions;
 use crate::timing;
 use core::mem;
 use log::debug;
@@ -92,6 +93,7 @@ struct Context<'a> {
     encinfo: EncInfo,
 
     // References to contextual data structures we need.
+    cfg: &'a ControlFlowGraph,
     domtree: &'a DominatorTree,
     liveness: &'a mut Liveness,
 
@@ -126,6 +128,7 @@ impl Coloring {
         &mut self,
         isa: &dyn TargetIsa,
         func: &mut Function,
+        cfg: &ControlFlowGraph,
         domtree: &DominatorTree,
         liveness: &mut Liveness,
         tracker: &mut LiveValueTracker,
@@ -137,6 +140,7 @@ impl Coloring {
             cur: EncCursor::new(func, isa),
             reginfo: isa.register_info(),
             encinfo: isa.encoding_info(),
+            cfg,
             domtree,
             liveness,
             divert: &mut self.divert,
@@ -166,13 +170,13 @@ impl<'a> Context<'a> {
         debug!("Coloring {}:", ebb);
         let mut regs = self.visit_ebb_header(ebb, tracker);
         tracker.drop_dead_params();
-        self.divert.clear();
 
         // Now go through the instructions in `ebb` and color the values they define.
         self.cur.goto_top(ebb);
         while let Some(inst) = self.cur.next_inst() {
             self.cur.use_srcloc(inst);
-            if !self.cur.func.dfg[inst].opcode().is_ghost() {
+            let opcode = self.cur.func.dfg[inst].opcode();
+            if !opcode.is_ghost() {
                 // This is an instruction which either has an encoding or carries ABI-related
                 // register allocation constraints.
                 let enc = self.cur.func.encodings[inst];
@@ -189,6 +193,54 @@ impl<'a> Context<'a> {
                 self.process_ghost_kills(kills, &mut regs);
             }
             tracker.drop_dead(inst);
+
+            // We are not able to insert any regmove for diversion or un-diversion after the first
+            // branch. Instead, we record the diversion to be restored at the entry of the next EBB,
+            // which should have a single predecessor.
+            if opcode.is_branch() && cfg!(feature = "basic-blocks") {
+                // The next instruction is necessarily an unconditional branch.
+                if let Some(branch) = self.cur.next_inst() {
+                    debug!(
+                        "Skip coloring {}\n    from {}\n    with diversions {}",
+                        self.cur.display_inst(branch),
+                        regs.input.display(&self.reginfo),
+                        self.divert.display(&self.reginfo)
+                    );
+                    use crate::ir::instructions::BranchInfo::*;
+                    let target = match self.cur.func.dfg.analyze_branch(branch) {
+                        NotABranch | Table(_, _) => panic!(
+                            "unexpected instruction {} after a conditional branch",
+                            self.cur.display_inst(branch)
+                        ),
+                        SingleDest(ebb, _) => ebb,
+                    };
+
+                    // We have a single branch with a single target, and an EBB with a single
+                    // predecessor. Thus we can forward the diversion set to the next EBB.
+                    if self.cfg.pred_iter(target).count() == 1 {
+                        // Transfer the diversion to the next EBB.
+                        self.divert
+                            .save_for_ebb(&mut self.cur.func.entry_diversions, target);
+                        debug!(
+                            "Set entry-diversion for {} to\n      {}",
+                            target,
+                            self.divert.display(&self.reginfo)
+                        );
+                    } else {
+                        debug_assert!(
+                            self.divert.is_empty(),
+                            "Divert set is non-empty after the terminator."
+                        );
+                    }
+                    assert_eq!(
+                        self.cur.next_inst(),
+                        None,
+                        "Unexpected instruction after a branch group."
+                    );
+                } else {
+                    assert!(opcode.is_terminator());
+                }
+            }
         }
     }
 
@@ -205,7 +257,15 @@ impl<'a> Context<'a> {
             self.domtree,
         );
 
+        // Copy the content of the registered diversions to be reused at the
+        // entry of this basic block.
         self.divert.at_ebb(&self.cur.func.entry_diversions, ebb);
+        debug!(
+            "Start {} with entry-diversion set to\n      {}",
+            ebb,
+            self.divert.display(&self.reginfo)
+        );
+
         if self.cur.func.layout.entry_block() == Some(ebb) {
             // Parameters on the entry block have ABI constraints.
             self.color_entry_params(tracker.live())
@@ -231,17 +291,35 @@ impl<'a> Context<'a> {
                 "Live-in: {}:{} in {}",
                 lv.value,
                 lv.affinity.display(&self.reginfo),
-                self.cur.func.locations[lv.value].display(&self.reginfo)
+                self.divert
+                    .get(lv.value, &self.cur.func.locations)
+                    .display(&self.reginfo)
             );
             if let Affinity::Reg(rci) = lv.affinity {
                 let rc = self.reginfo.rc(rci);
                 let loc = self.cur.func.locations[lv.value];
-                match loc {
-                    ValueLoc::Reg(reg) => regs.take(rc, reg, lv.is_local),
+                let reg = match loc {
+                    ValueLoc::Reg(reg) => reg,
                     ValueLoc::Unassigned => panic!("Live-in {} wasn't assigned", lv.value),
                     ValueLoc::Stack(ss) => {
                         panic!("Live-in {} is in {}, should be register", lv.value, ss)
                     }
+                };
+                if lv.is_local {
+                    regs.take(rc, reg, lv.is_local);
+                } else {
+                    let loc = self.divert.get(lv.value, &self.cur.func.locations);
+                    let reg_divert = match loc {
+                        ValueLoc::Reg(reg) => reg,
+                        ValueLoc::Unassigned => {
+                            panic!("Diversion: Live-in {} wasn't assigned", lv.value)
+                        }
+                        ValueLoc::Stack(ss) => panic!(
+                            "Diversion: Live-in {} is in {}, should be register",
+                            lv.value, ss
+                        ),
+                    };
+                    regs.take_divert(rc, reg, reg_divert);
                 }
             }
         }
@@ -1155,4 +1233,10 @@ impl AvailableRegs {
             self.global.take(rc, reg);
         }
     }
+
+    /// Take a diverted register from both sets for a non-local allocation.
+    pub fn take_divert(&mut self, rc: RegClass, reg: RegUnit, reg_divert: RegUnit) {
+        self.input.take(rc, reg_divert);
+        self.global.take(rc, reg);
+    }
 }
diff --git a/cranelift/codegen/src/regalloc/context.rs b/cranelift/codegen/src/regalloc/context.rs
index 84fe139d87..c5f5da4ecb 100644
--- a/cranelift/codegen/src/regalloc/context.rs
+++ b/cranelift/codegen/src/regalloc/context.rs
@@ -198,8 +198,14 @@ impl Context {
         }
 
         // Pass: Coloring.
-        self.coloring
-            .run(isa, func, domtree, &mut self.liveness, &mut self.tracker);
+        self.coloring.run(
+            isa,
+            func,
+            cfg,
+            domtree,
+            &mut self.liveness,
+            &mut self.tracker,
+        );
 
         // This function runs after register allocation has taken
         // place, meaning values have locations assigned already.
@@ -218,7 +224,7 @@ impl Context {
         if isa.flags().enable_verifier() {
             let ok = verify_context(func, cfg, domtree, isa, &mut errors).is_ok()
                 && verify_liveness(isa, func, cfg, &self.liveness, &mut errors).is_ok()
-                && verify_locations(isa, func, Some(&self.liveness), &mut errors).is_ok()
+                && verify_locations(isa, func, cfg, Some(&self.liveness), &mut errors).is_ok()
                 && verify_cssa(
                     func,
                     cfg,
diff --git a/cranelift/codegen/src/verifier/locations.rs b/cranelift/codegen/src/verifier/locations.rs
index 8c290bb281..bf1a4e1860 100644
--- a/cranelift/codegen/src/verifier/locations.rs
+++ b/cranelift/codegen/src/verifier/locations.rs
@@ -1,5 +1,6 @@
 //! Verify value locations.
 
+use crate::flowgraph::ControlFlowGraph;
 use crate::ir;
 use crate::isa;
 use crate::regalloc::liveness::Liveness;
@@ -21,6 +22,7 @@ use crate::verifier::{VerifierErrors, VerifierStepResult};
 pub fn verify_locations(
     isa: &dyn isa::TargetIsa,
     func: &ir::Function,
+    cfg: &ControlFlowGraph,
     liveness: Option<&Liveness>,
     errors: &mut VerifierErrors,
 ) -> VerifierStepResult<()> {
@@ -30,6 +32,7 @@ pub fn verify_locations(
         func,
         reginfo: isa.register_info(),
         encinfo: isa.encoding_info(),
+        cfg,
         liveness,
     };
     verifier.check_constraints(errors)?;
@@ -41,6 +44,7 @@ struct LocationVerifier<'a> {
     func: &'a ir::Function,
     reginfo: isa::RegInfo,
     encinfo: isa::EncInfo,
+    cfg: &'a ControlFlowGraph,
     liveness: Option<&'a Liveness>,
 }
 
@@ -53,6 +57,7 @@ impl<'a> LocationVerifier<'a> {
         for ebb in self.func.layout.ebbs() {
             divert.at_ebb(&self.func.entry_diversions, ebb);
 
+            let mut is_after_branch = false;
             for inst in self.func.layout.ebb_insts(ebb) {
                 let enc = self.func.encodings[inst];
 
@@ -70,10 +75,11 @@ impl<'a> LocationVerifier<'a> {
                 if opcode.is_return() {
                     self.check_return_abi(inst, &divert, errors)?;
                 } else if opcode.is_branch() && !divert.is_empty() {
-                    self.check_cfg_edges(inst, &divert, errors)?;
+                    self.check_cfg_edges(inst, &mut divert, is_after_branch, errors)?;
                 }
 
                 self.update_diversions(inst, &mut divert, errors)?;
+                is_after_branch = opcode.is_branch();
             }
         }
 
@@ -284,8 +290,9 @@ impl<'a> LocationVerifier<'a> {
             return fatal!(
                 errors,
                 inst,
-                "inconsistent with global location {}",
-                self.func.locations[arg].display(&self.reginfo)
+                "inconsistent with global location {} ({})",
+                self.func.locations[arg].display(&self.reginfo),
+                self.func.dfg.display_inst(inst, None)
             );
         }
 
@@ -299,37 +306,52 @@ impl<'a> LocationVerifier<'a> {
     fn check_cfg_edges(
         &self,
         inst: ir::Inst,
-        divert: &RegDiversions,
+        divert: &mut RegDiversions,
+        is_after_branch: bool,
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         use crate::ir::instructions::BranchInfo::*;
+        let dfg = &self.func.dfg;
+        let branch_kind = dfg.analyze_branch(inst);
 
         // We can only check CFG edges if we have a liveness analysis.
         let liveness = match self.liveness {
             Some(l) => l,
             None => return Ok(()),
         };
-        let dfg = &self.func.dfg;
 
-        match dfg.analyze_branch(inst) {
+        match branch_kind {
             NotABranch => panic!(
                 "No branch information for {}",
                 dfg.display_inst(inst, self.isa)
             ),
             SingleDest(ebb, _) => {
+                let unique_predecessor = self.cfg.pred_iter(ebb).count() == 1;
+                let mut val_to_remove = vec![];
                 for (&value, d) in divert.iter() {
                     let lr = &liveness[value];
-                    if lr.is_livein(ebb, liveness.context(&self.func.layout)) {
+                    if is_after_branch && unique_predecessor {
+                        // Forward diversions based on the targeted branch.
+                        if !lr.is_livein(ebb, liveness.context(&self.func.layout)) {
+                            val_to_remove.push(value)
+                        }
+                    } else if lr.is_livein(ebb, liveness.context(&self.func.layout)) {
                         return fatal!(
                             errors,
                             inst,
-                            "{} is diverted to {} and live in to {}",
+                            "SingleDest: {} is diverted to {} and live in to {}",
                             value,
                             d.to.display(&self.reginfo),
                             ebb
                         );
                     }
                 }
+                if is_after_branch && unique_predecessor {
+                    for val in val_to_remove.into_iter() {
+                        divert.remove(val);
+                    }
+                    debug_assert!(divert.check_ebb_entry(&self.func.entry_diversions, ebb));
+                }
             }
             Table(jt, ebb) => {
                 for (&value, d) in divert.iter() {
@@ -339,7 +361,7 @@ impl<'a> LocationVerifier<'a> {
                             return fatal!(
                                 errors,
                                 inst,
-                                "{} is diverted to {} and live in to {}",
+                                "Table.default: {} is diverted to {} and live in to {}",
                                 value,
                                 d.to.display(&self.reginfo),
                                 ebb
@@ -351,7 +373,7 @@ impl<'a> LocationVerifier<'a> {
                             return fatal!(
                                 errors,
                                 inst,
-                                "{} is diverted to {} and live in to {}",
+                                "Table.case: {} is diverted to {} and live in to {}",
                                 value,
                                 d.to.display(&self.reginfo),
                                 ebb
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
index 6d49a6e6d0..e970136ac8 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
@@ -1,6 +1,7 @@
 test compile
 
 target x86_64
+feature !"basic-blocks"
 ; regex: V=v\d+
 ; regex: EBB=ebb\d+
 
diff --git a/cranelift/filetests/filetests/safepoint/basic.clif b/cranelift/filetests/filetests/safepoint/basic.clif
index b47a5bf393..820594e85b 100644
--- a/cranelift/filetests/filetests/safepoint/basic.clif
+++ b/cranelift/filetests/filetests/safepoint/basic.clif
@@ -1,7 +1,7 @@
 test safepoint
-
 set enable_safepoints=true
 target x86_64
+feature !"basic-blocks"
 
 function %test(i32, r64, r64) -> r64 {
     ebb0(v0: i32, v1:r64, v2:r64):
diff --git a/cranelift/filetests/filetests/safepoint/call.clif b/cranelift/filetests/filetests/safepoint/call.clif
index 489ebd0438..a322445319 100644
--- a/cranelift/filetests/filetests/safepoint/call.clif
+++ b/cranelift/filetests/filetests/safepoint/call.clif
@@ -1,7 +1,7 @@
 test safepoint
-
 set enable_safepoints=true
 target x86_64
+feature !"basic-blocks"
 
 function %direct() -> r64 {
     fn0 = %none()
diff --git a/cranelift/src/bugpoint_test.clif b/cranelift/src/bugpoint_test.clif
index 20c34e2351..157411ed75 100644
--- a/cranelift/src/bugpoint_test.clif
+++ b/cranelift/src/bugpoint_test.clif
@@ -554,6 +554,9 @@ ebb18:
     v196 = load.i8 v195
     v197 = uextend.i32 v196
     brz v197, ebb19
+    jump ebb164
+
+ebb164:
     v198 = global_value.i64 gv12
     trap user0
 
@@ -572,6 +575,9 @@ ebb19:
     v209 = load.i8 v208
     v210 = uextend.i32 v209
     brz v210, ebb20
+    jump ebb163
+
+ebb163:
     v211 = global_value.i64 gv13
     trap user0
 
@@ -605,6 +611,9 @@ ebb22:
     v233 = load.i8 v232
     v234 = uextend.i32 v233
     brz v234, ebb23
+    jump ebb162
+
+ebb162:
     v235 = global_value.i64 gv16
     trap user0
 
@@ -630,6 +639,9 @@ ebb24:
     v252 = load.i8 v251
     v253 = uextend.i32 v252
     brz v253, ebb25
+    jump ebb161
+
+ebb161:
     v254 = global_value.i64 gv17
     trap user0
 
@@ -666,6 +678,9 @@ ebb27:
     v278 = load.i8 v277
     v279 = uextend.i32 v278
     brz v279, ebb28
+    jump ebb160
+
+ebb160:
     v280 = global_value.i64 gv18
     trap user0
 
@@ -681,6 +696,9 @@ ebb28:
     v289 = load.i8 v288
     v290 = uextend.i32 v289
     brz v290, ebb29
+    jump ebb159
+
+ebb159:
     v291 = global_value.i64 gv19
     trap user0
 
@@ -699,6 +717,9 @@ ebb29:
     v302 = load.i8 v301
     v303 = uextend.i32 v302
     brz v303, ebb30
+    jump ebb158
+
+ebb158:
     v304 = global_value.i64 gv20
     trap user0
 
@@ -714,6 +735,9 @@ ebb30:
     v313 = load.i8 v312
     v314 = uextend.i32 v313
     brz v314, ebb31
+    jump ebb157
+
+ebb157:
     v315 = global_value.i64 gv21
     trap user0
 
@@ -887,6 +911,9 @@ ebb49(v1006: i16):
     v411 = load.i8 v410
     v412 = uextend.i32 v411
     brz v412, ebb50
+    jump ebb156
+
+ebb156:
     v413 = global_value.i64 gv28
     trap user0
 
@@ -908,6 +935,9 @@ ebb50:
     v424 = load.i8 v423
     v425 = uextend.i32 v424
     brz v425, ebb51
+    jump ebb155
+
+ebb155:
     v426 = global_value.i64 gv29
     trap user0
 
@@ -922,6 +952,9 @@ ebb51:
     v432 = bint.i8 v431
     v433 = uextend.i32 v432
     brz v433, ebb52
+    jump ebb154
+
+ebb154:
     v434 = global_value.i64 gv30
     trap user0
 
@@ -941,6 +974,9 @@ ebb52:
     v447 = load.i8 v446
     v448 = uextend.i32 v447
     brz v448, ebb53
+    jump ebb153
+
+ebb153:
     v449 = global_value.i64 gv31
     trap user0
 
@@ -960,6 +996,9 @@ ebb53:
     v462 = load.i8 v461
     v463 = uextend.i32 v462
     brz v463, ebb54
+    jump ebb152
+
+ebb152:
     v464 = global_value.i64 gv32
     trap user0
 
@@ -976,6 +1015,9 @@ ebb54:
     v474 = load.i8 v473
     v475 = uextend.i32 v474
     brz v475, ebb55
+    jump ebb151
+
+ebb151:
     v476 = global_value.i64 gv33
     trap user0
 
@@ -1002,6 +1044,9 @@ ebb56:
     v493 = load.i8 v492
     v494 = uextend.i32 v493
     brz v494, ebb57
+    jump ebb150
+
+ebb150:
     v495 = global_value.i64 gv34
     trap user0
 
@@ -1017,6 +1062,9 @@ ebb57:
     v504 = load.i8 v503
     v505 = uextend.i32 v504
     brz v505, ebb58
+    jump ebb149
+
+ebb149:
     v506 = global_value.i64 gv35
     trap user0
 
@@ -1032,6 +1080,9 @@ ebb58:
     v517 = load.i8 v516
     v518 = uextend.i32 v517
     brz v518, ebb59
+    jump ebb148
+
+ebb148:
     v519 = global_value.i64 gv36
     trap user0
 
@@ -1049,6 +1100,9 @@ ebb59:
     v530 = load.i8 v529
     v531 = uextend.i32 v530
     brz v531, ebb60
+    jump ebb147
+
+ebb147:
     v532 = global_value.i64 gv37
     trap user0
 
@@ -1065,6 +1119,9 @@ ebb60:
     v542 = load.i8 v541
     v543 = uextend.i32 v542
     brz v543, ebb61
+    jump ebb146
+
+ebb146:
     v544 = global_value.i64 gv38
     trap user0
 
@@ -1118,6 +1175,9 @@ ebb62(v552: i32, v1009: i64, v1013: i64, v1016: i64, v1019: i64, v1022: i16, v10
     v556 = bint.i8 v555
     v557 = uextend.i32 v556
     brz v557, ebb63
+    jump ebb145
+
+ebb145:
     v558 = global_value.i64 gv39
     trap user0
 
@@ -1131,6 +1191,9 @@ ebb63:
     v566 = bint.i8 v565
     v567 = uextend.i32 v566
     brz v567, ebb64
+    jump ebb144
+
+ebb144:
     v568 = global_value.i64 gv40
     trap user0
 
@@ -1175,6 +1238,9 @@ ebb68(v584: i32):
     v593 = load.i8 v592
     v594 = uextend.i32 v593
     brz v594, ebb69
+    jump ebb143
+
+ebb143:
     v595 = global_value.i64 gv43
     trap user0
 
@@ -1185,6 +1251,9 @@ ebb69:
     v600 = bint.i8 v599
     v601 = uextend.i32 v600
     brnz v601, ebb70
+    jump ebb142
+
+ebb142:
     v602 = global_value.i64 gv44
     trap user0
 
@@ -1205,6 +1274,9 @@ ebb70:
     v618 = load.i8 v617
     v619 = uextend.i32 v618
     brz v619, ebb71
+    jump ebb141
+
+ebb141:
     v620 = global_value.i64 gv45
     trap user0
 
@@ -1225,6 +1297,9 @@ ebb71:
     v632 = load.i8 v631
     v633 = uextend.i32 v632
     brz v633, ebb72
+    jump ebb140
+
+ebb140:
     v634 = global_value.i64 gv46
     trap user0
 
@@ -1240,6 +1315,9 @@ ebb72:
     v644 = load.i8 v643
     v645 = uextend.i32 v644
     brz v645, ebb73
+    jump ebb139
+
+ebb139:
     v646 = global_value.i64 gv47
     trap user0
 
@@ -1266,6 +1344,9 @@ ebb74:
     v662 = load.i8 v661
     v663 = uextend.i32 v662
     brz v663, ebb75
+    jump ebb138
+
+ebb138:
     v664 = global_value.i64 gv48
     trap user0
 
@@ -1294,6 +1375,9 @@ ebb76:
     v686 = load.i8 v685
     v687 = uextend.i32 v686
     brz v687, ebb77
+    jump ebb137
+
+ebb137:
     v688 = global_value.i64 gv49
     trap user0
 
@@ -1465,6 +1549,9 @@ ebb96:
     v790 = load.i8 v789
     v791 = uextend.i32 v790
     brz v791, ebb97
+    jump ebb136
+
+ebb136:
     v792 = global_value.i64 gv58
     trap user0
 
@@ -1476,6 +1563,9 @@ ebb97:
     v797 = bint.i8 v796
     v798 = uextend.i32 v797
     brz v798, ebb98
+    jump ebb135
+
+ebb135:
     v799 = global_value.i64 gv59
     trap user0
 
@@ -1515,6 +1605,9 @@ ebb99(v804: i64, v1035: i64, v1037: i64, v1039: i64, v1044: i64, v1052: i16, v10
     v813 = load.i8 v812
     v814 = uextend.i32 v813
     brz v814, ebb100
+    jump ebb134
+
+ebb134:
     v815 = global_value.i64 gv60
     trap user0
 
@@ -1534,6 +1627,9 @@ ebb100:
     v826 = load.i8 v825
     v827 = uextend.i32 v826
     brz v827, ebb101
+    jump ebb133
+
+ebb133:
     v828 = global_value.i64 gv61
     trap user0
 
@@ -1555,6 +1651,9 @@ ebb101:
     v839 = load.i8 v838
     v840 = uextend.i32 v839
     brz v840, ebb102
+    jump ebb132
+
+ebb132:
     v841 = global_value.i64 gv62
     trap user0
 
@@ -1574,6 +1673,9 @@ ebb102:
     v852 = load.i8 v851
     v853 = uextend.i32 v852
     brz v853, ebb103
+    jump ebb131
+
+ebb131:
     v854 = global_value.i64 gv63
     trap user0
 
@@ -1591,6 +1693,9 @@ ebb103:
     v866 = load.i8 v865
     v867 = uextend.i32 v866
     brz v867, ebb104
+    jump ebb130
+
+ebb130:
     v868 = global_value.i64 gv64
     trap user0
 
@@ -1607,6 +1712,9 @@ ebb104:
     v878 = load.i8 v877
     v879 = uextend.i32 v878
     brz v879, ebb105
+    jump ebb129
+
+ebb129:
     v880 = global_value.i64 gv65
     trap user0
 
@@ -1654,6 +1762,9 @@ ebb109(v896: i64):
     v905 = load.i8 v904
     v906 = uextend.i32 v905
     brz v906, ebb110
+    jump ebb128
+
+ebb128:
     v907 = global_value.i64 gv68
     trap user0
 
@@ -1664,6 +1775,9 @@ ebb110:
     v912 = bint.i8 v911
     v913 = uextend.i32 v912
     brnz v913, ebb111
+    jump ebb127
+
+ebb127:
     v914 = global_value.i64 gv69
     trap user0
 
@@ -1684,6 +1798,9 @@ ebb111:
     v930 = load.i8 v929
     v931 = uextend.i32 v930
     brz v931, ebb112
+    jump ebb126
+
+ebb126:
     v932 = global_value.i64 gv70
     trap user0
 
@@ -1709,6 +1826,9 @@ ebb113:
     v948 = load.i8 v947
     v949 = uextend.i32 v948
     brz v949, ebb114
+    jump ebb125
+
+ebb125:
     v950 = global_value.i64 gv71
     trap user0
 
@@ -1737,6 +1857,9 @@ ebb115:
     v972 = load.i8 v971
     v973 = uextend.i32 v972
     brz v973, ebb116
+    jump ebb123
+
+ebb123:
     v974 = global_value.i64 gv72
     trap user0
 
@@ -1752,6 +1875,9 @@ ebb116:
     v984 = load.i8 v983
     v985 = uextend.i32 v984
     brz v985, ebb117
+    jump ebb122
+
+ebb122:
     v986 = global_value.i64 gv73
     trap user0
 
@@ -1775,6 +1901,9 @@ ebb119:
     v1001 = load.i8 v1000
     v1002 = uextend.i32 v1001
     brz v1002, ebb120
+    jump ebb121
+
+ebb121:
     v1003 = global_value.i64 gv74
     trap user0
 

From ea919489ee86d2b00e2e91bb318700d70fdd02f6 Mon Sep 17 00:00:00 2001
From: Ujjwal Sharma 
Date: Thu, 5 Sep 2019 18:33:13 +0530
Subject: [PATCH 2634/3084] [codegen] add encodings for iadd carry variants
 (#961)

* [codegen] add encodings for iadd carry variants

Add encodings for iadd carry variants (iadd_cout, iadd_cin, iadd_carry)
for x86_32, enabling the legalization for iadd.i64 to work.

* [codegen] remove support for iadd carry variants on riscv

Previously, the carry variants of iadd (iadd_cin, iadd_cout and
iadd_carry) were being legalized for isa/riscv since RISC architectures
lack a flags register.

This forced us to return and accept booleans for these operations, which
proved to be problematic and inconvenient, especially for x86.

This commit removes support for said statements and all dependent
statements for isa/riscv so that we can work on a better legalization
strategy in the future.

* [codegen] change operand type from bool to iflag for iadd carry variants

The type of the carry operands for the carry variants of the iadd
instruction (iadd_cin, iadd_cout, iadd_carry) was bool for compatibility
reasons for isa/riscv. Since support for these instructions on RISC
architectures has been temporarily suspended, we can safely change the
type to iflags.
---
 .../codegen/meta/src/isa/x86/encodings.rs     |  9 ++++
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 41 +++++++++++++++++++
 .../codegen/meta/src/shared/instructions.rs   |  4 +-
 cranelift/codegen/meta/src/shared/legalize.rs | 28 -------------
 cranelift/codegen/src/ir/dfg.rs               |  5 +--
 .../{expand-i32.clif => expand-i32.clif.bak}  |  3 ++
 ...egalize-i64.clif => legalize-i64.clif.bak} |  3 ++
 .../filetests/filetests/isa/x86/binary32.clif |  8 ++++
 .../filetests/isa/x86/legalize-i64.clif       | 16 ++++++++
 9 files changed, 84 insertions(+), 33 deletions(-)
 rename cranelift/filetests/filetests/isa/riscv/{expand-i32.clif => expand-i32.clif.bak} (87%)
 rename cranelift/filetests/filetests/isa/riscv/{legalize-i64.clif => legalize-i64.clif.bak} (94%)
 create mode 100644 cranelift/filetests/filetests/isa/x86/legalize-i64.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 02186983b3..ab929011b7 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -372,6 +372,9 @@ pub fn define(
     let fsub = shared.by_name("fsub");
     let func_addr = shared.by_name("func_addr");
     let iadd = shared.by_name("iadd");
+    let iadd_cout = shared.by_name("iadd_cout");
+    let iadd_cin = shared.by_name("iadd_cin");
+    let iadd_carry = shared.by_name("iadd_carry");
     let iadd_imm = shared.by_name("iadd_imm");
     let icmp = shared.by_name("icmp");
     let icmp_imm = shared.by_name("icmp_imm");
@@ -556,6 +559,8 @@ pub fn define(
     let rec_rfurm = r.template("rfurm");
     let rec_rmov = r.template("rmov");
     let rec_rr = r.template("rr");
+    let rec_rcin = r.template("rcin");
+    let rec_rcarry = r.template("rcarry");
     let rec_rrx = r.template("rrx");
     let rec_safepoint = r.recipe("safepoint");
     let rec_setf_abcd = r.template("setf_abcd");
@@ -611,6 +616,10 @@ pub fn define(
     let mut e = PerCpuModeEncodings::new();
 
     e.enc_i32_i64(iadd, rec_rr.opcodes(vec![0x01]));
+    e.enc_i32_i64(iadd_cout, rec_rr.opcodes(vec![0x01]));
+    e.enc_i32_i64(iadd_cin, rec_rcin.opcodes(vec![0x11]));
+    e.enc_i32_i64(iadd_carry, rec_rcarry.opcodes(vec![0x11]));
+
     e.enc_i32_i64(isub, rec_rr.opcodes(vec![0x29]));
     e.enc_i32_i64(band, rec_rr.opcodes(vec![0x21]));
     e.enc_i32_i64(bor, rec_rr.opcodes(vec![0x09]));
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index d126580804..6da640a172 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -2528,6 +2528,47 @@ pub fn define<'shared>(
             ),
     );
 
+    // Adding with carry
+
+    // XX /r, MR form. Add two GPR registers and get carry flag.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("rcin", f_ternary, 1)
+            .operands_in(vec![
+                OperandConstraint::RegClass(gpr),
+                OperandConstraint::RegClass(gpr),
+                OperandConstraint::FixedReg(reg_rflags),
+            ])
+            .operands_out(vec![0])
+            .clobbers_flags(true)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
+                    modrm_rr(in_reg0, in_reg1, sink);
+                "#,
+            ),
+    );
+
+    // XX /r, MR form. Add two GPR registers with carry flag.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("rcarry", f_ternary, 1)
+            .operands_in(vec![
+                OperandConstraint::RegClass(gpr),
+                OperandConstraint::RegClass(gpr),
+                OperandConstraint::FixedReg(reg_rflags),
+            ])
+            .operands_out(vec![
+                OperandConstraint::TiedInput(0),
+                OperandConstraint::FixedReg(reg_rflags),
+            ])
+            .clobbers_flags(true)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
+                    modrm_rr(in_reg0, in_reg1, sink);
+                "#,
+            ),
+    );
+
     // Compare and set flags.
 
     // XX /r, MR form. Compare two GPR registers and set flags.
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index a40a6dc247..02819be42b 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1864,8 +1864,8 @@ pub fn define(
     let a = &operand("a", iB);
     let x = &operand("x", iB);
     let y = &operand("y", iB);
-    let c_in = &operand_doc("c_in", b1, "Input carry flag");
-    let c_out = &operand_doc("c_out", b1, "Output carry flag");
+    let c_in = &operand_doc("c_in", iflags, "Input carry flag");
+    let c_out = &operand_doc("c_out", iflags, "Output carry flag");
     let b_in = &operand_doc("b_in", b1, "Input borrow flag");
     let b_out = &operand_doc("b_out", b1, "Output borrow flag");
 
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 004bdff95b..05c74fdc47 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -66,7 +66,6 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     let fcvt_from_uint = insts.by_name("fcvt_from_uint");
     let fneg = insts.by_name("fneg");
     let iadd = insts.by_name("iadd");
-    let iadd_carry = insts.by_name("iadd_carry");
     let iadd_cin = insts.by_name("iadd_cin");
     let iadd_cout = insts.by_name("iadd_cout");
     let iadd_imm = insts.by_name("iadd_imm");
@@ -168,8 +167,6 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     let c2 = var("c2");
     let c3 = var("c3");
     let c4 = var("c4");
-    let c_in = var("c_in");
-    let c_int = var("c_int");
     let d = var("d");
     let d1 = var("d1");
     let d2 = var("d2");
@@ -464,27 +461,12 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
 
     // Expand integer operations with carry for RISC architectures that don't have
     // the flags.
-    let intcc_ult = Literal::enumerator_for(intcc, "ult");
-    expand.legalize(
-        def!((a, c) = iadd_cout(x, y)),
-        vec![def!(a = iadd(x, y)), def!(c = icmp(intcc_ult, a, x))],
-    );
-
     let intcc_ugt = Literal::enumerator_for(intcc, "ugt");
     expand.legalize(
         def!((a, b) = isub_bout(x, y)),
         vec![def!(a = isub(x, y)), def!(b = icmp(intcc_ugt, a, x))],
     );
 
-    expand.legalize(
-        def!(a = iadd_cin(x, y, c)),
-        vec![
-            def!(a1 = iadd(x, y)),
-            def!(c_int = bint(c)),
-            def!(a = iadd(a1, c_int)),
-        ],
-    );
-
     expand.legalize(
         def!(a = isub_bin(x, y, b)),
         vec![
@@ -494,16 +476,6 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
         ],
     );
 
-    expand.legalize(
-        def!((a, c) = iadd_carry(x, y, c_in)),
-        vec![
-            def!((a1, c1) = iadd_cout(x, y)),
-            def!(c_int = bint(c_in)),
-            def!((a, c2) = iadd_cout(a1, c_int)),
-            def!(c = bor(c1, c2)),
-        ],
-    );
-
     expand.legalize(
         def!((a, b) = isub_borrow(x, y, b_in)),
         vec![
diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index 7392e893e7..24eeff4eec 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -1239,7 +1239,6 @@ mod tests {
 
     #[test]
     fn aliases() {
-        use crate::ir::condcodes::IntCC;
         use crate::ir::InstBuilder;
 
         let mut func = Function::new();
@@ -1264,9 +1263,9 @@ mod tests {
         pos.func.dfg.clear_results(iadd);
         pos.func.dfg.attach_result(iadd, s);
 
-        // Replace `iadd_cout` with a normal `iadd` and an `icmp`.
+        // Replace `iadd_cout` with a normal `iadd` and an `ifcmp`.
         pos.func.dfg.replace(iadd).iadd(v1, arg0);
-        let c2 = pos.ins().icmp(IntCC::UnsignedLessThan, s, v1);
+        let c2 = pos.ins().ifcmp(s, v1);
         pos.func.dfg.change_to_alias(c, c2);
 
         assert_eq!(pos.func.dfg.resolve_aliases(c2), c2);
diff --git a/cranelift/filetests/filetests/isa/riscv/expand-i32.clif b/cranelift/filetests/filetests/isa/riscv/expand-i32.clif.bak
similarity index 87%
rename from cranelift/filetests/filetests/isa/riscv/expand-i32.clif
rename to cranelift/filetests/filetests/isa/riscv/expand-i32.clif.bak
index eb63d7cdcd..cc2a9d726c 100644
--- a/cranelift/filetests/filetests/isa/riscv/expand-i32.clif
+++ b/cranelift/filetests/filetests/isa/riscv/expand-i32.clif.bak
@@ -1,3 +1,6 @@
+; TODO(ryzokuken): figure out a better legalization strategy for platforms that
+; platforms that don't have flags.
+
 ; Test the legalization of i32 instructions that don't have RISC-V versions.
 test legalizer
 
diff --git a/cranelift/filetests/filetests/isa/riscv/legalize-i64.clif b/cranelift/filetests/filetests/isa/riscv/legalize-i64.clif.bak
similarity index 94%
rename from cranelift/filetests/filetests/isa/riscv/legalize-i64.clif
rename to cranelift/filetests/filetests/isa/riscv/legalize-i64.clif.bak
index d043337a21..366472e7d3 100644
--- a/cranelift/filetests/filetests/isa/riscv/legalize-i64.clif
+++ b/cranelift/filetests/filetests/isa/riscv/legalize-i64.clif.bak
@@ -1,3 +1,6 @@
+; TODO(ryzokuken): figure out a better legalization strategy for platforms that
+; platforms that don't have flags.
+
 ; Test the legalization of i64 arithmetic instructions.
 test legalizer
 target riscv32 supports_m=1
diff --git a/cranelift/filetests/filetests/isa/x86/binary32.clif b/cranelift/filetests/filetests/isa/x86/binary32.clif
index 602b2d4f31..79680a939d 100644
--- a/cranelift/filetests/filetests/isa/x86/binary32.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary32.clif
@@ -469,6 +469,14 @@ ebb0:
     ; asm: mov    %cl,(%eax,%ebx,1)
     istore8_complex v601, v521+v522      ; bin: heap_oob 88 0c 18
 
+    ; Carry Addition
+    ; asm: addl %esi, %ecx
+    [-,%rcx,%rflags] v701, v702 = iadd_cout v1, v2  ; bin: 01 f1
+    ; asm: adcl %esi, %ecx
+    [-,%rcx] v703 = iadd_cin v1, v2, v702           ; bin: 11 f1
+    ; asm: adcl %esi, %ecx
+    [-,%rcx,%rflags] v704, v705 = iadd_carry v1, v2, v702 ; bin: 11 f1
+
     ; asm: testl %ecx, %ecx
     ; asm: je ebb1
     brz v1, ebb1                                ; bin: 85 c9 74 0e
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-i64.clif b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
new file mode 100644
index 0000000000..dc7d5696fe
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
@@ -0,0 +1,16 @@
+; Test the legalization of i64 instructions on x86_32.
+test legalizer
+target i686 haswell
+
+; regex: V=v\d+
+
+function %iadd(i64, i64) -> i64 {
+ebb0(v1: i64, v2: i64):
+    v10 = iadd v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; check: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; check: $(v10_lsb=$V), $(carry=$V) = iadd_cout $v1_lsb, $v2_lsb
+    ; check: $(v10_msb=$V) = iadd_cin $v1_msb, $v2_msb, $carry
+    ; check: v10 = iconcat $v10_lsb, $v10_msb
+    return v10
+}

From 26b88ae7b5bcfba1e4920eaa5dbf85fdb36cf1cd Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Thu, 5 Sep 2019 08:21:29 -0600
Subject: [PATCH 2635/3084] Limit redundant jump folding to only fold
 parameterless target blocks (#972)

---
 cranelift/codegen/src/binemit/relaxation.rs | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index 6648369cf3..bd09d473d2 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -169,6 +169,14 @@ fn try_fold_redundant_jump(
         }
     };
 
+    // For the moment, only attempt to fold a branch to an ebb that is parameterless.
+    // These blocks are mainly produced by critical edge splitting.
+    //
+    // TODO: Allow folding blocks that define SSA values and function as phi nodes.
+    if func.dfg.num_ebb_params(first_dest) != 0 {
+        return false;
+    }
+
     // Look at the first instruction of the first branch's destination.
     // If it is an unconditional branch, maybe the second jump can be bypassed.
     let second_inst = func.layout.first_inst(first_dest).expect("Instructions");

From 8a9384f86940112f5379ce7278ed136258de7ec0 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 3 Sep 2019 15:31:13 +0200
Subject: [PATCH 2636/3084] Tweak comments;

---
 .../codegen/meta/src/isa/x86/encodings.rs     |  3 +-
 cranelift/codegen/src/binemit/mod.rs          |  4 +-
 cranelift/codegen/src/binemit/relaxation.rs   |  3 +-
 cranelift/codegen/src/ir/constant.rs          | 60 +++++++++++--------
 cranelift/codegen/src/ir/immediates.rs        |  4 +-
 cranelift/reader/src/parser.rs                |  3 +-
 6 files changed, 44 insertions(+), 33 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index ab929011b7..9c7ff89615 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -279,7 +279,8 @@ impl PerCpuModeEncodings {
         }
     }
 
-    /// Add the same encoding to both X86_32 and X86_64; assumes configuration (e.g. REX, operand binding) has already happened
+    /// Add the same encoding to both X86_32 and X86_64; assumes configuration (e.g. REX, operand
+    /// binding) has already happened.
     fn enc_32_64_maybe_isap(
         &mut self,
         inst: impl Clone + Into,
diff --git a/cranelift/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
index d6ca75d2d9..0123e4dbd3 100644
--- a/cranelift/codegen/src/binemit/mod.rs
+++ b/cranelift/codegen/src/binemit/mod.rs
@@ -185,7 +185,7 @@ where
 
     sink.begin_jumptables();
 
-    // output jump tables
+    // Output jump tables.
     for (jt, jt_data) in func.jump_tables.iter() {
         let jt_offset = func.jt_offsets[jt];
         for ebb in jt_data.iter() {
@@ -196,7 +196,7 @@ where
 
     sink.begin_rodata();
 
-    // output constants
+    // Output constants.
     for (_, constant_data) in func.dfg.constants.iter() {
         for byte in constant_data.iter() {
             sink.put1(*byte)
diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index bd09d473d2..43f361885a 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -129,8 +129,7 @@ pub fn relax_branches(
 
     for (jt, jt_data) in func.jump_tables.iter() {
         func.jt_offsets[jt] = offset;
-        // TODO: this should be computed based on the min size needed to hold
-        //        the furthest branch.
+        // TODO: this should be computed based on the min size needed to hold the furthest branch.
         offset += jt_data.len() as u32 * 4;
     }
 
diff --git a/cranelift/codegen/src/ir/constant.rs b/cranelift/codegen/src/ir/constant.rs
index 50e93df1ab..c22380db55 100644
--- a/cranelift/codegen/src/ir/constant.rs
+++ b/cranelift/codegen/src/ir/constant.rs
@@ -1,9 +1,12 @@
 //! Constants
 //!
-//! The constant pool defined here allows cranelift to avoid emitting the same constant multiple
-//! times. As constants are inserted in the pool, a handle is returned; the handle is a cranelift
-//! Entity. Inserting the same data multiple times will always return the same handle. Future work
-//! could include: ensuring alignment of constants within the pool, bucketing constants by size.
+//! The constant pool defined here allows Cranelift to avoid emitting the same constant multiple
+//! times. As constants are inserted in the pool, a handle is returned; the handle is a Cranelift
+//! Entity. Inserting the same data multiple times will always return the same handle.
+//!
+//! Future work could include:
+//! - ensuring alignment of constants within the pool,
+//! - bucketing constants by size.
 
 use crate::ir::Constant;
 use cranelift_entity::EntityRef;
@@ -19,8 +22,8 @@ pub type ConstantOffset = u32;
 /// Inner type for storing data and offset together in the constant pool. The offset is optional
 /// because it must be set relative to the function code size (i.e. constants are emitted after the
 /// function body); because the function is not yet compiled when constants are inserted,
-/// [`set_offset`](crate::ir::ConstantPool::set_offset) must be called once a constant's
-/// offset from the beginning of the function is known (see
+/// [`set_offset`](crate::ir::ConstantPool::set_offset) must be called once a constant's offset
+/// from the beginning of the function is known (see
 /// [`relaxation.rs`](crate::binemit::relaxation)).
 #[derive(Clone)]
 pub struct ConstantPoolEntry {
@@ -30,7 +33,7 @@ pub struct ConstantPoolEntry {
 
 impl ConstantPoolEntry {
     fn new(data: ConstantData) -> Self {
-        ConstantPoolEntry { data, offset: None }
+        Self { data, offset: None }
     }
 
     /// Return the size of the constant at this entry.
@@ -44,21 +47,23 @@ impl ConstantPoolEntry {
     }
 }
 
-/// Maintains the mapping between a constant handle (i.e.
-/// [`Constant`](crate::ir::Constant)) and its constant data (i.e.
-/// [`ConstantData`](crate::ir::ConstantData)).
+/// Maintains the mapping between a constant handle (i.e.  [`Constant`](crate::ir::Constant)) and
+/// its constant data (i.e.  [`ConstantData`](crate::ir::ConstantData)).
 #[derive(Clone)]
 pub struct ConstantPool {
-    /// This mapping maintains the insertion order as long as Constants are created with sequentially increasing integers.
+    /// This mapping maintains the insertion order as long as Constants are created with
+    /// sequentially increasing integers.
     handles_to_values: BTreeMap,
-    /// This mapping is unordered (no need for lexicographic ordering) but allows us to map constant data back to handles.
+
+    /// This mapping is unordered (no need for lexicographic ordering) but allows us to map
+    /// constant data back to handles.
     values_to_handles: HashMap,
 }
 
 impl ConstantPool {
     /// Create a new constant pool instance.
     pub fn new() -> Self {
-        ConstantPool {
+        Self {
             handles_to_values: BTreeMap::new(),
             values_to_handles: HashMap::new(),
         }
@@ -75,10 +80,7 @@ impl ConstantPool {
     /// returned.
     pub fn insert(&mut self, constant_value: ConstantData) -> Constant {
         if self.values_to_handles.contains_key(&constant_value) {
-            self.values_to_handles
-                .get(&constant_value)
-                .expect("A constant handle must have a corresponding constant value; this is an implementation error in ConstantPool")
-                .clone()
+            self.values_to_handles.get(&constant_value).unwrap().clone()
         } else {
             let constant_handle = Constant::new(self.len());
             self.values_to_handles
@@ -94,16 +96,17 @@ impl ConstantPool {
     /// Retrieve the constant data given a handle.
     pub fn get(&self, constant_handle: Constant) -> &ConstantData {
         assert!(self.handles_to_values.contains_key(&constant_handle));
-        &self.handles_to_values
-            .get(&constant_handle)
-            .expect("A constant handle must have a corresponding constant value; was a constant handle created outside of the pool?")
-            .data
+        &self.handles_to_values.get(&constant_handle).unwrap().data
     }
 
     /// Assign an offset to a given constant, where the offset is the number of bytes from the
     /// beginning of the function to the beginning of the constant data inside the pool.
     pub fn set_offset(&mut self, constant_handle: Constant, constant_offset: ConstantOffset) {
-        assert!(self.handles_to_values.contains_key(&constant_handle), "A constant handle must have already been inserted into the pool; perhaps a constant pool was created outside of the pool?");
+        assert!(
+            self.handles_to_values.contains_key(&constant_handle),
+            "A constant handle must have already been inserted into the pool; perhaps a \
+             constant pool was created outside of the pool?"
+        );
         self.handles_to_values
             .entry(constant_handle)
             .and_modify(|e| e.offset = Some(constant_offset));
@@ -112,10 +115,17 @@ impl ConstantPool {
     /// Retrieve the offset of a given constant, where the offset is the number of bytes from the
     /// beginning of the function to the beginning of the constant data inside the pool.
     pub fn get_offset(&self, constant_handle: Constant) -> ConstantOffset {
-        self.handles_to_values.get(&constant_handle)
-            .expect("A constant handle must have a corresponding constant value; was a constant handle created outside of the pool?")
+        self.handles_to_values
+            .get(&constant_handle)
+            .expect(
+                "A constant handle must have a corresponding constant value; was a constant \
+                 handle created outside of the pool?",
+            )
             .offset
-            .expect("A constant offset has not yet been set; verify that `set_offset` has been called before this point")
+            .expect(
+                "A constant offset has not yet been set; verify that `set_offset` has been \
+                 called before this point",
+            )
     }
 
     /// Iterate over the constants in insertion order.
diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index d69aed4390..2c5ee27c47 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -292,7 +292,7 @@ impl FromStr for Uimm32 {
 
 /// A 128-bit unsigned integer immediate operand.
 ///
-/// This is used as an immediate value in SIMD instructions
+/// This is used as an immediate value in SIMD instructions.
 #[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)]
 pub struct Uimm128(pub [u8; 16]);
 
@@ -314,7 +314,7 @@ impl Uimm128 {
 }
 
 impl Display for Uimm128 {
-    // print a 128-bit vector in hexadecimal, e.g. 0x000102030405060708090a0b0c0d0e0f
+    // Print a 128-bit vector in hexadecimal, e.g. 0x000102030405060708090a0b0c0d0e0f.
     fn fmt(&self, f: &mut Formatter) -> fmt::Result {
         write!(f, "0x")?;
         let mut anything_written = false;
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 6971ca663d..7965a4a31a 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -595,7 +595,8 @@ impl<'a> Parser<'a> {
         }
     }
 
-    // Match and consume a Uimm128 immediate; due to size restrictions on InstructionData, Uimm128 is boxed in cranelift-codegen/meta/src/shared/immediates.rs
+    // Match and consume a Uimm128 immediate; due to size restrictions on InstructionData, Uimm128
+    // is boxed in cranelift-codegen/meta/src/shared/immediates.rs
     fn match_uimm128(&mut self, err_msg: &str) -> ParseResult {
         if let Some(Token::Integer(text)) = self.token() {
             self.consume();

From 0acddc08ea80497ad229e2927a592f18dca2c401 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 3 Sep 2019 16:32:32 +0200
Subject: [PATCH 2637/3084] [meta] Split FormatBuilder::imm to avoid the extra
 Into<> parameter type;

---
 cranelift/codegen/meta/src/cdsl/formats.rs   | 38 +++++++-------------
 cranelift/codegen/meta/src/shared/formats.rs | 26 ++++++++------
 2 files changed, 28 insertions(+), 36 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
index b716a8a976..a3249b4c6e 100644
--- a/cranelift/codegen/meta/src/cdsl/formats.rs
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -89,27 +89,6 @@ pub struct InstructionFormatBuilder {
     typevar_operand: Option,
 }
 
-pub struct ImmParameter {
-    kind: OperandKind,
-    member: &'static str,
-}
-impl Into for (&'static str, &OperandKind) {
-    fn into(self) -> ImmParameter {
-        ImmParameter {
-            kind: self.1.clone(),
-            member: self.0,
-        }
-    }
-}
-impl Into for &OperandKind {
-    fn into(self) -> ImmParameter {
-        ImmParameter {
-            kind: self.clone(),
-            member: self.default_member.unwrap(),
-        }
-    }
-}
-
 impl InstructionFormatBuilder {
     pub fn new(name: &'static str) -> Self {
         Self {
@@ -131,12 +110,21 @@ impl InstructionFormatBuilder {
         self
     }
 
-    pub fn imm(mut self, param: impl Into) -> Self {
-        let imm_param = param.into();
+    pub fn imm(mut self, operand_kind: &OperandKind) -> Self {
         let field = FormatField {
             immnum: self.imm_fields.len(),
-            kind: imm_param.kind,
-            member: imm_param.member,
+            kind: operand_kind.clone(),
+            member: operand_kind.default_member.unwrap(),
+        };
+        self.imm_fields.push(field);
+        self
+    }
+
+    pub fn imm_with_name(mut self, member: &'static str, operand_kind: &OperandKind) -> Self {
+        let field = FormatField {
+            immnum: self.imm_fields.len(),
+            kind: operand_kind.clone(),
+            member,
         };
         self.imm_fields.push(field);
         self
diff --git a/cranelift/codegen/meta/src/shared/formats.rs b/cranelift/codegen/meta/src/shared/formats.rs
index 394eb6d58f..2a16f250a1 100644
--- a/cranelift/codegen/meta/src/shared/formats.rs
+++ b/cranelift/codegen/meta/src/shared/formats.rs
@@ -60,10 +60,14 @@ pub fn define(immediates: &OperandKinds, entities: &OperandKinds) -> FormatRegis
     registry.insert(
         Builder::new("InsertLane")
             .value()
-            .imm(("lane", uimm8))
+            .imm_with_name("lane", uimm8)
             .value(),
     );
-    registry.insert(Builder::new("ExtractLane").value().imm(("lane", uimm8)));
+    registry.insert(
+        Builder::new("ExtractLane")
+            .value()
+            .imm_with_name("lane", uimm8),
+    );
 
     registry.insert(Builder::new("IntCompare").imm(intcc).value().value());
     registry.insert(Builder::new("IntCompareImm").imm(intcc).value().imm(imm64));
@@ -151,26 +155,26 @@ pub fn define(immediates: &OperandKinds, entities: &OperandKinds) -> FormatRegis
     registry.insert(
         Builder::new("RegMove")
             .value()
-            .imm(("src", regunit))
-            .imm(("dst", regunit)),
+            .imm_with_name("src", regunit)
+            .imm_with_name("dst", regunit),
     );
     registry.insert(
         Builder::new("CopySpecial")
-            .imm(("src", regunit))
-            .imm(("dst", regunit)),
+            .imm_with_name("src", regunit)
+            .imm_with_name("dst", regunit),
     );
-    registry.insert(Builder::new("CopyToSsa").imm(("src", regunit)));
+    registry.insert(Builder::new("CopyToSsa").imm_with_name("src", regunit));
     registry.insert(
         Builder::new("RegSpill")
             .value()
-            .imm(("src", regunit))
-            .imm(("dst", stack_slot)),
+            .imm_with_name("src", regunit)
+            .imm_with_name("dst", stack_slot),
     );
     registry.insert(
         Builder::new("RegFill")
             .value()
-            .imm(("src", stack_slot))
-            .imm(("dst", regunit)),
+            .imm_with_name("src", stack_slot)
+            .imm_with_name("dst", regunit),
     );
 
     registry.insert(Builder::new("Trap").imm(trapcode));

From 38656cce35a78d2bedcdf54008a51fc22cfbbd67 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 3 Sep 2019 18:14:58 +0200
Subject: [PATCH 2638/3084] [meta] Simplify and comment instruction building a
 bit;

---
 cranelift/codegen/meta/src/cdsl/formats.rs    |   4 +-
 .../codegen/meta/src/cdsl/instructions.rs     | 136 +++++++++---------
 2 files changed, 67 insertions(+), 73 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
index a3249b4c6e..a0dcbb67ae 100644
--- a/cranelift/codegen/meta/src/cdsl/formats.rs
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -189,7 +189,9 @@ impl FormatRegistry {
             if operand.is_value() {
                 num_values += 1;
             }
-            has_varargs = has_varargs || operand.is_varargs();
+            if !has_varargs {
+                has_varargs = operand.is_varargs();
+            }
             if let Some(imm_key) = operand.kind.imm_key() {
                 imm_keys.push(imm_key);
             }
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index f31e530771..da59beb2b0 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -113,9 +113,12 @@ pub struct InstructionContent {
     /// polymorphic, set otherwise.
     pub polymorphic_info: Option,
 
+    /// Indices in operands_in of input operands that are values.
     pub value_opnums: Vec,
-    pub value_results: Vec,
+    /// Indices in operands_in of input operands that are immediates or entities.
     pub imm_opnums: Vec,
+    /// Indices in operands_out of output operands that are values.
+    pub value_results: Vec,
 
     /// True for instructions that terminate the EBB.
     pub is_terminator: bool,
@@ -332,8 +335,6 @@ impl InstructionBuilder {
         let operands_in = self.operands_in.unwrap_or_else(Vec::new);
         let operands_out = self.operands_out.unwrap_or_else(Vec::new);
 
-        let format_index = format_registry.lookup(&operands_in);
-
         let mut value_opnums = Vec::new();
         let mut imm_opnums = Vec::new();
         for (i, op) in operands_in.iter().enumerate() {
@@ -346,13 +347,13 @@ impl InstructionBuilder {
             }
         }
 
-        let mut value_results = Vec::new();
-        for (i, op) in operands_out.iter().enumerate() {
-            if op.is_value() {
-                value_results.push(i);
-            }
-        }
+        let value_results = operands_out
+            .iter()
+            .enumerate()
+            .filter_map(|(i, op)| if op.is_value() { Some(i) } else { None })
+            .collect();
 
+        let format_index = format_registry.lookup(&operands_in);
         let format = format_registry.get(format_index);
         let polymorphic_info =
             verify_polymorphic(&operands_in, &operands_out, &format, &value_opnums);
@@ -461,12 +462,8 @@ fn verify_polymorphic(
     }
 
     // Verify the use of type variables.
-    let mut use_typevar_operand = false;
-    let mut ctrl_typevar = None;
-    let mut other_typevars = None;
-    let mut maybe_error_message = None;
-
     let tv_op = format.typevar_operand;
+    let mut maybe_error_message = None;
     if let Some(tv_op) = tv_op {
         if tv_op < value_opnums.len() {
             let op_num = value_opnums[tv_op];
@@ -475,11 +472,13 @@ fn verify_polymorphic(
             if (free_typevar.is_some() && tv == &free_typevar.unwrap())
                 || tv.singleton_type().is_some()
             {
-                match verify_ctrl_typevar(tv, &value_opnums, &operands_in, &operands_out) {
-                    Ok(typevars) => {
-                        other_typevars = Some(typevars);
-                        ctrl_typevar = Some(tv.clone());
-                        use_typevar_operand = true;
+                match is_ctrl_typevar_candidate(tv, &operands_in, &operands_out) {
+                    Ok(other_typevars) => {
+                        return Some(PolymorphicInfo {
+                            use_typevar_operand: true,
+                            ctrl_typevar: tv.clone(),
+                            other_typevars,
+                        });
                     }
                     Err(error_message) => {
                         maybe_error_message = Some(error_message);
@@ -489,33 +488,32 @@ fn verify_polymorphic(
         }
     };
 
-    if !use_typevar_operand {
-        if operands_out.len() == 0 {
-            match maybe_error_message {
-                Some(msg) => panic!(msg),
-                None => panic!("typevar_operand must be a free type variable"),
-            }
+    // If we reached here, it means the type variable indicated as the typevar operand couldn't
+    // control every other input and output type variable. We need to look at the result type
+    // variables.
+    if operands_out.len() == 0 {
+        // No result means no other possible type variable, so it's a type inference failure.
+        match maybe_error_message {
+            Some(msg) => panic!(msg),
+            None => panic!("typevar_operand must be a free type variable"),
         }
-
-        let tv = operands_out[0].type_var().unwrap();
-        let free_typevar = tv.free_typevar();
-        if free_typevar.is_some() && tv != &free_typevar.unwrap() {
-            panic!("first result must be a free type variable");
-        }
-
-        other_typevars =
-            Some(verify_ctrl_typevar(tv, &value_opnums, &operands_in, &operands_out).unwrap());
-        ctrl_typevar = Some(tv.clone());
     }
 
-    // rustc is not capable to determine this statically, so enforce it with options.
-    assert!(ctrl_typevar.is_some());
-    assert!(other_typevars.is_some());
+    // Otherwise, try to infer the controlling type variable by looking at the first result.
+    let tv = operands_out[0].type_var().unwrap();
+    let free_typevar = tv.free_typevar();
+    if free_typevar.is_some() && tv != &free_typevar.unwrap() {
+        panic!("first result must be a free type variable");
+    }
+
+    // At this point, if the next unwrap() fails, it means the output type couldn't be used as a
+    // controlling type variable either; panicking is the right behavior.
+    let other_typevars = is_ctrl_typevar_candidate(tv, &operands_in, &operands_out).unwrap();
 
     Some(PolymorphicInfo {
-        use_typevar_operand,
-        ctrl_typevar: ctrl_typevar.unwrap(),
-        other_typevars: other_typevars.unwrap(),
+        use_typevar_operand: false,
+        ctrl_typevar: tv.clone(),
+        other_typevars,
     })
 }
 
@@ -527,57 +525,51 @@ fn verify_polymorphic(
 ///
 /// All polymorphic results must be derived from `ctrl_typevar`.
 ///
-/// Return a vector of other type variables used, or panics.
-fn verify_ctrl_typevar(
+/// Return a vector of other type variables used, or a string explaining what went wrong.
+fn is_ctrl_typevar_candidate(
     ctrl_typevar: &TypeVar,
-    value_opnums: &Vec,
     operands_in: &Vec,
     operands_out: &Vec,
 ) -> Result, String> {
     let mut other_typevars = Vec::new();
 
     // Check value inputs.
-    for &op_num in value_opnums {
-        let typ = operands_in[op_num].type_var();
+    for input in operands_in {
+        if !input.is_value() {
+            continue;
+        }
 
-        let tv = if let Some(typ) = typ {
-            typ.free_typevar()
-        } else {
-            None
-        };
+        let typ = input.type_var().unwrap();
+        let free_typevar = typ.free_typevar();
 
         // Non-polymorphic or derived from ctrl_typevar is OK.
-        let tv = match tv {
-            Some(tv) => {
-                if &tv == ctrl_typevar {
-                    continue;
-                }
-                tv
-            }
-            None => continue,
-        };
+        if free_typevar.is_none() {
+            continue;
+        }
+        let free_typevar = free_typevar.unwrap();
+        if &free_typevar == ctrl_typevar {
+            continue;
+        }
 
         // No other derived typevars allowed.
-        if typ.is_some() && typ.unwrap() != &tv {
+        if typ != &free_typevar {
             return Err(format!(
-                "{:?}: type variable {} must be derived from {:?}",
-                operands_in[op_num],
-                typ.unwrap().name,
-                ctrl_typevar
+                "{:?}: type variable {} must be derived from {:?} while it is derived from {:?}",
+                input, typ.name, ctrl_typevar, free_typevar
             ));
         }
 
         // Other free type variables can only be used once each.
         for other_tv in &other_typevars {
-            if &tv == other_tv {
+            if &free_typevar == other_tv {
                 return Err(format!(
-                    "type variable {} can't be used more than once",
-                    tv.name
+                    "non-controlling type variable {} can't be used more than once",
+                    free_typevar.name
                 ));
             }
         }
 
-        other_typevars.push(tv);
+        other_typevars.push(free_typevar);
     }
 
     // Check outputs.
@@ -587,10 +579,10 @@ fn verify_ctrl_typevar(
         }
 
         let typ = result.type_var().unwrap();
-        let tv = typ.free_typevar();
+        let free_typevar = typ.free_typevar();
 
-        // Non-polymorphic or derived form ctrl_typevar is OK.
-        if tv.is_none() || &tv.unwrap() == ctrl_typevar {
+        // Non-polymorphic or derived from ctrl_typevar is OK.
+        if free_typevar.is_none() || &free_typevar.unwrap() == ctrl_typevar {
             continue;
         }
 

From 1c28d43f00fa99628bdbed74da100a62624bfca0 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 3 Sep 2019 18:23:21 +0200
Subject: [PATCH 2639/3084] [meta] Remove unused immfield index in the
 FormatField;

---
 cranelift/codegen/meta/src/cdsl/formats.rs | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
index a0dcbb67ae..cf82da180a 100644
--- a/cranelift/codegen/meta/src/cdsl/formats.rs
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -12,9 +12,6 @@ use cranelift_entity::{entity_impl, PrimaryMap};
 /// data type.
 #[derive(Debug)]
 pub struct FormatField {
-    /// Immediate operand number in parent.
-    immnum: usize,
-
     /// Immediate operand kind.
     pub kind: OperandKind,
 
@@ -53,7 +50,7 @@ pub struct InstructionFormat {
 
 impl fmt::Display for InstructionFormat {
     fn fmt(&self, fmt: &mut fmt::Formatter) -> Result<(), fmt::Error> {
-        let args = self
+        let imm_args = self
             .imm_fields
             .iter()
             .map(|field| format!("{}: {}", field.member, field.kind.name))
@@ -61,7 +58,7 @@ impl fmt::Display for InstructionFormat {
             .join(", ");
         fmt.write_fmt(format_args!(
             "{}(imms=({}), vals={})",
-            self.name, args, self.num_value_operands
+            self.name, imm_args, self.num_value_operands
         ))?;
         Ok(())
     }
@@ -112,7 +109,6 @@ impl InstructionFormatBuilder {
 
     pub fn imm(mut self, operand_kind: &OperandKind) -> Self {
         let field = FormatField {
-            immnum: self.imm_fields.len(),
             kind: operand_kind.clone(),
             member: operand_kind.default_member.unwrap(),
         };
@@ -122,7 +118,6 @@ impl InstructionFormatBuilder {
 
     pub fn imm_with_name(mut self, member: &'static str, operand_kind: &OperandKind) -> Self {
         let field = FormatField {
-            immnum: self.imm_fields.len(),
             kind: operand_kind.clone(),
             member,
         };

From dce8ad82294263e21ea6d91b9f98d13f3ad24cb4 Mon Sep 17 00:00:00 2001
From: Ujjwal Sharma 
Date: Sun, 1 Sep 2019 04:01:45 +0530
Subject: [PATCH 2640/3084] [codegen] add encodings for isub borrow variants

Add encodings for isub borrow variants (isub_bout, isub_bin,
isub_borrow) for x86_32, enabling the legalization for isub.i64 to work.

Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1576675
Bug: https://github.com/CraneStation/cranelift/issues/765
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 15 +++++++++++----
 cranelift/codegen/meta/src/isa/x86/recipes.rs |  4 ++--
 .../filetests/filetests/isa/x86/binary32.clif |  8 ++++++++
 .../filetests/isa/x86/legalize-i64.clif       | 19 +++++++++++++++----
 4 files changed, 36 insertions(+), 10 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 9c7ff89615..072cbb1b2a 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -397,6 +397,9 @@ pub fn define(
     let istore8 = shared.by_name("istore8");
     let istore8_complex = shared.by_name("istore8_complex");
     let isub = shared.by_name("isub");
+    let isub_bout = shared.by_name("isub_bout");
+    let isub_bin = shared.by_name("isub_bin");
+    let isub_borrow = shared.by_name("isub_borrow");
     let jump = shared.by_name("jump");
     let jump_table_base = shared.by_name("jump_table_base");
     let jump_table_entry = shared.by_name("jump_table_entry");
@@ -560,8 +563,8 @@ pub fn define(
     let rec_rfurm = r.template("rfurm");
     let rec_rmov = r.template("rmov");
     let rec_rr = r.template("rr");
-    let rec_rcin = r.template("rcin");
-    let rec_rcarry = r.template("rcarry");
+    let rec_rin = r.template("rin");
+    let rec_rio = r.template("rio");
     let rec_rrx = r.template("rrx");
     let rec_safepoint = r.recipe("safepoint");
     let rec_setf_abcd = r.template("setf_abcd");
@@ -618,10 +621,14 @@ pub fn define(
 
     e.enc_i32_i64(iadd, rec_rr.opcodes(vec![0x01]));
     e.enc_i32_i64(iadd_cout, rec_rr.opcodes(vec![0x01]));
-    e.enc_i32_i64(iadd_cin, rec_rcin.opcodes(vec![0x11]));
-    e.enc_i32_i64(iadd_carry, rec_rcarry.opcodes(vec![0x11]));
+    e.enc_i32_i64(iadd_cin, rec_rin.opcodes(vec![0x11]));
+    e.enc_i32_i64(iadd_carry, rec_rio.opcodes(vec![0x11]));
 
     e.enc_i32_i64(isub, rec_rr.opcodes(vec![0x29]));
+    e.enc_i32_i64(isub_bout, rec_rr.opcodes(vec![0x29]));
+    e.enc_i32_i64(isub_bin, rec_rin.opcodes(vec![0x19]));
+    e.enc_i32_i64(isub_borrow, rec_rio.opcodes(vec![0x19]));
+
     e.enc_i32_i64(band, rec_rr.opcodes(vec![0x21]));
     e.enc_i32_i64(bor, rec_rr.opcodes(vec![0x09]));
     e.enc_i32_i64(bxor, rec_rr.opcodes(vec![0x31]));
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 6da640a172..579da62bdd 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -2532,7 +2532,7 @@ pub fn define<'shared>(
 
     // XX /r, MR form. Add two GPR registers and get carry flag.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rcin", f_ternary, 1)
+        EncodingRecipeBuilder::new("rin", f_ternary, 1)
             .operands_in(vec![
                 OperandConstraint::RegClass(gpr),
                 OperandConstraint::RegClass(gpr),
@@ -2550,7 +2550,7 @@ pub fn define<'shared>(
 
     // XX /r, MR form. Add two GPR registers with carry flag.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rcarry", f_ternary, 1)
+        EncodingRecipeBuilder::new("rio", f_ternary, 1)
             .operands_in(vec![
                 OperandConstraint::RegClass(gpr),
                 OperandConstraint::RegClass(gpr),
diff --git a/cranelift/filetests/filetests/isa/x86/binary32.clif b/cranelift/filetests/filetests/isa/x86/binary32.clif
index 79680a939d..6f2ddaa84e 100644
--- a/cranelift/filetests/filetests/isa/x86/binary32.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary32.clif
@@ -477,6 +477,14 @@ ebb0:
     ; asm: adcl %esi, %ecx
     [-,%rcx,%rflags] v704, v705 = iadd_carry v1, v2, v702 ; bin: 11 f1
 
+    ; Borrow Subtraction
+    ; asm: subl %esi, %ecx
+    [-,%rcx,%rflags] v706, v707 = isub_bout v1, v2         ; bin: 29 f1
+    ; asm: sbbl %esi, %ecx
+    [-,%rcx] v708 = isub_bin v1, v2, v707                  ; bin: 19 f1
+    ; asm: sbbl %esi, %ecx
+    [-,%rcx,%rflags] v709, v710 = isub_borrow v1, v2, v707 ; bin: 19 f1
+
     ; asm: testl %ecx, %ecx
     ; asm: je ebb1
     brz v1, ebb1                                ; bin: 85 c9 74 0e
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-i64.clif b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
index dc7d5696fe..66cdbd245d 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
@@ -8,9 +8,20 @@ function %iadd(i64, i64) -> i64 {
 ebb0(v1: i64, v2: i64):
     v10 = iadd v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
-    ; check: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
-    ; check: $(v10_lsb=$V), $(carry=$V) = iadd_cout $v1_lsb, $v2_lsb
-    ; check: $(v10_msb=$V) = iadd_cin $v1_msb, $v2_msb, $carry
-    ; check: v10 = iconcat $v10_lsb, $v10_msb
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(v10_lsb=$V), $(carry=$V) = iadd_cout $v1_lsb, $v2_lsb
+    ; nextln: $(v10_msb=$V) = iadd_cin $v1_msb, $v2_msb, $carry
+    ; nextln: v10 = iconcat $v10_lsb, $v10_msb
+    return v10
+}
+
+function %isub(i64, i64) -> i64 {
+ebb0(v1: i64, v2: i64):
+    v10 = isub v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(v10_lsb=$V), $(borrow=$V) = isub_bout $v1_lsb, $v2_lsb
+    ; nextln: $(v10_msb=$V) = isub_bin $v1_msb, $v2_msb, $borrow
+    ; nextln: v10 = iconcat $v10_lsb, $v10_msb
     return v10
 }

From 9fb8bdd6d51ef8d1b9acee2528c7ee2a64d35038 Mon Sep 17 00:00:00 2001
From: Ujjwal Sharma 
Date: Wed, 4 Sep 2019 23:51:30 +0530
Subject: [PATCH 2641/3084] [codegen] remove support for isub borrow variants
 on riscv

Previously, the borrow variants of isub (isub_bin, isub_bout and
isub_borrow) were being legalized for isa/riscv since RISC architectures
lack a flags register.

This forced us to return and accept booleans for these operations, which
proved to be problematic and inconvenient, especially for x86.

This commit removes support for said statements and all dependent
statements for isa/riscv so that we can work on a better legalization
strategy in the future.
---
 cranelift/codegen/meta/src/shared/legalize.rs | 30 -------------------
 1 file changed, 30 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 05c74fdc47..709dab7962 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -86,7 +86,6 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     let istore16 = insts.by_name("istore16");
     let isub = insts.by_name("isub");
     let isub_bin = insts.by_name("isub_bin");
-    let isub_borrow = insts.by_name("isub_borrow");
     let isub_bout = insts.by_name("isub_bout");
     let load = insts.by_name("load");
     let popcnt = insts.by_name("popcnt");
@@ -160,8 +159,6 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     let b2 = var("b2");
     let b3 = var("b3");
     let b4 = var("b4");
-    let b_in = var("b_in");
-    let b_int = var("b_int");
     let c = var("c");
     let c1 = var("c1");
     let c2 = var("c2");
@@ -459,33 +456,6 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
         }
     }
 
-    // Expand integer operations with carry for RISC architectures that don't have
-    // the flags.
-    let intcc_ugt = Literal::enumerator_for(intcc, "ugt");
-    expand.legalize(
-        def!((a, b) = isub_bout(x, y)),
-        vec![def!(a = isub(x, y)), def!(b = icmp(intcc_ugt, a, x))],
-    );
-
-    expand.legalize(
-        def!(a = isub_bin(x, y, b)),
-        vec![
-            def!(a1 = isub(x, y)),
-            def!(b_int = bint(b)),
-            def!(a = isub(a1, b_int)),
-        ],
-    );
-
-    expand.legalize(
-        def!((a, b) = isub_borrow(x, y, b_in)),
-        vec![
-            def!((a1, b1) = isub_bout(x, y)),
-            def!(b_int = bint(b_in)),
-            def!((a, b2) = isub_bout(a1, b_int)),
-            def!(b = bor(b1, b2)),
-        ],
-    );
-
     // Expansions for fcvt_from_{u,s}int for smaller integer types.
     // These use expand and not widen because the controlling type variable for
     // these instructions are f32/f64, which are legalized as part of the expand

From 1a099f2e8c5c0c4ed0ff8d5887d894ef0b89b203 Mon Sep 17 00:00:00 2001
From: Ujjwal Sharma 
Date: Wed, 4 Sep 2019 23:58:31 +0530
Subject: [PATCH 2642/3084] [codegen] change operand type from bool to iflag
 for isub borrow variants

The type of the borrow operands for the borrow variants of the isub
instruction (isub_bin, isub_bout, isub_borrow) was bool for compatibility
reasons for isa/riscv. Since support for these instructions on RISC
architectures has been temporarily suspended, we can safely change the
type to iflags.
---
 cranelift/codegen/meta/src/shared/instructions.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 02819be42b..85c39d71b5 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1866,8 +1866,8 @@ pub fn define(
     let y = &operand("y", iB);
     let c_in = &operand_doc("c_in", iflags, "Input carry flag");
     let c_out = &operand_doc("c_out", iflags, "Output carry flag");
-    let b_in = &operand_doc("b_in", b1, "Input borrow flag");
-    let b_out = &operand_doc("b_out", b1, "Output borrow flag");
+    let b_in = &operand_doc("b_in", iflags, "Input borrow flag");
+    let b_out = &operand_doc("b_out", iflags, "Output borrow flag");
 
     ig.push(
         Inst::new(

From d858ebb4aad5d623e35f97b8043ea7503b99f75a Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 5 Sep 2019 17:36:04 -0700
Subject: [PATCH 2643/3084] Bump version to 0.42.0

---
 cranelift/Cargo.toml              | 28 ++++++++++++++--------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    | 10 +++++-----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 17 files changed, 61 insertions(+), 61 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index d714b67ad1..df8956cba9 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.41.0"
+version = "0.42.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,19 +19,19 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.41.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.41.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.41.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.41.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.41.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.41.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.41.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.41.0" }
-cranelift-module = { path = "cranelift-module", version = "0.41.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.41.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.41.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.41.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.41.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.42.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.42.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.42.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.42.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.42.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.42.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.42.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.42.0" }
+cranelift-module = { path = "cranelift-module", version = "0.42.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.42.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.42.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.42.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.42.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 3775c66c79..fee6d19b2e 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.41.0"
+version = "0.42.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.41.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.42.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index bd25b25ce3..1ea0c59e07 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.41.0"
+version = "0.42.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.41.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.41.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.42.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.42.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -27,7 +27,7 @@ serde = { version = "1.0.94", features = ["derive"], optional = true }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.41.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.42.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 4e5834d0d0..c623493e30 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.41.0"
+version = "0.42.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.41.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.42.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 3aecf29d9b..22b34d33d7 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.41.0"
+version = "0.42.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 26f497d3bd..71feb0c0de 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.41.0"
+version = "0.42.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.41.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.42.0" }
 faerie = "0.11.0"
 goblin = "0.0.24"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 6873cc17df..b8029c17e1 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.41.0"
+version = "0.42.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.41.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.41.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.41.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.42.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.42.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.42.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 32348e6311..c3e6d64a4f 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.41.0"
+version = "0.42.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
 target-lexicon = "0.8.0"
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 0ff8046fe8..15805b7878 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.41.0"
+version = "0.42.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.41.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.42.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 5023efaf5d..5c2adb2202 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.41.0"
+version = "0.42.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
 target-lexicon = "0.8.0"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index fc87ad6997..9b067ffa41 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.41.0"
+version = "0.42.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.41.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.42.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index a8f5425e35..ce7ab1997e 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.41.0"
+version="0.42.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 2678a91395..3c4ff264b1 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.41.0"
+version = "0.42.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0" }
 target-lexicon = "0.8.0"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 01d6de4900..a40ac22c1a 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.41.0"
+version = "0.42.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.41.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.42.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 3210717c7d..70c196ae1e 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.41.0"
+version = "0.42.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.41.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.41.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.42.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.42.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.41.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.41.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.41.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.42.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.42.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.42.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 3004135f4c..2967788619 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.41.0"
+version = "0.42.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.41.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.42.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 18f9b6f4a6..5621eab84f 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.41.0"
+version = "0.42.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.37.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.41.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.41.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.41.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.42.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.42.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From bafd79330dc8b473fa10dc769710028b23efb548 Mon Sep 17 00:00:00 2001
From: stefson 
Date: Fri, 6 Sep 2019 12:16:22 +0200
Subject: [PATCH 2644/3084] upgrade to latest target-lexicon 0.8.1

this fixes the compile for arm and aarch64

reported via https://github.com/CraneStation/cranelift/issues/977
---
 cranelift/Cargo.toml           | 2 +-
 cranelift/codegen/Cargo.toml   | 2 +-
 cranelift/faerie/Cargo.toml    | 2 +-
 cranelift/frontend/Cargo.toml  | 2 +-
 cranelift/fuzz/Cargo.toml      | 2 +-
 cranelift/native/Cargo.toml    | 2 +-
 cranelift/reader/Cargo.toml    | 2 +-
 cranelift/simplejit/Cargo.toml | 2 +-
 cranelift/wasm/Cargo.toml      | 2 +-
 9 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index df8956cba9..8fee3b6dbe 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -38,7 +38,7 @@ serde = "1.0.8"
 term = "0.6.1"
 capstone = { version = "0.6.0", optional = true }
 wabt = { version = "0.9.1", optional = true }
-target-lexicon = "0.8.0"
+target-lexicon = "0.8.1"
 pretty_env_logger = "0.3.0"
 file-per-thread-logger = "0.1.2"
 indicatif = "0.11.0"
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 1ea0c59e07..2c5ccb4d2e 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -18,7 +18,7 @@ cranelift-bforest = { path = "../cranelift-bforest", version = "0.42.0", default
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
-target-lexicon = "0.8.0"
+target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 71feb0c0de..2f8f2258f0 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -15,7 +15,7 @@ cranelift-module = { path = "../cranelift-module", version = "0.42.0" }
 faerie = "0.11.0"
 goblin = "0.0.24"
 failure = "0.1.2"
-target-lexicon = "0.8.0"
+target-lexicon = "0.8.1"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index c3e6d64a4f..3ef5a45641 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -12,7 +12,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
-target-lexicon = "0.8.0"
+target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 
diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
index adde8dce3a..f164491ef8 100644
--- a/cranelift/fuzz/Cargo.toml
+++ b/cranelift/fuzz/Cargo.toml
@@ -15,7 +15,7 @@ libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" }
 cranelift-codegen = { path = "../cranelift-codegen" }
 cranelift-wasm = { path = "../cranelift-wasm" }
 cranelift-reader = { path = "../cranelift-reader" }
-target-lexicon = "0.8.0"
+target-lexicon = "0.8.1"
 
 # Prevent this from interfering with workspaces
 [workspace]
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 5c2adb2202..b677ddae12 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -11,7 +11,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
-target-lexicon = "0.8.0"
+target-lexicon = "0.8.1"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
 raw-cpuid = "6.0.0"
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 3c4ff264b1..2a7216e8f7 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -11,7 +11,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0" }
-target-lexicon = "0.8.0"
+target-lexicon = "0.8.1"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 70c196ae1e..706a39f384 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -16,7 +16,7 @@ cranelift-native = { path = "../cranelift-native", version = "0.42.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
-target-lexicon = "0.8.0"
+target-lexicon = "0.8.1"
 memmap = { version = "0.7.0", optional = true } 
 
 [target.'cfg(target_os = "windows")'.dependencies]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 5621eab84f..c140de53a7 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -23,7 +23,7 @@ serde = { version = "1.0.94", features = ["derive"], optional = true }
 
 [dev-dependencies]
 wabt = "0.9.1"
-target-lexicon = "0.8.0"
+target-lexicon = "0.8.1"
 
 [features]
 default = ["std"]

From 29e3ec51c15f0999cae76a05785e6e7ce25245a2 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 4 Sep 2019 16:26:55 +0200
Subject: [PATCH 2645/3084] [meta] Introduce the Immediates structure instead
 of using dynamic lookup;

---
 .../codegen/meta/src/isa/riscv/encodings.rs   |   6 +-
 .../codegen/meta/src/isa/x86/instructions.rs  |   7 +-
 .../codegen/meta/src/isa/x86/legalize.rs      |  73 ++---
 cranelift/codegen/meta/src/isa/x86/mod.rs     |   1 +
 cranelift/codegen/meta/src/isa/x86/recipes.rs |   2 +-
 cranelift/codegen/meta/src/shared/formats.rs  | 131 ++++----
 .../codegen/meta/src/shared/immediates.rs     | 296 ++++++++++--------
 .../codegen/meta/src/shared/instructions.rs   |  81 +++--
 cranelift/codegen/meta/src/shared/legalize.rs | 111 ++++---
 cranelift/codegen/meta/src/shared/mod.rs      |  24 +-
 10 files changed, 369 insertions(+), 363 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/riscv/encodings.rs b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
index a5af1214aa..af5e471a53 100644
--- a/cranelift/codegen/meta/src/isa/riscv/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
@@ -230,8 +230,7 @@ pub fn define<'defs>(
          -> InstructionPredicateNode {
             let x = var_pool.create("x");
             let y = var_pool.create("y");
-            let cc =
-                Literal::enumerator_for(shared_defs.operand_kinds.by_name("intcc"), intcc_field);
+            let cc = Literal::enumerator_for(&shared_defs.imm.intcc, intcc_field);
             Apply::new(
                 bound_inst.clone().into(),
                 vec![Expr::Literal(cc), Expr::Var(x), Expr::Var(y)],
@@ -313,8 +312,7 @@ pub fn define<'defs>(
             let y = var_pool.create("y");
             let dest = var_pool.create("dest");
             let args = var_pool.create("args");
-            let cc =
-                Literal::enumerator_for(shared_defs.operand_kinds.by_name("intcc"), intcc_field);
+            let cc = Literal::enumerator_for(&shared_defs.imm.intcc, intcc_field);
             Apply::new(
                 bound_inst.clone().into(),
                 vec![
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 6464261938..742b632642 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -7,11 +7,13 @@ use crate::cdsl::instructions::{
 use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
 use crate::cdsl::types::ValueType;
 use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
-use crate::shared::{immediates, types, OperandKinds};
+use crate::shared::immediates::Immediates;
+use crate::shared::types;
 
 pub fn define(
     mut all_instructions: &mut AllInstructions,
     format_registry: &FormatRegistry,
+    immediates: &Immediates,
 ) -> InstructionGroup {
     let mut ig = InstructionGroupBuilder::new(
         "x86",
@@ -249,8 +251,7 @@ pub fn define(
         .operands_out(vec![y, rflags]),
     );
 
-    let immediates = OperandKinds::from(immediates::define());
-    let uimm8 = immediates.by_name("uimm8");
+    let uimm8 = &immediates.uimm8;
     let TxN = &TypeVar::new(
         "TxN",
         "A SIMD vector type",
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index a64f631168..158b05e4f1 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -58,12 +58,7 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
     let x86_umulx = x86_instructions.by_name("x86_umulx");
     let x86_smulx = x86_instructions.by_name("x86_smulx");
 
-    // List of immediates.
-    let floatcc = shared.operand_kinds.by_name("floatcc");
-    let imm64 = shared.operand_kinds.by_name("imm64");
-    let intcc = shared.operand_kinds.by_name("intcc");
-    let uimm8 = shared.operand_kinds.by_name("uimm8");
-    let ieee64 = shared.operand_kinds.by_name("ieee64");
+    let imm = &shared.imm;
 
     // Division and remainder.
     //
@@ -99,12 +94,12 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
     // `ucomiss` or `ucomisd` instruction. The remaining codes need legalization
     // patterns.
 
-    let floatcc_eq = Literal::enumerator_for(floatcc, "eq");
-    let floatcc_ord = Literal::enumerator_for(floatcc, "ord");
-    let floatcc_ueq = Literal::enumerator_for(floatcc, "ueq");
-    let floatcc_ne = Literal::enumerator_for(floatcc, "ne");
-    let floatcc_uno = Literal::enumerator_for(floatcc, "uno");
-    let floatcc_one = Literal::enumerator_for(floatcc, "one");
+    let floatcc_eq = Literal::enumerator_for(&imm.floatcc, "eq");
+    let floatcc_ord = Literal::enumerator_for(&imm.floatcc, "ord");
+    let floatcc_ueq = Literal::enumerator_for(&imm.floatcc, "ueq");
+    let floatcc_ne = Literal::enumerator_for(&imm.floatcc, "ne");
+    let floatcc_uno = Literal::enumerator_for(&imm.floatcc, "uno");
+    let floatcc_one = Literal::enumerator_for(&imm.floatcc, "one");
 
     // Equality needs an explicit `ord` test which checks the parity bit.
     group.legalize(
@@ -124,14 +119,14 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
         ],
     );
 
-    let floatcc_lt = &Literal::enumerator_for(floatcc, "lt");
-    let floatcc_gt = &Literal::enumerator_for(floatcc, "gt");
-    let floatcc_le = &Literal::enumerator_for(floatcc, "le");
-    let floatcc_ge = &Literal::enumerator_for(floatcc, "ge");
-    let floatcc_ugt = &Literal::enumerator_for(floatcc, "ugt");
-    let floatcc_ult = &Literal::enumerator_for(floatcc, "ult");
-    let floatcc_uge = &Literal::enumerator_for(floatcc, "uge");
-    let floatcc_ule = &Literal::enumerator_for(floatcc, "ule");
+    let floatcc_lt = &Literal::enumerator_for(&imm.floatcc, "lt");
+    let floatcc_gt = &Literal::enumerator_for(&imm.floatcc, "gt");
+    let floatcc_le = &Literal::enumerator_for(&imm.floatcc, "le");
+    let floatcc_ge = &Literal::enumerator_for(&imm.floatcc, "ge");
+    let floatcc_ugt = &Literal::enumerator_for(&imm.floatcc, "ugt");
+    let floatcc_ult = &Literal::enumerator_for(&imm.floatcc, "ult");
+    let floatcc_uge = &Literal::enumerator_for(&imm.floatcc, "uge");
+    let floatcc_ule = &Literal::enumerator_for(&imm.floatcc, "ule");
 
     // Inequalities that need to be reversed.
     for &(cc, rev_cc) in &[
@@ -165,9 +160,9 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
     let r2flags = var("r2flags");
     let index2 = var("index2");
 
-    let intcc_eq = Literal::enumerator_for(intcc, "eq");
-    let imm64_minus_one = Literal::constant(imm64, -1);
-    let imm64_63 = Literal::constant(imm64, 63);
+    let intcc_eq = Literal::enumerator_for(&imm.intcc, "eq");
+    let imm64_minus_one = Literal::constant(&imm.imm64, -1);
+    let imm64_63 = Literal::constant(&imm.imm64, 63);
     group.legalize(
         def!(a = clz.I64(x)),
         vec![
@@ -179,7 +174,7 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
         ],
     );
 
-    let imm64_31 = Literal::constant(imm64, 31);
+    let imm64_31 = Literal::constant(&imm.imm64, 31);
     group.legalize(
         def!(a = clz.I32(x)),
         vec![
@@ -191,7 +186,7 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
         ],
     );
 
-    let imm64_64 = Literal::constant(imm64, 64);
+    let imm64_64 = Literal::constant(&imm.imm64, 64);
     group.legalize(
         def!(a = ctz.I64(x)),
         vec![
@@ -201,7 +196,7 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
         ],
     );
 
-    let imm64_32 = Literal::constant(imm64, 32);
+    let imm64_32 = Literal::constant(&imm.imm64, 32);
     group.legalize(
         def!(a = ctz.I32(x)),
         vec![
@@ -232,13 +227,13 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
     let qc0F = var("qc0F");
     let qc01 = var("qc01");
 
-    let imm64_1 = Literal::constant(imm64, 1);
-    let imm64_4 = Literal::constant(imm64, 4);
+    let imm64_1 = Literal::constant(&imm.imm64, 1);
+    let imm64_4 = Literal::constant(&imm.imm64, 4);
     group.legalize(
         def!(qv16 = popcnt.I64(qv1)),
         vec![
             def!(qv3 = ushr_imm(qv1, imm64_1)),
-            def!(qc77 = iconst(Literal::constant(imm64, 0x7777777777777777))),
+            def!(qc77 = iconst(Literal::constant(&imm.imm64, 0x7777777777777777))),
             def!(qv4 = band(qv3, qc77)),
             def!(qv5 = isub(qv1, qv4)),
             def!(qv6 = ushr_imm(qv4, imm64_1)),
@@ -249,11 +244,11 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
             def!(qv11 = isub(qv8, qv10)),
             def!(qv12 = ushr_imm(qv11, imm64_4)),
             def!(qv13 = iadd(qv11, qv12)),
-            def!(qc0F = iconst(Literal::constant(imm64, 0x0F0F0F0F0F0F0F0F))),
+            def!(qc0F = iconst(Literal::constant(&imm.imm64, 0x0F0F0F0F0F0F0F0F))),
             def!(qv14 = band(qv13, qc0F)),
-            def!(qc01 = iconst(Literal::constant(imm64, 0x0101010101010101))),
+            def!(qc01 = iconst(Literal::constant(&imm.imm64, 0x0101010101010101))),
             def!(qv15 = imul(qv14, qc01)),
-            def!(qv16 = ushr_imm(qv15, Literal::constant(imm64, 56))),
+            def!(qv16 = ushr_imm(qv15, Literal::constant(&imm.imm64, 56))),
         ],
     );
 
@@ -281,7 +276,7 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
         def!(lv16 = popcnt.I32(lv1)),
         vec![
             def!(lv3 = ushr_imm(lv1, imm64_1)),
-            def!(lc77 = iconst(Literal::constant(imm64, 0x77777777))),
+            def!(lc77 = iconst(Literal::constant(&imm.imm64, 0x77777777))),
             def!(lv4 = band(lv3, lc77)),
             def!(lv5 = isub(lv1, lv4)),
             def!(lv6 = ushr_imm(lv4, imm64_1)),
@@ -292,11 +287,11 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
             def!(lv11 = isub(lv8, lv10)),
             def!(lv12 = ushr_imm(lv11, imm64_4)),
             def!(lv13 = iadd(lv11, lv12)),
-            def!(lc0F = iconst(Literal::constant(imm64, 0x0F0F0F0F))),
+            def!(lc0F = iconst(Literal::constant(&imm.imm64, 0x0F0F0F0F))),
             def!(lv14 = band(lv13, lc0F)),
-            def!(lc01 = iconst(Literal::constant(imm64, 0x01010101))),
+            def!(lc01 = iconst(Literal::constant(&imm.imm64, 0x01010101))),
             def!(lv15 = imul(lv14, lc01)),
-            def!(lv16 = ushr_imm(lv15, Literal::constant(imm64, 24))),
+            def!(lv16 = ushr_imm(lv15, Literal::constant(&imm.imm64, 24))),
         ],
     );
 
@@ -313,9 +308,9 @@ pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGrou
     .chain_with(shared.transform_groups.by_name("narrow").id);
 
     // SIMD
-    let uimm8_zero = Literal::constant(uimm8, 0x00);
-    let uimm8_one = Literal::constant(uimm8, 0x01);
-    let ieee64_zero = Literal::constant(ieee64, 0x00);
+    let uimm8_zero = Literal::constant(&imm.uimm8, 0x00);
+    let uimm8_one = Literal::constant(&imm.uimm8, 0x01);
+    let ieee64_zero = Literal::constant(&imm.ieee64, 0x00);
     let b = var("b");
     let c = var("c");
     let d = var("d");
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index 9f4a4a334a..5122c7b2ad 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -20,6 +20,7 @@ pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let inst_group = instructions::define(
         &mut shared_defs.all_instructions,
         &shared_defs.format_registry,
+        &shared_defs.imm,
     );
     legalize::define(shared_defs, &inst_group);
 
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 579da62bdd..8eb0fdac6d 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -332,7 +332,7 @@ pub fn define<'shared>(
 ) -> RecipeGroup<'shared> {
     // The set of floating point condition codes that are directly supported.
     // Other condition codes need to be reversed or expressed as two tests.
-    let floatcc = shared_defs.operand_kinds.by_name("floatcc");
+    let floatcc = &shared_defs.imm.floatcc;
     let supported_floatccs: Vec = ["ord", "uno", "one", "ueq", "gt", "ge", "ult", "ule"]
         .iter()
         .map(|name| Literal::enumerator_for(floatcc, name))
diff --git a/cranelift/codegen/meta/src/shared/formats.rs b/cranelift/codegen/meta/src/shared/formats.rs
index 2a16f250a1..2f79fdbdf1 100644
--- a/cranelift/codegen/meta/src/shared/formats.rs
+++ b/cranelift/codegen/meta/src/shared/formats.rs
@@ -1,22 +1,8 @@
 use crate::cdsl::formats::{FormatRegistry, InstructionFormatBuilder as Builder};
+use crate::shared::immediates::Immediates;
 use crate::shared::OperandKinds;
 
-pub fn define(immediates: &OperandKinds, entities: &OperandKinds) -> FormatRegistry {
-    // Shorthands for immediates.
-    let uimm8 = immediates.by_name("uimm8");
-    let uimm32 = immediates.by_name("uimm32");
-    let uimm128 = immediates.by_name("uimm128");
-    let imm64 = immediates.by_name("imm64");
-    let ieee32 = immediates.by_name("ieee32");
-    let ieee64 = immediates.by_name("ieee64");
-    let boolean = immediates.by_name("boolean");
-    let intcc = immediates.by_name("intcc");
-    let floatcc = immediates.by_name("floatcc");
-    let memflags = immediates.by_name("memflags");
-    let offset32 = immediates.by_name("offset32");
-    let trapcode = immediates.by_name("trapcode");
-    let regunit = immediates.by_name("regunit");
-
+pub fn define(imm: &Immediates, entities: &OperandKinds) -> FormatRegistry {
     // Shorthands for entities.
     let global_value = entities.by_name("global_value");
     let ebb = entities.by_name("ebb");
@@ -30,15 +16,15 @@ pub fn define(immediates: &OperandKinds, entities: &OperandKinds) -> FormatRegis
     let mut registry = FormatRegistry::new();
 
     registry.insert(Builder::new("Unary").value());
-    registry.insert(Builder::new("UnaryImm").imm(imm64));
-    registry.insert(Builder::new("UnaryImm128").imm(uimm128));
-    registry.insert(Builder::new("UnaryIeee32").imm(ieee32));
-    registry.insert(Builder::new("UnaryIeee64").imm(ieee64));
-    registry.insert(Builder::new("UnaryBool").imm(boolean));
+    registry.insert(Builder::new("UnaryImm").imm(&imm.imm64));
+    registry.insert(Builder::new("UnaryImm128").imm(&imm.uimm128));
+    registry.insert(Builder::new("UnaryIeee32").imm(&imm.ieee32));
+    registry.insert(Builder::new("UnaryIeee64").imm(&imm.ieee64));
+    registry.insert(Builder::new("UnaryBool").imm(&imm.boolean));
     registry.insert(Builder::new("UnaryGlobalValue").imm(global_value));
 
     registry.insert(Builder::new("Binary").value().value());
-    registry.insert(Builder::new("BinaryImm").value().imm(imm64));
+    registry.insert(Builder::new("BinaryImm").value().imm(&imm.imm64));
 
     // The select instructions are controlled by the second VALUE operand.
     // The first VALUE operand is the controlling flag which has a derived type.
@@ -60,43 +46,59 @@ pub fn define(immediates: &OperandKinds, entities: &OperandKinds) -> FormatRegis
     registry.insert(
         Builder::new("InsertLane")
             .value()
-            .imm_with_name("lane", uimm8)
+            .imm_with_name("lane", &imm.uimm8)
             .value(),
     );
     registry.insert(
         Builder::new("ExtractLane")
             .value()
-            .imm_with_name("lane", uimm8),
+            .imm_with_name("lane", &imm.uimm8),
     );
 
-    registry.insert(Builder::new("IntCompare").imm(intcc).value().value());
-    registry.insert(Builder::new("IntCompareImm").imm(intcc).value().imm(imm64));
-    registry.insert(Builder::new("IntCond").imm(intcc).value());
+    registry.insert(Builder::new("IntCompare").imm(&imm.intcc).value().value());
+    registry.insert(
+        Builder::new("IntCompareImm")
+            .imm(&imm.intcc)
+            .value()
+            .imm(&imm.imm64),
+    );
+    registry.insert(Builder::new("IntCond").imm(&imm.intcc).value());
 
-    registry.insert(Builder::new("FloatCompare").imm(floatcc).value().value());
-    registry.insert(Builder::new("FloatCond").imm(floatcc).value());;
+    registry.insert(
+        Builder::new("FloatCompare")
+            .imm(&imm.floatcc)
+            .value()
+            .value(),
+    );
+    registry.insert(Builder::new("FloatCond").imm(&imm.floatcc).value());;
 
-    registry.insert(Builder::new("IntSelect").imm(intcc).value().value().value());
+    registry.insert(
+        Builder::new("IntSelect")
+            .imm(&imm.intcc)
+            .value()
+            .value()
+            .value(),
+    );
 
     registry.insert(Builder::new("Jump").imm(ebb).varargs());
     registry.insert(Builder::new("Branch").value().imm(ebb).varargs());
     registry.insert(
         Builder::new("BranchInt")
-            .imm(intcc)
+            .imm(&imm.intcc)
             .value()
             .imm(ebb)
             .varargs(),
     );
     registry.insert(
         Builder::new("BranchFloat")
-            .imm(floatcc)
+            .imm(&imm.floatcc)
             .value()
             .imm(ebb)
             .varargs(),
     );
     registry.insert(
         Builder::new("BranchIcmp")
-            .imm(intcc)
+            .imm(&imm.intcc)
             .value()
             .value()
             .imm(ebb)
@@ -107,7 +109,7 @@ pub fn define(immediates: &OperandKinds, entities: &OperandKinds) -> FormatRegis
         Builder::new("BranchTableEntry")
             .value()
             .value()
-            .imm(uimm8)
+            .imm(&imm.uimm8)
             .imm(jump_table),
     );
     registry.insert(Builder::new("BranchTableBase").imm(jump_table));
@@ -117,74 +119,89 @@ pub fn define(immediates: &OperandKinds, entities: &OperandKinds) -> FormatRegis
     registry.insert(Builder::new("CallIndirect").imm(sig_ref).value().varargs());
     registry.insert(Builder::new("FuncAddr").imm(func_ref));
 
-    registry.insert(Builder::new("Load").imm(memflags).value().imm(offset32));
+    registry.insert(
+        Builder::new("Load")
+            .imm(&imm.memflags)
+            .value()
+            .imm(&imm.offset32),
+    );
     registry.insert(
         Builder::new("LoadComplex")
-            .imm(memflags)
+            .imm(&imm.memflags)
             .varargs()
-            .imm(offset32),
+            .imm(&imm.offset32),
     );
     registry.insert(
         Builder::new("Store")
-            .imm(memflags)
+            .imm(&imm.memflags)
             .value()
             .value()
-            .imm(offset32),
+            .imm(&imm.offset32),
     );
     registry.insert(
         Builder::new("StoreComplex")
-            .imm(memflags)
+            .imm(&imm.memflags)
             .value()
             .varargs()
-            .imm(offset32),
+            .imm(&imm.offset32),
     );
-    registry.insert(Builder::new("StackLoad").imm(stack_slot).imm(offset32));
+    registry.insert(Builder::new("StackLoad").imm(stack_slot).imm(&imm.offset32));
     registry.insert(
         Builder::new("StackStore")
             .value()
             .imm(stack_slot)
-            .imm(offset32),
+            .imm(&imm.offset32),
     );
 
     // Accessing a WebAssembly heap.
-    registry.insert(Builder::new("HeapAddr").imm(heap).value().imm(uimm32));
+    registry.insert(Builder::new("HeapAddr").imm(heap).value().imm(&imm.uimm32));
 
     // Accessing a WebAssembly table.
-    registry.insert(Builder::new("TableAddr").imm(table).value().imm(offset32));
+    registry.insert(
+        Builder::new("TableAddr")
+            .imm(table)
+            .value()
+            .imm(&imm.offset32),
+    );
 
     registry.insert(
         Builder::new("RegMove")
             .value()
-            .imm_with_name("src", regunit)
-            .imm_with_name("dst", regunit),
+            .imm_with_name("src", &imm.regunit)
+            .imm_with_name("dst", &imm.regunit),
     );
     registry.insert(
         Builder::new("CopySpecial")
-            .imm_with_name("src", regunit)
-            .imm_with_name("dst", regunit),
+            .imm_with_name("src", &imm.regunit)
+            .imm_with_name("dst", &imm.regunit),
     );
-    registry.insert(Builder::new("CopyToSsa").imm_with_name("src", regunit));
+    registry.insert(Builder::new("CopyToSsa").imm_with_name("src", &imm.regunit));
     registry.insert(
         Builder::new("RegSpill")
             .value()
-            .imm_with_name("src", regunit)
+            .imm_with_name("src", &imm.regunit)
             .imm_with_name("dst", stack_slot),
     );
     registry.insert(
         Builder::new("RegFill")
             .value()
             .imm_with_name("src", stack_slot)
-            .imm_with_name("dst", regunit),
+            .imm_with_name("dst", &imm.regunit),
     );
 
-    registry.insert(Builder::new("Trap").imm(trapcode));
-    registry.insert(Builder::new("CondTrap").value().imm(trapcode));
-    registry.insert(Builder::new("IntCondTrap").imm(intcc).value().imm(trapcode));
+    registry.insert(Builder::new("Trap").imm(&imm.trapcode));
+    registry.insert(Builder::new("CondTrap").value().imm(&imm.trapcode));
+    registry.insert(
+        Builder::new("IntCondTrap")
+            .imm(&imm.intcc)
+            .value()
+            .imm(&imm.trapcode),
+    );
     registry.insert(
         Builder::new("FloatCondTrap")
-            .imm(floatcc)
+            .imm(&imm.floatcc)
             .value()
-            .imm(trapcode),
+            .imm(&imm.trapcode),
     );
 
     registry
diff --git a/cranelift/codegen/meta/src/shared/immediates.rs b/cranelift/codegen/meta/src/shared/immediates.rs
index 5b8baca898..b7a7f712a2 100644
--- a/cranelift/codegen/meta/src/shared/immediates.rs
+++ b/cranelift/codegen/meta/src/shared/immediates.rs
@@ -2,154 +2,174 @@ use crate::cdsl::operands::{OperandKind, OperandKindBuilder as Builder};
 
 use std::collections::HashMap;
 
-pub fn define() -> Vec {
-    let mut kinds = Vec::new();
+pub struct Immediates {
+    /// A 64-bit immediate integer operand.
+    ///
+    /// This type of immediate integer can interact with SSA values with any IntType type.
+    pub imm64: OperandKind,
 
-    // A 64-bit immediate integer operand.
-    //
-    // This type of immediate integer can interact with SSA values with any
-    // IntType type.
-    let imm64 = Builder::new_imm("imm64")
-        .doc("A 64-bit immediate integer.")
-        .build();
-    kinds.push(imm64);
+    /// An unsigned 8-bit immediate integer operand.
+    ///
+    /// This small operand is used to indicate lane indexes in SIMD vectors and immediate bit
+    /// counts on shift instructions.
+    pub uimm8: OperandKind,
 
-    // An unsigned 8-bit immediate integer operand.
-    //
-    // This small operand is used to indicate lane indexes in SIMD vectors and
-    // immediate bit counts on shift instructions.
-    let uimm8 = Builder::new_imm("uimm8")
-        .doc("An 8-bit immediate unsigned integer.")
-        .build();
-    kinds.push(uimm8);
+    /// An unsigned 32-bit immediate integer operand.
+    pub uimm32: OperandKind,
 
-    // An unsigned 32-bit immediate integer operand.
-    let uimm32 = Builder::new_imm("uimm32")
-        .doc("A 32-bit immediate unsigned integer.")
-        .build();
-    kinds.push(uimm32);
+    /// An unsigned 128-bit immediate integer operand.
+    ///
+    /// This operand is used to pass entire 128-bit vectors as immediates to instructions like
+    /// const.
+    pub uimm128: OperandKind,
 
-    // An unsigned 128-bit immediate integer operand.
-    //
-    // This operand is used to pass entire 128-bit vectors as immediates to
-    // instructions like const.
-    let uimm128 = Builder::new_imm("uimm128")
-        .doc("A 128-bit immediate unsigned integer.")
-        .rust_type("ir::Constant")
-        .build();
-    kinds.push(uimm128);
+    /// A 32-bit immediate signed offset.
+    ///
+    /// This is used to represent an immediate address offset in load/store instructions.
+    pub offset32: OperandKind,
 
-    // A 32-bit immediate signed offset.
-    //
-    // This is used to represent an immediate address offset in load/store
-    // instructions.
-    let offset32 = Builder::new_imm("offset32")
-        .doc("A 32-bit immediate signed offset.")
-        .default_member("offset")
-        .build();
-    kinds.push(offset32);
+    /// A 32-bit immediate floating point operand.
+    ///
+    /// IEEE 754-2008 binary32 interchange format.
+    pub ieee32: OperandKind,
 
-    // A 32-bit immediate floating point operand.
-    //
-    // IEEE 754-2008 binary32 interchange format.
-    let ieee32 = Builder::new_imm("ieee32")
-        .doc("A 32-bit immediate floating point number.")
-        .build();
-    kinds.push(ieee32);
+    /// A 64-bit immediate floating point operand.
+    ///
+    /// IEEE 754-2008 binary64 interchange format.
+    pub ieee64: OperandKind,
 
-    // A 64-bit immediate floating point operand.
-    //
-    // IEEE 754-2008 binary64 interchange format.
-    let ieee64 = Builder::new_imm("ieee64")
-        .doc("A 64-bit immediate floating point number.")
-        .build();
-    kinds.push(ieee64);
+    /// An immediate boolean operand.
+    ///
+    /// This type of immediate boolean can interact with SSA values with any BoolType type.
+    pub boolean: OperandKind,
 
-    // An immediate boolean operand.
-    //
-    // This type of immediate boolean can interact with SSA values with any
-    // BoolType type.
-    let boolean = Builder::new_imm("boolean")
-        .doc("An immediate boolean.")
-        .rust_type("bool")
-        .build();
-    kinds.push(boolean);
+    /// A condition code for comparing integer values.
+    ///
+    /// This enumerated operand kind is used for the `icmp` instruction and corresponds to the
+    /// condcodes::IntCC` Rust type.
+    pub intcc: OperandKind,
 
-    // A condition code for comparing integer values.
-    // This enumerated operand kind is used for the `icmp` instruction and corresponds to the
-    // condcodes::IntCC` Rust type.
-    let mut intcc_values = HashMap::new();
-    intcc_values.insert("eq", "Equal");
-    intcc_values.insert("ne", "NotEqual");
-    intcc_values.insert("sge", "SignedGreaterThanOrEqual");
-    intcc_values.insert("sgt", "SignedGreaterThan");
-    intcc_values.insert("sle", "SignedLessThanOrEqual");
-    intcc_values.insert("slt", "SignedLessThan");
-    intcc_values.insert("uge", "UnsignedGreaterThanOrEqual");
-    intcc_values.insert("ugt", "UnsignedGreaterThan");
-    intcc_values.insert("ule", "UnsignedLessThanOrEqual");
-    intcc_values.insert("ult", "UnsignedLessThan");
-    let intcc = Builder::new_enum("intcc", intcc_values)
-        .doc("An integer comparison condition code.")
-        .default_member("cond")
-        .rust_type("ir::condcodes::IntCC")
-        .build();
-    kinds.push(intcc);
+    /// A condition code for comparing floating point values.
+    ///
+    /// This enumerated operand kind is used for the `fcmp` instruction and corresponds to the
+    /// `condcodes::FloatCC` Rust type.
+    pub floatcc: OperandKind,
 
-    // A condition code for comparing floating point values.  This enumerated operand kind is used
-    // for the `fcmp` instruction and corresponds to the `condcodes::FloatCC` Rust type.
-    let mut floatcc_values = HashMap::new();
-    floatcc_values.insert("ord", "Ordered");
-    floatcc_values.insert("uno", "Unordered");
-    floatcc_values.insert("eq", "Equal");
-    floatcc_values.insert("ne", "NotEqual");
-    floatcc_values.insert("one", "OrderedNotEqual");
-    floatcc_values.insert("ueq", "UnorderedOrEqual");
-    floatcc_values.insert("lt", "LessThan");
-    floatcc_values.insert("le", "LessThanOrEqual");
-    floatcc_values.insert("gt", "GreaterThan");
-    floatcc_values.insert("ge", "GreaterThanOrEqual");
-    floatcc_values.insert("ult", "UnorderedOrLessThan");
-    floatcc_values.insert("ule", "UnorderedOrLessThanOrEqual");
-    floatcc_values.insert("ugt", "UnorderedOrGreaterThan");
-    floatcc_values.insert("uge", "UnorderedOrGreaterThanOrEqual");
-    let floatcc = Builder::new_enum("floatcc", floatcc_values)
-        .doc("A floating point comparison condition code")
-        .default_member("cond")
-        .rust_type("ir::condcodes::FloatCC")
-        .build();
-    kinds.push(floatcc);
+    /// Flags for memory operations like `load` and `store`.
+    pub memflags: OperandKind,
 
-    // Flags for memory operations like :clif:inst:`load` and :clif:inst:`store`.
-    let memflags = Builder::new_imm("memflags")
-        .doc("Memory operation flags")
-        .default_member("flags")
-        .rust_type("ir::MemFlags")
-        .build();
-    kinds.push(memflags);
+    /// A register unit in the current target ISA.
+    pub regunit: OperandKind,
 
-    // A register unit in the current target ISA.
-    let regunit = Builder::new_imm("regunit")
-        .doc("A register unit in the target ISA")
-        .rust_type("isa::RegUnit")
-        .build();
-    kinds.push(regunit);
-
-    // A trap code indicating the reason for trapping.
-    //
-    // The Rust enum type also has a `User(u16)` variant for user-provided trap
-    // codes.
-    let mut trapcode_values = HashMap::new();
-    trapcode_values.insert("stk_ovf", "StackOverflow");
-    trapcode_values.insert("heap_oob", "HeapOutOfBounds");
-    trapcode_values.insert("int_ovf", "IntegerOverflow");
-    trapcode_values.insert("int_divz", "IntegerDivisionByZero");
-    let trapcode = Builder::new_enum("trapcode", trapcode_values)
-        .doc("A trap reason code.")
-        .default_member("code")
-        .rust_type("ir::TrapCode")
-        .build();
-    kinds.push(trapcode);
-
-    return kinds;
+    /// A trap code indicating the reason for trapping.
+    ///
+    /// The Rust enum type also has a `User(u16)` variant for user-provided trap codes.
+    pub trapcode: OperandKind,
+}
+
+impl Immediates {
+    pub fn new() -> Self {
+        Self {
+            imm64: Builder::new_imm("imm64")
+                .doc("A 64-bit immediate integer.")
+                .build(),
+
+            uimm8: Builder::new_imm("uimm8")
+                .doc("An 8-bit immediate unsigned integer.")
+                .build(),
+
+            uimm32: Builder::new_imm("uimm32")
+                .doc("A 32-bit immediate unsigned integer.")
+                .build(),
+
+            uimm128: Builder::new_imm("uimm128")
+                .doc("A 128-bit immediate unsigned integer.")
+                .rust_type("ir::Constant")
+                .build(),
+
+            offset32: Builder::new_imm("offset32")
+                .doc("A 32-bit immediate signed offset.")
+                .default_member("offset")
+                .build(),
+
+            ieee32: Builder::new_imm("ieee32")
+                .doc("A 32-bit immediate floating point number.")
+                .build(),
+
+            ieee64: Builder::new_imm("ieee64")
+                .doc("A 64-bit immediate floating point number.")
+                .build(),
+
+            boolean: Builder::new_imm("boolean")
+                .doc("An immediate boolean.")
+                .rust_type("bool")
+                .build(),
+
+            intcc: {
+                let mut intcc_values = HashMap::new();
+                intcc_values.insert("eq", "Equal");
+                intcc_values.insert("ne", "NotEqual");
+                intcc_values.insert("sge", "SignedGreaterThanOrEqual");
+                intcc_values.insert("sgt", "SignedGreaterThan");
+                intcc_values.insert("sle", "SignedLessThanOrEqual");
+                intcc_values.insert("slt", "SignedLessThan");
+                intcc_values.insert("uge", "UnsignedGreaterThanOrEqual");
+                intcc_values.insert("ugt", "UnsignedGreaterThan");
+                intcc_values.insert("ule", "UnsignedLessThanOrEqual");
+                intcc_values.insert("ult", "UnsignedLessThan");
+                Builder::new_enum("intcc", intcc_values)
+                    .doc("An integer comparison condition code.")
+                    .default_member("cond")
+                    .rust_type("ir::condcodes::IntCC")
+                    .build()
+            },
+
+            floatcc: {
+                let mut floatcc_values = HashMap::new();
+                floatcc_values.insert("ord", "Ordered");
+                floatcc_values.insert("uno", "Unordered");
+                floatcc_values.insert("eq", "Equal");
+                floatcc_values.insert("ne", "NotEqual");
+                floatcc_values.insert("one", "OrderedNotEqual");
+                floatcc_values.insert("ueq", "UnorderedOrEqual");
+                floatcc_values.insert("lt", "LessThan");
+                floatcc_values.insert("le", "LessThanOrEqual");
+                floatcc_values.insert("gt", "GreaterThan");
+                floatcc_values.insert("ge", "GreaterThanOrEqual");
+                floatcc_values.insert("ult", "UnorderedOrLessThan");
+                floatcc_values.insert("ule", "UnorderedOrLessThanOrEqual");
+                floatcc_values.insert("ugt", "UnorderedOrGreaterThan");
+                floatcc_values.insert("uge", "UnorderedOrGreaterThanOrEqual");
+                Builder::new_enum("floatcc", floatcc_values)
+                    .doc("A floating point comparison condition code")
+                    .default_member("cond")
+                    .rust_type("ir::condcodes::FloatCC")
+                    .build()
+            },
+
+            memflags: Builder::new_imm("memflags")
+                .doc("Memory operation flags")
+                .default_member("flags")
+                .rust_type("ir::MemFlags")
+                .build(),
+
+            regunit: Builder::new_imm("regunit")
+                .doc("A register unit in the target ISA")
+                .rust_type("isa::RegUnit")
+                .build(),
+
+            trapcode: {
+                let mut trapcode_values = HashMap::new();
+                trapcode_values.insert("stk_ovf", "StackOverflow");
+                trapcode_values.insert("heap_oob", "HeapOutOfBounds");
+                trapcode_values.insert("int_ovf", "IntegerOverflow");
+                trapcode_values.insert("int_divz", "IntegerDivisionByZero");
+                Builder::new_enum("trapcode", trapcode_values)
+                    .doc("A trap reason code.")
+                    .default_member("code")
+                    .rust_type("ir::TrapCode")
+                    .build()
+            },
+        }
+    }
 }
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 85c39d71b5..ad94efeb80 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -8,12 +8,13 @@ use crate::cdsl::operands::{create_operand as operand, create_operand_doc as ope
 use crate::cdsl::type_inference::Constraint::WiderOrEq;
 use crate::cdsl::types::{LaneType, ValueType};
 use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
+use crate::shared::immediates::Immediates;
 use crate::shared::{types, OperandKinds};
 
 pub fn define(
     all_instructions: &mut AllInstructions,
     format_registry: &FormatRegistry,
-    immediates: &OperandKinds,
+    imm: &Immediates,
     entities: &OperandKinds,
 ) -> InstructionGroup {
     let mut ig = InstructionGroupBuilder::new(
@@ -24,20 +25,6 @@ pub fn define(
     );
 
     // Operand kind shorthands.
-    let intcc = immediates.by_name("intcc");
-    let floatcc = immediates.by_name("floatcc");
-    let trapcode = immediates.by_name("trapcode");
-    let uimm8 = immediates.by_name("uimm8");
-    let uimm32 = immediates.by_name("uimm32");
-    let imm64 = immediates.by_name("imm64");
-    let uimm128 = immediates.by_name("uimm128");
-    let offset32 = immediates.by_name("offset32");
-    let memflags = immediates.by_name("memflags");
-    let ieee32 = immediates.by_name("ieee32");
-    let ieee64 = immediates.by_name("ieee64");
-    let boolean = immediates.by_name("boolean");
-    let regunit = immediates.by_name("regunit");
-
     let ebb = entities.by_name("ebb");
     let jump_table = entities.by_name("jump_table");
     let variable_args = entities.by_name("variable_args");
@@ -142,7 +129,7 @@ pub fn define(
 
     let addr = &operand("addr", iAddr);
     let c = &operand_doc("c", Testable, "Controlling value to test");
-    let Cond = &operand("Cond", intcc);
+    let Cond = &operand("Cond", &imm.intcc);
     let x = &operand("x", iB);
     let y = &operand("y", iB);
     let EBB = &operand_doc("EBB", ebb, "Destination extended basic block");
@@ -253,7 +240,7 @@ pub fn define(
         .is_branch(true),
     );
 
-    let Cond = &operand("Cond", floatcc);
+    let Cond = &operand("Cond", &imm.floatcc);
     let f = &operand("f", fflags);
 
     ig.push(
@@ -300,7 +287,7 @@ pub fn define(
     // These are the instructions which br_table legalizes to: they perform address computations,
     // using pointer-sized integers, so their type variables are more constrained.
     let x = &operand_doc("x", iAddr, "index into jump table");
-    let Size = &operand_doc("Size", uimm8, "Size in bytes");
+    let Size = &operand_doc("Size", &imm.uimm8, "Size in bytes");
 
     ig.push(
         Inst::new(
@@ -365,7 +352,7 @@ pub fn define(
         .can_store(true),
     );
 
-    let code = &operand("code", trapcode);
+    let code = &operand("code", &imm.trapcode);
 
     ig.push(
         Inst::new(
@@ -418,7 +405,7 @@ pub fn define(
         .can_trap(true),
     );
 
-    let Cond = &operand("Cond", intcc);
+    let Cond = &operand("Cond", &imm.intcc);
     let f = &operand("f", iflags);
 
     ig.push(
@@ -432,7 +419,7 @@ pub fn define(
         .can_trap(true),
     );
 
-    let Cond = &operand("Cond", floatcc);
+    let Cond = &operand("Cond", &imm.floatcc);
     let f = &operand("f", fflags);
 
     ig.push(
@@ -539,11 +526,11 @@ pub fn define(
     );
 
     let SS = &operand("SS", stack_slot);
-    let Offset = &operand_doc("Offset", offset32, "Byte offset from base address");
+    let Offset = &operand_doc("Offset", &imm.offset32, "Byte offset from base address");
     let x = &operand_doc("x", Mem, "Value to be stored");
     let a = &operand_doc("a", Mem, "Value loaded");
     let p = &operand("p", iAddr);
-    let MemFlags = &operand("MemFlags", memflags);
+    let MemFlags = &operand("MemFlags", &imm.memflags);
     let args = &operand_doc("args", variable_args, "Address arguments");
 
     ig.push(
@@ -876,7 +863,7 @@ pub fn define(
 
     let x = &operand_doc("x", Mem, "Value to be stored");
     let a = &operand_doc("a", Mem, "Value loaded");
-    let Offset = &operand_doc("Offset", offset32, "In-bounds offset into stack slot");
+    let Offset = &operand_doc("Offset", &imm.offset32, "In-bounds offset into stack slot");
 
     ig.push(
         Inst::new(
@@ -962,7 +949,7 @@ pub fn define(
 
     let H = &operand("H", heap);
     let p = &operand("p", HeapOffset);
-    let Size = &operand_doc("Size", uimm32, "Size in bytes");
+    let Size = &operand_doc("Size", &imm.uimm32, "Size in bytes");
 
     ig.push(
         Inst::new(
@@ -990,7 +977,7 @@ pub fn define(
     );
     let T = &operand("T", table);
     let p = &operand("p", TableOffset);
-    let Offset = &operand_doc("Offset", offset32, "Byte offset from element address");
+    let Offset = &operand_doc("Offset", &imm.offset32, "Byte offset from element address");
 
     ig.push(
         Inst::new(
@@ -1013,7 +1000,7 @@ pub fn define(
         .operands_out(vec![addr]),
     );
 
-    let N = &operand("N", imm64);
+    let N = &operand("N", &imm.imm64);
     let a = &operand_doc("a", Int, "A constant integer scalar or vector value");
 
     ig.push(
@@ -1030,7 +1017,7 @@ pub fn define(
         .operands_out(vec![a]),
     );
 
-    let N = &operand("N", ieee32);
+    let N = &operand("N", &imm.ieee32);
     let a = &operand_doc("a", f32_, "A constant f32 scalar value");
 
     ig.push(
@@ -1046,7 +1033,7 @@ pub fn define(
         .operands_out(vec![a]),
     );
 
-    let N = &operand("N", ieee64);
+    let N = &operand("N", &imm.ieee64);
     let a = &operand_doc("a", f64_, "A constant f64 scalar value");
 
     ig.push(
@@ -1062,7 +1049,7 @@ pub fn define(
         .operands_out(vec![a]),
     );
 
-    let N = &operand("N", boolean);
+    let N = &operand("N", &imm.boolean);
     let a = &operand_doc("a", Bool, "A constant boolean scalar or vector value");
 
     ig.push(
@@ -1079,7 +1066,11 @@ pub fn define(
         .operands_out(vec![a]),
     );
 
-    let N = &operand_doc("N", uimm128, "The 16 immediate bytes of a 128-bit vector");
+    let N = &operand_doc(
+        "N",
+        &imm.uimm128,
+        "The 16 immediate bytes of a 128-bit vector",
+    );
     let a = &operand_doc("a", TxN, "A constant vector value");
 
     ig.push(
@@ -1137,7 +1128,7 @@ pub fn define(
         .operands_out(vec![a]),
     );
 
-    let cc = &operand_doc("cc", intcc, "Controlling condition code");
+    let cc = &operand_doc("cc", &imm.intcc, "Controlling condition code");
     let flags = &operand_doc("flags", iflags, "The machine's flag register");
 
     ig.push(
@@ -1217,8 +1208,8 @@ pub fn define(
         .can_load(true),
     );
 
-    let src = &operand("src", regunit);
-    let dst = &operand("dst", regunit);
+    let src = &operand("src", &imm.regunit);
+    let dst = &operand("dst", &imm.regunit);
 
     ig.push(
         Inst::new(
@@ -1302,7 +1293,7 @@ pub fn define(
         .other_side_effects(true),
     );
 
-    let Offset = &operand_doc("Offset", imm64, "Offset from current stack pointer");
+    let Offset = &operand_doc("Offset", &imm.imm64, "Offset from current stack pointer");
 
     ig.push(
         Inst::new(
@@ -1319,7 +1310,7 @@ pub fn define(
         .other_side_effects(true),
     );
 
-    let Offset = &operand_doc("Offset", imm64, "Offset from current stack pointer");
+    let Offset = &operand_doc("Offset", &imm.imm64, "Offset from current stack pointer");
 
     ig.push(
         Inst::new(
@@ -1498,7 +1489,7 @@ pub fn define(
 
     let x = &operand_doc("x", TxN, "SIMD vector to modify");
     let y = &operand_doc("y", &TxN.lane_of(), "New lane value");
-    let Idx = &operand_doc("Idx", uimm8, "Lane index");
+    let Idx = &operand_doc("Idx", &imm.uimm8, "Lane index");
 
     ig.push(
         Inst::new(
@@ -1532,7 +1523,7 @@ pub fn define(
     );
 
     let a = &operand("a", &Int.as_bool());
-    let Cond = &operand("Cond", intcc);
+    let Cond = &operand("Cond", &imm.intcc);
     let x = &operand("x", Int);
     let y = &operand("y", Int);
 
@@ -1566,7 +1557,7 @@ pub fn define(
 
     let a = &operand("a", b1);
     let x = &operand("x", iB);
-    let Y = &operand("Y", imm64);
+    let Y = &operand("Y", &imm.imm64);
 
     ig.push(
         Inst::new(
@@ -1757,7 +1748,7 @@ pub fn define(
 
     let a = &operand("a", iB);
     let x = &operand("x", iB);
-    let Y = &operand("Y", imm64);
+    let Y = &operand("Y", &imm.imm64);
 
     ig.push(
         Inst::new(
@@ -2092,7 +2083,7 @@ pub fn define(
     );
 
     let x = &operand("x", iB);
-    let Y = &operand("Y", imm64);
+    let Y = &operand("Y", &imm.imm64);
     let a = &operand("a", iB);
 
     ig.push(
@@ -2145,7 +2136,7 @@ pub fn define(
 
     let x = &operand_doc("x", Int, "Scalar or vector value to shift");
     let y = &operand_doc("y", iB, "Number of bits to shift");
-    let Y = &operand("Y", imm64);
+    let Y = &operand("Y", &imm.imm64);
     let a = &operand("a", Int);
 
     ig.push(
@@ -2375,7 +2366,7 @@ pub fn define(
             .simd_lanes(Interval::All)
             .build(),
     );
-    let Cond = &operand("Cond", floatcc);
+    let Cond = &operand("Cond", &imm.floatcc);
     let x = &operand("x", Float);
     let y = &operand("y", Float);
     let a = &operand("a", &Float.as_bool());
@@ -2687,7 +2678,7 @@ pub fn define(
         .operands_out(vec![a]),
     );
 
-    let Cond = &operand("Cond", intcc);
+    let Cond = &operand("Cond", &imm.intcc);
     let f = &operand("f", iflags);
     let a = &operand("a", b1);
 
@@ -2705,7 +2696,7 @@ pub fn define(
         .operands_out(vec![a]),
     );
 
-    let Cond = &operand("Cond", floatcc);
+    let Cond = &operand("Cond", &imm.floatcc);
     let f = &operand("f", fflags);
 
     ig.push(
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 709dab7962..ce2faa7720 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -2,12 +2,11 @@ use crate::cdsl::ast::{var, ExprBuilder, Literal};
 use crate::cdsl::instructions::{Instruction, InstructionGroup};
 use crate::cdsl::xform::{TransformGroupBuilder, TransformGroups};
 
-use crate::shared::OperandKinds;
-
+use crate::shared::immediates::Immediates;
 use crate::shared::types::Float::{F32, F64};
 use crate::shared::types::Int::{I16, I32, I64, I8};
 
-pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformGroups {
+pub fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGroups {
     let mut narrow = TransformGroupBuilder::new(
         "narrow",
         r#"
@@ -139,12 +138,6 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     expand.custom_legalize(insts.by_name("stack_load"), "expand_stack_load");
     expand.custom_legalize(insts.by_name("stack_store"), "expand_stack_store");
 
-    // List of immediates.
-    let imm64 = immediates.by_name("imm64");
-    let ieee32 = immediates.by_name("ieee32");
-    let ieee64 = immediates.by_name("ieee64");
-    let intcc = immediates.by_name("intcc");
-
     // List of variables to reuse in patterns.
     let x = var("x");
     let y = var("y");
@@ -298,7 +291,7 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     }
 
     for &(int_ty, num) in &[(I8, 24), (I16, 16)] {
-        let imm = Literal::constant(imm64, -num);
+        let imm = Literal::constant(&imm.imm64, -num);
 
         widen.legalize(
             def!(a = clz.int_ty(b)),
@@ -322,7 +315,7 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     }
 
     for &(int_ty, num) in &[(I8, 1 << 8), (I16, 1 << 16)] {
-        let num = Literal::constant(imm64, num);
+        let num = Literal::constant(&imm.imm64, num);
         widen.legalize(
             def!(a = ctz.int_ty(b)),
             vec![
@@ -423,7 +416,7 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
         }
 
         for cc in &["eq", "ne", "ugt", "ult", "uge", "ule"] {
-            let w_cc = Literal::enumerator_for(intcc, cc);
+            let w_cc = Literal::enumerator_for(&imm.intcc, cc);
             widen.legalize(
                 def!(a = icmp_imm.int_ty(w_cc, b, c)),
                 vec![def!(x = uextend.I32(b)), def!(a = icmp_imm(w_cc, x, c))],
@@ -439,7 +432,7 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
         }
 
         for cc in &["sgt", "slt", "sge", "sle"] {
-            let w_cc = Literal::enumerator_for(intcc, cc);
+            let w_cc = Literal::enumerator_for(&imm.intcc, cc);
             widen.legalize(
                 def!(a = icmp_imm.int_ty(w_cc, b, c)),
                 vec![def!(x = sextend.I32(b)), def!(a = icmp_imm(w_cc, x, c))],
@@ -534,7 +527,7 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     }
 
     //# Expand bnot using xor.
-    let minus_one = Literal::constant(imm64, -1);
+    let minus_one = Literal::constant(&imm.imm64, -1);
     expand.legalize(
         def!(a = bnot(x)),
         vec![def!(y = iconst(minus_one)), def!(a = bxor(x, y))],
@@ -543,82 +536,82 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     //# Expand bitrev
     //# Adapted from Stack Overflow.
     //# https://stackoverflow.com/questions/746171/most-efficient-algorithm-for-bit-reversal-from-msb-lsb-to-lsb-msb-in-c
-    let imm64_1 = Literal::constant(imm64, 1);
-    let imm64_2 = Literal::constant(imm64, 2);
-    let imm64_4 = Literal::constant(imm64, 4);
+    let imm64_1 = Literal::constant(&imm.imm64, 1);
+    let imm64_2 = Literal::constant(&imm.imm64, 2);
+    let imm64_4 = Literal::constant(&imm.imm64, 4);
 
     widen.legalize(
         def!(a = bitrev.I8(x)),
         vec![
-            def!(a1 = band_imm(x, Literal::constant(imm64, 0xaa))),
+            def!(a1 = band_imm(x, Literal::constant(&imm.imm64, 0xaa))),
             def!(a2 = ushr_imm(a1, imm64_1)),
-            def!(a3 = band_imm(x, Literal::constant(imm64, 0x55))),
+            def!(a3 = band_imm(x, Literal::constant(&imm.imm64, 0x55))),
             def!(a4 = ishl_imm(a3, imm64_1)),
             def!(b = bor(a2, a4)),
-            def!(b1 = band_imm(b, Literal::constant(imm64, 0xcc))),
+            def!(b1 = band_imm(b, Literal::constant(&imm.imm64, 0xcc))),
             def!(b2 = ushr_imm(b1, imm64_2)),
-            def!(b3 = band_imm(b, Literal::constant(imm64, 0x33))),
+            def!(b3 = band_imm(b, Literal::constant(&imm.imm64, 0x33))),
             def!(b4 = ishl_imm(b3, imm64_2)),
             def!(c = bor(b2, b4)),
-            def!(c1 = band_imm(c, Literal::constant(imm64, 0xf0))),
+            def!(c1 = band_imm(c, Literal::constant(&imm.imm64, 0xf0))),
             def!(c2 = ushr_imm(c1, imm64_4)),
-            def!(c3 = band_imm(c, Literal::constant(imm64, 0x0f))),
+            def!(c3 = band_imm(c, Literal::constant(&imm.imm64, 0x0f))),
             def!(c4 = ishl_imm(c3, imm64_4)),
             def!(a = bor(c2, c4)),
         ],
     );
 
-    let imm64_8 = Literal::constant(imm64, 8);
+    let imm64_8 = Literal::constant(&imm.imm64, 8);
 
     widen.legalize(
         def!(a = bitrev.I16(x)),
         vec![
-            def!(a1 = band_imm(x, Literal::constant(imm64, 0xaaaa))),
+            def!(a1 = band_imm(x, Literal::constant(&imm.imm64, 0xaaaa))),
             def!(a2 = ushr_imm(a1, imm64_1)),
-            def!(a3 = band_imm(x, Literal::constant(imm64, 0x5555))),
+            def!(a3 = band_imm(x, Literal::constant(&imm.imm64, 0x5555))),
             def!(a4 = ishl_imm(a3, imm64_1)),
             def!(b = bor(a2, a4)),
-            def!(b1 = band_imm(b, Literal::constant(imm64, 0xcccc))),
+            def!(b1 = band_imm(b, Literal::constant(&imm.imm64, 0xcccc))),
             def!(b2 = ushr_imm(b1, imm64_2)),
-            def!(b3 = band_imm(b, Literal::constant(imm64, 0x3333))),
+            def!(b3 = band_imm(b, Literal::constant(&imm.imm64, 0x3333))),
             def!(b4 = ishl_imm(b3, imm64_2)),
             def!(c = bor(b2, b4)),
-            def!(c1 = band_imm(c, Literal::constant(imm64, 0xf0f0))),
+            def!(c1 = band_imm(c, Literal::constant(&imm.imm64, 0xf0f0))),
             def!(c2 = ushr_imm(c1, imm64_4)),
-            def!(c3 = band_imm(c, Literal::constant(imm64, 0x0f0f))),
+            def!(c3 = band_imm(c, Literal::constant(&imm.imm64, 0x0f0f))),
             def!(c4 = ishl_imm(c3, imm64_4)),
             def!(d = bor(c2, c4)),
-            def!(d1 = band_imm(d, Literal::constant(imm64, 0xff00))),
+            def!(d1 = band_imm(d, Literal::constant(&imm.imm64, 0xff00))),
             def!(d2 = ushr_imm(d1, imm64_8)),
-            def!(d3 = band_imm(d, Literal::constant(imm64, 0x00ff))),
+            def!(d3 = band_imm(d, Literal::constant(&imm.imm64, 0x00ff))),
             def!(d4 = ishl_imm(d3, imm64_8)),
             def!(a = bor(d2, d4)),
         ],
     );
 
-    let imm64_16 = Literal::constant(imm64, 16);
+    let imm64_16 = Literal::constant(&imm.imm64, 16);
 
     expand.legalize(
         def!(a = bitrev.I32(x)),
         vec![
-            def!(a1 = band_imm(x, Literal::constant(imm64, 0xaaaaaaaa))),
+            def!(a1 = band_imm(x, Literal::constant(&imm.imm64, 0xaaaaaaaa))),
             def!(a2 = ushr_imm(a1, imm64_1)),
-            def!(a3 = band_imm(x, Literal::constant(imm64, 0x55555555))),
+            def!(a3 = band_imm(x, Literal::constant(&imm.imm64, 0x55555555))),
             def!(a4 = ishl_imm(a3, imm64_1)),
             def!(b = bor(a2, a4)),
-            def!(b1 = band_imm(b, Literal::constant(imm64, 0xcccccccc))),
+            def!(b1 = band_imm(b, Literal::constant(&imm.imm64, 0xcccccccc))),
             def!(b2 = ushr_imm(b1, imm64_2)),
-            def!(b3 = band_imm(b, Literal::constant(imm64, 0x33333333))),
+            def!(b3 = band_imm(b, Literal::constant(&imm.imm64, 0x33333333))),
             def!(b4 = ishl_imm(b3, imm64_2)),
             def!(c = bor(b2, b4)),
-            def!(c1 = band_imm(c, Literal::constant(imm64, 0xf0f0f0f0))),
+            def!(c1 = band_imm(c, Literal::constant(&imm.imm64, 0xf0f0f0f0))),
             def!(c2 = ushr_imm(c1, imm64_4)),
-            def!(c3 = band_imm(c, Literal::constant(imm64, 0x0f0f0f0f))),
+            def!(c3 = band_imm(c, Literal::constant(&imm.imm64, 0x0f0f0f0f))),
             def!(c4 = ishl_imm(c3, imm64_4)),
             def!(d = bor(c2, c4)),
-            def!(d1 = band_imm(d, Literal::constant(imm64, 0xff00ff00))),
+            def!(d1 = band_imm(d, Literal::constant(&imm.imm64, 0xff00ff00))),
             def!(d2 = ushr_imm(d1, imm64_8)),
-            def!(d3 = band_imm(d, Literal::constant(imm64, 0x00ff00ff))),
+            def!(d3 = band_imm(d, Literal::constant(&imm.imm64, 0x00ff00ff))),
             def!(d4 = ishl_imm(d3, imm64_8)),
             def!(e = bor(d2, d4)),
             def!(e1 = ushr_imm(e, imm64_16)),
@@ -628,21 +621,21 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     );
 
     #[allow(overflowing_literals)]
-    let imm64_0xaaaaaaaaaaaaaaaa = Literal::constant(imm64, 0xaaaaaaaaaaaaaaaa);
-    let imm64_0x5555555555555555 = Literal::constant(imm64, 0x5555555555555555);
+    let imm64_0xaaaaaaaaaaaaaaaa = Literal::constant(&imm.imm64, 0xaaaaaaaaaaaaaaaa);
+    let imm64_0x5555555555555555 = Literal::constant(&imm.imm64, 0x5555555555555555);
     #[allow(overflowing_literals)]
-    let imm64_0xcccccccccccccccc = Literal::constant(imm64, 0xcccccccccccccccc);
-    let imm64_0x3333333333333333 = Literal::constant(imm64, 0x3333333333333333);
+    let imm64_0xcccccccccccccccc = Literal::constant(&imm.imm64, 0xcccccccccccccccc);
+    let imm64_0x3333333333333333 = Literal::constant(&imm.imm64, 0x3333333333333333);
     #[allow(overflowing_literals)]
-    let imm64_0xf0f0f0f0f0f0f0f0 = Literal::constant(imm64, 0xf0f0f0f0f0f0f0f0);
-    let imm64_0x0f0f0f0f0f0f0f0f = Literal::constant(imm64, 0x0f0f0f0f0f0f0f0f);
+    let imm64_0xf0f0f0f0f0f0f0f0 = Literal::constant(&imm.imm64, 0xf0f0f0f0f0f0f0f0);
+    let imm64_0x0f0f0f0f0f0f0f0f = Literal::constant(&imm.imm64, 0x0f0f0f0f0f0f0f0f);
     #[allow(overflowing_literals)]
-    let imm64_0xff00ff00ff00ff00 = Literal::constant(imm64, 0xff00ff00ff00ff00);
-    let imm64_0x00ff00ff00ff00ff = Literal::constant(imm64, 0x00ff00ff00ff00ff);
+    let imm64_0xff00ff00ff00ff00 = Literal::constant(&imm.imm64, 0xff00ff00ff00ff00);
+    let imm64_0x00ff00ff00ff00ff = Literal::constant(&imm.imm64, 0x00ff00ff00ff00ff);
     #[allow(overflowing_literals)]
-    let imm64_0xffff0000ffff0000 = Literal::constant(imm64, 0xffff0000ffff0000);
-    let imm64_0x0000ffff0000ffff = Literal::constant(imm64, 0x0000ffff0000ffff);
-    let imm64_32 = Literal::constant(imm64, 32);
+    let imm64_0xffff0000ffff0000 = Literal::constant(&imm.imm64, 0xffff0000ffff0000);
+    let imm64_0x0000ffff0000ffff = Literal::constant(&imm.imm64, 0x0000ffff0000ffff);
+    let imm64_32 = Literal::constant(&imm.imm64, 32);
 
     expand.legalize(
         def!(a = bitrev.I64(x)),
@@ -680,8 +673,12 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
 
     // Floating-point sign manipulations.
     for &(ty, const_inst, minus_zero) in &[
-        (F32, f32const, &Literal::bits(ieee32, 0x80000000)),
-        (F64, f64const, &Literal::bits(ieee64, 0x8000000000000000)),
+        (F32, f32const, &Literal::bits(&imm.ieee32, 0x80000000)),
+        (
+            F64,
+            f64const,
+            &Literal::bits(&imm.ieee64, 0x8000000000000000),
+        ),
     ] {
         expand.legalize(
             def!(a = fabs.ty(x)),
@@ -724,9 +721,9 @@ pub fn define(insts: &InstructionGroup, immediates: &OperandKinds) -> TransformG
     )
     .chain_with(expand_id);
 
-    let imm64_0 = Literal::constant(imm64, 0);
-    let intcc_ne = Literal::enumerator_for(intcc, "ne");
-    let intcc_eq = Literal::enumerator_for(intcc, "eq");
+    let imm64_0 = Literal::constant(&imm.imm64, 0);
+    let intcc_ne = Literal::enumerator_for(&imm.intcc, "ne");
+    let intcc_eq = Literal::enumerator_for(&imm.intcc, "eq");
 
     expand_flags.legalize(
         def!(trapnz(x, c)),
diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
index ba1425b844..837bf3c71d 100644
--- a/cranelift/codegen/meta/src/shared/mod.rs
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -14,11 +14,13 @@ use crate::cdsl::operands::OperandKind;
 use crate::cdsl::settings::SettingGroup;
 use crate::cdsl::xform::TransformGroups;
 
+use crate::shared::immediates::Immediates;
+
 pub struct Definitions {
     pub settings: SettingGroup,
     pub all_instructions: AllInstructions,
     pub instructions: InstructionGroup,
-    pub operand_kinds: OperandKinds,
+    pub imm: Immediates,
     pub format_registry: FormatRegistry,
     pub transform_groups: TransformGroups,
 }
@@ -26,28 +28,12 @@ pub struct Definitions {
 pub struct OperandKinds(Vec);
 
 impl OperandKinds {
-    pub fn new() -> Self {
-        Self(Vec::new())
-    }
-
     pub fn by_name(&self, name: &'static str) -> &OperandKind {
         self.0
             .iter()
             .find(|op| op.name == name)
             .expect(&format!("unknown Operand name: {}", name))
     }
-
-    pub fn push(&mut self, operand_kind: OperandKind) {
-        assert!(
-            self.0
-                .iter()
-                .find(|existing| existing.name == operand_kind.name)
-                .is_none(),
-            "trying to insert operand kind '{}' for the second time",
-            operand_kind.name
-        );
-        self.0.push(operand_kind);
-    }
 }
 
 impl From> for OperandKinds {
@@ -59,7 +45,7 @@ impl From> for OperandKinds {
 pub fn define() -> Definitions {
     let mut all_instructions = AllInstructions::new();
 
-    let immediates = OperandKinds(immediates::define());
+    let immediates = Immediates::new();
     let entities = OperandKinds(entities::define());
     let format_registry = formats::define(&immediates, &entities);
     let instructions = instructions::define(
@@ -74,7 +60,7 @@ pub fn define() -> Definitions {
         settings: settings::define(),
         all_instructions,
         instructions,
-        operand_kinds: immediates,
+        imm: immediates,
         format_registry,
         transform_groups,
     }

From 8fba449b7bc6d554dba0605a259bed56c798cc91 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 4 Sep 2019 16:37:15 +0200
Subject: [PATCH 2646/3084] [meta] Introduce the EntityRefs structure instead
 of using dynamic lookup;

---
 cranelift/codegen/meta/src/shared/entities.rs | 129 ++++++++++--------
 cranelift/codegen/meta/src/shared/formats.rs  |  76 ++++++-----
 .../codegen/meta/src/shared/instructions.rs   |  46 +++----
 cranelift/codegen/meta/src/shared/mod.rs      |  23 +---
 4 files changed, 137 insertions(+), 137 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/entities.rs b/cranelift/codegen/meta/src/shared/entities.rs
index 358ef8f23e..910722789b 100644
--- a/cranelift/codegen/meta/src/shared/entities.rs
+++ b/cranelift/codegen/meta/src/shared/entities.rs
@@ -1,65 +1,76 @@
 use crate::cdsl::operands::{OperandKind, OperandKindBuilder as Builder, OperandKindFields};
 
+pub struct EntityRefs {
+    /// A reference to an extended basic block in the same function.
+    /// This is primarliy used in control flow instructions.
+    pub ebb: OperandKind,
+
+    /// A reference to a stack slot declared in the function preamble.
+    pub stack_slot: OperandKind,
+
+    /// A reference to a global value.
+    pub global_value: OperandKind,
+
+    /// A reference to a function signature declared in the function preamble.
+    /// This is used to provide the call signature in a call_indirect instruction.
+    pub sig_ref: OperandKind,
+
+    /// A reference to an external function declared in the function preamble.
+    /// This is used to provide the callee and signature in a call instruction.
+    pub func_ref: OperandKind,
+
+    /// A reference to a jump table declared in the function preamble.
+    pub jump_table: OperandKind,
+
+    /// A reference to a heap declared in the function preamble.
+    pub heap: OperandKind,
+
+    /// A reference to a table declared in the function preamble.
+    pub table: OperandKind,
+
+    /// A variable-sized list of value operands. Use for Ebb and function call arguments.
+    pub varargs: OperandKind,
+}
+
+impl EntityRefs {
+    pub fn new() -> Self {
+        Self {
+            ebb: create("ebb", "An extended basic block in the same function.")
+                .default_member("destination")
+                .build(),
+
+            stack_slot: create("stack_slot", "A stack slot").build(),
+
+            global_value: create("global_value", "A global value.").build(),
+
+            sig_ref: create("sig_ref", "A function signature.").build(),
+
+            func_ref: create("func_ref", "An external function.").build(),
+
+            jump_table: create("jump_table", "A jump table.")
+                .default_member("table")
+                .build(),
+
+            heap: create("heap", "A heap.").build(),
+
+            table: create("table", "A table.").build(),
+
+            varargs: Builder::new("variable_args", OperandKindFields::VariableArgs)
+                .doc(
+                    r#"
+                        A variable size list of `value` operands.
+
+                        Use this to represent arguments passed to a function call, arguments
+                        passed to an extended basic block, or a variable number of results
+                        returned from an instruction.
+                    "#,
+                )
+                .build(),
+        }
+    }
+}
+
 /// Small helper to initialize an OperandBuilder with the right kind, for a given name and doc.
 fn create(name: &'static str, doc: &'static str) -> Builder {
     Builder::new(name, OperandKindFields::EntityRef).doc(doc)
 }
-
-pub fn define() -> Vec {
-    let mut kinds = Vec::new();
-
-    // A reference to an extended basic block in the same function.
-    // This is primarliy used in control flow instructions.
-    let ebb = create("ebb", "An extended basic block in the same function.")
-        .default_member("destination")
-        .build();
-    kinds.push(ebb);
-
-    // A reference to a stack slot declared in the function preamble.
-    let stack_slot = create("stack_slot", "A stack slot").build();
-    kinds.push(stack_slot);
-
-    // A reference to a global value.
-    let global_value = create("global_value", "A global value.").build();
-    kinds.push(global_value);
-
-    // A reference to a function signature declared in the function preamble.
-    // This is used to provide the call signature in a call_indirect instruction.
-    let sig_ref = create("sig_ref", "A function signature.").build();
-    kinds.push(sig_ref);
-
-    // A reference to an external function declared in the function preamble.
-    // This is used to provide the callee and signature in a call instruction.
-    let func_ref = create("func_ref", "An external function.").build();
-    kinds.push(func_ref);
-
-    // A reference to a jump table declared in the function preamble.
-    let jump_table = create("jump_table", "A jump table.")
-        .default_member("table")
-        .build();
-    kinds.push(jump_table);
-
-    // A reference to a heap declared in the function preamble.
-    let heap = create("heap", "A heap.").build();
-    kinds.push(heap);
-
-    // A reference to a table declared in the function preamble.
-    let table = create("table", "A table.").build();
-    kinds.push(table);
-
-    // A variable-sized list of value operands. Use for Ebb and function call arguments.
-    let varargs = Builder::new("variable_args", OperandKindFields::VariableArgs)
-        .doc(
-            r#"
-            A variable size list of `value` operands.
-
-            Use this to represent arguments passed to a function call, arguments
-            passed to an extended basic block, or a variable number of results
-            returned from an instruction.
-        "#,
-        )
-        .build();
-    kinds.push(varargs);
-
-    return kinds;
-}
diff --git a/cranelift/codegen/meta/src/shared/formats.rs b/cranelift/codegen/meta/src/shared/formats.rs
index 2f79fdbdf1..d631f337c2 100644
--- a/cranelift/codegen/meta/src/shared/formats.rs
+++ b/cranelift/codegen/meta/src/shared/formats.rs
@@ -1,18 +1,7 @@
 use crate::cdsl::formats::{FormatRegistry, InstructionFormatBuilder as Builder};
-use crate::shared::immediates::Immediates;
-use crate::shared::OperandKinds;
-
-pub fn define(imm: &Immediates, entities: &OperandKinds) -> FormatRegistry {
-    // Shorthands for entities.
-    let global_value = entities.by_name("global_value");
-    let ebb = entities.by_name("ebb");
-    let jump_table = entities.by_name("jump_table");
-    let func_ref = entities.by_name("func_ref");
-    let sig_ref = entities.by_name("sig_ref");
-    let stack_slot = entities.by_name("stack_slot");
-    let heap = entities.by_name("heap");
-    let table = entities.by_name("table");
+use crate::shared::{entities::EntityRefs, immediates::Immediates};
 
+pub fn define(imm: &Immediates, entities: &EntityRefs) -> FormatRegistry {
     let mut registry = FormatRegistry::new();
 
     registry.insert(Builder::new("Unary").value());
@@ -21,7 +10,7 @@ pub fn define(imm: &Immediates, entities: &OperandKinds) -> FormatRegistry {
     registry.insert(Builder::new("UnaryIeee32").imm(&imm.ieee32));
     registry.insert(Builder::new("UnaryIeee64").imm(&imm.ieee64));
     registry.insert(Builder::new("UnaryBool").imm(&imm.boolean));
-    registry.insert(Builder::new("UnaryGlobalValue").imm(global_value));
+    registry.insert(Builder::new("UnaryGlobalValue").imm(&entities.global_value));
 
     registry.insert(Builder::new("Binary").value().value());
     registry.insert(Builder::new("BinaryImm").value().imm(&imm.imm64));
@@ -80,20 +69,20 @@ pub fn define(imm: &Immediates, entities: &OperandKinds) -> FormatRegistry {
             .value(),
     );
 
-    registry.insert(Builder::new("Jump").imm(ebb).varargs());
-    registry.insert(Builder::new("Branch").value().imm(ebb).varargs());
+    registry.insert(Builder::new("Jump").imm(&entities.ebb).varargs());
+    registry.insert(Builder::new("Branch").value().imm(&entities.ebb).varargs());
     registry.insert(
         Builder::new("BranchInt")
             .imm(&imm.intcc)
             .value()
-            .imm(ebb)
+            .imm(&entities.ebb)
             .varargs(),
     );
     registry.insert(
         Builder::new("BranchFloat")
             .imm(&imm.floatcc)
             .value()
-            .imm(ebb)
+            .imm(&entities.ebb)
             .varargs(),
     );
     registry.insert(
@@ -101,23 +90,37 @@ pub fn define(imm: &Immediates, entities: &OperandKinds) -> FormatRegistry {
             .imm(&imm.intcc)
             .value()
             .value()
-            .imm(ebb)
+            .imm(&entities.ebb)
             .varargs(),
     );
-    registry.insert(Builder::new("BranchTable").value().imm(ebb).imm(jump_table));
+    registry.insert(
+        Builder::new("BranchTable")
+            .value()
+            .imm(&entities.ebb)
+            .imm(&entities.jump_table),
+    );
     registry.insert(
         Builder::new("BranchTableEntry")
             .value()
             .value()
             .imm(&imm.uimm8)
-            .imm(jump_table),
+            .imm(&entities.jump_table),
+    );
+    registry.insert(Builder::new("BranchTableBase").imm(&entities.jump_table));
+    registry.insert(
+        Builder::new("IndirectJump")
+            .value()
+            .imm(&entities.jump_table),
     );
-    registry.insert(Builder::new("BranchTableBase").imm(jump_table));
-    registry.insert(Builder::new("IndirectJump").value().imm(jump_table));
 
-    registry.insert(Builder::new("Call").imm(func_ref).varargs());
-    registry.insert(Builder::new("CallIndirect").imm(sig_ref).value().varargs());
-    registry.insert(Builder::new("FuncAddr").imm(func_ref));
+    registry.insert(Builder::new("Call").imm(&entities.func_ref).varargs());
+    registry.insert(
+        Builder::new("CallIndirect")
+            .imm(&entities.sig_ref)
+            .value()
+            .varargs(),
+    );
+    registry.insert(Builder::new("FuncAddr").imm(&entities.func_ref));
 
     registry.insert(
         Builder::new("Load")
@@ -145,21 +148,30 @@ pub fn define(imm: &Immediates, entities: &OperandKinds) -> FormatRegistry {
             .varargs()
             .imm(&imm.offset32),
     );
-    registry.insert(Builder::new("StackLoad").imm(stack_slot).imm(&imm.offset32));
+    registry.insert(
+        Builder::new("StackLoad")
+            .imm(&entities.stack_slot)
+            .imm(&imm.offset32),
+    );
     registry.insert(
         Builder::new("StackStore")
             .value()
-            .imm(stack_slot)
+            .imm(&entities.stack_slot)
             .imm(&imm.offset32),
     );
 
     // Accessing a WebAssembly heap.
-    registry.insert(Builder::new("HeapAddr").imm(heap).value().imm(&imm.uimm32));
+    registry.insert(
+        Builder::new("HeapAddr")
+            .imm(&entities.heap)
+            .value()
+            .imm(&imm.uimm32),
+    );
 
     // Accessing a WebAssembly table.
     registry.insert(
         Builder::new("TableAddr")
-            .imm(table)
+            .imm(&entities.table)
             .value()
             .imm(&imm.offset32),
     );
@@ -180,12 +192,12 @@ pub fn define(imm: &Immediates, entities: &OperandKinds) -> FormatRegistry {
         Builder::new("RegSpill")
             .value()
             .imm_with_name("src", &imm.regunit)
-            .imm_with_name("dst", stack_slot),
+            .imm_with_name("dst", &entities.stack_slot),
     );
     registry.insert(
         Builder::new("RegFill")
             .value()
-            .imm_with_name("src", stack_slot)
+            .imm_with_name("src", &entities.stack_slot)
             .imm_with_name("dst", &imm.regunit),
     );
 
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index ad94efeb80..9499bf190a 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -8,14 +8,14 @@ use crate::cdsl::operands::{create_operand as operand, create_operand_doc as ope
 use crate::cdsl::type_inference::Constraint::WiderOrEq;
 use crate::cdsl::types::{LaneType, ValueType};
 use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
-use crate::shared::immediates::Immediates;
-use crate::shared::{types, OperandKinds};
+use crate::shared::types;
+use crate::shared::{entities::EntityRefs, immediates::Immediates};
 
 pub fn define(
     all_instructions: &mut AllInstructions,
     format_registry: &FormatRegistry,
     imm: &Immediates,
-    entities: &OperandKinds,
+    entities: &EntityRefs,
 ) -> InstructionGroup {
     let mut ig = InstructionGroupBuilder::new(
         "base",
@@ -25,16 +25,6 @@ pub fn define(
     );
 
     // Operand kind shorthands.
-    let ebb = entities.by_name("ebb");
-    let jump_table = entities.by_name("jump_table");
-    let variable_args = entities.by_name("variable_args");
-    let func_ref = entities.by_name("func_ref");
-    let sig_ref = entities.by_name("sig_ref");
-    let stack_slot = entities.by_name("stack_slot");
-    let global_value = entities.by_name("global_value");
-    let heap = entities.by_name("heap");
-    let table = entities.by_name("table");
-
     let iflags: &TypeVar = &ValueType::Special(types::Flag::IFlags.into()).into();
     let fflags: &TypeVar = &ValueType::Special(types::Flag::FFlags.into()).into();
 
@@ -132,8 +122,8 @@ pub fn define(
     let Cond = &operand("Cond", &imm.intcc);
     let x = &operand("x", iB);
     let y = &operand("y", iB);
-    let EBB = &operand_doc("EBB", ebb, "Destination extended basic block");
-    let args = &operand_doc("args", variable_args, "EBB arguments");
+    let EBB = &operand_doc("EBB", &entities.ebb, "Destination extended basic block");
+    let args = &operand_doc("args", &entities.varargs, "EBB arguments");
 
     ig.push(
         Inst::new(
@@ -257,7 +247,7 @@ pub fn define(
     // The index into the br_table can be any type; legalizer will convert it to the right type.
     let x = &operand_doc("x", iB, "index into jump table");
     let entry = &operand_doc("entry", iAddr, "entry of jump table");
-    let JT = &operand("JT", jump_table);
+    let JT = &operand("JT", &entities.jump_table);
 
     ig.push(
         Inst::new(
@@ -433,7 +423,7 @@ pub fn define(
         .can_trap(true),
     );
 
-    let rvals = &operand_doc("rvals", variable_args, "return values");
+    let rvals = &operand_doc("rvals", &entities.varargs, "return values");
 
     ig.push(
         Inst::new(
@@ -467,8 +457,12 @@ pub fn define(
         .is_terminator(true),
     );
 
-    let FN = &operand_doc("FN", func_ref, "function to call, declared by `function`");
-    let args = &operand_doc("args", variable_args, "call arguments");
+    let FN = &operand_doc(
+        "FN",
+        &entities.func_ref,
+        "function to call, declared by `function`",
+    );
+    let args = &operand_doc("args", &entities.varargs, "call arguments");
 
     ig.push(
         Inst::new(
@@ -485,7 +479,7 @@ pub fn define(
         .is_call(true),
     );
 
-    let SIG = &operand_doc("SIG", sig_ref, "function signature");
+    let SIG = &operand_doc("SIG", &entities.sig_ref, "function signature");
     let callee = &operand_doc("callee", iAddr, "address of function to call");
 
     ig.push(
@@ -525,13 +519,13 @@ pub fn define(
         .operands_out(vec![addr]),
     );
 
-    let SS = &operand("SS", stack_slot);
+    let SS = &operand("SS", &entities.stack_slot);
     let Offset = &operand_doc("Offset", &imm.offset32, "Byte offset from base address");
     let x = &operand_doc("x", Mem, "Value to be stored");
     let a = &operand_doc("a", Mem, "Value loaded");
     let p = &operand("p", iAddr);
     let MemFlags = &operand("MemFlags", &imm.memflags);
-    let args = &operand_doc("args", variable_args, "Address arguments");
+    let args = &operand_doc("args", &entities.varargs, "Address arguments");
 
     ig.push(
         Inst::new(
@@ -917,7 +911,7 @@ pub fn define(
         .operands_out(vec![addr]),
     );
 
-    let GV = &operand("GV", global_value);
+    let GV = &operand("GV", &entities.global_value);
 
     ig.push(
         Inst::new(
@@ -947,7 +941,7 @@ pub fn define(
         TypeSetBuilder::new().ints(32..64).build(),
     );
 
-    let H = &operand("H", heap);
+    let H = &operand("H", &entities.heap);
     let p = &operand("p", HeapOffset);
     let Size = &operand_doc("Size", &imm.uimm32, "Size in bytes");
 
@@ -975,7 +969,7 @@ pub fn define(
         "An unsigned table offset",
         TypeSetBuilder::new().ints(32..64).build(),
     );
-    let T = &operand("T", table);
+    let T = &operand("T", &entities.table);
     let p = &operand("p", TableOffset);
     let Offset = &operand_doc("Offset", &imm.offset32, "Byte offset from element address");
 
@@ -1382,7 +1376,7 @@ pub fn define(
 
     let N = &operand_doc(
         "args",
-        variable_args,
+        &entities.varargs,
         "Variable number of args for Stackmap",
     );
 
diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
index 837bf3c71d..29c74caa06 100644
--- a/cranelift/codegen/meta/src/shared/mod.rs
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -1,6 +1,6 @@
 //! Shared definitions for the Cranelift intermediate language.
 
-pub mod entities;
+mod entities;
 pub mod formats;
 pub mod immediates;
 pub mod instructions;
@@ -10,10 +10,10 @@ pub mod types;
 
 use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::instructions::{AllInstructions, InstructionGroup};
-use crate::cdsl::operands::OperandKind;
 use crate::cdsl::settings::SettingGroup;
 use crate::cdsl::xform::TransformGroups;
 
+use crate::shared::entities::EntityRefs;
 use crate::shared::immediates::Immediates;
 
 pub struct Definitions {
@@ -25,28 +25,11 @@ pub struct Definitions {
     pub transform_groups: TransformGroups,
 }
 
-pub struct OperandKinds(Vec);
-
-impl OperandKinds {
-    pub fn by_name(&self, name: &'static str) -> &OperandKind {
-        self.0
-            .iter()
-            .find(|op| op.name == name)
-            .expect(&format!("unknown Operand name: {}", name))
-    }
-}
-
-impl From> for OperandKinds {
-    fn from(kinds: Vec) -> Self {
-        OperandKinds(kinds)
-    }
-}
-
 pub fn define() -> Definitions {
     let mut all_instructions = AllInstructions::new();
 
     let immediates = Immediates::new();
-    let entities = OperandKinds(entities::define());
+    let entities = EntityRefs::new();;
     let format_registry = formats::define(&immediates, &entities);
     let instructions = instructions::define(
         &mut all_instructions,

From d1d2e790b9aa0cbecc6a10a047b2ff0b790ca3d3 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 4 Sep 2019 17:50:51 +0200
Subject: [PATCH 2647/3084] [meta] Morph a few pub into pub(crate), and remove
 dead code;

---
 cranelift/codegen/meta/src/cdsl/ast.rs            | 15 ---------------
 cranelift/codegen/meta/src/cdsl/instructions.rs   |  5 -----
 cranelift/codegen/meta/src/cdsl/xform.rs          |  3 ---
 cranelift/codegen/meta/src/gen_encodings.rs       |  2 +-
 cranelift/codegen/meta/src/gen_inst.rs            |  2 +-
 cranelift/codegen/meta/src/isa/arm32/mod.rs       |  2 +-
 cranelift/codegen/meta/src/isa/arm64/mod.rs       |  2 +-
 cranelift/codegen/meta/src/isa/mod.rs             |  2 +-
 cranelift/codegen/meta/src/isa/riscv/encodings.rs |  2 +-
 cranelift/codegen/meta/src/isa/riscv/mod.rs       |  2 +-
 cranelift/codegen/meta/src/isa/riscv/recipes.rs   |  2 +-
 cranelift/codegen/meta/src/isa/x86/encodings.rs   |  2 +-
 .../codegen/meta/src/isa/x86/instructions.rs      |  2 +-
 cranelift/codegen/meta/src/isa/x86/legalize.rs    |  2 +-
 cranelift/codegen/meta/src/isa/x86/mod.rs         |  2 +-
 cranelift/codegen/meta/src/isa/x86/recipes.rs     |  2 +-
 cranelift/codegen/meta/src/shared/formats.rs      |  2 +-
 cranelift/codegen/meta/src/shared/immediates.rs   |  2 +-
 cranelift/codegen/meta/src/shared/instructions.rs |  2 +-
 cranelift/codegen/meta/src/shared/legalize.rs     |  2 +-
 cranelift/codegen/meta/src/shared/mod.rs          |  4 ++--
 21 files changed, 19 insertions(+), 42 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 5e1c192f0f..141ffd84c2 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -11,7 +11,6 @@ use std::fmt;
 pub enum Expr {
     Var(VarIndex),
     Literal(Literal),
-    Apply(Apply),
 }
 
 impl Expr {
@@ -39,7 +38,6 @@ impl Expr {
         match self {
             Expr::Var(var_index) => var_pool.get(*var_index).to_rust_code(),
             Expr::Literal(literal) => literal.to_rust_code(),
-            Expr::Apply(a) => a.to_rust_code(var_pool),
         }
     }
 }
@@ -81,9 +79,6 @@ impl DefPool {
     pub fn get(&self, index: DefIndex) -> &Def {
         self.pool.get(index).unwrap()
     }
-    pub fn get_mut(&mut self, index: DefIndex) -> &mut Def {
-        self.pool.get_mut(index).unwrap()
-    }
     pub fn next_index(&self) -> DefIndex {
         self.pool.next_key()
     }
@@ -430,16 +425,6 @@ impl Apply {
         format!("{}({})", inst_name, args)
     }
 
-    fn to_rust_code(&self, var_pool: &VarPool) -> String {
-        let args = self
-            .args
-            .iter()
-            .map(|arg| arg.to_rust_code(var_pool))
-            .collect::>()
-            .join(", ");
-        format!("{}({})", self.inst.name, args)
-    }
-
     pub fn inst_predicate(
         &self,
         format_registry: &FormatRegistry,
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index da59beb2b0..1689f2a5c0 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -4,7 +4,6 @@ use std::collections::HashMap;
 use std::fmt;
 use std::ops;
 use std::rc::Rc;
-use std::slice;
 
 use crate::cdsl::camel_case;
 use crate::cdsl::formats::{
@@ -72,10 +71,6 @@ pub struct InstructionGroup {
 }
 
 impl InstructionGroup {
-    pub fn iter(&self) -> slice::Iter {
-        self.instructions.iter()
-    }
-
     pub fn by_name(&self, name: &'static str) -> &Instruction {
         self.instructions
             .iter()
diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index e988bf12ac..b1a0234cd4 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -384,9 +384,6 @@ impl TransformGroups {
     pub fn get(&self, id: TransformGroupIndex) -> &TransformGroup {
         &self.groups[id]
     }
-    pub fn get_mut(&mut self, id: TransformGroupIndex) -> &mut TransformGroup {
-        self.groups.get_mut(id).unwrap()
-    }
     fn next_key(&self) -> TransformGroupIndex {
         self.groups.next_key()
     }
diff --git a/cranelift/codegen/meta/src/gen_encodings.rs b/cranelift/codegen/meta/src/gen_encodings.rs
index 71fcf17002..cfaa63f087 100644
--- a/cranelift/codegen/meta/src/gen_encodings.rs
+++ b/cranelift/codegen/meta/src/gen_encodings.rs
@@ -1126,7 +1126,7 @@ fn gen_isa(defs: &SharedDefinitions, isa: &TargetIsa, fmt: &mut Formatter) {
     fmt.line("};");
 }
 
-pub fn generate(
+pub(crate) fn generate(
     defs: &SharedDefinitions,
     isa: &TargetIsa,
     filename: &str,
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index 5dbb9dc095..7d8353dad3 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -1059,7 +1059,7 @@ fn gen_builder(instructions: &AllInstructions, formats: &FormatRegistry, fmt: &m
     fmt.line("}");
 }
 
-pub fn generate(
+pub(crate) fn generate(
     shared_defs: &SharedDefinitions,
     opcode_filename: &str,
     inst_builder_filename: &str,
diff --git a/cranelift/codegen/meta/src/isa/arm32/mod.rs b/cranelift/codegen/meta/src/isa/arm32/mod.rs
index d143a15163..fb41db2530 100644
--- a/cranelift/codegen/meta/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm32/mod.rs
@@ -49,7 +49,7 @@ fn define_regs() -> IsaRegs {
     regs.build()
 }
 
-pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
+pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let settings = define_settings(&shared_defs.settings);
     let regs = define_regs();
 
diff --git a/cranelift/codegen/meta/src/isa/arm64/mod.rs b/cranelift/codegen/meta/src/isa/arm64/mod.rs
index 195fc80ae6..697e9513c5 100644
--- a/cranelift/codegen/meta/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm64/mod.rs
@@ -45,7 +45,7 @@ fn define_registers() -> IsaRegs {
     regs.build()
 }
 
-pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
+pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let settings = define_settings(&shared_defs.settings);
     let regs = define_registers();
 
diff --git a/cranelift/codegen/meta/src/isa/mod.rs b/cranelift/codegen/meta/src/isa/mod.rs
index 0c2bb5eed0..6c5ba13888 100644
--- a/cranelift/codegen/meta/src/isa/mod.rs
+++ b/cranelift/codegen/meta/src/isa/mod.rs
@@ -55,7 +55,7 @@ impl fmt::Display for Isa {
     }
 }
 
-pub fn define(isas: &Vec, shared_defs: &mut SharedDefinitions) -> Vec {
+pub(crate) fn define(isas: &Vec, shared_defs: &mut SharedDefinitions) -> Vec {
     isas.iter()
         .map(|isa| match isa {
             Isa::Riscv => riscv::define(shared_defs),
diff --git a/cranelift/codegen/meta/src/isa/riscv/encodings.rs b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
index af5e471a53..02a3bc7b61 100644
--- a/cranelift/codegen/meta/src/isa/riscv/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
@@ -101,7 +101,7 @@ fn lui_bits() -> u16 {
     0b01101
 }
 
-pub fn define<'defs>(
+pub(crate) fn define<'defs>(
     shared_defs: &'defs SharedDefinitions,
     isa_settings: &SettingGroup,
     recipes: &'defs RecipeGroup,
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index 435b38eb34..a2683b5a8b 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -85,7 +85,7 @@ fn define_registers() -> IsaRegs {
     regs.build()
 }
 
-pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
+pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let settings = define_settings(&shared_defs.settings);
     let regs = define_registers();
 
diff --git a/cranelift/codegen/meta/src/isa/riscv/recipes.rs b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
index 40396b4ab3..f32bbd568c 100644
--- a/cranelift/codegen/meta/src/isa/riscv/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
@@ -50,7 +50,7 @@ impl<'formats> RecipeGroup<'formats> {
     }
 }
 
-pub fn define<'formats>(
+pub(crate) fn define<'formats>(
     shared_defs: &'formats SharedDefinitions,
     regs: &IsaRegs,
 ) -> RecipeGroup<'formats> {
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 072cbb1b2a..ecdff33740 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -318,7 +318,7 @@ impl PerCpuModeEncodings {
 
 // Definitions.
 
-pub fn define(
+pub(crate) fn define(
     shared_defs: &SharedDefinitions,
     settings: &SettingGroup,
     x86: &InstructionGroup,
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 742b632642..03730cdeac 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -10,7 +10,7 @@ use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
 use crate::shared::immediates::Immediates;
 use crate::shared::types;
 
-pub fn define(
+pub(crate) fn define(
     mut all_instructions: &mut AllInstructions,
     format_registry: &FormatRegistry,
     immediates: &Immediates,
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 158b05e4f1..2fd160de3a 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -6,7 +6,7 @@ use crate::shared::types::Float::F64;
 use crate::shared::types::Int::{I32, I64};
 use crate::shared::Definitions as SharedDefinitions;
 
-pub fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGroup) {
+pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGroup) {
     let mut group = TransformGroupBuilder::new(
         "x86_expand",
         r#"
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index 5122c7b2ad..10baf256a9 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -13,7 +13,7 @@ mod recipes;
 mod registers;
 mod settings;
 
-pub fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
+pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let settings = settings::define(&shared_defs.settings);
     let regs = registers::define();
 
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 8eb0fdac6d..6a369f34c6 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -325,7 +325,7 @@ fn valid_scale(format: &InstructionFormat) -> InstructionPredicate {
         })
 }
 
-pub fn define<'shared>(
+pub(crate) fn define<'shared>(
     shared_defs: &'shared SharedDefinitions,
     settings: &'shared SettingGroup,
     regs: &'shared IsaRegs,
diff --git a/cranelift/codegen/meta/src/shared/formats.rs b/cranelift/codegen/meta/src/shared/formats.rs
index d631f337c2..5309afc1b0 100644
--- a/cranelift/codegen/meta/src/shared/formats.rs
+++ b/cranelift/codegen/meta/src/shared/formats.rs
@@ -1,7 +1,7 @@
 use crate::cdsl::formats::{FormatRegistry, InstructionFormatBuilder as Builder};
 use crate::shared::{entities::EntityRefs, immediates::Immediates};
 
-pub fn define(imm: &Immediates, entities: &EntityRefs) -> FormatRegistry {
+pub(crate) fn define(imm: &Immediates, entities: &EntityRefs) -> FormatRegistry {
     let mut registry = FormatRegistry::new();
 
     registry.insert(Builder::new("Unary").value());
diff --git a/cranelift/codegen/meta/src/shared/immediates.rs b/cranelift/codegen/meta/src/shared/immediates.rs
index b7a7f712a2..30c1a73970 100644
--- a/cranelift/codegen/meta/src/shared/immediates.rs
+++ b/cranelift/codegen/meta/src/shared/immediates.rs
@@ -2,7 +2,7 @@ use crate::cdsl::operands::{OperandKind, OperandKindBuilder as Builder};
 
 use std::collections::HashMap;
 
-pub struct Immediates {
+pub(crate) struct Immediates {
     /// A 64-bit immediate integer operand.
     ///
     /// This type of immediate integer can interact with SSA values with any IntType type.
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 9499bf190a..f636141a37 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -11,7 +11,7 @@ use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
 use crate::shared::types;
 use crate::shared::{entities::EntityRefs, immediates::Immediates};
 
-pub fn define(
+pub(crate) fn define(
     all_instructions: &mut AllInstructions,
     format_registry: &FormatRegistry,
     imm: &Immediates,
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index ce2faa7720..90fe47d0fd 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -6,7 +6,7 @@ use crate::shared::immediates::Immediates;
 use crate::shared::types::Float::{F32, F64};
 use crate::shared::types::Int::{I16, I32, I64, I8};
 
-pub fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGroups {
+pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGroups {
     let mut narrow = TransformGroupBuilder::new(
         "narrow",
         r#"
diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
index 29c74caa06..12921da5ef 100644
--- a/cranelift/codegen/meta/src/shared/mod.rs
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -16,7 +16,7 @@ use crate::cdsl::xform::TransformGroups;
 use crate::shared::entities::EntityRefs;
 use crate::shared::immediates::Immediates;
 
-pub struct Definitions {
+pub(crate) struct Definitions {
     pub settings: SettingGroup,
     pub all_instructions: AllInstructions,
     pub instructions: InstructionGroup,
@@ -25,7 +25,7 @@ pub struct Definitions {
     pub transform_groups: TransformGroups,
 }
 
-pub fn define() -> Definitions {
+pub(crate) fn define() -> Definitions {
     let mut all_instructions = AllInstructions::new();
 
     let immediates = Immediates::new();

From 660b8b28b8bb021f5d06ddcbf90d8ec59e346ba9 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 28 Aug 2019 17:33:45 +0200
Subject: [PATCH 2648/3084] [codegen] Add a pinned register that's entirely
 under the control of the user;

---
 cranelift/codegen/meta/src/cdsl/regs.rs       |  11 ++
 cranelift/codegen/meta/src/gen_registers.rs   |   1 +
 .../codegen/meta/src/isa/x86/encodings.rs     |  11 ++
 cranelift/codegen/meta/src/isa/x86/recipes.rs |  20 ++++
 .../codegen/meta/src/isa/x86/registers.rs     |   3 +-
 .../codegen/meta/src/shared/instructions.rs   |  28 +++++
 cranelift/codegen/meta/src/shared/settings.rs |  11 ++
 cranelift/codegen/src/isa/registers.rs        |  15 +++
 cranelift/codegen/src/isa/x86/abi.rs          |  14 ++-
 cranelift/codegen/src/isa/x86/mod.rs          |   4 +-
 cranelift/codegen/src/regalloc/coloring.rs    | 110 ++++++++++++++----
 .../codegen/src/regalloc/register_set.rs      |   2 +
 cranelift/codegen/src/regalloc/solver.rs      |   8 +-
 cranelift/codegen/src/settings.rs             |   1 +
 cranelift/codegen/src/verifier/mod.rs         |  20 ++++
 15 files changed, 229 insertions(+), 30 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/regs.rs b/cranelift/codegen/meta/src/cdsl/regs.rs
index e7a3464d10..920b1f5426 100644
--- a/cranelift/codegen/meta/src/cdsl/regs.rs
+++ b/cranelift/codegen/meta/src/cdsl/regs.rs
@@ -11,6 +11,7 @@ pub struct RegBank {
     pub names: Vec<&'static str>,
     pub prefix: &'static str,
     pub pressure_tracking: bool,
+    pub pinned_reg: Option,
     pub toprcs: Vec,
     pub classes: Vec,
 }
@@ -23,6 +24,7 @@ impl RegBank {
         names: Vec<&'static str>,
         prefix: &'static str,
         pressure_tracking: bool,
+        pinned_reg: Option,
     ) -> Self {
         RegBank {
             name,
@@ -31,6 +33,7 @@ impl RegBank {
             names,
             prefix,
             pressure_tracking,
+            pinned_reg,
             toprcs: Vec::new(),
             classes: Vec::new(),
         }
@@ -183,6 +186,7 @@ pub struct RegBankBuilder {
     pub names: Vec<&'static str>,
     pub prefix: &'static str,
     pub pressure_tracking: Option,
+    pub pinned_reg: Option,
 }
 
 impl RegBankBuilder {
@@ -193,6 +197,7 @@ impl RegBankBuilder {
             names: vec![],
             prefix,
             pressure_tracking: None,
+            pinned_reg: None,
         }
     }
     pub fn units(mut self, units: u8) -> Self {
@@ -207,6 +212,11 @@ impl RegBankBuilder {
         self.pressure_tracking = Some(track);
         self
     }
+    pub fn pinned_reg(mut self, unit: u16) -> Self {
+        assert!(unit < (self.units as u16));
+        self.pinned_reg = Some(unit);
+        self
+    }
 }
 
 pub struct IsaRegsBuilder {
@@ -246,6 +256,7 @@ impl IsaRegsBuilder {
             builder
                 .pressure_tracking
                 .expect("Pressure tracking must be explicitly set"),
+            builder.pinned_reg,
         ))
     }
 
diff --git a/cranelift/codegen/meta/src/gen_registers.rs b/cranelift/codegen/meta/src/gen_registers.rs
index 5e0fdac57b..08edfa9e21 100644
--- a/cranelift/codegen/meta/src/gen_registers.rs
+++ b/cranelift/codegen/meta/src/gen_registers.rs
@@ -56,6 +56,7 @@ fn gen_regclass(isa: &TargetIsa, reg_class: &RegClass, fmt: &mut Formatter) {
         fmtln!(fmt, "first: {},", reg_bank.first_unit + reg_class.start);
         fmtln!(fmt, "subclasses: {:#x},", reg_class.subclass_mask());
         fmtln!(fmt, "mask: [{}],", mask);
+        fmtln!(fmt, "pinned_reg: {:?},", reg_bank.pinned_reg);
         fmtln!(fmt, "info: &INFO,");
     });
     fmtln!(fmt, "};");
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index ecdff33740..71f1042100 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -372,6 +372,7 @@ pub(crate) fn define(
     let fpromote = shared.by_name("fpromote");
     let fsub = shared.by_name("fsub");
     let func_addr = shared.by_name("func_addr");
+    let get_pinned_reg = shared.by_name("get_pinned_reg");
     let iadd = shared.by_name("iadd");
     let iadd_cout = shared.by_name("iadd_cout");
     let iadd_cin = shared.by_name("iadd_cin");
@@ -421,6 +422,7 @@ pub(crate) fn define(
     let scalar_to_vector = shared.by_name("scalar_to_vector");
     let selectif = shared.by_name("selectif");
     let sextend = shared.by_name("sextend");
+    let set_pinned_reg = shared.by_name("set_pinned_reg");
     let sload16 = shared.by_name("sload16");
     let sload16_complex = shared.by_name("sload16_complex");
     let sload32 = shared.by_name("sload32");
@@ -516,6 +518,7 @@ pub(crate) fn define(
     let rec_furm = r.template("furm");
     let rec_furm_reg_to_ssa = r.template("furm_reg_to_ssa");
     let rec_furmi_rnd = r.template("furmi_rnd");
+    let rec_get_pinned_reg = r.recipe("get_pinned_reg");
     let rec_got_fnaddr8 = r.template("got_fnaddr8");
     let rec_got_gvaddr8 = r.template("got_gvaddr8");
     let rec_gvaddr4 = r.template("gvaddr4");
@@ -569,6 +572,7 @@ pub(crate) fn define(
     let rec_safepoint = r.recipe("safepoint");
     let rec_setf_abcd = r.template("setf_abcd");
     let rec_seti_abcd = r.template("seti_abcd");
+    let rec_set_pinned_reg = r.template("set_pinned_reg");
     let rec_spaddr4_id = r.template("spaddr4_id");
     let rec_spaddr8_id = r.template("spaddr8_id");
     let rec_spillSib32 = r.template("spillSib32");
@@ -619,6 +623,13 @@ pub(crate) fn define(
     // Definitions.
     let mut e = PerCpuModeEncodings::new();
 
+    // The pinned reg is fixed to a certain value entirely user-controlled, so it generates nothing!
+    e.enc64_rec(get_pinned_reg.bind(I64), rec_get_pinned_reg, 0);
+    e.enc_x86_64(
+        set_pinned_reg.bind(I64),
+        rec_set_pinned_reg.opcodes(vec![0x89]).rex().w(),
+    );
+
     e.enc_i32_i64(iadd, rec_rr.opcodes(vec![0x01]));
     e.enc_i32_i64(iadd_cout, rec_rr.opcodes(vec![0x01]));
     e.enc_i32_i64(iadd_cin, rec_rin.opcodes(vec![0x11]));
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 6a369f34c6..a5db0c1b97 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -351,6 +351,7 @@ pub(crate) fn define<'shared>(
     let reg_rax = Register::new(gpr, regs.regunit_by_name(gpr, "rax"));
     let reg_rcx = Register::new(gpr, regs.regunit_by_name(gpr, "rcx"));
     let reg_rdx = Register::new(gpr, regs.regunit_by_name(gpr, "rdx"));
+    let reg_r15 = Register::new(gpr, regs.regunit_by_name(gpr, "r15"));
 
     // Stack operand with a 32-bit signed displacement from either RBP or RSP.
     let stack_gpr32 = Stack::new(gpr);
@@ -428,6 +429,25 @@ pub(crate) fn define<'shared>(
             .emit(""),
     );
 
+    recipes.add_recipe(
+        EncodingRecipeBuilder::new("get_pinned_reg", f_nullary, 0)
+            .operands_out(vec![reg_r15])
+            .emit(""),
+    );
+    // umr with a fixed register output that's r15.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("set_pinned_reg", f_unary, 1)
+            .operands_in(vec![gpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    let r15 = RU::r15.into();
+                    {{PUT_OP}}(bits, rex2(r15, in_reg0), sink);
+                    modrm_rr(r15, in_reg0, sink);
+                "#,
+            ),
+    );
+
     // No-op fills, created by late-stage redundant-fill removal.
     recipes.add_recipe(
         EncodingRecipeBuilder::new("fillnull", f_unary, 0)
diff --git a/cranelift/codegen/meta/src/isa/x86/registers.rs b/cranelift/codegen/meta/src/isa/x86/registers.rs
index 3039bafba2..4157084c15 100644
--- a/cranelift/codegen/meta/src/isa/x86/registers.rs
+++ b/cranelift/codegen/meta/src/isa/x86/registers.rs
@@ -6,7 +6,8 @@ pub fn define() -> IsaRegs {
     let builder = RegBankBuilder::new("IntRegs", "r")
         .units(16)
         .names(vec!["rax", "rcx", "rdx", "rbx", "rsp", "rbp", "rsi", "rdi"])
-        .track_pressure(true);
+        .track_pressure(true)
+        .pinned_reg(15);
     let int_regs = regs.add_bank(builder);
 
     let builder = RegBankBuilder::new("FloatRegs", "xmm")
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index f636141a37..67e2891068 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -964,6 +964,34 @@ pub(crate) fn define(
         .operands_out(vec![addr]),
     );
 
+    // Note this instruction is marked as having other side-effects, so GVN won't try to hoist it,
+    // which would result in it being subject to spilling. While not hoisting would generally hurt
+    // performance, since a computed value used many times may need to be regenerated before each
+    // use, it is not the case here: this instruction doesn't generate any code.  That's because,
+    // by definition the pinned register is never used by the register allocator, but is written to
+    // and read explicitly and exclusively by set_pinned_reg and get_pinned_reg.
+    ig.push(
+        Inst::new(
+            "get_pinned_reg",
+            r#"
+            Gets the content of the pinned register, when it's enabled.
+        "#,
+        )
+        .operands_out(vec![addr])
+        .other_side_effects(true),
+    );
+
+    ig.push(
+        Inst::new(
+            "set_pinned_reg",
+            r#"
+        Sets the content of the pinned register, when it's enabled.
+        "#,
+        )
+        .operands_in(vec![addr])
+        .other_side_effects(true),
+    );
+
     let TableOffset = &TypeVar::new(
         "TableOffset",
         "An unsigned table offset",
diff --git a/cranelift/codegen/meta/src/shared/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
index 6561ee9856..e7c567c345 100644
--- a/cranelift/codegen/meta/src/shared/settings.rs
+++ b/cranelift/codegen/meta/src/shared/settings.rs
@@ -84,6 +84,17 @@ pub fn define() -> SettingGroup {
         false,
     );
 
+    settings.add_bool(
+        "enable_pinned_reg",
+        r#"Enable the use of the pinned register.
+
+        This register is excluded from register allocation, and is completely under the control of
+        the end-user. It is possible to read it via the get_pinned_reg instruction, and to set it
+        with the set_pinned_reg instruction.
+        "#,
+        false,
+    );
+
     settings.add_bool("enable_simd", "Enable the use of SIMD instructions.", false);
 
     settings.add_bool(
diff --git a/cranelift/codegen/src/isa/registers.rs b/cranelift/codegen/src/isa/registers.rs
index 3c1b4262c1..f7fcdcac2e 100644
--- a/cranelift/codegen/src/isa/registers.rs
+++ b/cranelift/codegen/src/isa/registers.rs
@@ -154,6 +154,12 @@ pub struct RegClassData {
 
     /// The global `RegInfo` instance containing this register class.
     pub info: &'static RegInfo,
+
+    /// The "pinned" register of the associated register bank.
+    ///
+    /// This register must be non-volatile (callee-preserved) and must not be the fixed
+    /// output register of any instruction.
+    pub pinned_reg: Option,
 }
 
 impl RegClassData {
@@ -201,6 +207,15 @@ impl RegClassData {
     pub fn contains(&self, regunit: RegUnit) -> bool {
         self.mask[(regunit / 32) as usize] & (1u32 << (regunit % 32)) != 0
     }
+
+    /// If the pinned register is used, is the given regunit the pinned register of this class?
+    #[inline]
+    pub fn is_pinned_reg(&self, enabled: bool, regunit: RegUnit) -> bool {
+        enabled
+            && self
+                .pinned_reg
+                .map_or(false, |pinned_reg| pinned_reg == regunit)
+    }
 }
 
 impl fmt::Display for RegClassData {
diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 0072935229..6252a3fb77 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -89,9 +89,8 @@ impl ArgAssigner for Args {
                 let reg = FPR.unit(self.fpr_used);
                 self.fpr_used += 1;
                 return ArgumentLoc::Reg(reg).into();
-            } else {
-                return ValueConversion::VectorSplit.into();
             }
+            return ValueConversion::VectorSplit.into();
         }
 
         // Large integers and booleans are broken down to fit in a register.
@@ -229,7 +228,7 @@ pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass {
 }
 
 /// Get the set of allocatable registers for `func`.
-pub fn allocatable_registers(_func: &ir::Function, triple: &Triple) -> RegisterSet {
+pub fn allocatable_registers(triple: &Triple, flags: &shared_settings::Flags) -> RegisterSet {
     let mut regs = RegisterSet::new();
     regs.take(GPR, RU::rsp as RegUnit);
     regs.take(GPR, RU::rbp as RegUnit);
@@ -240,6 +239,15 @@ pub fn allocatable_registers(_func: &ir::Function, triple: &Triple) -> RegisterS
             regs.take(GPR, GPR.unit(i));
             regs.take(FPR, FPR.unit(i));
         }
+        if flags.enable_pinned_reg() {
+            unimplemented!("Pinned register not implemented on x86-32.");
+        }
+    } else {
+        // Choose r15 as the pinned register on 64-bits: it is non-volatile on native ABIs and
+        // isn't the fixed output register of any instruction.
+        if flags.enable_pinned_reg() {
+            regs.take(GPR, RU::r15 as RegUnit);
+        }
     }
 
     regs
diff --git a/cranelift/codegen/src/isa/x86/mod.rs b/cranelift/codegen/src/isa/x86/mod.rs
index 9244791ea8..52ab055440 100644
--- a/cranelift/codegen/src/isa/x86/mod.rs
+++ b/cranelift/codegen/src/isa/x86/mod.rs
@@ -119,8 +119,8 @@ impl TargetIsa for Isa {
         abi::regclass_for_abi_type(ty)
     }
 
-    fn allocatable_registers(&self, func: &ir::Function) -> regalloc::RegisterSet {
-        abi::allocatable_registers(func, &self.triple)
+    fn allocatable_registers(&self, _func: &ir::Function) -> regalloc::RegisterSet {
+        abi::allocatable_registers(&self.triple, &self.shared_flags)
     }
 
     #[cfg(feature = "testing_hooks")]
diff --git a/cranelift/codegen/src/regalloc/coloring.rs b/cranelift/codegen/src/regalloc/coloring.rs
index f23c738583..1e69c342a9 100644
--- a/cranelift/codegen/src/regalloc/coloring.rs
+++ b/cranelift/codegen/src/regalloc/coloring.rs
@@ -106,6 +106,8 @@ struct Context<'a> {
     // Pristine set of registers that the allocator can use.
     // This set remains immutable, we make clones.
     usable_regs: RegisterSet,
+
+    uses_pinned_reg: bool,
 }
 
 impl Coloring {
@@ -137,6 +139,7 @@ impl Coloring {
         debug!("Coloring for:\n{}", func.display(isa));
         let mut ctx = Context {
             usable_regs: isa.allocatable_registers(func),
+            uses_pinned_reg: isa.flags().enable_pinned_reg(),
             cur: EncCursor::new(func, isa),
             reginfo: isa.register_info(),
             encinfo: isa.encoding_info(),
@@ -151,6 +154,12 @@ impl Coloring {
 }
 
 impl<'a> Context<'a> {
+    /// Is the pinned register usage enabled, and is this register the pinned register?
+    #[inline]
+    fn is_pinned_reg(&self, rc: RegClass, reg: RegUnit) -> bool {
+        rc.is_pinned_reg(self.uses_pinned_reg, reg)
+    }
+
     /// Run the coloring algorithm.
     fn run(&mut self, tracker: &mut LiveValueTracker) {
         self.cur
@@ -425,9 +434,11 @@ impl<'a> Context<'a> {
 
         // Program the solver with register constraints for the input side.
         self.solver.reset(®s.input);
+
         if let Some(constraints) = constraints {
             self.program_input_constraints(inst, constraints.ins);
         }
+
         let call_sig = self.cur.func.dfg.call_signature(inst);
         if let Some(sig) = call_sig {
             self.program_input_abi(inst, AbiParams::Parameters(sig));
@@ -457,6 +468,7 @@ impl<'a> Context<'a> {
         if self.solver.has_fixed_input_conflicts() {
             self.divert_fixed_input_conflicts(tracker.live());
         }
+
         self.solver.inputs_done();
 
         // Update the live value tracker with this instruction.
@@ -467,6 +479,13 @@ impl<'a> Context<'a> {
             if let Affinity::Reg(rci) = lv.affinity {
                 let rc = self.reginfo.rc(rci);
                 let reg = self.divert.reg(lv.value, &self.cur.func.locations);
+
+                if self.is_pinned_reg(rc, reg) {
+                    // Don't kill the pinned reg, either in the local or global register sets.
+                    debug_assert!(lv.is_local, "pinned register SSA value can't be global");
+                    continue;
+                }
+
                 debug!(
                     "    kill {} in {} ({} {})",
                     lv.value,
@@ -506,6 +525,7 @@ impl<'a> Context<'a> {
                 );
             }
         }
+
         if let Some(sig) = call_sig {
             self.program_output_abi(
                 sig,
@@ -515,6 +535,7 @@ impl<'a> Context<'a> {
                 ®s.global,
             );
         }
+
         if let Some(constraints) = constraints {
             self.program_output_constraints(
                 inst,
@@ -596,16 +617,28 @@ impl<'a> Context<'a> {
 
             if let Affinity::Reg(rci) = lv.affinity {
                 let rc = self.reginfo.rc(rci);
+                let reg = loc.unwrap_reg();
+
+                debug_assert!(
+                    !self.is_pinned_reg(rc, reg)
+                        || self.cur.func.dfg[inst].opcode() == Opcode::GetPinnedReg,
+                    "pinned register may not be part of outputs for '{}'.",
+                    self.cur.func.dfg[inst].opcode()
+                );
+
+                if self.is_pinned_reg(rc, reg) {
+                    continue;
+                }
 
                 // Remove the dead defs.
                 if lv.endpoint == inst {
-                    regs.input.free(rc, loc.unwrap_reg());
+                    regs.input.free(rc, reg);
                     debug_assert!(lv.is_local);
                 }
 
                 // Track globals in their undiverted locations.
                 if !lv.is_local && !replace_global_defines {
-                    regs.global.take(rc, loc.unwrap_reg());
+                    regs.global.take(rc, reg);
                 }
             }
         }
@@ -626,14 +659,35 @@ impl<'a> Context<'a> {
             // already in a register.
             let cur_reg = self.divert.reg(value, &self.cur.func.locations);
             match op.kind {
-                ConstraintKind::FixedReg(regunit) | ConstraintKind::FixedTied(regunit) => {
+                ConstraintKind::FixedReg(regunit) => {
                     // Add the fixed constraint even if `cur_reg == regunit`.
                     // It is possible that we will want to convert the value to a variable later,
                     // and this identity assignment prevents that from happening.
                     self.solver
                         .reassign_in(value, op.regclass, cur_reg, regunit);
                 }
-                ConstraintKind::Reg | ConstraintKind::Tied(_) => {
+                ConstraintKind::FixedTied(regunit) => {
+                    // The pinned register may not be part of a fixed tied requirement. If this
+                    // becomes the case, then it must be changed to a different register.
+                    debug_assert!(
+                        !self.is_pinned_reg(op.regclass, regunit),
+                        "see comment above"
+                    );
+                    // See comment right above.
+                    self.solver
+                        .reassign_in(value, op.regclass, cur_reg, regunit);
+                }
+                ConstraintKind::Tied(_) => {
+                    if self.is_pinned_reg(op.regclass, cur_reg) {
+                        // Divert the pinned register; it shouldn't be reused for a tied input.
+                        if self.solver.can_add_var(op.regclass, cur_reg) {
+                            self.solver.add_var(value, op.regclass, cur_reg);
+                        }
+                    } else if !op.regclass.contains(cur_reg) {
+                        self.solver.add_var(value, op.regclass, cur_reg);
+                    }
+                }
+                ConstraintKind::Reg => {
                     if !op.regclass.contains(cur_reg) {
                         self.solver.add_var(value, op.regclass, cur_reg);
                     }
@@ -664,10 +718,13 @@ impl<'a> Context<'a> {
             match op.kind {
                 ConstraintKind::Reg | ConstraintKind::Tied(_) => {
                     let cur_reg = self.divert.reg(value, &self.cur.func.locations);
-                    // This is the opposite condition of `program_input_constraints()`.
-                    if op.regclass.contains(cur_reg) {
+
+                    // This is the opposite condition of `program_input_constraints()`. The pinned
+                    // register mustn't be added back as a variable.
+                    if op.regclass.contains(cur_reg) && !self.is_pinned_reg(op.regclass, cur_reg) {
                         // This code runs after calling `solver.inputs_done()` so we must identify
-                        // the new variable as killed or live-through.
+                        // the new variable as killed or live-through. Always special-case the
+                        // pinned register as a through variable.
                         let ctx = self.liveness.context(&self.cur.func.layout);
                         if self.liveness[value].killed_at(inst, ctx.order.pp_ebb(inst), ctx) {
                             self.solver.add_killed_var(value, op.regclass, cur_reg);
@@ -778,7 +835,7 @@ impl<'a> Context<'a> {
             if pred(lr, self.liveness.context(&self.cur.func.layout)) {
                 if let Affinity::Reg(rci) = lr.affinity {
                     let rc = self.reginfo.rc(rci);
-                    // Stack diversions should not be possible here. The only live transiently
+                    // Stack diversions should not be possible here. They only live transiently
                     // during `shuffle_inputs()`.
                     self.solver.reassign_in(
                         value,
@@ -797,8 +854,8 @@ impl<'a> Context<'a> {
         }
     }
 
-    // Find existing live values that conflict with the fixed input register constraints programmed
-    // into the constraint solver. Convert them to solver variables so they can be diverted.
+    /// Find existing live values that conflict with the fixed input register constraints programmed
+    /// into the constraint solver. Convert them to solver variables so they can be diverted.
     fn divert_fixed_input_conflicts(&mut self, live: &[LiveValue]) {
         for lv in live {
             if let Affinity::Reg(rci) = lv.affinity {
@@ -886,7 +943,9 @@ impl<'a> Context<'a> {
         reg: RegUnit,
         throughs: &[LiveValue],
     ) {
-        if !self.solver.add_fixed_output(rc, reg) {
+        // Pinned register is already unavailable in the solver, since it is copied in the
+        // available registers on entry.
+        if !self.is_pinned_reg(rc, reg) && !self.solver.add_fixed_output(rc, reg) {
             // The fixed output conflicts with some of the live-through registers.
             for lv in throughs {
                 if let Affinity::Reg(rci) = lv.affinity {
@@ -929,12 +988,12 @@ impl<'a> Context<'a> {
                     // Find the input operand we're tied to.
                     // The solver doesn't care about the output value.
                     let arg = self.cur.func.dfg.inst_args(inst)[num as usize];
-                    if let Some(reg) = self.solver.add_tied_input(
-                        arg,
-                        op.regclass,
-                        self.divert.reg(arg, &self.cur.func.locations),
-                        !lv.is_local,
-                    ) {
+                    let reg = self.divert.reg(arg, &self.cur.func.locations);
+
+                    if let Some(reg) =
+                        self.solver
+                            .add_tied_input(arg, op.regclass, reg, !lv.is_local)
+                    {
                         // The value we're tied to has been assigned to a fixed register.
                         // We need to make sure that fixed output register is compatible with the
                         // global register set.
@@ -1000,7 +1059,7 @@ impl<'a> Context<'a> {
                 let toprc2 = self.reginfo.toprc(rci);
                 let reg2 = self.divert.reg(lv.value, &self.cur.func.locations);
                 if rc.contains(reg2)
-                    && self.solver.can_add_var(lv.value, toprc2, reg2)
+                    && self.solver.can_add_var(toprc2, reg2)
                     && !self.is_live_on_outgoing_edge(lv.value)
                 {
                     self.solver.add_through_var(lv.value, toprc2, reg2);
@@ -1061,8 +1120,15 @@ impl<'a> Context<'a> {
         for m in self.solver.moves() {
             match *m {
                 Reg {
-                    value, from, to, ..
+                    value,
+                    from,
+                    to,
+                    rc,
                 } => {
+                    debug_assert!(
+                        !self.is_pinned_reg(rc, to),
+                        "pinned register used in a regmove"
+                    );
                     self.divert.regmove(value, from, to);
                     self.cur.ins().regmove(value, from, to);
                 }
@@ -1086,8 +1152,12 @@ impl<'a> Context<'a> {
                     value,
                     from_slot,
                     to,
-                    ..
+                    rc,
                 } => {
+                    debug_assert!(
+                        !self.is_pinned_reg(rc, to),
+                        "pinned register used in a regfill"
+                    );
                     // These slots are single use, so mark `ss` as available again.
                     let ss = slot[from_slot].take().expect("Using unallocated slot");
                     self.divert.regfill(value, ss, to);
diff --git a/cranelift/codegen/src/regalloc/register_set.rs b/cranelift/codegen/src/regalloc/register_set.rs
index fb7f208c59..e5edaa96d6 100644
--- a/cranelift/codegen/src/regalloc/register_set.rs
+++ b/cranelift/codegen/src/regalloc/register_set.rs
@@ -268,6 +268,7 @@ mod tests {
         subclasses: 0,
         mask: [0xf0000000, 0x0000000f, 0],
         info: &INFO,
+        pinned_reg: None,
     };
 
     const DPR: RegClass = &RegClassData {
@@ -280,6 +281,7 @@ mod tests {
         subclasses: 0,
         mask: [0x50000000, 0x0000000a, 0],
         info: &INFO,
+        pinned_reg: None,
     };
 
     const INFO: RegInfo = RegInfo {
diff --git a/cranelift/codegen/src/regalloc/solver.rs b/cranelift/codegen/src/regalloc/solver.rs
index ec517028cc..35e17b050d 100644
--- a/cranelift/codegen/src/regalloc/solver.rs
+++ b/cranelift/codegen/src/regalloc/solver.rs
@@ -877,6 +877,7 @@ impl Solver {
             let d = v.iter(&self.regs_in, &self.regs_out, global_regs).len();
             v.domain = cmp::min(d, u16::MAX as usize) as u16;
         }
+
         // Solve for vars with small domains first to increase the chance of finding a solution.
         //
         // Also consider this case:
@@ -949,9 +950,8 @@ impl Solver {
                     // live registers be diverted. We need to make it a non-global value.
                     if v.is_global && gregs.iter(rc).next().is_none() {
                         return Err(SolverError::Global(v.value));
-                    } else {
-                        return Err(SolverError::Divert(rc));
                     }
+                    return Err(SolverError::Divert(rc));
                 }
             };
 
@@ -976,7 +976,7 @@ impl Solver {
     }
 
     /// Check if `value` can be added as a variable to help find a solution.
-    pub fn can_add_var(&mut self, _value: Value, constraint: RegClass, from: RegUnit) -> bool {
+    pub fn can_add_var(&mut self, constraint: RegClass, from: RegUnit) -> bool {
         !self.regs_in.is_avail(constraint, from)
     }
 }
@@ -1030,7 +1030,7 @@ impl Solver {
         let mut avail = regs.clone();
         let mut i = 0;
         while i < self.moves.len() + self.fills.len() {
-            // Don't even look at the fills until we've spent all the moves. Deferring these let's
+            // Don't even look at the fills until we've spent all the moves. Deferring these lets
             // us potentially reuse the claimed registers to resolve multiple cycles.
             if i >= self.moves.len() {
                 self.moves.append(&mut self.fills);
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index f9bc9f6d1a..c22eb3f9a4 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -388,6 +388,7 @@ mod tests {
              avoid_div_traps = false\n\
              enable_float = true\n\
              enable_nan_canonicalization = false\n\
+             enable_pinned_reg = false\n\
              enable_simd = false\n\
              enable_atomics = true\n\
              enable_safepoints = false\n\
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 72fa1f7e17..5bc10d9d92 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -676,6 +676,26 @@ impl<'a> Verifier<'a> {
                 self.verify_value_list(inst, args, errors)?;
             }
 
+            NullAry {
+                opcode: Opcode::GetPinnedReg,
+            }
+            | Unary {
+                opcode: Opcode::SetPinnedReg,
+                ..
+            } => {
+                if let Some(isa) = &self.isa {
+                    if !isa.flags().enable_pinned_reg() {
+                        return fatal!(
+                            errors,
+                            inst,
+                            "GetPinnedReg/SetPinnedReg cannot be used without enable_pinned_reg"
+                        );
+                    }
+                } else {
+                    return fatal!(errors, inst, "GetPinnedReg/SetPinnedReg need an ISA!");
+                }
+            }
+
             // Exhaustive list so we can't forget to add new formats
             Unary { .. }
             | UnaryImm { .. }

From c1609b70e8f2b862d0487172d24604b11b0db0c4 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 28 Aug 2019 17:56:26 +0200
Subject: [PATCH 2649/3084] [codegen] Allow using the pinned register as the
 heap base via a setting;

---
 cranelift/codegen/meta/src/shared/settings.rs | 15 +++++++++
 cranelift/codegen/src/legalizer/heap.rs       | 33 ++++++++++++++-----
 cranelift/codegen/src/settings.rs             |  1 +
 3 files changed, 40 insertions(+), 9 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
index e7c567c345..63c5d3c01a 100644
--- a/cranelift/codegen/meta/src/shared/settings.rs
+++ b/cranelift/codegen/meta/src/shared/settings.rs
@@ -95,6 +95,21 @@ pub fn define() -> SettingGroup {
         false,
     );
 
+    settings.add_bool(
+        "use_pinned_reg_as_heap_base",
+        r#"Use the pinned register as the heap base.
+
+        Enabling this requires the enable_pinned_reg setting to be set to true. It enables a custom
+        legalization of the `heap_addr` instruction so it will use the pinned register as the heap
+        base, instead of fetching it from a global value.
+
+        Warning! Enabling this means that the pinned register *must* be maintained to contain the
+        heap base address at all times, during the lifetime of a function. Using the pinned
+        register for other purposes when this is set is very likely to cause crashes.
+        "#,
+        false,
+    );
+
     settings.add_bool("enable_simd", "Enable the use of SIMD instructions.", false);
 
     settings.add_bool(
diff --git a/cranelift/codegen/src/legalizer/heap.rs b/cranelift/codegen/src/legalizer/heap.rs
index 33f37155ee..dfada10dc5 100644
--- a/cranelift/codegen/src/legalizer/heap.rs
+++ b/cranelift/codegen/src/legalizer/heap.rs
@@ -14,7 +14,7 @@ pub fn expand_heap_addr(
     inst: ir::Inst,
     func: &mut ir::Function,
     cfg: &mut ControlFlowGraph,
-    _isa: &dyn TargetIsa,
+    isa: &dyn TargetIsa,
 ) {
     // Unpack the instruction.
     let (heap, offset, access_size) = match func.dfg[inst] {
@@ -32,16 +32,24 @@ pub fn expand_heap_addr(
 
     match func.heaps[heap].style {
         ir::HeapStyle::Dynamic { bound_gv } => {
-            dynamic_addr(inst, heap, offset, access_size, bound_gv, func)
-        }
-        ir::HeapStyle::Static { bound } => {
-            static_addr(inst, heap, offset, access_size, bound.into(), func, cfg)
+            dynamic_addr(isa, inst, heap, offset, access_size, bound_gv, func)
         }
+        ir::HeapStyle::Static { bound } => static_addr(
+            isa,
+            inst,
+            heap,
+            offset,
+            access_size,
+            bound.into(),
+            func,
+            cfg,
+        ),
     }
 }
 
 /// Expand a `heap_addr` for a dynamic heap.
 fn dynamic_addr(
+    isa: &dyn TargetIsa,
     inst: ir::Inst,
     heap: ir::Heap,
     offset: ir::Value,
@@ -82,11 +90,12 @@ fn dynamic_addr(
     }
     pos.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds);
 
-    compute_addr(inst, heap, addr_ty, offset, offset_ty, pos.func);
+    compute_addr(isa, inst, heap, addr_ty, offset, offset_ty, pos.func);
 }
 
 /// Expand a `heap_addr` for a static heap.
 fn static_addr(
+    isa: &dyn TargetIsa,
     inst: ir::Inst,
     heap: ir::Heap,
     offset: ir::Value,
@@ -134,11 +143,12 @@ fn static_addr(
         pos.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds);
     }
 
-    compute_addr(inst, heap, addr_ty, offset, offset_ty, pos.func);
+    compute_addr(isa, inst, heap, addr_ty, offset, offset_ty, pos.func);
 }
 
 /// Emit code for the base address computation of a `heap_addr` instruction.
 fn compute_addr(
+    isa: &dyn TargetIsa,
     inst: ir::Inst,
     heap: ir::Heap,
     addr_ty: ir::Type,
@@ -165,7 +175,12 @@ fn compute_addr(
     }
 
     // Add the heap base address base
-    let base_gv = pos.func.heaps[heap].base;
-    let base = pos.ins().global_value(addr_ty, base_gv);
+    let base = if isa.flags().enable_pinned_reg() && isa.flags().use_pinned_reg_as_heap_base() {
+        pos.ins().get_pinned_reg(isa.pointer_type())
+    } else {
+        let base_gv = pos.func.heaps[heap].base;
+        pos.ins().global_value(addr_ty, base_gv)
+    };
+
     pos.func.dfg.replace(inst).iadd(base, offset);
 }
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index c22eb3f9a4..99d3647cbd 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -389,6 +389,7 @@ mod tests {
              enable_float = true\n\
              enable_nan_canonicalization = false\n\
              enable_pinned_reg = false\n\
+             use_pinned_reg_as_heap_base = false\n\
              enable_simd = false\n\
              enable_atomics = true\n\
              enable_safepoints = false\n\

From dca2e7e9a754806c882bf4961856e6f83797ebeb Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 2 Sep 2019 17:34:59 +0200
Subject: [PATCH 2650/3084] Add basic test for the pinned register;

---
 .../filetests/isa/x86/pinned-reg.clif         | 74 +++++++++++++++++++
 1 file changed, 74 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/pinned-reg.clif

diff --git a/cranelift/filetests/filetests/isa/x86/pinned-reg.clif b/cranelift/filetests/filetests/isa/x86/pinned-reg.clif
new file mode 100644
index 0000000000..b8a16d4eb4
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/pinned-reg.clif
@@ -0,0 +1,74 @@
+test compile
+
+set enable_pinned_reg=true
+set use_pinned_reg_as_heap_base=true
+set opt_level=best
+
+target x86_64
+
+; regex: V=v\d+
+
+; r15 is the pinned heap register. It must not be rewritten, so it must not be
+; used as a tied output register.
+function %tied_input() -> i64 system_v {
+ebb0:
+    v1 = get_pinned_reg.i64
+    v2 = iadd_imm v1, 42
+    return v2
+}
+
+; check: ,%r15]
+; sameln: v1 = get_pinned_reg.i64
+; nextln: regmove v1, %r15 -> %rax
+; nextln: ,%rax]
+; sameln: iadd_imm v1, 42
+
+;; It musn't be used even if this is a tied input used twice.
+function %tied_twice() -> i64 system_v {
+ebb0:
+    v1 = get_pinned_reg.i64
+    v2 = iadd v1, v1
+    return v2
+}
+
+; check: ,%r15]
+; sameln: v1 = get_pinned_reg.i64
+; nextln: regmove v1, %r15 -> %rax
+; nextln: ,%rax]
+; sameln: iadd v1, v1
+
+function %uses() -> i64 system_v {
+ebb0:
+    v1 = get_pinned_reg.i64
+    v2 = iadd_imm v1, 42
+    v3 = get_pinned_reg.i64
+    v4 = iadd v2, v3
+    return v4
+}
+
+; check: ,%r15]
+; sameln: v1 = get_pinned_reg.i64
+; nextln: regmove v1, %r15 -> %rax
+; nextln: ,%rax]
+; sameln: iadd_imm v1, 42
+; nextln: ,%r15
+; sameln: v3 = get_pinned_reg.i64
+; nextln: ,%rax]
+; sameln: iadd v2, v3
+
+; When the pinned register is used as the heap base, the final load instruction
+; must use the %r15 register, since x86 implements the complex addressing mode.
+function u0:1(i64 vmctx) -> i64 system_v {
+    gv0 = vmctx
+    heap0 = static gv0, min 0x000a_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000, index_type i32
+
+ebb0(v42: i64):
+    v5 = iconst.i32 42
+    v6 = heap_addr.i64 heap0, v5, 0
+    v7 = load.i64 v6
+    return v7
+}
+
+; check: ,%r15]
+; sameln: $(heap_base=$V) = get_pinned_reg.i64
+; nextln: load_complex.i64 $heap_base+

From e35cf861db889b3b0bfe96c4de089c2bbd7c3e8b Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 6 Sep 2019 11:54:17 +0200
Subject: [PATCH 2651/3084] Fixes #984: Add a isa::lookup_by_name function;

This removes the explicit dependency on target-lexicon for the embedder,
which can instead use the ISA's name directly. It can simplify
dependency management, in particular avoid the need for synchronizing
the target-lexicon dependencies versions.

It also tweak the error when an ISA isn't built as part of Cranelift to
be a SupportDisabled error; this was dead code before this.
---
 cranelift/codegen/src/isa/mod.rs | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index 816e185f81..20e38ddeb4 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -66,7 +66,7 @@ use crate::timing;
 use core::fmt;
 use failure_derive::Fail;
 use std::boxed::Box;
-use target_lexicon::{Architecture, PointerWidth, Triple};
+use target_lexicon::{triple, Architecture, PointerWidth, Triple};
 
 #[cfg(feature = "riscv")]
 mod riscv;
@@ -97,13 +97,13 @@ macro_rules! isa_builder {
         };
         #[cfg(not(feature = $feature))]
         fn $name(_triple: Triple) -> Result {
-            Err(LookupError::Unsupported)
+            Err(LookupError::SupportDisabled)
         }
         $name
     }};
 }
 
-/// Look for a supported ISA with the given `name`.
+/// Look for an ISA for the given `triple`.
 /// Return a builder that can create a corresponding `TargetIsa`.
 pub fn lookup(triple: Triple) -> Result {
     match triple.architecture {
@@ -117,6 +117,13 @@ pub fn lookup(triple: Triple) -> Result {
     }
 }
 
+/// Look for a supported ISA with the given `name`.
+/// Return a builder that can create a corresponding `TargetIsa`.
+pub fn lookup_by_name(name: &str) -> Result {
+    use std::str::FromStr;
+    lookup(triple!(name))
+}
+
 /// Describes reason for target lookup failure
 #[derive(Fail, PartialEq, Eq, Copy, Clone, Debug)]
 pub enum LookupError {

From 891944dba17763a8f593e94cf4b306c220d6e464 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 6 Sep 2019 22:07:06 +0200
Subject: [PATCH 2652/3084] Generate basic-blocks instead of Ebb in
 frontend::switch. (#981)

---
 cranelift/frontend/src/switch.rs | 34 +++++++++++++++++++++++++++-----
 1 file changed, 29 insertions(+), 5 deletions(-)

diff --git a/cranelift/frontend/src/switch.rs b/cranelift/frontend/src/switch.rs
index 0db3ac348e..0b1ca746cb 100644
--- a/cranelift/frontend/src/switch.rs
+++ b/cranelift/frontend/src/switch.rs
@@ -167,12 +167,22 @@ impl Switch {
         contiguous_case_ranges: Vec,
         cases_and_jt_ebbs: &mut Vec<(EntryIndex, Ebb, Vec)>,
     ) {
+        let mut was_branch = false;
+        let ins_fallthrough_jump = |was_branch: bool, bx: &mut FunctionBuilder| {
+            if was_branch {
+                let ebb = bx.create_ebb();
+                bx.ins().jump(ebb, &[]);
+                bx.switch_to_block(ebb);
+            }
+        };
         for ContiguousCaseRange { first_index, ebbs } in contiguous_case_ranges.into_iter().rev() {
             match (ebbs.len(), first_index) {
                 (1, 0) => {
+                    ins_fallthrough_jump(was_branch, bx);
                     bx.ins().brz(val, ebbs[0], &[]);
                 }
                 (1, _) => {
+                    ins_fallthrough_jump(was_branch, bx);
                     let is_good_val = bx.ins().icmp_imm(IntCC::Equal, val, first_index as i64);
                     bx.ins().brnz(is_good_val, ebbs[0], &[]);
                 }
@@ -187,6 +197,7 @@ impl Switch {
                     return;
                 }
                 (_, _) => {
+                    ins_fallthrough_jump(was_branch, bx);
                     let jt_ebb = bx.create_ebb();
                     let is_good_val = bx.ins().icmp_imm(
                         IntCC::UnsignedGreaterThanOrEqual,
@@ -197,6 +208,7 @@ impl Switch {
                     cases_and_jt_ebbs.push((first_index, jt_ebb, ebbs));
                 }
             }
+            was_branch = true;
         }
 
         bx.ins().jump(otherwise, &[]);
@@ -359,7 +371,10 @@ ebb3:
     v1 = uextend.i32 v0
     v2 = icmp_imm eq v1, 2
     brnz v2, ebb2
-    brz v1, ebb1
+    jump ebb3
+
+ebb3:
+    brz.i32 v1, ebb1
     jump ebb0"
         );
     }
@@ -382,6 +397,9 @@ ebb0:
 ebb9:
     v3 = icmp_imm.i32 uge v1, 10
     brnz v3, ebb10
+    jump ebb11
+
+ebb11:
     v4 = icmp_imm.i32 eq v1, 7
     brnz v4, ebb4
     jump ebb0
@@ -389,9 +407,9 @@ ebb9:
 ebb8:
     v5 = icmp_imm.i32 eq v1, 5
     brnz v5, ebb3
-    jump ebb11
+    jump ebb12
 
-ebb11:
+ebb12:
     br_table.i32 v1, ebb0, jt0
 
 ebb10:
@@ -410,7 +428,10 @@ ebb10:
     v1 = uextend.i32 v0
     v2 = icmp_imm eq v1, 0x8000_0000_0000_0000
     brnz v2, ebb1
-    v3 = icmp_imm eq v1, 1
+    jump ebb3
+
+ebb3:
+    v3 = icmp_imm.i32 eq v1, 1
     brnz v3, ebb2
     jump ebb0"
         );
@@ -426,7 +447,10 @@ ebb10:
     v1 = uextend.i32 v0
     v2 = icmp_imm eq v1, 0x7fff_ffff_ffff_ffff
     brnz v2, ebb1
-    v3 = icmp_imm eq v1, 1
+    jump ebb3
+
+ebb3:
+    v3 = icmp_imm.i32 eq v1, 1
     brnz v3, ebb2
     jump ebb0"
         )

From 592f5445dd8ed19586b3ffbeb072720243f6cadc Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 6 Sep 2019 22:26:17 +0200
Subject: [PATCH 2653/3084] Fix legalize-br-table test case for basic blocks.
 (#990)

---
 .../isa/x86/legalize-br-table-bb.clif         | 31 +++++++++++++++++++
 .../filetests/isa/x86/legalize-br-table.clif  |  1 -
 2 files changed, 31 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/legalize-br-table-bb.clif

diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-table-bb.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-table-bb.clif
new file mode 100644
index 0000000000..69e4240764
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-table-bb.clif
@@ -0,0 +1,31 @@
+test compile
+target x86_64
+feature "basic-blocks"
+; regex: V=v\d+
+; regex: EBB=ebb\d+
+
+function u0:0(i64) system_v {
+    ss0 = explicit_slot 1
+    jt0 = jump_table [ebb1]
+
+ebb0(v0: i64):
+    v1 = stack_addr.i64 ss0
+    v2 = load.i8 v1
+    br_table v2, ebb2, jt0
+; check:     $(oob=$V) = ifcmp_imm $(idx=$V), 1
+; ebb2 is replaced by ebb1 by fold_redundant_jump
+; nextln:    brif uge $oob, ebb1
+; nextln:    fallthrough $(inb=$EBB)
+; check:   $inb:
+; nextln:    $(final_idx=$V) = uextend.i64 $idx
+; nextln:    $(base=$V) = jump_table_base.i64 jt0
+; nextln:    $(rel_addr=$V) = jump_table_entry $final_idx, $base, 4, jt0
+; nextln:    $(addr=$V) = iadd $base, $rel_addr
+; nextln:    indirect_jump_table_br $addr, jt0
+
+ebb2:
+    jump ebb1
+
+ebb1:
+    return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
index e970136ac8..63c24b0240 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
@@ -1,5 +1,4 @@
 test compile
-
 target x86_64
 feature !"basic-blocks"
 ; regex: V=v\d+

From 1c5711c12bc0ae250a20f4ae84d35373f021e280 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 6 Sep 2019 22:27:02 +0200
Subject: [PATCH 2654/3084] Basic Block: Fix IR builder library example. (#989)

---
 cranelift/frontend/src/lib.rs | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/cranelift/frontend/src/lib.rs b/cranelift/frontend/src/lib.rs
index 5b426d608f..eff5c655fd 100644
--- a/cranelift/frontend/src/lib.rs
+++ b/cranelift/frontend/src/lib.rs
@@ -50,10 +50,12 @@
 //!    jump block1
 //! block1:
 //!    z = z + y;
-//!    brnz y, block2;
+//!    brnz y, block3;
+//!    jump block2
+//! block2:
 //!    z = z - x;
 //!    return y
-//! block2:
+//! block3:
 //!    y = y - x
 //!    jump block1
 //! }
@@ -85,6 +87,7 @@
 //!         let block0 = builder.create_ebb();
 //!         let block1 = builder.create_ebb();
 //!         let block2 = builder.create_ebb();
+//!         let block3 = builder.create_ebb();
 //!         let x = Variable::new(0);
 //!         let y = Variable::new(1);
 //!         let z = Variable::new(2);
@@ -120,8 +123,12 @@
 //!         }
 //!         {
 //!             let arg = builder.use_var(y);
-//!             builder.ins().brnz(arg, block2, &[]);
+//!             builder.ins().brnz(arg, block3, &[]);
 //!         }
+//!         builder.ins().jump(block2, &[]);
+//!
+//!         builder.switch_to_block(block2);
+//!         builder.seal_block(block2);
 //!         {
 //!             let arg1 = builder.use_var(z);
 //!             let arg2 = builder.use_var(x);
@@ -133,8 +140,8 @@
 //!             builder.ins().return_(&[arg]);
 //!         }
 //!
-//!         builder.switch_to_block(block2);
-//!         builder.seal_block(block2);
+//!         builder.switch_to_block(block3);
+//!         builder.seal_block(block3);
 //!
 //!         {
 //!             let arg1 = builder.use_var(y);

From 3b0e244316028b367bc3f01c06eac7bf9104f8b3 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 6 Sep 2019 22:29:04 +0200
Subject: [PATCH 2655/3084] Fix binary64.clif to work with fold_redundant_jump.
 (#987)

---
 .../filetests/filetests/isa/x86/binary64.clif      | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/cranelift/filetests/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
index d1b862c85a..9b1f86beff 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64.clif
@@ -746,11 +746,14 @@ ebb7:
 
     ; asm: ebb1:
 ebb1:
-    return                                       ; bin: c3
+    return                                      ; bin: c3
 
     ; asm: ebb2:
 ebb2:
-    jump ebb1                                   ; bin: eb fd
+    ; Add a no-op instruction to prevent fold_redundant_jump from removing this block.
+    ; asm: notq %rcx
+    [-,%rcx]             v5000 = bnot v1        ; bin: 48 f7 d1
+    jump ebb1                                   ; bin: eb fa
 }
 
 ; CPU flag instructions.
@@ -1349,11 +1352,14 @@ ebb7:
 
     ; asm: ebb1x:
 ebb1:
-    return                                       ; bin: c3
+    return                                      ; bin: c3
 
     ; asm: ebb2x:
 ebb2:
-    jump ebb1                                   ; bin: eb fd
+    ; Add a no-op instruction to prevent fold_redundant_jump from removing this block.
+    ; asm: notl %ecx
+    [-,%rcx]             v5000 = bnot v1        ; bin: f7 d1
+    jump ebb1                                   ; bin: eb fb
 
 }
 

From c9a25abbc4a79a111fa208868ef068a5cc74e828 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 12 Jun 2019 19:24:47 +0200
Subject: [PATCH 2656/3084] Basic i128 support

---
 cranelift/codegen/meta/src/cdsl/types.rs          |  6 ++++--
 cranelift/codegen/meta/src/cdsl/typevar.rs        |  2 +-
 cranelift/codegen/meta/src/shared/instructions.rs |  6 +++---
 cranelift/codegen/meta/src/shared/types.rs        |  4 ++++
 cranelift/codegen/src/ir/types.rs                 | 14 ++++++++++++--
 cranelift/codegen/src/regalloc/reload.rs          |  2 +-
 cranelift/reader/src/lexer.rs                     |  1 +
 7 files changed, 26 insertions(+), 9 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
index eba239d1d7..53b0032803 100644
--- a/cranelift/codegen/meta/src/cdsl/types.rs
+++ b/cranelift/codegen/meta/src/cdsl/types.rs
@@ -219,8 +219,9 @@ impl LaneType {
                 LaneType::IntType(shared_types::Int::I16) => 6,
                 LaneType::IntType(shared_types::Int::I32) => 7,
                 LaneType::IntType(shared_types::Int::I64) => 8,
-                LaneType::FloatType(shared_types::Float::F32) => 9,
-                LaneType::FloatType(shared_types::Float::F64) => 10,
+                LaneType::IntType(shared_types::Int::I128) => 9,
+                LaneType::FloatType(shared_types::Float::F32) => 10,
+                LaneType::FloatType(shared_types::Float::F64) => 11,
             }
     }
 
@@ -241,6 +242,7 @@ impl LaneType {
             16 => shared_types::Int::I16,
             32 => shared_types::Int::I32,
             64 => shared_types::Int::I64,
+            128 => shared_types::Int::I128,
             _ => unreachable!("unxpected num bits for int"),
         })
     }
diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 9ae4c33fd8..27ec365227 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -9,7 +9,7 @@ use std::rc::Rc;
 use crate::cdsl::types::{BVType, LaneType, ReferenceType, SpecialType, ValueType};
 
 const MAX_LANES: u16 = 256;
-const MAX_BITS: u16 = 64;
+const MAX_BITS: u16 = 128;
 const MAX_BITVEC: u16 = MAX_BITS * MAX_LANES;
 
 /// Type variables can be used in place of concrete types when defining
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 67e2891068..843347ce95 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -3143,7 +3143,7 @@ pub(crate) fn define(
         "WideInt",
         "An integer type with lanes from `i16` upwards",
         TypeSetBuilder::new()
-            .ints(16..64)
+            .ints(16..128)
             .simd_lanes(Interval::All)
             .build(),
     );
@@ -3171,9 +3171,9 @@ pub(crate) fn define(
 
     let NarrowInt = &TypeVar::new(
         "NarrowInt",
-        "An integer type with lanes type to `i32`",
+        "An integer type with lanes type to `i64`",
         TypeSetBuilder::new()
-            .ints(8..32)
+            .ints(8..64)
             .simd_lanes(Interval::All)
             .build(),
     );
diff --git a/cranelift/codegen/meta/src/shared/types.rs b/cranelift/codegen/meta/src/shared/types.rs
index 266c30b3d8..60c546657e 100644
--- a/cranelift/codegen/meta/src/shared/types.rs
+++ b/cranelift/codegen/meta/src/shared/types.rs
@@ -51,6 +51,8 @@ pub enum Int {
     I32 = 32,
     /// 64-bit int.
     I64 = 64,
+    /// 128-bit int.
+    I128 = 128,
 }
 
 /// This provides an iterator through all of the supported int variants.
@@ -72,6 +74,7 @@ impl Iterator for IntIterator {
             1 => Some(Int::I16),
             2 => Some(Int::I32),
             3 => Some(Int::I64),
+            4 => Some(Int::I128),
             _ => return None,
         };
         self.index += 1;
@@ -199,6 +202,7 @@ mod iter_tests {
         assert_eq!(int_iter.next(), Some(Int::I16));
         assert_eq!(int_iter.next(), Some(Int::I32));
         assert_eq!(int_iter.next(), Some(Int::I64));
+        assert_eq!(int_iter.next(), Some(Int::I128));
         assert_eq!(int_iter.next(), None);
     }
 
diff --git a/cranelift/codegen/src/ir/types.rs b/cranelift/codegen/src/ir/types.rs
index 4eb72f3fc0..a5c1a99978 100644
--- a/cranelift/codegen/src/ir/types.rs
+++ b/cranelift/codegen/src/ir/types.rs
@@ -63,6 +63,7 @@ impl Type {
             B16 | I16 => 4,
             B32 | I32 | F32 | R32 => 5,
             B64 | I64 | F64 | R64 => 6,
+            I128 => 7,
             _ => 0,
         }
     }
@@ -75,6 +76,7 @@ impl Type {
             B16 | I16 => 16,
             B32 | I32 | F32 | R32 => 32,
             B64 | I64 | F64 | R64 => 64,
+            I128 => 128,
             _ => 0,
         }
     }
@@ -86,6 +88,7 @@ impl Type {
             16 => Some(I16),
             32 => Some(I32),
             64 => Some(I64),
+            128 => Some(I128),
             _ => None,
         }
     }
@@ -132,6 +135,7 @@ impl Type {
             I16 => I8,
             I32 => I16,
             I64 => I32,
+            I128 => I64,
             F64 => F32,
             B16 => B8,
             B32 => B16,
@@ -147,6 +151,7 @@ impl Type {
             I8 => I16,
             I16 => I32,
             I32 => I64,
+            I64 => I128,
             F32 => F64,
             B8 => B16,
             B16 => B32,
@@ -190,7 +195,7 @@ impl Type {
     /// Is this a scalar integer type?
     pub fn is_int(self) -> bool {
         match self {
-            I8 | I16 | I32 | I64 => true,
+            I8 | I16 | I32 | I64 | I128 => true,
             _ => false,
         }
     }
@@ -374,6 +379,7 @@ mod tests {
         assert_eq!(I16, I16.lane_type());
         assert_eq!(I32, I32.lane_type());
         assert_eq!(I64, I64.lane_type());
+        assert_eq!(I128, I128.lane_type());
         assert_eq!(F32, F32.lane_type());
         assert_eq!(F64, F64.lane_type());
         assert_eq!(B1, B1.by(8).unwrap().lane_type());
@@ -394,6 +400,7 @@ mod tests {
         assert_eq!(I16.lane_bits(), 16);
         assert_eq!(I32.lane_bits(), 32);
         assert_eq!(I64.lane_bits(), 64);
+        assert_eq!(I128.lane_bits(), 128);
         assert_eq!(F32.lane_bits(), 32);
         assert_eq!(F64.lane_bits(), 64);
         assert_eq!(R32.lane_bits(), 32);
@@ -415,6 +422,7 @@ mod tests {
         assert_eq!(I32.half_width(), Some(I16));
         assert_eq!(I32X4.half_width(), Some(I16X4));
         assert_eq!(I64.half_width(), Some(I32));
+        assert_eq!(I128.half_width(), Some(I64));
         assert_eq!(F32.half_width(), None);
         assert_eq!(F64.half_width(), Some(F32));
 
@@ -430,7 +438,8 @@ mod tests {
         assert_eq!(I16.double_width(), Some(I32));
         assert_eq!(I32.double_width(), Some(I64));
         assert_eq!(I32X4.double_width(), Some(I64X4));
-        assert_eq!(I64.double_width(), None);
+        assert_eq!(I64.double_width(), Some(I128));
+        assert_eq!(I128.double_width(), None);
         assert_eq!(F32.double_width(), Some(F64));
         assert_eq!(F64.double_width(), None);
     }
@@ -465,6 +474,7 @@ mod tests {
         assert_eq!(I16.to_string(), "i16");
         assert_eq!(I32.to_string(), "i32");
         assert_eq!(I64.to_string(), "i64");
+        assert_eq!(I128.to_string(), "i128");
         assert_eq!(F32.to_string(), "f32");
         assert_eq!(F64.to_string(), "f64");
         assert_eq!(R32.to_string(), "r32");
diff --git a/cranelift/codegen/src/regalloc/reload.rs b/cranelift/codegen/src/regalloc/reload.rs
index fb6b61ec6f..bbc198c45d 100644
--- a/cranelift/codegen/src/regalloc/reload.rs
+++ b/cranelift/codegen/src/regalloc/reload.rs
@@ -233,7 +233,7 @@ impl<'a> Context<'a> {
                             let dst_ty = self.cur.func.dfg.value_type(dst_val);
                             debug_assert!(src_ty == dst_ty);
                             // This limits the transformation to copies of the
-                            // types: I64 I32 I16 I8 F64 and F32, since that's
+                            // types: I128 I64 I32 I16 I8 F64 and F32, since that's
                             // the set of `copy_nop` encodings available.
                             src_ty.is_int() || src_ty.is_float()
                         }
diff --git a/cranelift/reader/src/lexer.rs b/cranelift/reader/src/lexer.rs
index 465e79e97f..151fbd7af5 100644
--- a/cranelift/reader/src/lexer.rs
+++ b/cranelift/reader/src/lexer.rs
@@ -365,6 +365,7 @@ impl<'a> Lexer<'a> {
             "i16" => types::I16,
             "i32" => types::I32,
             "i64" => types::I64,
+            "i128" => types::I128,
             "f32" => types::F32,
             "f64" => types::F64,
             "b1" => types::B1,

From 4305fe37a02e3f53915c1cc76335ff5b50b46c44 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 12 Jun 2019 19:55:00 +0200
Subject: [PATCH 2657/3084] Add test

---
 cranelift/filetests/filetests/isa/x86/i128.clif | 13 +++++++++++++
 1 file changed, 13 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/i128.clif

diff --git a/cranelift/filetests/filetests/isa/x86/i128.clif b/cranelift/filetests/filetests/isa/x86/i128.clif
new file mode 100644
index 0000000000..71dcac9261
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/i128.clif
@@ -0,0 +1,13 @@
+test compile
+target x86_64
+
+function u0:0(i128) -> i128 fast {
+ebb0(v0: i128):
+    v1 = iconst.i64 0
+    v2 = iconst.i64 42
+    v3 = iconcat.i64 v1, v2
+    return v3
+    ; check: v1 = iconst.i64 0
+    ; check: v2 = iconst.i64 42
+    ; check: return v1, v2, v7
+}

From b7ec05557547691cc05fe1dfee85f8a3de8aba27 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 12 Jun 2019 20:06:38 +0200
Subject: [PATCH 2658/3084] Use little endian byte order in i128 test

---
 cranelift/filetests/filetests/isa/x86/i128.clif | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/filetests/filetests/isa/x86/i128.clif b/cranelift/filetests/filetests/isa/x86/i128.clif
index 71dcac9261..c344a48ee2 100644
--- a/cranelift/filetests/filetests/isa/x86/i128.clif
+++ b/cranelift/filetests/filetests/isa/x86/i128.clif
@@ -5,9 +5,9 @@ function u0:0(i128) -> i128 fast {
 ebb0(v0: i128):
     v1 = iconst.i64 0
     v2 = iconst.i64 42
-    v3 = iconcat.i64 v1, v2
+    v3 = iconcat.i64 v2, v1
     return v3
     ; check: v1 = iconst.i64 0
     ; check: v2 = iconst.i64 42
-    ; check: return v1, v2, v7
+    ; check: return v2, v1, v7
 }

From c1553194a7e8c153f555aaed146aacdfd138f735 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Mon, 17 Jun 2019 19:51:21 +0200
Subject: [PATCH 2659/3084] Fix WideInt max size in insturctions.py

---
 cranelift/codegen/src/ir/types.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/ir/types.rs b/cranelift/codegen/src/ir/types.rs
index a5c1a99978..394ec8e024 100644
--- a/cranelift/codegen/src/ir/types.rs
+++ b/cranelift/codegen/src/ir/types.rs
@@ -10,7 +10,7 @@ use target_lexicon::{PointerWidth, Triple};
 /// field is present put no type is needed, such as the controlling type variable for a
 /// non-polymorphic instruction.
 ///
-/// Basic integer types: `I8`, `I16`, `I32`, and `I64`. These types are sign-agnostic.
+/// Basic integer types: `I8`, `I16`, `I32`, `I64`, and `I128`. These types are sign-agnostic.
 ///
 /// Basic floating point types: `F32` and `F64`. IEEE single and double precision.
 ///

From 6f7d57a71f6dbbac4fa1a74ddeb41a1149440f23 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Mon, 17 Jun 2019 20:08:46 +0200
Subject: [PATCH 2660/3084] Handle isplit when it is not the result of a
 legalization

---
 cranelift/codegen/src/legalizer/mod.rs        | 25 ++++++++++++++
 .../filetests/filetests/isa/x86/i128.clif     | 33 ++++++++++++++-----
 2 files changed, 49 insertions(+), 9 deletions(-)

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index e6f7bcb00d..c46fb8d81f 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -57,6 +57,31 @@ fn legalize_inst(
         }
     } else if opcode.is_branch() {
         split::simplify_branch_arguments(&mut pos.func.dfg, inst);
+    } else if opcode == ir::Opcode::Isplit {
+        pos.use_srcloc(inst);
+
+        let arg = match pos.func.dfg[inst] {
+            ir::InstructionData::Unary {
+                arg,
+                ..
+            } => pos.func.dfg.resolve_aliases(arg),
+            _ => panic!("Expected isplit: {}", pos.func.dfg.display_inst(inst, None)),
+        };
+
+        let res = pos.func.dfg.inst_results(inst).to_vec();
+        assert_eq!(res.len(), 2);
+        let (resl, resh) = (res[0], res[1]); // Prevent borrowck error
+
+        let curpos = pos.position();
+        let srcloc = pos.srcloc();
+        let (xl, xh) = split::isplit(pos.func, cfg, curpos, srcloc, arg);
+
+        pos.func.dfg.clear_results(inst);
+        pos.remove_inst();
+        pos.func.dfg.change_to_alias(resl, xl);
+        pos.func.dfg.change_to_alias(resh, xh);
+
+        return true;
     }
 
     match pos.func.update_encoding(inst, isa) {
diff --git a/cranelift/filetests/filetests/isa/x86/i128.clif b/cranelift/filetests/filetests/isa/x86/i128.clif
index c344a48ee2..3c50d219dc 100644
--- a/cranelift/filetests/filetests/isa/x86/i128.clif
+++ b/cranelift/filetests/filetests/isa/x86/i128.clif
@@ -1,13 +1,28 @@
 test compile
 target x86_64
 
-function u0:0(i128) -> i128 fast {
-ebb0(v0: i128):
-    v1 = iconst.i64 0
-    v2 = iconst.i64 42
-    v3 = iconcat.i64 v2, v1
-    return v3
-    ; check: v1 = iconst.i64 0
-    ; check: v2 = iconst.i64 42
-    ; check: return v2, v1, v7
+function u0:0(i64, i64) -> i128 fast {
+ebb0(v0: i64, v1: i64):
+;check: ebb0(v0: i64 [%rdi], v1: i64 [%rsi], v3: i64 [%rbp]):
+
+    v2 = iconcat.i64 v0, v1
+    ; check: regmove v0, %rdi -> %rax
+    ; check: regmove v1, %rsi -> %rdx
+
+    return v2
+    ; check: v4 = x86_pop.i64
+    ; check: return v0, v1, v4
+}
+
+function u0:1(i128) -> i64, i64 fast {
+ebb0(v0: i128):
+; check: ebb0(v3: i64 [%rdi], v4: i64 [%rsi], v5: i64 [%rbp]):
+
+    v1, v2 = isplit v0
+    ; check: regmove v3, %rdi -> %rax
+    ; check: regmove v4, %rsi -> %rdx
+
+    return v1, v2
+    ; check: v6 = x86_pop.i64
+    ; check: return v3, v4, v6
 }

From f04b334b20a4b928482f337594a3ea177579274d Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Mon, 17 Jun 2019 20:26:37 +0200
Subject: [PATCH 2661/3084] Rustfmt

---
 cranelift/codegen/src/legalizer/mod.rs | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index c46fb8d81f..db9f302cae 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -61,10 +61,7 @@ fn legalize_inst(
         pos.use_srcloc(inst);
 
         let arg = match pos.func.dfg[inst] {
-            ir::InstructionData::Unary {
-                arg,
-                ..
-            } => pos.func.dfg.resolve_aliases(arg),
+            ir::InstructionData::Unary { arg, .. } => pos.func.dfg.resolve_aliases(arg),
             _ => panic!("Expected isplit: {}", pos.func.dfg.display_inst(inst, None)),
         };
 

From 954a2007d2da5d0e20f41d43d182fd122f389345 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Mon, 17 Jun 2019 21:16:20 +0200
Subject: [PATCH 2662/3084] Fix isplit legalization

---
 cranelift/codegen/src/legalizer/mod.rs | 46 ++++++++++++++++++++------
 1 file changed, 35 insertions(+), 11 deletions(-)

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index db9f302cae..94154d7dfe 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -65,20 +65,44 @@ fn legalize_inst(
             _ => panic!("Expected isplit: {}", pos.func.dfg.display_inst(inst, None)),
         };
 
-        let res = pos.func.dfg.inst_results(inst).to_vec();
-        assert_eq!(res.len(), 2);
-        let (resl, resh) = (res[0], res[1]); // Prevent borrowck error
+        let should_replace = match pos.func.dfg.value_def(arg) {
+            ir::ValueDef::Result(inst, num) => {
+                if let ir::InstructionData::Binary { opcode, args, .. } = pos.func.dfg[inst] {
+                    opcode == ir::Opcode::Iconcat
+                } else {
+                    // `arg` was not created by an `iconcat` instruction. Don't try to resolve it,
+                    // as otherwise `split::isplit` will re-insert the original `isplit`, causing
+                    // an endless loop.
+                    false
+                }
+            }
+            ir::ValueDef::Param(ebb, num) => true,
+        };
 
-        let curpos = pos.position();
-        let srcloc = pos.srcloc();
-        let (xl, xh) = split::isplit(pos.func, cfg, curpos, srcloc, arg);
+        if should_replace {
+            let res = pos.func.dfg.inst_results(inst).to_vec();
+            assert_eq!(res.len(), 2);
+            let (resl, resh) = (res[0], res[1]); // Prevent borrowck error
 
-        pos.func.dfg.clear_results(inst);
-        pos.remove_inst();
-        pos.func.dfg.change_to_alias(resl, xl);
-        pos.func.dfg.change_to_alias(resh, xh);
+            dbg!(pos.position());
 
-        return true;
+            // Remove old isplit
+            pos.func.dfg.clear_results(inst);
+            pos.remove_inst();
+
+            dbg!(pos.position());
+
+            let curpos = pos.position();
+            let srcloc = pos.srcloc();
+            let (xl, xh) = split::isplit(pos.func, cfg, curpos, srcloc, arg);
+
+            pos.func.dfg.change_to_alias(resl, xl);
+            pos.func.dfg.change_to_alias(resh, xh);
+
+            dbg!(&pos.func);
+
+            return true;
+        }
     }
 
     match pos.func.update_encoding(inst, isa) {

From 44ecc9c3d0fb3097d2045c9c7d371d491f069907 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Thu, 27 Jun 2019 20:29:51 +0200
Subject: [PATCH 2663/3084] [WIP]

---
 cranelift/codegen/src/legalizer/mod.rs | 106 +++++++++++++++----------
 1 file changed, 64 insertions(+), 42 deletions(-)

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 94154d7dfe..ac16e16c34 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -21,6 +21,7 @@ use crate::ir::{self, InstBuilder, MemFlags};
 use crate::isa::TargetIsa;
 use crate::predicates;
 use crate::timing;
+use std::vec::Vec;
 
 mod boundary;
 mod call;
@@ -36,24 +37,29 @@ use self::heap::expand_heap_addr;
 use self::libcall::expand_as_libcall;
 use self::table::expand_table_addr;
 
-/// Legalize `inst` for `isa`. Return true if any changes to the code were
-/// made; return false if the instruction was successfully encoded as is.
+enum LegalizeInstResult {
+    Done,
+    Legalized,
+    SplitLegalizePending,
+}
+
+/// Legalize `inst` for `isa`.
 fn legalize_inst(
     inst: ir::Inst,
     pos: &mut FuncCursor,
     cfg: &mut ControlFlowGraph,
     isa: &dyn TargetIsa,
-) -> bool {
+) -> LegalizeInstResult {
     let opcode = pos.func.dfg[inst].opcode();
 
     // Check for ABI boundaries that need to be converted to the legalized signature.
     if opcode.is_call() {
         if boundary::handle_call_abi(inst, pos.func, cfg) {
-            return true;
+            return LegalizeInstResult::Legalized;
         }
     } else if opcode.is_return() {
         if boundary::handle_return_abi(inst, pos.func, cfg) {
-            return true;
+            return LegalizeInstResult::Legalized;
         }
     } else if opcode.is_branch() {
         split::simplify_branch_arguments(&mut pos.func.dfg, inst);
@@ -65,48 +71,48 @@ fn legalize_inst(
             _ => panic!("Expected isplit: {}", pos.func.dfg.display_inst(inst, None)),
         };
 
-        let should_replace = match pos.func.dfg.value_def(arg) {
+        match pos.func.dfg.value_def(arg) {
             ir::ValueDef::Result(inst, num) => {
                 if let ir::InstructionData::Binary { opcode, args, .. } = pos.func.dfg[inst] {
-                    opcode == ir::Opcode::Iconcat
+                    if opcode != ir::Opcode::Iconcat {
+                        return LegalizeInstResult::SplitLegalizePending;
+                    }
                 } else {
                     // `arg` was not created by an `iconcat` instruction. Don't try to resolve it,
                     // as otherwise `split::isplit` will re-insert the original `isplit`, causing
                     // an endless loop.
-                    false
+                    return LegalizeInstResult::SplitLegalizePending;
                 }
             }
-            ir::ValueDef::Param(ebb, num) => true,
-        };
-
-        if should_replace {
-            let res = pos.func.dfg.inst_results(inst).to_vec();
-            assert_eq!(res.len(), 2);
-            let (resl, resh) = (res[0], res[1]); // Prevent borrowck error
-
-            dbg!(pos.position());
-
-            // Remove old isplit
-            pos.func.dfg.clear_results(inst);
-            pos.remove_inst();
-
-            dbg!(pos.position());
-
-            let curpos = pos.position();
-            let srcloc = pos.srcloc();
-            let (xl, xh) = split::isplit(pos.func, cfg, curpos, srcloc, arg);
-
-            pos.func.dfg.change_to_alias(resl, xl);
-            pos.func.dfg.change_to_alias(resh, xh);
-
-            dbg!(&pos.func);
-
-            return true;
+            ir::ValueDef::Param(ebb, num) => {},
         }
+
+        let res = pos.func.dfg.inst_results(inst).to_vec();
+        assert_eq!(res.len(), 2);
+        let (resl, resh) = (res[0], res[1]); // Prevent borrowck error
+
+        dbg!(pos.position());
+
+        // Remove old isplit
+        pos.func.dfg.clear_results(inst);
+        pos.remove_inst();
+
+        dbg!(pos.position());
+
+        let curpos = pos.position();
+        let srcloc = pos.srcloc();
+        let (xl, xh) = split::isplit(pos.func, cfg, curpos, srcloc, arg);
+
+        pos.func.dfg.change_to_alias(resl, xl);
+        pos.func.dfg.change_to_alias(resh, xh);
+
+        dbg!(&pos.func);
+
+        return LegalizeInstResult::Legalized;
     }
 
     match pos.func.update_encoding(inst, isa) {
-        Ok(()) => false,
+        Ok(()) => LegalizeInstResult::Done,
         Err(action) => {
             // We should transform the instruction into legal equivalents.
             // If the current instruction was replaced, we need to double back and revisit
@@ -115,12 +121,16 @@ fn legalize_inst(
             // There's a risk of infinite looping here if the legalization patterns are
             // unsound. Should we attempt to detect that?
             if action(inst, pos.func, cfg, isa) {
-                return true;
+                return LegalizeInstResult::Legalized;
             }
 
             // We don't have any pattern expansion for this instruction either.
             // Try converting it to a library call as a last resort.
-            expand_as_libcall(inst, pos.func, isa)
+            if expand_as_libcall(inst, pos.func, isa) {
+                LegalizeInstResult::Legalized
+            } else {
+                LegalizeInstResult::Done
+            }
         }
     }
 }
@@ -139,6 +149,7 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
     func.encodings.resize(func.dfg.num_insts());
 
     let mut pos = FuncCursor::new(func);
+    let mut pending_splits = Vec::new();
 
     // Process EBBs in layout order. Some legalization actions may split the current EBB or append
     // new ones to the end. We need to make sure we visit those new EBBs too.
@@ -148,16 +159,27 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
         let mut prev_pos = pos.position();
 
         while let Some(inst) = pos.next_inst() {
-            if legalize_inst(inst, &mut pos, cfg, isa) {
-                // Go back and legalize the inserted return value conversion instructions.
-                pos.set_position(prev_pos);
-            } else {
+            match legalize_inst(inst, &mut pos, cfg, isa) {
                 // Remember this position in case we need to double back.
-                prev_pos = pos.position();
+                LegalizeInstResult::Done => prev_pos = pos.position(),
+
+                // Go back and legalize the inserted return value conversion instructions.
+                LegalizeInstResult::Legalized => pos.set_position(prev_pos),
+
+                // The argument of a `isplit` or `vsplit` instruction didn't resolve to a
+                // `iconcat` or `vconcat` instruction. Try again after legalizing the rest of
+                // the instructions.
+                LegalizeInstResult::SplitLegalizePending => pending_splits.push(inst),
             }
         }
     }
 
+    // Try legalizing `isplit` and `vsplit` instructions, which could not previously be legalized.
+    for inst in pending_splits {
+        //pos.goto_inst(inst);
+        //legalize_inst(inst, &mut pos, cfg, isa);
+    }
+
     // Now that we've lowered all br_tables, we don't need the jump tables anymore.
     if !isa.flags().jump_tables_enabled() {
         pos.func.jump_tables.clear();

From d9ee08c0884c48cc04c28346838ff5c11442f38e Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 29 Jun 2019 14:37:05 +0200
Subject: [PATCH 2664/3084] Fix bug when i128 ebb param is unused

---
 cranelift/codegen/src/legalizer/mod.rs        |   4 +
 cranelift/codegen/src/legalizer/split.rs      | 112 ++++++++++++------
 .../isa/x86/jump_i128_param_unused.clif       |  10 ++
 3 files changed, 90 insertions(+), 36 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/jump_i128_param_unused.clif

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index ac16e16c34..a2266da08f 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -174,6 +174,10 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
         }
     }
 
+    while let Some(ebb) = pos.next_ebb() {
+        split::split_ebb_params(pos.func, cfg, ebb);
+    }
+
     // Try legalizing `isplit` and `vsplit` instructions, which could not previously be legalized.
     for inst in pending_splits {
         //pos.goto_inst(inst);
diff --git a/cranelift/codegen/src/legalizer/split.rs b/cranelift/codegen/src/legalizer/split.rs
index 773df13216..7b3d8f8d5c 100644
--- a/cranelift/codegen/src/legalizer/split.rs
+++ b/cranelift/codegen/src/legalizer/split.rs
@@ -124,6 +124,36 @@ fn split_any(
     let pos = &mut FuncCursor::new(func).at_position(pos).with_srcloc(srcloc);
     let result = split_value(pos, value, concat, &mut repairs);
 
+    perform_repairs(pos, cfg, repairs);
+
+    result
+}
+
+pub fn split_ebb_params(
+    func: &mut ir::Function,
+    cfg: &ControlFlowGraph,
+    ebb: Ebb,
+) {
+    let mut repairs = Vec::new();
+    let pos = &mut FuncCursor::new(func).at_top(ebb);
+
+    for (num, ebb_param) in pos.func.dfg.ebb_params(ebb).to_vec().into_iter().enumerate() {
+        let ty = pos.func.dfg.value_type(ebb_param);
+        if ty != ir::types::I128 {
+            continue;
+        }
+
+        split_ebb_param(pos, ebb, num, ebb_param, Opcode::Iconcat, &mut repairs);
+    }
+
+    perform_repairs(pos, cfg, repairs);
+}
+
+fn perform_repairs(
+    pos: &mut FuncCursor,
+    cfg: &ControlFlowGraph,
+    mut repairs: Vec,
+) {
     // We have split the value requested, and now we may need to fix some EBB predecessors.
     while let Some(repair) = repairs.pop() {
         for BasicBlock { inst, .. } in cfg.pred_iter(repair.ebb) {
@@ -181,8 +211,6 @@ fn split_any(
             pos.func.dfg[inst].put_value_list(args);
         }
     }
-
-    result
 }
 
 /// Split a single value using the integer or vector semantics given by the `concat` opcode.
@@ -215,40 +243,7 @@ fn split_value(
             // This is an EBB parameter. We can split the parameter value unless this is the entry
             // block.
             if pos.func.layout.entry_block() != Some(ebb) {
-                // We are going to replace the parameter at `num` with two new arguments.
-                // Determine the new value types.
-                let ty = pos.func.dfg.value_type(value);
-                let split_type = match concat {
-                    Opcode::Iconcat => ty.half_width().expect("Invalid type for isplit"),
-                    Opcode::Vconcat => ty.half_vector().expect("Invalid type for vsplit"),
-                    _ => panic!("Unhandled concat opcode: {}", concat),
-                };
-
-                // Since the `repairs` stack potentially contains other parameter numbers for
-                // `ebb`, avoid shifting and renumbering EBB parameters. It could invalidate other
-                // `repairs` entries.
-                //
-                // Replace the original `value` with the low part, and append the high part at the
-                // end of the argument list.
-                let lo = pos.func.dfg.replace_ebb_param(value, split_type);
-                let hi_num = pos.func.dfg.num_ebb_params(ebb);
-                let hi = pos.func.dfg.append_ebb_param(ebb, split_type);
-                reuse = Some((lo, hi));
-
-                // Now the original value is dangling. Insert a concatenation instruction that can
-                // compute it from the two new parameters. This also serves as a record of what we
-                // did so a future call to this function doesn't have to redo the work.
-                //
-                // Note that it is safe to move `pos` here since `reuse` was set above, so we don't
-                // need to insert a split instruction before returning.
-                pos.goto_first_inst(ebb);
-                pos.ins()
-                    .with_result(value)
-                    .Binary(concat, split_type, lo, hi);
-
-                // Finally, splitting the EBB parameter is not enough. We also have to repair all
-                // of the predecessor instructions that branch here.
-                add_repair(concat, split_type, ebb, num, hi_num, repairs);
+                reuse = Some(split_ebb_param(pos, ebb, num, value, concat, repairs));
             }
         }
     }
@@ -267,6 +262,51 @@ fn split_value(
     }
 }
 
+fn split_ebb_param(
+    pos: &mut FuncCursor,
+    ebb: Ebb,
+    param_num: usize,
+    value: Value,
+    concat: Opcode,
+    repairs: &mut Vec,
+) -> (Value, Value) {
+    // We are going to replace the parameter at `num` with two new arguments.
+    // Determine the new value types.
+    let ty = pos.func.dfg.value_type(value);
+    let split_type = match concat {
+        Opcode::Iconcat => ty.half_width().expect("Invalid type for isplit"),
+        Opcode::Vconcat => ty.half_vector().expect("Invalid type for vsplit"),
+        _ => panic!("Unhandled concat opcode: {}", concat),
+    };
+
+    // Since the `repairs` stack potentially contains other parameter numbers for
+    // `ebb`, avoid shifting and renumbering EBB parameters. It could invalidate other
+    // `repairs` entries.
+    //
+    // Replace the original `value` with the low part, and append the high part at the
+    // end of the argument list.
+    let lo = pos.func.dfg.replace_ebb_param(value, split_type);
+    let hi_num = pos.func.dfg.num_ebb_params(ebb);
+    let hi = pos.func.dfg.append_ebb_param(ebb, split_type);
+
+    // Now the original value is dangling. Insert a concatenation instruction that can
+    // compute it from the two new parameters. This also serves as a record of what we
+    // did so a future call to this function doesn't have to redo the work.
+    //
+    // Note that it is safe to move `pos` here since `reuse` was set above, so we don't
+    // need to insert a split instruction before returning.
+    pos.goto_first_inst(ebb);
+    pos.ins()
+        .with_result(value)
+        .Binary(concat, split_type, lo, hi);
+
+    // Finally, splitting the EBB parameter is not enough. We also have to repair all
+    // of the predecessor instructions that branch here.
+    add_repair(concat, split_type, ebb, param_num, hi_num, repairs);
+
+    (lo, hi)
+}
+
 // Add a repair entry to the work list.
 fn add_repair(
     concat: Opcode,
diff --git a/cranelift/filetests/filetests/isa/x86/jump_i128_param_unused.clif b/cranelift/filetests/filetests/isa/x86/jump_i128_param_unused.clif
new file mode 100644
index 0000000000..9d96fcbe31
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/jump_i128_param_unused.clif
@@ -0,0 +1,10 @@
+test compile
+target x86_64
+
+function u0:0(i128) system_v {
+ebb0(v0: i128):
+    jump ebb1(v0)
+
+ebb1(v1: i128):
+    return
+}

From 8d0e8f893117e7e1806a37352b1680bda2b1ba0d Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 29 Jun 2019 16:21:52 +0200
Subject: [PATCH 2665/3084] [meta] Fix legalization in presence of varargs

---
 cranelift/codegen/meta/src/gen_legalizer.rs | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index 7b59844e60..67959b3c7c 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -61,10 +61,10 @@ fn unwrap_inst(
             fmtln!(fmt, "{},", field.member);
         }
 
-        if iform.num_value_operands == 1 {
-            fmt.line("arg,");
-        } else if iform.has_value_list || iform.num_value_operands > 1 {
+        if iform.has_value_list || iform.num_value_operands > 1 {
             fmt.line("ref args,");
+        } else if iform.num_value_operands == 1 {
+            fmt.line("arg,");
         }
 
         fmt.line("..");
@@ -87,6 +87,11 @@ fn unwrap_inst(
                 } else if op.is_value() {
                     let n = inst.value_opnums.iter().position(|&i| i == op_num).unwrap();
                     fmtln!(fmt, "func.dfg.resolve_aliases(args[{}]),", n);
+                } else if op.is_varargs() {
+                    let n = inst.imm_opnums.iter().chain(inst.value_opnums.iter()).max().map(|n| n + 1).unwrap_or(0);
+                    fmtln!(fmt, "\
+                        args.iter().skip({}).map(|&arg| func.dfg.resolve_aliases(arg)).collect::>(),\
+                    ", n);
                 }
             }
 
@@ -104,6 +109,14 @@ fn unwrap_inst(
     });
     fmtln!(fmt, "};");
 
+    assert_eq!(inst.operands_in.len(), apply.args.len());
+    for (i, op) in inst.operands_in.iter().enumerate() {
+        if op.is_varargs() {
+            let name = var_pool.get(apply.args[i].maybe_var().expect("vararg without name")).name;
+            fmtln!(fmt, "let {} = &{};", name, name);
+        }
+    }
+
     for &op_num in &inst.value_opnums {
         let arg = &apply.args[op_num];
         if let Some(var_index) = arg.maybe_var() {

From a43a3a5e9fdf8d135bc2bf8fbabc6bfc5f897de5 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 29 Jun 2019 16:22:31 +0200
Subject: [PATCH 2666/3084] [meta] Give a nicer error message when a
 legalization uses an incorrect number of arguments

---
 cranelift/codegen/meta/src/cdsl/xform.rs | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index b1a0234cd4..90d991f5e6 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -183,7 +183,12 @@ fn rewrite_expr(
     assert_eq!(
         apply_target.inst().operands_in.len(),
         dummy_args.len(),
-        "number of arguments in instruction is incorrect"
+        "number of arguments in instruction {} is incorrect\nexpected: {:?}",
+        apply_target.inst().name,
+        apply_target.inst().operands_in
+            .iter()
+            .map(|operand| format!("{}: {}", operand.name, operand.kind.name))
+            .collect::>(),
     );
 
     let mut args = Vec::new();

From 83ac6dd4d498f9cb0dce5c7f44e4bd0fd52a4da2 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 29 Jun 2019 16:23:07 +0200
Subject: [PATCH 2667/3084] [meta] Add some Debug derives

---
 cranelift/codegen/meta/src/cdsl/ast.rs            | 2 ++
 cranelift/codegen/meta/src/cdsl/instructions.rs   | 4 +++-
 cranelift/codegen/meta/src/cdsl/type_inference.rs | 2 +-
 3 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 141ffd84c2..798ee29d00 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -8,6 +8,7 @@ use cranelift_entity::{entity_impl, PrimaryMap};
 
 use std::fmt;
 
+#[derive(Debug)]
 pub enum Expr {
     Var(VarIndex),
     Literal(Literal),
@@ -363,6 +364,7 @@ impl VarPool {
 ///
 /// An `Apply` AST expression is created by using function call syntax on instructions. This
 /// applies to both bound and unbound polymorphic instructions.
+#[derive(Debug)]
 pub struct Apply {
     pub inst: Instruction,
     pub args: Vec,
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 1689f2a5c0..5f91771797 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -79,12 +79,14 @@ impl InstructionGroup {
     }
 }
 
+#[derive(Debug)]
 pub struct PolymorphicInfo {
     pub use_typevar_operand: bool,
     pub ctrl_typevar: TypeVar,
     pub other_typevars: Vec,
 }
 
+#[derive(Debug)]
 pub struct InstructionContent {
     /// Instruction mnemonic, also becomes opcode name.
     pub name: String,
@@ -139,7 +141,7 @@ pub struct InstructionContent {
     pub writes_cpu_flags: bool,
 }
 
-#[derive(Clone)]
+#[derive(Clone, Debug)]
 pub struct Instruction {
     content: Rc,
 }
diff --git a/cranelift/codegen/meta/src/cdsl/type_inference.rs b/cranelift/codegen/meta/src/cdsl/type_inference.rs
index 101cfa4104..a56d81463c 100644
--- a/cranelift/codegen/meta/src/cdsl/type_inference.rs
+++ b/cranelift/codegen/meta/src/cdsl/type_inference.rs
@@ -4,7 +4,7 @@ use crate::cdsl::typevar::{DerivedFunc, TypeSet, TypeVar};
 use std::collections::{HashMap, HashSet};
 use std::iter::FromIterator;
 
-#[derive(Hash, PartialEq, Eq)]
+#[derive(Debug, Hash, PartialEq, Eq)]
 pub enum Constraint {
     /// Constraint specifying that a type var tv1 must be wider than or equal to type var tv2 at
     /// runtime. This requires that:

From 762b5e494b442fda74cf3a9126426861cdad22d7 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 29 Jun 2019 16:38:53 +0200
Subject: [PATCH 2668/3084] Legalize brz.i128 and brnz.i128

---
 cranelift/codegen/meta/src/gen_legalizer.rs   |  5 ++++
 cranelift/codegen/meta/src/shared/legalize.rs | 26 ++++++++++++++++++-
 .../filetests/filetests/isa/x86/br-i128.clif  | 24 +++++++++++++++++
 3 files changed, 54 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/br-i128.clif

diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index 67959b3c7c..344bad7472 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -415,6 +415,11 @@ fn gen_transform<'a>(
             fmt.line("let removed = pos.remove_inst();");
             fmt.line("debug_assert_eq!(removed, inst);");
         }
+
+        if transform.def_pool.get(transform.src).apply.inst.is_branch {
+            fmt.line("cfg.recompute_ebb(pos.func, pos.current_ebb().unwrap());");
+        }
+
         fmt.line("return true;");
     });
     fmt.line("}");
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 90fe47d0fd..57ad9c89b8 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -4,7 +4,7 @@ use crate::cdsl::xform::{TransformGroupBuilder, TransformGroups};
 
 use crate::shared::immediates::Immediates;
 use crate::shared::types::Float::{F32, F64};
-use crate::shared::types::Int::{I16, I32, I64, I8};
+use crate::shared::types::Int::{I16, I128, I32, I64, I8};
 
 pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGroups {
     let mut narrow = TransformGroupBuilder::new(
@@ -49,6 +49,8 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     let bor = insts.by_name("bor");
     let bor_imm = insts.by_name("bor_imm");
     let bor_not = insts.by_name("bor_not");
+    let brnz = insts.by_name("brnz");
+    let brz = insts.by_name("brz");
     let br_icmp = insts.by_name("br_icmp");
     let br_table = insts.by_name("br_table");
     let bxor = insts.by_name("bxor");
@@ -177,9 +179,11 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     let al = var("al");
     let ah = var("ah");
     let cc = var("cc");
+    let ebb = var("ebb");
     let ptr = var("ptr");
     let flags = var("flags");
     let offset = var("off");
+    let vararg = var("vararg");
 
     narrow.legalize(
         def!(a = iadd(x, y)),
@@ -227,6 +231,26 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
         ],
     );
 
+    narrow.legalize(
+        def!(brz.I128(x, ebb, vararg)),
+        vec![
+            def!((xl, xh) = isplit(x)),
+            def!(a = icmp_imm(Literal::enumerator_for(intcc, "eq"), xl, Literal::constant(imm64, 0))),
+            def!(b = icmp_imm(Literal::enumerator_for(intcc, "eq"), xh, Literal::constant(imm64, 0))),
+            def!(c = band(a, b)),
+            def!(brz(c, ebb, vararg)),
+        ],
+    );
+
+    narrow.legalize(
+        def!(brnz.I128(x, ebb, vararg)),
+        vec![
+            def!((xl, xh) = isplit(x)),
+            def!(brnz(xl, ebb, vararg)),
+            def!(brnz(xh, ebb, vararg)),
+        ],
+    );
+
     // Widen instructions with one input operand.
     for &op in &[bnot, popcnt] {
         for &int_ty in &[I8, I16] {
diff --git a/cranelift/filetests/filetests/isa/x86/br-i128.clif b/cranelift/filetests/filetests/isa/x86/br-i128.clif
new file mode 100644
index 0000000000..a09db3f41b
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/br-i128.clif
@@ -0,0 +1,24 @@
+test compile
+target x86_64
+
+function u0:0(i128) -> i8 fast {
+ebb0(v0: i128):
+    brz v0, ebb1
+    v1 = iconst.i8 0
+    return v1
+
+ebb1:
+    v2 = iconst.i8 1
+    return v2
+}
+
+function u0:1(i128) -> i8 fast {
+ebb0(v0: i128):
+    brnz v0, ebb1
+    v1 = iconst.i8 0
+    return v1
+
+ebb1:
+    v2 = iconst.i8 1
+    return v2
+}

From c7a8b6c9e568905e2442412bd5fd4aa2e1c3e28e Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 29 Jun 2019 16:44:07 +0200
Subject: [PATCH 2669/3084] Remove some dbg! invocations

---
 cranelift/codegen/src/legalizer/mod.rs | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index a2266da08f..e0433cb05f 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -91,14 +91,10 @@ fn legalize_inst(
         assert_eq!(res.len(), 2);
         let (resl, resh) = (res[0], res[1]); // Prevent borrowck error
 
-        dbg!(pos.position());
-
         // Remove old isplit
         pos.func.dfg.clear_results(inst);
         pos.remove_inst();
 
-        dbg!(pos.position());
-
         let curpos = pos.position();
         let srcloc = pos.srcloc();
         let (xl, xh) = split::isplit(pos.func, cfg, curpos, srcloc, arg);
@@ -106,8 +102,6 @@ fn legalize_inst(
         pos.func.dfg.change_to_alias(resl, xl);
         pos.func.dfg.change_to_alias(resh, xh);
 
-        dbg!(&pos.func);
-
         return LegalizeInstResult::Legalized;
     }
 

From 599b48d95f348da07e8f9f8f880acdf1d92df24b Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 29 Jun 2019 17:01:54 +0200
Subject: [PATCH 2670/3084] Narrowing legalize some more bitops

---
 cranelift/codegen/meta/src/shared/legalize.rs | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 57ad9c89b8..26f4a4543d 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -207,7 +207,7 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
         ],
     );
 
-    for &bin_op in &[band, bor, bxor] {
+    for &bin_op in &[band, bor, bxor, band_not, bor_not, bxor_not] {
         narrow.legalize(
             def!(a = bin_op(x, y)),
             vec![
@@ -220,6 +220,16 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
         );
     }
 
+    narrow.legalize(
+        def!(a = bnot(x)),
+        vec![
+            def!((xl, xh) = isplit(x)),
+            def!(al = bnot(xl)),
+            def!(ah = bnot(xh)),
+            def!(a = iconcat(al, ah)),
+        ],
+    );
+
     narrow.legalize(
         def!(a = select(c, x, y)),
         vec![

From 0d5b87038ab89b11715123d96b41513d37bb7633 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 29 Jun 2019 17:06:17 +0200
Subject: [PATCH 2671/3084] Rustfmt

---
 cranelift/codegen/meta/src/cdsl/xform.rs      |  4 +++-
 cranelift/codegen/meta/src/gen_legalizer.rs   |  4 +++-
 cranelift/codegen/meta/src/shared/legalize.rs | 18 +++++++++++++---
 cranelift/codegen/src/legalizer/mod.rs        |  2 +-
 cranelift/codegen/src/legalizer/split.rs      | 21 +++++++++----------
 5 files changed, 32 insertions(+), 17 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index 90d991f5e6..b90d552b9a 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -185,7 +185,9 @@ fn rewrite_expr(
         dummy_args.len(),
         "number of arguments in instruction {} is incorrect\nexpected: {:?}",
         apply_target.inst().name,
-        apply_target.inst().operands_in
+        apply_target
+            .inst()
+            .operands_in
             .iter()
             .map(|operand| format!("{}: {}", operand.name, operand.kind.name))
             .collect::>(),
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index 344bad7472..8223aef0ed 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -112,7 +112,9 @@ fn unwrap_inst(
     assert_eq!(inst.operands_in.len(), apply.args.len());
     for (i, op) in inst.operands_in.iter().enumerate() {
         if op.is_varargs() {
-            let name = var_pool.get(apply.args[i].maybe_var().expect("vararg without name")).name;
+            let name = var_pool
+                .get(apply.args[i].maybe_var().expect("vararg without name"))
+                .name;
             fmtln!(fmt, "let {} = &{};", name, name);
         }
     }
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 26f4a4543d..63118033a9 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -4,7 +4,7 @@ use crate::cdsl::xform::{TransformGroupBuilder, TransformGroups};
 
 use crate::shared::immediates::Immediates;
 use crate::shared::types::Float::{F32, F64};
-use crate::shared::types::Int::{I16, I128, I32, I64, I8};
+use crate::shared::types::Int::{I128, I16, I32, I64, I8};
 
 pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGroups {
     let mut narrow = TransformGroupBuilder::new(
@@ -245,8 +245,20 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
         def!(brz.I128(x, ebb, vararg)),
         vec![
             def!((xl, xh) = isplit(x)),
-            def!(a = icmp_imm(Literal::enumerator_for(intcc, "eq"), xl, Literal::constant(imm64, 0))),
-            def!(b = icmp_imm(Literal::enumerator_for(intcc, "eq"), xh, Literal::constant(imm64, 0))),
+            def!(
+                a = icmp_imm(
+                    Literal::enumerator_for(intcc, "eq"),
+                    xl,
+                    Literal::constant(imm64, 0)
+                )
+            ),
+            def!(
+                b = icmp_imm(
+                    Literal::enumerator_for(intcc, "eq"),
+                    xh,
+                    Literal::constant(imm64, 0)
+                )
+            ),
             def!(c = band(a, b)),
             def!(brz(c, ebb, vararg)),
         ],
diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index e0433cb05f..c96f24e9b3 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -84,7 +84,7 @@ fn legalize_inst(
                     return LegalizeInstResult::SplitLegalizePending;
                 }
             }
-            ir::ValueDef::Param(ebb, num) => {},
+            ir::ValueDef::Param(ebb, num) => {}
         }
 
         let res = pos.func.dfg.inst_results(inst).to_vec();
diff --git a/cranelift/codegen/src/legalizer/split.rs b/cranelift/codegen/src/legalizer/split.rs
index 7b3d8f8d5c..f16dae161b 100644
--- a/cranelift/codegen/src/legalizer/split.rs
+++ b/cranelift/codegen/src/legalizer/split.rs
@@ -129,15 +129,18 @@ fn split_any(
     result
 }
 
-pub fn split_ebb_params(
-    func: &mut ir::Function,
-    cfg: &ControlFlowGraph,
-    ebb: Ebb,
-) {
+pub fn split_ebb_params(func: &mut ir::Function, cfg: &ControlFlowGraph, ebb: Ebb) {
     let mut repairs = Vec::new();
     let pos = &mut FuncCursor::new(func).at_top(ebb);
 
-    for (num, ebb_param) in pos.func.dfg.ebb_params(ebb).to_vec().into_iter().enumerate() {
+    for (num, ebb_param) in pos
+        .func
+        .dfg
+        .ebb_params(ebb)
+        .to_vec()
+        .into_iter()
+        .enumerate()
+    {
         let ty = pos.func.dfg.value_type(ebb_param);
         if ty != ir::types::I128 {
             continue;
@@ -149,11 +152,7 @@ pub fn split_ebb_params(
     perform_repairs(pos, cfg, repairs);
 }
 
-fn perform_repairs(
-    pos: &mut FuncCursor,
-    cfg: &ControlFlowGraph,
-    mut repairs: Vec,
-) {
+fn perform_repairs(pos: &mut FuncCursor, cfg: &ControlFlowGraph, mut repairs: Vec) {
     // We have split the value requested, and now we may need to fix some EBB predecessors.
     while let Some(repair) = repairs.pop() {
         for BasicBlock { inst, .. } in cfg.pred_iter(repair.ebb) {

From dce521fa1c8980982cc59da431d38b56178f6800 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 29 Jun 2019 17:17:02 +0200
Subject: [PATCH 2672/3084] Fix lone isplit, when the corresponding iconcat
 will be created later during legalization

---
 cranelift/codegen/src/legalizer/mod.rs        |  4 ++--
 .../isa/x86/legalize-isplit-backwards.clif    | 24 +++++++++++++++++++
 2 files changed, 26 insertions(+), 2 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/legalize-isplit-backwards.clif

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index c96f24e9b3..5377783544 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -174,8 +174,8 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
 
     // Try legalizing `isplit` and `vsplit` instructions, which could not previously be legalized.
     for inst in pending_splits {
-        //pos.goto_inst(inst);
-        //legalize_inst(inst, &mut pos, cfg, isa);
+        pos.goto_inst(inst);
+        legalize_inst(inst, &mut pos, cfg, isa);
     }
 
     // Now that we've lowered all br_tables, we don't need the jump tables anymore.
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-isplit-backwards.clif b/cranelift/filetests/filetests/isa/x86/legalize-isplit-backwards.clif
new file mode 100644
index 0000000000..43881fe09e
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/legalize-isplit-backwards.clif
@@ -0,0 +1,24 @@
+test compile
+target x86_64
+
+function u0:0(i128) -> i64, i64 fast {
+; check: ebb0(v4: i64 [%rdi], v5: i64 [%rsi], v8: i64 [%rbp]):
+ebb0(v0: i128):
+    jump ebb2
+
+ebb1:
+    ; When this `isplit` is legalized, the bnot below is not yet legalized,
+    ; so there isn't a corresponding `iconcat` yet. We should try legalization
+    ; for this `isplit` again once all instrucions have been legalized.
+    v2, v3 = isplit.i128 v1
+    ; return v6, v7
+    return v2, v3
+
+ebb2:
+    ; check: v6 = bnot.i64 v4
+    ; check: v2 -> v6
+    ; check: v7 = bnot.i64 v5
+    ; check: v3 -> v7
+    v1 = bnot.i128 v0
+    jump ebb1
+}

From 3ae78fdddee8cfaf12f4e06658c9bfdd128a97a7 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 29 Jun 2019 17:18:14 +0200
Subject: [PATCH 2673/3084] Fix warnings

---
 cranelift/codegen/src/legalizer/mod.rs | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 5377783544..bc78d2ba16 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -72,8 +72,11 @@ fn legalize_inst(
         };
 
         match pos.func.dfg.value_def(arg) {
-            ir::ValueDef::Result(inst, num) => {
-                if let ir::InstructionData::Binary { opcode, args, .. } = pos.func.dfg[inst] {
+            ir::ValueDef::Result(inst, _num) => {
+                if let ir::InstructionData::Binary {
+                    opcode, args: _, ..
+                } = pos.func.dfg[inst]
+                {
                     if opcode != ir::Opcode::Iconcat {
                         return LegalizeInstResult::SplitLegalizePending;
                     }
@@ -84,7 +87,7 @@ fn legalize_inst(
                     return LegalizeInstResult::SplitLegalizePending;
                 }
             }
-            ir::ValueDef::Param(ebb, num) => {}
+            ir::ValueDef::Param(_ebb, _num) => {}
         }
 
         let res = pos.func.dfg.inst_results(inst).to_vec();

From fa9602df80d60cea419eedc900175edfe467e09c Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 29 Jun 2019 17:40:36 +0200
Subject: [PATCH 2674/3084] Legalize load.i128 and store.i128

---
 .../codegen/meta/src/cdsl/instructions.rs     |  2 +-
 cranelift/codegen/meta/src/shared/legalize.rs | 32 ++++++++++++++++---
 .../filetests/filetests/isa/x86/i128.clif     | 18 +++++++++++
 3 files changed, 47 insertions(+), 5 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 5f91771797..b91d9f26cf 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -14,7 +14,7 @@ use crate::cdsl::type_inference::Constraint;
 use crate::cdsl::types::{LaneType, ReferenceType, ValueType, VectorType};
 use crate::cdsl::typevar::TypeVar;
 
-#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+#[derive(Copy, Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct OpcodeNumber(u32);
 entity_impl!(OpcodeNumber);
 
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 63118033a9..e006452b35 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -247,16 +247,16 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
             def!((xl, xh) = isplit(x)),
             def!(
                 a = icmp_imm(
-                    Literal::enumerator_for(intcc, "eq"),
+                    Literal::enumerator_for(&imm.intcc, "eq"),
                     xl,
-                    Literal::constant(imm64, 0)
+                    Literal::constant(&imm.imm64, 0)
                 )
             ),
             def!(
                 b = icmp_imm(
-                    Literal::enumerator_for(intcc, "eq"),
+                    Literal::enumerator_for(&imm.intcc, "eq"),
                     xh,
-                    Literal::constant(imm64, 0)
+                    Literal::constant(&imm.imm64, 0)
                 )
             ),
             def!(c = band(a, b)),
@@ -273,6 +273,30 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
         ],
     );
 
+    // FIXME generalize to any offset once offset+8 can be represented
+    narrow.legalize(
+        def!(a = load.I128(flags, ptr, Literal::constant(&imm.offset32, 0))),
+        vec![
+            def!(al = load.I64(flags, ptr, Literal::constant(&imm.offset32, 0))),
+            def!(ah = load.I64(flags, ptr, Literal::constant(&imm.offset32, 8))),
+            // `iconcat` expects the same byte order as stored in memory,
+            // so no need to swap depending on endianness.
+            def!(a = iconcat(al, ah)),
+        ],
+    );
+
+    // FIXME generalize to any offset once offset+8 can be represented
+    narrow.legalize(
+        def!(store.I128(flags, a, ptr, Literal::constant(&imm.offset32, 0))),
+        vec![
+            // `isplit` gives the same byte order as stored in memory,
+            // so no need to swap depending on endianness.
+            def!((al, ah) = isplit(a)),
+            def!(store.I64(flags, al, ptr, Literal::constant(&imm.offset32, 0))),
+            def!(store.I64(flags, ah, ptr, Literal::constant(&imm.offset32, 8))),
+        ],
+    );
+
     // Widen instructions with one input operand.
     for &op in &[bnot, popcnt] {
         for &int_ty in &[I8, I16] {
diff --git a/cranelift/filetests/filetests/isa/x86/i128.clif b/cranelift/filetests/filetests/isa/x86/i128.clif
index 3c50d219dc..044b01e58f 100644
--- a/cranelift/filetests/filetests/isa/x86/i128.clif
+++ b/cranelift/filetests/filetests/isa/x86/i128.clif
@@ -26,3 +26,21 @@ ebb0(v0: i128):
     ; check: v6 = x86_pop.i64
     ; check: return v3, v4, v6
 }
+
+function u0:1(i64, i128) fast {
+; check: ebb0(v0: i64 [%rdi], v2: i64 [%rsi], v3: i64 [%rdx], v4: i64 [%rbp]):
+ebb0(v0: i64, v1: i128):
+    ; check: store v2, v0
+    ; check: store v3, v0+8
+    store v1, v0
+    return
+}
+
+function u0:1(i64) -> i128 fast {
+ebb0(v0: i64):
+    ; check: v2 = load.i64 v0
+    ; check: v3 = load.i64 v0+8
+    v1 = load.i128 v0
+    ; check: return v2, v3, v5
+    return v1
+}

From 67593d997be7baf3fc7f3f0f133928e87ae6f1fb Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Mon, 8 Jul 2019 18:42:28 +0200
Subject: [PATCH 2675/3084] Add b128 type to fix tests

---
 cranelift/codegen/meta/src/cdsl/types.rs      | 16 +++++++------
 cranelift/codegen/meta/src/cdsl/typevar.rs    | 23 ++++++++++---------
 cranelift/codegen/meta/src/shared/types.rs    |  4 ++++
 cranelift/codegen/src/ir/types.rs             | 18 +++++++++++----
 .../filetests/filetests/isa/x86/i128.clif     |  4 ++--
 cranelift/reader/src/lexer.rs                 |  1 +
 6 files changed, 41 insertions(+), 25 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
index 53b0032803..f431bb3ed7 100644
--- a/cranelift/codegen/meta/src/cdsl/types.rs
+++ b/cranelift/codegen/meta/src/cdsl/types.rs
@@ -215,13 +215,14 @@ impl LaneType {
                 LaneType::BoolType(shared_types::Bool::B16) => 2,
                 LaneType::BoolType(shared_types::Bool::B32) => 3,
                 LaneType::BoolType(shared_types::Bool::B64) => 4,
-                LaneType::IntType(shared_types::Int::I8) => 5,
-                LaneType::IntType(shared_types::Int::I16) => 6,
-                LaneType::IntType(shared_types::Int::I32) => 7,
-                LaneType::IntType(shared_types::Int::I64) => 8,
-                LaneType::IntType(shared_types::Int::I128) => 9,
-                LaneType::FloatType(shared_types::Float::F32) => 10,
-                LaneType::FloatType(shared_types::Float::F64) => 11,
+                LaneType::BoolType(shared_types::Bool::B128) => 5,
+                LaneType::IntType(shared_types::Int::I8) => 6,
+                LaneType::IntType(shared_types::Int::I16) => 7,
+                LaneType::IntType(shared_types::Int::I32) => 8,
+                LaneType::IntType(shared_types::Int::I64) => 9,
+                LaneType::IntType(shared_types::Int::I128) => 10,
+                LaneType::FloatType(shared_types::Float::F32) => 11,
+                LaneType::FloatType(shared_types::Float::F64) => 12,
             }
     }
 
@@ -232,6 +233,7 @@ impl LaneType {
             16 => shared_types::Bool::B16,
             32 => shared_types::Bool::B32,
             64 => shared_types::Bool::B64,
+            128 => shared_types::Bool::B128,
             _ => unreachable!("unxpected num bits for bool"),
         })
     }
diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 27ec365227..71c2fd2e29 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -10,6 +10,7 @@ use crate::cdsl::types::{BVType, LaneType, ReferenceType, SpecialType, ValueType
 
 const MAX_LANES: u16 = 256;
 const MAX_BITS: u16 = 128;
+const MAX_FLOAT_BITS: u16 = 64;
 const MAX_BITVEC: u16 = MAX_BITS * MAX_LANES;
 
 /// Type variables can be used in place of concrete types when defining
@@ -177,7 +178,7 @@ impl TypeVar {
                     "can't double all integer types"
                 );
                 assert!(
-                    ts.floats.len() == 0 || *ts.floats.iter().max().unwrap() < MAX_BITS,
+                    ts.floats.len() == 0 || *ts.floats.iter().max().unwrap() < MAX_FLOAT_BITS,
                     "can't double all float types"
                 );
                 assert!(
@@ -503,7 +504,7 @@ impl TypeSet {
         copy.floats = NumSet::from_iter(
             self.floats
                 .iter()
-                .filter(|&&x| x < MAX_BITS)
+                .filter(|&&x| x < MAX_FLOAT_BITS)
                 .map(|&x| x * 2),
         );
         copy.bools = NumSet::from_iter(
@@ -621,7 +622,7 @@ impl TypeSet {
                 let mut copy = self.clone();
                 copy.bitvecs = NumSet::new();
                 if self.bools.contains(&1) {
-                    copy.ints = NumSet::from_iter(vec![8, 16, 32, 64]);
+                    copy.ints = NumSet::from_iter(vec![8, 16, 32, 64, 128]);
                     copy.floats = NumSet::from_iter(vec![32, 64]);
                 } else {
                     copy.ints = &self.bools - &NumSet::from_iter(vec![1]);
@@ -950,7 +951,7 @@ fn test_typevar_builder() {
     let type_set = TypeSetBuilder::new().ints(Interval::All).build();
     assert_eq!(type_set.lanes, num_set![1]);
     assert!(type_set.floats.is_empty());
-    assert_eq!(type_set.ints, num_set![8, 16, 32, 64]);
+    assert_eq!(type_set.ints, num_set![8, 16, 32, 64, 128]);
     assert!(type_set.bools.is_empty());
     assert!(type_set.bitvecs.is_empty());
     assert!(type_set.specials.is_empty());
@@ -959,7 +960,7 @@ fn test_typevar_builder() {
     assert_eq!(type_set.lanes, num_set![1]);
     assert!(type_set.floats.is_empty());
     assert!(type_set.ints.is_empty());
-    assert_eq!(type_set.bools, num_set![1, 8, 16, 32, 64]);
+    assert_eq!(type_set.bools, num_set![1, 8, 16, 32, 64, 128]);
     assert!(type_set.bitvecs.is_empty());
     assert!(type_set.specials.is_empty());
 
@@ -1101,7 +1102,7 @@ fn test_forward_images() {
     );
     assert_eq!(
         TypeSetBuilder::new().ints(32..64).build().double_width(),
-        TypeSetBuilder::new().ints(64..64).build()
+        TypeSetBuilder::new().ints(64..128).build()
     );
     assert_eq!(
         TypeSetBuilder::new().floats(32..32).build().double_width(),
@@ -1117,7 +1118,7 @@ fn test_forward_images() {
     );
     assert_eq!(
         TypeSetBuilder::new().bools(32..64).build().double_width(),
-        TypeSetBuilder::new().bools(64..64).build()
+        TypeSetBuilder::new().bools(64..128).build()
     );
 }
 
@@ -1145,7 +1146,7 @@ fn test_backward_images() {
     assert_eq!(
         TypeSetBuilder::new()
             .simd_lanes(1..4)
-            .bools(1..64)
+            .bools(1..128)
             .build()
             .preimage(DerivedFunc::AsBool),
         TypeSetBuilder::new()
@@ -1205,9 +1206,9 @@ fn test_backward_images() {
     // Half width.
     assert_eq!(
         TypeSetBuilder::new()
-            .ints(64..64)
+            .ints(128..128)
             .floats(64..64)
-            .bools(64..64)
+            .bools(128..128)
             .build()
             .preimage(DerivedFunc::HalfWidth)
             .size(),
@@ -1221,7 +1222,7 @@ fn test_backward_images() {
             .preimage(DerivedFunc::HalfWidth),
         TypeSetBuilder::new()
             .simd_lanes(64..256)
-            .bools(16..64)
+            .bools(16..128)
             .build(),
     );
 
diff --git a/cranelift/codegen/meta/src/shared/types.rs b/cranelift/codegen/meta/src/shared/types.rs
index 60c546657e..52fa9545c6 100644
--- a/cranelift/codegen/meta/src/shared/types.rs
+++ b/cranelift/codegen/meta/src/shared/types.rs
@@ -12,6 +12,8 @@ pub enum Bool {
     B32 = 32,
     /// 64-bit bool.
     B64 = 64,
+    /// 128-bit bool.
+    B128 = 128,
 }
 
 /// This provides an iterator through all of the supported bool variants.
@@ -34,6 +36,7 @@ impl Iterator for BoolIterator {
             2 => Some(Bool::B16),
             3 => Some(Bool::B32),
             4 => Some(Bool::B64),
+            5 => Some(Bool::B128),
             _ => return None,
         };
         self.index += 1;
@@ -192,6 +195,7 @@ mod iter_tests {
         assert_eq!(bool_iter.next(), Some(Bool::B16));
         assert_eq!(bool_iter.next(), Some(Bool::B32));
         assert_eq!(bool_iter.next(), Some(Bool::B64));
+        assert_eq!(bool_iter.next(), Some(Bool::B128));
         assert_eq!(bool_iter.next(), None);
     }
 
diff --git a/cranelift/codegen/src/ir/types.rs b/cranelift/codegen/src/ir/types.rs
index 394ec8e024..10fca8aaa8 100644
--- a/cranelift/codegen/src/ir/types.rs
+++ b/cranelift/codegen/src/ir/types.rs
@@ -14,7 +14,7 @@ use target_lexicon::{PointerWidth, Triple};
 ///
 /// Basic floating point types: `F32` and `F64`. IEEE single and double precision.
 ///
-/// Boolean types: `B1`, `B8`, `B16`, `B32`, and `B64`. These all encode 'true' or 'false'. The
+/// Boolean types: `B1`, `B8`, `B16`, `B32`, `B64`, and `B128`. These all encode 'true' or 'false'. The
 /// larger types use redundant bits.
 ///
 /// SIMD vector types have power-of-two lanes, up to 256. Lanes can be any int/float/bool type.
@@ -63,7 +63,7 @@ impl Type {
             B16 | I16 => 4,
             B32 | I32 | F32 | R32 => 5,
             B64 | I64 | F64 | R64 => 6,
-            I128 => 7,
+            B128 | I128 => 7,
             _ => 0,
         }
     }
@@ -76,7 +76,7 @@ impl Type {
             B16 | I16 => 16,
             B32 | I32 | F32 | R32 => 32,
             B64 | I64 | F64 | R64 => 64,
-            I128 => 128,
+            B128 | I128 => 128,
             _ => 0,
         }
     }
@@ -112,6 +112,7 @@ impl Type {
             B32 | I32 | F32 => B32,
             B64 | I64 | F64 => B64,
             R32 | R64 => panic!("Reference types should not convert to bool"),
+            B128 | I128 => B128,
             _ => B1,
         })
     }
@@ -140,6 +141,7 @@ impl Type {
             B16 => B8,
             B32 => B16,
             B64 => B32,
+            B128 => B64,
             _ => return None,
         }))
     }
@@ -156,6 +158,7 @@ impl Type {
             B8 => B16,
             B16 => B32,
             B32 => B64,
+            B64 => B128,
             _ => return None,
         }))
     }
@@ -187,7 +190,7 @@ impl Type {
     /// Is this a scalar boolean type?
     pub fn is_bool(self) -> bool {
         match self {
-            B1 | B8 | B16 | B32 | B64 => true,
+            B1 | B8 | B16 | B32 | B64 | B128 => true,
             _ => false,
         }
     }
@@ -375,6 +378,7 @@ mod tests {
         assert_eq!(B16, B16.lane_type());
         assert_eq!(B32, B32.lane_type());
         assert_eq!(B64, B64.lane_type());
+        assert_eq!(B128, B128.lane_type());
         assert_eq!(I8, I8.lane_type());
         assert_eq!(I16, I16.lane_type());
         assert_eq!(I32, I32.lane_type());
@@ -396,6 +400,7 @@ mod tests {
         assert_eq!(B16.lane_bits(), 16);
         assert_eq!(B32.lane_bits(), 32);
         assert_eq!(B64.lane_bits(), 64);
+        assert_eq!(B128.lane_bits(), 128);
         assert_eq!(I8.lane_bits(), 8);
         assert_eq!(I16.lane_bits(), 16);
         assert_eq!(I32.lane_bits(), 32);
@@ -417,6 +422,7 @@ mod tests {
         assert_eq!(B16.half_width(), Some(B8));
         assert_eq!(B32.half_width(), Some(B16));
         assert_eq!(B64.half_width(), Some(B32));
+        assert_eq!(B128.half_width(), Some(B64));
         assert_eq!(I8.half_width(), None);
         assert_eq!(I16.half_width(), Some(I8));
         assert_eq!(I32.half_width(), Some(I16));
@@ -433,7 +439,8 @@ mod tests {
         assert_eq!(B8.double_width(), Some(B16));
         assert_eq!(B16.double_width(), Some(B32));
         assert_eq!(B32.double_width(), Some(B64));
-        assert_eq!(B64.double_width(), None);
+        assert_eq!(B64.double_width(), Some(B128));
+        assert_eq!(B128.double_width(), None);
         assert_eq!(I8.double_width(), Some(I16));
         assert_eq!(I16.double_width(), Some(I32));
         assert_eq!(I32.double_width(), Some(I64));
@@ -470,6 +477,7 @@ mod tests {
         assert_eq!(B16.to_string(), "b16");
         assert_eq!(B32.to_string(), "b32");
         assert_eq!(B64.to_string(), "b64");
+        assert_eq!(B128.to_string(), "b128");
         assert_eq!(I8.to_string(), "i8");
         assert_eq!(I16.to_string(), "i16");
         assert_eq!(I32.to_string(), "i32");
diff --git a/cranelift/filetests/filetests/isa/x86/i128.clif b/cranelift/filetests/filetests/isa/x86/i128.clif
index 044b01e58f..41b2a80fce 100644
--- a/cranelift/filetests/filetests/isa/x86/i128.clif
+++ b/cranelift/filetests/filetests/isa/x86/i128.clif
@@ -27,7 +27,7 @@ ebb0(v0: i128):
     ; check: return v3, v4, v6
 }
 
-function u0:1(i64, i128) fast {
+function u0:2(i64, i128) fast {
 ; check: ebb0(v0: i64 [%rdi], v2: i64 [%rsi], v3: i64 [%rdx], v4: i64 [%rbp]):
 ebb0(v0: i64, v1: i128):
     ; check: store v2, v0
@@ -36,7 +36,7 @@ ebb0(v0: i64, v1: i128):
     return
 }
 
-function u0:1(i64) -> i128 fast {
+function u0:3(i64) -> i128 fast {
 ebb0(v0: i64):
     ; check: v2 = load.i64 v0
     ; check: v3 = load.i64 v0+8
diff --git a/cranelift/reader/src/lexer.rs b/cranelift/reader/src/lexer.rs
index 151fbd7af5..2432ab0551 100644
--- a/cranelift/reader/src/lexer.rs
+++ b/cranelift/reader/src/lexer.rs
@@ -373,6 +373,7 @@ impl<'a> Lexer<'a> {
             "b16" => types::B16,
             "b32" => types::B32,
             "b64" => types::B64,
+            "b128" => types::B128,
             "r32" => types::R32,
             "r64" => types::R64,
             _ => return None,

From acd454890c2aff60600e0d119cccb9a9255647ae Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Thu, 18 Jul 2019 12:54:59 +0200
Subject: [PATCH 2676/3084] Legalize load.i128 and store.i128 with arbitrary
 offsets

---
 cranelift/codegen/src/legalizer/mod.rs        | 60 +++++++++++++++++++
 .../filetests/filetests/isa/x86/i128.clif     | 14 ++---
 2 files changed, 67 insertions(+), 7 deletions(-)

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index bc78d2ba16..ad13a1aaf3 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -567,3 +567,63 @@ fn expand_stack_store(
     mflags.set_aligned();
     pos.func.dfg.replace(inst).store(mflags, val, addr, 0);
 }
+
+/// Split a load into two parts before `iconcat`ing the result together.
+fn narrow_load(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    _cfg: &mut ControlFlowGraph,
+    _isa: &dyn TargetIsa,
+) {
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+    pos.use_srcloc(inst);
+
+    let (ptr, offset, flags) = match pos.func.dfg[inst] {
+        ir::InstructionData::Load {
+            opcode: ir::Opcode::Load,
+            arg,
+            offset,
+            flags,
+        } => (arg, offset, flags),
+        _ => panic!("Expected load: {}", pos.func.dfg.display_inst(inst, None)),
+    };
+
+    let al = pos.ins().load(ir::types::I64, flags, ptr, offset);
+    let ah = pos.ins().load(
+        ir::types::I64,
+        flags,
+        ptr,
+        offset.try_add_i64(8).expect("load offset overflow"),
+    );
+    pos.func.dfg.replace(inst).iconcat(al, ah);
+}
+
+/// Split a store into two parts after `isplit`ing the value.
+fn narrow_store(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    _cfg: &mut ControlFlowGraph,
+    _isa: &dyn TargetIsa,
+) {
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+    pos.use_srcloc(inst);
+
+    let (val, ptr, offset, flags) = match pos.func.dfg[inst] {
+        ir::InstructionData::Store {
+            opcode: ir::Opcode::Store,
+            args,
+            offset,
+            flags,
+        } => (args[0], args[1], offset, flags),
+        _ => panic!("Expected store: {}", pos.func.dfg.display_inst(inst, None)),
+    };
+
+    let (al, ah) = pos.func.dfg.replace(inst).isplit(val);
+    pos.ins().store(flags, al, ptr, offset);
+    pos.ins().store(
+        flags,
+        ah,
+        ptr,
+        offset.try_add_i64(8).expect("store offset overflow"),
+    );
+}
diff --git a/cranelift/filetests/filetests/isa/x86/i128.clif b/cranelift/filetests/filetests/isa/x86/i128.clif
index 41b2a80fce..b710a7430e 100644
--- a/cranelift/filetests/filetests/isa/x86/i128.clif
+++ b/cranelift/filetests/filetests/isa/x86/i128.clif
@@ -28,19 +28,19 @@ ebb0(v0: i128):
 }
 
 function u0:2(i64, i128) fast {
-; check: ebb0(v0: i64 [%rdi], v2: i64 [%rsi], v3: i64 [%rdx], v4: i64 [%rbp]):
+; check: ebb0(v0: i64 [%rdi], v2: i64 [%rsi], v3: i64 [%rdx], v6: i64 [%rbp]):
 ebb0(v0: i64, v1: i128):
-    ; check: store v2, v0
-    ; check: store v3, v0+8
-    store v1, v0
+    ; check: store v2, v0+8
+    ; check: store v3, v0+16
+    store v1, v0+8
     return
 }
 
 function u0:3(i64) -> i128 fast {
 ebb0(v0: i64):
-    ; check: v2 = load.i64 v0
-    ; check: v3 = load.i64 v0+8
-    v1 = load.i128 v0
+    ; check: v2 = load.i64 v0+8
+    ; check: v3 = load.i64 v0+16
+    v1 = load.i128 v0+8
     ; check: return v2, v3, v5
     return v1
 }

From ffa1e946a749acaca8c5d244b4e05c0f56aa60fa Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Fri, 19 Jul 2019 12:26:33 +0200
Subject: [PATCH 2677/3084] Fix compilation

---
 cranelift/codegen/meta/src/cdsl/instructions.rs |  5 +++++
 cranelift/codegen/meta/src/isa/x86/encodings.rs | 13 ++++++++-----
 2 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index b91d9f26cf..f061be94ed 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -1127,6 +1127,11 @@ fn bind_vector(
     mut value_types: Vec,
 ) -> BoundInstruction {
     let num_lanes = vector_size_in_bits / lane_type.lane_bits();
+    assert!(
+        num_lanes >= 2,
+        "Minimum lane number for bind_vector is 2, found {}.",
+        num_lanes,
+    );
     let vector_type = ValueType::Vector(VectorType::new(lane_type, num_lanes));
     value_types.push(ValueTypeOrAny::ValueType(vector_type));
     verify_polymorphic_binding(&inst, &value_types);
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 71f1042100..1739ed3ab7 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -9,7 +9,7 @@ use crate::cdsl::instructions::{
 };
 use crate::cdsl::recipes::{EncodingRecipe, EncodingRecipeNumber, Recipes};
 use crate::cdsl::settings::{SettingGroup, SettingPredicateNumber};
-use crate::cdsl::types::ValueType;
+use crate::cdsl::types::{LaneType, ValueType};
 use crate::shared::types::Bool::{B1, B16, B32, B64, B8};
 use crate::shared::types::Float::{F32, F64};
 use crate::shared::types::Int::{I16, I32, I64, I8};
@@ -1735,6 +1735,8 @@ pub(crate) fn define(
     // legalize.rs for how this is done; once there, x86_pshuf* (below) is used for broadcasting the
     // value across the register
 
+    let allowed_simd_type = |t: &LaneType| t.lane_bits() >= 8 && t.lane_bits() < 128;
+
     // PSHUFB, 8-bit shuffle using two XMM registers
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
         let instruction = x86_pshufb.bind_vector_from_lane(ty, sse_vector_size);
@@ -1756,7 +1758,7 @@ pub(crate) fn define(
     // SIMD scalar_to_vector; this uses MOV to copy the scalar value to an XMM register; according
     // to the Intel manual: "When the destination operand is an XMM register, the source operand is
     // written to the low doubleword of the register and the regiser is zero-extended to 128 bits."
-    for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() >= 8) {
+    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         let instruction = scalar_to_vector.bind_vector_from_lane(ty, sse_vector_size);
         let template = rec_frurm.opcodes(vec![0x66, 0x0f, 0x6e]); // MOVD/MOVQ
         if ty.lane_bits() < 64 {
@@ -1816,8 +1818,9 @@ pub(crate) fn define(
     }
 
     // SIMD bitcast all 128-bit vectors to each other (for legalizing splat.x16x8)
-    for from_type in ValueType::all_lane_types().filter(|t| t.lane_bits() >= 8) {
-        for to_type in ValueType::all_lane_types().filter(|t| t.lane_bits() >= 8 && *t != from_type)
+    for from_type in ValueType::all_lane_types().filter(allowed_simd_type) {
+        for to_type in
+            ValueType::all_lane_types().filter(|t| allowed_simd_type(t) && *t != from_type)
         {
             let instruction = raw_bitcast
                 .bind_vector_from_lane(to_type, sse_vector_size)
@@ -1833,7 +1836,7 @@ pub(crate) fn define(
     // for that; alternately, constants could be loaded into XMM registers using a sequence like:
     // MOVQ + MOVHPD + MOVQ + MOVLPD (this allows the constants to be immediates instead of stored
     // in memory) but some performance measurements are needed.
-    for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() >= 8) {
+    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         let instruction = vconst.bind_vector_from_lane(ty, sse_vector_size);
         let template = rec_vconst.nonrex().opcodes(vec![0x0f, 0x10]);
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE

From 2426bce9ac3d0f73733574173d08a38137e95d1e Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Thu, 15 Aug 2019 13:24:34 +0200
Subject: [PATCH 2678/3084] Fix load.i64 and store legalization

---
 cranelift/codegen/src/legalizer/mod.rs           | 10 +++++++---
 .../filetests/isa/x86/load-store-narrow.clif     | 16 ++++++++++++++++
 2 files changed, 23 insertions(+), 3 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/load-store-narrow.clif

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index ad13a1aaf3..70fb614a2d 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -588,9 +588,12 @@ fn narrow_load(
         _ => panic!("Expected load: {}", pos.func.dfg.display_inst(inst, None)),
     };
 
-    let al = pos.ins().load(ir::types::I64, flags, ptr, offset);
+    let res_ty = pos.func.dfg.ctrl_typevar(inst);
+    let small_ty = res_ty.half_width().expect("Can't narrow load");
+
+    let al = pos.ins().load(small_ty, flags, ptr, offset);
     let ah = pos.ins().load(
-        ir::types::I64,
+        small_ty,
         flags,
         ptr,
         offset.try_add_i64(8).expect("load offset overflow"),
@@ -618,7 +621,7 @@ fn narrow_store(
         _ => panic!("Expected store: {}", pos.func.dfg.display_inst(inst, None)),
     };
 
-    let (al, ah) = pos.func.dfg.replace(inst).isplit(val);
+    let (al, ah) = pos.ins().isplit(val);
     pos.ins().store(flags, al, ptr, offset);
     pos.ins().store(
         flags,
@@ -626,4 +629,5 @@ fn narrow_store(
         ptr,
         offset.try_add_i64(8).expect("store offset overflow"),
     );
+    pos.remove_inst();
 }
diff --git a/cranelift/filetests/filetests/isa/x86/load-store-narrow.clif b/cranelift/filetests/filetests/isa/x86/load-store-narrow.clif
new file mode 100644
index 0000000000..5f95b92fc0
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/load-store-narrow.clif
@@ -0,0 +1,16 @@
+test compile
+target i686
+
+function u0:0(i64, i32) system_v {
+ebb0(v0: i64, v1: i32):
+    v2 = bor v0, v0
+    store v2, v1
+    return
+}
+
+function u0:1(i32) -> i64 system_v {
+ebb0(v1: i32):
+    v0 = load.i64 v1
+    v2 = bor v0, v0
+    return v2
+}

From f4cdd3007c388b0f257abf79b8f6478ed066f724 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Thu, 29 Aug 2019 10:35:32 +0200
Subject: [PATCH 2679/3084] [split] Prevent double legalization of isplit and
 vsplit

---
 cranelift/codegen/src/legalizer/mod.rs        |  9 +++++++--
 .../isa/x86/isplit-not-legalized-twice.clif   | 20 +++++++++++++++++++
 2 files changed, 27 insertions(+), 2 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/isplit-not-legalized-twice.clif

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 70fb614a2d..d00a2ccd79 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -21,6 +21,7 @@ use crate::ir::{self, InstBuilder, MemFlags};
 use crate::isa::TargetIsa;
 use crate::predicates;
 use crate::timing;
+use std::collections::BTreeSet;
 use std::vec::Vec;
 
 mod boundary;
@@ -146,7 +147,9 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
     func.encodings.resize(func.dfg.num_insts());
 
     let mut pos = FuncCursor::new(func);
-    let mut pending_splits = Vec::new();
+
+    // This must be a set to prevent trying to legalize `isplit` and `vsplit` twice in certain cases.
+    let mut pending_splits = BTreeSet::new();
 
     // Process EBBs in layout order. Some legalization actions may split the current EBB or append
     // new ones to the end. We need to make sure we visit those new EBBs too.
@@ -166,7 +169,9 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
                 // The argument of a `isplit` or `vsplit` instruction didn't resolve to a
                 // `iconcat` or `vconcat` instruction. Try again after legalizing the rest of
                 // the instructions.
-                LegalizeInstResult::SplitLegalizePending => pending_splits.push(inst),
+                LegalizeInstResult::SplitLegalizePending => {
+                    pending_splits.insert(inst);
+                }
             }
         }
     }
diff --git a/cranelift/filetests/filetests/isa/x86/isplit-not-legalized-twice.clif b/cranelift/filetests/filetests/isa/x86/isplit-not-legalized-twice.clif
new file mode 100644
index 0000000000..4b81a186da
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/isplit-not-legalized-twice.clif
@@ -0,0 +1,20 @@
+test compile
+target x86_64
+
+function u0:0(i64, i64) -> i128 system_v {
+ebb0(v0: i64, v1: i64):
+    trap user0
+
+ebb30:
+    v245 = iconst.i64 0
+    v246 = iconcat v245, v245
+    ; The next instruction used to be legalized twice, causing a panic the second time.
+    v250, v251 = isplit.i128 v370
+    v252, v253 = isplit v246
+    trap user0
+
+ebb45:
+    v369 = iconst.i64 0
+    v370 = load.i128 v369
+    trap user0
+}

From e2b2b520ebabeccc1485a9e4c5d6f6697f55a578 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Fri, 30 Aug 2019 19:07:39 +0200
Subject: [PATCH 2680/3084] Fix compilation

---
 cranelift/codegen/meta/src/isa/x86/encodings.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 1739ed3ab7..253491f01c 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1776,7 +1776,7 @@ pub(crate) fn define(
     insertlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41_simd))); // PINSRD
     insertlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41_simd))); // PINSRQ, only x86_64
 
-    for ty in ValueType::all_lane_types() {
+    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         if let Some((opcode, isap)) = insertlane_mapping.get(&ty.lane_bits()) {
             let instruction = insertlane.bind_vector_from_lane(ty, sse_vector_size);
             let template = rec_r_ib_unsigned_r.opcodes(opcode.clone());
@@ -1797,7 +1797,7 @@ pub(crate) fn define(
     extractlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41_simd))); // PEXTRD
     extractlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41_simd))); // PEXTRQ, only x86_64
 
-    for ty in ValueType::all_lane_types() {
+    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         if let Some((opcode, isap)) = extractlane_mapping.get(&ty.lane_bits()) {
             let instruction = extractlane.bind_vector_from_lane(ty, sse_vector_size);
             let template = rec_r_ib_unsigned_gpr.opcodes(opcode.clone());

From e8d4ef7c3de2c049cf16443e2fc3f7c08174dffd Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 31 Aug 2019 12:24:31 +0200
Subject: [PATCH 2681/3084] Fix review comments

---
 cranelift/codegen/meta/src/gen_legalizer.rs |  7 +++++++
 cranelift/codegen/src/legalizer/mod.rs      | 15 ++++++---------
 2 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index 8223aef0ed..da1fb1f58f 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -89,6 +89,8 @@ fn unwrap_inst(
                     fmtln!(fmt, "func.dfg.resolve_aliases(args[{}]),", n);
                 } else if op.is_varargs() {
                     let n = inst.imm_opnums.iter().chain(inst.value_opnums.iter()).max().map(|n| n + 1).unwrap_or(0);
+                    // We need to create a `Vec` here, as using a slice would result in a borrowck
+                    // error later on.
                     fmtln!(fmt, "\
                         args.iter().skip({}).map(|&arg| func.dfg.resolve_aliases(arg)).collect::>(),\
                     ", n);
@@ -115,6 +117,9 @@ fn unwrap_inst(
             let name = var_pool
                 .get(apply.args[i].maybe_var().expect("vararg without name"))
                 .name;
+
+            // Above name is set to an `Vec` representing the varargs. However it is expected to be
+            // `&[Value]` below, so we borrow it.
             fmtln!(fmt, "let {} = &{};", name, name);
         }
     }
@@ -419,6 +424,8 @@ fn gen_transform<'a>(
         }
 
         if transform.def_pool.get(transform.src).apply.inst.is_branch {
+            // A branch might have been legalized into multiple branches, so we need to recompute
+            // the cfg.
             fmt.line("cfg.recompute_ebb(pos.func, pos.current_ebb().unwrap());");
         }
 
diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index d00a2ccd79..0f24689d8c 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -75,12 +75,11 @@ fn legalize_inst(
         match pos.func.dfg.value_def(arg) {
             ir::ValueDef::Result(inst, _num) => {
                 if let ir::InstructionData::Binary {
-                    opcode, args: _, ..
+                    opcode: ir::Opcode::Iconcat,
+                    ..
                 } = pos.func.dfg[inst]
                 {
-                    if opcode != ir::Opcode::Iconcat {
-                        return LegalizeInstResult::SplitLegalizePending;
-                    }
+                    // `arg` was created by an `iconcat` instruction.
                 } else {
                     // `arg` was not created by an `iconcat` instruction. Don't try to resolve it,
                     // as otherwise `split::isplit` will re-insert the original `isplit`, causing
@@ -153,7 +152,9 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
 
     // Process EBBs in layout order. Some legalization actions may split the current EBB or append
     // new ones to the end. We need to make sure we visit those new EBBs too.
-    while let Some(_ebb) = pos.next_ebb() {
+    while let Some(ebb) = pos.next_ebb() {
+        split::split_ebb_params(pos.func, cfg, ebb);
+
         // Keep track of the cursor position before the instruction being processed, so we can
         // double back when replacing instructions.
         let mut prev_pos = pos.position();
@@ -176,10 +177,6 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
         }
     }
 
-    while let Some(ebb) = pos.next_ebb() {
-        split::split_ebb_params(pos.func, cfg, ebb);
-    }
-
     // Try legalizing `isplit` and `vsplit` instructions, which could not previously be legalized.
     for inst in pending_splits {
         pos.goto_inst(inst);

From 0273eb84e071a6f7ffb8c84f37c113132d68ef17 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 7 Sep 2019 11:51:23 +0200
Subject: [PATCH 2682/3084] Fix rebase

---
 cranelift/codegen/meta/src/shared/legalize.rs | 27 +++----------------
 1 file changed, 3 insertions(+), 24 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index e006452b35..9588272709 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -185,6 +185,9 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     let offset = var("off");
     let vararg = var("vararg");
 
+    narrow.custom_legalize(load, "narrow_load");
+    narrow.custom_legalize(store, "narrow_store");
+
     narrow.legalize(
         def!(a = iadd(x, y)),
         vec![
@@ -273,30 +276,6 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
         ],
     );
 
-    // FIXME generalize to any offset once offset+8 can be represented
-    narrow.legalize(
-        def!(a = load.I128(flags, ptr, Literal::constant(&imm.offset32, 0))),
-        vec![
-            def!(al = load.I64(flags, ptr, Literal::constant(&imm.offset32, 0))),
-            def!(ah = load.I64(flags, ptr, Literal::constant(&imm.offset32, 8))),
-            // `iconcat` expects the same byte order as stored in memory,
-            // so no need to swap depending on endianness.
-            def!(a = iconcat(al, ah)),
-        ],
-    );
-
-    // FIXME generalize to any offset once offset+8 can be represented
-    narrow.legalize(
-        def!(store.I128(flags, a, ptr, Literal::constant(&imm.offset32, 0))),
-        vec![
-            // `isplit` gives the same byte order as stored in memory,
-            // so no need to swap depending on endianness.
-            def!((al, ah) = isplit(a)),
-            def!(store.I64(flags, al, ptr, Literal::constant(&imm.offset32, 0))),
-            def!(store.I64(flags, ah, ptr, Literal::constant(&imm.offset32, 8))),
-        ],
-    );
-
     // Widen instructions with one input operand.
     for &op in &[bnot, popcnt] {
         for &int_ty in &[I8, I16] {

From 8fd112899090d684d6ec80cd184967495c1910e6 Mon Sep 17 00:00:00 2001
From: Aaron Power 
Date: Thu, 21 Mar 2019 10:32:42 +0100
Subject: [PATCH 2683/3084] Remove FunctionBuilderContext from API, and change
 FunctionBuilder API

---
 cranelift/frontend/src/frontend.rs            | 156 +++++++++---------
 cranelift/frontend/src/lib.rs                 |  19 +--
 cranelift/frontend/src/ssa.rs                 |   4 +-
 cranelift/frontend/src/switch.rs              |  16 +-
 cranelift/frontend/src/variable.rs            |   2 +-
 .../simplejit/examples/simplejit-minimal.rs   |  17 +-
 cranelift/simplejit/tests/basic.rs            |  37 +++--
 cranelift/umbrella/src/lib.rs                 |   2 +-
 cranelift/wasm/src/code_translator.rs         |  19 ++-
 cranelift/wasm/src/environ/dummy.rs           |   3 +-
 cranelift/wasm/src/func_translator.rs         |  59 +++----
 11 files changed, 162 insertions(+), 172 deletions(-)

diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 0961314559..29330a88df 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -20,22 +20,22 @@ use std::vec::Vec;
 /// In order to reduce memory reallocations when compiling multiple functions,
 /// `FunctionBuilderContext` holds various data structures which are cleared between
 /// functions, rather than dropped, preserving the underlying allocations.
-pub struct FunctionBuilderContext {
+struct FunctionBuilderContext {
     ssa: SSABuilder,
     ebbs: SecondaryMap,
     types: SecondaryMap,
 }
 
 /// Temporary object used to build a single Cranelift IR `Function`.
-pub struct FunctionBuilder<'a> {
+pub struct FunctionBuilder {
     /// The function currently being built.
     /// This field is public so the function can be re-borrowed.
-    pub func: &'a mut Function,
+    pub func: Function,
 
     /// Source location to assign to all new instructions.
     srcloc: ir::SourceLoc,
 
-    func_ctx: &'a mut FunctionBuilderContext,
+    func_ctx: FunctionBuilderContext,
     position: Position,
 }
 
@@ -75,8 +75,9 @@ impl Position {
 }
 
 impl FunctionBuilderContext {
-    /// Creates a FunctionBuilderContext structure. The structure is automatically cleared after
-    /// each [`FunctionBuilder`](struct.FunctionBuilder.html) completes translating a function.
+    /// Creates a FunctionBuilderContext structure. The structure is
+    /// automatically cleared after each `FunctionBuilder` completes translating
+    /// a function.
     pub fn new() -> Self {
         Self {
             ssa: SSABuilder::new(),
@@ -90,26 +91,22 @@ impl FunctionBuilderContext {
         self.ebbs.clear();
         self.types.clear();
     }
-
-    fn is_empty(&self) -> bool {
-        self.ssa.is_empty() && self.ebbs.is_empty() && self.types.is_empty()
-    }
 }
 
 /// Implementation of the [`InstBuilder`](cranelift_codegen::ir::InstBuilder) that has
 /// one convenience method per Cranelift IR instruction.
-pub struct FuncInstBuilder<'short, 'long: 'short> {
-    builder: &'short mut FunctionBuilder<'long>,
+pub struct FuncInstBuilder<'short> {
+    builder: &'short mut FunctionBuilder,
     ebb: Ebb,
 }
 
-impl<'short, 'long> FuncInstBuilder<'short, 'long> {
-    fn new(builder: &'short mut FunctionBuilder<'long>, ebb: Ebb) -> Self {
+impl<'short> FuncInstBuilder<'short> {
+    fn new(builder: &'short mut FunctionBuilder, ebb: Ebb) -> Self {
         Self { builder, ebb }
     }
 }
 
-impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
+impl<'short> InstBuilderBase<'short> for FuncInstBuilder<'short> {
     fn data_flow_graph(&self) -> &DataFlowGraph {
         &self.builder.func.dfg
     }
@@ -203,12 +200,9 @@ impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
 /// The first block for which you call `switch_to_block` will be assumed to be the beginning of
 /// the function.
 ///
-/// At creation, a `FunctionBuilder` instance borrows an already allocated `Function` which it
-/// modifies with the information stored in the mutable borrowed
-/// [`FunctionBuilderContext`](struct.FunctionBuilderContext.html). The function passed in
-/// argument should be newly created with
-/// [`Function::with_name_signature()`](Function::with_name_signature), whereas the
-/// `FunctionBuilderContext` can be kept as is between two function translations.
+/// At creation, a `FunctionBuilder` instance borrows an already allocated
+/// `Function`. The function passed in should be newly created with
+/// [`Function::with_name_signature()`](Function::with_name_signature).
 ///
 /// # Errors
 ///
@@ -216,15 +210,13 @@ impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
 /// function in a way that violate the coherence of the code. For instance: switching to a new
 /// `Ebb` when you haven't filled the current one with a terminator instruction, inserting a
 /// return instruction with arguments that don't match the function's signature.
-impl<'a> FunctionBuilder<'a> {
-    /// Creates a new FunctionBuilder structure that will operate on a `Function` using a
-    /// `FunctionBuilderContext`.
-    pub fn new(func: &'a mut Function, func_ctx: &'a mut FunctionBuilderContext) -> Self {
-        debug_assert!(func_ctx.is_empty());
+impl FunctionBuilder {
+    /// Creates a new FunctionBuilder structure that will operate on a `Function`.
+    pub fn new(func: Function) -> Self {
         Self {
             func,
             srcloc: Default::default(),
-            func_ctx,
+            func_ctx: FunctionBuilderContext::new(),
             position: Position::default(),
         }
     }
@@ -279,7 +271,7 @@ impl<'a> FunctionBuilder<'a> {
     /// created. Forgetting to call this method on every block will cause inconsistencies in the
     /// produced functions.
     pub fn seal_block(&mut self, ebb: Ebb) {
-        let side_effects = self.func_ctx.ssa.seal_ebb_header_block(ebb, self.func);
+        let side_effects = self.func_ctx.ssa.seal_ebb_header_block(ebb, &mut self.func);
         self.handle_ssa_side_effects(side_effects);
     }
 
@@ -290,7 +282,7 @@ impl<'a> FunctionBuilder<'a> {
     /// function can be used at the end of translating all blocks to ensure
     /// that everything is sealed.
     pub fn seal_all_blocks(&mut self) {
-        let side_effects = self.func_ctx.ssa.seal_all_ebb_header_blocks(self.func);
+        let side_effects = self.func_ctx.ssa.seal_all_ebb_header_blocks(&mut self.func);
         self.handle_ssa_side_effects(side_effects);
     }
 
@@ -311,7 +303,7 @@ impl<'a> FunctionBuilder<'a> {
             });
             self.func_ctx
                 .ssa
-                .use_var(self.func, var, ty, self.position.basic_block.unwrap())
+                .use_var(&mut self.func, var, ty, self.position.basic_block.unwrap())
         };
         self.handle_ssa_side_effects(side_effects);
         val
@@ -393,7 +385,7 @@ impl<'a> FunctionBuilder<'a> {
 
     /// Returns an object with the [`InstBuilder`](cranelift_codegen::ir::InstBuilder)
     /// trait that allows to conveniently append an instruction to the current `Ebb` being built.
-    pub fn ins<'short>(&'short mut self) -> FuncInstBuilder<'short, 'a> {
+    pub fn ins<'short>(&'short mut self) -> FuncInstBuilder<'short> {
         let ebb = self
             .position
             .ebb
@@ -423,7 +415,7 @@ impl<'a> FunctionBuilder<'a> {
     /// need to know about `FunctionBuilder` at all.
     pub fn cursor(&mut self) -> FuncCursor {
         self.ensure_inserted_ebb();
-        FuncCursor::new(self.func)
+        FuncCursor::new(&mut self.func)
             .with_srcloc(self.srcloc)
             .at_bottom(self.position.ebb.unwrap())
     }
@@ -459,10 +451,21 @@ impl<'a> FunctionBuilder<'a> {
         }
     }
 
-    /// Declare that translation of the current function is complete. This
-    /// resets the state of the `FunctionBuilder` in preparation to be used
+    /// Clears the current state of the `FunctionBuilder` while preserving
+    /// allocations for reuse in translating another function.
+    pub fn clear(&mut self) {
+        // Clear the state (but preserve the allocated buffers) in preparation
+        // for translation another function.
+        self.func_ctx.clear();
+        // Reset srcloc and position to initial states.
+        self.srcloc = Default::default();
+        self.position = Position::default();
+    }
+
+    /// Declare that translation of the current function is complete, and return the completed
+    /// function. This resets the state of the `FunctionBuilder` in preparation to be used
     /// for another function.
-    pub fn finalize(&mut self) {
+    pub fn finalize(&mut self) -> ir::Function {
         // Check that all the `Ebb`s are filled and sealed.
         debug_assert!(
             self.func_ctx
@@ -492,13 +495,8 @@ impl<'a> FunctionBuilder<'a> {
             }
         }
 
-        // Clear the state (but preserve the allocated buffers) in preparation
-        // for translation another function.
-        self.func_ctx.clear();
-
-        // Reset srcloc and position to initial states.
-        self.srcloc = Default::default();
-        self.position = Position::default();
+        self.clear();
+        std::mem::replace(&mut self.func, Function::new())
     }
 }
 
@@ -507,7 +505,7 @@ impl<'a> FunctionBuilder<'a> {
 /// performance of your translation perform more complex transformations to your Cranelift IR
 /// function. The functions below help you inspect the function you're creating and modify it
 /// in ways that can be unsafe if used incorrectly.
-impl<'a> FunctionBuilder<'a> {
+impl FunctionBuilder {
     /// Retrieves all the parameters for an `Ebb` currently inferred from the jump instructions
     /// inserted that target it and the SSA construction.
     pub fn ebb_params(&self, ebb: Ebb) -> &[Value] {
@@ -593,7 +591,7 @@ impl<'a> FunctionBuilder<'a> {
 }
 
 /// Helper functions
-impl<'a> FunctionBuilder<'a> {
+impl FunctionBuilder {
     /// Calls libc.memcpy
     ///
     /// Copies the `size` bytes from `src` to `dest`, assumes that `src + size`
@@ -853,7 +851,7 @@ fn greatest_divisible_power_of_two(size: u64) -> u64 {
 }
 
 // Helper functions
-impl<'a> FunctionBuilder<'a> {
+impl FunctionBuilder {
     fn move_to_next_basic_block(&mut self) {
         self.position.basic_block = PackedOption::from(
             self.func_ctx
@@ -888,7 +886,7 @@ impl<'a> FunctionBuilder<'a> {
 #[cfg(test)]
 mod tests {
     use super::greatest_divisible_power_of_two;
-    use crate::frontend::{FunctionBuilder, FunctionBuilderContext};
+    use crate::frontend::FunctionBuilder;
     use crate::Variable;
     use cranelift_codegen::entity::EntityRef;
     use cranelift_codegen::ir::types::*;
@@ -903,10 +901,9 @@ mod tests {
         sig.returns.push(AbiParam::new(I32));
         sig.params.push(AbiParam::new(I32));
 
-        let mut fn_ctx = FunctionBuilderContext::new();
-        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
-        {
-            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
+        let func = {
+            let func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+            let mut builder = FunctionBuilder::new(func);
 
             let block0 = builder.create_ebb();
             let block1 = builder.create_ebb();
@@ -981,8 +978,8 @@ mod tests {
                 builder.seal_all_blocks();
             }
 
-            builder.finalize();
-        }
+            builder.finalize()
+        };
 
         let flags = settings::Flags::new(settings::builder());
         // println!("{}", func.display(None));
@@ -1019,10 +1016,9 @@ mod tests {
         let mut sig = Signature::new(target.default_call_conv());
         sig.returns.push(AbiParam::new(I32));
 
-        let mut fn_ctx = FunctionBuilderContext::new();
-        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
-        {
-            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
+        let func = {
+            let func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+            let mut builder = FunctionBuilder::new(func);
 
             let block0 = builder.create_ebb();
             let x = Variable::new(0);
@@ -1041,8 +1037,8 @@ mod tests {
             builder.ins().return_(&[size]);
 
             builder.seal_all_blocks();
-            builder.finalize();
-        }
+            builder.finalize()
+        };
 
         assert_eq!(
             func.display(None).to_string(),
@@ -1080,10 +1076,9 @@ ebb0:
         let mut sig = Signature::new(target.default_call_conv());
         sig.returns.push(AbiParam::new(I32));
 
-        let mut fn_ctx = FunctionBuilderContext::new();
-        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
-        {
-            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
+        let func = {
+            let func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+            let mut builder = FunctionBuilder::new(func);
 
             let block0 = builder.create_ebb();
             let x = Variable::new(0);
@@ -1100,8 +1095,8 @@ ebb0:
             builder.ins().return_(&[dest]);
 
             builder.seal_all_blocks();
-            builder.finalize();
-        }
+            builder.finalize()
+        };
 
         assert_eq!(
             func.display(None).to_string(),
@@ -1137,10 +1132,9 @@ ebb0:
         let mut sig = Signature::new(target.default_call_conv());
         sig.returns.push(AbiParam::new(I32));
 
-        let mut fn_ctx = FunctionBuilderContext::new();
-        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
-        {
-            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
+        let func = {
+            let func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+            let mut builder = FunctionBuilder::new(func);
 
             let block0 = builder.create_ebb();
             let x = Variable::new(0);
@@ -1157,8 +1151,8 @@ ebb0:
             builder.ins().return_(&[dest]);
 
             builder.seal_all_blocks();
-            builder.finalize();
-        }
+            builder.finalize()
+        };
 
         assert_eq!(
             func.display(None).to_string(),
@@ -1197,10 +1191,9 @@ ebb0:
         let mut sig = Signature::new(target.default_call_conv());
         sig.returns.push(AbiParam::new(I32));
 
-        let mut fn_ctx = FunctionBuilderContext::new();
-        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
-        {
-            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
+        let func = {
+            let func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+            let mut builder = FunctionBuilder::new(func);
 
             let block0 = builder.create_ebb();
             let y = Variable::new(16);
@@ -1214,8 +1207,8 @@ ebb0:
             builder.ins().return_(&[dest]);
 
             builder.seal_all_blocks();
-            builder.finalize();
-        }
+            builder.finalize()
+        };
 
         assert_eq!(
             func.display(None).to_string(),
@@ -1249,10 +1242,9 @@ ebb0:
         let mut sig = Signature::new(target.default_call_conv());
         sig.returns.push(AbiParam::new(I32));
 
-        let mut fn_ctx = FunctionBuilderContext::new();
-        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
-        {
-            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
+        let func = {
+            let func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+            let mut builder = FunctionBuilder::new(func);
 
             let block0 = builder.create_ebb();
             let y = Variable::new(16);
@@ -1266,8 +1258,8 @@ ebb0:
             builder.ins().return_(&[dest]);
 
             builder.seal_all_blocks();
-            builder.finalize();
-        }
+            builder.finalize()
+        };
 
         assert_eq!(
             func.display(None).to_string(),
diff --git a/cranelift/frontend/src/lib.rs b/cranelift/frontend/src/lib.rs
index eff5c655fd..14cd7502dc 100644
--- a/cranelift/frontend/src/lib.rs
+++ b/cranelift/frontend/src/lib.rs
@@ -3,7 +3,7 @@
 //! Provides a straightforward way to create a Cranelift IR function and fill it with instructions
 //! corresponding to your source program written in another language.
 //!
-//! To get started, create an [`FunctionBuilderContext`](struct.FunctionBuilderContext.html) and
+//! To get started, create an [`Function`](../cranelift_codegen/ir/function/struct.Function.html) and
 //! pass it as an argument to a [`FunctionBuilder`](struct.FunctionBuilder.html).
 //!
 //! # Mutable variables and Cranelift IR values
@@ -61,7 +61,7 @@
 //! }
 //! ```
 //!
-//! Here is how you build the corresponding Cranelift IR function using `FunctionBuilderContext`:
+//! Here is how you build the corresponding Cranelift IR function using `Function`:
 //!
 //! ```rust
 //! extern crate cranelift_codegen;
@@ -73,16 +73,15 @@
 //! use cranelift_codegen::isa::CallConv;
 //! use cranelift_codegen::settings;
 //! use cranelift_codegen::verifier::verify_function;
-//! use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
+//! use cranelift_frontend::{FunctionBuilder, Variable};
 //!
 //! fn main() {
 //!     let mut sig = Signature::new(CallConv::SystemV);
 //!     sig.returns.push(AbiParam::new(I32));
 //!     sig.params.push(AbiParam::new(I32));
-//!     let mut fn_builder_ctx = FunctionBuilderContext::new();
-//!     let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
-//!     {
-//!         let mut builder = FunctionBuilder::new(&mut func, &mut fn_builder_ctx);
+//!     let func = {
+//!         let func = Function::with_name_signature(ExternalName::user(0, 0), sig);
+//!         let mut builder = FunctionBuilder::new(func);
 //!
 //!         let block0 = builder.create_ebb();
 //!         let block1 = builder.create_ebb();
@@ -152,8 +151,8 @@
 //!         builder.ins().jump(block1, &[]);
 //!         builder.seal_block(block1);
 //!
-//!         builder.finalize();
-//!     }
+//!         builder.finalize()
+//!     };
 //!
 //!     let flags = settings::Flags::new(settings::builder());
 //!     let res = verify_function(&func, &flags);
@@ -195,7 +194,7 @@ use hashmap_core::HashMap;
 #[cfg(feature = "std")]
 use std::collections::HashMap;
 
-pub use crate::frontend::{FunctionBuilder, FunctionBuilderContext};
+pub use crate::frontend::FunctionBuilder;
 pub use crate::switch::Switch;
 pub use crate::variable::Variable;
 
diff --git a/cranelift/frontend/src/ssa.rs b/cranelift/frontend/src/ssa.rs
index aa735f7747..2bdd068515 100644
--- a/cranelift/frontend/src/ssa.rs
+++ b/cranelift/frontend/src/ssa.rs
@@ -173,9 +173,7 @@ impl SSABuilder {
         self.variables.clear();
         self.blocks.clear();
         self.ebb_headers.clear();
-        debug_assert!(self.calls.is_empty());
-        debug_assert!(self.results.is_empty());
-        debug_assert!(self.side_effects.is_empty());
+        debug_assert!(self.is_empty());
     }
 
     /// Tests whether an `SSABuilder` is in a cleared state.
diff --git a/cranelift/frontend/src/switch.rs b/cranelift/frontend/src/switch.rs
index 0b1ca746cb..33b2001655 100644
--- a/cranelift/frontend/src/switch.rs
+++ b/cranelift/frontend/src/switch.rs
@@ -16,12 +16,11 @@ type EntryIndex = u64;
 /// # use cranelift_codegen::ir::types::*;
 /// # use cranelift_codegen::ir::{ExternalName, Function, Signature, InstBuilder};
 /// # use cranelift_codegen::isa::CallConv;
-/// # use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Switch};
+/// # use cranelift_frontend::{FunctionBuilder, Switch};
 /// #
 /// # let mut sig = Signature::new(CallConv::SystemV);
-/// # let mut fn_builder_ctx = FunctionBuilderContext::new();
 /// # let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
-/// # let mut builder = FunctionBuilder::new(&mut func, &mut fn_builder_ctx);
+/// # let mut builder = FunctionBuilder::new(func);
 /// #
 /// # let entry = builder.create_ebb();
 /// # builder.switch_to_block(entry);
@@ -289,16 +288,13 @@ impl ContiguousCaseRange {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use crate::frontend::FunctionBuilderContext;
     use cranelift_codegen::ir::Function;
     use std::string::ToString;
 
     macro_rules! setup {
         ($default:expr, [$($index:expr,)*]) => {{
-            let mut func = Function::new();
-            let mut func_ctx = FunctionBuilderContext::new();
-            {
-                let mut bx = FunctionBuilder::new(&mut func, &mut func_ctx);
+            let func = {
+                let mut bx = FunctionBuilder::new(Function::new());
                 let ebb = bx.create_ebb();
                 bx.switch_to_block(ebb);
                 let val = bx.ins().iconst(types::I8, 0);
@@ -308,7 +304,9 @@ mod tests {
                     switch.set_entry($index, ebb);
                 )*
                 switch.emit(&mut bx, val, Ebb::with_number($default).unwrap());
-            }
+                bx.seal_all_blocks();
+                bx.finalize()
+            };
             func
                 .to_string()
                 .trim_start_matches("function u0:0() fast {\n")
diff --git a/cranelift/frontend/src/variable.rs b/cranelift/frontend/src/variable.rs
index dddcd7490b..1a061beb97 100644
--- a/cranelift/frontend/src/variable.rs
+++ b/cranelift/frontend/src/variable.rs
@@ -1,6 +1,6 @@
 //! A basic `Variable` implementation.
 //!
-//! `FunctionBuilderContext`, `FunctionBuilder`, and related types have a `Variable`
+//! `FunctionBuilder` and related types have a `Variable`
 //! type parameter, to allow frontends that identify variables with
 //! their own index types to use them directly. Frontends which don't
 //! can use the `Variable` defined here.
diff --git a/cranelift/simplejit/examples/simplejit-minimal.rs b/cranelift/simplejit/examples/simplejit-minimal.rs
index 3b8e147830..2fc28367eb 100644
--- a/cranelift/simplejit/examples/simplejit-minimal.rs
+++ b/cranelift/simplejit/examples/simplejit-minimal.rs
@@ -7,7 +7,6 @@ fn main() {
     let mut module: Module =
         Module::new(SimpleJITBuilder::new(default_libcall_names()));
     let mut ctx = module.make_context();
-    let mut func_ctx = FunctionBuilderContext::new();
 
     let mut sig_a = module.make_signature();
     sig_a.params.push(AbiParam::new(types::I32));
@@ -25,8 +24,8 @@ fn main() {
 
     ctx.func.signature = sig_a;
     ctx.func.name = ExternalName::user(0, func_a.as_u32());
-    {
-        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
+    ctx.func = {
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(ctx.func);
         let ebb = bcx.create_ebb();
 
         bcx.switch_to_block(ebb);
@@ -36,15 +35,15 @@ fn main() {
         let add = bcx.ins().iadd(cst, param);
         bcx.ins().return_(&[add]);
         bcx.seal_all_blocks();
-        bcx.finalize();
-    }
+        bcx.finalize()
+    };
     module.define_function(func_a, &mut ctx).unwrap();
     module.clear_context(&mut ctx);
 
     ctx.func.signature = sig_b;
     ctx.func.name = ExternalName::user(0, func_b.as_u32());
-    {
-        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
+    ctx.func = {
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(ctx.func);
         let ebb = bcx.create_ebb();
 
         bcx.switch_to_block(ebb);
@@ -58,8 +57,8 @@ fn main() {
         };
         bcx.ins().return_(&[value]);
         bcx.seal_all_blocks();
-        bcx.finalize();
-    }
+        bcx.finalize()
+    };
     module.define_function(func_b, &mut ctx).unwrap();
     module.clear_context(&mut ctx);
 
diff --git a/cranelift/simplejit/tests/basic.rs b/cranelift/simplejit/tests/basic.rs
index a3932b1d6a..ea3f9cb5fd 100644
--- a/cranelift/simplejit/tests/basic.rs
+++ b/cranelift/simplejit/tests/basic.rs
@@ -37,14 +37,16 @@ fn define_simple_function(module: &mut Module) -> FuncId {
         .unwrap();
 
     let mut ctx = Context::new();
-    ctx.func = Function::with_name_signature(ExternalName::user(0, func_id.as_u32()), sig);
-    let mut func_ctx = FunctionBuilderContext::new();
-    {
-        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
+
+    ctx.func = {
+        let func = Function::with_name_signature(ExternalName::user(0, func_id.as_u32()), sig);
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(func);
         let ebb = bcx.create_ebb();
         bcx.switch_to_block(ebb);
         bcx.ins().return_(&[]);
-    }
+        bcx.seal_all_blocks();
+        bcx.finalize()
+    };
 
     module.define_function(func_id, &mut ctx).unwrap();
 
@@ -85,11 +87,9 @@ fn switch_error() {
         call_conv: CallConv::SystemV,
     };
 
-    let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
-
-    let mut func_ctx = FunctionBuilderContext::new();
-    {
-        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut func, &mut func_ctx);
+    let func = {
+        let func = Function::with_name_signature(ExternalName::user(0, 0), sig);
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(func);
         let start = bcx.create_ebb();
         let bb0 = bcx.create_ebb();
         let bb1 = bcx.create_ebb();
@@ -134,8 +134,8 @@ fn switch_error() {
         bcx.ins().return_(&[r]);
 
         bcx.seal_all_blocks();
-        bcx.finalize();
-    }
+        bcx.finalize()
+    };
 
     let flags = settings::Flags::new(settings::builder());
     match cranelift_codegen::verify_function(&func, &flags) {
@@ -164,10 +164,9 @@ fn libcall_function() {
         .unwrap();
 
     let mut ctx = Context::new();
-    ctx.func = Function::with_name_signature(ExternalName::user(0, func_id.as_u32()), sig);
-    let mut func_ctx = FunctionBuilderContext::new();
-    {
-        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
+    ctx.func = {
+        let func = Function::with_name_signature(ExternalName::user(0, func_id.as_u32()), sig);
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(func);
         let ebb = bcx.create_ebb();
         bcx.switch_to_block(ebb);
 
@@ -189,7 +188,11 @@ fn libcall_function() {
         bcx.call_memset(module.target_config(), buffer, zero, size);
 
         bcx.ins().return_(&[]);
-    }
+
+        bcx.seal_all_blocks();
+
+        bcx.finalize()
+    };
 
     module.define_function(func_id, &mut ctx).unwrap();
 
diff --git a/cranelift/umbrella/src/lib.rs b/cranelift/umbrella/src/lib.rs
index f97ed443ce..acb0e972d8 100644
--- a/cranelift/umbrella/src/lib.rs
+++ b/cranelift/umbrella/src/lib.rs
@@ -43,7 +43,7 @@ pub mod prelude {
     pub use crate::codegen::isa;
     pub use crate::codegen::settings::{self, Configurable};
 
-    pub use crate::frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
+    pub use crate::frontend::{FunctionBuilder, Variable};
 }
 
 /// Version number of this crate.
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index b87844bfd4..664d839023 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -81,7 +81,7 @@ pub fn translate_operator(
          *  `get_global` and `set_global` are handled by the environment.
          ***********************************************************************************/
         Operator::GetGlobal { global_index } => {
-            let val = match state.get_global(builder.func, *global_index, environ)? {
+            let val = match state.get_global(&mut builder.func, *global_index, environ)? {
                 GlobalVariable::Const(val) => val,
                 GlobalVariable::Memory { gv, offset, ty } => {
                     let addr = builder.ins().global_value(environ.pointer_type(), gv);
@@ -92,7 +92,7 @@ pub fn translate_operator(
             state.push1(val);
         }
         Operator::SetGlobal { global_index } => {
-            match state.get_global(builder.func, *global_index, environ)? {
+            match state.get_global(&mut builder.func, *global_index, environ)? {
                 GlobalVariable::Const(_) => panic!("global #{} is a constant", *global_index),
                 GlobalVariable::Memory { gv, offset, ty } => {
                     let addr = builder.ins().global_value(environ.pointer_type(), gv);
@@ -367,7 +367,8 @@ pub fn translate_operator(
          * argument referring to an index in the external functions table of the module.
          ************************************************************************************/
         Operator::Call { function_index } => {
-            let (fref, num_args) = state.get_direct_func(builder.func, *function_index, environ)?;
+            let (fref, num_args) =
+                state.get_direct_func(&mut builder.func, *function_index, environ)?;
             let call = environ.translate_call(
                 builder.cursor(),
                 FuncIndex::from_u32(*function_index),
@@ -388,8 +389,8 @@ pub fn translate_operator(
         Operator::CallIndirect { index, table_index } => {
             // `index` is the index of the function's signature and `table_index` is the index of
             // the table to search the function in.
-            let (sigref, num_args) = state.get_indirect_sig(builder.func, *index, environ)?;
-            let table = state.get_table(builder.func, *table_index, environ)?;
+            let (sigref, num_args) = state.get_indirect_sig(&mut builder.func, *index, environ)?;
+            let table = state.get_table(&mut builder.func, *table_index, environ)?;
             let callee = state.pop1();
             let call = environ.translate_call_indirect(
                 builder.cursor(),
@@ -417,13 +418,13 @@ pub fn translate_operator(
             // The WebAssembly MVP only supports one linear memory, but we expect the reserved
             // argument to be a memory index.
             let heap_index = MemoryIndex::from_u32(*reserved);
-            let heap = state.get_heap(builder.func, *reserved, environ)?;
+            let heap = state.get_heap(&mut builder.func, *reserved, environ)?;
             let val = state.pop1();
             state.push1(environ.translate_memory_grow(builder.cursor(), heap_index, heap, val)?)
         }
         Operator::MemorySize { reserved } => {
             let heap_index = MemoryIndex::from_u32(*reserved);
-            let heap = state.get_heap(builder.func, *reserved, environ)?;
+            let heap = state.get_heap(&mut builder.func, *reserved, environ)?;
             state.push1(environ.translate_memory_size(builder.cursor(), heap_index, heap)?);
         }
         /******************************* Load instructions ***********************************
@@ -1231,7 +1232,7 @@ fn translate_load(
 ) -> WasmResult<()> {
     let addr32 = state.pop1();
     // We don't yet support multiple linear memories.
-    let heap = state.get_heap(builder.func, 0, environ)?;
+    let heap = state.get_heap(&mut builder.func, 0, environ)?;
     let (base, offset) = get_heap_addr(heap, addr32, offset, environ.pointer_type(), builder);
     // Note that we don't set `is_aligned` here, even if the load instruction's
     // alignment immediate says it's aligned, because WebAssembly's immediate
@@ -1256,7 +1257,7 @@ fn translate_store(
     let val_ty = builder.func.dfg.value_type(val);
 
     // We don't yet support multiple linear memories.
-    let heap = state.get_heap(builder.func, 0, environ)?;
+    let heap = state.get_heap(&mut builder.func, 0, environ)?;
     let (base, offset) = get_heap_addr(heap, addr32, offset, environ.pointer_type(), builder);
     // See the comments in `translate_load` about the flags.
     let flags = MemFlags::new();
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index fcec4e648c..8199cadee6 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -533,8 +533,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
                 func.collect_debug_info();
             }
             self.trans
-                .translate(body_bytes, body_offset, &mut func, &mut func_environ)?;
-            func
+                .translate(body_bytes, body_offset, func, &mut func_environ)?
         };
         self.func_bytecode_sizes.push(body_bytes.len());
         self.info.function_bodies.push(func);
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index b49cffd474..ed5c51e59f 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -12,7 +12,7 @@ use crate::wasm_unsupported;
 use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir::{self, Ebb, InstBuilder, ValueLabel};
 use cranelift_codegen::timing;
-use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
+use cranelift_frontend::{FunctionBuilder, Variable};
 use log::info;
 use wasmparser::{self, BinaryReader};
 
@@ -22,7 +22,7 @@ use wasmparser::{self, BinaryReader};
 /// by a `FuncEnvironment` object. A single translator instance can be reused to translate multiple
 /// functions which will reduce heap allocation traffic.
 pub struct FuncTranslator {
-    func_ctx: FunctionBuilderContext,
+    builder: FunctionBuilder,
     state: TranslationState,
 }
 
@@ -30,7 +30,7 @@ impl FuncTranslator {
     /// Create a new translator.
     pub fn new() -> Self {
         Self {
-            func_ctx: FunctionBuilderContext::new(),
+            builder: FunctionBuilder::new(ir::Function::new()),
             state: TranslationState::new(),
         }
     }
@@ -57,9 +57,9 @@ impl FuncTranslator {
         &mut self,
         code: &[u8],
         code_offset: usize,
-        func: &mut ir::Function,
+        func: ir::Function,
         environ: &mut FE,
-    ) -> WasmResult<()> {
+    ) -> WasmResult {
         self.translate_from_reader(
             BinaryReader::new_with_offset(code, code_offset),
             func,
@@ -71,9 +71,9 @@ impl FuncTranslator {
     pub fn translate_from_reader(
         &mut self,
         mut reader: BinaryReader,
-        func: &mut ir::Function,
+        func: ir::Function,
         environ: &mut FE,
-    ) -> WasmResult<()> {
+    ) -> WasmResult {
         let _tt = timing::wasm_translate_function();
         info!(
             "translate({} bytes, {}{})",
@@ -84,31 +84,32 @@ impl FuncTranslator {
         debug_assert_eq!(func.dfg.num_ebbs(), 0, "Function must be empty");
         debug_assert_eq!(func.dfg.num_insts(), 0, "Function must be empty");
 
-        // This clears the `FunctionBuilderContext`.
-        let mut builder = FunctionBuilder::new(func, &mut self.func_ctx);
-        builder.set_srcloc(cur_srcloc(&reader));
-        let entry_block = builder.create_ebb();
-        builder.append_ebb_params_for_function_params(entry_block);
-        builder.switch_to_block(entry_block); // This also creates values for the arguments.
-        builder.seal_block(entry_block); // Declare all predecessors known.
+        self.builder.func = func;
+        self.builder.set_srcloc(cur_srcloc(&reader));
+        let entry_block = self.builder.create_ebb();
+        self.builder
+            .append_ebb_params_for_function_params(entry_block);
+        self.builder.switch_to_block(entry_block); // This also creates values for the arguments.
+        self.builder.seal_block(entry_block); // Declare all predecessors known.
 
         // Make sure the entry block is inserted in the layout before we make any callbacks to
         // `environ`. The callback functions may need to insert things in the entry block.
-        builder.ensure_inserted_ebb();
+        self.builder.ensure_inserted_ebb();
 
-        let num_params = declare_wasm_parameters(&mut builder, entry_block);
+        let num_params = declare_wasm_parameters(&mut self.builder, entry_block);
 
         // Set up the translation state with a single pushed control block representing the whole
         // function and its return values.
-        let exit_block = builder.create_ebb();
-        builder.append_ebb_params_for_function_returns(exit_block);
-        self.state.initialize(&builder.func.signature, exit_block);
+        let exit_block = self.builder.create_ebb();
+        self.builder
+            .append_ebb_params_for_function_returns(exit_block);
+        self.state
+            .initialize(&self.builder.func.signature, exit_block);
 
-        parse_local_decls(&mut reader, &mut builder, num_params, environ)?;
-        parse_function_body(reader, &mut builder, &mut self.state, environ)?;
+        parse_local_decls(&mut reader, &mut self.builder, num_params, environ)?;
+        parse_function_body(reader, &mut self.builder, &mut self.state, environ)?;
 
-        builder.finalize();
-        Ok(())
+        Ok(self.builder.finalize())
     }
 }
 
@@ -288,8 +289,8 @@ mod tests {
         ctx.func.signature.params.push(ir::AbiParam::new(I32));
         ctx.func.signature.returns.push(ir::AbiParam::new(I32));
 
-        trans
-            .translate(&BODY, 0, &mut ctx.func, &mut runtime.func_env())
+        ctx.func = trans
+            .translate(&BODY, 0, ctx.func, &mut runtime.func_env())
             .unwrap();
         debug!("{}", ctx.func.display(None));
         ctx.verify(&flags).unwrap();
@@ -327,8 +328,8 @@ mod tests {
         ctx.func.signature.params.push(ir::AbiParam::new(I32));
         ctx.func.signature.returns.push(ir::AbiParam::new(I32));
 
-        trans
-            .translate(&BODY, 0, &mut ctx.func, &mut runtime.func_env())
+        ctx.func = trans
+            .translate(&BODY, 0, ctx.func, &mut runtime.func_env())
             .unwrap();
         debug!("{}", ctx.func.display(None));
         ctx.verify(&flags).unwrap();
@@ -374,8 +375,8 @@ mod tests {
         ctx.func.name = ir::ExternalName::testcase("infloop");
         ctx.func.signature.returns.push(ir::AbiParam::new(I32));
 
-        trans
-            .translate(&BODY, 0, &mut ctx.func, &mut runtime.func_env())
+        ctx.func = trans
+            .translate(&BODY, 0, ctx.func, &mut runtime.func_env())
             .unwrap();
         debug!("{}", ctx.func.display(None));
         ctx.verify(&flags).unwrap();

From dfda794f559707545b8fdf3d1afd300ed41afb4c Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 13 Aug 2019 13:11:22 -0700
Subject: [PATCH 2684/3084] Add a custom section hook to `ModuleEnvironment`

This commit adds a hook to the `ModuleEnvironment` trait to learn when a
custom section in a wasm file is read. This hook can in theory be used
to parse and handle custom sections as they appear in the wasm file
without having to re-iterate over the wasm file after cranelift has
already parsed the wasm file.

The `translate_module` function is now less strict in that it doesn't
require sections to be in a particular order, but it's figured that the
wasm file is already validated elsewhere to verify the section order.
---
 cranelift/wasm/src/environ/spec.rs      |  10 ++
 cranelift/wasm/src/module_translator.rs | 193 +++++++++---------------
 2 files changed, 79 insertions(+), 124 deletions(-)

diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 59208ea562..b653091153 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -467,4 +467,14 @@ pub trait ModuleEnvironment<'data> {
         offset: usize,
         data: &'data [u8],
     ) -> WasmResult<()>;
+
+    /// Indicates that a custom section has been found in the wasm file
+    fn custom_section(
+        &mut self,
+        name: &'data str,
+        data: &'data [u8],
+    ) -> WasmResult<()> {
+        drop((name, data));
+        Ok(())
+    }
 }
diff --git a/cranelift/wasm/src/module_translator.rs b/cranelift/wasm/src/module_translator.rs
index b35d1d4e38..1ff8dd2dd8 100644
--- a/cranelift/wasm/src/module_translator.rs
+++ b/cranelift/wasm/src/module_translator.rs
@@ -18,133 +18,78 @@ pub fn translate_module<'data>(
     let _tt = timing::wasm_translate_module();
     let mut reader = ModuleReader::new(data)?;
 
-    reader.skip_custom_sections()?;
-    if reader.eof() {
-        return Ok(());
-    }
-    let mut section = reader.read()?;
+    while !reader.eof() {
+        let section = reader.read()?;
+        match section.code {
+            SectionCode::Type => {
+                let types = section.get_type_section_reader()?;
+                parse_type_section(types, environ)?;
+            }
 
-    if let SectionCode::Type = section.code {
-        let types = section.get_type_section_reader()?;
-        parse_type_section(types, environ)?;
+            SectionCode::Import => {
+                let imports = section.get_import_section_reader()?;
+                parse_import_section(imports, environ)?;
+            }
 
-        reader.skip_custom_sections()?;
-        if reader.eof() {
-            return Ok(());
+            SectionCode::Function => {
+                let functions = section.get_function_section_reader()?;
+                parse_function_section(functions, environ)?;
+            }
+
+            SectionCode::Table => {
+                let tables = section.get_table_section_reader()?;
+                parse_table_section(tables, environ)?;
+            }
+
+            SectionCode::Memory => {
+                let memories = section.get_memory_section_reader()?;
+                parse_memory_section(memories, environ)?;
+            }
+
+            SectionCode::Global => {
+                let globals = section.get_global_section_reader()?;
+                parse_global_section(globals, environ)?;
+            }
+
+            SectionCode::Export => {
+                let exports = section.get_export_section_reader()?;
+                parse_export_section(exports, environ)?;
+            }
+
+            SectionCode::Start => {
+                let start = section.get_start_section_content()?;
+                parse_start_section(start, environ)?;
+            }
+
+            SectionCode::Element => {
+                let elements = section.get_element_section_reader()?;
+                parse_element_section(elements, environ)?;
+            }
+
+            SectionCode::Code => {
+                let code = section.get_code_section_reader()?;
+                parse_code_section(code, environ)?;
+            }
+
+            SectionCode::Data => {
+                let data = section.get_data_section_reader()?;
+                parse_data_section(data, environ)?;
+            }
+
+            SectionCode::DataCount => {
+                return Err(WasmError::InvalidWebAssembly {
+                    message: "don't know how to handle the data count section yet",
+                    offset: reader.current_position(),
+                });
+            }
+
+            SectionCode::Custom { name, kind: _ } => {
+                let mut reader = section.get_binary_reader();
+                let len = reader.bytes_remaining();
+                let payload = reader.read_bytes(len)?;
+                environ.custom_section(name, payload)?;
+            }
         }
-        section = reader.read()?;
-    }
-
-    if let SectionCode::Import = section.code {
-        let imports = section.get_import_section_reader()?;
-        parse_import_section(imports, environ)?;
-
-        reader.skip_custom_sections()?;
-        if reader.eof() {
-            return Ok(());
-        }
-        section = reader.read()?;
-    }
-
-    if let SectionCode::Function = section.code {
-        let functions = section.get_function_section_reader()?;
-        parse_function_section(functions, environ)?;
-
-        reader.skip_custom_sections()?;
-        if reader.eof() {
-            return Ok(());
-        }
-        section = reader.read()?;
-    }
-
-    if let SectionCode::Table = section.code {
-        let tables = section.get_table_section_reader()?;
-        parse_table_section(tables, environ)?;
-
-        reader.skip_custom_sections()?;
-        if reader.eof() {
-            return Ok(());
-        }
-        section = reader.read()?;
-    }
-
-    if let SectionCode::Memory = section.code {
-        let memories = section.get_memory_section_reader()?;
-        parse_memory_section(memories, environ)?;
-
-        reader.skip_custom_sections()?;
-        if reader.eof() {
-            return Ok(());
-        }
-        section = reader.read()?;
-    }
-
-    if let SectionCode::Global = section.code {
-        let globals = section.get_global_section_reader()?;
-        parse_global_section(globals, environ)?;
-
-        reader.skip_custom_sections()?;
-        if reader.eof() {
-            return Ok(());
-        }
-        section = reader.read()?;
-    }
-
-    if let SectionCode::Export = section.code {
-        let exports = section.get_export_section_reader()?;
-        parse_export_section(exports, environ)?;
-
-        reader.skip_custom_sections()?;
-        if reader.eof() {
-            return Ok(());
-        }
-        section = reader.read()?;
-    }
-
-    if let SectionCode::Start = section.code {
-        let start = section.get_start_section_content()?;
-        parse_start_section(start, environ)?;
-
-        reader.skip_custom_sections()?;
-        if reader.eof() {
-            return Ok(());
-        }
-        section = reader.read()?;
-    }
-
-    if let SectionCode::Element = section.code {
-        let elements = section.get_element_section_reader()?;
-        parse_element_section(elements, environ)?;
-
-        reader.skip_custom_sections()?;
-        if reader.eof() {
-            return Ok(());
-        }
-        section = reader.read()?;
-    }
-
-    if let SectionCode::Code = section.code {
-        let code = section.get_code_section_reader()?;
-        parse_code_section(code, environ)?;
-
-        reader.skip_custom_sections()?;
-        if reader.eof() {
-            return Ok(());
-        }
-        section = reader.read()?;
-    }
-
-    if let SectionCode::Data = section.code {
-        let data = section.get_data_section_reader()?;
-        parse_data_section(data, environ)?;
-    }
-
-    reader.skip_custom_sections()?;
-    if !reader.eof() {
-        return Err(WasmError::InvalidWebAssembly {
-            message: "sections must occur at most once and in the prescribed order",
-            offset: reader.current_position(),
-        });
     }
 
     Ok(())

From 705bfacf109a718b616960e0fde24fb8a0a176e1 Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 13 Aug 2019 13:14:03 -0700
Subject: [PATCH 2685/3084] rustfmt

---
 cranelift/wasm/src/environ/spec.rs | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index b653091153..e01c9b3cf9 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -469,11 +469,7 @@ pub trait ModuleEnvironment<'data> {
     ) -> WasmResult<()>;
 
     /// Indicates that a custom section has been found in the wasm file
-    fn custom_section(
-        &mut self,
-        name: &'data str,
-        data: &'data [u8],
-    ) -> WasmResult<()> {
+    fn custom_section(&mut self, name: &'data str, data: &'data [u8]) -> WasmResult<()> {
         drop((name, data));
         Ok(())
     }

From 955cdd5f83bd694b1c87d7c69e25cbbee06d675e Mon Sep 17 00:00:00 2001
From: Julian Seward 
Date: Sat, 7 Sep 2019 17:34:08 +0200
Subject: [PATCH 2686/3084] VirtRegs::find: use SmallVec instead of Vec for
 val_stack.

Pushing on the `val_stack` vector is CL's biggest source of calls to
malloc/realloc/free, by some margin.  It accounts for about 27.7% of all heap
blocks allocated when compiling wasm_lua_binarytrees.  This change removes
pretty much all dynamic allocation by changing to a SmallVec<[Value; 8]>
instead.  A fixed size of 4 gets all the gains to be had, in testing, so 8
gives some safety margin and is harmless from a stack-use perspective: 8
Values will occupy 32 bytes.

As a bonus, this change also reduces the compiler's dynamic instruction count
by about 0.5%.
---
 cranelift/codegen/Cargo.toml               | 1 +
 cranelift/codegen/src/regalloc/virtregs.rs | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 2c5ccb4d2e..2d81a660e8 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -21,6 +21,7 @@ hashmap_core = { version = "0.1.9", optional = true }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
+smallvec = { version = "0.6.10" }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
 # machine code. Integration tests that need external dependencies can be
diff --git a/cranelift/codegen/src/regalloc/virtregs.rs b/cranelift/codegen/src/regalloc/virtregs.rs
index 584ad9c53a..fc267c3f61 100644
--- a/cranelift/codegen/src/regalloc/virtregs.rs
+++ b/cranelift/codegen/src/regalloc/virtregs.rs
@@ -21,6 +21,7 @@ use crate::packed_option::PackedOption;
 use crate::ref_slice::ref_slice;
 use core::cmp::Ordering;
 use core::fmt;
+use smallvec::SmallVec;
 use std::vec::Vec;
 
 /// A virtual register reference.
@@ -292,7 +293,7 @@ impl VirtRegs {
     /// Find the leader value and rank of the set containing `v`.
     /// Compress the path if needed.
     fn find(&mut self, mut val: Value) -> (Value, u32) {
-        let mut val_stack = vec![];
+        let mut val_stack = SmallVec::<[Value; 8]>::new();
         let found = loop {
             match UFEntry::decode(self.union_find[val]) {
                 UFEntry::Rank(rank) => break (val, rank),

From c6a4c60a0f3c028b498860e712963da23330527e Mon Sep 17 00:00:00 2001
From: Julian Seward 
Date: Sat, 7 Sep 2019 19:12:42 +0200
Subject: [PATCH 2687/3084] EbbHeaderBlockData::predecessors: use SmallVec
 instead of Vec

Allocations associated with pushes to EbbHeaderBlockData::predecessors account
for 4.9% of all heap allocation (calls) in CL.  This change avoids almost all
of them by changing it to be a SmallVec<[PredBlock; 4]>.  Dynamic instruction
count falls by 0.15%.
---
 cranelift/frontend/Cargo.toml |  1 +
 cranelift/frontend/src/ssa.rs | 12 ++++++++----
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 3ef5a45641..3cd8882201 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -15,6 +15,7 @@ cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
+smallvec = { version = "0.6.10" }
 
 [features]
 default = ["std"]
diff --git a/cranelift/frontend/src/ssa.rs b/cranelift/frontend/src/ssa.rs
index 2bdd068515..43300f4ded 100644
--- a/cranelift/frontend/src/ssa.rs
+++ b/cranelift/frontend/src/ssa.rs
@@ -16,6 +16,7 @@ use cranelift_codegen::ir::types::{F32, F64};
 use cranelift_codegen::ir::{Ebb, Function, Inst, InstBuilder, InstructionData, Type, Value};
 use cranelift_codegen::packed_option::PackedOption;
 use cranelift_codegen::packed_option::ReservedValue;
+use smallvec::SmallVec;
 use std::vec::Vec;
 
 /// Structure containing the data relevant the construction of SSA for a given function.
@@ -123,9 +124,11 @@ impl PredBlock {
     }
 }
 
+type PredBlockSmallVec = SmallVec<[PredBlock; 4]>;
+
 struct EbbHeaderBlockData {
     // The predecessors of the Ebb header block, with the block and branch instruction.
-    predecessors: Vec,
+    predecessors: PredBlockSmallVec,
     // A ebb header block is sealed if all of its predecessors have been declared.
     sealed: bool,
     // The ebb which this block is part of.
@@ -366,7 +369,7 @@ impl SSABuilder {
     /// Predecessors have to be added with `declare_ebb_predecessor`.
     pub fn declare_ebb_header_block(&mut self, ebb: Ebb) -> Block {
         let block = self.blocks.push(BlockData::EbbHeader(EbbHeaderBlockData {
-            predecessors: Vec::new(),
+            predecessors: PredBlockSmallVec::new(),
             sealed: false,
             ebb,
             undef_variables: Vec::new(),
@@ -587,7 +590,8 @@ impl SSABuilder {
                 // There is disagreement in the predecessors on which value to use so we have
                 // to keep the ebb argument. To avoid borrowing `self` for the whole loop,
                 // temporarily detach the predecessors list and replace it with an empty list.
-                let mut preds = mem::replace(self.predecessors_mut(dest_ebb), Vec::new());
+                let mut preds =
+                    mem::replace(self.predecessors_mut(dest_ebb), PredBlockSmallVec::new());
                 for &mut PredBlock {
                     block: ref mut pred_block,
                     branch: ref mut last_inst,
@@ -699,7 +703,7 @@ impl SSABuilder {
     }
 
     /// Same as predecessors, but for &mut.
-    fn predecessors_mut(&mut self, ebb: Ebb) -> &mut Vec {
+    fn predecessors_mut(&mut self, ebb: Ebb) -> &mut PredBlockSmallVec {
         let block = self.header_block(ebb);
         match self.blocks[block] {
             BlockData::EbbBody { .. } => panic!("should not happen"),

From d2443a75f352bde49dc529420cc3d8e4bb370c19 Mon Sep 17 00:00:00 2001
From: Julian Seward 
Date: Sat, 7 Sep 2019 18:07:51 +0200
Subject: [PATCH 2688/3084] legalizer/split.rs: simplify_branch_arguments: use
 SmallVec instead of Vec

This function is responsible for 8.5% of all heap allocation (calls) in CL.
This change avoids almost all of them by using a SmallVec::<[Value; 32]>
instead.  Dynamic instruction count falls by 0.25%.  The fixed size of 32 was
arrived at after profiling with fixed sizes of 1, 2, 4, 8, 16, 32, 64 and 128.
32 is as high as I can push it without the instruction count starting to creep
up again, and gets almost all the block-reduction win of 64 and 128.
---
 cranelift/codegen/src/legalizer/split.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/legalizer/split.rs b/cranelift/codegen/src/legalizer/split.rs
index f16dae161b..9f689922f1 100644
--- a/cranelift/codegen/src/legalizer/split.rs
+++ b/cranelift/codegen/src/legalizer/split.rs
@@ -68,6 +68,7 @@ use crate::cursor::{Cursor, CursorPosition, FuncCursor};
 use crate::flowgraph::{BasicBlock, ControlFlowGraph};
 use crate::ir::{self, Ebb, Inst, InstBuilder, InstructionData, Opcode, Type, Value, ValueDef};
 use core::iter;
+use smallvec::SmallVec;
 use std::vec::Vec;
 
 /// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values
@@ -373,7 +374,7 @@ fn resolve_splits(dfg: &ir::DataFlowGraph, value: Value) -> Value {
 /// After legalizing the instructions computing the value that was split, it is likely that we can
 /// avoid depending on the split instruction. Its input probably comes from a concatenation.
 pub fn simplify_branch_arguments(dfg: &mut ir::DataFlowGraph, branch: Inst) {
-    let mut new_args = Vec::new();
+    let mut new_args = SmallVec::<[Value; 32]>::new();
 
     for &arg in dfg.inst_args(branch) {
         let new_arg = resolve_splits(dfg, arg);

From 90b0b86f5c4ebe8956d31d2597a04505b91101d9 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 6 Sep 2019 16:14:55 +0200
Subject: [PATCH 2689/3084] Simplify isa_builder macro

---
 cranelift/codegen/src/isa/mod.rs | 21 ++++++++++-----------
 1 file changed, 10 insertions(+), 11 deletions(-)

diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index 20e38ddeb4..c36d29ce2b 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -88,18 +88,17 @@ pub mod registers;
 mod stack;
 
 /// Returns a builder that can create a corresponding `TargetIsa`
-/// or `Err(LookupError::Unsupported)` if not enabled.
+/// or `Err(LookupError::SupportDisabled)` if not enabled.
 macro_rules! isa_builder {
-    ($name:ident, $feature:tt) => {{
+    ($name: ident, $feature: tt, $triple: ident) => {{
         #[cfg(feature = $feature)]
-        fn $name(triple: Triple) -> Result {
-            Ok($name::isa_builder(triple))
-        };
+        {
+            Ok($name::isa_builder($triple))
+        }
         #[cfg(not(feature = $feature))]
-        fn $name(_triple: Triple) -> Result {
+        {
             Err(LookupError::SupportDisabled)
         }
-        $name
     }};
 }
 
@@ -107,12 +106,12 @@ macro_rules! isa_builder {
 /// Return a builder that can create a corresponding `TargetIsa`.
 pub fn lookup(triple: Triple) -> Result {
     match triple.architecture {
-        Architecture::Riscv32 | Architecture::Riscv64 => isa_builder!(riscv, "riscv")(triple),
+        Architecture::Riscv32 | Architecture::Riscv64 => isa_builder!(riscv, "riscv", triple),
         Architecture::I386 | Architecture::I586 | Architecture::I686 | Architecture::X86_64 => {
-            isa_builder!(x86, "x86")(triple)
+            isa_builder!(x86, "x86", triple)
         }
-        Architecture::Arm { .. } => isa_builder!(arm32, "arm32")(triple),
-        Architecture::Aarch64 { .. } => isa_builder!(arm64, "arm64")(triple),
+        Architecture::Arm { .. } => isa_builder!(arm32, "arm32", triple),
+        Architecture::Aarch64 { .. } => isa_builder!(arm64, "arm64", triple),
         _ => Err(LookupError::Unsupported),
     }
 }

From 3293ca6b69155c5ce60947f5d68428a091b34d94 Mon Sep 17 00:00:00 2001
From: Philip Craig 
Date: Mon, 6 May 2019 16:34:16 +1000
Subject: [PATCH 2690/3084] Add cranelift-object

---
 cranelift/Cargo.toml                |   1 +
 cranelift/codegen/src/ir/libcall.rs |   2 +-
 cranelift/faerie/src/backend.rs     |  25 +-
 cranelift/module/src/backend.rs     |  17 +-
 cranelift/module/src/module.rs      |  46 ++-
 cranelift/object/Cargo.toml         |  20 +
 cranelift/object/LICENSE            | 220 +++++++++++
 cranelift/object/README.md          |   4 +
 cranelift/object/src/backend.rs     | 584 ++++++++++++++++++++++++++++
 cranelift/object/src/lib.rs         |  38 ++
 cranelift/object/src/traps.rs       |  32 ++
 cranelift/simplejit/src/backend.rs  |  10 +-
 12 files changed, 977 insertions(+), 22 deletions(-)
 create mode 100644 cranelift/object/Cargo.toml
 create mode 100644 cranelift/object/LICENSE
 create mode 100644 cranelift/object/README.md
 create mode 100644 cranelift/object/src/backend.rs
 create mode 100644 cranelift/object/src/lib.rs
 create mode 100644 cranelift/object/src/traps.rs

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 8fee3b6dbe..c48ef7f71f 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -29,6 +29,7 @@ cranelift-native = { path = "cranelift-native", version = "0.42.0" }
 cranelift-filetests = { path = "cranelift-filetests", version = "0.42.0" }
 cranelift-module = { path = "cranelift-module", version = "0.42.0" }
 cranelift-faerie = { path = "cranelift-faerie", version = "0.42.0" }
+cranelift-object = { path = "cranelift-object", version = "0.42.0" }
 cranelift-simplejit = { path = "cranelift-simplejit", version = "0.42.0" }
 cranelift-preopt = { path = "cranelift-preopt", version = "0.42.0" }
 cranelift = { path = "cranelift-umbrella", version = "0.42.0" }
diff --git a/cranelift/codegen/src/ir/libcall.rs b/cranelift/codegen/src/ir/libcall.rs
index 59fb951a32..1e19891221 100644
--- a/cranelift/codegen/src/ir/libcall.rs
+++ b/cranelift/codegen/src/ir/libcall.rs
@@ -18,7 +18,7 @@ use serde::{Deserialize, Serialize};
 /// convention in the embedding VM's runtime library.
 ///
 /// This list is likely to grow over time.
-#[derive(Copy, Clone, Debug, PartialEq, Eq)]
+#[derive(Copy, Clone, Debug, PartialEq, Eq, Hash)]
 #[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub enum LibCall {
     /// probe for stack overflow. These are emitted for functions which need
diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index 4a4d189612..585866ca98 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -8,8 +8,8 @@ use cranelift_codegen::binemit::{
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::{self, binemit, ir};
 use cranelift_module::{
-    Backend, DataContext, DataDescription, Init, Linkage, ModuleError, ModuleNamespace,
-    ModuleResult,
+    Backend, DataContext, DataDescription, DataId, FuncId, Init, Linkage, ModuleError,
+    ModuleNamespace, ModuleResult,
 };
 use faerie;
 use failure::Error;
@@ -123,13 +123,20 @@ impl Backend for FaerieBackend {
         &*self.isa
     }
 
-    fn declare_function(&mut self, name: &str, linkage: Linkage) {
+    fn declare_function(&mut self, _id: FuncId, name: &str, linkage: Linkage) {
         self.artifact
             .declare(name, translate_function_linkage(linkage))
             .expect("inconsistent declarations");
     }
 
-    fn declare_data(&mut self, name: &str, linkage: Linkage, writable: bool, align: Option) {
+    fn declare_data(
+        &mut self,
+        _id: DataId,
+        name: &str,
+        linkage: Linkage,
+        writable: bool,
+        align: Option,
+    ) {
         self.artifact
             .declare(name, translate_data_linkage(linkage, writable, align))
             .expect("inconsistent declarations");
@@ -137,6 +144,7 @@ impl Backend for FaerieBackend {
 
     fn define_function(
         &mut self,
+        _id: FuncId,
         name: &str,
         ctx: &cranelift_codegen::Context,
         namespace: &ModuleNamespace,
@@ -194,6 +202,7 @@ impl Backend for FaerieBackend {
 
     fn define_data(
         &mut self,
+        _id: DataId,
         name: &str,
         _writable: bool,
         _align: Option,
@@ -274,6 +283,7 @@ impl Backend for FaerieBackend {
 
     fn finalize_function(
         &mut self,
+        _id: FuncId,
         _func: &FaerieCompiledFunction,
         _namespace: &ModuleNamespace,
     ) {
@@ -284,7 +294,12 @@ impl Backend for FaerieBackend {
         // Nothing to do.
     }
 
-    fn finalize_data(&mut self, _data: &FaerieCompiledData, _namespace: &ModuleNamespace) {
+    fn finalize_data(
+        &mut self,
+        _id: DataId,
+        _data: &FaerieCompiledData,
+        _namespace: &ModuleNamespace,
+    ) {
         // Nothing to do.
     }
 
diff --git a/cranelift/module/src/backend.rs b/cranelift/module/src/backend.rs
index 316c0a3e2b..89b610bdba 100644
--- a/cranelift/module/src/backend.rs
+++ b/cranelift/module/src/backend.rs
@@ -1,6 +1,8 @@
 //! Defines the `Backend` trait.
 
 use crate::DataContext;
+use crate::DataId;
+use crate::FuncId;
 use crate::Linkage;
 use crate::ModuleNamespace;
 use crate::ModuleResult;
@@ -56,16 +58,24 @@ where
     fn isa(&self) -> &dyn TargetIsa;
 
     /// Declare a function.
-    fn declare_function(&mut self, name: &str, linkage: Linkage);
+    fn declare_function(&mut self, id: FuncId, name: &str, linkage: Linkage);
 
     /// Declare a data object.
-    fn declare_data(&mut self, name: &str, linkage: Linkage, writable: bool, align: Option);
+    fn declare_data(
+        &mut self,
+        id: DataId,
+        name: &str,
+        linkage: Linkage,
+        writable: bool,
+        align: Option,
+    );
 
     /// Define a function, producing the function body from the given `Context`.
     ///
     /// Functions must be declared before being defined.
     fn define_function(
         &mut self,
+        id: FuncId,
         name: &str,
         ctx: &Context,
         namespace: &ModuleNamespace,
@@ -77,6 +87,7 @@ where
     /// Data objects must be declared before being defined.
     fn define_data(
         &mut self,
+        id: DataId,
         name: &str,
         writable: bool,
         align: Option,
@@ -107,6 +118,7 @@ where
     /// and `Export` entities referenced to be defined.
     fn finalize_function(
         &mut self,
+        id: FuncId,
         func: &Self::CompiledFunction,
         namespace: &ModuleNamespace,
     ) -> Self::FinalizedFunction;
@@ -118,6 +130,7 @@ where
     /// `Local` and `Export` entities referenced to be defined.
     fn finalize_data(
         &mut self,
+        id: DataId,
         data: &Self::CompiledData,
         namespace: &ModuleNamespace,
     ) -> Self::FinalizedData;
diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index d9cb1e00c5..f19dcfab21 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -224,26 +224,32 @@ impl ModuleContents
 where
     B: Backend,
 {
-    fn get_function_info(&self, name: &ir::ExternalName) -> &ModuleFunction {
+    fn get_function_id(&self, name: &ir::ExternalName) -> FuncId {
         if let ir::ExternalName::User { namespace, index } = *name {
             debug_assert_eq!(namespace, 0);
-            let func = FuncId::from_u32(index);
-            &self.functions[func]
+            FuncId::from_u32(index)
         } else {
             panic!("unexpected ExternalName kind {}", name)
         }
     }
 
-    /// Get the `DataDeclaration` for the function named by `name`.
-    fn get_data_info(&self, name: &ir::ExternalName) -> &ModuleData {
+    fn get_data_id(&self, name: &ir::ExternalName) -> DataId {
         if let ir::ExternalName::User { namespace, index } = *name {
             debug_assert_eq!(namespace, 1);
-            let data = DataId::from_u32(index);
-            &self.data_objects[data]
+            DataId::from_u32(index)
         } else {
             panic!("unexpected ExternalName kind {}", name)
         }
     }
+
+    fn get_function_info(&self, name: &ir::ExternalName) -> &ModuleFunction {
+        &self.functions[self.get_function_id(name)]
+    }
+
+    /// Get the `DataDeclaration` for the function named by `name`.
+    fn get_data_info(&self, name: &ir::ExternalName) -> &ModuleData {
+        &self.data_objects[self.get_data_id(name)]
+    }
 }
 
 /// This provides a view to the state of a module which allows `ir::ExternalName`s to be translated
@@ -259,12 +265,22 @@ impl<'a, B> ModuleNamespace<'a, B>
 where
     B: Backend,
 {
+    /// Get the `FuncId` for the function named by `name`.
+    pub fn get_function_id(&self, name: &ir::ExternalName) -> FuncId {
+        self.contents.get_function_id(name)
+    }
+
+    /// Get the `DataId` for the data object named by `name`.
+    pub fn get_data_id(&self, name: &ir::ExternalName) -> DataId {
+        self.contents.get_data_id(name)
+    }
+
     /// Get the `FunctionDeclaration` for the function named by `name`.
     pub fn get_function_decl(&self, name: &ir::ExternalName) -> &FunctionDeclaration {
         &self.contents.get_function_info(name).decl
     }
 
-    /// Get the `DataDeclaration` for the function named by `name`.
+    /// Get the `DataDeclaration` for the data object named by `name`.
     pub fn get_data_decl(&self, name: &ir::ExternalName) -> &DataDeclaration {
         &self.contents.get_data_info(name).decl
     }
@@ -407,7 +423,8 @@ where
                 FuncOrDataId::Func(id) => {
                     let existing = &mut self.contents.functions[id];
                     existing.merge(linkage, signature)?;
-                    self.backend.declare_function(name, existing.decl.linkage);
+                    self.backend
+                        .declare_function(id, name, existing.decl.linkage);
                     Ok(id)
                 }
                 FuncOrDataId::Data(..) => {
@@ -424,7 +441,7 @@ where
                     compiled: None,
                 });
                 entry.insert(FuncOrDataId::Func(id));
-                self.backend.declare_function(name, linkage);
+                self.backend.declare_function(id, name, linkage);
                 Ok(id)
             }
         }
@@ -451,6 +468,7 @@ where
                     let existing = &mut self.contents.data_objects[id];
                     existing.merge(linkage, writable, align);
                     self.backend.declare_data(
+                        id,
                         name,
                         existing.decl.linkage,
                         existing.decl.writable,
@@ -474,7 +492,8 @@ where
                     compiled: None,
                 });
                 entry.insert(FuncOrDataId::Data(id));
-                self.backend.declare_data(name, linkage, writable, align);
+                self.backend
+                    .declare_data(id, name, linkage, writable, align);
                 Ok(id)
             }
         }
@@ -536,7 +555,6 @@ where
             );
             ModuleError::Compilation(e)
         })?;
-
         let info = &self.contents.functions[func];
         if info.compiled.is_some() {
             return Err(ModuleError::DuplicateDefinition(info.decl.name.clone()));
@@ -546,6 +564,7 @@ where
         }
 
         let compiled = Some(self.backend.define_function(
+            func,
             &info.decl.name,
             ctx,
             &ModuleNamespace:: {
@@ -570,6 +589,7 @@ where
                 return Err(ModuleError::InvalidImportDefinition(info.decl.name.clone()));
             }
             Some(self.backend.define_data(
+                data,
                 &info.decl.name,
                 info.decl.writable,
                 info.decl.align,
@@ -638,6 +658,7 @@ where
             let info = &self.contents.functions[func];
             debug_assert!(info.decl.linkage.is_definable());
             self.backend.finalize_function(
+                func,
                 info.compiled
                     .as_ref()
                     .expect("function must be compiled before it can be finalized"),
@@ -650,6 +671,7 @@ where
             let info = &self.contents.data_objects[data];
             debug_assert!(info.decl.linkage.is_definable());
             self.backend.finalize_data(
+                data,
                 info.compiled
                     .as_ref()
                     .expect("data object must be compiled before it can be finalized"),
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
new file mode 100644
index 0000000000..22613d65db
--- /dev/null
+++ b/cranelift/object/Cargo.toml
@@ -0,0 +1,20 @@
+[package]
+name = "cranelift-object"
+version = "0.42.0"
+authors = ["The Cranelift Project Developers"]
+description = "Emit Cranelift output to native object files with `object`"
+repository = "https://github.com/CraneStation/cranelift"
+documentation = "https://cranelift.readthedocs.io/"
+license = "Apache-2.0 WITH LLVM-exception"
+readme = "README.md"
+edition = "2018"
+
+[dependencies]
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.42.0" }
+object = { version = "0.14.0", default-features = false, features = ["write"] }
+target-lexicon = "0.8.1"
+
+[badges]
+maintenance = { status = "experimental" }
+travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/cranelift/object/LICENSE b/cranelift/object/LICENSE
new file mode 100644
index 0000000000..f9d81955f4
--- /dev/null
+++ b/cranelift/object/LICENSE
@@ -0,0 +1,220 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
+
diff --git a/cranelift/object/README.md b/cranelift/object/README.md
new file mode 100644
index 0000000000..3a0fc51356
--- /dev/null
+++ b/cranelift/object/README.md
@@ -0,0 +1,4 @@
+This crate contains a library that enables
+[Cranelift](https://crates.io/crates/cranelift)
+to emit native object (".o") files, using the
+[object](https://crates.io/crates/object) library.
diff --git a/cranelift/object/src/backend.rs b/cranelift/object/src/backend.rs
new file mode 100644
index 0000000000..78dc246324
--- /dev/null
+++ b/cranelift/object/src/backend.rs
@@ -0,0 +1,584 @@
+//! Defines `ObjectBackend`.
+
+use crate::traps::{ObjectTrapSink, ObjectTrapSite};
+use cranelift_codegen::binemit::{
+    Addend, CodeOffset, NullStackmapSink, NullTrapSink, Reloc, RelocSink,
+};
+use cranelift_codegen::entity::SecondaryMap;
+use cranelift_codegen::isa::TargetIsa;
+use cranelift_codegen::{self, binemit, ir};
+use cranelift_module::{
+    Backend, DataContext, DataDescription, DataId, FuncId, Init, Linkage, ModuleNamespace,
+    ModuleResult,
+};
+use object::write::{Object, Relocation, SectionId, StandardSection, Symbol, SymbolId};
+use object::{RelocationEncoding, RelocationKind, SymbolKind, SymbolScope};
+use std::collections::HashMap;
+use target_lexicon::PointerWidth;
+
+#[derive(Debug)]
+/// Setting to enable collection of traps. Setting this to `Enabled` in
+/// `ObjectBuilder` means that `ObjectProduct` will contains trap sites.
+pub enum ObjectTrapCollection {
+    /// `ObjectProduct::traps` will be empty
+    Disabled,
+    /// `ObjectProduct::traps` will contain trap sites
+    Enabled,
+}
+
+/// A builder for `ObjectBackend`.
+pub struct ObjectBuilder {
+    isa: Box,
+    name: String,
+    collect_traps: ObjectTrapCollection,
+    libcall_names: Box String>,
+    function_alignment: u64,
+}
+
+impl ObjectBuilder {
+    /// Create a new `ObjectBuilder` using the given Cranelift target, that
+    /// can be passed to
+    /// [`Module::new`](cranelift_module/struct.Module.html#method.new].
+    ///
+    /// `collect_traps` setting determines whether trap information is collected in the
+    /// `ObjectProduct`.
+    ///
+    /// The `libcall_names` function provides a way to translate `cranelift_codegen`'s `ir::LibCall`
+    /// enum to symbols. LibCalls are inserted in the IR as part of the legalization for certain
+    /// floating point instructions, and for stack probes. If you don't know what to use for this
+    /// argument, use `cranelift_module::default_libcall_names()`.
+    pub fn new(
+        isa: Box,
+        name: String,
+        collect_traps: ObjectTrapCollection,
+        libcall_names: Box String>,
+    ) -> ModuleResult {
+        Ok(Self {
+            isa,
+            name,
+            collect_traps,
+            libcall_names,
+            function_alignment: 1,
+        })
+    }
+
+    /// Set the alignment used for functions.
+    pub fn function_alignment(&mut self, alignment: u64) -> &mut Self {
+        self.function_alignment = alignment;
+        self
+    }
+}
+
+/// A `ObjectBackend` implements `Backend` and emits ".o" files using the `object` library.
+///
+/// See the `ObjectBuilder` for a convenient way to construct `ObjectBackend` instances.
+pub struct ObjectBackend {
+    isa: Box,
+    object: Object,
+    functions: SecondaryMap>,
+    data_objects: SecondaryMap>,
+    traps: SecondaryMap>,
+    libcalls: HashMap,
+    libcall_names: Box String>,
+    collect_traps: ObjectTrapCollection,
+    function_alignment: u64,
+}
+
+impl Backend for ObjectBackend {
+    type Builder = ObjectBuilder;
+
+    type CompiledFunction = ObjectCompiledFunction;
+    type CompiledData = ObjectCompiledData;
+
+    // There's no need to return individual artifacts; we're writing them into
+    // the output file instead.
+    type FinalizedFunction = ();
+    type FinalizedData = ();
+
+    type Product = ObjectProduct;
+
+    /// Create a new `ObjectBackend` using the given Cranelift target.
+    fn new(builder: ObjectBuilder) -> Self {
+        let triple = builder.isa.triple();
+        let mut object = Object::new(triple.binary_format, triple.architecture);
+        object.add_file_symbol(builder.name.as_bytes().to_vec());
+        Self {
+            isa: builder.isa,
+            object,
+            functions: SecondaryMap::new(),
+            data_objects: SecondaryMap::new(),
+            traps: SecondaryMap::new(),
+            libcalls: HashMap::new(),
+            libcall_names: builder.libcall_names,
+            collect_traps: builder.collect_traps,
+            function_alignment: builder.function_alignment,
+        }
+    }
+
+    fn isa(&self) -> &dyn TargetIsa {
+        &*self.isa
+    }
+
+    fn declare_function(&mut self, id: FuncId, name: &str, linkage: Linkage) {
+        let (scope, weak) = translate_linkage(linkage);
+
+        if let Some(function) = self.functions[id] {
+            let symbol = self.object.symbol_mut(function);
+            symbol.scope = scope;
+            symbol.weak = weak;
+        } else {
+            let symbol_id = self.object.add_symbol(Symbol {
+                name: name.as_bytes().to_vec(),
+                value: 0,
+                size: 0,
+                kind: SymbolKind::Text,
+                scope,
+                weak,
+                section: None,
+            });
+            self.functions[id] = Some(symbol_id);
+        }
+    }
+
+    fn declare_data(
+        &mut self,
+        id: DataId,
+        name: &str,
+        linkage: Linkage,
+        _writable: bool,
+        _align: Option,
+    ) {
+        let (scope, weak) = translate_linkage(linkage);
+
+        if let Some(data) = self.data_objects[id] {
+            let symbol = self.object.symbol_mut(data);
+            symbol.scope = scope;
+            symbol.weak = weak;
+        } else {
+            let symbol_id = self.object.add_symbol(Symbol {
+                name: name.as_bytes().to_vec(),
+                value: 0,
+                size: 0,
+                kind: SymbolKind::Data,
+                scope,
+                weak,
+                section: None,
+            });
+            self.data_objects[id] = Some(symbol_id);
+        }
+    }
+
+    fn define_function(
+        &mut self,
+        func_id: FuncId,
+        _name: &str,
+        ctx: &cranelift_codegen::Context,
+        _namespace: &ModuleNamespace,
+        code_size: u32,
+    ) -> ModuleResult {
+        let mut code: Vec = vec![0; code_size as usize];
+        let mut reloc_sink = ObjectRelocSink::default();
+        let mut trap_sink = ObjectTrapSink::default();
+        let mut stackmap_sink = NullStackmapSink {};
+
+        if let ObjectTrapCollection::Enabled = self.collect_traps {
+            unsafe {
+                ctx.emit_to_memory(
+                    &*self.isa,
+                    code.as_mut_ptr(),
+                    &mut reloc_sink,
+                    &mut trap_sink,
+                    &mut stackmap_sink,
+                )
+            };
+        } else {
+            let mut trap_sink = NullTrapSink {};
+            unsafe {
+                ctx.emit_to_memory(
+                    &*self.isa,
+                    code.as_mut_ptr(),
+                    &mut reloc_sink,
+                    &mut trap_sink,
+                    &mut stackmap_sink,
+                )
+            };
+        }
+
+        let symbol = self.functions[func_id].unwrap();
+        let section = self.object.section_id(StandardSection::Text);
+        let offset = self
+            .object
+            .add_symbol_data(symbol, section, &code, self.function_alignment);
+        self.traps[func_id] = trap_sink.sites;
+        Ok(ObjectCompiledFunction {
+            offset,
+            size: code_size,
+            section,
+            relocs: reloc_sink.relocs,
+        })
+    }
+
+    fn define_data(
+        &mut self,
+        data_id: DataId,
+        _name: &str,
+        writable: bool,
+        align: Option,
+        data_ctx: &DataContext,
+        _namespace: &ModuleNamespace,
+    ) -> ModuleResult {
+        let &DataDescription {
+            ref init,
+            ref function_decls,
+            ref data_decls,
+            ref function_relocs,
+            ref data_relocs,
+        } = data_ctx.description();
+
+        let size = init.size();
+        let mut data = Vec::with_capacity(size);
+        match *init {
+            Init::Uninitialized => {
+                panic!("data is not initialized yet");
+            }
+            Init::Zeros { .. } => {
+                data.resize(size, 0);
+            }
+            Init::Bytes { ref contents } => {
+                data.extend_from_slice(contents);
+            }
+        }
+
+        let reloc_size = match self.isa.triple().pointer_width().unwrap() {
+            PointerWidth::U16 => 16,
+            PointerWidth::U32 => 32,
+            PointerWidth::U64 => 64,
+        };
+        let mut relocs = Vec::new();
+        for &(offset, id) in function_relocs {
+            relocs.push(RelocRecord {
+                offset,
+                name: function_decls[id].clone(),
+                kind: RelocationKind::Absolute,
+                encoding: RelocationEncoding::Generic,
+                size: reloc_size,
+                addend: 0,
+            });
+        }
+        for &(offset, id, addend) in data_relocs {
+            relocs.push(RelocRecord {
+                offset,
+                name: data_decls[id].clone(),
+                kind: RelocationKind::Absolute,
+                encoding: RelocationEncoding::Generic,
+                size: reloc_size,
+                addend,
+            });
+        }
+
+        let symbol = self.data_objects[data_id].unwrap();
+        let section = self.object.section_id(if writable {
+            StandardSection::Data
+        } else if relocs.is_empty() {
+            StandardSection::ReadOnlyData
+        } else {
+            StandardSection::ReadOnlyDataWithRel
+        });
+        let offset =
+            self.object
+                .add_symbol_data(symbol, section, &data, u64::from(align.unwrap_or(1)));
+        Ok(ObjectCompiledData {
+            offset,
+            section,
+            relocs,
+        })
+    }
+
+    fn write_data_funcaddr(
+        &mut self,
+        _data: &mut ObjectCompiledData,
+        _offset: usize,
+        _what: ir::FuncRef,
+    ) {
+        unimplemented!()
+    }
+
+    fn write_data_dataaddr(
+        &mut self,
+        _data: &mut ObjectCompiledData,
+        _offset: usize,
+        _what: ir::GlobalValue,
+        _usize: binemit::Addend,
+    ) {
+        unimplemented!()
+    }
+
+    fn finalize_function(
+        &mut self,
+        _id: FuncId,
+        func: &ObjectCompiledFunction,
+        namespace: &ModuleNamespace,
+    ) {
+        for &RelocRecord {
+            offset,
+            ref name,
+            kind,
+            encoding,
+            size,
+            addend,
+        } in &func.relocs
+        {
+            let offset = func.offset + offset as u64;
+            let symbol = self.get_symbol(namespace, name);
+            self.object
+                .add_relocation(
+                    func.section,
+                    Relocation {
+                        offset,
+                        size,
+                        kind,
+                        encoding,
+                        symbol,
+                        addend,
+                    },
+                )
+                .unwrap();
+        }
+    }
+
+    fn get_finalized_function(&self, _func: &ObjectCompiledFunction) {
+        // Nothing to do.
+    }
+
+    fn finalize_data(
+        &mut self,
+        _id: DataId,
+        data: &ObjectCompiledData,
+        namespace: &ModuleNamespace,
+    ) {
+        for &RelocRecord {
+            offset,
+            ref name,
+            kind,
+            encoding,
+            size,
+            addend,
+        } in &data.relocs
+        {
+            let offset = data.offset + offset as u64;
+            let symbol = self.get_symbol(namespace, name);
+            self.object
+                .add_relocation(
+                    data.section,
+                    Relocation {
+                        offset,
+                        size,
+                        kind,
+                        encoding,
+                        symbol,
+                        addend,
+                    },
+                )
+                .unwrap();
+        }
+    }
+
+    fn get_finalized_data(&self, _data: &ObjectCompiledData) {
+        // Nothing to do.
+    }
+
+    fn publish(&mut self) {
+        // Nothing to do.
+    }
+
+    fn finish(self) -> ObjectProduct {
+        ObjectProduct {
+            object: self.object,
+            functions: self.functions,
+            data_objects: self.data_objects,
+            traps: self.traps,
+        }
+    }
+}
+
+impl ObjectBackend {
+    // This should only be called during finalization because it creates
+    // symbols for missing libcalls.
+    fn get_symbol(
+        &mut self,
+        namespace: &ModuleNamespace,
+        name: &ir::ExternalName,
+    ) -> SymbolId {
+        match *name {
+            ir::ExternalName::User { .. } => {
+                if namespace.is_function(name) {
+                    let id = namespace.get_function_id(name);
+                    self.functions[id].unwrap()
+                } else {
+                    let id = namespace.get_data_id(name);
+                    self.data_objects[id].unwrap()
+                }
+            }
+            ir::ExternalName::LibCall(ref libcall) => {
+                let name = (self.libcall_names)(*libcall);
+                if let Some(symbol) = self.object.symbol_id(name.as_bytes()) {
+                    symbol
+                } else if let Some(symbol) = self.libcalls.get(libcall) {
+                    *symbol
+                } else {
+                    let symbol = self.object.add_symbol(Symbol {
+                        name: name.as_bytes().to_vec(),
+                        value: 0,
+                        size: 0,
+                        kind: SymbolKind::Text,
+                        scope: SymbolScope::Unknown,
+                        weak: false,
+                        section: None,
+                    });
+                    self.libcalls.insert(*libcall, symbol);
+                    symbol
+                }
+            }
+            _ => panic!("invalid ExternalName {}", name),
+        }
+    }
+}
+
+fn translate_linkage(linkage: Linkage) -> (SymbolScope, bool) {
+    let scope = match linkage {
+        Linkage::Import => SymbolScope::Unknown,
+        Linkage::Local => SymbolScope::Compilation,
+        Linkage::Export | Linkage::Preemptible => SymbolScope::Dynamic,
+    };
+    // TODO: this matches rustc_codegen_cranelift, but may be wrong.
+    let weak = linkage == Linkage::Preemptible;
+    (scope, weak)
+}
+
+#[derive(Clone)]
+pub struct ObjectCompiledFunction {
+    offset: u64,
+    size: u32,
+    section: SectionId,
+    relocs: Vec,
+}
+
+#[derive(Clone)]
+pub struct ObjectCompiledData {
+    offset: u64,
+    section: SectionId,
+    relocs: Vec,
+}
+
+/// This is the output of `Module`'s
+/// [`finish`](../cranelift_module/struct.Module.html#method.finish) function.
+/// It contains the generated `Object` and other information produced during
+/// compilation.
+pub struct ObjectProduct {
+    /// Object artifact with all functions and data from the module defined.
+    pub object: Object,
+    /// Symbol IDs for functions (both declared and defined).
+    pub functions: SecondaryMap>,
+    /// Symbol IDs for data objects (both declared and defined).
+    pub data_objects: SecondaryMap>,
+    /// Trap sites for defined functions.
+    pub traps: SecondaryMap>,
+}
+
+impl ObjectProduct {
+    /// Return the `SymbolId` for the given function.
+    #[inline]
+    pub fn function_symbol(&self, id: FuncId) -> SymbolId {
+        self.functions[id].unwrap()
+    }
+
+    /// Return the `SymbolId` for the given data object.
+    #[inline]
+    pub fn data_symbol(&self, id: DataId) -> SymbolId {
+        self.data_objects[id].unwrap()
+    }
+
+    /// Write the object bytes in memory.
+    #[inline]
+    pub fn emit(self) -> Result, String> {
+        self.object.write()
+    }
+}
+
+#[derive(Clone)]
+struct RelocRecord {
+    offset: CodeOffset,
+    name: ir::ExternalName,
+    kind: RelocationKind,
+    encoding: RelocationEncoding,
+    size: u8,
+    addend: Addend,
+}
+
+#[derive(Default)]
+struct ObjectRelocSink {
+    relocs: Vec,
+}
+
+impl RelocSink for ObjectRelocSink {
+    fn reloc_ebb(&mut self, _offset: CodeOffset, _reloc: Reloc, _ebb_offset: CodeOffset) {
+        unimplemented!();
+    }
+
+    fn reloc_external(
+        &mut self,
+        offset: CodeOffset,
+        reloc: Reloc,
+        name: &ir::ExternalName,
+        addend: Addend,
+    ) {
+        let (kind, encoding, size) = match reloc {
+            Reloc::Abs4 => (RelocationKind::Absolute, RelocationEncoding::Generic, 32),
+            Reloc::Abs8 => (RelocationKind::Absolute, RelocationEncoding::Generic, 64),
+            Reloc::X86PCRel4 => (RelocationKind::Relative, RelocationEncoding::Generic, 32),
+            Reloc::X86CallPCRel4 => (RelocationKind::Relative, RelocationEncoding::X86Branch, 32),
+            // TODO: Get Cranelift to tell us when we can use
+            // R_X86_64_GOTPCRELX/R_X86_64_REX_GOTPCRELX.
+            Reloc::X86CallPLTRel4 => (
+                RelocationKind::PltRelative,
+                RelocationEncoding::X86Branch,
+                32,
+            ),
+            Reloc::X86GOTPCRel4 => (RelocationKind::GotRelative, RelocationEncoding::Generic, 32),
+            // FIXME
+            _ => unimplemented!(),
+        };
+        self.relocs.push(RelocRecord {
+            offset,
+            name: name.clone(),
+            kind,
+            encoding,
+            size,
+            addend,
+        });
+    }
+
+    fn reloc_jt(&mut self, _offset: CodeOffset, reloc: Reloc, _jt: ir::JumpTable) {
+        match reloc {
+            Reloc::X86PCRelRodata4 => {
+                // Not necessary to record this unless we are going to split apart code and its
+                // jumptbl/rodata.
+            }
+            _ => {
+                panic!("Unhandled reloc");
+            }
+        }
+    }
+
+    fn reloc_constant(&mut self, _offset: CodeOffset, reloc: Reloc, _jt: ir::ConstantOffset) {
+        match reloc {
+            Reloc::X86PCRelRodata4 => {
+                // Not necessary to record this unless we are going to split apart code and its
+                // jumptbl/rodata.
+            }
+            _ => {
+                panic!("Unhandled reloc");
+            }
+        }
+    }
+}
diff --git a/cranelift/object/src/lib.rs b/cranelift/object/src/lib.rs
new file mode 100644
index 0000000000..162a0b47aa
--- /dev/null
+++ b/cranelift/object/src/lib.rs
@@ -0,0 +1,38 @@
+//! Top-level lib.rs for `cranelift_object`.
+//!
+//! Users of this module should not have to depend on `object` directly.
+
+#![deny(
+    missing_docs,
+    trivial_numeric_casts,
+    unused_extern_crates,
+    unstable_features
+)]
+#![warn(unused_import_braces)]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    allow(clippy::new_without_default, clippy::new_without_default_derive)
+)]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    warn(
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
+    )
+)]
+
+mod backend;
+mod traps;
+
+pub use crate::backend::{ObjectBackend, ObjectBuilder, ObjectProduct, ObjectTrapCollection};
+pub use crate::traps::{ObjectTrapSink, ObjectTrapSite};
+
+/// Version number of this crate.
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/cranelift/object/src/traps.rs b/cranelift/object/src/traps.rs
new file mode 100644
index 0000000000..a64f1e13fa
--- /dev/null
+++ b/cranelift/object/src/traps.rs
@@ -0,0 +1,32 @@
+//! Records every `TrapCode` that cranelift outputs during code generation,
+//! for every function in the module. This data may be useful at runtime.
+
+use cranelift_codegen::{binemit, ir};
+
+/// Record of the arguments cranelift passes to `TrapSink::trap`
+#[derive(Clone)]
+pub struct ObjectTrapSite {
+    /// Offset into function
+    pub offset: binemit::CodeOffset,
+    /// Source location given to cranelift
+    pub srcloc: ir::SourceLoc,
+    /// Trap code, as determined by cranelift
+    pub code: ir::TrapCode,
+}
+
+/// Record of the trap sites for a given function
+#[derive(Default, Clone)]
+pub struct ObjectTrapSink {
+    /// All trap sites collected in function
+    pub sites: Vec,
+}
+
+impl binemit::TrapSink for ObjectTrapSink {
+    fn trap(&mut self, offset: binemit::CodeOffset, srcloc: ir::SourceLoc, code: ir::TrapCode) {
+        self.sites.push(ObjectTrapSite {
+            offset,
+            srcloc,
+            code,
+        });
+    }
+}
diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index 372f112e8d..68d603af30 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -7,7 +7,8 @@ use cranelift_codegen::binemit::{
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::{self, ir, settings};
 use cranelift_module::{
-    Backend, DataContext, DataDescription, Init, Linkage, ModuleNamespace, ModuleResult,
+    Backend, DataContext, DataDescription, DataId, FuncId, Init, Linkage, ModuleNamespace,
+    ModuleResult,
 };
 use cranelift_native;
 #[cfg(not(windows))]
@@ -222,12 +223,13 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         &*self.isa
     }
 
-    fn declare_function(&mut self, _name: &str, _linkage: Linkage) {
+    fn declare_function(&mut self, _id: FuncId, _name: &str, _linkage: Linkage) {
         // Nothing to do.
     }
 
     fn declare_data(
         &mut self,
+        _id: DataId,
         _name: &str,
         _linkage: Linkage,
         _writable: bool,
@@ -238,6 +240,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
 
     fn define_function(
         &mut self,
+        _id: FuncId,
         name: &str,
         ctx: &cranelift_codegen::Context,
         _namespace: &ModuleNamespace,
@@ -283,6 +286,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
 
     fn define_data(
         &mut self,
+        _id: DataId,
         _name: &str,
         writable: bool,
         align: Option,
@@ -372,6 +376,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
 
     fn finalize_function(
         &mut self,
+        _id: FuncId,
         func: &Self::CompiledFunction,
         namespace: &ModuleNamespace,
     ) -> Self::FinalizedFunction {
@@ -425,6 +430,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
 
     fn finalize_data(
         &mut self,
+        _id: DataId,
         data: &Self::CompiledData,
         namespace: &ModuleNamespace,
     ) -> Self::FinalizedData {

From ac2ca6116b3e84c779af021cc4c9eab138d4c3ca Mon Sep 17 00:00:00 2001
From: data-pup <16364986+data-pup@users.noreply.github.com>
Date: Fri, 28 Jun 2019 19:29:53 +0000
Subject: [PATCH 2691/3084] allow module environment to parse name section

---
 cranelift/wasm/src/environ/dummy.rs       | 17 +++++++-
 cranelift/wasm/src/environ/spec.rs        |  8 ++++
 cranelift/wasm/src/module_translator.rs   | 12 +++++-
 cranelift/wasm/src/sections_translator.rs | 50 +++++++++++++++++++++--
 cranelift/wasm/tests/wasm_testsuite.rs    | 48 +++++++++++++++++-----
 5 files changed, 119 insertions(+), 16 deletions(-)

diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index 8199cadee6..98cdb3f44e 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -17,7 +17,7 @@ use cranelift_codegen::ir::immediates::{Offset32, Uimm64};
 use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::isa::TargetFrontendConfig;
-use cranelift_entity::{EntityRef, PrimaryMap};
+use cranelift_entity::{EntityRef, PrimaryMap, SecondaryMap};
 use std::boxed::Box;
 use std::string::String;
 use std::vec::Vec;
@@ -124,6 +124,9 @@ pub struct DummyEnvironment {
 
     /// Instructs to collect debug data during translation.
     debug_info: bool,
+
+    /// Function names.
+    function_names: SecondaryMap,
 }
 
 impl DummyEnvironment {
@@ -135,6 +138,7 @@ impl DummyEnvironment {
             func_bytecode_sizes: Vec::new(),
             return_mode,
             debug_info,
+            function_names: SecondaryMap::new(),
         }
     }
 
@@ -152,6 +156,12 @@ impl DummyEnvironment {
     pub fn get_num_func_imports(&self) -> usize {
         self.info.imported_funcs.len()
     }
+
+    /// Return the name of the function, if a name for the function with
+    /// the corresponding index exists.
+    pub fn get_func_name(&self, func_index: FuncIndex) -> Option<&str> {
+        self.function_names.get(func_index).map(String::as_ref)
+    }
 }
 
 /// The `FuncEnvironment` implementation for use by the `DummyEnvironment`.
@@ -539,4 +549,9 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         self.info.function_bodies.push(func);
         Ok(())
     }
+
+    fn declare_func_name(&mut self, func_index: FuncIndex, name: &'data str) -> WasmResult<()> {
+        self.function_names[func_index] = String::from(name);
+        Ok(())
+    }
 }
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index e01c9b3cf9..ba87cc992a 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -468,6 +468,14 @@ pub trait ModuleEnvironment<'data> {
         data: &'data [u8],
     ) -> WasmResult<()>;
 
+    /// Declares the name of a function to the environment.
+    ///
+    /// By default this does nothing, but implementations can use this to read
+    /// the function name subsection of the custom name section if desired.
+    fn declare_func_name(&mut self, _func_index: FuncIndex, _name: &'data str) -> WasmResult<()> {
+        Ok(())
+    }
+
     /// Indicates that a custom section has been found in the wasm file
     fn custom_section(&mut self, name: &'data str, data: &'data [u8]) -> WasmResult<()> {
         drop((name, data));
diff --git a/cranelift/wasm/src/module_translator.rs b/cranelift/wasm/src/module_translator.rs
index 1ff8dd2dd8..8d8b55397d 100644
--- a/cranelift/wasm/src/module_translator.rs
+++ b/cranelift/wasm/src/module_translator.rs
@@ -4,10 +4,10 @@ use crate::environ::{ModuleEnvironment, WasmError, WasmResult};
 use crate::sections_translator::{
     parse_code_section, parse_data_section, parse_element_section, parse_export_section,
     parse_function_section, parse_global_section, parse_import_section, parse_memory_section,
-    parse_start_section, parse_table_section, parse_type_section,
+    parse_name_section, parse_start_section, parse_table_section, parse_type_section,
 };
 use cranelift_codegen::timing;
-use wasmparser::{ModuleReader, SectionCode};
+use wasmparser::{CustomSectionKind, ModuleReader, SectionCode};
 
 /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cranelift IR
 /// [`Function`](cranelift_codegen::ir::Function).
@@ -83,6 +83,14 @@ pub fn translate_module<'data>(
                 });
             }
 
+            SectionCode::Custom {
+                kind: CustomSectionKind::Name,
+                name: _,
+            } => {
+                let names = section.get_name_section_reader()?;
+                parse_name_section(names, environ)?;
+            }
+
             SectionCode::Custom { name, kind: _ } => {
                 let mut reader = section.get_binary_reader();
                 let len = reader.bytes_remaining();
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index 4ca8a66b87..67830af3ff 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -12,7 +12,7 @@ use crate::translation_utils::{
     tabletype_to_type, type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory,
     MemoryIndex, SignatureIndex, Table, TableElementType, TableIndex,
 };
-use crate::wasm_unsupported;
+use crate::{wasm_unsupported, HashMap};
 use core::convert::TryFrom;
 use cranelift_codegen::ir::{self, AbiParam, Signature};
 use cranelift_entity::EntityRef;
@@ -21,8 +21,8 @@ use wasmparser::{
     self, CodeSectionReader, Data, DataKind, DataSectionReader, Element, ElementKind,
     ElementSectionReader, Export, ExportSectionReader, ExternalKind, FuncType,
     FunctionSectionReader, GlobalSectionReader, GlobalType, ImportSectionEntryType,
-    ImportSectionReader, MemorySectionReader, MemoryType, Operator, TableSectionReader,
-    TypeSectionReader,
+    ImportSectionReader, MemorySectionReader, MemoryType, NameSectionReader, Naming, NamingReader,
+    Operator, TableSectionReader, TypeSectionReader,
 };
 
 /// Parses the Type section of the wasm module.
@@ -351,3 +351,47 @@ pub fn parse_data_section<'data>(
 
     Ok(())
 }
+
+/// Parses the Name section of the wasm module.
+pub fn parse_name_section<'data>(
+    mut names: NameSectionReader<'data>,
+    environ: &mut dyn ModuleEnvironment<'data>,
+) -> WasmResult<()> {
+    while let Ok(subsection) = names.read() {
+        match subsection {
+            wasmparser::Name::Function(function_subsection) => {
+                if let Some(function_names) = function_subsection
+                    .get_map()
+                    .ok()
+                    .and_then(parse_function_name_subsection)
+                {
+                    for (index, name) in function_names {
+                        environ.declare_func_name(index, name)?;
+                    }
+                }
+                return Ok(());
+            }
+            wasmparser::Name::Local(_) | wasmparser::Name::Module(_) => {}
+        };
+    }
+    Ok(())
+}
+
+fn parse_function_name_subsection<'data>(
+    mut naming_reader: NamingReader<'data>,
+) -> Option> {
+    let mut function_names = HashMap::new();
+    for _ in 0..naming_reader.get_count() {
+        let Naming { index, name } = naming_reader.read().ok()?;
+        if function_names
+            .insert(FuncIndex::from_u32(index), name)
+            .is_some()
+        {
+            // If the function index has been previously seen, then we
+            // break out of the loop and early return `None`, because these
+            // should be unique.
+            return None;
+        }
+    }
+    return Some(function_names);
+}
diff --git a/cranelift/wasm/tests/wasm_testsuite.rs b/cranelift/wasm/tests/wasm_testsuite.rs
index fd46458d53..69db90ca6c 100644
--- a/cranelift/wasm/tests/wasm_testsuite.rs
+++ b/cranelift/wasm/tests/wasm_testsuite.rs
@@ -2,7 +2,7 @@ use cranelift_codegen::isa;
 use cranelift_codegen::print_errors::pretty_verifier_error;
 use cranelift_codegen::settings::{self, Flags};
 use cranelift_codegen::verifier;
-use cranelift_wasm::{translate_module, DummyEnvironment, ReturnMode};
+use cranelift_wasm::{translate_module, DummyEnvironment, FuncIndex, ReturnMode};
 use std::fs;
 use std::fs::File;
 use std::io;
@@ -10,7 +10,7 @@ use std::io::prelude::*;
 use std::path::Path;
 use std::str::FromStr;
 use target_lexicon::triple;
-use wabt::{wat2wasm_with_features, Features};
+use wabt::{wat2wasm_with_features, Features, Wat2Wasm};
 
 #[test]
 fn testsuite() {
@@ -31,17 +31,42 @@ fn testsuite() {
     let flags = Flags::new(settings::builder());
     for path in paths {
         let path = path.path();
-        handle_module(&path, &flags, ReturnMode::NormalReturns);
+        let data = read_module(&path);
+        handle_module(data, &flags, ReturnMode::NormalReturns);
     }
 }
 
 #[test]
 fn use_fallthrough_return() {
     let flags = Flags::new(settings::builder());
-    handle_module(
-        Path::new("../wasmtests/use_fallthrough_return.wat"),
-        &flags,
-        ReturnMode::FallthroughReturn,
+    let path = Path::new("../wasmtests/use_fallthrough_return.wat");
+    let data = read_module(&path);
+    handle_module(data, &flags, ReturnMode::FallthroughReturn);
+}
+
+#[test]
+fn use_name_section() {
+    let wat = r#"
+        (module $module_name
+            (func $func_name (local $loc_name i32)
+            )
+        )"#;
+    let data = Wat2Wasm::new()
+        .write_debug_names(true)
+        .convert(wat)
+        .unwrap_or_else(|e| panic!("error converting wat to wasm: {:?}", e));
+
+    let flags = Flags::new(settings::builder());
+    let triple = triple!("riscv64");
+    let isa = isa::lookup(triple).unwrap().finish(flags.clone());
+    let return_mode = ReturnMode::NormalReturns;
+    let mut dummy_environ = DummyEnvironment::new(isa.frontend_config(), return_mode, false);
+
+    translate_module(data.as_ref(), &mut dummy_environ).unwrap();
+
+    assert_eq!(
+        dummy_environ.get_func_name(FuncIndex::from_u32(0)).unwrap(),
+        "func_name"
     );
 }
 
@@ -52,10 +77,10 @@ fn read_file(path: &Path) -> io::Result> {
     Ok(buf)
 }
 
-fn handle_module(path: &Path, flags: &Flags, return_mode: ReturnMode) {
+fn read_module(path: &Path) -> Vec {
     let mut features = Features::new();
     features.enable_all();
-    let data = match path.extension() {
+    match path.extension() {
         None => {
             panic!("the file extension is not wasm or wat");
         }
@@ -72,7 +97,10 @@ fn handle_module(path: &Path, flags: &Flags, return_mode: ReturnMode) {
             }
             None | Some(&_) => panic!("the file extension for {:?} is not wasm or wat", path),
         },
-    };
+    }
+}
+
+fn handle_module(data: Vec, flags: &Flags, return_mode: ReturnMode) {
     let triple = triple!("riscv64");
     let isa = isa::lookup(triple).unwrap().finish(flags.clone());
     let mut dummy_environ = DummyEnvironment::new(isa.frontend_config(), return_mode, false);

From 345b2dc0cc03fd8b2200a0621b394172dbf6e9a2 Mon Sep 17 00:00:00 2001
From: Ujjwal Sharma 
Date: Tue, 10 Sep 2019 16:25:24 +0530
Subject: [PATCH 2692/3084] [codegen] add new recipe "rout" (#1014)

* [codegen] add new recipe "rout"

Add a new recipe "rout" intended to be used by arithematic operations
that output flags, currently being used for `iadd_cout` and `isub_bout`.

Fixes: https://github.com/CraneStation/cranelift/issues/1009
---
 .../codegen/meta/src/isa/x86/encodings.rs     |  5 +++--
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 19 ++++++++++++++++++-
 .../filetests/filetests/isa/x86/run-i64.clif  | 17 +++++++++++++++++
 3 files changed, 38 insertions(+), 3 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/run-i64.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 253491f01c..2ae60896d4 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -566,6 +566,7 @@ pub(crate) fn define(
     let rec_rfurm = r.template("rfurm");
     let rec_rmov = r.template("rmov");
     let rec_rr = r.template("rr");
+    let rec_rout = r.template("rout");
     let rec_rin = r.template("rin");
     let rec_rio = r.template("rio");
     let rec_rrx = r.template("rrx");
@@ -631,12 +632,12 @@ pub(crate) fn define(
     );
 
     e.enc_i32_i64(iadd, rec_rr.opcodes(vec![0x01]));
-    e.enc_i32_i64(iadd_cout, rec_rr.opcodes(vec![0x01]));
+    e.enc_i32_i64(iadd_cout, rec_rout.opcodes(vec![0x01]));
     e.enc_i32_i64(iadd_cin, rec_rin.opcodes(vec![0x11]));
     e.enc_i32_i64(iadd_carry, rec_rio.opcodes(vec![0x11]));
 
     e.enc_i32_i64(isub, rec_rr.opcodes(vec![0x29]));
-    e.enc_i32_i64(isub_bout, rec_rr.opcodes(vec![0x29]));
+    e.enc_i32_i64(isub_bout, rec_rout.opcodes(vec![0x29]));
     e.enc_i32_i64(isub_bin, rec_rin.opcodes(vec![0x19]));
     e.enc_i32_i64(isub_borrow, rec_rio.opcodes(vec![0x19]));
 
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index a5db0c1b97..3f14769dee 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -2548,7 +2548,24 @@ pub(crate) fn define<'shared>(
             ),
     );
 
-    // Adding with carry
+    // Arithematic with flag I/O.
+
+    // XX /r, MR form. Add two GPR registers and set carry flag.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("rout", f_binary, 1)
+            .operands_in(vec![gpr, gpr])
+            .operands_out(vec![
+                OperandConstraint::TiedInput(0),
+                OperandConstraint::FixedReg(reg_rflags),
+            ])
+            .clobbers_flags(true)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
+                    modrm_rr(in_reg0, in_reg1, sink);
+                "#,
+            ),
+    );
 
     // XX /r, MR form. Add two GPR registers and get carry flag.
     recipes.add_template_recipe(
diff --git a/cranelift/filetests/filetests/isa/x86/run-i64.clif b/cranelift/filetests/filetests/isa/x86/run-i64.clif
new file mode 100644
index 0000000000..fc1b155c10
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/run-i64.clif
@@ -0,0 +1,17 @@
+; Test i64 instructions on x86_32.
+test compile
+target i686 haswell
+
+function %iadd(i64, i64) -> i64 {
+ebb0(v1: i64, v2: i64):
+    v10 = iadd v1, v2
+    ; check: iadd_cout
+    return v10
+}
+
+function %isub(i64, i64) -> i64 {
+ebb0(v1: i64, v2: i64):
+    v10 = isub v1, v2
+    ; check: isub_bout
+    return v10
+}

From 81fa5e7696ee828f56f1e831dc43e5b0a7275c2b Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Fri, 6 Sep 2019 18:18:28 +0200
Subject: [PATCH 2693/3084] Add equivalent safepoint test cases for basic
 blocks.

---
 .../filetests/safepoint/basic-bb.clif         | 72 +++++++++++++++++++
 .../filetests/filetests/safepoint/call.clif   | 16 +++--
 2 files changed, 82 insertions(+), 6 deletions(-)
 create mode 100644 cranelift/filetests/filetests/safepoint/basic-bb.clif

diff --git a/cranelift/filetests/filetests/safepoint/basic-bb.clif b/cranelift/filetests/filetests/safepoint/basic-bb.clif
new file mode 100644
index 0000000000..b2ccedf559
--- /dev/null
+++ b/cranelift/filetests/filetests/safepoint/basic-bb.clif
@@ -0,0 +1,72 @@
+test safepoint
+set enable_safepoints=true
+target x86_64
+feature "basic-blocks"
+
+function %test(i32, r64, r64) -> r64 {
+    ebb0(v0: i32, v1:r64, v2:r64):
+        jump ebb1(v0)
+    ebb1(v3: i32):
+        v4 = irsub_imm v3, 1
+        jump ebb2(v4)
+    ebb2(v5: i32):
+        resumable_trap interrupt
+        brz v5, ebb1(v5)
+        jump ebb3
+    ebb3:
+        v6 = null.r64
+        v7 = is_null v6
+        brnz v7, ebb2(v0)
+        jump ebb4
+    ebb4:
+        brnz v0, ebb5
+        jump ebb6
+    ebb5:
+        return v1
+    ebb6:
+        return v2
+}
+
+; sameln: function %test(i32 [%rdi], r64 [%rsi], r64 [%rdx]) -> r64 [%rax] fast {
+; nextln: ebb0(v0: i32 [%rdi], v1: r64 [%rsi], v2: r64 [%rdx]):
+; nextln:   v10 = copy v0
+; nextln:   jump ebb1(v10)
+; nextln: 
+; nextln: ebb1(v3: i32 [%rax]):
+; nextln:   v8 = iconst.i32 1
+; nextln:   v4 = isub v8, v3
+; nextln:   jump ebb2(v4)
+; nextln: 
+; nextln: ebb2(v5: i32 [%rcx]):
+; nextln:   safepoint v1, v2
+; nextln:   resumable_trap interrupt
+; nextln:   brz v5, ebb7
+; nextln:   jump ebb3
+; nextln: 
+; nextln: ebb3:
+; nextln:   v6 = null.r64 
+; nextln:   v7 = is_null v6
+; nextln:   brnz v7, ebb8
+; nextln:   jump ebb4
+; nextln: 
+; nextln: ebb4:
+; nextln:   brnz.i32 v0, ebb5
+; nextln:   jump ebb6
+; nextln: 
+; nextln: ebb5:
+; nextln:   regmove.r64 v1, %rsi -> %rax
+; nextln:   return v1
+; nextln: 
+; nextln: ebb6:
+; nextln:   regmove.r64 v2, %rdx -> %rax
+; nextln:   return v2
+; nextln: 
+; nextln: ebb7:
+; nextln:   regmove.i32 v5, %rcx -> %rax
+; nextln:   jump ebb1(v5)
+; nextln: 
+; nextln: ebb8:
+; nextln:   v9 = copy.i32 v0
+; nextln:   regmove v9, %rax -> %rcx
+; nextln:   jump ebb2(v9)
+; nextln: }
diff --git a/cranelift/filetests/filetests/safepoint/call.clif b/cranelift/filetests/filetests/safepoint/call.clif
index a322445319..9e9583093b 100644
--- a/cranelift/filetests/filetests/safepoint/call.clif
+++ b/cranelift/filetests/filetests/safepoint/call.clif
@@ -1,7 +1,6 @@
 test safepoint
 set enable_safepoints=true
 target x86_64
-feature !"basic-blocks"
 
 function %direct() -> r64 {
     fn0 = %none()
@@ -12,9 +11,11 @@ ebb0:
     call fn0()
     v1 = call fn1()
     v2, v3 = call fn2()
-    brz v2, ebb1 
-    return v1
+    brz v2, ebb2
+    jump ebb1
 ebb1:
+    return v1
+ebb2:
     v4 = call fn1()
     return v3
 }
@@ -39,12 +40,15 @@ ebb1:
 ; nextln:   safepoint v1
 ; nextln:   v2, v10 = call_indirect sig2, v7()
 ; nextln:   v3 = spill v10
-; nextln:   brz v2, ebb1
-; nextln:   v11 = fill v1
+; nextln:   brz v2, ebb2
+; nextln:   jump ebb1
+; nextln: 
+; nextln: ebb1:
+; nextln:   v11 = fill.r64 v1
 ; nextln:   regmove v11, %r15 -> %rax
 ; nextln:   return v11
 ; nextln: 
-; nextln: ebb1:
+; nextln: ebb2:
 ; nextln:   v8 = func_addr.i64 fn1
 ; nextln:   safepoint v3
 ; nextln:   v4 = call_indirect sig1, v8()

From 63367d205cc129ebcd1a12c5ebdb04bc37ce4353 Mon Sep 17 00:00:00 2001
From: julian-seward1 
Date: Tue, 10 Sep 2019 15:06:10 +0200
Subject: [PATCH 2694/3084] Fix #975 (Structure of
 cranelift-wasm/src/translation_utils.rs causes many pointless heap
 allocations) (#1010)

This patch restricts the `Err(..)` return from `blocktype_to_type` to be
`Err(..)` only in the case where it really is an error to continue.  The three
use points of `blocktype_to_type` are changed to check for an `Err(..)` rather
than silently ignoring it.  There are also cosmetic changes to `type_to_type`
and `tabletype_to_type`.

When compiling wasm_lua_binarytrees, this reduces the number of blocks
allocated by CL by 1.9%.  Instruction count falls by 0.1%.

Details:

* `type_to_type` and `tabletype_to_type`:

   - Added the function name in the failure message

   - No functional change for non-error cases

   - Push the `Ok(..)` to expression leaves, where it really applies.  This
     corrects the misleading impression that, in the case of an unsupported
     type, the function returns `Ok` wrapped around whatever
     `wasm_unsupported` returns.  It doesn't do that, but it certainly reads
     like that.  This assumes that the LLVM backend will do tail merging, so
     the generated code will be unchanged.

* `blocktype_to_type`:

  - Change return type from `WasmResult` to `WasmResult>`

  - Manually inline the call to `type_to_type`, to make this function easier
    to read.

  - For the non-error case: map `TypeOrFuncType::Type(Type::EmptyBlockType)`
    to `Ok(None)` rather than `Err(..)`, since that's what all the call sites
    expect - For the error cases, add the function name in the failure
    messages

* cranelift-wasm/src/code_translator.rs

  - For the three uses of `blocktype_to_type`, use `?` to detect failures and
    drop out immediately, meaning that the code will no longer silently ignore
    errors.
---
 cranelift/wasm/src/code_translator.rs   |  6 +--
 cranelift/wasm/src/translation_utils.rs | 50 ++++++++++++++-----------
 2 files changed, 32 insertions(+), 24 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 664d839023..1801c6539b 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -133,7 +133,7 @@ pub fn translate_operator(
          ***********************************************************************************/
         Operator::Block { ty } => {
             let next = builder.create_ebb();
-            if let Ok(ty_cre) = blocktype_to_type(*ty) {
+            if let Some(ty_cre) = blocktype_to_type(*ty)? {
                 builder.append_ebb_param(next, ty_cre);
             }
             state.push_block(next, num_return_values(*ty)?);
@@ -141,7 +141,7 @@ pub fn translate_operator(
         Operator::Loop { ty } => {
             let loop_body = builder.create_ebb();
             let next = builder.create_ebb();
-            if let Ok(ty_cre) = blocktype_to_type(*ty) {
+            if let Some(ty_cre) = blocktype_to_type(*ty)? {
                 builder.append_ebb_param(next, ty_cre);
             }
             builder.ins().jump(loop_body, &[]);
@@ -168,7 +168,7 @@ pub fn translate_operator(
             //   and we add nothing;
             // - either the If have an Else clause, in that case the destination of this jump
             //   instruction will be changed later when we translate the Else operator.
-            if let Ok(ty_cre) = blocktype_to_type(*ty) {
+            if let Some(ty_cre) = blocktype_to_type(*ty)? {
                 builder.append_ebb_param(if_not, ty_cre);
             }
             state.push_if(jump_inst, if_not, num_return_values(*ty)?);
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index 90967d4102..f033b8660c 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -115,35 +115,43 @@ pub struct Memory {
 
 /// Helper function translating wasmparser types to Cranelift types when possible.
 pub fn type_to_type(ty: wasmparser::Type) -> WasmResult {
-    Ok(match ty {
-        wasmparser::Type::I32 => ir::types::I32,
-        wasmparser::Type::I64 => ir::types::I64,
-        wasmparser::Type::F32 => ir::types::F32,
-        wasmparser::Type::F64 => ir::types::F64,
-        ty => wasm_unsupported!("unsupported wasm type {:?}", ty),
-    })
+    match ty {
+        wasmparser::Type::I32 => Ok(ir::types::I32),
+        wasmparser::Type::I64 => Ok(ir::types::I64),
+        wasmparser::Type::F32 => Ok(ir::types::F32),
+        wasmparser::Type::F64 => Ok(ir::types::F64),
+        ty => wasm_unsupported!("type_to_type: wasm type {:?}", ty),
+    }
 }
 
 /// Helper function translating wasmparser possible table types to Cranelift types when possible,
 /// or None for Func tables.
 pub fn tabletype_to_type(ty: wasmparser::Type) -> WasmResult> {
-    Ok(match ty {
-        wasmparser::Type::I32 => Some(ir::types::I32),
-        wasmparser::Type::I64 => Some(ir::types::I64),
-        wasmparser::Type::F32 => Some(ir::types::F32),
-        wasmparser::Type::F64 => Some(ir::types::F64),
-        wasmparser::Type::AnyFunc => None,
-        ty => wasm_unsupported!("unsupported table wasm type {:?}", ty),
-    })
+    match ty {
+        wasmparser::Type::I32 => Ok(Some(ir::types::I32)),
+        wasmparser::Type::I64 => Ok(Some(ir::types::I64)),
+        wasmparser::Type::F32 => Ok(Some(ir::types::F32)),
+        wasmparser::Type::F64 => Ok(Some(ir::types::F64)),
+        wasmparser::Type::AnyFunc => Ok(None),
+        ty => wasm_unsupported!("tabletype_to_type: table wasm type {:?}", ty),
+    }
 }
 
 /// Helper function translating wasmparser block signatures to Cranelift types when possible.
-pub fn blocktype_to_type(ty: wasmparser::TypeOrFuncType) -> WasmResult {
-    match ty {
-        wasmparser::TypeOrFuncType::Type(ty) => type_to_type(ty),
-        wasmparser::TypeOrFuncType::FuncType(_) => {
-            wasm_unsupported!("multi-value block signature {:?}", ty);
-        }
+pub fn blocktype_to_type(ty_or_ft: wasmparser::TypeOrFuncType) -> WasmResult> {
+    match ty_or_ft {
+        wasmparser::TypeOrFuncType::Type(ty) => match ty {
+            wasmparser::Type::I32 => Ok(Some(ir::types::I32)),
+            wasmparser::Type::I64 => Ok(Some(ir::types::I64)),
+            wasmparser::Type::F32 => Ok(Some(ir::types::F32)),
+            wasmparser::Type::F64 => Ok(Some(ir::types::F64)),
+            wasmparser::Type::EmptyBlockType => Ok(None),
+            ty => wasm_unsupported!("blocktype_to_type: type {:?}", ty),
+        },
+        wasmparser::TypeOrFuncType::FuncType(_) => wasm_unsupported!(
+            "blocktype_to_type: multi-value block signature {:?}",
+            ty_or_ft
+        ),
     }
 }
 

From 4b085b9cf71638aae83b4d6450c115a7ecc4267c Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Tue, 10 Sep 2019 08:18:06 -0600
Subject: [PATCH 2695/3084] Avoid unnecessary reallocations in
 domtree::with_function() (#1011)

---
 cranelift/codegen/src/dominator_tree.rs |  8 +++++++-
 cranelift/codegen/src/ir/layout.rs      |  5 +++++
 cranelift/entity/src/map.rs             | 19 +++++++++++++++++++
 3 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/dominator_tree.rs b/cranelift/codegen/src/dominator_tree.rs
index feea33a0ef..8191c310eb 100644
--- a/cranelift/codegen/src/dominator_tree.rs
+++ b/cranelift/codegen/src/dominator_tree.rs
@@ -226,7 +226,13 @@ impl DominatorTree {
 
     /// Allocate and compute a dominator tree.
     pub fn with_function(func: &Function, cfg: &ControlFlowGraph) -> Self {
-        let mut domtree = Self::new();
+        let ebb_capacity = func.layout.ebb_capacity();
+        let mut domtree = Self {
+            nodes: SecondaryMap::with_capacity(ebb_capacity),
+            postorder: Vec::with_capacity(ebb_capacity),
+            stack: Vec::new(),
+            valid: false,
+        };
         domtree.compute(func, cfg);
         domtree
     }
diff --git a/cranelift/codegen/src/ir/layout.rs b/cranelift/codegen/src/ir/layout.rs
index e1015f7f3f..78a4628576 100644
--- a/cranelift/codegen/src/ir/layout.rs
+++ b/cranelift/codegen/src/ir/layout.rs
@@ -60,6 +60,11 @@ impl Layout {
         self.first_ebb = None;
         self.last_ebb = None;
     }
+
+    /// Returns the capacity of the `EbbData` map.
+    pub fn ebb_capacity(&self) -> usize {
+        self.ebbs.capacity()
+    }
 }
 
 /// Sequence numbers.
diff --git a/cranelift/entity/src/map.rs b/cranelift/entity/src/map.rs
index bb1b94aeca..f46c307426 100644
--- a/cranelift/entity/src/map.rs
+++ b/cranelift/entity/src/map.rs
@@ -52,6 +52,20 @@ where
         }
     }
 
+    /// Create a new, empty map with the specified capacity.
+    ///
+    /// The map will be able to hold exactly `capacity` elements without reallocating.
+    pub fn with_capacity(capacity: usize) -> Self
+    where
+        V: Default,
+    {
+        Self {
+            elems: Vec::with_capacity(capacity),
+            default: Default::default(),
+            unused: PhantomData,
+        }
+    }
+
     /// Create a new empty map with a specified default value.
     ///
     /// This constructor does not require V to implement Default.
@@ -63,6 +77,11 @@ where
         }
     }
 
+    /// Returns the number of elements the map can hold without reallocating.
+    pub fn capacity(&self) -> usize {
+        self.elems.capacity()
+    }
+
     /// Get the element at `k` if it exists.
     pub fn get(&self, k: K) -> Option<&V> {
         self.elems.get(k.index())

From 7f152611a49172765e4416ae0643c511c44f166c Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 21 Aug 2019 14:07:50 -0700
Subject: [PATCH 2696/3084] Log compiled and legalized functions

---
 cranelift/codegen/src/context.rs | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index 704d891776..5350b3444c 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -33,6 +33,7 @@ use crate::timing;
 use crate::unreachable_code::eliminate_unreachable_code;
 use crate::value_label::{build_value_labels_ranges, ComparableSourceLoc, ValueLabelsRanges};
 use crate::verifier::{verify_context, verify_locations, VerifierErrors, VerifierResult};
+use log::debug;
 use std::vec::Vec;
 
 /// Persistent data structures and compilation pipeline.
@@ -129,6 +130,7 @@ impl Context {
     pub fn compile(&mut self, isa: &dyn TargetIsa) -> CodegenResult {
         let _tt = timing::compile();
         self.verify_if(isa)?;
+        debug!("Compiling:\n{}", self.func.display(isa));
 
         self.compute_cfg();
         if isa.flags().opt_level() != OptLevel::Fastest {
@@ -158,7 +160,10 @@ impl Context {
             self.redundant_reload_remover(isa)?;
             self.shrink_instructions(isa)?;
         }
-        self.relax_branches(isa)
+        let result = self.relax_branches(isa);
+
+        debug!("Compiled:\n{}", self.func.display(isa));
+        result
     }
 
     /// Emit machine code directly into raw memory.
@@ -256,6 +261,7 @@ impl Context {
         self.domtree.clear();
         self.loop_analysis.clear();
         legalize_function(&mut self.func, &mut self.cfg, isa);
+        debug!("Legalized:\n{}", self.func.display(isa));
         self.verify_if(isa)
     }
 

From 8a6686ec45ebf7a26034b8f3b97499bd720e0eee Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 21 Aug 2019 14:59:17 -0700
Subject: [PATCH 2697/3084] Enable SSSE3 setting when detected on CPU

---
 cranelift/native/src/lib.rs | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/cranelift/native/src/lib.rs b/cranelift/native/src/lib.rs
index 0687e70173..cfa27cef86 100644
--- a/cranelift/native/src/lib.rs
+++ b/cranelift/native/src/lib.rs
@@ -59,6 +59,9 @@ fn parse_x86_cpuid(isa_builder: &mut isa::Builder) -> Result<(), &'static str> {
         if info.has_sse3() {
             isa_builder.enable("has_sse3").unwrap();
         }
+        if info.has_ssse3() {
+            isa_builder.enable("has_ssse3").unwrap();
+        }
         if info.has_sse41() {
             isa_builder.enable("has_sse41").unwrap();
         }

From ebc783e49be50a913983f10e41e4792eef545070 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 21 Aug 2019 15:15:27 -0700
Subject: [PATCH 2698/3084] Use raw_bitcast when legalizing splat

raw_bitcast matches the intent of this legalization more clearly (to simply change the CLIF type without changing any bits) and the additional null encodings added are necessary for later instructions
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 37 +++++++++++++------
 .../codegen/meta/src/isa/x86/legalize.rs      |  5 ++-
 .../filetests/isa/x86/legalize-splat.clif     |  2 +-
 3 files changed, 30 insertions(+), 14 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 2ae60896d4..4a04ef574d 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -234,7 +234,7 @@ impl PerCpuModeEncodings {
     }
     fn enc_both_isap(
         &mut self,
-        inst: BoundInstruction,
+        inst: impl Clone + Into,
         template: Template,
         isap: SettingPredicateNumber,
     ) {
@@ -243,7 +243,7 @@ impl PerCpuModeEncodings {
     }
     fn enc_both_instp(
         &mut self,
-        inst: BoundInstruction,
+        inst: impl Clone + Into,
         template: Template,
         instp: InstructionPredicateNode,
     ) {
@@ -1811,23 +1811,38 @@ pub(crate) fn define(
         }
     }
 
-    // SIMD bitcast f64 to all 8-bit-lane vectors (for legalizing splat.x8x16); assumes that f64 is stored in an XMM register
-    for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
-        let instruction = bitcast.bind_vector_from_lane(ty, sse_vector_size).bind(F64);
+    // helper for generating null encodings for FPRs on both 32- and 64-bit architectures
+    let mut null_encode_32_64 = |instruction: BoundInstruction| {
         e.enc32_rec(instruction.clone(), rec_null_fpr, 0);
         e.enc64_rec(instruction, rec_null_fpr, 0);
-    }
+    };
 
     // SIMD bitcast all 128-bit vectors to each other (for legalizing splat.x16x8)
     for from_type in ValueType::all_lane_types().filter(allowed_simd_type) {
         for to_type in
             ValueType::all_lane_types().filter(|t| allowed_simd_type(t) && *t != from_type)
         {
-            let instruction = raw_bitcast
-                .bind_vector_from_lane(to_type, sse_vector_size)
-                .bind_vector_from_lane(from_type, sse_vector_size);
-            e.enc32_rec(instruction.clone(), rec_null_fpr, 0);
-            e.enc64_rec(instruction, rec_null_fpr, 0);
+            null_encode_32_64(
+                raw_bitcast
+                    .bind_vector_from_lane(to_type, sse_vector_size)
+                    .bind_vector_from_lane(from_type, sse_vector_size),
+            );
+        }
+    }
+
+    // SIMD raw bitcast floats to vector (and back); assumes that floats are already stored in an XMM register
+    for float_type in &[F32, F64] {
+        for lane_type in ValueType::all_lane_types().filter(allowed_simd_type) {
+            null_encode_32_64(
+                raw_bitcast
+                    .bind_vector_from_lane(lane_type, sse_vector_size)
+                    .bind(*float_type),
+            );
+            null_encode_32_64(
+                raw_bitcast
+                    .bind(*float_type)
+                    .bind_vector_from_lane(lane_type, sse_vector_size),
+            );
         }
     }
 
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 2fd160de3a..d56beb8022 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -20,7 +20,6 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     // List of instructions.
     let insts = &shared.instructions;
     let band = insts.by_name("band");
-    let bitcast = insts.by_name("bitcast");
     let bor = insts.by_name("bor");
     let clz = insts.by_name("clz");
     let ctz = insts.by_name("ctz");
@@ -321,7 +320,9 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     // SIMD splat: 8-bits
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
         let splat_any8x16 = splat.bind_vector_from_lane(ty, sse_vector_size);
-        let bitcast_f64_to_any8x16 = bitcast.bind_vector_from_lane(ty, sse_vector_size).bind(F64);
+        let bitcast_f64_to_any8x16 = raw_bitcast
+            .bind_vector_from_lane(ty, sse_vector_size)
+            .bind(F64);
         narrow.legalize(
             def!(y = splat_any8x16(x)),
             vec![
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-splat.clif b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
index fa07f80c11..c0fc83ebe7 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
@@ -68,6 +68,6 @@ ebb0:
 ; nextln:     v0 = ireduce.i8 v2
 ; nextln:     v3 = scalar_to_vector.i8x16 v0
 ; nextln:     v4 = f64const 0.0
-; nextln:     v5 = bitcast.i8x16 v4
+; nextln:     v5 = raw_bitcast.i8x16 v4
 ; nextln:     v1 = x86_pshufb v3, v5
 ; nextln:     return v1

From f1363168a9dfc4552be76e1039f0971504683b25 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 21 Aug 2019 15:21:18 -0700
Subject: [PATCH 2699/3084] Translate the sign-extended and zero-extended
 versions of extract_lane

---
 cranelift/wasm/src/code_translator.rs | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 1801c6539b..e5ce5a0c32 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -940,6 +940,16 @@ pub fn translate_operator(
             let splatted = builder.ins().splat(ty, value_to_splat);
             state.push1(splatted)
         }
+        Operator::I8x16ExtractLaneS { lane } | Operator::I16x8ExtractLaneS { lane } => {
+            let vector = optionally_bitcast_vector(state.pop1(), type_of(op), builder);
+            let extracted = builder.ins().extractlane(vector, lane.clone());
+            state.push1(builder.ins().sextend(I32, extracted))
+        }
+        Operator::I8x16ExtractLaneU { lane } | Operator::I16x8ExtractLaneU { lane } => {
+            let vector = optionally_bitcast_vector(state.pop1(), type_of(op), builder);
+            state.push1(builder.ins().extractlane(vector, lane.clone()));
+            // on x86, PEXTRB zeroes the upper bits of the destination register of extractlane so uextend is elided; of course, this depends on extractlane being legalized to a PEXTRB
+        }
         Operator::I32x4ExtractLane { lane }
         | Operator::I64x2ExtractLane { lane }
         | Operator::F32x4ExtractLane { lane }
@@ -967,10 +977,6 @@ pub fn translate_operator(
         }
         Operator::V128Load { .. }
         | Operator::V128Store { .. }
-        | Operator::I8x16ExtractLaneS { .. }
-        | Operator::I8x16ExtractLaneU { .. }
-        | Operator::I16x8ExtractLaneS { .. }
-        | Operator::I16x8ExtractLaneU { .. }
         | Operator::V8x16Shuffle { .. }
         | Operator::I8x16Eq
         | Operator::I8x16Ne

From 00bedca274bbcb5275a1818527713dfc33a63d6f Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 21 Aug 2019 16:56:11 -0700
Subject: [PATCH 2700/3084] Avoid extra register movement when lowering the x86
 extractlane of a float vector

This commit is based on the assumption that floats are already stored in XMM registers in x86. When extracting a lane, cranelift was moving the float to a regular register and back to an XMM register; this change avoids this by shuffling the float value to the lowest bits of the XMM register. It also assumes that the upper bits can be left as is (instead of zeroing them out).
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 16 +++---
 .../codegen/meta/src/isa/x86/instructions.rs  | 17 ++++++
 .../codegen/meta/src/isa/x86/legalize.rs      |  3 +
 cranelift/codegen/src/isa/x86/enc_tables.rs   | 57 +++++++++++++++++++
 .../isa/x86/extractlane-binemit.clif          | 38 +++++++++++++
 .../filetests/isa/x86/extractlane-run.clif    | 31 ++++++++++
 .../filetests/isa/x86/extractlane.clif        | 35 ------------
 7 files changed, 154 insertions(+), 43 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/extractlane-run.clif
 delete mode 100644 cranelift/filetests/filetests/isa/x86/extractlane.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 4a04ef574d..30246b85ad 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -356,7 +356,6 @@ pub(crate) fn define(
     let copy_to_ssa = shared.by_name("copy_to_ssa");
     let ctz = shared.by_name("ctz");
     let debugtrap = shared.by_name("debugtrap");
-    let extractlane = shared.by_name("extractlane");
     let f32const = shared.by_name("f32const");
     let f64const = shared.by_name("f64const");
     let fadd = shared.by_name("fadd");
@@ -460,6 +459,7 @@ pub(crate) fn define(
     let x86_fmax = x86.by_name("x86_fmax");
     let x86_fmin = x86.by_name("x86_fmin");
     let x86_pop = x86.by_name("x86_pop");
+    let x86_pextr = x86.by_name("x86_pextr");
     let x86_pshufd = x86.by_name("x86_pshufd");
     let x86_pshufb = x86.by_name("x86_pshufb");
     let x86_push = x86.by_name("x86_push");
@@ -1791,16 +1791,16 @@ pub(crate) fn define(
     }
 
     // SIMD extractlane
-    let mut extractlane_mapping: HashMap, Option)> =
+    let mut x86_pextr_mapping: HashMap, Option)> =
         HashMap::new();
-    extractlane_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x14], Some(use_sse41_simd))); // PEXTRB
-    extractlane_mapping.insert(16, (vec![0x66, 0x0f, 0xc5], None)); // PEXTRW from zSSE2, SSE4.1 has a PEXTRW that can move to reg/m16 but the opcode is four bytes
-    extractlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41_simd))); // PEXTRD
-    extractlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41_simd))); // PEXTRQ, only x86_64
+    x86_pextr_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x14], Some(use_sse41))); // PEXTRB
+    x86_pextr_mapping.insert(16, (vec![0x66, 0x0f, 0xc5], None)); // PEXTRW from zSSE2, SSE4.1 has a PEXTRW that can move to reg/m16 but the opcode is four bytes
+    x86_pextr_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41))); // PEXTRD
+    x86_pextr_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41))); // PEXTRQ, only x86_64
 
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
-        if let Some((opcode, isap)) = extractlane_mapping.get(&ty.lane_bits()) {
-            let instruction = extractlane.bind_vector_from_lane(ty, sse_vector_size);
+        if let Some((opcode, isap)) = x86_pextr_mapping.get(&ty.lane_bits()) {
+            let instruction = x86_pextr.bind_vector_from_lane(ty, sse_vector_size);
             let template = rec_r_ib_unsigned_gpr.opcodes(opcode.clone());
             if ty.lane_bits() < 64 {
                 e.enc_32_64_maybe_isap(instruction, template.nonrex(), isap.clone());
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 03730cdeac..3f583c6edb 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -291,5 +291,22 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    let Idx = &operand_doc("Idx", uimm8, "Lane index");
+    let x = &operand("x", TxN);
+    let a = &operand("a", &TxN.lane_of());
+
+    ig.push(
+        Inst::new(
+            "x86_pextr",
+            r#"
+        Extract lane ``Idx`` from ``x``.
+        The lane index, ``Idx``, is an immediate value, not an SSA value. It
+        must indicate a valid lane index for the type of ``x``.
+        "#,
+        )
+        .operands_in(vec![x, Idx])
+        .operands_out(vec![a]),
+    );
+
     ig.build()
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index d56beb8022..4c2ebaefd4 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -23,6 +23,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let bor = insts.by_name("bor");
     let clz = insts.by_name("clz");
     let ctz = insts.by_name("ctz");
+    let extractlane = insts.by_name("extractlane");
     let f64const = insts.by_name("f64const");
     let fcmp = insts.by_name("fcmp");
     let fcvt_from_uint = insts.by_name("fcvt_from_uint");
@@ -379,5 +380,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         );
     }
 
+    narrow.custom_legalize(extractlane, "convert_extractlane");
+
     narrow.build_and_add_to(&mut shared.transform_groups);
 }
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index e0fc05178d..39bdb57845 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -5,6 +5,7 @@ use crate::bitset::BitSet;
 use crate::cursor::{Cursor, FuncCursor};
 use crate::flowgraph::ControlFlowGraph;
 use crate::ir::condcodes::{FloatCC, IntCC};
+use crate::ir::types::*;
 use crate::ir::{self, Function, Inst, InstBuilder};
 use crate::isa::constraints::*;
 use crate::isa::enc_tables::*;
@@ -893,3 +894,59 @@ fn expand_fcvt_to_uint_sat(
     cfg.recompute_ebb(pos.func, uint_large_ebb);
     cfg.recompute_ebb(pos.func, done);
 }
+
+/// Because floats already exist in XMM registers, we can keep them there when executing a CLIF
+/// extractlane instruction
+fn convert_extractlane(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    _cfg: &mut ControlFlowGraph,
+    _isa: &dyn TargetIsa,
+) {
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+    pos.use_srcloc(inst);
+
+    if let ir::InstructionData::ExtractLane {
+        opcode: ir::Opcode::Extractlane,
+        arg,
+        lane,
+    } = pos.func.dfg[inst]
+    {
+        // NOTE: the following legalization assumes that the upper bits of the XMM register do
+        // not need to be zeroed during extractlane.
+        let value_type = pos.func.dfg.value_type(arg);
+        if value_type.lane_type().is_float() {
+            // Floats are already in XMM registers and can stay there.
+            let shuffled = if lane != 0 {
+                // Replace the extractlane with a PSHUFD to get the float in the right place.
+                match value_type {
+                    F32X4 => {
+                        // Move the selected lane to the 0 lane.
+                        let shuffle_mask: u8 = 0b00_00_00_00 | lane;
+                        pos.ins().x86_pshufd(arg, shuffle_mask)
+                    }
+                    F64X2 => {
+                        assert_eq!(lane, 1);
+                        // Because we know the lane == 1, we move the upper 64 bits to the lower
+                        // 64 bits, leaving the top 64 bits as-is.
+                        let shuffle_mask = 0b11_10_11_10;
+                        let bitcast = pos.ins().raw_bitcast(F32X4, arg);
+                        pos.ins().x86_pshufd(bitcast, shuffle_mask)
+                    }
+                    _ => unreachable!(),
+                }
+            } else {
+                // Remove the extractlane instruction, leaving the float where it is.
+                arg
+            };
+            // Then we must bitcast to the right type.
+            pos.func
+                .dfg
+                .replace(inst)
+                .raw_bitcast(value_type.lane_type(), shuffled);
+        } else {
+            // For non-floats, lower with the usual PEXTR* instruction.
+            pos.func.dfg.replace(inst).x86_pextr(arg, lane);
+        }
+    }
+}
diff --git a/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif b/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif
new file mode 100644
index 0000000000..0a3b776a99
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif
@@ -0,0 +1,38 @@
+test binemit
+set enable_simd
+target x86_64 haswell
+
+; for extractlane, floats are legalized differently than integers and booleans; integers and booleans use x86_pextr
+; which is manually placed in the IR so that it can be binemit-tested
+
+function %test_extractlane_b8() {
+ebb0:
+[-, %rax]   v0 = bconst.b8 true
+[-, %xmm0]  v1 = splat.b8x16 v0
+[-, %rax]   v2 = x86_pextr v1, 10    ; bin: 66 0f 3a 14 c0 0a
+            return
+}
+
+function %test_extractlane_i16() {
+ebb0:
+[-, %rax]   v0 = iconst.i16 4
+[-, %xmm1]  v1 = splat.i16x8 v0
+[-, %rax]   v2 = x86_pextr v1, 4    ; bin: 66 0f c5 c8 04
+            return
+}
+
+function %test_extractlane_i32() {
+ebb0:
+[-, %rax]   v0 = iconst.i32 42
+[-, %xmm4]  v1 = splat.i32x4 v0
+[-, %rcx]   v2 = x86_pextr v1, 2    ; bin: 66 0f 3a 16 e1 02
+            return
+}
+
+function %test_extractlane_b64() {
+ebb0:
+[-, %rax]   v0 = bconst.b64 false
+[-, %xmm2]  v1 = splat.b64x2 v0
+[-, %rbx]   v2 = x86_pextr v1, 1    ; bin: 66 48 0f 3a 16 d3 01
+            return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/extractlane-run.clif b/cranelift/filetests/filetests/isa/x86/extractlane-run.clif
new file mode 100644
index 0000000000..ce8c00a933
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/extractlane-run.clif
@@ -0,0 +1,31 @@
+test run
+set enable_simd
+
+function %test_extractlane_b8() -> b8 {
+ebb0:
+    v1 = vconst.b8x16 [false false false false false false false false false false true false false
+    false false false]
+    v2 = extractlane v1, 10
+    return v2
+}
+; run
+
+function %test_extractlane_i16() -> b1 {
+ebb0:
+    v0 = vconst.i16x8 0x00080007000600050004000300020001
+    v1 = extractlane v0, 1
+    v2 = icmp_imm eq v1, 2
+    return v2
+}
+; run
+
+function %test_extractlane_f32() -> b1 {
+ebb0:
+    v0 = f32const 0x42.42
+    v1 = vconst.f32x4 [0x00.00 0x00.00 0x00.00 0x42.42]
+    v2 = extractlane v1, 3
+    v10 = f32const 0x42.42  ; TODO this should not be necessary, v0 should be re-usable
+    v3 = fcmp eq v2, v10
+    return v3
+}
+; run
diff --git a/cranelift/filetests/filetests/isa/x86/extractlane.clif b/cranelift/filetests/filetests/isa/x86/extractlane.clif
deleted file mode 100644
index e7a1ea898e..0000000000
--- a/cranelift/filetests/filetests/isa/x86/extractlane.clif
+++ /dev/null
@@ -1,35 +0,0 @@
-test binemit
-set enable_simd
-target x86_64 haswell
-
-function %test_extractlane_b8() {
-ebb0:
-[-, %rax]   v0 = bconst.b8 true
-[-, %xmm0]  v1 = splat.b8x16 v0
-[-, %rax]   v2 = extractlane v1, 10    ; bin: 66 0f 3a 14 c0 0a
-            return
-}
-
-function %test_extractlane_i16() {
-ebb0:
-[-, %rax]   v0 = iconst.i16 4
-[-, %xmm1]  v1 = splat.i16x8 v0
-[-, %rax]   v2 = extractlane v1, 4    ; bin: 66 0f c5 c8 04
-            return
-}
-
-function %test_extractlane_i32() {
-ebb0:
-[-, %rax]   v0 = iconst.i32 42
-[-, %xmm4]  v1 = splat.i32x4 v0
-[-, %rcx]   v2 = extractlane v1, 2    ; bin: 66 0f 3a 16 e1 02
-            return
-}
-
-function %test_extractlane_f64() {
-ebb0:
-[-, %rax]   v0 = f64const 0x0.0
-[-, %xmm2]  v1 = splat.f64x2 v0
-[-, %rbx]   v2 = extractlane v1, 1    ; bin: 66 48 0f 3a 16 d3 01
-            return
-}

From 3dfc68afb1b0771f41c189889a4070701be02897 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 22 Aug 2019 13:59:32 -0700
Subject: [PATCH 2701/3084] Avoid extra register movement when lowering the x86
 scalar_to_vector of a float value

---
 cranelift/codegen/meta/src/cdsl/types.rs      | 21 ++++++++
 .../codegen/meta/src/isa/x86/encodings.rs     | 52 ++++++++++++-------
 .../filetests/isa/x86/extractlane-run.clif    |  3 +-
 ...tor.clif => scalar_to_vector-binemit.clif} |  6 +--
 .../isa/x86/scalar_to_vector-compile.clif     | 19 +++++++
 5 files changed, 76 insertions(+), 25 deletions(-)
 rename cranelift/filetests/filetests/isa/x86/{scalar_to_vector.clif => scalar_to_vector-binemit.clif} (80%)
 create mode 100644 cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif

diff --git a/cranelift/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
index f431bb3ed7..92b9ab3a2f 100644
--- a/cranelift/codegen/meta/src/cdsl/types.rs
+++ b/cranelift/codegen/meta/src/cdsl/types.rs
@@ -264,6 +264,27 @@ impl LaneType {
             ValueType::Vector(VectorType::new(*self, lanes.into()))
         }
     }
+
+    pub fn is_float(&self) -> bool {
+        match self {
+            LaneType::FloatType(_) => true,
+            _ => false,
+        }
+    }
+
+    pub fn is_int(&self) -> bool {
+        match self {
+            LaneType::IntType(_) => true,
+            _ => false,
+        }
+    }
+
+    pub fn is_bool(&self) -> bool {
+        match self {
+            LaneType::BoolType(_) => true,
+            _ => false,
+        }
+    }
 }
 
 impl fmt::Display for LaneType {
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 30246b85ad..b95705f9bc 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -4,8 +4,8 @@ use std::collections::HashMap;
 
 use crate::cdsl::encodings::{Encoding, EncodingBuilder};
 use crate::cdsl::instructions::{
-    BoundInstruction, InstSpec, Instruction, InstructionGroup, InstructionPredicate,
-    InstructionPredicateNode, InstructionPredicateRegistry,
+    InstSpec, Instruction, InstructionGroup, InstructionPredicate, InstructionPredicateNode,
+    InstructionPredicateRegistry,
 };
 use crate::cdsl::recipes::{EncodingRecipe, EncodingRecipeNumber, Recipes};
 use crate::cdsl::settings::{SettingGroup, SettingPredicateNumber};
@@ -279,6 +279,17 @@ impl PerCpuModeEncodings {
         }
     }
 
+    /// Add the same encoding/recipe pairing to both X86_32 and X86_64
+    fn enc_32_64_rec(
+        &mut self,
+        inst: impl Clone + Into,
+        recipe: &EncodingRecipe,
+        bits: u16,
+    ) {
+        self.enc32_rec(inst.clone(), recipe, bits);
+        self.enc64_rec(inst, recipe, bits);
+    }
+
     /// Add the same encoding to both X86_32 and X86_64; assumes configuration (e.g. REX, operand
     /// binding) has already happened.
     fn enc_32_64_maybe_isap(
@@ -1761,12 +1772,16 @@ pub(crate) fn define(
     // written to the low doubleword of the register and the regiser is zero-extended to 128 bits."
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         let instruction = scalar_to_vector.bind_vector_from_lane(ty, sse_vector_size);
-        let template = rec_frurm.opcodes(vec![0x66, 0x0f, 0x6e]); // MOVD/MOVQ
-        if ty.lane_bits() < 64 {
-            // no 32-bit encodings for 64-bit widths
-            e.enc32(instruction.clone(), template.clone());
+        if ty.is_float() {
+            e.enc_32_64_rec(instruction, rec_null_fpr, 0);
+        } else {
+            let template = rec_frurm.opcodes(vec![0x66, 0x0f, 0x6e]); // MOVD/MOVQ
+            if ty.lane_bits() < 64 {
+                // no 32-bit encodings for 64-bit widths
+                e.enc32(instruction.clone(), template.clone());
+            }
+            e.enc_x86_64(instruction, template);
         }
-        e.enc_x86_64(instruction, template);
     }
 
     // SIMD insertlane
@@ -1811,37 +1826,34 @@ pub(crate) fn define(
         }
     }
 
-    // helper for generating null encodings for FPRs on both 32- and 64-bit architectures
-    let mut null_encode_32_64 = |instruction: BoundInstruction| {
-        e.enc32_rec(instruction.clone(), rec_null_fpr, 0);
-        e.enc64_rec(instruction, rec_null_fpr, 0);
-    };
-
     // SIMD bitcast all 128-bit vectors to each other (for legalizing splat.x16x8)
     for from_type in ValueType::all_lane_types().filter(allowed_simd_type) {
         for to_type in
             ValueType::all_lane_types().filter(|t| allowed_simd_type(t) && *t != from_type)
         {
-            null_encode_32_64(
-                raw_bitcast
-                    .bind_vector_from_lane(to_type, sse_vector_size)
-                    .bind_vector_from_lane(from_type, sse_vector_size),
-            );
+            let instruction = raw_bitcast
+                .bind_vector_from_lane(to_type, sse_vector_size)
+                .bind_vector_from_lane(from_type, sse_vector_size);
+            e.enc_32_64_rec(instruction, rec_null_fpr, 0);
         }
     }
 
     // SIMD raw bitcast floats to vector (and back); assumes that floats are already stored in an XMM register
     for float_type in &[F32, F64] {
         for lane_type in ValueType::all_lane_types().filter(allowed_simd_type) {
-            null_encode_32_64(
+            e.enc_32_64_rec(
                 raw_bitcast
                     .bind_vector_from_lane(lane_type, sse_vector_size)
                     .bind(*float_type),
+                rec_null_fpr,
+                0,
             );
-            null_encode_32_64(
+            e.enc_32_64_rec(
                 raw_bitcast
                     .bind(*float_type)
                     .bind_vector_from_lane(lane_type, sse_vector_size),
+                rec_null_fpr,
+                0,
             );
         }
     }
diff --git a/cranelift/filetests/filetests/isa/x86/extractlane-run.clif b/cranelift/filetests/filetests/isa/x86/extractlane-run.clif
index ce8c00a933..4590bd0673 100644
--- a/cranelift/filetests/filetests/isa/x86/extractlane-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/extractlane-run.clif
@@ -24,8 +24,7 @@ ebb0:
     v0 = f32const 0x42.42
     v1 = vconst.f32x4 [0x00.00 0x00.00 0x00.00 0x42.42]
     v2 = extractlane v1, 3
-    v10 = f32const 0x42.42  ; TODO this should not be necessary, v0 should be re-usable
-    v3 = fcmp eq v2, v10
+    v3 = fcmp eq v2, v0
     return v3
 }
 ; run
diff --git a/cranelift/filetests/filetests/isa/x86/scalar_to_vector.clif b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif
similarity index 80%
rename from cranelift/filetests/filetests/isa/x86/scalar_to_vector.clif
rename to cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif
index 51ddea3e7e..b26f3d2e6b 100644
--- a/cranelift/filetests/filetests/isa/x86/scalar_to_vector.clif
+++ b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif
@@ -17,10 +17,10 @@ ebb0:
             return
 }
 
-function %test_scalar_to_vector_f32() {
+function %test_scalar_to_vector_b32() {
 ebb0:
-[-, %rcx]   v0 = f32const 0x0.42
-[-, %xmm3]  v1 = scalar_to_vector.f32x4 v0    ; bin: 66 0f 6e d9
+[-, %rcx]   v0 = bconst.b32 false
+[-, %xmm3]  v1 = scalar_to_vector.b32x4 v0    ; bin: 66 0f 6e d9
             return
 }
 
diff --git a/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif
new file mode 100644
index 0000000000..2d2ab331f7
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif
@@ -0,0 +1,19 @@
+test compile
+set opt_level=best
+set probestack_enabled=false
+set enable_simd
+target x86_64
+
+; ensure that scalar_to_vector emits no instructions for floats (already exist in an XMM register)
+function %test_scalar_to_vector_f32() -> f32x4 baldrdash_system_v {
+ebb0:
+    v0 = f32const 0x0.42
+    v1 = scalar_to_vector.f32x4 v0
+    return v1
+}
+
+; check: ebb0
+; nextln: v2 = iconst.i32 0x3e84_0000
+; nextln: v0 = bitcast.f32 v2
+; nextln: [null_fpr#00,%xmm0]                 v1 = scalar_to_vector.f32x4 v0
+; nextln: return v1

From 295b2ef614e815cf27feacf553cac05d060e4f6b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 23 Aug 2019 11:38:29 -0700
Subject: [PATCH 2702/3084] Avoid extra register movement when lowering an x86
 insertlane to a float vector

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 49 +++++++++---
 .../codegen/meta/src/isa/x86/instructions.rs  | 79 +++++++++++++++++++
 .../codegen/meta/src/isa/x86/legalize.rs      |  1 +
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 21 +++++
 cranelift/codegen/src/isa/x86/enc_tables.rs   | 62 +++++++++++++++
 cranelift/codegen/src/verifier/locations.rs   |  6 +-
 .../filetests/isa/x86/extractlane-run.clif    | 38 +++++++++
 .../filetests/isa/x86/insertlane-binemit.clif | 42 ++++++++++
 .../filetests/isa/x86/insertlane-run.clif     | 48 +++++++++++
 .../filetests/isa/x86/insertlane.clif         | 39 ---------
 .../filetests/isa/x86/legalize-splat.clif     |  4 +-
 11 files changed, 334 insertions(+), 55 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/insertlane-binemit.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/insertlane-run.clif
 delete mode 100644 cranelift/filetests/filetests/isa/x86/insertlane.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index b95705f9bc..14b3c0eea9 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -396,7 +396,6 @@ pub(crate) fn define(
     let ifcmp_sp = shared.by_name("ifcmp_sp");
     let imul = shared.by_name("imul");
     let indirect_jump_table_br = shared.by_name("indirect_jump_table_br");
-    let insertlane = shared.by_name("insertlane");
     let ireduce = shared.by_name("ireduce");
     let ishl = shared.by_name("ishl");
     let ishl_imm = shared.by_name("ishl_imm");
@@ -469,8 +468,12 @@ pub(crate) fn define(
     let x86_cvtt2si = x86.by_name("x86_cvtt2si");
     let x86_fmax = x86.by_name("x86_fmax");
     let x86_fmin = x86.by_name("x86_fmin");
+    let x86_insertps = x86.by_name("x86_insertps");
+    let x86_movlhps = x86.by_name("x86_movlhps");
+    let x86_movsd = x86.by_name("x86_movsd");
     let x86_pop = x86.by_name("x86_pop");
     let x86_pextr = x86.by_name("x86_pextr");
+    let x86_pinsr = x86.by_name("x86_pinsr");
     let x86_pshufd = x86.by_name("x86_pshufd");
     let x86_pshufb = x86.by_name("x86_pshufb");
     let x86_push = x86.by_name("x86_push");
@@ -501,6 +504,7 @@ pub(crate) fn define(
     let rec_f64imm_z = r.template("f64imm_z");
     let rec_fa = r.template("fa");
     let rec_fax = r.template("fax");
+    let rec_fa_ib = r.template("fa_ib");
     let rec_fcmp = r.template("fcmp");
     let rec_fcscc = r.template("fcscc");
     let rec_ffillnull = r.recipe("ffillnull");
@@ -1785,16 +1789,16 @@ pub(crate) fn define(
     }
 
     // SIMD insertlane
-    let mut insertlane_mapping: HashMap, Option)> =
+    let mut x86_pinsr_mapping: HashMap, Option)> =
         HashMap::new();
-    insertlane_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x20], Some(use_sse41_simd))); // PINSRB
-    insertlane_mapping.insert(16, (vec![0x66, 0x0f, 0xc4], None)); // PINSRW from SSE2
-    insertlane_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41_simd))); // PINSRD
-    insertlane_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41_simd))); // PINSRQ, only x86_64
+    x86_pinsr_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x20], Some(use_sse41_simd))); // PINSRB
+    x86_pinsr_mapping.insert(16, (vec![0x66, 0x0f, 0xc4], None)); // PINSRW from SSE2
+    x86_pinsr_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41_simd))); // PINSRD
+    x86_pinsr_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41_simd))); // PINSRQ, only x86_64
 
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
-        if let Some((opcode, isap)) = insertlane_mapping.get(&ty.lane_bits()) {
-            let instruction = insertlane.bind_vector_from_lane(ty, sse_vector_size);
+        if let Some((opcode, isap)) = x86_pinsr_mapping.get(&ty.lane_bits()) {
+            let instruction = x86_pinsr.bind_vector_from_lane(ty, sse_vector_size);
             let template = rec_r_ib_unsigned_r.opcodes(opcode.clone());
             if ty.lane_bits() < 64 {
                 e.enc_32_64_maybe_isap(instruction, template.nonrex(), isap.clone());
@@ -1805,13 +1809,34 @@ pub(crate) fn define(
         }
     }
 
+    // for legalizing insertlane with floats, INSERTPS from SSE4.1
+    {
+        let instruction = x86_insertps.bind_vector_from_lane(F32, sse_vector_size);
+        let template = rec_fa_ib.nonrex().opcodes(vec![0x66, 0x0f, 0x3a, 0x21]);
+        e.enc_32_64_maybe_isap(instruction, template, Some(use_sse41_simd));
+    }
+
+    // for legalizing insertlane with floats,  MOVSD from SSE2
+    {
+        let instruction = x86_movsd.bind_vector_from_lane(F64, sse_vector_size);
+        let template = rec_fa.nonrex().opcodes(vec![0xf2, 0x0f, 0x10]);
+        e.enc_32_64_maybe_isap(instruction, template, None); // from SSE2
+    }
+
+    // for legalizing insertlane with floats, MOVLHPS from SSE
+    {
+        let instruction = x86_movlhps.bind_vector_from_lane(F64, sse_vector_size);
+        let template = rec_fa.nonrex().opcodes(vec![0x0f, 0x16]);
+        e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
+    }
+
     // SIMD extractlane
     let mut x86_pextr_mapping: HashMap, Option)> =
         HashMap::new();
-    x86_pextr_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x14], Some(use_sse41))); // PEXTRB
-    x86_pextr_mapping.insert(16, (vec![0x66, 0x0f, 0xc5], None)); // PEXTRW from zSSE2, SSE4.1 has a PEXTRW that can move to reg/m16 but the opcode is four bytes
-    x86_pextr_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41))); // PEXTRD
-    x86_pextr_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41))); // PEXTRQ, only x86_64
+    x86_pextr_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x14], Some(use_sse41_simd))); // PEXTRB
+    x86_pextr_mapping.insert(16, (vec![0x66, 0x0f, 0xc5], None)); // PEXTRW from SSE2, SSE4.1 has a PEXTRW that can move to reg/m16 but the opcode is four bytes
+    x86_pextr_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41_simd))); // PEXTRD
+    x86_pextr_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41_simd))); // PEXTRQ, only x86_64
 
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         if let Some((opcode, isap)) = x86_pextr_mapping.get(&ty.lane_bits()) {
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 3f583c6edb..b9f2496a85 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -308,5 +308,84 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    let IBxN = &TypeVar::new(
+        "IBxN",
+        "A SIMD vector type containing only booleans and integers",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .bools(Interval::All)
+            .simd_lanes(Interval::All)
+            .includes_scalars(false)
+            .build(),
+    );
+    let x = &operand("x", IBxN);
+    let y = &operand_doc("y", &IBxN.lane_of(), "New lane value");
+    let a = &operand("a", IBxN);
+
+    ig.push(
+        Inst::new(
+            "x86_pinsr",
+            r#"
+        Insert ``y`` into ``x`` at lane ``Idx``.
+        The lane index, ``Idx``, is an immediate value, not an SSA value. It
+        must indicate a valid lane index for the type of ``x``.
+        "#,
+        )
+        .operands_in(vec![x, Idx, y])
+        .operands_out(vec![a]),
+    );
+
+    let FxN = &TypeVar::new(
+        "FxN",
+        "A SIMD vector type containing floats",
+        TypeSetBuilder::new()
+            .floats(Interval::All)
+            .simd_lanes(Interval::All)
+            .includes_scalars(false)
+            .build(),
+    );
+    let x = &operand("x", FxN);
+    let y = &operand_doc("y", &FxN.lane_of(), "New lane value");
+    let a = &operand("a", FxN);
+
+    ig.push(
+        Inst::new(
+            "x86_insertps",
+            r#"
+        Insert a lane of ``y`` into ``x`` at using ``Idx`` to encode both which lane the value is 
+        extracted from and which it is inserted to. This is similar to x86_pinsr but inserts 
+        floats, which are already stored in an XMM register.
+        "#,
+        )
+        .operands_in(vec![x, Idx, y])
+        .operands_out(vec![a]),
+    );
+
+    let x = &operand("x", FxN);
+    let y = &operand("y", FxN);
+    let a = &operand("a", FxN);
+
+    ig.push(
+        Inst::new(
+            "x86_movsd",
+            r#"
+        Move the low 64 bits of the float vector ``y`` to the low 64 bits of float vector ``x``
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
+    ig.push(
+        Inst::new(
+            "x86_movlhps",
+            r#"
+        Move the low 64 bits of the float vector ``y`` to the high 64 bits of float vector ``x``
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
     ig.build()
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 4c2ebaefd4..555a93f9cb 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -381,6 +381,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     }
 
     narrow.custom_legalize(extractlane, "convert_extractlane");
+    narrow.custom_legalize(insertlane, "convert_insertlane");
 
     narrow.build_and_add_to(&mut shared.transform_groups);
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 3f14769dee..8176effc42 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -566,6 +566,27 @@ pub(crate) fn define<'shared>(
             ),
     );
 
+    // XX /r with FPR ins and outs. A form with a byte immediate.
+    {
+        let format = formats.get(f_insert_lane);
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("fa_ib", f_insert_lane, 2)
+                .operands_in(vec![fpr, fpr])
+                .operands_out(vec![0])
+                .inst_predicate(InstructionPredicate::new_is_unsigned_int(
+                    format, "lane", 8, 0,
+                ))
+                .emit(
+                    r#"
+                    {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                    modrm_rr(in_reg1, in_reg0, sink);
+                    let imm:i64 = lane.into();
+                    sink.put1(imm as u8);
+                "#,
+                ),
+        );
+    }
+
     // XX /n for a unary operation with extension bits.
     recipes.add_template_recipe(
         EncodingRecipeBuilder::new("ur", f_unary, 1)
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 39bdb57845..f67d7f0b69 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -950,3 +950,65 @@ fn convert_extractlane(
         }
     }
 }
+
+/// Because floats exist in XMM registers, we can keep them there when executing a CLIF
+/// insertlane instruction
+fn convert_insertlane(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    _cfg: &mut ControlFlowGraph,
+    _isa: &dyn TargetIsa,
+) {
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+    pos.use_srcloc(inst);
+
+    if let ir::InstructionData::InsertLane {
+        opcode: ir::Opcode::Insertlane,
+        args: [vector, replacement],
+        lane,
+    } = pos.func.dfg[inst]
+    {
+        let value_type = pos.func.dfg.value_type(vector);
+        if value_type.lane_type().is_float() {
+            // Floats are already in XMM registers and can stay there.
+            match value_type {
+                F32X4 => {
+                    assert!(lane > 0 && lane <= 3);
+                    let immediate = 0b00_00_00_00 | lane << 4;
+                    // Insert 32-bits from replacement (at index 00, bits 7:8) to vector (lane
+                    // shifted into bits 5:6).
+                    pos.func
+                        .dfg
+                        .replace(inst)
+                        .x86_insertps(vector, immediate, replacement)
+                }
+                F64X2 => {
+                    let replacement_as_vector = pos.ins().raw_bitcast(F64X2, replacement); // only necessary due to SSA types
+                    if lane == 0 {
+                        // Move the lowest quadword in replacement to vector without changing
+                        // the upper bits.
+                        pos.func
+                            .dfg
+                            .replace(inst)
+                            .x86_movsd(vector, replacement_as_vector)
+                    } else {
+                        assert_eq!(lane, 1);
+                        // Move the low 64 bits of replacement vector to the high 64 bits of the
+                        // vector.
+                        pos.func
+                            .dfg
+                            .replace(inst)
+                            .x86_movlhps(vector, replacement_as_vector)
+                    }
+                }
+                _ => unreachable!(),
+            };
+        } else {
+            // For non-floats, lower with the usual PINSR* instruction.
+            pos.func
+                .dfg
+                .replace(inst)
+                .x86_pinsr(vector, lane, replacement);
+        }
+    }
+}
diff --git a/cranelift/codegen/src/verifier/locations.rs b/cranelift/codegen/src/verifier/locations.rs
index bf1a4e1860..cf17ae13de 100644
--- a/cranelift/codegen/src/verifier/locations.rs
+++ b/cranelift/codegen/src/verifier/locations.rs
@@ -107,8 +107,10 @@ impl<'a> LocationVerifier<'a> {
         fatal!(
             errors,
             inst,
-            "{} constraints not satisfied",
-            self.encinfo.display(enc)
+            "{} constraints not satisfied in: {}\n{}",
+            self.encinfo.display(enc),
+            self.func.dfg.display_inst(inst, self.isa),
+            self.func.display(self.isa)
         )
     }
 
diff --git a/cranelift/filetests/filetests/isa/x86/extractlane-run.clif b/cranelift/filetests/filetests/isa/x86/extractlane-run.clif
index 4590bd0673..adb2e7b8e6 100644
--- a/cranelift/filetests/filetests/isa/x86/extractlane-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/extractlane-run.clif
@@ -28,3 +28,41 @@ ebb0:
     return v3
 }
 ; run
+
+function %test_extractlane_i32_with_vector_reuse() -> b1 {
+ebb0:
+    v0 = iconst.i32 42
+    v1 = iconst.i32 99
+
+    v2 = splat.i32x4 v0
+    v3 = insertlane v2, 2, v1
+
+    v4 = extractlane v3, 3
+    v5 = icmp eq v4, v0
+
+    v6 = extractlane v3, 2
+    v7 = icmp eq v6, v1
+
+    v8 = band v5, v7
+    return v8
+}
+; run
+
+function %test_extractlane_f32_with_vector_reuse() -> b1 {
+ebb0:
+    v0 = f32const 0x42.42
+    v1 = f32const 0x99.99
+
+    v2 = splat.f32x4 v0
+    v3 = insertlane v2, 2, v1
+
+    v4 = extractlane v3, 3
+    v5 = fcmp eq v4, v0
+
+    v6 = extractlane v3, 2
+    v7 = fcmp eq v6, v1
+
+    v8 = band v5, v7
+    return v8
+}
+; run
diff --git a/cranelift/filetests/filetests/isa/x86/insertlane-binemit.clif b/cranelift/filetests/filetests/isa/x86/insertlane-binemit.clif
new file mode 100644
index 0000000000..49048130c0
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/insertlane-binemit.clif
@@ -0,0 +1,42 @@
+test binemit
+set enable_simd
+target x86_64 haswell
+
+; for insertlane, floats are legalized differently than integers and booleans; integers and booleans use x86_pinsr
+; which is manually placed in the IR so that it can be binemit-tested
+
+function %test_insertlane_b8() {
+ebb0:
+[-, %rax]   v0 = bconst.b8 true
+[-, %rbx]   v1 = bconst.b8 false
+[-, %xmm0]  v2 = splat.b8x16 v0
+[-, %xmm0]  v3 = x86_pinsr v2, 10, v1    ; bin: 66 0f 3a 20 c3 0a
+            return
+}
+
+function %test_insertlane_i16() {
+ebb0:
+[-, %rax]   v0 = iconst.i16 4
+[-, %rbx]   v1 = iconst.i16 5
+[-, %xmm1]  v2 = splat.i16x8 v0
+[-, %xmm1]  v3 = x86_pinsr v2, 4, v1    ; bin: 66 0f c4 cb 04
+            return
+}
+
+function %test_insertlane_i32() {
+ebb0:
+[-, %rax]   v0 = iconst.i32 42
+[-, %rbx]   v1 = iconst.i32 99
+[-, %xmm4]  v2 = splat.i32x4 v0
+[-, %xmm4]  v3 = x86_pinsr v2, 2, v1    ; bin: 66 0f 3a 22 e3 02
+            return
+}
+
+function %test_insertlane_b64() {
+ebb0:
+[-, %rax]   v0 = bconst.b64 true
+[-, %rbx]   v1 = bconst.b64 false
+[-, %xmm2]  v2 = splat.b64x2 v0
+[-, %xmm2]  v3 = x86_pinsr v2, 1, v1    ; bin: 66 48 0f 3a 22 d3 01
+            return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/insertlane-run.clif b/cranelift/filetests/filetests/isa/x86/insertlane-run.clif
new file mode 100644
index 0000000000..92fb38202e
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/insertlane-run.clif
@@ -0,0 +1,48 @@
+test run
+set enable_simd
+
+; TODO once SIMD vector comparison is implemented, remove use of extractlane below
+
+function %test_insertlane_b8() -> b8 {
+ebb0:
+    v1 = bconst.b8 true
+    v2 = vconst.b8x16 [false false false false false false false false false false false false false
+     false false false]
+    v3 = insertlane v2, 10, v1
+    v4 = extractlane v3, 10
+    return v4
+}
+; run
+
+function %test_insertlane_f32() -> b1 {
+ebb0:
+    v0 = f32const 0x42.42
+    v1 = vconst.f32x4 0x00
+    v2 = insertlane v1, 1, v0
+    v3 = extractlane v2, 1
+    v4 = fcmp eq v3, v0
+    return v4
+}
+; run
+
+function %test_insertlane_f64_lane1() -> b1 {
+ebb0:
+    v0 = f64const 0x42.42
+    v1 = vconst.f64x2 0x00
+    v2 = insertlane v1, 1, v0
+    v3 = extractlane v2, 1
+    v4 = fcmp eq v3, v0
+    return v4
+}
+; run
+
+function %test_insertlane_f64_lane0() -> b1 {
+ebb0:
+    v0 = f64const 0x42.42
+    v1 = vconst.f64x2 0x00
+    v2 = insertlane v1, 0, v0
+    v3 = extractlane v2, 0
+    v4 = fcmp eq v3, v0
+    return v4
+}
+; run
diff --git a/cranelift/filetests/filetests/isa/x86/insertlane.clif b/cranelift/filetests/filetests/isa/x86/insertlane.clif
deleted file mode 100644
index c55dc40333..0000000000
--- a/cranelift/filetests/filetests/isa/x86/insertlane.clif
+++ /dev/null
@@ -1,39 +0,0 @@
-test binemit
-set enable_simd
-target x86_64 haswell
-
-function %test_insertlane_b8() {
-ebb0:
-[-, %rax]   v0 = bconst.b8 true
-[-, %rbx]   v1 = bconst.b8 false
-[-, %xmm0]  v2 = splat.b8x16 v0
-[-, %xmm0]  v3 = insertlane v2, 10, v1    ; bin: 66 0f 3a 20 c3 0a
-            return
-}
-
-function %test_insertlane_i16() {
-ebb0:
-[-, %rax]   v0 = iconst.i16 4
-[-, %rbx]   v1 = iconst.i16 5
-[-, %xmm1]  v2 = splat.i16x8 v0
-[-, %xmm1]  v3 = insertlane v2, 4, v1    ; bin: 66 0f c4 cb 04
-            return
-}
-
-function %test_insertlane_i32() {
-ebb0:
-[-, %rax]   v0 = iconst.i32 42
-[-, %rbx]   v1 = iconst.i32 99
-[-, %xmm4]  v2 = splat.i32x4 v0
-[-, %xmm4]  v3 = insertlane v2, 2, v1    ; bin: 66 0f 3a 22 e3 02
-            return
-}
-
-function %test_insertlane_f64() {
-ebb0:
-[-, %rax]   v0 = f64const 0x0.0
-[-, %rbx]   v1 = f64const 0x4.2
-[-, %xmm2]  v2 = splat.f64x2 v0
-[-, %xmm2]  v3 = insertlane v2, 1, v1    ; bin: 66 48 0f 3a 22 d3 01
-            return
-}
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-splat.clif b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
index c0fc83ebe7..19d61d529c 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
@@ -33,7 +33,7 @@ ebb0:
 ; check:   ebb0:
 ; nextln:     v0 = iconst.i64 42
 ; nextln:     v2 = scalar_to_vector.i64x2 v0
-; nextln:     v1 = insertlane v2, 1, v0
+; nextln:     v1 = x86_pinsr v2, 1, v0
 ; nextln:     return v1
 
 
@@ -48,7 +48,7 @@ ebb0:
 ; check:   ebb0:
 ; nextln:     v0 = bconst.b16 true
 ; nextln:     v2 = scalar_to_vector.b16x8 v0
-; nextln:     v3 = insertlane v2, 1, v0
+; nextln:     v3 = x86_pinsr v2, 1, v0
 ; nextln:     v4 = raw_bitcast.i32x4 v3
 ; nextln:     v5 = x86_pshufd v4, 0
 ; nextln:     v1 = raw_bitcast.b16x8 v5

From 6f1ed94e82227bd8a35d7232e8ea533cb68fd726 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 5 Sep 2019 09:26:11 -0700
Subject: [PATCH 2703/3084] Fix documentation

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 26 +++++++++++--------
 .../codegen/meta/src/shared/instructions.rs   | 12 ++++++---
 .../filetests/isa/x86/insertlane-binemit.clif |  4 +--
 3 files changed, 25 insertions(+), 17 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 14b3c0eea9..d773c2c662 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1744,16 +1744,17 @@ pub(crate) fn define(
     e.enc_both(ffcmp.bind(F32), rec_fcmp.opcodes(vec![0x0f, 0x2e]));
     e.enc_both(ffcmp.bind(F64), rec_fcmp.opcodes(vec![0x66, 0x0f, 0x2e]));
 
-    // SIMD vector size: eventually multiple vector sizes may be supported but for now only SSE-sized vectors are available
+    // SIMD vector size: eventually multiple vector sizes may be supported but for now only
+    // SSE-sized vectors are available.
     let sse_vector_size: u64 = 128;
 
     // SIMD splat: before x86 can use vector data, it must be moved to XMM registers; see
     // legalize.rs for how this is done; once there, x86_pshuf* (below) is used for broadcasting the
-    // value across the register
+    // value across the register.
 
     let allowed_simd_type = |t: &LaneType| t.lane_bits() >= 8 && t.lane_bits() < 128;
 
-    // PSHUFB, 8-bit shuffle using two XMM registers
+    // PSHUFB, 8-bit shuffle using two XMM registers.
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
         let instruction = x86_pshufb.bind_vector_from_lane(ty, sse_vector_size);
         let template = rec_fa.nonrex().opcodes(vec![0x66, 0x0f, 0x38, 00]);
@@ -1761,7 +1762,7 @@ pub(crate) fn define(
         e.enc64_isap(instruction, template, use_ssse3_simd);
     }
 
-    // PSHUFD, 32-bit shuffle using one XMM register and a u8 immediate
+    // PSHUFD, 32-bit shuffle using one XMM register and a u8 immediate.
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 32) {
         let instruction = x86_pshufd.bind_vector_from_lane(ty, sse_vector_size);
         let template = rec_r_ib_unsigned_fpr
@@ -1803,27 +1804,28 @@ pub(crate) fn define(
             if ty.lane_bits() < 64 {
                 e.enc_32_64_maybe_isap(instruction, template.nonrex(), isap.clone());
             } else {
-                // turns out the 64-bit widths have REX/W encodings and only are available on x86_64
+                // It turns out the 64-bit widths have REX/W encodings and only are available on
+                // x86_64.
                 e.enc64_maybe_isap(instruction, template.rex().w(), isap.clone());
             }
         }
     }
 
-    // for legalizing insertlane with floats, INSERTPS from SSE4.1
+    // For legalizing insertlane with floats, INSERTPS from SSE4.1.
     {
         let instruction = x86_insertps.bind_vector_from_lane(F32, sse_vector_size);
         let template = rec_fa_ib.nonrex().opcodes(vec![0x66, 0x0f, 0x3a, 0x21]);
         e.enc_32_64_maybe_isap(instruction, template, Some(use_sse41_simd));
     }
 
-    // for legalizing insertlane with floats,  MOVSD from SSE2
+    // For legalizing insertlane with floats,  MOVSD from SSE2.
     {
         let instruction = x86_movsd.bind_vector_from_lane(F64, sse_vector_size);
         let template = rec_fa.nonrex().opcodes(vec![0xf2, 0x0f, 0x10]);
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE2
     }
 
-    // for legalizing insertlane with floats, MOVLHPS from SSE
+    // For legalizing insertlane with floats, MOVLHPS from SSE.
     {
         let instruction = x86_movlhps.bind_vector_from_lane(F64, sse_vector_size);
         let template = rec_fa.nonrex().opcodes(vec![0x0f, 0x16]);
@@ -1845,13 +1847,14 @@ pub(crate) fn define(
             if ty.lane_bits() < 64 {
                 e.enc_32_64_maybe_isap(instruction, template.nonrex(), isap.clone());
             } else {
-                // turns out the 64-bit widths have REX/W encodings and only are available on x86_64
+                // It turns out the 64-bit widths have REX/W encodings and only are available on
+                // x86_64.
                 e.enc64_maybe_isap(instruction, template.rex().w(), isap.clone());
             }
         }
     }
 
-    // SIMD bitcast all 128-bit vectors to each other (for legalizing splat.x16x8)
+    // SIMD bitcast all 128-bit vectors to each other (for legalizing splat.x16x8).
     for from_type in ValueType::all_lane_types().filter(allowed_simd_type) {
         for to_type in
             ValueType::all_lane_types().filter(|t| allowed_simd_type(t) && *t != from_type)
@@ -1863,7 +1866,8 @@ pub(crate) fn define(
         }
     }
 
-    // SIMD raw bitcast floats to vector (and back); assumes that floats are already stored in an XMM register
+    // SIMD raw bitcast floats to vector (and back); assumes that floats are already stored in an
+    // XMM register.
     for float_type in &[F32, F64] {
         for lane_type in ValueType::all_lane_types().filter(allowed_simd_type) {
             e.enc_32_64_rec(
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 843347ce95..3ebebfe183 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1537,7 +1537,9 @@ pub(crate) fn define(
         Extract lane ``Idx`` from ``x``.
 
         The lane index, ``Idx``, is an immediate value, not an SSA value. It
-        must indicate a valid lane index for the type of ``x``.
+        must indicate a valid lane index for the type of ``x``. Note that the upper bits of ``a``
+        may or may not be zeroed depending on the ISA but the type system should prevent using 
+        ``a`` as anything other than the extracted value.
         "#,
         )
         .operands_in(vec![x, Idx])
@@ -2782,9 +2784,11 @@ pub(crate) fn define(
         Inst::new(
             "scalar_to_vector",
             r#"
-    Scalar To Vector -- move a value out of a scalar register and into a vector
-    register; the scalar will be moved to the lowest-order bits of the vector
-    register and any higher bits will be zeroed.
+    Scalar To Vector -- move a value out of a scalar register and into a vector register; the 
+    scalar will be moved to the lowest-order bits of the vector register. Note that this 
+    instruction is intended as a low-level legalization instruction and frontends should prefer 
+    insertlane; on certain architectures, scalar_to_vector may zero the highest-order bits for some
+    types (e.g. integers) but not for others (e.g. floats).
     "#,
         )
         .operands_in(vec![s])
diff --git a/cranelift/filetests/filetests/isa/x86/insertlane-binemit.clif b/cranelift/filetests/filetests/isa/x86/insertlane-binemit.clif
index 49048130c0..c388ed6fae 100644
--- a/cranelift/filetests/filetests/isa/x86/insertlane-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/insertlane-binemit.clif
@@ -2,8 +2,8 @@ test binemit
 set enable_simd
 target x86_64 haswell
 
-; for insertlane, floats are legalized differently than integers and booleans; integers and booleans use x86_pinsr
-; which is manually placed in the IR so that it can be binemit-tested
+; for insertlane, floats are legalized differently than integers and booleans; integers and
+; booleans use x86_pinsr which is manually placed in the IR so that it can be binemit-tested
 
 function %test_insertlane_b8() {
 ebb0:

From 3aa76b558ce9348b23ff9f14529032c0b5c6ca67 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 30 Aug 2019 13:01:03 +0200
Subject: [PATCH 2704/3084] Legalize i64.const by breaking it into two
 i32.const, on 32-bits platforms;

---
 cranelift/codegen/meta/src/shared/legalize.rs |  4 +++
 cranelift/codegen/src/legalizer/mod.rs        | 34 ++++++++++++++++++-
 .../filetests/isa/x86/legalize-custom.clif    |  8 -----
 .../isa/x86/legalize-f64const-x64.clif        | 13 +++++++
 .../filetests/legalizer/iconst-i64.clif       | 12 +++++++
 5 files changed, 62 insertions(+), 9 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/legalize-f64const-x64.clif
 create mode 100644 cranelift/filetests/filetests/legalizer/iconst-i64.clif

diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 9588272709..1c2c510460 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -188,6 +188,10 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     narrow.custom_legalize(load, "narrow_load");
     narrow.custom_legalize(store, "narrow_store");
 
+    // iconst.i64 can't be legalized in the meta langage (because integer literals can't be
+    // embedded as part of arguments), so use a custom legalization for now.
+    narrow.custom_legalize(iconst, "narrow_iconst");
+
     narrow.legalize(
         def!(a = iadd(x, y)),
         vec![
diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 0f24689d8c..2fd353c7b9 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -16,7 +16,7 @@
 use crate::bitset::BitSet;
 use crate::cursor::{Cursor, FuncCursor};
 use crate::flowgraph::ControlFlowGraph;
-use crate::ir::types::I32;
+use crate::ir::types::{I32, I64};
 use crate::ir::{self, InstBuilder, MemFlags};
 use crate::isa::TargetIsa;
 use crate::predicates;
@@ -633,3 +633,35 @@ fn narrow_store(
     );
     pos.remove_inst();
 }
+
+/// Expands an illegal iconst value by splitting it into two.
+fn narrow_iconst(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    _cfg: &mut ControlFlowGraph,
+    isa: &dyn TargetIsa,
+) {
+    let imm: i64 = if let ir::InstructionData::UnaryImm {
+        opcode: ir::Opcode::Iconst,
+        imm,
+    } = &func.dfg[inst]
+    {
+        (*imm).into()
+    } else {
+        panic!("unexpected instruction in narrow_iconst");
+    };
+
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+    pos.use_srcloc(inst);
+
+    let ty = pos.func.dfg.ctrl_typevar(inst);
+    if isa.pointer_bits() == 32 && ty == I64 {
+        let low = pos.ins().iconst(I32, imm & 0xffffffff);
+        let high = pos.ins().iconst(I32, imm >> 32);
+        // The instruction has as many results as iconcat, so no need to replace them.
+        pos.func.dfg.replace(inst).iconcat(low, high);
+        return;
+    }
+
+    unimplemented!("missing encoding or legalization for iconst.{:?}", ty);
+}
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-custom.clif b/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
index 747f4e819b..2657bfd497 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
@@ -62,14 +62,6 @@ ebb0:
     return v1
 }
 
-function %f64const() -> f64 {
-ebb0:
-    v1 = f64const 0x1.0p1
-    ; check: $(tmp=$V) = iconst.i64
-    ; check: v1 = bitcast.f64 $tmp
-    return v1
-}
-
 function %select_f64(f64, f64, i32) -> f64 {
 ebb0(v0: f64, v1: f64, v2: i32):
     v3 = select v2, v0, v1
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-f64const-x64.clif b/cranelift/filetests/filetests/isa/x86/legalize-f64const-x64.clif
new file mode 100644
index 0000000000..addafe90a3
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/legalize-f64const-x64.clif
@@ -0,0 +1,13 @@
+; Test the legalization of f64const.
+test legalizer
+target x86_64
+
+; regex: V=v\d+
+
+function %f64const() -> f64 {
+ebb0:
+    v1 = f64const 0x1.0p1
+    ; check: $(tmp=$V) = iconst.i64
+    ; check: v1 = bitcast.f64 $tmp
+    return v1
+}
diff --git a/cranelift/filetests/filetests/legalizer/iconst-i64.clif b/cranelift/filetests/filetests/legalizer/iconst-i64.clif
new file mode 100644
index 0000000000..a3c9168416
--- /dev/null
+++ b/cranelift/filetests/filetests/legalizer/iconst-i64.clif
@@ -0,0 +1,12 @@
+test legalizer
+target i686
+
+function %foo() -> i64 {
+ebb0:
+    v1 = iconst.i64 0x6400000042
+    return v1
+}
+
+; check: v2 = iconst.i32 66
+; check: v3 = iconst.i32 100
+; check: v1 = iconcat v2, v3

From cad20745876e3f7c45b2801fdb2b29cd4f4c402b Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 9 Sep 2019 18:18:41 +0200
Subject: [PATCH 2705/3084] [codegen] Don't simplify an operation if it would
 result in unnecessary legalization;

Converting something like iadd.i64 on a 32-bits architecture into a
iadd_imm.i64 will result in the instruction being legalized back to an
iadd.i64 later on, creating unnecessary churn.

This commit implements avoid doing so, and changes the target ISA to a
64-bits platform for tests than ran into this, as well as making sure
this won't happen on 32-bits platforms.
---
 cranelift/codegen/src/context.rs              |  2 +-
 cranelift/codegen/src/simple_preopt.rs        | 49 +++++++++------
 .../simple_preopt/div_by_const_indirect.clif  |  2 +-
 .../filetests/simple_preopt/simplify32.clif   | 61 +++++++++++++++++++
 .../{simplify.clif => simplify64.clif}        |  4 +-
 5 files changed, 98 insertions(+), 20 deletions(-)
 create mode 100644 cranelift/filetests/filetests/simple_preopt/simplify32.clif
 rename cranelift/filetests/filetests/simple_preopt/{simplify.clif => simplify64.clif} (99%)

diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index 5350b3444c..0f4e9ad420 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -243,7 +243,7 @@ impl Context {
 
     /// Perform pre-legalization rewrites on the function.
     pub fn preopt(&mut self, isa: &dyn TargetIsa) -> CodegenResult<()> {
-        do_preopt(&mut self.func, &mut self.cfg);
+        do_preopt(&mut self.func, &mut self.cfg, isa);
         self.verify_if(isa)?;
         Ok(())
     }
diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index 8219f6627c..221e8a57b0 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -16,6 +16,7 @@ use crate::ir::{
     types::{I16, I32, I64, I8},
     DataFlowGraph, Ebb, Function, Inst, InstBuilder, InstructionData, Type, Value,
 };
+use crate::isa::TargetIsa;
 use crate::timing;
 
 #[inline]
@@ -533,9 +534,14 @@ fn try_fold_extended_move(
 
 /// Apply basic simplifications.
 ///
-/// This folds constants with arithmetic to form `_imm` instructions, and other
-/// minor simplifications.
-fn simplify(pos: &mut FuncCursor, inst: Inst) {
+/// This folds constants with arithmetic to form `_imm` instructions, and other minor
+/// simplifications.
+///
+/// Doesn't apply some simplifications if the native word width (in bytes) is smaller than the
+/// controlling type's width of the instruction. This would result in an illegal instruction that
+/// would likely be expanded back into an instruction on smaller types with the same initial
+/// opcode, creating unnecessary churn.
+fn simplify(pos: &mut FuncCursor, inst: Inst, native_word_width: u32) {
     match pos.func.dfg[inst] {
         InstructionData::Binary { opcode, args } => {
             if let Some(mut imm) = resolve_imm64_value(&pos.func.dfg, args[1]) {
@@ -562,13 +568,15 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
                     _ => return,
                 };
                 let ty = pos.func.dfg.ctrl_typevar(inst);
-                pos.func
-                    .dfg
-                    .replace(inst)
-                    .BinaryImm(new_opcode, ty, imm, args[0]);
+                if ty.bytes() <= native_word_width {
+                    pos.func
+                        .dfg
+                        .replace(inst)
+                        .BinaryImm(new_opcode, ty, imm, args[0]);
 
-                // Repeat for BinaryImm simplification.
-                simplify(pos, inst);
+                    // Repeat for BinaryImm simplification.
+                    simplify(pos, inst, native_word_width);
+                }
             } else if let Some(imm) = resolve_imm64_value(&pos.func.dfg, args[0]) {
                 let new_opcode = match opcode {
                     Opcode::Iadd => Opcode::IaddImm,
@@ -580,10 +588,12 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
                     _ => return,
                 };
                 let ty = pos.func.dfg.ctrl_typevar(inst);
-                pos.func
-                    .dfg
-                    .replace(inst)
-                    .BinaryImm(new_opcode, ty, imm, args[1]);
+                if ty.bytes() <= native_word_width {
+                    pos.func
+                        .dfg
+                        .replace(inst)
+                        .BinaryImm(new_opcode, ty, imm, args[1]);
+                }
             }
         }
 
@@ -643,7 +653,9 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
                 }
 
                 Opcode::UshrImm | Opcode::SshrImm => {
-                    if try_fold_extended_move(pos, inst, opcode, arg, imm) {
+                    if pos.func.dfg.ctrl_typevar(inst).bytes() <= native_word_width
+                        && try_fold_extended_move(pos, inst, opcode, arg, imm)
+                    {
                         return;
                     }
                 }
@@ -686,7 +698,9 @@ fn simplify(pos: &mut FuncCursor, inst: Inst) {
         InstructionData::IntCompare { opcode, cond, args } => {
             debug_assert_eq!(opcode, Opcode::Icmp);
             if let Some(imm) = resolve_imm64_value(&pos.func.dfg, args[1]) {
-                pos.func.dfg.replace(inst).icmp_imm(cond, args[0], imm);
+                if pos.func.dfg.ctrl_typevar(inst).bytes() <= native_word_width {
+                    pos.func.dfg.replace(inst).icmp_imm(cond, args[0], imm);
+                }
             }
         }
 
@@ -937,13 +951,14 @@ fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst
 }
 
 /// The main pre-opt pass.
-pub fn do_preopt(func: &mut Function, cfg: &mut ControlFlowGraph) {
+pub fn do_preopt(func: &mut Function, cfg: &mut ControlFlowGraph, isa: &dyn TargetIsa) {
     let _tt = timing::preopt();
     let mut pos = FuncCursor::new(func);
+    let native_word_width = isa.pointer_bytes();
     while let Some(ebb) = pos.next_ebb() {
         while let Some(inst) = pos.next_inst() {
             // Apply basic simplifications.
-            simplify(&mut pos, inst);
+            simplify(&mut pos, inst, native_word_width as u32);
 
             // Try to transform divide-by-constant into simpler operations.
             if let Some(divrem_info) = get_div_info(inst, &pos.func.dfg) {
diff --git a/cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif b/cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif
index 5833b0f371..c111113197 100644
--- a/cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif
+++ b/cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif
@@ -1,5 +1,5 @@
 test simple_preopt
-target i686 baseline
+target x86_64 baseline
 
 ; Cases where the denominator is created by an iconst
 
diff --git a/cranelift/filetests/filetests/simple_preopt/simplify32.clif b/cranelift/filetests/filetests/simple_preopt/simplify32.clif
new file mode 100644
index 0000000000..45add1b7a3
--- /dev/null
+++ b/cranelift/filetests/filetests/simple_preopt/simplify32.clif
@@ -0,0 +1,61 @@
+test simple_preopt
+target i686
+
+;; 32-bits platforms.
+
+function %iadd_imm(i32) -> i32 {
+ebb0(v0: i32):
+    v1 = iconst.i32 2
+    v2 = iadd v0, v1
+    return v2
+}
+; sameln: function %iadd_imm
+; nextln: ebb0(v0: i32):
+; nextln:     v1 = iconst.i32 2
+; nextln:     v2 = iadd_imm v0, 2
+; nextln:     return v2
+; nextln: }
+
+function %isub_imm(i32) -> i32 {
+ebb0(v0: i32):
+    v1 = iconst.i32 2
+    v2 = isub v0, v1
+    return v2
+}
+; sameln: function %isub_imm
+; nextln: ebb0(v0: i32):
+; nextln:     v1 = iconst.i32 2
+; nextln:     v2 = iadd_imm v0, -2
+; nextln:     return v2
+; nextln: }
+
+function %icmp_imm(i32) -> i32 {
+ebb0(v0: i32):
+    v1 = iconst.i32 2
+    v2 = icmp slt v0, v1
+    v3 = bint.i32 v2
+    return v3
+}
+; sameln: function %icmp_imm
+; nextln: ebb0(v0: i32):
+; nextln:     v1 = iconst.i32 2
+; nextln:     v2 = icmp_imm slt v0, 2
+; nextln:     v3 = bint.i32 v2
+; nextln:     return v3
+; nextln: }
+
+;; Don't simplify operations that would get illegal because of lack of native
+;; support.
+function %iadd_imm(i64) -> i64 {
+ebb0(v0: i64):
+    v1 = iconst.i64 2
+    v2 = iadd v0, v1
+    return v2
+}
+; sameln: function %iadd_imm
+; nextln: ebb0(v0: i64):
+; nextln:     v1 = iconst.i64 2
+; nextln:     v2 = iadd v0, v1
+; nextln:     return v2
+; nextln: }
+
diff --git a/cranelift/filetests/filetests/simple_preopt/simplify.clif b/cranelift/filetests/filetests/simple_preopt/simplify64.clif
similarity index 99%
rename from cranelift/filetests/filetests/simple_preopt/simplify.clif
rename to cranelift/filetests/filetests/simple_preopt/simplify64.clif
index e74be4738a..db485ce773 100644
--- a/cranelift/filetests/filetests/simple_preopt/simplify.clif
+++ b/cranelift/filetests/filetests/simple_preopt/simplify64.clif
@@ -1,5 +1,7 @@
 test simple_preopt
-target i686
+target x86_64
+
+;; 64-bits platforms.
 
 function %iadd_imm(i32) -> i32 {
 ebb0(v0: i32):

From abb157315b91df6b3c165d519d8be0be6323e0f4 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Tue, 10 Sep 2019 23:02:11 +0200
Subject: [PATCH 2706/3084] Basic Blocks: Fix frontend::sample_function test
 case. (#1007)

---
 cranelift/frontend/src/frontend.rs | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 29330a88df..064ab7053c 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -908,6 +908,7 @@ mod tests {
             let block0 = builder.create_ebb();
             let block1 = builder.create_ebb();
             let block2 = builder.create_ebb();
+            let block3 = builder.create_ebb();
             let x = Variable::new(0);
             let y = Variable::new(1);
             let z = Variable::new(2);
@@ -945,7 +946,13 @@ mod tests {
             }
             {
                 let arg = builder.use_var(y);
-                builder.ins().brnz(arg, block2, &[]);
+                builder.ins().brnz(arg, block3, &[]);
+            }
+            builder.ins().jump(block2, &[]);
+
+            builder.switch_to_block(block2);
+            if !lazy_seal {
+                builder.seal_block(block2);
             }
             {
                 let arg1 = builder.use_var(z);
@@ -958,9 +965,9 @@ mod tests {
                 builder.ins().return_(&[arg]);
             }
 
-            builder.switch_to_block(block2);
+            builder.switch_to_block(block3);
             if !lazy_seal {
-                builder.seal_block(block2);
+                builder.seal_block(block3);
             }
 
             {

From 35c6d869f55e9ea263498ef56d1f68b36d3388cb Mon Sep 17 00:00:00 2001
From: Wander Lairson Costa 
Date: Tue, 10 Sep 2019 19:11:20 -0300
Subject: [PATCH 2707/3084] Replace region::protect by make_exec in
 cranelift-filetests

---
 cranelift/filetests/src/function_runner.rs | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/cranelift/filetests/src/function_runner.rs b/cranelift/filetests/src/function_runner.rs
index 8837596ed9..6f2cd4a83a 100644
--- a/cranelift/filetests/src/function_runner.rs
+++ b/cranelift/filetests/src/function_runner.rs
@@ -5,8 +5,6 @@ use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::{settings, Context};
 use cranelift_native::builder as host_isa_builder;
 use memmap::MmapMut;
-use region;
-use region::Protection;
 
 /// Run a function on a host
 pub struct FunctionRunner {
@@ -66,7 +64,8 @@ impl FunctionRunner {
             .map_err(|e| e.to_string())?;
         let mut code_page =
             MmapMut::map_anon(code_info.total_size as usize).map_err(|e| e.to_string())?;
-        let callable_fn: fn() -> bool = unsafe {
+
+        unsafe {
             context.emit_to_memory(
                 self.isa.as_ref(),
                 code_page.as_mut_ptr(),
@@ -74,15 +73,11 @@ impl FunctionRunner {
                 traps,
                 stackmaps,
             );
-            region::protect(
-                code_page.as_mut_ptr(),
-                code_page.len(),
-                Protection::ReadExecute,
-            )
-            .map_err(|e| e.to_string())?;
-            mem::transmute(code_page.as_mut_ptr())
         };
 
+        let code_page = code_page.make_exec().map_err(|e| e.to_string())?;
+        let callable_fn: fn() -> bool = unsafe { mem::transmute(code_page.as_ptr()) };
+
         // execute
         match callable_fn() {
             true => Ok(()),

From e8dc9ad8c91d1e06fc72e2e6cd1aacf968b58f79 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Wed, 11 Sep 2019 18:23:13 +0200
Subject: [PATCH 2708/3084] Fix x86/br-i128 test case to use basic blocks.

---
 cranelift/filetests/filetests/isa/x86/br-i128.clif | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/cranelift/filetests/filetests/isa/x86/br-i128.clif b/cranelift/filetests/filetests/isa/x86/br-i128.clif
index a09db3f41b..e3ac6a9d86 100644
--- a/cranelift/filetests/filetests/isa/x86/br-i128.clif
+++ b/cranelift/filetests/filetests/isa/x86/br-i128.clif
@@ -3,22 +3,28 @@ target x86_64
 
 function u0:0(i128) -> i8 fast {
 ebb0(v0: i128):
-    brz v0, ebb1
+    brz v0, ebb2
+    jump ebb1
+
+ebb1:
     v1 = iconst.i8 0
     return v1
 
-ebb1:
+ebb2:
     v2 = iconst.i8 1
     return v2
 }
 
 function u0:1(i128) -> i8 fast {
 ebb0(v0: i128):
-    brnz v0, ebb1
+    brnz v0, ebb2
+    jump ebb1
+
+ebb1:
     v1 = iconst.i8 0
     return v1
 
-ebb1:
+ebb2:
     v2 = iconst.i8 1
     return v2
 }

From 92a01c816d2676bce24bef63b21acf2e862af4e5 Mon Sep 17 00:00:00 2001
From: julian-seward1 
Date: Thu, 12 Sep 2019 11:09:35 +0200
Subject: [PATCH 2709/3084] Minor speedup tuning for SecondaryMap (#1020)

The `SecondaryMap` abstraction -- basically, resize-on-demand arrays with a
default value -- is very hot in Cranelift.  This small patch is the result of
many profiling runs.  It makes two changes:

* `fn index_mut` is changed to be `#[inline(always)]`, based on profile data.

* `fn index` and `fn index_mut` call `self.elems.resize()` directly, rather
  than via `self.resize()`.  The point of this is not to improve performance.
  Rather, it ensures that the public functions for `SecondaryMap` do not call
  each other.  When public interface functions call each other, it becomes
  difficult to interpret profiling results, because it's harder to see what
  fraction of costs for `SecondaryMap` as a whole come from outside the
  module, and what fraction is the result of "internal" calls to the external
  interface.

The overall result, for wasm_lua_binarytrees, is a 1.4% reduction in
instruction count for the compiler, and a 2.2% reduction in loads/stores.
---
 cranelift/entity/src/map.rs | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/cranelift/entity/src/map.rs b/cranelift/entity/src/map.rs
index f46c307426..77fe38ce92 100644
--- a/cranelift/entity/src/map.rs
+++ b/cranelift/entity/src/map.rs
@@ -83,16 +83,19 @@ where
     }
 
     /// Get the element at `k` if it exists.
+    #[inline(always)]
     pub fn get(&self, k: K) -> Option<&V> {
         self.elems.get(k.index())
     }
 
     /// Is this map completely empty?
+    #[inline(always)]
     pub fn is_empty(&self) -> bool {
         self.elems.is_empty()
     }
 
     /// Remove all entries from this map.
+    #[inline(always)]
     pub fn clear(&mut self) {
         self.elems.clear()
     }
@@ -123,7 +126,6 @@ where
     }
 
     /// Resize the map to have `n` entries by adding default entries as needed.
-    #[inline]
     pub fn resize(&mut self, n: usize) {
         self.elems.resize(n, self.default.clone());
     }
@@ -139,8 +141,9 @@ where
 {
     type Output = V;
 
+    #[inline(always)]
     fn index(&self, k: K) -> &V {
-        self.get(k).unwrap_or(&self.default)
+        self.elems.get(k.index()).unwrap_or(&self.default)
     }
 }
 
@@ -152,11 +155,11 @@ where
     K: EntityRef,
     V: Clone,
 {
-    #[inline]
+    #[inline(always)]
     fn index_mut(&mut self, k: K) -> &mut V {
         let i = k.index();
         if i >= self.elems.len() {
-            self.resize(i + 1);
+            self.elems.resize(i + 1, self.default.clone());
         }
         &mut self.elems[i]
     }

From cadd0ac65523a93ca39f84ae1b61037c21e840e8 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 12 Sep 2019 09:02:22 -0700
Subject: [PATCH 2710/3084] Remove unused import when cross-compiling, fixes
 #1022

---
 cranelift/native/src/lib.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/native/src/lib.rs b/cranelift/native/src/lib.rs
index cfa27cef86..9ad873e166 100644
--- a/cranelift/native/src/lib.rs
+++ b/cranelift/native/src/lib.rs
@@ -26,7 +26,6 @@
 #![no_std]
 
 use cranelift_codegen::isa;
-use cranelift_codegen::settings::Configurable;
 use target_lexicon::Triple;
 
 #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
@@ -50,6 +49,7 @@ pub fn builder() -> Result {
 
 #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
 fn parse_x86_cpuid(isa_builder: &mut isa::Builder) -> Result<(), &'static str> {
+    use cranelift_codegen::settings::Configurable;
     let cpuid = CpuId::new();
 
     if let Some(info) = cpuid.get_feature_info() {

From 3418fb6e18780fd1f885e047085e73568ed89184 Mon Sep 17 00:00:00 2001
From: Ujjwal Sharma 
Date: Fri, 13 Sep 2019 20:57:50 +0530
Subject: [PATCH 2711/3084] =?UTF-8?q?[codegen]=20reintroduce=20support=20f?=
 =?UTF-8?q?or=20carry=20and=20borrow=20instructions=20in=20RI=E2=80=A6=20(?=
 =?UTF-8?q?#1005)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Reintroduce support for iadd carry variants and isub borrow variants for
RISC ISAs which had been removed in
https://github.com/CraneStation/cranelift/pull/961 and
https://github.com/CraneStation/cranelift/pull/962 because of the lack
of a proper flags register in RISC architectures.
---
 cranelift/codegen/meta/src/isa/arm32/mod.rs   |   6 +-
 cranelift/codegen/meta/src/isa/arm64/mod.rs   |   4 +-
 cranelift/codegen/meta/src/isa/riscv/mod.rs   |   7 +-
 .../codegen/meta/src/isa/x86/encodings.rs     |  24 +--
 .../codegen/meta/src/isa/x86/legalize.rs      |   2 +-
 cranelift/codegen/meta/src/isa/x86/mod.rs     |   4 +-
 .../codegen/meta/src/shared/instructions.rs   | 138 ++++++++++++++-
 cranelift/codegen/meta/src/shared/legalize.rs | 162 +++++++++++++++---
 cranelift/codegen/src/ir/dfg.rs               |   4 +-
 cranelift/codegen/src/legalizer/heap.rs       |   2 +-
 .../{expand-i32.clif.bak => expand-i32.clif}  |   3 -
 ...egalize-i64.clif.bak => legalize-i64.clif} |   3 -
 .../filetests/filetests/isa/x86/binary32.clif |  12 +-
 .../filetests/isa/x86/legalize-i64.clif       |   8 +-
 .../filetests/filetests/isa/x86/run-i64.clif  |   4 +-
 .../parser/instruction_encoding.clif          |   4 +-
 .../filetests/filetests/parser/ternary.clif   |  24 +--
 17 files changed, 326 insertions(+), 85 deletions(-)
 rename cranelift/filetests/filetests/isa/riscv/{expand-i32.clif.bak => expand-i32.clif} (87%)
 rename cranelift/filetests/filetests/isa/riscv/{legalize-i64.clif.bak => legalize-i64.clif} (94%)

diff --git a/cranelift/codegen/meta/src/isa/arm32/mod.rs b/cranelift/codegen/meta/src/isa/arm32/mod.rs
index fb41db2530..1f3c05ad60 100644
--- a/cranelift/codegen/meta/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm32/mod.rs
@@ -66,9 +66,9 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let mut t32 = CpuMode::new("T32");
 
     // TODO refine these.
-    let narrow = shared_defs.transform_groups.by_name("narrow");
-    a32.legalize_default(narrow);
-    t32.legalize_default(narrow);
+    let narrow_flags = shared_defs.transform_groups.by_name("narrow_flags");
+    a32.legalize_default(narrow_flags);
+    t32.legalize_default(narrow_flags);
 
     let cpu_modes = vec![a32, t32];
 
diff --git a/cranelift/codegen/meta/src/isa/arm64/mod.rs b/cranelift/codegen/meta/src/isa/arm64/mod.rs
index 697e9513c5..15bfb736cc 100644
--- a/cranelift/codegen/meta/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm64/mod.rs
@@ -60,8 +60,8 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let mut a64 = CpuMode::new("A64");
 
     // TODO refine these.
-    let narrow = shared_defs.transform_groups.by_name("narrow");
-    a64.legalize_default(narrow);
+    let narrow_flags = shared_defs.transform_groups.by_name("narrow_flags");
+    a64.legalize_default(narrow_flags);
 
     let cpu_modes = vec![a64];
 
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index a2683b5a8b..14eb3267f6 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -102,15 +102,16 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let mut rv_64 = CpuMode::new("RV64");
 
     let expand = shared_defs.transform_groups.by_name("expand");
-    let narrow = shared_defs.transform_groups.by_name("narrow");
+    let narrow_no_flags = shared_defs.transform_groups.by_name("narrow_no_flags");
+
     rv_32.legalize_monomorphic(expand);
-    rv_32.legalize_default(narrow);
+    rv_32.legalize_default(narrow_no_flags);
     rv_32.legalize_type(I32, expand);
     rv_32.legalize_type(F32, expand);
     rv_32.legalize_type(F64, expand);
 
     rv_64.legalize_monomorphic(expand);
-    rv_64.legalize_default(narrow);
+    rv_64.legalize_default(narrow_no_flags);
     rv_64.legalize_type(I32, expand);
     rv_64.legalize_type(I64, expand);
     rv_64.legalize_type(F32, expand);
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index d773c2c662..f16ca93717 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -384,9 +384,9 @@ pub(crate) fn define(
     let func_addr = shared.by_name("func_addr");
     let get_pinned_reg = shared.by_name("get_pinned_reg");
     let iadd = shared.by_name("iadd");
-    let iadd_cout = shared.by_name("iadd_cout");
-    let iadd_cin = shared.by_name("iadd_cin");
-    let iadd_carry = shared.by_name("iadd_carry");
+    let iadd_ifcout = shared.by_name("iadd_ifcout");
+    let iadd_ifcin = shared.by_name("iadd_ifcin");
+    let iadd_ifcarry = shared.by_name("iadd_ifcarry");
     let iadd_imm = shared.by_name("iadd_imm");
     let icmp = shared.by_name("icmp");
     let icmp_imm = shared.by_name("icmp_imm");
@@ -407,9 +407,9 @@ pub(crate) fn define(
     let istore8 = shared.by_name("istore8");
     let istore8_complex = shared.by_name("istore8_complex");
     let isub = shared.by_name("isub");
-    let isub_bout = shared.by_name("isub_bout");
-    let isub_bin = shared.by_name("isub_bin");
-    let isub_borrow = shared.by_name("isub_borrow");
+    let isub_ifbout = shared.by_name("isub_ifbout");
+    let isub_ifbin = shared.by_name("isub_ifbin");
+    let isub_ifborrow = shared.by_name("isub_ifborrow");
     let jump = shared.by_name("jump");
     let jump_table_base = shared.by_name("jump_table_base");
     let jump_table_entry = shared.by_name("jump_table_entry");
@@ -647,14 +647,14 @@ pub(crate) fn define(
     );
 
     e.enc_i32_i64(iadd, rec_rr.opcodes(vec![0x01]));
-    e.enc_i32_i64(iadd_cout, rec_rout.opcodes(vec![0x01]));
-    e.enc_i32_i64(iadd_cin, rec_rin.opcodes(vec![0x11]));
-    e.enc_i32_i64(iadd_carry, rec_rio.opcodes(vec![0x11]));
+    e.enc_i32_i64(iadd_ifcout, rec_rout.opcodes(vec![0x01]));
+    e.enc_i32_i64(iadd_ifcin, rec_rin.opcodes(vec![0x11]));
+    e.enc_i32_i64(iadd_ifcarry, rec_rio.opcodes(vec![0x11]));
 
     e.enc_i32_i64(isub, rec_rr.opcodes(vec![0x29]));
-    e.enc_i32_i64(isub_bout, rec_rout.opcodes(vec![0x29]));
-    e.enc_i32_i64(isub_bin, rec_rin.opcodes(vec![0x19]));
-    e.enc_i32_i64(isub_borrow, rec_rio.opcodes(vec![0x19]));
+    e.enc_i32_i64(isub_ifbout, rec_rout.opcodes(vec![0x29]));
+    e.enc_i32_i64(isub_ifbin, rec_rin.opcodes(vec![0x19]));
+    e.enc_i32_i64(isub_ifborrow, rec_rio.opcodes(vec![0x19]));
 
     e.enc_i32_i64(band, rec_rr.opcodes(vec![0x21]));
     e.enc_i32_i64(bor, rec_rr.opcodes(vec![0x09]));
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 555a93f9cb..dfd7f84334 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -305,7 +305,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     Use x86-specific instructions if needed."#,
     )
     .isa("x86")
-    .chain_with(shared.transform_groups.by_name("narrow").id);
+    .chain_with(shared.transform_groups.by_name("narrow_flags").id);
 
     // SIMD
     let uimm8_zero = Literal::constant(&imm.uimm8, 0x00);
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index 10baf256a9..6ced509396 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -29,13 +29,13 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let mut x86_32 = CpuMode::new("I32");
 
     let expand_flags = shared_defs.transform_groups.by_name("expand_flags");
-    let narrow = shared_defs.transform_groups.by_name("narrow");
+    let narrow_flags = shared_defs.transform_groups.by_name("narrow_flags");
     let widen = shared_defs.transform_groups.by_name("widen");
     let x86_narrow = shared_defs.transform_groups.by_name("x86_narrow");
     let x86_expand = shared_defs.transform_groups.by_name("x86_expand");
 
     x86_32.legalize_monomorphic(expand_flags);
-    x86_32.legalize_default(narrow);
+    x86_32.legalize_default(narrow_flags);
     x86_32.legalize_type(B1, expand_flags);
     x86_32.legalize_type(I8, widen);
     x86_32.legalize_type(I16, widen);
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 3ebebfe183..bae1327b60 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1879,10 +1879,16 @@ pub(crate) fn define(
     let a = &operand("a", iB);
     let x = &operand("x", iB);
     let y = &operand("y", iB);
-    let c_in = &operand_doc("c_in", iflags, "Input carry flag");
-    let c_out = &operand_doc("c_out", iflags, "Output carry flag");
-    let b_in = &operand_doc("b_in", iflags, "Input borrow flag");
-    let b_out = &operand_doc("b_out", iflags, "Output borrow flag");
+
+    let c_in = &operand_doc("c_in", b1, "Input carry flag");
+    let c_out = &operand_doc("c_out", b1, "Output carry flag");
+    let b_in = &operand_doc("b_in", b1, "Input borrow flag");
+    let b_out = &operand_doc("b_out", b1, "Output borrow flag");
+
+    let c_if_in = &operand("c_in", iflags);
+    let c_if_out = &operand("c_out", iflags);
+    let b_if_in = &operand("b_in", iflags);
+    let b_if_out = &operand("b_out", iflags);
 
     ig.push(
         Inst::new(
@@ -1904,6 +1910,26 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    ig.push(
+        Inst::new(
+            "iadd_ifcin",
+            r#"
+        Add integers with carry in.
+
+        Same as `iadd` with an additional carry flag input. Computes:
+
+        ```text
+            a = x + y + c_{in} \pmod 2^B
+        ```
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y, c_if_in])
+        .operands_out(vec![a]),
+    );
+
     ig.push(
         Inst::new(
             "iadd_cout",
@@ -1925,6 +1951,27 @@ pub(crate) fn define(
         .operands_out(vec![a, c_out]),
     );
 
+    ig.push(
+        Inst::new(
+            "iadd_ifcout",
+            r#"
+        Add integers with carry out.
+
+        Same as `iadd` with an additional carry flag output.
+
+        ```text
+            a &= x + y \pmod 2^B \\
+            c_{out} &= x+y >= 2^B
+        ```
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a, c_if_out]),
+    );
+
     ig.push(
         Inst::new(
             "iadd_carry",
@@ -1946,6 +1993,27 @@ pub(crate) fn define(
         .operands_out(vec![a, c_out]),
     );
 
+    ig.push(
+        Inst::new(
+            "iadd_ifcarry",
+            r#"
+        Add integers with carry in and out.
+
+        Same as `iadd` with an additional carry flag input and output.
+
+        ```text
+            a &= x + y + c_{in} \pmod 2^B \\
+            c_{out} &= x + y + c_{in} >= 2^B
+        ```
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y, c_if_in])
+        .operands_out(vec![a, c_if_out]),
+    );
+
     ig.push(
         Inst::new(
             "isub_bin",
@@ -1966,6 +2034,26 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    ig.push(
+        Inst::new(
+            "isub_ifbin",
+            r#"
+        Subtract integers with borrow in.
+
+        Same as `isub` with an additional borrow flag input. Computes:
+
+        ```text
+            a = x - (y + b_{in}) \pmod 2^B
+        ```
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y, b_if_in])
+        .operands_out(vec![a]),
+    );
+
     ig.push(
         Inst::new(
             "isub_bout",
@@ -1987,6 +2075,27 @@ pub(crate) fn define(
         .operands_out(vec![a, b_out]),
     );
 
+    ig.push(
+        Inst::new(
+            "isub_ifbout",
+            r#"
+        Subtract integers with borrow out.
+
+        Same as `isub` with an additional borrow flag output.
+
+        ```text
+            a &= x - y \pmod 2^B \\
+            b_{out} &= x < y
+        ```
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a, b_if_out]),
+    );
+
     ig.push(
         Inst::new(
             "isub_borrow",
@@ -2008,6 +2117,27 @@ pub(crate) fn define(
         .operands_out(vec![a, b_out]),
     );
 
+    ig.push(
+        Inst::new(
+            "isub_ifborrow",
+            r#"
+        Subtract integers with borrow in and out.
+
+        Same as `isub` with an additional borrow flag input and output.
+
+        ```text
+            a &= x - (y + b_{in}) \pmod 2^B \\
+            b_{out} &= x < y + b_{in}
+        ```
+
+        Polymorphic over all scalar integer types, but does not support vector
+        types.
+        "#,
+        )
+        .operands_in(vec![x, y, b_if_in])
+        .operands_out(vec![a, b_if_out]),
+    );
+
     let bits = &TypeVar::new(
         "bits",
         "Any integer, float, or boolean scalar or vector type",
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 1c2c510460..f9cec3557e 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -69,6 +69,9 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     let iadd = insts.by_name("iadd");
     let iadd_cin = insts.by_name("iadd_cin");
     let iadd_cout = insts.by_name("iadd_cout");
+    let iadd_carry = insts.by_name("iadd_carry");
+    let iadd_ifcin = insts.by_name("iadd_ifcin");
+    let iadd_ifcout = insts.by_name("iadd_ifcout");
     let iadd_imm = insts.by_name("iadd_imm");
     let icmp = insts.by_name("icmp");
     let icmp_imm = insts.by_name("icmp_imm");
@@ -88,6 +91,9 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     let isub = insts.by_name("isub");
     let isub_bin = insts.by_name("isub_bin");
     let isub_bout = insts.by_name("isub_bout");
+    let isub_borrow = insts.by_name("isub_borrow");
+    let isub_ifbin = insts.by_name("isub_ifbin");
+    let isub_ifbout = insts.by_name("isub_ifbout");
     let load = insts.by_name("load");
     let popcnt = insts.by_name("popcnt");
     let rotl = insts.by_name("rotl");
@@ -154,11 +160,15 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     let b2 = var("b2");
     let b3 = var("b3");
     let b4 = var("b4");
+    let b_in = var("b_in");
+    let b_int = var("b_int");
     let c = var("c");
     let c1 = var("c1");
     let c2 = var("c2");
     let c3 = var("c3");
     let c4 = var("c4");
+    let c_in = var("c_in");
+    let c_int = var("c_int");
     let d = var("d");
     let d1 = var("d1");
     let d2 = var("d2");
@@ -192,28 +202,6 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     // embedded as part of arguments), so use a custom legalization for now.
     narrow.custom_legalize(iconst, "narrow_iconst");
 
-    narrow.legalize(
-        def!(a = iadd(x, y)),
-        vec![
-            def!((xl, xh) = isplit(x)),
-            def!((yl, yh) = isplit(y)),
-            def!((al, c) = iadd_cout(xl, yl)),
-            def!(ah = iadd_cin(xh, yh, c)),
-            def!(a = iconcat(al, ah)),
-        ],
-    );
-
-    narrow.legalize(
-        def!(a = isub(x, y)),
-        vec![
-            def!((xl, xh) = isplit(x)),
-            def!((yl, yh) = isplit(y)),
-            def!((al, b) = isub_bout(xl, yl)),
-            def!(ah = isub_bin(xh, yh, b)),
-            def!(a = iconcat(al, ah)),
-        ],
-    );
-
     for &bin_op in &[band, bor, bxor, band_not, bor_not, bxor_not] {
         narrow.legalize(
             def!(a = bin_op(x, y)),
@@ -502,6 +490,58 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
         }
     }
 
+    // Expand integer operations with carry for RISC architectures that don't have
+    // the flags.
+    let intcc_ult = Literal::enumerator_for(&imm.intcc, "ult");
+    expand.legalize(
+        def!((a, c) = iadd_cout(x, y)),
+        vec![def!(a = iadd(x, y)), def!(c = icmp(intcc_ult, a, x))],
+    );
+
+    let intcc_ugt = Literal::enumerator_for(&imm.intcc, "ugt");
+    expand.legalize(
+        def!((a, b) = isub_bout(x, y)),
+        vec![def!(a = isub(x, y)), def!(b = icmp(intcc_ugt, a, x))],
+    );
+
+    expand.legalize(
+        def!(a = iadd_cin(x, y, c)),
+        vec![
+            def!(a1 = iadd(x, y)),
+            def!(c_int = bint(c)),
+            def!(a = iadd(a1, c_int)),
+        ],
+    );
+
+    expand.legalize(
+        def!(a = isub_bin(x, y, b)),
+        vec![
+            def!(a1 = isub(x, y)),
+            def!(b_int = bint(b)),
+            def!(a = isub(a1, b_int)),
+        ],
+    );
+
+    expand.legalize(
+        def!((a, c) = iadd_carry(x, y, c_in)),
+        vec![
+            def!((a1, c1) = iadd_cout(x, y)),
+            def!(c_int = bint(c_in)),
+            def!((a, c2) = iadd_cout(a1, c_int)),
+            def!(c = bor(c1, c2)),
+        ],
+    );
+
+    expand.legalize(
+        def!((a, b) = isub_borrow(x, y, b_in)),
+        vec![
+            def!((a1, b1) = isub_bout(x, y)),
+            def!(b_int = bint(b_in)),
+            def!((a, b2) = isub_bout(a1, b_int)),
+            def!(b = bor(b1, b2)),
+        ],
+    );
+
     // Expansions for fcvt_from_{u,s}int for smaller integer types.
     // These use expand and not widen because the controlling type variable for
     // these instructions are f32/f64, which are legalized as part of the expand
@@ -758,7 +798,7 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
 
     let mut groups = TransformGroups::new();
 
-    narrow.build_and_add_to(&mut groups);
+    let narrow_id = narrow.build_and_add_to(&mut groups);
     let expand_id = expand.build_and_add_to(&mut groups);
 
     // Expansions using CPU flags.
@@ -796,6 +836,82 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
 
     expand_flags.build_and_add_to(&mut groups);
 
+    // Narrow legalizations using CPU flags.
+    let mut narrow_flags = TransformGroupBuilder::new(
+        "narrow_flags",
+        r#"
+        Narrow instructions for architectures with flags.
+
+        Narrow some instructions using CPU flags, then fall back to the normal
+        legalizations. Not all architectures support CPU flags, so these
+        patterns are kept separate.
+    "#,
+    )
+    .chain_with(narrow_id);
+
+    narrow_flags.legalize(
+        def!(a = iadd(x, y)),
+        vec![
+            def!((xl, xh) = isplit(x)),
+            def!((yl, yh) = isplit(y)),
+            def!((al, c) = iadd_ifcout(xl, yl)),
+            def!(ah = iadd_ifcin(xh, yh, c)),
+            def!(a = iconcat(al, ah)),
+        ],
+    );
+
+    narrow_flags.legalize(
+        def!(a = isub(x, y)),
+        vec![
+            def!((xl, xh) = isplit(x)),
+            def!((yl, yh) = isplit(y)),
+            def!((al, b) = isub_ifbout(xl, yl)),
+            def!(ah = isub_ifbin(xh, yh, b)),
+            def!(a = iconcat(al, ah)),
+        ],
+    );
+
+    narrow_flags.build_and_add_to(&mut groups);
+
+    // TODO(ryzokuken): figure out a way to legalize iadd_c* to iadd_ifc* (and
+    // similarly isub_b* to isub_ifb*) on expand_flags so that this isn't required.
+    // Narrow legalizations for ISAs that don't have CPU flags.
+    let mut narrow_no_flags = TransformGroupBuilder::new(
+        "narrow_no_flags",
+        r#"
+        Narrow instructions for architectures without flags.
+
+        Narrow some instructions avoiding the use of CPU flags, then fall back
+        to the normal legalizations. Not all architectures support CPU flags,
+        so these patterns are kept separate.
+    "#,
+    )
+    .chain_with(narrow_id);
+
+    narrow_no_flags.legalize(
+        def!(a = iadd(x, y)),
+        vec![
+            def!((xl, xh) = isplit(x)),
+            def!((yl, yh) = isplit(y)),
+            def!((al, c) = iadd_cout(xl, yl)),
+            def!(ah = iadd_cin(xh, yh, c)),
+            def!(a = iconcat(al, ah)),
+        ],
+    );
+
+    narrow_no_flags.legalize(
+        def!(a = isub(x, y)),
+        vec![
+            def!((xl, xh) = isplit(x)),
+            def!((yl, yh) = isplit(y)),
+            def!((al, b) = isub_bout(xl, yl)),
+            def!(ah = isub_bin(xh, yh, b)),
+            def!(a = iconcat(al, ah)),
+        ],
+    );
+
+    narrow_no_flags.build_and_add_to(&mut groups);
+
     // TODO The order of declarations unfortunately matters to be compatible with the Python code.
     // When it's all migrated, we can put this next to the narrow/expand build_and_add_to calls
     // above.
diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index 24eeff4eec..f4e2875952 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -1253,7 +1253,7 @@ mod tests {
         assert_eq!(pos.func.dfg.resolve_aliases(v1), v1);
 
         let arg0 = pos.func.dfg.append_ebb_param(ebb0, types::I32);
-        let (s, c) = pos.ins().iadd_cout(v1, arg0);
+        let (s, c) = pos.ins().iadd_ifcout(v1, arg0);
         let iadd = match pos.func.dfg.value_def(s) {
             ValueDef::Result(i, 0) => i,
             _ => panic!(),
@@ -1263,7 +1263,7 @@ mod tests {
         pos.func.dfg.clear_results(iadd);
         pos.func.dfg.attach_result(iadd, s);
 
-        // Replace `iadd_cout` with a normal `iadd` and an `ifcmp`.
+        // Replace `iadd_ifcout` with a normal `iadd` and an `ifcmp`.
         pos.func.dfg.replace(iadd).iadd(v1, arg0);
         let c2 = pos.ins().ifcmp(s, v1);
         pos.func.dfg.change_to_alias(c, c2);
diff --git a/cranelift/codegen/src/legalizer/heap.rs b/cranelift/codegen/src/legalizer/heap.rs
index dfada10dc5..332553e803 100644
--- a/cranelift/codegen/src/legalizer/heap.rs
+++ b/cranelift/codegen/src/legalizer/heap.rs
@@ -82,7 +82,7 @@ fn dynamic_addr(
     } else {
         // We need an overflow check for the adjusted offset.
         let access_size_val = pos.ins().iconst(offset_ty, access_size as i64);
-        let (adj_offset, overflow) = pos.ins().iadd_cout(offset, access_size_val);
+        let (adj_offset, overflow) = pos.ins().iadd_ifcout(offset, access_size_val);
         pos.ins().trapnz(overflow, ir::TrapCode::HeapOutOfBounds);
         oob = pos
             .ins()
diff --git a/cranelift/filetests/filetests/isa/riscv/expand-i32.clif.bak b/cranelift/filetests/filetests/isa/riscv/expand-i32.clif
similarity index 87%
rename from cranelift/filetests/filetests/isa/riscv/expand-i32.clif.bak
rename to cranelift/filetests/filetests/isa/riscv/expand-i32.clif
index cc2a9d726c..eb63d7cdcd 100644
--- a/cranelift/filetests/filetests/isa/riscv/expand-i32.clif.bak
+++ b/cranelift/filetests/filetests/isa/riscv/expand-i32.clif
@@ -1,6 +1,3 @@
-; TODO(ryzokuken): figure out a better legalization strategy for platforms that
-; platforms that don't have flags.
-
 ; Test the legalization of i32 instructions that don't have RISC-V versions.
 test legalizer
 
diff --git a/cranelift/filetests/filetests/isa/riscv/legalize-i64.clif.bak b/cranelift/filetests/filetests/isa/riscv/legalize-i64.clif
similarity index 94%
rename from cranelift/filetests/filetests/isa/riscv/legalize-i64.clif.bak
rename to cranelift/filetests/filetests/isa/riscv/legalize-i64.clif
index 366472e7d3..d043337a21 100644
--- a/cranelift/filetests/filetests/isa/riscv/legalize-i64.clif.bak
+++ b/cranelift/filetests/filetests/isa/riscv/legalize-i64.clif
@@ -1,6 +1,3 @@
-; TODO(ryzokuken): figure out a better legalization strategy for platforms that
-; platforms that don't have flags.
-
 ; Test the legalization of i64 arithmetic instructions.
 test legalizer
 target riscv32 supports_m=1
diff --git a/cranelift/filetests/filetests/isa/x86/binary32.clif b/cranelift/filetests/filetests/isa/x86/binary32.clif
index 6f2ddaa84e..aa50cf5335 100644
--- a/cranelift/filetests/filetests/isa/x86/binary32.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary32.clif
@@ -471,19 +471,19 @@ ebb0:
 
     ; Carry Addition
     ; asm: addl %esi, %ecx
-    [-,%rcx,%rflags] v701, v702 = iadd_cout v1, v2  ; bin: 01 f1
+    [-,%rcx,%rflags] v701, v702 = iadd_ifcout v1, v2  ; bin: 01 f1
     ; asm: adcl %esi, %ecx
-    [-,%rcx] v703 = iadd_cin v1, v2, v702           ; bin: 11 f1
+    [-,%rcx] v703 = iadd_ifcin v1, v2, v702           ; bin: 11 f1
     ; asm: adcl %esi, %ecx
-    [-,%rcx,%rflags] v704, v705 = iadd_carry v1, v2, v702 ; bin: 11 f1
+    [-,%rcx,%rflags] v704, v705 = iadd_ifcarry v1, v2, v702 ; bin: 11 f1
 
     ; Borrow Subtraction
     ; asm: subl %esi, %ecx
-    [-,%rcx,%rflags] v706, v707 = isub_bout v1, v2         ; bin: 29 f1
+    [-,%rcx,%rflags] v706, v707 = isub_ifbout v1, v2         ; bin: 29 f1
     ; asm: sbbl %esi, %ecx
-    [-,%rcx] v708 = isub_bin v1, v2, v707                  ; bin: 19 f1
+    [-,%rcx] v708 = isub_ifbin v1, v2, v707                  ; bin: 19 f1
     ; asm: sbbl %esi, %ecx
-    [-,%rcx,%rflags] v709, v710 = isub_borrow v1, v2, v707 ; bin: 19 f1
+    [-,%rcx,%rflags] v709, v710 = isub_ifborrow v1, v2, v707 ; bin: 19 f1
 
     ; asm: testl %ecx, %ecx
     ; asm: je ebb1
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-i64.clif b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
index 66cdbd245d..8ae88f736e 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
@@ -9,8 +9,8 @@ ebb0(v1: i64, v2: i64):
     v10 = iadd v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
-    ; nextln: $(v10_lsb=$V), $(carry=$V) = iadd_cout $v1_lsb, $v2_lsb
-    ; nextln: $(v10_msb=$V) = iadd_cin $v1_msb, $v2_msb, $carry
+    ; nextln: $(v10_lsb=$V), $(carry=$V) = iadd_ifcout $v1_lsb, $v2_lsb
+    ; nextln: $(v10_msb=$V) = iadd_ifcin $v1_msb, $v2_msb, $carry
     ; nextln: v10 = iconcat $v10_lsb, $v10_msb
     return v10
 }
@@ -20,8 +20,8 @@ ebb0(v1: i64, v2: i64):
     v10 = isub v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
-    ; nextln: $(v10_lsb=$V), $(borrow=$V) = isub_bout $v1_lsb, $v2_lsb
-    ; nextln: $(v10_msb=$V) = isub_bin $v1_msb, $v2_msb, $borrow
+    ; nextln: $(v10_lsb=$V), $(borrow=$V) = isub_ifbout $v1_lsb, $v2_lsb
+    ; nextln: $(v10_msb=$V) = isub_ifbin $v1_msb, $v2_msb, $borrow
     ; nextln: v10 = iconcat $v10_lsb, $v10_msb
     return v10
 }
diff --git a/cranelift/filetests/filetests/isa/x86/run-i64.clif b/cranelift/filetests/filetests/isa/x86/run-i64.clif
index fc1b155c10..6fae71966e 100644
--- a/cranelift/filetests/filetests/isa/x86/run-i64.clif
+++ b/cranelift/filetests/filetests/isa/x86/run-i64.clif
@@ -5,13 +5,13 @@ target i686 haswell
 function %iadd(i64, i64) -> i64 {
 ebb0(v1: i64, v2: i64):
     v10 = iadd v1, v2
-    ; check: iadd_cout
+    ; check: iadd_ifcout
     return v10
 }
 
 function %isub(i64, i64) -> i64 {
 ebb0(v1: i64, v2: i64):
     v10 = isub v1, v2
-    ; check: isub_bout
+    ; check: isub_ifbout
     return v10
 }
diff --git a/cranelift/filetests/filetests/parser/instruction_encoding.clif b/cranelift/filetests/filetests/parser/instruction_encoding.clif
index 9a405030a6..5f7ae26af3 100644
--- a/cranelift/filetests/filetests/parser/instruction_encoding.clif
+++ b/cranelift/filetests/filetests/parser/instruction_encoding.clif
@@ -8,7 +8,7 @@ function %foo(i32, i32) {
 ebb1(v0: i32 [%x8], v1: i32):
     [-,-] v2 = iadd v0, v1
     [-] trap heap_oob
-    [R#1234, %x5, %x11] v6, v7 = iadd_cout v2, v0
+    [R#1234, %x5, %x11] v6, v7 = iadd_ifcout v2, v0
     [Rshamt#beef, %x25] v8 = ishl_imm v6, 2
 @55 v9 = iadd v8, v7
 @a5 [Iret#5] return v0, v8
@@ -17,7 +17,7 @@ ebb1(v0: i32 [%x8], v1: i32):
 ; nextln: ebb1(v0: i32 [%x8], v1: i32):
 ; nextln:     [-,-]$WS v2 = iadd v0, v1
 ; nextln:     [-]$WS trap heap_oob
-; nextln:     [R#1234,%x5,%x11]$WS v6, v7 = iadd_cout v2, v0
+; nextln:     [R#1234,%x5,%x11]$WS v6, v7 = iadd_ifcout v2, v0
 ; nextln:     [Rshamt#beef,%x25]$WS v8 = ishl_imm v6, 2
 ; nextln:     @0055 [-,-]$WS v9 = iadd v8, v7
 ; nextln:     @00a5 [Iret#05]$WS return v0, v8
diff --git a/cranelift/filetests/filetests/parser/ternary.clif b/cranelift/filetests/filetests/parser/ternary.clif
index 75f09bb223..d17e5592c6 100644
--- a/cranelift/filetests/filetests/parser/ternary.clif
+++ b/cranelift/filetests/filetests/parser/ternary.clif
@@ -3,22 +3,22 @@ test verifier
 
 function %add_i96(i32, i32, i32, i32, i32, i32) -> i32, i32, i32 {
 ebb1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32):
-    v10, v11 = iadd_cout v1, v4
-    ;check: v10, v11 = iadd_cout v1, v4
-    v20, v21 = iadd_carry v2, v5, v11
-    ; check: v20, v21 = iadd_carry v2, v5, v11
-    v30 = iadd_cin v3, v6, v21
-    ; check: v30 = iadd_cin v3, v6, v21
+    v10, v11 = iadd_ifcout v1, v4
+    ;check: v10, v11 = iadd_ifcout v1, v4
+    v20, v21 = iadd_ifcarry v2, v5, v11
+    ; check: v20, v21 = iadd_ifcarry v2, v5, v11
+    v30 = iadd_ifcin v3, v6, v21
+    ; check: v30 = iadd_ifcin v3, v6, v21
     return v10, v20, v30
 }
 
 function %sub_i96(i32, i32, i32, i32, i32, i32) -> i32, i32, i32 {
 ebb1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32):
-    v10, v11 = isub_bout v1, v4
-    ;check: v10, v11 = isub_bout v1, v4
-    v20, v21 = isub_borrow v2, v5, v11
-    ; check: v20, v21 = isub_borrow v2, v5, v11
-    v30 = isub_bin v3, v6, v21
-    ; check: v30 = isub_bin v3, v6, v21
+    v10, v11 = isub_ifbout v1, v4
+    ;check: v10, v11 = isub_ifbout v1, v4
+    v20, v21 = isub_ifborrow v2, v5, v11
+    ; check: v20, v21 = isub_ifborrow v2, v5, v11
+    v30 = isub_ifbin v3, v6, v21
+    ; check: v30 = isub_ifbin v3, v6, v21
     return v10, v20, v30
 }

From 4aef679243a01d07dac57b81ae8da9507f40447d Mon Sep 17 00:00:00 2001
From: Rett Berg 
Date: Fri, 13 Sep 2019 21:52:57 -0700
Subject: [PATCH 2712/3084] fix reflink in CONTRIBUTING.md

---
 cranelift/CONTRIBUTING.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
index 4d97970a6c..97ce756eb9 100644
--- a/cranelift/CONTRIBUTING.md
+++ b/cranelift/CONTRIBUTING.md
@@ -50,7 +50,7 @@ things set in stone yet.
 
 We abide by our [Code of Conduct] and ask that you do as well.
 
-[Code of Conduct](CODE_OF_CONDUCT.md)
+[Code of Conduct]: CODE_OF_CONDUCT.md
 
 ## Coding Guidelines
 

From a51606bfeb6cccab86219b722d90b1dc4a7e74b0 Mon Sep 17 00:00:00 2001
From: Wander Lairson Costa 
Date: Sat, 14 Sep 2019 20:12:56 -0300
Subject: [PATCH 2713/3084] [frontend] Add message for pristine assertion
 violation

After we add any instruction to an EBB, we can't add EBB parameters
anymore. There is an assertion in `append_ebb_param` to detect this
error, but the backtrace generated doesn't give any hint what happened.

We add an error message to the assertion so the user can understand what
is wrong.

fixes #1003
---
 cranelift/frontend/src/frontend.rs | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 064ab7053c..37ac4b95dc 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -523,7 +523,10 @@ impl FunctionBuilder {
     /// **Note:** this function has to be called at the creation of the `Ebb` before adding
     /// instructions to it, otherwise this could interfere with SSA construction.
     pub fn append_ebb_param(&mut self, ebb: Ebb, ty: Type) -> Value {
-        debug_assert!(self.func_ctx.ebbs[ebb].pristine);
+        debug_assert!(
+            self.func_ctx.ebbs[ebb].pristine,
+            "You can't add EBB parameters after adding any instruction"
+        );
         debug_assert_eq!(
             self.func_ctx.ebbs[ebb].user_param_count,
             self.func.dfg.num_ebb_params(ebb)

From b95508c51a334e7c5cf0dff0bb9dd4d55fb098a0 Mon Sep 17 00:00:00 2001
From: julian-seward1 
Date: Mon, 16 Sep 2019 15:01:07 +0200
Subject: [PATCH 2714/3084] legalizer/split.rs: split_ebb_params: avoid
 pointless Vec allocations. (#1025)

This function is responsible for 2.2% of all heap allocation (calls) in CL.
This change avoids all of them in the (presumably) common case where none of
the parameters require splitting.  It also slightly reduces the compiler's
instruction count.
---
 cranelift/codegen/src/legalizer/split.rs | 28 +++++++++++++++---------
 1 file changed, 18 insertions(+), 10 deletions(-)

diff --git a/cranelift/codegen/src/legalizer/split.rs b/cranelift/codegen/src/legalizer/split.rs
index 9f689922f1..5e55419fad 100644
--- a/cranelift/codegen/src/legalizer/split.rs
+++ b/cranelift/codegen/src/legalizer/split.rs
@@ -131,19 +131,27 @@ fn split_any(
 }
 
 pub fn split_ebb_params(func: &mut ir::Function, cfg: &ControlFlowGraph, ebb: Ebb) {
-    let mut repairs = Vec::new();
     let pos = &mut FuncCursor::new(func).at_top(ebb);
+    let ebb_params = pos.func.dfg.ebb_params(ebb);
 
-    for (num, ebb_param) in pos
-        .func
-        .dfg
-        .ebb_params(ebb)
-        .to_vec()
-        .into_iter()
-        .enumerate()
+    // Add further splittable types here.
+    fn type_requires_splitting(ty: Type) -> bool {
+        ty == ir::types::I128
+    }
+
+    // A shortcut.  If none of the param types require splitting, exit now.  This helps because
+    // the loop below necessarily has to copy the ebb params into a new vector, so it's better to
+    // avoid doing so when possible.
+    if !ebb_params
+        .iter()
+        .any(|ebb_param| type_requires_splitting(pos.func.dfg.value_type(*ebb_param)))
     {
-        let ty = pos.func.dfg.value_type(ebb_param);
-        if ty != ir::types::I128 {
+        return;
+    }
+
+    let mut repairs = Vec::new();
+    for (num, ebb_param) in ebb_params.to_vec().into_iter().enumerate() {
+        if !type_requires_splitting(pos.func.dfg.value_type(ebb_param)) {
             continue;
         }
 

From 99380fad1af9c029ae2b77c083eb123bc2dc429b Mon Sep 17 00:00:00 2001
From: Andy Wortman 
Date: Mon, 16 Sep 2019 14:35:55 +0000
Subject: [PATCH 2715/3084] Use 'xor r, r' to set registers to 0 instead of mov
 (#766)

---
 .../codegen/meta/src/cdsl/instructions.rs     | 16 +++++++
 .../codegen/meta/src/isa/x86/encodings.rs     | 30 +++++++++++++
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 12 +++++
 cranelift/codegen/src/predicates.rs           |  6 +++
 .../floating-point-zero-constants-32bit.clif  | 17 +++++++
 .../x86/floating-point-zero-constants.clif    | 31 +++++++++++++
 .../filetests/isa/x86/isub_imm-i8.clif        |  8 ++--
 .../x86/optimized-zero-constants-32bit.clif   | 36 ++++++++++++++-
 .../isa/x86/optimized-zero-constants.clif     | 44 ++++++++++++++++++-
 9 files changed, 193 insertions(+), 7 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/floating-point-zero-constants-32bit.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/floating-point-zero-constants.clif

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index f061be94ed..2737099cc4 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -606,6 +606,8 @@ pub enum FormatPredicateKind {
     /// `2^scale`.
     IsUnsignedInt(usize, usize),
 
+    /// Is the immediate format field member an integer equal to zero?
+    IsZeroInt,
     /// Is the immediate format field member equal to zero? (float32 version)
     IsZero32BitFloat,
 
@@ -679,6 +681,9 @@ impl FormatPredicateNode {
                 "predicates::is_unsigned_int({}, {}, {})",
                 self.member_name, width, scale
             ),
+            FormatPredicateKind::IsZeroInt => {
+                format!("predicates::is_zero_int({})", self.member_name)
+            }
             FormatPredicateKind::IsZero32BitFloat => {
                 format!("predicates::is_zero_32_bit_float({})", self.member_name)
             }
@@ -891,6 +896,17 @@ impl InstructionPredicate {
         ))
     }
 
+    pub fn new_is_zero_int(
+        format: &InstructionFormat,
+        field_name: &'static str,
+    ) -> InstructionPredicateNode {
+        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
+            format,
+            field_name,
+            FormatPredicateKind::IsZeroInt,
+        ))
+    }
+
     pub fn new_is_zero_32bit_float(
         format: &InstructionFormat,
         field_name: &'static str,
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index f16ca93717..9bdb751b26 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -614,6 +614,7 @@ pub(crate) fn define(
     let rec_trapif = r.recipe("trapif");
     let rec_trapff = r.recipe("trapff");
     let rec_u_id = r.template("u_id");
+    let rec_u_id_z = r.template("u_id_z");
     let rec_umr = r.template("umr");
     let rec_umr_reg_to_ssa = r.template("umr_reg_to_ssa");
     let rec_ur = r.template("ur");
@@ -750,6 +751,35 @@ pub(crate) fn define(
     }
     e.enc64(bconst.bind(B64), rec_pu_id_bool.opcodes(vec![0xb8]).rex());
 
+    let is_zero_int = InstructionPredicate::new_is_zero_int(f_unary_imm, "imm");
+    e.enc_both_instp(
+        iconst.bind(I8),
+        rec_u_id_z.opcodes(vec![0x30]),
+        is_zero_int.clone(),
+    );
+    // You may expect that i16 encodings would have an 0x66 prefix on the opcode to indicate that
+    // encodings should be on 16-bit operands (f.ex, "xor %ax, %ax"). Cranelift currently does not
+    // know that it can drop the 0x66 prefix and clear the upper half of a 32-bit register in these
+    // scenarios, so we explicitly select a wider but permissible opcode.
+    //
+    // This effectively formalizes the i16->i32 widening that Cranelift performs when there isn't
+    // an appropriate i16 encoding available.
+    e.enc_both_instp(
+        iconst.bind(I16),
+        rec_u_id_z.opcodes(vec![0x31]),
+        is_zero_int.clone(),
+    );
+    e.enc_both_instp(
+        iconst.bind(I32),
+        rec_u_id_z.opcodes(vec![0x31]),
+        is_zero_int.clone(),
+    );
+    e.enc_x86_64_instp(
+        iconst.bind(I64),
+        rec_u_id_z.opcodes(vec![0x31]),
+        is_zero_int,
+    );
+
     // Shifts and rotates.
     // Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit
     // and 16-bit shifts would need explicit masking.
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 8176effc42..11a9972d98 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -1023,6 +1023,18 @@ pub(crate) fn define<'shared>(
             ),
     );
 
+    // XX+rd id unary with zero immediate.
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("u_id_z", f_unary_imm, 1)
+            .operands_out(vec![gpr])
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(out_reg0, out_reg0), sink);
+                    modrm_rr(out_reg0, out_reg0, sink);
+                "#,
+            ),
+    );
+
     // XX /n Unary with floating point 32-bit immediate equal to zero.
     {
         let format = formats.get(f_unary_ieee32);
diff --git a/cranelift/codegen/src/predicates.rs b/cranelift/codegen/src/predicates.rs
index 978e733c66..f900546111 100644
--- a/cranelift/codegen/src/predicates.rs
+++ b/cranelift/codegen/src/predicates.rs
@@ -11,6 +11,12 @@
 
 use crate::ir;
 
+/// Check that an integer value is zero.
+#[allow(dead_code)]
+pub fn is_zero_int>(x: T) -> bool {
+    x.into() == 0
+}
+
 /// Check that a 64-bit floating point value is zero.
 #[allow(dead_code)]
 pub fn is_zero_64_bit_float>(x: T) -> bool {
diff --git a/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants-32bit.clif b/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants-32bit.clif
new file mode 100644
index 0000000000..8021375558
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants-32bit.clif
@@ -0,0 +1,17 @@
+; Check that floating-point and integer constants equal to zero are optimized correctly.
+test binemit
+target i686
+
+function %foo() -> f32 fast {
+ebb0:
+  ; asm: xorps %xmm0, %xmm0
+  [-,%xmm0]    v0 = f32const 0.0     ; bin: 0f 57 c0
+  return v0
+}
+
+function %bar() -> f64 fast {
+ebb0:
+  ; asm: xorpd %xmm0, %xmm0
+  [-,%xmm0]    v1 = f64const 0.0     ; bin: 66 0f 57 c0
+  return v1
+}
diff --git a/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants.clif b/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants.clif
new file mode 100644
index 0000000000..049320870e
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants.clif
@@ -0,0 +1,31 @@
+; Check that floating-point constants equal to zero are optimized correctly.
+test binemit
+target x86_64
+
+function %zero_const_32bit_no_rex() -> f32 fast {
+ebb0:
+  ; asm: xorps %xmm0, %xmm0
+  [-,%xmm0]    v0 = f32const 0.0     ; bin: 40 0f 57 c0
+  return v0
+}
+
+function %zero_const_32bit_rex() -> f32 fast {
+ebb0:
+  ; asm: xorps %xmm8, %xmm8
+  [-,%xmm8]    v1 = f32const 0.0     ; bin: 45 0f 57 c0
+  return v1
+}
+
+function %zero_const_64bit_no_rex() -> f64 fast {
+ebb0:
+  ; asm: xorpd %xmm0, %xmm0
+  [-,%xmm0]    v0 = f64const 0.0     ; bin: 66 40 0f 57 c0
+  return v0
+}
+
+function %zero_const_64bit_rex() -> f64 fast {
+ebb0:
+  ; asm: xorpd %xmm8, %xmm8
+  [-,%xmm8]    v1 = f64const 0.0     ; bin: 66 45 0f 57 c0
+  return v1
+}
diff --git a/cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif b/cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif
index 8958b1afa4..1ca70ebbbe 100644
--- a/cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif
@@ -5,9 +5,9 @@ function u0:0(i8) -> i8 fast {
 ebb0(v0: i8):
     v1 = iconst.i8 0
     v2 = isub v1, v0
-    ; check: v4 = uextend.i32 v0
-    ; nextln: v6 = iconst.i32 0
-    ; nextln = isub v6, v4
-    ; nextln = ireduce.i8 v5
+    ; check: v3 = uextend.i32 v0
+    ; nextln: v5 = iconst.i32 0
+    ; nextln = isub v5, v3
+    ; nextln = ireduce.i8 v4
     return v2
 }
diff --git a/cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif
index c90e455527..21f936c4b9 100644
--- a/cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif
+++ b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif
@@ -1,5 +1,6 @@
-; Check that floating-point constants equal to zero are optimized correctly.
+; Check that floating-point and integer constants equal to zero are optimized correctly.
 test binemit
+set opt_level=best
 target i686
 
 function %foo() -> f32 fast {
@@ -16,3 +17,36 @@ ebb0:
   return v1
 }
 
+function %zero_dword() -> i32 fast {
+ebb0:
+  ; asm: xor %eax, %eax
+  [-,%rax]     v0 = iconst.i32 0     ; bin: 31 c0
+  ; asm: xor %edi, %edi
+  [-,%rdi]     v1 = iconst.i32 0     ; bin: 31 ff
+  return v0
+}
+
+function %zero_word() -> i16 fast {
+ebb0:
+  ; while you may expect this to be encoded like 6631c0, aka
+  ; xor %ax, %ax, the upper 16 bits of the register used for
+  ; i16 are left undefined, so it's not wrong to clear them.
+  ;
+  ; discarding the 66 prefix is shorter, so this test expects
+  ; that we do so.
+  ;
+  ; asm: xor %eax, %eax
+  [-,%rax]     v0 = iconst.i16 0     ; bin: 31 c0
+  ; asm: xor %edi, %edi
+  [-,%rdi]     v1 = iconst.i16 0     ; bin: 31 ff
+  return v0
+}
+
+function %zero_byte() -> i8 fast {
+ebb0:
+  ; asm: xor %al, %al
+  [-,%rax]     v0 = iconst.i8 0     ; bin: 30 c0
+  ; asm: xor %dh, %dh
+  [-,%rdi]     v1 = iconst.i8 0     ; bin: 30 ff
+  return v0
+}
diff --git a/cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif
index 44060e9b97..4a1ad00ff4 100644
--- a/cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif
+++ b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif
@@ -1,11 +1,12 @@
 ; Check that floating-point constants equal to zero are optimized correctly.
 test binemit
+set opt_level=best
 target x86_64
 
 function %zero_const_32bit_no_rex() -> f32 fast {
 ebb0:
   ; asm: xorps %xmm0, %xmm0
-  [-,%xmm0]    v0 = f32const 0.0     ; bin: 40 0f 57 c0
+  [-,%xmm0]    v0 = f32const 0.0     ; bin: 0f 57 c0
   return v0
 }
 
@@ -19,7 +20,7 @@ ebb0:
 function %zero_const_64bit_no_rex() -> f64 fast {
 ebb0:
   ; asm: xorpd %xmm0, %xmm0
-  [-,%xmm0]    v0 = f64const 0.0     ; bin: 66 40 0f 57 c0
+  [-,%xmm0]    v0 = f64const 0.0     ; bin: 66 0f 57 c0
   return v0
 }
 
@@ -30,3 +31,42 @@ ebb0:
   return v1
 }
 
+function %imm_zero_register() -> i64 fast {
+ebb0:
+  ; asm: xor %eax, %eax
+  [-,%rax]     v0 = iconst.i64 0     ; bin: 31 c0
+  ; asm: xor %edi, %edi
+  [-,%rdi]     v1 = iconst.i64 0     ; bin: 31 ff
+  ; asm: xor %r8, r8
+  [-,%r8]      v2 = iconst.i64 0     ; bin: 45 31 c0
+  ; asm: xor %r15, %r15
+  [-,%r15]     v4 = iconst.i64 0     ; bin: 45 31 ff
+  return v0
+}
+
+function %zero_word() -> i16 fast {
+ebb0:
+  ; while you may expect this to be encoded like 6631c0, aka
+  ; xor %ax, %ax, the upper 16 bits of the register used for
+  ; i16 are left undefined, so it's not wrong to clear them.
+  ;
+  ; discarding the 66 prefix is shorter, so this test expects
+  ; that we do so.
+  ;
+  ; asm: xor %eax, %eax
+  [-,%rax]     v0 = iconst.i16 0     ; bin: 31 c0
+  ; asm: xor %edi, %edi
+  [-,%rdi]     v1 = iconst.i16 0     ; bin: 31 ff
+  return v0
+}
+
+function %zero_byte() -> i8 fast {
+ebb0:
+  ; asm: xor %r8b, %r8b
+  [-,%r15]     v0 = iconst.i8 0     ; bin: 45 30 ff
+  ; asm: xor %al, %al
+  [-,%rax]     v1 = iconst.i8 0     ; bin: 30 c0
+  ; asm: xor %dh, %dh
+  [-,%rdi]     v2 = iconst.i8 0     ; bin: 30 ff
+  return v0
+}

From f8d0e6ecab19ad85c4f5e41ed93fe4e7a5122a7d Mon Sep 17 00:00:00 2001
From: bookmoons <35854232+bookmoons@users.noreply.github.com>
Date: Tue, 17 Sep 2019 01:12:31 -0400
Subject: [PATCH 2716/3084] Continuous fuzzing (#944)

Enables automated fuzzing on Fuzzit. Runs fuzz regression tests
every push and PR. Runs full fuzzing every push. Fuzzit emails
if it finds crashes.

Uses the existing fuzz targets:
* translate-module - Fuzz valid WebAssembly modules.
* reader-parse - Fuzz IR text format parsing.
---
 .azure-pipelines.yml                     | 24 +++++++++++++++
 cranelift/README.md                      |  1 +
 cranelift/ci/fuzzit.sh                   | 38 ++++++++++++++++++++++++
 cranelift/fuzz/fuzz_reader_parse_test.rs |  3 +-
 4 files changed, 65 insertions(+), 1 deletion(-)
 create mode 100755 cranelift/ci/fuzzit.sh

diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml
index a18a45652e..9e0a4697eb 100644
--- a/.azure-pipelines.yml
+++ b/.azure-pipelines.yml
@@ -97,6 +97,30 @@ jobs:
         RUST_BACKTRACE: 1
       condition: and(succeeded(), eq(variables['toolchain'], 'nightly'))
 
+- job: Fuzz_regression
+  displayName: Fuzz regression
+  pool:
+    vmImage: "ubuntu-16.04"
+  variables:
+    toolchain: nightly
+  steps:
+    - template: ci/azure-install-rust.yml
+    - bash: cargo install cargo-fuzz
+    - bash: ci/fuzzit.sh local-regression
+
+- job: Fuzz
+  condition: ne(variables['Build.Reason'], 'PullRequest')
+  pool:
+    vmImage: "ubuntu-16.04"
+  variables:
+    toolchain: nightly
+  steps:
+    - template: ci/azure-install-rust.yml
+    - bash: cargo install cargo-fuzz
+    - bash: ci/fuzzit.sh fuzzing
+      env:
+        FUZZIT_API_KEY: $(FUZZIT_API_KEY)
+
 - job: Build
   strategy:
     matrix:
diff --git a/cranelift/README.md b/cranelift/README.md
index 3bfa2dc6cd..171055d030 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -8,6 +8,7 @@ into executable machine code.
 
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
 [![Travis Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
+[![Fuzzit Status](https://app.fuzzit.dev/badge?org_id=CraneStation)](https://app.fuzzit.dev/orgs/CraneStation/dashboard)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
 ![Minimum rustc 1.37](https://img.shields.io/badge/rustc-1.37+-green.svg)
 
diff --git a/cranelift/ci/fuzzit.sh b/cranelift/ci/fuzzit.sh
new file mode 100755
index 0000000000..c08fb3c64e
--- /dev/null
+++ b/cranelift/ci/fuzzit.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+set -xe
+
+# Validate arguments
+if [ "$#" -ne 1 ]; then
+    cat << EOF
+Usage: $0 
+
+Types are:
+local-regression - Run corpus and past crashes locally to catch regressions.
+fuzzing - Submit for long run fuzzing on Fuzzit.
+EOF
+    exit 1
+fi
+
+# Configure
+set -xe
+NAME=cranelift
+TYPE=$1
+FUZZIT_VERSION=2.4.46
+
+# Setup
+if [[ ! -f fuzzit || ! `./fuzzit --version` =~ $FUZZIT_VERSION$ ]]; then
+    wget -q -O fuzzit https://github.com/fuzzitdev/fuzzit/releases/download/v$FUZZIT_VERSION/fuzzit_Linux_x86_64
+    chmod a+x fuzzit
+fi
+./fuzzit --version
+
+# Fuzz
+function fuzz {
+    FUZZER=$1
+    TARGET=$2
+    cargo fuzz run $FUZZER -- -runs=0
+    ./fuzzit --version
+    ./fuzzit create job --type $TYPE $NAME/$TARGET ./fuzz/target/x86_64-unknown-linux-gnu/debug/$FUZZER
+}
+fuzz fuzz_translate_module translate-module
+fuzz fuzz_reader_parse_test reader-parse
diff --git a/cranelift/fuzz/fuzz_reader_parse_test.rs b/cranelift/fuzz/fuzz_reader_parse_test.rs
index cccf135916..4f17898264 100644
--- a/cranelift/fuzz/fuzz_reader_parse_test.rs
+++ b/cranelift/fuzz/fuzz_reader_parse_test.rs
@@ -6,6 +6,7 @@ use std::str;
 
 fuzz_target!(|data: &[u8]| {
     if let Ok(s) = str::from_utf8(data) {
-        let _ = cranelift_reader::parse_test(s, None, None);
+        let options = cranelift_reader::ParseOptions::default();
+        let _ = cranelift_reader::parse_test(s, options);
     }
 });

From 8d62d5f72445bf081d7245ce815d6c8c87489872 Mon Sep 17 00:00:00 2001
From: Till Schneidereit 
Date: Tue, 17 Sep 2019 15:06:37 +0900
Subject: [PATCH 2717/3084] Revert "Continuous fuzzing (#944)" (#1036)

This reverts commit 374fd238b06860a1e6e3f8bb9eb598afe91b90fe.
---
 .azure-pipelines.yml                     | 24 ---------------
 cranelift/README.md                      |  1 -
 cranelift/ci/fuzzit.sh                   | 38 ------------------------
 cranelift/fuzz/fuzz_reader_parse_test.rs |  3 +-
 4 files changed, 1 insertion(+), 65 deletions(-)
 delete mode 100755 cranelift/ci/fuzzit.sh

diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml
index 9e0a4697eb..a18a45652e 100644
--- a/.azure-pipelines.yml
+++ b/.azure-pipelines.yml
@@ -97,30 +97,6 @@ jobs:
         RUST_BACKTRACE: 1
       condition: and(succeeded(), eq(variables['toolchain'], 'nightly'))
 
-- job: Fuzz_regression
-  displayName: Fuzz regression
-  pool:
-    vmImage: "ubuntu-16.04"
-  variables:
-    toolchain: nightly
-  steps:
-    - template: ci/azure-install-rust.yml
-    - bash: cargo install cargo-fuzz
-    - bash: ci/fuzzit.sh local-regression
-
-- job: Fuzz
-  condition: ne(variables['Build.Reason'], 'PullRequest')
-  pool:
-    vmImage: "ubuntu-16.04"
-  variables:
-    toolchain: nightly
-  steps:
-    - template: ci/azure-install-rust.yml
-    - bash: cargo install cargo-fuzz
-    - bash: ci/fuzzit.sh fuzzing
-      env:
-        FUZZIT_API_KEY: $(FUZZIT_API_KEY)
-
 - job: Build
   strategy:
     matrix:
diff --git a/cranelift/README.md b/cranelift/README.md
index 171055d030..3bfa2dc6cd 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -8,7 +8,6 @@ into executable machine code.
 
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
 [![Travis Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
-[![Fuzzit Status](https://app.fuzzit.dev/badge?org_id=CraneStation)](https://app.fuzzit.dev/orgs/CraneStation/dashboard)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
 ![Minimum rustc 1.37](https://img.shields.io/badge/rustc-1.37+-green.svg)
 
diff --git a/cranelift/ci/fuzzit.sh b/cranelift/ci/fuzzit.sh
deleted file mode 100755
index c08fb3c64e..0000000000
--- a/cranelift/ci/fuzzit.sh
+++ /dev/null
@@ -1,38 +0,0 @@
-#!/bin/bash
-set -xe
-
-# Validate arguments
-if [ "$#" -ne 1 ]; then
-    cat << EOF
-Usage: $0 
-
-Types are:
-local-regression - Run corpus and past crashes locally to catch regressions.
-fuzzing - Submit for long run fuzzing on Fuzzit.
-EOF
-    exit 1
-fi
-
-# Configure
-set -xe
-NAME=cranelift
-TYPE=$1
-FUZZIT_VERSION=2.4.46
-
-# Setup
-if [[ ! -f fuzzit || ! `./fuzzit --version` =~ $FUZZIT_VERSION$ ]]; then
-    wget -q -O fuzzit https://github.com/fuzzitdev/fuzzit/releases/download/v$FUZZIT_VERSION/fuzzit_Linux_x86_64
-    chmod a+x fuzzit
-fi
-./fuzzit --version
-
-# Fuzz
-function fuzz {
-    FUZZER=$1
-    TARGET=$2
-    cargo fuzz run $FUZZER -- -runs=0
-    ./fuzzit --version
-    ./fuzzit create job --type $TYPE $NAME/$TARGET ./fuzz/target/x86_64-unknown-linux-gnu/debug/$FUZZER
-}
-fuzz fuzz_translate_module translate-module
-fuzz fuzz_reader_parse_test reader-parse
diff --git a/cranelift/fuzz/fuzz_reader_parse_test.rs b/cranelift/fuzz/fuzz_reader_parse_test.rs
index 4f17898264..cccf135916 100644
--- a/cranelift/fuzz/fuzz_reader_parse_test.rs
+++ b/cranelift/fuzz/fuzz_reader_parse_test.rs
@@ -6,7 +6,6 @@ use std::str;
 
 fuzz_target!(|data: &[u8]| {
     if let Ok(s) = str::from_utf8(data) {
-        let options = cranelift_reader::ParseOptions::default();
-        let _ = cranelift_reader::parse_test(s, options);
+        let _ = cranelift_reader::parse_test(s, None, None);
     }
 });

From 5426e42a27f24fbbb217d8c9a511e46162198866 Mon Sep 17 00:00:00 2001
From: Erin Power 
Date: Sat, 14 Sep 2019 17:08:10 +0200
Subject: [PATCH 2718/3084] Revert "Remove FunctionBuilderContext from API, and
 change FunctionBuilder API"

This reverts commit 39e638af99dbe6537bc935bfb1a74669b62877b3.
---
 cranelift/frontend/src/frontend.rs            | 156 +++++++++---------
 cranelift/frontend/src/lib.rs                 |  19 ++-
 cranelift/frontend/src/ssa.rs                 |   4 +-
 cranelift/frontend/src/switch.rs              |  16 +-
 cranelift/frontend/src/variable.rs            |   2 +-
 .../simplejit/examples/simplejit-minimal.rs   |  17 +-
 cranelift/simplejit/tests/basic.rs            |  37 ++---
 cranelift/umbrella/src/lib.rs                 |   2 +-
 cranelift/wasm/src/code_translator.rs         |  19 +--
 cranelift/wasm/src/environ/dummy.rs           |   3 +-
 cranelift/wasm/src/func_translator.rs         |  59 ++++---
 11 files changed, 172 insertions(+), 162 deletions(-)

diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 37ac4b95dc..fbd2428e1c 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -20,22 +20,22 @@ use std::vec::Vec;
 /// In order to reduce memory reallocations when compiling multiple functions,
 /// `FunctionBuilderContext` holds various data structures which are cleared between
 /// functions, rather than dropped, preserving the underlying allocations.
-struct FunctionBuilderContext {
+pub struct FunctionBuilderContext {
     ssa: SSABuilder,
     ebbs: SecondaryMap,
     types: SecondaryMap,
 }
 
 /// Temporary object used to build a single Cranelift IR `Function`.
-pub struct FunctionBuilder {
+pub struct FunctionBuilder<'a> {
     /// The function currently being built.
     /// This field is public so the function can be re-borrowed.
-    pub func: Function,
+    pub func: &'a mut Function,
 
     /// Source location to assign to all new instructions.
     srcloc: ir::SourceLoc,
 
-    func_ctx: FunctionBuilderContext,
+    func_ctx: &'a mut FunctionBuilderContext,
     position: Position,
 }
 
@@ -75,9 +75,8 @@ impl Position {
 }
 
 impl FunctionBuilderContext {
-    /// Creates a FunctionBuilderContext structure. The structure is
-    /// automatically cleared after each `FunctionBuilder` completes translating
-    /// a function.
+    /// Creates a FunctionBuilderContext structure. The structure is automatically cleared after
+    /// each [`FunctionBuilder`](struct.FunctionBuilder.html) completes translating a function.
     pub fn new() -> Self {
         Self {
             ssa: SSABuilder::new(),
@@ -91,22 +90,26 @@ impl FunctionBuilderContext {
         self.ebbs.clear();
         self.types.clear();
     }
+
+    fn is_empty(&self) -> bool {
+        self.ssa.is_empty() && self.ebbs.is_empty() && self.types.is_empty()
+    }
 }
 
 /// Implementation of the [`InstBuilder`](cranelift_codegen::ir::InstBuilder) that has
 /// one convenience method per Cranelift IR instruction.
-pub struct FuncInstBuilder<'short> {
-    builder: &'short mut FunctionBuilder,
+pub struct FuncInstBuilder<'short, 'long: 'short> {
+    builder: &'short mut FunctionBuilder<'long>,
     ebb: Ebb,
 }
 
-impl<'short> FuncInstBuilder<'short> {
-    fn new(builder: &'short mut FunctionBuilder, ebb: Ebb) -> Self {
+impl<'short, 'long> FuncInstBuilder<'short, 'long> {
+    fn new(builder: &'short mut FunctionBuilder<'long>, ebb: Ebb) -> Self {
         Self { builder, ebb }
     }
 }
 
-impl<'short> InstBuilderBase<'short> for FuncInstBuilder<'short> {
+impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
     fn data_flow_graph(&self) -> &DataFlowGraph {
         &self.builder.func.dfg
     }
@@ -200,9 +203,12 @@ impl<'short> InstBuilderBase<'short> for FuncInstBuilder<'short> {
 /// The first block for which you call `switch_to_block` will be assumed to be the beginning of
 /// the function.
 ///
-/// At creation, a `FunctionBuilder` instance borrows an already allocated
-/// `Function`. The function passed in should be newly created with
-/// [`Function::with_name_signature()`](Function::with_name_signature).
+/// At creation, a `FunctionBuilder` instance borrows an already allocated `Function` which it
+/// modifies with the information stored in the mutable borrowed
+/// [`FunctionBuilderContext`](struct.FunctionBuilderContext.html). The function passed in
+/// argument should be newly created with
+/// [`Function::with_name_signature()`](Function::with_name_signature), whereas the
+/// `FunctionBuilderContext` can be kept as is between two function translations.
 ///
 /// # Errors
 ///
@@ -210,13 +216,15 @@ impl<'short> InstBuilderBase<'short> for FuncInstBuilder<'short> {
 /// function in a way that violate the coherence of the code. For instance: switching to a new
 /// `Ebb` when you haven't filled the current one with a terminator instruction, inserting a
 /// return instruction with arguments that don't match the function's signature.
-impl FunctionBuilder {
-    /// Creates a new FunctionBuilder structure that will operate on a `Function`.
-    pub fn new(func: Function) -> Self {
+impl<'a> FunctionBuilder<'a> {
+    /// Creates a new FunctionBuilder structure that will operate on a `Function` using a
+    /// `FunctionBuilderContext`.
+    pub fn new(func: &'a mut Function, func_ctx: &'a mut FunctionBuilderContext) -> Self {
+        debug_assert!(func_ctx.is_empty());
         Self {
             func,
             srcloc: Default::default(),
-            func_ctx: FunctionBuilderContext::new(),
+            func_ctx,
             position: Position::default(),
         }
     }
@@ -271,7 +279,7 @@ impl FunctionBuilder {
     /// created. Forgetting to call this method on every block will cause inconsistencies in the
     /// produced functions.
     pub fn seal_block(&mut self, ebb: Ebb) {
-        let side_effects = self.func_ctx.ssa.seal_ebb_header_block(ebb, &mut self.func);
+        let side_effects = self.func_ctx.ssa.seal_ebb_header_block(ebb, self.func);
         self.handle_ssa_side_effects(side_effects);
     }
 
@@ -282,7 +290,7 @@ impl FunctionBuilder {
     /// function can be used at the end of translating all blocks to ensure
     /// that everything is sealed.
     pub fn seal_all_blocks(&mut self) {
-        let side_effects = self.func_ctx.ssa.seal_all_ebb_header_blocks(&mut self.func);
+        let side_effects = self.func_ctx.ssa.seal_all_ebb_header_blocks(self.func);
         self.handle_ssa_side_effects(side_effects);
     }
 
@@ -303,7 +311,7 @@ impl FunctionBuilder {
             });
             self.func_ctx
                 .ssa
-                .use_var(&mut self.func, var, ty, self.position.basic_block.unwrap())
+                .use_var(self.func, var, ty, self.position.basic_block.unwrap())
         };
         self.handle_ssa_side_effects(side_effects);
         val
@@ -385,7 +393,7 @@ impl FunctionBuilder {
 
     /// Returns an object with the [`InstBuilder`](cranelift_codegen::ir::InstBuilder)
     /// trait that allows to conveniently append an instruction to the current `Ebb` being built.
-    pub fn ins<'short>(&'short mut self) -> FuncInstBuilder<'short> {
+    pub fn ins<'short>(&'short mut self) -> FuncInstBuilder<'short, 'a> {
         let ebb = self
             .position
             .ebb
@@ -415,7 +423,7 @@ impl FunctionBuilder {
     /// need to know about `FunctionBuilder` at all.
     pub fn cursor(&mut self) -> FuncCursor {
         self.ensure_inserted_ebb();
-        FuncCursor::new(&mut self.func)
+        FuncCursor::new(self.func)
             .with_srcloc(self.srcloc)
             .at_bottom(self.position.ebb.unwrap())
     }
@@ -451,21 +459,10 @@ impl FunctionBuilder {
         }
     }
 
-    /// Clears the current state of the `FunctionBuilder` while preserving
-    /// allocations for reuse in translating another function.
-    pub fn clear(&mut self) {
-        // Clear the state (but preserve the allocated buffers) in preparation
-        // for translation another function.
-        self.func_ctx.clear();
-        // Reset srcloc and position to initial states.
-        self.srcloc = Default::default();
-        self.position = Position::default();
-    }
-
-    /// Declare that translation of the current function is complete, and return the completed
-    /// function. This resets the state of the `FunctionBuilder` in preparation to be used
+    /// Declare that translation of the current function is complete. This
+    /// resets the state of the `FunctionBuilder` in preparation to be used
     /// for another function.
-    pub fn finalize(&mut self) -> ir::Function {
+    pub fn finalize(&mut self) {
         // Check that all the `Ebb`s are filled and sealed.
         debug_assert!(
             self.func_ctx
@@ -495,8 +492,13 @@ impl FunctionBuilder {
             }
         }
 
-        self.clear();
-        std::mem::replace(&mut self.func, Function::new())
+        // Clear the state (but preserve the allocated buffers) in preparation
+        // for translation another function.
+        self.func_ctx.clear();
+
+        // Reset srcloc and position to initial states.
+        self.srcloc = Default::default();
+        self.position = Position::default();
     }
 }
 
@@ -505,7 +507,7 @@ impl FunctionBuilder {
 /// performance of your translation perform more complex transformations to your Cranelift IR
 /// function. The functions below help you inspect the function you're creating and modify it
 /// in ways that can be unsafe if used incorrectly.
-impl FunctionBuilder {
+impl<'a> FunctionBuilder<'a> {
     /// Retrieves all the parameters for an `Ebb` currently inferred from the jump instructions
     /// inserted that target it and the SSA construction.
     pub fn ebb_params(&self, ebb: Ebb) -> &[Value] {
@@ -594,7 +596,7 @@ impl FunctionBuilder {
 }
 
 /// Helper functions
-impl FunctionBuilder {
+impl<'a> FunctionBuilder<'a> {
     /// Calls libc.memcpy
     ///
     /// Copies the `size` bytes from `src` to `dest`, assumes that `src + size`
@@ -854,7 +856,7 @@ fn greatest_divisible_power_of_two(size: u64) -> u64 {
 }
 
 // Helper functions
-impl FunctionBuilder {
+impl<'a> FunctionBuilder<'a> {
     fn move_to_next_basic_block(&mut self) {
         self.position.basic_block = PackedOption::from(
             self.func_ctx
@@ -889,7 +891,7 @@ impl FunctionBuilder {
 #[cfg(test)]
 mod tests {
     use super::greatest_divisible_power_of_two;
-    use crate::frontend::FunctionBuilder;
+    use crate::frontend::{FunctionBuilder, FunctionBuilderContext};
     use crate::Variable;
     use cranelift_codegen::entity::EntityRef;
     use cranelift_codegen::ir::types::*;
@@ -904,9 +906,10 @@ mod tests {
         sig.returns.push(AbiParam::new(I32));
         sig.params.push(AbiParam::new(I32));
 
-        let func = {
-            let func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
-            let mut builder = FunctionBuilder::new(func);
+        let mut fn_ctx = FunctionBuilderContext::new();
+        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+        {
+            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
 
             let block0 = builder.create_ebb();
             let block1 = builder.create_ebb();
@@ -988,8 +991,8 @@ mod tests {
                 builder.seal_all_blocks();
             }
 
-            builder.finalize()
-        };
+            builder.finalize();
+        }
 
         let flags = settings::Flags::new(settings::builder());
         // println!("{}", func.display(None));
@@ -1026,9 +1029,10 @@ mod tests {
         let mut sig = Signature::new(target.default_call_conv());
         sig.returns.push(AbiParam::new(I32));
 
-        let func = {
-            let func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
-            let mut builder = FunctionBuilder::new(func);
+        let mut fn_ctx = FunctionBuilderContext::new();
+        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+        {
+            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
 
             let block0 = builder.create_ebb();
             let x = Variable::new(0);
@@ -1047,8 +1051,8 @@ mod tests {
             builder.ins().return_(&[size]);
 
             builder.seal_all_blocks();
-            builder.finalize()
-        };
+            builder.finalize();
+        }
 
         assert_eq!(
             func.display(None).to_string(),
@@ -1086,9 +1090,10 @@ ebb0:
         let mut sig = Signature::new(target.default_call_conv());
         sig.returns.push(AbiParam::new(I32));
 
-        let func = {
-            let func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
-            let mut builder = FunctionBuilder::new(func);
+        let mut fn_ctx = FunctionBuilderContext::new();
+        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+        {
+            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
 
             let block0 = builder.create_ebb();
             let x = Variable::new(0);
@@ -1105,8 +1110,8 @@ ebb0:
             builder.ins().return_(&[dest]);
 
             builder.seal_all_blocks();
-            builder.finalize()
-        };
+            builder.finalize();
+        }
 
         assert_eq!(
             func.display(None).to_string(),
@@ -1142,9 +1147,10 @@ ebb0:
         let mut sig = Signature::new(target.default_call_conv());
         sig.returns.push(AbiParam::new(I32));
 
-        let func = {
-            let func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
-            let mut builder = FunctionBuilder::new(func);
+        let mut fn_ctx = FunctionBuilderContext::new();
+        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+        {
+            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
 
             let block0 = builder.create_ebb();
             let x = Variable::new(0);
@@ -1161,8 +1167,8 @@ ebb0:
             builder.ins().return_(&[dest]);
 
             builder.seal_all_blocks();
-            builder.finalize()
-        };
+            builder.finalize();
+        }
 
         assert_eq!(
             func.display(None).to_string(),
@@ -1201,9 +1207,10 @@ ebb0:
         let mut sig = Signature::new(target.default_call_conv());
         sig.returns.push(AbiParam::new(I32));
 
-        let func = {
-            let func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
-            let mut builder = FunctionBuilder::new(func);
+        let mut fn_ctx = FunctionBuilderContext::new();
+        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+        {
+            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
 
             let block0 = builder.create_ebb();
             let y = Variable::new(16);
@@ -1217,8 +1224,8 @@ ebb0:
             builder.ins().return_(&[dest]);
 
             builder.seal_all_blocks();
-            builder.finalize()
-        };
+            builder.finalize();
+        }
 
         assert_eq!(
             func.display(None).to_string(),
@@ -1252,9 +1259,10 @@ ebb0:
         let mut sig = Signature::new(target.default_call_conv());
         sig.returns.push(AbiParam::new(I32));
 
-        let func = {
-            let func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
-            let mut builder = FunctionBuilder::new(func);
+        let mut fn_ctx = FunctionBuilderContext::new();
+        let mut func = Function::with_name_signature(ExternalName::testcase("sample"), sig);
+        {
+            let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
 
             let block0 = builder.create_ebb();
             let y = Variable::new(16);
@@ -1268,8 +1276,8 @@ ebb0:
             builder.ins().return_(&[dest]);
 
             builder.seal_all_blocks();
-            builder.finalize()
-        };
+            builder.finalize();
+        }
 
         assert_eq!(
             func.display(None).to_string(),
diff --git a/cranelift/frontend/src/lib.rs b/cranelift/frontend/src/lib.rs
index 14cd7502dc..eff5c655fd 100644
--- a/cranelift/frontend/src/lib.rs
+++ b/cranelift/frontend/src/lib.rs
@@ -3,7 +3,7 @@
 //! Provides a straightforward way to create a Cranelift IR function and fill it with instructions
 //! corresponding to your source program written in another language.
 //!
-//! To get started, create an [`Function`](../cranelift_codegen/ir/function/struct.Function.html) and
+//! To get started, create an [`FunctionBuilderContext`](struct.FunctionBuilderContext.html) and
 //! pass it as an argument to a [`FunctionBuilder`](struct.FunctionBuilder.html).
 //!
 //! # Mutable variables and Cranelift IR values
@@ -61,7 +61,7 @@
 //! }
 //! ```
 //!
-//! Here is how you build the corresponding Cranelift IR function using `Function`:
+//! Here is how you build the corresponding Cranelift IR function using `FunctionBuilderContext`:
 //!
 //! ```rust
 //! extern crate cranelift_codegen;
@@ -73,15 +73,16 @@
 //! use cranelift_codegen::isa::CallConv;
 //! use cranelift_codegen::settings;
 //! use cranelift_codegen::verifier::verify_function;
-//! use cranelift_frontend::{FunctionBuilder, Variable};
+//! use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
 //!
 //! fn main() {
 //!     let mut sig = Signature::new(CallConv::SystemV);
 //!     sig.returns.push(AbiParam::new(I32));
 //!     sig.params.push(AbiParam::new(I32));
-//!     let func = {
-//!         let func = Function::with_name_signature(ExternalName::user(0, 0), sig);
-//!         let mut builder = FunctionBuilder::new(func);
+//!     let mut fn_builder_ctx = FunctionBuilderContext::new();
+//!     let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
+//!     {
+//!         let mut builder = FunctionBuilder::new(&mut func, &mut fn_builder_ctx);
 //!
 //!         let block0 = builder.create_ebb();
 //!         let block1 = builder.create_ebb();
@@ -151,8 +152,8 @@
 //!         builder.ins().jump(block1, &[]);
 //!         builder.seal_block(block1);
 //!
-//!         builder.finalize()
-//!     };
+//!         builder.finalize();
+//!     }
 //!
 //!     let flags = settings::Flags::new(settings::builder());
 //!     let res = verify_function(&func, &flags);
@@ -194,7 +195,7 @@ use hashmap_core::HashMap;
 #[cfg(feature = "std")]
 use std::collections::HashMap;
 
-pub use crate::frontend::FunctionBuilder;
+pub use crate::frontend::{FunctionBuilder, FunctionBuilderContext};
 pub use crate::switch::Switch;
 pub use crate::variable::Variable;
 
diff --git a/cranelift/frontend/src/ssa.rs b/cranelift/frontend/src/ssa.rs
index 43300f4ded..b3b74fac9b 100644
--- a/cranelift/frontend/src/ssa.rs
+++ b/cranelift/frontend/src/ssa.rs
@@ -176,7 +176,9 @@ impl SSABuilder {
         self.variables.clear();
         self.blocks.clear();
         self.ebb_headers.clear();
-        debug_assert!(self.is_empty());
+        debug_assert!(self.calls.is_empty());
+        debug_assert!(self.results.is_empty());
+        debug_assert!(self.side_effects.is_empty());
     }
 
     /// Tests whether an `SSABuilder` is in a cleared state.
diff --git a/cranelift/frontend/src/switch.rs b/cranelift/frontend/src/switch.rs
index 33b2001655..0b1ca746cb 100644
--- a/cranelift/frontend/src/switch.rs
+++ b/cranelift/frontend/src/switch.rs
@@ -16,11 +16,12 @@ type EntryIndex = u64;
 /// # use cranelift_codegen::ir::types::*;
 /// # use cranelift_codegen::ir::{ExternalName, Function, Signature, InstBuilder};
 /// # use cranelift_codegen::isa::CallConv;
-/// # use cranelift_frontend::{FunctionBuilder, Switch};
+/// # use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Switch};
 /// #
 /// # let mut sig = Signature::new(CallConv::SystemV);
+/// # let mut fn_builder_ctx = FunctionBuilderContext::new();
 /// # let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
-/// # let mut builder = FunctionBuilder::new(func);
+/// # let mut builder = FunctionBuilder::new(&mut func, &mut fn_builder_ctx);
 /// #
 /// # let entry = builder.create_ebb();
 /// # builder.switch_to_block(entry);
@@ -288,13 +289,16 @@ impl ContiguousCaseRange {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::frontend::FunctionBuilderContext;
     use cranelift_codegen::ir::Function;
     use std::string::ToString;
 
     macro_rules! setup {
         ($default:expr, [$($index:expr,)*]) => {{
-            let func = {
-                let mut bx = FunctionBuilder::new(Function::new());
+            let mut func = Function::new();
+            let mut func_ctx = FunctionBuilderContext::new();
+            {
+                let mut bx = FunctionBuilder::new(&mut func, &mut func_ctx);
                 let ebb = bx.create_ebb();
                 bx.switch_to_block(ebb);
                 let val = bx.ins().iconst(types::I8, 0);
@@ -304,9 +308,7 @@ mod tests {
                     switch.set_entry($index, ebb);
                 )*
                 switch.emit(&mut bx, val, Ebb::with_number($default).unwrap());
-                bx.seal_all_blocks();
-                bx.finalize()
-            };
+            }
             func
                 .to_string()
                 .trim_start_matches("function u0:0() fast {\n")
diff --git a/cranelift/frontend/src/variable.rs b/cranelift/frontend/src/variable.rs
index 1a061beb97..dddcd7490b 100644
--- a/cranelift/frontend/src/variable.rs
+++ b/cranelift/frontend/src/variable.rs
@@ -1,6 +1,6 @@
 //! A basic `Variable` implementation.
 //!
-//! `FunctionBuilder` and related types have a `Variable`
+//! `FunctionBuilderContext`, `FunctionBuilder`, and related types have a `Variable`
 //! type parameter, to allow frontends that identify variables with
 //! their own index types to use them directly. Frontends which don't
 //! can use the `Variable` defined here.
diff --git a/cranelift/simplejit/examples/simplejit-minimal.rs b/cranelift/simplejit/examples/simplejit-minimal.rs
index 2fc28367eb..3b8e147830 100644
--- a/cranelift/simplejit/examples/simplejit-minimal.rs
+++ b/cranelift/simplejit/examples/simplejit-minimal.rs
@@ -7,6 +7,7 @@ fn main() {
     let mut module: Module =
         Module::new(SimpleJITBuilder::new(default_libcall_names()));
     let mut ctx = module.make_context();
+    let mut func_ctx = FunctionBuilderContext::new();
 
     let mut sig_a = module.make_signature();
     sig_a.params.push(AbiParam::new(types::I32));
@@ -24,8 +25,8 @@ fn main() {
 
     ctx.func.signature = sig_a;
     ctx.func.name = ExternalName::user(0, func_a.as_u32());
-    ctx.func = {
-        let mut bcx: FunctionBuilder = FunctionBuilder::new(ctx.func);
+    {
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
         let ebb = bcx.create_ebb();
 
         bcx.switch_to_block(ebb);
@@ -35,15 +36,15 @@ fn main() {
         let add = bcx.ins().iadd(cst, param);
         bcx.ins().return_(&[add]);
         bcx.seal_all_blocks();
-        bcx.finalize()
-    };
+        bcx.finalize();
+    }
     module.define_function(func_a, &mut ctx).unwrap();
     module.clear_context(&mut ctx);
 
     ctx.func.signature = sig_b;
     ctx.func.name = ExternalName::user(0, func_b.as_u32());
-    ctx.func = {
-        let mut bcx: FunctionBuilder = FunctionBuilder::new(ctx.func);
+    {
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
         let ebb = bcx.create_ebb();
 
         bcx.switch_to_block(ebb);
@@ -57,8 +58,8 @@ fn main() {
         };
         bcx.ins().return_(&[value]);
         bcx.seal_all_blocks();
-        bcx.finalize()
-    };
+        bcx.finalize();
+    }
     module.define_function(func_b, &mut ctx).unwrap();
     module.clear_context(&mut ctx);
 
diff --git a/cranelift/simplejit/tests/basic.rs b/cranelift/simplejit/tests/basic.rs
index ea3f9cb5fd..a3932b1d6a 100644
--- a/cranelift/simplejit/tests/basic.rs
+++ b/cranelift/simplejit/tests/basic.rs
@@ -37,16 +37,14 @@ fn define_simple_function(module: &mut Module) -> FuncId {
         .unwrap();
 
     let mut ctx = Context::new();
-
-    ctx.func = {
-        let func = Function::with_name_signature(ExternalName::user(0, func_id.as_u32()), sig);
-        let mut bcx: FunctionBuilder = FunctionBuilder::new(func);
+    ctx.func = Function::with_name_signature(ExternalName::user(0, func_id.as_u32()), sig);
+    let mut func_ctx = FunctionBuilderContext::new();
+    {
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
         let ebb = bcx.create_ebb();
         bcx.switch_to_block(ebb);
         bcx.ins().return_(&[]);
-        bcx.seal_all_blocks();
-        bcx.finalize()
-    };
+    }
 
     module.define_function(func_id, &mut ctx).unwrap();
 
@@ -87,9 +85,11 @@ fn switch_error() {
         call_conv: CallConv::SystemV,
     };
 
-    let func = {
-        let func = Function::with_name_signature(ExternalName::user(0, 0), sig);
-        let mut bcx: FunctionBuilder = FunctionBuilder::new(func);
+    let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
+
+    let mut func_ctx = FunctionBuilderContext::new();
+    {
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut func, &mut func_ctx);
         let start = bcx.create_ebb();
         let bb0 = bcx.create_ebb();
         let bb1 = bcx.create_ebb();
@@ -134,8 +134,8 @@ fn switch_error() {
         bcx.ins().return_(&[r]);
 
         bcx.seal_all_blocks();
-        bcx.finalize()
-    };
+        bcx.finalize();
+    }
 
     let flags = settings::Flags::new(settings::builder());
     match cranelift_codegen::verify_function(&func, &flags) {
@@ -164,9 +164,10 @@ fn libcall_function() {
         .unwrap();
 
     let mut ctx = Context::new();
-    ctx.func = {
-        let func = Function::with_name_signature(ExternalName::user(0, func_id.as_u32()), sig);
-        let mut bcx: FunctionBuilder = FunctionBuilder::new(func);
+    ctx.func = Function::with_name_signature(ExternalName::user(0, func_id.as_u32()), sig);
+    let mut func_ctx = FunctionBuilderContext::new();
+    {
+        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
         let ebb = bcx.create_ebb();
         bcx.switch_to_block(ebb);
 
@@ -188,11 +189,7 @@ fn libcall_function() {
         bcx.call_memset(module.target_config(), buffer, zero, size);
 
         bcx.ins().return_(&[]);
-
-        bcx.seal_all_blocks();
-
-        bcx.finalize()
-    };
+    }
 
     module.define_function(func_id, &mut ctx).unwrap();
 
diff --git a/cranelift/umbrella/src/lib.rs b/cranelift/umbrella/src/lib.rs
index acb0e972d8..f97ed443ce 100644
--- a/cranelift/umbrella/src/lib.rs
+++ b/cranelift/umbrella/src/lib.rs
@@ -43,7 +43,7 @@ pub mod prelude {
     pub use crate::codegen::isa;
     pub use crate::codegen::settings::{self, Configurable};
 
-    pub use crate::frontend::{FunctionBuilder, Variable};
+    pub use crate::frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
 }
 
 /// Version number of this crate.
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index e5ce5a0c32..c420147550 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -81,7 +81,7 @@ pub fn translate_operator(
          *  `get_global` and `set_global` are handled by the environment.
          ***********************************************************************************/
         Operator::GetGlobal { global_index } => {
-            let val = match state.get_global(&mut builder.func, *global_index, environ)? {
+            let val = match state.get_global(builder.func, *global_index, environ)? {
                 GlobalVariable::Const(val) => val,
                 GlobalVariable::Memory { gv, offset, ty } => {
                     let addr = builder.ins().global_value(environ.pointer_type(), gv);
@@ -92,7 +92,7 @@ pub fn translate_operator(
             state.push1(val);
         }
         Operator::SetGlobal { global_index } => {
-            match state.get_global(&mut builder.func, *global_index, environ)? {
+            match state.get_global(builder.func, *global_index, environ)? {
                 GlobalVariable::Const(_) => panic!("global #{} is a constant", *global_index),
                 GlobalVariable::Memory { gv, offset, ty } => {
                     let addr = builder.ins().global_value(environ.pointer_type(), gv);
@@ -367,8 +367,7 @@ pub fn translate_operator(
          * argument referring to an index in the external functions table of the module.
          ************************************************************************************/
         Operator::Call { function_index } => {
-            let (fref, num_args) =
-                state.get_direct_func(&mut builder.func, *function_index, environ)?;
+            let (fref, num_args) = state.get_direct_func(builder.func, *function_index, environ)?;
             let call = environ.translate_call(
                 builder.cursor(),
                 FuncIndex::from_u32(*function_index),
@@ -389,8 +388,8 @@ pub fn translate_operator(
         Operator::CallIndirect { index, table_index } => {
             // `index` is the index of the function's signature and `table_index` is the index of
             // the table to search the function in.
-            let (sigref, num_args) = state.get_indirect_sig(&mut builder.func, *index, environ)?;
-            let table = state.get_table(&mut builder.func, *table_index, environ)?;
+            let (sigref, num_args) = state.get_indirect_sig(builder.func, *index, environ)?;
+            let table = state.get_table(builder.func, *table_index, environ)?;
             let callee = state.pop1();
             let call = environ.translate_call_indirect(
                 builder.cursor(),
@@ -418,13 +417,13 @@ pub fn translate_operator(
             // The WebAssembly MVP only supports one linear memory, but we expect the reserved
             // argument to be a memory index.
             let heap_index = MemoryIndex::from_u32(*reserved);
-            let heap = state.get_heap(&mut builder.func, *reserved, environ)?;
+            let heap = state.get_heap(builder.func, *reserved, environ)?;
             let val = state.pop1();
             state.push1(environ.translate_memory_grow(builder.cursor(), heap_index, heap, val)?)
         }
         Operator::MemorySize { reserved } => {
             let heap_index = MemoryIndex::from_u32(*reserved);
-            let heap = state.get_heap(&mut builder.func, *reserved, environ)?;
+            let heap = state.get_heap(builder.func, *reserved, environ)?;
             state.push1(environ.translate_memory_size(builder.cursor(), heap_index, heap)?);
         }
         /******************************* Load instructions ***********************************
@@ -1238,7 +1237,7 @@ fn translate_load(
 ) -> WasmResult<()> {
     let addr32 = state.pop1();
     // We don't yet support multiple linear memories.
-    let heap = state.get_heap(&mut builder.func, 0, environ)?;
+    let heap = state.get_heap(builder.func, 0, environ)?;
     let (base, offset) = get_heap_addr(heap, addr32, offset, environ.pointer_type(), builder);
     // Note that we don't set `is_aligned` here, even if the load instruction's
     // alignment immediate says it's aligned, because WebAssembly's immediate
@@ -1263,7 +1262,7 @@ fn translate_store(
     let val_ty = builder.func.dfg.value_type(val);
 
     // We don't yet support multiple linear memories.
-    let heap = state.get_heap(&mut builder.func, 0, environ)?;
+    let heap = state.get_heap(builder.func, 0, environ)?;
     let (base, offset) = get_heap_addr(heap, addr32, offset, environ.pointer_type(), builder);
     // See the comments in `translate_load` about the flags.
     let flags = MemFlags::new();
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index 98cdb3f44e..fde1e6b19d 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -543,7 +543,8 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
                 func.collect_debug_info();
             }
             self.trans
-                .translate(body_bytes, body_offset, func, &mut func_environ)?
+                .translate(body_bytes, body_offset, &mut func, &mut func_environ)?;
+            func
         };
         self.func_bytecode_sizes.push(body_bytes.len());
         self.info.function_bodies.push(func);
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index ed5c51e59f..b49cffd474 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -12,7 +12,7 @@ use crate::wasm_unsupported;
 use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir::{self, Ebb, InstBuilder, ValueLabel};
 use cranelift_codegen::timing;
-use cranelift_frontend::{FunctionBuilder, Variable};
+use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
 use log::info;
 use wasmparser::{self, BinaryReader};
 
@@ -22,7 +22,7 @@ use wasmparser::{self, BinaryReader};
 /// by a `FuncEnvironment` object. A single translator instance can be reused to translate multiple
 /// functions which will reduce heap allocation traffic.
 pub struct FuncTranslator {
-    builder: FunctionBuilder,
+    func_ctx: FunctionBuilderContext,
     state: TranslationState,
 }
 
@@ -30,7 +30,7 @@ impl FuncTranslator {
     /// Create a new translator.
     pub fn new() -> Self {
         Self {
-            builder: FunctionBuilder::new(ir::Function::new()),
+            func_ctx: FunctionBuilderContext::new(),
             state: TranslationState::new(),
         }
     }
@@ -57,9 +57,9 @@ impl FuncTranslator {
         &mut self,
         code: &[u8],
         code_offset: usize,
-        func: ir::Function,
+        func: &mut ir::Function,
         environ: &mut FE,
-    ) -> WasmResult {
+    ) -> WasmResult<()> {
         self.translate_from_reader(
             BinaryReader::new_with_offset(code, code_offset),
             func,
@@ -71,9 +71,9 @@ impl FuncTranslator {
     pub fn translate_from_reader(
         &mut self,
         mut reader: BinaryReader,
-        func: ir::Function,
+        func: &mut ir::Function,
         environ: &mut FE,
-    ) -> WasmResult {
+    ) -> WasmResult<()> {
         let _tt = timing::wasm_translate_function();
         info!(
             "translate({} bytes, {}{})",
@@ -84,32 +84,31 @@ impl FuncTranslator {
         debug_assert_eq!(func.dfg.num_ebbs(), 0, "Function must be empty");
         debug_assert_eq!(func.dfg.num_insts(), 0, "Function must be empty");
 
-        self.builder.func = func;
-        self.builder.set_srcloc(cur_srcloc(&reader));
-        let entry_block = self.builder.create_ebb();
-        self.builder
-            .append_ebb_params_for_function_params(entry_block);
-        self.builder.switch_to_block(entry_block); // This also creates values for the arguments.
-        self.builder.seal_block(entry_block); // Declare all predecessors known.
+        // This clears the `FunctionBuilderContext`.
+        let mut builder = FunctionBuilder::new(func, &mut self.func_ctx);
+        builder.set_srcloc(cur_srcloc(&reader));
+        let entry_block = builder.create_ebb();
+        builder.append_ebb_params_for_function_params(entry_block);
+        builder.switch_to_block(entry_block); // This also creates values for the arguments.
+        builder.seal_block(entry_block); // Declare all predecessors known.
 
         // Make sure the entry block is inserted in the layout before we make any callbacks to
         // `environ`. The callback functions may need to insert things in the entry block.
-        self.builder.ensure_inserted_ebb();
+        builder.ensure_inserted_ebb();
 
-        let num_params = declare_wasm_parameters(&mut self.builder, entry_block);
+        let num_params = declare_wasm_parameters(&mut builder, entry_block);
 
         // Set up the translation state with a single pushed control block representing the whole
         // function and its return values.
-        let exit_block = self.builder.create_ebb();
-        self.builder
-            .append_ebb_params_for_function_returns(exit_block);
-        self.state
-            .initialize(&self.builder.func.signature, exit_block);
+        let exit_block = builder.create_ebb();
+        builder.append_ebb_params_for_function_returns(exit_block);
+        self.state.initialize(&builder.func.signature, exit_block);
 
-        parse_local_decls(&mut reader, &mut self.builder, num_params, environ)?;
-        parse_function_body(reader, &mut self.builder, &mut self.state, environ)?;
+        parse_local_decls(&mut reader, &mut builder, num_params, environ)?;
+        parse_function_body(reader, &mut builder, &mut self.state, environ)?;
 
-        Ok(self.builder.finalize())
+        builder.finalize();
+        Ok(())
     }
 }
 
@@ -289,8 +288,8 @@ mod tests {
         ctx.func.signature.params.push(ir::AbiParam::new(I32));
         ctx.func.signature.returns.push(ir::AbiParam::new(I32));
 
-        ctx.func = trans
-            .translate(&BODY, 0, ctx.func, &mut runtime.func_env())
+        trans
+            .translate(&BODY, 0, &mut ctx.func, &mut runtime.func_env())
             .unwrap();
         debug!("{}", ctx.func.display(None));
         ctx.verify(&flags).unwrap();
@@ -328,8 +327,8 @@ mod tests {
         ctx.func.signature.params.push(ir::AbiParam::new(I32));
         ctx.func.signature.returns.push(ir::AbiParam::new(I32));
 
-        ctx.func = trans
-            .translate(&BODY, 0, ctx.func, &mut runtime.func_env())
+        trans
+            .translate(&BODY, 0, &mut ctx.func, &mut runtime.func_env())
             .unwrap();
         debug!("{}", ctx.func.display(None));
         ctx.verify(&flags).unwrap();
@@ -375,8 +374,8 @@ mod tests {
         ctx.func.name = ir::ExternalName::testcase("infloop");
         ctx.func.signature.returns.push(ir::AbiParam::new(I32));
 
-        ctx.func = trans
-            .translate(&BODY, 0, ctx.func, &mut runtime.func_env())
+        trans
+            .translate(&BODY, 0, &mut ctx.func, &mut runtime.func_env())
             .unwrap();
         debug!("{}", ctx.func.display(None));
         ctx.verify(&flags).unwrap();

From 863ac809d9968ba0ba2fe9765cbe98acd4bae2de Mon Sep 17 00:00:00 2001
From: Wander Lairson Costa 
Date: Mon, 16 Sep 2019 18:59:23 -0300
Subject: [PATCH 2719/3084] [codegen] Check for downcasting in bitcast
 instruction

Bitcasting to a smaller size is invalid.

closes #854
---
 cranelift/codegen/src/verifier/mod.rs         | 29 +++++++++++++++++++
 .../filetests/filetests/verifier/bitcast.clif | 23 +++++++++++++++
 2 files changed, 52 insertions(+)
 create mode 100644 cranelift/filetests/filetests/verifier/bitcast.clif

diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 5bc10d9d92..2244947f0f 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -696,6 +696,13 @@ impl<'a> Verifier<'a> {
                 }
             }
 
+            Unary {
+                opcode: Opcode::Bitcast,
+                arg,
+            } => {
+                self.verify_bitcast(inst, arg, errors)?;
+            }
+
             // Exhaustive list so we can't forget to add new formats
             Unary { .. }
             | UnaryImm { .. }
@@ -981,6 +988,28 @@ impl<'a> Verifier<'a> {
         }
     }
 
+    fn verify_bitcast(
+        &self,
+        inst: Inst,
+        arg: Value,
+        errors: &mut VerifierErrors,
+    ) -> VerifierStepResult<()> {
+        let typ = self.func.dfg.ctrl_typevar(inst);
+        let value_type = self.func.dfg.value_type(arg);
+
+        if typ.lane_bits() < value_type.lane_bits() {
+            fatal!(
+                errors,
+                inst,
+                "The bitcast argument {} doesn't fit in a type of {} bits",
+                arg,
+                typ.lane_bits()
+            )
+        } else {
+            Ok(())
+        }
+    }
+
     fn domtree_integrity(
         &self,
         domtree: &DominatorTree,
diff --git a/cranelift/filetests/filetests/verifier/bitcast.clif b/cranelift/filetests/filetests/verifier/bitcast.clif
new file mode 100644
index 0000000000..eb5303cfc0
--- /dev/null
+++ b/cranelift/filetests/filetests/verifier/bitcast.clif
@@ -0,0 +1,23 @@
+test verifier
+
+; bitcast between two types of equal size if ok
+function %valid_bitcast1(i32) -> f32 { ; Ok
+ebb0(v0: i32):
+    v1 = bitcast.f32 v0
+    return v1
+}
+
+; bitcast to a type larger than the operand is ok
+function %valid_bitcast2(i32) -> i64 { ; Ok
+ebb0(v0: i32):
+    v1 = bitcast.i64 v0
+    return v1
+}
+
+; bitcast to a smaller type is not ok
+function %bad_bitcast(i64) -> i32 {
+ebb0(v0: i64):
+    v1 = bitcast.i32 v0 ; error: The bitcast argument v0 doesn't fit in a type of 32 bits
+    return v1
+}
+

From 9b852fde093b458a95ca14161f2419d35d119ed9 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 13 Sep 2019 16:41:46 -0700
Subject: [PATCH 2720/3084] Add verifier check to ensure each SIMD lane index
 is within bounds, fixes #1016

---
 cranelift/codegen/src/verifier/mod.rs         | 54 ++++++++++++++-----
 .../isa/x86/extractlane-binemit.clif          |  4 +-
 .../filetests/verifier/simd-lane-index.clif   | 41 ++++++++++++++
 3 files changed, 83 insertions(+), 16 deletions(-)
 create mode 100644 cranelift/filetests/filetests/verifier/simd-lane-index.clif

diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 2244947f0f..c76090e57d 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -1775,21 +1775,47 @@ impl<'a> Verifier<'a> {
     ) -> VerifierStepResult<()> {
         let inst_data = &self.func.dfg[inst];
 
-        // If this is some sort of a store instruction, get the memflags, else, just return.
-        let memflags = match *inst_data {
+        match *inst_data {
             ir::InstructionData::Store { flags, .. }
-            | ir::InstructionData::StoreComplex { flags, .. } => flags,
-            _ => return Ok(()),
-        };
-
-        if memflags.readonly() {
-            fatal!(
-                errors,
-                inst,
-                "A store instruction cannot have the `readonly` MemFlag"
-            )
-        } else {
-            Ok(())
+            | ir::InstructionData::StoreComplex { flags, .. } => {
+                if flags.readonly() {
+                    fatal!(
+                        errors,
+                        inst,
+                        "A store instruction cannot have the `readonly` MemFlag"
+                    )
+                } else {
+                    Ok(())
+                }
+            }
+            ir::InstructionData::ExtractLane {
+                opcode: ir::instructions::Opcode::Extractlane,
+                lane,
+                arg,
+                ..
+            }
+            | ir::InstructionData::InsertLane {
+                opcode: ir::instructions::Opcode::Insertlane,
+                lane,
+                args: [arg, _],
+                ..
+            } => {
+                // We must be specific about the opcodes above because other instructions are using
+                // the ExtractLane/InsertLane formats.
+                let ty = self.func.dfg.value_type(arg);
+                if u16::from(lane) >= ty.lane_count() {
+                    fatal!(
+                        errors,
+                        inst,
+                        "The lane {} does not index into the type {}",
+                        lane,
+                        ty
+                    )
+                } else {
+                    Ok(())
+                }
+            }
+            _ => Ok(()),
         }
     }
 
diff --git a/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif b/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif
index 0a3b776a99..86f16315cf 100644
--- a/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif
@@ -2,8 +2,8 @@ test binemit
 set enable_simd
 target x86_64 haswell
 
-; for extractlane, floats are legalized differently than integers and booleans; integers and booleans use x86_pextr
-; which is manually placed in the IR so that it can be binemit-tested
+; for extractlane, floats are legalized differently than integers and booleans; integers and
+; booleans use x86_pextr which is manually placed in the IR so that it can be binemit-tested
 
 function %test_extractlane_b8() {
 ebb0:
diff --git a/cranelift/filetests/filetests/verifier/simd-lane-index.clif b/cranelift/filetests/filetests/verifier/simd-lane-index.clif
new file mode 100644
index 0000000000..064254c0e2
--- /dev/null
+++ b/cranelift/filetests/filetests/verifier/simd-lane-index.clif
@@ -0,0 +1,41 @@
+test verifier
+set enable_simd
+target x86_64
+
+function %insertlane_i32x4() {
+ebb0:
+    v0 = vconst.i32x4 [0 0 0 0]
+    v1 = iconst.i32 42
+    v2 = insertlane v0, 4, v1 ; error: The lane 4 does not index into the type i32x4
+    return
+}
+
+function %insertlane_b16x8() {
+ebb0:
+    v0 = vconst.b16x8 [false false false false false false false false]
+    v1 = bconst.b16 true
+    v2 = insertlane v0, 8, v1 ; error: The lane 8 does not index into the type b16x8
+    return
+}
+
+function %insertlane_f64x2() {
+ebb0:
+    v0 = vconst.f64x2 0x00
+    v1 = f64const 0x0.1
+    v2 = insertlane v0, 2, v1 ; error: The lane 2 does not index into the type f64x2
+    return
+}
+
+function %extractlane_i32x4() {
+ebb0:
+    v0 = vconst.i32x4 [0 0 0 0]
+    v1 = extractlane v0, 4 ; error: The lane 4 does not index into the type i32x4
+    return
+}
+
+function %extractlane_b8x16() {
+ebb0:
+    v0 = vconst.b8x16 0x00
+    v1 = extractlane v0, 16 ; error: The lane 16 does not index into the type b8x16
+    return
+}

From 156938facf1c31d001ff272cc259460513162279 Mon Sep 17 00:00:00 2001
From: bookmoons <35854232+bookmoons@users.noreply.github.com>
Date: Wed, 18 Sep 2019 01:35:30 -0400
Subject: [PATCH 2721/3084] Continuous fuzzing with Fuzzit (#1042)

Enables automated fuzzing on Fuzzit. Runs fuzz regression tests
every push and PR. Runs full fuzzing every push. Fuzzit emails
if it finds crashes.

Uses the existing fuzz targets:
* translate-module - Fuzz valid WebAssembly modules.
* reader-parse - Fuzz IR text format parsing.
---
 .azure-pipelines.yml                     | 24 +++++++++++++++
 cranelift/README.md                      |  1 +
 cranelift/ci/fuzzit.sh                   | 38 ++++++++++++++++++++++++
 cranelift/fuzz/fuzz_reader_parse_test.rs |  3 +-
 4 files changed, 65 insertions(+), 1 deletion(-)
 create mode 100755 cranelift/ci/fuzzit.sh

diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml
index a18a45652e..9e0a4697eb 100644
--- a/.azure-pipelines.yml
+++ b/.azure-pipelines.yml
@@ -97,6 +97,30 @@ jobs:
         RUST_BACKTRACE: 1
       condition: and(succeeded(), eq(variables['toolchain'], 'nightly'))
 
+- job: Fuzz_regression
+  displayName: Fuzz regression
+  pool:
+    vmImage: "ubuntu-16.04"
+  variables:
+    toolchain: nightly
+  steps:
+    - template: ci/azure-install-rust.yml
+    - bash: cargo install cargo-fuzz
+    - bash: ci/fuzzit.sh local-regression
+
+- job: Fuzz
+  condition: ne(variables['Build.Reason'], 'PullRequest')
+  pool:
+    vmImage: "ubuntu-16.04"
+  variables:
+    toolchain: nightly
+  steps:
+    - template: ci/azure-install-rust.yml
+    - bash: cargo install cargo-fuzz
+    - bash: ci/fuzzit.sh fuzzing
+      env:
+        FUZZIT_API_KEY: $(FUZZIT_API_KEY)
+
 - job: Build
   strategy:
     matrix:
diff --git a/cranelift/README.md b/cranelift/README.md
index 3bfa2dc6cd..171055d030 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -8,6 +8,7 @@ into executable machine code.
 
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
 [![Travis Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
+[![Fuzzit Status](https://app.fuzzit.dev/badge?org_id=CraneStation)](https://app.fuzzit.dev/orgs/CraneStation/dashboard)
 [![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
 ![Minimum rustc 1.37](https://img.shields.io/badge/rustc-1.37+-green.svg)
 
diff --git a/cranelift/ci/fuzzit.sh b/cranelift/ci/fuzzit.sh
new file mode 100755
index 0000000000..c08fb3c64e
--- /dev/null
+++ b/cranelift/ci/fuzzit.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+set -xe
+
+# Validate arguments
+if [ "$#" -ne 1 ]; then
+    cat << EOF
+Usage: $0 
+
+Types are:
+local-regression - Run corpus and past crashes locally to catch regressions.
+fuzzing - Submit for long run fuzzing on Fuzzit.
+EOF
+    exit 1
+fi
+
+# Configure
+set -xe
+NAME=cranelift
+TYPE=$1
+FUZZIT_VERSION=2.4.46
+
+# Setup
+if [[ ! -f fuzzit || ! `./fuzzit --version` =~ $FUZZIT_VERSION$ ]]; then
+    wget -q -O fuzzit https://github.com/fuzzitdev/fuzzit/releases/download/v$FUZZIT_VERSION/fuzzit_Linux_x86_64
+    chmod a+x fuzzit
+fi
+./fuzzit --version
+
+# Fuzz
+function fuzz {
+    FUZZER=$1
+    TARGET=$2
+    cargo fuzz run $FUZZER -- -runs=0
+    ./fuzzit --version
+    ./fuzzit create job --type $TYPE $NAME/$TARGET ./fuzz/target/x86_64-unknown-linux-gnu/debug/$FUZZER
+}
+fuzz fuzz_translate_module translate-module
+fuzz fuzz_reader_parse_test reader-parse
diff --git a/cranelift/fuzz/fuzz_reader_parse_test.rs b/cranelift/fuzz/fuzz_reader_parse_test.rs
index cccf135916..4f17898264 100644
--- a/cranelift/fuzz/fuzz_reader_parse_test.rs
+++ b/cranelift/fuzz/fuzz_reader_parse_test.rs
@@ -6,6 +6,7 @@ use std::str;
 
 fuzz_target!(|data: &[u8]| {
     if let Ok(s) = str::from_utf8(data) {
-        let _ = cranelift_reader::parse_test(s, None, None);
+        let options = cranelift_reader::ParseOptions::default();
+        let _ = cranelift_reader::parse_test(s, options);
     }
 });

From add6a4f2690ab94cdd8a3c66ca935b0351985542 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 18 Sep 2019 10:06:15 +0200
Subject: [PATCH 2722/3084] Correctly zero extend operand of fcvt_from_uint for
 8ints and 16bit ints (#997)

Fixes #996
---
 cranelift/codegen/meta/src/shared/legalize.rs    | 16 +++-------------
 cranelift/codegen/src/isa/x86/enc_tables.rs      | 16 ++++++++++------
 .../filetests/isa/x86/saturating-float-cast.clif | 13 +++++++++++++
 3 files changed, 26 insertions(+), 19 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/saturating-float-cast.clif

diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index f9cec3557e..30de84a198 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -64,7 +64,6 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     let f64const = insts.by_name("f64const");
     let fcopysign = insts.by_name("fcopysign");
     let fcvt_from_sint = insts.by_name("fcvt_from_sint");
-    let fcvt_from_uint = insts.by_name("fcvt_from_uint");
     let fneg = insts.by_name("fneg");
     let iadd = insts.by_name("iadd");
     let iadd_cin = insts.by_name("iadd_cin");
@@ -542,21 +541,12 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
         ],
     );
 
-    // Expansions for fcvt_from_{u,s}int for smaller integer types.
-    // These use expand and not widen because the controlling type variable for
-    // these instructions are f32/f64, which are legalized as part of the expand
+    // Expansion for fcvt_from_sint for smaller integer types.
+    // This uses expand and not widen because the controlling type variable for
+    // this instruction is f32/f64, which is legalized as part of the expand
     // group.
     for &dest_ty in &[F32, F64] {
         for &src_ty in &[I8, I16] {
-            let bound_inst = fcvt_from_uint.bind(dest_ty).bind(src_ty);
-            expand.legalize(
-                def!(a = bound_inst(b)),
-                vec![
-                    def!(x = uextend.I32(b)),
-                    def!(a = fcvt_from_uint.dest_ty(x)),
-                ],
-            );
-
             let bound_inst = fcvt_from_sint.bind(dest_ty).bind(src_ty);
             expand.legalize(
                 def!(a = bound_inst(b)),
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index f67d7f0b69..732fcd9628 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -406,12 +406,16 @@ fn expand_fcvt_from_uint(
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
 
-    // Conversion from unsigned 32-bit is easy on x86-64.
-    // TODO: This should be guarded by an ISA check.
-    if xty == ir::types::I32 {
-        let wide = pos.ins().uextend(ir::types::I64, x);
-        pos.func.dfg.replace(inst).fcvt_from_sint(ty, wide);
-        return;
+    // Conversion from an unsigned int smaller than 64bit is easy on x86-64.
+    match xty {
+        ir::types::I8 | ir::types::I16 | ir::types::I32 => {
+            // TODO: This should be guarded by an ISA check.
+            let wide = pos.ins().uextend(ir::types::I64, x);
+            pos.func.dfg.replace(inst).fcvt_from_sint(ty, wide);
+            return;
+        }
+        ir::types::I64 => {}
+        _ => unimplemented!(),
     }
 
     let old_ebb = pos.func.layout.pp_ebb(inst);
diff --git a/cranelift/filetests/filetests/isa/x86/saturating-float-cast.clif b/cranelift/filetests/filetests/isa/x86/saturating-float-cast.clif
new file mode 100644
index 0000000000..5986e1f864
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/saturating-float-cast.clif
@@ -0,0 +1,13 @@
+test compile
+target x86_64
+
+function u0:0() -> f32 system_v {
+ebb0:
+    v0 = iconst.i8 255
+; check: v2 = iconst.i32 255
+; nextln: v0 = ireduce.i8 v2
+    v1 = fcvt_from_uint.f32 v0
+; nextln: v3 = uextend.i64 v0
+; nextln: v1 = fcvt_from_sint.f32 v3
+    return v1
+}

From c0274eca24291f85a309300e2733cbd80b5c1969 Mon Sep 17 00:00:00 2001
From: AnthonyMikh 
Date: Wed, 18 Sep 2019 11:11:32 +0300
Subject: [PATCH 2723/3084] Simplify and rename `Stackmap::from_vec` (#1032)

Co-Authored-By: bjorn3 
---
 cranelift/codegen/src/binemit/stackmap.rs | 49 ++++++++++++-----------
 1 file changed, 25 insertions(+), 24 deletions(-)

diff --git a/cranelift/codegen/src/binemit/stackmap.rs b/cranelift/codegen/src/binemit/stackmap.rs
index 8d0d2a3204..4051b47e19 100644
--- a/cranelift/codegen/src/binemit/stackmap.rs
+++ b/cranelift/codegen/src/binemit/stackmap.rs
@@ -3,10 +3,13 @@ use crate::ir;
 use crate::isa::TargetIsa;
 use std::vec::Vec;
 
+type Num = u32;
+const NUM_BITS: usize = std::mem::size_of::() * 8;
+
 /// Wrapper class for longer bit vectors that cannot be represented by a single BitSet.
 #[derive(Clone, Debug)]
 pub struct Stackmap {
-    bitmap: Vec>,
+    bitmap: Vec>,
 }
 
 impl Stackmap {
@@ -52,34 +55,32 @@ impl Stackmap {
             }
         }
 
-        Stackmap::from_vec(&vec)
+        Stackmap::from_slice(&vec)
     }
 
-    /// Create a vec of Bitsets from a vec of bools.
-    pub fn from_vec(vec: &Vec) -> Self {
-        let mut rem = vec.len();
-        let num_word = ((rem as f32) / 32.0).ceil() as usize;
+    /// Create a vec of Bitsets from a slice of bools.
+    pub fn from_slice(vec: &[bool]) -> Self {
+        let len = vec.len();
+        let num_word = len / NUM_BITS + (len % NUM_BITS != 0) as usize;
         let mut bitmap = Vec::with_capacity(num_word);
 
-        for i in 0..num_word {
+        for segment in vec.chunks(NUM_BITS) {
             let mut curr_word = 0;
-            let count = if rem > 32 { 32 } else { rem };
-            for j in 0..count {
-                if vec[i * 32 + j] {
-                    curr_word |= 1 << j;
+            for (i, set) in segment.iter().enumerate() {
+                if *set {
+                    curr_word |= 1 << i;
                 }
             }
-            bitmap.push(BitSet::(curr_word));
-            rem -= count;
+            bitmap.push(BitSet(curr_word));
         }
         Self { bitmap }
     }
 
     /// Returns a specified bit.
     pub fn get_bit(&self, bit_index: usize) -> bool {
-        assert!(bit_index < 32 * self.bitmap.len());
-        let word_index = bit_index / 32;
-        let word_offset = (bit_index % 32) as u8;
+        assert!(bit_index < NUM_BITS * self.bitmap.len());
+        let word_index = bit_index / NUM_BITS;
+        let word_offset = (bit_index % NUM_BITS) as u8;
         self.bitmap[word_index].contains(word_offset)
     }
 }
@@ -91,26 +92,26 @@ mod tests {
     #[test]
     fn stackmaps() {
         let vec: Vec = Vec::new();
-        assert!(Stackmap::from_vec(&vec).bitmap.is_empty());
+        assert!(Stackmap::from_slice(&vec).bitmap.is_empty());
 
-        let mut vec: [bool; 32] = Default::default();
+        let mut vec: [bool; NUM_BITS] = Default::default();
         let set_true_idx = [5, 7, 24, 31];
 
-        for idx in set_true_idx.iter() {
-            vec[*idx] = true;
+        for &idx in &set_true_idx {
+            vec[idx] = true;
         }
 
         let mut vec = vec.to_vec();
         assert_eq!(
-            vec![BitSet::(2164261024)],
-            Stackmap::from_vec(&vec).bitmap
+            vec![BitSet::(2164261024)],
+            Stackmap::from_slice(&vec).bitmap
         );
 
         vec.push(false);
         vec.push(true);
-        let res = Stackmap::from_vec(&vec);
+        let res = Stackmap::from_slice(&vec);
         assert_eq!(
-            vec![BitSet::(2164261024), BitSet::(2)],
+            vec![BitSet::(2164261024), BitSet::(2)],
             res.bitmap
         );
 

From 3c6795decf180e1befc6c0617e042a37a174f85c Mon Sep 17 00:00:00 2001
From: Erin Power 
Date: Tue, 17 Sep 2019 18:53:38 +0200
Subject: [PATCH 2724/3084] Fixed broken pipe race condition in
 ensure_installed

---
 cranelift/test-all.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index 48e53116ea..d3ad831274 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -55,7 +55,7 @@ ensure_installed() {
     program="$1"
     toolchain="${2:-stable}"
     if has_toolchain $toolchain; then
-        if cargo +$toolchain install --list | grep -q $program; then
+        if grep -q $program <(cargo +$toolchain install --list); then
             echo "$program found"
         else
             echo "installing $program"

From e252e0f5b75ef41fb6b67f04ccd6361f7aa48ea0 Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Wed, 18 Sep 2019 05:14:29 -0400
Subject: [PATCH 2725/3084] fix broken link in cranelift-object

---
 cranelift/object/src/backend.rs | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/cranelift/object/src/backend.rs b/cranelift/object/src/backend.rs
index 78dc246324..09f209bb3f 100644
--- a/cranelift/object/src/backend.rs
+++ b/cranelift/object/src/backend.rs
@@ -37,8 +37,7 @@ pub struct ObjectBuilder {
 
 impl ObjectBuilder {
     /// Create a new `ObjectBuilder` using the given Cranelift target, that
-    /// can be passed to
-    /// [`Module::new`](cranelift_module/struct.Module.html#method.new].
+    /// can be passed to [`Module::new`](cranelift_module::Module::new).
     ///
     /// `collect_traps` setting determines whether trap information is collected in the
     /// `ObjectProduct`.

From 59139c6c468920f69378f9d256da2c70f1ca2feb Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Wed, 18 Sep 2019 19:48:57 -0400
Subject: [PATCH 2726/3084] Add more documentation for entities, fixes #1038
 (#1041)

Also fixes broken links to point to docs.rs; see
https://github.com/CraneStation/cranelift/pull/1041#issuecomment-532356880,
there doesn't currently seem to be a way to link to a type in a
dependent crate.
---
 cranelift/codegen/src/ir/entities.rs | 123 +++++++++++++++++++++++++--
 1 file changed, 114 insertions(+), 9 deletions(-)

diff --git a/cranelift/codegen/src/ir/entities.rs b/cranelift/codegen/src/ir/entities.rs
index ab78e0f3a8..1e09d18840 100644
--- a/cranelift/codegen/src/ir/entities.rs
+++ b/cranelift/codegen/src/ir/entities.rs
@@ -25,7 +25,12 @@ use core::u32;
 #[cfg(feature = "enable-serde")]
 use serde::{Deserialize, Serialize};
 
-/// An opaque reference to an extended basic block in a function.
+/// An opaque reference to an [extended basic
+/// block](https://en.wikipedia.org/wiki/Extended_basic_block) in a
+/// [`Function`](super::function::Function).
+///
+/// You can get an `Ebb` using
+/// [`FunctionBuilder::create_ebb`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.create_ebb)
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct Ebb(u32);
 entity_impl!(Ebb, "ebb");
@@ -44,6 +49,18 @@ impl Ebb {
 }
 
 /// An opaque reference to an SSA value.
+///
+/// You can get a constant `Value` from the following
+/// [`InstBuilder`](super::InstBuilder) instructions:
+///
+/// - [`iconst`](super::InstBuilder::iconst) for integer constants
+/// - [`f32const`](super::InstBuilder::f32const) for 32-bit float constants
+/// - [`f64const`](super::InstBuilder::f64const) for 64-bit float constants
+/// - [`bconst`](super::InstBuilder::bconst) for boolean constants
+/// - [`vconst`](super::InstBuilder::vconst) for vector constants
+/// - [`null`](super::InstBuilder::null) for null reference constants
+///
+/// Any `InstBuilder` instruction that has an output will also return a `Value`.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct Value(u32);
 entity_impl!(Value, "v");
@@ -62,12 +79,34 @@ impl Value {
     }
 }
 
-/// An opaque reference to an instruction in a function.
+/// An opaque reference to an instruction in a [`Function`](super::Function).
+///
+/// Most usage of `Inst` is internal. `Inst`ructions are returned by
+/// [`InstBuilder`](super::InstBuilder) instructions that do not return a
+/// [`Value`], such as control flow and trap instructions.
+///
+/// If you look around the API, you can find many inventive uses for `Inst`,
+/// such as [annotating specific instructions with a comment][inst_comment]
+/// or [performing reflection at compile time](super::DataFlowGraph::analyze_branch)
+/// on the type of instruction.
+///
+/// [inst_comment]: https://github.com/bjorn3/rustc_codegen_cranelift/blob/0f8814fd6da3d436a90549d4bb19b94034f2b19c/src/pretty_clif.rs
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct Inst(u32);
 entity_impl!(Inst, "inst");
 
 /// An opaque reference to a stack slot.
+///
+/// Stack slots represent an address on the
+/// [call stack](https://en.wikipedia.org/wiki/Call_stack).
+///
+/// `StackSlot`s can be created with
+/// [`FunctionBuilder::create_stackslot`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.create_stack_slot).
+///
+/// `StackSlot`s are most often used with
+/// [`stack_addr`](super::InstBuilder::stack_addr),
+/// [`stack_load`](super::InstBuilder::stack_load), and
+/// [`stack_store`](super::InstBuilder::stack_store).
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
 #[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct StackSlot(u32);
@@ -87,6 +126,22 @@ impl StackSlot {
 }
 
 /// An opaque reference to a global value.
+///
+/// A `GlobalValue` is a [`Value`](Value) that will be live across the entire
+/// function lifetime. It can be preloaded from other global values.
+///
+/// You can create a `GlobalValue` in the following ways:
+///
+/// - When compiling to WASM, you can use it to load values from a
+/// [`VmContext`](super::GlobalValueData::VMContext) using
+/// [`FuncEnvironment::make_global`](https://docs.rs/cranelift-wasm/*/cranelift_wasm/trait.FuncEnvironment.html#tymethod.make_global).
+/// - When compiling to native code, you can use it for objects in static memory with
+/// [`Module::declare_data_in_func`](https://docs.rs/cranelift-module/*/cranelift_module/struct.Module.html#method.declare_data_in_func).
+/// - For any compilation target, it can be registered with
+/// [`FunctionBuilder::create_global_value`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.create_global_value).
+///
+/// `GlobalValue`s can be retrieved with
+/// [`InstBuilder:global_value`](super::InstBuilder::global_value).
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
 pub struct GlobalValue(u32);
 entity_impl!(GlobalValue, "gv");
@@ -104,7 +159,11 @@ impl GlobalValue {
     }
 }
 
-/// An opaque reference to a constant
+/// An opaque reference to a constant.
+///
+/// You can store [`ConstantData`](super::ConstantData) in a
+/// [`ConstantPool`](super::ConstantPool) for efficient storage and retrieval.
+/// See [`ConstantPool::insert`](super::ConstantPool::insert).
 #[derive(Copy, Clone, PartialEq, Eq, Hash, Ord, PartialOrd)]
 pub struct Constant(u32);
 entity_impl!(Constant, "const");
@@ -122,7 +181,16 @@ impl Constant {
     }
 }
 
-/// An opaque reference to a jump table.
+/// An opaque reference to a [jump table](https://en.wikipedia.org/wiki/Branch_table).
+///
+/// `JumpTable`s are used for indirect branching and are specialized for dense,
+/// 0-based jump offsets. If you want a jump table which doesn't start at 0,
+/// or is not contiguous, consider using a [`Switch`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.Switch.html) instead.
+///
+/// `JumpTable` are used with [`br_table`](super::InstBuilder::br_table).
+///
+/// `JumpTable`s can be created with
+/// [`create_jump_table`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.create_jump_table).
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
 #[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct JumpTable(u32);
@@ -141,7 +209,22 @@ impl JumpTable {
     }
 }
 
-/// A reference to an external function.
+/// An opaque reference to another [`Function`](super::Function).
+///
+/// `FuncRef`s are used for [direct](super::InstBuilder::call) function calls
+/// and by [`func_addr`](super::InstBuilder::func_addr) for use in
+/// [indirect](super::InstBuilder::call_indirect) function calls.
+///
+/// `FuncRef`s can be created with
+///
+/// - [`FunctionBuilder::import_function`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.import_function)
+/// for external functions
+/// - [`Module::declare_func_in_func`](https://docs.rs/cranelift-module/*/cranelift_module/struct.Module.html#method.declare_func_in_func)
+/// for functions declared elsewhere in the same native
+/// [`Module`](https://docs.rs/cranelift-module/*/cranelift_module/struct.Module.html)
+/// - [`FuncEnvironment::make_direct_func`](https://docs.rs/cranelift-wasm/*/cranelift_wasm/trait.FuncEnvironment.html#tymethod.make_direct_func)
+/// for functions declared in the same WebAssembly
+/// [`FuncEnvironment`](https://docs.rs/cranelift-wasm/*/cranelift_wasm/trait.FuncEnvironment.html#tymethod.make_direct_func)
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
 pub struct FuncRef(u32);
 entity_impl!(FuncRef, "fn");
@@ -159,7 +242,18 @@ impl FuncRef {
     }
 }
 
-/// A reference to a function signature.
+/// An opaque reference to a function [`Signature`](super::Signature).
+///
+/// `SigRef`s are used to declare a function with
+/// [`FunctionBuiler::import_function`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.import_function)
+/// as well as to make an [indirect function call](super::InstBuilder::call_indirect).
+///
+/// `SigRef`s can be created with
+/// [`FunctionBuilder::import_signature`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.import_signature).
+///
+/// You can retrieve the [`Signature`](super::Signature) that was used to create a `SigRef` with
+/// [`FunctionBuilder::signature`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.signature) or
+/// [`func.dfg.signatures`](super::dfg::DataFlowGraph::signatures).
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
 pub struct SigRef(u32);
 entity_impl!(SigRef, "sig");
@@ -177,7 +271,12 @@ impl SigRef {
     }
 }
 
-/// A reference to a heap.
+/// An opaque reference to a [heap](https://en.wikipedia.org/wiki/Memory_management#DYNAMIC).
+///
+/// Heaps are used to access dynamically allocated memory through
+/// [`heap_addr`](super::InstBuilder::heap_addr).
+///
+/// To create a heap, use [`FunctionBuilder::create_heap`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.create_heap).
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
 pub struct Heap(u32);
 entity_impl!(Heap, "heap");
@@ -195,7 +294,13 @@ impl Heap {
     }
 }
 
-/// A reference to a table.
+/// An opaque reference to a [WebAssembly
+/// table](https://developer.mozilla.org/en-US/docs/WebAssembly/Understanding_the_text_format#WebAssembly_tables).
+///
+/// `Table`s are used to store a list of function references.
+/// They can be created with [`FuncEnvironment::make_table`](https://docs.rs/cranelift-wasm/*/cranelift_wasm/trait.FuncEnvironment.html#tymethod.make_table).
+/// They can be used with
+/// [`FuncEnvironment::translate_call_indirect`](https://docs.rs/cranelift-wasm/*/cranelift_wasm/trait.FuncEnvironment.html#tymethod.translate_call_indirect).
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
 pub struct Table(u32);
 entity_impl!(Table, "table");
@@ -213,7 +318,7 @@ impl Table {
     }
 }
 
-/// A reference to any of the entities defined in this module that can appear in CLIF IR.
+/// An opaque reference to any of the entities defined in this module that can appear in CLIF IR.
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
 pub enum AnyEntity {
     /// The whole function.

From 9e088e41640bf03b019df245e3c5d92b38ee0ac1 Mon Sep 17 00:00:00 2001
From: julian-seward1 
Date: Thu, 19 Sep 2019 18:51:25 +0200
Subject: [PATCH 2727/3084] Reorganise optimisation level settings, and make
 the insn shrink pass optional (#1044)

This patch:

* removes the "default" opt level, on the basis that it has no definition and
  is referred to nowhere in the compiler.

* renames the "fastest" level to "none".  The resulting set of transformations
  is unchanged.

* renames the "best" level to "speed_and_size".  The resulting set of
  transformations is unchanged.

* adds a new level, "speed".  This is the same as "speed_and_size" except that
  it omits transformations aimed only at reducing code size.  Currently it
  omits only the insn shrinking pass.
---
 cranelift/codegen/meta/src/shared/settings.rs    |  9 +++++----
 cranelift/codegen/src/context.rs                 | 14 ++++++++------
 cranelift/codegen/src/settings.rs                | 16 +++++++++-------
 .../filetests/isa/x86/allones_funcaddrs32.clif   |  2 +-
 .../filetests/isa/x86/allones_funcaddrs64.clif   |  2 +-
 .../x86/baseline_clz_ctz_popcount_encoding.clif  |  2 +-
 .../filetests/filetests/isa/x86/binary32.clif    |  2 +-
 .../filetests/isa/x86/binary64-float.clif        |  2 +-
 .../filetests/isa/x86/binary64-pic.clif          |  2 +-
 .../filetests/filetests/isa/x86/binary64.clif    |  2 +-
 .../filetests/filetests/isa/x86/isub_imm-i8.clif |  1 +
 .../filetests/isa/x86/legalize-br-table.clif     |  1 +
 .../filetests/isa/x86/legalize-call.clif         |  2 +-
 .../isa/x86/optimized-zero-constants-32bit.clif  |  2 +-
 .../isa/x86/optimized-zero-constants.clif        |  2 +-
 .../filetests/filetests/isa/x86/pinned-reg.clif  |  2 +-
 .../filetests/isa/x86/prologue-epilogue.clif     |  2 +-
 .../filetests/isa/x86/relax_branch.clif          |  2 +-
 .../isa/x86/scalar_to_vector-binemit.clif        |  2 +-
 .../isa/x86/scalar_to_vector-compile.clif        |  2 +-
 .../filetests/isa/x86/shrink-multiple-uses.clif  |  2 +-
 .../filetests/filetests/isa/x86/shrink.clif      |  2 +-
 .../filetests/isa/x86/stack-addr64.clif          |  2 +-
 .../filetests/isa/x86/stack-load-store64.clif    |  2 +-
 .../filetests/isa/x86/vconst-binemit.clif        |  2 +-
 .../filetests/isa/x86/windows_fastcall_x64.clif  |  2 +-
 cranelift/filetests/src/test_binemit.rs          |  2 +-
 27 files changed, 46 insertions(+), 39 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
index 63c5d3c01a..43fe357ab7 100644
--- a/cranelift/codegen/meta/src/shared/settings.rs
+++ b/cranelift/codegen/meta/src/shared/settings.rs
@@ -8,11 +8,12 @@ pub fn define() -> SettingGroup {
         r#"
         Optimization level:
 
-        - default: Very profitable optimizations enabled, none slow.
-        - best: Enable all optimizations
-        - fastest: Optimize for compile time by disabling most optimizations.
+        - none: Minimise compile time by disabling most optimizations.
+        - speed: Generate the fastest possible code
+        - speed_and_size: like "speed", but also perform transformations
+          aimed at reducing code size.
         "#,
-        vec!["default", "best", "fastest"],
+        vec!["none", "speed", "speed_and_size"],
     );
 
     settings.add_bool(
diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index 0f4e9ad420..8bbe2893de 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -132,18 +132,18 @@ impl Context {
         self.verify_if(isa)?;
         debug!("Compiling:\n{}", self.func.display(isa));
 
+        let opt_level = isa.flags().opt_level();
+
         self.compute_cfg();
-        if isa.flags().opt_level() != OptLevel::Fastest {
+        if opt_level != OptLevel::None {
             self.preopt(isa)?;
         }
         if isa.flags().enable_nan_canonicalization() {
             self.canonicalize_nans(isa)?;
         }
         self.legalize(isa)?;
-        if isa.flags().opt_level() != OptLevel::Fastest {
+        if opt_level != OptLevel::None {
             self.postopt(isa)?;
-        }
-        if isa.flags().opt_level() == OptLevel::Best {
             self.compute_domtree();
             self.compute_loop_analysis();
             self.licm(isa)?;
@@ -151,13 +151,15 @@ impl Context {
         }
         self.compute_domtree();
         self.eliminate_unreachable_code(isa)?;
-        if isa.flags().opt_level() != OptLevel::Fastest {
+        if opt_level != OptLevel::None {
             self.dce(isa)?;
         }
         self.regalloc(isa)?;
         self.prologue_epilogue(isa)?;
-        if isa.flags().opt_level() == OptLevel::Best {
+        if opt_level == OptLevel::Speed || opt_level == OptLevel::SpeedAndSize {
             self.redundant_reload_remover(isa)?;
+        }
+        if opt_level == OptLevel::SpeedAndSize {
             self.shrink_instructions(isa)?;
         }
         let result = self.relax_branches(isa);
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index 99d3647cbd..c310972170 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -14,10 +14,10 @@
 //! use cranelift_codegen::settings::{self, Configurable};
 //!
 //! let mut b = settings::builder();
-//! b.set("opt_level", "fastest");
+//! b.set("opt_level", "speed_and_size");
 //!
 //! let f = settings::Flags::new(b);
-//! assert_eq!(f.opt_level(), settings::OptLevel::Fastest);
+//! assert_eq!(f.opt_level(), settings::OptLevel::SpeedAndSize);
 //! ```
 
 use crate::constant_hash::{probe, simple_hash};
@@ -378,7 +378,7 @@ mod tests {
         assert_eq!(
             f.to_string(),
             "[shared]\n\
-             opt_level = \"default\"\n\
+             opt_level = \"none\"\n\
              libcall_call_conv = \"isa_default\"\n\
              baldrdash_prologue_words = 0\n\
              probestack_size_log2 = 12\n\
@@ -398,7 +398,7 @@ mod tests {
              probestack_func_adjusts_sp = false\n\
              jump_tables_enabled = true\n"
         );
-        assert_eq!(f.opt_level(), super::OptLevel::Default);
+        assert_eq!(f.opt_level(), super::OptLevel::None);
         assert_eq!(f.enable_simd(), false);
         assert_eq!(f.baldrdash_prologue_words(), 0);
     }
@@ -428,13 +428,15 @@ mod tests {
         );
         assert_eq!(
             b.set("opt_level", "true"),
-            Err(BadValue("any among default, best, fastest".to_string()))
+            Err(BadValue(
+                "any among none, speed, speed_and_size".to_string()
+            ))
         );
-        assert_eq!(b.set("opt_level", "best"), Ok(()));
+        assert_eq!(b.set("opt_level", "speed"), Ok(()));
         assert_eq!(b.set("enable_simd", "0"), Ok(()));
 
         let f = Flags::new(b);
         assert_eq!(f.enable_simd(), false);
-        assert_eq!(f.opt_level(), super::OptLevel::Best);
+        assert_eq!(f.opt_level(), super::OptLevel::Speed);
     }
 }
diff --git a/cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif
index b6d7542fc3..5a6a5b9708 100644
--- a/cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif
+++ b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif
@@ -1,6 +1,6 @@
 ; binary emission of 32-bit code.
 test binemit
-set opt_level=best
+set opt_level=speed_and_size
 set allones_funcaddrs
 target i686 haswell
 
diff --git a/cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif
index 06ab24ed80..617db5a445 100644
--- a/cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif
+++ b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif
@@ -1,6 +1,6 @@
 ; binary emission of 64-bit code.
 test binemit
-set opt_level=best
+set opt_level=speed_and_size
 set allones_funcaddrs
 target x86_64 haswell
 
diff --git a/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif b/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif
index b2a0c9617f..31c4016dc0 100644
--- a/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif
+++ b/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif
@@ -1,5 +1,5 @@
 test binemit
-set opt_level=best
+set opt_level=speed_and_size
 target x86_64 baseline
 
 ; The binary encodings can be verified with the command:
diff --git a/cranelift/filetests/filetests/isa/x86/binary32.clif b/cranelift/filetests/filetests/isa/x86/binary32.clif
index aa50cf5335..cc8f638014 100644
--- a/cranelift/filetests/filetests/isa/x86/binary32.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary32.clif
@@ -1,6 +1,6 @@
 ; binary emission of x86-32 code.
 test binemit
-set opt_level=best
+set opt_level=speed_and_size
 target i686 haswell
 
 ; The binary encodings can be verified with the command:
diff --git a/cranelift/filetests/filetests/isa/x86/binary64-float.clif b/cranelift/filetests/filetests/isa/x86/binary64-float.clif
index 831079070d..171a3db7c9 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64-float.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64-float.clif
@@ -1,6 +1,6 @@
 ; Binary emission of 64-bit floating point code.
 test binemit
-set opt_level=best
+set opt_level=speed_and_size
 target x86_64 haswell
 
 ; The binary encodings can be verified with the command:
diff --git a/cranelift/filetests/filetests/isa/x86/binary64-pic.clif b/cranelift/filetests/filetests/isa/x86/binary64-pic.clif
index adda09da70..3f3d86288c 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64-pic.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64-pic.clif
@@ -1,6 +1,6 @@
 ; binary emission of 64-bit code.
 test binemit
-set opt_level=best
+set opt_level=speed_and_size
 set is_pic
 target x86_64 haswell
 
diff --git a/cranelift/filetests/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
index 9b1f86beff..0430cd78a5 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64.clif
@@ -1,6 +1,6 @@
 ; binary emission of x86-64 code.
 test binemit
-set opt_level=best
+set opt_level=speed_and_size
 target x86_64 haswell
 
 ; The binary encodings can be verified with the command:
diff --git a/cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif b/cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif
index 1ca70ebbbe..35698c9abc 100644
--- a/cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif
@@ -1,4 +1,5 @@
 test compile
+set opt_level=speed_and_size
 target x86_64
 
 function u0:0(i8) -> i8 fast {
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
index 63c24b0240..51ab2d08d5 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
@@ -1,4 +1,5 @@
 test compile
+set opt_level=speed_and_size
 target x86_64
 feature !"basic-blocks"
 ; regex: V=v\d+
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-call.clif b/cranelift/filetests/filetests/isa/x86/legalize-call.clif
index b66e7e3a53..240b075374 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-call.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-call.clif
@@ -1,6 +1,6 @@
 ; Test legalization of a non-colocated call in 64-bit non-PIC mode.
 test legalizer
-set opt_level=best
+set opt_level=speed_and_size
 target x86_64 haswell
 
 function %call() {
diff --git a/cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif
index 21f936c4b9..7dbbcc86e0 100644
--- a/cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif
+++ b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif
@@ -1,6 +1,6 @@
 ; Check that floating-point and integer constants equal to zero are optimized correctly.
 test binemit
-set opt_level=best
+set opt_level=speed_and_size
 target i686
 
 function %foo() -> f32 fast {
diff --git a/cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif
index 4a1ad00ff4..807466e84c 100644
--- a/cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif
+++ b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif
@@ -1,6 +1,6 @@
 ; Check that floating-point constants equal to zero are optimized correctly.
 test binemit
-set opt_level=best
+set opt_level=speed_and_size
 target x86_64
 
 function %zero_const_32bit_no_rex() -> f32 fast {
diff --git a/cranelift/filetests/filetests/isa/x86/pinned-reg.clif b/cranelift/filetests/filetests/isa/x86/pinned-reg.clif
index b8a16d4eb4..2a447a6d9d 100644
--- a/cranelift/filetests/filetests/isa/x86/pinned-reg.clif
+++ b/cranelift/filetests/filetests/isa/x86/pinned-reg.clif
@@ -2,7 +2,7 @@ test compile
 
 set enable_pinned_reg=true
 set use_pinned_reg_as_heap_base=true
-set opt_level=best
+set opt_level=speed_and_size
 
 target x86_64
 
diff --git a/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif b/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
index ec67358804..f2fd3c68ee 100644
--- a/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
+++ b/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
@@ -1,5 +1,5 @@
 test compile
-set opt_level=best
+set opt_level=speed_and_size
 set is_pic
 target x86_64 haswell
 
diff --git a/cranelift/filetests/filetests/isa/x86/relax_branch.clif b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
index 7b86aef0e0..15c7e876a3 100644
--- a/cranelift/filetests/filetests/isa/x86/relax_branch.clif
+++ b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
@@ -1,5 +1,5 @@
 test binemit
-set opt_level=best
+set opt_level=speed_and_size
 set avoid_div_traps
 set baldrdash_prologue_words=3
 set allones_funcaddrs
diff --git a/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif
index b26f3d2e6b..149d54b03f 100644
--- a/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif
@@ -1,5 +1,5 @@
 test binemit
-set opt_level=best
+set opt_level=speed_and_size
 set enable_simd
 target x86_64
 
diff --git a/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif
index 2d2ab331f7..6db3d12f40 100644
--- a/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif
+++ b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif
@@ -1,5 +1,5 @@
 test compile
-set opt_level=best
+set opt_level=speed_and_size
 set probestack_enabled=false
 set enable_simd
 target x86_64
diff --git a/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif b/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
index c0a34f2ecc..358d098a6a 100644
--- a/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
+++ b/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
@@ -1,5 +1,5 @@
 test shrink
-set opt_level=best
+set opt_level=speed_and_size
 target x86_64
 
 function %test_multiple_uses(i32 [%rdi]) -> i32 {
diff --git a/cranelift/filetests/filetests/isa/x86/shrink.clif b/cranelift/filetests/filetests/isa/x86/shrink.clif
index f0d78af226..b0d3174ece 100644
--- a/cranelift/filetests/filetests/isa/x86/shrink.clif
+++ b/cranelift/filetests/filetests/isa/x86/shrink.clif
@@ -1,5 +1,5 @@
 test binemit
-set opt_level=best
+set opt_level=speed_and_size
 target x86_64
 
 ; Test that instruction shrinking eliminates REX prefixes when possible.
diff --git a/cranelift/filetests/filetests/isa/x86/stack-addr64.clif b/cranelift/filetests/filetests/isa/x86/stack-addr64.clif
index a333f2cd5d..c80d190907 100644
--- a/cranelift/filetests/filetests/isa/x86/stack-addr64.clif
+++ b/cranelift/filetests/filetests/isa/x86/stack-addr64.clif
@@ -1,6 +1,6 @@
 ; binary emission of stack address instructions on x86-64.
 test binemit
-set opt_level=fastest
+set opt_level=none
 target x86_64 haswell
 
 ; The binary encodings can be verified with the command:
diff --git a/cranelift/filetests/filetests/isa/x86/stack-load-store64.clif b/cranelift/filetests/filetests/isa/x86/stack-load-store64.clif
index c1854e623a..3c0e2c8c0e 100644
--- a/cranelift/filetests/filetests/isa/x86/stack-load-store64.clif
+++ b/cranelift/filetests/filetests/isa/x86/stack-load-store64.clif
@@ -1,6 +1,6 @@
 ; legalization of stack load and store instructions on x86-64.
 test legalizer
-set opt_level=fastest
+set opt_level=none
 target x86_64 haswell
 
 function %stack_load_and_store() {
diff --git a/cranelift/filetests/filetests/isa/x86/vconst-binemit.clif b/cranelift/filetests/filetests/isa/x86/vconst-binemit.clif
index f7b9ce4627..b07dc0fd4e 100644
--- a/cranelift/filetests/filetests/isa/x86/vconst-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/vconst-binemit.clif
@@ -1,5 +1,5 @@
 test binemit
-set opt_level=best
+set opt_level=speed_and_size
 set enable_simd
 target x86_64
 
diff --git a/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
index 14b46e579f..a621abfe9f 100644
--- a/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
+++ b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
@@ -1,5 +1,5 @@
 test compile
-set opt_level=best
+set opt_level=speed_and_size
 set is_pic
 target x86_64 haswell
 
diff --git a/cranelift/filetests/src/test_binemit.rs b/cranelift/filetests/src/test_binemit.rs
index 75bd86008f..b291f61446 100644
--- a/cranelift/filetests/src/test_binemit.rs
+++ b/cranelift/filetests/src/test_binemit.rs
@@ -162,7 +162,7 @@ impl SubTest for TestBinEmit {
                                 recipe_constraints.satisfied(inst, &divert, &func)
                             });
 
-                        if opt_level == OptLevel::Best {
+                        if opt_level == OptLevel::SpeedAndSize {
                             // Get the smallest legal encoding
                             legal_encodings
                                 .min_by_key(|&e| encinfo.byte_size(e, inst, &divert, &func))

From af1499ce99f099a2eb044399fa1e6062478a8731 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 26 Aug 2019 14:50:05 -0700
Subject: [PATCH 2728/3084] Add x86 implementation of shuffle

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 11 ++-
 .../codegen/meta/src/isa/x86/legalize.rs      |  2 +
 cranelift/codegen/meta/src/isa/x86/recipes.rs |  6 +-
 cranelift/codegen/meta/src/shared/formats.rs  |  8 +-
 .../codegen/meta/src/shared/immediates.rs     | 12 +++
 .../codegen/meta/src/shared/instructions.rs   | 37 +++++++++-
 cranelift/codegen/src/ir/dfg.rs               | 10 ++-
 cranelift/codegen/src/ir/entities.rs          | 23 ++++++
 cranelift/codegen/src/ir/mod.rs               |  3 +-
 cranelift/codegen/src/isa/x86/enc_tables.rs   | 74 +++++++++++++++++++
 cranelift/codegen/src/verifier/mod.rs         |  3 +-
 cranelift/codegen/src/write.rs                | 19 +++--
 .../filetests/isa/x86/shuffle-legalize.clif   | 31 ++++++++
 .../filetests/isa/x86/shuffle-run.clif        | 44 +++++++++++
 .../filetests/isa/x86/vconst-rodata.clif      |  1 -
 cranelift/reader/src/parser.rs                | 47 +++++++-----
 cranelift/serde/src/serde_clif_json.rs        | 36 ++++++---
 cranelift/wasm/src/code_translator.rs         | 13 +++-
 18 files changed, 336 insertions(+), 44 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/shuffle-legalize.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/shuffle-run.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 9bdb751b26..8824329c77 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1785,7 +1785,7 @@ pub(crate) fn define(
     let allowed_simd_type = |t: &LaneType| t.lane_bits() >= 8 && t.lane_bits() < 128;
 
     // PSHUFB, 8-bit shuffle using two XMM registers.
-    for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
+    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         let instruction = x86_pshufb.bind_vector_from_lane(ty, sse_vector_size);
         let template = rec_fa.nonrex().opcodes(vec![0x66, 0x0f, 0x38, 00]);
         e.enc32_isap(instruction.clone(), template.clone(), use_ssse3_simd);
@@ -1804,7 +1804,7 @@ pub(crate) fn define(
 
     // SIMD scalar_to_vector; this uses MOV to copy the scalar value to an XMM register; according
     // to the Intel manual: "When the destination operand is an XMM register, the source operand is
-    // written to the low doubleword of the register and the regiser is zero-extended to 128 bits."
+    // written to the low doubleword of the register and the register is zero-extended to 128 bits."
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         let instruction = scalar_to_vector.bind_vector_from_lane(ty, sse_vector_size);
         if ty.is_float() {
@@ -1929,6 +1929,13 @@ pub(crate) fn define(
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
     }
 
+    // SIMD bor using ORPS
+    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
+        let instruction = bor.bind_vector_from_lane(ty, sse_vector_size);
+        let template = rec_fa.nonrex().opcodes(vec![0x0f, 0x56]);
+        e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
+    }
+
     // Reference type instructions
 
     // Null references implemented as iconst 0.
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index dfd7f84334..e37759e892 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -45,6 +45,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let selectif = insts.by_name("selectif");
     let smulhi = insts.by_name("smulhi");
     let splat = insts.by_name("splat");
+    let shuffle = insts.by_name("shuffle");
     let srem = insts.by_name("srem");
     let udiv = insts.by_name("udiv");
     let umulhi = insts.by_name("umulhi");
@@ -380,6 +381,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         );
     }
 
+    narrow.custom_legalize(shuffle, "convert_shuffle");
     narrow.custom_legalize(extractlane, "convert_extractlane");
     narrow.custom_legalize(insertlane, "convert_insertlane");
 
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 11a9972d98..bee51883a4 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -396,11 +396,11 @@ pub(crate) fn define<'shared>(
     let f_trap = formats.by_name("Trap");
     let f_unary = formats.by_name("Unary");
     let f_unary_bool = formats.by_name("UnaryBool");
+    let f_unary_const = formats.by_name("UnaryConst");
     let f_unary_global_value = formats.by_name("UnaryGlobalValue");
     let f_unary_ieee32 = formats.by_name("UnaryIeee32");
     let f_unary_ieee64 = formats.by_name("UnaryIeee64");
     let f_unary_imm = formats.by_name("UnaryImm");
-    let f_unary_imm128 = formats.by_name("UnaryImm128");
 
     // Predicates shorthands.
     let use_sse41 = settings.predicate_by_name("use_sse41");
@@ -2437,14 +2437,14 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("vconst", f_unary_imm128, 5)
+        EncodingRecipeBuilder::new("vconst", f_unary_const, 5)
             .operands_out(vec![fpr])
             .clobbers_flags(false)
             .emit(
                 r#"
                     {{PUT_OP}}(bits, rex2(0, out_reg0), sink);
                     modrm_riprel(out_reg0, sink);
-                    const_disp4(imm, func, sink);
+                    const_disp4(constant_handle, func, sink);
                 "#,
             ),
     );
diff --git a/cranelift/codegen/meta/src/shared/formats.rs b/cranelift/codegen/meta/src/shared/formats.rs
index 5309afc1b0..74a2f6bb8f 100644
--- a/cranelift/codegen/meta/src/shared/formats.rs
+++ b/cranelift/codegen/meta/src/shared/formats.rs
@@ -6,10 +6,10 @@ pub(crate) fn define(imm: &Immediates, entities: &EntityRefs) -> FormatRegistry
 
     registry.insert(Builder::new("Unary").value());
     registry.insert(Builder::new("UnaryImm").imm(&imm.imm64));
-    registry.insert(Builder::new("UnaryImm128").imm(&imm.uimm128));
     registry.insert(Builder::new("UnaryIeee32").imm(&imm.ieee32));
     registry.insert(Builder::new("UnaryIeee64").imm(&imm.ieee64));
     registry.insert(Builder::new("UnaryBool").imm(&imm.boolean));
+    registry.insert(Builder::new("UnaryConst").imm(&imm.pool_constant));
     registry.insert(Builder::new("UnaryGlobalValue").imm(&entities.global_value));
 
     registry.insert(Builder::new("Binary").value().value());
@@ -43,6 +43,12 @@ pub(crate) fn define(imm: &Immediates, entities: &EntityRefs) -> FormatRegistry
             .value()
             .imm_with_name("lane", &imm.uimm8),
     );
+    registry.insert(
+        Builder::new("Shuffle")
+            .value()
+            .value()
+            .imm_with_name("mask", &imm.uimm128),
+    );
 
     registry.insert(Builder::new("IntCompare").imm(&imm.intcc).value().value());
     registry.insert(
diff --git a/cranelift/codegen/meta/src/shared/immediates.rs b/cranelift/codegen/meta/src/shared/immediates.rs
index 30c1a73970..0b5e84c521 100644
--- a/cranelift/codegen/meta/src/shared/immediates.rs
+++ b/cranelift/codegen/meta/src/shared/immediates.rs
@@ -23,6 +23,12 @@ pub(crate) struct Immediates {
     /// const.
     pub uimm128: OperandKind,
 
+    /// A constant stored in the constant pool.
+    ///
+    /// This operand is used to pass constants to instructions like vconst while storing the
+    /// actual bytes in the constant pool.
+    pub pool_constant: OperandKind,
+
     /// A 32-bit immediate signed offset.
     ///
     /// This is used to represent an immediate address offset in load/store instructions.
@@ -84,6 +90,12 @@ impl Immediates {
 
             uimm128: Builder::new_imm("uimm128")
                 .doc("A 128-bit immediate unsigned integer.")
+                .rust_type("ir::Immediate")
+                .build(),
+
+            pool_constant: Builder::new_imm("poolConstant")
+                .doc("A constant stored in the constant pool.")
+                .default_member("constant_handle")
                 .rust_type("ir::Constant")
                 .build(),
 
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index bae1327b60..b52a8dafed 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1090,7 +1090,7 @@ pub(crate) fn define(
 
     let N = &operand_doc(
         "N",
-        &imm.uimm128,
+        &imm.pool_constant,
         "The 16 immediate bytes of a 128-bit vector",
     );
     let a = &operand_doc("a", TxN, "A constant vector value");
@@ -1108,6 +1108,41 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    let mask = &operand_doc(
+        "mask",
+        &imm.uimm128,
+        "The 16 immediate bytes used for selecting the elements to shuffle",
+    );
+    let Tx16 = &TypeVar::new(
+        "Tx16",
+        "A SIMD vector with exactly 16 lanes of 8-bit values; eventually this may support other \
+         lane counts and widths",
+        TypeSetBuilder::new()
+            .ints(8..8)
+            .bools(8..8)
+            .simd_lanes(16..16)
+            .includes_scalars(false)
+            .build(),
+    );
+    let a = &operand_doc("a", Tx16, "A vector value");
+    let b = &operand_doc("b", Tx16, "A vector value");
+
+    ig.push(
+        Inst::new(
+            "shuffle",
+            r#"
+        SIMD vector shuffle.
+        
+        Shuffle two vectors using the given immediate bytes. For each of the 16 bytes of the
+        immediate, a value i of 0-15 selects the i-th element of the first vector and a value i of 
+        16-31 selects the (i-16)th element of the second vector. Immediate values outside of the 
+        0-31 range place a 0 in the resulting vector lane.
+        "#,
+        )
+        .operands_in(vec![a, b, mask])
+        .operands_out(vec![a]),
+    );
+
     let a = &operand_doc("a", Ref, "A constant reference null value");
 
     ig.push(
diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index f4e2875952..5b3054b59d 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -5,7 +5,7 @@ use crate::ir;
 use crate::ir::builder::ReplaceBuilder;
 use crate::ir::extfunc::ExtFuncData;
 use crate::ir::instructions::{BranchInfo, CallInfo, InstructionData};
-use crate::ir::{types, ConstantPool};
+use crate::ir::{types, ConstantPool, Immediate};
 use crate::ir::{
     Ebb, FuncRef, Inst, SigRef, Signature, Type, Value, ValueLabelAssignments, ValueList,
     ValueListPool,
@@ -19,6 +19,7 @@ use core::mem;
 use core::ops::{Index, IndexMut};
 use core::u16;
 use std::collections::HashMap;
+use std::vec::Vec;
 
 /// A data flow graph defines all instructions and extended basic blocks in a function as well as
 /// the data flow dependencies between them. The DFG also tracks values which can be either
@@ -70,6 +71,9 @@ pub struct DataFlowGraph {
 
     /// Constants used within the function
     pub constants: ConstantPool,
+
+    /// Stores large immediates that otherwise will not fit on InstructionData
+    pub immediates: PrimaryMap>,
 }
 
 impl DataFlowGraph {
@@ -85,6 +89,7 @@ impl DataFlowGraph {
             ext_funcs: PrimaryMap::new(),
             values_labels: None,
             constants: ConstantPool::new(),
+            immediates: PrimaryMap::new(),
         }
     }
 
@@ -98,7 +103,8 @@ impl DataFlowGraph {
         self.signatures.clear();
         self.ext_funcs.clear();
         self.values_labels = None;
-        self.constants.clear()
+        self.constants.clear();
+        self.immediates.clear();
     }
 
     /// Get the total number of instructions created in this function, whether they are currently
diff --git a/cranelift/codegen/src/ir/entities.rs b/cranelift/codegen/src/ir/entities.rs
index 1e09d18840..1f8e1fc6a2 100644
--- a/cranelift/codegen/src/ir/entities.rs
+++ b/cranelift/codegen/src/ir/entities.rs
@@ -181,6 +181,29 @@ impl Constant {
     }
 }
 
+/// An opaque reference to an immediate.
+///
+/// Some immediates (e.g. SIMD shuffle masks) are too large to store in the
+/// [`InstructionData`](super::instructions::InstructionData) struct and therefore must be
+/// tracked separately in [`DataFlowGraph::immediates`](super::dfg::DataFlowGraph). `Immediate`
+/// provides a way to reference values stored there.
+#[derive(Copy, Clone, PartialEq, Eq, Hash)]
+pub struct Immediate(u32);
+entity_impl!(Immediate, "imm");
+
+impl Immediate {
+    /// Create an immediate reference from its number.
+    ///
+    /// This method is for use by the parser.
+    pub fn with_number(n: u32) -> Option {
+        if n < u32::MAX {
+            Some(Immediate(n))
+        } else {
+            None
+        }
+    }
+}
+
 /// An opaque reference to a [jump table](https://en.wikipedia.org/wiki/Branch_table).
 ///
 /// `JumpTable`s are used for indirect branching and are specialized for dense,
diff --git a/cranelift/codegen/src/ir/mod.rs b/cranelift/codegen/src/ir/mod.rs
index 930f5d496d..2a0293b31b 100644
--- a/cranelift/codegen/src/ir/mod.rs
+++ b/cranelift/codegen/src/ir/mod.rs
@@ -31,7 +31,8 @@ pub use crate::ir::builder::{InsertBuilder, InstBuilder, InstBuilderBase, InstIn
 pub use crate::ir::constant::{ConstantData, ConstantOffset, ConstantPool};
 pub use crate::ir::dfg::{DataFlowGraph, ValueDef};
 pub use crate::ir::entities::{
-    Constant, Ebb, FuncRef, GlobalValue, Heap, Inst, JumpTable, SigRef, StackSlot, Table, Value,
+    Constant, Ebb, FuncRef, GlobalValue, Heap, Immediate, Inst, JumpTable, SigRef, StackSlot,
+    Table, Value,
 };
 pub use crate::ir::extfunc::{
     AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData, Signature,
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 732fcd9628..94333116e6 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -899,6 +899,80 @@ fn expand_fcvt_to_uint_sat(
     cfg.recompute_ebb(pos.func, done);
 }
 
+/// Convert shuffle instructions.
+fn convert_shuffle(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    _cfg: &mut ControlFlowGraph,
+    _isa: &dyn TargetIsa,
+) {
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+    pos.use_srcloc(inst);
+
+    if let ir::InstructionData::Shuffle { args, mask, .. } = pos.func.dfg[inst] {
+        // A mask-building helper: in 128-bit SIMD, 0-15 indicate which lane to read from and a 1
+        // in the most significant position zeroes the lane.
+        let zero_unknown_lane_index = |b: u8| if b > 15 { 0b10000000 } else { b };
+
+        // We only have to worry about aliasing here because copies will be introduced later (in
+        // regalloc).
+        let a = pos.func.dfg.resolve_aliases(args[0]);
+        let b = pos.func.dfg.resolve_aliases(args[1]);
+        let mask = pos
+            .func
+            .dfg
+            .immediates
+            .get(mask)
+            .expect("The shuffle immediate should have been recorded before this point")
+            .clone();
+        if a == b {
+            // PSHUFB the first argument (since it is the same as the second).
+            let constructed_mask = mask
+                .iter()
+                // If the mask is greater than 15 it still may be referring to a lane in b.
+                .map(|&b| if b > 15 { b.wrapping_sub(16) } else { b })
+                .map(zero_unknown_lane_index)
+                .collect();
+            let handle = pos.func.dfg.constants.insert(constructed_mask);
+            // Move the built mask into another XMM register.
+            let a_type = pos.func.dfg.value_type(a);
+            let mask_value = pos.ins().vconst(a_type, handle);
+            // Shuffle the single incoming argument.
+            pos.func.dfg.replace(inst).x86_pshufb(a, mask_value);
+        } else {
+            // PSHUFB the first argument, placing zeroes for unused lanes.
+            let constructed_mask = mask.iter().cloned().map(zero_unknown_lane_index).collect();
+            let handle = pos.func.dfg.constants.insert(constructed_mask);
+            // Move the built mask into another XMM register.
+            let a_type = pos.func.dfg.value_type(a);
+            let mask_value = pos.ins().vconst(a_type, handle);
+            // Shuffle the first argument.
+            let shuffled_first_arg = pos.ins().x86_pshufb(a, mask_value);
+
+            // PSHUFB the second argument, placing zeroes for unused lanes.
+            let constructed_mask = mask
+                .iter()
+                .map(|b| b.wrapping_sub(16))
+                .map(zero_unknown_lane_index)
+                .collect();
+            let handle = pos.func.dfg.constants.insert(constructed_mask);
+            // Move the built mask into another XMM register.
+            let b_type = pos.func.dfg.value_type(b);
+            let mask_value = pos.ins().vconst(b_type, handle);
+            // Shuffle the second argument.
+            let shuffled_second_arg = pos.ins().x86_pshufb(b, mask_value);
+
+            // OR the vectors together to form the final shuffled value.
+            pos.func
+                .dfg
+                .replace(inst)
+                .bor(shuffled_first_arg, shuffled_second_arg);
+
+            // TODO when AVX512 is enabled we should replace this sequence with a single VPERMB
+        };
+    }
+}
+
 /// Because floats already exist in XMM registers, we can keep them there when executing a CLIF
 /// extractlane instruction
 fn convert_extractlane(
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index c76090e57d..21f0c72cae 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -706,7 +706,6 @@ impl<'a> Verifier<'a> {
             // Exhaustive list so we can't forget to add new formats
             Unary { .. }
             | UnaryImm { .. }
-            | UnaryImm128 { .. }
             | UnaryIeee32 { .. }
             | UnaryIeee64 { .. }
             | UnaryBool { .. }
@@ -715,6 +714,8 @@ impl<'a> Verifier<'a> {
             | Ternary { .. }
             | InsertLane { .. }
             | ExtractLane { .. }
+            | UnaryConst { .. }
+            | Shuffle { .. }
             | IntCompare { .. }
             | IntCompareImm { .. }
             | IntCond { .. }
diff --git a/cranelift/codegen/src/write.rs b/cranelift/codegen/src/write.rs
index 4b616cab2e..e3c8bdb2fa 100644
--- a/cranelift/codegen/src/write.rs
+++ b/cranelift/codegen/src/write.rs
@@ -488,11 +488,6 @@ pub fn write_operands(
     match dfg[inst] {
         Unary { arg, .. } => write!(w, " {}", arg),
         UnaryImm { imm, .. } => write!(w, " {}", imm),
-        UnaryImm128 { imm, .. } => {
-            let data = dfg.constants.get(imm);
-            let uimm128 = Uimm128::from(&data[..]);
-            write!(w, " {}", uimm128)
-        }
         UnaryIeee32 { imm, .. } => write!(w, " {}", imm),
         UnaryIeee64 { imm, .. } => write!(w, " {}", imm),
         UnaryBool { imm, .. } => write!(w, " {}", imm),
@@ -510,6 +505,20 @@ pub fn write_operands(
         NullAry { .. } => write!(w, " "),
         InsertLane { lane, args, .. } => write!(w, " {}, {}, {}", args[0], lane, args[1]),
         ExtractLane { lane, arg, .. } => write!(w, " {}, {}", arg, lane),
+        UnaryConst {
+            constant_handle, ..
+        } => {
+            let data = dfg.constants.get(constant_handle);
+            let uimm128 = Uimm128::from(&data[..]);
+            write!(w, " {}", uimm128)
+        }
+        Shuffle { mask, args, .. } => {
+            let data = dfg.immediates.get(mask).expect(
+                "Expected the shuffle mask to already be inserted into the immediates table",
+            );
+            let uimm128 = Uimm128::from(&data[..]);
+            write!(w, " {}, {}, {}", args[0], args[1], uimm128)
+        }
         IntCompare { cond, args, .. } => write!(w, " {} {}, {}", cond, args[0], args[1]),
         IntCompareImm { cond, arg, imm, .. } => write!(w, " {} {}, {}", cond, arg, imm),
         IntCond { cond, arg, .. } => write!(w, " {} {}", cond, arg),
diff --git a/cranelift/filetests/filetests/isa/x86/shuffle-legalize.clif b/cranelift/filetests/filetests/isa/x86/shuffle-legalize.clif
new file mode 100644
index 0000000000..d192489448
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/shuffle-legalize.clif
@@ -0,0 +1,31 @@
+test legalizer
+set enable_simd
+target x86_64 skylake
+
+function %test_shuffle_different_ssa_values() -> i8x16 {
+ebb0:
+    v0 = vconst.i8x16 0x00
+    v1 = vconst.i8x16 0x01
+    v2 = shuffle v0, v1, 0x11000000000000000000000000000000     ; pick the second lane of v1, the rest use the first lane of v0
+    return v2
+}
+
+; check:  v1 = vconst.i8x16 0x01
+; nextln: v3 = vconst.i8x16 0x80000000000000000000000000000000
+; nextln: v4 = x86_pshufb v0, v3
+; nextln: v5 = vconst.i8x16 0x01808080808080808080808080808080
+; nextln: v6 = x86_pshufb v1, v5
+; nextln: v2 = bor v4, v6
+
+
+
+function %test_shuffle_same_ssa_value() -> i8x16 {
+ebb0:
+    v1 = vconst.i8x16 0x01
+    v2 = shuffle v1, v1, 0x13000000000000000000000000000000     ; pick the fourth lane of v1 and the rest from the first lane of v1
+    return v2
+}
+
+; check:  v1 = vconst.i8x16 0x01
+; nextln: v3 = vconst.i8x16 0x03000000000000000000000000000000
+; nextln: v2 = x86_pshufb v1, v3
diff --git a/cranelift/filetests/filetests/isa/x86/shuffle-run.clif b/cranelift/filetests/filetests/isa/x86/shuffle-run.clif
new file mode 100644
index 0000000000..60fd7d7b25
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/shuffle-run.clif
@@ -0,0 +1,44 @@
+test run
+set enable_simd
+
+function %test_shuffle_different_ssa_values() -> b1 {
+ebb0:
+    v0 = vconst.i8x16 0x00
+    v1 = vconst.i8x16 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 42]
+    v2 = shuffle v0, v1, [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 31]     ; use the first lane of v0 throughout except use the last lane of v1
+    v3 = extractlane.i8x16 v2, 15
+    v4 = iconst.i8 42
+    v5 = icmp eq v3, v4
+    return v5
+}
+
+; run
+
+function %test_shuffle_same_ssa_value() -> b1 {
+ebb0:
+    v0 = vconst.i8x16 0x01000000_00000000_00000000_00000000     ; note where lane 15 is when written with hexadecimal syntax
+    v1 = shuffle v0, v0, 0x0f0f0f0f_0f0f0f0f_0f0f0f0f_0f0f0f0f  ; use the last lane of v0 to fill all lanes
+    v2 = extractlane.i8x16 v1, 4
+    v3 = iconst.i8 0x01
+    v4 = icmp eq v2, v3
+    return v4
+}
+
+; run
+
+function %compare_shuffle() -> b1 {
+ebb0:
+    v1 = vconst.i32x4 [0 1 2 3]
+    v2 = raw_bitcast.i8x16  v1 ; we have to cast because shuffle is type-limited to Tx16
+    ; keep each lane in place from the first vector
+    v3 = shuffle v2, v2, [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15]
+    v4 = raw_bitcast.i32x4 v3
+    v5 = extractlane.i32x4 v4, 3
+    v6 = icmp_imm eq v5, 3
+    v7 = extractlane.i32x4 v4, 0
+    v8 = icmp_imm eq v7, 0
+    v9 = band v6, v8
+    return v9
+}
+
+; run
diff --git a/cranelift/filetests/filetests/isa/x86/vconst-rodata.clif b/cranelift/filetests/filetests/isa/x86/vconst-rodata.clif
index 99e8455ed4..34c203dce6 100644
--- a/cranelift/filetests/filetests/isa/x86/vconst-rodata.clif
+++ b/cranelift/filetests/filetests/isa/x86/vconst-rodata.clif
@@ -1,6 +1,5 @@
 test rodata
 set enable_simd=true
-set probestack_enabled=false
 target x86_64 haswell
 
 function %test_vconst_i32() -> i32x4 {
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 7965a4a31a..bc9436fe0c 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -2243,23 +2243,6 @@ impl<'a> Parser<'a> {
                 opcode,
                 imm: self.match_imm64("expected immediate integer operand")?,
             },
-            InstructionFormat::UnaryImm128 => match explicit_control_type {
-                None => {
-                    return err!(
-                        self.loc,
-                        "Expected {:?} to have a controlling type variable, e.g. inst.i32x4",
-                        opcode
-                    )
-                }
-                Some(ty) => {
-                    let uimm128 = self.match_uimm128_or_literals(ty)?;
-                    let constant_handle = ctx.function.dfg.constants.insert(uimm128.0.to_vec());
-                    InstructionData::UnaryImm128 {
-                        opcode,
-                        imm: constant_handle,
-                    }
-                }
-            },
             InstructionFormat::UnaryIeee32 => InstructionData::UnaryIeee32 {
                 opcode,
                 imm: self.match_ieee32("expected immediate 32-bit float operand")?,
@@ -2442,6 +2425,36 @@ impl<'a> Parser<'a> {
                 let lane = self.match_uimm8("expected lane number")?;
                 InstructionData::ExtractLane { opcode, lane, arg }
             }
+            InstructionFormat::UnaryConst => match explicit_control_type {
+                None => {
+                    return err!(
+                        self.loc,
+                        "Expected {:?} to have a controlling type variable, e.g. inst.i32x4",
+                        opcode
+                    )
+                }
+                Some(controlling_type) => {
+                    let uimm128 = self.match_uimm128_or_literals(controlling_type)?;
+                    let constant_handle = ctx.function.dfg.constants.insert(uimm128.to_vec());
+                    InstructionData::UnaryConst {
+                        opcode,
+                        constant_handle,
+                    }
+                }
+            },
+            InstructionFormat::Shuffle => {
+                let a = self.match_value("expected SSA value first operand")?;
+                self.match_token(Token::Comma, "expected ',' between operands")?;
+                let b = self.match_value("expected SSA value second operand")?;
+                self.match_token(Token::Comma, "expected ',' between operands")?;
+                let uimm128 = self.match_uimm128_or_literals(I8X16)?;
+                let mask = ctx.function.dfg.immediates.push(uimm128.to_vec());
+                InstructionData::Shuffle {
+                    opcode,
+                    mask,
+                    args: [a, b],
+                }
+            }
             InstructionFormat::IntCompare => {
                 let cond = self.match_enum("expected intcc condition code")?;
                 let lhs = self.match_value("expected SSA value first operand")?;
diff --git a/cranelift/serde/src/serde_clif_json.rs b/cranelift/serde/src/serde_clif_json.rs
index 90935e9234..0d19ee5fa0 100644
--- a/cranelift/serde/src/serde_clif_json.rs
+++ b/cranelift/serde/src/serde_clif_json.rs
@@ -1,4 +1,3 @@
-use cranelift_codegen::ir::immediates::Uimm128;
 use cranelift_codegen::ir::{Ebb, Function, Inst, InstructionData, Signature};
 use serde_derive::{Deserialize, Serialize};
 
@@ -59,6 +58,11 @@ pub enum SerInstData {
         arg: String,
         lane: String,
     },
+    Shuffle {
+        opcode: String,
+        args: [String; 2],
+        mask: String,
+    },
     IntCompare {
         opcode: String,
         args: [String; 2],
@@ -262,14 +266,6 @@ pub fn get_inst_data(inst_index: Inst, func: &Function) -> SerInstData {
             opcode: opcode.to_string(),
             imm: imm.to_string(),
         },
-        InstructionData::UnaryImm128 { opcode, imm } => {
-            let data = func.dfg.constants.get(imm);
-            let uimm128 = Uimm128::from(&data[..]);
-            SerInstData::UnaryImm {
-                opcode: opcode.to_string(),
-                imm: uimm128.to_string(),
-            }
-        }
         InstructionData::UnaryIeee32 { opcode, imm } => SerInstData::UnaryIeee32 {
             opcode: opcode.to_string(),
             imm: imm.to_string(),
@@ -340,6 +336,28 @@ pub fn get_inst_data(inst_index: Inst, func: &Function) -> SerInstData {
             arg: arg.to_string(),
             lane: lane.to_string(),
         },
+        InstructionData::UnaryConst {
+            opcode,
+            constant_handle,
+        } => {
+            let constant = func.dfg.constants.get(constant_handle);
+            SerInstData::UnaryImm {
+                opcode: opcode.to_string(),
+                imm: format!("{:?}", constant),
+            }
+        }
+        InstructionData::Shuffle { opcode, args, mask } => {
+            let mask = func
+                .dfg
+                .immediates
+                .get(mask)
+                .expect("Expected shuffle mask to already be inserted in immediate mapping");
+            SerInstData::Shuffle {
+                opcode: opcode.to_string(),
+                args: [args[0].to_string(), args[1].to_string()],
+                mask: format!("{:?}", mask),
+            }
+        }
         InstructionData::IntCompare { opcode, args, cond } => {
             let hold_args = [args[0].to_string(), args[1].to_string()];
             SerInstData::IntCompare {
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index c420147550..a7bb331a39 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -974,9 +974,20 @@ pub fn translate_operator(
                 builder,
             ))
         }
+        Operator::V8x16Shuffle { lanes, .. } => {
+            let (vector_a, vector_b) = state.pop2();
+            let a = optionally_bitcast_vector(vector_a, I8X16, builder);
+            let b = optionally_bitcast_vector(vector_b, I8X16, builder);
+            let mask = builder.func.dfg.immediates.push(lanes.to_vec());
+            let shuffled = builder.ins().shuffle(a, b, mask);
+            state.push1(shuffled)
+            // At this point the original types of a and b are lost; users of this value (i.e. this
+            // WASM-to-CLIF translator) may need to raw_bitcast for type-correctness. This is due
+            // to WASM using the less specific v128 type for certain operations and more specific
+            // types (e.g. i8x16) for others.
+        }
         Operator::V128Load { .. }
         | Operator::V128Store { .. }
-        | Operator::V8x16Shuffle { .. }
         | Operator::I8x16Eq
         | Operator::I8x16Ne
         | Operator::I8x16LtS

From e72434e58feb7379a3d08e0c73f67eb06b08e5b4 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 27 Aug 2019 16:03:56 -0700
Subject: [PATCH 2729/3084] Add boolean encodings for x86

Includes and, or, xor, not, and regmove; TODO re-factor PerCpuModeEncodings to avoid code duplication
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 23 +++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 8824329c77..9ea29fb932 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -150,6 +150,20 @@ impl PerCpuModeEncodings {
         self.enc64(inst.bind(I64), template.rex().w());
     }
 
+    /// Add encodings for `inst.b32` to X86_32.
+    /// Add encodings for `inst.b32` to X86_64 with and without REX.
+    /// Add encodings for `inst.b64` to X86_64 with a REX.W prefix.
+    fn enc_b32_b64(&mut self, inst: impl Into, template: Template) {
+        let inst: InstSpec = inst.into();
+        self.enc32(inst.bind(B32), template.nonrex());
+
+        // REX-less encoding must come after REX encoding so we don't use it by default. Otherwise
+        // reg-alloc would never use r8 and up.
+        self.enc64(inst.bind(B32), template.rex());
+        self.enc64(inst.bind(B32), template.nonrex());
+        self.enc64(inst.bind(B64), template.rex().w());
+    }
+
     /// Add encodings for `inst.i32` to X86_32.
     /// Add encodings for `inst.i32` to X86_64 with a REX prefix.
     /// Add encodings for `inst.i64` to X86_64 with a REX.W prefix.
@@ -658,11 +672,15 @@ pub(crate) fn define(
     e.enc_i32_i64(isub_ifborrow, rec_rio.opcodes(vec![0x19]));
 
     e.enc_i32_i64(band, rec_rr.opcodes(vec![0x21]));
+    e.enc_b32_b64(band, rec_rr.opcodes(vec![0x21]));
     e.enc_i32_i64(bor, rec_rr.opcodes(vec![0x09]));
+    e.enc_b32_b64(bor, rec_rr.opcodes(vec![0x09]));
     e.enc_i32_i64(bxor, rec_rr.opcodes(vec![0x31]));
+    e.enc_b32_b64(bxor, rec_rr.opcodes(vec![0x31]));
 
     // x86 has a bitwise not instruction NOT.
     e.enc_i32_i64(bnot, rec_ur.opcodes(vec![0xf7]).rrr(2));
+    e.enc_b32_b64(bnot, rec_ur.opcodes(vec![0xf7]).rrr(2));
 
     // Also add a `b1` encodings for the logic instructions.
     // TODO: Should this be done with 8-bit instructions? It would improve partial register
@@ -690,7 +708,12 @@ pub(crate) fn define(
         e.enc32(regmove.bind(ty), rec_rmov.opcodes(vec![0x89]));
         e.enc64(regmove.bind(ty), rec_rmov.opcodes(vec![0x89]).rex());
     }
+    for &ty in &[B8, B16, B32] {
+        e.enc32(regmove.bind(ty), rec_rmov.opcodes(vec![0x89]));
+        e.enc64(regmove.bind(ty), rec_rmov.opcodes(vec![0x89]).rex());
+    }
     e.enc64(regmove.bind(I64), rec_rmov.opcodes(vec![0x89]).rex().w());
+    e.enc64(regmove.bind(B64), rec_rmov.opcodes(vec![0x89]).rex().w());
     e.enc_both(regmove.bind(B1), rec_rmov.opcodes(vec![0x89]));
     e.enc_both(regmove.bind(I8), rec_rmov.opcodes(vec![0x89]));
     e.enc32(regmove.bind_ref(R32), rec_rmov.opcodes(vec![0x89]));

From 20c67f243e9a5ba2c1e691e82172afe3a2789236 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 27 Aug 2019 16:04:08 -0700
Subject: [PATCH 2730/3084] Add boolean shuffle test

---
 .../filetests/filetests/isa/x86/shuffle-run.clif | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/cranelift/filetests/filetests/isa/x86/shuffle-run.clif b/cranelift/filetests/filetests/isa/x86/shuffle-run.clif
index 60fd7d7b25..bc9eecb689 100644
--- a/cranelift/filetests/filetests/isa/x86/shuffle-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/shuffle-run.clif
@@ -42,3 +42,19 @@ ebb0:
 }
 
 ; run
+
+
+function %compare_shuffle() -> b32 {
+ebb0:
+    v1 = vconst.b32x4 [true false true false]
+    v2 = raw_bitcast.b8x16 v1 ; we have to cast because shuffle is type-limited to Tx16
+    ; pair up the true values to make the entire vector true
+    v3 = shuffle v2, v2, [0 1 2 3 0 1 2 3 8 9 10 11 8 9 10 11]
+    v4 = raw_bitcast.b32x4 v3
+    v5 = extractlane v4, 3
+    v6 = extractlane v4, 0
+    v7 = band v5, v6
+    return v7
+}
+
+; run

From 1e74d011110fd9ec5fa236404a3f9e8681eff0da Mon Sep 17 00:00:00 2001
From: Wander Lairson Costa 
Date: Thu, 19 Sep 2019 15:26:17 -0300
Subject: [PATCH 2731/3084] Check for types::INVALID in the function signature,
 resolves #493 (#1046)

---
 cranelift/codegen/src/verifier/mod.rs | 67 ++++++++++++++++++++++++++-
 1 file changed, 66 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 21f0c72cae..02cc3f0c52 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -1838,12 +1838,51 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
+    fn typecheck_function_signature(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
+        self.func
+            .signature
+            .params
+            .iter()
+            .enumerate()
+            .filter(|(_, ¶m)| param.value_type == types::INVALID)
+            .for_each(|(i, _)| {
+                report!(
+                    errors,
+                    AnyEntity::Function,
+                    "Parameter at position {} has an invalid type",
+                    i
+                );
+            });
+
+        self.func
+            .signature
+            .returns
+            .iter()
+            .enumerate()
+            .filter(|(_, &ret)| ret.value_type == types::INVALID)
+            .for_each(|(i, _)| {
+                report!(
+                    errors,
+                    AnyEntity::Function,
+                    "Return value at position {} has an invalid type",
+                    i
+                )
+            });
+
+        if errors.has_error() {
+            Err(())
+        } else {
+            Ok(())
+        }
+    }
+
     pub fn run(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         self.verify_global_values(errors)?;
         self.verify_heaps(errors)?;
         self.verify_tables(errors)?;
         self.verify_jump_tables(errors)?;
         self.typecheck_entry_block_params(errors)?;
+        self.typecheck_function_signature(errors)?;
 
         for ebb in self.func.layout.ebbs() {
             for inst in self.func.layout.ebb_insts(ebb) {
@@ -1870,7 +1909,7 @@ mod tests {
     use super::{Verifier, VerifierError, VerifierErrors};
     use crate::entity::EntityList;
     use crate::ir::instructions::{InstructionData, Opcode};
-    use crate::ir::Function;
+    use crate::ir::{types, AbiParam, Function};
     use crate::settings;
 
     macro_rules! assert_err_with_msg {
@@ -1929,4 +1968,30 @@ mod tests {
 
         assert_err_with_msg!(errors, "instruction format");
     }
+
+    #[test]
+    fn test_function_invalid_param() {
+        let mut func = Function::new();
+        func.signature.params.push(AbiParam::new(types::INVALID));
+
+        let mut errors = VerifierErrors::default();
+        let flags = &settings::Flags::new(settings::builder());
+        let verifier = Verifier::new(&func, flags.into());
+
+        let _ = verifier.typecheck_function_signature(&mut errors);
+        assert_err_with_msg!(errors, "Parameter at position 0 has an invalid type");
+    }
+
+    #[test]
+    fn test_function_invalid_return_value() {
+        let mut func = Function::new();
+        func.signature.returns.push(AbiParam::new(types::INVALID));
+
+        let mut errors = VerifierErrors::default();
+        let flags = &settings::Flags::new(settings::builder());
+        let verifier = Verifier::new(&func, flags.into());
+
+        let _ = verifier.typecheck_function_signature(&mut errors);
+        assert_err_with_msg!(errors, "Return value at position 0 has an invalid type");
+    }
 }

From 13ac951004c0b8765bf14b5597c22102ec6be789 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 6 Sep 2019 09:54:19 -0700
Subject: [PATCH 2732/3084] Allow global initialization of SIMD vectors

---
 cranelift/wasm/src/translation_utils.rs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index f033b8660c..726d6f7c63 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -65,7 +65,7 @@ pub struct Global {
     pub initializer: GlobalInit,
 }
 
-/// Globals are initialized via the four `const` operators or by referring to another import.
+/// Globals are initialized via the `const` operators or by referring to another import.
 #[derive(Debug, Clone, Copy, Hash)]
 pub enum GlobalInit {
     /// An `i32.const`.
@@ -76,6 +76,8 @@ pub enum GlobalInit {
     F32Const(u32),
     /// An `f64.const`.
     F64Const(u64),
+    /// A `vconst`.
+    V128Const([u8; 16]),
     /// A `get_global` of another global.
     GetGlobal(GlobalIndex),
     ///< The global is imported from, and thus initialized by, a different module.

From ddd1680e7637ca9459597f94fc1548f41167ff88 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 6 Sep 2019 10:54:41 -0700
Subject: [PATCH 2733/3084] Add V128 -> I8X16 type conversions

---
 cranelift/wasm/src/translation_utils.rs | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index 726d6f7c63..7dd774a626 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -122,6 +122,7 @@ pub fn type_to_type(ty: wasmparser::Type) -> WasmResult {
         wasmparser::Type::I64 => Ok(ir::types::I64),
         wasmparser::Type::F32 => Ok(ir::types::F32),
         wasmparser::Type::F64 => Ok(ir::types::F64),
+        wasmparser::Type::V128 => Ok(ir::types::I8X16),
         ty => wasm_unsupported!("type_to_type: wasm type {:?}", ty),
     }
 }
@@ -134,6 +135,7 @@ pub fn tabletype_to_type(ty: wasmparser::Type) -> WasmResult> {
         wasmparser::Type::I64 => Ok(Some(ir::types::I64)),
         wasmparser::Type::F32 => Ok(Some(ir::types::F32)),
         wasmparser::Type::F64 => Ok(Some(ir::types::F64)),
+        wasmparser::Type::V128 => Ok(Some(ir::types::I8X16)),
         wasmparser::Type::AnyFunc => Ok(None),
         ty => wasm_unsupported!("tabletype_to_type: table wasm type {:?}", ty),
     }
@@ -147,6 +149,7 @@ pub fn blocktype_to_type(ty_or_ft: wasmparser::TypeOrFuncType) -> WasmResult Ok(Some(ir::types::I64)),
             wasmparser::Type::F32 => Ok(Some(ir::types::F32)),
             wasmparser::Type::F64 => Ok(Some(ir::types::F64)),
+            wasmparser::Type::V128 => Ok(Some(ir::types::I8X16)),
             wasmparser::Type::EmptyBlockType => Ok(None),
             ty => wasm_unsupported!("blocktype_to_type: type {:?}", ty),
         },
@@ -175,7 +178,8 @@ pub fn num_return_values(ty: wasmparser::TypeOrFuncType) -> WasmResult {
             wasmparser::Type::I32
             | wasmparser::Type::F32
             | wasmparser::Type::I64
-            | wasmparser::Type::F64 => Ok(1),
+            | wasmparser::Type::F64
+            | wasmparser::Type::V128 => Ok(1),
             ty => wasm_unsupported!("unsupported return value type {:?}", ty),
         },
         wasmparser::TypeOrFuncType::FuncType(_) => {

From 7e6913e362deca7dffe2f3d5e3a849a36a857a87 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 6 Sep 2019 11:21:28 -0700
Subject: [PATCH 2734/3084] Add x86 encodings for vector store, load, fill,
 spill, and regmove

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 40 +++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 9ea29fb932..016f3bc9a8 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -293,6 +293,12 @@ impl PerCpuModeEncodings {
         }
     }
 
+    /// Add the same encoding/template pairing to both X86_32 and X86_64
+    fn enc_32_64(&mut self, inst: impl Clone + Into, template: Template) {
+        self.enc32(inst.clone(), template.clone());
+        self.enc64(inst, template);
+    }
+
     /// Add the same encoding/recipe pairing to both X86_32 and X86_64
     fn enc_32_64_rec(
         &mut self,
@@ -1959,6 +1965,40 @@ pub(crate) fn define(
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
     }
 
+    // SIMD register movement: store, load, spill, fill, regmove. All of these use encodings of
+    // MOVUPS and MOVAPS from SSE (TODO ideally all of these would either use MOVAPS when we have
+    // alignment or type-specific encodings, see https://github.com/CraneStation/cranelift/issues/1039).
+    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
+        // Store
+        let bound_store = store.bind_vector_from_lane(ty, sse_vector_size).bind_any();
+        e.enc_32_64(bound_store.clone(), rec_fst.opcodes(vec![0x0f, 0x11]));
+        e.enc_32_64(bound_store.clone(), rec_fstDisp8.opcodes(vec![0x0f, 0x11]));
+        e.enc_32_64(bound_store, rec_fstDisp32.opcodes(vec![0x0f, 0x11]));
+
+        // Load
+        let bound_load = load.bind_vector_from_lane(ty, sse_vector_size).bind_any();
+        e.enc_32_64(bound_load.clone(), rec_fld.opcodes(vec![0x0f, 0x10]));
+        e.enc_32_64(bound_load.clone(), rec_fldDisp8.opcodes(vec![0x0f, 0x10]));
+        e.enc_32_64(bound_load, rec_fldDisp32.opcodes(vec![0x0f, 0x10]));
+
+        // Spill
+        let bound_spill = spill.bind_vector_from_lane(ty, sse_vector_size);
+        e.enc_32_64(bound_spill, rec_fspillSib32.opcodes(vec![0x0f, 0x11]));
+        let bound_regspill = regspill.bind_vector_from_lane(ty, sse_vector_size);
+        e.enc_32_64(bound_regspill, rec_fregspill32.opcodes(vec![0x0f, 0x11]));
+
+        // Fill
+        let bound_fill = fill.bind_vector_from_lane(ty, sse_vector_size);
+        e.enc_32_64(bound_fill, rec_fillSib32.opcodes(vec![0x0f, 0x10]));
+        let bound_regfill = regfill.bind_vector_from_lane(ty, sse_vector_size);
+        e.enc_32_64(bound_regfill, rec_fregfill32.opcodes(vec![0x0f, 0x10]));
+
+        // Regmove
+        let bound_regmove = regmove.bind_vector_from_lane(ty, sse_vector_size);
+        e.enc_32_64(bound_regmove.clone(), rec_rmov.opcodes(vec![0x0f, 0x28]));
+        e.enc_32_64(bound_regmove, rec_frmov.opcodes(vec![0x0f, 0x28]));
+    }
+
     // Reference type instructions
 
     // Null references implemented as iconst 0.

From 6cbc6e8bfb9c0734bb80738af717e546001119f9 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 6 Sep 2019 11:43:11 -0700
Subject: [PATCH 2735/3084] Handle use of SIMD vector globals and locals

---
 cranelift/wasm/src/func_translator.rs     | 4 ++++
 cranelift/wasm/src/sections_translator.rs | 4 ++++
 cranelift/wasm/src/translation_utils.rs   | 3 ++-
 3 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index b49cffd474..cced9d87e6 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -179,6 +179,10 @@ fn declare_locals(
         I64 => builder.ins().iconst(ir::types::I64, 0),
         F32 => builder.ins().f32const(ir::immediates::Ieee32::with_bits(0)),
         F64 => builder.ins().f64const(ir::immediates::Ieee64::with_bits(0)),
+        V128 => {
+            let constant_handle = builder.func.dfg.constants.insert([0; 16].to_vec());
+            builder.ins().vconst(ir::types::I8X16, constant_handle)
+        }
         AnyRef => builder.ins().null(environ.reference_type()),
         ty => wasm_unsupported!("unsupported local type {:?}", ty),
     };
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index 67830af3ff..47aeb495cb 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -14,6 +14,7 @@ use crate::translation_utils::{
 };
 use crate::{wasm_unsupported, HashMap};
 use core::convert::TryFrom;
+use cranelift_codegen::ir::immediates::Uimm128;
 use cranelift_codegen::ir::{self, AbiParam, Signature};
 use cranelift_entity::EntityRef;
 use std::vec::Vec;
@@ -201,6 +202,9 @@ pub fn parse_global_section(
             Operator::I64Const { value } => GlobalInit::I64Const(value),
             Operator::F32Const { value } => GlobalInit::F32Const(value.bits()),
             Operator::F64Const { value } => GlobalInit::F64Const(value.bits()),
+            Operator::V128Const { value } => {
+                GlobalInit::V128Const(Uimm128::from(value.bytes().to_vec().as_slice()))
+            }
             Operator::GetGlobal { global_index } => {
                 GlobalInit::GetGlobal(GlobalIndex::from_u32(global_index))
             }
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index 7dd774a626..23d2c2892d 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -4,6 +4,7 @@ use crate::wasm_unsupported;
 use core::u32;
 use cranelift_codegen::entity::entity_impl;
 use cranelift_codegen::ir;
+use cranelift_codegen::ir::immediates::Uimm128;
 #[cfg(feature = "enable-serde")]
 use serde::{Deserialize, Serialize};
 use wasmparser;
@@ -77,7 +78,7 @@ pub enum GlobalInit {
     /// An `f64.const`.
     F64Const(u64),
     /// A `vconst`.
-    V128Const([u8; 16]),
+    V128Const(Uimm128),
     /// A `get_global` of another global.
     GetGlobal(GlobalIndex),
     ///< The global is imported from, and thus initialized by, a different module.

From cd426cb7bcc81d49a4f06c78e50ab88192543fd2 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 9 Sep 2019 16:31:42 -0700
Subject: [PATCH 2736/3084] Rename Uimm128 to V128Imm

---
 cranelift/codegen/src/ir/immediates.rs    | 80 +++++++++++------------
 cranelift/codegen/src/write.rs            | 10 +--
 cranelift/reader/src/parser.rs            | 10 +--
 cranelift/wasm/src/sections_translator.rs |  4 +-
 cranelift/wasm/src/translation_utils.rs   |  4 +-
 5 files changed, 54 insertions(+), 54 deletions(-)

diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index 2c5ee27c47..371c05bdd9 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -290,13 +290,13 @@ impl FromStr for Uimm32 {
     }
 }
 
-/// A 128-bit unsigned integer immediate operand.
+/// A 128-bit immediate operand.
 ///
 /// This is used as an immediate value in SIMD instructions.
 #[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)]
-pub struct Uimm128(pub [u8; 16]);
+pub struct V128Imm(pub [u8; 16]);
 
-impl Uimm128 {
+impl V128Imm {
     /// Iterate over the bytes in the constant
     pub fn bytes(&self) -> impl Iterator {
         self.0.iter()
@@ -313,7 +313,7 @@ impl Uimm128 {
     }
 }
 
-impl Display for Uimm128 {
+impl Display for V128Imm {
     // Print a 128-bit vector in hexadecimal, e.g. 0x000102030405060708090a0b0c0d0e0f.
     fn fmt(&self, f: &mut Formatter) -> fmt::Result {
         write!(f, "0x")?;
@@ -333,24 +333,24 @@ impl Display for Uimm128 {
     }
 }
 
-impl From for Uimm128 {
+impl From for V128Imm {
     fn from(x: u64) -> Self {
         let mut buffer: [u8; 16] = [0; 16]; // zero-fill
         (0..8).for_each(|byte| buffer[byte] = (x >> (byte as u64 * 8) & 0xff) as u8); // insert each byte from the u64 into v in little-endian order
-        Uimm128(buffer)
+        V128Imm(buffer)
     }
 }
 
-impl From<&[u8]> for Uimm128 {
+impl From<&[u8]> for V128Imm {
     fn from(slice: &[u8]) -> Self {
         assert_eq!(slice.len(), 16);
         let mut buffer = [0; 16];
         buffer.copy_from_slice(slice);
-        Uimm128(buffer)
+        V128Imm(buffer)
     }
 }
 
-impl FromStr for Uimm128 {
+impl FromStr for V128Imm {
     type Err = &'static str;
 
     // parse a 128-bit vector from a hexadecimal string, formatted as above
@@ -384,7 +384,7 @@ impl FromStr for Uimm128 {
                     position = position.wrapping_sub(1); // should only wrap on the last iteration
                 }
 
-                Ok(Uimm128(buffer))
+                Ok(V128Imm(buffer))
             }
         }
     }
@@ -396,7 +396,7 @@ impl FromStr for Uimm128 {
 ///  - this requires the input type (i.e. $ty) to implement ToBytes
 macro_rules! construct_uimm128_from_iterator_of {
     ( $ty:ident, $lanes:expr ) => {
-        impl FromIterator<$ty> for Uimm128 {
+        impl FromIterator<$ty> for V128Imm {
             fn from_iter>(iter: T) -> Self {
                 let mut buffer: [u8; 16] = [0; 16];
                 iter.into_iter()
@@ -405,14 +405,14 @@ macro_rules! construct_uimm128_from_iterator_of {
                     .flat_map(|b| b)
                     .enumerate()
                     .for_each(|(i, b)| buffer[i] = b);
-                Uimm128(buffer)
+                V128Imm(buffer)
             }
         }
     };
 }
 
 /// Special case for booleans since we have to decide the bit-width based on the number of items
-impl FromIterator for Uimm128 {
+impl FromIterator for V128Imm {
     fn from_iter>(iter: T) -> Self {
         let bools = Vec::from_iter(iter);
         let count = bools.len();
@@ -425,7 +425,7 @@ impl FromIterator for Uimm128 {
             .enumerate()
             .map(|(i, &b)| (i * step, if b { 1 } else { 0 }))
             .for_each(|(i, b)| buffer[i] = b);
-        Uimm128(buffer)
+        V128Imm(buffer)
     }
 }
 
@@ -1069,51 +1069,51 @@ mod tests {
 
     #[test]
     fn format_uimm128() {
-        assert_eq!(Uimm128::from(0).to_string(), "0x00");
-        assert_eq!(Uimm128::from(42).to_string(), "0x2a");
-        assert_eq!(Uimm128::from(3735928559).to_string(), "0xdeadbeef");
+        assert_eq!(V128Imm::from(0).to_string(), "0x00");
+        assert_eq!(V128Imm::from(42).to_string(), "0x2a");
+        assert_eq!(V128Imm::from(3735928559).to_string(), "0xdeadbeef");
         assert_eq!(
-            Uimm128::from(0x0102030405060708).to_string(),
+            V128Imm::from(0x0102030405060708).to_string(),
             "0x0102030405060708"
         );
     }
 
     #[test]
     fn parse_uimm128() {
-        parse_ok::("0x00", "0x00");
-        parse_ok::("0x00000042", "0x42");
-        parse_ok::(
+        parse_ok::("0x00", "0x00");
+        parse_ok::("0x00000042", "0x42");
+        parse_ok::(
             "0x0102030405060708090a0b0c0d0e0f00",
             "0x0102030405060708090a0b0c0d0e0f00",
         );
-        parse_ok::("0x_0000_0043_21", "0x4321");
+        parse_ok::("0x_0000_0043_21", "0x4321");
 
-        parse_err::("", "Expected a hexadecimal string, e.g. 0x1234");
-        parse_err::("0x", "Expected a hexadecimal string, e.g. 0x1234");
-        parse_err::(
+        parse_err::("", "Expected a hexadecimal string, e.g. 0x1234");
+        parse_err::("0x", "Expected a hexadecimal string, e.g. 0x1234");
+        parse_err::(
             "0x042",
             "Hexadecimal string must have an even number of digits",
         );
-        parse_err::(
+        parse_err::(
             "0x00000000000000000000000000000000000000000000000000",
             "Hexadecimal string has too many digits to fit in a 128-bit vector",
         );
-        parse_err::("0xrstu", "Unable to parse as hexadecimal");
-        parse_err::("0x__", "Hexadecimal string must have some digits");
+        parse_err::("0xrstu", "Unable to parse as hexadecimal");
+        parse_err::("0x__", "Hexadecimal string must have some digits");
     }
 
     #[test]
     fn uimm128_equivalence() {
         assert_eq!(
-            "0x01".parse::().unwrap().0,
+            "0x01".parse::().unwrap().0,
             [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
         );
         assert_eq!(
-            Uimm128::from_iter(vec![1, 0, 0, 0]).0,
+            V128Imm::from_iter(vec![1, 0, 0, 0]).0,
             [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
         );
         assert_eq!(
-            Uimm128::from(1).0,
+            V128Imm::from(1).0,
             [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
         );
     }
@@ -1121,19 +1121,19 @@ mod tests {
     #[test]
     fn uimm128_endianness() {
         assert_eq!(
-            "0x42".parse::().unwrap().0,
+            "0x42".parse::().unwrap().0,
             [0x42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
         );
         assert_eq!(
-            "0x00".parse::().unwrap().0,
+            "0x00".parse::().unwrap().0,
             [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
         );
         assert_eq!(
-            "0x12345678".parse::().unwrap().0,
+            "0x12345678".parse::().unwrap().0,
             [0x78, 0x56, 0x34, 0x12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
         );
         assert_eq!(
-            "0x1234_5678".parse::().unwrap().0,
+            "0x1234_5678".parse::().unwrap().0,
             [0x78, 0x56, 0x34, 0x12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
         );
     }
@@ -1141,17 +1141,17 @@ mod tests {
     #[test]
     fn uimm128_from_iter() {
         assert_eq!(
-            Uimm128::from_iter(vec![4, 3, 2, 1]).0,
+            V128Imm::from_iter(vec![4, 3, 2, 1]).0,
             [4, 0, 0, 0, 3, 0, 0, 0, 2, 0, 0, 0, 1, 0, 0, 0]
         );
 
         assert_eq!(
-            Uimm128::from_iter(vec![false, true]).0,
+            V128Imm::from_iter(vec![false, true]).0,
             [/* false */ 0, 0, 0, 0, 0, 0, 0, 0, /* true */ 1, 0, 0, 0, 0, 0, 0, 0]
         );
 
         assert_eq!(
-            Uimm128::from_iter(vec![false, true, false, true, false, true, false, true]).0,
+            V128Imm::from_iter(vec![false, true, false, true, false, true, false, true]).0,
             [0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0]
         );
 
@@ -1160,7 +1160,7 @@ mod tests {
             1 as u8, 2, 3, 4, 5, 6, 7, 8, 9, 0xa, 0xb, 0xc, 0xd, 0xe, 0xf, 0,
         ];
         assert_eq!(
-            Uimm128::from_iter(u8s).0,
+            V128Imm::from_iter(u8s).0,
             [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 0]
         );
 
@@ -1170,7 +1170,7 @@ mod tests {
             .map(|&f| Ieee32::from(f))
             .collect();
         assert_eq!(
-            Uimm128::from_iter(ieee32s).0,
+            V128Imm::from_iter(ieee32s).0,
             [
                 /* 32.4 == */ 0x9a, 0x99, 0x01, 0x42, /* 0 == */ 0, 0, 0, 0,
                 /* 1 == */ 0, 0, 0x80, 0x3f, /* 6.6666 == */ 0xca, 0x54, 0xd5, 0x40,
diff --git a/cranelift/codegen/src/write.rs b/cranelift/codegen/src/write.rs
index e3c8bdb2fa..8d974c1ec5 100644
--- a/cranelift/codegen/src/write.rs
+++ b/cranelift/codegen/src/write.rs
@@ -5,7 +5,7 @@
 
 use crate::entity::SecondaryMap;
 use crate::ir::entities::AnyEntity;
-use crate::ir::immediates::Uimm128;
+use crate::ir::immediates::V128Imm;
 use crate::ir::{
     DataFlowGraph, DisplayFunctionAnnotations, Ebb, Function, Inst, SigRef, Type, Value, ValueDef,
     ValueLoc,
@@ -509,15 +509,15 @@ pub fn write_operands(
             constant_handle, ..
         } => {
             let data = dfg.constants.get(constant_handle);
-            let uimm128 = Uimm128::from(&data[..]);
-            write!(w, " {}", uimm128)
+            let v128 = V128Imm::from(&data[..]);
+            write!(w, " {}", v128)
         }
         Shuffle { mask, args, .. } => {
             let data = dfg.immediates.get(mask).expect(
                 "Expected the shuffle mask to already be inserted into the immediates table",
             );
-            let uimm128 = Uimm128::from(&data[..]);
-            write!(w, " {}, {}, {}", args[0], args[1], uimm128)
+            let v128 = V128Imm::from(&data[..]);
+            write!(w, " {}, {}, {}", args[0], args[1], v128)
         }
         IntCompare { cond, args, .. } => write!(w, " {} {}, {}", cond, args[0], args[1]),
         IntCompareImm { cond, arg, imm, .. } => write!(w, " {} {}, {}", cond, arg, imm),
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index bc9436fe0c..711b4d3cc1 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -9,7 +9,7 @@ use crate::testfile::{Comment, Details, Feature, TestFile};
 use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::entities::AnyEntity;
-use cranelift_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm128, Uimm32, Uimm64};
+use cranelift_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32, Uimm64, V128Imm};
 use cranelift_codegen::ir::instructions::{InstructionData, InstructionFormat, VariableArgs};
 use cranelift_codegen::ir::types::INVALID;
 use cranelift_codegen::ir::types::*;
@@ -597,7 +597,7 @@ impl<'a> Parser<'a> {
 
     // Match and consume a Uimm128 immediate; due to size restrictions on InstructionData, Uimm128
     // is boxed in cranelift-codegen/meta/src/shared/immediates.rs
-    fn match_uimm128(&mut self, err_msg: &str) -> ParseResult {
+    fn match_uimm128(&mut self, err_msg: &str) -> ParseResult {
         if let Some(Token::Integer(text)) = self.token() {
             self.consume();
             // Lexer just gives us raw text that looks like hex code.
@@ -614,7 +614,7 @@ impl<'a> Parser<'a> {
     }
 
     // Match and consume either a hexadecimal Uimm128 immediate (e.g. 0x000102...) or its literal list form (e.g. [0 1 2...])
-    fn match_uimm128_or_literals(&mut self, controlling_type: Type) -> ParseResult {
+    fn match_uimm128_or_literals(&mut self, controlling_type: Type) -> ParseResult {
         if self.optional(Token::LBracket) {
             // parse using a list of values, e.g. vconst.i32x4 [0 1 2 3]
             let uimm128 = self.parse_literals_to_uimm128(controlling_type)?;
@@ -838,7 +838,7 @@ impl<'a> Parser<'a> {
     }
 
     /// Parse a list of literals (i.e. integers, floats, booleans); e.g.
-    fn parse_literals_to_uimm128(&mut self, ty: Type) -> ParseResult {
+    fn parse_literals_to_uimm128(&mut self, ty: Type) -> ParseResult {
         macro_rules! consume {
             ( $ty:ident, $match_fn:expr ) => {{
                 assert!($ty.is_vector());
@@ -846,7 +846,7 @@ impl<'a> Parser<'a> {
                 for _ in 0..$ty.lane_count() {
                     v.push($match_fn?);
                 }
-                Uimm128::from_iter(v)
+                V128Imm::from_iter(v)
             }};
         }
 
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index 47aeb495cb..c8c66529ad 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -14,7 +14,7 @@ use crate::translation_utils::{
 };
 use crate::{wasm_unsupported, HashMap};
 use core::convert::TryFrom;
-use cranelift_codegen::ir::immediates::Uimm128;
+use cranelift_codegen::ir::immediates::V128Imm;
 use cranelift_codegen::ir::{self, AbiParam, Signature};
 use cranelift_entity::EntityRef;
 use std::vec::Vec;
@@ -203,7 +203,7 @@ pub fn parse_global_section(
             Operator::F32Const { value } => GlobalInit::F32Const(value.bits()),
             Operator::F64Const { value } => GlobalInit::F64Const(value.bits()),
             Operator::V128Const { value } => {
-                GlobalInit::V128Const(Uimm128::from(value.bytes().to_vec().as_slice()))
+                GlobalInit::V128Const(V128Imm::from(value.bytes().to_vec().as_slice()))
             }
             Operator::GetGlobal { global_index } => {
                 GlobalInit::GetGlobal(GlobalIndex::from_u32(global_index))
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index 23d2c2892d..95553c6e48 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -4,7 +4,7 @@ use crate::wasm_unsupported;
 use core::u32;
 use cranelift_codegen::entity::entity_impl;
 use cranelift_codegen::ir;
-use cranelift_codegen::ir::immediates::Uimm128;
+use cranelift_codegen::ir::immediates::V128Imm;
 #[cfg(feature = "enable-serde")]
 use serde::{Deserialize, Serialize};
 use wasmparser;
@@ -78,7 +78,7 @@ pub enum GlobalInit {
     /// An `f64.const`.
     F64Const(u64),
     /// A `vconst`.
-    V128Const(Uimm128),
+    V128Const(V128Imm),
     /// A `get_global` of another global.
     GetGlobal(GlobalIndex),
     ///< The global is imported from, and thus initialized by, a different module.

From 766cf8ddfdc1de9d13f2a44094701cb691d2f1bf Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 11 Sep 2019 10:58:21 -0700
Subject: [PATCH 2737/3084] Add x86 implemention for SIMD iadd

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 11 ++++
 .../filetests/isa/x86/iadd-simd.clif          | 50 +++++++++++++++++++
 cranelift/wasm/src/code_translator.rs         |  8 +--
 3 files changed, 65 insertions(+), 4 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/iadd-simd.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 016f3bc9a8..1096fb7d4c 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1999,6 +1999,17 @@ pub(crate) fn define(
         e.enc_32_64(bound_regmove, rec_frmov.opcodes(vec![0x0f, 0x28]));
     }
 
+    // SIMD integer addition
+    for (ty, opcodes) in &[
+        (I8, &[0x66, 0x0f, 0xfc]),  // PADDB from SSE2
+        (I16, &[0x66, 0x0f, 0xfd]), // PADDW from SSE2
+        (I32, &[0x66, 0x0f, 0xfe]), // PADDD from SSE2
+        (I64, &[0x66, 0x0f, 0xd4]), // PADDQ from SSE2
+    ] {
+        let iadd = iadd.bind_vector_from_lane(ty.clone(), sse_vector_size);
+        e.enc_32_64(iadd, rec_fa.opcodes(opcodes.to_vec()));
+    }
+
     // Reference type instructions
 
     // Null references implemented as iconst 0.
diff --git a/cranelift/filetests/filetests/isa/x86/iadd-simd.clif b/cranelift/filetests/filetests/isa/x86/iadd-simd.clif
new file mode 100644
index 0000000000..0884a97f55
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/iadd-simd.clif
@@ -0,0 +1,50 @@
+test run
+test binemit
+set enable_simd
+target x86_64 skylake
+
+function %iadd_i32x4() -> b1 {
+ebb0:
+[-, %xmm0]    v0 = vconst.i32x4 [1 1 1 1]
+[-, %xmm1]    v1 = vconst.i32x4 [1 2 3 4]
+[-, %xmm0]    v2 = iadd v0, v1      ; bin: 66 0f fe c1
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 2
+
+    v5 = extractlane v2, 3
+    v6 = icmp_imm eq v5, 5
+    ; TODO replace extractlanes with vector comparison
+
+    v7 = band v4, v6
+    return v7
+}
+
+; run
+
+function %iadd_i8x16_with_overflow() -> b1 {
+ebb0:
+[-, %xmm0]    v0 = vconst.i8x16 [255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255]
+[-, %xmm7]    v1 = vconst.i8x16 [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2]
+[-, %xmm0]    v2 = iadd v0, v1      ; bin: 66 0f fc c7
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 1
+    ; TODO replace extractlane with vector comparison
+
+    return v4
+}
+
+; run
+
+function %iadd_i16x8(i16x8, i16x8) -> i16x8 {
+ebb0(v0: i16x8 [%xmm1], v1: i16x8 [%xmm2]):
+[-, %xmm1]  v2 = iadd v0, v1      ; bin: 66 0f fd ca
+            return v2
+}
+
+function %iadd_i64x2(i64x2, i64x2) -> i64x2 {
+ebb0(v0: i64x2 [%xmm3], v1: i64x2 [%xmm4]):
+[-, %xmm3]  v2 = iadd v0, v1      ; bin: 66 0f d4 dc
+            return v2
+}
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index a7bb331a39..fef3215c52 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -986,6 +986,10 @@ pub fn translate_operator(
             // to WASM using the less specific v128 type for certain operations and more specific
             // types (e.g. i8x16) for others.
         }
+        Operator::I8x16Add | Operator::I16x8Add | Operator::I32x4Add | Operator::I64x2Add => {
+            let (a, b) = state.pop2();
+            state.push1(builder.ins().iadd(a, b))
+        }
         Operator::V128Load { .. }
         | Operator::V128Store { .. }
         | Operator::I8x16Eq
@@ -1041,7 +1045,6 @@ pub fn translate_operator(
         | Operator::I8x16Shl
         | Operator::I8x16ShrS
         | Operator::I8x16ShrU
-        | Operator::I8x16Add
         | Operator::I8x16AddSaturateS
         | Operator::I8x16AddSaturateU
         | Operator::I8x16Sub
@@ -1054,7 +1057,6 @@ pub fn translate_operator(
         | Operator::I16x8Shl
         | Operator::I16x8ShrS
         | Operator::I16x8ShrU
-        | Operator::I16x8Add
         | Operator::I16x8AddSaturateS
         | Operator::I16x8AddSaturateU
         | Operator::I16x8Sub
@@ -1067,7 +1069,6 @@ pub fn translate_operator(
         | Operator::I32x4Shl
         | Operator::I32x4ShrS
         | Operator::I32x4ShrU
-        | Operator::I32x4Add
         | Operator::I32x4Sub
         | Operator::I32x4Mul
         | Operator::I64x2Neg
@@ -1076,7 +1077,6 @@ pub fn translate_operator(
         | Operator::I64x2Shl
         | Operator::I64x2ShrS
         | Operator::I64x2ShrU
-        | Operator::I64x2Add
         | Operator::I64x2Sub
         | Operator::F32x4Abs
         | Operator::F32x4Neg

From fe25abeb0d2e81e55833f7f8503f5a1036ee367f Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 11 Sep 2019 11:44:29 -0700
Subject: [PATCH 2738/3084] Add x86 encodings for vector copy, copy_nop,
 fill_nop

---
 cranelift/codegen/meta/src/isa/x86/encodings.rs | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 1096fb7d4c..1843383714 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1992,11 +1992,19 @@ pub(crate) fn define(
         e.enc_32_64(bound_fill, rec_fillSib32.opcodes(vec![0x0f, 0x10]));
         let bound_regfill = regfill.bind_vector_from_lane(ty, sse_vector_size);
         e.enc_32_64(bound_regfill, rec_fregfill32.opcodes(vec![0x0f, 0x10]));
+        let bound_fill_nop = fill_nop.bind_vector_from_lane(ty, sse_vector_size);
+        e.enc_32_64_rec(bound_fill_nop, rec_ffillnull, 0);
 
         // Regmove
         let bound_regmove = regmove.bind_vector_from_lane(ty, sse_vector_size);
         e.enc_32_64(bound_regmove.clone(), rec_rmov.opcodes(vec![0x0f, 0x28]));
         e.enc_32_64(bound_regmove, rec_frmov.opcodes(vec![0x0f, 0x28]));
+
+        // Copy
+        let bound_copy = copy.bind_vector_from_lane(ty, sse_vector_size);
+        e.enc_32_64(bound_copy, rec_furm.opcodes(vec![0x0f, 0x28])); // MOVAPS from SSE
+        let bound_copy_nop = copy_nop.bind_vector_from_lane(ty, sse_vector_size);
+        e.enc_32_64_rec(bound_copy_nop, rec_stacknull, 0);
     }
 
     // SIMD integer addition

From 5c5eabb8d04e197a55700df23036e1874605ce42 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Thu, 19 Sep 2019 12:23:51 -0700
Subject: [PATCH 2739/3084] Bump version to 0.43.0

---
 cranelift/Cargo.toml              | 30 +++++++++++++++---------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    | 10 +++++-----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/object/Cargo.toml       |  6 +++---
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 18 files changed, 65 insertions(+), 65 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index c48ef7f71f..56f9b6e94c 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.42.0"
+version = "0.43.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.42.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.42.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.42.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.42.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.42.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.42.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.42.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.42.0" }
-cranelift-module = { path = "cranelift-module", version = "0.42.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.42.0" }
-cranelift-object = { path = "cranelift-object", version = "0.42.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.42.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.42.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.42.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.43.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.43.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.43.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.43.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.43.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.43.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.43.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.43.0" }
+cranelift-module = { path = "cranelift-module", version = "0.43.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.43.0" }
+cranelift-object = { path = "cranelift-object", version = "0.43.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.43.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.43.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.43.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index fee6d19b2e..211e3ca035 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.42.0"
+version = "0.43.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.42.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.43.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 2d81a660e8..07ca18a9d3 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.42.0"
+version = "0.43.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.42.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.42.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.43.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.43.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -28,7 +28,7 @@ smallvec = { version = "0.6.10" }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.42.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.43.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index c623493e30..c239203133 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.42.0"
+version = "0.43.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.42.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.43.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 22b34d33d7..5f54feb483 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.42.0"
+version = "0.43.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 2f8f2258f0..280e762f68 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.42.0"
+version = "0.43.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.42.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.43.0" }
 faerie = "0.11.0"
 goblin = "0.0.24"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index b8029c17e1..41fe2cb511 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.42.0"
+version = "0.43.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.42.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.42.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.42.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.43.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.43.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.43.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 3cd8882201..22c55b4b8c 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.42.0"
+version = "0.43.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", default-features = false }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 15805b7878..a0c48bcf42 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.42.0"
+version = "0.43.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.42.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.43.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index b677ddae12..b0e8bf9027 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.42.0"
+version = "0.43.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", default-features = false }
 target-lexicon = "0.8.1"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 22613d65db..72ffe00fbe 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.42.0"
+version = "0.43.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.42.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.43.0" }
 object = { version = "0.14.0", default-features = false, features = ["write"] }
 target-lexicon = "0.8.1"
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 9b067ffa41..c85bc98b2b 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.42.0"
+version = "0.43.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.42.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.43.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index ce7ab1997e..782e4bf27e 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.42.0"
+version="0.43.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 2a7216e8f7..2c342cfbc0 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.42.0"
+version = "0.43.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0" }
 target-lexicon = "0.8.1"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index a40ac22c1a..8125888cd5 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.42.0"
+version = "0.43.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.42.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.43.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 706a39f384..fad5eca066 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.42.0"
+version = "0.43.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.42.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.42.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.43.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.43.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.42.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.42.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.42.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.43.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.43.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.43.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 2967788619..dcae5fd681 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.42.0"
+version = "0.43.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.42.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.43.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index c140de53a7..d3cc1f3736 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.42.0"
+version = "0.43.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.37.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.42.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.42.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.42.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.43.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.43.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 9b8e7b511e8b4c21742c660e3213c1b8de44f611 Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Thu, 19 Sep 2019 09:46:22 -0700
Subject: [PATCH 2740/3084] tidy: Remove extra semicolons

These were causing compilation warnings.
---
 cranelift/codegen/meta/src/cdsl/typevar.rs   | 2 +-
 cranelift/codegen/meta/src/shared/formats.rs | 2 +-
 cranelift/codegen/meta/src/shared/mod.rs     | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 71c2fd2e29..78e702cacf 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -851,7 +851,7 @@ impl TypeSetBuilder {
 
     pub fn build(self) -> TypeSet {
         let min_lanes = if self.includes_scalars { 1 } else { 2 };
-;
+
         let bools = range_to_set(self.bools.to_range(1..MAX_BITS, None))
             .into_iter()
             .filter(legal_bool)
diff --git a/cranelift/codegen/meta/src/shared/formats.rs b/cranelift/codegen/meta/src/shared/formats.rs
index 74a2f6bb8f..73aaa42634 100644
--- a/cranelift/codegen/meta/src/shared/formats.rs
+++ b/cranelift/codegen/meta/src/shared/formats.rs
@@ -65,7 +65,7 @@ pub(crate) fn define(imm: &Immediates, entities: &EntityRefs) -> FormatRegistry
             .value()
             .value(),
     );
-    registry.insert(Builder::new("FloatCond").imm(&imm.floatcc).value());;
+    registry.insert(Builder::new("FloatCond").imm(&imm.floatcc).value());
 
     registry.insert(
         Builder::new("IntSelect")
diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
index 12921da5ef..73cdb4254d 100644
--- a/cranelift/codegen/meta/src/shared/mod.rs
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -29,7 +29,7 @@ pub(crate) fn define() -> Definitions {
     let mut all_instructions = AllInstructions::new();
 
     let immediates = Immediates::new();
-    let entities = EntityRefs::new();;
+    let entities = EntityRefs::new();
     let format_registry = formats::define(&immediates, &entities);
     let instructions = instructions::define(
         &mut all_instructions,

From 2330ca7e2c599d6e70cce7203ff0cfa70a066863 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 20 Sep 2019 13:44:42 -0700
Subject: [PATCH 2741/3084] Fix incorrect regmove and fill encodings for SIMD
 types

- `fill` attempted to use a GPR recipe, `fillSib32`, instead of its FPR equivalent, `ffillSib32` (code compiled without error but incorrect instructions were allowed, e.g. `v1 = regmove v0, %rdi -> %xmm0`
- `regmove` could be encoded with a GPR recipe, `rmov`, which hid the above incorrectness; now only FPR-to-FPR regmoves are allowed using the `frmov` recipe
---
 cranelift/codegen/meta/src/isa/x86/encodings.rs | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 1843383714..741f77d755 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1989,7 +1989,7 @@ pub(crate) fn define(
 
         // Fill
         let bound_fill = fill.bind_vector_from_lane(ty, sse_vector_size);
-        e.enc_32_64(bound_fill, rec_fillSib32.opcodes(vec![0x0f, 0x10]));
+        e.enc_32_64(bound_fill, rec_ffillSib32.opcodes(vec![0x0f, 0x10]));
         let bound_regfill = regfill.bind_vector_from_lane(ty, sse_vector_size);
         e.enc_32_64(bound_regfill, rec_fregfill32.opcodes(vec![0x0f, 0x10]));
         let bound_fill_nop = fill_nop.bind_vector_from_lane(ty, sse_vector_size);
@@ -1997,7 +1997,6 @@ pub(crate) fn define(
 
         // Regmove
         let bound_regmove = regmove.bind_vector_from_lane(ty, sse_vector_size);
-        e.enc_32_64(bound_regmove.clone(), rec_rmov.opcodes(vec![0x0f, 0x28]));
         e.enc_32_64(bound_regmove, rec_frmov.opcodes(vec![0x0f, 0x28]));
 
         // Copy

From 411545ed9adf7f0d50ada008fd2fc380b8248518 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 20 Sep 2019 14:11:15 -0700
Subject: [PATCH 2742/3084] Bump version to 0.43.1

---
 cranelift/Cargo.toml              | 30 +++++++++++++++---------------
 cranelift/bforest/Cargo.toml      |  4 ++--
 cranelift/codegen/Cargo.toml      |  8 ++++----
 cranelift/codegen/meta/Cargo.toml |  4 ++--
 cranelift/entity/Cargo.toml       |  2 +-
 cranelift/faerie/Cargo.toml       |  6 +++---
 cranelift/filetests/Cargo.toml    | 10 +++++-----
 cranelift/frontend/Cargo.toml     |  4 ++--
 cranelift/module/Cargo.toml       |  6 +++---
 cranelift/native/Cargo.toml       |  4 ++--
 cranelift/object/Cargo.toml       |  6 +++---
 cranelift/preopt/Cargo.toml       |  6 +++---
 cranelift/publish-all.sh          |  2 +-
 cranelift/reader/Cargo.toml       |  4 ++--
 cranelift/serde/Cargo.toml        |  6 +++---
 cranelift/simplejit/Cargo.toml    | 14 +++++++-------
 cranelift/umbrella/Cargo.toml     |  6 +++---
 cranelift/wasm/Cargo.toml         |  8 ++++----
 18 files changed, 65 insertions(+), 65 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 56f9b6e94c..df1224038a 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.43.0"
+version = "0.43.1"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.43.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.43.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.43.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.43.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.43.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.43.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.43.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.43.0" }
-cranelift-module = { path = "cranelift-module", version = "0.43.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.43.0" }
-cranelift-object = { path = "cranelift-object", version = "0.43.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.43.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.43.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.43.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.43.1" }
+cranelift-entity = { path = "cranelift-entity", version = "0.43.1" }
+cranelift-reader = { path = "cranelift-reader", version = "0.43.1" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.43.1" }
+cranelift-serde = { path = "cranelift-serde", version = "0.43.1", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.43.1", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.43.1" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.43.1" }
+cranelift-module = { path = "cranelift-module", version = "0.43.1" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.43.1" }
+cranelift-object = { path = "cranelift-object", version = "0.43.1" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.43.1" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.43.1" }
+cranelift = { path = "cranelift-umbrella", version = "0.43.1" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 211e3ca035..4726440e70 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.43.0"
+version = "0.43.1"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.43.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.43.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 07ca18a9d3..251440cee0 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.43.0"
+version = "0.43.1"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,8 +13,8 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.43.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.43.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.43.1", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.43.1", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -28,7 +28,7 @@ smallvec = { version = "0.6.10" }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.43.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.43.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index c239203133..d869c412ba 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.43.0"
+version = "0.43.1"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../../cranelift-entity", version = "0.43.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.43.1", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 5f54feb483..d71e63ec5e 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.43.0"
+version = "0.43.1"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 280e762f68..98dccdcf45 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.43.0"
+version = "0.43.1"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.43.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1" }
+cranelift-module = { path = "../cranelift-module", version = "0.43.1" }
 faerie = "0.11.0"
 goblin = "0.0.24"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 41fe2cb511..5a7d2aab11 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.43.0"
+version = "0.43.1"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.43.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.43.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.43.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.43.1" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.43.1" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.43.1" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 22c55b4b8c..919b9b4bd7 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.43.0"
+version = "0.43.1"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", default-features = false }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index a0c48bcf42..ac5eff0822 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.43.0"
+version = "0.43.1"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.43.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.43.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index b0e8bf9027..35576d1d3b 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.43.0"
+version = "0.43.1"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", default-features = false }
 target-lexicon = "0.8.1"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 72ffe00fbe..d75fb054f3 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.43.0"
+version = "0.43.1"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.43.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1" }
+cranelift-module = { path = "../cranelift-module", version = "0.43.1" }
 object = { version = "0.14.0", default-features = false, features = ["write"] }
 target-lexicon = "0.8.1"
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index c85bc98b2b..2bf1a1a528 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.43.0"
+version = "0.43.1"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.43.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.43.1", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 782e4bf27e..e52228a9c3 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.43.0"
+version="0.43.1"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 2c342cfbc0..769a5b4550 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.43.0"
+version = "0.43.1"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1" }
 target-lexicon = "0.8.1"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 8125888cd5..16848ee23e 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.43.0"
+version = "0.43.1"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.43.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.43.1" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index fad5eca066..7078e483b4 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.43.0"
+version = "0.43.1"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.43.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.43.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1" }
+cranelift-module = { path = "../cranelift-module", version = "0.43.1" }
+cranelift-native = { path = "../cranelift-native", version = "0.43.1" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.43.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.43.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.43.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.43.1" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.43.1" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.43.1" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index dcae5fd681..3f605832a0 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.43.0"
+version = "0.43.1"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.43.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.43.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index d3cc1f3736..a87e4a4cce 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.43.0"
+version = "0.43.1"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.37.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.43.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.43.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.43.1", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.43.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 178241625c71d3b3b354d754ac24ad2b993ed2ae Mon Sep 17 00:00:00 2001
From: Anthony Ramine 
Date: Thu, 19 Sep 2019 12:23:42 +0200
Subject: [PATCH 2743/3084] Use slice::from_ref and slice::from_mut

---
 cranelift/codegen/meta/src/gen_inst.rs     |  9 +++++++--
 cranelift/codegen/src/ir/instructions.rs   |  1 -
 cranelift/codegen/src/lib.rs               |  1 -
 cranelift/codegen/src/ref_slice.rs         | 18 ------------------
 cranelift/codegen/src/regalloc/virtregs.rs |  4 ++--
 5 files changed, 9 insertions(+), 24 deletions(-)
 delete mode 100644 cranelift/codegen/src/ref_slice.rs

diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index 7d8353dad3..ed28a83794 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -97,9 +97,14 @@ fn gen_instruction_data(registry: &FormatRegistry, fmt: &mut Formatter) {
 
 fn gen_arguments_method(registry: &FormatRegistry, fmt: &mut Formatter, is_mut: bool) {
     let (method, mut_, rslice, as_slice) = if is_mut {
-        ("arguments_mut", "mut ", "ref_slice_mut", "as_mut_slice")
+        (
+            "arguments_mut",
+            "mut ",
+            "core::slice::from_mut",
+            "as_mut_slice",
+        )
     } else {
-        ("arguments", "", "ref_slice", "as_slice")
+        ("arguments", "", "core::slice::from_ref", "as_slice")
     };
 
     fmtln!(
diff --git a/cranelift/codegen/src/ir/instructions.rs b/cranelift/codegen/src/ir/instructions.rs
index 30e8d14689..b3c3d0d211 100644
--- a/cranelift/codegen/src/ir/instructions.rs
+++ b/cranelift/codegen/src/ir/instructions.rs
@@ -18,7 +18,6 @@ use crate::isa;
 
 use crate::bitset::BitSet;
 use crate::entity;
-use crate::ref_slice::{ref_slice, ref_slice_mut};
 
 /// Some instructions use an external list of argument values because there is not enough space in
 /// the 16-byte `InstructionData` struct. These value lists are stored in a memory pool in
diff --git a/cranelift/codegen/src/lib.rs b/cranelift/codegen/src/lib.rs
index 1e97dfd6e0..73f2114561 100644
--- a/cranelift/codegen/src/lib.rs
+++ b/cranelift/codegen/src/lib.rs
@@ -95,7 +95,6 @@ mod partition_slice;
 mod postopt;
 mod predicates;
 mod redundant_reload_remover;
-mod ref_slice;
 mod regalloc;
 mod result;
 mod scoped_hash_map;
diff --git a/cranelift/codegen/src/ref_slice.rs b/cranelift/codegen/src/ref_slice.rs
deleted file mode 100644
index 2fad921cea..0000000000
--- a/cranelift/codegen/src/ref_slice.rs
+++ /dev/null
@@ -1,18 +0,0 @@
-//! Functions for converting a reference into a singleton slice.
-//!
-//! See also the [`ref_slice` crate](https://crates.io/crates/ref_slice).
-//!
-//! We define the functions here to avoid external dependencies, and to ensure that they are
-//! inlined in this crate.
-//!
-//! Despite their using an unsafe block, these functions are completely safe.
-
-use core::slice;
-
-pub fn ref_slice(s: &T) -> &[T] {
-    unsafe { slice::from_raw_parts(s, 1) }
-}
-
-pub fn ref_slice_mut(s: &mut T) -> &mut [T] {
-    unsafe { slice::from_raw_parts_mut(s, 1) }
-}
diff --git a/cranelift/codegen/src/regalloc/virtregs.rs b/cranelift/codegen/src/regalloc/virtregs.rs
index fc267c3f61..df2b480490 100644
--- a/cranelift/codegen/src/regalloc/virtregs.rs
+++ b/cranelift/codegen/src/regalloc/virtregs.rs
@@ -18,9 +18,9 @@ use crate::entity::{EntityList, ListPool};
 use crate::entity::{Keys, PrimaryMap, SecondaryMap};
 use crate::ir::{Function, Value};
 use crate::packed_option::PackedOption;
-use crate::ref_slice::ref_slice;
 use core::cmp::Ordering;
 use core::fmt;
+use core::slice;
 use smallvec::SmallVec;
 use std::vec::Vec;
 
@@ -104,7 +104,7 @@ impl VirtRegs {
         'a: 'b,
     {
         self.get(*value)
-            .map_or_else(|| ref_slice(value), |vr| self.values(vr))
+            .map_or_else(|| slice::from_ref(value), |vr| self.values(vr))
     }
 
     /// Check if `a` and `b` belong to the same congruence class.

From 26accbadf8218a8ffde222ffa7673d15fa981e55 Mon Sep 17 00:00:00 2001
From: Anthony Ramine 
Date: Thu, 19 Sep 2019 12:24:39 +0200
Subject: [PATCH 2744/3084] Use f32::to_bits and f64::to_bits

---
 cranelift/codegen/src/ir/immediates.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index 371c05bdd9..c46b5ac0d3 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -809,7 +809,7 @@ impl Ieee32 {
 
     /// Create a new `Ieee32` representing the number `x`.
     pub fn with_float(x: f32) -> Self {
-        Ieee32(unsafe { mem::transmute(x) })
+        Ieee32(x.to_bits())
     }
 
     /// Get the bitwise representation.
@@ -882,7 +882,7 @@ impl Ieee64 {
 
     /// Create a new `Ieee64` representing the number `x`.
     pub fn with_float(x: f64) -> Self {
-        Ieee64(unsafe { mem::transmute(x) })
+        Ieee64(x.to_bits())
     }
 
     /// Get the bitwise representation.

From eeb3159fe9bb758e768e1a843101eb01c3152aac Mon Sep 17 00:00:00 2001
From: Anthony Ramine 
Date: Sun, 22 Sep 2019 14:31:54 +0200
Subject: [PATCH 2745/3084] Make wasm_unsupported be composeable

---
 cranelift/wasm/src/code_translator.rs     |  6 ++--
 cranelift/wasm/src/environ/spec.rs        |  2 +-
 cranelift/wasm/src/func_translator.rs     |  2 +-
 cranelift/wasm/src/sections_translator.rs | 34 +++++++++++++++++++----
 cranelift/wasm/src/translation_utils.rs   | 17 +++++++-----
 5 files changed, 43 insertions(+), 18 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index fef3215c52..0bf7f4b0b4 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -907,7 +907,7 @@ pub fn translate_operator(
         | Operator::I64AtomicRmw16UCmpxchg { .. }
         | Operator::I64AtomicRmw32UCmpxchg { .. }
         | Operator::Fence { .. } => {
-            wasm_unsupported!("proposed thread operator {:?}", op);
+            return Err(wasm_unsupported!("proposed thread operator {:?}", op));
         }
         Operator::MemoryInit { .. }
         | Operator::DataDrop { .. }
@@ -920,7 +920,7 @@ pub fn translate_operator(
         | Operator::TableSet { .. }
         | Operator::TableGrow { .. }
         | Operator::TableSize { .. } => {
-            wasm_unsupported!("proposed bulk memory operator {:?}", op);
+            return Err(wasm_unsupported!("proposed bulk memory operator {:?}", op));
         }
         Operator::V128Const { value } => {
             let handle = builder.func.dfg.constants.insert(value.bytes().to_vec());
@@ -1109,7 +1109,7 @@ pub fn translate_operator(
         | Operator::I16x8LoadSplat { .. }
         | Operator::I32x4LoadSplat { .. }
         | Operator::I64x2LoadSplat { .. } => {
-            wasm_unsupported!("proposed SIMD operator {:?}", op);
+            return Err(wasm_unsupported!("proposed SIMD operator {:?}", op));
         }
     };
     Ok(())
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index ba87cc992a..bce06e4cd0 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -80,7 +80,7 @@ pub enum WasmError {
 /// on the arguments to this macro.
 #[macro_export]
 macro_rules! wasm_unsupported {
-    ($($arg:tt)*) => { return Err($crate::environ::WasmError::Unsupported(format!($($arg)*))) }
+    ($($arg:tt)*) => { $crate::environ::WasmError::Unsupported(format!($($arg)*)) }
 }
 
 impl From for WasmError {
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index cced9d87e6..ef1eecfa57 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -184,7 +184,7 @@ fn declare_locals(
             builder.ins().vconst(ir::types::I8X16, constant_handle)
         }
         AnyRef => builder.ins().null(environ.reference_type()),
-        ty => wasm_unsupported!("unsupported local type {:?}", ty),
+        ty => return Err(wasm_unsupported!("unsupported local type {:?}", ty)),
     };
 
     let ty = builder.func.dfg.value_type(zeroval);
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index c8c66529ad..a961a71ca9 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -53,7 +53,12 @@ pub fn parse_type_section(
                 }));
                 environ.declare_signature(sig)?;
             }
-            ty => wasm_unsupported!("unsupported type in type section: {:?}", ty),
+            ty => {
+                return Err(wasm_unsupported!(
+                    "unsupported type in type section: {:?}",
+                    ty
+                ))
+            }
         }
     }
     Ok(())
@@ -209,7 +214,10 @@ pub fn parse_global_section(
                 GlobalInit::GetGlobal(GlobalIndex::from_u32(global_index))
             }
             ref s => {
-                wasm_unsupported!("unsupported init expr in global section: {:?}", s);
+                return Err(wasm_unsupported!(
+                    "unsupported init expr in global section: {:?}",
+                    s
+                ));
             }
         };
         let global = Global {
@@ -284,7 +292,10 @@ pub fn parse_element_section<'data>(
                     (Some(GlobalIndex::from_u32(global_index)), 0)
                 }
                 ref s => {
-                    wasm_unsupported!("unsupported init expr in element section: {:?}", s);
+                    return Err(wasm_unsupported!(
+                        "unsupported init expr in element section: {:?}",
+                        s
+                    ));
                 }
             };
             let items_reader = items.get_items_reader()?;
@@ -300,7 +311,10 @@ pub fn parse_element_section<'data>(
                 elems.into_boxed_slice(),
             )?
         } else {
-            wasm_unsupported!("unsupported passive elements section: {:?}", kind);
+            return Err(wasm_unsupported!(
+                "unsupported passive elements section: {:?}",
+                kind
+            ));
         }
     }
     Ok(())
@@ -340,7 +354,12 @@ pub fn parse_data_section<'data>(
                 Operator::GetGlobal { global_index } => {
                     (Some(GlobalIndex::from_u32(global_index)), 0)
                 }
-                ref s => wasm_unsupported!("unsupported init expr in data section: {:?}", s),
+                ref s => {
+                    return Err(wasm_unsupported!(
+                        "unsupported init expr in data section: {:?}",
+                        s
+                    ))
+                }
             };
             environ.declare_data_initialization(
                 MemoryIndex::from_u32(memory_index),
@@ -349,7 +368,10 @@ pub fn parse_data_section<'data>(
                 data,
             )?;
         } else {
-            wasm_unsupported!("unsupported passive data section: {:?}", kind);
+            return Err(wasm_unsupported!(
+                "unsupported passive data section: {:?}",
+                kind
+            ));
         }
     }
 
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index 95553c6e48..df910c534b 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -124,7 +124,7 @@ pub fn type_to_type(ty: wasmparser::Type) -> WasmResult {
         wasmparser::Type::F32 => Ok(ir::types::F32),
         wasmparser::Type::F64 => Ok(ir::types::F64),
         wasmparser::Type::V128 => Ok(ir::types::I8X16),
-        ty => wasm_unsupported!("type_to_type: wasm type {:?}", ty),
+        ty => Err(wasm_unsupported!("type_to_type: wasm type {:?}", ty)),
     }
 }
 
@@ -138,7 +138,10 @@ pub fn tabletype_to_type(ty: wasmparser::Type) -> WasmResult> {
         wasmparser::Type::F64 => Ok(Some(ir::types::F64)),
         wasmparser::Type::V128 => Ok(Some(ir::types::I8X16)),
         wasmparser::Type::AnyFunc => Ok(None),
-        ty => wasm_unsupported!("tabletype_to_type: table wasm type {:?}", ty),
+        ty => Err(wasm_unsupported!(
+            "tabletype_to_type: table wasm type {:?}",
+            ty
+        )),
     }
 }
 
@@ -152,12 +155,12 @@ pub fn blocktype_to_type(ty_or_ft: wasmparser::TypeOrFuncType) -> WasmResult Ok(Some(ir::types::F64)),
             wasmparser::Type::V128 => Ok(Some(ir::types::I8X16)),
             wasmparser::Type::EmptyBlockType => Ok(None),
-            ty => wasm_unsupported!("blocktype_to_type: type {:?}", ty),
+            ty => Err(wasm_unsupported!("blocktype_to_type: type {:?}", ty)),
         },
-        wasmparser::TypeOrFuncType::FuncType(_) => wasm_unsupported!(
+        wasmparser::TypeOrFuncType::FuncType(_) => Err(wasm_unsupported!(
             "blocktype_to_type: multi-value block signature {:?}",
             ty_or_ft
-        ),
+        )),
     }
 }
 
@@ -181,10 +184,10 @@ pub fn num_return_values(ty: wasmparser::TypeOrFuncType) -> WasmResult {
             | wasmparser::Type::I64
             | wasmparser::Type::F64
             | wasmparser::Type::V128 => Ok(1),
-            ty => wasm_unsupported!("unsupported return value type {:?}", ty),
+            ty => Err(wasm_unsupported!("unsupported return value type {:?}", ty)),
         },
         wasmparser::TypeOrFuncType::FuncType(_) => {
-            wasm_unsupported!("multi-value block signature {:?}", ty);
+            Err(wasm_unsupported!("multi-value block signature {:?}", ty))
         }
     }
 }

From 3ab8eed7d327793b28b777a61b95b96276d2f695 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 23 Sep 2019 12:57:47 +0200
Subject: [PATCH 2746/3084] Remove unused core::mem import;

---
 cranelift/codegen/src/ir/immediates.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index c46b5ac0d3..408fe17cf3 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -6,7 +6,6 @@
 
 use core::fmt::{self, Display, Formatter};
 use core::iter::FromIterator;
-use core::mem;
 use core::str::{from_utf8, FromStr};
 use core::{i32, u32};
 use std::vec::Vec;
@@ -931,6 +930,7 @@ impl IntoBytes for Ieee64 {
 mod tests {
     use super::*;
     use core::fmt::Display;
+    use core::mem;
     use core::str::FromStr;
     use core::{f32, f64};
     use std::string::ToString;

From c2587c9d619903deafa974ba06f6b96a0bc1fb5f Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 17 Sep 2019 10:52:31 +0200
Subject: [PATCH 2747/3084] [meta] Remove Literal's kind field;

---
 cranelift/codegen/meta/src/cdsl/ast.rs | 121 ++++++++++++-------------
 1 file changed, 58 insertions(+), 63 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 798ee29d00..5c6256836f 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -8,7 +8,6 @@ use cranelift_entity::{entity_impl, PrimaryMap};
 
 use std::fmt;
 
-#[derive(Debug)]
 pub enum Expr {
     Var(VarIndex),
     Literal(Literal),
@@ -95,54 +94,37 @@ impl DefPool {
 pub struct DefIndex(u32);
 entity_impl!(DefIndex);
 
-#[derive(Debug, Clone)]
-enum LiteralValue {
+#[derive(Clone, Debug)]
+pub enum Literal {
     /// A value of an enumerated immediate operand.
     ///
     /// Some immediate operand kinds like `intcc` and `floatcc` have an enumerated range of values
     /// corresponding to a Rust enum type. An `Enumerator` object is an AST leaf node representing one
     /// of the values.
-    Enumerator(&'static str),
+    Enumerator {
+        rust_type: String,
+        value: &'static str,
+    },
 
     /// A bitwise value of an immediate operand, used for bitwise exact floating point constants.
-    Bits(u64),
+    Bits { rust_type: String, value: u64 },
 
     /// A value of an integer immediate operand.
     Int(i64),
 }
 
-#[derive(Clone)]
-pub struct Literal {
-    kind: OperandKind,
-    value: LiteralValue,
-}
-
-impl fmt::Debug for Literal {
-    fn fmt(&self, fmt: &mut fmt::Formatter) -> Result<(), fmt::Error> {
-        write!(
-            fmt,
-            "Literal(kind={}, value={:?})",
-            self.kind.name, self.value
-        )
-    }
-}
-
 impl Literal {
     pub fn enumerator_for(kind: &OperandKind, value: &'static str) -> Self {
-        if let OperandKindFields::ImmEnum(values) = &kind.fields {
-            assert!(
-                values.get(value).is_some(),
-                format!(
-                    "nonexistent value '{}' in enumeration '{}'",
-                    value, kind.name
-                )
-            );
-        } else {
-            panic!("enumerator is for enum values");
-        }
-        Self {
-            kind: kind.clone(),
-            value: LiteralValue::Enumerator(value),
+        let value = match &kind.fields {
+            OperandKindFields::ImmEnum(values) => values.get(value).expect(&format!(
+                "nonexistent value '{}' in enumeration '{}'",
+                value, kind.name
+            )),
+            _ => panic!("enumerator is for enum values"),
+        };
+        Literal::Enumerator {
+            rust_type: kind.rust_type.clone(),
+            value,
         }
     }
 
@@ -151,36 +133,25 @@ impl Literal {
             OperandKindFields::ImmValue => {}
             _ => panic!("bits_of is for immediate scalar types"),
         }
-        Self {
-            kind: kind.clone(),
-            value: LiteralValue::Bits(bits),
+        Literal::Bits {
+            rust_type: kind.rust_type.clone(),
+            value: bits,
         }
     }
 
     pub fn constant(kind: &OperandKind, value: i64) -> Self {
         match kind.fields {
             OperandKindFields::ImmValue => {}
-            _ => panic!("bits_of is for immediate scalar types"),
-        }
-        Self {
-            kind: kind.clone(),
-            value: LiteralValue::Int(value),
+            _ => panic!("constant is for immediate scalar types"),
         }
+        Literal::Int(value)
     }
 
     pub fn to_rust_code(&self) -> String {
-        let maybe_values = match &self.kind.fields {
-            OperandKindFields::ImmEnum(values) => Some(values),
-            OperandKindFields::ImmValue => None,
-            _ => panic!("impossible per construction"),
-        };
-
-        match self.value {
-            LiteralValue::Enumerator(value) => {
-                format!("{}::{}", self.kind.rust_type, maybe_values.unwrap()[value])
-            }
-            LiteralValue::Bits(bits) => format!("{}::with_bits({:#x})", self.kind.rust_type, bits),
-            LiteralValue::Int(val) => val.to_string(),
+        match self {
+            Literal::Enumerator { rust_type, value } => format!("{}::{}", rust_type, value),
+            Literal::Bits { rust_type, value } => format!("{}::with_bits({:#x})", rust_type, value),
+            Literal::Int(val) => val.to_string(),
         }
     }
 }
@@ -364,7 +335,6 @@ impl VarPool {
 ///
 /// An `Apply` AST expression is created by using function call syntax on instructions. This
 /// applies to both bound and unbound polymorphic instructions.
-#[derive(Debug)]
 pub struct Apply {
     pub inst: Instruction,
     pub args: Vec,
@@ -395,13 +365,38 @@ impl Apply {
             let arg = &args[imm_index];
             if let Some(literal) = arg.maybe_literal() {
                 let op = &inst.operands_in[imm_index];
-                assert!(
-                    op.kind.name == literal.kind.name,
-                    format!(
-                        "Passing literal of kind {} to field of wrong kind {}",
-                        literal.kind.name, op.kind.name
-                    )
-                );
+                match &op.kind.fields {
+                    OperandKindFields::ImmEnum(values) => {
+                        if let Literal::Enumerator { value, .. } = literal {
+                            assert!(
+                                values.iter().any(|(_key, v)| v == value),
+                                "Nonexistent enum value '{}' passed to field of kind '{}' -- \
+                                 did you use the right enum?",
+                                value,
+                                op.kind.name
+                            );
+                        } else {
+                            panic!(
+                                "Passed non-enum field value {:?} to field of kind {}",
+                                literal, op.kind.name
+                            );
+                        }
+                    }
+                    OperandKindFields::ImmValue => match &literal {
+                        Literal::Enumerator { value, .. } => panic!(
+                            "Expected immediate value in immediate field of kind '{}', \
+                             obtained enum value '{}'",
+                            op.kind.name, value
+                        ),
+                        Literal::Bits { .. } | Literal::Int(_) => {}
+                    },
+                    _ => {
+                        panic!(
+                            "Literal passed to non-literal field of kind {}",
+                            op.kind.name
+                        );
+                    }
+                }
             }
         }
 

From f0244516c543ce8d3bc5a386ed52b440581c7dcb Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 17 Sep 2019 10:57:28 +0200
Subject: [PATCH 2748/3084] [meta] Make more things pub(crate) instead of pub;

This could help the compiler find unused fields/methods. It didn't find any
during this migration.
---
 cranelift/codegen/meta/src/cdsl/ast.rs        | 30 +++++++++----------
 cranelift/codegen/meta/src/cdsl/cpu_modes.rs  |  2 +-
 cranelift/codegen/meta/src/cdsl/encodings.rs  |  6 ++--
 cranelift/codegen/meta/src/cdsl/isa.rs        |  2 +-
 .../codegen/meta/src/cdsl/type_inference.rs   |  4 +--
 cranelift/codegen/meta/src/cdsl/xform.rs      | 10 +++----
 cranelift/codegen/meta/src/constant_hash.rs   |  4 +--
 cranelift/codegen/meta/src/default_map.rs     |  2 +-
 cranelift/codegen/meta/src/gen_binemit.rs     |  2 +-
 cranelift/codegen/meta/src/gen_legalizer.rs   |  2 +-
 cranelift/codegen/meta/src/gen_registers.rs   |  2 +-
 cranelift/codegen/meta/src/gen_settings.rs    |  2 +-
 cranelift/codegen/meta/src/gen_types.rs       |  2 +-
 .../codegen/meta/src/isa/riscv/encodings.rs   |  2 +-
 .../codegen/meta/src/isa/x86/encodings.rs     |  2 +-
 cranelift/codegen/meta/src/isa/x86/recipes.rs |  2 +-
 16 files changed, 38 insertions(+), 38 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 5c6256836f..b2b1e2f3e8 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -8,7 +8,7 @@ use cranelift_entity::{entity_impl, PrimaryMap};
 
 use std::fmt;
 
-pub enum Expr {
+pub(crate) enum Expr {
     Var(VarIndex),
     Literal(Literal),
 }
@@ -43,7 +43,7 @@ impl Expr {
 }
 
 /// An AST definition associates a set of variables with the values produced by an expression.
-pub struct Def {
+pub(crate) struct Def {
     pub apply: Apply,
     pub defined_vars: Vec,
 }
@@ -66,7 +66,7 @@ impl Def {
     }
 }
 
-pub struct DefPool {
+pub(crate) struct DefPool {
     pool: PrimaryMap,
 }
 
@@ -91,11 +91,11 @@ impl DefPool {
 }
 
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
-pub struct DefIndex(u32);
+pub(crate) struct DefIndex(u32);
 entity_impl!(DefIndex);
 
 #[derive(Clone, Debug)]
-pub enum Literal {
+pub(crate) enum Literal {
     /// A value of an enumerated immediate operand.
     ///
     /// Some immediate operand kinds like `intcc` and `floatcc` have an enumerated range of values
@@ -157,7 +157,7 @@ impl Literal {
 }
 
 #[derive(Clone, Copy, Debug)]
-pub enum PatternPosition {
+pub(crate) enum PatternPosition {
     Source,
     Destination,
 }
@@ -179,7 +179,7 @@ pub enum PatternPosition {
 /// deleted immediately.
 ///
 /// Temporary values are defined only in the destination pattern.
-pub struct Var {
+pub(crate) struct Var {
     pub name: &'static str,
 
     /// The `Def` defining this variable in a source pattern.
@@ -307,10 +307,10 @@ impl fmt::Debug for Var {
 }
 
 #[derive(Copy, Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
-pub struct VarIndex(u32);
+pub(crate) struct VarIndex(u32);
 entity_impl!(VarIndex);
 
-pub struct VarPool {
+pub(crate) struct VarPool {
     pool: PrimaryMap,
 }
 
@@ -335,7 +335,7 @@ impl VarPool {
 ///
 /// An `Apply` AST expression is created by using function call syntax on instructions. This
 /// applies to both bound and unbound polymorphic instructions.
-pub struct Apply {
+pub(crate) struct Apply {
     pub inst: Instruction,
     pub args: Vec,
     pub value_types: Vec,
@@ -506,14 +506,14 @@ impl Apply {
 
 // Simple helpers for legalize actions construction.
 
-pub enum DummyExpr {
+pub(crate) enum DummyExpr {
     Var(DummyVar),
     Literal(Literal),
     Apply(InstSpec, Vec),
 }
 
 #[derive(Clone)]
-pub struct DummyVar {
+pub(crate) struct DummyVar {
     pub name: &'static str,
 }
 
@@ -528,16 +528,16 @@ impl Into for Literal {
     }
 }
 
-pub fn var(name: &'static str) -> DummyVar {
+pub(crate) fn var(name: &'static str) -> DummyVar {
     DummyVar { name }
 }
 
-pub struct DummyDef {
+pub(crate) struct DummyDef {
     pub expr: DummyExpr,
     pub defined_vars: Vec,
 }
 
-pub struct ExprBuilder {
+pub(crate) struct ExprBuilder {
     expr: DummyExpr,
 }
 
diff --git a/cranelift/codegen/meta/src/cdsl/cpu_modes.rs b/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
index 4b4406d2d3..6436d2e4a2 100644
--- a/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
+++ b/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
@@ -5,7 +5,7 @@ use crate::cdsl::encodings::Encoding;
 use crate::cdsl::types::{LaneType, ValueType};
 use crate::cdsl::xform::{TransformGroup, TransformGroupIndex};
 
-pub struct CpuMode {
+pub(crate) struct CpuMode {
     pub name: &'static str,
     default_legalize: Option,
     monomorphic_legalize: Option,
diff --git a/cranelift/codegen/meta/src/cdsl/encodings.rs b/cranelift/codegen/meta/src/cdsl/encodings.rs
index cd05eb9b38..77d6bc7374 100644
--- a/cranelift/codegen/meta/src/cdsl/encodings.rs
+++ b/cranelift/codegen/meta/src/cdsl/encodings.rs
@@ -20,7 +20,7 @@ use crate::cdsl::types::ValueType;
 /// 3. With operands providing constraints: `icmp.i32(intcc.eq, x, y)`.
 ///
 /// If the instruction is polymorphic, all type variables must be provided.
-pub struct EncodingContent {
+pub(crate) struct EncodingContent {
     /// The `Instruction` or `BoundInstruction` being encoded.
     inst: InstSpec,
 
@@ -49,9 +49,9 @@ impl EncodingContent {
     }
 }
 
-pub type Encoding = Rc;
+pub(crate) type Encoding = Rc;
 
-pub struct EncodingBuilder {
+pub(crate) struct EncodingBuilder {
     inst: InstSpec,
     recipe: EncodingRecipeNumber,
     encbits: u16,
diff --git a/cranelift/codegen/meta/src/cdsl/isa.rs b/cranelift/codegen/meta/src/cdsl/isa.rs
index e561ce2e9c..512105d09a 100644
--- a/cranelift/codegen/meta/src/cdsl/isa.rs
+++ b/cranelift/codegen/meta/src/cdsl/isa.rs
@@ -8,7 +8,7 @@ use crate::cdsl::regs::IsaRegs;
 use crate::cdsl::settings::SettingGroup;
 use crate::cdsl::xform::{TransformGroupIndex, TransformGroups};
 
-pub struct TargetIsa {
+pub(crate) struct TargetIsa {
     pub name: &'static str,
     pub instructions: InstructionGroup,
     pub settings: SettingGroup,
diff --git a/cranelift/codegen/meta/src/cdsl/type_inference.rs b/cranelift/codegen/meta/src/cdsl/type_inference.rs
index a56d81463c..5181739969 100644
--- a/cranelift/codegen/meta/src/cdsl/type_inference.rs
+++ b/cranelift/codegen/meta/src/cdsl/type_inference.rs
@@ -122,7 +122,7 @@ enum TypeEnvRank {
 }
 
 /// Class encapsulating the necessary bookkeeping for type inference.
-pub struct TypeEnvironment {
+pub(crate) struct TypeEnvironment {
     vars: HashSet,
     ranks: HashMap,
     equivalency_map: HashMap,
@@ -602,7 +602,7 @@ fn infer_definition(
 }
 
 /// Perform type inference on an transformation. Return an updated type environment or error.
-pub fn infer_transform(
+pub(crate) fn infer_transform(
     src: DefIndex,
     dst: &Vec,
     def_pool: &DefPool,
diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index b90d552b9a..6875fa4518 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -17,7 +17,7 @@ use std::iter::FromIterator;
 /// cases when it applies.
 ///
 /// The source pattern can contain only a single instruction.
-pub struct Transform {
+pub(crate) struct Transform {
     pub src: DefIndex,
     pub dst: Vec,
     pub var_pool: VarPool,
@@ -268,7 +268,7 @@ fn rewrite_def_list(
 }
 
 /// A group of related transformations.
-pub struct TransformGroup {
+pub(crate) struct TransformGroup {
     pub name: &'static str,
     pub doc: &'static str,
     pub chain_with: Option,
@@ -294,10 +294,10 @@ impl TransformGroup {
 }
 
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
-pub struct TransformGroupIndex(u32);
+pub(crate) struct TransformGroupIndex(u32);
 entity_impl!(TransformGroupIndex);
 
-pub struct TransformGroupBuilder {
+pub(crate) struct TransformGroupBuilder {
     name: &'static str,
     doc: &'static str,
     chain_with: Option,
@@ -369,7 +369,7 @@ impl TransformGroupBuilder {
     }
 }
 
-pub struct TransformGroups {
+pub(crate) struct TransformGroups {
     groups: PrimaryMap,
 }
 
diff --git a/cranelift/codegen/meta/src/constant_hash.rs b/cranelift/codegen/meta/src/constant_hash.rs
index 4090e4d77d..ce6214efa8 100644
--- a/cranelift/codegen/meta/src/constant_hash.rs
+++ b/cranelift/codegen/meta/src/constant_hash.rs
@@ -1,6 +1,6 @@
 use std::iter;
 
-pub fn simple_hash(s: &str) -> usize {
+pub(crate) fn simple_hash(s: &str) -> usize {
     let mut h: u32 = 5381;
     for c in s.chars() {
         h = (h ^ c as u32).wrapping_add(h.rotate_right(6));
@@ -11,7 +11,7 @@ pub fn simple_hash(s: &str) -> usize {
 /// Compute an open addressed, quadratically probed hash table containing
 /// `items`. The returned table is a list containing the elements of the
 /// iterable `items` and `None` in unused slots.
-pub fn generate_table<'cont, T, I: iter::Iterator, H: Fn(&T) -> usize>(
+pub(crate) fn generate_table<'cont, T, I: iter::Iterator, H: Fn(&T) -> usize>(
     items: I,
     num_items: usize,
     hash_function: H,
diff --git a/cranelift/codegen/meta/src/default_map.rs b/cranelift/codegen/meta/src/default_map.rs
index da6d471439..0a2f7fc41a 100644
--- a/cranelift/codegen/meta/src/default_map.rs
+++ b/cranelift/codegen/meta/src/default_map.rs
@@ -1,7 +1,7 @@
 use std::collections::HashMap;
 use std::hash::Hash;
 
-pub trait MapWithDefault {
+pub(crate) trait MapWithDefault {
     fn get_or_default(&mut self, k: K) -> &mut V;
 }
 
diff --git a/cranelift/codegen/meta/src/gen_binemit.rs b/cranelift/codegen/meta/src/gen_binemit.rs
index 53929fe8c5..fed145cbd4 100644
--- a/cranelift/codegen/meta/src/gen_binemit.rs
+++ b/cranelift/codegen/meta/src/gen_binemit.rs
@@ -211,7 +211,7 @@ fn gen_isa(formats: &FormatRegistry, isa_name: &str, recipes: &Recipes, fmt: &mu
     fmt.line("}");
 }
 
-pub fn generate(
+pub(crate) fn generate(
     formats: &FormatRegistry,
     isa_name: &str,
     recipes: &Recipes,
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index da1fb1f58f..4153fb8c32 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -577,7 +577,7 @@ fn gen_isa(
 }
 
 /// Generate the legalizer files.
-pub fn generate(
+pub(crate) fn generate(
     isas: &Vec,
     format_registry: &FormatRegistry,
     transform_groups: &TransformGroups,
diff --git a/cranelift/codegen/meta/src/gen_registers.rs b/cranelift/codegen/meta/src/gen_registers.rs
index 08edfa9e21..8f778f8601 100644
--- a/cranelift/codegen/meta/src/gen_registers.rs
+++ b/cranelift/codegen/meta/src/gen_registers.rs
@@ -133,7 +133,7 @@ fn gen_isa(isa: &TargetIsa, fmt: &mut Formatter) {
     fmtln!(fmt, "}");
 }
 
-pub fn generate(isa: &TargetIsa, filename: &str, out_dir: &str) -> Result<(), error::Error> {
+pub(crate) fn generate(isa: &TargetIsa, filename: &str, out_dir: &str) -> Result<(), error::Error> {
     let mut fmt = Formatter::new();
     gen_isa(&isa, &mut fmt);
     fmt.update_file(filename, out_dir)?;
diff --git a/cranelift/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
index cdd07d78bd..84c81cddc4 100644
--- a/cranelift/codegen/meta/src/gen_settings.rs
+++ b/cranelift/codegen/meta/src/gen_settings.rs
@@ -431,7 +431,7 @@ fn gen_group(group: &SettingGroup, parent: ParentGroup, fmt: &mut Formatter) {
     gen_display(group, fmt);
 }
 
-pub fn generate(
+pub(crate) fn generate(
     settings: &SettingGroup,
     parent_group: ParentGroup,
     filename: &str,
diff --git a/cranelift/codegen/meta/src/gen_types.rs b/cranelift/codegen/meta/src/gen_types.rs
index d4b4c60d65..6ced212b8d 100644
--- a/cranelift/codegen/meta/src/gen_types.rs
+++ b/cranelift/codegen/meta/src/gen_types.rs
@@ -68,7 +68,7 @@ fn emit_types(fmt: &mut srcgen::Formatter) -> Result<(), error::Error> {
 }
 
 /// Generate the types file.
-pub fn generate(filename: &str, out_dir: &str) -> Result<(), error::Error> {
+pub(crate) fn generate(filename: &str, out_dir: &str) -> Result<(), error::Error> {
     let mut fmt = srcgen::Formatter::new();
     emit_types(&mut fmt)?;
     fmt.update_file(filename, out_dir)?;
diff --git a/cranelift/codegen/meta/src/isa/riscv/encodings.rs b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
index 02a3bc7b61..6ddc13b0a3 100644
--- a/cranelift/codegen/meta/src/isa/riscv/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
@@ -18,7 +18,7 @@ fn enc(inst: impl Into, recipe: EncodingRecipeNumber, bits: u16) -> En
     EncodingBuilder::new(inst.into(), recipe, bits)
 }
 
-pub struct PerCpuModeEncodings<'defs> {
+pub(crate) struct PerCpuModeEncodings<'defs> {
     pub inst_pred_reg: InstructionPredicateRegistry,
     pub enc32: Vec,
     pub enc64: Vec,
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 741f77d755..d8b070b69e 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -18,7 +18,7 @@ use crate::shared::Definitions as SharedDefinitions;
 
 use super::recipes::{RecipeGroup, Template};
 
-pub struct PerCpuModeEncodings {
+pub(crate) struct PerCpuModeEncodings {
     pub enc32: Vec,
     pub enc64: Vec,
     pub recipes: Recipes,
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index bee51883a4..b30a35e3f5 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -13,7 +13,7 @@ use crate::shared::Definitions as SharedDefinitions;
 /// Helper data structure to create recipes and template recipes.
 /// It contains all the recipes and recipe templates that might be used in the encodings crate of
 /// this same directory.
-pub struct RecipeGroup<'builder> {
+pub(crate) struct RecipeGroup<'builder> {
     /// Memoized format pointer, to pass it to builders later.
     formats: &'builder FormatRegistry,
 

From 947fce194ea1103920c01533babe9e66a55dceb9 Mon Sep 17 00:00:00 2001
From: Erin Power 
Date: Sat, 31 Aug 2019 10:09:14 +0200
Subject: [PATCH 2749/3084] Replaced instances of SparseSet with EntitySet

---
 cranelift/bforest/src/pool.rs           | 16 +++--
 cranelift/codegen/src/topo_order.rs     |  9 +--
 cranelift/codegen/src/verifier/flags.rs |  4 +-
 cranelift/entity/src/set.rs             | 93 +++++++++++++++++++++++--
 4 files changed, 105 insertions(+), 17 deletions(-)

diff --git a/cranelift/bforest/src/pool.rs b/cranelift/bforest/src/pool.rs
index 1b51cfe56b..0848089eb6 100644
--- a/cranelift/bforest/src/pool.rs
+++ b/cranelift/bforest/src/pool.rs
@@ -83,7 +83,7 @@ impl NodePool {
         NodeData: fmt::Display,
         F::Key: fmt::Display,
     {
-        use crate::entity::SparseSet;
+        use crate::entity::EntitySet;
         use core::borrow::Borrow;
         use core::cmp::Ordering;
         use std::vec::Vec;
@@ -94,7 +94,13 @@ impl NodePool {
             assert!(size > 0, "Root must have more than one sub-tree");
         }
 
-        let mut done = SparseSet::new();
+        let mut done = match self[node] {
+            NodeData::Inner { size, .. } | NodeData::Leaf { size, .. } => {
+                EntitySet::with_capacity(size.into())
+            }
+            _ => EntitySet::new(),
+        };
+
         let mut todo = Vec::new();
 
         // Todo-list entries are:
@@ -104,11 +110,7 @@ impl NodePool {
         todo.push((None, node, None));
 
         while let Some((lkey, node, rkey)) = todo.pop() {
-            assert_eq!(
-                done.insert(node),
-                None,
-                "Node appears more than once in tree"
-            );
+            assert!(done.insert(node), "Node appears more than once in tree");
             let mut lower = lkey;
 
             match self[node] {
diff --git a/cranelift/codegen/src/topo_order.rs b/cranelift/codegen/src/topo_order.rs
index 7dcec7274b..8824f4cc0a 100644
--- a/cranelift/codegen/src/topo_order.rs
+++ b/cranelift/codegen/src/topo_order.rs
@@ -1,7 +1,7 @@
 //! Topological order of EBBs, according to the dominator tree.
 
 use crate::dominator_tree::DominatorTree;
-use crate::entity::SparseSet;
+use crate::entity::EntitySet;
 use crate::ir::{Ebb, Layout};
 use std::vec::Vec;
 
@@ -19,7 +19,7 @@ pub struct TopoOrder {
     next: usize,
 
     /// Set of visited EBBs.
-    visited: SparseSet,
+    visited: EntitySet,
 
     /// Stack of EBBs to be visited next, already in `visited`.
     stack: Vec,
@@ -31,7 +31,7 @@ impl TopoOrder {
         Self {
             preferred: Vec::new(),
             next: 0,
-            visited: SparseSet::new(),
+            visited: EntitySet::new(),
             stack: Vec::new(),
         }
     }
@@ -64,6 +64,7 @@ impl TopoOrder {
     /// - All EBBs in the `preferred` iterator given to `reset` will be returned.
     /// - All dominators are visited before the EBB returned.
     pub fn next(&mut self, layout: &Layout, domtree: &DominatorTree) -> Option {
+        self.visited.resize(layout.ebb_capacity());
         // Any entries in `stack` should be returned immediately. They have already been added to
         // `visited`.
         while self.stack.is_empty() {
@@ -73,7 +74,7 @@ impl TopoOrder {
                     // We have the next EBB in the preferred order.
                     self.next += 1;
                     // Push it along with any non-visited dominators.
-                    while self.visited.insert(ebb).is_none() {
+                    while self.visited.insert(ebb) {
                         self.stack.push(ebb);
                         match domtree.idom(ebb) {
                             Some(idom) => ebb = layout.inst_ebb(idom).expect("idom not in layout"),
diff --git a/cranelift/codegen/src/verifier/flags.rs b/cranelift/codegen/src/verifier/flags.rs
index 43a235def5..d39b6e0487 100644
--- a/cranelift/codegen/src/verifier/flags.rs
+++ b/cranelift/codegen/src/verifier/flags.rs
@@ -1,6 +1,6 @@
 //! Verify CPU flags values.
 
-use crate::entity::{SecondaryMap, SparseSet};
+use crate::entity::{EntitySet, SecondaryMap};
 use crate::flowgraph::{BasicBlock, ControlFlowGraph};
 use crate::ir;
 use crate::ir::instructions::BranchInfo;
@@ -50,7 +50,7 @@ impl<'a> FlagsVerifier<'a> {
     fn check(&mut self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         // List of EBBs that need to be processed. EBBs may be re-added to this list when we detect
         // that one of their successor blocks needs a live-in flags value.
-        let mut worklist = SparseSet::new();
+        let mut worklist = EntitySet::with_capacity(self.func.layout.ebb_capacity());
         for ebb in self.func.layout.ebbs() {
             worklist.insert(ebb);
         }
diff --git a/cranelift/entity/src/set.rs b/cranelift/entity/src/set.rs
index a4759a1712..fa74ba453f 100644
--- a/cranelift/entity/src/set.rs
+++ b/cranelift/entity/src/set.rs
@@ -33,6 +33,14 @@ where
         }
     }
 
+    /// Creates a new empty set with the specified capacity.
+    pub fn with_capacity(capacity: usize) -> Self {
+        Self {
+            elems: Vec::with_capacity((capacity + 7) / 8),
+            ..Self::new()
+        }
+    }
+
     /// Get the element at `k` if it exists.
     pub fn contains(&self, k: K) -> bool {
         let index = k.index();
@@ -45,9 +53,11 @@ where
 
     /// Is this set completely empty?
     pub fn is_empty(&self) -> bool {
-        // Note that this implementation will become incorrect should it ever become possible
-        // to remove elements from an `EntitySet`.
-        self.len == 0
+        if self.len != 0 {
+            false
+        } else {
+            self.elems.iter().all(|&e| e == 0)
+        }
     }
 
     /// Returns the cardinality of the set.  More precisely, it returns the number of calls to
@@ -93,6 +103,34 @@ where
         self.elems[index / 8] |= 1 << (index % 8);
         result
     }
+
+    /// Removes and returns the entity from the set if it exists.
+    pub fn pop(&mut self) -> Option {
+        if self.len == 0 {
+            return None;
+        }
+
+        // Clear the last known entity in the list.
+        let last_index = self.len - 1;
+        self.elems[last_index / 8] &= !(1 << (last_index % 8));
+
+        // Set the length to the next last stored entity or zero if we pop'ed
+        // the last entity.
+        self.len = self
+            .elems
+            .iter()
+            .enumerate()
+            .rev()
+            .find(|(_, &byte)| byte != 0)
+            // Map `i` from byte index to bit level index.
+            // `(i + 1) * 8` = Last bit in byte.
+            // `last - byte.leading_zeros()` = last set bit in byte.
+            // `as usize` won't ever truncate as the potential range is `0..=8`.
+            .map(|(i, byte)| ((i + 1) * 8) - byte.leading_zeros() as usize)
+            .unwrap_or(0);
+
+        Some(K::new(last_index))
+    }
 }
 
 #[cfg(test)]
@@ -101,7 +139,7 @@ mod tests {
     use core::u32;
 
     // `EntityRef` impl for testing.
-    #[derive(Clone, Copy, Debug, PartialEq, Eq)]
+    #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
     struct E(u32);
 
     impl EntityRef for E {
@@ -159,4 +197,51 @@ mod tests {
         m.clear();
         assert!(m.is_empty());
     }
+
+    #[test]
+    fn pop_ordered() {
+        let r0 = E(0);
+        let r1 = E(1);
+        let r2 = E(2);
+        let mut m = EntitySet::new();
+        m.insert(r0);
+        m.insert(r1);
+        m.insert(r2);
+
+        assert_eq!(r2, m.pop().unwrap());
+        assert_eq!(r1, m.pop().unwrap());
+        assert_eq!(r0, m.pop().unwrap());
+        assert!(m.pop().is_none());
+        assert!(m.pop().is_none());
+    }
+
+    #[test]
+    fn pop_unordered() {
+        let mut ebbs = [
+            E(0),
+            E(1),
+            E(6),
+            E(7),
+            E(5),
+            E(9),
+            E(10),
+            E(2),
+            E(3),
+            E(11),
+            E(12),
+        ];
+
+        let mut m = EntitySet::new();
+        for &ebb in &ebbs {
+            m.insert(ebb);
+        }
+        assert_eq!(m.len, 13);
+        ebbs.sort();
+
+        for &ebb in ebbs.iter().rev() {
+            assert_eq!(ebb, m.pop().unwrap());
+        }
+
+        assert!(m.is_empty());
+    }
 }

From 4052bc04ee90f84395b275f24cfeb253ba27e118 Mon Sep 17 00:00:00 2001
From: Pat Hickey 
Date: Mon, 23 Sep 2019 16:47:13 -0700
Subject: [PATCH 2750/3084] cranelift-wasm: upgrade to wasmparser 0.39.1
 (#1068)

Uses new SectionContent api, which is simpler than matching on
SectionCode and then getting the right reader, & avoids possible panics
---
 cranelift/wasm/Cargo.toml               |  2 +-
 cranelift/wasm/src/module_translator.rs | 68 +++++++++++--------------
 2 files changed, 30 insertions(+), 40 deletions(-)

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index a87e4a4cce..71a2a7ca31 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -11,7 +11,7 @@ keywords = ["webassembly", "wasm"]
 edition = "2018"
 
 [dependencies]
-wasmparser = { version = "0.37.0", default-features = false }
+wasmparser = { version = "0.39.1", default-features = false }
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.43.1", default-features = false }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.43.1", default-features = false }
diff --git a/cranelift/wasm/src/module_translator.rs b/cranelift/wasm/src/module_translator.rs
index 8d8b55397d..462f79ab24 100644
--- a/cranelift/wasm/src/module_translator.rs
+++ b/cranelift/wasm/src/module_translator.rs
@@ -7,7 +7,7 @@ use crate::sections_translator::{
     parse_name_section, parse_start_section, parse_table_section, parse_type_section,
 };
 use cranelift_codegen::timing;
-use wasmparser::{CustomSectionKind, ModuleReader, SectionCode};
+use wasmparser::{CustomSectionContent, ModuleReader, SectionContent};
 
 /// Translate a sequence of bytes forming a valid Wasm binary into a list of valid Cranelift IR
 /// [`Function`](cranelift_codegen::ir::Function).
@@ -20,83 +20,73 @@ pub fn translate_module<'data>(
 
     while !reader.eof() {
         let section = reader.read()?;
-        match section.code {
-            SectionCode::Type => {
-                let types = section.get_type_section_reader()?;
+        match section.content()? {
+            SectionContent::Type(types) => {
                 parse_type_section(types, environ)?;
             }
 
-            SectionCode::Import => {
-                let imports = section.get_import_section_reader()?;
+            SectionContent::Import(imports) => {
                 parse_import_section(imports, environ)?;
             }
 
-            SectionCode::Function => {
-                let functions = section.get_function_section_reader()?;
+            SectionContent::Function(functions) => {
                 parse_function_section(functions, environ)?;
             }
 
-            SectionCode::Table => {
-                let tables = section.get_table_section_reader()?;
+            SectionContent::Table(tables) => {
                 parse_table_section(tables, environ)?;
             }
 
-            SectionCode::Memory => {
-                let memories = section.get_memory_section_reader()?;
+            SectionContent::Memory(memories) => {
                 parse_memory_section(memories, environ)?;
             }
 
-            SectionCode::Global => {
-                let globals = section.get_global_section_reader()?;
+            SectionContent::Global(globals) => {
                 parse_global_section(globals, environ)?;
             }
 
-            SectionCode::Export => {
-                let exports = section.get_export_section_reader()?;
+            SectionContent::Export(exports) => {
                 parse_export_section(exports, environ)?;
             }
 
-            SectionCode::Start => {
-                let start = section.get_start_section_content()?;
+            SectionContent::Start(start) => {
                 parse_start_section(start, environ)?;
             }
 
-            SectionCode::Element => {
-                let elements = section.get_element_section_reader()?;
+            SectionContent::Element(elements) => {
                 parse_element_section(elements, environ)?;
             }
 
-            SectionCode::Code => {
-                let code = section.get_code_section_reader()?;
+            SectionContent::Code(code) => {
                 parse_code_section(code, environ)?;
             }
 
-            SectionCode::Data => {
-                let data = section.get_data_section_reader()?;
+            SectionContent::Data(data) => {
                 parse_data_section(data, environ)?;
             }
 
-            SectionCode::DataCount => {
+            SectionContent::DataCount(_) => {
                 return Err(WasmError::InvalidWebAssembly {
                     message: "don't know how to handle the data count section yet",
                     offset: reader.current_position(),
                 });
             }
 
-            SectionCode::Custom {
-                kind: CustomSectionKind::Name,
-                name: _,
-            } => {
-                let names = section.get_name_section_reader()?;
-                parse_name_section(names, environ)?;
-            }
-
-            SectionCode::Custom { name, kind: _ } => {
-                let mut reader = section.get_binary_reader();
-                let len = reader.bytes_remaining();
-                let payload = reader.read_bytes(len)?;
-                environ.custom_section(name, payload)?;
-            }
+            SectionContent::Custom {
+                name,
+                binary,
+                content,
+            } => match content {
+                Some(CustomSectionContent::Name(names)) => {
+                    parse_name_section(names, environ)?;
+                }
+                _ => {
+                    let mut reader = binary.clone();
+                    let len = reader.bytes_remaining();
+                    let payload = reader.read_bytes(len)?;
+                    environ.custom_section(name, payload)?;
+                }
+            },
         }
     }
 

From 614fb7b5268508ebeb7642ebe9780ff5d9f8e7a7 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Mon, 16 Sep 2019 15:39:03 +0200
Subject: [PATCH 2751/3084] Factor out the var_index function.

---
 cranelift/codegen/meta/src/cdsl/xform.rs | 55 ++++++++++++------------
 1 file changed, 28 insertions(+), 27 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index 6875fa4518..d0a543baec 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -138,6 +138,27 @@ impl Transform {
     }
 }
 
+/// Inserts, if not present, a name in the `symbol_table`. Then returns its index in the variable
+/// pool `var_pool`. If the variable was not present in the symbol table, then add it to the list of
+/// `defined_vars`.
+fn var_index(
+    name: &'static str,
+    symbol_table: &mut SymbolTable,
+    defined_vars: &mut Vec,
+    var_pool: &mut VarPool,
+) -> VarIndex {
+    match symbol_table.get(name) {
+        Some(&existing_var) => existing_var,
+        None => {
+            // Materialize the variable.
+            let new_var = var_pool.create(name);
+            symbol_table.insert(name, new_var);
+            defined_vars.push(new_var);
+            new_var
+        }
+    }
+}
+
 /// Given a list of symbols defined in a Def, rewrite them to local symbols. Yield the new locals.
 fn rewrite_defined_vars(
     position: PatternPosition,
@@ -149,16 +170,7 @@ fn rewrite_defined_vars(
 ) -> Vec {
     let mut new_defined_vars = Vec::new();
     for var in &dummy_def.defined_vars {
-        let own_var = match symbol_table.get(var.name) {
-            Some(&existing_var) => existing_var,
-            None => {
-                // Materialize the variable.
-                let new_var = var_pool.create(var.name);
-                symbol_table.insert(var.name, new_var);
-                defined_vars.push(new_var);
-                new_var
-            }
-        };
+        let own_var = var_index(var.name, symbol_table, defined_vars, var_pool);
         var_pool.get_mut(own_var).set_def(position, def_index);
         new_defined_vars.push(own_var);
     }
@@ -197,23 +209,12 @@ fn rewrite_expr(
     for (i, arg) in dummy_args.into_iter().enumerate() {
         match arg {
             DummyExpr::Var(var) => {
-                let own_var = match symbol_table.get(var.name) {
-                    Some(&own_var) => {
-                        let var = var_pool.get(own_var);
-                        assert!(
-                            var.is_input() || var.get_def(position).is_some(),
-                            format!("{:?} used as both input and def", var)
-                        );
-                        own_var
-                    }
-                    None => {
-                        // First time we're using this variable.
-                        let own_var = var_pool.create(var.name);
-                        symbol_table.insert(var.name, own_var);
-                        input_vars.push(own_var);
-                        own_var
-                    }
-                };
+                let own_var = var_index(var.name, symbol_table, input_vars, var_pool);
+                let var = var_pool.get(own_var);
+                assert!(
+                    var.is_input() || var.get_def(position).is_some(),
+                    format!("{:?} used as both input and def", var)
+                );
                 args.push(Expr::Var(own_var));
             }
             DummyExpr::Literal(literal) => {

From 0c1f17d6dd05f67ca10f7bb984403693110740a9 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Mon, 16 Sep 2019 15:41:30 +0200
Subject: [PATCH 2752/3084] Add empty_vararg literal to generate jump
 instruction.

---
 cranelift/codegen/meta/src/cdsl/ast.rs | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index b2b1e2f3e8..910ffe24b2 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -111,6 +111,9 @@ pub(crate) enum Literal {
 
     /// A value of an integer immediate operand.
     Int(i64),
+
+    /// A empty list of variable set of arguments.
+    EmptyVarArgs,
 }
 
 impl Literal {
@@ -147,11 +150,16 @@ impl Literal {
         Literal::Int(value)
     }
 
+    pub fn empty_vararg() -> Self {
+        Literal::EmptyVarArgs
+    }
+
     pub fn to_rust_code(&self) -> String {
         match self {
             Literal::Enumerator { rust_type, value } => format!("{}::{}", rust_type, value),
             Literal::Bits { rust_type, value } => format!("{}::with_bits({:#x})", rust_type, value),
             Literal::Int(val) => val.to_string(),
+            Literal::EmptyVarArgs => "&[]".into(),
         }
     }
 }
@@ -388,7 +396,7 @@ impl Apply {
                              obtained enum value '{}'",
                             op.kind.name, value
                         ),
-                        Literal::Bits { .. } | Literal::Int(_) => {}
+                        Literal::Bits { .. } | Literal::Int(_) | Literal::EmptyVarArgs => {}
                     },
                     _ => {
                         panic!(

From 26cfbafb32c4056353abe36debf9e3bcd65ada6f Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Mon, 16 Sep 2019 15:46:20 +0200
Subject: [PATCH 2753/3084] Add ebb macro to insert a new blocks in
 legalization output.

---
 cranelift/codegen/meta/src/cdsl/ast.rs      | 68 ++++++++++++++++++++-
 cranelift/codegen/meta/src/cdsl/xform.rs    | 58 ++++++++++++++----
 cranelift/codegen/meta/src/gen_legalizer.rs | 40 ++++++++++--
 3 files changed, 148 insertions(+), 18 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 910ffe24b2..5618927bb4 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -4,9 +4,10 @@ use crate::cdsl::operands::{OperandKind, OperandKindFields};
 use crate::cdsl::types::ValueType;
 use crate::cdsl::typevar::{TypeSetBuilder, TypeVar};
 
-use cranelift_entity::{entity_impl, PrimaryMap};
+use cranelift_entity::{entity_impl, PrimaryMap, SparseMap, SparseMapValue};
 
 use std::fmt;
+use std::iter::IntoIterator;
 
 pub(crate) enum Expr {
     Var(VarIndex),
@@ -82,7 +83,7 @@ impl DefPool {
     pub fn next_index(&self) -> DefIndex {
         self.pool.next_key()
     }
-    pub fn create(&mut self, apply: Apply, defined_vars: Vec) -> DefIndex {
+    pub fn create_inst(&mut self, apply: Apply, defined_vars: Vec) -> DefIndex {
         self.pool.push(Def {
             apply,
             defined_vars,
@@ -94,6 +95,55 @@ impl DefPool {
 pub(crate) struct DefIndex(u32);
 entity_impl!(DefIndex);
 
+/// A definition which would lead to generate a block creation.
+#[derive(Clone)]
+pub(crate) struct Block {
+    /// Instruction index after which the block entry is set.
+    pub location: DefIndex,
+    /// Variable holding the new created block.
+    pub name: VarIndex,
+}
+
+pub(crate) struct BlockPool {
+    pool: SparseMap,
+}
+
+impl SparseMapValue for Block {
+    fn key(&self) -> DefIndex {
+        self.location
+    }
+}
+
+impl BlockPool {
+    pub fn new() -> Self {
+        Self {
+            pool: SparseMap::new(),
+        }
+    }
+    pub fn get(&self, index: DefIndex) -> Option<&Block> {
+        self.pool.get(index)
+    }
+    pub fn create_block(&mut self, name: VarIndex, location: DefIndex) {
+        if self.pool.contains_key(location) {
+            panic!("Attempt to insert 2 blocks after the same instruction")
+        }
+        self.pool.insert(Block { location, name });
+    }
+    pub fn is_empty(&self) -> bool {
+        self.pool.is_empty()
+    }
+}
+
+// Implement IntoIterator such that we can iterate over blocks which are in the block pool.
+impl<'a> IntoIterator for &'a BlockPool {
+    type Item = <&'a SparseMap as IntoIterator>::Item;
+    type IntoIter = <&'a SparseMap as IntoIterator>::IntoIter;
+
+    fn into_iter(self) -> Self::IntoIter {
+        self.pool.into_iter()
+    }
+}
+
 #[derive(Clone, Debug)]
 pub(crate) enum Literal {
     /// A value of an enumerated immediate operand.
@@ -518,6 +568,7 @@ pub(crate) enum DummyExpr {
     Var(DummyVar),
     Literal(Literal),
     Apply(InstSpec, Vec),
+    Block(DummyVar),
 }
 
 #[derive(Clone)]
@@ -561,6 +612,11 @@ impl ExprBuilder {
             defined_vars,
         }
     }
+
+    pub fn block(name: DummyVar) -> Self {
+        let expr = DummyExpr::Block(name);
+        Self { expr }
+    }
 }
 
 macro_rules! def_rhs {
@@ -592,3 +648,11 @@ macro_rules! def {
         def_rhs!($($tt)*).assign_to(Vec::new())
     }
 }
+
+// Helper macro to define legalization recipes.
+macro_rules! ebb {
+    // An basic block definition, splitting the current block in 2.
+    ($block: ident) => {
+        ExprBuilder::block($block).assign_to(Vec::new())
+    };
+}
diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index d0a543baec..6304469718 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -1,5 +1,6 @@
 use crate::cdsl::ast::{
-    Apply, DefIndex, DefPool, DummyDef, DummyExpr, Expr, PatternPosition, VarIndex, VarPool,
+    Apply, BlockPool, DefIndex, DefPool, DummyDef, DummyExpr, Expr, PatternPosition, VarIndex,
+    VarPool,
 };
 use crate::cdsl::instructions::Instruction;
 use crate::cdsl::type_inference::{infer_transform, TypeEnvironment};
@@ -22,6 +23,7 @@ pub(crate) struct Transform {
     pub dst: Vec,
     pub var_pool: VarPool,
     pub def_pool: DefPool,
+    pub block_pool: BlockPool,
     pub type_env: TypeEnvironment,
 }
 
@@ -31,6 +33,7 @@ impl Transform {
     fn new(src: DummyDef, dst: Vec) -> Self {
         let mut var_pool = VarPool::new();
         let mut def_pool = DefPool::new();
+        let mut block_pool = BlockPool::new();
 
         let mut input_vars: Vec = Vec::new();
         let mut defined_vars: Vec = Vec::new();
@@ -47,6 +50,7 @@ impl Transform {
             &mut defined_vars,
             &mut var_pool,
             &mut def_pool,
+            &mut block_pool,
         )[0];
 
         let num_src_inputs = input_vars.len();
@@ -59,6 +63,7 @@ impl Transform {
             &mut defined_vars,
             &mut var_pool,
             &mut def_pool,
+            &mut block_pool,
         );
 
         // Sanity checks.
@@ -122,6 +127,7 @@ impl Transform {
             dst,
             var_pool,
             def_pool,
+            block_pool,
             type_env,
         }
     }
@@ -224,6 +230,9 @@ fn rewrite_expr(
             DummyExpr::Apply(..) => {
                 panic!("Recursive apply is not allowed.");
             }
+            DummyExpr::Block(_block) => {
+                panic!("Blocks are not valid arguments.");
+            }
         }
     }
 
@@ -238,8 +247,18 @@ fn rewrite_def_list(
     defined_vars: &mut Vec,
     var_pool: &mut VarPool,
     def_pool: &mut DefPool,
+    block_pool: &mut BlockPool,
 ) -> Vec {
     let mut new_defs = Vec::new();
+    // Register variable names of new blocks first as a block name can be used to jump forward. Thus
+    // the name has to be registered first to avoid misinterpreting it as an input-var.
+    for dummy_def in dummy_defs.iter() {
+        if let DummyExpr::Block(ref var) = dummy_def.expr {
+            var_index(var.name, symbol_table, defined_vars, var_pool);
+        }
+    }
+
+    // Iterate over the definitions and blocks, to map variables names to inputs or outputs.
     for dummy_def in dummy_defs {
         let def_index = def_pool.next_index();
 
@@ -251,19 +270,34 @@ fn rewrite_def_list(
             defined_vars,
             var_pool,
         );
-        let new_apply = rewrite_expr(position, dummy_def.expr, symbol_table, input_vars, var_pool);
+        if let DummyExpr::Block(var) = dummy_def.expr {
+            let var_index = *symbol_table
+                .get(var.name)
+                .or_else(|| {
+                    panic!(
+                        "Block {} was not registered during the first visit",
+                        var.name
+                    )
+                })
+                .unwrap();
+            var_pool.get_mut(var_index).set_def(position, def_index);
+            block_pool.create_block(var_index, def_index);
+        } else {
+            let new_apply =
+                rewrite_expr(position, dummy_def.expr, symbol_table, input_vars, var_pool);
 
-        assert!(
-            def_pool.next_index() == def_index,
-            "shouldn't have created new defs in the meanwhile"
-        );
-        assert_eq!(
-            new_apply.inst.value_results.len(),
-            new_defined_vars.len(),
-            "number of Var results in instruction is incorrect"
-        );
+            assert!(
+                def_pool.next_index() == def_index,
+                "shouldn't have created new defs in the meanwhile"
+            );
+            assert_eq!(
+                new_apply.inst.value_results.len(),
+                new_defined_vars.len(),
+                "number of Var results in instruction is incorrect"
+            );
 
-        new_defs.push(def_pool.create(new_apply, new_defined_vars));
+            new_defs.push(def_pool.create_inst(new_apply, new_defined_vars));
+        }
     }
     new_defs
 }
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index 4153fb8c32..bceed3fac6 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -401,14 +401,30 @@ fn gen_transform<'a>(
     // Guard the actual expansion by `predicate`.
     fmt.line("if predicate {");
     fmt.indent(|fmt| {
+        // If we are adding some blocks, we need to recall the original block, such that we can
+        // recompute it.
+        if !transform.block_pool.is_empty() {
+            fmt.line("let orig_ebb = pos.current_ebb().unwrap();");
+        }
+
         // If we're going to delete `inst`, we need to detach its results first so they can be
         // reattached during pattern expansion.
         if !replace_inst {
             fmt.line("pos.func.dfg.clear_results(inst);");
         }
 
+        // Emit new block creation.
+        for block in &transform.block_pool {
+            let var = transform.var_pool.get(block.name);
+            fmtln!(fmt, "let {} = pos.func.dfg.make_ebb();", var.name);
+        }
+
         // Emit the destination pattern.
         for &def_index in &transform.dst {
+            if let Some(block) = transform.block_pool.get(def_index) {
+                let var = transform.var_pool.get(block.name);
+                fmtln!(fmt, "pos.insert_ebb({});", var.name);
+            }
             emit_dst_inst(
                 transform.def_pool.get(def_index),
                 &transform.def_pool,
@@ -417,16 +433,32 @@ fn gen_transform<'a>(
             );
         }
 
+        // Insert a new block after the last instruction, if needed.
+        let def_next_index = transform.def_pool.next_index();
+        if let Some(block) = transform.block_pool.get(def_next_index) {
+            let var = transform.var_pool.get(block.name);
+            fmtln!(fmt, "pos.insert_ebb({});", var.name);
+        }
+
         // Delete the original instruction if we didn't have an opportunity to replace it.
         if !replace_inst {
             fmt.line("let removed = pos.remove_inst();");
             fmt.line("debug_assert_eq!(removed, inst);");
         }
 
-        if transform.def_pool.get(transform.src).apply.inst.is_branch {
-            // A branch might have been legalized into multiple branches, so we need to recompute
-            // the cfg.
-            fmt.line("cfg.recompute_ebb(pos.func, pos.current_ebb().unwrap());");
+        if transform.block_pool.is_empty() {
+            if transform.def_pool.get(transform.src).apply.inst.is_branch {
+                // A branch might have been legalized into multiple branches, so we need to recompute
+                // the cfg.
+                fmt.line("cfg.recompute_ebb(pos.func, pos.current_ebb().unwrap());");
+            }
+        } else {
+            // Update CFG for the new blocks.
+            fmt.line("cfg.recompute_ebb(pos.func, orig_ebb);");
+            for block in &transform.block_pool {
+                let var = transform.var_pool.get(block.name);
+                fmtln!(fmt, "cfg.recompute_ebb(pos.func, {});", var.name);
+            }
         }
 
         fmt.line("return true;");

From 694de912a5a1f6752bf0b44d5b320b359bcb1696 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Mon, 16 Sep 2019 15:50:34 +0200
Subject: [PATCH 2754/3084] Update brnz.i128 legalization to use non-extended
 basic blocks.

---
 cranelift/codegen/meta/src/shared/legalize.rs | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 30de84a198..fef4c1e1ec 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -93,6 +93,7 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     let isub_borrow = insts.by_name("isub_borrow");
     let isub_ifbin = insts.by_name("isub_ifbin");
     let isub_ifbout = insts.by_name("isub_ifbout");
+    let jump = insts.by_name("jump");
     let load = insts.by_name("load");
     let popcnt = insts.by_name("popcnt");
     let rotl = insts.by_name("rotl");
@@ -189,6 +190,8 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     let ah = var("ah");
     let cc = var("cc");
     let ebb = var("ebb");
+    let ebb1 = var("ebb1");
+    let ebb2 = var("ebb2");
     let ptr = var("ptr");
     let flags = var("flags");
     let offset = var("off");
@@ -259,11 +262,13 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     );
 
     narrow.legalize(
-        def!(brnz.I128(x, ebb, vararg)),
+        def!(brnz.I128(x, ebb1, vararg)),
         vec![
             def!((xl, xh) = isplit(x)),
-            def!(brnz(xl, ebb, vararg)),
-            def!(brnz(xh, ebb, vararg)),
+            def!(brnz(xl, ebb1, vararg)),
+            def!(jump(ebb2, Literal::empty_vararg())),
+            ebb!(ebb2),
+            def!(brnz(xh, ebb1, vararg)),
         ],
     );
 

From 702155b19b796663e4e3e8c76cf3a35ef0963229 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 28 Aug 2019 15:29:40 -0700
Subject: [PATCH 2755/3084] Optimize vconst for x86 when immediate contains all
 zeroes or ones

Instead of using MOVUPS to expensively load bits from memory, this change uses a predicate to optimize vconst without a memory access:
 - when the 128-bit immediate is all zeroes in all bits, use PXOR to zero out an XMM register
 - when the 128-bit immediate is all ones in all bits, use PCMPEQB to set an XMM register to all ones

This leaves the constant data in the constant pool, which may increase code size (TODO)
---
 .../codegen/meta/src/cdsl/instructions.rs     | 36 ++++++++++++++++
 .../codegen/meta/src/isa/x86/encodings.rs     | 43 ++++++++++++++++++-
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 12 ++++++
 cranelift/codegen/src/predicates.rs           | 27 ++++++++++++
 .../filetests/isa/x86/vconst-binemit.clif     |  4 +-
 .../filetests/isa/x86/vconst-opt-run.clif     | 23 ++++++++++
 .../filetests/isa/x86/vconst-opt.clif         | 12 ++++++
 7 files changed, 154 insertions(+), 3 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/vconst-opt-run.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/vconst-opt.clif

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 2737099cc4..fdc5ad8bf8 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -614,6 +614,12 @@ pub enum FormatPredicateKind {
     /// Is the immediate format field member equal to zero? (float64 version)
     IsZero64BitFloat,
 
+    /// Is the immediate format field member equal zero in all lanes?
+    IsAllZeroes128Bit,
+
+    /// Does the immediate format field member have ones in all bits of all lanes?
+    IsAllOnes128Bit,
+
     /// Has the value list (in member_name) the size specified in parameter?
     LengthEquals(usize),
 
@@ -690,6 +696,14 @@ impl FormatPredicateNode {
             FormatPredicateKind::IsZero64BitFloat => {
                 format!("predicates::is_zero_64_bit_float({})", self.member_name)
             }
+            FormatPredicateKind::IsAllZeroes128Bit => format!(
+                "predicates::is_all_zeroes_128_bit(func.dfg.constants.get({}))",
+                self.member_name
+            ),
+            FormatPredicateKind::IsAllOnes128Bit => format!(
+                "predicates::is_all_ones_128_bit(func.dfg.constants.get({}))",
+                self.member_name
+            ),
             FormatPredicateKind::LengthEquals(num) => format!(
                 "predicates::has_length_of({}, {}, func)",
                 self.member_name, num
@@ -929,6 +943,28 @@ impl InstructionPredicate {
         ))
     }
 
+    pub fn new_is_all_zeroes_128bit(
+        format: &InstructionFormat,
+        field_name: &'static str,
+    ) -> InstructionPredicateNode {
+        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
+            format,
+            field_name,
+            FormatPredicateKind::IsAllZeroes128Bit,
+        ))
+    }
+
+    pub fn new_is_all_ones_128bit(
+        format: &InstructionFormat,
+        field_name: &'static str,
+    ) -> InstructionPredicateNode {
+        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
+            format,
+            field_name,
+            FormatPredicateKind::IsAllOnes128Bit,
+        ))
+    }
+
     pub fn new_length_equals(format: &InstructionFormat, size: usize) -> InstructionPredicateNode {
         assert!(
             format.has_value_list,
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index d8b070b69e..6c578c458a 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -310,6 +310,20 @@ impl PerCpuModeEncodings {
         self.enc64_rec(inst, recipe, bits);
     }
 
+    /// Add the same encoding to both X86_32 and X86_64; assumes configuration (e.g. REX, operand binding) has already happened
+    fn enc_32_64_func(
+        &mut self,
+        inst: impl Clone + Into,
+        template: Template,
+        builder_closure: T,
+    ) where
+        T: FnOnce(EncodingBuilder) -> EncodingBuilder,
+    {
+        let encoding = self.make_encoding(inst.into(), template, builder_closure);
+        self.enc32.push(encoding.clone());
+        self.enc64.push(encoding);
+    }
+
     /// Add the same encoding to both X86_32 and X86_64; assumes configuration (e.g. REX, operand
     /// binding) has already happened.
     fn enc_32_64_maybe_isap(
@@ -642,6 +656,7 @@ pub(crate) fn define(
     let rec_urm_noflags = r.template("urm_noflags");
     let rec_urm_noflags_abcd = r.template("urm_noflags_abcd");
     let rec_vconst = r.template("vconst");
+    let rec_vconst_optimized = r.template("vconst_optimized");
 
     // Predicates shorthands.
     let all_ones_funcaddrs_and_not_is_pic =
@@ -1671,7 +1686,7 @@ pub(crate) fn define(
     );
     e.enc_x86_64_instp(
         f64const,
-        rec_f64imm_z.opcodes(vec![0x66, 0x0f, 0x57]),
+        rec_f64imm_z.opcodes(vec![0x66, 0x0f, 0x57]), // XORPD from SSE2
         is_zero_64_bit_float,
     );
 
@@ -1946,6 +1961,32 @@ pub(crate) fn define(
         }
     }
 
+    // SIMD vconst for special cases (all zeroes, all ones)
+    // this must be encoded prior to the MOVUPS implementation (below) so the compiler sees this
+    // encoding first
+    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
+        let f_unary_const = formats.get(formats.by_name("UnaryConst"));
+        let instruction = vconst.bind_vector_from_lane(ty, sse_vector_size);
+
+        let is_zero_128bit =
+            InstructionPredicate::new_is_all_zeroes_128bit(f_unary_const, "constant_handle");
+        let template = rec_vconst_optimized
+            .nonrex()
+            .opcodes(vec![0x66, 0x0f, 0xef]); // PXOR from SSE2
+        e.enc_32_64_func(instruction.clone(), template, |builder| {
+            builder.inst_predicate(is_zero_128bit)
+        });
+
+        let is_ones_128bit =
+            InstructionPredicate::new_is_all_ones_128bit(f_unary_const, "constant_handle");
+        let template = rec_vconst_optimized
+            .nonrex()
+            .opcodes(vec![0x66, 0x0f, 0x74]); // PCMPEQB from SSE2
+        e.enc_32_64_func(instruction, template, |builder| {
+            builder.inst_predicate(is_ones_128bit)
+        });
+    }
+
     // SIMD vconst using MOVUPS
     // TODO it would be ideal if eventually this became the more efficient MOVAPS but we would have
     // to guarantee that the constants are aligned when emitted and there is currently no mechanism
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index b30a35e3f5..b9a10c86f6 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -2449,6 +2449,18 @@ pub(crate) fn define<'shared>(
             ),
     );
 
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("vconst_optimized", f_unary_const, 1)
+            .operands_out(vec![fpr])
+            .clobbers_flags(false)
+            .emit(
+                r#"
+                    {{PUT_OP}}(bits, rex2(out_reg0, out_reg0), sink);
+                    modrm_rr(out_reg0, out_reg0, sink);
+                "#,
+            ),
+    );
+
     recipes.add_template_recipe(
         EncodingRecipeBuilder::new("jt_base", f_branch_table_base, 5)
             .operands_out(vec![gpr])
diff --git a/cranelift/codegen/src/predicates.rs b/cranelift/codegen/src/predicates.rs
index f900546111..16672b145b 100644
--- a/cranelift/codegen/src/predicates.rs
+++ b/cranelift/codegen/src/predicates.rs
@@ -31,6 +31,18 @@ pub fn is_zero_32_bit_float>(x: T) -> bool {
     x32.bits() == 0
 }
 
+/// Check that a 128-bit vector contains all zeroes.
+#[allow(dead_code)]
+pub fn is_all_zeroes_128_bit<'b, T: PartialEq<&'b [u8; 16]>>(x: T) -> bool {
+    x.eq(&&[0; 16])
+}
+
+/// Check that a 128-bit vector contains all ones.
+#[allow(dead_code)]
+pub fn is_all_ones_128_bit<'b, T: PartialEq<&'b [u8; 16]>>(x: T) -> bool {
+    x.eq(&&[0xff; 16])
+}
+
 /// Check that `x` is the same as `y`.
 #[allow(dead_code)]
 pub fn is_equal + Copy>(x: T, y: O) -> bool {
@@ -109,4 +121,19 @@ mod tests {
         assert!(!is_signed_int(x1, 16, 4));
         assert!(!is_signed_int(x2, 16, 4));
     }
+
+    #[test]
+    fn is_all_zeroes() {
+        assert!(is_all_zeroes_128_bit(&[0; 16]));
+        assert!(is_all_zeroes_128_bit(vec![
+            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+        ]));
+        assert!(!is_all_zeroes_128_bit(&[1; 16]));
+    }
+
+    #[test]
+    fn is_all_ones() {
+        assert!(!is_all_ones_128_bit(&[0; 16]));
+        assert!(is_all_ones_128_bit(&[0xff; 16]));
+    }
 }
diff --git a/cranelift/filetests/filetests/isa/x86/vconst-binemit.clif b/cranelift/filetests/filetests/isa/x86/vconst-binemit.clif
index b07dc0fd4e..2d6f862679 100644
--- a/cranelift/filetests/filetests/isa/x86/vconst-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/vconst-binemit.clif
@@ -5,7 +5,7 @@ target x86_64
 
 function %test_vconst_b8() {
 ebb0:
-[-, %xmm2]  v0 = vconst.b8x16 0x00  ; bin: 0f 10 15 00000008 PCRelRodata4(15)
-[-, %xmm3]  v1 = vconst.b8x16 0x01  ; bin: 0f 10 1d 00000011 PCRelRodata4(31)
+[-, %xmm2]  v0 = vconst.b8x16 0x01  ; bin: 0f 10 15 00000008 PCRelRodata4(15)
+[-, %xmm3]  v1 = vconst.b8x16 0x02  ; bin: 0f 10 1d 00000011 PCRelRodata4(31)
             return
 }
diff --git a/cranelift/filetests/filetests/isa/x86/vconst-opt-run.clif b/cranelift/filetests/filetests/isa/x86/vconst-opt-run.clif
new file mode 100644
index 0000000000..487ff4f844
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/vconst-opt-run.clif
@@ -0,0 +1,23 @@
+test run
+set enable_simd
+target x86_64
+
+; TODO move to vconst-run.clif
+
+function %test_vconst_zeroes() -> b1 {
+ebb0:
+    v0 = vconst.i8x16 0x00
+    v1 = extractlane v0, 4
+    v2 = icmp_imm eq v1, 0
+    return v2
+}
+; run
+
+function %test_vconst_ones() -> b1 {
+ebb0:
+    v0 = vconst.i8x16 0xffffffffffffffffffffffffffffffff
+    v1 = extractlane v0, 2
+    v2 = icmp_imm eq v1, 0xff
+    return v2
+}
+; run
diff --git a/cranelift/filetests/filetests/isa/x86/vconst-opt.clif b/cranelift/filetests/filetests/isa/x86/vconst-opt.clif
new file mode 100644
index 0000000000..4daeed8abe
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/vconst-opt.clif
@@ -0,0 +1,12 @@
+test binemit
+set enable_simd
+target x86_64
+
+; TODO move to vconst-compile.clif or vconst-binemit.clif
+
+function %test_vconst_optimizations() {
+ebb0:
+[-, %xmm4]  v0 = vconst.b8x16 0x00                                  ; bin: 66 0f ef e4
+[-, %xmm7]  v1 = vconst.b8x16 0xffffffffffffffffffffffffffffffff    ; bin: 66 0f 74 ff
+            return
+}

From a3db30d97e646c68a83f06a0086e125dffdd87ed Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 3 Sep 2019 15:15:55 -0700
Subject: [PATCH 2756/3084] Add x86 encoding for SIMD `icmp eq`

Also adds a predicate for matching the `eq` IntCC code (TODO this should be replaced by something more general)
---
 .../codegen/meta/src/cdsl/instructions.rs     | 19 ++++++++++
 .../codegen/meta/src/isa/x86/encodings.rs     | 26 ++++++++++++++
 .../codegen/meta/src/isa/x86/legalize.rs      |  1 +
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 13 +++++++
 cranelift/codegen/src/predicates.rs           | 16 +++++++++
 .../filetests/isa/x86/icmp-compile.clif       | 35 +++++++++++++++++++
 .../filetests/filetests/isa/x86/icmp-run.clif | 24 +++++++++++++
 7 files changed, 134 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/icmp-compile.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/icmp-run.clif

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index fdc5ad8bf8..0ad5c1397a 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -628,6 +628,9 @@ pub enum FormatPredicateKind {
 
     /// Is the referenced data object colocated?
     IsColocatedData,
+
+    /// Does the operation have a specific condition code?
+    HasConditionCode(&'static str),
 }
 
 #[derive(Clone, Hash, PartialEq, Eq)]
@@ -714,6 +717,10 @@ impl FormatPredicateNode {
             FormatPredicateKind::IsColocatedData => {
                 format!("predicates::is_colocated_data({}, func)", self.member_name)
             }
+            FormatPredicateKind::HasConditionCode(code) => format!(
+                "predicates::match_condition_code_to_str({}, \"{}\")",
+                self.member_name, code
+            ),
         }
     }
 }
@@ -997,6 +1004,18 @@ impl InstructionPredicate {
         ))
     }
 
+    pub fn new_has_condition_code(
+        format: &InstructionFormat,
+        condition_code: &'static str,
+        field_name: &'static str,
+    ) -> InstructionPredicateNode {
+        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
+            format,
+            field_name,
+            FormatPredicateKind::HasConditionCode(condition_code),
+        ))
+    }
+
     pub fn and(mut self, new_node: InstructionPredicateNode) -> Self {
         let node = self.node;
         let mut and_nodes = match node {
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 6c578c458a..453ecc37dc 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -573,6 +573,7 @@ pub(crate) fn define(
     let rec_gvaddr4 = r.template("gvaddr4");
     let rec_gvaddr8 = r.template("gvaddr8");
     let rec_icscc = r.template("icscc");
+    let rec_icscc_fpr = r.template("icscc_fpr");
     let rec_icscc_ib = r.template("icscc_ib");
     let rec_icscc_id = r.template("icscc_id");
     let rec_indirect_jmp = r.template("indirect_jmp");
@@ -2058,6 +2059,31 @@ pub(crate) fn define(
         e.enc_32_64(iadd, rec_fa.opcodes(opcodes.to_vec()));
     }
 
+    // SIMD icmp using PCMPEQ*
+    let mut pcmpeq_mapping: HashMap, Option)> =
+        HashMap::new();
+    pcmpeq_mapping.insert(8, (vec![0x66, 0x0f, 0x74], None)); // PCMPEQB from SSE2
+    pcmpeq_mapping.insert(16, (vec![0x66, 0x0f, 0x75], None)); // PCMPEQW from SSE2
+    pcmpeq_mapping.insert(32, (vec![0x66, 0x0f, 0x76], None)); // PCMPEQD from SSE2
+    pcmpeq_mapping.insert(64, (vec![0x66, 0x0f, 0x38, 0x29], Some(use_sse41_simd))); // PCMPEQQ from SSE4.1
+    for ty in ValueType::all_lane_types().filter(|t| t.is_int() && allowed_simd_type(t)) {
+        if let Some((opcodes, isa_predicate)) = pcmpeq_mapping.get(&ty.lane_bits()) {
+            let instruction = icmp.bind_vector_from_lane(ty, sse_vector_size);
+            let f_int_compare = formats.get(formats.by_name("IntCompare"));
+            let has_eq_condition_code =
+                InstructionPredicate::new_has_condition_code(f_int_compare, "eq", "cond");
+            let template = rec_icscc_fpr.nonrex().opcodes(opcodes.clone());
+            e.enc_32_64_func(instruction, template, |builder| {
+                let builder = builder.inst_predicate(has_eq_condition_code);
+                if let Some(p) = isa_predicate {
+                    builder.isa_predicate(*p)
+                } else {
+                    builder
+                }
+            });
+        }
+    }
+
     // Reference type instructions
 
     // Null references implemented as iconst 0.
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index e37759e892..57057ce6bb 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -329,6 +329,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
             def!(y = splat_any8x16(x)),
             vec![
                 def!(a = scalar_to_vector(x)), // move into the lowest 8 bits of an XMM register
+                // TODO replace the following two instructions with `vconst(0)` when this is possible; see https://github.com/CraneStation/cranelift/issues/1052
                 def!(b = f64const(ieee64_zero)), // zero out a different XMM register; the shuffle mask for moving the lowest byte to all other byte lanes is 0x0
                 def!(c = bitcast_f64_to_any8x16(b)), // no instruction emitted; informs the SSA that the 0 in b can be used as a vector of this type
                 def!(y = x86_pshufb(a, c)), // PSHUFB takes two XMM operands, one of which is a shuffle mask (i.e. b)
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index b9a10c86f6..da602b9973 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -2940,6 +2940,19 @@ pub(crate) fn define<'shared>(
             ),
     );
 
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("icscc_fpr", f_int_compare, 1)
+            .operands_in(vec![fpr, fpr])
+            .operands_out(vec![0])
+            .emit(
+                r#"
+                    // Comparison instruction.
+                    {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                    modrm_rr(in_reg1, in_reg0, sink);
+                "#,
+            ),
+    );
+
     {
         let format = formats.get(f_int_compare_imm);
 
diff --git a/cranelift/codegen/src/predicates.rs b/cranelift/codegen/src/predicates.rs
index 16672b145b..cb56fb0100 100644
--- a/cranelift/codegen/src/predicates.rs
+++ b/cranelift/codegen/src/predicates.rs
@@ -10,6 +10,8 @@
 //! dead code warning.
 
 use crate::ir;
+use crate::ir::condcodes::IntCC;
+use std::string::ToString;
 
 /// Check that an integer value is zero.
 #[allow(dead_code)]
@@ -83,6 +85,14 @@ pub fn has_length_of(value_list: &ir::ValueList, num: usize, func: &ir::Function
     value_list.len(&func.dfg.value_lists) == num
 }
 
+#[allow(dead_code)]
+pub fn match_condition_code_to_str(
+    condition_code: IntCC,
+    stringified_condition_code: &str,
+) -> bool {
+    condition_code.to_string().eq(stringified_condition_code)
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -136,4 +146,10 @@ mod tests {
         assert!(!is_all_ones_128_bit(&[0; 16]));
         assert!(is_all_ones_128_bit(&[0xff; 16]));
     }
+
+    #[test]
+    fn condition_code() {
+        assert!(match_condition_code_to_str(IntCC::Equal, "eq"));
+        assert!(!match_condition_code_to_str(IntCC::Equal, "ne"));
+    }
 }
diff --git a/cranelift/filetests/filetests/isa/x86/icmp-compile.clif b/cranelift/filetests/filetests/isa/x86/icmp-compile.clif
new file mode 100644
index 0000000000..cf9cb3ff07
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/icmp-compile.clif
@@ -0,0 +1,35 @@
+test binemit
+set enable_simd
+target x86_64 skylake
+
+function %icmp_i8x16() {
+ebb0:
+[-, %xmm3]  v0 = vconst.i8x16 0x00                                  ; bin: 66 0f ef db
+[-, %xmm4]  v1 = vconst.i8x16 0xffffffffffffffffffffffffffffffff    ; bin: 66 0f 74 e4
+[-, %xmm3]  v2 = icmp eq v0, v1                                     ; bin: 66 0f 74 dc
+            return
+}
+
+function %icmp_i16x8() {
+ebb0:
+[-, %xmm0]  v0 = vconst.i16x8 0x00
+[-, %xmm7]  v1 = vconst.i16x8 0xffffffffffffffffffffffffffffffff
+[-, %xmm0]  v2 = icmp eq v0, v1                                     ; bin: 66 0f 75 c7
+            return
+}
+
+function %icmp_i32x4() {
+ebb0:
+[-, %xmm0]  v0 = vconst.i32x4 0x00
+[-, %xmm4]  v1 = vconst.i32x4 0xffffffffffffffffffffffffffffffff
+[-, %xmm0]  v2 = icmp eq v0, v1                                     ; bin: 66 0f 76 c4
+            return
+}
+
+function %icmp_i64x2() {
+ebb0:
+[-, %xmm0]  v0 = vconst.i64x2 0x00
+[-, %xmm1]  v1 = vconst.i64x2 0xffffffffffffffffffffffffffffffff
+[-, %xmm0]  v2 = icmp eq v0, v1                                     ; bin: 66 0f 38 29 c1
+            return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/icmp-run.clif b/cranelift/filetests/filetests/isa/x86/icmp-run.clif
new file mode 100644
index 0000000000..c470af662a
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/icmp-run.clif
@@ -0,0 +1,24 @@
+test run
+set enable_simd
+
+function %run_icmp_i8x16() -> b8 {
+ebb0:
+    v0 = vconst.i8x16 0x00
+    v1 = vconst.i8x16 0x00
+    v2 = icmp eq v0, v1
+    v3 = extractlane v2, 0
+    return v3
+}
+
+; run
+
+function %run_icmp_i64x2() -> b64 {
+ebb0:
+    v0 = vconst.i64x2 0xffffffffffffffffffffffffffffffff
+    v1 = vconst.i64x2 0xffffffffffffffffffffffffffffffff
+    v2 = icmp eq v0, v1
+    v3 = extractlane v2, 1
+    return v3
+}
+
+; run

From c648fa3d06ef12ffa23da28909dec89ca216530b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 23 Sep 2019 11:04:50 -0700
Subject: [PATCH 2757/3084] Move condcodes.rs to a new crate,
 cranelift-codegen-shared

This move allows the `IntCC`/`FloatCC` enums to be used in both meta (for predicate matching) and in codegen. To avoid breaking any code dependent on the previous location of condcodes.rs (`cranelift-codegen/src/condcodes.rs`), the module is re-exported under `cranelift_codegen::ir`.
---
 cranelift/codegen/Cargo.toml                  |   1 +
 cranelift/codegen/meta/Cargo.toml             |   1 +
 cranelift/codegen/shared/Cargo.toml           |  11 +
 cranelift/codegen/shared/LICENSE              | 220 ++++++++++++++++++
 cranelift/codegen/shared/README.md            |   2 +
 .../{src/ir => shared/src}/condcodes.rs       |   0
 cranelift/codegen/shared/src/lib.rs           |  26 +++
 cranelift/codegen/src/ir/mod.rs               |   2 +-
 cranelift/publish-all.sh                      |   4 +-
 9 files changed, 264 insertions(+), 3 deletions(-)
 create mode 100644 cranelift/codegen/shared/Cargo.toml
 create mode 100644 cranelift/codegen/shared/LICENSE
 create mode 100644 cranelift/codegen/shared/README.md
 rename cranelift/codegen/{src/ir => shared/src}/condcodes.rs (100%)
 create mode 100644 cranelift/codegen/shared/src/lib.rs

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 251440cee0..633711930d 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -13,6 +13,7 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
+cranelift-codegen-shared = { path = "./shared", version = "0.43.1" }
 cranelift-entity = { path = "../cranelift-entity", version = "0.43.1", default-features = false }
 cranelift-bforest = { path = "../cranelift-bforest", version = "0.43.1", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index d869c412ba..d8bf77a6a9 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -9,6 +9,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
+cranelift-codegen-shared = { path = "../shared", version = "0.43.1" }
 cranelift-entity = { path = "../../cranelift-entity", version = "0.43.1", default-features = false }
 
 [badges]
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
new file mode 100644
index 0000000000..2ceba9eaf1
--- /dev/null
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -0,0 +1,11 @@
+[package]
+authors = ["The Cranelift Project Developers"]
+name = "cranelift-codegen-shared"
+version = "0.43.1"
+description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
+license = "Apache-2.0 WITH LLVM-exception"
+repository = "https://github.com/CraneStation/cranelift"
+readme = "README.md"
+edition = "2018"
+
+[dependencies]
diff --git a/cranelift/codegen/shared/LICENSE b/cranelift/codegen/shared/LICENSE
new file mode 100644
index 0000000000..f9d81955f4
--- /dev/null
+++ b/cranelift/codegen/shared/LICENSE
@@ -0,0 +1,220 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+--- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.
+
diff --git a/cranelift/codegen/shared/README.md b/cranelift/codegen/shared/README.md
new file mode 100644
index 0000000000..54f9f5d6d2
--- /dev/null
+++ b/cranelift/codegen/shared/README.md
@@ -0,0 +1,2 @@
+This crate contains shared definitions for use in both `cranelift-codegen-meta` and `cranelift
+-codegen`.
diff --git a/cranelift/codegen/src/ir/condcodes.rs b/cranelift/codegen/shared/src/condcodes.rs
similarity index 100%
rename from cranelift/codegen/src/ir/condcodes.rs
rename to cranelift/codegen/shared/src/condcodes.rs
diff --git a/cranelift/codegen/shared/src/lib.rs b/cranelift/codegen/shared/src/lib.rs
new file mode 100644
index 0000000000..87de6e7951
--- /dev/null
+++ b/cranelift/codegen/shared/src/lib.rs
@@ -0,0 +1,26 @@
+//! This library contains code that is common to both the `cranelift-codegen` and
+//! `cranelift-codegen-meta` libraries.
+
+#![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
+#![warn(unused_import_braces)]
+#![cfg_attr(feature = "std", deny(unstable_features))]
+#![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
+#![cfg_attr(
+    feature = "cargo-clippy",
+    warn(
+        clippy::float_arithmetic,
+        clippy::mut_mut,
+        clippy::nonminimal_bool,
+        clippy::option_map_unwrap_or,
+        clippy::option_map_unwrap_or_else,
+        clippy::print_stdout,
+        clippy::unicode_not_nfc,
+        clippy::use_self
+    )
+)]
+
+pub mod condcodes;
+
+/// Version number of this crate.
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/cranelift/codegen/src/ir/mod.rs b/cranelift/codegen/src/ir/mod.rs
index 2a0293b31b..2ee589f6af 100644
--- a/cranelift/codegen/src/ir/mod.rs
+++ b/cranelift/codegen/src/ir/mod.rs
@@ -1,7 +1,6 @@
 //! Representation of Cranelift IR functions.
 
 mod builder;
-pub mod condcodes;
 pub mod constant;
 pub mod dfg;
 pub mod entities;
@@ -55,6 +54,7 @@ pub use crate::ir::table::TableData;
 pub use crate::ir::trapcode::TrapCode;
 pub use crate::ir::types::Type;
 pub use crate::ir::valueloc::{ArgumentLoc, ValueLoc};
+pub use cranelift_codegen_shared::condcodes;
 
 use crate::binemit;
 use crate::entity::{entity_impl, PrimaryMap, SecondaryMap};
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index e52228a9c3..8480a4183f 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -15,7 +15,7 @@ version="0.43.1"
 #
 # The main Cargo.toml in the top-level directory is the cranelift-tools crate which we don't publish.
 echo "Updating crate versions to $version"
-for crate in . cranelift-* cranelift-codegen/meta; do
+for crate in . cranelift-* cranelift-codegen/shared cranelift-codegen/meta; do
     # Update the version number of this crate to $version.
     sed -i.bk -e "s/^version = .*/version = \"$version\"/" \
         "$crate/Cargo.toml"
@@ -38,7 +38,7 @@ echo git tag v$version
 echo git push
 echo git push origin v$version
 for crate in \
-    entity bforest codegen/meta codegen frontend native \
+    entity bforest codegen/shared codegen/meta codegen frontend native \
     preopt \
     reader wasm module \
     faerie umbrella simplejit

From 636ef98024d08b5e512ea34b1c907e4fb78fd793 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 23 Sep 2019 11:22:34 -0700
Subject: [PATCH 2758/3084] Use existing `is_equal` predicate with the
 newly-shared condition codes

This removes the `HasConditionCode(&'static str)` predicate and the associated issues with that.
---
 cranelift/codegen/meta/src/cdsl/instructions.rs |  7 ++++---
 cranelift/codegen/meta/src/isa/x86/encodings.rs |  3 ++-
 cranelift/codegen/src/predicates.rs             | 16 ----------------
 3 files changed, 6 insertions(+), 20 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 0ad5c1397a..178077e456 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -13,6 +13,7 @@ use crate::cdsl::operands::Operand;
 use crate::cdsl::type_inference::Constraint;
 use crate::cdsl::types::{LaneType, ReferenceType, ValueType, VectorType};
 use crate::cdsl::typevar::TypeVar;
+use cranelift_codegen_shared::condcodes::IntCC;
 
 #[derive(Copy, Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct OpcodeNumber(u32);
@@ -630,7 +631,7 @@ pub enum FormatPredicateKind {
     IsColocatedData,
 
     /// Does the operation have a specific condition code?
-    HasConditionCode(&'static str),
+    HasConditionCode(IntCC),
 }
 
 #[derive(Clone, Hash, PartialEq, Eq)]
@@ -718,7 +719,7 @@ impl FormatPredicateNode {
                 format!("predicates::is_colocated_data({}, func)", self.member_name)
             }
             FormatPredicateKind::HasConditionCode(code) => format!(
-                "predicates::match_condition_code_to_str({}, \"{}\")",
+                "predicates::is_equal({}, IntCC::{:?})",
                 self.member_name, code
             ),
         }
@@ -1006,7 +1007,7 @@ impl InstructionPredicate {
 
     pub fn new_has_condition_code(
         format: &InstructionFormat,
-        condition_code: &'static str,
+        condition_code: IntCC,
         field_name: &'static str,
     ) -> InstructionPredicateNode {
         InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 453ecc37dc..fc385d512a 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1,5 +1,6 @@
 #![allow(non_snake_case)]
 
+use cranelift_codegen_shared::condcodes::IntCC;
 use std::collections::HashMap;
 
 use crate::cdsl::encodings::{Encoding, EncodingBuilder};
@@ -2071,7 +2072,7 @@ pub(crate) fn define(
             let instruction = icmp.bind_vector_from_lane(ty, sse_vector_size);
             let f_int_compare = formats.get(formats.by_name("IntCompare"));
             let has_eq_condition_code =
-                InstructionPredicate::new_has_condition_code(f_int_compare, "eq", "cond");
+                InstructionPredicate::new_has_condition_code(f_int_compare, IntCC::Equal, "cond");
             let template = rec_icscc_fpr.nonrex().opcodes(opcodes.clone());
             e.enc_32_64_func(instruction, template, |builder| {
                 let builder = builder.inst_predicate(has_eq_condition_code);
diff --git a/cranelift/codegen/src/predicates.rs b/cranelift/codegen/src/predicates.rs
index cb56fb0100..16672b145b 100644
--- a/cranelift/codegen/src/predicates.rs
+++ b/cranelift/codegen/src/predicates.rs
@@ -10,8 +10,6 @@
 //! dead code warning.
 
 use crate::ir;
-use crate::ir::condcodes::IntCC;
-use std::string::ToString;
 
 /// Check that an integer value is zero.
 #[allow(dead_code)]
@@ -85,14 +83,6 @@ pub fn has_length_of(value_list: &ir::ValueList, num: usize, func: &ir::Function
     value_list.len(&func.dfg.value_lists) == num
 }
 
-#[allow(dead_code)]
-pub fn match_condition_code_to_str(
-    condition_code: IntCC,
-    stringified_condition_code: &str,
-) -> bool {
-    condition_code.to_string().eq(stringified_condition_code)
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -146,10 +136,4 @@ mod tests {
         assert!(!is_all_ones_128_bit(&[0; 16]));
         assert!(is_all_ones_128_bit(&[0xff; 16]));
     }
-
-    #[test]
-    fn condition_code() {
-        assert!(match_condition_code_to_str(IntCC::Equal, "eq"));
-        assert!(!match_condition_code_to_str(IntCC::Equal, "ne"));
-    }
 }

From 1431ab5201b5affdf8b08838bed39153188f40eb Mon Sep 17 00:00:00 2001
From: Artur Jamro 
Date: Tue, 24 Sep 2019 09:28:40 -0700
Subject: [PATCH 2759/3084] Derive serde traits for TrapCode

---
 cranelift/codegen/src/ir/trapcode.rs | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/cranelift/codegen/src/ir/trapcode.rs b/cranelift/codegen/src/ir/trapcode.rs
index 273a77670c..5621e1d91c 100644
--- a/cranelift/codegen/src/ir/trapcode.rs
+++ b/cranelift/codegen/src/ir/trapcode.rs
@@ -2,11 +2,14 @@
 
 use core::fmt::{self, Display, Formatter};
 use core::str::FromStr;
+#[cfg(feature = "enable-serde")]
+use serde::{Deserialize, Serialize};
 
 /// A trap code describing the reason for a trap.
 ///
 /// All trap instructions have an explicit trap code.
 #[derive(Clone, Copy, PartialEq, Eq, Debug, Hash)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub enum TrapCode {
     /// The current stack space was exhausted.
     ///

From 6e131e5347fcdc33c7ffbdaa0205a2aa29f30765 Mon Sep 17 00:00:00 2001
From: Ujjwal Sharma 
Date: Sat, 7 Sep 2019 00:10:54 +0530
Subject: [PATCH 2760/3084] [codegen] add intcc conditions for reading carry
 flag

Add conditions to IntCC for checking the carry flag (Carry, NotCarry).

Fixes: https://github.com/CraneStation/cranelift/issues/980
---
 cranelift/codegen/meta/src/shared/instructions.rs | 1 +
 cranelift/codegen/src/isa/arm32/mod.rs            | 8 ++++++++
 cranelift/codegen/src/isa/arm64/mod.rs            | 8 ++++++++
 cranelift/codegen/src/isa/mod.rs                  | 6 ++++++
 cranelift/codegen/src/isa/riscv/mod.rs            | 8 ++++++++
 cranelift/codegen/src/isa/x86/mod.rs              | 8 ++++++++
 cranelift/codegen/src/legalizer/heap.rs           | 6 +++++-
 7 files changed, 44 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index b52a8dafed..5461aada55 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1586,6 +1586,7 @@ pub(crate) fn define(
     let x = &operand("x", Int);
     let y = &operand("y", Int);
 
+    // TODO(ryzokuken): Add documentation for unsigned overflow.
     ig.push(
         Inst::new(
             "icmp",
diff --git a/cranelift/codegen/src/isa/arm32/mod.rs b/cranelift/codegen/src/isa/arm32/mod.rs
index ced1b88767..ed5854593b 100644
--- a/cranelift/codegen/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/src/isa/arm32/mod.rs
@@ -126,6 +126,14 @@ impl TargetIsa for Isa {
     fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut MemoryCodeSink) {
         emit_function(func, binemit::emit_inst, sink, self)
     }
+
+    fn unsigned_add_overflow_condition(&self) -> ir::condcodes::IntCC {
+        ir::condcodes::IntCC::UnsignedLessThan
+    }
+
+    fn unsigned_sub_overflow_condition(&self) -> ir::condcodes::IntCC {
+        ir::condcodes::IntCC::UnsignedGreaterThanOrEqual
+    }
 }
 
 impl fmt::Display for Isa {
diff --git a/cranelift/codegen/src/isa/arm64/mod.rs b/cranelift/codegen/src/isa/arm64/mod.rs
index d787524a6a..71ae93c0fe 100644
--- a/cranelift/codegen/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/src/isa/arm64/mod.rs
@@ -114,6 +114,14 @@ impl TargetIsa for Isa {
     fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut MemoryCodeSink) {
         emit_function(func, binemit::emit_inst, sink, self)
     }
+
+    fn unsigned_add_overflow_condition(&self) -> ir::condcodes::IntCC {
+        ir::condcodes::IntCC::UnsignedLessThan
+    }
+
+    fn unsigned_sub_overflow_condition(&self) -> ir::condcodes::IntCC {
+        ir::condcodes::IntCC::UnsignedGreaterThanOrEqual
+    }
 }
 
 impl fmt::Display for Isa {
diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index c36d29ce2b..6457eb9ad7 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -371,4 +371,10 @@ pub trait TargetIsa: fmt::Display + Sync {
 
     /// Emit a whole function into memory.
     fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut binemit::MemoryCodeSink);
+
+    /// IntCC condition for Unsigned Addition Overflow (Carry).
+    fn unsigned_add_overflow_condition(&self) -> ir::condcodes::IntCC;
+
+    /// IntCC condition for Unsigned Subtraction Overflow (Borrow/Carry).
+    fn unsigned_sub_overflow_condition(&self) -> ir::condcodes::IntCC;
 }
diff --git a/cranelift/codegen/src/isa/riscv/mod.rs b/cranelift/codegen/src/isa/riscv/mod.rs
index 233e92a3bb..93ef3787f0 100644
--- a/cranelift/codegen/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/src/isa/riscv/mod.rs
@@ -121,6 +121,14 @@ impl TargetIsa for Isa {
     fn emit_function_to_memory(&self, func: &ir::Function, sink: &mut MemoryCodeSink) {
         emit_function(func, binemit::emit_inst, sink, self)
     }
+
+    fn unsigned_add_overflow_condition(&self) -> ir::condcodes::IntCC {
+        unimplemented!()
+    }
+
+    fn unsigned_sub_overflow_condition(&self) -> ir::condcodes::IntCC {
+        unimplemented!()
+    }
 }
 
 #[cfg(test)]
diff --git a/cranelift/codegen/src/isa/x86/mod.rs b/cranelift/codegen/src/isa/x86/mod.rs
index 52ab055440..8136170cd7 100644
--- a/cranelift/codegen/src/isa/x86/mod.rs
+++ b/cranelift/codegen/src/isa/x86/mod.rs
@@ -142,6 +142,14 @@ impl TargetIsa for Isa {
         let _tt = timing::prologue_epilogue();
         abi::prologue_epilogue(func, self)
     }
+
+    fn unsigned_add_overflow_condition(&self) -> ir::condcodes::IntCC {
+        ir::condcodes::IntCC::UnsignedLessThan
+    }
+
+    fn unsigned_sub_overflow_condition(&self) -> ir::condcodes::IntCC {
+        ir::condcodes::IntCC::UnsignedLessThan
+    }
 }
 
 impl fmt::Display for Isa {
diff --git a/cranelift/codegen/src/legalizer/heap.rs b/cranelift/codegen/src/legalizer/heap.rs
index 332553e803..a6d9d9637d 100644
--- a/cranelift/codegen/src/legalizer/heap.rs
+++ b/cranelift/codegen/src/legalizer/heap.rs
@@ -83,7 +83,11 @@ fn dynamic_addr(
         // We need an overflow check for the adjusted offset.
         let access_size_val = pos.ins().iconst(offset_ty, access_size as i64);
         let (adj_offset, overflow) = pos.ins().iadd_ifcout(offset, access_size_val);
-        pos.ins().trapnz(overflow, ir::TrapCode::HeapOutOfBounds);
+        pos.ins().trapif(
+            isa.unsigned_add_overflow_condition(),
+            overflow,
+            ir::TrapCode::HeapOutOfBounds,
+        );
         oob = pos
             .ins()
             .icmp(IntCC::UnsignedGreaterThan, adj_offset, bound);

From dfdd504edc598149bf6f432cf90fe0f383f3340f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 24 Sep 2019 16:28:17 -0700
Subject: [PATCH 2761/3084] Bump version to 0.44.0

---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  2 +-
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 19 files changed, 68 insertions(+), 68 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index df1224038a..ef5d513f09 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.43.1"
+version = "0.44.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.43.1" }
-cranelift-entity = { path = "cranelift-entity", version = "0.43.1" }
-cranelift-reader = { path = "cranelift-reader", version = "0.43.1" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.43.1" }
-cranelift-serde = { path = "cranelift-serde", version = "0.43.1", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.43.1", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.43.1" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.43.1" }
-cranelift-module = { path = "cranelift-module", version = "0.43.1" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.43.1" }
-cranelift-object = { path = "cranelift-object", version = "0.43.1" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.43.1" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.43.1" }
-cranelift = { path = "cranelift-umbrella", version = "0.43.1" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.44.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.44.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.44.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.44.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.44.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.44.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.44.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.44.0" }
+cranelift-module = { path = "cranelift-module", version = "0.44.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.44.0" }
+cranelift-object = { path = "cranelift-object", version = "0.44.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.44.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.44.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.44.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 4726440e70..960827da52 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.43.1"
+version = "0.44.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.43.1", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 633711930d..eb6a1094c8 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.43.1"
+version = "0.44.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.43.1" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.43.1", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.43.1", default-features = false }
+cranelift-codegen-shared = { path = "./shared", version = "0.44.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.44.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
@@ -29,7 +29,7 @@ smallvec = { version = "0.6.10" }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.43.1", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.44.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index d8bf77a6a9..2d75957355 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.43.1"
+version = "0.44.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.43.1" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.43.1", default-features = false }
+cranelift-codegen-shared = { path = "../shared", version = "0.44.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.44.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 2ceba9eaf1..d58b92fd50 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.43.1"
+version = "0.44.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index d71e63ec5e..65b9bc7aaa 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.43.1"
+version = "0.44.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 98dccdcf45..26d3e5defe 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.43.1"
+version = "0.44.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1" }
-cranelift-module = { path = "../cranelift-module", version = "0.43.1" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.44.0" }
 faerie = "0.11.0"
 goblin = "0.0.24"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 5a7d2aab11..acbeeebb5e 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.43.1"
+version = "0.44.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.43.1" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.43.1" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.43.1" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.44.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.44.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.44.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 919b9b4bd7..e30bde657e 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.43.1"
+version = "0.44.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index ac5eff0822..890c1a5778 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.43.1"
+version = "0.44.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.43.1", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 35576d1d3b..54f05fa369 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.43.1"
+version = "0.44.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
 target-lexicon = "0.8.1"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index d75fb054f3..5cb413e181 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.43.1"
+version = "0.44.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1" }
-cranelift-module = { path = "../cranelift-module", version = "0.43.1" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.44.0" }
 object = { version = "0.14.0", default-features = false, features = ["write"] }
 target-lexicon = "0.8.1"
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 2bf1a1a528..64f246da46 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.43.1"
+version = "0.44.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.43.1", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 8480a4183f..d424fe7718 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.43.1"
+version="0.44.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 769a5b4550..3d97d552a3 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.43.1"
+version = "0.44.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0" }
 target-lexicon = "0.8.1"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 16848ee23e..81b3a2a09f 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.43.1"
+version = "0.44.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.43.1" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.44.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 7078e483b4..a41d9d9f7b 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.43.1"
+version = "0.44.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1" }
-cranelift-module = { path = "../cranelift-module", version = "0.43.1" }
-cranelift-native = { path = "../cranelift-native", version = "0.43.1" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.44.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.44.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.43.1" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.43.1" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.43.1" }
+cranelift = { path = "../cranelift-umbrella", version = "0.44.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.44.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.44.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 3f605832a0..c77cd048d9 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.43.1"
+version = "0.44.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.43.1", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.44.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 71a2a7ca31..fd7f51608d 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.43.1"
+version = "0.44.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.39.1", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.43.1", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.43.1", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.43.1", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.44.0", default-features = false }
 hashmap_core = { version = "0.1.9", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 43a891dfa23d1ed838a7eebef601a38120d53ac5 Mon Sep 17 00:00:00 2001
From: Ujjwal Sharma 
Date: Sat, 7 Sep 2019 17:34:38 +0530
Subject: [PATCH 2762/3084] [codegen] add intcc conditions for reading overflow
 flag

Add conditions to IntCC for checking the overflow flag (Overflow,
NotOverflow).
---
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 48 +++----------------
 .../codegen/meta/src/shared/immediates.rs     |  2 +
 .../codegen/meta/src/shared/instructions.rs   |  1 +
 cranelift/codegen/shared/src/condcodes.rs     | 16 ++++++-
 cranelift/codegen/src/isa/x86/binemit.rs      |  4 +-
 .../filetests/filetests/isa/x86/binary32.clif |  4 ++
 .../filetests/filetests/isa/x86/binary64.clif |  4 ++
 7 files changed, 34 insertions(+), 45 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index da602b9973..13d9dcf833 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -2920,21 +2920,9 @@ pub(crate) fn define<'shared>(
                     {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
                     modrm_rr(in_reg0, in_reg1, sink);
                     // `setCC` instruction, no REX.
-                    use crate::ir::condcodes::IntCC::*;
-                    let setcc = match cond {
-                        Equal => 0x94,
-                        NotEqual => 0x95,
-                        SignedLessThan => 0x9c,
-                        SignedGreaterThanOrEqual => 0x9d,
-                        SignedGreaterThan => 0x9f,
-                        SignedLessThanOrEqual => 0x9e,
-                        UnsignedLessThan => 0x92,
-                        UnsignedGreaterThanOrEqual => 0x93,
-                        UnsignedGreaterThan => 0x97,
-                        UnsignedLessThanOrEqual => 0x96,
-                    };
+                    let setcc = 0x90 | icc2opc(cond);
                     sink.put1(0x0f);
-                    sink.put1(setcc);
+                    sink.put1(setcc as u8);
                     modrm_rr(out_reg0, 0, sink);
                 "#,
             ),
@@ -2971,21 +2959,9 @@ pub(crate) fn define<'shared>(
                         let imm: i64 = imm.into();
                         sink.put1(imm as u8);
                         // `setCC` instruction, no REX.
-                        use crate::ir::condcodes::IntCC::*;
-                        let setcc = match cond {
-                            Equal => 0x94,
-                            NotEqual => 0x95,
-                            SignedLessThan => 0x9c,
-                            SignedGreaterThanOrEqual => 0x9d,
-                            SignedGreaterThan => 0x9f,
-                            SignedLessThanOrEqual => 0x9e,
-                            UnsignedLessThan => 0x92,
-                            UnsignedGreaterThanOrEqual => 0x93,
-                            UnsignedGreaterThan => 0x97,
-                            UnsignedLessThanOrEqual => 0x96,
-                        };
+                        let setcc = 0x90 | icc2opc(cond);
                         sink.put1(0x0f);
-                        sink.put1(setcc);
+                        sink.put1(setcc as u8);
                         modrm_rr(out_reg0, 0, sink);
                     "#,
                 ),
@@ -3006,21 +2982,9 @@ pub(crate) fn define<'shared>(
                         let imm: i64 = imm.into();
                         sink.put4(imm as u32);
                         // `setCC` instruction, no REX.
-                        use crate::ir::condcodes::IntCC::*;
-                        let setcc = match cond {
-                            Equal => 0x94,
-                            NotEqual => 0x95,
-                            SignedLessThan => 0x9c,
-                            SignedGreaterThanOrEqual => 0x9d,
-                            SignedGreaterThan => 0x9f,
-                            SignedLessThanOrEqual => 0x9e,
-                            UnsignedLessThan => 0x92,
-                            UnsignedGreaterThanOrEqual => 0x93,
-                            UnsignedGreaterThan => 0x97,
-                            UnsignedLessThanOrEqual => 0x96,
-                        };
+                        let setcc = 0x90 | icc2opc(cond);
                         sink.put1(0x0f);
-                        sink.put1(setcc);
+                        sink.put1(setcc as u8);
                         modrm_rr(out_reg0, 0, sink);
                     "#,
                 ),
diff --git a/cranelift/codegen/meta/src/shared/immediates.rs b/cranelift/codegen/meta/src/shared/immediates.rs
index 0b5e84c521..f9c114a6a0 100644
--- a/cranelift/codegen/meta/src/shared/immediates.rs
+++ b/cranelift/codegen/meta/src/shared/immediates.rs
@@ -129,6 +129,8 @@ impl Immediates {
                 intcc_values.insert("ugt", "UnsignedGreaterThan");
                 intcc_values.insert("ule", "UnsignedLessThanOrEqual");
                 intcc_values.insert("ult", "UnsignedLessThan");
+                intcc_values.insert("of", "Overflow");
+                intcc_values.insert("nof", "NotOverflow");
                 Builder::new_enum("intcc", intcc_values)
                     .doc("An integer comparison condition code.")
                     .default_member("cond")
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 5461aada55..679ee1a2c1 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1587,6 +1587,7 @@ pub(crate) fn define(
     let y = &operand("y", Int);
 
     // TODO(ryzokuken): Add documentation for unsigned overflow.
+    // TODO(ryzokuken): Add documentation for signed overflow.
     ig.push(
         Inst::new(
             "icmp",
diff --git a/cranelift/codegen/shared/src/condcodes.rs b/cranelift/codegen/shared/src/condcodes.rs
index 743e30954d..1171eef2f6 100644
--- a/cranelift/codegen/shared/src/condcodes.rs
+++ b/cranelift/codegen/shared/src/condcodes.rs
@@ -51,6 +51,10 @@ pub enum IntCC {
     UnsignedGreaterThan,
     /// Unsigned `<=`.
     UnsignedLessThanOrEqual,
+    /// Signed Overflow.
+    Overflow,
+    /// Signed No Overflow.
+    NotOverflow,
 }
 
 impl CondCode for IntCC {
@@ -67,6 +71,8 @@ impl CondCode for IntCC {
             UnsignedGreaterThanOrEqual => UnsignedLessThan,
             UnsignedGreaterThan => UnsignedLessThanOrEqual,
             UnsignedLessThanOrEqual => UnsignedGreaterThan,
+            Overflow => NotOverflow,
+            NotOverflow => Overflow,
         }
     }
 
@@ -83,6 +89,8 @@ impl CondCode for IntCC {
             UnsignedGreaterThanOrEqual => UnsignedLessThanOrEqual,
             UnsignedLessThan => UnsignedGreaterThan,
             UnsignedLessThanOrEqual => UnsignedGreaterThanOrEqual,
+            Overflow => Overflow,
+            NotOverflow => NotOverflow,
         }
     }
 }
@@ -101,6 +109,8 @@ impl Display for IntCC {
             UnsignedGreaterThanOrEqual => "uge",
             UnsignedLessThan => "ult",
             UnsignedLessThanOrEqual => "ule",
+            Overflow => "of",
+            NotOverflow => "nof",
         })
     }
 }
@@ -121,6 +131,8 @@ impl FromStr for IntCC {
             "ugt" => Ok(UnsignedGreaterThan),
             "ule" => Ok(UnsignedLessThanOrEqual),
             "ult" => Ok(UnsignedLessThan),
+            "of" => Ok(Overflow),
+            "nof" => Ok(NotOverflow),
             _ => Err(()),
         }
     }
@@ -270,7 +282,7 @@ mod tests {
     use super::*;
     use std::string::ToString;
 
-    static INT_ALL: [IntCC; 10] = [
+    static INT_ALL: [IntCC; 12] = [
         IntCC::Equal,
         IntCC::NotEqual,
         IntCC::SignedLessThan,
@@ -281,6 +293,8 @@ mod tests {
         IntCC::UnsignedGreaterThanOrEqual,
         IntCC::UnsignedGreaterThan,
         IntCC::UnsignedLessThanOrEqual,
+        IntCC::Overflow,
+        IntCC::NotOverflow,
     ];
 
     #[test]
diff --git a/cranelift/codegen/src/isa/x86/binemit.rs b/cranelift/codegen/src/isa/x86/binemit.rs
index a5924ed030..cda6ae5807 100644
--- a/cranelift/codegen/src/isa/x86/binemit.rs
+++ b/cranelift/codegen/src/isa/x86/binemit.rs
@@ -272,8 +272,8 @@ fn sib(scale: u8, index: RegUnit, base: RegUnit, sink: &m
 fn icc2opc(cond: IntCC) -> u16 {
     use crate::ir::condcodes::IntCC::*;
     match cond {
-        // 0x0 = Overflow.
-        // 0x1 = !Overflow.
+        Overflow => 0x0,
+        NotOverflow => 0x1,
         UnsignedLessThan => 0x2,
         UnsignedGreaterThanOrEqual => 0x3,
         Equal => 0x4,
diff --git a/cranelift/filetests/filetests/isa/x86/binary32.clif b/cranelift/filetests/filetests/isa/x86/binary32.clif
index cc8f638014..5db78ee2e0 100644
--- a/cranelift/filetests/filetests/isa/x86/binary32.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary32.clif
@@ -664,6 +664,10 @@ ebb11:
     trapif ugt v11, user0                          ; bin: 76 02 user0 0f 0b
     ; asm: jnbe .+4; ud2
     trapif ule v11, user0                          ; bin: 77 02 user0 0f 0b
+    ; asm: jo .+4; ud2
+    trapif of v11, user0                          ; bin: 71 02 user0 0f 0b
+    ; asm: jno .+4; ud2
+    trapif nof v11, user0                          ; bin: 70 02 user0 0f 0b
 
     ; Stack check.
     ; asm: cmpl %esp, %ecx
diff --git a/cranelift/filetests/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
index 0430cd78a5..a3fcede60d 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64.clif
@@ -862,6 +862,10 @@ ebb11:
     trapif ugt v11, user0                          ; bin: 76 02 user0 0f 0b
     ; asm: jnbe .+4; ud2
     trapif ule v11, user0                          ; bin: 77 02 user0 0f 0b
+    ; asm: jo .+4; ud2
+    trapif of v11, user0                          ; bin: 71 02 user0 0f 0b
+    ; asm: jno .+4; ud2
+    trapif nof v11, user0                          ; bin: 70 02 user0 0f 0b
 
     ; Debug trap.
     debugtrap ; bin: cc

From b036ab77dc1f2a2f7feaf2508b90a84926f6b073 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 13 Sep 2019 17:57:14 +0200
Subject: [PATCH 2763/3084] Update comment about the live range implementation
 details;

---
 cranelift/codegen/src/regalloc/liverange.rs | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/cranelift/codegen/src/regalloc/liverange.rs b/cranelift/codegen/src/regalloc/liverange.rs
index 5629ea40ec..b0d3c6fed6 100644
--- a/cranelift/codegen/src/regalloc/liverange.rs
+++ b/cranelift/codegen/src/regalloc/liverange.rs
@@ -58,8 +58,16 @@
 //!
 //! # Implementation notes
 //!
-//! A few notes about the implementation of this data structure. This should not concern someone
-//! only looking to use the public interface.
+//! A few notes about the implementation of the live intervals field `liveins`. This should not
+//! concern someone only looking to use the public interface.
+//!
+//! ## Current representation
+//!
+//! Our current implementation uses a B-tree map with the necessary interface for an efficient
+//! implementation of coalescing, implemented as a generic data-structure bforest::Map.
+//!
+//! A `BTreeMap` could have been used for the live-in intervals, but it doesn't provide
+//! the necessary API to make coalescing easy, nor does it optimize for our types' sizes.
 //!
 //! ## EBB ordering
 //!
@@ -99,13 +107,6 @@
 //! It is more complicated to work with, though, so it is probably not worth it. The performance
 //! benefits of switching to a numerical EBB order only appears if the binary search is doing
 //! EBB-EBB comparisons.
-//!
-//! ## B-tree representation
-//!
-//! A `BTreeMap` could also be used for the live-in intervals. It looks like the
-//! standard library B-tree doesn't provide the necessary interface for an efficient implementation
-//! of coalescing, so we would need to roll our own.
-//!
 
 use crate::bforest;
 use crate::entity::SparseMapValue;

From 59f5f12c60b070445bb31c7fb79867ea13ea3783 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 12 Sep 2019 14:23:12 +0200
Subject: [PATCH 2764/3084] [codegen] Rename GenLiveRange to GenericLiveRange;

(to avoid confuson with Gen interpreted as Generator)
---
 cranelift/codegen/src/ir/layout.rs          |  2 +-
 cranelift/codegen/src/regalloc/liverange.rs | 32 ++++++++++++---------
 2 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/cranelift/codegen/src/ir/layout.rs b/cranelift/codegen/src/ir/layout.rs
index 78a4628576..632c03dc59 100644
--- a/cranelift/codegen/src/ir/layout.rs
+++ b/cranelift/codegen/src/ir/layout.rs
@@ -698,7 +698,7 @@ impl Layout {
 
 #[derive(Clone, Debug, Default)]
 struct InstNode {
-    // The Ebb containing this instruction, or `None` if the instruction is not yet inserted.
+    /// The Ebb containing this instruction, or `None` if the instruction is not yet inserted.
     ebb: PackedOption,
     prev: PackedOption,
     next: PackedOption,
diff --git a/cranelift/codegen/src/regalloc/liverange.rs b/cranelift/codegen/src/regalloc/liverange.rs
index b0d3c6fed6..49d7f23cb2 100644
--- a/cranelift/codegen/src/regalloc/liverange.rs
+++ b/cranelift/codegen/src/regalloc/liverange.rs
@@ -142,13 +142,13 @@ use core::marker::PhantomData;
 /// Inserting new instructions in the layout is safe, but removing instructions is not. Besides the
 /// instructions using or defining their value, `LiveRange` structs can contain references to
 /// branch and jump instructions.
-pub type LiveRange = GenLiveRange;
+pub type LiveRange = GenericLiveRange;
 
 /// Generic live range implementation.
 ///
 /// The intended generic parameter is `PO=Layout`, but tests are simpler with a mock order.
 /// Use `LiveRange` instead of using this generic directly.
-pub struct GenLiveRange {
+pub struct GenericLiveRange {
     /// The value described by this live range.
     /// This member can't be modified in case the live range is stored in a `SparseMap`.
     value: Value,
@@ -216,7 +216,7 @@ impl<'a, PO: ProgramOrder> bforest::Comparator for Cmp<'a, PO> {
     }
 }
 
-impl GenLiveRange {
+impl GenericLiveRange {
     /// Create a new live range for `value` defined at `def`.
     ///
     /// The live range will be created as dead, but it can be extended with `extend_in_ebb()`.
@@ -307,7 +307,7 @@ impl GenLiveRange {
             c.insert(ebb, to);
         }
 
-        // Now `c` to left pointing at an interval that ends in `to`.
+        // Now `c` is left pointing at an interval that ends in `to`.
         debug_assert_eq!(c.value(), Some(to));
 
         // See if it can be coalesced with the following interval.
@@ -449,7 +449,7 @@ impl GenLiveRange {
 }
 
 /// Allow a `LiveRange` to be stored in a `SparseMap` indexed by values.
-impl SparseMapValue for GenLiveRange {
+impl SparseMapValue for GenericLiveRange {
     fn key(&self) -> Value {
         self.value
     }
@@ -457,7 +457,7 @@ impl SparseMapValue for GenLiveRange {
 
 #[cfg(test)]
 mod tests {
-    use super::{GenLiveRange, LiveRangeContext};
+    use super::{GenericLiveRange, LiveRangeContext};
     use crate::bforest;
     use crate::entity::EntityRef;
     use crate::ir::{Ebb, Inst, Value};
@@ -510,7 +510,11 @@ mod tests {
         }
 
         // Validate the live range invariants.
-        fn validate(&self, lr: &GenLiveRange, forest: &bforest::MapForest) {
+        fn validate(
+            &self,
+            lr: &GenericLiveRange,
+            forest: &bforest::MapForest,
+        ) {
             // The def interval must cover a single EBB.
             let def_ebb = self.pp_ebb(lr.def_begin);
             assert_eq!(def_ebb, self.pp_ebb(lr.def_end));
@@ -554,7 +558,7 @@ mod tests {
         let i1 = Inst::new(1);
         let i2 = Inst::new(2);
         let e2 = Ebb::new(2);
-        let lr = GenLiveRange::new(v0, i1.into(), Default::default());
+        let lr = GenericLiveRange::new(v0, i1.into(), Default::default());
         let forest = &bforest::MapForest::new();
         let ctx = LiveRangeContext::new(PO, forest);
         assert!(lr.is_dead());
@@ -574,7 +578,7 @@ mod tests {
     fn dead_arg_range() {
         let v0 = Value::new(0);
         let e2 = Ebb::new(2);
-        let lr = GenLiveRange::new(v0, e2.into(), Default::default());
+        let lr = GenericLiveRange::new(v0, e2.into(), Default::default());
         let forest = &bforest::MapForest::new();
         let ctx = LiveRangeContext::new(PO, forest);
         assert!(lr.is_dead());
@@ -593,7 +597,7 @@ mod tests {
         let i11 = Inst::new(11);
         let i12 = Inst::new(12);
         let i13 = Inst::new(13);
-        let mut lr = GenLiveRange::new(v0, i11.into(), Default::default());
+        let mut lr = GenericLiveRange::new(v0, i11.into(), Default::default());
         let forest = &mut bforest::MapForest::new();
 
         assert_eq!(lr.extend_in_ebb(e10, i13, PO, forest), false);
@@ -617,7 +621,7 @@ mod tests {
         let i11 = Inst::new(11);
         let i12 = Inst::new(12);
         let i13 = Inst::new(13);
-        let mut lr = GenLiveRange::new(v0, e10.into(), Default::default());
+        let mut lr = GenericLiveRange::new(v0, e10.into(), Default::default());
         let forest = &mut bforest::MapForest::new();
 
         // Extending a dead EBB argument in its own block should not indicate that a live-in
@@ -652,7 +656,7 @@ mod tests {
         let i21 = Inst::new(21);
         let i22 = Inst::new(22);
         let i23 = Inst::new(23);
-        let mut lr = GenLiveRange::new(v0, i11.into(), Default::default());
+        let mut lr = GenericLiveRange::new(v0, i11.into(), Default::default());
         let forest = &mut bforest::MapForest::new();
 
         assert_eq!(lr.extend_in_ebb(e10, i12, PO, forest), false);
@@ -691,7 +695,7 @@ mod tests {
         let i31 = Inst::new(31);
         let e40 = Ebb::new(40);
         let i41 = Inst::new(41);
-        let mut lr = GenLiveRange::new(v0, i11.into(), Default::default());
+        let mut lr = GenericLiveRange::new(v0, i11.into(), Default::default());
         let forest = &mut bforest::MapForest::new();
 
         assert_eq!(lr.extend_in_ebb(e30, i31, PO, forest), true);
@@ -717,7 +721,7 @@ mod tests {
             [(e20, i41)]
         );
 
-        let mut lr = GenLiveRange::new(v0, i11.into(), Default::default());
+        let mut lr = GenericLiveRange::new(v0, i11.into(), Default::default());
 
         assert_eq!(lr.extend_in_ebb(e40, i41, PO, forest), true);
         assert_eq!(

From b9b1c842e9606c16f3dc17e28f6c80a068434634 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 12 Sep 2019 14:43:31 +0200
Subject: [PATCH 2765/3084] Use LiveRange::reaches_use instead of
 reimplementing it in liveness checks;

---
 cranelift/codegen/src/verifier/liveness.rs | 32 ++++++----------------
 1 file changed, 9 insertions(+), 23 deletions(-)

diff --git a/cranelift/codegen/src/verifier/liveness.rs b/cranelift/codegen/src/verifier/liveness.rs
index c396a46803..11cde21e6f 100644
--- a/cranelift/codegen/src/verifier/liveness.rs
+++ b/cranelift/codegen/src/verifier/liveness.rs
@@ -2,13 +2,12 @@
 
 use crate::flowgraph::{BasicBlock, ControlFlowGraph};
 use crate::ir::entities::AnyEntity;
-use crate::ir::{ExpandedProgramPoint, Function, Inst, ProgramOrder, ProgramPoint, Value};
+use crate::ir::{ExpandedProgramPoint, Function, ProgramPoint, Value};
 use crate::isa::TargetIsa;
 use crate::regalloc::liveness::Liveness;
 use crate::regalloc::liverange::LiveRange;
 use crate::timing;
 use crate::verifier::{VerifierErrors, VerifierStepResult};
-use core::cmp::Ordering;
 
 /// Verify liveness information for `func`.
 ///
@@ -65,6 +64,7 @@ impl<'a> LivenessVerifier<'a> {
 
     /// Check all instructions.
     fn check_insts(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
+        let lr_ctx = self.liveness.context(&self.func.layout);
         for ebb in self.func.layout.ebbs() {
             for inst in self.func.layout.ebb_insts(ebb) {
                 let encoding = self.func.encodings[inst];
@@ -106,7 +106,9 @@ impl<'a> LivenessVerifier<'a> {
                         Some(lr) => lr,
                         None => return fatal!(errors, inst, "{} has no live range", val),
                     };
-                    if !self.live_at_use(lr, inst) {
+
+                    debug_assert!(lr_ctx.order.inst_ebb(inst).unwrap() == ebb);
+                    if !lr.reaches_use(inst, ebb, lr_ctx) {
                         return fatal!(errors, inst, "{} is not live at this use", val);
                     }
 
@@ -126,24 +128,6 @@ impl<'a> LivenessVerifier<'a> {
         Ok(())
     }
 
-    /// Is `lr` live at the use `inst`?
-    fn live_at_use(&self, lr: &LiveRange, inst: Inst) -> bool {
-        let ctx = self.liveness.context(&self.func.layout);
-
-        // Check if `inst` is in the def range, not including the def itself.
-        if ctx.order.cmp(lr.def(), inst) == Ordering::Less
-            && ctx.order.cmp(inst, lr.def_local_end()) != Ordering::Greater
-        {
-            return true;
-        }
-
-        // Otherwise see if `inst` is in one of the live-in ranges.
-        match lr.livein_local_end(ctx.order.inst_ebb(inst).unwrap(), ctx) {
-            Some(end) => ctx.order.cmp(inst, end) != Ordering::Greater,
-            None => false,
-        }
-    }
-
     /// Check the integrity of the live range `lr`.
     fn check_lr(
         &self,
@@ -220,11 +204,13 @@ impl<'a> LivenessVerifier<'a> {
                 }
             };
 
+            let lr_ctx = self.liveness.context(&self.func.layout);
+
             // Check all the EBBs in the interval independently.
             loop {
                 // If `val` is live-in at `ebb`, it must be live at all the predecessors.
-                for BasicBlock { inst: pred, .. } in self.cfg.pred_iter(ebb) {
-                    if !self.live_at_use(lr, pred) {
+                for BasicBlock { inst: pred, ebb } in self.cfg.pred_iter(ebb) {
+                    if !lr.reaches_use(pred, ebb, lr_ctx) {
                         return fatal!(
                             errors,
                             pred,

From 5beb10e77a8cdd7203597ab4d1d72518482e861e Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 12 Sep 2019 19:32:43 +0200
Subject: [PATCH 2766/3084] Regalloc: remove the transient LiveRangeContext
 data structure;

---
 cranelift/codegen/src/regalloc/coalescing.rs  |  29 ++--
 cranelift/codegen/src/regalloc/coloring.rs    |  28 ++--
 .../src/regalloc/live_value_tracker.rs        |   3 +-
 cranelift/codegen/src/regalloc/liveness.rs    |  12 +-
 cranelift/codegen/src/regalloc/liverange.rs   | 136 +++++-------------
 cranelift/codegen/src/regalloc/spilling.rs    |   7 +-
 cranelift/codegen/src/value_label.rs          |   9 +-
 cranelift/codegen/src/verifier/cssa.rs        |   8 +-
 cranelift/codegen/src/verifier/liveness.rs    |  11 +-
 cranelift/codegen/src/verifier/locations.rs   |   8 +-
 10 files changed, 108 insertions(+), 143 deletions(-)

diff --git a/cranelift/codegen/src/regalloc/coalescing.rs b/cranelift/codegen/src/regalloc/coalescing.rs
index 4ba19ebbf4..8a1f1d81bb 100644
--- a/cranelift/codegen/src/regalloc/coalescing.rs
+++ b/cranelift/codegen/src/regalloc/coalescing.rs
@@ -199,7 +199,8 @@ impl<'a> Context<'a> {
                 if self.liveness[param].reaches_use(
                     pred_inst,
                     pred_ebb,
-                    self.liveness.context(&self.func.layout),
+                    self.liveness.forest(),
+                    &self.func.layout,
                 ) {
                     self.isolate_param(ebb, param);
                 }
@@ -240,7 +241,6 @@ impl<'a> Context<'a> {
             // `ebb`, it can never be used as an EBB argument.
             let interference = {
                 let lr = &self.liveness[arg];
-                let ctx = self.liveness.context(&self.func.layout);
 
                 // There are two ways the argument value can interfere with `ebb`:
                 //
@@ -255,7 +255,7 @@ impl<'a> Context<'a> {
                 );
 
                 // The only other possibility is that `arg` is live-in to `ebb`.
-                lr.is_livein(ebb, ctx)
+                lr.is_livein(ebb, self.liveness.forest(), &self.func.layout)
             };
 
             if interference {
@@ -435,8 +435,12 @@ impl<'a> Context<'a> {
 
             // Check for interference between `parent` and `value`. Since `parent` dominates
             // `value`, we only have to check if it overlaps the definition.
-            let ctx = self.liveness.context(&self.func.layout);
-            if self.liveness[parent.value].overlaps_def(node.def, node.ebb, ctx) {
+            if self.liveness[parent.value].overlaps_def(
+                node.def,
+                node.ebb,
+                self.liveness.forest(),
+                &self.func.layout,
+            ) {
                 // The two values are interfering, so they can't be in the same virtual register.
                 debug!("-> interference: {} overlaps def of {}", parent, value);
                 return false;
@@ -593,7 +597,6 @@ impl<'a> Context<'a> {
         // This gives us the closest dominating value def for each of the values.
         self.forest.clear();
         self.values.clear();
-        let ctx = self.liveness.context(&self.func.layout);
         for node in nodes {
             // Accumulate ordered values for the new vreg.
             if node.is_value() {
@@ -623,7 +626,12 @@ impl<'a> Context<'a> {
                 // Check if the parent value interferes with the virtual copy.
                 let inst = node.def.unwrap_inst();
                 if node.set_id != parent.set_id
-                    && self.liveness[parent.value].reaches_use(inst, node.ebb, ctx)
+                    && self.liveness[parent.value].reaches_use(
+                        inst,
+                        node.ebb,
+                        self.liveness.forest(),
+                        &self.func.layout,
+                    )
                 {
                     debug!(
                         " - interference: {} overlaps vcopy at {}:{}",
@@ -647,7 +655,12 @@ impl<'a> Context<'a> {
             // Both node and parent are values, so check for interference.
             debug_assert!(node.is_value() && parent.is_value());
             if node.set_id != parent.set_id
-                && self.liveness[parent.value].overlaps_def(node.def, node.ebb, ctx)
+                && self.liveness[parent.value].overlaps_def(
+                    node.def,
+                    node.ebb,
+                    self.liveness.forest(),
+                    &self.func.layout,
+                )
             {
                 // The two values are interfering.
                 debug!(" - interference: {} overlaps def of {}", parent, node.value);
diff --git a/cranelift/codegen/src/regalloc/coloring.rs b/cranelift/codegen/src/regalloc/coloring.rs
index 1e69c342a9..23a546a479 100644
--- a/cranelift/codegen/src/regalloc/coloring.rs
+++ b/cranelift/codegen/src/regalloc/coloring.rs
@@ -54,7 +54,7 @@ use crate::regalloc::affinity::Affinity;
 use crate::regalloc::diversion::RegDiversions;
 use crate::regalloc::live_value_tracker::{LiveValue, LiveValueTracker};
 use crate::regalloc::liveness::Liveness;
-use crate::regalloc::liverange::{LiveRange, LiveRangeContext};
+use crate::regalloc::liverange::{LiveRange, LiveRangeForest};
 use crate::regalloc::register_set::RegisterSet;
 use crate::regalloc::solver::{Solver, SolverError};
 use crate::timing;
@@ -461,7 +461,7 @@ impl<'a> Context<'a> {
                     "Can't handle EBB arguments: {}",
                     self.cur.display_inst(inst)
                 );
-                self.undivert_regs(|lr, _| !lr.is_local());
+                self.undivert_regs(|lr, _, _| !lr.is_local());
             }
         }
 
@@ -725,8 +725,13 @@ impl<'a> Context<'a> {
                         // This code runs after calling `solver.inputs_done()` so we must identify
                         // the new variable as killed or live-through. Always special-case the
                         // pinned register as a through variable.
-                        let ctx = self.liveness.context(&self.cur.func.layout);
-                        if self.liveness[value].killed_at(inst, ctx.order.pp_ebb(inst), ctx) {
+                        let layout = &self.cur.func.layout;
+                        if self.liveness[value].killed_at(
+                            inst,
+                            layout.pp_ebb(inst),
+                            self.liveness.forest(),
+                            layout,
+                        ) {
                             self.solver.add_killed_var(value, op.regclass, cur_reg);
                         } else {
                             self.solver.add_through_var(value, op.regclass, cur_reg);
@@ -755,7 +760,7 @@ impl<'a> Context<'a> {
         //
         // Values with a global live range that are not live in to `dest` could appear as branch
         // arguments, so they can't always be un-diverted.
-        self.undivert_regs(|lr, ctx| lr.is_livein(dest, ctx));
+        self.undivert_regs(|lr, forest, layout| lr.is_livein(dest, forest, layout));
 
         // Now handle the EBB arguments.
         let br_args = self.cur.func.dfg.inst_variable_args(inst);
@@ -825,14 +830,14 @@ impl<'a> Context<'a> {
     /// are reallocated to their global register assignments.
     fn undivert_regs(&mut self, mut pred: Pred)
     where
-        Pred: FnMut(&LiveRange, LiveRangeContext) -> bool,
+        Pred: FnMut(&LiveRange, &LiveRangeForest, &Layout) -> bool,
     {
         for (&value, rdiv) in self.divert.iter() {
             let lr = self
                 .liveness
                 .get(value)
                 .expect("Missing live range for diverted register");
-            if pred(lr, self.liveness.context(&self.cur.func.layout)) {
+            if pred(lr, self.liveness.forest(), &self.cur.func.layout) {
                 if let Affinity::Reg(rci) = lr.affinity {
                     let rc = self.reginfo.rc(rci);
                     // Stack diversions should not be possible here. They only live transiently
@@ -1080,20 +1085,21 @@ impl<'a> Context<'a> {
         use crate::ir::instructions::BranchInfo::*;
 
         let inst = self.cur.current_inst().expect("Not on an instruction");
-        let ctx = self.liveness.context(&self.cur.func.layout);
+        let layout = &self.cur.func.layout;
+        let forest = self.liveness.forest();
         match self.cur.func.dfg.analyze_branch(inst) {
             NotABranch => false,
             SingleDest(ebb, _) => {
                 let lr = &self.liveness[value];
-                lr.is_livein(ebb, ctx)
+                lr.is_livein(ebb, forest, layout)
             }
             Table(jt, ebb) => {
                 let lr = &self.liveness[value];
                 !lr.is_local()
-                    && (ebb.map_or(false, |ebb| lr.is_livein(ebb, ctx))
+                    && (ebb.map_or(false, |ebb| lr.is_livein(ebb, forest, layout))
                         || self.cur.func.jump_tables[jt]
                             .iter()
-                            .any(|ebb| lr.is_livein(*ebb, ctx)))
+                            .any(|ebb| lr.is_livein(*ebb, forest, layout)))
             }
         }
     }
diff --git a/cranelift/codegen/src/regalloc/live_value_tracker.rs b/cranelift/codegen/src/regalloc/live_value_tracker.rs
index 1b08862e37..adfe56c410 100644
--- a/cranelift/codegen/src/regalloc/live_value_tracker.rs
+++ b/cranelift/codegen/src/regalloc/live_value_tracker.rs
@@ -191,7 +191,6 @@ impl LiveValueTracker {
                 .idom_sets
                 .get(&idom)
                 .expect("No stored live set for dominator");
-            let ctx = liveness.context(layout);
             // Get just the values that are live-in to `ebb`.
             for &value in idom_live_list.as_slice(&self.idom_pool) {
                 let lr = liveness
@@ -199,7 +198,7 @@ impl LiveValueTracker {
                     .expect("Immediate dominator value has no live range");
 
                 // Check if this value is live-in here.
-                if let Some(endpoint) = lr.livein_local_end(ebb, ctx) {
+                if let Some(endpoint) = lr.livein_local_end(ebb, liveness.forest(), layout) {
                     self.live.push(value, endpoint, lr);
                 }
             }
diff --git a/cranelift/codegen/src/regalloc/liveness.rs b/cranelift/codegen/src/regalloc/liveness.rs
index 7b31aedd54..1ace6bf6d1 100644
--- a/cranelift/codegen/src/regalloc/liveness.rs
+++ b/cranelift/codegen/src/regalloc/liveness.rs
@@ -181,7 +181,7 @@ use crate::ir::dfg::ValueDef;
 use crate::ir::{Ebb, Function, Inst, Layout, ProgramPoint, Value};
 use crate::isa::{EncInfo, OperandConstraint, TargetIsa};
 use crate::regalloc::affinity::Affinity;
-use crate::regalloc::liverange::{LiveRange, LiveRangeContext, LiveRangeForest};
+use crate::regalloc::liverange::{LiveRange, LiveRangeForest};
 use crate::timing;
 use core::mem;
 use core::ops::Index;
@@ -314,16 +314,16 @@ impl Liveness {
         }
     }
 
+    /// Current forest storage.
+    pub fn forest(&self) -> &LiveRangeForest {
+        &self.forest
+    }
+
     /// Current live ranges.
     pub fn ranges(&self) -> &LiveRangeSet {
         &self.ranges
     }
 
-    /// Get a context needed for working with a `LiveRange`.
-    pub fn context<'a>(&'a self, layout: &'a Layout) -> LiveRangeContext<'a, Layout> {
-        LiveRangeContext::new(layout, &self.forest)
-    }
-
     /// Clear all data structures in this liveness analysis.
     pub fn clear(&mut self) {
         self.ranges.clear();
diff --git a/cranelift/codegen/src/regalloc/liverange.rs b/cranelift/codegen/src/regalloc/liverange.rs
index 49d7f23cb2..6465b9ab8e 100644
--- a/cranelift/codegen/src/regalloc/liverange.rs
+++ b/cranelift/codegen/src/regalloc/liverange.rs
@@ -179,32 +179,6 @@ pub struct GenericLiveRange {
     po: PhantomData<*const PO>,
 }
 
-/// Context information needed to query a `LiveRange`.
-pub struct LiveRangeContext<'a, PO: 'a + ProgramOrder> {
-    /// Ordering of EBBs.
-    pub order: &'a PO,
-    /// Memory pool.
-    pub forest: &'a bforest::MapForest,
-}
-
-impl<'a, PO: ProgramOrder> LiveRangeContext<'a, PO> {
-    /// Make a new context.
-    pub fn new(order: &'a PO, forest: &'a bforest::MapForest) -> Self {
-        Self { order, forest }
-    }
-}
-
-impl<'a, PO: ProgramOrder> Clone for LiveRangeContext<'a, PO> {
-    fn clone(&self) -> Self {
-        LiveRangeContext {
-            order: self.order,
-            forest: self.forest,
-        }
-    }
-}
-
-impl<'a, PO: ProgramOrder> Copy for LiveRangeContext<'a, PO> {}
-
 /// Forest of B-trees used for storing live ranges.
 pub type LiveRangeForest = bforest::MapForest;
 
@@ -371,13 +345,13 @@ impl GenericLiveRange {
     /// If the live range is live through all of `ebb`, the terminator of `ebb` is a correct
     /// answer, but it is also possible that an even later program point is returned. So don't
     /// depend on the returned `Inst` to belong to `ebb`.
-    pub fn livein_local_end(&self, ebb: Ebb, ctx: LiveRangeContext) -> Option {
-        let cmp = Cmp(ctx.order);
+    pub fn livein_local_end(&self, ebb: Ebb, forest: &LiveRangeForest, order: &PO) -> Option {
+        let cmp = Cmp(order);
         self.liveins
-            .get_or_less(ebb, ctx.forest, &cmp)
+            .get_or_less(ebb, forest, &cmp)
             .and_then(|(_, inst)| {
                 // We have an entry that ends at `inst`.
-                if ctx.order.cmp(inst, ebb) == Ordering::Greater {
+                if order.cmp(inst, ebb) == Ordering::Greater {
                     Some(inst)
                 } else {
                     None
@@ -388,16 +362,16 @@ impl GenericLiveRange {
     /// Is this value live-in to `ebb`?
     ///
     /// An EBB argument is not considered to be live in.
-    pub fn is_livein(&self, ebb: Ebb, ctx: LiveRangeContext) -> bool {
-        self.livein_local_end(ebb, ctx).is_some()
+    pub fn is_livein(&self, ebb: Ebb, forest: &LiveRangeForest, order: &PO) -> bool {
+        self.livein_local_end(ebb, forest, order).is_some()
     }
 
     /// Get all the live-in intervals.
     ///
     /// Note that the intervals are stored in a compressed form so each entry may span multiple
     /// EBBs where the value is live in.
-    pub fn liveins<'a>(&'a self, ctx: LiveRangeContext<'a, PO>) -> bforest::MapIter<'a, Ebb, Inst> {
-        self.liveins.iter(ctx.forest)
+    pub fn liveins<'a>(&'a self, forest: &'a LiveRangeForest) -> bforest::MapIter<'a, Ebb, Inst> {
+        self.liveins.iter(forest)
     }
 
     /// Check if this live range overlaps a definition in `ebb`.
@@ -405,7 +379,8 @@ impl GenericLiveRange {
         &self,
         def: ExpandedProgramPoint,
         ebb: Ebb,
-        ctx: LiveRangeContext,
+        forest: &LiveRangeForest,
+        order: &PO,
     ) -> bool {
         // Two defs at the same program point always overlap, even if one is dead.
         if def == self.def_begin.into() {
@@ -413,38 +388,39 @@ impl GenericLiveRange {
         }
 
         // Check for an overlap with the local range.
-        if ctx.order.cmp(def, self.def_begin) != Ordering::Less
-            && ctx.order.cmp(def, self.def_end) == Ordering::Less
+        if order.cmp(def, self.def_begin) != Ordering::Less
+            && order.cmp(def, self.def_end) == Ordering::Less
         {
             return true;
         }
 
         // Check for an overlap with a live-in range.
-        match self.livein_local_end(ebb, ctx) {
-            Some(inst) => ctx.order.cmp(def, inst) == Ordering::Less,
+        match self.livein_local_end(ebb, forest, order) {
+            Some(inst) => order.cmp(def, inst) == Ordering::Less,
             None => false,
         }
     }
 
     /// Check if this live range reaches a use at `user` in `ebb`.
-    pub fn reaches_use(&self, user: Inst, ebb: Ebb, ctx: LiveRangeContext) -> bool {
+    pub fn reaches_use(&self, user: Inst, ebb: Ebb, forest: &LiveRangeForest, order: &PO) -> bool {
         // Check for an overlap with the local range.
-        if ctx.order.cmp(user, self.def_begin) == Ordering::Greater
-            && ctx.order.cmp(user, self.def_end) != Ordering::Greater
+        if order.cmp(user, self.def_begin) == Ordering::Greater
+            && order.cmp(user, self.def_end) != Ordering::Greater
         {
             return true;
         }
 
         // Check for an overlap with a live-in range.
-        match self.livein_local_end(ebb, ctx) {
-            Some(inst) => ctx.order.cmp(user, inst) != Ordering::Greater,
+        match self.livein_local_end(ebb, forest, order) {
+            Some(inst) => order.cmp(user, inst) != Ordering::Greater,
             None => false,
         }
     }
 
     /// Check if this live range is killed at `user` in `ebb`.
-    pub fn killed_at(&self, user: Inst, ebb: Ebb, ctx: LiveRangeContext) -> bool {
-        self.def_local_end() == user.into() || self.livein_local_end(ebb, ctx) == Some(user)
+    pub fn killed_at(&self, user: Inst, ebb: Ebb, forest: &LiveRangeForest, order: &PO) -> bool {
+        self.def_local_end() == user.into()
+            || self.livein_local_end(ebb, forest, order) == Some(user)
     }
 }
 
@@ -457,7 +433,7 @@ impl SparseMapValue for GenericLiveRange {
 
 #[cfg(test)]
 mod tests {
-    use super::{GenericLiveRange, LiveRangeContext};
+    use super::GenericLiveRange;
     use crate::bforest;
     use crate::entity::EntityRef;
     use crate::ir::{Ebb, Inst, Value};
@@ -560,18 +536,17 @@ mod tests {
         let e2 = Ebb::new(2);
         let lr = GenericLiveRange::new(v0, i1.into(), Default::default());
         let forest = &bforest::MapForest::new();
-        let ctx = LiveRangeContext::new(PO, forest);
         assert!(lr.is_dead());
         assert!(lr.is_local());
         assert_eq!(lr.def(), i1.into());
         assert_eq!(lr.def_local_end(), i1.into());
-        assert_eq!(lr.livein_local_end(e2, ctx), None);
-        PO.validate(&lr, ctx.forest);
+        assert_eq!(lr.livein_local_end(e2, forest, PO), None);
+        PO.validate(&lr, forest);
 
         // A dead live range overlaps its own def program point.
-        assert!(lr.overlaps_def(i1.into(), e0, ctx));
-        assert!(!lr.overlaps_def(i2.into(), e0, ctx));
-        assert!(!lr.overlaps_def(e0.into(), e0, ctx));
+        assert!(lr.overlaps_def(i1.into(), e0, forest, PO));
+        assert!(!lr.overlaps_def(i2.into(), e0, forest, PO));
+        assert!(!lr.overlaps_def(e0.into(), e0, forest, PO));
     }
 
     #[test]
@@ -580,14 +555,13 @@ mod tests {
         let e2 = Ebb::new(2);
         let lr = GenericLiveRange::new(v0, e2.into(), Default::default());
         let forest = &bforest::MapForest::new();
-        let ctx = LiveRangeContext::new(PO, forest);
         assert!(lr.is_dead());
         assert!(lr.is_local());
         assert_eq!(lr.def(), e2.into());
         assert_eq!(lr.def_local_end(), e2.into());
         // The def interval of an EBB argument does not count as live-in.
-        assert_eq!(lr.livein_local_end(e2, ctx), None);
-        PO.validate(&lr, ctx.forest);
+        assert_eq!(lr.livein_local_end(e2, forest, PO), None);
+        PO.validate(&lr, forest);
     }
 
     #[test]
@@ -664,25 +638,16 @@ mod tests {
         // Adding a live-in interval.
         assert_eq!(lr.extend_in_ebb(e20, i22, PO, forest), true);
         PO.validate(&lr, forest);
-        assert_eq!(
-            lr.livein_local_end(e20, LiveRangeContext::new(PO, forest)),
-            Some(i22)
-        );
+        assert_eq!(lr.livein_local_end(e20, forest, PO), Some(i22));
 
         // Non-extending the live-in.
         assert_eq!(lr.extend_in_ebb(e20, i21, PO, forest), false);
-        assert_eq!(
-            lr.livein_local_end(e20, LiveRangeContext::new(PO, forest)),
-            Some(i22)
-        );
+        assert_eq!(lr.livein_local_end(e20, forest, PO), Some(i22));
 
         // Extending the existing live-in.
         assert_eq!(lr.extend_in_ebb(e20, i23, PO, forest), false);
         PO.validate(&lr, forest);
-        assert_eq!(
-            lr.livein_local_end(e20, LiveRangeContext::new(PO, forest)),
-            Some(i23)
-        );
+        assert_eq!(lr.livein_local_end(e20, forest, PO), Some(i23));
     }
 
     #[test]
@@ -699,52 +664,29 @@ mod tests {
         let forest = &mut bforest::MapForest::new();
 
         assert_eq!(lr.extend_in_ebb(e30, i31, PO, forest), true);
-        assert_eq!(
-            lr.liveins(LiveRangeContext::new(PO, forest))
-                .collect::>(),
-            [(e30, i31)]
-        );
+        assert_eq!(lr.liveins(forest).collect::>(), [(e30, i31)]);
 
         // Coalesce to previous
         assert_eq!(lr.extend_in_ebb(e40, i41, PO, forest), true);
-        assert_eq!(
-            lr.liveins(LiveRangeContext::new(PO, forest))
-                .collect::>(),
-            [(e30, i41)]
-        );
+        assert_eq!(lr.liveins(forest).collect::>(), [(e30, i41)]);
 
         // Coalesce to next
         assert_eq!(lr.extend_in_ebb(e20, i21, PO, forest), true);
-        assert_eq!(
-            lr.liveins(LiveRangeContext::new(PO, forest))
-                .collect::>(),
-            [(e20, i41)]
-        );
+        assert_eq!(lr.liveins(forest).collect::>(), [(e20, i41)]);
 
         let mut lr = GenericLiveRange::new(v0, i11.into(), Default::default());
 
         assert_eq!(lr.extend_in_ebb(e40, i41, PO, forest), true);
-        assert_eq!(
-            lr.liveins(LiveRangeContext::new(PO, forest))
-                .collect::>(),
-            [(e40, i41)]
-        );
+        assert_eq!(lr.liveins(forest).collect::>(), [(e40, i41)]);
 
         assert_eq!(lr.extend_in_ebb(e20, i21, PO, forest), true);
         assert_eq!(
-            lr.liveins(LiveRangeContext::new(PO, forest))
-                .collect::>(),
+            lr.liveins(forest).collect::>(),
             [(e20, i21), (e40, i41)]
         );
 
         // Coalesce to previous and next
         assert_eq!(lr.extend_in_ebb(e30, i31, PO, forest), true);
-        assert_eq!(
-            lr.liveins(LiveRangeContext::new(PO, forest))
-                .collect::>(),
-            [(e20, i41)]
-        );
+        assert_eq!(lr.liveins(forest).collect::>(), [(e20, i41)]);
     }
-
-    // TODO: Add more tests that exercise the binary search algorithm.
 }
diff --git a/cranelift/codegen/src/regalloc/spilling.rs b/cranelift/codegen/src/regalloc/spilling.rs
index f45dd41c94..0012ad01d8 100644
--- a/cranelift/codegen/src/regalloc/spilling.rs
+++ b/cranelift/codegen/src/regalloc/spilling.rs
@@ -319,17 +319,18 @@ impl<'a> Context<'a> {
             for (idx, (op, &arg)) in constraints.ins.iter().zip(args).enumerate() {
                 let mut reguse = RegUse::new(arg, idx, op.regclass.into());
                 let lr = &self.liveness[arg];
-                let ctx = self.liveness.context(&self.cur.func.layout);
                 match op.kind {
                     ConstraintKind::Stack => continue,
                     ConstraintKind::FixedReg(_) => reguse.fixed = true,
                     ConstraintKind::Tied(_) => {
                         // A tied operand must kill the used value.
-                        reguse.tied = !lr.killed_at(inst, ebb, ctx);
+                        reguse.tied =
+                            !lr.killed_at(inst, ebb, self.liveness.forest(), &self.cur.func.layout);
                     }
                     ConstraintKind::FixedTied(_) => {
                         reguse.fixed = true;
-                        reguse.tied = !lr.killed_at(inst, ebb, ctx);
+                        reguse.tied =
+                            !lr.killed_at(inst, ebb, self.liveness.forest(), &self.cur.func.layout);
                     }
                     ConstraintKind::Reg => {}
                 }
diff --git a/cranelift/codegen/src/value_label.rs b/cranelift/codegen/src/value_label.rs
index 2bd3bdc13d..d3d1e0672c 100644
--- a/cranelift/codegen/src/value_label.rs
+++ b/cranelift/codegen/src/value_label.rs
@@ -96,8 +96,8 @@ where
     ebbs.sort_by_key(|ebb| func.offsets[*ebb]); // Ensure inst offsets always increase
     let encinfo = isa.encoding_info();
     let values_locations = &func.locations;
-    let liveness_context = regalloc.liveness().context(&func.layout);
     let liveness_ranges = regalloc.liveness().ranges();
+    let liveness_forest = regalloc.liveness().forest();
 
     let mut ranges = HashMap::new();
     let mut add_range = |label, range: (u32, u32), loc: ValueLoc| {
@@ -126,7 +126,10 @@ where
             // Remove killed values.
             tracked_values.retain(|(x, label, start_offset, last_loc)| {
                 let range = liveness_ranges.get(*x);
-                if range.expect("value").killed_at(inst, ebb, liveness_context) {
+                if range
+                    .expect("value")
+                    .killed_at(inst, ebb, &liveness_forest, &func.layout)
+                {
                     add_range(*label, (*start_offset, end_offset), *last_loc);
                     return false;
                 }
@@ -173,7 +176,7 @@ where
                 // Ignore dead/inactive Values.
                 let range = liveness_ranges.get(*v);
                 match range {
-                    Some(r) => r.reaches_use(inst, ebb, liveness_context),
+                    Some(r) => r.reaches_use(inst, ebb, &liveness_forest, &func.layout),
                     None => false,
                 }
             });
diff --git a/cranelift/codegen/src/verifier/cssa.rs b/cranelift/codegen/src/verifier/cssa.rs
index 6014ff8030..3878f096d1 100644
--- a/cranelift/codegen/src/verifier/cssa.rs
+++ b/cranelift/codegen/src/verifier/cssa.rs
@@ -118,8 +118,12 @@ impl<'a> CssaVerifier<'a> {
                     if self.preorder.dominates(prev_ebb, def_ebb)
                         && self.domtree.dominates(prev_def, def, &self.func.layout)
                     {
-                        let ctx = self.liveness.context(&self.func.layout);
-                        if self.liveness[prev_val].overlaps_def(def, def_ebb, ctx) {
+                        if self.liveness[prev_val].overlaps_def(
+                            def,
+                            def_ebb,
+                            self.liveness.forest(),
+                            &self.func.layout,
+                        ) {
                             return fatal!(
                                 errors,
                                 val,
diff --git a/cranelift/codegen/src/verifier/liveness.rs b/cranelift/codegen/src/verifier/liveness.rs
index 11cde21e6f..37b59c154d 100644
--- a/cranelift/codegen/src/verifier/liveness.rs
+++ b/cranelift/codegen/src/verifier/liveness.rs
@@ -64,7 +64,6 @@ impl<'a> LivenessVerifier<'a> {
 
     /// Check all instructions.
     fn check_insts(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
-        let lr_ctx = self.liveness.context(&self.func.layout);
         for ebb in self.func.layout.ebbs() {
             for inst in self.func.layout.ebb_insts(ebb) {
                 let encoding = self.func.encodings[inst];
@@ -107,8 +106,8 @@ impl<'a> LivenessVerifier<'a> {
                         None => return fatal!(errors, inst, "{} has no live range", val),
                     };
 
-                    debug_assert!(lr_ctx.order.inst_ebb(inst).unwrap() == ebb);
-                    if !lr.reaches_use(inst, ebb, lr_ctx) {
+                    debug_assert!(self.func.layout.inst_ebb(inst).unwrap() == ebb);
+                    if !lr.reaches_use(inst, ebb, self.liveness.forest(), &self.func.layout) {
                         return fatal!(errors, inst, "{} is not live at this use", val);
                     }
 
@@ -180,7 +179,7 @@ impl<'a> LivenessVerifier<'a> {
         }
 
         // Now check the live-in intervals against the CFG.
-        for (mut ebb, end) in lr.liveins(self.liveness.context(l)) {
+        for (mut ebb, end) in lr.liveins(self.liveness.forest()) {
             if !l.is_ebb_inserted(ebb) {
                 return fatal!(
                     errors,
@@ -204,13 +203,11 @@ impl<'a> LivenessVerifier<'a> {
                 }
             };
 
-            let lr_ctx = self.liveness.context(&self.func.layout);
-
             // Check all the EBBs in the interval independently.
             loop {
                 // If `val` is live-in at `ebb`, it must be live at all the predecessors.
                 for BasicBlock { inst: pred, ebb } in self.cfg.pred_iter(ebb) {
-                    if !lr.reaches_use(pred, ebb, lr_ctx) {
+                    if !lr.reaches_use(pred, ebb, self.liveness.forest(), &self.func.layout) {
                         return fatal!(
                             errors,
                             pred,
diff --git a/cranelift/codegen/src/verifier/locations.rs b/cranelift/codegen/src/verifier/locations.rs
index cf17ae13de..d32bb7e4bd 100644
--- a/cranelift/codegen/src/verifier/locations.rs
+++ b/cranelift/codegen/src/verifier/locations.rs
@@ -334,10 +334,10 @@ impl<'a> LocationVerifier<'a> {
                     let lr = &liveness[value];
                     if is_after_branch && unique_predecessor {
                         // Forward diversions based on the targeted branch.
-                        if !lr.is_livein(ebb, liveness.context(&self.func.layout)) {
+                        if !lr.is_livein(ebb, liveness.forest(), &self.func.layout) {
                             val_to_remove.push(value)
                         }
-                    } else if lr.is_livein(ebb, liveness.context(&self.func.layout)) {
+                    } else if lr.is_livein(ebb, liveness.forest(), &self.func.layout) {
                         return fatal!(
                             errors,
                             inst,
@@ -359,7 +359,7 @@ impl<'a> LocationVerifier<'a> {
                 for (&value, d) in divert.iter() {
                     let lr = &liveness[value];
                     if let Some(ebb) = ebb {
-                        if lr.is_livein(ebb, liveness.context(&self.func.layout)) {
+                        if lr.is_livein(ebb, liveness.forest(), &self.func.layout) {
                             return fatal!(
                                 errors,
                                 inst,
@@ -371,7 +371,7 @@ impl<'a> LocationVerifier<'a> {
                         }
                     }
                     for ebb in self.func.jump_tables[jt].iter() {
-                        if lr.is_livein(*ebb, liveness.context(&self.func.layout)) {
+                        if lr.is_livein(*ebb, liveness.forest(), &self.func.layout) {
                             return fatal!(
                                 errors,
                                 inst,

From 46ab1b4103d21d78ac2240c7e5b20fc07ba91b4a Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 13 Sep 2019 18:54:38 +0200
Subject: [PATCH 2767/3084] Liverange: use a macro to make Order comparisons
 simpler to read;

---
 cranelift/codegen/src/regalloc/liverange.rs | 42 +++++++++++++--------
 1 file changed, 26 insertions(+), 16 deletions(-)

diff --git a/cranelift/codegen/src/regalloc/liverange.rs b/cranelift/codegen/src/regalloc/liverange.rs
index 6465b9ab8e..fb8fcc5fff 100644
--- a/cranelift/codegen/src/regalloc/liverange.rs
+++ b/cranelift/codegen/src/regalloc/liverange.rs
@@ -190,6 +190,22 @@ impl<'a, PO: ProgramOrder> bforest::Comparator for Cmp<'a, PO> {
     }
 }
 
+/// A simple helper macro to make comparisons more natural to read.
+macro_rules! cmp {
+    ($order:ident, $a:ident > $b:expr) => {
+        $order.cmp($a, $b) == Ordering::Greater
+    };
+    ($order:ident, $a:ident >= $b:expr) => {
+        $order.cmp($a, $b) != Ordering::Less
+    };
+    ($order:ident, $a:ident < $b:expr) => {
+        $order.cmp($a, $b) == Ordering::Less
+    };
+    ($order:ident, $a:ident <= $b:expr) => {
+        $order.cmp($a, $b) != Ordering::Greater
+    };
+}
+
 impl GenericLiveRange {
     /// Create a new live range for `value` defined at `def`.
     ///
@@ -227,15 +243,13 @@ impl GenericLiveRange {
         //
         // We're assuming here that `to` never precedes `def_begin` in the same EBB, but we can't
         // check it without a method for getting `to`'s EBB.
-        if order.cmp(ebb, self.def_end) != Ordering::Greater
-            && order.cmp(to, self.def_begin) != Ordering::Less
-        {
+        if cmp!(order, ebb <= self.def_end) && cmp!(order, to >= self.def_begin) {
             let to_pp = to.into();
             debug_assert_ne!(
                 to_pp, self.def_begin,
                 "Can't use value in the defining instruction."
             );
-            if order.cmp(to, self.def_end) == Ordering::Greater {
+            if cmp!(order, to > self.def_end) {
                 self.def_end = to_pp;
             }
             return false;
@@ -249,7 +263,7 @@ impl GenericLiveRange {
         if let Some(end) = c.goto(ebb) {
             // There's an interval beginning at `ebb`. See if it extends.
             first_time_livein = false;
-            if order.cmp(end, to) == Ordering::Less {
+            if cmp!(order, end < to) {
                 *c.value_mut().unwrap() = to;
             } else {
                 return first_time_livein;
@@ -257,10 +271,10 @@ impl GenericLiveRange {
         } else if let Some((_, end)) = c.prev() {
             // There's no interval beginning at `ebb`, but we could still be live-in at `ebb` with
             // a coalesced interval that begins before and ends after.
-            if order.cmp(end, ebb) == Ordering::Greater {
+            if cmp!(order, end > ebb) {
                 // Yep, the previous interval overlaps `ebb`.
                 first_time_livein = false;
-                if order.cmp(end, to) == Ordering::Less {
+                if cmp!(order, end < to) {
                     *c.value_mut().unwrap() = to;
                 } else {
                     return first_time_livein;
@@ -351,7 +365,7 @@ impl GenericLiveRange {
             .get_or_less(ebb, forest, &cmp)
             .and_then(|(_, inst)| {
                 // We have an entry that ends at `inst`.
-                if order.cmp(inst, ebb) == Ordering::Greater {
+                if cmp!(order, inst > ebb) {
                     Some(inst)
                 } else {
                     None
@@ -388,15 +402,13 @@ impl GenericLiveRange {
         }
 
         // Check for an overlap with the local range.
-        if order.cmp(def, self.def_begin) != Ordering::Less
-            && order.cmp(def, self.def_end) == Ordering::Less
-        {
+        if cmp!(order, def >= self.def_begin) && cmp!(order, def < self.def_end) {
             return true;
         }
 
         // Check for an overlap with a live-in range.
         match self.livein_local_end(ebb, forest, order) {
-            Some(inst) => order.cmp(def, inst) == Ordering::Less,
+            Some(inst) => cmp!(order, def < inst),
             None => false,
         }
     }
@@ -404,15 +416,13 @@ impl GenericLiveRange {
     /// Check if this live range reaches a use at `user` in `ebb`.
     pub fn reaches_use(&self, user: Inst, ebb: Ebb, forest: &LiveRangeForest, order: &PO) -> bool {
         // Check for an overlap with the local range.
-        if order.cmp(user, self.def_begin) == Ordering::Greater
-            && order.cmp(user, self.def_end) != Ordering::Greater
-        {
+        if cmp!(order, user > self.def_begin) && cmp!(order, user <= self.def_end) {
             return true;
         }
 
         // Check for an overlap with a live-in range.
         match self.livein_local_end(ebb, forest, order) {
-            Some(inst) => order.cmp(user, inst) != Ordering::Greater,
+            Some(inst) => cmp!(order, user <= inst),
             None => false,
         }
     }

From 8c3072c774de3a2c419867eccc355011c83b7f47 Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Wed, 25 Sep 2019 12:07:54 -0700
Subject: [PATCH 2768/3084] Combine `VisibleTranslationState` and
 `TranslationState` (#1076)

`VisibleTranslationState` was a wrapper around a `TranslationState` that was
meant to public API consumers outside of this crate. However, the internal
`TranslationState` and all its methods were still publicly exposed! This commit
simplifies and remedies the situation by combining them into a single
`TranslationState` type. Most of its methods are only `pub(crate)` now, not
visible to the entire world. The only methods that are `pub` are the ones that
`VisibleTranslationState` exposed.
---
 cranelift/wasm/src/environ/spec.rs    |  6 +--
 cranelift/wasm/src/func_translator.rs |  6 +--
 cranelift/wasm/src/lib.rs             |  2 +-
 cranelift/wasm/src/state.rs           | 78 ++++++++++++---------------
 4 files changed, 42 insertions(+), 50 deletions(-)

diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index bce06e4cd0..e83e675e72 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -6,7 +6,7 @@
 //!
 //! [Wasmtime]: https://github.com/CraneStation/wasmtime
 
-use crate::state::VisibleTranslationState;
+use crate::state::TranslationState;
 use crate::translation_utils::{
     FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
 };
@@ -281,7 +281,7 @@ pub trait FuncEnvironment {
         &mut self,
         _op: &Operator,
         _builder: &mut FunctionBuilder,
-        _state: &VisibleTranslationState,
+        _state: &TranslationState,
     ) -> WasmResult<()> {
         Ok(())
     }
@@ -292,7 +292,7 @@ pub trait FuncEnvironment {
         &mut self,
         _op: &Operator,
         _builder: &mut FunctionBuilder,
-        _state: &VisibleTranslationState,
+        _state: &TranslationState,
     ) -> WasmResult<()> {
         Ok(())
     }
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index ef1eecfa57..f947ddd065 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -6,7 +6,7 @@
 
 use crate::code_translator::translate_operator;
 use crate::environ::{FuncEnvironment, ReturnMode, WasmResult};
-use crate::state::{TranslationState, VisibleTranslationState};
+use crate::state::TranslationState;
 use crate::translation_utils::get_vmctx_value_label;
 use crate::wasm_unsupported;
 use cranelift_codegen::entity::EntityRef;
@@ -215,9 +215,9 @@ fn parse_function_body(
     while !state.control_stack.is_empty() {
         builder.set_srcloc(cur_srcloc(&reader));
         let op = reader.read_operator()?;
-        environ.before_translate_operator(&op, builder, &VisibleTranslationState::new(state))?;
+        environ.before_translate_operator(&op, builder, state)?;
         translate_operator(&op, builder, state, environ)?;
-        environ.after_translate_operator(&op, builder, &VisibleTranslationState::new(state))?;
+        environ.after_translate_operator(&op, builder, state)?;
     }
 
     // The final `End` operator left us in the exit block where we need to manually add a return
diff --git a/cranelift/wasm/src/lib.rs b/cranelift/wasm/src/lib.rs
index 3c1065228b..d47feb631d 100644
--- a/cranelift/wasm/src/lib.rs
+++ b/cranelift/wasm/src/lib.rs
@@ -62,7 +62,7 @@ pub use crate::environ::{
 };
 pub use crate::func_translator::FuncTranslator;
 pub use crate::module_translator::translate_module;
-pub use crate::state::VisibleTranslationState;
+pub use crate::state::TranslationState;
 pub use crate::translation_utils::{
     get_vmctx_value_label, DefinedFuncIndex, DefinedGlobalIndex, DefinedMemoryIndex,
     DefinedTableIndex, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex,
diff --git a/cranelift/wasm/src/state.rs b/cranelift/wasm/src/state.rs
index 7016035b72..7359482893 100644
--- a/cranelift/wasm/src/state.rs
+++ b/cranelift/wasm/src/state.rs
@@ -124,28 +124,6 @@ impl ControlStackFrame {
     }
 }
 
-/// VisibleTranslationState wraps a TranslationState with an interface appropriate for users
-/// outside this `cranelift-wasm`.
-///
-/// VisibleTranslationState is currently very minimal (only exposing reachability information), but
-/// is anticipated to grow in the future, with functions to inspect or modify the wasm operand
-/// stack for example.
-pub struct VisibleTranslationState<'a> {
-    state: &'a TranslationState,
-}
-
-impl<'a> VisibleTranslationState<'a> {
-    /// Build a VisibleTranslationState from an existing TranslationState
-    pub fn new(state: &'a TranslationState) -> Self {
-        VisibleTranslationState { state }
-    }
-
-    /// True if the current translation state expresses reachable code, false if it is unreachable
-    pub fn reachable(&self) -> bool {
-        self.state.reachable
-    }
-}
-
 /// Contains information passed along during the translation and that records:
 ///
 /// - The current value and control stacks.
@@ -154,12 +132,12 @@ impl<'a> VisibleTranslationState<'a> {
 pub struct TranslationState {
     /// A stack of values corresponding to the active values in the input wasm function at this
     /// point.
-    pub stack: Vec,
+    pub(crate) stack: Vec,
     /// A stack of active control flow operations at this point in the input wasm function.
-    pub control_stack: Vec,
+    pub(crate) control_stack: Vec,
     /// Is the current translation state still reachable? This is false when translating operators
     /// like End, Return, or Unreachable.
-    pub reachable: bool,
+    pub(crate) reachable: bool,
 
     // Map of global variables that have already been created by `FuncEnvironment::make_global`.
     globals: HashMap,
@@ -181,9 +159,18 @@ pub struct TranslationState {
     functions: HashMap,
 }
 
+// Public methods that are exposed to non-`cranelift_wasm` API consumers.
+impl TranslationState {
+    /// True if the current translation state expresses reachable code, false if it is unreachable.
+    #[inline]
+    pub fn reachable(&self) -> bool {
+        self.reachable
+    }
+}
+
 impl TranslationState {
     /// Construct a new, empty, `TranslationState`
-    pub fn new() -> Self {
+    pub(crate) fn new() -> Self {
         Self {
             stack: Vec::new(),
             control_stack: Vec::new(),
@@ -211,7 +198,7 @@ impl TranslationState {
     ///
     /// This resets the state to containing only a single block representing the whole function.
     /// The exit block is the last block in the function which will contain the return instruction.
-    pub fn initialize(&mut self, sig: &ir::Signature, exit_block: Ebb) {
+    pub(crate) fn initialize(&mut self, sig: &ir::Signature, exit_block: Ebb) {
         self.clear();
         self.push_block(
             exit_block,
@@ -223,34 +210,34 @@ impl TranslationState {
     }
 
     /// Push a value.
-    pub fn push1(&mut self, val: Value) {
+    pub(crate) fn push1(&mut self, val: Value) {
         self.stack.push(val);
     }
 
     /// Push multiple values.
-    pub fn pushn(&mut self, vals: &[Value]) {
+    pub(crate) fn pushn(&mut self, vals: &[Value]) {
         self.stack.extend_from_slice(vals);
     }
 
     /// Pop one value.
-    pub fn pop1(&mut self) -> Value {
+    pub(crate) fn pop1(&mut self) -> Value {
         self.stack.pop().unwrap()
     }
 
     /// Peek at the top of the stack without popping it.
-    pub fn peek1(&self) -> Value {
+    pub(crate) fn peek1(&self) -> Value {
         *self.stack.last().unwrap()
     }
 
     /// Pop two values. Return them in the order they were pushed.
-    pub fn pop2(&mut self) -> (Value, Value) {
+    pub(crate) fn pop2(&mut self) -> (Value, Value) {
         let v2 = self.stack.pop().unwrap();
         let v1 = self.stack.pop().unwrap();
         (v1, v2)
     }
 
     /// Pop three values. Return them in the order they were pushed.
-    pub fn pop3(&mut self) -> (Value, Value, Value) {
+    pub(crate) fn pop3(&mut self) -> (Value, Value, Value) {
         let v3 = self.stack.pop().unwrap();
         let v2 = self.stack.pop().unwrap();
         let v1 = self.stack.pop().unwrap();
@@ -260,18 +247,18 @@ impl TranslationState {
     /// Pop the top `n` values on the stack.
     ///
     /// The popped values are not returned. Use `peekn` to look at them before popping.
-    pub fn popn(&mut self, n: usize) {
+    pub(crate) fn popn(&mut self, n: usize) {
         let new_len = self.stack.len() - n;
         self.stack.truncate(new_len);
     }
 
     /// Peek at the top `n` values on the stack in the order they were pushed.
-    pub fn peekn(&self, n: usize) -> &[Value] {
+    pub(crate) fn peekn(&self, n: usize) -> &[Value] {
         &self.stack[self.stack.len() - n..]
     }
 
     /// Push a block on the control stack.
-    pub fn push_block(&mut self, following_code: Ebb, num_result_types: usize) {
+    pub(crate) fn push_block(&mut self, following_code: Ebb, num_result_types: usize) {
         self.control_stack.push(ControlStackFrame::Block {
             destination: following_code,
             original_stack_size: self.stack.len(),
@@ -281,7 +268,7 @@ impl TranslationState {
     }
 
     /// Push a loop on the control stack.
-    pub fn push_loop(&mut self, header: Ebb, following_code: Ebb, num_result_types: usize) {
+    pub(crate) fn push_loop(&mut self, header: Ebb, following_code: Ebb, num_result_types: usize) {
         self.control_stack.push(ControlStackFrame::Loop {
             header,
             destination: following_code,
@@ -291,7 +278,12 @@ impl TranslationState {
     }
 
     /// Push an if on the control stack.
-    pub fn push_if(&mut self, branch_inst: Inst, following_code: Ebb, num_result_types: usize) {
+    pub(crate) fn push_if(
+        &mut self,
+        branch_inst: Inst,
+        following_code: Ebb,
+        num_result_types: usize,
+    ) {
         self.control_stack.push(ControlStackFrame::If {
             branch_inst,
             destination: following_code,
@@ -308,7 +300,7 @@ impl TranslationState {
     /// Get the `GlobalVariable` reference that should be used to access the global variable
     /// `index`. Create the reference if necessary.
     /// Also return the WebAssembly type of the global.
-    pub fn get_global(
+    pub(crate) fn get_global(
         &mut self,
         func: &mut ir::Function,
         index: u32,
@@ -323,7 +315,7 @@ impl TranslationState {
 
     /// Get the `Heap` reference that should be used to access linear memory `index`.
     /// Create the reference if necessary.
-    pub fn get_heap(
+    pub(crate) fn get_heap(
         &mut self,
         func: &mut ir::Function,
         index: u32,
@@ -338,7 +330,7 @@ impl TranslationState {
 
     /// Get the `Table` reference that should be used to access table `index`.
     /// Create the reference if necessary.
-    pub fn get_table(
+    pub(crate) fn get_table(
         &mut self,
         func: &mut ir::Function,
         index: u32,
@@ -355,7 +347,7 @@ impl TranslationState {
     /// `index`. Also return the number of WebAssembly arguments in the signature.
     ///
     /// Create the signature if necessary.
-    pub fn get_indirect_sig(
+    pub(crate) fn get_indirect_sig(
         &mut self,
         func: &mut ir::Function,
         index: u32,
@@ -375,7 +367,7 @@ impl TranslationState {
     /// `index`. Also return the number of WebAssembly arguments in the signature.
     ///
     /// Create the function reference if necessary.
-    pub fn get_direct_func(
+    pub(crate) fn get_direct_func(
         &mut self,
         func: &mut ir::Function,
         index: u32,

From 3d5346a90b8dd273001f4d178118508891c86441 Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Wed, 25 Sep 2019 19:59:49 -0600
Subject: [PATCH 2769/3084] Name opcodes statically in isa/x86. Closes #1051
 (#1079)

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 895 ++++++++----------
 cranelift/codegen/meta/src/isa/x86/mod.rs     |   1 +
 cranelift/codegen/meta/src/isa/x86/opcodes.rs | 377 ++++++++
 cranelift/codegen/meta/src/isa/x86/recipes.rs |   8 +-
 4 files changed, 770 insertions(+), 511 deletions(-)
 create mode 100644 cranelift/codegen/meta/src/isa/x86/opcodes.rs

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index fc385d512a..01f25d9d7f 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -17,6 +17,8 @@ use crate::shared::types::Int::{I16, I32, I64, I8};
 use crate::shared::types::Reference::{R32, R64};
 use crate::shared::Definitions as SharedDefinitions;
 
+use crate::isa::x86::opcodes::*;
+
 use super::recipes::{RecipeGroup, Template};
 
 pub(crate) struct PerCpuModeEncodings {
@@ -681,91 +683,91 @@ pub(crate) fn define(
     e.enc64_rec(get_pinned_reg.bind(I64), rec_get_pinned_reg, 0);
     e.enc_x86_64(
         set_pinned_reg.bind(I64),
-        rec_set_pinned_reg.opcodes(vec![0x89]).rex().w(),
+        rec_set_pinned_reg.opcodes(&MOV_STORE).rex().w(),
     );
 
-    e.enc_i32_i64(iadd, rec_rr.opcodes(vec![0x01]));
-    e.enc_i32_i64(iadd_ifcout, rec_rout.opcodes(vec![0x01]));
-    e.enc_i32_i64(iadd_ifcin, rec_rin.opcodes(vec![0x11]));
-    e.enc_i32_i64(iadd_ifcarry, rec_rio.opcodes(vec![0x11]));
+    e.enc_i32_i64(iadd, rec_rr.opcodes(&ADD));
+    e.enc_i32_i64(iadd_ifcout, rec_rout.opcodes(&ADD));
+    e.enc_i32_i64(iadd_ifcin, rec_rin.opcodes(&ADC));
+    e.enc_i32_i64(iadd_ifcarry, rec_rio.opcodes(&ADC));
 
-    e.enc_i32_i64(isub, rec_rr.opcodes(vec![0x29]));
-    e.enc_i32_i64(isub_ifbout, rec_rout.opcodes(vec![0x29]));
-    e.enc_i32_i64(isub_ifbin, rec_rin.opcodes(vec![0x19]));
-    e.enc_i32_i64(isub_ifborrow, rec_rio.opcodes(vec![0x19]));
+    e.enc_i32_i64(isub, rec_rr.opcodes(&SUB));
+    e.enc_i32_i64(isub_ifbout, rec_rout.opcodes(&SUB));
+    e.enc_i32_i64(isub_ifbin, rec_rin.opcodes(&SBB));
+    e.enc_i32_i64(isub_ifborrow, rec_rio.opcodes(&SBB));
 
-    e.enc_i32_i64(band, rec_rr.opcodes(vec![0x21]));
-    e.enc_b32_b64(band, rec_rr.opcodes(vec![0x21]));
-    e.enc_i32_i64(bor, rec_rr.opcodes(vec![0x09]));
-    e.enc_b32_b64(bor, rec_rr.opcodes(vec![0x09]));
-    e.enc_i32_i64(bxor, rec_rr.opcodes(vec![0x31]));
-    e.enc_b32_b64(bxor, rec_rr.opcodes(vec![0x31]));
+    e.enc_i32_i64(band, rec_rr.opcodes(&AND));
+    e.enc_b32_b64(band, rec_rr.opcodes(&AND));
+    e.enc_i32_i64(bor, rec_rr.opcodes(&OR));
+    e.enc_b32_b64(bor, rec_rr.opcodes(&OR));
+    e.enc_i32_i64(bxor, rec_rr.opcodes(&XOR));
+    e.enc_b32_b64(bxor, rec_rr.opcodes(&XOR));
 
     // x86 has a bitwise not instruction NOT.
-    e.enc_i32_i64(bnot, rec_ur.opcodes(vec![0xf7]).rrr(2));
-    e.enc_b32_b64(bnot, rec_ur.opcodes(vec![0xf7]).rrr(2));
+    e.enc_i32_i64(bnot, rec_ur.opcodes(&NOT).rrr(2));
+    e.enc_b32_b64(bnot, rec_ur.opcodes(&NOT).rrr(2));
 
     // Also add a `b1` encodings for the logic instructions.
     // TODO: Should this be done with 8-bit instructions? It would improve partial register
     // dependencies.
-    e.enc_both(band.bind(B1), rec_rr.opcodes(vec![0x21]));
-    e.enc_both(bor.bind(B1), rec_rr.opcodes(vec![0x09]));
-    e.enc_both(bxor.bind(B1), rec_rr.opcodes(vec![0x31]));
+    e.enc_both(band.bind(B1), rec_rr.opcodes(&AND));
+    e.enc_both(bor.bind(B1), rec_rr.opcodes(&OR));
+    e.enc_both(bxor.bind(B1), rec_rr.opcodes(&XOR));
 
-    e.enc_i32_i64(imul, rec_rrx.opcodes(vec![0x0f, 0xaf]));
-    e.enc_i32_i64(x86_sdivmodx, rec_div.opcodes(vec![0xf7]).rrr(7));
-    e.enc_i32_i64(x86_udivmodx, rec_div.opcodes(vec![0xf7]).rrr(6));
+    e.enc_i32_i64(imul, rec_rrx.opcodes(&IMUL));
+    e.enc_i32_i64(x86_sdivmodx, rec_div.opcodes(&IDIV).rrr(7));
+    e.enc_i32_i64(x86_udivmodx, rec_div.opcodes(&DIV).rrr(6));
 
-    e.enc_i32_i64(x86_smulx, rec_mulx.opcodes(vec![0xf7]).rrr(5));
-    e.enc_i32_i64(x86_umulx, rec_mulx.opcodes(vec![0xf7]).rrr(4));
+    e.enc_i32_i64(x86_smulx, rec_mulx.opcodes(&IMUL_RDX_RAX).rrr(5));
+    e.enc_i32_i64(x86_umulx, rec_mulx.opcodes(&MUL).rrr(4));
 
-    e.enc_i32_i64(copy, rec_umr.opcodes(vec![0x89]));
-    e.enc_r32_r64_rex_only(copy, rec_umr.opcodes(vec![0x89]));
-    e.enc_both(copy.bind(B1), rec_umr.opcodes(vec![0x89]));
-    e.enc_both(copy.bind(I8), rec_umr.opcodes(vec![0x89]));
-    e.enc_both(copy.bind(I16), rec_umr.opcodes(vec![0x89]));
+    e.enc_i32_i64(copy, rec_umr.opcodes(&MOV_STORE));
+    e.enc_r32_r64_rex_only(copy, rec_umr.opcodes(&MOV_STORE));
+    e.enc_both(copy.bind(B1), rec_umr.opcodes(&MOV_STORE));
+    e.enc_both(copy.bind(I8), rec_umr.opcodes(&MOV_STORE));
+    e.enc_both(copy.bind(I16), rec_umr.opcodes(&MOV_STORE));
 
     // TODO For x86-64, only define REX forms for now, since we can't describe the
     // special regunit immediate operands with the current constraint language.
     for &ty in &[I8, I16, I32] {
-        e.enc32(regmove.bind(ty), rec_rmov.opcodes(vec![0x89]));
-        e.enc64(regmove.bind(ty), rec_rmov.opcodes(vec![0x89]).rex());
+        e.enc32(regmove.bind(ty), rec_rmov.opcodes(&MOV_STORE));
+        e.enc64(regmove.bind(ty), rec_rmov.opcodes(&MOV_STORE).rex());
     }
     for &ty in &[B8, B16, B32] {
-        e.enc32(regmove.bind(ty), rec_rmov.opcodes(vec![0x89]));
-        e.enc64(regmove.bind(ty), rec_rmov.opcodes(vec![0x89]).rex());
+        e.enc32(regmove.bind(ty), rec_rmov.opcodes(&MOV_STORE));
+        e.enc64(regmove.bind(ty), rec_rmov.opcodes(&MOV_STORE).rex());
     }
-    e.enc64(regmove.bind(I64), rec_rmov.opcodes(vec![0x89]).rex().w());
-    e.enc64(regmove.bind(B64), rec_rmov.opcodes(vec![0x89]).rex().w());
-    e.enc_both(regmove.bind(B1), rec_rmov.opcodes(vec![0x89]));
-    e.enc_both(regmove.bind(I8), rec_rmov.opcodes(vec![0x89]));
-    e.enc32(regmove.bind_ref(R32), rec_rmov.opcodes(vec![0x89]));
-    e.enc64(regmove.bind_ref(R32), rec_rmov.opcodes(vec![0x89]).rex());
+    e.enc64(regmove.bind(I64), rec_rmov.opcodes(&MOV_STORE).rex().w());
+    e.enc64(regmove.bind(B64), rec_rmov.opcodes(&MOV_STORE).rex().w());
+    e.enc_both(regmove.bind(B1), rec_rmov.opcodes(&MOV_STORE));
+    e.enc_both(regmove.bind(I8), rec_rmov.opcodes(&MOV_STORE));
+    e.enc32(regmove.bind_ref(R32), rec_rmov.opcodes(&MOV_STORE));
+    e.enc64(regmove.bind_ref(R32), rec_rmov.opcodes(&MOV_STORE).rex());
     e.enc64(
         regmove.bind_ref(R64),
-        rec_rmov.opcodes(vec![0x89]).rex().w(),
+        rec_rmov.opcodes(&MOV_STORE).rex().w(),
     );
 
-    e.enc_i32_i64(iadd_imm, rec_r_ib.opcodes(vec![0x83]).rrr(0));
-    e.enc_i32_i64(iadd_imm, rec_r_id.opcodes(vec![0x81]).rrr(0));
+    e.enc_i32_i64(iadd_imm, rec_r_ib.opcodes(&ADD_IMM8_SIGN_EXTEND).rrr(0));
+    e.enc_i32_i64(iadd_imm, rec_r_id.opcodes(&ADD_IMM).rrr(0));
 
-    e.enc_i32_i64(band_imm, rec_r_ib.opcodes(vec![0x83]).rrr(4));
-    e.enc_i32_i64(band_imm, rec_r_id.opcodes(vec![0x81]).rrr(4));
+    e.enc_i32_i64(band_imm, rec_r_ib.opcodes(&AND_IMM8_SIGN_EXTEND).rrr(4));
+    e.enc_i32_i64(band_imm, rec_r_id.opcodes(&AND_IMM).rrr(4));
 
-    e.enc_i32_i64(bor_imm, rec_r_ib.opcodes(vec![0x83]).rrr(1));
-    e.enc_i32_i64(bor_imm, rec_r_id.opcodes(vec![0x81]).rrr(1));
+    e.enc_i32_i64(bor_imm, rec_r_ib.opcodes(&OR_IMM8_SIGN_EXTEND).rrr(1));
+    e.enc_i32_i64(bor_imm, rec_r_id.opcodes(&OR_IMM).rrr(1));
 
-    e.enc_i32_i64(bxor_imm, rec_r_ib.opcodes(vec![0x83]).rrr(6));
-    e.enc_i32_i64(bxor_imm, rec_r_id.opcodes(vec![0x81]).rrr(6));
+    e.enc_i32_i64(bxor_imm, rec_r_ib.opcodes(&XOR_IMM8_SIGN_EXTEND).rrr(6));
+    e.enc_i32_i64(bxor_imm, rec_r_id.opcodes(&XOR_IMM).rrr(6));
 
     // TODO: band_imm.i64 with an unsigned 32-bit immediate can be encoded as band_imm.i32. Can
     // even use the single-byte immediate for 0xffff_ffXX masks.
 
     // Immediate constants.
-    e.enc32(iconst.bind(I32), rec_pu_id.opcodes(vec![0xb8]));
+    e.enc32(iconst.bind(I32), rec_pu_id.opcodes(&MOV_IMM));
 
-    e.enc64(iconst.bind(I32), rec_pu_id.rex().opcodes(vec![0xb8]));
-    e.enc64(iconst.bind(I32), rec_pu_id.opcodes(vec![0xb8]));
+    e.enc64(iconst.bind(I32), rec_pu_id.rex().opcodes(&MOV_IMM));
+    e.enc64(iconst.bind(I32), rec_pu_id.opcodes(&MOV_IMM));
 
     // The 32-bit immediate movl also zero-extends to 64 bits.
     let f_unary_imm = formats.get(formats.by_name("UnaryImm"));
@@ -773,34 +775,32 @@ pub(crate) fn define(
 
     e.enc64_func(
         iconst.bind(I64),
-        rec_pu_id.opcodes(vec![0xb8]).rex(),
+        rec_pu_id.opcodes(&MOV_IMM).rex(),
         |encoding| encoding.inst_predicate(is_unsigned_int32.clone()),
     );
-    e.enc64_func(
-        iconst.bind(I64),
-        rec_pu_id.opcodes(vec![0xb8]),
-        |encoding| encoding.inst_predicate(is_unsigned_int32),
-    );
+    e.enc64_func(iconst.bind(I64), rec_pu_id.opcodes(&MOV_IMM), |encoding| {
+        encoding.inst_predicate(is_unsigned_int32)
+    });
 
     // Sign-extended 32-bit immediate.
     e.enc64(
         iconst.bind(I64),
-        rec_u_id.rex().opcodes(vec![0xc7]).rrr(0).w(),
+        rec_u_id.rex().opcodes(&MOV_IMM_SIGNEXTEND).rrr(0).w(),
     );
 
-    // Finally, the 0xb8 opcode takes an 8-byte immediate with a REX.W prefix.
-    e.enc64(iconst.bind(I64), rec_pu_iq.opcodes(vec![0xb8]).rex().w());
+    // Finally, the MOV_IMM opcode takes an 8-byte immediate with a REX.W prefix.
+    e.enc64(iconst.bind(I64), rec_pu_iq.opcodes(&MOV_IMM).rex().w());
 
     // Bool constants (uses MOV)
     for &ty in &[B1, B8, B16, B32] {
-        e.enc_both(bconst.bind(ty), rec_pu_id_bool.opcodes(vec![0xb8]));
+        e.enc_both(bconst.bind(ty), rec_pu_id_bool.opcodes(&MOV_IMM));
     }
-    e.enc64(bconst.bind(B64), rec_pu_id_bool.opcodes(vec![0xb8]).rex());
+    e.enc64(bconst.bind(B64), rec_pu_id_bool.opcodes(&MOV_IMM).rex());
 
     let is_zero_int = InstructionPredicate::new_is_zero_int(f_unary_imm, "imm");
     e.enc_both_instp(
         iconst.bind(I8),
-        rec_u_id_z.opcodes(vec![0x30]),
+        rec_u_id_z.opcodes(&XORB),
         is_zero_int.clone(),
     );
     // You may expect that i16 encodings would have an 0x66 prefix on the opcode to indicate that
@@ -812,19 +812,15 @@ pub(crate) fn define(
     // an appropriate i16 encoding available.
     e.enc_both_instp(
         iconst.bind(I16),
-        rec_u_id_z.opcodes(vec![0x31]),
+        rec_u_id_z.opcodes(&XOR),
         is_zero_int.clone(),
     );
     e.enc_both_instp(
         iconst.bind(I32),
-        rec_u_id_z.opcodes(vec![0x31]),
+        rec_u_id_z.opcodes(&XOR),
         is_zero_int.clone(),
     );
-    e.enc_x86_64_instp(
-        iconst.bind(I64),
-        rec_u_id_z.opcodes(vec![0x31]),
-        is_zero_int,
-    );
+    e.enc_x86_64_instp(iconst.bind(I64), rec_u_id_z.opcodes(&XOR), is_zero_int);
 
     // Shifts and rotates.
     // Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit
@@ -835,97 +831,49 @@ pub(crate) fn define(
         // to bind any.
         e.enc32(
             inst.bind(I32).bind_any(),
-            rec_rc.opcodes(vec![0xd3]).rrr(rrr),
+            rec_rc.opcodes(&ROTATE_CL).rrr(rrr),
         );
         e.enc64(
             inst.bind(I64).bind_any(),
-            rec_rc.opcodes(vec![0xd3]).rrr(rrr).rex().w(),
+            rec_rc.opcodes(&ROTATE_CL).rrr(rrr).rex().w(),
         );
         e.enc64(
             inst.bind(I32).bind_any(),
-            rec_rc.opcodes(vec![0xd3]).rrr(rrr).rex(),
+            rec_rc.opcodes(&ROTATE_CL).rrr(rrr).rex(),
         );
         e.enc64(
             inst.bind(I32).bind_any(),
-            rec_rc.opcodes(vec![0xd3]).rrr(rrr),
+            rec_rc.opcodes(&ROTATE_CL).rrr(rrr),
         );
     }
 
-    for &(inst, rrr) in &[
-        (rotl_imm, 0),
-        (rotr_imm, 1),
-        (ishl_imm, 4),
-        (ushr_imm, 5),
-        (sshr_imm, 7),
-    ] {
-        e.enc_i32_i64(inst, rec_r_ib.opcodes(vec![0xc1]).rrr(rrr));
-    }
+    e.enc_i32_i64(rotl_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(0));
+    e.enc_i32_i64(rotr_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(1));
+    e.enc_i32_i64(ishl_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(4));
+    e.enc_i32_i64(ushr_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(5));
+    e.enc_i32_i64(sshr_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(7));
 
     // Population count.
-    e.enc32_isap(
-        popcnt.bind(I32),
-        rec_urm.opcodes(vec![0xf3, 0x0f, 0xb8]),
-        use_popcnt,
-    );
+    e.enc32_isap(popcnt.bind(I32), rec_urm.opcodes(&POPCNT), use_popcnt);
     e.enc64_isap(
         popcnt.bind(I64),
-        rec_urm.opcodes(vec![0xf3, 0x0f, 0xb8]).rex().w(),
-        use_popcnt,
-    );
-    e.enc64_isap(
-        popcnt.bind(I32),
-        rec_urm.opcodes(vec![0xf3, 0x0f, 0xb8]).rex(),
-        use_popcnt,
-    );
-    e.enc64_isap(
-        popcnt.bind(I32),
-        rec_urm.opcodes(vec![0xf3, 0x0f, 0xb8]),
+        rec_urm.opcodes(&POPCNT).rex().w(),
         use_popcnt,
     );
+    e.enc64_isap(popcnt.bind(I32), rec_urm.opcodes(&POPCNT).rex(), use_popcnt);
+    e.enc64_isap(popcnt.bind(I32), rec_urm.opcodes(&POPCNT), use_popcnt);
 
     // Count leading zero bits.
-    e.enc32_isap(
-        clz.bind(I32),
-        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbd]),
-        use_lzcnt,
-    );
-    e.enc64_isap(
-        clz.bind(I64),
-        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbd]).rex().w(),
-        use_lzcnt,
-    );
-    e.enc64_isap(
-        clz.bind(I32),
-        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbd]).rex(),
-        use_lzcnt,
-    );
-    e.enc64_isap(
-        clz.bind(I32),
-        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbd]),
-        use_lzcnt,
-    );
+    e.enc32_isap(clz.bind(I32), rec_urm.opcodes(&LZCNT), use_lzcnt);
+    e.enc64_isap(clz.bind(I64), rec_urm.opcodes(&LZCNT).rex().w(), use_lzcnt);
+    e.enc64_isap(clz.bind(I32), rec_urm.opcodes(&LZCNT).rex(), use_lzcnt);
+    e.enc64_isap(clz.bind(I32), rec_urm.opcodes(&LZCNT), use_lzcnt);
 
     // Count trailing zero bits.
-    e.enc32_isap(
-        ctz.bind(I32),
-        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbc]),
-        use_bmi1,
-    );
-    e.enc64_isap(
-        ctz.bind(I64),
-        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbc]).rex().w(),
-        use_bmi1,
-    );
-    e.enc64_isap(
-        ctz.bind(I32),
-        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbc]).rex(),
-        use_bmi1,
-    );
-    e.enc64_isap(
-        ctz.bind(I32),
-        rec_urm.opcodes(vec![0xf3, 0x0f, 0xbc]),
-        use_bmi1,
-    );
+    e.enc32_isap(ctz.bind(I32), rec_urm.opcodes(&TZCNT), use_bmi1);
+    e.enc64_isap(ctz.bind(I64), rec_urm.opcodes(&TZCNT).rex().w(), use_bmi1);
+    e.enc64_isap(ctz.bind(I32), rec_urm.opcodes(&TZCNT).rex(), use_bmi1);
+    e.enc64_isap(ctz.bind(I32), rec_urm.opcodes(&TZCNT), use_bmi1);
 
     // Loads and stores.
     let f_load_complex = formats.get(formats.by_name("LoadComplex"));
@@ -934,41 +882,41 @@ pub(crate) fn define(
     for recipe in &[rec_ldWithIndex, rec_ldWithIndexDisp8, rec_ldWithIndexDisp32] {
         e.enc_i32_i64_instp(
             load_complex,
-            recipe.opcodes(vec![0x8b]),
+            recipe.opcodes(&MOV_LOAD),
             is_load_complex_length_two.clone(),
         );
         e.enc_x86_64_instp(
             uload32_complex,
-            recipe.opcodes(vec![0x8b]),
+            recipe.opcodes(&MOV_LOAD),
             is_load_complex_length_two.clone(),
         );
 
         e.enc64_instp(
             sload32_complex,
-            recipe.opcodes(vec![0x63]).rex().w(),
+            recipe.opcodes(&MOVSXD).rex().w(),
             is_load_complex_length_two.clone(),
         );
 
         e.enc_i32_i64_instp(
             uload16_complex,
-            recipe.opcodes(vec![0x0f, 0xb7]),
+            recipe.opcodes(&MOVZX_WORD),
             is_load_complex_length_two.clone(),
         );
         e.enc_i32_i64_instp(
             sload16_complex,
-            recipe.opcodes(vec![0x0f, 0xbf]),
+            recipe.opcodes(&MOVSX_WORD),
             is_load_complex_length_two.clone(),
         );
 
         e.enc_i32_i64_instp(
             uload8_complex,
-            recipe.opcodes(vec![0x0f, 0xb6]),
+            recipe.opcodes(&MOVZX_BYTE),
             is_load_complex_length_two.clone(),
         );
 
         e.enc_i32_i64_instp(
             sload8_complex,
-            recipe.opcodes(vec![0x0f, 0xbe]),
+            recipe.opcodes(&MOVSX_BYTE),
             is_load_complex_length_two.clone(),
         );
     }
@@ -979,22 +927,22 @@ pub(crate) fn define(
     for recipe in &[rec_stWithIndex, rec_stWithIndexDisp8, rec_stWithIndexDisp32] {
         e.enc_i32_i64_instp(
             store_complex,
-            recipe.opcodes(vec![0x89]),
+            recipe.opcodes(&MOV_STORE),
             is_store_complex_length_three.clone(),
         );
         e.enc_x86_64_instp(
             istore32_complex,
-            recipe.opcodes(vec![0x89]),
+            recipe.opcodes(&MOV_STORE),
             is_store_complex_length_three.clone(),
         );
         e.enc_both_instp(
             istore16_complex.bind(I32),
-            recipe.opcodes(vec![0x66, 0x89]),
+            recipe.opcodes(&MOV_STORE_16),
             is_store_complex_length_three.clone(),
         );
         e.enc_x86_64_instp(
             istore16_complex.bind(I64),
-            recipe.opcodes(vec![0x66, 0x89]),
+            recipe.opcodes(&MOV_STORE_16),
             is_store_complex_length_three.clone(),
         );
     }
@@ -1006,20 +954,20 @@ pub(crate) fn define(
     ] {
         e.enc_both_instp(
             istore8_complex.bind(I32),
-            recipe.opcodes(vec![0x88]),
+            recipe.opcodes(&MOV_BYTE_STORE),
             is_store_complex_length_three.clone(),
         );
         e.enc_x86_64_instp(
             istore8_complex.bind(I64),
-            recipe.opcodes(vec![0x88]),
+            recipe.opcodes(&MOV_BYTE_STORE),
             is_store_complex_length_three.clone(),
         );
     }
 
     for recipe in &[rec_st, rec_stDisp8, rec_stDisp32] {
-        e.enc_i32_i64_ld_st(store, true, recipe.opcodes(vec![0x89]));
-        e.enc_x86_64(istore32.bind(I64).bind_any(), recipe.opcodes(vec![0x89]));
-        e.enc_i32_i64_ld_st(istore16, false, recipe.opcodes(vec![0x66, 0x89]));
+        e.enc_i32_i64_ld_st(store, true, recipe.opcodes(&MOV_STORE));
+        e.enc_x86_64(istore32.bind(I64).bind_any(), recipe.opcodes(&MOV_STORE));
+        e.enc_i32_i64_ld_st(istore16, false, recipe.opcodes(&MOV_STORE_16));
     }
 
     // Byte stores are more complicated because the registers they can address
@@ -1027,40 +975,46 @@ pub(crate) fn define(
     // the corresponding st* recipes when a REX prefix is applied.
 
     for recipe in &[rec_st_abcd, rec_stDisp8_abcd, rec_stDisp32_abcd] {
-        e.enc_both(istore8.bind(I32).bind_any(), recipe.opcodes(vec![0x88]));
-        e.enc_x86_64(istore8.bind(I64).bind_any(), recipe.opcodes(vec![0x88]));
+        e.enc_both(
+            istore8.bind(I32).bind_any(),
+            recipe.opcodes(&MOV_BYTE_STORE),
+        );
+        e.enc_x86_64(
+            istore8.bind(I64).bind_any(),
+            recipe.opcodes(&MOV_BYTE_STORE),
+        );
     }
 
-    e.enc_i32_i64(spill, rec_spillSib32.opcodes(vec![0x89]));
-    e.enc_i32_i64(regspill, rec_regspill32.opcodes(vec![0x89]));
-    e.enc_r32_r64_rex_only(spill, rec_spillSib32.opcodes(vec![0x89]));
-    e.enc_r32_r64_rex_only(regspill, rec_regspill32.opcodes(vec![0x89]));
+    e.enc_i32_i64(spill, rec_spillSib32.opcodes(&MOV_STORE));
+    e.enc_i32_i64(regspill, rec_regspill32.opcodes(&MOV_STORE));
+    e.enc_r32_r64_rex_only(spill, rec_spillSib32.opcodes(&MOV_STORE));
+    e.enc_r32_r64_rex_only(regspill, rec_regspill32.opcodes(&MOV_STORE));
 
     // Use a 32-bit write for spilling `b1`, `i8` and `i16` to avoid
     // constraining the permitted registers.
     // See MIN_SPILL_SLOT_SIZE which makes this safe.
 
-    e.enc_both(spill.bind(B1), rec_spillSib32.opcodes(vec![0x89]));
-    e.enc_both(regspill.bind(B1), rec_regspill32.opcodes(vec![0x89]));
+    e.enc_both(spill.bind(B1), rec_spillSib32.opcodes(&MOV_STORE));
+    e.enc_both(regspill.bind(B1), rec_regspill32.opcodes(&MOV_STORE));
     for &ty in &[I8, I16] {
-        e.enc_both(spill.bind(ty), rec_spillSib32.opcodes(vec![0x89]));
-        e.enc_both(regspill.bind(ty), rec_regspill32.opcodes(vec![0x89]));
+        e.enc_both(spill.bind(ty), rec_spillSib32.opcodes(&MOV_STORE));
+        e.enc_both(regspill.bind(ty), rec_regspill32.opcodes(&MOV_STORE));
     }
 
     for recipe in &[rec_ld, rec_ldDisp8, rec_ldDisp32] {
-        e.enc_i32_i64_ld_st(load, true, recipe.opcodes(vec![0x8b]));
-        e.enc_x86_64(uload32.bind(I64), recipe.opcodes(vec![0x8b]));
-        e.enc64(sload32.bind(I64), recipe.opcodes(vec![0x63]).rex().w());
-        e.enc_i32_i64_ld_st(uload16, true, recipe.opcodes(vec![0x0f, 0xb7]));
-        e.enc_i32_i64_ld_st(sload16, true, recipe.opcodes(vec![0x0f, 0xbf]));
-        e.enc_i32_i64_ld_st(uload8, true, recipe.opcodes(vec![0x0f, 0xb6]));
-        e.enc_i32_i64_ld_st(sload8, true, recipe.opcodes(vec![0x0f, 0xbe]));
+        e.enc_i32_i64_ld_st(load, true, recipe.opcodes(&MOV_LOAD));
+        e.enc_x86_64(uload32.bind(I64), recipe.opcodes(&MOV_LOAD));
+        e.enc64(sload32.bind(I64), recipe.opcodes(&MOVSXD).rex().w());
+        e.enc_i32_i64_ld_st(uload16, true, recipe.opcodes(&MOVZX_WORD));
+        e.enc_i32_i64_ld_st(sload16, true, recipe.opcodes(&MOVSX_WORD));
+        e.enc_i32_i64_ld_st(uload8, true, recipe.opcodes(&MOVZX_BYTE));
+        e.enc_i32_i64_ld_st(sload8, true, recipe.opcodes(&MOVSX_BYTE));
     }
 
-    e.enc_i32_i64(fill, rec_fillSib32.opcodes(vec![0x8b]));
-    e.enc_i32_i64(regfill, rec_regfill32.opcodes(vec![0x8b]));
-    e.enc_r32_r64_rex_only(fill, rec_fillSib32.opcodes(vec![0x8b]));
-    e.enc_r32_r64_rex_only(regfill, rec_regfill32.opcodes(vec![0x8b]));
+    e.enc_i32_i64(fill, rec_fillSib32.opcodes(&MOV_LOAD));
+    e.enc_i32_i64(regfill, rec_regfill32.opcodes(&MOV_LOAD));
+    e.enc_r32_r64_rex_only(fill, rec_fillSib32.opcodes(&MOV_LOAD));
+    e.enc_r32_r64_rex_only(regfill, rec_regfill32.opcodes(&MOV_LOAD));
 
     // No-op fills, created by late-stage redundant-fill removal.
     for &ty in &[I64, I32, I16, I8] {
@@ -1076,44 +1030,44 @@ pub(crate) fn define(
 
     // Load 32 bits from `b1`, `i8` and `i16` spill slots. See `spill.b1` above.
 
-    e.enc_both(fill.bind(B1), rec_fillSib32.opcodes(vec![0x8b]));
-    e.enc_both(regfill.bind(B1), rec_regfill32.opcodes(vec![0x8b]));
+    e.enc_both(fill.bind(B1), rec_fillSib32.opcodes(&MOV_LOAD));
+    e.enc_both(regfill.bind(B1), rec_regfill32.opcodes(&MOV_LOAD));
     for &ty in &[I8, I16] {
-        e.enc_both(fill.bind(ty), rec_fillSib32.opcodes(vec![0x8b]));
-        e.enc_both(regfill.bind(ty), rec_regfill32.opcodes(vec![0x8b]));
+        e.enc_both(fill.bind(ty), rec_fillSib32.opcodes(&MOV_LOAD));
+        e.enc_both(regfill.bind(ty), rec_regfill32.opcodes(&MOV_LOAD));
     }
 
     // Push and Pop.
-    e.enc32(x86_push.bind(I32), rec_pushq.opcodes(vec![0x50]));
-    e.enc_x86_64(x86_push.bind(I64), rec_pushq.opcodes(vec![0x50]));
+    e.enc32(x86_push.bind(I32), rec_pushq.opcodes(&PUSH_REG));
+    e.enc_x86_64(x86_push.bind(I64), rec_pushq.opcodes(&PUSH_REG));
 
-    e.enc32(x86_pop.bind(I32), rec_popq.opcodes(vec![0x58]));
-    e.enc_x86_64(x86_pop.bind(I64), rec_popq.opcodes(vec![0x58]));
+    e.enc32(x86_pop.bind(I32), rec_popq.opcodes(&POP_REG));
+    e.enc_x86_64(x86_pop.bind(I64), rec_popq.opcodes(&POP_REG));
 
     // Copy Special
     // For x86-64, only define REX forms for now, since we can't describe the
     // special regunit immediate operands with the current constraint language.
-    e.enc64(copy_special, rec_copysp.opcodes(vec![0x89]).rex().w());
-    e.enc32(copy_special, rec_copysp.opcodes(vec![0x89]));
+    e.enc64(copy_special, rec_copysp.opcodes(&MOV_STORE).rex().w());
+    e.enc32(copy_special, rec_copysp.opcodes(&MOV_STORE));
 
     // Copy to SSA.  These have to be done with special _rex_only encoders, because the standard
     // machinery for deciding whether a REX.{RXB} prefix is needed doesn't take into account
     // the source register, which is specified directly in the instruction.
-    e.enc_i32_i64_rex_only(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(vec![0x89]));
-    e.enc_r32_r64_rex_only(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(vec![0x89]));
-    e.enc_both_rex_only(copy_to_ssa.bind(B1), rec_umr_reg_to_ssa.opcodes(vec![0x89]));
-    e.enc_both_rex_only(copy_to_ssa.bind(I8), rec_umr_reg_to_ssa.opcodes(vec![0x89]));
+    e.enc_i32_i64_rex_only(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(&MOV_STORE));
+    e.enc_r32_r64_rex_only(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(&MOV_STORE));
+    e.enc_both_rex_only(copy_to_ssa.bind(B1), rec_umr_reg_to_ssa.opcodes(&MOV_STORE));
+    e.enc_both_rex_only(copy_to_ssa.bind(I8), rec_umr_reg_to_ssa.opcodes(&MOV_STORE));
     e.enc_both_rex_only(
         copy_to_ssa.bind(I16),
-        rec_umr_reg_to_ssa.opcodes(vec![0x89]),
+        rec_umr_reg_to_ssa.opcodes(&MOV_STORE),
     );
     e.enc_both_rex_only(
         copy_to_ssa.bind(F64),
-        rec_furm_reg_to_ssa.opcodes(vec![0xf2, 0x0f, 0x10]),
+        rec_furm_reg_to_ssa.opcodes(&MOVSD_LOAD),
     );
     e.enc_both_rex_only(
         copy_to_ssa.bind(F32),
-        rec_furm_reg_to_ssa.opcodes(vec![0xf3, 0x0f, 0x10]),
+        rec_furm_reg_to_ssa.opcodes(&MOVSS_LOAD),
     );
 
     // Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
@@ -1129,204 +1083,159 @@ pub(crate) fn define(
     }
 
     // Adjust SP down by a dynamic value (or up, with a negative operand).
-    e.enc32(adjust_sp_down.bind(I32), rec_adjustsp.opcodes(vec![0x29]));
+    e.enc32(adjust_sp_down.bind(I32), rec_adjustsp.opcodes(&SUB));
     e.enc64(
         adjust_sp_down.bind(I64),
-        rec_adjustsp.opcodes(vec![0x29]).rex().w(),
+        rec_adjustsp.opcodes(&SUB).rex().w(),
     );
 
     // Adjust SP up by an immediate (or down, with a negative immediate).
-    e.enc32(adjust_sp_up_imm, rec_adjustsp_ib.opcodes(vec![0x83]));
-    e.enc32(adjust_sp_up_imm, rec_adjustsp_id.opcodes(vec![0x81]));
+    e.enc32(adjust_sp_up_imm, rec_adjustsp_ib.opcodes(&CMP_IMM8));
+    e.enc32(adjust_sp_up_imm, rec_adjustsp_id.opcodes(&CMP_IMM));
     e.enc64(
         adjust_sp_up_imm,
-        rec_adjustsp_ib.opcodes(vec![0x83]).rex().w(),
+        rec_adjustsp_ib.opcodes(&CMP_IMM8).rex().w(),
     );
     e.enc64(
         adjust_sp_up_imm,
-        rec_adjustsp_id.opcodes(vec![0x81]).rex().w(),
+        rec_adjustsp_id.opcodes(&CMP_IMM).rex().w(),
     );
 
     // Adjust SP down by an immediate (or up, with a negative immediate).
     e.enc32(
         adjust_sp_down_imm,
-        rec_adjustsp_ib.opcodes(vec![0x83]).rrr(5),
+        rec_adjustsp_ib.opcodes(&CMP_IMM8).rrr(5),
     );
-    e.enc32(
+    e.enc32(adjust_sp_down_imm, rec_adjustsp_id.opcodes(&CMP_IMM).rrr(5));
+    e.enc64(
         adjust_sp_down_imm,
-        rec_adjustsp_id.opcodes(vec![0x81]).rrr(5),
+        rec_adjustsp_ib.opcodes(&CMP_IMM8).rrr(5).rex().w(),
     );
     e.enc64(
         adjust_sp_down_imm,
-        rec_adjustsp_ib.opcodes(vec![0x83]).rrr(5).rex().w(),
-    );
-    e.enc64(
-        adjust_sp_down_imm,
-        rec_adjustsp_id.opcodes(vec![0x81]).rrr(5).rex().w(),
+        rec_adjustsp_id.opcodes(&CMP_IMM).rrr(5).rex().w(),
     );
 
     // Float loads and stores.
+    e.enc_both(load.bind(F32).bind_any(), rec_fld.opcodes(&MOVSS_LOAD));
+    e.enc_both(load.bind(F32).bind_any(), rec_fldDisp8.opcodes(&MOVSS_LOAD));
     e.enc_both(
         load.bind(F32).bind_any(),
-        rec_fld.opcodes(vec![0xf3, 0x0f, 0x10]),
-    );
-    e.enc_both(
-        load.bind(F32).bind_any(),
-        rec_fldDisp8.opcodes(vec![0xf3, 0x0f, 0x10]),
-    );
-    e.enc_both(
-        load.bind(F32).bind_any(),
-        rec_fldDisp32.opcodes(vec![0xf3, 0x0f, 0x10]),
+        rec_fldDisp32.opcodes(&MOVSS_LOAD),
     );
 
     e.enc_both(
         load_complex.bind(F32),
-        rec_fldWithIndex.opcodes(vec![0xf3, 0x0f, 0x10]),
+        rec_fldWithIndex.opcodes(&MOVSS_LOAD),
     );
     e.enc_both(
         load_complex.bind(F32),
-        rec_fldWithIndexDisp8.opcodes(vec![0xf3, 0x0f, 0x10]),
+        rec_fldWithIndexDisp8.opcodes(&MOVSS_LOAD),
     );
     e.enc_both(
         load_complex.bind(F32),
-        rec_fldWithIndexDisp32.opcodes(vec![0xf3, 0x0f, 0x10]),
+        rec_fldWithIndexDisp32.opcodes(&MOVSS_LOAD),
     );
 
+    e.enc_both(load.bind(F64).bind_any(), rec_fld.opcodes(&MOVSD_LOAD));
+    e.enc_both(load.bind(F64).bind_any(), rec_fldDisp8.opcodes(&MOVSD_LOAD));
     e.enc_both(
         load.bind(F64).bind_any(),
-        rec_fld.opcodes(vec![0xf2, 0x0f, 0x10]),
-    );
-    e.enc_both(
-        load.bind(F64).bind_any(),
-        rec_fldDisp8.opcodes(vec![0xf2, 0x0f, 0x10]),
-    );
-    e.enc_both(
-        load.bind(F64).bind_any(),
-        rec_fldDisp32.opcodes(vec![0xf2, 0x0f, 0x10]),
+        rec_fldDisp32.opcodes(&MOVSD_LOAD),
     );
 
     e.enc_both(
         load_complex.bind(F64),
-        rec_fldWithIndex.opcodes(vec![0xf2, 0x0f, 0x10]),
+        rec_fldWithIndex.opcodes(&MOVSD_LOAD),
     );
     e.enc_both(
         load_complex.bind(F64),
-        rec_fldWithIndexDisp8.opcodes(vec![0xf2, 0x0f, 0x10]),
+        rec_fldWithIndexDisp8.opcodes(&MOVSD_LOAD),
     );
     e.enc_both(
         load_complex.bind(F64),
-        rec_fldWithIndexDisp32.opcodes(vec![0xf2, 0x0f, 0x10]),
+        rec_fldWithIndexDisp32.opcodes(&MOVSD_LOAD),
     );
 
+    e.enc_both(store.bind(F32).bind_any(), rec_fst.opcodes(&MOVSS_STORE));
     e.enc_both(
         store.bind(F32).bind_any(),
-        rec_fst.opcodes(vec![0xf3, 0x0f, 0x11]),
+        rec_fstDisp8.opcodes(&MOVSS_STORE),
     );
     e.enc_both(
         store.bind(F32).bind_any(),
-        rec_fstDisp8.opcodes(vec![0xf3, 0x0f, 0x11]),
-    );
-    e.enc_both(
-        store.bind(F32).bind_any(),
-        rec_fstDisp32.opcodes(vec![0xf3, 0x0f, 0x11]),
+        rec_fstDisp32.opcodes(&MOVSS_STORE),
     );
 
     e.enc_both(
         store_complex.bind(F32),
-        rec_fstWithIndex.opcodes(vec![0xf3, 0x0f, 0x11]),
+        rec_fstWithIndex.opcodes(&MOVSS_STORE),
     );
     e.enc_both(
         store_complex.bind(F32),
-        rec_fstWithIndexDisp8.opcodes(vec![0xf3, 0x0f, 0x11]),
+        rec_fstWithIndexDisp8.opcodes(&MOVSS_STORE),
     );
     e.enc_both(
         store_complex.bind(F32),
-        rec_fstWithIndexDisp32.opcodes(vec![0xf3, 0x0f, 0x11]),
+        rec_fstWithIndexDisp32.opcodes(&MOVSS_STORE),
     );
 
+    e.enc_both(store.bind(F64).bind_any(), rec_fst.opcodes(&MOVSD_STORE));
     e.enc_both(
         store.bind(F64).bind_any(),
-        rec_fst.opcodes(vec![0xf2, 0x0f, 0x11]),
+        rec_fstDisp8.opcodes(&MOVSD_STORE),
     );
     e.enc_both(
         store.bind(F64).bind_any(),
-        rec_fstDisp8.opcodes(vec![0xf2, 0x0f, 0x11]),
-    );
-    e.enc_both(
-        store.bind(F64).bind_any(),
-        rec_fstDisp32.opcodes(vec![0xf2, 0x0f, 0x11]),
+        rec_fstDisp32.opcodes(&MOVSD_STORE),
     );
 
     e.enc_both(
         store_complex.bind(F64),
-        rec_fstWithIndex.opcodes(vec![0xf2, 0x0f, 0x11]),
+        rec_fstWithIndex.opcodes(&MOVSD_STORE),
     );
     e.enc_both(
         store_complex.bind(F64),
-        rec_fstWithIndexDisp8.opcodes(vec![0xf2, 0x0f, 0x11]),
+        rec_fstWithIndexDisp8.opcodes(&MOVSD_STORE),
     );
     e.enc_both(
         store_complex.bind(F64),
-        rec_fstWithIndexDisp32.opcodes(vec![0xf2, 0x0f, 0x11]),
+        rec_fstWithIndexDisp32.opcodes(&MOVSD_STORE),
     );
 
-    e.enc_both(
-        fill.bind(F32),
-        rec_ffillSib32.opcodes(vec![0xf3, 0x0f, 0x10]),
-    );
-    e.enc_both(
-        regfill.bind(F32),
-        rec_fregfill32.opcodes(vec![0xf3, 0x0f, 0x10]),
-    );
-    e.enc_both(
-        fill.bind(F64),
-        rec_ffillSib32.opcodes(vec![0xf2, 0x0f, 0x10]),
-    );
-    e.enc_both(
-        regfill.bind(F64),
-        rec_fregfill32.opcodes(vec![0xf2, 0x0f, 0x10]),
-    );
+    e.enc_both(fill.bind(F32), rec_ffillSib32.opcodes(&MOVSS_LOAD));
+    e.enc_both(regfill.bind(F32), rec_fregfill32.opcodes(&MOVSS_LOAD));
+    e.enc_both(fill.bind(F64), rec_ffillSib32.opcodes(&MOVSD_LOAD));
+    e.enc_both(regfill.bind(F64), rec_fregfill32.opcodes(&MOVSD_LOAD));
 
-    e.enc_both(
-        spill.bind(F32),
-        rec_fspillSib32.opcodes(vec![0xf3, 0x0f, 0x11]),
-    );
-    e.enc_both(
-        regspill.bind(F32),
-        rec_fregspill32.opcodes(vec![0xf3, 0x0f, 0x11]),
-    );
-    e.enc_both(
-        spill.bind(F64),
-        rec_fspillSib32.opcodes(vec![0xf2, 0x0f, 0x11]),
-    );
-    e.enc_both(
-        regspill.bind(F64),
-        rec_fregspill32.opcodes(vec![0xf2, 0x0f, 0x11]),
-    );
+    e.enc_both(spill.bind(F32), rec_fspillSib32.opcodes(&MOVSS_STORE));
+    e.enc_both(regspill.bind(F32), rec_fregspill32.opcodes(&MOVSS_STORE));
+    e.enc_both(spill.bind(F64), rec_fspillSib32.opcodes(&MOVSD_STORE));
+    e.enc_both(regspill.bind(F64), rec_fregspill32.opcodes(&MOVSD_STORE));
 
     // Function addresses.
 
     // Non-PIC, all-ones funcaddresses.
     e.enc32_isap(
         func_addr.bind(I32),
-        rec_fnaddr4.opcodes(vec![0xb8]),
+        rec_fnaddr4.opcodes(&MOV_IMM),
         not_all_ones_funcaddrs_and_not_is_pic,
     );
     e.enc64_isap(
         func_addr.bind(I64),
-        rec_fnaddr8.opcodes(vec![0xb8]).rex().w(),
+        rec_fnaddr8.opcodes(&MOV_IMM).rex().w(),
         not_all_ones_funcaddrs_and_not_is_pic,
     );
 
     // Non-PIC, all-zeros funcaddresses.
     e.enc32_isap(
         func_addr.bind(I32),
-        rec_allones_fnaddr4.opcodes(vec![0xb8]),
+        rec_allones_fnaddr4.opcodes(&MOV_IMM),
         all_ones_funcaddrs_and_not_is_pic,
     );
     e.enc64_isap(
         func_addr.bind(I64),
-        rec_allones_fnaddr8.opcodes(vec![0xb8]).rex().w(),
+        rec_allones_fnaddr8.opcodes(&MOV_IMM).rex().w(),
         all_ones_funcaddrs_and_not_is_pic,
     );
 
@@ -1335,14 +1244,14 @@ pub(crate) fn define(
     let is_colocated_func = InstructionPredicate::new_is_colocated_func(f_func_addr, "func_ref");
     e.enc64_instp(
         func_addr.bind(I64),
-        rec_pcrel_fnaddr8.opcodes(vec![0x8d]).rex().w(),
+        rec_pcrel_fnaddr8.opcodes(&LEA).rex().w(),
         is_colocated_func,
     );
 
     // 64-bit, non-colocated, PIC.
     e.enc64_isap(
         func_addr.bind(I64),
-        rec_got_fnaddr8.opcodes(vec![0x8b]).rex().w(),
+        rec_got_fnaddr8.opcodes(&MOV_LOAD).rex().w(),
         is_pic,
     );
 
@@ -1351,19 +1260,19 @@ pub(crate) fn define(
     // Non-PIC.
     e.enc32_isap(
         symbol_value.bind(I32),
-        rec_gvaddr4.opcodes(vec![0xb8]),
+        rec_gvaddr4.opcodes(&MOV_IMM),
         not_is_pic,
     );
     e.enc64_isap(
         symbol_value.bind(I64),
-        rec_gvaddr8.opcodes(vec![0xb8]).rex().w(),
+        rec_gvaddr8.opcodes(&MOV_IMM).rex().w(),
         not_is_pic,
     );
 
     // PIC, colocated.
     e.enc64_func(
         symbol_value.bind(I64),
-        rec_pcrel_gvaddr8.opcodes(vec![0x8d]).rex().w(),
+        rec_pcrel_gvaddr8.opcodes(&LEA).rex().w(),
         |encoding| {
             encoding
                 .isa_predicate(is_pic)
@@ -1374,7 +1283,7 @@ pub(crate) fn define(
     // PIC, non-colocated.
     e.enc64_isap(
         symbol_value.bind(I64),
-        rec_got_gvaddr8.opcodes(vec![0x8b]).rex().w(),
+        rec_got_gvaddr8.opcodes(&MOV_LOAD).rex().w(),
         is_pic,
     );
 
@@ -1382,102 +1291,102 @@ pub(crate) fn define(
     //
     // TODO: Add encoding rules for stack_load and stack_store, so that they
     // don't get legalized to stack_addr + load/store.
-    e.enc32(stack_addr.bind(I32), rec_spaddr4_id.opcodes(vec![0x8d]));
-    e.enc64(
-        stack_addr.bind(I64),
-        rec_spaddr8_id.opcodes(vec![0x8d]).rex().w(),
-    );
+    e.enc32(stack_addr.bind(I32), rec_spaddr4_id.opcodes(&LEA));
+    e.enc64(stack_addr.bind(I64), rec_spaddr8_id.opcodes(&LEA).rex().w());
 
     // Call/return
 
     // 32-bit, both PIC and non-PIC.
-    e.enc32(call, rec_call_id.opcodes(vec![0xe8]));
+    e.enc32(call, rec_call_id.opcodes(&CALL_RELATIVE));
 
     // 64-bit, colocated, both PIC and non-PIC. Use the call instruction's pc-relative field.
     let f_call = formats.get(formats.by_name("Call"));
     let is_colocated_func = InstructionPredicate::new_is_colocated_func(f_call, "func_ref");
-    e.enc64_instp(call, rec_call_id.opcodes(vec![0xe8]), is_colocated_func);
+    e.enc64_instp(call, rec_call_id.opcodes(&CALL_RELATIVE), is_colocated_func);
 
     // 64-bit, non-colocated, PIC. There is no 64-bit non-colocated non-PIC version, since non-PIC
     // is currently using the large model, which requires calls be lowered to
     // func_addr+call_indirect.
-    e.enc64_isap(call, rec_call_plt_id.opcodes(vec![0xe8]), is_pic);
+    e.enc64_isap(call, rec_call_plt_id.opcodes(&CALL_RELATIVE), is_pic);
 
     e.enc32(
         call_indirect.bind(I32),
-        rec_call_r.opcodes(vec![0xff]).rrr(2),
+        rec_call_r.opcodes(&JUMP_ABSOLUTE).rrr(2),
     );
     e.enc64(
         call_indirect.bind(I64),
-        rec_call_r.opcodes(vec![0xff]).rrr(2).rex(),
+        rec_call_r.opcodes(&JUMP_ABSOLUTE).rrr(2).rex(),
     );
     e.enc64(
         call_indirect.bind(I64),
-        rec_call_r.opcodes(vec![0xff]).rrr(2),
+        rec_call_r.opcodes(&JUMP_ABSOLUTE).rrr(2),
     );
 
-    e.enc32(return_, rec_ret.opcodes(vec![0xc3]));
-    e.enc64(return_, rec_ret.opcodes(vec![0xc3]));
+    e.enc32(return_, rec_ret.opcodes(&RET_NEAR));
+    e.enc64(return_, rec_ret.opcodes(&RET_NEAR));
 
     // Branches.
-    e.enc32(jump, rec_jmpb.opcodes(vec![0xeb]));
-    e.enc64(jump, rec_jmpb.opcodes(vec![0xeb]));
-    e.enc32(jump, rec_jmpd.opcodes(vec![0xe9]));
-    e.enc64(jump, rec_jmpd.opcodes(vec![0xe9]));
+    e.enc32(jump, rec_jmpb.opcodes(&JUMP_SHORT));
+    e.enc64(jump, rec_jmpb.opcodes(&JUMP_SHORT));
+    e.enc32(jump, rec_jmpd.opcodes(&JUMP_NEAR_RELATIVE));
+    e.enc64(jump, rec_jmpd.opcodes(&JUMP_NEAR_RELATIVE));
 
-    e.enc_both(brif, rec_brib.opcodes(vec![0x70]));
-    e.enc_both(brif, rec_brid.opcodes(vec![0x0f, 0x80]));
+    e.enc_both(brif, rec_brib.opcodes(&JUMP_SHORT_IF_OVERFLOW));
+    e.enc_both(brif, rec_brid.opcodes(&JUMP_NEAR_IF_OVERFLOW));
 
     // Not all float condition codes are legal, see `supported_floatccs`.
-    e.enc_both(brff, rec_brfb.opcodes(vec![0x70]));
-    e.enc_both(brff, rec_brfd.opcodes(vec![0x0f, 0x80]));
+    e.enc_both(brff, rec_brfb.opcodes(&JUMP_SHORT_IF_OVERFLOW));
+    e.enc_both(brff, rec_brfd.opcodes(&JUMP_NEAR_IF_OVERFLOW));
 
     // Note that the tjccd opcode will be prefixed with 0x0f.
-    e.enc_i32_i64(brz, rec_tjccb.opcodes(vec![0x74]));
-    e.enc_i32_i64(brz, rec_tjccd.opcodes(vec![0x84]));
-    e.enc_i32_i64(brnz, rec_tjccb.opcodes(vec![0x75]));
-    e.enc_i32_i64(brnz, rec_tjccd.opcodes(vec![0x85]));
+    e.enc_i32_i64(brz, rec_tjccb.opcodes(&JUMP_SHORT_IF_EQUAL));
+    e.enc_i32_i64(brz, rec_tjccd.opcodes(&TEST_BYTE_REG));
+    e.enc_i32_i64(brnz, rec_tjccb.opcodes(&JUMP_SHORT_IF_NOT_EQUAL));
+    e.enc_i32_i64(brnz, rec_tjccd.opcodes(&TEST_REG));
 
     // Branch on a b1 value in a register only looks at the low 8 bits. See also
     // bint encodings below.
     //
     // Start with the worst-case encoding for X86_32 only. The register allocator
     // can't handle a branch with an ABCD-constrained operand.
-    e.enc32(brz.bind(B1), rec_t8jccd_long.opcodes(vec![0x84]));
-    e.enc32(brnz.bind(B1), rec_t8jccd_long.opcodes(vec![0x85]));
+    e.enc32(brz.bind(B1), rec_t8jccd_long.opcodes(&TEST_BYTE_REG));
+    e.enc32(brnz.bind(B1), rec_t8jccd_long.opcodes(&TEST_REG));
 
-    e.enc_both(brz.bind(B1), rec_t8jccb_abcd.opcodes(vec![0x74]));
-    e.enc_both(brz.bind(B1), rec_t8jccd_abcd.opcodes(vec![0x84]));
-    e.enc_both(brnz.bind(B1), rec_t8jccb_abcd.opcodes(vec![0x75]));
-    e.enc_both(brnz.bind(B1), rec_t8jccd_abcd.opcodes(vec![0x85]));
+    e.enc_both(brz.bind(B1), rec_t8jccb_abcd.opcodes(&JUMP_SHORT_IF_EQUAL));
+    e.enc_both(brz.bind(B1), rec_t8jccd_abcd.opcodes(&TEST_BYTE_REG));
+    e.enc_both(
+        brnz.bind(B1),
+        rec_t8jccb_abcd.opcodes(&JUMP_SHORT_IF_NOT_EQUAL),
+    );
+    e.enc_both(brnz.bind(B1), rec_t8jccd_abcd.opcodes(&TEST_REG));
 
     // Jump tables.
     e.enc64(
         jump_table_entry.bind(I64),
-        rec_jt_entry.opcodes(vec![0x63]).rex().w(),
+        rec_jt_entry.opcodes(&MOVSXD).rex().w(),
     );
-    e.enc32(jump_table_entry.bind(I32), rec_jt_entry.opcodes(vec![0x8b]));
+    e.enc32(jump_table_entry.bind(I32), rec_jt_entry.opcodes(&MOV_LOAD));
 
     e.enc64(
         jump_table_base.bind(I64),
-        rec_jt_base.opcodes(vec![0x8d]).rex().w(),
+        rec_jt_base.opcodes(&LEA).rex().w(),
     );
-    e.enc32(jump_table_base.bind(I32), rec_jt_base.opcodes(vec![0x8d]));
+    e.enc32(jump_table_base.bind(I32), rec_jt_base.opcodes(&LEA));
 
     e.enc_x86_64(
         indirect_jump_table_br.bind(I64),
-        rec_indirect_jmp.opcodes(vec![0xff]).rrr(4),
+        rec_indirect_jmp.opcodes(&JUMP_ABSOLUTE).rrr(4),
     );
     e.enc32(
         indirect_jump_table_br.bind(I32),
-        rec_indirect_jmp.opcodes(vec![0xff]).rrr(4),
+        rec_indirect_jmp.opcodes(&JUMP_ABSOLUTE).rrr(4),
     );
 
     // Trap as ud2
-    e.enc32(trap, rec_trap.opcodes(vec![0x0f, 0x0b]));
-    e.enc64(trap, rec_trap.opcodes(vec![0x0f, 0x0b]));
-    e.enc32(resumable_trap, rec_trap.opcodes(vec![0x0f, 0x0b]));
-    e.enc64(resumable_trap, rec_trap.opcodes(vec![0x0f, 0x0b]));
+    e.enc32(trap, rec_trap.opcodes(&UNDEFINED2));
+    e.enc64(trap, rec_trap.opcodes(&UNDEFINED2));
+    e.enc32(resumable_trap, rec_trap.opcodes(&UNDEFINED2));
+    e.enc64(resumable_trap, rec_trap.opcodes(&UNDEFINED2));
 
     // Debug trap as int3
     e.enc32_rec(debugtrap, rec_debugtrap, 0);
@@ -1489,31 +1398,28 @@ pub(crate) fn define(
     e.enc64_rec(trapff, rec_trapff, 0);
 
     // Comparisons
-    e.enc_i32_i64(icmp, rec_icscc.opcodes(vec![0x39]));
-    e.enc_i32_i64(icmp_imm, rec_icscc_ib.opcodes(vec![0x83]).rrr(7));
-    e.enc_i32_i64(icmp_imm, rec_icscc_id.opcodes(vec![0x81]).rrr(7));
-    e.enc_i32_i64(ifcmp, rec_rcmp.opcodes(vec![0x39]));
-    e.enc_i32_i64(ifcmp_imm, rec_rcmp_ib.opcodes(vec![0x83]).rrr(7));
-    e.enc_i32_i64(ifcmp_imm, rec_rcmp_id.opcodes(vec![0x81]).rrr(7));
+    e.enc_i32_i64(icmp, rec_icscc.opcodes(&CMP_REG));
+    e.enc_i32_i64(icmp_imm, rec_icscc_ib.opcodes(&CMP_IMM8).rrr(7));
+    e.enc_i32_i64(icmp_imm, rec_icscc_id.opcodes(&CMP_IMM).rrr(7));
+    e.enc_i32_i64(ifcmp, rec_rcmp.opcodes(&CMP_REG));
+    e.enc_i32_i64(ifcmp_imm, rec_rcmp_ib.opcodes(&CMP_IMM8).rrr(7));
+    e.enc_i32_i64(ifcmp_imm, rec_rcmp_id.opcodes(&CMP_IMM).rrr(7));
     // TODO: We could special-case ifcmp_imm(x, 0) to TEST(x, x).
 
-    e.enc32(ifcmp_sp.bind(I32), rec_rcmp_sp.opcodes(vec![0x39]));
-    e.enc64(
-        ifcmp_sp.bind(I64),
-        rec_rcmp_sp.opcodes(vec![0x39]).rex().w(),
-    );
+    e.enc32(ifcmp_sp.bind(I32), rec_rcmp_sp.opcodes(&CMP_REG));
+    e.enc64(ifcmp_sp.bind(I64), rec_rcmp_sp.opcodes(&CMP_REG).rex().w());
 
     // Convert flags to bool.
     // This encodes `b1` as an 8-bit low register with the value 0 or 1.
-    e.enc_both(trueif, rec_seti_abcd.opcodes(vec![0x0f, 0x90]));
-    e.enc_both(trueff, rec_setf_abcd.opcodes(vec![0x0f, 0x90]));
+    e.enc_both(trueif, rec_seti_abcd.opcodes(&SET_BYTE_IF_OVERFLOW));
+    e.enc_both(trueff, rec_setf_abcd.opcodes(&SET_BYTE_IF_OVERFLOW));
 
     // Conditional move (a.k.a integer select).
-    e.enc_i32_i64(selectif, rec_cmov.opcodes(vec![0x0f, 0x40]));
+    e.enc_i32_i64(selectif, rec_cmov.opcodes(&CMOV_OVERFLOW));
 
     // Bit scan forwards and reverse
-    e.enc_i32_i64(x86_bsf, rec_bsf_and_bsr.opcodes(vec![0x0f, 0xbc]));
-    e.enc_i32_i64(x86_bsr, rec_bsf_and_bsr.opcodes(vec![0x0f, 0xbd]));
+    e.enc_i32_i64(x86_bsf, rec_bsf_and_bsr.opcodes(&BIT_SCAN_FORWARD));
+    e.enc_i32_i64(x86_bsr, rec_bsf_and_bsr.opcodes(&BIT_SCAN_REVERSE));
 
     // Convert bool to int.
     //
@@ -1523,24 +1429,24 @@ pub(crate) fn define(
     // Encode movzbq as movzbl, because it's equivalent and shorter.
     e.enc32(
         bint.bind(I32).bind(B1),
-        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xb6]),
+        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
     );
 
     e.enc64(
         bint.bind(I64).bind(B1),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xb6]).rex(),
+        rec_urm_noflags.opcodes(&MOVZX_BYTE).rex(),
     );
     e.enc64(
         bint.bind(I64).bind(B1),
-        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xb6]),
+        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
     );
     e.enc64(
         bint.bind(I32).bind(B1),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xb6]).rex(),
+        rec_urm_noflags.opcodes(&MOVZX_BYTE).rex(),
     );
     e.enc64(
         bint.bind(I32).bind(B1),
-        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xb6]),
+        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
     );
 
     // Numerical conversions.
@@ -1563,103 +1469,103 @@ pub(crate) fn define(
     // movsbl
     e.enc32(
         sextend.bind(I32).bind(I8),
-        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xbe]),
+        rec_urm_noflags_abcd.opcodes(&MOVSX_BYTE),
     );
     e.enc64(
         sextend.bind(I32).bind(I8),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xbe]).rex(),
+        rec_urm_noflags.opcodes(&MOVSX_BYTE).rex(),
     );
     e.enc64(
         sextend.bind(I32).bind(I8),
-        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xbe]),
+        rec_urm_noflags_abcd.opcodes(&MOVSX_BYTE),
     );
 
     // movswl
     e.enc32(
         sextend.bind(I32).bind(I16),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xbf]),
+        rec_urm_noflags.opcodes(&MOVSX_WORD),
     );
     e.enc64(
         sextend.bind(I32).bind(I16),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xbf]).rex(),
+        rec_urm_noflags.opcodes(&MOVSX_WORD).rex(),
     );
     e.enc64(
         sextend.bind(I32).bind(I16),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xbf]),
+        rec_urm_noflags.opcodes(&MOVSX_WORD),
     );
 
     // movsbq
     e.enc64(
         sextend.bind(I64).bind(I8),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xbe]).rex().w(),
+        rec_urm_noflags.opcodes(&MOVSX_BYTE).rex().w(),
     );
 
     // movswq
     e.enc64(
         sextend.bind(I64).bind(I16),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xbf]).rex().w(),
+        rec_urm_noflags.opcodes(&MOVSX_WORD).rex().w(),
     );
 
     // movslq
     e.enc64(
         sextend.bind(I64).bind(I32),
-        rec_urm_noflags.opcodes(vec![0x63]).rex().w(),
+        rec_urm_noflags.opcodes(&MOVSXD).rex().w(),
     );
 
     // movzbl
     e.enc32(
         uextend.bind(I32).bind(I8),
-        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xb6]),
+        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
     );
     e.enc64(
         uextend.bind(I32).bind(I8),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xb6]).rex(),
+        rec_urm_noflags.opcodes(&MOVZX_BYTE).rex(),
     );
     e.enc64(
         uextend.bind(I32).bind(I8),
-        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xb6]),
+        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
     );
 
     // movzwl
     e.enc32(
         uextend.bind(I32).bind(I16),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xb7]),
+        rec_urm_noflags.opcodes(&MOVZX_WORD),
     );
     e.enc64(
         uextend.bind(I32).bind(I16),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xb7]).rex(),
+        rec_urm_noflags.opcodes(&MOVZX_WORD).rex(),
     );
     e.enc64(
         uextend.bind(I32).bind(I16),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xb7]),
+        rec_urm_noflags.opcodes(&MOVZX_WORD),
     );
 
     // movzbq, encoded as movzbl because it's equivalent and shorter.
     e.enc64(
         uextend.bind(I64).bind(I8),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xb6]).rex(),
+        rec_urm_noflags.opcodes(&MOVZX_BYTE).rex(),
     );
     e.enc64(
         uextend.bind(I64).bind(I8),
-        rec_urm_noflags_abcd.opcodes(vec![0x0f, 0xb6]),
+        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
     );
 
     // movzwq, encoded as movzwl because it's equivalent and shorter
     e.enc64(
         uextend.bind(I64).bind(I16),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xb7]).rex(),
+        rec_urm_noflags.opcodes(&MOVZX_WORD).rex(),
     );
     e.enc64(
         uextend.bind(I64).bind(I16),
-        rec_urm_noflags.opcodes(vec![0x0f, 0xb7]),
+        rec_urm_noflags.opcodes(&MOVZX_WORD),
     );
 
     // A 32-bit register copy clears the high 32 bits.
     e.enc64(
         uextend.bind(I64).bind(I32),
-        rec_umr.opcodes(vec![0x89]).rex(),
+        rec_umr.opcodes(&MOV_STORE).rex(),
     );
-    e.enc64(uextend.bind(I64).bind(I32), rec_umr.opcodes(vec![0x89]));
+    e.enc64(uextend.bind(I64).bind(I32), rec_umr.opcodes(&MOV_STORE));
 
     // Floating point
 
@@ -1669,7 +1575,7 @@ pub(crate) fn define(
     let is_zero_32_bit_float = InstructionPredicate::new_is_zero_32bit_float(f_unary_ieee32, "imm");
     e.enc32_instp(
         f32const,
-        rec_f32imm_z.opcodes(vec![0x0f, 0x57]),
+        rec_f32imm_z.opcodes(&XORPS),
         is_zero_32_bit_float.clone(),
     );
 
@@ -1677,148 +1583,133 @@ pub(crate) fn define(
     let is_zero_64_bit_float = InstructionPredicate::new_is_zero_64bit_float(f_unary_ieee64, "imm");
     e.enc32_instp(
         f64const,
-        rec_f64imm_z.opcodes(vec![0x66, 0x0f, 0x57]),
+        rec_f64imm_z.opcodes(&XORPD),
         is_zero_64_bit_float.clone(),
     );
 
-    e.enc_x86_64_instp(
-        f32const,
-        rec_f32imm_z.opcodes(vec![0x0f, 0x57]),
-        is_zero_32_bit_float,
-    );
-    e.enc_x86_64_instp(
-        f64const,
-        rec_f64imm_z.opcodes(vec![0x66, 0x0f, 0x57]), // XORPD from SSE2
-        is_zero_64_bit_float,
-    );
+    e.enc_x86_64_instp(f32const, rec_f32imm_z.opcodes(&XORPS), is_zero_32_bit_float);
+    e.enc_x86_64_instp(f64const, rec_f64imm_z.opcodes(&XORPD), is_zero_64_bit_float);
 
     // movd
     e.enc_both(
         bitcast.bind(F32).bind(I32),
-        rec_frurm.opcodes(vec![0x66, 0x0f, 0x6e]),
+        rec_frurm.opcodes(&MOVD_LOAD_XMM),
     );
     e.enc_both(
         bitcast.bind(I32).bind(F32),
-        rec_rfumr.opcodes(vec![0x66, 0x0f, 0x7e]),
+        rec_rfumr.opcodes(&MOVD_STORE_XMM),
     );
 
     // movq
     e.enc64(
         bitcast.bind(F64).bind(I64),
-        rec_frurm.opcodes(vec![0x66, 0x0f, 0x6e]).rex().w(),
+        rec_frurm.opcodes(&MOVD_LOAD_XMM).rex().w(),
     );
     e.enc64(
         bitcast.bind(I64).bind(F64),
-        rec_rfumr.opcodes(vec![0x66, 0x0f, 0x7e]).rex().w(),
+        rec_rfumr.opcodes(&MOVD_STORE_XMM).rex().w(),
     );
 
     // movaps
-    e.enc_both(copy.bind(F32), rec_furm.opcodes(vec![0x0f, 0x28]));
-    e.enc_both(copy.bind(F64), rec_furm.opcodes(vec![0x0f, 0x28]));
+    e.enc_both(copy.bind(F32), rec_furm.opcodes(&MOVAPS_LOAD));
+    e.enc_both(copy.bind(F64), rec_furm.opcodes(&MOVAPS_LOAD));
 
     // TODO For x86-64, only define REX forms for now, since we can't describe the special regunit
     // immediate operands with the current constraint language.
-    e.enc32(regmove.bind(F32), rec_frmov.opcodes(vec![0x0f, 0x28]));
-    e.enc64(regmove.bind(F32), rec_frmov.opcodes(vec![0x0f, 0x28]).rex());
+    e.enc32(regmove.bind(F32), rec_frmov.opcodes(&MOVAPS_LOAD));
+    e.enc64(regmove.bind(F32), rec_frmov.opcodes(&MOVAPS_LOAD).rex());
 
     // TODO For x86-64, only define REX forms for now, since we can't describe the special regunit
     // immediate operands with the current constraint language.
-    e.enc32(regmove.bind(F64), rec_frmov.opcodes(vec![0x0f, 0x28]));
-    e.enc64(regmove.bind(F64), rec_frmov.opcodes(vec![0x0f, 0x28]).rex());
+    e.enc32(regmove.bind(F64), rec_frmov.opcodes(&MOVAPS_LOAD));
+    e.enc64(regmove.bind(F64), rec_frmov.opcodes(&MOVAPS_LOAD).rex());
 
     // cvtsi2ss
-    e.enc_i32_i64(
-        fcvt_from_sint.bind(F32),
-        rec_frurm.opcodes(vec![0xf3, 0x0f, 0x2a]),
-    );
+    e.enc_i32_i64(fcvt_from_sint.bind(F32), rec_frurm.opcodes(&CVTSI2SS));
 
     // cvtsi2sd
-    e.enc_i32_i64(
-        fcvt_from_sint.bind(F64),
-        rec_frurm.opcodes(vec![0xf2, 0x0f, 0x2a]),
-    );
+    e.enc_i32_i64(fcvt_from_sint.bind(F64), rec_frurm.opcodes(&CVTSI2SD));
 
     // cvtss2sd
-    e.enc_both(
-        fpromote.bind(F64).bind(F32),
-        rec_furm.opcodes(vec![0xf3, 0x0f, 0x5a]),
-    );
+    e.enc_both(fpromote.bind(F64).bind(F32), rec_furm.opcodes(&CVTSS2SD));
 
     // cvtsd2ss
-    e.enc_both(
-        fdemote.bind(F32).bind(F64),
-        rec_furm.opcodes(vec![0xf2, 0x0f, 0x5a]),
-    );
+    e.enc_both(fdemote.bind(F32).bind(F64), rec_furm.opcodes(&CVTSD2SS));
 
     // cvttss2si
     e.enc_both(
         x86_cvtt2si.bind(I32).bind(F32),
-        rec_rfurm.opcodes(vec![0xf3, 0x0f, 0x2c]),
+        rec_rfurm.opcodes(&CVTTSS2SI),
     );
     e.enc64(
         x86_cvtt2si.bind(I64).bind(F32),
-        rec_rfurm.opcodes(vec![0xf3, 0x0f, 0x2c]).rex().w(),
+        rec_rfurm.opcodes(&CVTTSS2SI).rex().w(),
     );
 
     // cvttsd2si
     e.enc_both(
         x86_cvtt2si.bind(I32).bind(F64),
-        rec_rfurm.opcodes(vec![0xf2, 0x0f, 0x2c]),
+        rec_rfurm.opcodes(&CVTTSD2SI),
     );
     e.enc64(
         x86_cvtt2si.bind(I64).bind(F64),
-        rec_rfurm.opcodes(vec![0xf2, 0x0f, 0x2c]).rex().w(),
+        rec_rfurm.opcodes(&CVTTSD2SI).rex().w(),
     );
 
     // Exact square roots.
-    e.enc_both(sqrt.bind(F32), rec_furm.opcodes(vec![0xf3, 0x0f, 0x51]));
-    e.enc_both(sqrt.bind(F64), rec_furm.opcodes(vec![0xf2, 0x0f, 0x51]));
+    e.enc_both(sqrt.bind(F32), rec_furm.opcodes(&SQRTSS));
+    e.enc_both(sqrt.bind(F64), rec_furm.opcodes(&SQRTSD));
 
     // Rounding. The recipe looks at the opcode to pick an immediate.
     for inst in &[nearest, floor, ceil, trunc] {
-        e.enc_both_isap(
-            inst.bind(F32),
-            rec_furmi_rnd.opcodes(vec![0x66, 0x0f, 0x3a, 0x0a]),
-            use_sse41,
-        );
-        e.enc_both_isap(
-            inst.bind(F64),
-            rec_furmi_rnd.opcodes(vec![0x66, 0x0f, 0x3a, 0x0b]),
-            use_sse41,
-        );
+        e.enc_both_isap(inst.bind(F32), rec_furmi_rnd.opcodes(&ROUNDSS), use_sse41);
+        e.enc_both_isap(inst.bind(F64), rec_furmi_rnd.opcodes(&ROUNDSD), use_sse41);
     }
 
     // Binary arithmetic ops.
-    for &(inst, opc) in &[
-        (fadd, 0x58),
-        (fsub, 0x5c),
-        (fmul, 0x59),
-        (fdiv, 0x5e),
-        (x86_fmin, 0x5d),
-        (x86_fmax, 0x5f),
-    ] {
-        e.enc_both(inst.bind(F32), rec_fa.opcodes(vec![0xf3, 0x0f, opc]));
-        e.enc_both(inst.bind(F64), rec_fa.opcodes(vec![0xf2, 0x0f, opc]));
-    }
+    e.enc_both(fadd.bind(F32), rec_fa.opcodes(&ADDSS));
+    e.enc_both(fadd.bind(F64), rec_fa.opcodes(&ADDSD));
+
+    e.enc_both(fsub.bind(F32), rec_fa.opcodes(&SUBSS));
+    e.enc_both(fsub.bind(F64), rec_fa.opcodes(&SUBSD));
+
+    e.enc_both(fmul.bind(F32), rec_fa.opcodes(&MULSS));
+    e.enc_both(fmul.bind(F64), rec_fa.opcodes(&MULSD));
+
+    e.enc_both(fdiv.bind(F32), rec_fa.opcodes(&DIVSS));
+    e.enc_both(fdiv.bind(F64), rec_fa.opcodes(&DIVSD));
+
+    e.enc_both(x86_fmin.bind(F32), rec_fa.opcodes(&MINSS));
+    e.enc_both(x86_fmin.bind(F64), rec_fa.opcodes(&MINSD));
+
+    e.enc_both(x86_fmax.bind(F32), rec_fa.opcodes(&MAXSS));
+    e.enc_both(x86_fmax.bind(F64), rec_fa.opcodes(&MAXSD));
 
     // Binary bitwise ops.
-    for &(inst, opc) in &[(band, 0x54), (bor, 0x56), (bxor, 0x57)] {
-        e.enc_both(inst.bind(F32), rec_fa.opcodes(vec![0x0f, opc]));
-        e.enc_both(inst.bind(F64), rec_fa.opcodes(vec![0x0f, opc]));
-    }
+    //
+    // The F64 version is intentionally encoded using the single-precision opcode:
+    // the operation is identical and the encoding is one byte shorter.
+    e.enc_both(band.bind(F32), rec_fa.opcodes(&ANDPS));
+    e.enc_both(band.bind(F64), rec_fa.opcodes(&ANDPS));
+
+    e.enc_both(bor.bind(F32), rec_fa.opcodes(&ORPS));
+    e.enc_both(bor.bind(F64), rec_fa.opcodes(&ORPS));
+
+    e.enc_both(bxor.bind(F32), rec_fa.opcodes(&XORPS));
+    e.enc_both(bxor.bind(F64), rec_fa.opcodes(&XORPS));
 
     // The `andnps(x,y)` instruction computes `~x&y`, while band_not(x,y)` is `x&~y.
-    e.enc_both(band_not.bind(F32), rec_fax.opcodes(vec![0x0f, 0x55]));
-    e.enc_both(band_not.bind(F64), rec_fax.opcodes(vec![0x0f, 0x55]));
+    e.enc_both(band_not.bind(F32), rec_fax.opcodes(&ANDNPS));
+    e.enc_both(band_not.bind(F64), rec_fax.opcodes(&ANDNPS));
 
     // Comparisons.
     //
     // This only covers the condition codes in `supported_floatccs`, the rest are
     // handled by legalization patterns.
-    e.enc_both(fcmp.bind(F32), rec_fcscc.opcodes(vec![0x0f, 0x2e]));
-    e.enc_both(fcmp.bind(F64), rec_fcscc.opcodes(vec![0x66, 0x0f, 0x2e]));
-    e.enc_both(ffcmp.bind(F32), rec_fcmp.opcodes(vec![0x0f, 0x2e]));
-    e.enc_both(ffcmp.bind(F64), rec_fcmp.opcodes(vec![0x66, 0x0f, 0x2e]));
+    e.enc_both(fcmp.bind(F32), rec_fcscc.opcodes(&UCOMISS));
+    e.enc_both(fcmp.bind(F64), rec_fcscc.opcodes(&UCOMISD));
+    e.enc_both(ffcmp.bind(F32), rec_fcmp.opcodes(&UCOMISS));
+    e.enc_both(ffcmp.bind(F64), rec_fcmp.opcodes(&UCOMISD));
 
     // SIMD vector size: eventually multiple vector sizes may be supported but for now only
     // SSE-sized vectors are available.
@@ -1833,7 +1724,7 @@ pub(crate) fn define(
     // PSHUFB, 8-bit shuffle using two XMM registers.
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         let instruction = x86_pshufb.bind_vector_from_lane(ty, sse_vector_size);
-        let template = rec_fa.nonrex().opcodes(vec![0x66, 0x0f, 0x38, 00]);
+        let template = rec_fa.nonrex().opcodes(&PSHUFB);
         e.enc32_isap(instruction.clone(), template.clone(), use_ssse3_simd);
         e.enc64_isap(instruction, template, use_ssse3_simd);
     }
@@ -1841,9 +1732,7 @@ pub(crate) fn define(
     // PSHUFD, 32-bit shuffle using one XMM register and a u8 immediate.
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 32) {
         let instruction = x86_pshufd.bind_vector_from_lane(ty, sse_vector_size);
-        let template = rec_r_ib_unsigned_fpr
-            .nonrex()
-            .opcodes(vec![0x66, 0x0f, 0x70]);
+        let template = rec_r_ib_unsigned_fpr.nonrex().opcodes(&PSHUFD);
         e.enc32(instruction.clone(), template.clone());
         e.enc64(instruction, template);
     }
@@ -1856,7 +1745,7 @@ pub(crate) fn define(
         if ty.is_float() {
             e.enc_32_64_rec(instruction, rec_null_fpr, 0);
         } else {
-            let template = rec_frurm.opcodes(vec![0x66, 0x0f, 0x6e]); // MOVD/MOVQ
+            let template = rec_frurm.opcodes(&MOVD_LOAD_XMM);
             if ty.lane_bits() < 64 {
                 // no 32-bit encodings for 64-bit widths
                 e.enc32(instruction.clone(), template.clone());
@@ -1866,17 +1755,17 @@ pub(crate) fn define(
     }
 
     // SIMD insertlane
-    let mut x86_pinsr_mapping: HashMap, Option)> =
+    let mut x86_pinsr_mapping: HashMap)> =
         HashMap::new();
-    x86_pinsr_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x20], Some(use_sse41_simd))); // PINSRB
-    x86_pinsr_mapping.insert(16, (vec![0x66, 0x0f, 0xc4], None)); // PINSRW from SSE2
-    x86_pinsr_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41_simd))); // PINSRD
-    x86_pinsr_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x22], Some(use_sse41_simd))); // PINSRQ, only x86_64
+    x86_pinsr_mapping.insert(8, (&PINSRB, Some(use_sse41_simd)));
+    x86_pinsr_mapping.insert(16, (&PINSRW, None));
+    x86_pinsr_mapping.insert(32, (&PINSR, Some(use_sse41_simd)));
+    x86_pinsr_mapping.insert(64, (&PINSR, Some(use_sse41_simd)));
 
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         if let Some((opcode, isap)) = x86_pinsr_mapping.get(&ty.lane_bits()) {
             let instruction = x86_pinsr.bind_vector_from_lane(ty, sse_vector_size);
-            let template = rec_r_ib_unsigned_r.opcodes(opcode.clone());
+            let template = rec_r_ib_unsigned_r.opcodes(opcode);
             if ty.lane_bits() < 64 {
                 e.enc_32_64_maybe_isap(instruction, template.nonrex(), isap.clone());
             } else {
@@ -1890,36 +1779,36 @@ pub(crate) fn define(
     // For legalizing insertlane with floats, INSERTPS from SSE4.1.
     {
         let instruction = x86_insertps.bind_vector_from_lane(F32, sse_vector_size);
-        let template = rec_fa_ib.nonrex().opcodes(vec![0x66, 0x0f, 0x3a, 0x21]);
+        let template = rec_fa_ib.nonrex().opcodes(&INSERTPS);
         e.enc_32_64_maybe_isap(instruction, template, Some(use_sse41_simd));
     }
 
     // For legalizing insertlane with floats,  MOVSD from SSE2.
     {
         let instruction = x86_movsd.bind_vector_from_lane(F64, sse_vector_size);
-        let template = rec_fa.nonrex().opcodes(vec![0xf2, 0x0f, 0x10]);
+        let template = rec_fa.nonrex().opcodes(&MOVSD_LOAD);
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE2
     }
 
     // For legalizing insertlane with floats, MOVLHPS from SSE.
     {
         let instruction = x86_movlhps.bind_vector_from_lane(F64, sse_vector_size);
-        let template = rec_fa.nonrex().opcodes(vec![0x0f, 0x16]);
+        let template = rec_fa.nonrex().opcodes(&MOVLHPS);
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
     }
 
     // SIMD extractlane
-    let mut x86_pextr_mapping: HashMap, Option)> =
+    let mut x86_pextr_mapping: HashMap)> =
         HashMap::new();
-    x86_pextr_mapping.insert(8, (vec![0x66, 0x0f, 0x3a, 0x14], Some(use_sse41_simd))); // PEXTRB
-    x86_pextr_mapping.insert(16, (vec![0x66, 0x0f, 0xc5], None)); // PEXTRW from SSE2, SSE4.1 has a PEXTRW that can move to reg/m16 but the opcode is four bytes
-    x86_pextr_mapping.insert(32, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41_simd))); // PEXTRD
-    x86_pextr_mapping.insert(64, (vec![0x66, 0x0f, 0x3a, 0x16], Some(use_sse41_simd))); // PEXTRQ, only x86_64
+    x86_pextr_mapping.insert(8, (&PEXTRB, Some(use_sse41_simd)));
+    x86_pextr_mapping.insert(16, (&PEXTRW_SSE2, None));
+    x86_pextr_mapping.insert(32, (&PEXTR, Some(use_sse41_simd)));
+    x86_pextr_mapping.insert(64, (&PEXTR, Some(use_sse41_simd)));
 
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         if let Some((opcode, isap)) = x86_pextr_mapping.get(&ty.lane_bits()) {
             let instruction = x86_pextr.bind_vector_from_lane(ty, sse_vector_size);
-            let template = rec_r_ib_unsigned_gpr.opcodes(opcode.clone());
+            let template = rec_r_ib_unsigned_gpr.opcodes(opcode);
             if ty.lane_bits() < 64 {
                 e.enc_32_64_maybe_isap(instruction, template.nonrex(), isap.clone());
             } else {
@@ -1972,18 +1861,14 @@ pub(crate) fn define(
 
         let is_zero_128bit =
             InstructionPredicate::new_is_all_zeroes_128bit(f_unary_const, "constant_handle");
-        let template = rec_vconst_optimized
-            .nonrex()
-            .opcodes(vec![0x66, 0x0f, 0xef]); // PXOR from SSE2
+        let template = rec_vconst_optimized.nonrex().opcodes(&PXOR);
         e.enc_32_64_func(instruction.clone(), template, |builder| {
             builder.inst_predicate(is_zero_128bit)
         });
 
         let is_ones_128bit =
             InstructionPredicate::new_is_all_ones_128bit(f_unary_const, "constant_handle");
-        let template = rec_vconst_optimized
-            .nonrex()
-            .opcodes(vec![0x66, 0x0f, 0x74]); // PCMPEQB from SSE2
+        let template = rec_vconst_optimized.nonrex().opcodes(&PCMPEQB);
         e.enc_32_64_func(instruction, template, |builder| {
             builder.inst_predicate(is_ones_128bit)
         });
@@ -1997,14 +1882,14 @@ pub(crate) fn define(
     // in memory) but some performance measurements are needed.
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         let instruction = vconst.bind_vector_from_lane(ty, sse_vector_size);
-        let template = rec_vconst.nonrex().opcodes(vec![0x0f, 0x10]);
+        let template = rec_vconst.nonrex().opcodes(&MOVUPS_LOAD);
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
     }
 
     // SIMD bor using ORPS
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         let instruction = bor.bind_vector_from_lane(ty, sse_vector_size);
-        let template = rec_fa.nonrex().opcodes(vec![0x0f, 0x56]);
+        let template = rec_fa.nonrex().opcodes(&ORPS);
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
     }
 
@@ -2014,66 +1899,60 @@ pub(crate) fn define(
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         // Store
         let bound_store = store.bind_vector_from_lane(ty, sse_vector_size).bind_any();
-        e.enc_32_64(bound_store.clone(), rec_fst.opcodes(vec![0x0f, 0x11]));
-        e.enc_32_64(bound_store.clone(), rec_fstDisp8.opcodes(vec![0x0f, 0x11]));
-        e.enc_32_64(bound_store, rec_fstDisp32.opcodes(vec![0x0f, 0x11]));
+        e.enc_32_64(bound_store.clone(), rec_fst.opcodes(&MOVUPS_STORE));
+        e.enc_32_64(bound_store.clone(), rec_fstDisp8.opcodes(&MOVUPS_STORE));
+        e.enc_32_64(bound_store, rec_fstDisp32.opcodes(&MOVUPS_STORE));
 
         // Load
         let bound_load = load.bind_vector_from_lane(ty, sse_vector_size).bind_any();
-        e.enc_32_64(bound_load.clone(), rec_fld.opcodes(vec![0x0f, 0x10]));
-        e.enc_32_64(bound_load.clone(), rec_fldDisp8.opcodes(vec![0x0f, 0x10]));
-        e.enc_32_64(bound_load, rec_fldDisp32.opcodes(vec![0x0f, 0x10]));
+        e.enc_32_64(bound_load.clone(), rec_fld.opcodes(&MOVUPS_LOAD));
+        e.enc_32_64(bound_load.clone(), rec_fldDisp8.opcodes(&MOVUPS_LOAD));
+        e.enc_32_64(bound_load, rec_fldDisp32.opcodes(&MOVUPS_LOAD));
 
         // Spill
         let bound_spill = spill.bind_vector_from_lane(ty, sse_vector_size);
-        e.enc_32_64(bound_spill, rec_fspillSib32.opcodes(vec![0x0f, 0x11]));
+        e.enc_32_64(bound_spill, rec_fspillSib32.opcodes(&MOVUPS_STORE));
         let bound_regspill = regspill.bind_vector_from_lane(ty, sse_vector_size);
-        e.enc_32_64(bound_regspill, rec_fregspill32.opcodes(vec![0x0f, 0x11]));
+        e.enc_32_64(bound_regspill, rec_fregspill32.opcodes(&MOVUPS_STORE));
 
         // Fill
         let bound_fill = fill.bind_vector_from_lane(ty, sse_vector_size);
-        e.enc_32_64(bound_fill, rec_ffillSib32.opcodes(vec![0x0f, 0x10]));
+        e.enc_32_64(bound_fill, rec_ffillSib32.opcodes(&MOVUPS_LOAD));
         let bound_regfill = regfill.bind_vector_from_lane(ty, sse_vector_size);
-        e.enc_32_64(bound_regfill, rec_fregfill32.opcodes(vec![0x0f, 0x10]));
+        e.enc_32_64(bound_regfill, rec_fregfill32.opcodes(&MOVUPS_LOAD));
         let bound_fill_nop = fill_nop.bind_vector_from_lane(ty, sse_vector_size);
         e.enc_32_64_rec(bound_fill_nop, rec_ffillnull, 0);
 
         // Regmove
         let bound_regmove = regmove.bind_vector_from_lane(ty, sse_vector_size);
-        e.enc_32_64(bound_regmove, rec_frmov.opcodes(vec![0x0f, 0x28]));
+        e.enc_32_64(bound_regmove, rec_frmov.opcodes(&MOVAPS_LOAD));
 
         // Copy
         let bound_copy = copy.bind_vector_from_lane(ty, sse_vector_size);
-        e.enc_32_64(bound_copy, rec_furm.opcodes(vec![0x0f, 0x28])); // MOVAPS from SSE
+        e.enc_32_64(bound_copy, rec_furm.opcodes(&MOVAPS_LOAD));
         let bound_copy_nop = copy_nop.bind_vector_from_lane(ty, sse_vector_size);
         e.enc_32_64_rec(bound_copy_nop, rec_stacknull, 0);
     }
 
     // SIMD integer addition
-    for (ty, opcodes) in &[
-        (I8, &[0x66, 0x0f, 0xfc]),  // PADDB from SSE2
-        (I16, &[0x66, 0x0f, 0xfd]), // PADDW from SSE2
-        (I32, &[0x66, 0x0f, 0xfe]), // PADDD from SSE2
-        (I64, &[0x66, 0x0f, 0xd4]), // PADDQ from SSE2
-    ] {
+    for (ty, opcodes) in &[(I8, &PADDB), (I16, &PADDW), (I32, &PADDD), (I64, &PADDQ)] {
         let iadd = iadd.bind_vector_from_lane(ty.clone(), sse_vector_size);
-        e.enc_32_64(iadd, rec_fa.opcodes(opcodes.to_vec()));
+        e.enc_32_64(iadd, rec_fa.opcodes(*opcodes));
     }
 
     // SIMD icmp using PCMPEQ*
-    let mut pcmpeq_mapping: HashMap, Option)> =
-        HashMap::new();
-    pcmpeq_mapping.insert(8, (vec![0x66, 0x0f, 0x74], None)); // PCMPEQB from SSE2
-    pcmpeq_mapping.insert(16, (vec![0x66, 0x0f, 0x75], None)); // PCMPEQW from SSE2
-    pcmpeq_mapping.insert(32, (vec![0x66, 0x0f, 0x76], None)); // PCMPEQD from SSE2
-    pcmpeq_mapping.insert(64, (vec![0x66, 0x0f, 0x38, 0x29], Some(use_sse41_simd))); // PCMPEQQ from SSE4.1
+    let mut pcmpeq_mapping: HashMap)> = HashMap::new();
+    pcmpeq_mapping.insert(8, (&PCMPEQB, None));
+    pcmpeq_mapping.insert(16, (&PCMPEQW, None));
+    pcmpeq_mapping.insert(32, (&PCMPEQD, None));
+    pcmpeq_mapping.insert(64, (&PCMPEQQ, Some(use_sse41_simd)));
     for ty in ValueType::all_lane_types().filter(|t| t.is_int() && allowed_simd_type(t)) {
         if let Some((opcodes, isa_predicate)) = pcmpeq_mapping.get(&ty.lane_bits()) {
             let instruction = icmp.bind_vector_from_lane(ty, sse_vector_size);
             let f_int_compare = formats.get(formats.by_name("IntCompare"));
             let has_eq_condition_code =
                 InstructionPredicate::new_has_condition_code(f_int_compare, IntCC::Equal, "cond");
-            let template = rec_icscc_fpr.nonrex().opcodes(opcodes.clone());
+            let template = rec_icscc_fpr.nonrex().opcodes(*opcodes);
             e.enc_32_64_func(instruction, template, |builder| {
                 let builder = builder.inst_predicate(has_eq_condition_code);
                 if let Some(p) = isa_predicate {
@@ -2088,13 +1967,13 @@ pub(crate) fn define(
     // Reference type instructions
 
     // Null references implemented as iconst 0.
-    e.enc32(null.bind_ref(R32), rec_pu_id_ref.opcodes(vec![0xb8]));
+    e.enc32(null.bind_ref(R32), rec_pu_id_ref.opcodes(&MOV_IMM));
 
-    e.enc64(null.bind_ref(R64), rec_pu_id_ref.rex().opcodes(vec![0xb8]));
-    e.enc64(null.bind_ref(R64), rec_pu_id_ref.opcodes(vec![0xb8]));
+    e.enc64(null.bind_ref(R64), rec_pu_id_ref.rex().opcodes(&MOV_IMM));
+    e.enc64(null.bind_ref(R64), rec_pu_id_ref.opcodes(&MOV_IMM));
 
     // is_null, implemented by testing whether the value is 0.
-    e.enc_r32_r64_rex_only(is_null, rec_is_zero.opcodes(vec![0x85]));
+    e.enc_r32_r64_rex_only(is_null, rec_is_zero.opcodes(&TEST_REG));
 
     // safepoint instruction calls sink, no actual encoding.
     e.enc32_rec(safepoint, rec_safepoint, 0);
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index 6ced509396..e18c54ad64 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -9,6 +9,7 @@ use crate::shared::Definitions as SharedDefinitions;
 mod encodings;
 mod instructions;
 mod legalize;
+mod opcodes;
 mod recipes;
 mod registers;
 mod settings;
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
new file mode 100644
index 0000000000..12b60e532b
--- /dev/null
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -0,0 +1,377 @@
+//! Static, named definitions of instruction opcodes.
+
+/// Empty opcode for use as a default.
+pub static EMPTY: [u8; 0] = [];
+
+/// Add with carry flag r{16,32,64} to r/m of the same size.
+pub static ADC: [u8; 1] = [0x11];
+
+/// Add r{16,32,64} to r/m of the same size.
+pub static ADD: [u8; 1] = [0x01];
+
+/// Add imm{16,32} to r/m{16,32,64}, possibly sign-extended.
+pub static ADD_IMM: [u8; 1] = [0x81];
+
+/// Add sign-extended imm8 to r/m{16,32,64}.
+pub static ADD_IMM8_SIGN_EXTEND: [u8; 1] = [0x83];
+
+/// Add the low double-precision floating-point value from xmm2/mem to xmm1
+/// and store the result in xmm1.
+pub static ADDSD: [u8; 3] = [0xf2, 0x0f, 0x58];
+
+/// Add the low single-precision floating-point value from xmm2/mem to xmm1
+/// and store the result in xmm1.
+pub static ADDSS: [u8; 3] = [0xf3, 0x0f, 0x58];
+
+/// r/m{16,32,64} AND register of the same size (Intel docs have a typo).
+pub static AND: [u8; 1] = [0x21];
+
+/// imm{16,32} AND r/m{16,32,64}, possibly sign-extended.
+pub static AND_IMM: [u8; 1] = [0x81];
+
+/// r/m{16,32,64} AND sign-extended imm8.
+pub static AND_IMM8_SIGN_EXTEND: [u8; 1] = [0x83];
+
+/// Return the bitwise logical AND NOT of packed single-precision floating-point
+/// values in xmm1 and xmm2/mem.
+pub static ANDNPS: [u8; 2] = [0x0f, 0x55];
+
+/// Return the bitwise logical AND of packed single-precision floating-point values
+/// in xmm1 and xmm2/mem.
+pub static ANDPS: [u8; 2] = [0x0f, 0x54];
+
+/// Bit scan forward (stores index of first encountered 1 from the front).
+pub static BIT_SCAN_FORWARD: [u8; 2] = [0x0f, 0xbc];
+
+/// Bit scan reverse (stores index of first encountered 1 from the back).
+pub static BIT_SCAN_REVERSE: [u8; 2] = [0x0f, 0xbd];
+
+/// Call near, relative, displacement relative to next instruction (sign-extended).
+pub static CALL_RELATIVE: [u8; 1] = [0xe8];
+
+/// Move r/m{16,32,64} if overflow (OF=1).
+pub static CMOV_OVERFLOW: [u8; 2] = [0x0f, 0x40];
+
+/// Compare imm{16,32} with r/m{16,32,64} (sign-extended if 64).
+pub static CMP_IMM: [u8; 1] = [0x81];
+
+/// Compare imm8 with r/m{16,32,64}.
+pub static CMP_IMM8: [u8; 1] = [0x83];
+
+/// Compare r{16,32,64} with r/m of the same size.
+pub static CMP_REG: [u8; 1] = [0x39];
+
+/// Convert scalar double-precision floating-point value to scalar single-precision
+/// floating-point value.
+pub static CVTSD2SS: [u8; 3] = [0xf2, 0x0f, 0x5a];
+
+/// Convert doubleword integer to scalar double-precision floating-point value.
+pub static CVTSI2SD: [u8; 3] = [0xf2, 0x0f, 0x2a];
+
+/// Convert doubleword integer to scalar single-precision floating-point value.
+pub static CVTSI2SS: [u8; 3] = [0xf3, 0x0f, 0x2a];
+
+/// Convert scalar single-precision floating-point value to scalar double-precision
+/// float-point value.
+pub static CVTSS2SD: [u8; 3] = [0xf3, 0x0f, 0x5a];
+
+/// Convert with truncation scalar double-precision floating-point value to signed
+/// integer.
+pub static CVTTSD2SI: [u8; 3] = [0xf2, 0x0f, 0x2c];
+
+/// Convert with truncation scalar single-precision floating-point value to integer.
+pub static CVTTSS2SI: [u8; 3] = [0xf3, 0x0f, 0x2c];
+
+/// Unsigned divide for {16,32,64}-bit.
+pub static DIV: [u8; 1] = [0xf7];
+
+/// Divide low double-precision floating-point value in xmm1 by low double-precision
+/// floating-point value in xmm2/m64.
+pub static DIVSD: [u8; 3] = [0xf2, 0x0f, 0x5e];
+
+/// Divide low single-precision floating-point value in xmm1 by low single-precision
+/// floating-point value in xmm2/m32.
+pub static DIVSS: [u8; 3] = [0xf3, 0x0f, 0x5e];
+
+/// Signed divide for {16,32,64}-bit.
+pub static IDIV: [u8; 1] = [0xf7];
+
+/// Signed multiply for {16,32,64}-bit, generic registers.
+pub static IMUL: [u8; 2] = [0x0f, 0xaf];
+
+/// Signed multiply for {16,32,64}-bit, storing into RDX:RAX.
+pub static IMUL_RDX_RAX: [u8; 1] = [0xf7];
+
+/// Insert scalar single-precision floating-point value.
+pub static INSERTPS: [u8; 4] = [0x66, 0x0f, 0x3a, 0x21];
+
+/// Either:
+///  1. Jump near, absolute indirect, RIP = 64-bit offset from register or memory.
+///  2. Jump far, absolute indirect, address given in m16:64.
+pub static JUMP_ABSOLUTE: [u8; 1] = [0xff];
+
+/// Jump near, relative, RIP = RIP + 32-bit displacement sign extended to 64 bits.
+pub static JUMP_NEAR_RELATIVE: [u8; 1] = [0xe9];
+
+/// Jump near (rel32) if overflow (OF=1).
+pub static JUMP_NEAR_IF_OVERFLOW: [u8; 2] = [0x0f, 0x80];
+
+/// Jump short, relative, RIP = RIP + 8-bit displacement sign extended to 64 bits.
+pub static JUMP_SHORT: [u8; 1] = [0xeb];
+
+/// Jump short (rel8) if equal (ZF=1).
+pub static JUMP_SHORT_IF_EQUAL: [u8; 1] = [0x74];
+
+/// Jump short (rel8) if not equal (ZF=0).
+pub static JUMP_SHORT_IF_NOT_EQUAL: [u8; 1] = [0x75];
+
+/// Jump short (rel8) if overflow (OF=1).
+pub static JUMP_SHORT_IF_OVERFLOW: [u8; 1] = [0x70];
+
+/// Store effective address for m in register r{16,32,64}.
+pub static LEA: [u8; 1] = [0x8d];
+
+/// Count the number of leading zero bits.
+pub static LZCNT: [u8; 3] = [0xf3, 0x0f, 0xbd];
+
+/// Return the maximum scalar double-precision floating-point value between
+/// xmm2/m64 and xmm1.
+pub static MAXSD: [u8; 3] = [0xf2, 0x0f, 0x5f];
+
+/// Return the maximum scalar single-precision floating-point value between
+/// xmm2/m32 and xmm1.
+pub static MAXSS: [u8; 3] = [0xf3, 0x0f, 0x5f];
+
+/// Return the minimum scalar double-precision floating-point value between
+/// xmm2/m64 and xmm1.
+pub static MINSD: [u8; 3] = [0xf2, 0x0f, 0x5d];
+
+/// Return the minimum scalar single-precision floating-point value between
+/// xmm2/m32 and xmm1.
+pub static MINSS: [u8; 3] = [0xf3, 0x0f, 0x5d];
+
+/// Move r8 to r/m8.
+pub static MOV_BYTE_STORE: [u8; 1] = [0x88];
+
+/// Move imm{16,32,64} to same-sized register.
+pub static MOV_IMM: [u8; 1] = [0xb8];
+
+/// Move imm{16,32} to r{16,32,64}, sign-extended if 64-bit target.
+pub static MOV_IMM_SIGNEXTEND: [u8; 1] = [0xc7];
+
+/// Move {r/m16, r/m32, r/m64} to same-sized register.
+pub static MOV_LOAD: [u8; 1] = [0x8b];
+
+/// Move r16 to r/m16.
+pub static MOV_STORE_16: [u8; 2] = [0x66, 0x89];
+
+/// Move {r16, r32, r64} to same-sized register or memory.
+pub static MOV_STORE: [u8; 1] = [0x89];
+
+/// Move aligned packed single-precision floating-point values from x/m to xmm (SSE).
+pub static MOVAPS_LOAD: [u8; 2] = [0x0f, 0x28];
+
+/// Move doubleword from r/m32 to xmm (SSE2). Quadword with REX prefix.
+pub static MOVD_LOAD_XMM: [u8; 3] = [0x66, 0x0f, 0x6e];
+
+/// Move doubleword from xmm to r/m32 (SSE2). Quadword with REX prefix.
+pub static MOVD_STORE_XMM: [u8; 3] = [0x66, 0x0f, 0x7e];
+
+/// Move packed single-precision floating-point values low to high (SSE).
+pub static MOVLHPS: [u8; 2] = [0x0f, 0x16];
+
+/// Move scalar double-precision floating-point value (from reg/mem to reg).
+pub static MOVSD_LOAD: [u8; 3] = [0xf2, 0x0f, 0x10];
+
+/// Move scalar double-precision floating-point value (from reg to reg/mem).
+pub static MOVSD_STORE: [u8; 3] = [0xf2, 0x0f, 0x11];
+
+/// Move scalar single-precision floating-point value (from reg to reg/mem).
+pub static MOVSS_STORE: [u8; 3] = [0xf3, 0x0f, 0x11];
+
+/// Move scalar single-precision floating-point-value (from reg/mem to reg).
+pub static MOVSS_LOAD: [u8; 3] = [0xf3, 0x0f, 0x10];
+
+/// Move byte to register with sign-extension.
+pub static MOVSX_BYTE: [u8; 2] = [0x0f, 0xbe];
+
+/// Move word to register with sign-extension.
+pub static MOVSX_WORD: [u8; 2] = [0x0f, 0xbf];
+
+/// Move doubleword to register with sign-extension.
+pub static MOVSXD: [u8; 1] = [0x63];
+
+/// Move unaligned packed single-precision floating-point from x/m to xmm (SSE).
+pub static MOVUPS_LOAD: [u8; 2] = [0x0f, 0x10];
+
+/// Move unaligned packed single-precision floating-point value from xmm to x/m (SSE).
+pub static MOVUPS_STORE: [u8; 2] = [0x0f, 0x11];
+
+/// Move byte to register with zero-extension.
+pub static MOVZX_BYTE: [u8; 2] = [0x0f, 0xb6];
+
+/// Move word to register with zero-extension.
+pub static MOVZX_WORD: [u8; 2] = [0x0f, 0xb7];
+
+/// Unsigned multiply for {16,32,64}-bit.
+pub static MUL: [u8; 1] = [0xf7];
+
+/// Multiply the low double-precision floating-point value in xmm2/m64 by the
+/// low double-precision floating-point value in xmm1.
+pub static MULSD: [u8; 3] = [0xf2, 0x0f, 0x59];
+
+/// Multiply the low single-precision floating-point value in xmm2/m32 by the
+/// low single-precision floating-point value in xmm1.
+pub static MULSS: [u8; 3] = [0xf3, 0x0f, 0x59];
+
+/// Reverse each bit of r/m{16,32,64}.
+pub static NOT: [u8; 1] = [0xf7];
+
+/// r{16,32,64} OR register of same size.
+pub static OR: [u8; 1] = [0x09];
+
+/// imm{16,32} OR r/m{16,32,64}, possibly sign-extended.
+pub static OR_IMM: [u8; 1] = [0x81];
+
+/// r/m{16,32,64} OR sign-extended imm8.
+pub static OR_IMM8_SIGN_EXTEND: [u8; 1] = [0x83];
+
+/// Return the bitwise logical OR of packed single-precision values in xmm and x/m (SSE).
+pub static ORPS: [u8; 2] = [0x0f, 0x56];
+
+/// Add packed byte integers from xmm2/m128 and xmm1 (SSE2).
+pub static PADDB: [u8; 3] = [0x66, 0x0f, 0xfc];
+
+/// Add packed doubleword integers from xmm2/m128 and xmm1 (SSE2).
+pub static PADDD: [u8; 3] = [0x66, 0x0f, 0xfe];
+
+/// Add packed quadword integers from xmm2/m128 and xmm1 (SSE2).
+pub static PADDQ: [u8; 3] = [0x66, 0x0f, 0xd4];
+
+/// Add packed word integers from xmm2/m128 and xmm1 (SSE2).
+pub static PADDW: [u8; 3] = [0x66, 0x0f, 0xfd];
+
+/// Compare packed data for equal (SSE2).
+pub static PCMPEQB: [u8; 3] = [0x66, 0x0f, 0x74];
+
+/// Compare packed data for equal (SSE2).
+pub static PCMPEQD: [u8; 3] = [0x66, 0x0f, 0x76];
+
+/// Compare packed data for equal (SSE4.1).
+pub static PCMPEQQ: [u8; 4] = [0x66, 0x0f, 0x38, 0x29];
+
+/// Compare packed data for equal (SSE2).
+pub static PCMPEQW: [u8; 3] = [0x66, 0x0f, 0x75];
+
+/// Extract doubleword or quadword, depending on REX.W (SSE4.1).
+pub static PEXTR: [u8; 4] = [0x66, 0x0f, 0x3a, 0x16];
+
+/// Extract byte (SSE4.1).
+pub static PEXTRB: [u8; 4] = [0x66, 0x0f, 0x3a, 0x14];
+
+/// Extract word (SSE2). There is a 4-byte SSE4.1 variant that can also move to m/16.
+pub static PEXTRW_SSE2: [u8; 3] = [0x66, 0x0f, 0xc5];
+
+/// Insert doubleword or quadword, depending on REX.W (SSE4.1).
+pub static PINSR: [u8; 4] = [0x66, 0x0f, 0x3a, 0x22];
+
+/// Insert byte (SSE4.1).
+pub static PINSRB: [u8; 4] = [0x66, 0x0f, 0x3a, 0x20];
+
+/// Insert word (SSE2).
+pub static PINSRW: [u8; 3] = [0x66, 0x0f, 0xc4];
+
+/// Pop top of stack into r{16,32,64}; increment stack pointer.
+pub static POP_REG: [u8; 1] = [0x58];
+
+/// Returns the count of number of bits set to 1.
+pub static POPCNT: [u8; 3] = [0xf3, 0x0f, 0xb8];
+
+/// Shuffle bytes in xmm1 according to contents of xmm2/m128 (SSE3).
+pub static PSHUFB: [u8; 4] = [0x66, 0x0f, 0x38, 0x00];
+
+/// Shuffle the doublewords in xmm2/m128 based on the encoding in imm8 and
+/// store the result in xmm1 (SSE2).
+pub static PSHUFD: [u8; 3] = [0x66, 0x0f, 0x70];
+
+/// Push r{16,32,64}.
+pub static PUSH_REG: [u8; 1] = [0x50];
+
+/// Logical exclusive OR (SSE2).
+pub static PXOR: [u8; 3] = [0x66, 0x0f, 0xef];
+
+/// Near return to calling procedure.
+pub static RET_NEAR: [u8; 1] = [0xc3];
+
+/// General rotation opcode. Kind of rotation depends on encoding.
+pub static ROTATE_CL: [u8; 1] = [0xd3];
+
+/// General rotation opcode. Kind of rotation depends on encoding.
+pub static ROTATE_IMM8: [u8; 1] = [0xc1];
+
+/// Round scalar doubl-precision floating-point values.
+pub static ROUNDSD: [u8; 4] = [0x66, 0x0f, 0x3a, 0x0b];
+
+/// Round scalar single-precision floating-point values.
+pub static ROUNDSS: [u8; 4] = [0x66, 0x0f, 0x3a, 0x0a];
+
+/// Subtract with borrow r{16,32,64} from r/m of the same size.
+pub static SBB: [u8; 1] = [0x19];
+
+/// Set byte if overflow (OF=1).
+pub static SET_BYTE_IF_OVERFLOW: [u8; 2] = [0x0f, 0x90];
+
+/// Compute square root of scalar double-precision floating-point value.
+pub static SQRTSD: [u8; 3] = [0xf2, 0x0f, 0x51];
+
+/// Compute square root of scalar single-precision value.
+pub static SQRTSS: [u8; 3] = [0xf3, 0x0f, 0x51];
+
+/// Subtract r{16,32,64} from r/m of same size.
+pub static SUB: [u8; 1] = [0x29];
+
+/// Subtract the low double-precision floating-point value in xmm2/m64 from xmm1
+/// and store the result in xmm1.
+pub static SUBSD: [u8; 3] = [0xf2, 0x0f, 0x5c];
+
+/// Subtract the low single-precision floating-point value in xmm2/m32 from xmm1
+/// and store the result in xmm1.
+pub static SUBSS: [u8; 3] = [0xf3, 0x0f, 0x5c];
+
+/// AND r8 with r/m8; set SF, ZF, PF according to result.
+pub static TEST_BYTE_REG: [u8; 1] = [0x84];
+
+/// AND {r16, r32, r64} with r/m of the same size; set SF, ZF, PF according to result.
+pub static TEST_REG: [u8; 1] = [0x85];
+
+/// Count the number of trailing zero bits.
+pub static TZCNT: [u8; 3] = [0xf3, 0x0f, 0xbc];
+
+/// Compare low double-precision floating-point values in xmm1 and xmm2/mem64
+/// and set the EFLAGS flags accordingly.
+pub static UCOMISD: [u8; 3] = [0x66, 0x0f, 0x2e];
+
+/// Compare low single-precision floating-point values in xmm1 and xmm2/mem32
+/// and set the EFLAGS flags accordingly.
+pub static UCOMISS: [u8; 2] = [0x0f, 0x2e];
+
+/// Raise invalid opcode instruction.
+pub static UNDEFINED2: [u8; 2] = [0x0f, 0x0b];
+
+/// imm{16,32} XOR r/m{16,32,64}, possibly sign-extended.
+pub static XOR_IMM: [u8; 1] = [0x81];
+
+/// r/m{16,32,64} XOR sign-extended imm8.
+pub static XOR_IMM8_SIGN_EXTEND: [u8; 1] = [0x83];
+
+/// r/m{16,32,64} XOR register of the same size.
+pub static XOR: [u8; 1] = [0x31];
+
+/// r/m8 XOR r8.
+pub static XORB: [u8; 1] = [0x30];
+
+/// Bitwise logical XOR of packed double-precision floating-point values.
+pub static XORPD: [u8; 3] = [0x66, 0x0f, 0x57];
+
+/// Bitwise logical XOR of packed single-precision floating-point values.
+pub static XORPS: [u8; 2] = [0x0f, 0x57];
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 13d9dcf833..ce0999c04c 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -10,6 +10,8 @@ use crate::cdsl::regs::IsaRegs;
 use crate::cdsl::settings::SettingGroup;
 use crate::shared::Definitions as SharedDefinitions;
 
+use crate::isa::x86::opcodes;
+
 /// Helper data structure to create recipes and template recipes.
 /// It contains all the recipes and recipe templates that might be used in the encodings crate of
 /// this same directory.
@@ -186,7 +188,7 @@ pub struct Template<'builder> {
     /// Value of the RRR bits (between 0 and 0b111).
     rrr_bits: u16,
     /// Opcode bytes.
-    op_bytes: Vec,
+    op_bytes: &'static [u8],
 }
 
 impl<'builder> Template<'builder> {
@@ -204,7 +206,7 @@ impl<'builder> Template<'builder> {
             rex: false,
             w_bit: 0,
             rrr_bits: 0,
-            op_bytes: Vec::new(),
+            op_bytes: &opcodes::EMPTY,
         }
     }
 
@@ -226,7 +228,7 @@ impl<'builder> Template<'builder> {
     }
 
     // Copy setters.
-    pub fn opcodes(&self, op_bytes: Vec) -> Self {
+    pub fn opcodes(&self, op_bytes: &'static [u8]) -> Self {
         assert!(!op_bytes.is_empty());
         let mut copy = self.clone();
         copy.op_bytes = op_bytes;

From 40f6d3b753719161545bf89457ca3f7ce24291ba Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Tue, 24 Sep 2019 18:19:18 +0200
Subject: [PATCH 2770/3084] Set speed-and-size optimization level for
 legalize-br-table-bb.clif test case.

---
 cranelift/filetests/filetests/isa/x86/legalize-br-table-bb.clif | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-table-bb.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-table-bb.clif
index 69e4240764..78d66d43af 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-table-bb.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-table-bb.clif
@@ -1,4 +1,5 @@
 test compile
+set opt_level=speed_and_size
 target x86_64
 feature "basic-blocks"
 ; regex: V=v\d+

From a1f6457e8af856ff0e39735bbd2544cc3c3edaf2 Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Thu, 26 Sep 2019 12:00:03 -0400
Subject: [PATCH 2771/3084] Allow building without std (#1069)

Closes https://github.com/CraneStation/cranelift/issues/1067
---
 cranelift/codegen/Cargo.toml                    |  6 +++---
 cranelift/codegen/src/binemit/memorysink.rs     |  2 +-
 cranelift/codegen/src/binemit/stackmap.rs       |  4 ++--
 cranelift/codegen/src/divconst_magic_numbers.rs |  2 --
 cranelift/codegen/src/ir/constant.rs            |  3 ++-
 cranelift/codegen/src/ir/dfg.rs                 |  2 +-
 cranelift/codegen/src/lib.rs                    |  2 +-
 cranelift/codegen/src/scoped_hash_map.rs        | 13 ++++++++++++-
 cranelift/codegen/src/value_label.rs            | 11 ++++++-----
 cranelift/codegen/src/write.rs                  |  2 +-
 cranelift/entity/src/map.rs                     |  2 +-
 cranelift/frontend/Cargo.toml                   |  4 ++--
 cranelift/frontend/src/frontend.rs              |  2 +-
 cranelift/frontend/src/lib.rs                   |  4 ++--
 cranelift/module/Cargo.toml                     |  4 ++--
 cranelift/module/src/lib.rs                     |  2 +-
 cranelift/wasm/Cargo.toml                       |  4 ++--
 cranelift/wasm/src/lib.rs                       |  5 +++--
 18 files changed, 43 insertions(+), 31 deletions(-)

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index eb6a1094c8..e38f06d608 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -18,7 +18,7 @@ cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-f
 cranelift-bforest = { path = "../cranelift-bforest", version = "0.44.0", default-features = false }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
-hashmap_core = { version = "0.1.9", optional = true }
+hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
@@ -43,10 +43,10 @@ std = [
     "cranelift-codegen-meta/std"
 ]
 
-# The "core" features enables use of "hashmap_core" since core doesn't have
+# The "core" features enables use of "hashbrown" since core doesn't have
 # a HashMap implementation, and a workaround for Cargo #4866.
 core = [
-    "hashmap_core",
+    "hashbrown",
     "cranelift-codegen-meta/core"
 ]
 
diff --git a/cranelift/codegen/src/binemit/memorysink.rs b/cranelift/codegen/src/binemit/memorysink.rs
index ffb8b44b49..ac3e82bea2 100644
--- a/cranelift/codegen/src/binemit/memorysink.rs
+++ b/cranelift/codegen/src/binemit/memorysink.rs
@@ -99,7 +99,7 @@ impl<'a> MemoryCodeSink<'a> {
         unsafe {
             #[cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
             write_unaligned(self.data.offset(self.offset) as *mut T, x);
-            self.offset += std::mem::size_of::() as isize;
+            self.offset += core::mem::size_of::() as isize;
         }
     }
 }
diff --git a/cranelift/codegen/src/binemit/stackmap.rs b/cranelift/codegen/src/binemit/stackmap.rs
index 4051b47e19..5d6844dcf3 100644
--- a/cranelift/codegen/src/binemit/stackmap.rs
+++ b/cranelift/codegen/src/binemit/stackmap.rs
@@ -4,7 +4,7 @@ use crate::isa::TargetIsa;
 use std::vec::Vec;
 
 type Num = u32;
-const NUM_BITS: usize = std::mem::size_of::() * 8;
+const NUM_BITS: usize = core::mem::size_of::() * 8;
 
 /// Wrapper class for longer bit vectors that cannot be represented by a single BitSet.
 #[derive(Clone, Debug)]
@@ -20,7 +20,7 @@ impl Stackmap {
         isa: &dyn TargetIsa,
     ) -> Self {
         let loc = &func.locations;
-        let mut live_ref_in_stack_slot = std::collections::HashSet::new();
+        let mut live_ref_in_stack_slot = crate::HashSet::new();
         // References can be in registers, and live registers values are pushed onto the stack before calls and traps.
         // TODO: Implement register maps. If a register containing a reference is spilled and reused after a safepoint,
         // it could contain a stale reference value if the garbage collector relocated the value.
diff --git a/cranelift/codegen/src/divconst_magic_numbers.rs b/cranelift/codegen/src/divconst_magic_numbers.rs
index b155496ab9..7b302e9b1a 100644
--- a/cranelift/codegen/src/divconst_magic_numbers.rs
+++ b/cranelift/codegen/src/divconst_magic_numbers.rs
@@ -1078,8 +1078,6 @@ mod tests {
                 d -= 1;
             }
         }
-
         assert_eq!(n_tests_done, 50_148_000);
     }
-
 }
diff --git a/cranelift/codegen/src/ir/constant.rs b/cranelift/codegen/src/ir/constant.rs
index c22380db55..9f21209c4c 100644
--- a/cranelift/codegen/src/ir/constant.rs
+++ b/cranelift/codegen/src/ir/constant.rs
@@ -9,8 +9,9 @@
 //! - bucketing constants by size.
 
 use crate::ir::Constant;
+use crate::HashMap;
 use cranelift_entity::EntityRef;
-use std::collections::{BTreeMap, HashMap};
+use std::collections::BTreeMap;
 use std::vec::Vec;
 
 /// This type describes the actual constant data.
diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index 5b3054b59d..e4ec9c6be8 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -13,12 +13,12 @@ use crate::ir::{
 use crate::isa::TargetIsa;
 use crate::packed_option::ReservedValue;
 use crate::write::write_operands;
+use crate::HashMap;
 use core::fmt;
 use core::iter;
 use core::mem;
 use core::ops::{Index, IndexMut};
 use core::u16;
-use std::collections::HashMap;
 use std::vec::Vec;
 
 /// A data flow graph defines all instructions and extended basic blocks in a function as well as
diff --git a/cranelift/codegen/src/lib.rs b/cranelift/codegen/src/lib.rs
index 73f2114561..0e61171d14 100644
--- a/cranelift/codegen/src/lib.rs
+++ b/cranelift/codegen/src/lib.rs
@@ -50,7 +50,7 @@ extern crate alloc as std;
 extern crate std;
 
 #[cfg(not(feature = "std"))]
-use hashmap_core::{map as hash_map, HashMap, HashSet};
+use hashbrown::{hash_map, HashMap, HashSet};
 #[cfg(feature = "std")]
 use std::collections::{hash_map, HashMap, HashSet};
 
diff --git a/cranelift/codegen/src/scoped_hash_map.rs b/cranelift/codegen/src/scoped_hash_map.rs
index fe22e26ce3..809d22132a 100644
--- a/cranelift/codegen/src/scoped_hash_map.rs
+++ b/cranelift/codegen/src/scoped_hash_map.rs
@@ -8,6 +8,11 @@ use crate::fx::FxHashMap;
 use core::hash::Hash;
 use core::mem;
 
+#[cfg(not(feature = "std"))]
+use crate::fx::FxHasher;
+#[cfg(not(feature = "std"))]
+type Hasher = core::hash::BuildHasherDefault;
+
 struct Val {
     value: V,
     next_key: Option,
@@ -16,7 +21,10 @@ struct Val {
 
 /// A view into an occupied entry in a `ScopedHashMap`. It is part of the `Entry` enum.
 pub struct OccupiedEntry<'a, K: 'a, V: 'a> {
+    #[cfg(feature = "std")]
     entry: super::hash_map::OccupiedEntry<'a, K, Val>,
+    #[cfg(not(feature = "std"))]
+    entry: super::hash_map::OccupiedEntry<'a, K, Val, Hasher>,
 }
 
 impl<'a, K, V> OccupiedEntry<'a, K, V> {
@@ -28,12 +36,15 @@ impl<'a, K, V> OccupiedEntry<'a, K, V> {
 
 /// A view into a vacant entry in a `ScopedHashMap`. It is part of the `Entry` enum.
 pub struct VacantEntry<'a, K: 'a, V: 'a> {
+    #[cfg(feature = "std")]
     entry: super::hash_map::VacantEntry<'a, K, Val>,
+    #[cfg(not(feature = "std"))]
+    entry: super::hash_map::VacantEntry<'a, K, Val, Hasher>,
     next_key: Option,
     depth: usize,
 }
 
-impl<'a, K, V> VacantEntry<'a, K, V> {
+impl<'a, K: Hash, V> VacantEntry<'a, K, V> {
     /// Sets the value of the entry with the `VacantEntry`'s key.
     pub fn insert(self, value: V) {
         self.entry.insert(Val {
diff --git a/cranelift/codegen/src/value_label.rs b/cranelift/codegen/src/value_label.rs
index d3d1e0672c..b6ad71faf8 100644
--- a/cranelift/codegen/src/value_label.rs
+++ b/cranelift/codegen/src/value_label.rs
@@ -1,11 +1,12 @@
 use crate::ir::{Function, SourceLoc, Value, ValueLabel, ValueLabelAssignments, ValueLoc};
 use crate::isa::TargetIsa;
 use crate::regalloc::{Context, RegDiversions};
-use std::cmp::Ordering;
-use std::collections::{BTreeMap, HashMap};
-use std::iter::Iterator;
-use std::ops::Bound::*;
-use std::ops::Deref;
+use crate::HashMap;
+use core::cmp::Ordering;
+use core::iter::Iterator;
+use core::ops::Bound::*;
+use core::ops::Deref;
+use std::collections::BTreeMap;
 use std::vec::Vec;
 
 #[cfg(feature = "enable-serde")]
diff --git a/cranelift/codegen/src/write.rs b/cranelift/codegen/src/write.rs
index 8d974c1ec5..8f5f81d23e 100644
--- a/cranelift/codegen/src/write.rs
+++ b/cranelift/codegen/src/write.rs
@@ -13,8 +13,8 @@ use crate::ir::{
 use crate::isa::{RegInfo, TargetIsa};
 use crate::packed_option::ReservedValue;
 use crate::value_label::ValueLabelsRanges;
+use crate::HashSet;
 use core::fmt::{self, Write};
-use std::collections::HashSet;
 use std::string::String;
 use std::vec::Vec;
 
diff --git a/cranelift/entity/src/map.rs b/cranelift/entity/src/map.rs
index 77fe38ce92..884f812fb2 100644
--- a/cranelift/entity/src/map.rs
+++ b/cranelift/entity/src/map.rs
@@ -3,6 +3,7 @@
 use crate::iter::{Iter, IterMut};
 use crate::keys::Keys;
 use crate::EntityRef;
+use core::cmp::min;
 use core::marker::PhantomData;
 use core::ops::{Index, IndexMut};
 use core::slice;
@@ -12,7 +13,6 @@ use serde::{
     ser::{SerializeSeq, Serializer},
     Deserialize, Serialize,
 };
-use std::cmp::min;
 use std::vec::Vec;
 
 /// A mapping `K -> V` for densely indexed entity references.
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index e30bde657e..0f8aefe91d 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -14,13 +14,13 @@ edition = "2018"
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
-hashmap_core = { version = "0.1.9", optional = true }
+hashbrown = { version = "0.6", optional = true }
 smallvec = { version = "0.6.10" }
 
 [features]
 default = ["std"]
 std = ["cranelift-codegen/std"]
-core = ["hashmap_core", "cranelift-codegen/core"]
+core = ["hashbrown", "cranelift-codegen/core"]
 
 # Temporary feature that enforces basic block semantics.
 basic-blocks = ["cranelift-codegen/basic-blocks"]
diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index fbd2428e1c..6204071618 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -341,7 +341,7 @@ impl<'a> FunctionBuilder<'a> {
     /// This will not do anything unless `func.dfg.collect_debug_info` is called first.
     pub fn set_val_label(&mut self, val: Value, label: ValueLabel) {
         if let Some(values_labels) = self.func.dfg.values_labels.as_mut() {
-            use std::collections::hash_map::Entry;
+            use crate::hash_map::Entry;
 
             let start = ValueLabelStart {
                 from: self.srcloc,
diff --git a/cranelift/frontend/src/lib.rs b/cranelift/frontend/src/lib.rs
index eff5c655fd..5e53944901 100644
--- a/cranelift/frontend/src/lib.rs
+++ b/cranelift/frontend/src/lib.rs
@@ -191,9 +191,9 @@ extern crate alloc as std;
 extern crate std;
 
 #[cfg(not(feature = "std"))]
-use hashmap_core::HashMap;
+use hashbrown::{hash_map, HashMap};
 #[cfg(feature = "std")]
-use std::collections::HashMap;
+use std::collections::{hash_map, HashMap};
 
 pub use crate::frontend::{FunctionBuilder, FunctionBuilderContext};
 pub use crate::switch::Switch;
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 890c1a5778..2ce44cc32c 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -13,14 +13,14 @@ edition = "2018"
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
-hashmap_core = { version = "0.1.9", optional = true }
+hashbrown = { version = "0.6", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
 
 [features]
 default = ["std"]
 std = ["cranelift-codegen/std", "cranelift-entity/std"]
-core = ["hashmap_core", "cranelift-codegen/core"]
+core = ["hashbrown", "cranelift-codegen/core"]
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/module/src/lib.rs b/cranelift/module/src/lib.rs
index 7cb9c10f22..0122171e9a 100644
--- a/cranelift/module/src/lib.rs
+++ b/cranelift/module/src/lib.rs
@@ -28,7 +28,7 @@ extern crate alloc as std;
 extern crate std;
 
 #[cfg(not(feature = "std"))]
-use hashmap_core::{map as hash_map, HashMap};
+use hashbrown::{hash_map, HashMap};
 #[cfg(feature = "std")]
 use std::collections::{hash_map, HashMap};
 
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index fd7f51608d..5e2a9e1cbe 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -15,7 +15,7 @@ wasmparser = { version = "0.39.1", default-features = false }
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.44.0", default-features = false }
-hashmap_core = { version = "0.1.9", optional = true }
+hashbrown = { version = "0.6", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
@@ -28,7 +28,7 @@ target-lexicon = "0.8.1"
 [features]
 default = ["std"]
 std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std"]
-core = ["hashmap_core", "cranelift-codegen/core", "cranelift-frontend/core", "wasmparser/core"]
+core = ["hashbrown", "cranelift-codegen/core", "cranelift-frontend/core", "wasmparser/core"]
 enable-serde = ["serde"]
 
 # Temporary feature that enforces basic block semantics.
diff --git a/cranelift/wasm/src/lib.rs b/cranelift/wasm/src/lib.rs
index d47feb631d..297c9df5f0 100644
--- a/cranelift/wasm/src/lib.rs
+++ b/cranelift/wasm/src/lib.rs
@@ -37,9 +37,10 @@ extern crate alloc as std;
 extern crate std;
 
 #[cfg(not(feature = "std"))]
-use hashmap_core::{
+use hashbrown::{
+    hash_map,
     hash_map::Entry::{Occupied, Vacant},
-    map as hash_map, HashMap,
+    HashMap,
 };
 #[cfg(feature = "std")]
 use std::collections::{

From e45ef24d3bf5bccaf456a15cea1140e10595bdee Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 25 Sep 2019 12:40:27 -0700
Subject: [PATCH 2772/3084] Convert SIMD load and store to their respective
 CLIF instructions

---
 cranelift/wasm/src/code_translator.rs | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 0bf7f4b0b4..3c004e363c 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -500,6 +500,11 @@ pub fn translate_operator(
         } => {
             translate_load(*offset, ir::Opcode::Load, F64, builder, state, environ)?;
         }
+        Operator::V128Load {
+            memarg: MemoryImmediate { flags: _, offset },
+        } => {
+            translate_load(*offset, ir::Opcode::Load, I8X16, builder, state, environ)?;
+        }
         /****************************** Store instructions ***********************************
          * Wasm specifies an integer alignment flag but we drop it in Cranelift.
          * The memory base address is provided by the environment.
@@ -539,6 +544,11 @@ pub fn translate_operator(
         } => {
             translate_store(*offset, ir::Opcode::Istore32, builder, state, environ)?;
         }
+        Operator::V128Store {
+            memarg: MemoryImmediate { flags: _, offset },
+        } => {
+            translate_store(*offset, ir::Opcode::Store, builder, state, environ)?;
+        }
         /****************************** Nullary Operators ************************************/
         Operator::I32Const { value } => state.push1(builder.ins().iconst(I32, i64::from(*value))),
         Operator::I64Const { value } => state.push1(builder.ins().iconst(I64, *value)),
@@ -990,9 +1000,7 @@ pub fn translate_operator(
             let (a, b) = state.pop2();
             state.push1(builder.ins().iadd(a, b))
         }
-        Operator::V128Load { .. }
-        | Operator::V128Store { .. }
-        | Operator::I8x16Eq
+        Operator::I8x16Eq
         | Operator::I8x16Ne
         | Operator::I8x16LtS
         | Operator::I8x16LtU

From 9f3c5b967e77156ed2acd4848b893050790b3bea Mon Sep 17 00:00:00 2001
From: Ujjwal Sharma 
Date: Thu, 26 Sep 2019 03:38:38 +0530
Subject: [PATCH 2773/3084] [codegen] add documentation for overflow

Add documentation to the icmp instruction text for both signed and
unsigned overflow, making it very clear why unsigned overflow is
complicated and where to find it.
---
 .../codegen/meta/src/shared/instructions.rs   | 27 ++++++++++---------
 1 file changed, 15 insertions(+), 12 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 679ee1a2c1..1f0e629a2e 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1586,8 +1586,6 @@ pub(crate) fn define(
     let x = &operand("x", Int);
     let y = &operand("y", Int);
 
-    // TODO(ryzokuken): Add documentation for unsigned overflow.
-    // TODO(ryzokuken): Add documentation for signed overflow.
     ig.push(
         Inst::new(
             "icmp",
@@ -1597,16 +1595,21 @@ pub(crate) fn define(
         The condition code determines if the operands are interpreted as signed
         or unsigned integers.
 
-        ====== ======== =========
-        Signed Unsigned Condition
-        ====== ======== =========
-        eq     eq       Equal
-        ne     ne       Not equal
-        slt    ult      Less than
-        sge    uge      Greater than or equal
-        sgt    ugt      Greater than
-        sle    ule      Less than or equal
-        ====== ======== =========
+        | Signed | Unsigned | Condition             |
+        |--------|----------|-----------------------|
+        | eq     | eq       | Equal                 |
+        | ne     | ne       | Not equal             |
+        | slt    | ult      | Less than             |
+        | sge    | uge      | Greater than or equal |
+        | sgt    | ugt      | Greater than          |
+        | sle    | ule      | Less than or equal    |
+        | of     | *        | Overflow              |
+        | nof    | *        | No Overflow           |
+
+        \* The unsigned version of overflow conditions have ISA-specific
+        semantics and thus have been kept as methods on the TargetIsa trait as
+        [unsigned_add_overflow_condition][isa::TargetIsa::unsigned_add_overflow_condition] and
+        [unsigned_sub_overflow_condition][isa::TargetIsa::unsigned_sub_overflow_condition].
 
         When this instruction compares integer vectors, it returns a boolean
         vector of lane-wise comparisons.

From c3d01756a33d9a8276c65b39fd432ffb09495f28 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 25 Sep 2019 14:09:11 +0200
Subject: [PATCH 2774/3084] Baldrdash: uses ECX for the WasmTableCallSigReg on
 x86 32-bits;

---
 cranelift/codegen/src/isa/x86/abi.rs               |  9 ++++++++-
 .../filetests/isa/x86/baldrdash-table-sig-reg.clif | 14 ++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/baldrdash-table-sig-reg.clif

diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 6252a3fb77..6bd766724e 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -120,7 +120,14 @@ impl ArgAssigner for Args {
                     .into();
                 }
                 // This is SpiderMonkey's `WasmTableCallSigReg`.
-                ArgumentPurpose::SignatureId => return ArgumentLoc::Reg(RU::r10 as RegUnit).into(),
+                ArgumentPurpose::SignatureId => {
+                    return ArgumentLoc::Reg(if self.pointer_bits == 64 {
+                        RU::r10
+                    } else {
+                        RU::rcx
+                    } as RegUnit)
+                    .into()
+                }
                 _ => {}
             }
         }
diff --git a/cranelift/filetests/filetests/isa/x86/baldrdash-table-sig-reg.clif b/cranelift/filetests/filetests/isa/x86/baldrdash-table-sig-reg.clif
new file mode 100644
index 0000000000..73aaaeb283
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/baldrdash-table-sig-reg.clif
@@ -0,0 +1,14 @@
+test compile
+set probestack_enabled=false
+target i686
+
+function u0:0(i32 vmctx) baldrdash_system_v {
+    sig0 = (i32 vmctx, i32 sigid) baldrdash_system_v
+
+ebb0(v0: i32):
+    v2 = iconst.i32 0
+    v8 = iconst.i32 0
+    v9 = iconst.i32 0
+    call_indirect sig0, v8(v9, v2)
+    trap user0
+}

From 79784dfaf66f0427ee8258020ba6546e00c93290 Mon Sep 17 00:00:00 2001
From: John Gallagher 
Date: Mon, 30 Sep 2019 09:38:03 -0400
Subject: [PATCH 2775/3084] Change signature of all() function

---
 cranelift/codegen/meta/src/isa/mod.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/mod.rs b/cranelift/codegen/meta/src/isa/mod.rs
index 6c5ba13888..3b90f7fe13 100644
--- a/cranelift/codegen/meta/src/isa/mod.rs
+++ b/cranelift/codegen/meta/src/isa/mod.rs
@@ -38,8 +38,8 @@ impl Isa {
     }
 
     /// Returns all supported isa targets.
-    pub fn all() -> [Isa; 4] {
-        [Isa::Riscv, Isa::X86, Isa::Arm32, Isa::Arm64]
+    pub fn all() -> &'static [Isa] {
+        &[Isa::Riscv, Isa::X86, Isa::Arm32, Isa::Arm64]
     }
 }
 

From 4e3cb25983552a6d18514516d0be5b08b03e77d5 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 13 Sep 2019 18:58:50 +0200
Subject: [PATCH 2776/3084] Use a sorted array for (Ebb, Inst) interval again
 (fixes #1084);

---
 cranelift/codegen/src/regalloc/coalescing.rs  |  30 +-
 cranelift/codegen/src/regalloc/coloring.rs    |  24 +-
 .../src/regalloc/live_value_tracker.rs        |   2 +-
 cranelift/codegen/src/regalloc/liveness.rs    |  29 +-
 cranelift/codegen/src/regalloc/liverange.rs   | 342 +++++++++---------
 cranelift/codegen/src/regalloc/spilling.rs    |   6 +-
 cranelift/codegen/src/value_label.rs          |   8 +-
 cranelift/codegen/src/verifier/cssa.rs        |   7 +-
 cranelift/codegen/src/verifier/liveness.rs    |   6 +-
 cranelift/codegen/src/verifier/locations.rs   |   8 +-
 10 files changed, 212 insertions(+), 250 deletions(-)

diff --git a/cranelift/codegen/src/regalloc/coalescing.rs b/cranelift/codegen/src/regalloc/coalescing.rs
index 8a1f1d81bb..b27a105d65 100644
--- a/cranelift/codegen/src/regalloc/coalescing.rs
+++ b/cranelift/codegen/src/regalloc/coalescing.rs
@@ -196,12 +196,7 @@ impl<'a> Context<'a> {
             // to be live at the use.
             for i in 0..num_params {
                 let param = self.func.dfg.ebb_params(ebb)[i];
-                if self.liveness[param].reaches_use(
-                    pred_inst,
-                    pred_ebb,
-                    self.liveness.forest(),
-                    &self.func.layout,
-                ) {
+                if self.liveness[param].reaches_use(pred_inst, pred_ebb, &self.func.layout) {
                     self.isolate_param(ebb, param);
                 }
             }
@@ -255,7 +250,7 @@ impl<'a> Context<'a> {
                 );
 
                 // The only other possibility is that `arg` is live-in to `ebb`.
-                lr.is_livein(ebb, self.liveness.forest(), &self.func.layout)
+                lr.is_livein(ebb, &self.func.layout)
             };
 
             if interference {
@@ -435,12 +430,7 @@ impl<'a> Context<'a> {
 
             // Check for interference between `parent` and `value`. Since `parent` dominates
             // `value`, we only have to check if it overlaps the definition.
-            if self.liveness[parent.value].overlaps_def(
-                node.def,
-                node.ebb,
-                self.liveness.forest(),
-                &self.func.layout,
-            ) {
+            if self.liveness[parent.value].overlaps_def(node.def, node.ebb, &self.func.layout) {
                 // The two values are interfering, so they can't be in the same virtual register.
                 debug!("-> interference: {} overlaps def of {}", parent, value);
                 return false;
@@ -626,12 +616,7 @@ impl<'a> Context<'a> {
                 // Check if the parent value interferes with the virtual copy.
                 let inst = node.def.unwrap_inst();
                 if node.set_id != parent.set_id
-                    && self.liveness[parent.value].reaches_use(
-                        inst,
-                        node.ebb,
-                        self.liveness.forest(),
-                        &self.func.layout,
-                    )
+                    && self.liveness[parent.value].reaches_use(inst, node.ebb, &self.func.layout)
                 {
                     debug!(
                         " - interference: {} overlaps vcopy at {}:{}",
@@ -655,12 +640,7 @@ impl<'a> Context<'a> {
             // Both node and parent are values, so check for interference.
             debug_assert!(node.is_value() && parent.is_value());
             if node.set_id != parent.set_id
-                && self.liveness[parent.value].overlaps_def(
-                    node.def,
-                    node.ebb,
-                    self.liveness.forest(),
-                    &self.func.layout,
-                )
+                && self.liveness[parent.value].overlaps_def(node.def, node.ebb, &self.func.layout)
             {
                 // The two values are interfering.
                 debug!(" - interference: {} overlaps def of {}", parent, node.value);
diff --git a/cranelift/codegen/src/regalloc/coloring.rs b/cranelift/codegen/src/regalloc/coloring.rs
index 23a546a479..f7fb50129f 100644
--- a/cranelift/codegen/src/regalloc/coloring.rs
+++ b/cranelift/codegen/src/regalloc/coloring.rs
@@ -54,7 +54,7 @@ use crate::regalloc::affinity::Affinity;
 use crate::regalloc::diversion::RegDiversions;
 use crate::regalloc::live_value_tracker::{LiveValue, LiveValueTracker};
 use crate::regalloc::liveness::Liveness;
-use crate::regalloc::liverange::{LiveRange, LiveRangeForest};
+use crate::regalloc::liverange::LiveRange;
 use crate::regalloc::register_set::RegisterSet;
 use crate::regalloc::solver::{Solver, SolverError};
 use crate::timing;
@@ -461,7 +461,7 @@ impl<'a> Context<'a> {
                     "Can't handle EBB arguments: {}",
                     self.cur.display_inst(inst)
                 );
-                self.undivert_regs(|lr, _, _| !lr.is_local());
+                self.undivert_regs(|lr, _| !lr.is_local());
             }
         }
 
@@ -726,12 +726,7 @@ impl<'a> Context<'a> {
                         // the new variable as killed or live-through. Always special-case the
                         // pinned register as a through variable.
                         let layout = &self.cur.func.layout;
-                        if self.liveness[value].killed_at(
-                            inst,
-                            layout.pp_ebb(inst),
-                            self.liveness.forest(),
-                            layout,
-                        ) {
+                        if self.liveness[value].killed_at(inst, layout.pp_ebb(inst), layout) {
                             self.solver.add_killed_var(value, op.regclass, cur_reg);
                         } else {
                             self.solver.add_through_var(value, op.regclass, cur_reg);
@@ -760,7 +755,7 @@ impl<'a> Context<'a> {
         //
         // Values with a global live range that are not live in to `dest` could appear as branch
         // arguments, so they can't always be un-diverted.
-        self.undivert_regs(|lr, forest, layout| lr.is_livein(dest, forest, layout));
+        self.undivert_regs(|lr, layout| lr.is_livein(dest, layout));
 
         // Now handle the EBB arguments.
         let br_args = self.cur.func.dfg.inst_variable_args(inst);
@@ -830,14 +825,14 @@ impl<'a> Context<'a> {
     /// are reallocated to their global register assignments.
     fn undivert_regs(&mut self, mut pred: Pred)
     where
-        Pred: FnMut(&LiveRange, &LiveRangeForest, &Layout) -> bool,
+        Pred: FnMut(&LiveRange, &Layout) -> bool,
     {
         for (&value, rdiv) in self.divert.iter() {
             let lr = self
                 .liveness
                 .get(value)
                 .expect("Missing live range for diverted register");
-            if pred(lr, self.liveness.forest(), &self.cur.func.layout) {
+            if pred(lr, &self.cur.func.layout) {
                 if let Affinity::Reg(rci) = lr.affinity {
                     let rc = self.reginfo.rc(rci);
                     // Stack diversions should not be possible here. They only live transiently
@@ -1086,20 +1081,19 @@ impl<'a> Context<'a> {
 
         let inst = self.cur.current_inst().expect("Not on an instruction");
         let layout = &self.cur.func.layout;
-        let forest = self.liveness.forest();
         match self.cur.func.dfg.analyze_branch(inst) {
             NotABranch => false,
             SingleDest(ebb, _) => {
                 let lr = &self.liveness[value];
-                lr.is_livein(ebb, forest, layout)
+                lr.is_livein(ebb, layout)
             }
             Table(jt, ebb) => {
                 let lr = &self.liveness[value];
                 !lr.is_local()
-                    && (ebb.map_or(false, |ebb| lr.is_livein(ebb, forest, layout))
+                    && (ebb.map_or(false, |ebb| lr.is_livein(ebb, layout))
                         || self.cur.func.jump_tables[jt]
                             .iter()
-                            .any(|ebb| lr.is_livein(*ebb, forest, layout)))
+                            .any(|ebb| lr.is_livein(*ebb, layout)))
             }
         }
     }
diff --git a/cranelift/codegen/src/regalloc/live_value_tracker.rs b/cranelift/codegen/src/regalloc/live_value_tracker.rs
index adfe56c410..5e17bb3557 100644
--- a/cranelift/codegen/src/regalloc/live_value_tracker.rs
+++ b/cranelift/codegen/src/regalloc/live_value_tracker.rs
@@ -198,7 +198,7 @@ impl LiveValueTracker {
                     .expect("Immediate dominator value has no live range");
 
                 // Check if this value is live-in here.
-                if let Some(endpoint) = lr.livein_local_end(ebb, liveness.forest(), layout) {
+                if let Some(endpoint) = lr.livein_local_end(ebb, layout) {
                     self.live.push(value, endpoint, lr);
                 }
             }
diff --git a/cranelift/codegen/src/regalloc/liveness.rs b/cranelift/codegen/src/regalloc/liveness.rs
index 1ace6bf6d1..419e35bc98 100644
--- a/cranelift/codegen/src/regalloc/liveness.rs
+++ b/cranelift/codegen/src/regalloc/liveness.rs
@@ -181,7 +181,7 @@ use crate::ir::dfg::ValueDef;
 use crate::ir::{Ebb, Function, Inst, Layout, ProgramPoint, Value};
 use crate::isa::{EncInfo, OperandConstraint, TargetIsa};
 use crate::regalloc::affinity::Affinity;
-use crate::regalloc::liverange::{LiveRange, LiveRangeForest};
+use crate::regalloc::liverange::LiveRange;
 use crate::timing;
 use core::mem;
 use core::ops::Index;
@@ -249,14 +249,13 @@ fn extend_to_use(
     worklist: &mut Vec,
     func: &Function,
     cfg: &ControlFlowGraph,
-    forest: &mut LiveRangeForest,
 ) {
     // This is our scratch working space, and we'll leave it empty when we return.
     debug_assert!(worklist.is_empty());
 
     // Extend the range locally in `ebb`.
     // If there already was a live interval in that block, we're done.
-    if lr.extend_in_ebb(ebb, to, &func.layout, forest) {
+    if lr.extend_in_ebb(ebb, to, &func.layout) {
         worklist.push(ebb);
     }
 
@@ -277,7 +276,7 @@ fn extend_to_use(
             inst: branch,
         } in cfg.pred_iter(livein)
         {
-            if lr.extend_in_ebb(pred, branch, &func.layout, forest) {
+            if lr.extend_in_ebb(pred, branch, &func.layout) {
                 // This predecessor EBB also became live-in. We need to process it later.
                 worklist.push(pred);
             }
@@ -292,9 +291,6 @@ pub struct Liveness {
     /// The live ranges that have been computed so far.
     ranges: LiveRangeSet,
 
-    /// Memory pool for the live ranges.
-    forest: LiveRangeForest,
-
     /// Working space for the `extend_to_use` algorithm.
     /// This vector is always empty, except for inside that function.
     /// It lives here to avoid repeated allocation of scratch memory.
@@ -309,16 +305,10 @@ impl Liveness {
     pub fn new() -> Self {
         Self {
             ranges: LiveRangeSet::new(),
-            forest: LiveRangeForest::new(),
             worklist: Vec::new(),
         }
     }
 
-    /// Current forest storage.
-    pub fn forest(&self) -> &LiveRangeForest {
-        &self.forest
-    }
-
     /// Current live ranges.
     pub fn ranges(&self) -> &LiveRangeSet {
         &self.ranges
@@ -327,7 +317,6 @@ impl Liveness {
     /// Clear all data structures in this liveness analysis.
     pub fn clear(&mut self) {
         self.ranges.clear();
-        self.forest.clear();
         self.worklist.clear();
     }
 
@@ -376,7 +365,7 @@ impl Liveness {
     ) -> &mut Affinity {
         debug_assert_eq!(Some(ebb), layout.inst_ebb(user));
         let lr = self.ranges.get_mut(value).expect("Value has no live range");
-        let livein = lr.extend_in_ebb(ebb, user, layout, &mut self.forest);
+        let livein = lr.extend_in_ebb(ebb, user, layout);
         debug_assert!(!livein, "{} should already be live in {}", value, ebb);
         &mut lr.affinity
     }
@@ -431,15 +420,7 @@ impl Liveness {
                     let lr = get_or_create(&mut self.ranges, arg, isa, func, &encinfo);
 
                     // Extend the live range to reach this use.
-                    extend_to_use(
-                        lr,
-                        ebb,
-                        inst,
-                        &mut self.worklist,
-                        func,
-                        cfg,
-                        &mut self.forest,
-                    );
+                    extend_to_use(lr, ebb, inst, &mut self.worklist, func, cfg);
 
                     // Apply operand constraint, ignoring any variable arguments after the fixed
                     // operands described by `operand_constraints`. Variable arguments are either
diff --git a/cranelift/codegen/src/regalloc/liverange.rs b/cranelift/codegen/src/regalloc/liverange.rs
index fb8fcc5fff..1d30650fa1 100644
--- a/cranelift/codegen/src/regalloc/liverange.rs
+++ b/cranelift/codegen/src/regalloc/liverange.rs
@@ -63,11 +63,11 @@
 //!
 //! ## Current representation
 //!
-//! Our current implementation uses a B-tree map with the necessary interface for an efficient
-//! implementation of coalescing, implemented as a generic data-structure bforest::Map.
-//!
-//! A `BTreeMap` could have been used for the live-in intervals, but it doesn't provide
-//! the necessary API to make coalescing easy, nor does it optimize for our types' sizes.
+//! Our current implementation uses a sorted array of compressed intervals, represented by their
+//! boundaries (Ebb, Inst), sorted by Ebb. This is a simple data structure, enables coalescing of
+//! intervals easily, and shows some nice performance behavior. See
+//! https://github.com/CraneStation/cranelift/issues/1084 for benchmarks against using a
+//! bforest::Map.
 //!
 //! ## EBB ordering
 //!
@@ -107,13 +107,19 @@
 //! It is more complicated to work with, though, so it is probably not worth it. The performance
 //! benefits of switching to a numerical EBB order only appears if the binary search is doing
 //! EBB-EBB comparisons.
+//!
+//! A `BTreeMap` could have been used for the live-in intervals, but it doesn't provide
+//! the necessary API to make coalescing easy, nor does it optimize for our types' sizes.
+//!
+//! Even the specialized `bforest::Map` implementation is slower than a plain sorted
+//! array, see https://github.com/CraneStation/cranelift/issues/1084 for details.
 
-use crate::bforest;
 use crate::entity::SparseMapValue;
 use crate::ir::{Ebb, ExpandedProgramPoint, Inst, Layout, ProgramOrder, ProgramPoint, Value};
 use crate::regalloc::affinity::Affinity;
 use core::cmp::Ordering;
 use core::marker::PhantomData;
+use smallvec::SmallVec;
 
 /// Global live range of a single SSA value.
 ///
@@ -144,6 +150,12 @@ use core::marker::PhantomData;
 /// branch and jump instructions.
 pub type LiveRange = GenericLiveRange;
 
+// See comment of liveins below.
+pub struct Interval {
+    begin: Ebb,
+    end: Inst,
+}
+
 /// Generic live range implementation.
 ///
 /// The intended generic parameter is `PO=Layout`, but tests are simpler with a mock order.
@@ -167,29 +179,18 @@ pub struct GenericLiveRange {
 
     /// Additional live-in intervals sorted in program order.
     ///
-    /// This map is empty for most values which are only used in one EBB.
+    /// This vector is empty for most values which are only used in one EBB.
     ///
-    /// A map entry `ebb -> inst` means that the live range is live-in to `ebb`, continuing up to
+    /// An entry `ebb -> inst` means that the live range is live-in to `ebb`, continuing up to
     /// `inst` which may belong to a later EBB in the program order.
     ///
     /// The entries are non-overlapping, and none of them overlap the EBB where the value is
     /// defined.
-    liveins: bforest::Map,
+    liveins: SmallVec<[Interval; 2]>,
 
     po: PhantomData<*const PO>,
 }
 
-/// Forest of B-trees used for storing live ranges.
-pub type LiveRangeForest = bforest::MapForest;
-
-struct Cmp<'a, PO: ProgramOrder + 'a>(&'a PO);
-
-impl<'a, PO: ProgramOrder> bforest::Comparator for Cmp<'a, PO> {
-    fn cmp(&self, a: Ebb, b: Ebb) -> Ordering {
-        self.0.cmp(a, b)
-    }
-}
-
 /// A simple helper macro to make comparisons more natural to read.
 macro_rules! cmp {
     ($order:ident, $a:ident > $b:expr) => {
@@ -216,11 +217,26 @@ impl GenericLiveRange {
             affinity,
             def_begin: def,
             def_end: def,
-            liveins: bforest::Map::new(),
+            liveins: SmallVec::new(),
             po: PhantomData,
         }
     }
 
+    /// Finds an entry in the compressed set of live-in intervals that contains `ebb`, or return
+    /// the position where to insert such a new entry.
+    fn lookup_entry_containing_ebb(&self, ebb: Ebb, order: &PO) -> Result {
+        self.liveins
+            .binary_search_by(|interval| order.cmp(interval.begin, ebb))
+            .or_else(|n| {
+                // The previous interval's end might cover the searched ebb.
+                if n > 0 && cmp!(order, ebb <= self.liveins[n - 1].end) {
+                    Ok(n - 1)
+                } else {
+                    Err(n)
+                }
+            })
+    }
+
     /// Extend the local interval for `ebb` so it reaches `to` which must belong to `ebb`.
     /// Create a live-in interval if necessary.
     ///
@@ -232,83 +248,101 @@ impl GenericLiveRange {
     ///
     /// The return value can be used to detect if we just learned that the value is live-in to
     /// `ebb`. This can trigger recursive extensions in `ebb`'s CFG predecessor blocks.
-    pub fn extend_in_ebb(
-        &mut self,
-        ebb: Ebb,
-        to: Inst,
-        order: &PO,
-        forest: &mut bforest::MapForest,
-    ) -> bool {
+    pub fn extend_in_ebb(&mut self, ebb: Ebb, inst: Inst, order: &PO) -> bool {
         // First check if we're extending the def interval.
         //
-        // We're assuming here that `to` never precedes `def_begin` in the same EBB, but we can't
-        // check it without a method for getting `to`'s EBB.
-        if cmp!(order, ebb <= self.def_end) && cmp!(order, to >= self.def_begin) {
-            let to_pp = to.into();
+        // We're assuming here that `inst` never precedes `def_begin` in the same EBB, but we can't
+        // check it without a method for getting `inst`'s EBB.
+        if cmp!(order, ebb <= self.def_end) && cmp!(order, inst >= self.def_begin) {
+            let inst_pp = inst.into();
             debug_assert_ne!(
-                to_pp, self.def_begin,
+                inst_pp, self.def_begin,
                 "Can't use value in the defining instruction."
             );
-            if cmp!(order, to > self.def_end) {
-                self.def_end = to_pp;
+            if cmp!(order, inst > self.def_end) {
+                self.def_end = inst_pp;
             }
             return false;
         }
 
         // Now check if we're extending any of the existing live-in intervals.
-        let cmp = Cmp(order);
-        let mut c = self.liveins.cursor(forest, &cmp);
-        let first_time_livein;
-
-        if let Some(end) = c.goto(ebb) {
-            // There's an interval beginning at `ebb`. See if it extends.
-            first_time_livein = false;
-            if cmp!(order, end < to) {
-                *c.value_mut().unwrap() = to;
-            } else {
-                return first_time_livein;
-            }
-        } else if let Some((_, end)) = c.prev() {
-            // There's no interval beginning at `ebb`, but we could still be live-in at `ebb` with
-            // a coalesced interval that begins before and ends after.
-            if cmp!(order, end > ebb) {
-                // Yep, the previous interval overlaps `ebb`.
-                first_time_livein = false;
-                if cmp!(order, end < to) {
-                    *c.value_mut().unwrap() = to;
-                } else {
-                    return first_time_livein;
+        match self.lookup_entry_containing_ebb(ebb, order) {
+            Ok(n) => {
+                // We found one interval and might need to extend it.
+                if cmp!(order, inst <= self.liveins[n].end) {
+                    // Both interval parts are already included in a compressed interval.
+                    return false;
                 }
-            } else {
-                first_time_livein = true;
-                // The current interval does not overlap `ebb`, but it may still be possible to
-                // coalesce with it.
-                if order.is_ebb_gap(end, ebb) {
-                    *c.value_mut().unwrap() = to;
-                } else {
-                    c.insert(ebb, to);
+
+                // If the instruction at the end is the last instruction before the next block,
+                // coalesce the two intervals:
+                // [ival.begin; ival.end] + [next.begin; next.end] = [ival.begin; next.end]
+                if let Some(next) = &self.liveins.get(n + 1) {
+                    if order.is_ebb_gap(inst, next.begin) {
+                        // At this point we can choose to remove the current interval or the next
+                        // one; remove the next one to avoid one memory move.
+                        let next_end = next.end;
+                        debug_assert!(cmp!(order, next_end > self.liveins[n].end));
+                        self.liveins[n].end = next_end;
+                        self.liveins.remove(n + 1);
+                        return false;
+                    }
                 }
+
+                // We can't coalesce, just extend the interval.
+                self.liveins[n].end = inst;
+                false
             }
-        } else {
-            // There is no existing interval before `ebb`.
-            first_time_livein = true;
-            c.insert(ebb, to);
-        }
 
-        // Now `c` is left pointing at an interval that ends in `to`.
-        debug_assert_eq!(c.value(), Some(to));
+            Err(n) => {
+                // No interval was found containing the current EBB: we need to insert a new one,
+                // unless there's a coalescing opportunity with the previous or next one.
+                let coalesce_next = self
+                    .liveins
+                    .get(n)
+                    .filter(|next| order.is_ebb_gap(inst, next.begin))
+                    .is_some();
+                let coalesce_prev = self
+                    .liveins
+                    .get(n.wrapping_sub(1))
+                    .filter(|prev| order.is_ebb_gap(prev.end, ebb))
+                    .is_some();
 
-        // See if it can be coalesced with the following interval.
-        if let Some((next_ebb, next_end)) = c.next() {
-            if order.is_ebb_gap(to, next_ebb) {
-                // Remove this interval and extend the previous end point to `next_end`.
-                c.remove();
-                c.prev();
-                *c.value_mut().unwrap() = next_end;
+                match (coalesce_prev, coalesce_next) {
+                    // The new interval is the missing hole between prev and next: we can merge
+                    // them all together.
+                    (true, true) => {
+                        let prev_end = self.liveins[n - 1].end;
+                        debug_assert!(cmp!(order, prev_end <= self.liveins[n].end));
+                        self.liveins[n - 1].end = self.liveins[n].end;
+                        self.liveins.remove(n);
+                    }
+
+                    // Coalesce only with the previous or next one.
+                    (true, false) => {
+                        debug_assert!(cmp!(order, inst >= self.liveins[n - 1].end));
+                        self.liveins[n - 1].end = inst;
+                    }
+                    (false, true) => {
+                        debug_assert!(cmp!(order, ebb <= self.liveins[n].begin));
+                        self.liveins[n].begin = ebb;
+                    }
+
+                    (false, false) => {
+                        // No coalescing opportunity, we have to insert.
+                        self.liveins.insert(
+                            n,
+                            Interval {
+                                begin: ebb,
+                                end: inst,
+                            },
+                        );
+                    }
+                }
+
+                true
             }
         }
-
-        first_time_livein
     }
 
     /// Is this the live range of a dead value?
@@ -359,43 +393,39 @@ impl GenericLiveRange {
     /// If the live range is live through all of `ebb`, the terminator of `ebb` is a correct
     /// answer, but it is also possible that an even later program point is returned. So don't
     /// depend on the returned `Inst` to belong to `ebb`.
-    pub fn livein_local_end(&self, ebb: Ebb, forest: &LiveRangeForest, order: &PO) -> Option {
-        let cmp = Cmp(order);
-        self.liveins
-            .get_or_less(ebb, forest, &cmp)
-            .and_then(|(_, inst)| {
-                // We have an entry that ends at `inst`.
-                if cmp!(order, inst > ebb) {
-                    Some(inst)
+    pub fn livein_local_end(&self, ebb: Ebb, order: &PO) -> Option {
+        self.lookup_entry_containing_ebb(ebb, order)
+            .and_then(|i| {
+                let inst = self.liveins[i].end;
+                if cmp!(order, ebb < inst) {
+                    Ok(inst)
                 } else {
-                    None
+                    // Can be any error type, really, since it's discarded by ok().
+                    Err(i)
                 }
             })
+            .ok()
     }
 
     /// Is this value live-in to `ebb`?
     ///
     /// An EBB argument is not considered to be live in.
-    pub fn is_livein(&self, ebb: Ebb, forest: &LiveRangeForest, order: &PO) -> bool {
-        self.livein_local_end(ebb, forest, order).is_some()
+    pub fn is_livein(&self, ebb: Ebb, order: &PO) -> bool {
+        self.livein_local_end(ebb, order).is_some()
     }
 
     /// Get all the live-in intervals.
     ///
     /// Note that the intervals are stored in a compressed form so each entry may span multiple
     /// EBBs where the value is live in.
-    pub fn liveins<'a>(&'a self, forest: &'a LiveRangeForest) -> bforest::MapIter<'a, Ebb, Inst> {
-        self.liveins.iter(forest)
+    pub fn liveins<'a>(&'a self) -> impl Iterator + 'a {
+        self.liveins
+            .iter()
+            .map(|interval| (interval.begin, interval.end))
     }
 
     /// Check if this live range overlaps a definition in `ebb`.
-    pub fn overlaps_def(
-        &self,
-        def: ExpandedProgramPoint,
-        ebb: Ebb,
-        forest: &LiveRangeForest,
-        order: &PO,
-    ) -> bool {
+    pub fn overlaps_def(&self, def: ExpandedProgramPoint, ebb: Ebb, order: &PO) -> bool {
         // Two defs at the same program point always overlap, even if one is dead.
         if def == self.def_begin.into() {
             return true;
@@ -407,30 +437,29 @@ impl GenericLiveRange {
         }
 
         // Check for an overlap with a live-in range.
-        match self.livein_local_end(ebb, forest, order) {
+        match self.livein_local_end(ebb, order) {
             Some(inst) => cmp!(order, def < inst),
             None => false,
         }
     }
 
     /// Check if this live range reaches a use at `user` in `ebb`.
-    pub fn reaches_use(&self, user: Inst, ebb: Ebb, forest: &LiveRangeForest, order: &PO) -> bool {
+    pub fn reaches_use(&self, user: Inst, ebb: Ebb, order: &PO) -> bool {
         // Check for an overlap with the local range.
         if cmp!(order, user > self.def_begin) && cmp!(order, user <= self.def_end) {
             return true;
         }
 
         // Check for an overlap with a live-in range.
-        match self.livein_local_end(ebb, forest, order) {
+        match self.livein_local_end(ebb, order) {
             Some(inst) => cmp!(order, user <= inst),
             None => false,
         }
     }
 
     /// Check if this live range is killed at `user` in `ebb`.
-    pub fn killed_at(&self, user: Inst, ebb: Ebb, forest: &LiveRangeForest, order: &PO) -> bool {
-        self.def_local_end() == user.into()
-            || self.livein_local_end(ebb, forest, order) == Some(user)
+    pub fn killed_at(&self, user: Inst, ebb: Ebb, order: &PO) -> bool {
+        self.def_local_end() == user.into() || self.livein_local_end(ebb, order) == Some(user)
     }
 }
 
@@ -443,8 +472,7 @@ impl SparseMapValue for GenericLiveRange {
 
 #[cfg(test)]
 mod tests {
-    use super::GenericLiveRange;
-    use crate::bforest;
+    use super::{GenericLiveRange, Interval};
     use crate::entity::EntityRef;
     use crate::ir::{Ebb, Inst, Value};
     use crate::ir::{ExpandedProgramPoint, ProgramOrder};
@@ -496,11 +524,7 @@ mod tests {
         }
 
         // Validate the live range invariants.
-        fn validate(
-            &self,
-            lr: &GenericLiveRange,
-            forest: &bforest::MapForest,
-        ) {
+        fn validate(&self, lr: &GenericLiveRange) {
             // The def interval must cover a single EBB.
             let def_ebb = self.pp_ebb(lr.def_begin);
             assert_eq!(def_ebb, self.pp_ebb(lr.def_end));
@@ -516,7 +540,10 @@ mod tests {
 
             // Check the live-in intervals.
             let mut prev_end = None;
-            for (begin, end) in lr.liveins.iter(forest) {
+            for Interval { begin, end } in lr.liveins.iter() {
+                let begin = *begin;
+                let end = *end;
+
                 assert_eq!(self.cmp(begin, end), Ordering::Less);
                 if let Some(e) = prev_end {
                     assert_eq!(self.cmp(e, begin), Ordering::Less);
@@ -545,18 +572,17 @@ mod tests {
         let i2 = Inst::new(2);
         let e2 = Ebb::new(2);
         let lr = GenericLiveRange::new(v0, i1.into(), Default::default());
-        let forest = &bforest::MapForest::new();
         assert!(lr.is_dead());
         assert!(lr.is_local());
         assert_eq!(lr.def(), i1.into());
         assert_eq!(lr.def_local_end(), i1.into());
-        assert_eq!(lr.livein_local_end(e2, forest, PO), None);
-        PO.validate(&lr, forest);
+        assert_eq!(lr.livein_local_end(e2, PO), None);
+        PO.validate(&lr);
 
         // A dead live range overlaps its own def program point.
-        assert!(lr.overlaps_def(i1.into(), e0, forest, PO));
-        assert!(!lr.overlaps_def(i2.into(), e0, forest, PO));
-        assert!(!lr.overlaps_def(e0.into(), e0, forest, PO));
+        assert!(lr.overlaps_def(i1.into(), e0, PO));
+        assert!(!lr.overlaps_def(i2.into(), e0, PO));
+        assert!(!lr.overlaps_def(e0.into(), e0, PO));
     }
 
     #[test]
@@ -564,14 +590,13 @@ mod tests {
         let v0 = Value::new(0);
         let e2 = Ebb::new(2);
         let lr = GenericLiveRange::new(v0, e2.into(), Default::default());
-        let forest = &bforest::MapForest::new();
         assert!(lr.is_dead());
         assert!(lr.is_local());
         assert_eq!(lr.def(), e2.into());
         assert_eq!(lr.def_local_end(), e2.into());
         // The def interval of an EBB argument does not count as live-in.
-        assert_eq!(lr.livein_local_end(e2, forest, PO), None);
-        PO.validate(&lr, forest);
+        assert_eq!(lr.livein_local_end(e2, PO), None);
+        PO.validate(&lr);
     }
 
     #[test]
@@ -582,18 +607,17 @@ mod tests {
         let i12 = Inst::new(12);
         let i13 = Inst::new(13);
         let mut lr = GenericLiveRange::new(v0, i11.into(), Default::default());
-        let forest = &mut bforest::MapForest::new();
 
-        assert_eq!(lr.extend_in_ebb(e10, i13, PO, forest), false);
-        PO.validate(&lr, forest);
+        assert_eq!(lr.extend_in_ebb(e10, i13, PO), false);
+        PO.validate(&lr);
         assert!(!lr.is_dead());
         assert!(lr.is_local());
         assert_eq!(lr.def(), i11.into());
         assert_eq!(lr.def_local_end(), i13.into());
 
         // Extending to an already covered inst should not change anything.
-        assert_eq!(lr.extend_in_ebb(e10, i12, PO, forest), false);
-        PO.validate(&lr, forest);
+        assert_eq!(lr.extend_in_ebb(e10, i12, PO), false);
+        PO.validate(&lr);
         assert_eq!(lr.def(), i11.into());
         assert_eq!(lr.def_local_end(), i13.into());
     }
@@ -606,26 +630,25 @@ mod tests {
         let i12 = Inst::new(12);
         let i13 = Inst::new(13);
         let mut lr = GenericLiveRange::new(v0, e10.into(), Default::default());
-        let forest = &mut bforest::MapForest::new();
 
         // Extending a dead EBB argument in its own block should not indicate that a live-in
         // interval was created.
-        assert_eq!(lr.extend_in_ebb(e10, i12, PO, forest), false);
-        PO.validate(&lr, forest);
+        assert_eq!(lr.extend_in_ebb(e10, i12, PO), false);
+        PO.validate(&lr);
         assert!(!lr.is_dead());
         assert!(lr.is_local());
         assert_eq!(lr.def(), e10.into());
         assert_eq!(lr.def_local_end(), i12.into());
 
         // Extending to an already covered inst should not change anything.
-        assert_eq!(lr.extend_in_ebb(e10, i11, PO, forest), false);
-        PO.validate(&lr, forest);
+        assert_eq!(lr.extend_in_ebb(e10, i11, PO), false);
+        PO.validate(&lr);
         assert_eq!(lr.def(), e10.into());
         assert_eq!(lr.def_local_end(), i12.into());
 
         // Extending further.
-        assert_eq!(lr.extend_in_ebb(e10, i13, PO, forest), false);
-        PO.validate(&lr, forest);
+        assert_eq!(lr.extend_in_ebb(e10, i13, PO), false);
+        PO.validate(&lr);
         assert_eq!(lr.def(), e10.into());
         assert_eq!(lr.def_local_end(), i13.into());
     }
@@ -641,23 +664,22 @@ mod tests {
         let i22 = Inst::new(22);
         let i23 = Inst::new(23);
         let mut lr = GenericLiveRange::new(v0, i11.into(), Default::default());
-        let forest = &mut bforest::MapForest::new();
 
-        assert_eq!(lr.extend_in_ebb(e10, i12, PO, forest), false);
+        assert_eq!(lr.extend_in_ebb(e10, i12, PO), false);
 
         // Adding a live-in interval.
-        assert_eq!(lr.extend_in_ebb(e20, i22, PO, forest), true);
-        PO.validate(&lr, forest);
-        assert_eq!(lr.livein_local_end(e20, forest, PO), Some(i22));
+        assert_eq!(lr.extend_in_ebb(e20, i22, PO), true);
+        PO.validate(&lr);
+        assert_eq!(lr.livein_local_end(e20, PO), Some(i22));
 
         // Non-extending the live-in.
-        assert_eq!(lr.extend_in_ebb(e20, i21, PO, forest), false);
-        assert_eq!(lr.livein_local_end(e20, forest, PO), Some(i22));
+        assert_eq!(lr.extend_in_ebb(e20, i21, PO), false);
+        assert_eq!(lr.livein_local_end(e20, PO), Some(i22));
 
         // Extending the existing live-in.
-        assert_eq!(lr.extend_in_ebb(e20, i23, PO, forest), false);
-        PO.validate(&lr, forest);
-        assert_eq!(lr.livein_local_end(e20, forest, PO), Some(i23));
+        assert_eq!(lr.extend_in_ebb(e20, i23, PO), false);
+        PO.validate(&lr);
+        assert_eq!(lr.livein_local_end(e20, PO), Some(i23));
     }
 
     #[test]
@@ -671,32 +693,28 @@ mod tests {
         let e40 = Ebb::new(40);
         let i41 = Inst::new(41);
         let mut lr = GenericLiveRange::new(v0, i11.into(), Default::default());
-        let forest = &mut bforest::MapForest::new();
 
-        assert_eq!(lr.extend_in_ebb(e30, i31, PO, forest), true);
-        assert_eq!(lr.liveins(forest).collect::>(), [(e30, i31)]);
+        assert_eq!(lr.extend_in_ebb(e30, i31, PO,), true);
+        assert_eq!(lr.liveins().collect::>(), [(e30, i31)]);
 
         // Coalesce to previous
-        assert_eq!(lr.extend_in_ebb(e40, i41, PO, forest), true);
-        assert_eq!(lr.liveins(forest).collect::>(), [(e30, i41)]);
+        assert_eq!(lr.extend_in_ebb(e40, i41, PO,), true);
+        assert_eq!(lr.liveins().collect::>(), [(e30, i41)]);
 
         // Coalesce to next
-        assert_eq!(lr.extend_in_ebb(e20, i21, PO, forest), true);
-        assert_eq!(lr.liveins(forest).collect::>(), [(e20, i41)]);
+        assert_eq!(lr.extend_in_ebb(e20, i21, PO,), true);
+        assert_eq!(lr.liveins().collect::>(), [(e20, i41)]);
 
         let mut lr = GenericLiveRange::new(v0, i11.into(), Default::default());
 
-        assert_eq!(lr.extend_in_ebb(e40, i41, PO, forest), true);
-        assert_eq!(lr.liveins(forest).collect::>(), [(e40, i41)]);
+        assert_eq!(lr.extend_in_ebb(e40, i41, PO,), true);
+        assert_eq!(lr.liveins().collect::>(), [(e40, i41)]);
 
-        assert_eq!(lr.extend_in_ebb(e20, i21, PO, forest), true);
-        assert_eq!(
-            lr.liveins(forest).collect::>(),
-            [(e20, i21), (e40, i41)]
-        );
+        assert_eq!(lr.extend_in_ebb(e20, i21, PO,), true);
+        assert_eq!(lr.liveins().collect::>(), [(e20, i21), (e40, i41)]);
 
         // Coalesce to previous and next
-        assert_eq!(lr.extend_in_ebb(e30, i31, PO, forest), true);
-        assert_eq!(lr.liveins(forest).collect::>(), [(e20, i41)]);
+        assert_eq!(lr.extend_in_ebb(e30, i31, PO,), true);
+        assert_eq!(lr.liveins().collect::>(), [(e20, i41)]);
     }
 }
diff --git a/cranelift/codegen/src/regalloc/spilling.rs b/cranelift/codegen/src/regalloc/spilling.rs
index 0012ad01d8..ace368d798 100644
--- a/cranelift/codegen/src/regalloc/spilling.rs
+++ b/cranelift/codegen/src/regalloc/spilling.rs
@@ -324,13 +324,11 @@ impl<'a> Context<'a> {
                     ConstraintKind::FixedReg(_) => reguse.fixed = true,
                     ConstraintKind::Tied(_) => {
                         // A tied operand must kill the used value.
-                        reguse.tied =
-                            !lr.killed_at(inst, ebb, self.liveness.forest(), &self.cur.func.layout);
+                        reguse.tied = !lr.killed_at(inst, ebb, &self.cur.func.layout);
                     }
                     ConstraintKind::FixedTied(_) => {
                         reguse.fixed = true;
-                        reguse.tied =
-                            !lr.killed_at(inst, ebb, self.liveness.forest(), &self.cur.func.layout);
+                        reguse.tied = !lr.killed_at(inst, ebb, &self.cur.func.layout);
                     }
                     ConstraintKind::Reg => {}
                 }
diff --git a/cranelift/codegen/src/value_label.rs b/cranelift/codegen/src/value_label.rs
index b6ad71faf8..3d5802f12e 100644
--- a/cranelift/codegen/src/value_label.rs
+++ b/cranelift/codegen/src/value_label.rs
@@ -98,7 +98,6 @@ where
     let encinfo = isa.encoding_info();
     let values_locations = &func.locations;
     let liveness_ranges = regalloc.liveness().ranges();
-    let liveness_forest = regalloc.liveness().forest();
 
     let mut ranges = HashMap::new();
     let mut add_range = |label, range: (u32, u32), loc: ValueLoc| {
@@ -127,10 +126,7 @@ where
             // Remove killed values.
             tracked_values.retain(|(x, label, start_offset, last_loc)| {
                 let range = liveness_ranges.get(*x);
-                if range
-                    .expect("value")
-                    .killed_at(inst, ebb, &liveness_forest, &func.layout)
-                {
+                if range.expect("value").killed_at(inst, ebb, &func.layout) {
                     add_range(*label, (*start_offset, end_offset), *last_loc);
                     return false;
                 }
@@ -177,7 +173,7 @@ where
                 // Ignore dead/inactive Values.
                 let range = liveness_ranges.get(*v);
                 match range {
-                    Some(r) => r.reaches_use(inst, ebb, &liveness_forest, &func.layout),
+                    Some(r) => r.reaches_use(inst, ebb, &func.layout),
                     None => false,
                 }
             });
diff --git a/cranelift/codegen/src/verifier/cssa.rs b/cranelift/codegen/src/verifier/cssa.rs
index 3878f096d1..85c6a638c2 100644
--- a/cranelift/codegen/src/verifier/cssa.rs
+++ b/cranelift/codegen/src/verifier/cssa.rs
@@ -118,12 +118,7 @@ impl<'a> CssaVerifier<'a> {
                     if self.preorder.dominates(prev_ebb, def_ebb)
                         && self.domtree.dominates(prev_def, def, &self.func.layout)
                     {
-                        if self.liveness[prev_val].overlaps_def(
-                            def,
-                            def_ebb,
-                            self.liveness.forest(),
-                            &self.func.layout,
-                        ) {
+                        if self.liveness[prev_val].overlaps_def(def, def_ebb, &self.func.layout) {
                             return fatal!(
                                 errors,
                                 val,
diff --git a/cranelift/codegen/src/verifier/liveness.rs b/cranelift/codegen/src/verifier/liveness.rs
index 37b59c154d..218bb60fdc 100644
--- a/cranelift/codegen/src/verifier/liveness.rs
+++ b/cranelift/codegen/src/verifier/liveness.rs
@@ -107,7 +107,7 @@ impl<'a> LivenessVerifier<'a> {
                     };
 
                     debug_assert!(self.func.layout.inst_ebb(inst).unwrap() == ebb);
-                    if !lr.reaches_use(inst, ebb, self.liveness.forest(), &self.func.layout) {
+                    if !lr.reaches_use(inst, ebb, &self.func.layout) {
                         return fatal!(errors, inst, "{} is not live at this use", val);
                     }
 
@@ -179,7 +179,7 @@ impl<'a> LivenessVerifier<'a> {
         }
 
         // Now check the live-in intervals against the CFG.
-        for (mut ebb, end) in lr.liveins(self.liveness.forest()) {
+        for (mut ebb, end) in lr.liveins() {
             if !l.is_ebb_inserted(ebb) {
                 return fatal!(
                     errors,
@@ -207,7 +207,7 @@ impl<'a> LivenessVerifier<'a> {
             loop {
                 // If `val` is live-in at `ebb`, it must be live at all the predecessors.
                 for BasicBlock { inst: pred, ebb } in self.cfg.pred_iter(ebb) {
-                    if !lr.reaches_use(pred, ebb, self.liveness.forest(), &self.func.layout) {
+                    if !lr.reaches_use(pred, ebb, &self.func.layout) {
                         return fatal!(
                             errors,
                             pred,
diff --git a/cranelift/codegen/src/verifier/locations.rs b/cranelift/codegen/src/verifier/locations.rs
index d32bb7e4bd..6a32cec0f6 100644
--- a/cranelift/codegen/src/verifier/locations.rs
+++ b/cranelift/codegen/src/verifier/locations.rs
@@ -334,10 +334,10 @@ impl<'a> LocationVerifier<'a> {
                     let lr = &liveness[value];
                     if is_after_branch && unique_predecessor {
                         // Forward diversions based on the targeted branch.
-                        if !lr.is_livein(ebb, liveness.forest(), &self.func.layout) {
+                        if !lr.is_livein(ebb, &self.func.layout) {
                             val_to_remove.push(value)
                         }
-                    } else if lr.is_livein(ebb, liveness.forest(), &self.func.layout) {
+                    } else if lr.is_livein(ebb, &self.func.layout) {
                         return fatal!(
                             errors,
                             inst,
@@ -359,7 +359,7 @@ impl<'a> LocationVerifier<'a> {
                 for (&value, d) in divert.iter() {
                     let lr = &liveness[value];
                     if let Some(ebb) = ebb {
-                        if lr.is_livein(ebb, liveness.forest(), &self.func.layout) {
+                        if lr.is_livein(ebb, &self.func.layout) {
                             return fatal!(
                                 errors,
                                 inst,
@@ -371,7 +371,7 @@ impl<'a> LocationVerifier<'a> {
                         }
                     }
                     for ebb in self.func.jump_tables[jt].iter() {
-                        if lr.is_livein(*ebb, liveness.forest(), &self.func.layout) {
+                        if lr.is_livein(*ebb, &self.func.layout) {
                             return fatal!(
                                 errors,
                                 inst,

From ca1df499a075159c6f4288834601cb62c6c739d5 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 17 Sep 2019 13:38:52 -0700
Subject: [PATCH 2777/3084] Add x86 encoding for isub

---
 .../codegen/meta/src/isa/x86/encodings.rs     |  6 +++
 cranelift/codegen/meta/src/isa/x86/opcodes.rs | 12 ++++++
 .../{iadd-simd.clif => simd-arithmetic.clif}  | 37 +++++++++++++++++++
 cranelift/wasm/src/code_translator.rs         |  8 ++--
 4 files changed, 59 insertions(+), 4 deletions(-)
 rename cranelift/filetests/filetests/isa/x86/{iadd-simd.clif => simd-arithmetic.clif} (57%)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 01f25d9d7f..b752b6bfae 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1940,6 +1940,12 @@ pub(crate) fn define(
         e.enc_32_64(iadd, rec_fa.opcodes(*opcodes));
     }
 
+    // SIMD integer subtraction
+    for (ty, opcodes) in &[(I8, &PSUBB), (I16, &PSUBW), (I32, &PSUBD), (I64, &PSUBQ)] {
+        let isub = isub.bind_vector_from_lane(ty.clone(), sse_vector_size);
+        e.enc_32_64(isub, rec_fa.opcodes(*opcodes));
+    }
+
     // SIMD icmp using PCMPEQ*
     let mut pcmpeq_mapping: HashMap)> = HashMap::new();
     pcmpeq_mapping.insert(8, (&PCMPEQB, None));
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index 12b60e532b..6ae740883c 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -294,6 +294,18 @@ pub static PSHUFB: [u8; 4] = [0x66, 0x0f, 0x38, 0x00];
 /// store the result in xmm1 (SSE2).
 pub static PSHUFD: [u8; 3] = [0x66, 0x0f, 0x70];
 
+/// Subtract packed byte integers in xmm2/m128 from packed byte integers in xmm1 (SSE2).
+pub static PSUBB: [u8; 3] = [0x66, 0x0f, 0xf8];
+
+/// Subtract packed word integers in xmm2/m128 from packed word integers in xmm1 (SSE2).
+pub static PSUBW: [u8; 3] = [0x66, 0x0f, 0xf9];
+
+/// Subtract packed doubleword integers in xmm2/m128 from doubleword byte integers in xmm1 (SSE2).
+pub static PSUBD: [u8; 3] = [0x66, 0x0f, 0xfa];
+
+/// Subtract packed quadword integers in xmm2/m128 from xmm1 (SSE2).
+pub static PSUBQ: [u8; 3] = [0x66, 0x0f, 0xfb];
+
 /// Push r{16,32,64}.
 pub static PUSH_REG: [u8; 1] = [0x50];
 
diff --git a/cranelift/filetests/filetests/isa/x86/iadd-simd.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
similarity index 57%
rename from cranelift/filetests/filetests/isa/x86/iadd-simd.clif
rename to cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
index 0884a97f55..64b6e854e7 100644
--- a/cranelift/filetests/filetests/isa/x86/iadd-simd.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
@@ -48,3 +48,40 @@ ebb0(v0: i64x2 [%xmm3], v1: i64x2 [%xmm4]):
 [-, %xmm3]  v2 = iadd v0, v1      ; bin: 66 0f d4 dc
             return v2
 }
+
+function %isub_i32x4() -> b1 {
+ebb0:
+[-, %xmm3]    v0 = vconst.i32x4 [1 1 1 1]
+[-, %xmm5]    v1 = vconst.i32x4 [1 2 3 4]
+[-, %xmm3]    v2 = isub v0, v1  ; bin: 66 0f fa dd
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 0
+
+    v5 = extractlane v2, 1
+    v6 = icmp_imm eq v5, 0xffffffff
+    ; TODO replace extractlanes with vector comparison
+
+    v7 = band v4, v6
+    return v7
+}
+
+; run
+
+function %isub_i64x2(i64x2, i64x2) -> i64x2 {
+ebb0(v0: i64x2 [%xmm0], v1: i64x2 [%xmm1]):
+[-, %xmm0]  v2 = isub v0, v1      ; bin: 66 0f fb c1
+    return v2
+}
+
+function %isub_i16x8(i16x8, i16x8) -> i16x8 {
+ebb0(v0: i16x8 [%xmm3], v1: i16x8 [%xmm4]):
+[-, %xmm3]  v2 = isub v0, v1      ; bin: 66 0f f9 dc
+    return v2
+}
+
+function %isub_i8x16(i8x16, i8x16) -> i8x16 {
+ebb0(v0: i8x16 [%xmm3], v1: i8x16 [%xmm4]):
+[-, %xmm3]  v2 = isub v0, v1      ; bin: 66 0f f8 dc
+    return v2
+}
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 3c004e363c..9dbd9d7463 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1000,6 +1000,10 @@ pub fn translate_operator(
             let (a, b) = state.pop2();
             state.push1(builder.ins().iadd(a, b))
         }
+        Operator::I8x16Sub | Operator::I16x8Sub | Operator::I32x4Sub | Operator::I64x2Sub => {
+            let (a, b) = state.pop2();
+            state.push1(builder.ins().isub(a, b))
+        }
         Operator::I8x16Eq
         | Operator::I8x16Ne
         | Operator::I8x16LtS
@@ -1055,7 +1059,6 @@ pub fn translate_operator(
         | Operator::I8x16ShrU
         | Operator::I8x16AddSaturateS
         | Operator::I8x16AddSaturateU
-        | Operator::I8x16Sub
         | Operator::I8x16SubSaturateS
         | Operator::I8x16SubSaturateU
         | Operator::I8x16Mul
@@ -1067,7 +1070,6 @@ pub fn translate_operator(
         | Operator::I16x8ShrU
         | Operator::I16x8AddSaturateS
         | Operator::I16x8AddSaturateU
-        | Operator::I16x8Sub
         | Operator::I16x8SubSaturateS
         | Operator::I16x8SubSaturateU
         | Operator::I16x8Mul
@@ -1077,7 +1079,6 @@ pub fn translate_operator(
         | Operator::I32x4Shl
         | Operator::I32x4ShrS
         | Operator::I32x4ShrU
-        | Operator::I32x4Sub
         | Operator::I32x4Mul
         | Operator::I64x2Neg
         | Operator::I64x2AnyTrue
@@ -1085,7 +1086,6 @@ pub fn translate_operator(
         | Operator::I64x2Shl
         | Operator::I64x2ShrS
         | Operator::I64x2ShrU
-        | Operator::I64x2Sub
         | Operator::F32x4Abs
         | Operator::F32x4Neg
         | Operator::F32x4Sqrt

From ba393afd4d01f87b265ae7e4cf0ad95e740a4592 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 18 Sep 2019 10:50:01 -0700
Subject: [PATCH 2778/3084] Add x86 legalization for SIMD ineg

---
 .../codegen/meta/src/isa/x86/legalize.rs      |  2 ++
 .../codegen/meta/src/shared/instructions.rs   | 11 ++++++
 cranelift/codegen/src/isa/x86/enc_tables.rs   | 25 +++++++++++++
 .../filetests/isa/x86/simd-arithmetic.clif    | 35 +++++++++++++++++++
 cranelift/wasm/src/code_translator.rs         |  8 ++---
 5 files changed, 77 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 57057ce6bb..61389729f5 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -36,6 +36,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let iadd = insts.by_name("iadd");
     let iconst = insts.by_name("iconst");
     let imul = insts.by_name("imul");
+    let ineg = insts.by_name("ineg");
     let insertlane = insts.by_name("insertlane");
     let isub = insts.by_name("isub");
     let popcnt = insts.by_name("popcnt");
@@ -385,6 +386,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     narrow.custom_legalize(shuffle, "convert_shuffle");
     narrow.custom_legalize(extractlane, "convert_extractlane");
     narrow.custom_legalize(insertlane, "convert_insertlane");
+    narrow.custom_legalize(ineg, "convert_ineg");
 
     narrow.build_and_add_to(&mut shared.transform_groups);
 }
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 1f0e629a2e..d8f5474c29 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1704,6 +1704,17 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    ig.push(
+        Inst::new(
+            "ineg",
+            r#"
+        Integer negation: `a := -x \pmod{2^B}`.
+        "#,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a]),
+    );
+
     ig.push(
         Inst::new(
             "imul",
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 94333116e6..93adca0c25 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -5,6 +5,7 @@ use crate::bitset::BitSet;
 use crate::cursor::{Cursor, FuncCursor};
 use crate::flowgraph::ControlFlowGraph;
 use crate::ir::condcodes::{FloatCC, IntCC};
+use crate::ir::immediates::V128Imm;
 use crate::ir::types::*;
 use crate::ir::{self, Function, Inst, InstBuilder};
 use crate::isa::constraints::*;
@@ -1090,3 +1091,27 @@ fn convert_insertlane(
         }
     }
 }
+
+/// For SIMD negation, convert an `ineg` to a `vconst + isub`.
+fn convert_ineg(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    _cfg: &mut ControlFlowGraph,
+    _isa: &dyn TargetIsa,
+) {
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+    pos.use_srcloc(inst);
+
+    if let ir::InstructionData::Unary {
+        opcode: ir::Opcode::Ineg,
+        arg,
+    } = pos.func.dfg[inst]
+    {
+        let value_type = pos.func.dfg.value_type(arg);
+        if value_type.is_vector() && value_type.lane_type().is_int() {
+            let zero_immediate = pos.func.dfg.constants.insert(V128Imm::from(0).to_vec());
+            let zero_value = pos.ins().vconst(value_type, zero_immediate); // this should be legalized to a PXOR
+            pos.func.dfg.replace(inst).isub(zero_value, arg);
+        }
+    }
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
index 64b6e854e7..8244177728 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
@@ -1,5 +1,6 @@
 test run
 test binemit
+test legalizer
 set enable_simd
 target x86_64 skylake
 
@@ -85,3 +86,37 @@ ebb0(v0: i8x16 [%xmm3], v1: i8x16 [%xmm4]):
 [-, %xmm3]  v2 = isub v0, v1      ; bin: 66 0f f8 dc
     return v2
 }
+
+function %ineg_i32x4() -> b1 {
+ebb0:
+    v0 = vconst.i32x4 [1 1 1 1]
+    v2 = ineg v0
+    ; check: v5 = vconst.i32x4 0x00
+    ; nextln: v2 = isub v5, v0
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, -1
+
+    return v4 ; bin: c3
+}
+; run
+
+function %ineg_legalized() {
+ebb0:
+    v0 = vconst.i8x16 0x00
+    v1 = ineg v0
+    ; check: v6 = vconst.i8x16 0x00
+    ; nextln: v1 = isub v6, v0
+
+    v2 = raw_bitcast.i16x8 v0
+    v3 = ineg v2
+    ; check: v7 = vconst.i16x8 0x00
+    ; nextln: v3 = isub v7, v2
+
+    v4 = raw_bitcast.i64x2 v0
+    v5 = ineg v4
+    ; check: v8 = vconst.i64x2 0x00
+    ; nextln: v5 = isub v8, v4
+
+    return ; bin: c3
+}
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 9dbd9d7463..ce2aa8dab7 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1004,6 +1004,10 @@ pub fn translate_operator(
             let (a, b) = state.pop2();
             state.push1(builder.ins().isub(a, b))
         }
+        Operator::I8x16Neg | Operator::I16x8Neg | Operator::I32x4Neg | Operator::I64x2Neg => {
+            let a = state.pop1();
+            state.push1(builder.ins().ineg(a))
+        }
         Operator::I8x16Eq
         | Operator::I8x16Ne
         | Operator::I8x16LtS
@@ -1051,7 +1055,6 @@ pub fn translate_operator(
         | Operator::V128Or
         | Operator::V128Xor
         | Operator::V128Bitselect
-        | Operator::I8x16Neg
         | Operator::I8x16AnyTrue
         | Operator::I8x16AllTrue
         | Operator::I8x16Shl
@@ -1062,7 +1065,6 @@ pub fn translate_operator(
         | Operator::I8x16SubSaturateS
         | Operator::I8x16SubSaturateU
         | Operator::I8x16Mul
-        | Operator::I16x8Neg
         | Operator::I16x8AnyTrue
         | Operator::I16x8AllTrue
         | Operator::I16x8Shl
@@ -1073,14 +1075,12 @@ pub fn translate_operator(
         | Operator::I16x8SubSaturateS
         | Operator::I16x8SubSaturateU
         | Operator::I16x8Mul
-        | Operator::I32x4Neg
         | Operator::I32x4AnyTrue
         | Operator::I32x4AllTrue
         | Operator::I32x4Shl
         | Operator::I32x4ShrS
         | Operator::I32x4ShrU
         | Operator::I32x4Mul
-        | Operator::I64x2Neg
         | Operator::I64x2AnyTrue
         | Operator::I64x2AllTrue
         | Operator::I64x2Shl

From c932f9b2b5745f3bc049306ddea49ec691a5c941 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 18 Sep 2019 13:45:26 -0700
Subject: [PATCH 2779/3084] Add parsing of 16-bit signed integers

---
 cranelift/codegen/src/ir/immediates.rs |  7 +++++++
 cranelift/reader/src/parser.rs         | 15 ++++++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index 408fe17cf3..eb6dfad34b 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -22,6 +22,12 @@ impl IntoBytes for u8 {
     }
 }
 
+impl IntoBytes for i16 {
+    fn into_bytes(self) -> Vec {
+        self.to_le_bytes().to_vec()
+    }
+}
+
 impl IntoBytes for i32 {
     fn into_bytes(self) -> Vec {
         self.to_le_bytes().to_vec()
@@ -429,6 +435,7 @@ impl FromIterator for V128Imm {
 }
 
 construct_uimm128_from_iterator_of!(u8, 16);
+construct_uimm128_from_iterator_of!(i16, 8);
 construct_uimm128_from_iterator_of!(i32, 4);
 construct_uimm128_from_iterator_of!(Ieee32, 4);
 construct_uimm128_from_iterator_of!(Imm64, 2);
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 711b4d3cc1..82437d7c6c 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -665,6 +665,19 @@ impl<'a> Parser<'a> {
         }
     }
 
+    // Match and consume a signed 16-bit immediate.
+    fn match_imm16(&mut self, err_msg: &str) -> ParseResult {
+        if let Some(Token::Integer(text)) = self.token() {
+            self.consume();
+            // Lexer just gives us raw text that looks like an integer.
+            // Parse it as a i16 to check for overflow and other issues.
+            text.parse()
+                .map_err(|_| self.error("expected i16 decimal immediate"))
+        } else {
+            err!(self.loc, err_msg)
+        }
+    }
+
     // Match and consume an i32 immediate.
     // This is used for stack argument byte offsets.
     fn match_imm32(&mut self, err_msg: &str) -> ParseResult {
@@ -855,7 +868,7 @@ impl<'a> Parser<'a> {
         } else {
             let uimm128 = match ty.lane_type() {
                 I8 => consume!(ty, self.match_uimm8("Expected an 8-bit unsigned integer")),
-                I16 => unimplemented!(), // TODO no 16-bit match yet
+                I16 => consume!(ty, self.match_imm16("Expected a 16-bit integer")),
                 I32 => consume!(ty, self.match_imm32("Expected a 32-bit integer")),
                 I64 => consume!(ty, self.match_imm64("Expected a 64-bit integer")),
                 F32 => consume!(ty, self.match_ieee32("Expected a 32-bit float...")),

From 168ad7fda3cb52c6c9ffa12684b281ba54a8ab01 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 18 Sep 2019 14:37:31 -0700
Subject: [PATCH 2780/3084] Fix 16-bit x86_pextr encoding

The x86 ISA has (at least) two encodings for PEXTRW:
 1. in the SSE2 opcode (66 0f c5) the XMM operand uses r/m and the GPR operand uses reg
 2. in the SSE4.1 opcode (66 0f 3a 15) the XMM operand uses reg and the GPR operand uses r/m

This changes the 16-bit x86_pextr encoding from 1 to 2 to match the other PEXTR* implementations (all #2 style).
---
 cranelift/codegen/meta/src/isa/x86/encodings.rs | 17 ++++++++---------
 cranelift/codegen/meta/src/isa/x86/opcodes.rs   |  4 ++--
 .../filetests/isa/x86/extractlane-binemit.clif  |  2 +-
 3 files changed, 11 insertions(+), 12 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index b752b6bfae..f1df48f13b 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1798,23 +1798,22 @@ pub(crate) fn define(
     }
 
     // SIMD extractlane
-    let mut x86_pextr_mapping: HashMap)> =
-        HashMap::new();
-    x86_pextr_mapping.insert(8, (&PEXTRB, Some(use_sse41_simd)));
-    x86_pextr_mapping.insert(16, (&PEXTRW_SSE2, None));
-    x86_pextr_mapping.insert(32, (&PEXTR, Some(use_sse41_simd)));
-    x86_pextr_mapping.insert(64, (&PEXTR, Some(use_sse41_simd)));
+    let mut x86_pextr_mapping: HashMap = HashMap::new();
+    x86_pextr_mapping.insert(8, &PEXTRB);
+    x86_pextr_mapping.insert(16, &PEXTRW);
+    x86_pextr_mapping.insert(32, &PEXTR);
+    x86_pextr_mapping.insert(64, &PEXTR);
 
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
-        if let Some((opcode, isap)) = x86_pextr_mapping.get(&ty.lane_bits()) {
+        if let Some(opcode) = x86_pextr_mapping.get(&ty.lane_bits()) {
             let instruction = x86_pextr.bind_vector_from_lane(ty, sse_vector_size);
             let template = rec_r_ib_unsigned_gpr.opcodes(opcode);
             if ty.lane_bits() < 64 {
-                e.enc_32_64_maybe_isap(instruction, template.nonrex(), isap.clone());
+                e.enc_32_64_maybe_isap(instruction, template.nonrex(), Some(use_sse41_simd));
             } else {
                 // It turns out the 64-bit widths have REX/W encodings and only are available on
                 // x86_64.
-                e.enc64_maybe_isap(instruction, template.rex().w(), isap.clone());
+                e.enc64_maybe_isap(instruction, template.rex().w(), Some(use_sse41_simd));
             }
         }
     }
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index 6ae740883c..706774a3d8 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -269,8 +269,8 @@ pub static PEXTR: [u8; 4] = [0x66, 0x0f, 0x3a, 0x16];
 /// Extract byte (SSE4.1).
 pub static PEXTRB: [u8; 4] = [0x66, 0x0f, 0x3a, 0x14];
 
-/// Extract word (SSE2). There is a 4-byte SSE4.1 variant that can also move to m/16.
-pub static PEXTRW_SSE2: [u8; 3] = [0x66, 0x0f, 0xc5];
+/// Extract word (SSE4.1). There is a 3-byte SSE2 variant that can also move to m/16.
+pub static PEXTRW: [u8; 4] = [0x66, 0x0f, 0x3a, 0x15];
 
 /// Insert doubleword or quadword, depending on REX.W (SSE4.1).
 pub static PINSR: [u8; 4] = [0x66, 0x0f, 0x3a, 0x22];
diff --git a/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif b/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif
index 86f16315cf..d1478b99d0 100644
--- a/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif
@@ -17,7 +17,7 @@ function %test_extractlane_i16() {
 ebb0:
 [-, %rax]   v0 = iconst.i16 4
 [-, %xmm1]  v1 = splat.i16x8 v0
-[-, %rax]   v2 = x86_pextr v1, 4    ; bin: 66 0f c5 c8 04
+[-, %rax]   v2 = x86_pextr v1, 4    ; bin: 66 0f 3a 15 c8 04
             return
 }
 

From 630cb3ee62e625d329d5eabb8cb6f518ff7d83d9 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 18 Sep 2019 14:44:06 -0700
Subject: [PATCH 2781/3084] Add x86 encoding for SIMD imul

Only i16x8 and i32x4 are encoded in this commit mainly because i8x16 and i64x2 do not have simple encodings in x86. i64x2 is not required by the SIMD spec and there is discussion (https://github.com/WebAssembly/simd/pull/98#issuecomment-530092217) about removing i8x16.
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 10 +++++
 cranelift/codegen/meta/src/isa/x86/opcodes.rs |  8 ++++
 .../codegen/meta/src/shared/instructions.rs   |  3 +-
 .../filetests/isa/x86/simd-arithmetic.clif    | 44 +++++++++++++++++++
 cranelift/wasm/src/code_translator.rs         |  6 ++-
 5 files changed, 67 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index f1df48f13b..680e0f0764 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1945,6 +1945,16 @@ pub(crate) fn define(
         e.enc_32_64(isub, rec_fa.opcodes(*opcodes));
     }
 
+    // SIMD integer multiplication: the x86 ISA does not have instructions for multiplying I8x16
+    // and I64x2 and these are (at the time of writing) not necessary for WASM SIMD.
+    for (ty, opcodes, isap) in &[
+        (I16, &PMULLW[..], None),
+        (I32, &PMULLD[..], Some(use_sse41_simd)),
+    ] {
+        let imul = imul.bind_vector_from_lane(ty.clone(), sse_vector_size);
+        e.enc_32_64_maybe_isap(imul, rec_fa.opcodes(opcodes), *isap);
+    }
+
     // SIMD icmp using PCMPEQ*
     let mut pcmpeq_mapping: HashMap)> = HashMap::new();
     pcmpeq_mapping.insert(8, (&PCMPEQB, None));
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index 706774a3d8..33b7d71c38 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -281,6 +281,14 @@ pub static PINSRB: [u8; 4] = [0x66, 0x0f, 0x3a, 0x20];
 /// Insert word (SSE2).
 pub static PINSRW: [u8; 3] = [0x66, 0x0f, 0xc4];
 
+/// Multiply the packed signed word integers in xmm1 and xmm2/m128, and store the low 16 bits of
+/// the results in xmm1 (SSE2).
+pub static PMULLW: [u8; 3] = [0x66, 0x0f, 0xd5];
+
+/// Multiply the packed doubleword signed integers in xmm1 and xmm2/m128 and store the low 32
+/// bits of each product in xmm1 (SSE4.1).
+pub static PMULLD: [u8; 4] = [0x66, 0x0f, 0x38, 0x40];
+
 /// Pop top of stack into r{16,32,64}; increment stack pointer.
 pub static POP_REG: [u8; 1] = [0x58];
 
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index d8f5474c29..49c7900e9b 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1722,8 +1722,7 @@ pub(crate) fn define(
         Wrapping integer multiplication: `a := x y \pmod{2^B}`.
 
         This instruction does not depend on the signed/unsigned interpretation
-        of the
-        operands.
+        of the operands.
 
         Polymorphic over all integer types (vector and scalar).
         "#,
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
index 8244177728..e2714a91dc 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
@@ -120,3 +120,47 @@ ebb0:
 
     return ; bin: c3
 }
+
+function %imul_i32x4() -> b1 {
+ebb0:
+[-, %xmm0]    v0 = vconst.i32x4 [-1 0 1 -2147483647] ; e.g. -2147483647 == 0x80_00_00_01
+[-, %xmm1]    v1 = vconst.i32x4 [2 2 2 2]
+[-, %xmm0]    v2 = imul v0, v1 ; bin: 66 0f 38 40 c1
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, -2
+
+    v5 = extractlane v2, 1
+    v6 = icmp_imm eq v5, 0
+
+    v7 = extractlane v2, 3
+    v8 = icmp_imm eq v7, 2 ; 0x80_00_00_01 * 2 == 0x1_00_00_00_02 (and the 1 is dropped)
+
+    v9 = band v4, v6
+    v10 = band v8, v9
+    return v10
+}
+; run
+
+function %imul_i16x8() -> b1 {
+ebb0:
+[-, %xmm1]    v0 = vconst.i16x8 [-1 0 1 32767 0 0 0 0] ; e.g. 32767 == 0x7f_ff
+[-, %xmm2]    v1 = vconst.i16x8 [2 2 2 2 0 0 0 0]
+[-, %xmm1]    v2 = imul v0, v1 ; bin: 66 0f d5 ca
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 0xfffe ; TODO -2 will not work here and below because v3 is being
+    ; uextend-ed, not sextend-ed
+
+    v5 = extractlane v2, 1
+    v6 = icmp_imm eq v5, 0
+
+    v7 = extractlane v2, 3
+    v8 = icmp_imm eq v7, 0xfffe ; 0x7f_ff * 2 == 0xff_fe
+
+    v9 = band v4, v6
+    v10 = band v8, v9
+
+    return v4
+}
+; run
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index ce2aa8dab7..8313212af9 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1008,6 +1008,10 @@ pub fn translate_operator(
             let a = state.pop1();
             state.push1(builder.ins().ineg(a))
         }
+        Operator::I16x8Mul | Operator::I32x4Mul => {
+            let (a, b) = state.pop2();
+            state.push1(builder.ins().imul(a, b))
+        }
         Operator::I8x16Eq
         | Operator::I8x16Ne
         | Operator::I8x16LtS
@@ -1074,13 +1078,11 @@ pub fn translate_operator(
         | Operator::I16x8AddSaturateU
         | Operator::I16x8SubSaturateS
         | Operator::I16x8SubSaturateU
-        | Operator::I16x8Mul
         | Operator::I32x4AnyTrue
         | Operator::I32x4AllTrue
         | Operator::I32x4Shl
         | Operator::I32x4ShrS
         | Operator::I32x4ShrU
-        | Operator::I32x4Mul
         | Operator::I64x2AnyTrue
         | Operator::I64x2AllTrue
         | Operator::I64x2Shl

From 21144068d441dbd3dc4d192122fc121401df4117 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 18 Sep 2019 15:56:15 -0700
Subject: [PATCH 2782/3084] Add saturating addition with a SIMD encoding

This includes the new instructions `sadd_sat` and `uadd_sat` and only encodes the i8x16 and i16x8 types; these are what is needed for implementing the SIMD spec (see https://github.com/WebAssembly/simd/blob/master/proposals/simd/SIMD.md#saturating-integer-addition).
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 20 ++++++++++++
 cranelift/codegen/meta/src/isa/x86/opcodes.rs | 12 +++++++
 .../codegen/meta/src/shared/instructions.rs   | 32 +++++++++++++++++++
 .../filetests/isa/x86/simd-arithmetic.clif    | 26 +++++++++++++++
 cranelift/wasm/src/code_translator.rs         | 12 ++++---
 5 files changed, 98 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 680e0f0764..7a2d725030 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -464,6 +464,7 @@ pub(crate) fn define(
     let rotl_imm = shared.by_name("rotl_imm");
     let rotr = shared.by_name("rotr");
     let rotr_imm = shared.by_name("rotr_imm");
+    let sadd_sat = shared.by_name("sadd_sat");
     let safepoint = shared.by_name("safepoint");
     let scalar_to_vector = shared.by_name("scalar_to_vector");
     let selectif = shared.by_name("selectif");
@@ -490,6 +491,7 @@ pub(crate) fn define(
     let trueff = shared.by_name("trueff");
     let trueif = shared.by_name("trueif");
     let trunc = shared.by_name("trunc");
+    let uadd_sat = shared.by_name("uadd_sat");
     let uextend = shared.by_name("uextend");
     let uload16 = shared.by_name("uload16");
     let uload16_complex = shared.by_name("uload16_complex");
@@ -1939,6 +1941,24 @@ pub(crate) fn define(
         e.enc_32_64(iadd, rec_fa.opcodes(*opcodes));
     }
 
+    // SIMD integer saturating addition
+    e.enc_32_64(
+        sadd_sat.bind_vector_from_lane(I8, sse_vector_size),
+        rec_fa.opcodes(&PADDSB),
+    );
+    e.enc_32_64(
+        sadd_sat.bind_vector_from_lane(I16, sse_vector_size),
+        rec_fa.opcodes(&PADDSW),
+    );
+    e.enc_32_64(
+        uadd_sat.bind_vector_from_lane(I8, sse_vector_size),
+        rec_fa.opcodes(&PADDUSB),
+    );
+    e.enc_32_64(
+        uadd_sat.bind_vector_from_lane(I16, sse_vector_size),
+        rec_fa.opcodes(&PADDUSW),
+    );
+
     // SIMD integer subtraction
     for (ty, opcodes) in &[(I8, &PSUBB), (I16, &PSUBW), (I32, &PSUBD), (I64, &PSUBQ)] {
         let isub = isub.bind_vector_from_lane(ty.clone(), sse_vector_size);
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index 33b7d71c38..9d93f0cf14 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -251,6 +251,18 @@ pub static PADDQ: [u8; 3] = [0x66, 0x0f, 0xd4];
 /// Add packed word integers from xmm2/m128 and xmm1 (SSE2).
 pub static PADDW: [u8; 3] = [0x66, 0x0f, 0xfd];
 
+/// Add packed signed byte integers from xmm2/m128 and xmm1 saturate the results (SSE).
+pub static PADDSB: [u8; 3] = [0x66, 0x0f, 0xec];
+
+/// Add packed signed word integers from xmm2/m128 and xmm1 saturate the results (SSE).
+pub static PADDSW: [u8; 3] = [0x66, 0x0f, 0xed];
+
+/// Add packed unsigned byte integers from xmm2/m128 and xmm1 saturate the results (SSE).
+pub static PADDUSB: [u8; 3] = [0x66, 0x0f, 0xdc];
+
+/// Add packed unsigned word integers from xmm2/m128 and xmm1 saturate the results (SSE).
+pub static PADDUSW: [u8; 3] = [0x66, 0x0f, 0xdd];
+
 /// Compare packed data for equal (SSE2).
 pub static PCMPEQB: [u8; 3] = [0x66, 0x0f, 0x74];
 
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 49c7900e9b..27eb3a3498 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1690,6 +1690,38 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    ig.push(
+        Inst::new(
+            "uadd_sat",
+            r#"
+        Add with unsigned saturation.
+
+        This is similar to `iadd` but the operands are interpreted as unsigned integers and their 
+        summed result, instead of wrapping, will be saturated to the highest unsigned integer for
+        the controlling type (e.g. `0xFF` for i8).
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
+    ig.push(
+        Inst::new(
+            "sadd_sat",
+            r#"
+        Add with signed saturation.
+
+        This is similar to `iadd` but the operands are interpreted as signed integers and their 
+        summed result, instead of wrapping, will be saturated to the lowest or highest 
+        signed integer for the controlling type (e.g. `0x80` or `0x7F` for i8). For example, 
+        since an `iadd_ssat.i8` of `0x70` and `0x70` is greater than `0x7F`, the result will be 
+        clamped to `0x7F`.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
     ig.push(
         Inst::new(
             "isub",
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
index e2714a91dc..c82c5deb68 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
@@ -164,3 +164,29 @@ ebb0:
     return v4
 }
 ; run
+
+function %sadd_sat_i8x16() -> b1 {
+ebb0:
+[-, %xmm2]    v0 = vconst.i8x16 [127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
+[-, %xmm3]    v1 = vconst.i8x16 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
+
+[-, %xmm2]    v2 = sadd_sat v0, v1 ; bin: 66 0f ec d3
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 127
+
+    return v4
+}
+; run
+
+function %uadd_sat_i16x8() -> b1 {
+ebb0:
+[-, %xmm2]    v0 = vconst.i16x8 [-1 0 0 0 0 0 0 0]
+[-, %xmm3]    v1 = vconst.i16x8 [-1 1 1 1 1 1 1 1]
+
+[-, %xmm2]    v2 = uadd_sat v0, v1 ; bin: 66 0f dd d3
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 65535
+
+    return v4
+}
+; run
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 8313212af9..85623f95fb 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1000,6 +1000,14 @@ pub fn translate_operator(
             let (a, b) = state.pop2();
             state.push1(builder.ins().iadd(a, b))
         }
+        Operator::I8x16AddSaturateS | Operator::I16x8AddSaturateS => {
+            let (a, b) = state.pop2();
+            state.push1(builder.ins().sadd_sat(a, b))
+        }
+        Operator::I8x16AddSaturateU | Operator::I16x8AddSaturateU => {
+            let (a, b) = state.pop2();
+            state.push1(builder.ins().uadd_sat(a, b))
+        }
         Operator::I8x16Sub | Operator::I16x8Sub | Operator::I32x4Sub | Operator::I64x2Sub => {
             let (a, b) = state.pop2();
             state.push1(builder.ins().isub(a, b))
@@ -1064,8 +1072,6 @@ pub fn translate_operator(
         | Operator::I8x16Shl
         | Operator::I8x16ShrS
         | Operator::I8x16ShrU
-        | Operator::I8x16AddSaturateS
-        | Operator::I8x16AddSaturateU
         | Operator::I8x16SubSaturateS
         | Operator::I8x16SubSaturateU
         | Operator::I8x16Mul
@@ -1074,8 +1080,6 @@ pub fn translate_operator(
         | Operator::I16x8Shl
         | Operator::I16x8ShrS
         | Operator::I16x8ShrU
-        | Operator::I16x8AddSaturateS
-        | Operator::I16x8AddSaturateU
         | Operator::I16x8SubSaturateS
         | Operator::I16x8SubSaturateU
         | Operator::I32x4AnyTrue

From 90c49a2f7c58c225d83b6b786caaac0638a89515 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 18 Sep 2019 16:28:13 -0700
Subject: [PATCH 2783/3084] Add saturating subtraction with a SIMD encoding

This includes the new instructions `ssub_sat` and `usub_sat` and only encodes the i8x16 and i16x8 types; these are what is needed for implementing the SIMD spec (see https://github.com/WebAssembly/simd/blob/master/proposals/simd/SIMD.md#saturating-integer-subtraction).
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 20 +++++++++++++
 cranelift/codegen/meta/src/isa/x86/opcodes.rs | 16 ++++++++++
 .../codegen/meta/src/shared/instructions.rs   | 30 +++++++++++++++++++
 .../filetests/isa/x86/simd-arithmetic.clif    | 28 +++++++++++++++++
 cranelift/wasm/src/code_translator.rs         | 12 +++++---
 5 files changed, 102 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 7a2d725030..1e92940487 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -480,6 +480,7 @@ pub(crate) fn define(
     let sqrt = shared.by_name("sqrt");
     let sshr = shared.by_name("sshr");
     let sshr_imm = shared.by_name("sshr_imm");
+    let ssub_sat = shared.by_name("ssub_sat");
     let stack_addr = shared.by_name("stack_addr");
     let store = shared.by_name("store");
     let store_complex = shared.by_name("store_complex");
@@ -501,6 +502,7 @@ pub(crate) fn define(
     let uload8_complex = shared.by_name("uload8_complex");
     let ushr = shared.by_name("ushr");
     let ushr_imm = shared.by_name("ushr_imm");
+    let usub_sat = shared.by_name("usub_sat");
     let vconst = shared.by_name("vconst");
     let x86_bsf = x86.by_name("x86_bsf");
     let x86_bsr = x86.by_name("x86_bsr");
@@ -1965,6 +1967,24 @@ pub(crate) fn define(
         e.enc_32_64(isub, rec_fa.opcodes(*opcodes));
     }
 
+    // SIMD integer saturating subtraction
+    e.enc_32_64(
+        ssub_sat.bind_vector_from_lane(I8, sse_vector_size),
+        rec_fa.opcodes(&PSUBSB),
+    );
+    e.enc_32_64(
+        ssub_sat.bind_vector_from_lane(I16, sse_vector_size),
+        rec_fa.opcodes(&PSUBSW),
+    );
+    e.enc_32_64(
+        usub_sat.bind_vector_from_lane(I8, sse_vector_size),
+        rec_fa.opcodes(&PSUBUSB),
+    );
+    e.enc_32_64(
+        usub_sat.bind_vector_from_lane(I16, sse_vector_size),
+        rec_fa.opcodes(&PSUBUSW),
+    );
+
     // SIMD integer multiplication: the x86 ISA does not have instructions for multiplying I8x16
     // and I64x2 and these are (at the time of writing) not necessary for WASM SIMD.
     for (ty, opcodes, isap) in &[
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index 9d93f0cf14..f81d2423ea 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -326,6 +326,22 @@ pub static PSUBD: [u8; 3] = [0x66, 0x0f, 0xfa];
 /// Subtract packed quadword integers in xmm2/m128 from xmm1 (SSE2).
 pub static PSUBQ: [u8; 3] = [0x66, 0x0f, 0xfb];
 
+/// Subtract packed signed byte integers in xmm2/m128 from packed signed byte integers in xmm1
+/// and saturate results (SSE2).
+pub static PSUBSB: [u8; 3] = [0x66, 0x0f, 0xe8];
+
+/// Subtract packed signed word integers in xmm2/m128 from packed signed word integers in xmm1
+/// and saturate results (SSE2).
+pub static PSUBSW: [u8; 3] = [0x66, 0x0f, 0xe9];
+
+/// Subtract packed unsigned byte integers in xmm2/m128 from packed unsigned byte integers in xmm1
+/// and saturate results (SSE2).
+pub static PSUBUSB: [u8; 3] = [0x66, 0x0f, 0xd8];
+
+/// Subtract packed unsigned word integers in xmm2/m128 from packed unsigned word integers in xmm1
+/// and saturate results (SSE2).
+pub static PSUBUSW: [u8; 3] = [0x66, 0x0f, 0xd9];
+
 /// Push r{16,32,64}.
 pub static PUSH_REG: [u8; 1] = [0x50];
 
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 27eb3a3498..b9070c7f39 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1736,6 +1736,36 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    ig.push(
+        Inst::new(
+            "usub_sat",
+            r#"
+        Subtract with unsigned saturation.
+
+        This is similar to `isub` but the operands are interpreted as unsigned integers and their 
+        difference, instead of wrapping, will be saturated to the lowest unsigned integer for
+        the controlling type (e.g. `0x00` for i8).
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
+    ig.push(
+        Inst::new(
+            "ssub_sat",
+            r#"
+        Subtract with signed saturation.
+
+        This is similar to `isub` but the operands are interpreted as signed integers and their 
+        difference, instead of wrapping, will be saturated to the lowest or highest 
+        signed integer for the controlling type (e.g. `0x80` or `0x7F` for i8).
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
     ig.push(
         Inst::new(
             "ineg",
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
index c82c5deb68..c9d6c4c372 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
@@ -190,3 +190,31 @@ ebb0:
     return v4
 }
 ; run
+
+function %sub_sat_i8x16() -> b1 {
+ebb0:
+[-, %xmm2]    v0 = vconst.i8x16 [128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] ; 120 == 0x80 == -128
+[-, %xmm3]    v1 = vconst.i8x16 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
+
+[-, %xmm2]    v2 = ssub_sat v0, v1 ; bin: 66 0f e8 d3
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 0x80 ; still -128, TODO it's unclear why I can't use -128 here
+
+    ; now re-use 0x80 as an unsigned 128
+[-, %xmm2]    v5 = usub_sat v0, v2 ; bin: 66 0f d8 d2
+    v6 = extractlane v5, 0
+    v7 = icmp_imm eq v6, 0
+
+    v8 = band v4, v7
+    return v8
+}
+; run
+
+function %sub_sat_i16x8() {
+ebb0:
+[-, %xmm3]    v0 = vconst.i16x8 [0 0 0 0 0 0 0 0]
+[-, %xmm5]    v1 = vconst.i16x8 [1 1 1 1 1 1 1 1]
+[-, %xmm3]    v2 = ssub_sat v0, v1 ; bin: 66 0f e9 dd
+[-, %xmm3]    v3 = usub_sat v0, v1 ; bin: 66 0f d9 dd
+    return
+}
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 85623f95fb..ad5345768d 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1012,6 +1012,14 @@ pub fn translate_operator(
             let (a, b) = state.pop2();
             state.push1(builder.ins().isub(a, b))
         }
+        Operator::I8x16SubSaturateS | Operator::I16x8SubSaturateS => {
+            let (a, b) = state.pop2();
+            state.push1(builder.ins().ssub_sat(a, b))
+        }
+        Operator::I8x16SubSaturateU | Operator::I16x8SubSaturateU => {
+            let (a, b) = state.pop2();
+            state.push1(builder.ins().usub_sat(a, b))
+        }
         Operator::I8x16Neg | Operator::I16x8Neg | Operator::I32x4Neg | Operator::I64x2Neg => {
             let a = state.pop1();
             state.push1(builder.ins().ineg(a))
@@ -1072,16 +1080,12 @@ pub fn translate_operator(
         | Operator::I8x16Shl
         | Operator::I8x16ShrS
         | Operator::I8x16ShrU
-        | Operator::I8x16SubSaturateS
-        | Operator::I8x16SubSaturateU
         | Operator::I8x16Mul
         | Operator::I16x8AnyTrue
         | Operator::I16x8AllTrue
         | Operator::I16x8Shl
         | Operator::I16x8ShrS
         | Operator::I16x8ShrU
-        | Operator::I16x8SubSaturateS
-        | Operator::I16x8SubSaturateU
         | Operator::I32x4AnyTrue
         | Operator::I32x4AllTrue
         | Operator::I32x4Shl

From d8967bb58abf3b1f3e2deb9fcb2e76d8e37d679a Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 30 Sep 2019 10:13:37 -0700
Subject: [PATCH 2784/3084] Split SIMD arithmetic tests by test directive

---
 ...etic.clif => simd-arithmetic-binemit.clif} |  63 +------
 .../isa/x86/simd-arithmetic-legalize.clif     |  36 ++++
 .../isa/x86/simd-arithmetic-run.clif          | 155 ++++++++++++++++++
 3 files changed, 200 insertions(+), 54 deletions(-)
 rename cranelift/filetests/filetests/isa/x86/{simd-arithmetic.clif => simd-arithmetic-binemit.clif} (78%)
 create mode 100644 cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif

diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif
similarity index 78%
rename from cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
rename to cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif
index c9d6c4c372..cc2d7f03e1 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif
@@ -1,6 +1,4 @@
-test run
 test binemit
-test legalizer
 set enable_simd
 target x86_64 skylake
 
@@ -15,14 +13,11 @@ ebb0:
 
     v5 = extractlane v2, 3
     v6 = icmp_imm eq v5, 5
-    ; TODO replace extractlanes with vector comparison
 
     v7 = band v4, v6
     return v7
 }
 
-; run
-
 function %iadd_i8x16_with_overflow() -> b1 {
 ebb0:
 [-, %xmm0]    v0 = vconst.i8x16 [255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255]
@@ -31,13 +26,10 @@ ebb0:
 
     v3 = extractlane v2, 0
     v4 = icmp_imm eq v3, 1
-    ; TODO replace extractlane with vector comparison
 
     return v4
 }
 
-; run
-
 function %iadd_i16x8(i16x8, i16x8) -> i16x8 {
 ebb0(v0: i16x8 [%xmm1], v1: i16x8 [%xmm2]):
 [-, %xmm1]  v2 = iadd v0, v1      ; bin: 66 0f fd ca
@@ -61,14 +53,11 @@ ebb0:
 
     v5 = extractlane v2, 1
     v6 = icmp_imm eq v5, 0xffffffff
-    ; TODO replace extractlanes with vector comparison
 
     v7 = band v4, v6
     return v7
 }
 
-; run
-
 function %isub_i64x2(i64x2, i64x2) -> i64x2 {
 ebb0(v0: i64x2 [%xmm0], v1: i64x2 [%xmm1]):
 [-, %xmm0]  v2 = isub v0, v1      ; bin: 66 0f fb c1
@@ -87,40 +76,6 @@ ebb0(v0: i8x16 [%xmm3], v1: i8x16 [%xmm4]):
     return v2
 }
 
-function %ineg_i32x4() -> b1 {
-ebb0:
-    v0 = vconst.i32x4 [1 1 1 1]
-    v2 = ineg v0
-    ; check: v5 = vconst.i32x4 0x00
-    ; nextln: v2 = isub v5, v0
-
-    v3 = extractlane v2, 0
-    v4 = icmp_imm eq v3, -1
-
-    return v4 ; bin: c3
-}
-; run
-
-function %ineg_legalized() {
-ebb0:
-    v0 = vconst.i8x16 0x00
-    v1 = ineg v0
-    ; check: v6 = vconst.i8x16 0x00
-    ; nextln: v1 = isub v6, v0
-
-    v2 = raw_bitcast.i16x8 v0
-    v3 = ineg v2
-    ; check: v7 = vconst.i16x8 0x00
-    ; nextln: v3 = isub v7, v2
-
-    v4 = raw_bitcast.i64x2 v0
-    v5 = ineg v4
-    ; check: v8 = vconst.i64x2 0x00
-    ; nextln: v5 = isub v8, v4
-
-    return ; bin: c3
-}
-
 function %imul_i32x4() -> b1 {
 ebb0:
 [-, %xmm0]    v0 = vconst.i32x4 [-1 0 1 -2147483647] ; e.g. -2147483647 == 0x80_00_00_01
@@ -140,7 +95,7 @@ ebb0:
     v10 = band v8, v9
     return v10
 }
-; run
+
 
 function %imul_i16x8() -> b1 {
 ebb0:
@@ -149,8 +104,8 @@ ebb0:
 [-, %xmm1]    v2 = imul v0, v1 ; bin: 66 0f d5 ca
 
     v3 = extractlane v2, 0
-    v4 = icmp_imm eq v3, 0xfffe ; TODO -2 will not work here and below because v3 is being
-    ; uextend-ed, not sextend-ed
+    v4 = icmp_imm eq v3, 0xfffe ; 0xfffe == -2; -2 will not work here and below because v3 is
+    ; being uextend-ed, not sextend-ed
 
     v5 = extractlane v2, 1
     v6 = icmp_imm eq v5, 0
@@ -163,7 +118,7 @@ ebb0:
 
     return v4
 }
-; run
+
 
 function %sadd_sat_i8x16() -> b1 {
 ebb0:
@@ -176,7 +131,7 @@ ebb0:
 
     return v4
 }
-; run
+
 
 function %uadd_sat_i16x8() -> b1 {
 ebb0:
@@ -189,16 +144,16 @@ ebb0:
 
     return v4
 }
-; run
+
 
 function %sub_sat_i8x16() -> b1 {
 ebb0:
-[-, %xmm2]    v0 = vconst.i8x16 [128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] ; 120 == 0x80 == -128
+[-, %xmm2]    v0 = vconst.i8x16 [128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] ; 128 == 0x80 == -128
 [-, %xmm3]    v1 = vconst.i8x16 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
 
 [-, %xmm2]    v2 = ssub_sat v0, v1 ; bin: 66 0f e8 d3
     v3 = extractlane v2, 0
-    v4 = icmp_imm eq v3, 0x80 ; still -128, TODO it's unclear why I can't use -128 here
+    v4 = icmp_imm eq v3, 0x80 ; 0x80 == -128
 
     ; now re-use 0x80 as an unsigned 128
 [-, %xmm2]    v5 = usub_sat v0, v2 ; bin: 66 0f d8 d2
@@ -208,7 +163,7 @@ ebb0:
     v8 = band v4, v7
     return v8
 }
-; run
+
 
 function %sub_sat_i16x8() {
 ebb0:
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
new file mode 100644
index 0000000000..6155204899
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
@@ -0,0 +1,36 @@
+test legalizer
+set enable_simd
+target x86_64 skylake
+
+function %ineg_i32x4() -> b1 {
+ebb0:
+    v0 = vconst.i32x4 [1 1 1 1]
+    v2 = ineg v0
+    ; check: v5 = vconst.i32x4 0x00
+    ; nextln: v2 = isub v5, v0
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, -1
+
+    return v4
+}
+
+function %ineg_legalized() {
+ebb0:
+    v0 = vconst.i8x16 0x00
+    v1 = ineg v0
+    ; check: v6 = vconst.i8x16 0x00
+    ; nextln: v1 = isub v6, v0
+
+    v2 = raw_bitcast.i16x8 v0
+    v3 = ineg v2
+    ; check: v7 = vconst.i16x8 0x00
+    ; nextln: v3 = isub v7, v2
+
+    v4 = raw_bitcast.i64x2 v0
+    v5 = ineg v4
+    ; check: v8 = vconst.i64x2 0x00
+    ; nextln: v5 = isub v8, v4
+
+    return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
new file mode 100644
index 0000000000..22bcf11bdd
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
@@ -0,0 +1,155 @@
+test run
+set enable_simd
+target x86_64 skylake
+
+function %iadd_i32x4() -> b1 {
+ebb0:
+    v0 = vconst.i32x4 [1 1 1 1]
+    v1 = vconst.i32x4 [1 2 3 4]
+    v2 = iadd v0, v1
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 2
+
+    v5 = extractlane v2, 3
+    v6 = icmp_imm eq v5, 5
+    ; TODO replace extractlanes with vector comparison
+
+    v7 = band v4, v6
+    return v7
+}
+; run
+
+function %iadd_i8x16_with_overflow() -> b1 {
+ebb0:
+    v0 = vconst.i8x16 [255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255]
+    v1 = vconst.i8x16 [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2]
+    v2 = iadd v0, v1
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 1
+    ; TODO replace extractlane with vector comparison
+
+    return v4
+}
+; run
+
+function %isub_i32x4() -> b1 {
+ebb0:
+    v0 = vconst.i32x4 [1 1 1 1]
+    v1 = vconst.i32x4 [1 2 3 4]
+    v2 = isub v0, v1
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 0
+
+    v5 = extractlane v2, 1
+    v6 = icmp_imm eq v5, 0xffffffff
+    ; TODO replace extractlanes with vector comparison
+
+    v7 = band v4, v6
+    return v7
+}
+; run
+
+
+function %ineg_i32x4() -> b1 {
+ebb0:
+    v0 = vconst.i32x4 [1 1 1 1]
+    v2 = ineg v0
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, -1
+
+    return v4
+}
+; run
+
+function %imul_i32x4() -> b1 {
+ebb0:
+    v0 = vconst.i32x4 [-1 0 1 -2147483647] ; e.g. -2147483647 == 0x80_00_00_01
+    v1 = vconst.i32x4 [2 2 2 2]
+    v2 = imul v0, v1
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, -2
+
+    v5 = extractlane v2, 1
+    v6 = icmp_imm eq v5, 0
+
+    v7 = extractlane v2, 3
+    v8 = icmp_imm eq v7, 2 ; 0x80_00_00_01 * 2 == 0x1_00_00_00_02 (and the 1 is dropped)
+
+    v9 = band v4, v6
+    v10 = band v8, v9
+    return v10
+}
+; run
+
+function %imul_i16x8() -> b1 {
+ebb0:
+    v0 = vconst.i16x8 [-1 0 1 32767 0 0 0 0] ; e.g. 32767 == 0x7f_ff
+    v1 = vconst.i16x8 [2 2 2 2 0 0 0 0]
+    v2 = imul v0, v1
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 0xfffe ; 0xfffe == -2; -2 will not work here and below because v3 is
+    ; being uextend-ed, not sextend-ed
+
+    v5 = extractlane v2, 1
+    v6 = icmp_imm eq v5, 0
+
+    v7 = extractlane v2, 3
+    v8 = icmp_imm eq v7, 0xfffe ; 0x7f_ff * 2 == 0xff_fe
+
+    v9 = band v4, v6
+    v10 = band v8, v9
+
+    return v4
+}
+; run
+
+function %sadd_sat_i8x16() -> b1 {
+ebb0:
+    v0 = vconst.i8x16 [127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
+    v1 = vconst.i8x16 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
+
+    v2 = sadd_sat v0, v1
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 127
+
+    return v4
+}
+; run
+
+function %uadd_sat_i16x8() -> b1 {
+ebb0:
+    v0 = vconst.i16x8 [-1 0 0 0 0 0 0 0]
+    v1 = vconst.i16x8 [-1 1 1 1 1 1 1 1]
+
+    v2 = uadd_sat v0, v1
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 65535
+
+    return v4
+}
+; run
+
+function %sub_sat_i8x16() -> b1 {
+ebb0:
+    v0 = vconst.i8x16 [128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] ; 128 == 0x80 == -128
+    v1 = vconst.i8x16 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
+
+    v2 = ssub_sat v0, v1
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 0x80 ; 0x80 == -128
+
+    ; now re-use 0x80 as an unsigned 128
+    v5 = usub_sat v0, v2
+    v6 = extractlane v5, 0
+    v7 = icmp_imm eq v6, 0
+
+    v8 = band v4, v7
+    return v8
+}
+; run

From 0d50462a93671ae75561cdb491d2e0db01d7ab19 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 1 Oct 2019 10:18:11 +0200
Subject: [PATCH 2785/3084] Fixes #1091: Use match statements instead of
 HashMaps in x86 encodings;

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 104 +++++++++---------
 1 file changed, 51 insertions(+), 53 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 1e92940487..da08462e9a 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1759,24 +1759,22 @@ pub(crate) fn define(
     }
 
     // SIMD insertlane
-    let mut x86_pinsr_mapping: HashMap)> =
-        HashMap::new();
-    x86_pinsr_mapping.insert(8, (&PINSRB, Some(use_sse41_simd)));
-    x86_pinsr_mapping.insert(16, (&PINSRW, None));
-    x86_pinsr_mapping.insert(32, (&PINSR, Some(use_sse41_simd)));
-    x86_pinsr_mapping.insert(64, (&PINSR, Some(use_sse41_simd)));
-
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
-        if let Some((opcode, isap)) = x86_pinsr_mapping.get(&ty.lane_bits()) {
-            let instruction = x86_pinsr.bind_vector_from_lane(ty, sse_vector_size);
-            let template = rec_r_ib_unsigned_r.opcodes(opcode);
-            if ty.lane_bits() < 64 {
-                e.enc_32_64_maybe_isap(instruction, template.nonrex(), isap.clone());
-            } else {
-                // It turns out the 64-bit widths have REX/W encodings and only are available on
-                // x86_64.
-                e.enc64_maybe_isap(instruction, template.rex().w(), isap.clone());
-            }
+        let (opcode, isap): (&[_], _) = match ty.lane_bits() {
+            8 => (&PINSRB, Some(use_sse41_simd)),
+            16 => (&PINSRW, None),
+            32 | 64 => (&PINSR, Some(use_sse41_simd)),
+            _ => panic!("invalid size for SIMD insertlane"),
+        };
+
+        let instruction = x86_pinsr.bind_vector_from_lane(ty, sse_vector_size);
+        let template = rec_r_ib_unsigned_r.opcodes(opcode);
+        if ty.lane_bits() < 64 {
+            e.enc_32_64_maybe_isap(instruction, template.nonrex(), isap);
+        } else {
+            // It turns out the 64-bit widths have REX/W encodings and only are available on
+            // x86_64.
+            e.enc64_maybe_isap(instruction, template.rex().w(), isap);
         }
     }
 
@@ -1802,23 +1800,22 @@ pub(crate) fn define(
     }
 
     // SIMD extractlane
-    let mut x86_pextr_mapping: HashMap = HashMap::new();
-    x86_pextr_mapping.insert(8, &PEXTRB);
-    x86_pextr_mapping.insert(16, &PEXTRW);
-    x86_pextr_mapping.insert(32, &PEXTR);
-    x86_pextr_mapping.insert(64, &PEXTR);
-
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
-        if let Some(opcode) = x86_pextr_mapping.get(&ty.lane_bits()) {
-            let instruction = x86_pextr.bind_vector_from_lane(ty, sse_vector_size);
-            let template = rec_r_ib_unsigned_gpr.opcodes(opcode);
-            if ty.lane_bits() < 64 {
-                e.enc_32_64_maybe_isap(instruction, template.nonrex(), Some(use_sse41_simd));
-            } else {
-                // It turns out the 64-bit widths have REX/W encodings and only are available on
-                // x86_64.
-                e.enc64_maybe_isap(instruction, template.rex().w(), Some(use_sse41_simd));
-            }
+        let opcode = match ty.lane_bits() {
+            8 => &PEXTRB,
+            16 => &PEXTRW,
+            32 | 64 => &PEXTR,
+            _ => panic!("invalid size for SIMD extractlane"),
+        };
+
+        let instruction = x86_pextr.bind_vector_from_lane(ty, sse_vector_size);
+        let template = rec_r_ib_unsigned_gpr.opcodes(opcode);
+        if ty.lane_bits() < 64 {
+            e.enc_32_64_maybe_isap(instruction, template.nonrex(), Some(use_sse41_simd));
+        } else {
+            // It turns out the 64-bit widths have REX/W encodings and only are available on
+            // x86_64.
+            e.enc64_maybe_isap(instruction, template.rex().w(), Some(use_sse41_simd));
         }
     }
 
@@ -1996,27 +1993,28 @@ pub(crate) fn define(
     }
 
     // SIMD icmp using PCMPEQ*
-    let mut pcmpeq_mapping: HashMap)> = HashMap::new();
-    pcmpeq_mapping.insert(8, (&PCMPEQB, None));
-    pcmpeq_mapping.insert(16, (&PCMPEQW, None));
-    pcmpeq_mapping.insert(32, (&PCMPEQD, None));
-    pcmpeq_mapping.insert(64, (&PCMPEQQ, Some(use_sse41_simd)));
     for ty in ValueType::all_lane_types().filter(|t| t.is_int() && allowed_simd_type(t)) {
-        if let Some((opcodes, isa_predicate)) = pcmpeq_mapping.get(&ty.lane_bits()) {
-            let instruction = icmp.bind_vector_from_lane(ty, sse_vector_size);
-            let f_int_compare = formats.get(formats.by_name("IntCompare"));
-            let has_eq_condition_code =
-                InstructionPredicate::new_has_condition_code(f_int_compare, IntCC::Equal, "cond");
-            let template = rec_icscc_fpr.nonrex().opcodes(*opcodes);
-            e.enc_32_64_func(instruction, template, |builder| {
-                let builder = builder.inst_predicate(has_eq_condition_code);
-                if let Some(p) = isa_predicate {
-                    builder.isa_predicate(*p)
-                } else {
-                    builder
-                }
-            });
-        }
+        let (opcodes, isa_predicate): (&[_], _) = match ty.lane_bits() {
+            8 => (&PCMPEQB, None),
+            16 => (&PCMPEQW, None),
+            32 => (&PCMPEQD, None),
+            64 => (&PCMPEQQ, Some(use_sse41_simd)),
+            _ => panic!("invalid size for SIMD icmp"),
+        };
+
+        let instruction = icmp.bind_vector_from_lane(ty, sse_vector_size);
+        let f_int_compare = formats.get(formats.by_name("IntCompare"));
+        let has_eq_condition_code =
+            InstructionPredicate::new_has_condition_code(f_int_compare, IntCC::Equal, "cond");
+        let template = rec_icscc_fpr.nonrex().opcodes(opcodes);
+        e.enc_32_64_func(instruction, template, |builder| {
+            let builder = builder.inst_predicate(has_eq_condition_code);
+            if let Some(p) = isa_predicate {
+                builder.isa_predicate(p)
+            } else {
+                builder
+            }
+        });
     }
 
     // Reference type instructions

From 093b3b34267d80bd58650c336938d493799c2a6b Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Tue, 1 Oct 2019 21:32:37 +0200
Subject: [PATCH 2786/3084] Fix brz.i128

Fixes #1066
---
 cranelift/codegen/meta/src/shared/legalize.rs      |  2 +-
 cranelift/filetests/filetests/isa/x86/br-i128.clif | 12 ++++++++++++
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index fef4c1e1ec..7ac4e8af13 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -257,7 +257,7 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
                 )
             ),
             def!(c = band(a, b)),
-            def!(brz(c, ebb, vararg)),
+            def!(brnz(c, ebb, vararg)),
         ],
     );
 
diff --git a/cranelift/filetests/filetests/isa/x86/br-i128.clif b/cranelift/filetests/filetests/isa/x86/br-i128.clif
index e3ac6a9d86..442be8d6c8 100644
--- a/cranelift/filetests/filetests/isa/x86/br-i128.clif
+++ b/cranelift/filetests/filetests/isa/x86/br-i128.clif
@@ -4,6 +4,11 @@ target x86_64
 function u0:0(i128) -> i8 fast {
 ebb0(v0: i128):
     brz v0, ebb2
+    ; check: v0 = iconcat v3, v4
+    ; nextln: v5 = icmp_imm eq v3, 0
+    ; nextln: v6 = icmp_imm eq v4, 0
+    ; nextln: v7 = band v5, v6
+    ; nextln: brnz v7, ebb2
     jump ebb1
 
 ebb1:
@@ -18,7 +23,14 @@ ebb2:
 function u0:1(i128) -> i8 fast {
 ebb0(v0: i128):
     brnz v0, ebb2
+    ; check: v0 = iconcat v3, v4
+    ; nextln: brnz v3, ebb2
+    ; nextln: fallthrough ebb3
+
+    ; check: ebb3:
+    ; nextln: brnz.i64 v4, ebb2
     jump ebb1
+    ; nextln: fallthrough ebb1
 
 ebb1:
     v1 = iconst.i8 0

From e0005f1e6cfaf26f09fb188a88831216c4197646 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Tue, 1 Oct 2019 14:45:08 +0200
Subject: [PATCH 2787/3084] Insert split-edge of conditional branches before
 the destination.

---
 .../codegen/src/regalloc/branch_splitting.rs  | 37 ++++---------------
 .../filetests/regalloc/coalesce-bb.clif       | 12 +++---
 .../filetests/regalloc/reload-208-bb.clif     | 10 +++--
 .../filetests/regalloc/x86-regres-bb.clif     |  7 ++--
 .../filetests/safepoint/basic-bb.clif         | 18 ++++-----
 5 files changed, 32 insertions(+), 52 deletions(-)

diff --git a/cranelift/codegen/src/regalloc/branch_splitting.rs b/cranelift/codegen/src/regalloc/branch_splitting.rs
index e44f452296..a3f118b3d6 100644
--- a/cranelift/codegen/src/regalloc/branch_splitting.rs
+++ b/cranelift/codegen/src/regalloc/branch_splitting.rs
@@ -22,7 +22,6 @@ pub fn run(
 ) {
     let mut ctx = Context {
         has_new_blocks: false,
-        has_fallthrough_return: None,
         cur: EncCursor::new(func, isa),
         domtree,
         topo,
@@ -35,12 +34,6 @@ struct Context<'a> {
     /// True if new blocks were inserted.
     has_new_blocks: bool,
 
-    /// Record whether newly inserted empty blocks should be inserted last, or before the last, to
-    /// avoid disturbing the expected control flow of `fallthroug_return` statements.
-    ///
-    /// This value is computed when needed. The Option wraps the computed value if any.
-    has_fallthrough_return: Option,
-
     /// Current instruction as well as reference to function and ISA.
     cur: EncCursor<'a>,
 
@@ -89,7 +82,13 @@ impl<'a> Context<'a> {
         // If there are any parameters, split the edge.
         if self.should_split_edge(target) {
             // Create the block the branch will jump to.
-            let new_ebb = self.make_empty_ebb();
+            let new_ebb = self.cur.func.dfg.make_ebb();
+
+            // Insert the new block before the destination, such that it can fallthrough in the
+            // target block.
+            assert_ne!(Some(target), self.cur.layout().entry_block());
+            self.cur.layout_mut().insert_ebb(new_ebb, target);
+            self.has_new_blocks = true;
 
             // Extract the arguments of the branch instruction, split the Ebb parameters and the
             // branch arguments
@@ -157,28 +156,6 @@ impl<'a> Context<'a> {
         }
     }
 
-    // A new ebb must be inserted before the last ebb because the last ebb may have a
-    // fallthrough_return and can't have anything after it.
-    fn make_empty_ebb(&mut self) -> Ebb {
-        let last_ebb = self.cur.layout().last_ebb().unwrap();
-        if self.has_fallthrough_return == None {
-            let last_inst = self.cur.layout().last_inst(last_ebb).unwrap();
-            self.has_fallthrough_return =
-                Some(self.cur.func.dfg[last_inst].opcode() == Opcode::FallthroughReturn);
-        }
-        let new_ebb = self.cur.func.dfg.make_ebb();
-        if self.has_fallthrough_return == Some(true) {
-            // Insert before the last block which has a fallthrough_return
-            // instruction.
-            self.cur.layout_mut().insert_ebb(new_ebb, last_ebb);
-        } else {
-            // Insert after the last block.
-            self.cur.layout_mut().insert_ebb_after(new_ebb, last_ebb);
-        }
-        self.has_new_blocks = true;
-        new_ebb
-    }
-
     /// Returns whether we should introduce a new branch.
     fn should_split_edge(&self, target: Ebb) -> bool {
         // We should split the edge if the target has any parameters.
diff --git a/cranelift/filetests/filetests/regalloc/coalesce-bb.clif b/cranelift/filetests/filetests/regalloc/coalesce-bb.clif
index 384ef2ca4c..7718df7d17 100644
--- a/cranelift/filetests/filetests/regalloc/coalesce-bb.clif
+++ b/cranelift/filetests/filetests/regalloc/coalesce-bb.clif
@@ -103,21 +103,21 @@ ebb0(v0: i32):
     v2 = iconst.i32 2
     jump ebb1(v1, v2)
 
+    ; check:  $(splitEdge=$EBB):
+    ; check:    $(nv11b=$V) = copy.i32 v11
+    ; not:      copy
+    ; check:    jump ebb1($nv11b, v12)
+
 ebb1(v10: i32, v11: i32):
     ; v11 needs to be isolated because it interferes with v10.
     ; check: ebb1(v10: i32 [$LOC], $(nv11a=$V): i32 [$LOC])
     ; check: v11 = copy $nv11a
     v12 = iadd v10, v11
     v13 = icmp ult v12, v0
-    ; check:  brnz v13, $(splitEdge=$EBB)
+    ; check:  brnz v13, $splitEdge
     brnz v13, ebb1(v11, v12)
     jump ebb2
 
-    ; check:  $splitEdge:
-    ; check:    $(nv11b=$V) = copy.i32 v11
-    ; not:      copy
-    ; check:   jump ebb1($nv11b, v12)
-
 ebb2:
     return v12
 }
diff --git a/cranelift/filetests/filetests/regalloc/reload-208-bb.clif b/cranelift/filetests/filetests/regalloc/reload-208-bb.clif
index 47438d7d61..d77e4d8ebc 100644
--- a/cranelift/filetests/filetests/regalloc/reload-208-bb.clif
+++ b/cranelift/filetests/filetests/regalloc/reload-208-bb.clif
@@ -44,6 +44,11 @@ ebb5:
     brnz v6, ebb2
     jump ebb3(v4)
 
+    ; check: ebb5:
+    ; check:   jump ebb3(v4)
+    ; check: $(splitEdge=$EBB):
+    ; nextln:  jump ebb3(v9)
+
 ebb3(v7: i32):
     call fn1(v0, v7)
     v26 = iconst.i32 0x4ffe
@@ -61,13 +66,10 @@ ebb6:
     v8 = iadd v25, v23
     v9 = load.i32 v8+56
     ; check: v9 = spill
-    ; check: brnz $V, $(splitEdge=$EBB)
+    ; check: brnz $V, $splitEdge
     brnz v9, ebb3(v9)
     jump ebb4
 
-    ; check: $splitEdge:
-    ; nextln:  jump ebb3(v9)
-
 ebb4:
     jump ebb2
 
diff --git a/cranelift/filetests/filetests/regalloc/x86-regres-bb.clif b/cranelift/filetests/filetests/regalloc/x86-regres-bb.clif
index 28eba22a64..060164034a 100644
--- a/cranelift/filetests/filetests/regalloc/x86-regres-bb.clif
+++ b/cranelift/filetests/filetests/regalloc/x86-regres-bb.clif
@@ -19,6 +19,9 @@ ebb0(v0: i32):
     v3 = iconst.i32 0
     jump ebb2(v3, v2, v0)
 
+    ; check: $(splitEdge=$EBB):
+    ; check:   jump ebb2($V, $V, v9)
+
 ebb2(v4: i32, v5: i32, v7: i32):
     ; check: ebb2
     v6 = iadd v4, v5
@@ -33,11 +36,9 @@ ebb2(v4: i32, v5: i32, v7: i32):
     ;
     ; We should be able to handle this situation without making copies of v9.
     brnz v9, ebb2(v5, v6, v9)
-    ; check:   brnz v9, $(splitEdge=$EBB)
+    ; check:   brnz v9, $splitEdge
     jump ebb3
 
-    ; check: $splitEdge:
-    ; check:   jump ebb2($V, $V, v9)
 ebb3:
     return v5
 }
diff --git a/cranelift/filetests/filetests/safepoint/basic-bb.clif b/cranelift/filetests/filetests/safepoint/basic-bb.clif
index b2ccedf559..14a43d09a6 100644
--- a/cranelift/filetests/filetests/safepoint/basic-bb.clif
+++ b/cranelift/filetests/filetests/safepoint/basic-bb.clif
@@ -32,11 +32,20 @@ function %test(i32, r64, r64) -> r64 {
 ; nextln:   v10 = copy v0
 ; nextln:   jump ebb1(v10)
 ; nextln: 
+; nextln: ebb7:
+; nextln:   regmove.i32 v5, %rcx -> %rax
+; nextln:   jump ebb1(v5)
+; nextln: 
 ; nextln: ebb1(v3: i32 [%rax]):
 ; nextln:   v8 = iconst.i32 1
 ; nextln:   v4 = isub v8, v3
 ; nextln:   jump ebb2(v4)
 ; nextln: 
+; nextln: ebb8:
+; nextln:   v9 = copy.i32 v0
+; nextln:   regmove v9, %rax -> %rcx
+; nextln:   jump ebb2(v9)
+; nextln: 
 ; nextln: ebb2(v5: i32 [%rcx]):
 ; nextln:   safepoint v1, v2
 ; nextln:   resumable_trap interrupt
@@ -60,13 +69,4 @@ function %test(i32, r64, r64) -> r64 {
 ; nextln: ebb6:
 ; nextln:   regmove.r64 v2, %rdx -> %rax
 ; nextln:   return v2
-; nextln: 
-; nextln: ebb7:
-; nextln:   regmove.i32 v5, %rcx -> %rax
-; nextln:   jump ebb1(v5)
-; nextln: 
-; nextln: ebb8:
-; nextln:   v9 = copy.i32 v0
-; nextln:   regmove v9, %rax -> %rcx
-; nextln:   jump ebb2(v9)
 ; nextln: }

From a9d9f9f1359c4e2c59f42100560803339d84908e Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 2 Oct 2019 11:10:19 +0200
Subject: [PATCH 2788/3084] Write a run test for brz.i128;

---
 .../filetests/isa/x86/br-i128-run.clif        | 38 +++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/br-i128-run.clif

diff --git a/cranelift/filetests/filetests/isa/x86/br-i128-run.clif b/cranelift/filetests/filetests/isa/x86/br-i128-run.clif
new file mode 100644
index 0000000000..bde3238462
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/br-i128-run.clif
@@ -0,0 +1,38 @@
+test run
+target x86_64
+
+function %br_false() -> b1 {
+ebb0:
+    v10 = iconst.i64 0x42
+    v11 = iconst.i64 0x00
+    v0 = iconcat v10, v11
+    brz v0, ebb2
+    jump ebb1
+
+ebb1:
+    v1 = bconst.b1 true
+    return v1
+
+ebb2:
+    v2 = bconst.b1 false
+    return v2
+}
+; run
+
+function %br_true() -> b1 {
+ebb0:
+    v10 = iconst.i64 0x00
+    v11 = iconst.i64 0x00
+    v0 = iconcat v10, v11
+    brz v0, ebb2
+    jump ebb1
+
+ebb1:
+    v1 = bconst.b1 false
+    return v1
+
+ebb2:
+    v2 = bconst.b1 true
+    return v2
+}
+; run

From dadfbcd32b79739ae8e8a5a0a1228d2a3de852bb Mon Sep 17 00:00:00 2001
From: Erin Power 
Date: Mon, 23 Sep 2019 17:55:49 +0200
Subject: [PATCH 2789/3084] Re-export ReplaceBuilder

ReplaceBuilder is available in the public API through
`DataFlowGraph::replace`, however it's documentation is not available
through rustdoc as the type isn't publicly importable.
---
 cranelift/codegen/src/ir/mod.rs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/ir/mod.rs b/cranelift/codegen/src/ir/mod.rs
index 2ee589f6af..9c03d569db 100644
--- a/cranelift/codegen/src/ir/mod.rs
+++ b/cranelift/codegen/src/ir/mod.rs
@@ -26,7 +26,9 @@ mod valueloc;
 #[cfg(feature = "enable-serde")]
 use serde::{Deserialize, Serialize};
 
-pub use crate::ir::builder::{InsertBuilder, InstBuilder, InstBuilderBase, InstInserterBase};
+pub use crate::ir::builder::{
+    InsertBuilder, InstBuilder, InstBuilderBase, InstInserterBase, ReplaceBuilder,
+};
 pub use crate::ir::constant::{ConstantData, ConstantOffset, ConstantPool};
 pub use crate::ir::dfg::{DataFlowGraph, ValueDef};
 pub use crate::ir::entities::{

From d25e611946cd14587b16a710e72d3f8285dff493 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 28 Sep 2019 15:39:58 +0200
Subject: [PATCH 2790/3084] Remove std feature from cranelift-bforest

---
 cranelift/bforest/Cargo.toml  | 5 -----
 cranelift/bforest/src/lib.rs  | 8 +-------
 cranelift/bforest/src/map.rs  | 6 +++---
 cranelift/bforest/src/node.rs | 2 +-
 cranelift/bforest/src/pool.rs | 2 +-
 cranelift/bforest/src/set.rs  | 6 +++---
 cranelift/codegen/Cargo.toml  | 3 +--
 7 files changed, 10 insertions(+), 22 deletions(-)

diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 960827da52..4dc95212e9 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -14,11 +14,6 @@ edition = "2018"
 [dependencies]
 cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
 
-[features]
-default = ["std"]
-std = ["cranelift-entity/std"]
-core = []
-
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/cranelift/bforest/src/lib.rs b/cranelift/bforest/src/lib.rs
index 5d33068b7a..2f5c1060a0 100644
--- a/cranelift/bforest/src/lib.rs
+++ b/cranelift/bforest/src/lib.rs
@@ -15,7 +15,6 @@
 
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "std", warn(unstable_features))]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
@@ -34,13 +33,8 @@
 #![no_std]
 
 #[cfg(test)]
-#[cfg(not(feature = "std"))]
 #[macro_use]
-extern crate alloc as std;
-#[cfg(test)]
-#[cfg(feature = "std")]
-#[macro_use]
-extern crate std;
+extern crate alloc;
 
 #[macro_use]
 extern crate cranelift_entity as entity;
diff --git a/cranelift/bforest/src/map.rs b/cranelift/bforest/src/map.rs
index 3cf8421a30..931253cae5 100644
--- a/cranelift/bforest/src/map.rs
+++ b/cranelift/bforest/src/map.rs
@@ -231,7 +231,7 @@ where
 
     /// Get a text version of the path to `key`.
     fn tpath>(&self, key: K, forest: &MapForest, comp: &C) -> String {
-        use std::string::ToString;
+        use alloc::string::ToString;
         match self.root.expand() {
             None => "map(empty)".to_string(),
             Some(root) => {
@@ -420,7 +420,7 @@ where
 
     /// Get a text version of the path to the current position.
     fn tpath(&self) -> String {
-        use std::string::ToString;
+        use alloc::string::ToString;
         self.path.to_string()
     }
 }
@@ -430,7 +430,7 @@ mod tests {
     use super::super::NodeData;
     use super::*;
     use core::mem;
-    use std::vec::Vec;
+    use alloc::vec::Vec;
 
     #[test]
     fn node_size() {
diff --git a/cranelift/bforest/src/node.rs b/cranelift/bforest/src/node.rs
index f0dbc4d7f5..260494b171 100644
--- a/cranelift/bforest/src/node.rs
+++ b/cranelift/bforest/src/node.rs
@@ -585,7 +585,7 @@ where
 mod tests {
     use super::*;
     use core::mem;
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     // Forest impl for a set implementation.
     struct TF();
diff --git a/cranelift/bforest/src/pool.rs b/cranelift/bforest/src/pool.rs
index 0848089eb6..6b4bae020d 100644
--- a/cranelift/bforest/src/pool.rs
+++ b/cranelift/bforest/src/pool.rs
@@ -86,7 +86,7 @@ impl NodePool {
         use crate::entity::EntitySet;
         use core::borrow::Borrow;
         use core::cmp::Ordering;
-        use std::vec::Vec;
+        use alloc::vec::Vec;
 
         // The root node can't be an inner node with just a single sub-tree. It should have been
         // pruned.
diff --git a/cranelift/bforest/src/set.rs b/cranelift/bforest/src/set.rs
index 72e517561f..d22cb8285e 100644
--- a/cranelift/bforest/src/set.rs
+++ b/cranelift/bforest/src/set.rs
@@ -6,7 +6,7 @@ use crate::packed_option::PackedOption;
 use core::fmt;
 use core::marker::PhantomData;
 #[cfg(test)]
-use std::string::String;
+use alloc::string::String;
 
 /// Tag type defining forest types for a set.
 struct SetTypes(PhantomData);
@@ -321,7 +321,7 @@ where
 
     /// Get a text version of the path to the current position.
     fn tpath(&self) -> String {
-        use std::string::ToString;
+        use alloc::string::ToString;
         self.path.to_string()
     }
 }
@@ -358,7 +358,7 @@ mod tests {
     use super::super::NodeData;
     use super::*;
     use core::mem;
-    use std::vec::Vec;
+    use alloc::vec::Vec;
 
     #[test]
     fn node_size() {
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index e38f06d608..dae5e033c7 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -15,7 +15,7 @@ edition = "2018"
 [dependencies]
 cranelift-codegen-shared = { path = "./shared", version = "0.44.0" }
 cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.44.0", default-features = false }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.44.0" }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashbrown = { version = "0.6", optional = true }
@@ -39,7 +39,6 @@ default = ["std"]
 # features need to be enabled.
 std = [
     "cranelift-entity/std",
-    "cranelift-bforest/std",
     "cranelift-codegen-meta/std"
 ]
 

From a114423d0a649079586d8081eb854176df9e7fd9 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 28 Sep 2019 15:46:03 +0200
Subject: [PATCH 2791/3084] Remove std feature from cranelift-entity

---
 cranelift/codegen/Cargo.toml        | 7 ++-----
 cranelift/codegen/meta/Cargo.toml   | 6 +++---
 cranelift/entity/Cargo.toml         | 3 ---
 cranelift/entity/src/boxed_slice.rs | 4 ++--
 cranelift/entity/src/lib.rs         | 8 +-------
 cranelift/entity/src/list.rs        | 2 +-
 cranelift/entity/src/map.rs         | 4 ++--
 cranelift/entity/src/primary.rs     | 4 ++--
 cranelift/entity/src/set.rs         | 2 +-
 cranelift/entity/src/sparse.rs      | 2 +-
 cranelift/module/Cargo.toml         | 4 ++--
 cranelift/preopt/Cargo.toml         | 4 ++--
 cranelift/wasm/Cargo.toml           | 2 +-
 13 files changed, 20 insertions(+), 32 deletions(-)

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index dae5e033c7..2739160aa2 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -14,7 +14,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen-shared = { path = "./shared", version = "0.44.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.44.0" }
 cranelift-bforest = { path = "../cranelift-bforest", version = "0.44.0" }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
@@ -37,10 +37,7 @@ default = ["std"]
 # The "std" feature enables use of libstd. The "core" feature enables use
 # of some minimal std-like replacement libraries. At least one of these two
 # features need to be enabled.
-std = [
-    "cranelift-entity/std",
-    "cranelift-codegen-meta/std"
-]
+std = ["cranelift-codegen-meta/std"]
 
 # The "core" features enables use of "hashbrown" since core doesn't have
 # a HashMap implementation, and a workaround for Cargo #4866.
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 2d75957355..74cc0accd3 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -10,7 +10,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen-shared = { path = "../shared", version = "0.44.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.44.0", default-features = false }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.44.0" }
 
 [badges]
 maintenance = { status = "experimental" }
@@ -18,6 +18,6 @@ travis-ci = { repository = "CraneStation/cranelift" }
 
 [features]
 default = ["std"]
-std = ["cranelift-entity/std"]
+std = []
 # The "core" feature enables a workaround for Cargo #4866.
-core = ["cranelift-entity/core"]
+core = []
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 65b9bc7aaa..e6f1ec6310 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -15,9 +15,6 @@ edition = "2018"
 serde = { version = "1.0.94", features = ["derive"], optional = true }
 
 [features]
-default = ["std"]
-std = []
-core = []
 enable-serde = ["serde"]
 
 [badges]
diff --git a/cranelift/entity/src/boxed_slice.rs b/cranelift/entity/src/boxed_slice.rs
index 2030c6b536..302cab0235 100644
--- a/cranelift/entity/src/boxed_slice.rs
+++ b/cranelift/entity/src/boxed_slice.rs
@@ -6,7 +6,7 @@ use crate::EntityRef;
 use core::marker::PhantomData;
 use core::ops::{Index, IndexMut};
 use core::slice;
-use std::boxed::Box;
+use alloc::boxed::Box;
 
 /// A slice mapping `K -> V` allocating dense entity references.
 ///
@@ -141,7 +141,7 @@ where
 mod tests {
     use super::*;
     use crate::primary::PrimaryMap;
-    use std::vec::Vec;
+    use alloc::vec::Vec;
 
     // `EntityRef` impl for testing.
     #[derive(Clone, Copy, Debug, PartialEq, Eq)]
diff --git a/cranelift/entity/src/lib.rs b/cranelift/entity/src/lib.rs
index aa10264ab8..793a8af27a 100644
--- a/cranelift/entity/src/lib.rs
+++ b/cranelift/entity/src/lib.rs
@@ -31,7 +31,6 @@
 
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
-#![cfg_attr(feature = "std", deny(unstable_features))]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
 #![cfg_attr(
     feature = "cargo-clippy",
@@ -52,12 +51,7 @@
 )]
 #![no_std]
 
-#[cfg(not(feature = "std"))]
-#[macro_use]
-extern crate alloc as std;
-#[cfg(feature = "std")]
-#[macro_use]
-extern crate std;
+extern crate alloc;
 
 // Re-export core so that the macros works with both std and no_std crates
 #[doc(hidden)]
diff --git a/cranelift/entity/src/list.rs b/cranelift/entity/src/list.rs
index 009b3d70b1..263a981bb3 100644
--- a/cranelift/entity/src/list.rs
+++ b/cranelift/entity/src/list.rs
@@ -3,7 +3,7 @@ use crate::packed_option::ReservedValue;
 use crate::EntityRef;
 use core::marker::PhantomData;
 use core::mem;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// A small list of entity references allocated from a pool.
 ///
diff --git a/cranelift/entity/src/map.rs b/cranelift/entity/src/map.rs
index 884f812fb2..937c4a49d9 100644
--- a/cranelift/entity/src/map.rs
+++ b/cranelift/entity/src/map.rs
@@ -13,7 +13,7 @@ use serde::{
     ser::{SerializeSeq, Serializer},
     Deserialize, Serialize,
 };
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// A mapping `K -> V` for densely indexed entity references.
 ///
@@ -222,7 +222,7 @@ where
     where
         D: Deserializer<'de>,
     {
-        use std::fmt;
+        use alloc::fmt;
         struct SecondaryMapVisitor {
             unused: PhantomData V>,
         }
diff --git a/cranelift/entity/src/primary.rs b/cranelift/entity/src/primary.rs
index 9cde5e779d..a464fe60f8 100644
--- a/cranelift/entity/src/primary.rs
+++ b/cranelift/entity/src/primary.rs
@@ -9,8 +9,8 @@ use core::ops::{Index, IndexMut};
 use core::slice;
 #[cfg(feature = "enable-serde")]
 use serde::{Deserialize, Serialize};
-use std::boxed::Box;
-use std::vec::Vec;
+use alloc::boxed::Box;
+use alloc::vec::Vec;
 
 /// A primary mapping `K -> V` allocating dense entity references.
 ///
diff --git a/cranelift/entity/src/set.rs b/cranelift/entity/src/set.rs
index fa74ba453f..2379fe2a7d 100644
--- a/cranelift/entity/src/set.rs
+++ b/cranelift/entity/src/set.rs
@@ -3,7 +3,7 @@
 use crate::keys::Keys;
 use crate::EntityRef;
 use core::marker::PhantomData;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// A set of `K` for densely indexed entity references.
 ///
diff --git a/cranelift/entity/src/sparse.rs b/cranelift/entity/src/sparse.rs
index 83c519f47f..43d8d4d431 100644
--- a/cranelift/entity/src/sparse.rs
+++ b/cranelift/entity/src/sparse.rs
@@ -12,7 +12,7 @@ use crate::EntityRef;
 use core::mem;
 use core::slice;
 use core::u32;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Trait for extracting keys from values stored in a `SparseMap`.
 ///
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 2ce44cc32c..c67957c6f6 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -12,14 +12,14 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.44.0" }
 hashbrown = { version = "0.6", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
 
 [features]
 default = ["std"]
-std = ["cranelift-codegen/std", "cranelift-entity/std"]
+std = ["cranelift-codegen/std"]
 core = ["hashbrown", "cranelift-codegen/core"]
 
 [badges]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 64f246da46..c18188cf77 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -13,14 +13,14 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.44.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
 
 [features]
 default = ["std"]
-std = ["cranelift-codegen/std", "cranelift-entity/std"]
+std = ["cranelift-codegen/std"]
 core = ["cranelift-codegen/core"]
 
 [badges]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 5e2a9e1cbe..c1e4bdd80b 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -13,7 +13,7 @@ edition = "2018"
 [dependencies]
 wasmparser = { version = "0.39.1", default-features = false }
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.44.0" }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.44.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }

From 10e226f9ff67daea9d948573821cb4f8492f1402 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 28 Sep 2019 15:52:23 +0200
Subject: [PATCH 2792/3084] Always use extern crate std in cranelift-codegen

---
 cranelift/codegen/src/abi.rs                         | 2 +-
 cranelift/codegen/src/binemit/relaxation.rs          | 2 +-
 cranelift/codegen/src/binemit/stackmap.rs            | 4 ++--
 cranelift/codegen/src/cfg_printer.rs                 | 2 +-
 cranelift/codegen/src/context.rs                     | 2 +-
 cranelift/codegen/src/dominator_tree.rs              | 2 +-
 cranelift/codegen/src/flowgraph.rs                   | 2 +-
 cranelift/codegen/src/ir/constant.rs                 | 4 ++--
 cranelift/codegen/src/ir/dfg.rs                      | 4 ++--
 cranelift/codegen/src/ir/entities.rs                 | 2 +-
 cranelift/codegen/src/ir/extfunc.rs                  | 4 ++--
 cranelift/codegen/src/ir/extname.rs                  | 2 +-
 cranelift/codegen/src/ir/immediates.rs               | 4 ++--
 cranelift/codegen/src/ir/instructions.rs             | 4 ++--
 cranelift/codegen/src/ir/jumptable.rs                | 4 ++--
 cranelift/codegen/src/ir/layout.rs                   | 2 +-
 cranelift/codegen/src/ir/libcall.rs                  | 2 +-
 cranelift/codegen/src/ir/mod.rs                      | 2 +-
 cranelift/codegen/src/ir/progpoint.rs                | 2 +-
 cranelift/codegen/src/ir/sourceloc.rs                | 2 +-
 cranelift/codegen/src/ir/stackslot.rs                | 4 ++--
 cranelift/codegen/src/ir/trapcode.rs                 | 2 +-
 cranelift/codegen/src/ir/types.rs                    | 2 +-
 cranelift/codegen/src/isa/arm32/mod.rs               | 2 +-
 cranelift/codegen/src/isa/arm32/registers.rs         | 2 +-
 cranelift/codegen/src/isa/arm64/mod.rs               | 2 +-
 cranelift/codegen/src/isa/arm64/registers.rs         | 2 +-
 cranelift/codegen/src/isa/mod.rs                     | 6 +++---
 cranelift/codegen/src/isa/riscv/mod.rs               | 4 ++--
 cranelift/codegen/src/isa/riscv/registers.rs         | 2 +-
 cranelift/codegen/src/isa/riscv/settings.rs          | 2 +-
 cranelift/codegen/src/isa/x86/mod.rs                 | 2 +-
 cranelift/codegen/src/isa/x86/registers.rs           | 2 +-
 cranelift/codegen/src/iterators.rs                   | 2 +-
 cranelift/codegen/src/legalizer/boundary.rs          | 2 +-
 cranelift/codegen/src/legalizer/libcall.rs           | 2 +-
 cranelift/codegen/src/legalizer/mod.rs               | 6 +++---
 cranelift/codegen/src/legalizer/split.rs             | 2 +-
 cranelift/codegen/src/lib.rs                         | 7 +++----
 cranelift/codegen/src/licm.rs                        | 2 +-
 cranelift/codegen/src/loop_analysis.rs               | 4 ++--
 cranelift/codegen/src/partition_slice.rs             | 2 +-
 cranelift/codegen/src/print_errors.rs                | 6 +++---
 cranelift/codegen/src/redundant_reload_remover.rs    | 2 +-
 cranelift/codegen/src/regalloc/branch_splitting.rs   | 2 +-
 cranelift/codegen/src/regalloc/coalescing.rs         | 2 +-
 cranelift/codegen/src/regalloc/live_value_tracker.rs | 2 +-
 cranelift/codegen/src/regalloc/liveness.rs           | 2 +-
 cranelift/codegen/src/regalloc/liverange.rs          | 2 +-
 cranelift/codegen/src/regalloc/pressure.rs           | 2 +-
 cranelift/codegen/src/regalloc/register_set.rs       | 2 +-
 cranelift/codegen/src/regalloc/reload.rs             | 2 +-
 cranelift/codegen/src/regalloc/safepoint.rs          | 2 +-
 cranelift/codegen/src/regalloc/solver.rs             | 4 ++--
 cranelift/codegen/src/regalloc/spilling.rs           | 2 +-
 cranelift/codegen/src/regalloc/virtregs.rs           | 2 +-
 cranelift/codegen/src/settings.rs                    | 6 +++---
 cranelift/codegen/src/simple_gvn.rs                  | 2 +-
 cranelift/codegen/src/timing.rs                      | 2 +-
 cranelift/codegen/src/topo_order.rs                  | 2 +-
 cranelift/codegen/src/value_label.rs                 | 4 ++--
 cranelift/codegen/src/verifier/mod.rs                | 6 +++---
 cranelift/codegen/src/write.rs                       | 6 +++---
 63 files changed, 89 insertions(+), 90 deletions(-)

diff --git a/cranelift/codegen/src/abi.rs b/cranelift/codegen/src/abi.rs
index 4c0b8d4dbc..8564fd5a24 100644
--- a/cranelift/codegen/src/abi.rs
+++ b/cranelift/codegen/src/abi.rs
@@ -5,7 +5,7 @@
 
 use crate::ir::{AbiParam, ArgumentExtension, ArgumentLoc, Type};
 use core::cmp::Ordering;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Legalization action to perform on a single argument or return value when converting a
 /// signature.
diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index 43f361885a..806129e9af 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -227,7 +227,7 @@ fn try_fold_redundant_jump(
     }
 
     // Build a value list of first_args (unchanged) followed by second_params (rewritten).
-    let arguments_vec: std::vec::Vec<_> = first_args
+    let arguments_vec: alloc::vec::Vec<_> = first_args
         .iter()
         .chain(second_params.iter())
         .map(|x| *x)
diff --git a/cranelift/codegen/src/binemit/stackmap.rs b/cranelift/codegen/src/binemit/stackmap.rs
index 5d6844dcf3..4d513f7322 100644
--- a/cranelift/codegen/src/binemit/stackmap.rs
+++ b/cranelift/codegen/src/binemit/stackmap.rs
@@ -1,7 +1,7 @@
 use crate::bitset::BitSet;
 use crate::ir;
 use crate::isa::TargetIsa;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 type Num = u32;
 const NUM_BITS: usize = core::mem::size_of::() * 8;
@@ -42,7 +42,7 @@ impl Stackmap {
         let frame_size = stack.frame_size.unwrap();
         let word_size = ir::stackslot::StackSize::from(isa.pointer_bytes());
         let num_words = (frame_size / word_size) as usize;
-        let mut vec = std::vec::Vec::with_capacity(num_words);
+        let mut vec = alloc::vec::Vec::with_capacity(num_words);
 
         vec.resize(num_words, false);
 
diff --git a/cranelift/codegen/src/cfg_printer.rs b/cranelift/codegen/src/cfg_printer.rs
index e3c62820cd..c67e8b1eec 100644
--- a/cranelift/codegen/src/cfg_printer.rs
+++ b/cranelift/codegen/src/cfg_printer.rs
@@ -1,7 +1,7 @@
 //! The `CFGPrinter` utility.
 
 use core::fmt::{Display, Formatter, Result, Write};
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 use crate::entity::SecondaryMap;
 use crate::flowgraph::{BasicBlock, ControlFlowGraph};
diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index 8bbe2893de..aad08a5481 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -34,7 +34,7 @@ use crate::unreachable_code::eliminate_unreachable_code;
 use crate::value_label::{build_value_labels_ranges, ComparableSourceLoc, ValueLabelsRanges};
 use crate::verifier::{verify_context, verify_locations, VerifierErrors, VerifierResult};
 use log::debug;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Persistent data structures and compilation pipeline.
 pub struct Context {
diff --git a/cranelift/codegen/src/dominator_tree.rs b/cranelift/codegen/src/dominator_tree.rs
index 8191c310eb..737019d315 100644
--- a/cranelift/codegen/src/dominator_tree.rs
+++ b/cranelift/codegen/src/dominator_tree.rs
@@ -9,7 +9,7 @@ use crate::timing;
 use core::cmp;
 use core::cmp::Ordering;
 use core::mem;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// RPO numbers are not first assigned in a contiguous way but as multiples of STRIDE, to leave
 /// room for modifications of the dominator tree.
diff --git a/cranelift/codegen/src/flowgraph.rs b/cranelift/codegen/src/flowgraph.rs
index 645b7d4fff..f53cfa6bff 100644
--- a/cranelift/codegen/src/flowgraph.rs
+++ b/cranelift/codegen/src/flowgraph.rs
@@ -214,7 +214,7 @@ mod tests {
     use super::*;
     use crate::cursor::{Cursor, FuncCursor};
     use crate::ir::{types, Function, InstBuilder};
-    use std::vec::Vec;
+    use alloc::vec::Vec;
 
     #[test]
     fn empty() {
diff --git a/cranelift/codegen/src/ir/constant.rs b/cranelift/codegen/src/ir/constant.rs
index 9f21209c4c..c4d7cb0898 100644
--- a/cranelift/codegen/src/ir/constant.rs
+++ b/cranelift/codegen/src/ir/constant.rs
@@ -11,8 +11,8 @@
 use crate::ir::Constant;
 use crate::HashMap;
 use cranelift_entity::EntityRef;
-use std::collections::BTreeMap;
-use std::vec::Vec;
+use alloc::collections::BTreeMap;
+use alloc::vec::Vec;
 
 /// This type describes the actual constant data.
 pub type ConstantData = Vec;
diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index e4ec9c6be8..34ed10622d 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -19,7 +19,7 @@ use core::iter;
 use core::mem;
 use core::ops::{Index, IndexMut};
 use core::u16;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// A data flow graph defines all instructions and extended basic blocks in a function as well as
 /// the data flow dependencies between them. The DFG also tracks values which can be either
@@ -1093,7 +1093,7 @@ mod tests {
     use crate::cursor::{Cursor, FuncCursor};
     use crate::ir::types;
     use crate::ir::{Function, InstructionData, Opcode, TrapCode};
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn make_inst() {
diff --git a/cranelift/codegen/src/ir/entities.rs b/cranelift/codegen/src/ir/entities.rs
index 1f8e1fc6a2..552a9f3ae0 100644
--- a/cranelift/codegen/src/ir/entities.rs
+++ b/cranelift/codegen/src/ir/entities.rs
@@ -456,7 +456,7 @@ impl From
 for AnyEntity {
 mod tests {
     use super::*;
     use core::u32;
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn value_with_number() {
diff --git a/cranelift/codegen/src/ir/extfunc.rs b/cranelift/codegen/src/ir/extfunc.rs
index 0b74fd3d24..f09b428833 100644
--- a/cranelift/codegen/src/ir/extfunc.rs
+++ b/cranelift/codegen/src/ir/extfunc.rs
@@ -9,7 +9,7 @@ use crate::ir::{ArgumentLoc, ExternalName, SigRef, Type};
 use crate::isa::{CallConv, RegInfo, RegUnit};
 use core::fmt;
 use core::str::FromStr;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Function signature.
 ///
@@ -335,7 +335,7 @@ impl fmt::Display for ExtFuncData {
 mod tests {
     use super::*;
     use crate::ir::types::{B8, F32, I32};
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn argument_type() {
diff --git a/cranelift/codegen/src/ir/extname.rs b/cranelift/codegen/src/ir/extname.rs
index 5fd8600882..d153db9cbd 100644
--- a/cranelift/codegen/src/ir/extname.rs
+++ b/cranelift/codegen/src/ir/extname.rs
@@ -121,7 +121,7 @@ mod tests {
     use super::ExternalName;
     use crate::ir::LibCall;
     use core::u32;
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn display_testcase() {
diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index eb6dfad34b..5d3807671f 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -8,7 +8,7 @@ use core::fmt::{self, Display, Formatter};
 use core::iter::FromIterator;
 use core::str::{from_utf8, FromStr};
 use core::{i32, u32};
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Convert a type into a vector of bytes; all implementors in this file must use little-endian
 /// orderings of bytes to match WebAssembly's little-endianness.
@@ -940,7 +940,7 @@ mod tests {
     use core::mem;
     use core::str::FromStr;
     use core::{f32, f64};
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn format_imm64() {
diff --git a/cranelift/codegen/src/ir/instructions.rs b/cranelift/codegen/src/ir/instructions.rs
index b3c3d0d211..97c84110aa 100644
--- a/cranelift/codegen/src/ir/instructions.rs
+++ b/cranelift/codegen/src/ir/instructions.rs
@@ -9,7 +9,7 @@
 use core::fmt::{self, Display, Formatter};
 use core::ops::{Deref, DerefMut};
 use core::str::FromStr;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 use crate::ir;
 use crate::ir::types;
@@ -560,7 +560,7 @@ pub enum ResolvedConstraint {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn opcodes() {
diff --git a/cranelift/codegen/src/ir/jumptable.rs b/cranelift/codegen/src/ir/jumptable.rs
index a40ee48e74..596e668e89 100644
--- a/cranelift/codegen/src/ir/jumptable.rs
+++ b/cranelift/codegen/src/ir/jumptable.rs
@@ -6,7 +6,7 @@
 use crate::ir::entities::Ebb;
 use core::fmt::{self, Display, Formatter};
 use core::slice::{Iter, IterMut};
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Contents of a jump table.
 ///
@@ -85,7 +85,7 @@ mod tests {
     use super::JumpTableData;
     use crate::entity::EntityRef;
     use crate::ir::Ebb;
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn empty() {
diff --git a/cranelift/codegen/src/ir/layout.rs b/cranelift/codegen/src/ir/layout.rs
index 632c03dc59..3d1f5e975c 100644
--- a/cranelift/codegen/src/ir/layout.rs
+++ b/cranelift/codegen/src/ir/layout.rs
@@ -751,7 +751,7 @@ mod tests {
     use crate::entity::EntityRef;
     use crate::ir::{Ebb, Inst, ProgramOrder, SourceLoc};
     use core::cmp::Ordering;
-    use std::vec::Vec;
+    use alloc::vec::Vec;
 
     struct LayoutCursor<'f> {
         /// Borrowed function layout. Public so it can be re-borrowed from this cursor.
diff --git a/cranelift/codegen/src/ir/libcall.rs b/cranelift/codegen/src/ir/libcall.rs
index 1e19891221..52c4c40dad 100644
--- a/cranelift/codegen/src/ir/libcall.rs
+++ b/cranelift/codegen/src/ir/libcall.rs
@@ -209,7 +209,7 @@ fn make_funcref(
 #[cfg(test)]
 mod tests {
     use super::*;
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn display() {
diff --git a/cranelift/codegen/src/ir/mod.rs b/cranelift/codegen/src/ir/mod.rs
index 9c03d569db..8b89227ff2 100644
--- a/cranelift/codegen/src/ir/mod.rs
+++ b/cranelift/codegen/src/ir/mod.rs
@@ -100,7 +100,7 @@ pub struct ValueLabelStart {
 #[derive(Debug, Clone)]
 pub enum ValueLabelAssignments {
     /// Original value labels assigned at transform.
-    Starts(std::vec::Vec),
+    Starts(alloc::vec::Vec),
 
     /// A value alias to original value.
     Alias {
diff --git a/cranelift/codegen/src/ir/progpoint.rs b/cranelift/codegen/src/ir/progpoint.rs
index bd03c120cd..32c349aa00 100644
--- a/cranelift/codegen/src/ir/progpoint.rs
+++ b/cranelift/codegen/src/ir/progpoint.rs
@@ -148,7 +148,7 @@ mod tests {
     use super::*;
     use crate::entity::EntityRef;
     use crate::ir::{Ebb, Inst};
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn convert() {
diff --git a/cranelift/codegen/src/ir/sourceloc.rs b/cranelift/codegen/src/ir/sourceloc.rs
index a0d051d3aa..b8f162366f 100644
--- a/cranelift/codegen/src/ir/sourceloc.rs
+++ b/cranelift/codegen/src/ir/sourceloc.rs
@@ -54,7 +54,7 @@ impl fmt::Display for SourceLoc {
 #[cfg(test)]
 mod tests {
     use crate::ir::SourceLoc;
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn display() {
diff --git a/cranelift/codegen/src/ir/stackslot.rs b/cranelift/codegen/src/ir/stackslot.rs
index 0b488582d4..fb526ab50e 100644
--- a/cranelift/codegen/src/ir/stackslot.rs
+++ b/cranelift/codegen/src/ir/stackslot.rs
@@ -11,7 +11,7 @@ use core::fmt;
 use core::ops::{Index, IndexMut};
 use core::slice;
 use core::str::FromStr;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 #[cfg(feature = "enable-serde")]
 use serde::{Deserialize, Serialize};
@@ -343,7 +343,7 @@ mod tests {
     use super::*;
     use crate::ir::types;
     use crate::ir::Function;
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn stack_slot() {
diff --git a/cranelift/codegen/src/ir/trapcode.rs b/cranelift/codegen/src/ir/trapcode.rs
index 5621e1d91c..3f8ffe3fb5 100644
--- a/cranelift/codegen/src/ir/trapcode.rs
+++ b/cranelift/codegen/src/ir/trapcode.rs
@@ -103,7 +103,7 @@ impl FromStr for TrapCode {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     // Everything but user-defined codes.
     const CODES: [TrapCode; 11] = [
diff --git a/cranelift/codegen/src/ir/types.rs b/cranelift/codegen/src/ir/types.rs
index 10fca8aaa8..4ada74b070 100644
--- a/cranelift/codegen/src/ir/types.rs
+++ b/cranelift/codegen/src/ir/types.rs
@@ -363,7 +363,7 @@ impl Default for Type {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn basic_scalars() {
diff --git a/cranelift/codegen/src/isa/arm32/mod.rs b/cranelift/codegen/src/isa/arm32/mod.rs
index ed5854593b..f377560581 100644
--- a/cranelift/codegen/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/src/isa/arm32/mod.rs
@@ -16,7 +16,7 @@ use crate::isa::Builder as IsaBuilder;
 use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
 use core::fmt;
-use std::boxed::Box;
+use alloc::boxed::Box;
 use target_lexicon::{Architecture, Triple};
 
 #[allow(dead_code)]
diff --git a/cranelift/codegen/src/isa/arm32/registers.rs b/cranelift/codegen/src/isa/arm32/registers.rs
index 70549528cf..df555b4043 100644
--- a/cranelift/codegen/src/isa/arm32/registers.rs
+++ b/cranelift/codegen/src/isa/arm32/registers.rs
@@ -8,7 +8,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-arm32.rs"));
 mod tests {
     use super::{D, GPR, INFO, S};
     use crate::isa::RegUnit;
-    use std::string::{String, ToString};
+    use alloc::string::{String, ToString};
 
     #[test]
     fn unit_encodings() {
diff --git a/cranelift/codegen/src/isa/arm64/mod.rs b/cranelift/codegen/src/isa/arm64/mod.rs
index 71ae93c0fe..84afc33cbd 100644
--- a/cranelift/codegen/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/src/isa/arm64/mod.rs
@@ -16,7 +16,7 @@ use crate::isa::Builder as IsaBuilder;
 use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
 use core::fmt;
-use std::boxed::Box;
+use alloc::boxed::Box;
 use target_lexicon::Triple;
 
 #[allow(dead_code)]
diff --git a/cranelift/codegen/src/isa/arm64/registers.rs b/cranelift/codegen/src/isa/arm64/registers.rs
index 91ca256bbe..c02f6b7d4d 100644
--- a/cranelift/codegen/src/isa/arm64/registers.rs
+++ b/cranelift/codegen/src/isa/arm64/registers.rs
@@ -8,7 +8,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-arm64.rs"));
 mod tests {
     use super::INFO;
     use crate::isa::RegUnit;
-    use std::string::{String, ToString};
+    use alloc::string::{String, ToString};
 
     #[test]
     fn unit_encodings() {
diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index 6457eb9ad7..8c6b3bc203 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -25,7 +25,7 @@
 //! # fn main() {
 //! use cranelift_codegen::isa;
 //! use cranelift_codegen::settings::{self, Configurable};
-//! use std::str::FromStr;
+//! use alloc::str::FromStr;
 //! use target_lexicon::Triple;
 //!
 //! let shared_builder = settings::builder();
@@ -65,7 +65,7 @@ use crate::settings::SetResult;
 use crate::timing;
 use core::fmt;
 use failure_derive::Fail;
-use std::boxed::Box;
+use alloc::boxed::Box;
 use target_lexicon::{triple, Architecture, PointerWidth, Triple};
 
 #[cfg(feature = "riscv")]
@@ -119,7 +119,7 @@ pub fn lookup(triple: Triple) -> Result {
 /// Look for a supported ISA with the given `name`.
 /// Return a builder that can create a corresponding `TargetIsa`.
 pub fn lookup_by_name(name: &str) -> Result {
-    use std::str::FromStr;
+    use alloc::str::FromStr;
     lookup(triple!(name))
 }
 
diff --git a/cranelift/codegen/src/isa/riscv/mod.rs b/cranelift/codegen/src/isa/riscv/mod.rs
index 93ef3787f0..e4bd6c9bbd 100644
--- a/cranelift/codegen/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/src/isa/riscv/mod.rs
@@ -16,7 +16,7 @@ use crate::isa::Builder as IsaBuilder;
 use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
 use core::fmt;
-use std::boxed::Box;
+use alloc::boxed::Box;
 use target_lexicon::{PointerWidth, Triple};
 
 #[allow(dead_code)]
@@ -138,7 +138,7 @@ mod tests {
     use crate::isa;
     use crate::settings::{self, Configurable};
     use core::str::FromStr;
-    use std::string::{String, ToString};
+    use alloc::string::{String, ToString};
     use target_lexicon::triple;
 
     fn encstr(isa: &dyn isa::TargetIsa, enc: Result) -> String {
diff --git a/cranelift/codegen/src/isa/riscv/registers.rs b/cranelift/codegen/src/isa/riscv/registers.rs
index 4f167bae8f..9043b7f65f 100644
--- a/cranelift/codegen/src/isa/riscv/registers.rs
+++ b/cranelift/codegen/src/isa/riscv/registers.rs
@@ -8,7 +8,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-riscv.rs"));
 mod tests {
     use super::{FPR, GPR, INFO};
     use crate::isa::RegUnit;
-    use std::string::{String, ToString};
+    use alloc::string::{String, ToString};
 
     #[test]
     fn unit_encodings() {
diff --git a/cranelift/codegen/src/isa/riscv/settings.rs b/cranelift/codegen/src/isa/riscv/settings.rs
index 24c0e2af11..40aa3bed2b 100644
--- a/cranelift/codegen/src/isa/riscv/settings.rs
+++ b/cranelift/codegen/src/isa/riscv/settings.rs
@@ -12,7 +12,7 @@ include!(concat!(env!("OUT_DIR"), "/settings-riscv.rs"));
 mod tests {
     use super::{builder, Flags};
     use crate::settings::{self, Configurable};
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn display_default() {
diff --git a/cranelift/codegen/src/isa/x86/mod.rs b/cranelift/codegen/src/isa/x86/mod.rs
index 8136170cd7..35ca485bd8 100644
--- a/cranelift/codegen/src/isa/x86/mod.rs
+++ b/cranelift/codegen/src/isa/x86/mod.rs
@@ -18,7 +18,7 @@ use crate::regalloc;
 use crate::result::CodegenResult;
 use crate::timing;
 use core::fmt;
-use std::boxed::Box;
+use alloc::boxed::Box;
 use target_lexicon::{PointerWidth, Triple};
 
 #[allow(dead_code)]
diff --git a/cranelift/codegen/src/isa/x86/registers.rs b/cranelift/codegen/src/isa/x86/registers.rs
index a5ebf438d4..130cf41d79 100644
--- a/cranelift/codegen/src/isa/x86/registers.rs
+++ b/cranelift/codegen/src/isa/x86/registers.rs
@@ -8,7 +8,7 @@ include!(concat!(env!("OUT_DIR"), "/registers-x86.rs"));
 mod tests {
     use super::*;
     use crate::isa::RegUnit;
-    use std::string::{String, ToString};
+    use alloc::string::{String, ToString};
 
     #[test]
     fn unit_encodings() {
diff --git a/cranelift/codegen/src/iterators.rs b/cranelift/codegen/src/iterators.rs
index 71cb8cbdf6..ca9c4ab26b 100644
--- a/cranelift/codegen/src/iterators.rs
+++ b/cranelift/codegen/src/iterators.rs
@@ -44,7 +44,7 @@ where
 
 #[cfg(test)]
 mod tests {
-    use std::vec::Vec;
+    use alloc::vec::Vec;
 
     #[test]
     fn adjpairs() {
diff --git a/cranelift/codegen/src/legalizer/boundary.rs b/cranelift/codegen/src/legalizer/boundary.rs
index 8912743dd5..d95d6d4239 100644
--- a/cranelift/codegen/src/legalizer/boundary.rs
+++ b/cranelift/codegen/src/legalizer/boundary.rs
@@ -28,7 +28,7 @@ use crate::ir::{
 use crate::isa::TargetIsa;
 use crate::legalizer::split::{isplit, vsplit};
 use log::debug;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Legalize all the function signatures in `func`.
 ///
diff --git a/cranelift/codegen/src/legalizer/libcall.rs b/cranelift/codegen/src/legalizer/libcall.rs
index 1b11ae7593..b34a90b519 100644
--- a/cranelift/codegen/src/legalizer/libcall.rs
+++ b/cranelift/codegen/src/legalizer/libcall.rs
@@ -4,7 +4,7 @@ use crate::ir;
 use crate::ir::{get_libcall_funcref, InstBuilder};
 use crate::isa::{CallConv, TargetIsa};
 use crate::legalizer::boundary::legalize_libcall_signature;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Try to expand `inst` as a library call, returning true is successful.
 pub fn expand_as_libcall(inst: ir::Inst, func: &mut ir::Function, isa: &dyn TargetIsa) -> bool {
diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 2fd353c7b9..f090b14008 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -21,8 +21,8 @@ use crate::ir::{self, InstBuilder, MemFlags};
 use crate::isa::TargetIsa;
 use crate::predicates;
 use crate::timing;
-use std::collections::BTreeSet;
-use std::vec::Vec;
+use alloc::collections::BTreeSet;
+use alloc::vec::Vec;
 
 mod boundary;
 mod call;
@@ -373,7 +373,7 @@ fn expand_br_table_conds(
     let table_size = func.jump_tables[table].len();
     let mut cond_failed_ebb = vec![];
     if table_size >= 1 {
-        cond_failed_ebb = std::vec::Vec::with_capacity(table_size - 1);
+        cond_failed_ebb = alloc::vec::Vec::with_capacity(table_size - 1);
         for _ in 0..table_size - 1 {
             cond_failed_ebb.push(func.dfg.make_ebb());
         }
diff --git a/cranelift/codegen/src/legalizer/split.rs b/cranelift/codegen/src/legalizer/split.rs
index 5e55419fad..c844fba214 100644
--- a/cranelift/codegen/src/legalizer/split.rs
+++ b/cranelift/codegen/src/legalizer/split.rs
@@ -69,7 +69,7 @@ use crate::flowgraph::{BasicBlock, ControlFlowGraph};
 use crate::ir::{self, Ebb, Inst, InstBuilder, InstructionData, Opcode, Type, Value, ValueDef};
 use core::iter;
 use smallvec::SmallVec;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values
 /// if possible.
diff --git a/cranelift/codegen/src/lib.rs b/cranelift/codegen/src/lib.rs
index 0e61171d14..ab22e71459 100644
--- a/cranelift/codegen/src/lib.rs
+++ b/cranelift/codegen/src/lib.rs
@@ -35,16 +35,15 @@
         clippy::nonminimal_bool,
         clippy::option_map_unwrap_or,
         clippy::option_map_unwrap_or_else,
-        clippy::print_stdout,
+        clippy::print_allocout,
         clippy::unicode_not_nfc,
         clippy::use_self
     )
 )]
 #![no_std]
 
-#[cfg(not(feature = "std"))]
-#[macro_use]
-extern crate alloc as std;
+extern crate alloc;
+
 #[cfg(feature = "std")]
 #[macro_use]
 extern crate std;
diff --git a/cranelift/codegen/src/licm.rs b/cranelift/codegen/src/licm.rs
index 2866da87d7..284d50c5ea 100644
--- a/cranelift/codegen/src/licm.rs
+++ b/cranelift/codegen/src/licm.rs
@@ -11,7 +11,7 @@ use crate::ir::{
 use crate::isa::TargetIsa;
 use crate::loop_analysis::{Loop, LoopAnalysis};
 use crate::timing;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Performs the LICM pass by detecting loops within the CFG and moving
 /// loop-invariant instructions out of them.
diff --git a/cranelift/codegen/src/loop_analysis.rs b/cranelift/codegen/src/loop_analysis.rs
index ea26332b40..40de9afdff 100644
--- a/cranelift/codegen/src/loop_analysis.rs
+++ b/cranelift/codegen/src/loop_analysis.rs
@@ -9,7 +9,7 @@ use crate::flowgraph::{BasicBlock, ControlFlowGraph};
 use crate::ir::{Ebb, Function, Layout};
 use crate::packed_option::PackedOption;
 use crate::timing;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// A opaque reference to a code loop.
 #[derive(Copy, Clone, PartialEq, Eq, Hash)]
@@ -237,7 +237,7 @@ mod tests {
     use crate::flowgraph::ControlFlowGraph;
     use crate::ir::{types, Function, InstBuilder};
     use crate::loop_analysis::{Loop, LoopAnalysis};
-    use std::vec::Vec;
+    use alloc::vec::Vec;
 
     #[test]
     fn nested_loops_detection() {
diff --git a/cranelift/codegen/src/partition_slice.rs b/cranelift/codegen/src/partition_slice.rs
index a4ee129008..959f8c1102 100644
--- a/cranelift/codegen/src/partition_slice.rs
+++ b/cranelift/codegen/src/partition_slice.rs
@@ -53,7 +53,7 @@ where
 #[cfg(test)]
 mod tests {
     use super::partition_slice;
-    use std::vec::Vec;
+    use alloc::vec::Vec;
 
     fn check(x: &[u32], want: &[u32]) {
         assert_eq!(x.len(), want.len());
diff --git a/cranelift/codegen/src/print_errors.rs b/cranelift/codegen/src/print_errors.rs
index a18b5d6955..52169a3880 100644
--- a/cranelift/codegen/src/print_errors.rs
+++ b/cranelift/codegen/src/print_errors.rs
@@ -10,9 +10,9 @@ use crate::verifier::{VerifierError, VerifierErrors};
 use crate::write::{decorate_function, FuncWriter, PlainWriter};
 use core::fmt;
 use core::fmt::Write;
-use std::boxed::Box;
-use std::string::{String, ToString};
-use std::vec::Vec;
+use alloc::boxed::Box;
+use alloc::string::{String, ToString};
+use alloc::vec::Vec;
 
 /// Pretty-print a verifier error.
 pub fn pretty_verifier_error<'a>(
diff --git a/cranelift/codegen/src/redundant_reload_remover.rs b/cranelift/codegen/src/redundant_reload_remover.rs
index 9bcc3fbc9d..a0b6764454 100644
--- a/cranelift/codegen/src/redundant_reload_remover.rs
+++ b/cranelift/codegen/src/redundant_reload_remover.rs
@@ -14,7 +14,7 @@ use crate::isa::{RegInfo, RegUnit, TargetIsa};
 use crate::regalloc::RegDiversions;
 use core::convert::TryInto;
 use cranelift_entity::{PrimaryMap, SecondaryMap};
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 // =============================================================================================
 // A description of the redundant-fill-removal algorithm
diff --git a/cranelift/codegen/src/regalloc/branch_splitting.rs b/cranelift/codegen/src/regalloc/branch_splitting.rs
index a3f118b3d6..3a9e6c87bc 100644
--- a/cranelift/codegen/src/regalloc/branch_splitting.rs
+++ b/cranelift/codegen/src/regalloc/branch_splitting.rs
@@ -4,7 +4,7 @@
 //! between a conditional branch and the following terminator.
 #![cfg(feature = "basic-blocks")]
 
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 use crate::cursor::{Cursor, EncCursor};
 use crate::dominator_tree::DominatorTree;
diff --git a/cranelift/codegen/src/regalloc/coalescing.rs b/cranelift/codegen/src/regalloc/coalescing.rs
index b27a105d65..11846ff45b 100644
--- a/cranelift/codegen/src/regalloc/coalescing.rs
+++ b/cranelift/codegen/src/regalloc/coalescing.rs
@@ -22,7 +22,7 @@ use core::fmt;
 use core::iter;
 use core::slice;
 use log::debug;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 // # Implementation
 //
diff --git a/cranelift/codegen/src/regalloc/live_value_tracker.rs b/cranelift/codegen/src/regalloc/live_value_tracker.rs
index 5e17bb3557..7faed970a7 100644
--- a/cranelift/codegen/src/regalloc/live_value_tracker.rs
+++ b/cranelift/codegen/src/regalloc/live_value_tracker.rs
@@ -12,7 +12,7 @@ use crate::partition_slice::partition_slice;
 use crate::regalloc::affinity::Affinity;
 use crate::regalloc::liveness::Liveness;
 use crate::regalloc::liverange::LiveRange;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 type ValueList = EntityList;
 
diff --git a/cranelift/codegen/src/regalloc/liveness.rs b/cranelift/codegen/src/regalloc/liveness.rs
index 419e35bc98..ba7ce60459 100644
--- a/cranelift/codegen/src/regalloc/liveness.rs
+++ b/cranelift/codegen/src/regalloc/liveness.rs
@@ -185,7 +185,7 @@ use crate::regalloc::liverange::LiveRange;
 use crate::timing;
 use core::mem;
 use core::ops::Index;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// A set of live ranges, indexed by value number.
 type LiveRangeSet = SparseMap;
diff --git a/cranelift/codegen/src/regalloc/liverange.rs b/cranelift/codegen/src/regalloc/liverange.rs
index 1d30650fa1..5ababa53b4 100644
--- a/cranelift/codegen/src/regalloc/liverange.rs
+++ b/cranelift/codegen/src/regalloc/liverange.rs
@@ -477,7 +477,7 @@ mod tests {
     use crate::ir::{Ebb, Inst, Value};
     use crate::ir::{ExpandedProgramPoint, ProgramOrder};
     use core::cmp::Ordering;
-    use std::vec::Vec;
+    use alloc::vec::Vec;
 
     // Dummy program order which simply compares indexes.
     // It is assumed that EBBs have indexes that are multiples of 10, and instructions have indexes
diff --git a/cranelift/codegen/src/regalloc/pressure.rs b/cranelift/codegen/src/regalloc/pressure.rs
index 3b15a1c6e2..2cac3bd804 100644
--- a/cranelift/codegen/src/regalloc/pressure.rs
+++ b/cranelift/codegen/src/regalloc/pressure.rs
@@ -277,7 +277,7 @@ mod tests {
     use crate::regalloc::RegisterSet;
     use core::borrow::Borrow;
     use core::str::FromStr;
-    use std::boxed::Box;
+    use alloc::boxed::Box;
     use target_lexicon::triple;
 
     // Make an arm32 `TargetIsa`, if possible.
diff --git a/cranelift/codegen/src/regalloc/register_set.rs b/cranelift/codegen/src/regalloc/register_set.rs
index e5edaa96d6..52b8a6fa0a 100644
--- a/cranelift/codegen/src/regalloc/register_set.rs
+++ b/cranelift/codegen/src/regalloc/register_set.rs
@@ -255,7 +255,7 @@ impl fmt::Display for RegisterSet {
 mod tests {
     use super::*;
     use crate::isa::registers::{RegClass, RegClassData};
-    use std::vec::Vec;
+    use alloc::vec::Vec;
 
     // Register classes for testing.
     const GPR: RegClass = &RegClassData {
diff --git a/cranelift/codegen/src/regalloc/reload.rs b/cranelift/codegen/src/regalloc/reload.rs
index bbc198c45d..ed04e4fc64 100644
--- a/cranelift/codegen/src/regalloc/reload.rs
+++ b/cranelift/codegen/src/regalloc/reload.rs
@@ -22,7 +22,7 @@ use crate::regalloc::liveness::Liveness;
 use crate::timing;
 use crate::topo_order::TopoOrder;
 use log::debug;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Reusable data structures for the reload pass.
 pub struct Reload {
diff --git a/cranelift/codegen/src/regalloc/safepoint.rs b/cranelift/codegen/src/regalloc/safepoint.rs
index 9b27b6227d..ba846190f3 100644
--- a/cranelift/codegen/src/regalloc/safepoint.rs
+++ b/cranelift/codegen/src/regalloc/safepoint.rs
@@ -4,7 +4,7 @@ use crate::ir::{Function, InstBuilder, InstructionData, Opcode, TrapCode};
 use crate::isa::TargetIsa;
 use crate::regalloc::live_value_tracker::LiveValueTracker;
 use crate::regalloc::liveness::Liveness;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 fn insert_and_encode_safepoint<'f>(
     pos: &mut FuncCursor<'f>,
diff --git a/cranelift/codegen/src/regalloc/solver.rs b/cranelift/codegen/src/regalloc/solver.rs
index 35e17b050d..8d651f143c 100644
--- a/cranelift/codegen/src/regalloc/solver.rs
+++ b/cranelift/codegen/src/regalloc/solver.rs
@@ -109,7 +109,7 @@ use core::fmt;
 use core::mem;
 use core::u16;
 use log::debug;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// A variable in the constraint problem.
 ///
@@ -1160,7 +1160,7 @@ mod tests {
     use crate::isa::{RegClass, RegInfo, RegUnit, TargetIsa};
     use crate::regalloc::RegisterSet;
     use core::str::FromStr;
-    use std::boxed::Box;
+    use alloc::boxed::Box;
     use target_lexicon::triple;
 
     // Make an arm32 `TargetIsa`, if possible.
diff --git a/cranelift/codegen/src/regalloc/spilling.rs b/cranelift/codegen/src/regalloc/spilling.rs
index ace368d798..3b42bcb5e5 100644
--- a/cranelift/codegen/src/regalloc/spilling.rs
+++ b/cranelift/codegen/src/regalloc/spilling.rs
@@ -29,7 +29,7 @@ use crate::timing;
 use crate::topo_order::TopoOrder;
 use core::fmt;
 use log::debug;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Return a top-level register class which contains `unit`.
 fn toprc_containing_regunit(unit: RegUnit, reginfo: &RegInfo) -> RegClass {
diff --git a/cranelift/codegen/src/regalloc/virtregs.rs b/cranelift/codegen/src/regalloc/virtregs.rs
index df2b480490..e3944b3cc2 100644
--- a/cranelift/codegen/src/regalloc/virtregs.rs
+++ b/cranelift/codegen/src/regalloc/virtregs.rs
@@ -22,7 +22,7 @@ use core::cmp::Ordering;
 use core::fmt;
 use core::slice;
 use smallvec::SmallVec;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// A virtual register reference.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index c310972170..99e1562cc5 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -25,8 +25,8 @@ use crate::isa::TargetIsa;
 use core::fmt;
 use core::str;
 use failure_derive::Fail;
-use std::boxed::Box;
-use std::string::{String, ToString};
+use alloc::boxed::Box;
+use alloc::string::{String, ToString};
 
 /// A string-based configurator for settings groups.
 ///
@@ -369,7 +369,7 @@ mod tests {
     use super::Configurable;
     use super::SetError::*;
     use super::{builder, Flags};
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn display_default() {
diff --git a/cranelift/codegen/src/simple_gvn.rs b/cranelift/codegen/src/simple_gvn.rs
index 21a0b9beb9..f88f6f7a5c 100644
--- a/cranelift/codegen/src/simple_gvn.rs
+++ b/cranelift/codegen/src/simple_gvn.rs
@@ -7,7 +7,7 @@ use crate::scoped_hash_map::ScopedHashMap;
 use crate::timing;
 use core::cell::{Ref, RefCell};
 use core::hash::{Hash, Hasher};
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Test whether the given opcode is unsafe to even consider for GVN.
 fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool {
diff --git a/cranelift/codegen/src/timing.rs b/cranelift/codegen/src/timing.rs
index 2a2d393803..34ba801593 100644
--- a/cranelift/codegen/src/timing.rs
+++ b/cranelift/codegen/src/timing.rs
@@ -252,7 +252,7 @@ mod details {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn display() {
diff --git a/cranelift/codegen/src/topo_order.rs b/cranelift/codegen/src/topo_order.rs
index 8824f4cc0a..647a8da941 100644
--- a/cranelift/codegen/src/topo_order.rs
+++ b/cranelift/codegen/src/topo_order.rs
@@ -3,7 +3,7 @@
 use crate::dominator_tree::DominatorTree;
 use crate::entity::EntitySet;
 use crate::ir::{Ebb, Layout};
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Present EBBs in a topological order such that all dominating EBBs are guaranteed to be visited
 /// before the current EBB.
diff --git a/cranelift/codegen/src/value_label.rs b/cranelift/codegen/src/value_label.rs
index 3d5802f12e..e11df7b9e8 100644
--- a/cranelift/codegen/src/value_label.rs
+++ b/cranelift/codegen/src/value_label.rs
@@ -6,8 +6,8 @@ use core::cmp::Ordering;
 use core::iter::Iterator;
 use core::ops::Bound::*;
 use core::ops::Deref;
-use std::collections::BTreeMap;
-use std::vec::Vec;
+use alloc::collections::BTreeMap;
+use alloc::vec::Vec;
 
 #[cfg(feature = "enable-serde")]
 use serde::{Deserialize, Serialize};
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 02cc3f0c52..fe9f1b10de 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -75,9 +75,9 @@ use crate::timing;
 use core::cmp::Ordering;
 use core::fmt::{self, Display, Formatter, Write};
 use failure_derive::Fail;
-use std::collections::BTreeSet;
-use std::string::String;
-use std::vec::Vec;
+use alloc::collections::BTreeSet;
+use alloc::string::String;
+use alloc::vec::Vec;
 
 pub use self::cssa::verify_cssa;
 pub use self::liveness::verify_liveness;
diff --git a/cranelift/codegen/src/write.rs b/cranelift/codegen/src/write.rs
index 8f5f81d23e..bc4e4ca58c 100644
--- a/cranelift/codegen/src/write.rs
+++ b/cranelift/codegen/src/write.rs
@@ -15,8 +15,8 @@ use crate::packed_option::ReservedValue;
 use crate::value_label::ValueLabelsRanges;
 use crate::HashSet;
 use core::fmt::{self, Write};
-use std::string::String;
-use std::vec::Vec;
+use alloc::string::String;
+use alloc::vec::Vec;
 
 /// A `FuncWriter` used to decorate functions during printing.
 pub trait FuncWriter {
@@ -759,7 +759,7 @@ mod tests {
     use crate::cursor::{Cursor, CursorPosition, FuncCursor};
     use crate::ir::types;
     use crate::ir::{ExternalName, Function, InstBuilder, StackSlotData, StackSlotKind};
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     #[test]
     fn basic() {

From 74556d45ae9558cf5a7097bb8839351dcda0d4fc Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 28 Sep 2019 15:55:07 +0200
Subject: [PATCH 2793/3084] Remove std feature from cranelift-codegen-meta

---
 cranelift/codegen/Cargo.toml      | 9 +++------
 cranelift/codegen/meta/Cargo.toml | 6 ------
 2 files changed, 3 insertions(+), 12 deletions(-)

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 2739160aa2..7281ad3092 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -29,7 +29,7 @@ smallvec = { version = "0.6.10" }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.44.0", default-features = false }
+cranelift-codegen-meta = { path = "meta", version = "0.44.0" }
 
 [features]
 default = ["std"]
@@ -37,14 +37,11 @@ default = ["std"]
 # The "std" feature enables use of libstd. The "core" feature enables use
 # of some minimal std-like replacement libraries. At least one of these two
 # features need to be enabled.
-std = ["cranelift-codegen-meta/std"]
+std = []
 
 # The "core" features enables use of "hashbrown" since core doesn't have
 # a HashMap implementation, and a workaround for Cargo #4866.
-core = [
-    "hashbrown",
-    "cranelift-codegen-meta/core"
-]
+core = ["hashbrown"]
 
 # This enables some additional functions useful for writing tests, but which
 # can significantly increase the size of the library.
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 74cc0accd3..63624d3f1e 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -15,9 +15,3 @@ cranelift-entity = { path = "../../cranelift-entity", version = "0.44.0" }
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }
-
-[features]
-default = ["std"]
-std = []
-# The "core" feature enables a workaround for Cargo #4866.
-core = []

From 1a99ac6b4a378bac654cffb0061c96a3b5f2254a Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 28 Sep 2019 15:57:43 +0200
Subject: [PATCH 2794/3084] Always use extern crate std in cranelift-frontend

---
 cranelift/frontend/src/frontend.rs | 4 ++--
 cranelift/frontend/src/lib.rs      | 5 ++---
 cranelift/frontend/src/ssa.rs      | 2 +-
 cranelift/frontend/src/switch.rs   | 4 ++--
 4 files changed, 7 insertions(+), 8 deletions(-)

diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 6204071618..15488f7c2a 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -13,7 +13,7 @@ use cranelift_codegen::ir::{
 };
 use cranelift_codegen::isa::{TargetFrontendConfig, TargetIsa};
 use cranelift_codegen::packed_option::PackedOption;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Structure used for translating a series of functions into Cranelift IR.
 ///
@@ -899,7 +899,7 @@ mod tests {
     use cranelift_codegen::isa::CallConv;
     use cranelift_codegen::settings;
     use cranelift_codegen::verifier::verify_function;
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     fn sample_function(lazy_seal: bool) {
         let mut sig = Signature::new(CallConv::SystemV);
diff --git a/cranelift/frontend/src/lib.rs b/cranelift/frontend/src/lib.rs
index 5e53944901..6357eae9bb 100644
--- a/cranelift/frontend/src/lib.rs
+++ b/cranelift/frontend/src/lib.rs
@@ -183,9 +183,8 @@
 )]
 #![no_std]
 
-#[cfg(not(feature = "std"))]
-#[macro_use]
-extern crate alloc as std;
+extern crate alloc;
+
 #[cfg(feature = "std")]
 #[macro_use]
 extern crate std;
diff --git a/cranelift/frontend/src/ssa.rs b/cranelift/frontend/src/ssa.rs
index b3b74fac9b..0175c6381d 100644
--- a/cranelift/frontend/src/ssa.rs
+++ b/cranelift/frontend/src/ssa.rs
@@ -17,7 +17,7 @@ use cranelift_codegen::ir::{Ebb, Function, Inst, InstBuilder, InstructionData, T
 use cranelift_codegen::packed_option::PackedOption;
 use cranelift_codegen::packed_option::ReservedValue;
 use smallvec::SmallVec;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 /// Structure containing the data relevant the construction of SSA for a given function.
 ///
diff --git a/cranelift/frontend/src/switch.rs b/cranelift/frontend/src/switch.rs
index 0b1ca746cb..7353d7d049 100644
--- a/cranelift/frontend/src/switch.rs
+++ b/cranelift/frontend/src/switch.rs
@@ -3,7 +3,7 @@ use crate::frontend::FunctionBuilder;
 use cranelift_codegen::ir::condcodes::IntCC;
 use cranelift_codegen::ir::*;
 use log::debug;
-use std::vec::Vec;
+use alloc::vec::Vec;
 
 type EntryIndex = u64;
 
@@ -291,7 +291,7 @@ mod tests {
     use super::*;
     use crate::frontend::FunctionBuilderContext;
     use cranelift_codegen::ir::Function;
-    use std::string::ToString;
+    use alloc::string::ToString;
 
     macro_rules! setup {
         ($default:expr, [$($index:expr,)*]) => {{

From 9037e7a6012788e7117842baf93a6ca32640cc96 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 28 Sep 2019 15:59:08 +0200
Subject: [PATCH 2795/3084] Remove std feature from cranelift-preopt

---
 cranelift/preopt/Cargo.toml | 5 -----
 cranelift/preopt/src/lib.rs | 7 -------
 2 files changed, 12 deletions(-)

diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index c18188cf77..6dc55e1432 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -18,11 +18,6 @@ cranelift-entity = { path = "../cranelift-entity", version = "0.44.0" }
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
 
-[features]
-default = ["std"]
-std = ["cranelift-codegen/std"]
-core = ["cranelift-codegen/core"]
-
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/cranelift/preopt/src/lib.rs b/cranelift/preopt/src/lib.rs
index c35eea14a2..027460bc59 100644
--- a/cranelift/preopt/src/lib.rs
+++ b/cranelift/preopt/src/lib.rs
@@ -20,13 +20,6 @@
 )]
 #![no_std]
 
-#[cfg(not(feature = "std"))]
-#[macro_use]
-extern crate alloc as std;
-#[cfg(feature = "std")]
-#[macro_use]
-extern crate std;
-
 mod constant_folding;
 
 use cranelift_codegen::{isa::TargetIsa, settings::FlagsOrIsa, CodegenResult, Context};

From c274d81b5b4fa2ed835d4c060152f1230ec9a5fb Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 28 Sep 2019 16:09:57 +0200
Subject: [PATCH 2796/3084] Fix it

---
 cranelift/codegen/src/isa/mod.rs | 2 +-
 cranelift/codegen/src/lib.rs     | 2 ++
 cranelift/frontend/src/lib.rs    | 2 ++
 3 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index 8c6b3bc203..fd118f1286 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -25,7 +25,7 @@
 //! # fn main() {
 //! use cranelift_codegen::isa;
 //! use cranelift_codegen::settings::{self, Configurable};
-//! use alloc::str::FromStr;
+//! use std::str::FromStr;
 //! use target_lexicon::Triple;
 //!
 //! let shared_builder = settings::builder();
diff --git a/cranelift/codegen/src/lib.rs b/cranelift/codegen/src/lib.rs
index ab22e71459..4b6cfc33ca 100644
--- a/cranelift/codegen/src/lib.rs
+++ b/cranelift/codegen/src/lib.rs
@@ -42,6 +42,8 @@
 )]
 #![no_std]
 
+#[allow(unused_imports)] // #[macro_use] is required for no_std
+#[macro_use]
 extern crate alloc;
 
 #[cfg(feature = "std")]
diff --git a/cranelift/frontend/src/lib.rs b/cranelift/frontend/src/lib.rs
index 6357eae9bb..0801be740d 100644
--- a/cranelift/frontend/src/lib.rs
+++ b/cranelift/frontend/src/lib.rs
@@ -183,6 +183,8 @@
 )]
 #![no_std]
 
+#[allow(unused_imports)] // #[macro_use] is required for no_std
+#[macro_use]
 extern crate alloc;
 
 #[cfg(feature = "std")]

From bb8fa40ef0ead8bbaa23e4f0ab1d1eda2d22681c Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 28 Sep 2019 16:43:00 +0200
Subject: [PATCH 2797/3084] Rustfmt

---
 cranelift/bforest/src/map.rs                      | 2 +-
 cranelift/bforest/src/node.rs                     | 2 +-
 cranelift/bforest/src/pool.rs                     | 2 +-
 cranelift/bforest/src/set.rs                      | 6 +++---
 cranelift/codegen/src/abi.rs                      | 2 +-
 cranelift/codegen/src/cfg_printer.rs              | 2 +-
 cranelift/codegen/src/context.rs                  | 2 +-
 cranelift/codegen/src/dominator_tree.rs           | 2 +-
 cranelift/codegen/src/ir/constant.rs              | 2 +-
 cranelift/codegen/src/ir/dfg.rs                   | 2 +-
 cranelift/codegen/src/ir/entities.rs              | 2 +-
 cranelift/codegen/src/ir/extfunc.rs               | 2 +-
 cranelift/codegen/src/ir/extname.rs               | 2 +-
 cranelift/codegen/src/ir/immediates.rs            | 4 ++--
 cranelift/codegen/src/ir/instructions.rs          | 2 +-
 cranelift/codegen/src/ir/jumptable.rs             | 2 +-
 cranelift/codegen/src/ir/layout.rs                | 2 +-
 cranelift/codegen/src/ir/stackslot.rs             | 2 +-
 cranelift/codegen/src/isa/arm32/mod.rs            | 2 +-
 cranelift/codegen/src/isa/arm64/mod.rs            | 2 +-
 cranelift/codegen/src/isa/mod.rs                  | 2 +-
 cranelift/codegen/src/isa/riscv/mod.rs            | 4 ++--
 cranelift/codegen/src/isa/x86/mod.rs              | 2 +-
 cranelift/codegen/src/legalizer/boundary.rs       | 2 +-
 cranelift/codegen/src/legalizer/split.rs          | 2 +-
 cranelift/codegen/src/print_errors.rs             | 4 ++--
 cranelift/codegen/src/redundant_reload_remover.rs | 2 +-
 cranelift/codegen/src/regalloc/coalescing.rs      | 2 +-
 cranelift/codegen/src/regalloc/liveness.rs        | 2 +-
 cranelift/codegen/src/regalloc/liverange.rs       | 2 +-
 cranelift/codegen/src/regalloc/pressure.rs        | 2 +-
 cranelift/codegen/src/regalloc/reload.rs          | 2 +-
 cranelift/codegen/src/regalloc/solver.rs          | 4 ++--
 cranelift/codegen/src/regalloc/spilling.rs        | 2 +-
 cranelift/codegen/src/regalloc/virtregs.rs        | 2 +-
 cranelift/codegen/src/settings.rs                 | 4 ++--
 cranelift/codegen/src/simple_gvn.rs               | 2 +-
 cranelift/codegen/src/value_label.rs              | 4 ++--
 cranelift/codegen/src/verifier/mod.rs             | 6 +++---
 cranelift/codegen/src/write.rs                    | 2 +-
 cranelift/entity/src/boxed_slice.rs               | 2 +-
 cranelift/entity/src/list.rs                      | 2 +-
 cranelift/entity/src/map.rs                       | 2 +-
 cranelift/entity/src/primary.rs                   | 4 ++--
 cranelift/entity/src/set.rs                       | 2 +-
 cranelift/entity/src/sparse.rs                    | 2 +-
 cranelift/frontend/src/frontend.rs                | 4 ++--
 cranelift/frontend/src/ssa.rs                     | 2 +-
 cranelift/frontend/src/switch.rs                  | 4 ++--
 49 files changed, 62 insertions(+), 62 deletions(-)

diff --git a/cranelift/bforest/src/map.rs b/cranelift/bforest/src/map.rs
index 931253cae5..fb7d4d9ddf 100644
--- a/cranelift/bforest/src/map.rs
+++ b/cranelift/bforest/src/map.rs
@@ -429,8 +429,8 @@ where
 mod tests {
     use super::super::NodeData;
     use super::*;
-    use core::mem;
     use alloc::vec::Vec;
+    use core::mem;
 
     #[test]
     fn node_size() {
diff --git a/cranelift/bforest/src/node.rs b/cranelift/bforest/src/node.rs
index 260494b171..589bc2b5e0 100644
--- a/cranelift/bforest/src/node.rs
+++ b/cranelift/bforest/src/node.rs
@@ -584,8 +584,8 @@ where
 #[cfg(test)]
 mod tests {
     use super::*;
-    use core::mem;
     use alloc::string::ToString;
+    use core::mem;
 
     // Forest impl for a set implementation.
     struct TF();
diff --git a/cranelift/bforest/src/pool.rs b/cranelift/bforest/src/pool.rs
index 6b4bae020d..e4744d2bcb 100644
--- a/cranelift/bforest/src/pool.rs
+++ b/cranelift/bforest/src/pool.rs
@@ -84,9 +84,9 @@ impl NodePool {
         F::Key: fmt::Display,
     {
         use crate::entity::EntitySet;
+        use alloc::vec::Vec;
         use core::borrow::Borrow;
         use core::cmp::Ordering;
-        use alloc::vec::Vec;
 
         // The root node can't be an inner node with just a single sub-tree. It should have been
         // pruned.
diff --git a/cranelift/bforest/src/set.rs b/cranelift/bforest/src/set.rs
index d22cb8285e..e7761a63d9 100644
--- a/cranelift/bforest/src/set.rs
+++ b/cranelift/bforest/src/set.rs
@@ -3,10 +3,10 @@
 use super::{Comparator, Forest, Node, NodeData, NodePool, Path, SetValue, INNER_SIZE};
 use crate::packed_option::PackedOption;
 #[cfg(test)]
+use alloc::string::String;
+#[cfg(test)]
 use core::fmt;
 use core::marker::PhantomData;
-#[cfg(test)]
-use alloc::string::String;
 
 /// Tag type defining forest types for a set.
 struct SetTypes(PhantomData);
@@ -357,8 +357,8 @@ where
 mod tests {
     use super::super::NodeData;
     use super::*;
-    use core::mem;
     use alloc::vec::Vec;
+    use core::mem;
 
     #[test]
     fn node_size() {
diff --git a/cranelift/codegen/src/abi.rs b/cranelift/codegen/src/abi.rs
index 8564fd5a24..e63f21b8d3 100644
--- a/cranelift/codegen/src/abi.rs
+++ b/cranelift/codegen/src/abi.rs
@@ -4,8 +4,8 @@
 //! `TargetIsa::legalize_signature()` method.
 
 use crate::ir::{AbiParam, ArgumentExtension, ArgumentLoc, Type};
-use core::cmp::Ordering;
 use alloc::vec::Vec;
+use core::cmp::Ordering;
 
 /// Legalization action to perform on a single argument or return value when converting a
 /// signature.
diff --git a/cranelift/codegen/src/cfg_printer.rs b/cranelift/codegen/src/cfg_printer.rs
index c67e8b1eec..ed47475295 100644
--- a/cranelift/codegen/src/cfg_printer.rs
+++ b/cranelift/codegen/src/cfg_printer.rs
@@ -1,7 +1,7 @@
 //! The `CFGPrinter` utility.
 
-use core::fmt::{Display, Formatter, Result, Write};
 use alloc::vec::Vec;
+use core::fmt::{Display, Formatter, Result, Write};
 
 use crate::entity::SecondaryMap;
 use crate::flowgraph::{BasicBlock, ControlFlowGraph};
diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index aad08a5481..ccf216d214 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -33,8 +33,8 @@ use crate::timing;
 use crate::unreachable_code::eliminate_unreachable_code;
 use crate::value_label::{build_value_labels_ranges, ComparableSourceLoc, ValueLabelsRanges};
 use crate::verifier::{verify_context, verify_locations, VerifierErrors, VerifierResult};
-use log::debug;
 use alloc::vec::Vec;
+use log::debug;
 
 /// Persistent data structures and compilation pipeline.
 pub struct Context {
diff --git a/cranelift/codegen/src/dominator_tree.rs b/cranelift/codegen/src/dominator_tree.rs
index 737019d315..67446b21f6 100644
--- a/cranelift/codegen/src/dominator_tree.rs
+++ b/cranelift/codegen/src/dominator_tree.rs
@@ -6,10 +6,10 @@ use crate::ir::instructions::BranchInfo;
 use crate::ir::{Ebb, ExpandedProgramPoint, Function, Inst, Layout, ProgramOrder, Value};
 use crate::packed_option::PackedOption;
 use crate::timing;
+use alloc::vec::Vec;
 use core::cmp;
 use core::cmp::Ordering;
 use core::mem;
-use alloc::vec::Vec;
 
 /// RPO numbers are not first assigned in a contiguous way but as multiples of STRIDE, to leave
 /// room for modifications of the dominator tree.
diff --git a/cranelift/codegen/src/ir/constant.rs b/cranelift/codegen/src/ir/constant.rs
index c4d7cb0898..ff30d67734 100644
--- a/cranelift/codegen/src/ir/constant.rs
+++ b/cranelift/codegen/src/ir/constant.rs
@@ -10,9 +10,9 @@
 
 use crate::ir::Constant;
 use crate::HashMap;
-use cranelift_entity::EntityRef;
 use alloc::collections::BTreeMap;
 use alloc::vec::Vec;
+use cranelift_entity::EntityRef;
 
 /// This type describes the actual constant data.
 pub type ConstantData = Vec;
diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index 34ed10622d..0ff6a263c4 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -14,12 +14,12 @@ use crate::isa::TargetIsa;
 use crate::packed_option::ReservedValue;
 use crate::write::write_operands;
 use crate::HashMap;
+use alloc::vec::Vec;
 use core::fmt;
 use core::iter;
 use core::mem;
 use core::ops::{Index, IndexMut};
 use core::u16;
-use alloc::vec::Vec;
 
 /// A data flow graph defines all instructions and extended basic blocks in a function as well as
 /// the data flow dependencies between them. The DFG also tracks values which can be either
diff --git a/cranelift/codegen/src/ir/entities.rs b/cranelift/codegen/src/ir/entities.rs
index 552a9f3ae0..1883fea258 100644
--- a/cranelift/codegen/src/ir/entities.rs
+++ b/cranelift/codegen/src/ir/entities.rs
@@ -455,8 +455,8 @@ impl From
 for AnyEntity {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use core::u32;
     use alloc::string::ToString;
+    use core::u32;
 
     #[test]
     fn value_with_number() {
diff --git a/cranelift/codegen/src/ir/extfunc.rs b/cranelift/codegen/src/ir/extfunc.rs
index f09b428833..112fb31b7b 100644
--- a/cranelift/codegen/src/ir/extfunc.rs
+++ b/cranelift/codegen/src/ir/extfunc.rs
@@ -7,9 +7,9 @@
 
 use crate::ir::{ArgumentLoc, ExternalName, SigRef, Type};
 use crate::isa::{CallConv, RegInfo, RegUnit};
+use alloc::vec::Vec;
 use core::fmt;
 use core::str::FromStr;
-use alloc::vec::Vec;
 
 /// Function signature.
 ///
diff --git a/cranelift/codegen/src/ir/extname.rs b/cranelift/codegen/src/ir/extname.rs
index d153db9cbd..3a47699acc 100644
--- a/cranelift/codegen/src/ir/extname.rs
+++ b/cranelift/codegen/src/ir/extname.rs
@@ -120,8 +120,8 @@ impl FromStr for ExternalName {
 mod tests {
     use super::ExternalName;
     use crate::ir::LibCall;
-    use core::u32;
     use alloc::string::ToString;
+    use core::u32;
 
     #[test]
     fn display_testcase() {
diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index 5d3807671f..27320328fe 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -4,11 +4,11 @@
 //! Each type here should have a corresponding definition in the
 //! `cranelift-codegen/meta/src/shared/immediates` crate in the meta language.
 
+use alloc::vec::Vec;
 use core::fmt::{self, Display, Formatter};
 use core::iter::FromIterator;
 use core::str::{from_utf8, FromStr};
 use core::{i32, u32};
-use alloc::vec::Vec;
 
 /// Convert a type into a vector of bytes; all implementors in this file must use little-endian
 /// orderings of bytes to match WebAssembly's little-endianness.
@@ -936,11 +936,11 @@ impl IntoBytes for Ieee64 {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use alloc::string::ToString;
     use core::fmt::Display;
     use core::mem;
     use core::str::FromStr;
     use core::{f32, f64};
-    use alloc::string::ToString;
 
     #[test]
     fn format_imm64() {
diff --git a/cranelift/codegen/src/ir/instructions.rs b/cranelift/codegen/src/ir/instructions.rs
index 97c84110aa..7eb504483f 100644
--- a/cranelift/codegen/src/ir/instructions.rs
+++ b/cranelift/codegen/src/ir/instructions.rs
@@ -6,10 +6,10 @@
 //! A large part of this module is auto-generated from the instruction descriptions in the meta
 //! directory.
 
+use alloc::vec::Vec;
 use core::fmt::{self, Display, Formatter};
 use core::ops::{Deref, DerefMut};
 use core::str::FromStr;
-use alloc::vec::Vec;
 
 use crate::ir;
 use crate::ir::types;
diff --git a/cranelift/codegen/src/ir/jumptable.rs b/cranelift/codegen/src/ir/jumptable.rs
index 596e668e89..b412e807ac 100644
--- a/cranelift/codegen/src/ir/jumptable.rs
+++ b/cranelift/codegen/src/ir/jumptable.rs
@@ -4,9 +4,9 @@
 //! The actual table of destinations is stored in a `JumpTableData` struct defined in this module.
 
 use crate::ir::entities::Ebb;
+use alloc::vec::Vec;
 use core::fmt::{self, Display, Formatter};
 use core::slice::{Iter, IterMut};
-use alloc::vec::Vec;
 
 /// Contents of a jump table.
 ///
diff --git a/cranelift/codegen/src/ir/layout.rs b/cranelift/codegen/src/ir/layout.rs
index 3d1f5e975c..7b624a92d2 100644
--- a/cranelift/codegen/src/ir/layout.rs
+++ b/cranelift/codegen/src/ir/layout.rs
@@ -750,8 +750,8 @@ mod tests {
     use crate::cursor::{Cursor, CursorPosition};
     use crate::entity::EntityRef;
     use crate::ir::{Ebb, Inst, ProgramOrder, SourceLoc};
-    use core::cmp::Ordering;
     use alloc::vec::Vec;
+    use core::cmp::Ordering;
 
     struct LayoutCursor<'f> {
         /// Borrowed function layout. Public so it can be re-borrowed from this cursor.
diff --git a/cranelift/codegen/src/ir/stackslot.rs b/cranelift/codegen/src/ir/stackslot.rs
index fb526ab50e..00dee43af4 100644
--- a/cranelift/codegen/src/ir/stackslot.rs
+++ b/cranelift/codegen/src/ir/stackslot.rs
@@ -6,12 +6,12 @@
 use crate::entity::{Iter, IterMut, Keys, PrimaryMap};
 use crate::ir::{StackSlot, Type};
 use crate::packed_option::PackedOption;
+use alloc::vec::Vec;
 use core::cmp;
 use core::fmt;
 use core::ops::{Index, IndexMut};
 use core::slice;
 use core::str::FromStr;
-use alloc::vec::Vec;
 
 #[cfg(feature = "enable-serde")]
 use serde::{Deserialize, Serialize};
diff --git a/cranelift/codegen/src/isa/arm32/mod.rs b/cranelift/codegen/src/isa/arm32/mod.rs
index f377560581..6c00ef9089 100644
--- a/cranelift/codegen/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/src/isa/arm32/mod.rs
@@ -15,8 +15,8 @@ use crate::isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encoding
 use crate::isa::Builder as IsaBuilder;
 use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
-use core::fmt;
 use alloc::boxed::Box;
+use core::fmt;
 use target_lexicon::{Architecture, Triple};
 
 #[allow(dead_code)]
diff --git a/cranelift/codegen/src/isa/arm64/mod.rs b/cranelift/codegen/src/isa/arm64/mod.rs
index 84afc33cbd..26c28329bc 100644
--- a/cranelift/codegen/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/src/isa/arm64/mod.rs
@@ -15,8 +15,8 @@ use crate::isa::enc_tables::{lookup_enclist, Encodings};
 use crate::isa::Builder as IsaBuilder;
 use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
-use core::fmt;
 use alloc::boxed::Box;
+use core::fmt;
 use target_lexicon::Triple;
 
 #[allow(dead_code)]
diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index fd118f1286..067dff318f 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -63,9 +63,9 @@ use crate::result::CodegenResult;
 use crate::settings;
 use crate::settings::SetResult;
 use crate::timing;
+use alloc::boxed::Box;
 use core::fmt;
 use failure_derive::Fail;
-use alloc::boxed::Box;
 use target_lexicon::{triple, Architecture, PointerWidth, Triple};
 
 #[cfg(feature = "riscv")]
diff --git a/cranelift/codegen/src/isa/riscv/mod.rs b/cranelift/codegen/src/isa/riscv/mod.rs
index e4bd6c9bbd..79aaeaddf9 100644
--- a/cranelift/codegen/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/src/isa/riscv/mod.rs
@@ -15,8 +15,8 @@ use crate::isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encoding
 use crate::isa::Builder as IsaBuilder;
 use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
-use core::fmt;
 use alloc::boxed::Box;
+use core::fmt;
 use target_lexicon::{PointerWidth, Triple};
 
 #[allow(dead_code)]
@@ -137,8 +137,8 @@ mod tests {
     use crate::ir::{Function, InstructionData, Opcode};
     use crate::isa;
     use crate::settings::{self, Configurable};
-    use core::str::FromStr;
     use alloc::string::{String, ToString};
+    use core::str::FromStr;
     use target_lexicon::triple;
 
     fn encstr(isa: &dyn isa::TargetIsa, enc: Result) -> String {
diff --git a/cranelift/codegen/src/isa/x86/mod.rs b/cranelift/codegen/src/isa/x86/mod.rs
index 35ca485bd8..19cc41c99d 100644
--- a/cranelift/codegen/src/isa/x86/mod.rs
+++ b/cranelift/codegen/src/isa/x86/mod.rs
@@ -17,8 +17,8 @@ use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
 use crate::result::CodegenResult;
 use crate::timing;
-use core::fmt;
 use alloc::boxed::Box;
+use core::fmt;
 use target_lexicon::{PointerWidth, Triple};
 
 #[allow(dead_code)]
diff --git a/cranelift/codegen/src/legalizer/boundary.rs b/cranelift/codegen/src/legalizer/boundary.rs
index d95d6d4239..479208751f 100644
--- a/cranelift/codegen/src/legalizer/boundary.rs
+++ b/cranelift/codegen/src/legalizer/boundary.rs
@@ -27,8 +27,8 @@ use crate::ir::{
 };
 use crate::isa::TargetIsa;
 use crate::legalizer::split::{isplit, vsplit};
-use log::debug;
 use alloc::vec::Vec;
+use log::debug;
 
 /// Legalize all the function signatures in `func`.
 ///
diff --git a/cranelift/codegen/src/legalizer/split.rs b/cranelift/codegen/src/legalizer/split.rs
index c844fba214..a3fa29f267 100644
--- a/cranelift/codegen/src/legalizer/split.rs
+++ b/cranelift/codegen/src/legalizer/split.rs
@@ -67,9 +67,9 @@
 use crate::cursor::{Cursor, CursorPosition, FuncCursor};
 use crate::flowgraph::{BasicBlock, ControlFlowGraph};
 use crate::ir::{self, Ebb, Inst, InstBuilder, InstructionData, Opcode, Type, Value, ValueDef};
+use alloc::vec::Vec;
 use core::iter;
 use smallvec::SmallVec;
-use alloc::vec::Vec;
 
 /// Split `value` into two values using the `isplit` semantics. Do this by reusing existing values
 /// if possible.
diff --git a/cranelift/codegen/src/print_errors.rs b/cranelift/codegen/src/print_errors.rs
index 52169a3880..78ae325630 100644
--- a/cranelift/codegen/src/print_errors.rs
+++ b/cranelift/codegen/src/print_errors.rs
@@ -8,11 +8,11 @@ use crate::isa::TargetIsa;
 use crate::result::CodegenError;
 use crate::verifier::{VerifierError, VerifierErrors};
 use crate::write::{decorate_function, FuncWriter, PlainWriter};
-use core::fmt;
-use core::fmt::Write;
 use alloc::boxed::Box;
 use alloc::string::{String, ToString};
 use alloc::vec::Vec;
+use core::fmt;
+use core::fmt::Write;
 
 /// Pretty-print a verifier error.
 pub fn pretty_verifier_error<'a>(
diff --git a/cranelift/codegen/src/redundant_reload_remover.rs b/cranelift/codegen/src/redundant_reload_remover.rs
index a0b6764454..11f13f5256 100644
--- a/cranelift/codegen/src/redundant_reload_remover.rs
+++ b/cranelift/codegen/src/redundant_reload_remover.rs
@@ -12,9 +12,9 @@ use crate::ir::{
 };
 use crate::isa::{RegInfo, RegUnit, TargetIsa};
 use crate::regalloc::RegDiversions;
+use alloc::vec::Vec;
 use core::convert::TryInto;
 use cranelift_entity::{PrimaryMap, SecondaryMap};
-use alloc::vec::Vec;
 
 // =============================================================================================
 // A description of the redundant-fill-removal algorithm
diff --git a/cranelift/codegen/src/regalloc/coalescing.rs b/cranelift/codegen/src/regalloc/coalescing.rs
index 11846ff45b..c408b912fa 100644
--- a/cranelift/codegen/src/regalloc/coalescing.rs
+++ b/cranelift/codegen/src/regalloc/coalescing.rs
@@ -17,12 +17,12 @@ use crate::regalloc::affinity::Affinity;
 use crate::regalloc::liveness::Liveness;
 use crate::regalloc::virtregs::{VirtReg, VirtRegs};
 use crate::timing;
+use alloc::vec::Vec;
 use core::cmp;
 use core::fmt;
 use core::iter;
 use core::slice;
 use log::debug;
-use alloc::vec::Vec;
 
 // # Implementation
 //
diff --git a/cranelift/codegen/src/regalloc/liveness.rs b/cranelift/codegen/src/regalloc/liveness.rs
index ba7ce60459..1b9784a198 100644
--- a/cranelift/codegen/src/regalloc/liveness.rs
+++ b/cranelift/codegen/src/regalloc/liveness.rs
@@ -183,9 +183,9 @@ use crate::isa::{EncInfo, OperandConstraint, TargetIsa};
 use crate::regalloc::affinity::Affinity;
 use crate::regalloc::liverange::LiveRange;
 use crate::timing;
+use alloc::vec::Vec;
 use core::mem;
 use core::ops::Index;
-use alloc::vec::Vec;
 
 /// A set of live ranges, indexed by value number.
 type LiveRangeSet = SparseMap;
diff --git a/cranelift/codegen/src/regalloc/liverange.rs b/cranelift/codegen/src/regalloc/liverange.rs
index 5ababa53b4..266c4ca09e 100644
--- a/cranelift/codegen/src/regalloc/liverange.rs
+++ b/cranelift/codegen/src/regalloc/liverange.rs
@@ -476,8 +476,8 @@ mod tests {
     use crate::entity::EntityRef;
     use crate::ir::{Ebb, Inst, Value};
     use crate::ir::{ExpandedProgramPoint, ProgramOrder};
-    use core::cmp::Ordering;
     use alloc::vec::Vec;
+    use core::cmp::Ordering;
 
     // Dummy program order which simply compares indexes.
     // It is assumed that EBBs have indexes that are multiples of 10, and instructions have indexes
diff --git a/cranelift/codegen/src/regalloc/pressure.rs b/cranelift/codegen/src/regalloc/pressure.rs
index 2cac3bd804..c7fc1fff9a 100644
--- a/cranelift/codegen/src/regalloc/pressure.rs
+++ b/cranelift/codegen/src/regalloc/pressure.rs
@@ -275,9 +275,9 @@ mod tests {
     use super::Pressure;
     use crate::isa::{RegClass, TargetIsa};
     use crate::regalloc::RegisterSet;
+    use alloc::boxed::Box;
     use core::borrow::Borrow;
     use core::str::FromStr;
-    use alloc::boxed::Box;
     use target_lexicon::triple;
 
     // Make an arm32 `TargetIsa`, if possible.
diff --git a/cranelift/codegen/src/regalloc/reload.rs b/cranelift/codegen/src/regalloc/reload.rs
index ed04e4fc64..0f4f595346 100644
--- a/cranelift/codegen/src/regalloc/reload.rs
+++ b/cranelift/codegen/src/regalloc/reload.rs
@@ -21,8 +21,8 @@ use crate::regalloc::live_value_tracker::{LiveValue, LiveValueTracker};
 use crate::regalloc::liveness::Liveness;
 use crate::timing;
 use crate::topo_order::TopoOrder;
-use log::debug;
 use alloc::vec::Vec;
+use log::debug;
 
 /// Reusable data structures for the reload pass.
 pub struct Reload {
diff --git a/cranelift/codegen/src/regalloc/solver.rs b/cranelift/codegen/src/regalloc/solver.rs
index 8d651f143c..5b63b2825a 100644
--- a/cranelift/codegen/src/regalloc/solver.rs
+++ b/cranelift/codegen/src/regalloc/solver.rs
@@ -104,12 +104,12 @@ use crate::entity::{SparseMap, SparseMapValue};
 use crate::ir::Value;
 use crate::isa::{RegClass, RegUnit};
 use crate::regalloc::register_set::RegSetIter;
+use alloc::vec::Vec;
 use core::cmp;
 use core::fmt;
 use core::mem;
 use core::u16;
 use log::debug;
-use alloc::vec::Vec;
 
 /// A variable in the constraint problem.
 ///
@@ -1159,8 +1159,8 @@ mod tests {
     use crate::ir::Value;
     use crate::isa::{RegClass, RegInfo, RegUnit, TargetIsa};
     use crate::regalloc::RegisterSet;
-    use core::str::FromStr;
     use alloc::boxed::Box;
+    use core::str::FromStr;
     use target_lexicon::triple;
 
     // Make an arm32 `TargetIsa`, if possible.
diff --git a/cranelift/codegen/src/regalloc/spilling.rs b/cranelift/codegen/src/regalloc/spilling.rs
index 3b42bcb5e5..e7127b9606 100644
--- a/cranelift/codegen/src/regalloc/spilling.rs
+++ b/cranelift/codegen/src/regalloc/spilling.rs
@@ -27,9 +27,9 @@ use crate::regalloc::pressure::Pressure;
 use crate::regalloc::virtregs::VirtRegs;
 use crate::timing;
 use crate::topo_order::TopoOrder;
+use alloc::vec::Vec;
 use core::fmt;
 use log::debug;
-use alloc::vec::Vec;
 
 /// Return a top-level register class which contains `unit`.
 fn toprc_containing_regunit(unit: RegUnit, reginfo: &RegInfo) -> RegClass {
diff --git a/cranelift/codegen/src/regalloc/virtregs.rs b/cranelift/codegen/src/regalloc/virtregs.rs
index e3944b3cc2..628c49c286 100644
--- a/cranelift/codegen/src/regalloc/virtregs.rs
+++ b/cranelift/codegen/src/regalloc/virtregs.rs
@@ -18,11 +18,11 @@ use crate::entity::{EntityList, ListPool};
 use crate::entity::{Keys, PrimaryMap, SecondaryMap};
 use crate::ir::{Function, Value};
 use crate::packed_option::PackedOption;
+use alloc::vec::Vec;
 use core::cmp::Ordering;
 use core::fmt;
 use core::slice;
 use smallvec::SmallVec;
-use alloc::vec::Vec;
 
 /// A virtual register reference.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index 99e1562cc5..44fafd0b64 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -22,11 +22,11 @@
 
 use crate::constant_hash::{probe, simple_hash};
 use crate::isa::TargetIsa;
+use alloc::boxed::Box;
+use alloc::string::{String, ToString};
 use core::fmt;
 use core::str;
 use failure_derive::Fail;
-use alloc::boxed::Box;
-use alloc::string::{String, ToString};
 
 /// A string-based configurator for settings groups.
 ///
diff --git a/cranelift/codegen/src/simple_gvn.rs b/cranelift/codegen/src/simple_gvn.rs
index f88f6f7a5c..2130634c47 100644
--- a/cranelift/codegen/src/simple_gvn.rs
+++ b/cranelift/codegen/src/simple_gvn.rs
@@ -5,9 +5,9 @@ use crate::dominator_tree::DominatorTree;
 use crate::ir::{Function, Inst, InstructionData, Opcode, Type};
 use crate::scoped_hash_map::ScopedHashMap;
 use crate::timing;
+use alloc::vec::Vec;
 use core::cell::{Ref, RefCell};
 use core::hash::{Hash, Hasher};
-use alloc::vec::Vec;
 
 /// Test whether the given opcode is unsafe to even consider for GVN.
 fn trivially_unsafe_for_gvn(opcode: Opcode) -> bool {
diff --git a/cranelift/codegen/src/value_label.rs b/cranelift/codegen/src/value_label.rs
index e11df7b9e8..b597fdd585 100644
--- a/cranelift/codegen/src/value_label.rs
+++ b/cranelift/codegen/src/value_label.rs
@@ -2,12 +2,12 @@ use crate::ir::{Function, SourceLoc, Value, ValueLabel, ValueLabelAssignments, V
 use crate::isa::TargetIsa;
 use crate::regalloc::{Context, RegDiversions};
 use crate::HashMap;
+use alloc::collections::BTreeMap;
+use alloc::vec::Vec;
 use core::cmp::Ordering;
 use core::iter::Iterator;
 use core::ops::Bound::*;
 use core::ops::Deref;
-use alloc::collections::BTreeMap;
-use alloc::vec::Vec;
 
 #[cfg(feature = "enable-serde")]
 use serde::{Deserialize, Serialize};
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index fe9f1b10de..7fc33dfd67 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -72,12 +72,12 @@ use crate::isa::TargetIsa;
 use crate::iterators::IteratorExtras;
 use crate::settings::FlagsOrIsa;
 use crate::timing;
-use core::cmp::Ordering;
-use core::fmt::{self, Display, Formatter, Write};
-use failure_derive::Fail;
 use alloc::collections::BTreeSet;
 use alloc::string::String;
 use alloc::vec::Vec;
+use core::cmp::Ordering;
+use core::fmt::{self, Display, Formatter, Write};
+use failure_derive::Fail;
 
 pub use self::cssa::verify_cssa;
 pub use self::liveness::verify_liveness;
diff --git a/cranelift/codegen/src/write.rs b/cranelift/codegen/src/write.rs
index bc4e4ca58c..ae093f3919 100644
--- a/cranelift/codegen/src/write.rs
+++ b/cranelift/codegen/src/write.rs
@@ -14,9 +14,9 @@ use crate::isa::{RegInfo, TargetIsa};
 use crate::packed_option::ReservedValue;
 use crate::value_label::ValueLabelsRanges;
 use crate::HashSet;
-use core::fmt::{self, Write};
 use alloc::string::String;
 use alloc::vec::Vec;
+use core::fmt::{self, Write};
 
 /// A `FuncWriter` used to decorate functions during printing.
 pub trait FuncWriter {
diff --git a/cranelift/entity/src/boxed_slice.rs b/cranelift/entity/src/boxed_slice.rs
index 302cab0235..8d01b60b7f 100644
--- a/cranelift/entity/src/boxed_slice.rs
+++ b/cranelift/entity/src/boxed_slice.rs
@@ -3,10 +3,10 @@
 use crate::iter::{Iter, IterMut};
 use crate::keys::Keys;
 use crate::EntityRef;
+use alloc::boxed::Box;
 use core::marker::PhantomData;
 use core::ops::{Index, IndexMut};
 use core::slice;
-use alloc::boxed::Box;
 
 /// A slice mapping `K -> V` allocating dense entity references.
 ///
diff --git a/cranelift/entity/src/list.rs b/cranelift/entity/src/list.rs
index 263a981bb3..68cab8166b 100644
--- a/cranelift/entity/src/list.rs
+++ b/cranelift/entity/src/list.rs
@@ -1,9 +1,9 @@
 //! Small lists of entity references.
 use crate::packed_option::ReservedValue;
 use crate::EntityRef;
+use alloc::vec::Vec;
 use core::marker::PhantomData;
 use core::mem;
-use alloc::vec::Vec;
 
 /// A small list of entity references allocated from a pool.
 ///
diff --git a/cranelift/entity/src/map.rs b/cranelift/entity/src/map.rs
index 937c4a49d9..7eb889e8b4 100644
--- a/cranelift/entity/src/map.rs
+++ b/cranelift/entity/src/map.rs
@@ -3,6 +3,7 @@
 use crate::iter::{Iter, IterMut};
 use crate::keys::Keys;
 use crate::EntityRef;
+use alloc::vec::Vec;
 use core::cmp::min;
 use core::marker::PhantomData;
 use core::ops::{Index, IndexMut};
@@ -13,7 +14,6 @@ use serde::{
     ser::{SerializeSeq, Serializer},
     Deserialize, Serialize,
 };
-use alloc::vec::Vec;
 
 /// A mapping `K -> V` for densely indexed entity references.
 ///
diff --git a/cranelift/entity/src/primary.rs b/cranelift/entity/src/primary.rs
index a464fe60f8..974033addd 100644
--- a/cranelift/entity/src/primary.rs
+++ b/cranelift/entity/src/primary.rs
@@ -3,14 +3,14 @@ use crate::boxed_slice::BoxedSlice;
 use crate::iter::{Iter, IterMut};
 use crate::keys::Keys;
 use crate::EntityRef;
+use alloc::boxed::Box;
+use alloc::vec::Vec;
 use core::iter::FromIterator;
 use core::marker::PhantomData;
 use core::ops::{Index, IndexMut};
 use core::slice;
 #[cfg(feature = "enable-serde")]
 use serde::{Deserialize, Serialize};
-use alloc::boxed::Box;
-use alloc::vec::Vec;
 
 /// A primary mapping `K -> V` allocating dense entity references.
 ///
diff --git a/cranelift/entity/src/set.rs b/cranelift/entity/src/set.rs
index 2379fe2a7d..c54ae4b31d 100644
--- a/cranelift/entity/src/set.rs
+++ b/cranelift/entity/src/set.rs
@@ -2,8 +2,8 @@
 
 use crate::keys::Keys;
 use crate::EntityRef;
-use core::marker::PhantomData;
 use alloc::vec::Vec;
+use core::marker::PhantomData;
 
 /// A set of `K` for densely indexed entity references.
 ///
diff --git a/cranelift/entity/src/sparse.rs b/cranelift/entity/src/sparse.rs
index 43d8d4d431..57d971b281 100644
--- a/cranelift/entity/src/sparse.rs
+++ b/cranelift/entity/src/sparse.rs
@@ -9,10 +9,10 @@
 
 use crate::map::SecondaryMap;
 use crate::EntityRef;
+use alloc::vec::Vec;
 use core::mem;
 use core::slice;
 use core::u32;
-use alloc::vec::Vec;
 
 /// Trait for extracting keys from values stored in a `SparseMap`.
 ///
diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 15488f7c2a..85a7725930 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -1,6 +1,7 @@
 //! A frontend for building Cranelift IR from other languages.
 use crate::ssa::{Block, SSABuilder, SideEffects};
 use crate::variable::Variable;
+use alloc::vec::Vec;
 use cranelift_codegen::cursor::{Cursor, FuncCursor};
 use cranelift_codegen::entity::{EntitySet, SecondaryMap};
 use cranelift_codegen::ir;
@@ -13,7 +14,6 @@ use cranelift_codegen::ir::{
 };
 use cranelift_codegen::isa::{TargetFrontendConfig, TargetIsa};
 use cranelift_codegen::packed_option::PackedOption;
-use alloc::vec::Vec;
 
 /// Structure used for translating a series of functions into Cranelift IR.
 ///
@@ -893,13 +893,13 @@ mod tests {
     use super::greatest_divisible_power_of_two;
     use crate::frontend::{FunctionBuilder, FunctionBuilderContext};
     use crate::Variable;
+    use alloc::string::ToString;
     use cranelift_codegen::entity::EntityRef;
     use cranelift_codegen::ir::types::*;
     use cranelift_codegen::ir::{AbiParam, ExternalName, Function, InstBuilder, Signature};
     use cranelift_codegen::isa::CallConv;
     use cranelift_codegen::settings;
     use cranelift_codegen::verifier::verify_function;
-    use alloc::string::ToString;
 
     fn sample_function(lazy_seal: bool) {
         let mut sig = Signature::new(CallConv::SystemV);
diff --git a/cranelift/frontend/src/ssa.rs b/cranelift/frontend/src/ssa.rs
index 0175c6381d..dcc729ebd5 100644
--- a/cranelift/frontend/src/ssa.rs
+++ b/cranelift/frontend/src/ssa.rs
@@ -6,6 +6,7 @@
 //! Lecture Notes in Computer Science, vol 7791. Springer, Berlin, Heidelberg
 
 use crate::Variable;
+use alloc::vec::Vec;
 use core::mem;
 use core::u32;
 use cranelift_codegen::cursor::{Cursor, FuncCursor};
@@ -17,7 +18,6 @@ use cranelift_codegen::ir::{Ebb, Function, Inst, InstBuilder, InstructionData, T
 use cranelift_codegen::packed_option::PackedOption;
 use cranelift_codegen::packed_option::ReservedValue;
 use smallvec::SmallVec;
-use alloc::vec::Vec;
 
 /// Structure containing the data relevant the construction of SSA for a given function.
 ///
diff --git a/cranelift/frontend/src/switch.rs b/cranelift/frontend/src/switch.rs
index 7353d7d049..25cba38ac3 100644
--- a/cranelift/frontend/src/switch.rs
+++ b/cranelift/frontend/src/switch.rs
@@ -1,9 +1,9 @@
 use super::HashMap;
 use crate::frontend::FunctionBuilder;
+use alloc::vec::Vec;
 use cranelift_codegen::ir::condcodes::IntCC;
 use cranelift_codegen::ir::*;
 use log::debug;
-use alloc::vec::Vec;
 
 type EntryIndex = u64;
 
@@ -290,8 +290,8 @@ impl ContiguousCaseRange {
 mod tests {
     use super::*;
     use crate::frontend::FunctionBuilderContext;
-    use cranelift_codegen::ir::Function;
     use alloc::string::ToString;
+    use cranelift_codegen::ir::Function;
 
     macro_rules! setup {
         ($default:expr, [$($index:expr,)*]) => {{

From a111ba8d1105c290d09eab9721f628e55f98651d Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 28 Sep 2019 19:51:54 +0200
Subject: [PATCH 2798/3084] Enable all-arch feature in test-no_std.sh

---
 cranelift/test-no_std.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh
index 43750a38ce..9f878d9f0b 100755
--- a/cranelift/test-no_std.sh
+++ b/cranelift/test-no_std.sh
@@ -21,10 +21,10 @@ for LIB in $LIBS; do
     pushd "$LIB" >/dev/null
 
     # Test with just "core" enabled.
-    cargo +nightly test --no-default-features --features core
+    cargo +nightly test --no-default-features --features "core all-arch"
 
     # Test with "core" and "std" enabled at the same time.
-    cargo +nightly test --features core
+    cargo +nightly test --features "core all-arch"
 
     popd >/dev/null
 done

From 277d9810416762088b28f0a8950f90c37f192df1 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 2 Oct 2019 11:13:33 -0700
Subject: [PATCH 2799/3084] Rewrite another `std::string::String` to `alloc`.

---
 cranelift/bforest/src/map.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/bforest/src/map.rs b/cranelift/bforest/src/map.rs
index fb7d4d9ddf..7d7e456623 100644
--- a/cranelift/bforest/src/map.rs
+++ b/cranelift/bforest/src/map.rs
@@ -6,7 +6,7 @@ use crate::packed_option::PackedOption;
 use core::fmt;
 use core::marker::PhantomData;
 #[cfg(test)]
-use std::string::String;
+use alloc::string::String;
 
 /// Tag type defining forest types for a map.
 struct MapTypes(PhantomData<(K, V)>);

From f9d802fb1dfad597af98767cdfe7e24f26dd0ebb Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 2 Oct 2019 11:22:40 -0700
Subject: [PATCH 2800/3084] Format with rustfmt.

---
 cranelift/bforest/src/map.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/bforest/src/map.rs b/cranelift/bforest/src/map.rs
index 7d7e456623..79ac018a98 100644
--- a/cranelift/bforest/src/map.rs
+++ b/cranelift/bforest/src/map.rs
@@ -3,10 +3,10 @@
 use super::{Comparator, Forest, Node, NodeData, NodePool, Path, INNER_SIZE};
 use crate::packed_option::PackedOption;
 #[cfg(test)]
+use alloc::string::String;
+#[cfg(test)]
 use core::fmt;
 use core::marker::PhantomData;
-#[cfg(test)]
-use alloc::string::String;
 
 /// Tag type defining forest types for a map.
 struct MapTypes(PhantomData<(K, V)>);

From 10be3e4ba8c7168e1f933a268309f1e5021bb53b Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Wed, 2 Oct 2019 12:40:35 -0700
Subject: [PATCH 2801/3084] cranelift-wasm: support multi-value Wasm (#1049)

This commit introduces initial support for multi-value Wasm. Wasm blocks and
calls can now take and return an arbitrary number of values.

The encoding for multi-value blocks means that we need to keep the contents of
the "Types" section around when translating function bodies. To do this, we
introduce a `WasmTypesMap` type that maps the type indices to their parameters
and returns, construct it when parsing the "Types" section, and shepherd it
through a bunch of functions and methods when translating function bodies.
---
 cranelift/src/clif-util.rs                |   8 +
 cranelift/src/wasm.rs                     |   7 +
 cranelift/wasm/src/code_translator.rs     | 243 +++++++++++++++-------
 cranelift/wasm/src/environ/dummy.rs       |  14 +-
 cranelift/wasm/src/environ/mod.rs         |   1 +
 cranelift/wasm/src/environ/spec.rs        |  20 ++
 cranelift/wasm/src/func_translator.rs     |  41 +++-
 cranelift/wasm/src/lib.rs                 |   2 +-
 cranelift/wasm/src/module_translator.rs   |   7 +-
 cranelift/wasm/src/sections_translator.rs |  15 +-
 cranelift/wasm/src/state.rs               | 116 +++++++++--
 cranelift/wasm/src/translation_utils.rs   |  87 +++++---
 cranelift/wasm/tests/wasm_testsuite.rs    |   1 +
 cranelift/wasmtests/fac-multi-value.wat   |  19 ++
 cranelift/wasmtests/multi-0.wat           |   3 +
 cranelift/wasmtests/multi-1.wat           |   6 +
 cranelift/wasmtests/multi-10.wat          |  10 +
 cranelift/wasmtests/multi-11.wat          |   7 +
 cranelift/wasmtests/multi-12.wat          |   9 +
 cranelift/wasmtests/multi-13.wat          |  10 +
 cranelift/wasmtests/multi-14.wat          |  10 +
 cranelift/wasmtests/multi-15.wat          |  22 ++
 cranelift/wasmtests/multi-2.wat           |   6 +
 cranelift/wasmtests/multi-3.wat           |  13 ++
 cranelift/wasmtests/multi-4.wat           |  13 ++
 cranelift/wasmtests/multi-5.wat           |  11 +
 cranelift/wasmtests/multi-6.wat           |  11 +
 cranelift/wasmtests/multi-7.wat           |   9 +
 cranelift/wasmtests/multi-8.wat           |  12 ++
 cranelift/wasmtests/multi-9.wat           |  15 ++
 30 files changed, 610 insertions(+), 138 deletions(-)
 create mode 100644 cranelift/wasmtests/fac-multi-value.wat
 create mode 100644 cranelift/wasmtests/multi-0.wat
 create mode 100644 cranelift/wasmtests/multi-1.wat
 create mode 100644 cranelift/wasmtests/multi-10.wat
 create mode 100644 cranelift/wasmtests/multi-11.wat
 create mode 100644 cranelift/wasmtests/multi-12.wat
 create mode 100644 cranelift/wasmtests/multi-13.wat
 create mode 100644 cranelift/wasmtests/multi-14.wat
 create mode 100644 cranelift/wasmtests/multi-15.wat
 create mode 100644 cranelift/wasmtests/multi-2.wat
 create mode 100644 cranelift/wasmtests/multi-3.wat
 create mode 100644 cranelift/wasmtests/multi-4.wat
 create mode 100644 cranelift/wasmtests/multi-5.wat
 create mode 100644 cranelift/wasmtests/multi-6.wat
 create mode 100644 cranelift/wasmtests/multi-7.wat
 create mode 100644 cranelift/wasmtests/multi-8.wat
 create mode 100644 cranelift/wasmtests/multi-9.wat

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 19402d4bff..676e9d334f 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -113,6 +113,12 @@ fn add_enable_simd_flag<'a>() -> clap::Arg<'a, 'a> {
         .help("Enable WASM's SIMD operations")
 }
 
+fn add_enable_multi_value<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("enable-multi-value")
+        .long("enable-multi-value")
+        .help("Enable WASM's multi-value support")
+}
+
 /// Returns a vector of clap value options and changes these options into a vector of strings
 fn get_vec(argument_vec: Option) -> Vec {
     let mut ret_vec: Vec = Vec::new();
@@ -144,6 +150,7 @@ fn add_wasm_or_compile<'a>(cmd: &str) -> clap::App<'a, 'a> {
         .arg(add_input_file_arg())
         .arg(add_debug_flag())
         .arg(add_enable_simd_flag())
+        .arg(add_enable_multi_value())
 }
 
 fn handle_debug_flag(debug: bool) {
@@ -304,6 +311,7 @@ fn main() {
                     rest_cmd.is_present("time-passes"),
                     rest_cmd.is_present("value-ranges"),
                     rest_cmd.is_present("enable-simd"),
+                    rest_cmd.is_present("enable-multi-value"),
                 )
             };
 
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index cc55238822..d2752bdb36 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -50,6 +50,7 @@ pub fn run(
     flag_report_times: bool,
     flag_calc_value_ranges: bool,
     flag_enable_simd: bool,
+    flag_enable_multi_value: bool,
 ) -> Result<(), String> {
     let parsed = parse_sets_and_triple(flag_set, flag_triple)?;
 
@@ -66,6 +67,7 @@ pub fn run(
             flag_report_times,
             flag_calc_value_ranges,
             flag_enable_simd,
+            flag_enable_multi_value,
             &path.to_path_buf(),
             &name,
             parsed.as_fisa(),
@@ -84,6 +86,7 @@ fn handle_module(
     flag_report_times: bool,
     flag_calc_value_ranges: bool,
     flag_enable_simd: bool,
+    flag_enable_multi_value: bool,
     path: &PathBuf,
     name: &str,
     fisa: FlagsOrIsa,
@@ -104,6 +107,10 @@ fn handle_module(
         if flag_enable_simd {
             features.enable_simd();
         }
+        if flag_enable_multi_value {
+            features.enable_multi_value();
+        }
+
         module_binary = match wat2wasm_with_features(&module_binary, features) {
             Ok(data) => data,
             Err(e) => return Err(e.to_string()),
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index ad5345768d..4416410664 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -23,10 +23,10 @@
 //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as
 //! argument.
 use super::{hash_map, HashMap};
-use crate::environ::{FuncEnvironment, GlobalVariable, ReturnMode, WasmResult};
-use crate::state::{ControlStackFrame, TranslationState};
+use crate::environ::{FuncEnvironment, GlobalVariable, ReturnMode, WasmResult, WasmTypesMap};
+use crate::state::{ControlStackFrame, ElseData, TranslationState};
 use crate::translation_utils::{
-    blocktype_to_type, f32_translation, f64_translation, num_return_values,
+    blocktype_params_results, ebb_with_params, f32_translation, f64_translation,
 };
 use crate::translation_utils::{FuncIndex, MemoryIndex, SignatureIndex, TableIndex};
 use crate::wasm_unsupported;
@@ -43,13 +43,14 @@ use wasmparser::{MemoryImmediate, Operator};
 /// Translates wasm operators into Cranelift IR instructions. Returns `true` if it inserted
 /// a return.
 pub fn translate_operator(
+    wasm_types: &WasmTypesMap,
     op: &Operator,
     builder: &mut FunctionBuilder,
     state: &mut TranslationState,
     environ: &mut FE,
 ) -> WasmResult<()> {
     if !state.reachable {
-        translate_unreachable_operator(&op, builder, state);
+        translate_unreachable_operator(wasm_types, &op, builder, state)?;
         return Ok(());
     }
 
@@ -132,27 +133,52 @@ pub fn translate_operator(
          *  possible `Ebb`'s arguments values.
          ***********************************************************************************/
         Operator::Block { ty } => {
-            let next = builder.create_ebb();
-            if let Some(ty_cre) = blocktype_to_type(*ty)? {
-                builder.append_ebb_param(next, ty_cre);
-            }
-            state.push_block(next, num_return_values(*ty)?);
+            let (params, results) = blocktype_params_results(wasm_types, *ty)?;
+            let next = ebb_with_params(builder, results)?;
+            state.push_block(next, params.len(), results.len());
         }
         Operator::Loop { ty } => {
-            let loop_body = builder.create_ebb();
-            let next = builder.create_ebb();
-            if let Some(ty_cre) = blocktype_to_type(*ty)? {
-                builder.append_ebb_param(next, ty_cre);
-            }
-            builder.ins().jump(loop_body, &[]);
-            state.push_loop(loop_body, next, num_return_values(*ty)?);
+            let (params, results) = blocktype_params_results(wasm_types, *ty)?;
+            let loop_body = ebb_with_params(builder, params)?;
+            let next = ebb_with_params(builder, results)?;
+            builder.ins().jump(loop_body, state.peekn(params.len()));
+            state.push_loop(loop_body, next, params.len(), results.len());
+
+            // Pop the initial `Ebb` actuals and replace them with the `Ebb`'s
+            // params since control flow joins at the top of the loop.
+            state.popn(params.len());
+            state.stack.extend_from_slice(builder.ebb_params(loop_body));
+
             builder.switch_to_block(loop_body);
             environ.translate_loop_header(builder.cursor())?;
         }
         Operator::If { ty } => {
             let val = state.pop1();
-            let if_not = builder.create_ebb();
-            let jump_inst = builder.ins().brz(val, if_not, &[]);
+
+            let (params, results) = blocktype_params_results(wasm_types, *ty)?;
+            let (destination, else_data) = if params == results {
+                // It is possible there is no `else` block, so we will only
+                // allocate an ebb for it if/when we find the `else`. For now,
+                // we if the condition isn't true, then we jump directly to the
+                // destination ebb following the whole `if...end`. If we do end
+                // up discovering an `else`, then we will allocate an ebb for it
+                // and go back and patch the jump.
+                let destination = ebb_with_params(builder, results)?;
+                let branch_inst = builder
+                    .ins()
+                    .brz(val, destination, state.peekn(params.len()));
+                (destination, ElseData::NoElse { branch_inst })
+            } else {
+                // The `if` type signature is not valid without an `else` block,
+                // so we eagerly allocate the `else` block here.
+                let destination = ebb_with_params(builder, results)?;
+                let else_block = ebb_with_params(builder, params)?;
+                builder
+                    .ins()
+                    .brz(val, else_block, state.peekn(params.len()));
+                builder.seal_block(else_block);
+                (destination, ElseData::WithElse { else_block })
+            };
 
             #[cfg(feature = "basic-blocks")]
             {
@@ -168,41 +194,63 @@ pub fn translate_operator(
             //   and we add nothing;
             // - either the If have an Else clause, in that case the destination of this jump
             //   instruction will be changed later when we translate the Else operator.
-            if let Some(ty_cre) = blocktype_to_type(*ty)? {
-                builder.append_ebb_param(if_not, ty_cre);
-            }
-            state.push_if(jump_inst, if_not, num_return_values(*ty)?);
+            state.push_if(destination, else_data, params.len(), results.len(), *ty);
         }
         Operator::Else => {
-            // We take the control frame pushed by the if, use its ebb as the else body
-            // and push a new control frame with a new ebb for the code after the if/then/else
-            // At the end of the then clause we jump to the destination
             let i = state.control_stack.len() - 1;
-            let (destination, return_count, branch_inst, ref mut reachable_from_top) =
-                match state.control_stack[i] {
-                    ControlStackFrame::If {
-                        destination,
-                        num_return_values,
-                        branch_inst,
-                        reachable_from_top,
-                        ..
-                    } => (
-                        destination,
-                        num_return_values,
-                        branch_inst,
-                        reachable_from_top,
-                    ),
-                    _ => panic!("should not happen"),
-                };
-            // The if has an else, so there's no branch to the end from the top.
-            *reachable_from_top = false;
-            builder.ins().jump(destination, state.peekn(return_count));
-            state.popn(return_count);
-            // We change the target of the branch instruction
-            let else_ebb = builder.create_ebb();
-            builder.change_jump_destination(branch_inst, else_ebb);
-            builder.seal_block(else_ebb);
-            builder.switch_to_block(else_ebb);
+            match state.control_stack[i] {
+                ControlStackFrame::If {
+                    else_data: ElseData::NoElse { branch_inst },
+                    ref mut reachable_from_top,
+                    blocktype,
+                    ..
+                } => {
+                    // We take the control frame pushed by the if, use its ebb
+                    // as the else body and push a new control frame with a new
+                    // ebb for the code after the if/then/else. At the end of the
+                    // then clause we jump to the destination.
+
+                    // The `if` has an `else`, so there's no branch to the end from the top.
+                    *reachable_from_top = false;
+
+                    let (params, _results) = blocktype_params_results(wasm_types, blocktype)?;
+                    let else_ebb = ebb_with_params(builder, params)?;
+                    builder.ins().jump(else_ebb, state.peekn(params.len()));
+                    state.popn(params.len());
+
+                    // You might be expecting that we push the parameters for this
+                    // `else` block here, something like this:
+                    //
+                    //     state.pushn(&control_stack_frame.params);
+                    //
+                    // We don't do that because they are already on the top of the stack
+                    // for us: we pushed the parameters twice when we saw the initial
+                    // `if` so that we wouldn't have to save the parameters in the
+                    // `ControlStackFrame` as another `Vec` allocation.
+
+                    builder.change_jump_destination(branch_inst, else_ebb);
+                    builder.seal_block(else_ebb);
+                    builder.switch_to_block(else_ebb);
+
+                    // NB: we don't bother updating the control frame's
+                    // `ElseData` because nothing else will read it.
+                }
+                ControlStackFrame::If {
+                    destination,
+                    num_return_values,
+                    else_data: ElseData::WithElse { else_block, .. },
+                    reachable_from_top,
+                    ..
+                } => {
+                    debug_assert!(!reachable_from_top);
+                    builder
+                        .ins()
+                        .jump(destination, state.peekn(num_return_values));
+                    state.popn(num_return_values);
+                    builder.switch_to_block(else_block);
+                }
+                _ => unreachable!(),
+            }
         }
         Operator::End => {
             let frame = state.control_stack.pop().unwrap();
@@ -211,6 +259,12 @@ pub fn translate_operator(
                 builder
                     .ins()
                     .jump(frame.following_code(), state.peekn(return_count));
+                // You might expect that if we just finished an `if` block that
+                // didn't have a corresponding `else` block, then we would clean
+                // up our duplicate set of parameters that we pushed earlier
+                // right here. However, we don't have to explicitly do that,
+                // since we truncate the stack back to the original height
+                // below.
             }
             builder.switch_to_block(frame.following_code());
             builder.seal_block(frame.following_code());
@@ -1139,40 +1193,71 @@ pub fn translate_operator(
 /// are dropped but special ones like `End` or `Else` signal the potential end of the unreachable
 /// portion so the translation state must be updated accordingly.
 fn translate_unreachable_operator(
+    wasm_types: &WasmTypesMap,
     op: &Operator,
     builder: &mut FunctionBuilder,
     state: &mut TranslationState,
-) {
+) -> WasmResult<()> {
     match *op {
-        Operator::If { ty: _ } => {
+        Operator::If { ty } => {
             // Push a placeholder control stack entry. The if isn't reachable,
             // so we don't have any branches anywhere.
-            state.push_if(ir::Inst::reserved_value(), ir::Ebb::reserved_value(), 0);
+            state.push_if(
+                ir::Ebb::reserved_value(),
+                ElseData::NoElse {
+                    branch_inst: ir::Inst::reserved_value(),
+                },
+                0,
+                0,
+                ty,
+            );
         }
         Operator::Loop { ty: _ } | Operator::Block { ty: _ } => {
-            state.push_block(ir::Ebb::reserved_value(), 0);
+            state.push_block(ir::Ebb::reserved_value(), 0, 0);
         }
         Operator::Else => {
             let i = state.control_stack.len() - 1;
-            if let ControlStackFrame::If {
-                branch_inst,
-                ref mut reachable_from_top,
-                ..
-            } = state.control_stack[i]
-            {
-                if *reachable_from_top {
-                    // We have a branch from the top of the if to the else.
-                    state.reachable = true;
-                    // And because there's an else, there can no longer be a
-                    // branch from the top directly to the end.
-                    *reachable_from_top = false;
+            match state.control_stack[i] {
+                ControlStackFrame::If {
+                    else_data: ElseData::NoElse { branch_inst },
+                    ref mut reachable_from_top,
+                    blocktype,
+                    ..
+                } => {
+                    if *reachable_from_top {
+                        // We have a branch from the top of the if to the else.
+                        state.reachable = true;
+                        // And because there's an else, there can no longer be a
+                        // branch from the top directly to the end.
+                        *reachable_from_top = false;
 
-                    // We change the target of the branch instruction
-                    let else_ebb = builder.create_ebb();
-                    builder.change_jump_destination(branch_inst, else_ebb);
-                    builder.seal_block(else_ebb);
-                    builder.switch_to_block(else_ebb);
+                        let (params, _results) = blocktype_params_results(wasm_types, blocktype)?;
+                        let else_ebb = ebb_with_params(builder, params)?;
+
+                        // We change the target of the branch instruction.
+                        builder.change_jump_destination(branch_inst, else_ebb);
+                        builder.seal_block(else_ebb);
+                        builder.switch_to_block(else_ebb);
+
+                        // Again, no need to push the parameters for the `else`,
+                        // since we already did when we saw the original `if`. See
+                        // the comment for translating `Operator::Else` in
+                        // `translate_operator` for details.
+                    }
                 }
+                ControlStackFrame::If {
+                    else_data: ElseData::WithElse { else_block, .. },
+                    reachable_from_top,
+                    ..
+                } => {
+                    debug_assert!(
+                        !reachable_from_top,
+                        "should not be reachable from top if we have an else block"
+                    );
+                    builder.switch_to_block(else_block);
+                    state.reachable = true;
+                }
+                _ => unreachable!(),
             }
         }
         Operator::End => {
@@ -1192,7 +1277,17 @@ fn translate_unreachable_operator(
                     false
                 }
                 ControlStackFrame::If {
-                    reachable_from_top, ..
+                    else_data: ElseData::WithElse { .. },
+                    reachable_from_top,
+                    ..
+                } => {
+                    debug_assert!(!reachable_from_top);
+                    true
+                }
+                ControlStackFrame::If {
+                    else_data: ElseData::NoElse { .. },
+                    reachable_from_top,
+                    ..
                 } => {
                     // A reachable if without an else has a branch from the top
                     // directly to the bottom.
@@ -1216,6 +1311,8 @@ fn translate_unreachable_operator(
             // We don't translate because this is unreachable code
         }
     }
+
+    Ok(())
 }
 
 /// Get the address+offset to use for a heap access.
@@ -1342,7 +1439,7 @@ fn translate_br_if_args(
         // code that comes after it
         frame.set_branched_to_exit();
         let return_count = if frame.is_loop() {
-            0
+            frame.num_param_values()
         } else {
             frame.num_return_values()
         };
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index fde1e6b19d..cd84c00a28 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -5,7 +5,9 @@
 //! [wasmtime-environ]: https://crates.io/crates/wasmtime-environ
 //! [Wasmtime]: https://github.com/CraneStation/wasmtime
 
-use crate::environ::{FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmResult};
+use crate::environ::{
+    FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmResult, WasmTypesMap,
+};
 use crate::func_translator::FuncTranslator;
 use crate::translation_utils::{
     DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table,
@@ -529,6 +531,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
 
     fn define_function_body(
         &mut self,
+        wasm_types: &WasmTypesMap,
         body_bytes: &'data [u8],
         body_offset: usize,
     ) -> WasmResult<()> {
@@ -542,8 +545,13 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
             if self.debug_info {
                 func.collect_debug_info();
             }
-            self.trans
-                .translate(body_bytes, body_offset, &mut func, &mut func_environ)?;
+            self.trans.translate(
+                wasm_types,
+                body_bytes,
+                body_offset,
+                &mut func,
+                &mut func_environ,
+            )?;
             func
         };
         self.func_bytecode_sizes.push(body_bytes.len());
diff --git a/cranelift/wasm/src/environ/mod.rs b/cranelift/wasm/src/environ/mod.rs
index 4b7405ea7b..ac29af7c8f 100644
--- a/cranelift/wasm/src/environ/mod.rs
+++ b/cranelift/wasm/src/environ/mod.rs
@@ -7,4 +7,5 @@ mod spec;
 pub use crate::environ::dummy::DummyEnvironment;
 pub use crate::environ::spec::{
     FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmError, WasmResult,
+    WasmTypesMap,
 };
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index e83e675e72..87607755e4 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -15,6 +15,7 @@ use cranelift_codegen::cursor::FuncCursor;
 use cranelift_codegen::ir::immediates::Offset32;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::isa::TargetFrontendConfig;
+use cranelift_entity::PrimaryMap;
 use cranelift_frontend::FunctionBuilder;
 use failure_derive::Fail;
 use std::boxed::Box;
@@ -103,6 +104,24 @@ pub enum ReturnMode {
     FallthroughReturn,
 }
 
+/// A map containing a Wasm module's original, raw signatures.
+///
+/// This is used for translating multi-value Wasm blocks inside functions, which
+/// are encoded to refer to their type signature via index.
+#[derive(Debug)]
+pub struct WasmTypesMap {
+    pub(crate) inner:
+        PrimaryMap, Box<[wasmparser::Type]>)>,
+}
+
+impl WasmTypesMap {
+    pub(crate) fn new() -> Self {
+        WasmTypesMap {
+            inner: PrimaryMap::new(),
+        }
+    }
+}
+
 /// Environment affecting the translation of a single WebAssembly function.
 ///
 /// A `FuncEnvironment` trait object is required to translate a WebAssembly function to Cranelift
@@ -449,6 +468,7 @@ pub trait ModuleEnvironment<'data> {
     /// functions is already provided by `reserve_func_types`.
     fn define_function_body(
         &mut self,
+        wasm_types: &WasmTypesMap,
         body_bytes: &'data [u8],
         body_offset: usize,
     ) -> WasmResult<()>;
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index f947ddd065..0f9f979e92 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -5,7 +5,7 @@
 //! WebAssembly module and the runtime environment.
 
 use crate::code_translator::translate_operator;
-use crate::environ::{FuncEnvironment, ReturnMode, WasmResult};
+use crate::environ::{FuncEnvironment, ReturnMode, WasmResult, WasmTypesMap};
 use crate::state::TranslationState;
 use crate::translation_utils::get_vmctx_value_label;
 use crate::wasm_unsupported;
@@ -55,12 +55,14 @@ impl FuncTranslator {
     ///
     pub fn translate(
         &mut self,
+        wasm_types: &WasmTypesMap,
         code: &[u8],
         code_offset: usize,
         func: &mut ir::Function,
         environ: &mut FE,
     ) -> WasmResult<()> {
         self.translate_from_reader(
+            wasm_types,
             BinaryReader::new_with_offset(code, code_offset),
             func,
             environ,
@@ -70,6 +72,7 @@ impl FuncTranslator {
     /// Translate a binary WebAssembly function from a `BinaryReader`.
     pub fn translate_from_reader(
         &mut self,
+        wasm_types: &WasmTypesMap,
         mut reader: BinaryReader,
         func: &mut ir::Function,
         environ: &mut FE,
@@ -105,7 +108,7 @@ impl FuncTranslator {
         self.state.initialize(&builder.func.signature, exit_block);
 
         parse_local_decls(&mut reader, &mut builder, num_params, environ)?;
-        parse_function_body(reader, &mut builder, &mut self.state, environ)?;
+        parse_function_body(wasm_types, reader, &mut builder, &mut self.state, environ)?;
 
         builder.finalize();
         Ok(())
@@ -203,6 +206,7 @@ fn declare_locals(
 /// This assumes that the local variable declarations have already been parsed and function
 /// arguments and locals are declared in the builder.
 fn parse_function_body(
+    wasm_types: &WasmTypesMap,
     mut reader: BinaryReader,
     builder: &mut FunctionBuilder,
     state: &mut TranslationState,
@@ -216,7 +220,7 @@ fn parse_function_body(
         builder.set_srcloc(cur_srcloc(&reader));
         let op = reader.read_operator()?;
         environ.before_translate_operator(&op, builder, state)?;
-        translate_operator(&op, builder, state, environ)?;
+        translate_operator(wasm_types, &op, builder, state, environ)?;
         environ.after_translate_operator(&op, builder, state)?;
     }
 
@@ -254,7 +258,7 @@ fn cur_srcloc(reader: &BinaryReader) -> ir::SourceLoc {
 #[cfg(test)]
 mod tests {
     use super::{FuncTranslator, ReturnMode};
-    use crate::environ::DummyEnvironment;
+    use crate::environ::{DummyEnvironment, WasmTypesMap};
     use cranelift_codegen::ir::types::I32;
     use cranelift_codegen::{ir, isa, settings, Context};
     use log::debug;
@@ -286,6 +290,7 @@ mod tests {
             false,
         );
 
+        let wasm_types = WasmTypesMap::new();
         let mut ctx = Context::new();
 
         ctx.func.name = ir::ExternalName::testcase("small1");
@@ -293,7 +298,13 @@ mod tests {
         ctx.func.signature.returns.push(ir::AbiParam::new(I32));
 
         trans
-            .translate(&BODY, 0, &mut ctx.func, &mut runtime.func_env())
+            .translate(
+                &wasm_types,
+                &BODY,
+                0,
+                &mut ctx.func,
+                &mut runtime.func_env(),
+            )
             .unwrap();
         debug!("{}", ctx.func.display(None));
         ctx.verify(&flags).unwrap();
@@ -325,6 +336,8 @@ mod tests {
             ReturnMode::NormalReturns,
             false,
         );
+
+        let wasm_types = WasmTypesMap::new();
         let mut ctx = Context::new();
 
         ctx.func.name = ir::ExternalName::testcase("small2");
@@ -332,7 +345,13 @@ mod tests {
         ctx.func.signature.returns.push(ir::AbiParam::new(I32));
 
         trans
-            .translate(&BODY, 0, &mut ctx.func, &mut runtime.func_env())
+            .translate(
+                &wasm_types,
+                &BODY,
+                0,
+                &mut ctx.func,
+                &mut runtime.func_env(),
+            )
             .unwrap();
         debug!("{}", ctx.func.display(None));
         ctx.verify(&flags).unwrap();
@@ -373,13 +392,21 @@ mod tests {
             ReturnMode::NormalReturns,
             false,
         );
+
+        let wasm_types = WasmTypesMap::new();
         let mut ctx = Context::new();
 
         ctx.func.name = ir::ExternalName::testcase("infloop");
         ctx.func.signature.returns.push(ir::AbiParam::new(I32));
 
         trans
-            .translate(&BODY, 0, &mut ctx.func, &mut runtime.func_env())
+            .translate(
+                &wasm_types,
+                &BODY,
+                0,
+                &mut ctx.func,
+                &mut runtime.func_env(),
+            )
             .unwrap();
         debug!("{}", ctx.func.display(None));
         ctx.verify(&flags).unwrap();
diff --git a/cranelift/wasm/src/lib.rs b/cranelift/wasm/src/lib.rs
index 297c9df5f0..f4fa3a7358 100644
--- a/cranelift/wasm/src/lib.rs
+++ b/cranelift/wasm/src/lib.rs
@@ -59,7 +59,7 @@ mod translation_utils;
 
 pub use crate::environ::{
     DummyEnvironment, FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmError,
-    WasmResult,
+    WasmResult, WasmTypesMap,
 };
 pub use crate::func_translator::FuncTranslator;
 pub use crate::module_translator::translate_module;
diff --git a/cranelift/wasm/src/module_translator.rs b/cranelift/wasm/src/module_translator.rs
index 462f79ab24..4076b8c740 100644
--- a/cranelift/wasm/src/module_translator.rs
+++ b/cranelift/wasm/src/module_translator.rs
@@ -1,6 +1,6 @@
 //! Translation skeleton that traverses the whole WebAssembly module and call helper functions
 //! to deal with each part of it.
-use crate::environ::{ModuleEnvironment, WasmError, WasmResult};
+use crate::environ::{ModuleEnvironment, WasmError, WasmResult, WasmTypesMap};
 use crate::sections_translator::{
     parse_code_section, parse_data_section, parse_element_section, parse_export_section,
     parse_function_section, parse_global_section, parse_import_section, parse_memory_section,
@@ -17,12 +17,13 @@ pub fn translate_module<'data>(
 ) -> WasmResult<()> {
     let _tt = timing::wasm_translate_module();
     let mut reader = ModuleReader::new(data)?;
+    let mut wasm_types = WasmTypesMap::new();
 
     while !reader.eof() {
         let section = reader.read()?;
         match section.content()? {
             SectionContent::Type(types) => {
-                parse_type_section(types, environ)?;
+                parse_type_section(types, &mut wasm_types, environ)?;
             }
 
             SectionContent::Import(imports) => {
@@ -58,7 +59,7 @@ pub fn translate_module<'data>(
             }
 
             SectionContent::Code(code) => {
-                parse_code_section(code, environ)?;
+                parse_code_section(code, &wasm_types, environ)?;
             }
 
             SectionContent::Data(data) => {
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index a961a71ca9..d14852f609 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -7,7 +7,7 @@
 //! The special case of the initialize expressions for table elements offsets or global variables
 //! is handled, according to the semantics of WebAssembly, to only specific expressions that are
 //! interpreted on the fly.
-use crate::environ::{ModuleEnvironment, WasmResult};
+use crate::environ::{ModuleEnvironment, WasmResult, WasmTypesMap};
 use crate::translation_utils::{
     tabletype_to_type, type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory,
     MemoryIndex, SignatureIndex, Table, TableElementType, TableIndex,
@@ -29,16 +29,19 @@ use wasmparser::{
 /// Parses the Type section of the wasm module.
 pub fn parse_type_section(
     types: TypeSectionReader,
+    wasm_types: &mut WasmTypesMap,
     environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
-    environ.reserve_signatures(types.get_count())?;
+    let count = types.get_count();
+    wasm_types.inner.reserve(count as usize);
+    environ.reserve_signatures(count)?;
 
     for entry in types {
         match entry? {
             FuncType {
                 form: wasmparser::Type::Func,
-                ref params,
-                ref returns,
+                params,
+                returns,
             } => {
                 let mut sig = Signature::new(environ.target_config().default_call_conv);
                 sig.params.extend(params.iter().map(|ty| {
@@ -52,6 +55,7 @@ pub fn parse_type_section(
                     AbiParam::new(cret_arg)
                 }));
                 environ.declare_signature(sig)?;
+                wasm_types.inner.push((params, returns));
             }
             ty => {
                 return Err(wasm_unsupported!(
@@ -323,13 +327,14 @@ pub fn parse_element_section<'data>(
 /// Parses the Code section of the wasm module.
 pub fn parse_code_section<'data>(
     code: CodeSectionReader<'data>,
+    wasm_types: &WasmTypesMap,
     environ: &mut dyn ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
     for body in code {
         let mut reader = body?.get_binary_reader();
         let size = reader.bytes_remaining();
         let offset = reader.original_position();
-        environ.define_function_body(reader.read_bytes(size)?, offset)?;
+        environ.define_function_body(wasm_types, reader.read_bytes(size)?, offset)?;
     }
     Ok(())
 }
diff --git a/cranelift/wasm/src/state.rs b/cranelift/wasm/src/state.rs
index 7359482893..cd84434e2d 100644
--- a/cranelift/wasm/src/state.rs
+++ b/cranelift/wasm/src/state.rs
@@ -9,6 +9,24 @@ use crate::translation_utils::{FuncIndex, GlobalIndex, MemoryIndex, SignatureInd
 use cranelift_codegen::ir::{self, Ebb, Inst, Value};
 use std::vec::Vec;
 
+/// Information about the presence of an associated `else` for an `if`, or the
+/// lack thereof.
+#[derive(Debug)]
+pub enum ElseData {
+    /// The `if` does not already have an `else` block.
+    NoElse {
+        /// If we discover that we need an `else` block, this is the jump
+        /// instruction that needs to be fixed up to point to the new `else`
+        /// block rather than the destination block after the `if...end`.
+        branch_inst: Inst,
+    },
+    /// We have already allocated an `else` block.
+    WithElse {
+        /// This is the `else` block.
+        else_block: Ebb,
+    },
+}
+
 /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following
 /// fields:
 ///
@@ -23,14 +41,17 @@ use std::vec::Vec;
 pub enum ControlStackFrame {
     If {
         destination: Ebb,
-        branch_inst: Inst,
+        else_data: ElseData,
+        num_param_values: usize,
         num_return_values: usize,
         original_stack_size: usize,
         exit_is_branched_to: bool,
         reachable_from_top: bool,
+        blocktype: wasmparser::TypeOrFuncType,
     },
     Block {
         destination: Ebb,
+        num_param_values: usize,
         num_return_values: usize,
         original_stack_size: usize,
         exit_is_branched_to: bool,
@@ -38,6 +59,7 @@ pub enum ControlStackFrame {
     Loop {
         destination: Ebb,
         header: Ebb,
+        num_param_values: usize,
         num_return_values: usize,
         original_stack_size: usize,
     },
@@ -58,6 +80,19 @@ impl ControlStackFrame {
             } => num_return_values,
         }
     }
+    pub fn num_param_values(&self) -> usize {
+        match *self {
+            ControlStackFrame::If {
+                num_param_values, ..
+            }
+            | ControlStackFrame::Block {
+                num_param_values, ..
+            }
+            | ControlStackFrame::Loop {
+                num_param_values, ..
+            } => num_param_values,
+        }
+    }
     pub fn following_code(&self) -> Ebb {
         match *self {
             ControlStackFrame::If { destination, .. }
@@ -202,6 +237,7 @@ impl TranslationState {
         self.clear();
         self.push_block(
             exit_block,
+            0,
             sig.returns
                 .iter()
                 .filter(|arg| arg.purpose == ir::ArgumentPurpose::Normal)
@@ -221,12 +257,17 @@ impl TranslationState {
 
     /// Pop one value.
     pub(crate) fn pop1(&mut self) -> Value {
-        self.stack.pop().unwrap()
+        self.stack
+            .pop()
+            .expect("attempted to pop a value from an empty stack")
     }
 
     /// Peek at the top of the stack without popping it.
     pub(crate) fn peek1(&self) -> Value {
-        *self.stack.last().unwrap()
+        *self
+            .stack
+            .last()
+            .expect("attempted to peek at a value on an empty stack")
     }
 
     /// Pop two values. Return them in the order they were pushed.
@@ -248,31 +289,58 @@ impl TranslationState {
     ///
     /// The popped values are not returned. Use `peekn` to look at them before popping.
     pub(crate) fn popn(&mut self, n: usize) {
+        debug_assert!(
+            n <= self.stack.len(),
+            "popn({}) but stack only has {} values",
+            n,
+            self.stack.len()
+        );
         let new_len = self.stack.len() - n;
         self.stack.truncate(new_len);
     }
 
     /// Peek at the top `n` values on the stack in the order they were pushed.
     pub(crate) fn peekn(&self, n: usize) -> &[Value] {
+        debug_assert!(
+            n <= self.stack.len(),
+            "peekn({}) but stack only has {} values",
+            n,
+            self.stack.len()
+        );
         &self.stack[self.stack.len() - n..]
     }
 
     /// Push a block on the control stack.
-    pub(crate) fn push_block(&mut self, following_code: Ebb, num_result_types: usize) {
+    pub(crate) fn push_block(
+        &mut self,
+        following_code: Ebb,
+        num_param_types: usize,
+        num_result_types: usize,
+    ) {
+        debug_assert!(num_param_types <= self.stack.len());
         self.control_stack.push(ControlStackFrame::Block {
             destination: following_code,
-            original_stack_size: self.stack.len(),
+            original_stack_size: self.stack.len() - num_param_types,
+            num_param_values: num_param_types,
             num_return_values: num_result_types,
             exit_is_branched_to: false,
         });
     }
 
     /// Push a loop on the control stack.
-    pub(crate) fn push_loop(&mut self, header: Ebb, following_code: Ebb, num_result_types: usize) {
+    pub(crate) fn push_loop(
+        &mut self,
+        header: Ebb,
+        following_code: Ebb,
+        num_param_types: usize,
+        num_result_types: usize,
+    ) {
+        debug_assert!(num_param_types <= self.stack.len());
         self.control_stack.push(ControlStackFrame::Loop {
             header,
             destination: following_code,
-            original_stack_size: self.stack.len(),
+            original_stack_size: self.stack.len() - num_param_types,
+            num_param_values: num_param_types,
             num_return_values: num_result_types,
         });
     }
@@ -280,17 +348,39 @@ impl TranslationState {
     /// Push an if on the control stack.
     pub(crate) fn push_if(
         &mut self,
-        branch_inst: Inst,
-        following_code: Ebb,
+        destination: Ebb,
+        else_data: ElseData,
+        num_param_types: usize,
         num_result_types: usize,
+        blocktype: wasmparser::TypeOrFuncType,
     ) {
+        debug_assert!(num_param_types <= self.stack.len());
+
+        // Push a second copy of our `if`'s parameters on the stack. This lets
+        // us avoid saving them on the side in the `ControlStackFrame` for our
+        // `else` block (if it exists), which would require a second heap
+        // allocation. See also the comment in `translate_operator` for
+        // `Operator::Else`.
+        self.stack.reserve(num_param_types);
+        for i in (self.stack.len() - num_param_types)..self.stack.len() {
+            let val = self.stack[i];
+            self.stack.push(val);
+        }
+
+        let has_else = match else_data {
+            ElseData::NoElse { .. } => false,
+            ElseData::WithElse { .. } => true,
+        };
+
         self.control_stack.push(ControlStackFrame::If {
-            branch_inst,
-            destination: following_code,
-            original_stack_size: self.stack.len(),
+            destination,
+            else_data,
+            original_stack_size: self.stack.len() - num_param_types,
+            num_param_values: num_param_types,
             num_return_values: num_result_types,
             exit_is_branched_to: false,
-            reachable_from_top: self.reachable,
+            reachable_from_top: self.reachable && !has_else,
+            blocktype,
         });
     }
 }
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index df910c534b..aba5ce5178 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -1,10 +1,11 @@
 //! Helper functions and structures for the translation.
-use crate::environ::WasmResult;
+use crate::environ::{WasmResult, WasmTypesMap};
 use crate::wasm_unsupported;
 use core::u32;
 use cranelift_codegen::entity::entity_impl;
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::immediates::V128Imm;
+use cranelift_frontend::FunctionBuilder;
 #[cfg(feature = "enable-serde")]
 use serde::{Deserialize, Serialize};
 use wasmparser;
@@ -145,23 +146,61 @@ pub fn tabletype_to_type(ty: wasmparser::Type) -> WasmResult> {
     }
 }
 
-/// Helper function translating wasmparser block signatures to Cranelift types when possible.
-pub fn blocktype_to_type(ty_or_ft: wasmparser::TypeOrFuncType) -> WasmResult> {
-    match ty_or_ft {
+/// Get the parameter and result types for the given Wasm blocktype.
+pub fn blocktype_params_results(
+    wasm_types: &WasmTypesMap,
+    ty_or_ft: wasmparser::TypeOrFuncType,
+) -> WasmResult<(&[wasmparser::Type], &[wasmparser::Type])> {
+    Ok(match ty_or_ft {
         wasmparser::TypeOrFuncType::Type(ty) => match ty {
-            wasmparser::Type::I32 => Ok(Some(ir::types::I32)),
-            wasmparser::Type::I64 => Ok(Some(ir::types::I64)),
-            wasmparser::Type::F32 => Ok(Some(ir::types::F32)),
-            wasmparser::Type::F64 => Ok(Some(ir::types::F64)),
-            wasmparser::Type::V128 => Ok(Some(ir::types::I8X16)),
-            wasmparser::Type::EmptyBlockType => Ok(None),
-            ty => Err(wasm_unsupported!("blocktype_to_type: type {:?}", ty)),
+            wasmparser::Type::I32 => (&[], &[wasmparser::Type::I32]),
+            wasmparser::Type::I64 => (&[], &[wasmparser::Type::I64]),
+            wasmparser::Type::F32 => (&[], &[wasmparser::Type::F32]),
+            wasmparser::Type::F64 => (&[], &[wasmparser::Type::F64]),
+            wasmparser::Type::V128 => (&[], &[wasmparser::Type::V128]),
+            wasmparser::Type::EmptyBlockType => (&[], &[]),
+            ty => return Err(wasm_unsupported!("blocktype_params_results: type {:?}", ty)),
         },
-        wasmparser::TypeOrFuncType::FuncType(_) => Err(wasm_unsupported!(
-            "blocktype_to_type: multi-value block signature {:?}",
-            ty_or_ft
-        )),
+        wasmparser::TypeOrFuncType::FuncType(ty_index) => {
+            let sig_idx = SignatureIndex::from_u32(ty_index);
+            let (ref params, ref returns) = wasm_types.inner[sig_idx];
+            (&*params, &*returns)
+        }
+    })
+}
+
+/// Create an `Ebb` with the given Wasm parameters.
+pub fn ebb_with_params(
+    builder: &mut FunctionBuilder,
+    params: &[wasmparser::Type],
+) -> WasmResult {
+    let ebb = builder.create_ebb();
+    for ty in params.iter() {
+        match ty {
+            wasmparser::Type::I32 => {
+                builder.append_ebb_param(ebb, ir::types::I32);
+            }
+            wasmparser::Type::I64 => {
+                builder.append_ebb_param(ebb, ir::types::I64);
+            }
+            wasmparser::Type::F32 => {
+                builder.append_ebb_param(ebb, ir::types::F32);
+            }
+            wasmparser::Type::F64 => {
+                builder.append_ebb_param(ebb, ir::types::F64);
+            }
+            wasmparser::Type::V128 => {
+                builder.append_ebb_param(ebb, ir::types::I8X16);
+            }
+            ty => {
+                return Err(wasm_unsupported!(
+                    "ebb_with_params: type {:?} in multi-value block's signature",
+                    ty
+                ))
+            }
+        }
     }
+    Ok(ebb)
 }
 
 /// Turns a `wasmparser` `f32` into a `Cranelift` one.
@@ -174,24 +213,6 @@ pub fn f64_translation(x: wasmparser::Ieee64) -> ir::immediates::Ieee64 {
     ir::immediates::Ieee64::with_bits(x.bits())
 }
 
-/// Translate a `wasmparser` type into its `Cranelift` equivalent, when possible
-pub fn num_return_values(ty: wasmparser::TypeOrFuncType) -> WasmResult {
-    match ty {
-        wasmparser::TypeOrFuncType::Type(ty) => match ty {
-            wasmparser::Type::EmptyBlockType => Ok(0),
-            wasmparser::Type::I32
-            | wasmparser::Type::F32
-            | wasmparser::Type::I64
-            | wasmparser::Type::F64
-            | wasmparser::Type::V128 => Ok(1),
-            ty => Err(wasm_unsupported!("unsupported return value type {:?}", ty)),
-        },
-        wasmparser::TypeOrFuncType::FuncType(_) => {
-            Err(wasm_unsupported!("multi-value block signature {:?}", ty))
-        }
-    }
-}
-
 /// Special VMContext value label. It is tracked as 0xffff_fffe label.
 pub fn get_vmctx_value_label() -> ir::ValueLabel {
     const VMCTX_LABEL: u32 = 0xffff_fffe;
diff --git a/cranelift/wasm/tests/wasm_testsuite.rs b/cranelift/wasm/tests/wasm_testsuite.rs
index 69db90ca6c..8c0d8e510b 100644
--- a/cranelift/wasm/tests/wasm_testsuite.rs
+++ b/cranelift/wasm/tests/wasm_testsuite.rs
@@ -31,6 +31,7 @@ fn testsuite() {
     let flags = Flags::new(settings::builder());
     for path in paths {
         let path = path.path();
+        println!("=== {} ===", path.display());
         let data = read_module(&path);
         handle_module(data, &flags, ReturnMode::NormalReturns);
     }
diff --git a/cranelift/wasmtests/fac-multi-value.wat b/cranelift/wasmtests/fac-multi-value.wat
new file mode 100644
index 0000000000..7a4d5c0fab
--- /dev/null
+++ b/cranelift/wasmtests/fac-multi-value.wat
@@ -0,0 +1,19 @@
+(module
+  ;; Iterative factorial without locals.
+  (func $pick0 (param i64) (result i64 i64)
+    (get_local 0) (get_local 0)
+  )
+  (func $pick1 (param i64 i64) (result i64 i64 i64)
+    (get_local 0) (get_local 1) (get_local 0)
+  )
+  (func (export "fac-ssa") (param i64) (result i64)
+    (i64.const 1) (get_local 0)
+    (loop $l (param i64 i64) (result i64)
+      (call $pick1) (call $pick1) (i64.mul)
+      (call $pick1) (i64.const 1) (i64.sub)
+      (call $pick0) (i64.const 0) (i64.gt_u)
+      (br_if $l)
+      (drop) (return)
+    )
+  )
+)
diff --git a/cranelift/wasmtests/multi-0.wat b/cranelift/wasmtests/multi-0.wat
new file mode 100644
index 0000000000..d1cc24c596
--- /dev/null
+++ b/cranelift/wasmtests/multi-0.wat
@@ -0,0 +1,3 @@
+(module
+  (func (export "i64.dup") (param i64) (result i64 i64)
+    (get_local 0) (get_local 0)))
diff --git a/cranelift/wasmtests/multi-1.wat b/cranelift/wasmtests/multi-1.wat
new file mode 100644
index 0000000000..a814647419
--- /dev/null
+++ b/cranelift/wasmtests/multi-1.wat
@@ -0,0 +1,6 @@
+(module
+  (func (export "multiBlock") (param i64 i32) (result i32 i64 f64)
+    (local.get 1)
+    (local.get 0)
+    (block (param i32 i64) (result i32 i64 f64)
+      (f64.const 1234.5))))
diff --git a/cranelift/wasmtests/multi-10.wat b/cranelift/wasmtests/multi-10.wat
new file mode 100644
index 0000000000..01fbf42941
--- /dev/null
+++ b/cranelift/wasmtests/multi-10.wat
@@ -0,0 +1,10 @@
+(module
+  (func (export "f") (param i64 i32) (result i64 i64)
+    (local.get 0)
+    (local.get 1)
+    ;; If with else. Fewer params than results.
+    (if (param i64) (result i64 i64)
+      (then
+        (i64.const -1))
+      (else
+        (i64.const -2)))))
diff --git a/cranelift/wasmtests/multi-11.wat b/cranelift/wasmtests/multi-11.wat
new file mode 100644
index 0000000000..1ae75889bc
--- /dev/null
+++ b/cranelift/wasmtests/multi-11.wat
@@ -0,0 +1,7 @@
+(module
+  (func (export "multiLoop") (param i64) (result i64 i64)
+    (local.get 0)
+    ;; Fewer params than results.
+    (loop (param i64) (result i64 i64)
+      i64.const 42
+      return)))
diff --git a/cranelift/wasmtests/multi-12.wat b/cranelift/wasmtests/multi-12.wat
new file mode 100644
index 0000000000..9a3e4f7fb5
--- /dev/null
+++ b/cranelift/wasmtests/multi-12.wat
@@ -0,0 +1,9 @@
+(module
+  (func (export "multiLoop") (param i64 i64 i64) (result i64 i64)
+    (local.get 2)
+    (local.get 1)
+    (local.get 0)
+    ;; More params than results.
+    (loop (param i64 i64 i64) (result i64 i64)
+      drop
+      return)))
diff --git a/cranelift/wasmtests/multi-13.wat b/cranelift/wasmtests/multi-13.wat
new file mode 100644
index 0000000000..4f4846300e
--- /dev/null
+++ b/cranelift/wasmtests/multi-13.wat
@@ -0,0 +1,10 @@
+(module
+  (func (export "as-if-then") (param i32 i32) (result i32)
+    (block (result i32)
+      (if (result i32) (local.get 0)
+        (then (br 1 (i32.const 3)))
+        (else (local.get 1))
+      )
+    )
+  )
+)
diff --git a/cranelift/wasmtests/multi-14.wat b/cranelift/wasmtests/multi-14.wat
new file mode 100644
index 0000000000..26d0cb596a
--- /dev/null
+++ b/cranelift/wasmtests/multi-14.wat
@@ -0,0 +1,10 @@
+(module
+  (func (export "as-if-else") (param i32 i32) (result i32)
+    (block (result i32)
+      (if (result i32) (local.get 0)
+        (then (local.get 1))
+        (else (br 1 (i32.const 4)))
+      )
+    )
+  )
+)
diff --git a/cranelift/wasmtests/multi-15.wat b/cranelift/wasmtests/multi-15.wat
new file mode 100644
index 0000000000..2f017bd6aa
--- /dev/null
+++ b/cranelift/wasmtests/multi-15.wat
@@ -0,0 +1,22 @@
+(module
+  (func (export "large-sig")
+    (param i32 i64 f32 f32 i32 f64 f32 i32 i32 i32 f32 f64 f64 f64 i32 i32 f32)
+    (result f64 f32 i32 i32 i32 i64 f32 i32 i32 f32 f64 f64 i32 f32 i32 f64)
+    (local.get 5)
+    (local.get 2)
+    (local.get 0)
+    (local.get 8)
+    (local.get 7)
+    (local.get 1)
+    (local.get 3)
+    (local.get 9)
+    (local.get 4)
+    (local.get 6)
+    (local.get 13)
+    (local.get 11)
+    (local.get 15)
+    (local.get 16)
+    (local.get 14)
+    (local.get 12)
+  )
+)
diff --git a/cranelift/wasmtests/multi-2.wat b/cranelift/wasmtests/multi-2.wat
new file mode 100644
index 0000000000..6f2a7378b1
--- /dev/null
+++ b/cranelift/wasmtests/multi-2.wat
@@ -0,0 +1,6 @@
+(module
+  (func (export "multiLoop") (param i64 i64) (result i64 i64)
+    (local.get 1)
+    (local.get 0)
+    (loop (param i64 i64) (result i64 i64)
+       return)))
diff --git a/cranelift/wasmtests/multi-3.wat b/cranelift/wasmtests/multi-3.wat
new file mode 100644
index 0000000000..d58071f9c7
--- /dev/null
+++ b/cranelift/wasmtests/multi-3.wat
@@ -0,0 +1,13 @@
+(module
+  (func (export "multiIf") (param i32 i64 i64) (result i64 i64)
+    (local.get 2)
+    (local.get 1)
+    (local.get 0)
+    (if (param i64 i64) (result i64 i64)
+      (then return)
+      ;; Hits the code path for an `else` after a block that ends unreachable.
+      (else
+        (drop)
+        (drop)
+        (i64.const 0)
+        (i64.const 0)))))
diff --git a/cranelift/wasmtests/multi-4.wat b/cranelift/wasmtests/multi-4.wat
new file mode 100644
index 0000000000..9c028531d3
--- /dev/null
+++ b/cranelift/wasmtests/multi-4.wat
@@ -0,0 +1,13 @@
+(module
+  (func (export "multiIf2") (param i32 i64 i64) (result i64 i64)
+    (local.get 2)
+    (local.get 1)
+    (local.get 0)
+    (if (param i64 i64) (result i64 i64)
+      (then
+        i64.add
+        i64.const 1)
+      ;; Hits the code path for an `else` after a block that does not end unreachable.
+      (else
+        i64.sub
+        i64.const 2))))
diff --git a/cranelift/wasmtests/multi-5.wat b/cranelift/wasmtests/multi-5.wat
new file mode 100644
index 0000000000..92944770f9
--- /dev/null
+++ b/cranelift/wasmtests/multi-5.wat
@@ -0,0 +1,11 @@
+(module
+  (func (export "foo")
+    i32.const 1
+    i64.const 2
+    ;; More params than results.
+    (block (param i32 i64) (result i32)
+      drop
+    )
+    drop
+  )
+)
diff --git a/cranelift/wasmtests/multi-6.wat b/cranelift/wasmtests/multi-6.wat
new file mode 100644
index 0000000000..c1135a1187
--- /dev/null
+++ b/cranelift/wasmtests/multi-6.wat
@@ -0,0 +1,11 @@
+(module
+  (func (export "foo")
+    i32.const 1
+    ;; Fewer params than results.
+    (block (param i32) (result i32 i64)
+      i64.const 2
+    )
+    drop
+    drop
+  )
+)
diff --git a/cranelift/wasmtests/multi-7.wat b/cranelift/wasmtests/multi-7.wat
new file mode 100644
index 0000000000..c4545ba26c
--- /dev/null
+++ b/cranelift/wasmtests/multi-7.wat
@@ -0,0 +1,9 @@
+(module
+  (func (export "f") (param i64 i32) (result i64)
+    (local.get 0)
+    (local.get 1)
+    ;; If with no else. Same number of params and results.
+    (if (param i64) (result i64)
+      (then
+        (drop)
+        (i64.const -1)))))
diff --git a/cranelift/wasmtests/multi-8.wat b/cranelift/wasmtests/multi-8.wat
new file mode 100644
index 0000000000..1bc23f9f5d
--- /dev/null
+++ b/cranelift/wasmtests/multi-8.wat
@@ -0,0 +1,12 @@
+(module
+  (func (export "f") (param i64 i32) (result i64)
+    (local.get 0)
+    (local.get 1)
+    ;; If with else. Same number of params and results.
+    (if (param i64) (result i64)
+      (then
+        (drop)
+        (i64.const -1))
+      (else
+        (drop)
+        (i64.const -2)))))
diff --git a/cranelift/wasmtests/multi-9.wat b/cranelift/wasmtests/multi-9.wat
new file mode 100644
index 0000000000..d0cecf71b2
--- /dev/null
+++ b/cranelift/wasmtests/multi-9.wat
@@ -0,0 +1,15 @@
+(module
+  (func (export "f") (param i64 i32) (result i64)
+    (local.get 0)
+    (local.get 1)
+    (local.get 1)
+    ;; If with else. More params than results.
+    (if (param i64 i32) (result i64)
+      (then
+        (drop)
+        (drop)
+        (i64.const -1))
+      (else
+        (drop)
+        (drop)
+        (i64.const -2)))))

From b3cf7f911ba901b7a5b2268e31fb6af8725feb5b Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Wed, 2 Oct 2019 15:41:43 -0700
Subject: [PATCH 2802/3084] clif-util: Make the `-t` flag work with the `wasm`
 subcommand (#1105)

* clif-util: Make the `-t` flag work with the `wasm` subcommand

The presence of the flag was checked in the code, so it was essentially already
supported, but it was not properly configured when constructing
the CLI arguments parser.

* clif-util: also enable the `-c` flag for the `wasm` subcommand

Similar to the parent commit, this functionality is already supported, just a
mix up of the CLI parser construction made it not show up.
---
 cranelift/src/clif-util.rs | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 676e9d334f..0c6110f024 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -119,6 +119,18 @@ fn add_enable_multi_value<'a>() -> clap::Arg<'a, 'a> {
         .help("Enable WASM's multi-value support")
 }
 
+fn add_just_decode_flag<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("just-decode")
+        .short("t")
+        .help("Just decode into Cranelift IR")
+}
+
+fn add_check_translation_flag<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("check-translation")
+        .short("c")
+        .help("Just checks the correctness of Cranelift IR translated from WebAssembly")
+}
+
 /// Returns a vector of clap value options and changes these options into a vector of strings
 fn get_vec(argument_vec: Option) -> Vec {
     let mut ret_vec: Vec = Vec::new();
@@ -151,6 +163,8 @@ fn add_wasm_or_compile<'a>(cmd: &str) -> clap::App<'a, 'a> {
         .arg(add_debug_flag())
         .arg(add_enable_simd_flag())
         .arg(add_enable_multi_value())
+        .arg(add_just_decode_flag())
+        .arg(add_check_translation_flag())
 }
 
 fn handle_debug_flag(debug: bool) {
@@ -191,17 +205,7 @@ fn main() {
                 .arg(add_input_file_arg())
                 .arg(add_debug_flag()),
         )
-        .subcommand(
-            add_wasm_or_compile("compile")
-                .arg(
-                    Arg::with_name("just-decode")
-                        .short("t")
-                        .help("Just decode WebAssembly to Cranelift IR"),
-                )
-                .arg(Arg::with_name("check-translation").short("c").help(
-                    "Just checks the correctness of Cranelift IR translated from WebAssembly",
-                )),
-        )
+        .subcommand(add_wasm_or_compile("compile"))
         .subcommand(
             add_wasm_or_compile("wasm").arg(
                 Arg::with_name("value-ranges")

From 913d26841a798d82b00d0a9b58b99e5c997f43e1 Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Wed, 2 Oct 2019 16:28:46 -0700
Subject: [PATCH 2803/3084] cranelift-wasm: Jump to the destination block at
 end of consequent

This commit fixes a bug where at the end of an `if..else..end`'s consequent
block, we would sometimes erroneously jump to the `else` block instead of to the
following destination block. Not good!
---
 cranelift/wasm/src/code_translator.rs | 3 ++-
 cranelift/wasmtests/multi-16.wat      | 9 +++++++++
 2 files changed, 11 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/wasmtests/multi-16.wat

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 4416410664..4b6eb8ed32 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -203,6 +203,7 @@ pub fn translate_operator(
                     else_data: ElseData::NoElse { branch_inst },
                     ref mut reachable_from_top,
                     blocktype,
+                    destination,
                     ..
                 } => {
                     // We take the control frame pushed by the if, use its ebb
@@ -215,7 +216,7 @@ pub fn translate_operator(
 
                     let (params, _results) = blocktype_params_results(wasm_types, blocktype)?;
                     let else_ebb = ebb_with_params(builder, params)?;
-                    builder.ins().jump(else_ebb, state.peekn(params.len()));
+                    builder.ins().jump(destination, state.peekn(params.len()));
                     state.popn(params.len());
 
                     // You might be expecting that we push the parameters for this
diff --git a/cranelift/wasmtests/multi-16.wat b/cranelift/wasmtests/multi-16.wat
new file mode 100644
index 0000000000..1e60aa8cc8
--- /dev/null
+++ b/cranelift/wasmtests/multi-16.wat
@@ -0,0 +1,9 @@
+(module
+  (func (export "param") (param i32) (result i32)
+    (i32.const 1)
+    (if (param i32) (result i32) (local.get 0)
+      (then (i32.const 2) (i32.add))
+      (else (i32.const -2) (i32.add))
+    )
+  )
+)

From a6af10725709d1de90680681987936145ec825da Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Thu, 3 Oct 2019 12:41:53 -0700
Subject: [PATCH 2804/3084] deps: bump wasmparser dependency to 0.39.2 (#1112)

This has a bug fix for Wasm multi-value blocks that is necessary to getting the
spec tests passing.
---
 cranelift/wasm/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index c1e4bdd80b..81266bd227 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -11,7 +11,7 @@ keywords = ["webassembly", "wasm"]
 edition = "2018"
 
 [dependencies]
-wasmparser = { version = "0.39.1", default-features = false }
+wasmparser = { version = "0.39.2", default-features = false }
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.44.0" }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.44.0", default-features = false }

From 13676cafd1dbee3bff3cd0879f799e8666969572 Mon Sep 17 00:00:00 2001
From: julian-seward1 
Date: Tue, 8 Oct 2019 18:13:35 +0200
Subject: [PATCH 2805/3084] RedundantReloadRemover::run: use cached state
 rather than allocating it new for each function. (#1118)

RedundantReloadRemover participates in the state-recycling machinery
implemented in cranelift-codegen/src/context.rs, whose goal it is to cache
per-pass state so it can be used for compilation of multiple functions without
reallocation.  Unfortunately RedundantReloadRemover::run simply ignores the
cached state and reallocates it new for each function.  This patch fixes that.
This reduces the number of malloc'd blocks by about 2%.
---
 cranelift/codegen/src/redundant_reload_remover.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/redundant_reload_remover.rs b/cranelift/codegen/src/redundant_reload_remover.rs
index 11f13f5256..b1928f03b8 100644
--- a/cranelift/codegen/src/redundant_reload_remover.rs
+++ b/cranelift/codegen/src/redundant_reload_remover.rs
@@ -887,7 +887,7 @@ impl RedundantReloadRemover {
             cur: EncCursor::new(func, isa),
             reginfo: isa.register_info(),
             cfg: cfg,
-            state: &mut RedundantReloadRemover::new(),
+            state: self,
         };
         let mut total_regunits = 0;
         for rb in isa.register_info().banks {

From cbbd94db02db187b79712e2b2128f7ffae17547e Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 8 Oct 2019 16:09:26 +0200
Subject: [PATCH 2806/3084] Allow wrap-around when subtracting type size to
 immediate in try_fold_extended_move;

---
 cranelift/codegen/src/simple_preopt.rs             |  2 +-
 .../fold-extended-move-wraparound.clif             | 14 ++++++++++++++
 2 files changed, 15 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/filetests/filetests/simple_preopt/fold-extended-move-wraparound.clif

diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index 221e8a57b0..73e22451c9 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -508,7 +508,7 @@ fn try_fold_extended_move(
             }
 
             let imm_bits: i64 = imm.into();
-            let ireduce_ty = match dest_ty.lane_bits() as i64 - imm_bits {
+            let ireduce_ty = match (dest_ty.lane_bits() as i64).wrapping_sub(imm_bits) {
                 8 => I8,
                 16 => I16,
                 32 => I32,
diff --git a/cranelift/filetests/filetests/simple_preopt/fold-extended-move-wraparound.clif b/cranelift/filetests/filetests/simple_preopt/fold-extended-move-wraparound.clif
new file mode 100644
index 0000000000..074507a786
--- /dev/null
+++ b/cranelift/filetests/filetests/simple_preopt/fold-extended-move-wraparound.clif
@@ -0,0 +1,14 @@
+test simple_preopt
+target x86_64
+
+function %wraparound(i64 vmctx) -> f32 system_v {
+    gv0 = vmctx
+    gv1 = iadd_imm.i64 gv0, 48
+
+ebb35(v0: i64):
+    v88 = iconst.i64 0
+    v89 = iconst.i64 0x8000_0000_0000_0000
+    v90 = ishl_imm v88, 0x8000_0000_0000_0000
+    v91 = sshr v90, v89; check: sshr_imm v90, 0x8000_0000_0000_0000
+    trap user0
+}

From ece9450a2fe79966c4729a098eb28e90b2f05af2 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 9 Oct 2019 06:20:30 -0700
Subject: [PATCH 2807/3084] Bump version to 0.45.0

---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  2 +-
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 19 files changed, 68 insertions(+), 68 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index ef5d513f09..9229936326 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.44.0"
+version = "0.45.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.44.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.44.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.44.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.44.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.44.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.44.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.44.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.44.0" }
-cranelift-module = { path = "cranelift-module", version = "0.44.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.44.0" }
-cranelift-object = { path = "cranelift-object", version = "0.44.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.44.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.44.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.44.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.45.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.45.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.45.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.45.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.45.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.45.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.45.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.45.0" }
+cranelift-module = { path = "cranelift-module", version = "0.45.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.45.0" }
+cranelift-object = { path = "cranelift-object", version = "0.45.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.45.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.45.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.45.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 4dc95212e9..a8aa5387ad 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.44.0"
+version = "0.45.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.44.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.45.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 7281ad3092..9759c93491 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.44.0"
+version = "0.45.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.44.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.44.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.44.0" }
+cranelift-codegen-shared = { path = "./shared", version = "0.45.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.45.0" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.45.0" }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashbrown = { version = "0.6", optional = true }
@@ -29,7 +29,7 @@ smallvec = { version = "0.6.10" }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.44.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.45.0" }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 63624d3f1e..ae4bd31aff 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.44.0"
+version = "0.45.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.44.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.44.0" }
+cranelift-codegen-shared = { path = "../shared", version = "0.45.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.45.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index d58b92fd50..079515a49f 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.44.0"
+version = "0.45.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index e6f1ec6310..e80350d003 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.44.0"
+version = "0.45.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 26d3e5defe..7b1177126e 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.44.0"
+version = "0.45.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.44.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.45.0" }
 faerie = "0.11.0"
 goblin = "0.0.24"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index acbeeebb5e..c737bdf260 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.44.0"
+version = "0.45.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.44.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.44.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.44.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.45.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.45.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.45.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 0f8aefe91d..6c437680c3 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.44.0"
+version = "0.45.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", default-features = false }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index c67957c6f6..570ca26912 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.44.0"
+version = "0.45.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.44.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.45.0" }
 hashbrown = { version = "0.6", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 54f05fa369..9f3af19cbf 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.44.0"
+version = "0.45.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", default-features = false }
 target-lexicon = "0.8.1"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 5cb413e181..acefc54516 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.44.0"
+version = "0.45.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.44.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.45.0" }
 object = { version = "0.14.0", default-features = false, features = ["write"] }
 target-lexicon = "0.8.1"
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 6dc55e1432..5d0d1ebd3e 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.44.0"
+version = "0.45.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.44.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.45.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index d424fe7718..51aaef2a12 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.44.0"
+version="0.45.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 3d97d552a3..3119aaf263 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.44.0"
+version = "0.45.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0" }
 target-lexicon = "0.8.1"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 81b3a2a09f..17b988dca4 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.44.0"
+version = "0.45.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.44.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.45.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index a41d9d9f7b..6ea16fa229 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.44.0"
+version = "0.45.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.44.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.44.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.45.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.45.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.44.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.44.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.44.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.45.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.45.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.45.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index c77cd048d9..6de7e230e2 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.44.0"
+version = "0.45.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.44.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.45.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 81266bd227..5dd19f3998 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.44.0"
+version = "0.45.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.39.2", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.44.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.44.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.44.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.45.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.45.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From c6ed6b7247ef4b9d19d95a54bf15bb4a52d8fe4f Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 9 Oct 2019 06:28:47 -0700
Subject: [PATCH 2808/3084] "std" builds need to enable the "std" feature in
 cranelift-codegen.

---
 cranelift/preopt/Cargo.toml | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 5d0d1ebd3e..3f8c87be37 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -18,6 +18,11 @@ cranelift-entity = { path = "../cranelift-entity", version = "0.45.0" }
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
 
+[features]
+default = ["std"]
+std = ["cranelift-codegen/std"]
+core = ["cranelift-codegen/core"]
+
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }

From 68b671e0eed88085ec10cbc828bd1c98db10840f Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 5 Oct 2019 17:12:51 +0200
Subject: [PATCH 2809/3084] Fix isplit legalization for ebb params when jumping
 forward

Fixes #1106
---
 cranelift/codegen/src/legalizer/mod.rs        | 11 ++++++++--
 .../isa/x86/i128-isplit-forward-jump.clif     | 22 +++++++++++++++++++
 2 files changed, 31 insertions(+), 2 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/i128-isplit-forward-jump.clif

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index f090b14008..b406506f48 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -146,6 +146,15 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
     func.encodings.resize(func.dfg.num_insts());
 
     let mut pos = FuncCursor::new(func);
+    let func_begin = pos.position();
+
+    // Split ebb params before trying to legalize instructions, so that the newly introduced
+    // isplit instructions get legalized.
+    while let Some(ebb) = pos.next_ebb() {
+        split::split_ebb_params(pos.func, cfg, ebb);
+    }
+
+    pos.set_position(func_begin);
 
     // This must be a set to prevent trying to legalize `isplit` and `vsplit` twice in certain cases.
     let mut pending_splits = BTreeSet::new();
@@ -153,8 +162,6 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
     // Process EBBs in layout order. Some legalization actions may split the current EBB or append
     // new ones to the end. We need to make sure we visit those new EBBs too.
     while let Some(ebb) = pos.next_ebb() {
-        split::split_ebb_params(pos.func, cfg, ebb);
-
         // Keep track of the cursor position before the instruction being processed, so we can
         // double back when replacing instructions.
         let mut prev_pos = pos.position();
diff --git a/cranelift/filetests/filetests/isa/x86/i128-isplit-forward-jump.clif b/cranelift/filetests/filetests/isa/x86/i128-isplit-forward-jump.clif
new file mode 100644
index 0000000000..863f39612f
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/i128-isplit-forward-jump.clif
@@ -0,0 +1,22 @@
+test compile
+target x86_64
+
+function u0:0() -> i128 system_v {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = iconst.i64 0
+    v2 = iconcat v0, v1
+    jump ebb5
+
+ebb2:
+    jump ebb4(v27)
+
+ebb4(v23: i128):
+    return v23
+
+ebb5:
+    v27 = bxor.i128 v2, v2
+    v32 = iconst.i32 0
+    brz v32, ebb2
+    trap user0
+}

From c8128539d07224cf46102db8945298d9ff17301c Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 9 Oct 2019 13:59:12 +0200
Subject: [PATCH 2810/3084] cleanup: remove spurious macro_use in
 cranelift-bforest;

---
 cranelift/bforest/src/lib.rs | 1 -
 1 file changed, 1 deletion(-)

diff --git a/cranelift/bforest/src/lib.rs b/cranelift/bforest/src/lib.rs
index 2f5c1060a0..26e5b8efcf 100644
--- a/cranelift/bforest/src/lib.rs
+++ b/cranelift/bforest/src/lib.rs
@@ -33,7 +33,6 @@
 #![no_std]
 
 #[cfg(test)]
-#[macro_use]
 extern crate alloc;
 
 #[macro_use]

From 19444649e7f5c230a22a1a3142003ab91a8b7d86 Mon Sep 17 00:00:00 2001
From: Ujjwal Sharma 
Date: Tue, 1 Oct 2019 14:26:35 +0530
Subject: [PATCH 2811/3084] [codegen] add to_static_str method to IntCC

Add a method to_static_str to objects of type IntCC which consumes the
object to basically do the opposite of IntCC::new.

Refs: https://github.com/CraneStation/cranelift/pull/1081#discussion_r329042331
---
 cranelift/codegen/shared/src/condcodes.rs | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/shared/src/condcodes.rs b/cranelift/codegen/shared/src/condcodes.rs
index 1171eef2f6..d151c49466 100644
--- a/cranelift/codegen/shared/src/condcodes.rs
+++ b/cranelift/codegen/shared/src/condcodes.rs
@@ -95,10 +95,11 @@ impl CondCode for IntCC {
     }
 }
 
-impl Display for IntCC {
-    fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+impl IntCC {
+    /// Get the corresponding string condition code for the IntCC object.
+    pub fn to_static_str(self) -> &'static str {
         use self::IntCC::*;
-        f.write_str(match *self {
+        match self {
             Equal => "eq",
             NotEqual => "ne",
             SignedGreaterThan => "sgt",
@@ -111,7 +112,13 @@ impl Display for IntCC {
             UnsignedLessThanOrEqual => "ule",
             Overflow => "of",
             NotOverflow => "nof",
-        })
+        }
+    }
+}
+
+impl Display for IntCC {
+    fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+        f.write_str(self.to_static_str())
     }
 }
 

From c062f12d7ccc86ee30cb3ca47b425d9f44911da1 Mon Sep 17 00:00:00 2001
From: Ujjwal Sharma 
Date: Thu, 26 Sep 2019 21:59:56 +0530
Subject: [PATCH 2812/3084] [codegen] legalize icmp for 64 and 128 bit operands

Add legalizations for icmp and icmp_imm for i64 and i128 operands for
the narrow legalization set, allowing 32-bit ISAs (like x86-32) to
compare 64-bit integers and all ISAs to compare 128-bit integers.

Fixes: https://github.com/bnjbvr/cranelift-x86/issues/2
---
 cranelift/codegen/meta/src/shared/legalize.rs |  64 ++++
 cranelift/codegen/shared/src/condcodes.rs     |  26 ++
 cranelift/codegen/src/legalizer/mod.rs        |  64 ++++
 .../filetests/isa/x86/icmp-i128.clif          |  52 +++
 .../filetests/isa/x86/legalize-i64.clif       | 298 ++++++++++++++++++
 5 files changed, 504 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/icmp-i128.clif

diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 7ac4e8af13..bf046f5ddd 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -5,6 +5,7 @@ use crate::cdsl::xform::{TransformGroupBuilder, TransformGroups};
 use crate::shared::immediates::Immediates;
 use crate::shared::types::Float::{F32, F64};
 use crate::shared::types::Int::{I128, I16, I32, I64, I8};
+use cranelift_codegen_shared::condcodes::{CondCode, IntCC};
 
 pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGroups {
     let mut narrow = TransformGroupBuilder::new(
@@ -272,6 +273,69 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
         ],
     );
 
+    // TODO(ryzokuken): benchmark this and decide if branching is a faster
+    // approach than evaluating boolean expressions.
+
+    narrow.custom_legalize(icmp_imm, "narrow_icmp_imm");
+
+    let intcc_eq = Literal::enumerator_for(&imm.intcc, "eq");
+    let intcc_ne = Literal::enumerator_for(&imm.intcc, "ne");
+    for &(int_ty, int_ty_half) in &[(I64, I32), (I128, I64)] {
+        narrow.legalize(
+            def!(b = icmp.int_ty(intcc_eq, x, y)),
+            vec![
+                def!((xl, xh) = isplit(x)),
+                def!((yl, yh) = isplit(y)),
+                def!(b1 = icmp.int_ty_half(intcc_eq, xl, yl)),
+                def!(b2 = icmp.int_ty_half(intcc_eq, xh, yh)),
+                def!(b = band(b1, b2)),
+            ],
+        );
+
+        narrow.legalize(
+            def!(b = icmp.int_ty(intcc_ne, x, y)),
+            vec![
+                def!((xl, xh) = isplit(x)),
+                def!((yl, yh) = isplit(y)),
+                def!(b1 = icmp.int_ty_half(intcc_ne, xl, yl)),
+                def!(b2 = icmp.int_ty_half(intcc_ne, xh, yh)),
+                def!(b = bor(b1, b2)),
+            ],
+        );
+
+        use IntCC::*;
+        for cc in &[
+            SignedGreaterThan,
+            SignedGreaterThanOrEqual,
+            SignedLessThan,
+            SignedLessThanOrEqual,
+            UnsignedGreaterThan,
+            UnsignedGreaterThanOrEqual,
+            UnsignedLessThan,
+            UnsignedLessThanOrEqual,
+        ] {
+            let intcc_cc = Literal::enumerator_for(&imm.intcc, cc.to_static_str());
+            let cc1 = Literal::enumerator_for(&imm.intcc, cc.without_equal().to_static_str());
+            let cc2 =
+                Literal::enumerator_for(&imm.intcc, cc.inverse().without_equal().to_static_str());
+            let cc3 = Literal::enumerator_for(&imm.intcc, cc.unsigned().to_static_str());
+            narrow.legalize(
+                def!(b = icmp.int_ty(intcc_cc, x, y)),
+                vec![
+                    def!((xl, xh) = isplit(x)),
+                    def!((yl, yh) = isplit(y)),
+                    // X = cc1 || (!cc2 && cc3)
+                    def!(b1 = icmp.int_ty_half(cc1, xh, yh)),
+                    def!(b2 = icmp.int_ty_half(cc2, xh, yh)),
+                    def!(b3 = icmp.int_ty_half(cc3, xl, yl)),
+                    def!(c1 = bnot(b2)),
+                    def!(c2 = band(c1, b3)),
+                    def!(b = bor(b1, c2)),
+                ],
+            );
+        }
+    }
+
     // Widen instructions with one input operand.
     for &op in &[bnot, popcnt] {
         for &int_ty in &[I8, I16] {
diff --git a/cranelift/codegen/shared/src/condcodes.rs b/cranelift/codegen/shared/src/condcodes.rs
index d151c49466..03ae865ce4 100644
--- a/cranelift/codegen/shared/src/condcodes.rs
+++ b/cranelift/codegen/shared/src/condcodes.rs
@@ -96,6 +96,32 @@ impl CondCode for IntCC {
 }
 
 impl IntCC {
+    /// Get the corresponding IntCC with the equal component removed.
+    /// For conditions without a zero component, this is a no-op.
+    pub fn without_equal(self) -> Self {
+        use self::IntCC::*;
+        match self {
+            SignedGreaterThan | SignedGreaterThanOrEqual => SignedGreaterThan,
+            SignedLessThan | SignedLessThanOrEqual => SignedLessThan,
+            UnsignedGreaterThan | UnsignedGreaterThanOrEqual => UnsignedGreaterThan,
+            UnsignedLessThan | UnsignedLessThanOrEqual => UnsignedLessThan,
+            _ => self,
+        }
+    }
+
+    /// Get the corresponding IntCC with the signed component removed.
+    /// For conditions without a signed component, this is a no-op.
+    pub fn unsigned(self) -> Self {
+        use self::IntCC::*;
+        match self {
+            SignedGreaterThan | UnsignedGreaterThan => UnsignedGreaterThan,
+            SignedGreaterThanOrEqual | UnsignedGreaterThanOrEqual => UnsignedGreaterThanOrEqual,
+            SignedLessThan | UnsignedLessThan => UnsignedLessThan,
+            SignedLessThanOrEqual | UnsignedLessThanOrEqual => UnsignedLessThanOrEqual,
+            _ => self,
+        }
+    }
+
     /// Get the corresponding string condition code for the IntCC object.
     pub fn to_static_str(self) -> &'static str {
         use self::IntCC::*;
diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index b406506f48..b0fdac9ea9 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -672,3 +672,67 @@ fn narrow_iconst(
 
     unimplemented!("missing encoding or legalization for iconst.{:?}", ty);
 }
+
+fn narrow_icmp_imm(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    _cfg: &mut ControlFlowGraph,
+    _isa: &dyn TargetIsa,
+) {
+    use crate::ir::condcodes::{CondCode, IntCC};
+
+    let (arg, cond, imm): (ir::Value, IntCC, i64) = match func.dfg[inst] {
+        ir::InstructionData::IntCompareImm {
+            opcode: ir::Opcode::IcmpImm,
+            arg,
+            cond,
+            imm,
+        } => (arg, cond, imm.into()),
+        _ => panic!("unexpected instruction in narrow_icmp_imm"),
+    };
+
+    let mut pos = FuncCursor::new(func).at_inst(inst);
+    pos.use_srcloc(inst);
+
+    let ty = pos.func.dfg.ctrl_typevar(inst);
+    let ty_half = ty.half_width().unwrap();
+
+    let imm_low = pos
+        .ins()
+        .iconst(ty_half, imm & (1u128 << ty_half.bits() - 1) as i64);
+    let imm_high = pos
+        .ins()
+        .iconst(ty_half, imm.wrapping_shr(ty_half.bits().into()));
+    let (arg_low, arg_high) = pos.ins().isplit(arg);
+
+    match cond {
+        IntCC::Equal => {
+            let res_low = pos.ins().icmp(cond, arg_low, imm_low);
+            let res_high = pos.ins().icmp(cond, arg_high, imm_high);
+            pos.func.dfg.replace(inst).band(res_low, res_high);
+        }
+        IntCC::NotEqual => {
+            let res_low = pos.ins().icmp(cond, arg_low, imm_low);
+            let res_high = pos.ins().icmp(cond, arg_high, imm_high);
+            pos.func.dfg.replace(inst).bor(res_low, res_high);
+        }
+        IntCC::SignedGreaterThan
+        | IntCC::SignedGreaterThanOrEqual
+        | IntCC::SignedLessThan
+        | IntCC::SignedLessThanOrEqual
+        | IntCC::UnsignedGreaterThan
+        | IntCC::UnsignedGreaterThanOrEqual
+        | IntCC::UnsignedLessThan
+        | IntCC::UnsignedLessThanOrEqual => {
+            let b1 = pos.ins().icmp(cond.without_equal(), arg_high, imm_high);
+            let b2 = pos
+                .ins()
+                .icmp(cond.inverse().without_equal(), arg_high, imm_high);
+            let b3 = pos.ins().icmp(cond.unsigned(), arg_low, imm_low);
+            let c1 = pos.ins().bnot(b2);
+            let c2 = pos.ins().band(c1, b3);
+            pos.func.dfg.replace(inst).bor(b1, c2);
+        }
+        _ => unimplemented!("missing legalization for condition {:?}", cond),
+    }
+}
diff --git a/cranelift/filetests/filetests/isa/x86/icmp-i128.clif b/cranelift/filetests/filetests/isa/x86/icmp-i128.clif
new file mode 100644
index 0000000000..8816d22d69
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/icmp-i128.clif
@@ -0,0 +1,52 @@
+test run
+target x86_64 haswell
+
+function %test_icmp_eq_i128() -> b1 {
+ebb0:
+    v11 = iconst.i64 0x0
+    v12 = iconst.i64 0x0
+    v1 = iconcat v11, v12
+    v21 = iconst.i64 0x0
+    v22 = iconst.i64 0x0
+    v2 = iconcat v21, v22
+    v10 = icmp.i128 eq v1, v2
+    return v10
+}
+
+; run
+
+function %test_icmp_imm_eq_i128() -> b1 {
+ebb0:
+    v11 = iconst.i64 0x0
+    v12 = iconst.i64 0x0
+    v1 = iconcat v11, v12
+    v10 = icmp_imm.i128 eq v1, 0x0
+    return v10
+}
+
+; run
+
+function %test_icmp_ne_i128() -> b1 {
+ebb0:
+    v11 = iconst.i64 0x0
+    v12 = iconst.i64 0x0
+    v1 = iconcat v11, v12
+    v21 = iconst.i64 0x0
+    v22 = iconst.i64 0x1
+    v2 = iconcat v21, v22
+    v10 = icmp.i128 ne v1, v2
+    return v10
+}
+
+; run
+
+function %test_icmp_imm_ne_i128() -> b1 {
+ebb0:
+    v11 = iconst.i64 0x0
+    v12 = iconst.i64 0x0
+    v1 = iconcat v11, v12
+    v10 = icmp_imm.i128 ne v1, 0x1
+    return v10
+}
+
+; run
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-i64.clif b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
index 8ae88f736e..2723bd69b8 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
@@ -25,3 +25,301 @@ ebb0(v1: i64, v2: i64):
     ; nextln: v10 = iconcat $v10_lsb, $v10_msb
     return v10
 }
+
+function %icmp_eq(i64, i64) -> b1 {
+ebb0(v1: i64, v2: i64):
+    v10 = icmp eq v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(v10_lsb=$V) = icmp eq $v1_lsb, $v2_lsb
+    ; nextln: $(v10_msb=$V) = icmp eq $v1_msb, $v2_msb
+    ; nextln: v10 = band $v10_lsb, $v10_msb
+    return v10
+}
+
+function %icmp_imm_eq(i64) -> b1 {
+ebb0(v1: i64):
+    v10 = icmp_imm eq v1, 0
+    ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
+    ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
+    ; nextln: v1 = iconcat $(v1_lsb_a=$V), $(v1_msb_a=$V)
+    ; nextln: $(v2_lsb=$V) = iconst.i32 0
+    ; nextln: $(v2_msb=$V) = iconst.i32 0
+    ; nextln: $(v10_lsb=$V) = icmp eq $v1_lsb, $v2_lsb
+    ; nextln: $(v10_msb=$V) = icmp eq $v1_msb, $v2_msb
+    ; nextln: v10 = band $v10_lsb, $v10_msb
+    return v10
+}
+
+function %icmp_ne(i64, i64) -> b1 {
+ebb0(v1: i64, v2: i64):
+    v10 = icmp ne v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(v10_lsb=$V) = icmp ne $v1_lsb, $v2_lsb
+    ; nextln: $(v10_msb=$V) = icmp ne $v1_msb, $v2_msb
+    ; nextln: v10 = bor $v10_lsb, $v10_msb
+    return v10
+}
+
+function %icmp_imm_ne(i64) -> b1 {
+ebb0(v1: i64):
+    v10 = icmp_imm ne v1, 0
+    ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
+    ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
+    ; nextln: v1 = iconcat $(v1_lsb_a=$V), $(v1_msb_a=$V)
+    ; nextln: $(v2_lsb=$V) = iconst.i32 0
+    ; nextln: $(v2_msb=$V) = iconst.i32 0
+    ; nextln: $(v10_lsb=$V) = icmp ne $v1_lsb, $v2_lsb
+    ; nextln: $(v10_msb=$V) = icmp ne $v1_msb, $v2_msb
+    ; nextln: v10 = bor $v10_lsb, $v10_msb
+    return v10
+}
+
+function %icmp_sgt(i64, i64) -> b1 {
+ebb0(v1: i64, v2: i64):
+    v10 = icmp sgt v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(b1=$V) = icmp sgt $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp slt $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp ugt $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_imm_sgt(i64) -> b1 {
+ebb0(v1: i64):
+    v10 = icmp_imm sgt v1, 0
+    ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
+    ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
+    ; nextln: v1 = iconcat $(v1_lsb_a=$V), $(v1_msb_a=$V)
+    ; nextln: $(v2_lsb=$V) = iconst.i32 0
+    ; nextln: $(v2_msb=$V) = iconst.i32 0
+    ; nextln: $(b1=$V) = icmp sgt $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp slt $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp ugt $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_sge(i64, i64) -> b1 {
+ebb0(v1: i64, v2: i64):
+    v10 = icmp sge v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(b1=$V) = icmp sgt $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp slt $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp uge $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_imm_sge(i64) -> b1 {
+ebb0(v1: i64):
+    v10 = icmp_imm sge v1, 0
+    ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
+    ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
+    ; nextln: v1 = iconcat $(v1_lsb_a=$V), $(v1_msb_a=$V)
+    ; nextln: $(v2_lsb=$V) = iconst.i32 0
+    ; nextln: $(v2_msb=$V) = iconst.i32 0
+    ; nextln: $(b1=$V) = icmp sgt $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp slt $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp uge $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_slt(i64, i64) -> b1 {
+ebb0(v1: i64, v2: i64):
+    v10 = icmp slt v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(b1=$V) = icmp slt $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp sgt $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp ult $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_imm_slt(i64) -> b1 {
+ebb0(v1: i64):
+    v10 = icmp_imm slt v1, 0
+    ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
+    ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
+    ; nextln: v1 = iconcat $(v1_lsb_a=$V), $(v1_msb_a=$V)
+    ; nextln: $(v2_lsb=$V) = iconst.i32 0
+    ; nextln: $(v2_msb=$V) = iconst.i32 0
+    ; nextln: $(b1=$V) = icmp slt $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp sgt $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp ult $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_sle(i64, i64) -> b1 {
+ebb0(v1: i64, v2: i64):
+    v10 = icmp sle v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(b1=$V) = icmp slt $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp sgt $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp ule $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_imm_sle(i64) -> b1 {
+ebb0(v1: i64):
+    v10 = icmp_imm sle v1, 0
+    ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
+    ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
+    ; nextln: v1 = iconcat $(v1_lsb_a=$V), $(v1_msb_a=$V)
+    ; nextln: $(v2_lsb=$V) = iconst.i32 0
+    ; nextln: $(v2_msb=$V) = iconst.i32 0
+    ; nextln: $(b1=$V) = icmp slt $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp sgt $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp ule $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_ugt(i64, i64) -> b1 {
+ebb0(v1: i64, v2: i64):
+    v10 = icmp ugt v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(b1=$V) = icmp ugt $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp ult $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp ugt $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_imm_ugt(i64) -> b1 {
+ebb0(v1: i64):
+    v10 = icmp_imm ugt v1, 0
+    ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
+    ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
+    ; nextln: v1 = iconcat $(v1_lsb_a=$V), $(v1_msb_a=$V)
+    ; nextln: $(v2_lsb=$V) = iconst.i32 0
+    ; nextln: $(v2_msb=$V) = iconst.i32 0
+    ; nextln: $(b1=$V) = icmp ugt $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp ult $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp ugt $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_uge(i64, i64) -> b1 {
+ebb0(v1: i64, v2: i64):
+    v10 = icmp uge v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(b1=$V) = icmp ugt $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp ult $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp uge $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_imm_uge(i64) -> b1 {
+ebb0(v1: i64):
+    v10 = icmp_imm uge v1, 0
+    ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
+    ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
+    ; nextln: v1 = iconcat $(v1_lsb_a=$V), $(v1_msb_a=$V)
+    ; nextln: $(v2_lsb=$V) = iconst.i32 0
+    ; nextln: $(v2_msb=$V) = iconst.i32 0
+    ; nextln: $(b1=$V) = icmp ugt $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp ult $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp uge $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_ult(i64, i64) -> b1 {
+ebb0(v1: i64, v2: i64):
+    v10 = icmp ult v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(b1=$V) = icmp ult $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp ugt $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp ult $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_imm_ult(i64) -> b1 {
+ebb0(v1: i64):
+    v10 = icmp_imm ult v1, 0
+    ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
+    ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
+    ; nextln: v1 = iconcat $(v1_lsb_a=$V), $(v1_msb_a=$V)
+    ; nextln: $(v2_lsb=$V) = iconst.i32 0
+    ; nextln: $(v2_msb=$V) = iconst.i32 0
+    ; nextln: $(b1=$V) = icmp ult $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp ugt $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp ult $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_ule(i64, i64) -> b1 {
+ebb0(v1: i64, v2: i64):
+    v10 = icmp ule v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(b1=$V) = icmp ult $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp ugt $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp ule $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}
+
+function %icmp_imm_ule(i64) -> b1 {
+ebb0(v1: i64):
+    v10 = icmp_imm ule v1, 0
+    ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
+    ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
+    ; nextln: v1 = iconcat $(v1_lsb_a=$V), $(v1_msb_a=$V)
+    ; nextln: $(v2_lsb=$V) = iconst.i32 0
+    ; nextln: $(v2_msb=$V) = iconst.i32 0
+    ; nextln: $(b1=$V) = icmp ult $v1_msb, $v2_msb
+    ; nextln: $(b2=$V) = icmp ugt $v1_msb, $v2_msb
+    ; nextln: $(b3=$V) = icmp ule $v1_lsb, $v2_lsb
+    ; nextln: $(c1=$V) = bnot $b2
+    ; nextln: $(c2=$V) = band $c1, $b3
+    ; nextln: v10 = bor $b1, $c2
+    return v10
+}

From d404368deac765e1549d295e8a679646e2bd2182 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 9 Oct 2019 16:35:33 +0200
Subject: [PATCH 2813/3084] Share constant_hash code between the meta and
 codegen crates;

---
 cranelift/codegen/meta/src/constant_hash.rs   | 55 -------------
 cranelift/codegen/meta/src/gen_encodings.rs   |  2 +-
 cranelift/codegen/meta/src/gen_inst.rs        |  2 +-
 cranelift/codegen/meta/src/gen_settings.rs    |  6 +-
 cranelift/codegen/meta/src/lib.rs             |  1 -
 cranelift/codegen/shared/src/constant_hash.rs | 79 +++++++++++++++++++
 cranelift/codegen/shared/src/lib.rs           |  1 +
 cranelift/codegen/src/constant_hash.rs        | 32 ++------
 8 files changed, 94 insertions(+), 84 deletions(-)
 delete mode 100644 cranelift/codegen/meta/src/constant_hash.rs
 create mode 100644 cranelift/codegen/shared/src/constant_hash.rs

diff --git a/cranelift/codegen/meta/src/constant_hash.rs b/cranelift/codegen/meta/src/constant_hash.rs
deleted file mode 100644
index ce6214efa8..0000000000
--- a/cranelift/codegen/meta/src/constant_hash.rs
+++ /dev/null
@@ -1,55 +0,0 @@
-use std::iter;
-
-pub(crate) fn simple_hash(s: &str) -> usize {
-    let mut h: u32 = 5381;
-    for c in s.chars() {
-        h = (h ^ c as u32).wrapping_add(h.rotate_right(6));
-    }
-    h as usize
-}
-
-/// Compute an open addressed, quadratically probed hash table containing
-/// `items`. The returned table is a list containing the elements of the
-/// iterable `items` and `None` in unused slots.
-pub(crate) fn generate_table<'cont, T, I: iter::Iterator, H: Fn(&T) -> usize>(
-    items: I,
-    num_items: usize,
-    hash_function: H,
-) -> Vec> {
-    let size = (1.20 * num_items as f64) as usize;
-    // TODO do we really need the multiply by two here?
-    let size = if size.is_power_of_two() {
-        size * 2
-    } else {
-        size.next_power_of_two()
-    };
-
-    let mut table = vec![None; size];
-
-    for i in items {
-        let mut h = hash_function(&i) % size;
-        let mut s = 0;
-        while table[h].is_some() {
-            s += 1;
-            h = (h + s) % size;
-        }
-        table[h] = Some(i);
-    }
-
-    table
-}
-
-#[test]
-fn test_generate_table() {
-    let v = vec!["Hello".to_string(), "world".to_string()];
-    let table = generate_table(v.iter(), v.len(), |s| simple_hash(&s));
-    assert_eq!(
-        table,
-        vec![
-            None,
-            Some(&"Hello".to_string()),
-            Some(&"world".to_string()),
-            None
-        ]
-    );
-}
diff --git a/cranelift/codegen/meta/src/gen_encodings.rs b/cranelift/codegen/meta/src/gen_encodings.rs
index cfaa63f087..a2d1da62a7 100644
--- a/cranelift/codegen/meta/src/gen_encodings.rs
+++ b/cranelift/codegen/meta/src/gen_encodings.rs
@@ -49,6 +49,7 @@ use std::collections::{BTreeMap, HashMap, HashSet};
 use std::convert::TryFrom;
 use std::iter::FromIterator;
 
+use cranelift_codegen_shared::constant_hash::generate_table;
 use cranelift_entity::EntityRef;
 
 use crate::error;
@@ -66,7 +67,6 @@ use crate::cdsl::xform::TransformGroupIndex;
 
 use crate::shared::Definitions as SharedDefinitions;
 
-use crate::constant_hash::generate_table;
 use crate::default_map::MapWithDefault;
 use crate::unique_table::UniqueSeqTable;
 
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index ed28a83794..85a853150e 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -1,5 +1,6 @@
 use std::fmt;
 
+use cranelift_codegen_shared::constant_hash;
 use cranelift_entity::EntityRef;
 
 use crate::cdsl::camel_case;
@@ -10,7 +11,6 @@ use crate::cdsl::typevar::{TypeSet, TypeVar};
 
 use crate::shared::Definitions as SharedDefinitions;
 
-use crate::constant_hash;
 use crate::error;
 use crate::srcgen::{Formatter, Match};
 use crate::unique_table::{UniqueSeqTable, UniqueTable};
diff --git a/cranelift/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
index 84c81cddc4..f17a7ea0eb 100644
--- a/cranelift/codegen/meta/src/gen_settings.rs
+++ b/cranelift/codegen/meta/src/gen_settings.rs
@@ -1,12 +1,14 @@
+use std::collections::HashMap;
+
+use cranelift_codegen_shared::constant_hash::{generate_table, simple_hash};
+
 use crate::cdsl::camel_case;
 use crate::cdsl::settings::{
     BoolSetting, Predicate, Preset, Setting, SettingGroup, SpecificSetting,
 };
-use crate::constant_hash::{generate_table, simple_hash};
 use crate::error;
 use crate::srcgen::{Formatter, Match};
 use crate::unique_table::UniqueSeqTable;
-use std::collections::HashMap;
 
 pub enum ParentGroup {
     None,
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index ef450e7875..92d8721334 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -13,7 +13,6 @@ mod gen_registers;
 mod gen_settings;
 mod gen_types;
 
-mod constant_hash;
 mod default_map;
 mod shared;
 mod unique_table;
diff --git a/cranelift/codegen/shared/src/constant_hash.rs b/cranelift/codegen/shared/src/constant_hash.rs
new file mode 100644
index 0000000000..668dc9f657
--- /dev/null
+++ b/cranelift/codegen/shared/src/constant_hash.rs
@@ -0,0 +1,79 @@
+//! Build support for precomputed constant hash tables.
+//!
+//! This module can generate constant hash tables using open addressing and quadratic probing.
+//!
+//! The hash tables are arrays that are guaranteed to:
+//!
+//! - Have a power-of-two size.
+//! - Contain at least one empty slot.
+//!
+//! This module provides build meta support for lookups in these tables, as well as the shared hash
+//! function used for probing.
+
+use std::iter;
+
+/// A primitive hash function for matching opcodes.
+pub fn simple_hash(s: &str) -> usize {
+    let mut h: u32 = 5381;
+    for c in s.chars() {
+        h = (h ^ c as u32).wrapping_add(h.rotate_right(6));
+    }
+    h as usize
+}
+
+/// Compute an open addressed, quadratically probed hash table containing
+/// `items`. The returned table is a list containing the elements of the
+/// iterable `items` and `None` in unused slots.
+pub fn generate_table<'cont, T, I: iter::Iterator, H: Fn(&T) -> usize>(
+    items: I,
+    num_items: usize,
+    hash_function: H,
+) -> Vec> {
+    let size = (1.20 * num_items as f64) as usize;
+    // TODO do we really need the multiply by two here?
+    let size = if size.is_power_of_two() {
+        size * 2
+    } else {
+        size.next_power_of_two()
+    };
+
+    let mut table = vec![None; size];
+
+    for i in items {
+        let mut h = hash_function(&i) % size;
+        let mut s = 0;
+        while table[h].is_some() {
+            s += 1;
+            h = (h + s) % size;
+        }
+        table[h] = Some(i);
+    }
+
+    table
+}
+
+#[cfg(test)]
+mod tests {
+    use super::{generate_table, simple_hash};
+
+    #[test]
+    fn basic() {
+        assert_eq!(simple_hash("Hello"), 0x2fa70c01);
+        assert_eq!(simple_hash("world"), 0x5b0c31d5);
+    }
+
+    #[test]
+    fn test_generate_table() {
+        let v = vec!["Hello".to_string(), "world".to_string()];
+        let table = generate_table(v.iter(), v.len(), |s| simple_hash(&s));
+        assert_eq!(
+            table,
+            vec![
+                None,
+                Some(&"Hello".to_string()),
+                Some(&"world".to_string()),
+                None
+            ]
+        );
+    }
+}
diff --git a/cranelift/codegen/shared/src/lib.rs b/cranelift/codegen/shared/src/lib.rs
index 87de6e7951..478fb0526f 100644
--- a/cranelift/codegen/shared/src/lib.rs
+++ b/cranelift/codegen/shared/src/lib.rs
@@ -21,6 +21,7 @@
 )]
 
 pub mod condcodes;
+pub mod constant_hash;
 
 /// Version number of this crate.
 pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/cranelift/codegen/src/constant_hash.rs b/cranelift/codegen/src/constant_hash.rs
index 5785a89825..1de2a2edb4 100644
--- a/cranelift/codegen/src/constant_hash.rs
+++ b/cranelift/codegen/src/constant_hash.rs
@@ -1,13 +1,18 @@
 //! Runtime support for precomputed constant hash tables.
 //!
-//! The `cranelift-codegen/meta/src/constant_hash.rs` Rust crate can generate constant hash tables
-//! using open addressing and quadratic probing. The hash tables are arrays that are guaranteed to:
+//! The shared module with the same name can generate constant hash tables using open addressing
+//! and quadratic probing.
+//!
+//! The hash tables are arrays that are guaranteed to:
 //!
 //! - Have a power-of-two size.
 //! - Contain at least one empty slot.
 //!
 //! This module provides runtime support for lookups in these tables.
 
+// Re-export entities from constant_hash for simplicity of use.
+pub use cranelift_codegen_shared::constant_hash::*;
+
 /// Trait that must be implemented by the entries in a constant hash table.
 pub trait Table {
     /// Get the number of entries in this table which must be a power of two.
@@ -47,6 +52,7 @@ pub fn probe + ?Sized>(
 
         // Quadratic probing.
         step += 1;
+
         // When `table.len()` is a power of two, it can be proven that `idx` will visit all
         // entries. This means that this loop will always terminate if the hash table has even
         // one unused entry.
@@ -54,25 +60,3 @@ pub fn probe + ?Sized>(
         idx += step;
     }
 }
-
-/// A primitive hash function for matching opcodes.
-/// Must match `cranelift-codegen/meta/src/constant_hash.rs`.
-pub fn simple_hash(s: &str) -> usize {
-    let mut h: u32 = 5381;
-    for c in s.chars() {
-        h = (h ^ c as u32).wrapping_add(h.rotate_right(6));
-    }
-    h as usize
-}
-
-#[cfg(test)]
-mod tests {
-    use super::simple_hash;
-
-    #[test]
-    fn basic() {
-        // c.f. `meta/src/constant_hash.rs` tests.
-        assert_eq!(simple_hash("Hello"), 0x2fa70c01);
-        assert_eq!(simple_hash("world"), 0x5b0c31d5);
-    }
-}

From 097fa0c7b11a0c1f0ef43965a872d63548e3824f Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 9 Oct 2019 17:29:01 +0200
Subject: [PATCH 2814/3084] Clarify a comment in constant_hash::generate_table;

---
 cranelift/codegen/shared/src/constant_hash.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/shared/src/constant_hash.rs b/cranelift/codegen/shared/src/constant_hash.rs
index 668dc9f657..e16d58f5bc 100644
--- a/cranelift/codegen/shared/src/constant_hash.rs
+++ b/cranelift/codegen/shared/src/constant_hash.rs
@@ -30,7 +30,8 @@ pub fn generate_table<'cont, T, I: iter::Iterator, H: Fn(&T) ->
     hash_function: H,
 ) -> Vec> {
     let size = (1.20 * num_items as f64) as usize;
-    // TODO do we really need the multiply by two here?
+
+    // Probing code's stop condition relies on the table having one vacant entry at least.
     let size = if size.is_power_of_two() {
         size * 2
     } else {

From f668869508b61ef190a6c9ecf2f798c57132c6a5 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 9 Oct 2019 17:38:29 +0200
Subject: [PATCH 2815/3084] Share constants between codegen and the meta crate;

---
 cranelift/codegen/meta/src/cdsl/regs.rs    | 17 ++++++----
 cranelift/codegen/meta/src/cdsl/types.rs   | 21 ++----------
 cranelift/codegen/shared/src/constants.rs  | 30 +++++++++++++++++
 cranelift/codegen/shared/src/lib.rs        |  1 +
 cranelift/codegen/src/ir/types.rs          | 19 ++++-------
 cranelift/codegen/src/isa/registers.rs     | 39 +++++++++++++---------
 cranelift/codegen/src/regalloc/pressure.rs |  7 ++--
 7 files changed, 79 insertions(+), 55 deletions(-)
 create mode 100644 cranelift/codegen/shared/src/constants.rs

diff --git a/cranelift/codegen/meta/src/cdsl/regs.rs b/cranelift/codegen/meta/src/cdsl/regs.rs
index 920b1f5426..d60835ab3f 100644
--- a/cranelift/codegen/meta/src/cdsl/regs.rs
+++ b/cranelift/codegen/meta/src/cdsl/regs.rs
@@ -1,3 +1,4 @@
+use cranelift_codegen_shared::constants;
 use cranelift_entity::{entity_impl, EntityRef, PrimaryMap};
 
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
@@ -364,19 +365,21 @@ impl IsaRegsBuilder {
             }
         }
 
-        // This limit should be coordinated with the `RegClassMask` and `RegClassIndex` types in
-        // isa/registers.rs of the non-meta code.
-        assert!(self.classes.len() <= 32, "Too many register classes");
+        assert!(
+            self.classes.len() <= constants::MAX_NUM_REG_CLASSES,
+            "Too many register classes"
+        );
 
-        // The maximum number of top-level register classes which have pressure tracking should be
-        // kept in sync with the MAX_TRACKED_TOPRCS constant in isa/registers.rs of the non-meta
-        // code.
         let num_toplevel = self
             .classes
             .values()
             .filter(|x| x.toprc == x.index && self.banks.get(x.bank).unwrap().pressure_tracking)
             .count();
-        assert!(num_toplevel <= 4, "Too many top-level register classes");
+
+        assert!(
+            num_toplevel <= constants::MAX_TRACKED_TOP_RCS,
+            "Too many top-level register classes"
+        );
 
         IsaRegs::new(self.banks, self.classes)
     }
diff --git a/cranelift/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
index 92b9ab3a2f..33ce0acfc1 100644
--- a/cranelift/codegen/meta/src/cdsl/types.rs
+++ b/cranelift/codegen/meta/src/cdsl/types.rs
@@ -1,24 +1,9 @@
 //! Cranelift ValueType hierarchy
 
-// Temporary disabled: Unused at the moment.
-// use std::collections::HashMap;
-
 use std::fmt;
 
 use crate::shared::types as shared_types;
-
-// Numbering scheme for value types:
-//
-// 0: Void
-// 0x01-0x6f: Special types
-// 0x70-0x7d: Lane types
-// 0x7e-0x7f: Reference types
-// 0x80-0xff: Vector types
-//
-// Vector types are encoded with the lane type in the low 4 bits and log2(lanes)
-// in the high 4 bits, giving a range of 2-256 lanes.
-static LANE_BASE: u8 = 0x70;
-static REFERENCE_BASE: u8 = 0x7E;
+use cranelift_codegen_shared::constants;
 
 // Rust name prefix used for the `rust_name` method.
 static _RUST_NAME_PREFIX: &'static str = "ir::types::";
@@ -208,7 +193,7 @@ impl LaneType {
 
     /// Find the unique number associated with this lane type.
     pub fn number(self) -> u8 {
-        LANE_BASE
+        constants::LANE_BASE
             + match self {
                 LaneType::BoolType(shared_types::Bool::B1) => 0,
                 LaneType::BoolType(shared_types::Bool::B8) => 1,
@@ -579,7 +564,7 @@ impl ReferenceType {
 
     /// Find the unique number associated with this reference type.
     pub fn number(self) -> u8 {
-        REFERENCE_BASE
+        constants::REFERENCE_BASE
             + match self {
                 ReferenceType(shared_types::Reference::R32) => 0,
                 ReferenceType(shared_types::Reference::R64) => 1,
diff --git a/cranelift/codegen/shared/src/constants.rs b/cranelift/codegen/shared/src/constants.rs
new file mode 100644
index 0000000000..b3f1377856
--- /dev/null
+++ b/cranelift/codegen/shared/src/constants.rs
@@ -0,0 +1,30 @@
+//! This module contains constants that are shared between the codegen and the meta crate, so they
+//! are kept in sync.
+
+// Numbering scheme for value types:
+//
+// 0: Void
+// 0x01-0x6f: Special types
+// 0x70-0x7d: Lane types
+// 0x7e-0x7f: Reference types
+// 0x80-0xff: Vector types
+//
+// Vector types are encoded with the lane type in the low 4 bits and log2(lanes)
+// in the high 4 bits, giving a range of 2-256 lanes.
+
+/// Start of the lane types.
+pub const LANE_BASE: u8 = 0x70;
+
+/// Base for reference types.
+pub const REFERENCE_BASE: u8 = 0x7E;
+
+/// Start of the 2-lane vector types.
+pub const VECTOR_BASE: u8 = 0x80;
+
+// Some constants about register classes and types.
+
+/// Guaranteed maximum number of top-level register classes with pressure tracking in any ISA.
+pub const MAX_TRACKED_TOP_RCS: usize = 4;
+
+/// Guaranteed maximum number of register classes in any ISA.
+pub const MAX_NUM_REG_CLASSES: usize = 32;
diff --git a/cranelift/codegen/shared/src/lib.rs b/cranelift/codegen/shared/src/lib.rs
index 478fb0526f..a2c5f3df33 100644
--- a/cranelift/codegen/shared/src/lib.rs
+++ b/cranelift/codegen/shared/src/lib.rs
@@ -22,6 +22,7 @@
 
 pub mod condcodes;
 pub mod constant_hash;
+pub mod constants;
 
 /// Version number of this crate.
 pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/cranelift/codegen/src/ir/types.rs b/cranelift/codegen/src/ir/types.rs
index 4ada74b070..4bdec45268 100644
--- a/cranelift/codegen/src/ir/types.rs
+++ b/cranelift/codegen/src/ir/types.rs
@@ -2,6 +2,7 @@
 
 use core::default::Default;
 use core::fmt::{self, Debug, Display, Formatter};
+use cranelift_codegen_shared::constants;
 use target_lexicon::{PointerWidth, Triple};
 
 /// The type of an SSA value.
@@ -25,12 +26,6 @@ pub struct Type(u8);
 /// Not a valid type. Can't be loaded or stored. Can't be part of a SIMD vector.
 pub const INVALID: Type = Type(0);
 
-/// Start of the lane types. See also `meta/src/cdsl/types.rs`.
-const LANE_BASE: u8 = 0x70;
-
-/// Start of the 2-lane vector types.
-const VECTOR_BASE: u8 = LANE_BASE + 16;
-
 // Include code generated by `cranelift-codegen/meta/gen_types.rs`. This file contains constant
 // definitions for all the scalar types as well as common vector types for 64, 128, 256, and
 // 512-bit SIMD vectors.
@@ -41,10 +36,10 @@ impl Type {
     ///
     /// A lane type is the same as a SIMD vector type with one lane, so it returns itself.
     pub fn lane_type(self) -> Self {
-        if self.0 < VECTOR_BASE {
+        if self.0 < constants::VECTOR_BASE {
             self
         } else {
-            Self(LANE_BASE | (self.0 & 0x0f))
+            Self(constants::LANE_BASE | (self.0 & 0x0f))
         }
     }
 
@@ -170,21 +165,21 @@ impl Type {
 
     /// Is this a special type?
     pub fn is_special(self) -> bool {
-        self.0 < LANE_BASE
+        self.0 < constants::LANE_BASE
     }
 
     /// Is this a lane type?
     ///
     /// This is a scalar type that can also appear as the lane type of a SIMD vector.
     pub fn is_lane(self) -> bool {
-        LANE_BASE <= self.0 && self.0 < VECTOR_BASE
+        constants::LANE_BASE <= self.0 && self.0 < constants::VECTOR_BASE
     }
 
     /// Is this a SIMD vector type?
     ///
     /// A vector type has 2 or more lanes.
     pub fn is_vector(self) -> bool {
-        self.0 >= VECTOR_BASE
+        self.0 >= constants::VECTOR_BASE
     }
 
     /// Is this a scalar boolean type?
@@ -234,7 +229,7 @@ impl Type {
     ///
     /// A scalar type is the same as a SIMD vector type with one lane, so it returns 0.
     pub fn log2_lane_count(self) -> u8 {
-        self.0.saturating_sub(LANE_BASE) >> 4
+        self.0.saturating_sub(constants::LANE_BASE) >> 4
     }
 
     /// Get the number of lanes in this SIMD vector type.
diff --git a/cranelift/codegen/src/isa/registers.rs b/cranelift/codegen/src/isa/registers.rs
index f7fcdcac2e..d3544904e0 100644
--- a/cranelift/codegen/src/isa/registers.rs
+++ b/cranelift/codegen/src/isa/registers.rs
@@ -12,25 +12,16 @@ use core::fmt;
 /// The register allocator will enforce that each register unit only gets used for one thing.
 pub type RegUnit = u16;
 
+/// A bit mask indexed by register classes.
+///
+/// The size of this type is determined by the ISA with the most register classes.
+pub type RegClassMask = u32;
+
 /// A bit mask indexed by register units.
 ///
 /// The size of this type is determined by the target ISA that has the most register units defined.
 /// Currently that is arm32 which has 64+16 units.
-///
-/// This type should be coordinated with meta/src/cdsl/regs.rs.
-pub type RegUnitMask = [u32; 3];
-
-/// A bit mask indexed by register classes.
-///
-/// The size of this type is determined by the ISA with the most register classes.
-///
-/// This type should be coordinated with meta/src/cdsl/regs.rs.
-pub type RegClassMask = u32;
-
-/// Guaranteed maximum number of top-level register classes with pressure tracking in any ISA.
-///
-/// This can be increased, but should be coordinated with meta/src/cdsl/regs.rs.
-pub const MAX_TRACKED_TOPRCS: usize = 4;
+pub type RegUnitMask = [RegClassMask; 3];
 
 /// The register units in a target ISA are divided into disjoint register banks. Each bank covers a
 /// contiguous range of register units.
@@ -338,3 +329,21 @@ impl<'a> fmt::Display for DisplayRegUnit<'a> {
         }
     }
 }
+
+#[test]
+fn assert_sizes() {
+    use cranelift_codegen_shared::constants;
+    use std::mem::size_of;
+
+    // In these tests, size_of returns number of bytes: we actually want the number of bits, so
+    // multiply these by 8.
+    assert!(
+        (size_of::() * 8) <= constants::MAX_NUM_REG_CLASSES,
+        "need to bump MAX_NUM_REG_CLASSES or change RegClassMask type"
+    );
+
+    assert!(
+        constants::MAX_NUM_REG_CLASSES < (1 << (size_of::() * 8)),
+        "need to change RegClassIndex's type to a wider type"
+    );
+}
diff --git a/cranelift/codegen/src/regalloc/pressure.rs b/cranelift/codegen/src/regalloc/pressure.rs
index c7fc1fff9a..73c7fbc1a3 100644
--- a/cranelift/codegen/src/regalloc/pressure.rs
+++ b/cranelift/codegen/src/regalloc/pressure.rs
@@ -36,11 +36,12 @@
 // Remove once we're using the pressure tracker.
 #![allow(dead_code)]
 
-use crate::isa::registers::{RegClass, RegClassMask, RegInfo, MAX_TRACKED_TOPRCS};
+use crate::isa::registers::{RegClass, RegClassMask, RegInfo};
 use crate::regalloc::RegisterSet;
 use core::cmp::min;
 use core::fmt;
 use core::iter::ExactSizeIterator;
+use cranelift_codegen_shared::constants::MAX_TRACKED_TOP_RCS;
 
 /// Information per top-level register class.
 ///
@@ -76,7 +77,7 @@ pub struct Pressure {
     aliased: RegClassMask,
 
     // Current register counts per top-level register class.
-    toprc: [TopRC; MAX_TRACKED_TOPRCS],
+    toprc: [TopRC; MAX_TRACKED_TOP_RCS],
 }
 
 impl Pressure {
@@ -105,7 +106,7 @@ impl Pressure {
             } else {
                 // This bank has no pressure tracking, so its top-level register classes may exceed
                 // `MAX_TRACKED_TOPRCS`. Fill in dummy entries.
-                for rc in &mut p.toprc[first..min(first + num, MAX_TRACKED_TOPRCS)] {
+                for rc in &mut p.toprc[first..min(first + num, MAX_TRACKED_TOP_RCS)] {
                     // These aren't used if we don't set the `aliased` bit.
                     rc.first_toprc = !0;
                     rc.limit = !0;

From f1c25c2c5ab0445873116b75687123cab8fca597 Mon Sep 17 00:00:00 2001
From: Ujjwal Sharma 
Date: Thu, 3 Oct 2019 04:33:59 +0530
Subject: [PATCH 2816/3084] [codegen] legalize imul for 64-bit and 128-bit
 operands

Add a legalization that legalizes imul.I64 for 32-bit ISAs and imul.I128
for 64-bit (and subsequently 32-bit) ISAs.

Refs: https://github.com/bnjbvr/cranelift-x86/issues/4
---
 cranelift/codegen/meta/src/shared/legalize.rs | 20 +++++++++++++++++++
 .../filetests/isa/x86/imul-i128.clif          | 20 +++++++++++++++++++
 .../filetests/isa/x86/legalize-i128.clif      | 20 +++++++++++++++++++
 .../filetests/isa/x86/legalize-i64.clif       | 15 ++++++++++++++
 4 files changed, 75 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/imul-i128.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/legalize-i128.clif

diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index bf046f5ddd..aef8cca89a 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -115,6 +115,7 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     let uextend = insts.by_name("uextend");
     let uload8 = insts.by_name("uload8");
     let uload16 = insts.by_name("uload16");
+    let umulhi = insts.by_name("umulhi");
     let ushr = insts.by_name("ushr");
     let ushr_imm = insts.by_name("ushr_imm");
     let urem = insts.by_name("urem");
@@ -336,6 +337,25 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
         }
     }
 
+    // TODO(ryzokuken): explore the perf diff w/ x86_umulx and consider have a
+    // separate legalization for x86.
+    for &ty in &[I64, I128] {
+        narrow.legalize(
+            def!(a = imul.ty(x, y)),
+            vec![
+                def!((xl, xh) = isplit(x)),
+                def!((yl, yh) = isplit(y)),
+                def!(a1 = imul(xh, yl)),
+                def!(a2 = imul(xl, yh)),
+                def!(a3 = iadd(a1, a2)),
+                def!(a4 = umulhi(xl, yl)),
+                def!(ah = iadd(a3, a4)),
+                def!(al = imul(xl, yl)),
+                def!(a = iconcat(al, ah)),
+            ],
+        );
+    }
+
     // Widen instructions with one input operand.
     for &op in &[bnot, popcnt] {
         for &int_ty in &[I8, I16] {
diff --git a/cranelift/filetests/filetests/isa/x86/imul-i128.clif b/cranelift/filetests/filetests/isa/x86/imul-i128.clif
new file mode 100644
index 0000000000..2d683a32dd
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/imul-i128.clif
@@ -0,0 +1,20 @@
+test run
+target x86_64 haswell
+
+function %test_imul_i128() -> b1 {
+ebb0:
+    v11 = iconst.i64 0xf2347ac4503f1e24
+    v12 = iconst.i64 0x0098fe985354ab06
+    v1 = iconcat v11, v12
+    v21 = iconst.i64 0xf606ba453589ef89
+    v22 = iconst.i64 0x042e1f3054ca7432
+    v2 = iconcat v21, v22
+    v31 = iconst.i64 0xbe2044b2742ebd44
+    v32 = iconst.i64 0xa363ce3b6849f307
+    v3 = iconcat v31, v32
+    v4 = imul v1, v2
+    v5 = icmp eq v3, v4
+    return v5
+}
+
+; run
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-i128.clif b/cranelift/filetests/filetests/isa/x86/legalize-i128.clif
new file mode 100644
index 0000000000..db071ba3c7
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/legalize-i128.clif
@@ -0,0 +1,20 @@
+; Test the legalization of i128 instructions on x86_64.
+test legalizer
+target x86_64 haswell
+
+; regex: V=v\d+
+
+function %imul(i128, i128) -> i128 {
+ebb0(v1: i128, v2: i128):
+    v10 = imul v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(v11=$V) = imul $v1_msb, $v2_lsb
+    ; nextln: $(v12=$V) = imul $v1_lsb, $v2_msb
+    ; nextln: $(v13=$V) = iadd $v11, $v12
+    ; nextln: $(v99=$V), $(v14=$V) = x86_umulx $v1_lsb, $v2_lsb
+    ; nextln: $(v10_msb=$V) = iadd $v13, $v14
+    ; nextln: $(v10_lsb=$V) = imul $v1_lsb, $v2_lsb
+    ; nextln: v10 = iconcat $v10_lsb, $v10_msb
+    return v10
+}
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-i64.clif b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
index 2723bd69b8..a484818a34 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
@@ -26,6 +26,21 @@ ebb0(v1: i64, v2: i64):
     return v10
 }
 
+function %imul(i64, i64) -> i64 {
+ebb0(v1: i64, v2: i64):
+    v10 = imul v1, v2
+    ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
+    ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
+    ; nextln: $(v11=$V) = imul $v1_msb, $v2_lsb
+    ; nextln: $(v12=$V) = imul $v1_lsb, $v2_msb
+    ; nextln: $(v13=$V) = iadd $v11, $v12
+    ; nextln: $(v99=$V), $(v14=$V) = x86_umulx $v1_lsb, $v2_lsb
+    ; nextln: $(v10_msb=$V) = iadd $v13, $v14
+    ; nextln: $(v10_lsb=$V) = imul $v1_lsb, $v2_lsb
+    ; nextln: v10 = iconcat $v10_lsb, $v10_msb
+    return v10
+}
+
 function %icmp_eq(i64, i64) -> b1 {
 ebb0(v1: i64, v2: i64):
     v10 = icmp eq v1, v2

From 6d690e52756bfd7c2210cba59959a026b8f3f2ce Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 10 Oct 2019 08:54:46 -0700
Subject: [PATCH 2817/3084] Allow binding immediates to instructions (#1012)

This change should make the code more clear (and less code) when adding encodings for instructions with specific immediates; e.g., a constant with a 0 immediate could be encoded as an XOR with something like `const.bind(...)` without explicitly creating the necessary predicates. It has several parts:
* Introduce Bindable trait to instructions
* Convert all instruction bindings to use Bindable::bind()
* Add ability to bind immediates to BoundInstruction
This is an attempt to reduce some of the issues in #955.
---
 cranelift/codegen/meta/src/cdsl/encodings.rs  |  37 +-
 .../codegen/meta/src/cdsl/instructions.rs     | 369 ++++++++++++------
 .../codegen/meta/src/isa/riscv/encodings.rs   | 204 +++++-----
 .../codegen/meta/src/isa/x86/encodings.rs     | 184 ++++-----
 .../codegen/meta/src/isa/x86/legalize.rs      |  22 +-
 cranelift/codegen/meta/src/shared/legalize.rs |   2 +-
 6 files changed, 477 insertions(+), 341 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/encodings.rs b/cranelift/codegen/meta/src/cdsl/encodings.rs
index 77d6bc7374..540d3402ae 100644
--- a/cranelift/codegen/meta/src/cdsl/encodings.rs
+++ b/cranelift/codegen/meta/src/cdsl/encodings.rs
@@ -1,5 +1,4 @@
-use std::rc::Rc;
-
+use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::instructions::{
     InstSpec, Instruction, InstructionPredicate, InstructionPredicateNode,
     InstructionPredicateNumber, InstructionPredicateRegistry, ValueTypeOrAny,
@@ -7,6 +6,8 @@ use crate::cdsl::instructions::{
 use crate::cdsl::recipes::{EncodingRecipeNumber, Recipes};
 use crate::cdsl::settings::SettingPredicateNumber;
 use crate::cdsl::types::ValueType;
+use std::rc::Rc;
+use std::string::ToString;
 
 /// Encoding for a concrete instruction.
 ///
@@ -61,19 +62,25 @@ pub(crate) struct EncodingBuilder {
 }
 
 impl EncodingBuilder {
-    pub fn new(inst: InstSpec, recipe: EncodingRecipeNumber, encbits: u16) -> Self {
+    pub fn new(
+        inst: InstSpec,
+        recipe: EncodingRecipeNumber,
+        encbits: u16,
+        formats: &FormatRegistry,
+    ) -> Self {
         let (inst_predicate, bound_type) = match &inst {
             InstSpec::Bound(inst) => {
                 let other_typevars = &inst.inst.polymorphic_info.as_ref().unwrap().other_typevars;
 
-                assert!(
-                    inst.value_types.len() == other_typevars.len() + 1,
+                assert_eq!(
+                    inst.value_types.len(),
+                    other_typevars.len() + 1,
                     "partially bound polymorphic instruction"
                 );
 
                 // Add secondary type variables to the instruction predicate.
                 let value_types = &inst.value_types;
-                let mut inst_predicate = None;
+                let mut inst_predicate: Option = None;
                 for (typevar, value_type) in other_typevars.iter().zip(value_types.iter().skip(1)) {
                     let value_type = match value_type {
                         ValueTypeOrAny::Any => continue,
@@ -84,6 +91,24 @@ impl EncodingBuilder {
                     inst_predicate = Some(type_predicate.into());
                 }
 
+                // Add immediate value predicates
+                for (immediate_value, immediate_operand) in inst
+                    .immediate_values
+                    .iter()
+                    .zip(inst.inst.operands_in.iter().filter(|o| o.is_immediate()))
+                {
+                    let immediate_predicate = InstructionPredicate::new_is_field_equal(
+                        formats.get(inst.inst.format),
+                        immediate_operand.name,
+                        immediate_value.to_string(),
+                    );
+                    inst_predicate = if let Some(type_predicate) = inst_predicate {
+                        Some(type_predicate.and(immediate_predicate))
+                    } else {
+                        Some(immediate_predicate.into())
+                    }
+                }
+
                 let ctrl_type = value_types[0]
                     .clone()
                     .expect("Controlling type shouldn't be Any");
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 178077e456..5d47c1a76b 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -2,6 +2,7 @@ use cranelift_entity::{entity_impl, PrimaryMap};
 
 use std::collections::HashMap;
 use std::fmt;
+use std::fmt::{Display, Error, Formatter};
 use std::ops;
 use std::rc::Rc;
 
@@ -13,6 +14,7 @@ use crate::cdsl::operands::Operand;
 use crate::cdsl::type_inference::Constraint;
 use crate::cdsl::types::{LaneType, ReferenceType, ValueType, VectorType};
 use crate::cdsl::typevar::TypeVar;
+use crate::shared::types::{Bool, Float, Int, Reference};
 use cranelift_codegen_shared::condcodes::IntCC;
 
 #[derive(Copy, Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
@@ -80,6 +82,14 @@ impl InstructionGroup {
     }
 }
 
+/// Instructions can have parameters bound to them to specialize them for more specific encodings
+/// (e.g. the encoding for adding two float types may be different than that of adding two
+/// integer types)
+pub trait Bindable {
+    /// Bind a parameter to an instruction
+    fn bind(&self, parameter: impl Into) -> BoundInstruction;
+}
+
 #[derive(Debug)]
 pub struct PolymorphicInfo {
     pub use_typevar_operand: bool,
@@ -173,30 +183,11 @@ impl Instruction {
             None => Vec::new(),
         }
     }
+}
 
-    pub fn bind(&self, lane_type: impl Into) -> BoundInstruction {
-        bind(self.clone(), Some(lane_type.into()), Vec::new())
-    }
-
-    pub fn bind_ref(&self, reference_type: impl Into) -> BoundInstruction {
-        bind_ref(self.clone(), Some(reference_type.into()), Vec::new())
-    }
-
-    pub fn bind_vector_from_lane(
-        &self,
-        lane_type: impl Into,
-        vector_size_in_bits: u64,
-    ) -> BoundInstruction {
-        bind_vector(
-            self.clone(),
-            lane_type.into(),
-            vector_size_in_bits,
-            Vec::new(),
-        )
-    }
-
-    pub fn bind_any(&self) -> BoundInstruction {
-        bind(self.clone(), None, Vec::new())
+impl Bindable for Instruction {
+    fn bind(&self, parameter: impl Into) -> BoundInstruction {
+        BoundInstruction::new(self).bind(parameter)
     }
 }
 
@@ -407,36 +398,163 @@ impl ValueTypeOrAny {
     }
 }
 
+/// The number of bits in the vector
+type VectorBitWidth = u64;
+
+/// An parameter used for binding instructions to specific types or values
+pub enum BindParameter {
+    Any,
+    Lane(LaneType),
+    Vector(LaneType, VectorBitWidth),
+    Reference(ReferenceType),
+    Immediate(Immediate),
+}
+
+/// Constructor for more easily building vector parameters from any lane type
+pub fn vector(parameter: impl Into, vector_size: VectorBitWidth) -> BindParameter {
+    BindParameter::Vector(parameter.into(), vector_size)
+}
+
+impl From for BindParameter {
+    fn from(ty: Int) -> Self {
+        BindParameter::Lane(ty.into())
+    }
+}
+
+impl From for BindParameter {
+    fn from(ty: Bool) -> Self {
+        BindParameter::Lane(ty.into())
+    }
+}
+
+impl From for BindParameter {
+    fn from(ty: Float) -> Self {
+        BindParameter::Lane(ty.into())
+    }
+}
+
+impl From for BindParameter {
+    fn from(ty: LaneType) -> Self {
+        BindParameter::Lane(ty)
+    }
+}
+
+impl From for BindParameter {
+    fn from(ty: Reference) -> Self {
+        BindParameter::Reference(ty.into())
+    }
+}
+
+impl From for BindParameter {
+    fn from(imm: Immediate) -> Self {
+        BindParameter::Immediate(imm)
+    }
+}
+
+#[derive(Clone)]
+pub enum Immediate {
+    UInt8(u8),
+    UInt128(u128),
+}
+
+impl Display for Immediate {
+    fn fmt(&self, f: &mut Formatter) -> Result<(), Error> {
+        match self {
+            Immediate::UInt8(x) => write!(f, "{}", x),
+            Immediate::UInt128(x) => write!(f, "{}", x),
+        }
+    }
+}
+
 #[derive(Clone)]
 pub struct BoundInstruction {
     pub inst: Instruction,
     pub value_types: Vec,
+    pub immediate_values: Vec,
 }
 
 impl BoundInstruction {
-    pub fn bind(self, lane_type: impl Into) -> BoundInstruction {
-        bind(self.inst, Some(lane_type.into()), self.value_types)
+    /// Construct a new bound instruction (with nothing bound yet) from an instruction
+    fn new(inst: &Instruction) -> Self {
+        BoundInstruction {
+            inst: inst.clone(),
+            value_types: vec![],
+            immediate_values: vec![],
+        }
     }
 
-    pub fn bind_ref(self, reference_type: impl Into) -> BoundInstruction {
-        bind_ref(self.inst, Some(reference_type.into()), self.value_types)
-    }
+    /// Verify that the bindings for a BoundInstruction are correct.
+    fn verify_bindings(&self) -> Result<(), String> {
+        // Verify that binding types to the instruction does not violate the polymorphic rules.
+        if !self.value_types.is_empty() {
+            match &self.inst.polymorphic_info {
+                Some(poly) => {
+                    if self.value_types.len() > 1 + poly.other_typevars.len() {
+                        return Err(format!(
+                            "trying to bind too many types for {}",
+                            self.inst.name
+                        ));
+                    }
+                }
+                None => {
+                    return Err(format!(
+                        "trying to bind a type for {} which is not a polymorphic instruction",
+                        self.inst.name
+                    ));
+                }
+            }
+        }
 
-    pub fn bind_vector_from_lane(
-        self,
-        lane_type: impl Into,
-        vector_size_in_bits: u64,
-    ) -> BoundInstruction {
-        bind_vector(
-            self.inst,
-            lane_type.into(),
-            vector_size_in_bits,
-            self.value_types,
-        )
-    }
+        // Verify that only the right number of immediates are bound.
+        let immediate_count = self
+            .inst
+            .operands_in
+            .iter()
+            .filter(|o| o.is_immediate())
+            .count();
+        if self.immediate_values.len() > immediate_count {
+            return Err(format!(
+                "trying to bind too many immediates ({}) to instruction {} which only expects {} \
+                 immediates",
+                self.immediate_values.len(),
+                self.inst.name,
+                immediate_count
+            ));
+        }
 
-    pub fn bind_any(self) -> BoundInstruction {
-        bind(self.inst, None, self.value_types)
+        Ok(())
+    }
+}
+
+impl Bindable for BoundInstruction {
+    fn bind(&self, parameter: impl Into) -> BoundInstruction {
+        let mut modified = self.clone();
+        match parameter.into() {
+            BindParameter::Any => modified.value_types.push(ValueTypeOrAny::Any),
+            BindParameter::Lane(lane_type) => modified
+                .value_types
+                .push(ValueTypeOrAny::ValueType(lane_type.into())),
+            BindParameter::Vector(lane_type, vector_size_in_bits) => {
+                let num_lanes = vector_size_in_bits / lane_type.lane_bits();
+                assert!(
+                    num_lanes >= 2,
+                    "Minimum lane number for bind_vector is 2, found {}.",
+                    num_lanes,
+                );
+                let vector_type = ValueType::Vector(VectorType::new(lane_type, num_lanes));
+                modified
+                    .value_types
+                    .push(ValueTypeOrAny::ValueType(vector_type));
+            }
+            BindParameter::Reference(reference_type) => {
+                modified
+                    .value_types
+                    .push(ValueTypeOrAny::ValueType(reference_type.into()));
+            }
+            BindParameter::Immediate(immediate) => modified.immediate_values.push(immediate),
+        }
+        modified.verify_bindings().unwrap();
+        modified
     }
 }
 
@@ -1124,17 +1242,13 @@ impl InstSpec {
             InstSpec::Bound(bound_inst) => &bound_inst.inst,
         }
     }
-    pub fn bind(&self, lane_type: impl Into) -> BoundInstruction {
-        match self {
-            InstSpec::Inst(inst) => inst.bind(lane_type),
-            InstSpec::Bound(inst) => inst.clone().bind(lane_type),
-        }
-    }
+}
 
-    pub fn bind_ref(&self, reference_type: impl Into) -> BoundInstruction {
+impl Bindable for InstSpec {
+    fn bind(&self, parameter: impl Into) -> BoundInstruction {
         match self {
-            InstSpec::Inst(inst) => inst.bind_ref(reference_type),
-            InstSpec::Bound(inst) => inst.clone().bind_ref(reference_type),
+            InstSpec::Inst(inst) => inst.bind(parameter.into()),
+            InstSpec::Bound(inst) => inst.bind(parameter.into()),
         }
     }
 }
@@ -1151,79 +1265,94 @@ impl Into for BoundInstruction {
     }
 }
 
-/// Helper bind reused by {Bound,}Instruction::bind.
-fn bind(
-    inst: Instruction,
-    lane_type: Option,
-    mut value_types: Vec,
-) -> BoundInstruction {
-    match lane_type {
-        Some(lane_type) => {
-            value_types.push(ValueTypeOrAny::ValueType(lane_type.into()));
-        }
-        None => {
-            value_types.push(ValueTypeOrAny::Any);
-        }
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::cdsl::formats::InstructionFormatBuilder;
+    use crate::cdsl::operands::{OperandBuilder, OperandKindBuilder, OperandKindFields};
+    use crate::cdsl::typevar::TypeSetBuilder;
+    use crate::shared::types::Int::{I32, I64};
+
+    fn field_to_operand(index: usize, field: OperandKindFields) -> Operand {
+        // pretend the index string is &'static
+        let name = Box::leak(index.to_string().into_boxed_str());
+        let kind = OperandKindBuilder::new(name, field).build();
+        let operand = OperandBuilder::new(name, kind).build();
+        operand
     }
 
-    verify_polymorphic_binding(&inst, &value_types);
-
-    BoundInstruction { inst, value_types }
-}
-
-/// Helper bind for reference types reused by {Bound,}Instruction::bind_ref.
-fn bind_ref(
-    inst: Instruction,
-    reference_type: Option,
-    mut value_types: Vec,
-) -> BoundInstruction {
-    match reference_type {
-        Some(reference_type) => {
-            value_types.push(ValueTypeOrAny::ValueType(reference_type.into()));
-        }
-        None => {
-            value_types.push(ValueTypeOrAny::Any);
-        }
+    fn field_to_operands(types: Vec) -> Vec {
+        types
+            .iter()
+            .enumerate()
+            .map(|(i, f)| field_to_operand(i, f.clone()))
+            .collect()
     }
 
-    verify_polymorphic_binding(&inst, &value_types);
-
-    BoundInstruction { inst, value_types }
-}
-
-/// Helper bind for vector types reused by {Bound,}Instruction::bind.
-fn bind_vector(
-    inst: Instruction,
-    lane_type: LaneType,
-    vector_size_in_bits: u64,
-    mut value_types: Vec,
-) -> BoundInstruction {
-    let num_lanes = vector_size_in_bits / lane_type.lane_bits();
-    assert!(
-        num_lanes >= 2,
-        "Minimum lane number for bind_vector is 2, found {}.",
-        num_lanes,
-    );
-    let vector_type = ValueType::Vector(VectorType::new(lane_type, num_lanes));
-    value_types.push(ValueTypeOrAny::ValueType(vector_type));
-    verify_polymorphic_binding(&inst, &value_types);
-    BoundInstruction { inst, value_types }
-}
-
-/// Helper to verify that binding types to the instruction does not violate polymorphic rules
-fn verify_polymorphic_binding(inst: &Instruction, value_types: &Vec) {
-    match &inst.polymorphic_info {
-        Some(poly) => {
-            assert!(
-                value_types.len() <= 1 + poly.other_typevars.len(),
-                format!("trying to bind too many types for {}", inst.name)
-            );
-        }
-        None => {
-            panic!(format!(
-                "trying to bind a type for {} which is not a polymorphic instruction",
-                inst.name
-            ));
+    fn build_fake_instruction(
+        inputs: Vec,
+        outputs: Vec,
+    ) -> Instruction {
+        // setup a format from the input operands
+        let mut formats = FormatRegistry::new();
+        let mut format = InstructionFormatBuilder::new("fake");
+        for (i, f) in inputs.iter().enumerate() {
+            match f {
+                OperandKindFields::TypeVar(_) => format = format.value(),
+                OperandKindFields::ImmValue => {
+                    format = format.imm(&field_to_operand(i, f.clone()).kind)
+                }
+                _ => {}
+            };
         }
+        formats.insert(format);
+
+        // create the fake instruction
+        InstructionBuilder::new("fake", "A fake instruction for testing.")
+            .operands_in(field_to_operands(inputs).iter().collect())
+            .operands_out(field_to_operands(outputs).iter().collect())
+            .build(&formats, OpcodeNumber(42))
+    }
+
+    #[test]
+    fn ensure_bound_instructions_can_bind_lane_types() {
+        let type1 = TypeSetBuilder::new().ints(8..64).build();
+        let in1 = OperandKindFields::TypeVar(TypeVar::new("a", "...", type1));
+        let inst = build_fake_instruction(vec![in1], vec![]);
+        inst.bind(LaneType::IntType(I32));
+    }
+
+    #[test]
+    fn ensure_bound_instructions_can_bind_immediates() {
+        let inst = build_fake_instruction(vec![OperandKindFields::ImmValue], vec![]);
+        let bound_inst = inst.bind(Immediate::UInt8(42));
+        assert!(bound_inst.verify_bindings().is_ok());
+    }
+
+    #[test]
+    #[should_panic]
+    fn ensure_instructions_fail_to_bind() {
+        let inst = build_fake_instruction(vec![], vec![]);
+        inst.bind(BindParameter::Lane(LaneType::IntType(I32)));
+        // trying to bind to an instruction with no inputs should fail
+    }
+
+    #[test]
+    #[should_panic]
+    fn ensure_bound_instructions_fail_to_bind_too_many_types() {
+        let type1 = TypeSetBuilder::new().ints(8..64).build();
+        let in1 = OperandKindFields::TypeVar(TypeVar::new("a", "...", type1));
+        let inst = build_fake_instruction(vec![in1], vec![]);
+        inst.bind(LaneType::IntType(I32))
+            .bind(LaneType::IntType(I64));
+    }
+
+    #[test]
+    #[should_panic]
+    fn ensure_instructions_fail_to_bind_too_many_immediates() {
+        let inst = build_fake_instruction(vec![OperandKindFields::ImmValue], vec![]);
+        inst.bind(BindParameter::Immediate(Immediate::UInt8(0)))
+            .bind(BindParameter::Immediate(Immediate::UInt8(1)));
+        // trying to bind too many immediates to an instruction  should fail
     }
 }
diff --git a/cranelift/codegen/meta/src/isa/riscv/encodings.rs b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
index 6ddc13b0a3..21ad3c469c 100644
--- a/cranelift/codegen/meta/src/isa/riscv/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
@@ -1,7 +1,7 @@
 use crate::cdsl::ast::{Apply, Expr, Literal, VarPool};
 use crate::cdsl::encodings::{Encoding, EncodingBuilder};
 use crate::cdsl::instructions::{
-    BoundInstruction, InstSpec, InstructionPredicateNode, InstructionPredicateRegistry,
+    Bindable, BoundInstruction, InstSpec, InstructionPredicateNode, InstructionPredicateRegistry,
 };
 use crate::cdsl::recipes::{EncodingRecipeNumber, Recipes};
 use crate::cdsl::settings::SettingGroup;
@@ -13,27 +13,34 @@ use crate::shared::types::Reference::{R32, R64};
 use crate::shared::Definitions as SharedDefinitions;
 
 use super::recipes::RecipeGroup;
-
-fn enc(inst: impl Into, recipe: EncodingRecipeNumber, bits: u16) -> EncodingBuilder {
-    EncodingBuilder::new(inst.into(), recipe, bits)
-}
+use crate::cdsl::formats::FormatRegistry;
 
 pub(crate) struct PerCpuModeEncodings<'defs> {
     pub inst_pred_reg: InstructionPredicateRegistry,
     pub enc32: Vec,
     pub enc64: Vec,
     recipes: &'defs Recipes,
+    formats: &'defs FormatRegistry,
 }
 
 impl<'defs> PerCpuModeEncodings<'defs> {
-    fn new(recipes: &'defs Recipes) -> Self {
+    fn new(recipes: &'defs Recipes, formats: &'defs FormatRegistry) -> Self {
         Self {
             inst_pred_reg: InstructionPredicateRegistry::new(),
             enc32: Vec::new(),
             enc64: Vec::new(),
             recipes,
+            formats,
         }
     }
+    fn enc(
+        &self,
+        inst: impl Into,
+        recipe: EncodingRecipeNumber,
+        bits: u16,
+    ) -> EncodingBuilder {
+        EncodingBuilder::new(inst.into(), recipe, bits, self.formats)
+    }
     fn add32(&mut self, encoding: EncodingBuilder) {
         self.enc32
             .push(encoding.build(self.recipes, &mut self.inst_pred_reg));
@@ -169,7 +176,7 @@ pub(crate) fn define<'defs>(
     let use_m = isa_settings.predicate_by_name("use_m");
 
     // Definitions.
-    let mut e = PerCpuModeEncodings::new(&recipes.recipes);
+    let mut e = PerCpuModeEncodings::new(&recipes.recipes, &shared_defs.format_registry);
 
     // Basic arithmetic binary instructions are encoded in an R-type instruction.
     for &(inst, inst_imm, f3, f7) in &[
@@ -179,26 +186,26 @@ pub(crate) fn define<'defs>(
         (bor, Some(bor_imm), 0b110, 0b0000000),
         (band, Some(band_imm), 0b111, 0b0000000),
     ] {
-        e.add32(enc(inst.bind(I32), r_r, op_bits(f3, f7)));
-        e.add64(enc(inst.bind(I64), r_r, op_bits(f3, f7)));
+        e.add32(e.enc(inst.bind(I32), r_r, op_bits(f3, f7)));
+        e.add64(e.enc(inst.bind(I64), r_r, op_bits(f3, f7)));
 
         // Immediate versions for add/xor/or/and.
         if let Some(inst_imm) = inst_imm {
-            e.add32(enc(inst_imm.bind(I32), r_ii, opimm_bits(f3, 0)));
-            e.add64(enc(inst_imm.bind(I64), r_ii, opimm_bits(f3, 0)));
+            e.add32(e.enc(inst_imm.bind(I32), r_ii, opimm_bits(f3, 0)));
+            e.add64(e.enc(inst_imm.bind(I64), r_ii, opimm_bits(f3, 0)));
         }
     }
 
     // 32-bit ops in RV64.
-    e.add64(enc(iadd.bind(I32), r_r, op32_bits(0b000, 0b0000000)));
-    e.add64(enc(isub.bind(I32), r_r, op32_bits(0b000, 0b0100000)));
+    e.add64(e.enc(iadd.bind(I32), r_r, op32_bits(0b000, 0b0000000)));
+    e.add64(e.enc(isub.bind(I32), r_r, op32_bits(0b000, 0b0100000)));
     // There are no andiw/oriw/xoriw variations.
-    e.add64(enc(iadd_imm.bind(I32), r_ii, opimm32_bits(0b000, 0)));
+    e.add64(e.enc(iadd_imm.bind(I32), r_ii, opimm32_bits(0b000, 0)));
 
     // Use iadd_imm with %x0 to materialize constants.
-    e.add32(enc(iconst.bind(I32), r_iz, opimm_bits(0b0, 0)));
-    e.add64(enc(iconst.bind(I32), r_iz, opimm_bits(0b0, 0)));
-    e.add64(enc(iconst.bind(I64), r_iz, opimm_bits(0b0, 0)));
+    e.add32(e.enc(iconst.bind(I32), r_iz, opimm_bits(0b0, 0)));
+    e.add64(e.enc(iconst.bind(I32), r_iz, opimm_bits(0b0, 0)));
+    e.add64(e.enc(iconst.bind(I64), r_iz, opimm_bits(0b0, 0)));
 
     // Dynamic shifts have the same masking semantics as the clif base instructions.
     for &(inst, inst_imm, f3, f7) in &[
@@ -206,17 +213,17 @@ pub(crate) fn define<'defs>(
         (ushr, ushr_imm, 0b101, 0b0),
         (sshr, sshr_imm, 0b101, 0b100000),
     ] {
-        e.add32(enc(inst.bind(I32).bind(I32), r_r, op_bits(f3, f7)));
-        e.add64(enc(inst.bind(I64).bind(I64), r_r, op_bits(f3, f7)));
-        e.add64(enc(inst.bind(I32).bind(I32), r_r, op32_bits(f3, f7)));
+        e.add32(e.enc(inst.bind(I32).bind(I32), r_r, op_bits(f3, f7)));
+        e.add64(e.enc(inst.bind(I64).bind(I64), r_r, op_bits(f3, f7)));
+        e.add64(e.enc(inst.bind(I32).bind(I32), r_r, op32_bits(f3, f7)));
         // Allow i32 shift amounts in 64-bit shifts.
-        e.add64(enc(inst.bind(I64).bind(I32), r_r, op_bits(f3, f7)));
-        e.add64(enc(inst.bind(I32).bind(I64), r_r, op32_bits(f3, f7)));
+        e.add64(e.enc(inst.bind(I64).bind(I32), r_r, op_bits(f3, f7)));
+        e.add64(e.enc(inst.bind(I32).bind(I64), r_r, op32_bits(f3, f7)));
 
         // Immediate shifts.
-        e.add32(enc(inst_imm.bind(I32), r_rshamt, opimm_bits(f3, f7)));
-        e.add64(enc(inst_imm.bind(I64), r_rshamt, opimm_bits(f3, f7)));
-        e.add64(enc(inst_imm.bind(I32), r_rshamt, opimm32_bits(f3, f7)));
+        e.add32(e.enc(inst_imm.bind(I32), r_rshamt, opimm_bits(f3, f7)));
+        e.add64(e.enc(inst_imm.bind(I64), r_rshamt, opimm_bits(f3, f7)));
+        e.add64(e.enc(inst_imm.bind(I32), r_rshamt, opimm32_bits(f3, f7)));
     }
 
     // Signed and unsigned integer 'less than'. There are no 'w' variants for comparing 32-bit
@@ -242,20 +249,20 @@ pub(crate) fn define<'defs>(
         let icmp_i32 = icmp.bind(I32);
         let icmp_i64 = icmp.bind(I64);
         e.add32(
-            enc(icmp_i32.clone(), r_ricmp, op_bits(0b010, 0b0000000))
+            e.enc(icmp_i32.clone(), r_ricmp, op_bits(0b010, 0b0000000))
                 .inst_predicate(icmp_instp(&icmp_i32, "slt")),
         );
         e.add64(
-            enc(icmp_i64.clone(), r_ricmp, op_bits(0b010, 0b0000000))
+            e.enc(icmp_i64.clone(), r_ricmp, op_bits(0b010, 0b0000000))
                 .inst_predicate(icmp_instp(&icmp_i64, "slt")),
         );
 
         e.add32(
-            enc(icmp_i32.clone(), r_ricmp, op_bits(0b011, 0b0000000))
+            e.enc(icmp_i32.clone(), r_ricmp, op_bits(0b011, 0b0000000))
                 .inst_predicate(icmp_instp(&icmp_i32, "ult")),
         );
         e.add64(
-            enc(icmp_i64.clone(), r_ricmp, op_bits(0b011, 0b0000000))
+            e.enc(icmp_i64.clone(), r_ricmp, op_bits(0b011, 0b0000000))
                 .inst_predicate(icmp_instp(&icmp_i64, "ult")),
         );
 
@@ -263,42 +270,51 @@ pub(crate) fn define<'defs>(
         let icmp_i32 = icmp_imm.bind(I32);
         let icmp_i64 = icmp_imm.bind(I64);
         e.add32(
-            enc(icmp_i32.clone(), r_iicmp, opimm_bits(0b010, 0))
+            e.enc(icmp_i32.clone(), r_iicmp, opimm_bits(0b010, 0))
                 .inst_predicate(icmp_instp(&icmp_i32, "slt")),
         );
         e.add64(
-            enc(icmp_i64.clone(), r_iicmp, opimm_bits(0b010, 0))
+            e.enc(icmp_i64.clone(), r_iicmp, opimm_bits(0b010, 0))
                 .inst_predicate(icmp_instp(&icmp_i64, "slt")),
         );
 
         e.add32(
-            enc(icmp_i32.clone(), r_iicmp, opimm_bits(0b011, 0))
+            e.enc(icmp_i32.clone(), r_iicmp, opimm_bits(0b011, 0))
                 .inst_predicate(icmp_instp(&icmp_i32, "ult")),
         );
         e.add64(
-            enc(icmp_i64.clone(), r_iicmp, opimm_bits(0b011, 0))
+            e.enc(icmp_i64.clone(), r_iicmp, opimm_bits(0b011, 0))
                 .inst_predicate(icmp_instp(&icmp_i64, "ult")),
         );
     }
 
     // Integer constants with the low 12 bits clear are materialized by lui.
-    e.add32(enc(iconst.bind(I32), r_u, lui_bits()));
-    e.add64(enc(iconst.bind(I32), r_u, lui_bits()));
-    e.add64(enc(iconst.bind(I64), r_u, lui_bits()));
+    e.add32(e.enc(iconst.bind(I32), r_u, lui_bits()));
+    e.add64(e.enc(iconst.bind(I32), r_u, lui_bits()));
+    e.add64(e.enc(iconst.bind(I64), r_u, lui_bits()));
 
     // "M" Standard Extension for Integer Multiplication and Division.
     // Gated by the `use_m` flag.
-    e.add32(enc(imul.bind(I32), r_r, op_bits(0b000, 0b00000001)).isa_predicate(use_m));
-    e.add64(enc(imul.bind(I64), r_r, op_bits(0b000, 0b00000001)).isa_predicate(use_m));
-    e.add64(enc(imul.bind(I32), r_r, op32_bits(0b000, 0b00000001)).isa_predicate(use_m));
+    e.add32(
+        e.enc(imul.bind(I32), r_r, op_bits(0b000, 0b00000001))
+            .isa_predicate(use_m),
+    );
+    e.add64(
+        e.enc(imul.bind(I64), r_r, op_bits(0b000, 0b00000001))
+            .isa_predicate(use_m),
+    );
+    e.add64(
+        e.enc(imul.bind(I32), r_r, op32_bits(0b000, 0b00000001))
+            .isa_predicate(use_m),
+    );
 
     // Control flow.
 
     // Unconditional branches.
-    e.add32(enc(jump, r_uj, jal_bits()));
-    e.add64(enc(jump, r_uj, jal_bits()));
-    e.add32(enc(call, r_uj_call, jal_bits()));
-    e.add64(enc(call, r_uj_call, jal_bits()));
+    e.add32(e.enc(jump, r_uj, jal_bits()));
+    e.add64(e.enc(jump, r_uj, jal_bits()));
+    e.add32(e.enc(call, r_uj_call, jal_bits()));
+    e.add64(e.enc(call, r_uj_call, jal_bits()));
 
     // Conditional branches.
     {
@@ -338,101 +354,81 @@ pub(crate) fn define<'defs>(
             ("uge", 0b111),
         ] {
             e.add32(
-                enc(br_icmp_i32.clone(), r_sb, branch_bits(f3))
+                e.enc(br_icmp_i32.clone(), r_sb, branch_bits(f3))
                     .inst_predicate(br_icmp_instp(&br_icmp_i32, cond)),
             );
             e.add64(
-                enc(br_icmp_i64.clone(), r_sb, branch_bits(f3))
+                e.enc(br_icmp_i64.clone(), r_sb, branch_bits(f3))
                     .inst_predicate(br_icmp_instp(&br_icmp_i64, cond)),
             );
         }
     }
 
     for &(inst, f3) in &[(brz, 0b000), (brnz, 0b001)] {
-        e.add32(enc(inst.bind(I32), r_sb_zero, branch_bits(f3)));
-        e.add64(enc(inst.bind(I64), r_sb_zero, branch_bits(f3)));
-        e.add32(enc(inst.bind(B1), r_sb_zero, branch_bits(f3)));
-        e.add64(enc(inst.bind(B1), r_sb_zero, branch_bits(f3)));
+        e.add32(e.enc(inst.bind(I32), r_sb_zero, branch_bits(f3)));
+        e.add64(e.enc(inst.bind(I64), r_sb_zero, branch_bits(f3)));
+        e.add32(e.enc(inst.bind(B1), r_sb_zero, branch_bits(f3)));
+        e.add64(e.enc(inst.bind(B1), r_sb_zero, branch_bits(f3)));
     }
 
     // Returns are a special case of jalr_bits using %x1 to hold the return address.
     // The return address is provided by a special-purpose `link` return value that
     // is added by legalize_signature().
-    e.add32(enc(return_, r_iret, jalr_bits()));
-    e.add64(enc(return_, r_iret, jalr_bits()));
-    e.add32(enc(call_indirect.bind(I32), r_icall, jalr_bits()));
-    e.add64(enc(call_indirect.bind(I64), r_icall, jalr_bits()));
+    e.add32(e.enc(return_, r_iret, jalr_bits()));
+    e.add64(e.enc(return_, r_iret, jalr_bits()));
+    e.add32(e.enc(call_indirect.bind(I32), r_icall, jalr_bits()));
+    e.add64(e.enc(call_indirect.bind(I64), r_icall, jalr_bits()));
 
     // Spill and fill.
-    e.add32(enc(spill.bind(I32), r_gp_sp, store_bits(0b010)));
-    e.add64(enc(spill.bind(I32), r_gp_sp, store_bits(0b010)));
-    e.add64(enc(spill.bind(I64), r_gp_sp, store_bits(0b011)));
-    e.add32(enc(fill.bind(I32), r_gp_fi, load_bits(0b010)));
-    e.add64(enc(fill.bind(I32), r_gp_fi, load_bits(0b010)));
-    e.add64(enc(fill.bind(I64), r_gp_fi, load_bits(0b011)));
+    e.add32(e.enc(spill.bind(I32), r_gp_sp, store_bits(0b010)));
+    e.add64(e.enc(spill.bind(I32), r_gp_sp, store_bits(0b010)));
+    e.add64(e.enc(spill.bind(I64), r_gp_sp, store_bits(0b011)));
+    e.add32(e.enc(fill.bind(I32), r_gp_fi, load_bits(0b010)));
+    e.add64(e.enc(fill.bind(I32), r_gp_fi, load_bits(0b010)));
+    e.add64(e.enc(fill.bind(I64), r_gp_fi, load_bits(0b011)));
 
     // No-op fills, created by late-stage redundant-fill removal.
     for &ty in &[I64, I32] {
-        e.add64(enc(fill_nop.bind(ty), r_fillnull, 0));
-        e.add32(enc(fill_nop.bind(ty), r_fillnull, 0));
+        e.add64(e.enc(fill_nop.bind(ty), r_fillnull, 0));
+        e.add32(e.enc(fill_nop.bind(ty), r_fillnull, 0));
     }
-    e.add64(enc(fill_nop.bind(B1), r_fillnull, 0));
-    e.add32(enc(fill_nop.bind(B1), r_fillnull, 0));
+    e.add64(e.enc(fill_nop.bind(B1), r_fillnull, 0));
+    e.add32(e.enc(fill_nop.bind(B1), r_fillnull, 0));
 
     // Register copies.
-    e.add32(enc(copy.bind(I32), r_icopy, opimm_bits(0b000, 0)));
-    e.add64(enc(copy.bind(I64), r_icopy, opimm_bits(0b000, 0)));
-    e.add64(enc(copy.bind(I32), r_icopy, opimm32_bits(0b000, 0)));
+    e.add32(e.enc(copy.bind(I32), r_icopy, opimm_bits(0b000, 0)));
+    e.add64(e.enc(copy.bind(I64), r_icopy, opimm_bits(0b000, 0)));
+    e.add64(e.enc(copy.bind(I32), r_icopy, opimm32_bits(0b000, 0)));
 
-    e.add32(enc(regmove.bind(I32), r_irmov, opimm_bits(0b000, 0)));
-    e.add64(enc(regmove.bind(I64), r_irmov, opimm_bits(0b000, 0)));
-    e.add64(enc(regmove.bind(I32), r_irmov, opimm32_bits(0b000, 0)));
+    e.add32(e.enc(regmove.bind(I32), r_irmov, opimm_bits(0b000, 0)));
+    e.add64(e.enc(regmove.bind(I64), r_irmov, opimm_bits(0b000, 0)));
+    e.add64(e.enc(regmove.bind(I32), r_irmov, opimm32_bits(0b000, 0)));
 
-    e.add32(enc(copy.bind(B1), r_icopy, opimm_bits(0b000, 0)));
-    e.add64(enc(copy.bind(B1), r_icopy, opimm_bits(0b000, 0)));
-    e.add32(enc(regmove.bind(B1), r_irmov, opimm_bits(0b000, 0)));
-    e.add64(enc(regmove.bind(B1), r_irmov, opimm_bits(0b000, 0)));
+    e.add32(e.enc(copy.bind(B1), r_icopy, opimm_bits(0b000, 0)));
+    e.add64(e.enc(copy.bind(B1), r_icopy, opimm_bits(0b000, 0)));
+    e.add32(e.enc(regmove.bind(B1), r_irmov, opimm_bits(0b000, 0)));
+    e.add64(e.enc(regmove.bind(B1), r_irmov, opimm_bits(0b000, 0)));
 
     // Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
     // into a no-op.
     // The same encoding is generated for both the 64- and 32-bit architectures.
     for &ty in &[I64, I32, I16, I8] {
-        e.add32(enc(copy_nop.bind(ty), r_stacknull, 0));
-        e.add64(enc(copy_nop.bind(ty), r_stacknull, 0));
+        e.add32(e.enc(copy_nop.bind(ty), r_stacknull, 0));
+        e.add64(e.enc(copy_nop.bind(ty), r_stacknull, 0));
     }
     for &ty in &[F64, F32] {
-        e.add32(enc(copy_nop.bind(ty), r_stacknull, 0));
-        e.add64(enc(copy_nop.bind(ty), r_stacknull, 0));
+        e.add32(e.enc(copy_nop.bind(ty), r_stacknull, 0));
+        e.add64(e.enc(copy_nop.bind(ty), r_stacknull, 0));
     }
 
     // Copy-to-SSA
-    e.add32(enc(
-        copy_to_ssa.bind(I32),
-        r_copytossa,
-        opimm_bits(0b000, 0),
-    ));
-    e.add64(enc(
-        copy_to_ssa.bind(I64),
-        r_copytossa,
-        opimm_bits(0b000, 0),
-    ));
-    e.add64(enc(
-        copy_to_ssa.bind(I32),
-        r_copytossa,
-        opimm32_bits(0b000, 0),
-    ));
-    e.add32(enc(copy_to_ssa.bind(B1), r_copytossa, opimm_bits(0b000, 0)));
-    e.add64(enc(copy_to_ssa.bind(B1), r_copytossa, opimm_bits(0b000, 0)));
-    e.add32(enc(
-        copy_to_ssa.bind_ref(R32),
-        r_copytossa,
-        opimm_bits(0b000, 0),
-    ));
-    e.add64(enc(
-        copy_to_ssa.bind_ref(R64),
-        r_copytossa,
-        opimm_bits(0b000, 0),
-    ));
+    e.add32(e.enc(copy_to_ssa.bind(I32), r_copytossa, opimm_bits(0b000, 0)));
+    e.add64(e.enc(copy_to_ssa.bind(I64), r_copytossa, opimm_bits(0b000, 0)));
+    e.add64(e.enc(copy_to_ssa.bind(I32), r_copytossa, opimm32_bits(0b000, 0)));
+    e.add32(e.enc(copy_to_ssa.bind(B1), r_copytossa, opimm_bits(0b000, 0)));
+    e.add64(e.enc(copy_to_ssa.bind(B1), r_copytossa, opimm_bits(0b000, 0)));
+    e.add32(e.enc(copy_to_ssa.bind(R32), r_copytossa, opimm_bits(0b000, 0)));
+    e.add64(e.enc(copy_to_ssa.bind(R64), r_copytossa, opimm_bits(0b000, 0)));
 
     e
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index da08462e9a..c9b1ed8b42 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -5,8 +5,8 @@ use std::collections::HashMap;
 
 use crate::cdsl::encodings::{Encoding, EncodingBuilder};
 use crate::cdsl::instructions::{
-    InstSpec, Instruction, InstructionGroup, InstructionPredicate, InstructionPredicateNode,
-    InstructionPredicateRegistry,
+    vector, Bindable, InstSpec, Instruction, InstructionGroup, InstructionPredicate,
+    InstructionPredicateNode, InstructionPredicateRegistry,
 };
 use crate::cdsl::recipes::{EncodingRecipe, EncodingRecipeNumber, Recipes};
 use crate::cdsl::settings::{SettingGroup, SettingPredicateNumber};
@@ -20,23 +20,27 @@ use crate::shared::Definitions as SharedDefinitions;
 use crate::isa::x86::opcodes::*;
 
 use super::recipes::{RecipeGroup, Template};
+use crate::cdsl::formats::FormatRegistry;
+use crate::cdsl::instructions::BindParameter::Any;
 
-pub(crate) struct PerCpuModeEncodings {
+pub(crate) struct PerCpuModeEncodings<'defs> {
     pub enc32: Vec,
     pub enc64: Vec,
     pub recipes: Recipes,
     recipes_by_name: HashMap,
     pub inst_pred_reg: InstructionPredicateRegistry,
+    formats: &'defs FormatRegistry,
 }
 
-impl PerCpuModeEncodings {
-    fn new() -> Self {
+impl<'defs> PerCpuModeEncodings<'defs> {
+    fn new(formats: &'defs FormatRegistry) -> Self {
         Self {
             enc32: Vec::new(),
             enc64: Vec::new(),
             recipes: Recipes::new(),
             recipes_by_name: HashMap::new(),
             inst_pred_reg: InstructionPredicateRegistry::new(),
+            formats,
         }
     }
 
@@ -69,7 +73,7 @@ impl PerCpuModeEncodings {
     {
         let (recipe, bits) = template.build();
         let recipe_number = self.add_recipe(recipe);
-        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits);
+        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits, self.formats);
         builder_closure(builder).build(&self.recipes, &mut self.inst_pred_reg)
     }
 
@@ -101,7 +105,7 @@ impl PerCpuModeEncodings {
     }
     fn enc32_rec(&mut self, inst: impl Into, recipe: &EncodingRecipe, bits: u16) {
         let recipe_number = self.add_recipe(recipe.clone());
-        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits);
+        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits, self.formats);
         let encoding = builder.build(&self.recipes, &mut self.inst_pred_reg);
         self.enc32.push(encoding);
     }
@@ -134,7 +138,7 @@ impl PerCpuModeEncodings {
     }
     fn enc64_rec(&mut self, inst: impl Into, recipe: &EncodingRecipe, bits: u16) {
         let recipe_number = self.add_recipe(recipe.clone());
-        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits);
+        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits, self.formats);
         let encoding = builder.build(&self.recipes, &mut self.inst_pred_reg);
         self.enc64.push(encoding);
     }
@@ -207,8 +211,8 @@ impl PerCpuModeEncodings {
     /// Add encodings for `inst.r64` to X86_64 with a REX.W prefix.
     fn enc_r32_r64_rex_only(&mut self, inst: impl Into, template: Template) {
         let inst: InstSpec = inst.into();
-        self.enc32(inst.bind_ref(R32), template.nonrex());
-        self.enc64(inst.bind_ref(R64), template.rex().w());
+        self.enc32(inst.bind(R32), template.nonrex());
+        self.enc64(inst.bind(R64), template.rex().w());
     }
 
     /// Add encodings for `inst` to X86_64 with and without a REX prefix.
@@ -281,18 +285,18 @@ impl PerCpuModeEncodings {
     /// Add encodings for `inst.i64` to X86_64 with a REX prefix, using the `w_bit`
     /// argument to determine whether or not to set the REX.W bit.
     fn enc_i32_i64_ld_st(&mut self, inst: &Instruction, w_bit: bool, template: Template) {
-        self.enc32(inst.clone().bind(I32).bind_any(), template.clone());
+        self.enc32(inst.clone().bind(I32).bind(Any), template.clone());
 
         // REX-less encoding must come after REX encoding so we don't use it by
         // default. Otherwise reg-alloc would never use r8 and up.
-        self.enc64(inst.clone().bind(I32).bind_any(), template.clone().rex());
-        self.enc64(inst.clone().bind(I32).bind_any(), template.clone());
+        self.enc64(inst.clone().bind(I32).bind(Any), template.clone().rex());
+        self.enc64(inst.clone().bind(I32).bind(Any), template.clone());
 
         if w_bit {
-            self.enc64(inst.clone().bind(I64).bind_any(), template.rex().w());
+            self.enc64(inst.clone().bind(I64).bind(Any), template.rex().w());
         } else {
-            self.enc64(inst.clone().bind(I64).bind_any(), template.clone().rex());
-            self.enc64(inst.clone().bind(I64).bind_any(), template);
+            self.enc64(inst.clone().bind(I64).bind(Any), template.clone().rex());
+            self.enc64(inst.clone().bind(I64).bind(Any), template);
         }
     }
 
@@ -366,12 +370,12 @@ impl PerCpuModeEncodings {
 
 // Definitions.
 
-pub(crate) fn define(
-    shared_defs: &SharedDefinitions,
+pub(crate) fn define<'defs>(
+    shared_defs: &'defs SharedDefinitions,
     settings: &SettingGroup,
     x86: &InstructionGroup,
     r: &RecipeGroup,
-) -> PerCpuModeEncodings {
+) -> PerCpuModeEncodings<'defs> {
     let shared = &shared_defs.instructions;
     let formats = &shared_defs.format_registry;
 
@@ -681,7 +685,7 @@ pub(crate) fn define(
     let use_sse41_simd = settings.predicate_by_name("use_sse41_simd");
 
     // Definitions.
-    let mut e = PerCpuModeEncodings::new();
+    let mut e = PerCpuModeEncodings::new(formats);
 
     // The pinned reg is fixed to a certain value entirely user-controlled, so it generates nothing!
     e.enc64_rec(get_pinned_reg.bind(I64), rec_get_pinned_reg, 0);
@@ -742,15 +746,11 @@ pub(crate) fn define(
         e.enc64(regmove.bind(ty), rec_rmov.opcodes(&MOV_STORE).rex());
     }
     e.enc64(regmove.bind(I64), rec_rmov.opcodes(&MOV_STORE).rex().w());
-    e.enc64(regmove.bind(B64), rec_rmov.opcodes(&MOV_STORE).rex().w());
     e.enc_both(regmove.bind(B1), rec_rmov.opcodes(&MOV_STORE));
     e.enc_both(regmove.bind(I8), rec_rmov.opcodes(&MOV_STORE));
-    e.enc32(regmove.bind_ref(R32), rec_rmov.opcodes(&MOV_STORE));
-    e.enc64(regmove.bind_ref(R32), rec_rmov.opcodes(&MOV_STORE).rex());
-    e.enc64(
-        regmove.bind_ref(R64),
-        rec_rmov.opcodes(&MOV_STORE).rex().w(),
-    );
+    e.enc32(regmove.bind(R32), rec_rmov.opcodes(&MOV_STORE));
+    e.enc64(regmove.bind(R32), rec_rmov.opcodes(&MOV_STORE).rex());
+    e.enc64(regmove.bind(R64), rec_rmov.opcodes(&MOV_STORE).rex().w());
 
     e.enc_i32_i64(iadd_imm, rec_r_ib.opcodes(&ADD_IMM8_SIGN_EXTEND).rrr(0));
     e.enc_i32_i64(iadd_imm, rec_r_id.opcodes(&ADD_IMM).rrr(0));
@@ -834,19 +834,19 @@ pub(crate) fn define(
         // Cannot use enc_i32_i64 for this pattern because instructions require
         // to bind any.
         e.enc32(
-            inst.bind(I32).bind_any(),
+            inst.bind(I32).bind(Any),
             rec_rc.opcodes(&ROTATE_CL).rrr(rrr),
         );
         e.enc64(
-            inst.bind(I64).bind_any(),
+            inst.bind(I64).bind(Any),
             rec_rc.opcodes(&ROTATE_CL).rrr(rrr).rex().w(),
         );
         e.enc64(
-            inst.bind(I32).bind_any(),
+            inst.bind(I32).bind(Any),
             rec_rc.opcodes(&ROTATE_CL).rrr(rrr).rex(),
         );
         e.enc64(
-            inst.bind(I32).bind_any(),
+            inst.bind(I32).bind(Any),
             rec_rc.opcodes(&ROTATE_CL).rrr(rrr),
         );
     }
@@ -970,7 +970,7 @@ pub(crate) fn define(
 
     for recipe in &[rec_st, rec_stDisp8, rec_stDisp32] {
         e.enc_i32_i64_ld_st(store, true, recipe.opcodes(&MOV_STORE));
-        e.enc_x86_64(istore32.bind(I64).bind_any(), recipe.opcodes(&MOV_STORE));
+        e.enc_x86_64(istore32.bind(I64).bind(Any), recipe.opcodes(&MOV_STORE));
         e.enc_i32_i64_ld_st(istore16, false, recipe.opcodes(&MOV_STORE_16));
     }
 
@@ -979,14 +979,8 @@ pub(crate) fn define(
     // the corresponding st* recipes when a REX prefix is applied.
 
     for recipe in &[rec_st_abcd, rec_stDisp8_abcd, rec_stDisp32_abcd] {
-        e.enc_both(
-            istore8.bind(I32).bind_any(),
-            recipe.opcodes(&MOV_BYTE_STORE),
-        );
-        e.enc_x86_64(
-            istore8.bind(I64).bind_any(),
-            recipe.opcodes(&MOV_BYTE_STORE),
-        );
+        e.enc_both(istore8.bind(I32).bind(Any), recipe.opcodes(&MOV_BYTE_STORE));
+        e.enc_x86_64(istore8.bind(I64).bind(Any), recipe.opcodes(&MOV_BYTE_STORE));
     }
 
     e.enc_i32_i64(spill, rec_spillSib32.opcodes(&MOV_STORE));
@@ -1121,12 +1115,9 @@ pub(crate) fn define(
     );
 
     // Float loads and stores.
-    e.enc_both(load.bind(F32).bind_any(), rec_fld.opcodes(&MOVSS_LOAD));
-    e.enc_both(load.bind(F32).bind_any(), rec_fldDisp8.opcodes(&MOVSS_LOAD));
-    e.enc_both(
-        load.bind(F32).bind_any(),
-        rec_fldDisp32.opcodes(&MOVSS_LOAD),
-    );
+    e.enc_both(load.bind(F32).bind(Any), rec_fld.opcodes(&MOVSS_LOAD));
+    e.enc_both(load.bind(F32).bind(Any), rec_fldDisp8.opcodes(&MOVSS_LOAD));
+    e.enc_both(load.bind(F32).bind(Any), rec_fldDisp32.opcodes(&MOVSS_LOAD));
 
     e.enc_both(
         load_complex.bind(F32),
@@ -1141,12 +1132,9 @@ pub(crate) fn define(
         rec_fldWithIndexDisp32.opcodes(&MOVSS_LOAD),
     );
 
-    e.enc_both(load.bind(F64).bind_any(), rec_fld.opcodes(&MOVSD_LOAD));
-    e.enc_both(load.bind(F64).bind_any(), rec_fldDisp8.opcodes(&MOVSD_LOAD));
-    e.enc_both(
-        load.bind(F64).bind_any(),
-        rec_fldDisp32.opcodes(&MOVSD_LOAD),
-    );
+    e.enc_both(load.bind(F64).bind(Any), rec_fld.opcodes(&MOVSD_LOAD));
+    e.enc_both(load.bind(F64).bind(Any), rec_fldDisp8.opcodes(&MOVSD_LOAD));
+    e.enc_both(load.bind(F64).bind(Any), rec_fldDisp32.opcodes(&MOVSD_LOAD));
 
     e.enc_both(
         load_complex.bind(F64),
@@ -1161,13 +1149,13 @@ pub(crate) fn define(
         rec_fldWithIndexDisp32.opcodes(&MOVSD_LOAD),
     );
 
-    e.enc_both(store.bind(F32).bind_any(), rec_fst.opcodes(&MOVSS_STORE));
+    e.enc_both(store.bind(F32).bind(Any), rec_fst.opcodes(&MOVSS_STORE));
     e.enc_both(
-        store.bind(F32).bind_any(),
+        store.bind(F32).bind(Any),
         rec_fstDisp8.opcodes(&MOVSS_STORE),
     );
     e.enc_both(
-        store.bind(F32).bind_any(),
+        store.bind(F32).bind(Any),
         rec_fstDisp32.opcodes(&MOVSS_STORE),
     );
 
@@ -1184,13 +1172,13 @@ pub(crate) fn define(
         rec_fstWithIndexDisp32.opcodes(&MOVSS_STORE),
     );
 
-    e.enc_both(store.bind(F64).bind_any(), rec_fst.opcodes(&MOVSD_STORE));
+    e.enc_both(store.bind(F64).bind(Any), rec_fst.opcodes(&MOVSD_STORE));
     e.enc_both(
-        store.bind(F64).bind_any(),
+        store.bind(F64).bind(Any),
         rec_fstDisp8.opcodes(&MOVSD_STORE),
     );
     e.enc_both(
-        store.bind(F64).bind_any(),
+        store.bind(F64).bind(Any),
         rec_fstDisp32.opcodes(&MOVSD_STORE),
     );
 
@@ -1727,7 +1715,7 @@ pub(crate) fn define(
 
     // PSHUFB, 8-bit shuffle using two XMM registers.
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
-        let instruction = x86_pshufb.bind_vector_from_lane(ty, sse_vector_size);
+        let instruction = x86_pshufb.bind(vector(ty, sse_vector_size));
         let template = rec_fa.nonrex().opcodes(&PSHUFB);
         e.enc32_isap(instruction.clone(), template.clone(), use_ssse3_simd);
         e.enc64_isap(instruction, template, use_ssse3_simd);
@@ -1735,7 +1723,7 @@ pub(crate) fn define(
 
     // PSHUFD, 32-bit shuffle using one XMM register and a u8 immediate.
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 32) {
-        let instruction = x86_pshufd.bind_vector_from_lane(ty, sse_vector_size);
+        let instruction = x86_pshufd.bind(vector(ty, sse_vector_size));
         let template = rec_r_ib_unsigned_fpr.nonrex().opcodes(&PSHUFD);
         e.enc32(instruction.clone(), template.clone());
         e.enc64(instruction, template);
@@ -1745,7 +1733,7 @@ pub(crate) fn define(
     // to the Intel manual: "When the destination operand is an XMM register, the source operand is
     // written to the low doubleword of the register and the register is zero-extended to 128 bits."
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
-        let instruction = scalar_to_vector.bind_vector_from_lane(ty, sse_vector_size);
+        let instruction = scalar_to_vector.bind(vector(ty, sse_vector_size));
         if ty.is_float() {
             e.enc_32_64_rec(instruction, rec_null_fpr, 0);
         } else {
@@ -1767,7 +1755,7 @@ pub(crate) fn define(
             _ => panic!("invalid size for SIMD insertlane"),
         };
 
-        let instruction = x86_pinsr.bind_vector_from_lane(ty, sse_vector_size);
+        let instruction = x86_pinsr.bind(vector(ty, sse_vector_size));
         let template = rec_r_ib_unsigned_r.opcodes(opcode);
         if ty.lane_bits() < 64 {
             e.enc_32_64_maybe_isap(instruction, template.nonrex(), isap);
@@ -1780,21 +1768,21 @@ pub(crate) fn define(
 
     // For legalizing insertlane with floats, INSERTPS from SSE4.1.
     {
-        let instruction = x86_insertps.bind_vector_from_lane(F32, sse_vector_size);
+        let instruction = x86_insertps.bind(vector(F32, sse_vector_size));
         let template = rec_fa_ib.nonrex().opcodes(&INSERTPS);
         e.enc_32_64_maybe_isap(instruction, template, Some(use_sse41_simd));
     }
 
     // For legalizing insertlane with floats,  MOVSD from SSE2.
     {
-        let instruction = x86_movsd.bind_vector_from_lane(F64, sse_vector_size);
+        let instruction = x86_movsd.bind(vector(F64, sse_vector_size));
         let template = rec_fa.nonrex().opcodes(&MOVSD_LOAD);
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE2
     }
 
     // For legalizing insertlane with floats, MOVLHPS from SSE.
     {
-        let instruction = x86_movlhps.bind_vector_from_lane(F64, sse_vector_size);
+        let instruction = x86_movlhps.bind(vector(F64, sse_vector_size));
         let template = rec_fa.nonrex().opcodes(&MOVLHPS);
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
     }
@@ -1808,7 +1796,7 @@ pub(crate) fn define(
             _ => panic!("invalid size for SIMD extractlane"),
         };
 
-        let instruction = x86_pextr.bind_vector_from_lane(ty, sse_vector_size);
+        let instruction = x86_pextr.bind(vector(ty, sse_vector_size));
         let template = rec_r_ib_unsigned_gpr.opcodes(opcode);
         if ty.lane_bits() < 64 {
             e.enc_32_64_maybe_isap(instruction, template.nonrex(), Some(use_sse41_simd));
@@ -1825,8 +1813,8 @@ pub(crate) fn define(
             ValueType::all_lane_types().filter(|t| allowed_simd_type(t) && *t != from_type)
         {
             let instruction = raw_bitcast
-                .bind_vector_from_lane(to_type, sse_vector_size)
-                .bind_vector_from_lane(from_type, sse_vector_size);
+                .bind(vector(to_type, sse_vector_size))
+                .bind(vector(from_type, sse_vector_size));
             e.enc_32_64_rec(instruction, rec_null_fpr, 0);
         }
     }
@@ -1837,7 +1825,7 @@ pub(crate) fn define(
         for lane_type in ValueType::all_lane_types().filter(allowed_simd_type) {
             e.enc_32_64_rec(
                 raw_bitcast
-                    .bind_vector_from_lane(lane_type, sse_vector_size)
+                    .bind(vector(lane_type, sse_vector_size))
                     .bind(*float_type),
                 rec_null_fpr,
                 0,
@@ -1845,7 +1833,7 @@ pub(crate) fn define(
             e.enc_32_64_rec(
                 raw_bitcast
                     .bind(*float_type)
-                    .bind_vector_from_lane(lane_type, sse_vector_size),
+                    .bind(vector(lane_type, sse_vector_size)),
                 rec_null_fpr,
                 0,
             );
@@ -1857,7 +1845,7 @@ pub(crate) fn define(
     // encoding first
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         let f_unary_const = formats.get(formats.by_name("UnaryConst"));
-        let instruction = vconst.bind_vector_from_lane(ty, sse_vector_size);
+        let instruction = vconst.bind(vector(ty, sse_vector_size));
 
         let is_zero_128bit =
             InstructionPredicate::new_is_all_zeroes_128bit(f_unary_const, "constant_handle");
@@ -1881,14 +1869,14 @@ pub(crate) fn define(
     // MOVQ + MOVHPD + MOVQ + MOVLPD (this allows the constants to be immediates instead of stored
     // in memory) but some performance measurements are needed.
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
-        let instruction = vconst.bind_vector_from_lane(ty, sse_vector_size);
+        let instruction = vconst.bind(vector(ty, sse_vector_size));
         let template = rec_vconst.nonrex().opcodes(&MOVUPS_LOAD);
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
     }
 
     // SIMD bor using ORPS
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
-        let instruction = bor.bind_vector_from_lane(ty, sse_vector_size);
+        let instruction = bor.bind(vector(ty, sse_vector_size));
         let template = rec_fa.nonrex().opcodes(&ORPS);
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
     }
@@ -1898,87 +1886,87 @@ pub(crate) fn define(
     // alignment or type-specific encodings, see https://github.com/CraneStation/cranelift/issues/1039).
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         // Store
-        let bound_store = store.bind_vector_from_lane(ty, sse_vector_size).bind_any();
+        let bound_store = store.bind(vector(ty, sse_vector_size)).bind(Any);
         e.enc_32_64(bound_store.clone(), rec_fst.opcodes(&MOVUPS_STORE));
         e.enc_32_64(bound_store.clone(), rec_fstDisp8.opcodes(&MOVUPS_STORE));
         e.enc_32_64(bound_store, rec_fstDisp32.opcodes(&MOVUPS_STORE));
 
         // Load
-        let bound_load = load.bind_vector_from_lane(ty, sse_vector_size).bind_any();
+        let bound_load = load.bind(vector(ty, sse_vector_size)).bind(Any);
         e.enc_32_64(bound_load.clone(), rec_fld.opcodes(&MOVUPS_LOAD));
         e.enc_32_64(bound_load.clone(), rec_fldDisp8.opcodes(&MOVUPS_LOAD));
         e.enc_32_64(bound_load, rec_fldDisp32.opcodes(&MOVUPS_LOAD));
 
         // Spill
-        let bound_spill = spill.bind_vector_from_lane(ty, sse_vector_size);
+        let bound_spill = spill.bind(vector(ty, sse_vector_size));
         e.enc_32_64(bound_spill, rec_fspillSib32.opcodes(&MOVUPS_STORE));
-        let bound_regspill = regspill.bind_vector_from_lane(ty, sse_vector_size);
+        let bound_regspill = regspill.bind(vector(ty, sse_vector_size));
         e.enc_32_64(bound_regspill, rec_fregspill32.opcodes(&MOVUPS_STORE));
 
         // Fill
-        let bound_fill = fill.bind_vector_from_lane(ty, sse_vector_size);
+        let bound_fill = fill.bind(vector(ty, sse_vector_size));
         e.enc_32_64(bound_fill, rec_ffillSib32.opcodes(&MOVUPS_LOAD));
-        let bound_regfill = regfill.bind_vector_from_lane(ty, sse_vector_size);
+        let bound_regfill = regfill.bind(vector(ty, sse_vector_size));
         e.enc_32_64(bound_regfill, rec_fregfill32.opcodes(&MOVUPS_LOAD));
-        let bound_fill_nop = fill_nop.bind_vector_from_lane(ty, sse_vector_size);
+        let bound_fill_nop = fill_nop.bind(vector(ty, sse_vector_size));
         e.enc_32_64_rec(bound_fill_nop, rec_ffillnull, 0);
 
         // Regmove
-        let bound_regmove = regmove.bind_vector_from_lane(ty, sse_vector_size);
+        let bound_regmove = regmove.bind(vector(ty, sse_vector_size));
         e.enc_32_64(bound_regmove, rec_frmov.opcodes(&MOVAPS_LOAD));
 
         // Copy
-        let bound_copy = copy.bind_vector_from_lane(ty, sse_vector_size);
+        let bound_copy = copy.bind(vector(ty, sse_vector_size));
         e.enc_32_64(bound_copy, rec_furm.opcodes(&MOVAPS_LOAD));
-        let bound_copy_nop = copy_nop.bind_vector_from_lane(ty, sse_vector_size);
+        let bound_copy_nop = copy_nop.bind(vector(ty, sse_vector_size));
         e.enc_32_64_rec(bound_copy_nop, rec_stacknull, 0);
     }
 
     // SIMD integer addition
     for (ty, opcodes) in &[(I8, &PADDB), (I16, &PADDW), (I32, &PADDD), (I64, &PADDQ)] {
-        let iadd = iadd.bind_vector_from_lane(ty.clone(), sse_vector_size);
+        let iadd = iadd.bind(vector(ty.clone(), sse_vector_size));
         e.enc_32_64(iadd, rec_fa.opcodes(*opcodes));
     }
 
     // SIMD integer saturating addition
     e.enc_32_64(
-        sadd_sat.bind_vector_from_lane(I8, sse_vector_size),
+        sadd_sat.bind(vector(I8, sse_vector_size)),
         rec_fa.opcodes(&PADDSB),
     );
     e.enc_32_64(
-        sadd_sat.bind_vector_from_lane(I16, sse_vector_size),
+        sadd_sat.bind(vector(I16, sse_vector_size)),
         rec_fa.opcodes(&PADDSW),
     );
     e.enc_32_64(
-        uadd_sat.bind_vector_from_lane(I8, sse_vector_size),
+        uadd_sat.bind(vector(I8, sse_vector_size)),
         rec_fa.opcodes(&PADDUSB),
     );
     e.enc_32_64(
-        uadd_sat.bind_vector_from_lane(I16, sse_vector_size),
+        uadd_sat.bind(vector(I16, sse_vector_size)),
         rec_fa.opcodes(&PADDUSW),
     );
 
     // SIMD integer subtraction
     for (ty, opcodes) in &[(I8, &PSUBB), (I16, &PSUBW), (I32, &PSUBD), (I64, &PSUBQ)] {
-        let isub = isub.bind_vector_from_lane(ty.clone(), sse_vector_size);
+        let isub = isub.bind(vector(ty.clone(), sse_vector_size));
         e.enc_32_64(isub, rec_fa.opcodes(*opcodes));
     }
 
     // SIMD integer saturating subtraction
     e.enc_32_64(
-        ssub_sat.bind_vector_from_lane(I8, sse_vector_size),
+        ssub_sat.bind(vector(I8, sse_vector_size)),
         rec_fa.opcodes(&PSUBSB),
     );
     e.enc_32_64(
-        ssub_sat.bind_vector_from_lane(I16, sse_vector_size),
+        ssub_sat.bind(vector(I16, sse_vector_size)),
         rec_fa.opcodes(&PSUBSW),
     );
     e.enc_32_64(
-        usub_sat.bind_vector_from_lane(I8, sse_vector_size),
+        usub_sat.bind(vector(I8, sse_vector_size)),
         rec_fa.opcodes(&PSUBUSB),
     );
     e.enc_32_64(
-        usub_sat.bind_vector_from_lane(I16, sse_vector_size),
+        usub_sat.bind(vector(I16, sse_vector_size)),
         rec_fa.opcodes(&PSUBUSW),
     );
 
@@ -1988,7 +1976,7 @@ pub(crate) fn define(
         (I16, &PMULLW[..], None),
         (I32, &PMULLD[..], Some(use_sse41_simd)),
     ] {
-        let imul = imul.bind_vector_from_lane(ty.clone(), sse_vector_size);
+        let imul = imul.bind(vector(ty.clone(), sse_vector_size));
         e.enc_32_64_maybe_isap(imul, rec_fa.opcodes(opcodes), *isap);
     }
 
@@ -2002,7 +1990,7 @@ pub(crate) fn define(
             _ => panic!("invalid size for SIMD icmp"),
         };
 
-        let instruction = icmp.bind_vector_from_lane(ty, sse_vector_size);
+        let instruction = icmp.bind(vector(ty, sse_vector_size));
         let f_int_compare = formats.get(formats.by_name("IntCompare"));
         let has_eq_condition_code =
             InstructionPredicate::new_has_condition_code(f_int_compare, IntCC::Equal, "cond");
@@ -2020,10 +2008,10 @@ pub(crate) fn define(
     // Reference type instructions
 
     // Null references implemented as iconst 0.
-    e.enc32(null.bind_ref(R32), rec_pu_id_ref.opcodes(&MOV_IMM));
+    e.enc32(null.bind(R32), rec_pu_id_ref.opcodes(&MOV_IMM));
 
-    e.enc64(null.bind_ref(R64), rec_pu_id_ref.rex().opcodes(&MOV_IMM));
-    e.enc64(null.bind_ref(R64), rec_pu_id_ref.opcodes(&MOV_IMM));
+    e.enc64(null.bind(R64), rec_pu_id_ref.rex().opcodes(&MOV_IMM));
+    e.enc64(null.bind(R64), rec_pu_id_ref.opcodes(&MOV_IMM));
 
     // is_null, implemented by testing whether the value is 0.
     e.enc_r32_r64_rex_only(is_null, rec_is_zero.opcodes(&TEST_REG));
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 61389729f5..f5c1c43015 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::ast::{var, ExprBuilder, Literal};
-use crate::cdsl::instructions::InstructionGroup;
+use crate::cdsl::instructions::{vector, Bindable, InstructionGroup};
 use crate::cdsl::types::ValueType;
 use crate::cdsl::xform::TransformGroupBuilder;
 use crate::shared::types::Float::F64;
@@ -322,10 +322,8 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
 
     // SIMD splat: 8-bits
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
-        let splat_any8x16 = splat.bind_vector_from_lane(ty, sse_vector_size);
-        let bitcast_f64_to_any8x16 = raw_bitcast
-            .bind_vector_from_lane(ty, sse_vector_size)
-            .bind(F64);
+        let splat_any8x16 = splat.bind(vector(ty, sse_vector_size));
+        let bitcast_f64_to_any8x16 = raw_bitcast.bind(vector(ty, sse_vector_size)).bind(F64);
         narrow.legalize(
             def!(y = splat_any8x16(x)),
             vec![
@@ -340,13 +338,13 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
 
     // SIMD splat: 16-bits
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 16) {
-        let splat_x16x8 = splat.bind_vector_from_lane(ty, sse_vector_size);
+        let splat_x16x8 = splat.bind(vector(ty, sse_vector_size));
         let raw_bitcast_any16x8_to_i32x4 = raw_bitcast
-            .bind_vector_from_lane(I32, sse_vector_size)
-            .bind_vector_from_lane(ty, sse_vector_size);
+            .bind(vector(I32, sse_vector_size))
+            .bind(vector(ty, sse_vector_size));
         let raw_bitcast_i32x4_to_any16x8 = raw_bitcast
-            .bind_vector_from_lane(ty, sse_vector_size)
-            .bind_vector_from_lane(I32, sse_vector_size);
+            .bind(vector(ty, sse_vector_size))
+            .bind(vector(I32, sse_vector_size));
         narrow.legalize(
             def!(y = splat_x16x8(x)),
             vec![
@@ -361,7 +359,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
 
     // SIMD splat: 32-bits
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 32) {
-        let splat_any32x4 = splat.bind_vector_from_lane(ty, sse_vector_size);
+        let splat_any32x4 = splat.bind(vector(ty, sse_vector_size));
         narrow.legalize(
             def!(y = splat_any32x4(x)),
             vec![
@@ -373,7 +371,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
 
     // SIMD splat: 64-bits
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 64) {
-        let splat_any64x2 = splat.bind_vector_from_lane(ty, sse_vector_size);
+        let splat_any64x2 = splat.bind(vector(ty, sse_vector_size));
         narrow.legalize(
             def!(y = splat_any64x2(x)),
             vec![
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index aef8cca89a..2606dfb60d 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -1,5 +1,5 @@
 use crate::cdsl::ast::{var, ExprBuilder, Literal};
-use crate::cdsl::instructions::{Instruction, InstructionGroup};
+use crate::cdsl::instructions::{Bindable, Instruction, InstructionGroup};
 use crate::cdsl::xform::{TransformGroupBuilder, TransformGroups};
 
 use crate::shared::immediates::Immediates;

From 9c159ac17c7fe9337722e17b063c190f08bdcce0 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 10 Oct 2019 12:46:51 +0200
Subject: [PATCH 2818/3084] Cleanup: Mark ebb as unused in legalization;

---
 cranelift/codegen/src/legalizer/mod.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index b0fdac9ea9..77c67fb2ae 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -161,7 +161,7 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
 
     // Process EBBs in layout order. Some legalization actions may split the current EBB or append
     // new ones to the end. We need to make sure we visit those new EBBs too.
-    while let Some(ebb) = pos.next_ebb() {
+    while let Some(_ebb) = pos.next_ebb() {
         // Keep track of the cursor position before the instruction being processed, so we can
         // double back when replacing instructions.
         let mut prev_pos = pos.position();

From 5e879962757e1f00545f25a502d9c3de5b97bdf5 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 10 Oct 2019 21:19:53 +0200
Subject: [PATCH 2819/3084] [wasm] Make the WasmTypeMap constructor public;
 (#1125)

---
 cranelift/wasm/src/environ/spec.rs | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 87607755e4..499483116f 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -115,8 +115,9 @@ pub struct WasmTypesMap {
 }
 
 impl WasmTypesMap {
-    pub(crate) fn new() -> Self {
-        WasmTypesMap {
+    /// Creates a new type map.
+    pub fn new() -> Self {
+        Self {
             inner: PrimaryMap::new(),
         }
     }

From 96d51cb1e8223137300bd2ef5f17074443e007d6 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 30 Sep 2019 11:27:29 -0700
Subject: [PATCH 2820/3084] Switch x86 SIMD bor from ORPS to POR encoding

There are two reasons for this change:
 1. it reduces confusion; using the `POR` encoding will match the future encodings of `band` and `bxor` and the `ORPS` encoding may be confusing as it is intended for floating-point operations
 2. `POR` has slightly more throughput: it only has to wait 0.33 cycles to execute again on all Intel architectures above Core whereas `ORPS` must wait 1 cycle on architectures older than Skylake (Intel Optimization Reference Manual, C.3)

`POR` does add one additional byte to the encoding and requires SSE2 so the `ORPS` opcode is left in for future use.
---
 cranelift/codegen/meta/src/isa/x86/encodings.rs | 13 ++++++-------
 cranelift/codegen/meta/src/isa/x86/opcodes.rs   |  3 +++
 2 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index c9b1ed8b42..a42c3872ef 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1874,13 +1874,6 @@ pub(crate) fn define<'defs>(
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
     }
 
-    // SIMD bor using ORPS
-    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
-        let instruction = bor.bind(vector(ty, sse_vector_size));
-        let template = rec_fa.nonrex().opcodes(&ORPS);
-        e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
-    }
-
     // SIMD register movement: store, load, spill, fill, regmove. All of these use encodings of
     // MOVUPS and MOVAPS from SSE (TODO ideally all of these would either use MOVAPS when we have
     // alignment or type-specific encodings, see https://github.com/CraneStation/cranelift/issues/1039).
@@ -1980,6 +1973,12 @@ pub(crate) fn define<'defs>(
         e.enc_32_64_maybe_isap(imul, rec_fa.opcodes(opcodes), *isap);
     }
 
+    // SIMD bor
+    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
+        let bor = bor.bind(vector(ty, sse_vector_size));
+        e.enc_32_64(bor, rec_fa.nonrex().opcodes(&POR));
+    }
+
     // SIMD icmp using PCMPEQ*
     for ty in ValueType::all_lane_types().filter(|t| t.is_int() && allowed_simd_type(t)) {
         let (opcodes, isa_predicate): (&[_], _) = match ty.lane_bits() {
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index f81d2423ea..d315de0113 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -307,6 +307,9 @@ pub static POP_REG: [u8; 1] = [0x58];
 /// Returns the count of number of bits set to 1.
 pub static POPCNT: [u8; 3] = [0xf3, 0x0f, 0xb8];
 
+/// Bitwise OR of xmm2/m128 and xmm1 (SSE2).
+pub static POR: [u8; 3] = [0x66, 0x0f, 0xeb];
+
 /// Shuffle bytes in xmm1 according to contents of xmm2/m128 (SSE3).
 pub static PSHUFB: [u8; 4] = [0x66, 0x0f, 0x38, 0x00];
 

From 4cdc1e76a489205c5aef550ea4ff6511c26ad3fa Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 30 Sep 2019 11:44:31 -0700
Subject: [PATCH 2821/3084] Add x86 SIMD band

---
 cranelift/codegen/meta/src/isa/x86/encodings.rs   |  7 ++++++-
 cranelift/codegen/meta/src/isa/x86/opcodes.rs     |  3 +++
 .../filetests/isa/x86/simd-logical-binemit.clif   | 15 +++++++++++++++
 3 files changed, 24 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index a42c3872ef..f63a15e3ae 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1973,8 +1973,13 @@ pub(crate) fn define<'defs>(
         e.enc_32_64_maybe_isap(imul, rec_fa.opcodes(opcodes), *isap);
     }
 
-    // SIMD bor
+    // SIMD logical operations
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
+        // band
+        let band = band.bind(vector(ty, sse_vector_size));
+        e.enc_32_64(band, rec_fa.opcodes(&PAND));
+
+        // bor
         let bor = bor.bind(vector(ty, sse_vector_size));
         e.enc_32_64(bor, rec_fa.nonrex().opcodes(&POR));
     }
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index d315de0113..b7f223eb27 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -263,6 +263,9 @@ pub static PADDUSB: [u8; 3] = [0x66, 0x0f, 0xdc];
 /// Add packed unsigned word integers from xmm2/m128 and xmm1 saturate the results (SSE).
 pub static PADDUSW: [u8; 3] = [0x66, 0x0f, 0xdd];
 
+/// Bitwise AND of xmm2/m128 and xmm1 (SSE2).
+pub static PAND: [u8; 3] = [0x66, 0x0f, 0xdb];
+
 /// Compare packed data for equal (SSE2).
 pub static PCMPEQB: [u8; 3] = [0x66, 0x0f, 0x74];
 
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
new file mode 100644
index 0000000000..9500175b86
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
@@ -0,0 +1,15 @@
+test binemit
+set enable_simd
+target x86_64 skylake
+
+function %bor_b16x8(b16x8, b16x8) -> b16x8 {
+ebb0(v0: b16x8 [%xmm2], v1: b16x8 [%xmm1]):
+[-, %xmm2]  v2 = bor v0, v1     ; bin: 66 0f eb d1
+            return v2
+}
+
+function %band_b64x2(b64x2, b64x2) -> b64x2 {
+ebb0(v0: b64x2 [%xmm6], v1: b64x2 [%xmm3]):
+[-, %xmm6]  v2 = band v0, v1      ; bin: 66 0f db f3
+            return v2
+}

From dbe7dd59da45ef97ca523e83448898f05c503695 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 30 Sep 2019 11:48:20 -0700
Subject: [PATCH 2822/3084] Add x86 SIMD bxor

---
 cranelift/codegen/meta/src/isa/x86/encodings.rs        | 10 +++++++---
 .../filetests/isa/x86/simd-logical-binemit.clif        |  6 ++++++
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index f63a15e3ae..dcdc4d0196 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1975,13 +1975,17 @@ pub(crate) fn define<'defs>(
 
     // SIMD logical operations
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
-        // band
+        // and
         let band = band.bind(vector(ty, sse_vector_size));
         e.enc_32_64(band, rec_fa.opcodes(&PAND));
 
-        // bor
+        // or
         let bor = bor.bind(vector(ty, sse_vector_size));
-        e.enc_32_64(bor, rec_fa.nonrex().opcodes(&POR));
+        e.enc_32_64(bor, rec_fa.opcodes(&POR));
+
+        // xor
+        let bxor = bxor.bind(vector(ty, sse_vector_size));
+        e.enc_32_64(bxor, rec_fa.opcodes(&PXOR));
     }
 
     // SIMD icmp using PCMPEQ*
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
index 9500175b86..dd0365b016 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
@@ -13,3 +13,9 @@ ebb0(v0: b64x2 [%xmm6], v1: b64x2 [%xmm3]):
 [-, %xmm6]  v2 = band v0, v1      ; bin: 66 0f db f3
             return v2
 }
+
+function %bxor_b32x4(b32x4, b32x4) -> b32x4 {
+ebb0(v0: b32x4 [%xmm4], v1: b32x4 [%xmm0]):
+[-, %xmm4]  v2 = bxor v0, v1      ; bin: 66 0f ef e0
+            return v2
+}

From 4c56516d3f5f1d88fb64cbf0d3c6039e330f996e Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 2 Oct 2019 10:20:31 -0700
Subject: [PATCH 2823/3084] Allow creating constants in legalization AST

This adds a `DummyConstant` structure that is converted to something like `let const0 = pos.func.dfg.constants.insert(...)` in `gen_legalizer.rs`. This allows us to create constants during legalization with something like `let ones = constant(vec![0xff; 16])` and then use `ones` within a `def!` block, e.g.: `def!(a = vconst(ones))`. One unfortunate side-effect of this change is that, because the names of the constants in `ConstPool` are dynamic, the `VarPool` and `SymbolTable` structures that previously operated on `&'static str` types now must operate on `String` types; however, since this is a change to the meta code-generation, it should result in no runtime performance impact.
---
 cranelift/codegen/meta/src/cdsl/ast.rs      | 114 ++++++++++++++++++--
 cranelift/codegen/meta/src/cdsl/xform.rs    |  47 +++++---
 cranelift/codegen/meta/src/gen_legalizer.rs |  18 +++-
 3 files changed, 154 insertions(+), 25 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 5618927bb4..9014c4df5f 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -54,7 +54,7 @@ impl Def {
         let results = self
             .defined_vars
             .iter()
-            .map(|&x| var_pool.get(x).name)
+            .map(|&x| var_pool.get(x).name.as_str())
             .collect::>();
 
         let results = if results.len() == 1 {
@@ -238,7 +238,7 @@ pub(crate) enum PatternPosition {
 ///
 /// Temporary values are defined only in the destination pattern.
 pub(crate) struct Var {
-    pub name: &'static str,
+    pub name: String,
 
     /// The `Def` defining this variable in a source pattern.
     pub src_def: Option,
@@ -254,7 +254,7 @@ pub(crate) struct Var {
 }
 
 impl Var {
-    fn new(name: &'static str) -> Self {
+    fn new(name: String) -> Self {
         Self {
             name,
             src_def: None,
@@ -346,7 +346,7 @@ impl Var {
     }
 
     pub fn to_rust_code(&self) -> String {
-        self.name.into()
+        self.name.clone()
     }
     fn rust_type(&self) -> String {
         self.type_var.as_ref().unwrap().to_rust_code()
@@ -384,8 +384,51 @@ impl VarPool {
     pub fn get_mut(&mut self, index: VarIndex) -> &mut Var {
         self.pool.get_mut(index).unwrap()
     }
-    pub fn create(&mut self, name: &'static str) -> VarIndex {
-        self.pool.push(Var::new(name))
+    pub fn create(&mut self, name: impl Into) -> VarIndex {
+        self.pool.push(Var::new(name.into()))
+    }
+}
+
+/// Contains constants created in the AST that must be inserted into the true [ConstantPool]
+/// (cranelift_codegen::ir::ConstantPool) when the legalizer code is generated. The constant data
+/// is named in the order it is inserted; inserting data using [insert]
+/// (cranelift_codegen_meta::cdsl::ast::insert) will avoid duplicates.
+pub(crate) struct ConstPool {
+    pool: Vec>,
+}
+
+impl ConstPool {
+    /// Create an empty constant pool.
+    pub fn new() -> Self {
+        Self { pool: vec![] }
+    }
+
+    /// Create a name for a constant from its position in the pool.
+    fn create_name(position: usize) -> String {
+        format!("const{}", position)
+    }
+
+    /// Insert constant data into the pool, returning the name of the variable used to reference it.
+    /// This method will search for data that matches the new data and return the existing constant
+    /// name to avoid duplicates.
+    pub fn insert(&mut self, data: Vec) -> String {
+        let possible_position = self.pool.iter().position(|d| d == &data);
+        let position = if let Some(found_position) = possible_position {
+            found_position
+        } else {
+            let new_position = self.pool.len();
+            self.pool.push(data);
+            new_position
+        };
+        ConstPool::create_name(position)
+    }
+
+    /// Iterate over the name/value pairs in the pool.
+    pub fn iter(&self) -> impl Iterator)> {
+        self.pool
+            .iter()
+            .enumerate()
+            .map(|(i, v)| (ConstPool::create_name(i), v))
     }
 }
 
@@ -567,13 +610,14 @@ impl Apply {
 pub(crate) enum DummyExpr {
     Var(DummyVar),
     Literal(Literal),
+    Constant(DummyConstant),
     Apply(InstSpec, Vec),
     Block(DummyVar),
 }
 
 #[derive(Clone)]
 pub(crate) struct DummyVar {
-    pub name: &'static str,
+    pub name: String,
 }
 
 impl Into for DummyVar {
@@ -587,8 +631,23 @@ impl Into for Literal {
     }
 }
 
-pub(crate) fn var(name: &'static str) -> DummyVar {
-    DummyVar { name }
+#[derive(Clone)]
+pub(crate) struct DummyConstant(pub(crate) Vec);
+
+pub(crate) fn constant(data: Vec) -> DummyConstant {
+    DummyConstant(data)
+}
+
+impl Into for DummyConstant {
+    fn into(self) -> DummyExpr {
+        DummyExpr::Constant(self)
+    }
+}
+
+pub(crate) fn var(name: &str) -> DummyVar {
+    DummyVar {
+        name: name.to_owned(),
+    }
 }
 
 pub(crate) struct DummyDef {
@@ -656,3 +715,40 @@ macro_rules! ebb {
         ExprBuilder::block($block).assign_to(Vec::new())
     };
 }
+
+#[cfg(test)]
+mod tests {
+    use crate::cdsl::ast::ConstPool;
+
+    #[test]
+    fn const_pool_returns_var_names() {
+        let mut c = ConstPool::new();
+        assert_eq!(c.insert([0, 1, 2].to_vec()), "const0");
+        assert_eq!(c.insert([1, 2, 3].to_vec()), "const1");
+    }
+
+    #[test]
+    fn const_pool_avoids_duplicates() {
+        let data = [0, 1, 2].to_vec();
+        let mut c = ConstPool::new();
+        assert_eq!(c.pool.len(), 0);
+
+        assert_eq!(c.insert(data.clone()), "const0");
+        assert_eq!(c.pool.len(), 1);
+
+        assert_eq!(c.insert(data), "const0");
+        assert_eq!(c.pool.len(), 1);
+    }
+
+    #[test]
+    fn const_pool_iterates() {
+        let mut c = ConstPool::new();
+        c.insert([0, 1, 2].to_vec());
+        c.insert([3, 4, 5].to_vec());
+
+        let mut iter = c.iter();
+        assert_eq!(iter.next(), Some(("const0".to_owned(), &vec![0, 1, 2])));
+        assert_eq!(iter.next(), Some(("const1".to_owned(), &vec![3, 4, 5])));
+        assert_eq!(iter.next(), None);
+    }
+}
diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index 6304469718..916bdc0a32 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -1,6 +1,6 @@
 use crate::cdsl::ast::{
-    Apply, BlockPool, DefIndex, DefPool, DummyDef, DummyExpr, Expr, PatternPosition, VarIndex,
-    VarPool,
+    Apply, BlockPool, ConstPool, DefIndex, DefPool, DummyDef, DummyExpr, Expr, PatternPosition,
+    VarIndex, VarPool,
 };
 use crate::cdsl::instructions::Instruction;
 use crate::cdsl::type_inference::{infer_transform, TypeEnvironment};
@@ -24,16 +24,18 @@ pub(crate) struct Transform {
     pub var_pool: VarPool,
     pub def_pool: DefPool,
     pub block_pool: BlockPool,
+    pub const_pool: ConstPool,
     pub type_env: TypeEnvironment,
 }
 
-type SymbolTable = HashMap<&'static str, VarIndex>;
+type SymbolTable = HashMap;
 
 impl Transform {
     fn new(src: DummyDef, dst: Vec) -> Self {
         let mut var_pool = VarPool::new();
         let mut def_pool = DefPool::new();
         let mut block_pool = BlockPool::new();
+        let mut const_pool = ConstPool::new();
 
         let mut input_vars: Vec = Vec::new();
         let mut defined_vars: Vec = Vec::new();
@@ -51,6 +53,7 @@ impl Transform {
             &mut var_pool,
             &mut def_pool,
             &mut block_pool,
+            &mut const_pool,
         )[0];
 
         let num_src_inputs = input_vars.len();
@@ -64,6 +67,7 @@ impl Transform {
             &mut var_pool,
             &mut def_pool,
             &mut block_pool,
+            &mut const_pool,
         );
 
         // Sanity checks.
@@ -128,6 +132,7 @@ impl Transform {
             var_pool,
             def_pool,
             block_pool,
+            const_pool,
             type_env,
         }
     }
@@ -148,16 +153,17 @@ impl Transform {
 /// pool `var_pool`. If the variable was not present in the symbol table, then add it to the list of
 /// `defined_vars`.
 fn var_index(
-    name: &'static str,
+    name: &str,
     symbol_table: &mut SymbolTable,
     defined_vars: &mut Vec,
     var_pool: &mut VarPool,
 ) -> VarIndex {
-    match symbol_table.get(name) {
+    let name = name.to_string();
+    match symbol_table.get(&name) {
         Some(&existing_var) => existing_var,
         None => {
             // Materialize the variable.
-            let new_var = var_pool.create(name);
+            let new_var = var_pool.create(name.clone());
             symbol_table.insert(name, new_var);
             defined_vars.push(new_var);
             new_var
@@ -176,7 +182,7 @@ fn rewrite_defined_vars(
 ) -> Vec {
     let mut new_defined_vars = Vec::new();
     for var in &dummy_def.defined_vars {
-        let own_var = var_index(var.name, symbol_table, defined_vars, var_pool);
+        let own_var = var_index(&var.name, symbol_table, defined_vars, var_pool);
         var_pool.get_mut(own_var).set_def(position, def_index);
         new_defined_vars.push(own_var);
     }
@@ -190,6 +196,7 @@ fn rewrite_expr(
     symbol_table: &mut SymbolTable,
     input_vars: &mut Vec,
     var_pool: &mut VarPool,
+    const_pool: &mut ConstPool,
 ) -> Apply {
     let (apply_target, dummy_args) = if let DummyExpr::Apply(apply_target, dummy_args) = dummy_expr
     {
@@ -215,7 +222,7 @@ fn rewrite_expr(
     for (i, arg) in dummy_args.into_iter().enumerate() {
         match arg {
             DummyExpr::Var(var) => {
-                let own_var = var_index(var.name, symbol_table, input_vars, var_pool);
+                let own_var = var_index(&var.name, symbol_table, input_vars, var_pool);
                 let var = var_pool.get(own_var);
                 assert!(
                     var.is_input() || var.get_def(position).is_some(),
@@ -227,6 +234,15 @@ fn rewrite_expr(
                 assert!(!apply_target.inst().operands_in[i].is_value());
                 args.push(Expr::Literal(literal));
             }
+            DummyExpr::Constant(constant) => {
+                let const_name = const_pool.insert(constant.0);
+                // Here we abuse var_index by passing an empty, immediately-dropped vector to
+                // `defined_vars`; the reason for this is that unlike the `Var` case above,
+                // constants will create a variable that is not an input variable (it is tracked
+                // instead by ConstPool).
+                let const_var = var_index(&const_name, symbol_table, &mut vec![], var_pool);
+                args.push(Expr::Var(const_var));
+            }
             DummyExpr::Apply(..) => {
                 panic!("Recursive apply is not allowed.");
             }
@@ -248,13 +264,14 @@ fn rewrite_def_list(
     var_pool: &mut VarPool,
     def_pool: &mut DefPool,
     block_pool: &mut BlockPool,
+    const_pool: &mut ConstPool,
 ) -> Vec {
     let mut new_defs = Vec::new();
     // Register variable names of new blocks first as a block name can be used to jump forward. Thus
     // the name has to be registered first to avoid misinterpreting it as an input-var.
     for dummy_def in dummy_defs.iter() {
         if let DummyExpr::Block(ref var) = dummy_def.expr {
-            var_index(var.name, symbol_table, defined_vars, var_pool);
+            var_index(&var.name, symbol_table, defined_vars, var_pool);
         }
     }
 
@@ -272,7 +289,7 @@ fn rewrite_def_list(
         );
         if let DummyExpr::Block(var) = dummy_def.expr {
             let var_index = *symbol_table
-                .get(var.name)
+                .get(&var.name)
                 .or_else(|| {
                     panic!(
                         "Block {} was not registered during the first visit",
@@ -283,8 +300,14 @@ fn rewrite_def_list(
             var_pool.get_mut(var_index).set_def(position, def_index);
             block_pool.create_block(var_index, def_index);
         } else {
-            let new_apply =
-                rewrite_expr(position, dummy_def.expr, symbol_table, input_vars, var_pool);
+            let new_apply = rewrite_expr(
+                position,
+                dummy_def.expr,
+                symbol_table,
+                input_vars,
+                var_pool,
+                const_pool,
+            );
 
             assert!(
                 def_pool.next_index() == def_index,
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index bceed3fac6..8a52960f42 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -43,7 +43,7 @@ fn unwrap_inst(
         .args
         .iter()
         .map(|arg| match arg.maybe_var() {
-            Some(var_index) => var_pool.get(var_index).name,
+            Some(var_index) => var_pool.get(var_index).name.as_ref(),
             None => "_",
         })
         .collect::>()
@@ -114,7 +114,7 @@ fn unwrap_inst(
     assert_eq!(inst.operands_in.len(), apply.args.len());
     for (i, op) in inst.operands_in.iter().enumerate() {
         if op.is_varargs() {
-            let name = var_pool
+            let name = &var_pool
                 .get(apply.args[i].maybe_var().expect("vararg without name"))
                 .name;
 
@@ -283,8 +283,8 @@ fn emit_dst_inst(def: &Def, def_pool: &DefPool, var_pool: &VarPool, fmt: &mut Fo
         let vars = def
             .defined_vars
             .iter()
-            .map(|&var_index| var_pool.get(var_index).name)
-            .collect::>();
+            .map(|&var_index| var_pool.get(var_index).name.as_ref())
+            .collect::>();
         if vars.len() == 1 {
             vars[0].to_string()
         } else {
@@ -401,6 +401,16 @@ fn gen_transform<'a>(
     // Guard the actual expansion by `predicate`.
     fmt.line("if predicate {");
     fmt.indent(|fmt| {
+        // Emit any constants that must be created before use.
+        for (name, value) in transform.const_pool.iter() {
+            fmtln!(
+                fmt,
+                "let {} = pos.func.dfg.constants.insert(vec!{:?});",
+                name,
+                value
+            );
+        }
+
         // If we are adding some blocks, we need to recall the original block, such that we can
         // recompute it.
         if !transform.block_pool.is_empty() {

From 1f728c179761d713b49998cf463eb2c96ced09b7 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 2 Oct 2019 10:24:20 -0700
Subject: [PATCH 2824/3084] Add x86 legalization for SIMD bnot

---
 cranelift/codegen/meta/src/isa/x86/legalize.rs | 18 ++++++++++++++++--
 cranelift/codegen/src/ir/immediates.rs         |  6 +++---
 .../isa/x86/simd-logical-legalize.clif         | 11 +++++++++++
 .../filetests/isa/x86/simd-logical-rodata.clif | 11 +++++++++++
 4 files changed, 41 insertions(+), 5 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/simd-logical-rodata.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index f5c1c43015..00cbcb3a92 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -1,6 +1,6 @@
-use crate::cdsl::ast::{var, ExprBuilder, Literal};
+use crate::cdsl::ast::{constant, var, ExprBuilder, Literal};
 use crate::cdsl::instructions::{vector, Bindable, InstructionGroup};
-use crate::cdsl::types::ValueType;
+use crate::cdsl::types::{LaneType, ValueType};
 use crate::cdsl::xform::TransformGroupBuilder;
 use crate::shared::types::Float::F64;
 use crate::shared::types::Int::{I32, I64};
@@ -21,6 +21,8 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let insts = &shared.instructions;
     let band = insts.by_name("band");
     let bor = insts.by_name("bor");
+    let bnot = insts.by_name("bnot");
+    let bxor = insts.by_name("bxor");
     let clz = insts.by_name("clz");
     let ctz = insts.by_name("ctz");
     let extractlane = insts.by_name("extractlane");
@@ -52,6 +54,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let umulhi = insts.by_name("umulhi");
     let ushr_imm = insts.by_name("ushr_imm");
     let urem = insts.by_name("urem");
+    let vconst = insts.by_name("vconst");
 
     let x86_bsf = x86_instructions.by_name("x86_bsf");
     let x86_bsr = x86_instructions.by_name("x86_bsr");
@@ -319,6 +322,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
 
     // SIMD vector size: eventually multiple vector sizes may be supported but for now only SSE-sized vectors are available
     let sse_vector_size: u64 = 128;
+    let allowed_simd_type = |t: &LaneType| t.lane_bits() >= 8 && t.lane_bits() < 128;
 
     // SIMD splat: 8-bits
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
@@ -381,6 +385,16 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         );
     }
 
+    // SIMD bnot
+    let ones = constant(vec![0xff; 16]);
+    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
+        let bnot = bnot.bind(vector(ty, sse_vector_size));
+        narrow.legalize(
+            def!(y = bnot(x)),
+            vec![def!(a = vconst(ones)), def!(y = bxor(a, x))],
+        );
+    }
+
     narrow.custom_legalize(shuffle, "convert_shuffle");
     narrow.custom_legalize(extractlane, "convert_extractlane");
     narrow.custom_legalize(insertlane, "convert_insertlane");
diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index 27320328fe..8ca70e5a44 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -302,17 +302,17 @@ impl FromStr for Uimm32 {
 pub struct V128Imm(pub [u8; 16]);
 
 impl V128Imm {
-    /// Iterate over the bytes in the constant
+    /// Iterate over the bytes in the constant.
     pub fn bytes(&self) -> impl Iterator {
         self.0.iter()
     }
 
-    /// Convert the immediate into a vector
+    /// Convert the immediate into a vector.
     pub fn to_vec(self) -> Vec {
         self.0.to_vec()
     }
 
-    /// Convert the immediate into a slice
+    /// Convert the immediate into a slice.
     pub fn as_slice(&self) -> &[u8] {
         &self.0[..]
     }
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif
new file mode 100644
index 0000000000..be00fe7278
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif
@@ -0,0 +1,11 @@
+test legalizer
+set enable_simd
+target x86_64 skylake
+
+function %bnot_b32x4(b32x4) -> b32x4 {
+ebb0(v0: b32x4):
+    v1 = bnot v0
+    ; check: v2 = vconst.b32x4 0xffffffffffffffffffffffffffffffff
+    ; nextln: v1 = bxor v2, v0
+    return v1
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-rodata.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-rodata.clif
new file mode 100644
index 0000000000..1c5fb89733
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-rodata.clif
@@ -0,0 +1,11 @@
+test rodata
+set enable_simd
+target x86_64 skylake
+
+function %bnot_b32x4(b32x4) -> b32x4 {
+ebb0(v0: b32x4):
+    v1 = bnot v0
+    return v1
+}
+
+; sameln: [FF, FF, FF, FF, FF, FF, FF, FF, FF, FF, FF, FF, FF, FF, FF, FF]

From b19f804ed554992a358a80012df66048bebb829b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 2 Oct 2019 14:02:58 -0700
Subject: [PATCH 2825/3084] Convert WASM logical operators to CLIF

---
 cranelift/wasm/src/code_translator.rs | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 4b6eb8ed32..a6aa6a32d1 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1083,6 +1083,22 @@ pub fn translate_operator(
             let (a, b) = state.pop2();
             state.push1(builder.ins().imul(a, b))
         }
+        Operator::V128Not => {
+            let a = state.pop1();
+            state.push1(builder.ins().bnot(a));
+        }
+        Operator::V128And => {
+            let (a, b) = state.pop2();
+            state.push1(builder.ins().band(a, b));
+        }
+        Operator::V128Or => {
+            let (a, b) = state.pop2();
+            state.push1(builder.ins().bor(a, b));
+        }
+        Operator::V128Xor => {
+            let (a, b) = state.pop2();
+            state.push1(builder.ins().bxor(a, b));
+        }
         Operator::I8x16Eq
         | Operator::I8x16Ne
         | Operator::I8x16LtS
@@ -1125,10 +1141,6 @@ pub fn translate_operator(
         | Operator::F64x2Gt
         | Operator::F64x2Le
         | Operator::F64x2Ge
-        | Operator::V128Not
-        | Operator::V128And
-        | Operator::V128Or
-        | Operator::V128Xor
         | Operator::V128Bitselect
         | Operator::I8x16AnyTrue
         | Operator::I8x16AllTrue

From ca53090f1bc2ce61fa440576168e259a80eff121 Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Fri, 11 Oct 2019 12:37:17 -0700
Subject: [PATCH 2826/3084] cranelift-wasm: Create `ModuleTranslationState` and
 polish API a little (#1111)

* cranelift-wasm: replace `WasmTypesMap` with `ModuleTranslationState`

The `ModuleTranslationState` contains information decoded from the Wasm module
that must be referenced during each Wasm function's translation.

This is only for data that is maintained by `cranelift-wasm` itself, as opposed
to being maintained by the embedder. Data that is maintained by the embedder is
represented with `ModuleEnvironment`.

A `ModuleTranslationState` is returned by `translate_module`, and can then be
used when translating functions from that module.

* cranelift-wasm: rename `TranslationState` to `FuncTranslationState`

To disambiguate a bit with the new `ModuleTranslationState`.

* cranelift-wasm: Reorganize the internal `state` module into submodules

One module for the `ModuleTranslationState` and another for the
`FuncTranslationState`.

* cranelift-wasm: replace `FuncTranslator` with methods on `ModuleTranslationState`

`FuncTranslator` was two methods that always took ownership of `self`, so it
didn't really make sense as an object as opposed to two different functions, or
in this case methods on the object that actually persists for a longer time.

I think this improves ergonomics nicely.

Before:

```rust
let module_translation = translate_module(...)?;
for body in func_bodies {
    let mut translator = FuncTranslator::new();
    translator.translate(body, ...)?;
}
```

After:

```rust
let module_translation = translate_module(...)?;
for body in func_bodies {
    module_translation.translate_func(body, ...)?;
}
```

Note that this commit does not remove `FuncTranslator`. It still exists, but is
just a wrapper over the `ModuleTranslationState` methods, and it is marked
deprecated, so that downstream users get a heads up. This should make the
transition easier.

* Revert "cranelift-wasm: replace `FuncTranslator` with methods on `ModuleTranslationState`"

This reverts commit 075f9ae933bcaae39348b61287c8f78a4009340d.
---
 cranelift/wasm/src/code_translator.rs         | 38 ++++++++--------
 cranelift/wasm/src/environ/dummy.rs           |  9 ++--
 cranelift/wasm/src/environ/mod.rs             |  1 -
 cranelift/wasm/src/environ/spec.rs            | 28 ++----------
 cranelift/wasm/src/func_translator.rs         | 43 +++++++++++--------
 cranelift/wasm/src/lib.rs                     |  5 ++-
 cranelift/wasm/src/module_translator.rs       | 13 +++---
 cranelift/wasm/src/sections_translator.rs     | 13 +++---
 .../src/{state.rs => state/func_state.rs}     | 21 +++++----
 cranelift/wasm/src/state/mod.rs               | 14 ++++++
 cranelift/wasm/src/state/module_state.rs      | 27 ++++++++++++
 cranelift/wasm/src/translation_utils.rs       |  7 +--
 12 files changed, 127 insertions(+), 92 deletions(-)
 rename cranelift/wasm/src/{state.rs => state/func_state.rs} (96%)
 create mode 100644 cranelift/wasm/src/state/mod.rs
 create mode 100644 cranelift/wasm/src/state/module_state.rs

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index a6aa6a32d1..bc75157a88 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -23,8 +23,8 @@
 //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as
 //! argument.
 use super::{hash_map, HashMap};
-use crate::environ::{FuncEnvironment, GlobalVariable, ReturnMode, WasmResult, WasmTypesMap};
-use crate::state::{ControlStackFrame, ElseData, TranslationState};
+use crate::environ::{FuncEnvironment, GlobalVariable, ReturnMode, WasmResult};
+use crate::state::{ControlStackFrame, ElseData, FuncTranslationState, ModuleTranslationState};
 use crate::translation_utils::{
     blocktype_params_results, ebb_with_params, f32_translation, f64_translation,
 };
@@ -43,14 +43,14 @@ use wasmparser::{MemoryImmediate, Operator};
 /// Translates wasm operators into Cranelift IR instructions. Returns `true` if it inserted
 /// a return.
 pub fn translate_operator(
-    wasm_types: &WasmTypesMap,
+    module_translation_state: &ModuleTranslationState,
     op: &Operator,
     builder: &mut FunctionBuilder,
-    state: &mut TranslationState,
+    state: &mut FuncTranslationState,
     environ: &mut FE,
 ) -> WasmResult<()> {
     if !state.reachable {
-        translate_unreachable_operator(wasm_types, &op, builder, state)?;
+        translate_unreachable_operator(module_translation_state, &op, builder, state)?;
         return Ok(());
     }
 
@@ -133,12 +133,12 @@ pub fn translate_operator(
          *  possible `Ebb`'s arguments values.
          ***********************************************************************************/
         Operator::Block { ty } => {
-            let (params, results) = blocktype_params_results(wasm_types, *ty)?;
+            let (params, results) = blocktype_params_results(module_translation_state, *ty)?;
             let next = ebb_with_params(builder, results)?;
             state.push_block(next, params.len(), results.len());
         }
         Operator::Loop { ty } => {
-            let (params, results) = blocktype_params_results(wasm_types, *ty)?;
+            let (params, results) = blocktype_params_results(module_translation_state, *ty)?;
             let loop_body = ebb_with_params(builder, params)?;
             let next = ebb_with_params(builder, results)?;
             builder.ins().jump(loop_body, state.peekn(params.len()));
@@ -155,7 +155,7 @@ pub fn translate_operator(
         Operator::If { ty } => {
             let val = state.pop1();
 
-            let (params, results) = blocktype_params_results(wasm_types, *ty)?;
+            let (params, results) = blocktype_params_results(module_translation_state, *ty)?;
             let (destination, else_data) = if params == results {
                 // It is possible there is no `else` block, so we will only
                 // allocate an ebb for it if/when we find the `else`. For now,
@@ -214,7 +214,8 @@ pub fn translate_operator(
                     // The `if` has an `else`, so there's no branch to the end from the top.
                     *reachable_from_top = false;
 
-                    let (params, _results) = blocktype_params_results(wasm_types, blocktype)?;
+                    let (params, _results) =
+                        blocktype_params_results(module_translation_state, blocktype)?;
                     let else_ebb = ebb_with_params(builder, params)?;
                     builder.ins().jump(destination, state.peekn(params.len()));
                     state.popn(params.len());
@@ -1206,10 +1207,10 @@ pub fn translate_operator(
 /// are dropped but special ones like `End` or `Else` signal the potential end of the unreachable
 /// portion so the translation state must be updated accordingly.
 fn translate_unreachable_operator(
-    wasm_types: &WasmTypesMap,
+    module_translation_state: &ModuleTranslationState,
     op: &Operator,
     builder: &mut FunctionBuilder,
-    state: &mut TranslationState,
+    state: &mut FuncTranslationState,
 ) -> WasmResult<()> {
     match *op {
         Operator::If { ty } => {
@@ -1244,7 +1245,8 @@ fn translate_unreachable_operator(
                         // branch from the top directly to the end.
                         *reachable_from_top = false;
 
-                        let (params, _results) = blocktype_params_results(wasm_types, blocktype)?;
+                        let (params, _results) =
+                            blocktype_params_results(module_translation_state, blocktype)?;
                         let else_ebb = ebb_with_params(builder, params)?;
 
                         // We change the target of the branch instruction.
@@ -1371,7 +1373,7 @@ fn translate_load(
     opcode: ir::Opcode,
     result_ty: Type,
     builder: &mut FunctionBuilder,
-    state: &mut TranslationState,
+    state: &mut FuncTranslationState,
     environ: &mut FE,
 ) -> WasmResult<()> {
     let addr32 = state.pop1();
@@ -1394,7 +1396,7 @@ fn translate_store(
     offset: u32,
     opcode: ir::Opcode,
     builder: &mut FunctionBuilder,
-    state: &mut TranslationState,
+    state: &mut FuncTranslationState,
     environ: &mut FE,
 ) -> WasmResult<()> {
     let (addr32, val) = state.pop2();
@@ -1411,13 +1413,13 @@ fn translate_store(
     Ok(())
 }
 
-fn translate_icmp(cc: IntCC, builder: &mut FunctionBuilder, state: &mut TranslationState) {
+fn translate_icmp(cc: IntCC, builder: &mut FunctionBuilder, state: &mut FuncTranslationState) {
     let (arg0, arg1) = state.pop2();
     let val = builder.ins().icmp(cc, arg0, arg1);
     state.push1(builder.ins().bint(I32, val));
 }
 
-fn translate_fcmp(cc: FloatCC, builder: &mut FunctionBuilder, state: &mut TranslationState) {
+fn translate_fcmp(cc: FloatCC, builder: &mut FunctionBuilder, state: &mut FuncTranslationState) {
     let (arg0, arg1) = state.pop2();
     let val = builder.ins().fcmp(cc, arg0, arg1);
     state.push1(builder.ins().bint(I32, val));
@@ -1426,7 +1428,7 @@ fn translate_fcmp(cc: FloatCC, builder: &mut FunctionBuilder, state: &mut Transl
 fn translate_br_if(
     relative_depth: u32,
     builder: &mut FunctionBuilder,
-    state: &mut TranslationState,
+    state: &mut FuncTranslationState,
 ) {
     let val = state.pop1();
     let (br_destination, inputs) = translate_br_if_args(relative_depth, state);
@@ -1443,7 +1445,7 @@ fn translate_br_if(
 
 fn translate_br_if_args(
     relative_depth: u32,
-    state: &mut TranslationState,
+    state: &mut FuncTranslationState,
 ) -> (ir::Ebb, &[ir::Value]) {
     let i = state.control_stack.len() - 1 - (relative_depth as usize);
     let (return_count, br_destination) = {
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index cd84c00a28..2d3137f250 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -5,10 +5,9 @@
 //! [wasmtime-environ]: https://crates.io/crates/wasmtime-environ
 //! [Wasmtime]: https://github.com/CraneStation/wasmtime
 
-use crate::environ::{
-    FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmResult, WasmTypesMap,
-};
+use crate::environ::{FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmResult};
 use crate::func_translator::FuncTranslator;
+use crate::state::ModuleTranslationState;
 use crate::translation_utils::{
     DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table,
     TableIndex,
@@ -531,7 +530,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
 
     fn define_function_body(
         &mut self,
-        wasm_types: &WasmTypesMap,
+        module_translation_state: &ModuleTranslationState,
         body_bytes: &'data [u8],
         body_offset: usize,
     ) -> WasmResult<()> {
@@ -546,7 +545,7 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
                 func.collect_debug_info();
             }
             self.trans.translate(
-                wasm_types,
+                module_translation_state,
                 body_bytes,
                 body_offset,
                 &mut func,
diff --git a/cranelift/wasm/src/environ/mod.rs b/cranelift/wasm/src/environ/mod.rs
index ac29af7c8f..4b7405ea7b 100644
--- a/cranelift/wasm/src/environ/mod.rs
+++ b/cranelift/wasm/src/environ/mod.rs
@@ -7,5 +7,4 @@ mod spec;
 pub use crate::environ::dummy::DummyEnvironment;
 pub use crate::environ::spec::{
     FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmError, WasmResult,
-    WasmTypesMap,
 };
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 499483116f..6e44f22c74 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -6,7 +6,7 @@
 //!
 //! [Wasmtime]: https://github.com/CraneStation/wasmtime
 
-use crate::state::TranslationState;
+use crate::state::{FuncTranslationState, ModuleTranslationState};
 use crate::translation_utils::{
     FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
 };
@@ -15,7 +15,6 @@ use cranelift_codegen::cursor::FuncCursor;
 use cranelift_codegen::ir::immediates::Offset32;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::isa::TargetFrontendConfig;
-use cranelift_entity::PrimaryMap;
 use cranelift_frontend::FunctionBuilder;
 use failure_derive::Fail;
 use std::boxed::Box;
@@ -104,25 +103,6 @@ pub enum ReturnMode {
     FallthroughReturn,
 }
 
-/// A map containing a Wasm module's original, raw signatures.
-///
-/// This is used for translating multi-value Wasm blocks inside functions, which
-/// are encoded to refer to their type signature via index.
-#[derive(Debug)]
-pub struct WasmTypesMap {
-    pub(crate) inner:
-        PrimaryMap, Box<[wasmparser::Type]>)>,
-}
-
-impl WasmTypesMap {
-    /// Creates a new type map.
-    pub fn new() -> Self {
-        Self {
-            inner: PrimaryMap::new(),
-        }
-    }
-}
-
 /// Environment affecting the translation of a single WebAssembly function.
 ///
 /// A `FuncEnvironment` trait object is required to translate a WebAssembly function to Cranelift
@@ -301,7 +281,7 @@ pub trait FuncEnvironment {
         &mut self,
         _op: &Operator,
         _builder: &mut FunctionBuilder,
-        _state: &TranslationState,
+        _state: &FuncTranslationState,
     ) -> WasmResult<()> {
         Ok(())
     }
@@ -312,7 +292,7 @@ pub trait FuncEnvironment {
         &mut self,
         _op: &Operator,
         _builder: &mut FunctionBuilder,
-        _state: &TranslationState,
+        _state: &FuncTranslationState,
     ) -> WasmResult<()> {
         Ok(())
     }
@@ -469,7 +449,7 @@ pub trait ModuleEnvironment<'data> {
     /// functions is already provided by `reserve_func_types`.
     fn define_function_body(
         &mut self,
-        wasm_types: &WasmTypesMap,
+        module_translation_state: &ModuleTranslationState,
         body_bytes: &'data [u8],
         body_offset: usize,
     ) -> WasmResult<()>;
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index 0f9f979e92..39b8dd43ca 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -5,8 +5,8 @@
 //! WebAssembly module and the runtime environment.
 
 use crate::code_translator::translate_operator;
-use crate::environ::{FuncEnvironment, ReturnMode, WasmResult, WasmTypesMap};
-use crate::state::TranslationState;
+use crate::environ::{FuncEnvironment, ReturnMode, WasmResult};
+use crate::state::{FuncTranslationState, ModuleTranslationState};
 use crate::translation_utils::get_vmctx_value_label;
 use crate::wasm_unsupported;
 use cranelift_codegen::entity::EntityRef;
@@ -23,7 +23,7 @@ use wasmparser::{self, BinaryReader};
 /// functions which will reduce heap allocation traffic.
 pub struct FuncTranslator {
     func_ctx: FunctionBuilderContext,
-    state: TranslationState,
+    state: FuncTranslationState,
 }
 
 impl FuncTranslator {
@@ -31,7 +31,7 @@ impl FuncTranslator {
     pub fn new() -> Self {
         Self {
             func_ctx: FunctionBuilderContext::new(),
-            state: TranslationState::new(),
+            state: FuncTranslationState::new(),
         }
     }
 
@@ -55,14 +55,14 @@ impl FuncTranslator {
     ///
     pub fn translate(
         &mut self,
-        wasm_types: &WasmTypesMap,
+        module_translation_state: &ModuleTranslationState,
         code: &[u8],
         code_offset: usize,
         func: &mut ir::Function,
         environ: &mut FE,
     ) -> WasmResult<()> {
         self.translate_from_reader(
-            wasm_types,
+            module_translation_state,
             BinaryReader::new_with_offset(code, code_offset),
             func,
             environ,
@@ -72,7 +72,7 @@ impl FuncTranslator {
     /// Translate a binary WebAssembly function from a `BinaryReader`.
     pub fn translate_from_reader(
         &mut self,
-        wasm_types: &WasmTypesMap,
+        module_translation_state: &ModuleTranslationState,
         mut reader: BinaryReader,
         func: &mut ir::Function,
         environ: &mut FE,
@@ -108,7 +108,13 @@ impl FuncTranslator {
         self.state.initialize(&builder.func.signature, exit_block);
 
         parse_local_decls(&mut reader, &mut builder, num_params, environ)?;
-        parse_function_body(wasm_types, reader, &mut builder, &mut self.state, environ)?;
+        parse_function_body(
+            module_translation_state,
+            reader,
+            &mut builder,
+            &mut self.state,
+            environ,
+        )?;
 
         builder.finalize();
         Ok(())
@@ -206,10 +212,10 @@ fn declare_locals(
 /// This assumes that the local variable declarations have already been parsed and function
 /// arguments and locals are declared in the builder.
 fn parse_function_body(
-    wasm_types: &WasmTypesMap,
+    module_translation_state: &ModuleTranslationState,
     mut reader: BinaryReader,
     builder: &mut FunctionBuilder,
-    state: &mut TranslationState,
+    state: &mut FuncTranslationState,
     environ: &mut FE,
 ) -> WasmResult<()> {
     // The control stack is initialized with a single block representing the whole function.
@@ -220,7 +226,7 @@ fn parse_function_body(
         builder.set_srcloc(cur_srcloc(&reader));
         let op = reader.read_operator()?;
         environ.before_translate_operator(&op, builder, state)?;
-        translate_operator(wasm_types, &op, builder, state, environ)?;
+        translate_operator(module_translation_state, &op, builder, state, environ)?;
         environ.after_translate_operator(&op, builder, state)?;
     }
 
@@ -258,7 +264,8 @@ fn cur_srcloc(reader: &BinaryReader) -> ir::SourceLoc {
 #[cfg(test)]
 mod tests {
     use super::{FuncTranslator, ReturnMode};
-    use crate::environ::{DummyEnvironment, WasmTypesMap};
+    use crate::environ::DummyEnvironment;
+    use crate::ModuleTranslationState;
     use cranelift_codegen::ir::types::I32;
     use cranelift_codegen::{ir, isa, settings, Context};
     use log::debug;
@@ -290,7 +297,7 @@ mod tests {
             false,
         );
 
-        let wasm_types = WasmTypesMap::new();
+        let module_translation_state = ModuleTranslationState::new();
         let mut ctx = Context::new();
 
         ctx.func.name = ir::ExternalName::testcase("small1");
@@ -299,7 +306,7 @@ mod tests {
 
         trans
             .translate(
-                &wasm_types,
+                &module_translation_state,
                 &BODY,
                 0,
                 &mut ctx.func,
@@ -337,7 +344,7 @@ mod tests {
             false,
         );
 
-        let wasm_types = WasmTypesMap::new();
+        let module_translation_state = ModuleTranslationState::new();
         let mut ctx = Context::new();
 
         ctx.func.name = ir::ExternalName::testcase("small2");
@@ -346,7 +353,7 @@ mod tests {
 
         trans
             .translate(
-                &wasm_types,
+                &module_translation_state,
                 &BODY,
                 0,
                 &mut ctx.func,
@@ -393,7 +400,7 @@ mod tests {
             false,
         );
 
-        let wasm_types = WasmTypesMap::new();
+        let module_translation_state = ModuleTranslationState::new();
         let mut ctx = Context::new();
 
         ctx.func.name = ir::ExternalName::testcase("infloop");
@@ -401,7 +408,7 @@ mod tests {
 
         trans
             .translate(
-                &wasm_types,
+                &module_translation_state,
                 &BODY,
                 0,
                 &mut ctx.func,
diff --git a/cranelift/wasm/src/lib.rs b/cranelift/wasm/src/lib.rs
index f4fa3a7358..a50bd79320 100644
--- a/cranelift/wasm/src/lib.rs
+++ b/cranelift/wasm/src/lib.rs
@@ -59,11 +59,12 @@ mod translation_utils;
 
 pub use crate::environ::{
     DummyEnvironment, FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmError,
-    WasmResult, WasmTypesMap,
+    WasmResult,
 };
 pub use crate::func_translator::FuncTranslator;
 pub use crate::module_translator::translate_module;
-pub use crate::state::TranslationState;
+pub use crate::state::func_state::FuncTranslationState;
+pub use crate::state::module_state::ModuleTranslationState;
 pub use crate::translation_utils::{
     get_vmctx_value_label, DefinedFuncIndex, DefinedGlobalIndex, DefinedMemoryIndex,
     DefinedTableIndex, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex,
diff --git a/cranelift/wasm/src/module_translator.rs b/cranelift/wasm/src/module_translator.rs
index 4076b8c740..257daa1f20 100644
--- a/cranelift/wasm/src/module_translator.rs
+++ b/cranelift/wasm/src/module_translator.rs
@@ -1,11 +1,12 @@
 //! Translation skeleton that traverses the whole WebAssembly module and call helper functions
 //! to deal with each part of it.
-use crate::environ::{ModuleEnvironment, WasmError, WasmResult, WasmTypesMap};
+use crate::environ::{ModuleEnvironment, WasmError, WasmResult};
 use crate::sections_translator::{
     parse_code_section, parse_data_section, parse_element_section, parse_export_section,
     parse_function_section, parse_global_section, parse_import_section, parse_memory_section,
     parse_name_section, parse_start_section, parse_table_section, parse_type_section,
 };
+use crate::state::ModuleTranslationState;
 use cranelift_codegen::timing;
 use wasmparser::{CustomSectionContent, ModuleReader, SectionContent};
 
@@ -14,16 +15,16 @@ use wasmparser::{CustomSectionContent, ModuleReader, SectionContent};
 pub fn translate_module<'data>(
     data: &'data [u8],
     environ: &mut dyn ModuleEnvironment<'data>,
-) -> WasmResult<()> {
+) -> WasmResult {
     let _tt = timing::wasm_translate_module();
     let mut reader = ModuleReader::new(data)?;
-    let mut wasm_types = WasmTypesMap::new();
+    let mut module_translation_state = ModuleTranslationState::new();
 
     while !reader.eof() {
         let section = reader.read()?;
         match section.content()? {
             SectionContent::Type(types) => {
-                parse_type_section(types, &mut wasm_types, environ)?;
+                parse_type_section(types, &mut module_translation_state, environ)?;
             }
 
             SectionContent::Import(imports) => {
@@ -59,7 +60,7 @@ pub fn translate_module<'data>(
             }
 
             SectionContent::Code(code) => {
-                parse_code_section(code, &wasm_types, environ)?;
+                parse_code_section(code, &module_translation_state, environ)?;
             }
 
             SectionContent::Data(data) => {
@@ -91,5 +92,5 @@ pub fn translate_module<'data>(
         }
     }
 
-    Ok(())
+    Ok(module_translation_state)
 }
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index d14852f609..cc479c8a3e 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -7,7 +7,8 @@
 //! The special case of the initialize expressions for table elements offsets or global variables
 //! is handled, according to the semantics of WebAssembly, to only specific expressions that are
 //! interpreted on the fly.
-use crate::environ::{ModuleEnvironment, WasmResult, WasmTypesMap};
+use crate::environ::{ModuleEnvironment, WasmResult};
+use crate::state::ModuleTranslationState;
 use crate::translation_utils::{
     tabletype_to_type, type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory,
     MemoryIndex, SignatureIndex, Table, TableElementType, TableIndex,
@@ -29,11 +30,11 @@ use wasmparser::{
 /// Parses the Type section of the wasm module.
 pub fn parse_type_section(
     types: TypeSectionReader,
-    wasm_types: &mut WasmTypesMap,
+    module_translation_state: &mut ModuleTranslationState,
     environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
     let count = types.get_count();
-    wasm_types.inner.reserve(count as usize);
+    module_translation_state.wasm_types.reserve(count as usize);
     environ.reserve_signatures(count)?;
 
     for entry in types {
@@ -55,7 +56,7 @@ pub fn parse_type_section(
                     AbiParam::new(cret_arg)
                 }));
                 environ.declare_signature(sig)?;
-                wasm_types.inner.push((params, returns));
+                module_translation_state.wasm_types.push((params, returns));
             }
             ty => {
                 return Err(wasm_unsupported!(
@@ -327,14 +328,14 @@ pub fn parse_element_section<'data>(
 /// Parses the Code section of the wasm module.
 pub fn parse_code_section<'data>(
     code: CodeSectionReader<'data>,
-    wasm_types: &WasmTypesMap,
+    module_translation_state: &ModuleTranslationState,
     environ: &mut dyn ModuleEnvironment<'data>,
 ) -> WasmResult<()> {
     for body in code {
         let mut reader = body?.get_binary_reader();
         let size = reader.bytes_remaining();
         let offset = reader.original_position();
-        environ.define_function_body(wasm_types, reader.read_bytes(size)?, offset)?;
+        environ.define_function_body(module_translation_state, reader.read_bytes(size)?, offset)?;
     }
     Ok(())
 }
diff --git a/cranelift/wasm/src/state.rs b/cranelift/wasm/src/state/func_state.rs
similarity index 96%
rename from cranelift/wasm/src/state.rs
rename to cranelift/wasm/src/state/func_state.rs
index cd84434e2d..4b3627e008 100644
--- a/cranelift/wasm/src/state.rs
+++ b/cranelift/wasm/src/state/func_state.rs
@@ -1,11 +1,14 @@
-//! WebAssembly function translation state.
+//! WebAssembly module and function translation state.
 //!
-//! The `TranslationState` struct defined in this module is used to keep track of the WebAssembly
+//! The `ModuleTranslationState` struct defined in this module is used to keep track of data about
+//! the whole WebAssembly module, such as the decoded type signatures.
+//!
+//! The `FuncTranslationState` struct defined in this module is used to keep track of the WebAssembly
 //! value and control stacks during the translation of a single function.
 
-use super::{HashMap, Occupied, Vacant};
 use crate::environ::{FuncEnvironment, GlobalVariable, WasmResult};
 use crate::translation_utils::{FuncIndex, GlobalIndex, MemoryIndex, SignatureIndex, TableIndex};
+use crate::{HashMap, Occupied, Vacant};
 use cranelift_codegen::ir::{self, Ebb, Inst, Value};
 use std::vec::Vec;
 
@@ -159,12 +162,12 @@ impl ControlStackFrame {
     }
 }
 
-/// Contains information passed along during the translation and that records:
+/// Contains information passed along during a function's translation and that records:
 ///
 /// - The current value and control stacks.
 /// - The depth of the two unreachable control blocks stacks, that are manipulated when translating
 ///   unreachable code;
-pub struct TranslationState {
+pub struct FuncTranslationState {
     /// A stack of values corresponding to the active values in the input wasm function at this
     /// point.
     pub(crate) stack: Vec,
@@ -195,7 +198,7 @@ pub struct TranslationState {
 }
 
 // Public methods that are exposed to non-`cranelift_wasm` API consumers.
-impl TranslationState {
+impl FuncTranslationState {
     /// True if the current translation state expresses reachable code, false if it is unreachable.
     #[inline]
     pub fn reachable(&self) -> bool {
@@ -203,8 +206,8 @@ impl TranslationState {
     }
 }
 
-impl TranslationState {
-    /// Construct a new, empty, `TranslationState`
+impl FuncTranslationState {
+    /// Construct a new, empty, `FuncTranslationState`
     pub(crate) fn new() -> Self {
         Self {
             stack: Vec::new(),
@@ -386,7 +389,7 @@ impl TranslationState {
 }
 
 /// Methods for handling entity references.
-impl TranslationState {
+impl FuncTranslationState {
     /// Get the `GlobalVariable` reference that should be used to access the global variable
     /// `index`. Create the reference if necessary.
     /// Also return the WebAssembly type of the global.
diff --git a/cranelift/wasm/src/state/mod.rs b/cranelift/wasm/src/state/mod.rs
new file mode 100644
index 0000000000..730dc8beb5
--- /dev/null
+++ b/cranelift/wasm/src/state/mod.rs
@@ -0,0 +1,14 @@
+//! WebAssembly module and function translation state.
+//!
+//! The `ModuleTranslationState` struct defined in this module is used to keep track of data about
+//! the whole WebAssembly module, such as the decoded type signatures.
+//!
+//! The `FuncTranslationState` struct defined in this module is used to keep track of the WebAssembly
+//! value and control stacks during the translation of a single function.
+
+pub(crate) mod func_state;
+pub(crate) mod module_state;
+
+// Re-export for convenience.
+pub(crate) use func_state::*;
+pub(crate) use module_state::*;
diff --git a/cranelift/wasm/src/state/module_state.rs b/cranelift/wasm/src/state/module_state.rs
new file mode 100644
index 0000000000..b2cbca77eb
--- /dev/null
+++ b/cranelift/wasm/src/state/module_state.rs
@@ -0,0 +1,27 @@
+use crate::translation_utils::SignatureIndex;
+use cranelift_entity::PrimaryMap;
+use std::boxed::Box;
+
+/// Contains information decoded from the Wasm module that must be referenced
+/// during each Wasm function's translation.
+///
+/// This is only for data that is maintained by `cranelift-wasm` itself, as
+/// opposed to being maintained by the embedder. Data that is maintained by the
+/// embedder is represented with `ModuleEnvironment`.
+#[derive(Debug)]
+pub struct ModuleTranslationState {
+    /// A map containing a Wasm module's original, raw signatures.
+    ///
+    /// This is used for translating multi-value Wasm blocks inside functions,
+    /// which are encoded to refer to their type signature via index.
+    pub(crate) wasm_types:
+        PrimaryMap, Box<[wasmparser::Type]>)>,
+}
+
+impl ModuleTranslationState {
+    pub(crate) fn new() -> Self {
+        ModuleTranslationState {
+            wasm_types: PrimaryMap::new(),
+        }
+    }
+}
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index aba5ce5178..1e5e963b4a 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -1,5 +1,6 @@
 //! Helper functions and structures for the translation.
-use crate::environ::{WasmResult, WasmTypesMap};
+use crate::environ::WasmResult;
+use crate::state::ModuleTranslationState;
 use crate::wasm_unsupported;
 use core::u32;
 use cranelift_codegen::entity::entity_impl;
@@ -148,7 +149,7 @@ pub fn tabletype_to_type(ty: wasmparser::Type) -> WasmResult> {
 
 /// Get the parameter and result types for the given Wasm blocktype.
 pub fn blocktype_params_results(
-    wasm_types: &WasmTypesMap,
+    module_translation_state: &ModuleTranslationState,
     ty_or_ft: wasmparser::TypeOrFuncType,
 ) -> WasmResult<(&[wasmparser::Type], &[wasmparser::Type])> {
     Ok(match ty_or_ft {
@@ -163,7 +164,7 @@ pub fn blocktype_params_results(
         },
         wasmparser::TypeOrFuncType::FuncType(ty_index) => {
             let sig_idx = SignatureIndex::from_u32(ty_index);
-            let (ref params, ref returns) = wasm_types.inner[sig_idx];
+            let (ref params, ref returns) = module_translation_state.wasm_types[sig_idx];
             (&*params, &*returns)
         }
     })

From 350b3b24064060225eed2f7579071de3f8ef0c06 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 4 Jul 2019 18:28:12 +0200
Subject: [PATCH 2827/3084] [meta] Avoid unwrapping instructions several times
 during legalization;

This avoids doing multiple unpacking of the InstructionData for a single
legalization, improving readability and reducing size of the generated
code. For instance, icmp had to unpack the format once per IntCC
condition code.
---
 cranelift/codegen/meta/src/cdsl/ast.rs        |   2 +-
 .../codegen/meta/src/cdsl/instructions.rs     |  20 +--
 cranelift/codegen/meta/src/gen_encodings.rs   |   4 +-
 cranelift/codegen/meta/src/gen_legalizer.rs   | 139 ++++++++++++++----
 .../codegen/meta/src/isa/x86/legalize.rs      |  23 ++-
 5 files changed, 131 insertions(+), 57 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 9014c4df5f..f047a74e55 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -63,7 +63,7 @@ impl Def {
             format!("({})", results.join(", "))
         };
 
-        format!("{} << {}", results, self.apply.to_comment_string(var_pool))
+        format!("{} := {}", results, self.apply.to_comment_string(var_pool))
     }
 }
 
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 5d47c1a76b..a62cf39e05 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -856,14 +856,14 @@ pub enum TypePredicateNode {
 }
 
 impl TypePredicateNode {
-    fn rust_predicate(&self) -> String {
+    fn rust_predicate(&self, func_str: &str) -> String {
         match self {
             TypePredicateNode::TypeVarCheck(index, value_type_name) => format!(
-                "func.dfg.value_type(args[{}]) == {}",
-                index, value_type_name
+                "{}.dfg.value_type(args[{}]) == {}",
+                func_str, index, value_type_name
             ),
             TypePredicateNode::CtrlTypeVarCheck(value_type_name) => {
-                format!("func.dfg.ctrl_typevar(inst) == {}", value_type_name)
+                format!("{}.dfg.ctrl_typevar(inst) == {}", func_str, value_type_name)
             }
         }
     }
@@ -884,18 +884,18 @@ pub enum InstructionPredicateNode {
 }
 
 impl InstructionPredicateNode {
-    fn rust_predicate(&self) -> String {
+    fn rust_predicate(&self, func_str: &str) -> String {
         match self {
             InstructionPredicateNode::FormatPredicate(node) => node.rust_predicate(),
-            InstructionPredicateNode::TypePredicate(node) => node.rust_predicate(),
+            InstructionPredicateNode::TypePredicate(node) => node.rust_predicate(func_str),
             InstructionPredicateNode::And(nodes) => nodes
                 .iter()
-                .map(|x| x.rust_predicate())
+                .map(|x| x.rust_predicate(func_str))
                 .collect::>()
                 .join(" && "),
             InstructionPredicateNode::Or(nodes) => nodes
                 .iter()
-                .map(|x| x.rust_predicate())
+                .map(|x| x.rust_predicate(func_str))
                 .collect::>()
                 .join(" || "),
         }
@@ -1169,9 +1169,9 @@ impl InstructionPredicate {
         self
     }
 
-    pub fn rust_predicate(&self) -> String {
+    pub fn rust_predicate(&self, func_str: &str) -> String {
         match &self.node {
-            Some(root) => root.rust_predicate(),
+            Some(root) => root.rust_predicate(func_str),
             None => "true".into(),
         }
     }
diff --git a/cranelift/codegen/meta/src/gen_encodings.rs b/cranelift/codegen/meta/src/gen_encodings.rs
index a2d1da62a7..1a1a439c49 100644
--- a/cranelift/codegen/meta/src/gen_encodings.rs
+++ b/cranelift/codegen/meta/src/gen_encodings.rs
@@ -78,7 +78,7 @@ use crate::unique_table::UniqueSeqTable;
 fn emit_instp(instp: &InstructionPredicate, has_func: bool, fmt: &mut Formatter) {
     if instp.is_type_predicate() {
         fmt.line("let args = inst.arguments(&func.dfg.value_lists);");
-        fmt.line(instp.rust_predicate());
+        fmt.line(instp.rust_predicate("func"));
         return;
     }
 
@@ -127,7 +127,7 @@ fn emit_instp(instp: &InstructionPredicate, has_func: bool, fmt: &mut Formatter)
             // Silence dead argument.
             fmt.line("let _ = func;");
         }
-        fmtln!(fmt, "return {};", instp.rust_predicate());
+        fmtln!(fmt, "return {};", instp.rust_predicate("func"));
     });
     fmtln!(fmt, "}");
 
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index 8a52960f42..ac79f8af33 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -1,6 +1,7 @@
-use crate::cdsl::ast::{Def, DefPool, VarPool};
+use crate::cdsl::ast::{Def, DefPool, Expr, VarPool};
 use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::operands::Operand;
 use crate::cdsl::type_inference::Constraint;
 use crate::cdsl::typevar::{TypeSet, TypeVar};
 use crate::cdsl::xform::{Transform, TransformGroup, TransformGroups};
@@ -34,7 +35,7 @@ fn unwrap_inst(
     let iform = format_registry.get(inst.format);
 
     fmt.comment(format!(
-        "Unwrap {}",
+        "Unwrap fields from instruction format {}",
         def.to_comment_string(&transform.var_pool)
     ));
 
@@ -42,19 +43,54 @@ fn unwrap_inst(
     let arg_names = apply
         .args
         .iter()
-        .map(|arg| match arg.maybe_var() {
-            Some(var_index) => var_pool.get(var_index).name.as_ref(),
-            None => "_",
+        .enumerate()
+        .map(|(arg_num, arg)| match &arg {
+            Expr::Var(var_index) => var_pool.get(*var_index).name.as_ref(),
+            Expr::Literal(_) => {
+                let n = inst.imm_opnums.iter().position(|&i| i == arg_num).unwrap();
+                iform.imm_fields[n].member
+            }
         })
         .collect::>()
         .join(", ");
 
+    // May we need "args" in the values consumed by predicates?
+    let emit_args = iform.num_value_operands >= 1 || iform.has_value_list;
+
+    // We need a tuple:
+    // - if there's at least one value operand, then we emit a variable for the value, and the
+    // value list as args.
+    // - otherwise, if there's the count of immediate operands added to the presence of a value list exceeds one.
+    let need_tuple = if iform.num_value_operands >= 1 {
+        true
+    } else {
+        let mut imm_and_varargs = inst
+            .operands_in
+            .iter()
+            .filter(|op| op.is_immediate())
+            .count();
+        if iform.has_value_list {
+            imm_and_varargs += 1;
+        }
+        imm_and_varargs > 1
+    };
+
+    let maybe_args = if emit_args { ", args" } else { "" };
+    let defined_values = format!("{}{}", arg_names, maybe_args);
+
+    let tuple_or_value = if need_tuple {
+        format!("({})", defined_values)
+    } else {
+        defined_values
+    };
+
     fmtln!(
         fmt,
-        "let ({}, predicate) = if let crate::ir::InstructionData::{} {{",
-        arg_names,
+        "let {} = if let ir::InstructionData::{} {{",
+        tuple_or_value,
         iform.name
     );
+
     fmt.indent(|fmt| {
         // Fields are encoded directly.
         for field in &iform.imm_fields {
@@ -69,47 +105,61 @@ fn unwrap_inst(
 
         fmt.line("..");
         fmt.outdented_line("} = pos.func.dfg[inst] {");
-        fmt.line("let func = &pos.func;");
 
         if iform.has_value_list {
-            fmt.line("let args = args.as_slice(&func.dfg.value_lists);");
+            fmt.line("let args = args.as_slice(&pos.func.dfg.value_lists);");
         } else if iform.num_value_operands == 1 {
             fmt.line("let args = [arg];")
         }
 
         // Generate the values for the tuple.
-        fmt.line("(");
-        fmt.indent(|fmt| {
-            for (op_num, op) in inst.operands_in.iter().enumerate() {
+        let emit_one_value =
+            |fmt: &mut Formatter, needs_comma: bool, op_num: usize, op: &Operand| {
+                let comma = if needs_comma { "," } else { "" };
                 if op.is_immediate() {
                     let n = inst.imm_opnums.iter().position(|&i| i == op_num).unwrap();
-                    fmtln!(fmt, "{},", iform.imm_fields[n].member);
+                    fmtln!(fmt, "{}{}", iform.imm_fields[n].member, comma);
                 } else if op.is_value() {
                     let n = inst.value_opnums.iter().position(|&i| i == op_num).unwrap();
-                    fmtln!(fmt, "func.dfg.resolve_aliases(args[{}]),", n);
+                    fmtln!(fmt, "pos.func.dfg.resolve_aliases(args[{}]),", n);
                 } else if op.is_varargs() {
                     let n = inst.imm_opnums.iter().chain(inst.value_opnums.iter()).max().map(|n| n + 1).unwrap_or(0);
                     // We need to create a `Vec` here, as using a slice would result in a borrowck
                     // error later on.
                     fmtln!(fmt, "\
-                        args.iter().skip({}).map(|&arg| func.dfg.resolve_aliases(arg)).collect::>(),\
+                        args.iter().skip({}).map(|&arg| pos.func.dfg.resolve_aliases(arg)).collect::>(),\
                     ", n);
+                } else {
+                    // This is a value list argument.
+                    assert!(iform.has_value_list);
                 }
-            }
+            };
 
-            // Evaluate the instruction predicate if any.
-            fmt.multi_line(
-                &apply
-                    .inst_predicate_with_ctrl_typevar(format_registry, var_pool)
-                    .rust_predicate(),
-            );
-        });
-        fmt.line(")");
+        if need_tuple {
+            fmt.line("(");
+            fmt.indent(|fmt| {
+                for (op_num, op) in inst.operands_in.iter().enumerate() {
+                    let needs_comma = emit_args || op_num + 1 < inst.operands_in.len();
+                    emit_one_value(fmt, needs_comma, op_num, op);
+                }
+                if emit_args {
+                    fmt.line("args");
+                }
+            });
+            fmt.line(")");
+        } else {
+            // Only one of these can be true at the same time, otherwise we'd need a tuple.
+            emit_one_value(fmt, false, 0, &inst.operands_in[0]);
+            if emit_args {
+                fmt.line("args");
+            }
+        }
 
         fmt.outdented_line("} else {");
         fmt.line(r#"unreachable!("bad instruction format")"#);
     });
     fmtln!(fmt, "};");
+    fmt.empty_line();
 
     assert_eq!(inst.operands_in.len(), apply.args.len());
     for (i, op) in inst.operands_in.iter().enumerate() {
@@ -385,21 +435,37 @@ fn emit_dst_inst(def: &Def, def_pool: &DefPool, var_pool: &VarPool, fmt: &mut Fo
 /// `inst: Inst` is the variable to be replaced. It is pointed to by `pos: Cursor`.
 /// `dfg: DataFlowGraph` is available and mutable.
 fn gen_transform<'a>(
+    replace_inst: bool,
     transform: &'a Transform,
     format_registry: &FormatRegistry,
     type_sets: &mut UniqueTable<'a, TypeSet>,
     fmt: &mut Formatter,
 ) {
-    // Unwrap the source instruction, create local variables for the input variables.
-    let replace_inst = unwrap_inst(&transform, format_registry, fmt);
+    // Evaluate the instruction predicate if any.
+    let apply = &transform.def_pool.get(transform.src).apply;
 
-    // Emit any runtime checks; these will rebind `predicate` emitted by unwrap_inst().
+    let inst_predicate = apply
+        .inst_predicate_with_ctrl_typevar(format_registry, &transform.var_pool)
+        .rust_predicate("pos.func");
+
+    let has_extra_constraints = !transform.type_env.constraints.is_empty();
+    if has_extra_constraints {
+        // Extra constraints rely on the predicate being a variable that we can rebind as we add
+        // more constraint predicates.
+        fmt.multi_line(&format!("let predicate = {};", inst_predicate));
+    }
+
+    // Emit any runtime checks; these will rebind `predicate` emitted right above.
     for constraint in &transform.type_env.constraints {
         emit_runtime_typecheck(constraint, type_sets, fmt);
     }
 
     // Guard the actual expansion by `predicate`.
-    fmt.line("if predicate {");
+    if has_extra_constraints {
+        fmt.line("if predicate {");
+    } else {
+        fmt.multi_line(&format!("if {} {{", inst_predicate))
+    }
     fmt.indent(|fmt| {
         // Emit any constants that must be created before use.
         for (name, value) in transform.const_pool.iter() {
@@ -525,8 +591,17 @@ fn gen_transform_group<'a>(
                 for camel_name in sorted_inst_names {
                     fmtln!(fmt, "ir::Opcode::{} => {{", camel_name);
                     fmt.indent(|fmt| {
-                        for transform in inst_to_transforms.get(camel_name).unwrap() {
-                            gen_transform(transform, format_registry, type_sets, fmt);
+                        let transforms = inst_to_transforms.get(camel_name).unwrap();
+
+                        // Unwrap the source instruction, create local variables for the input variables.
+                        let replace_inst = unwrap_inst(&transforms[0], format_registry, fmt);
+                        fmt.empty_line();
+
+                        for (i, transform) in transforms.into_iter().enumerate() {
+                            if i > 0 {
+                                fmt.empty_line();
+                            }
+                            gen_transform(replace_inst, transform, format_registry, type_sets, fmt);
                         }
                     });
                     fmtln!(fmt, "}");
@@ -540,7 +615,7 @@ fn gen_transform_group<'a>(
                 for (inst_camel_name, func_name) in sorted_custom_legalizes {
                     fmtln!(fmt, "ir::Opcode::{} => {{", inst_camel_name);
                     fmt.indent(|fmt| {
-                        fmtln!(fmt, "{}(inst, pos.func, cfg, isa);", func_name);
+                        fmtln!(fmt, "{}(inst, func, cfg, isa);", func_name);
                         fmt.line("return true;");
                     });
                     fmtln!(fmt, "}");
@@ -558,7 +633,7 @@ fn gen_transform_group<'a>(
         match &group.chain_with {
             Some(group_id) => fmtln!(
                 fmt,
-                "{}(inst, pos.func, cfg, isa)",
+                "{}(inst, func, cfg, isa)",
                 transform_groups.get(*group_id).rust_name()
             ),
             None => fmt.line("false"),
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 00cbcb3a92..04951c3d5b 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -212,7 +212,9 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     );
 
     // Population count for baseline x86_64
-    let qv1 = var("qv1");
+    let x = var("x");
+    let r = var("r");
+
     let qv3 = var("qv3");
     let qv4 = var("qv4");
     let qv5 = var("qv5");
@@ -226,7 +228,6 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let qv13 = var("qv13");
     let qv14 = var("qv14");
     let qv15 = var("qv15");
-    let qv16 = var("qv16");
     let qc77 = var("qc77");
     #[allow(non_snake_case)]
     let qc0F = var("qc0F");
@@ -235,12 +236,12 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let imm64_1 = Literal::constant(&imm.imm64, 1);
     let imm64_4 = Literal::constant(&imm.imm64, 4);
     group.legalize(
-        def!(qv16 = popcnt.I64(qv1)),
+        def!(r = popcnt.I64(x)),
         vec![
-            def!(qv3 = ushr_imm(qv1, imm64_1)),
+            def!(qv3 = ushr_imm(x, imm64_1)),
             def!(qc77 = iconst(Literal::constant(&imm.imm64, 0x7777777777777777))),
             def!(qv4 = band(qv3, qc77)),
-            def!(qv5 = isub(qv1, qv4)),
+            def!(qv5 = isub(x, qv4)),
             def!(qv6 = ushr_imm(qv4, imm64_1)),
             def!(qv7 = band(qv6, qc77)),
             def!(qv8 = isub(qv5, qv7)),
@@ -253,11 +254,10 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
             def!(qv14 = band(qv13, qc0F)),
             def!(qc01 = iconst(Literal::constant(&imm.imm64, 0x0101010101010101))),
             def!(qv15 = imul(qv14, qc01)),
-            def!(qv16 = ushr_imm(qv15, Literal::constant(&imm.imm64, 56))),
+            def!(r = ushr_imm(qv15, Literal::constant(&imm.imm64, 56))),
         ],
     );
 
-    let lv1 = var("lv1");
     let lv3 = var("lv3");
     let lv4 = var("lv4");
     let lv5 = var("lv5");
@@ -271,19 +271,18 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let lv13 = var("lv13");
     let lv14 = var("lv14");
     let lv15 = var("lv15");
-    let lv16 = var("lv16");
     let lc77 = var("lc77");
     #[allow(non_snake_case)]
     let lc0F = var("lc0F");
     let lc01 = var("lc01");
 
     group.legalize(
-        def!(lv16 = popcnt.I32(lv1)),
+        def!(r = popcnt.I32(x)),
         vec![
-            def!(lv3 = ushr_imm(lv1, imm64_1)),
+            def!(lv3 = ushr_imm(x, imm64_1)),
             def!(lc77 = iconst(Literal::constant(&imm.imm64, 0x77777777))),
             def!(lv4 = band(lv3, lc77)),
-            def!(lv5 = isub(lv1, lv4)),
+            def!(lv5 = isub(x, lv4)),
             def!(lv6 = ushr_imm(lv4, imm64_1)),
             def!(lv7 = band(lv6, lc77)),
             def!(lv8 = isub(lv5, lv7)),
@@ -296,7 +295,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
             def!(lv14 = band(lv13, lc0F)),
             def!(lc01 = iconst(Literal::constant(&imm.imm64, 0x01010101))),
             def!(lv15 = imul(lv14, lc01)),
-            def!(lv16 = ushr_imm(lv15, Literal::constant(&imm.imm64, 24))),
+            def!(r = ushr_imm(lv15, Literal::constant(&imm.imm64, 24))),
         ],
     );
 

From 687604d33a1132d552bfc627283c2248fca6fdde Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 4 Jul 2019 19:48:10 +0200
Subject: [PATCH 2828/3084] [meta] Legalization: don't emit a spurious `if
 true` for transforms that always apply;

This enhances readability of the generated legalizer code by replacing
`if true { body }` with `body`.
---
 .../codegen/meta/src/cdsl/instructions.rs     | 19 +++++-----
 cranelift/codegen/meta/src/gen_encodings.rs   |  6 ++--
 cranelift/codegen/meta/src/gen_legalizer.rs   | 35 +++++++++++++------
 3 files changed, 38 insertions(+), 22 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index a62cf39e05..699e30cb3f 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -1169,17 +1169,18 @@ impl InstructionPredicate {
         self
     }
 
-    pub fn rust_predicate(&self, func_str: &str) -> String {
-        match &self.node {
-            Some(root) => root.rust_predicate(func_str),
-            None => "true".into(),
-        }
+    pub fn rust_predicate(&self, func_str: &str) -> Option {
+        self.node.as_ref().map(|root| root.rust_predicate(func_str))
     }
 
-    /// Returns true if the predicate only depends on type parameters (and not on an instruction
-    /// format).
-    pub fn is_type_predicate(&self) -> bool {
-        self.node.as_ref().unwrap().is_type_predicate()
+    /// Returns the type predicate if this is one, or None otherwise.
+    pub fn type_predicate(&self, func_str: &str) -> Option {
+        let node = self.node.as_ref().unwrap();
+        if node.is_type_predicate() {
+            Some(node.rust_predicate(func_str))
+        } else {
+            None
+        }
     }
 
     /// Returns references to all the nodes that are leaves in the condition (i.e. by flattening
diff --git a/cranelift/codegen/meta/src/gen_encodings.rs b/cranelift/codegen/meta/src/gen_encodings.rs
index 1a1a439c49..86046cfde1 100644
--- a/cranelift/codegen/meta/src/gen_encodings.rs
+++ b/cranelift/codegen/meta/src/gen_encodings.rs
@@ -76,9 +76,9 @@ use crate::unique_table::UniqueSeqTable;
 /// The generated code is an `if let` pattern match that falls through if the instruction has an
 /// unexpected format. This should lead to a panic.
 fn emit_instp(instp: &InstructionPredicate, has_func: bool, fmt: &mut Formatter) {
-    if instp.is_type_predicate() {
+    if let Some(type_predicate) = instp.type_predicate("func") {
         fmt.line("let args = inst.arguments(&func.dfg.value_lists);");
-        fmt.line(instp.rust_predicate("func"));
+        fmt.line(type_predicate);
         return;
     }
 
@@ -127,7 +127,7 @@ fn emit_instp(instp: &InstructionPredicate, has_func: bool, fmt: &mut Formatter)
             // Silence dead argument.
             fmt.line("let _ = func;");
         }
-        fmtln!(fmt, "return {};", instp.rust_predicate("func"));
+        fmtln!(fmt, "return {};", instp.rust_predicate("func").unwrap());
     });
     fmtln!(fmt, "}");
 
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index ac79f8af33..d71f97b10d 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -452,7 +452,11 @@ fn gen_transform<'a>(
     if has_extra_constraints {
         // Extra constraints rely on the predicate being a variable that we can rebind as we add
         // more constraint predicates.
-        fmt.multi_line(&format!("let predicate = {};", inst_predicate));
+        if let Some(pred) = &inst_predicate {
+            fmt.multi_line(&format!("let predicate = {};", pred));
+        } else {
+            fmt.line("let predicate = true;");
+        }
     }
 
     // Emit any runtime checks; these will rebind `predicate` emitted right above.
@@ -460,13 +464,7 @@ fn gen_transform<'a>(
         emit_runtime_typecheck(constraint, type_sets, fmt);
     }
 
-    // Guard the actual expansion by `predicate`.
-    if has_extra_constraints {
-        fmt.line("if predicate {");
-    } else {
-        fmt.multi_line(&format!("if {} {{", inst_predicate))
-    }
-    fmt.indent(|fmt| {
+    let do_expand = |fmt: &mut Formatter| {
         // Emit any constants that must be created before use.
         for (name, value) in transform.const_pool.iter() {
             fmtln!(
@@ -538,8 +536,25 @@ fn gen_transform<'a>(
         }
 
         fmt.line("return true;");
-    });
-    fmt.line("}");
+    };
+
+    // Guard the actual expansion by `predicate`.
+    if has_extra_constraints {
+        fmt.line("if predicate {");
+        fmt.indent(|fmt| {
+            do_expand(fmt);
+        });
+        fmt.line("}");
+    } else if let Some(pred) = &inst_predicate {
+        fmt.multi_line(&format!("if {} {{", pred));
+        fmt.indent(|fmt| {
+            do_expand(fmt);
+        });
+        fmt.line("}");
+    } else {
+        // Unconditional transform (there was no predicate), just emit it.
+        do_expand(fmt);
+    }
 }
 
 fn gen_transform_group<'a>(

From d3ef80147b02fc64795769348b46ad7ac8980dda Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 1 Oct 2019 17:21:52 +0200
Subject: [PATCH 2829/3084] [meta] Simplify handling of variable arguments in
 legalization;

If a legalization contains varargs, it defers creating the binding for
the varargs, making the generated code easier to understand.
---
 cranelift/codegen/meta/src/gen_legalizer.rs | 27 +++++++++++----------
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index d71f97b10d..13c4351fb4 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -44,6 +44,10 @@ fn unwrap_inst(
         .args
         .iter()
         .enumerate()
+        .filter(|(arg_num, _)| {
+            // Variable args are specially handled after extracting args.
+            !inst.operands_in[*arg_num].is_varargs()
+        })
         .map(|(arg_num, arg)| match &arg {
             Expr::Var(var_index) => var_pool.get(*var_index).name.as_ref(),
             Expr::Literal(_) => {
@@ -122,16 +126,9 @@ fn unwrap_inst(
                 } else if op.is_value() {
                     let n = inst.value_opnums.iter().position(|&i| i == op_num).unwrap();
                     fmtln!(fmt, "pos.func.dfg.resolve_aliases(args[{}]),", n);
-                } else if op.is_varargs() {
-                    let n = inst.imm_opnums.iter().chain(inst.value_opnums.iter()).max().map(|n| n + 1).unwrap_or(0);
-                    // We need to create a `Vec` here, as using a slice would result in a borrowck
-                    // error later on.
-                    fmtln!(fmt, "\
-                        args.iter().skip({}).map(|&arg| pos.func.dfg.resolve_aliases(arg)).collect::>(),\
-                    ", n);
                 } else {
-                    // This is a value list argument.
-                    assert!(iform.has_value_list);
+                    // This is a value list argument or a varargs.
+                    assert!(iform.has_value_list || op.is_varargs());
                 }
             };
 
@@ -167,10 +164,14 @@ fn unwrap_inst(
             let name = &var_pool
                 .get(apply.args[i].maybe_var().expect("vararg without name"))
                 .name;
-
-            // Above name is set to an `Vec` representing the varargs. However it is expected to be
-            // `&[Value]` below, so we borrow it.
-            fmtln!(fmt, "let {} = &{};", name, name);
+            let n = inst
+                .imm_opnums
+                .iter()
+                .chain(inst.value_opnums.iter())
+                .max()
+                .copied()
+                .unwrap_or(0);
+            fmtln!(fmt, "let {} = &Vec::from(&args[{}..]);", name, n);
         }
     }
 

From 566a1436347593f1892c0b9551142191e9cf4e50 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 1 Oct 2019 18:19:37 +0200
Subject: [PATCH 2830/3084] [meta] Add pub(crate) to more types;

This caught one unused method, allowing us to remove it.
---
 .../codegen/meta/src/cdsl/instructions.rs     | 33 ++++++++++---------
 cranelift/codegen/meta/src/cdsl/recipes.rs    |  8 ++---
 .../codegen/meta/src/cdsl/type_inference.rs   |  2 +-
 .../codegen/meta/src/isa/riscv/recipes.rs     |  2 +-
 cranelift/codegen/meta/src/isa/x86/recipes.rs |  2 +-
 5 files changed, 24 insertions(+), 23 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 699e30cb3f..0d27159cd9 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -18,12 +18,12 @@ use crate::shared::types::{Bool, Float, Int, Reference};
 use cranelift_codegen_shared::condcodes::IntCC;
 
 #[derive(Copy, Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
-pub struct OpcodeNumber(u32);
+pub(crate) struct OpcodeNumber(u32);
 entity_impl!(OpcodeNumber);
 
-pub type AllInstructions = PrimaryMap;
+pub(crate) type AllInstructions = PrimaryMap;
 
-pub struct InstructionGroupBuilder<'format_reg, 'all_inst> {
+pub(crate) struct InstructionGroupBuilder<'format_reg, 'all_inst> {
     _name: &'static str,
     _doc: &'static str,
     format_registry: &'format_reg FormatRegistry,
@@ -67,7 +67,7 @@ impl<'format_reg, 'all_inst> InstructionGroupBuilder<'format_reg, 'all_inst> {
 /// Every instruction must belong to exactly one instruction group. A given
 /// target architecture can support instructions from multiple groups, and it
 /// does not necessarily support all instructions in a group.
-pub struct InstructionGroup {
+pub(crate) struct InstructionGroup {
     _name: &'static str,
     _doc: &'static str,
     instructions: Vec,
@@ -85,20 +85,20 @@ impl InstructionGroup {
 /// Instructions can have parameters bound to them to specialize them for more specific encodings
 /// (e.g. the encoding for adding two float types may be different than that of adding two
 /// integer types)
-pub trait Bindable {
+pub(crate) trait Bindable {
     /// Bind a parameter to an instruction
     fn bind(&self, parameter: impl Into) -> BoundInstruction;
 }
 
 #[derive(Debug)]
-pub struct PolymorphicInfo {
+pub(crate) struct PolymorphicInfo {
     pub use_typevar_operand: bool,
     pub ctrl_typevar: TypeVar,
     pub other_typevars: Vec,
 }
 
 #[derive(Debug)]
-pub struct InstructionContent {
+pub(crate) struct InstructionContent {
     /// Instruction mnemonic, also becomes opcode name.
     pub name: String,
     pub camel_name: String,
@@ -153,7 +153,7 @@ pub struct InstructionContent {
 }
 
 #[derive(Clone, Debug)]
-pub struct Instruction {
+pub(crate) struct Instruction {
     content: Rc,
 }
 
@@ -221,7 +221,7 @@ impl fmt::Display for Instruction {
     }
 }
 
-pub struct InstructionBuilder {
+pub(crate) struct InstructionBuilder {
     name: String,
     doc: String,
     operands_in: Option>,
@@ -384,7 +384,7 @@ impl InstructionBuilder {
 
 /// A thin wrapper like Option, but with more precise semantics.
 #[derive(Clone)]
-pub enum ValueTypeOrAny {
+pub(crate) enum ValueTypeOrAny {
     ValueType(ValueType),
     Any,
 }
@@ -467,7 +467,7 @@ impl Display for Immediate {
 }
 
 #[derive(Clone)]
-pub struct BoundInstruction {
+pub(crate) struct BoundInstruction {
     pub inst: Instruction,
     pub value_types: Vec,
     pub immediate_values: Vec,
@@ -941,7 +941,7 @@ impl InstructionPredicateNode {
 }
 
 #[derive(Clone, Hash, PartialEq, Eq)]
-pub struct InstructionPredicate {
+pub(crate) struct InstructionPredicate {
     node: Option,
 }
 
@@ -1191,15 +1191,16 @@ impl InstructionPredicate {
 }
 
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
-pub struct InstructionPredicateNumber(u32);
+pub(crate) struct InstructionPredicateNumber(u32);
 entity_impl!(InstructionPredicateNumber);
 
-pub type InstructionPredicateMap = PrimaryMap;
+pub(crate) type InstructionPredicateMap =
+    PrimaryMap;
 
 /// A registry of predicates to help deduplicating them, during Encodings construction. When the
 /// construction process is over, it needs to be extracted with `extract` and associated to the
 /// TargetIsa.
-pub struct InstructionPredicateRegistry {
+pub(crate) struct InstructionPredicateRegistry {
     /// Maps a predicate number to its actual predicate.
     map: InstructionPredicateMap,
 
@@ -1231,7 +1232,7 @@ impl InstructionPredicateRegistry {
 }
 
 /// An instruction specification, containing an instruction that has bound types or not.
-pub enum InstSpec {
+pub(crate) enum InstSpec {
     Inst(Instruction),
     Bound(BoundInstruction),
 }
diff --git a/cranelift/codegen/meta/src/cdsl/recipes.rs b/cranelift/codegen/meta/src/cdsl/recipes.rs
index 8157e08864..ed31cbbfd8 100644
--- a/cranelift/codegen/meta/src/cdsl/recipes.rs
+++ b/cranelift/codegen/meta/src/cdsl/recipes.rs
@@ -103,7 +103,7 @@ impl Into for Stack {
 /// branch instructions. It is an `(origin, bits)` tuple describing the exact
 /// range that can be encoded in a branch instruction.
 #[derive(Clone)]
-pub struct EncodingRecipe {
+pub(crate) struct EncodingRecipe {
     /// Short mnemonic name for this recipe.
     pub name: String,
 
@@ -158,13 +158,13 @@ impl PartialEq for EncodingRecipe {
 impl Eq for EncodingRecipe {}
 
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
-pub struct EncodingRecipeNumber(u32);
+pub(crate) struct EncodingRecipeNumber(u32);
 entity_impl!(EncodingRecipeNumber);
 
-pub type Recipes = PrimaryMap;
+pub(crate) type Recipes = PrimaryMap;
 
 #[derive(Clone)]
-pub struct EncodingRecipeBuilder {
+pub(crate) struct EncodingRecipeBuilder {
     pub name: String,
     format: InstructionFormatIndex,
     pub base_size: u64,
diff --git a/cranelift/codegen/meta/src/cdsl/type_inference.rs b/cranelift/codegen/meta/src/cdsl/type_inference.rs
index 5181739969..9cb894d911 100644
--- a/cranelift/codegen/meta/src/cdsl/type_inference.rs
+++ b/cranelift/codegen/meta/src/cdsl/type_inference.rs
@@ -5,7 +5,7 @@ use std::collections::{HashMap, HashSet};
 use std::iter::FromIterator;
 
 #[derive(Debug, Hash, PartialEq, Eq)]
-pub enum Constraint {
+pub(crate) enum Constraint {
     /// Constraint specifying that a type var tv1 must be wider than or equal to type var tv2 at
     /// runtime. This requires that:
     /// 1) They have the same number of lanes
diff --git a/cranelift/codegen/meta/src/isa/riscv/recipes.rs b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
index f32bbd568c..b1297479ec 100644
--- a/cranelift/codegen/meta/src/isa/riscv/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
@@ -7,7 +7,7 @@ use crate::cdsl::regs::IsaRegs;
 use crate::shared::Definitions as SharedDefinitions;
 
 /// An helper to create recipes and use them when defining the RISCV encodings.
-pub struct RecipeGroup<'formats> {
+pub(crate) struct RecipeGroup<'formats> {
     /// Memoized format registry, to pass it to the builders.
     formats: &'formats FormatRegistry,
 
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index ce0999c04c..8063e9441f 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -164,7 +164,7 @@ fn replace_nonrex_constraints(
 /// Encodings, in encodings.rs. This is an idiosyncrasy of the x86 meta-language, and could be
 /// reconsidered later.
 #[derive(Clone)]
-pub struct Template<'builder> {
+pub(crate) struct Template<'builder> {
     /// Mapping of format indexes to format data, used in the build() method.
     formats: &'builder FormatRegistry,
 

From 1ccf056baf7f847b2788f0bc22a39d11eb2c948a Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 15 Oct 2019 10:25:12 +0200
Subject: [PATCH 2831/3084] [wasm] Make the ModuleTranslationState ctor public.

It's useful for consumers which don't want to translate a whole module,
but just need translation of functions.
---
 cranelift/wasm/src/state/module_state.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/cranelift/wasm/src/state/module_state.rs b/cranelift/wasm/src/state/module_state.rs
index b2cbca77eb..306b17a068 100644
--- a/cranelift/wasm/src/state/module_state.rs
+++ b/cranelift/wasm/src/state/module_state.rs
@@ -19,7 +19,8 @@ pub struct ModuleTranslationState {
 }
 
 impl ModuleTranslationState {
-    pub(crate) fn new() -> Self {
+    /// Creates a new empty ModuleTranslationState.
+    pub fn new() -> Self {
         ModuleTranslationState {
             wasm_types: PrimaryMap::new(),
         }

From 48ccb3e051a74b54e6286aff2e1e92be0a24febe Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 15 Oct 2019 10:15:06 +0200
Subject: [PATCH 2832/3084] [ci] Disable cargo fuzz on macos nightly;

It has started breaking in ways unrelated to Cranelift, making it hard
to spot real CI failures in Cranelift. We should re-enable it at some
point, but disable it in the meanwhile.
---
 .azure-pipelines.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml
index 9e0a4697eb..f3935c5af8 100644
--- a/.azure-pipelines.yml
+++ b/.azure-pipelines.yml
@@ -88,6 +88,7 @@ jobs:
     - bash: cargo install cargo-fuzz
       displayName: Install cargo-fuzz
       condition: and(succeeded(), eq(variables['toolchain'], 'nightly'))
+
     - bash: |
         fuzz_module="ffaefab69523eb11935a9b420d58826c8ea65c4c"
         cargo fuzz run fuzz_translate_module \
@@ -96,6 +97,7 @@ jobs:
       env:
         RUST_BACKTRACE: 1
       condition: and(succeeded(), eq(variables['toolchain'], 'nightly'))
+      continueOnError: true
 
 - job: Fuzz_regression
   displayName: Fuzz regression

From a5efd2a625bacd436ddcd6f6c985239f0314a05b Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 11 Oct 2019 16:44:40 +0200
Subject: [PATCH 2833/3084] [bugpoint] Cosmetic improvements;

- Mostly Rust improvements to make code look more idiomatic.
- Also reuses the code memory accross compilation, to avoid many
memory allocations.
---
 cranelift/src/bugpoint.rs | 123 ++++++++++++++++----------------------
 cranelift/src/disasm.rs   |   6 +-
 2 files changed, 56 insertions(+), 73 deletions(-)

diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index 90d475199f..4091f1b9a0 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -48,9 +48,7 @@ pub fn run(
         match reduce(isa, func, verbose) {
             Ok((func, crash_msg)) => {
                 println!("Crash message: {}", crash_msg);
-
                 println!("\n{}", func);
-
                 println!(
                     "{} ebbs {} insts -> {} ebbs {} insts",
                     orig_ebb_count,
@@ -69,6 +67,7 @@ pub fn run(
 enum MutationKind {
     /// The mutation reduced the amount of instructions or ebbs.
     Shrinked,
+
     /// The mutation only changed an instruction. Performing another round of mutations may only
     /// reduce the test case if another mutation shrank the test case.
     Changed,
@@ -83,7 +82,7 @@ trait Mutator {
 
     fn reduce(
         &mut self,
-        ccc: &mut CrashCheckContext,
+        context: &mut CrashCheckContext,
         mut func: Function,
         progress_bar_prefix: String,
         verbose: bool,
@@ -111,7 +110,7 @@ trait Mutator {
 
             progress.set_message(&msg);
 
-            match ccc.check_for_crash(&mutated_func) {
+            match context.check_for_crash(&mutated_func) {
                 CheckResult::Succeed => {
                     // Shrinking didn't hit the problem anymore, discard changes.
                     continue;
@@ -170,28 +169,17 @@ impl Mutator for RemoveInst {
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
-        if let Some((prev_ebb, prev_inst)) =
-            next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst)
-        {
+        next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst).map(|(prev_ebb, prev_inst)| {
             func.layout.remove_inst(prev_inst);
-            if func.layout.ebb_insts(prev_ebb).next().is_none() {
+            let msg = if func.layout.ebb_insts(prev_ebb).next().is_none() {
                 // Make sure empty ebbs are removed, as `next_inst_ret_prev` depends on non empty ebbs
                 func.layout.remove_ebb(prev_ebb);
-                Some((
-                    func,
-                    format!("Remove inst {} and empty ebb {}", prev_inst, prev_ebb),
-                    MutationKind::Shrinked,
-                ))
+                format!("Remove inst {} and empty ebb {}", prev_inst, prev_ebb)
             } else {
-                Some((
-                    func,
-                    format!("Remove inst {}", prev_inst),
-                    MutationKind::Shrinked,
-                ))
-            }
-        } else {
-            None
-        }
+                format!("Remove inst {}", prev_inst)
+            };
+            (func, msg, MutationKind::Shrinked)
+        })
     }
 }
 
@@ -222,24 +210,18 @@ impl Mutator for ReplaceInstWithIconst {
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
-        if let Some((_prev_ebb, prev_inst)) =
-            next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst)
-        {
+        next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst).map(|(_prev_ebb, prev_inst)| {
             let results = func.dfg.inst_results(prev_inst);
-            if results.len() == 1 {
+            let msg = if results.len() == 1 {
                 let ty = func.dfg.value_type(results[0]);
                 func.dfg.replace(prev_inst).iconst(ty, 0);
-                Some((
-                    func,
-                    format!("Replace inst {} with iconst.{}", prev_inst, ty),
-                    MutationKind::Changed,
-                ))
+                format!("Replace inst {} with iconst.{}", prev_inst, ty)
             } else {
-                Some((func, format!(""), MutationKind::Changed))
-            }
-        } else {
-            None
-        }
+                // Returns something so the harness tries replacement on following instructions.
+                format!("")
+            };
+            (func, msg, MutationKind::Changed)
+        })
     }
 }
 
@@ -270,18 +252,14 @@ impl Mutator for ReplaceInstWithTrap {
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
-        if let Some((_prev_ebb, prev_inst)) =
-            next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst)
-        {
+        next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst).map(|(_prev_ebb, prev_inst)| {
             func.dfg.replace(prev_inst).trap(TrapCode::User(0));
-            Some((
+            (
                 func,
                 format!("Replace inst {} with trap", prev_inst),
                 MutationKind::Changed,
-            ))
-        } else {
-            None
-        }
+            )
+        })
     }
 }
 
@@ -308,20 +286,18 @@ impl Mutator for RemoveEbb {
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
-        if let Some(next_ebb) = func.layout.next_ebb(self.ebb) {
+        func.layout.next_ebb(self.ebb).map(|next_ebb| {
             self.ebb = next_ebb;
             while let Some(inst) = func.layout.last_inst(self.ebb) {
                 func.layout.remove_inst(inst);
             }
             func.layout.remove_ebb(self.ebb);
-            Some((
+            (
                 func,
                 format!("Remove ebb {}", next_ebb),
                 MutationKind::Shrinked,
-            ))
-        } else {
-            None
-        }
+            )
+        })
     }
 }
 
@@ -568,13 +544,13 @@ fn next_inst_ret_prev(func: &Function, ebb: &mut Ebb, inst: &mut Inst) -> Option
     if let Some(next_inst) = func.layout.next_inst(*inst) {
         *inst = next_inst;
         return Some(prev);
-    } else if let Some(next_ebb) = func.layout.next_ebb(*ebb) {
+    }
+    if let Some(next_ebb) = func.layout.next_ebb(*ebb) {
         *ebb = next_ebb;
         *inst = func.layout.first_inst(*ebb).expect("no inst");
         return Some(prev);
-    } else {
-        return None;
     }
+    None
 }
 
 fn ebb_count(func: &Function) -> usize {
@@ -601,12 +577,12 @@ fn reduce(
     mut func: Function,
     verbose: bool,
 ) -> Result<(Function, String), String> {
-    let mut ccc = CrashCheckContext::new(isa);
+    let mut context = CrashCheckContext::new(isa);
 
-    match ccc.check_for_crash(&func) {
+    match context.check_for_crash(&func) {
         CheckResult::Succeed => {
             return Err(format!(
-                "Given function compiled successfully or gave an verifier error."
+                "Given function compiled successfully or gave a verifier error."
             ));
         }
         CheckResult::Crash(_) => {}
@@ -619,17 +595,17 @@ fn reduce(
         let mut phase = 0;
 
         loop {
-            let mut mutator = match phase {
-                0 => Box::new(RemoveInst::new(&func)) as Box,
-                1 => Box::new(ReplaceInstWithIconst::new(&func)) as Box,
-                2 => Box::new(ReplaceInstWithTrap::new(&func)) as Box,
-                3 => Box::new(RemoveEbb::new(&func)) as Box,
-                4 => Box::new(RemoveUnusedEntities::new()) as Box,
+            let mut mutator: Box = match phase {
+                0 => Box::new(RemoveInst::new(&func)),
+                1 => Box::new(ReplaceInstWithIconst::new(&func)),
+                2 => Box::new(ReplaceInstWithTrap::new(&func)),
+                3 => Box::new(RemoveEbb::new(&func)),
+                4 => Box::new(RemoveUnusedEntities::new()),
                 _ => break,
             };
 
             func = mutator.reduce(
-                &mut ccc,
+                &mut context,
                 func,
                 format!("pass {}", pass_idx),
                 verbose,
@@ -646,7 +622,7 @@ fn reduce(
         }
     }
 
-    let crash_msg = match ccc.check_for_crash(&func) {
+    let crash_msg = match context.check_for_crash(&func) {
         CheckResult::Succeed => unreachable!("Used to crash, but doesn't anymore???"),
         CheckResult::Crash(crash_msg) => crash_msg,
     };
@@ -658,18 +634,23 @@ struct CrashCheckContext<'a> {
     /// Cached `Context`, to prevent repeated allocation.
     context: Context,
 
+    /// Cached code memory, to prevent repeated allocation.
+    code_memory: Vec,
+
     /// The target isa to compile for.
     isa: &'a dyn TargetIsa,
 }
 
 fn get_panic_string(panic: Box) -> String {
     let panic = match panic.downcast::<&'static str>() {
-        Ok(panic_msg) => panic_msg.to_owned(),
+        Ok(panic_msg) => {
+            return panic_msg.to_string();
+        }
         Err(panic) => panic,
     };
     match panic.downcast::() {
         Ok(panic_msg) => *panic_msg,
-        Err(_) => "Box".to_owned(),
+        Err(_) => "Box".to_string(),
     }
 }
 
@@ -685,6 +666,7 @@ impl<'a> CrashCheckContext<'a> {
     fn new(isa: &'a dyn TargetIsa) -> Self {
         CrashCheckContext {
             context: Context::new(),
+            code_memory: Vec::new(),
             isa,
         }
     }
@@ -692,6 +674,8 @@ impl<'a> CrashCheckContext<'a> {
     #[cfg_attr(test, allow(unreachable_code))]
     fn check_for_crash(&mut self, func: &Function) -> CheckResult {
         self.context.clear();
+        self.code_memory.clear();
+
         self.context.func = func.clone();
 
         use std::io::Write;
@@ -702,9 +686,9 @@ impl<'a> CrashCheckContext<'a> {
         })) {
             Ok(Some(_)) => return CheckResult::Succeed,
             Ok(None) => {}
-            // The verifier panicked. compiling it will probably give the same panic.
+            // The verifier panicked. Compiling it will probably give the same panic.
             // We treat it as succeeding to make it possible to reduce for the actual error.
-            // FIXME prevent verifier panic on removing ebb1
+            // FIXME prevent verifier panic on removing ebb0.
             Err(_) => return CheckResult::Succeed,
         }
 
@@ -732,11 +716,10 @@ impl<'a> CrashCheckContext<'a> {
             let mut relocs = PrintRelocs::new(false);
             let mut traps = PrintTraps::new(false);
             let mut stackmaps = PrintStackmaps::new(false);
-            let mut mem = vec![];
 
             let _ = self.context.compile_and_emit(
                 self.isa,
-                &mut mem,
+                &mut self.code_memory,
                 &mut relocs,
                 &mut traps,
                 &mut stackmaps,
diff --git a/cranelift/src/disasm.rs b/cranelift/src/disasm.rs
index 9cd6851de7..78bf92999e 100644
--- a/cranelift/src/disasm.rs
+++ b/cranelift/src/disasm.rs
@@ -9,7 +9,7 @@ pub struct PrintRelocs {
 }
 
 impl PrintRelocs {
-    pub fn new(flag_print: bool) -> PrintRelocs {
+    pub fn new(flag_print: bool) -> Self {
         Self {
             flag_print,
             text: String::new(),
@@ -80,7 +80,7 @@ pub struct PrintTraps {
 }
 
 impl PrintTraps {
-    pub fn new(flag_print: bool) -> PrintTraps {
+    pub fn new(flag_print: bool) -> Self {
         Self {
             flag_print,
             text: String::new(),
@@ -102,7 +102,7 @@ pub struct PrintStackmaps {
 }
 
 impl PrintStackmaps {
-    pub fn new(flag_print: bool) -> PrintStackmaps {
+    pub fn new(flag_print: bool) -> Self {
         Self {
             flag_print,
             text: String::new(),

From 0340ddbb65949786b7c13c465abcc20bb90b9e5f Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 11 Oct 2019 17:00:43 +0200
Subject: [PATCH 2834/3084] [bugpoint] Move test content to the tests/
 directory;

---
 cranelift/src/bugpoint.rs                   | 86 +--------------------
 cranelift/{src => tests}/bugpoint_test.clif |  0
 cranelift/tests/bugpoint_test_expected.clif | 76 ++++++++++++++++++
 3 files changed, 79 insertions(+), 83 deletions(-)
 rename cranelift/{src => tests}/bugpoint_test.clif (100%)
 create mode 100644 cranelift/tests/bugpoint_test_expected.clif

diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index 4091f1b9a0..c2a845e1b8 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -742,7 +742,8 @@ mod tests {
 
     #[test]
     fn test_reduce() {
-        const TEST: &'static str = include_str!("./bugpoint_test.clif");
+        const TEST: &'static str = include_str!("../tests/bugpoint_test.clif");
+        const EXPECTED: &'static str = include_str!("../tests/bugpoint_test_expected.clif");
 
         let test_file = parse_test(TEST, ParseOptions::default()).unwrap();
 
@@ -752,89 +753,8 @@ mod tests {
 
         for (func, _) in test_file.functions {
             let (func, crash_msg) = reduce(isa, func, false).expect("Couldn't reduce test case");
-
             assert_eq!(crash_msg, "test crash");
-
-            assert_eq!(
-                format!("{}", func),
-                "function u0:0(i64, i64, i64) system_v {
-    sig0 = (i64, i64, i16, i64, i64, i64, i64, i64) system_v
-    fn0 = u0:95 sig0
-
-ebb0(v0: i64, v1: i64, v2: i64):
-    v113 -> v1
-    v124 -> v1
-    v136 -> v1
-    v148 -> v1
-    v160 -> v1
-    v185 -> v1
-    v222 -> v1
-    v237 -> v1
-    v241 -> v1
-    v256 -> v1
-    v262 -> v1
-    v105 = iconst.i64 0
-    trap user0
-
-ebb99(v804: i64, v1035: i64, v1037: i64, v1039: i64, v1044: i64, v1052: i16, v1057: i64):
-    v817 -> v1035
-    v830 -> v1037
-    v844 -> v1039
-    v857 -> v1039
-    v939 -> v1039
-    v1042 -> v1039
-    v1050 -> v1039
-    v908 -> v1044
-    v917 -> v1044
-    v921 -> v1044
-    v1043 -> v1044
-    v960 -> v1052
-    v990 -> v1052
-    v1051 -> v1052
-    v1055 -> v1052
-    v963 -> v1057
-    v1056 -> v1057
-    v1060 -> v1057
-    trap user0
-
-ebb101:
-    v829 = iconst.i64 0
-    v935 -> v829
-    v962 -> v829
-    v992 -> v829
-    v1036 -> v829
-    v1049 -> v829
-    trap user0
-
-ebb102:
-    v842 = iconst.i64 0
-    v976 -> v842
-    v989 -> v842
-    v1038 -> v842
-    v1061 -> v842
-    trap user0
-
-ebb105:
-    v883 = iconst.i64 0
-    v934 -> v883
-    v961 -> v883
-    v991 -> v883
-    v1005 -> v883
-    v1048 -> v883
-    trap user0
-
-ebb114:
-    v951 = iconst.i64 0
-    v988 -> v951
-    trap user0
-
-ebb117:
-    v987 = iconst.i64 0
-    call fn0(v0, v105, v1052, v883, v829, v987, v951, v842)
-    trap user0
-}
-"
-            );
+            assert_eq!(format!("{}", func), EXPECTED.replace("\r\n", "\n"));
         }
     }
 }
diff --git a/cranelift/src/bugpoint_test.clif b/cranelift/tests/bugpoint_test.clif
similarity index 100%
rename from cranelift/src/bugpoint_test.clif
rename to cranelift/tests/bugpoint_test.clif
diff --git a/cranelift/tests/bugpoint_test_expected.clif b/cranelift/tests/bugpoint_test_expected.clif
new file mode 100644
index 0000000000..0382b856f1
--- /dev/null
+++ b/cranelift/tests/bugpoint_test_expected.clif
@@ -0,0 +1,76 @@
+function u0:0(i64, i64, i64) system_v {
+    sig0 = (i64, i64, i16, i64, i64, i64, i64, i64) system_v
+    fn0 = u0:95 sig0
+
+ebb0(v0: i64, v1: i64, v2: i64):
+    v113 -> v1
+    v124 -> v1
+    v136 -> v1
+    v148 -> v1
+    v160 -> v1
+    v185 -> v1
+    v222 -> v1
+    v237 -> v1
+    v241 -> v1
+    v256 -> v1
+    v262 -> v1
+    v105 = iconst.i64 0
+    trap user0
+
+ebb99(v804: i64, v1035: i64, v1037: i64, v1039: i64, v1044: i64, v1052: i16, v1057: i64):
+    v817 -> v1035
+    v830 -> v1037
+    v844 -> v1039
+    v857 -> v1039
+    v939 -> v1039
+    v1042 -> v1039
+    v1050 -> v1039
+    v908 -> v1044
+    v917 -> v1044
+    v921 -> v1044
+    v1043 -> v1044
+    v960 -> v1052
+    v990 -> v1052
+    v1051 -> v1052
+    v1055 -> v1052
+    v963 -> v1057
+    v1056 -> v1057
+    v1060 -> v1057
+    trap user0
+
+ebb101:
+    v829 = iconst.i64 0
+    v935 -> v829
+    v962 -> v829
+    v992 -> v829
+    v1036 -> v829
+    v1049 -> v829
+    trap user0
+
+ebb102:
+    v842 = iconst.i64 0
+    v976 -> v842
+    v989 -> v842
+    v1038 -> v842
+    v1061 -> v842
+    trap user0
+
+ebb105:
+    v883 = iconst.i64 0
+    v934 -> v883
+    v961 -> v883
+    v991 -> v883
+    v1005 -> v883
+    v1048 -> v883
+    trap user0
+
+ebb114:
+    v951 = iconst.i64 0
+    v988 -> v951
+    trap user0
+
+ebb117:
+    v987 = iconst.i64 0
+    call fn0(v0, v105, v1052, v883, v829, v987, v951, v842)
+    trap user0
+}

From 6b7304cb14a3ea44c88d5182a87a8771c5c37e4f Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 11 Oct 2019 18:59:19 +0200
Subject: [PATCH 2835/3084] [bugpoint] Implement replacing a single instruction
 by several ones;

This allows replacing a function that has N results with N instructions
with the same result type. It also narrows down typing, so that
instructions creating F32/F64 values are replaced with a constant of the
correct type.
---
 cranelift/src/bugpoint.rs          | 118 +++++++++++++++++++++--------
 cranelift/tests/bugpoint_test.clif |   3 +-
 2 files changed, 88 insertions(+), 33 deletions(-)

diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index c2a845e1b8..bc6826eef6 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -2,8 +2,10 @@
 
 use crate::disasm::{PrintRelocs, PrintStackmaps, PrintTraps};
 use crate::utils::{parse_sets_and_triple, read_to_string};
+use cranelift_codegen::cursor::{Cursor, FuncCursor};
+use cranelift_codegen::ir::types::{F32, F64};
 use cranelift_codegen::ir::{
-    Ebb, FuncRef, Function, GlobalValueData, Inst, InstBuilder, InstructionData, StackSlots,
+    self, Ebb, FuncRef, Function, GlobalValueData, Inst, InstBuilder, InstructionData, StackSlots,
     TrapCode,
 };
 use cranelift_codegen::isa::TargetIsa;
@@ -65,8 +67,8 @@ pub fn run(
 }
 
 enum MutationKind {
-    /// The mutation reduced the amount of instructions or ebbs.
-    Shrinked,
+    /// The mutation raised or reduced the amount of instructions or ebbs.
+    ExpandedOrShrinked,
 
     /// The mutation only changed an instruction. Performing another round of mutations may only
     /// reduce the test case if another mutation shrank the test case.
@@ -118,18 +120,15 @@ trait Mutator {
                 CheckResult::Crash(_) => {
                     // Panic remained while shrinking, make changes definitive.
                     func = mutated_func;
-                    match mutation_kind {
-                        MutationKind::Shrinked => {
+                    let verb = match mutation_kind {
+                        MutationKind::ExpandedOrShrinked => {
                             *should_keep_reducing = true;
-                            if verbose {
-                                progress.println(format!("{}: shrink", msg));
-                            }
-                        }
-                        MutationKind::Changed => {
-                            if verbose {
-                                progress.println(format!("{}: changed", msg));
-                            }
+                            "shrink"
                         }
+                        MutationKind::Changed => "changed",
+                    };
+                    if verbose {
+                        progress.println(format!("{}: {}", msg, verb));
                     }
                 }
             }
@@ -178,18 +177,18 @@ impl Mutator for RemoveInst {
             } else {
                 format!("Remove inst {}", prev_inst)
             };
-            (func, msg, MutationKind::Shrinked)
+            (func, msg, MutationKind::ExpandedOrShrinked)
         })
     }
 }
 
-/// Try to replace instructions with `iconst`.
-struct ReplaceInstWithIconst {
+/// Try to replace instructions with `iconst` or `fconst`.
+struct ReplaceInstWithConst {
     ebb: Ebb,
     inst: Inst,
 }
 
-impl ReplaceInstWithIconst {
+impl ReplaceInstWithConst {
     fn new(func: &Function) -> Self {
         let first_ebb = func.layout.entry_block().unwrap();
         let first_inst = func.layout.first_inst(first_ebb).unwrap();
@@ -198,11 +197,27 @@ impl ReplaceInstWithIconst {
             inst: first_inst,
         }
     }
+
+    fn const_for_type<'f, T: InstBuilder<'f>>(builder: T, ty: ir::Type) -> &'static str {
+        // Try to keep the result type consistent, and default to an integer type
+        // otherwise: this will cover all the cases for f32/f64 and integer types, or
+        // create verifier errors otherwise.
+        if ty == F32 {
+            builder.f32const(0.0);
+            "f32const"
+        } else if ty == F64 {
+            builder.f64const(0.0);
+            "f64const"
+        } else {
+            builder.iconst(ty, 0);
+            "iconst"
+        }
+    }
 }
 
-impl Mutator for ReplaceInstWithIconst {
+impl Mutator for ReplaceInstWithConst {
     fn name(&self) -> &'static str {
-        "replace inst with iconst"
+        "replace inst with const"
     }
 
     fn mutation_count(&self, func: &Function) -> Option {
@@ -211,16 +226,57 @@ impl Mutator for ReplaceInstWithIconst {
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
         next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst).map(|(_prev_ebb, prev_inst)| {
-            let results = func.dfg.inst_results(prev_inst);
-            let msg = if results.len() == 1 {
-                let ty = func.dfg.value_type(results[0]);
-                func.dfg.replace(prev_inst).iconst(ty, 0);
-                format!("Replace inst {} with iconst.{}", prev_inst, ty)
-            } else {
-                // Returns something so the harness tries replacement on following instructions.
-                format!("")
-            };
-            (func, msg, MutationKind::Changed)
+            let num_results = func.dfg.inst_results(prev_inst).len();
+
+            if num_results == 0 {
+                // Short-circuit: lie and say we've changed something so subsequent instructions
+                // still get replaced.
+                return (func, format!(""), MutationKind::Changed);
+            }
+
+            if num_results == 1 {
+                let ty = func.dfg.value_type(func.dfg.first_result(prev_inst));
+                let new_inst_name = Self::const_for_type(func.dfg.replace(prev_inst), ty);
+                return (
+                    func,
+                    format!("Replace inst {} with {}.", prev_inst, new_inst_name),
+                    MutationKind::Changed,
+                );
+            }
+
+            // At least 2 results. Replace each instruction with as many const instructions as
+            // there are results.
+            let mut pos = FuncCursor::new(&mut func).at_inst(prev_inst);
+
+            // Copy result SSA names into our own vector; otherwise we couldn't mutably borrow pos
+            // in the loop below.
+            let results = pos
+                .func
+                .dfg
+                .inst_results(prev_inst)
+                .iter()
+                .cloned()
+                .collect::>();
+
+            // Detach results from the previous instruction, since we're going to reuse them.
+            pos.func.dfg.clear_results(prev_inst);
+
+            let mut inst_names = Vec::new();
+            for r in results {
+                let ty = pos.func.dfg.value_type(r);
+                let builder = pos.ins().with_results([Some(r)]);
+                let new_inst_name = Self::const_for_type(builder, ty);
+                inst_names.push(new_inst_name);
+            }
+
+            // Remove the instruction.
+            assert_eq!(pos.remove_inst(), prev_inst);
+
+            (
+                func,
+                format!("Replace inst {} with {}", prev_inst, inst_names.join(" / ")),
+                MutationKind::ExpandedOrShrinked,
+            )
         })
     }
 }
@@ -295,7 +351,7 @@ impl Mutator for RemoveEbb {
             (
                 func,
                 format!("Remove ebb {}", next_ebb),
-                MutationKind::Shrinked,
+                MutationKind::ExpandedOrShrinked,
             )
         })
     }
@@ -597,7 +653,7 @@ fn reduce(
         loop {
             let mut mutator: Box = match phase {
                 0 => Box::new(RemoveInst::new(&func)),
-                1 => Box::new(ReplaceInstWithIconst::new(&func)),
+                1 => Box::new(ReplaceInstWithConst::new(&func)),
                 2 => Box::new(ReplaceInstWithTrap::new(&func)),
                 3 => Box::new(RemoveEbb::new(&func)),
                 4 => Box::new(RemoveUnusedEntities::new()),
diff --git a/cranelift/tests/bugpoint_test.clif b/cranelift/tests/bugpoint_test.clif
index 157411ed75..772b36d58e 100644
--- a/cranelift/tests/bugpoint_test.clif
+++ b/cranelift/tests/bugpoint_test.clif
@@ -300,8 +300,7 @@ ebb0(v0: i64, v1: i64, v2: i64):
     v241 -> v1
     v256 -> v1
     v262 -> v1
-    v3 = stack_addr.i64 ss0
-    v4 = load.i64 aligned v2
+    v3, v4 = x86_sdivmodx v0, v1, v2
     store aligned v4, v3
     v5 = load.i64 aligned v2+8
     store aligned v5, v3+8

From 012fca61f9a07987e1844fccc6f5cf623ee03edb Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 14 Oct 2019 15:37:28 +0200
Subject: [PATCH 2836/3084] [bugpoint] Use a unique progress bar and simplify
 Mutator trait;

---
 cranelift/src/bugpoint.rs | 121 +++++++++++++++++---------------------
 1 file changed, 54 insertions(+), 67 deletions(-)

diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index bc6826eef6..f8c266fe4b 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -77,68 +77,8 @@ enum MutationKind {
 
 trait Mutator {
     fn name(&self) -> &'static str;
-
     fn mutation_count(&self, func: &Function) -> Option;
-
     fn mutate(&mut self, func: Function) -> Option<(Function, String, MutationKind)>;
-
-    fn reduce(
-        &mut self,
-        context: &mut CrashCheckContext,
-        mut func: Function,
-        progress_bar_prefix: String,
-        verbose: bool,
-        should_keep_reducing: &mut bool,
-    ) -> Function {
-        let progress = ProgressBar::with_draw_target(
-            self.mutation_count(&func).unwrap_or(0) as u64,
-            ProgressDrawTarget::stdout(),
-        );
-        progress.set_style(
-            ProgressStyle::default_bar().template("{bar:60} {prefix:40} {pos:>4}/{len:>4} {msg}"),
-        );
-
-        progress.set_prefix(&(progress_bar_prefix + &format!(" phase {}", self.name())));
-
-        for _ in 0..10000 {
-            progress.inc(1);
-
-            let (mutated_func, msg, mutation_kind) = match self.mutate(func.clone()) {
-                Some(res) => res,
-                None => {
-                    break;
-                }
-            };
-
-            progress.set_message(&msg);
-
-            match context.check_for_crash(&mutated_func) {
-                CheckResult::Succeed => {
-                    // Shrinking didn't hit the problem anymore, discard changes.
-                    continue;
-                }
-                CheckResult::Crash(_) => {
-                    // Panic remained while shrinking, make changes definitive.
-                    func = mutated_func;
-                    let verb = match mutation_kind {
-                        MutationKind::ExpandedOrShrinked => {
-                            *should_keep_reducing = true;
-                            "shrink"
-                        }
-                        MutationKind::Changed => "changed",
-                    };
-                    if verbose {
-                        progress.println(format!("{}: {}", msg, verb));
-                    }
-                }
-            }
-        }
-
-        progress.set_message("done");
-        progress.finish();
-
-        func
-    }
 }
 
 /// Try to remove instructions.
@@ -646,6 +586,11 @@ fn reduce(
 
     resolve_aliases(&mut func);
 
+    let progress_bar = ProgressBar::with_draw_target(0, ProgressDrawTarget::stdout());
+    progress_bar.set_style(
+        ProgressStyle::default_bar().template("{bar:60} {prefix:40} {pos:>4}/{len:>4} {msg}"),
+    );
+
     for pass_idx in 0..100 {
         let mut should_keep_reducing = false;
         let mut phase = 0;
@@ -660,17 +605,57 @@ fn reduce(
                 _ => break,
             };
 
-            func = mutator.reduce(
-                &mut context,
-                func,
-                format!("pass {}", pass_idx),
-                verbose,
-                &mut should_keep_reducing,
-            );
+            progress_bar.set_prefix(&format!("pass {} phase {}", pass_idx, mutator.name()));
+            progress_bar.set_length(mutator.mutation_count(&func).unwrap() as u64);
+
+            // Reset progress bar.
+            progress_bar.set_position(0);
+            progress_bar.set_draw_delta(0);
+
+            for _ in 0..10000 {
+                progress_bar.inc(1);
+
+                let (mutated_func, msg, mutation_kind) = match mutator.mutate(func.clone()) {
+                    Some(res) => res,
+                    None => {
+                        break;
+                    }
+                };
+
+                progress_bar.set_message(&msg);
+
+                match context.check_for_crash(&mutated_func) {
+                    CheckResult::Succeed => {
+                        // Shrinking didn't hit the problem anymore, discard changes.
+                        continue;
+                    }
+                    CheckResult::Crash(_) => {
+                        // Panic remained while shrinking, make changes definitive.
+                        func = mutated_func;
+                        let verb = match mutation_kind {
+                            MutationKind::ExpandedOrShrinked => {
+                                should_keep_reducing = true;
+                                "shrink"
+                            }
+                            MutationKind::Changed => "changed",
+                        };
+                        if verbose {
+                            progress_bar.println(format!("{}: {}", msg, verb));
+                        }
+                    }
+                }
+            }
 
             phase += 1;
         }
 
+        progress_bar.println(format!(
+            "After pass {}, remaining insts/ebbs: {}/{}",
+            pass_idx,
+            inst_count(&func),
+            ebb_count(&func)
+        ));
+
         if !should_keep_reducing {
             // No new shrinking opportunities have been found this pass. This means none will ever
             // be found. Skip the rest of the passes over the function.
@@ -678,6 +663,8 @@ fn reduce(
         }
     }
 
+    progress_bar.finish();
+
     let crash_msg = match context.check_for_crash(&func) {
         CheckResult::Succeed => unreachable!("Used to crash, but doesn't anymore???"),
         CheckResult::Crash(crash_msg) => crash_msg,

From 735d4c7aefa71a1e5b4d013bb2b38a9b203f2486 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 14 Oct 2019 15:39:28 +0200
Subject: [PATCH 2837/3084] [bugpoint] Make the mutation_count non optional;

---
 cranelift/src/bugpoint.rs | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index f8c266fe4b..0fc4f619e8 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -77,7 +77,7 @@ enum MutationKind {
 
 trait Mutator {
     fn name(&self) -> &'static str;
-    fn mutation_count(&self, func: &Function) -> Option;
+    fn mutation_count(&self, func: &Function) -> usize;
     fn mutate(&mut self, func: Function) -> Option<(Function, String, MutationKind)>;
 }
 
@@ -103,8 +103,8 @@ impl Mutator for RemoveInst {
         "remove inst"
     }
 
-    fn mutation_count(&self, func: &Function) -> Option {
-        Some(inst_count(func))
+    fn mutation_count(&self, func: &Function) -> usize {
+        inst_count(func)
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
@@ -160,8 +160,8 @@ impl Mutator for ReplaceInstWithConst {
         "replace inst with const"
     }
 
-    fn mutation_count(&self, func: &Function) -> Option {
-        Some(inst_count(func))
+    fn mutation_count(&self, func: &Function) -> usize {
+        inst_count(func)
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
@@ -243,8 +243,8 @@ impl Mutator for ReplaceInstWithTrap {
         "replace inst with trap"
     }
 
-    fn mutation_count(&self, func: &Function) -> Option {
-        Some(inst_count(func))
+    fn mutation_count(&self, func: &Function) -> usize {
+        inst_count(func)
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
@@ -277,8 +277,8 @@ impl Mutator for RemoveEbb {
         "remove ebb"
     }
 
-    fn mutation_count(&self, func: &Function) -> Option {
-        Some(ebb_count(func))
+    fn mutation_count(&self, func: &Function) -> usize {
+        ebb_count(func)
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
@@ -313,8 +313,8 @@ impl Mutator for RemoveUnusedEntities {
         "remove unused entities"
     }
 
-    fn mutation_count(&self, _func: &Function) -> Option {
-        Some(4)
+    fn mutation_count(&self, _func: &Function) -> usize {
+        4
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
@@ -606,7 +606,7 @@ fn reduce(
             };
 
             progress_bar.set_prefix(&format!("pass {} phase {}", pass_idx, mutator.name()));
-            progress_bar.set_length(mutator.mutation_count(&func).unwrap() as u64);
+            progress_bar.set_length(mutator.mutation_count(&func) as u64);
 
             // Reset progress bar.
             progress_bar.set_position(0);

From ab42f322d4ed4f3f047249eb9eba1e190a74b006 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 14 Oct 2019 15:46:44 +0200
Subject: [PATCH 2838/3084] [bugpoint] Don't test for a crash when a mutation
 doesn't change anything;

---
 cranelift/src/bugpoint.rs | 59 ++++++++++++++++++++++++++-------------
 1 file changed, 39 insertions(+), 20 deletions(-)

diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index 0fc4f619e8..cb5f6b58dd 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -66,19 +66,23 @@ pub fn run(
     Ok(())
 }
 
-enum MutationKind {
+enum ProgressStatus {
     /// The mutation raised or reduced the amount of instructions or ebbs.
     ExpandedOrShrinked,
 
     /// The mutation only changed an instruction. Performing another round of mutations may only
     /// reduce the test case if another mutation shrank the test case.
     Changed,
+
+    /// No need to re-test if the program crashes, because the mutation had no effect, but we want
+    /// to keep on iterating.
+    Skip,
 }
 
 trait Mutator {
     fn name(&self) -> &'static str;
     fn mutation_count(&self, func: &Function) -> usize;
-    fn mutate(&mut self, func: Function) -> Option<(Function, String, MutationKind)>;
+    fn mutate(&mut self, func: Function) -> Option<(Function, String, ProgressStatus)>;
 }
 
 /// Try to remove instructions.
@@ -107,7 +111,7 @@ impl Mutator for RemoveInst {
         inst_count(func)
     }
 
-    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
+    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, ProgressStatus)> {
         next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst).map(|(prev_ebb, prev_inst)| {
             func.layout.remove_inst(prev_inst);
             let msg = if func.layout.ebb_insts(prev_ebb).next().is_none() {
@@ -117,7 +121,7 @@ impl Mutator for RemoveInst {
             } else {
                 format!("Remove inst {}", prev_inst)
             };
-            (func, msg, MutationKind::ExpandedOrShrinked)
+            (func, msg, ProgressStatus::ExpandedOrShrinked)
         })
     }
 }
@@ -164,14 +168,17 @@ impl Mutator for ReplaceInstWithConst {
         inst_count(func)
     }
 
-    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
+    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, ProgressStatus)> {
         next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst).map(|(_prev_ebb, prev_inst)| {
             let num_results = func.dfg.inst_results(prev_inst).len();
 
-            if num_results == 0 {
-                // Short-circuit: lie and say we've changed something so subsequent instructions
-                // still get replaced.
-                return (func, format!(""), MutationKind::Changed);
+            let opcode = func.dfg[prev_inst].opcode();
+            if num_results == 0
+                || opcode == ir::Opcode::Iconst
+                || opcode == ir::Opcode::F32const
+                || opcode == ir::Opcode::F64const
+            {
+                return (func, format!(""), ProgressStatus::Skip);
             }
 
             if num_results == 1 {
@@ -180,7 +187,7 @@ impl Mutator for ReplaceInstWithConst {
                 return (
                     func,
                     format!("Replace inst {} with {}.", prev_inst, new_inst_name),
-                    MutationKind::Changed,
+                    ProgressStatus::Changed,
                 );
             }
 
@@ -215,7 +222,7 @@ impl Mutator for ReplaceInstWithConst {
             (
                 func,
                 format!("Replace inst {} with {}", prev_inst, inst_names.join(" / ")),
-                MutationKind::ExpandedOrShrinked,
+                ProgressStatus::ExpandedOrShrinked,
             )
         })
     }
@@ -247,13 +254,18 @@ impl Mutator for ReplaceInstWithTrap {
         inst_count(func)
     }
 
-    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
+    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, ProgressStatus)> {
         next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst).map(|(_prev_ebb, prev_inst)| {
-            func.dfg.replace(prev_inst).trap(TrapCode::User(0));
+            let status = if func.dfg[prev_inst].opcode() == ir::Opcode::Trap {
+                ProgressStatus::Skip
+            } else {
+                func.dfg.replace(prev_inst).trap(TrapCode::User(0));
+                ProgressStatus::Changed
+            };
             (
                 func,
                 format!("Replace inst {} with trap", prev_inst),
-                MutationKind::Changed,
+                status,
             )
         })
     }
@@ -281,7 +293,7 @@ impl Mutator for RemoveEbb {
         ebb_count(func)
     }
 
-    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
+    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, ProgressStatus)> {
         func.layout.next_ebb(self.ebb).map(|next_ebb| {
             self.ebb = next_ebb;
             while let Some(inst) = func.layout.last_inst(self.ebb) {
@@ -291,7 +303,7 @@ impl Mutator for RemoveEbb {
             (
                 func,
                 format!("Remove ebb {}", next_ebb),
-                MutationKind::ExpandedOrShrinked,
+                ProgressStatus::ExpandedOrShrinked,
             )
         })
     }
@@ -317,7 +329,7 @@ impl Mutator for RemoveUnusedEntities {
         4
     }
 
-    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, MutationKind)> {
+    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, ProgressStatus)> {
         let name = match self.kind {
             0 => {
                 let mut ext_func_usage_map = HashMap::new();
@@ -531,7 +543,7 @@ impl Mutator for RemoveUnusedEntities {
             _ => return None,
         };
         self.kind += 1;
-        Some((func, name.to_owned(), MutationKind::Changed))
+        Some((func, name.to_owned(), ProgressStatus::Changed))
     }
 }
 
@@ -622,6 +634,12 @@ fn reduce(
                     }
                 };
 
+                if let ProgressStatus::Skip = mutation_kind {
+                    // The mutator didn't change anything, but we want to try more mutator
+                    // iterations.
+                    continue;
+                }
+
                 progress_bar.set_message(&msg);
 
                 match context.check_for_crash(&mutated_func) {
@@ -633,11 +651,12 @@ fn reduce(
                         // Panic remained while shrinking, make changes definitive.
                         func = mutated_func;
                         let verb = match mutation_kind {
-                            MutationKind::ExpandedOrShrinked => {
+                            ProgressStatus::ExpandedOrShrinked => {
                                 should_keep_reducing = true;
                                 "shrink"
                             }
-                            MutationKind::Changed => "changed",
+                            ProgressStatus::Changed => "changed",
+                            ProgressStatus::Skip => unreachable!(),
                         };
                         if verbose {
                             progress_bar.println(format!("{}: {}", msg, verb));

From 5b274ed3ba033495f5af5f4a6fad86bdceb7dcdb Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 14 Oct 2019 15:49:47 +0200
Subject: [PATCH 2839/3084] [bugpoint] Merge consecutive blocks (fixes #1124);

---
 cranelift/Cargo.toml      |   1 +
 cranelift/src/bugpoint.rs | 151 ++++++++++++++++++++++++++++++++++++--
 2 files changed, 146 insertions(+), 6 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 9229936326..deef8c430d 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -57,5 +57,6 @@ basic-blocks = ["cranelift-codegen/basic-blocks", "cranelift-frontend/basic-bloc
 # necessary.
 [profile.release]
 debug = true
+# debug-assertions = true # uncomment to make bugpoint blazingly fast!
 [profile.bench]
 debug = true
diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index cb5f6b58dd..2d753548b8 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -3,6 +3,7 @@
 use crate::disasm::{PrintRelocs, PrintStackmaps, PrintTraps};
 use crate::utils::{parse_sets_and_triple, read_to_string};
 use cranelift_codegen::cursor::{Cursor, FuncCursor};
+use cranelift_codegen::flowgraph::ControlFlowGraph;
 use cranelift_codegen::ir::types::{F32, F64};
 use cranelift_codegen::ir::{
     self, Ebb, FuncRef, Function, GlobalValueData, Inst, InstBuilder, InstructionData, StackSlots,
@@ -83,6 +84,10 @@ trait Mutator {
     fn name(&self) -> &'static str;
     fn mutation_count(&self, func: &Function) -> usize;
     fn mutate(&mut self, func: Function) -> Option<(Function, String, ProgressStatus)>;
+
+    /// Gets called when the returned mutated function kept on causing the crash. This can be used
+    /// to update position of the next item to look at. Does nothing by default.
+    fn did_crash(&mut self) {}
 }
 
 /// Try to remove instructions.
@@ -547,6 +552,113 @@ impl Mutator for RemoveUnusedEntities {
     }
 }
 
+struct MergeBlocks {
+    ebb: Ebb,
+    prev_ebb: Option,
+}
+
+impl MergeBlocks {
+    fn new(func: &Function) -> Self {
+        Self {
+            ebb: func.layout.entry_block().unwrap(),
+            prev_ebb: None,
+        }
+    }
+}
+
+impl Mutator for MergeBlocks {
+    fn name(&self) -> &'static str {
+        "merge blocks"
+    }
+
+    fn mutation_count(&self, func: &Function) -> usize {
+        // N ebbs may result in at most N-1 merges.
+        ebb_count(func) - 1
+    }
+
+    fn mutate(&mut self, mut func: Function) -> Option<(Function, String, ProgressStatus)> {
+        let ebb = match func.layout.next_ebb(self.ebb) {
+            Some(ebb) => ebb,
+            None => return None,
+        };
+
+        self.ebb = ebb;
+
+        let mut cfg = ControlFlowGraph::new();
+        cfg.compute(&func);
+
+        if cfg.pred_iter(ebb).count() != 1 {
+            return Some((
+                func,
+                format!("did nothing for {}", ebb),
+                ProgressStatus::Skip,
+            ));
+        }
+
+        let pred = cfg.pred_iter(ebb).next().unwrap();
+
+        #[cfg(feature = "basic-blocks")]
+        {
+            // If the branch instruction that lead us to this block is preceded by another branch
+            // instruction, then we have a conditional jump sequence that we should not break by
+            // replacing the second instruction by more of them.
+            if let Some(pred_pred_inst) = func.layout.prev_inst(pred.inst) {
+                if func.dfg[pred_pred_inst].opcode().is_branch() {
+                    return Some((
+                        func,
+                        format!("did nothing for {}", ebb),
+                        ProgressStatus::Skip,
+                    ));
+                }
+            }
+        }
+
+        assert!(func.dfg.ebb_params(ebb).len() == func.dfg.inst_variable_args(pred.inst).len());
+
+        // If there were any EBB parameters in ebb, then the last instruction in pred will
+        // fill these parameters. Make the EBB params aliases of the terminator arguments.
+        for (ebb_param, arg) in func
+            .dfg
+            .detach_ebb_params(ebb)
+            .as_slice(&func.dfg.value_lists)
+            .iter()
+            .cloned()
+            .zip(func.dfg.inst_variable_args(pred.inst).iter().cloned())
+            .collect::>()
+        {
+            if ebb_param != arg {
+                func.dfg.change_to_alias(ebb_param, arg);
+            }
+        }
+
+        // Remove the terminator branch to the current EBB.
+        func.layout.remove_inst(pred.inst);
+
+        // Move all the instructions to the predecessor.
+        while let Some(inst) = func.layout.first_inst(ebb) {
+            func.layout.remove_inst(inst);
+            func.layout.append_inst(inst, pred.ebb);
+        }
+
+        // Remove the predecessor EBB.
+        func.layout.remove_ebb(ebb);
+
+        // Record the previous EBB: if we caused a crash (as signaled by a call to did_crash), then
+        // we'll start back to this EBB.
+        self.prev_ebb = Some(pred.ebb);
+
+        return Some((
+            func,
+            format!("merged {} and {}", pred.ebb, ebb),
+            ProgressStatus::ExpandedOrShrinked,
+        ));
+    }
+
+    fn did_crash(&mut self) {
+        self.ebb = self.prev_ebb.unwrap();
+    }
+}
+
 fn next_inst_ret_prev(func: &Function, ebb: &mut Ebb, inst: &mut Inst) -> Option<(Ebb, Inst)> {
     let prev = (*ebb, *inst);
     if let Some(next_inst) = func.layout.next_inst(*inst) {
@@ -614,6 +726,7 @@ fn reduce(
                 2 => Box::new(ReplaceInstWithTrap::new(&func)),
                 3 => Box::new(RemoveEbb::new(&func)),
                 4 => Box::new(RemoveUnusedEntities::new()),
+                5 => Box::new(MergeBlocks::new(&func)),
                 _ => break,
             };
 
@@ -644,12 +757,16 @@ fn reduce(
 
                 match context.check_for_crash(&mutated_func) {
                     CheckResult::Succeed => {
-                        // Shrinking didn't hit the problem anymore, discard changes.
+                        // Mutating didn't hit the problem anymore, discard changes.
                         continue;
                     }
                     CheckResult::Crash(_) => {
-                        // Panic remained while shrinking, make changes definitive.
+                        // Panic remained while mutating, make changes definitive.
                         func = mutated_func;
+
+                        // Notify the mutator that the mutation was successful.
+                        mutator.did_crash();
+
                         let verb = match mutation_kind {
                             ProgressStatus::ExpandedOrShrinked => {
                                 should_keep_reducing = true;
@@ -669,10 +786,15 @@ fn reduce(
         }
 
         progress_bar.println(format!(
-            "After pass {}, remaining insts/ebbs: {}/{}",
+            "After pass {}, remaining insts/ebbs: {}/{} ({})",
             pass_idx,
             inst_count(&func),
-            ebb_count(&func)
+            ebb_count(&func),
+            if should_keep_reducing {
+                "will keep reducing"
+            } else {
+                "stop reducing"
+            }
         ));
 
         if !should_keep_reducing {
@@ -814,9 +936,26 @@ mod tests {
         let isa = test_file.isa_spec.unique_isa().expect("Unknown isa");
 
         for (func, _) in test_file.functions {
-            let (func, crash_msg) = reduce(isa, func, false).expect("Couldn't reduce test case");
+            let (reduced_func, crash_msg) =
+                reduce(isa, func, false).expect("Couldn't reduce test case");
             assert_eq!(crash_msg, "test crash");
-            assert_eq!(format!("{}", func), EXPECTED.replace("\r\n", "\n"));
+
+            let (func_reduced_twice, crash_msg) =
+                reduce(isa, reduced_func.clone(), false).expect("Couldn't re-reduce test case");
+            assert_eq!(crash_msg, "test crash");
+
+            assert_eq!(
+                ebb_count(&func_reduced_twice),
+                ebb_count(&reduced_func),
+                "reduction wasn't maximal for ebbs"
+            );
+            assert_eq!(
+                inst_count(&func_reduced_twice),
+                inst_count(&reduced_func),
+                "reduction wasn't maximal for insts"
+            );
+
+            assert_eq!(format!("{}", reduced_func), EXPECTED.replace("\r\n", "\n"));
         }
     }
 }

From ac4e93f97160e282452250cf9009fa5469557d33 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 15 Oct 2019 09:43:28 -0700
Subject: [PATCH 2840/3084] Bump version to 0.46.0

---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  2 +-
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 19 files changed, 68 insertions(+), 68 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index deef8c430d..b0f5e91e7d 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.45.0"
+version = "0.46.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.45.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.45.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.45.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.45.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.45.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.45.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.45.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.45.0" }
-cranelift-module = { path = "cranelift-module", version = "0.45.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.45.0" }
-cranelift-object = { path = "cranelift-object", version = "0.45.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.45.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.45.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.45.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.46.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.46.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.46.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.46.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.46.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.46.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.46.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.46.0" }
+cranelift-module = { path = "cranelift-module", version = "0.46.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.46.0" }
+cranelift-object = { path = "cranelift-object", version = "0.46.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.46.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.46.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.46.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index a8aa5387ad..bea45dcc6f 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.45.0"
+version = "0.46.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.45.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.46.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 9759c93491..563033ef95 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.45.0"
+version = "0.46.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.45.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.45.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.45.0" }
+cranelift-codegen-shared = { path = "./shared", version = "0.46.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.46.0" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.46.0" }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashbrown = { version = "0.6", optional = true }
@@ -29,7 +29,7 @@ smallvec = { version = "0.6.10" }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.45.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.46.0" }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index ae4bd31aff..49757a07aa 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.45.0"
+version = "0.46.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.45.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.45.0" }
+cranelift-codegen-shared = { path = "../shared", version = "0.46.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.46.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 079515a49f..7dd740670d 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.45.0"
+version = "0.46.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index e80350d003..b138268444 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.45.0"
+version = "0.46.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 7b1177126e..363a81fcd6 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.45.0"
+version = "0.46.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.45.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.46.0" }
 faerie = "0.11.0"
 goblin = "0.0.24"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index c737bdf260..144c1d0ab5 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.45.0"
+version = "0.46.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.45.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.45.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.45.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.46.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.46.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.46.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 6c437680c3..a9ef6c3481 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.45.0"
+version = "0.46.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", default-features = false }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 570ca26912..5d62d08d20 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.45.0"
+version = "0.46.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.45.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.46.0" }
 hashbrown = { version = "0.6", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 9f3af19cbf..ae57953f2d 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.45.0"
+version = "0.46.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", default-features = false }
 target-lexicon = "0.8.1"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index acefc54516..1e69a5e061 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.45.0"
+version = "0.46.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.45.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.46.0" }
 object = { version = "0.14.0", default-features = false, features = ["write"] }
 target-lexicon = "0.8.1"
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 3f8c87be37..6ab4e86f94 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.45.0"
+version = "0.46.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.45.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.46.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 51aaef2a12..456aa7845e 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.45.0"
+version="0.46.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 3119aaf263..a8ef0c4af7 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.45.0"
+version = "0.46.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0" }
 target-lexicon = "0.8.1"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 17b988dca4..6376186373 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.45.0"
+version = "0.46.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.45.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.46.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 6ea16fa229..214a730255 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.45.0"
+version = "0.46.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.45.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.45.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.46.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.46.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.45.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.45.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.45.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.46.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.46.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.46.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 6de7e230e2..8b0ca3297d 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.45.0"
+version = "0.46.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.45.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.46.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 5dd19f3998..ebec419cae 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.45.0"
+version = "0.46.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.39.2", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.45.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.45.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.45.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.46.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.46.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From bae0257fc3efb16769e155e0d822f35bbf75dd93 Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Mon, 14 Oct 2019 14:04:05 -0700
Subject: [PATCH 2841/3084] cranelift-wasm: Fix reachability tracking for `if
 .. else .. end`

We weren't previously keeping track of quite the right information for whether
an `if .. else .. end`'s following block was reachable or not. It should be
reachable if the head is reachable and either the consequent or alternative end
reachable (and therefore fall through to the following block) or do an early
`br_if` to it.

This commit rejiggers `ControlStackFrame::If` to keep track of reachability at
the end of the consequent (we don't need to keep track of it at the end of the
alternative, since that is simply `state.reachable`) and adds Wasm tests for
every reachability situation we can encounter with `if .. else .. end`.

Fixes #1132
---
 cranelift/wasm/src/code_translator.rs         | 163 +++++++++---------
 cranelift/wasm/src/state/func_state.rs        |  29 +++-
 .../if-reachability-translation-0.wat         |  12 ++
 .../if-reachability-translation-1.wat         |  12 ++
 .../if-reachability-translation-2.wat         |  12 ++
 .../if-reachability-translation-3.wat         |  12 ++
 .../if-reachability-translation-4.wat         |  12 ++
 .../if-reachability-translation-5.wat         |  14 ++
 .../if-reachability-translation-6.wat         |  14 ++
 9 files changed, 194 insertions(+), 86 deletions(-)
 create mode 100644 cranelift/wasmtests/if-reachability-translation-0.wat
 create mode 100644 cranelift/wasmtests/if-reachability-translation-1.wat
 create mode 100644 cranelift/wasmtests/if-reachability-translation-2.wat
 create mode 100644 cranelift/wasmtests/if-reachability-translation-3.wat
 create mode 100644 cranelift/wasmtests/if-reachability-translation-4.wat
 create mode 100644 cranelift/wasmtests/if-reachability-translation-5.wat
 create mode 100644 cranelift/wasmtests/if-reachability-translation-6.wat

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index bc75157a88..6dd7deb1ff 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -200,62 +200,69 @@ pub fn translate_operator(
             let i = state.control_stack.len() - 1;
             match state.control_stack[i] {
                 ControlStackFrame::If {
-                    else_data: ElseData::NoElse { branch_inst },
-                    ref mut reachable_from_top,
+                    ref else_data,
+                    head_is_reachable,
+                    ref mut consequent_ends_reachable,
+                    num_return_values,
                     blocktype,
                     destination,
                     ..
                 } => {
-                    // We take the control frame pushed by the if, use its ebb
-                    // as the else body and push a new control frame with a new
-                    // ebb for the code after the if/then/else. At the end of the
-                    // then clause we jump to the destination.
+                    // We finished the consequent, so record its final
+                    // reachability state.
+                    debug_assert!(consequent_ends_reachable.is_none());
+                    *consequent_ends_reachable = Some(state.reachable);
 
-                    // The `if` has an `else`, so there's no branch to the end from the top.
-                    *reachable_from_top = false;
+                    if head_is_reachable {
+                        // We have a branch from the head of the `if` to the `else`.
+                        state.reachable = true;
 
-                    let (params, _results) =
-                        blocktype_params_results(module_translation_state, blocktype)?;
-                    let else_ebb = ebb_with_params(builder, params)?;
-                    builder.ins().jump(destination, state.peekn(params.len()));
-                    state.popn(params.len());
+                        // Ensure we have an ebb for the `else` block (it may have
+                        // already been pre-allocated, see `ElseData` for details).
+                        let else_ebb = match *else_data {
+                            ElseData::NoElse { branch_inst } => {
+                                let (params, _results) =
+                                    blocktype_params_results(module_translation_state, blocktype)?;
+                                debug_assert_eq!(params.len(), num_return_values);
+                                let else_ebb = ebb_with_params(builder, params)?;
+                                builder.ins().jump(destination, state.peekn(params.len()));
+                                state.popn(params.len());
 
-                    // You might be expecting that we push the parameters for this
-                    // `else` block here, something like this:
-                    //
-                    //     state.pushn(&control_stack_frame.params);
-                    //
-                    // We don't do that because they are already on the top of the stack
-                    // for us: we pushed the parameters twice when we saw the initial
-                    // `if` so that we wouldn't have to save the parameters in the
-                    // `ControlStackFrame` as another `Vec` allocation.
+                                builder.change_jump_destination(branch_inst, else_ebb);
+                                builder.seal_block(else_ebb);
+                                else_ebb
+                            }
+                            ElseData::WithElse { else_block } => {
+                                builder
+                                    .ins()
+                                    .jump(destination, state.peekn(num_return_values));
+                                state.popn(num_return_values);
+                                else_block
+                            }
+                        };
 
-                    builder.change_jump_destination(branch_inst, else_ebb);
-                    builder.seal_block(else_ebb);
-                    builder.switch_to_block(else_ebb);
+                        // You might be expecting that we push the parameters for this
+                        // `else` block here, something like this:
+                        //
+                        //     state.pushn(&control_stack_frame.params);
+                        //
+                        // We don't do that because they are already on the top of the stack
+                        // for us: we pushed the parameters twice when we saw the initial
+                        // `if` so that we wouldn't have to save the parameters in the
+                        // `ControlStackFrame` as another `Vec` allocation.
 
-                    // NB: we don't bother updating the control frame's
-                    // `ElseData` because nothing else will read it.
-                }
-                ControlStackFrame::If {
-                    destination,
-                    num_return_values,
-                    else_data: ElseData::WithElse { else_block, .. },
-                    reachable_from_top,
-                    ..
-                } => {
-                    debug_assert!(!reachable_from_top);
-                    builder
-                        .ins()
-                        .jump(destination, state.peekn(num_return_values));
-                    state.popn(num_return_values);
-                    builder.switch_to_block(else_block);
+                        builder.switch_to_block(else_ebb);
+
+                        // We don't bother updating the control frame's `ElseData`
+                        // to `WithElse` because nothing else will read it.
+                    }
                 }
                 _ => unreachable!(),
             }
         }
         Operator::End => {
             let frame = state.control_stack.pop().unwrap();
+
             if !builder.is_unreachable() || !builder.is_pristine() {
                 let return_count = frame.num_return_values();
                 builder
@@ -1212,6 +1219,7 @@ fn translate_unreachable_operator(
     builder: &mut FunctionBuilder,
     state: &mut FuncTranslationState,
 ) -> WasmResult<()> {
+    debug_assert!(!state.reachable);
     match *op {
         Operator::If { ty } => {
             // Push a placeholder control stack entry. The if isn't reachable,
@@ -1233,25 +1241,33 @@ fn translate_unreachable_operator(
             let i = state.control_stack.len() - 1;
             match state.control_stack[i] {
                 ControlStackFrame::If {
-                    else_data: ElseData::NoElse { branch_inst },
-                    ref mut reachable_from_top,
+                    ref else_data,
+                    head_is_reachable,
+                    ref mut consequent_ends_reachable,
                     blocktype,
                     ..
                 } => {
-                    if *reachable_from_top {
-                        // We have a branch from the top of the if to the else.
+                    debug_assert!(consequent_ends_reachable.is_none());
+                    *consequent_ends_reachable = Some(state.reachable);
+
+                    if head_is_reachable {
+                        // We have a branch from the head of the `if` to the `else`.
                         state.reachable = true;
-                        // And because there's an else, there can no longer be a
-                        // branch from the top directly to the end.
-                        *reachable_from_top = false;
 
-                        let (params, _results) =
-                            blocktype_params_results(module_translation_state, blocktype)?;
-                        let else_ebb = ebb_with_params(builder, params)?;
+                        let else_ebb = match *else_data {
+                            ElseData::NoElse { branch_inst } => {
+                                let (params, _results) =
+                                    blocktype_params_results(module_translation_state, blocktype)?;
+                                let else_ebb = ebb_with_params(builder, params)?;
+
+                                // We change the target of the branch instruction.
+                                builder.change_jump_destination(branch_inst, else_ebb);
+                                builder.seal_block(else_ebb);
+                                else_ebb
+                            }
+                            ElseData::WithElse { else_block } => else_block,
+                        };
 
-                        // We change the target of the branch instruction.
-                        builder.change_jump_destination(branch_inst, else_ebb);
-                        builder.seal_block(else_ebb);
                         builder.switch_to_block(else_ebb);
 
                         // Again, no need to push the parameters for the `else`,
@@ -1260,18 +1276,6 @@ fn translate_unreachable_operator(
                         // `translate_operator` for details.
                     }
                 }
-                ControlStackFrame::If {
-                    else_data: ElseData::WithElse { else_block, .. },
-                    reachable_from_top,
-                    ..
-                } => {
-                    debug_assert!(
-                        !reachable_from_top,
-                        "should not be reachable from top if we have an else block"
-                    );
-                    builder.switch_to_block(else_block);
-                    state.reachable = true;
-                }
                 _ => unreachable!(),
             }
         }
@@ -1291,23 +1295,24 @@ fn translate_unreachable_operator(
                     // And loops can't have branches to the end.
                     false
                 }
+                // If we never set `consequent_ends_reachable` then that means
+                // we are finishing the consequent now, and there was no
+                // `else`. Whether the following block is reachable depends only
+                // on if the head was reachable.
                 ControlStackFrame::If {
-                    else_data: ElseData::WithElse { .. },
-                    reachable_from_top,
+                    head_is_reachable,
+                    consequent_ends_reachable: None,
                     ..
-                } => {
-                    debug_assert!(!reachable_from_top);
-                    true
-                }
+                } => head_is_reachable,
+                // Since we are only in this function when in unreachable code,
+                // we know that the alternative just ended unreachable. Whether
+                // the following block is reachable depends on if the consequent
+                // ended reachable or not.
                 ControlStackFrame::If {
-                    else_data: ElseData::NoElse { .. },
-                    reachable_from_top,
+                    head_is_reachable,
+                    consequent_ends_reachable: Some(consequent_ends_reachable),
                     ..
-                } => {
-                    // A reachable if without an else has a branch from the top
-                    // directly to the bottom.
-                    reachable_from_top
-                }
+                } => head_is_reachable && consequent_ends_reachable,
                 // All other control constructs are already handled.
                 _ => false,
             };
diff --git a/cranelift/wasm/src/state/func_state.rs b/cranelift/wasm/src/state/func_state.rs
index 4b3627e008..88d83ab846 100644
--- a/cranelift/wasm/src/state/func_state.rs
+++ b/cranelift/wasm/src/state/func_state.rs
@@ -17,13 +17,22 @@ use std::vec::Vec;
 #[derive(Debug)]
 pub enum ElseData {
     /// The `if` does not already have an `else` block.
+    ///
+    /// This doesn't mean that it will never have an `else`, just that we
+    /// haven't seen it yet.
     NoElse {
         /// If we discover that we need an `else` block, this is the jump
         /// instruction that needs to be fixed up to point to the new `else`
         /// block rather than the destination block after the `if...end`.
         branch_inst: Inst,
     },
+
     /// We have already allocated an `else` block.
+    ///
+    /// Usually we don't know whether we will hit an `if .. end` or an `if
+    /// .. else .. end`, but sometimes we can tell based on the block's type
+    /// signature that the signature is not valid if there isn't an `else`. In
+    /// these cases, we pre-allocate the `else` block.
     WithElse {
         /// This is the `else` block.
         else_block: Ebb,
@@ -49,8 +58,18 @@ pub enum ControlStackFrame {
         num_return_values: usize,
         original_stack_size: usize,
         exit_is_branched_to: bool,
-        reachable_from_top: bool,
         blocktype: wasmparser::TypeOrFuncType,
+        /// Was the head of the `if` reachable?
+        head_is_reachable: bool,
+        /// What was the reachability at the end of the consequent?
+        ///
+        /// This is `None` until we're finished translating the consequent, and
+        /// is set to `Some` either by hitting an `else` when we will begin
+        /// translating the alternative, or by hitting an `end` in which case
+        /// there is no alternative.
+        consequent_ends_reachable: Option,
+        // Note: no need for `alternative_ends_reachable` because that is just
+        // `state.reachable` when we hit the `end` in the `if .. else .. end`.
     },
     Block {
         destination: Ebb,
@@ -370,11 +389,6 @@ impl FuncTranslationState {
             self.stack.push(val);
         }
 
-        let has_else = match else_data {
-            ElseData::NoElse { .. } => false,
-            ElseData::WithElse { .. } => true,
-        };
-
         self.control_stack.push(ControlStackFrame::If {
             destination,
             else_data,
@@ -382,7 +396,8 @@ impl FuncTranslationState {
             num_param_values: num_param_types,
             num_return_values: num_result_types,
             exit_is_branched_to: false,
-            reachable_from_top: self.reachable && !has_else,
+            head_is_reachable: self.reachable,
+            consequent_ends_reachable: None,
             blocktype,
         });
     }
diff --git a/cranelift/wasmtests/if-reachability-translation-0.wat b/cranelift/wasmtests/if-reachability-translation-0.wat
new file mode 100644
index 0000000000..54145a9d4e
--- /dev/null
+++ b/cranelift/wasmtests/if-reachability-translation-0.wat
@@ -0,0 +1,12 @@
+;; An unreachable `if` means that the consequent, alternative, and following
+;; block are also unreachable.
+
+(module
+  (func (param i32) (result i32)
+    unreachable
+    if  ;; label = @2
+      nop
+    else
+      nop
+    end
+    i32.const 0))
diff --git a/cranelift/wasmtests/if-reachability-translation-1.wat b/cranelift/wasmtests/if-reachability-translation-1.wat
new file mode 100644
index 0000000000..6e1e6121f4
--- /dev/null
+++ b/cranelift/wasmtests/if-reachability-translation-1.wat
@@ -0,0 +1,12 @@
+;; Reachable `if` head and reachable consequent and alternative means that the
+;; following block is also reachable.
+
+(module
+  (func (param i32) (result i32)
+    local.get 0
+    if  ;; label = @2
+      nop
+    else
+      nop
+    end
+    i32.const 0))
diff --git a/cranelift/wasmtests/if-reachability-translation-2.wat b/cranelift/wasmtests/if-reachability-translation-2.wat
new file mode 100644
index 0000000000..4bbaf99820
--- /dev/null
+++ b/cranelift/wasmtests/if-reachability-translation-2.wat
@@ -0,0 +1,12 @@
+;; Reachable `if` head and unreachable consequent and reachable alternative
+;; means that the following block is also reachable.
+
+(module
+  (func (param i32) (result i32)
+    local.get 0
+    if
+      unreachable
+    else
+      nop
+    end
+    i32.const 0))
diff --git a/cranelift/wasmtests/if-reachability-translation-3.wat b/cranelift/wasmtests/if-reachability-translation-3.wat
new file mode 100644
index 0000000000..72251cba16
--- /dev/null
+++ b/cranelift/wasmtests/if-reachability-translation-3.wat
@@ -0,0 +1,12 @@
+;; Reachable `if` head and consequent and unreachable alternative means that the
+;; following block is also reachable.
+
+(module
+  (func (param i32) (result i32)
+    local.get 0
+    if
+      nop
+    else
+      unreachable
+    end
+    i32.const 0))
diff --git a/cranelift/wasmtests/if-reachability-translation-4.wat b/cranelift/wasmtests/if-reachability-translation-4.wat
new file mode 100644
index 0000000000..b8a4069430
--- /dev/null
+++ b/cranelift/wasmtests/if-reachability-translation-4.wat
@@ -0,0 +1,12 @@
+;; Reachable `if` head and unreachable consequent and alternative means that the
+;; following block is unreachable.
+
+(module
+  (func (param i32) (result i32)
+    local.get 0
+    if
+      unreachable
+    else
+      unreachable
+    end
+    i32.const 0))
diff --git a/cranelift/wasmtests/if-reachability-translation-5.wat b/cranelift/wasmtests/if-reachability-translation-5.wat
new file mode 100644
index 0000000000..7b1f665e05
--- /dev/null
+++ b/cranelift/wasmtests/if-reachability-translation-5.wat
@@ -0,0 +1,14 @@
+;; Reachable `if` head and unreachable consequent and alternative, but with a
+;; branch out of the consequent, means that the following block is reachable.
+
+(module
+  (func (param i32 i32) (result i32)
+    local.get 0
+    if
+      local.get 1
+      br_if 0
+      unreachable
+    else
+      unreachable
+    end
+    i32.const 0))
diff --git a/cranelift/wasmtests/if-reachability-translation-6.wat b/cranelift/wasmtests/if-reachability-translation-6.wat
new file mode 100644
index 0000000000..d9da824f14
--- /dev/null
+++ b/cranelift/wasmtests/if-reachability-translation-6.wat
@@ -0,0 +1,14 @@
+;; Reachable `if` head and unreachable consequent and alternative, but with a
+;; branch out of the alternative, means that the following block is reachable.
+
+(module
+  (func (param i32 i32) (result i32)
+    local.get 0
+    if
+      unreachable
+    else
+      local.get 1
+      br_if 0
+      unreachable
+    end
+    i32.const 0))

From 50b7d2827d9e0a2e70bb716187cb0d26359f86e3 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 15 Oct 2019 11:11:48 -0700
Subject: [PATCH 2842/3084] Bump version to 0.46.1

---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  2 +-
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 19 files changed, 68 insertions(+), 68 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index b0f5e91e7d..d2404d2ad2 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.46.0"
+version = "0.46.1"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.46.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.46.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.46.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.46.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.46.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.46.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.46.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.46.0" }
-cranelift-module = { path = "cranelift-module", version = "0.46.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.46.0" }
-cranelift-object = { path = "cranelift-object", version = "0.46.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.46.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.46.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.46.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.46.1" }
+cranelift-entity = { path = "cranelift-entity", version = "0.46.1" }
+cranelift-reader = { path = "cranelift-reader", version = "0.46.1" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.46.1" }
+cranelift-serde = { path = "cranelift-serde", version = "0.46.1", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.46.1", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.46.1" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.46.1" }
+cranelift-module = { path = "cranelift-module", version = "0.46.1" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.46.1" }
+cranelift-object = { path = "cranelift-object", version = "0.46.1" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.46.1" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.46.1" }
+cranelift = { path = "cranelift-umbrella", version = "0.46.1" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index bea45dcc6f..9abe721b71 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.46.0"
+version = "0.46.1"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.46.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.46.1", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 563033ef95..ccda65d612 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.46.0"
+version = "0.46.1"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.46.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.46.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.46.0" }
+cranelift-codegen-shared = { path = "./shared", version = "0.46.1" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.46.1" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.46.1" }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }
 hashbrown = { version = "0.6", optional = true }
@@ -29,7 +29,7 @@ smallvec = { version = "0.6.10" }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.46.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.46.1" }
 
 [features]
 default = ["std"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 49757a07aa..c11a7333db 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.46.0"
+version = "0.46.1"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.46.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.46.0" }
+cranelift-codegen-shared = { path = "../shared", version = "0.46.1" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.46.1" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 7dd740670d..098ed8b6cc 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.46.0"
+version = "0.46.1"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index b138268444..152520be38 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.46.0"
+version = "0.46.1"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 363a81fcd6..c5b471da1e 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.46.0"
+version = "0.46.1"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.46.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1" }
+cranelift-module = { path = "../cranelift-module", version = "0.46.1" }
 faerie = "0.11.0"
 goblin = "0.0.24"
 failure = "0.1.2"
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 144c1d0ab5..5a359c8e22 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.46.0"
+version = "0.46.1"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.46.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.46.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.46.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.46.1" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.46.1" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.46.1" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index a9ef6c3481..950e3d2623 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.46.0"
+version = "0.46.1"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default-features = false }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 5d62d08d20..5923cd792f 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.46.0"
+version = "0.46.1"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.46.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.46.1" }
 hashbrown = { version = "0.6", optional = true }
 failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index ae57953f2d..5af8f3a7a3 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.46.0"
+version = "0.46.1"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default-features = false }
 target-lexicon = "0.8.1"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 1e69a5e061..8ad641984f 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.46.0"
+version = "0.46.1"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.46.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1" }
+cranelift-module = { path = "../cranelift-module", version = "0.46.1" }
 object = { version = "0.14.0", default-features = false, features = ["write"] }
 target-lexicon = "0.8.1"
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 6ab4e86f94..fae98ff23f 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.46.0"
+version = "0.46.1"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.46.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.46.1" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 456aa7845e..b5f832b3a0 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.46.0"
+version="0.46.1"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index a8ef0c4af7..14feaee8cb 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.46.0"
+version = "0.46.1"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1" }
 target-lexicon = "0.8.1"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 6376186373..348b24d65c 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.46.0"
+version = "0.46.1"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.46.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.46.1" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 214a730255..637a08cb90 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.46.0"
+version = "0.46.1"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,9 +10,9 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0" }
-cranelift-module = { path = "../cranelift-module", version = "0.46.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.46.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1" }
+cranelift-module = { path = "../cranelift-module", version = "0.46.1" }
+cranelift-native = { path = "../cranelift-native", version = "0.46.1" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -27,9 +27,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.46.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.46.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.46.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.46.1" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.46.1" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.46.1" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 8b0ca3297d..912ec3cf0d 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.46.0"
+version = "0.46.1"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.46.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.46.1", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index ebec419cae..df97b7260a 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.46.0"
+version = "0.46.1"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.39.2", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.46.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.46.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.46.1" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.46.1", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 failure = { version = "0.1.1", default-features = false, features = ["derive"] }
 failure_derive = { version = "0.1.1", default-features = false }

From 1600dba634b35c17b74616f9ddd3220f05e2c1be Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 30 Sep 2019 14:56:03 -0700
Subject: [PATCH 2843/3084] Make ConstantData a container for any-size constant
 values

Previously, ConstantData was a type alias for `Vec` which prevented it from having an implementation; this meant that `V128Imm` and `&[u8; 16]` were used in places that otherwise could have accepted types of different byte lengths.
---
 .../codegen/meta/src/cdsl/instructions.rs     | 20 ++---
 .../codegen/meta/src/isa/x86/encodings.rs     |  4 +-
 cranelift/codegen/src/ir/constant.rs          | 85 +++++++++++++++++--
 cranelift/codegen/src/isa/x86/enc_tables.rs   |  8 +-
 cranelift/codegen/src/predicates.rs           | 27 +++---
 cranelift/codegen/src/write.rs                |  5 +-
 6 files changed, 113 insertions(+), 36 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 0d27159cd9..464aa6e248 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -734,10 +734,10 @@ pub enum FormatPredicateKind {
     IsZero64BitFloat,
 
     /// Is the immediate format field member equal zero in all lanes?
-    IsAllZeroes128Bit,
+    IsAllZeroes,
 
     /// Does the immediate format field member have ones in all bits of all lanes?
-    IsAllOnes128Bit,
+    IsAllOnes,
 
     /// Has the value list (in member_name) the size specified in parameter?
     LengthEquals(usize),
@@ -818,12 +818,12 @@ impl FormatPredicateNode {
             FormatPredicateKind::IsZero64BitFloat => {
                 format!("predicates::is_zero_64_bit_float({})", self.member_name)
             }
-            FormatPredicateKind::IsAllZeroes128Bit => format!(
-                "predicates::is_all_zeroes_128_bit(func.dfg.constants.get({}))",
+            FormatPredicateKind::IsAllZeroes => format!(
+                "predicates::is_all_zeroes(func.dfg.constants.get({}))",
                 self.member_name
             ),
-            FormatPredicateKind::IsAllOnes128Bit => format!(
-                "predicates::is_all_ones_128_bit(func.dfg.constants.get({}))",
+            FormatPredicateKind::IsAllOnes => format!(
+                "predicates::is_all_ones(func.dfg.constants.get({}))",
                 self.member_name
             ),
             FormatPredicateKind::LengthEquals(num) => format!(
@@ -1069,25 +1069,25 @@ impl InstructionPredicate {
         ))
     }
 
-    pub fn new_is_all_zeroes_128bit(
+    pub fn new_is_all_zeroes(
         format: &InstructionFormat,
         field_name: &'static str,
     ) -> InstructionPredicateNode {
         InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
             format,
             field_name,
-            FormatPredicateKind::IsAllZeroes128Bit,
+            FormatPredicateKind::IsAllZeroes,
         ))
     }
 
-    pub fn new_is_all_ones_128bit(
+    pub fn new_is_all_ones(
         format: &InstructionFormat,
         field_name: &'static str,
     ) -> InstructionPredicateNode {
         InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
             format,
             field_name,
-            FormatPredicateKind::IsAllOnes128Bit,
+            FormatPredicateKind::IsAllOnes,
         ))
     }
 
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index dcdc4d0196..a029e8170f 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1848,14 +1848,14 @@ pub(crate) fn define<'defs>(
         let instruction = vconst.bind(vector(ty, sse_vector_size));
 
         let is_zero_128bit =
-            InstructionPredicate::new_is_all_zeroes_128bit(f_unary_const, "constant_handle");
+            InstructionPredicate::new_is_all_zeroes(f_unary_const, "constant_handle");
         let template = rec_vconst_optimized.nonrex().opcodes(&PXOR);
         e.enc_32_64_func(instruction.clone(), template, |builder| {
             builder.inst_predicate(is_zero_128bit)
         });
 
         let is_ones_128bit =
-            InstructionPredicate::new_is_all_ones_128bit(f_unary_const, "constant_handle");
+            InstructionPredicate::new_is_all_ones(f_unary_const, "constant_handle");
         let template = rec_vconst_optimized.nonrex().opcodes(&PCMPEQB);
         e.enc_32_64_func(instruction, template, |builder| {
             builder.inst_predicate(is_ones_128bit)
diff --git a/cranelift/codegen/src/ir/constant.rs b/cranelift/codegen/src/ir/constant.rs
index ff30d67734..bd69e60c6f 100644
--- a/cranelift/codegen/src/ir/constant.rs
+++ b/cranelift/codegen/src/ir/constant.rs
@@ -12,10 +12,65 @@ use crate::ir::Constant;
 use crate::HashMap;
 use alloc::collections::BTreeMap;
 use alloc::vec::Vec;
+use core::fmt;
+use core::slice::Iter;
 use cranelift_entity::EntityRef;
 
-/// This type describes the actual constant data.
-pub type ConstantData = Vec;
+/// This type describes the actual constant data. Note that the bytes stored in this structure are
+/// expected to be in little-endian order; this is due to ease-of-use when interacting with
+/// WebAssembly values, which are [little-endian by design]
+/// (https://github.com/WebAssembly/design/blob/master/Portability.md).
+#[derive(Clone, Hash, Eq, PartialEq, Debug)]
+pub struct ConstantData(Vec);
+
+impl From> for ConstantData {
+    fn from(v: Vec) -> Self {
+        Self(v)
+    }
+}
+
+impl From<&[u8]> for ConstantData {
+    fn from(v: &[u8]) -> Self {
+        Self(v.to_vec())
+    }
+}
+
+impl ConstantData {
+    /// Return the number of bytes in the constant.
+    pub fn len(&self) -> usize {
+        self.0.len()
+    }
+
+    /// Iterate over the constant's bytes.
+    pub fn iter(&self) -> Iter {
+        self.0.iter()
+    }
+}
+
+impl fmt::Display for ConstantData {
+    /// Print the constant data in hexadecimal format, e.g. 0x000102030405060708090a0b0c0d0e0f.
+    /// This function will flip the stored order of bytes--little-endian--to the more readable
+    /// big-endian ordering. Any zero bytes in high-order bytes will be discarded in the formatted
+    /// string.
+    ///
+    /// ```
+    /// use cranelift_codegen::ir::ConstantData;
+    /// let data = ConstantData::from([3, 2, 1, 0, 0].as_ref()); // note the little-endian order
+    /// assert_eq!(data.to_string(), "0x010203");
+    /// ```
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "0x")?;
+        let mut bytes_written = 0;
+        for b in self.0.iter().rev().skip_while(|&&b| b == 0) {
+            write!(f, "{:02x}", b)?;
+            bytes_written += 1;
+        }
+        if bytes_written < 1 {
+            write!(f, "00")?;
+        }
+        Ok(())
+    }
+}
 
 /// This type describes an offset in bytes within a constant pool.
 pub type ConstantOffset = u32;
@@ -79,7 +134,8 @@ impl ConstantPool {
     /// Insert constant data into the pool, returning a handle for later referencing; when constant
     /// data is inserted that is a duplicate of previous constant data, the existing handle will be
     /// returned.
-    pub fn insert(&mut self, constant_value: ConstantData) -> Constant {
+    pub fn insert(&mut self, constant_value: impl Into) -> Constant {
+        let constant_value = constant_value.into();
         if self.values_to_handles.contains_key(&constant_value) {
             self.values_to_handles.get(&constant_value).unwrap().clone()
         } else {
@@ -153,6 +209,7 @@ impl ConstantPool {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use std::string::ToString;
 
     #[test]
     fn empty() {
@@ -194,7 +251,7 @@ mod tests {
         sut.insert(vec![4, 5, 6]);
         sut.insert(vec![1, 2, 3]);
         let data = sut.iter().map(|(_, v)| v).collect::>();
-        assert_eq!(data, vec![&vec![1, 2, 3], &vec![4, 5, 6]]);
+        assert_eq!(data, vec![&vec![1, 2, 3].into(), &vec![4, 5, 6].into()]);
     }
 
     #[test]
@@ -202,7 +259,7 @@ mod tests {
         let mut sut = ConstantPool::new();
         let data = vec![1, 2, 3];
         let handle = sut.insert(data.clone());
-        assert_eq!(sut.get(handle), &data);
+        assert_eq!(sut.get(handle), &data.into());
     }
 
     #[test]
@@ -228,4 +285,22 @@ mod tests {
         let a = sut.insert(vec![1]);
         sut.get_offset(a); // panics, set_offset should have been called
     }
+
+    #[test]
+    fn display_constant_data() {
+        assert_eq!(ConstantData::from([0].as_ref()).to_string(), "0x00");
+        assert_eq!(ConstantData::from([42].as_ref()).to_string(), "0x2a");
+        assert_eq!(
+            ConstantData::from([3, 2, 1, 0].as_ref()).to_string(),
+            "0x010203"
+        );
+        assert_eq!(
+            ConstantData::from(3735928559u32.to_le_bytes().as_ref()).to_string(),
+            "0xdeadbeef"
+        );
+        assert_eq!(
+            ConstantData::from(0x0102030405060708u64.to_le_bytes().as_ref()).to_string(),
+            "0x0102030405060708"
+        );
+    }
 }
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 93adca0c25..64c0499d40 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -16,6 +16,7 @@ use crate::isa::RegUnit;
 use crate::isa::{self, TargetIsa};
 use crate::predicates;
 use crate::regalloc::RegDiversions;
+use std::vec::Vec;
 
 include!(concat!(env!("OUT_DIR"), "/encoding-x86.rs"));
 include!(concat!(env!("OUT_DIR"), "/legalize-x86.rs"));
@@ -928,7 +929,7 @@ fn convert_shuffle(
             .clone();
         if a == b {
             // PSHUFB the first argument (since it is the same as the second).
-            let constructed_mask = mask
+            let constructed_mask: Vec = mask
                 .iter()
                 // If the mask is greater than 15 it still may be referring to a lane in b.
                 .map(|&b| if b > 15 { b.wrapping_sub(16) } else { b })
@@ -942,7 +943,8 @@ fn convert_shuffle(
             pos.func.dfg.replace(inst).x86_pshufb(a, mask_value);
         } else {
             // PSHUFB the first argument, placing zeroes for unused lanes.
-            let constructed_mask = mask.iter().cloned().map(zero_unknown_lane_index).collect();
+            let constructed_mask: Vec =
+                mask.iter().cloned().map(zero_unknown_lane_index).collect();
             let handle = pos.func.dfg.constants.insert(constructed_mask);
             // Move the built mask into another XMM register.
             let a_type = pos.func.dfg.value_type(a);
@@ -951,7 +953,7 @@ fn convert_shuffle(
             let shuffled_first_arg = pos.ins().x86_pshufb(a, mask_value);
 
             // PSHUFB the second argument, placing zeroes for unused lanes.
-            let constructed_mask = mask
+            let constructed_mask: Vec = mask
                 .iter()
                 .map(|b| b.wrapping_sub(16))
                 .map(zero_unknown_lane_index)
diff --git a/cranelift/codegen/src/predicates.rs b/cranelift/codegen/src/predicates.rs
index 16672b145b..374e2bbbff 100644
--- a/cranelift/codegen/src/predicates.rs
+++ b/cranelift/codegen/src/predicates.rs
@@ -10,6 +10,7 @@
 //! dead code warning.
 
 use crate::ir;
+use crate::ir::ConstantData;
 
 /// Check that an integer value is zero.
 #[allow(dead_code)]
@@ -33,14 +34,14 @@ pub fn is_zero_32_bit_float>(x: T) -> bool {
 
 /// Check that a 128-bit vector contains all zeroes.
 #[allow(dead_code)]
-pub fn is_all_zeroes_128_bit<'b, T: PartialEq<&'b [u8; 16]>>(x: T) -> bool {
-    x.eq(&&[0; 16])
+pub fn is_all_zeroes(x: &ConstantData) -> bool {
+    x.iter().all(|&f| f == 0)
 }
 
 /// Check that a 128-bit vector contains all ones.
 #[allow(dead_code)]
-pub fn is_all_ones_128_bit<'b, T: PartialEq<&'b [u8; 16]>>(x: T) -> bool {
-    x.eq(&&[0xff; 16])
+pub fn is_all_ones(x: &ConstantData) -> bool {
+    x.iter().all(|&f| f == 0xff)
 }
 
 /// Check that `x` is the same as `y`.
@@ -123,17 +124,17 @@ mod tests {
     }
 
     #[test]
-    fn is_all_zeroes() {
-        assert!(is_all_zeroes_128_bit(&[0; 16]));
-        assert!(is_all_zeroes_128_bit(vec![
-            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
-        ]));
-        assert!(!is_all_zeroes_128_bit(&[1; 16]));
+    fn check_is_all_zeroes() {
+        assert!(is_all_zeroes(&[0; 16].as_ref().into()));
+        assert!(is_all_zeroes(
+            &vec![0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0].into()
+        ));
+        assert!(!is_all_zeroes(&[1; 16].as_ref().into()));
     }
 
     #[test]
-    fn is_all_ones() {
-        assert!(!is_all_ones_128_bit(&[0; 16]));
-        assert!(is_all_ones_128_bit(&[0xff; 16]));
+    fn check_is_all_ones() {
+        assert!(!is_all_ones(&[0; 16].as_ref().into()));
+        assert!(is_all_ones(&[0xff; 16].as_ref().into()));
     }
 }
diff --git a/cranelift/codegen/src/write.rs b/cranelift/codegen/src/write.rs
index ae093f3919..910d1b988e 100644
--- a/cranelift/codegen/src/write.rs
+++ b/cranelift/codegen/src/write.rs
@@ -508,9 +508,8 @@ pub fn write_operands(
         UnaryConst {
             constant_handle, ..
         } => {
-            let data = dfg.constants.get(constant_handle);
-            let v128 = V128Imm::from(&data[..]);
-            write!(w, " {}", v128)
+            let constant_data = dfg.constants.get(constant_handle);
+            write!(w, " {}", constant_data)
         }
         Shuffle { mask, args, .. } => {
             let data = dfg.immediates.get(mask).expect(

From a03f905d0842bbceb15c601b91ef8f3c97f610fa Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 1 Oct 2019 20:13:17 -0700
Subject: [PATCH 2844/3084] Replace V128Imm functionality with ConstantData

This moves most original uses of V128Imm (e.g. in parsing) to ConstantData and shifts the unit tests from V128Imm to ConstantData.
---
 cranelift/codegen/src/ir/constant.rs        | 180 ++++++++++++++-
 cranelift/codegen/src/ir/dfg.rs             |   5 +-
 cranelift/codegen/src/ir/immediates.rs      | 237 +-------------------
 cranelift/codegen/src/isa/x86/enc_tables.rs |   9 +-
 cranelift/codegen/src/write.rs              |   4 +-
 cranelift/reader/src/parser.rs              |  93 +++++---
 cranelift/wasm/src/code_translator.rs       |   7 +-
 7 files changed, 263 insertions(+), 272 deletions(-)

diff --git a/cranelift/codegen/src/ir/constant.rs b/cranelift/codegen/src/ir/constant.rs
index bd69e60c6f..f29c8fbada 100644
--- a/cranelift/codegen/src/ir/constant.rs
+++ b/cranelift/codegen/src/ir/constant.rs
@@ -8,19 +8,21 @@
 //! - ensuring alignment of constants within the pool,
 //! - bucketing constants by size.
 
+use crate::ir::immediates::{IntoBytes, V128Imm};
 use crate::ir::Constant;
 use crate::HashMap;
 use alloc::collections::BTreeMap;
 use alloc::vec::Vec;
 use core::fmt;
 use core::slice::Iter;
+use core::str::{from_utf8, FromStr};
 use cranelift_entity::EntityRef;
 
 /// This type describes the actual constant data. Note that the bytes stored in this structure are
 /// expected to be in little-endian order; this is due to ease-of-use when interacting with
 /// WebAssembly values, which are [little-endian by design]
 /// (https://github.com/WebAssembly/design/blob/master/Portability.md).
-#[derive(Clone, Hash, Eq, PartialEq, Debug)]
+#[derive(Clone, Hash, Eq, PartialEq, Debug, Default)]
 pub struct ConstantData(Vec);
 
 impl From> for ConstantData {
@@ -35,16 +37,47 @@ impl From<&[u8]> for ConstantData {
     }
 }
 
+impl From for ConstantData {
+    fn from(v: V128Imm) -> Self {
+        Self(v.to_vec())
+    }
+}
+
 impl ConstantData {
     /// Return the number of bytes in the constant.
     pub fn len(&self) -> usize {
         self.0.len()
     }
 
+    /// Convert the data to a vector.
+    pub fn to_vec(self) -> Vec {
+        self.0
+    }
+
     /// Iterate over the constant's bytes.
     pub fn iter(&self) -> Iter {
         self.0.iter()
     }
+
+    /// Add new bytes to the constant data.
+    pub fn append(mut self, bytes: impl IntoBytes) -> Self {
+        let mut to_add = bytes.into_bytes();
+        self.0.append(&mut to_add);
+        self
+    }
+
+    /// Expand the size of the constant data to `expected_size` number of bytes by adding zeroes
+    /// in the high-order byte slots.
+    pub fn expand_to(mut self, expected_size: usize) -> Self {
+        if self.len() > expected_size {
+            panic!(
+                "The constant data is already expanded beyond {} bytes",
+                expected_size
+            )
+        }
+        self.0.resize(expected_size, 0);
+        self
+    }
 }
 
 impl fmt::Display for ConstantData {
@@ -72,6 +105,50 @@ impl fmt::Display for ConstantData {
     }
 }
 
+impl FromStr for ConstantData {
+    type Err = &'static str;
+
+    /// Parse a hexadecimal string to `ConstantData`. This is the inverse of [ConstantData::fmt]
+    /// (cranelift_codegen::ir::ConstantData::fmt).
+    ///
+    /// ```
+    /// use cranelift_codegen::ir::ConstantData;
+    /// let c: ConstantData = "0x000102".parse().unwrap();
+    /// assert_eq!(c.to_vec(), [2, 1, 0]);
+    /// ```
+    fn from_str(s: &str) -> Result {
+        if s.len() <= 2 || &s[0..2] != "0x" {
+            return Err("Expected a hexadecimal string, e.g. 0x1234");
+        }
+
+        // clean and check the string
+        let cleaned: Vec = s[2..]
+            .as_bytes()
+            .iter()
+            .filter(|&&b| b as char != '_')
+            .cloned()
+            .collect(); // remove 0x prefix and any intervening _ characters
+
+        if cleaned.len() == 0 {
+            Err("Hexadecimal string must have some digits")
+        } else if cleaned.len() % 2 != 0 {
+            Err("Hexadecimal string must have an even number of digits")
+        } else if cleaned.len() > 32 {
+            Err("Hexadecimal string has too many digits to fit in a 128-bit vector")
+        } else {
+            let mut buffer = Vec::with_capacity((s.len() - 2) / 2);
+            for i in (0..cleaned.len()).step_by(2) {
+                let pair = from_utf8(&cleaned[i..i + 2])
+                    .or_else(|_| Err("Unable to parse hexadecimal pair as UTF-8"))?;
+                let byte = u8::from_str_radix(pair, 16)
+                    .or_else(|_| Err("Unable to parse as hexadecimal"))?;
+                buffer.insert(0, byte);
+            }
+            Ok(ConstantData(buffer))
+        }
+    }
+}
+
 /// This type describes an offset in bytes within a constant pool.
 pub type ConstantOffset = u32;
 
@@ -303,4 +380,105 @@ mod tests {
             "0x0102030405060708"
         );
     }
+
+    #[test]
+    fn iterate_over_constant_data() {
+        let c = ConstantData::from([1, 2, 3].as_ref());
+        let mut iter = c.iter();
+        assert_eq!(iter.next(), Some(&1));
+        assert_eq!(iter.next(), Some(&2));
+        assert_eq!(iter.next(), Some(&3));
+        assert_eq!(iter.next(), None);
+    }
+
+    #[test]
+    fn add_to_constant_data() {
+        let d = ConstantData::from([1, 2].as_ref());
+        let e = d.append(i16::from(3u8));
+        assert_eq!(e.to_vec(), vec![1, 2, 3, 0])
+    }
+
+    #[test]
+    fn extend_constant_data() {
+        let d = ConstantData::from([1, 2].as_ref());
+        assert_eq!(d.expand_to(4).to_vec(), vec![1, 2, 0, 0])
+    }
+
+    #[test]
+    #[should_panic]
+    fn extend_constant_data_to_invalid_length() {
+        ConstantData::from([1, 2].as_ref()).expand_to(1);
+    }
+
+    #[test]
+    fn parse_constant_data_and_restringify() {
+        // Verify that parsing of `from` succeeds and stringifies to `to`.
+        fn parse_ok(from: &str, to: &str) {
+            let parsed = from.parse::().unwrap();
+            assert_eq!(parsed.to_string(), to);
+        }
+
+        // Verify that parsing of `from` fails with `error_msg`.
+        fn parse_err(from: &str, error_msg: &str) {
+            let parsed = from.parse::();
+            assert!(
+                parsed.is_err(),
+                "Expected a parse error but parsing succeeded: {}",
+                from
+            );
+            assert_eq!(parsed.err().unwrap(), error_msg);
+        }
+
+        parse_ok("0x00", "0x00");
+        parse_ok("0x00000042", "0x42");
+        parse_ok(
+            "0x0102030405060708090a0b0c0d0e0f00",
+            "0x0102030405060708090a0b0c0d0e0f00",
+        );
+        parse_ok("0x_0000_0043_21", "0x4321");
+
+        parse_err("", "Expected a hexadecimal string, e.g. 0x1234");
+        parse_err("0x", "Expected a hexadecimal string, e.g. 0x1234");
+        parse_err(
+            "0x042",
+            "Hexadecimal string must have an even number of digits",
+        );
+        parse_err(
+            "0x00000000000000000000000000000000000000000000000000",
+            "Hexadecimal string has too many digits to fit in a 128-bit vector",
+        );
+        parse_err("0xrstu", "Unable to parse as hexadecimal");
+        parse_err("0x__", "Hexadecimal string must have some digits");
+    }
+
+    #[test]
+    fn verify_stored_bytes_in_constant_data() {
+        assert_eq!("0x01".parse::().unwrap().to_vec(), [1]);
+        assert_eq!(ConstantData::from([1, 0].as_ref()).0, [1, 0]);
+        assert_eq!(ConstantData::from(vec![1, 0, 0, 0]).0, [1, 0, 0, 0]);
+    }
+
+    #[test]
+    fn check_constant_data_endianness_as_uimm128() {
+        fn parse_to_uimm128(from: &str) -> Vec {
+            from.parse::().unwrap().expand_to(16).to_vec()
+        }
+
+        assert_eq!(
+            parse_to_uimm128("0x42"),
+            [0x42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+        );
+        assert_eq!(
+            parse_to_uimm128("0x00"),
+            [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+        );
+        assert_eq!(
+            parse_to_uimm128("0x12345678"),
+            [0x78, 0x56, 0x34, 0x12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+        );
+        assert_eq!(
+            parse_to_uimm128("0x1234_5678"),
+            [0x78, 0x56, 0x34, 0x12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+        );
+    }
 }
diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index 0ff6a263c4..b41f035822 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -5,7 +5,7 @@ use crate::ir;
 use crate::ir::builder::ReplaceBuilder;
 use crate::ir::extfunc::ExtFuncData;
 use crate::ir::instructions::{BranchInfo, CallInfo, InstructionData};
-use crate::ir::{types, ConstantPool, Immediate};
+use crate::ir::{types, ConstantData, ConstantPool, Immediate};
 use crate::ir::{
     Ebb, FuncRef, Inst, SigRef, Signature, Type, Value, ValueLabelAssignments, ValueList,
     ValueListPool,
@@ -14,7 +14,6 @@ use crate::isa::TargetIsa;
 use crate::packed_option::ReservedValue;
 use crate::write::write_operands;
 use crate::HashMap;
-use alloc::vec::Vec;
 use core::fmt;
 use core::iter;
 use core::mem;
@@ -73,7 +72,7 @@ pub struct DataFlowGraph {
     pub constants: ConstantPool,
 
     /// Stores large immediates that otherwise will not fit on InstructionData
-    pub immediates: PrimaryMap>,
+    pub immediates: PrimaryMap,
 }
 
 impl DataFlowGraph {
diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index 8ca70e5a44..8222f612a4 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -6,13 +6,13 @@
 
 use alloc::vec::Vec;
 use core::fmt::{self, Display, Formatter};
-use core::iter::FromIterator;
-use core::str::{from_utf8, FromStr};
+use core::str::FromStr;
 use core::{i32, u32};
 
 /// Convert a type into a vector of bytes; all implementors in this file must use little-endian
 /// orderings of bytes to match WebAssembly's little-endianness.
-trait IntoBytes {
+pub trait IntoBytes {
+    /// Return the little-endian byte representation of the implementing type.
     fn into_bytes(self) -> Vec;
 }
 
@@ -34,6 +34,12 @@ impl IntoBytes for i32 {
     }
 }
 
+impl IntoBytes for Vec {
+    fn into_bytes(self) -> Vec {
+        self
+    }
+}
+
 /// 64-bit immediate signed integer operand.
 ///
 /// An `Imm64` operand can also be used to represent immediate values of smaller integer types by
@@ -318,34 +324,6 @@ impl V128Imm {
     }
 }
 
-impl Display for V128Imm {
-    // Print a 128-bit vector in hexadecimal, e.g. 0x000102030405060708090a0b0c0d0e0f.
-    fn fmt(&self, f: &mut Formatter) -> fmt::Result {
-        write!(f, "0x")?;
-        let mut anything_written = false;
-        for &b in self.0.iter().rev() {
-            if b == 0 && !anything_written {
-                continue;
-            } else {
-                anything_written = true;
-                write!(f, "{:02x}", b)?;
-            }
-        }
-        if !anything_written {
-            write!(f, "00")?;
-        }
-        Ok(())
-    }
-}
-
-impl From for V128Imm {
-    fn from(x: u64) -> Self {
-        let mut buffer: [u8; 16] = [0; 16]; // zero-fill
-        (0..8).for_each(|byte| buffer[byte] = (x >> (byte as u64 * 8) & 0xff) as u8); // insert each byte from the u64 into v in little-endian order
-        V128Imm(buffer)
-    }
-}
-
 impl From<&[u8]> for V128Imm {
     fn from(slice: &[u8]) -> Self {
         assert_eq!(slice.len(), 16);
@@ -355,92 +333,6 @@ impl From<&[u8]> for V128Imm {
     }
 }
 
-impl FromStr for V128Imm {
-    type Err = &'static str;
-
-    // parse a 128-bit vector from a hexadecimal string, formatted as above
-    fn from_str(s: &str) -> Result {
-        if s.len() <= 2 || &s[0..2] != "0x" {
-            Err("Expected a hexadecimal string, e.g. 0x1234")
-        } else {
-            // clean and check the string
-            let cleaned: Vec = s[2..]
-                .as_bytes()
-                .iter()
-                .filter(|&&b| b as char != '_')
-                .cloned()
-                .collect(); // remove 0x prefix and any intervening _ characters
-
-            if cleaned.len() == 0 {
-                Err("Hexadecimal string must have some digits")
-            } else if cleaned.len() % 2 != 0 {
-                Err("Hexadecimal string must have an even number of digits")
-            } else if cleaned.len() > 32 {
-                Err("Hexadecimal string has too many digits to fit in a 128-bit vector")
-            } else {
-                let mut buffer = [0; 16]; // zero-fill the buffer
-                let mut position = cleaned.len() / 2 - 1; // since Uimm128 is little-endian but the string is not, we write from back to front but must start at the highest position required by the string
-                for i in (0..cleaned.len()).step_by(2) {
-                    let pair = from_utf8(&cleaned[i..i + 2])
-                        .or_else(|_| Err("Unable to parse hexadecimal pair as UTF-8"))?;
-                    let byte = u8::from_str_radix(pair, 16)
-                        .or_else(|_| Err("Unable to parse as hexadecimal"))?;
-                    buffer[position] = byte;
-                    position = position.wrapping_sub(1); // should only wrap on the last iteration
-                }
-
-                Ok(V128Imm(buffer))
-            }
-        }
-    }
-}
-
-/// Implement a way to convert an iterator of immediates to a Uimm128:
-///  - this expects the items in reverse order (e.g. last lane first) which is the natural output of pushing items into a vector
-///  - this may not fully consume the iterator or may fail if it cannot take the expected number of items
-///  - this requires the input type (i.e. $ty) to implement ToBytes
-macro_rules! construct_uimm128_from_iterator_of {
-    ( $ty:ident, $lanes:expr ) => {
-        impl FromIterator<$ty> for V128Imm {
-            fn from_iter>(iter: T) -> Self {
-                let mut buffer: [u8; 16] = [0; 16];
-                iter.into_iter()
-                    .take($lanes)
-                    .map(|f| f.into_bytes())
-                    .flat_map(|b| b)
-                    .enumerate()
-                    .for_each(|(i, b)| buffer[i] = b);
-                V128Imm(buffer)
-            }
-        }
-    };
-}
-
-/// Special case for booleans since we have to decide the bit-width based on the number of items
-impl FromIterator for V128Imm {
-    fn from_iter>(iter: T) -> Self {
-        let bools = Vec::from_iter(iter);
-        let count = bools.len();
-        assert!(count > 0 && count <= 16); // ensure we don't have too many booleans
-        assert_eq!(count & (count - 1), 0); // ensure count is a power of two, see https://stackoverflow.com/questions/600293
-        let mut buffer: [u8; 16] = [0; 16];
-        let step = 16 / count;
-        bools
-            .iter()
-            .enumerate()
-            .map(|(i, &b)| (i * step, if b { 1 } else { 0 }))
-            .for_each(|(i, b)| buffer[i] = b);
-        V128Imm(buffer)
-    }
-}
-
-construct_uimm128_from_iterator_of!(u8, 16);
-construct_uimm128_from_iterator_of!(i16, 8);
-construct_uimm128_from_iterator_of!(i32, 4);
-construct_uimm128_from_iterator_of!(Ieee32, 4);
-construct_uimm128_from_iterator_of!(Imm64, 2);
-construct_uimm128_from_iterator_of!(Ieee64, 2);
-
 /// 32-bit signed immediate offset.
 ///
 /// This is used to encode an immediate offset for load/store instructions. All supported ISAs have
@@ -1074,117 +966,6 @@ mod tests {
         parse_err::("0x0_0000_0000_0000_0000", "Too many hexadecimal digits");
     }
 
-    #[test]
-    fn format_uimm128() {
-        assert_eq!(V128Imm::from(0).to_string(), "0x00");
-        assert_eq!(V128Imm::from(42).to_string(), "0x2a");
-        assert_eq!(V128Imm::from(3735928559).to_string(), "0xdeadbeef");
-        assert_eq!(
-            V128Imm::from(0x0102030405060708).to_string(),
-            "0x0102030405060708"
-        );
-    }
-
-    #[test]
-    fn parse_uimm128() {
-        parse_ok::("0x00", "0x00");
-        parse_ok::("0x00000042", "0x42");
-        parse_ok::(
-            "0x0102030405060708090a0b0c0d0e0f00",
-            "0x0102030405060708090a0b0c0d0e0f00",
-        );
-        parse_ok::("0x_0000_0043_21", "0x4321");
-
-        parse_err::("", "Expected a hexadecimal string, e.g. 0x1234");
-        parse_err::("0x", "Expected a hexadecimal string, e.g. 0x1234");
-        parse_err::(
-            "0x042",
-            "Hexadecimal string must have an even number of digits",
-        );
-        parse_err::(
-            "0x00000000000000000000000000000000000000000000000000",
-            "Hexadecimal string has too many digits to fit in a 128-bit vector",
-        );
-        parse_err::("0xrstu", "Unable to parse as hexadecimal");
-        parse_err::("0x__", "Hexadecimal string must have some digits");
-    }
-
-    #[test]
-    fn uimm128_equivalence() {
-        assert_eq!(
-            "0x01".parse::().unwrap().0,
-            [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
-        );
-        assert_eq!(
-            V128Imm::from_iter(vec![1, 0, 0, 0]).0,
-            [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
-        );
-        assert_eq!(
-            V128Imm::from(1).0,
-            [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
-        );
-    }
-
-    #[test]
-    fn uimm128_endianness() {
-        assert_eq!(
-            "0x42".parse::().unwrap().0,
-            [0x42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
-        );
-        assert_eq!(
-            "0x00".parse::().unwrap().0,
-            [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
-        );
-        assert_eq!(
-            "0x12345678".parse::().unwrap().0,
-            [0x78, 0x56, 0x34, 0x12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
-        );
-        assert_eq!(
-            "0x1234_5678".parse::().unwrap().0,
-            [0x78, 0x56, 0x34, 0x12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
-        );
-    }
-
-    #[test]
-    fn uimm128_from_iter() {
-        assert_eq!(
-            V128Imm::from_iter(vec![4, 3, 2, 1]).0,
-            [4, 0, 0, 0, 3, 0, 0, 0, 2, 0, 0, 0, 1, 0, 0, 0]
-        );
-
-        assert_eq!(
-            V128Imm::from_iter(vec![false, true]).0,
-            [/* false */ 0, 0, 0, 0, 0, 0, 0, 0, /* true */ 1, 0, 0, 0, 0, 0, 0, 0]
-        );
-
-        assert_eq!(
-            V128Imm::from_iter(vec![false, true, false, true, false, true, false, true]).0,
-            [0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0]
-        );
-
-        #[allow(trivial_numeric_casts)]
-        let u8s = vec![
-            1 as u8, 2, 3, 4, 5, 6, 7, 8, 9, 0xa, 0xb, 0xc, 0xd, 0xe, 0xf, 0,
-        ];
-        assert_eq!(
-            V128Imm::from_iter(u8s).0,
-            [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 0]
-        );
-
-        #[allow(trivial_numeric_casts)]
-        let ieee32s: Vec = vec![32.4 as f32, 0.0, 1.0, 6.6666]
-            .iter()
-            .map(|&f| Ieee32::from(f))
-            .collect();
-        assert_eq!(
-            V128Imm::from_iter(ieee32s).0,
-            [
-                /* 32.4 == */ 0x9a, 0x99, 0x01, 0x42, /* 0 == */ 0, 0, 0, 0,
-                /* 1 == */ 0, 0, 0x80, 0x3f, /* 6.6666 == */ 0xca, 0x54, 0xd5, 0x40,
-            ]
-        )
-    }
-
     #[test]
     fn format_offset32() {
         assert_eq!(Offset32(0).to_string(), "");
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 64c0499d40..4b01e298d1 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -5,9 +5,8 @@ use crate::bitset::BitSet;
 use crate::cursor::{Cursor, FuncCursor};
 use crate::flowgraph::ControlFlowGraph;
 use crate::ir::condcodes::{FloatCC, IntCC};
-use crate::ir::immediates::V128Imm;
 use crate::ir::types::*;
-use crate::ir::{self, Function, Inst, InstBuilder};
+use crate::ir::{self, ConstantData, Function, Inst, InstBuilder};
 use crate::isa::constraints::*;
 use crate::isa::enc_tables::*;
 use crate::isa::encoding::base_size;
@@ -1111,7 +1110,11 @@ fn convert_ineg(
     {
         let value_type = pos.func.dfg.value_type(arg);
         if value_type.is_vector() && value_type.lane_type().is_int() {
-            let zero_immediate = pos.func.dfg.constants.insert(V128Imm::from(0).to_vec());
+            let zero_immediate = pos
+                .func
+                .dfg
+                .constants
+                .insert(ConstantData::from(vec![0; 16]));
             let zero_value = pos.ins().vconst(value_type, zero_immediate); // this should be legalized to a PXOR
             pos.func.dfg.replace(inst).isub(zero_value, arg);
         }
diff --git a/cranelift/codegen/src/write.rs b/cranelift/codegen/src/write.rs
index 910d1b988e..f2758e0c30 100644
--- a/cranelift/codegen/src/write.rs
+++ b/cranelift/codegen/src/write.rs
@@ -5,7 +5,6 @@
 
 use crate::entity::SecondaryMap;
 use crate::ir::entities::AnyEntity;
-use crate::ir::immediates::V128Imm;
 use crate::ir::{
     DataFlowGraph, DisplayFunctionAnnotations, Ebb, Function, Inst, SigRef, Type, Value, ValueDef,
     ValueLoc,
@@ -515,8 +514,7 @@ pub fn write_operands(
             let data = dfg.immediates.get(mask).expect(
                 "Expected the shuffle mask to already be inserted into the immediates table",
             );
-            let v128 = V128Imm::from(&data[..]);
-            write!(w, " {}, {}, {}", args[0], args[1], v128)
+            write!(w, " {}, {}, {}", args[0], args[1], data)
         }
         IntCompare { cond, args, .. } => write!(w, " {} {}, {}", cond, args[0], args[1]),
         IntCompareImm { cond, arg, imm, .. } => write!(w, " {} {}, {}", cond, arg, imm),
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 82437d7c6c..b4ca9b7ccd 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -9,20 +9,19 @@ use crate::testfile::{Comment, Details, Feature, TestFile};
 use cranelift_codegen::entity::EntityRef;
 use cranelift_codegen::ir;
 use cranelift_codegen::ir::entities::AnyEntity;
-use cranelift_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32, Uimm64, V128Imm};
+use cranelift_codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Offset32, Uimm32, Uimm64};
 use cranelift_codegen::ir::instructions::{InstructionData, InstructionFormat, VariableArgs};
 use cranelift_codegen::ir::types::INVALID;
 use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{
-    AbiParam, ArgumentExtension, ArgumentLoc, Ebb, ExtFuncData, ExternalName, FuncRef, Function,
-    GlobalValue, GlobalValueData, Heap, HeapData, HeapStyle, JumpTable, JumpTableData, MemFlags,
-    Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind, Table, TableData, Type,
-    Value, ValueLoc,
+    AbiParam, ArgumentExtension, ArgumentLoc, ConstantData, Ebb, ExtFuncData, ExternalName,
+    FuncRef, Function, GlobalValue, GlobalValueData, Heap, HeapData, HeapStyle, JumpTable,
+    JumpTableData, MemFlags, Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind,
+    Table, TableData, Type, Value, ValueLoc,
 };
 use cranelift_codegen::isa::{self, CallConv, Encoding, RegUnit, TargetIsa};
 use cranelift_codegen::packed_option::ReservedValue;
 use cranelift_codegen::{settings, timing};
-use std::iter::FromIterator;
 use std::mem;
 use std::str::FromStr;
 use std::{u16, u32};
@@ -595,16 +594,13 @@ impl<'a> Parser<'a> {
         }
     }
 
-    // Match and consume a Uimm128 immediate; due to size restrictions on InstructionData, Uimm128
-    // is boxed in cranelift-codegen/meta/src/shared/immediates.rs
-    fn match_uimm128(&mut self, err_msg: &str) -> ParseResult {
+    // Match and consume a hexadeximal immediate
+    fn match_hexadecimal_constant(&mut self, err_msg: &str) -> ParseResult {
         if let Some(Token::Integer(text)) = self.token() {
             self.consume();
-            // Lexer just gives us raw text that looks like hex code.
-            // Parse it as an Uimm128 to check for overflow and other issues.
             text.parse().map_err(|e| {
                 self.error(&format!(
-                    "expected u128 hexadecimal immediate, failed to parse: {}",
+                    "expected hexadecimal immediate, failed to parse: {}",
                     e
                 ))
             })
@@ -614,15 +610,27 @@ impl<'a> Parser<'a> {
     }
 
     // Match and consume either a hexadecimal Uimm128 immediate (e.g. 0x000102...) or its literal list form (e.g. [0 1 2...])
-    fn match_uimm128_or_literals(&mut self, controlling_type: Type) -> ParseResult {
-        if self.optional(Token::LBracket) {
+    fn match_constant_data(&mut self, controlling_type: Type) -> ParseResult {
+        let expected_size = controlling_type.bytes() as usize;
+        let constant_data = if self.optional(Token::LBracket) {
             // parse using a list of values, e.g. vconst.i32x4 [0 1 2 3]
             let uimm128 = self.parse_literals_to_uimm128(controlling_type)?;
             self.match_token(Token::RBracket, "expected a terminating right bracket")?;
-            Ok(uimm128)
+            uimm128
         } else {
-            // parse using a hexadecimal value
-            self.match_uimm128("expected an immediate hexadecimal operand")
+            // parse using a hexadecimal value, e.g. 0x000102...
+            let uimm128 =
+                self.match_hexadecimal_constant("expected an immediate hexadecimal operand")?;
+            uimm128.expand_to(expected_size)
+        };
+
+        if constant_data.len() == expected_size {
+            Ok(constant_data)
+        } else {
+            Err(self.error(&format!(
+                "expected parsed constant to have {} bytes",
+                expected_size
+            )))
         }
     }
 
@@ -851,29 +859,42 @@ impl<'a> Parser<'a> {
     }
 
     /// Parse a list of literals (i.e. integers, floats, booleans); e.g.
-    fn parse_literals_to_uimm128(&mut self, ty: Type) -> ParseResult {
+    fn parse_literals_to_uimm128(&mut self, ty: Type) -> ParseResult {
         macro_rules! consume {
             ( $ty:ident, $match_fn:expr ) => {{
                 assert!($ty.is_vector());
-                let mut v = Vec::with_capacity($ty.lane_count() as usize);
+                let mut data = ConstantData::default();
                 for _ in 0..$ty.lane_count() {
-                    v.push($match_fn?);
+                    data = data.append($match_fn);
                 }
-                V128Imm::from_iter(v)
+                data
             }};
         }
 
+        fn boolean_to_vec(value: bool, ty: Type) -> Vec {
+            let lane_size = ty.bytes() / u32::from(ty.lane_count());
+            if lane_size < 1 {
+                panic!("The boolean lane must have a byte size greater than zero.");
+            }
+            let mut buffer = vec![0; lane_size as usize];
+            buffer[0] = if value { 1 } else { 0 };
+            buffer
+        }
+
         if !ty.is_vector() {
             err!(self.loc, "Expected a controlling vector type, not {}", ty)
         } else {
             let uimm128 = match ty.lane_type() {
-                I8 => consume!(ty, self.match_uimm8("Expected an 8-bit unsigned integer")),
-                I16 => consume!(ty, self.match_imm16("Expected a 16-bit integer")),
-                I32 => consume!(ty, self.match_imm32("Expected a 32-bit integer")),
-                I64 => consume!(ty, self.match_imm64("Expected a 64-bit integer")),
-                F32 => consume!(ty, self.match_ieee32("Expected a 32-bit float...")),
-                F64 => consume!(ty, self.match_ieee64("Expected a 64-bit float")),
-                b if b.is_bool() => consume!(ty, self.match_bool("Expected a boolean")),
+                I8 => consume!(ty, self.match_uimm8("Expected an 8-bit unsigned integer")?),
+                I16 => consume!(ty, self.match_imm16("Expected a 16-bit integer")?),
+                I32 => consume!(ty, self.match_imm32("Expected a 32-bit integer")?),
+                I64 => consume!(ty, self.match_imm64("Expected a 64-bit integer")?),
+                F32 => consume!(ty, self.match_ieee32("Expected a 32-bit float...")?),
+                F64 => consume!(ty, self.match_ieee64("Expected a 64-bit float")?),
+                b if b.is_bool() => consume!(
+                    ty,
+                    boolean_to_vec(self.match_bool("Expected a boolean")?, ty)
+                ),
                 _ => return err!(self.loc, "Expected a type of: float, int, bool"),
             };
             Ok(uimm128)
@@ -2447,8 +2468,8 @@ impl<'a> Parser<'a> {
                     )
                 }
                 Some(controlling_type) => {
-                    let uimm128 = self.match_uimm128_or_literals(controlling_type)?;
-                    let constant_handle = ctx.function.dfg.constants.insert(uimm128.to_vec());
+                    let uimm128 = self.match_constant_data(controlling_type)?;
+                    let constant_handle = ctx.function.dfg.constants.insert(uimm128);
                     InstructionData::UnaryConst {
                         opcode,
                         constant_handle,
@@ -2460,8 +2481,8 @@ impl<'a> Parser<'a> {
                 self.match_token(Token::Comma, "expected ',' between operands")?;
                 let b = self.match_value("expected SSA value second operand")?;
                 self.match_token(Token::Comma, "expected ',' between operands")?;
-                let uimm128 = self.match_uimm128_or_literals(I8X16)?;
-                let mask = ctx.function.dfg.immediates.push(uimm128.to_vec());
+                let uimm128 = self.match_constant_data(I8X16)?;
+                let mask = ctx.function.dfg.immediates.push(uimm128);
                 InstructionData::Shuffle {
                     opcode,
                     mask,
@@ -3263,4 +3284,12 @@ mod tests {
         cannot_parse_as_uimm128!("1 2 3", I32X4);
         cannot_parse_as_uimm128!(" ", F32X4);
     }
+
+    #[test]
+    fn parse_constant_from_booleans() {
+        let c = Parser::new("true false true false")
+            .parse_literals_to_uimm128(B32X4)
+            .unwrap();
+        assert_eq!(c.to_vec(), [1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0])
+    }
 }
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 6dd7deb1ff..9f2aee1f91 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -33,7 +33,9 @@ use crate::wasm_unsupported;
 use core::{i32, u32};
 use cranelift_codegen::ir::condcodes::{FloatCC, IntCC};
 use cranelift_codegen::ir::types::*;
-use cranelift_codegen::ir::{self, InstBuilder, JumpTableData, MemFlags, Value, ValueLabel};
+use cranelift_codegen::ir::{
+    self, ConstantData, InstBuilder, JumpTableData, MemFlags, Value, ValueLabel,
+};
 use cranelift_codegen::packed_option::ReservedValue;
 use cranelift_frontend::{FunctionBuilder, Variable};
 use wasmparser::{MemoryImmediate, Operator};
@@ -1051,7 +1053,8 @@ pub fn translate_operator(
             let (vector_a, vector_b) = state.pop2();
             let a = optionally_bitcast_vector(vector_a, I8X16, builder);
             let b = optionally_bitcast_vector(vector_b, I8X16, builder);
-            let mask = builder.func.dfg.immediates.push(lanes.to_vec());
+            let lanes = ConstantData::from(lanes.as_ref());
+            let mask = builder.func.dfg.immediates.push(lanes);
             let shuffled = builder.ins().shuffle(a, b, mask);
             state.push1(shuffled)
             // At this point the original types of a and b are lost; users of this value (i.e. this

From ff93564c55c28f0236f1118000cb6c263948e1d5 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 9 Oct 2019 11:09:40 -0700
Subject: [PATCH 2845/3084] Update predicate documentation to match new
 ConstantData parameter

---
 cranelift/codegen/src/predicates.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/src/predicates.rs b/cranelift/codegen/src/predicates.rs
index 374e2bbbff..5812163e09 100644
--- a/cranelift/codegen/src/predicates.rs
+++ b/cranelift/codegen/src/predicates.rs
@@ -32,13 +32,13 @@ pub fn is_zero_32_bit_float>(x: T) -> bool {
     x32.bits() == 0
 }
 
-/// Check that a 128-bit vector contains all zeroes.
+/// Check that a constant contains all zeroes.
 #[allow(dead_code)]
 pub fn is_all_zeroes(x: &ConstantData) -> bool {
     x.iter().all(|&f| f == 0)
 }
 
-/// Check that a 128-bit vector contains all ones.
+/// Check that a constant contains all ones.
 #[allow(dead_code)]
 pub fn is_all_ones(x: &ConstantData) -> bool {
     x.iter().all(|&f| f == 0xff)

From a69b0fc22122d1dee19933e5a19f6abb581a5a8d Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 9 Oct 2019 15:02:05 -0700
Subject: [PATCH 2846/3084] Change match_literals_to_uimm128 to
 match_literals_to_constant_data

---
 cranelift/reader/src/parser.rs | 45 +++++++++++++++++-----------------
 1 file changed, 23 insertions(+), 22 deletions(-)

diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index b4ca9b7ccd..1b1698caf4 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -614,7 +614,7 @@ impl<'a> Parser<'a> {
         let expected_size = controlling_type.bytes() as usize;
         let constant_data = if self.optional(Token::LBracket) {
             // parse using a list of values, e.g. vconst.i32x4 [0 1 2 3]
-            let uimm128 = self.parse_literals_to_uimm128(controlling_type)?;
+            let uimm128 = self.parse_literals_to_constant_data(controlling_type)?;
             self.match_token(Token::RBracket, "expected a terminating right bracket")?;
             uimm128
         } else {
@@ -858,8 +858,9 @@ impl<'a> Parser<'a> {
         }
     }
 
-    /// Parse a list of literals (i.e. integers, floats, booleans); e.g.
-    fn parse_literals_to_uimm128(&mut self, ty: Type) -> ParseResult {
+    /// Parse a list of literals (i.e. integers, floats, booleans); e.g. `0 1 2 3`, usually as
+    /// part of something like `vconst.i32x4 [0 1 2 3]`.
+    fn parse_literals_to_constant_data(&mut self, ty: Type) -> ParseResult {
         macro_rules! consume {
             ( $ty:ident, $match_fn:expr ) => {{
                 assert!($ty.is_vector());
@@ -884,7 +885,7 @@ impl<'a> Parser<'a> {
         if !ty.is_vector() {
             err!(self.loc, "Expected a controlling vector type, not {}", ty)
         } else {
-            let uimm128 = match ty.lane_type() {
+            let constant_data = match ty.lane_type() {
                 I8 => consume!(ty, self.match_uimm8("Expected an 8-bit unsigned integer")?),
                 I16 => consume!(ty, self.match_imm16("Expected a 16-bit integer")?),
                 I32 => consume!(ty, self.match_imm32("Expected a 32-bit integer")?),
@@ -897,7 +898,7 @@ impl<'a> Parser<'a> {
                 ),
                 _ => return err!(self.loc, "Expected a type of: float, int, bool"),
             };
-            Ok(uimm128)
+            Ok(constant_data)
         }
     }
 
@@ -3256,39 +3257,39 @@ mod tests {
 
     #[test]
     fn uimm128() {
-        macro_rules! parse_as_uimm128 {
+        macro_rules! parse_as_constant_data {
             ($text:expr, $type:expr) => {{
-                Parser::new($text).parse_literals_to_uimm128($type)
+                Parser::new($text).parse_literals_to_constant_data($type)
             }};
         }
-        macro_rules! can_parse_as_uimm128 {
+        macro_rules! can_parse_as_constant_data {
             ($text:expr, $type:expr) => {{
-                assert!(parse_as_uimm128!($text, $type).is_ok())
+                assert!(parse_as_constant_data!($text, $type).is_ok())
             }};
         }
-        macro_rules! cannot_parse_as_uimm128 {
+        macro_rules! cannot_parse_as_constant_data {
             ($text:expr, $type:expr) => {{
-                assert!(parse_as_uimm128!($text, $type).is_err())
+                assert!(parse_as_constant_data!($text, $type).is_err())
             }};
         }
 
-        can_parse_as_uimm128!("1 2 3 4", I32X4);
-        can_parse_as_uimm128!("1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16", I8X16);
-        can_parse_as_uimm128!("0x1.1 0x2.2 0x3.3 0x4.4", F32X4);
-        can_parse_as_uimm128!("true false true false true false true false", B16X8);
-        can_parse_as_uimm128!("0 -1", I64X2);
-        can_parse_as_uimm128!("true false", B64X2);
-        can_parse_as_uimm128!("true true true true true", B32X4); // note that parse_literals_to_uimm128 will leave extra tokens unconsumed
+        can_parse_as_constant_data!("1 2 3 4", I32X4);
+        can_parse_as_constant_data!("1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16", I8X16);
+        can_parse_as_constant_data!("0x1.1 0x2.2 0x3.3 0x4.4", F32X4);
+        can_parse_as_constant_data!("true false true false true false true false", B16X8);
+        can_parse_as_constant_data!("0 -1", I64X2);
+        can_parse_as_constant_data!("true false", B64X2);
+        can_parse_as_constant_data!("true true true true true", B32X4); // note that parse_literals_to_constant_data will leave extra tokens unconsumed
 
-        cannot_parse_as_uimm128!("0x0 0x1 0x2 0x3", I32X4);
-        cannot_parse_as_uimm128!("1 2 3", I32X4);
-        cannot_parse_as_uimm128!(" ", F32X4);
+        cannot_parse_as_constant_data!("0x0 0x1 0x2 0x3", I32X4);
+        cannot_parse_as_constant_data!("1 2 3", I32X4);
+        cannot_parse_as_constant_data!(" ", F32X4);
     }
 
     #[test]
     fn parse_constant_from_booleans() {
         let c = Parser::new("true false true false")
-            .parse_literals_to_uimm128(B32X4)
+            .parse_literals_to_constant_data(B32X4)
             .unwrap();
         assert_eq!(c.to_vec(), [1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0])
     }

From 67733bd2fc7f976b30ae7ec47416c305117b5157 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 11 Oct 2019 09:26:08 -0700
Subject: [PATCH 2847/3084] Use ConstantData exclusively for inserting data
 into the constant pool

Previously we allowed anything that could be converted into ConstantData (e.g. a Vec).
---
 cranelift/codegen/meta/src/gen_legalizer.rs |  2 +-
 cranelift/codegen/src/ir/constant.rs        | 35 ++++++++++++---------
 cranelift/codegen/src/isa/x86/enc_tables.rs | 16 +++-------
 cranelift/wasm/src/code_translator.rs       |  3 +-
 cranelift/wasm/src/func_translator.rs       |  2 +-
 5 files changed, 30 insertions(+), 28 deletions(-)

diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index 13c4351fb4..f4033c64b5 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -470,7 +470,7 @@ fn gen_transform<'a>(
         for (name, value) in transform.const_pool.iter() {
             fmtln!(
                 fmt,
-                "let {} = pos.func.dfg.constants.insert(vec!{:?});",
+                "let {} = pos.func.dfg.constants.insert(vec!{:?}.into());",
                 name,
                 value
             );
diff --git a/cranelift/codegen/src/ir/constant.rs b/cranelift/codegen/src/ir/constant.rs
index f29c8fbada..1bd9fc8b71 100644
--- a/cranelift/codegen/src/ir/constant.rs
+++ b/cranelift/codegen/src/ir/constant.rs
@@ -14,6 +14,7 @@ use crate::HashMap;
 use alloc::collections::BTreeMap;
 use alloc::vec::Vec;
 use core::fmt;
+use core::iter::FromIterator;
 use core::slice::Iter;
 use core::str::{from_utf8, FromStr};
 use cranelift_entity::EntityRef;
@@ -25,6 +26,13 @@ use cranelift_entity::EntityRef;
 #[derive(Clone, Hash, Eq, PartialEq, Debug, Default)]
 pub struct ConstantData(Vec);
 
+impl FromIterator for ConstantData {
+    fn from_iter>(iter: T) -> Self {
+        let v = iter.into_iter().collect();
+        ConstantData(v)
+    }
+}
+
 impl From> for ConstantData {
     fn from(v: Vec) -> Self {
         Self(v)
@@ -211,8 +219,7 @@ impl ConstantPool {
     /// Insert constant data into the pool, returning a handle for later referencing; when constant
     /// data is inserted that is a duplicate of previous constant data, the existing handle will be
     /// returned.
-    pub fn insert(&mut self, constant_value: impl Into) -> Constant {
-        let constant_value = constant_value.into();
+    pub fn insert(&mut self, constant_value: ConstantData) -> Constant {
         if self.values_to_handles.contains_key(&constant_value) {
             self.values_to_handles.get(&constant_value).unwrap().clone()
         } else {
@@ -297,24 +304,24 @@ mod tests {
     #[test]
     fn insert() {
         let mut sut = ConstantPool::new();
-        sut.insert(vec![1, 2, 3]);
-        sut.insert(vec![4, 5, 6]);
+        sut.insert(vec![1, 2, 3].into());
+        sut.insert(vec![4, 5, 6].into());
         assert_eq!(sut.len(), 2);
     }
 
     #[test]
     fn insert_duplicate() {
         let mut sut = ConstantPool::new();
-        let a = sut.insert(vec![1, 2, 3]);
-        sut.insert(vec![4, 5, 6]);
-        let b = sut.insert(vec![1, 2, 3]);
+        let a = sut.insert(vec![1, 2, 3].into());
+        sut.insert(vec![4, 5, 6].into());
+        let b = sut.insert(vec![1, 2, 3].into());
         assert_eq!(a, b);
     }
 
     #[test]
     fn clear() {
         let mut sut = ConstantPool::new();
-        sut.insert(vec![1, 2, 3]);
+        sut.insert(vec![1, 2, 3].into());
         assert_eq!(sut.len(), 1);
 
         sut.clear();
@@ -324,9 +331,9 @@ mod tests {
     #[test]
     fn iteration_order() {
         let mut sut = ConstantPool::new();
-        sut.insert(vec![1, 2, 3]);
-        sut.insert(vec![4, 5, 6]);
-        sut.insert(vec![1, 2, 3]);
+        sut.insert(vec![1, 2, 3].into());
+        sut.insert(vec![4, 5, 6].into());
+        sut.insert(vec![1, 2, 3].into());
         let data = sut.iter().map(|(_, v)| v).collect::>();
         assert_eq!(data, vec![&vec![1, 2, 3].into(), &vec![4, 5, 6].into()]);
     }
@@ -335,7 +342,7 @@ mod tests {
     fn get() {
         let mut sut = ConstantPool::new();
         let data = vec![1, 2, 3];
-        let handle = sut.insert(data.clone());
+        let handle = sut.insert(data.clone().into());
         assert_eq!(sut.get(handle), &data.into());
     }
 
@@ -350,7 +357,7 @@ mod tests {
     #[test]
     fn get_offset() {
         let mut sut = ConstantPool::new();
-        let a = sut.insert(vec![1]);
+        let a = sut.insert(vec![1].into());
         sut.set_offset(a, 42);
         assert_eq!(sut.get_offset(a), 42)
     }
@@ -359,7 +366,7 @@ mod tests {
     #[should_panic]
     fn get_nonexistent_offset() {
         let mut sut = ConstantPool::new();
-        let a = sut.insert(vec![1]);
+        let a = sut.insert(vec![1].into());
         sut.get_offset(a); // panics, set_offset should have been called
     }
 
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 4b01e298d1..13e9d4ecef 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -6,7 +6,7 @@ use crate::cursor::{Cursor, FuncCursor};
 use crate::flowgraph::ControlFlowGraph;
 use crate::ir::condcodes::{FloatCC, IntCC};
 use crate::ir::types::*;
-use crate::ir::{self, ConstantData, Function, Inst, InstBuilder};
+use crate::ir::{self, Function, Inst, InstBuilder};
 use crate::isa::constraints::*;
 use crate::isa::enc_tables::*;
 use crate::isa::encoding::base_size;
@@ -15,7 +15,6 @@ use crate::isa::RegUnit;
 use crate::isa::{self, TargetIsa};
 use crate::predicates;
 use crate::regalloc::RegDiversions;
-use std::vec::Vec;
 
 include!(concat!(env!("OUT_DIR"), "/encoding-x86.rs"));
 include!(concat!(env!("OUT_DIR"), "/legalize-x86.rs"));
@@ -928,7 +927,7 @@ fn convert_shuffle(
             .clone();
         if a == b {
             // PSHUFB the first argument (since it is the same as the second).
-            let constructed_mask: Vec = mask
+            let constructed_mask = mask
                 .iter()
                 // If the mask is greater than 15 it still may be referring to a lane in b.
                 .map(|&b| if b > 15 { b.wrapping_sub(16) } else { b })
@@ -942,8 +941,7 @@ fn convert_shuffle(
             pos.func.dfg.replace(inst).x86_pshufb(a, mask_value);
         } else {
             // PSHUFB the first argument, placing zeroes for unused lanes.
-            let constructed_mask: Vec =
-                mask.iter().cloned().map(zero_unknown_lane_index).collect();
+            let constructed_mask = mask.iter().cloned().map(zero_unknown_lane_index).collect();
             let handle = pos.func.dfg.constants.insert(constructed_mask);
             // Move the built mask into another XMM register.
             let a_type = pos.func.dfg.value_type(a);
@@ -952,7 +950,7 @@ fn convert_shuffle(
             let shuffled_first_arg = pos.ins().x86_pshufb(a, mask_value);
 
             // PSHUFB the second argument, placing zeroes for unused lanes.
-            let constructed_mask: Vec = mask
+            let constructed_mask = mask
                 .iter()
                 .map(|b| b.wrapping_sub(16))
                 .map(zero_unknown_lane_index)
@@ -1110,11 +1108,7 @@ fn convert_ineg(
     {
         let value_type = pos.func.dfg.value_type(arg);
         if value_type.is_vector() && value_type.lane_type().is_int() {
-            let zero_immediate = pos
-                .func
-                .dfg
-                .constants
-                .insert(ConstantData::from(vec![0; 16]));
+            let zero_immediate = pos.func.dfg.constants.insert(vec![0; 16].into());
             let zero_value = pos.ins().vconst(value_type, zero_immediate); // this should be legalized to a PXOR
             pos.func.dfg.replace(inst).isub(zero_value, arg);
         }
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 9f2aee1f91..c5a56f0ebe 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -998,7 +998,8 @@ pub fn translate_operator(
             return Err(wasm_unsupported!("proposed bulk memory operator {:?}", op));
         }
         Operator::V128Const { value } => {
-            let handle = builder.func.dfg.constants.insert(value.bytes().to_vec());
+            let data = value.bytes().to_vec().into();
+            let handle = builder.func.dfg.constants.insert(data);
             let value = builder.ins().vconst(I8X16, handle);
             // the v128.const is typed in CLIF as a I8x16 but raw_bitcast to a different type before use
             state.push1(value)
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index 39b8dd43ca..08d248157f 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -189,7 +189,7 @@ fn declare_locals(
         F32 => builder.ins().f32const(ir::immediates::Ieee32::with_bits(0)),
         F64 => builder.ins().f64const(ir::immediates::Ieee64::with_bits(0)),
         V128 => {
-            let constant_handle = builder.func.dfg.constants.insert([0; 16].to_vec());
+            let constant_handle = builder.func.dfg.constants.insert([0; 16].to_vec().into());
             builder.ins().vconst(ir::types::I8X16, constant_handle)
         }
         AnyRef => builder.ins().null(environ.reference_type()),

From 6460fe705fed7db7015b2bae06ceaa3e3b8e4587 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 2 Oct 2019 13:38:54 -0700
Subject: [PATCH 2848/3084] Add x86 SIMD ishl

Only the shifts with applicable SSE2 instructions (i.e. 16-64 bit width) are implemented here.
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 21 ++++++++++
 .../codegen/meta/src/isa/x86/instructions.rs  | 35 +++++++++++++++++
 .../codegen/meta/src/isa/x86/legalize.rs      | 15 ++++++-
 cranelift/codegen/meta/src/isa/x86/opcodes.rs |  9 +++++
 .../isa/x86/simd-bitwise-binemit.clif         | 21 ++++++++++
 .../isa/x86/simd-bitwise-legalize.clif        | 13 +++++++
 .../filetests/isa/x86/simd-bitwise-run.clif   | 39 +++++++++++++++++++
 7 files changed, 152 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index a029e8170f..f7f8964905 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -521,6 +521,7 @@ pub(crate) fn define<'defs>(
     let x86_pinsr = x86.by_name("x86_pinsr");
     let x86_pshufd = x86.by_name("x86_pshufd");
     let x86_pshufb = x86.by_name("x86_pshufb");
+    let x86_psll = x86.by_name("x86_psll");
     let x86_push = x86.by_name("x86_push");
     let x86_sdivmodx = x86.by_name("x86_sdivmodx");
     let x86_smulx = x86.by_name("x86_smulx");
@@ -1988,6 +1989,26 @@ pub(crate) fn define<'defs>(
         e.enc_32_64(bxor, rec_fa.opcodes(&PXOR));
     }
 
+    // SIMD bitcast from I32/I64 to the low bits of a vector (e.g. I64x2); this register movement
+    // allows SIMD shifts to be legalized more easily. TODO ideally this would be typed as an
+    // I128x1 but restrictions on the type builder prevent this; the general idea here is that
+    // the upper bits are all zeroed and do not form parts of any separate lane. See
+    // https://github.com/CraneStation/cranelift/issues/1146.
+    e.enc_both(
+        bitcast.bind(vector(I64, sse_vector_size)).bind(I32),
+        rec_frurm.opcodes(&MOVD_LOAD_XMM),
+    );
+    e.enc64(
+        bitcast.bind(vector(I64, sse_vector_size)).bind(I64),
+        rec_frurm.opcodes(&MOVD_LOAD_XMM).rex().w(),
+    );
+
+    // SIMD shift left
+    for (ty, opcodes) in &[(I16, &PSLLW), (I32, &PSLLD), (I64, &PSLLQ)] {
+        let x86_psll = x86_psll.bind(vector(*ty, sse_vector_size));
+        e.enc_32_64(x86_psll, rec_fa.opcodes(*opcodes));
+    }
+
     // SIMD icmp using PCMPEQ*
     for ty in ValueType::all_lane_types().filter(|t| t.is_int() && allowed_simd_type(t)) {
         let (opcodes, isa_predicate): (&[_], _) = match ty.lane_bits() {
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index b9f2496a85..c8839e78a8 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -387,5 +387,40 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    let IxN = &TypeVar::new(
+        "IxN",
+        "A SIMD vector type containing integers",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .simd_lanes(Interval::All)
+            .includes_scalars(false)
+            .build(),
+    );
+    let I64x2 = &TypeVar::new(
+        "I64x2",
+        "A SIMD vector type containing one large integer (the upper lane is concatenated with \
+         the lower lane to form the integer)",
+        TypeSetBuilder::new()
+            .ints(64..64)
+            .simd_lanes(2..2)
+            .includes_scalars(false)
+            .build(),
+    );
+    let x = &operand_doc("x", IxN, "Vector value to shift");
+    let y = &operand_doc("y", I64x2, "Number of bits to shift");
+    let a = &operand("a", IxN);
+    ig.push(
+        Inst::new(
+            "x86_psll",
+            r#"
+        Shift Packed Data Left Logical -- This implements the behavior of the shared instruction 
+        ``ishl`` but alters the shift operand to live in an XMM register as expected by the PSSL*
+        family of instructions.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
     ig.build()
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 04951c3d5b..8b71bfd637 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -3,7 +3,7 @@ use crate::cdsl::instructions::{vector, Bindable, InstructionGroup};
 use crate::cdsl::types::{LaneType, ValueType};
 use crate::cdsl::xform::TransformGroupBuilder;
 use crate::shared::types::Float::F64;
-use crate::shared::types::Int::{I32, I64};
+use crate::shared::types::Int::{I16, I32, I64};
 use crate::shared::Definitions as SharedDefinitions;
 
 pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGroup) {
@@ -20,6 +20,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     // List of instructions.
     let insts = &shared.instructions;
     let band = insts.by_name("band");
+    let bitcast = insts.by_name("bitcast");
     let bor = insts.by_name("bor");
     let bnot = insts.by_name("bnot");
     let bxor = insts.by_name("bxor");
@@ -40,6 +41,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let imul = insts.by_name("imul");
     let ineg = insts.by_name("ineg");
     let insertlane = insts.by_name("insertlane");
+    let ishl = insts.by_name("ishl");
     let isub = insts.by_name("isub");
     let popcnt = insts.by_name("popcnt");
     let raw_bitcast = insts.by_name("raw_bitcast");
@@ -60,6 +62,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let x86_bsr = x86_instructions.by_name("x86_bsr");
     let x86_pshufb = x86_instructions.by_name("x86_pshufb");
     let x86_pshufd = x86_instructions.by_name("x86_pshufd");
+    let x86_psll = x86_instructions.by_name("x86_psll");
     let x86_umulx = x86_instructions.by_name("x86_umulx");
     let x86_smulx = x86_instructions.by_name("x86_smulx");
 
@@ -394,6 +397,16 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         );
     }
 
+    // SIMD shift left
+    for ty in &[I16, I32, I64] {
+        let ishl = ishl.bind(vector(*ty, sse_vector_size));
+        let bitcast = bitcast.bind(vector(I64, sse_vector_size));
+        narrow.legalize(
+            def!(a = ishl(x, y)),
+            vec![def!(b = bitcast(y)), def!(a = x86_psll(x, b))],
+        );
+    }
+
     narrow.custom_legalize(shuffle, "convert_shuffle");
     narrow.custom_legalize(extractlane, "convert_extractlane");
     narrow.custom_legalize(insertlane, "convert_insertlane");
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index b7f223eb27..6e3859d848 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -320,6 +320,15 @@ pub static PSHUFB: [u8; 4] = [0x66, 0x0f, 0x38, 0x00];
 /// store the result in xmm1 (SSE2).
 pub static PSHUFD: [u8; 3] = [0x66, 0x0f, 0x70];
 
+/// Shift words in xmm1 left by xmm2/m128 while shifting in 0s (SSE2).
+pub static PSLLW: [u8; 3] = [0x66, 0x0f, 0xf1];
+
+/// Shift doublewords in xmm1 left by xmm2/m128 while shifting in 0s (SSE2).
+pub static PSLLD: [u8; 3] = [0x66, 0x0f, 0xf2];
+
+/// Shift quadwords in xmm1 left by xmm2/m128 while shifting in 0s (SSE2).
+pub static PSLLQ: [u8; 3] = [0x66, 0x0f, 0xf3];
+
 /// Subtract packed byte integers in xmm2/m128 from packed byte integers in xmm1 (SSE2).
 pub static PSUBB: [u8; 3] = [0x66, 0x0f, 0xf8];
 
diff --git a/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif
new file mode 100644
index 0000000000..5cfb4375d7
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif
@@ -0,0 +1,21 @@
+test binemit
+set enable_simd
+target x86_64 skylake
+
+function %ishl_i16x8(i16x8, i64x2) -> i16x8 {
+ebb0(v0: i16x8 [%xmm2], v1: i64x2 [%xmm1]):
+[-, %xmm2]  v2 = x86_psll v0, v1     ; bin: 66 0f f1 d1
+            return v2
+}
+
+function %ishl_i32x4(i32x4, i64x2) -> i32x4 {
+ebb0(v0: i32x4 [%xmm4], v1: i64x2 [%xmm0]):
+[-, %xmm4]  v2 = x86_psll v0, v1      ; bin: 66 0f f2 e0
+            return v2
+}
+
+function %ishl_i64x2(i64x2, i64x2) -> i64x2 {
+ebb0(v0: i64x2 [%xmm6], v1: i64x2 [%xmm3]):
+[-, %xmm6]  v2 = x86_psll v0, v1      ; bin: 66 0f f3 f3
+            return v2
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif
new file mode 100644
index 0000000000..5c2893950d
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif
@@ -0,0 +1,13 @@
+test legalizer
+set enable_simd
+target x86_64 skylake
+
+function %ishl_i32x4() -> i32x4 {
+ebb0:
+    v0 = iconst.i32 1
+    v1 = vconst.i32x4 [1 2 4 8]
+    v2 = ishl v1, v0
+    ; check: v3 = bitcast.i64x2 v0
+    ; nextln: v2 = x86_psll v1, v3
+    return v2
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif b/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
new file mode 100644
index 0000000000..224b3d5470
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
@@ -0,0 +1,39 @@
+test run
+set enable_simd
+target x86_64 skylake
+
+; TODO: once available, replace all lane extraction with `icmp + all_ones`
+
+function %ishl_i32x4() -> b1 {
+ebb0:
+    v0 = iconst.i32 1
+    v1 = vconst.i32x4 [1 2 4 8]
+    v2 = ishl v1, v0
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 2
+
+    v5 = extractlane v2, 3
+    v6 = icmp_imm eq v5, 16
+
+    v7 = band v4, v6
+    return v7
+}
+; run
+
+function %ishl_too_large_i16x8() -> b1 {
+ebb0:
+    v0 = iconst.i32 17 ; note that this will shift off the end of each lane
+    v1 = vconst.i16x8 [1 2 4 8 16 32 64 128]
+    v2 = ishl v1, v0
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 0
+
+    v5 = extractlane v2, 3
+    v6 = icmp_imm eq v5, 0
+
+    v7 = band v4, v6
+    return v7
+}
+; run

From 808885ce56e8c2b7c0934ac79f5ca3142e093849 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 2 Oct 2019 13:55:44 -0700
Subject: [PATCH 2849/3084] Translate WASM shl to CLIF ishl

Note how, according to the spec (see https://github.com/WebAssembly/simd/blob/master/proposals/simd/SIMD.md#left-shift-by-scalar), the shift count is computed modulo the lane width. E.g., a shift count of 17 on an i16x8 should not result in all zeroes as it does with Cranelift's `ishl` and x86's `PSSLW`--it should shift once to the left.
---
 cranelift/wasm/src/code_translator.rs | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index c5a56f0ebe..e2514909ec 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1111,6 +1111,15 @@ pub fn translate_operator(
             let (a, b) = state.pop2();
             state.push1(builder.ins().bxor(a, b));
         }
+        Operator::I16x8Shl | Operator::I32x4Shl | Operator::I64x2Shl => {
+            let (a, b) = state.pop2();
+            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
+            let bitwidth = i64::from(builder.func.dfg.value_type(a).bits());
+            // The spec expects to shift with `b mod lanewidth`; so, e.g., for 16 bit lane-width
+            // we do `b AND 15`; this means fewer instructions than `iconst + urem`.
+            let b_mod_bitwidth = builder.ins().band_imm(b, bitwidth - 1);
+            state.push1(builder.ins().ishl(bitcast_a, b_mod_bitwidth))
+        }
         Operator::I8x16Eq
         | Operator::I8x16Ne
         | Operator::I8x16LtS
@@ -1162,17 +1171,14 @@ pub fn translate_operator(
         | Operator::I8x16Mul
         | Operator::I16x8AnyTrue
         | Operator::I16x8AllTrue
-        | Operator::I16x8Shl
         | Operator::I16x8ShrS
         | Operator::I16x8ShrU
         | Operator::I32x4AnyTrue
         | Operator::I32x4AllTrue
-        | Operator::I32x4Shl
         | Operator::I32x4ShrS
         | Operator::I32x4ShrU
         | Operator::I64x2AnyTrue
         | Operator::I64x2AllTrue
-        | Operator::I64x2Shl
         | Operator::I64x2ShrS
         | Operator::I64x2ShrU
         | Operator::F32x4Abs

From f1904bffea935bcc68f1e9666bf018c46653e144 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 7 Oct 2019 10:38:35 -0700
Subject: [PATCH 2850/3084] Add x86 SIMD sshr and ushr

Only the shifts with applicable SSE2 instructions are implemented here: PSRL* (for ushr) only has 16-64 bit instructions and PSRA* (for sshr) only has 16-32 bit instructions.
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 14 ++++
 .../codegen/meta/src/isa/x86/instructions.rs  | 26 ++++++-
 .../codegen/meta/src/isa/x86/legalize.rs      | 26 ++++++-
 cranelift/codegen/meta/src/isa/x86/opcodes.rs | 15 ++++
 .../isa/x86/simd-bitwise-binemit.clif         | 30 ++++++++
 .../isa/x86/simd-bitwise-legalize.clif        | 20 ++++++
 .../filetests/isa/x86/simd-bitwise-run.clif   | 68 +++++++++++++++++++
 7 files changed, 197 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index f7f8964905..8457369929 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -522,6 +522,8 @@ pub(crate) fn define<'defs>(
     let x86_pshufd = x86.by_name("x86_pshufd");
     let x86_pshufb = x86.by_name("x86_pshufb");
     let x86_psll = x86.by_name("x86_psll");
+    let x86_psra = x86.by_name("x86_psra");
+    let x86_psrl = x86.by_name("x86_psrl");
     let x86_push = x86.by_name("x86_push");
     let x86_sdivmodx = x86.by_name("x86_sdivmodx");
     let x86_smulx = x86.by_name("x86_smulx");
@@ -2009,6 +2011,18 @@ pub(crate) fn define<'defs>(
         e.enc_32_64(x86_psll, rec_fa.opcodes(*opcodes));
     }
 
+    // SIMD shift right (logical)
+    for (ty, opcodes) in &[(I16, &PSRLW), (I32, &PSRLD), (I64, &PSRLQ)] {
+        let x86_psrl = x86_psrl.bind(vector(*ty, sse_vector_size));
+        e.enc_32_64(x86_psrl, rec_fa.opcodes(*opcodes));
+    }
+
+    // SIMD shift right (arithmetic)
+    for (ty, opcodes) in &[(I16, &PSRAW), (I32, &PSRAD)] {
+        let x86_psra = x86_psra.bind(vector(*ty, sse_vector_size));
+        e.enc_32_64(x86_psra, rec_fa.opcodes(*opcodes));
+    }
+
     // SIMD icmp using PCMPEQ*
     for ty in ValueType::all_lane_types().filter(|t| t.is_int() && allowed_simd_type(t)) {
         let (opcodes, isa_predicate): (&[_], _) = match ty.lane_bits() {
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index c8839e78a8..6ed1e88999 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -414,13 +414,37 @@ pub(crate) fn define(
             "x86_psll",
             r#"
         Shift Packed Data Left Logical -- This implements the behavior of the shared instruction 
-        ``ishl`` but alters the shift operand to live in an XMM register as expected by the PSSL*
+        ``ishl`` but alters the shift operand to live in an XMM register as expected by the PSLL*
         family of instructions.
         "#,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
     );
+    ig.push(
+        Inst::new(
+            "x86_psrl",
+            r#"
+        Shift Packed Data Right Logical -- This implements the behavior of the shared instruction 
+        ``ushr`` but alters the shift operand to live in an XMM register as expected by the PSRL*
+        family of instructions.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+    ig.push(
+        Inst::new(
+            "x86_psra",
+            r#"
+        Shift Packed Data Right Arithmetic -- This implements the behavior of the shared 
+        instruction ``sshr`` but alters the shift operand to live in an XMM register as expected by 
+        the PSRA* family of instructions.
+        "#,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
 
     ig.build()
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 8b71bfd637..8d9033d3a9 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -52,10 +52,12 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let splat = insts.by_name("splat");
     let shuffle = insts.by_name("shuffle");
     let srem = insts.by_name("srem");
+    let sshr = insts.by_name("sshr");
     let udiv = insts.by_name("udiv");
     let umulhi = insts.by_name("umulhi");
     let ushr_imm = insts.by_name("ushr_imm");
     let urem = insts.by_name("urem");
+    let ushr = insts.by_name("ushr");
     let vconst = insts.by_name("vconst");
 
     let x86_bsf = x86_instructions.by_name("x86_bsf");
@@ -63,6 +65,8 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let x86_pshufb = x86_instructions.by_name("x86_pshufb");
     let x86_pshufd = x86_instructions.by_name("x86_pshufd");
     let x86_psll = x86_instructions.by_name("x86_psll");
+    let x86_psra = x86_instructions.by_name("x86_psra");
+    let x86_psrl = x86_instructions.by_name("x86_psrl");
     let x86_umulx = x86_instructions.by_name("x86_umulx");
     let x86_smulx = x86_instructions.by_name("x86_smulx");
 
@@ -397,7 +401,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         );
     }
 
-    // SIMD shift left
+    // SIMD shift left (logical)
     for ty in &[I16, I32, I64] {
         let ishl = ishl.bind(vector(*ty, sse_vector_size));
         let bitcast = bitcast.bind(vector(I64, sse_vector_size));
@@ -407,6 +411,26 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         );
     }
 
+    // SIMD shift right (logical)
+    for ty in &[I16, I32, I64] {
+        let ushr = ushr.bind(vector(*ty, sse_vector_size));
+        let bitcast = bitcast.bind(vector(I64, sse_vector_size));
+        narrow.legalize(
+            def!(a = ushr(x, y)),
+            vec![def!(b = bitcast(y)), def!(a = x86_psrl(x, b))],
+        );
+    }
+
+    // SIMD shift left (arithmetic)
+    for ty in &[I16, I32, I64] {
+        let sshr = sshr.bind(vector(*ty, sse_vector_size));
+        let bitcast = bitcast.bind(vector(I64, sse_vector_size));
+        narrow.legalize(
+            def!(a = sshr(x, y)),
+            vec![def!(b = bitcast(y)), def!(a = x86_psra(x, b))],
+        );
+    }
+
     narrow.custom_legalize(shuffle, "convert_shuffle");
     narrow.custom_legalize(extractlane, "convert_extractlane");
     narrow.custom_legalize(insertlane, "convert_insertlane");
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index 6e3859d848..0fa7c8a7f7 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -329,6 +329,21 @@ pub static PSLLD: [u8; 3] = [0x66, 0x0f, 0xf2];
 /// Shift quadwords in xmm1 left by xmm2/m128 while shifting in 0s (SSE2).
 pub static PSLLQ: [u8; 3] = [0x66, 0x0f, 0xf3];
 
+/// Shift words in xmm1 right by xmm2/m128 while shifting in 0s (SSE2).
+pub static PSRLW: [u8; 3] = [0x66, 0x0f, 0xd1];
+
+/// Shift doublewords in xmm1 right by xmm2/m128 while shifting in 0s (SSE2).
+pub static PSRLD: [u8; 3] = [0x66, 0x0f, 0xd2];
+
+/// Shift quadwords in xmm1 right by xmm2/m128 while shifting in 0s (SSE2).
+pub static PSRLQ: [u8; 3] = [0x66, 0x0f, 0xd3];
+
+/// Shift words in xmm1 right by xmm2/m128 while shifting in sign bits (SSE2).
+pub static PSRAW: [u8; 3] = [0x66, 0x0f, 0xe1];
+
+/// Shift doublewords in xmm1 right by xmm2/m128 while shifting in sign bits (SSE2).
+pub static PSRAD: [u8; 3] = [0x66, 0x0f, 0xe2];
+
 /// Subtract packed byte integers in xmm2/m128 from packed byte integers in xmm1 (SSE2).
 pub static PSUBB: [u8; 3] = [0x66, 0x0f, 0xf8];
 
diff --git a/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif
index 5cfb4375d7..2a6530f7b5 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif
@@ -19,3 +19,33 @@ ebb0(v0: i64x2 [%xmm6], v1: i64x2 [%xmm3]):
 [-, %xmm6]  v2 = x86_psll v0, v1      ; bin: 66 0f f3 f3
             return v2
 }
+
+function %ushr_i16x8(i16x8, i64x2) -> i16x8 {
+ebb0(v0: i16x8 [%xmm2], v1: i64x2 [%xmm1]):
+[-, %xmm2]  v2 = x86_psrl v0, v1     ; bin: 66 0f d1 d1
+            return v2
+}
+
+function %ushr_i32x4(i32x4, i64x2) -> i32x4 {
+ebb0(v0: i32x4 [%xmm4], v1: i64x2 [%xmm0]):
+[-, %xmm4]  v2 = x86_psrl v0, v1      ; bin: 66 0f d2 e0
+            return v2
+}
+
+function %ushr_i64x2(i64x2, i64x2) -> i64x2 {
+ebb0(v0: i64x2 [%xmm6], v1: i64x2 [%xmm3]):
+[-, %xmm6]  v2 = x86_psrl v0, v1      ; bin: 66 0f d3 f3
+            return v2
+}
+
+function %sshr_i16x8(i16x8, i64x2) -> i16x8 {
+ebb0(v0: i16x8 [%xmm2], v1: i64x2 [%xmm1]):
+[-, %xmm2]  v2 = x86_psra v0, v1     ; bin: 66 0f e1 d1
+            return v2
+}
+
+function %sshr_i32x4(i32x4, i64x2) -> i32x4 {
+ebb0(v0: i32x4 [%xmm4], v1: i64x2 [%xmm0]):
+[-, %xmm4]  v2 = x86_psra v0, v1      ; bin: 66 0f e2 e0
+            return v2
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif
index 5c2893950d..9c728eb208 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif
@@ -11,3 +11,23 @@ ebb0:
     ; nextln: v2 = x86_psll v1, v3
     return v2
 }
+
+function %ushr_i64x2() -> i64x2 {
+ebb0:
+    v0 = iconst.i32 1
+    v1 = vconst.i64x2 [1 2]
+    v2 = ushr v1, v0
+    ; check: v3 = bitcast.i64x2 v0
+    ; nextln: v2 = x86_psrl v1, v3
+    return v2
+}
+
+function %sshr_i16x8() -> i16x8 {
+ebb0:
+    v0 = iconst.i32 1
+    v1 = vconst.i16x8 [1 2 4 8 16 32 64 128]
+    v2 = sshr v1, v0
+    ; check: v3 = bitcast.i64x2 v0
+    ; nextln: v2 = x86_psra v1, v3
+    return v2
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif b/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
index 224b3d5470..07c50bee0a 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
@@ -37,3 +37,71 @@ ebb0:
     return v7
 }
 ; run
+
+function %ushr_i64x2() -> b1 {
+ebb0:
+    v0 = iconst.i32 1
+    v1 = vconst.i64x2 [1 2]
+    v2 = ushr v1, v0
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 0
+
+    v5 = extractlane v2, 1
+    v6 = icmp_imm eq v5, 1
+
+    v7 = band v4, v6
+    return v7
+}
+; run
+
+function %ushr_too_large_i32x4() -> b1 {
+ebb0:
+    v0 = iconst.i32 33 ; note that this will shift off the end of each lane
+    v1 = vconst.i32x4 [1 2 4 8]
+    v2 = ushr v1, v0
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 0
+
+    v5 = extractlane v2, 3
+    v6 = icmp_imm eq v5, 0
+
+    v7 = band v4, v6
+    return v7
+}
+; run
+
+function %sshr_i16x8() -> b1 {
+ebb0:
+    v0 = iconst.i32 1
+    v1 = vconst.i16x8 [-1 2 4 8 -16 32 64 128]
+    v2 = sshr v1, v0
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 0xffff ; because of the shifted-in sign-bit, this remains 0xffff == -1
+
+    v5 = extractlane v2, 4
+    v6 = icmp_imm eq v5, 0xfff8 ; -16 has been shifted to -8 == 0xfff8
+
+    v7 = band v4, v6
+    return v7
+}
+; run
+
+function %sshr_too_large_i32x4() -> b1 {
+ebb0:
+    v0 = iconst.i32 33 ; note that this will shift off the end of each lane
+    v1 = vconst.i32x4 [1 2 4 -8]
+    v2 = sshr v1, v0
+
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 0
+
+    v5 = extractlane v2, 3
+    v6 = icmp_imm eq v5, 0xffff_ffff ; shifting in the sign-bit repeatedly fills the result with 1s
+
+    v7 = band v4, v6
+    return v7
+}
+; run

From 19a980363e72fbf5d08c6f783b0ee55eb95fd3fb Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 7 Oct 2019 10:48:10 -0700
Subject: [PATCH 2851/3084] Translate WASM shr to CLIF sshr and ushr

As with shift left, the spec requires that the shift count is computed modulo the lane width (see https://github.com/WebAssembly/simd/blob/master/proposals/simd/SIMD.md#left-shift-by-scalar).
---
 cranelift/wasm/src/code_translator.rs | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index e2514909ec..8039635db4 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1120,6 +1120,24 @@ pub fn translate_operator(
             let b_mod_bitwidth = builder.ins().band_imm(b, bitwidth - 1);
             state.push1(builder.ins().ishl(bitcast_a, b_mod_bitwidth))
         }
+        Operator::I16x8ShrU | Operator::I32x4ShrU | Operator::I64x2ShrU => {
+            let (a, b) = state.pop2();
+            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
+            let bitwidth = i64::from(builder.func.dfg.value_type(a).bits());
+            // The spec expects to shift with `b mod lanewidth`; so, e.g., for 16 bit lane-width
+            // we do `b AND 15`; this means fewer instructions than `iconst + urem`.
+            let b_mod_bitwidth = builder.ins().band_imm(b, bitwidth - 1);
+            state.push1(builder.ins().ushr(bitcast_a, b_mod_bitwidth))
+        }
+        Operator::I16x8ShrS | Operator::I32x4ShrS => {
+            let (a, b) = state.pop2();
+            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
+            let bitwidth = i64::from(builder.func.dfg.value_type(a).bits());
+            // The spec expects to shift with `b mod lanewidth`; so, e.g., for 16 bit lane-width
+            // we do `b AND 15`; this means fewer instructions than `iconst + urem`.
+            let b_mod_bitwidth = builder.ins().band_imm(b, bitwidth - 1);
+            state.push1(builder.ins().sshr(bitcast_a, b_mod_bitwidth))
+        }
         Operator::I8x16Eq
         | Operator::I8x16Ne
         | Operator::I8x16LtS
@@ -1171,16 +1189,11 @@ pub fn translate_operator(
         | Operator::I8x16Mul
         | Operator::I16x8AnyTrue
         | Operator::I16x8AllTrue
-        | Operator::I16x8ShrS
-        | Operator::I16x8ShrU
         | Operator::I32x4AnyTrue
         | Operator::I32x4AllTrue
-        | Operator::I32x4ShrS
-        | Operator::I32x4ShrU
         | Operator::I64x2AnyTrue
         | Operator::I64x2AllTrue
         | Operator::I64x2ShrS
-        | Operator::I64x2ShrU
         | Operator::F32x4Abs
         | Operator::F32x4Neg
         | Operator::F32x4Sqrt

From e15b720d56b703a839fa01d35854d1d06fbc575e Mon Sep 17 00:00:00 2001
From: "dependabot-preview[bot]"
 <27856297+dependabot-preview[bot]@users.noreply.github.com>
Date: Wed, 16 Oct 2019 10:07:07 +0000
Subject: [PATCH 2852/3084] Update indicatif requirement from 0.11.0 to 0.12.0

Updates the requirements on [indicatif](https://github.com/mitsuhiko/indicatif) to permit the latest version.
- [Release notes](https://github.com/mitsuhiko/indicatif/releases)
- [Commits](https://github.com/mitsuhiko/indicatif/compare/0.11.0...0.12.0)

Signed-off-by: dependabot-preview[bot] 
---
 cranelift/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index d2404d2ad2..8ccacc3e8a 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -42,7 +42,7 @@ wabt = { version = "0.9.1", optional = true }
 target-lexicon = "0.8.1"
 pretty_env_logger = "0.3.0"
 file-per-thread-logger = "0.1.2"
-indicatif = "0.11.0"
+indicatif = "0.12.0"
 walkdir = "2.2"
 
 [features]

From 7c31ce40c4723dae4816f45fce54878eb7c6fbce Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Wed, 16 Oct 2019 19:35:35 +0200
Subject: [PATCH 2853/3084] i128-isplit-forward-jump.clif: BB conditional
 branches can only be followed by a jump statement.

---
 .../filetests/filetests/isa/x86/i128-isplit-forward-jump.clif  | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/cranelift/filetests/filetests/isa/x86/i128-isplit-forward-jump.clif b/cranelift/filetests/filetests/isa/x86/i128-isplit-forward-jump.clif
index 863f39612f..b5144203ac 100644
--- a/cranelift/filetests/filetests/isa/x86/i128-isplit-forward-jump.clif
+++ b/cranelift/filetests/filetests/isa/x86/i128-isplit-forward-jump.clif
@@ -18,5 +18,8 @@ ebb5:
     v27 = bxor.i128 v2, v2
     v32 = iconst.i32 0
     brz v32, ebb2
+    jump ebb6
+
+ebb6:
     trap user0
 }

From 69d2f40c106b0e19982b36dcbf36f5b78a514248 Mon Sep 17 00:00:00 2001
From: "Nicolas B. Pierron" 
Date: Mon, 9 Sep 2019 14:37:28 +0200
Subject: [PATCH 2854/3084] Fix #796: Enable basic-blocks by default.

---
 cranelift/Cargo.toml          | 2 +-
 cranelift/codegen/Cargo.toml  | 2 +-
 cranelift/frontend/Cargo.toml | 2 +-
 cranelift/wasm/Cargo.toml     | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 8ccacc3e8a..667104bcd4 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -46,7 +46,7 @@ indicatif = "0.12.0"
 walkdir = "2.2"
 
 [features]
-default = ["disas", "wasm", "cranelift-codegen/all-arch"]
+default = ["disas", "wasm", "cranelift-codegen/all-arch", "basic-blocks"]
 disas = ["capstone"]
 wasm = ["wabt", "cranelift-wasm"]
 basic-blocks = ["cranelift-codegen/basic-blocks", "cranelift-frontend/basic-blocks",
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index ccda65d612..fc7a9023a8 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -32,7 +32,7 @@ smallvec = { version = "0.6.10" }
 cranelift-codegen-meta = { path = "meta", version = "0.46.1" }
 
 [features]
-default = ["std"]
+default = ["std", "basic-blocks"]
 
 # The "std" feature enables use of libstd. The "core" feature enables use
 # of some minimal std-like replacement libraries. At least one of these two
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 950e3d2623..02a61441af 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -18,7 +18,7 @@ hashbrown = { version = "0.6", optional = true }
 smallvec = { version = "0.6.10" }
 
 [features]
-default = ["std"]
+default = ["std", "basic-blocks"]
 std = ["cranelift-codegen/std"]
 core = ["hashbrown", "cranelift-codegen/core"]
 
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index df97b7260a..80feba60c6 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -26,7 +26,7 @@ wabt = "0.9.1"
 target-lexicon = "0.8.1"
 
 [features]
-default = ["std"]
+default = ["std", "basic-blocks"]
 std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std"]
 core = ["hashbrown", "cranelift-codegen/core", "cranelift-frontend/core", "wasmparser/core"]
 enable-serde = ["serde"]

From 05cc8823c24f14a028ec71a92433b1ff9666d596 Mon Sep 17 00:00:00 2001
From: "dependabot-preview[bot]"
 <27856297+dependabot-preview[bot]@users.noreply.github.com>
Date: Thu, 17 Oct 2019 14:56:17 +0000
Subject: [PATCH 2855/3084] Update raw-cpuid requirement from 6.0.0 to 7.0.3

Updates the requirements on [raw-cpuid](https://github.com/gz/rust-cpuid) to permit the latest version.
- [Release notes](https://github.com/gz/rust-cpuid/releases)
- [Commits](https://github.com/gz/rust-cpuid/compare/6.0...7.0.3)

Signed-off-by: dependabot-preview[bot] 
---
 cranelift/native/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 5af8f3a7a3..10399d046f 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -14,7 +14,7 @@ cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default
 target-lexicon = "0.8.1"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
-raw-cpuid = "6.0.0"
+raw-cpuid = "7.0.3"
 
 [features]
 default = ["std"]

From beca77c2f81baf24f974a658f56b1d3532d4d67c Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 8 Oct 2019 18:03:58 +0200
Subject: [PATCH 2856/3084] Regalloc: rename "constraint" to "rc" and "op" to
 "constraint";

---
 cranelift/codegen/src/regalloc/coloring.rs | 80 ++++++++++++----------
 cranelift/codegen/src/regalloc/pressure.rs | 16 ++---
 cranelift/codegen/src/regalloc/solver.rs   | 73 +++++++++-----------
 3 files changed, 84 insertions(+), 85 deletions(-)

diff --git a/cranelift/codegen/src/regalloc/coloring.rs b/cranelift/codegen/src/regalloc/coloring.rs
index f7fb50129f..c6c48bf78c 100644
--- a/cranelift/codegen/src/regalloc/coloring.rs
+++ b/cranelift/codegen/src/regalloc/coloring.rs
@@ -588,8 +588,8 @@ impl<'a> Context<'a> {
         // Copy register assignments from tied inputs to tied outputs.
         if let Some(constraints) = constraints {
             if constraints.tied_ops {
-                for (op, lv) in constraints.outs.iter().zip(defs) {
-                    if let ConstraintKind::Tied(num) = op.kind {
+                for (constraint, lv) in constraints.outs.iter().zip(defs) {
+                    if let ConstraintKind::Tied(num) = constraint.kind {
                         let arg = self.cur.func.dfg.inst_args(inst)[num as usize];
                         let reg = self.divert.reg(arg, &self.cur.func.locations);
                         self.cur.func.locations[lv.value] = ValueLoc::Reg(reg);
@@ -650,46 +650,46 @@ impl<'a> Context<'a> {
 
     /// Program the input-side constraints for `inst` into the constraint solver.
     fn program_input_constraints(&mut self, inst: Inst, constraints: &[OperandConstraint]) {
-        for (op, &value) in constraints
+        for (constraint, &arg_val) in constraints
             .iter()
             .zip(self.cur.func.dfg.inst_args(inst))
-            .filter(|&(op, _)| op.kind != ConstraintKind::Stack)
+            .filter(|&(constraint, _)| constraint.kind != ConstraintKind::Stack)
         {
             // Reload pass is supposed to ensure that all arguments to register operands are
             // already in a register.
-            let cur_reg = self.divert.reg(value, &self.cur.func.locations);
-            match op.kind {
+            let cur_reg = self.divert.reg(arg_val, &self.cur.func.locations);
+            match constraint.kind {
                 ConstraintKind::FixedReg(regunit) => {
                     // Add the fixed constraint even if `cur_reg == regunit`.
                     // It is possible that we will want to convert the value to a variable later,
                     // and this identity assignment prevents that from happening.
                     self.solver
-                        .reassign_in(value, op.regclass, cur_reg, regunit);
+                        .reassign_in(arg_val, constraint.regclass, cur_reg, regunit);
                 }
                 ConstraintKind::FixedTied(regunit) => {
                     // The pinned register may not be part of a fixed tied requirement. If this
                     // becomes the case, then it must be changed to a different register.
                     debug_assert!(
-                        !self.is_pinned_reg(op.regclass, regunit),
+                        !self.is_pinned_reg(constraint.regclass, regunit),
                         "see comment above"
                     );
                     // See comment right above.
                     self.solver
-                        .reassign_in(value, op.regclass, cur_reg, regunit);
+                        .reassign_in(arg_val, constraint.regclass, cur_reg, regunit);
                 }
                 ConstraintKind::Tied(_) => {
-                    if self.is_pinned_reg(op.regclass, cur_reg) {
+                    if self.is_pinned_reg(constraint.regclass, cur_reg) {
                         // Divert the pinned register; it shouldn't be reused for a tied input.
-                        if self.solver.can_add_var(op.regclass, cur_reg) {
-                            self.solver.add_var(value, op.regclass, cur_reg);
+                        if self.solver.can_add_var(constraint.regclass, cur_reg) {
+                            self.solver.add_var(arg_val, constraint.regclass, cur_reg);
                         }
-                    } else if !op.regclass.contains(cur_reg) {
-                        self.solver.add_var(value, op.regclass, cur_reg);
+                    } else if !constraint.regclass.contains(cur_reg) {
+                        self.solver.add_var(arg_val, constraint.regclass, cur_reg);
                     }
                 }
                 ConstraintKind::Reg => {
-                    if !op.regclass.contains(cur_reg) {
-                        self.solver.add_var(value, op.regclass, cur_reg);
+                    if !constraint.regclass.contains(cur_reg) {
+                        self.solver.add_var(arg_val, constraint.regclass, cur_reg);
                     }
                 }
                 ConstraintKind::Stack => unreachable!(),
@@ -714,22 +714,25 @@ impl<'a> Context<'a> {
             .expect("Current instruction not encoded")
             .ins;
 
-        for (op, &value) in constraints.iter().zip(self.cur.func.dfg.inst_args(inst)) {
-            match op.kind {
+        for (constraint, &arg_val) in constraints.iter().zip(self.cur.func.dfg.inst_args(inst)) {
+            match constraint.kind {
                 ConstraintKind::Reg | ConstraintKind::Tied(_) => {
-                    let cur_reg = self.divert.reg(value, &self.cur.func.locations);
+                    let cur_reg = self.divert.reg(arg_val, &self.cur.func.locations);
 
                     // This is the opposite condition of `program_input_constraints()`. The pinned
                     // register mustn't be added back as a variable.
-                    if op.regclass.contains(cur_reg) && !self.is_pinned_reg(op.regclass, cur_reg) {
+                    if constraint.regclass.contains(cur_reg)
+                        && !self.is_pinned_reg(constraint.regclass, cur_reg)
+                    {
                         // This code runs after calling `solver.inputs_done()` so we must identify
-                        // the new variable as killed or live-through. Always special-case the
-                        // pinned register as a through variable.
+                        // the new variable as killed or live-through.
                         let layout = &self.cur.func.layout;
-                        if self.liveness[value].killed_at(inst, layout.pp_ebb(inst), layout) {
-                            self.solver.add_killed_var(value, op.regclass, cur_reg);
+                        if self.liveness[arg_val].killed_at(inst, layout.pp_ebb(inst), layout) {
+                            self.solver
+                                .add_killed_var(arg_val, constraint.regclass, cur_reg);
                         } else {
-                            self.solver.add_through_var(value, op.regclass, cur_reg);
+                            self.solver
+                                .add_through_var(arg_val, constraint.regclass, cur_reg);
                         }
                     }
                 }
@@ -862,6 +865,10 @@ impl<'a> Context<'a> {
                 let toprc = self.reginfo.toprc(rci);
                 let reg = self.divert.reg(lv.value, &self.cur.func.locations);
                 if self.solver.is_fixed_input_conflict(toprc, reg) {
+                    debug!(
+                        "adding var to divert fixed input conflict for {}",
+                        toprc.info.display_regunit(reg)
+                    );
                     self.solver.add_var(lv.value, toprc, reg);
                 }
             }
@@ -879,15 +886,15 @@ impl<'a> Context<'a> {
         replace_global_defines: &mut bool,
         global_regs: &RegisterSet,
     ) {
-        for (op, lv) in constraints.iter().zip(defs) {
-            match op.kind {
+        for (constraint, lv) in constraints.iter().zip(defs) {
+            match constraint.kind {
                 ConstraintKind::FixedReg(reg) | ConstraintKind::FixedTied(reg) => {
-                    self.add_fixed_output(lv.value, op.regclass, reg, throughs);
-                    if !lv.is_local && !global_regs.is_avail(op.regclass, reg) {
+                    self.add_fixed_output(lv.value, constraint.regclass, reg, throughs);
+                    if !lv.is_local && !global_regs.is_avail(constraint.regclass, reg) {
                         debug!(
                             "Fixed output {} in {}:{} is not available in global regs",
                             lv.value,
-                            op.regclass,
+                            constraint.regclass,
                             self.reginfo.display_regunit(reg)
                         );
                         *replace_global_defines = true;
@@ -976,13 +983,14 @@ impl<'a> Context<'a> {
         replace_global_defines: &mut bool,
         global_regs: &RegisterSet,
     ) {
-        for (op, lv) in constraints.iter().zip(defs) {
-            match op.kind {
+        for (constraint, lv) in constraints.iter().zip(defs) {
+            match constraint.kind {
                 ConstraintKind::FixedReg(_)
                 | ConstraintKind::FixedTied(_)
                 | ConstraintKind::Stack => continue,
                 ConstraintKind::Reg => {
-                    self.solver.add_def(lv.value, op.regclass, !lv.is_local);
+                    self.solver
+                        .add_def(lv.value, constraint.regclass, !lv.is_local);
                 }
                 ConstraintKind::Tied(num) => {
                     // Find the input operand we're tied to.
@@ -992,16 +1000,16 @@ impl<'a> Context<'a> {
 
                     if let Some(reg) =
                         self.solver
-                            .add_tied_input(arg, op.regclass, reg, !lv.is_local)
+                            .add_tied_input(arg, constraint.regclass, reg, !lv.is_local)
                     {
                         // The value we're tied to has been assigned to a fixed register.
                         // We need to make sure that fixed output register is compatible with the
                         // global register set.
-                        if !lv.is_local && !global_regs.is_avail(op.regclass, reg) {
+                        if !lv.is_local && !global_regs.is_avail(constraint.regclass, reg) {
                             debug!(
                                 "Tied output {} in {}:{} is not available in global regs",
                                 lv.value,
-                                op.regclass,
+                                constraint.regclass,
                                 self.reginfo.display_regunit(reg)
                             );
                             *replace_global_defines = true;
diff --git a/cranelift/codegen/src/regalloc/pressure.rs b/cranelift/codegen/src/regalloc/pressure.rs
index 73c7fbc1a3..3783a78e28 100644
--- a/cranelift/codegen/src/regalloc/pressure.rs
+++ b/cranelift/codegen/src/regalloc/pressure.rs
@@ -48,20 +48,20 @@ use cranelift_codegen_shared::constants::MAX_TRACKED_TOP_RCS;
 /// Everything but the counts is static information computed from the constructor arguments.
 #[derive(Default)]
 struct TopRC {
-    // Number of registers currently used from this register class.
+    /// Number of registers currently used from this register class.
     base_count: u32,
     transient_count: u32,
 
-    // Max number of registers that can be allocated.
+    /// Max number of registers that can be allocated.
     limit: u32,
 
-    // Register units per register.
+    /// Register units per register.
     width: u8,
 
-    // The first aliasing top-level RC.
+    /// The first aliasing top-level RC.
     first_toprc: u8,
 
-    // The number of aliasing top-level RCs.
+    /// The number of aliasing top-level RCs.
     num_toprcs: u8,
 }
 
@@ -72,11 +72,11 @@ impl TopRC {
 }
 
 pub struct Pressure {
-    // Bit mask of top-level register classes that are aliased by other top-level register classes.
-    // Unaliased register classes can use a simpler interference algorithm.
+    /// Bit mask of top-level register classes that are aliased by other top-level register classes.
+    /// Unaliased register classes can use a simpler interference algorithm.
     aliased: RegClassMask,
 
-    // Current register counts per top-level register class.
+    /// Current register counts per top-level register class.
     toprc: [TopRC; MAX_TRACKED_TOP_RCS],
 }
 
diff --git a/cranelift/codegen/src/regalloc/solver.rs b/cranelift/codegen/src/regalloc/solver.rs
index 5b63b2825a..5a3e1df908 100644
--- a/cranelift/codegen/src/regalloc/solver.rs
+++ b/cranelift/codegen/src/regalloc/solver.rs
@@ -265,6 +265,7 @@ pub enum Move {
         from: RegUnit,
         to: RegUnit,
     },
+    #[allow(dead_code)] // rustc doesn't see it isn't dead.
     Spill {
         value: Value,
         rc: RegClass,
@@ -283,7 +284,7 @@ impl Move {
     /// Create a register move from an assignment, but not for identity assignments.
     fn with_assignment(a: &Assignment) -> Option {
         if a.from != a.to {
-            Some(Move::Reg {
+            Some(Self::Reg {
                 value: a.value,
                 from: a.from,
                 to: a.to,
@@ -298,16 +299,16 @@ impl Move {
     #[cfg_attr(feature = "cargo-clippy", allow(clippy::wrong_self_convention))]
     fn from_reg(&self) -> Option<(RegClass, RegUnit)> {
         match *self {
-            Move::Reg { rc, from, .. } | Move::Spill { rc, from, .. } => Some((rc, from)),
-            Move::Fill { .. } => None,
+            Self::Reg { rc, from, .. } | Self::Spill { rc, from, .. } => Some((rc, from)),
+            Self::Fill { .. } => None,
         }
     }
 
     /// Get the "to" register and register class, if possible.
     fn to_reg(&self) -> Option<(RegClass, RegUnit)> {
         match *self {
-            Move::Reg { rc, to, .. } | Move::Fill { rc, to, .. } => Some((rc, to)),
-            Move::Spill { .. } => None,
+            Self::Reg { rc, to, .. } | Self::Fill { rc, to, .. } => Some((rc, to)),
+            Self::Spill { .. } => None,
         }
     }
 
@@ -315,8 +316,8 @@ impl Move {
     fn replace_to_reg(&mut self, new: RegUnit) -> RegUnit {
         mem::replace(
             match *self {
-                Move::Reg { ref mut to, .. } | Move::Fill { ref mut to, .. } => to,
-                Move::Spill { .. } => panic!("No to register in a spill {}", self),
+                Self::Reg { ref mut to, .. } | Self::Fill { ref mut to, .. } => to,
+                Self::Spill { .. } => panic!("No to register in a spill {}", self),
             },
             new,
         )
@@ -325,13 +326,13 @@ impl Move {
     /// Convert this `Reg` move to a spill to `slot` and return the old "to" register.
     fn change_to_spill(&mut self, slot: usize) -> RegUnit {
         match self.clone() {
-            Move::Reg {
+            Self::Reg {
                 value,
                 rc,
                 from,
                 to,
             } => {
-                *self = Move::Spill {
+                *self = Self::Spill {
                     value,
                     rc,
                     from,
@@ -346,14 +347,14 @@ impl Move {
     /// Get the value being moved.
     fn value(&self) -> Value {
         match *self {
-            Move::Reg { value, .. } | Move::Fill { value, .. } | Move::Spill { value, .. } => value,
+            Self::Reg { value, .. } | Self::Fill { value, .. } | Self::Spill { value, .. } => value,
         }
     }
 
     /// Get the associated register class.
     fn rc(&self) -> RegClass {
         match *self {
-            Move::Reg { rc, .. } | Move::Fill { rc, .. } | Move::Spill { rc, .. } => rc,
+            Self::Reg { rc, .. } | Self::Fill { rc, .. } | Self::Spill { rc, .. } => rc,
         }
     }
 }
@@ -361,7 +362,7 @@ impl Move {
 impl fmt::Display for Move {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
-            Move::Reg {
+            Self::Reg {
                 value,
                 from,
                 to,
@@ -374,7 +375,7 @@ impl fmt::Display for Move {
                 rc.info.display_regunit(from),
                 rc.info.display_regunit(to)
             ),
-            Move::Spill {
+            Self::Spill {
                 value,
                 from,
                 to_slot,
@@ -387,7 +388,7 @@ impl fmt::Display for Move {
                 rc.info.display_regunit(from),
                 to_slot
             ),
-            Move::Fill {
+            Self::Fill {
                 value,
                 from_slot,
                 to,
@@ -593,67 +594,57 @@ impl Solver {
     /// instruction.
     ///
     /// This function should be called after `inputs_done()` only. Use `add_var()` before.
-    pub fn add_killed_var(&mut self, value: Value, constraint: RegClass, from: RegUnit) {
+    pub fn add_killed_var(&mut self, value: Value, rc: RegClass, from: RegUnit) {
         debug!(
             "add_killed_var({}:{}, from={})",
             value,
-            constraint,
-            constraint.info.display_regunit(from)
+            rc,
+            rc.info.display_regunit(from)
         );
         debug_assert!(self.inputs_done);
-        self.add_live_var(value, constraint, from, false);
+        self.add_live_var(value, rc, from, false);
     }
 
     /// Add an extra input-side variable representing a value that is live through the current
     /// instruction.
     ///
     /// This function should be called after `inputs_done()` only. Use `add_var()` before.
-    pub fn add_through_var(&mut self, value: Value, constraint: RegClass, from: RegUnit) {
+    pub fn add_through_var(&mut self, value: Value, rc: RegClass, from: RegUnit) {
         debug!(
             "add_through_var({}:{}, from={})",
             value,
-            constraint,
-            constraint.info.display_regunit(from)
+            rc,
+            rc.info.display_regunit(from)
         );
         debug_assert!(self.inputs_done);
-        self.add_live_var(value, constraint, from, true);
+        self.add_live_var(value, rc, from, true);
     }
 
     /// Shared code for `add_var`, `add_killed_var`, and `add_through_var`.
     ///
     /// Add a variable that is live before the instruction, and possibly live through. Merge
     /// constraints if the value has already been added as a variable or fixed assignment.
-    fn add_live_var(
-        &mut self,
-        value: Value,
-        constraint: RegClass,
-        from: RegUnit,
-        live_through: bool,
-    ) {
+    fn add_live_var(&mut self, value: Value, rc: RegClass, from: RegUnit, live_through: bool) {
         // Check for existing entries for this value.
-        if self.regs_in.is_avail(constraint, from) {
+        if self.regs_in.is_avail(rc, from) {
             // There could be an existing variable entry.
             if let Some(v) = self.vars.iter_mut().find(|v| v.value == value) {
                 // We have an existing variable entry for `value`. Combine the constraints.
-                if let Some(rc) = v.constraint.intersect(constraint) {
+                if let Some(rc) = v.constraint.intersect(rc) {
                     debug!("-> combining constraint with {} yields {}", v, rc);
                     v.constraint = rc;
                     return;
                 } else {
                     // The spiller should have made sure the same value is not used with disjoint
                     // constraints.
-                    panic!("Incompatible constraints: {} + {}", constraint, v)
+                    panic!("Incompatible constraints: {} + {}", rc, v)
                 }
             }
 
             // No variable, then it must be a fixed reassignment.
             if let Some(a) = self.assignments.get(value) {
                 debug!("-> already fixed assignment {}", a);
-                debug_assert!(
-                    constraint.contains(a.to),
-                    "Incompatible constraints for {}",
-                    value
-                );
+                debug_assert!(rc.contains(a.to), "Incompatible constraints for {}", value);
                 return;
             }
 
@@ -661,12 +652,12 @@ impl Solver {
             panic!("Wrong from register for {}", value);
         }
 
-        let new_var = Variable::new_live(value, constraint, from, live_through);
+        let new_var = Variable::new_live(value, rc, from, live_through);
         debug!("-> new var: {}", new_var);
 
-        self.regs_in.free(constraint, from);
+        self.regs_in.free(rc, from);
         if self.inputs_done && live_through {
-            self.regs_out.free(constraint, from);
+            self.regs_out.free(rc, from);
         }
         self.vars.push(new_var);
     }
@@ -1007,7 +998,7 @@ impl Solver {
         self.moves
             .extend(self.assignments.values().filter_map(Move::with_assignment));
 
-        if !(self.moves.is_empty()) {
+        if !self.moves.is_empty() {
             debug!("collect_moves: {}", DisplayList(&self.moves));
         }
     }

From a3f55cdf1fc6ea6cd49f199e235260ee38232460 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 9 Oct 2019 13:51:07 +0200
Subject: [PATCH 2857/3084] Regalloc solver: check that a variable doesn't
 exist to test if it can be added (fixes #1123);

This situation could be triggered that can_add_var would return true
while a variable was already added for the given register.

For instance, when we have a reassignment (because of a fixed register
input requirement) and a fixed input conflict on the same fixed
register, this register will not be available in the regs_in set after
inputs_done (because of the fixed input conflict diversion) but will
have its own variable.
---
 cranelift/codegen/src/regalloc/solver.rs      |   3 +-
 .../regalloc/solver-fixedconflict-var-2.clif  | 100 ++++++++++
 .../regalloc/solver-fixedconflict-var-3.clif  | 137 ++++++++++++++
 .../regalloc/solver-fixedconflict-var.clif    | 173 ++++++++++++++++++
 4 files changed, 412 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-2.clif
 create mode 100644 cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif
 create mode 100644 cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif

diff --git a/cranelift/codegen/src/regalloc/solver.rs b/cranelift/codegen/src/regalloc/solver.rs
index 5a3e1df908..96ce702fdc 100644
--- a/cranelift/codegen/src/regalloc/solver.rs
+++ b/cranelift/codegen/src/regalloc/solver.rs
@@ -626,7 +626,7 @@ impl Solver {
     /// constraints if the value has already been added as a variable or fixed assignment.
     fn add_live_var(&mut self, value: Value, rc: RegClass, from: RegUnit, live_through: bool) {
         // Check for existing entries for this value.
-        if self.regs_in.is_avail(rc, from) {
+        if !self.can_add_var(rc, from) {
             // There could be an existing variable entry.
             if let Some(v) = self.vars.iter_mut().find(|v| v.value == value) {
                 // We have an existing variable entry for `value`. Combine the constraints.
@@ -969,6 +969,7 @@ impl Solver {
     /// Check if `value` can be added as a variable to help find a solution.
     pub fn can_add_var(&mut self, constraint: RegClass, from: RegUnit) -> bool {
         !self.regs_in.is_avail(constraint, from)
+            && !self.vars.iter().any(|var| var.from == Some(from))
     }
 }
 
diff --git a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-2.clif b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-2.clif
new file mode 100644
index 0000000000..be64db792d
--- /dev/null
+++ b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-2.clif
@@ -0,0 +1,100 @@
+test compile
+set opt_level=speed
+set enable_pinned_reg=true
+target x86_64 haswell
+
+function u0:0(i32, i32, i32, i64 vmctx) -> i64 uext system_v {
+ebb0(v0: i32, v1: i32, v2: i32, v3: i64):
+    v236 = iconst.i32 0x4de9_bd37
+    v424 = iconst.i32 0
+    jump ebb37(v424)
+
+ebb37(v65: i32):
+    v433 = iconst.i32 0
+    jump ebb40(v433)
+
+ebb40(v70: i32):
+    v75 = iconst.i32 0
+    v259 = iconst.i32 0
+    v78 -> v259
+    v449 = iconst.i32 0
+    v450, v451 = x86_sdivmodx v75, v449, v259
+    v79 -> v450
+    v269 = iconst.i32 0
+    v270 = ushr_imm v269, 31
+    v271 = iadd v269, v270
+    v98 -> v271
+    v100 = iconst.i32 -31
+    v272 = iconst.i32 0x4de9_bd37
+    v490, v273 = x86_smulx v100, v272
+    v493 = iconst.i32 0
+    jump ebb61(v493)
+
+ebb61(v103: i32):
+    v104 = iconst.i32 -23
+    v105 = iconst.i32 -23
+    v106 = popcnt v105
+    v500 = sshr_imm v104, 31
+    v501 = iconst.i32 0
+    jump ebb64(v501)
+
+ebb64(v107: i32):
+    v108 = iconst.i32 0
+    v109 = iconst.i32 0
+    v278 = iconst.i32 0
+    v507, v279 = x86_smulx v109, v278
+    v280 = isub v279, v109
+    v281 = sshr_imm v280, 11
+    v282 = iconst.i32 0
+    v283 = iadd v281, v282
+    v111 -> v283
+    v112 = rotr v108, v283
+    jump ebb65
+
+ebb65:
+    v509 = iconst.i32 0
+    v510, v511 = x86_sdivmodx v107, v509, v112
+    v113 -> v510
+    v114 = iconst.i32 0
+    v517 = iconst.i32 0
+    v518, v519 = x86_sdivmodx v103, v517, v114
+    v115 -> v518
+    v534 = iconst.i32 0
+    v122 -> v534
+    v541 = iconst.i32 0
+    v542, v543 = x86_sdivmodx v271, v541, v122
+    v123 -> v542
+    v289 = iconst.i32 0
+    v125 -> v289
+    v550 = iconst.i32 0
+    v551, v552 = x86_sdivmodx v79, v550, v289
+    v126 -> v551
+    v130 = iconst.i32 0
+    v558 = iconst.i32 0
+    v559, v560 = x86_sdivmodx v70, v558, v130
+    v131 -> v559
+    v305 = iconst.i32 0
+    v140 -> v305
+    v577 = iconst.i32 0
+    v578, v579 = x86_sdivmodx v65, v577, v305
+    v141 -> v578
+    v166 = iconst.i32 0
+    v167 = iconst.i32 -31
+    v318 = iconst.i32 0x4de9_bd37
+    v650, v319 = x86_smulx v167, v318
+    v320 = isub v319, v167
+    v321 = sshr_imm v320, 4
+    v322 = iconst.i32 0
+    v323 = iadd v321, v322
+    v169 -> v323
+    v652 = iconst.i32 0
+    v653, v654 = x86_sdivmodx v166, v652, v323
+    v170 -> v653
+    v171 = iconst.i32 -23
+    v172 = iconst.i32 -23
+    v173 = popcnt v172
+    v174 = popcnt v173
+    v660 = sshr_imm v171, 31
+    v661, v662 = x86_sdivmodx v171, v660, v174
+    trap user0
+}
diff --git a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif
new file mode 100644
index 0000000000..13a613d1bf
--- /dev/null
+++ b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif
@@ -0,0 +1,137 @@
+test compile
+set opt_level=speed
+set enable_pinned_reg=true
+target x86_64 haswell
+
+function u0:0(i32, i32, i32, i64 vmctx) -> i64 uext system_v {
+ebb0(v0: i32, v1: i32, v2: i32, v3: i64):
+    v5 = iconst.i32 -8
+    v114 = iconst.i32 0
+    v16 = iconst.i32 -8
+    v17 = popcnt v16
+    v192 = ifcmp_imm v17, -1
+    brif eq v192, ebb12
+    trap user0
+
+ebb12:
+    v122 = iconst.i32 0
+    v123 = ushr_imm v122, 31
+    v124 = iadd v122, v123
+    v20 -> v124
+    v25 = iconst.i32 -19
+    v204 = iconst.i32 0
+    v31 -> v204
+    v210 = ifcmp_imm v31, -1
+    brif eq v210, ebb18
+    trap user0
+
+ebb18:
+    v215 = iconst.i32 0
+    jump ebb19(v215)
+
+ebb19(v32: i32):
+    v35 = iconst.i32 0
+    v218 = ifcmp_imm v35, -1
+    brif eq v218, ebb21
+    trap user0
+
+ebb21:
+    v223 = iconst.i32 0
+    jump ebb22(v223)
+
+ebb22(v36: i32):
+    v136 = iconst.i32 0
+    v40 -> v136
+    v227 = ifcmp_imm v136, -1
+    brif eq v227, ebb24
+    trap user0
+
+ebb24:
+    v232 = iconst.i32 0
+    jump ebb25(v232)
+
+ebb25(v41: i32):
+    v142 = iconst.i32 0
+    v45 -> v142
+    v236 = ifcmp_imm v142, -1
+    brif eq v236, ebb27
+    trap user0
+
+ebb27:
+    v241 = iconst.i32 0
+    jump ebb28(v241)
+
+ebb28(v46: i32):
+    v49 = iconst.i32 0
+    v244 = ifcmp_imm v49, -1
+    brif eq v244, ebb30
+    trap user0
+
+ebb30:
+    v254 = iconst.i32 0
+    v53 -> v254
+    v54 = iconst.i32 -23
+    v55 = popcnt v54
+    v143 = iconst.i32 0x4de9_bd37
+    v260, v144 = x86_smulx v55, v143
+    v145 = iconst.i32 0
+    v146 = sshr_imm v145, 4
+    v147 = iconst.i32 0
+    v148 = iadd v146, v147
+    v57 -> v148
+    v58 = ishl v53, v148
+    jump ebb35
+
+ebb35:
+    v262 = iconst.i32 0
+    v263, v264 = x86_sdivmodx v46, v262, v58
+    v59 -> v263
+    v270 = iconst.i32 0
+    v271, v272 = x86_sdivmodx v41, v270, v59
+    v60 -> v271
+    v61 = f32const 0.0
+    v280 = iconst.i32 0
+    v281 = fcmp uno v61, v61
+    brnz v281, ebb41(v280)
+    trap user0
+
+ebb41(v62: i32):
+    v157 = iconst.i32 0
+    v158 = sshr_imm v157, 4
+    v159 = iconst.i32 0
+    v160 = iadd v158, v159
+    v75 -> v160
+    v308 = ifcmp_imm v160, -1
+    brif eq v308, ebb52
+    trap user0
+
+ebb52:
+    v87 = iconst.i32 -23
+    v88 = iconst.i32 -23
+    v89 = popcnt v88
+    v161 = iconst.i32 0x4de9_bd37
+    v324, v162 = x86_smulx v89, v161
+    v163 = isub v162, v89
+    v164 = sshr_imm v163, 4
+    v165 = iconst.i32 0
+    v166 = iadd v164, v165
+    v91 -> v166
+    v326 = iconst.i32 0
+    v327, v328 = x86_sdivmodx v87, v326, v166
+    v92 -> v327
+    v351 = iconst.i32 0
+    v99 -> v351
+    v358 = iconst.i32 0
+    v359, v360 = x86_sdivmodx v36, v358, v99
+    v100 -> v359
+    v102 = iconst.i32 0
+    v103 = rotr.i32 v32, v102
+    v366 = iconst.i32 0
+    v367, v368 = x86_sdivmodx v25, v366, v103
+    v104 -> v367
+    v383 = iconst.i32 0
+    v107 -> v383
+    v390 = iconst.i32 0
+    v391, v392 = x86_sdivmodx v124, v390, v107
+    trap user0
+}
diff --git a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
new file mode 100644
index 0000000000..fc643ddbd4
--- /dev/null
+++ b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
@@ -0,0 +1,173 @@
+test compile
+set opt_level=speed
+set enable_pinned_reg=true
+target x86_64 haswell
+
+;; Test for the issue #1123; https://github.com/CraneStation/cranelift/issues/1123
+
+function u0:0(i32, i32, i32, i64 vmctx) -> i64 uext system_v {
+ebb0(v0: i32, v1: i32, v2: i32, v3: i64):
+    v351 = iconst.i32 0x4de9_bd37
+    v31 = iconst.i32 -23
+    v35 = iconst.i32 0
+    v36 = iconst.i32 -31
+    v357 = iconst.i32 0x4de9_bd37
+    v530, v358 = x86_smulx v36, v357
+    v359 = isub v358, v36
+    v360 = sshr_imm v359, 4
+    v361 = iconst.i32 0
+    v362 = iadd v360, v361
+    v38 -> v362
+    v532 = sshr_imm v35, 31
+    v533, v534 = x86_sdivmodx v35, v532, v362
+    v39 -> v533
+    v53 = iconst.i32 0
+    v547 = ifcmp_imm v53, -1
+    brif eq v547, ebb30
+    trap user0
+
+ebb30:
+    v75 = iconst.i32 0
+    v581 = ifcmp_imm v75, -1
+    brif eq v581, ebb42
+    trap user0
+
+ebb42:
+    v136 = iconst.i32 0
+    v691 = ifcmp_imm v136, -1
+    brif eq v691, ebb81
+    trap user0
+
+ebb81:
+    v158 = iconst.i32 0
+    v725 = ifcmp_imm v158, -1
+    brif eq v725, ebb93
+    trap user0
+
+ebb93:
+    v760 = iconst.i32 0
+    jump ebb106(v760)
+
+ebb106(v175: i32):
+    v179 = iconst.i32 0
+    v180 = icmp_imm eq v179, 0
+    v183 = iconst.i32 0
+    v766 = ifcmp_imm v183, -1
+    brif eq v766, ebb108
+    trap user0
+
+ebb108:
+    v771 = iconst.i32 0
+    jump ebb109(v771)
+
+ebb109(v184: i32):
+    v785 = iconst.i32 0
+    v193 -> v785
+    v791 = ifcmp_imm v193, -1
+    brif eq v791, ebb117
+    trap user0
+
+ebb117:
+    v796 = iconst.i32 0
+    jump ebb118(v796)
+
+ebb118(v194: i32):
+    v203 = iconst.i32 -63
+    v809 = iconst.i32 0
+    v207 -> v809
+    v815 = ifcmp_imm v207, -1
+    brif eq v815, ebb126
+    trap user0
+
+ebb126:
+    v209 = iconst.i32 0
+    v823 = ifcmp_imm v209, -1
+    brif eq v823, ebb129
+    trap user0
+
+ebb129:
+    v213 = iconst.i32 -23
+    v214 = iconst.i32 -19
+    v215 = icmp_imm eq v214, 0
+    v216 = bint.i32 v215
+    v217 = popcnt v216
+    v435 = iconst.i32 0x7df7_df7d
+    v831, v436 = x86_smulx v217, v435
+    v437 = isub v436, v217
+    v438 = sshr_imm v437, 5
+    v439 = ushr_imm v438, 31
+    v440 = iadd v438, v439
+    v219 -> v440
+    v220 = rotr v213, v440
+    v229 = iconst.i32 0
+    v841 = iconst.i32 0
+    v842, v843 = x86_sdivmodx v194, v841, v229
+    v230 -> v842
+    v849 = iconst.i32 0
+    v850, v851 = x86_sdivmodx v184, v849, v230
+    v231 -> v850
+    v232 = iconst.i32 0
+    v857 = iconst.i32 0
+    v858, v859 = x86_sdivmodx v175, v857, v232
+    v233 -> v858
+    v915 = iconst.i32 0
+    jump ebb163(v915)
+
+ebb163(v253: i32):
+    v255 = iconst.i32 0
+    v256 = iconst.i32 -23
+    v257 = iconst.i32 -19
+    v258 = icmp_imm eq v257, 0
+    v259 = bint.i32 v258
+    v260 = popcnt v259
+    v447 = iconst.i32 0x7df7_df7d
+    v921, v448 = x86_smulx v260, v447
+    v449 = isub v448, v260
+    v450 = sshr_imm v449, 5
+    v451 = ushr_imm v450, 31
+    v452 = iadd v450, v451
+    v262 -> v452
+    v263 = rotr v256, v452
+    v264 = popcnt v263
+    v265 = popcnt v264
+    v266 = popcnt v265
+    v267 = rotr v255, v266
+    v268 = popcnt v267
+    v923 = iconst.i32 0
+    v924, v925 = x86_sdivmodx v253, v923, v268
+    v269 -> v924
+    v276 = iconst.i32 0
+    v277 = iconst.i32 -63
+    v278 = popcnt v277
+    v947 = iconst.i32 0
+    v948, v949 = x86_sdivmodx v276, v947, v278
+    v279 -> v948
+    v309 = iconst.i32 0
+    v310 = iconst.i32 0
+    v311 = iconst.i32 0
+    v312 = icmp_imm eq v311, 0
+    v313 = bint.i32 v312
+    v314 = rotr v310, v313
+    v315 = iconst.i32 -31
+    v464 = iconst.i32 0
+    v1020, v465 = x86_smulx v315, v464
+    v466 = isub v465, v315
+    v467 = sshr_imm v466, 4
+    v468 = iconst.i32 0
+    v469 = iadd v467, v468
+    v317 -> v469
+    v1022 = iconst.i32 0
+    v1023, v1024 = x86_sdivmodx v314, v1022, v469
+    v318 -> v1023
+    v320 = iconst.i32 0
+    v321 = iconst.i32 -19
+    v322 = popcnt v321
+    v1030 = iconst.i32 0
+    v1031, v1032 = x86_sdivmodx v320, v1030, v322
+    v323 -> v1031
+    v1047 = iconst.i32 0
+    v325 -> v1047
+    v1054 = sshr_imm v309, 31
+    v1055, v1056 = x86_sdivmodx v309, v1054, v325
+    trap user0
+}

From 46b44ad82d9370ec92b9388b53bbbd1bcc2a24c8 Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Thu, 17 Oct 2019 11:30:38 -0600
Subject: [PATCH 2858/3084] Increase legibility of the SSABuilder (#1142)

---
 cranelift/frontend/src/ssa.rs | 160 +++++++++++++++++++++-------------
 1 file changed, 99 insertions(+), 61 deletions(-)

diff --git a/cranelift/frontend/src/ssa.rs b/cranelift/frontend/src/ssa.rs
index dcc729ebd5..7a1f440a1e 100644
--- a/cranelift/frontend/src/ssa.rs
+++ b/cranelift/frontend/src/ssa.rs
@@ -4,6 +4,8 @@
 //! Zwinkau A. (2013) Simple and Efficient Construction of Static Single Assignment Form.
 //! In: Jhala R., De Bosschere K. (eds) Compiler Construction. CC 2013.
 //! Lecture Notes in Computer Science, vol 7791. Springer, Berlin, Heidelberg
+//!
+//! https://link.springer.com/content/pdf/10.1007/978-3-642-37051-9_6.pdf
 
 use crate::Variable;
 use alloc::vec::Vec;
@@ -35,20 +37,24 @@ use smallvec::SmallVec;
 /// and it is said _sealed_ if all of its predecessors have been declared. Only filled predecessors
 /// can be declared.
 pub struct SSABuilder {
-    // Records for every variable and for every relevant block, the last definition of
-    // the variable in the block.
     // TODO: Consider a sparse representation rather than SecondaryMap-of-SecondaryMap.
+    /// Records for every variable and for every relevant block, the last definition of
+    /// the variable in the block.
     variables: SecondaryMap>>,
-    // Records the position of the basic blocks and the list of values used but not defined in the
-    // block.
+
+    /// Records the position of the basic blocks and the list of values used but not defined in the
+    /// block.
     blocks: PrimaryMap,
-    // Records the basic blocks at the beginning of the `Ebb`s.
+
+    /// Records the basic blocks at the beginning of the `Ebb`s.
     ebb_headers: SecondaryMap>,
 
-    // Call and result stacks for use in the `use_var`/`predecessors_lookup` state machine.
+    /// Call stack for use in the `use_var`/`predecessors_lookup` state machine.
     calls: Vec,
+    /// Result stack for use in the `use_var`/`predecessors_lookup` state machine.
     results: Vec,
-    // Side effects accumulated in the `use_var`/`predecessors_lookup` state machine.
+
+    /// Side effects accumulated in the `use_var`/`predecessors_lookup` state machine.
     side_effects: SideEffects,
 }
 
@@ -200,6 +206,7 @@ enum ZeroOneOrMore {
     More,
 }
 
+/// Cases used internally by `use_var_nonlocal()` for avoiding the borrow checker.
 #[derive(Debug)]
 enum UseVarCases {
     Unsealed(Value),
@@ -245,6 +252,7 @@ fn emit_zero(ty: Type, mut cur: FuncCursor) -> Value {
         panic!("unimplemented type: {:?}", ty)
     }
 }
+
 /// The following methods are the API of the SSA builder. Here is how it should be used when
 /// translating to Cranelift IR:
 ///
@@ -288,36 +296,45 @@ impl SSABuilder {
         ty: Type,
         block: Block,
     ) -> (Value, SideEffects) {
-        // First we lookup for the current definition of the variable in this block
+        // First, try Local Value Numbering (Algorithm 1 in the paper).
+        // If the variable already has a known Value in this block, use that.
         if let Some(var_defs) = self.variables.get(var) {
             if let Some(val) = var_defs[block].expand() {
                 return (val, SideEffects::new());
             }
         }
 
-        // Otherwise, we have to do a non-local lookup.
+        // Otherwise, use Global Value Numbering (Algorithm 2 in the paper).
+        // This resolves the Value with respect to its predecessors.
         debug_assert!(self.calls.is_empty());
         debug_assert!(self.results.is_empty());
         debug_assert!(self.side_effects.is_empty());
+
+        // Prepare the 'calls' and 'results' stacks for the state machine.
         self.use_var_nonlocal(func, var, ty, block);
-        (
-            self.run_state_machine(func, var, ty),
-            mem::replace(&mut self.side_effects, SideEffects::new()),
-        )
+
+        let value = self.run_state_machine(func, var, ty);
+        let side_effects = mem::replace(&mut self.side_effects, SideEffects::new());
+
+        (value, side_effects)
     }
 
-    /// Resolve a use of `var` in `block` in the case where there's no prior def
-    /// in `block`.
+    /// Resolve the minimal SSA Value of `var` in `block` by traversing predecessors.
+    ///
+    /// This function sets up state for `run_state_machine()` but does not execute it.
     fn use_var_nonlocal(&mut self, func: &mut Function, var: Variable, ty: Type, block: Block) {
+        // This function is split into two parts to appease the borrow checker.
+        // Part 1: With a mutable borrow of self, update the DataFlowGraph if necessary.
         let case = match self.blocks[block] {
             BlockData::EbbHeader(ref mut data) => {
                 // The block has multiple predecessors so we append an Ebb parameter that
                 // will serve as a value.
                 if data.sealed {
                     if data.predecessors.len() == 1 {
-                        // Only one predecessor, straightforward case
+                        // Optimize the common case of one predecessor: no param needed.
                         UseVarCases::SealedOnePredecessor(data.predecessors[0].block)
                     } else {
+                        // Break potential cycles by eagerly adding an operandless param.
                         let val = func.dfg.append_ebb_param(data.ebb, ty);
                         UseVarCases::SealedMultiplePredecessors(val, data.ebb)
                     }
@@ -329,22 +346,26 @@ impl SSABuilder {
             }
             BlockData::EbbBody { predecessor: pred } => UseVarCases::SealedOnePredecessor(pred),
         };
+
+        // Part 2: Prepare SSABuilder state for run_state_machine().
         match case {
-            // The block has a single predecessor, we look into it.
             UseVarCases::SealedOnePredecessor(pred) => {
+                // Get the Value directly from the single predecessor.
                 self.calls.push(Call::FinishSealedOnePredecessor(block));
                 self.calls.push(Call::UseVar(pred));
             }
-            // The block has multiple predecessors, we register the EBB parameter as the current
-            // definition for the variable.
             UseVarCases::Unsealed(val) => {
+                // Define the operandless param added above to prevent lookup cycles.
                 self.def_var(var, val, block);
+
+                // Nothing more can be known at this point.
                 self.results.push(val);
             }
             UseVarCases::SealedMultiplePredecessors(val, ebb) => {
-                // If multiple predecessor we look up a use_var in each of them:
-                // if they all yield the same value no need for an EBB parameter
+                // Define the operandless param added above to prevent lookup cycles.
                 self.def_var(var, val, block);
+
+                // Look up a use_var for each precessor.
                 self.begin_predecessors_lookup(val, ebb);
             }
         }
@@ -487,30 +508,47 @@ impl SSABuilder {
         }
     }
 
-    /// Look up in the predecessors of an Ebb the def for a value an decides whether or not
-    /// to keep the eeb arg, and act accordingly. Returns the chosen value and optionally a
-    /// list of Ebb that are the middle of newly created critical edges splits.
+    /// Given the local SSA Value of a Variable in an Ebb, perform a recursive lookup on
+    /// predecessors to determine if it is redundant with another Value earlier in the CFG.
+    ///
+    /// If such a Value exists and is redundant, the local Value is replaced by the
+    /// corresponding non-local Value. If the original Value was an Ebb parameter,
+    /// the parameter may be removed if redundant. Parameters are placed eagerly by callers
+    /// to avoid infinite loops when looking up a Value for an Ebb that is in a CFG loop.
+    ///
+    /// Doing this lookup for each Value in each Ebb preserves SSA form during construction.
+    ///
+    /// Returns the chosen Value.
+    ///
+    /// ## Arguments
+    ///
+    /// `sentinel` is a dummy Ebb parameter inserted by `use_var_nonlocal()`.
+    /// Its purpose is to allow detection of CFG cycles while traversing predecessors.
+    ///
+    /// The `sentinel: Value` and the `ty: Type` are describing the `var: Variable`
+    /// that is being looked up.
     fn predecessors_lookup(
         &mut self,
         func: &mut Function,
-        temp_arg_val: Value,
-        temp_arg_var: Variable,
+        sentinel: Value,
+        var: Variable,
         ty: Type,
-        dest_ebb: Ebb,
+        ebb: Ebb,
     ) -> Value {
         debug_assert!(self.calls.is_empty());
         debug_assert!(self.results.is_empty());
         // self.side_effects may be non-empty here so that callers can
         // accumulate side effects over multiple calls.
-        self.begin_predecessors_lookup(temp_arg_val, dest_ebb);
-        self.run_state_machine(func, temp_arg_var, ty)
+        self.begin_predecessors_lookup(sentinel, ebb);
+        self.run_state_machine(func, var, ty)
     }
 
-    /// Initiate use lookups in all predecessors of `dest_ebb`, and arrange for a call
-    /// to `finish_predecessors_lookup` once they complete.
-    fn begin_predecessors_lookup(&mut self, temp_arg_val: Value, dest_ebb: Ebb) {
+    /// Set up state for `run_state_machine()` to initiate non-local use lookups
+    /// in all predecessors of `dest_ebb`, and arrange for a call to
+    /// `finish_predecessors_lookup` once they complete.
+    fn begin_predecessors_lookup(&mut self, sentinel: Value, dest_ebb: Ebb) {
         self.calls
-            .push(Call::FinishPredecessorsLookup(temp_arg_val, dest_ebb));
+            .push(Call::FinishPredecessorsLookup(sentinel, dest_ebb));
         // Iterate over the predecessors.
         let mut calls = mem::replace(&mut self.calls, Vec::new());
         calls.extend(
@@ -527,31 +565,36 @@ impl SSABuilder {
     fn finish_predecessors_lookup(
         &mut self,
         func: &mut Function,
-        temp_arg_val: Value,
-        temp_arg_var: Variable,
+        sentinel: Value,
+        var: Variable,
         dest_ebb: Ebb,
     ) {
         let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero;
 
-        // Iterate over the predecessors.
-        for _ in 0..self.predecessors(dest_ebb).len() {
-            // For each predecessor, we query what is the local SSA value corresponding
-            // to var and we put it as an argument of the branch instruction.
-            let pred_val = self.results.pop().unwrap();
+        // Determine how many predecessors are yielding unique, non-temporary Values.
+        let num_predecessors = self.predecessors(dest_ebb).len();
+        for &pred_val in self.results.iter().rev().take(num_predecessors) {
             match pred_values {
                 ZeroOneOrMore::Zero => {
-                    if pred_val != temp_arg_val {
+                    if pred_val != sentinel {
                         pred_values = ZeroOneOrMore::One(pred_val);
                     }
                 }
                 ZeroOneOrMore::One(old_val) => {
-                    if pred_val != temp_arg_val && pred_val != old_val {
+                    if pred_val != sentinel && pred_val != old_val {
                         pred_values = ZeroOneOrMore::More;
+                        break;
                     }
                 }
-                ZeroOneOrMore::More => {}
+                ZeroOneOrMore::More => {
+                    break;
+                }
             }
         }
+
+        // Those predecessors' Values have been examined: pop all their results.
+        self.results.truncate(self.results.len() - num_predecessors);
+
         let result_val = match pred_values {
             ZeroOneOrMore::Zero => {
                 // The variable is used but never defined before. This is an irregularity in the
@@ -562,11 +605,11 @@ impl SSABuilder {
                 }
                 self.side_effects.instructions_added_to_ebbs.push(dest_ebb);
                 let zero = emit_zero(
-                    func.dfg.value_type(temp_arg_val),
+                    func.dfg.value_type(sentinel),
                     FuncCursor::new(func).at_first_insertion_point(dest_ebb),
                 );
-                func.dfg.remove_ebb_param(temp_arg_val);
-                func.dfg.change_to_alias(temp_arg_val, zero);
+                func.dfg.remove_ebb_param(sentinel);
+                func.dfg.change_to_alias(sentinel, zero);
                 zero
             }
             ZeroOneOrMore::One(pred_val) => {
@@ -577,15 +620,15 @@ impl SSABuilder {
                 // Resolve aliases eagerly so that we can check for cyclic aliasing,
                 // which can occur in unreachable code.
                 let mut resolved = func.dfg.resolve_aliases(pred_val);
-                if temp_arg_val == resolved {
+                if sentinel == resolved {
                     // Cycle detected. Break it by creating a zero value.
                     resolved = emit_zero(
-                        func.dfg.value_type(temp_arg_val),
+                        func.dfg.value_type(sentinel),
                         FuncCursor::new(func).at_first_insertion_point(dest_ebb),
                     );
                 }
-                func.dfg.remove_ebb_param(temp_arg_val);
-                func.dfg.change_to_alias(temp_arg_val, resolved);
+                func.dfg.remove_ebb_param(sentinel);
+                func.dfg.change_to_alias(sentinel, resolved);
                 resolved
             }
             ZeroOneOrMore::More => {
@@ -600,20 +643,15 @@ impl SSABuilder {
                 } in &mut preds
                 {
                     // We already did a full `use_var` above, so we can do just the fast path.
-                    let pred_val = self
-                        .variables
-                        .get(temp_arg_var)
-                        .unwrap()
-                        .get(*pred_block)
-                        .unwrap()
-                        .unwrap();
+                    let block_map = self.variables.get(var).unwrap();
+                    let pred_val = block_map.get(*pred_block).unwrap().unwrap();
                     let jump_arg = self.append_jump_argument(
                         func,
                         *last_inst,
                         *pred_block,
                         dest_ebb,
                         pred_val,
-                        temp_arg_var,
+                        var,
                     );
                     if let Some((middle_ebb, middle_block, middle_jump_inst)) = jump_arg {
                         *pred_block = middle_block;
@@ -625,7 +663,7 @@ impl SSABuilder {
                 debug_assert!(self.predecessors(dest_ebb).is_empty());
                 *self.predecessors_mut(dest_ebb) = preds;
 
-                temp_arg_val
+                sentinel
             }
         };
 
@@ -743,8 +781,8 @@ impl SSABuilder {
                 Call::FinishSealedOnePredecessor(block) => {
                     self.finish_sealed_one_predecessor(var, block);
                 }
-                Call::FinishPredecessorsLookup(temp_arg_val, dest_ebb) => {
-                    self.finish_predecessors_lookup(func, temp_arg_val, var, dest_ebb);
+                Call::FinishPredecessorsLookup(sentinel, dest_ebb) => {
+                    self.finish_predecessors_lookup(func, sentinel, var, dest_ebb);
                 }
             }
         }

From fad6bb1a5c8b4d52b84297f8611948f77494ae7a Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Thu, 17 Oct 2019 11:35:33 -0600
Subject: [PATCH 2859/3084] Fix build by marking tests as incompatible with
 basic-blocks. Closes #1152

---
 .../filetests/filetests/regalloc/solver-fixedconflict-var-3.clif | 1 +
 .../filetests/filetests/regalloc/solver-fixedconflict-var.clif   | 1 +
 2 files changed, 2 insertions(+)

diff --git a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif
index 13a613d1bf..c06c22f6b3 100644
--- a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif
+++ b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif
@@ -2,6 +2,7 @@ test compile
 set opt_level=speed
 set enable_pinned_reg=true
 target x86_64 haswell
+feature !"basic-blocks"
 
 function u0:0(i32, i32, i32, i64 vmctx) -> i64 uext system_v {
 ebb0(v0: i32, v1: i32, v2: i32, v3: i64):
diff --git a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
index fc643ddbd4..25bcd405f8 100644
--- a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
+++ b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
@@ -2,6 +2,7 @@ test compile
 set opt_level=speed
 set enable_pinned_reg=true
 target x86_64 haswell
+feature !"basic-blocks"
 
 ;; Test for the issue #1123; https://github.com/CraneStation/cranelift/issues/1123
 

From 8f74333662921a749222ecffd720bb8258123696 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 8 Oct 2019 20:32:52 -0700
Subject: [PATCH 2860/3084] Add x86 SIMD band_not

---
 .../codegen/meta/src/isa/x86/encodings.rs     |  4 ++++
 cranelift/codegen/meta/src/isa/x86/opcodes.rs |  3 +++
 .../isa/x86/simd-logical-binemit.clif         |  6 +++++
 .../filetests/isa/x86/simd-logical-run.clif   | 23 +++++++++++++++++++
 4 files changed, 36 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/simd-logical-run.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 8457369929..f957dd7791 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1982,6 +1982,10 @@ pub(crate) fn define<'defs>(
         let band = band.bind(vector(ty, sse_vector_size));
         e.enc_32_64(band, rec_fa.opcodes(&PAND));
 
+        // and not (note flipped recipe operands to match band_not order)
+        let band_not = band_not.bind(vector(ty, sse_vector_size));
+        e.enc_32_64(band_not, rec_fax.opcodes(&PANDN));
+
         // or
         let bor = bor.bind(vector(ty, sse_vector_size));
         e.enc_32_64(bor, rec_fa.opcodes(&POR));
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index 0fa7c8a7f7..2df259f37e 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -266,6 +266,9 @@ pub static PADDUSW: [u8; 3] = [0x66, 0x0f, 0xdd];
 /// Bitwise AND of xmm2/m128 and xmm1 (SSE2).
 pub static PAND: [u8; 3] = [0x66, 0x0f, 0xdb];
 
+/// Bitwise AND NOT of xmm2/m128 and xmm1 (SSE2).
+pub static PANDN: [u8; 3] = [0x66, 0x0f, 0xdf];
+
 /// Compare packed data for equal (SSE2).
 pub static PCMPEQB: [u8; 3] = [0x66, 0x0f, 0x74];
 
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
index dd0365b016..835eb8ca2f 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
@@ -19,3 +19,9 @@ ebb0(v0: b32x4 [%xmm4], v1: b32x4 [%xmm0]):
 [-, %xmm4]  v2 = bxor v0, v1      ; bin: 66 0f ef e0
             return v2
 }
+
+function %band_not_b64x2(b64x2, b64x2) -> b64x2 {
+ebb0(v0: b64x2 [%xmm6], v1: b64x2 [%xmm3]):
+[-, %xmm3]  v2 = band_not v0, v1      ; bin: 66 0f df de
+            return v2
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif
new file mode 100644
index 0000000000..6ab5db0c49
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif
@@ -0,0 +1,23 @@
+test run
+set enable_simd
+target x86_64 skylake
+
+function %bnot() -> b32 {
+ebb0:
+    v0 = vconst.b32x4 [true true true false]
+    v1 = bnot v0
+    v2 = extractlane v1, 3
+    return v2
+}
+; run
+
+function %band_not() -> b1 {
+ebb0:
+    v0 = vconst.i16x8 [1 0 0 0 0 0 0 0]
+    v1 = vconst.i16x8 [0 0 0 0 0 0 0 0]
+    v2 = band_not v0, v1
+    v3 = extractlane v2, 0
+    v4 = icmp_imm eq v3, 1
+    return v4
+}
+; run

From b927c5551180173bbef579161e7631bda470a75b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 11 Oct 2019 15:12:46 -0700
Subject: [PATCH 2861/3084] Add SIMD bitselect instruction and x86 legalization

This new instructions matches the `bitselect` behavior described in the WASM SIMD spec (https://github.com/WebAssembly/simd/blob/master/proposals/simd/SIMD.md#bitwise-select)
---
 .../codegen/meta/src/isa/x86/legalize.rs      | 15 +++++++++++++
 .../codegen/meta/src/shared/instructions.rs   | 16 ++++++++++++++
 .../isa/x86/simd-bitwise-legalize.clif        | 12 ++++++++++
 .../filetests/isa/x86/simd-bitwise-run.clif   | 22 +++++++++++++++++++
 4 files changed, 65 insertions(+)

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 8d9033d3a9..df0b88dc04 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -20,7 +20,9 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     // List of instructions.
     let insts = &shared.instructions;
     let band = insts.by_name("band");
+    let band_not = insts.by_name("band_not");
     let bitcast = insts.by_name("bitcast");
+    let bitselect = insts.by_name("bitselect");
     let bor = insts.by_name("bor");
     let bnot = insts.by_name("bnot");
     let bxor = insts.by_name("bxor");
@@ -431,6 +433,19 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         );
     }
 
+    // SIMD select
+    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
+        let bitselect = bitselect.bind(vector(ty, sse_vector_size)); // must bind both x/y and c
+        narrow.legalize(
+            def!(d = bitselect(c, x, y)),
+            vec![
+                def!(a = band(x, c)),
+                def!(b = band_not(y, c)),
+                def!(d = bor(a, b)),
+            ],
+        );
+    }
+
     narrow.custom_legalize(shuffle, "convert_shuffle");
     narrow.custom_legalize(extractlane, "convert_extractlane");
     narrow.custom_legalize(insertlane, "convert_insertlane");
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index b9070c7f39..e13871fa3c 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1199,6 +1199,22 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    let c = &operand_doc("c", Any, "Controlling value to test");
+    ig.push(
+        Inst::new(
+            "bitselect",
+            r#"
+        Conditional select of bits.
+
+        For each bit in `c`, this instruction selects the corresponding bit from `x` if the bit 
+        in `c` is 1 and the corresponding bit from `y` if the bit in `c` is 0. See also: 
+        `select`, `vselect`.
+        "#,
+        )
+        .operands_in(vec![c, x, y])
+        .operands_out(vec![a]),
+    );
+
     let x = &operand("x", Any);
 
     ig.push(
diff --git a/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif
index 9c728eb208..e8391c8a73 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif
@@ -31,3 +31,15 @@ ebb0:
     ; nextln: v2 = x86_psra v1, v3
     return v2
 }
+
+function %bitselect_i16x8() -> i16x8 {
+ebb0:
+    v0 = vconst.i16x8 [0 0 0 0 0 0 0 0]
+    v1 = vconst.i16x8 [0 0 0 0 0 0 0 0]
+    v2 = vconst.i16x8 [0 0 0 0 0 0 0 0]
+    v3 = bitselect v0, v1, v2
+    ; check: v4 = band v1, v0
+    ; nextln: v5 = band_not v2, v0
+    ; nextln: v3 = bor v4, v5
+    return v3
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif b/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
index 07c50bee0a..0c6eac6a10 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
@@ -105,3 +105,25 @@ ebb0:
     return v7
 }
 ; run
+
+function %bitselect_i8x16() -> b1 {
+ebb0:
+    v0 = vconst.i8x16 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255]  ; the selector vector
+    v1 = vconst.i8x16 [127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 42] ; for each 1-bit in v0 the bit of v1 is selected
+    v2 = vconst.i8x16 [42 0 0 0 0 0 0 0 0 0 0 0 0 0 0 127] ; for each 0-bit in v0 the bit of v2 is selected
+    v3 = bitselect v0, v1, v2
+
+    v4 = extractlane v3, 0
+    v5 = icmp_imm eq v4, 42
+
+    v6 = extractlane v3, 1
+    v7 = icmp_imm eq v6, 0
+
+    v8 = extractlane v3, 15
+    v9 = icmp_imm eq v8, 42
+
+    v10 = band v5, v7
+    v11 = band v10, v9
+    return v11
+}
+; run

From 135f9eb4a645d563691fe37b3f3f57c08e651332 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 11 Oct 2019 15:16:46 -0700
Subject: [PATCH 2862/3084] Translate WASM bitselect to CLIF

---
 cranelift/wasm/src/code_translator.rs | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 8039635db4..01c72fab2a 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1138,6 +1138,15 @@ pub fn translate_operator(
             let b_mod_bitwidth = builder.ins().band_imm(b, bitwidth - 1);
             state.push1(builder.ins().sshr(bitcast_a, b_mod_bitwidth))
         }
+        Operator::V128Bitselect => {
+            let (a, b, c) = state.pop3();
+            let bitcast_a = optionally_bitcast_vector(a, I8X16, builder);
+            let bitcast_b = optionally_bitcast_vector(b, I8X16, builder);
+            let bitcast_c = optionally_bitcast_vector(c, I8X16, builder);
+            // The CLIF operand ordering is slightly different and the types of all three
+            // operands must match (hence the bitcast).
+            state.push1(builder.ins().bitselect(bitcast_c, bitcast_a, bitcast_b))
+        }
         Operator::I8x16Eq
         | Operator::I8x16Ne
         | Operator::I8x16LtS
@@ -1180,7 +1189,6 @@ pub fn translate_operator(
         | Operator::F64x2Gt
         | Operator::F64x2Le
         | Operator::F64x2Ge
-        | Operator::V128Bitselect
         | Operator::I8x16AnyTrue
         | Operator::I8x16AllTrue
         | Operator::I8x16Shl

From d102bf9b61234860ba9b5dbbc5784f6c71351df7 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 18 Oct 2019 16:05:01 +0200
Subject: [PATCH 2863/3084] Add allow(dead_code) for Immediate until it's used
 in legalization patterns; (#1157)

---
 cranelift/codegen/meta/src/cdsl/instructions.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 464aa6e248..4acbd4a645 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -452,6 +452,7 @@ impl From for BindParameter {
 }
 
 #[derive(Clone)]
+#[allow(dead_code)] // TODO(#1150): remove this once we use it in legalization patterns.
 pub enum Immediate {
     UInt8(u8),
     UInt128(u128),

From 8fece43aa126cf26345c948844c6b46f042789d0 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 18 Oct 2019 15:25:33 +0200
Subject: [PATCH 2864/3084] [meta] Use a type alias for Instruction;

... instead of embedding a Rc, just make it a type
alias.
---
 .../codegen/meta/src/cdsl/instructions.rs     | 71 ++++++++-----------
 1 file changed, 29 insertions(+), 42 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 4acbd4a645..0e18ced843 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -3,7 +3,6 @@ use cranelift_entity::{entity_impl, PrimaryMap};
 use std::collections::HashMap;
 use std::fmt;
 use std::fmt::{Display, Error, Formatter};
-use std::ops;
 use std::rc::Rc;
 
 use crate::cdsl::camel_case;
@@ -152,19 +151,7 @@ pub(crate) struct InstructionContent {
     pub writes_cpu_flags: bool,
 }
 
-#[derive(Clone, Debug)]
-pub(crate) struct Instruction {
-    content: Rc,
-}
-
-impl ops::Deref for Instruction {
-    type Target = InstructionContent;
-    fn deref(&self) -> &Self::Target {
-        &*self.content
-    }
-}
-
-impl Instruction {
+impl InstructionContent {
     pub fn snake_name(&self) -> &str {
         if &self.name == "return" {
             "return_"
@@ -185,13 +172,15 @@ impl Instruction {
     }
 }
 
+pub(crate) type Instruction = Rc;
+
 impl Bindable for Instruction {
     fn bind(&self, parameter: impl Into) -> BoundInstruction {
         BoundInstruction::new(self).bind(parameter)
     }
 }
 
-impl fmt::Display for Instruction {
+impl fmt::Display for InstructionContent {
     fn fmt(&self, fmt: &mut fmt::Formatter) -> Result<(), fmt::Error> {
         if self.operands_out.len() > 0 {
             let operands_out = self
@@ -352,33 +341,31 @@ impl InstructionBuilder {
 
         let camel_name = camel_case(&self.name);
 
-        Instruction {
-            content: Rc::new(InstructionContent {
-                name: self.name,
-                camel_name,
-                opcode_number,
-                doc: self.doc,
-                operands_in,
-                operands_out,
-                constraints: self.constraints.unwrap_or_else(Vec::new),
-                format: format_index,
-                polymorphic_info,
-                value_opnums,
-                value_results,
-                imm_opnums,
-                is_terminator: self.is_terminator,
-                is_branch: self.is_branch,
-                is_indirect_branch: self.is_indirect_branch,
-                is_call: self.is_call,
-                is_return: self.is_return,
-                is_ghost: self.is_ghost,
-                can_load: self.can_load,
-                can_store: self.can_store,
-                can_trap: self.can_trap,
-                other_side_effects: self.other_side_effects,
-                writes_cpu_flags,
-            }),
-        }
+        Rc::new(InstructionContent {
+            name: self.name,
+            camel_name,
+            opcode_number,
+            doc: self.doc,
+            operands_in,
+            operands_out,
+            constraints: self.constraints.unwrap_or_else(Vec::new),
+            format: format_index,
+            polymorphic_info,
+            value_opnums,
+            value_results,
+            imm_opnums,
+            is_terminator: self.is_terminator,
+            is_branch: self.is_branch,
+            is_indirect_branch: self.is_indirect_branch,
+            is_call: self.is_call,
+            is_return: self.is_return,
+            is_ghost: self.is_ghost,
+            can_load: self.can_load,
+            can_store: self.can_store,
+            can_trap: self.can_trap,
+            other_side_effects: self.other_side_effects,
+            writes_cpu_flags,
+        })
     }
 }
 

From d3e694fbe71086d9521a79fb3a1e6f4daf4a092e Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 18 Oct 2019 16:04:51 +0200
Subject: [PATCH 2865/3084] [meta] Remove unused InstructionGroup::{name, doc};

---
 cranelift/codegen/meta/src/cdsl/instructions.rs    | 10 ----------
 cranelift/codegen/meta/src/cdsl/xform.rs           |  2 +-
 cranelift/codegen/meta/src/isa/arm32/mod.rs        |  2 --
 cranelift/codegen/meta/src/isa/arm64/mod.rs        |  2 --
 cranelift/codegen/meta/src/isa/riscv/mod.rs        |  2 --
 cranelift/codegen/meta/src/isa/x86/instructions.rs |  7 +------
 cranelift/codegen/meta/src/shared/instructions.rs  |  7 +------
 7 files changed, 3 insertions(+), 29 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 0e18ced843..e642a512a7 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -23,8 +23,6 @@ entity_impl!(OpcodeNumber);
 pub(crate) type AllInstructions = PrimaryMap;
 
 pub(crate) struct InstructionGroupBuilder<'format_reg, 'all_inst> {
-    _name: &'static str,
-    _doc: &'static str,
     format_registry: &'format_reg FormatRegistry,
     all_instructions: &'all_inst mut AllInstructions,
     own_instructions: Vec,
@@ -32,14 +30,10 @@ pub(crate) struct InstructionGroupBuilder<'format_reg, 'all_inst> {
 
 impl<'format_reg, 'all_inst> InstructionGroupBuilder<'format_reg, 'all_inst> {
     pub fn new(
-        name: &'static str,
-        doc: &'static str,
         all_instructions: &'all_inst mut AllInstructions,
         format_registry: &'format_reg FormatRegistry,
     ) -> Self {
         Self {
-            _name: name,
-            _doc: doc,
             format_registry,
             all_instructions,
             own_instructions: Vec::new(),
@@ -56,8 +50,6 @@ impl<'format_reg, 'all_inst> InstructionGroupBuilder<'format_reg, 'all_inst> {
 
     pub fn build(self) -> InstructionGroup {
         InstructionGroup {
-            _name: self._name,
-            _doc: self._doc,
             instructions: self.own_instructions,
         }
     }
@@ -67,8 +59,6 @@ impl<'format_reg, 'all_inst> InstructionGroupBuilder<'format_reg, 'all_inst> {
 /// target architecture can support instructions from multiple groups, and it
 /// does not necessarily support all instructions in a group.
 pub(crate) struct InstructionGroup {
-    _name: &'static str,
-    _doc: &'static str,
     instructions: Vec,
 }
 
diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index 916bdc0a32..6980998d70 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -471,7 +471,7 @@ fn test_double_custom_legalization() {
     let mut dummy_all = AllInstructions::new();
     let mut format = FormatRegistry::new();
     format.insert(InstructionFormatBuilder::new("nullary"));
-    let mut inst_group = InstructionGroupBuilder::new("test", "", &mut dummy_all, &format);
+    let mut inst_group = InstructionGroupBuilder::new(&mut dummy_all, &format);
     inst_group.push(InstructionBuilder::new("dummy", "doc"));
     let inst_group = inst_group.build();
     let dummy_inst = inst_group.by_name("dummy");
diff --git a/cranelift/codegen/meta/src/isa/arm32/mod.rs b/cranelift/codegen/meta/src/isa/arm32/mod.rs
index 1f3c05ad60..152d57d598 100644
--- a/cranelift/codegen/meta/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm32/mod.rs
@@ -54,8 +54,6 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let regs = define_regs();
 
     let inst_group = InstructionGroupBuilder::new(
-        "arm32",
-        "arm32 specific instruction set",
         &mut shared_defs.all_instructions,
         &shared_defs.format_registry,
     )
diff --git a/cranelift/codegen/meta/src/isa/arm64/mod.rs b/cranelift/codegen/meta/src/isa/arm64/mod.rs
index 15bfb736cc..3811cc25a6 100644
--- a/cranelift/codegen/meta/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm64/mod.rs
@@ -50,8 +50,6 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let regs = define_registers();
 
     let inst_group = InstructionGroupBuilder::new(
-        "arm64",
-        "arm64 specific instruction set",
         &mut shared_defs.all_instructions,
         &shared_defs.format_registry,
     )
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index 14eb3267f6..ef173176c0 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -90,8 +90,6 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let regs = define_registers();
 
     let inst_group = InstructionGroupBuilder::new(
-        "riscv",
-        "riscv specific instruction set",
         &mut shared_defs.all_instructions,
         &shared_defs.format_registry,
     )
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 6ed1e88999..d5d02b1ac5 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -15,12 +15,7 @@ pub(crate) fn define(
     format_registry: &FormatRegistry,
     immediates: &Immediates,
 ) -> InstructionGroup {
-    let mut ig = InstructionGroupBuilder::new(
-        "x86",
-        "x86 specific instruction set",
-        &mut all_instructions,
-        format_registry,
-    );
+    let mut ig = InstructionGroupBuilder::new(&mut all_instructions, format_registry);
 
     let iflags: &TypeVar = &ValueType::Special(types::Flag::IFlags.into()).into();
 
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index e13871fa3c..f6d181def2 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -17,12 +17,7 @@ pub(crate) fn define(
     imm: &Immediates,
     entities: &EntityRefs,
 ) -> InstructionGroup {
-    let mut ig = InstructionGroupBuilder::new(
-        "base",
-        "Shared base instruction set",
-        all_instructions,
-        format_registry,
-    );
+    let mut ig = InstructionGroupBuilder::new(all_instructions, format_registry);
 
     // Operand kind shorthands.
     let iflags: &TypeVar = &ValueType::Special(types::Flag::IFlags.into()).into();

From 9e9a7626d7f4019dd10b39654ccf6bb2988615d2 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 18 Oct 2019 16:45:51 +0200
Subject: [PATCH 2866/3084] [meta] Use a ref-counted pointer to an
 InstructionFormat in instructions;

This avoids a lot of dereferences, and InstructionFormat are immutable
once they're created. It removes a lot of code that was keeping the
FormatRegistry around, just in case we needed the format. This is more
in line with the way we create Instructions, and make it easy to
reference InstructionFormats in general.
---
 cranelift/codegen/meta/src/cdsl/ast.rs        |  29 ++--
 cranelift/codegen/meta/src/cdsl/encodings.rs  |  12 +-
 cranelift/codegen/meta/src/cdsl/formats.rs    |  46 ++---
 .../codegen/meta/src/cdsl/instructions.rs     |  15 +-
 cranelift/codegen/meta/src/cdsl/recipes.rs    |  23 +--
 cranelift/codegen/meta/src/gen_binemit.rs     |  15 +-
 cranelift/codegen/meta/src/gen_inst.rs        |  19 +-
 cranelift/codegen/meta/src/gen_legalizer.rs   |  43 +----
 .../codegen/meta/src/isa/riscv/encodings.rs   |  13 +-
 .../codegen/meta/src/isa/riscv/recipes.rs     |  46 ++---
 .../codegen/meta/src/isa/x86/encodings.rs     |  64 +++----
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 164 +++++++++---------
 cranelift/codegen/meta/src/lib.rs             |   9 +-
 13 files changed, 222 insertions(+), 276 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index f047a74e55..3a994f27ed 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -1,4 +1,3 @@
-use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::instructions::{InstSpec, Instruction, InstructionPredicate};
 use crate::cdsl::operands::{OperandKind, OperandKindFields};
 use crate::cdsl::types::ValueType;
@@ -523,23 +522,23 @@ impl Apply {
         format!("{}({})", inst_name, args)
     }
 
-    pub fn inst_predicate(
-        &self,
-        format_registry: &FormatRegistry,
-        var_pool: &VarPool,
-    ) -> InstructionPredicate {
-        let iform = format_registry.get(self.inst.format);
-
+    pub fn inst_predicate(&self, var_pool: &VarPool) -> InstructionPredicate {
         let mut pred = InstructionPredicate::new();
-        for (format_field, &op_num) in iform.imm_fields.iter().zip(self.inst.imm_opnums.iter()) {
+        for (format_field, &op_num) in self
+            .inst
+            .format
+            .imm_fields
+            .iter()
+            .zip(self.inst.imm_opnums.iter())
+        {
             let arg = &self.args[op_num];
             if arg.maybe_var().is_some() {
                 // Ignore free variables for now.
                 continue;
             }
             pred = pred.and(InstructionPredicate::new_is_field_equal_ast(
-                iform,
-                &format_field,
+                &*self.inst.format,
+                format_field,
                 arg.to_rust_code(var_pool),
             ));
         }
@@ -565,12 +564,8 @@ impl Apply {
     }
 
     /// Same as `inst_predicate()`, but also check the controlling type variable.
-    pub fn inst_predicate_with_ctrl_typevar(
-        &self,
-        format_registry: &FormatRegistry,
-        var_pool: &VarPool,
-    ) -> InstructionPredicate {
-        let mut pred = self.inst_predicate(format_registry, var_pool);
+    pub fn inst_predicate_with_ctrl_typevar(&self, var_pool: &VarPool) -> InstructionPredicate {
+        let mut pred = self.inst_predicate(var_pool);
 
         if !self.value_types.is_empty() {
             let bound_type = &self.value_types[0];
diff --git a/cranelift/codegen/meta/src/cdsl/encodings.rs b/cranelift/codegen/meta/src/cdsl/encodings.rs
index 540d3402ae..6797672a87 100644
--- a/cranelift/codegen/meta/src/cdsl/encodings.rs
+++ b/cranelift/codegen/meta/src/cdsl/encodings.rs
@@ -1,4 +1,3 @@
-use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::instructions::{
     InstSpec, Instruction, InstructionPredicate, InstructionPredicateNode,
     InstructionPredicateNumber, InstructionPredicateRegistry, ValueTypeOrAny,
@@ -62,12 +61,7 @@ pub(crate) struct EncodingBuilder {
 }
 
 impl EncodingBuilder {
-    pub fn new(
-        inst: InstSpec,
-        recipe: EncodingRecipeNumber,
-        encbits: u16,
-        formats: &FormatRegistry,
-    ) -> Self {
+    pub fn new(inst: InstSpec, recipe: EncodingRecipeNumber, encbits: u16) -> Self {
         let (inst_predicate, bound_type) = match &inst {
             InstSpec::Bound(inst) => {
                 let other_typevars = &inst.inst.polymorphic_info.as_ref().unwrap().other_typevars;
@@ -98,7 +92,7 @@ impl EncodingBuilder {
                     .zip(inst.inst.operands_in.iter().filter(|o| o.is_immediate()))
                 {
                     let immediate_predicate = InstructionPredicate::new_is_field_equal(
-                        formats.get(inst.inst.format),
+                        &inst.inst.format,
                         immediate_operand.name,
                         immediate_value.to_string(),
                     );
@@ -158,7 +152,7 @@ impl EncodingBuilder {
 
         let inst = self.inst.inst();
         assert!(
-            inst.format == recipes[self.recipe].format,
+            Rc::ptr_eq(&inst.format, &recipes[self.recipe].format),
             format!(
                 "Inst {} and recipe {} must have the same format!",
                 inst.name, recipes[self.recipe].name
diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
index cf82da180a..0e71f37a97 100644
--- a/cranelift/codegen/meta/src/cdsl/formats.rs
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -2,10 +2,9 @@ use crate::cdsl::operands::{Operand, OperandKind};
 
 use std::collections::{HashMap, HashSet};
 use std::fmt;
+use std::rc::Rc;
 use std::slice;
 
-use cranelift_entity::{entity_impl, PrimaryMap};
-
 /// An immediate field in an instruction format.
 ///
 /// This corresponds to a single member of a variant of the `InstructionData`
@@ -152,15 +151,10 @@ impl InstructionFormatBuilder {
     }
 }
 
-#[derive(Copy, Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
-pub struct InstructionFormatIndex(u32);
-entity_impl!(InstructionFormatIndex);
-
 pub struct FormatRegistry {
-    /// Map (immediate kinds names, number of values, has varargs) to an instruction format index
-    /// in the actual map.
-    sig_to_index: HashMap<(Vec, usize, bool), InstructionFormatIndex>,
-    map: PrimaryMap,
+    /// Map (immediate kinds names, number of values, has varargs) to an instruction format.
+    sig_to_index: HashMap<(Vec, usize, bool), usize>,
+    formats: Vec>,
     name_set: HashSet<&'static str>,
 }
 
@@ -168,14 +162,14 @@ impl FormatRegistry {
     pub fn new() -> Self {
         Self {
             sig_to_index: HashMap::new(),
-            map: PrimaryMap::new(),
+            formats: Vec::new(),
             name_set: HashSet::new(),
         }
     }
 
     /// Find an existing instruction format that matches the given lists of instruction inputs and
     /// outputs.
-    pub fn lookup(&self, operands_in: &Vec) -> InstructionFormatIndex {
+    pub fn lookup(&self, operands_in: &Vec) -> &Rc {
         let mut imm_keys = Vec::new();
         let mut num_values = 0;
         let mut has_varargs = false;
@@ -193,22 +187,19 @@ impl FormatRegistry {
         }
 
         let sig = (imm_keys, num_values, has_varargs);
-        *self
+        let index = *self
             .sig_to_index
             .get(&sig)
-            .expect("unknown InstructionFormat; please define it in shared/formats.rs first")
+            .expect("unknown InstructionFormat; please define it in shared/formats.rs first");
+        &self.formats[index]
     }
 
-    pub fn by_name(&self, name: &str) -> InstructionFormatIndex {
-        self.map
+    pub fn by_name(&self, name: &str) -> &Rc {
+        &self
+            .formats
             .iter()
-            .find(|(_key, value)| value.name == name)
+            .find(|format| format.name == name)
             .unwrap_or_else(|| panic!("format with name '{}' doesn't exist", name))
-            .0
-    }
-
-    pub fn get(&self, index: InstructionFormatIndex) -> &InstructionFormat {
-        self.map.get(index).unwrap()
     }
 
     pub fn insert(&mut self, inst_format: InstructionFormatBuilder) {
@@ -230,17 +221,18 @@ impl FormatRegistry {
             .collect();
         let key = (imm_keys, format.num_value_operands, format.has_value_list);
 
-        let index = self.map.push(format);
+        let index = self.formats.len();
+        self.formats.push(Rc::new(format));
         if let Some(already_inserted) = self.sig_to_index.insert(key, index) {
             panic!(
                 "duplicate InstructionFormat: trying to insert '{}' while '{}' already has the same structure.",
-                self.map.get(index).unwrap().name,
-                self.map.get(already_inserted).unwrap().name
+                self.formats[index].name,
+                self.formats[already_inserted].name
             );
         }
     }
 
-    pub fn iter(&self) -> slice::Iter {
-        self.map.values()
+    pub fn iter(&self) -> slice::Iter> {
+        self.formats.iter()
     }
 }
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index e642a512a7..fb92b85572 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -6,9 +6,7 @@ use std::fmt::{Display, Error, Formatter};
 use std::rc::Rc;
 
 use crate::cdsl::camel_case;
-use crate::cdsl::formats::{
-    FormatField, FormatRegistry, InstructionFormat, InstructionFormatIndex,
-};
+use crate::cdsl::formats::{FormatField, FormatRegistry, InstructionFormat};
 use crate::cdsl::operands::Operand;
 use crate::cdsl::type_inference::Constraint;
 use crate::cdsl::types::{LaneType, ReferenceType, ValueType, VectorType};
@@ -104,7 +102,7 @@ pub(crate) struct InstructionContent {
     pub constraints: Vec,
 
     /// Instruction format, automatically derived from the input operands.
-    pub format: InstructionFormatIndex,
+    pub format: Rc,
 
     /// One of the input or output operands is a free type variable. None if the instruction is not
     /// polymorphic, set otherwise.
@@ -321,8 +319,7 @@ impl InstructionBuilder {
             .filter_map(|(i, op)| if op.is_value() { Some(i) } else { None })
             .collect();
 
-        let format_index = format_registry.lookup(&operands_in);
-        let format = format_registry.get(format_index);
+        let format = format_registry.lookup(&operands_in).clone();
         let polymorphic_info =
             verify_polymorphic(&operands_in, &operands_out, &format, &value_opnums);
 
@@ -339,7 +336,7 @@ impl InstructionBuilder {
             operands_in,
             operands_out,
             constraints: self.constraints.unwrap_or_else(Vec::new),
-            format: format_index,
+            format,
             polymorphic_info,
             value_opnums,
             value_results,
@@ -1093,9 +1090,9 @@ impl InstructionPredicate {
     }
 
     pub fn new_is_colocated_data(format_registry: &FormatRegistry) -> InstructionPredicateNode {
-        let format = format_registry.get(format_registry.by_name("UnaryGlobalValue"));
+        let format = format_registry.by_name("UnaryGlobalValue");
         InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
-            format,
+            &*format,
             "global_value",
             FormatPredicateKind::IsColocatedData,
         ))
diff --git a/cranelift/codegen/meta/src/cdsl/recipes.rs b/cranelift/codegen/meta/src/cdsl/recipes.rs
index ed31cbbfd8..20b5c077cb 100644
--- a/cranelift/codegen/meta/src/cdsl/recipes.rs
+++ b/cranelift/codegen/meta/src/cdsl/recipes.rs
@@ -1,6 +1,8 @@
+use std::rc::Rc;
+
 use cranelift_entity::{entity_impl, PrimaryMap};
 
-use crate::cdsl::formats::{FormatRegistry, InstructionFormatIndex};
+use crate::cdsl::formats::InstructionFormat;
 use crate::cdsl::instructions::InstructionPredicate;
 use crate::cdsl::regs::RegClassIndex;
 use crate::cdsl::settings::SettingPredicateNumber;
@@ -108,7 +110,7 @@ pub(crate) struct EncodingRecipe {
     pub name: String,
 
     /// Associated instruction format.
-    pub format: InstructionFormatIndex,
+    pub format: Rc,
 
     /// Base number of bytes in the binary encoded instruction.
     pub base_size: u64,
@@ -141,7 +143,7 @@ pub(crate) struct EncodingRecipe {
 // Implement PartialEq ourselves: take all the fields into account but the name.
 impl PartialEq for EncodingRecipe {
     fn eq(&self, other: &Self) -> bool {
-        self.format == other.format
+        Rc::ptr_eq(&self.format, &other.format)
             && self.base_size == other.base_size
             && self.operands_in == other.operands_in
             && self.operands_out == other.operands_out
@@ -166,7 +168,7 @@ pub(crate) type Recipes = PrimaryMap;
 #[derive(Clone)]
 pub(crate) struct EncodingRecipeBuilder {
     pub name: String,
-    format: InstructionFormatIndex,
+    format: Rc,
     pub base_size: u64,
     pub operands_in: Option>,
     pub operands_out: Option>,
@@ -179,10 +181,10 @@ pub(crate) struct EncodingRecipeBuilder {
 }
 
 impl EncodingRecipeBuilder {
-    pub fn new(name: impl Into, format: InstructionFormatIndex, base_size: u64) -> Self {
+    pub fn new(name: impl Into, format: &Rc, base_size: u64) -> Self {
         Self {
             name: name.into(),
-            format,
+            format: format.clone(),
             base_size,
             operands_in: None,
             operands_out: None,
@@ -250,18 +252,17 @@ impl EncodingRecipeBuilder {
         self
     }
 
-    pub fn build(self, formats: &FormatRegistry) -> EncodingRecipe {
+    pub fn build(self) -> EncodingRecipe {
         let operands_in = self.operands_in.unwrap_or(Vec::new());
         let operands_out = self.operands_out.unwrap_or(Vec::new());
 
         // The number of input constraints must match the number of format input operands.
-        if !formats.get(self.format).has_value_list {
-            let format = formats.get(self.format);
+        if !self.format.has_value_list {
             assert!(
-                operands_in.len() == format.num_value_operands,
+                operands_in.len() == self.format.num_value_operands,
                 format!(
                     "missing operand constraints for recipe {} (format {})",
-                    self.name, format.name
+                    self.name, self.format.name
                 )
             );
         }
diff --git a/cranelift/codegen/meta/src/gen_binemit.rs b/cranelift/codegen/meta/src/gen_binemit.rs
index fed145cbd4..3d7b08a227 100644
--- a/cranelift/codegen/meta/src/gen_binemit.rs
+++ b/cranelift/codegen/meta/src/gen_binemit.rs
@@ -5,7 +5,6 @@ use cranelift_entity::EntityRef;
 use crate::error;
 use crate::srcgen::Formatter;
 
-use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::recipes::{EncodingRecipe, OperandConstraint, Recipes};
 
 /// Generate code to handle a single recipe.
@@ -14,11 +13,12 @@ use crate::cdsl::recipes::{EncodingRecipe, OperandConstraint, Recipes};
 /// - Determine register locations for operands with register constraints.
 /// - Determine stack slot locations for operands with stack constraints.
 /// - Call hand-written code for the actual emission.
-fn gen_recipe(formats: &FormatRegistry, recipe: &EncodingRecipe, fmt: &mut Formatter) {
-    let inst_format = formats.get(recipe.format);
+fn gen_recipe(recipe: &EncodingRecipe, fmt: &mut Formatter) {
+    let inst_format = &recipe.format;
     let num_value_ops = inst_format.num_value_operands;
 
-    // TODO: Set want_args to true for only MultiAry instructions instead of all formats with value list.
+    // TODO: Set want_args to true for only MultiAry instructions instead of all formats with value
+    // list.
     let want_args = inst_format.has_value_list
         || recipe.operands_in.iter().any(|c| match c {
             OperandConstraint::RegClass(_) | OperandConstraint::Stack(_) => true,
@@ -148,7 +148,7 @@ fn unwrap_values(
     varlist
 }
 
-fn gen_isa(formats: &FormatRegistry, isa_name: &str, recipes: &Recipes, fmt: &mut Formatter) {
+fn gen_isa(isa_name: &str, recipes: &Recipes, fmt: &mut Formatter) {
     fmt.doc_comment(format!(
         "Emit binary machine code for `inst` for the {} ISA.",
         isa_name
@@ -193,7 +193,7 @@ fn gen_isa(formats: &FormatRegistry, isa_name: &str, recipes: &Recipes, fmt: &mu
                 fmt.comment(format!("Recipe {}", recipe.name));
                 fmtln!(fmt, "{} => {{", i.index());
                 fmt.indent(|fmt| {
-                    gen_recipe(formats, recipe, fmt);
+                    gen_recipe(recipe, fmt);
                 });
                 fmt.line("}");
             }
@@ -212,14 +212,13 @@ fn gen_isa(formats: &FormatRegistry, isa_name: &str, recipes: &Recipes, fmt: &mu
 }
 
 pub(crate) fn generate(
-    formats: &FormatRegistry,
     isa_name: &str,
     recipes: &Recipes,
     binemit_filename: &str,
     out_dir: &str,
 ) -> Result<(), error::Error> {
     let mut fmt = Formatter::new();
-    gen_isa(formats, isa_name, recipes, &mut fmt);
+    gen_isa(isa_name, recipes, &mut fmt);
     fmt.update_file(binemit_filename, out_dir)?;
     Ok(())
 }
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index 85a853150e..053eba2517 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -400,7 +400,7 @@ fn gen_bool_accessor bool>(
     fmt.empty_line();
 }
 
-fn gen_opcodes<'a>(all_inst: &AllInstructions, formats: &FormatRegistry, fmt: &mut Formatter) {
+fn gen_opcodes<'a>(all_inst: &AllInstructions, fmt: &mut Formatter) {
     fmt.doc_comment(
         r#"
         An instruction opcode.
@@ -418,13 +418,12 @@ fn gen_opcodes<'a>(all_inst: &AllInstructions, formats: &FormatRegistry, fmt: &m
     fmt.indent(|fmt| {
         let mut is_first_opcode = true;
         for inst in all_inst.values() {
-            let format = formats.get(inst.format);
-            fmt.doc_comment(format!("`{}`. ({})", inst, format.name));
+            fmt.doc_comment(format!("`{}`. ({})", inst, inst.format.name));
 
             // Document polymorphism.
             if let Some(poly) = &inst.polymorphic_info {
                 if poly.use_typevar_operand {
-                    let op_num = inst.value_opnums[format.typevar_operand.unwrap()];
+                    let op_num = inst.value_opnums[inst.format.typevar_operand.unwrap()];
                     fmt.doc_comment(format!(
                         "Type inferred from `{}`.",
                         inst.operands_in[op_num].name
@@ -537,8 +536,12 @@ fn gen_opcodes<'a>(all_inst: &AllInstructions, formats: &FormatRegistry, fmt: &m
     );
     fmt.indent(|fmt| {
         for inst in all_inst.values() {
-            let format = formats.get(inst.format);
-            fmtln!(fmt, "InstructionFormat::{}, // {}", format.name, inst.name);
+            fmtln!(
+                fmt,
+                "InstructionFormat::{}, // {}",
+                inst.format.name,
+                inst.name
+            );
         }
     });
     fmtln!(fmt, "];");
@@ -1055,7 +1058,7 @@ fn gen_builder(instructions: &AllInstructions, formats: &FormatRegistry, fmt: &m
     fmt.line("pub trait InstBuilder<'f>: InstBuilderBase<'f> {");
     fmt.indent(|fmt| {
         for inst in instructions.values() {
-            gen_inst_builder(inst, formats.get(inst.format), fmt);
+            gen_inst_builder(inst, &*inst.format, fmt);
         }
         for format in formats.iter() {
             gen_format_constructor(format, fmt);
@@ -1080,7 +1083,7 @@ pub(crate) fn generate(
     fmt.empty_line();
     gen_instruction_data_impl(format_registry, &mut fmt);
     fmt.empty_line();
-    gen_opcodes(all_inst, format_registry, &mut fmt);
+    gen_opcodes(all_inst, &mut fmt);
     gen_type_constraints(all_inst, &mut fmt);
     fmt.update_file(opcode_filename, out_dir)?;
 
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index f4033c64b5..c93262701d 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -1,5 +1,4 @@
 use crate::cdsl::ast::{Def, DefPool, Expr, VarPool};
-use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::operands::Operand;
 use crate::cdsl::type_inference::Constraint;
@@ -21,18 +20,14 @@ use std::iter::FromIterator;
 ///
 /// Also create a local variable named `predicate` with the value of the evaluated instruction
 /// predicate, or `true` if the node has no predicate.
-fn unwrap_inst(
-    transform: &Transform,
-    format_registry: &FormatRegistry,
-    fmt: &mut Formatter,
-) -> bool {
+fn unwrap_inst(transform: &Transform, fmt: &mut Formatter) -> bool {
     let var_pool = &transform.var_pool;
     let def_pool = &transform.def_pool;
 
     let def = def_pool.get(transform.src);
     let apply = &def.apply;
     let inst = &apply.inst;
-    let iform = format_registry.get(inst.format);
+    let iform = &inst.format;
 
     fmt.comment(format!(
         "Unwrap fields from instruction format {}",
@@ -438,7 +433,6 @@ fn emit_dst_inst(def: &Def, def_pool: &DefPool, var_pool: &VarPool, fmt: &mut Fo
 fn gen_transform<'a>(
     replace_inst: bool,
     transform: &'a Transform,
-    format_registry: &FormatRegistry,
     type_sets: &mut UniqueTable<'a, TypeSet>,
     fmt: &mut Formatter,
 ) {
@@ -446,7 +440,7 @@ fn gen_transform<'a>(
     let apply = &transform.def_pool.get(transform.src).apply;
 
     let inst_predicate = apply
-        .inst_predicate_with_ctrl_typevar(format_registry, &transform.var_pool)
+        .inst_predicate_with_ctrl_typevar(&transform.var_pool)
         .rust_predicate("pos.func");
 
     let has_extra_constraints = !transform.type_env.constraints.is_empty();
@@ -560,7 +554,6 @@ fn gen_transform<'a>(
 
 fn gen_transform_group<'a>(
     group: &'a TransformGroup,
-    format_registry: &FormatRegistry,
     transform_groups: &TransformGroups,
     type_sets: &mut UniqueTable<'a, TypeSet>,
     fmt: &mut Formatter,
@@ -610,14 +603,14 @@ fn gen_transform_group<'a>(
                         let transforms = inst_to_transforms.get(camel_name).unwrap();
 
                         // Unwrap the source instruction, create local variables for the input variables.
-                        let replace_inst = unwrap_inst(&transforms[0], format_registry, fmt);
+                        let replace_inst = unwrap_inst(&transforms[0], fmt);
                         fmt.empty_line();
 
                         for (i, transform) in transforms.into_iter().enumerate() {
                             if i > 0 {
                                 fmt.empty_line();
                             }
-                            gen_transform(replace_inst, transform, format_registry, type_sets, fmt);
+                            gen_transform(replace_inst, transform, type_sets, fmt);
                         }
                     });
                     fmtln!(fmt, "}");
@@ -665,7 +658,6 @@ fn gen_transform_group<'a>(
 /// Generate `TYPE_SETS` and `LEGALIZE_ACTIONS` tables.
 fn gen_isa(
     isa: &TargetIsa,
-    format_registry: &FormatRegistry,
     transform_groups: &TransformGroups,
     shared_group_names: &mut HashSet<&'static str>,
     fmt: &mut Formatter,
@@ -679,13 +671,7 @@ fn gen_isa(
                     isa_name == isa.name,
                     "ISA-specific legalizations must be used by the same ISA"
                 );
-                gen_transform_group(
-                    group,
-                    format_registry,
-                    transform_groups,
-                    &mut type_sets,
-                    fmt,
-                );
+                gen_transform_group(group, transform_groups, &mut type_sets, fmt);
             }
             None => {
                 shared_group_names.insert(group.name);
@@ -712,7 +698,6 @@ fn gen_isa(
 /// Generate the legalizer files.
 pub(crate) fn generate(
     isas: &Vec,
-    format_registry: &FormatRegistry,
     transform_groups: &TransformGroups,
     filename_prefix: &str,
     out_dir: &str,
@@ -721,13 +706,7 @@ pub(crate) fn generate(
 
     for isa in isas {
         let mut fmt = Formatter::new();
-        gen_isa(
-            isa,
-            format_registry,
-            transform_groups,
-            &mut shared_group_names,
-            &mut fmt,
-        );
+        gen_isa(isa, transform_groups, &mut shared_group_names, &mut fmt);
         fmt.update_file(format!("{}-{}.rs", filename_prefix, isa.name), out_dir)?;
     }
 
@@ -738,13 +717,7 @@ pub(crate) fn generate(
     sorted_shared_group_names.sort();
     for group_name in &sorted_shared_group_names {
         let group = transform_groups.by_name(group_name);
-        gen_transform_group(
-            group,
-            format_registry,
-            transform_groups,
-            &mut type_sets,
-            &mut fmt,
-        );
+        gen_transform_group(group, transform_groups, &mut type_sets, &mut fmt);
     }
     gen_typesets_table(&type_sets, &mut fmt);
     fmt.update_file(format!("{}r.rs", filename_prefix), out_dir)?;
diff --git a/cranelift/codegen/meta/src/isa/riscv/encodings.rs b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
index 21ad3c469c..936579b2bc 100644
--- a/cranelift/codegen/meta/src/isa/riscv/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
@@ -13,24 +13,21 @@ use crate::shared::types::Reference::{R32, R64};
 use crate::shared::Definitions as SharedDefinitions;
 
 use super::recipes::RecipeGroup;
-use crate::cdsl::formats::FormatRegistry;
 
 pub(crate) struct PerCpuModeEncodings<'defs> {
     pub inst_pred_reg: InstructionPredicateRegistry,
     pub enc32: Vec,
     pub enc64: Vec,
     recipes: &'defs Recipes,
-    formats: &'defs FormatRegistry,
 }
 
 impl<'defs> PerCpuModeEncodings<'defs> {
-    fn new(recipes: &'defs Recipes, formats: &'defs FormatRegistry) -> Self {
+    fn new(recipes: &'defs Recipes) -> Self {
         Self {
             inst_pred_reg: InstructionPredicateRegistry::new(),
             enc32: Vec::new(),
             enc64: Vec::new(),
             recipes,
-            formats,
         }
     }
     fn enc(
@@ -39,7 +36,7 @@ impl<'defs> PerCpuModeEncodings<'defs> {
         recipe: EncodingRecipeNumber,
         bits: u16,
     ) -> EncodingBuilder {
-        EncodingBuilder::new(inst.into(), recipe, bits, self.formats)
+        EncodingBuilder::new(inst.into(), recipe, bits)
     }
     fn add32(&mut self, encoding: EncodingBuilder) {
         self.enc32
@@ -176,7 +173,7 @@ pub(crate) fn define<'defs>(
     let use_m = isa_settings.predicate_by_name("use_m");
 
     // Definitions.
-    let mut e = PerCpuModeEncodings::new(&recipes.recipes, &shared_defs.format_registry);
+    let mut e = PerCpuModeEncodings::new(&recipes.recipes);
 
     // Basic arithmetic binary instructions are encoded in an R-type instruction.
     for &(inst, inst_imm, f3, f7) in &[
@@ -242,7 +239,7 @@ pub(crate) fn define<'defs>(
                 bound_inst.clone().into(),
                 vec![Expr::Literal(cc), Expr::Var(x), Expr::Var(y)],
             )
-            .inst_predicate(&shared_defs.format_registry, &var_pool)
+            .inst_predicate(&var_pool)
             .unwrap()
         };
 
@@ -339,7 +336,7 @@ pub(crate) fn define<'defs>(
                     Expr::Var(args),
                 ],
             )
-            .inst_predicate(&shared_defs.format_registry, &var_pool)
+            .inst_predicate(&var_pool)
             .unwrap()
         };
 
diff --git a/cranelift/codegen/meta/src/isa/riscv/recipes.rs b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
index b1297479ec..ab70595b34 100644
--- a/cranelift/codegen/meta/src/isa/riscv/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
@@ -1,16 +1,12 @@
 use std::collections::HashMap;
 
-use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::instructions::InstructionPredicate;
 use crate::cdsl::recipes::{EncodingRecipeBuilder, EncodingRecipeNumber, Recipes, Stack};
 use crate::cdsl::regs::IsaRegs;
 use crate::shared::Definitions as SharedDefinitions;
 
 /// An helper to create recipes and use them when defining the RISCV encodings.
-pub(crate) struct RecipeGroup<'formats> {
-    /// Memoized format registry, to pass it to the builders.
-    formats: &'formats FormatRegistry,
-
+pub(crate) struct RecipeGroup {
     /// The actualy list of recipes explicitly created in this file.
     pub recipes: Recipes,
 
@@ -18,10 +14,9 @@ pub(crate) struct RecipeGroup<'formats> {
     name_to_recipe: HashMap,
 }
 
-impl<'formats> RecipeGroup<'formats> {
-    fn new(formats: &'formats FormatRegistry) -> Self {
+impl RecipeGroup {
+    fn new() -> Self {
         Self {
-            formats,
             recipes: Recipes::new(),
             name_to_recipe: HashMap::new(),
         }
@@ -33,7 +28,7 @@ impl<'formats> RecipeGroup<'formats> {
             format!("riscv recipe '{}' created twice", builder.name)
         );
         let name = builder.name.clone();
-        let number = self.recipes.push(builder.build(self.formats));
+        let number = self.recipes.push(builder.build());
         self.name_to_recipe.insert(name, number);
     }
 
@@ -50,13 +45,10 @@ impl<'formats> RecipeGroup<'formats> {
     }
 }
 
-pub(crate) fn define<'formats>(
-    shared_defs: &'formats SharedDefinitions,
-    regs: &IsaRegs,
-) -> RecipeGroup<'formats> {
+pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeGroup {
+    // Format shorthands.
     let formats = &shared_defs.format_registry;
 
-    // Format shorthands.
     let f_binary = formats.by_name("Binary");
     let f_binary_imm = formats.by_name("BinaryImm");
     let f_branch = formats.by_name("Branch");
@@ -76,7 +68,7 @@ pub(crate) fn define<'formats>(
     let gpr = regs.class_by_name("GPR");
 
     // Definitions.
-    let mut recipes = RecipeGroup::new(&shared_defs.format_registry);
+    let mut recipes = RecipeGroup::new();
 
     // R-type 32-bit instructions: These are mostly binary arithmetic instructions.
     // The encbits are `opcode[6:2] | (funct3 << 5) | (funct7 << 8)
@@ -103,36 +95,42 @@ pub(crate) fn define<'formats>(
             .emit("put_r(bits, in_reg0, in_reg1, out_reg0, sink);"),
     );
 
-    let format = formats.get(f_binary_imm);
     recipes.push(
         EncodingRecipeBuilder::new("Ii", f_binary_imm, 4)
             .operands_in(vec![gpr])
             .operands_out(vec![gpr])
             .inst_predicate(InstructionPredicate::new_is_signed_int(
-                format, "imm", 12, 0,
+                &*f_binary_imm,
+                "imm",
+                12,
+                0,
             ))
             .emit("put_i(bits, in_reg0, imm.into(), out_reg0, sink);"),
     );
 
     // I-type instruction with a hardcoded %x0 rs1.
-    let format = formats.get(f_unary_imm);
     recipes.push(
         EncodingRecipeBuilder::new("Iz", f_unary_imm, 4)
             .operands_out(vec![gpr])
             .inst_predicate(InstructionPredicate::new_is_signed_int(
-                format, "imm", 12, 0,
+                &*f_unary_imm,
+                "imm",
+                12,
+                0,
             ))
             .emit("put_i(bits, 0, imm.into(), out_reg0, sink);"),
     );
 
     // I-type encoding of an integer comparison.
-    let format = formats.get(f_int_compare_imm);
     recipes.push(
         EncodingRecipeBuilder::new("Iicmp", f_int_compare_imm, 4)
             .operands_in(vec![gpr])
             .operands_out(vec![gpr])
             .inst_predicate(InstructionPredicate::new_is_signed_int(
-                format, "imm", 12, 0,
+                &*f_int_compare_imm,
+                "imm",
+                12,
+                0,
             ))
             .emit("put_i(bits, in_reg0, imm.into(), out_reg0, sink);"),
     );
@@ -195,12 +193,14 @@ pub(crate) fn define<'formats>(
     );
 
     // U-type instructions have a 20-bit immediate that targets bits 12-31.
-    let format = formats.get(f_unary_imm);
     recipes.push(
         EncodingRecipeBuilder::new("U", f_unary_imm, 4)
             .operands_out(vec![gpr])
             .inst_predicate(InstructionPredicate::new_is_signed_int(
-                format, "imm", 32, 12,
+                &*f_unary_imm,
+                "imm",
+                32,
+                12,
             ))
             .emit("put_u(bits, imm.into(), out_reg0, sink);"),
     );
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index f957dd7791..a9a6f700ac 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -20,27 +20,24 @@ use crate::shared::Definitions as SharedDefinitions;
 use crate::isa::x86::opcodes::*;
 
 use super::recipes::{RecipeGroup, Template};
-use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::instructions::BindParameter::Any;
 
-pub(crate) struct PerCpuModeEncodings<'defs> {
+pub(crate) struct PerCpuModeEncodings {
     pub enc32: Vec,
     pub enc64: Vec,
     pub recipes: Recipes,
     recipes_by_name: HashMap,
     pub inst_pred_reg: InstructionPredicateRegistry,
-    formats: &'defs FormatRegistry,
 }
 
-impl<'defs> PerCpuModeEncodings<'defs> {
-    fn new(formats: &'defs FormatRegistry) -> Self {
+impl PerCpuModeEncodings {
+    fn new() -> Self {
         Self {
             enc32: Vec::new(),
             enc64: Vec::new(),
             recipes: Recipes::new(),
             recipes_by_name: HashMap::new(),
             inst_pred_reg: InstructionPredicateRegistry::new(),
-            formats,
         }
     }
 
@@ -73,7 +70,7 @@ impl<'defs> PerCpuModeEncodings<'defs> {
     {
         let (recipe, bits) = template.build();
         let recipe_number = self.add_recipe(recipe);
-        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits, self.formats);
+        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits);
         builder_closure(builder).build(&self.recipes, &mut self.inst_pred_reg)
     }
 
@@ -105,7 +102,7 @@ impl<'defs> PerCpuModeEncodings<'defs> {
     }
     fn enc32_rec(&mut self, inst: impl Into, recipe: &EncodingRecipe, bits: u16) {
         let recipe_number = self.add_recipe(recipe.clone());
-        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits, self.formats);
+        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits);
         let encoding = builder.build(&self.recipes, &mut self.inst_pred_reg);
         self.enc32.push(encoding);
     }
@@ -138,7 +135,7 @@ impl<'defs> PerCpuModeEncodings<'defs> {
     }
     fn enc64_rec(&mut self, inst: impl Into, recipe: &EncodingRecipe, bits: u16) {
         let recipe_number = self.add_recipe(recipe.clone());
-        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits, self.formats);
+        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits);
         let encoding = builder.build(&self.recipes, &mut self.inst_pred_reg);
         self.enc64.push(encoding);
     }
@@ -370,12 +367,12 @@ impl<'defs> PerCpuModeEncodings<'defs> {
 
 // Definitions.
 
-pub(crate) fn define<'defs>(
-    shared_defs: &'defs SharedDefinitions,
+pub(crate) fn define(
+    shared_defs: &SharedDefinitions,
     settings: &SettingGroup,
     x86: &InstructionGroup,
     r: &RecipeGroup,
-) -> PerCpuModeEncodings<'defs> {
+) -> PerCpuModeEncodings {
     let shared = &shared_defs.instructions;
     let formats = &shared_defs.format_registry;
 
@@ -688,7 +685,7 @@ pub(crate) fn define<'defs>(
     let use_sse41_simd = settings.predicate_by_name("use_sse41_simd");
 
     // Definitions.
-    let mut e = PerCpuModeEncodings::new(formats);
+    let mut e = PerCpuModeEncodings::new();
 
     // The pinned reg is fixed to a certain value entirely user-controlled, so it generates nothing!
     e.enc64_rec(get_pinned_reg.bind(I64), rec_get_pinned_reg, 0);
@@ -777,8 +774,8 @@ pub(crate) fn define<'defs>(
     e.enc64(iconst.bind(I32), rec_pu_id.opcodes(&MOV_IMM));
 
     // The 32-bit immediate movl also zero-extends to 64 bits.
-    let f_unary_imm = formats.get(formats.by_name("UnaryImm"));
-    let is_unsigned_int32 = InstructionPredicate::new_is_unsigned_int(f_unary_imm, "imm", 32, 0);
+    let f_unary_imm = formats.by_name("UnaryImm");
+    let is_unsigned_int32 = InstructionPredicate::new_is_unsigned_int(&*f_unary_imm, "imm", 32, 0);
 
     e.enc64_func(
         iconst.bind(I64),
@@ -883,8 +880,8 @@ pub(crate) fn define<'defs>(
     e.enc64_isap(ctz.bind(I32), rec_urm.opcodes(&TZCNT), use_bmi1);
 
     // Loads and stores.
-    let f_load_complex = formats.get(formats.by_name("LoadComplex"));
-    let is_load_complex_length_two = InstructionPredicate::new_length_equals(f_load_complex, 2);
+    let f_load_complex = formats.by_name("LoadComplex");
+    let is_load_complex_length_two = InstructionPredicate::new_length_equals(&*f_load_complex, 2);
 
     for recipe in &[rec_ldWithIndex, rec_ldWithIndexDisp8, rec_ldWithIndexDisp32] {
         e.enc_i32_i64_instp(
@@ -928,8 +925,9 @@ pub(crate) fn define<'defs>(
         );
     }
 
-    let f_store_complex = formats.get(formats.by_name("StoreComplex"));
-    let is_store_complex_length_three = InstructionPredicate::new_length_equals(f_store_complex, 3);
+    let f_store_complex = formats.by_name("StoreComplex");
+    let is_store_complex_length_three =
+        InstructionPredicate::new_length_equals(&*f_store_complex, 3);
 
     for recipe in &[rec_stWithIndex, rec_stWithIndexDisp8, rec_stWithIndexDisp32] {
         e.enc_i32_i64_instp(
@@ -1235,8 +1233,8 @@ pub(crate) fn define<'defs>(
     );
 
     // 64-bit, colocated, both PIC and non-PIC. Use the lea instruction's pc-relative field.
-    let f_func_addr = formats.get(formats.by_name("FuncAddr"));
-    let is_colocated_func = InstructionPredicate::new_is_colocated_func(f_func_addr, "func_ref");
+    let f_func_addr = formats.by_name("FuncAddr");
+    let is_colocated_func = InstructionPredicate::new_is_colocated_func(&*f_func_addr, "func_ref");
     e.enc64_instp(
         func_addr.bind(I64),
         rec_pcrel_fnaddr8.opcodes(&LEA).rex().w(),
@@ -1295,8 +1293,8 @@ pub(crate) fn define<'defs>(
     e.enc32(call, rec_call_id.opcodes(&CALL_RELATIVE));
 
     // 64-bit, colocated, both PIC and non-PIC. Use the call instruction's pc-relative field.
-    let f_call = formats.get(formats.by_name("Call"));
-    let is_colocated_func = InstructionPredicate::new_is_colocated_func(f_call, "func_ref");
+    let f_call = formats.by_name("Call");
+    let is_colocated_func = InstructionPredicate::new_is_colocated_func(&*f_call, "func_ref");
     e.enc64_instp(call, rec_call_id.opcodes(&CALL_RELATIVE), is_colocated_func);
 
     // 64-bit, non-colocated, PIC. There is no 64-bit non-colocated non-PIC version, since non-PIC
@@ -1566,16 +1564,18 @@ pub(crate) fn define<'defs>(
 
     // Floating-point constants equal to 0.0 can be encoded using either `xorps` or `xorpd`, for
     // 32-bit and 64-bit floats respectively.
-    let f_unary_ieee32 = formats.get(formats.by_name("UnaryIeee32"));
-    let is_zero_32_bit_float = InstructionPredicate::new_is_zero_32bit_float(f_unary_ieee32, "imm");
+    let f_unary_ieee32 = formats.by_name("UnaryIeee32");
+    let is_zero_32_bit_float =
+        InstructionPredicate::new_is_zero_32bit_float(&*f_unary_ieee32, "imm");
     e.enc32_instp(
         f32const,
         rec_f32imm_z.opcodes(&XORPS),
         is_zero_32_bit_float.clone(),
     );
 
-    let f_unary_ieee64 = formats.get(formats.by_name("UnaryIeee64"));
-    let is_zero_64_bit_float = InstructionPredicate::new_is_zero_64bit_float(f_unary_ieee64, "imm");
+    let f_unary_ieee64 = formats.by_name("UnaryIeee64");
+    let is_zero_64_bit_float =
+        InstructionPredicate::new_is_zero_64bit_float(&*f_unary_ieee64, "imm");
     e.enc32_instp(
         f64const,
         rec_f64imm_z.opcodes(&XORPD),
@@ -1847,18 +1847,18 @@ pub(crate) fn define<'defs>(
     // this must be encoded prior to the MOVUPS implementation (below) so the compiler sees this
     // encoding first
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
-        let f_unary_const = formats.get(formats.by_name("UnaryConst"));
+        let f_unary_const = formats.by_name("UnaryConst");
         let instruction = vconst.bind(vector(ty, sse_vector_size));
 
         let is_zero_128bit =
-            InstructionPredicate::new_is_all_zeroes(f_unary_const, "constant_handle");
+            InstructionPredicate::new_is_all_zeroes(&*f_unary_const, "constant_handle");
         let template = rec_vconst_optimized.nonrex().opcodes(&PXOR);
         e.enc_32_64_func(instruction.clone(), template, |builder| {
             builder.inst_predicate(is_zero_128bit)
         });
 
         let is_ones_128bit =
-            InstructionPredicate::new_is_all_ones(f_unary_const, "constant_handle");
+            InstructionPredicate::new_is_all_ones(&*f_unary_const, "constant_handle");
         let template = rec_vconst_optimized.nonrex().opcodes(&PCMPEQB);
         e.enc_32_64_func(instruction, template, |builder| {
             builder.inst_predicate(is_ones_128bit)
@@ -2038,9 +2038,9 @@ pub(crate) fn define<'defs>(
         };
 
         let instruction = icmp.bind(vector(ty, sse_vector_size));
-        let f_int_compare = formats.get(formats.by_name("IntCompare"));
+        let f_int_compare = formats.by_name("IntCompare");
         let has_eq_condition_code =
-            InstructionPredicate::new_has_condition_code(f_int_compare, IntCC::Equal, "cond");
+            InstructionPredicate::new_has_condition_code(&*f_int_compare, IntCC::Equal, "cond");
         let template = rec_icscc_fpr.nonrex().opcodes(opcodes);
         e.enc_32_64_func(instruction, template, |builder| {
             let builder = builder.inst_predicate(has_eq_condition_code);
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 8063e9441f..bf91a6326c 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -1,7 +1,7 @@
 use std::rc::Rc;
 
 use crate::cdsl::ast::Literal;
-use crate::cdsl::formats::{FormatRegistry, InstructionFormat};
+use crate::cdsl::formats::InstructionFormat;
 use crate::cdsl::instructions::InstructionPredicate;
 use crate::cdsl::recipes::{
     EncodingRecipe, EncodingRecipeBuilder, OperandConstraint, Register, Stack,
@@ -16,9 +16,6 @@ use crate::isa::x86::opcodes;
 /// It contains all the recipes and recipe templates that might be used in the encodings crate of
 /// this same directory.
 pub(crate) struct RecipeGroup<'builder> {
-    /// Memoized format pointer, to pass it to builders later.
-    formats: &'builder FormatRegistry,
-
     /// Memoized registers description, to pass it to builders later.
     regs: &'builder IsaRegs,
 
@@ -31,19 +28,18 @@ pub(crate) struct RecipeGroup<'builder> {
 }
 
 impl<'builder> RecipeGroup<'builder> {
-    fn new(formats: &'builder FormatRegistry, regs: &'builder IsaRegs) -> Self {
+    fn new(regs: &'builder IsaRegs) -> Self {
         Self {
-            formats,
             regs,
             recipes: Vec::new(),
             templates: Vec::new(),
         }
     }
     fn add_recipe(&mut self, recipe: EncodingRecipeBuilder) {
-        self.recipes.push(recipe.build(self.formats));
+        self.recipes.push(recipe.build());
     }
     fn add_template_recipe(&mut self, recipe: EncodingRecipeBuilder) -> Rc> {
-        let template = Rc::new(Template::new(recipe, self.formats, self.regs));
+        let template = Rc::new(Template::new(recipe, self.regs));
         self.templates.push(template.clone());
         template
     }
@@ -165,9 +161,6 @@ fn replace_nonrex_constraints(
 /// reconsidered later.
 #[derive(Clone)]
 pub(crate) struct Template<'builder> {
-    /// Mapping of format indexes to format data, used in the build() method.
-    formats: &'builder FormatRegistry,
-
     /// Description of registers, used in the build() method.
     regs: &'builder IsaRegs,
 
@@ -192,13 +185,8 @@ pub(crate) struct Template<'builder> {
 }
 
 impl<'builder> Template<'builder> {
-    fn new(
-        recipe: EncodingRecipeBuilder,
-        formats: &'builder FormatRegistry,
-        regs: &'builder IsaRegs,
-    ) -> Self {
+    fn new(recipe: EncodingRecipeBuilder, regs: &'builder IsaRegs) -> Self {
         Self {
-            formats,
             regs,
             recipe,
             requires_prefix: false,
@@ -293,7 +281,7 @@ impl<'builder> Template<'builder> {
             self.recipe.operands_out = Some(replace_nonrex_constraints(self.regs, operands_out));
         }
 
-        (self.recipe.build(self.formats), bits)
+        (self.recipe.build(), bits)
     }
 }
 
@@ -340,8 +328,6 @@ pub(crate) fn define<'shared>(
         .map(|name| Literal::enumerator_for(floatcc, name))
         .collect();
 
-    let formats = &shared_defs.format_registry;
-
     // Register classes shorthands.
     let abcd = regs.class_by_name("ABCD");
     let gpr = regs.class_by_name("GPR");
@@ -360,6 +346,8 @@ pub(crate) fn define<'shared>(
     let stack_fpr32 = Stack::new(fpr);
 
     // Format shorthands, prefixed with f_.
+    let formats = &shared_defs.format_registry;
+
     let f_binary = formats.by_name("Binary");
     let f_binary_imm = formats.by_name("BinaryImm");
     let f_branch = formats.by_name("Branch");
@@ -408,7 +396,7 @@ pub(crate) fn define<'shared>(
     let use_sse41 = settings.predicate_by_name("use_sse41");
 
     // Definitions.
-    let mut recipes = RecipeGroup::new(formats, regs);
+    let mut recipes = RecipeGroup::new(regs);
 
     // A null unary instruction that takes a GPR register. Can be used for identity copies and
     // no-op conversions.
@@ -501,7 +489,7 @@ pub(crate) fn define<'shared>(
             .clobbers_flags(false)
             .inst_predicate(supported_floatccs_predicate(
                 &supported_floatccs,
-                formats.get(f_float_cond_trap),
+                &*f_float_cond_trap,
             ))
             .emit(
                 r#"
@@ -570,13 +558,15 @@ pub(crate) fn define<'shared>(
 
     // XX /r with FPR ins and outs. A form with a byte immediate.
     {
-        let format = formats.get(f_insert_lane);
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("fa_ib", f_insert_lane, 2)
                 .operands_in(vec![fpr, fpr])
                 .operands_out(vec![0])
                 .inst_predicate(InstructionPredicate::new_is_unsigned_int(
-                    format, "lane", 8, 0,
+                    &*f_insert_lane,
+                    "lane",
+                    8,
+                    0,
                 ))
                 .emit(
                     r#"
@@ -686,7 +676,6 @@ pub(crate) fn define<'shared>(
                     modrm_rr(in_reg0, out_reg0, sink);
                 "#,
                 ),
-            formats,
             regs,
         )
         .when_prefixed(urm_noflags),
@@ -850,12 +839,16 @@ pub(crate) fn define<'shared>(
 
     // XX /n ib with 8-bit immediate sign-extended.
     {
-        let format = formats.get(f_binary_imm);
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("r_ib", f_binary_imm, 2)
                 .operands_in(vec![gpr])
                 .operands_out(vec![0])
-                .inst_predicate(InstructionPredicate::new_is_signed_int(format, "imm", 8, 0))
+                .inst_predicate(InstructionPredicate::new_is_signed_int(
+                    &*f_binary_imm,
+                    "imm",
+                    8,
+                    0,
+                ))
                 .emit(
                     r#"
                         {{PUT_OP}}(bits, rex1(in_reg0), sink);
@@ -872,7 +865,10 @@ pub(crate) fn define<'shared>(
                 .operands_in(vec![gpr])
                 .operands_out(vec![0])
                 .inst_predicate(InstructionPredicate::new_is_signed_int(
-                    format, "imm", 32, 0,
+                    &*f_binary_imm,
+                    "imm",
+                    32,
+                    0,
                 ))
                 .emit(
                     r#"
@@ -887,13 +883,15 @@ pub(crate) fn define<'shared>(
 
     // XX /r ib with 8-bit unsigned immediate (e.g. for pshufd)
     {
-        let format = formats.get(f_extract_lane);
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("r_ib_unsigned_fpr", f_extract_lane, 2)
                 .operands_in(vec![fpr])
                 .operands_out(vec![fpr])
                 .inst_predicate(InstructionPredicate::new_is_unsigned_int(
-                    format, "lane", 8, 0,
+                    &*f_extract_lane,
+                    "lane",
+                    8,
+                    0,
                 )) // TODO if the format name is changed then "lane" should be renamed to something more appropriate--ordering mask? broadcast immediate?
                 .emit(
                     r#"
@@ -908,13 +906,12 @@ pub(crate) fn define<'shared>(
 
     // XX /r ib with 8-bit unsigned immediate (e.g. for extractlane)
     {
-        let format = formats.get(f_extract_lane);
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("r_ib_unsigned_gpr", f_extract_lane, 2)
                 .operands_in(vec![fpr])
                 .operands_out(vec![gpr])
                 .inst_predicate(InstructionPredicate::new_is_unsigned_int(
-                    format, "lane", 8, 0,
+                    &*f_extract_lane, "lane", 8, 0,
                 ))
                 .emit(
                     r#"
@@ -929,13 +926,15 @@ pub(crate) fn define<'shared>(
 
     // XX /r ib with 8-bit unsigned immediate (e.g. for insertlane)
     {
-        let format = formats.get(f_insert_lane);
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("r_ib_unsigned_r", f_insert_lane, 2)
                 .operands_in(vec![fpr, gpr])
                 .operands_out(vec![0])
                 .inst_predicate(InstructionPredicate::new_is_unsigned_int(
-                    format, "lane", 8, 0,
+                    &*f_insert_lane,
+                    "lane",
+                    8,
+                    0,
                 ))
                 .emit(
                     r#"
@@ -950,12 +949,14 @@ pub(crate) fn define<'shared>(
 
     {
         // XX /n id with 32-bit immediate sign-extended. UnaryImm version.
-        let format = formats.get(f_unary_imm);
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("u_id", f_unary_imm, 5)
                 .operands_out(vec![gpr])
                 .inst_predicate(InstructionPredicate::new_is_signed_int(
-                    format, "imm", 32, 0,
+                    &*f_unary_imm,
+                    "imm",
+                    32,
+                    0,
                 ))
                 .emit(
                     r#"
@@ -1039,11 +1040,13 @@ pub(crate) fn define<'shared>(
 
     // XX /n Unary with floating point 32-bit immediate equal to zero.
     {
-        let format = formats.get(f_unary_ieee32);
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("f32imm_z", f_unary_ieee32, 1)
                 .operands_out(vec![fpr])
-                .inst_predicate(InstructionPredicate::new_is_zero_32bit_float(format, "imm"))
+                .inst_predicate(InstructionPredicate::new_is_zero_32bit_float(
+                    &*f_unary_ieee32,
+                    "imm",
+                ))
                 .emit(
                     r#"
                         {{PUT_OP}}(bits, rex2(out_reg0, out_reg0), sink);
@@ -1055,11 +1058,13 @@ pub(crate) fn define<'shared>(
 
     // XX /n Unary with floating point 64-bit immediate equal to zero.
     {
-        let format = formats.get(f_unary_ieee64);
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("f64imm_z", f_unary_ieee64, 1)
                 .operands_out(vec![fpr])
-                .inst_predicate(InstructionPredicate::new_is_zero_64bit_float(format, "imm"))
+                .inst_predicate(InstructionPredicate::new_is_zero_64bit_float(
+                    &*f_unary_ieee64,
+                    "imm",
+                ))
                 .emit(
                     r#"
                         {{PUT_OP}}(bits, rex2(out_reg0, out_reg0), sink);
@@ -1114,10 +1119,14 @@ pub(crate) fn define<'shared>(
     );
 
     {
-        let format = formats.get(f_unary_imm);
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("adjustsp_ib", f_unary_imm, 2)
-                .inst_predicate(InstructionPredicate::new_is_signed_int(format, "imm", 8, 0))
+                .inst_predicate(InstructionPredicate::new_is_signed_int(
+                    &*f_unary_imm,
+                    "imm",
+                    8,
+                    0,
+                ))
                 .emit(
                     r#"
                         {{PUT_OP}}(bits, rex1(RU::rsp.into()), sink);
@@ -1131,7 +1140,10 @@ pub(crate) fn define<'shared>(
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("adjustsp_id", f_unary_imm, 5)
                 .inst_predicate(InstructionPredicate::new_is_signed_int(
-                    format, "imm", 32, 0,
+                    &*f_unary_imm,
+                    "imm",
+                    32,
+                    0,
                 ))
                 .emit(
                     r#"
@@ -1350,10 +1362,10 @@ pub(crate) fn define<'shared>(
 
     {
         // Simple stores.
-        let format = formats.get(f_store);
 
         // A predicate asking if the offset is zero.
-        let has_no_offset = InstructionPredicate::new_is_field_equal(format, "offset", "0".into());
+        let has_no_offset =
+            InstructionPredicate::new_is_field_equal(&*f_store, "offset", "0".into());
 
         // XX /r register-indirect store with no offset.
         let st = recipes.add_template_recipe(
@@ -1407,7 +1419,6 @@ pub(crate) fn define<'shared>(
                         }
                     "#,
                     ),
-                formats,
                 regs,
             )
             .when_prefixed(st),
@@ -1439,7 +1450,7 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let has_small_offset = InstructionPredicate::new_is_signed_int(format, "offset", 8, 0);
+        let has_small_offset = InstructionPredicate::new_is_signed_int(&*f_store, "offset", 8, 0);
 
         // XX /r register-indirect store with 8-bit offset.
         let st_disp8 = recipes.add_template_recipe(
@@ -1491,7 +1502,6 @@ pub(crate) fn define<'shared>(
                         sink.put1(offset as u8);
                     "#,
                     ),
-                formats,
                 regs,
             )
             .when_prefixed(st_disp8),
@@ -1570,7 +1580,6 @@ pub(crate) fn define<'shared>(
                         sink.put4(offset as u32);
                     "#,
                     ),
-                formats,
                 regs,
             )
             .when_prefixed(st_disp32),
@@ -1603,10 +1612,10 @@ pub(crate) fn define<'shared>(
 
     {
         // Complex stores.
-        let format = formats.get(f_store_complex);
 
         // A predicate asking if the offset is zero.
-        let has_no_offset = InstructionPredicate::new_is_field_equal(format, "offset", "0".into());
+        let has_no_offset =
+            InstructionPredicate::new_is_field_equal(&*f_store_complex, "offset", "0".into());
 
         // XX /r register-indirect store with index and no offset.
         recipes.add_template_recipe(
@@ -1687,7 +1696,8 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let has_small_offset = InstructionPredicate::new_is_signed_int(format, "offset", 8, 0);
+        let has_small_offset =
+            InstructionPredicate::new_is_signed_int(&*f_store_complex, "offset", 8, 0);
 
         // XX /r register-indirect store with index and 8-bit offset.
         recipes.add_template_recipe(
@@ -1750,7 +1760,8 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let has_big_offset = InstructionPredicate::new_is_signed_int(format, "offset", 32, 0);
+        let has_big_offset =
+            InstructionPredicate::new_is_signed_int(&*f_store_complex, "offset", 32, 0);
 
         // XX /r register-indirect store with index and 32-bit offset.
         recipes.add_template_recipe(
@@ -1890,10 +1901,10 @@ pub(crate) fn define<'shared>(
 
     {
         // Simple loads.
-        let format = formats.get(f_load);
 
         // A predicate asking if the offset is zero.
-        let has_no_offset = InstructionPredicate::new_is_field_equal(format, "offset", "0".into());
+        let has_no_offset =
+            InstructionPredicate::new_is_field_equal(&*f_load, "offset", "0".into());
 
         // XX /r load with no offset.
         recipes.add_template_recipe(
@@ -1949,7 +1960,7 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let has_small_offset = InstructionPredicate::new_is_signed_int(format, "offset", 8, 0);
+        let has_small_offset = InstructionPredicate::new_is_signed_int(&*f_load, "offset", 8, 0);
 
         // XX /r load with 8-bit offset.
         recipes.add_template_recipe(
@@ -2003,7 +2014,7 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let has_big_offset = InstructionPredicate::new_is_signed_int(format, "offset", 32, 0);
+        let has_big_offset = InstructionPredicate::new_is_signed_int(&*f_load, "offset", 32, 0);
 
         // XX /r load with 32-bit offset.
         recipes.add_template_recipe(
@@ -2060,10 +2071,10 @@ pub(crate) fn define<'shared>(
 
     {
         // Complex loads.
-        let format = formats.get(f_load_complex);
 
         // A predicate asking if the offset is zero.
-        let has_no_offset = InstructionPredicate::new_is_field_equal(format, "offset", "0".into());
+        let has_no_offset =
+            InstructionPredicate::new_is_field_equal(&*f_load_complex, "offset", "0".into());
 
         // XX /r load with index and no offset.
         recipes.add_template_recipe(
@@ -2119,7 +2130,8 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let has_small_offset = InstructionPredicate::new_is_signed_int(format, "offset", 8, 0);
+        let has_small_offset =
+            InstructionPredicate::new_is_signed_int(&*f_load_complex, "offset", 8, 0);
 
         // XX /r load with index and 8-bit offset.
         recipes.add_template_recipe(
@@ -2163,7 +2175,8 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let has_big_offset = InstructionPredicate::new_is_signed_int(format, "offset", 32, 0);
+        let has_big_offset =
+            InstructionPredicate::new_is_signed_int(&*f_load_complex, "offset", 32, 0);
 
         // XX /r load with index and 32-bit offset.
         recipes.add_template_recipe(
@@ -2377,7 +2390,7 @@ pub(crate) fn define<'shared>(
             .clobbers_flags(false)
             .inst_predicate(supported_floatccs_predicate(
                 &supported_floatccs,
-                formats.get(f_branch_float),
+                &*f_branch_float,
             ))
             .emit(
                 r#"
@@ -2394,7 +2407,7 @@ pub(crate) fn define<'shared>(
             .clobbers_flags(false)
             .inst_predicate(supported_floatccs_predicate(
                 &supported_floatccs,
-                formats.get(f_branch_float),
+                &*f_branch_float,
             ))
             .emit(
                 r#"
@@ -2421,7 +2434,7 @@ pub(crate) fn define<'shared>(
             .operands_in(vec![gpr, gpr])
             .operands_out(vec![gpr])
             .clobbers_flags(false)
-            .inst_predicate(valid_scale(formats.get(f_branch_table_entry)))
+            .inst_predicate(valid_scale(&*f_branch_table_entry))
             .compute_size("size_plus_maybe_offset_for_in_reg_1")
             .emit(
                 r#"
@@ -2499,7 +2512,6 @@ pub(crate) fn define<'shared>(
                     modrm_r_bits(out_reg0, bits, sink);
                 "#,
                 ),
-            formats,
             regs,
         )
         .requires_prefix(true),
@@ -2517,7 +2529,6 @@ pub(crate) fn define<'shared>(
                     modrm_r_bits(out_reg0, bits, sink);
                 "#,
                 ),
-            formats,
             regs,
         )
         .when_prefixed(seti),
@@ -2535,7 +2546,6 @@ pub(crate) fn define<'shared>(
                     modrm_r_bits(out_reg0, bits, sink);
                 "#,
                 ),
-            formats,
             regs,
         )
         .requires_prefix(true),
@@ -2553,7 +2563,6 @@ pub(crate) fn define<'shared>(
                     modrm_r_bits(out_reg0, bits, sink);
                 "#,
                 ),
-            formats,
             regs,
         )
         .when_prefixed(setf),
@@ -2695,9 +2704,7 @@ pub(crate) fn define<'shared>(
     );
 
     {
-        let format = formats.get(f_binary_imm);
-
-        let has_small_offset = InstructionPredicate::new_is_signed_int(format, "imm", 8, 0);
+        let has_small_offset = InstructionPredicate::new_is_signed_int(&*f_binary_imm, "imm", 8, 0);
 
         // XX /n, MI form with imm8.
         recipes.add_template_recipe(
@@ -2715,7 +2722,7 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let has_big_offset = InstructionPredicate::new_is_signed_int(format, "imm", 32, 0);
+        let has_big_offset = InstructionPredicate::new_is_signed_int(&*f_binary_imm, "imm", 32, 0);
 
         // XX /n, MI form with imm32.
         recipes.add_template_recipe(
@@ -2798,7 +2805,6 @@ pub(crate) fn define<'shared>(
                     disp1(destination, func, sink);
                 "#,
                 ),
-            formats,
             regs,
         )
         .requires_prefix(true),
@@ -2819,7 +2825,6 @@ pub(crate) fn define<'shared>(
                     disp1(destination, func, sink);
                 "#,
                 ),
-            formats,
             regs,
         )
         .when_prefixed(t8jccb),
@@ -2841,7 +2846,6 @@ pub(crate) fn define<'shared>(
                     disp4(destination, func, sink);
                 "#,
                 ),
-            formats,
             regs,
         )
         .requires_prefix(true),
@@ -2863,7 +2867,6 @@ pub(crate) fn define<'shared>(
                     disp4(destination, func, sink);
                 "#,
                 ),
-            formats,
             regs,
         )
         .when_prefixed(t8jccd),
@@ -2944,9 +2947,8 @@ pub(crate) fn define<'shared>(
     );
 
     {
-        let format = formats.get(f_int_compare_imm);
-
-        let is_small_imm = InstructionPredicate::new_is_signed_int(format, "imm", 8, 0);
+        let is_small_imm =
+            InstructionPredicate::new_is_signed_int(&*f_int_compare_imm, "imm", 8, 0);
 
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("icscc_ib", f_int_compare_imm, 2 + 3)
@@ -2969,7 +2971,7 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let is_big_imm = InstructionPredicate::new_is_signed_int(format, "imm", 32, 0);
+        let is_big_imm = InstructionPredicate::new_is_signed_int(&*f_int_compare_imm, "imm", 32, 0);
 
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("icscc_id", f_int_compare_imm, 5 + 3)
@@ -3014,7 +3016,7 @@ pub(crate) fn define<'shared>(
             .operands_out(vec![abcd])
             .inst_predicate(supported_floatccs_predicate(
                 &supported_floatccs,
-                formats.get(f_float_compare),
+                &*f_float_compare,
             ))
             .emit(
                 r#"
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index 92d8721334..1513b09e18 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -39,13 +39,7 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
 
     gen_inst::generate(&shared_defs, "opcodes.rs", "inst_builder.rs", &out_dir)?;
 
-    gen_legalizer::generate(
-        &isas,
-        &shared_defs.format_registry,
-        &shared_defs.transform_groups,
-        "legalize",
-        &out_dir,
-    )?;
+    gen_legalizer::generate(&isas, &shared_defs.transform_groups, "legalize", &out_dir)?;
 
     for isa in isas {
         gen_registers::generate(&isa, &format!("registers-{}.rs", isa.name), &out_dir)?;
@@ -65,7 +59,6 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
         )?;
 
         gen_binemit::generate(
-            &shared_defs.format_registry,
             &isa.name,
             &isa.recipes,
             &format!("binemit-{}.rs", isa.name),

From 0243b642e36c00d7ef3ba31165be04856f7c47df Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 18 Oct 2019 19:24:03 +0200
Subject: [PATCH 2867/3084] [meta] Remove name lookups in formats;

This does a lot at once, since there was no clear way to split the three
commits:

- Instruction need to be passed an explicit InstructionFormat,
- InstructionFormat deduplication is checked once all entities have been
defined;
---
 cranelift/codegen/meta/src/cdsl/formats.rs    | 122 +---
 .../codegen/meta/src/cdsl/instructions.rs     |  94 +++-
 cranelift/codegen/meta/src/cdsl/operands.rs   |   4 +-
 cranelift/codegen/meta/src/cdsl/xform.rs      |  11 +-
 cranelift/codegen/meta/src/gen_inst.rs        |  58 +-
 cranelift/codegen/meta/src/isa/arm32/mod.rs   |   6 +-
 cranelift/codegen/meta/src/isa/arm64/mod.rs   |   6 +-
 cranelift/codegen/meta/src/isa/riscv/mod.rs   |   6 +-
 .../codegen/meta/src/isa/riscv/recipes.rs     |  72 +--
 .../codegen/meta/src/isa/x86/encodings.rs     |  41 +-
 .../codegen/meta/src/isa/x86/instructions.rs  |  44 +-
 cranelift/codegen/meta/src/isa/x86/mod.rs     |   2 +-
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 427 +++++++-------
 cranelift/codegen/meta/src/lib.rs             |  17 +-
 cranelift/codegen/meta/src/shared/formats.rs  | 523 ++++++++++--------
 .../codegen/meta/src/shared/instructions.rs   | 232 +++++++-
 cranelift/codegen/meta/src/shared/mod.rs      |  67 ++-
 17 files changed, 1002 insertions(+), 730 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
index 0e71f37a97..d03929b5f6 100644
--- a/cranelift/codegen/meta/src/cdsl/formats.rs
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -1,9 +1,6 @@
-use crate::cdsl::operands::{Operand, OperandKind};
-
-use std::collections::{HashMap, HashSet};
+use crate::cdsl::operands::OperandKind;
 use std::fmt;
 use std::rc::Rc;
-use std::slice;
 
 /// An immediate field in an instruction format.
 ///
@@ -30,7 +27,7 @@ pub struct FormatField {
 ///
 /// All instruction formats must be predefined in the meta shared/formats.rs module.
 #[derive(Debug)]
-pub struct InstructionFormat {
+pub(crate) struct InstructionFormat {
     /// Instruction format name in CamelCase. This is used as a Rust variant name in both the
     /// `InstructionData` and `InstructionFormat` enums.
     pub name: &'static str,
@@ -47,6 +44,14 @@ pub struct InstructionFormat {
     pub typevar_operand: Option,
 }
 
+/// A tuple serving as a key to deduplicate InstructionFormat.
+#[derive(Hash, PartialEq, Eq)]
+pub(crate) struct FormatStructure {
+    pub num_value_operands: usize,
+    pub has_value_list: bool,
+    pub imm_field_names: Vec<&'static str>,
+}
+
 impl fmt::Display for InstructionFormat {
     fn fmt(&self, fmt: &mut fmt::Formatter) -> Result<(), fmt::Error> {
         let imm_args = self
@@ -75,9 +80,22 @@ impl InstructionFormat {
                 )
             })
     }
+
+    /// Returns a tuple that uniquely identifies the structure.
+    pub fn structure(&self) -> FormatStructure {
+        FormatStructure {
+            num_value_operands: self.num_value_operands,
+            has_value_list: self.has_value_list,
+            imm_field_names: self
+                .imm_fields
+                .iter()
+                .map(|field| field.kind.name)
+                .collect::>(),
+        }
+    }
 }
 
-pub struct InstructionFormatBuilder {
+pub(crate) struct InstructionFormatBuilder {
     name: &'static str,
     num_value_operands: usize,
     has_value_list: bool,
@@ -131,7 +149,7 @@ impl InstructionFormatBuilder {
         self
     }
 
-    pub fn build(self) -> InstructionFormat {
+    pub fn build(self) -> Rc {
         let typevar_operand = if self.typevar_operand.is_some() {
             self.typevar_operand
         } else if self.has_value_list || self.num_value_operands > 0 {
@@ -141,98 +159,12 @@ impl InstructionFormatBuilder {
             None
         };
 
-        InstructionFormat {
+        Rc::new(InstructionFormat {
             name: self.name,
             num_value_operands: self.num_value_operands,
             has_value_list: self.has_value_list,
             imm_fields: self.imm_fields,
             typevar_operand,
-        }
-    }
-}
-
-pub struct FormatRegistry {
-    /// Map (immediate kinds names, number of values, has varargs) to an instruction format.
-    sig_to_index: HashMap<(Vec, usize, bool), usize>,
-    formats: Vec>,
-    name_set: HashSet<&'static str>,
-}
-
-impl FormatRegistry {
-    pub fn new() -> Self {
-        Self {
-            sig_to_index: HashMap::new(),
-            formats: Vec::new(),
-            name_set: HashSet::new(),
-        }
-    }
-
-    /// Find an existing instruction format that matches the given lists of instruction inputs and
-    /// outputs.
-    pub fn lookup(&self, operands_in: &Vec) -> &Rc {
-        let mut imm_keys = Vec::new();
-        let mut num_values = 0;
-        let mut has_varargs = false;
-
-        for operand in operands_in.iter() {
-            if operand.is_value() {
-                num_values += 1;
-            }
-            if !has_varargs {
-                has_varargs = operand.is_varargs();
-            }
-            if let Some(imm_key) = operand.kind.imm_key() {
-                imm_keys.push(imm_key);
-            }
-        }
-
-        let sig = (imm_keys, num_values, has_varargs);
-        let index = *self
-            .sig_to_index
-            .get(&sig)
-            .expect("unknown InstructionFormat; please define it in shared/formats.rs first");
-        &self.formats[index]
-    }
-
-    pub fn by_name(&self, name: &str) -> &Rc {
-        &self
-            .formats
-            .iter()
-            .find(|format| format.name == name)
-            .unwrap_or_else(|| panic!("format with name '{}' doesn't exist", name))
-    }
-
-    pub fn insert(&mut self, inst_format: InstructionFormatBuilder) {
-        let name = &inst_format.name;
-        if !self.name_set.insert(name) {
-            panic!(
-                "Trying to add an InstructionFormat named {}, but it already exists!",
-                name
-            );
-        }
-
-        let format = inst_format.build();
-
-        // Compute key.
-        let imm_keys = format
-            .imm_fields
-            .iter()
-            .map(|field| field.kind.imm_key().unwrap())
-            .collect();
-        let key = (imm_keys, format.num_value_operands, format.has_value_list);
-
-        let index = self.formats.len();
-        self.formats.push(Rc::new(format));
-        if let Some(already_inserted) = self.sig_to_index.insert(key, index) {
-            panic!(
-                "duplicate InstructionFormat: trying to insert '{}' while '{}' already has the same structure.",
-                self.formats[index].name,
-                self.formats[already_inserted].name
-            );
-        }
-    }
-
-    pub fn iter(&self) -> slice::Iter> {
-        self.formats.iter()
+        })
     }
 }
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index fb92b85572..7663d5e22a 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -1,3 +1,4 @@
+use cranelift_codegen_shared::condcodes::IntCC;
 use cranelift_entity::{entity_impl, PrimaryMap};
 
 use std::collections::HashMap;
@@ -6,13 +7,14 @@ use std::fmt::{Display, Error, Formatter};
 use std::rc::Rc;
 
 use crate::cdsl::camel_case;
-use crate::cdsl::formats::{FormatField, FormatRegistry, InstructionFormat};
+use crate::cdsl::formats::{FormatField, InstructionFormat};
 use crate::cdsl::operands::Operand;
 use crate::cdsl::type_inference::Constraint;
 use crate::cdsl::types::{LaneType, ReferenceType, ValueType, VectorType};
 use crate::cdsl::typevar::TypeVar;
+
+use crate::shared::formats::Formats;
 use crate::shared::types::{Bool, Float, Int, Reference};
-use cranelift_codegen_shared::condcodes::IntCC;
 
 #[derive(Copy, Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub(crate) struct OpcodeNumber(u32);
@@ -20,19 +22,14 @@ entity_impl!(OpcodeNumber);
 
 pub(crate) type AllInstructions = PrimaryMap;
 
-pub(crate) struct InstructionGroupBuilder<'format_reg, 'all_inst> {
-    format_registry: &'format_reg FormatRegistry,
+pub(crate) struct InstructionGroupBuilder<'all_inst> {
     all_instructions: &'all_inst mut AllInstructions,
     own_instructions: Vec,
 }
 
-impl<'format_reg, 'all_inst> InstructionGroupBuilder<'format_reg, 'all_inst> {
-    pub fn new(
-        all_instructions: &'all_inst mut AllInstructions,
-        format_registry: &'format_reg FormatRegistry,
-    ) -> Self {
+impl<'all_inst> InstructionGroupBuilder<'all_inst> {
+    pub fn new(all_instructions: &'all_inst mut AllInstructions) -> Self {
         Self {
-            format_registry,
             all_instructions,
             own_instructions: Vec::new(),
         }
@@ -40,7 +37,7 @@ impl<'format_reg, 'all_inst> InstructionGroupBuilder<'format_reg, 'all_inst> {
 
     pub fn push(&mut self, builder: InstructionBuilder) {
         let opcode_number = OpcodeNumber(self.all_instructions.next_key().as_u32());
-        let inst = builder.build(self.format_registry, opcode_number);
+        let inst = builder.build(opcode_number);
         // Note this clone is cheap, since Instruction is a Rc<> wrapper for InstructionContent.
         self.own_instructions.push(inst.clone());
         self.all_instructions.push(inst);
@@ -201,6 +198,7 @@ impl fmt::Display for InstructionContent {
 pub(crate) struct InstructionBuilder {
     name: String,
     doc: String,
+    format: Rc,
     operands_in: Option>,
     operands_out: Option>,
     constraints: Option>,
@@ -219,10 +217,11 @@ pub(crate) struct InstructionBuilder {
 }
 
 impl InstructionBuilder {
-    pub fn new>(name: S, doc: S) -> Self {
+    pub fn new>(name: S, doc: S, format: &Rc) -> Self {
         Self {
             name: name.into(),
             doc: doc.into(),
+            format: format.clone(),
             operands_in: None,
             operands_out: None,
             constraints: None,
@@ -297,7 +296,7 @@ impl InstructionBuilder {
         self
     }
 
-    fn build(self, format_registry: &FormatRegistry, opcode_number: OpcodeNumber) -> Instruction {
+    fn build(self, opcode_number: OpcodeNumber) -> Instruction {
         let operands_in = self.operands_in.unwrap_or_else(Vec::new);
         let operands_out = self.operands_out.unwrap_or_else(Vec::new);
 
@@ -319,9 +318,10 @@ impl InstructionBuilder {
             .filter_map(|(i, op)| if op.is_value() { Some(i) } else { None })
             .collect();
 
-        let format = format_registry.lookup(&operands_in).clone();
+        verify_format(&self.name, &operands_in, &self.format);
+
         let polymorphic_info =
-            verify_polymorphic(&operands_in, &operands_out, &format, &value_opnums);
+            verify_polymorphic(&operands_in, &operands_out, &self.format, &value_opnums);
 
         // Infer from output operands whether an instruciton clobbers CPU flags or not.
         let writes_cpu_flags = operands_out.iter().any(|op| op.is_cpu_flags());
@@ -336,7 +336,7 @@ impl InstructionBuilder {
             operands_in,
             operands_out,
             constraints: self.constraints.unwrap_or_else(Vec::new),
-            format,
+            format: self.format,
             polymorphic_info,
             value_opnums,
             value_results,
@@ -533,6 +533,57 @@ impl Bindable for BoundInstruction {
     }
 }
 
+/// Checks that the input operands actually match the given format.
+fn verify_format(inst_name: &str, operands_in: &Vec, format: &InstructionFormat) {
+    // A format is defined by:
+    // - its number of input value operands,
+    // - its number and names of input immediate operands,
+    // - whether it has a value list or not.
+    let mut num_values = 0;
+    let mut imm_index = 0;
+
+    for operand in operands_in.iter() {
+        if operand.is_varargs() {
+            assert!(
+                format.has_value_list,
+                "instruction {} has varargs, but its format {} doesn't have a value list; you may \
+                 need to use a different format.",
+                inst_name, format.name
+            );
+        }
+        if operand.is_value() {
+            num_values += 1;
+        }
+        if let Some(imm_name) = operand.kind.imm_name() {
+            if let Some(format_field) = format.imm_fields.get(imm_index) {
+                assert_eq!(
+                    format_field.kind.name, imm_name,
+                    "{}th operand of {} should be {} (according to format), not {} (according to \
+                     inst definition). You may need to use a different format.",
+                    imm_index, inst_name, format_field.kind.name, imm_name
+                );
+                imm_index += 1;
+            }
+        }
+    }
+
+    assert_eq!(
+        num_values, format.num_value_operands,
+        "inst {} doesnt' have as many value input operand as its format {} declares; you may need \
+         to use a different format.",
+        inst_name, format.name
+    );
+
+    assert_eq!(
+        imm_index,
+        format.imm_fields.len(),
+        "inst {} doesn't have as many immediate input \
+         operands as its format {} declares; you may need to use a different format.",
+        inst_name,
+        format.name
+    );
+}
+
 /// Check if this instruction is polymorphic, and verify its use of type variables.
 fn verify_polymorphic(
     operands_in: &Vec,
@@ -1089,8 +1140,8 @@ impl InstructionPredicate {
         ))
     }
 
-    pub fn new_is_colocated_data(format_registry: &FormatRegistry) -> InstructionPredicateNode {
-        let format = format_registry.by_name("UnaryGlobalValue");
+    pub fn new_is_colocated_data(formats: &Formats) -> InstructionPredicateNode {
+        let format = &formats.unary_global_value;
         InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
             &*format,
             "global_value",
@@ -1271,7 +1322,6 @@ mod test {
         outputs: Vec,
     ) -> Instruction {
         // setup a format from the input operands
-        let mut formats = FormatRegistry::new();
         let mut format = InstructionFormatBuilder::new("fake");
         for (i, f) in inputs.iter().enumerate() {
             match f {
@@ -1282,13 +1332,13 @@ mod test {
                 _ => {}
             };
         }
-        formats.insert(format);
+        let format = format.build();
 
         // create the fake instruction
-        InstructionBuilder::new("fake", "A fake instruction for testing.")
+        InstructionBuilder::new("fake", "A fake instruction for testing.", &format)
             .operands_in(field_to_operands(inputs).iter().collect())
             .operands_out(field_to_operands(outputs).iter().collect())
-            .build(&formats, OpcodeNumber(42))
+            .build(OpcodeNumber(42))
     }
 
     #[test]
diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
index a219584c9e..631f0807aa 100644
--- a/cranelift/codegen/meta/src/cdsl/operands.rs
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -138,11 +138,11 @@ pub struct OperandKind {
 }
 
 impl OperandKind {
-    pub fn imm_key(&self) -> Option {
+    pub fn imm_name(&self) -> Option<&str> {
         match self.fields {
             OperandKindFields::ImmEnum(_)
             | OperandKindFields::ImmValue
-            | OperandKindFields::EntityRef => Some(self.name.to_string()),
+            | OperandKindFields::EntityRef => Some(&self.name),
             _ => None,
         }
     }
diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index 6980998d70..e9798c404a 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -465,14 +465,15 @@ impl TransformGroups {
 #[test]
 #[should_panic]
 fn test_double_custom_legalization() {
-    use crate::cdsl::formats::{FormatRegistry, InstructionFormatBuilder};
+    use crate::cdsl::formats::InstructionFormatBuilder;
     use crate::cdsl::instructions::{AllInstructions, InstructionBuilder, InstructionGroupBuilder};
 
+    let nullary = InstructionFormatBuilder::new("nullary").build();
+
     let mut dummy_all = AllInstructions::new();
-    let mut format = FormatRegistry::new();
-    format.insert(InstructionFormatBuilder::new("nullary"));
-    let mut inst_group = InstructionGroupBuilder::new(&mut dummy_all, &format);
-    inst_group.push(InstructionBuilder::new("dummy", "doc"));
+    let mut inst_group = InstructionGroupBuilder::new(&mut dummy_all);
+    inst_group.push(InstructionBuilder::new("dummy", "doc", &nullary));
+
     let inst_group = inst_group.build();
     let dummy_inst = inst_group.by_name("dummy");
 
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index 053eba2517..c1a6277b95 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -4,13 +4,11 @@ use cranelift_codegen_shared::constant_hash;
 use cranelift_entity::EntityRef;
 
 use crate::cdsl::camel_case;
-use crate::cdsl::formats::{FormatRegistry, InstructionFormat};
+use crate::cdsl::formats::InstructionFormat;
 use crate::cdsl::instructions::{AllInstructions, Instruction};
 use crate::cdsl::operands::Operand;
 use crate::cdsl::typevar::{TypeSet, TypeVar};
 
-use crate::shared::Definitions as SharedDefinitions;
-
 use crate::error;
 use crate::srcgen::{Formatter, Match};
 use crate::unique_table::{UniqueSeqTable, UniqueTable};
@@ -19,7 +17,7 @@ use crate::unique_table::{UniqueSeqTable, UniqueTable};
 const TYPESET_LIMIT: usize = 0xff;
 
 /// Generate an instruction format enumeration.
-fn gen_formats(registry: &FormatRegistry, fmt: &mut Formatter) {
+fn gen_formats(formats: &Vec<&InstructionFormat>, fmt: &mut Formatter) {
     fmt.doc_comment(
         r#"
         An instruction format
@@ -32,7 +30,7 @@ fn gen_formats(registry: &FormatRegistry, fmt: &mut Formatter) {
     fmt.line("#[derive(Copy, Clone, PartialEq, Eq, Debug)]");
     fmt.line("pub enum InstructionFormat {");
     fmt.indent(|fmt| {
-        for format in registry.iter() {
+        for format in formats {
             fmt.doc_comment(format.to_string());
             fmtln!(fmt, "{},", format.name);
         }
@@ -47,7 +45,7 @@ fn gen_formats(registry: &FormatRegistry, fmt: &mut Formatter) {
         fmt.line("fn from(inst: &'a InstructionData) -> Self {");
         fmt.indent(|fmt| {
             let mut m = Match::new("*inst");
-            for format in registry.iter() {
+            for format in formats {
                 m.arm(
                     format!("InstructionData::{}", format.name),
                     vec![".."],
@@ -67,12 +65,12 @@ fn gen_formats(registry: &FormatRegistry, fmt: &mut Formatter) {
 /// Every variant must contain an `opcode` field. The size of `InstructionData` should be kept at
 /// 16 bytes on 64-bit architectures. If more space is needed to represent an instruction, use a
 /// `ValueList` to store the additional information out of line.
-fn gen_instruction_data(registry: &FormatRegistry, fmt: &mut Formatter) {
+fn gen_instruction_data(formats: &Vec<&InstructionFormat>, fmt: &mut Formatter) {
     fmt.line("#[derive(Clone, Debug)]");
     fmt.line("#[allow(missing_docs)]");
     fmt.line("pub enum InstructionData {");
     fmt.indent(|fmt| {
-        for format in registry.iter() {
+        for format in formats {
             fmtln!(fmt, "{} {{", format.name);
             fmt.indent(|fmt| {
                 fmt.line("opcode: Opcode,");
@@ -95,7 +93,7 @@ fn gen_instruction_data(registry: &FormatRegistry, fmt: &mut Formatter) {
     fmt.line("}");
 }
 
-fn gen_arguments_method(registry: &FormatRegistry, fmt: &mut Formatter, is_mut: bool) {
+fn gen_arguments_method(formats: &Vec<&InstructionFormat>, fmt: &mut Formatter, is_mut: bool) {
     let (method, mut_, rslice, as_slice) = if is_mut {
         (
             "arguments_mut",
@@ -117,7 +115,7 @@ fn gen_arguments_method(registry: &FormatRegistry, fmt: &mut Formatter, is_mut:
     );
     fmt.indent(|fmt| {
         let mut m = Match::new("*self");
-        for format in registry.iter() {
+        for format in formats {
             let name = format!("InstructionData::{}", format.name);
 
             // Formats with a value list put all of their arguments in the list. We don't split
@@ -165,14 +163,14 @@ fn gen_arguments_method(registry: &FormatRegistry, fmt: &mut Formatter, is_mut:
 /// - `pub fn put_value_list(&mut self, args: ir::ValueList>`
 /// - `pub fn eq(&self, &other: Self, &pool) -> bool`
 /// - `pub fn hash(&self, state: &mut H, &pool)`
-fn gen_instruction_data_impl(registry: &FormatRegistry, fmt: &mut Formatter) {
+fn gen_instruction_data_impl(formats: &Vec<&InstructionFormat>, fmt: &mut Formatter) {
     fmt.line("impl InstructionData {");
     fmt.indent(|fmt| {
         fmt.doc_comment("Get the opcode of this instruction.");
         fmt.line("pub fn opcode(&self) -> Opcode {");
         fmt.indent(|fmt| {
             let mut m = Match::new("*self");
-            for format in registry.iter() {
+            for format in formats {
                 m.arm(format!("InstructionData::{}", format.name), vec!["opcode", ".."],
                       "opcode".to_string());
             }
@@ -185,7 +183,7 @@ fn gen_instruction_data_impl(registry: &FormatRegistry, fmt: &mut Formatter) {
         fmt.line("pub fn typevar_operand(&self, pool: &ir::ValueListPool) -> Option {");
         fmt.indent(|fmt| {
             let mut m = Match::new("*self");
-            for format in registry.iter() {
+            for format in formats {
                 let name = format!("InstructionData::{}", format.name);
                 if format.typevar_operand.is_none() {
                     m.arm(name, vec![".."], "None".to_string());
@@ -208,12 +206,12 @@ fn gen_instruction_data_impl(registry: &FormatRegistry, fmt: &mut Formatter) {
         fmt.empty_line();
 
         fmt.doc_comment("Get the value arguments to this instruction.");
-        gen_arguments_method(registry, fmt, false);
+        gen_arguments_method(formats, fmt, false);
         fmt.empty_line();
 
         fmt.doc_comment(r#"Get mutable references to the value arguments to this
                         instruction."#);
-        gen_arguments_method(registry, fmt, true);
+        gen_arguments_method(formats, fmt, true);
         fmt.empty_line();
 
         fmt.doc_comment(r#"
@@ -227,7 +225,7 @@ fn gen_instruction_data_impl(registry: &FormatRegistry, fmt: &mut Formatter) {
         fmt.indent(|fmt| {
             let mut m = Match::new("*self");
 
-            for format in registry.iter() {
+            for format in formats {
                 if format.has_value_list {
                     m.arm(format!("InstructionData::{}", format.name),
                     vec!["ref mut args", ".."],
@@ -254,7 +252,7 @@ fn gen_instruction_data_impl(registry: &FormatRegistry, fmt: &mut Formatter) {
         fmt.indent(|fmt| {
             fmt.line("let args = match *self {");
             fmt.indent(|fmt| {
-                for format in registry.iter() {
+                for format in formats {
                     if format.has_value_list {
                         fmtln!(fmt, "InstructionData::{} {{ ref mut args, .. }} => args,", format.name);
                     }
@@ -284,7 +282,7 @@ fn gen_instruction_data_impl(registry: &FormatRegistry, fmt: &mut Formatter) {
 
             fmt.line("match (self, other) {");
             fmt.indent(|fmt| {
-                for format in registry.iter() {
+                for format in formats {
                     let name = format!("&InstructionData::{}", format.name);
                     let mut members = vec!["opcode"];
 
@@ -336,7 +334,7 @@ fn gen_instruction_data_impl(registry: &FormatRegistry, fmt: &mut Formatter) {
         fmt.indent(|fmt| {
             fmt.line("match *self {");
             fmt.indent(|fmt| {
-                for format in registry.iter() {
+                for format in formats {
                     let name = format!("InstructionData::{}", format.name);
                     let mut members = vec!["opcode"];
 
@@ -1037,7 +1035,11 @@ fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Fo
 }
 
 /// Generate a Builder trait with methods for all instructions.
-fn gen_builder(instructions: &AllInstructions, formats: &FormatRegistry, fmt: &mut Formatter) {
+fn gen_builder(
+    instructions: &AllInstructions,
+    formats: &Vec<&InstructionFormat>,
+    fmt: &mut Formatter,
+) {
     fmt.doc_comment(
         r#"
         Convenience methods for building instructions.
@@ -1060,7 +1062,7 @@ fn gen_builder(instructions: &AllInstructions, formats: &FormatRegistry, fmt: &m
         for inst in instructions.values() {
             gen_inst_builder(inst, &*inst.format, fmt);
         }
-        for format in formats.iter() {
+        for format in formats {
             gen_format_constructor(format, fmt);
         }
     });
@@ -1068,20 +1070,18 @@ fn gen_builder(instructions: &AllInstructions, formats: &FormatRegistry, fmt: &m
 }
 
 pub(crate) fn generate(
-    shared_defs: &SharedDefinitions,
+    formats: Vec<&InstructionFormat>,
+    all_inst: &AllInstructions,
     opcode_filename: &str,
     inst_builder_filename: &str,
     out_dir: &str,
 ) -> Result<(), error::Error> {
-    let format_registry = &shared_defs.format_registry;
-    let all_inst = &shared_defs.all_instructions;
-
     // Opcodes.
     let mut fmt = Formatter::new();
-    gen_formats(format_registry, &mut fmt);
-    gen_instruction_data(format_registry, &mut fmt);
+    gen_formats(&formats, &mut fmt);
+    gen_instruction_data(&formats, &mut fmt);
     fmt.empty_line();
-    gen_instruction_data_impl(format_registry, &mut fmt);
+    gen_instruction_data_impl(&formats, &mut fmt);
     fmt.empty_line();
     gen_opcodes(all_inst, &mut fmt);
     gen_type_constraints(all_inst, &mut fmt);
@@ -1089,7 +1089,7 @@ pub(crate) fn generate(
 
     // Instruction builder.
     let mut fmt = Formatter::new();
-    gen_builder(all_inst, format_registry, &mut fmt);
+    gen_builder(all_inst, &formats, &mut fmt);
     fmt.update_file(inst_builder_filename, out_dir)?;
 
     Ok(())
diff --git a/cranelift/codegen/meta/src/isa/arm32/mod.rs b/cranelift/codegen/meta/src/isa/arm32/mod.rs
index 152d57d598..5cb1761a7e 100644
--- a/cranelift/codegen/meta/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm32/mod.rs
@@ -53,11 +53,7 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let settings = define_settings(&shared_defs.settings);
     let regs = define_regs();
 
-    let inst_group = InstructionGroupBuilder::new(
-        &mut shared_defs.all_instructions,
-        &shared_defs.format_registry,
-    )
-    .build();
+    let inst_group = InstructionGroupBuilder::new(&mut shared_defs.all_instructions).build();
 
     // CPU modes for 32-bit ARM and Thumb2.
     let mut a32 = CpuMode::new("A32");
diff --git a/cranelift/codegen/meta/src/isa/arm64/mod.rs b/cranelift/codegen/meta/src/isa/arm64/mod.rs
index 3811cc25a6..3440c8af82 100644
--- a/cranelift/codegen/meta/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/meta/src/isa/arm64/mod.rs
@@ -49,11 +49,7 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let settings = define_settings(&shared_defs.settings);
     let regs = define_registers();
 
-    let inst_group = InstructionGroupBuilder::new(
-        &mut shared_defs.all_instructions,
-        &shared_defs.format_registry,
-    )
-    .build();
+    let inst_group = InstructionGroupBuilder::new(&mut shared_defs.all_instructions).build();
 
     let mut a64 = CpuMode::new("A64");
 
diff --git a/cranelift/codegen/meta/src/isa/riscv/mod.rs b/cranelift/codegen/meta/src/isa/riscv/mod.rs
index ef173176c0..801e61a3d2 100644
--- a/cranelift/codegen/meta/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/mod.rs
@@ -89,11 +89,7 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let settings = define_settings(&shared_defs.settings);
     let regs = define_registers();
 
-    let inst_group = InstructionGroupBuilder::new(
-        &mut shared_defs.all_instructions,
-        &shared_defs.format_registry,
-    )
-    .build();
+    let inst_group = InstructionGroupBuilder::new(&mut shared_defs.all_instructions).build();
 
     // CPU modes for 32-bit and 64-bit operation.
     let mut rv_32 = CpuMode::new("RV32");
diff --git a/cranelift/codegen/meta/src/isa/riscv/recipes.rs b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
index ab70595b34..9f68ed79bd 100644
--- a/cranelift/codegen/meta/src/isa/riscv/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
@@ -46,23 +46,7 @@ impl RecipeGroup {
 }
 
 pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeGroup {
-    // Format shorthands.
-    let formats = &shared_defs.format_registry;
-
-    let f_binary = formats.by_name("Binary");
-    let f_binary_imm = formats.by_name("BinaryImm");
-    let f_branch = formats.by_name("Branch");
-    let f_branch_icmp = formats.by_name("BranchIcmp");
-    let f_call = formats.by_name("Call");
-    let f_call_indirect = formats.by_name("CallIndirect");
-    let f_copy_to_ssa = formats.by_name("CopyToSsa");
-    let f_int_compare = formats.by_name("IntCompare");
-    let f_int_compare_imm = formats.by_name("IntCompareImm");
-    let f_jump = formats.by_name("Jump");
-    let f_multiary = formats.by_name("MultiAry");
-    let f_regmove = formats.by_name("RegMove");
-    let f_unary = formats.by_name("Unary");
-    let f_unary_imm = formats.by_name("UnaryImm");
+    let formats = &shared_defs.formats;
 
     // Register classes shorthands.
     let gpr = regs.class_by_name("GPR");
@@ -73,7 +57,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
     // R-type 32-bit instructions: These are mostly binary arithmetic instructions.
     // The encbits are `opcode[6:2] | (funct3 << 5) | (funct7 << 8)
     recipes.push(
-        EncodingRecipeBuilder::new("R", f_binary, 4)
+        EncodingRecipeBuilder::new("R", &formats.binary, 4)
             .operands_in(vec![gpr, gpr])
             .operands_out(vec![gpr])
             .emit("put_r(bits, in_reg0, in_reg1, out_reg0, sink);"),
@@ -81,7 +65,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // R-type with an immediate shift amount instead of rs2.
     recipes.push(
-        EncodingRecipeBuilder::new("Rshamt", f_binary_imm, 4)
+        EncodingRecipeBuilder::new("Rshamt", &formats.binary_imm, 4)
             .operands_in(vec![gpr])
             .operands_out(vec![gpr])
             .emit("put_rshamt(bits, in_reg0, imm.into(), out_reg0, sink);"),
@@ -89,18 +73,18 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // R-type encoding of an integer comparison.
     recipes.push(
-        EncodingRecipeBuilder::new("Ricmp", f_int_compare, 4)
+        EncodingRecipeBuilder::new("Ricmp", &formats.int_compare, 4)
             .operands_in(vec![gpr, gpr])
             .operands_out(vec![gpr])
             .emit("put_r(bits, in_reg0, in_reg1, out_reg0, sink);"),
     );
 
     recipes.push(
-        EncodingRecipeBuilder::new("Ii", f_binary_imm, 4)
+        EncodingRecipeBuilder::new("Ii", &formats.binary_imm, 4)
             .operands_in(vec![gpr])
             .operands_out(vec![gpr])
             .inst_predicate(InstructionPredicate::new_is_signed_int(
-                &*f_binary_imm,
+                &*formats.binary_imm,
                 "imm",
                 12,
                 0,
@@ -110,10 +94,10 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // I-type instruction with a hardcoded %x0 rs1.
     recipes.push(
-        EncodingRecipeBuilder::new("Iz", f_unary_imm, 4)
+        EncodingRecipeBuilder::new("Iz", &formats.unary_imm, 4)
             .operands_out(vec![gpr])
             .inst_predicate(InstructionPredicate::new_is_signed_int(
-                &*f_unary_imm,
+                &*&formats.unary_imm,
                 "imm",
                 12,
                 0,
@@ -123,11 +107,11 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // I-type encoding of an integer comparison.
     recipes.push(
-        EncodingRecipeBuilder::new("Iicmp", f_int_compare_imm, 4)
+        EncodingRecipeBuilder::new("Iicmp", &formats.int_compare_imm, 4)
             .operands_in(vec![gpr])
             .operands_out(vec![gpr])
             .inst_predicate(InstructionPredicate::new_is_signed_int(
-                &*f_int_compare_imm,
+                &*&formats.int_compare_imm,
                 "imm",
                 12,
                 0,
@@ -137,8 +121,9 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // I-type encoding for `jalr` as a return instruction. We won't use the immediate offset.  The
     // variable return values are not encoded.
-    recipes.push(EncodingRecipeBuilder::new("Iret", f_multiary, 4).emit(
-        r#"
+    recipes.push(
+        EncodingRecipeBuilder::new("Iret", &formats.multiary, 4).emit(
+            r#"
                     // Return instructions are always a jalr to %x1.
                     // The return address is provided as a special-purpose link argument.
                     put_i(
@@ -149,11 +134,12 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
                         sink,
                     );
                 "#,
-    ));
+        ),
+    );
 
     // I-type encoding for `jalr` as a call_indirect.
     recipes.push(
-        EncodingRecipeBuilder::new("Icall", f_call_indirect, 4)
+        EncodingRecipeBuilder::new("Icall", &formats.call_indirect, 4)
             .operands_in(vec![gpr])
             .emit(
                 r#"
@@ -171,7 +157,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // Copy of a GPR is implemented as addi x, 0.
     recipes.push(
-        EncodingRecipeBuilder::new("Icopy", f_unary, 4)
+        EncodingRecipeBuilder::new("Icopy", &formats.unary, 4)
             .operands_in(vec![gpr])
             .operands_out(vec![gpr])
             .emit("put_i(bits, in_reg0, 0, out_reg0, sink);"),
@@ -179,14 +165,14 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // Same for a GPR regmove.
     recipes.push(
-        EncodingRecipeBuilder::new("Irmov", f_regmove, 4)
+        EncodingRecipeBuilder::new("Irmov", &formats.reg_move, 4)
             .operands_in(vec![gpr])
             .emit("put_i(bits, src, 0, dst, sink);"),
     );
 
     // Same for copy-to-SSA -- GPR regmove.
     recipes.push(
-        EncodingRecipeBuilder::new("copytossa", f_copy_to_ssa, 4)
+        EncodingRecipeBuilder::new("copytossa", &formats.copy_to_ssa, 4)
             // No operands_in to mention, because a source register is specified directly.
             .operands_out(vec![gpr])
             .emit("put_i(bits, src, 0, out_reg0, sink);"),
@@ -194,10 +180,10 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // U-type instructions have a 20-bit immediate that targets bits 12-31.
     recipes.push(
-        EncodingRecipeBuilder::new("U", f_unary_imm, 4)
+        EncodingRecipeBuilder::new("U", &formats.unary_imm, 4)
             .operands_out(vec![gpr])
             .inst_predicate(InstructionPredicate::new_is_signed_int(
-                &*f_unary_imm,
+                &*&formats.unary_imm,
                 "imm",
                 32,
                 12,
@@ -207,7 +193,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // UJ-type unconditional branch instructions.
     recipes.push(
-        EncodingRecipeBuilder::new("UJ", f_jump, 4)
+        EncodingRecipeBuilder::new("UJ", &formats.jump, 4)
             .branch_range((0, 21))
             .emit(
                 r#"
@@ -218,7 +204,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
             ),
     );
 
-    recipes.push(EncodingRecipeBuilder::new("UJcall", f_call, 4).emit(
+    recipes.push(EncodingRecipeBuilder::new("UJcall", &formats.call, 4).emit(
         r#"
                     sink.reloc_external(Reloc::RiscvCall,
                                         &func.dfg.ext_funcs[func_ref].name,
@@ -230,7 +216,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // SB-type branch instructions.
     recipes.push(
-        EncodingRecipeBuilder::new("SB", f_branch_icmp, 4)
+        EncodingRecipeBuilder::new("SB", &formats.branch_icmp, 4)
             .operands_in(vec![gpr, gpr])
             .branch_range((0, 13))
             .emit(
@@ -244,7 +230,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // SB-type branch instruction with rs2 fixed to zero.
     recipes.push(
-        EncodingRecipeBuilder::new("SBzero", f_branch, 4)
+        EncodingRecipeBuilder::new("SBzero", &formats.branch, 4)
             .operands_in(vec![gpr])
             .branch_range((0, 13))
             .emit(
@@ -258,7 +244,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // Spill of a GPR.
     recipes.push(
-        EncodingRecipeBuilder::new("GPsp", f_unary, 4)
+        EncodingRecipeBuilder::new("GPsp", &formats.unary, 4)
             .operands_in(vec![gpr])
             .operands_out(vec![Stack::new(gpr)])
             .emit("unimplemented!();"),
@@ -266,7 +252,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // Fill of a GPR.
     recipes.push(
-        EncodingRecipeBuilder::new("GPfi", f_unary, 4)
+        EncodingRecipeBuilder::new("GPfi", &formats.unary, 4)
             .operands_in(vec![Stack::new(gpr)])
             .operands_out(vec![gpr])
             .emit("unimplemented!();"),
@@ -274,7 +260,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // Stack-slot to same stack-slot copy, which is guaranteed to turn into a no-op.
     recipes.push(
-        EncodingRecipeBuilder::new("stacknull", f_unary, 0)
+        EncodingRecipeBuilder::new("stacknull", &formats.unary, 0)
             .operands_in(vec![Stack::new(gpr)])
             .operands_out(vec![Stack::new(gpr)])
             .emit(""),
@@ -282,7 +268,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
 
     // No-op fills, created by late-stage redundant-fill removal.
     recipes.push(
-        EncodingRecipeBuilder::new("fillnull", f_unary, 0)
+        EncodingRecipeBuilder::new("fillnull", &formats.unary, 0)
             .operands_in(vec![Stack::new(gpr)])
             .operands_out(vec![gpr])
             .clobbers_flags(false)
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index a9a6f700ac..94ade711a1 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -374,7 +374,7 @@ pub(crate) fn define(
     r: &RecipeGroup,
 ) -> PerCpuModeEncodings {
     let shared = &shared_defs.instructions;
-    let formats = &shared_defs.format_registry;
+    let formats = &shared_defs.formats;
 
     // Shorthands for instructions.
     let adjust_sp_down = shared.by_name("adjust_sp_down");
@@ -774,8 +774,8 @@ pub(crate) fn define(
     e.enc64(iconst.bind(I32), rec_pu_id.opcodes(&MOV_IMM));
 
     // The 32-bit immediate movl also zero-extends to 64 bits.
-    let f_unary_imm = formats.by_name("UnaryImm");
-    let is_unsigned_int32 = InstructionPredicate::new_is_unsigned_int(&*f_unary_imm, "imm", 32, 0);
+    let is_unsigned_int32 =
+        InstructionPredicate::new_is_unsigned_int(&*formats.unary_imm, "imm", 32, 0);
 
     e.enc64_func(
         iconst.bind(I64),
@@ -801,7 +801,7 @@ pub(crate) fn define(
     }
     e.enc64(bconst.bind(B64), rec_pu_id_bool.opcodes(&MOV_IMM).rex());
 
-    let is_zero_int = InstructionPredicate::new_is_zero_int(f_unary_imm, "imm");
+    let is_zero_int = InstructionPredicate::new_is_zero_int(&formats.unary_imm, "imm");
     e.enc_both_instp(
         iconst.bind(I8),
         rec_u_id_z.opcodes(&XORB),
@@ -880,8 +880,8 @@ pub(crate) fn define(
     e.enc64_isap(ctz.bind(I32), rec_urm.opcodes(&TZCNT), use_bmi1);
 
     // Loads and stores.
-    let f_load_complex = formats.by_name("LoadComplex");
-    let is_load_complex_length_two = InstructionPredicate::new_length_equals(&*f_load_complex, 2);
+    let is_load_complex_length_two =
+        InstructionPredicate::new_length_equals(&*formats.load_complex, 2);
 
     for recipe in &[rec_ldWithIndex, rec_ldWithIndexDisp8, rec_ldWithIndexDisp32] {
         e.enc_i32_i64_instp(
@@ -925,9 +925,8 @@ pub(crate) fn define(
         );
     }
 
-    let f_store_complex = formats.by_name("StoreComplex");
     let is_store_complex_length_three =
-        InstructionPredicate::new_length_equals(&*f_store_complex, 3);
+        InstructionPredicate::new_length_equals(&*formats.store_complex, 3);
 
     for recipe in &[rec_stWithIndex, rec_stWithIndexDisp8, rec_stWithIndexDisp32] {
         e.enc_i32_i64_instp(
@@ -1233,8 +1232,8 @@ pub(crate) fn define(
     );
 
     // 64-bit, colocated, both PIC and non-PIC. Use the lea instruction's pc-relative field.
-    let f_func_addr = formats.by_name("FuncAddr");
-    let is_colocated_func = InstructionPredicate::new_is_colocated_func(&*f_func_addr, "func_ref");
+    let is_colocated_func =
+        InstructionPredicate::new_is_colocated_func(&*formats.func_addr, "func_ref");
     e.enc64_instp(
         func_addr.bind(I64),
         rec_pcrel_fnaddr8.opcodes(&LEA).rex().w(),
@@ -1293,8 +1292,7 @@ pub(crate) fn define(
     e.enc32(call, rec_call_id.opcodes(&CALL_RELATIVE));
 
     // 64-bit, colocated, both PIC and non-PIC. Use the call instruction's pc-relative field.
-    let f_call = formats.by_name("Call");
-    let is_colocated_func = InstructionPredicate::new_is_colocated_func(&*f_call, "func_ref");
+    let is_colocated_func = InstructionPredicate::new_is_colocated_func(&*formats.call, "func_ref");
     e.enc64_instp(call, rec_call_id.opcodes(&CALL_RELATIVE), is_colocated_func);
 
     // 64-bit, non-colocated, PIC. There is no 64-bit non-colocated non-PIC version, since non-PIC
@@ -1564,18 +1562,16 @@ pub(crate) fn define(
 
     // Floating-point constants equal to 0.0 can be encoded using either `xorps` or `xorpd`, for
     // 32-bit and 64-bit floats respectively.
-    let f_unary_ieee32 = formats.by_name("UnaryIeee32");
     let is_zero_32_bit_float =
-        InstructionPredicate::new_is_zero_32bit_float(&*f_unary_ieee32, "imm");
+        InstructionPredicate::new_is_zero_32bit_float(&*formats.unary_ieee32, "imm");
     e.enc32_instp(
         f32const,
         rec_f32imm_z.opcodes(&XORPS),
         is_zero_32_bit_float.clone(),
     );
 
-    let f_unary_ieee64 = formats.by_name("UnaryIeee64");
     let is_zero_64_bit_float =
-        InstructionPredicate::new_is_zero_64bit_float(&*f_unary_ieee64, "imm");
+        InstructionPredicate::new_is_zero_64bit_float(&*formats.unary_ieee64, "imm");
     e.enc32_instp(
         f64const,
         rec_f64imm_z.opcodes(&XORPD),
@@ -1847,18 +1843,17 @@ pub(crate) fn define(
     // this must be encoded prior to the MOVUPS implementation (below) so the compiler sees this
     // encoding first
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
-        let f_unary_const = formats.by_name("UnaryConst");
         let instruction = vconst.bind(vector(ty, sse_vector_size));
 
         let is_zero_128bit =
-            InstructionPredicate::new_is_all_zeroes(&*f_unary_const, "constant_handle");
+            InstructionPredicate::new_is_all_zeroes(&*formats.unary_const, "constant_handle");
         let template = rec_vconst_optimized.nonrex().opcodes(&PXOR);
         e.enc_32_64_func(instruction.clone(), template, |builder| {
             builder.inst_predicate(is_zero_128bit)
         });
 
         let is_ones_128bit =
-            InstructionPredicate::new_is_all_ones(&*f_unary_const, "constant_handle");
+            InstructionPredicate::new_is_all_ones(&*formats.unary_const, "constant_handle");
         let template = rec_vconst_optimized.nonrex().opcodes(&PCMPEQB);
         e.enc_32_64_func(instruction, template, |builder| {
             builder.inst_predicate(is_ones_128bit)
@@ -2038,9 +2033,11 @@ pub(crate) fn define(
         };
 
         let instruction = icmp.bind(vector(ty, sse_vector_size));
-        let f_int_compare = formats.by_name("IntCompare");
-        let has_eq_condition_code =
-            InstructionPredicate::new_has_condition_code(&*f_int_compare, IntCC::Equal, "cond");
+        let has_eq_condition_code = InstructionPredicate::new_has_condition_code(
+            &*formats.int_compare,
+            IntCC::Equal,
+            "cond",
+        );
         let template = rec_icscc_fpr.nonrex().opcodes(opcodes);
         e.enc_32_64_func(instruction, template, |builder| {
             let builder = builder.inst_predicate(has_eq_condition_code);
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index d5d02b1ac5..21e51982ce 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -1,21 +1,22 @@
 #![allow(non_snake_case)]
 
-use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::instructions::{
     AllInstructions, InstructionBuilder as Inst, InstructionGroup, InstructionGroupBuilder,
 };
 use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
 use crate::cdsl::types::ValueType;
 use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
+
+use crate::shared::formats::Formats;
 use crate::shared::immediates::Immediates;
 use crate::shared::types;
 
 pub(crate) fn define(
     mut all_instructions: &mut AllInstructions,
-    format_registry: &FormatRegistry,
+    formats: &Formats,
     immediates: &Immediates,
 ) -> InstructionGroup {
-    let mut ig = InstructionGroupBuilder::new(&mut all_instructions, format_registry);
+    let mut ig = InstructionGroupBuilder::new(&mut all_instructions);
 
     let iflags: &TypeVar = &ValueType::Special(types::Flag::IFlags.into()).into();
 
@@ -43,6 +44,7 @@ pub(crate) fn define(
 
         Return both quotient and remainder.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![nlo, nhi, d])
         .operands_out(vec![q, r])
@@ -62,6 +64,7 @@ pub(crate) fn define(
 
         Return both quotient and remainder.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![nlo, nhi, d])
         .operands_out(vec![q, r])
@@ -82,6 +85,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![argL, argR])
         .operands_out(vec![resLo, resHi]),
@@ -96,6 +100,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![argL, argR])
         .operands_out(vec![resLo, resHi]),
@@ -132,6 +137,7 @@ pub(crate) fn define(
 
         This instruction does not trap.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -154,6 +160,7 @@ pub(crate) fn define(
         When the two operands don't compare as LT, `y` is returned unchanged,
         even if it is a signalling NaN.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -172,6 +179,7 @@ pub(crate) fn define(
         When the two operands don't compare as GT, `y` is returned unchanged,
         even if it is a signalling NaN.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -190,6 +198,7 @@ pub(crate) fn define(
     This is polymorphic in i32 and i64. However, it is only implemented for i64
     in 64-bit mode, and only for i32 in 32-bit mode.
     "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .other_side_effects(true)
@@ -208,6 +217,7 @@ pub(crate) fn define(
     This is polymorphic in i32 and i64. However, it is only implemented for i64
     in 64-bit mode, and only for i32 in 32-bit mode.
     "#,
+            &formats.nullary,
         )
         .operands_out(vec![x])
         .other_side_effects(true)
@@ -229,6 +239,7 @@ pub(crate) fn define(
     This is polymorphic in i32 and i64. It is implemented for both i64 and
     i32 in 64-bit mode, and only for i32 in 32-bit mode.
     "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![y, rflags]),
@@ -241,6 +252,7 @@ pub(crate) fn define(
     Bit Scan Forwards -- returns the bit-index of the least significant 1
     in the word. Is otherwise identical to 'bsr', just above.
     "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![y, rflags]),
@@ -269,6 +281,7 @@ pub(crate) fn define(
     Packed Shuffle Doublewords -- copies data from either memory or lanes in an extended
     register and re-orders the data according to the passed immediate byte.
     "#,
+            &formats.extract_lane,
         )
         .operands_in(vec![a, i]) // TODO allow copying from memory here (need more permissive type than TxN)
         .operands_out(vec![a]),
@@ -281,6 +294,7 @@ pub(crate) fn define(
     Packed Shuffle Bytes -- re-orders data in an extended register using a shuffle
     mask from either memory or another extended register
     "#,
+            &formats.binary,
         )
         .operands_in(vec![a, b]) // TODO allow re-ordering from memory here (need more permissive type than TxN)
         .operands_out(vec![a]),
@@ -298,6 +312,7 @@ pub(crate) fn define(
         The lane index, ``Idx``, is an immediate value, not an SSA value. It
         must indicate a valid lane index for the type of ``x``.
         "#,
+            &formats.extract_lane,
         )
         .operands_in(vec![x, Idx])
         .operands_out(vec![a]),
@@ -325,6 +340,7 @@ pub(crate) fn define(
         The lane index, ``Idx``, is an immediate value, not an SSA value. It
         must indicate a valid lane index for the type of ``x``.
         "#,
+            &formats.insert_lane,
         )
         .operands_in(vec![x, Idx, y])
         .operands_out(vec![a]),
@@ -347,10 +363,11 @@ pub(crate) fn define(
         Inst::new(
             "x86_insertps",
             r#"
-        Insert a lane of ``y`` into ``x`` at using ``Idx`` to encode both which lane the value is 
-        extracted from and which it is inserted to. This is similar to x86_pinsr but inserts 
+        Insert a lane of ``y`` into ``x`` at using ``Idx`` to encode both which lane the value is
+        extracted from and which it is inserted to. This is similar to x86_pinsr but inserts
         floats, which are already stored in an XMM register.
         "#,
+            &formats.insert_lane,
         )
         .operands_in(vec![x, Idx, y])
         .operands_out(vec![a]),
@@ -366,6 +383,7 @@ pub(crate) fn define(
             r#"
         Move the low 64 bits of the float vector ``y`` to the low 64 bits of float vector ``x``
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -377,6 +395,7 @@ pub(crate) fn define(
             r#"
         Move the low 64 bits of the float vector ``y`` to the high 64 bits of float vector ``x``
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -401,41 +420,48 @@ pub(crate) fn define(
             .includes_scalars(false)
             .build(),
     );
+
     let x = &operand_doc("x", IxN, "Vector value to shift");
     let y = &operand_doc("y", I64x2, "Number of bits to shift");
     let a = &operand("a", IxN);
+
     ig.push(
         Inst::new(
             "x86_psll",
             r#"
-        Shift Packed Data Left Logical -- This implements the behavior of the shared instruction 
+        Shift Packed Data Left Logical -- This implements the behavior of the shared instruction
         ``ishl`` but alters the shift operand to live in an XMM register as expected by the PSLL*
         family of instructions.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
     );
+
     ig.push(
         Inst::new(
             "x86_psrl",
             r#"
-        Shift Packed Data Right Logical -- This implements the behavior of the shared instruction 
+        Shift Packed Data Right Logical -- This implements the behavior of the shared instruction
         ``ushr`` but alters the shift operand to live in an XMM register as expected by the PSRL*
         family of instructions.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
     );
+
     ig.push(
         Inst::new(
             "x86_psra",
             r#"
-        Shift Packed Data Right Arithmetic -- This implements the behavior of the shared 
-        instruction ``sshr`` but alters the shift operand to live in an XMM register as expected by 
+        Shift Packed Data Right Arithmetic -- This implements the behavior of the shared
+        instruction ``sshr`` but alters the shift operand to live in an XMM register as expected by
         the PSRA* family of instructions.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index e18c54ad64..1322368265 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -20,7 +20,7 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
 
     let inst_group = instructions::define(
         &mut shared_defs.all_instructions,
-        &shared_defs.format_registry,
+        &shared_defs.formats,
         &shared_defs.imm,
     );
     legalize::define(shared_defs, &inst_group);
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index bf91a6326c..c1fd7a2c17 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -345,52 +345,7 @@ pub(crate) fn define<'shared>(
     let stack_gpr32 = Stack::new(gpr);
     let stack_fpr32 = Stack::new(fpr);
 
-    // Format shorthands, prefixed with f_.
-    let formats = &shared_defs.format_registry;
-
-    let f_binary = formats.by_name("Binary");
-    let f_binary_imm = formats.by_name("BinaryImm");
-    let f_branch = formats.by_name("Branch");
-    let f_branch_float = formats.by_name("BranchFloat");
-    let f_branch_int = formats.by_name("BranchInt");
-    let f_branch_table_entry = formats.by_name("BranchTableEntry");
-    let f_branch_table_base = formats.by_name("BranchTableBase");
-    let f_call = formats.by_name("Call");
-    let f_call_indirect = formats.by_name("CallIndirect");
-    let f_copy_special = formats.by_name("CopySpecial");
-    let f_copy_to_ssa = formats.by_name("CopyToSsa");
-    let f_extract_lane = formats.by_name("ExtractLane"); // TODO this would preferably retrieve a BinaryImm8 format but because formats are compared structurally and ExtractLane has the same structure this is impossible--if we rename ExtractLane, it may even impact parsing
-    let f_float_compare = formats.by_name("FloatCompare");
-    let f_float_cond = formats.by_name("FloatCond");
-    let f_float_cond_trap = formats.by_name("FloatCondTrap");
-    let f_func_addr = formats.by_name("FuncAddr");
-    let f_indirect_jump = formats.by_name("IndirectJump");
-    let f_insert_lane = formats.by_name("InsertLane");
-    let f_int_compare = formats.by_name("IntCompare");
-    let f_int_compare_imm = formats.by_name("IntCompareImm");
-    let f_int_cond = formats.by_name("IntCond");
-    let f_int_cond_trap = formats.by_name("IntCondTrap");
-    let f_int_select = formats.by_name("IntSelect");
-    let f_jump = formats.by_name("Jump");
-    let f_load = formats.by_name("Load");
-    let f_load_complex = formats.by_name("LoadComplex");
-    let f_multiary = formats.by_name("MultiAry");
-    let f_nullary = formats.by_name("NullAry");
-    let f_reg_fill = formats.by_name("RegFill");
-    let f_reg_move = formats.by_name("RegMove");
-    let f_reg_spill = formats.by_name("RegSpill");
-    let f_stack_load = formats.by_name("StackLoad");
-    let f_store = formats.by_name("Store");
-    let f_store_complex = formats.by_name("StoreComplex");
-    let f_ternary = formats.by_name("Ternary");
-    let f_trap = formats.by_name("Trap");
-    let f_unary = formats.by_name("Unary");
-    let f_unary_bool = formats.by_name("UnaryBool");
-    let f_unary_const = formats.by_name("UnaryConst");
-    let f_unary_global_value = formats.by_name("UnaryGlobalValue");
-    let f_unary_ieee32 = formats.by_name("UnaryIeee32");
-    let f_unary_ieee64 = formats.by_name("UnaryIeee64");
-    let f_unary_imm = formats.by_name("UnaryImm");
+    let formats = &shared_defs.formats;
 
     // Predicates shorthands.
     let use_sse41 = settings.predicate_by_name("use_sse41");
@@ -401,32 +356,32 @@ pub(crate) fn define<'shared>(
     // A null unary instruction that takes a GPR register. Can be used for identity copies and
     // no-op conversions.
     recipes.add_recipe(
-        EncodingRecipeBuilder::new("null", f_unary, 0)
+        EncodingRecipeBuilder::new("null", &formats.unary, 0)
             .operands_in(vec![gpr])
             .operands_out(vec![0])
             .emit(""),
     );
     recipes.add_recipe(
-        EncodingRecipeBuilder::new("null_fpr", f_unary, 0)
+        EncodingRecipeBuilder::new("null_fpr", &formats.unary, 0)
             .operands_in(vec![fpr])
             .operands_out(vec![0])
             .emit(""),
     );
     recipes.add_recipe(
-        EncodingRecipeBuilder::new("stacknull", f_unary, 0)
+        EncodingRecipeBuilder::new("stacknull", &formats.unary, 0)
             .operands_in(vec![stack_gpr32])
             .operands_out(vec![stack_gpr32])
             .emit(""),
     );
 
     recipes.add_recipe(
-        EncodingRecipeBuilder::new("get_pinned_reg", f_nullary, 0)
+        EncodingRecipeBuilder::new("get_pinned_reg", &formats.nullary, 0)
             .operands_out(vec![reg_r15])
             .emit(""),
     );
     // umr with a fixed register output that's r15.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("set_pinned_reg", f_unary, 1)
+        EncodingRecipeBuilder::new("set_pinned_reg", &formats.unary, 1)
             .operands_in(vec![gpr])
             .clobbers_flags(false)
             .emit(
@@ -440,25 +395,26 @@ pub(crate) fn define<'shared>(
 
     // No-op fills, created by late-stage redundant-fill removal.
     recipes.add_recipe(
-        EncodingRecipeBuilder::new("fillnull", f_unary, 0)
+        EncodingRecipeBuilder::new("fillnull", &formats.unary, 0)
             .operands_in(vec![stack_gpr32])
             .operands_out(vec![gpr])
             .clobbers_flags(false)
             .emit(""),
     );
     recipes.add_recipe(
-        EncodingRecipeBuilder::new("ffillnull", f_unary, 0)
+        EncodingRecipeBuilder::new("ffillnull", &formats.unary, 0)
             .operands_in(vec![stack_gpr32])
             .operands_out(vec![fpr])
             .clobbers_flags(false)
             .emit(""),
     );
 
-    recipes
-        .add_recipe(EncodingRecipeBuilder::new("debugtrap", f_nullary, 1).emit("sink.put1(0xcc);"));
+    recipes.add_recipe(
+        EncodingRecipeBuilder::new("debugtrap", &formats.nullary, 1).emit("sink.put1(0xcc);"),
+    );
 
     // XX opcode, no ModR/M.
-    recipes.add_template_recipe(EncodingRecipeBuilder::new("trap", f_trap, 0).emit(
+    recipes.add_template_recipe(EncodingRecipeBuilder::new("trap", &formats.trap, 0).emit(
         r#"
             sink.trap(code, func.srclocs[inst]);
             {{PUT_OP}}(bits, BASE_REX, sink);
@@ -467,7 +423,7 @@ pub(crate) fn define<'shared>(
 
     // Macro: conditional jump over a ud2.
     recipes.add_recipe(
-        EncodingRecipeBuilder::new("trapif", f_int_cond_trap, 4)
+        EncodingRecipeBuilder::new("trapif", &formats.int_cond_trap, 4)
             .operands_in(vec![reg_rflags])
             .clobbers_flags(false)
             .emit(
@@ -484,12 +440,12 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_recipe(
-        EncodingRecipeBuilder::new("trapff", f_float_cond_trap, 4)
+        EncodingRecipeBuilder::new("trapff", &formats.float_cond_trap, 4)
             .operands_in(vec![reg_rflags])
             .clobbers_flags(false)
             .inst_predicate(supported_floatccs_predicate(
                 &supported_floatccs,
-                &*f_float_cond_trap,
+                &*formats.float_cond_trap,
             ))
             .emit(
                 r#"
@@ -506,7 +462,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rr", f_binary, 1)
+        EncodingRecipeBuilder::new("rr", &formats.binary, 1)
             .operands_in(vec![gpr, gpr])
             .operands_out(vec![0])
             .emit(
@@ -519,7 +475,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r with operands swapped. (RM form).
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rrx", f_binary, 1)
+        EncodingRecipeBuilder::new("rrx", &formats.binary, 1)
             .operands_in(vec![gpr, gpr])
             .operands_out(vec![0])
             .emit(
@@ -532,7 +488,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r with FPR ins and outs. A form.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("fa", f_binary, 1)
+        EncodingRecipeBuilder::new("fa", &formats.binary, 1)
             .operands_in(vec![fpr, fpr])
             .operands_out(vec![0])
             .emit(
@@ -545,7 +501,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r with FPR ins and outs. A form with input operands swapped.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("fax", f_binary, 1)
+        EncodingRecipeBuilder::new("fax", &formats.binary, 1)
             .operands_in(vec![fpr, fpr])
             .operands_out(vec![1])
             .emit(
@@ -559,11 +515,11 @@ pub(crate) fn define<'shared>(
     // XX /r with FPR ins and outs. A form with a byte immediate.
     {
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("fa_ib", f_insert_lane, 2)
+            EncodingRecipeBuilder::new("fa_ib", &formats.insert_lane, 2)
                 .operands_in(vec![fpr, fpr])
                 .operands_out(vec![0])
                 .inst_predicate(InstructionPredicate::new_is_unsigned_int(
-                    &*f_insert_lane,
+                    &*formats.insert_lane,
                     "lane",
                     8,
                     0,
@@ -581,7 +537,7 @@ pub(crate) fn define<'shared>(
 
     // XX /n for a unary operation with extension bits.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("ur", f_unary, 1)
+        EncodingRecipeBuilder::new("ur", &formats.unary, 1)
             .operands_in(vec![gpr])
             .operands_out(vec![0])
             .emit(
@@ -595,7 +551,7 @@ pub(crate) fn define<'shared>(
     // XX /r, but for a unary operator with separate input/output register, like
     // copies. MR form, preserving flags.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("umr", f_unary, 1)
+        EncodingRecipeBuilder::new("umr", &formats.unary, 1)
             .operands_in(vec![gpr])
             .operands_out(vec![gpr])
             .clobbers_flags(false)
@@ -609,7 +565,7 @@ pub(crate) fn define<'shared>(
 
     // Same as umr, but with FPR -> GPR registers.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rfumr", f_unary, 1)
+        EncodingRecipeBuilder::new("rfumr", &formats.unary, 1)
             .operands_in(vec![fpr])
             .operands_out(vec![gpr])
             .clobbers_flags(false)
@@ -623,7 +579,7 @@ pub(crate) fn define<'shared>(
 
     // Same as umr, but with the source register specified directly.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("umr_reg_to_ssa", f_copy_to_ssa, 1)
+        EncodingRecipeBuilder::new("umr_reg_to_ssa", &formats.copy_to_ssa, 1)
             // No operands_in to mention, because a source register is specified directly.
             .operands_out(vec![gpr])
             .clobbers_flags(false)
@@ -638,7 +594,7 @@ pub(crate) fn define<'shared>(
     // XX /r, but for a unary operator with separate input/output register.
     // RM form. Clobbers FLAGS.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("urm", f_unary, 1)
+        EncodingRecipeBuilder::new("urm", &formats.unary, 1)
             .operands_in(vec![gpr])
             .operands_out(vec![gpr])
             .emit(
@@ -651,7 +607,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r. Same as urm, but doesn't clobber FLAGS.
     let urm_noflags = recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("urm_noflags", f_unary, 1)
+        EncodingRecipeBuilder::new("urm_noflags", &formats.unary, 1)
             .operands_in(vec![gpr])
             .operands_out(vec![gpr])
             .clobbers_flags(false)
@@ -666,7 +622,7 @@ pub(crate) fn define<'shared>(
     // XX /r. Same as urm_noflags, but input limited to ABCD.
     recipes.add_template(
         Template::new(
-            EncodingRecipeBuilder::new("urm_noflags_abcd", f_unary, 1)
+            EncodingRecipeBuilder::new("urm_noflags_abcd", &formats.unary, 1)
                 .operands_in(vec![abcd])
                 .operands_out(vec![gpr])
                 .clobbers_flags(false)
@@ -683,7 +639,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r, RM form, FPR -> FPR.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("furm", f_unary, 1)
+        EncodingRecipeBuilder::new("furm", &formats.unary, 1)
             .operands_in(vec![fpr])
             .operands_out(vec![fpr])
             .clobbers_flags(false)
@@ -697,7 +653,7 @@ pub(crate) fn define<'shared>(
 
     // Same as furm, but with the source register specified directly.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("furm_reg_to_ssa", f_copy_to_ssa, 1)
+        EncodingRecipeBuilder::new("furm_reg_to_ssa", &formats.copy_to_ssa, 1)
             // No operands_in to mention, because a source register is specified directly.
             .operands_out(vec![fpr])
             .clobbers_flags(false)
@@ -711,7 +667,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r, RM form, GPR -> FPR.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("frurm", f_unary, 1)
+        EncodingRecipeBuilder::new("frurm", &formats.unary, 1)
             .operands_in(vec![gpr])
             .operands_out(vec![fpr])
             .clobbers_flags(false)
@@ -725,7 +681,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r, RM form, FPR -> GPR.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rfurm", f_unary, 1)
+        EncodingRecipeBuilder::new("rfurm", &formats.unary, 1)
             .operands_in(vec![fpr])
             .operands_out(vec![gpr])
             .clobbers_flags(false)
@@ -739,7 +695,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r, RMI form for one of the roundXX SSE 4.1 instructions.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("furmi_rnd", f_unary, 2)
+        EncodingRecipeBuilder::new("furmi_rnd", &formats.unary, 2)
             .operands_in(vec![fpr])
             .operands_out(vec![fpr])
             .isa_predicate(use_sse41)
@@ -760,7 +716,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r, for regmove instructions.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rmov", f_reg_move, 1)
+        EncodingRecipeBuilder::new("rmov", &formats.reg_move, 1)
             .operands_in(vec![gpr])
             .clobbers_flags(false)
             .emit(
@@ -773,7 +729,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r, for regmove instructions (FPR version, RM encoded).
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("frmov", f_reg_move, 1)
+        EncodingRecipeBuilder::new("frmov", &formats.reg_move, 1)
             .operands_in(vec![fpr])
             .clobbers_flags(false)
             .emit(
@@ -786,7 +742,7 @@ pub(crate) fn define<'shared>(
 
     // XX /n with one arg in %rcx, for shifts.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rc", f_binary, 1)
+        EncodingRecipeBuilder::new("rc", &formats.binary, 1)
             .operands_in(vec![
                 OperandConstraint::RegClass(gpr),
                 OperandConstraint::FixedReg(reg_rcx),
@@ -802,7 +758,7 @@ pub(crate) fn define<'shared>(
 
     // XX /n for division: inputs in %rax, %rdx, r. Outputs in %rax, %rdx.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("div", f_ternary, 1)
+        EncodingRecipeBuilder::new("div", &formats.ternary, 1)
             .operands_in(vec![
                 OperandConstraint::FixedReg(reg_rax),
                 OperandConstraint::FixedReg(reg_rdx),
@@ -820,7 +776,7 @@ pub(crate) fn define<'shared>(
 
     // XX /n for {s,u}mulx: inputs in %rax, r. Outputs in %rdx(hi):%rax(lo)
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("mulx", f_binary, 1)
+        EncodingRecipeBuilder::new("mulx", &formats.binary, 1)
             .operands_in(vec![
                 OperandConstraint::FixedReg(reg_rax),
                 OperandConstraint::RegClass(gpr),
@@ -840,11 +796,11 @@ pub(crate) fn define<'shared>(
     // XX /n ib with 8-bit immediate sign-extended.
     {
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("r_ib", f_binary_imm, 2)
+            EncodingRecipeBuilder::new("r_ib", &formats.binary_imm, 2)
                 .operands_in(vec![gpr])
                 .operands_out(vec![0])
                 .inst_predicate(InstructionPredicate::new_is_signed_int(
-                    &*f_binary_imm,
+                    &*formats.binary_imm,
                     "imm",
                     8,
                     0,
@@ -861,11 +817,11 @@ pub(crate) fn define<'shared>(
 
         // XX /n id with 32-bit immediate sign-extended.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("r_id", f_binary_imm, 5)
+            EncodingRecipeBuilder::new("r_id", &formats.binary_imm, 5)
                 .operands_in(vec![gpr])
                 .operands_out(vec![0])
                 .inst_predicate(InstructionPredicate::new_is_signed_int(
-                    &*f_binary_imm,
+                    &*formats.binary_imm,
                     "imm",
                     32,
                     0,
@@ -884,11 +840,11 @@ pub(crate) fn define<'shared>(
     // XX /r ib with 8-bit unsigned immediate (e.g. for pshufd)
     {
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("r_ib_unsigned_fpr", f_extract_lane, 2)
+            EncodingRecipeBuilder::new("r_ib_unsigned_fpr", &formats.extract_lane, 2)
                 .operands_in(vec![fpr])
                 .operands_out(vec![fpr])
                 .inst_predicate(InstructionPredicate::new_is_unsigned_int(
-                    &*f_extract_lane,
+                    &*formats.extract_lane,
                     "lane",
                     8,
                     0,
@@ -907,11 +863,11 @@ pub(crate) fn define<'shared>(
     // XX /r ib with 8-bit unsigned immediate (e.g. for extractlane)
     {
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("r_ib_unsigned_gpr", f_extract_lane, 2)
+            EncodingRecipeBuilder::new("r_ib_unsigned_gpr", &formats.extract_lane, 2)
                 .operands_in(vec![fpr])
                 .operands_out(vec![gpr])
                 .inst_predicate(InstructionPredicate::new_is_unsigned_int(
-                    &*f_extract_lane, "lane", 8, 0,
+                    &*formats.extract_lane, "lane", 8, 0,
                 ))
                 .emit(
                     r#"
@@ -927,11 +883,11 @@ pub(crate) fn define<'shared>(
     // XX /r ib with 8-bit unsigned immediate (e.g. for insertlane)
     {
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("r_ib_unsigned_r", f_insert_lane, 2)
+            EncodingRecipeBuilder::new("r_ib_unsigned_r", &formats.insert_lane, 2)
                 .operands_in(vec![fpr, gpr])
                 .operands_out(vec![0])
                 .inst_predicate(InstructionPredicate::new_is_unsigned_int(
-                    &*f_insert_lane,
+                    &*formats.insert_lane,
                     "lane",
                     8,
                     0,
@@ -950,10 +906,10 @@ pub(crate) fn define<'shared>(
     {
         // XX /n id with 32-bit immediate sign-extended. UnaryImm version.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("u_id", f_unary_imm, 5)
+            EncodingRecipeBuilder::new("u_id", &formats.unary_imm, 5)
                 .operands_out(vec![gpr])
                 .inst_predicate(InstructionPredicate::new_is_signed_int(
-                    &*f_unary_imm,
+                    &*formats.unary_imm,
                     "imm",
                     32,
                     0,
@@ -971,7 +927,7 @@ pub(crate) fn define<'shared>(
 
     // XX+rd id unary with 32-bit immediate. Note no recipe predicate.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("pu_id", f_unary_imm, 4)
+        EncodingRecipeBuilder::new("pu_id", &formats.unary_imm, 4)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -986,7 +942,7 @@ pub(crate) fn define<'shared>(
 
     // XX+rd id unary with bool immediate. Note no recipe predicate.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("pu_id_bool", f_unary_bool, 4)
+        EncodingRecipeBuilder::new("pu_id_bool", &formats.unary_bool, 4)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1001,7 +957,7 @@ pub(crate) fn define<'shared>(
 
     // XX+rd id nullary with 0 as 32-bit immediate. Note no recipe predicate.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("pu_id_ref", f_nullary, 4)
+        EncodingRecipeBuilder::new("pu_id_ref", &formats.nullary, 4)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1015,7 +971,7 @@ pub(crate) fn define<'shared>(
 
     // XX+rd iq unary with 64-bit immediate.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("pu_iq", f_unary_imm, 8)
+        EncodingRecipeBuilder::new("pu_iq", &formats.unary_imm, 8)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1028,7 +984,7 @@ pub(crate) fn define<'shared>(
 
     // XX+rd id unary with zero immediate.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("u_id_z", f_unary_imm, 1)
+        EncodingRecipeBuilder::new("u_id_z", &formats.unary_imm, 1)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1041,10 +997,10 @@ pub(crate) fn define<'shared>(
     // XX /n Unary with floating point 32-bit immediate equal to zero.
     {
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("f32imm_z", f_unary_ieee32, 1)
+            EncodingRecipeBuilder::new("f32imm_z", &formats.unary_ieee32, 1)
                 .operands_out(vec![fpr])
                 .inst_predicate(InstructionPredicate::new_is_zero_32bit_float(
-                    &*f_unary_ieee32,
+                    &*formats.unary_ieee32,
                     "imm",
                 ))
                 .emit(
@@ -1059,10 +1015,10 @@ pub(crate) fn define<'shared>(
     // XX /n Unary with floating point 64-bit immediate equal to zero.
     {
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("f64imm_z", f_unary_ieee64, 1)
+            EncodingRecipeBuilder::new("f64imm_z", &formats.unary_ieee64, 1)
                 .operands_out(vec![fpr])
                 .inst_predicate(InstructionPredicate::new_is_zero_64bit_float(
-                    &*f_unary_ieee64,
+                    &*formats.unary_ieee64,
                     "imm",
                 ))
                 .emit(
@@ -1075,7 +1031,7 @@ pub(crate) fn define<'shared>(
     }
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("pushq", f_unary, 0)
+        EncodingRecipeBuilder::new("pushq", &formats.unary, 0)
             .operands_in(vec![gpr])
             .emit(
                 r#"
@@ -1086,7 +1042,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("popq", f_nullary, 0)
+        EncodingRecipeBuilder::new("popq", &formats.nullary, 0)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1097,7 +1053,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r, for regmove instructions.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("copysp", f_copy_special, 1)
+        EncodingRecipeBuilder::new("copysp", &formats.copy_special, 1)
             .clobbers_flags(false)
             .emit(
                 r#"
@@ -1108,7 +1064,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("adjustsp", f_unary, 1)
+        EncodingRecipeBuilder::new("adjustsp", &formats.unary, 1)
             .operands_in(vec![gpr])
             .emit(
                 r#"
@@ -1120,9 +1076,9 @@ pub(crate) fn define<'shared>(
 
     {
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("adjustsp_ib", f_unary_imm, 2)
+            EncodingRecipeBuilder::new("adjustsp_ib", &formats.unary_imm, 2)
                 .inst_predicate(InstructionPredicate::new_is_signed_int(
-                    &*f_unary_imm,
+                    &*formats.unary_imm,
                     "imm",
                     8,
                     0,
@@ -1138,9 +1094,9 @@ pub(crate) fn define<'shared>(
         );
 
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("adjustsp_id", f_unary_imm, 5)
+            EncodingRecipeBuilder::new("adjustsp_id", &formats.unary_imm, 5)
                 .inst_predicate(InstructionPredicate::new_is_signed_int(
-                    &*f_unary_imm,
+                    &*formats.unary_imm,
                     "imm",
                     32,
                     0,
@@ -1158,7 +1114,7 @@ pub(crate) fn define<'shared>(
 
     // XX+rd id with Abs4 function relocation.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("fnaddr4", f_func_addr, 4)
+        EncodingRecipeBuilder::new("fnaddr4", &formats.func_addr, 4)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1173,7 +1129,7 @@ pub(crate) fn define<'shared>(
 
     // XX+rd iq with Abs8 function relocation.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("fnaddr8", f_func_addr, 8)
+        EncodingRecipeBuilder::new("fnaddr8", &formats.func_addr, 8)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1188,7 +1144,7 @@ pub(crate) fn define<'shared>(
 
     // Similar to fnaddr4, but writes !0 (this is used by BaldrMonkey).
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("allones_fnaddr4", f_func_addr, 4)
+        EncodingRecipeBuilder::new("allones_fnaddr4", &formats.func_addr, 4)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1204,7 +1160,7 @@ pub(crate) fn define<'shared>(
 
     // Similar to fnaddr8, but writes !0 (this is used by BaldrMonkey).
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("allones_fnaddr8", f_func_addr, 8)
+        EncodingRecipeBuilder::new("allones_fnaddr8", &formats.func_addr, 8)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1219,7 +1175,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("pcrel_fnaddr8", f_func_addr, 5)
+        EncodingRecipeBuilder::new("pcrel_fnaddr8", &formats.func_addr, 5)
             .operands_out(vec![gpr])
             // rex2 gets passed 0 for r/m register because the upper bit of
             // r/m doesn't get decoded when in rip-relative addressing mode.
@@ -1238,7 +1194,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("got_fnaddr8", f_func_addr, 5)
+        EncodingRecipeBuilder::new("got_fnaddr8", &formats.func_addr, 5)
             .operands_out(vec![gpr])
             // rex2 gets passed 0 for r/m register because the upper bit of
             // r/m doesn't get decoded when in rip-relative addressing mode.
@@ -1258,7 +1214,7 @@ pub(crate) fn define<'shared>(
 
     // XX+rd id with Abs4 globalsym relocation.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("gvaddr4", f_unary_global_value, 4)
+        EncodingRecipeBuilder::new("gvaddr4", &formats.unary_global_value, 4)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1273,7 +1229,7 @@ pub(crate) fn define<'shared>(
 
     // XX+rd iq with Abs8 globalsym relocation.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("gvaddr8", f_unary_global_value, 8)
+        EncodingRecipeBuilder::new("gvaddr8", &formats.unary_global_value, 8)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1288,7 +1244,7 @@ pub(crate) fn define<'shared>(
 
     // XX+rd iq with PCRel4 globalsym relocation.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("pcrel_gvaddr8", f_unary_global_value, 5)
+        EncodingRecipeBuilder::new("pcrel_gvaddr8", &formats.unary_global_value, 5)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1306,7 +1262,7 @@ pub(crate) fn define<'shared>(
 
     // XX+rd iq with Abs8 globalsym relocation.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("got_gvaddr8", f_unary_global_value, 5)
+        EncodingRecipeBuilder::new("got_gvaddr8", &formats.unary_global_value, 5)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1327,7 +1283,7 @@ pub(crate) fn define<'shared>(
     // TODO Alternative forms for 8-bit immediates, when applicable.
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("spaddr4_id", f_stack_load, 6)
+        EncodingRecipeBuilder::new("spaddr4_id", &formats.stack_load, 6)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1343,7 +1299,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("spaddr8_id", f_stack_load, 6)
+        EncodingRecipeBuilder::new("spaddr8_id", &formats.stack_load, 6)
             .operands_out(vec![gpr])
             .emit(
                 r#"
@@ -1365,11 +1321,11 @@ pub(crate) fn define<'shared>(
 
         // A predicate asking if the offset is zero.
         let has_no_offset =
-            InstructionPredicate::new_is_field_equal(&*f_store, "offset", "0".into());
+            InstructionPredicate::new_is_field_equal(&*formats.store, "offset", "0".into());
 
         // XX /r register-indirect store with no offset.
         let st = recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("st", f_store, 1)
+            EncodingRecipeBuilder::new("st", &formats.store, 1)
                 .operands_in(vec![gpr, gpr])
                 .inst_predicate(has_no_offset.clone())
                 .clobbers_flags(false)
@@ -1397,7 +1353,7 @@ pub(crate) fn define<'shared>(
         // Only ABCD allowed for stored value. This is for byte stores with no REX.
         recipes.add_template(
             Template::new(
-                EncodingRecipeBuilder::new("st_abcd", f_store, 1)
+                EncodingRecipeBuilder::new("st_abcd", &formats.store, 1)
                     .operands_in(vec![abcd, gpr])
                     .inst_predicate(has_no_offset.clone())
                     .clobbers_flags(false)
@@ -1426,7 +1382,7 @@ pub(crate) fn define<'shared>(
 
         // XX /r register-indirect store of FPR with no offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("fst", f_store, 1)
+            EncodingRecipeBuilder::new("fst", &formats.store, 1)
                 .operands_in(vec![fpr, gpr])
                 .inst_predicate(has_no_offset.clone())
                 .clobbers_flags(false)
@@ -1450,11 +1406,12 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let has_small_offset = InstructionPredicate::new_is_signed_int(&*f_store, "offset", 8, 0);
+        let has_small_offset =
+            InstructionPredicate::new_is_signed_int(&*formats.store, "offset", 8, 0);
 
         // XX /r register-indirect store with 8-bit offset.
         let st_disp8 = recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("stDisp8", f_store, 2)
+            EncodingRecipeBuilder::new("stDisp8", &formats.store, 2)
                 .operands_in(vec![gpr, gpr])
                 .inst_predicate(has_small_offset.clone())
                 .clobbers_flags(false)
@@ -1481,7 +1438,7 @@ pub(crate) fn define<'shared>(
         // Only ABCD allowed for stored value. This is for byte stores with no REX.
         recipes.add_template(
             Template::new(
-                EncodingRecipeBuilder::new("stDisp8_abcd", f_store, 2)
+                EncodingRecipeBuilder::new("stDisp8_abcd", &formats.store, 2)
                     .operands_in(vec![abcd, gpr])
                     .inst_predicate(has_small_offset.clone())
                     .clobbers_flags(false)
@@ -1509,7 +1466,7 @@ pub(crate) fn define<'shared>(
 
         // XX /r register-indirect store with 8-bit offset of FPR.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("fstDisp8", f_store, 2)
+            EncodingRecipeBuilder::new("fstDisp8", &formats.store, 2)
                 .operands_in(vec![fpr, gpr])
                 .inst_predicate(has_small_offset.clone())
                 .clobbers_flags(false)
@@ -1534,7 +1491,7 @@ pub(crate) fn define<'shared>(
 
         // XX /r register-indirect store with 32-bit offset.
         let st_disp32 = recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("stDisp32", f_store, 5)
+            EncodingRecipeBuilder::new("stDisp32", &formats.store, 5)
                 .operands_in(vec![gpr, gpr])
                 .clobbers_flags(false)
                 .compute_size("size_plus_maybe_sib_for_in_reg_1")
@@ -1560,7 +1517,7 @@ pub(crate) fn define<'shared>(
         // Only ABCD allowed for stored value. This is for byte stores with no REX.
         recipes.add_template(
             Template::new(
-                EncodingRecipeBuilder::new("stDisp32_abcd", f_store, 5)
+                EncodingRecipeBuilder::new("stDisp32_abcd", &formats.store, 5)
                     .operands_in(vec![abcd, gpr])
                     .clobbers_flags(false)
                     .compute_size("size_plus_maybe_sib_for_in_reg_1")
@@ -1587,7 +1544,7 @@ pub(crate) fn define<'shared>(
 
         // XX /r register-indirect store with 32-bit offset of FPR.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("fstDisp32", f_store, 5)
+            EncodingRecipeBuilder::new("fstDisp32", &formats.store, 5)
                 .operands_in(vec![fpr, gpr])
                 .clobbers_flags(false)
                 .compute_size("size_plus_maybe_sib_for_in_reg_1")
@@ -1615,11 +1572,11 @@ pub(crate) fn define<'shared>(
 
         // A predicate asking if the offset is zero.
         let has_no_offset =
-            InstructionPredicate::new_is_field_equal(&*f_store_complex, "offset", "0".into());
+            InstructionPredicate::new_is_field_equal(&*formats.store_complex, "offset", "0".into());
 
         // XX /r register-indirect store with index and no offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("stWithIndex", f_store_complex, 2)
+            EncodingRecipeBuilder::new("stWithIndex", &formats.store_complex, 2)
                 .operands_in(vec![gpr, gpr, gpr])
                 .inst_predicate(has_no_offset.clone())
                 .clobbers_flags(false)
@@ -1646,7 +1603,7 @@ pub(crate) fn define<'shared>(
         // XX /r register-indirect store with index and no offset.
         // Only ABCD allowed for stored value. This is for byte stores with no REX.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("stWithIndex_abcd", f_store_complex, 2)
+            EncodingRecipeBuilder::new("stWithIndex_abcd", &formats.store_complex, 2)
                 .operands_in(vec![abcd, gpr, gpr])
                 .inst_predicate(has_no_offset.clone())
                 .clobbers_flags(false)
@@ -1672,7 +1629,7 @@ pub(crate) fn define<'shared>(
 
         // XX /r register-indirect store with index and no offset of FPR.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("fstWithIndex", f_store_complex, 2)
+            EncodingRecipeBuilder::new("fstWithIndex", &formats.store_complex, 2)
                 .operands_in(vec![fpr, gpr, gpr])
                 .inst_predicate(has_no_offset.clone())
                 .clobbers_flags(false)
@@ -1697,11 +1654,11 @@ pub(crate) fn define<'shared>(
         );
 
         let has_small_offset =
-            InstructionPredicate::new_is_signed_int(&*f_store_complex, "offset", 8, 0);
+            InstructionPredicate::new_is_signed_int(&*formats.store_complex, "offset", 8, 0);
 
         // XX /r register-indirect store with index and 8-bit offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("stWithIndexDisp8", f_store_complex, 3)
+            EncodingRecipeBuilder::new("stWithIndexDisp8", &formats.store_complex, 3)
                 .operands_in(vec![gpr, gpr, gpr])
                 .inst_predicate(has_small_offset.clone())
                 .clobbers_flags(false)
@@ -1722,7 +1679,7 @@ pub(crate) fn define<'shared>(
         // XX /r register-indirect store with index and 8-bit offset.
         // Only ABCD allowed for stored value. This is for byte stores with no REX.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("stWithIndexDisp8_abcd", f_store_complex, 3)
+            EncodingRecipeBuilder::new("stWithIndexDisp8_abcd", &formats.store_complex, 3)
                 .operands_in(vec![abcd, gpr, gpr])
                 .inst_predicate(has_small_offset.clone())
                 .clobbers_flags(false)
@@ -1742,7 +1699,7 @@ pub(crate) fn define<'shared>(
 
         // XX /r register-indirect store with index and 8-bit offset of FPR.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("fstWithIndexDisp8", f_store_complex, 3)
+            EncodingRecipeBuilder::new("fstWithIndexDisp8", &formats.store_complex, 3)
                 .operands_in(vec![fpr, gpr, gpr])
                 .inst_predicate(has_small_offset.clone())
                 .clobbers_flags(false)
@@ -1761,11 +1718,11 @@ pub(crate) fn define<'shared>(
         );
 
         let has_big_offset =
-            InstructionPredicate::new_is_signed_int(&*f_store_complex, "offset", 32, 0);
+            InstructionPredicate::new_is_signed_int(&*formats.store_complex, "offset", 32, 0);
 
         // XX /r register-indirect store with index and 32-bit offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("stWithIndexDisp32", f_store_complex, 6)
+            EncodingRecipeBuilder::new("stWithIndexDisp32", &formats.store_complex, 6)
                 .operands_in(vec![gpr, gpr, gpr])
                 .inst_predicate(has_big_offset.clone())
                 .clobbers_flags(false)
@@ -1786,7 +1743,7 @@ pub(crate) fn define<'shared>(
         // XX /r register-indirect store with index and 32-bit offset.
         // Only ABCD allowed for stored value. This is for byte stores with no REX.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("stWithIndexDisp32_abcd", f_store_complex, 6)
+            EncodingRecipeBuilder::new("stWithIndexDisp32_abcd", &formats.store_complex, 6)
                 .operands_in(vec![abcd, gpr, gpr])
                 .inst_predicate(has_big_offset.clone())
                 .clobbers_flags(false)
@@ -1806,7 +1763,7 @@ pub(crate) fn define<'shared>(
 
         // XX /r register-indirect store with index and 32-bit offset of FPR.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("fstWithIndexDisp32", f_store_complex, 6)
+            EncodingRecipeBuilder::new("fstWithIndexDisp32", &formats.store_complex, 6)
                 .operands_in(vec![fpr, gpr, gpr])
                 .inst_predicate(has_big_offset.clone())
                 .clobbers_flags(false)
@@ -1827,7 +1784,7 @@ pub(crate) fn define<'shared>(
 
     // Unary spill with SIB and 32-bit displacement.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("spillSib32", f_unary, 6)
+        EncodingRecipeBuilder::new("spillSib32", &formats.unary, 6)
             .operands_in(vec![gpr])
             .operands_out(vec![stack_gpr32])
             .clobbers_flags(false)
@@ -1845,7 +1802,7 @@ pub(crate) fn define<'shared>(
 
     // Like spillSib32, but targeting an FPR rather than a GPR.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("fspillSib32", f_unary, 6)
+        EncodingRecipeBuilder::new("fspillSib32", &formats.unary, 6)
             .operands_in(vec![fpr])
             .operands_out(vec![stack_fpr32])
             .clobbers_flags(false)
@@ -1863,7 +1820,7 @@ pub(crate) fn define<'shared>(
 
     // Regspill using RSP-relative addressing.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("regspill32", f_reg_spill, 6)
+        EncodingRecipeBuilder::new("regspill32", &formats.reg_spill, 6)
             .operands_in(vec![gpr])
             .clobbers_flags(false)
             .emit(
@@ -1881,7 +1838,7 @@ pub(crate) fn define<'shared>(
 
     // Like regspill32, but targeting an FPR rather than a GPR.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("fregspill32", f_reg_spill, 6)
+        EncodingRecipeBuilder::new("fregspill32", &formats.reg_spill, 6)
             .operands_in(vec![fpr])
             .clobbers_flags(false)
             .emit(
@@ -1904,11 +1861,11 @@ pub(crate) fn define<'shared>(
 
         // A predicate asking if the offset is zero.
         let has_no_offset =
-            InstructionPredicate::new_is_field_equal(&*f_load, "offset", "0".into());
+            InstructionPredicate::new_is_field_equal(&*formats.load, "offset", "0".into());
 
         // XX /r load with no offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("ld", f_load, 1)
+            EncodingRecipeBuilder::new("ld", &formats.load, 1)
                 .operands_in(vec![gpr])
                 .operands_out(vec![gpr])
                 .inst_predicate(has_no_offset.clone())
@@ -1935,7 +1892,7 @@ pub(crate) fn define<'shared>(
 
         // XX /r float load with no offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("fld", f_load, 1)
+            EncodingRecipeBuilder::new("fld", &formats.load, 1)
                 .operands_in(vec![gpr])
                 .operands_out(vec![fpr])
                 .inst_predicate(has_no_offset.clone())
@@ -1960,11 +1917,12 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let has_small_offset = InstructionPredicate::new_is_signed_int(&*f_load, "offset", 8, 0);
+        let has_small_offset =
+            InstructionPredicate::new_is_signed_int(&*formats.load, "offset", 8, 0);
 
         // XX /r load with 8-bit offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("ldDisp8", f_load, 2)
+            EncodingRecipeBuilder::new("ldDisp8", &formats.load, 2)
                 .operands_in(vec![gpr])
                 .operands_out(vec![gpr])
                 .inst_predicate(has_small_offset.clone())
@@ -1990,7 +1948,7 @@ pub(crate) fn define<'shared>(
 
         // XX /r float load with 8-bit offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("fldDisp8", f_load, 2)
+            EncodingRecipeBuilder::new("fldDisp8", &formats.load, 2)
                 .operands_in(vec![gpr])
                 .operands_out(vec![fpr])
                 .inst_predicate(has_small_offset.clone())
@@ -2014,11 +1972,12 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let has_big_offset = InstructionPredicate::new_is_signed_int(&*f_load, "offset", 32, 0);
+        let has_big_offset =
+            InstructionPredicate::new_is_signed_int(&*formats.load, "offset", 32, 0);
 
         // XX /r load with 32-bit offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("ldDisp32", f_load, 5)
+            EncodingRecipeBuilder::new("ldDisp32", &formats.load, 5)
                 .operands_in(vec![gpr])
                 .operands_out(vec![gpr])
                 .inst_predicate(has_big_offset.clone())
@@ -2044,7 +2003,7 @@ pub(crate) fn define<'shared>(
 
         // XX /r float load with 32-bit offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("fldDisp32", f_load, 5)
+            EncodingRecipeBuilder::new("fldDisp32", &formats.load, 5)
                 .operands_in(vec![gpr])
                 .operands_out(vec![fpr])
                 .inst_predicate(has_big_offset.clone())
@@ -2074,11 +2033,11 @@ pub(crate) fn define<'shared>(
 
         // A predicate asking if the offset is zero.
         let has_no_offset =
-            InstructionPredicate::new_is_field_equal(&*f_load_complex, "offset", "0".into());
+            InstructionPredicate::new_is_field_equal(&*formats.load_complex, "offset", "0".into());
 
         // XX /r load with index and no offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("ldWithIndex", f_load_complex, 2)
+            EncodingRecipeBuilder::new("ldWithIndex", &formats.load_complex, 2)
                 .operands_in(vec![gpr, gpr])
                 .operands_out(vec![gpr])
                 .inst_predicate(has_no_offset.clone())
@@ -2105,7 +2064,7 @@ pub(crate) fn define<'shared>(
 
         // XX /r float load with index and no offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("fldWithIndex", f_load_complex, 2)
+            EncodingRecipeBuilder::new("fldWithIndex", &formats.load_complex, 2)
                 .operands_in(vec![gpr, gpr])
                 .operands_out(vec![fpr])
                 .inst_predicate(has_no_offset.clone())
@@ -2131,11 +2090,11 @@ pub(crate) fn define<'shared>(
         );
 
         let has_small_offset =
-            InstructionPredicate::new_is_signed_int(&*f_load_complex, "offset", 8, 0);
+            InstructionPredicate::new_is_signed_int(&*formats.load_complex, "offset", 8, 0);
 
         // XX /r load with index and 8-bit offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("ldWithIndexDisp8", f_load_complex, 3)
+            EncodingRecipeBuilder::new("ldWithIndexDisp8", &formats.load_complex, 3)
                 .operands_in(vec![gpr, gpr])
                 .operands_out(vec![gpr])
                 .inst_predicate(has_small_offset.clone())
@@ -2156,7 +2115,7 @@ pub(crate) fn define<'shared>(
 
         // XX /r float load with 8-bit offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("fldWithIndexDisp8", f_load_complex, 3)
+            EncodingRecipeBuilder::new("fldWithIndexDisp8", &formats.load_complex, 3)
                 .operands_in(vec![gpr, gpr])
                 .operands_out(vec![fpr])
                 .inst_predicate(has_small_offset.clone())
@@ -2176,11 +2135,11 @@ pub(crate) fn define<'shared>(
         );
 
         let has_big_offset =
-            InstructionPredicate::new_is_signed_int(&*f_load_complex, "offset", 32, 0);
+            InstructionPredicate::new_is_signed_int(&*formats.load_complex, "offset", 32, 0);
 
         // XX /r load with index and 32-bit offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("ldWithIndexDisp32", f_load_complex, 6)
+            EncodingRecipeBuilder::new("ldWithIndexDisp32", &formats.load_complex, 6)
                 .operands_in(vec![gpr, gpr])
                 .operands_out(vec![gpr])
                 .inst_predicate(has_big_offset.clone())
@@ -2201,7 +2160,7 @@ pub(crate) fn define<'shared>(
 
         // XX /r float load with index and 32-bit offset.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("fldWithIndexDisp32", f_load_complex, 6)
+            EncodingRecipeBuilder::new("fldWithIndexDisp32", &formats.load_complex, 6)
                 .operands_in(vec![gpr, gpr])
                 .operands_out(vec![fpr])
                 .inst_predicate(has_big_offset.clone())
@@ -2223,7 +2182,7 @@ pub(crate) fn define<'shared>(
 
     // Unary fill with SIB and 32-bit displacement.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("fillSib32", f_unary, 6)
+        EncodingRecipeBuilder::new("fillSib32", &formats.unary, 6)
             .operands_in(vec![stack_gpr32])
             .operands_out(vec![gpr])
             .clobbers_flags(false)
@@ -2240,7 +2199,7 @@ pub(crate) fn define<'shared>(
 
     // Like fillSib32, but targeting an FPR rather than a GPR.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("ffillSib32", f_unary, 6)
+        EncodingRecipeBuilder::new("ffillSib32", &formats.unary, 6)
             .operands_in(vec![stack_fpr32])
             .operands_out(vec![fpr])
             .clobbers_flags(false)
@@ -2257,7 +2216,7 @@ pub(crate) fn define<'shared>(
 
     // Regfill with RSP-relative 32-bit displacement.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("regfill32", f_reg_fill, 6)
+        EncodingRecipeBuilder::new("regfill32", &formats.reg_fill, 6)
             .operands_in(vec![stack_gpr32])
             .clobbers_flags(false)
             .emit(
@@ -2274,7 +2233,7 @@ pub(crate) fn define<'shared>(
 
     // Like regfill32, but targeting an FPR rather than a GPR.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("fregfill32", f_reg_fill, 6)
+        EncodingRecipeBuilder::new("fregfill32", &formats.reg_fill, 6)
             .operands_in(vec![stack_fpr32])
             .clobbers_flags(false)
             .emit(
@@ -2291,8 +2250,9 @@ pub(crate) fn define<'shared>(
 
     // Call/return.
 
-    recipes.add_template_recipe(EncodingRecipeBuilder::new("call_id", f_call, 4).emit(
-        r#"
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("call_id", &formats.call, 4).emit(
+            r#"
             sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
             {{PUT_OP}}(bits, BASE_REX, sink);
             // The addend adjusts for the difference between the end of the
@@ -2302,10 +2262,12 @@ pub(crate) fn define<'shared>(
                                 -4);
             sink.put4(0);
         "#,
-    ));
+        ),
+    );
 
-    recipes.add_template_recipe(EncodingRecipeBuilder::new("call_plt_id", f_call, 4).emit(
-        r#"
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("call_plt_id", &formats.call, 4).emit(
+            r#"
             sink.trap(TrapCode::StackOverflow, func.srclocs[inst]);
             {{PUT_OP}}(bits, BASE_REX, sink);
             sink.reloc_external(Reloc::X86CallPLTRel4,
@@ -2313,10 +2275,11 @@ pub(crate) fn define<'shared>(
                                 -4);
             sink.put4(0);
         "#,
-    ));
+        ),
+    );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("call_r", f_call_indirect, 1)
+        EncodingRecipeBuilder::new("call_r", &formats.call_indirect, 1)
             .operands_in(vec![gpr])
             .emit(
                 r#"
@@ -2328,13 +2291,14 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("ret", f_multiary, 0).emit("{{PUT_OP}}(bits, BASE_REX, sink);"),
+        EncodingRecipeBuilder::new("ret", &formats.multiary, 0)
+            .emit("{{PUT_OP}}(bits, BASE_REX, sink);"),
     );
 
     // Branches.
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("jmpb", f_jump, 1)
+        EncodingRecipeBuilder::new("jmpb", &formats.jump, 1)
             .branch_range((1, 8))
             .clobbers_flags(false)
             .emit(
@@ -2346,7 +2310,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("jmpd", f_jump, 4)
+        EncodingRecipeBuilder::new("jmpd", &formats.jump, 4)
             .branch_range((4, 32))
             .clobbers_flags(false)
             .emit(
@@ -2358,7 +2322,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("brib", f_branch_int, 1)
+        EncodingRecipeBuilder::new("brib", &formats.branch_int, 1)
             .operands_in(vec![reg_rflags])
             .branch_range((1, 8))
             .clobbers_flags(false)
@@ -2371,7 +2335,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("brid", f_branch_int, 4)
+        EncodingRecipeBuilder::new("brid", &formats.branch_int, 4)
             .operands_in(vec![reg_rflags])
             .branch_range((4, 32))
             .clobbers_flags(false)
@@ -2384,13 +2348,13 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("brfb", f_branch_float, 1)
+        EncodingRecipeBuilder::new("brfb", &formats.branch_float, 1)
             .operands_in(vec![reg_rflags])
             .branch_range((1, 8))
             .clobbers_flags(false)
             .inst_predicate(supported_floatccs_predicate(
                 &supported_floatccs,
-                &*f_branch_float,
+                &*formats.branch_float,
             ))
             .emit(
                 r#"
@@ -2401,13 +2365,13 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("brfd", f_branch_float, 4)
+        EncodingRecipeBuilder::new("brfd", &formats.branch_float, 4)
             .operands_in(vec![reg_rflags])
             .branch_range((4, 32))
             .clobbers_flags(false)
             .inst_predicate(supported_floatccs_predicate(
                 &supported_floatccs,
-                &*f_branch_float,
+                &*formats.branch_float,
             ))
             .emit(
                 r#"
@@ -2418,7 +2382,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("indirect_jmp", f_indirect_jump, 1)
+        EncodingRecipeBuilder::new("indirect_jmp", &formats.indirect_jump, 1)
             .operands_in(vec![gpr])
             .clobbers_flags(false)
             .emit(
@@ -2430,11 +2394,11 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("jt_entry", f_branch_table_entry, 2)
+        EncodingRecipeBuilder::new("jt_entry", &formats.branch_table_entry, 2)
             .operands_in(vec![gpr, gpr])
             .operands_out(vec![gpr])
             .clobbers_flags(false)
-            .inst_predicate(valid_scale(&*f_branch_table_entry))
+            .inst_predicate(valid_scale(&*formats.branch_table_entry))
             .compute_size("size_plus_maybe_offset_for_in_reg_1")
             .emit(
                 r#"
@@ -2452,7 +2416,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("vconst", f_unary_const, 5)
+        EncodingRecipeBuilder::new("vconst", &formats.unary_const, 5)
             .operands_out(vec![fpr])
             .clobbers_flags(false)
             .emit(
@@ -2465,7 +2429,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("vconst_optimized", f_unary_const, 1)
+        EncodingRecipeBuilder::new("vconst_optimized", &formats.unary_const, 1)
             .operands_out(vec![fpr])
             .clobbers_flags(false)
             .emit(
@@ -2477,7 +2441,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("jt_base", f_branch_table_base, 5)
+        EncodingRecipeBuilder::new("jt_base", &formats.branch_table_base, 5)
             .operands_out(vec![gpr])
             .clobbers_flags(false)
             .emit(
@@ -2502,7 +2466,7 @@ pub(crate) fn define<'shared>(
 
     let seti = recipes.add_template(
         Template::new(
-            EncodingRecipeBuilder::new("seti", f_int_cond, 1)
+            EncodingRecipeBuilder::new("seti", &formats.int_cond, 1)
                 .operands_in(vec![reg_rflags])
                 .operands_out(vec![gpr])
                 .clobbers_flags(false)
@@ -2519,7 +2483,7 @@ pub(crate) fn define<'shared>(
 
     recipes.add_template(
         Template::new(
-            EncodingRecipeBuilder::new("seti_abcd", f_int_cond, 1)
+            EncodingRecipeBuilder::new("seti_abcd", &formats.int_cond, 1)
                 .operands_in(vec![reg_rflags])
                 .operands_out(vec![abcd])
                 .clobbers_flags(false)
@@ -2536,7 +2500,7 @@ pub(crate) fn define<'shared>(
 
     let setf = recipes.add_template(
         Template::new(
-            EncodingRecipeBuilder::new("setf", f_float_cond, 1)
+            EncodingRecipeBuilder::new("setf", &formats.float_cond, 1)
                 .operands_in(vec![reg_rflags])
                 .operands_out(vec![gpr])
                 .clobbers_flags(false)
@@ -2553,7 +2517,7 @@ pub(crate) fn define<'shared>(
 
     recipes.add_template(
         Template::new(
-            EncodingRecipeBuilder::new("setf_abcd", f_float_cond, 1)
+            EncodingRecipeBuilder::new("setf_abcd", &formats.float_cond, 1)
                 .operands_in(vec![reg_rflags])
                 .operands_out(vec![abcd])
                 .clobbers_flags(false)
@@ -2572,7 +2536,7 @@ pub(crate) fn define<'shared>(
     // (maybe-REX.W) 0F 4x modrm(r,r)
     // 1 byte, modrm(r,r), is after the opcode
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("cmov", f_int_select, 1)
+        EncodingRecipeBuilder::new("cmov", &formats.int_select, 1)
             .operands_in(vec![
                 OperandConstraint::FixedReg(reg_rflags),
                 OperandConstraint::RegClass(gpr),
@@ -2590,7 +2554,7 @@ pub(crate) fn define<'shared>(
 
     // Bit scan forwards and reverse
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("bsf_and_bsr", f_unary, 1)
+        EncodingRecipeBuilder::new("bsf_and_bsr", &formats.unary, 1)
             .operands_in(vec![gpr])
             .operands_out(vec![
                 OperandConstraint::RegClass(gpr),
@@ -2608,7 +2572,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r, MR form. Add two GPR registers and set carry flag.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rout", f_binary, 1)
+        EncodingRecipeBuilder::new("rout", &formats.binary, 1)
             .operands_in(vec![gpr, gpr])
             .operands_out(vec![
                 OperandConstraint::TiedInput(0),
@@ -2625,7 +2589,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r, MR form. Add two GPR registers and get carry flag.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rin", f_ternary, 1)
+        EncodingRecipeBuilder::new("rin", &formats.ternary, 1)
             .operands_in(vec![
                 OperandConstraint::RegClass(gpr),
                 OperandConstraint::RegClass(gpr),
@@ -2643,7 +2607,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r, MR form. Add two GPR registers with carry flag.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rio", f_ternary, 1)
+        EncodingRecipeBuilder::new("rio", &formats.ternary, 1)
             .operands_in(vec![
                 OperandConstraint::RegClass(gpr),
                 OperandConstraint::RegClass(gpr),
@@ -2666,7 +2630,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r, MR form. Compare two GPR registers and set flags.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rcmp", f_binary, 1)
+        EncodingRecipeBuilder::new("rcmp", &formats.binary, 1)
             .operands_in(vec![gpr, gpr])
             .operands_out(vec![reg_rflags])
             .emit(
@@ -2679,7 +2643,7 @@ pub(crate) fn define<'shared>(
 
     // Same as rcmp, but second operand is the stack pointer.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rcmp_sp", f_unary, 1)
+        EncodingRecipeBuilder::new("rcmp_sp", &formats.unary, 1)
             .operands_in(vec![gpr])
             .operands_out(vec![reg_rflags])
             .emit(
@@ -2692,7 +2656,7 @@ pub(crate) fn define<'shared>(
 
     // XX /r, RM form. Compare two FPR registers and set flags.
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("fcmp", f_binary, 1)
+        EncodingRecipeBuilder::new("fcmp", &formats.binary, 1)
             .operands_in(vec![fpr, fpr])
             .operands_out(vec![reg_rflags])
             .emit(
@@ -2704,11 +2668,12 @@ pub(crate) fn define<'shared>(
     );
 
     {
-        let has_small_offset = InstructionPredicate::new_is_signed_int(&*f_binary_imm, "imm", 8, 0);
+        let has_small_offset =
+            InstructionPredicate::new_is_signed_int(&*formats.binary_imm, "imm", 8, 0);
 
         // XX /n, MI form with imm8.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("rcmp_ib", f_binary_imm, 2)
+            EncodingRecipeBuilder::new("rcmp_ib", &formats.binary_imm, 2)
                 .operands_in(vec![gpr])
                 .operands_out(vec![reg_rflags])
                 .inst_predicate(has_small_offset)
@@ -2722,11 +2687,12 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let has_big_offset = InstructionPredicate::new_is_signed_int(&*f_binary_imm, "imm", 32, 0);
+        let has_big_offset =
+            InstructionPredicate::new_is_signed_int(&*formats.binary_imm, "imm", 32, 0);
 
         // XX /n, MI form with imm32.
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("rcmp_id", f_binary_imm, 5)
+            EncodingRecipeBuilder::new("rcmp_id", &formats.binary_imm, 5)
                 .operands_in(vec![gpr])
                 .operands_out(vec![reg_rflags])
                 .inst_predicate(has_big_offset)
@@ -2756,7 +2722,7 @@ pub(crate) fn define<'shared>(
     // Bits 8-15 control the test instruction which always has opcode byte 0x85.
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("tjccb", f_branch, 1 + 2)
+        EncodingRecipeBuilder::new("tjccb", &formats.branch, 1 + 2)
             .operands_in(vec![gpr])
             .branch_range((3, 8))
             .emit(
@@ -2772,7 +2738,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("tjccd", f_branch, 1 + 6)
+        EncodingRecipeBuilder::new("tjccd", &formats.branch, 1 + 6)
             .operands_in(vec![gpr])
             .branch_range((7, 32))
             .emit(
@@ -2792,7 +2758,7 @@ pub(crate) fn define<'shared>(
 
     let t8jccb = recipes.add_template(
         Template::new(
-            EncodingRecipeBuilder::new("t8jccb", f_branch, 1 + 2)
+            EncodingRecipeBuilder::new("t8jccb", &formats.branch, 1 + 2)
                 .operands_in(vec![gpr])
                 .branch_range((3, 8))
                 .emit(
@@ -2812,7 +2778,7 @@ pub(crate) fn define<'shared>(
 
     recipes.add_template(
         Template::new(
-            EncodingRecipeBuilder::new("t8jccb_abcd", f_branch, 1 + 2)
+            EncodingRecipeBuilder::new("t8jccb_abcd", &formats.branch, 1 + 2)
                 .operands_in(vec![abcd])
                 .branch_range((3, 8))
                 .emit(
@@ -2832,7 +2798,7 @@ pub(crate) fn define<'shared>(
 
     let t8jccd = recipes.add_template(
         Template::new(
-            EncodingRecipeBuilder::new("t8jccd", f_branch, 1 + 6)
+            EncodingRecipeBuilder::new("t8jccd", &formats.branch, 1 + 6)
                 .operands_in(vec![gpr])
                 .branch_range((7, 32))
                 .emit(
@@ -2853,7 +2819,7 @@ pub(crate) fn define<'shared>(
 
     recipes.add_template(
         Template::new(
-            EncodingRecipeBuilder::new("t8jccd_abcd", f_branch, 1 + 6)
+            EncodingRecipeBuilder::new("t8jccd_abcd", &formats.branch, 1 + 6)
                 .operands_in(vec![abcd])
                 .branch_range((7, 32))
                 .emit(
@@ -2879,7 +2845,7 @@ pub(crate) fn define<'shared>(
     // a 0xff immediate.
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("t8jccd_long", f_branch, 5 + 6)
+        EncodingRecipeBuilder::new("t8jccd_long", &formats.branch, 5 + 6)
             .operands_in(vec![gpr])
             .branch_range((11, 32))
             .emit(
@@ -2916,7 +2882,7 @@ pub(crate) fn define<'shared>(
     // The omission of a `when_prefixed` alternative is deliberate here.
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("icscc", f_int_compare, 1 + 3)
+        EncodingRecipeBuilder::new("icscc", &formats.int_compare, 1 + 3)
             .operands_in(vec![gpr, gpr])
             .operands_out(vec![abcd])
             .emit(
@@ -2934,7 +2900,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("icscc_fpr", f_int_compare, 1)
+        EncodingRecipeBuilder::new("icscc_fpr", &formats.int_compare, 1)
             .operands_in(vec![fpr, fpr])
             .operands_out(vec![0])
             .emit(
@@ -2948,10 +2914,10 @@ pub(crate) fn define<'shared>(
 
     {
         let is_small_imm =
-            InstructionPredicate::new_is_signed_int(&*f_int_compare_imm, "imm", 8, 0);
+            InstructionPredicate::new_is_signed_int(&*formats.int_compare_imm, "imm", 8, 0);
 
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("icscc_ib", f_int_compare_imm, 2 + 3)
+            EncodingRecipeBuilder::new("icscc_ib", &formats.int_compare_imm, 2 + 3)
                 .operands_in(vec![gpr])
                 .operands_out(vec![abcd])
                 .inst_predicate(is_small_imm)
@@ -2971,10 +2937,11 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
-        let is_big_imm = InstructionPredicate::new_is_signed_int(&*f_int_compare_imm, "imm", 32, 0);
+        let is_big_imm =
+            InstructionPredicate::new_is_signed_int(&*formats.int_compare_imm, "imm", 32, 0);
 
         recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("icscc_id", f_int_compare_imm, 5 + 3)
+            EncodingRecipeBuilder::new("icscc_id", &formats.int_compare_imm, 5 + 3)
                 .operands_in(vec![gpr])
                 .operands_out(vec![abcd])
                 .inst_predicate(is_big_imm)
@@ -3011,12 +2978,12 @@ pub(crate) fn define<'shared>(
     // The omission of a `when_prefixed` alternative is deliberate here.
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("fcscc", f_float_compare, 1 + 3)
+        EncodingRecipeBuilder::new("fcscc", &formats.float_compare, 1 + 3)
             .operands_in(vec![fpr, fpr])
             .operands_out(vec![abcd])
             .inst_predicate(supported_floatccs_predicate(
                 &supported_floatccs,
-                &*f_float_compare,
+                &*formats.float_compare,
             ))
             .emit(
                 r#"
@@ -3050,7 +3017,7 @@ pub(crate) fn define<'shared>(
     );
 
     recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("is_zero", f_unary, 2 + 2)
+        EncodingRecipeBuilder::new("is_zero", &formats.unary, 2 + 2)
             .operands_in(vec![gpr])
             .operands_out(vec![abcd])
             .emit(
@@ -3066,11 +3033,13 @@ pub(crate) fn define<'shared>(
             ),
     );
 
-    recipes.add_recipe(EncodingRecipeBuilder::new("safepoint", f_multiary, 0).emit(
-        r#"
+    recipes.add_recipe(
+        EncodingRecipeBuilder::new("safepoint", &formats.multiary, 0).emit(
+            r#"
             sink.add_stackmap(args, func, isa);
         "#,
-    ));
+        ),
+    );
 
     recipes
 }
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index 1513b09e18..00fae4b455 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -23,7 +23,8 @@ pub fn isa_from_arch(arch: &str) -> Result {
 
 /// Generates all the Rust source files used in Cranelift from the meta-language.
 pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error> {
-    // Common definitions.
+    // Create all the definitions:
+    // - common definitions.
     let mut shared_defs = shared::define();
 
     gen_settings::generate(
@@ -34,10 +35,20 @@ pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error>
     )?;
     gen_types::generate("types.rs", &out_dir)?;
 
-    // Per ISA definitions.
+    // - per ISA definitions.
     let isas = isa::define(isas, &mut shared_defs);
 
-    gen_inst::generate(&shared_defs, "opcodes.rs", "inst_builder.rs", &out_dir)?;
+    // At this point, all definitions are done.
+    let all_formats = shared_defs.verify_instruction_formats();
+
+    // Generate all the code.
+    gen_inst::generate(
+        all_formats,
+        &shared_defs.all_instructions,
+        "opcodes.rs",
+        "inst_builder.rs",
+        &out_dir,
+    )?;
 
     gen_legalizer::generate(&isas, &shared_defs.transform_groups, "legalize", &out_dir)?;
 
diff --git a/cranelift/codegen/meta/src/shared/formats.rs b/cranelift/codegen/meta/src/shared/formats.rs
index 73aaa42634..3cc3d343d7 100644
--- a/cranelift/codegen/meta/src/shared/formats.rs
+++ b/cranelift/codegen/meta/src/shared/formats.rs
@@ -1,226 +1,303 @@
-use crate::cdsl::formats::{FormatRegistry, InstructionFormatBuilder as Builder};
+use crate::cdsl::formats::{InstructionFormat, InstructionFormatBuilder as Builder};
 use crate::shared::{entities::EntityRefs, immediates::Immediates};
+use std::rc::Rc;
 
-pub(crate) fn define(imm: &Immediates, entities: &EntityRefs) -> FormatRegistry {
-    let mut registry = FormatRegistry::new();
-
-    registry.insert(Builder::new("Unary").value());
-    registry.insert(Builder::new("UnaryImm").imm(&imm.imm64));
-    registry.insert(Builder::new("UnaryIeee32").imm(&imm.ieee32));
-    registry.insert(Builder::new("UnaryIeee64").imm(&imm.ieee64));
-    registry.insert(Builder::new("UnaryBool").imm(&imm.boolean));
-    registry.insert(Builder::new("UnaryConst").imm(&imm.pool_constant));
-    registry.insert(Builder::new("UnaryGlobalValue").imm(&entities.global_value));
-
-    registry.insert(Builder::new("Binary").value().value());
-    registry.insert(Builder::new("BinaryImm").value().imm(&imm.imm64));
-
-    // The select instructions are controlled by the second VALUE operand.
-    // The first VALUE operand is the controlling flag which has a derived type.
-    // The fma instruction has the same constraint on all inputs.
-    registry.insert(
-        Builder::new("Ternary")
-            .value()
-            .value()
-            .value()
-            .typevar_operand(1),
-    );
-
-    // Catch-all for instructions with many outputs and inputs and no immediate
-    // operands.
-    registry.insert(Builder::new("MultiAry").varargs());
-
-    registry.insert(Builder::new("NullAry"));
-
-    registry.insert(
-        Builder::new("InsertLane")
-            .value()
-            .imm_with_name("lane", &imm.uimm8)
-            .value(),
-    );
-    registry.insert(
-        Builder::new("ExtractLane")
-            .value()
-            .imm_with_name("lane", &imm.uimm8),
-    );
-    registry.insert(
-        Builder::new("Shuffle")
-            .value()
-            .value()
-            .imm_with_name("mask", &imm.uimm128),
-    );
-
-    registry.insert(Builder::new("IntCompare").imm(&imm.intcc).value().value());
-    registry.insert(
-        Builder::new("IntCompareImm")
-            .imm(&imm.intcc)
-            .value()
-            .imm(&imm.imm64),
-    );
-    registry.insert(Builder::new("IntCond").imm(&imm.intcc).value());
-
-    registry.insert(
-        Builder::new("FloatCompare")
-            .imm(&imm.floatcc)
-            .value()
-            .value(),
-    );
-    registry.insert(Builder::new("FloatCond").imm(&imm.floatcc).value());
-
-    registry.insert(
-        Builder::new("IntSelect")
-            .imm(&imm.intcc)
-            .value()
-            .value()
-            .value(),
-    );
-
-    registry.insert(Builder::new("Jump").imm(&entities.ebb).varargs());
-    registry.insert(Builder::new("Branch").value().imm(&entities.ebb).varargs());
-    registry.insert(
-        Builder::new("BranchInt")
-            .imm(&imm.intcc)
-            .value()
-            .imm(&entities.ebb)
-            .varargs(),
-    );
-    registry.insert(
-        Builder::new("BranchFloat")
-            .imm(&imm.floatcc)
-            .value()
-            .imm(&entities.ebb)
-            .varargs(),
-    );
-    registry.insert(
-        Builder::new("BranchIcmp")
-            .imm(&imm.intcc)
-            .value()
-            .value()
-            .imm(&entities.ebb)
-            .varargs(),
-    );
-    registry.insert(
-        Builder::new("BranchTable")
-            .value()
-            .imm(&entities.ebb)
-            .imm(&entities.jump_table),
-    );
-    registry.insert(
-        Builder::new("BranchTableEntry")
-            .value()
-            .value()
-            .imm(&imm.uimm8)
-            .imm(&entities.jump_table),
-    );
-    registry.insert(Builder::new("BranchTableBase").imm(&entities.jump_table));
-    registry.insert(
-        Builder::new("IndirectJump")
-            .value()
-            .imm(&entities.jump_table),
-    );
-
-    registry.insert(Builder::new("Call").imm(&entities.func_ref).varargs());
-    registry.insert(
-        Builder::new("CallIndirect")
-            .imm(&entities.sig_ref)
-            .value()
-            .varargs(),
-    );
-    registry.insert(Builder::new("FuncAddr").imm(&entities.func_ref));
-
-    registry.insert(
-        Builder::new("Load")
-            .imm(&imm.memflags)
-            .value()
-            .imm(&imm.offset32),
-    );
-    registry.insert(
-        Builder::new("LoadComplex")
-            .imm(&imm.memflags)
-            .varargs()
-            .imm(&imm.offset32),
-    );
-    registry.insert(
-        Builder::new("Store")
-            .imm(&imm.memflags)
-            .value()
-            .value()
-            .imm(&imm.offset32),
-    );
-    registry.insert(
-        Builder::new("StoreComplex")
-            .imm(&imm.memflags)
-            .value()
-            .varargs()
-            .imm(&imm.offset32),
-    );
-    registry.insert(
-        Builder::new("StackLoad")
-            .imm(&entities.stack_slot)
-            .imm(&imm.offset32),
-    );
-    registry.insert(
-        Builder::new("StackStore")
-            .value()
-            .imm(&entities.stack_slot)
-            .imm(&imm.offset32),
-    );
-
-    // Accessing a WebAssembly heap.
-    registry.insert(
-        Builder::new("HeapAddr")
-            .imm(&entities.heap)
-            .value()
-            .imm(&imm.uimm32),
-    );
-
-    // Accessing a WebAssembly table.
-    registry.insert(
-        Builder::new("TableAddr")
-            .imm(&entities.table)
-            .value()
-            .imm(&imm.offset32),
-    );
-
-    registry.insert(
-        Builder::new("RegMove")
-            .value()
-            .imm_with_name("src", &imm.regunit)
-            .imm_with_name("dst", &imm.regunit),
-    );
-    registry.insert(
-        Builder::new("CopySpecial")
-            .imm_with_name("src", &imm.regunit)
-            .imm_with_name("dst", &imm.regunit),
-    );
-    registry.insert(Builder::new("CopyToSsa").imm_with_name("src", &imm.regunit));
-    registry.insert(
-        Builder::new("RegSpill")
-            .value()
-            .imm_with_name("src", &imm.regunit)
-            .imm_with_name("dst", &entities.stack_slot),
-    );
-    registry.insert(
-        Builder::new("RegFill")
-            .value()
-            .imm_with_name("src", &entities.stack_slot)
-            .imm_with_name("dst", &imm.regunit),
-    );
-
-    registry.insert(Builder::new("Trap").imm(&imm.trapcode));
-    registry.insert(Builder::new("CondTrap").value().imm(&imm.trapcode));
-    registry.insert(
-        Builder::new("IntCondTrap")
-            .imm(&imm.intcc)
-            .value()
-            .imm(&imm.trapcode),
-    );
-    registry.insert(
-        Builder::new("FloatCondTrap")
-            .imm(&imm.floatcc)
-            .value()
-            .imm(&imm.trapcode),
-    );
-
-    registry
+pub(crate) struct Formats {
+    pub(crate) binary: Rc,
+    pub(crate) binary_imm: Rc,
+    pub(crate) branch: Rc,
+    pub(crate) branch_float: Rc,
+    pub(crate) branch_icmp: Rc,
+    pub(crate) branch_int: Rc,
+    pub(crate) branch_table: Rc,
+    pub(crate) branch_table_base: Rc,
+    pub(crate) branch_table_entry: Rc,
+    pub(crate) call: Rc,
+    pub(crate) call_indirect: Rc,
+    pub(crate) cond_trap: Rc,
+    pub(crate) copy_special: Rc,
+    pub(crate) copy_to_ssa: Rc,
+    pub(crate) extract_lane: Rc,
+    pub(crate) float_compare: Rc,
+    pub(crate) float_cond: Rc,
+    pub(crate) float_cond_trap: Rc,
+    pub(crate) func_addr: Rc,
+    pub(crate) heap_addr: Rc,
+    pub(crate) indirect_jump: Rc,
+    pub(crate) insert_lane: Rc,
+    pub(crate) int_compare: Rc,
+    pub(crate) int_compare_imm: Rc,
+    pub(crate) int_cond: Rc,
+    pub(crate) int_cond_trap: Rc,
+    pub(crate) int_select: Rc,
+    pub(crate) jump: Rc,
+    pub(crate) load: Rc,
+    pub(crate) load_complex: Rc,
+    pub(crate) multiary: Rc,
+    pub(crate) nullary: Rc,
+    pub(crate) reg_fill: Rc,
+    pub(crate) reg_move: Rc,
+    pub(crate) reg_spill: Rc,
+    pub(crate) shuffle: Rc,
+    pub(crate) stack_load: Rc,
+    pub(crate) stack_store: Rc,
+    pub(crate) store: Rc,
+    pub(crate) store_complex: Rc,
+    pub(crate) table_addr: Rc,
+    pub(crate) ternary: Rc,
+    pub(crate) trap: Rc,
+    pub(crate) unary: Rc,
+    pub(crate) unary_bool: Rc,
+    pub(crate) unary_const: Rc,
+    pub(crate) unary_global_value: Rc,
+    pub(crate) unary_ieee32: Rc,
+    pub(crate) unary_ieee64: Rc,
+    pub(crate) unary_imm: Rc,
+}
+
+impl Formats {
+    pub fn new(imm: &Immediates, entities: &EntityRefs) -> Self {
+        Self {
+            unary: Builder::new("Unary").value().build(),
+
+            unary_imm: Builder::new("UnaryImm").imm(&imm.imm64).build(),
+
+            unary_ieee32: Builder::new("UnaryIeee32").imm(&imm.ieee32).build(),
+
+            unary_ieee64: Builder::new("UnaryIeee64").imm(&imm.ieee64).build(),
+
+            unary_bool: Builder::new("UnaryBool").imm(&imm.boolean).build(),
+
+            unary_const: Builder::new("UnaryConst").imm(&imm.pool_constant).build(),
+
+            unary_global_value: Builder::new("UnaryGlobalValue")
+                .imm(&entities.global_value)
+                .build(),
+
+            binary: Builder::new("Binary").value().value().build(),
+
+            binary_imm: Builder::new("BinaryImm").value().imm(&imm.imm64).build(),
+
+            // The select instructions are controlled by the second VALUE operand.
+            // The first VALUE operand is the controlling flag which has a derived type.
+            // The fma instruction has the same constraint on all inputs.
+            ternary: Builder::new("Ternary")
+                .value()
+                .value()
+                .value()
+                .typevar_operand(1)
+                .build(),
+
+            // Catch-all for instructions with many outputs and inputs and no immediate
+            // operands.
+            multiary: Builder::new("MultiAry").varargs().build(),
+
+            nullary: Builder::new("NullAry").build(),
+
+            insert_lane: Builder::new("InsertLane")
+                .value()
+                .imm_with_name("lane", &imm.uimm8)
+                .value()
+                .build(),
+
+            extract_lane: Builder::new("ExtractLane")
+                .value()
+                .imm_with_name("lane", &imm.uimm8)
+                .build(),
+
+            shuffle: Builder::new("Shuffle")
+                .value()
+                .value()
+                .imm_with_name("mask", &imm.uimm128)
+                .build(),
+
+            int_compare: Builder::new("IntCompare")
+                .imm(&imm.intcc)
+                .value()
+                .value()
+                .build(),
+
+            int_compare_imm: Builder::new("IntCompareImm")
+                .imm(&imm.intcc)
+                .value()
+                .imm(&imm.imm64)
+                .build(),
+
+            int_cond: Builder::new("IntCond").imm(&imm.intcc).value().build(),
+
+            float_compare: Builder::new("FloatCompare")
+                .imm(&imm.floatcc)
+                .value()
+                .value()
+                .build(),
+
+            float_cond: Builder::new("FloatCond").imm(&imm.floatcc).value().build(),
+
+            int_select: Builder::new("IntSelect")
+                .imm(&imm.intcc)
+                .value()
+                .value()
+                .value()
+                .build(),
+
+            jump: Builder::new("Jump").imm(&entities.ebb).varargs().build(),
+
+            branch: Builder::new("Branch")
+                .value()
+                .imm(&entities.ebb)
+                .varargs()
+                .build(),
+
+            branch_int: Builder::new("BranchInt")
+                .imm(&imm.intcc)
+                .value()
+                .imm(&entities.ebb)
+                .varargs()
+                .build(),
+
+            branch_float: Builder::new("BranchFloat")
+                .imm(&imm.floatcc)
+                .value()
+                .imm(&entities.ebb)
+                .varargs()
+                .build(),
+
+            branch_icmp: Builder::new("BranchIcmp")
+                .imm(&imm.intcc)
+                .value()
+                .value()
+                .imm(&entities.ebb)
+                .varargs()
+                .build(),
+
+            branch_table: Builder::new("BranchTable")
+                .value()
+                .imm(&entities.ebb)
+                .imm(&entities.jump_table)
+                .build(),
+
+            branch_table_entry: Builder::new("BranchTableEntry")
+                .value()
+                .value()
+                .imm(&imm.uimm8)
+                .imm(&entities.jump_table)
+                .build(),
+
+            branch_table_base: Builder::new("BranchTableBase")
+                .imm(&entities.jump_table)
+                .build(),
+
+            indirect_jump: Builder::new("IndirectJump")
+                .value()
+                .imm(&entities.jump_table)
+                .build(),
+
+            call: Builder::new("Call")
+                .imm(&entities.func_ref)
+                .varargs()
+                .build(),
+
+            call_indirect: Builder::new("CallIndirect")
+                .imm(&entities.sig_ref)
+                .value()
+                .varargs()
+                .build(),
+
+            func_addr: Builder::new("FuncAddr").imm(&entities.func_ref).build(),
+
+            load: Builder::new("Load")
+                .imm(&imm.memflags)
+                .value()
+                .imm(&imm.offset32)
+                .build(),
+
+            load_complex: Builder::new("LoadComplex")
+                .imm(&imm.memflags)
+                .varargs()
+                .imm(&imm.offset32)
+                .build(),
+
+            store: Builder::new("Store")
+                .imm(&imm.memflags)
+                .value()
+                .value()
+                .imm(&imm.offset32)
+                .build(),
+
+            store_complex: Builder::new("StoreComplex")
+                .imm(&imm.memflags)
+                .value()
+                .varargs()
+                .imm(&imm.offset32)
+                .build(),
+
+            stack_load: Builder::new("StackLoad")
+                .imm(&entities.stack_slot)
+                .imm(&imm.offset32)
+                .build(),
+
+            stack_store: Builder::new("StackStore")
+                .value()
+                .imm(&entities.stack_slot)
+                .imm(&imm.offset32)
+                .build(),
+
+            // Accessing a WebAssembly heap.
+            heap_addr: Builder::new("HeapAddr")
+                .imm(&entities.heap)
+                .value()
+                .imm(&imm.uimm32)
+                .build(),
+
+            // Accessing a WebAssembly table.
+            table_addr: Builder::new("TableAddr")
+                .imm(&entities.table)
+                .value()
+                .imm(&imm.offset32)
+                .build(),
+
+            reg_move: Builder::new("RegMove")
+                .value()
+                .imm_with_name("src", &imm.regunit)
+                .imm_with_name("dst", &imm.regunit)
+                .build(),
+
+            copy_special: Builder::new("CopySpecial")
+                .imm_with_name("src", &imm.regunit)
+                .imm_with_name("dst", &imm.regunit)
+                .build(),
+
+            copy_to_ssa: Builder::new("CopyToSsa")
+                .imm_with_name("src", &imm.regunit)
+                .build(),
+
+            reg_spill: Builder::new("RegSpill")
+                .value()
+                .imm_with_name("src", &imm.regunit)
+                .imm_with_name("dst", &entities.stack_slot)
+                .build(),
+
+            reg_fill: Builder::new("RegFill")
+                .value()
+                .imm_with_name("src", &entities.stack_slot)
+                .imm_with_name("dst", &imm.regunit)
+                .build(),
+
+            trap: Builder::new("Trap").imm(&imm.trapcode).build(),
+
+            cond_trap: Builder::new("CondTrap").value().imm(&imm.trapcode).build(),
+
+            int_cond_trap: Builder::new("IntCondTrap")
+                .imm(&imm.intcc)
+                .value()
+                .imm(&imm.trapcode)
+                .build(),
+
+            float_cond_trap: Builder::new("FloatCondTrap")
+                .imm(&imm.floatcc)
+                .value()
+                .imm(&imm.trapcode)
+                .build(),
+        }
+    }
 }
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index f6d181def2..1980730d95 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1,6 +1,5 @@
 #![allow(non_snake_case)]
 
-use crate::cdsl::formats::FormatRegistry;
 use crate::cdsl::instructions::{
     AllInstructions, InstructionBuilder as Inst, InstructionGroup, InstructionGroupBuilder,
 };
@@ -8,16 +7,17 @@ use crate::cdsl::operands::{create_operand as operand, create_operand_doc as ope
 use crate::cdsl::type_inference::Constraint::WiderOrEq;
 use crate::cdsl::types::{LaneType, ValueType};
 use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
+use crate::shared::formats::Formats;
 use crate::shared::types;
 use crate::shared::{entities::EntityRefs, immediates::Immediates};
 
 pub(crate) fn define(
     all_instructions: &mut AllInstructions,
-    format_registry: &FormatRegistry,
+    formats: &Formats,
     imm: &Immediates,
     entities: &EntityRefs,
 ) -> InstructionGroup {
-    let mut ig = InstructionGroupBuilder::new(all_instructions, format_registry);
+    let mut ig = InstructionGroupBuilder::new(all_instructions);
 
     // Operand kind shorthands.
     let iflags: &TypeVar = &ValueType::Special(types::Flag::IFlags.into()).into();
@@ -130,6 +130,7 @@ pub(crate) fn define(
         EBB arguments. The number and types of arguments must match the
         destination EBB.
         "#,
+            &formats.jump,
         )
         .operands_in(vec![EBB, args])
         .is_terminator(true)
@@ -149,6 +150,7 @@ pub(crate) fn define(
         relaxation pass. There is no reason to use this instruction outside
         that pass.
         "#,
+            &formats.jump,
         )
         .operands_in(vec![EBB, args])
         .is_terminator(true)
@@ -164,6 +166,7 @@ pub(crate) fn define(
         If ``c`` is a `b1` value, take the branch when ``c`` is false. If
         ``c`` is an integer value, take the branch when ``c = 0``.
         "#,
+            &formats.branch,
         )
         .operands_in(vec![c, EBB, args])
         .is_branch(true),
@@ -178,6 +181,7 @@ pub(crate) fn define(
         If ``c`` is a `b1` value, take the branch when ``c`` is true. If
         ``c`` is an integer value, take the branch when ``c != 0``.
         "#,
+            &formats.branch,
         )
         .operands_in(vec![c, EBB, args])
         .is_branch(true),
@@ -207,6 +211,7 @@ pub(crate) fn define(
         implement all or some of the condition codes. The instruction can also
         be used to represent *macro-op fusion* on architectures like Intel's.
         "#,
+            &formats.branch_icmp,
         )
         .operands_in(vec![Cond, x, y, EBB, args])
         .is_branch(true),
@@ -220,6 +225,7 @@ pub(crate) fn define(
             r#"
         Branch when condition is true in integer CPU flags.
         "#,
+            &formats.branch_int,
         )
         .operands_in(vec![Cond, f, EBB, args])
         .is_branch(true),
@@ -234,6 +240,7 @@ pub(crate) fn define(
             r#"
         Branch when condition is true in floating point CPU flags.
         "#,
+            &formats.branch_float,
         )
         .operands_in(vec![Cond, f, EBB, args])
         .is_branch(true),
@@ -263,6 +270,7 @@ pub(crate) fn define(
         function in a dynamic library, that will typically use
         ``call_indirect``.
         "#,
+            &formats.branch_table,
         )
         .operands_in(vec![x, EBB, JT])
         .is_terminator(true)
@@ -287,6 +295,7 @@ pub(crate) fn define(
     Currently, the only type supported is entries which are relative to the
     base of the jump table.
     "#,
+            &formats.branch_table_entry,
         )
         .operands_in(vec![x, addr, Size, JT])
         .operands_out(vec![entry])
@@ -304,6 +313,7 @@ pub(crate) fn define(
     load an entry using ``jump_table_entry``, then use this instruction to add
     the relative base back to it.
     "#,
+            &formats.branch_table_base,
         )
         .operands_in(vec![JT])
         .operands_out(vec![addr]),
@@ -318,6 +328,7 @@ pub(crate) fn define(
     Unconditionally jump via a jump table entry that was previously loaded
     with the ``jump_table_entry`` instruction.
     "#,
+            &formats.indirect_jump,
         )
         .operands_in(vec![addr, JT])
         .is_indirect_branch(true)
@@ -331,6 +342,7 @@ pub(crate) fn define(
             r#"
     Encodes an assembly debug trap.
     "#,
+            &formats.nullary,
         )
         .other_side_effects(true)
         .can_load(true)
@@ -345,6 +357,7 @@ pub(crate) fn define(
             r#"
         Terminate execution unconditionally.
         "#,
+            &formats.trap,
         )
         .operands_in(vec![code])
         .can_trap(true)
@@ -359,6 +372,7 @@ pub(crate) fn define(
 
         if ``c`` is non-zero, execution continues at the following instruction.
         "#,
+            &formats.cond_trap,
         )
         .operands_in(vec![c, code])
         .can_trap(true),
@@ -372,6 +386,7 @@ pub(crate) fn define(
 
         This instruction allows non-conditional traps to be used as non-terminal instructions.
         "#,
+            &formats.trap,
         )
         .operands_in(vec![code])
         .can_trap(true),
@@ -385,6 +400,7 @@ pub(crate) fn define(
 
         if ``c`` is zero, execution continues at the following instruction.
         "#,
+            &formats.cond_trap,
         )
         .operands_in(vec![c, code])
         .can_trap(true),
@@ -399,6 +415,7 @@ pub(crate) fn define(
             r#"
         Trap when condition is true in integer CPU flags.
         "#,
+            &formats.int_cond_trap,
         )
         .operands_in(vec![Cond, f, code])
         .can_trap(true),
@@ -413,6 +430,7 @@ pub(crate) fn define(
             r#"
         Trap when condition is true in floating point CPU flags.
         "#,
+            &formats.float_cond_trap,
         )
         .operands_in(vec![Cond, f, code])
         .can_trap(true),
@@ -430,6 +448,7 @@ pub(crate) fn define(
         provided return values. The list of return values must match the
         function signature's return types.
         "#,
+            &formats.multiary,
         )
         .operands_in(vec![rvals])
         .is_return(true)
@@ -446,6 +465,7 @@ pub(crate) fn define(
         a custom epilogue, which will then perform the real return. This
         instruction has no encoding.
         "#,
+            &formats.multiary,
         )
         .operands_in(vec![rvals])
         .is_return(true)
@@ -468,6 +488,7 @@ pub(crate) fn define(
         Call a function which has been declared in the preamble. The argument
         types must match the function's signature.
         "#,
+            &formats.call,
         )
         .operands_in(vec![FN, args])
         .operands_out(vec![rvals])
@@ -491,6 +512,7 @@ pub(crate) fn define(
         `table_addr` and `load` are used to obtain a native address
         from a table.
         "#,
+            &formats.call_indirect,
         )
         .operands_in(vec![SIG, callee, args])
         .operands_out(vec![rvals])
@@ -509,6 +531,7 @@ pub(crate) fn define(
         are too far away to be addressable by a direct `call`
         instruction.
         "#,
+            &formats.func_addr,
         )
         .operands_in(vec![FN])
         .operands_out(vec![addr]),
@@ -531,6 +554,7 @@ pub(crate) fn define(
         This is a polymorphic instruction that can load any value type which
         has a memory representation.
         "#,
+            &formats.load,
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
@@ -546,6 +570,7 @@ pub(crate) fn define(
         This is a polymorphic instruction that can load any value type which
         has a memory representation.
         "#,
+            &formats.load_complex,
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
@@ -561,6 +586,7 @@ pub(crate) fn define(
         This is a polymorphic instruction that can store any value type with a
         memory representation.
         "#,
+            &formats.store,
         )
         .operands_in(vec![MemFlags, x, p, Offset])
         .can_store(true),
@@ -575,6 +601,7 @@ pub(crate) fn define(
         This is a polymorphic instruction that can store any value type with a
         memory representation.
         "#,
+            &formats.store_complex,
         )
         .operands_in(vec![MemFlags, x, args, Offset])
         .can_store(true),
@@ -596,6 +623,7 @@ pub(crate) fn define(
 
         This is equivalent to ``load.i8`` followed by ``uextend``.
         "#,
+            &formats.load,
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
@@ -610,6 +638,7 @@ pub(crate) fn define(
 
         This is equivalent to ``load.i8`` followed by ``uextend``.
         "#,
+            &formats.load_complex,
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
@@ -624,6 +653,7 @@ pub(crate) fn define(
 
         This is equivalent to ``load.i8`` followed by ``sextend``.
         "#,
+            &formats.load,
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
@@ -638,6 +668,7 @@ pub(crate) fn define(
 
         This is equivalent to ``load.i8`` followed by ``sextend``.
         "#,
+            &formats.load_complex,
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
@@ -652,6 +683,7 @@ pub(crate) fn define(
 
         This is equivalent to ``ireduce.i8`` followed by ``store.i8``.
         "#,
+            &formats.store,
         )
         .operands_in(vec![MemFlags, x, p, Offset])
         .can_store(true),
@@ -665,6 +697,7 @@ pub(crate) fn define(
 
         This is equivalent to ``ireduce.i8`` followed by ``store.i8``.
         "#,
+            &formats.store_complex,
         )
         .operands_in(vec![MemFlags, x, args, Offset])
         .can_store(true),
@@ -686,6 +719,7 @@ pub(crate) fn define(
 
         This is equivalent to ``load.i16`` followed by ``uextend``.
         "#,
+            &formats.load,
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
@@ -700,6 +734,7 @@ pub(crate) fn define(
 
         This is equivalent to ``load.i16`` followed by ``uextend``.
         "#,
+            &formats.load_complex,
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
@@ -714,6 +749,7 @@ pub(crate) fn define(
 
         This is equivalent to ``load.i16`` followed by ``sextend``.
         "#,
+            &formats.load,
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
@@ -728,6 +764,7 @@ pub(crate) fn define(
 
         This is equivalent to ``load.i16`` followed by ``sextend``.
         "#,
+            &formats.load_complex,
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
@@ -742,6 +779,7 @@ pub(crate) fn define(
 
         This is equivalent to ``ireduce.i16`` followed by ``store.i16``.
         "#,
+            &formats.store,
         )
         .operands_in(vec![MemFlags, x, p, Offset])
         .can_store(true),
@@ -755,6 +793,7 @@ pub(crate) fn define(
 
         This is equivalent to ``ireduce.i16`` followed by ``store.i16``.
         "#,
+            &formats.store_complex,
         )
         .operands_in(vec![MemFlags, x, args, Offset])
         .can_store(true),
@@ -776,6 +815,7 @@ pub(crate) fn define(
 
         This is equivalent to ``load.i32`` followed by ``uextend``.
         "#,
+            &formats.load,
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
@@ -790,6 +830,7 @@ pub(crate) fn define(
 
         This is equivalent to ``load.i32`` followed by ``uextend``.
         "#,
+            &formats.load_complex,
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
@@ -804,6 +845,7 @@ pub(crate) fn define(
 
         This is equivalent to ``load.i32`` followed by ``sextend``.
         "#,
+            &formats.load,
         )
         .operands_in(vec![MemFlags, p, Offset])
         .operands_out(vec![a])
@@ -818,6 +860,7 @@ pub(crate) fn define(
 
         This is equivalent to ``load.i32`` followed by ``sextend``.
         "#,
+            &formats.load_complex,
         )
         .operands_in(vec![MemFlags, args, Offset])
         .operands_out(vec![a])
@@ -832,6 +875,7 @@ pub(crate) fn define(
 
         This is equivalent to ``ireduce.i32`` followed by ``store.i32``.
         "#,
+            &formats.store,
         )
         .operands_in(vec![MemFlags, x, p, Offset])
         .can_store(true),
@@ -845,6 +889,7 @@ pub(crate) fn define(
 
         This is equivalent to ``ireduce.i32`` followed by ``store.i32``.
         "#,
+            &formats.store_complex,
         )
         .operands_in(vec![MemFlags, x, args, Offset])
         .can_store(true),
@@ -867,6 +912,7 @@ pub(crate) fn define(
         access cannot go out of bounds, i.e.
         `sizeof(a) + Offset <= sizeof(SS)`.
         "#,
+            &formats.stack_load,
         )
         .operands_in(vec![SS, Offset])
         .operands_out(vec![a])
@@ -886,6 +932,7 @@ pub(crate) fn define(
         access cannot go out of bounds, i.e.
         `sizeof(a) + Offset <= sizeof(SS)`.
         "#,
+            &formats.stack_store,
         )
         .operands_in(vec![x, SS, Offset])
         .can_store(true),
@@ -901,6 +948,7 @@ pub(crate) fn define(
         refer to a byte inside the stack slot:
         `0 <= Offset < sizeof(SS)`.
         "#,
+            &formats.stack_load,
         )
         .operands_in(vec![SS, Offset])
         .operands_out(vec![addr]),
@@ -914,6 +962,7 @@ pub(crate) fn define(
             r#"
         Compute the value of global GV.
         "#,
+            &formats.unary_global_value,
         )
         .operands_in(vec![GV])
         .operands_out(vec![a]),
@@ -925,6 +974,7 @@ pub(crate) fn define(
             r#"
         Compute the value of global GV, which is a symbolic value.
         "#,
+            &formats.unary_global_value,
         )
         .operands_in(vec![GV])
         .operands_out(vec![a]),
@@ -954,6 +1004,7 @@ pub(crate) fn define(
            heap's base address.
         2. If ``p + Size`` is greater than the heap bound, generate a trap.
         "#,
+            &formats.heap_addr,
         )
         .operands_in(vec![H, p, Size])
         .operands_out(vec![addr]),
@@ -971,6 +1022,7 @@ pub(crate) fn define(
             r#"
             Gets the content of the pinned register, when it's enabled.
         "#,
+            &formats.nullary,
         )
         .operands_out(vec![addr])
         .other_side_effects(true),
@@ -982,6 +1034,7 @@ pub(crate) fn define(
             r#"
         Sets the content of the pinned register, when it's enabled.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![addr])
         .other_side_effects(true),
@@ -1012,6 +1065,7 @@ pub(crate) fn define(
            base address.
         2. If ``p`` is greater than the table bound, generate a trap.
         "#,
+            &formats.table_addr,
         )
         .operands_in(vec![T, p, Offset])
         .operands_out(vec![addr]),
@@ -1029,6 +1083,7 @@ pub(crate) fn define(
         Create a scalar integer SSA value with an immediate constant value, or
         an integer vector where all the lanes have the same value.
         "#,
+            &formats.unary_imm,
         )
         .operands_in(vec![N])
         .operands_out(vec![a]),
@@ -1045,6 +1100,7 @@ pub(crate) fn define(
 
         Create a `f32` SSA value with an immediate constant value.
         "#,
+            &formats.unary_ieee32,
         )
         .operands_in(vec![N])
         .operands_out(vec![a]),
@@ -1061,6 +1117,7 @@ pub(crate) fn define(
 
         Create a `f64` SSA value with an immediate constant value.
         "#,
+            &formats.unary_ieee64,
         )
         .operands_in(vec![N])
         .operands_out(vec![a]),
@@ -1078,6 +1135,7 @@ pub(crate) fn define(
         Create a scalar boolean SSA value with an immediate constant value, or
         a boolean vector where all the lanes have the same value.
         "#,
+            &formats.unary_bool,
         )
         .operands_in(vec![N])
         .operands_out(vec![a]),
@@ -1098,6 +1156,7 @@ pub(crate) fn define(
 
         Construct a vector with the given immediate bytes.
         "#,
+            &formats.unary_const,
         )
         .operands_in(vec![N])
         .operands_out(vec![a]),
@@ -1127,12 +1186,13 @@ pub(crate) fn define(
             "shuffle",
             r#"
         SIMD vector shuffle.
-        
+
         Shuffle two vectors using the given immediate bytes. For each of the 16 bytes of the
-        immediate, a value i of 0-15 selects the i-th element of the first vector and a value i of 
-        16-31 selects the (i-16)th element of the second vector. Immediate values outside of the 
+        immediate, a value i of 0-15 selects the i-th element of the first vector and a value i of
+        16-31 selects the (i-16)th element of the second vector. Immediate values outside of the
         0-31 range place a 0 in the resulting vector lane.
         "#,
+            &formats.shuffle,
         )
         .operands_in(vec![a, b, mask])
         .operands_out(vec![a]),
@@ -1148,6 +1208,7 @@ pub(crate) fn define(
 
         Create a scalar reference SSA value with a constant null value.
         "#,
+            &formats.nullary,
         )
         .operands_out(vec![a]),
     );
@@ -1155,10 +1216,11 @@ pub(crate) fn define(
     ig.push(Inst::new(
         "nop",
         r#"
-        Just a dummy instruction
+        Just a dummy instruction.
 
-        Note: this doesn't compile to a machine code nop
+        Note: this doesn't compile to a machine code nop.
         "#,
+        &formats.nullary,
     ));
 
     let c = &operand_doc("c", Testable, "Controlling value to test");
@@ -1175,6 +1237,7 @@ pub(crate) fn define(
         This instruction selects whole values. Use `vselect` for
         lane-wise selection.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![c, x, y])
         .operands_out(vec![a]),
@@ -1189,6 +1252,7 @@ pub(crate) fn define(
             r#"
         Conditional select, dependent on integer condition codes.
         "#,
+            &formats.int_select,
         )
         .operands_in(vec![cc, flags, x, y])
         .operands_out(vec![a]),
@@ -1201,10 +1265,11 @@ pub(crate) fn define(
             r#"
         Conditional select of bits.
 
-        For each bit in `c`, this instruction selects the corresponding bit from `x` if the bit 
-        in `c` is 1 and the corresponding bit from `y` if the bit in `c` is 0. See also: 
+        For each bit in `c`, this instruction selects the corresponding bit from `x` if the bit
+        in `c` is 1 and the corresponding bit from `y` if the bit in `c` is 0. See also:
         `select`, `vselect`.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![c, x, y])
         .operands_out(vec![a]),
@@ -1225,6 +1290,7 @@ pub(crate) fn define(
         instruction transformations, and the register allocator needs a way of
         representing register copies.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -1239,6 +1305,7 @@ pub(crate) fn define(
         This instruction behaves exactly like `copy`, but the result
         value is assigned to a spill slot.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
@@ -1254,6 +1321,7 @@ pub(crate) fn define(
         This instruction behaves exactly like `copy`, but creates a new
         SSA value for the spilled input value.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
@@ -1270,6 +1338,7 @@ pub(crate) fn define(
         registers and stack slots have been assigned.  It is used to replace `fill`s that have
         been identified as redundant.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
@@ -1294,6 +1363,7 @@ pub(crate) fn define(
         before the value leaves the EBB. At the entry to a new EBB, all live
         values must be in their originally assigned registers.
         "#,
+            &formats.reg_move,
         )
         .operands_in(vec![x, src, dst])
         .other_side_effects(true),
@@ -1310,6 +1380,7 @@ pub(crate) fn define(
         special registers, e.g. copying the stack register to the frame
         register in a function prologue.
         "#,
+            &formats.copy_special,
         )
         .operands_in(vec![src, dst])
         .other_side_effects(true),
@@ -1326,6 +1397,7 @@ pub(crate) fn define(
         of ''copy_special''.  This instruction is internal and should not be created by
         Cranelift users.
         "#,
+            &formats.copy_to_ssa,
         )
         .operands_in(vec![src])
         .operands_out(vec![a])
@@ -1341,6 +1413,7 @@ pub(crate) fn define(
 
         This instruction copies its input, preserving the value type.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -1356,6 +1429,7 @@ pub(crate) fn define(
 
     This instruction is used to adjust the stack pointer by a dynamic amount.
     "#,
+            &formats.unary,
         )
         .operands_in(vec![delta])
         .other_side_effects(true),
@@ -1373,6 +1447,7 @@ pub(crate) fn define(
     prologues and epilogues. ``Offset`` is constrained to the size of a signed
     32-bit integer.
     "#,
+            &formats.unary_imm,
         )
         .operands_in(vec![Offset])
         .other_side_effects(true),
@@ -1391,6 +1466,7 @@ pub(crate) fn define(
     prologues and epilogues. ``Offset`` is constrained to the size of a signed
     32-bit integer.
     "#,
+            &formats.unary_imm,
         )
         .operands_in(vec![Offset])
         .other_side_effects(true),
@@ -1407,6 +1483,7 @@ pub(crate) fn define(
     This is like `ifcmp` where ``addr`` is the LHS operand and the stack
     pointer is the RHS.
     "#,
+            &formats.unary,
         )
         .operands_in(vec![addr])
         .operands_out(vec![f]),
@@ -1425,6 +1502,7 @@ pub(crate) fn define(
 
         See also `regmove`.
         "#,
+            &formats.reg_spill,
         )
         .operands_in(vec![x, src, SS])
         .other_side_effects(true),
@@ -1443,6 +1521,7 @@ pub(crate) fn define(
 
         See also `regmove`.
         "#,
+            &formats.reg_fill,
         )
         .operands_in(vec![x, SS, dst])
         .other_side_effects(true),
@@ -1461,6 +1540,7 @@ pub(crate) fn define(
         This instruction will provide live reference values at a point in
         the function. It can only be used by the compiler.
         "#,
+            &formats.multiary,
         )
         .operands_in(vec![N])
         .other_side_effects(true),
@@ -1480,6 +1560,7 @@ pub(crate) fn define(
         the lanes from ``x``. The result may be two scalars if ``x`` only had
         two lanes.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![lo, hi])
@@ -1515,6 +1596,7 @@ pub(crate) fn define(
 
         It is possible to form a vector by concatenating two scalars.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a])
@@ -1535,6 +1617,7 @@ pub(crate) fn define(
         Select lanes from ``x`` or ``y`` controlled by the lanes of the boolean
         vector ``c``.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![c, x, y])
         .operands_out(vec![a]),
@@ -1550,6 +1633,7 @@ pub(crate) fn define(
 
         Return a vector whose lanes are all ``x``.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -1568,6 +1652,7 @@ pub(crate) fn define(
         The lane index, ``Idx``, is an immediate value, not an SSA value. It
         must indicate a valid lane index for the type of ``x``.
         "#,
+            &formats.insert_lane,
         )
         .operands_in(vec![x, Idx, y])
         .operands_out(vec![a]),
@@ -1584,9 +1669,10 @@ pub(crate) fn define(
 
         The lane index, ``Idx``, is an immediate value, not an SSA value. It
         must indicate a valid lane index for the type of ``x``. Note that the upper bits of ``a``
-        may or may not be zeroed depending on the ISA but the type system should prevent using 
+        may or may not be zeroed depending on the ISA but the type system should prevent using
         ``a`` as anything other than the extracted value.
         "#,
+            &formats.extract_lane,
         )
         .operands_in(vec![x, Idx])
         .operands_out(vec![a]),
@@ -1625,6 +1711,7 @@ pub(crate) fn define(
         When this instruction compares integer vectors, it returns a boolean
         vector of lane-wise comparisons.
         "#,
+            &formats.int_compare,
         )
         .operands_in(vec![Cond, x, y])
         .operands_out(vec![a]),
@@ -1646,6 +1733,7 @@ pub(crate) fn define(
         This instruction can only compare scalars. Use `icmp` for
         lane-wise vector comparisons.
         "#,
+            &formats.int_compare_imm,
         )
         .operands_in(vec![Cond, x, Y])
         .operands_out(vec![a]),
@@ -1664,6 +1752,7 @@ pub(crate) fn define(
         Compare two scalar integer values and return integer CPU flags
         representing the result.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![f]),
@@ -1678,6 +1767,7 @@ pub(crate) fn define(
         Like `icmp_imm`, but returns integer CPU flags instead of testing
         a specific condition code.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![f]),
@@ -1696,6 +1786,7 @@ pub(crate) fn define(
         This instruction does not depend on the signed/unsigned interpretation
         of the operands.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -1707,10 +1798,11 @@ pub(crate) fn define(
             r#"
         Add with unsigned saturation.
 
-        This is similar to `iadd` but the operands are interpreted as unsigned integers and their 
+        This is similar to `iadd` but the operands are interpreted as unsigned integers and their
         summed result, instead of wrapping, will be saturated to the highest unsigned integer for
         the controlling type (e.g. `0xFF` for i8).
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -1722,12 +1814,13 @@ pub(crate) fn define(
             r#"
         Add with signed saturation.
 
-        This is similar to `iadd` but the operands are interpreted as signed integers and their 
-        summed result, instead of wrapping, will be saturated to the lowest or highest 
-        signed integer for the controlling type (e.g. `0x80` or `0x7F` for i8). For example, 
-        since an `iadd_ssat.i8` of `0x70` and `0x70` is greater than `0x7F`, the result will be 
+        This is similar to `iadd` but the operands are interpreted as signed integers and their
+        summed result, instead of wrapping, will be saturated to the lowest or highest
+        signed integer for the controlling type (e.g. `0x80` or `0x7F` for i8). For example,
+        since an `iadd_ssat.i8` of `0x70` and `0x70` is greater than `0x7F`, the result will be
         clamped to `0x7F`.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -1742,6 +1835,7 @@ pub(crate) fn define(
         This instruction does not depend on the signed/unsigned interpretation
         of the operands.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -1753,10 +1847,11 @@ pub(crate) fn define(
             r#"
         Subtract with unsigned saturation.
 
-        This is similar to `isub` but the operands are interpreted as unsigned integers and their 
+        This is similar to `isub` but the operands are interpreted as unsigned integers and their
         difference, instead of wrapping, will be saturated to the lowest unsigned integer for
         the controlling type (e.g. `0x00` for i8).
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -1768,10 +1863,11 @@ pub(crate) fn define(
             r#"
         Subtract with signed saturation.
 
-        This is similar to `isub` but the operands are interpreted as signed integers and their 
-        difference, instead of wrapping, will be saturated to the lowest or highest 
+        This is similar to `isub` but the operands are interpreted as signed integers and their
+        difference, instead of wrapping, will be saturated to the lowest or highest
         signed integer for the controlling type (e.g. `0x80` or `0x7F` for i8).
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -1783,6 +1879,7 @@ pub(crate) fn define(
             r#"
         Integer negation: `a := -x \pmod{2^B}`.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -1799,6 +1896,7 @@ pub(crate) fn define(
 
         Polymorphic over all integer types (vector and scalar).
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -1814,6 +1912,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -1829,6 +1928,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -1842,6 +1942,7 @@ pub(crate) fn define(
 
         This operation traps if the divisor is zero.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a])
@@ -1859,6 +1960,7 @@ pub(crate) fn define(
         representable in `B` bits two's complement. This only happens
         when `x = -2^{B-1}, y = -1`.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a])
@@ -1873,6 +1975,7 @@ pub(crate) fn define(
 
         This operation traps if the divisor is zero.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a])
@@ -1887,6 +1990,7 @@ pub(crate) fn define(
 
         This operation traps if the divisor is zero.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a])
@@ -1908,6 +2012,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -1922,6 +2027,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -1935,6 +2041,7 @@ pub(crate) fn define(
 
         This operation traps if the divisor is zero.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -1950,6 +2057,7 @@ pub(crate) fn define(
         representable in `B` bits two's complement. This only happens
         when `x = -2^{B-1}, Y = -1`.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -1963,6 +2071,7 @@ pub(crate) fn define(
 
         This operation traps if the divisor is zero.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -1976,6 +2085,7 @@ pub(crate) fn define(
 
         This operation traps if the divisor is zero.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -1994,6 +2104,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -2028,6 +2139,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![x, y, c_in])
         .operands_out(vec![a]),
@@ -2048,6 +2160,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![x, y, c_if_in])
         .operands_out(vec![a]),
@@ -2069,6 +2182,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a, c_out]),
@@ -2090,6 +2204,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a, c_if_out]),
@@ -2111,6 +2226,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![x, y, c_in])
         .operands_out(vec![a, c_out]),
@@ -2132,6 +2248,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![x, y, c_if_in])
         .operands_out(vec![a, c_if_out]),
@@ -2152,6 +2269,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![x, y, b_in])
         .operands_out(vec![a]),
@@ -2172,6 +2290,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![x, y, b_if_in])
         .operands_out(vec![a]),
@@ -2193,6 +2312,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a, b_out]),
@@ -2214,6 +2334,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a, b_if_out]),
@@ -2235,6 +2356,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![x, y, b_in])
         .operands_out(vec![a, b_out]),
@@ -2256,6 +2378,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![x, y, b_if_in])
         .operands_out(vec![a, b_if_out]),
@@ -2282,6 +2405,7 @@ pub(crate) fn define(
             r#"
         Bitwise and.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2293,6 +2417,7 @@ pub(crate) fn define(
             r#"
         Bitwise or.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2304,6 +2429,7 @@ pub(crate) fn define(
             r#"
         Bitwise xor.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2315,6 +2441,7 @@ pub(crate) fn define(
             r#"
         Bitwise not.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2328,6 +2455,7 @@ pub(crate) fn define(
 
         Computes `x & ~y`.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2341,6 +2469,7 @@ pub(crate) fn define(
 
         Computes `x | ~y`.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2354,6 +2483,7 @@ pub(crate) fn define(
 
         Computes `x ^ ~y`.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2374,6 +2504,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -2390,6 +2521,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -2406,6 +2538,7 @@ pub(crate) fn define(
         Polymorphic over all scalar integer types, but does not support vector
         types.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -2424,6 +2557,7 @@ pub(crate) fn define(
 
         Rotate the bits in ``x`` by ``y`` places.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2437,6 +2571,7 @@ pub(crate) fn define(
 
         Rotate the bits in ``x`` by ``y`` places.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2448,6 +2583,7 @@ pub(crate) fn define(
             r#"
         Rotate left by immediate.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -2459,6 +2595,7 @@ pub(crate) fn define(
             r#"
         Rotate right by immediate.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -2480,6 +2617,7 @@ pub(crate) fn define(
             a &:= x \cdot 2^s \pmod{2^B}.
         ```
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2502,6 +2640,7 @@ pub(crate) fn define(
             a &:= \lfloor x \cdot 2^{-s} \rfloor.
         ```
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2517,6 +2656,7 @@ pub(crate) fn define(
 
         The shift amount is masked to the size of the register.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2530,6 +2670,7 @@ pub(crate) fn define(
 
         The shift amount is masked to the size of ``x``.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -2543,6 +2684,7 @@ pub(crate) fn define(
 
         The shift amount is masked to the size of the register.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -2556,6 +2698,7 @@ pub(crate) fn define(
 
         The shift amount is masked to the size of the register.
         "#,
+            &formats.binary_imm,
         )
         .operands_in(vec![x, Y])
         .operands_out(vec![a]),
@@ -2572,6 +2715,7 @@ pub(crate) fn define(
 
         Reverses the bits in ``x``.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2587,6 +2731,7 @@ pub(crate) fn define(
         reaching the first one bit. When ``x`` is zero, returns the size of x
         in bits.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2602,6 +2747,7 @@ pub(crate) fn define(
         consecutive bits identical to the sign bit. When ``x`` is 0 or -1,
         returns one less than the size of x in bits.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2617,6 +2763,7 @@ pub(crate) fn define(
         reaching the first one bit. When ``x`` is zero, returns the size of x
         in bits.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2630,6 +2777,7 @@ pub(crate) fn define(
 
         Count the number of one bits in ``x``.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2712,6 +2860,7 @@ pub(crate) fn define(
         When this instruction compares floating point vectors, it returns a
         boolean vector with the results of lane-wise comparisons.
         "#,
+            &formats.float_compare,
         )
         .operands_in(vec![Cond, x, y])
         .operands_out(vec![a]),
@@ -2728,6 +2877,7 @@ pub(crate) fn define(
         Compares two numbers like `fcmp`, but returns floating point CPU
         flags instead of testing a specific condition.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![f]),
@@ -2744,6 +2894,7 @@ pub(crate) fn define(
             r#"
         Floating point addition.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2755,6 +2906,7 @@ pub(crate) fn define(
             r#"
         Floating point subtraction.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2766,6 +2918,7 @@ pub(crate) fn define(
             r#"
         Floating point multiplication.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2781,6 +2934,7 @@ pub(crate) fn define(
         `udiv`, this can't trap. Division by zero is infinity or
         NaN, depending on the dividend.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2792,6 +2946,7 @@ pub(crate) fn define(
             r#"
         Floating point square root.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2806,6 +2961,7 @@ pub(crate) fn define(
         Computes `a := xy+z` without any intermediate rounding of the
         product.
         "#,
+            &formats.ternary,
         )
         .operands_in(vec![x, y, z])
         .operands_out(vec![a]),
@@ -2821,6 +2977,7 @@ pub(crate) fn define(
 
         Note that this is a pure bitwise operation.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2836,6 +2993,7 @@ pub(crate) fn define(
 
         Note that this is a pure bitwise operation.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2856,6 +3014,7 @@ pub(crate) fn define(
         Note that this is a pure bitwise operation. The sign bit from ``y`` is
         copied to the sign bit of ``x``.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2871,6 +3030,7 @@ pub(crate) fn define(
 
         If either operand is NaN, this returns a NaN.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2886,6 +3046,7 @@ pub(crate) fn define(
 
         If either operand is NaN, this returns a NaN.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
@@ -2899,6 +3060,7 @@ pub(crate) fn define(
             r#"
         Round floating point round to integral, towards positive infinity.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2910,6 +3072,7 @@ pub(crate) fn define(
             r#"
         Round floating point round to integral, towards negative infinity.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2921,6 +3084,7 @@ pub(crate) fn define(
             r#"
         Round floating point round to integral, towards zero.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2933,6 +3097,7 @@ pub(crate) fn define(
         Round floating point round to integral, towards nearest with ties to
         even.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2950,6 +3115,7 @@ pub(crate) fn define(
         The condition code determines if the reference type in question is
         null or not.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -2968,6 +3134,7 @@ pub(crate) fn define(
         Check the CPU flags in ``f`` against the ``Cond`` condition code and
         return true when the condition code is satisfied.
         "#,
+            &formats.int_cond,
         )
         .operands_in(vec![Cond, f])
         .operands_out(vec![a]),
@@ -2985,6 +3152,7 @@ pub(crate) fn define(
         Check the CPU flags in ``f`` against the ``Cond`` condition code and
         return true when the condition code is satisfied.
         "#,
+            &formats.float_cond,
         )
         .operands_in(vec![Cond, f])
         .operands_out(vec![a]),
@@ -3003,6 +3171,7 @@ pub(crate) fn define(
         size. A bitcast is equivalent to storing one type and loading the other
         type from the same address.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -3025,6 +3194,7 @@ pub(crate) fn define(
         lane is a raw_bitcast of the corresponding operand lane. TODO there is
         currently no mechanism for enforcing the bit width constraint.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -3037,12 +3207,13 @@ pub(crate) fn define(
         Inst::new(
             "scalar_to_vector",
             r#"
-    Scalar To Vector -- move a value out of a scalar register and into a vector register; the 
-    scalar will be moved to the lowest-order bits of the vector register. Note that this 
-    instruction is intended as a low-level legalization instruction and frontends should prefer 
+    Scalar To Vector -- move a value out of a scalar register and into a vector register; the
+    scalar will be moved to the lowest-order bits of the vector register. Note that this
+    instruction is intended as a low-level legalization instruction and frontends should prefer
     insertlane; on certain architectures, scalar_to_vector may zero the highest-order bits for some
     types (e.g. integers) but not for others (e.g. floats).
     "#,
+            &formats.unary,
         )
         .operands_in(vec![s])
         .operands_out(vec![a]),
@@ -3079,6 +3250,7 @@ pub(crate) fn define(
         and each lane must not have more bits that the input lanes. If the
         input and output types are the same, this is a no-op.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
@@ -3106,6 +3278,7 @@ pub(crate) fn define(
         and each lane must not have fewer bits that the input lanes. If the
         input and output types are the same, this is a no-op.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
@@ -3132,6 +3305,7 @@ pub(crate) fn define(
         True maps to 1 and false maps to 0. The result type must have the same
         number of vector lanes as the input.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -3146,6 +3320,7 @@ pub(crate) fn define(
         True maps to all 1s and false maps to all 0s. The result type must have
         the same number of vector lanes as the input.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -3185,6 +3360,7 @@ pub(crate) fn define(
         and each lane must not have more bits that the input lanes. If the
         input and output types are the same, this is a no-op.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
@@ -3216,6 +3392,7 @@ pub(crate) fn define(
         and each lane must not have fewer bits that the input lanes. If the
         input and output types are the same, this is a no-op.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
@@ -3236,6 +3413,7 @@ pub(crate) fn define(
         and each lane must not have fewer bits that the input lanes. If the
         input and output types are the same, this is a no-op.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
@@ -3269,6 +3447,7 @@ pub(crate) fn define(
         and the result lanes must not have fewer bits than the input lanes. If
         the input and output types are the same, this is a no-op.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
@@ -3291,6 +3470,7 @@ pub(crate) fn define(
         and the result lanes must not have more bits than the input lanes. If
         the input and output types are the same, this is a no-op.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
@@ -3312,6 +3492,7 @@ pub(crate) fn define(
 
         The result type must have the same number of vector lanes as the input.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
@@ -3326,6 +3507,7 @@ pub(crate) fn define(
         saturates the input instead of trapping. NaN and negative values are
         converted to 0.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -3343,6 +3525,7 @@ pub(crate) fn define(
 
         The result type must have the same number of vector lanes as the input.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a])
@@ -3356,6 +3539,7 @@ pub(crate) fn define(
         Convert floating point to signed integer as fcvt_to_sint does, but
         saturates the input instead of trapping. NaN values are converted to 0.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -3375,6 +3559,7 @@ pub(crate) fn define(
 
         The result type must have the same number of vector lanes as the input.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -3391,6 +3576,7 @@ pub(crate) fn define(
 
         The result type must have the same number of vector lanes as the input.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![a]),
@@ -3420,6 +3606,7 @@ pub(crate) fn define(
         Returns the low half of `x` and the high half of `x` as two independent
         values.
         "#,
+            &formats.unary,
         )
         .operands_in(vec![x])
         .operands_out(vec![lo, hi])
@@ -3453,6 +3640,7 @@ pub(crate) fn define(
         the same number of lanes as the inputs, but the lanes are twice the
         size.
         "#,
+            &formats.binary,
         )
         .operands_in(vec![lo, hi])
         .operands_out(vec![a])
diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
index 73cdb4254d..808b6bbc49 100644
--- a/cranelift/codegen/meta/src/shared/mod.rs
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -8,20 +8,25 @@ pub mod legalize;
 pub mod settings;
 pub mod types;
 
-use crate::cdsl::formats::FormatRegistry;
+use crate::cdsl::formats::{FormatStructure, InstructionFormat};
 use crate::cdsl::instructions::{AllInstructions, InstructionGroup};
 use crate::cdsl::settings::SettingGroup;
 use crate::cdsl::xform::TransformGroups;
 
 use crate::shared::entities::EntityRefs;
+use crate::shared::formats::Formats;
 use crate::shared::immediates::Immediates;
 
+use std::collections::HashMap;
+use std::iter::FromIterator;
+use std::rc::Rc;
+
 pub(crate) struct Definitions {
     pub settings: SettingGroup,
     pub all_instructions: AllInstructions,
     pub instructions: InstructionGroup,
     pub imm: Immediates,
-    pub format_registry: FormatRegistry,
+    pub formats: Formats,
     pub transform_groups: TransformGroups,
 }
 
@@ -30,13 +35,9 @@ pub(crate) fn define() -> Definitions {
 
     let immediates = Immediates::new();
     let entities = EntityRefs::new();
-    let format_registry = formats::define(&immediates, &entities);
-    let instructions = instructions::define(
-        &mut all_instructions,
-        &format_registry,
-        &immediates,
-        &entities,
-    );
+    let formats = Formats::new(&immediates, &entities);
+    let instructions =
+        instructions::define(&mut all_instructions, &formats, &immediates, &entities);
     let transform_groups = legalize::define(&instructions, &immediates);
 
     Definitions {
@@ -44,7 +45,53 @@ pub(crate) fn define() -> Definitions {
         all_instructions,
         instructions,
         imm: immediates,
-        format_registry,
+        formats,
         transform_groups,
     }
 }
+
+impl Definitions {
+    /// Verifies certain properties of formats.
+    ///
+    /// - Formats must be uniquely named: if two formats have the same name, they must refer to the
+    /// same data. Otherwise, two format variants in the codegen crate would have the same name.
+    /// - Formats must be structurally different from each other. Otherwise, this would lead to
+    /// code duplicate in the codegen crate.
+    ///
+    /// Returns a list of all the instruction formats effectively used.
+    pub fn verify_instruction_formats(&self) -> Vec<&InstructionFormat> {
+        let mut format_names: HashMap<&'static str, &Rc> = HashMap::new();
+
+        // A structure is: number of input value operands / whether there's varargs or not / names
+        // of immediate fields.
+        let mut format_structures: HashMap = HashMap::new();
+
+        for inst in self.all_instructions.values() {
+            // Check name.
+            if let Some(existing_format) = format_names.get(&inst.format.name) {
+                assert!(
+                    Rc::ptr_eq(&existing_format, &inst.format),
+                    "formats must uniquely named; there's a\
+                     conflict on the name '{}', please make sure it is used only once.",
+                    existing_format.name
+                );
+            } else {
+                format_names.insert(inst.format.name, &inst.format);
+            }
+
+            // Check structure.
+            let key = inst.format.structure();
+            if let Some(existing_format) = format_structures.get(&key) {
+                assert_eq!(
+                    existing_format.name, inst.format.name,
+                    "duplicate instruction formats {} and {}; please remove one.",
+                    existing_format.name, inst.format.name
+                );
+            } else {
+                format_structures.insert(key, &inst.format);
+            }
+        }
+
+        Vec::from_iter(format_structures.into_iter().map(|(_, v)| v))
+    }
+}

From f721821bec49f1b34dfda5cb7cc79f18abd0ae47 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 17 Oct 2019 17:15:56 +0200
Subject: [PATCH 2868/3084] [contribution] Add Github issue templates;

---
 cranelift/.github/ISSUE_TEMPLATE/bug-report.md  | 16 ++++++++++++++++
 cranelift/.github/ISSUE_TEMPLATE/improvement.md | 15 +++++++++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 cranelift/.github/ISSUE_TEMPLATE/bug-report.md
 create mode 100644 cranelift/.github/ISSUE_TEMPLATE/improvement.md

diff --git a/cranelift/.github/ISSUE_TEMPLATE/bug-report.md b/cranelift/.github/ISSUE_TEMPLATE/bug-report.md
new file mode 100644
index 0000000000..8038372339
--- /dev/null
+++ b/cranelift/.github/ISSUE_TEMPLATE/bug-report.md
@@ -0,0 +1,16 @@
+---
+name: "Bug report"
+about: "Report a bug or a crash in Cranelift."
+labels: 'bug'
+---
+
+Thanks for opening a bug report on Cranelift! Please answer the questions below
+if they're relevant and delete this text before submitting.
+
+- What are the steps to reproduce the issue? Can you include a CLIF test case,
+  ideally reduced with the `bugpoint` clif-util command?
+- What do you expect to happen? What does actually happen? Does it panic, and
+  if so, with which assertion?
+- Which Cranelift version / commit hash / branch are you using?
+- If relevant, can you include some extra information about your environment?
+  (Rust version, operating system, architecture...)
diff --git a/cranelift/.github/ISSUE_TEMPLATE/improvement.md b/cranelift/.github/ISSUE_TEMPLATE/improvement.md
new file mode 100644
index 0000000000..16e082466e
--- /dev/null
+++ b/cranelift/.github/ISSUE_TEMPLATE/improvement.md
@@ -0,0 +1,15 @@
+---
+name: "Improvement"
+about: "A feature request or code improvement."
+---
+
+Please try to describe precisely what you would like to do in Cranelift and/or
+expect from it. You can answer the questions below if they're relevant and
+delete this text before submitting. Thanks for opening an issue!
+
+- What is the feature or code improvement you would like to do in Cranelift?
+- What is the value of adding this in Cranelift?
+- Do you have an implementation plan, and/or ideas for data structures or
+  algorithms to use?
+- Have you considered alternative implementations? If so, how are they better
+  or worse than your proposal?

From 499413dbc4eaf0ef31de38b3f0d64a167e776047 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 17 Oct 2019 17:27:08 +0200
Subject: [PATCH 2869/3084] [contribution] Add a pull request template.

---
 cranelift/.github/pull_request_template.md | 12 ++++++++++++
 1 file changed, 12 insertions(+)
 create mode 100644 cranelift/.github/pull_request_template.md

diff --git a/cranelift/.github/pull_request_template.md b/cranelift/.github/pull_request_template.md
new file mode 100644
index 0000000000..d125ec7a7f
--- /dev/null
+++ b/cranelift/.github/pull_request_template.md
@@ -0,0 +1,12 @@
+- [ ] This has been discussed in issue #..., or if not, please tell us why
+  here.
+- [ ] A short description of what this does, why it is needed; if the
+  description becomes long, the matter should probably be discussed in an issue
+  first.
+- [ ] This PR contains test cases, if meaningful.
+- [ ] A reviewer from the core maintainer team has been assigned for this PR.
+  If you don't know who could review this, please indicate so and/or ping
+  `bnjbvr`. The list of suggested reviewers on the right can help you.
+
+

From 873465e7a9d6e133044d5e660e7beb82d5a2d30b Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 17 Oct 2019 17:39:44 +0200
Subject: [PATCH 2870/3084] [contribution] Update contribution guidelines.

---
 cranelift/CONTRIBUTING.md | 41 ++++++++++++++++++++++++++++++++-------
 1 file changed, 34 insertions(+), 7 deletions(-)

diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
index 97ce756eb9..e5e5b9410f 100644
--- a/cranelift/CONTRIBUTING.md
+++ b/cranelift/CONTRIBUTING.md
@@ -69,8 +69,8 @@ version. See the [rustfmt quickstart] for setup.
 [format-all.sh] is a script for running the appropriate version of rustfmt,
 which may be convenient when there are multiple versions installed.
 
-[rustfmt-preview]: https://github.com/rust-lang-nursery/rustfmt
-[rustfmt quickstart]: https://github.com/rust-lang-nursery/rustfmt#quick-start
+[rustfmt-preview]: https://github.com/rust-lang/rustfmt
+[rustfmt quickstart]: https://github.com/rust-lang/rustfmt#quick-start
 [format-all.sh]: https://github.com/CraneStation/cranelift/blob/master/format-all.sh
 
 ### Rustc version support
@@ -90,7 +90,20 @@ We use [issues] for asking questions and tracking bugs and unimplemented
 features, and [pull requests] (PRs) for tracking and reviewing code
 submissions.
 
-When submitting PRs:
+### Before submitting a PR
+
+Consider opening an issue to talk about it. PRs without corresponding issues
+are appropriate for fairly narrow technical matters, not for fixes to
+user-facing bugs or for feature implementations, especially when those features
+might have multiple implementation strategies that usefully could be discussed.
+
+Our issue templates might help you through the process.
+
+### When submitting PRs
+
+ - Please fill in the pull request template as appropriate. It is usually
+   helpful, it speeds up the review process and helps understanding the changes
+   brought by the PR.
 
  - Write clear commit messages that start with a one-line summary of the
    change (and if it's difficult to summarize in one line, consider
@@ -106,8 +119,14 @@ When submitting PRs:
  - For pull requests that fix existing issues, use [issue keywords]. Note that
    not all pull requests need to have accompanying issues.
 
-Anyone may submit a pull request, and anyone may comment on or review others'
-pull requests. Pull requests are merged by members of the [Core Team].
+ - Assign the review to somebody from the [Core Team], either using suggestions
+   in the list proposed by Github, or somebody else if you have a specific
+   person in mind.
+
+ - When updating your pull request, please make sure to re-request review if
+   the request has been cancelled.
+
+### Focused commits or squashing
 
 We generally squash sequences of incremental-development commits together into
 logical commits (though keeping logical commits focused). Developers may do
@@ -115,8 +134,16 @@ this themselves before submitting a PR or during the PR process, or Core Team
 members may do it when merging a PR. Ideally, the continuous-integration tests
 should pass at each logical commit.
 
-Core Team members may push minor changes directly, though should create PRs
-for significant changes.
+### Review and merge
+
+Anyone may submit a pull request, and anyone may comment on or review others'
+pull requests. However, one review from somebody in the [Core Team] is required
+before the Core Team merges it.
+
+Even Core Team members should create PRs for every change, including minor work
+items (version bump, removing warnings, etc.): this is helpful to keep track of
+what has happened on the repository. Very minor changes may be merged without a
+review, although it is always preferred to have one.
 
 [issues]: https://guides.github.com/features/issues/
 [pull requests]: https://help.github.com/articles/about-pull-requests/

From 186effc420b26b104ff9ed63ef1c94927c706a02 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 18 Oct 2019 15:35:27 -0700
Subject: [PATCH 2871/3084] Add x86 SIMD vany_true and x86_ptest

In order to implement SIMD's any_true (https://github.com/WebAssembly/simd/blob/master/proposals/simd/SIMD.md#any-lane-true), we must legalize some instruction (I chose `vany_true`) to a sequence of `PTEST` and `SETNZ`. To emit `PTEST` I added the new CLIF instruction `x86_ptest` and used CLIF's `trueif ne` for `SETNZ`.
---
 .../codegen/meta/src/isa/x86/encodings.rs     |  5 +++++
 .../codegen/meta/src/isa/x86/instructions.rs  | 19 +++++++++++++++++++
 .../codegen/meta/src/isa/x86/legalize.rs      | 13 +++++++++++++
 cranelift/codegen/meta/src/isa/x86/opcodes.rs |  4 ++++
 .../codegen/meta/src/shared/instructions.rs   | 16 ++++++++++++++++
 .../isa/x86/simd-logical-binemit.clif         |  6 ++++++
 .../isa/x86/simd-logical-legalize.clif        |  8 ++++++++
 .../filetests/isa/x86/simd-logical-run.clif   | 18 ++++++++++++++++++
 8 files changed, 89 insertions(+)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 94ade711a1..3a8f8b696c 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -521,6 +521,7 @@ pub(crate) fn define(
     let x86_psll = x86.by_name("x86_psll");
     let x86_psra = x86.by_name("x86_psra");
     let x86_psrl = x86.by_name("x86_psrl");
+    let x86_ptest = x86.by_name("x86_ptest");
     let x86_push = x86.by_name("x86_push");
     let x86_sdivmodx = x86.by_name("x86_sdivmodx");
     let x86_smulx = x86.by_name("x86_smulx");
@@ -1988,6 +1989,10 @@ pub(crate) fn define(
         // xor
         let bxor = bxor.bind(vector(ty, sse_vector_size));
         e.enc_32_64(bxor, rec_fa.opcodes(&PXOR));
+
+        // ptest
+        let x86_ptest = x86_ptest.bind(vector(ty, sse_vector_size));
+        e.enc_32_64_maybe_isap(x86_ptest, rec_fcmp.opcodes(&PTEST), Some(use_sse41_simd));
     }
 
     // SIMD bitcast from I32/I64 to the low bits of a vector (e.g. I64x2); this register movement
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 21e51982ce..366df755b9 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -467,5 +467,24 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    let x = &operand("x", TxN);
+    let y = &operand("y", TxN);
+    let f = &operand("f", iflags);
+    ig.push(
+        Inst::new(
+            "x86_ptest",
+            r#"
+        Logical Compare -- PTEST will set the ZF flag if all bits in the result are 0 of the 
+        bitwise AND of the first source operand (first operand) and the second source operand 
+        (second operand). PTEST sets the CF flag if all bits in the result are 0 of the bitwise  
+        AND of the second source operand (second operand) and the logical NOT of the destination
+        operand (first operand).
+        "#,
+            &formats.binary,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![f]),
+    );
+
     ig.build()
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index df0b88dc04..8af7f3447a 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -55,12 +55,14 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let shuffle = insts.by_name("shuffle");
     let srem = insts.by_name("srem");
     let sshr = insts.by_name("sshr");
+    let trueif = insts.by_name("trueif");
     let udiv = insts.by_name("udiv");
     let umulhi = insts.by_name("umulhi");
     let ushr_imm = insts.by_name("ushr_imm");
     let urem = insts.by_name("urem");
     let ushr = insts.by_name("ushr");
     let vconst = insts.by_name("vconst");
+    let vany_true = insts.by_name("vany_true");
 
     let x86_bsf = x86_instructions.by_name("x86_bsf");
     let x86_bsr = x86_instructions.by_name("x86_bsr");
@@ -69,6 +71,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let x86_psll = x86_instructions.by_name("x86_psll");
     let x86_psra = x86_instructions.by_name("x86_psra");
     let x86_psrl = x86_instructions.by_name("x86_psrl");
+    let x86_ptest = x86_instructions.by_name("x86_ptest");
     let x86_umulx = x86_instructions.by_name("x86_umulx");
     let x86_smulx = x86_instructions.by_name("x86_smulx");
 
@@ -446,6 +449,16 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         );
     }
 
+    // SIMD vany_true
+    let ne = Literal::enumerator_for(&imm.intcc, "ne");
+    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
+        let vany_true = vany_true.bind(vector(ty, sse_vector_size));
+        narrow.legalize(
+            def!(y = vany_true(x)),
+            vec![def!(a = x86_ptest(x, x)), def!(y = trueif(ne, a))],
+        );
+    }
+
     narrow.custom_legalize(shuffle, "convert_shuffle");
     narrow.custom_legalize(extractlane, "convert_extractlane");
     narrow.custom_legalize(insertlane, "convert_insertlane");
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index 2df259f37e..0491028810 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -375,6 +375,10 @@ pub static PSUBUSB: [u8; 3] = [0x66, 0x0f, 0xd8];
 /// and saturate results (SSE2).
 pub static PSUBUSW: [u8; 3] = [0x66, 0x0f, 0xd9];
 
+/// Set ZF if xmm2/m128 AND xmm1 result is all 0s; set CF if xmm2/m128 AND NOT xmm1 result is all
+/// 0s (SSE4.1).
+pub static PTEST: [u8; 4] = [0x66, 0x0f, 0x38, 0x17];
+
 /// Push r{16,32,64}.
 pub static PUSH_REG: [u8; 1] = [0x50];
 
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 1980730d95..6d0b79ccfa 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1623,6 +1623,22 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    let s = &operand("s", b1);
+
+    ig.push(
+        Inst::new(
+            "vany_true",
+            r#"
+        Reduce a vector to a scalar boolean.
+
+        Return a scalar boolean true if any lane in ``a`` is non-zero, false otherwise.
+        "#,
+            &formats.unary,
+        )
+        .operands_in(vec![a])
+        .operands_out(vec![s]),
+    );
+
     let x = &operand("x", &TxN.lane_of());
 
     ig.push(
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
index 835eb8ca2f..6d6a3fac31 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
@@ -25,3 +25,9 @@ ebb0(v0: b64x2 [%xmm6], v1: b64x2 [%xmm3]):
 [-, %xmm3]  v2 = band_not v0, v1      ; bin: 66 0f df de
             return v2
 }
+
+function %x86_ptest_f64x2(f64x2, f64x2) {
+ebb0(v0: f64x2 [%xmm0], v1: f64x2 [%xmm2]):
+[-, %rflags]  v2 = x86_ptest v0, v1      ; bin: 66 0f 38 17 c2
+            return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif
index be00fe7278..925e18573c 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif
@@ -9,3 +9,11 @@ ebb0(v0: b32x4):
     ; nextln: v1 = bxor v2, v0
     return v1
 }
+
+function %vany_true_b32x4(b32x4) -> b1 {
+ebb0(v0: b32x4):
+    v1 = vany_true v0
+    ; check: v2 = x86_ptest v0, v0
+    ; nextln: v1 = trueif ne v2
+    return v1
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif
index 6ab5db0c49..2a85c44b04 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif
@@ -21,3 +21,21 @@ ebb0:
     return v4
 }
 ; run
+
+function %vany_true_i16x8() -> b1 {
+ebb0:
+    v0 = vconst.i16x8 [1 0 0 0 0 0 0 0]
+    v1 = vany_true v0
+    return v1
+}
+; run
+
+function %vany_true_b32x4() -> b1 {
+ebb0:
+    v0 = vconst.b32x4 [false false false false]
+    v1 = vany_true v0
+    v2 = bint.i32 v1
+    v3 = icmp_imm eq v2, 0
+    return v3
+}
+; run

From 65e18df12ff9c8a451e8be773b12477a9701ca22 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 18 Oct 2019 15:40:17 -0700
Subject: [PATCH 2872/3084] Translate WASM any_true to CLIF

---
 cranelift/wasm/src/code_translator.rs | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 01c72fab2a..f4bdc601bb 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1147,6 +1147,13 @@ pub fn translate_operator(
             // operands must match (hence the bitcast).
             state.push1(builder.ins().bitselect(bitcast_c, bitcast_a, bitcast_b))
         }
+        Operator::I8x16AnyTrue
+        | Operator::I16x8AnyTrue
+        | Operator::I32x4AnyTrue
+        | Operator::I64x2AnyTrue => {
+            let bool_result = builder.ins().vany_true(state.pop1());
+            state.push1(builder.ins().bint(I32, bool_result))
+        }
         Operator::I8x16Eq
         | Operator::I8x16Ne
         | Operator::I8x16LtS
@@ -1189,17 +1196,13 @@ pub fn translate_operator(
         | Operator::F64x2Gt
         | Operator::F64x2Le
         | Operator::F64x2Ge
-        | Operator::I8x16AnyTrue
         | Operator::I8x16AllTrue
         | Operator::I8x16Shl
         | Operator::I8x16ShrS
         | Operator::I8x16ShrU
         | Operator::I8x16Mul
-        | Operator::I16x8AnyTrue
         | Operator::I16x8AllTrue
-        | Operator::I32x4AnyTrue
         | Operator::I32x4AllTrue
-        | Operator::I64x2AnyTrue
         | Operator::I64x2AllTrue
         | Operator::I64x2ShrS
         | Operator::F32x4Abs

From 879ccf871ac1be37152a20a404e99b77fe06be7e Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 18 Oct 2019 16:21:06 -0700
Subject: [PATCH 2873/3084] Add x86 SIMD vall_true

In order to implement SIMD's all_true (https://github.com/WebAssembly/simd/blob/master/proposals/simd/SIMD.md#all-lanes-true), we must legalize some instruction (I chose `vall_true`) to a comparison against 0 and a similar reduction as vany_true using `PTEST` and `SETNZ`. Since `icmp` only allows integers but `vall_true` could allow more vector types, `raw_bitcast` is used to convert the lane types into integers, e.g. b32x4 to i32x4. To do so without runtime type-checking, the `raw_bitcast` instruction (which emits no instruction) can now bitcast from any vector type to the same type, e.g. i32x4 to i32x4.
---
 .../codegen/meta/src/isa/x86/legalize.rs      | 36 +++++++++++++++++++
 .../codegen/meta/src/shared/instructions.rs   | 14 ++++++++
 .../isa/x86/simd-logical-legalize.clif        | 10 ++++++
 .../filetests/isa/x86/simd-logical-run.clif   | 18 ++++++++++
 4 files changed, 78 insertions(+)

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 8af7f3447a..68ca67685f 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -39,6 +39,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let fmax = insts.by_name("fmax");
     let fmin = insts.by_name("fmin");
     let iadd = insts.by_name("iadd");
+    let icmp = insts.by_name("icmp");
     let iconst = insts.by_name("iconst");
     let imul = insts.by_name("imul");
     let ineg = insts.by_name("ineg");
@@ -62,6 +63,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let urem = insts.by_name("urem");
     let ushr = insts.by_name("ushr");
     let vconst = insts.by_name("vconst");
+    let vall_true = insts.by_name("vall_true");
     let vany_true = insts.by_name("vany_true");
 
     let x86_bsf = x86_instructions.by_name("x86_bsf");
@@ -459,6 +461,40 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         );
     }
 
+    // SIMD vall_true
+    let zeroes = constant(vec![0x00; 16]);
+    let eq = Literal::enumerator_for(&imm.intcc, "eq");
+    for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
+        let vall_true = vall_true.bind(vector(ty, sse_vector_size));
+        if ty.is_int() {
+            // In the common case (Wasm's integer-only all_true), we do not require a bitcast.
+            narrow.legalize(
+                def!(y = vall_true(x)),
+                vec![
+                    def!(a = vconst(zeroes)),
+                    def!(c = icmp(eq, x, a)),
+                    def!(d = x86_ptest(c, c)),
+                    def!(y = trueif(eq, d)),
+                ],
+            );
+        } else {
+            // However, to support other types we must bitcast them to an integer vector to use
+            // icmp.
+            let lane_type_as_int = LaneType::int_from_bits(ty.lane_bits() as u16);
+            let raw_bitcast_to_int = raw_bitcast.bind(vector(lane_type_as_int, sse_vector_size));
+            narrow.legalize(
+                def!(y = vall_true(x)),
+                vec![
+                    def!(a = vconst(zeroes)),
+                    def!(b = raw_bitcast_to_int(x)),
+                    def!(c = icmp(eq, b, a)),
+                    def!(d = x86_ptest(c, c)),
+                    def!(y = trueif(eq, d)),
+                ],
+            );
+        }
+    }
+
     narrow.custom_legalize(shuffle, "convert_shuffle");
     narrow.custom_legalize(extractlane, "convert_extractlane");
     narrow.custom_legalize(insertlane, "convert_insertlane");
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 6d0b79ccfa..532fee6c87 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1639,6 +1639,20 @@ pub(crate) fn define(
         .operands_out(vec![s]),
     );
 
+    ig.push(
+        Inst::new(
+            "vall_true",
+            r#"
+        Reduce a vector to a scalar boolean.
+
+        Return a scalar boolean true if all lanes in ``i`` are non-zero, false otherwise.
+        "#,
+            &formats.unary,
+        )
+        .operands_in(vec![a])
+        .operands_out(vec![s]),
+    );
+
     let x = &operand("x", &TxN.lane_of());
 
     ig.push(
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif
index 925e18573c..2e13f79b9b 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif
@@ -17,3 +17,13 @@ ebb0(v0: b32x4):
     ; nextln: v1 = trueif ne v2
     return v1
 }
+
+function %vall_true_i64x2(i64x2) -> b1 {
+ebb0(v0: i64x2):
+    v1 = vall_true v0
+    ; check: v2 = vconst.i64x2 0x00
+    ; nextln: v3 = icmp eq v0, v2
+    ; nextln: v4 = x86_ptest v3, v3
+    ; nextln: v1 = trueif eq v4
+    return v1
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif
index 2a85c44b04..9b525f2e10 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif
@@ -39,3 +39,21 @@ ebb0:
     return v3
 }
 ; run
+
+function %vall_true_i16x8() -> b1 {
+ebb0:
+    v0 = vconst.i16x8 [1 0 0 0 0 0 0 0]
+    v1 = vall_true v0
+    v2 = bint.i32 v1
+    v3 = icmp_imm eq v2, 0
+    return v3
+}
+; run
+
+function %vall_true_b32x4() -> b1 {
+ebb0:
+    v0 = vconst.b32x4 [true true true true]
+    v1 = vall_true v0
+    return v1
+}
+; run

From 77035b44a62f05e2f36d0496de89948690151bf5 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 18 Oct 2019 16:23:31 -0700
Subject: [PATCH 2874/3084] Translate WASM all_true to CLIF

---
 cranelift/wasm/src/code_translator.rs | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index f4bdc601bb..709d40894f 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1154,6 +1154,13 @@ pub fn translate_operator(
             let bool_result = builder.ins().vany_true(state.pop1());
             state.push1(builder.ins().bint(I32, bool_result))
         }
+        Operator::I8x16AllTrue
+        | Operator::I16x8AllTrue
+        | Operator::I32x4AllTrue
+        | Operator::I64x2AllTrue => {
+            let bool_result = builder.ins().vall_true(state.pop1());
+            state.push1(builder.ins().bint(I32, bool_result))
+        }
         Operator::I8x16Eq
         | Operator::I8x16Ne
         | Operator::I8x16LtS
@@ -1196,14 +1203,10 @@ pub fn translate_operator(
         | Operator::F64x2Gt
         | Operator::F64x2Le
         | Operator::F64x2Ge
-        | Operator::I8x16AllTrue
         | Operator::I8x16Shl
         | Operator::I8x16ShrS
         | Operator::I8x16ShrU
         | Operator::I8x16Mul
-        | Operator::I16x8AllTrue
-        | Operator::I32x4AllTrue
-        | Operator::I64x2AllTrue
         | Operator::I64x2ShrS
         | Operator::F32x4Abs
         | Operator::F32x4Neg

From 2b6ea31621c97ca05975eda308fbeda8b9e161d5 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Thu, 10 Oct 2019 15:58:13 +0200
Subject: [PATCH 2875/3084] [wasm] Include more large tests;

---
 cranelift/wasmtests/embenchen_fannkuch.wat | 16725 +++++++++++++++++++
 cranelift/wasmtests/embenchen_fasta.wat    | 16547 ++++++++++++++++++
 cranelift/wasmtests/embenchen_ifs.wat      | 15771 +++++++++++++++++
 cranelift/wasmtests/embenchen_primes.wat   | 15334 +++++++++++++++++
 cranelift/wasmtests/rust_fannkuch.wat      |  2511 +++
 5 files changed, 66888 insertions(+)
 create mode 100644 cranelift/wasmtests/embenchen_fannkuch.wat
 create mode 100644 cranelift/wasmtests/embenchen_fasta.wat
 create mode 100644 cranelift/wasmtests/embenchen_ifs.wat
 create mode 100644 cranelift/wasmtests/embenchen_primes.wat
 create mode 100644 cranelift/wasmtests/rust_fannkuch.wat

diff --git a/cranelift/wasmtests/embenchen_fannkuch.wat b/cranelift/wasmtests/embenchen_fannkuch.wat
new file mode 100644
index 0000000000..c61b5e6e2f
--- /dev/null
+++ b/cranelift/wasmtests/embenchen_fannkuch.wat
@@ -0,0 +1,16725 @@
+(module
+ (type $0 (func (param i32 i32 i32) (result i32)))
+ (type $1 (func (param i32) (result i32)))
+ (type $2 (func (param i32)))
+ (type $3 (func (result i32)))
+ (type $4 (func (param i32 i32) (result i32)))
+ (type $5 (func (param i32 i32)))
+ (type $6 (func))
+ (type $7 (func (param i32 i32 i32 i32 i32) (result i32)))
+ (type $8 (func (param i32 i32 i32)))
+ (type $9 (func (param i64 i32) (result i32)))
+ (type $10 (func (param i32 i32 i32 i32 i32)))
+ (type $11 (func (param f64 i32) (result f64)))
+ (type $12 (func (param i32 i32 i32 i32) (result i32)))
+ (import "env" "memory" (memory $16 2048 2048))
+ (data (i32.const 1024) "\04\04\00\00\05")
+ (data (i32.const 1040) "\01")
+ (data (i32.const 1064) "\01\00\00\00\02\00\00\00<\10\00\00\00\04")
+ (data (i32.const 1088) "\01")
+ (data (i32.const 1103) "\n\ff\ff\ff\ff")
+ (data (i32.const 1140) "error: %d\n\00Pfannkuchen(%d) = %d.\n\00%d\00\11\00\n\00\11\11\11\00\00\00\00\05\00\00\00\00\00\00\t\00\00\00\00\0b")
+ (data (i32.const 1209) "\11\00\0f\n\11\11\11\03\n\07\00\01\13\t\0b\0b\00\00\t\06\0b\00\00\0b\00\06\11\00\00\00\11\11\11")
+ (data (i32.const 1258) "\0b")
+ (data (i32.const 1267) "\11\00\n\n\11\11\11\00\n\00\00\02\00\t\0b\00\00\00\t\00\0b\00\00\0b")
+ (data (i32.const 1316) "\0c")
+ (data (i32.const 1328) "\0c\00\00\00\00\0c\00\00\00\00\t\0c\00\00\00\00\00\0c\00\00\0c")
+ (data (i32.const 1374) "\0e")
+ (data (i32.const 1386) "\0d\00\00\00\04\0d\00\00\00\00\t\0e\00\00\00\00\00\0e\00\00\0e")
+ (data (i32.const 1432) "\10")
+ (data (i32.const 1444) "\0f\00\00\00\00\0f\00\00\00\00\t\10\00\00\00\00\00\10\00\00\10\00\00\12\00\00\00\12\12\12")
+ (data (i32.const 1499) "\12\00\00\00\12\12\12\00\00\00\00\00\00\t")
+ (data (i32.const 1548) "\0b")
+ (data (i32.const 1560) "\n\00\00\00\00\n\00\00\00\00\t\0b\00\00\00\00\00\0b\00\00\0b")
+ (data (i32.const 1606) "\0c")
+ (data (i32.const 1618) "\0c\00\00\00\00\0c\00\00\00\00\t\0c\00\00\00\00\00\0c\00\00\0c\00\000123456789ABCDEF-+   0X0x\00(null)\00-0X+0X 0X-0x+0x 0x\00inf\00INF\00nan\00NAN\00.\00T!\"\19\0d\01\02\03\11K\1c\0c\10\04\0b\1d\12\1e\'hnopqb \05\06\0f\13\14\15\1a\08\16\07($\17\18\t\n\0e\1b\1f%#\83\82}&*+<=>?CGJMXYZ[\\]^_`acdefgijklrstyz{|\00Illegal byte sequence\00Domain error\00Result not representable\00Not a tty\00Permission denied\00Operation not permitted\00No such file or directory\00No such process\00File exists\00Value too large for data type\00No space left on device\00Out of memory\00Resource busy\00Interrupted system call\00Resource temporarily unavailable\00Invalid seek\00Cross-device link\00Read-only file system\00Directory not empty\00Connection reset by peer\00Operation timed out\00Connection refused\00Host is down\00Host is unreachable\00Address in use\00Broken pipe\00I/O error\00No such device or address\00Block device required\00No such device\00Not a directory\00Is a directory\00Text file busy\00Exec format error\00Invalid argument\00Argument list too long\00Symbolic link loop\00Filename too long\00Too many open files in system\00No file descriptors available\00Bad file descriptor\00No child process\00Bad address\00File too large\00Too many links\00No locks available\00Resource deadlock would occur\00State not recoverable\00Previous owner died\00Operation canceled\00Function not implemented\00No message of desired type\00Identifier removed\00Device not a stream\00No data available\00Device timeout\00Out of streams resources\00Link has been severed\00Protocol error\00Bad message\00File descriptor in bad state\00Not a socket\00Destination address required\00Message too large\00Protocol wrong type for socket\00Protocol not available\00Protocol not supported\00Socket type not supported\00Not supported\00Protocol family not supported\00Address family not supported by protocol\00Address not available\00Network is down\00Network unreachable\00Connection reset by network\00Connection aborted\00No buffer space available\00Socket is connected\00Socket not connected\00Cannot send after socket shutdown\00Operation already in progress\00Operation in progress\00Stale file handle\00Remote I/O error\00Quota exceeded\00No medium found\00Wrong medium type\00No error information")
+ (import "env" "table" (table $timport$17 8 8 funcref))
+ (elem (global.get $gimport$19) $45 $9 $46 $14 $10 $15 $47 $16)
+ (import "env" "DYNAMICTOP_PTR" (global $gimport$0 i32))
+ (import "env" "STACKTOP" (global $gimport$1 i32))
+ (import "env" "STACK_MAX" (global $gimport$2 i32))
+ (import "env" "memoryBase" (global $gimport$18 i32))
+ (import "env" "tableBase" (global $gimport$19 i32))
+ (import "env" "abort" (func $fimport$3 (param i32)))
+ (import "env" "enlargeMemory" (func $fimport$4 (result i32)))
+ (import "env" "getTotalMemory" (func $fimport$5 (result i32)))
+ (import "env" "abortOnCannotGrowMemory" (func $fimport$6 (result i32)))
+ (import "env" "_pthread_cleanup_pop" (func $fimport$7 (param i32)))
+ (import "env" "___syscall6" (func $fimport$8 (param i32 i32) (result i32)))
+ (import "env" "_pthread_cleanup_push" (func $fimport$9 (param i32 i32)))
+ (import "env" "_abort" (func $fimport$10))
+ (import "env" "___setErrNo" (func $fimport$11 (param i32)))
+ (import "env" "_emscripten_memcpy_big" (func $fimport$12 (param i32 i32 i32) (result i32)))
+ (import "env" "___syscall54" (func $fimport$13 (param i32 i32) (result i32)))
+ (import "env" "___syscall140" (func $fimport$14 (param i32 i32) (result i32)))
+ (import "env" "___syscall146" (func $fimport$15 (param i32 i32) (result i32)))
+ (global $global$0 (mut i32) (global.get $gimport$0))
+ (global $global$1 (mut i32) (global.get $gimport$1))
+ (global $global$2 (mut i32) (global.get $gimport$2))
+ (global $global$3 (mut i32) (i32.const 0))
+ (global $global$4 (mut i32) (i32.const 0))
+ (global $global$5 (mut i32) (i32.const 0))
+ (export "_sbrk" (func $38))
+ (export "_free" (func $36))
+ (export "_main" (func $8))
+ (export "_pthread_self" (func $41))
+ (export "_memset" (func $39))
+ (export "_malloc" (func $35))
+ (export "_memcpy" (func $40))
+ (export "___errno_location" (func $12))
+ (export "runPostSets" (func $37))
+ (export "stackAlloc" (func $0))
+ (export "stackSave" (func $1))
+ (export "stackRestore" (func $2))
+ (export "establishStackSpace" (func $3))
+ (export "setThrew" (func $4))
+ (export "setTempRet0" (func $5))
+ (export "getTempRet0" (func $6))
+ (export "dynCall_ii" (func $42))
+ (export "dynCall_iiii" (func $43))
+ (export "dynCall_vi" (func $44))
+ (func $0 (; 13 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (local.get $0)
+    )
+   )
+   (global.set $global$1
+    (i32.and
+     (i32.add
+      (global.get $global$1)
+      (i32.const 15)
+     )
+     (i32.const -16)
+    )
+   )
+   (local.get $1)
+  )
+ )
+ (func $1 (; 14 ;) (type $3) (result i32)
+  (global.get $global$1)
+ )
+ (func $2 (; 15 ;) (type $2) (param $0 i32)
+  (global.set $global$1
+   (local.get $0)
+  )
+ )
+ (func $3 (; 16 ;) (type $5) (param $0 i32) (param $1 i32)
+  (block $label$1
+   (global.set $global$1
+    (local.get $0)
+   )
+   (global.set $global$2
+    (local.get $1)
+   )
+  )
+ )
+ (func $4 (; 17 ;) (type $5) (param $0 i32) (param $1 i32)
+  (if
+   (i32.eqz
+    (global.get $global$3)
+   )
+   (block
+    (global.set $global$3
+     (local.get $0)
+    )
+    (global.set $global$4
+     (local.get $1)
+    )
+   )
+  )
+ )
+ (func $5 (; 18 ;) (type $2) (param $0 i32)
+  (global.set $global$5
+   (local.get $0)
+  )
+ )
+ (func $6 (; 19 ;) (type $3) (result i32)
+  (global.get $global$5)
+ )
+ (func $7 (; 20 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (local $16 i32)
+  (local $17 i32)
+  (block $label$1 (result i32)
+   (local.set $3
+    (call $35
+     (local.tee $15
+      (i32.shl
+       (local.tee $4
+        (i32.load offset=4
+         (local.get $0)
+        )
+       )
+       (i32.const 2)
+      )
+     )
+    )
+   )
+   (local.set $6
+    (call $35
+     (local.get $15)
+    )
+   )
+   (local.set $10
+    (call $35
+     (local.get $15)
+    )
+   )
+   (if
+    (local.tee $2
+     (i32.gt_s
+      (local.get $4)
+      (i32.const 0)
+     )
+    )
+    (block
+     (local.set $1
+      (i32.const 0)
+     )
+     (loop $label$3
+      (i32.store
+       (i32.add
+        (local.get $3)
+        (i32.shl
+         (local.get $1)
+         (i32.const 2)
+        )
+       )
+       (local.get $1)
+      )
+      (br_if $label$3
+       (i32.ne
+        (local.tee $1
+         (i32.add
+          (local.get $1)
+          (i32.const 1)
+         )
+        )
+        (local.get $4)
+       )
+      )
+     )
+     (i32.store
+      (i32.add
+       (local.get $3)
+       (i32.shl
+        (local.tee $0
+         (i32.load
+          (local.get $0)
+         )
+        )
+        (i32.const 2)
+       )
+      )
+      (local.tee $11
+       (i32.add
+        (local.get $4)
+        (i32.const -1)
+       )
+      )
+     )
+     (i32.store
+      (local.tee $14
+       (i32.add
+        (local.get $3)
+        (i32.shl
+         (local.get $11)
+         (i32.const 2)
+        )
+       )
+      )
+      (local.get $0)
+     )
+     (if
+      (local.get $2)
+      (block
+       (local.set $0
+        (i32.const 0)
+       )
+       (local.set $1
+        (local.get $4)
+       )
+       (loop $label$5
+        (block $label$6
+         (if
+          (i32.gt_s
+           (local.get $1)
+           (i32.const 1)
+          )
+          (loop $label$8
+           (i32.store
+            (i32.add
+             (local.get $10)
+             (i32.shl
+              (local.tee $2
+               (i32.add
+                (local.get $1)
+                (i32.const -1)
+               )
+              )
+              (i32.const 2)
+             )
+            )
+            (local.get $1)
+           )
+           (if
+            (i32.gt_s
+             (local.get $2)
+             (i32.const 1)
+            )
+            (block
+             (local.set $1
+              (local.get $2)
+             )
+             (br $label$8)
+            )
+            (local.set $2
+             (i32.const 1)
+            )
+           )
+          )
+          (local.set $2
+           (local.get $1)
+          )
+         )
+         (if
+          (local.tee $7
+           (i32.load
+            (local.get $3)
+           )
+          )
+          (if
+           (i32.ne
+            (i32.load
+             (local.get $14)
+            )
+            (local.get $11)
+           )
+           (block
+            (drop
+             (call $40
+              (local.get $6)
+              (local.get $3)
+              (local.get $15)
+             )
+            )
+            (local.set $8
+             (i32.const 0)
+            )
+            (local.set $9
+             (i32.load
+              (local.get $6)
+             )
+            )
+            (loop $label$14
+             (if
+              (i32.gt_s
+               (local.tee $1
+                (i32.add
+                 (local.get $9)
+                 (i32.const -1)
+                )
+               )
+               (i32.const 1)
+              )
+              (block
+               (local.set $5
+                (i32.const 1)
+               )
+               (loop $label$16
+                (local.set $17
+                 (i32.load
+                  (local.tee $12
+                   (i32.add
+                    (local.get $6)
+                    (i32.shl
+                     (local.get $5)
+                     (i32.const 2)
+                    )
+                   )
+                  )
+                 )
+                )
+                (i32.store
+                 (local.get $12)
+                 (i32.load
+                  (local.tee $12
+                   (i32.add
+                    (local.get $6)
+                    (i32.shl
+                     (local.get $1)
+                     (i32.const 2)
+                    )
+                   )
+                  )
+                 )
+                )
+                (i32.store
+                 (local.get $12)
+                 (local.get $17)
+                )
+                (br_if $label$16
+                 (i32.lt_s
+                  (local.tee $5
+                   (i32.add
+                    (local.get $5)
+                    (i32.const 1)
+                   )
+                  )
+                  (local.tee $1
+                   (i32.add
+                    (local.get $1)
+                    (i32.const -1)
+                   )
+                  )
+                 )
+                )
+               )
+              )
+             )
+             (local.set $5
+              (i32.add
+               (local.get $8)
+               (i32.const 1)
+              )
+             )
+             (local.set $1
+              (i32.load
+               (local.tee $12
+                (i32.add
+                 (local.get $6)
+                 (i32.shl
+                  (local.get $9)
+                  (i32.const 2)
+                 )
+                )
+               )
+              )
+             )
+             (i32.store
+              (local.get $12)
+              (local.get $9)
+             )
+             (if
+              (local.get $1)
+              (block
+               (local.set $8
+                (local.get $5)
+               )
+               (local.set $9
+                (local.get $1)
+               )
+               (br $label$14)
+              )
+             )
+            )
+            (if
+             (i32.le_s
+              (local.get $0)
+              (local.get $8)
+             )
+             (local.set $0
+              (local.get $5)
+             )
+            )
+           )
+          )
+         )
+         (if
+          (i32.lt_s
+           (local.get $2)
+           (local.get $11)
+          )
+          (local.set $1
+           (local.get $2)
+          )
+          (block
+           (local.set $1
+            (i32.const 31)
+           )
+           (br $label$6)
+          )
+         )
+         (loop $label$21
+          (if
+           (i32.gt_s
+            (local.get $1)
+            (i32.const 0)
+           )
+           (block
+            (local.set $2
+             (i32.const 0)
+            )
+            (loop $label$23
+             (i32.store
+              (i32.add
+               (local.get $3)
+               (i32.shl
+                (local.get $2)
+                (i32.const 2)
+               )
+              )
+              (i32.load
+               (i32.add
+                (local.get $3)
+                (i32.shl
+                 (local.tee $2
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 1)
+                  )
+                 )
+                 (i32.const 2)
+                )
+               )
+              )
+             )
+             (br_if $label$23
+              (i32.lt_s
+               (local.get $2)
+               (local.get $1)
+              )
+             )
+             (local.set $2
+              (local.get $1)
+             )
+            )
+           )
+           (local.set $2
+            (i32.const 0)
+           )
+          )
+          (i32.store
+           (i32.add
+            (local.get $3)
+            (i32.shl
+             (local.get $2)
+             (i32.const 2)
+            )
+           )
+           (local.get $7)
+          )
+          (local.set $5
+           (i32.load
+            (local.tee $2
+             (i32.add
+              (local.get $10)
+              (i32.shl
+               (local.get $1)
+               (i32.const 2)
+              )
+             )
+            )
+           )
+          )
+          (i32.store
+           (local.get $2)
+           (i32.add
+            (local.get $5)
+            (i32.const -1)
+           )
+          )
+          (br_if $label$5
+           (i32.gt_s
+            (local.get $5)
+            (i32.const 1)
+           )
+          )
+          (if
+           (i32.lt_s
+            (local.tee $1
+             (i32.add
+              (local.get $1)
+              (i32.const 1)
+             )
+            )
+            (local.get $11)
+           )
+           (block
+            (local.set $7
+             (i32.load
+              (local.get $3)
+             )
+            )
+            (br $label$21)
+           )
+           (block
+            (local.set $1
+             (i32.const 31)
+            )
+            (br $label$6)
+           )
+          )
+         )
+        )
+       )
+       (if
+        (i32.eq
+         (local.get $1)
+         (i32.const 31)
+        )
+        (block
+         (call $36
+          (local.get $3)
+         )
+         (call $36
+          (local.get $6)
+         )
+         (call $36
+          (local.get $10)
+         )
+         (return
+          (local.get $0)
+         )
+        )
+       )
+      )
+      (block
+       (local.set $16
+        (local.get $14)
+       )
+       (local.set $13
+        (local.get $11)
+       )
+      )
+     )
+    )
+    (block
+     (i32.store
+      (i32.add
+       (local.get $3)
+       (i32.shl
+        (local.tee $0
+         (i32.load
+          (local.get $0)
+         )
+        )
+        (i32.const 2)
+       )
+      )
+      (local.tee $13
+       (i32.add
+        (local.get $4)
+        (i32.const -1)
+       )
+      )
+     )
+     (i32.store
+      (local.tee $16
+       (i32.add
+        (local.get $3)
+        (i32.shl
+         (local.get $13)
+         (i32.const 2)
+        )
+       )
+      )
+      (local.get $0)
+     )
+    )
+   )
+   (local.set $0
+    (i32.const 0)
+   )
+   (local.set $1
+    (local.get $4)
+   )
+   (loop $label$30
+    (block $label$31
+     (if
+      (i32.gt_s
+       (local.get $1)
+       (i32.const 1)
+      )
+      (loop $label$33
+       (i32.store
+        (i32.add
+         (local.get $10)
+         (i32.shl
+          (local.tee $2
+           (i32.add
+            (local.get $1)
+            (i32.const -1)
+           )
+          )
+          (i32.const 2)
+         )
+        )
+        (local.get $1)
+       )
+       (if
+        (i32.gt_s
+         (local.get $2)
+         (i32.const 1)
+        )
+        (block
+         (local.set $1
+          (local.get $2)
+         )
+         (br $label$33)
+        )
+        (local.set $2
+         (i32.const 1)
+        )
+       )
+      )
+      (local.set $2
+       (local.get $1)
+      )
+     )
+     (if
+      (local.tee $9
+       (i32.load
+        (local.get $3)
+       )
+      )
+      (if
+       (i32.ne
+        (i32.load
+         (local.get $16)
+        )
+        (local.get $13)
+       )
+       (block
+        (local.set $5
+         (i32.const 0)
+        )
+        (local.set $8
+         (i32.load
+          (local.get $6)
+         )
+        )
+        (loop $label$39
+         (if
+          (i32.gt_s
+           (local.tee $1
+            (i32.add
+             (local.get $8)
+             (i32.const -1)
+            )
+           )
+           (i32.const 1)
+          )
+          (block
+           (local.set $4
+            (i32.const 1)
+           )
+           (loop $label$41
+            (local.set $14
+             (i32.load
+              (local.tee $7
+               (i32.add
+                (local.get $6)
+                (i32.shl
+                 (local.get $4)
+                 (i32.const 2)
+                )
+               )
+              )
+             )
+            )
+            (i32.store
+             (local.get $7)
+             (i32.load
+              (local.tee $7
+               (i32.add
+                (local.get $6)
+                (i32.shl
+                 (local.get $1)
+                 (i32.const 2)
+                )
+               )
+              )
+             )
+            )
+            (i32.store
+             (local.get $7)
+             (local.get $14)
+            )
+            (br_if $label$41
+             (i32.lt_s
+              (local.tee $4
+               (i32.add
+                (local.get $4)
+                (i32.const 1)
+               )
+              )
+              (local.tee $1
+               (i32.add
+                (local.get $1)
+                (i32.const -1)
+               )
+              )
+             )
+            )
+           )
+          )
+         )
+         (local.set $4
+          (i32.add
+           (local.get $5)
+           (i32.const 1)
+          )
+         )
+         (local.set $1
+          (i32.load
+           (local.tee $7
+            (i32.add
+             (local.get $6)
+             (i32.shl
+              (local.get $8)
+              (i32.const 2)
+             )
+            )
+           )
+          )
+         )
+         (i32.store
+          (local.get $7)
+          (local.get $8)
+         )
+         (if
+          (local.get $1)
+          (block
+           (local.set $5
+            (local.get $4)
+           )
+           (local.set $8
+            (local.get $1)
+           )
+           (br $label$39)
+          )
+         )
+        )
+        (if
+         (i32.le_s
+          (local.get $0)
+          (local.get $5)
+         )
+         (local.set $0
+          (local.get $4)
+         )
+        )
+       )
+      )
+     )
+     (if
+      (i32.lt_s
+       (local.get $2)
+       (local.get $13)
+      )
+      (local.set $1
+       (local.get $2)
+      )
+      (block
+       (local.set $1
+        (i32.const 31)
+       )
+       (br $label$31)
+      )
+     )
+     (loop $label$46
+      (if
+       (i32.gt_s
+        (local.get $1)
+        (i32.const 0)
+       )
+       (block
+        (local.set $2
+         (i32.const 0)
+        )
+        (loop $label$48
+         (i32.store
+          (i32.add
+           (local.get $3)
+           (i32.shl
+            (local.get $2)
+            (i32.const 2)
+           )
+          )
+          (i32.load
+           (i32.add
+            (local.get $3)
+            (i32.shl
+             (local.tee $2
+              (i32.add
+               (local.get $2)
+               (i32.const 1)
+              )
+             )
+             (i32.const 2)
+            )
+           )
+          )
+         )
+         (br_if $label$48
+          (i32.lt_s
+           (local.get $2)
+           (local.get $1)
+          )
+         )
+         (local.set $2
+          (local.get $1)
+         )
+        )
+       )
+       (local.set $2
+        (i32.const 0)
+       )
+      )
+      (i32.store
+       (i32.add
+        (local.get $3)
+        (i32.shl
+         (local.get $2)
+         (i32.const 2)
+        )
+       )
+       (local.get $9)
+      )
+      (local.set $4
+       (i32.load
+        (local.tee $2
+         (i32.add
+          (local.get $10)
+          (i32.shl
+           (local.get $1)
+           (i32.const 2)
+          )
+         )
+        )
+       )
+      )
+      (i32.store
+       (local.get $2)
+       (i32.add
+        (local.get $4)
+        (i32.const -1)
+       )
+      )
+      (br_if $label$30
+       (i32.gt_s
+        (local.get $4)
+        (i32.const 1)
+       )
+      )
+      (if
+       (i32.lt_s
+        (local.tee $1
+         (i32.add
+          (local.get $1)
+          (i32.const 1)
+         )
+        )
+        (local.get $13)
+       )
+       (block
+        (local.set $9
+         (i32.load
+          (local.get $3)
+         )
+        )
+        (br $label$46)
+       )
+       (block
+        (local.set $1
+         (i32.const 31)
+        )
+        (br $label$31)
+       )
+      )
+     )
+    )
+   )
+   (if
+    (i32.eq
+     (local.get $1)
+     (i32.const 31)
+    )
+    (block
+     (call $36
+      (local.get $3)
+     )
+     (call $36
+      (local.get $6)
+     )
+     (call $36
+      (local.get $10)
+     )
+     (return
+      (local.get $0)
+     )
+    )
+   )
+   (i32.const 0)
+  )
+ )
+ (func $8 (; 21 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (block $label$1 (result i32)
+   (local.set $5
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 32)
+    )
+   )
+   (local.set $7
+    (i32.add
+     (local.get $5)
+     (i32.const 16)
+    )
+   )
+   (local.set $10
+    (i32.add
+     (local.get $5)
+     (i32.const 8)
+    )
+   )
+   (local.set $2
+    (local.get $5)
+   )
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (i32.le_s
+       (local.get $0)
+       (i32.const 1)
+      )
+     )
+     (block $label$4
+      (block $label$5
+       (block $label$6
+        (block $label$7
+         (block $label$8
+          (block $label$9
+           (block $label$10
+            (br_table $label$5 $label$10 $label$8 $label$9 $label$7 $label$6 $label$4
+             (i32.sub
+              (local.tee $0
+               (i32.load8_s
+                (i32.load offset=4
+                 (local.get $1)
+                )
+               )
+              )
+              (i32.const 48)
+             )
+            )
+           )
+           (local.set $3
+            (i32.const 9)
+           )
+           (br $label$2)
+          )
+          (br $label$3)
+         )
+         (local.set $3
+          (i32.const 10)
+         )
+         (br $label$2)
+        )
+        (local.set $3
+         (i32.const 11)
+        )
+        (br $label$2)
+       )
+       (local.set $3
+        (i32.const 12)
+       )
+       (br $label$2)
+      )
+      (global.set $global$1
+       (local.get $5)
+      )
+      (return
+       (i32.const 0)
+      )
+     )
+     (i32.store
+      (local.get $2)
+      (i32.add
+       (local.get $0)
+       (i32.const -48)
+      )
+     )
+     (drop
+      (call $33
+       (i32.const 1140)
+       (local.get $2)
+      )
+     )
+     (global.set $global$1
+      (local.get $5)
+     )
+     (return
+      (i32.const -1)
+     )
+    )
+    (local.set $3
+     (i32.const 11)
+    )
+   )
+   (local.set $6
+    (i32.add
+     (local.get $3)
+     (i32.const -1)
+    )
+   )
+   (local.set $2
+    (i32.const 0)
+   )
+   (local.set $0
+    (i32.const 0)
+   )
+   (loop $label$11
+    (i32.store
+     (local.tee $1
+      (call $35
+       (i32.const 12)
+      )
+     )
+     (local.get $0)
+    )
+    (i32.store offset=4
+     (local.get $1)
+     (local.get $3)
+    )
+    (i32.store offset=8
+     (local.get $1)
+     (local.get $2)
+    )
+    (if
+     (i32.ne
+      (local.tee $0
+       (i32.add
+        (local.get $0)
+        (i32.const 1)
+       )
+      )
+      (local.get $6)
+     )
+     (block
+      (local.set $2
+       (local.get $1)
+      )
+      (br $label$11)
+     )
+    )
+   )
+   (local.set $4
+    (call $35
+     (local.tee $0
+      (i32.shl
+       (local.get $3)
+       (i32.const 2)
+      )
+     )
+    )
+   )
+   (local.set $8
+    (call $35
+     (local.get $0)
+    )
+   )
+   (local.set $0
+    (i32.const 0)
+   )
+   (loop $label$13
+    (i32.store
+     (i32.add
+      (local.get $4)
+      (i32.shl
+       (local.get $0)
+       (i32.const 2)
+      )
+     )
+     (local.get $0)
+    )
+    (br_if $label$13
+     (i32.ne
+      (local.tee $0
+       (i32.add
+        (local.get $0)
+        (i32.const 1)
+       )
+      )
+      (local.get $3)
+     )
+    )
+   )
+   (local.set $0
+    (local.get $3)
+   )
+   (local.set $6
+    (i32.const 30)
+   )
+   (loop $label$14
+    (block $label$15
+     (local.set $2
+      (i32.const 0)
+     )
+     (loop $label$16
+      (i32.store
+       (local.get $10)
+       (i32.add
+        (i32.load
+         (i32.add
+          (local.get $4)
+          (i32.shl
+           (local.get $2)
+           (i32.const 2)
+          )
+         )
+        )
+        (i32.const 1)
+       )
+      )
+      (drop
+       (call $33
+        (i32.const 1174)
+        (local.get $10)
+       )
+      )
+      (br_if $label$16
+       (i32.ne
+        (local.tee $2
+         (i32.add
+          (local.get $2)
+          (i32.const 1)
+         )
+        )
+        (local.get $3)
+       )
+      )
+     )
+     (drop
+      (call $34
+       (i32.const 10)
+      )
+     )
+     (if
+      (i32.gt_s
+       (local.get $0)
+       (i32.const 1)
+      )
+      (loop $label$18
+       (i32.store
+        (i32.add
+         (local.get $8)
+         (i32.shl
+          (local.tee $2
+           (i32.add
+            (local.get $0)
+            (i32.const -1)
+           )
+          )
+          (i32.const 2)
+         )
+        )
+        (local.get $0)
+       )
+       (if
+        (i32.gt_s
+         (local.get $2)
+         (i32.const 1)
+        )
+        (block
+         (local.set $0
+          (local.get $2)
+         )
+         (br $label$18)
+        )
+        (local.set $0
+         (i32.const 1)
+        )
+       )
+      )
+      (br_if $label$15
+       (i32.eq
+        (local.get $0)
+        (local.get $3)
+       )
+      )
+     )
+     (local.set $6
+      (i32.add
+       (local.get $6)
+       (i32.const -1)
+      )
+     )
+     (loop $label$22
+      (block $label$23
+       (local.set $9
+        (i32.load
+         (local.get $4)
+        )
+       )
+       (if
+        (i32.gt_s
+         (local.get $0)
+         (i32.const 0)
+        )
+        (block
+         (local.set $2
+          (i32.const 0)
+         )
+         (loop $label$25
+          (i32.store
+           (i32.add
+            (local.get $4)
+            (i32.shl
+             (local.get $2)
+             (i32.const 2)
+            )
+           )
+           (i32.load
+            (i32.add
+             (local.get $4)
+             (i32.shl
+              (local.tee $2
+               (i32.add
+                (local.get $2)
+                (i32.const 1)
+               )
+              )
+              (i32.const 2)
+             )
+            )
+           )
+          )
+          (br_if $label$25
+           (i32.lt_s
+            (local.get $2)
+            (local.get $0)
+           )
+          )
+          (local.set $2
+           (local.get $0)
+          )
+         )
+        )
+        (local.set $2
+         (i32.const 0)
+        )
+       )
+       (i32.store
+        (i32.add
+         (local.get $4)
+         (i32.shl
+          (local.get $2)
+          (i32.const 2)
+         )
+        )
+        (local.get $9)
+       )
+       (local.set $2
+        (i32.load
+         (local.tee $9
+          (i32.add
+           (local.get $8)
+           (i32.shl
+            (local.get $0)
+            (i32.const 2)
+           )
+          )
+         )
+        )
+       )
+       (i32.store
+        (local.get $9)
+        (i32.add
+         (local.get $2)
+         (i32.const -1)
+        )
+       )
+       (br_if $label$23
+        (i32.gt_s
+         (local.get $2)
+         (i32.const 1)
+        )
+       )
+       (br_if $label$22
+        (i32.ne
+         (local.tee $0
+          (i32.add
+           (local.get $0)
+           (i32.const 1)
+          )
+         )
+         (local.get $3)
+        )
+       )
+       (br $label$15)
+      )
+     )
+     (br_if $label$14
+      (local.get $6)
+     )
+    )
+   )
+   (call $36
+    (local.get $4)
+   )
+   (call $36
+    (local.get $8)
+   )
+   (if
+    (local.get $1)
+    (block
+     (local.set $0
+      (i32.const 0)
+     )
+     (loop $label$28
+      (if
+       (i32.lt_s
+        (local.get $0)
+        (local.tee $2
+         (call $7
+          (local.get $1)
+         )
+        )
+       )
+       (local.set $0
+        (local.get $2)
+       )
+      )
+      (local.set $2
+       (i32.load offset=8
+        (local.get $1)
+       )
+      )
+      (call $36
+       (local.get $1)
+      )
+      (if
+       (local.get $2)
+       (block
+        (local.set $1
+         (local.get $2)
+        )
+        (br $label$28)
+       )
+      )
+     )
+    )
+    (local.set $0
+     (i32.const 0)
+    )
+   )
+   (i32.store
+    (local.get $7)
+    (local.get $3)
+   )
+   (i32.store offset=4
+    (local.get $7)
+    (local.get $0)
+   )
+   (drop
+    (call $33
+     (i32.const 1151)
+     (local.get $7)
+    )
+   )
+   (global.set $global$1
+    (local.get $5)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $9 (; 22 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (i32.store
+    (local.tee $2
+     (local.get $1)
+    )
+    (i32.load offset=60
+     (local.get $0)
+    )
+   )
+   (local.set $0
+    (call $11
+     (call $fimport$8
+      (i32.const 6)
+      (local.get $2)
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $1)
+   )
+   (local.get $0)
+  )
+ )
+ (func $10 (; 23 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 32)
+    )
+   )
+   (i32.store
+    (local.tee $3
+     (local.get $4)
+    )
+    (i32.load offset=60
+     (local.get $0)
+    )
+   )
+   (i32.store offset=4
+    (local.get $3)
+    (i32.const 0)
+   )
+   (i32.store offset=8
+    (local.get $3)
+    (local.get $1)
+   )
+   (i32.store offset=12
+    (local.get $3)
+    (local.tee $0
+     (i32.add
+      (local.get $4)
+      (i32.const 20)
+     )
+    )
+   )
+   (i32.store offset=16
+    (local.get $3)
+    (local.get $2)
+   )
+   (local.set $0
+    (if (result i32)
+     (i32.lt_s
+      (call $11
+       (call $fimport$14
+        (i32.const 140)
+        (local.get $3)
+       )
+      )
+      (i32.const 0)
+     )
+     (block (result i32)
+      (i32.store
+       (local.get $0)
+       (i32.const -1)
+      )
+      (i32.const -1)
+     )
+     (i32.load
+      (local.get $0)
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $4)
+   )
+   (local.get $0)
+  )
+ )
+ (func $11 (; 24 ;) (type $1) (param $0 i32) (result i32)
+  (if (result i32)
+   (i32.gt_u
+    (local.get $0)
+    (i32.const -4096)
+   )
+   (block (result i32)
+    (i32.store
+     (call $12)
+     (i32.sub
+      (i32.const 0)
+      (local.get $0)
+     )
+    )
+    (i32.const -1)
+   )
+   (local.get $0)
+  )
+ )
+ (func $12 (; 25 ;) (type $3) (result i32)
+  (i32.const 3648)
+ )
+ (func $13 (; 26 ;) (type $2) (param $0 i32)
+  (nop)
+ )
+ (func $14 (; 27 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 80)
+    )
+   )
+   (local.set $3
+    (local.get $4)
+   )
+   (local.set $5
+    (i32.add
+     (local.get $4)
+     (i32.const 12)
+    )
+   )
+   (i32.store offset=36
+    (local.get $0)
+    (i32.const 3)
+   )
+   (if
+    (i32.eqz
+     (i32.and
+      (i32.load
+       (local.get $0)
+      )
+      (i32.const 64)
+     )
+    )
+    (block
+     (i32.store
+      (local.get $3)
+      (i32.load offset=60
+       (local.get $0)
+      )
+     )
+     (i32.store offset=4
+      (local.get $3)
+      (i32.const 21505)
+     )
+     (i32.store offset=8
+      (local.get $3)
+      (local.get $5)
+     )
+     (if
+      (call $fimport$13
+       (i32.const 54)
+       (local.get $3)
+      )
+      (i32.store8 offset=75
+       (local.get $0)
+       (i32.const -1)
+      )
+     )
+    )
+   )
+   (local.set $0
+    (call $15
+     (local.get $0)
+     (local.get $1)
+     (local.get $2)
+    )
+   )
+   (global.set $global$1
+    (local.get $4)
+   )
+   (local.get $0)
+  )
+ )
+ (func $15 (; 28 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (block $label$1 (result i32)
+   (local.set $8
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 48)
+    )
+   )
+   (local.set $9
+    (i32.add
+     (local.get $8)
+     (i32.const 16)
+    )
+   )
+   (local.set $10
+    (local.get $8)
+   )
+   (i32.store
+    (local.tee $3
+     (i32.add
+      (local.get $8)
+      (i32.const 32)
+     )
+    )
+    (local.tee $4
+     (i32.load
+      (local.tee $6
+       (i32.add
+        (local.get $0)
+        (i32.const 28)
+       )
+      )
+     )
+    )
+   )
+   (i32.store offset=4
+    (local.get $3)
+    (local.tee $5
+     (i32.sub
+      (i32.load
+       (local.tee $11
+        (i32.add
+         (local.get $0)
+         (i32.const 20)
+        )
+       )
+      )
+      (local.get $4)
+     )
+    )
+   )
+   (i32.store offset=8
+    (local.get $3)
+    (local.get $1)
+   )
+   (i32.store offset=12
+    (local.get $3)
+    (local.get $2)
+   )
+   (local.set $13
+    (i32.add
+     (local.get $0)
+     (i32.const 60)
+    )
+   )
+   (local.set $14
+    (i32.add
+     (local.get $0)
+     (i32.const 44)
+    )
+   )
+   (local.set $1
+    (local.get $3)
+   )
+   (local.set $4
+    (i32.const 2)
+   )
+   (local.set $12
+    (i32.add
+     (local.get $5)
+     (local.get $2)
+    )
+   )
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (loop $label$5
+       (if
+        (i32.load
+         (i32.const 3604)
+        )
+        (block
+         (call $fimport$9
+          (i32.const 1)
+          (local.get $0)
+         )
+         (i32.store
+          (local.get $10)
+          (i32.load
+           (local.get $13)
+          )
+         )
+         (i32.store offset=4
+          (local.get $10)
+          (local.get $1)
+         )
+         (i32.store offset=8
+          (local.get $10)
+          (local.get $4)
+         )
+         (local.set $3
+          (call $11
+           (call $fimport$15
+            (i32.const 146)
+            (local.get $10)
+           )
+          )
+         )
+         (call $fimport$7
+          (i32.const 0)
+         )
+        )
+        (block
+         (i32.store
+          (local.get $9)
+          (i32.load
+           (local.get $13)
+          )
+         )
+         (i32.store offset=4
+          (local.get $9)
+          (local.get $1)
+         )
+         (i32.store offset=8
+          (local.get $9)
+          (local.get $4)
+         )
+         (local.set $3
+          (call $11
+           (call $fimport$15
+            (i32.const 146)
+            (local.get $9)
+           )
+          )
+         )
+        )
+       )
+       (br_if $label$4
+        (i32.eq
+         (local.get $12)
+         (local.get $3)
+        )
+       )
+       (br_if $label$3
+        (i32.lt_s
+         (local.get $3)
+         (i32.const 0)
+        )
+       )
+       (local.set $5
+        (if (result i32)
+         (i32.gt_u
+          (local.get $3)
+          (local.tee $5
+           (i32.load offset=4
+            (local.get $1)
+           )
+          )
+         )
+         (block (result i32)
+          (i32.store
+           (local.get $6)
+           (local.tee $7
+            (i32.load
+             (local.get $14)
+            )
+           )
+          )
+          (i32.store
+           (local.get $11)
+           (local.get $7)
+          )
+          (local.set $7
+           (i32.load offset=12
+            (local.get $1)
+           )
+          )
+          (local.set $1
+           (i32.add
+            (local.get $1)
+            (i32.const 8)
+           )
+          )
+          (local.set $4
+           (i32.add
+            (local.get $4)
+            (i32.const -1)
+           )
+          )
+          (i32.sub
+           (local.get $3)
+           (local.get $5)
+          )
+         )
+         (if (result i32)
+          (i32.eq
+           (local.get $4)
+           (i32.const 2)
+          )
+          (block (result i32)
+           (i32.store
+            (local.get $6)
+            (i32.add
+             (i32.load
+              (local.get $6)
+             )
+             (local.get $3)
+            )
+           )
+           (local.set $7
+            (local.get $5)
+           )
+           (local.set $4
+            (i32.const 2)
+           )
+           (local.get $3)
+          )
+          (block (result i32)
+           (local.set $7
+            (local.get $5)
+           )
+           (local.get $3)
+          )
+         )
+        )
+       )
+       (i32.store
+        (local.get $1)
+        (i32.add
+         (i32.load
+          (local.get $1)
+         )
+         (local.get $5)
+        )
+       )
+       (i32.store offset=4
+        (local.get $1)
+        (i32.sub
+         (local.get $7)
+         (local.get $5)
+        )
+       )
+       (local.set $12
+        (i32.sub
+         (local.get $12)
+         (local.get $3)
+        )
+       )
+       (br $label$5)
+      )
+     )
+     (i32.store offset=16
+      (local.get $0)
+      (i32.add
+       (local.tee $1
+        (i32.load
+         (local.get $14)
+        )
+       )
+       (i32.load offset=48
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store
+      (local.get $6)
+      (local.get $1)
+     )
+     (i32.store
+      (local.get $11)
+      (local.get $1)
+     )
+     (br $label$2)
+    )
+    (i32.store offset=16
+     (local.get $0)
+     (i32.const 0)
+    )
+    (i32.store
+     (local.get $6)
+     (i32.const 0)
+    )
+    (i32.store
+     (local.get $11)
+     (i32.const 0)
+    )
+    (i32.store
+     (local.get $0)
+     (i32.or
+      (i32.load
+       (local.get $0)
+      )
+      (i32.const 32)
+     )
+    )
+    (local.set $2
+     (if (result i32)
+      (i32.eq
+       (local.get $4)
+       (i32.const 2)
+      )
+      (i32.const 0)
+      (i32.sub
+       (local.get $2)
+       (i32.load offset=4
+        (local.get $1)
+       )
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $8)
+   )
+   (local.get $2)
+  )
+ )
+ (func $16 (; 29 ;) (type $2) (param $0 i32)
+  (if
+   (i32.eqz
+    (i32.load offset=68
+     (local.get $0)
+    )
+   )
+   (call $13
+    (local.get $0)
+   )
+  )
+ )
+ (func $17 (; 30 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $5
+    (i32.and
+     (local.get $1)
+     (i32.const 255)
+    )
+   )
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (if
+       (i32.and
+        (local.tee $4
+         (i32.ne
+          (local.get $2)
+          (i32.const 0)
+         )
+        )
+        (i32.ne
+         (i32.and
+          (local.get $0)
+          (i32.const 3)
+         )
+         (i32.const 0)
+        )
+       )
+       (block
+        (local.set $4
+         (i32.and
+          (local.get $1)
+          (i32.const 255)
+         )
+        )
+        (local.set $3
+         (local.get $2)
+        )
+        (local.set $2
+         (local.get $0)
+        )
+        (loop $label$6
+         (if
+          (i32.eq
+           (i32.load8_s
+            (local.get $2)
+           )
+           (i32.shr_s
+            (i32.shl
+             (local.get $4)
+             (i32.const 24)
+            )
+            (i32.const 24)
+           )
+          )
+          (block
+           (local.set $0
+            (local.get $3)
+           )
+           (br $label$3)
+          )
+         )
+         (br_if $label$6
+          (i32.and
+           (local.tee $0
+            (i32.ne
+             (local.tee $3
+              (i32.add
+               (local.get $3)
+               (i32.const -1)
+              )
+             )
+             (i32.const 0)
+            )
+           )
+           (i32.ne
+            (i32.and
+             (local.tee $2
+              (i32.add
+               (local.get $2)
+               (i32.const 1)
+              )
+             )
+             (i32.const 3)
+            )
+            (i32.const 0)
+           )
+          )
+         )
+         (br $label$4)
+        )
+       )
+       (block
+        (local.set $3
+         (local.get $2)
+        )
+        (local.set $2
+         (local.get $0)
+        )
+        (local.set $0
+         (local.get $4)
+        )
+       )
+      )
+     )
+     (if
+      (local.get $0)
+      (block
+       (local.set $0
+        (local.get $3)
+       )
+       (br $label$3)
+      )
+      (local.set $0
+       (i32.const 0)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.ne
+      (i32.load8_s
+       (local.get $2)
+      )
+      (i32.shr_s
+       (i32.shl
+        (local.tee $1
+         (i32.and
+          (local.get $1)
+          (i32.const 255)
+         )
+        )
+        (i32.const 24)
+       )
+       (i32.const 24)
+      )
+     )
+     (block
+      (local.set $3
+       (i32.mul
+        (local.get $5)
+        (i32.const 16843009)
+       )
+      )
+      (block $label$12
+       (block $label$13
+        (br_if $label$13
+         (i32.le_u
+          (local.get $0)
+          (i32.const 3)
+         )
+        )
+        (loop $label$14
+         (if
+          (i32.eqz
+           (i32.and
+            (i32.xor
+             (i32.and
+              (local.tee $4
+               (i32.xor
+                (i32.load
+                 (local.get $2)
+                )
+                (local.get $3)
+               )
+              )
+              (i32.const -2139062144)
+             )
+             (i32.const -2139062144)
+            )
+            (i32.add
+             (local.get $4)
+             (i32.const -16843009)
+            )
+           )
+          )
+          (block
+           (local.set $2
+            (i32.add
+             (local.get $2)
+             (i32.const 4)
+            )
+           )
+           (br_if $label$14
+            (i32.gt_u
+             (local.tee $0
+              (i32.add
+               (local.get $0)
+               (i32.const -4)
+              )
+             )
+             (i32.const 3)
+            )
+           )
+           (br $label$13)
+          )
+         )
+        )
+        (br $label$12)
+       )
+       (if
+        (i32.eqz
+         (local.get $0)
+        )
+        (block
+         (local.set $0
+          (i32.const 0)
+         )
+         (br $label$2)
+        )
+       )
+      )
+      (loop $label$17
+       (br_if $label$2
+        (i32.eq
+         (i32.load8_s
+          (local.get $2)
+         )
+         (i32.shr_s
+          (i32.shl
+           (local.get $1)
+           (i32.const 24)
+          )
+          (i32.const 24)
+         )
+        )
+       )
+       (local.set $2
+        (i32.add
+         (local.get $2)
+         (i32.const 1)
+        )
+       )
+       (br_if $label$17
+        (local.tee $0
+         (i32.add
+          (local.get $0)
+          (i32.const -1)
+         )
+        )
+       )
+       (local.set $0
+        (i32.const 0)
+       )
+      )
+     )
+    )
+   )
+   (if (result i32)
+    (local.get $0)
+    (local.get $2)
+    (i32.const 0)
+   )
+  )
+ )
+ (func $18 (; 31 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 224)
+    )
+   )
+   (local.set $5
+    (i32.add
+     (local.get $4)
+     (i32.const 136)
+    )
+   )
+   (i64.store align=4
+    (local.tee $3
+     (i32.add
+      (local.get $4)
+      (i32.const 80)
+     )
+    )
+    (i64.const 0)
+   )
+   (i64.store offset=8 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i64.store offset=16 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i64.store offset=24 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i64.store offset=32 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i32.store
+    (local.tee $6
+     (i32.add
+      (local.get $4)
+      (i32.const 120)
+     )
+    )
+    (i32.load
+     (local.get $2)
+    )
+   )
+   (if
+    (i32.lt_s
+     (call $19
+      (i32.const 0)
+      (local.get $1)
+      (local.get $6)
+      (local.tee $2
+       (local.get $4)
+      )
+      (local.get $3)
+     )
+     (i32.const 0)
+    )
+    (local.set $1
+     (i32.const -1)
+    )
+    (block
+     (local.set $12
+      (if (result i32)
+       (i32.gt_s
+        (i32.load offset=76
+         (local.get $0)
+        )
+        (i32.const -1)
+       )
+       (call $20
+        (local.get $0)
+       )
+       (i32.const 0)
+      )
+     )
+     (local.set $7
+      (i32.load
+       (local.get $0)
+      )
+     )
+     (if
+      (i32.lt_s
+       (i32.load8_s offset=74
+        (local.get $0)
+       )
+       (i32.const 1)
+      )
+      (i32.store
+       (local.get $0)
+       (i32.and
+        (local.get $7)
+        (i32.const -33)
+       )
+      )
+     )
+     (if
+      (i32.load
+       (local.tee $8
+        (i32.add
+         (local.get $0)
+         (i32.const 48)
+        )
+       )
+      )
+      (local.set $1
+       (call $19
+        (local.get $0)
+        (local.get $1)
+        (local.get $6)
+        (local.get $2)
+        (local.get $3)
+       )
+      )
+      (block
+       (local.set $10
+        (i32.load
+         (local.tee $9
+          (i32.add
+           (local.get $0)
+           (i32.const 44)
+          )
+         )
+        )
+       )
+       (i32.store
+        (local.get $9)
+        (local.get $5)
+       )
+       (i32.store
+        (local.tee $13
+         (i32.add
+          (local.get $0)
+          (i32.const 28)
+         )
+        )
+        (local.get $5)
+       )
+       (i32.store
+        (local.tee $11
+         (i32.add
+          (local.get $0)
+          (i32.const 20)
+         )
+        )
+        (local.get $5)
+       )
+       (i32.store
+        (local.get $8)
+        (i32.const 80)
+       )
+       (i32.store
+        (local.tee $14
+         (i32.add
+          (local.get $0)
+          (i32.const 16)
+         )
+        )
+        (i32.add
+         (local.get $5)
+         (i32.const 80)
+        )
+       )
+       (local.set $1
+        (call $19
+         (local.get $0)
+         (local.get $1)
+         (local.get $6)
+         (local.get $2)
+         (local.get $3)
+        )
+       )
+       (if
+        (local.get $10)
+        (block
+         (drop
+          (call_indirect (type $0)
+           (local.get $0)
+           (i32.const 0)
+           (i32.const 0)
+           (i32.add
+            (i32.and
+             (i32.load offset=36
+              (local.get $0)
+             )
+             (i32.const 3)
+            )
+            (i32.const 2)
+           )
+          )
+         )
+         (if
+          (i32.eqz
+           (i32.load
+            (local.get $11)
+           )
+          )
+          (local.set $1
+           (i32.const -1)
+          )
+         )
+         (i32.store
+          (local.get $9)
+          (local.get $10)
+         )
+         (i32.store
+          (local.get $8)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $14)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $13)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $11)
+          (i32.const 0)
+         )
+        )
+       )
+      )
+     )
+     (i32.store
+      (local.get $0)
+      (i32.or
+       (local.tee $2
+        (i32.load
+         (local.get $0)
+        )
+       )
+       (i32.and
+        (local.get $7)
+        (i32.const 32)
+       )
+      )
+     )
+     (if
+      (local.get $12)
+      (call $13
+       (local.get $0)
+      )
+     )
+     (if
+      (i32.and
+       (local.get $2)
+       (i32.const 32)
+      )
+      (local.set $1
+       (i32.const -1)
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $4)
+   )
+   (local.get $1)
+  )
+ )
+ (func $19 (; 32 ;) (type $7) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (param $4 i32) (result i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (local $16 i32)
+  (local $17 i32)
+  (local $18 i32)
+  (local $19 i32)
+  (local $20 i32)
+  (local $21 i32)
+  (local $22 i32)
+  (local $23 i32)
+  (local $24 i32)
+  (local $25 i32)
+  (local $26 i32)
+  (local $27 i32)
+  (local $28 i32)
+  (local $29 i32)
+  (local $30 i32)
+  (local $31 i32)
+  (local $32 i32)
+  (local $33 i32)
+  (local $34 i32)
+  (local $35 i32)
+  (local $36 i32)
+  (local $37 i32)
+  (local $38 i32)
+  (local $39 i32)
+  (local $40 i32)
+  (local $41 i32)
+  (local $42 i32)
+  (local $43 i32)
+  (local $44 i32)
+  (local $45 i32)
+  (local $46 i32)
+  (local $47 i32)
+  (local $48 i32)
+  (local $49 i32)
+  (local $50 i64)
+  (local $51 i64)
+  (local $52 f64)
+  (local $53 f64)
+  (block $label$1 (result i32)
+   (local.set $23
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 624)
+    )
+   )
+   (local.set $20
+    (i32.add
+     (local.get $23)
+     (i32.const 16)
+    )
+   )
+   (local.set $16
+    (local.get $23)
+   )
+   (local.set $36
+    (i32.add
+     (local.get $23)
+     (i32.const 528)
+    )
+   )
+   (local.set $30
+    (i32.ne
+     (local.get $0)
+     (i32.const 0)
+    )
+   )
+   (local.set $38
+    (local.tee $21
+     (i32.add
+      (local.tee $17
+       (i32.add
+        (local.get $23)
+        (i32.const 536)
+       )
+      )
+      (i32.const 40)
+     )
+    )
+   )
+   (local.set $39
+    (i32.add
+     (local.get $17)
+     (i32.const 39)
+    )
+   )
+   (local.set $42
+    (i32.add
+     (local.tee $37
+      (i32.add
+       (local.get $23)
+       (i32.const 8)
+      )
+     )
+     (i32.const 4)
+    )
+   )
+   (local.set $43
+    (i32.sub
+     (i32.const 0)
+     (local.tee $27
+      (local.tee $19
+       (i32.add
+        (local.get $23)
+        (i32.const 588)
+       )
+      )
+     )
+    )
+   )
+   (local.set $33
+    (i32.add
+     (local.tee $17
+      (i32.add
+       (local.get $23)
+       (i32.const 576)
+      )
+     )
+     (i32.const 12)
+    )
+   )
+   (local.set $40
+    (i32.add
+     (local.get $17)
+     (i32.const 11)
+    )
+   )
+   (local.set $44
+    (i32.sub
+     (local.tee $28
+      (local.get $33)
+     )
+     (local.get $27)
+    )
+   )
+   (local.set $45
+    (i32.sub
+     (i32.const -2)
+     (local.get $27)
+    )
+   )
+   (local.set $46
+    (i32.add
+     (local.get $28)
+     (i32.const 2)
+    )
+   )
+   (local.set $48
+    (i32.add
+     (local.tee $47
+      (i32.add
+       (local.get $23)
+       (i32.const 24)
+      )
+     )
+     (i32.const 288)
+    )
+   )
+   (local.set $41
+    (local.tee $31
+     (i32.add
+      (local.get $19)
+      (i32.const 9)
+     )
+    )
+   )
+   (local.set $34
+    (i32.add
+     (local.get $19)
+     (i32.const 8)
+    )
+   )
+   (local.set $15
+    (i32.const 0)
+   )
+   (local.set $10
+    (i32.const 0)
+   )
+   (local.set $17
+    (i32.const 0)
+   )
+   (block $label$2
+    (block $label$3
+     (loop $label$4
+      (block $label$5
+       (if
+        (i32.gt_s
+         (local.get $15)
+         (i32.const -1)
+        )
+        (local.set $15
+         (if (result i32)
+          (i32.gt_s
+           (local.get $10)
+           (i32.sub
+            (i32.const 2147483647)
+            (local.get $15)
+           )
+          )
+          (block (result i32)
+           (i32.store
+            (call $12)
+            (i32.const 75)
+           )
+           (i32.const -1)
+          )
+          (i32.add
+           (local.get $10)
+           (local.get $15)
+          )
+         )
+        )
+       )
+       (br_if $label$3
+        (i32.eqz
+         (i32.shr_s
+          (i32.shl
+           (local.tee $5
+            (i32.load8_s
+             (local.get $1)
+            )
+           )
+           (i32.const 24)
+          )
+          (i32.const 24)
+         )
+        )
+       )
+       (local.set $11
+        (local.get $1)
+       )
+       (block $label$9
+        (block $label$10
+         (loop $label$11
+          (block $label$12
+           (block $label$13
+            (block $label$14
+             (block $label$15
+              (br_table $label$14 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$15 $label$13
+               (i32.sub
+                (i32.shr_s
+                 (i32.shl
+                  (local.get $5)
+                  (i32.const 24)
+                 )
+                 (i32.const 24)
+                )
+                (i32.const 0)
+               )
+              )
+             )
+             (local.set $5
+              (local.get $11)
+             )
+             (br $label$10)
+            )
+            (local.set $5
+             (local.get $11)
+            )
+            (br $label$12)
+           )
+           (local.set $5
+            (i32.load8_s
+             (local.tee $11
+              (i32.add
+               (local.get $11)
+               (i32.const 1)
+              )
+             )
+            )
+           )
+           (br $label$11)
+          )
+         )
+         (br $label$9)
+        )
+        (loop $label$16
+         (br_if $label$9
+          (i32.ne
+           (i32.load8_s offset=1
+            (local.get $5)
+           )
+           (i32.const 37)
+          )
+         )
+         (local.set $11
+          (i32.add
+           (local.get $11)
+           (i32.const 1)
+          )
+         )
+         (br_if $label$16
+          (i32.eq
+           (i32.load8_s
+            (local.tee $5
+             (i32.add
+              (local.get $5)
+              (i32.const 2)
+             )
+            )
+           )
+           (i32.const 37)
+          )
+         )
+        )
+       )
+       (local.set $10
+        (i32.sub
+         (local.get $11)
+         (local.get $1)
+        )
+       )
+       (if
+        (local.get $30)
+        (if
+         (i32.eqz
+          (i32.and
+           (i32.load
+            (local.get $0)
+           )
+           (i32.const 32)
+          )
+         )
+         (drop
+          (call $21
+           (local.get $1)
+           (local.get $10)
+           (local.get $0)
+          )
+         )
+        )
+       )
+       (if
+        (local.get $10)
+        (block
+         (local.set $1
+          (local.get $5)
+         )
+         (br $label$4)
+        )
+       )
+       (local.set $10
+        (if (result i32)
+         (i32.lt_u
+          (local.tee $9
+           (i32.add
+            (i32.shr_s
+             (i32.shl
+              (local.tee $10
+               (i32.load8_s
+                (local.tee $11
+                 (i32.add
+                  (local.get $5)
+                  (i32.const 1)
+                 )
+                )
+               )
+              )
+              (i32.const 24)
+             )
+             (i32.const 24)
+            )
+            (i32.const -48)
+           )
+          )
+          (i32.const 10)
+         )
+         (block (result i32)
+          (local.set $10
+           (i32.add
+            (local.get $5)
+            (i32.const 3)
+           )
+          )
+          (if
+           (local.tee $12
+            (i32.eq
+             (i32.load8_s offset=2
+              (local.get $5)
+             )
+             (i32.const 36)
+            )
+           )
+           (local.set $11
+            (local.get $10)
+           )
+          )
+          (if
+           (local.get $12)
+           (local.set $17
+            (i32.const 1)
+           )
+          )
+          (local.set $5
+           (i32.load8_s
+            (local.get $11)
+           )
+          )
+          (if
+           (i32.eqz
+            (local.get $12)
+           )
+           (local.set $9
+            (i32.const -1)
+           )
+          )
+          (local.get $17)
+         )
+         (block (result i32)
+          (local.set $5
+           (local.get $10)
+          )
+          (local.set $9
+           (i32.const -1)
+          )
+          (local.get $17)
+         )
+        )
+       )
+       (block $label$25
+        (if
+         (i32.lt_u
+          (local.tee $12
+           (i32.add
+            (i32.shr_s
+             (i32.shl
+              (local.get $5)
+              (i32.const 24)
+             )
+             (i32.const 24)
+            )
+            (i32.const -32)
+           )
+          )
+          (i32.const 32)
+         )
+         (block
+          (local.set $17
+           (i32.const 0)
+          )
+          (loop $label$27
+           (br_if $label$25
+            (i32.eqz
+             (i32.and
+              (i32.shl
+               (i32.const 1)
+               (local.get $12)
+              )
+              (i32.const 75913)
+             )
+            )
+           )
+           (local.set $17
+            (i32.or
+             (i32.shl
+              (i32.const 1)
+              (i32.add
+               (i32.shr_s
+                (i32.shl
+                 (local.get $5)
+                 (i32.const 24)
+                )
+                (i32.const 24)
+               )
+               (i32.const -32)
+              )
+             )
+             (local.get $17)
+            )
+           )
+           (br_if $label$27
+            (i32.lt_u
+             (local.tee $12
+              (i32.add
+               (i32.shr_s
+                (i32.shl
+                 (local.tee $5
+                  (i32.load8_s
+                   (local.tee $11
+                    (i32.add
+                     (local.get $11)
+                     (i32.const 1)
+                    )
+                   )
+                  )
+                 )
+                 (i32.const 24)
+                )
+                (i32.const 24)
+               )
+               (i32.const -32)
+              )
+             )
+             (i32.const 32)
+            )
+           )
+          )
+         )
+         (local.set $17
+          (i32.const 0)
+         )
+        )
+       )
+       (block $label$29
+        (if
+         (i32.eq
+          (i32.shr_s
+           (i32.shl
+            (local.get $5)
+            (i32.const 24)
+           )
+           (i32.const 24)
+          )
+          (i32.const 42)
+         )
+         (block
+          (local.set $11
+           (block $label$31 (result i32)
+            (block $label$32
+             (br_if $label$32
+              (i32.ge_u
+               (local.tee $12
+                (i32.add
+                 (i32.shr_s
+                  (i32.shl
+                   (local.tee $5
+                    (i32.load8_s
+                     (local.tee $7
+                      (i32.add
+                       (local.get $11)
+                       (i32.const 1)
+                      )
+                     )
+                    )
+                   )
+                   (i32.const 24)
+                  )
+                  (i32.const 24)
+                 )
+                 (i32.const -48)
+                )
+               )
+               (i32.const 10)
+              )
+             )
+             (br_if $label$32
+              (i32.ne
+               (i32.load8_s offset=2
+                (local.get $11)
+               )
+               (i32.const 36)
+              )
+             )
+             (i32.store
+              (i32.add
+               (local.get $4)
+               (i32.shl
+                (local.get $12)
+                (i32.const 2)
+               )
+              )
+              (i32.const 10)
+             )
+             (local.set $8
+              (i32.const 1)
+             )
+             (local.set $10
+              (i32.wrap_i64
+               (i64.load
+                (i32.add
+                 (local.get $3)
+                 (i32.shl
+                  (i32.add
+                   (i32.load8_s
+                    (local.get $7)
+                   )
+                   (i32.const -48)
+                  )
+                  (i32.const 3)
+                 )
+                )
+               )
+              )
+             )
+             (br $label$31
+              (i32.add
+               (local.get $11)
+               (i32.const 3)
+              )
+             )
+            )
+            (if
+             (local.get $10)
+             (block
+              (local.set $15
+               (i32.const -1)
+              )
+              (br $label$5)
+             )
+            )
+            (if
+             (i32.eqz
+              (local.get $30)
+             )
+             (block
+              (local.set $12
+               (local.get $17)
+              )
+              (local.set $17
+               (i32.const 0)
+              )
+              (local.set $11
+               (local.get $7)
+              )
+              (local.set $10
+               (i32.const 0)
+              )
+              (br $label$29)
+             )
+            )
+            (local.set $10
+             (i32.load
+              (local.tee $11
+               (i32.and
+                (i32.add
+                 (i32.load
+                  (local.get $2)
+                 )
+                 (i32.const 3)
+                )
+                (i32.const -4)
+               )
+              )
+             )
+            )
+            (i32.store
+             (local.get $2)
+             (i32.add
+              (local.get $11)
+              (i32.const 4)
+             )
+            )
+            (local.set $8
+             (i32.const 0)
+            )
+            (local.get $7)
+           )
+          )
+          (local.set $12
+           (i32.or
+            (local.get $17)
+            (i32.const 8192)
+           )
+          )
+          (local.set $7
+           (i32.sub
+            (i32.const 0)
+            (local.get $10)
+           )
+          )
+          (local.set $5
+           (i32.load8_s
+            (local.get $11)
+           )
+          )
+          (if
+           (i32.eqz
+            (local.tee $6
+             (i32.lt_s
+              (local.get $10)
+              (i32.const 0)
+             )
+            )
+           )
+           (local.set $12
+            (local.get $17)
+           )
+          )
+          (local.set $17
+           (local.get $8)
+          )
+          (if
+           (local.get $6)
+           (local.set $10
+            (local.get $7)
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.tee $12
+            (i32.add
+             (i32.shr_s
+              (i32.shl
+               (local.get $5)
+               (i32.const 24)
+              )
+              (i32.const 24)
+             )
+             (i32.const -48)
+            )
+           )
+           (i32.const 10)
+          )
+          (block
+           (local.set $7
+            (i32.const 0)
+           )
+           (local.set $5
+            (local.get $12)
+           )
+           (loop $label$39
+            (local.set $7
+             (i32.add
+              (i32.mul
+               (local.get $7)
+               (i32.const 10)
+              )
+              (local.get $5)
+             )
+            )
+            (br_if $label$39
+             (i32.lt_u
+              (local.tee $5
+               (i32.add
+                (i32.shr_s
+                 (i32.shl
+                  (local.tee $12
+                   (i32.load8_s
+                    (local.tee $11
+                     (i32.add
+                      (local.get $11)
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                  )
+                  (i32.const 24)
+                 )
+                 (i32.const 24)
+                )
+                (i32.const -48)
+               )
+              )
+              (i32.const 10)
+             )
+            )
+           )
+           (if
+            (i32.lt_s
+             (local.get $7)
+             (i32.const 0)
+            )
+            (block
+             (local.set $15
+              (i32.const -1)
+             )
+             (br $label$5)
+            )
+            (block
+             (local.set $5
+              (local.get $12)
+             )
+             (local.set $12
+              (local.get $17)
+             )
+             (local.set $17
+              (local.get $10)
+             )
+             (local.set $10
+              (local.get $7)
+             )
+            )
+           )
+          )
+          (block
+           (local.set $12
+            (local.get $17)
+           )
+           (local.set $17
+            (local.get $10)
+           )
+           (local.set $10
+            (i32.const 0)
+           )
+          )
+         )
+        )
+       )
+       (block $label$43
+        (if
+         (i32.eq
+          (i32.shr_s
+           (i32.shl
+            (local.get $5)
+            (i32.const 24)
+           )
+           (i32.const 24)
+          )
+          (i32.const 46)
+         )
+         (block
+          (if
+           (i32.ne
+            (i32.shr_s
+             (i32.shl
+              (local.tee $5
+               (i32.load8_s
+                (local.tee $7
+                 (i32.add
+                  (local.get $11)
+                  (i32.const 1)
+                 )
+                )
+               )
+              )
+              (i32.const 24)
+             )
+             (i32.const 24)
+            )
+            (i32.const 42)
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.tee $5
+               (i32.add
+                (i32.shr_s
+                 (i32.shl
+                  (local.get $5)
+                  (i32.const 24)
+                 )
+                 (i32.const 24)
+                )
+                (i32.const -48)
+               )
+              )
+              (i32.const 10)
+             )
+             (block
+              (local.set $11
+               (local.get $7)
+              )
+              (local.set $7
+               (i32.const 0)
+              )
+             )
+             (block
+              (local.set $5
+               (i32.const 0)
+              )
+              (local.set $11
+               (local.get $7)
+              )
+              (br $label$43)
+             )
+            )
+            (loop $label$48
+             (local.set $5
+              (i32.add
+               (i32.mul
+                (local.get $7)
+                (i32.const 10)
+               )
+               (local.get $5)
+              )
+             )
+             (br_if $label$43
+              (i32.ge_u
+               (local.tee $8
+                (i32.add
+                 (i32.load8_s
+                  (local.tee $11
+                   (i32.add
+                    (local.get $11)
+                    (i32.const 1)
+                   )
+                  )
+                 )
+                 (i32.const -48)
+                )
+               )
+               (i32.const 10)
+              )
+             )
+             (local.set $7
+              (local.get $5)
+             )
+             (local.set $5
+              (local.get $8)
+             )
+             (br $label$48)
+            )
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.tee $5
+             (i32.add
+              (i32.load8_s
+               (local.tee $7
+                (i32.add
+                 (local.get $11)
+                 (i32.const 2)
+                )
+               )
+              )
+              (i32.const -48)
+             )
+            )
+            (i32.const 10)
+           )
+           (if
+            (i32.eq
+             (i32.load8_s offset=3
+              (local.get $11)
+             )
+             (i32.const 36)
+            )
+            (block
+             (i32.store
+              (i32.add
+               (local.get $4)
+               (i32.shl
+                (local.get $5)
+                (i32.const 2)
+               )
+              )
+              (i32.const 10)
+             )
+             (local.set $5
+              (i32.wrap_i64
+               (i64.load
+                (i32.add
+                 (local.get $3)
+                 (i32.shl
+                  (i32.add
+                   (i32.load8_s
+                    (local.get $7)
+                   )
+                   (i32.const -48)
+                  )
+                  (i32.const 3)
+                 )
+                )
+               )
+              )
+             )
+             (local.set $11
+              (i32.add
+               (local.get $11)
+               (i32.const 4)
+              )
+             )
+             (br $label$43)
+            )
+           )
+          )
+          (if
+           (local.get $17)
+           (block
+            (local.set $15
+             (i32.const -1)
+            )
+            (br $label$5)
+           )
+          )
+          (local.set $11
+           (if (result i32)
+            (local.get $30)
+            (block (result i32)
+             (local.set $5
+              (i32.load
+               (local.tee $11
+                (i32.and
+                 (i32.add
+                  (i32.load
+                   (local.get $2)
+                  )
+                  (i32.const 3)
+                 )
+                 (i32.const -4)
+                )
+               )
+              )
+             )
+             (i32.store
+              (local.get $2)
+              (i32.add
+               (local.get $11)
+               (i32.const 4)
+              )
+             )
+             (local.get $7)
+            )
+            (block (result i32)
+             (local.set $5
+              (i32.const 0)
+             )
+             (local.get $7)
+            )
+           )
+          )
+         )
+         (local.set $5
+          (i32.const -1)
+         )
+        )
+       )
+       (local.set $7
+        (local.get $11)
+       )
+       (local.set $8
+        (i32.const 0)
+       )
+       (loop $label$55
+        (if
+         (i32.gt_u
+          (local.tee $6
+           (i32.add
+            (i32.load8_s
+             (local.get $7)
+            )
+            (i32.const -65)
+           )
+          )
+          (i32.const 57)
+         )
+         (block
+          (local.set $15
+           (i32.const -1)
+          )
+          (br $label$5)
+         )
+        )
+        (local.set $11
+         (i32.add
+          (local.get $7)
+          (i32.const 1)
+         )
+        )
+        (if
+         (i32.lt_u
+          (i32.add
+           (local.tee $6
+            (i32.and
+             (local.tee $13
+              (i32.load8_s
+               (i32.add
+                (i32.add
+                 (i32.mul
+                  (local.get $8)
+                  (i32.const 58)
+                 )
+                 (i32.const 1177)
+                )
+                (local.get $6)
+               )
+              )
+             )
+             (i32.const 255)
+            )
+           )
+           (i32.const -1)
+          )
+          (i32.const 8)
+         )
+         (block
+          (local.set $7
+           (local.get $11)
+          )
+          (local.set $8
+           (local.get $6)
+          )
+          (br $label$55)
+         )
+        )
+       )
+       (if
+        (i32.eqz
+         (i32.shr_s
+          (i32.shl
+           (local.get $13)
+           (i32.const 24)
+          )
+          (i32.const 24)
+         )
+        )
+        (block
+         (local.set $15
+          (i32.const -1)
+         )
+         (br $label$5)
+        )
+       )
+       (local.set $14
+        (i32.gt_s
+         (local.get $9)
+         (i32.const -1)
+        )
+       )
+       (block $label$59
+        (block $label$60
+         (if
+          (i32.eq
+           (i32.shr_s
+            (i32.shl
+             (local.get $13)
+             (i32.const 24)
+            )
+            (i32.const 24)
+           )
+           (i32.const 19)
+          )
+          (if
+           (local.get $14)
+           (block
+            (local.set $15
+             (i32.const -1)
+            )
+            (br $label$5)
+           )
+           (br $label$60)
+          )
+          (block
+           (if
+            (local.get $14)
+            (block
+             (i32.store
+              (i32.add
+               (local.get $4)
+               (i32.shl
+                (local.get $9)
+                (i32.const 2)
+               )
+              )
+              (local.get $6)
+             )
+             (i64.store
+              (local.get $16)
+              (i64.load
+               (i32.add
+                (local.get $3)
+                (i32.shl
+                 (local.get $9)
+                 (i32.const 3)
+                )
+               )
+              )
+             )
+             (br $label$60)
+            )
+           )
+           (if
+            (i32.eqz
+             (local.get $30)
+            )
+            (block
+             (local.set $15
+              (i32.const 0)
+             )
+             (br $label$5)
+            )
+           )
+           (call $22
+            (local.get $16)
+            (local.get $6)
+            (local.get $2)
+           )
+          )
+         )
+         (br $label$59)
+        )
+        (if
+         (i32.eqz
+          (local.get $30)
+         )
+         (block
+          (local.set $10
+           (i32.const 0)
+          )
+          (local.set $1
+           (local.get $11)
+          )
+          (br $label$4)
+         )
+        )
+       )
+       (local.set $9
+        (i32.and
+         (local.tee $7
+          (i32.load8_s
+           (local.get $7)
+          )
+         )
+         (i32.const -33)
+        )
+       )
+       (if
+        (i32.eqz
+         (i32.and
+          (i32.ne
+           (local.get $8)
+           (i32.const 0)
+          )
+          (i32.eq
+           (i32.and
+            (local.get $7)
+            (i32.const 15)
+           )
+           (i32.const 3)
+          )
+         )
+        )
+        (local.set $9
+         (local.get $7)
+        )
+       )
+       (local.set $7
+        (i32.and
+         (local.get $12)
+         (i32.const -65537)
+        )
+       )
+       (if
+        (i32.and
+         (local.get $12)
+         (i32.const 8192)
+        )
+        (local.set $12
+         (local.get $7)
+        )
+       )
+       (block $label$70
+        (block $label$71
+         (block $label$72
+          (block $label$73
+           (block $label$74
+            (block $label$75
+             (block $label$76
+              (block $label$77
+               (block $label$78
+                (block $label$79
+                 (block $label$80
+                  (block $label$81
+                   (block $label$82
+                    (block $label$83
+                     (block $label$84
+                      (block $label$85
+                       (block $label$86
+                        (block $label$87
+                         (block $label$88
+                          (block $label$89
+                           (br_table $label$78 $label$77 $label$80 $label$77 $label$78 $label$78 $label$78 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$79 $label$77 $label$77 $label$77 $label$77 $label$87 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$78 $label$77 $label$83 $label$85 $label$78 $label$78 $label$78 $label$77 $label$85 $label$77 $label$77 $label$77 $label$82 $label$89 $label$86 $label$88 $label$77 $label$77 $label$81 $label$77 $label$84 $label$77 $label$77 $label$87 $label$77
+                            (i32.sub
+                             (local.get $9)
+                             (i32.const 65)
+                            )
+                           )
+                          )
+                          (block $label$90
+                           (block $label$91
+                            (block $label$92
+                             (block $label$93
+                              (block $label$94
+                               (block $label$95
+                                (block $label$96
+                                 (block $label$97
+                                  (br_table $label$97 $label$96 $label$95 $label$94 $label$93 $label$90 $label$92 $label$91 $label$90
+                                   (i32.sub
+                                    (i32.shr_s
+                                     (i32.shl
+                                      (i32.and
+                                       (local.get $8)
+                                       (i32.const 255)
+                                      )
+                                      (i32.const 24)
+                                     )
+                                     (i32.const 24)
+                                    )
+                                    (i32.const 0)
+                                   )
+                                  )
+                                 )
+                                 (i32.store
+                                  (i32.load
+                                   (local.get $16)
+                                  )
+                                  (local.get $15)
+                                 )
+                                 (local.set $10
+                                  (i32.const 0)
+                                 )
+                                 (local.set $1
+                                  (local.get $11)
+                                 )
+                                 (br $label$4)
+                                )
+                                (i32.store
+                                 (i32.load
+                                  (local.get $16)
+                                 )
+                                 (local.get $15)
+                                )
+                                (local.set $10
+                                 (i32.const 0)
+                                )
+                                (local.set $1
+                                 (local.get $11)
+                                )
+                                (br $label$4)
+                               )
+                               (i64.store
+                                (i32.load
+                                 (local.get $16)
+                                )
+                                (i64.extend_i32_s
+                                 (local.get $15)
+                                )
+                               )
+                               (local.set $10
+                                (i32.const 0)
+                               )
+                               (local.set $1
+                                (local.get $11)
+                               )
+                               (br $label$4)
+                              )
+                              (i32.store16
+                               (i32.load
+                                (local.get $16)
+                               )
+                               (local.get $15)
+                              )
+                              (local.set $10
+                               (i32.const 0)
+                              )
+                              (local.set $1
+                               (local.get $11)
+                              )
+                              (br $label$4)
+                             )
+                             (i32.store8
+                              (i32.load
+                               (local.get $16)
+                              )
+                              (local.get $15)
+                             )
+                             (local.set $10
+                              (i32.const 0)
+                             )
+                             (local.set $1
+                              (local.get $11)
+                             )
+                             (br $label$4)
+                            )
+                            (i32.store
+                             (i32.load
+                              (local.get $16)
+                             )
+                             (local.get $15)
+                            )
+                            (local.set $10
+                             (i32.const 0)
+                            )
+                            (local.set $1
+                             (local.get $11)
+                            )
+                            (br $label$4)
+                           )
+                           (i64.store
+                            (i32.load
+                             (local.get $16)
+                            )
+                            (i64.extend_i32_s
+                             (local.get $15)
+                            )
+                           )
+                           (local.set $10
+                            (i32.const 0)
+                           )
+                           (local.set $1
+                            (local.get $11)
+                           )
+                           (br $label$4)
+                          )
+                          (local.set $10
+                           (i32.const 0)
+                          )
+                          (local.set $1
+                           (local.get $11)
+                          )
+                          (br $label$4)
+                         )
+                         (local.set $12
+                          (i32.or
+                           (local.get $12)
+                           (i32.const 8)
+                          )
+                         )
+                         (if
+                          (i32.le_u
+                           (local.get $5)
+                           (i32.const 8)
+                          )
+                          (local.set $5
+                           (i32.const 8)
+                          )
+                         )
+                         (local.set $9
+                          (i32.const 120)
+                         )
+                         (br $label$76)
+                        )
+                        (br $label$76)
+                       )
+                       (if
+                        (i64.eq
+                         (local.tee $50
+                          (i64.load
+                           (local.get $16)
+                          )
+                         )
+                         (i64.const 0)
+                        )
+                        (local.set $7
+                         (local.get $21)
+                        )
+                        (block
+                         (local.set $1
+                          (local.get $21)
+                         )
+                         (loop $label$101
+                          (i64.store8
+                           (local.tee $1
+                            (i32.add
+                             (local.get $1)
+                             (i32.const -1)
+                            )
+                           )
+                           (i64.or
+                            (i64.and
+                             (local.get $50)
+                             (i64.const 7)
+                            )
+                            (i64.const 48)
+                           )
+                          )
+                          (br_if $label$101
+                           (i64.ne
+                            (local.tee $50
+                             (i64.shr_u
+                              (local.get $50)
+                              (i64.const 3)
+                             )
+                            )
+                            (i64.const 0)
+                           )
+                          )
+                          (local.set $7
+                           (local.get $1)
+                          )
+                         )
+                        )
+                       )
+                       (if
+                        (i32.and
+                         (local.get $12)
+                         (i32.const 8)
+                        )
+                        (block
+                         (local.set $8
+                          (i32.add
+                           (local.tee $1
+                            (i32.sub
+                             (local.get $38)
+                             (local.get $7)
+                            )
+                           )
+                           (i32.const 1)
+                          )
+                         )
+                         (if
+                          (i32.le_s
+                           (local.get $5)
+                           (local.get $1)
+                          )
+                          (local.set $5
+                           (local.get $8)
+                          )
+                         )
+                         (local.set $6
+                          (i32.const 0)
+                         )
+                         (local.set $8
+                          (i32.const 1657)
+                         )
+                         (br $label$71)
+                        )
+                        (block
+                         (local.set $6
+                          (i32.const 0)
+                         )
+                         (local.set $8
+                          (i32.const 1657)
+                         )
+                         (br $label$71)
+                        )
+                       )
+                      )
+                      (if
+                       (i64.lt_s
+                        (local.tee $50
+                         (i64.load
+                          (local.get $16)
+                         )
+                        )
+                        (i64.const 0)
+                       )
+                       (block
+                        (i64.store
+                         (local.get $16)
+                         (local.tee $50
+                          (i64.sub
+                           (i64.const 0)
+                           (local.get $50)
+                          )
+                         )
+                        )
+                        (local.set $6
+                         (i32.const 1)
+                        )
+                        (local.set $8
+                         (i32.const 1657)
+                        )
+                        (br $label$75)
+                       )
+                      )
+                      (if
+                       (i32.and
+                        (local.get $12)
+                        (i32.const 2048)
+                       )
+                       (block
+                        (local.set $6
+                         (i32.const 1)
+                        )
+                        (local.set $8
+                         (i32.const 1658)
+                        )
+                        (br $label$75)
+                       )
+                       (block
+                        (local.set $6
+                         (local.tee $1
+                          (i32.and
+                           (local.get $12)
+                           (i32.const 1)
+                          )
+                         )
+                        )
+                        (local.set $8
+                         (if (result i32)
+                          (local.get $1)
+                          (i32.const 1659)
+                          (i32.const 1657)
+                         )
+                        )
+                        (br $label$75)
+                       )
+                      )
+                     )
+                     (local.set $50
+                      (i64.load
+                       (local.get $16)
+                      )
+                     )
+                     (local.set $6
+                      (i32.const 0)
+                     )
+                     (local.set $8
+                      (i32.const 1657)
+                     )
+                     (br $label$75)
+                    )
+                    (i64.store8
+                     (local.get $39)
+                     (i64.load
+                      (local.get $16)
+                     )
+                    )
+                    (local.set $1
+                     (local.get $39)
+                    )
+                    (local.set $12
+                     (local.get $7)
+                    )
+                    (local.set $7
+                     (i32.const 1)
+                    )
+                    (local.set $6
+                     (i32.const 0)
+                    )
+                    (local.set $8
+                     (i32.const 1657)
+                    )
+                    (local.set $5
+                     (local.get $21)
+                    )
+                    (br $label$70)
+                   )
+                   (local.set $1
+                    (call $24
+                     (i32.load
+                      (call $12)
+                     )
+                    )
+                   )
+                   (br $label$74)
+                  )
+                  (if
+                   (i32.eqz
+                    (local.tee $1
+                     (i32.load
+                      (local.get $16)
+                     )
+                    )
+                   )
+                   (local.set $1
+                    (i32.const 1667)
+                   )
+                  )
+                  (br $label$74)
+                 )
+                 (i64.store32
+                  (local.get $37)
+                  (i64.load
+                   (local.get $16)
+                  )
+                 )
+                 (i32.store
+                  (local.get $42)
+                  (i32.const 0)
+                 )
+                 (i32.store
+                  (local.get $16)
+                  (local.get $37)
+                 )
+                 (local.set $7
+                  (local.get $37)
+                 )
+                 (local.set $6
+                  (i32.const -1)
+                 )
+                 (br $label$73)
+                )
+                (local.set $7
+                 (i32.load
+                  (local.get $16)
+                 )
+                )
+                (if
+                 (local.get $5)
+                 (block
+                  (local.set $6
+                   (local.get $5)
+                  )
+                  (br $label$73)
+                 )
+                 (block
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (i32.const 0)
+                   (local.get $12)
+                  )
+                  (local.set $1
+                   (i32.const 0)
+                  )
+                  (br $label$72)
+                 )
+                )
+               )
+               (local.set $52
+                (f64.load
+                 (local.get $16)
+                )
+               )
+               (i32.store
+                (local.get $20)
+                (i32.const 0)
+               )
+               (local.set $26
+                (if (result i32)
+                 (i64.lt_s
+                  (i64.reinterpret_f64
+                   (local.get $52)
+                  )
+                  (i64.const 0)
+                 )
+                 (block (result i32)
+                  (local.set $24
+                   (i32.const 1)
+                  )
+                  (local.set $52
+                   (f64.neg
+                    (local.get $52)
+                   )
+                  )
+                  (i32.const 1674)
+                 )
+                 (block (result i32)
+                  (local.set $1
+                   (i32.and
+                    (local.get $12)
+                    (i32.const 1)
+                   )
+                  )
+                  (if (result i32)
+                   (i32.and
+                    (local.get $12)
+                    (i32.const 2048)
+                   )
+                   (block (result i32)
+                    (local.set $24
+                     (i32.const 1)
+                    )
+                    (i32.const 1677)
+                   )
+                   (block (result i32)
+                    (local.set $24
+                     (local.get $1)
+                    )
+                    (if (result i32)
+                     (local.get $1)
+                     (i32.const 1680)
+                     (i32.const 1675)
+                    )
+                   )
+                  )
+                 )
+                )
+               )
+               (block $label$119
+                (if
+                 (i64.lt_u
+                  (i64.and
+                   (i64.reinterpret_f64
+                    (local.get $52)
+                   )
+                   (i64.const 9218868437227405312)
+                  )
+                  (i64.const 9218868437227405312)
+                 )
+                 (block
+                  (if
+                   (local.tee $1
+                    (f64.ne
+                     (local.tee $52
+                      (f64.mul
+                       (call $27
+                        (local.get $52)
+                        (local.get $20)
+                       )
+                       (f64.const 2)
+                      )
+                     )
+                     (f64.const 0)
+                    )
+                   )
+                   (i32.store
+                    (local.get $20)
+                    (i32.add
+                     (i32.load
+                      (local.get $20)
+                     )
+                     (i32.const -1)
+                    )
+                   )
+                  )
+                  (if
+                   (i32.eq
+                    (local.tee $22
+                     (i32.or
+                      (local.get $9)
+                      (i32.const 32)
+                     )
+                    )
+                    (i32.const 97)
+                   )
+                   (block
+                    (local.set $1
+                     (i32.add
+                      (local.get $26)
+                      (i32.const 9)
+                     )
+                    )
+                    (if
+                     (local.tee $6
+                      (i32.and
+                       (local.get $9)
+                       (i32.const 32)
+                      )
+                     )
+                     (local.set $26
+                      (local.get $1)
+                     )
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.or
+                       (i32.gt_u
+                        (local.get $5)
+                        (i32.const 11)
+                       )
+                       (i32.eqz
+                        (local.tee $1
+                         (i32.sub
+                          (i32.const 12)
+                          (local.get $5)
+                         )
+                        )
+                       )
+                      )
+                     )
+                     (block
+                      (local.set $53
+                       (f64.const 8)
+                      )
+                      (loop $label$125
+                       (local.set $53
+                        (f64.mul
+                         (local.get $53)
+                         (f64.const 16)
+                        )
+                       )
+                       (br_if $label$125
+                        (local.tee $1
+                         (i32.add
+                          (local.get $1)
+                          (i32.const -1)
+                         )
+                        )
+                       )
+                      )
+                      (local.set $52
+                       (if (result f64)
+                        (i32.eq
+                         (i32.load8_s
+                          (local.get $26)
+                         )
+                         (i32.const 45)
+                        )
+                        (f64.neg
+                         (f64.add
+                          (local.get $53)
+                          (f64.sub
+                           (f64.neg
+                            (local.get $52)
+                           )
+                           (local.get $53)
+                          )
+                         )
+                        )
+                        (f64.sub
+                         (f64.add
+                          (local.get $52)
+                          (local.get $53)
+                         )
+                         (local.get $53)
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (local.set $1
+                     (i32.sub
+                      (i32.const 0)
+                      (local.tee $7
+                       (i32.load
+                        (local.get $20)
+                       )
+                      )
+                     )
+                    )
+                    (if
+                     (i32.eq
+                      (local.tee $1
+                       (call $23
+                        (i64.extend_i32_s
+                         (if (result i32)
+                          (i32.lt_s
+                           (local.get $7)
+                           (i32.const 0)
+                          )
+                          (local.get $1)
+                          (local.get $7)
+                         )
+                        )
+                        (local.get $33)
+                       )
+                      )
+                      (local.get $33)
+                     )
+                     (block
+                      (i32.store8
+                       (local.get $40)
+                       (i32.const 48)
+                      )
+                      (local.set $1
+                       (local.get $40)
+                      )
+                     )
+                    )
+                    (local.set $13
+                     (i32.or
+                      (local.get $24)
+                      (i32.const 2)
+                     )
+                    )
+                    (i32.store8
+                     (i32.add
+                      (local.get $1)
+                      (i32.const -1)
+                     )
+                     (i32.add
+                      (i32.and
+                       (i32.shr_s
+                        (local.get $7)
+                        (i32.const 31)
+                       )
+                       (i32.const 2)
+                      )
+                      (i32.const 43)
+                     )
+                    )
+                    (i32.store8
+                     (local.tee $8
+                      (i32.add
+                       (local.get $1)
+                       (i32.const -2)
+                      )
+                     )
+                     (i32.add
+                      (local.get $9)
+                      (i32.const 15)
+                     )
+                    )
+                    (local.set $9
+                     (i32.lt_s
+                      (local.get $5)
+                      (i32.const 1)
+                     )
+                    )
+                    (local.set $14
+                     (i32.eqz
+                      (i32.and
+                       (local.get $12)
+                       (i32.const 8)
+                      )
+                     )
+                    )
+                    (local.set $1
+                     (local.get $19)
+                    )
+                    (loop $label$131
+                     (i32.store8
+                      (local.get $1)
+                      (i32.or
+                       (i32.load8_u
+                        (i32.add
+                         (local.tee $7
+                          (i32.trunc_f64_s
+                           (local.get $52)
+                          )
+                         )
+                         (i32.const 1641)
+                        )
+                       )
+                       (local.get $6)
+                      )
+                     )
+                     (local.set $52
+                      (f64.mul
+                       (f64.sub
+                        (local.get $52)
+                        (f64.convert_i32_s
+                         (local.get $7)
+                        )
+                       )
+                       (f64.const 16)
+                      )
+                     )
+                     (local.set $1
+                      (block $label$132 (result i32)
+                       (if (result i32)
+                        (i32.eq
+                         (i32.sub
+                          (local.tee $7
+                           (i32.add
+                            (local.get $1)
+                            (i32.const 1)
+                           )
+                          )
+                          (local.get $27)
+                         )
+                         (i32.const 1)
+                        )
+                        (block (result i32)
+                         (drop
+                          (br_if $label$132
+                           (local.get $7)
+                           (i32.and
+                            (local.get $14)
+                            (i32.and
+                             (local.get $9)
+                             (f64.eq
+                              (local.get $52)
+                              (f64.const 0)
+                             )
+                            )
+                           )
+                          )
+                         )
+                         (i32.store8
+                          (local.get $7)
+                          (i32.const 46)
+                         )
+                         (i32.add
+                          (local.get $1)
+                          (i32.const 2)
+                         )
+                        )
+                        (local.get $7)
+                       )
+                      )
+                     )
+                     (br_if $label$131
+                      (f64.ne
+                       (local.get $52)
+                       (f64.const 0)
+                      )
+                     )
+                    )
+                    (local.set $6
+                     (i32.sub
+                      (i32.add
+                       (local.get $46)
+                       (local.get $5)
+                      )
+                      (local.tee $7
+                       (local.get $8)
+                      )
+                     )
+                    )
+                    (local.set $9
+                     (i32.add
+                      (i32.sub
+                       (local.get $44)
+                       (local.get $7)
+                      )
+                      (local.get $1)
+                     )
+                    )
+                    (call $25
+                     (local.get $0)
+                     (i32.const 32)
+                     (local.get $10)
+                     (local.tee $5
+                      (i32.add
+                       (if (result i32)
+                        (i32.and
+                         (i32.ne
+                          (local.get $5)
+                          (i32.const 0)
+                         )
+                         (i32.lt_s
+                          (i32.add
+                           (local.get $45)
+                           (local.get $1)
+                          )
+                          (local.get $5)
+                         )
+                        )
+                        (local.get $6)
+                        (local.tee $6
+                         (local.get $9)
+                        )
+                       )
+                       (local.get $13)
+                      )
+                     )
+                     (local.get $12)
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $21
+                       (local.get $26)
+                       (local.get $13)
+                       (local.get $0)
+                      )
+                     )
+                    )
+                    (call $25
+                     (local.get $0)
+                     (i32.const 48)
+                     (local.get $10)
+                     (local.get $5)
+                     (i32.xor
+                      (local.get $12)
+                      (i32.const 65536)
+                     )
+                    )
+                    (local.set $1
+                     (i32.sub
+                      (local.get $1)
+                      (local.get $27)
+                     )
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $21
+                       (local.get $19)
+                       (local.get $1)
+                       (local.get $0)
+                      )
+                     )
+                    )
+                    (call $25
+                     (local.get $0)
+                     (i32.const 48)
+                     (i32.sub
+                      (local.get $6)
+                      (i32.add
+                       (local.get $1)
+                       (local.tee $1
+                        (i32.sub
+                         (local.get $28)
+                         (local.get $7)
+                        )
+                       )
+                      )
+                     )
+                     (i32.const 0)
+                     (i32.const 0)
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $21
+                       (local.get $8)
+                       (local.get $1)
+                       (local.get $0)
+                      )
+                     )
+                    )
+                    (call $25
+                     (local.get $0)
+                     (i32.const 32)
+                     (local.get $10)
+                     (local.get $5)
+                     (i32.xor
+                      (local.get $12)
+                      (i32.const 8192)
+                     )
+                    )
+                    (if
+                     (i32.ge_s
+                      (local.get $5)
+                      (local.get $10)
+                     )
+                     (local.set $10
+                      (local.get $5)
+                     )
+                    )
+                    (br $label$119)
+                   )
+                  )
+                  (if
+                   (local.get $1)
+                   (block
+                    (i32.store
+                     (local.get $20)
+                     (local.tee $6
+                      (i32.add
+                       (i32.load
+                        (local.get $20)
+                       )
+                       (i32.const -28)
+                      )
+                     )
+                    )
+                    (local.set $52
+                     (f64.mul
+                      (local.get $52)
+                      (f64.const 268435456)
+                     )
+                    )
+                   )
+                   (local.set $6
+                    (i32.load
+                     (local.get $20)
+                    )
+                   )
+                  )
+                  (local.set $8
+                   (local.tee $7
+                    (if (result i32)
+                     (i32.lt_s
+                      (local.get $6)
+                      (i32.const 0)
+                     )
+                     (local.get $47)
+                     (local.get $48)
+                    )
+                   )
+                  )
+                  (loop $label$145
+                   (i32.store
+                    (local.get $8)
+                    (local.tee $1
+                     (i32.trunc_f64_s
+                      (local.get $52)
+                     )
+                    )
+                   )
+                   (local.set $8
+                    (i32.add
+                     (local.get $8)
+                     (i32.const 4)
+                    )
+                   )
+                   (br_if $label$145
+                    (f64.ne
+                     (local.tee $52
+                      (f64.mul
+                       (f64.sub
+                        (local.get $52)
+                        (f64.convert_i32_u
+                         (local.get $1)
+                        )
+                       )
+                       (f64.const 1e9)
+                      )
+                     )
+                     (f64.const 0)
+                    )
+                   )
+                  )
+                  (if
+                   (i32.gt_s
+                    (local.get $6)
+                    (i32.const 0)
+                   )
+                   (block
+                    (local.set $1
+                     (local.get $7)
+                    )
+                    (loop $label$147
+                     (local.set $14
+                      (if (result i32)
+                       (i32.gt_s
+                        (local.get $6)
+                        (i32.const 29)
+                       )
+                       (i32.const 29)
+                       (local.get $6)
+                      )
+                     )
+                     (block $label$150
+                      (if
+                       (i32.ge_u
+                        (local.tee $6
+                         (i32.add
+                          (local.get $8)
+                          (i32.const -4)
+                         )
+                        )
+                        (local.get $1)
+                       )
+                       (block
+                        (local.set $50
+                         (i64.extend_i32_u
+                          (local.get $14)
+                         )
+                        )
+                        (local.set $13
+                         (i32.const 0)
+                        )
+                        (loop $label$152
+                         (i64.store32
+                          (local.get $6)
+                          (i64.rem_u
+                           (local.tee $51
+                            (i64.add
+                             (i64.shl
+                              (i64.extend_i32_u
+                               (i32.load
+                                (local.get $6)
+                               )
+                              )
+                              (local.get $50)
+                             )
+                             (i64.extend_i32_u
+                              (local.get $13)
+                             )
+                            )
+                           )
+                           (i64.const 1000000000)
+                          )
+                         )
+                         (local.set $13
+                          (i32.wrap_i64
+                           (i64.div_u
+                            (local.get $51)
+                            (i64.const 1000000000)
+                           )
+                          )
+                         )
+                         (br_if $label$152
+                          (i32.ge_u
+                           (local.tee $6
+                            (i32.add
+                             (local.get $6)
+                             (i32.const -4)
+                            )
+                           )
+                           (local.get $1)
+                          )
+                         )
+                        )
+                        (br_if $label$150
+                         (i32.eqz
+                          (local.get $13)
+                         )
+                        )
+                        (i32.store
+                         (local.tee $1
+                          (i32.add
+                           (local.get $1)
+                           (i32.const -4)
+                          )
+                         )
+                         (local.get $13)
+                        )
+                       )
+                      )
+                     )
+                     (loop $label$153
+                      (if
+                       (i32.gt_u
+                        (local.get $8)
+                        (local.get $1)
+                       )
+                       (if
+                        (i32.eqz
+                         (i32.load
+                          (local.tee $6
+                           (i32.add
+                            (local.get $8)
+                            (i32.const -4)
+                           )
+                          )
+                         )
+                        )
+                        (block
+                         (local.set $8
+                          (local.get $6)
+                         )
+                         (br $label$153)
+                        )
+                       )
+                      )
+                     )
+                     (i32.store
+                      (local.get $20)
+                      (local.tee $6
+                       (i32.sub
+                        (i32.load
+                         (local.get $20)
+                        )
+                        (local.get $14)
+                       )
+                      )
+                     )
+                     (br_if $label$147
+                      (i32.gt_s
+                       (local.get $6)
+                       (i32.const 0)
+                      )
+                     )
+                    )
+                   )
+                   (local.set $1
+                    (local.get $7)
+                   )
+                  )
+                  (local.set $18
+                   (if (result i32)
+                    (i32.lt_s
+                     (local.get $5)
+                     (i32.const 0)
+                    )
+                    (i32.const 6)
+                    (local.get $5)
+                   )
+                  )
+                  (if
+                   (i32.lt_s
+                    (local.get $6)
+                    (i32.const 0)
+                   )
+                   (block
+                    (local.set $14
+                     (i32.add
+                      (i32.div_s
+                       (i32.add
+                        (local.get $18)
+                        (i32.const 25)
+                       )
+                       (i32.const 9)
+                      )
+                      (i32.const 1)
+                     )
+                    )
+                    (local.set $25
+                     (i32.eq
+                      (local.get $22)
+                      (i32.const 102)
+                     )
+                    )
+                    (local.set $5
+                     (local.get $8)
+                    )
+                    (loop $label$160
+                     (if
+                      (i32.gt_s
+                       (local.tee $13
+                        (i32.sub
+                         (i32.const 0)
+                         (local.get $6)
+                        )
+                       )
+                       (i32.const 9)
+                      )
+                      (local.set $13
+                       (i32.const 9)
+                      )
+                     )
+                     (block $label$162
+                      (if
+                       (i32.lt_u
+                        (local.get $1)
+                        (local.get $5)
+                       )
+                       (block
+                        (local.set $29
+                         (i32.add
+                          (i32.shl
+                           (i32.const 1)
+                           (local.get $13)
+                          )
+                          (i32.const -1)
+                         )
+                        )
+                        (local.set $35
+                         (i32.shr_u
+                          (i32.const 1000000000)
+                          (local.get $13)
+                         )
+                        )
+                        (local.set $6
+                         (i32.const 0)
+                        )
+                        (local.set $8
+                         (local.get $1)
+                        )
+                        (loop $label$164
+                         (i32.store
+                          (local.get $8)
+                          (i32.add
+                           (i32.shr_u
+                            (local.tee $32
+                             (i32.load
+                              (local.get $8)
+                             )
+                            )
+                            (local.get $13)
+                           )
+                           (local.get $6)
+                          )
+                         )
+                         (local.set $6
+                          (i32.mul
+                           (i32.and
+                            (local.get $32)
+                            (local.get $29)
+                           )
+                           (local.get $35)
+                          )
+                         )
+                         (br_if $label$164
+                          (i32.lt_u
+                           (local.tee $8
+                            (i32.add
+                             (local.get $8)
+                             (i32.const 4)
+                            )
+                           )
+                           (local.get $5)
+                          )
+                         )
+                        )
+                        (local.set $8
+                         (i32.add
+                          (local.get $1)
+                          (i32.const 4)
+                         )
+                        )
+                        (if
+                         (i32.eqz
+                          (i32.load
+                           (local.get $1)
+                          )
+                         )
+                         (local.set $1
+                          (local.get $8)
+                         )
+                        )
+                        (br_if $label$162
+                         (i32.eqz
+                          (local.get $6)
+                         )
+                        )
+                        (i32.store
+                         (local.get $5)
+                         (local.get $6)
+                        )
+                        (local.set $5
+                         (i32.add
+                          (local.get $5)
+                          (i32.const 4)
+                         )
+                        )
+                       )
+                       (block
+                        (local.set $8
+                         (i32.add
+                          (local.get $1)
+                          (i32.const 4)
+                         )
+                        )
+                        (if
+                         (i32.eqz
+                          (i32.load
+                           (local.get $1)
+                          )
+                         )
+                         (local.set $1
+                          (local.get $8)
+                         )
+                        )
+                       )
+                      )
+                     )
+                     (local.set $6
+                      (i32.add
+                       (local.tee $8
+                        (if (result i32)
+                         (local.get $25)
+                         (local.get $7)
+                         (local.get $1)
+                        )
+                       )
+                       (i32.shl
+                        (local.get $14)
+                        (i32.const 2)
+                       )
+                      )
+                     )
+                     (if
+                      (i32.gt_s
+                       (i32.shr_s
+                        (i32.sub
+                         (local.get $5)
+                         (local.get $8)
+                        )
+                        (i32.const 2)
+                       )
+                       (local.get $14)
+                      )
+                      (local.set $5
+                       (local.get $6)
+                      )
+                     )
+                     (i32.store
+                      (local.get $20)
+                      (local.tee $6
+                       (i32.add
+                        (i32.load
+                         (local.get $20)
+                        )
+                        (local.get $13)
+                       )
+                      )
+                     )
+                     (br_if $label$160
+                      (i32.lt_s
+                       (local.get $6)
+                       (i32.const 0)
+                      )
+                     )
+                     (local.set $13
+                      (local.get $5)
+                     )
+                    )
+                   )
+                   (local.set $13
+                    (local.get $8)
+                   )
+                  )
+                  (local.set $25
+                   (local.get $7)
+                  )
+                  (block $label$172
+                   (if
+                    (i32.lt_u
+                     (local.get $1)
+                     (local.get $13)
+                    )
+                    (block
+                     (local.set $5
+                      (i32.mul
+                       (i32.shr_s
+                        (i32.sub
+                         (local.get $25)
+                         (local.get $1)
+                        )
+                        (i32.const 2)
+                       )
+                       (i32.const 9)
+                      )
+                     )
+                     (br_if $label$172
+                      (i32.lt_u
+                       (local.tee $6
+                        (i32.load
+                         (local.get $1)
+                        )
+                       )
+                       (i32.const 10)
+                      )
+                     )
+                     (local.set $8
+                      (i32.const 10)
+                     )
+                     (loop $label$174
+                      (local.set $5
+                       (i32.add
+                        (local.get $5)
+                        (i32.const 1)
+                       )
+                      )
+                      (br_if $label$174
+                       (i32.ge_u
+                        (local.get $6)
+                        (local.tee $8
+                         (i32.mul
+                          (local.get $8)
+                          (i32.const 10)
+                         )
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (local.set $5
+                     (i32.const 0)
+                    )
+                   )
+                  )
+                  (local.set $29
+                   (i32.eq
+                    (local.get $22)
+                    (i32.const 103)
+                   )
+                  )
+                  (local.set $35
+                   (i32.ne
+                    (local.get $18)
+                    (i32.const 0)
+                   )
+                  )
+                  (if
+                   (i32.lt_s
+                    (local.tee $8
+                     (i32.add
+                      (i32.sub
+                       (local.get $18)
+                       (if (result i32)
+                        (i32.ne
+                         (local.get $22)
+                         (i32.const 102)
+                        )
+                        (local.get $5)
+                        (i32.const 0)
+                       )
+                      )
+                      (i32.shr_s
+                       (i32.shl
+                        (i32.and
+                         (local.get $35)
+                         (local.get $29)
+                        )
+                        (i32.const 31)
+                       )
+                       (i32.const 31)
+                      )
+                     )
+                    )
+                    (i32.add
+                     (i32.mul
+                      (i32.shr_s
+                       (i32.sub
+                        (local.get $13)
+                        (local.get $25)
+                       )
+                       (i32.const 2)
+                      )
+                      (i32.const 9)
+                     )
+                     (i32.const -9)
+                    )
+                   )
+                   (block
+                    (if
+                     (i32.lt_s
+                      (local.tee $8
+                       (i32.add
+                        (i32.rem_s
+                         (local.tee $14
+                          (i32.add
+                           (local.get $8)
+                           (i32.const 9216)
+                          )
+                         )
+                         (i32.const 9)
+                        )
+                        (i32.const 1)
+                       )
+                      )
+                      (i32.const 9)
+                     )
+                     (block
+                      (local.set $6
+                       (i32.const 10)
+                      )
+                      (loop $label$180
+                       (local.set $6
+                        (i32.mul
+                         (local.get $6)
+                         (i32.const 10)
+                        )
+                       )
+                       (br_if $label$180
+                        (i32.ne
+                         (local.tee $8
+                          (i32.add
+                           (local.get $8)
+                           (i32.const 1)
+                          )
+                         )
+                         (i32.const 9)
+                        )
+                       )
+                      )
+                     )
+                     (local.set $6
+                      (i32.const 10)
+                     )
+                    )
+                    (local.set $14
+                     (i32.rem_u
+                      (local.tee $22
+                       (i32.load
+                        (local.tee $8
+                         (i32.add
+                          (i32.add
+                           (local.get $7)
+                           (i32.const 4)
+                          )
+                          (i32.shl
+                           (i32.add
+                            (i32.div_s
+                             (local.get $14)
+                             (i32.const 9)
+                            )
+                            (i32.const -1024)
+                           )
+                           (i32.const 2)
+                          )
+                         )
+                        )
+                       )
+                      )
+                      (local.get $6)
+                     )
+                    )
+                    (block $label$182
+                     (if
+                      (i32.eqz
+                       (i32.and
+                        (local.tee $32
+                         (i32.eq
+                          (i32.add
+                           (local.get $8)
+                           (i32.const 4)
+                          )
+                          (local.get $13)
+                         )
+                        )
+                        (i32.eqz
+                         (local.get $14)
+                        )
+                       )
+                      )
+                      (block
+                       (local.set $52
+                        (if (result f64)
+                         (i32.lt_u
+                          (local.get $14)
+                          (local.tee $49
+                           (i32.div_s
+                            (local.get $6)
+                            (i32.const 2)
+                           )
+                          )
+                         )
+                         (f64.const 0.5)
+                         (if (result f64)
+                          (i32.and
+                           (local.get $32)
+                           (i32.eq
+                            (local.get $14)
+                            (local.get $49)
+                           )
+                          )
+                          (f64.const 1)
+                          (f64.const 1.5)
+                         )
+                        )
+                       )
+                       (local.set $53
+                        (if (result f64)
+                         (i32.and
+                          (i32.div_u
+                           (local.get $22)
+                           (local.get $6)
+                          )
+                          (i32.const 1)
+                         )
+                         (f64.const 9007199254740994)
+                         (f64.const 9007199254740992)
+                        )
+                       )
+                       (block $label$190
+                        (if
+                         (local.get $24)
+                         (block
+                          (br_if $label$190
+                           (i32.ne
+                            (i32.load8_s
+                             (local.get $26)
+                            )
+                            (i32.const 45)
+                           )
+                          )
+                          (local.set $53
+                           (f64.neg
+                            (local.get $53)
+                           )
+                          )
+                          (local.set $52
+                           (f64.neg
+                            (local.get $52)
+                           )
+                          )
+                         )
+                        )
+                       )
+                       (i32.store
+                        (local.get $8)
+                        (local.tee $14
+                         (i32.sub
+                          (local.get $22)
+                          (local.get $14)
+                         )
+                        )
+                       )
+                       (br_if $label$182
+                        (f64.eq
+                         (f64.add
+                          (local.get $53)
+                          (local.get $52)
+                         )
+                         (local.get $53)
+                        )
+                       )
+                       (i32.store
+                        (local.get $8)
+                        (local.tee $5
+                         (i32.add
+                          (local.get $14)
+                          (local.get $6)
+                         )
+                        )
+                       )
+                       (if
+                        (i32.gt_u
+                         (local.get $5)
+                         (i32.const 999999999)
+                        )
+                        (loop $label$193
+                         (i32.store
+                          (local.get $8)
+                          (i32.const 0)
+                         )
+                         (if
+                          (i32.lt_u
+                           (local.tee $8
+                            (i32.add
+                             (local.get $8)
+                             (i32.const -4)
+                            )
+                           )
+                           (local.get $1)
+                          )
+                          (i32.store
+                           (local.tee $1
+                            (i32.add
+                             (local.get $1)
+                             (i32.const -4)
+                            )
+                           )
+                           (i32.const 0)
+                          )
+                         )
+                         (i32.store
+                          (local.get $8)
+                          (local.tee $5
+                           (i32.add
+                            (i32.load
+                             (local.get $8)
+                            )
+                            (i32.const 1)
+                           )
+                          )
+                         )
+                         (br_if $label$193
+                          (i32.gt_u
+                           (local.get $5)
+                           (i32.const 999999999)
+                          )
+                         )
+                        )
+                       )
+                       (local.set $5
+                        (i32.mul
+                         (i32.shr_s
+                          (i32.sub
+                           (local.get $25)
+                           (local.get $1)
+                          )
+                          (i32.const 2)
+                         )
+                         (i32.const 9)
+                        )
+                       )
+                       (br_if $label$182
+                        (i32.lt_u
+                         (local.tee $14
+                          (i32.load
+                           (local.get $1)
+                          )
+                         )
+                         (i32.const 10)
+                        )
+                       )
+                       (local.set $6
+                        (i32.const 10)
+                       )
+                       (loop $label$195
+                        (local.set $5
+                         (i32.add
+                          (local.get $5)
+                          (i32.const 1)
+                         )
+                        )
+                        (br_if $label$195
+                         (i32.ge_u
+                          (local.get $14)
+                          (local.tee $6
+                           (i32.mul
+                            (local.get $6)
+                            (i32.const 10)
+                           )
+                          )
+                         )
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (local.set $14
+                     (local.get $1)
+                    )
+                    (local.set $6
+                     (local.get $5)
+                    )
+                    (if
+                     (i32.le_u
+                      (local.get $13)
+                      (local.tee $8
+                       (i32.add
+                        (local.get $8)
+                        (i32.const 4)
+                       )
+                      )
+                     )
+                     (local.set $8
+                      (local.get $13)
+                     )
+                    )
+                   )
+                   (block
+                    (local.set $14
+                     (local.get $1)
+                    )
+                    (local.set $6
+                     (local.get $5)
+                    )
+                    (local.set $8
+                     (local.get $13)
+                    )
+                   )
+                  )
+                  (local.set $32
+                   (i32.sub
+                    (i32.const 0)
+                    (local.get $6)
+                   )
+                  )
+                  (loop $label$198
+                   (block $label$199
+                    (if
+                     (i32.le_u
+                      (local.get $8)
+                      (local.get $14)
+                     )
+                     (block
+                      (local.set $22
+                       (i32.const 0)
+                      )
+                      (br $label$199)
+                     )
+                    )
+                    (if
+                     (i32.load
+                      (local.tee $1
+                       (i32.add
+                        (local.get $8)
+                        (i32.const -4)
+                       )
+                      )
+                     )
+                     (local.set $22
+                      (i32.const 1)
+                     )
+                     (block
+                      (local.set $8
+                       (local.get $1)
+                      )
+                      (br $label$198)
+                     )
+                    )
+                   )
+                  )
+                  (block $label$203
+                   (if
+                    (local.get $29)
+                    (block
+                     (local.set $1
+                      (if (result i32)
+                       (i32.and
+                        (i32.gt_s
+                         (local.tee $1
+                          (i32.add
+                           (i32.xor
+                            (i32.and
+                             (local.get $35)
+                             (i32.const 1)
+                            )
+                            (i32.const 1)
+                           )
+                           (local.get $18)
+                          )
+                         )
+                         (local.get $6)
+                        )
+                        (i32.gt_s
+                         (local.get $6)
+                         (i32.const -5)
+                        )
+                       )
+                       (block (result i32)
+                        (local.set $5
+                         (i32.add
+                          (local.get $9)
+                          (i32.const -1)
+                         )
+                        )
+                        (i32.sub
+                         (i32.add
+                          (local.get $1)
+                          (i32.const -1)
+                         )
+                         (local.get $6)
+                        )
+                       )
+                       (block (result i32)
+                        (local.set $5
+                         (i32.add
+                          (local.get $9)
+                          (i32.const -2)
+                         )
+                        )
+                        (i32.add
+                         (local.get $1)
+                         (i32.const -1)
+                        )
+                       )
+                      )
+                     )
+                     (br_if $label$203
+                      (local.tee $13
+                       (i32.and
+                        (local.get $12)
+                        (i32.const 8)
+                       )
+                      )
+                     )
+                     (block $label$207
+                      (if
+                       (local.get $22)
+                       (block
+                        (if
+                         (i32.eqz
+                          (local.tee $18
+                           (i32.load
+                            (i32.add
+                             (local.get $8)
+                             (i32.const -4)
+                            )
+                           )
+                          )
+                         )
+                         (block
+                          (local.set $9
+                           (i32.const 9)
+                          )
+                          (br $label$207)
+                         )
+                        )
+                        (if
+                         (i32.rem_u
+                          (local.get $18)
+                          (i32.const 10)
+                         )
+                         (block
+                          (local.set $9
+                           (i32.const 0)
+                          )
+                          (br $label$207)
+                         )
+                         (block
+                          (local.set $13
+                           (i32.const 10)
+                          )
+                          (local.set $9
+                           (i32.const 0)
+                          )
+                         )
+                        )
+                        (loop $label$212
+                         (local.set $9
+                          (i32.add
+                           (local.get $9)
+                           (i32.const 1)
+                          )
+                         )
+                         (br_if $label$212
+                          (i32.eqz
+                           (i32.rem_u
+                            (local.get $18)
+                            (local.tee $13
+                             (i32.mul
+                              (local.get $13)
+                              (i32.const 10)
+                             )
+                            )
+                           )
+                          )
+                         )
+                        )
+                       )
+                       (local.set $9
+                        (i32.const 9)
+                       )
+                      )
+                     )
+                     (local.set $18
+                      (i32.add
+                       (i32.mul
+                        (i32.shr_s
+                         (i32.sub
+                          (local.get $8)
+                          (local.get $25)
+                         )
+                         (i32.const 2)
+                        )
+                        (i32.const 9)
+                       )
+                       (i32.const -9)
+                      )
+                     )
+                     (if
+                      (i32.eq
+                       (i32.or
+                        (local.get $5)
+                        (i32.const 32)
+                       )
+                       (i32.const 102)
+                      )
+                      (block
+                       (local.set $13
+                        (i32.const 0)
+                       )
+                       (if
+                        (i32.ge_s
+                         (local.get $1)
+                         (if (result i32)
+                          (i32.lt_s
+                           (local.tee $9
+                            (i32.sub
+                             (local.get $18)
+                             (local.get $9)
+                            )
+                           )
+                           (i32.const 0)
+                          )
+                          (local.tee $9
+                           (i32.const 0)
+                          )
+                          (local.get $9)
+                         )
+                        )
+                        (local.set $1
+                         (local.get $9)
+                        )
+                       )
+                      )
+                      (block
+                       (local.set $13
+                        (i32.const 0)
+                       )
+                       (if
+                        (i32.ge_s
+                         (local.get $1)
+                         (if (result i32)
+                          (i32.lt_s
+                           (local.tee $9
+                            (i32.sub
+                             (i32.add
+                              (local.get $18)
+                              (local.get $6)
+                             )
+                             (local.get $9)
+                            )
+                           )
+                           (i32.const 0)
+                          )
+                          (local.tee $9
+                           (i32.const 0)
+                          )
+                          (local.get $9)
+                         )
+                        )
+                        (local.set $1
+                         (local.get $9)
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (block
+                     (local.set $13
+                      (i32.and
+                       (local.get $12)
+                       (i32.const 8)
+                      )
+                     )
+                     (local.set $1
+                      (local.get $18)
+                     )
+                     (local.set $5
+                      (local.get $9)
+                     )
+                    )
+                   )
+                  )
+                  (if
+                   (local.tee $25
+                    (i32.eq
+                     (i32.or
+                      (local.get $5)
+                      (i32.const 32)
+                     )
+                     (i32.const 102)
+                    )
+                   )
+                   (block
+                    (local.set $9
+                     (i32.const 0)
+                    )
+                    (if
+                     (i32.le_s
+                      (local.get $6)
+                      (i32.const 0)
+                     )
+                     (local.set $6
+                      (i32.const 0)
+                     )
+                    )
+                   )
+                   (block
+                    (if
+                     (i32.lt_s
+                      (i32.sub
+                       (local.get $28)
+                       (local.tee $9
+                        (call $23
+                         (i64.extend_i32_s
+                          (if (result i32)
+                           (i32.lt_s
+                            (local.get $6)
+                            (i32.const 0)
+                           )
+                           (local.get $32)
+                           (local.get $6)
+                          )
+                         )
+                         (local.get $33)
+                        )
+                       )
+                      )
+                      (i32.const 2)
+                     )
+                     (loop $label$229
+                      (i32.store8
+                       (local.tee $9
+                        (i32.add
+                         (local.get $9)
+                         (i32.const -1)
+                        )
+                       )
+                       (i32.const 48)
+                      )
+                      (br_if $label$229
+                       (i32.lt_s
+                        (i32.sub
+                         (local.get $28)
+                         (local.get $9)
+                        )
+                        (i32.const 2)
+                       )
+                      )
+                     )
+                    )
+                    (i32.store8
+                     (i32.add
+                      (local.get $9)
+                      (i32.const -1)
+                     )
+                     (i32.add
+                      (i32.and
+                       (i32.shr_s
+                        (local.get $6)
+                        (i32.const 31)
+                       )
+                       (i32.const 2)
+                      )
+                      (i32.const 43)
+                     )
+                    )
+                    (i32.store8
+                     (local.tee $6
+                      (i32.add
+                       (local.get $9)
+                       (i32.const -2)
+                      )
+                     )
+                     (local.get $5)
+                    )
+                    (local.set $9
+                     (local.get $6)
+                    )
+                    (local.set $6
+                     (i32.sub
+                      (local.get $28)
+                      (local.get $6)
+                     )
+                    )
+                   )
+                  )
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.tee $18
+                    (i32.add
+                     (i32.add
+                      (i32.add
+                       (i32.add
+                        (local.get $24)
+                        (i32.const 1)
+                       )
+                       (local.get $1)
+                      )
+                      (i32.ne
+                       (local.tee $29
+                        (i32.or
+                         (local.get $1)
+                         (local.get $13)
+                        )
+                       )
+                       (i32.const 0)
+                      )
+                     )
+                     (local.get $6)
+                    )
+                   )
+                   (local.get $12)
+                  )
+                  (if
+                   (i32.eqz
+                    (i32.and
+                     (i32.load
+                      (local.get $0)
+                     )
+                     (i32.const 32)
+                    )
+                   )
+                   (drop
+                    (call $21
+                     (local.get $26)
+                     (local.get $24)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (call $25
+                   (local.get $0)
+                   (i32.const 48)
+                   (local.get $10)
+                   (local.get $18)
+                   (i32.xor
+                    (local.get $12)
+                    (i32.const 65536)
+                   )
+                  )
+                  (block $label$231
+                   (if
+                    (local.get $25)
+                    (block
+                     (local.set $6
+                      (local.tee $9
+                       (if (result i32)
+                        (i32.gt_u
+                         (local.get $14)
+                         (local.get $7)
+                        )
+                        (local.get $7)
+                        (local.get $14)
+                       )
+                      )
+                     )
+                     (loop $label$235
+                      (local.set $5
+                       (call $23
+                        (i64.extend_i32_u
+                         (i32.load
+                          (local.get $6)
+                         )
+                        )
+                        (local.get $31)
+                       )
+                      )
+                      (block $label$236
+                       (if
+                        (i32.eq
+                         (local.get $6)
+                         (local.get $9)
+                        )
+                        (block
+                         (br_if $label$236
+                          (i32.ne
+                           (local.get $5)
+                           (local.get $31)
+                          )
+                         )
+                         (i32.store8
+                          (local.get $34)
+                          (i32.const 48)
+                         )
+                         (local.set $5
+                          (local.get $34)
+                         )
+                        )
+                        (block
+                         (br_if $label$236
+                          (i32.le_u
+                           (local.get $5)
+                           (local.get $19)
+                          )
+                         )
+                         (drop
+                          (call $39
+                           (local.get $19)
+                           (i32.const 48)
+                           (i32.sub
+                            (local.get $5)
+                            (local.get $27)
+                           )
+                          )
+                         )
+                         (loop $label$239
+                          (br_if $label$239
+                           (i32.gt_u
+                            (local.tee $5
+                             (i32.add
+                              (local.get $5)
+                              (i32.const -1)
+                             )
+                            )
+                            (local.get $19)
+                           )
+                          )
+                         )
+                        )
+                       )
+                      )
+                      (if
+                       (i32.eqz
+                        (i32.and
+                         (i32.load
+                          (local.get $0)
+                         )
+                         (i32.const 32)
+                        )
+                       )
+                       (drop
+                        (call $21
+                         (local.get $5)
+                         (i32.sub
+                          (local.get $41)
+                          (local.get $5)
+                         )
+                         (local.get $0)
+                        )
+                       )
+                      )
+                      (if
+                       (i32.le_u
+                        (local.tee $5
+                         (i32.add
+                          (local.get $6)
+                          (i32.const 4)
+                         )
+                        )
+                        (local.get $7)
+                       )
+                       (block
+                        (local.set $6
+                         (local.get $5)
+                        )
+                        (br $label$235)
+                       )
+                      )
+                     )
+                     (block $label$242
+                      (if
+                       (local.get $29)
+                       (block
+                        (br_if $label$242
+                         (i32.and
+                          (i32.load
+                           (local.get $0)
+                          )
+                          (i32.const 32)
+                         )
+                        )
+                        (drop
+                         (call $21
+                          (i32.const 1709)
+                          (i32.const 1)
+                          (local.get $0)
+                         )
+                        )
+                       )
+                      )
+                     )
+                     (if
+                      (i32.and
+                       (i32.gt_s
+                        (local.get $1)
+                        (i32.const 0)
+                       )
+                       (i32.lt_u
+                        (local.get $5)
+                        (local.get $8)
+                       )
+                      )
+                      (loop $label$245
+                       (if
+                        (i32.gt_u
+                         (local.tee $7
+                          (call $23
+                           (i64.extend_i32_u
+                            (i32.load
+                             (local.get $5)
+                            )
+                           )
+                           (local.get $31)
+                          )
+                         )
+                         (local.get $19)
+                        )
+                        (block
+                         (drop
+                          (call $39
+                           (local.get $19)
+                           (i32.const 48)
+                           (i32.sub
+                            (local.get $7)
+                            (local.get $27)
+                           )
+                          )
+                         )
+                         (loop $label$247
+                          (br_if $label$247
+                           (i32.gt_u
+                            (local.tee $7
+                             (i32.add
+                              (local.get $7)
+                              (i32.const -1)
+                             )
+                            )
+                            (local.get $19)
+                           )
+                          )
+                         )
+                        )
+                       )
+                       (if
+                        (i32.eqz
+                         (i32.and
+                          (i32.load
+                           (local.get $0)
+                          )
+                          (i32.const 32)
+                         )
+                        )
+                        (drop
+                         (call $21
+                          (local.get $7)
+                          (if (result i32)
+                           (i32.gt_s
+                            (local.get $1)
+                            (i32.const 9)
+                           )
+                           (i32.const 9)
+                           (local.get $1)
+                          )
+                          (local.get $0)
+                         )
+                        )
+                       )
+                       (local.set $7
+                        (i32.add
+                         (local.get $1)
+                         (i32.const -9)
+                        )
+                       )
+                       (if
+                        (i32.and
+                         (i32.gt_s
+                          (local.get $1)
+                          (i32.const 9)
+                         )
+                         (i32.lt_u
+                          (local.tee $5
+                           (i32.add
+                            (local.get $5)
+                            (i32.const 4)
+                           )
+                          )
+                          (local.get $8)
+                         )
+                        )
+                        (block
+                         (local.set $1
+                          (local.get $7)
+                         )
+                         (br $label$245)
+                        )
+                        (local.set $1
+                         (local.get $7)
+                        )
+                       )
+                      )
+                     )
+                     (call $25
+                      (local.get $0)
+                      (i32.const 48)
+                      (i32.add
+                       (local.get $1)
+                       (i32.const 9)
+                      )
+                      (i32.const 9)
+                      (i32.const 0)
+                     )
+                    )
+                    (block
+                     (local.set $5
+                      (i32.add
+                       (local.get $14)
+                       (i32.const 4)
+                      )
+                     )
+                     (if
+                      (i32.eqz
+                       (local.get $22)
+                      )
+                      (local.set $8
+                       (local.get $5)
+                      )
+                     )
+                     (if
+                      (i32.gt_s
+                       (local.get $1)
+                       (i32.const -1)
+                      )
+                      (block
+                       (local.set $13
+                        (i32.eqz
+                         (local.get $13)
+                        )
+                       )
+                       (local.set $7
+                        (local.get $14)
+                       )
+                       (local.set $5
+                        (local.get $1)
+                       )
+                       (loop $label$256
+                        (if
+                         (i32.eq
+                          (local.tee $1
+                           (call $23
+                            (i64.extend_i32_u
+                             (i32.load
+                              (local.get $7)
+                             )
+                            )
+                            (local.get $31)
+                           )
+                          )
+                          (local.get $31)
+                         )
+                         (block
+                          (i32.store8
+                           (local.get $34)
+                           (i32.const 48)
+                          )
+                          (local.set $1
+                           (local.get $34)
+                          )
+                         )
+                        )
+                        (block $label$258
+                         (if
+                          (i32.eq
+                           (local.get $7)
+                           (local.get $14)
+                          )
+                          (block
+                           (if
+                            (i32.eqz
+                             (i32.and
+                              (i32.load
+                               (local.get $0)
+                              )
+                              (i32.const 32)
+                             )
+                            )
+                            (drop
+                             (call $21
+                              (local.get $1)
+                              (i32.const 1)
+                              (local.get $0)
+                             )
+                            )
+                           )
+                           (local.set $1
+                            (i32.add
+                             (local.get $1)
+                             (i32.const 1)
+                            )
+                           )
+                           (br_if $label$258
+                            (i32.and
+                             (local.get $13)
+                             (i32.lt_s
+                              (local.get $5)
+                              (i32.const 1)
+                             )
+                            )
+                           )
+                           (br_if $label$258
+                            (i32.and
+                             (i32.load
+                              (local.get $0)
+                             )
+                             (i32.const 32)
+                            )
+                           )
+                           (drop
+                            (call $21
+                             (i32.const 1709)
+                             (i32.const 1)
+                             (local.get $0)
+                            )
+                           )
+                          )
+                          (block
+                           (br_if $label$258
+                            (i32.le_u
+                             (local.get $1)
+                             (local.get $19)
+                            )
+                           )
+                           (drop
+                            (call $39
+                             (local.get $19)
+                             (i32.const 48)
+                             (i32.add
+                              (local.get $1)
+                              (local.get $43)
+                             )
+                            )
+                           )
+                           (loop $label$262
+                            (br_if $label$262
+                             (i32.gt_u
+                              (local.tee $1
+                               (i32.add
+                                (local.get $1)
+                                (i32.const -1)
+                               )
+                              )
+                              (local.get $19)
+                             )
+                            )
+                           )
+                          )
+                         )
+                        )
+                        (local.set $6
+                         (i32.sub
+                          (local.get $41)
+                          (local.get $1)
+                         )
+                        )
+                        (if
+                         (i32.eqz
+                          (i32.and
+                           (i32.load
+                            (local.get $0)
+                           )
+                           (i32.const 32)
+                          )
+                         )
+                         (drop
+                          (call $21
+                           (local.get $1)
+                           (if (result i32)
+                            (i32.gt_s
+                             (local.get $5)
+                             (local.get $6)
+                            )
+                            (local.get $6)
+                            (local.get $5)
+                           )
+                           (local.get $0)
+                          )
+                         )
+                        )
+                        (br_if $label$256
+                         (i32.and
+                          (i32.lt_u
+                           (local.tee $7
+                            (i32.add
+                             (local.get $7)
+                             (i32.const 4)
+                            )
+                           )
+                           (local.get $8)
+                          )
+                          (i32.gt_s
+                           (local.tee $5
+                            (i32.sub
+                             (local.get $5)
+                             (local.get $6)
+                            )
+                           )
+                           (i32.const -1)
+                          )
+                         )
+                        )
+                        (local.set $1
+                         (local.get $5)
+                        )
+                       )
+                      )
+                     )
+                     (call $25
+                      (local.get $0)
+                      (i32.const 48)
+                      (i32.add
+                       (local.get $1)
+                       (i32.const 18)
+                      )
+                      (i32.const 18)
+                      (i32.const 0)
+                     )
+                     (br_if $label$231
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $21
+                       (local.get $9)
+                       (i32.sub
+                        (local.get $28)
+                        (local.get $9)
+                       )
+                       (local.get $0)
+                      )
+                     )
+                    )
+                   )
+                  )
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.get $18)
+                   (i32.xor
+                    (local.get $12)
+                    (i32.const 8192)
+                   )
+                  )
+                  (if
+                   (i32.ge_s
+                    (local.get $18)
+                    (local.get $10)
+                   )
+                   (local.set $10
+                    (local.get $18)
+                   )
+                  )
+                 )
+                 (block
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.tee $8
+                    (i32.add
+                     (if (result i32)
+                      (local.tee $6
+                       (i32.or
+                        (f64.ne
+                         (local.get $52)
+                         (local.get $52)
+                        )
+                        (i32.const 0)
+                       )
+                      )
+                      (local.tee $24
+                       (i32.const 0)
+                      )
+                      (local.get $24)
+                     )
+                     (i32.const 3)
+                    )
+                   )
+                   (local.get $7)
+                  )
+                  (if
+                   (i32.eqz
+                    (i32.and
+                     (local.tee $1
+                      (i32.load
+                       (local.get $0)
+                      )
+                     )
+                     (i32.const 32)
+                    )
+                   )
+                   (block
+                    (drop
+                     (call $21
+                      (local.get $26)
+                      (local.get $24)
+                      (local.get $0)
+                     )
+                    )
+                    (local.set $1
+                     (i32.load
+                      (local.get $0)
+                     )
+                    )
+                   )
+                  )
+                  (local.set $7
+                   (if (result i32)
+                    (local.tee $5
+                     (i32.ne
+                      (i32.and
+                       (local.get $9)
+                       (i32.const 32)
+                      )
+                      (i32.const 0)
+                     )
+                    )
+                    (i32.const 1693)
+                    (i32.const 1697)
+                   )
+                  )
+                  (local.set $5
+                   (if (result i32)
+                    (local.get $5)
+                    (i32.const 1701)
+                    (i32.const 1705)
+                   )
+                  )
+                  (if
+                   (i32.eqz
+                    (local.get $6)
+                   )
+                   (local.set $5
+                    (local.get $7)
+                   )
+                  )
+                  (if
+                   (i32.eqz
+                    (i32.and
+                     (local.get $1)
+                     (i32.const 32)
+                    )
+                   )
+                   (drop
+                    (call $21
+                     (local.get $5)
+                     (i32.const 3)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.get $8)
+                   (i32.xor
+                    (local.get $12)
+                    (i32.const 8192)
+                   )
+                  )
+                  (if
+                   (i32.ge_s
+                    (local.get $8)
+                    (local.get $10)
+                   )
+                   (local.set $10
+                    (local.get $8)
+                   )
+                  )
+                 )
+                )
+               )
+               (local.set $1
+                (local.get $11)
+               )
+               (br $label$4)
+              )
+              (local.set $7
+               (local.get $5)
+              )
+              (local.set $6
+               (i32.const 0)
+              )
+              (local.set $8
+               (i32.const 1657)
+              )
+              (local.set $5
+               (local.get $21)
+              )
+              (br $label$70)
+             )
+             (local.set $7
+              (i32.and
+               (local.get $9)
+               (i32.const 32)
+              )
+             )
+             (local.set $7
+              (if (result i32)
+               (i64.eq
+                (local.tee $50
+                 (i64.load
+                  (local.get $16)
+                 )
+                )
+                (i64.const 0)
+               )
+               (block (result i32)
+                (local.set $50
+                 (i64.const 0)
+                )
+                (local.get $21)
+               )
+               (block (result i32)
+                (local.set $1
+                 (local.get $21)
+                )
+                (loop $label$280
+                 (i32.store8
+                  (local.tee $1
+                   (i32.add
+                    (local.get $1)
+                    (i32.const -1)
+                   )
+                  )
+                  (i32.or
+                   (i32.load8_u
+                    (i32.add
+                     (i32.and
+                      (i32.wrap_i64
+                       (local.get $50)
+                      )
+                      (i32.const 15)
+                     )
+                     (i32.const 1641)
+                    )
+                   )
+                   (local.get $7)
+                  )
+                 )
+                 (br_if $label$280
+                  (i64.ne
+                   (local.tee $50
+                    (i64.shr_u
+                     (local.get $50)
+                     (i64.const 4)
+                    )
+                   )
+                   (i64.const 0)
+                  )
+                 )
+                )
+                (local.set $50
+                 (i64.load
+                  (local.get $16)
+                 )
+                )
+                (local.get $1)
+               )
+              )
+             )
+             (local.set $8
+              (i32.add
+               (i32.shr_s
+                (local.get $9)
+                (i32.const 4)
+               )
+               (i32.const 1657)
+              )
+             )
+             (if
+              (local.tee $1
+               (i32.or
+                (i32.eqz
+                 (i32.and
+                  (local.get $12)
+                  (i32.const 8)
+                 )
+                )
+                (i64.eq
+                 (local.get $50)
+                 (i64.const 0)
+                )
+               )
+              )
+              (local.set $8
+               (i32.const 1657)
+              )
+             )
+             (local.set $6
+              (if (result i32)
+               (local.get $1)
+               (i32.const 0)
+               (i32.const 2)
+              )
+             )
+             (br $label$71)
+            )
+            (local.set $7
+             (call $23
+              (local.get $50)
+              (local.get $21)
+             )
+            )
+            (br $label$71)
+           )
+           (local.set $14
+            (i32.eqz
+             (local.tee $13
+              (call $17
+               (local.get $1)
+               (i32.const 0)
+               (local.get $5)
+              )
+             )
+            )
+           )
+           (local.set $8
+            (i32.sub
+             (local.get $13)
+             (local.get $1)
+            )
+           )
+           (local.set $9
+            (i32.add
+             (local.get $1)
+             (local.get $5)
+            )
+           )
+           (local.set $12
+            (local.get $7)
+           )
+           (local.set $7
+            (if (result i32)
+             (local.get $14)
+             (local.get $5)
+             (local.get $8)
+            )
+           )
+           (local.set $6
+            (i32.const 0)
+           )
+           (local.set $8
+            (i32.const 1657)
+           )
+           (local.set $5
+            (if (result i32)
+             (local.get $14)
+             (local.get $9)
+             (local.get $13)
+            )
+           )
+           (br $label$70)
+          )
+          (local.set $1
+           (i32.const 0)
+          )
+          (local.set $5
+           (i32.const 0)
+          )
+          (local.set $8
+           (local.get $7)
+          )
+          (loop $label$288
+           (block $label$289
+            (br_if $label$289
+             (i32.eqz
+              (local.tee $9
+               (i32.load
+                (local.get $8)
+               )
+              )
+             )
+            )
+            (br_if $label$289
+             (i32.or
+              (i32.lt_s
+               (local.tee $5
+                (call $26
+                 (local.get $36)
+                 (local.get $9)
+                )
+               )
+               (i32.const 0)
+              )
+              (i32.gt_u
+               (local.get $5)
+               (i32.sub
+                (local.get $6)
+                (local.get $1)
+               )
+              )
+             )
+            )
+            (local.set $8
+             (i32.add
+              (local.get $8)
+              (i32.const 4)
+             )
+            )
+            (br_if $label$288
+             (i32.gt_u
+              (local.get $6)
+              (local.tee $1
+               (i32.add
+                (local.get $5)
+                (local.get $1)
+               )
+              )
+             )
+            )
+           )
+          )
+          (if
+           (i32.lt_s
+            (local.get $5)
+            (i32.const 0)
+           )
+           (block
+            (local.set $15
+             (i32.const -1)
+            )
+            (br $label$5)
+           )
+          )
+          (call $25
+           (local.get $0)
+           (i32.const 32)
+           (local.get $10)
+           (local.get $1)
+           (local.get $12)
+          )
+          (if
+           (local.get $1)
+           (block
+            (local.set $5
+             (i32.const 0)
+            )
+            (loop $label$292
+             (br_if $label$72
+              (i32.eqz
+               (local.tee $8
+                (i32.load
+                 (local.get $7)
+                )
+               )
+              )
+             )
+             (br_if $label$72
+              (i32.gt_s
+               (local.tee $5
+                (i32.add
+                 (local.tee $8
+                  (call $26
+                   (local.get $36)
+                   (local.get $8)
+                  )
+                 )
+                 (local.get $5)
+                )
+               )
+               (local.get $1)
+              )
+             )
+             (if
+              (i32.eqz
+               (i32.and
+                (i32.load
+                 (local.get $0)
+                )
+                (i32.const 32)
+               )
+              )
+              (drop
+               (call $21
+                (local.get $36)
+                (local.get $8)
+                (local.get $0)
+               )
+              )
+             )
+             (local.set $7
+              (i32.add
+               (local.get $7)
+               (i32.const 4)
+              )
+             )
+             (br_if $label$292
+              (i32.lt_u
+               (local.get $5)
+               (local.get $1)
+              )
+             )
+             (br $label$72)
+            )
+           )
+           (block
+            (local.set $1
+             (i32.const 0)
+            )
+            (br $label$72)
+           )
+          )
+         )
+         (call $25
+          (local.get $0)
+          (i32.const 32)
+          (local.get $10)
+          (local.get $1)
+          (i32.xor
+           (local.get $12)
+           (i32.const 8192)
+          )
+         )
+         (if
+          (i32.le_s
+           (local.get $10)
+           (local.get $1)
+          )
+          (local.set $10
+           (local.get $1)
+          )
+         )
+         (local.set $1
+          (local.get $11)
+         )
+         (br $label$4)
+        )
+        (local.set $1
+         (i32.and
+          (local.get $12)
+          (i32.const -65537)
+         )
+        )
+        (if
+         (i32.gt_s
+          (local.get $5)
+          (i32.const -1)
+         )
+         (local.set $12
+          (local.get $1)
+         )
+        )
+        (local.set $5
+         (if (result i32)
+          (i32.or
+           (local.get $5)
+           (local.tee $9
+            (i64.ne
+             (i64.load
+              (local.get $16)
+             )
+             (i64.const 0)
+            )
+           )
+          )
+          (block (result i32)
+           (local.set $1
+            (local.get $7)
+           )
+           (if
+            (i32.gt_s
+             (local.get $5)
+             (local.tee $7
+              (i32.add
+               (i32.xor
+                (i32.and
+                 (local.get $9)
+                 (i32.const 1)
+                )
+                (i32.const 1)
+               )
+               (i32.sub
+                (local.get $38)
+                (local.get $7)
+               )
+              )
+             )
+            )
+            (local.set $7
+             (local.get $5)
+            )
+           )
+           (local.get $21)
+          )
+          (block (result i32)
+           (local.set $1
+            (local.get $21)
+           )
+           (local.set $7
+            (i32.const 0)
+           )
+           (local.get $21)
+          )
+         )
+        )
+       )
+       (call $25
+        (local.get $0)
+        (i32.const 32)
+        (if (result i32)
+         (i32.lt_s
+          (local.get $10)
+          (local.tee $5
+           (i32.add
+            (if (result i32)
+             (i32.lt_s
+              (local.get $7)
+              (local.tee $9
+               (i32.sub
+                (local.get $5)
+                (local.get $1)
+               )
+              )
+             )
+             (local.tee $7
+              (local.get $9)
+             )
+             (local.get $7)
+            )
+            (local.get $6)
+           )
+          )
+         )
+         (local.tee $10
+          (local.get $5)
+         )
+         (local.get $10)
+        )
+        (local.get $5)
+        (local.get $12)
+       )
+       (if
+        (i32.eqz
+         (i32.and
+          (i32.load
+           (local.get $0)
+          )
+          (i32.const 32)
+         )
+        )
+        (drop
+         (call $21
+          (local.get $8)
+          (local.get $6)
+          (local.get $0)
+         )
+        )
+       )
+       (call $25
+        (local.get $0)
+        (i32.const 48)
+        (local.get $10)
+        (local.get $5)
+        (i32.xor
+         (local.get $12)
+         (i32.const 65536)
+        )
+       )
+       (call $25
+        (local.get $0)
+        (i32.const 48)
+        (local.get $7)
+        (local.get $9)
+        (i32.const 0)
+       )
+       (if
+        (i32.eqz
+         (i32.and
+          (i32.load
+           (local.get $0)
+          )
+          (i32.const 32)
+         )
+        )
+        (drop
+         (call $21
+          (local.get $1)
+          (local.get $9)
+          (local.get $0)
+         )
+        )
+       )
+       (call $25
+        (local.get $0)
+        (i32.const 32)
+        (local.get $10)
+        (local.get $5)
+        (i32.xor
+         (local.get $12)
+         (i32.const 8192)
+        )
+       )
+       (local.set $1
+        (local.get $11)
+       )
+       (br $label$4)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.eqz
+      (local.get $0)
+     )
+     (if
+      (local.get $17)
+      (block
+       (local.set $0
+        (i32.const 1)
+       )
+       (loop $label$308
+        (if
+         (local.tee $1
+          (i32.load
+           (i32.add
+            (local.get $4)
+            (i32.shl
+             (local.get $0)
+             (i32.const 2)
+            )
+           )
+          )
+         )
+         (block
+          (call $22
+           (i32.add
+            (local.get $3)
+            (i32.shl
+             (local.get $0)
+             (i32.const 3)
+            )
+           )
+           (local.get $1)
+           (local.get $2)
+          )
+          (br_if $label$308
+           (i32.lt_s
+            (local.tee $0
+             (i32.add
+              (local.get $0)
+              (i32.const 1)
+             )
+            )
+            (i32.const 10)
+           )
+          )
+          (local.set $15
+           (i32.const 1)
+          )
+          (br $label$2)
+         )
+        )
+       )
+       (loop $label$310
+        (if
+         (i32.load
+          (i32.add
+           (local.get $4)
+           (i32.shl
+            (local.get $0)
+            (i32.const 2)
+           )
+          )
+         )
+         (block
+          (local.set $15
+           (i32.const -1)
+          )
+          (br $label$2)
+         )
+        )
+        (br_if $label$310
+         (i32.lt_s
+          (local.tee $0
+           (i32.add
+            (local.get $0)
+            (i32.const 1)
+           )
+          )
+          (i32.const 10)
+         )
+        )
+        (local.set $15
+         (i32.const 1)
+        )
+       )
+      )
+      (local.set $15
+       (i32.const 0)
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $23)
+   )
+   (local.get $15)
+  )
+ )
+ (func $20 (; 33 ;) (type $1) (param $0 i32) (result i32)
+  (i32.const 0)
+ )
+ (func $21 (; 34 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (block $label$1 (result i32)
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (local.tee $3
+       (i32.load
+        (local.tee $4
+         (i32.add
+          (local.get $2)
+          (i32.const 16)
+         )
+        )
+       )
+      )
+     )
+     (if
+      (call $30
+       (local.get $2)
+      )
+      (local.set $3
+       (i32.const 0)
+      )
+      (block
+       (local.set $3
+        (i32.load
+         (local.get $4)
+        )
+       )
+       (br $label$3)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.lt_u
+      (i32.sub
+       (local.get $3)
+       (local.tee $4
+        (i32.load
+         (local.tee $5
+          (i32.add
+           (local.get $2)
+           (i32.const 20)
+          )
+         )
+        )
+       )
+      )
+      (local.get $1)
+     )
+     (block
+      (local.set $3
+       (call_indirect (type $0)
+        (local.get $2)
+        (local.get $0)
+        (local.get $1)
+        (i32.add
+         (i32.and
+          (i32.load offset=36
+           (local.get $2)
+          )
+          (i32.const 3)
+         )
+         (i32.const 2)
+        )
+       )
+      )
+      (br $label$2)
+     )
+    )
+    (local.set $2
+     (block $label$7 (result i32)
+      (if (result i32)
+       (i32.gt_s
+        (i32.load8_s offset=75
+         (local.get $2)
+        )
+        (i32.const -1)
+       )
+       (block (result i32)
+        (local.set $3
+         (local.get $1)
+        )
+        (loop $label$9
+         (drop
+          (br_if $label$7
+           (i32.const 0)
+           (i32.eqz
+            (local.get $3)
+           )
+          )
+         )
+         (if
+          (i32.ne
+           (i32.load8_s
+            (i32.add
+             (local.get $0)
+             (local.tee $6
+              (i32.add
+               (local.get $3)
+               (i32.const -1)
+              )
+             )
+            )
+           )
+           (i32.const 10)
+          )
+          (block
+           (local.set $3
+            (local.get $6)
+           )
+           (br $label$9)
+          )
+         )
+        )
+        (br_if $label$2
+         (i32.lt_u
+          (call_indirect (type $0)
+           (local.get $2)
+           (local.get $0)
+           (local.get $3)
+           (i32.add
+            (i32.and
+             (i32.load offset=36
+              (local.get $2)
+             )
+             (i32.const 3)
+            )
+            (i32.const 2)
+           )
+          )
+          (local.get $3)
+         )
+        )
+        (local.set $4
+         (i32.load
+          (local.get $5)
+         )
+        )
+        (local.set $1
+         (i32.sub
+          (local.get $1)
+          (local.get $3)
+         )
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (local.get $3)
+         )
+        )
+        (local.get $3)
+       )
+       (i32.const 0)
+      )
+     )
+    )
+    (drop
+     (call $40
+      (local.get $4)
+      (local.get $0)
+      (local.get $1)
+     )
+    )
+    (i32.store
+     (local.get $5)
+     (i32.add
+      (i32.load
+       (local.get $5)
+      )
+      (local.get $1)
+     )
+    )
+    (local.set $3
+     (i32.add
+      (local.get $2)
+      (local.get $1)
+     )
+    )
+   )
+   (local.get $3)
+  )
+ )
+ (func $22 (; 35 ;) (type $8) (param $0 i32) (param $1 i32) (param $2 i32)
+  (local $3 i32)
+  (local $4 i64)
+  (local $5 f64)
+  (block $label$1
+   (if
+    (i32.le_u
+     (local.get $1)
+     (i32.const 20)
+    )
+    (block $label$3
+     (block $label$4
+      (block $label$5
+       (block $label$6
+        (block $label$7
+         (block $label$8
+          (block $label$9
+           (block $label$10
+            (block $label$11
+             (block $label$12
+              (block $label$13
+               (br_table $label$13 $label$12 $label$11 $label$10 $label$9 $label$8 $label$7 $label$6 $label$5 $label$4 $label$3
+                (i32.sub
+                 (local.get $1)
+                 (i32.const 9)
+                )
+               )
+              )
+              (local.set $3
+               (i32.load
+                (local.tee $1
+                 (i32.and
+                  (i32.add
+                   (i32.load
+                    (local.get $2)
+                   )
+                   (i32.const 3)
+                  )
+                  (i32.const -4)
+                 )
+                )
+               )
+              )
+              (i32.store
+               (local.get $2)
+               (i32.add
+                (local.get $1)
+                (i32.const 4)
+               )
+              )
+              (i32.store
+               (local.get $0)
+               (local.get $3)
+              )
+              (br $label$1)
+             )
+             (local.set $3
+              (i32.load
+               (local.tee $1
+                (i32.and
+                 (i32.add
+                  (i32.load
+                   (local.get $2)
+                  )
+                  (i32.const 3)
+                 )
+                 (i32.const -4)
+                )
+               )
+              )
+             )
+             (i32.store
+              (local.get $2)
+              (i32.add
+               (local.get $1)
+               (i32.const 4)
+              )
+             )
+             (i64.store
+              (local.get $0)
+              (i64.extend_i32_s
+               (local.get $3)
+              )
+             )
+             (br $label$1)
+            )
+            (local.set $3
+             (i32.load
+              (local.tee $1
+               (i32.and
+                (i32.add
+                 (i32.load
+                  (local.get $2)
+                 )
+                 (i32.const 3)
+                )
+                (i32.const -4)
+               )
+              )
+             )
+            )
+            (i32.store
+             (local.get $2)
+             (i32.add
+              (local.get $1)
+              (i32.const 4)
+             )
+            )
+            (i64.store
+             (local.get $0)
+             (i64.extend_i32_u
+              (local.get $3)
+             )
+            )
+            (br $label$1)
+           )
+           (local.set $4
+            (i64.load
+             (local.tee $1
+              (i32.and
+               (i32.add
+                (i32.load
+                 (local.get $2)
+                )
+                (i32.const 7)
+               )
+               (i32.const -8)
+              )
+             )
+            )
+           )
+           (i32.store
+            (local.get $2)
+            (i32.add
+             (local.get $1)
+             (i32.const 8)
+            )
+           )
+           (i64.store
+            (local.get $0)
+            (local.get $4)
+           )
+           (br $label$1)
+          )
+          (local.set $3
+           (i32.load
+            (local.tee $1
+             (i32.and
+              (i32.add
+               (i32.load
+                (local.get $2)
+               )
+               (i32.const 3)
+              )
+              (i32.const -4)
+             )
+            )
+           )
+          )
+          (i32.store
+           (local.get $2)
+           (i32.add
+            (local.get $1)
+            (i32.const 4)
+           )
+          )
+          (i64.store
+           (local.get $0)
+           (i64.extend_i32_s
+            (i32.shr_s
+             (i32.shl
+              (i32.and
+               (local.get $3)
+               (i32.const 65535)
+              )
+              (i32.const 16)
+             )
+             (i32.const 16)
+            )
+           )
+          )
+          (br $label$1)
+         )
+         (local.set $3
+          (i32.load
+           (local.tee $1
+            (i32.and
+             (i32.add
+              (i32.load
+               (local.get $2)
+              )
+              (i32.const 3)
+             )
+             (i32.const -4)
+            )
+           )
+          )
+         )
+         (i32.store
+          (local.get $2)
+          (i32.add
+           (local.get $1)
+           (i32.const 4)
+          )
+         )
+         (i64.store
+          (local.get $0)
+          (i64.extend_i32_u
+           (i32.and
+            (local.get $3)
+            (i32.const 65535)
+           )
+          )
+         )
+         (br $label$1)
+        )
+        (local.set $3
+         (i32.load
+          (local.tee $1
+           (i32.and
+            (i32.add
+             (i32.load
+              (local.get $2)
+             )
+             (i32.const 3)
+            )
+            (i32.const -4)
+           )
+          )
+         )
+        )
+        (i32.store
+         (local.get $2)
+         (i32.add
+          (local.get $1)
+          (i32.const 4)
+         )
+        )
+        (i64.store
+         (local.get $0)
+         (i64.extend_i32_s
+          (i32.shr_s
+           (i32.shl
+            (i32.and
+             (local.get $3)
+             (i32.const 255)
+            )
+            (i32.const 24)
+           )
+           (i32.const 24)
+          )
+         )
+        )
+        (br $label$1)
+       )
+       (local.set $3
+        (i32.load
+         (local.tee $1
+          (i32.and
+           (i32.add
+            (i32.load
+             (local.get $2)
+            )
+            (i32.const 3)
+           )
+           (i32.const -4)
+          )
+         )
+        )
+       )
+       (i32.store
+        (local.get $2)
+        (i32.add
+         (local.get $1)
+         (i32.const 4)
+        )
+       )
+       (i64.store
+        (local.get $0)
+        (i64.extend_i32_u
+         (i32.and
+          (local.get $3)
+          (i32.const 255)
+         )
+        )
+       )
+       (br $label$1)
+      )
+      (local.set $5
+       (f64.load
+        (local.tee $1
+         (i32.and
+          (i32.add
+           (i32.load
+            (local.get $2)
+           )
+           (i32.const 7)
+          )
+          (i32.const -8)
+         )
+        )
+       )
+      )
+      (i32.store
+       (local.get $2)
+       (i32.add
+        (local.get $1)
+        (i32.const 8)
+       )
+      )
+      (f64.store
+       (local.get $0)
+       (local.get $5)
+      )
+      (br $label$1)
+     )
+     (local.set $5
+      (f64.load
+       (local.tee $1
+        (i32.and
+         (i32.add
+          (i32.load
+           (local.get $2)
+          )
+          (i32.const 7)
+         )
+         (i32.const -8)
+        )
+       )
+      )
+     )
+     (i32.store
+      (local.get $2)
+      (i32.add
+       (local.get $1)
+       (i32.const 8)
+      )
+     )
+     (f64.store
+      (local.get $0)
+      (local.get $5)
+     )
+    )
+   )
+  )
+ )
+ (func $23 (; 36 ;) (type $9) (param $0 i64) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i64)
+  (block $label$1 (result i32)
+   (local.set $2
+    (i32.wrap_i64
+     (local.get $0)
+    )
+   )
+   (if
+    (i64.gt_u
+     (local.get $0)
+     (i64.const 4294967295)
+    )
+    (block
+     (loop $label$3
+      (i64.store8
+       (local.tee $1
+        (i32.add
+         (local.get $1)
+         (i32.const -1)
+        )
+       )
+       (i64.or
+        (i64.rem_u
+         (local.get $0)
+         (i64.const 10)
+        )
+        (i64.const 48)
+       )
+      )
+      (local.set $4
+       (i64.div_u
+        (local.get $0)
+        (i64.const 10)
+       )
+      )
+      (if
+       (i64.gt_u
+        (local.get $0)
+        (i64.const 42949672959)
+       )
+       (block
+        (local.set $0
+         (local.get $4)
+        )
+        (br $label$3)
+       )
+      )
+     )
+     (local.set $2
+      (i32.wrap_i64
+       (local.get $4)
+      )
+     )
+    )
+   )
+   (if
+    (local.get $2)
+    (loop $label$6
+     (i32.store8
+      (local.tee $1
+       (i32.add
+        (local.get $1)
+        (i32.const -1)
+       )
+      )
+      (i32.or
+       (i32.rem_u
+        (local.get $2)
+        (i32.const 10)
+       )
+       (i32.const 48)
+      )
+     )
+     (local.set $3
+      (i32.div_u
+       (local.get $2)
+       (i32.const 10)
+      )
+     )
+     (if
+      (i32.ge_u
+       (local.get $2)
+       (i32.const 10)
+      )
+      (block
+       (local.set $2
+        (local.get $3)
+       )
+       (br $label$6)
+      )
+     )
+    )
+   )
+   (local.get $1)
+  )
+ )
+ (func $24 (; 37 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (i32.const 0)
+   )
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (loop $label$5
+       (br_if $label$4
+        (i32.eq
+         (i32.load8_u
+          (i32.add
+           (local.get $1)
+           (i32.const 1711)
+          )
+         )
+         (local.get $0)
+        )
+       )
+       (br_if $label$5
+        (i32.ne
+         (local.tee $1
+          (i32.add
+           (local.get $1)
+           (i32.const 1)
+          )
+         )
+         (i32.const 87)
+        )
+       )
+       (local.set $1
+        (i32.const 87)
+       )
+       (local.set $0
+        (i32.const 1799)
+       )
+       (br $label$3)
+      )
+     )
+     (if
+      (local.get $1)
+      (block
+       (local.set $0
+        (i32.const 1799)
+       )
+       (br $label$3)
+      )
+      (local.set $0
+       (i32.const 1799)
+      )
+     )
+     (br $label$2)
+    )
+    (loop $label$8
+     (local.set $2
+      (local.get $0)
+     )
+     (loop $label$9
+      (local.set $0
+       (i32.add
+        (local.get $2)
+        (i32.const 1)
+       )
+      )
+      (if
+       (i32.load8_s
+        (local.get $2)
+       )
+       (block
+        (local.set $2
+         (local.get $0)
+        )
+        (br $label$9)
+       )
+      )
+     )
+     (br_if $label$8
+      (local.tee $1
+       (i32.add
+        (local.get $1)
+        (i32.const -1)
+       )
+      )
+     )
+    )
+   )
+   (local.get $0)
+  )
+ )
+ (func $25 (; 38 ;) (type $10) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (param $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (block $label$1
+   (local.set $7
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 256)
+    )
+   )
+   (local.set $6
+    (local.get $7)
+   )
+   (block $label$2
+    (if
+     (i32.and
+      (i32.gt_s
+       (local.get $2)
+       (local.get $3)
+      )
+      (i32.eqz
+       (i32.and
+        (local.get $4)
+        (i32.const 73728)
+       )
+      )
+     )
+     (block
+      (drop
+       (call $39
+        (local.get $6)
+        (local.get $1)
+        (if (result i32)
+         (i32.gt_u
+          (local.tee $5
+           (i32.sub
+            (local.get $2)
+            (local.get $3)
+           )
+          )
+          (i32.const 256)
+         )
+         (i32.const 256)
+         (local.get $5)
+        )
+       )
+      )
+      (local.set $4
+       (i32.eqz
+        (i32.and
+         (local.tee $1
+          (i32.load
+           (local.get $0)
+          )
+         )
+         (i32.const 32)
+        )
+       )
+      )
+      (if
+       (i32.gt_u
+        (local.get $5)
+        (i32.const 255)
+       )
+       (block
+        (loop $label$7
+         (if
+          (local.get $4)
+          (block
+           (drop
+            (call $21
+             (local.get $6)
+             (i32.const 256)
+             (local.get $0)
+            )
+           )
+           (local.set $1
+            (i32.load
+             (local.get $0)
+            )
+           )
+          )
+         )
+         (local.set $4
+          (i32.eqz
+           (i32.and
+            (local.get $1)
+            (i32.const 32)
+           )
+          )
+         )
+         (br_if $label$7
+          (i32.gt_u
+           (local.tee $5
+            (i32.add
+             (local.get $5)
+             (i32.const -256)
+            )
+           )
+           (i32.const 255)
+          )
+         )
+        )
+        (br_if $label$2
+         (i32.eqz
+          (local.get $4)
+         )
+        )
+        (local.set $5
+         (i32.and
+          (i32.sub
+           (local.get $2)
+           (local.get $3)
+          )
+          (i32.const 255)
+         )
+        )
+       )
+       (br_if $label$2
+        (i32.eqz
+         (local.get $4)
+        )
+       )
+      )
+      (drop
+       (call $21
+        (local.get $6)
+        (local.get $5)
+        (local.get $0)
+       )
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $7)
+   )
+  )
+ )
+ (func $26 (; 39 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (if (result i32)
+   (local.get $0)
+   (call $29
+    (local.get $0)
+    (local.get $1)
+    (i32.const 0)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $27 (; 40 ;) (type $11) (param $0 f64) (param $1 i32) (result f64)
+  (call $28
+   (local.get $0)
+   (local.get $1)
+  )
+ )
+ (func $28 (; 41 ;) (type $11) (param $0 f64) (param $1 i32) (result f64)
+  (local $2 i64)
+  (local $3 i64)
+  (block $label$1 (result f64)
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (block $label$5
+       (br_table $label$5 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$4 $label$3
+        (i32.sub
+         (i32.shr_s
+          (i32.shl
+           (i32.and
+            (i32.and
+             (i32.wrap_i64
+              (local.tee $3
+               (i64.shr_u
+                (local.tee $2
+                 (i64.reinterpret_f64
+                  (local.get $0)
+                 )
+                )
+                (i64.const 52)
+               )
+              )
+             )
+             (i32.const 65535)
+            )
+            (i32.const 2047)
+           )
+           (i32.const 16)
+          )
+          (i32.const 16)
+         )
+         (i32.const 0)
+        )
+       )
+      )
+      (i32.store
+       (local.get $1)
+       (if (result i32)
+        (f64.ne
+         (local.get $0)
+         (f64.const 0)
+        )
+        (block (result i32)
+         (local.set $0
+          (call $28
+           (f64.mul
+            (local.get $0)
+            (f64.const 18446744073709551615)
+           )
+           (local.get $1)
+          )
+         )
+         (i32.add
+          (i32.load
+           (local.get $1)
+          )
+          (i32.const -64)
+         )
+        )
+        (i32.const 0)
+       )
+      )
+      (br $label$2)
+     )
+     (br $label$2)
+    )
+    (i32.store
+     (local.get $1)
+     (i32.add
+      (i32.and
+       (i32.wrap_i64
+        (local.get $3)
+       )
+       (i32.const 2047)
+      )
+      (i32.const -1022)
+     )
+    )
+    (local.set $0
+     (f64.reinterpret_i64
+      (i64.or
+       (i64.and
+        (local.get $2)
+        (i64.const -9218868437227405313)
+       )
+       (i64.const 4602678819172646912)
+      )
+     )
+    )
+   )
+   (local.get $0)
+  )
+ )
+ (func $29 (; 42 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (block $label$1 (result i32)
+   (if (result i32)
+    (local.get $0)
+    (block (result i32)
+     (if
+      (i32.lt_u
+       (local.get $1)
+       (i32.const 128)
+      )
+      (block
+       (i32.store8
+        (local.get $0)
+        (local.get $1)
+       )
+       (br $label$1
+        (i32.const 1)
+       )
+      )
+     )
+     (if
+      (i32.lt_u
+       (local.get $1)
+       (i32.const 2048)
+      )
+      (block
+       (i32.store8
+        (local.get $0)
+        (i32.or
+         (i32.shr_u
+          (local.get $1)
+          (i32.const 6)
+         )
+         (i32.const 192)
+        )
+       )
+       (i32.store8 offset=1
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (local.get $1)
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (br $label$1
+        (i32.const 2)
+       )
+      )
+     )
+     (if
+      (i32.or
+       (i32.lt_u
+        (local.get $1)
+        (i32.const 55296)
+       )
+       (i32.eq
+        (i32.and
+         (local.get $1)
+         (i32.const -8192)
+        )
+        (i32.const 57344)
+       )
+      )
+      (block
+       (i32.store8
+        (local.get $0)
+        (i32.or
+         (i32.shr_u
+          (local.get $1)
+          (i32.const 12)
+         )
+         (i32.const 224)
+        )
+       )
+       (i32.store8 offset=1
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (i32.shr_u
+           (local.get $1)
+           (i32.const 6)
+          )
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.store8 offset=2
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (local.get $1)
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (br $label$1
+        (i32.const 3)
+       )
+      )
+     )
+     (if (result i32)
+      (i32.lt_u
+       (i32.add
+        (local.get $1)
+        (i32.const -65536)
+       )
+       (i32.const 1048576)
+      )
+      (block (result i32)
+       (i32.store8
+        (local.get $0)
+        (i32.or
+         (i32.shr_u
+          (local.get $1)
+          (i32.const 18)
+         )
+         (i32.const 240)
+        )
+       )
+       (i32.store8 offset=1
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (i32.shr_u
+           (local.get $1)
+           (i32.const 12)
+          )
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.store8 offset=2
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (i32.shr_u
+           (local.get $1)
+           (i32.const 6)
+          )
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.store8 offset=3
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (local.get $1)
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.const 4)
+      )
+      (block (result i32)
+       (i32.store
+        (call $12)
+        (i32.const 84)
+       )
+       (i32.const -1)
+      )
+     )
+    )
+    (i32.const 1)
+   )
+  )
+ )
+ (func $30 (; 43 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (i32.load8_s
+     (local.tee $2
+      (i32.add
+       (local.get $0)
+       (i32.const 74)
+      )
+     )
+    )
+   )
+   (i32.store8
+    (local.get $2)
+    (i32.or
+     (i32.add
+      (local.get $1)
+      (i32.const 255)
+     )
+     (local.get $1)
+    )
+   )
+   (local.tee $0
+    (if (result i32)
+     (i32.and
+      (local.tee $1
+       (i32.load
+        (local.get $0)
+       )
+      )
+      (i32.const 8)
+     )
+     (block (result i32)
+      (i32.store
+       (local.get $0)
+       (i32.or
+        (local.get $1)
+        (i32.const 32)
+       )
+      )
+      (i32.const -1)
+     )
+     (block (result i32)
+      (i32.store offset=8
+       (local.get $0)
+       (i32.const 0)
+      )
+      (i32.store offset=4
+       (local.get $0)
+       (i32.const 0)
+      )
+      (i32.store offset=28
+       (local.get $0)
+       (local.tee $1
+        (i32.load offset=44
+         (local.get $0)
+        )
+       )
+      )
+      (i32.store offset=20
+       (local.get $0)
+       (local.get $1)
+      )
+      (i32.store offset=16
+       (local.get $0)
+       (i32.add
+        (local.get $1)
+        (i32.load offset=48
+         (local.get $0)
+        )
+       )
+      )
+      (i32.const 0)
+     )
+    )
+   )
+  )
+ )
+ (func $31 (; 44 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (block $label$1 (result i32)
+   (local.set $3
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (i32.store8
+    (local.tee $4
+     (local.get $3)
+    )
+    (local.tee $7
+     (i32.and
+      (local.get $1)
+      (i32.const 255)
+     )
+    )
+   )
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (local.tee $5
+       (i32.load
+        (local.tee $2
+         (i32.add
+          (local.get $0)
+          (i32.const 16)
+         )
+        )
+       )
+      )
+     )
+     (if
+      (call $30
+       (local.get $0)
+      )
+      (local.set $1
+       (i32.const -1)
+      )
+      (block
+       (local.set $5
+        (i32.load
+         (local.get $2)
+        )
+       )
+       (br $label$3)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.lt_u
+      (local.tee $6
+       (i32.load
+        (local.tee $2
+         (i32.add
+          (local.get $0)
+          (i32.const 20)
+         )
+        )
+       )
+      )
+      (local.get $5)
+     )
+     (if
+      (i32.ne
+       (local.tee $1
+        (i32.and
+         (local.get $1)
+         (i32.const 255)
+        )
+       )
+       (i32.load8_s offset=75
+        (local.get $0)
+       )
+      )
+      (block
+       (i32.store
+        (local.get $2)
+        (i32.add
+         (local.get $6)
+         (i32.const 1)
+        )
+       )
+       (i32.store8
+        (local.get $6)
+        (local.get $7)
+       )
+       (br $label$2)
+      )
+     )
+    )
+    (local.set $1
+     (if (result i32)
+      (i32.eq
+       (call_indirect (type $0)
+        (local.get $0)
+        (local.get $4)
+        (i32.const 1)
+        (i32.add
+         (i32.and
+          (i32.load offset=36
+           (local.get $0)
+          )
+          (i32.const 3)
+         )
+         (i32.const 2)
+        )
+       )
+       (i32.const 1)
+      )
+      (i32.load8_u
+       (local.get $4)
+      )
+      (i32.const -1)
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $3)
+   )
+   (local.get $1)
+  )
+ )
+ (func $32 (; 45 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (block $label$1 (result i32)
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (i32.lt_s
+       (i32.load offset=76
+        (local.get $1)
+       )
+       (i32.const 0)
+      )
+     )
+     (br_if $label$3
+      (i32.eqz
+       (call $20
+        (local.get $1)
+       )
+      )
+     )
+     (local.set $0
+      (block $label$4 (result i32)
+       (block $label$5
+        (br_if $label$5
+         (i32.eq
+          (i32.load8_s offset=75
+           (local.get $1)
+          )
+          (local.get $0)
+         )
+        )
+        (br_if $label$5
+         (i32.ge_u
+          (local.tee $2
+           (i32.load
+            (local.tee $3
+             (i32.add
+              (local.get $1)
+              (i32.const 20)
+             )
+            )
+           )
+          )
+          (i32.load offset=16
+           (local.get $1)
+          )
+         )
+        )
+        (i32.store
+         (local.get $3)
+         (i32.add
+          (local.get $2)
+          (i32.const 1)
+         )
+        )
+        (i32.store8
+         (local.get $2)
+         (local.get $0)
+        )
+        (br $label$4
+         (i32.and
+          (local.get $0)
+          (i32.const 255)
+         )
+        )
+       )
+       (call $31
+        (local.get $1)
+        (local.get $0)
+       )
+      )
+     )
+     (call $13
+      (local.get $1)
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.ne
+      (i32.load8_s offset=75
+       (local.get $1)
+      )
+      (local.get $0)
+     )
+     (if
+      (i32.lt_u
+       (local.tee $2
+        (i32.load
+         (local.tee $3
+          (i32.add
+           (local.get $1)
+           (i32.const 20)
+          )
+         )
+        )
+       )
+       (i32.load offset=16
+        (local.get $1)
+       )
+      )
+      (block
+       (i32.store
+        (local.get $3)
+        (i32.add
+         (local.get $2)
+         (i32.const 1)
+        )
+       )
+       (i32.store8
+        (local.get $2)
+        (local.get $0)
+       )
+       (local.set $0
+        (i32.and
+         (local.get $0)
+         (i32.const 255)
+        )
+       )
+       (br $label$2)
+      )
+     )
+    )
+    (local.set $0
+     (call $31
+      (local.get $1)
+      (local.get $0)
+     )
+    )
+   )
+   (local.get $0)
+  )
+ )
+ (func $33 (; 46 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (block $label$1 (result i32)
+   (local.set $2
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (i32.store
+    (local.tee $3
+     (local.get $2)
+    )
+    (local.get $1)
+   )
+   (local.set $0
+    (call $18
+     (i32.load
+      (i32.const 1024)
+     )
+     (local.get $0)
+     (local.get $3)
+    )
+   )
+   (global.set $global$1
+    (local.get $2)
+   )
+   (local.get $0)
+  )
+ )
+ (func $34 (; 47 ;) (type $1) (param $0 i32) (result i32)
+  (call $32
+   (local.get $0)
+   (i32.load
+    (i32.const 1024)
+   )
+  )
+ )
+ (func $35 (; 48 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (local $16 i32)
+  (local $17 i32)
+  (local $18 i32)
+  (local $19 i32)
+  (local $20 i32)
+  (local $21 i32)
+  (block $label$1 (result i32)
+   (local.set $14
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (local.set $18
+    (local.get $14)
+   )
+   (block $label$2
+    (if
+     (i32.lt_u
+      (local.get $0)
+      (i32.const 245)
+     )
+     (block
+      (local.set $3
+       (i32.and
+        (i32.add
+         (local.get $0)
+         (i32.const 11)
+        )
+        (i32.const -8)
+       )
+      )
+      (if
+       (i32.and
+        (local.tee $0
+         (i32.shr_u
+          (local.tee $8
+           (i32.load
+            (i32.const 3652)
+           )
+          )
+          (local.tee $2
+           (i32.shr_u
+            (if (result i32)
+             (i32.lt_u
+              (local.get $0)
+              (i32.const 11)
+             )
+             (local.tee $3
+              (i32.const 16)
+             )
+             (local.get $3)
+            )
+            (i32.const 3)
+           )
+          )
+         )
+        )
+        (i32.const 3)
+       )
+       (block
+        (local.set $4
+         (i32.load
+          (local.tee $1
+           (i32.add
+            (local.tee $7
+             (i32.load
+              (local.tee $3
+               (i32.add
+                (local.tee $2
+                 (i32.add
+                  (i32.shl
+                   (i32.shl
+                    (local.tee $5
+                     (i32.add
+                      (i32.xor
+                       (i32.and
+                        (local.get $0)
+                        (i32.const 1)
+                       )
+                       (i32.const 1)
+                      )
+                      (local.get $2)
+                     )
+                    )
+                    (i32.const 1)
+                   )
+                   (i32.const 2)
+                  )
+                  (i32.const 3692)
+                 )
+                )
+                (i32.const 8)
+               )
+              )
+             )
+            )
+            (i32.const 8)
+           )
+          )
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $2)
+          (local.get $4)
+         )
+         (i32.store
+          (i32.const 3652)
+          (i32.and
+           (local.get $8)
+           (i32.xor
+            (i32.shl
+             (i32.const 1)
+             (local.get $5)
+            )
+            (i32.const -1)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $4)
+            (i32.load
+             (i32.const 3668)
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $0
+              (i32.add
+               (local.get $4)
+               (i32.const 12)
+              )
+             )
+            )
+            (local.get $7)
+           )
+           (block
+            (i32.store
+             (local.get $0)
+             (local.get $2)
+            )
+            (i32.store
+             (local.get $3)
+             (local.get $4)
+            )
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (i32.store offset=4
+         (local.get $7)
+         (i32.or
+          (local.tee $0
+           (i32.shl
+            (local.get $5)
+            (i32.const 3)
+           )
+          )
+          (i32.const 3)
+         )
+        )
+        (i32.store
+         (local.tee $0
+          (i32.add
+           (i32.add
+            (local.get $7)
+            (local.get $0)
+           )
+           (i32.const 4)
+          )
+         )
+         (i32.or
+          (i32.load
+           (local.get $0)
+          )
+          (i32.const 1)
+         )
+        )
+        (global.set $global$1
+         (local.get $14)
+        )
+        (return
+         (local.get $1)
+        )
+       )
+      )
+      (if
+       (i32.gt_u
+        (local.get $3)
+        (local.tee $16
+         (i32.load
+          (i32.const 3660)
+         )
+        )
+       )
+       (block
+        (if
+         (local.get $0)
+         (block
+          (local.set $5
+           (i32.and
+            (i32.shr_u
+             (local.tee $0
+              (i32.add
+               (i32.and
+                (local.tee $0
+                 (i32.and
+                  (i32.shl
+                   (local.get $0)
+                   (local.get $2)
+                  )
+                  (i32.or
+                   (local.tee $0
+                    (i32.shl
+                     (i32.const 2)
+                     (local.get $2)
+                    )
+                   )
+                   (i32.sub
+                    (i32.const 0)
+                    (local.get $0)
+                   )
+                  )
+                 )
+                )
+                (i32.sub
+                 (i32.const 0)
+                 (local.get $0)
+                )
+               )
+               (i32.const -1)
+              )
+             )
+             (i32.const 12)
+            )
+            (i32.const 16)
+           )
+          )
+          (local.set $12
+           (i32.load
+            (local.tee $5
+             (i32.add
+              (local.tee $9
+               (i32.load
+                (local.tee $2
+                 (i32.add
+                  (local.tee $4
+                   (i32.add
+                    (i32.shl
+                     (i32.shl
+                      (local.tee $11
+                       (i32.add
+                        (i32.or
+                         (i32.or
+                          (i32.or
+                           (i32.or
+                            (local.tee $0
+                             (i32.and
+                              (i32.shr_u
+                               (local.tee $2
+                                (i32.shr_u
+                                 (local.get $0)
+                                 (local.get $5)
+                                )
+                               )
+                               (i32.const 5)
+                              )
+                              (i32.const 8)
+                             )
+                            )
+                            (local.get $5)
+                           )
+                           (local.tee $0
+                            (i32.and
+                             (i32.shr_u
+                              (local.tee $2
+                               (i32.shr_u
+                                (local.get $2)
+                                (local.get $0)
+                               )
+                              )
+                              (i32.const 2)
+                             )
+                             (i32.const 4)
+                            )
+                           )
+                          )
+                          (local.tee $0
+                           (i32.and
+                            (i32.shr_u
+                             (local.tee $2
+                              (i32.shr_u
+                               (local.get $2)
+                               (local.get $0)
+                              )
+                             )
+                             (i32.const 1)
+                            )
+                            (i32.const 2)
+                           )
+                          )
+                         )
+                         (local.tee $0
+                          (i32.and
+                           (i32.shr_u
+                            (local.tee $2
+                             (i32.shr_u
+                              (local.get $2)
+                              (local.get $0)
+                             )
+                            )
+                            (i32.const 1)
+                           )
+                           (i32.const 1)
+                          )
+                         )
+                        )
+                        (i32.shr_u
+                         (local.get $2)
+                         (local.get $0)
+                        )
+                       )
+                      )
+                      (i32.const 1)
+                     )
+                     (i32.const 2)
+                    )
+                    (i32.const 3692)
+                   )
+                  )
+                  (i32.const 8)
+                 )
+                )
+               )
+              )
+              (i32.const 8)
+             )
+            )
+           )
+          )
+          (if
+           (i32.eq
+            (local.get $4)
+            (local.get $12)
+           )
+           (i32.store
+            (i32.const 3652)
+            (local.tee $7
+             (i32.and
+              (local.get $8)
+              (i32.xor
+               (i32.shl
+                (i32.const 1)
+                (local.get $11)
+               )
+               (i32.const -1)
+              )
+             )
+            )
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.get $12)
+              (i32.load
+               (i32.const 3668)
+              )
+             )
+             (call $fimport$10)
+            )
+            (if
+             (i32.eq
+              (i32.load
+               (local.tee $0
+                (i32.add
+                 (local.get $12)
+                 (i32.const 12)
+                )
+               )
+              )
+              (local.get $9)
+             )
+             (block
+              (i32.store
+               (local.get $0)
+               (local.get $4)
+              )
+              (i32.store
+               (local.get $2)
+               (local.get $12)
+              )
+              (local.set $7
+               (local.get $8)
+              )
+             )
+             (call $fimport$10)
+            )
+           )
+          )
+          (i32.store offset=4
+           (local.get $9)
+           (i32.or
+            (local.get $3)
+            (i32.const 3)
+           )
+          )
+          (i32.store offset=4
+           (local.tee $4
+            (i32.add
+             (local.get $9)
+             (local.get $3)
+            )
+           )
+           (i32.or
+            (local.tee $11
+             (i32.sub
+              (i32.shl
+               (local.get $11)
+               (i32.const 3)
+              )
+              (local.get $3)
+             )
+            )
+            (i32.const 1)
+           )
+          )
+          (i32.store
+           (i32.add
+            (local.get $4)
+            (local.get $11)
+           )
+           (local.get $11)
+          )
+          (if
+           (local.get $16)
+           (block
+            (local.set $9
+             (i32.load
+              (i32.const 3672)
+             )
+            )
+            (local.set $2
+             (i32.add
+              (i32.shl
+               (i32.shl
+                (local.tee $0
+                 (i32.shr_u
+                  (local.get $16)
+                  (i32.const 3)
+                 )
+                )
+                (i32.const 1)
+               )
+               (i32.const 2)
+              )
+              (i32.const 3692)
+             )
+            )
+            (if
+             (i32.and
+              (local.get $7)
+              (local.tee $0
+               (i32.shl
+                (i32.const 1)
+                (local.get $0)
+               )
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.tee $0
+                (i32.load
+                 (local.tee $3
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 8)
+                  )
+                 )
+                )
+               )
+               (i32.load
+                (i32.const 3668)
+               )
+              )
+              (call $fimport$10)
+              (block
+               (local.set $6
+                (local.get $3)
+               )
+               (local.set $1
+                (local.get $0)
+               )
+              )
+             )
+             (block
+              (i32.store
+               (i32.const 3652)
+               (i32.or
+                (local.get $7)
+                (local.get $0)
+               )
+              )
+              (local.set $6
+               (i32.add
+                (local.get $2)
+                (i32.const 8)
+               )
+              )
+              (local.set $1
+               (local.get $2)
+              )
+             )
+            )
+            (i32.store
+             (local.get $6)
+             (local.get $9)
+            )
+            (i32.store offset=12
+             (local.get $1)
+             (local.get $9)
+            )
+            (i32.store offset=8
+             (local.get $9)
+             (local.get $1)
+            )
+            (i32.store offset=12
+             (local.get $9)
+             (local.get $2)
+            )
+           )
+          )
+          (i32.store
+           (i32.const 3660)
+           (local.get $11)
+          )
+          (i32.store
+           (i32.const 3672)
+           (local.get $4)
+          )
+          (global.set $global$1
+           (local.get $14)
+          )
+          (return
+           (local.get $5)
+          )
+         )
+        )
+        (if
+         (local.tee $6
+          (i32.load
+           (i32.const 3656)
+          )
+         )
+         (block
+          (local.set $2
+           (i32.and
+            (i32.shr_u
+             (local.tee $0
+              (i32.add
+               (i32.and
+                (local.get $6)
+                (i32.sub
+                 (i32.const 0)
+                 (local.get $6)
+                )
+               )
+               (i32.const -1)
+              )
+             )
+             (i32.const 12)
+            )
+            (i32.const 16)
+           )
+          )
+          (local.set $9
+           (i32.sub
+            (i32.and
+             (i32.load offset=4
+              (local.tee $2
+               (i32.load
+                (i32.add
+                 (i32.shl
+                  (i32.add
+                   (i32.or
+                    (i32.or
+                     (i32.or
+                      (i32.or
+                       (local.tee $0
+                        (i32.and
+                         (i32.shr_u
+                          (local.tee $1
+                           (i32.shr_u
+                            (local.get $0)
+                            (local.get $2)
+                           )
+                          )
+                          (i32.const 5)
+                         )
+                         (i32.const 8)
+                        )
+                       )
+                       (local.get $2)
+                      )
+                      (local.tee $0
+                       (i32.and
+                        (i32.shr_u
+                         (local.tee $1
+                          (i32.shr_u
+                           (local.get $1)
+                           (local.get $0)
+                          )
+                         )
+                         (i32.const 2)
+                        )
+                        (i32.const 4)
+                       )
+                      )
+                     )
+                     (local.tee $0
+                      (i32.and
+                       (i32.shr_u
+                        (local.tee $1
+                         (i32.shr_u
+                          (local.get $1)
+                          (local.get $0)
+                         )
+                        )
+                        (i32.const 1)
+                       )
+                       (i32.const 2)
+                      )
+                     )
+                    )
+                    (local.tee $0
+                     (i32.and
+                      (i32.shr_u
+                       (local.tee $1
+                        (i32.shr_u
+                         (local.get $1)
+                         (local.get $0)
+                        )
+                       )
+                       (i32.const 1)
+                      )
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                   (i32.shr_u
+                    (local.get $1)
+                    (local.get $0)
+                   )
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 3956)
+                )
+               )
+              )
+             )
+             (i32.const -8)
+            )
+            (local.get $3)
+           )
+          )
+          (local.set $1
+           (local.get $2)
+          )
+          (loop $label$25
+           (block $label$26
+            (if
+             (i32.eqz
+              (local.tee $0
+               (i32.load offset=16
+                (local.get $1)
+               )
+              )
+             )
+             (br_if $label$26
+              (i32.eqz
+               (local.tee $0
+                (i32.load offset=20
+                 (local.get $1)
+                )
+               )
+              )
+             )
+            )
+            (if
+             (local.tee $7
+              (i32.lt_u
+               (local.tee $1
+                (i32.sub
+                 (i32.and
+                  (i32.load offset=4
+                   (local.get $0)
+                  )
+                  (i32.const -8)
+                 )
+                 (local.get $3)
+                )
+               )
+               (local.get $9)
+              )
+             )
+             (local.set $9
+              (local.get $1)
+             )
+            )
+            (local.set $1
+             (local.get $0)
+            )
+            (if
+             (local.get $7)
+             (local.set $2
+              (local.get $0)
+             )
+            )
+            (br $label$25)
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.get $2)
+            (local.tee $12
+             (i32.load
+              (i32.const 3668)
+             )
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.ge_u
+            (local.get $2)
+            (local.tee $13
+             (i32.add
+              (local.get $2)
+              (local.get $3)
+             )
+            )
+           )
+           (call $fimport$10)
+          )
+          (local.set $15
+           (i32.load offset=24
+            (local.get $2)
+           )
+          )
+          (block $label$32
+           (if
+            (i32.eq
+             (local.tee $0
+              (i32.load offset=12
+               (local.get $2)
+              )
+             )
+             (local.get $2)
+            )
+            (block
+             (if
+              (i32.eqz
+               (local.tee $0
+                (i32.load
+                 (local.tee $1
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 20)
+                  )
+                 )
+                )
+               )
+              )
+              (if
+               (i32.eqz
+                (local.tee $0
+                 (i32.load
+                  (local.tee $1
+                   (i32.add
+                    (local.get $2)
+                    (i32.const 16)
+                   )
+                  )
+                 )
+                )
+               )
+               (block
+                (local.set $4
+                 (i32.const 0)
+                )
+                (br $label$32)
+               )
+              )
+             )
+             (loop $label$36
+              (if
+               (local.tee $7
+                (i32.load
+                 (local.tee $11
+                  (i32.add
+                   (local.get $0)
+                   (i32.const 20)
+                  )
+                 )
+                )
+               )
+               (block
+                (local.set $0
+                 (local.get $7)
+                )
+                (local.set $1
+                 (local.get $11)
+                )
+                (br $label$36)
+               )
+              )
+              (if
+               (local.tee $7
+                (i32.load
+                 (local.tee $11
+                  (i32.add
+                   (local.get $0)
+                   (i32.const 16)
+                  )
+                 )
+                )
+               )
+               (block
+                (local.set $0
+                 (local.get $7)
+                )
+                (local.set $1
+                 (local.get $11)
+                )
+                (br $label$36)
+               )
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.get $1)
+               (local.get $12)
+              )
+              (call $fimport$10)
+              (block
+               (i32.store
+                (local.get $1)
+                (i32.const 0)
+               )
+               (local.set $4
+                (local.get $0)
+               )
+              )
+             )
+            )
+            (block
+             (if
+              (i32.lt_u
+               (local.tee $11
+                (i32.load offset=8
+                 (local.get $2)
+                )
+               )
+               (local.get $12)
+              )
+              (call $fimport$10)
+             )
+             (if
+              (i32.ne
+               (i32.load
+                (local.tee $7
+                 (i32.add
+                  (local.get $11)
+                  (i32.const 12)
+                 )
+                )
+               )
+               (local.get $2)
+              )
+              (call $fimport$10)
+             )
+             (if
+              (i32.eq
+               (i32.load
+                (local.tee $1
+                 (i32.add
+                  (local.get $0)
+                  (i32.const 8)
+                 )
+                )
+               )
+               (local.get $2)
+              )
+              (block
+               (i32.store
+                (local.get $7)
+                (local.get $0)
+               )
+               (i32.store
+                (local.get $1)
+                (local.get $11)
+               )
+               (local.set $4
+                (local.get $0)
+               )
+              )
+              (call $fimport$10)
+             )
+            )
+           )
+          )
+          (block $label$46
+           (if
+            (local.get $15)
+            (block
+             (if
+              (i32.eq
+               (local.get $2)
+               (i32.load
+                (local.tee $0
+                 (i32.add
+                  (i32.shl
+                   (local.tee $1
+                    (i32.load offset=28
+                     (local.get $2)
+                    )
+                   )
+                   (i32.const 2)
+                  )
+                  (i32.const 3956)
+                 )
+                )
+               )
+              )
+              (block
+               (i32.store
+                (local.get $0)
+                (local.get $4)
+               )
+               (if
+                (i32.eqz
+                 (local.get $4)
+                )
+                (block
+                 (i32.store
+                  (i32.const 3656)
+                  (i32.and
+                   (local.get $6)
+                   (i32.xor
+                    (i32.shl
+                     (i32.const 1)
+                     (local.get $1)
+                    )
+                    (i32.const -1)
+                   )
+                  )
+                 )
+                 (br $label$46)
+                )
+               )
+              )
+              (block
+               (if
+                (i32.lt_u
+                 (local.get $15)
+                 (i32.load
+                  (i32.const 3668)
+                 )
+                )
+                (call $fimport$10)
+               )
+               (if
+                (i32.eq
+                 (i32.load
+                  (local.tee $0
+                   (i32.add
+                    (local.get $15)
+                    (i32.const 16)
+                   )
+                  )
+                 )
+                 (local.get $2)
+                )
+                (i32.store
+                 (local.get $0)
+                 (local.get $4)
+                )
+                (i32.store offset=20
+                 (local.get $15)
+                 (local.get $4)
+                )
+               )
+               (br_if $label$46
+                (i32.eqz
+                 (local.get $4)
+                )
+               )
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.get $4)
+               (local.tee $0
+                (i32.load
+                 (i32.const 3668)
+                )
+               )
+              )
+              (call $fimport$10)
+             )
+             (i32.store offset=24
+              (local.get $4)
+              (local.get $15)
+             )
+             (if
+              (local.tee $1
+               (i32.load offset=16
+                (local.get $2)
+               )
+              )
+              (if
+               (i32.lt_u
+                (local.get $1)
+                (local.get $0)
+               )
+               (call $fimport$10)
+               (block
+                (i32.store offset=16
+                 (local.get $4)
+                 (local.get $1)
+                )
+                (i32.store offset=24
+                 (local.get $1)
+                 (local.get $4)
+                )
+               )
+              )
+             )
+             (if
+              (local.tee $0
+               (i32.load offset=20
+                (local.get $2)
+               )
+              )
+              (if
+               (i32.lt_u
+                (local.get $0)
+                (i32.load
+                 (i32.const 3668)
+                )
+               )
+               (call $fimport$10)
+               (block
+                (i32.store offset=20
+                 (local.get $4)
+                 (local.get $0)
+                )
+                (i32.store offset=24
+                 (local.get $0)
+                 (local.get $4)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.get $9)
+            (i32.const 16)
+           )
+           (block
+            (i32.store offset=4
+             (local.get $2)
+             (i32.or
+              (local.tee $0
+               (i32.add
+                (local.get $9)
+                (local.get $3)
+               )
+              )
+              (i32.const 3)
+             )
+            )
+            (i32.store
+             (local.tee $0
+              (i32.add
+               (i32.add
+                (local.get $2)
+                (local.get $0)
+               )
+               (i32.const 4)
+              )
+             )
+             (i32.or
+              (i32.load
+               (local.get $0)
+              )
+              (i32.const 1)
+             )
+            )
+           )
+           (block
+            (i32.store offset=4
+             (local.get $2)
+             (i32.or
+              (local.get $3)
+              (i32.const 3)
+             )
+            )
+            (i32.store offset=4
+             (local.get $13)
+             (i32.or
+              (local.get $9)
+              (i32.const 1)
+             )
+            )
+            (i32.store
+             (i32.add
+              (local.get $13)
+              (local.get $9)
+             )
+             (local.get $9)
+            )
+            (if
+             (local.get $16)
+             (block
+              (local.set $7
+               (i32.load
+                (i32.const 3672)
+               )
+              )
+              (local.set $3
+               (i32.add
+                (i32.shl
+                 (i32.shl
+                  (local.tee $0
+                   (i32.shr_u
+                    (local.get $16)
+                    (i32.const 3)
+                   )
+                  )
+                  (i32.const 1)
+                 )
+                 (i32.const 2)
+                )
+                (i32.const 3692)
+               )
+              )
+              (if
+               (i32.and
+                (local.get $8)
+                (local.tee $0
+                 (i32.shl
+                  (i32.const 1)
+                  (local.get $0)
+                 )
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.tee $0
+                  (i32.load
+                   (local.tee $1
+                    (i32.add
+                     (local.get $3)
+                     (i32.const 8)
+                    )
+                   )
+                  )
+                 )
+                 (i32.load
+                  (i32.const 3668)
+                 )
+                )
+                (call $fimport$10)
+                (block
+                 (local.set $10
+                  (local.get $1)
+                 )
+                 (local.set $5
+                  (local.get $0)
+                 )
+                )
+               )
+               (block
+                (i32.store
+                 (i32.const 3652)
+                 (i32.or
+                  (local.get $8)
+                  (local.get $0)
+                 )
+                )
+                (local.set $10
+                 (i32.add
+                  (local.get $3)
+                  (i32.const 8)
+                 )
+                )
+                (local.set $5
+                 (local.get $3)
+                )
+               )
+              )
+              (i32.store
+               (local.get $10)
+               (local.get $7)
+              )
+              (i32.store offset=12
+               (local.get $5)
+               (local.get $7)
+              )
+              (i32.store offset=8
+               (local.get $7)
+               (local.get $5)
+              )
+              (i32.store offset=12
+               (local.get $7)
+               (local.get $3)
+              )
+             )
+            )
+            (i32.store
+             (i32.const 3660)
+             (local.get $9)
+            )
+            (i32.store
+             (i32.const 3672)
+             (local.get $13)
+            )
+           )
+          )
+          (global.set $global$1
+           (local.get $14)
+          )
+          (return
+           (i32.add
+            (local.get $2)
+            (i32.const 8)
+           )
+          )
+         )
+         (local.set $0
+          (local.get $3)
+         )
+        )
+       )
+       (local.set $0
+        (local.get $3)
+       )
+      )
+     )
+     (if
+      (i32.gt_u
+       (local.get $0)
+       (i32.const -65)
+      )
+      (local.set $0
+       (i32.const -1)
+      )
+      (block
+       (local.set $7
+        (i32.and
+         (local.tee $0
+          (i32.add
+           (local.get $0)
+           (i32.const 11)
+          )
+         )
+         (i32.const -8)
+        )
+       )
+       (if
+        (local.tee $5
+         (i32.load
+          (i32.const 3656)
+         )
+        )
+        (block
+         (local.set $17
+          (if (result i32)
+           (local.tee $0
+            (i32.shr_u
+             (local.get $0)
+             (i32.const 8)
+            )
+           )
+           (if (result i32)
+            (i32.gt_u
+             (local.get $7)
+             (i32.const 16777215)
+            )
+            (i32.const 31)
+            (i32.or
+             (i32.and
+              (i32.shr_u
+               (local.get $7)
+               (i32.add
+                (local.tee $0
+                 (i32.add
+                  (i32.sub
+                   (i32.const 14)
+                   (i32.or
+                    (i32.or
+                     (local.tee $0
+                      (i32.and
+                       (i32.shr_u
+                        (i32.add
+                         (local.tee $1
+                          (i32.shl
+                           (local.get $0)
+                           (local.tee $3
+                            (i32.and
+                             (i32.shr_u
+                              (i32.add
+                               (local.get $0)
+                               (i32.const 1048320)
+                              )
+                              (i32.const 16)
+                             )
+                             (i32.const 8)
+                            )
+                           )
+                          )
+                         )
+                         (i32.const 520192)
+                        )
+                        (i32.const 16)
+                       )
+                       (i32.const 4)
+                      )
+                     )
+                     (local.get $3)
+                    )
+                    (local.tee $0
+                     (i32.and
+                      (i32.shr_u
+                       (i32.add
+                        (local.tee $1
+                         (i32.shl
+                          (local.get $1)
+                          (local.get $0)
+                         )
+                        )
+                        (i32.const 245760)
+                       )
+                       (i32.const 16)
+                      )
+                      (i32.const 2)
+                     )
+                    )
+                   )
+                  )
+                  (i32.shr_u
+                   (i32.shl
+                    (local.get $1)
+                    (local.get $0)
+                   )
+                   (i32.const 15)
+                  )
+                 )
+                )
+                (i32.const 7)
+               )
+              )
+              (i32.const 1)
+             )
+             (i32.shl
+              (local.get $0)
+              (i32.const 1)
+             )
+            )
+           )
+           (i32.const 0)
+          )
+         )
+         (local.set $3
+          (i32.sub
+           (i32.const 0)
+           (local.get $7)
+          )
+         )
+         (block $label$78
+          (block $label$79
+           (block $label$80
+            (if
+             (local.tee $1
+              (i32.load
+               (i32.add
+                (i32.shl
+                 (local.get $17)
+                 (i32.const 2)
+                )
+                (i32.const 3956)
+               )
+              )
+             )
+             (block
+              (local.set $0
+               (i32.sub
+                (i32.const 25)
+                (i32.shr_u
+                 (local.get $17)
+                 (i32.const 1)
+                )
+               )
+              )
+              (local.set $4
+               (i32.const 0)
+              )
+              (local.set $10
+               (i32.shl
+                (local.get $7)
+                (if (result i32)
+                 (i32.eq
+                  (local.get $17)
+                  (i32.const 31)
+                 )
+                 (i32.const 0)
+                 (local.get $0)
+                )
+               )
+              )
+              (local.set $0
+               (i32.const 0)
+              )
+              (loop $label$84
+               (if
+                (i32.lt_u
+                 (local.tee $6
+                  (i32.sub
+                   (i32.and
+                    (i32.load offset=4
+                     (local.get $1)
+                    )
+                    (i32.const -8)
+                   )
+                   (local.get $7)
+                  )
+                 )
+                 (local.get $3)
+                )
+                (if
+                 (local.get $6)
+                 (block
+                  (local.set $3
+                   (local.get $6)
+                  )
+                  (local.set $0
+                   (local.get $1)
+                  )
+                 )
+                 (block
+                  (local.set $3
+                   (i32.const 0)
+                  )
+                  (local.set $0
+                   (local.get $1)
+                  )
+                  (br $label$79)
+                 )
+                )
+               )
+               (local.set $1
+                (if (result i32)
+                 (i32.or
+                  (i32.eqz
+                   (local.tee $19
+                    (i32.load offset=20
+                     (local.get $1)
+                    )
+                   )
+                  )
+                  (i32.eq
+                   (local.get $19)
+                   (local.tee $6
+                    (i32.load
+                     (i32.add
+                      (i32.add
+                       (local.get $1)
+                       (i32.const 16)
+                      )
+                      (i32.shl
+                       (i32.shr_u
+                        (local.get $10)
+                        (i32.const 31)
+                       )
+                       (i32.const 2)
+                      )
+                     )
+                    )
+                   )
+                  )
+                 )
+                 (local.get $4)
+                 (local.get $19)
+                )
+               )
+               (local.set $10
+                (i32.shl
+                 (local.get $10)
+                 (i32.xor
+                  (i32.and
+                   (local.tee $4
+                    (i32.eqz
+                     (local.get $6)
+                    )
+                   )
+                   (i32.const 1)
+                  )
+                  (i32.const 1)
+                 )
+                )
+               )
+               (if
+                (local.get $4)
+                (block
+                 (local.set $4
+                  (local.get $1)
+                 )
+                 (local.set $1
+                  (local.get $0)
+                 )
+                 (br $label$80)
+                )
+                (block
+                 (local.set $4
+                  (local.get $1)
+                 )
+                 (local.set $1
+                  (local.get $6)
+                 )
+                 (br $label$84)
+                )
+               )
+              )
+             )
+             (block
+              (local.set $4
+               (i32.const 0)
+              )
+              (local.set $1
+               (i32.const 0)
+              )
+             )
+            )
+           )
+           (br_if $label$79
+            (local.tee $0
+             (if (result i32)
+              (i32.and
+               (i32.eqz
+                (local.get $4)
+               )
+               (i32.eqz
+                (local.get $1)
+               )
+              )
+              (block (result i32)
+               (if
+                (i32.eqz
+                 (local.tee $0
+                  (i32.and
+                   (local.get $5)
+                   (i32.or
+                    (local.tee $0
+                     (i32.shl
+                      (i32.const 2)
+                      (local.get $17)
+                     )
+                    )
+                    (i32.sub
+                     (i32.const 0)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                 )
+                )
+                (block
+                 (local.set $0
+                  (local.get $7)
+                 )
+                 (br $label$2)
+                )
+               )
+               (local.set $10
+                (i32.and
+                 (i32.shr_u
+                  (local.tee $0
+                   (i32.add
+                    (i32.and
+                     (local.get $0)
+                     (i32.sub
+                      (i32.const 0)
+                      (local.get $0)
+                     )
+                    )
+                    (i32.const -1)
+                   )
+                  )
+                  (i32.const 12)
+                 )
+                 (i32.const 16)
+                )
+               )
+               (i32.load
+                (i32.add
+                 (i32.shl
+                  (i32.add
+                   (i32.or
+                    (i32.or
+                     (i32.or
+                      (i32.or
+                       (local.tee $0
+                        (i32.and
+                         (i32.shr_u
+                          (local.tee $4
+                           (i32.shr_u
+                            (local.get $0)
+                            (local.get $10)
+                           )
+                          )
+                          (i32.const 5)
+                         )
+                         (i32.const 8)
+                        )
+                       )
+                       (local.get $10)
+                      )
+                      (local.tee $0
+                       (i32.and
+                        (i32.shr_u
+                         (local.tee $4
+                          (i32.shr_u
+                           (local.get $4)
+                           (local.get $0)
+                          )
+                         )
+                         (i32.const 2)
+                        )
+                        (i32.const 4)
+                       )
+                      )
+                     )
+                     (local.tee $0
+                      (i32.and
+                       (i32.shr_u
+                        (local.tee $4
+                         (i32.shr_u
+                          (local.get $4)
+                          (local.get $0)
+                         )
+                        )
+                        (i32.const 1)
+                       )
+                       (i32.const 2)
+                      )
+                     )
+                    )
+                    (local.tee $0
+                     (i32.and
+                      (i32.shr_u
+                       (local.tee $4
+                        (i32.shr_u
+                         (local.get $4)
+                         (local.get $0)
+                        )
+                       )
+                       (i32.const 1)
+                      )
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                   (i32.shr_u
+                    (local.get $4)
+                    (local.get $0)
+                   )
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 3956)
+                )
+               )
+              )
+              (local.get $4)
+             )
+            )
+           )
+           (local.set $4
+            (local.get $1)
+           )
+           (br $label$78)
+          )
+          (loop $label$96
+           (if
+            (local.tee $10
+             (i32.lt_u
+              (local.tee $4
+               (i32.sub
+                (i32.and
+                 (i32.load offset=4
+                  (local.get $0)
+                 )
+                 (i32.const -8)
+                )
+                (local.get $7)
+               )
+              )
+              (local.get $3)
+             )
+            )
+            (local.set $3
+             (local.get $4)
+            )
+           )
+           (if
+            (local.get $10)
+            (local.set $1
+             (local.get $0)
+            )
+           )
+           (if
+            (local.tee $4
+             (i32.load offset=16
+              (local.get $0)
+             )
+            )
+            (block
+             (local.set $0
+              (local.get $4)
+             )
+             (br $label$96)
+            )
+           )
+           (br_if $label$96
+            (local.tee $0
+             (i32.load offset=20
+              (local.get $0)
+             )
+            )
+           )
+           (local.set $4
+            (local.get $1)
+           )
+          )
+         )
+         (if
+          (local.get $4)
+          (if
+           (i32.lt_u
+            (local.get $3)
+            (i32.sub
+             (i32.load
+              (i32.const 3660)
+             )
+             (local.get $7)
+            )
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.get $4)
+              (local.tee $12
+               (i32.load
+                (i32.const 3668)
+               )
+              )
+             )
+             (call $fimport$10)
+            )
+            (if
+             (i32.ge_u
+              (local.get $4)
+              (local.tee $6
+               (i32.add
+                (local.get $4)
+                (local.get $7)
+               )
+              )
+             )
+             (call $fimport$10)
+            )
+            (local.set $10
+             (i32.load offset=24
+              (local.get $4)
+             )
+            )
+            (block $label$104
+             (if
+              (i32.eq
+               (local.tee $0
+                (i32.load offset=12
+                 (local.get $4)
+                )
+               )
+               (local.get $4)
+              )
+              (block
+               (if
+                (i32.eqz
+                 (local.tee $0
+                  (i32.load
+                   (local.tee $1
+                    (i32.add
+                     (local.get $4)
+                     (i32.const 20)
+                    )
+                   )
+                  )
+                 )
+                )
+                (if
+                 (i32.eqz
+                  (local.tee $0
+                   (i32.load
+                    (local.tee $1
+                     (i32.add
+                      (local.get $4)
+                      (i32.const 16)
+                     )
+                    )
+                   )
+                  )
+                 )
+                 (block
+                  (local.set $13
+                   (i32.const 0)
+                  )
+                  (br $label$104)
+                 )
+                )
+               )
+               (loop $label$108
+                (if
+                 (local.tee $11
+                  (i32.load
+                   (local.tee $9
+                    (i32.add
+                     (local.get $0)
+                     (i32.const 20)
+                    )
+                   )
+                  )
+                 )
+                 (block
+                  (local.set $0
+                   (local.get $11)
+                  )
+                  (local.set $1
+                   (local.get $9)
+                  )
+                  (br $label$108)
+                 )
+                )
+                (if
+                 (local.tee $11
+                  (i32.load
+                   (local.tee $9
+                    (i32.add
+                     (local.get $0)
+                     (i32.const 16)
+                    )
+                   )
+                  )
+                 )
+                 (block
+                  (local.set $0
+                   (local.get $11)
+                  )
+                  (local.set $1
+                   (local.get $9)
+                  )
+                  (br $label$108)
+                 )
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $1)
+                 (local.get $12)
+                )
+                (call $fimport$10)
+                (block
+                 (i32.store
+                  (local.get $1)
+                  (i32.const 0)
+                 )
+                 (local.set $13
+                  (local.get $0)
+                 )
+                )
+               )
+              )
+              (block
+               (if
+                (i32.lt_u
+                 (local.tee $9
+                  (i32.load offset=8
+                   (local.get $4)
+                  )
+                 )
+                 (local.get $12)
+                )
+                (call $fimport$10)
+               )
+               (if
+                (i32.ne
+                 (i32.load
+                  (local.tee $11
+                   (i32.add
+                    (local.get $9)
+                    (i32.const 12)
+                   )
+                  )
+                 )
+                 (local.get $4)
+                )
+                (call $fimport$10)
+               )
+               (if
+                (i32.eq
+                 (i32.load
+                  (local.tee $1
+                   (i32.add
+                    (local.get $0)
+                    (i32.const 8)
+                   )
+                  )
+                 )
+                 (local.get $4)
+                )
+                (block
+                 (i32.store
+                  (local.get $11)
+                  (local.get $0)
+                 )
+                 (i32.store
+                  (local.get $1)
+                  (local.get $9)
+                 )
+                 (local.set $13
+                  (local.get $0)
+                 )
+                )
+                (call $fimport$10)
+               )
+              )
+             )
+            )
+            (block $label$118
+             (if
+              (local.get $10)
+              (block
+               (if
+                (i32.eq
+                 (local.get $4)
+                 (i32.load
+                  (local.tee $0
+                   (i32.add
+                    (i32.shl
+                     (local.tee $1
+                      (i32.load offset=28
+                       (local.get $4)
+                      )
+                     )
+                     (i32.const 2)
+                    )
+                    (i32.const 3956)
+                   )
+                  )
+                 )
+                )
+                (block
+                 (i32.store
+                  (local.get $0)
+                  (local.get $13)
+                 )
+                 (if
+                  (i32.eqz
+                   (local.get $13)
+                  )
+                  (block
+                   (i32.store
+                    (i32.const 3656)
+                    (local.tee $2
+                     (i32.and
+                      (local.get $5)
+                      (i32.xor
+                       (i32.shl
+                        (i32.const 1)
+                        (local.get $1)
+                       )
+                       (i32.const -1)
+                      )
+                     )
+                    )
+                   )
+                   (br $label$118)
+                  )
+                 )
+                )
+                (block
+                 (if
+                  (i32.lt_u
+                   (local.get $10)
+                   (i32.load
+                    (i32.const 3668)
+                   )
+                  )
+                  (call $fimport$10)
+                 )
+                 (if
+                  (i32.eq
+                   (i32.load
+                    (local.tee $0
+                     (i32.add
+                      (local.get $10)
+                      (i32.const 16)
+                     )
+                    )
+                   )
+                   (local.get $4)
+                  )
+                  (i32.store
+                   (local.get $0)
+                   (local.get $13)
+                  )
+                  (i32.store offset=20
+                   (local.get $10)
+                   (local.get $13)
+                  )
+                 )
+                 (if
+                  (i32.eqz
+                   (local.get $13)
+                  )
+                  (block
+                   (local.set $2
+                    (local.get $5)
+                   )
+                   (br $label$118)
+                  )
+                 )
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $13)
+                 (local.tee $0
+                  (i32.load
+                   (i32.const 3668)
+                  )
+                 )
+                )
+                (call $fimport$10)
+               )
+               (i32.store offset=24
+                (local.get $13)
+                (local.get $10)
+               )
+               (if
+                (local.tee $1
+                 (i32.load offset=16
+                  (local.get $4)
+                 )
+                )
+                (if
+                 (i32.lt_u
+                  (local.get $1)
+                  (local.get $0)
+                 )
+                 (call $fimport$10)
+                 (block
+                  (i32.store offset=16
+                   (local.get $13)
+                   (local.get $1)
+                  )
+                  (i32.store offset=24
+                   (local.get $1)
+                   (local.get $13)
+                  )
+                 )
+                )
+               )
+               (if
+                (local.tee $0
+                 (i32.load offset=20
+                  (local.get $4)
+                 )
+                )
+                (if
+                 (i32.lt_u
+                  (local.get $0)
+                  (i32.load
+                   (i32.const 3668)
+                  )
+                 )
+                 (call $fimport$10)
+                 (block
+                  (i32.store offset=20
+                   (local.get $13)
+                   (local.get $0)
+                  )
+                  (i32.store offset=24
+                   (local.get $0)
+                   (local.get $13)
+                  )
+                  (local.set $2
+                   (local.get $5)
+                  )
+                 )
+                )
+                (local.set $2
+                 (local.get $5)
+                )
+               )
+              )
+              (local.set $2
+               (local.get $5)
+              )
+             )
+            )
+            (block $label$136
+             (if
+              (i32.lt_u
+               (local.get $3)
+               (i32.const 16)
+              )
+              (block
+               (i32.store offset=4
+                (local.get $4)
+                (i32.or
+                 (local.tee $0
+                  (i32.add
+                   (local.get $3)
+                   (local.get $7)
+                  )
+                 )
+                 (i32.const 3)
+                )
+               )
+               (i32.store
+                (local.tee $0
+                 (i32.add
+                  (i32.add
+                   (local.get $4)
+                   (local.get $0)
+                  )
+                  (i32.const 4)
+                 )
+                )
+                (i32.or
+                 (i32.load
+                  (local.get $0)
+                 )
+                 (i32.const 1)
+                )
+               )
+              )
+              (block
+               (i32.store offset=4
+                (local.get $4)
+                (i32.or
+                 (local.get $7)
+                 (i32.const 3)
+                )
+               )
+               (i32.store offset=4
+                (local.get $6)
+                (i32.or
+                 (local.get $3)
+                 (i32.const 1)
+                )
+               )
+               (i32.store
+                (i32.add
+                 (local.get $6)
+                 (local.get $3)
+                )
+                (local.get $3)
+               )
+               (local.set $0
+                (i32.shr_u
+                 (local.get $3)
+                 (i32.const 3)
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $3)
+                 (i32.const 256)
+                )
+                (block
+                 (local.set $3
+                  (i32.add
+                   (i32.shl
+                    (i32.shl
+                     (local.get $0)
+                     (i32.const 1)
+                    )
+                    (i32.const 2)
+                   )
+                   (i32.const 3692)
+                  )
+                 )
+                 (if
+                  (i32.and
+                   (local.tee $1
+                    (i32.load
+                     (i32.const 3652)
+                    )
+                   )
+                   (local.tee $0
+                    (i32.shl
+                     (i32.const 1)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (if
+                   (i32.lt_u
+                    (local.tee $0
+                     (i32.load
+                      (local.tee $1
+                       (i32.add
+                        (local.get $3)
+                        (i32.const 8)
+                       )
+                      )
+                     )
+                    )
+                    (i32.load
+                     (i32.const 3668)
+                    )
+                   )
+                   (call $fimport$10)
+                   (block
+                    (local.set $16
+                     (local.get $1)
+                    )
+                    (local.set $8
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (block
+                   (i32.store
+                    (i32.const 3652)
+                    (i32.or
+                     (local.get $1)
+                     (local.get $0)
+                    )
+                   )
+                   (local.set $16
+                    (i32.add
+                     (local.get $3)
+                     (i32.const 8)
+                    )
+                   )
+                   (local.set $8
+                    (local.get $3)
+                   )
+                  )
+                 )
+                 (i32.store
+                  (local.get $16)
+                  (local.get $6)
+                 )
+                 (i32.store offset=12
+                  (local.get $8)
+                  (local.get $6)
+                 )
+                 (i32.store offset=8
+                  (local.get $6)
+                  (local.get $8)
+                 )
+                 (i32.store offset=12
+                  (local.get $6)
+                  (local.get $3)
+                 )
+                 (br $label$136)
+                )
+               )
+               (local.set $1
+                (i32.add
+                 (i32.shl
+                  (local.tee $5
+                   (if (result i32)
+                    (local.tee $0
+                     (i32.shr_u
+                      (local.get $3)
+                      (i32.const 8)
+                     )
+                    )
+                    (if (result i32)
+                     (i32.gt_u
+                      (local.get $3)
+                      (i32.const 16777215)
+                     )
+                     (i32.const 31)
+                     (i32.or
+                      (i32.and
+                       (i32.shr_u
+                        (local.get $3)
+                        (i32.add
+                         (local.tee $0
+                          (i32.add
+                           (i32.sub
+                            (i32.const 14)
+                            (i32.or
+                             (i32.or
+                              (local.tee $0
+                               (i32.and
+                                (i32.shr_u
+                                 (i32.add
+                                  (local.tee $1
+                                   (i32.shl
+                                    (local.get $0)
+                                    (local.tee $5
+                                     (i32.and
+                                      (i32.shr_u
+                                       (i32.add
+                                        (local.get $0)
+                                        (i32.const 1048320)
+                                       )
+                                       (i32.const 16)
+                                      )
+                                      (i32.const 8)
+                                     )
+                                    )
+                                   )
+                                  )
+                                  (i32.const 520192)
+                                 )
+                                 (i32.const 16)
+                                )
+                                (i32.const 4)
+                               )
+                              )
+                              (local.get $5)
+                             )
+                             (local.tee $0
+                              (i32.and
+                               (i32.shr_u
+                                (i32.add
+                                 (local.tee $1
+                                  (i32.shl
+                                   (local.get $1)
+                                   (local.get $0)
+                                  )
+                                 )
+                                 (i32.const 245760)
+                                )
+                                (i32.const 16)
+                               )
+                               (i32.const 2)
+                              )
+                             )
+                            )
+                           )
+                           (i32.shr_u
+                            (i32.shl
+                             (local.get $1)
+                             (local.get $0)
+                            )
+                            (i32.const 15)
+                           )
+                          )
+                         )
+                         (i32.const 7)
+                        )
+                       )
+                       (i32.const 1)
+                      )
+                      (i32.shl
+                       (local.get $0)
+                       (i32.const 1)
+                      )
+                     )
+                    )
+                    (i32.const 0)
+                   )
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 3956)
+                )
+               )
+               (i32.store offset=28
+                (local.get $6)
+                (local.get $5)
+               )
+               (i32.store offset=4
+                (local.tee $0
+                 (i32.add
+                  (local.get $6)
+                  (i32.const 16)
+                 )
+                )
+                (i32.const 0)
+               )
+               (i32.store
+                (local.get $0)
+                (i32.const 0)
+               )
+               (if
+                (i32.eqz
+                 (i32.and
+                  (local.get $2)
+                  (local.tee $0
+                   (i32.shl
+                    (i32.const 1)
+                    (local.get $5)
+                   )
+                  )
+                 )
+                )
+                (block
+                 (i32.store
+                  (i32.const 3656)
+                  (i32.or
+                   (local.get $2)
+                   (local.get $0)
+                  )
+                 )
+                 (i32.store
+                  (local.get $1)
+                  (local.get $6)
+                 )
+                 (i32.store offset=24
+                  (local.get $6)
+                  (local.get $1)
+                 )
+                 (i32.store offset=12
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (i32.store offset=8
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (br $label$136)
+                )
+               )
+               (local.set $0
+                (i32.load
+                 (local.get $1)
+                )
+               )
+               (local.set $1
+                (i32.sub
+                 (i32.const 25)
+                 (i32.shr_u
+                  (local.get $5)
+                  (i32.const 1)
+                 )
+                )
+               )
+               (local.set $5
+                (i32.shl
+                 (local.get $3)
+                 (if (result i32)
+                  (i32.eq
+                   (local.get $5)
+                   (i32.const 31)
+                  )
+                  (i32.const 0)
+                  (local.get $1)
+                 )
+                )
+               )
+               (block $label$151
+                (block $label$152
+                 (block $label$153
+                  (loop $label$154
+                   (br_if $label$152
+                    (i32.eq
+                     (i32.and
+                      (i32.load offset=4
+                       (local.get $0)
+                      )
+                      (i32.const -8)
+                     )
+                     (local.get $3)
+                    )
+                   )
+                   (local.set $2
+                    (i32.shl
+                     (local.get $5)
+                     (i32.const 1)
+                    )
+                   )
+                   (br_if $label$153
+                    (i32.eqz
+                     (local.tee $1
+                      (i32.load
+                       (local.tee $5
+                        (i32.add
+                         (i32.add
+                          (local.get $0)
+                          (i32.const 16)
+                         )
+                         (i32.shl
+                          (i32.shr_u
+                           (local.get $5)
+                           (i32.const 31)
+                          )
+                          (i32.const 2)
+                         )
+                        )
+                       )
+                      )
+                     )
+                    )
+                   )
+                   (local.set $5
+                    (local.get $2)
+                   )
+                   (local.set $0
+                    (local.get $1)
+                   )
+                   (br $label$154)
+                  )
+                 )
+                 (if
+                  (i32.lt_u
+                   (local.get $5)
+                   (i32.load
+                    (i32.const 3668)
+                   )
+                  )
+                  (call $fimport$10)
+                  (block
+                   (i32.store
+                    (local.get $5)
+                    (local.get $6)
+                   )
+                   (i32.store offset=24
+                    (local.get $6)
+                    (local.get $0)
+                   )
+                   (i32.store offset=12
+                    (local.get $6)
+                    (local.get $6)
+                   )
+                   (i32.store offset=8
+                    (local.get $6)
+                    (local.get $6)
+                   )
+                   (br $label$136)
+                  )
+                 )
+                 (br $label$151)
+                )
+                (if
+                 (i32.and
+                  (i32.ge_u
+                   (local.tee $2
+                    (i32.load
+                     (local.tee $3
+                      (i32.add
+                       (local.get $0)
+                       (i32.const 8)
+                      )
+                     )
+                    )
+                   )
+                   (local.tee $1
+                    (i32.load
+                     (i32.const 3668)
+                    )
+                   )
+                  )
+                  (i32.ge_u
+                   (local.get $0)
+                   (local.get $1)
+                  )
+                 )
+                 (block
+                  (i32.store offset=12
+                   (local.get $2)
+                   (local.get $6)
+                  )
+                  (i32.store
+                   (local.get $3)
+                   (local.get $6)
+                  )
+                  (i32.store offset=8
+                   (local.get $6)
+                   (local.get $2)
+                  )
+                  (i32.store offset=12
+                   (local.get $6)
+                   (local.get $0)
+                  )
+                  (i32.store offset=24
+                   (local.get $6)
+                   (i32.const 0)
+                  )
+                 )
+                 (call $fimport$10)
+                )
+               )
+              )
+             )
+            )
+            (global.set $global$1
+             (local.get $14)
+            )
+            (return
+             (i32.add
+              (local.get $4)
+              (i32.const 8)
+             )
+            )
+           )
+           (local.set $0
+            (local.get $7)
+           )
+          )
+          (local.set $0
+           (local.get $7)
+          )
+         )
+        )
+        (local.set $0
+         (local.get $7)
+        )
+       )
+      )
+     )
+    )
+   )
+   (if
+    (i32.ge_u
+     (local.tee $1
+      (i32.load
+       (i32.const 3660)
+      )
+     )
+     (local.get $0)
+    )
+    (block
+     (local.set $2
+      (i32.load
+       (i32.const 3672)
+      )
+     )
+     (if
+      (i32.gt_u
+       (local.tee $3
+        (i32.sub
+         (local.get $1)
+         (local.get $0)
+        )
+       )
+       (i32.const 15)
+      )
+      (block
+       (i32.store
+        (i32.const 3672)
+        (local.tee $1
+         (i32.add
+          (local.get $2)
+          (local.get $0)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3660)
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $1)
+        (i32.or
+         (local.get $3)
+         (i32.const 1)
+        )
+       )
+       (i32.store
+        (i32.add
+         (local.get $1)
+         (local.get $3)
+        )
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $2)
+        (i32.or
+         (local.get $0)
+         (i32.const 3)
+        )
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3660)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 3672)
+        (i32.const 0)
+       )
+       (i32.store offset=4
+        (local.get $2)
+        (i32.or
+         (local.get $1)
+         (i32.const 3)
+        )
+       )
+       (i32.store
+        (local.tee $0
+         (i32.add
+          (i32.add
+           (local.get $2)
+           (local.get $1)
+          )
+          (i32.const 4)
+         )
+        )
+        (i32.or
+         (i32.load
+          (local.get $0)
+         )
+         (i32.const 1)
+        )
+       )
+      )
+     )
+     (global.set $global$1
+      (local.get $14)
+     )
+     (return
+      (i32.add
+       (local.get $2)
+       (i32.const 8)
+      )
+     )
+    )
+   )
+   (if
+    (i32.gt_u
+     (local.tee $10
+      (i32.load
+       (i32.const 3664)
+      )
+     )
+     (local.get $0)
+    )
+    (block
+     (i32.store
+      (i32.const 3664)
+      (local.tee $3
+       (i32.sub
+        (local.get $10)
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store
+      (i32.const 3676)
+      (local.tee $1
+       (i32.add
+        (local.tee $2
+         (i32.load
+          (i32.const 3676)
+         )
+        )
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store offset=4
+      (local.get $1)
+      (i32.or
+       (local.get $3)
+       (i32.const 1)
+      )
+     )
+     (i32.store offset=4
+      (local.get $2)
+      (i32.or
+       (local.get $0)
+       (i32.const 3)
+      )
+     )
+     (global.set $global$1
+      (local.get $14)
+     )
+     (return
+      (i32.add
+       (local.get $2)
+       (i32.const 8)
+      )
+     )
+    )
+   )
+   (if
+    (i32.le_u
+     (local.tee $6
+      (i32.and
+       (local.tee $8
+        (i32.add
+         (local.tee $1
+          (if (result i32)
+           (i32.load
+            (i32.const 4124)
+           )
+           (i32.load
+            (i32.const 4132)
+           )
+           (block (result i32)
+            (i32.store
+             (i32.const 4132)
+             (i32.const 4096)
+            )
+            (i32.store
+             (i32.const 4128)
+             (i32.const 4096)
+            )
+            (i32.store
+             (i32.const 4136)
+             (i32.const -1)
+            )
+            (i32.store
+             (i32.const 4140)
+             (i32.const -1)
+            )
+            (i32.store
+             (i32.const 4144)
+             (i32.const 0)
+            )
+            (i32.store
+             (i32.const 4096)
+             (i32.const 0)
+            )
+            (i32.store
+             (local.get $18)
+             (local.tee $1
+              (i32.xor
+               (i32.and
+                (local.get $18)
+                (i32.const -16)
+               )
+               (i32.const 1431655768)
+              )
+             )
+            )
+            (i32.store
+             (i32.const 4124)
+             (local.get $1)
+            )
+            (i32.const 4096)
+           )
+          )
+         )
+         (local.tee $13
+          (i32.add
+           (local.get $0)
+           (i32.const 47)
+          )
+         )
+        )
+       )
+       (local.tee $4
+        (i32.sub
+         (i32.const 0)
+         (local.get $1)
+        )
+       )
+      )
+     )
+     (local.get $0)
+    )
+    (block
+     (global.set $global$1
+      (local.get $14)
+     )
+     (return
+      (i32.const 0)
+     )
+    )
+   )
+   (if
+    (local.tee $2
+     (i32.load
+      (i32.const 4092)
+     )
+    )
+    (if
+     (i32.or
+      (i32.le_u
+       (local.tee $1
+        (i32.add
+         (local.tee $3
+          (i32.load
+           (i32.const 4084)
+          )
+         )
+         (local.get $6)
+        )
+       )
+       (local.get $3)
+      )
+      (i32.gt_u
+       (local.get $1)
+       (local.get $2)
+      )
+     )
+     (block
+      (global.set $global$1
+       (local.get $14)
+      )
+      (return
+       (i32.const 0)
+      )
+     )
+    )
+   )
+   (local.set $7
+    (i32.add
+     (local.get $0)
+     (i32.const 48)
+    )
+   )
+   (block $label$171
+    (block $label$172
+     (if
+      (i32.eqz
+       (i32.and
+        (i32.load
+         (i32.const 4096)
+        )
+        (i32.const 4)
+       )
+      )
+      (block
+       (block $label$174
+        (block $label$175
+         (block $label$176
+          (br_if $label$176
+           (i32.eqz
+            (local.tee $3
+             (i32.load
+              (i32.const 3676)
+             )
+            )
+           )
+          )
+          (local.set $2
+           (i32.const 4100)
+          )
+          (loop $label$177
+           (block $label$178
+            (if
+             (i32.le_u
+              (local.tee $1
+               (i32.load
+                (local.get $2)
+               )
+              )
+              (local.get $3)
+             )
+             (br_if $label$178
+              (i32.gt_u
+               (i32.add
+                (local.get $1)
+                (i32.load
+                 (local.tee $5
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 4)
+                  )
+                 )
+                )
+               )
+               (local.get $3)
+              )
+             )
+            )
+            (br_if $label$176
+             (i32.eqz
+              (local.tee $1
+               (i32.load offset=8
+                (local.get $2)
+               )
+              )
+             )
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (br $label$177)
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.tee $3
+             (i32.and
+              (i32.sub
+               (local.get $8)
+               (local.get $10)
+              )
+              (local.get $4)
+             )
+            )
+            (i32.const 2147483647)
+           )
+           (if
+            (i32.eq
+             (local.tee $1
+              (call $38
+               (local.get $3)
+              )
+             )
+             (i32.add
+              (i32.load
+               (local.get $2)
+              )
+              (i32.load
+               (local.get $5)
+              )
+             )
+            )
+            (br_if $label$172
+             (i32.ne
+              (local.get $1)
+              (i32.const -1)
+             )
+            )
+            (block
+             (local.set $2
+              (local.get $1)
+             )
+             (local.set $1
+              (local.get $3)
+             )
+             (br $label$175)
+            )
+           )
+          )
+          (br $label$174)
+         )
+         (if
+          (i32.ne
+           (local.tee $1
+            (call $38
+             (i32.const 0)
+            )
+           )
+           (i32.const -1)
+          )
+          (block
+           (local.set $2
+            (i32.sub
+             (i32.and
+              (i32.add
+               (local.tee $5
+                (i32.add
+                 (local.tee $2
+                  (i32.load
+                   (i32.const 4128)
+                  )
+                 )
+                 (i32.const -1)
+                )
+               )
+               (local.tee $3
+                (local.get $1)
+               )
+              )
+              (i32.sub
+               (i32.const 0)
+               (local.get $2)
+              )
+             )
+             (local.get $3)
+            )
+           )
+           (local.set $4
+            (i32.add
+             (local.tee $3
+              (i32.add
+               (if (result i32)
+                (i32.and
+                 (local.get $5)
+                 (local.get $3)
+                )
+                (local.get $2)
+                (i32.const 0)
+               )
+               (local.get $6)
+              )
+             )
+             (local.tee $5
+              (i32.load
+               (i32.const 4084)
+              )
+             )
+            )
+           )
+           (if
+            (i32.and
+             (i32.gt_u
+              (local.get $3)
+              (local.get $0)
+             )
+             (i32.lt_u
+              (local.get $3)
+              (i32.const 2147483647)
+             )
+            )
+            (block
+             (if
+              (local.tee $2
+               (i32.load
+                (i32.const 4092)
+               )
+              )
+              (br_if $label$174
+               (i32.or
+                (i32.le_u
+                 (local.get $4)
+                 (local.get $5)
+                )
+                (i32.gt_u
+                 (local.get $4)
+                 (local.get $2)
+                )
+               )
+              )
+             )
+             (br_if $label$172
+              (i32.eq
+               (local.tee $2
+                (call $38
+                 (local.get $3)
+                )
+               )
+               (local.get $1)
+              )
+             )
+             (local.set $1
+              (local.get $3)
+             )
+             (br $label$175)
+            )
+           )
+          )
+         )
+         (br $label$174)
+        )
+        (local.set $5
+         (i32.sub
+          (i32.const 0)
+          (local.get $1)
+         )
+        )
+        (if
+         (i32.and
+          (i32.gt_u
+           (local.get $7)
+           (local.get $1)
+          )
+          (i32.and
+           (i32.lt_u
+            (local.get $1)
+            (i32.const 2147483647)
+           )
+           (i32.ne
+            (local.get $2)
+            (i32.const -1)
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.tee $3
+            (i32.and
+             (i32.add
+              (i32.sub
+               (local.get $13)
+               (local.get $1)
+              )
+              (local.tee $3
+               (i32.load
+                (i32.const 4132)
+               )
+              )
+             )
+             (i32.sub
+              (i32.const 0)
+              (local.get $3)
+             )
+            )
+           )
+           (i32.const 2147483647)
+          )
+          (if
+           (i32.eq
+            (call $38
+             (local.get $3)
+            )
+            (i32.const -1)
+           )
+           (block
+            (drop
+             (call $38
+              (local.get $5)
+             )
+            )
+            (br $label$174)
+           )
+           (local.set $3
+            (i32.add
+             (local.get $3)
+             (local.get $1)
+            )
+           )
+          )
+          (local.set $3
+           (local.get $1)
+          )
+         )
+         (local.set $3
+          (local.get $1)
+         )
+        )
+        (if
+         (i32.ne
+          (local.get $2)
+          (i32.const -1)
+         )
+         (block
+          (local.set $1
+           (local.get $2)
+          )
+          (br $label$172)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 4096)
+        (i32.or
+         (i32.load
+          (i32.const 4096)
+         )
+         (i32.const 4)
+        )
+       )
+      )
+     )
+     (if
+      (i32.lt_u
+       (local.get $6)
+       (i32.const 2147483647)
+      )
+      (if
+       (i32.and
+        (i32.lt_u
+         (local.tee $1
+          (call $38
+           (local.get $6)
+          )
+         )
+         (local.tee $3
+          (call $38
+           (i32.const 0)
+          )
+         )
+        )
+        (i32.and
+         (i32.ne
+          (local.get $1)
+          (i32.const -1)
+         )
+         (i32.ne
+          (local.get $3)
+          (i32.const -1)
+         )
+        )
+       )
+       (br_if $label$172
+        (i32.gt_u
+         (local.tee $3
+          (i32.sub
+           (local.get $3)
+           (local.get $1)
+          )
+         )
+         (i32.add
+          (local.get $0)
+          (i32.const 40)
+         )
+        )
+       )
+      )
+     )
+     (br $label$171)
+    )
+    (i32.store
+     (i32.const 4084)
+     (local.tee $2
+      (i32.add
+       (i32.load
+        (i32.const 4084)
+       )
+       (local.get $3)
+      )
+     )
+    )
+    (if
+     (i32.gt_u
+      (local.get $2)
+      (i32.load
+       (i32.const 4088)
+      )
+     )
+     (i32.store
+      (i32.const 4088)
+      (local.get $2)
+     )
+    )
+    (block $label$198
+     (if
+      (local.tee $8
+       (i32.load
+        (i32.const 3676)
+       )
+      )
+      (block
+       (local.set $2
+        (i32.const 4100)
+       )
+       (block $label$200
+        (block $label$201
+         (loop $label$202
+          (br_if $label$201
+           (i32.eq
+            (local.get $1)
+            (i32.add
+             (local.tee $4
+              (i32.load
+               (local.get $2)
+              )
+             )
+             (local.tee $5
+              (i32.load
+               (local.tee $7
+                (i32.add
+                 (local.get $2)
+                 (i32.const 4)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (br_if $label$202
+           (local.tee $2
+            (i32.load offset=8
+             (local.get $2)
+            )
+           )
+          )
+         )
+         (br $label$200)
+        )
+        (if
+         (i32.eqz
+          (i32.and
+           (i32.load offset=12
+            (local.get $2)
+           )
+           (i32.const 8)
+          )
+         )
+         (if
+          (i32.and
+           (i32.lt_u
+            (local.get $8)
+            (local.get $1)
+           )
+           (i32.ge_u
+            (local.get $8)
+            (local.get $4)
+           )
+          )
+          (block
+           (i32.store
+            (local.get $7)
+            (i32.add
+             (local.get $5)
+             (local.get $3)
+            )
+           )
+           (local.set $5
+            (i32.load
+             (i32.const 3664)
+            )
+           )
+           (local.set $1
+            (i32.and
+             (i32.sub
+              (i32.const 0)
+              (local.tee $2
+               (i32.add
+                (local.get $8)
+                (i32.const 8)
+               )
+              )
+             )
+             (i32.const 7)
+            )
+           )
+           (i32.store
+            (i32.const 3676)
+            (local.tee $2
+             (i32.add
+              (local.get $8)
+              (if (result i32)
+               (i32.and
+                (local.get $2)
+                (i32.const 7)
+               )
+               (local.get $1)
+               (local.tee $1
+                (i32.const 0)
+               )
+              )
+             )
+            )
+           )
+           (i32.store
+            (i32.const 3664)
+            (local.tee $1
+             (i32.add
+              (i32.sub
+               (local.get $3)
+               (local.get $1)
+              )
+              (local.get $5)
+             )
+            )
+           )
+           (i32.store offset=4
+            (local.get $2)
+            (i32.or
+             (local.get $1)
+             (i32.const 1)
+            )
+           )
+           (i32.store offset=4
+            (i32.add
+             (local.get $2)
+             (local.get $1)
+            )
+            (i32.const 40)
+           )
+           (i32.store
+            (i32.const 3680)
+            (i32.load
+             (i32.const 4140)
+            )
+           )
+           (br $label$198)
+          )
+         )
+        )
+       )
+       (if
+        (i32.lt_u
+         (local.get $1)
+         (local.tee $2
+          (i32.load
+           (i32.const 3668)
+          )
+         )
+        )
+        (block
+         (i32.store
+          (i32.const 3668)
+          (local.get $1)
+         )
+         (local.set $2
+          (local.get $1)
+         )
+        )
+       )
+       (local.set $10
+        (i32.add
+         (local.get $1)
+         (local.get $3)
+        )
+       )
+       (local.set $5
+        (i32.const 4100)
+       )
+       (block $label$208
+        (block $label$209
+         (loop $label$210
+          (br_if $label$209
+           (i32.eq
+            (i32.load
+             (local.get $5)
+            )
+            (local.get $10)
+           )
+          )
+          (br_if $label$210
+           (local.tee $5
+            (i32.load offset=8
+             (local.get $5)
+            )
+           )
+          )
+          (local.set $5
+           (i32.const 4100)
+          )
+         )
+         (br $label$208)
+        )
+        (if
+         (i32.and
+          (i32.load offset=12
+           (local.get $5)
+          )
+          (i32.const 8)
+         )
+         (local.set $5
+          (i32.const 4100)
+         )
+         (block
+          (i32.store
+           (local.get $5)
+           (local.get $1)
+          )
+          (i32.store
+           (local.tee $5
+            (i32.add
+             (local.get $5)
+             (i32.const 4)
+            )
+           )
+           (i32.add
+            (i32.load
+             (local.get $5)
+            )
+            (local.get $3)
+           )
+          )
+          (local.set $7
+           (i32.and
+            (i32.sub
+             (i32.const 0)
+             (local.tee $4
+              (i32.add
+               (local.get $1)
+               (i32.const 8)
+              )
+             )
+            )
+            (i32.const 7)
+           )
+          )
+          (local.set $3
+           (i32.and
+            (i32.sub
+             (i32.const 0)
+             (local.tee $5
+              (i32.add
+               (local.get $10)
+               (i32.const 8)
+              )
+             )
+            )
+            (i32.const 7)
+           )
+          )
+          (local.set $6
+           (i32.add
+            (local.tee $13
+             (i32.add
+              (local.get $1)
+              (if (result i32)
+               (i32.and
+                (local.get $4)
+                (i32.const 7)
+               )
+               (local.get $7)
+               (i32.const 0)
+              )
+             )
+            )
+            (local.get $0)
+           )
+          )
+          (local.set $7
+           (i32.sub
+            (i32.sub
+             (local.tee $4
+              (i32.add
+               (local.get $10)
+               (if (result i32)
+                (i32.and
+                 (local.get $5)
+                 (i32.const 7)
+                )
+                (local.get $3)
+                (i32.const 0)
+               )
+              )
+             )
+             (local.get $13)
+            )
+            (local.get $0)
+           )
+          )
+          (i32.store offset=4
+           (local.get $13)
+           (i32.or
+            (local.get $0)
+            (i32.const 3)
+           )
+          )
+          (block $label$217
+           (if
+            (i32.eq
+             (local.get $4)
+             (local.get $8)
+            )
+            (block
+             (i32.store
+              (i32.const 3664)
+              (local.tee $0
+               (i32.add
+                (i32.load
+                 (i32.const 3664)
+                )
+                (local.get $7)
+               )
+              )
+             )
+             (i32.store
+              (i32.const 3676)
+              (local.get $6)
+             )
+             (i32.store offset=4
+              (local.get $6)
+              (i32.or
+               (local.get $0)
+               (i32.const 1)
+              )
+             )
+            )
+            (block
+             (if
+              (i32.eq
+               (local.get $4)
+               (i32.load
+                (i32.const 3672)
+               )
+              )
+              (block
+               (i32.store
+                (i32.const 3660)
+                (local.tee $0
+                 (i32.add
+                  (i32.load
+                   (i32.const 3660)
+                  )
+                  (local.get $7)
+                 )
+                )
+               )
+               (i32.store
+                (i32.const 3672)
+                (local.get $6)
+               )
+               (i32.store offset=4
+                (local.get $6)
+                (i32.or
+                 (local.get $0)
+                 (i32.const 1)
+                )
+               )
+               (i32.store
+                (i32.add
+                 (local.get $6)
+                 (local.get $0)
+                )
+                (local.get $0)
+               )
+               (br $label$217)
+              )
+             )
+             (i32.store
+              (local.tee $0
+               (i32.add
+                (local.tee $0
+                 (if (result i32)
+                  (i32.eq
+                   (i32.and
+                    (local.tee $0
+                     (i32.load offset=4
+                      (local.get $4)
+                     )
+                    )
+                    (i32.const 3)
+                   )
+                   (i32.const 1)
+                  )
+                  (block (result i32)
+                   (local.set $11
+                    (i32.and
+                     (local.get $0)
+                     (i32.const -8)
+                    )
+                   )
+                   (local.set $1
+                    (i32.shr_u
+                     (local.get $0)
+                     (i32.const 3)
+                    )
+                   )
+                   (block $label$222
+                    (if
+                     (i32.lt_u
+                      (local.get $0)
+                      (i32.const 256)
+                     )
+                     (block
+                      (local.set $5
+                       (i32.load offset=12
+                        (local.get $4)
+                       )
+                      )
+                      (block $label$224
+                       (if
+                        (i32.ne
+                         (local.tee $3
+                          (i32.load offset=8
+                           (local.get $4)
+                          )
+                         )
+                         (local.tee $0
+                          (i32.add
+                           (i32.shl
+                            (i32.shl
+                             (local.get $1)
+                             (i32.const 1)
+                            )
+                            (i32.const 2)
+                           )
+                           (i32.const 3692)
+                          )
+                         )
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.get $3)
+                           (local.get $2)
+                          )
+                          (call $fimport$10)
+                         )
+                         (br_if $label$224
+                          (i32.eq
+                           (i32.load offset=12
+                            (local.get $3)
+                           )
+                           (local.get $4)
+                          )
+                         )
+                         (call $fimport$10)
+                        )
+                       )
+                      )
+                      (if
+                       (i32.eq
+                        (local.get $5)
+                        (local.get $3)
+                       )
+                       (block
+                        (i32.store
+                         (i32.const 3652)
+                         (i32.and
+                          (i32.load
+                           (i32.const 3652)
+                          )
+                          (i32.xor
+                           (i32.shl
+                            (i32.const 1)
+                            (local.get $1)
+                           )
+                           (i32.const -1)
+                          )
+                         )
+                        )
+                        (br $label$222)
+                       )
+                      )
+                      (block $label$228
+                       (if
+                        (i32.eq
+                         (local.get $5)
+                         (local.get $0)
+                        )
+                        (local.set $20
+                         (i32.add
+                          (local.get $5)
+                          (i32.const 8)
+                         )
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.get $5)
+                           (local.get $2)
+                          )
+                          (call $fimport$10)
+                         )
+                         (if
+                          (i32.eq
+                           (i32.load
+                            (local.tee $0
+                             (i32.add
+                              (local.get $5)
+                              (i32.const 8)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (block
+                           (local.set $20
+                            (local.get $0)
+                           )
+                           (br $label$228)
+                          )
+                         )
+                         (call $fimport$10)
+                        )
+                       )
+                      )
+                      (i32.store offset=12
+                       (local.get $3)
+                       (local.get $5)
+                      )
+                      (i32.store
+                       (local.get $20)
+                       (local.get $3)
+                      )
+                     )
+                     (block
+                      (local.set $8
+                       (i32.load offset=24
+                        (local.get $4)
+                       )
+                      )
+                      (block $label$234
+                       (if
+                        (i32.eq
+                         (local.tee $0
+                          (i32.load offset=12
+                           (local.get $4)
+                          )
+                         )
+                         (local.get $4)
+                        )
+                        (block
+                         (if
+                          (i32.eqz
+                           (local.tee $0
+                            (i32.load
+                             (local.tee $1
+                              (i32.add
+                               (local.tee $3
+                                (i32.add
+                                 (local.get $4)
+                                 (i32.const 16)
+                                )
+                               )
+                               (i32.const 4)
+                              )
+                             )
+                            )
+                           )
+                          )
+                          (if
+                           (local.tee $0
+                            (i32.load
+                             (local.get $3)
+                            )
+                           )
+                           (local.set $1
+                            (local.get $3)
+                           )
+                           (block
+                            (local.set $12
+                             (i32.const 0)
+                            )
+                            (br $label$234)
+                           )
+                          )
+                         )
+                         (loop $label$239
+                          (if
+                           (local.tee $3
+                            (i32.load
+                             (local.tee $5
+                              (i32.add
+                               (local.get $0)
+                               (i32.const 20)
+                              )
+                             )
+                            )
+                           )
+                           (block
+                            (local.set $0
+                             (local.get $3)
+                            )
+                            (local.set $1
+                             (local.get $5)
+                            )
+                            (br $label$239)
+                           )
+                          )
+                          (if
+                           (local.tee $3
+                            (i32.load
+                             (local.tee $5
+                              (i32.add
+                               (local.get $0)
+                               (i32.const 16)
+                              )
+                             )
+                            )
+                           )
+                           (block
+                            (local.set $0
+                             (local.get $3)
+                            )
+                            (local.set $1
+                             (local.get $5)
+                            )
+                            (br $label$239)
+                           )
+                          )
+                         )
+                         (if
+                          (i32.lt_u
+                           (local.get $1)
+                           (local.get $2)
+                          )
+                          (call $fimport$10)
+                          (block
+                           (i32.store
+                            (local.get $1)
+                            (i32.const 0)
+                           )
+                           (local.set $12
+                            (local.get $0)
+                           )
+                          )
+                         )
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.tee $5
+                            (i32.load offset=8
+                             (local.get $4)
+                            )
+                           )
+                           (local.get $2)
+                          )
+                          (call $fimport$10)
+                         )
+                         (if
+                          (i32.ne
+                           (i32.load
+                            (local.tee $3
+                             (i32.add
+                              (local.get $5)
+                              (i32.const 12)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (call $fimport$10)
+                         )
+                         (if
+                          (i32.eq
+                           (i32.load
+                            (local.tee $1
+                             (i32.add
+                              (local.get $0)
+                              (i32.const 8)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (block
+                           (i32.store
+                            (local.get $3)
+                            (local.get $0)
+                           )
+                           (i32.store
+                            (local.get $1)
+                            (local.get $5)
+                           )
+                           (local.set $12
+                            (local.get $0)
+                           )
+                          )
+                          (call $fimport$10)
+                         )
+                        )
+                       )
+                      )
+                      (br_if $label$222
+                       (i32.eqz
+                        (local.get $8)
+                       )
+                      )
+                      (block $label$249
+                       (if
+                        (i32.eq
+                         (local.get $4)
+                         (i32.load
+                          (local.tee $0
+                           (i32.add
+                            (i32.shl
+                             (local.tee $1
+                              (i32.load offset=28
+                               (local.get $4)
+                              )
+                             )
+                             (i32.const 2)
+                            )
+                            (i32.const 3956)
+                           )
+                          )
+                         )
+                        )
+                        (block
+                         (i32.store
+                          (local.get $0)
+                          (local.get $12)
+                         )
+                         (br_if $label$249
+                          (local.get $12)
+                         )
+                         (i32.store
+                          (i32.const 3656)
+                          (i32.and
+                           (i32.load
+                            (i32.const 3656)
+                           )
+                           (i32.xor
+                            (i32.shl
+                             (i32.const 1)
+                             (local.get $1)
+                            )
+                            (i32.const -1)
+                           )
+                          )
+                         )
+                         (br $label$222)
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.get $8)
+                           (i32.load
+                            (i32.const 3668)
+                           )
+                          )
+                          (call $fimport$10)
+                         )
+                         (if
+                          (i32.eq
+                           (i32.load
+                            (local.tee $0
+                             (i32.add
+                              (local.get $8)
+                              (i32.const 16)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (i32.store
+                           (local.get $0)
+                           (local.get $12)
+                          )
+                          (i32.store offset=20
+                           (local.get $8)
+                           (local.get $12)
+                          )
+                         )
+                         (br_if $label$222
+                          (i32.eqz
+                           (local.get $12)
+                          )
+                         )
+                        )
+                       )
+                      )
+                      (if
+                       (i32.lt_u
+                        (local.get $12)
+                        (local.tee $1
+                         (i32.load
+                          (i32.const 3668)
+                         )
+                        )
+                       )
+                       (call $fimport$10)
+                      )
+                      (i32.store offset=24
+                       (local.get $12)
+                       (local.get $8)
+                      )
+                      (if
+                       (local.tee $3
+                        (i32.load
+                         (local.tee $0
+                          (i32.add
+                           (local.get $4)
+                           (i32.const 16)
+                          )
+                         )
+                        )
+                       )
+                       (if
+                        (i32.lt_u
+                         (local.get $3)
+                         (local.get $1)
+                        )
+                        (call $fimport$10)
+                        (block
+                         (i32.store offset=16
+                          (local.get $12)
+                          (local.get $3)
+                         )
+                         (i32.store offset=24
+                          (local.get $3)
+                          (local.get $12)
+                         )
+                        )
+                       )
+                      )
+                      (br_if $label$222
+                       (i32.eqz
+                        (local.tee $0
+                         (i32.load offset=4
+                          (local.get $0)
+                         )
+                        )
+                       )
+                      )
+                      (if
+                       (i32.lt_u
+                        (local.get $0)
+                        (i32.load
+                         (i32.const 3668)
+                        )
+                       )
+                       (call $fimport$10)
+                       (block
+                        (i32.store offset=20
+                         (local.get $12)
+                         (local.get $0)
+                        )
+                        (i32.store offset=24
+                         (local.get $0)
+                         (local.get $12)
+                        )
+                       )
+                      )
+                     )
+                    )
+                   )
+                   (local.set $7
+                    (i32.add
+                     (local.get $11)
+                     (local.get $7)
+                    )
+                   )
+                   (i32.add
+                    (local.get $4)
+                    (local.get $11)
+                   )
+                  )
+                  (local.get $4)
+                 )
+                )
+                (i32.const 4)
+               )
+              )
+              (i32.and
+               (i32.load
+                (local.get $0)
+               )
+               (i32.const -2)
+              )
+             )
+             (i32.store offset=4
+              (local.get $6)
+              (i32.or
+               (local.get $7)
+               (i32.const 1)
+              )
+             )
+             (i32.store
+              (i32.add
+               (local.get $6)
+               (local.get $7)
+              )
+              (local.get $7)
+             )
+             (local.set $0
+              (i32.shr_u
+               (local.get $7)
+               (i32.const 3)
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.get $7)
+               (i32.const 256)
+              )
+              (block
+               (local.set $3
+                (i32.add
+                 (i32.shl
+                  (i32.shl
+                   (local.get $0)
+                   (i32.const 1)
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 3692)
+                )
+               )
+               (block $label$263
+                (if
+                 (i32.and
+                  (local.tee $1
+                   (i32.load
+                    (i32.const 3652)
+                   )
+                  )
+                  (local.tee $0
+                   (i32.shl
+                    (i32.const 1)
+                    (local.get $0)
+                   )
+                  )
+                 )
+                 (block
+                  (if
+                   (i32.ge_u
+                    (local.tee $0
+                     (i32.load
+                      (local.tee $1
+                       (i32.add
+                        (local.get $3)
+                        (i32.const 8)
+                       )
+                      )
+                     )
+                    )
+                    (i32.load
+                     (i32.const 3668)
+                    )
+                   )
+                   (block
+                    (local.set $21
+                     (local.get $1)
+                    )
+                    (local.set $9
+                     (local.get $0)
+                    )
+                    (br $label$263)
+                   )
+                  )
+                  (call $fimport$10)
+                 )
+                 (block
+                  (i32.store
+                   (i32.const 3652)
+                   (i32.or
+                    (local.get $1)
+                    (local.get $0)
+                   )
+                  )
+                  (local.set $21
+                   (i32.add
+                    (local.get $3)
+                    (i32.const 8)
+                   )
+                  )
+                  (local.set $9
+                   (local.get $3)
+                  )
+                 )
+                )
+               )
+               (i32.store
+                (local.get $21)
+                (local.get $6)
+               )
+               (i32.store offset=12
+                (local.get $9)
+                (local.get $6)
+               )
+               (i32.store offset=8
+                (local.get $6)
+                (local.get $9)
+               )
+               (i32.store offset=12
+                (local.get $6)
+                (local.get $3)
+               )
+               (br $label$217)
+              )
+             )
+             (local.set $3
+              (i32.add
+               (i32.shl
+                (local.tee $2
+                 (block $label$267 (result i32)
+                  (if (result i32)
+                   (local.tee $0
+                    (i32.shr_u
+                     (local.get $7)
+                     (i32.const 8)
+                    )
+                   )
+                   (block (result i32)
+                    (drop
+                     (br_if $label$267
+                      (i32.const 31)
+                      (i32.gt_u
+                       (local.get $7)
+                       (i32.const 16777215)
+                      )
+                     )
+                    )
+                    (i32.or
+                     (i32.and
+                      (i32.shr_u
+                       (local.get $7)
+                       (i32.add
+                        (local.tee $0
+                         (i32.add
+                          (i32.sub
+                           (i32.const 14)
+                           (i32.or
+                            (i32.or
+                             (local.tee $0
+                              (i32.and
+                               (i32.shr_u
+                                (i32.add
+                                 (local.tee $1
+                                  (i32.shl
+                                   (local.get $0)
+                                   (local.tee $3
+                                    (i32.and
+                                     (i32.shr_u
+                                      (i32.add
+                                       (local.get $0)
+                                       (i32.const 1048320)
+                                      )
+                                      (i32.const 16)
+                                     )
+                                     (i32.const 8)
+                                    )
+                                   )
+                                  )
+                                 )
+                                 (i32.const 520192)
+                                )
+                                (i32.const 16)
+                               )
+                               (i32.const 4)
+                              )
+                             )
+                             (local.get $3)
+                            )
+                            (local.tee $0
+                             (i32.and
+                              (i32.shr_u
+                               (i32.add
+                                (local.tee $1
+                                 (i32.shl
+                                  (local.get $1)
+                                  (local.get $0)
+                                 )
+                                )
+                                (i32.const 245760)
+                               )
+                               (i32.const 16)
+                              )
+                              (i32.const 2)
+                             )
+                            )
+                           )
+                          )
+                          (i32.shr_u
+                           (i32.shl
+                            (local.get $1)
+                            (local.get $0)
+                           )
+                           (i32.const 15)
+                          )
+                         )
+                        )
+                        (i32.const 7)
+                       )
+                      )
+                      (i32.const 1)
+                     )
+                     (i32.shl
+                      (local.get $0)
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                   (i32.const 0)
+                  )
+                 )
+                )
+                (i32.const 2)
+               )
+               (i32.const 3956)
+              )
+             )
+             (i32.store offset=28
+              (local.get $6)
+              (local.get $2)
+             )
+             (i32.store offset=4
+              (local.tee $0
+               (i32.add
+                (local.get $6)
+                (i32.const 16)
+               )
+              )
+              (i32.const 0)
+             )
+             (i32.store
+              (local.get $0)
+              (i32.const 0)
+             )
+             (if
+              (i32.eqz
+               (i32.and
+                (local.tee $1
+                 (i32.load
+                  (i32.const 3656)
+                 )
+                )
+                (local.tee $0
+                 (i32.shl
+                  (i32.const 1)
+                  (local.get $2)
+                 )
+                )
+               )
+              )
+              (block
+               (i32.store
+                (i32.const 3656)
+                (i32.or
+                 (local.get $1)
+                 (local.get $0)
+                )
+               )
+               (i32.store
+                (local.get $3)
+                (local.get $6)
+               )
+               (i32.store offset=24
+                (local.get $6)
+                (local.get $3)
+               )
+               (i32.store offset=12
+                (local.get $6)
+                (local.get $6)
+               )
+               (i32.store offset=8
+                (local.get $6)
+                (local.get $6)
+               )
+               (br $label$217)
+              )
+             )
+             (local.set $0
+              (i32.load
+               (local.get $3)
+              )
+             )
+             (local.set $1
+              (i32.sub
+               (i32.const 25)
+               (i32.shr_u
+                (local.get $2)
+                (i32.const 1)
+               )
+              )
+             )
+             (local.set $2
+              (i32.shl
+               (local.get $7)
+               (if (result i32)
+                (i32.eq
+                 (local.get $2)
+                 (i32.const 31)
+                )
+                (i32.const 0)
+                (local.get $1)
+               )
+              )
+             )
+             (block $label$273
+              (block $label$274
+               (block $label$275
+                (loop $label$276
+                 (br_if $label$274
+                  (i32.eq
+                   (i32.and
+                    (i32.load offset=4
+                     (local.get $0)
+                    )
+                    (i32.const -8)
+                   )
+                   (local.get $7)
+                  )
+                 )
+                 (local.set $3
+                  (i32.shl
+                   (local.get $2)
+                   (i32.const 1)
+                  )
+                 )
+                 (br_if $label$275
+                  (i32.eqz
+                   (local.tee $1
+                    (i32.load
+                     (local.tee $2
+                      (i32.add
+                       (i32.add
+                        (local.get $0)
+                        (i32.const 16)
+                       )
+                       (i32.shl
+                        (i32.shr_u
+                         (local.get $2)
+                         (i32.const 31)
+                        )
+                        (i32.const 2)
+                       )
+                      )
+                     )
+                    )
+                   )
+                  )
+                 )
+                 (local.set $2
+                  (local.get $3)
+                 )
+                 (local.set $0
+                  (local.get $1)
+                 )
+                 (br $label$276)
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $2)
+                 (i32.load
+                  (i32.const 3668)
+                 )
+                )
+                (call $fimport$10)
+                (block
+                 (i32.store
+                  (local.get $2)
+                  (local.get $6)
+                 )
+                 (i32.store offset=24
+                  (local.get $6)
+                  (local.get $0)
+                 )
+                 (i32.store offset=12
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (i32.store offset=8
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (br $label$217)
+                )
+               )
+               (br $label$273)
+              )
+              (if
+               (i32.and
+                (i32.ge_u
+                 (local.tee $2
+                  (i32.load
+                   (local.tee $3
+                    (i32.add
+                     (local.get $0)
+                     (i32.const 8)
+                    )
+                   )
+                  )
+                 )
+                 (local.tee $1
+                  (i32.load
+                   (i32.const 3668)
+                  )
+                 )
+                )
+                (i32.ge_u
+                 (local.get $0)
+                 (local.get $1)
+                )
+               )
+               (block
+                (i32.store offset=12
+                 (local.get $2)
+                 (local.get $6)
+                )
+                (i32.store
+                 (local.get $3)
+                 (local.get $6)
+                )
+                (i32.store offset=8
+                 (local.get $6)
+                 (local.get $2)
+                )
+                (i32.store offset=12
+                 (local.get $6)
+                 (local.get $0)
+                )
+                (i32.store offset=24
+                 (local.get $6)
+                 (i32.const 0)
+                )
+               )
+               (call $fimport$10)
+              )
+             )
+            )
+           )
+          )
+          (global.set $global$1
+           (local.get $14)
+          )
+          (return
+           (i32.add
+            (local.get $13)
+            (i32.const 8)
+           )
+          )
+         )
+        )
+       )
+       (loop $label$281
+        (block $label$282
+         (if
+          (i32.le_u
+           (local.tee $2
+            (i32.load
+             (local.get $5)
+            )
+           )
+           (local.get $8)
+          )
+          (br_if $label$282
+           (i32.gt_u
+            (local.tee $13
+             (i32.add
+              (local.get $2)
+              (i32.load offset=4
+               (local.get $5)
+              )
+             )
+            )
+            (local.get $8)
+           )
+          )
+         )
+         (local.set $5
+          (i32.load offset=8
+           (local.get $5)
+          )
+         )
+         (br $label$281)
+        )
+       )
+       (local.set $2
+        (i32.and
+         (i32.sub
+          (i32.const 0)
+          (local.tee $5
+           (i32.add
+            (local.tee $7
+             (i32.add
+              (local.get $13)
+              (i32.const -47)
+             )
+            )
+            (i32.const 8)
+           )
+          )
+         )
+         (i32.const 7)
+        )
+       )
+       (local.set $10
+        (i32.add
+         (local.tee $7
+          (if (result i32)
+           (i32.lt_u
+            (local.tee $2
+             (i32.add
+              (local.get $7)
+              (if (result i32)
+               (i32.and
+                (local.get $5)
+                (i32.const 7)
+               )
+               (local.get $2)
+               (i32.const 0)
+              )
+             )
+            )
+            (local.tee $12
+             (i32.add
+              (local.get $8)
+              (i32.const 16)
+             )
+            )
+           )
+           (local.get $8)
+           (local.get $2)
+          )
+         )
+         (i32.const 8)
+        )
+       )
+       (local.set $5
+        (i32.add
+         (local.get $7)
+         (i32.const 24)
+        )
+       )
+       (local.set $9
+        (i32.add
+         (local.get $3)
+         (i32.const -40)
+        )
+       )
+       (local.set $2
+        (i32.and
+         (i32.sub
+          (i32.const 0)
+          (local.tee $4
+           (i32.add
+            (local.get $1)
+            (i32.const 8)
+           )
+          )
+         )
+         (i32.const 7)
+        )
+       )
+       (i32.store
+        (i32.const 3676)
+        (local.tee $4
+         (i32.add
+          (local.get $1)
+          (if (result i32)
+           (i32.and
+            (local.get $4)
+            (i32.const 7)
+           )
+           (local.get $2)
+           (local.tee $2
+            (i32.const 0)
+           )
+          )
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3664)
+        (local.tee $2
+         (i32.sub
+          (local.get $9)
+          (local.get $2)
+         )
+        )
+       )
+       (i32.store offset=4
+        (local.get $4)
+        (i32.or
+         (local.get $2)
+         (i32.const 1)
+        )
+       )
+       (i32.store offset=4
+        (i32.add
+         (local.get $4)
+         (local.get $2)
+        )
+        (i32.const 40)
+       )
+       (i32.store
+        (i32.const 3680)
+        (i32.load
+         (i32.const 4140)
+        )
+       )
+       (i32.store
+        (local.tee $2
+         (i32.add
+          (local.get $7)
+          (i32.const 4)
+         )
+        )
+        (i32.const 27)
+       )
+       (i64.store align=4
+        (local.get $10)
+        (i64.load align=4
+         (i32.const 4100)
+        )
+       )
+       (i64.store offset=8 align=4
+        (local.get $10)
+        (i64.load align=4
+         (i32.const 4108)
+        )
+       )
+       (i32.store
+        (i32.const 4100)
+        (local.get $1)
+       )
+       (i32.store
+        (i32.const 4104)
+        (local.get $3)
+       )
+       (i32.store
+        (i32.const 4112)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 4108)
+        (local.get $10)
+       )
+       (local.set $1
+        (local.get $5)
+       )
+       (loop $label$290
+        (i32.store
+         (local.tee $1
+          (i32.add
+           (local.get $1)
+           (i32.const 4)
+          )
+         )
+         (i32.const 7)
+        )
+        (br_if $label$290
+         (i32.lt_u
+          (i32.add
+           (local.get $1)
+           (i32.const 4)
+          )
+          (local.get $13)
+         )
+        )
+       )
+       (if
+        (i32.ne
+         (local.get $7)
+         (local.get $8)
+        )
+        (block
+         (i32.store
+          (local.get $2)
+          (i32.and
+           (i32.load
+            (local.get $2)
+           )
+           (i32.const -2)
+          )
+         )
+         (i32.store offset=4
+          (local.get $8)
+          (i32.or
+           (local.tee $4
+            (i32.sub
+             (local.get $7)
+             (local.get $8)
+            )
+           )
+           (i32.const 1)
+          )
+         )
+         (i32.store
+          (local.get $7)
+          (local.get $4)
+         )
+         (local.set $1
+          (i32.shr_u
+           (local.get $4)
+           (i32.const 3)
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $4)
+           (i32.const 256)
+          )
+          (block
+           (local.set $2
+            (i32.add
+             (i32.shl
+              (i32.shl
+               (local.get $1)
+               (i32.const 1)
+              )
+              (i32.const 2)
+             )
+             (i32.const 3692)
+            )
+           )
+           (if
+            (i32.and
+             (local.tee $3
+              (i32.load
+               (i32.const 3652)
+              )
+             )
+             (local.tee $1
+              (i32.shl
+               (i32.const 1)
+               (local.get $1)
+              )
+             )
+            )
+            (if
+             (i32.lt_u
+              (local.tee $1
+               (i32.load
+                (local.tee $3
+                 (i32.add
+                  (local.get $2)
+                  (i32.const 8)
+                 )
+                )
+               )
+              )
+              (i32.load
+               (i32.const 3668)
+              )
+             )
+             (call $fimport$10)
+             (block
+              (local.set $15
+               (local.get $3)
+              )
+              (local.set $11
+               (local.get $1)
+              )
+             )
+            )
+            (block
+             (i32.store
+              (i32.const 3652)
+              (i32.or
+               (local.get $3)
+               (local.get $1)
+              )
+             )
+             (local.set $15
+              (i32.add
+               (local.get $2)
+               (i32.const 8)
+              )
+             )
+             (local.set $11
+              (local.get $2)
+             )
+            )
+           )
+           (i32.store
+            (local.get $15)
+            (local.get $8)
+           )
+           (i32.store offset=12
+            (local.get $11)
+            (local.get $8)
+           )
+           (i32.store offset=8
+            (local.get $8)
+            (local.get $11)
+           )
+           (i32.store offset=12
+            (local.get $8)
+            (local.get $2)
+           )
+           (br $label$198)
+          )
+         )
+         (local.set $2
+          (i32.add
+           (i32.shl
+            (local.tee $5
+             (if (result i32)
+              (local.tee $1
+               (i32.shr_u
+                (local.get $4)
+                (i32.const 8)
+               )
+              )
+              (if (result i32)
+               (i32.gt_u
+                (local.get $4)
+                (i32.const 16777215)
+               )
+               (i32.const 31)
+               (i32.or
+                (i32.and
+                 (i32.shr_u
+                  (local.get $4)
+                  (i32.add
+                   (local.tee $1
+                    (i32.add
+                     (i32.sub
+                      (i32.const 14)
+                      (i32.or
+                       (i32.or
+                        (local.tee $1
+                         (i32.and
+                          (i32.shr_u
+                           (i32.add
+                            (local.tee $3
+                             (i32.shl
+                              (local.get $1)
+                              (local.tee $2
+                               (i32.and
+                                (i32.shr_u
+                                 (i32.add
+                                  (local.get $1)
+                                  (i32.const 1048320)
+                                 )
+                                 (i32.const 16)
+                                )
+                                (i32.const 8)
+                               )
+                              )
+                             )
+                            )
+                            (i32.const 520192)
+                           )
+                           (i32.const 16)
+                          )
+                          (i32.const 4)
+                         )
+                        )
+                        (local.get $2)
+                       )
+                       (local.tee $1
+                        (i32.and
+                         (i32.shr_u
+                          (i32.add
+                           (local.tee $3
+                            (i32.shl
+                             (local.get $3)
+                             (local.get $1)
+                            )
+                           )
+                           (i32.const 245760)
+                          )
+                          (i32.const 16)
+                         )
+                         (i32.const 2)
+                        )
+                       )
+                      )
+                     )
+                     (i32.shr_u
+                      (i32.shl
+                       (local.get $3)
+                       (local.get $1)
+                      )
+                      (i32.const 15)
+                     )
+                    )
+                   )
+                   (i32.const 7)
+                  )
+                 )
+                 (i32.const 1)
+                )
+                (i32.shl
+                 (local.get $1)
+                 (i32.const 1)
+                )
+               )
+              )
+              (i32.const 0)
+             )
+            )
+            (i32.const 2)
+           )
+           (i32.const 3956)
+          )
+         )
+         (i32.store offset=28
+          (local.get $8)
+          (local.get $5)
+         )
+         (i32.store offset=20
+          (local.get $8)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $12)
+          (i32.const 0)
+         )
+         (if
+          (i32.eqz
+           (i32.and
+            (local.tee $3
+             (i32.load
+              (i32.const 3656)
+             )
+            )
+            (local.tee $1
+             (i32.shl
+              (i32.const 1)
+              (local.get $5)
+             )
+            )
+           )
+          )
+          (block
+           (i32.store
+            (i32.const 3656)
+            (i32.or
+             (local.get $3)
+             (local.get $1)
+            )
+           )
+           (i32.store
+            (local.get $2)
+            (local.get $8)
+           )
+           (i32.store offset=24
+            (local.get $8)
+            (local.get $2)
+           )
+           (i32.store offset=12
+            (local.get $8)
+            (local.get $8)
+           )
+           (i32.store offset=8
+            (local.get $8)
+            (local.get $8)
+           )
+           (br $label$198)
+          )
+         )
+         (local.set $1
+          (i32.load
+           (local.get $2)
+          )
+         )
+         (local.set $3
+          (i32.sub
+           (i32.const 25)
+           (i32.shr_u
+            (local.get $5)
+            (i32.const 1)
+           )
+          )
+         )
+         (local.set $5
+          (i32.shl
+           (local.get $4)
+           (if (result i32)
+            (i32.eq
+             (local.get $5)
+             (i32.const 31)
+            )
+            (i32.const 0)
+            (local.get $3)
+           )
+          )
+         )
+         (block $label$304
+          (block $label$305
+           (block $label$306
+            (loop $label$307
+             (br_if $label$305
+              (i32.eq
+               (i32.and
+                (i32.load offset=4
+                 (local.get $1)
+                )
+                (i32.const -8)
+               )
+               (local.get $4)
+              )
+             )
+             (local.set $2
+              (i32.shl
+               (local.get $5)
+               (i32.const 1)
+              )
+             )
+             (br_if $label$306
+              (i32.eqz
+               (local.tee $3
+                (i32.load
+                 (local.tee $5
+                  (i32.add
+                   (i32.add
+                    (local.get $1)
+                    (i32.const 16)
+                   )
+                   (i32.shl
+                    (i32.shr_u
+                     (local.get $5)
+                     (i32.const 31)
+                    )
+                    (i32.const 2)
+                   )
+                  )
+                 )
+                )
+               )
+              )
+             )
+             (local.set $5
+              (local.get $2)
+             )
+             (local.set $1
+              (local.get $3)
+             )
+             (br $label$307)
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $5)
+             (i32.load
+              (i32.const 3668)
+             )
+            )
+            (call $fimport$10)
+            (block
+             (i32.store
+              (local.get $5)
+              (local.get $8)
+             )
+             (i32.store offset=24
+              (local.get $8)
+              (local.get $1)
+             )
+             (i32.store offset=12
+              (local.get $8)
+              (local.get $8)
+             )
+             (i32.store offset=8
+              (local.get $8)
+              (local.get $8)
+             )
+             (br $label$198)
+            )
+           )
+           (br $label$304)
+          )
+          (if
+           (i32.and
+            (i32.ge_u
+             (local.tee $5
+              (i32.load
+               (local.tee $2
+                (i32.add
+                 (local.get $1)
+                 (i32.const 8)
+                )
+               )
+              )
+             )
+             (local.tee $3
+              (i32.load
+               (i32.const 3668)
+              )
+             )
+            )
+            (i32.ge_u
+             (local.get $1)
+             (local.get $3)
+            )
+           )
+           (block
+            (i32.store offset=12
+             (local.get $5)
+             (local.get $8)
+            )
+            (i32.store
+             (local.get $2)
+             (local.get $8)
+            )
+            (i32.store offset=8
+             (local.get $8)
+             (local.get $5)
+            )
+            (i32.store offset=12
+             (local.get $8)
+             (local.get $1)
+            )
+            (i32.store offset=24
+             (local.get $8)
+             (i32.const 0)
+            )
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+       )
+      )
+      (block
+       (if
+        (i32.or
+         (i32.eqz
+          (local.tee $2
+           (i32.load
+            (i32.const 3668)
+           )
+          )
+         )
+         (i32.lt_u
+          (local.get $1)
+          (local.get $2)
+         )
+        )
+        (i32.store
+         (i32.const 3668)
+         (local.get $1)
+        )
+       )
+       (i32.store
+        (i32.const 4100)
+        (local.get $1)
+       )
+       (i32.store
+        (i32.const 4104)
+        (local.get $3)
+       )
+       (i32.store
+        (i32.const 4112)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 3688)
+        (i32.load
+         (i32.const 4124)
+        )
+       )
+       (i32.store
+        (i32.const 3684)
+        (i32.const -1)
+       )
+       (local.set $2
+        (i32.const 0)
+       )
+       (loop $label$314
+        (i32.store offset=12
+         (local.tee $5
+          (i32.add
+           (i32.shl
+            (i32.shl
+             (local.get $2)
+             (i32.const 1)
+            )
+            (i32.const 2)
+           )
+           (i32.const 3692)
+          )
+         )
+         (local.get $5)
+        )
+        (i32.store offset=8
+         (local.get $5)
+         (local.get $5)
+        )
+        (br_if $label$314
+         (i32.ne
+          (local.tee $2
+           (i32.add
+            (local.get $2)
+            (i32.const 1)
+           )
+          )
+          (i32.const 32)
+         )
+        )
+       )
+       (local.set $5
+        (i32.add
+         (local.get $3)
+         (i32.const -40)
+        )
+       )
+       (local.set $3
+        (i32.and
+         (i32.sub
+          (i32.const 0)
+          (local.tee $2
+           (i32.add
+            (local.get $1)
+            (i32.const 8)
+           )
+          )
+         )
+         (i32.const 7)
+        )
+       )
+       (i32.store
+        (i32.const 3676)
+        (local.tee $3
+         (i32.add
+          (local.get $1)
+          (local.tee $1
+           (if (result i32)
+            (i32.and
+             (local.get $2)
+             (i32.const 7)
+            )
+            (local.get $3)
+            (i32.const 0)
+           )
+          )
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3664)
+        (local.tee $1
+         (i32.sub
+          (local.get $5)
+          (local.get $1)
+         )
+        )
+       )
+       (i32.store offset=4
+        (local.get $3)
+        (i32.or
+         (local.get $1)
+         (i32.const 1)
+        )
+       )
+       (i32.store offset=4
+        (i32.add
+         (local.get $3)
+         (local.get $1)
+        )
+        (i32.const 40)
+       )
+       (i32.store
+        (i32.const 3680)
+        (i32.load
+         (i32.const 4140)
+        )
+       )
+      )
+     )
+    )
+    (if
+     (i32.gt_u
+      (local.tee $1
+       (i32.load
+        (i32.const 3664)
+       )
+      )
+      (local.get $0)
+     )
+     (block
+      (i32.store
+       (i32.const 3664)
+       (local.tee $3
+        (i32.sub
+         (local.get $1)
+         (local.get $0)
+        )
+       )
+      )
+      (i32.store
+       (i32.const 3676)
+       (local.tee $1
+        (i32.add
+         (local.tee $2
+          (i32.load
+           (i32.const 3676)
+          )
+         )
+         (local.get $0)
+        )
+       )
+      )
+      (i32.store offset=4
+       (local.get $1)
+       (i32.or
+        (local.get $3)
+        (i32.const 1)
+       )
+      )
+      (i32.store offset=4
+       (local.get $2)
+       (i32.or
+        (local.get $0)
+        (i32.const 3)
+       )
+      )
+      (global.set $global$1
+       (local.get $14)
+      )
+      (return
+       (i32.add
+        (local.get $2)
+        (i32.const 8)
+       )
+      )
+     )
+    )
+   )
+   (i32.store
+    (call $12)
+    (i32.const 12)
+   )
+   (global.set $global$1
+    (local.get $14)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $36 (; 49 ;) (type $2) (param $0 i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (block $label$1
+   (if
+    (i32.eqz
+     (local.get $0)
+    )
+    (return)
+   )
+   (if
+    (i32.lt_u
+     (local.tee $1
+      (i32.add
+       (local.get $0)
+       (i32.const -8)
+      )
+     )
+     (local.tee $11
+      (i32.load
+       (i32.const 3668)
+      )
+     )
+    )
+    (call $fimport$10)
+   )
+   (if
+    (i32.eq
+     (local.tee $8
+      (i32.and
+       (local.tee $0
+        (i32.load
+         (i32.add
+          (local.get $0)
+          (i32.const -4)
+         )
+        )
+       )
+       (i32.const 3)
+      )
+     )
+     (i32.const 1)
+    )
+    (call $fimport$10)
+   )
+   (local.set $6
+    (i32.add
+     (local.get $1)
+     (local.tee $4
+      (i32.and
+       (local.get $0)
+       (i32.const -8)
+      )
+     )
+    )
+   )
+   (block $label$5
+    (if
+     (i32.and
+      (local.get $0)
+      (i32.const 1)
+     )
+     (block
+      (local.set $3
+       (local.get $1)
+      )
+      (local.set $2
+       (local.get $4)
+      )
+     )
+     (block
+      (if
+       (i32.eqz
+        (local.get $8)
+       )
+       (return)
+      )
+      (if
+       (i32.lt_u
+        (local.tee $0
+         (i32.add
+          (local.get $1)
+          (i32.sub
+           (i32.const 0)
+           (local.tee $8
+            (i32.load
+             (local.get $1)
+            )
+           )
+          )
+         )
+        )
+        (local.get $11)
+       )
+       (call $fimport$10)
+      )
+      (local.set $1
+       (i32.add
+        (local.get $8)
+        (local.get $4)
+       )
+      )
+      (if
+       (i32.eq
+        (local.get $0)
+        (i32.load
+         (i32.const 3672)
+        )
+       )
+       (block
+        (if
+         (i32.ne
+          (i32.and
+           (local.tee $3
+            (i32.load
+             (local.tee $2
+              (i32.add
+               (local.get $6)
+               (i32.const 4)
+              )
+             )
+            )
+           )
+           (i32.const 3)
+          )
+          (i32.const 3)
+         )
+         (block
+          (local.set $3
+           (local.get $0)
+          )
+          (local.set $2
+           (local.get $1)
+          )
+          (br $label$5)
+         )
+        )
+        (i32.store
+         (i32.const 3660)
+         (local.get $1)
+        )
+        (i32.store
+         (local.get $2)
+         (i32.and
+          (local.get $3)
+          (i32.const -2)
+         )
+        )
+        (i32.store offset=4
+         (local.get $0)
+         (i32.or
+          (local.get $1)
+          (i32.const 1)
+         )
+        )
+        (i32.store
+         (i32.add
+          (local.get $0)
+          (local.get $1)
+         )
+         (local.get $1)
+        )
+        (return)
+       )
+      )
+      (local.set $10
+       (i32.shr_u
+        (local.get $8)
+        (i32.const 3)
+       )
+      )
+      (if
+       (i32.lt_u
+        (local.get $8)
+        (i32.const 256)
+       )
+       (block
+        (local.set $3
+         (i32.load offset=12
+          (local.get $0)
+         )
+        )
+        (if
+         (i32.ne
+          (local.tee $4
+           (i32.load offset=8
+            (local.get $0)
+           )
+          )
+          (local.tee $2
+           (i32.add
+            (i32.shl
+             (i32.shl
+              (local.get $10)
+              (i32.const 1)
+             )
+             (i32.const 2)
+            )
+            (i32.const 3692)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $4)
+            (local.get $11)
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.ne
+            (i32.load offset=12
+             (local.get $4)
+            )
+            (local.get $0)
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $3)
+          (local.get $4)
+         )
+         (block
+          (i32.store
+           (i32.const 3652)
+           (i32.and
+            (i32.load
+             (i32.const 3652)
+            )
+            (i32.xor
+             (i32.shl
+              (i32.const 1)
+              (local.get $10)
+             )
+             (i32.const -1)
+            )
+           )
+          )
+          (local.set $3
+           (local.get $0)
+          )
+          (local.set $2
+           (local.get $1)
+          )
+          (br $label$5)
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $3)
+          (local.get $2)
+         )
+         (local.set $5
+          (i32.add
+           (local.get $3)
+           (i32.const 8)
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $3)
+            (local.get $11)
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $2
+              (i32.add
+               (local.get $3)
+               (i32.const 8)
+              )
+             )
+            )
+            (local.get $0)
+           )
+           (local.set $5
+            (local.get $2)
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (i32.store offset=12
+         (local.get $4)
+         (local.get $3)
+        )
+        (i32.store
+         (local.get $5)
+         (local.get $4)
+        )
+        (local.set $3
+         (local.get $0)
+        )
+        (local.set $2
+         (local.get $1)
+        )
+        (br $label$5)
+       )
+      )
+      (local.set $12
+       (i32.load offset=24
+        (local.get $0)
+       )
+      )
+      (block $label$22
+       (if
+        (i32.eq
+         (local.tee $4
+          (i32.load offset=12
+           (local.get $0)
+          )
+         )
+         (local.get $0)
+        )
+        (block
+         (if
+          (local.tee $4
+           (i32.load
+            (local.tee $8
+             (i32.add
+              (local.tee $5
+               (i32.add
+                (local.get $0)
+                (i32.const 16)
+               )
+              )
+              (i32.const 4)
+             )
+            )
+           )
+          )
+          (local.set $5
+           (local.get $8)
+          )
+          (if
+           (i32.eqz
+            (local.tee $4
+             (i32.load
+              (local.get $5)
+             )
+            )
+           )
+           (block
+            (local.set $7
+             (i32.const 0)
+            )
+            (br $label$22)
+           )
+          )
+         )
+         (loop $label$27
+          (if
+           (local.tee $10
+            (i32.load
+             (local.tee $8
+              (i32.add
+               (local.get $4)
+               (i32.const 20)
+              )
+             )
+            )
+           )
+           (block
+            (local.set $4
+             (local.get $10)
+            )
+            (local.set $5
+             (local.get $8)
+            )
+            (br $label$27)
+           )
+          )
+          (if
+           (local.tee $10
+            (i32.load
+             (local.tee $8
+              (i32.add
+               (local.get $4)
+               (i32.const 16)
+              )
+             )
+            )
+           )
+           (block
+            (local.set $4
+             (local.get $10)
+            )
+            (local.set $5
+             (local.get $8)
+            )
+            (br $label$27)
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $5)
+           (local.get $11)
+          )
+          (call $fimport$10)
+          (block
+           (i32.store
+            (local.get $5)
+            (i32.const 0)
+           )
+           (local.set $7
+            (local.get $4)
+           )
+          )
+         )
+        )
+        (block
+         (if
+          (i32.lt_u
+           (local.tee $5
+            (i32.load offset=8
+             (local.get $0)
+            )
+           )
+           (local.get $11)
+          )
+          (call $fimport$10)
+         )
+         (if
+          (i32.ne
+           (i32.load
+            (local.tee $8
+             (i32.add
+              (local.get $5)
+              (i32.const 12)
+             )
+            )
+           )
+           (local.get $0)
+          )
+          (call $fimport$10)
+         )
+         (if
+          (i32.eq
+           (i32.load
+            (local.tee $10
+             (i32.add
+              (local.get $4)
+              (i32.const 8)
+             )
+            )
+           )
+           (local.get $0)
+          )
+          (block
+           (i32.store
+            (local.get $8)
+            (local.get $4)
+           )
+           (i32.store
+            (local.get $10)
+            (local.get $5)
+           )
+           (local.set $7
+            (local.get $4)
+           )
+          )
+          (call $fimport$10)
+         )
+        )
+       )
+      )
+      (if
+       (local.get $12)
+       (block
+        (if
+         (i32.eq
+          (local.get $0)
+          (i32.load
+           (local.tee $5
+            (i32.add
+             (i32.shl
+              (local.tee $4
+               (i32.load offset=28
+                (local.get $0)
+               )
+              )
+              (i32.const 2)
+             )
+             (i32.const 3956)
+            )
+           )
+          )
+         )
+         (block
+          (i32.store
+           (local.get $5)
+           (local.get $7)
+          )
+          (if
+           (i32.eqz
+            (local.get $7)
+           )
+           (block
+            (i32.store
+             (i32.const 3656)
+             (i32.and
+              (i32.load
+               (i32.const 3656)
+              )
+              (i32.xor
+               (i32.shl
+                (i32.const 1)
+                (local.get $4)
+               )
+               (i32.const -1)
+              )
+             )
+            )
+            (local.set $3
+             (local.get $0)
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (br $label$5)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $12)
+            (i32.load
+             (i32.const 3668)
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $4
+              (i32.add
+               (local.get $12)
+               (i32.const 16)
+              )
+             )
+            )
+            (local.get $0)
+           )
+           (i32.store
+            (local.get $4)
+            (local.get $7)
+           )
+           (i32.store offset=20
+            (local.get $12)
+            (local.get $7)
+           )
+          )
+          (if
+           (i32.eqz
+            (local.get $7)
+           )
+           (block
+            (local.set $3
+             (local.get $0)
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (br $label$5)
+           )
+          )
+         )
+        )
+        (if
+         (i32.lt_u
+          (local.get $7)
+          (local.tee $5
+           (i32.load
+            (i32.const 3668)
+           )
+          )
+         )
+         (call $fimport$10)
+        )
+        (i32.store offset=24
+         (local.get $7)
+         (local.get $12)
+        )
+        (if
+         (local.tee $4
+          (i32.load
+           (local.tee $8
+            (i32.add
+             (local.get $0)
+             (i32.const 16)
+            )
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $4)
+           (local.get $5)
+          )
+          (call $fimport$10)
+          (block
+           (i32.store offset=16
+            (local.get $7)
+            (local.get $4)
+           )
+           (i32.store offset=24
+            (local.get $4)
+            (local.get $7)
+           )
+          )
+         )
+        )
+        (if
+         (local.tee $4
+          (i32.load offset=4
+           (local.get $8)
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $4)
+           (i32.load
+            (i32.const 3668)
+           )
+          )
+          (call $fimport$10)
+          (block
+           (i32.store offset=20
+            (local.get $7)
+            (local.get $4)
+           )
+           (i32.store offset=24
+            (local.get $4)
+            (local.get $7)
+           )
+           (local.set $3
+            (local.get $0)
+           )
+           (local.set $2
+            (local.get $1)
+           )
+          )
+         )
+         (block
+          (local.set $3
+           (local.get $0)
+          )
+          (local.set $2
+           (local.get $1)
+          )
+         )
+        )
+       )
+       (block
+        (local.set $3
+         (local.get $0)
+        )
+        (local.set $2
+         (local.get $1)
+        )
+       )
+      )
+     )
+    )
+   )
+   (if
+    (i32.ge_u
+     (local.get $3)
+     (local.get $6)
+    )
+    (call $fimport$10)
+   )
+   (if
+    (i32.eqz
+     (i32.and
+      (local.tee $0
+       (i32.load
+        (local.tee $1
+         (i32.add
+          (local.get $6)
+          (i32.const 4)
+         )
+        )
+       )
+      )
+      (i32.const 1)
+     )
+    )
+    (call $fimport$10)
+   )
+   (if
+    (i32.and
+     (local.get $0)
+     (i32.const 2)
+    )
+    (block
+     (i32.store
+      (local.get $1)
+      (i32.and
+       (local.get $0)
+       (i32.const -2)
+      )
+     )
+     (i32.store offset=4
+      (local.get $3)
+      (i32.or
+       (local.get $2)
+       (i32.const 1)
+      )
+     )
+     (i32.store
+      (i32.add
+       (local.get $3)
+       (local.get $2)
+      )
+      (local.get $2)
+     )
+    )
+    (block
+     (if
+      (i32.eq
+       (local.get $6)
+       (i32.load
+        (i32.const 3676)
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3664)
+        (local.tee $0
+         (i32.add
+          (i32.load
+           (i32.const 3664)
+          )
+          (local.get $2)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3676)
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $3)
+        (i32.or
+         (local.get $0)
+         (i32.const 1)
+        )
+       )
+       (if
+        (i32.ne
+         (local.get $3)
+         (i32.load
+          (i32.const 3672)
+         )
+        )
+        (return)
+       )
+       (i32.store
+        (i32.const 3672)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 3660)
+        (i32.const 0)
+       )
+       (return)
+      )
+     )
+     (if
+      (i32.eq
+       (local.get $6)
+       (i32.load
+        (i32.const 3672)
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3660)
+        (local.tee $0
+         (i32.add
+          (i32.load
+           (i32.const 3660)
+          )
+          (local.get $2)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3672)
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $3)
+        (i32.or
+         (local.get $0)
+         (i32.const 1)
+        )
+       )
+       (i32.store
+        (i32.add
+         (local.get $3)
+         (local.get $0)
+        )
+        (local.get $0)
+       )
+       (return)
+      )
+     )
+     (local.set $5
+      (i32.add
+       (i32.and
+        (local.get $0)
+        (i32.const -8)
+       )
+       (local.get $2)
+      )
+     )
+     (local.set $4
+      (i32.shr_u
+       (local.get $0)
+       (i32.const 3)
+      )
+     )
+     (block $label$61
+      (if
+       (i32.lt_u
+        (local.get $0)
+        (i32.const 256)
+       )
+       (block
+        (local.set $2
+         (i32.load offset=12
+          (local.get $6)
+         )
+        )
+        (if
+         (i32.ne
+          (local.tee $1
+           (i32.load offset=8
+            (local.get $6)
+           )
+          )
+          (local.tee $0
+           (i32.add
+            (i32.shl
+             (i32.shl
+              (local.get $4)
+              (i32.const 1)
+             )
+             (i32.const 2)
+            )
+            (i32.const 3692)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $1)
+            (i32.load
+             (i32.const 3668)
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.ne
+            (i32.load offset=12
+             (local.get $1)
+            )
+            (local.get $6)
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $2)
+          (local.get $1)
+         )
+         (block
+          (i32.store
+           (i32.const 3652)
+           (i32.and
+            (i32.load
+             (i32.const 3652)
+            )
+            (i32.xor
+             (i32.shl
+              (i32.const 1)
+              (local.get $4)
+             )
+             (i32.const -1)
+            )
+           )
+          )
+          (br $label$61)
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $2)
+          (local.get $0)
+         )
+         (local.set $14
+          (i32.add
+           (local.get $2)
+           (i32.const 8)
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $2)
+            (i32.load
+             (i32.const 3668)
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $0
+              (i32.add
+               (local.get $2)
+               (i32.const 8)
+              )
+             )
+            )
+            (local.get $6)
+           )
+           (local.set $14
+            (local.get $0)
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (i32.store offset=12
+         (local.get $1)
+         (local.get $2)
+        )
+        (i32.store
+         (local.get $14)
+         (local.get $1)
+        )
+       )
+       (block
+        (local.set $7
+         (i32.load offset=24
+          (local.get $6)
+         )
+        )
+        (block $label$73
+         (if
+          (i32.eq
+           (local.tee $0
+            (i32.load offset=12
+             (local.get $6)
+            )
+           )
+           (local.get $6)
+          )
+          (block
+           (if
+            (local.tee $0
+             (i32.load
+              (local.tee $1
+               (i32.add
+                (local.tee $2
+                 (i32.add
+                  (local.get $6)
+                  (i32.const 16)
+                 )
+                )
+                (i32.const 4)
+               )
+              )
+             )
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (if
+             (i32.eqz
+              (local.tee $0
+               (i32.load
+                (local.get $2)
+               )
+              )
+             )
+             (block
+              (local.set $9
+               (i32.const 0)
+              )
+              (br $label$73)
+             )
+            )
+           )
+           (loop $label$78
+            (if
+             (local.tee $4
+              (i32.load
+               (local.tee $1
+                (i32.add
+                 (local.get $0)
+                 (i32.const 20)
+                )
+               )
+              )
+             )
+             (block
+              (local.set $0
+               (local.get $4)
+              )
+              (local.set $2
+               (local.get $1)
+              )
+              (br $label$78)
+             )
+            )
+            (if
+             (local.tee $4
+              (i32.load
+               (local.tee $1
+                (i32.add
+                 (local.get $0)
+                 (i32.const 16)
+                )
+               )
+              )
+             )
+             (block
+              (local.set $0
+               (local.get $4)
+              )
+              (local.set $2
+               (local.get $1)
+              )
+              (br $label$78)
+             )
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $2)
+             (i32.load
+              (i32.const 3668)
+             )
+            )
+            (call $fimport$10)
+            (block
+             (i32.store
+              (local.get $2)
+              (i32.const 0)
+             )
+             (local.set $9
+              (local.get $0)
+             )
+            )
+           )
+          )
+          (block
+           (if
+            (i32.lt_u
+             (local.tee $2
+              (i32.load offset=8
+               (local.get $6)
+              )
+             )
+             (i32.load
+              (i32.const 3668)
+             )
+            )
+            (call $fimport$10)
+           )
+           (if
+            (i32.ne
+             (i32.load
+              (local.tee $1
+               (i32.add
+                (local.get $2)
+                (i32.const 12)
+               )
+              )
+             )
+             (local.get $6)
+            )
+            (call $fimport$10)
+           )
+           (if
+            (i32.eq
+             (i32.load
+              (local.tee $4
+               (i32.add
+                (local.get $0)
+                (i32.const 8)
+               )
+              )
+             )
+             (local.get $6)
+            )
+            (block
+             (i32.store
+              (local.get $1)
+              (local.get $0)
+             )
+             (i32.store
+              (local.get $4)
+              (local.get $2)
+             )
+             (local.set $9
+              (local.get $0)
+             )
+            )
+            (call $fimport$10)
+           )
+          )
+         )
+        )
+        (if
+         (local.get $7)
+         (block
+          (if
+           (i32.eq
+            (local.get $6)
+            (i32.load
+             (local.tee $2
+              (i32.add
+               (i32.shl
+                (local.tee $0
+                 (i32.load offset=28
+                  (local.get $6)
+                 )
+                )
+                (i32.const 2)
+               )
+               (i32.const 3956)
+              )
+             )
+            )
+           )
+           (block
+            (i32.store
+             (local.get $2)
+             (local.get $9)
+            )
+            (if
+             (i32.eqz
+              (local.get $9)
+             )
+             (block
+              (i32.store
+               (i32.const 3656)
+               (i32.and
+                (i32.load
+                 (i32.const 3656)
+                )
+                (i32.xor
+                 (i32.shl
+                  (i32.const 1)
+                  (local.get $0)
+                 )
+                 (i32.const -1)
+                )
+               )
+              )
+              (br $label$61)
+             )
+            )
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.get $7)
+              (i32.load
+               (i32.const 3668)
+              )
+             )
+             (call $fimport$10)
+            )
+            (if
+             (i32.eq
+              (i32.load
+               (local.tee $0
+                (i32.add
+                 (local.get $7)
+                 (i32.const 16)
+                )
+               )
+              )
+              (local.get $6)
+             )
+             (i32.store
+              (local.get $0)
+              (local.get $9)
+             )
+             (i32.store offset=20
+              (local.get $7)
+              (local.get $9)
+             )
+            )
+            (br_if $label$61
+             (i32.eqz
+              (local.get $9)
+             )
+            )
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.get $9)
+            (local.tee $2
+             (i32.load
+              (i32.const 3668)
+             )
+            )
+           )
+           (call $fimport$10)
+          )
+          (i32.store offset=24
+           (local.get $9)
+           (local.get $7)
+          )
+          (if
+           (local.tee $0
+            (i32.load
+             (local.tee $1
+              (i32.add
+               (local.get $6)
+               (i32.const 16)
+              )
+             )
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $0)
+             (local.get $2)
+            )
+            (call $fimport$10)
+            (block
+             (i32.store offset=16
+              (local.get $9)
+              (local.get $0)
+             )
+             (i32.store offset=24
+              (local.get $0)
+              (local.get $9)
+             )
+            )
+           )
+          )
+          (if
+           (local.tee $0
+            (i32.load offset=4
+             (local.get $1)
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $0)
+             (i32.load
+              (i32.const 3668)
+             )
+            )
+            (call $fimport$10)
+            (block
+             (i32.store offset=20
+              (local.get $9)
+              (local.get $0)
+             )
+             (i32.store offset=24
+              (local.get $0)
+              (local.get $9)
+             )
+            )
+           )
+          )
+         )
+        )
+       )
+      )
+     )
+     (i32.store offset=4
+      (local.get $3)
+      (i32.or
+       (local.get $5)
+       (i32.const 1)
+      )
+     )
+     (i32.store
+      (i32.add
+       (local.get $3)
+       (local.get $5)
+      )
+      (local.get $5)
+     )
+     (if
+      (i32.eq
+       (local.get $3)
+       (i32.load
+        (i32.const 3672)
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3660)
+        (local.get $5)
+       )
+       (return)
+      )
+      (local.set $2
+       (local.get $5)
+      )
+     )
+    )
+   )
+   (local.set $1
+    (i32.shr_u
+     (local.get $2)
+     (i32.const 3)
+    )
+   )
+   (if
+    (i32.lt_u
+     (local.get $2)
+     (i32.const 256)
+    )
+    (block
+     (local.set $0
+      (i32.add
+       (i32.shl
+        (i32.shl
+         (local.get $1)
+         (i32.const 1)
+        )
+        (i32.const 2)
+       )
+       (i32.const 3692)
+      )
+     )
+     (if
+      (i32.and
+       (local.tee $2
+        (i32.load
+         (i32.const 3652)
+        )
+       )
+       (local.tee $1
+        (i32.shl
+         (i32.const 1)
+         (local.get $1)
+        )
+       )
+      )
+      (if
+       (i32.lt_u
+        (local.tee $1
+         (i32.load
+          (local.tee $2
+           (i32.add
+            (local.get $0)
+            (i32.const 8)
+           )
+          )
+         )
+        )
+        (i32.load
+         (i32.const 3668)
+        )
+       )
+       (call $fimport$10)
+       (block
+        (local.set $15
+         (local.get $2)
+        )
+        (local.set $13
+         (local.get $1)
+        )
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3652)
+        (i32.or
+         (local.get $2)
+         (local.get $1)
+        )
+       )
+       (local.set $15
+        (i32.add
+         (local.get $0)
+         (i32.const 8)
+        )
+       )
+       (local.set $13
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store
+      (local.get $15)
+      (local.get $3)
+     )
+     (i32.store offset=12
+      (local.get $13)
+      (local.get $3)
+     )
+     (i32.store offset=8
+      (local.get $3)
+      (local.get $13)
+     )
+     (i32.store offset=12
+      (local.get $3)
+      (local.get $0)
+     )
+     (return)
+    )
+   )
+   (local.set $0
+    (i32.add
+     (i32.shl
+      (local.tee $1
+       (if (result i32)
+        (local.tee $0
+         (i32.shr_u
+          (local.get $2)
+          (i32.const 8)
+         )
+        )
+        (if (result i32)
+         (i32.gt_u
+          (local.get $2)
+          (i32.const 16777215)
+         )
+         (i32.const 31)
+         (i32.or
+          (i32.and
+           (i32.shr_u
+            (local.get $2)
+            (i32.add
+             (local.tee $0
+              (i32.add
+               (i32.sub
+                (i32.const 14)
+                (i32.or
+                 (i32.or
+                  (local.tee $4
+                   (i32.and
+                    (i32.shr_u
+                     (i32.add
+                      (local.tee $1
+                       (i32.shl
+                        (local.get $0)
+                        (local.tee $0
+                         (i32.and
+                          (i32.shr_u
+                           (i32.add
+                            (local.get $0)
+                            (i32.const 1048320)
+                           )
+                           (i32.const 16)
+                          )
+                          (i32.const 8)
+                         )
+                        )
+                       )
+                      )
+                      (i32.const 520192)
+                     )
+                     (i32.const 16)
+                    )
+                    (i32.const 4)
+                   )
+                  )
+                  (local.get $0)
+                 )
+                 (local.tee $1
+                  (i32.and
+                   (i32.shr_u
+                    (i32.add
+                     (local.tee $0
+                      (i32.shl
+                       (local.get $1)
+                       (local.get $4)
+                      )
+                     )
+                     (i32.const 245760)
+                    )
+                    (i32.const 16)
+                   )
+                   (i32.const 2)
+                  )
+                 )
+                )
+               )
+               (i32.shr_u
+                (i32.shl
+                 (local.get $0)
+                 (local.get $1)
+                )
+                (i32.const 15)
+               )
+              )
+             )
+             (i32.const 7)
+            )
+           )
+           (i32.const 1)
+          )
+          (i32.shl
+           (local.get $0)
+           (i32.const 1)
+          )
+         )
+        )
+        (i32.const 0)
+       )
+      )
+      (i32.const 2)
+     )
+     (i32.const 3956)
+    )
+   )
+   (i32.store offset=28
+    (local.get $3)
+    (local.get $1)
+   )
+   (i32.store offset=20
+    (local.get $3)
+    (i32.const 0)
+   )
+   (i32.store offset=16
+    (local.get $3)
+    (i32.const 0)
+   )
+   (block $label$113
+    (if
+     (i32.and
+      (local.tee $4
+       (i32.load
+        (i32.const 3656)
+       )
+      )
+      (local.tee $5
+       (i32.shl
+        (i32.const 1)
+        (local.get $1)
+       )
+      )
+     )
+     (block
+      (local.set $0
+       (i32.load
+        (local.get $0)
+       )
+      )
+      (local.set $4
+       (i32.sub
+        (i32.const 25)
+        (i32.shr_u
+         (local.get $1)
+         (i32.const 1)
+        )
+       )
+      )
+      (local.set $1
+       (i32.shl
+        (local.get $2)
+        (if (result i32)
+         (i32.eq
+          (local.get $1)
+          (i32.const 31)
+         )
+         (i32.const 0)
+         (local.get $4)
+        )
+       )
+      )
+      (block $label$117
+       (block $label$118
+        (block $label$119
+         (loop $label$120
+          (br_if $label$118
+           (i32.eq
+            (i32.and
+             (i32.load offset=4
+              (local.get $0)
+             )
+             (i32.const -8)
+            )
+            (local.get $2)
+           )
+          )
+          (local.set $4
+           (i32.shl
+            (local.get $1)
+            (i32.const 1)
+           )
+          )
+          (br_if $label$119
+           (i32.eqz
+            (local.tee $5
+             (i32.load
+              (local.tee $1
+               (i32.add
+                (i32.add
+                 (local.get $0)
+                 (i32.const 16)
+                )
+                (i32.shl
+                 (i32.shr_u
+                  (local.get $1)
+                  (i32.const 31)
+                 )
+                 (i32.const 2)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (local.set $1
+           (local.get $4)
+          )
+          (local.set $0
+           (local.get $5)
+          )
+          (br $label$120)
+         )
+        )
+        (if
+         (i32.lt_u
+          (local.get $1)
+          (i32.load
+           (i32.const 3668)
+          )
+         )
+         (call $fimport$10)
+         (block
+          (i32.store
+           (local.get $1)
+           (local.get $3)
+          )
+          (i32.store offset=24
+           (local.get $3)
+           (local.get $0)
+          )
+          (i32.store offset=12
+           (local.get $3)
+           (local.get $3)
+          )
+          (i32.store offset=8
+           (local.get $3)
+           (local.get $3)
+          )
+          (br $label$113)
+         )
+        )
+        (br $label$117)
+       )
+       (if
+        (i32.and
+         (i32.ge_u
+          (local.tee $2
+           (i32.load
+            (local.tee $1
+             (i32.add
+              (local.get $0)
+              (i32.const 8)
+             )
+            )
+           )
+          )
+          (local.tee $4
+           (i32.load
+            (i32.const 3668)
+           )
+          )
+         )
+         (i32.ge_u
+          (local.get $0)
+          (local.get $4)
+         )
+        )
+        (block
+         (i32.store offset=12
+          (local.get $2)
+          (local.get $3)
+         )
+         (i32.store
+          (local.get $1)
+          (local.get $3)
+         )
+         (i32.store offset=8
+          (local.get $3)
+          (local.get $2)
+         )
+         (i32.store offset=12
+          (local.get $3)
+          (local.get $0)
+         )
+         (i32.store offset=24
+          (local.get $3)
+          (i32.const 0)
+         )
+        )
+        (call $fimport$10)
+       )
+      )
+     )
+     (block
+      (i32.store
+       (i32.const 3656)
+       (i32.or
+        (local.get $4)
+        (local.get $5)
+       )
+      )
+      (i32.store
+       (local.get $0)
+       (local.get $3)
+      )
+      (i32.store offset=24
+       (local.get $3)
+       (local.get $0)
+      )
+      (i32.store offset=12
+       (local.get $3)
+       (local.get $3)
+      )
+      (i32.store offset=8
+       (local.get $3)
+       (local.get $3)
+      )
+     )
+    )
+   )
+   (i32.store
+    (i32.const 3684)
+    (local.tee $0
+     (i32.add
+      (i32.load
+       (i32.const 3684)
+      )
+      (i32.const -1)
+     )
+    )
+   )
+   (if
+    (local.get $0)
+    (return)
+    (local.set $0
+     (i32.const 4108)
+    )
+   )
+   (loop $label$128
+    (local.set $0
+     (i32.add
+      (local.tee $2
+       (i32.load
+        (local.get $0)
+       )
+      )
+      (i32.const 8)
+     )
+    )
+    (br_if $label$128
+     (local.get $2)
+    )
+   )
+   (i32.store
+    (i32.const 3684)
+    (i32.const -1)
+   )
+  )
+ )
+ (func $37 (; 50 ;) (type $6)
+  (nop)
+ )
+ (func $38 (; 51 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (i32.add
+     (local.tee $2
+      (i32.load
+       (global.get $global$0)
+      )
+     )
+     (local.tee $0
+      (i32.and
+       (i32.add
+        (local.get $0)
+        (i32.const 15)
+       )
+       (i32.const -16)
+      )
+     )
+    )
+   )
+   (if
+    (i32.or
+     (i32.and
+      (i32.gt_s
+       (local.get $0)
+       (i32.const 0)
+      )
+      (i32.lt_s
+       (local.get $1)
+       (local.get $2)
+      )
+     )
+     (i32.lt_s
+      (local.get $1)
+      (i32.const 0)
+     )
+    )
+    (block
+     (drop
+      (call $fimport$6)
+     )
+     (call $fimport$11
+      (i32.const 12)
+     )
+     (return
+      (i32.const -1)
+     )
+    )
+   )
+   (i32.store
+    (global.get $global$0)
+    (local.get $1)
+   )
+   (if
+    (i32.gt_s
+     (local.get $1)
+     (call $fimport$5)
+    )
+    (if
+     (i32.eqz
+      (call $fimport$4)
+     )
+     (block
+      (call $fimport$11
+       (i32.const 12)
+      )
+      (i32.store
+       (global.get $global$0)
+       (local.get $2)
+      )
+      (return
+       (i32.const -1)
+      )
+     )
+    )
+   )
+   (local.get $2)
+  )
+ )
+ (func $39 (; 52 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (i32.add
+     (local.get $0)
+     (local.get $2)
+    )
+   )
+   (if
+    (i32.ge_s
+     (local.get $2)
+     (i32.const 20)
+    )
+    (block
+     (local.set $1
+      (i32.and
+       (local.get $1)
+       (i32.const 255)
+      )
+     )
+     (if
+      (local.tee $3
+       (i32.and
+        (local.get $0)
+        (i32.const 3)
+       )
+      )
+      (block
+       (local.set $3
+        (i32.sub
+         (i32.add
+          (local.get $0)
+          (i32.const 4)
+         )
+         (local.get $3)
+        )
+       )
+       (loop $label$4
+        (if
+         (i32.lt_s
+          (local.get $0)
+          (local.get $3)
+         )
+         (block
+          (i32.store8
+           (local.get $0)
+           (local.get $1)
+          )
+          (local.set $0
+           (i32.add
+            (local.get $0)
+            (i32.const 1)
+           )
+          )
+          (br $label$4)
+         )
+        )
+       )
+      )
+     )
+     (local.set $3
+      (i32.or
+       (i32.or
+        (i32.or
+         (local.get $1)
+         (i32.shl
+          (local.get $1)
+          (i32.const 8)
+         )
+        )
+        (i32.shl
+         (local.get $1)
+         (i32.const 16)
+        )
+       )
+       (i32.shl
+        (local.get $1)
+        (i32.const 24)
+       )
+      )
+     )
+     (local.set $5
+      (i32.and
+       (local.get $4)
+       (i32.const -4)
+      )
+     )
+     (loop $label$6
+      (if
+       (i32.lt_s
+        (local.get $0)
+        (local.get $5)
+       )
+       (block
+        (i32.store
+         (local.get $0)
+         (local.get $3)
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (i32.const 4)
+         )
+        )
+        (br $label$6)
+       )
+      )
+     )
+    )
+   )
+   (loop $label$8
+    (if
+     (i32.lt_s
+      (local.get $0)
+      (local.get $4)
+     )
+     (block
+      (i32.store8
+       (local.get $0)
+       (local.get $1)
+      )
+      (local.set $0
+       (i32.add
+        (local.get $0)
+        (i32.const 1)
+       )
+      )
+      (br $label$8)
+     )
+    )
+   )
+   (i32.sub
+    (local.get $0)
+    (local.get $2)
+   )
+  )
+ )
+ (func $40 (; 53 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (block $label$1 (result i32)
+   (if
+    (i32.ge_s
+     (local.get $2)
+     (i32.const 4096)
+    )
+    (return
+     (call $fimport$12
+      (local.get $0)
+      (local.get $1)
+      (local.get $2)
+     )
+    )
+   )
+   (local.set $3
+    (local.get $0)
+   )
+   (if
+    (i32.eq
+     (i32.and
+      (local.get $0)
+      (i32.const 3)
+     )
+     (i32.and
+      (local.get $1)
+      (i32.const 3)
+     )
+    )
+    (block
+     (loop $label$4
+      (if
+       (i32.and
+        (local.get $0)
+        (i32.const 3)
+       )
+       (block
+        (if
+         (i32.eqz
+          (local.get $2)
+         )
+         (return
+          (local.get $3)
+         )
+        )
+        (i32.store8
+         (local.get $0)
+         (i32.load8_s
+          (local.get $1)
+         )
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (i32.const 1)
+         )
+        )
+        (local.set $1
+         (i32.add
+          (local.get $1)
+          (i32.const 1)
+         )
+        )
+        (local.set $2
+         (i32.sub
+          (local.get $2)
+          (i32.const 1)
+         )
+        )
+        (br $label$4)
+       )
+      )
+     )
+     (loop $label$7
+      (if
+       (i32.ge_s
+        (local.get $2)
+        (i32.const 4)
+       )
+       (block
+        (i32.store
+         (local.get $0)
+         (i32.load
+          (local.get $1)
+         )
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (i32.const 4)
+         )
+        )
+        (local.set $1
+         (i32.add
+          (local.get $1)
+          (i32.const 4)
+         )
+        )
+        (local.set $2
+         (i32.sub
+          (local.get $2)
+          (i32.const 4)
+         )
+        )
+        (br $label$7)
+       )
+      )
+     )
+    )
+   )
+   (loop $label$9
+    (if
+     (i32.gt_s
+      (local.get $2)
+      (i32.const 0)
+     )
+     (block
+      (i32.store8
+       (local.get $0)
+       (i32.load8_s
+        (local.get $1)
+       )
+      )
+      (local.set $0
+       (i32.add
+        (local.get $0)
+        (i32.const 1)
+       )
+      )
+      (local.set $1
+       (i32.add
+        (local.get $1)
+        (i32.const 1)
+       )
+      )
+      (local.set $2
+       (i32.sub
+        (local.get $2)
+        (i32.const 1)
+       )
+      )
+      (br $label$9)
+     )
+    )
+   )
+   (local.get $3)
+  )
+ )
+ (func $41 (; 54 ;) (type $3) (result i32)
+  (i32.const 0)
+ )
+ (func $42 (; 55 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (call_indirect (type $1)
+   (local.get $1)
+   (i32.add
+    (i32.and
+     (local.get $0)
+     (i32.const 1)
+    )
+    (i32.const 0)
+   )
+  )
+ )
+ (func $43 (; 56 ;) (type $12) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (result i32)
+  (call_indirect (type $0)
+   (local.get $1)
+   (local.get $2)
+   (local.get $3)
+   (i32.add
+    (i32.and
+     (local.get $0)
+     (i32.const 3)
+    )
+    (i32.const 2)
+   )
+  )
+ )
+ (func $44 (; 57 ;) (type $5) (param $0 i32) (param $1 i32)
+  (call_indirect (type $2)
+   (local.get $1)
+   (i32.add
+    (i32.and
+     (local.get $0)
+     (i32.const 1)
+    )
+    (i32.const 6)
+   )
+  )
+ )
+ (func $45 (; 58 ;) (type $1) (param $0 i32) (result i32)
+  (block $label$1 (result i32)
+   (call $fimport$3
+    (i32.const 0)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $46 (; 59 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (block $label$1 (result i32)
+   (call $fimport$3
+    (i32.const 1)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $47 (; 60 ;) (type $2) (param $0 i32)
+  (call $fimport$3
+   (i32.const 2)
+  )
+ )
+)
+
diff --git a/cranelift/wasmtests/embenchen_fasta.wat b/cranelift/wasmtests/embenchen_fasta.wat
new file mode 100644
index 0000000000..54baa68e15
--- /dev/null
+++ b/cranelift/wasmtests/embenchen_fasta.wat
@@ -0,0 +1,16547 @@
+(module
+ (type $0 (func (param i32 i32 i32) (result i32)))
+ (type $1 (func))
+ (type $2 (func (param i32) (result i32)))
+ (type $3 (func (param i32)))
+ (type $4 (func (result i32)))
+ (type $5 (func (param i32 i32)))
+ (type $6 (func (param i32 i32) (result i32)))
+ (type $7 (func (param i32 i32 i32 i32 i32) (result i32)))
+ (type $8 (func (param i32 i32 i32)))
+ (type $9 (func (param i64 i32) (result i32)))
+ (type $10 (func (param i32 i32 i32 i32 i32)))
+ (type $11 (func (param f64 i32) (result f64)))
+ (type $12 (func (param i32 i32 i32 i32) (result i32)))
+ (import "env" "memory" (memory $16 2048 2048))
+ (data (i32.const 1024) "&\02\00\00a\00\00\00q=\8a>\00\00\00\00c\00\00\00\8f\c2\f5=\00\00\00\00g\00\00\00\8f\c2\f5=\00\00\00\00t\00\00\00q=\8a>\00\00\00\00B\00\00\00\n\d7\a3<\00\00\00\00D\00\00\00\n\d7\a3<\00\00\00\00H\00\00\00\n\d7\a3<\00\00\00\00K\00\00\00\n\d7\a3<\00\00\00\00M\00\00\00\n\d7\a3<\00\00\00\00N\00\00\00\n\d7\a3<\00\00\00\00R\00\00\00\n\d7\a3<\00\00\00\00S\00\00\00\n\d7\a3<\00\00\00\00V\00\00\00\n\d7\a3<\00\00\00\00W\00\00\00\n\d7\a3<\00\00\00\00Y\00\00\00\n\d7\a3<")
+ (data (i32.const 1220) "a\00\00\00\e9\1c\9b>\00\00\00\00c\00\00\00r\bdJ>\00\00\00\00g\00\00\00\d7IJ>\00\00\00\00t\00\00\00r_\9a>")
+ (data (i32.const 1280) "\04\05\00\00\05")
+ (data (i32.const 1296) "\01")
+ (data (i32.const 1320) "\01\00\00\00\02\00\00\00L\12\00\00\00\04")
+ (data (i32.const 1344) "\01")
+ (data (i32.const 1359) "\n\ff\ff\ff\ff")
+ (data (i32.const 1396) "*\00\00\00error: %d\n\00GGCCGGGCGCGGTGGCTCACGCCTGTAATCCCAGCACTTTGGGAGGCCGAGGCGGGCGGATCACCTGAGGTCAGGAGTTCGAGACCAGCCTGGCCAACATGGTGAAACCCCGTCTCTACTAAAAATACAAAAATTAGCCGGGCGTGGTGGCGCGCGCCTGTAATCCCAGCTACTCGGGAGGCTGAGGCAGGAGAATCGCTTGAACCCGGGAGGCGGAGGTTGCAGTGAGCCGAGATCGCGCCACTGCACTCCAGCCTGGGCGACAGAGCGAGACTCCGTCTCAAAAA\00\11\00\n\00\11\11\11\00\00\00\00\05\00\00\00\00\00\00\t\00\00\00\00\0b")
+ (data (i32.const 1731) "\11\00\0f\n\11\11\11\03\n\07\00\01\13\t\0b\0b\00\00\t\06\0b\00\00\0b\00\06\11\00\00\00\11\11\11")
+ (data (i32.const 1780) "\0b")
+ (data (i32.const 1789) "\11\00\n\n\11\11\11\00\n\00\00\02\00\t\0b\00\00\00\t\00\0b\00\00\0b")
+ (data (i32.const 1838) "\0c")
+ (data (i32.const 1850) "\0c\00\00\00\00\0c\00\00\00\00\t\0c\00\00\00\00\00\0c\00\00\0c")
+ (data (i32.const 1896) "\0e")
+ (data (i32.const 1908) "\0d\00\00\00\04\0d\00\00\00\00\t\0e\00\00\00\00\00\0e\00\00\0e")
+ (data (i32.const 1954) "\10")
+ (data (i32.const 1966) "\0f\00\00\00\00\0f\00\00\00\00\t\10\00\00\00\00\00\10\00\00\10\00\00\12\00\00\00\12\12\12")
+ (data (i32.const 2021) "\12\00\00\00\12\12\12\00\00\00\00\00\00\t")
+ (data (i32.const 2070) "\0b")
+ (data (i32.const 2082) "\n\00\00\00\00\n\00\00\00\00\t\0b\00\00\00\00\00\0b\00\00\0b")
+ (data (i32.const 2128) "\0c")
+ (data (i32.const 2140) "\0c\00\00\00\00\0c\00\00\00\00\t\0c\00\00\00\00\00\0c\00\00\0c\00\000123456789ABCDEF-+   0X0x\00(null)\00-0X+0X 0X-0x+0x 0x\00inf\00INF\00nan\00NAN\00.\00T!\"\19\0d\01\02\03\11K\1c\0c\10\04\0b\1d\12\1e\'hnopqb \05\06\0f\13\14\15\1a\08\16\07($\17\18\t\n\0e\1b\1f%#\83\82}&*+<=>?CGJMXYZ[\\]^_`acdefgijklrstyz{|\00Illegal byte sequence\00Domain error\00Result not representable\00Not a tty\00Permission denied\00Operation not permitted\00No such file or directory\00No such process\00File exists\00Value too large for data type\00No space left on device\00Out of memory\00Resource busy\00Interrupted system call\00Resource temporarily unavailable\00Invalid seek\00Cross-device link\00Read-only file system\00Directory not empty\00Connection reset by peer\00Operation timed out\00Connection refused\00Host is down\00Host is unreachable\00Address in use\00Broken pipe\00I/O error\00No such device or address\00Block device required\00No such device\00Not a directory\00Is a directory\00Text file busy\00Exec format error\00Invalid argument\00Argument list too long\00Symbolic link loop\00Filename too long\00Too many open files in system\00No file descriptors available\00Bad file descriptor\00No child process\00Bad address\00File too large\00Too many links\00No locks available\00Resource deadlock would occur\00State not recoverable\00Previous owner died\00Operation canceled\00Function not implemented\00No message of desired type\00Identifier removed\00Device not a stream\00No data available\00Device timeout\00Out of streams resources\00Link has been severed\00Protocol error\00Bad message\00File descriptor in bad state\00Not a socket\00Destination address required\00Message too large\00Protocol wrong type for socket\00Protocol not available\00Protocol not supported\00Socket type not supported\00Not supported\00Protocol family not supported\00Address family not supported by protocol\00Address not available\00Network is down\00Network unreachable\00Connection reset by network\00Connection aborted\00No buffer space available\00Socket is connected\00Socket not connected\00Cannot send after socket shutdown\00Operation already in progress\00Operation in progress\00Stale file handle\00Remote I/O error\00Quota exceeded\00No medium found\00Wrong medium type\00No error information")
+ (import "env" "table" (table $timport$17 9 9 funcref))
+ (elem (global.get $gimport$19) $53 $9 $54 $14 $10 $15 $55 $16 $56)
+ (import "env" "DYNAMICTOP_PTR" (global $gimport$0 i32))
+ (import "env" "STACKTOP" (global $gimport$1 i32))
+ (import "env" "STACK_MAX" (global $gimport$2 i32))
+ (import "env" "memoryBase" (global $gimport$18 i32))
+ (import "env" "tableBase" (global $gimport$19 i32))
+ (import "env" "abort" (func $fimport$3 (param i32)))
+ (import "env" "enlargeMemory" (func $fimport$4 (result i32)))
+ (import "env" "getTotalMemory" (func $fimport$5 (result i32)))
+ (import "env" "abortOnCannotGrowMemory" (func $fimport$6 (result i32)))
+ (import "env" "_pthread_cleanup_pop" (func $fimport$7 (param i32)))
+ (import "env" "_abort" (func $fimport$8))
+ (import "env" "_pthread_cleanup_push" (func $fimport$9 (param i32 i32)))
+ (import "env" "___syscall6" (func $fimport$10 (param i32 i32) (result i32)))
+ (import "env" "___setErrNo" (func $fimport$11 (param i32)))
+ (import "env" "_emscripten_memcpy_big" (func $fimport$12 (param i32 i32 i32) (result i32)))
+ (import "env" "___syscall54" (func $fimport$13 (param i32 i32) (result i32)))
+ (import "env" "___syscall140" (func $fimport$14 (param i32 i32) (result i32)))
+ (import "env" "___syscall146" (func $fimport$15 (param i32 i32) (result i32)))
+ (global $global$0 (mut i32) (global.get $gimport$0))
+ (global $global$1 (mut i32) (global.get $gimport$1))
+ (global $global$2 (mut i32) (global.get $gimport$2))
+ (global $global$3 (mut i32) (i32.const 0))
+ (global $global$4 (mut i32) (i32.const 0))
+ (global $global$5 (mut i32) (i32.const 0))
+ (export "_sbrk" (func $45))
+ (export "_free" (func $38))
+ (export "_main" (func $7))
+ (export "_pthread_self" (func $48))
+ (export "_memset" (func $46))
+ (export "_malloc" (func $37))
+ (export "_memcpy" (func $47))
+ (export "___errno_location" (func $12))
+ (export "runPostSets" (func $44))
+ (export "stackAlloc" (func $0))
+ (export "stackSave" (func $1))
+ (export "stackRestore" (func $2))
+ (export "establishStackSpace" (func $3))
+ (export "setThrew" (func $4))
+ (export "setTempRet0" (func $5))
+ (export "getTempRet0" (func $6))
+ (export "dynCall_ii" (func $49))
+ (export "dynCall_iiii" (func $50))
+ (export "dynCall_vi" (func $51))
+ (export "dynCall_v" (func $52))
+ (func $0 (; 13 ;) (type $2) (param $0 i32) (result i32)
+  (local $1 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (local.get $0)
+    )
+   )
+   (global.set $global$1
+    (i32.and
+     (i32.add
+      (global.get $global$1)
+      (i32.const 15)
+     )
+     (i32.const -16)
+    )
+   )
+   (local.get $1)
+  )
+ )
+ (func $1 (; 14 ;) (type $4) (result i32)
+  (global.get $global$1)
+ )
+ (func $2 (; 15 ;) (type $3) (param $0 i32)
+  (global.set $global$1
+   (local.get $0)
+  )
+ )
+ (func $3 (; 16 ;) (type $5) (param $0 i32) (param $1 i32)
+  (block $label$1
+   (global.set $global$1
+    (local.get $0)
+   )
+   (global.set $global$2
+    (local.get $1)
+   )
+  )
+ )
+ (func $4 (; 17 ;) (type $5) (param $0 i32) (param $1 i32)
+  (if
+   (i32.eqz
+    (global.get $global$3)
+   )
+   (block
+    (global.set $global$3
+     (local.get $0)
+    )
+    (global.set $global$4
+     (local.get $1)
+    )
+   )
+  )
+ )
+ (func $5 (; 18 ;) (type $3) (param $0 i32)
+  (global.set $global$5
+   (local.get $0)
+  )
+ )
+ (func $6 (; 19 ;) (type $4) (result i32)
+  (global.get $global$5)
+ )
+ (func $7 (; 20 ;) (type $6) (param $0 i32) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 f32)
+  (local $12 f32)
+  (local $13 f64)
+  (block $label$1 (result i32)
+   (local.set $5
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 4256)
+    )
+   )
+   (local.set $3
+    (local.get $5)
+   )
+   (local.set $6
+    (i32.add
+     (local.get $5)
+     (i32.const 2128)
+    )
+   )
+   (local.set $7
+    (i32.add
+     (local.get $5)
+     (i32.const 8)
+    )
+   )
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (i32.le_s
+       (local.get $0)
+       (i32.const 1)
+      )
+     )
+     (block $label$4
+      (block $label$5
+       (block $label$6
+        (block $label$7
+         (block $label$8
+          (block $label$9
+           (block $label$10
+            (br_table $label$5 $label$10 $label$8 $label$9 $label$7 $label$6 $label$4
+             (i32.sub
+              (local.tee $0
+               (i32.load8_s
+                (i32.load offset=4
+                 (local.get $1)
+                )
+               )
+              )
+              (i32.const 48)
+             )
+            )
+           )
+           (local.set $4
+            (i32.const 950000)
+           )
+           (br $label$2)
+          )
+          (br $label$3)
+         )
+         (local.set $4
+          (i32.const 9500000)
+         )
+         (br $label$2)
+        )
+        (local.set $4
+         (i32.const 95000000)
+        )
+        (br $label$2)
+       )
+       (local.set $4
+        (i32.const 190000000)
+       )
+       (br $label$2)
+      )
+      (global.set $global$1
+       (local.get $5)
+      )
+      (return
+       (i32.const 0)
+      )
+     )
+     (i32.store
+      (local.get $3)
+      (i32.add
+       (local.get $0)
+       (i32.const -48)
+      )
+     )
+     (drop
+      (call $34
+       (i32.const 1400)
+       (local.get $3)
+      )
+     )
+     (global.set $global$1
+      (local.get $5)
+     )
+     (return
+      (i32.const -1)
+     )
+    )
+    (local.set $4
+     (i32.const 19000000)
+    )
+   )
+   (drop
+    (call $47
+     (local.tee $8
+      (call $40
+       (i32.const 347)
+      )
+     )
+     (i32.const 1411)
+     (i32.const 287)
+    )
+   )
+   (i64.store align=1
+    (local.tee $0
+     (i32.add
+      (local.get $8)
+      (i32.const 287)
+     )
+    )
+    (i64.load align=1
+     (i32.const 1411)
+    )
+   )
+   (i64.store offset=8 align=1
+    (local.get $0)
+    (i64.load align=1
+     (i32.const 1419)
+    )
+   )
+   (i64.store offset=16 align=1
+    (local.get $0)
+    (i64.load align=1
+     (i32.const 1427)
+    )
+   )
+   (i64.store offset=24 align=1
+    (local.get $0)
+    (i64.load align=1
+     (i32.const 1435)
+    )
+   )
+   (i64.store offset=32 align=1
+    (local.get $0)
+    (i64.load align=1
+     (i32.const 1443)
+    )
+   )
+   (i64.store offset=40 align=1
+    (local.get $0)
+    (i64.load align=1
+     (i32.const 1451)
+    )
+   )
+   (i64.store offset=48 align=1
+    (local.get $0)
+    (i64.load align=1
+     (i32.const 1459)
+    )
+   )
+   (i32.store offset=56 align=1
+    (local.get $0)
+    (i32.load align=1
+     (i32.const 1467)
+    )
+   )
+   (local.set $0
+    (i32.shl
+     (local.get $4)
+     (i32.const 1)
+    )
+   )
+   (local.set $1
+    (i32.const 0)
+   )
+   (loop $label$11
+    (drop
+     (call $47
+      (local.tee $2
+       (call $40
+        (i32.add
+         (local.tee $3
+          (if (result i32)
+           (i32.lt_u
+            (local.get $0)
+            (i32.const 60)
+           )
+           (local.get $0)
+           (i32.const 60)
+          )
+         )
+         (i32.const 2)
+        )
+       )
+      )
+      (i32.add
+       (local.get $8)
+       (local.get $1)
+      )
+      (local.get $3)
+     )
+    )
+    (i32.store8
+     (i32.add
+      (local.get $2)
+      (local.get $3)
+     )
+     (i32.const 0)
+    )
+    (if
+     (i32.gt_s
+      (local.tee $10
+       (call $31
+        (local.get $2)
+       )
+      )
+      (local.tee $9
+       (i32.load
+        (i32.const 1024)
+       )
+      )
+     )
+     (if
+      (i32.gt_s
+       (local.get $9)
+       (i32.const 0)
+      )
+      (block
+       (i32.store8
+        (i32.add
+         (local.get $2)
+         (local.get $9)
+        )
+        (i32.const 0)
+       )
+       (drop
+        (call $35
+         (local.get $2)
+        )
+       )
+       (i32.store
+        (i32.const 1024)
+        (i32.const 0)
+       )
+      )
+     )
+     (block
+      (drop
+       (call $35
+        (local.get $2)
+       )
+      )
+      (i32.store
+       (i32.const 1024)
+       (i32.sub
+        (i32.load
+         (i32.const 1024)
+        )
+        (local.get $10)
+       )
+      )
+     )
+    )
+    (call $41
+     (local.get $2)
+    )
+    (local.set $1
+     (i32.add
+      (local.tee $2
+       (i32.add
+        (local.get $3)
+        (local.get $1)
+       )
+      )
+      (i32.const -287)
+     )
+    )
+    (if
+     (i32.le_u
+      (local.get $2)
+      (i32.const 287)
+     )
+     (local.set $1
+      (local.get $2)
+     )
+    )
+    (br_if $label$11
+     (local.tee $0
+      (i32.sub
+       (local.get $0)
+       (local.get $3)
+      )
+     )
+    )
+   )
+   (call $42
+    (local.get $8)
+   )
+   (if
+    (i32.load
+     (i32.const 1028)
+    )
+    (block
+     (local.set $0
+      (i32.const 1028)
+     )
+     (local.set $11
+      (f32.const 0)
+     )
+     (loop $label$19
+      (local.set $12
+       (f32.demote_f64
+        (if (result f64)
+         (f64.lt
+          (local.tee $13
+           (f64.promote_f32
+            (local.tee $11
+             (f32.add
+              (local.get $11)
+              (f32.load
+               (local.tee $1
+                (i32.add
+                 (local.get $0)
+                 (i32.const 4)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (f64.const 1)
+         )
+         (local.get $13)
+         (f64.const 1)
+        )
+       )
+      )
+      (f32.store
+       (local.get $1)
+       (local.get $12)
+      )
+      (i32.store offset=8
+       (local.get $0)
+       (i32.trunc_f32_s
+        (f32.mul
+         (local.get $12)
+         (f32.const 512)
+        )
+       )
+      )
+      (br_if $label$19
+       (i32.load
+        (local.tee $0
+         (i32.add
+          (local.get $0)
+          (i32.const 12)
+         )
+        )
+       )
+      )
+      (local.set $1
+       (i32.const 0)
+      )
+      (local.set $0
+       (i32.const 1028)
+      )
+     )
+    )
+    (block
+     (local.set $1
+      (i32.const 0)
+     )
+     (local.set $0
+      (i32.const 1028)
+     )
+    )
+   )
+   (loop $label$23
+    (loop $label$24
+     (local.set $3
+      (i32.add
+       (local.get $0)
+       (i32.const 12)
+      )
+     )
+     (if
+      (i32.and
+       (i32.gt_u
+        (local.get $1)
+        (local.tee $2
+         (i32.load offset=8
+          (local.get $0)
+         )
+        )
+       )
+       (i32.ne
+        (local.get $2)
+        (i32.const 0)
+       )
+      )
+      (block
+       (local.set $0
+        (local.get $3)
+       )
+       (br $label$24)
+      )
+     )
+    )
+    (i32.store
+     (i32.add
+      (local.get $6)
+      (i32.shl
+       (local.get $1)
+       (i32.const 2)
+      )
+     )
+     (local.get $0)
+    )
+    (br_if $label$23
+     (i32.ne
+      (local.tee $1
+       (i32.add
+        (local.get $1)
+        (i32.const 1)
+       )
+      )
+      (i32.const 513)
+     )
+    )
+   )
+   (i32.store
+    (i32.add
+     (local.get $6)
+     (i32.const 2116)
+    )
+    (i32.const 0)
+   )
+   (local.set $0
+    (i32.mul
+     (local.get $4)
+     (i32.const 3)
+    )
+   )
+   (loop $label$26
+    (call $8
+     (local.get $6)
+     (local.tee $1
+      (if (result i32)
+       (i32.lt_u
+        (local.get $0)
+        (i32.const 60)
+       )
+       (local.get $0)
+       (i32.const 60)
+      )
+     )
+    )
+    (br_if $label$26
+     (local.tee $0
+      (i32.sub
+       (local.get $0)
+       (local.get $1)
+      )
+     )
+    )
+   )
+   (if
+    (i32.load
+     (i32.const 1220)
+    )
+    (block
+     (local.set $0
+      (i32.const 1220)
+     )
+     (local.set $11
+      (f32.const 0)
+     )
+     (loop $label$30
+      (local.set $12
+       (f32.demote_f64
+        (if (result f64)
+         (f64.lt
+          (local.tee $13
+           (f64.promote_f32
+            (local.tee $11
+             (f32.add
+              (local.get $11)
+              (f32.load
+               (local.tee $1
+                (i32.add
+                 (local.get $0)
+                 (i32.const 4)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (f64.const 1)
+         )
+         (local.get $13)
+         (f64.const 1)
+        )
+       )
+      )
+      (f32.store
+       (local.get $1)
+       (local.get $12)
+      )
+      (i32.store offset=8
+       (local.get $0)
+       (i32.trunc_f32_s
+        (f32.mul
+         (local.get $12)
+         (f32.const 512)
+        )
+       )
+      )
+      (br_if $label$30
+       (i32.load
+        (local.tee $0
+         (i32.add
+          (local.get $0)
+          (i32.const 12)
+         )
+        )
+       )
+      )
+      (local.set $1
+       (i32.const 0)
+      )
+      (local.set $0
+       (i32.const 1220)
+      )
+     )
+    )
+    (block
+     (local.set $1
+      (i32.const 0)
+     )
+     (local.set $0
+      (i32.const 1220)
+     )
+    )
+   )
+   (loop $label$34
+    (loop $label$35
+     (local.set $3
+      (i32.add
+       (local.get $0)
+       (i32.const 12)
+      )
+     )
+     (if
+      (i32.and
+       (i32.gt_u
+        (local.get $1)
+        (local.tee $2
+         (i32.load offset=8
+          (local.get $0)
+         )
+        )
+       )
+       (i32.ne
+        (local.get $2)
+        (i32.const 0)
+       )
+      )
+      (block
+       (local.set $0
+        (local.get $3)
+       )
+       (br $label$35)
+      )
+     )
+    )
+    (i32.store
+     (i32.add
+      (local.get $7)
+      (i32.shl
+       (local.get $1)
+       (i32.const 2)
+      )
+     )
+     (local.get $0)
+    )
+    (br_if $label$34
+     (i32.ne
+      (local.tee $1
+       (i32.add
+        (local.get $1)
+        (i32.const 1)
+       )
+      )
+      (i32.const 513)
+     )
+    )
+   )
+   (i32.store
+    (i32.add
+     (local.get $7)
+     (i32.const 2116)
+    )
+    (i32.const 0)
+   )
+   (local.set $0
+    (i32.mul
+     (local.get $4)
+     (i32.const 5)
+    )
+   )
+   (loop $label$37
+    (call $8
+     (local.get $7)
+     (local.tee $1
+      (if (result i32)
+       (i32.lt_u
+        (local.get $0)
+        (i32.const 60)
+       )
+       (local.get $0)
+       (i32.const 60)
+      )
+     )
+    )
+    (br_if $label$37
+     (local.tee $0
+      (i32.sub
+       (local.get $0)
+       (local.get $1)
+      )
+     )
+    )
+    (local.set $0
+     (i32.const 0)
+    )
+   )
+   (global.set $global$1
+    (local.get $5)
+   )
+   (local.get $0)
+  )
+ )
+ (func $8 (; 21 ;) (type $5) (param $0 i32) (param $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 f32)
+  (block $label$1
+   (if
+    (local.get $1)
+    (block
+     (local.set $3
+      (i32.const 0)
+     )
+     (local.set $2
+      (i32.load
+       (i32.const 1396)
+      )
+     )
+     (loop $label$3
+      (local.set $2
+       (i32.load
+        (i32.add
+         (local.get $0)
+         (i32.shl
+          (i32.trunc_f32_s
+           (f32.mul
+            (local.tee $6
+             (f32.div
+              (f32.convert_i32_u
+               (local.tee $4
+                (i32.rem_u
+                 (i32.add
+                  (i32.mul
+                   (local.get $2)
+                   (i32.const 3877)
+                  )
+                  (i32.const 29573)
+                 )
+                 (i32.const 139968)
+                )
+               )
+              )
+              (f32.const 139968)
+             )
+            )
+            (f32.const 512)
+           )
+          )
+          (i32.const 2)
+         )
+        )
+       )
+      )
+      (loop $label$4
+       (local.set $5
+        (i32.add
+         (local.get $2)
+         (i32.const 12)
+        )
+       )
+       (if
+        (f32.lt
+         (f32.load offset=4
+          (local.get $2)
+         )
+         (local.get $6)
+        )
+        (block
+         (local.set $2
+          (local.get $5)
+         )
+         (br $label$4)
+        )
+       )
+      )
+      (i32.store8
+       (i32.add
+        (i32.add
+         (local.get $0)
+         (i32.const 2052)
+        )
+        (local.get $3)
+       )
+       (i32.load
+        (local.get $2)
+       )
+      )
+      (if
+       (i32.ne
+        (local.tee $3
+         (i32.add
+          (local.get $3)
+          (i32.const 1)
+         )
+        )
+        (local.get $1)
+       )
+       (block
+        (local.set $2
+         (local.get $4)
+        )
+        (br $label$3)
+       )
+      )
+     )
+     (i32.store
+      (i32.const 1396)
+      (local.get $4)
+     )
+    )
+   )
+   (i32.store8
+    (i32.add
+     (i32.add
+      (local.get $0)
+      (i32.const 2052)
+     )
+     (local.get $1)
+    )
+    (i32.const 10)
+   )
+   (i32.store8
+    (i32.add
+     (i32.add
+      (local.get $0)
+      (i32.const 2052)
+     )
+     (local.tee $1
+      (i32.add
+       (local.get $1)
+       (i32.const 1)
+      )
+     )
+    )
+    (i32.const 0)
+   )
+   (i32.store
+    (i32.add
+     (local.get $0)
+     (i32.const 2116)
+    )
+    (local.get $1)
+   )
+   (if
+    (i32.le_s
+     (local.tee $3
+      (call $31
+       (local.tee $1
+        (i32.add
+         (local.get $0)
+         (i32.const 2052)
+        )
+       )
+      )
+     )
+     (local.tee $2
+      (i32.load
+       (i32.const 1024)
+      )
+     )
+    )
+    (block
+     (drop
+      (call $35
+       (local.get $1)
+      )
+     )
+     (i32.store
+      (i32.const 1024)
+      (i32.sub
+       (i32.load
+        (i32.const 1024)
+       )
+       (local.get $3)
+      )
+     )
+     (return)
+    )
+   )
+   (if
+    (i32.le_s
+     (local.get $2)
+     (i32.const 0)
+    )
+    (return)
+   )
+   (i32.store8
+    (i32.add
+     (i32.add
+      (local.get $0)
+      (i32.const 2052)
+     )
+     (local.get $2)
+    )
+    (i32.const 0)
+   )
+   (drop
+    (call $35
+     (local.get $1)
+    )
+   )
+   (i32.store8
+    (i32.add
+     (i32.add
+      (local.get $0)
+      (i32.const 2052)
+     )
+     (i32.load
+      (i32.const 1024)
+     )
+    )
+    (i32.const 122)
+   )
+   (i32.store
+    (i32.const 1024)
+    (i32.const 0)
+   )
+  )
+ )
+ (func $9 (; 22 ;) (type $2) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (i32.store
+    (local.tee $2
+     (local.get $1)
+    )
+    (i32.load offset=60
+     (local.get $0)
+    )
+   )
+   (local.set $0
+    (call $11
+     (call $fimport$10
+      (i32.const 6)
+      (local.get $2)
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $1)
+   )
+   (local.get $0)
+  )
+ )
+ (func $10 (; 23 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 32)
+    )
+   )
+   (i32.store
+    (local.tee $3
+     (local.get $4)
+    )
+    (i32.load offset=60
+     (local.get $0)
+    )
+   )
+   (i32.store offset=4
+    (local.get $3)
+    (i32.const 0)
+   )
+   (i32.store offset=8
+    (local.get $3)
+    (local.get $1)
+   )
+   (i32.store offset=12
+    (local.get $3)
+    (local.tee $0
+     (i32.add
+      (local.get $4)
+      (i32.const 20)
+     )
+    )
+   )
+   (i32.store offset=16
+    (local.get $3)
+    (local.get $2)
+   )
+   (local.set $0
+    (if (result i32)
+     (i32.lt_s
+      (call $11
+       (call $fimport$14
+        (i32.const 140)
+        (local.get $3)
+       )
+      )
+      (i32.const 0)
+     )
+     (block (result i32)
+      (i32.store
+       (local.get $0)
+       (i32.const -1)
+      )
+      (i32.const -1)
+     )
+     (i32.load
+      (local.get $0)
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $4)
+   )
+   (local.get $0)
+  )
+ )
+ (func $11 (; 24 ;) (type $2) (param $0 i32) (result i32)
+  (if (result i32)
+   (i32.gt_u
+    (local.get $0)
+    (i32.const -4096)
+   )
+   (block (result i32)
+    (i32.store
+     (call $12)
+     (i32.sub
+      (i32.const 0)
+      (local.get $0)
+     )
+    )
+    (i32.const -1)
+   )
+   (local.get $0)
+  )
+ )
+ (func $12 (; 25 ;) (type $4) (result i32)
+  (i32.const 4172)
+ )
+ (func $13 (; 26 ;) (type $3) (param $0 i32)
+  (nop)
+ )
+ (func $14 (; 27 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 80)
+    )
+   )
+   (local.set $3
+    (local.get $4)
+   )
+   (local.set $5
+    (i32.add
+     (local.get $4)
+     (i32.const 12)
+    )
+   )
+   (i32.store offset=36
+    (local.get $0)
+    (i32.const 3)
+   )
+   (if
+    (i32.eqz
+     (i32.and
+      (i32.load
+       (local.get $0)
+      )
+      (i32.const 64)
+     )
+    )
+    (block
+     (i32.store
+      (local.get $3)
+      (i32.load offset=60
+       (local.get $0)
+      )
+     )
+     (i32.store offset=4
+      (local.get $3)
+      (i32.const 21505)
+     )
+     (i32.store offset=8
+      (local.get $3)
+      (local.get $5)
+     )
+     (if
+      (call $fimport$13
+       (i32.const 54)
+       (local.get $3)
+      )
+      (i32.store8 offset=75
+       (local.get $0)
+       (i32.const -1)
+      )
+     )
+    )
+   )
+   (local.set $0
+    (call $15
+     (local.get $0)
+     (local.get $1)
+     (local.get $2)
+    )
+   )
+   (global.set $global$1
+    (local.get $4)
+   )
+   (local.get $0)
+  )
+ )
+ (func $15 (; 28 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (block $label$1 (result i32)
+   (local.set $8
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 48)
+    )
+   )
+   (local.set $9
+    (i32.add
+     (local.get $8)
+     (i32.const 16)
+    )
+   )
+   (local.set $10
+    (local.get $8)
+   )
+   (i32.store
+    (local.tee $3
+     (i32.add
+      (local.get $8)
+      (i32.const 32)
+     )
+    )
+    (local.tee $4
+     (i32.load
+      (local.tee $6
+       (i32.add
+        (local.get $0)
+        (i32.const 28)
+       )
+      )
+     )
+    )
+   )
+   (i32.store offset=4
+    (local.get $3)
+    (local.tee $5
+     (i32.sub
+      (i32.load
+       (local.tee $11
+        (i32.add
+         (local.get $0)
+         (i32.const 20)
+        )
+       )
+      )
+      (local.get $4)
+     )
+    )
+   )
+   (i32.store offset=8
+    (local.get $3)
+    (local.get $1)
+   )
+   (i32.store offset=12
+    (local.get $3)
+    (local.get $2)
+   )
+   (local.set $13
+    (i32.add
+     (local.get $0)
+     (i32.const 60)
+    )
+   )
+   (local.set $14
+    (i32.add
+     (local.get $0)
+     (i32.const 44)
+    )
+   )
+   (local.set $1
+    (local.get $3)
+   )
+   (local.set $4
+    (i32.const 2)
+   )
+   (local.set $12
+    (i32.add
+     (local.get $5)
+     (local.get $2)
+    )
+   )
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (loop $label$5
+       (if
+        (i32.load
+         (i32.const 4128)
+        )
+        (block
+         (call $fimport$9
+          (i32.const 1)
+          (local.get $0)
+         )
+         (i32.store
+          (local.get $10)
+          (i32.load
+           (local.get $13)
+          )
+         )
+         (i32.store offset=4
+          (local.get $10)
+          (local.get $1)
+         )
+         (i32.store offset=8
+          (local.get $10)
+          (local.get $4)
+         )
+         (local.set $3
+          (call $11
+           (call $fimport$15
+            (i32.const 146)
+            (local.get $10)
+           )
+          )
+         )
+         (call $fimport$7
+          (i32.const 0)
+         )
+        )
+        (block
+         (i32.store
+          (local.get $9)
+          (i32.load
+           (local.get $13)
+          )
+         )
+         (i32.store offset=4
+          (local.get $9)
+          (local.get $1)
+         )
+         (i32.store offset=8
+          (local.get $9)
+          (local.get $4)
+         )
+         (local.set $3
+          (call $11
+           (call $fimport$15
+            (i32.const 146)
+            (local.get $9)
+           )
+          )
+         )
+        )
+       )
+       (br_if $label$4
+        (i32.eq
+         (local.get $12)
+         (local.get $3)
+        )
+       )
+       (br_if $label$3
+        (i32.lt_s
+         (local.get $3)
+         (i32.const 0)
+        )
+       )
+       (local.set $5
+        (if (result i32)
+         (i32.gt_u
+          (local.get $3)
+          (local.tee $5
+           (i32.load offset=4
+            (local.get $1)
+           )
+          )
+         )
+         (block (result i32)
+          (i32.store
+           (local.get $6)
+           (local.tee $7
+            (i32.load
+             (local.get $14)
+            )
+           )
+          )
+          (i32.store
+           (local.get $11)
+           (local.get $7)
+          )
+          (local.set $7
+           (i32.load offset=12
+            (local.get $1)
+           )
+          )
+          (local.set $1
+           (i32.add
+            (local.get $1)
+            (i32.const 8)
+           )
+          )
+          (local.set $4
+           (i32.add
+            (local.get $4)
+            (i32.const -1)
+           )
+          )
+          (i32.sub
+           (local.get $3)
+           (local.get $5)
+          )
+         )
+         (if (result i32)
+          (i32.eq
+           (local.get $4)
+           (i32.const 2)
+          )
+          (block (result i32)
+           (i32.store
+            (local.get $6)
+            (i32.add
+             (i32.load
+              (local.get $6)
+             )
+             (local.get $3)
+            )
+           )
+           (local.set $7
+            (local.get $5)
+           )
+           (local.set $4
+            (i32.const 2)
+           )
+           (local.get $3)
+          )
+          (block (result i32)
+           (local.set $7
+            (local.get $5)
+           )
+           (local.get $3)
+          )
+         )
+        )
+       )
+       (i32.store
+        (local.get $1)
+        (i32.add
+         (i32.load
+          (local.get $1)
+         )
+         (local.get $5)
+        )
+       )
+       (i32.store offset=4
+        (local.get $1)
+        (i32.sub
+         (local.get $7)
+         (local.get $5)
+        )
+       )
+       (local.set $12
+        (i32.sub
+         (local.get $12)
+         (local.get $3)
+        )
+       )
+       (br $label$5)
+      )
+     )
+     (i32.store offset=16
+      (local.get $0)
+      (i32.add
+       (local.tee $1
+        (i32.load
+         (local.get $14)
+        )
+       )
+       (i32.load offset=48
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store
+      (local.get $6)
+      (local.get $1)
+     )
+     (i32.store
+      (local.get $11)
+      (local.get $1)
+     )
+     (br $label$2)
+    )
+    (i32.store offset=16
+     (local.get $0)
+     (i32.const 0)
+    )
+    (i32.store
+     (local.get $6)
+     (i32.const 0)
+    )
+    (i32.store
+     (local.get $11)
+     (i32.const 0)
+    )
+    (i32.store
+     (local.get $0)
+     (i32.or
+      (i32.load
+       (local.get $0)
+      )
+      (i32.const 32)
+     )
+    )
+    (local.set $2
+     (if (result i32)
+      (i32.eq
+       (local.get $4)
+       (i32.const 2)
+      )
+      (i32.const 0)
+      (i32.sub
+       (local.get $2)
+       (i32.load offset=4
+        (local.get $1)
+       )
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $8)
+   )
+   (local.get $2)
+  )
+ )
+ (func $16 (; 29 ;) (type $3) (param $0 i32)
+  (if
+   (i32.eqz
+    (i32.load offset=68
+     (local.get $0)
+    )
+   )
+   (call $13
+    (local.get $0)
+   )
+  )
+ )
+ (func $17 (; 30 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $5
+    (i32.and
+     (local.get $1)
+     (i32.const 255)
+    )
+   )
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (if
+       (i32.and
+        (local.tee $4
+         (i32.ne
+          (local.get $2)
+          (i32.const 0)
+         )
+        )
+        (i32.ne
+         (i32.and
+          (local.get $0)
+          (i32.const 3)
+         )
+         (i32.const 0)
+        )
+       )
+       (block
+        (local.set $4
+         (i32.and
+          (local.get $1)
+          (i32.const 255)
+         )
+        )
+        (local.set $3
+         (local.get $2)
+        )
+        (local.set $2
+         (local.get $0)
+        )
+        (loop $label$6
+         (if
+          (i32.eq
+           (i32.load8_s
+            (local.get $2)
+           )
+           (i32.shr_s
+            (i32.shl
+             (local.get $4)
+             (i32.const 24)
+            )
+            (i32.const 24)
+           )
+          )
+          (block
+           (local.set $0
+            (local.get $3)
+           )
+           (br $label$3)
+          )
+         )
+         (br_if $label$6
+          (i32.and
+           (local.tee $0
+            (i32.ne
+             (local.tee $3
+              (i32.add
+               (local.get $3)
+               (i32.const -1)
+              )
+             )
+             (i32.const 0)
+            )
+           )
+           (i32.ne
+            (i32.and
+             (local.tee $2
+              (i32.add
+               (local.get $2)
+               (i32.const 1)
+              )
+             )
+             (i32.const 3)
+            )
+            (i32.const 0)
+           )
+          )
+         )
+         (br $label$4)
+        )
+       )
+       (block
+        (local.set $3
+         (local.get $2)
+        )
+        (local.set $2
+         (local.get $0)
+        )
+        (local.set $0
+         (local.get $4)
+        )
+       )
+      )
+     )
+     (if
+      (local.get $0)
+      (block
+       (local.set $0
+        (local.get $3)
+       )
+       (br $label$3)
+      )
+      (local.set $0
+       (i32.const 0)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.ne
+      (i32.load8_s
+       (local.get $2)
+      )
+      (i32.shr_s
+       (i32.shl
+        (local.tee $1
+         (i32.and
+          (local.get $1)
+          (i32.const 255)
+         )
+        )
+        (i32.const 24)
+       )
+       (i32.const 24)
+      )
+     )
+     (block
+      (local.set $3
+       (i32.mul
+        (local.get $5)
+        (i32.const 16843009)
+       )
+      )
+      (block $label$12
+       (block $label$13
+        (br_if $label$13
+         (i32.le_u
+          (local.get $0)
+          (i32.const 3)
+         )
+        )
+        (loop $label$14
+         (if
+          (i32.eqz
+           (i32.and
+            (i32.xor
+             (i32.and
+              (local.tee $4
+               (i32.xor
+                (i32.load
+                 (local.get $2)
+                )
+                (local.get $3)
+               )
+              )
+              (i32.const -2139062144)
+             )
+             (i32.const -2139062144)
+            )
+            (i32.add
+             (local.get $4)
+             (i32.const -16843009)
+            )
+           )
+          )
+          (block
+           (local.set $2
+            (i32.add
+             (local.get $2)
+             (i32.const 4)
+            )
+           )
+           (br_if $label$14
+            (i32.gt_u
+             (local.tee $0
+              (i32.add
+               (local.get $0)
+               (i32.const -4)
+              )
+             )
+             (i32.const 3)
+            )
+           )
+           (br $label$13)
+          )
+         )
+        )
+        (br $label$12)
+       )
+       (if
+        (i32.eqz
+         (local.get $0)
+        )
+        (block
+         (local.set $0
+          (i32.const 0)
+         )
+         (br $label$2)
+        )
+       )
+      )
+      (loop $label$17
+       (br_if $label$2
+        (i32.eq
+         (i32.load8_s
+          (local.get $2)
+         )
+         (i32.shr_s
+          (i32.shl
+           (local.get $1)
+           (i32.const 24)
+          )
+          (i32.const 24)
+         )
+        )
+       )
+       (local.set $2
+        (i32.add
+         (local.get $2)
+         (i32.const 1)
+        )
+       )
+       (br_if $label$17
+        (local.tee $0
+         (i32.add
+          (local.get $0)
+          (i32.const -1)
+         )
+        )
+       )
+       (local.set $0
+        (i32.const 0)
+       )
+      )
+     )
+    )
+   )
+   (if (result i32)
+    (local.get $0)
+    (local.get $2)
+    (i32.const 0)
+   )
+  )
+ )
+ (func $18 (; 31 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 224)
+    )
+   )
+   (local.set $5
+    (i32.add
+     (local.get $4)
+     (i32.const 136)
+    )
+   )
+   (i64.store align=4
+    (local.tee $3
+     (i32.add
+      (local.get $4)
+      (i32.const 80)
+     )
+    )
+    (i64.const 0)
+   )
+   (i64.store offset=8 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i64.store offset=16 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i64.store offset=24 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i64.store offset=32 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i32.store
+    (local.tee $6
+     (i32.add
+      (local.get $4)
+      (i32.const 120)
+     )
+    )
+    (i32.load
+     (local.get $2)
+    )
+   )
+   (if
+    (i32.lt_s
+     (call $19
+      (i32.const 0)
+      (local.get $1)
+      (local.get $6)
+      (local.tee $2
+       (local.get $4)
+      )
+      (local.get $3)
+     )
+     (i32.const 0)
+    )
+    (local.set $1
+     (i32.const -1)
+    )
+    (block
+     (local.set $12
+      (if (result i32)
+       (i32.gt_s
+        (i32.load offset=76
+         (local.get $0)
+        )
+        (i32.const -1)
+       )
+       (call $20
+        (local.get $0)
+       )
+       (i32.const 0)
+      )
+     )
+     (local.set $7
+      (i32.load
+       (local.get $0)
+      )
+     )
+     (if
+      (i32.lt_s
+       (i32.load8_s offset=74
+        (local.get $0)
+       )
+       (i32.const 1)
+      )
+      (i32.store
+       (local.get $0)
+       (i32.and
+        (local.get $7)
+        (i32.const -33)
+       )
+      )
+     )
+     (if
+      (i32.load
+       (local.tee $8
+        (i32.add
+         (local.get $0)
+         (i32.const 48)
+        )
+       )
+      )
+      (local.set $1
+       (call $19
+        (local.get $0)
+        (local.get $1)
+        (local.get $6)
+        (local.get $2)
+        (local.get $3)
+       )
+      )
+      (block
+       (local.set $10
+        (i32.load
+         (local.tee $9
+          (i32.add
+           (local.get $0)
+           (i32.const 44)
+          )
+         )
+        )
+       )
+       (i32.store
+        (local.get $9)
+        (local.get $5)
+       )
+       (i32.store
+        (local.tee $13
+         (i32.add
+          (local.get $0)
+          (i32.const 28)
+         )
+        )
+        (local.get $5)
+       )
+       (i32.store
+        (local.tee $11
+         (i32.add
+          (local.get $0)
+          (i32.const 20)
+         )
+        )
+        (local.get $5)
+       )
+       (i32.store
+        (local.get $8)
+        (i32.const 80)
+       )
+       (i32.store
+        (local.tee $14
+         (i32.add
+          (local.get $0)
+          (i32.const 16)
+         )
+        )
+        (i32.add
+         (local.get $5)
+         (i32.const 80)
+        )
+       )
+       (local.set $1
+        (call $19
+         (local.get $0)
+         (local.get $1)
+         (local.get $6)
+         (local.get $2)
+         (local.get $3)
+        )
+       )
+       (if
+        (local.get $10)
+        (block
+         (drop
+          (call_indirect (type $0)
+           (local.get $0)
+           (i32.const 0)
+           (i32.const 0)
+           (i32.add
+            (i32.and
+             (i32.load offset=36
+              (local.get $0)
+             )
+             (i32.const 3)
+            )
+            (i32.const 2)
+           )
+          )
+         )
+         (if
+          (i32.eqz
+           (i32.load
+            (local.get $11)
+           )
+          )
+          (local.set $1
+           (i32.const -1)
+          )
+         )
+         (i32.store
+          (local.get $9)
+          (local.get $10)
+         )
+         (i32.store
+          (local.get $8)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $14)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $13)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $11)
+          (i32.const 0)
+         )
+        )
+       )
+      )
+     )
+     (i32.store
+      (local.get $0)
+      (i32.or
+       (local.tee $2
+        (i32.load
+         (local.get $0)
+        )
+       )
+       (i32.and
+        (local.get $7)
+        (i32.const 32)
+       )
+      )
+     )
+     (if
+      (local.get $12)
+      (call $13
+       (local.get $0)
+      )
+     )
+     (if
+      (i32.and
+       (local.get $2)
+       (i32.const 32)
+      )
+      (local.set $1
+       (i32.const -1)
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $4)
+   )
+   (local.get $1)
+  )
+ )
+ (func $19 (; 32 ;) (type $7) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (param $4 i32) (result i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (local $16 i32)
+  (local $17 i32)
+  (local $18 i32)
+  (local $19 i32)
+  (local $20 i32)
+  (local $21 i32)
+  (local $22 i32)
+  (local $23 i32)
+  (local $24 i32)
+  (local $25 i32)
+  (local $26 i32)
+  (local $27 i32)
+  (local $28 i32)
+  (local $29 i32)
+  (local $30 i32)
+  (local $31 i32)
+  (local $32 i32)
+  (local $33 i32)
+  (local $34 i32)
+  (local $35 i32)
+  (local $36 i32)
+  (local $37 i32)
+  (local $38 i32)
+  (local $39 i32)
+  (local $40 i32)
+  (local $41 i32)
+  (local $42 i32)
+  (local $43 i32)
+  (local $44 i32)
+  (local $45 i32)
+  (local $46 i32)
+  (local $47 i32)
+  (local $48 i32)
+  (local $49 i32)
+  (local $50 i64)
+  (local $51 i64)
+  (local $52 f64)
+  (local $53 f64)
+  (block $label$1 (result i32)
+   (local.set $23
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 624)
+    )
+   )
+   (local.set $20
+    (i32.add
+     (local.get $23)
+     (i32.const 16)
+    )
+   )
+   (local.set $16
+    (local.get $23)
+   )
+   (local.set $36
+    (i32.add
+     (local.get $23)
+     (i32.const 528)
+    )
+   )
+   (local.set $30
+    (i32.ne
+     (local.get $0)
+     (i32.const 0)
+    )
+   )
+   (local.set $38
+    (local.tee $21
+     (i32.add
+      (local.tee $17
+       (i32.add
+        (local.get $23)
+        (i32.const 536)
+       )
+      )
+      (i32.const 40)
+     )
+    )
+   )
+   (local.set $39
+    (i32.add
+     (local.get $17)
+     (i32.const 39)
+    )
+   )
+   (local.set $42
+    (i32.add
+     (local.tee $37
+      (i32.add
+       (local.get $23)
+       (i32.const 8)
+      )
+     )
+     (i32.const 4)
+    )
+   )
+   (local.set $43
+    (i32.sub
+     (i32.const 0)
+     (local.tee $27
+      (local.tee $19
+       (i32.add
+        (local.get $23)
+        (i32.const 588)
+       )
+      )
+     )
+    )
+   )
+   (local.set $33
+    (i32.add
+     (local.tee $17
+      (i32.add
+       (local.get $23)
+       (i32.const 576)
+      )
+     )
+     (i32.const 12)
+    )
+   )
+   (local.set $40
+    (i32.add
+     (local.get $17)
+     (i32.const 11)
+    )
+   )
+   (local.set $44
+    (i32.sub
+     (local.tee $28
+      (local.get $33)
+     )
+     (local.get $27)
+    )
+   )
+   (local.set $45
+    (i32.sub
+     (i32.const -2)
+     (local.get $27)
+    )
+   )
+   (local.set $46
+    (i32.add
+     (local.get $28)
+     (i32.const 2)
+    )
+   )
+   (local.set $48
+    (i32.add
+     (local.tee $47
+      (i32.add
+       (local.get $23)
+       (i32.const 24)
+      )
+     )
+     (i32.const 288)
+    )
+   )
+   (local.set $41
+    (local.tee $31
+     (i32.add
+      (local.get $19)
+      (i32.const 9)
+     )
+    )
+   )
+   (local.set $34
+    (i32.add
+     (local.get $19)
+     (i32.const 8)
+    )
+   )
+   (local.set $15
+    (i32.const 0)
+   )
+   (local.set $10
+    (i32.const 0)
+   )
+   (local.set $17
+    (i32.const 0)
+   )
+   (block $label$2
+    (block $label$3
+     (loop $label$4
+      (block $label$5
+       (if
+        (i32.gt_s
+         (local.get $15)
+         (i32.const -1)
+        )
+        (local.set $15
+         (if (result i32)
+          (i32.gt_s
+           (local.get $10)
+           (i32.sub
+            (i32.const 2147483647)
+            (local.get $15)
+           )
+          )
+          (block (result i32)
+           (i32.store
+            (call $12)
+            (i32.const 75)
+           )
+           (i32.const -1)
+          )
+          (i32.add
+           (local.get $10)
+           (local.get $15)
+          )
+         )
+        )
+       )
+       (br_if $label$3
+        (i32.eqz
+         (i32.shr_s
+          (i32.shl
+           (local.tee $5
+            (i32.load8_s
+             (local.get $1)
+            )
+           )
+           (i32.const 24)
+          )
+          (i32.const 24)
+         )
+        )
+       )
+       (local.set $11
+        (local.get $1)
+       )
+       (block $label$9
+        (block $label$10
+         (loop $label$11
+          (block $label$12
+           (block $label$13
+            (block $label$14
+             (block $label$15
+              (br_table $label$14 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$15 $label$13
+               (i32.sub
+                (i32.shr_s
+                 (i32.shl
+                  (local.get $5)
+                  (i32.const 24)
+                 )
+                 (i32.const 24)
+                )
+                (i32.const 0)
+               )
+              )
+             )
+             (local.set $5
+              (local.get $11)
+             )
+             (br $label$10)
+            )
+            (local.set $5
+             (local.get $11)
+            )
+            (br $label$12)
+           )
+           (local.set $5
+            (i32.load8_s
+             (local.tee $11
+              (i32.add
+               (local.get $11)
+               (i32.const 1)
+              )
+             )
+            )
+           )
+           (br $label$11)
+          )
+         )
+         (br $label$9)
+        )
+        (loop $label$16
+         (br_if $label$9
+          (i32.ne
+           (i32.load8_s offset=1
+            (local.get $5)
+           )
+           (i32.const 37)
+          )
+         )
+         (local.set $11
+          (i32.add
+           (local.get $11)
+           (i32.const 1)
+          )
+         )
+         (br_if $label$16
+          (i32.eq
+           (i32.load8_s
+            (local.tee $5
+             (i32.add
+              (local.get $5)
+              (i32.const 2)
+             )
+            )
+           )
+           (i32.const 37)
+          )
+         )
+        )
+       )
+       (local.set $10
+        (i32.sub
+         (local.get $11)
+         (local.get $1)
+        )
+       )
+       (if
+        (local.get $30)
+        (if
+         (i32.eqz
+          (i32.and
+           (i32.load
+            (local.get $0)
+           )
+           (i32.const 32)
+          )
+         )
+         (drop
+          (call $21
+           (local.get $1)
+           (local.get $10)
+           (local.get $0)
+          )
+         )
+        )
+       )
+       (if
+        (local.get $10)
+        (block
+         (local.set $1
+          (local.get $5)
+         )
+         (br $label$4)
+        )
+       )
+       (local.set $10
+        (if (result i32)
+         (i32.lt_u
+          (local.tee $9
+           (i32.add
+            (i32.shr_s
+             (i32.shl
+              (local.tee $10
+               (i32.load8_s
+                (local.tee $11
+                 (i32.add
+                  (local.get $5)
+                  (i32.const 1)
+                 )
+                )
+               )
+              )
+              (i32.const 24)
+             )
+             (i32.const 24)
+            )
+            (i32.const -48)
+           )
+          )
+          (i32.const 10)
+         )
+         (block (result i32)
+          (local.set $10
+           (i32.add
+            (local.get $5)
+            (i32.const 3)
+           )
+          )
+          (if
+           (local.tee $12
+            (i32.eq
+             (i32.load8_s offset=2
+              (local.get $5)
+             )
+             (i32.const 36)
+            )
+           )
+           (local.set $11
+            (local.get $10)
+           )
+          )
+          (if
+           (local.get $12)
+           (local.set $17
+            (i32.const 1)
+           )
+          )
+          (local.set $5
+           (i32.load8_s
+            (local.get $11)
+           )
+          )
+          (if
+           (i32.eqz
+            (local.get $12)
+           )
+           (local.set $9
+            (i32.const -1)
+           )
+          )
+          (local.get $17)
+         )
+         (block (result i32)
+          (local.set $5
+           (local.get $10)
+          )
+          (local.set $9
+           (i32.const -1)
+          )
+          (local.get $17)
+         )
+        )
+       )
+       (block $label$25
+        (if
+         (i32.lt_u
+          (local.tee $12
+           (i32.add
+            (i32.shr_s
+             (i32.shl
+              (local.get $5)
+              (i32.const 24)
+             )
+             (i32.const 24)
+            )
+            (i32.const -32)
+           )
+          )
+          (i32.const 32)
+         )
+         (block
+          (local.set $17
+           (i32.const 0)
+          )
+          (loop $label$27
+           (br_if $label$25
+            (i32.eqz
+             (i32.and
+              (i32.shl
+               (i32.const 1)
+               (local.get $12)
+              )
+              (i32.const 75913)
+             )
+            )
+           )
+           (local.set $17
+            (i32.or
+             (i32.shl
+              (i32.const 1)
+              (i32.add
+               (i32.shr_s
+                (i32.shl
+                 (local.get $5)
+                 (i32.const 24)
+                )
+                (i32.const 24)
+               )
+               (i32.const -32)
+              )
+             )
+             (local.get $17)
+            )
+           )
+           (br_if $label$27
+            (i32.lt_u
+             (local.tee $12
+              (i32.add
+               (i32.shr_s
+                (i32.shl
+                 (local.tee $5
+                  (i32.load8_s
+                   (local.tee $11
+                    (i32.add
+                     (local.get $11)
+                     (i32.const 1)
+                    )
+                   )
+                  )
+                 )
+                 (i32.const 24)
+                )
+                (i32.const 24)
+               )
+               (i32.const -32)
+              )
+             )
+             (i32.const 32)
+            )
+           )
+          )
+         )
+         (local.set $17
+          (i32.const 0)
+         )
+        )
+       )
+       (block $label$29
+        (if
+         (i32.eq
+          (i32.shr_s
+           (i32.shl
+            (local.get $5)
+            (i32.const 24)
+           )
+           (i32.const 24)
+          )
+          (i32.const 42)
+         )
+         (block
+          (local.set $11
+           (block $label$31 (result i32)
+            (block $label$32
+             (br_if $label$32
+              (i32.ge_u
+               (local.tee $12
+                (i32.add
+                 (i32.shr_s
+                  (i32.shl
+                   (local.tee $5
+                    (i32.load8_s
+                     (local.tee $7
+                      (i32.add
+                       (local.get $11)
+                       (i32.const 1)
+                      )
+                     )
+                    )
+                   )
+                   (i32.const 24)
+                  )
+                  (i32.const 24)
+                 )
+                 (i32.const -48)
+                )
+               )
+               (i32.const 10)
+              )
+             )
+             (br_if $label$32
+              (i32.ne
+               (i32.load8_s offset=2
+                (local.get $11)
+               )
+               (i32.const 36)
+              )
+             )
+             (i32.store
+              (i32.add
+               (local.get $4)
+               (i32.shl
+                (local.get $12)
+                (i32.const 2)
+               )
+              )
+              (i32.const 10)
+             )
+             (local.set $8
+              (i32.const 1)
+             )
+             (local.set $10
+              (i32.wrap_i64
+               (i64.load
+                (i32.add
+                 (local.get $3)
+                 (i32.shl
+                  (i32.add
+                   (i32.load8_s
+                    (local.get $7)
+                   )
+                   (i32.const -48)
+                  )
+                  (i32.const 3)
+                 )
+                )
+               )
+              )
+             )
+             (br $label$31
+              (i32.add
+               (local.get $11)
+               (i32.const 3)
+              )
+             )
+            )
+            (if
+             (local.get $10)
+             (block
+              (local.set $15
+               (i32.const -1)
+              )
+              (br $label$5)
+             )
+            )
+            (if
+             (i32.eqz
+              (local.get $30)
+             )
+             (block
+              (local.set $12
+               (local.get $17)
+              )
+              (local.set $17
+               (i32.const 0)
+              )
+              (local.set $11
+               (local.get $7)
+              )
+              (local.set $10
+               (i32.const 0)
+              )
+              (br $label$29)
+             )
+            )
+            (local.set $10
+             (i32.load
+              (local.tee $11
+               (i32.and
+                (i32.add
+                 (i32.load
+                  (local.get $2)
+                 )
+                 (i32.const 3)
+                )
+                (i32.const -4)
+               )
+              )
+             )
+            )
+            (i32.store
+             (local.get $2)
+             (i32.add
+              (local.get $11)
+              (i32.const 4)
+             )
+            )
+            (local.set $8
+             (i32.const 0)
+            )
+            (local.get $7)
+           )
+          )
+          (local.set $12
+           (i32.or
+            (local.get $17)
+            (i32.const 8192)
+           )
+          )
+          (local.set $7
+           (i32.sub
+            (i32.const 0)
+            (local.get $10)
+           )
+          )
+          (local.set $5
+           (i32.load8_s
+            (local.get $11)
+           )
+          )
+          (if
+           (i32.eqz
+            (local.tee $6
+             (i32.lt_s
+              (local.get $10)
+              (i32.const 0)
+             )
+            )
+           )
+           (local.set $12
+            (local.get $17)
+           )
+          )
+          (local.set $17
+           (local.get $8)
+          )
+          (if
+           (local.get $6)
+           (local.set $10
+            (local.get $7)
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.tee $12
+            (i32.add
+             (i32.shr_s
+              (i32.shl
+               (local.get $5)
+               (i32.const 24)
+              )
+              (i32.const 24)
+             )
+             (i32.const -48)
+            )
+           )
+           (i32.const 10)
+          )
+          (block
+           (local.set $7
+            (i32.const 0)
+           )
+           (local.set $5
+            (local.get $12)
+           )
+           (loop $label$39
+            (local.set $7
+             (i32.add
+              (i32.mul
+               (local.get $7)
+               (i32.const 10)
+              )
+              (local.get $5)
+             )
+            )
+            (br_if $label$39
+             (i32.lt_u
+              (local.tee $5
+               (i32.add
+                (i32.shr_s
+                 (i32.shl
+                  (local.tee $12
+                   (i32.load8_s
+                    (local.tee $11
+                     (i32.add
+                      (local.get $11)
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                  )
+                  (i32.const 24)
+                 )
+                 (i32.const 24)
+                )
+                (i32.const -48)
+               )
+              )
+              (i32.const 10)
+             )
+            )
+           )
+           (if
+            (i32.lt_s
+             (local.get $7)
+             (i32.const 0)
+            )
+            (block
+             (local.set $15
+              (i32.const -1)
+             )
+             (br $label$5)
+            )
+            (block
+             (local.set $5
+              (local.get $12)
+             )
+             (local.set $12
+              (local.get $17)
+             )
+             (local.set $17
+              (local.get $10)
+             )
+             (local.set $10
+              (local.get $7)
+             )
+            )
+           )
+          )
+          (block
+           (local.set $12
+            (local.get $17)
+           )
+           (local.set $17
+            (local.get $10)
+           )
+           (local.set $10
+            (i32.const 0)
+           )
+          )
+         )
+        )
+       )
+       (block $label$43
+        (if
+         (i32.eq
+          (i32.shr_s
+           (i32.shl
+            (local.get $5)
+            (i32.const 24)
+           )
+           (i32.const 24)
+          )
+          (i32.const 46)
+         )
+         (block
+          (if
+           (i32.ne
+            (i32.shr_s
+             (i32.shl
+              (local.tee $5
+               (i32.load8_s
+                (local.tee $7
+                 (i32.add
+                  (local.get $11)
+                  (i32.const 1)
+                 )
+                )
+               )
+              )
+              (i32.const 24)
+             )
+             (i32.const 24)
+            )
+            (i32.const 42)
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.tee $5
+               (i32.add
+                (i32.shr_s
+                 (i32.shl
+                  (local.get $5)
+                  (i32.const 24)
+                 )
+                 (i32.const 24)
+                )
+                (i32.const -48)
+               )
+              )
+              (i32.const 10)
+             )
+             (block
+              (local.set $11
+               (local.get $7)
+              )
+              (local.set $7
+               (i32.const 0)
+              )
+             )
+             (block
+              (local.set $5
+               (i32.const 0)
+              )
+              (local.set $11
+               (local.get $7)
+              )
+              (br $label$43)
+             )
+            )
+            (loop $label$48
+             (local.set $5
+              (i32.add
+               (i32.mul
+                (local.get $7)
+                (i32.const 10)
+               )
+               (local.get $5)
+              )
+             )
+             (br_if $label$43
+              (i32.ge_u
+               (local.tee $8
+                (i32.add
+                 (i32.load8_s
+                  (local.tee $11
+                   (i32.add
+                    (local.get $11)
+                    (i32.const 1)
+                   )
+                  )
+                 )
+                 (i32.const -48)
+                )
+               )
+               (i32.const 10)
+              )
+             )
+             (local.set $7
+              (local.get $5)
+             )
+             (local.set $5
+              (local.get $8)
+             )
+             (br $label$48)
+            )
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.tee $5
+             (i32.add
+              (i32.load8_s
+               (local.tee $7
+                (i32.add
+                 (local.get $11)
+                 (i32.const 2)
+                )
+               )
+              )
+              (i32.const -48)
+             )
+            )
+            (i32.const 10)
+           )
+           (if
+            (i32.eq
+             (i32.load8_s offset=3
+              (local.get $11)
+             )
+             (i32.const 36)
+            )
+            (block
+             (i32.store
+              (i32.add
+               (local.get $4)
+               (i32.shl
+                (local.get $5)
+                (i32.const 2)
+               )
+              )
+              (i32.const 10)
+             )
+             (local.set $5
+              (i32.wrap_i64
+               (i64.load
+                (i32.add
+                 (local.get $3)
+                 (i32.shl
+                  (i32.add
+                   (i32.load8_s
+                    (local.get $7)
+                   )
+                   (i32.const -48)
+                  )
+                  (i32.const 3)
+                 )
+                )
+               )
+              )
+             )
+             (local.set $11
+              (i32.add
+               (local.get $11)
+               (i32.const 4)
+              )
+             )
+             (br $label$43)
+            )
+           )
+          )
+          (if
+           (local.get $17)
+           (block
+            (local.set $15
+             (i32.const -1)
+            )
+            (br $label$5)
+           )
+          )
+          (local.set $11
+           (if (result i32)
+            (local.get $30)
+            (block (result i32)
+             (local.set $5
+              (i32.load
+               (local.tee $11
+                (i32.and
+                 (i32.add
+                  (i32.load
+                   (local.get $2)
+                  )
+                  (i32.const 3)
+                 )
+                 (i32.const -4)
+                )
+               )
+              )
+             )
+             (i32.store
+              (local.get $2)
+              (i32.add
+               (local.get $11)
+               (i32.const 4)
+              )
+             )
+             (local.get $7)
+            )
+            (block (result i32)
+             (local.set $5
+              (i32.const 0)
+             )
+             (local.get $7)
+            )
+           )
+          )
+         )
+         (local.set $5
+          (i32.const -1)
+         )
+        )
+       )
+       (local.set $7
+        (local.get $11)
+       )
+       (local.set $8
+        (i32.const 0)
+       )
+       (loop $label$55
+        (if
+         (i32.gt_u
+          (local.tee $6
+           (i32.add
+            (i32.load8_s
+             (local.get $7)
+            )
+            (i32.const -65)
+           )
+          )
+          (i32.const 57)
+         )
+         (block
+          (local.set $15
+           (i32.const -1)
+          )
+          (br $label$5)
+         )
+        )
+        (local.set $11
+         (i32.add
+          (local.get $7)
+          (i32.const 1)
+         )
+        )
+        (if
+         (i32.lt_u
+          (i32.add
+           (local.tee $6
+            (i32.and
+             (local.tee $13
+              (i32.load8_s
+               (i32.add
+                (i32.add
+                 (i32.mul
+                  (local.get $8)
+                  (i32.const 58)
+                 )
+                 (i32.const 1699)
+                )
+                (local.get $6)
+               )
+              )
+             )
+             (i32.const 255)
+            )
+           )
+           (i32.const -1)
+          )
+          (i32.const 8)
+         )
+         (block
+          (local.set $7
+           (local.get $11)
+          )
+          (local.set $8
+           (local.get $6)
+          )
+          (br $label$55)
+         )
+        )
+       )
+       (if
+        (i32.eqz
+         (i32.shr_s
+          (i32.shl
+           (local.get $13)
+           (i32.const 24)
+          )
+          (i32.const 24)
+         )
+        )
+        (block
+         (local.set $15
+          (i32.const -1)
+         )
+         (br $label$5)
+        )
+       )
+       (local.set $14
+        (i32.gt_s
+         (local.get $9)
+         (i32.const -1)
+        )
+       )
+       (block $label$59
+        (block $label$60
+         (if
+          (i32.eq
+           (i32.shr_s
+            (i32.shl
+             (local.get $13)
+             (i32.const 24)
+            )
+            (i32.const 24)
+           )
+           (i32.const 19)
+          )
+          (if
+           (local.get $14)
+           (block
+            (local.set $15
+             (i32.const -1)
+            )
+            (br $label$5)
+           )
+           (br $label$60)
+          )
+          (block
+           (if
+            (local.get $14)
+            (block
+             (i32.store
+              (i32.add
+               (local.get $4)
+               (i32.shl
+                (local.get $9)
+                (i32.const 2)
+               )
+              )
+              (local.get $6)
+             )
+             (i64.store
+              (local.get $16)
+              (i64.load
+               (i32.add
+                (local.get $3)
+                (i32.shl
+                 (local.get $9)
+                 (i32.const 3)
+                )
+               )
+              )
+             )
+             (br $label$60)
+            )
+           )
+           (if
+            (i32.eqz
+             (local.get $30)
+            )
+            (block
+             (local.set $15
+              (i32.const 0)
+             )
+             (br $label$5)
+            )
+           )
+           (call $22
+            (local.get $16)
+            (local.get $6)
+            (local.get $2)
+           )
+          )
+         )
+         (br $label$59)
+        )
+        (if
+         (i32.eqz
+          (local.get $30)
+         )
+         (block
+          (local.set $10
+           (i32.const 0)
+          )
+          (local.set $1
+           (local.get $11)
+          )
+          (br $label$4)
+         )
+        )
+       )
+       (local.set $9
+        (i32.and
+         (local.tee $7
+          (i32.load8_s
+           (local.get $7)
+          )
+         )
+         (i32.const -33)
+        )
+       )
+       (if
+        (i32.eqz
+         (i32.and
+          (i32.ne
+           (local.get $8)
+           (i32.const 0)
+          )
+          (i32.eq
+           (i32.and
+            (local.get $7)
+            (i32.const 15)
+           )
+           (i32.const 3)
+          )
+         )
+        )
+        (local.set $9
+         (local.get $7)
+        )
+       )
+       (local.set $7
+        (i32.and
+         (local.get $12)
+         (i32.const -65537)
+        )
+       )
+       (if
+        (i32.and
+         (local.get $12)
+         (i32.const 8192)
+        )
+        (local.set $12
+         (local.get $7)
+        )
+       )
+       (block $label$70
+        (block $label$71
+         (block $label$72
+          (block $label$73
+           (block $label$74
+            (block $label$75
+             (block $label$76
+              (block $label$77
+               (block $label$78
+                (block $label$79
+                 (block $label$80
+                  (block $label$81
+                   (block $label$82
+                    (block $label$83
+                     (block $label$84
+                      (block $label$85
+                       (block $label$86
+                        (block $label$87
+                         (block $label$88
+                          (block $label$89
+                           (br_table $label$78 $label$77 $label$80 $label$77 $label$78 $label$78 $label$78 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$79 $label$77 $label$77 $label$77 $label$77 $label$87 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$78 $label$77 $label$83 $label$85 $label$78 $label$78 $label$78 $label$77 $label$85 $label$77 $label$77 $label$77 $label$82 $label$89 $label$86 $label$88 $label$77 $label$77 $label$81 $label$77 $label$84 $label$77 $label$77 $label$87 $label$77
+                            (i32.sub
+                             (local.get $9)
+                             (i32.const 65)
+                            )
+                           )
+                          )
+                          (block $label$90
+                           (block $label$91
+                            (block $label$92
+                             (block $label$93
+                              (block $label$94
+                               (block $label$95
+                                (block $label$96
+                                 (block $label$97
+                                  (br_table $label$97 $label$96 $label$95 $label$94 $label$93 $label$90 $label$92 $label$91 $label$90
+                                   (i32.sub
+                                    (i32.shr_s
+                                     (i32.shl
+                                      (i32.and
+                                       (local.get $8)
+                                       (i32.const 255)
+                                      )
+                                      (i32.const 24)
+                                     )
+                                     (i32.const 24)
+                                    )
+                                    (i32.const 0)
+                                   )
+                                  )
+                                 )
+                                 (i32.store
+                                  (i32.load
+                                   (local.get $16)
+                                  )
+                                  (local.get $15)
+                                 )
+                                 (local.set $10
+                                  (i32.const 0)
+                                 )
+                                 (local.set $1
+                                  (local.get $11)
+                                 )
+                                 (br $label$4)
+                                )
+                                (i32.store
+                                 (i32.load
+                                  (local.get $16)
+                                 )
+                                 (local.get $15)
+                                )
+                                (local.set $10
+                                 (i32.const 0)
+                                )
+                                (local.set $1
+                                 (local.get $11)
+                                )
+                                (br $label$4)
+                               )
+                               (i64.store
+                                (i32.load
+                                 (local.get $16)
+                                )
+                                (i64.extend_i32_s
+                                 (local.get $15)
+                                )
+                               )
+                               (local.set $10
+                                (i32.const 0)
+                               )
+                               (local.set $1
+                                (local.get $11)
+                               )
+                               (br $label$4)
+                              )
+                              (i32.store16
+                               (i32.load
+                                (local.get $16)
+                               )
+                               (local.get $15)
+                              )
+                              (local.set $10
+                               (i32.const 0)
+                              )
+                              (local.set $1
+                               (local.get $11)
+                              )
+                              (br $label$4)
+                             )
+                             (i32.store8
+                              (i32.load
+                               (local.get $16)
+                              )
+                              (local.get $15)
+                             )
+                             (local.set $10
+                              (i32.const 0)
+                             )
+                             (local.set $1
+                              (local.get $11)
+                             )
+                             (br $label$4)
+                            )
+                            (i32.store
+                             (i32.load
+                              (local.get $16)
+                             )
+                             (local.get $15)
+                            )
+                            (local.set $10
+                             (i32.const 0)
+                            )
+                            (local.set $1
+                             (local.get $11)
+                            )
+                            (br $label$4)
+                           )
+                           (i64.store
+                            (i32.load
+                             (local.get $16)
+                            )
+                            (i64.extend_i32_s
+                             (local.get $15)
+                            )
+                           )
+                           (local.set $10
+                            (i32.const 0)
+                           )
+                           (local.set $1
+                            (local.get $11)
+                           )
+                           (br $label$4)
+                          )
+                          (local.set $10
+                           (i32.const 0)
+                          )
+                          (local.set $1
+                           (local.get $11)
+                          )
+                          (br $label$4)
+                         )
+                         (local.set $12
+                          (i32.or
+                           (local.get $12)
+                           (i32.const 8)
+                          )
+                         )
+                         (if
+                          (i32.le_u
+                           (local.get $5)
+                           (i32.const 8)
+                          )
+                          (local.set $5
+                           (i32.const 8)
+                          )
+                         )
+                         (local.set $9
+                          (i32.const 120)
+                         )
+                         (br $label$76)
+                        )
+                        (br $label$76)
+                       )
+                       (if
+                        (i64.eq
+                         (local.tee $50
+                          (i64.load
+                           (local.get $16)
+                          )
+                         )
+                         (i64.const 0)
+                        )
+                        (local.set $7
+                         (local.get $21)
+                        )
+                        (block
+                         (local.set $1
+                          (local.get $21)
+                         )
+                         (loop $label$101
+                          (i64.store8
+                           (local.tee $1
+                            (i32.add
+                             (local.get $1)
+                             (i32.const -1)
+                            )
+                           )
+                           (i64.or
+                            (i64.and
+                             (local.get $50)
+                             (i64.const 7)
+                            )
+                            (i64.const 48)
+                           )
+                          )
+                          (br_if $label$101
+                           (i64.ne
+                            (local.tee $50
+                             (i64.shr_u
+                              (local.get $50)
+                              (i64.const 3)
+                             )
+                            )
+                            (i64.const 0)
+                           )
+                          )
+                          (local.set $7
+                           (local.get $1)
+                          )
+                         )
+                        )
+                       )
+                       (if
+                        (i32.and
+                         (local.get $12)
+                         (i32.const 8)
+                        )
+                        (block
+                         (local.set $8
+                          (i32.add
+                           (local.tee $1
+                            (i32.sub
+                             (local.get $38)
+                             (local.get $7)
+                            )
+                           )
+                           (i32.const 1)
+                          )
+                         )
+                         (if
+                          (i32.le_s
+                           (local.get $5)
+                           (local.get $1)
+                          )
+                          (local.set $5
+                           (local.get $8)
+                          )
+                         )
+                         (local.set $6
+                          (i32.const 0)
+                         )
+                         (local.set $8
+                          (i32.const 2179)
+                         )
+                         (br $label$71)
+                        )
+                        (block
+                         (local.set $6
+                          (i32.const 0)
+                         )
+                         (local.set $8
+                          (i32.const 2179)
+                         )
+                         (br $label$71)
+                        )
+                       )
+                      )
+                      (if
+                       (i64.lt_s
+                        (local.tee $50
+                         (i64.load
+                          (local.get $16)
+                         )
+                        )
+                        (i64.const 0)
+                       )
+                       (block
+                        (i64.store
+                         (local.get $16)
+                         (local.tee $50
+                          (i64.sub
+                           (i64.const 0)
+                           (local.get $50)
+                          )
+                         )
+                        )
+                        (local.set $6
+                         (i32.const 1)
+                        )
+                        (local.set $8
+                         (i32.const 2179)
+                        )
+                        (br $label$75)
+                       )
+                      )
+                      (if
+                       (i32.and
+                        (local.get $12)
+                        (i32.const 2048)
+                       )
+                       (block
+                        (local.set $6
+                         (i32.const 1)
+                        )
+                        (local.set $8
+                         (i32.const 2180)
+                        )
+                        (br $label$75)
+                       )
+                       (block
+                        (local.set $6
+                         (local.tee $1
+                          (i32.and
+                           (local.get $12)
+                           (i32.const 1)
+                          )
+                         )
+                        )
+                        (local.set $8
+                         (if (result i32)
+                          (local.get $1)
+                          (i32.const 2181)
+                          (i32.const 2179)
+                         )
+                        )
+                        (br $label$75)
+                       )
+                      )
+                     )
+                     (local.set $50
+                      (i64.load
+                       (local.get $16)
+                      )
+                     )
+                     (local.set $6
+                      (i32.const 0)
+                     )
+                     (local.set $8
+                      (i32.const 2179)
+                     )
+                     (br $label$75)
+                    )
+                    (i64.store8
+                     (local.get $39)
+                     (i64.load
+                      (local.get $16)
+                     )
+                    )
+                    (local.set $1
+                     (local.get $39)
+                    )
+                    (local.set $12
+                     (local.get $7)
+                    )
+                    (local.set $7
+                     (i32.const 1)
+                    )
+                    (local.set $6
+                     (i32.const 0)
+                    )
+                    (local.set $8
+                     (i32.const 2179)
+                    )
+                    (local.set $5
+                     (local.get $21)
+                    )
+                    (br $label$70)
+                   )
+                   (local.set $1
+                    (call $24
+                     (i32.load
+                      (call $12)
+                     )
+                    )
+                   )
+                   (br $label$74)
+                  )
+                  (if
+                   (i32.eqz
+                    (local.tee $1
+                     (i32.load
+                      (local.get $16)
+                     )
+                    )
+                   )
+                   (local.set $1
+                    (i32.const 2189)
+                   )
+                  )
+                  (br $label$74)
+                 )
+                 (i64.store32
+                  (local.get $37)
+                  (i64.load
+                   (local.get $16)
+                  )
+                 )
+                 (i32.store
+                  (local.get $42)
+                  (i32.const 0)
+                 )
+                 (i32.store
+                  (local.get $16)
+                  (local.get $37)
+                 )
+                 (local.set $7
+                  (local.get $37)
+                 )
+                 (local.set $6
+                  (i32.const -1)
+                 )
+                 (br $label$73)
+                )
+                (local.set $7
+                 (i32.load
+                  (local.get $16)
+                 )
+                )
+                (if
+                 (local.get $5)
+                 (block
+                  (local.set $6
+                   (local.get $5)
+                  )
+                  (br $label$73)
+                 )
+                 (block
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (i32.const 0)
+                   (local.get $12)
+                  )
+                  (local.set $1
+                   (i32.const 0)
+                  )
+                  (br $label$72)
+                 )
+                )
+               )
+               (local.set $52
+                (f64.load
+                 (local.get $16)
+                )
+               )
+               (i32.store
+                (local.get $20)
+                (i32.const 0)
+               )
+               (local.set $26
+                (if (result i32)
+                 (i64.lt_s
+                  (i64.reinterpret_f64
+                   (local.get $52)
+                  )
+                  (i64.const 0)
+                 )
+                 (block (result i32)
+                  (local.set $24
+                   (i32.const 1)
+                  )
+                  (local.set $52
+                   (f64.neg
+                    (local.get $52)
+                   )
+                  )
+                  (i32.const 2196)
+                 )
+                 (block (result i32)
+                  (local.set $1
+                   (i32.and
+                    (local.get $12)
+                    (i32.const 1)
+                   )
+                  )
+                  (if (result i32)
+                   (i32.and
+                    (local.get $12)
+                    (i32.const 2048)
+                   )
+                   (block (result i32)
+                    (local.set $24
+                     (i32.const 1)
+                    )
+                    (i32.const 2199)
+                   )
+                   (block (result i32)
+                    (local.set $24
+                     (local.get $1)
+                    )
+                    (if (result i32)
+                     (local.get $1)
+                     (i32.const 2202)
+                     (i32.const 2197)
+                    )
+                   )
+                  )
+                 )
+                )
+               )
+               (block $label$119
+                (if
+                 (i64.lt_u
+                  (i64.and
+                   (i64.reinterpret_f64
+                    (local.get $52)
+                   )
+                   (i64.const 9218868437227405312)
+                  )
+                  (i64.const 9218868437227405312)
+                 )
+                 (block
+                  (if
+                   (local.tee $1
+                    (f64.ne
+                     (local.tee $52
+                      (f64.mul
+                       (call $27
+                        (local.get $52)
+                        (local.get $20)
+                       )
+                       (f64.const 2)
+                      )
+                     )
+                     (f64.const 0)
+                    )
+                   )
+                   (i32.store
+                    (local.get $20)
+                    (i32.add
+                     (i32.load
+                      (local.get $20)
+                     )
+                     (i32.const -1)
+                    )
+                   )
+                  )
+                  (if
+                   (i32.eq
+                    (local.tee $22
+                     (i32.or
+                      (local.get $9)
+                      (i32.const 32)
+                     )
+                    )
+                    (i32.const 97)
+                   )
+                   (block
+                    (local.set $1
+                     (i32.add
+                      (local.get $26)
+                      (i32.const 9)
+                     )
+                    )
+                    (if
+                     (local.tee $6
+                      (i32.and
+                       (local.get $9)
+                       (i32.const 32)
+                      )
+                     )
+                     (local.set $26
+                      (local.get $1)
+                     )
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.or
+                       (i32.gt_u
+                        (local.get $5)
+                        (i32.const 11)
+                       )
+                       (i32.eqz
+                        (local.tee $1
+                         (i32.sub
+                          (i32.const 12)
+                          (local.get $5)
+                         )
+                        )
+                       )
+                      )
+                     )
+                     (block
+                      (local.set $53
+                       (f64.const 8)
+                      )
+                      (loop $label$125
+                       (local.set $53
+                        (f64.mul
+                         (local.get $53)
+                         (f64.const 16)
+                        )
+                       )
+                       (br_if $label$125
+                        (local.tee $1
+                         (i32.add
+                          (local.get $1)
+                          (i32.const -1)
+                         )
+                        )
+                       )
+                      )
+                      (local.set $52
+                       (if (result f64)
+                        (i32.eq
+                         (i32.load8_s
+                          (local.get $26)
+                         )
+                         (i32.const 45)
+                        )
+                        (f64.neg
+                         (f64.add
+                          (local.get $53)
+                          (f64.sub
+                           (f64.neg
+                            (local.get $52)
+                           )
+                           (local.get $53)
+                          )
+                         )
+                        )
+                        (f64.sub
+                         (f64.add
+                          (local.get $52)
+                          (local.get $53)
+                         )
+                         (local.get $53)
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (local.set $1
+                     (i32.sub
+                      (i32.const 0)
+                      (local.tee $7
+                       (i32.load
+                        (local.get $20)
+                       )
+                      )
+                     )
+                    )
+                    (if
+                     (i32.eq
+                      (local.tee $1
+                       (call $23
+                        (i64.extend_i32_s
+                         (if (result i32)
+                          (i32.lt_s
+                           (local.get $7)
+                           (i32.const 0)
+                          )
+                          (local.get $1)
+                          (local.get $7)
+                         )
+                        )
+                        (local.get $33)
+                       )
+                      )
+                      (local.get $33)
+                     )
+                     (block
+                      (i32.store8
+                       (local.get $40)
+                       (i32.const 48)
+                      )
+                      (local.set $1
+                       (local.get $40)
+                      )
+                     )
+                    )
+                    (local.set $13
+                     (i32.or
+                      (local.get $24)
+                      (i32.const 2)
+                     )
+                    )
+                    (i32.store8
+                     (i32.add
+                      (local.get $1)
+                      (i32.const -1)
+                     )
+                     (i32.add
+                      (i32.and
+                       (i32.shr_s
+                        (local.get $7)
+                        (i32.const 31)
+                       )
+                       (i32.const 2)
+                      )
+                      (i32.const 43)
+                     )
+                    )
+                    (i32.store8
+                     (local.tee $8
+                      (i32.add
+                       (local.get $1)
+                       (i32.const -2)
+                      )
+                     )
+                     (i32.add
+                      (local.get $9)
+                      (i32.const 15)
+                     )
+                    )
+                    (local.set $9
+                     (i32.lt_s
+                      (local.get $5)
+                      (i32.const 1)
+                     )
+                    )
+                    (local.set $14
+                     (i32.eqz
+                      (i32.and
+                       (local.get $12)
+                       (i32.const 8)
+                      )
+                     )
+                    )
+                    (local.set $1
+                     (local.get $19)
+                    )
+                    (loop $label$131
+                     (i32.store8
+                      (local.get $1)
+                      (i32.or
+                       (i32.load8_u
+                        (i32.add
+                         (local.tee $7
+                          (i32.trunc_f64_s
+                           (local.get $52)
+                          )
+                         )
+                         (i32.const 2163)
+                        )
+                       )
+                       (local.get $6)
+                      )
+                     )
+                     (local.set $52
+                      (f64.mul
+                       (f64.sub
+                        (local.get $52)
+                        (f64.convert_i32_s
+                         (local.get $7)
+                        )
+                       )
+                       (f64.const 16)
+                      )
+                     )
+                     (local.set $1
+                      (block $label$132 (result i32)
+                       (if (result i32)
+                        (i32.eq
+                         (i32.sub
+                          (local.tee $7
+                           (i32.add
+                            (local.get $1)
+                            (i32.const 1)
+                           )
+                          )
+                          (local.get $27)
+                         )
+                         (i32.const 1)
+                        )
+                        (block (result i32)
+                         (drop
+                          (br_if $label$132
+                           (local.get $7)
+                           (i32.and
+                            (local.get $14)
+                            (i32.and
+                             (local.get $9)
+                             (f64.eq
+                              (local.get $52)
+                              (f64.const 0)
+                             )
+                            )
+                           )
+                          )
+                         )
+                         (i32.store8
+                          (local.get $7)
+                          (i32.const 46)
+                         )
+                         (i32.add
+                          (local.get $1)
+                          (i32.const 2)
+                         )
+                        )
+                        (local.get $7)
+                       )
+                      )
+                     )
+                     (br_if $label$131
+                      (f64.ne
+                       (local.get $52)
+                       (f64.const 0)
+                      )
+                     )
+                    )
+                    (local.set $6
+                     (i32.sub
+                      (i32.add
+                       (local.get $46)
+                       (local.get $5)
+                      )
+                      (local.tee $7
+                       (local.get $8)
+                      )
+                     )
+                    )
+                    (local.set $9
+                     (i32.add
+                      (i32.sub
+                       (local.get $44)
+                       (local.get $7)
+                      )
+                      (local.get $1)
+                     )
+                    )
+                    (call $25
+                     (local.get $0)
+                     (i32.const 32)
+                     (local.get $10)
+                     (local.tee $5
+                      (i32.add
+                       (if (result i32)
+                        (i32.and
+                         (i32.ne
+                          (local.get $5)
+                          (i32.const 0)
+                         )
+                         (i32.lt_s
+                          (i32.add
+                           (local.get $45)
+                           (local.get $1)
+                          )
+                          (local.get $5)
+                         )
+                        )
+                        (local.get $6)
+                        (local.tee $6
+                         (local.get $9)
+                        )
+                       )
+                       (local.get $13)
+                      )
+                     )
+                     (local.get $12)
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $21
+                       (local.get $26)
+                       (local.get $13)
+                       (local.get $0)
+                      )
+                     )
+                    )
+                    (call $25
+                     (local.get $0)
+                     (i32.const 48)
+                     (local.get $10)
+                     (local.get $5)
+                     (i32.xor
+                      (local.get $12)
+                      (i32.const 65536)
+                     )
+                    )
+                    (local.set $1
+                     (i32.sub
+                      (local.get $1)
+                      (local.get $27)
+                     )
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $21
+                       (local.get $19)
+                       (local.get $1)
+                       (local.get $0)
+                      )
+                     )
+                    )
+                    (call $25
+                     (local.get $0)
+                     (i32.const 48)
+                     (i32.sub
+                      (local.get $6)
+                      (i32.add
+                       (local.get $1)
+                       (local.tee $1
+                        (i32.sub
+                         (local.get $28)
+                         (local.get $7)
+                        )
+                       )
+                      )
+                     )
+                     (i32.const 0)
+                     (i32.const 0)
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $21
+                       (local.get $8)
+                       (local.get $1)
+                       (local.get $0)
+                      )
+                     )
+                    )
+                    (call $25
+                     (local.get $0)
+                     (i32.const 32)
+                     (local.get $10)
+                     (local.get $5)
+                     (i32.xor
+                      (local.get $12)
+                      (i32.const 8192)
+                     )
+                    )
+                    (if
+                     (i32.ge_s
+                      (local.get $5)
+                      (local.get $10)
+                     )
+                     (local.set $10
+                      (local.get $5)
+                     )
+                    )
+                    (br $label$119)
+                   )
+                  )
+                  (if
+                   (local.get $1)
+                   (block
+                    (i32.store
+                     (local.get $20)
+                     (local.tee $6
+                      (i32.add
+                       (i32.load
+                        (local.get $20)
+                       )
+                       (i32.const -28)
+                      )
+                     )
+                    )
+                    (local.set $52
+                     (f64.mul
+                      (local.get $52)
+                      (f64.const 268435456)
+                     )
+                    )
+                   )
+                   (local.set $6
+                    (i32.load
+                     (local.get $20)
+                    )
+                   )
+                  )
+                  (local.set $8
+                   (local.tee $7
+                    (if (result i32)
+                     (i32.lt_s
+                      (local.get $6)
+                      (i32.const 0)
+                     )
+                     (local.get $47)
+                     (local.get $48)
+                    )
+                   )
+                  )
+                  (loop $label$145
+                   (i32.store
+                    (local.get $8)
+                    (local.tee $1
+                     (i32.trunc_f64_s
+                      (local.get $52)
+                     )
+                    )
+                   )
+                   (local.set $8
+                    (i32.add
+                     (local.get $8)
+                     (i32.const 4)
+                    )
+                   )
+                   (br_if $label$145
+                    (f64.ne
+                     (local.tee $52
+                      (f64.mul
+                       (f64.sub
+                        (local.get $52)
+                        (f64.convert_i32_u
+                         (local.get $1)
+                        )
+                       )
+                       (f64.const 1e9)
+                      )
+                     )
+                     (f64.const 0)
+                    )
+                   )
+                  )
+                  (if
+                   (i32.gt_s
+                    (local.get $6)
+                    (i32.const 0)
+                   )
+                   (block
+                    (local.set $1
+                     (local.get $7)
+                    )
+                    (loop $label$147
+                     (local.set $14
+                      (if (result i32)
+                       (i32.gt_s
+                        (local.get $6)
+                        (i32.const 29)
+                       )
+                       (i32.const 29)
+                       (local.get $6)
+                      )
+                     )
+                     (block $label$150
+                      (if
+                       (i32.ge_u
+                        (local.tee $6
+                         (i32.add
+                          (local.get $8)
+                          (i32.const -4)
+                         )
+                        )
+                        (local.get $1)
+                       )
+                       (block
+                        (local.set $50
+                         (i64.extend_i32_u
+                          (local.get $14)
+                         )
+                        )
+                        (local.set $13
+                         (i32.const 0)
+                        )
+                        (loop $label$152
+                         (i64.store32
+                          (local.get $6)
+                          (i64.rem_u
+                           (local.tee $51
+                            (i64.add
+                             (i64.shl
+                              (i64.extend_i32_u
+                               (i32.load
+                                (local.get $6)
+                               )
+                              )
+                              (local.get $50)
+                             )
+                             (i64.extend_i32_u
+                              (local.get $13)
+                             )
+                            )
+                           )
+                           (i64.const 1000000000)
+                          )
+                         )
+                         (local.set $13
+                          (i32.wrap_i64
+                           (i64.div_u
+                            (local.get $51)
+                            (i64.const 1000000000)
+                           )
+                          )
+                         )
+                         (br_if $label$152
+                          (i32.ge_u
+                           (local.tee $6
+                            (i32.add
+                             (local.get $6)
+                             (i32.const -4)
+                            )
+                           )
+                           (local.get $1)
+                          )
+                         )
+                        )
+                        (br_if $label$150
+                         (i32.eqz
+                          (local.get $13)
+                         )
+                        )
+                        (i32.store
+                         (local.tee $1
+                          (i32.add
+                           (local.get $1)
+                           (i32.const -4)
+                          )
+                         )
+                         (local.get $13)
+                        )
+                       )
+                      )
+                     )
+                     (loop $label$153
+                      (if
+                       (i32.gt_u
+                        (local.get $8)
+                        (local.get $1)
+                       )
+                       (if
+                        (i32.eqz
+                         (i32.load
+                          (local.tee $6
+                           (i32.add
+                            (local.get $8)
+                            (i32.const -4)
+                           )
+                          )
+                         )
+                        )
+                        (block
+                         (local.set $8
+                          (local.get $6)
+                         )
+                         (br $label$153)
+                        )
+                       )
+                      )
+                     )
+                     (i32.store
+                      (local.get $20)
+                      (local.tee $6
+                       (i32.sub
+                        (i32.load
+                         (local.get $20)
+                        )
+                        (local.get $14)
+                       )
+                      )
+                     )
+                     (br_if $label$147
+                      (i32.gt_s
+                       (local.get $6)
+                       (i32.const 0)
+                      )
+                     )
+                    )
+                   )
+                   (local.set $1
+                    (local.get $7)
+                   )
+                  )
+                  (local.set $18
+                   (if (result i32)
+                    (i32.lt_s
+                     (local.get $5)
+                     (i32.const 0)
+                    )
+                    (i32.const 6)
+                    (local.get $5)
+                   )
+                  )
+                  (if
+                   (i32.lt_s
+                    (local.get $6)
+                    (i32.const 0)
+                   )
+                   (block
+                    (local.set $14
+                     (i32.add
+                      (i32.div_s
+                       (i32.add
+                        (local.get $18)
+                        (i32.const 25)
+                       )
+                       (i32.const 9)
+                      )
+                      (i32.const 1)
+                     )
+                    )
+                    (local.set $25
+                     (i32.eq
+                      (local.get $22)
+                      (i32.const 102)
+                     )
+                    )
+                    (local.set $5
+                     (local.get $8)
+                    )
+                    (loop $label$160
+                     (if
+                      (i32.gt_s
+                       (local.tee $13
+                        (i32.sub
+                         (i32.const 0)
+                         (local.get $6)
+                        )
+                       )
+                       (i32.const 9)
+                      )
+                      (local.set $13
+                       (i32.const 9)
+                      )
+                     )
+                     (block $label$162
+                      (if
+                       (i32.lt_u
+                        (local.get $1)
+                        (local.get $5)
+                       )
+                       (block
+                        (local.set $29
+                         (i32.add
+                          (i32.shl
+                           (i32.const 1)
+                           (local.get $13)
+                          )
+                          (i32.const -1)
+                         )
+                        )
+                        (local.set $35
+                         (i32.shr_u
+                          (i32.const 1000000000)
+                          (local.get $13)
+                         )
+                        )
+                        (local.set $6
+                         (i32.const 0)
+                        )
+                        (local.set $8
+                         (local.get $1)
+                        )
+                        (loop $label$164
+                         (i32.store
+                          (local.get $8)
+                          (i32.add
+                           (i32.shr_u
+                            (local.tee $32
+                             (i32.load
+                              (local.get $8)
+                             )
+                            )
+                            (local.get $13)
+                           )
+                           (local.get $6)
+                          )
+                         )
+                         (local.set $6
+                          (i32.mul
+                           (i32.and
+                            (local.get $32)
+                            (local.get $29)
+                           )
+                           (local.get $35)
+                          )
+                         )
+                         (br_if $label$164
+                          (i32.lt_u
+                           (local.tee $8
+                            (i32.add
+                             (local.get $8)
+                             (i32.const 4)
+                            )
+                           )
+                           (local.get $5)
+                          )
+                         )
+                        )
+                        (local.set $8
+                         (i32.add
+                          (local.get $1)
+                          (i32.const 4)
+                         )
+                        )
+                        (if
+                         (i32.eqz
+                          (i32.load
+                           (local.get $1)
+                          )
+                         )
+                         (local.set $1
+                          (local.get $8)
+                         )
+                        )
+                        (br_if $label$162
+                         (i32.eqz
+                          (local.get $6)
+                         )
+                        )
+                        (i32.store
+                         (local.get $5)
+                         (local.get $6)
+                        )
+                        (local.set $5
+                         (i32.add
+                          (local.get $5)
+                          (i32.const 4)
+                         )
+                        )
+                       )
+                       (block
+                        (local.set $8
+                         (i32.add
+                          (local.get $1)
+                          (i32.const 4)
+                         )
+                        )
+                        (if
+                         (i32.eqz
+                          (i32.load
+                           (local.get $1)
+                          )
+                         )
+                         (local.set $1
+                          (local.get $8)
+                         )
+                        )
+                       )
+                      )
+                     )
+                     (local.set $6
+                      (i32.add
+                       (local.tee $8
+                        (if (result i32)
+                         (local.get $25)
+                         (local.get $7)
+                         (local.get $1)
+                        )
+                       )
+                       (i32.shl
+                        (local.get $14)
+                        (i32.const 2)
+                       )
+                      )
+                     )
+                     (if
+                      (i32.gt_s
+                       (i32.shr_s
+                        (i32.sub
+                         (local.get $5)
+                         (local.get $8)
+                        )
+                        (i32.const 2)
+                       )
+                       (local.get $14)
+                      )
+                      (local.set $5
+                       (local.get $6)
+                      )
+                     )
+                     (i32.store
+                      (local.get $20)
+                      (local.tee $6
+                       (i32.add
+                        (i32.load
+                         (local.get $20)
+                        )
+                        (local.get $13)
+                       )
+                      )
+                     )
+                     (br_if $label$160
+                      (i32.lt_s
+                       (local.get $6)
+                       (i32.const 0)
+                      )
+                     )
+                     (local.set $13
+                      (local.get $5)
+                     )
+                    )
+                   )
+                   (local.set $13
+                    (local.get $8)
+                   )
+                  )
+                  (local.set $25
+                   (local.get $7)
+                  )
+                  (block $label$172
+                   (if
+                    (i32.lt_u
+                     (local.get $1)
+                     (local.get $13)
+                    )
+                    (block
+                     (local.set $5
+                      (i32.mul
+                       (i32.shr_s
+                        (i32.sub
+                         (local.get $25)
+                         (local.get $1)
+                        )
+                        (i32.const 2)
+                       )
+                       (i32.const 9)
+                      )
+                     )
+                     (br_if $label$172
+                      (i32.lt_u
+                       (local.tee $6
+                        (i32.load
+                         (local.get $1)
+                        )
+                       )
+                       (i32.const 10)
+                      )
+                     )
+                     (local.set $8
+                      (i32.const 10)
+                     )
+                     (loop $label$174
+                      (local.set $5
+                       (i32.add
+                        (local.get $5)
+                        (i32.const 1)
+                       )
+                      )
+                      (br_if $label$174
+                       (i32.ge_u
+                        (local.get $6)
+                        (local.tee $8
+                         (i32.mul
+                          (local.get $8)
+                          (i32.const 10)
+                         )
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (local.set $5
+                     (i32.const 0)
+                    )
+                   )
+                  )
+                  (local.set $29
+                   (i32.eq
+                    (local.get $22)
+                    (i32.const 103)
+                   )
+                  )
+                  (local.set $35
+                   (i32.ne
+                    (local.get $18)
+                    (i32.const 0)
+                   )
+                  )
+                  (if
+                   (i32.lt_s
+                    (local.tee $8
+                     (i32.add
+                      (i32.sub
+                       (local.get $18)
+                       (if (result i32)
+                        (i32.ne
+                         (local.get $22)
+                         (i32.const 102)
+                        )
+                        (local.get $5)
+                        (i32.const 0)
+                       )
+                      )
+                      (i32.shr_s
+                       (i32.shl
+                        (i32.and
+                         (local.get $35)
+                         (local.get $29)
+                        )
+                        (i32.const 31)
+                       )
+                       (i32.const 31)
+                      )
+                     )
+                    )
+                    (i32.add
+                     (i32.mul
+                      (i32.shr_s
+                       (i32.sub
+                        (local.get $13)
+                        (local.get $25)
+                       )
+                       (i32.const 2)
+                      )
+                      (i32.const 9)
+                     )
+                     (i32.const -9)
+                    )
+                   )
+                   (block
+                    (if
+                     (i32.lt_s
+                      (local.tee $8
+                       (i32.add
+                        (i32.rem_s
+                         (local.tee $14
+                          (i32.add
+                           (local.get $8)
+                           (i32.const 9216)
+                          )
+                         )
+                         (i32.const 9)
+                        )
+                        (i32.const 1)
+                       )
+                      )
+                      (i32.const 9)
+                     )
+                     (block
+                      (local.set $6
+                       (i32.const 10)
+                      )
+                      (loop $label$180
+                       (local.set $6
+                        (i32.mul
+                         (local.get $6)
+                         (i32.const 10)
+                        )
+                       )
+                       (br_if $label$180
+                        (i32.ne
+                         (local.tee $8
+                          (i32.add
+                           (local.get $8)
+                           (i32.const 1)
+                          )
+                         )
+                         (i32.const 9)
+                        )
+                       )
+                      )
+                     )
+                     (local.set $6
+                      (i32.const 10)
+                     )
+                    )
+                    (local.set $14
+                     (i32.rem_u
+                      (local.tee $22
+                       (i32.load
+                        (local.tee $8
+                         (i32.add
+                          (i32.add
+                           (local.get $7)
+                           (i32.const 4)
+                          )
+                          (i32.shl
+                           (i32.add
+                            (i32.div_s
+                             (local.get $14)
+                             (i32.const 9)
+                            )
+                            (i32.const -1024)
+                           )
+                           (i32.const 2)
+                          )
+                         )
+                        )
+                       )
+                      )
+                      (local.get $6)
+                     )
+                    )
+                    (block $label$182
+                     (if
+                      (i32.eqz
+                       (i32.and
+                        (local.tee $32
+                         (i32.eq
+                          (i32.add
+                           (local.get $8)
+                           (i32.const 4)
+                          )
+                          (local.get $13)
+                         )
+                        )
+                        (i32.eqz
+                         (local.get $14)
+                        )
+                       )
+                      )
+                      (block
+                       (local.set $52
+                        (if (result f64)
+                         (i32.lt_u
+                          (local.get $14)
+                          (local.tee $49
+                           (i32.div_s
+                            (local.get $6)
+                            (i32.const 2)
+                           )
+                          )
+                         )
+                         (f64.const 0.5)
+                         (if (result f64)
+                          (i32.and
+                           (local.get $32)
+                           (i32.eq
+                            (local.get $14)
+                            (local.get $49)
+                           )
+                          )
+                          (f64.const 1)
+                          (f64.const 1.5)
+                         )
+                        )
+                       )
+                       (local.set $53
+                        (if (result f64)
+                         (i32.and
+                          (i32.div_u
+                           (local.get $22)
+                           (local.get $6)
+                          )
+                          (i32.const 1)
+                         )
+                         (f64.const 9007199254740994)
+                         (f64.const 9007199254740992)
+                        )
+                       )
+                       (block $label$190
+                        (if
+                         (local.get $24)
+                         (block
+                          (br_if $label$190
+                           (i32.ne
+                            (i32.load8_s
+                             (local.get $26)
+                            )
+                            (i32.const 45)
+                           )
+                          )
+                          (local.set $53
+                           (f64.neg
+                            (local.get $53)
+                           )
+                          )
+                          (local.set $52
+                           (f64.neg
+                            (local.get $52)
+                           )
+                          )
+                         )
+                        )
+                       )
+                       (i32.store
+                        (local.get $8)
+                        (local.tee $14
+                         (i32.sub
+                          (local.get $22)
+                          (local.get $14)
+                         )
+                        )
+                       )
+                       (br_if $label$182
+                        (f64.eq
+                         (f64.add
+                          (local.get $53)
+                          (local.get $52)
+                         )
+                         (local.get $53)
+                        )
+                       )
+                       (i32.store
+                        (local.get $8)
+                        (local.tee $5
+                         (i32.add
+                          (local.get $14)
+                          (local.get $6)
+                         )
+                        )
+                       )
+                       (if
+                        (i32.gt_u
+                         (local.get $5)
+                         (i32.const 999999999)
+                        )
+                        (loop $label$193
+                         (i32.store
+                          (local.get $8)
+                          (i32.const 0)
+                         )
+                         (if
+                          (i32.lt_u
+                           (local.tee $8
+                            (i32.add
+                             (local.get $8)
+                             (i32.const -4)
+                            )
+                           )
+                           (local.get $1)
+                          )
+                          (i32.store
+                           (local.tee $1
+                            (i32.add
+                             (local.get $1)
+                             (i32.const -4)
+                            )
+                           )
+                           (i32.const 0)
+                          )
+                         )
+                         (i32.store
+                          (local.get $8)
+                          (local.tee $5
+                           (i32.add
+                            (i32.load
+                             (local.get $8)
+                            )
+                            (i32.const 1)
+                           )
+                          )
+                         )
+                         (br_if $label$193
+                          (i32.gt_u
+                           (local.get $5)
+                           (i32.const 999999999)
+                          )
+                         )
+                        )
+                       )
+                       (local.set $5
+                        (i32.mul
+                         (i32.shr_s
+                          (i32.sub
+                           (local.get $25)
+                           (local.get $1)
+                          )
+                          (i32.const 2)
+                         )
+                         (i32.const 9)
+                        )
+                       )
+                       (br_if $label$182
+                        (i32.lt_u
+                         (local.tee $14
+                          (i32.load
+                           (local.get $1)
+                          )
+                         )
+                         (i32.const 10)
+                        )
+                       )
+                       (local.set $6
+                        (i32.const 10)
+                       )
+                       (loop $label$195
+                        (local.set $5
+                         (i32.add
+                          (local.get $5)
+                          (i32.const 1)
+                         )
+                        )
+                        (br_if $label$195
+                         (i32.ge_u
+                          (local.get $14)
+                          (local.tee $6
+                           (i32.mul
+                            (local.get $6)
+                            (i32.const 10)
+                           )
+                          )
+                         )
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (local.set $14
+                     (local.get $1)
+                    )
+                    (local.set $6
+                     (local.get $5)
+                    )
+                    (if
+                     (i32.le_u
+                      (local.get $13)
+                      (local.tee $8
+                       (i32.add
+                        (local.get $8)
+                        (i32.const 4)
+                       )
+                      )
+                     )
+                     (local.set $8
+                      (local.get $13)
+                     )
+                    )
+                   )
+                   (block
+                    (local.set $14
+                     (local.get $1)
+                    )
+                    (local.set $6
+                     (local.get $5)
+                    )
+                    (local.set $8
+                     (local.get $13)
+                    )
+                   )
+                  )
+                  (local.set $32
+                   (i32.sub
+                    (i32.const 0)
+                    (local.get $6)
+                   )
+                  )
+                  (loop $label$198
+                   (block $label$199
+                    (if
+                     (i32.le_u
+                      (local.get $8)
+                      (local.get $14)
+                     )
+                     (block
+                      (local.set $22
+                       (i32.const 0)
+                      )
+                      (br $label$199)
+                     )
+                    )
+                    (if
+                     (i32.load
+                      (local.tee $1
+                       (i32.add
+                        (local.get $8)
+                        (i32.const -4)
+                       )
+                      )
+                     )
+                     (local.set $22
+                      (i32.const 1)
+                     )
+                     (block
+                      (local.set $8
+                       (local.get $1)
+                      )
+                      (br $label$198)
+                     )
+                    )
+                   )
+                  )
+                  (block $label$203
+                   (if
+                    (local.get $29)
+                    (block
+                     (local.set $1
+                      (if (result i32)
+                       (i32.and
+                        (i32.gt_s
+                         (local.tee $1
+                          (i32.add
+                           (i32.xor
+                            (i32.and
+                             (local.get $35)
+                             (i32.const 1)
+                            )
+                            (i32.const 1)
+                           )
+                           (local.get $18)
+                          )
+                         )
+                         (local.get $6)
+                        )
+                        (i32.gt_s
+                         (local.get $6)
+                         (i32.const -5)
+                        )
+                       )
+                       (block (result i32)
+                        (local.set $5
+                         (i32.add
+                          (local.get $9)
+                          (i32.const -1)
+                         )
+                        )
+                        (i32.sub
+                         (i32.add
+                          (local.get $1)
+                          (i32.const -1)
+                         )
+                         (local.get $6)
+                        )
+                       )
+                       (block (result i32)
+                        (local.set $5
+                         (i32.add
+                          (local.get $9)
+                          (i32.const -2)
+                         )
+                        )
+                        (i32.add
+                         (local.get $1)
+                         (i32.const -1)
+                        )
+                       )
+                      )
+                     )
+                     (br_if $label$203
+                      (local.tee $13
+                       (i32.and
+                        (local.get $12)
+                        (i32.const 8)
+                       )
+                      )
+                     )
+                     (block $label$207
+                      (if
+                       (local.get $22)
+                       (block
+                        (if
+                         (i32.eqz
+                          (local.tee $18
+                           (i32.load
+                            (i32.add
+                             (local.get $8)
+                             (i32.const -4)
+                            )
+                           )
+                          )
+                         )
+                         (block
+                          (local.set $9
+                           (i32.const 9)
+                          )
+                          (br $label$207)
+                         )
+                        )
+                        (if
+                         (i32.rem_u
+                          (local.get $18)
+                          (i32.const 10)
+                         )
+                         (block
+                          (local.set $9
+                           (i32.const 0)
+                          )
+                          (br $label$207)
+                         )
+                         (block
+                          (local.set $13
+                           (i32.const 10)
+                          )
+                          (local.set $9
+                           (i32.const 0)
+                          )
+                         )
+                        )
+                        (loop $label$212
+                         (local.set $9
+                          (i32.add
+                           (local.get $9)
+                           (i32.const 1)
+                          )
+                         )
+                         (br_if $label$212
+                          (i32.eqz
+                           (i32.rem_u
+                            (local.get $18)
+                            (local.tee $13
+                             (i32.mul
+                              (local.get $13)
+                              (i32.const 10)
+                             )
+                            )
+                           )
+                          )
+                         )
+                        )
+                       )
+                       (local.set $9
+                        (i32.const 9)
+                       )
+                      )
+                     )
+                     (local.set $18
+                      (i32.add
+                       (i32.mul
+                        (i32.shr_s
+                         (i32.sub
+                          (local.get $8)
+                          (local.get $25)
+                         )
+                         (i32.const 2)
+                        )
+                        (i32.const 9)
+                       )
+                       (i32.const -9)
+                      )
+                     )
+                     (if
+                      (i32.eq
+                       (i32.or
+                        (local.get $5)
+                        (i32.const 32)
+                       )
+                       (i32.const 102)
+                      )
+                      (block
+                       (local.set $13
+                        (i32.const 0)
+                       )
+                       (if
+                        (i32.ge_s
+                         (local.get $1)
+                         (if (result i32)
+                          (i32.lt_s
+                           (local.tee $9
+                            (i32.sub
+                             (local.get $18)
+                             (local.get $9)
+                            )
+                           )
+                           (i32.const 0)
+                          )
+                          (local.tee $9
+                           (i32.const 0)
+                          )
+                          (local.get $9)
+                         )
+                        )
+                        (local.set $1
+                         (local.get $9)
+                        )
+                       )
+                      )
+                      (block
+                       (local.set $13
+                        (i32.const 0)
+                       )
+                       (if
+                        (i32.ge_s
+                         (local.get $1)
+                         (if (result i32)
+                          (i32.lt_s
+                           (local.tee $9
+                            (i32.sub
+                             (i32.add
+                              (local.get $18)
+                              (local.get $6)
+                             )
+                             (local.get $9)
+                            )
+                           )
+                           (i32.const 0)
+                          )
+                          (local.tee $9
+                           (i32.const 0)
+                          )
+                          (local.get $9)
+                         )
+                        )
+                        (local.set $1
+                         (local.get $9)
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (block
+                     (local.set $13
+                      (i32.and
+                       (local.get $12)
+                       (i32.const 8)
+                      )
+                     )
+                     (local.set $1
+                      (local.get $18)
+                     )
+                     (local.set $5
+                      (local.get $9)
+                     )
+                    )
+                   )
+                  )
+                  (if
+                   (local.tee $25
+                    (i32.eq
+                     (i32.or
+                      (local.get $5)
+                      (i32.const 32)
+                     )
+                     (i32.const 102)
+                    )
+                   )
+                   (block
+                    (local.set $9
+                     (i32.const 0)
+                    )
+                    (if
+                     (i32.le_s
+                      (local.get $6)
+                      (i32.const 0)
+                     )
+                     (local.set $6
+                      (i32.const 0)
+                     )
+                    )
+                   )
+                   (block
+                    (if
+                     (i32.lt_s
+                      (i32.sub
+                       (local.get $28)
+                       (local.tee $9
+                        (call $23
+                         (i64.extend_i32_s
+                          (if (result i32)
+                           (i32.lt_s
+                            (local.get $6)
+                            (i32.const 0)
+                           )
+                           (local.get $32)
+                           (local.get $6)
+                          )
+                         )
+                         (local.get $33)
+                        )
+                       )
+                      )
+                      (i32.const 2)
+                     )
+                     (loop $label$229
+                      (i32.store8
+                       (local.tee $9
+                        (i32.add
+                         (local.get $9)
+                         (i32.const -1)
+                        )
+                       )
+                       (i32.const 48)
+                      )
+                      (br_if $label$229
+                       (i32.lt_s
+                        (i32.sub
+                         (local.get $28)
+                         (local.get $9)
+                        )
+                        (i32.const 2)
+                       )
+                      )
+                     )
+                    )
+                    (i32.store8
+                     (i32.add
+                      (local.get $9)
+                      (i32.const -1)
+                     )
+                     (i32.add
+                      (i32.and
+                       (i32.shr_s
+                        (local.get $6)
+                        (i32.const 31)
+                       )
+                       (i32.const 2)
+                      )
+                      (i32.const 43)
+                     )
+                    )
+                    (i32.store8
+                     (local.tee $6
+                      (i32.add
+                       (local.get $9)
+                       (i32.const -2)
+                      )
+                     )
+                     (local.get $5)
+                    )
+                    (local.set $9
+                     (local.get $6)
+                    )
+                    (local.set $6
+                     (i32.sub
+                      (local.get $28)
+                      (local.get $6)
+                     )
+                    )
+                   )
+                  )
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.tee $18
+                    (i32.add
+                     (i32.add
+                      (i32.add
+                       (i32.add
+                        (local.get $24)
+                        (i32.const 1)
+                       )
+                       (local.get $1)
+                      )
+                      (i32.ne
+                       (local.tee $29
+                        (i32.or
+                         (local.get $1)
+                         (local.get $13)
+                        )
+                       )
+                       (i32.const 0)
+                      )
+                     )
+                     (local.get $6)
+                    )
+                   )
+                   (local.get $12)
+                  )
+                  (if
+                   (i32.eqz
+                    (i32.and
+                     (i32.load
+                      (local.get $0)
+                     )
+                     (i32.const 32)
+                    )
+                   )
+                   (drop
+                    (call $21
+                     (local.get $26)
+                     (local.get $24)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (call $25
+                   (local.get $0)
+                   (i32.const 48)
+                   (local.get $10)
+                   (local.get $18)
+                   (i32.xor
+                    (local.get $12)
+                    (i32.const 65536)
+                   )
+                  )
+                  (block $label$231
+                   (if
+                    (local.get $25)
+                    (block
+                     (local.set $6
+                      (local.tee $9
+                       (if (result i32)
+                        (i32.gt_u
+                         (local.get $14)
+                         (local.get $7)
+                        )
+                        (local.get $7)
+                        (local.get $14)
+                       )
+                      )
+                     )
+                     (loop $label$235
+                      (local.set $5
+                       (call $23
+                        (i64.extend_i32_u
+                         (i32.load
+                          (local.get $6)
+                         )
+                        )
+                        (local.get $31)
+                       )
+                      )
+                      (block $label$236
+                       (if
+                        (i32.eq
+                         (local.get $6)
+                         (local.get $9)
+                        )
+                        (block
+                         (br_if $label$236
+                          (i32.ne
+                           (local.get $5)
+                           (local.get $31)
+                          )
+                         )
+                         (i32.store8
+                          (local.get $34)
+                          (i32.const 48)
+                         )
+                         (local.set $5
+                          (local.get $34)
+                         )
+                        )
+                        (block
+                         (br_if $label$236
+                          (i32.le_u
+                           (local.get $5)
+                           (local.get $19)
+                          )
+                         )
+                         (drop
+                          (call $46
+                           (local.get $19)
+                           (i32.const 48)
+                           (i32.sub
+                            (local.get $5)
+                            (local.get $27)
+                           )
+                          )
+                         )
+                         (loop $label$239
+                          (br_if $label$239
+                           (i32.gt_u
+                            (local.tee $5
+                             (i32.add
+                              (local.get $5)
+                              (i32.const -1)
+                             )
+                            )
+                            (local.get $19)
+                           )
+                          )
+                         )
+                        )
+                       )
+                      )
+                      (if
+                       (i32.eqz
+                        (i32.and
+                         (i32.load
+                          (local.get $0)
+                         )
+                         (i32.const 32)
+                        )
+                       )
+                       (drop
+                        (call $21
+                         (local.get $5)
+                         (i32.sub
+                          (local.get $41)
+                          (local.get $5)
+                         )
+                         (local.get $0)
+                        )
+                       )
+                      )
+                      (if
+                       (i32.le_u
+                        (local.tee $5
+                         (i32.add
+                          (local.get $6)
+                          (i32.const 4)
+                         )
+                        )
+                        (local.get $7)
+                       )
+                       (block
+                        (local.set $6
+                         (local.get $5)
+                        )
+                        (br $label$235)
+                       )
+                      )
+                     )
+                     (block $label$242
+                      (if
+                       (local.get $29)
+                       (block
+                        (br_if $label$242
+                         (i32.and
+                          (i32.load
+                           (local.get $0)
+                          )
+                          (i32.const 32)
+                         )
+                        )
+                        (drop
+                         (call $21
+                          (i32.const 2231)
+                          (i32.const 1)
+                          (local.get $0)
+                         )
+                        )
+                       )
+                      )
+                     )
+                     (if
+                      (i32.and
+                       (i32.gt_s
+                        (local.get $1)
+                        (i32.const 0)
+                       )
+                       (i32.lt_u
+                        (local.get $5)
+                        (local.get $8)
+                       )
+                      )
+                      (loop $label$245
+                       (if
+                        (i32.gt_u
+                         (local.tee $7
+                          (call $23
+                           (i64.extend_i32_u
+                            (i32.load
+                             (local.get $5)
+                            )
+                           )
+                           (local.get $31)
+                          )
+                         )
+                         (local.get $19)
+                        )
+                        (block
+                         (drop
+                          (call $46
+                           (local.get $19)
+                           (i32.const 48)
+                           (i32.sub
+                            (local.get $7)
+                            (local.get $27)
+                           )
+                          )
+                         )
+                         (loop $label$247
+                          (br_if $label$247
+                           (i32.gt_u
+                            (local.tee $7
+                             (i32.add
+                              (local.get $7)
+                              (i32.const -1)
+                             )
+                            )
+                            (local.get $19)
+                           )
+                          )
+                         )
+                        )
+                       )
+                       (if
+                        (i32.eqz
+                         (i32.and
+                          (i32.load
+                           (local.get $0)
+                          )
+                          (i32.const 32)
+                         )
+                        )
+                        (drop
+                         (call $21
+                          (local.get $7)
+                          (if (result i32)
+                           (i32.gt_s
+                            (local.get $1)
+                            (i32.const 9)
+                           )
+                           (i32.const 9)
+                           (local.get $1)
+                          )
+                          (local.get $0)
+                         )
+                        )
+                       )
+                       (local.set $7
+                        (i32.add
+                         (local.get $1)
+                         (i32.const -9)
+                        )
+                       )
+                       (if
+                        (i32.and
+                         (i32.gt_s
+                          (local.get $1)
+                          (i32.const 9)
+                         )
+                         (i32.lt_u
+                          (local.tee $5
+                           (i32.add
+                            (local.get $5)
+                            (i32.const 4)
+                           )
+                          )
+                          (local.get $8)
+                         )
+                        )
+                        (block
+                         (local.set $1
+                          (local.get $7)
+                         )
+                         (br $label$245)
+                        )
+                        (local.set $1
+                         (local.get $7)
+                        )
+                       )
+                      )
+                     )
+                     (call $25
+                      (local.get $0)
+                      (i32.const 48)
+                      (i32.add
+                       (local.get $1)
+                       (i32.const 9)
+                      )
+                      (i32.const 9)
+                      (i32.const 0)
+                     )
+                    )
+                    (block
+                     (local.set $5
+                      (i32.add
+                       (local.get $14)
+                       (i32.const 4)
+                      )
+                     )
+                     (if
+                      (i32.eqz
+                       (local.get $22)
+                      )
+                      (local.set $8
+                       (local.get $5)
+                      )
+                     )
+                     (if
+                      (i32.gt_s
+                       (local.get $1)
+                       (i32.const -1)
+                      )
+                      (block
+                       (local.set $13
+                        (i32.eqz
+                         (local.get $13)
+                        )
+                       )
+                       (local.set $7
+                        (local.get $14)
+                       )
+                       (local.set $5
+                        (local.get $1)
+                       )
+                       (loop $label$256
+                        (if
+                         (i32.eq
+                          (local.tee $1
+                           (call $23
+                            (i64.extend_i32_u
+                             (i32.load
+                              (local.get $7)
+                             )
+                            )
+                            (local.get $31)
+                           )
+                          )
+                          (local.get $31)
+                         )
+                         (block
+                          (i32.store8
+                           (local.get $34)
+                           (i32.const 48)
+                          )
+                          (local.set $1
+                           (local.get $34)
+                          )
+                         )
+                        )
+                        (block $label$258
+                         (if
+                          (i32.eq
+                           (local.get $7)
+                           (local.get $14)
+                          )
+                          (block
+                           (if
+                            (i32.eqz
+                             (i32.and
+                              (i32.load
+                               (local.get $0)
+                              )
+                              (i32.const 32)
+                             )
+                            )
+                            (drop
+                             (call $21
+                              (local.get $1)
+                              (i32.const 1)
+                              (local.get $0)
+                             )
+                            )
+                           )
+                           (local.set $1
+                            (i32.add
+                             (local.get $1)
+                             (i32.const 1)
+                            )
+                           )
+                           (br_if $label$258
+                            (i32.and
+                             (local.get $13)
+                             (i32.lt_s
+                              (local.get $5)
+                              (i32.const 1)
+                             )
+                            )
+                           )
+                           (br_if $label$258
+                            (i32.and
+                             (i32.load
+                              (local.get $0)
+                             )
+                             (i32.const 32)
+                            )
+                           )
+                           (drop
+                            (call $21
+                             (i32.const 2231)
+                             (i32.const 1)
+                             (local.get $0)
+                            )
+                           )
+                          )
+                          (block
+                           (br_if $label$258
+                            (i32.le_u
+                             (local.get $1)
+                             (local.get $19)
+                            )
+                           )
+                           (drop
+                            (call $46
+                             (local.get $19)
+                             (i32.const 48)
+                             (i32.add
+                              (local.get $1)
+                              (local.get $43)
+                             )
+                            )
+                           )
+                           (loop $label$262
+                            (br_if $label$262
+                             (i32.gt_u
+                              (local.tee $1
+                               (i32.add
+                                (local.get $1)
+                                (i32.const -1)
+                               )
+                              )
+                              (local.get $19)
+                             )
+                            )
+                           )
+                          )
+                         )
+                        )
+                        (local.set $6
+                         (i32.sub
+                          (local.get $41)
+                          (local.get $1)
+                         )
+                        )
+                        (if
+                         (i32.eqz
+                          (i32.and
+                           (i32.load
+                            (local.get $0)
+                           )
+                           (i32.const 32)
+                          )
+                         )
+                         (drop
+                          (call $21
+                           (local.get $1)
+                           (if (result i32)
+                            (i32.gt_s
+                             (local.get $5)
+                             (local.get $6)
+                            )
+                            (local.get $6)
+                            (local.get $5)
+                           )
+                           (local.get $0)
+                          )
+                         )
+                        )
+                        (br_if $label$256
+                         (i32.and
+                          (i32.lt_u
+                           (local.tee $7
+                            (i32.add
+                             (local.get $7)
+                             (i32.const 4)
+                            )
+                           )
+                           (local.get $8)
+                          )
+                          (i32.gt_s
+                           (local.tee $5
+                            (i32.sub
+                             (local.get $5)
+                             (local.get $6)
+                            )
+                           )
+                           (i32.const -1)
+                          )
+                         )
+                        )
+                        (local.set $1
+                         (local.get $5)
+                        )
+                       )
+                      )
+                     )
+                     (call $25
+                      (local.get $0)
+                      (i32.const 48)
+                      (i32.add
+                       (local.get $1)
+                       (i32.const 18)
+                      )
+                      (i32.const 18)
+                      (i32.const 0)
+                     )
+                     (br_if $label$231
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $21
+                       (local.get $9)
+                       (i32.sub
+                        (local.get $28)
+                        (local.get $9)
+                       )
+                       (local.get $0)
+                      )
+                     )
+                    )
+                   )
+                  )
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.get $18)
+                   (i32.xor
+                    (local.get $12)
+                    (i32.const 8192)
+                   )
+                  )
+                  (if
+                   (i32.ge_s
+                    (local.get $18)
+                    (local.get $10)
+                   )
+                   (local.set $10
+                    (local.get $18)
+                   )
+                  )
+                 )
+                 (block
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.tee $8
+                    (i32.add
+                     (if (result i32)
+                      (local.tee $6
+                       (i32.or
+                        (f64.ne
+                         (local.get $52)
+                         (local.get $52)
+                        )
+                        (i32.const 0)
+                       )
+                      )
+                      (local.tee $24
+                       (i32.const 0)
+                      )
+                      (local.get $24)
+                     )
+                     (i32.const 3)
+                    )
+                   )
+                   (local.get $7)
+                  )
+                  (if
+                   (i32.eqz
+                    (i32.and
+                     (local.tee $1
+                      (i32.load
+                       (local.get $0)
+                      )
+                     )
+                     (i32.const 32)
+                    )
+                   )
+                   (block
+                    (drop
+                     (call $21
+                      (local.get $26)
+                      (local.get $24)
+                      (local.get $0)
+                     )
+                    )
+                    (local.set $1
+                     (i32.load
+                      (local.get $0)
+                     )
+                    )
+                   )
+                  )
+                  (local.set $7
+                   (if (result i32)
+                    (local.tee $5
+                     (i32.ne
+                      (i32.and
+                       (local.get $9)
+                       (i32.const 32)
+                      )
+                      (i32.const 0)
+                     )
+                    )
+                    (i32.const 2215)
+                    (i32.const 2219)
+                   )
+                  )
+                  (local.set $5
+                   (if (result i32)
+                    (local.get $5)
+                    (i32.const 2223)
+                    (i32.const 2227)
+                   )
+                  )
+                  (if
+                   (i32.eqz
+                    (local.get $6)
+                   )
+                   (local.set $5
+                    (local.get $7)
+                   )
+                  )
+                  (if
+                   (i32.eqz
+                    (i32.and
+                     (local.get $1)
+                     (i32.const 32)
+                    )
+                   )
+                   (drop
+                    (call $21
+                     (local.get $5)
+                     (i32.const 3)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.get $8)
+                   (i32.xor
+                    (local.get $12)
+                    (i32.const 8192)
+                   )
+                  )
+                  (if
+                   (i32.ge_s
+                    (local.get $8)
+                    (local.get $10)
+                   )
+                   (local.set $10
+                    (local.get $8)
+                   )
+                  )
+                 )
+                )
+               )
+               (local.set $1
+                (local.get $11)
+               )
+               (br $label$4)
+              )
+              (local.set $7
+               (local.get $5)
+              )
+              (local.set $6
+               (i32.const 0)
+              )
+              (local.set $8
+               (i32.const 2179)
+              )
+              (local.set $5
+               (local.get $21)
+              )
+              (br $label$70)
+             )
+             (local.set $7
+              (i32.and
+               (local.get $9)
+               (i32.const 32)
+              )
+             )
+             (local.set $7
+              (if (result i32)
+               (i64.eq
+                (local.tee $50
+                 (i64.load
+                  (local.get $16)
+                 )
+                )
+                (i64.const 0)
+               )
+               (block (result i32)
+                (local.set $50
+                 (i64.const 0)
+                )
+                (local.get $21)
+               )
+               (block (result i32)
+                (local.set $1
+                 (local.get $21)
+                )
+                (loop $label$280
+                 (i32.store8
+                  (local.tee $1
+                   (i32.add
+                    (local.get $1)
+                    (i32.const -1)
+                   )
+                  )
+                  (i32.or
+                   (i32.load8_u
+                    (i32.add
+                     (i32.and
+                      (i32.wrap_i64
+                       (local.get $50)
+                      )
+                      (i32.const 15)
+                     )
+                     (i32.const 2163)
+                    )
+                   )
+                   (local.get $7)
+                  )
+                 )
+                 (br_if $label$280
+                  (i64.ne
+                   (local.tee $50
+                    (i64.shr_u
+                     (local.get $50)
+                     (i64.const 4)
+                    )
+                   )
+                   (i64.const 0)
+                  )
+                 )
+                )
+                (local.set $50
+                 (i64.load
+                  (local.get $16)
+                 )
+                )
+                (local.get $1)
+               )
+              )
+             )
+             (local.set $8
+              (i32.add
+               (i32.shr_s
+                (local.get $9)
+                (i32.const 4)
+               )
+               (i32.const 2179)
+              )
+             )
+             (if
+              (local.tee $1
+               (i32.or
+                (i32.eqz
+                 (i32.and
+                  (local.get $12)
+                  (i32.const 8)
+                 )
+                )
+                (i64.eq
+                 (local.get $50)
+                 (i64.const 0)
+                )
+               )
+              )
+              (local.set $8
+               (i32.const 2179)
+              )
+             )
+             (local.set $6
+              (if (result i32)
+               (local.get $1)
+               (i32.const 0)
+               (i32.const 2)
+              )
+             )
+             (br $label$71)
+            )
+            (local.set $7
+             (call $23
+              (local.get $50)
+              (local.get $21)
+             )
+            )
+            (br $label$71)
+           )
+           (local.set $14
+            (i32.eqz
+             (local.tee $13
+              (call $17
+               (local.get $1)
+               (i32.const 0)
+               (local.get $5)
+              )
+             )
+            )
+           )
+           (local.set $8
+            (i32.sub
+             (local.get $13)
+             (local.get $1)
+            )
+           )
+           (local.set $9
+            (i32.add
+             (local.get $1)
+             (local.get $5)
+            )
+           )
+           (local.set $12
+            (local.get $7)
+           )
+           (local.set $7
+            (if (result i32)
+             (local.get $14)
+             (local.get $5)
+             (local.get $8)
+            )
+           )
+           (local.set $6
+            (i32.const 0)
+           )
+           (local.set $8
+            (i32.const 2179)
+           )
+           (local.set $5
+            (if (result i32)
+             (local.get $14)
+             (local.get $9)
+             (local.get $13)
+            )
+           )
+           (br $label$70)
+          )
+          (local.set $1
+           (i32.const 0)
+          )
+          (local.set $5
+           (i32.const 0)
+          )
+          (local.set $8
+           (local.get $7)
+          )
+          (loop $label$288
+           (block $label$289
+            (br_if $label$289
+             (i32.eqz
+              (local.tee $9
+               (i32.load
+                (local.get $8)
+               )
+              )
+             )
+            )
+            (br_if $label$289
+             (i32.or
+              (i32.lt_s
+               (local.tee $5
+                (call $26
+                 (local.get $36)
+                 (local.get $9)
+                )
+               )
+               (i32.const 0)
+              )
+              (i32.gt_u
+               (local.get $5)
+               (i32.sub
+                (local.get $6)
+                (local.get $1)
+               )
+              )
+             )
+            )
+            (local.set $8
+             (i32.add
+              (local.get $8)
+              (i32.const 4)
+             )
+            )
+            (br_if $label$288
+             (i32.gt_u
+              (local.get $6)
+              (local.tee $1
+               (i32.add
+                (local.get $5)
+                (local.get $1)
+               )
+              )
+             )
+            )
+           )
+          )
+          (if
+           (i32.lt_s
+            (local.get $5)
+            (i32.const 0)
+           )
+           (block
+            (local.set $15
+             (i32.const -1)
+            )
+            (br $label$5)
+           )
+          )
+          (call $25
+           (local.get $0)
+           (i32.const 32)
+           (local.get $10)
+           (local.get $1)
+           (local.get $12)
+          )
+          (if
+           (local.get $1)
+           (block
+            (local.set $5
+             (i32.const 0)
+            )
+            (loop $label$292
+             (br_if $label$72
+              (i32.eqz
+               (local.tee $8
+                (i32.load
+                 (local.get $7)
+                )
+               )
+              )
+             )
+             (br_if $label$72
+              (i32.gt_s
+               (local.tee $5
+                (i32.add
+                 (local.tee $8
+                  (call $26
+                   (local.get $36)
+                   (local.get $8)
+                  )
+                 )
+                 (local.get $5)
+                )
+               )
+               (local.get $1)
+              )
+             )
+             (if
+              (i32.eqz
+               (i32.and
+                (i32.load
+                 (local.get $0)
+                )
+                (i32.const 32)
+               )
+              )
+              (drop
+               (call $21
+                (local.get $36)
+                (local.get $8)
+                (local.get $0)
+               )
+              )
+             )
+             (local.set $7
+              (i32.add
+               (local.get $7)
+               (i32.const 4)
+              )
+             )
+             (br_if $label$292
+              (i32.lt_u
+               (local.get $5)
+               (local.get $1)
+              )
+             )
+             (br $label$72)
+            )
+           )
+           (block
+            (local.set $1
+             (i32.const 0)
+            )
+            (br $label$72)
+           )
+          )
+         )
+         (call $25
+          (local.get $0)
+          (i32.const 32)
+          (local.get $10)
+          (local.get $1)
+          (i32.xor
+           (local.get $12)
+           (i32.const 8192)
+          )
+         )
+         (if
+          (i32.le_s
+           (local.get $10)
+           (local.get $1)
+          )
+          (local.set $10
+           (local.get $1)
+          )
+         )
+         (local.set $1
+          (local.get $11)
+         )
+         (br $label$4)
+        )
+        (local.set $1
+         (i32.and
+          (local.get $12)
+          (i32.const -65537)
+         )
+        )
+        (if
+         (i32.gt_s
+          (local.get $5)
+          (i32.const -1)
+         )
+         (local.set $12
+          (local.get $1)
+         )
+        )
+        (local.set $5
+         (if (result i32)
+          (i32.or
+           (local.get $5)
+           (local.tee $9
+            (i64.ne
+             (i64.load
+              (local.get $16)
+             )
+             (i64.const 0)
+            )
+           )
+          )
+          (block (result i32)
+           (local.set $1
+            (local.get $7)
+           )
+           (if
+            (i32.gt_s
+             (local.get $5)
+             (local.tee $7
+              (i32.add
+               (i32.xor
+                (i32.and
+                 (local.get $9)
+                 (i32.const 1)
+                )
+                (i32.const 1)
+               )
+               (i32.sub
+                (local.get $38)
+                (local.get $7)
+               )
+              )
+             )
+            )
+            (local.set $7
+             (local.get $5)
+            )
+           )
+           (local.get $21)
+          )
+          (block (result i32)
+           (local.set $1
+            (local.get $21)
+           )
+           (local.set $7
+            (i32.const 0)
+           )
+           (local.get $21)
+          )
+         )
+        )
+       )
+       (call $25
+        (local.get $0)
+        (i32.const 32)
+        (if (result i32)
+         (i32.lt_s
+          (local.get $10)
+          (local.tee $5
+           (i32.add
+            (if (result i32)
+             (i32.lt_s
+              (local.get $7)
+              (local.tee $9
+               (i32.sub
+                (local.get $5)
+                (local.get $1)
+               )
+              )
+             )
+             (local.tee $7
+              (local.get $9)
+             )
+             (local.get $7)
+            )
+            (local.get $6)
+           )
+          )
+         )
+         (local.tee $10
+          (local.get $5)
+         )
+         (local.get $10)
+        )
+        (local.get $5)
+        (local.get $12)
+       )
+       (if
+        (i32.eqz
+         (i32.and
+          (i32.load
+           (local.get $0)
+          )
+          (i32.const 32)
+         )
+        )
+        (drop
+         (call $21
+          (local.get $8)
+          (local.get $6)
+          (local.get $0)
+         )
+        )
+       )
+       (call $25
+        (local.get $0)
+        (i32.const 48)
+        (local.get $10)
+        (local.get $5)
+        (i32.xor
+         (local.get $12)
+         (i32.const 65536)
+        )
+       )
+       (call $25
+        (local.get $0)
+        (i32.const 48)
+        (local.get $7)
+        (local.get $9)
+        (i32.const 0)
+       )
+       (if
+        (i32.eqz
+         (i32.and
+          (i32.load
+           (local.get $0)
+          )
+          (i32.const 32)
+         )
+        )
+        (drop
+         (call $21
+          (local.get $1)
+          (local.get $9)
+          (local.get $0)
+         )
+        )
+       )
+       (call $25
+        (local.get $0)
+        (i32.const 32)
+        (local.get $10)
+        (local.get $5)
+        (i32.xor
+         (local.get $12)
+         (i32.const 8192)
+        )
+       )
+       (local.set $1
+        (local.get $11)
+       )
+       (br $label$4)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.eqz
+      (local.get $0)
+     )
+     (if
+      (local.get $17)
+      (block
+       (local.set $0
+        (i32.const 1)
+       )
+       (loop $label$308
+        (if
+         (local.tee $1
+          (i32.load
+           (i32.add
+            (local.get $4)
+            (i32.shl
+             (local.get $0)
+             (i32.const 2)
+            )
+           )
+          )
+         )
+         (block
+          (call $22
+           (i32.add
+            (local.get $3)
+            (i32.shl
+             (local.get $0)
+             (i32.const 3)
+            )
+           )
+           (local.get $1)
+           (local.get $2)
+          )
+          (br_if $label$308
+           (i32.lt_s
+            (local.tee $0
+             (i32.add
+              (local.get $0)
+              (i32.const 1)
+             )
+            )
+            (i32.const 10)
+           )
+          )
+          (local.set $15
+           (i32.const 1)
+          )
+          (br $label$2)
+         )
+        )
+       )
+       (loop $label$310
+        (if
+         (i32.load
+          (i32.add
+           (local.get $4)
+           (i32.shl
+            (local.get $0)
+            (i32.const 2)
+           )
+          )
+         )
+         (block
+          (local.set $15
+           (i32.const -1)
+          )
+          (br $label$2)
+         )
+        )
+        (br_if $label$310
+         (i32.lt_s
+          (local.tee $0
+           (i32.add
+            (local.get $0)
+            (i32.const 1)
+           )
+          )
+          (i32.const 10)
+         )
+        )
+        (local.set $15
+         (i32.const 1)
+        )
+       )
+      )
+      (local.set $15
+       (i32.const 0)
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $23)
+   )
+   (local.get $15)
+  )
+ )
+ (func $20 (; 33 ;) (type $2) (param $0 i32) (result i32)
+  (i32.const 0)
+ )
+ (func $21 (; 34 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (block $label$1 (result i32)
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (local.tee $3
+       (i32.load
+        (local.tee $4
+         (i32.add
+          (local.get $2)
+          (i32.const 16)
+         )
+        )
+       )
+      )
+     )
+     (if
+      (call $30
+       (local.get $2)
+      )
+      (local.set $3
+       (i32.const 0)
+      )
+      (block
+       (local.set $3
+        (i32.load
+         (local.get $4)
+        )
+       )
+       (br $label$3)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.lt_u
+      (i32.sub
+       (local.get $3)
+       (local.tee $4
+        (i32.load
+         (local.tee $5
+          (i32.add
+           (local.get $2)
+           (i32.const 20)
+          )
+         )
+        )
+       )
+      )
+      (local.get $1)
+     )
+     (block
+      (local.set $3
+       (call_indirect (type $0)
+        (local.get $2)
+        (local.get $0)
+        (local.get $1)
+        (i32.add
+         (i32.and
+          (i32.load offset=36
+           (local.get $2)
+          )
+          (i32.const 3)
+         )
+         (i32.const 2)
+        )
+       )
+      )
+      (br $label$2)
+     )
+    )
+    (local.set $2
+     (block $label$7 (result i32)
+      (if (result i32)
+       (i32.gt_s
+        (i32.load8_s offset=75
+         (local.get $2)
+        )
+        (i32.const -1)
+       )
+       (block (result i32)
+        (local.set $3
+         (local.get $1)
+        )
+        (loop $label$9
+         (drop
+          (br_if $label$7
+           (i32.const 0)
+           (i32.eqz
+            (local.get $3)
+           )
+          )
+         )
+         (if
+          (i32.ne
+           (i32.load8_s
+            (i32.add
+             (local.get $0)
+             (local.tee $6
+              (i32.add
+               (local.get $3)
+               (i32.const -1)
+              )
+             )
+            )
+           )
+           (i32.const 10)
+          )
+          (block
+           (local.set $3
+            (local.get $6)
+           )
+           (br $label$9)
+          )
+         )
+        )
+        (br_if $label$2
+         (i32.lt_u
+          (call_indirect (type $0)
+           (local.get $2)
+           (local.get $0)
+           (local.get $3)
+           (i32.add
+            (i32.and
+             (i32.load offset=36
+              (local.get $2)
+             )
+             (i32.const 3)
+            )
+            (i32.const 2)
+           )
+          )
+          (local.get $3)
+         )
+        )
+        (local.set $4
+         (i32.load
+          (local.get $5)
+         )
+        )
+        (local.set $1
+         (i32.sub
+          (local.get $1)
+          (local.get $3)
+         )
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (local.get $3)
+         )
+        )
+        (local.get $3)
+       )
+       (i32.const 0)
+      )
+     )
+    )
+    (drop
+     (call $47
+      (local.get $4)
+      (local.get $0)
+      (local.get $1)
+     )
+    )
+    (i32.store
+     (local.get $5)
+     (i32.add
+      (i32.load
+       (local.get $5)
+      )
+      (local.get $1)
+     )
+    )
+    (local.set $3
+     (i32.add
+      (local.get $2)
+      (local.get $1)
+     )
+    )
+   )
+   (local.get $3)
+  )
+ )
+ (func $22 (; 35 ;) (type $8) (param $0 i32) (param $1 i32) (param $2 i32)
+  (local $3 i32)
+  (local $4 i64)
+  (local $5 f64)
+  (block $label$1
+   (if
+    (i32.le_u
+     (local.get $1)
+     (i32.const 20)
+    )
+    (block $label$3
+     (block $label$4
+      (block $label$5
+       (block $label$6
+        (block $label$7
+         (block $label$8
+          (block $label$9
+           (block $label$10
+            (block $label$11
+             (block $label$12
+              (block $label$13
+               (br_table $label$13 $label$12 $label$11 $label$10 $label$9 $label$8 $label$7 $label$6 $label$5 $label$4 $label$3
+                (i32.sub
+                 (local.get $1)
+                 (i32.const 9)
+                )
+               )
+              )
+              (local.set $3
+               (i32.load
+                (local.tee $1
+                 (i32.and
+                  (i32.add
+                   (i32.load
+                    (local.get $2)
+                   )
+                   (i32.const 3)
+                  )
+                  (i32.const -4)
+                 )
+                )
+               )
+              )
+              (i32.store
+               (local.get $2)
+               (i32.add
+                (local.get $1)
+                (i32.const 4)
+               )
+              )
+              (i32.store
+               (local.get $0)
+               (local.get $3)
+              )
+              (br $label$1)
+             )
+             (local.set $3
+              (i32.load
+               (local.tee $1
+                (i32.and
+                 (i32.add
+                  (i32.load
+                   (local.get $2)
+                  )
+                  (i32.const 3)
+                 )
+                 (i32.const -4)
+                )
+               )
+              )
+             )
+             (i32.store
+              (local.get $2)
+              (i32.add
+               (local.get $1)
+               (i32.const 4)
+              )
+             )
+             (i64.store
+              (local.get $0)
+              (i64.extend_i32_s
+               (local.get $3)
+              )
+             )
+             (br $label$1)
+            )
+            (local.set $3
+             (i32.load
+              (local.tee $1
+               (i32.and
+                (i32.add
+                 (i32.load
+                  (local.get $2)
+                 )
+                 (i32.const 3)
+                )
+                (i32.const -4)
+               )
+              )
+             )
+            )
+            (i32.store
+             (local.get $2)
+             (i32.add
+              (local.get $1)
+              (i32.const 4)
+             )
+            )
+            (i64.store
+             (local.get $0)
+             (i64.extend_i32_u
+              (local.get $3)
+             )
+            )
+            (br $label$1)
+           )
+           (local.set $4
+            (i64.load
+             (local.tee $1
+              (i32.and
+               (i32.add
+                (i32.load
+                 (local.get $2)
+                )
+                (i32.const 7)
+               )
+               (i32.const -8)
+              )
+             )
+            )
+           )
+           (i32.store
+            (local.get $2)
+            (i32.add
+             (local.get $1)
+             (i32.const 8)
+            )
+           )
+           (i64.store
+            (local.get $0)
+            (local.get $4)
+           )
+           (br $label$1)
+          )
+          (local.set $3
+           (i32.load
+            (local.tee $1
+             (i32.and
+              (i32.add
+               (i32.load
+                (local.get $2)
+               )
+               (i32.const 3)
+              )
+              (i32.const -4)
+             )
+            )
+           )
+          )
+          (i32.store
+           (local.get $2)
+           (i32.add
+            (local.get $1)
+            (i32.const 4)
+           )
+          )
+          (i64.store
+           (local.get $0)
+           (i64.extend_i32_s
+            (i32.shr_s
+             (i32.shl
+              (i32.and
+               (local.get $3)
+               (i32.const 65535)
+              )
+              (i32.const 16)
+             )
+             (i32.const 16)
+            )
+           )
+          )
+          (br $label$1)
+         )
+         (local.set $3
+          (i32.load
+           (local.tee $1
+            (i32.and
+             (i32.add
+              (i32.load
+               (local.get $2)
+              )
+              (i32.const 3)
+             )
+             (i32.const -4)
+            )
+           )
+          )
+         )
+         (i32.store
+          (local.get $2)
+          (i32.add
+           (local.get $1)
+           (i32.const 4)
+          )
+         )
+         (i64.store
+          (local.get $0)
+          (i64.extend_i32_u
+           (i32.and
+            (local.get $3)
+            (i32.const 65535)
+           )
+          )
+         )
+         (br $label$1)
+        )
+        (local.set $3
+         (i32.load
+          (local.tee $1
+           (i32.and
+            (i32.add
+             (i32.load
+              (local.get $2)
+             )
+             (i32.const 3)
+            )
+            (i32.const -4)
+           )
+          )
+         )
+        )
+        (i32.store
+         (local.get $2)
+         (i32.add
+          (local.get $1)
+          (i32.const 4)
+         )
+        )
+        (i64.store
+         (local.get $0)
+         (i64.extend_i32_s
+          (i32.shr_s
+           (i32.shl
+            (i32.and
+             (local.get $3)
+             (i32.const 255)
+            )
+            (i32.const 24)
+           )
+           (i32.const 24)
+          )
+         )
+        )
+        (br $label$1)
+       )
+       (local.set $3
+        (i32.load
+         (local.tee $1
+          (i32.and
+           (i32.add
+            (i32.load
+             (local.get $2)
+            )
+            (i32.const 3)
+           )
+           (i32.const -4)
+          )
+         )
+        )
+       )
+       (i32.store
+        (local.get $2)
+        (i32.add
+         (local.get $1)
+         (i32.const 4)
+        )
+       )
+       (i64.store
+        (local.get $0)
+        (i64.extend_i32_u
+         (i32.and
+          (local.get $3)
+          (i32.const 255)
+         )
+        )
+       )
+       (br $label$1)
+      )
+      (local.set $5
+       (f64.load
+        (local.tee $1
+         (i32.and
+          (i32.add
+           (i32.load
+            (local.get $2)
+           )
+           (i32.const 7)
+          )
+          (i32.const -8)
+         )
+        )
+       )
+      )
+      (i32.store
+       (local.get $2)
+       (i32.add
+        (local.get $1)
+        (i32.const 8)
+       )
+      )
+      (f64.store
+       (local.get $0)
+       (local.get $5)
+      )
+      (br $label$1)
+     )
+     (local.set $5
+      (f64.load
+       (local.tee $1
+        (i32.and
+         (i32.add
+          (i32.load
+           (local.get $2)
+          )
+          (i32.const 7)
+         )
+         (i32.const -8)
+        )
+       )
+      )
+     )
+     (i32.store
+      (local.get $2)
+      (i32.add
+       (local.get $1)
+       (i32.const 8)
+      )
+     )
+     (f64.store
+      (local.get $0)
+      (local.get $5)
+     )
+    )
+   )
+  )
+ )
+ (func $23 (; 36 ;) (type $9) (param $0 i64) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i64)
+  (block $label$1 (result i32)
+   (local.set $2
+    (i32.wrap_i64
+     (local.get $0)
+    )
+   )
+   (if
+    (i64.gt_u
+     (local.get $0)
+     (i64.const 4294967295)
+    )
+    (block
+     (loop $label$3
+      (i64.store8
+       (local.tee $1
+        (i32.add
+         (local.get $1)
+         (i32.const -1)
+        )
+       )
+       (i64.or
+        (i64.rem_u
+         (local.get $0)
+         (i64.const 10)
+        )
+        (i64.const 48)
+       )
+      )
+      (local.set $4
+       (i64.div_u
+        (local.get $0)
+        (i64.const 10)
+       )
+      )
+      (if
+       (i64.gt_u
+        (local.get $0)
+        (i64.const 42949672959)
+       )
+       (block
+        (local.set $0
+         (local.get $4)
+        )
+        (br $label$3)
+       )
+      )
+     )
+     (local.set $2
+      (i32.wrap_i64
+       (local.get $4)
+      )
+     )
+    )
+   )
+   (if
+    (local.get $2)
+    (loop $label$6
+     (i32.store8
+      (local.tee $1
+       (i32.add
+        (local.get $1)
+        (i32.const -1)
+       )
+      )
+      (i32.or
+       (i32.rem_u
+        (local.get $2)
+        (i32.const 10)
+       )
+       (i32.const 48)
+      )
+     )
+     (local.set $3
+      (i32.div_u
+       (local.get $2)
+       (i32.const 10)
+      )
+     )
+     (if
+      (i32.ge_u
+       (local.get $2)
+       (i32.const 10)
+      )
+      (block
+       (local.set $2
+        (local.get $3)
+       )
+       (br $label$6)
+      )
+     )
+    )
+   )
+   (local.get $1)
+  )
+ )
+ (func $24 (; 37 ;) (type $2) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (i32.const 0)
+   )
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (loop $label$5
+       (br_if $label$4
+        (i32.eq
+         (i32.load8_u
+          (i32.add
+           (local.get $1)
+           (i32.const 2233)
+          )
+         )
+         (local.get $0)
+        )
+       )
+       (br_if $label$5
+        (i32.ne
+         (local.tee $1
+          (i32.add
+           (local.get $1)
+           (i32.const 1)
+          )
+         )
+         (i32.const 87)
+        )
+       )
+       (local.set $1
+        (i32.const 87)
+       )
+       (local.set $0
+        (i32.const 2321)
+       )
+       (br $label$3)
+      )
+     )
+     (if
+      (local.get $1)
+      (block
+       (local.set $0
+        (i32.const 2321)
+       )
+       (br $label$3)
+      )
+      (local.set $0
+       (i32.const 2321)
+      )
+     )
+     (br $label$2)
+    )
+    (loop $label$8
+     (local.set $2
+      (local.get $0)
+     )
+     (loop $label$9
+      (local.set $0
+       (i32.add
+        (local.get $2)
+        (i32.const 1)
+       )
+      )
+      (if
+       (i32.load8_s
+        (local.get $2)
+       )
+       (block
+        (local.set $2
+         (local.get $0)
+        )
+        (br $label$9)
+       )
+      )
+     )
+     (br_if $label$8
+      (local.tee $1
+       (i32.add
+        (local.get $1)
+        (i32.const -1)
+       )
+      )
+     )
+    )
+   )
+   (local.get $0)
+  )
+ )
+ (func $25 (; 38 ;) (type $10) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (param $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (block $label$1
+   (local.set $7
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 256)
+    )
+   )
+   (local.set $6
+    (local.get $7)
+   )
+   (block $label$2
+    (if
+     (i32.and
+      (i32.gt_s
+       (local.get $2)
+       (local.get $3)
+      )
+      (i32.eqz
+       (i32.and
+        (local.get $4)
+        (i32.const 73728)
+       )
+      )
+     )
+     (block
+      (drop
+       (call $46
+        (local.get $6)
+        (local.get $1)
+        (if (result i32)
+         (i32.gt_u
+          (local.tee $5
+           (i32.sub
+            (local.get $2)
+            (local.get $3)
+           )
+          )
+          (i32.const 256)
+         )
+         (i32.const 256)
+         (local.get $5)
+        )
+       )
+      )
+      (local.set $4
+       (i32.eqz
+        (i32.and
+         (local.tee $1
+          (i32.load
+           (local.get $0)
+          )
+         )
+         (i32.const 32)
+        )
+       )
+      )
+      (if
+       (i32.gt_u
+        (local.get $5)
+        (i32.const 255)
+       )
+       (block
+        (loop $label$7
+         (if
+          (local.get $4)
+          (block
+           (drop
+            (call $21
+             (local.get $6)
+             (i32.const 256)
+             (local.get $0)
+            )
+           )
+           (local.set $1
+            (i32.load
+             (local.get $0)
+            )
+           )
+          )
+         )
+         (local.set $4
+          (i32.eqz
+           (i32.and
+            (local.get $1)
+            (i32.const 32)
+           )
+          )
+         )
+         (br_if $label$7
+          (i32.gt_u
+           (local.tee $5
+            (i32.add
+             (local.get $5)
+             (i32.const -256)
+            )
+           )
+           (i32.const 255)
+          )
+         )
+        )
+        (br_if $label$2
+         (i32.eqz
+          (local.get $4)
+         )
+        )
+        (local.set $5
+         (i32.and
+          (i32.sub
+           (local.get $2)
+           (local.get $3)
+          )
+          (i32.const 255)
+         )
+        )
+       )
+       (br_if $label$2
+        (i32.eqz
+         (local.get $4)
+        )
+       )
+      )
+      (drop
+       (call $21
+        (local.get $6)
+        (local.get $5)
+        (local.get $0)
+       )
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $7)
+   )
+  )
+ )
+ (func $26 (; 39 ;) (type $6) (param $0 i32) (param $1 i32) (result i32)
+  (if (result i32)
+   (local.get $0)
+   (call $29
+    (local.get $0)
+    (local.get $1)
+    (i32.const 0)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $27 (; 40 ;) (type $11) (param $0 f64) (param $1 i32) (result f64)
+  (call $28
+   (local.get $0)
+   (local.get $1)
+  )
+ )
+ (func $28 (; 41 ;) (type $11) (param $0 f64) (param $1 i32) (result f64)
+  (local $2 i64)
+  (local $3 i64)
+  (block $label$1 (result f64)
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (block $label$5
+       (br_table $label$5 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$4 $label$3
+        (i32.sub
+         (i32.shr_s
+          (i32.shl
+           (i32.and
+            (i32.and
+             (i32.wrap_i64
+              (local.tee $3
+               (i64.shr_u
+                (local.tee $2
+                 (i64.reinterpret_f64
+                  (local.get $0)
+                 )
+                )
+                (i64.const 52)
+               )
+              )
+             )
+             (i32.const 65535)
+            )
+            (i32.const 2047)
+           )
+           (i32.const 16)
+          )
+          (i32.const 16)
+         )
+         (i32.const 0)
+        )
+       )
+      )
+      (i32.store
+       (local.get $1)
+       (if (result i32)
+        (f64.ne
+         (local.get $0)
+         (f64.const 0)
+        )
+        (block (result i32)
+         (local.set $0
+          (call $28
+           (f64.mul
+            (local.get $0)
+            (f64.const 18446744073709551615)
+           )
+           (local.get $1)
+          )
+         )
+         (i32.add
+          (i32.load
+           (local.get $1)
+          )
+          (i32.const -64)
+         )
+        )
+        (i32.const 0)
+       )
+      )
+      (br $label$2)
+     )
+     (br $label$2)
+    )
+    (i32.store
+     (local.get $1)
+     (i32.add
+      (i32.and
+       (i32.wrap_i64
+        (local.get $3)
+       )
+       (i32.const 2047)
+      )
+      (i32.const -1022)
+     )
+    )
+    (local.set $0
+     (f64.reinterpret_i64
+      (i64.or
+       (i64.and
+        (local.get $2)
+        (i64.const -9218868437227405313)
+       )
+       (i64.const 4602678819172646912)
+      )
+     )
+    )
+   )
+   (local.get $0)
+  )
+ )
+ (func $29 (; 42 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (block $label$1 (result i32)
+   (if (result i32)
+    (local.get $0)
+    (block (result i32)
+     (if
+      (i32.lt_u
+       (local.get $1)
+       (i32.const 128)
+      )
+      (block
+       (i32.store8
+        (local.get $0)
+        (local.get $1)
+       )
+       (br $label$1
+        (i32.const 1)
+       )
+      )
+     )
+     (if
+      (i32.lt_u
+       (local.get $1)
+       (i32.const 2048)
+      )
+      (block
+       (i32.store8
+        (local.get $0)
+        (i32.or
+         (i32.shr_u
+          (local.get $1)
+          (i32.const 6)
+         )
+         (i32.const 192)
+        )
+       )
+       (i32.store8 offset=1
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (local.get $1)
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (br $label$1
+        (i32.const 2)
+       )
+      )
+     )
+     (if
+      (i32.or
+       (i32.lt_u
+        (local.get $1)
+        (i32.const 55296)
+       )
+       (i32.eq
+        (i32.and
+         (local.get $1)
+         (i32.const -8192)
+        )
+        (i32.const 57344)
+       )
+      )
+      (block
+       (i32.store8
+        (local.get $0)
+        (i32.or
+         (i32.shr_u
+          (local.get $1)
+          (i32.const 12)
+         )
+         (i32.const 224)
+        )
+       )
+       (i32.store8 offset=1
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (i32.shr_u
+           (local.get $1)
+           (i32.const 6)
+          )
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.store8 offset=2
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (local.get $1)
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (br $label$1
+        (i32.const 3)
+       )
+      )
+     )
+     (if (result i32)
+      (i32.lt_u
+       (i32.add
+        (local.get $1)
+        (i32.const -65536)
+       )
+       (i32.const 1048576)
+      )
+      (block (result i32)
+       (i32.store8
+        (local.get $0)
+        (i32.or
+         (i32.shr_u
+          (local.get $1)
+          (i32.const 18)
+         )
+         (i32.const 240)
+        )
+       )
+       (i32.store8 offset=1
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (i32.shr_u
+           (local.get $1)
+           (i32.const 12)
+          )
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.store8 offset=2
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (i32.shr_u
+           (local.get $1)
+           (i32.const 6)
+          )
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.store8 offset=3
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (local.get $1)
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.const 4)
+      )
+      (block (result i32)
+       (i32.store
+        (call $12)
+        (i32.const 84)
+       )
+       (i32.const -1)
+      )
+     )
+    )
+    (i32.const 1)
+   )
+  )
+ )
+ (func $30 (; 43 ;) (type $2) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (i32.load8_s
+     (local.tee $2
+      (i32.add
+       (local.get $0)
+       (i32.const 74)
+      )
+     )
+    )
+   )
+   (i32.store8
+    (local.get $2)
+    (i32.or
+     (i32.add
+      (local.get $1)
+      (i32.const 255)
+     )
+     (local.get $1)
+    )
+   )
+   (local.tee $0
+    (if (result i32)
+     (i32.and
+      (local.tee $1
+       (i32.load
+        (local.get $0)
+       )
+      )
+      (i32.const 8)
+     )
+     (block (result i32)
+      (i32.store
+       (local.get $0)
+       (i32.or
+        (local.get $1)
+        (i32.const 32)
+       )
+      )
+      (i32.const -1)
+     )
+     (block (result i32)
+      (i32.store offset=8
+       (local.get $0)
+       (i32.const 0)
+      )
+      (i32.store offset=4
+       (local.get $0)
+       (i32.const 0)
+      )
+      (i32.store offset=28
+       (local.get $0)
+       (local.tee $1
+        (i32.load offset=44
+         (local.get $0)
+        )
+       )
+      )
+      (i32.store offset=20
+       (local.get $0)
+       (local.get $1)
+      )
+      (i32.store offset=16
+       (local.get $0)
+       (i32.add
+        (local.get $1)
+        (i32.load offset=48
+         (local.get $0)
+        )
+       )
+      )
+      (i32.const 0)
+     )
+    )
+   )
+  )
+ )
+ (func $31 (; 44 ;) (type $2) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (block $label$1 (result i32)
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (i32.eqz
+       (i32.and
+        (local.tee $2
+         (local.get $0)
+        )
+        (i32.const 3)
+       )
+      )
+     )
+     (local.set $1
+      (local.get $2)
+     )
+     (loop $label$4
+      (if
+       (i32.eqz
+        (i32.load8_s
+         (local.get $0)
+        )
+       )
+       (block
+        (local.set $0
+         (local.get $1)
+        )
+        (br $label$2)
+       )
+      )
+      (br_if $label$4
+       (i32.and
+        (local.tee $1
+         (local.tee $0
+          (i32.add
+           (local.get $0)
+           (i32.const 1)
+          )
+         )
+        )
+        (i32.const 3)
+       )
+      )
+      (br $label$3)
+     )
+    )
+    (loop $label$6
+     (local.set $1
+      (i32.add
+       (local.get $0)
+       (i32.const 4)
+      )
+     )
+     (if
+      (i32.eqz
+       (i32.and
+        (i32.xor
+         (i32.and
+          (local.tee $3
+           (i32.load
+            (local.get $0)
+           )
+          )
+          (i32.const -2139062144)
+         )
+         (i32.const -2139062144)
+        )
+        (i32.add
+         (local.get $3)
+         (i32.const -16843009)
+        )
+       )
+      )
+      (block
+       (local.set $0
+        (local.get $1)
+       )
+       (br $label$6)
+      )
+     )
+    )
+    (if
+     (i32.shr_s
+      (i32.shl
+       (i32.and
+        (local.get $3)
+        (i32.const 255)
+       )
+       (i32.const 24)
+      )
+      (i32.const 24)
+     )
+     (loop $label$9
+      (br_if $label$9
+       (i32.load8_s
+        (local.tee $0
+         (i32.add
+          (local.get $0)
+          (i32.const 1)
+         )
+        )
+       )
+      )
+     )
+    )
+   )
+   (i32.sub
+    (local.get $0)
+    (local.get $2)
+   )
+  )
+ )
+ (func $32 (; 45 ;) (type $6) (param $0 i32) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (block $label$1 (result i32)
+   (local.set $3
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (i32.store8
+    (local.tee $4
+     (local.get $3)
+    )
+    (local.tee $7
+     (i32.and
+      (local.get $1)
+      (i32.const 255)
+     )
+    )
+   )
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (local.tee $5
+       (i32.load
+        (local.tee $2
+         (i32.add
+          (local.get $0)
+          (i32.const 16)
+         )
+        )
+       )
+      )
+     )
+     (if
+      (call $30
+       (local.get $0)
+      )
+      (local.set $1
+       (i32.const -1)
+      )
+      (block
+       (local.set $5
+        (i32.load
+         (local.get $2)
+        )
+       )
+       (br $label$3)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.lt_u
+      (local.tee $6
+       (i32.load
+        (local.tee $2
+         (i32.add
+          (local.get $0)
+          (i32.const 20)
+         )
+        )
+       )
+      )
+      (local.get $5)
+     )
+     (if
+      (i32.ne
+       (local.tee $1
+        (i32.and
+         (local.get $1)
+         (i32.const 255)
+        )
+       )
+       (i32.load8_s offset=75
+        (local.get $0)
+       )
+      )
+      (block
+       (i32.store
+        (local.get $2)
+        (i32.add
+         (local.get $6)
+         (i32.const 1)
+        )
+       )
+       (i32.store8
+        (local.get $6)
+        (local.get $7)
+       )
+       (br $label$2)
+      )
+     )
+    )
+    (local.set $1
+     (if (result i32)
+      (i32.eq
+       (call_indirect (type $0)
+        (local.get $0)
+        (local.get $4)
+        (i32.const 1)
+        (i32.add
+         (i32.and
+          (i32.load offset=36
+           (local.get $0)
+          )
+          (i32.const 3)
+         )
+         (i32.const 2)
+        )
+       )
+       (i32.const 1)
+      )
+      (i32.load8_u
+       (local.get $4)
+      )
+      (i32.const -1)
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $3)
+   )
+   (local.get $1)
+  )
+ )
+ (func $33 (; 46 ;) (type $12) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (result i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (i32.mul
+     (local.get $2)
+     (local.get $1)
+    )
+   )
+   (if
+    (i32.gt_s
+     (i32.load offset=76
+      (local.get $3)
+     )
+     (i32.const -1)
+    )
+    (block
+     (local.set $5
+      (i32.eqz
+       (call $20
+        (local.get $3)
+       )
+      )
+     )
+     (local.set $0
+      (call $21
+       (local.get $0)
+       (local.get $4)
+       (local.get $3)
+      )
+     )
+     (if
+      (i32.eqz
+       (local.get $5)
+      )
+      (call $13
+       (local.get $3)
+      )
+     )
+    )
+    (local.set $0
+     (call $21
+      (local.get $0)
+      (local.get $4)
+      (local.get $3)
+     )
+    )
+   )
+   (if
+    (i32.ne
+     (local.get $0)
+     (local.get $4)
+    )
+    (local.set $2
+     (i32.div_u
+      (local.get $0)
+      (local.get $1)
+     )
+    )
+   )
+   (local.get $2)
+  )
+ )
+ (func $34 (; 47 ;) (type $6) (param $0 i32) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (block $label$1 (result i32)
+   (local.set $2
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (i32.store
+    (local.tee $3
+     (local.get $2)
+    )
+    (local.get $1)
+   )
+   (local.set $0
+    (call $18
+     (i32.load
+      (i32.const 1280)
+     )
+     (local.get $0)
+     (local.get $3)
+    )
+   )
+   (global.set $global$1
+    (local.get $2)
+   )
+   (local.get $0)
+  )
+ )
+ (func $35 (; 48 ;) (type $2) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (block $label$1 (result i32)
+   (local.set $2
+    (if (result i32)
+     (i32.gt_s
+      (i32.load offset=76
+       (local.tee $1
+        (i32.load
+         (i32.const 1280)
+        )
+       )
+      )
+      (i32.const -1)
+     )
+     (call $20
+      (local.get $1)
+     )
+     (i32.const 0)
+    )
+   )
+   (local.set $0
+    (block $label$4 (result i32)
+     (if (result i32)
+      (i32.lt_s
+       (call $36
+        (local.get $0)
+        (local.get $1)
+       )
+       (i32.const 0)
+      )
+      (i32.const 1)
+      (block (result i32)
+       (if
+        (i32.ne
+         (i32.load8_s offset=75
+          (local.get $1)
+         )
+         (i32.const 10)
+        )
+        (if
+         (i32.lt_u
+          (local.tee $0
+           (i32.load
+            (local.tee $3
+             (i32.add
+              (local.get $1)
+              (i32.const 20)
+             )
+            )
+           )
+          )
+          (i32.load offset=16
+           (local.get $1)
+          )
+         )
+         (block
+          (i32.store
+           (local.get $3)
+           (i32.add
+            (local.get $0)
+            (i32.const 1)
+           )
+          )
+          (i32.store8
+           (local.get $0)
+           (i32.const 10)
+          )
+          (br $label$4
+           (i32.const 0)
+          )
+         )
+        )
+       )
+       (i32.lt_s
+        (call $32
+         (local.get $1)
+         (i32.const 10)
+        )
+        (i32.const 0)
+       )
+      )
+     )
+    )
+   )
+   (if
+    (local.get $2)
+    (call $13
+     (local.get $1)
+    )
+   )
+   (i32.shr_s
+    (i32.shl
+     (local.get $0)
+     (i32.const 31)
+    )
+    (i32.const 31)
+   )
+  )
+ )
+ (func $36 (; 49 ;) (type $6) (param $0 i32) (param $1 i32) (result i32)
+  (i32.add
+   (call $33
+    (local.get $0)
+    (call $31
+     (local.get $0)
+    )
+    (i32.const 1)
+    (local.get $1)
+   )
+   (i32.const -1)
+  )
+ )
+ (func $37 (; 50 ;) (type $2) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (local $16 i32)
+  (local $17 i32)
+  (local $18 i32)
+  (local $19 i32)
+  (local $20 i32)
+  (local $21 i32)
+  (block $label$1 (result i32)
+   (local.set $14
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (local.set $18
+    (local.get $14)
+   )
+   (block $label$2
+    (if
+     (i32.lt_u
+      (local.get $0)
+      (i32.const 245)
+     )
+     (block
+      (local.set $3
+       (i32.and
+        (i32.add
+         (local.get $0)
+         (i32.const 11)
+        )
+        (i32.const -8)
+       )
+      )
+      (if
+       (i32.and
+        (local.tee $0
+         (i32.shr_u
+          (local.tee $8
+           (i32.load
+            (i32.const 4176)
+           )
+          )
+          (local.tee $2
+           (i32.shr_u
+            (if (result i32)
+             (i32.lt_u
+              (local.get $0)
+              (i32.const 11)
+             )
+             (local.tee $3
+              (i32.const 16)
+             )
+             (local.get $3)
+            )
+            (i32.const 3)
+           )
+          )
+         )
+        )
+        (i32.const 3)
+       )
+       (block
+        (local.set $4
+         (i32.load
+          (local.tee $1
+           (i32.add
+            (local.tee $7
+             (i32.load
+              (local.tee $3
+               (i32.add
+                (local.tee $2
+                 (i32.add
+                  (i32.shl
+                   (i32.shl
+                    (local.tee $5
+                     (i32.add
+                      (i32.xor
+                       (i32.and
+                        (local.get $0)
+                        (i32.const 1)
+                       )
+                       (i32.const 1)
+                      )
+                      (local.get $2)
+                     )
+                    )
+                    (i32.const 1)
+                   )
+                   (i32.const 2)
+                  )
+                  (i32.const 4216)
+                 )
+                )
+                (i32.const 8)
+               )
+              )
+             )
+            )
+            (i32.const 8)
+           )
+          )
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $2)
+          (local.get $4)
+         )
+         (i32.store
+          (i32.const 4176)
+          (i32.and
+           (local.get $8)
+           (i32.xor
+            (i32.shl
+             (i32.const 1)
+             (local.get $5)
+            )
+            (i32.const -1)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $4)
+            (i32.load
+             (i32.const 4192)
+            )
+           )
+           (call $fimport$8)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $0
+              (i32.add
+               (local.get $4)
+               (i32.const 12)
+              )
+             )
+            )
+            (local.get $7)
+           )
+           (block
+            (i32.store
+             (local.get $0)
+             (local.get $2)
+            )
+            (i32.store
+             (local.get $3)
+             (local.get $4)
+            )
+           )
+           (call $fimport$8)
+          )
+         )
+        )
+        (i32.store offset=4
+         (local.get $7)
+         (i32.or
+          (local.tee $0
+           (i32.shl
+            (local.get $5)
+            (i32.const 3)
+           )
+          )
+          (i32.const 3)
+         )
+        )
+        (i32.store
+         (local.tee $0
+          (i32.add
+           (i32.add
+            (local.get $7)
+            (local.get $0)
+           )
+           (i32.const 4)
+          )
+         )
+         (i32.or
+          (i32.load
+           (local.get $0)
+          )
+          (i32.const 1)
+         )
+        )
+        (global.set $global$1
+         (local.get $14)
+        )
+        (return
+         (local.get $1)
+        )
+       )
+      )
+      (if
+       (i32.gt_u
+        (local.get $3)
+        (local.tee $16
+         (i32.load
+          (i32.const 4184)
+         )
+        )
+       )
+       (block
+        (if
+         (local.get $0)
+         (block
+          (local.set $5
+           (i32.and
+            (i32.shr_u
+             (local.tee $0
+              (i32.add
+               (i32.and
+                (local.tee $0
+                 (i32.and
+                  (i32.shl
+                   (local.get $0)
+                   (local.get $2)
+                  )
+                  (i32.or
+                   (local.tee $0
+                    (i32.shl
+                     (i32.const 2)
+                     (local.get $2)
+                    )
+                   )
+                   (i32.sub
+                    (i32.const 0)
+                    (local.get $0)
+                   )
+                  )
+                 )
+                )
+                (i32.sub
+                 (i32.const 0)
+                 (local.get $0)
+                )
+               )
+               (i32.const -1)
+              )
+             )
+             (i32.const 12)
+            )
+            (i32.const 16)
+           )
+          )
+          (local.set $12
+           (i32.load
+            (local.tee $5
+             (i32.add
+              (local.tee $9
+               (i32.load
+                (local.tee $2
+                 (i32.add
+                  (local.tee $4
+                   (i32.add
+                    (i32.shl
+                     (i32.shl
+                      (local.tee $11
+                       (i32.add
+                        (i32.or
+                         (i32.or
+                          (i32.or
+                           (i32.or
+                            (local.tee $0
+                             (i32.and
+                              (i32.shr_u
+                               (local.tee $2
+                                (i32.shr_u
+                                 (local.get $0)
+                                 (local.get $5)
+                                )
+                               )
+                               (i32.const 5)
+                              )
+                              (i32.const 8)
+                             )
+                            )
+                            (local.get $5)
+                           )
+                           (local.tee $0
+                            (i32.and
+                             (i32.shr_u
+                              (local.tee $2
+                               (i32.shr_u
+                                (local.get $2)
+                                (local.get $0)
+                               )
+                              )
+                              (i32.const 2)
+                             )
+                             (i32.const 4)
+                            )
+                           )
+                          )
+                          (local.tee $0
+                           (i32.and
+                            (i32.shr_u
+                             (local.tee $2
+                              (i32.shr_u
+                               (local.get $2)
+                               (local.get $0)
+                              )
+                             )
+                             (i32.const 1)
+                            )
+                            (i32.const 2)
+                           )
+                          )
+                         )
+                         (local.tee $0
+                          (i32.and
+                           (i32.shr_u
+                            (local.tee $2
+                             (i32.shr_u
+                              (local.get $2)
+                              (local.get $0)
+                             )
+                            )
+                            (i32.const 1)
+                           )
+                           (i32.const 1)
+                          )
+                         )
+                        )
+                        (i32.shr_u
+                         (local.get $2)
+                         (local.get $0)
+                        )
+                       )
+                      )
+                      (i32.const 1)
+                     )
+                     (i32.const 2)
+                    )
+                    (i32.const 4216)
+                   )
+                  )
+                  (i32.const 8)
+                 )
+                )
+               )
+              )
+              (i32.const 8)
+             )
+            )
+           )
+          )
+          (if
+           (i32.eq
+            (local.get $4)
+            (local.get $12)
+           )
+           (i32.store
+            (i32.const 4176)
+            (local.tee $7
+             (i32.and
+              (local.get $8)
+              (i32.xor
+               (i32.shl
+                (i32.const 1)
+                (local.get $11)
+               )
+               (i32.const -1)
+              )
+             )
+            )
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.get $12)
+              (i32.load
+               (i32.const 4192)
+              )
+             )
+             (call $fimport$8)
+            )
+            (if
+             (i32.eq
+              (i32.load
+               (local.tee $0
+                (i32.add
+                 (local.get $12)
+                 (i32.const 12)
+                )
+               )
+              )
+              (local.get $9)
+             )
+             (block
+              (i32.store
+               (local.get $0)
+               (local.get $4)
+              )
+              (i32.store
+               (local.get $2)
+               (local.get $12)
+              )
+              (local.set $7
+               (local.get $8)
+              )
+             )
+             (call $fimport$8)
+            )
+           )
+          )
+          (i32.store offset=4
+           (local.get $9)
+           (i32.or
+            (local.get $3)
+            (i32.const 3)
+           )
+          )
+          (i32.store offset=4
+           (local.tee $4
+            (i32.add
+             (local.get $9)
+             (local.get $3)
+            )
+           )
+           (i32.or
+            (local.tee $11
+             (i32.sub
+              (i32.shl
+               (local.get $11)
+               (i32.const 3)
+              )
+              (local.get $3)
+             )
+            )
+            (i32.const 1)
+           )
+          )
+          (i32.store
+           (i32.add
+            (local.get $4)
+            (local.get $11)
+           )
+           (local.get $11)
+          )
+          (if
+           (local.get $16)
+           (block
+            (local.set $9
+             (i32.load
+              (i32.const 4196)
+             )
+            )
+            (local.set $2
+             (i32.add
+              (i32.shl
+               (i32.shl
+                (local.tee $0
+                 (i32.shr_u
+                  (local.get $16)
+                  (i32.const 3)
+                 )
+                )
+                (i32.const 1)
+               )
+               (i32.const 2)
+              )
+              (i32.const 4216)
+             )
+            )
+            (if
+             (i32.and
+              (local.get $7)
+              (local.tee $0
+               (i32.shl
+                (i32.const 1)
+                (local.get $0)
+               )
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.tee $0
+                (i32.load
+                 (local.tee $3
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 8)
+                  )
+                 )
+                )
+               )
+               (i32.load
+                (i32.const 4192)
+               )
+              )
+              (call $fimport$8)
+              (block
+               (local.set $6
+                (local.get $3)
+               )
+               (local.set $1
+                (local.get $0)
+               )
+              )
+             )
+             (block
+              (i32.store
+               (i32.const 4176)
+               (i32.or
+                (local.get $7)
+                (local.get $0)
+               )
+              )
+              (local.set $6
+               (i32.add
+                (local.get $2)
+                (i32.const 8)
+               )
+              )
+              (local.set $1
+               (local.get $2)
+              )
+             )
+            )
+            (i32.store
+             (local.get $6)
+             (local.get $9)
+            )
+            (i32.store offset=12
+             (local.get $1)
+             (local.get $9)
+            )
+            (i32.store offset=8
+             (local.get $9)
+             (local.get $1)
+            )
+            (i32.store offset=12
+             (local.get $9)
+             (local.get $2)
+            )
+           )
+          )
+          (i32.store
+           (i32.const 4184)
+           (local.get $11)
+          )
+          (i32.store
+           (i32.const 4196)
+           (local.get $4)
+          )
+          (global.set $global$1
+           (local.get $14)
+          )
+          (return
+           (local.get $5)
+          )
+         )
+        )
+        (if
+         (local.tee $6
+          (i32.load
+           (i32.const 4180)
+          )
+         )
+         (block
+          (local.set $2
+           (i32.and
+            (i32.shr_u
+             (local.tee $0
+              (i32.add
+               (i32.and
+                (local.get $6)
+                (i32.sub
+                 (i32.const 0)
+                 (local.get $6)
+                )
+               )
+               (i32.const -1)
+              )
+             )
+             (i32.const 12)
+            )
+            (i32.const 16)
+           )
+          )
+          (local.set $9
+           (i32.sub
+            (i32.and
+             (i32.load offset=4
+              (local.tee $2
+               (i32.load
+                (i32.add
+                 (i32.shl
+                  (i32.add
+                   (i32.or
+                    (i32.or
+                     (i32.or
+                      (i32.or
+                       (local.tee $0
+                        (i32.and
+                         (i32.shr_u
+                          (local.tee $1
+                           (i32.shr_u
+                            (local.get $0)
+                            (local.get $2)
+                           )
+                          )
+                          (i32.const 5)
+                         )
+                         (i32.const 8)
+                        )
+                       )
+                       (local.get $2)
+                      )
+                      (local.tee $0
+                       (i32.and
+                        (i32.shr_u
+                         (local.tee $1
+                          (i32.shr_u
+                           (local.get $1)
+                           (local.get $0)
+                          )
+                         )
+                         (i32.const 2)
+                        )
+                        (i32.const 4)
+                       )
+                      )
+                     )
+                     (local.tee $0
+                      (i32.and
+                       (i32.shr_u
+                        (local.tee $1
+                         (i32.shr_u
+                          (local.get $1)
+                          (local.get $0)
+                         )
+                        )
+                        (i32.const 1)
+                       )
+                       (i32.const 2)
+                      )
+                     )
+                    )
+                    (local.tee $0
+                     (i32.and
+                      (i32.shr_u
+                       (local.tee $1
+                        (i32.shr_u
+                         (local.get $1)
+                         (local.get $0)
+                        )
+                       )
+                       (i32.const 1)
+                      )
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                   (i32.shr_u
+                    (local.get $1)
+                    (local.get $0)
+                   )
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 4480)
+                )
+               )
+              )
+             )
+             (i32.const -8)
+            )
+            (local.get $3)
+           )
+          )
+          (local.set $1
+           (local.get $2)
+          )
+          (loop $label$25
+           (block $label$26
+            (if
+             (i32.eqz
+              (local.tee $0
+               (i32.load offset=16
+                (local.get $1)
+               )
+              )
+             )
+             (br_if $label$26
+              (i32.eqz
+               (local.tee $0
+                (i32.load offset=20
+                 (local.get $1)
+                )
+               )
+              )
+             )
+            )
+            (if
+             (local.tee $7
+              (i32.lt_u
+               (local.tee $1
+                (i32.sub
+                 (i32.and
+                  (i32.load offset=4
+                   (local.get $0)
+                  )
+                  (i32.const -8)
+                 )
+                 (local.get $3)
+                )
+               )
+               (local.get $9)
+              )
+             )
+             (local.set $9
+              (local.get $1)
+             )
+            )
+            (local.set $1
+             (local.get $0)
+            )
+            (if
+             (local.get $7)
+             (local.set $2
+              (local.get $0)
+             )
+            )
+            (br $label$25)
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.get $2)
+            (local.tee $12
+             (i32.load
+              (i32.const 4192)
+             )
+            )
+           )
+           (call $fimport$8)
+          )
+          (if
+           (i32.ge_u
+            (local.get $2)
+            (local.tee $13
+             (i32.add
+              (local.get $2)
+              (local.get $3)
+             )
+            )
+           )
+           (call $fimport$8)
+          )
+          (local.set $15
+           (i32.load offset=24
+            (local.get $2)
+           )
+          )
+          (block $label$32
+           (if
+            (i32.eq
+             (local.tee $0
+              (i32.load offset=12
+               (local.get $2)
+              )
+             )
+             (local.get $2)
+            )
+            (block
+             (if
+              (i32.eqz
+               (local.tee $0
+                (i32.load
+                 (local.tee $1
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 20)
+                  )
+                 )
+                )
+               )
+              )
+              (if
+               (i32.eqz
+                (local.tee $0
+                 (i32.load
+                  (local.tee $1
+                   (i32.add
+                    (local.get $2)
+                    (i32.const 16)
+                   )
+                  )
+                 )
+                )
+               )
+               (block
+                (local.set $4
+                 (i32.const 0)
+                )
+                (br $label$32)
+               )
+              )
+             )
+             (loop $label$36
+              (if
+               (local.tee $7
+                (i32.load
+                 (local.tee $11
+                  (i32.add
+                   (local.get $0)
+                   (i32.const 20)
+                  )
+                 )
+                )
+               )
+               (block
+                (local.set $0
+                 (local.get $7)
+                )
+                (local.set $1
+                 (local.get $11)
+                )
+                (br $label$36)
+               )
+              )
+              (if
+               (local.tee $7
+                (i32.load
+                 (local.tee $11
+                  (i32.add
+                   (local.get $0)
+                   (i32.const 16)
+                  )
+                 )
+                )
+               )
+               (block
+                (local.set $0
+                 (local.get $7)
+                )
+                (local.set $1
+                 (local.get $11)
+                )
+                (br $label$36)
+               )
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.get $1)
+               (local.get $12)
+              )
+              (call $fimport$8)
+              (block
+               (i32.store
+                (local.get $1)
+                (i32.const 0)
+               )
+               (local.set $4
+                (local.get $0)
+               )
+              )
+             )
+            )
+            (block
+             (if
+              (i32.lt_u
+               (local.tee $11
+                (i32.load offset=8
+                 (local.get $2)
+                )
+               )
+               (local.get $12)
+              )
+              (call $fimport$8)
+             )
+             (if
+              (i32.ne
+               (i32.load
+                (local.tee $7
+                 (i32.add
+                  (local.get $11)
+                  (i32.const 12)
+                 )
+                )
+               )
+               (local.get $2)
+              )
+              (call $fimport$8)
+             )
+             (if
+              (i32.eq
+               (i32.load
+                (local.tee $1
+                 (i32.add
+                  (local.get $0)
+                  (i32.const 8)
+                 )
+                )
+               )
+               (local.get $2)
+              )
+              (block
+               (i32.store
+                (local.get $7)
+                (local.get $0)
+               )
+               (i32.store
+                (local.get $1)
+                (local.get $11)
+               )
+               (local.set $4
+                (local.get $0)
+               )
+              )
+              (call $fimport$8)
+             )
+            )
+           )
+          )
+          (block $label$46
+           (if
+            (local.get $15)
+            (block
+             (if
+              (i32.eq
+               (local.get $2)
+               (i32.load
+                (local.tee $0
+                 (i32.add
+                  (i32.shl
+                   (local.tee $1
+                    (i32.load offset=28
+                     (local.get $2)
+                    )
+                   )
+                   (i32.const 2)
+                  )
+                  (i32.const 4480)
+                 )
+                )
+               )
+              )
+              (block
+               (i32.store
+                (local.get $0)
+                (local.get $4)
+               )
+               (if
+                (i32.eqz
+                 (local.get $4)
+                )
+                (block
+                 (i32.store
+                  (i32.const 4180)
+                  (i32.and
+                   (local.get $6)
+                   (i32.xor
+                    (i32.shl
+                     (i32.const 1)
+                     (local.get $1)
+                    )
+                    (i32.const -1)
+                   )
+                  )
+                 )
+                 (br $label$46)
+                )
+               )
+              )
+              (block
+               (if
+                (i32.lt_u
+                 (local.get $15)
+                 (i32.load
+                  (i32.const 4192)
+                 )
+                )
+                (call $fimport$8)
+               )
+               (if
+                (i32.eq
+                 (i32.load
+                  (local.tee $0
+                   (i32.add
+                    (local.get $15)
+                    (i32.const 16)
+                   )
+                  )
+                 )
+                 (local.get $2)
+                )
+                (i32.store
+                 (local.get $0)
+                 (local.get $4)
+                )
+                (i32.store offset=20
+                 (local.get $15)
+                 (local.get $4)
+                )
+               )
+               (br_if $label$46
+                (i32.eqz
+                 (local.get $4)
+                )
+               )
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.get $4)
+               (local.tee $0
+                (i32.load
+                 (i32.const 4192)
+                )
+               )
+              )
+              (call $fimport$8)
+             )
+             (i32.store offset=24
+              (local.get $4)
+              (local.get $15)
+             )
+             (if
+              (local.tee $1
+               (i32.load offset=16
+                (local.get $2)
+               )
+              )
+              (if
+               (i32.lt_u
+                (local.get $1)
+                (local.get $0)
+               )
+               (call $fimport$8)
+               (block
+                (i32.store offset=16
+                 (local.get $4)
+                 (local.get $1)
+                )
+                (i32.store offset=24
+                 (local.get $1)
+                 (local.get $4)
+                )
+               )
+              )
+             )
+             (if
+              (local.tee $0
+               (i32.load offset=20
+                (local.get $2)
+               )
+              )
+              (if
+               (i32.lt_u
+                (local.get $0)
+                (i32.load
+                 (i32.const 4192)
+                )
+               )
+               (call $fimport$8)
+               (block
+                (i32.store offset=20
+                 (local.get $4)
+                 (local.get $0)
+                )
+                (i32.store offset=24
+                 (local.get $0)
+                 (local.get $4)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.get $9)
+            (i32.const 16)
+           )
+           (block
+            (i32.store offset=4
+             (local.get $2)
+             (i32.or
+              (local.tee $0
+               (i32.add
+                (local.get $9)
+                (local.get $3)
+               )
+              )
+              (i32.const 3)
+             )
+            )
+            (i32.store
+             (local.tee $0
+              (i32.add
+               (i32.add
+                (local.get $2)
+                (local.get $0)
+               )
+               (i32.const 4)
+              )
+             )
+             (i32.or
+              (i32.load
+               (local.get $0)
+              )
+              (i32.const 1)
+             )
+            )
+           )
+           (block
+            (i32.store offset=4
+             (local.get $2)
+             (i32.or
+              (local.get $3)
+              (i32.const 3)
+             )
+            )
+            (i32.store offset=4
+             (local.get $13)
+             (i32.or
+              (local.get $9)
+              (i32.const 1)
+             )
+            )
+            (i32.store
+             (i32.add
+              (local.get $13)
+              (local.get $9)
+             )
+             (local.get $9)
+            )
+            (if
+             (local.get $16)
+             (block
+              (local.set $7
+               (i32.load
+                (i32.const 4196)
+               )
+              )
+              (local.set $3
+               (i32.add
+                (i32.shl
+                 (i32.shl
+                  (local.tee $0
+                   (i32.shr_u
+                    (local.get $16)
+                    (i32.const 3)
+                   )
+                  )
+                  (i32.const 1)
+                 )
+                 (i32.const 2)
+                )
+                (i32.const 4216)
+               )
+              )
+              (if
+               (i32.and
+                (local.get $8)
+                (local.tee $0
+                 (i32.shl
+                  (i32.const 1)
+                  (local.get $0)
+                 )
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.tee $0
+                  (i32.load
+                   (local.tee $1
+                    (i32.add
+                     (local.get $3)
+                     (i32.const 8)
+                    )
+                   )
+                  )
+                 )
+                 (i32.load
+                  (i32.const 4192)
+                 )
+                )
+                (call $fimport$8)
+                (block
+                 (local.set $10
+                  (local.get $1)
+                 )
+                 (local.set $5
+                  (local.get $0)
+                 )
+                )
+               )
+               (block
+                (i32.store
+                 (i32.const 4176)
+                 (i32.or
+                  (local.get $8)
+                  (local.get $0)
+                 )
+                )
+                (local.set $10
+                 (i32.add
+                  (local.get $3)
+                  (i32.const 8)
+                 )
+                )
+                (local.set $5
+                 (local.get $3)
+                )
+               )
+              )
+              (i32.store
+               (local.get $10)
+               (local.get $7)
+              )
+              (i32.store offset=12
+               (local.get $5)
+               (local.get $7)
+              )
+              (i32.store offset=8
+               (local.get $7)
+               (local.get $5)
+              )
+              (i32.store offset=12
+               (local.get $7)
+               (local.get $3)
+              )
+             )
+            )
+            (i32.store
+             (i32.const 4184)
+             (local.get $9)
+            )
+            (i32.store
+             (i32.const 4196)
+             (local.get $13)
+            )
+           )
+          )
+          (global.set $global$1
+           (local.get $14)
+          )
+          (return
+           (i32.add
+            (local.get $2)
+            (i32.const 8)
+           )
+          )
+         )
+         (local.set $0
+          (local.get $3)
+         )
+        )
+       )
+       (local.set $0
+        (local.get $3)
+       )
+      )
+     )
+     (if
+      (i32.gt_u
+       (local.get $0)
+       (i32.const -65)
+      )
+      (local.set $0
+       (i32.const -1)
+      )
+      (block
+       (local.set $7
+        (i32.and
+         (local.tee $0
+          (i32.add
+           (local.get $0)
+           (i32.const 11)
+          )
+         )
+         (i32.const -8)
+        )
+       )
+       (if
+        (local.tee $5
+         (i32.load
+          (i32.const 4180)
+         )
+        )
+        (block
+         (local.set $17
+          (if (result i32)
+           (local.tee $0
+            (i32.shr_u
+             (local.get $0)
+             (i32.const 8)
+            )
+           )
+           (if (result i32)
+            (i32.gt_u
+             (local.get $7)
+             (i32.const 16777215)
+            )
+            (i32.const 31)
+            (i32.or
+             (i32.and
+              (i32.shr_u
+               (local.get $7)
+               (i32.add
+                (local.tee $0
+                 (i32.add
+                  (i32.sub
+                   (i32.const 14)
+                   (i32.or
+                    (i32.or
+                     (local.tee $0
+                      (i32.and
+                       (i32.shr_u
+                        (i32.add
+                         (local.tee $1
+                          (i32.shl
+                           (local.get $0)
+                           (local.tee $3
+                            (i32.and
+                             (i32.shr_u
+                              (i32.add
+                               (local.get $0)
+                               (i32.const 1048320)
+                              )
+                              (i32.const 16)
+                             )
+                             (i32.const 8)
+                            )
+                           )
+                          )
+                         )
+                         (i32.const 520192)
+                        )
+                        (i32.const 16)
+                       )
+                       (i32.const 4)
+                      )
+                     )
+                     (local.get $3)
+                    )
+                    (local.tee $0
+                     (i32.and
+                      (i32.shr_u
+                       (i32.add
+                        (local.tee $1
+                         (i32.shl
+                          (local.get $1)
+                          (local.get $0)
+                         )
+                        )
+                        (i32.const 245760)
+                       )
+                       (i32.const 16)
+                      )
+                      (i32.const 2)
+                     )
+                    )
+                   )
+                  )
+                  (i32.shr_u
+                   (i32.shl
+                    (local.get $1)
+                    (local.get $0)
+                   )
+                   (i32.const 15)
+                  )
+                 )
+                )
+                (i32.const 7)
+               )
+              )
+              (i32.const 1)
+             )
+             (i32.shl
+              (local.get $0)
+              (i32.const 1)
+             )
+            )
+           )
+           (i32.const 0)
+          )
+         )
+         (local.set $3
+          (i32.sub
+           (i32.const 0)
+           (local.get $7)
+          )
+         )
+         (block $label$78
+          (block $label$79
+           (block $label$80
+            (if
+             (local.tee $1
+              (i32.load
+               (i32.add
+                (i32.shl
+                 (local.get $17)
+                 (i32.const 2)
+                )
+                (i32.const 4480)
+               )
+              )
+             )
+             (block
+              (local.set $0
+               (i32.sub
+                (i32.const 25)
+                (i32.shr_u
+                 (local.get $17)
+                 (i32.const 1)
+                )
+               )
+              )
+              (local.set $4
+               (i32.const 0)
+              )
+              (local.set $10
+               (i32.shl
+                (local.get $7)
+                (if (result i32)
+                 (i32.eq
+                  (local.get $17)
+                  (i32.const 31)
+                 )
+                 (i32.const 0)
+                 (local.get $0)
+                )
+               )
+              )
+              (local.set $0
+               (i32.const 0)
+              )
+              (loop $label$84
+               (if
+                (i32.lt_u
+                 (local.tee $6
+                  (i32.sub
+                   (i32.and
+                    (i32.load offset=4
+                     (local.get $1)
+                    )
+                    (i32.const -8)
+                   )
+                   (local.get $7)
+                  )
+                 )
+                 (local.get $3)
+                )
+                (if
+                 (local.get $6)
+                 (block
+                  (local.set $3
+                   (local.get $6)
+                  )
+                  (local.set $0
+                   (local.get $1)
+                  )
+                 )
+                 (block
+                  (local.set $3
+                   (i32.const 0)
+                  )
+                  (local.set $0
+                   (local.get $1)
+                  )
+                  (br $label$79)
+                 )
+                )
+               )
+               (local.set $1
+                (if (result i32)
+                 (i32.or
+                  (i32.eqz
+                   (local.tee $19
+                    (i32.load offset=20
+                     (local.get $1)
+                    )
+                   )
+                  )
+                  (i32.eq
+                   (local.get $19)
+                   (local.tee $6
+                    (i32.load
+                     (i32.add
+                      (i32.add
+                       (local.get $1)
+                       (i32.const 16)
+                      )
+                      (i32.shl
+                       (i32.shr_u
+                        (local.get $10)
+                        (i32.const 31)
+                       )
+                       (i32.const 2)
+                      )
+                     )
+                    )
+                   )
+                  )
+                 )
+                 (local.get $4)
+                 (local.get $19)
+                )
+               )
+               (local.set $10
+                (i32.shl
+                 (local.get $10)
+                 (i32.xor
+                  (i32.and
+                   (local.tee $4
+                    (i32.eqz
+                     (local.get $6)
+                    )
+                   )
+                   (i32.const 1)
+                  )
+                  (i32.const 1)
+                 )
+                )
+               )
+               (if
+                (local.get $4)
+                (block
+                 (local.set $4
+                  (local.get $1)
+                 )
+                 (local.set $1
+                  (local.get $0)
+                 )
+                 (br $label$80)
+                )
+                (block
+                 (local.set $4
+                  (local.get $1)
+                 )
+                 (local.set $1
+                  (local.get $6)
+                 )
+                 (br $label$84)
+                )
+               )
+              )
+             )
+             (block
+              (local.set $4
+               (i32.const 0)
+              )
+              (local.set $1
+               (i32.const 0)
+              )
+             )
+            )
+           )
+           (br_if $label$79
+            (local.tee $0
+             (if (result i32)
+              (i32.and
+               (i32.eqz
+                (local.get $4)
+               )
+               (i32.eqz
+                (local.get $1)
+               )
+              )
+              (block (result i32)
+               (if
+                (i32.eqz
+                 (local.tee $0
+                  (i32.and
+                   (local.get $5)
+                   (i32.or
+                    (local.tee $0
+                     (i32.shl
+                      (i32.const 2)
+                      (local.get $17)
+                     )
+                    )
+                    (i32.sub
+                     (i32.const 0)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                 )
+                )
+                (block
+                 (local.set $0
+                  (local.get $7)
+                 )
+                 (br $label$2)
+                )
+               )
+               (local.set $10
+                (i32.and
+                 (i32.shr_u
+                  (local.tee $0
+                   (i32.add
+                    (i32.and
+                     (local.get $0)
+                     (i32.sub
+                      (i32.const 0)
+                      (local.get $0)
+                     )
+                    )
+                    (i32.const -1)
+                   )
+                  )
+                  (i32.const 12)
+                 )
+                 (i32.const 16)
+                )
+               )
+               (i32.load
+                (i32.add
+                 (i32.shl
+                  (i32.add
+                   (i32.or
+                    (i32.or
+                     (i32.or
+                      (i32.or
+                       (local.tee $0
+                        (i32.and
+                         (i32.shr_u
+                          (local.tee $4
+                           (i32.shr_u
+                            (local.get $0)
+                            (local.get $10)
+                           )
+                          )
+                          (i32.const 5)
+                         )
+                         (i32.const 8)
+                        )
+                       )
+                       (local.get $10)
+                      )
+                      (local.tee $0
+                       (i32.and
+                        (i32.shr_u
+                         (local.tee $4
+                          (i32.shr_u
+                           (local.get $4)
+                           (local.get $0)
+                          )
+                         )
+                         (i32.const 2)
+                        )
+                        (i32.const 4)
+                       )
+                      )
+                     )
+                     (local.tee $0
+                      (i32.and
+                       (i32.shr_u
+                        (local.tee $4
+                         (i32.shr_u
+                          (local.get $4)
+                          (local.get $0)
+                         )
+                        )
+                        (i32.const 1)
+                       )
+                       (i32.const 2)
+                      )
+                     )
+                    )
+                    (local.tee $0
+                     (i32.and
+                      (i32.shr_u
+                       (local.tee $4
+                        (i32.shr_u
+                         (local.get $4)
+                         (local.get $0)
+                        )
+                       )
+                       (i32.const 1)
+                      )
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                   (i32.shr_u
+                    (local.get $4)
+                    (local.get $0)
+                   )
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 4480)
+                )
+               )
+              )
+              (local.get $4)
+             )
+            )
+           )
+           (local.set $4
+            (local.get $1)
+           )
+           (br $label$78)
+          )
+          (loop $label$96
+           (if
+            (local.tee $10
+             (i32.lt_u
+              (local.tee $4
+               (i32.sub
+                (i32.and
+                 (i32.load offset=4
+                  (local.get $0)
+                 )
+                 (i32.const -8)
+                )
+                (local.get $7)
+               )
+              )
+              (local.get $3)
+             )
+            )
+            (local.set $3
+             (local.get $4)
+            )
+           )
+           (if
+            (local.get $10)
+            (local.set $1
+             (local.get $0)
+            )
+           )
+           (if
+            (local.tee $4
+             (i32.load offset=16
+              (local.get $0)
+             )
+            )
+            (block
+             (local.set $0
+              (local.get $4)
+             )
+             (br $label$96)
+            )
+           )
+           (br_if $label$96
+            (local.tee $0
+             (i32.load offset=20
+              (local.get $0)
+             )
+            )
+           )
+           (local.set $4
+            (local.get $1)
+           )
+          )
+         )
+         (if
+          (local.get $4)
+          (if
+           (i32.lt_u
+            (local.get $3)
+            (i32.sub
+             (i32.load
+              (i32.const 4184)
+             )
+             (local.get $7)
+            )
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.get $4)
+              (local.tee $12
+               (i32.load
+                (i32.const 4192)
+               )
+              )
+             )
+             (call $fimport$8)
+            )
+            (if
+             (i32.ge_u
+              (local.get $4)
+              (local.tee $6
+               (i32.add
+                (local.get $4)
+                (local.get $7)
+               )
+              )
+             )
+             (call $fimport$8)
+            )
+            (local.set $10
+             (i32.load offset=24
+              (local.get $4)
+             )
+            )
+            (block $label$104
+             (if
+              (i32.eq
+               (local.tee $0
+                (i32.load offset=12
+                 (local.get $4)
+                )
+               )
+               (local.get $4)
+              )
+              (block
+               (if
+                (i32.eqz
+                 (local.tee $0
+                  (i32.load
+                   (local.tee $1
+                    (i32.add
+                     (local.get $4)
+                     (i32.const 20)
+                    )
+                   )
+                  )
+                 )
+                )
+                (if
+                 (i32.eqz
+                  (local.tee $0
+                   (i32.load
+                    (local.tee $1
+                     (i32.add
+                      (local.get $4)
+                      (i32.const 16)
+                     )
+                    )
+                   )
+                  )
+                 )
+                 (block
+                  (local.set $13
+                   (i32.const 0)
+                  )
+                  (br $label$104)
+                 )
+                )
+               )
+               (loop $label$108
+                (if
+                 (local.tee $11
+                  (i32.load
+                   (local.tee $9
+                    (i32.add
+                     (local.get $0)
+                     (i32.const 20)
+                    )
+                   )
+                  )
+                 )
+                 (block
+                  (local.set $0
+                   (local.get $11)
+                  )
+                  (local.set $1
+                   (local.get $9)
+                  )
+                  (br $label$108)
+                 )
+                )
+                (if
+                 (local.tee $11
+                  (i32.load
+                   (local.tee $9
+                    (i32.add
+                     (local.get $0)
+                     (i32.const 16)
+                    )
+                   )
+                  )
+                 )
+                 (block
+                  (local.set $0
+                   (local.get $11)
+                  )
+                  (local.set $1
+                   (local.get $9)
+                  )
+                  (br $label$108)
+                 )
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $1)
+                 (local.get $12)
+                )
+                (call $fimport$8)
+                (block
+                 (i32.store
+                  (local.get $1)
+                  (i32.const 0)
+                 )
+                 (local.set $13
+                  (local.get $0)
+                 )
+                )
+               )
+              )
+              (block
+               (if
+                (i32.lt_u
+                 (local.tee $9
+                  (i32.load offset=8
+                   (local.get $4)
+                  )
+                 )
+                 (local.get $12)
+                )
+                (call $fimport$8)
+               )
+               (if
+                (i32.ne
+                 (i32.load
+                  (local.tee $11
+                   (i32.add
+                    (local.get $9)
+                    (i32.const 12)
+                   )
+                  )
+                 )
+                 (local.get $4)
+                )
+                (call $fimport$8)
+               )
+               (if
+                (i32.eq
+                 (i32.load
+                  (local.tee $1
+                   (i32.add
+                    (local.get $0)
+                    (i32.const 8)
+                   )
+                  )
+                 )
+                 (local.get $4)
+                )
+                (block
+                 (i32.store
+                  (local.get $11)
+                  (local.get $0)
+                 )
+                 (i32.store
+                  (local.get $1)
+                  (local.get $9)
+                 )
+                 (local.set $13
+                  (local.get $0)
+                 )
+                )
+                (call $fimport$8)
+               )
+              )
+             )
+            )
+            (block $label$118
+             (if
+              (local.get $10)
+              (block
+               (if
+                (i32.eq
+                 (local.get $4)
+                 (i32.load
+                  (local.tee $0
+                   (i32.add
+                    (i32.shl
+                     (local.tee $1
+                      (i32.load offset=28
+                       (local.get $4)
+                      )
+                     )
+                     (i32.const 2)
+                    )
+                    (i32.const 4480)
+                   )
+                  )
+                 )
+                )
+                (block
+                 (i32.store
+                  (local.get $0)
+                  (local.get $13)
+                 )
+                 (if
+                  (i32.eqz
+                   (local.get $13)
+                  )
+                  (block
+                   (i32.store
+                    (i32.const 4180)
+                    (local.tee $2
+                     (i32.and
+                      (local.get $5)
+                      (i32.xor
+                       (i32.shl
+                        (i32.const 1)
+                        (local.get $1)
+                       )
+                       (i32.const -1)
+                      )
+                     )
+                    )
+                   )
+                   (br $label$118)
+                  )
+                 )
+                )
+                (block
+                 (if
+                  (i32.lt_u
+                   (local.get $10)
+                   (i32.load
+                    (i32.const 4192)
+                   )
+                  )
+                  (call $fimport$8)
+                 )
+                 (if
+                  (i32.eq
+                   (i32.load
+                    (local.tee $0
+                     (i32.add
+                      (local.get $10)
+                      (i32.const 16)
+                     )
+                    )
+                   )
+                   (local.get $4)
+                  )
+                  (i32.store
+                   (local.get $0)
+                   (local.get $13)
+                  )
+                  (i32.store offset=20
+                   (local.get $10)
+                   (local.get $13)
+                  )
+                 )
+                 (if
+                  (i32.eqz
+                   (local.get $13)
+                  )
+                  (block
+                   (local.set $2
+                    (local.get $5)
+                   )
+                   (br $label$118)
+                  )
+                 )
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $13)
+                 (local.tee $0
+                  (i32.load
+                   (i32.const 4192)
+                  )
+                 )
+                )
+                (call $fimport$8)
+               )
+               (i32.store offset=24
+                (local.get $13)
+                (local.get $10)
+               )
+               (if
+                (local.tee $1
+                 (i32.load offset=16
+                  (local.get $4)
+                 )
+                )
+                (if
+                 (i32.lt_u
+                  (local.get $1)
+                  (local.get $0)
+                 )
+                 (call $fimport$8)
+                 (block
+                  (i32.store offset=16
+                   (local.get $13)
+                   (local.get $1)
+                  )
+                  (i32.store offset=24
+                   (local.get $1)
+                   (local.get $13)
+                  )
+                 )
+                )
+               )
+               (if
+                (local.tee $0
+                 (i32.load offset=20
+                  (local.get $4)
+                 )
+                )
+                (if
+                 (i32.lt_u
+                  (local.get $0)
+                  (i32.load
+                   (i32.const 4192)
+                  )
+                 )
+                 (call $fimport$8)
+                 (block
+                  (i32.store offset=20
+                   (local.get $13)
+                   (local.get $0)
+                  )
+                  (i32.store offset=24
+                   (local.get $0)
+                   (local.get $13)
+                  )
+                  (local.set $2
+                   (local.get $5)
+                  )
+                 )
+                )
+                (local.set $2
+                 (local.get $5)
+                )
+               )
+              )
+              (local.set $2
+               (local.get $5)
+              )
+             )
+            )
+            (block $label$136
+             (if
+              (i32.lt_u
+               (local.get $3)
+               (i32.const 16)
+              )
+              (block
+               (i32.store offset=4
+                (local.get $4)
+                (i32.or
+                 (local.tee $0
+                  (i32.add
+                   (local.get $3)
+                   (local.get $7)
+                  )
+                 )
+                 (i32.const 3)
+                )
+               )
+               (i32.store
+                (local.tee $0
+                 (i32.add
+                  (i32.add
+                   (local.get $4)
+                   (local.get $0)
+                  )
+                  (i32.const 4)
+                 )
+                )
+                (i32.or
+                 (i32.load
+                  (local.get $0)
+                 )
+                 (i32.const 1)
+                )
+               )
+              )
+              (block
+               (i32.store offset=4
+                (local.get $4)
+                (i32.or
+                 (local.get $7)
+                 (i32.const 3)
+                )
+               )
+               (i32.store offset=4
+                (local.get $6)
+                (i32.or
+                 (local.get $3)
+                 (i32.const 1)
+                )
+               )
+               (i32.store
+                (i32.add
+                 (local.get $6)
+                 (local.get $3)
+                )
+                (local.get $3)
+               )
+               (local.set $0
+                (i32.shr_u
+                 (local.get $3)
+                 (i32.const 3)
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $3)
+                 (i32.const 256)
+                )
+                (block
+                 (local.set $3
+                  (i32.add
+                   (i32.shl
+                    (i32.shl
+                     (local.get $0)
+                     (i32.const 1)
+                    )
+                    (i32.const 2)
+                   )
+                   (i32.const 4216)
+                  )
+                 )
+                 (if
+                  (i32.and
+                   (local.tee $1
+                    (i32.load
+                     (i32.const 4176)
+                    )
+                   )
+                   (local.tee $0
+                    (i32.shl
+                     (i32.const 1)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (if
+                   (i32.lt_u
+                    (local.tee $0
+                     (i32.load
+                      (local.tee $1
+                       (i32.add
+                        (local.get $3)
+                        (i32.const 8)
+                       )
+                      )
+                     )
+                    )
+                    (i32.load
+                     (i32.const 4192)
+                    )
+                   )
+                   (call $fimport$8)
+                   (block
+                    (local.set $16
+                     (local.get $1)
+                    )
+                    (local.set $8
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (block
+                   (i32.store
+                    (i32.const 4176)
+                    (i32.or
+                     (local.get $1)
+                     (local.get $0)
+                    )
+                   )
+                   (local.set $16
+                    (i32.add
+                     (local.get $3)
+                     (i32.const 8)
+                    )
+                   )
+                   (local.set $8
+                    (local.get $3)
+                   )
+                  )
+                 )
+                 (i32.store
+                  (local.get $16)
+                  (local.get $6)
+                 )
+                 (i32.store offset=12
+                  (local.get $8)
+                  (local.get $6)
+                 )
+                 (i32.store offset=8
+                  (local.get $6)
+                  (local.get $8)
+                 )
+                 (i32.store offset=12
+                  (local.get $6)
+                  (local.get $3)
+                 )
+                 (br $label$136)
+                )
+               )
+               (local.set $1
+                (i32.add
+                 (i32.shl
+                  (local.tee $5
+                   (if (result i32)
+                    (local.tee $0
+                     (i32.shr_u
+                      (local.get $3)
+                      (i32.const 8)
+                     )
+                    )
+                    (if (result i32)
+                     (i32.gt_u
+                      (local.get $3)
+                      (i32.const 16777215)
+                     )
+                     (i32.const 31)
+                     (i32.or
+                      (i32.and
+                       (i32.shr_u
+                        (local.get $3)
+                        (i32.add
+                         (local.tee $0
+                          (i32.add
+                           (i32.sub
+                            (i32.const 14)
+                            (i32.or
+                             (i32.or
+                              (local.tee $0
+                               (i32.and
+                                (i32.shr_u
+                                 (i32.add
+                                  (local.tee $1
+                                   (i32.shl
+                                    (local.get $0)
+                                    (local.tee $5
+                                     (i32.and
+                                      (i32.shr_u
+                                       (i32.add
+                                        (local.get $0)
+                                        (i32.const 1048320)
+                                       )
+                                       (i32.const 16)
+                                      )
+                                      (i32.const 8)
+                                     )
+                                    )
+                                   )
+                                  )
+                                  (i32.const 520192)
+                                 )
+                                 (i32.const 16)
+                                )
+                                (i32.const 4)
+                               )
+                              )
+                              (local.get $5)
+                             )
+                             (local.tee $0
+                              (i32.and
+                               (i32.shr_u
+                                (i32.add
+                                 (local.tee $1
+                                  (i32.shl
+                                   (local.get $1)
+                                   (local.get $0)
+                                  )
+                                 )
+                                 (i32.const 245760)
+                                )
+                                (i32.const 16)
+                               )
+                               (i32.const 2)
+                              )
+                             )
+                            )
+                           )
+                           (i32.shr_u
+                            (i32.shl
+                             (local.get $1)
+                             (local.get $0)
+                            )
+                            (i32.const 15)
+                           )
+                          )
+                         )
+                         (i32.const 7)
+                        )
+                       )
+                       (i32.const 1)
+                      )
+                      (i32.shl
+                       (local.get $0)
+                       (i32.const 1)
+                      )
+                     )
+                    )
+                    (i32.const 0)
+                   )
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 4480)
+                )
+               )
+               (i32.store offset=28
+                (local.get $6)
+                (local.get $5)
+               )
+               (i32.store offset=4
+                (local.tee $0
+                 (i32.add
+                  (local.get $6)
+                  (i32.const 16)
+                 )
+                )
+                (i32.const 0)
+               )
+               (i32.store
+                (local.get $0)
+                (i32.const 0)
+               )
+               (if
+                (i32.eqz
+                 (i32.and
+                  (local.get $2)
+                  (local.tee $0
+                   (i32.shl
+                    (i32.const 1)
+                    (local.get $5)
+                   )
+                  )
+                 )
+                )
+                (block
+                 (i32.store
+                  (i32.const 4180)
+                  (i32.or
+                   (local.get $2)
+                   (local.get $0)
+                  )
+                 )
+                 (i32.store
+                  (local.get $1)
+                  (local.get $6)
+                 )
+                 (i32.store offset=24
+                  (local.get $6)
+                  (local.get $1)
+                 )
+                 (i32.store offset=12
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (i32.store offset=8
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (br $label$136)
+                )
+               )
+               (local.set $0
+                (i32.load
+                 (local.get $1)
+                )
+               )
+               (local.set $1
+                (i32.sub
+                 (i32.const 25)
+                 (i32.shr_u
+                  (local.get $5)
+                  (i32.const 1)
+                 )
+                )
+               )
+               (local.set $5
+                (i32.shl
+                 (local.get $3)
+                 (if (result i32)
+                  (i32.eq
+                   (local.get $5)
+                   (i32.const 31)
+                  )
+                  (i32.const 0)
+                  (local.get $1)
+                 )
+                )
+               )
+               (block $label$151
+                (block $label$152
+                 (block $label$153
+                  (loop $label$154
+                   (br_if $label$152
+                    (i32.eq
+                     (i32.and
+                      (i32.load offset=4
+                       (local.get $0)
+                      )
+                      (i32.const -8)
+                     )
+                     (local.get $3)
+                    )
+                   )
+                   (local.set $2
+                    (i32.shl
+                     (local.get $5)
+                     (i32.const 1)
+                    )
+                   )
+                   (br_if $label$153
+                    (i32.eqz
+                     (local.tee $1
+                      (i32.load
+                       (local.tee $5
+                        (i32.add
+                         (i32.add
+                          (local.get $0)
+                          (i32.const 16)
+                         )
+                         (i32.shl
+                          (i32.shr_u
+                           (local.get $5)
+                           (i32.const 31)
+                          )
+                          (i32.const 2)
+                         )
+                        )
+                       )
+                      )
+                     )
+                    )
+                   )
+                   (local.set $5
+                    (local.get $2)
+                   )
+                   (local.set $0
+                    (local.get $1)
+                   )
+                   (br $label$154)
+                  )
+                 )
+                 (if
+                  (i32.lt_u
+                   (local.get $5)
+                   (i32.load
+                    (i32.const 4192)
+                   )
+                  )
+                  (call $fimport$8)
+                  (block
+                   (i32.store
+                    (local.get $5)
+                    (local.get $6)
+                   )
+                   (i32.store offset=24
+                    (local.get $6)
+                    (local.get $0)
+                   )
+                   (i32.store offset=12
+                    (local.get $6)
+                    (local.get $6)
+                   )
+                   (i32.store offset=8
+                    (local.get $6)
+                    (local.get $6)
+                   )
+                   (br $label$136)
+                  )
+                 )
+                 (br $label$151)
+                )
+                (if
+                 (i32.and
+                  (i32.ge_u
+                   (local.tee $2
+                    (i32.load
+                     (local.tee $3
+                      (i32.add
+                       (local.get $0)
+                       (i32.const 8)
+                      )
+                     )
+                    )
+                   )
+                   (local.tee $1
+                    (i32.load
+                     (i32.const 4192)
+                    )
+                   )
+                  )
+                  (i32.ge_u
+                   (local.get $0)
+                   (local.get $1)
+                  )
+                 )
+                 (block
+                  (i32.store offset=12
+                   (local.get $2)
+                   (local.get $6)
+                  )
+                  (i32.store
+                   (local.get $3)
+                   (local.get $6)
+                  )
+                  (i32.store offset=8
+                   (local.get $6)
+                   (local.get $2)
+                  )
+                  (i32.store offset=12
+                   (local.get $6)
+                   (local.get $0)
+                  )
+                  (i32.store offset=24
+                   (local.get $6)
+                   (i32.const 0)
+                  )
+                 )
+                 (call $fimport$8)
+                )
+               )
+              )
+             )
+            )
+            (global.set $global$1
+             (local.get $14)
+            )
+            (return
+             (i32.add
+              (local.get $4)
+              (i32.const 8)
+             )
+            )
+           )
+           (local.set $0
+            (local.get $7)
+           )
+          )
+          (local.set $0
+           (local.get $7)
+          )
+         )
+        )
+        (local.set $0
+         (local.get $7)
+        )
+       )
+      )
+     )
+    )
+   )
+   (if
+    (i32.ge_u
+     (local.tee $1
+      (i32.load
+       (i32.const 4184)
+      )
+     )
+     (local.get $0)
+    )
+    (block
+     (local.set $2
+      (i32.load
+       (i32.const 4196)
+      )
+     )
+     (if
+      (i32.gt_u
+       (local.tee $3
+        (i32.sub
+         (local.get $1)
+         (local.get $0)
+        )
+       )
+       (i32.const 15)
+      )
+      (block
+       (i32.store
+        (i32.const 4196)
+        (local.tee $1
+         (i32.add
+          (local.get $2)
+          (local.get $0)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 4184)
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $1)
+        (i32.or
+         (local.get $3)
+         (i32.const 1)
+        )
+       )
+       (i32.store
+        (i32.add
+         (local.get $1)
+         (local.get $3)
+        )
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $2)
+        (i32.or
+         (local.get $0)
+         (i32.const 3)
+        )
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 4184)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 4196)
+        (i32.const 0)
+       )
+       (i32.store offset=4
+        (local.get $2)
+        (i32.or
+         (local.get $1)
+         (i32.const 3)
+        )
+       )
+       (i32.store
+        (local.tee $0
+         (i32.add
+          (i32.add
+           (local.get $2)
+           (local.get $1)
+          )
+          (i32.const 4)
+         )
+        )
+        (i32.or
+         (i32.load
+          (local.get $0)
+         )
+         (i32.const 1)
+        )
+       )
+      )
+     )
+     (global.set $global$1
+      (local.get $14)
+     )
+     (return
+      (i32.add
+       (local.get $2)
+       (i32.const 8)
+      )
+     )
+    )
+   )
+   (if
+    (i32.gt_u
+     (local.tee $10
+      (i32.load
+       (i32.const 4188)
+      )
+     )
+     (local.get $0)
+    )
+    (block
+     (i32.store
+      (i32.const 4188)
+      (local.tee $3
+       (i32.sub
+        (local.get $10)
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store
+      (i32.const 4200)
+      (local.tee $1
+       (i32.add
+        (local.tee $2
+         (i32.load
+          (i32.const 4200)
+         )
+        )
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store offset=4
+      (local.get $1)
+      (i32.or
+       (local.get $3)
+       (i32.const 1)
+      )
+     )
+     (i32.store offset=4
+      (local.get $2)
+      (i32.or
+       (local.get $0)
+       (i32.const 3)
+      )
+     )
+     (global.set $global$1
+      (local.get $14)
+     )
+     (return
+      (i32.add
+       (local.get $2)
+       (i32.const 8)
+      )
+     )
+    )
+   )
+   (if
+    (i32.le_u
+     (local.tee $6
+      (i32.and
+       (local.tee $8
+        (i32.add
+         (local.tee $1
+          (if (result i32)
+           (i32.load
+            (i32.const 4648)
+           )
+           (i32.load
+            (i32.const 4656)
+           )
+           (block (result i32)
+            (i32.store
+             (i32.const 4656)
+             (i32.const 4096)
+            )
+            (i32.store
+             (i32.const 4652)
+             (i32.const 4096)
+            )
+            (i32.store
+             (i32.const 4660)
+             (i32.const -1)
+            )
+            (i32.store
+             (i32.const 4664)
+             (i32.const -1)
+            )
+            (i32.store
+             (i32.const 4668)
+             (i32.const 0)
+            )
+            (i32.store
+             (i32.const 4620)
+             (i32.const 0)
+            )
+            (i32.store
+             (local.get $18)
+             (local.tee $1
+              (i32.xor
+               (i32.and
+                (local.get $18)
+                (i32.const -16)
+               )
+               (i32.const 1431655768)
+              )
+             )
+            )
+            (i32.store
+             (i32.const 4648)
+             (local.get $1)
+            )
+            (i32.const 4096)
+           )
+          )
+         )
+         (local.tee $13
+          (i32.add
+           (local.get $0)
+           (i32.const 47)
+          )
+         )
+        )
+       )
+       (local.tee $4
+        (i32.sub
+         (i32.const 0)
+         (local.get $1)
+        )
+       )
+      )
+     )
+     (local.get $0)
+    )
+    (block
+     (global.set $global$1
+      (local.get $14)
+     )
+     (return
+      (i32.const 0)
+     )
+    )
+   )
+   (if
+    (local.tee $2
+     (i32.load
+      (i32.const 4616)
+     )
+    )
+    (if
+     (i32.or
+      (i32.le_u
+       (local.tee $1
+        (i32.add
+         (local.tee $3
+          (i32.load
+           (i32.const 4608)
+          )
+         )
+         (local.get $6)
+        )
+       )
+       (local.get $3)
+      )
+      (i32.gt_u
+       (local.get $1)
+       (local.get $2)
+      )
+     )
+     (block
+      (global.set $global$1
+       (local.get $14)
+      )
+      (return
+       (i32.const 0)
+      )
+     )
+    )
+   )
+   (local.set $7
+    (i32.add
+     (local.get $0)
+     (i32.const 48)
+    )
+   )
+   (block $label$171
+    (block $label$172
+     (if
+      (i32.eqz
+       (i32.and
+        (i32.load
+         (i32.const 4620)
+        )
+        (i32.const 4)
+       )
+      )
+      (block
+       (block $label$174
+        (block $label$175
+         (block $label$176
+          (br_if $label$176
+           (i32.eqz
+            (local.tee $3
+             (i32.load
+              (i32.const 4200)
+             )
+            )
+           )
+          )
+          (local.set $2
+           (i32.const 4624)
+          )
+          (loop $label$177
+           (block $label$178
+            (if
+             (i32.le_u
+              (local.tee $1
+               (i32.load
+                (local.get $2)
+               )
+              )
+              (local.get $3)
+             )
+             (br_if $label$178
+              (i32.gt_u
+               (i32.add
+                (local.get $1)
+                (i32.load
+                 (local.tee $5
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 4)
+                  )
+                 )
+                )
+               )
+               (local.get $3)
+              )
+             )
+            )
+            (br_if $label$176
+             (i32.eqz
+              (local.tee $1
+               (i32.load offset=8
+                (local.get $2)
+               )
+              )
+             )
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (br $label$177)
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.tee $3
+             (i32.and
+              (i32.sub
+               (local.get $8)
+               (local.get $10)
+              )
+              (local.get $4)
+             )
+            )
+            (i32.const 2147483647)
+           )
+           (if
+            (i32.eq
+             (local.tee $1
+              (call $45
+               (local.get $3)
+              )
+             )
+             (i32.add
+              (i32.load
+               (local.get $2)
+              )
+              (i32.load
+               (local.get $5)
+              )
+             )
+            )
+            (br_if $label$172
+             (i32.ne
+              (local.get $1)
+              (i32.const -1)
+             )
+            )
+            (block
+             (local.set $2
+              (local.get $1)
+             )
+             (local.set $1
+              (local.get $3)
+             )
+             (br $label$175)
+            )
+           )
+          )
+          (br $label$174)
+         )
+         (if
+          (i32.ne
+           (local.tee $1
+            (call $45
+             (i32.const 0)
+            )
+           )
+           (i32.const -1)
+          )
+          (block
+           (local.set $2
+            (i32.sub
+             (i32.and
+              (i32.add
+               (local.tee $5
+                (i32.add
+                 (local.tee $2
+                  (i32.load
+                   (i32.const 4652)
+                  )
+                 )
+                 (i32.const -1)
+                )
+               )
+               (local.tee $3
+                (local.get $1)
+               )
+              )
+              (i32.sub
+               (i32.const 0)
+               (local.get $2)
+              )
+             )
+             (local.get $3)
+            )
+           )
+           (local.set $4
+            (i32.add
+             (local.tee $3
+              (i32.add
+               (if (result i32)
+                (i32.and
+                 (local.get $5)
+                 (local.get $3)
+                )
+                (local.get $2)
+                (i32.const 0)
+               )
+               (local.get $6)
+              )
+             )
+             (local.tee $5
+              (i32.load
+               (i32.const 4608)
+              )
+             )
+            )
+           )
+           (if
+            (i32.and
+             (i32.gt_u
+              (local.get $3)
+              (local.get $0)
+             )
+             (i32.lt_u
+              (local.get $3)
+              (i32.const 2147483647)
+             )
+            )
+            (block
+             (if
+              (local.tee $2
+               (i32.load
+                (i32.const 4616)
+               )
+              )
+              (br_if $label$174
+               (i32.or
+                (i32.le_u
+                 (local.get $4)
+                 (local.get $5)
+                )
+                (i32.gt_u
+                 (local.get $4)
+                 (local.get $2)
+                )
+               )
+              )
+             )
+             (br_if $label$172
+              (i32.eq
+               (local.tee $2
+                (call $45
+                 (local.get $3)
+                )
+               )
+               (local.get $1)
+              )
+             )
+             (local.set $1
+              (local.get $3)
+             )
+             (br $label$175)
+            )
+           )
+          )
+         )
+         (br $label$174)
+        )
+        (local.set $5
+         (i32.sub
+          (i32.const 0)
+          (local.get $1)
+         )
+        )
+        (if
+         (i32.and
+          (i32.gt_u
+           (local.get $7)
+           (local.get $1)
+          )
+          (i32.and
+           (i32.lt_u
+            (local.get $1)
+            (i32.const 2147483647)
+           )
+           (i32.ne
+            (local.get $2)
+            (i32.const -1)
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.tee $3
+            (i32.and
+             (i32.add
+              (i32.sub
+               (local.get $13)
+               (local.get $1)
+              )
+              (local.tee $3
+               (i32.load
+                (i32.const 4656)
+               )
+              )
+             )
+             (i32.sub
+              (i32.const 0)
+              (local.get $3)
+             )
+            )
+           )
+           (i32.const 2147483647)
+          )
+          (if
+           (i32.eq
+            (call $45
+             (local.get $3)
+            )
+            (i32.const -1)
+           )
+           (block
+            (drop
+             (call $45
+              (local.get $5)
+             )
+            )
+            (br $label$174)
+           )
+           (local.set $3
+            (i32.add
+             (local.get $3)
+             (local.get $1)
+            )
+           )
+          )
+          (local.set $3
+           (local.get $1)
+          )
+         )
+         (local.set $3
+          (local.get $1)
+         )
+        )
+        (if
+         (i32.ne
+          (local.get $2)
+          (i32.const -1)
+         )
+         (block
+          (local.set $1
+           (local.get $2)
+          )
+          (br $label$172)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 4620)
+        (i32.or
+         (i32.load
+          (i32.const 4620)
+         )
+         (i32.const 4)
+        )
+       )
+      )
+     )
+     (if
+      (i32.lt_u
+       (local.get $6)
+       (i32.const 2147483647)
+      )
+      (if
+       (i32.and
+        (i32.lt_u
+         (local.tee $1
+          (call $45
+           (local.get $6)
+          )
+         )
+         (local.tee $3
+          (call $45
+           (i32.const 0)
+          )
+         )
+        )
+        (i32.and
+         (i32.ne
+          (local.get $1)
+          (i32.const -1)
+         )
+         (i32.ne
+          (local.get $3)
+          (i32.const -1)
+         )
+        )
+       )
+       (br_if $label$172
+        (i32.gt_u
+         (local.tee $3
+          (i32.sub
+           (local.get $3)
+           (local.get $1)
+          )
+         )
+         (i32.add
+          (local.get $0)
+          (i32.const 40)
+         )
+        )
+       )
+      )
+     )
+     (br $label$171)
+    )
+    (i32.store
+     (i32.const 4608)
+     (local.tee $2
+      (i32.add
+       (i32.load
+        (i32.const 4608)
+       )
+       (local.get $3)
+      )
+     )
+    )
+    (if
+     (i32.gt_u
+      (local.get $2)
+      (i32.load
+       (i32.const 4612)
+      )
+     )
+     (i32.store
+      (i32.const 4612)
+      (local.get $2)
+     )
+    )
+    (block $label$198
+     (if
+      (local.tee $8
+       (i32.load
+        (i32.const 4200)
+       )
+      )
+      (block
+       (local.set $2
+        (i32.const 4624)
+       )
+       (block $label$200
+        (block $label$201
+         (loop $label$202
+          (br_if $label$201
+           (i32.eq
+            (local.get $1)
+            (i32.add
+             (local.tee $4
+              (i32.load
+               (local.get $2)
+              )
+             )
+             (local.tee $5
+              (i32.load
+               (local.tee $7
+                (i32.add
+                 (local.get $2)
+                 (i32.const 4)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (br_if $label$202
+           (local.tee $2
+            (i32.load offset=8
+             (local.get $2)
+            )
+           )
+          )
+         )
+         (br $label$200)
+        )
+        (if
+         (i32.eqz
+          (i32.and
+           (i32.load offset=12
+            (local.get $2)
+           )
+           (i32.const 8)
+          )
+         )
+         (if
+          (i32.and
+           (i32.lt_u
+            (local.get $8)
+            (local.get $1)
+           )
+           (i32.ge_u
+            (local.get $8)
+            (local.get $4)
+           )
+          )
+          (block
+           (i32.store
+            (local.get $7)
+            (i32.add
+             (local.get $5)
+             (local.get $3)
+            )
+           )
+           (local.set $5
+            (i32.load
+             (i32.const 4188)
+            )
+           )
+           (local.set $1
+            (i32.and
+             (i32.sub
+              (i32.const 0)
+              (local.tee $2
+               (i32.add
+                (local.get $8)
+                (i32.const 8)
+               )
+              )
+             )
+             (i32.const 7)
+            )
+           )
+           (i32.store
+            (i32.const 4200)
+            (local.tee $2
+             (i32.add
+              (local.get $8)
+              (if (result i32)
+               (i32.and
+                (local.get $2)
+                (i32.const 7)
+               )
+               (local.get $1)
+               (local.tee $1
+                (i32.const 0)
+               )
+              )
+             )
+            )
+           )
+           (i32.store
+            (i32.const 4188)
+            (local.tee $1
+             (i32.add
+              (i32.sub
+               (local.get $3)
+               (local.get $1)
+              )
+              (local.get $5)
+             )
+            )
+           )
+           (i32.store offset=4
+            (local.get $2)
+            (i32.or
+             (local.get $1)
+             (i32.const 1)
+            )
+           )
+           (i32.store offset=4
+            (i32.add
+             (local.get $2)
+             (local.get $1)
+            )
+            (i32.const 40)
+           )
+           (i32.store
+            (i32.const 4204)
+            (i32.load
+             (i32.const 4664)
+            )
+           )
+           (br $label$198)
+          )
+         )
+        )
+       )
+       (if
+        (i32.lt_u
+         (local.get $1)
+         (local.tee $2
+          (i32.load
+           (i32.const 4192)
+          )
+         )
+        )
+        (block
+         (i32.store
+          (i32.const 4192)
+          (local.get $1)
+         )
+         (local.set $2
+          (local.get $1)
+         )
+        )
+       )
+       (local.set $10
+        (i32.add
+         (local.get $1)
+         (local.get $3)
+        )
+       )
+       (local.set $5
+        (i32.const 4624)
+       )
+       (block $label$208
+        (block $label$209
+         (loop $label$210
+          (br_if $label$209
+           (i32.eq
+            (i32.load
+             (local.get $5)
+            )
+            (local.get $10)
+           )
+          )
+          (br_if $label$210
+           (local.tee $5
+            (i32.load offset=8
+             (local.get $5)
+            )
+           )
+          )
+          (local.set $5
+           (i32.const 4624)
+          )
+         )
+         (br $label$208)
+        )
+        (if
+         (i32.and
+          (i32.load offset=12
+           (local.get $5)
+          )
+          (i32.const 8)
+         )
+         (local.set $5
+          (i32.const 4624)
+         )
+         (block
+          (i32.store
+           (local.get $5)
+           (local.get $1)
+          )
+          (i32.store
+           (local.tee $5
+            (i32.add
+             (local.get $5)
+             (i32.const 4)
+            )
+           )
+           (i32.add
+            (i32.load
+             (local.get $5)
+            )
+            (local.get $3)
+           )
+          )
+          (local.set $7
+           (i32.and
+            (i32.sub
+             (i32.const 0)
+             (local.tee $4
+              (i32.add
+               (local.get $1)
+               (i32.const 8)
+              )
+             )
+            )
+            (i32.const 7)
+           )
+          )
+          (local.set $3
+           (i32.and
+            (i32.sub
+             (i32.const 0)
+             (local.tee $5
+              (i32.add
+               (local.get $10)
+               (i32.const 8)
+              )
+             )
+            )
+            (i32.const 7)
+           )
+          )
+          (local.set $6
+           (i32.add
+            (local.tee $13
+             (i32.add
+              (local.get $1)
+              (if (result i32)
+               (i32.and
+                (local.get $4)
+                (i32.const 7)
+               )
+               (local.get $7)
+               (i32.const 0)
+              )
+             )
+            )
+            (local.get $0)
+           )
+          )
+          (local.set $7
+           (i32.sub
+            (i32.sub
+             (local.tee $4
+              (i32.add
+               (local.get $10)
+               (if (result i32)
+                (i32.and
+                 (local.get $5)
+                 (i32.const 7)
+                )
+                (local.get $3)
+                (i32.const 0)
+               )
+              )
+             )
+             (local.get $13)
+            )
+            (local.get $0)
+           )
+          )
+          (i32.store offset=4
+           (local.get $13)
+           (i32.or
+            (local.get $0)
+            (i32.const 3)
+           )
+          )
+          (block $label$217
+           (if
+            (i32.eq
+             (local.get $4)
+             (local.get $8)
+            )
+            (block
+             (i32.store
+              (i32.const 4188)
+              (local.tee $0
+               (i32.add
+                (i32.load
+                 (i32.const 4188)
+                )
+                (local.get $7)
+               )
+              )
+             )
+             (i32.store
+              (i32.const 4200)
+              (local.get $6)
+             )
+             (i32.store offset=4
+              (local.get $6)
+              (i32.or
+               (local.get $0)
+               (i32.const 1)
+              )
+             )
+            )
+            (block
+             (if
+              (i32.eq
+               (local.get $4)
+               (i32.load
+                (i32.const 4196)
+               )
+              )
+              (block
+               (i32.store
+                (i32.const 4184)
+                (local.tee $0
+                 (i32.add
+                  (i32.load
+                   (i32.const 4184)
+                  )
+                  (local.get $7)
+                 )
+                )
+               )
+               (i32.store
+                (i32.const 4196)
+                (local.get $6)
+               )
+               (i32.store offset=4
+                (local.get $6)
+                (i32.or
+                 (local.get $0)
+                 (i32.const 1)
+                )
+               )
+               (i32.store
+                (i32.add
+                 (local.get $6)
+                 (local.get $0)
+                )
+                (local.get $0)
+               )
+               (br $label$217)
+              )
+             )
+             (i32.store
+              (local.tee $0
+               (i32.add
+                (local.tee $0
+                 (if (result i32)
+                  (i32.eq
+                   (i32.and
+                    (local.tee $0
+                     (i32.load offset=4
+                      (local.get $4)
+                     )
+                    )
+                    (i32.const 3)
+                   )
+                   (i32.const 1)
+                  )
+                  (block (result i32)
+                   (local.set $11
+                    (i32.and
+                     (local.get $0)
+                     (i32.const -8)
+                    )
+                   )
+                   (local.set $1
+                    (i32.shr_u
+                     (local.get $0)
+                     (i32.const 3)
+                    )
+                   )
+                   (block $label$222
+                    (if
+                     (i32.lt_u
+                      (local.get $0)
+                      (i32.const 256)
+                     )
+                     (block
+                      (local.set $5
+                       (i32.load offset=12
+                        (local.get $4)
+                       )
+                      )
+                      (block $label$224
+                       (if
+                        (i32.ne
+                         (local.tee $3
+                          (i32.load offset=8
+                           (local.get $4)
+                          )
+                         )
+                         (local.tee $0
+                          (i32.add
+                           (i32.shl
+                            (i32.shl
+                             (local.get $1)
+                             (i32.const 1)
+                            )
+                            (i32.const 2)
+                           )
+                           (i32.const 4216)
+                          )
+                         )
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.get $3)
+                           (local.get $2)
+                          )
+                          (call $fimport$8)
+                         )
+                         (br_if $label$224
+                          (i32.eq
+                           (i32.load offset=12
+                            (local.get $3)
+                           )
+                           (local.get $4)
+                          )
+                         )
+                         (call $fimport$8)
+                        )
+                       )
+                      )
+                      (if
+                       (i32.eq
+                        (local.get $5)
+                        (local.get $3)
+                       )
+                       (block
+                        (i32.store
+                         (i32.const 4176)
+                         (i32.and
+                          (i32.load
+                           (i32.const 4176)
+                          )
+                          (i32.xor
+                           (i32.shl
+                            (i32.const 1)
+                            (local.get $1)
+                           )
+                           (i32.const -1)
+                          )
+                         )
+                        )
+                        (br $label$222)
+                       )
+                      )
+                      (block $label$228
+                       (if
+                        (i32.eq
+                         (local.get $5)
+                         (local.get $0)
+                        )
+                        (local.set $20
+                         (i32.add
+                          (local.get $5)
+                          (i32.const 8)
+                         )
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.get $5)
+                           (local.get $2)
+                          )
+                          (call $fimport$8)
+                         )
+                         (if
+                          (i32.eq
+                           (i32.load
+                            (local.tee $0
+                             (i32.add
+                              (local.get $5)
+                              (i32.const 8)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (block
+                           (local.set $20
+                            (local.get $0)
+                           )
+                           (br $label$228)
+                          )
+                         )
+                         (call $fimport$8)
+                        )
+                       )
+                      )
+                      (i32.store offset=12
+                       (local.get $3)
+                       (local.get $5)
+                      )
+                      (i32.store
+                       (local.get $20)
+                       (local.get $3)
+                      )
+                     )
+                     (block
+                      (local.set $8
+                       (i32.load offset=24
+                        (local.get $4)
+                       )
+                      )
+                      (block $label$234
+                       (if
+                        (i32.eq
+                         (local.tee $0
+                          (i32.load offset=12
+                           (local.get $4)
+                          )
+                         )
+                         (local.get $4)
+                        )
+                        (block
+                         (if
+                          (i32.eqz
+                           (local.tee $0
+                            (i32.load
+                             (local.tee $1
+                              (i32.add
+                               (local.tee $3
+                                (i32.add
+                                 (local.get $4)
+                                 (i32.const 16)
+                                )
+                               )
+                               (i32.const 4)
+                              )
+                             )
+                            )
+                           )
+                          )
+                          (if
+                           (local.tee $0
+                            (i32.load
+                             (local.get $3)
+                            )
+                           )
+                           (local.set $1
+                            (local.get $3)
+                           )
+                           (block
+                            (local.set $12
+                             (i32.const 0)
+                            )
+                            (br $label$234)
+                           )
+                          )
+                         )
+                         (loop $label$239
+                          (if
+                           (local.tee $3
+                            (i32.load
+                             (local.tee $5
+                              (i32.add
+                               (local.get $0)
+                               (i32.const 20)
+                              )
+                             )
+                            )
+                           )
+                           (block
+                            (local.set $0
+                             (local.get $3)
+                            )
+                            (local.set $1
+                             (local.get $5)
+                            )
+                            (br $label$239)
+                           )
+                          )
+                          (if
+                           (local.tee $3
+                            (i32.load
+                             (local.tee $5
+                              (i32.add
+                               (local.get $0)
+                               (i32.const 16)
+                              )
+                             )
+                            )
+                           )
+                           (block
+                            (local.set $0
+                             (local.get $3)
+                            )
+                            (local.set $1
+                             (local.get $5)
+                            )
+                            (br $label$239)
+                           )
+                          )
+                         )
+                         (if
+                          (i32.lt_u
+                           (local.get $1)
+                           (local.get $2)
+                          )
+                          (call $fimport$8)
+                          (block
+                           (i32.store
+                            (local.get $1)
+                            (i32.const 0)
+                           )
+                           (local.set $12
+                            (local.get $0)
+                           )
+                          )
+                         )
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.tee $5
+                            (i32.load offset=8
+                             (local.get $4)
+                            )
+                           )
+                           (local.get $2)
+                          )
+                          (call $fimport$8)
+                         )
+                         (if
+                          (i32.ne
+                           (i32.load
+                            (local.tee $3
+                             (i32.add
+                              (local.get $5)
+                              (i32.const 12)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (call $fimport$8)
+                         )
+                         (if
+                          (i32.eq
+                           (i32.load
+                            (local.tee $1
+                             (i32.add
+                              (local.get $0)
+                              (i32.const 8)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (block
+                           (i32.store
+                            (local.get $3)
+                            (local.get $0)
+                           )
+                           (i32.store
+                            (local.get $1)
+                            (local.get $5)
+                           )
+                           (local.set $12
+                            (local.get $0)
+                           )
+                          )
+                          (call $fimport$8)
+                         )
+                        )
+                       )
+                      )
+                      (br_if $label$222
+                       (i32.eqz
+                        (local.get $8)
+                       )
+                      )
+                      (block $label$249
+                       (if
+                        (i32.eq
+                         (local.get $4)
+                         (i32.load
+                          (local.tee $0
+                           (i32.add
+                            (i32.shl
+                             (local.tee $1
+                              (i32.load offset=28
+                               (local.get $4)
+                              )
+                             )
+                             (i32.const 2)
+                            )
+                            (i32.const 4480)
+                           )
+                          )
+                         )
+                        )
+                        (block
+                         (i32.store
+                          (local.get $0)
+                          (local.get $12)
+                         )
+                         (br_if $label$249
+                          (local.get $12)
+                         )
+                         (i32.store
+                          (i32.const 4180)
+                          (i32.and
+                           (i32.load
+                            (i32.const 4180)
+                           )
+                           (i32.xor
+                            (i32.shl
+                             (i32.const 1)
+                             (local.get $1)
+                            )
+                            (i32.const -1)
+                           )
+                          )
+                         )
+                         (br $label$222)
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.get $8)
+                           (i32.load
+                            (i32.const 4192)
+                           )
+                          )
+                          (call $fimport$8)
+                         )
+                         (if
+                          (i32.eq
+                           (i32.load
+                            (local.tee $0
+                             (i32.add
+                              (local.get $8)
+                              (i32.const 16)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (i32.store
+                           (local.get $0)
+                           (local.get $12)
+                          )
+                          (i32.store offset=20
+                           (local.get $8)
+                           (local.get $12)
+                          )
+                         )
+                         (br_if $label$222
+                          (i32.eqz
+                           (local.get $12)
+                          )
+                         )
+                        )
+                       )
+                      )
+                      (if
+                       (i32.lt_u
+                        (local.get $12)
+                        (local.tee $1
+                         (i32.load
+                          (i32.const 4192)
+                         )
+                        )
+                       )
+                       (call $fimport$8)
+                      )
+                      (i32.store offset=24
+                       (local.get $12)
+                       (local.get $8)
+                      )
+                      (if
+                       (local.tee $3
+                        (i32.load
+                         (local.tee $0
+                          (i32.add
+                           (local.get $4)
+                           (i32.const 16)
+                          )
+                         )
+                        )
+                       )
+                       (if
+                        (i32.lt_u
+                         (local.get $3)
+                         (local.get $1)
+                        )
+                        (call $fimport$8)
+                        (block
+                         (i32.store offset=16
+                          (local.get $12)
+                          (local.get $3)
+                         )
+                         (i32.store offset=24
+                          (local.get $3)
+                          (local.get $12)
+                         )
+                        )
+                       )
+                      )
+                      (br_if $label$222
+                       (i32.eqz
+                        (local.tee $0
+                         (i32.load offset=4
+                          (local.get $0)
+                         )
+                        )
+                       )
+                      )
+                      (if
+                       (i32.lt_u
+                        (local.get $0)
+                        (i32.load
+                         (i32.const 4192)
+                        )
+                       )
+                       (call $fimport$8)
+                       (block
+                        (i32.store offset=20
+                         (local.get $12)
+                         (local.get $0)
+                        )
+                        (i32.store offset=24
+                         (local.get $0)
+                         (local.get $12)
+                        )
+                       )
+                      )
+                     )
+                    )
+                   )
+                   (local.set $7
+                    (i32.add
+                     (local.get $11)
+                     (local.get $7)
+                    )
+                   )
+                   (i32.add
+                    (local.get $4)
+                    (local.get $11)
+                   )
+                  )
+                  (local.get $4)
+                 )
+                )
+                (i32.const 4)
+               )
+              )
+              (i32.and
+               (i32.load
+                (local.get $0)
+               )
+               (i32.const -2)
+              )
+             )
+             (i32.store offset=4
+              (local.get $6)
+              (i32.or
+               (local.get $7)
+               (i32.const 1)
+              )
+             )
+             (i32.store
+              (i32.add
+               (local.get $6)
+               (local.get $7)
+              )
+              (local.get $7)
+             )
+             (local.set $0
+              (i32.shr_u
+               (local.get $7)
+               (i32.const 3)
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.get $7)
+               (i32.const 256)
+              )
+              (block
+               (local.set $3
+                (i32.add
+                 (i32.shl
+                  (i32.shl
+                   (local.get $0)
+                   (i32.const 1)
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 4216)
+                )
+               )
+               (block $label$263
+                (if
+                 (i32.and
+                  (local.tee $1
+                   (i32.load
+                    (i32.const 4176)
+                   )
+                  )
+                  (local.tee $0
+                   (i32.shl
+                    (i32.const 1)
+                    (local.get $0)
+                   )
+                  )
+                 )
+                 (block
+                  (if
+                   (i32.ge_u
+                    (local.tee $0
+                     (i32.load
+                      (local.tee $1
+                       (i32.add
+                        (local.get $3)
+                        (i32.const 8)
+                       )
+                      )
+                     )
+                    )
+                    (i32.load
+                     (i32.const 4192)
+                    )
+                   )
+                   (block
+                    (local.set $21
+                     (local.get $1)
+                    )
+                    (local.set $9
+                     (local.get $0)
+                    )
+                    (br $label$263)
+                   )
+                  )
+                  (call $fimport$8)
+                 )
+                 (block
+                  (i32.store
+                   (i32.const 4176)
+                   (i32.or
+                    (local.get $1)
+                    (local.get $0)
+                   )
+                  )
+                  (local.set $21
+                   (i32.add
+                    (local.get $3)
+                    (i32.const 8)
+                   )
+                  )
+                  (local.set $9
+                   (local.get $3)
+                  )
+                 )
+                )
+               )
+               (i32.store
+                (local.get $21)
+                (local.get $6)
+               )
+               (i32.store offset=12
+                (local.get $9)
+                (local.get $6)
+               )
+               (i32.store offset=8
+                (local.get $6)
+                (local.get $9)
+               )
+               (i32.store offset=12
+                (local.get $6)
+                (local.get $3)
+               )
+               (br $label$217)
+              )
+             )
+             (local.set $3
+              (i32.add
+               (i32.shl
+                (local.tee $2
+                 (block $label$267 (result i32)
+                  (if (result i32)
+                   (local.tee $0
+                    (i32.shr_u
+                     (local.get $7)
+                     (i32.const 8)
+                    )
+                   )
+                   (block (result i32)
+                    (drop
+                     (br_if $label$267
+                      (i32.const 31)
+                      (i32.gt_u
+                       (local.get $7)
+                       (i32.const 16777215)
+                      )
+                     )
+                    )
+                    (i32.or
+                     (i32.and
+                      (i32.shr_u
+                       (local.get $7)
+                       (i32.add
+                        (local.tee $0
+                         (i32.add
+                          (i32.sub
+                           (i32.const 14)
+                           (i32.or
+                            (i32.or
+                             (local.tee $0
+                              (i32.and
+                               (i32.shr_u
+                                (i32.add
+                                 (local.tee $1
+                                  (i32.shl
+                                   (local.get $0)
+                                   (local.tee $3
+                                    (i32.and
+                                     (i32.shr_u
+                                      (i32.add
+                                       (local.get $0)
+                                       (i32.const 1048320)
+                                      )
+                                      (i32.const 16)
+                                     )
+                                     (i32.const 8)
+                                    )
+                                   )
+                                  )
+                                 )
+                                 (i32.const 520192)
+                                )
+                                (i32.const 16)
+                               )
+                               (i32.const 4)
+                              )
+                             )
+                             (local.get $3)
+                            )
+                            (local.tee $0
+                             (i32.and
+                              (i32.shr_u
+                               (i32.add
+                                (local.tee $1
+                                 (i32.shl
+                                  (local.get $1)
+                                  (local.get $0)
+                                 )
+                                )
+                                (i32.const 245760)
+                               )
+                               (i32.const 16)
+                              )
+                              (i32.const 2)
+                             )
+                            )
+                           )
+                          )
+                          (i32.shr_u
+                           (i32.shl
+                            (local.get $1)
+                            (local.get $0)
+                           )
+                           (i32.const 15)
+                          )
+                         )
+                        )
+                        (i32.const 7)
+                       )
+                      )
+                      (i32.const 1)
+                     )
+                     (i32.shl
+                      (local.get $0)
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                   (i32.const 0)
+                  )
+                 )
+                )
+                (i32.const 2)
+               )
+               (i32.const 4480)
+              )
+             )
+             (i32.store offset=28
+              (local.get $6)
+              (local.get $2)
+             )
+             (i32.store offset=4
+              (local.tee $0
+               (i32.add
+                (local.get $6)
+                (i32.const 16)
+               )
+              )
+              (i32.const 0)
+             )
+             (i32.store
+              (local.get $0)
+              (i32.const 0)
+             )
+             (if
+              (i32.eqz
+               (i32.and
+                (local.tee $1
+                 (i32.load
+                  (i32.const 4180)
+                 )
+                )
+                (local.tee $0
+                 (i32.shl
+                  (i32.const 1)
+                  (local.get $2)
+                 )
+                )
+               )
+              )
+              (block
+               (i32.store
+                (i32.const 4180)
+                (i32.or
+                 (local.get $1)
+                 (local.get $0)
+                )
+               )
+               (i32.store
+                (local.get $3)
+                (local.get $6)
+               )
+               (i32.store offset=24
+                (local.get $6)
+                (local.get $3)
+               )
+               (i32.store offset=12
+                (local.get $6)
+                (local.get $6)
+               )
+               (i32.store offset=8
+                (local.get $6)
+                (local.get $6)
+               )
+               (br $label$217)
+              )
+             )
+             (local.set $0
+              (i32.load
+               (local.get $3)
+              )
+             )
+             (local.set $1
+              (i32.sub
+               (i32.const 25)
+               (i32.shr_u
+                (local.get $2)
+                (i32.const 1)
+               )
+              )
+             )
+             (local.set $2
+              (i32.shl
+               (local.get $7)
+               (if (result i32)
+                (i32.eq
+                 (local.get $2)
+                 (i32.const 31)
+                )
+                (i32.const 0)
+                (local.get $1)
+               )
+              )
+             )
+             (block $label$273
+              (block $label$274
+               (block $label$275
+                (loop $label$276
+                 (br_if $label$274
+                  (i32.eq
+                   (i32.and
+                    (i32.load offset=4
+                     (local.get $0)
+                    )
+                    (i32.const -8)
+                   )
+                   (local.get $7)
+                  )
+                 )
+                 (local.set $3
+                  (i32.shl
+                   (local.get $2)
+                   (i32.const 1)
+                  )
+                 )
+                 (br_if $label$275
+                  (i32.eqz
+                   (local.tee $1
+                    (i32.load
+                     (local.tee $2
+                      (i32.add
+                       (i32.add
+                        (local.get $0)
+                        (i32.const 16)
+                       )
+                       (i32.shl
+                        (i32.shr_u
+                         (local.get $2)
+                         (i32.const 31)
+                        )
+                        (i32.const 2)
+                       )
+                      )
+                     )
+                    )
+                   )
+                  )
+                 )
+                 (local.set $2
+                  (local.get $3)
+                 )
+                 (local.set $0
+                  (local.get $1)
+                 )
+                 (br $label$276)
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $2)
+                 (i32.load
+                  (i32.const 4192)
+                 )
+                )
+                (call $fimport$8)
+                (block
+                 (i32.store
+                  (local.get $2)
+                  (local.get $6)
+                 )
+                 (i32.store offset=24
+                  (local.get $6)
+                  (local.get $0)
+                 )
+                 (i32.store offset=12
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (i32.store offset=8
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (br $label$217)
+                )
+               )
+               (br $label$273)
+              )
+              (if
+               (i32.and
+                (i32.ge_u
+                 (local.tee $2
+                  (i32.load
+                   (local.tee $3
+                    (i32.add
+                     (local.get $0)
+                     (i32.const 8)
+                    )
+                   )
+                  )
+                 )
+                 (local.tee $1
+                  (i32.load
+                   (i32.const 4192)
+                  )
+                 )
+                )
+                (i32.ge_u
+                 (local.get $0)
+                 (local.get $1)
+                )
+               )
+               (block
+                (i32.store offset=12
+                 (local.get $2)
+                 (local.get $6)
+                )
+                (i32.store
+                 (local.get $3)
+                 (local.get $6)
+                )
+                (i32.store offset=8
+                 (local.get $6)
+                 (local.get $2)
+                )
+                (i32.store offset=12
+                 (local.get $6)
+                 (local.get $0)
+                )
+                (i32.store offset=24
+                 (local.get $6)
+                 (i32.const 0)
+                )
+               )
+               (call $fimport$8)
+              )
+             )
+            )
+           )
+          )
+          (global.set $global$1
+           (local.get $14)
+          )
+          (return
+           (i32.add
+            (local.get $13)
+            (i32.const 8)
+           )
+          )
+         )
+        )
+       )
+       (loop $label$281
+        (block $label$282
+         (if
+          (i32.le_u
+           (local.tee $2
+            (i32.load
+             (local.get $5)
+            )
+           )
+           (local.get $8)
+          )
+          (br_if $label$282
+           (i32.gt_u
+            (local.tee $13
+             (i32.add
+              (local.get $2)
+              (i32.load offset=4
+               (local.get $5)
+              )
+             )
+            )
+            (local.get $8)
+           )
+          )
+         )
+         (local.set $5
+          (i32.load offset=8
+           (local.get $5)
+          )
+         )
+         (br $label$281)
+        )
+       )
+       (local.set $2
+        (i32.and
+         (i32.sub
+          (i32.const 0)
+          (local.tee $5
+           (i32.add
+            (local.tee $7
+             (i32.add
+              (local.get $13)
+              (i32.const -47)
+             )
+            )
+            (i32.const 8)
+           )
+          )
+         )
+         (i32.const 7)
+        )
+       )
+       (local.set $10
+        (i32.add
+         (local.tee $7
+          (if (result i32)
+           (i32.lt_u
+            (local.tee $2
+             (i32.add
+              (local.get $7)
+              (if (result i32)
+               (i32.and
+                (local.get $5)
+                (i32.const 7)
+               )
+               (local.get $2)
+               (i32.const 0)
+              )
+             )
+            )
+            (local.tee $12
+             (i32.add
+              (local.get $8)
+              (i32.const 16)
+             )
+            )
+           )
+           (local.get $8)
+           (local.get $2)
+          )
+         )
+         (i32.const 8)
+        )
+       )
+       (local.set $5
+        (i32.add
+         (local.get $7)
+         (i32.const 24)
+        )
+       )
+       (local.set $9
+        (i32.add
+         (local.get $3)
+         (i32.const -40)
+        )
+       )
+       (local.set $2
+        (i32.and
+         (i32.sub
+          (i32.const 0)
+          (local.tee $4
+           (i32.add
+            (local.get $1)
+            (i32.const 8)
+           )
+          )
+         )
+         (i32.const 7)
+        )
+       )
+       (i32.store
+        (i32.const 4200)
+        (local.tee $4
+         (i32.add
+          (local.get $1)
+          (if (result i32)
+           (i32.and
+            (local.get $4)
+            (i32.const 7)
+           )
+           (local.get $2)
+           (local.tee $2
+            (i32.const 0)
+           )
+          )
+         )
+        )
+       )
+       (i32.store
+        (i32.const 4188)
+        (local.tee $2
+         (i32.sub
+          (local.get $9)
+          (local.get $2)
+         )
+        )
+       )
+       (i32.store offset=4
+        (local.get $4)
+        (i32.or
+         (local.get $2)
+         (i32.const 1)
+        )
+       )
+       (i32.store offset=4
+        (i32.add
+         (local.get $4)
+         (local.get $2)
+        )
+        (i32.const 40)
+       )
+       (i32.store
+        (i32.const 4204)
+        (i32.load
+         (i32.const 4664)
+        )
+       )
+       (i32.store
+        (local.tee $2
+         (i32.add
+          (local.get $7)
+          (i32.const 4)
+         )
+        )
+        (i32.const 27)
+       )
+       (i64.store align=4
+        (local.get $10)
+        (i64.load align=4
+         (i32.const 4624)
+        )
+       )
+       (i64.store offset=8 align=4
+        (local.get $10)
+        (i64.load align=4
+         (i32.const 4632)
+        )
+       )
+       (i32.store
+        (i32.const 4624)
+        (local.get $1)
+       )
+       (i32.store
+        (i32.const 4628)
+        (local.get $3)
+       )
+       (i32.store
+        (i32.const 4636)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 4632)
+        (local.get $10)
+       )
+       (local.set $1
+        (local.get $5)
+       )
+       (loop $label$290
+        (i32.store
+         (local.tee $1
+          (i32.add
+           (local.get $1)
+           (i32.const 4)
+          )
+         )
+         (i32.const 7)
+        )
+        (br_if $label$290
+         (i32.lt_u
+          (i32.add
+           (local.get $1)
+           (i32.const 4)
+          )
+          (local.get $13)
+         )
+        )
+       )
+       (if
+        (i32.ne
+         (local.get $7)
+         (local.get $8)
+        )
+        (block
+         (i32.store
+          (local.get $2)
+          (i32.and
+           (i32.load
+            (local.get $2)
+           )
+           (i32.const -2)
+          )
+         )
+         (i32.store offset=4
+          (local.get $8)
+          (i32.or
+           (local.tee $4
+            (i32.sub
+             (local.get $7)
+             (local.get $8)
+            )
+           )
+           (i32.const 1)
+          )
+         )
+         (i32.store
+          (local.get $7)
+          (local.get $4)
+         )
+         (local.set $1
+          (i32.shr_u
+           (local.get $4)
+           (i32.const 3)
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $4)
+           (i32.const 256)
+          )
+          (block
+           (local.set $2
+            (i32.add
+             (i32.shl
+              (i32.shl
+               (local.get $1)
+               (i32.const 1)
+              )
+              (i32.const 2)
+             )
+             (i32.const 4216)
+            )
+           )
+           (if
+            (i32.and
+             (local.tee $3
+              (i32.load
+               (i32.const 4176)
+              )
+             )
+             (local.tee $1
+              (i32.shl
+               (i32.const 1)
+               (local.get $1)
+              )
+             )
+            )
+            (if
+             (i32.lt_u
+              (local.tee $1
+               (i32.load
+                (local.tee $3
+                 (i32.add
+                  (local.get $2)
+                  (i32.const 8)
+                 )
+                )
+               )
+              )
+              (i32.load
+               (i32.const 4192)
+              )
+             )
+             (call $fimport$8)
+             (block
+              (local.set $15
+               (local.get $3)
+              )
+              (local.set $11
+               (local.get $1)
+              )
+             )
+            )
+            (block
+             (i32.store
+              (i32.const 4176)
+              (i32.or
+               (local.get $3)
+               (local.get $1)
+              )
+             )
+             (local.set $15
+              (i32.add
+               (local.get $2)
+               (i32.const 8)
+              )
+             )
+             (local.set $11
+              (local.get $2)
+             )
+            )
+           )
+           (i32.store
+            (local.get $15)
+            (local.get $8)
+           )
+           (i32.store offset=12
+            (local.get $11)
+            (local.get $8)
+           )
+           (i32.store offset=8
+            (local.get $8)
+            (local.get $11)
+           )
+           (i32.store offset=12
+            (local.get $8)
+            (local.get $2)
+           )
+           (br $label$198)
+          )
+         )
+         (local.set $2
+          (i32.add
+           (i32.shl
+            (local.tee $5
+             (if (result i32)
+              (local.tee $1
+               (i32.shr_u
+                (local.get $4)
+                (i32.const 8)
+               )
+              )
+              (if (result i32)
+               (i32.gt_u
+                (local.get $4)
+                (i32.const 16777215)
+               )
+               (i32.const 31)
+               (i32.or
+                (i32.and
+                 (i32.shr_u
+                  (local.get $4)
+                  (i32.add
+                   (local.tee $1
+                    (i32.add
+                     (i32.sub
+                      (i32.const 14)
+                      (i32.or
+                       (i32.or
+                        (local.tee $1
+                         (i32.and
+                          (i32.shr_u
+                           (i32.add
+                            (local.tee $3
+                             (i32.shl
+                              (local.get $1)
+                              (local.tee $2
+                               (i32.and
+                                (i32.shr_u
+                                 (i32.add
+                                  (local.get $1)
+                                  (i32.const 1048320)
+                                 )
+                                 (i32.const 16)
+                                )
+                                (i32.const 8)
+                               )
+                              )
+                             )
+                            )
+                            (i32.const 520192)
+                           )
+                           (i32.const 16)
+                          )
+                          (i32.const 4)
+                         )
+                        )
+                        (local.get $2)
+                       )
+                       (local.tee $1
+                        (i32.and
+                         (i32.shr_u
+                          (i32.add
+                           (local.tee $3
+                            (i32.shl
+                             (local.get $3)
+                             (local.get $1)
+                            )
+                           )
+                           (i32.const 245760)
+                          )
+                          (i32.const 16)
+                         )
+                         (i32.const 2)
+                        )
+                       )
+                      )
+                     )
+                     (i32.shr_u
+                      (i32.shl
+                       (local.get $3)
+                       (local.get $1)
+                      )
+                      (i32.const 15)
+                     )
+                    )
+                   )
+                   (i32.const 7)
+                  )
+                 )
+                 (i32.const 1)
+                )
+                (i32.shl
+                 (local.get $1)
+                 (i32.const 1)
+                )
+               )
+              )
+              (i32.const 0)
+             )
+            )
+            (i32.const 2)
+           )
+           (i32.const 4480)
+          )
+         )
+         (i32.store offset=28
+          (local.get $8)
+          (local.get $5)
+         )
+         (i32.store offset=20
+          (local.get $8)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $12)
+          (i32.const 0)
+         )
+         (if
+          (i32.eqz
+           (i32.and
+            (local.tee $3
+             (i32.load
+              (i32.const 4180)
+             )
+            )
+            (local.tee $1
+             (i32.shl
+              (i32.const 1)
+              (local.get $5)
+             )
+            )
+           )
+          )
+          (block
+           (i32.store
+            (i32.const 4180)
+            (i32.or
+             (local.get $3)
+             (local.get $1)
+            )
+           )
+           (i32.store
+            (local.get $2)
+            (local.get $8)
+           )
+           (i32.store offset=24
+            (local.get $8)
+            (local.get $2)
+           )
+           (i32.store offset=12
+            (local.get $8)
+            (local.get $8)
+           )
+           (i32.store offset=8
+            (local.get $8)
+            (local.get $8)
+           )
+           (br $label$198)
+          )
+         )
+         (local.set $1
+          (i32.load
+           (local.get $2)
+          )
+         )
+         (local.set $3
+          (i32.sub
+           (i32.const 25)
+           (i32.shr_u
+            (local.get $5)
+            (i32.const 1)
+           )
+          )
+         )
+         (local.set $5
+          (i32.shl
+           (local.get $4)
+           (if (result i32)
+            (i32.eq
+             (local.get $5)
+             (i32.const 31)
+            )
+            (i32.const 0)
+            (local.get $3)
+           )
+          )
+         )
+         (block $label$304
+          (block $label$305
+           (block $label$306
+            (loop $label$307
+             (br_if $label$305
+              (i32.eq
+               (i32.and
+                (i32.load offset=4
+                 (local.get $1)
+                )
+                (i32.const -8)
+               )
+               (local.get $4)
+              )
+             )
+             (local.set $2
+              (i32.shl
+               (local.get $5)
+               (i32.const 1)
+              )
+             )
+             (br_if $label$306
+              (i32.eqz
+               (local.tee $3
+                (i32.load
+                 (local.tee $5
+                  (i32.add
+                   (i32.add
+                    (local.get $1)
+                    (i32.const 16)
+                   )
+                   (i32.shl
+                    (i32.shr_u
+                     (local.get $5)
+                     (i32.const 31)
+                    )
+                    (i32.const 2)
+                   )
+                  )
+                 )
+                )
+               )
+              )
+             )
+             (local.set $5
+              (local.get $2)
+             )
+             (local.set $1
+              (local.get $3)
+             )
+             (br $label$307)
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $5)
+             (i32.load
+              (i32.const 4192)
+             )
+            )
+            (call $fimport$8)
+            (block
+             (i32.store
+              (local.get $5)
+              (local.get $8)
+             )
+             (i32.store offset=24
+              (local.get $8)
+              (local.get $1)
+             )
+             (i32.store offset=12
+              (local.get $8)
+              (local.get $8)
+             )
+             (i32.store offset=8
+              (local.get $8)
+              (local.get $8)
+             )
+             (br $label$198)
+            )
+           )
+           (br $label$304)
+          )
+          (if
+           (i32.and
+            (i32.ge_u
+             (local.tee $5
+              (i32.load
+               (local.tee $2
+                (i32.add
+                 (local.get $1)
+                 (i32.const 8)
+                )
+               )
+              )
+             )
+             (local.tee $3
+              (i32.load
+               (i32.const 4192)
+              )
+             )
+            )
+            (i32.ge_u
+             (local.get $1)
+             (local.get $3)
+            )
+           )
+           (block
+            (i32.store offset=12
+             (local.get $5)
+             (local.get $8)
+            )
+            (i32.store
+             (local.get $2)
+             (local.get $8)
+            )
+            (i32.store offset=8
+             (local.get $8)
+             (local.get $5)
+            )
+            (i32.store offset=12
+             (local.get $8)
+             (local.get $1)
+            )
+            (i32.store offset=24
+             (local.get $8)
+             (i32.const 0)
+            )
+           )
+           (call $fimport$8)
+          )
+         )
+        )
+       )
+      )
+      (block
+       (if
+        (i32.or
+         (i32.eqz
+          (local.tee $2
+           (i32.load
+            (i32.const 4192)
+           )
+          )
+         )
+         (i32.lt_u
+          (local.get $1)
+          (local.get $2)
+         )
+        )
+        (i32.store
+         (i32.const 4192)
+         (local.get $1)
+        )
+       )
+       (i32.store
+        (i32.const 4624)
+        (local.get $1)
+       )
+       (i32.store
+        (i32.const 4628)
+        (local.get $3)
+       )
+       (i32.store
+        (i32.const 4636)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 4212)
+        (i32.load
+         (i32.const 4648)
+        )
+       )
+       (i32.store
+        (i32.const 4208)
+        (i32.const -1)
+       )
+       (local.set $2
+        (i32.const 0)
+       )
+       (loop $label$314
+        (i32.store offset=12
+         (local.tee $5
+          (i32.add
+           (i32.shl
+            (i32.shl
+             (local.get $2)
+             (i32.const 1)
+            )
+            (i32.const 2)
+           )
+           (i32.const 4216)
+          )
+         )
+         (local.get $5)
+        )
+        (i32.store offset=8
+         (local.get $5)
+         (local.get $5)
+        )
+        (br_if $label$314
+         (i32.ne
+          (local.tee $2
+           (i32.add
+            (local.get $2)
+            (i32.const 1)
+           )
+          )
+          (i32.const 32)
+         )
+        )
+       )
+       (local.set $5
+        (i32.add
+         (local.get $3)
+         (i32.const -40)
+        )
+       )
+       (local.set $3
+        (i32.and
+         (i32.sub
+          (i32.const 0)
+          (local.tee $2
+           (i32.add
+            (local.get $1)
+            (i32.const 8)
+           )
+          )
+         )
+         (i32.const 7)
+        )
+       )
+       (i32.store
+        (i32.const 4200)
+        (local.tee $3
+         (i32.add
+          (local.get $1)
+          (local.tee $1
+           (if (result i32)
+            (i32.and
+             (local.get $2)
+             (i32.const 7)
+            )
+            (local.get $3)
+            (i32.const 0)
+           )
+          )
+         )
+        )
+       )
+       (i32.store
+        (i32.const 4188)
+        (local.tee $1
+         (i32.sub
+          (local.get $5)
+          (local.get $1)
+         )
+        )
+       )
+       (i32.store offset=4
+        (local.get $3)
+        (i32.or
+         (local.get $1)
+         (i32.const 1)
+        )
+       )
+       (i32.store offset=4
+        (i32.add
+         (local.get $3)
+         (local.get $1)
+        )
+        (i32.const 40)
+       )
+       (i32.store
+        (i32.const 4204)
+        (i32.load
+         (i32.const 4664)
+        )
+       )
+      )
+     )
+    )
+    (if
+     (i32.gt_u
+      (local.tee $1
+       (i32.load
+        (i32.const 4188)
+       )
+      )
+      (local.get $0)
+     )
+     (block
+      (i32.store
+       (i32.const 4188)
+       (local.tee $3
+        (i32.sub
+         (local.get $1)
+         (local.get $0)
+        )
+       )
+      )
+      (i32.store
+       (i32.const 4200)
+       (local.tee $1
+        (i32.add
+         (local.tee $2
+          (i32.load
+           (i32.const 4200)
+          )
+         )
+         (local.get $0)
+        )
+       )
+      )
+      (i32.store offset=4
+       (local.get $1)
+       (i32.or
+        (local.get $3)
+        (i32.const 1)
+       )
+      )
+      (i32.store offset=4
+       (local.get $2)
+       (i32.or
+        (local.get $0)
+        (i32.const 3)
+       )
+      )
+      (global.set $global$1
+       (local.get $14)
+      )
+      (return
+       (i32.add
+        (local.get $2)
+        (i32.const 8)
+       )
+      )
+     )
+    )
+   )
+   (i32.store
+    (call $12)
+    (i32.const 12)
+   )
+   (global.set $global$1
+    (local.get $14)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $38 (; 51 ;) (type $3) (param $0 i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (block $label$1
+   (if
+    (i32.eqz
+     (local.get $0)
+    )
+    (return)
+   )
+   (if
+    (i32.lt_u
+     (local.tee $1
+      (i32.add
+       (local.get $0)
+       (i32.const -8)
+      )
+     )
+     (local.tee $11
+      (i32.load
+       (i32.const 4192)
+      )
+     )
+    )
+    (call $fimport$8)
+   )
+   (if
+    (i32.eq
+     (local.tee $8
+      (i32.and
+       (local.tee $0
+        (i32.load
+         (i32.add
+          (local.get $0)
+          (i32.const -4)
+         )
+        )
+       )
+       (i32.const 3)
+      )
+     )
+     (i32.const 1)
+    )
+    (call $fimport$8)
+   )
+   (local.set $6
+    (i32.add
+     (local.get $1)
+     (local.tee $4
+      (i32.and
+       (local.get $0)
+       (i32.const -8)
+      )
+     )
+    )
+   )
+   (block $label$5
+    (if
+     (i32.and
+      (local.get $0)
+      (i32.const 1)
+     )
+     (block
+      (local.set $3
+       (local.get $1)
+      )
+      (local.set $2
+       (local.get $4)
+      )
+     )
+     (block
+      (if
+       (i32.eqz
+        (local.get $8)
+       )
+       (return)
+      )
+      (if
+       (i32.lt_u
+        (local.tee $0
+         (i32.add
+          (local.get $1)
+          (i32.sub
+           (i32.const 0)
+           (local.tee $8
+            (i32.load
+             (local.get $1)
+            )
+           )
+          )
+         )
+        )
+        (local.get $11)
+       )
+       (call $fimport$8)
+      )
+      (local.set $1
+       (i32.add
+        (local.get $8)
+        (local.get $4)
+       )
+      )
+      (if
+       (i32.eq
+        (local.get $0)
+        (i32.load
+         (i32.const 4196)
+        )
+       )
+       (block
+        (if
+         (i32.ne
+          (i32.and
+           (local.tee $3
+            (i32.load
+             (local.tee $2
+              (i32.add
+               (local.get $6)
+               (i32.const 4)
+              )
+             )
+            )
+           )
+           (i32.const 3)
+          )
+          (i32.const 3)
+         )
+         (block
+          (local.set $3
+           (local.get $0)
+          )
+          (local.set $2
+           (local.get $1)
+          )
+          (br $label$5)
+         )
+        )
+        (i32.store
+         (i32.const 4184)
+         (local.get $1)
+        )
+        (i32.store
+         (local.get $2)
+         (i32.and
+          (local.get $3)
+          (i32.const -2)
+         )
+        )
+        (i32.store offset=4
+         (local.get $0)
+         (i32.or
+          (local.get $1)
+          (i32.const 1)
+         )
+        )
+        (i32.store
+         (i32.add
+          (local.get $0)
+          (local.get $1)
+         )
+         (local.get $1)
+        )
+        (return)
+       )
+      )
+      (local.set $10
+       (i32.shr_u
+        (local.get $8)
+        (i32.const 3)
+       )
+      )
+      (if
+       (i32.lt_u
+        (local.get $8)
+        (i32.const 256)
+       )
+       (block
+        (local.set $3
+         (i32.load offset=12
+          (local.get $0)
+         )
+        )
+        (if
+         (i32.ne
+          (local.tee $4
+           (i32.load offset=8
+            (local.get $0)
+           )
+          )
+          (local.tee $2
+           (i32.add
+            (i32.shl
+             (i32.shl
+              (local.get $10)
+              (i32.const 1)
+             )
+             (i32.const 2)
+            )
+            (i32.const 4216)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $4)
+            (local.get $11)
+           )
+           (call $fimport$8)
+          )
+          (if
+           (i32.ne
+            (i32.load offset=12
+             (local.get $4)
+            )
+            (local.get $0)
+           )
+           (call $fimport$8)
+          )
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $3)
+          (local.get $4)
+         )
+         (block
+          (i32.store
+           (i32.const 4176)
+           (i32.and
+            (i32.load
+             (i32.const 4176)
+            )
+            (i32.xor
+             (i32.shl
+              (i32.const 1)
+              (local.get $10)
+             )
+             (i32.const -1)
+            )
+           )
+          )
+          (local.set $3
+           (local.get $0)
+          )
+          (local.set $2
+           (local.get $1)
+          )
+          (br $label$5)
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $3)
+          (local.get $2)
+         )
+         (local.set $5
+          (i32.add
+           (local.get $3)
+           (i32.const 8)
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $3)
+            (local.get $11)
+           )
+           (call $fimport$8)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $2
+              (i32.add
+               (local.get $3)
+               (i32.const 8)
+              )
+             )
+            )
+            (local.get $0)
+           )
+           (local.set $5
+            (local.get $2)
+           )
+           (call $fimport$8)
+          )
+         )
+        )
+        (i32.store offset=12
+         (local.get $4)
+         (local.get $3)
+        )
+        (i32.store
+         (local.get $5)
+         (local.get $4)
+        )
+        (local.set $3
+         (local.get $0)
+        )
+        (local.set $2
+         (local.get $1)
+        )
+        (br $label$5)
+       )
+      )
+      (local.set $12
+       (i32.load offset=24
+        (local.get $0)
+       )
+      )
+      (block $label$22
+       (if
+        (i32.eq
+         (local.tee $4
+          (i32.load offset=12
+           (local.get $0)
+          )
+         )
+         (local.get $0)
+        )
+        (block
+         (if
+          (local.tee $4
+           (i32.load
+            (local.tee $8
+             (i32.add
+              (local.tee $5
+               (i32.add
+                (local.get $0)
+                (i32.const 16)
+               )
+              )
+              (i32.const 4)
+             )
+            )
+           )
+          )
+          (local.set $5
+           (local.get $8)
+          )
+          (if
+           (i32.eqz
+            (local.tee $4
+             (i32.load
+              (local.get $5)
+             )
+            )
+           )
+           (block
+            (local.set $7
+             (i32.const 0)
+            )
+            (br $label$22)
+           )
+          )
+         )
+         (loop $label$27
+          (if
+           (local.tee $10
+            (i32.load
+             (local.tee $8
+              (i32.add
+               (local.get $4)
+               (i32.const 20)
+              )
+             )
+            )
+           )
+           (block
+            (local.set $4
+             (local.get $10)
+            )
+            (local.set $5
+             (local.get $8)
+            )
+            (br $label$27)
+           )
+          )
+          (if
+           (local.tee $10
+            (i32.load
+             (local.tee $8
+              (i32.add
+               (local.get $4)
+               (i32.const 16)
+              )
+             )
+            )
+           )
+           (block
+            (local.set $4
+             (local.get $10)
+            )
+            (local.set $5
+             (local.get $8)
+            )
+            (br $label$27)
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $5)
+           (local.get $11)
+          )
+          (call $fimport$8)
+          (block
+           (i32.store
+            (local.get $5)
+            (i32.const 0)
+           )
+           (local.set $7
+            (local.get $4)
+           )
+          )
+         )
+        )
+        (block
+         (if
+          (i32.lt_u
+           (local.tee $5
+            (i32.load offset=8
+             (local.get $0)
+            )
+           )
+           (local.get $11)
+          )
+          (call $fimport$8)
+         )
+         (if
+          (i32.ne
+           (i32.load
+            (local.tee $8
+             (i32.add
+              (local.get $5)
+              (i32.const 12)
+             )
+            )
+           )
+           (local.get $0)
+          )
+          (call $fimport$8)
+         )
+         (if
+          (i32.eq
+           (i32.load
+            (local.tee $10
+             (i32.add
+              (local.get $4)
+              (i32.const 8)
+             )
+            )
+           )
+           (local.get $0)
+          )
+          (block
+           (i32.store
+            (local.get $8)
+            (local.get $4)
+           )
+           (i32.store
+            (local.get $10)
+            (local.get $5)
+           )
+           (local.set $7
+            (local.get $4)
+           )
+          )
+          (call $fimport$8)
+         )
+        )
+       )
+      )
+      (if
+       (local.get $12)
+       (block
+        (if
+         (i32.eq
+          (local.get $0)
+          (i32.load
+           (local.tee $5
+            (i32.add
+             (i32.shl
+              (local.tee $4
+               (i32.load offset=28
+                (local.get $0)
+               )
+              )
+              (i32.const 2)
+             )
+             (i32.const 4480)
+            )
+           )
+          )
+         )
+         (block
+          (i32.store
+           (local.get $5)
+           (local.get $7)
+          )
+          (if
+           (i32.eqz
+            (local.get $7)
+           )
+           (block
+            (i32.store
+             (i32.const 4180)
+             (i32.and
+              (i32.load
+               (i32.const 4180)
+              )
+              (i32.xor
+               (i32.shl
+                (i32.const 1)
+                (local.get $4)
+               )
+               (i32.const -1)
+              )
+             )
+            )
+            (local.set $3
+             (local.get $0)
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (br $label$5)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $12)
+            (i32.load
+             (i32.const 4192)
+            )
+           )
+           (call $fimport$8)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $4
+              (i32.add
+               (local.get $12)
+               (i32.const 16)
+              )
+             )
+            )
+            (local.get $0)
+           )
+           (i32.store
+            (local.get $4)
+            (local.get $7)
+           )
+           (i32.store offset=20
+            (local.get $12)
+            (local.get $7)
+           )
+          )
+          (if
+           (i32.eqz
+            (local.get $7)
+           )
+           (block
+            (local.set $3
+             (local.get $0)
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (br $label$5)
+           )
+          )
+         )
+        )
+        (if
+         (i32.lt_u
+          (local.get $7)
+          (local.tee $5
+           (i32.load
+            (i32.const 4192)
+           )
+          )
+         )
+         (call $fimport$8)
+        )
+        (i32.store offset=24
+         (local.get $7)
+         (local.get $12)
+        )
+        (if
+         (local.tee $4
+          (i32.load
+           (local.tee $8
+            (i32.add
+             (local.get $0)
+             (i32.const 16)
+            )
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $4)
+           (local.get $5)
+          )
+          (call $fimport$8)
+          (block
+           (i32.store offset=16
+            (local.get $7)
+            (local.get $4)
+           )
+           (i32.store offset=24
+            (local.get $4)
+            (local.get $7)
+           )
+          )
+         )
+        )
+        (if
+         (local.tee $4
+          (i32.load offset=4
+           (local.get $8)
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $4)
+           (i32.load
+            (i32.const 4192)
+           )
+          )
+          (call $fimport$8)
+          (block
+           (i32.store offset=20
+            (local.get $7)
+            (local.get $4)
+           )
+           (i32.store offset=24
+            (local.get $4)
+            (local.get $7)
+           )
+           (local.set $3
+            (local.get $0)
+           )
+           (local.set $2
+            (local.get $1)
+           )
+          )
+         )
+         (block
+          (local.set $3
+           (local.get $0)
+          )
+          (local.set $2
+           (local.get $1)
+          )
+         )
+        )
+       )
+       (block
+        (local.set $3
+         (local.get $0)
+        )
+        (local.set $2
+         (local.get $1)
+        )
+       )
+      )
+     )
+    )
+   )
+   (if
+    (i32.ge_u
+     (local.get $3)
+     (local.get $6)
+    )
+    (call $fimport$8)
+   )
+   (if
+    (i32.eqz
+     (i32.and
+      (local.tee $0
+       (i32.load
+        (local.tee $1
+         (i32.add
+          (local.get $6)
+          (i32.const 4)
+         )
+        )
+       )
+      )
+      (i32.const 1)
+     )
+    )
+    (call $fimport$8)
+   )
+   (if
+    (i32.and
+     (local.get $0)
+     (i32.const 2)
+    )
+    (block
+     (i32.store
+      (local.get $1)
+      (i32.and
+       (local.get $0)
+       (i32.const -2)
+      )
+     )
+     (i32.store offset=4
+      (local.get $3)
+      (i32.or
+       (local.get $2)
+       (i32.const 1)
+      )
+     )
+     (i32.store
+      (i32.add
+       (local.get $3)
+       (local.get $2)
+      )
+      (local.get $2)
+     )
+    )
+    (block
+     (if
+      (i32.eq
+       (local.get $6)
+       (i32.load
+        (i32.const 4200)
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 4188)
+        (local.tee $0
+         (i32.add
+          (i32.load
+           (i32.const 4188)
+          )
+          (local.get $2)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 4200)
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $3)
+        (i32.or
+         (local.get $0)
+         (i32.const 1)
+        )
+       )
+       (if
+        (i32.ne
+         (local.get $3)
+         (i32.load
+          (i32.const 4196)
+         )
+        )
+        (return)
+       )
+       (i32.store
+        (i32.const 4196)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 4184)
+        (i32.const 0)
+       )
+       (return)
+      )
+     )
+     (if
+      (i32.eq
+       (local.get $6)
+       (i32.load
+        (i32.const 4196)
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 4184)
+        (local.tee $0
+         (i32.add
+          (i32.load
+           (i32.const 4184)
+          )
+          (local.get $2)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 4196)
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $3)
+        (i32.or
+         (local.get $0)
+         (i32.const 1)
+        )
+       )
+       (i32.store
+        (i32.add
+         (local.get $3)
+         (local.get $0)
+        )
+        (local.get $0)
+       )
+       (return)
+      )
+     )
+     (local.set $5
+      (i32.add
+       (i32.and
+        (local.get $0)
+        (i32.const -8)
+       )
+       (local.get $2)
+      )
+     )
+     (local.set $4
+      (i32.shr_u
+       (local.get $0)
+       (i32.const 3)
+      )
+     )
+     (block $label$61
+      (if
+       (i32.lt_u
+        (local.get $0)
+        (i32.const 256)
+       )
+       (block
+        (local.set $2
+         (i32.load offset=12
+          (local.get $6)
+         )
+        )
+        (if
+         (i32.ne
+          (local.tee $1
+           (i32.load offset=8
+            (local.get $6)
+           )
+          )
+          (local.tee $0
+           (i32.add
+            (i32.shl
+             (i32.shl
+              (local.get $4)
+              (i32.const 1)
+             )
+             (i32.const 2)
+            )
+            (i32.const 4216)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $1)
+            (i32.load
+             (i32.const 4192)
+            )
+           )
+           (call $fimport$8)
+          )
+          (if
+           (i32.ne
+            (i32.load offset=12
+             (local.get $1)
+            )
+            (local.get $6)
+           )
+           (call $fimport$8)
+          )
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $2)
+          (local.get $1)
+         )
+         (block
+          (i32.store
+           (i32.const 4176)
+           (i32.and
+            (i32.load
+             (i32.const 4176)
+            )
+            (i32.xor
+             (i32.shl
+              (i32.const 1)
+              (local.get $4)
+             )
+             (i32.const -1)
+            )
+           )
+          )
+          (br $label$61)
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $2)
+          (local.get $0)
+         )
+         (local.set $14
+          (i32.add
+           (local.get $2)
+           (i32.const 8)
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $2)
+            (i32.load
+             (i32.const 4192)
+            )
+           )
+           (call $fimport$8)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $0
+              (i32.add
+               (local.get $2)
+               (i32.const 8)
+              )
+             )
+            )
+            (local.get $6)
+           )
+           (local.set $14
+            (local.get $0)
+           )
+           (call $fimport$8)
+          )
+         )
+        )
+        (i32.store offset=12
+         (local.get $1)
+         (local.get $2)
+        )
+        (i32.store
+         (local.get $14)
+         (local.get $1)
+        )
+       )
+       (block
+        (local.set $7
+         (i32.load offset=24
+          (local.get $6)
+         )
+        )
+        (block $label$73
+         (if
+          (i32.eq
+           (local.tee $0
+            (i32.load offset=12
+             (local.get $6)
+            )
+           )
+           (local.get $6)
+          )
+          (block
+           (if
+            (local.tee $0
+             (i32.load
+              (local.tee $1
+               (i32.add
+                (local.tee $2
+                 (i32.add
+                  (local.get $6)
+                  (i32.const 16)
+                 )
+                )
+                (i32.const 4)
+               )
+              )
+             )
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (if
+             (i32.eqz
+              (local.tee $0
+               (i32.load
+                (local.get $2)
+               )
+              )
+             )
+             (block
+              (local.set $9
+               (i32.const 0)
+              )
+              (br $label$73)
+             )
+            )
+           )
+           (loop $label$78
+            (if
+             (local.tee $4
+              (i32.load
+               (local.tee $1
+                (i32.add
+                 (local.get $0)
+                 (i32.const 20)
+                )
+               )
+              )
+             )
+             (block
+              (local.set $0
+               (local.get $4)
+              )
+              (local.set $2
+               (local.get $1)
+              )
+              (br $label$78)
+             )
+            )
+            (if
+             (local.tee $4
+              (i32.load
+               (local.tee $1
+                (i32.add
+                 (local.get $0)
+                 (i32.const 16)
+                )
+               )
+              )
+             )
+             (block
+              (local.set $0
+               (local.get $4)
+              )
+              (local.set $2
+               (local.get $1)
+              )
+              (br $label$78)
+             )
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $2)
+             (i32.load
+              (i32.const 4192)
+             )
+            )
+            (call $fimport$8)
+            (block
+             (i32.store
+              (local.get $2)
+              (i32.const 0)
+             )
+             (local.set $9
+              (local.get $0)
+             )
+            )
+           )
+          )
+          (block
+           (if
+            (i32.lt_u
+             (local.tee $2
+              (i32.load offset=8
+               (local.get $6)
+              )
+             )
+             (i32.load
+              (i32.const 4192)
+             )
+            )
+            (call $fimport$8)
+           )
+           (if
+            (i32.ne
+             (i32.load
+              (local.tee $1
+               (i32.add
+                (local.get $2)
+                (i32.const 12)
+               )
+              )
+             )
+             (local.get $6)
+            )
+            (call $fimport$8)
+           )
+           (if
+            (i32.eq
+             (i32.load
+              (local.tee $4
+               (i32.add
+                (local.get $0)
+                (i32.const 8)
+               )
+              )
+             )
+             (local.get $6)
+            )
+            (block
+             (i32.store
+              (local.get $1)
+              (local.get $0)
+             )
+             (i32.store
+              (local.get $4)
+              (local.get $2)
+             )
+             (local.set $9
+              (local.get $0)
+             )
+            )
+            (call $fimport$8)
+           )
+          )
+         )
+        )
+        (if
+         (local.get $7)
+         (block
+          (if
+           (i32.eq
+            (local.get $6)
+            (i32.load
+             (local.tee $2
+              (i32.add
+               (i32.shl
+                (local.tee $0
+                 (i32.load offset=28
+                  (local.get $6)
+                 )
+                )
+                (i32.const 2)
+               )
+               (i32.const 4480)
+              )
+             )
+            )
+           )
+           (block
+            (i32.store
+             (local.get $2)
+             (local.get $9)
+            )
+            (if
+             (i32.eqz
+              (local.get $9)
+             )
+             (block
+              (i32.store
+               (i32.const 4180)
+               (i32.and
+                (i32.load
+                 (i32.const 4180)
+                )
+                (i32.xor
+                 (i32.shl
+                  (i32.const 1)
+                  (local.get $0)
+                 )
+                 (i32.const -1)
+                )
+               )
+              )
+              (br $label$61)
+             )
+            )
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.get $7)
+              (i32.load
+               (i32.const 4192)
+              )
+             )
+             (call $fimport$8)
+            )
+            (if
+             (i32.eq
+              (i32.load
+               (local.tee $0
+                (i32.add
+                 (local.get $7)
+                 (i32.const 16)
+                )
+               )
+              )
+              (local.get $6)
+             )
+             (i32.store
+              (local.get $0)
+              (local.get $9)
+             )
+             (i32.store offset=20
+              (local.get $7)
+              (local.get $9)
+             )
+            )
+            (br_if $label$61
+             (i32.eqz
+              (local.get $9)
+             )
+            )
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.get $9)
+            (local.tee $2
+             (i32.load
+              (i32.const 4192)
+             )
+            )
+           )
+           (call $fimport$8)
+          )
+          (i32.store offset=24
+           (local.get $9)
+           (local.get $7)
+          )
+          (if
+           (local.tee $0
+            (i32.load
+             (local.tee $1
+              (i32.add
+               (local.get $6)
+               (i32.const 16)
+              )
+             )
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $0)
+             (local.get $2)
+            )
+            (call $fimport$8)
+            (block
+             (i32.store offset=16
+              (local.get $9)
+              (local.get $0)
+             )
+             (i32.store offset=24
+              (local.get $0)
+              (local.get $9)
+             )
+            )
+           )
+          )
+          (if
+           (local.tee $0
+            (i32.load offset=4
+             (local.get $1)
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $0)
+             (i32.load
+              (i32.const 4192)
+             )
+            )
+            (call $fimport$8)
+            (block
+             (i32.store offset=20
+              (local.get $9)
+              (local.get $0)
+             )
+             (i32.store offset=24
+              (local.get $0)
+              (local.get $9)
+             )
+            )
+           )
+          )
+         )
+        )
+       )
+      )
+     )
+     (i32.store offset=4
+      (local.get $3)
+      (i32.or
+       (local.get $5)
+       (i32.const 1)
+      )
+     )
+     (i32.store
+      (i32.add
+       (local.get $3)
+       (local.get $5)
+      )
+      (local.get $5)
+     )
+     (if
+      (i32.eq
+       (local.get $3)
+       (i32.load
+        (i32.const 4196)
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 4184)
+        (local.get $5)
+       )
+       (return)
+      )
+      (local.set $2
+       (local.get $5)
+      )
+     )
+    )
+   )
+   (local.set $1
+    (i32.shr_u
+     (local.get $2)
+     (i32.const 3)
+    )
+   )
+   (if
+    (i32.lt_u
+     (local.get $2)
+     (i32.const 256)
+    )
+    (block
+     (local.set $0
+      (i32.add
+       (i32.shl
+        (i32.shl
+         (local.get $1)
+         (i32.const 1)
+        )
+        (i32.const 2)
+       )
+       (i32.const 4216)
+      )
+     )
+     (if
+      (i32.and
+       (local.tee $2
+        (i32.load
+         (i32.const 4176)
+        )
+       )
+       (local.tee $1
+        (i32.shl
+         (i32.const 1)
+         (local.get $1)
+        )
+       )
+      )
+      (if
+       (i32.lt_u
+        (local.tee $1
+         (i32.load
+          (local.tee $2
+           (i32.add
+            (local.get $0)
+            (i32.const 8)
+           )
+          )
+         )
+        )
+        (i32.load
+         (i32.const 4192)
+        )
+       )
+       (call $fimport$8)
+       (block
+        (local.set $15
+         (local.get $2)
+        )
+        (local.set $13
+         (local.get $1)
+        )
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 4176)
+        (i32.or
+         (local.get $2)
+         (local.get $1)
+        )
+       )
+       (local.set $15
+        (i32.add
+         (local.get $0)
+         (i32.const 8)
+        )
+       )
+       (local.set $13
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store
+      (local.get $15)
+      (local.get $3)
+     )
+     (i32.store offset=12
+      (local.get $13)
+      (local.get $3)
+     )
+     (i32.store offset=8
+      (local.get $3)
+      (local.get $13)
+     )
+     (i32.store offset=12
+      (local.get $3)
+      (local.get $0)
+     )
+     (return)
+    )
+   )
+   (local.set $0
+    (i32.add
+     (i32.shl
+      (local.tee $1
+       (if (result i32)
+        (local.tee $0
+         (i32.shr_u
+          (local.get $2)
+          (i32.const 8)
+         )
+        )
+        (if (result i32)
+         (i32.gt_u
+          (local.get $2)
+          (i32.const 16777215)
+         )
+         (i32.const 31)
+         (i32.or
+          (i32.and
+           (i32.shr_u
+            (local.get $2)
+            (i32.add
+             (local.tee $0
+              (i32.add
+               (i32.sub
+                (i32.const 14)
+                (i32.or
+                 (i32.or
+                  (local.tee $4
+                   (i32.and
+                    (i32.shr_u
+                     (i32.add
+                      (local.tee $1
+                       (i32.shl
+                        (local.get $0)
+                        (local.tee $0
+                         (i32.and
+                          (i32.shr_u
+                           (i32.add
+                            (local.get $0)
+                            (i32.const 1048320)
+                           )
+                           (i32.const 16)
+                          )
+                          (i32.const 8)
+                         )
+                        )
+                       )
+                      )
+                      (i32.const 520192)
+                     )
+                     (i32.const 16)
+                    )
+                    (i32.const 4)
+                   )
+                  )
+                  (local.get $0)
+                 )
+                 (local.tee $1
+                  (i32.and
+                   (i32.shr_u
+                    (i32.add
+                     (local.tee $0
+                      (i32.shl
+                       (local.get $1)
+                       (local.get $4)
+                      )
+                     )
+                     (i32.const 245760)
+                    )
+                    (i32.const 16)
+                   )
+                   (i32.const 2)
+                  )
+                 )
+                )
+               )
+               (i32.shr_u
+                (i32.shl
+                 (local.get $0)
+                 (local.get $1)
+                )
+                (i32.const 15)
+               )
+              )
+             )
+             (i32.const 7)
+            )
+           )
+           (i32.const 1)
+          )
+          (i32.shl
+           (local.get $0)
+           (i32.const 1)
+          )
+         )
+        )
+        (i32.const 0)
+       )
+      )
+      (i32.const 2)
+     )
+     (i32.const 4480)
+    )
+   )
+   (i32.store offset=28
+    (local.get $3)
+    (local.get $1)
+   )
+   (i32.store offset=20
+    (local.get $3)
+    (i32.const 0)
+   )
+   (i32.store offset=16
+    (local.get $3)
+    (i32.const 0)
+   )
+   (block $label$113
+    (if
+     (i32.and
+      (local.tee $4
+       (i32.load
+        (i32.const 4180)
+       )
+      )
+      (local.tee $5
+       (i32.shl
+        (i32.const 1)
+        (local.get $1)
+       )
+      )
+     )
+     (block
+      (local.set $0
+       (i32.load
+        (local.get $0)
+       )
+      )
+      (local.set $4
+       (i32.sub
+        (i32.const 25)
+        (i32.shr_u
+         (local.get $1)
+         (i32.const 1)
+        )
+       )
+      )
+      (local.set $1
+       (i32.shl
+        (local.get $2)
+        (if (result i32)
+         (i32.eq
+          (local.get $1)
+          (i32.const 31)
+         )
+         (i32.const 0)
+         (local.get $4)
+        )
+       )
+      )
+      (block $label$117
+       (block $label$118
+        (block $label$119
+         (loop $label$120
+          (br_if $label$118
+           (i32.eq
+            (i32.and
+             (i32.load offset=4
+              (local.get $0)
+             )
+             (i32.const -8)
+            )
+            (local.get $2)
+           )
+          )
+          (local.set $4
+           (i32.shl
+            (local.get $1)
+            (i32.const 1)
+           )
+          )
+          (br_if $label$119
+           (i32.eqz
+            (local.tee $5
+             (i32.load
+              (local.tee $1
+               (i32.add
+                (i32.add
+                 (local.get $0)
+                 (i32.const 16)
+                )
+                (i32.shl
+                 (i32.shr_u
+                  (local.get $1)
+                  (i32.const 31)
+                 )
+                 (i32.const 2)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (local.set $1
+           (local.get $4)
+          )
+          (local.set $0
+           (local.get $5)
+          )
+          (br $label$120)
+         )
+        )
+        (if
+         (i32.lt_u
+          (local.get $1)
+          (i32.load
+           (i32.const 4192)
+          )
+         )
+         (call $fimport$8)
+         (block
+          (i32.store
+           (local.get $1)
+           (local.get $3)
+          )
+          (i32.store offset=24
+           (local.get $3)
+           (local.get $0)
+          )
+          (i32.store offset=12
+           (local.get $3)
+           (local.get $3)
+          )
+          (i32.store offset=8
+           (local.get $3)
+           (local.get $3)
+          )
+          (br $label$113)
+         )
+        )
+        (br $label$117)
+       )
+       (if
+        (i32.and
+         (i32.ge_u
+          (local.tee $2
+           (i32.load
+            (local.tee $1
+             (i32.add
+              (local.get $0)
+              (i32.const 8)
+             )
+            )
+           )
+          )
+          (local.tee $4
+           (i32.load
+            (i32.const 4192)
+           )
+          )
+         )
+         (i32.ge_u
+          (local.get $0)
+          (local.get $4)
+         )
+        )
+        (block
+         (i32.store offset=12
+          (local.get $2)
+          (local.get $3)
+         )
+         (i32.store
+          (local.get $1)
+          (local.get $3)
+         )
+         (i32.store offset=8
+          (local.get $3)
+          (local.get $2)
+         )
+         (i32.store offset=12
+          (local.get $3)
+          (local.get $0)
+         )
+         (i32.store offset=24
+          (local.get $3)
+          (i32.const 0)
+         )
+        )
+        (call $fimport$8)
+       )
+      )
+     )
+     (block
+      (i32.store
+       (i32.const 4180)
+       (i32.or
+        (local.get $4)
+        (local.get $5)
+       )
+      )
+      (i32.store
+       (local.get $0)
+       (local.get $3)
+      )
+      (i32.store offset=24
+       (local.get $3)
+       (local.get $0)
+      )
+      (i32.store offset=12
+       (local.get $3)
+       (local.get $3)
+      )
+      (i32.store offset=8
+       (local.get $3)
+       (local.get $3)
+      )
+     )
+    )
+   )
+   (i32.store
+    (i32.const 4208)
+    (local.tee $0
+     (i32.add
+      (i32.load
+       (i32.const 4208)
+      )
+      (i32.const -1)
+     )
+    )
+   )
+   (if
+    (local.get $0)
+    (return)
+    (local.set $0
+     (i32.const 4632)
+    )
+   )
+   (loop $label$128
+    (local.set $0
+     (i32.add
+      (local.tee $2
+       (i32.load
+        (local.get $0)
+       )
+      )
+      (i32.const 8)
+     )
+    )
+    (br_if $label$128
+     (local.get $2)
+    )
+   )
+   (i32.store
+    (i32.const 4208)
+    (i32.const -1)
+   )
+  )
+ )
+ (func $39 (; 52 ;) (type $2) (param $0 i32) (result i32)
+  (local $1 i32)
+  (block $label$1 (result i32)
+   (if
+    (i32.eqz
+     (local.get $0)
+    )
+    (local.set $0
+     (i32.const 1)
+    )
+   )
+   (loop $label$3
+    (block $label$4
+     (if
+      (local.tee $1
+       (call $37
+        (local.get $0)
+       )
+      )
+      (block
+       (local.set $0
+        (local.get $1)
+       )
+       (br $label$4)
+      )
+     )
+     (if
+      (local.tee $1
+       (call $43)
+      )
+      (block
+       (call_indirect (type $1)
+        (i32.add
+         (i32.and
+          (local.get $1)
+          (i32.const 0)
+         )
+         (i32.const 8)
+        )
+       )
+       (br $label$3)
+      )
+      (local.set $0
+       (i32.const 0)
+      )
+     )
+    )
+   )
+   (local.get $0)
+  )
+ )
+ (func $40 (; 53 ;) (type $2) (param $0 i32) (result i32)
+  (call $39
+   (local.get $0)
+  )
+ )
+ (func $41 (; 54 ;) (type $3) (param $0 i32)
+  (call $38
+   (local.get $0)
+  )
+ )
+ (func $42 (; 55 ;) (type $3) (param $0 i32)
+  (call $41
+   (local.get $0)
+  )
+ )
+ (func $43 (; 56 ;) (type $4) (result i32)
+  (local $0 i32)
+  (block $label$1 (result i32)
+   (i32.store
+    (i32.const 4672)
+    (i32.add
+     (local.tee $0
+      (i32.load
+       (i32.const 4672)
+      )
+     )
+     (i32.const 0)
+    )
+   )
+   (local.get $0)
+  )
+ )
+ (func $44 (; 57 ;) (type $1)
+  (nop)
+ )
+ (func $45 (; 58 ;) (type $2) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (i32.add
+     (local.tee $2
+      (i32.load
+       (global.get $global$0)
+      )
+     )
+     (local.tee $0
+      (i32.and
+       (i32.add
+        (local.get $0)
+        (i32.const 15)
+       )
+       (i32.const -16)
+      )
+     )
+    )
+   )
+   (if
+    (i32.or
+     (i32.and
+      (i32.gt_s
+       (local.get $0)
+       (i32.const 0)
+      )
+      (i32.lt_s
+       (local.get $1)
+       (local.get $2)
+      )
+     )
+     (i32.lt_s
+      (local.get $1)
+      (i32.const 0)
+     )
+    )
+    (block
+     (drop
+      (call $fimport$6)
+     )
+     (call $fimport$11
+      (i32.const 12)
+     )
+     (return
+      (i32.const -1)
+     )
+    )
+   )
+   (i32.store
+    (global.get $global$0)
+    (local.get $1)
+   )
+   (if
+    (i32.gt_s
+     (local.get $1)
+     (call $fimport$5)
+    )
+    (if
+     (i32.eqz
+      (call $fimport$4)
+     )
+     (block
+      (call $fimport$11
+       (i32.const 12)
+      )
+      (i32.store
+       (global.get $global$0)
+       (local.get $2)
+      )
+      (return
+       (i32.const -1)
+      )
+     )
+    )
+   )
+   (local.get $2)
+  )
+ )
+ (func $46 (; 59 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (i32.add
+     (local.get $0)
+     (local.get $2)
+    )
+   )
+   (if
+    (i32.ge_s
+     (local.get $2)
+     (i32.const 20)
+    )
+    (block
+     (local.set $1
+      (i32.and
+       (local.get $1)
+       (i32.const 255)
+      )
+     )
+     (if
+      (local.tee $3
+       (i32.and
+        (local.get $0)
+        (i32.const 3)
+       )
+      )
+      (block
+       (local.set $3
+        (i32.sub
+         (i32.add
+          (local.get $0)
+          (i32.const 4)
+         )
+         (local.get $3)
+        )
+       )
+       (loop $label$4
+        (if
+         (i32.lt_s
+          (local.get $0)
+          (local.get $3)
+         )
+         (block
+          (i32.store8
+           (local.get $0)
+           (local.get $1)
+          )
+          (local.set $0
+           (i32.add
+            (local.get $0)
+            (i32.const 1)
+           )
+          )
+          (br $label$4)
+         )
+        )
+       )
+      )
+     )
+     (local.set $3
+      (i32.or
+       (i32.or
+        (i32.or
+         (local.get $1)
+         (i32.shl
+          (local.get $1)
+          (i32.const 8)
+         )
+        )
+        (i32.shl
+         (local.get $1)
+         (i32.const 16)
+        )
+       )
+       (i32.shl
+        (local.get $1)
+        (i32.const 24)
+       )
+      )
+     )
+     (local.set $5
+      (i32.and
+       (local.get $4)
+       (i32.const -4)
+      )
+     )
+     (loop $label$6
+      (if
+       (i32.lt_s
+        (local.get $0)
+        (local.get $5)
+       )
+       (block
+        (i32.store
+         (local.get $0)
+         (local.get $3)
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (i32.const 4)
+         )
+        )
+        (br $label$6)
+       )
+      )
+     )
+    )
+   )
+   (loop $label$8
+    (if
+     (i32.lt_s
+      (local.get $0)
+      (local.get $4)
+     )
+     (block
+      (i32.store8
+       (local.get $0)
+       (local.get $1)
+      )
+      (local.set $0
+       (i32.add
+        (local.get $0)
+        (i32.const 1)
+       )
+      )
+      (br $label$8)
+     )
+    )
+   )
+   (i32.sub
+    (local.get $0)
+    (local.get $2)
+   )
+  )
+ )
+ (func $47 (; 60 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (block $label$1 (result i32)
+   (if
+    (i32.ge_s
+     (local.get $2)
+     (i32.const 4096)
+    )
+    (return
+     (call $fimport$12
+      (local.get $0)
+      (local.get $1)
+      (local.get $2)
+     )
+    )
+   )
+   (local.set $3
+    (local.get $0)
+   )
+   (if
+    (i32.eq
+     (i32.and
+      (local.get $0)
+      (i32.const 3)
+     )
+     (i32.and
+      (local.get $1)
+      (i32.const 3)
+     )
+    )
+    (block
+     (loop $label$4
+      (if
+       (i32.and
+        (local.get $0)
+        (i32.const 3)
+       )
+       (block
+        (if
+         (i32.eqz
+          (local.get $2)
+         )
+         (return
+          (local.get $3)
+         )
+        )
+        (i32.store8
+         (local.get $0)
+         (i32.load8_s
+          (local.get $1)
+         )
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (i32.const 1)
+         )
+        )
+        (local.set $1
+         (i32.add
+          (local.get $1)
+          (i32.const 1)
+         )
+        )
+        (local.set $2
+         (i32.sub
+          (local.get $2)
+          (i32.const 1)
+         )
+        )
+        (br $label$4)
+       )
+      )
+     )
+     (loop $label$7
+      (if
+       (i32.ge_s
+        (local.get $2)
+        (i32.const 4)
+       )
+       (block
+        (i32.store
+         (local.get $0)
+         (i32.load
+          (local.get $1)
+         )
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (i32.const 4)
+         )
+        )
+        (local.set $1
+         (i32.add
+          (local.get $1)
+          (i32.const 4)
+         )
+        )
+        (local.set $2
+         (i32.sub
+          (local.get $2)
+          (i32.const 4)
+         )
+        )
+        (br $label$7)
+       )
+      )
+     )
+    )
+   )
+   (loop $label$9
+    (if
+     (i32.gt_s
+      (local.get $2)
+      (i32.const 0)
+     )
+     (block
+      (i32.store8
+       (local.get $0)
+       (i32.load8_s
+        (local.get $1)
+       )
+      )
+      (local.set $0
+       (i32.add
+        (local.get $0)
+        (i32.const 1)
+       )
+      )
+      (local.set $1
+       (i32.add
+        (local.get $1)
+        (i32.const 1)
+       )
+      )
+      (local.set $2
+       (i32.sub
+        (local.get $2)
+        (i32.const 1)
+       )
+      )
+      (br $label$9)
+     )
+    )
+   )
+   (local.get $3)
+  )
+ )
+ (func $48 (; 61 ;) (type $4) (result i32)
+  (i32.const 0)
+ )
+ (func $49 (; 62 ;) (type $6) (param $0 i32) (param $1 i32) (result i32)
+  (call_indirect (type $2)
+   (local.get $1)
+   (i32.add
+    (i32.and
+     (local.get $0)
+     (i32.const 1)
+    )
+    (i32.const 0)
+   )
+  )
+ )
+ (func $50 (; 63 ;) (type $12) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (result i32)
+  (call_indirect (type $0)
+   (local.get $1)
+   (local.get $2)
+   (local.get $3)
+   (i32.add
+    (i32.and
+     (local.get $0)
+     (i32.const 3)
+    )
+    (i32.const 2)
+   )
+  )
+ )
+ (func $51 (; 64 ;) (type $5) (param $0 i32) (param $1 i32)
+  (call_indirect (type $3)
+   (local.get $1)
+   (i32.add
+    (i32.and
+     (local.get $0)
+     (i32.const 1)
+    )
+    (i32.const 6)
+   )
+  )
+ )
+ (func $52 (; 65 ;) (type $3) (param $0 i32)
+  (call_indirect (type $1)
+   (i32.add
+    (i32.and
+     (local.get $0)
+     (i32.const 0)
+    )
+    (i32.const 8)
+   )
+  )
+ )
+ (func $53 (; 66 ;) (type $2) (param $0 i32) (result i32)
+  (block $label$1 (result i32)
+   (call $fimport$3
+    (i32.const 0)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $54 (; 67 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (block $label$1 (result i32)
+   (call $fimport$3
+    (i32.const 1)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $55 (; 68 ;) (type $3) (param $0 i32)
+  (call $fimport$3
+   (i32.const 2)
+  )
+ )
+ (func $56 (; 69 ;) (type $1)
+  (call $fimport$3
+   (i32.const 3)
+  )
+ )
+)
+
diff --git a/cranelift/wasmtests/embenchen_ifs.wat b/cranelift/wasmtests/embenchen_ifs.wat
new file mode 100644
index 0000000000..e5ada5702f
--- /dev/null
+++ b/cranelift/wasmtests/embenchen_ifs.wat
@@ -0,0 +1,15771 @@
+(module
+ (type $0 (func (param i32 i32 i32) (result i32)))
+ (type $1 (func (param i32) (result i32)))
+ (type $2 (func (param i32)))
+ (type $3 (func (result i32)))
+ (type $4 (func (param i32 i32) (result i32)))
+ (type $5 (func (param i32 i32)))
+ (type $6 (func))
+ (type $7 (func (param i32 i32 i32 i32 i32) (result i32)))
+ (type $8 (func (param i32 i32 i32)))
+ (type $9 (func (param i64 i32) (result i32)))
+ (type $10 (func (param i32 i32 i32 i32 i32)))
+ (type $11 (func (param f64 i32) (result f64)))
+ (type $12 (func (param i32 i32 i32 i32) (result i32)))
+ (import "env" "memory" (memory $16 2048 2048))
+ (data (i32.const 1024) "\04\04\00\00\05")
+ (data (i32.const 1040) "\01")
+ (data (i32.const 1064) "\01\00\00\00\02\00\00\00,\10\00\00\00\04")
+ (data (i32.const 1088) "\01")
+ (data (i32.const 1103) "\n\ff\ff\ff\ff")
+ (data (i32.const 1140) "error: %d\\n\00ok\00\11\00\n\00\11\11\11\00\00\00\00\05\00\00\00\00\00\00\t\00\00\00\00\0b")
+ (data (i32.const 1187) "\11\00\0f\n\11\11\11\03\n\07\00\01\13\t\0b\0b\00\00\t\06\0b\00\00\0b\00\06\11\00\00\00\11\11\11")
+ (data (i32.const 1236) "\0b")
+ (data (i32.const 1245) "\11\00\n\n\11\11\11\00\n\00\00\02\00\t\0b\00\00\00\t\00\0b\00\00\0b")
+ (data (i32.const 1294) "\0c")
+ (data (i32.const 1306) "\0c\00\00\00\00\0c\00\00\00\00\t\0c\00\00\00\00\00\0c\00\00\0c")
+ (data (i32.const 1352) "\0e")
+ (data (i32.const 1364) "\0d\00\00\00\04\0d\00\00\00\00\t\0e\00\00\00\00\00\0e\00\00\0e")
+ (data (i32.const 1410) "\10")
+ (data (i32.const 1422) "\0f\00\00\00\00\0f\00\00\00\00\t\10\00\00\00\00\00\10\00\00\10\00\00\12\00\00\00\12\12\12")
+ (data (i32.const 1477) "\12\00\00\00\12\12\12\00\00\00\00\00\00\t")
+ (data (i32.const 1526) "\0b")
+ (data (i32.const 1538) "\n\00\00\00\00\n\00\00\00\00\t\0b\00\00\00\00\00\0b\00\00\0b")
+ (data (i32.const 1584) "\0c")
+ (data (i32.const 1596) "\0c\00\00\00\00\0c\00\00\00\00\t\0c\00\00\00\00\00\0c\00\00\0c\00\000123456789ABCDEF-+   0X0x\00(null)\00-0X+0X 0X-0x+0x 0x\00inf\00INF\00nan\00NAN\00.\00T!\"\19\0d\01\02\03\11K\1c\0c\10\04\0b\1d\12\1e\'hnopqb \05\06\0f\13\14\15\1a\08\16\07($\17\18\t\n\0e\1b\1f%#\83\82}&*+<=>?CGJMXYZ[\\]^_`acdefgijklrstyz{|\00Illegal byte sequence\00Domain error\00Result not representable\00Not a tty\00Permission denied\00Operation not permitted\00No such file or directory\00No such process\00File exists\00Value too large for data type\00No space left on device\00Out of memory\00Resource busy\00Interrupted system call\00Resource temporarily unavailable\00Invalid seek\00Cross-device link\00Read-only file system\00Directory not empty\00Connection reset by peer\00Operation timed out\00Connection refused\00Host is down\00Host is unreachable\00Address in use\00Broken pipe\00I/O error\00No such device or address\00Block device required\00No such device\00Not a directory\00Is a directory\00Text file busy\00Exec format error\00Invalid argument\00Argument list too long\00Symbolic link loop\00Filename too long\00Too many open files in system\00No file descriptors available\00Bad file descriptor\00No child process\00Bad address\00File too large\00Too many links\00No locks available\00Resource deadlock would occur\00State not recoverable\00Previous owner died\00Operation canceled\00Function not implemented\00No message of desired type\00Identifier removed\00Device not a stream\00No data available\00Device timeout\00Out of streams resources\00Link has been severed\00Protocol error\00Bad message\00File descriptor in bad state\00Not a socket\00Destination address required\00Message too large\00Protocol wrong type for socket\00Protocol not available\00Protocol not supported\00Socket type not supported\00Not supported\00Protocol family not supported\00Address family not supported by protocol\00Address not available\00Network is down\00Network unreachable\00Connection reset by network\00Connection aborted\00No buffer space available\00Socket is connected\00Socket not connected\00Cannot send after socket shutdown\00Operation already in progress\00Operation in progress\00Stale file handle\00Remote I/O error\00Quota exceeded\00No medium found\00Wrong medium type\00No error information")
+ (import "env" "table" (table $timport$17 8 8 funcref))
+ (elem (global.get $gimport$19) $47 $9 $48 $14 $10 $15 $49 $16)
+ (import "env" "DYNAMICTOP_PTR" (global $gimport$0 i32))
+ (import "env" "STACKTOP" (global $gimport$1 i32))
+ (import "env" "STACK_MAX" (global $gimport$2 i32))
+ (import "env" "memoryBase" (global $gimport$18 i32))
+ (import "env" "tableBase" (global $gimport$19 i32))
+ (import "env" "abort" (func $fimport$3 (param i32)))
+ (import "env" "enlargeMemory" (func $fimport$4 (result i32)))
+ (import "env" "getTotalMemory" (func $fimport$5 (result i32)))
+ (import "env" "abortOnCannotGrowMemory" (func $fimport$6 (result i32)))
+ (import "env" "_pthread_cleanup_pop" (func $fimport$7 (param i32)))
+ (import "env" "___syscall6" (func $fimport$8 (param i32 i32) (result i32)))
+ (import "env" "_pthread_cleanup_push" (func $fimport$9 (param i32 i32)))
+ (import "env" "_abort" (func $fimport$10))
+ (import "env" "___setErrNo" (func $fimport$11 (param i32)))
+ (import "env" "_emscripten_memcpy_big" (func $fimport$12 (param i32 i32 i32) (result i32)))
+ (import "env" "___syscall54" (func $fimport$13 (param i32 i32) (result i32)))
+ (import "env" "___syscall140" (func $fimport$14 (param i32 i32) (result i32)))
+ (import "env" "___syscall146" (func $fimport$15 (param i32 i32) (result i32)))
+ (global $global$0 (mut i32) (global.get $gimport$0))
+ (global $global$1 (mut i32) (global.get $gimport$1))
+ (global $global$2 (mut i32) (global.get $gimport$2))
+ (global $global$3 (mut i32) (i32.const 0))
+ (global $global$4 (mut i32) (i32.const 0))
+ (global $global$5 (mut i32) (i32.const 0))
+ (export "_sbrk" (func $40))
+ (export "_free" (func $38))
+ (export "_main" (func $8))
+ (export "_pthread_self" (func $43))
+ (export "_memset" (func $41))
+ (export "_malloc" (func $37))
+ (export "_memcpy" (func $42))
+ (export "___errno_location" (func $12))
+ (export "runPostSets" (func $39))
+ (export "stackAlloc" (func $0))
+ (export "stackSave" (func $1))
+ (export "stackRestore" (func $2))
+ (export "establishStackSpace" (func $3))
+ (export "setThrew" (func $4))
+ (export "setTempRet0" (func $5))
+ (export "getTempRet0" (func $6))
+ (export "dynCall_ii" (func $44))
+ (export "dynCall_iiii" (func $45))
+ (export "dynCall_vi" (func $46))
+ (func $0 (; 13 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (local.get $0)
+    )
+   )
+   (global.set $global$1
+    (i32.and
+     (i32.add
+      (global.get $global$1)
+      (i32.const 15)
+     )
+     (i32.const -16)
+    )
+   )
+   (local.get $1)
+  )
+ )
+ (func $1 (; 14 ;) (type $3) (result i32)
+  (global.get $global$1)
+ )
+ (func $2 (; 15 ;) (type $2) (param $0 i32)
+  (global.set $global$1
+   (local.get $0)
+  )
+ )
+ (func $3 (; 16 ;) (type $5) (param $0 i32) (param $1 i32)
+  (block $label$1
+   (global.set $global$1
+    (local.get $0)
+   )
+   (global.set $global$2
+    (local.get $1)
+   )
+  )
+ )
+ (func $4 (; 17 ;) (type $5) (param $0 i32) (param $1 i32)
+  (if
+   (i32.eqz
+    (global.get $global$3)
+   )
+   (block
+    (global.set $global$3
+     (local.get $0)
+    )
+    (global.set $global$4
+     (local.get $1)
+    )
+   )
+  )
+ )
+ (func $5 (; 18 ;) (type $2) (param $0 i32)
+  (global.set $global$5
+   (local.get $0)
+  )
+ )
+ (func $6 (; 19 ;) (type $3) (result i32)
+  (global.get $global$5)
+ )
+ (func $7 (; 20 ;) (type $3) (result i32)
+  (local $0 i32)
+  (block $label$1 (result i32)
+   (i32.store
+    (i32.const 3584)
+    (i32.add
+     (local.tee $0
+      (i32.load
+       (i32.const 3584)
+      )
+     )
+     (i32.const 1)
+    )
+   )
+   (i32.and
+    (local.get $0)
+    (i32.const 16384)
+   )
+  )
+ )
+ (func $8 (; 21 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (local.set $2
+    (local.get $4)
+   )
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (i32.le_s
+       (local.get $0)
+       (i32.const 1)
+      )
+     )
+     (block $label$4
+      (block $label$5
+       (block $label$6
+        (block $label$7
+         (block $label$8
+          (block $label$9
+           (block $label$10
+            (br_table $label$5 $label$10 $label$8 $label$9 $label$7 $label$6 $label$4
+             (i32.sub
+              (local.tee $0
+               (i32.load8_s
+                (i32.load offset=4
+                 (local.get $1)
+                )
+               )
+              )
+              (i32.const 48)
+             )
+            )
+           )
+           (local.set $3
+            (i32.const 75)
+           )
+           (br $label$2)
+          )
+          (br $label$3)
+         )
+         (local.set $3
+          (i32.const 625)
+         )
+         (br $label$2)
+        )
+        (local.set $3
+         (i32.const 6250)
+        )
+        (br $label$2)
+       )
+       (local.set $3
+        (i32.const 12500)
+       )
+       (br $label$2)
+      )
+      (global.set $global$1
+       (local.get $4)
+      )
+      (return
+       (i32.const 0)
+      )
+     )
+     (i32.store
+      (local.get $2)
+      (i32.add
+       (local.get $0)
+       (i32.const -48)
+      )
+     )
+     (drop
+      (call $34
+       (i32.const 1140)
+       (local.get $2)
+      )
+     )
+     (global.set $global$1
+      (local.get $4)
+     )
+     (return
+      (i32.const -1)
+     )
+    )
+    (local.set $3
+     (i32.const 1250)
+    )
+   )
+   (local.set $1
+    (i32.const 0)
+   )
+   (local.set $0
+    (i32.const 0)
+   )
+   (loop $label$11
+    (local.set $2
+     (i32.const 0)
+    )
+    (loop $label$12
+     (local.set $0
+      (block $label$13 (result i32)
+       (block $label$14
+        (br_if $label$14
+         (i32.eqz
+          (call $7)
+         )
+        )
+        (br_if $label$14
+         (i32.eqz
+          (call $7)
+         )
+        )
+        (br $label$13
+         (i32.add
+          (local.get $0)
+          (i32.const 17)
+         )
+        )
+       )
+       (i32.add
+        (local.get $0)
+        (i32.const 19)
+       )
+      )
+     )
+     (block $label$15
+      (block $label$16
+       (br_if $label$16
+        (call $7)
+       )
+       (br_if $label$16
+        (call $7)
+       )
+       (br $label$15)
+      )
+      (local.set $0
+       (i32.add
+        (local.get $0)
+        (i32.const 23)
+       )
+      )
+     )
+     (br_if $label$12
+      (i32.lt_s
+       (local.tee $2
+        (i32.add
+         (local.get $2)
+         (i32.const 1)
+        )
+       )
+       (local.get $3)
+      )
+     )
+    )
+    (br_if $label$11
+     (i32.ne
+      (local.tee $1
+       (i32.add
+        (local.get $1)
+        (i32.const 1)
+       )
+      )
+      (i32.const 27000)
+     )
+    )
+   )
+   (drop
+    (call $35
+     (i32.const 1152)
+    )
+   )
+   (global.set $global$1
+    (local.get $4)
+   )
+   (local.get $0)
+  )
+ )
+ (func $9 (; 22 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (i32.store
+    (local.tee $2
+     (local.get $1)
+    )
+    (i32.load offset=60
+     (local.get $0)
+    )
+   )
+   (local.set $0
+    (call $11
+     (call $fimport$8
+      (i32.const 6)
+      (local.get $2)
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $1)
+   )
+   (local.get $0)
+  )
+ )
+ (func $10 (; 23 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 32)
+    )
+   )
+   (i32.store
+    (local.tee $3
+     (local.get $4)
+    )
+    (i32.load offset=60
+     (local.get $0)
+    )
+   )
+   (i32.store offset=4
+    (local.get $3)
+    (i32.const 0)
+   )
+   (i32.store offset=8
+    (local.get $3)
+    (local.get $1)
+   )
+   (i32.store offset=12
+    (local.get $3)
+    (local.tee $0
+     (i32.add
+      (local.get $4)
+      (i32.const 20)
+     )
+    )
+   )
+   (i32.store offset=16
+    (local.get $3)
+    (local.get $2)
+   )
+   (local.set $0
+    (if (result i32)
+     (i32.lt_s
+      (call $11
+       (call $fimport$14
+        (i32.const 140)
+        (local.get $3)
+       )
+      )
+      (i32.const 0)
+     )
+     (block (result i32)
+      (i32.store
+       (local.get $0)
+       (i32.const -1)
+      )
+      (i32.const -1)
+     )
+     (i32.load
+      (local.get $0)
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $4)
+   )
+   (local.get $0)
+  )
+ )
+ (func $11 (; 24 ;) (type $1) (param $0 i32) (result i32)
+  (if (result i32)
+   (i32.gt_u
+    (local.get $0)
+    (i32.const -4096)
+   )
+   (block (result i32)
+    (i32.store
+     (call $12)
+     (i32.sub
+      (i32.const 0)
+      (local.get $0)
+     )
+    )
+    (i32.const -1)
+   )
+   (local.get $0)
+  )
+ )
+ (func $12 (; 25 ;) (type $3) (result i32)
+  (i32.const 3632)
+ )
+ (func $13 (; 26 ;) (type $2) (param $0 i32)
+  (nop)
+ )
+ (func $14 (; 27 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 80)
+    )
+   )
+   (local.set $3
+    (local.get $4)
+   )
+   (local.set $5
+    (i32.add
+     (local.get $4)
+     (i32.const 12)
+    )
+   )
+   (i32.store offset=36
+    (local.get $0)
+    (i32.const 3)
+   )
+   (if
+    (i32.eqz
+     (i32.and
+      (i32.load
+       (local.get $0)
+      )
+      (i32.const 64)
+     )
+    )
+    (block
+     (i32.store
+      (local.get $3)
+      (i32.load offset=60
+       (local.get $0)
+      )
+     )
+     (i32.store offset=4
+      (local.get $3)
+      (i32.const 21505)
+     )
+     (i32.store offset=8
+      (local.get $3)
+      (local.get $5)
+     )
+     (if
+      (call $fimport$13
+       (i32.const 54)
+       (local.get $3)
+      )
+      (i32.store8 offset=75
+       (local.get $0)
+       (i32.const -1)
+      )
+     )
+    )
+   )
+   (local.set $0
+    (call $15
+     (local.get $0)
+     (local.get $1)
+     (local.get $2)
+    )
+   )
+   (global.set $global$1
+    (local.get $4)
+   )
+   (local.get $0)
+  )
+ )
+ (func $15 (; 28 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (block $label$1 (result i32)
+   (local.set $8
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 48)
+    )
+   )
+   (local.set $9
+    (i32.add
+     (local.get $8)
+     (i32.const 16)
+    )
+   )
+   (local.set $10
+    (local.get $8)
+   )
+   (i32.store
+    (local.tee $3
+     (i32.add
+      (local.get $8)
+      (i32.const 32)
+     )
+    )
+    (local.tee $4
+     (i32.load
+      (local.tee $6
+       (i32.add
+        (local.get $0)
+        (i32.const 28)
+       )
+      )
+     )
+    )
+   )
+   (i32.store offset=4
+    (local.get $3)
+    (local.tee $5
+     (i32.sub
+      (i32.load
+       (local.tee $11
+        (i32.add
+         (local.get $0)
+         (i32.const 20)
+        )
+       )
+      )
+      (local.get $4)
+     )
+    )
+   )
+   (i32.store offset=8
+    (local.get $3)
+    (local.get $1)
+   )
+   (i32.store offset=12
+    (local.get $3)
+    (local.get $2)
+   )
+   (local.set $13
+    (i32.add
+     (local.get $0)
+     (i32.const 60)
+    )
+   )
+   (local.set $14
+    (i32.add
+     (local.get $0)
+     (i32.const 44)
+    )
+   )
+   (local.set $1
+    (local.get $3)
+   )
+   (local.set $4
+    (i32.const 2)
+   )
+   (local.set $12
+    (i32.add
+     (local.get $5)
+     (local.get $2)
+    )
+   )
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (loop $label$5
+       (if
+        (i32.load
+         (i32.const 3588)
+        )
+        (block
+         (call $fimport$9
+          (i32.const 1)
+          (local.get $0)
+         )
+         (i32.store
+          (local.get $10)
+          (i32.load
+           (local.get $13)
+          )
+         )
+         (i32.store offset=4
+          (local.get $10)
+          (local.get $1)
+         )
+         (i32.store offset=8
+          (local.get $10)
+          (local.get $4)
+         )
+         (local.set $3
+          (call $11
+           (call $fimport$15
+            (i32.const 146)
+            (local.get $10)
+           )
+          )
+         )
+         (call $fimport$7
+          (i32.const 0)
+         )
+        )
+        (block
+         (i32.store
+          (local.get $9)
+          (i32.load
+           (local.get $13)
+          )
+         )
+         (i32.store offset=4
+          (local.get $9)
+          (local.get $1)
+         )
+         (i32.store offset=8
+          (local.get $9)
+          (local.get $4)
+         )
+         (local.set $3
+          (call $11
+           (call $fimport$15
+            (i32.const 146)
+            (local.get $9)
+           )
+          )
+         )
+        )
+       )
+       (br_if $label$4
+        (i32.eq
+         (local.get $12)
+         (local.get $3)
+        )
+       )
+       (br_if $label$3
+        (i32.lt_s
+         (local.get $3)
+         (i32.const 0)
+        )
+       )
+       (local.set $5
+        (if (result i32)
+         (i32.gt_u
+          (local.get $3)
+          (local.tee $5
+           (i32.load offset=4
+            (local.get $1)
+           )
+          )
+         )
+         (block (result i32)
+          (i32.store
+           (local.get $6)
+           (local.tee $7
+            (i32.load
+             (local.get $14)
+            )
+           )
+          )
+          (i32.store
+           (local.get $11)
+           (local.get $7)
+          )
+          (local.set $7
+           (i32.load offset=12
+            (local.get $1)
+           )
+          )
+          (local.set $1
+           (i32.add
+            (local.get $1)
+            (i32.const 8)
+           )
+          )
+          (local.set $4
+           (i32.add
+            (local.get $4)
+            (i32.const -1)
+           )
+          )
+          (i32.sub
+           (local.get $3)
+           (local.get $5)
+          )
+         )
+         (if (result i32)
+          (i32.eq
+           (local.get $4)
+           (i32.const 2)
+          )
+          (block (result i32)
+           (i32.store
+            (local.get $6)
+            (i32.add
+             (i32.load
+              (local.get $6)
+             )
+             (local.get $3)
+            )
+           )
+           (local.set $7
+            (local.get $5)
+           )
+           (local.set $4
+            (i32.const 2)
+           )
+           (local.get $3)
+          )
+          (block (result i32)
+           (local.set $7
+            (local.get $5)
+           )
+           (local.get $3)
+          )
+         )
+        )
+       )
+       (i32.store
+        (local.get $1)
+        (i32.add
+         (i32.load
+          (local.get $1)
+         )
+         (local.get $5)
+        )
+       )
+       (i32.store offset=4
+        (local.get $1)
+        (i32.sub
+         (local.get $7)
+         (local.get $5)
+        )
+       )
+       (local.set $12
+        (i32.sub
+         (local.get $12)
+         (local.get $3)
+        )
+       )
+       (br $label$5)
+      )
+     )
+     (i32.store offset=16
+      (local.get $0)
+      (i32.add
+       (local.tee $1
+        (i32.load
+         (local.get $14)
+        )
+       )
+       (i32.load offset=48
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store
+      (local.get $6)
+      (local.get $1)
+     )
+     (i32.store
+      (local.get $11)
+      (local.get $1)
+     )
+     (br $label$2)
+    )
+    (i32.store offset=16
+     (local.get $0)
+     (i32.const 0)
+    )
+    (i32.store
+     (local.get $6)
+     (i32.const 0)
+    )
+    (i32.store
+     (local.get $11)
+     (i32.const 0)
+    )
+    (i32.store
+     (local.get $0)
+     (i32.or
+      (i32.load
+       (local.get $0)
+      )
+      (i32.const 32)
+     )
+    )
+    (local.set $2
+     (if (result i32)
+      (i32.eq
+       (local.get $4)
+       (i32.const 2)
+      )
+      (i32.const 0)
+      (i32.sub
+       (local.get $2)
+       (i32.load offset=4
+        (local.get $1)
+       )
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $8)
+   )
+   (local.get $2)
+  )
+ )
+ (func $16 (; 29 ;) (type $2) (param $0 i32)
+  (if
+   (i32.eqz
+    (i32.load offset=68
+     (local.get $0)
+    )
+   )
+   (call $13
+    (local.get $0)
+   )
+  )
+ )
+ (func $17 (; 30 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $5
+    (i32.and
+     (local.get $1)
+     (i32.const 255)
+    )
+   )
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (if
+       (i32.and
+        (local.tee $4
+         (i32.ne
+          (local.get $2)
+          (i32.const 0)
+         )
+        )
+        (i32.ne
+         (i32.and
+          (local.get $0)
+          (i32.const 3)
+         )
+         (i32.const 0)
+        )
+       )
+       (block
+        (local.set $4
+         (i32.and
+          (local.get $1)
+          (i32.const 255)
+         )
+        )
+        (local.set $3
+         (local.get $2)
+        )
+        (local.set $2
+         (local.get $0)
+        )
+        (loop $label$6
+         (if
+          (i32.eq
+           (i32.load8_s
+            (local.get $2)
+           )
+           (i32.shr_s
+            (i32.shl
+             (local.get $4)
+             (i32.const 24)
+            )
+            (i32.const 24)
+           )
+          )
+          (block
+           (local.set $0
+            (local.get $3)
+           )
+           (br $label$3)
+          )
+         )
+         (br_if $label$6
+          (i32.and
+           (local.tee $0
+            (i32.ne
+             (local.tee $3
+              (i32.add
+               (local.get $3)
+               (i32.const -1)
+              )
+             )
+             (i32.const 0)
+            )
+           )
+           (i32.ne
+            (i32.and
+             (local.tee $2
+              (i32.add
+               (local.get $2)
+               (i32.const 1)
+              )
+             )
+             (i32.const 3)
+            )
+            (i32.const 0)
+           )
+          )
+         )
+         (br $label$4)
+        )
+       )
+       (block
+        (local.set $3
+         (local.get $2)
+        )
+        (local.set $2
+         (local.get $0)
+        )
+        (local.set $0
+         (local.get $4)
+        )
+       )
+      )
+     )
+     (if
+      (local.get $0)
+      (block
+       (local.set $0
+        (local.get $3)
+       )
+       (br $label$3)
+      )
+      (local.set $0
+       (i32.const 0)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.ne
+      (i32.load8_s
+       (local.get $2)
+      )
+      (i32.shr_s
+       (i32.shl
+        (local.tee $1
+         (i32.and
+          (local.get $1)
+          (i32.const 255)
+         )
+        )
+        (i32.const 24)
+       )
+       (i32.const 24)
+      )
+     )
+     (block
+      (local.set $3
+       (i32.mul
+        (local.get $5)
+        (i32.const 16843009)
+       )
+      )
+      (block $label$12
+       (block $label$13
+        (br_if $label$13
+         (i32.le_u
+          (local.get $0)
+          (i32.const 3)
+         )
+        )
+        (loop $label$14
+         (if
+          (i32.eqz
+           (i32.and
+            (i32.xor
+             (i32.and
+              (local.tee $4
+               (i32.xor
+                (i32.load
+                 (local.get $2)
+                )
+                (local.get $3)
+               )
+              )
+              (i32.const -2139062144)
+             )
+             (i32.const -2139062144)
+            )
+            (i32.add
+             (local.get $4)
+             (i32.const -16843009)
+            )
+           )
+          )
+          (block
+           (local.set $2
+            (i32.add
+             (local.get $2)
+             (i32.const 4)
+            )
+           )
+           (br_if $label$14
+            (i32.gt_u
+             (local.tee $0
+              (i32.add
+               (local.get $0)
+               (i32.const -4)
+              )
+             )
+             (i32.const 3)
+            )
+           )
+           (br $label$13)
+          )
+         )
+        )
+        (br $label$12)
+       )
+       (if
+        (i32.eqz
+         (local.get $0)
+        )
+        (block
+         (local.set $0
+          (i32.const 0)
+         )
+         (br $label$2)
+        )
+       )
+      )
+      (loop $label$17
+       (br_if $label$2
+        (i32.eq
+         (i32.load8_s
+          (local.get $2)
+         )
+         (i32.shr_s
+          (i32.shl
+           (local.get $1)
+           (i32.const 24)
+          )
+          (i32.const 24)
+         )
+        )
+       )
+       (local.set $2
+        (i32.add
+         (local.get $2)
+         (i32.const 1)
+        )
+       )
+       (br_if $label$17
+        (local.tee $0
+         (i32.add
+          (local.get $0)
+          (i32.const -1)
+         )
+        )
+       )
+       (local.set $0
+        (i32.const 0)
+       )
+      )
+     )
+    )
+   )
+   (if (result i32)
+    (local.get $0)
+    (local.get $2)
+    (i32.const 0)
+   )
+  )
+ )
+ (func $18 (; 31 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 224)
+    )
+   )
+   (local.set $5
+    (i32.add
+     (local.get $4)
+     (i32.const 136)
+    )
+   )
+   (i64.store align=4
+    (local.tee $3
+     (i32.add
+      (local.get $4)
+      (i32.const 80)
+     )
+    )
+    (i64.const 0)
+   )
+   (i64.store offset=8 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i64.store offset=16 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i64.store offset=24 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i64.store offset=32 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i32.store
+    (local.tee $6
+     (i32.add
+      (local.get $4)
+      (i32.const 120)
+     )
+    )
+    (i32.load
+     (local.get $2)
+    )
+   )
+   (if
+    (i32.lt_s
+     (call $19
+      (i32.const 0)
+      (local.get $1)
+      (local.get $6)
+      (local.tee $2
+       (local.get $4)
+      )
+      (local.get $3)
+     )
+     (i32.const 0)
+    )
+    (local.set $1
+     (i32.const -1)
+    )
+    (block
+     (local.set $12
+      (if (result i32)
+       (i32.gt_s
+        (i32.load offset=76
+         (local.get $0)
+        )
+        (i32.const -1)
+       )
+       (call $20
+        (local.get $0)
+       )
+       (i32.const 0)
+      )
+     )
+     (local.set $7
+      (i32.load
+       (local.get $0)
+      )
+     )
+     (if
+      (i32.lt_s
+       (i32.load8_s offset=74
+        (local.get $0)
+       )
+       (i32.const 1)
+      )
+      (i32.store
+       (local.get $0)
+       (i32.and
+        (local.get $7)
+        (i32.const -33)
+       )
+      )
+     )
+     (if
+      (i32.load
+       (local.tee $8
+        (i32.add
+         (local.get $0)
+         (i32.const 48)
+        )
+       )
+      )
+      (local.set $1
+       (call $19
+        (local.get $0)
+        (local.get $1)
+        (local.get $6)
+        (local.get $2)
+        (local.get $3)
+       )
+      )
+      (block
+       (local.set $10
+        (i32.load
+         (local.tee $9
+          (i32.add
+           (local.get $0)
+           (i32.const 44)
+          )
+         )
+        )
+       )
+       (i32.store
+        (local.get $9)
+        (local.get $5)
+       )
+       (i32.store
+        (local.tee $13
+         (i32.add
+          (local.get $0)
+          (i32.const 28)
+         )
+        )
+        (local.get $5)
+       )
+       (i32.store
+        (local.tee $11
+         (i32.add
+          (local.get $0)
+          (i32.const 20)
+         )
+        )
+        (local.get $5)
+       )
+       (i32.store
+        (local.get $8)
+        (i32.const 80)
+       )
+       (i32.store
+        (local.tee $14
+         (i32.add
+          (local.get $0)
+          (i32.const 16)
+         )
+        )
+        (i32.add
+         (local.get $5)
+         (i32.const 80)
+        )
+       )
+       (local.set $1
+        (call $19
+         (local.get $0)
+         (local.get $1)
+         (local.get $6)
+         (local.get $2)
+         (local.get $3)
+        )
+       )
+       (if
+        (local.get $10)
+        (block
+         (drop
+          (call_indirect (type $0)
+           (local.get $0)
+           (i32.const 0)
+           (i32.const 0)
+           (i32.add
+            (i32.and
+             (i32.load offset=36
+              (local.get $0)
+             )
+             (i32.const 3)
+            )
+            (i32.const 2)
+           )
+          )
+         )
+         (if
+          (i32.eqz
+           (i32.load
+            (local.get $11)
+           )
+          )
+          (local.set $1
+           (i32.const -1)
+          )
+         )
+         (i32.store
+          (local.get $9)
+          (local.get $10)
+         )
+         (i32.store
+          (local.get $8)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $14)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $13)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $11)
+          (i32.const 0)
+         )
+        )
+       )
+      )
+     )
+     (i32.store
+      (local.get $0)
+      (i32.or
+       (local.tee $2
+        (i32.load
+         (local.get $0)
+        )
+       )
+       (i32.and
+        (local.get $7)
+        (i32.const 32)
+       )
+      )
+     )
+     (if
+      (local.get $12)
+      (call $13
+       (local.get $0)
+      )
+     )
+     (if
+      (i32.and
+       (local.get $2)
+       (i32.const 32)
+      )
+      (local.set $1
+       (i32.const -1)
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $4)
+   )
+   (local.get $1)
+  )
+ )
+ (func $19 (; 32 ;) (type $7) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (param $4 i32) (result i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (local $16 i32)
+  (local $17 i32)
+  (local $18 i32)
+  (local $19 i32)
+  (local $20 i32)
+  (local $21 i32)
+  (local $22 i32)
+  (local $23 i32)
+  (local $24 i32)
+  (local $25 i32)
+  (local $26 i32)
+  (local $27 i32)
+  (local $28 i32)
+  (local $29 i32)
+  (local $30 i32)
+  (local $31 i32)
+  (local $32 i32)
+  (local $33 i32)
+  (local $34 i32)
+  (local $35 i32)
+  (local $36 i32)
+  (local $37 i32)
+  (local $38 i32)
+  (local $39 i32)
+  (local $40 i32)
+  (local $41 i32)
+  (local $42 i32)
+  (local $43 i32)
+  (local $44 i32)
+  (local $45 i32)
+  (local $46 i32)
+  (local $47 i32)
+  (local $48 i32)
+  (local $49 i32)
+  (local $50 i64)
+  (local $51 i64)
+  (local $52 f64)
+  (local $53 f64)
+  (block $label$1 (result i32)
+   (local.set $23
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 624)
+    )
+   )
+   (local.set $20
+    (i32.add
+     (local.get $23)
+     (i32.const 16)
+    )
+   )
+   (local.set $16
+    (local.get $23)
+   )
+   (local.set $36
+    (i32.add
+     (local.get $23)
+     (i32.const 528)
+    )
+   )
+   (local.set $30
+    (i32.ne
+     (local.get $0)
+     (i32.const 0)
+    )
+   )
+   (local.set $38
+    (local.tee $21
+     (i32.add
+      (local.tee $17
+       (i32.add
+        (local.get $23)
+        (i32.const 536)
+       )
+      )
+      (i32.const 40)
+     )
+    )
+   )
+   (local.set $39
+    (i32.add
+     (local.get $17)
+     (i32.const 39)
+    )
+   )
+   (local.set $42
+    (i32.add
+     (local.tee $37
+      (i32.add
+       (local.get $23)
+       (i32.const 8)
+      )
+     )
+     (i32.const 4)
+    )
+   )
+   (local.set $43
+    (i32.sub
+     (i32.const 0)
+     (local.tee $27
+      (local.tee $19
+       (i32.add
+        (local.get $23)
+        (i32.const 588)
+       )
+      )
+     )
+    )
+   )
+   (local.set $33
+    (i32.add
+     (local.tee $17
+      (i32.add
+       (local.get $23)
+       (i32.const 576)
+      )
+     )
+     (i32.const 12)
+    )
+   )
+   (local.set $40
+    (i32.add
+     (local.get $17)
+     (i32.const 11)
+    )
+   )
+   (local.set $44
+    (i32.sub
+     (local.tee $28
+      (local.get $33)
+     )
+     (local.get $27)
+    )
+   )
+   (local.set $45
+    (i32.sub
+     (i32.const -2)
+     (local.get $27)
+    )
+   )
+   (local.set $46
+    (i32.add
+     (local.get $28)
+     (i32.const 2)
+    )
+   )
+   (local.set $48
+    (i32.add
+     (local.tee $47
+      (i32.add
+       (local.get $23)
+       (i32.const 24)
+      )
+     )
+     (i32.const 288)
+    )
+   )
+   (local.set $41
+    (local.tee $31
+     (i32.add
+      (local.get $19)
+      (i32.const 9)
+     )
+    )
+   )
+   (local.set $34
+    (i32.add
+     (local.get $19)
+     (i32.const 8)
+    )
+   )
+   (local.set $15
+    (i32.const 0)
+   )
+   (local.set $10
+    (i32.const 0)
+   )
+   (local.set $17
+    (i32.const 0)
+   )
+   (block $label$2
+    (block $label$3
+     (loop $label$4
+      (block $label$5
+       (if
+        (i32.gt_s
+         (local.get $15)
+         (i32.const -1)
+        )
+        (local.set $15
+         (if (result i32)
+          (i32.gt_s
+           (local.get $10)
+           (i32.sub
+            (i32.const 2147483647)
+            (local.get $15)
+           )
+          )
+          (block (result i32)
+           (i32.store
+            (call $12)
+            (i32.const 75)
+           )
+           (i32.const -1)
+          )
+          (i32.add
+           (local.get $10)
+           (local.get $15)
+          )
+         )
+        )
+       )
+       (br_if $label$3
+        (i32.eqz
+         (i32.shr_s
+          (i32.shl
+           (local.tee $5
+            (i32.load8_s
+             (local.get $1)
+            )
+           )
+           (i32.const 24)
+          )
+          (i32.const 24)
+         )
+        )
+       )
+       (local.set $11
+        (local.get $1)
+       )
+       (block $label$9
+        (block $label$10
+         (loop $label$11
+          (block $label$12
+           (block $label$13
+            (block $label$14
+             (block $label$15
+              (br_table $label$14 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$15 $label$13
+               (i32.sub
+                (i32.shr_s
+                 (i32.shl
+                  (local.get $5)
+                  (i32.const 24)
+                 )
+                 (i32.const 24)
+                )
+                (i32.const 0)
+               )
+              )
+             )
+             (local.set $5
+              (local.get $11)
+             )
+             (br $label$10)
+            )
+            (local.set $5
+             (local.get $11)
+            )
+            (br $label$12)
+           )
+           (local.set $5
+            (i32.load8_s
+             (local.tee $11
+              (i32.add
+               (local.get $11)
+               (i32.const 1)
+              )
+             )
+            )
+           )
+           (br $label$11)
+          )
+         )
+         (br $label$9)
+        )
+        (loop $label$16
+         (br_if $label$9
+          (i32.ne
+           (i32.load8_s offset=1
+            (local.get $5)
+           )
+           (i32.const 37)
+          )
+         )
+         (local.set $11
+          (i32.add
+           (local.get $11)
+           (i32.const 1)
+          )
+         )
+         (br_if $label$16
+          (i32.eq
+           (i32.load8_s
+            (local.tee $5
+             (i32.add
+              (local.get $5)
+              (i32.const 2)
+             )
+            )
+           )
+           (i32.const 37)
+          )
+         )
+        )
+       )
+       (local.set $10
+        (i32.sub
+         (local.get $11)
+         (local.get $1)
+        )
+       )
+       (if
+        (local.get $30)
+        (if
+         (i32.eqz
+          (i32.and
+           (i32.load
+            (local.get $0)
+           )
+           (i32.const 32)
+          )
+         )
+         (drop
+          (call $21
+           (local.get $1)
+           (local.get $10)
+           (local.get $0)
+          )
+         )
+        )
+       )
+       (if
+        (local.get $10)
+        (block
+         (local.set $1
+          (local.get $5)
+         )
+         (br $label$4)
+        )
+       )
+       (local.set $10
+        (if (result i32)
+         (i32.lt_u
+          (local.tee $9
+           (i32.add
+            (i32.shr_s
+             (i32.shl
+              (local.tee $10
+               (i32.load8_s
+                (local.tee $11
+                 (i32.add
+                  (local.get $5)
+                  (i32.const 1)
+                 )
+                )
+               )
+              )
+              (i32.const 24)
+             )
+             (i32.const 24)
+            )
+            (i32.const -48)
+           )
+          )
+          (i32.const 10)
+         )
+         (block (result i32)
+          (local.set $10
+           (i32.add
+            (local.get $5)
+            (i32.const 3)
+           )
+          )
+          (if
+           (local.tee $12
+            (i32.eq
+             (i32.load8_s offset=2
+              (local.get $5)
+             )
+             (i32.const 36)
+            )
+           )
+           (local.set $11
+            (local.get $10)
+           )
+          )
+          (if
+           (local.get $12)
+           (local.set $17
+            (i32.const 1)
+           )
+          )
+          (local.set $5
+           (i32.load8_s
+            (local.get $11)
+           )
+          )
+          (if
+           (i32.eqz
+            (local.get $12)
+           )
+           (local.set $9
+            (i32.const -1)
+           )
+          )
+          (local.get $17)
+         )
+         (block (result i32)
+          (local.set $5
+           (local.get $10)
+          )
+          (local.set $9
+           (i32.const -1)
+          )
+          (local.get $17)
+         )
+        )
+       )
+       (block $label$25
+        (if
+         (i32.lt_u
+          (local.tee $12
+           (i32.add
+            (i32.shr_s
+             (i32.shl
+              (local.get $5)
+              (i32.const 24)
+             )
+             (i32.const 24)
+            )
+            (i32.const -32)
+           )
+          )
+          (i32.const 32)
+         )
+         (block
+          (local.set $17
+           (i32.const 0)
+          )
+          (loop $label$27
+           (br_if $label$25
+            (i32.eqz
+             (i32.and
+              (i32.shl
+               (i32.const 1)
+               (local.get $12)
+              )
+              (i32.const 75913)
+             )
+            )
+           )
+           (local.set $17
+            (i32.or
+             (i32.shl
+              (i32.const 1)
+              (i32.add
+               (i32.shr_s
+                (i32.shl
+                 (local.get $5)
+                 (i32.const 24)
+                )
+                (i32.const 24)
+               )
+               (i32.const -32)
+              )
+             )
+             (local.get $17)
+            )
+           )
+           (br_if $label$27
+            (i32.lt_u
+             (local.tee $12
+              (i32.add
+               (i32.shr_s
+                (i32.shl
+                 (local.tee $5
+                  (i32.load8_s
+                   (local.tee $11
+                    (i32.add
+                     (local.get $11)
+                     (i32.const 1)
+                    )
+                   )
+                  )
+                 )
+                 (i32.const 24)
+                )
+                (i32.const 24)
+               )
+               (i32.const -32)
+              )
+             )
+             (i32.const 32)
+            )
+           )
+          )
+         )
+         (local.set $17
+          (i32.const 0)
+         )
+        )
+       )
+       (block $label$29
+        (if
+         (i32.eq
+          (i32.shr_s
+           (i32.shl
+            (local.get $5)
+            (i32.const 24)
+           )
+           (i32.const 24)
+          )
+          (i32.const 42)
+         )
+         (block
+          (local.set $11
+           (block $label$31 (result i32)
+            (block $label$32
+             (br_if $label$32
+              (i32.ge_u
+               (local.tee $12
+                (i32.add
+                 (i32.shr_s
+                  (i32.shl
+                   (local.tee $5
+                    (i32.load8_s
+                     (local.tee $7
+                      (i32.add
+                       (local.get $11)
+                       (i32.const 1)
+                      )
+                     )
+                    )
+                   )
+                   (i32.const 24)
+                  )
+                  (i32.const 24)
+                 )
+                 (i32.const -48)
+                )
+               )
+               (i32.const 10)
+              )
+             )
+             (br_if $label$32
+              (i32.ne
+               (i32.load8_s offset=2
+                (local.get $11)
+               )
+               (i32.const 36)
+              )
+             )
+             (i32.store
+              (i32.add
+               (local.get $4)
+               (i32.shl
+                (local.get $12)
+                (i32.const 2)
+               )
+              )
+              (i32.const 10)
+             )
+             (local.set $8
+              (i32.const 1)
+             )
+             (local.set $10
+              (i32.wrap_i64
+               (i64.load
+                (i32.add
+                 (local.get $3)
+                 (i32.shl
+                  (i32.add
+                   (i32.load8_s
+                    (local.get $7)
+                   )
+                   (i32.const -48)
+                  )
+                  (i32.const 3)
+                 )
+                )
+               )
+              )
+             )
+             (br $label$31
+              (i32.add
+               (local.get $11)
+               (i32.const 3)
+              )
+             )
+            )
+            (if
+             (local.get $10)
+             (block
+              (local.set $15
+               (i32.const -1)
+              )
+              (br $label$5)
+             )
+            )
+            (if
+             (i32.eqz
+              (local.get $30)
+             )
+             (block
+              (local.set $12
+               (local.get $17)
+              )
+              (local.set $17
+               (i32.const 0)
+              )
+              (local.set $11
+               (local.get $7)
+              )
+              (local.set $10
+               (i32.const 0)
+              )
+              (br $label$29)
+             )
+            )
+            (local.set $10
+             (i32.load
+              (local.tee $11
+               (i32.and
+                (i32.add
+                 (i32.load
+                  (local.get $2)
+                 )
+                 (i32.const 3)
+                )
+                (i32.const -4)
+               )
+              )
+             )
+            )
+            (i32.store
+             (local.get $2)
+             (i32.add
+              (local.get $11)
+              (i32.const 4)
+             )
+            )
+            (local.set $8
+             (i32.const 0)
+            )
+            (local.get $7)
+           )
+          )
+          (local.set $12
+           (i32.or
+            (local.get $17)
+            (i32.const 8192)
+           )
+          )
+          (local.set $7
+           (i32.sub
+            (i32.const 0)
+            (local.get $10)
+           )
+          )
+          (local.set $5
+           (i32.load8_s
+            (local.get $11)
+           )
+          )
+          (if
+           (i32.eqz
+            (local.tee $6
+             (i32.lt_s
+              (local.get $10)
+              (i32.const 0)
+             )
+            )
+           )
+           (local.set $12
+            (local.get $17)
+           )
+          )
+          (local.set $17
+           (local.get $8)
+          )
+          (if
+           (local.get $6)
+           (local.set $10
+            (local.get $7)
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.tee $12
+            (i32.add
+             (i32.shr_s
+              (i32.shl
+               (local.get $5)
+               (i32.const 24)
+              )
+              (i32.const 24)
+             )
+             (i32.const -48)
+            )
+           )
+           (i32.const 10)
+          )
+          (block
+           (local.set $7
+            (i32.const 0)
+           )
+           (local.set $5
+            (local.get $12)
+           )
+           (loop $label$39
+            (local.set $7
+             (i32.add
+              (i32.mul
+               (local.get $7)
+               (i32.const 10)
+              )
+              (local.get $5)
+             )
+            )
+            (br_if $label$39
+             (i32.lt_u
+              (local.tee $5
+               (i32.add
+                (i32.shr_s
+                 (i32.shl
+                  (local.tee $12
+                   (i32.load8_s
+                    (local.tee $11
+                     (i32.add
+                      (local.get $11)
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                  )
+                  (i32.const 24)
+                 )
+                 (i32.const 24)
+                )
+                (i32.const -48)
+               )
+              )
+              (i32.const 10)
+             )
+            )
+           )
+           (if
+            (i32.lt_s
+             (local.get $7)
+             (i32.const 0)
+            )
+            (block
+             (local.set $15
+              (i32.const -1)
+             )
+             (br $label$5)
+            )
+            (block
+             (local.set $5
+              (local.get $12)
+             )
+             (local.set $12
+              (local.get $17)
+             )
+             (local.set $17
+              (local.get $10)
+             )
+             (local.set $10
+              (local.get $7)
+             )
+            )
+           )
+          )
+          (block
+           (local.set $12
+            (local.get $17)
+           )
+           (local.set $17
+            (local.get $10)
+           )
+           (local.set $10
+            (i32.const 0)
+           )
+          )
+         )
+        )
+       )
+       (block $label$43
+        (if
+         (i32.eq
+          (i32.shr_s
+           (i32.shl
+            (local.get $5)
+            (i32.const 24)
+           )
+           (i32.const 24)
+          )
+          (i32.const 46)
+         )
+         (block
+          (if
+           (i32.ne
+            (i32.shr_s
+             (i32.shl
+              (local.tee $5
+               (i32.load8_s
+                (local.tee $7
+                 (i32.add
+                  (local.get $11)
+                  (i32.const 1)
+                 )
+                )
+               )
+              )
+              (i32.const 24)
+             )
+             (i32.const 24)
+            )
+            (i32.const 42)
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.tee $5
+               (i32.add
+                (i32.shr_s
+                 (i32.shl
+                  (local.get $5)
+                  (i32.const 24)
+                 )
+                 (i32.const 24)
+                )
+                (i32.const -48)
+               )
+              )
+              (i32.const 10)
+             )
+             (block
+              (local.set $11
+               (local.get $7)
+              )
+              (local.set $7
+               (i32.const 0)
+              )
+             )
+             (block
+              (local.set $5
+               (i32.const 0)
+              )
+              (local.set $11
+               (local.get $7)
+              )
+              (br $label$43)
+             )
+            )
+            (loop $label$48
+             (local.set $5
+              (i32.add
+               (i32.mul
+                (local.get $7)
+                (i32.const 10)
+               )
+               (local.get $5)
+              )
+             )
+             (br_if $label$43
+              (i32.ge_u
+               (local.tee $8
+                (i32.add
+                 (i32.load8_s
+                  (local.tee $11
+                   (i32.add
+                    (local.get $11)
+                    (i32.const 1)
+                   )
+                  )
+                 )
+                 (i32.const -48)
+                )
+               )
+               (i32.const 10)
+              )
+             )
+             (local.set $7
+              (local.get $5)
+             )
+             (local.set $5
+              (local.get $8)
+             )
+             (br $label$48)
+            )
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.tee $5
+             (i32.add
+              (i32.load8_s
+               (local.tee $7
+                (i32.add
+                 (local.get $11)
+                 (i32.const 2)
+                )
+               )
+              )
+              (i32.const -48)
+             )
+            )
+            (i32.const 10)
+           )
+           (if
+            (i32.eq
+             (i32.load8_s offset=3
+              (local.get $11)
+             )
+             (i32.const 36)
+            )
+            (block
+             (i32.store
+              (i32.add
+               (local.get $4)
+               (i32.shl
+                (local.get $5)
+                (i32.const 2)
+               )
+              )
+              (i32.const 10)
+             )
+             (local.set $5
+              (i32.wrap_i64
+               (i64.load
+                (i32.add
+                 (local.get $3)
+                 (i32.shl
+                  (i32.add
+                   (i32.load8_s
+                    (local.get $7)
+                   )
+                   (i32.const -48)
+                  )
+                  (i32.const 3)
+                 )
+                )
+               )
+              )
+             )
+             (local.set $11
+              (i32.add
+               (local.get $11)
+               (i32.const 4)
+              )
+             )
+             (br $label$43)
+            )
+           )
+          )
+          (if
+           (local.get $17)
+           (block
+            (local.set $15
+             (i32.const -1)
+            )
+            (br $label$5)
+           )
+          )
+          (local.set $11
+           (if (result i32)
+            (local.get $30)
+            (block (result i32)
+             (local.set $5
+              (i32.load
+               (local.tee $11
+                (i32.and
+                 (i32.add
+                  (i32.load
+                   (local.get $2)
+                  )
+                  (i32.const 3)
+                 )
+                 (i32.const -4)
+                )
+               )
+              )
+             )
+             (i32.store
+              (local.get $2)
+              (i32.add
+               (local.get $11)
+               (i32.const 4)
+              )
+             )
+             (local.get $7)
+            )
+            (block (result i32)
+             (local.set $5
+              (i32.const 0)
+             )
+             (local.get $7)
+            )
+           )
+          )
+         )
+         (local.set $5
+          (i32.const -1)
+         )
+        )
+       )
+       (local.set $7
+        (local.get $11)
+       )
+       (local.set $8
+        (i32.const 0)
+       )
+       (loop $label$55
+        (if
+         (i32.gt_u
+          (local.tee $6
+           (i32.add
+            (i32.load8_s
+             (local.get $7)
+            )
+            (i32.const -65)
+           )
+          )
+          (i32.const 57)
+         )
+         (block
+          (local.set $15
+           (i32.const -1)
+          )
+          (br $label$5)
+         )
+        )
+        (local.set $11
+         (i32.add
+          (local.get $7)
+          (i32.const 1)
+         )
+        )
+        (if
+         (i32.lt_u
+          (i32.add
+           (local.tee $6
+            (i32.and
+             (local.tee $13
+              (i32.load8_s
+               (i32.add
+                (i32.add
+                 (i32.mul
+                  (local.get $8)
+                  (i32.const 58)
+                 )
+                 (i32.const 1155)
+                )
+                (local.get $6)
+               )
+              )
+             )
+             (i32.const 255)
+            )
+           )
+           (i32.const -1)
+          )
+          (i32.const 8)
+         )
+         (block
+          (local.set $7
+           (local.get $11)
+          )
+          (local.set $8
+           (local.get $6)
+          )
+          (br $label$55)
+         )
+        )
+       )
+       (if
+        (i32.eqz
+         (i32.shr_s
+          (i32.shl
+           (local.get $13)
+           (i32.const 24)
+          )
+          (i32.const 24)
+         )
+        )
+        (block
+         (local.set $15
+          (i32.const -1)
+         )
+         (br $label$5)
+        )
+       )
+       (local.set $14
+        (i32.gt_s
+         (local.get $9)
+         (i32.const -1)
+        )
+       )
+       (block $label$59
+        (block $label$60
+         (if
+          (i32.eq
+           (i32.shr_s
+            (i32.shl
+             (local.get $13)
+             (i32.const 24)
+            )
+            (i32.const 24)
+           )
+           (i32.const 19)
+          )
+          (if
+           (local.get $14)
+           (block
+            (local.set $15
+             (i32.const -1)
+            )
+            (br $label$5)
+           )
+           (br $label$60)
+          )
+          (block
+           (if
+            (local.get $14)
+            (block
+             (i32.store
+              (i32.add
+               (local.get $4)
+               (i32.shl
+                (local.get $9)
+                (i32.const 2)
+               )
+              )
+              (local.get $6)
+             )
+             (i64.store
+              (local.get $16)
+              (i64.load
+               (i32.add
+                (local.get $3)
+                (i32.shl
+                 (local.get $9)
+                 (i32.const 3)
+                )
+               )
+              )
+             )
+             (br $label$60)
+            )
+           )
+           (if
+            (i32.eqz
+             (local.get $30)
+            )
+            (block
+             (local.set $15
+              (i32.const 0)
+             )
+             (br $label$5)
+            )
+           )
+           (call $22
+            (local.get $16)
+            (local.get $6)
+            (local.get $2)
+           )
+          )
+         )
+         (br $label$59)
+        )
+        (if
+         (i32.eqz
+          (local.get $30)
+         )
+         (block
+          (local.set $10
+           (i32.const 0)
+          )
+          (local.set $1
+           (local.get $11)
+          )
+          (br $label$4)
+         )
+        )
+       )
+       (local.set $9
+        (i32.and
+         (local.tee $7
+          (i32.load8_s
+           (local.get $7)
+          )
+         )
+         (i32.const -33)
+        )
+       )
+       (if
+        (i32.eqz
+         (i32.and
+          (i32.ne
+           (local.get $8)
+           (i32.const 0)
+          )
+          (i32.eq
+           (i32.and
+            (local.get $7)
+            (i32.const 15)
+           )
+           (i32.const 3)
+          )
+         )
+        )
+        (local.set $9
+         (local.get $7)
+        )
+       )
+       (local.set $7
+        (i32.and
+         (local.get $12)
+         (i32.const -65537)
+        )
+       )
+       (if
+        (i32.and
+         (local.get $12)
+         (i32.const 8192)
+        )
+        (local.set $12
+         (local.get $7)
+        )
+       )
+       (block $label$70
+        (block $label$71
+         (block $label$72
+          (block $label$73
+           (block $label$74
+            (block $label$75
+             (block $label$76
+              (block $label$77
+               (block $label$78
+                (block $label$79
+                 (block $label$80
+                  (block $label$81
+                   (block $label$82
+                    (block $label$83
+                     (block $label$84
+                      (block $label$85
+                       (block $label$86
+                        (block $label$87
+                         (block $label$88
+                          (block $label$89
+                           (br_table $label$78 $label$77 $label$80 $label$77 $label$78 $label$78 $label$78 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$79 $label$77 $label$77 $label$77 $label$77 $label$87 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$78 $label$77 $label$83 $label$85 $label$78 $label$78 $label$78 $label$77 $label$85 $label$77 $label$77 $label$77 $label$82 $label$89 $label$86 $label$88 $label$77 $label$77 $label$81 $label$77 $label$84 $label$77 $label$77 $label$87 $label$77
+                            (i32.sub
+                             (local.get $9)
+                             (i32.const 65)
+                            )
+                           )
+                          )
+                          (block $label$90
+                           (block $label$91
+                            (block $label$92
+                             (block $label$93
+                              (block $label$94
+                               (block $label$95
+                                (block $label$96
+                                 (block $label$97
+                                  (br_table $label$97 $label$96 $label$95 $label$94 $label$93 $label$90 $label$92 $label$91 $label$90
+                                   (i32.sub
+                                    (i32.shr_s
+                                     (i32.shl
+                                      (i32.and
+                                       (local.get $8)
+                                       (i32.const 255)
+                                      )
+                                      (i32.const 24)
+                                     )
+                                     (i32.const 24)
+                                    )
+                                    (i32.const 0)
+                                   )
+                                  )
+                                 )
+                                 (i32.store
+                                  (i32.load
+                                   (local.get $16)
+                                  )
+                                  (local.get $15)
+                                 )
+                                 (local.set $10
+                                  (i32.const 0)
+                                 )
+                                 (local.set $1
+                                  (local.get $11)
+                                 )
+                                 (br $label$4)
+                                )
+                                (i32.store
+                                 (i32.load
+                                  (local.get $16)
+                                 )
+                                 (local.get $15)
+                                )
+                                (local.set $10
+                                 (i32.const 0)
+                                )
+                                (local.set $1
+                                 (local.get $11)
+                                )
+                                (br $label$4)
+                               )
+                               (i64.store
+                                (i32.load
+                                 (local.get $16)
+                                )
+                                (i64.extend_i32_s
+                                 (local.get $15)
+                                )
+                               )
+                               (local.set $10
+                                (i32.const 0)
+                               )
+                               (local.set $1
+                                (local.get $11)
+                               )
+                               (br $label$4)
+                              )
+                              (i32.store16
+                               (i32.load
+                                (local.get $16)
+                               )
+                               (local.get $15)
+                              )
+                              (local.set $10
+                               (i32.const 0)
+                              )
+                              (local.set $1
+                               (local.get $11)
+                              )
+                              (br $label$4)
+                             )
+                             (i32.store8
+                              (i32.load
+                               (local.get $16)
+                              )
+                              (local.get $15)
+                             )
+                             (local.set $10
+                              (i32.const 0)
+                             )
+                             (local.set $1
+                              (local.get $11)
+                             )
+                             (br $label$4)
+                            )
+                            (i32.store
+                             (i32.load
+                              (local.get $16)
+                             )
+                             (local.get $15)
+                            )
+                            (local.set $10
+                             (i32.const 0)
+                            )
+                            (local.set $1
+                             (local.get $11)
+                            )
+                            (br $label$4)
+                           )
+                           (i64.store
+                            (i32.load
+                             (local.get $16)
+                            )
+                            (i64.extend_i32_s
+                             (local.get $15)
+                            )
+                           )
+                           (local.set $10
+                            (i32.const 0)
+                           )
+                           (local.set $1
+                            (local.get $11)
+                           )
+                           (br $label$4)
+                          )
+                          (local.set $10
+                           (i32.const 0)
+                          )
+                          (local.set $1
+                           (local.get $11)
+                          )
+                          (br $label$4)
+                         )
+                         (local.set $12
+                          (i32.or
+                           (local.get $12)
+                           (i32.const 8)
+                          )
+                         )
+                         (if
+                          (i32.le_u
+                           (local.get $5)
+                           (i32.const 8)
+                          )
+                          (local.set $5
+                           (i32.const 8)
+                          )
+                         )
+                         (local.set $9
+                          (i32.const 120)
+                         )
+                         (br $label$76)
+                        )
+                        (br $label$76)
+                       )
+                       (if
+                        (i64.eq
+                         (local.tee $50
+                          (i64.load
+                           (local.get $16)
+                          )
+                         )
+                         (i64.const 0)
+                        )
+                        (local.set $7
+                         (local.get $21)
+                        )
+                        (block
+                         (local.set $1
+                          (local.get $21)
+                         )
+                         (loop $label$101
+                          (i64.store8
+                           (local.tee $1
+                            (i32.add
+                             (local.get $1)
+                             (i32.const -1)
+                            )
+                           )
+                           (i64.or
+                            (i64.and
+                             (local.get $50)
+                             (i64.const 7)
+                            )
+                            (i64.const 48)
+                           )
+                          )
+                          (br_if $label$101
+                           (i64.ne
+                            (local.tee $50
+                             (i64.shr_u
+                              (local.get $50)
+                              (i64.const 3)
+                             )
+                            )
+                            (i64.const 0)
+                           )
+                          )
+                          (local.set $7
+                           (local.get $1)
+                          )
+                         )
+                        )
+                       )
+                       (if
+                        (i32.and
+                         (local.get $12)
+                         (i32.const 8)
+                        )
+                        (block
+                         (local.set $8
+                          (i32.add
+                           (local.tee $1
+                            (i32.sub
+                             (local.get $38)
+                             (local.get $7)
+                            )
+                           )
+                           (i32.const 1)
+                          )
+                         )
+                         (if
+                          (i32.le_s
+                           (local.get $5)
+                           (local.get $1)
+                          )
+                          (local.set $5
+                           (local.get $8)
+                          )
+                         )
+                         (local.set $6
+                          (i32.const 0)
+                         )
+                         (local.set $8
+                          (i32.const 1635)
+                         )
+                         (br $label$71)
+                        )
+                        (block
+                         (local.set $6
+                          (i32.const 0)
+                         )
+                         (local.set $8
+                          (i32.const 1635)
+                         )
+                         (br $label$71)
+                        )
+                       )
+                      )
+                      (if
+                       (i64.lt_s
+                        (local.tee $50
+                         (i64.load
+                          (local.get $16)
+                         )
+                        )
+                        (i64.const 0)
+                       )
+                       (block
+                        (i64.store
+                         (local.get $16)
+                         (local.tee $50
+                          (i64.sub
+                           (i64.const 0)
+                           (local.get $50)
+                          )
+                         )
+                        )
+                        (local.set $6
+                         (i32.const 1)
+                        )
+                        (local.set $8
+                         (i32.const 1635)
+                        )
+                        (br $label$75)
+                       )
+                      )
+                      (if
+                       (i32.and
+                        (local.get $12)
+                        (i32.const 2048)
+                       )
+                       (block
+                        (local.set $6
+                         (i32.const 1)
+                        )
+                        (local.set $8
+                         (i32.const 1636)
+                        )
+                        (br $label$75)
+                       )
+                       (block
+                        (local.set $6
+                         (local.tee $1
+                          (i32.and
+                           (local.get $12)
+                           (i32.const 1)
+                          )
+                         )
+                        )
+                        (local.set $8
+                         (if (result i32)
+                          (local.get $1)
+                          (i32.const 1637)
+                          (i32.const 1635)
+                         )
+                        )
+                        (br $label$75)
+                       )
+                      )
+                     )
+                     (local.set $50
+                      (i64.load
+                       (local.get $16)
+                      )
+                     )
+                     (local.set $6
+                      (i32.const 0)
+                     )
+                     (local.set $8
+                      (i32.const 1635)
+                     )
+                     (br $label$75)
+                    )
+                    (i64.store8
+                     (local.get $39)
+                     (i64.load
+                      (local.get $16)
+                     )
+                    )
+                    (local.set $1
+                     (local.get $39)
+                    )
+                    (local.set $12
+                     (local.get $7)
+                    )
+                    (local.set $7
+                     (i32.const 1)
+                    )
+                    (local.set $6
+                     (i32.const 0)
+                    )
+                    (local.set $8
+                     (i32.const 1635)
+                    )
+                    (local.set $5
+                     (local.get $21)
+                    )
+                    (br $label$70)
+                   )
+                   (local.set $1
+                    (call $24
+                     (i32.load
+                      (call $12)
+                     )
+                    )
+                   )
+                   (br $label$74)
+                  )
+                  (if
+                   (i32.eqz
+                    (local.tee $1
+                     (i32.load
+                      (local.get $16)
+                     )
+                    )
+                   )
+                   (local.set $1
+                    (i32.const 1645)
+                   )
+                  )
+                  (br $label$74)
+                 )
+                 (i64.store32
+                  (local.get $37)
+                  (i64.load
+                   (local.get $16)
+                  )
+                 )
+                 (i32.store
+                  (local.get $42)
+                  (i32.const 0)
+                 )
+                 (i32.store
+                  (local.get $16)
+                  (local.get $37)
+                 )
+                 (local.set $7
+                  (local.get $37)
+                 )
+                 (local.set $6
+                  (i32.const -1)
+                 )
+                 (br $label$73)
+                )
+                (local.set $7
+                 (i32.load
+                  (local.get $16)
+                 )
+                )
+                (if
+                 (local.get $5)
+                 (block
+                  (local.set $6
+                   (local.get $5)
+                  )
+                  (br $label$73)
+                 )
+                 (block
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (i32.const 0)
+                   (local.get $12)
+                  )
+                  (local.set $1
+                   (i32.const 0)
+                  )
+                  (br $label$72)
+                 )
+                )
+               )
+               (local.set $52
+                (f64.load
+                 (local.get $16)
+                )
+               )
+               (i32.store
+                (local.get $20)
+                (i32.const 0)
+               )
+               (local.set $26
+                (if (result i32)
+                 (i64.lt_s
+                  (i64.reinterpret_f64
+                   (local.get $52)
+                  )
+                  (i64.const 0)
+                 )
+                 (block (result i32)
+                  (local.set $24
+                   (i32.const 1)
+                  )
+                  (local.set $52
+                   (f64.neg
+                    (local.get $52)
+                   )
+                  )
+                  (i32.const 1652)
+                 )
+                 (block (result i32)
+                  (local.set $1
+                   (i32.and
+                    (local.get $12)
+                    (i32.const 1)
+                   )
+                  )
+                  (if (result i32)
+                   (i32.and
+                    (local.get $12)
+                    (i32.const 2048)
+                   )
+                   (block (result i32)
+                    (local.set $24
+                     (i32.const 1)
+                    )
+                    (i32.const 1655)
+                   )
+                   (block (result i32)
+                    (local.set $24
+                     (local.get $1)
+                    )
+                    (if (result i32)
+                     (local.get $1)
+                     (i32.const 1658)
+                     (i32.const 1653)
+                    )
+                   )
+                  )
+                 )
+                )
+               )
+               (block $label$119
+                (if
+                 (i64.lt_u
+                  (i64.and
+                   (i64.reinterpret_f64
+                    (local.get $52)
+                   )
+                   (i64.const 9218868437227405312)
+                  )
+                  (i64.const 9218868437227405312)
+                 )
+                 (block
+                  (if
+                   (local.tee $1
+                    (f64.ne
+                     (local.tee $52
+                      (f64.mul
+                       (call $27
+                        (local.get $52)
+                        (local.get $20)
+                       )
+                       (f64.const 2)
+                      )
+                     )
+                     (f64.const 0)
+                    )
+                   )
+                   (i32.store
+                    (local.get $20)
+                    (i32.add
+                     (i32.load
+                      (local.get $20)
+                     )
+                     (i32.const -1)
+                    )
+                   )
+                  )
+                  (if
+                   (i32.eq
+                    (local.tee $22
+                     (i32.or
+                      (local.get $9)
+                      (i32.const 32)
+                     )
+                    )
+                    (i32.const 97)
+                   )
+                   (block
+                    (local.set $1
+                     (i32.add
+                      (local.get $26)
+                      (i32.const 9)
+                     )
+                    )
+                    (if
+                     (local.tee $6
+                      (i32.and
+                       (local.get $9)
+                       (i32.const 32)
+                      )
+                     )
+                     (local.set $26
+                      (local.get $1)
+                     )
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.or
+                       (i32.gt_u
+                        (local.get $5)
+                        (i32.const 11)
+                       )
+                       (i32.eqz
+                        (local.tee $1
+                         (i32.sub
+                          (i32.const 12)
+                          (local.get $5)
+                         )
+                        )
+                       )
+                      )
+                     )
+                     (block
+                      (local.set $53
+                       (f64.const 8)
+                      )
+                      (loop $label$125
+                       (local.set $53
+                        (f64.mul
+                         (local.get $53)
+                         (f64.const 16)
+                        )
+                       )
+                       (br_if $label$125
+                        (local.tee $1
+                         (i32.add
+                          (local.get $1)
+                          (i32.const -1)
+                         )
+                        )
+                       )
+                      )
+                      (local.set $52
+                       (if (result f64)
+                        (i32.eq
+                         (i32.load8_s
+                          (local.get $26)
+                         )
+                         (i32.const 45)
+                        )
+                        (f64.neg
+                         (f64.add
+                          (local.get $53)
+                          (f64.sub
+                           (f64.neg
+                            (local.get $52)
+                           )
+                           (local.get $53)
+                          )
+                         )
+                        )
+                        (f64.sub
+                         (f64.add
+                          (local.get $52)
+                          (local.get $53)
+                         )
+                         (local.get $53)
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (local.set $1
+                     (i32.sub
+                      (i32.const 0)
+                      (local.tee $7
+                       (i32.load
+                        (local.get $20)
+                       )
+                      )
+                     )
+                    )
+                    (if
+                     (i32.eq
+                      (local.tee $1
+                       (call $23
+                        (i64.extend_i32_s
+                         (if (result i32)
+                          (i32.lt_s
+                           (local.get $7)
+                           (i32.const 0)
+                          )
+                          (local.get $1)
+                          (local.get $7)
+                         )
+                        )
+                        (local.get $33)
+                       )
+                      )
+                      (local.get $33)
+                     )
+                     (block
+                      (i32.store8
+                       (local.get $40)
+                       (i32.const 48)
+                      )
+                      (local.set $1
+                       (local.get $40)
+                      )
+                     )
+                    )
+                    (local.set $13
+                     (i32.or
+                      (local.get $24)
+                      (i32.const 2)
+                     )
+                    )
+                    (i32.store8
+                     (i32.add
+                      (local.get $1)
+                      (i32.const -1)
+                     )
+                     (i32.add
+                      (i32.and
+                       (i32.shr_s
+                        (local.get $7)
+                        (i32.const 31)
+                       )
+                       (i32.const 2)
+                      )
+                      (i32.const 43)
+                     )
+                    )
+                    (i32.store8
+                     (local.tee $8
+                      (i32.add
+                       (local.get $1)
+                       (i32.const -2)
+                      )
+                     )
+                     (i32.add
+                      (local.get $9)
+                      (i32.const 15)
+                     )
+                    )
+                    (local.set $9
+                     (i32.lt_s
+                      (local.get $5)
+                      (i32.const 1)
+                     )
+                    )
+                    (local.set $14
+                     (i32.eqz
+                      (i32.and
+                       (local.get $12)
+                       (i32.const 8)
+                      )
+                     )
+                    )
+                    (local.set $1
+                     (local.get $19)
+                    )
+                    (loop $label$131
+                     (i32.store8
+                      (local.get $1)
+                      (i32.or
+                       (i32.load8_u
+                        (i32.add
+                         (local.tee $7
+                          (i32.trunc_f64_s
+                           (local.get $52)
+                          )
+                         )
+                         (i32.const 1619)
+                        )
+                       )
+                       (local.get $6)
+                      )
+                     )
+                     (local.set $52
+                      (f64.mul
+                       (f64.sub
+                        (local.get $52)
+                        (f64.convert_i32_s
+                         (local.get $7)
+                        )
+                       )
+                       (f64.const 16)
+                      )
+                     )
+                     (local.set $1
+                      (block $label$132 (result i32)
+                       (if (result i32)
+                        (i32.eq
+                         (i32.sub
+                          (local.tee $7
+                           (i32.add
+                            (local.get $1)
+                            (i32.const 1)
+                           )
+                          )
+                          (local.get $27)
+                         )
+                         (i32.const 1)
+                        )
+                        (block (result i32)
+                         (drop
+                          (br_if $label$132
+                           (local.get $7)
+                           (i32.and
+                            (local.get $14)
+                            (i32.and
+                             (local.get $9)
+                             (f64.eq
+                              (local.get $52)
+                              (f64.const 0)
+                             )
+                            )
+                           )
+                          )
+                         )
+                         (i32.store8
+                          (local.get $7)
+                          (i32.const 46)
+                         )
+                         (i32.add
+                          (local.get $1)
+                          (i32.const 2)
+                         )
+                        )
+                        (local.get $7)
+                       )
+                      )
+                     )
+                     (br_if $label$131
+                      (f64.ne
+                       (local.get $52)
+                       (f64.const 0)
+                      )
+                     )
+                    )
+                    (local.set $6
+                     (i32.sub
+                      (i32.add
+                       (local.get $46)
+                       (local.get $5)
+                      )
+                      (local.tee $7
+                       (local.get $8)
+                      )
+                     )
+                    )
+                    (local.set $9
+                     (i32.add
+                      (i32.sub
+                       (local.get $44)
+                       (local.get $7)
+                      )
+                      (local.get $1)
+                     )
+                    )
+                    (call $25
+                     (local.get $0)
+                     (i32.const 32)
+                     (local.get $10)
+                     (local.tee $5
+                      (i32.add
+                       (if (result i32)
+                        (i32.and
+                         (i32.ne
+                          (local.get $5)
+                          (i32.const 0)
+                         )
+                         (i32.lt_s
+                          (i32.add
+                           (local.get $45)
+                           (local.get $1)
+                          )
+                          (local.get $5)
+                         )
+                        )
+                        (local.get $6)
+                        (local.tee $6
+                         (local.get $9)
+                        )
+                       )
+                       (local.get $13)
+                      )
+                     )
+                     (local.get $12)
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $21
+                       (local.get $26)
+                       (local.get $13)
+                       (local.get $0)
+                      )
+                     )
+                    )
+                    (call $25
+                     (local.get $0)
+                     (i32.const 48)
+                     (local.get $10)
+                     (local.get $5)
+                     (i32.xor
+                      (local.get $12)
+                      (i32.const 65536)
+                     )
+                    )
+                    (local.set $1
+                     (i32.sub
+                      (local.get $1)
+                      (local.get $27)
+                     )
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $21
+                       (local.get $19)
+                       (local.get $1)
+                       (local.get $0)
+                      )
+                     )
+                    )
+                    (call $25
+                     (local.get $0)
+                     (i32.const 48)
+                     (i32.sub
+                      (local.get $6)
+                      (i32.add
+                       (local.get $1)
+                       (local.tee $1
+                        (i32.sub
+                         (local.get $28)
+                         (local.get $7)
+                        )
+                       )
+                      )
+                     )
+                     (i32.const 0)
+                     (i32.const 0)
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $21
+                       (local.get $8)
+                       (local.get $1)
+                       (local.get $0)
+                      )
+                     )
+                    )
+                    (call $25
+                     (local.get $0)
+                     (i32.const 32)
+                     (local.get $10)
+                     (local.get $5)
+                     (i32.xor
+                      (local.get $12)
+                      (i32.const 8192)
+                     )
+                    )
+                    (if
+                     (i32.ge_s
+                      (local.get $5)
+                      (local.get $10)
+                     )
+                     (local.set $10
+                      (local.get $5)
+                     )
+                    )
+                    (br $label$119)
+                   )
+                  )
+                  (if
+                   (local.get $1)
+                   (block
+                    (i32.store
+                     (local.get $20)
+                     (local.tee $6
+                      (i32.add
+                       (i32.load
+                        (local.get $20)
+                       )
+                       (i32.const -28)
+                      )
+                     )
+                    )
+                    (local.set $52
+                     (f64.mul
+                      (local.get $52)
+                      (f64.const 268435456)
+                     )
+                    )
+                   )
+                   (local.set $6
+                    (i32.load
+                     (local.get $20)
+                    )
+                   )
+                  )
+                  (local.set $8
+                   (local.tee $7
+                    (if (result i32)
+                     (i32.lt_s
+                      (local.get $6)
+                      (i32.const 0)
+                     )
+                     (local.get $47)
+                     (local.get $48)
+                    )
+                   )
+                  )
+                  (loop $label$145
+                   (i32.store
+                    (local.get $8)
+                    (local.tee $1
+                     (i32.trunc_f64_s
+                      (local.get $52)
+                     )
+                    )
+                   )
+                   (local.set $8
+                    (i32.add
+                     (local.get $8)
+                     (i32.const 4)
+                    )
+                   )
+                   (br_if $label$145
+                    (f64.ne
+                     (local.tee $52
+                      (f64.mul
+                       (f64.sub
+                        (local.get $52)
+                        (f64.convert_i32_u
+                         (local.get $1)
+                        )
+                       )
+                       (f64.const 1e9)
+                      )
+                     )
+                     (f64.const 0)
+                    )
+                   )
+                  )
+                  (if
+                   (i32.gt_s
+                    (local.get $6)
+                    (i32.const 0)
+                   )
+                   (block
+                    (local.set $1
+                     (local.get $7)
+                    )
+                    (loop $label$147
+                     (local.set $14
+                      (if (result i32)
+                       (i32.gt_s
+                        (local.get $6)
+                        (i32.const 29)
+                       )
+                       (i32.const 29)
+                       (local.get $6)
+                      )
+                     )
+                     (block $label$150
+                      (if
+                       (i32.ge_u
+                        (local.tee $6
+                         (i32.add
+                          (local.get $8)
+                          (i32.const -4)
+                         )
+                        )
+                        (local.get $1)
+                       )
+                       (block
+                        (local.set $50
+                         (i64.extend_i32_u
+                          (local.get $14)
+                         )
+                        )
+                        (local.set $13
+                         (i32.const 0)
+                        )
+                        (loop $label$152
+                         (i64.store32
+                          (local.get $6)
+                          (i64.rem_u
+                           (local.tee $51
+                            (i64.add
+                             (i64.shl
+                              (i64.extend_i32_u
+                               (i32.load
+                                (local.get $6)
+                               )
+                              )
+                              (local.get $50)
+                             )
+                             (i64.extend_i32_u
+                              (local.get $13)
+                             )
+                            )
+                           )
+                           (i64.const 1000000000)
+                          )
+                         )
+                         (local.set $13
+                          (i32.wrap_i64
+                           (i64.div_u
+                            (local.get $51)
+                            (i64.const 1000000000)
+                           )
+                          )
+                         )
+                         (br_if $label$152
+                          (i32.ge_u
+                           (local.tee $6
+                            (i32.add
+                             (local.get $6)
+                             (i32.const -4)
+                            )
+                           )
+                           (local.get $1)
+                          )
+                         )
+                        )
+                        (br_if $label$150
+                         (i32.eqz
+                          (local.get $13)
+                         )
+                        )
+                        (i32.store
+                         (local.tee $1
+                          (i32.add
+                           (local.get $1)
+                           (i32.const -4)
+                          )
+                         )
+                         (local.get $13)
+                        )
+                       )
+                      )
+                     )
+                     (loop $label$153
+                      (if
+                       (i32.gt_u
+                        (local.get $8)
+                        (local.get $1)
+                       )
+                       (if
+                        (i32.eqz
+                         (i32.load
+                          (local.tee $6
+                           (i32.add
+                            (local.get $8)
+                            (i32.const -4)
+                           )
+                          )
+                         )
+                        )
+                        (block
+                         (local.set $8
+                          (local.get $6)
+                         )
+                         (br $label$153)
+                        )
+                       )
+                      )
+                     )
+                     (i32.store
+                      (local.get $20)
+                      (local.tee $6
+                       (i32.sub
+                        (i32.load
+                         (local.get $20)
+                        )
+                        (local.get $14)
+                       )
+                      )
+                     )
+                     (br_if $label$147
+                      (i32.gt_s
+                       (local.get $6)
+                       (i32.const 0)
+                      )
+                     )
+                    )
+                   )
+                   (local.set $1
+                    (local.get $7)
+                   )
+                  )
+                  (local.set $18
+                   (if (result i32)
+                    (i32.lt_s
+                     (local.get $5)
+                     (i32.const 0)
+                    )
+                    (i32.const 6)
+                    (local.get $5)
+                   )
+                  )
+                  (if
+                   (i32.lt_s
+                    (local.get $6)
+                    (i32.const 0)
+                   )
+                   (block
+                    (local.set $14
+                     (i32.add
+                      (i32.div_s
+                       (i32.add
+                        (local.get $18)
+                        (i32.const 25)
+                       )
+                       (i32.const 9)
+                      )
+                      (i32.const 1)
+                     )
+                    )
+                    (local.set $25
+                     (i32.eq
+                      (local.get $22)
+                      (i32.const 102)
+                     )
+                    )
+                    (local.set $5
+                     (local.get $8)
+                    )
+                    (loop $label$160
+                     (if
+                      (i32.gt_s
+                       (local.tee $13
+                        (i32.sub
+                         (i32.const 0)
+                         (local.get $6)
+                        )
+                       )
+                       (i32.const 9)
+                      )
+                      (local.set $13
+                       (i32.const 9)
+                      )
+                     )
+                     (block $label$162
+                      (if
+                       (i32.lt_u
+                        (local.get $1)
+                        (local.get $5)
+                       )
+                       (block
+                        (local.set $29
+                         (i32.add
+                          (i32.shl
+                           (i32.const 1)
+                           (local.get $13)
+                          )
+                          (i32.const -1)
+                         )
+                        )
+                        (local.set $35
+                         (i32.shr_u
+                          (i32.const 1000000000)
+                          (local.get $13)
+                         )
+                        )
+                        (local.set $6
+                         (i32.const 0)
+                        )
+                        (local.set $8
+                         (local.get $1)
+                        )
+                        (loop $label$164
+                         (i32.store
+                          (local.get $8)
+                          (i32.add
+                           (i32.shr_u
+                            (local.tee $32
+                             (i32.load
+                              (local.get $8)
+                             )
+                            )
+                            (local.get $13)
+                           )
+                           (local.get $6)
+                          )
+                         )
+                         (local.set $6
+                          (i32.mul
+                           (i32.and
+                            (local.get $32)
+                            (local.get $29)
+                           )
+                           (local.get $35)
+                          )
+                         )
+                         (br_if $label$164
+                          (i32.lt_u
+                           (local.tee $8
+                            (i32.add
+                             (local.get $8)
+                             (i32.const 4)
+                            )
+                           )
+                           (local.get $5)
+                          )
+                         )
+                        )
+                        (local.set $8
+                         (i32.add
+                          (local.get $1)
+                          (i32.const 4)
+                         )
+                        )
+                        (if
+                         (i32.eqz
+                          (i32.load
+                           (local.get $1)
+                          )
+                         )
+                         (local.set $1
+                          (local.get $8)
+                         )
+                        )
+                        (br_if $label$162
+                         (i32.eqz
+                          (local.get $6)
+                         )
+                        )
+                        (i32.store
+                         (local.get $5)
+                         (local.get $6)
+                        )
+                        (local.set $5
+                         (i32.add
+                          (local.get $5)
+                          (i32.const 4)
+                         )
+                        )
+                       )
+                       (block
+                        (local.set $8
+                         (i32.add
+                          (local.get $1)
+                          (i32.const 4)
+                         )
+                        )
+                        (if
+                         (i32.eqz
+                          (i32.load
+                           (local.get $1)
+                          )
+                         )
+                         (local.set $1
+                          (local.get $8)
+                         )
+                        )
+                       )
+                      )
+                     )
+                     (local.set $6
+                      (i32.add
+                       (local.tee $8
+                        (if (result i32)
+                         (local.get $25)
+                         (local.get $7)
+                         (local.get $1)
+                        )
+                       )
+                       (i32.shl
+                        (local.get $14)
+                        (i32.const 2)
+                       )
+                      )
+                     )
+                     (if
+                      (i32.gt_s
+                       (i32.shr_s
+                        (i32.sub
+                         (local.get $5)
+                         (local.get $8)
+                        )
+                        (i32.const 2)
+                       )
+                       (local.get $14)
+                      )
+                      (local.set $5
+                       (local.get $6)
+                      )
+                     )
+                     (i32.store
+                      (local.get $20)
+                      (local.tee $6
+                       (i32.add
+                        (i32.load
+                         (local.get $20)
+                        )
+                        (local.get $13)
+                       )
+                      )
+                     )
+                     (br_if $label$160
+                      (i32.lt_s
+                       (local.get $6)
+                       (i32.const 0)
+                      )
+                     )
+                     (local.set $13
+                      (local.get $5)
+                     )
+                    )
+                   )
+                   (local.set $13
+                    (local.get $8)
+                   )
+                  )
+                  (local.set $25
+                   (local.get $7)
+                  )
+                  (block $label$172
+                   (if
+                    (i32.lt_u
+                     (local.get $1)
+                     (local.get $13)
+                    )
+                    (block
+                     (local.set $5
+                      (i32.mul
+                       (i32.shr_s
+                        (i32.sub
+                         (local.get $25)
+                         (local.get $1)
+                        )
+                        (i32.const 2)
+                       )
+                       (i32.const 9)
+                      )
+                     )
+                     (br_if $label$172
+                      (i32.lt_u
+                       (local.tee $6
+                        (i32.load
+                         (local.get $1)
+                        )
+                       )
+                       (i32.const 10)
+                      )
+                     )
+                     (local.set $8
+                      (i32.const 10)
+                     )
+                     (loop $label$174
+                      (local.set $5
+                       (i32.add
+                        (local.get $5)
+                        (i32.const 1)
+                       )
+                      )
+                      (br_if $label$174
+                       (i32.ge_u
+                        (local.get $6)
+                        (local.tee $8
+                         (i32.mul
+                          (local.get $8)
+                          (i32.const 10)
+                         )
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (local.set $5
+                     (i32.const 0)
+                    )
+                   )
+                  )
+                  (local.set $29
+                   (i32.eq
+                    (local.get $22)
+                    (i32.const 103)
+                   )
+                  )
+                  (local.set $35
+                   (i32.ne
+                    (local.get $18)
+                    (i32.const 0)
+                   )
+                  )
+                  (if
+                   (i32.lt_s
+                    (local.tee $8
+                     (i32.add
+                      (i32.sub
+                       (local.get $18)
+                       (if (result i32)
+                        (i32.ne
+                         (local.get $22)
+                         (i32.const 102)
+                        )
+                        (local.get $5)
+                        (i32.const 0)
+                       )
+                      )
+                      (i32.shr_s
+                       (i32.shl
+                        (i32.and
+                         (local.get $35)
+                         (local.get $29)
+                        )
+                        (i32.const 31)
+                       )
+                       (i32.const 31)
+                      )
+                     )
+                    )
+                    (i32.add
+                     (i32.mul
+                      (i32.shr_s
+                       (i32.sub
+                        (local.get $13)
+                        (local.get $25)
+                       )
+                       (i32.const 2)
+                      )
+                      (i32.const 9)
+                     )
+                     (i32.const -9)
+                    )
+                   )
+                   (block
+                    (if
+                     (i32.lt_s
+                      (local.tee $8
+                       (i32.add
+                        (i32.rem_s
+                         (local.tee $14
+                          (i32.add
+                           (local.get $8)
+                           (i32.const 9216)
+                          )
+                         )
+                         (i32.const 9)
+                        )
+                        (i32.const 1)
+                       )
+                      )
+                      (i32.const 9)
+                     )
+                     (block
+                      (local.set $6
+                       (i32.const 10)
+                      )
+                      (loop $label$180
+                       (local.set $6
+                        (i32.mul
+                         (local.get $6)
+                         (i32.const 10)
+                        )
+                       )
+                       (br_if $label$180
+                        (i32.ne
+                         (local.tee $8
+                          (i32.add
+                           (local.get $8)
+                           (i32.const 1)
+                          )
+                         )
+                         (i32.const 9)
+                        )
+                       )
+                      )
+                     )
+                     (local.set $6
+                      (i32.const 10)
+                     )
+                    )
+                    (local.set $14
+                     (i32.rem_u
+                      (local.tee $22
+                       (i32.load
+                        (local.tee $8
+                         (i32.add
+                          (i32.add
+                           (local.get $7)
+                           (i32.const 4)
+                          )
+                          (i32.shl
+                           (i32.add
+                            (i32.div_s
+                             (local.get $14)
+                             (i32.const 9)
+                            )
+                            (i32.const -1024)
+                           )
+                           (i32.const 2)
+                          )
+                         )
+                        )
+                       )
+                      )
+                      (local.get $6)
+                     )
+                    )
+                    (block $label$182
+                     (if
+                      (i32.eqz
+                       (i32.and
+                        (local.tee $32
+                         (i32.eq
+                          (i32.add
+                           (local.get $8)
+                           (i32.const 4)
+                          )
+                          (local.get $13)
+                         )
+                        )
+                        (i32.eqz
+                         (local.get $14)
+                        )
+                       )
+                      )
+                      (block
+                       (local.set $52
+                        (if (result f64)
+                         (i32.lt_u
+                          (local.get $14)
+                          (local.tee $49
+                           (i32.div_s
+                            (local.get $6)
+                            (i32.const 2)
+                           )
+                          )
+                         )
+                         (f64.const 0.5)
+                         (if (result f64)
+                          (i32.and
+                           (local.get $32)
+                           (i32.eq
+                            (local.get $14)
+                            (local.get $49)
+                           )
+                          )
+                          (f64.const 1)
+                          (f64.const 1.5)
+                         )
+                        )
+                       )
+                       (local.set $53
+                        (if (result f64)
+                         (i32.and
+                          (i32.div_u
+                           (local.get $22)
+                           (local.get $6)
+                          )
+                          (i32.const 1)
+                         )
+                         (f64.const 9007199254740994)
+                         (f64.const 9007199254740992)
+                        )
+                       )
+                       (block $label$190
+                        (if
+                         (local.get $24)
+                         (block
+                          (br_if $label$190
+                           (i32.ne
+                            (i32.load8_s
+                             (local.get $26)
+                            )
+                            (i32.const 45)
+                           )
+                          )
+                          (local.set $53
+                           (f64.neg
+                            (local.get $53)
+                           )
+                          )
+                          (local.set $52
+                           (f64.neg
+                            (local.get $52)
+                           )
+                          )
+                         )
+                        )
+                       )
+                       (i32.store
+                        (local.get $8)
+                        (local.tee $14
+                         (i32.sub
+                          (local.get $22)
+                          (local.get $14)
+                         )
+                        )
+                       )
+                       (br_if $label$182
+                        (f64.eq
+                         (f64.add
+                          (local.get $53)
+                          (local.get $52)
+                         )
+                         (local.get $53)
+                        )
+                       )
+                       (i32.store
+                        (local.get $8)
+                        (local.tee $5
+                         (i32.add
+                          (local.get $14)
+                          (local.get $6)
+                         )
+                        )
+                       )
+                       (if
+                        (i32.gt_u
+                         (local.get $5)
+                         (i32.const 999999999)
+                        )
+                        (loop $label$193
+                         (i32.store
+                          (local.get $8)
+                          (i32.const 0)
+                         )
+                         (if
+                          (i32.lt_u
+                           (local.tee $8
+                            (i32.add
+                             (local.get $8)
+                             (i32.const -4)
+                            )
+                           )
+                           (local.get $1)
+                          )
+                          (i32.store
+                           (local.tee $1
+                            (i32.add
+                             (local.get $1)
+                             (i32.const -4)
+                            )
+                           )
+                           (i32.const 0)
+                          )
+                         )
+                         (i32.store
+                          (local.get $8)
+                          (local.tee $5
+                           (i32.add
+                            (i32.load
+                             (local.get $8)
+                            )
+                            (i32.const 1)
+                           )
+                          )
+                         )
+                         (br_if $label$193
+                          (i32.gt_u
+                           (local.get $5)
+                           (i32.const 999999999)
+                          )
+                         )
+                        )
+                       )
+                       (local.set $5
+                        (i32.mul
+                         (i32.shr_s
+                          (i32.sub
+                           (local.get $25)
+                           (local.get $1)
+                          )
+                          (i32.const 2)
+                         )
+                         (i32.const 9)
+                        )
+                       )
+                       (br_if $label$182
+                        (i32.lt_u
+                         (local.tee $14
+                          (i32.load
+                           (local.get $1)
+                          )
+                         )
+                         (i32.const 10)
+                        )
+                       )
+                       (local.set $6
+                        (i32.const 10)
+                       )
+                       (loop $label$195
+                        (local.set $5
+                         (i32.add
+                          (local.get $5)
+                          (i32.const 1)
+                         )
+                        )
+                        (br_if $label$195
+                         (i32.ge_u
+                          (local.get $14)
+                          (local.tee $6
+                           (i32.mul
+                            (local.get $6)
+                            (i32.const 10)
+                           )
+                          )
+                         )
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (local.set $14
+                     (local.get $1)
+                    )
+                    (local.set $6
+                     (local.get $5)
+                    )
+                    (if
+                     (i32.le_u
+                      (local.get $13)
+                      (local.tee $8
+                       (i32.add
+                        (local.get $8)
+                        (i32.const 4)
+                       )
+                      )
+                     )
+                     (local.set $8
+                      (local.get $13)
+                     )
+                    )
+                   )
+                   (block
+                    (local.set $14
+                     (local.get $1)
+                    )
+                    (local.set $6
+                     (local.get $5)
+                    )
+                    (local.set $8
+                     (local.get $13)
+                    )
+                   )
+                  )
+                  (local.set $32
+                   (i32.sub
+                    (i32.const 0)
+                    (local.get $6)
+                   )
+                  )
+                  (loop $label$198
+                   (block $label$199
+                    (if
+                     (i32.le_u
+                      (local.get $8)
+                      (local.get $14)
+                     )
+                     (block
+                      (local.set $22
+                       (i32.const 0)
+                      )
+                      (br $label$199)
+                     )
+                    )
+                    (if
+                     (i32.load
+                      (local.tee $1
+                       (i32.add
+                        (local.get $8)
+                        (i32.const -4)
+                       )
+                      )
+                     )
+                     (local.set $22
+                      (i32.const 1)
+                     )
+                     (block
+                      (local.set $8
+                       (local.get $1)
+                      )
+                      (br $label$198)
+                     )
+                    )
+                   )
+                  )
+                  (block $label$203
+                   (if
+                    (local.get $29)
+                    (block
+                     (local.set $1
+                      (if (result i32)
+                       (i32.and
+                        (i32.gt_s
+                         (local.tee $1
+                          (i32.add
+                           (i32.xor
+                            (i32.and
+                             (local.get $35)
+                             (i32.const 1)
+                            )
+                            (i32.const 1)
+                           )
+                           (local.get $18)
+                          )
+                         )
+                         (local.get $6)
+                        )
+                        (i32.gt_s
+                         (local.get $6)
+                         (i32.const -5)
+                        )
+                       )
+                       (block (result i32)
+                        (local.set $5
+                         (i32.add
+                          (local.get $9)
+                          (i32.const -1)
+                         )
+                        )
+                        (i32.sub
+                         (i32.add
+                          (local.get $1)
+                          (i32.const -1)
+                         )
+                         (local.get $6)
+                        )
+                       )
+                       (block (result i32)
+                        (local.set $5
+                         (i32.add
+                          (local.get $9)
+                          (i32.const -2)
+                         )
+                        )
+                        (i32.add
+                         (local.get $1)
+                         (i32.const -1)
+                        )
+                       )
+                      )
+                     )
+                     (br_if $label$203
+                      (local.tee $13
+                       (i32.and
+                        (local.get $12)
+                        (i32.const 8)
+                       )
+                      )
+                     )
+                     (block $label$207
+                      (if
+                       (local.get $22)
+                       (block
+                        (if
+                         (i32.eqz
+                          (local.tee $18
+                           (i32.load
+                            (i32.add
+                             (local.get $8)
+                             (i32.const -4)
+                            )
+                           )
+                          )
+                         )
+                         (block
+                          (local.set $9
+                           (i32.const 9)
+                          )
+                          (br $label$207)
+                         )
+                        )
+                        (if
+                         (i32.rem_u
+                          (local.get $18)
+                          (i32.const 10)
+                         )
+                         (block
+                          (local.set $9
+                           (i32.const 0)
+                          )
+                          (br $label$207)
+                         )
+                         (block
+                          (local.set $13
+                           (i32.const 10)
+                          )
+                          (local.set $9
+                           (i32.const 0)
+                          )
+                         )
+                        )
+                        (loop $label$212
+                         (local.set $9
+                          (i32.add
+                           (local.get $9)
+                           (i32.const 1)
+                          )
+                         )
+                         (br_if $label$212
+                          (i32.eqz
+                           (i32.rem_u
+                            (local.get $18)
+                            (local.tee $13
+                             (i32.mul
+                              (local.get $13)
+                              (i32.const 10)
+                             )
+                            )
+                           )
+                          )
+                         )
+                        )
+                       )
+                       (local.set $9
+                        (i32.const 9)
+                       )
+                      )
+                     )
+                     (local.set $18
+                      (i32.add
+                       (i32.mul
+                        (i32.shr_s
+                         (i32.sub
+                          (local.get $8)
+                          (local.get $25)
+                         )
+                         (i32.const 2)
+                        )
+                        (i32.const 9)
+                       )
+                       (i32.const -9)
+                      )
+                     )
+                     (if
+                      (i32.eq
+                       (i32.or
+                        (local.get $5)
+                        (i32.const 32)
+                       )
+                       (i32.const 102)
+                      )
+                      (block
+                       (local.set $13
+                        (i32.const 0)
+                       )
+                       (if
+                        (i32.ge_s
+                         (local.get $1)
+                         (if (result i32)
+                          (i32.lt_s
+                           (local.tee $9
+                            (i32.sub
+                             (local.get $18)
+                             (local.get $9)
+                            )
+                           )
+                           (i32.const 0)
+                          )
+                          (local.tee $9
+                           (i32.const 0)
+                          )
+                          (local.get $9)
+                         )
+                        )
+                        (local.set $1
+                         (local.get $9)
+                        )
+                       )
+                      )
+                      (block
+                       (local.set $13
+                        (i32.const 0)
+                       )
+                       (if
+                        (i32.ge_s
+                         (local.get $1)
+                         (if (result i32)
+                          (i32.lt_s
+                           (local.tee $9
+                            (i32.sub
+                             (i32.add
+                              (local.get $18)
+                              (local.get $6)
+                             )
+                             (local.get $9)
+                            )
+                           )
+                           (i32.const 0)
+                          )
+                          (local.tee $9
+                           (i32.const 0)
+                          )
+                          (local.get $9)
+                         )
+                        )
+                        (local.set $1
+                         (local.get $9)
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (block
+                     (local.set $13
+                      (i32.and
+                       (local.get $12)
+                       (i32.const 8)
+                      )
+                     )
+                     (local.set $1
+                      (local.get $18)
+                     )
+                     (local.set $5
+                      (local.get $9)
+                     )
+                    )
+                   )
+                  )
+                  (if
+                   (local.tee $25
+                    (i32.eq
+                     (i32.or
+                      (local.get $5)
+                      (i32.const 32)
+                     )
+                     (i32.const 102)
+                    )
+                   )
+                   (block
+                    (local.set $9
+                     (i32.const 0)
+                    )
+                    (if
+                     (i32.le_s
+                      (local.get $6)
+                      (i32.const 0)
+                     )
+                     (local.set $6
+                      (i32.const 0)
+                     )
+                    )
+                   )
+                   (block
+                    (if
+                     (i32.lt_s
+                      (i32.sub
+                       (local.get $28)
+                       (local.tee $9
+                        (call $23
+                         (i64.extend_i32_s
+                          (if (result i32)
+                           (i32.lt_s
+                            (local.get $6)
+                            (i32.const 0)
+                           )
+                           (local.get $32)
+                           (local.get $6)
+                          )
+                         )
+                         (local.get $33)
+                        )
+                       )
+                      )
+                      (i32.const 2)
+                     )
+                     (loop $label$229
+                      (i32.store8
+                       (local.tee $9
+                        (i32.add
+                         (local.get $9)
+                         (i32.const -1)
+                        )
+                       )
+                       (i32.const 48)
+                      )
+                      (br_if $label$229
+                       (i32.lt_s
+                        (i32.sub
+                         (local.get $28)
+                         (local.get $9)
+                        )
+                        (i32.const 2)
+                       )
+                      )
+                     )
+                    )
+                    (i32.store8
+                     (i32.add
+                      (local.get $9)
+                      (i32.const -1)
+                     )
+                     (i32.add
+                      (i32.and
+                       (i32.shr_s
+                        (local.get $6)
+                        (i32.const 31)
+                       )
+                       (i32.const 2)
+                      )
+                      (i32.const 43)
+                     )
+                    )
+                    (i32.store8
+                     (local.tee $6
+                      (i32.add
+                       (local.get $9)
+                       (i32.const -2)
+                      )
+                     )
+                     (local.get $5)
+                    )
+                    (local.set $9
+                     (local.get $6)
+                    )
+                    (local.set $6
+                     (i32.sub
+                      (local.get $28)
+                      (local.get $6)
+                     )
+                    )
+                   )
+                  )
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.tee $18
+                    (i32.add
+                     (i32.add
+                      (i32.add
+                       (i32.add
+                        (local.get $24)
+                        (i32.const 1)
+                       )
+                       (local.get $1)
+                      )
+                      (i32.ne
+                       (local.tee $29
+                        (i32.or
+                         (local.get $1)
+                         (local.get $13)
+                        )
+                       )
+                       (i32.const 0)
+                      )
+                     )
+                     (local.get $6)
+                    )
+                   )
+                   (local.get $12)
+                  )
+                  (if
+                   (i32.eqz
+                    (i32.and
+                     (i32.load
+                      (local.get $0)
+                     )
+                     (i32.const 32)
+                    )
+                   )
+                   (drop
+                    (call $21
+                     (local.get $26)
+                     (local.get $24)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (call $25
+                   (local.get $0)
+                   (i32.const 48)
+                   (local.get $10)
+                   (local.get $18)
+                   (i32.xor
+                    (local.get $12)
+                    (i32.const 65536)
+                   )
+                  )
+                  (block $label$231
+                   (if
+                    (local.get $25)
+                    (block
+                     (local.set $6
+                      (local.tee $9
+                       (if (result i32)
+                        (i32.gt_u
+                         (local.get $14)
+                         (local.get $7)
+                        )
+                        (local.get $7)
+                        (local.get $14)
+                       )
+                      )
+                     )
+                     (loop $label$235
+                      (local.set $5
+                       (call $23
+                        (i64.extend_i32_u
+                         (i32.load
+                          (local.get $6)
+                         )
+                        )
+                        (local.get $31)
+                       )
+                      )
+                      (block $label$236
+                       (if
+                        (i32.eq
+                         (local.get $6)
+                         (local.get $9)
+                        )
+                        (block
+                         (br_if $label$236
+                          (i32.ne
+                           (local.get $5)
+                           (local.get $31)
+                          )
+                         )
+                         (i32.store8
+                          (local.get $34)
+                          (i32.const 48)
+                         )
+                         (local.set $5
+                          (local.get $34)
+                         )
+                        )
+                        (block
+                         (br_if $label$236
+                          (i32.le_u
+                           (local.get $5)
+                           (local.get $19)
+                          )
+                         )
+                         (drop
+                          (call $41
+                           (local.get $19)
+                           (i32.const 48)
+                           (i32.sub
+                            (local.get $5)
+                            (local.get $27)
+                           )
+                          )
+                         )
+                         (loop $label$239
+                          (br_if $label$239
+                           (i32.gt_u
+                            (local.tee $5
+                             (i32.add
+                              (local.get $5)
+                              (i32.const -1)
+                             )
+                            )
+                            (local.get $19)
+                           )
+                          )
+                         )
+                        )
+                       )
+                      )
+                      (if
+                       (i32.eqz
+                        (i32.and
+                         (i32.load
+                          (local.get $0)
+                         )
+                         (i32.const 32)
+                        )
+                       )
+                       (drop
+                        (call $21
+                         (local.get $5)
+                         (i32.sub
+                          (local.get $41)
+                          (local.get $5)
+                         )
+                         (local.get $0)
+                        )
+                       )
+                      )
+                      (if
+                       (i32.le_u
+                        (local.tee $5
+                         (i32.add
+                          (local.get $6)
+                          (i32.const 4)
+                         )
+                        )
+                        (local.get $7)
+                       )
+                       (block
+                        (local.set $6
+                         (local.get $5)
+                        )
+                        (br $label$235)
+                       )
+                      )
+                     )
+                     (block $label$242
+                      (if
+                       (local.get $29)
+                       (block
+                        (br_if $label$242
+                         (i32.and
+                          (i32.load
+                           (local.get $0)
+                          )
+                          (i32.const 32)
+                         )
+                        )
+                        (drop
+                         (call $21
+                          (i32.const 1687)
+                          (i32.const 1)
+                          (local.get $0)
+                         )
+                        )
+                       )
+                      )
+                     )
+                     (if
+                      (i32.and
+                       (i32.gt_s
+                        (local.get $1)
+                        (i32.const 0)
+                       )
+                       (i32.lt_u
+                        (local.get $5)
+                        (local.get $8)
+                       )
+                      )
+                      (loop $label$245
+                       (if
+                        (i32.gt_u
+                         (local.tee $7
+                          (call $23
+                           (i64.extend_i32_u
+                            (i32.load
+                             (local.get $5)
+                            )
+                           )
+                           (local.get $31)
+                          )
+                         )
+                         (local.get $19)
+                        )
+                        (block
+                         (drop
+                          (call $41
+                           (local.get $19)
+                           (i32.const 48)
+                           (i32.sub
+                            (local.get $7)
+                            (local.get $27)
+                           )
+                          )
+                         )
+                         (loop $label$247
+                          (br_if $label$247
+                           (i32.gt_u
+                            (local.tee $7
+                             (i32.add
+                              (local.get $7)
+                              (i32.const -1)
+                             )
+                            )
+                            (local.get $19)
+                           )
+                          )
+                         )
+                        )
+                       )
+                       (if
+                        (i32.eqz
+                         (i32.and
+                          (i32.load
+                           (local.get $0)
+                          )
+                          (i32.const 32)
+                         )
+                        )
+                        (drop
+                         (call $21
+                          (local.get $7)
+                          (if (result i32)
+                           (i32.gt_s
+                            (local.get $1)
+                            (i32.const 9)
+                           )
+                           (i32.const 9)
+                           (local.get $1)
+                          )
+                          (local.get $0)
+                         )
+                        )
+                       )
+                       (local.set $7
+                        (i32.add
+                         (local.get $1)
+                         (i32.const -9)
+                        )
+                       )
+                       (if
+                        (i32.and
+                         (i32.gt_s
+                          (local.get $1)
+                          (i32.const 9)
+                         )
+                         (i32.lt_u
+                          (local.tee $5
+                           (i32.add
+                            (local.get $5)
+                            (i32.const 4)
+                           )
+                          )
+                          (local.get $8)
+                         )
+                        )
+                        (block
+                         (local.set $1
+                          (local.get $7)
+                         )
+                         (br $label$245)
+                        )
+                        (local.set $1
+                         (local.get $7)
+                        )
+                       )
+                      )
+                     )
+                     (call $25
+                      (local.get $0)
+                      (i32.const 48)
+                      (i32.add
+                       (local.get $1)
+                       (i32.const 9)
+                      )
+                      (i32.const 9)
+                      (i32.const 0)
+                     )
+                    )
+                    (block
+                     (local.set $5
+                      (i32.add
+                       (local.get $14)
+                       (i32.const 4)
+                      )
+                     )
+                     (if
+                      (i32.eqz
+                       (local.get $22)
+                      )
+                      (local.set $8
+                       (local.get $5)
+                      )
+                     )
+                     (if
+                      (i32.gt_s
+                       (local.get $1)
+                       (i32.const -1)
+                      )
+                      (block
+                       (local.set $13
+                        (i32.eqz
+                         (local.get $13)
+                        )
+                       )
+                       (local.set $7
+                        (local.get $14)
+                       )
+                       (local.set $5
+                        (local.get $1)
+                       )
+                       (loop $label$256
+                        (if
+                         (i32.eq
+                          (local.tee $1
+                           (call $23
+                            (i64.extend_i32_u
+                             (i32.load
+                              (local.get $7)
+                             )
+                            )
+                            (local.get $31)
+                           )
+                          )
+                          (local.get $31)
+                         )
+                         (block
+                          (i32.store8
+                           (local.get $34)
+                           (i32.const 48)
+                          )
+                          (local.set $1
+                           (local.get $34)
+                          )
+                         )
+                        )
+                        (block $label$258
+                         (if
+                          (i32.eq
+                           (local.get $7)
+                           (local.get $14)
+                          )
+                          (block
+                           (if
+                            (i32.eqz
+                             (i32.and
+                              (i32.load
+                               (local.get $0)
+                              )
+                              (i32.const 32)
+                             )
+                            )
+                            (drop
+                             (call $21
+                              (local.get $1)
+                              (i32.const 1)
+                              (local.get $0)
+                             )
+                            )
+                           )
+                           (local.set $1
+                            (i32.add
+                             (local.get $1)
+                             (i32.const 1)
+                            )
+                           )
+                           (br_if $label$258
+                            (i32.and
+                             (local.get $13)
+                             (i32.lt_s
+                              (local.get $5)
+                              (i32.const 1)
+                             )
+                            )
+                           )
+                           (br_if $label$258
+                            (i32.and
+                             (i32.load
+                              (local.get $0)
+                             )
+                             (i32.const 32)
+                            )
+                           )
+                           (drop
+                            (call $21
+                             (i32.const 1687)
+                             (i32.const 1)
+                             (local.get $0)
+                            )
+                           )
+                          )
+                          (block
+                           (br_if $label$258
+                            (i32.le_u
+                             (local.get $1)
+                             (local.get $19)
+                            )
+                           )
+                           (drop
+                            (call $41
+                             (local.get $19)
+                             (i32.const 48)
+                             (i32.add
+                              (local.get $1)
+                              (local.get $43)
+                             )
+                            )
+                           )
+                           (loop $label$262
+                            (br_if $label$262
+                             (i32.gt_u
+                              (local.tee $1
+                               (i32.add
+                                (local.get $1)
+                                (i32.const -1)
+                               )
+                              )
+                              (local.get $19)
+                             )
+                            )
+                           )
+                          )
+                         )
+                        )
+                        (local.set $6
+                         (i32.sub
+                          (local.get $41)
+                          (local.get $1)
+                         )
+                        )
+                        (if
+                         (i32.eqz
+                          (i32.and
+                           (i32.load
+                            (local.get $0)
+                           )
+                           (i32.const 32)
+                          )
+                         )
+                         (drop
+                          (call $21
+                           (local.get $1)
+                           (if (result i32)
+                            (i32.gt_s
+                             (local.get $5)
+                             (local.get $6)
+                            )
+                            (local.get $6)
+                            (local.get $5)
+                           )
+                           (local.get $0)
+                          )
+                         )
+                        )
+                        (br_if $label$256
+                         (i32.and
+                          (i32.lt_u
+                           (local.tee $7
+                            (i32.add
+                             (local.get $7)
+                             (i32.const 4)
+                            )
+                           )
+                           (local.get $8)
+                          )
+                          (i32.gt_s
+                           (local.tee $5
+                            (i32.sub
+                             (local.get $5)
+                             (local.get $6)
+                            )
+                           )
+                           (i32.const -1)
+                          )
+                         )
+                        )
+                        (local.set $1
+                         (local.get $5)
+                        )
+                       )
+                      )
+                     )
+                     (call $25
+                      (local.get $0)
+                      (i32.const 48)
+                      (i32.add
+                       (local.get $1)
+                       (i32.const 18)
+                      )
+                      (i32.const 18)
+                      (i32.const 0)
+                     )
+                     (br_if $label$231
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $21
+                       (local.get $9)
+                       (i32.sub
+                        (local.get $28)
+                        (local.get $9)
+                       )
+                       (local.get $0)
+                      )
+                     )
+                    )
+                   )
+                  )
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.get $18)
+                   (i32.xor
+                    (local.get $12)
+                    (i32.const 8192)
+                   )
+                  )
+                  (if
+                   (i32.ge_s
+                    (local.get $18)
+                    (local.get $10)
+                   )
+                   (local.set $10
+                    (local.get $18)
+                   )
+                  )
+                 )
+                 (block
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.tee $8
+                    (i32.add
+                     (if (result i32)
+                      (local.tee $6
+                       (i32.or
+                        (f64.ne
+                         (local.get $52)
+                         (local.get $52)
+                        )
+                        (i32.const 0)
+                       )
+                      )
+                      (local.tee $24
+                       (i32.const 0)
+                      )
+                      (local.get $24)
+                     )
+                     (i32.const 3)
+                    )
+                   )
+                   (local.get $7)
+                  )
+                  (if
+                   (i32.eqz
+                    (i32.and
+                     (local.tee $1
+                      (i32.load
+                       (local.get $0)
+                      )
+                     )
+                     (i32.const 32)
+                    )
+                   )
+                   (block
+                    (drop
+                     (call $21
+                      (local.get $26)
+                      (local.get $24)
+                      (local.get $0)
+                     )
+                    )
+                    (local.set $1
+                     (i32.load
+                      (local.get $0)
+                     )
+                    )
+                   )
+                  )
+                  (local.set $7
+                   (if (result i32)
+                    (local.tee $5
+                     (i32.ne
+                      (i32.and
+                       (local.get $9)
+                       (i32.const 32)
+                      )
+                      (i32.const 0)
+                     )
+                    )
+                    (i32.const 1671)
+                    (i32.const 1675)
+                   )
+                  )
+                  (local.set $5
+                   (if (result i32)
+                    (local.get $5)
+                    (i32.const 1679)
+                    (i32.const 1683)
+                   )
+                  )
+                  (if
+                   (i32.eqz
+                    (local.get $6)
+                   )
+                   (local.set $5
+                    (local.get $7)
+                   )
+                  )
+                  (if
+                   (i32.eqz
+                    (i32.and
+                     (local.get $1)
+                     (i32.const 32)
+                    )
+                   )
+                   (drop
+                    (call $21
+                     (local.get $5)
+                     (i32.const 3)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (call $25
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.get $8)
+                   (i32.xor
+                    (local.get $12)
+                    (i32.const 8192)
+                   )
+                  )
+                  (if
+                   (i32.ge_s
+                    (local.get $8)
+                    (local.get $10)
+                   )
+                   (local.set $10
+                    (local.get $8)
+                   )
+                  )
+                 )
+                )
+               )
+               (local.set $1
+                (local.get $11)
+               )
+               (br $label$4)
+              )
+              (local.set $7
+               (local.get $5)
+              )
+              (local.set $6
+               (i32.const 0)
+              )
+              (local.set $8
+               (i32.const 1635)
+              )
+              (local.set $5
+               (local.get $21)
+              )
+              (br $label$70)
+             )
+             (local.set $7
+              (i32.and
+               (local.get $9)
+               (i32.const 32)
+              )
+             )
+             (local.set $7
+              (if (result i32)
+               (i64.eq
+                (local.tee $50
+                 (i64.load
+                  (local.get $16)
+                 )
+                )
+                (i64.const 0)
+               )
+               (block (result i32)
+                (local.set $50
+                 (i64.const 0)
+                )
+                (local.get $21)
+               )
+               (block (result i32)
+                (local.set $1
+                 (local.get $21)
+                )
+                (loop $label$280
+                 (i32.store8
+                  (local.tee $1
+                   (i32.add
+                    (local.get $1)
+                    (i32.const -1)
+                   )
+                  )
+                  (i32.or
+                   (i32.load8_u
+                    (i32.add
+                     (i32.and
+                      (i32.wrap_i64
+                       (local.get $50)
+                      )
+                      (i32.const 15)
+                     )
+                     (i32.const 1619)
+                    )
+                   )
+                   (local.get $7)
+                  )
+                 )
+                 (br_if $label$280
+                  (i64.ne
+                   (local.tee $50
+                    (i64.shr_u
+                     (local.get $50)
+                     (i64.const 4)
+                    )
+                   )
+                   (i64.const 0)
+                  )
+                 )
+                )
+                (local.set $50
+                 (i64.load
+                  (local.get $16)
+                 )
+                )
+                (local.get $1)
+               )
+              )
+             )
+             (local.set $8
+              (i32.add
+               (i32.shr_s
+                (local.get $9)
+                (i32.const 4)
+               )
+               (i32.const 1635)
+              )
+             )
+             (if
+              (local.tee $1
+               (i32.or
+                (i32.eqz
+                 (i32.and
+                  (local.get $12)
+                  (i32.const 8)
+                 )
+                )
+                (i64.eq
+                 (local.get $50)
+                 (i64.const 0)
+                )
+               )
+              )
+              (local.set $8
+               (i32.const 1635)
+              )
+             )
+             (local.set $6
+              (if (result i32)
+               (local.get $1)
+               (i32.const 0)
+               (i32.const 2)
+              )
+             )
+             (br $label$71)
+            )
+            (local.set $7
+             (call $23
+              (local.get $50)
+              (local.get $21)
+             )
+            )
+            (br $label$71)
+           )
+           (local.set $14
+            (i32.eqz
+             (local.tee $13
+              (call $17
+               (local.get $1)
+               (i32.const 0)
+               (local.get $5)
+              )
+             )
+            )
+           )
+           (local.set $8
+            (i32.sub
+             (local.get $13)
+             (local.get $1)
+            )
+           )
+           (local.set $9
+            (i32.add
+             (local.get $1)
+             (local.get $5)
+            )
+           )
+           (local.set $12
+            (local.get $7)
+           )
+           (local.set $7
+            (if (result i32)
+             (local.get $14)
+             (local.get $5)
+             (local.get $8)
+            )
+           )
+           (local.set $6
+            (i32.const 0)
+           )
+           (local.set $8
+            (i32.const 1635)
+           )
+           (local.set $5
+            (if (result i32)
+             (local.get $14)
+             (local.get $9)
+             (local.get $13)
+            )
+           )
+           (br $label$70)
+          )
+          (local.set $1
+           (i32.const 0)
+          )
+          (local.set $5
+           (i32.const 0)
+          )
+          (local.set $8
+           (local.get $7)
+          )
+          (loop $label$288
+           (block $label$289
+            (br_if $label$289
+             (i32.eqz
+              (local.tee $9
+               (i32.load
+                (local.get $8)
+               )
+              )
+             )
+            )
+            (br_if $label$289
+             (i32.or
+              (i32.lt_s
+               (local.tee $5
+                (call $26
+                 (local.get $36)
+                 (local.get $9)
+                )
+               )
+               (i32.const 0)
+              )
+              (i32.gt_u
+               (local.get $5)
+               (i32.sub
+                (local.get $6)
+                (local.get $1)
+               )
+              )
+             )
+            )
+            (local.set $8
+             (i32.add
+              (local.get $8)
+              (i32.const 4)
+             )
+            )
+            (br_if $label$288
+             (i32.gt_u
+              (local.get $6)
+              (local.tee $1
+               (i32.add
+                (local.get $5)
+                (local.get $1)
+               )
+              )
+             )
+            )
+           )
+          )
+          (if
+           (i32.lt_s
+            (local.get $5)
+            (i32.const 0)
+           )
+           (block
+            (local.set $15
+             (i32.const -1)
+            )
+            (br $label$5)
+           )
+          )
+          (call $25
+           (local.get $0)
+           (i32.const 32)
+           (local.get $10)
+           (local.get $1)
+           (local.get $12)
+          )
+          (if
+           (local.get $1)
+           (block
+            (local.set $5
+             (i32.const 0)
+            )
+            (loop $label$292
+             (br_if $label$72
+              (i32.eqz
+               (local.tee $8
+                (i32.load
+                 (local.get $7)
+                )
+               )
+              )
+             )
+             (br_if $label$72
+              (i32.gt_s
+               (local.tee $5
+                (i32.add
+                 (local.tee $8
+                  (call $26
+                   (local.get $36)
+                   (local.get $8)
+                  )
+                 )
+                 (local.get $5)
+                )
+               )
+               (local.get $1)
+              )
+             )
+             (if
+              (i32.eqz
+               (i32.and
+                (i32.load
+                 (local.get $0)
+                )
+                (i32.const 32)
+               )
+              )
+              (drop
+               (call $21
+                (local.get $36)
+                (local.get $8)
+                (local.get $0)
+               )
+              )
+             )
+             (local.set $7
+              (i32.add
+               (local.get $7)
+               (i32.const 4)
+              )
+             )
+             (br_if $label$292
+              (i32.lt_u
+               (local.get $5)
+               (local.get $1)
+              )
+             )
+             (br $label$72)
+            )
+           )
+           (block
+            (local.set $1
+             (i32.const 0)
+            )
+            (br $label$72)
+           )
+          )
+         )
+         (call $25
+          (local.get $0)
+          (i32.const 32)
+          (local.get $10)
+          (local.get $1)
+          (i32.xor
+           (local.get $12)
+           (i32.const 8192)
+          )
+         )
+         (if
+          (i32.le_s
+           (local.get $10)
+           (local.get $1)
+          )
+          (local.set $10
+           (local.get $1)
+          )
+         )
+         (local.set $1
+          (local.get $11)
+         )
+         (br $label$4)
+        )
+        (local.set $1
+         (i32.and
+          (local.get $12)
+          (i32.const -65537)
+         )
+        )
+        (if
+         (i32.gt_s
+          (local.get $5)
+          (i32.const -1)
+         )
+         (local.set $12
+          (local.get $1)
+         )
+        )
+        (local.set $5
+         (if (result i32)
+          (i32.or
+           (local.get $5)
+           (local.tee $9
+            (i64.ne
+             (i64.load
+              (local.get $16)
+             )
+             (i64.const 0)
+            )
+           )
+          )
+          (block (result i32)
+           (local.set $1
+            (local.get $7)
+           )
+           (if
+            (i32.gt_s
+             (local.get $5)
+             (local.tee $7
+              (i32.add
+               (i32.xor
+                (i32.and
+                 (local.get $9)
+                 (i32.const 1)
+                )
+                (i32.const 1)
+               )
+               (i32.sub
+                (local.get $38)
+                (local.get $7)
+               )
+              )
+             )
+            )
+            (local.set $7
+             (local.get $5)
+            )
+           )
+           (local.get $21)
+          )
+          (block (result i32)
+           (local.set $1
+            (local.get $21)
+           )
+           (local.set $7
+            (i32.const 0)
+           )
+           (local.get $21)
+          )
+         )
+        )
+       )
+       (call $25
+        (local.get $0)
+        (i32.const 32)
+        (if (result i32)
+         (i32.lt_s
+          (local.get $10)
+          (local.tee $5
+           (i32.add
+            (if (result i32)
+             (i32.lt_s
+              (local.get $7)
+              (local.tee $9
+               (i32.sub
+                (local.get $5)
+                (local.get $1)
+               )
+              )
+             )
+             (local.tee $7
+              (local.get $9)
+             )
+             (local.get $7)
+            )
+            (local.get $6)
+           )
+          )
+         )
+         (local.tee $10
+          (local.get $5)
+         )
+         (local.get $10)
+        )
+        (local.get $5)
+        (local.get $12)
+       )
+       (if
+        (i32.eqz
+         (i32.and
+          (i32.load
+           (local.get $0)
+          )
+          (i32.const 32)
+         )
+        )
+        (drop
+         (call $21
+          (local.get $8)
+          (local.get $6)
+          (local.get $0)
+         )
+        )
+       )
+       (call $25
+        (local.get $0)
+        (i32.const 48)
+        (local.get $10)
+        (local.get $5)
+        (i32.xor
+         (local.get $12)
+         (i32.const 65536)
+        )
+       )
+       (call $25
+        (local.get $0)
+        (i32.const 48)
+        (local.get $7)
+        (local.get $9)
+        (i32.const 0)
+       )
+       (if
+        (i32.eqz
+         (i32.and
+          (i32.load
+           (local.get $0)
+          )
+          (i32.const 32)
+         )
+        )
+        (drop
+         (call $21
+          (local.get $1)
+          (local.get $9)
+          (local.get $0)
+         )
+        )
+       )
+       (call $25
+        (local.get $0)
+        (i32.const 32)
+        (local.get $10)
+        (local.get $5)
+        (i32.xor
+         (local.get $12)
+         (i32.const 8192)
+        )
+       )
+       (local.set $1
+        (local.get $11)
+       )
+       (br $label$4)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.eqz
+      (local.get $0)
+     )
+     (if
+      (local.get $17)
+      (block
+       (local.set $0
+        (i32.const 1)
+       )
+       (loop $label$308
+        (if
+         (local.tee $1
+          (i32.load
+           (i32.add
+            (local.get $4)
+            (i32.shl
+             (local.get $0)
+             (i32.const 2)
+            )
+           )
+          )
+         )
+         (block
+          (call $22
+           (i32.add
+            (local.get $3)
+            (i32.shl
+             (local.get $0)
+             (i32.const 3)
+            )
+           )
+           (local.get $1)
+           (local.get $2)
+          )
+          (br_if $label$308
+           (i32.lt_s
+            (local.tee $0
+             (i32.add
+              (local.get $0)
+              (i32.const 1)
+             )
+            )
+            (i32.const 10)
+           )
+          )
+          (local.set $15
+           (i32.const 1)
+          )
+          (br $label$2)
+         )
+        )
+       )
+       (loop $label$310
+        (if
+         (i32.load
+          (i32.add
+           (local.get $4)
+           (i32.shl
+            (local.get $0)
+            (i32.const 2)
+           )
+          )
+         )
+         (block
+          (local.set $15
+           (i32.const -1)
+          )
+          (br $label$2)
+         )
+        )
+        (br_if $label$310
+         (i32.lt_s
+          (local.tee $0
+           (i32.add
+            (local.get $0)
+            (i32.const 1)
+           )
+          )
+          (i32.const 10)
+         )
+        )
+        (local.set $15
+         (i32.const 1)
+        )
+       )
+      )
+      (local.set $15
+       (i32.const 0)
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $23)
+   )
+   (local.get $15)
+  )
+ )
+ (func $20 (; 33 ;) (type $1) (param $0 i32) (result i32)
+  (i32.const 0)
+ )
+ (func $21 (; 34 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (block $label$1 (result i32)
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (local.tee $3
+       (i32.load
+        (local.tee $4
+         (i32.add
+          (local.get $2)
+          (i32.const 16)
+         )
+        )
+       )
+      )
+     )
+     (if
+      (call $30
+       (local.get $2)
+      )
+      (local.set $3
+       (i32.const 0)
+      )
+      (block
+       (local.set $3
+        (i32.load
+         (local.get $4)
+        )
+       )
+       (br $label$3)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.lt_u
+      (i32.sub
+       (local.get $3)
+       (local.tee $4
+        (i32.load
+         (local.tee $5
+          (i32.add
+           (local.get $2)
+           (i32.const 20)
+          )
+         )
+        )
+       )
+      )
+      (local.get $1)
+     )
+     (block
+      (local.set $3
+       (call_indirect (type $0)
+        (local.get $2)
+        (local.get $0)
+        (local.get $1)
+        (i32.add
+         (i32.and
+          (i32.load offset=36
+           (local.get $2)
+          )
+          (i32.const 3)
+         )
+         (i32.const 2)
+        )
+       )
+      )
+      (br $label$2)
+     )
+    )
+    (local.set $2
+     (block $label$7 (result i32)
+      (if (result i32)
+       (i32.gt_s
+        (i32.load8_s offset=75
+         (local.get $2)
+        )
+        (i32.const -1)
+       )
+       (block (result i32)
+        (local.set $3
+         (local.get $1)
+        )
+        (loop $label$9
+         (drop
+          (br_if $label$7
+           (i32.const 0)
+           (i32.eqz
+            (local.get $3)
+           )
+          )
+         )
+         (if
+          (i32.ne
+           (i32.load8_s
+            (i32.add
+             (local.get $0)
+             (local.tee $6
+              (i32.add
+               (local.get $3)
+               (i32.const -1)
+              )
+             )
+            )
+           )
+           (i32.const 10)
+          )
+          (block
+           (local.set $3
+            (local.get $6)
+           )
+           (br $label$9)
+          )
+         )
+        )
+        (br_if $label$2
+         (i32.lt_u
+          (call_indirect (type $0)
+           (local.get $2)
+           (local.get $0)
+           (local.get $3)
+           (i32.add
+            (i32.and
+             (i32.load offset=36
+              (local.get $2)
+             )
+             (i32.const 3)
+            )
+            (i32.const 2)
+           )
+          )
+          (local.get $3)
+         )
+        )
+        (local.set $4
+         (i32.load
+          (local.get $5)
+         )
+        )
+        (local.set $1
+         (i32.sub
+          (local.get $1)
+          (local.get $3)
+         )
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (local.get $3)
+         )
+        )
+        (local.get $3)
+       )
+       (i32.const 0)
+      )
+     )
+    )
+    (drop
+     (call $42
+      (local.get $4)
+      (local.get $0)
+      (local.get $1)
+     )
+    )
+    (i32.store
+     (local.get $5)
+     (i32.add
+      (i32.load
+       (local.get $5)
+      )
+      (local.get $1)
+     )
+    )
+    (local.set $3
+     (i32.add
+      (local.get $2)
+      (local.get $1)
+     )
+    )
+   )
+   (local.get $3)
+  )
+ )
+ (func $22 (; 35 ;) (type $8) (param $0 i32) (param $1 i32) (param $2 i32)
+  (local $3 i32)
+  (local $4 i64)
+  (local $5 f64)
+  (block $label$1
+   (if
+    (i32.le_u
+     (local.get $1)
+     (i32.const 20)
+    )
+    (block $label$3
+     (block $label$4
+      (block $label$5
+       (block $label$6
+        (block $label$7
+         (block $label$8
+          (block $label$9
+           (block $label$10
+            (block $label$11
+             (block $label$12
+              (block $label$13
+               (br_table $label$13 $label$12 $label$11 $label$10 $label$9 $label$8 $label$7 $label$6 $label$5 $label$4 $label$3
+                (i32.sub
+                 (local.get $1)
+                 (i32.const 9)
+                )
+               )
+              )
+              (local.set $3
+               (i32.load
+                (local.tee $1
+                 (i32.and
+                  (i32.add
+                   (i32.load
+                    (local.get $2)
+                   )
+                   (i32.const 3)
+                  )
+                  (i32.const -4)
+                 )
+                )
+               )
+              )
+              (i32.store
+               (local.get $2)
+               (i32.add
+                (local.get $1)
+                (i32.const 4)
+               )
+              )
+              (i32.store
+               (local.get $0)
+               (local.get $3)
+              )
+              (br $label$1)
+             )
+             (local.set $3
+              (i32.load
+               (local.tee $1
+                (i32.and
+                 (i32.add
+                  (i32.load
+                   (local.get $2)
+                  )
+                  (i32.const 3)
+                 )
+                 (i32.const -4)
+                )
+               )
+              )
+             )
+             (i32.store
+              (local.get $2)
+              (i32.add
+               (local.get $1)
+               (i32.const 4)
+              )
+             )
+             (i64.store
+              (local.get $0)
+              (i64.extend_i32_s
+               (local.get $3)
+              )
+             )
+             (br $label$1)
+            )
+            (local.set $3
+             (i32.load
+              (local.tee $1
+               (i32.and
+                (i32.add
+                 (i32.load
+                  (local.get $2)
+                 )
+                 (i32.const 3)
+                )
+                (i32.const -4)
+               )
+              )
+             )
+            )
+            (i32.store
+             (local.get $2)
+             (i32.add
+              (local.get $1)
+              (i32.const 4)
+             )
+            )
+            (i64.store
+             (local.get $0)
+             (i64.extend_i32_u
+              (local.get $3)
+             )
+            )
+            (br $label$1)
+           )
+           (local.set $4
+            (i64.load
+             (local.tee $1
+              (i32.and
+               (i32.add
+                (i32.load
+                 (local.get $2)
+                )
+                (i32.const 7)
+               )
+               (i32.const -8)
+              )
+             )
+            )
+           )
+           (i32.store
+            (local.get $2)
+            (i32.add
+             (local.get $1)
+             (i32.const 8)
+            )
+           )
+           (i64.store
+            (local.get $0)
+            (local.get $4)
+           )
+           (br $label$1)
+          )
+          (local.set $3
+           (i32.load
+            (local.tee $1
+             (i32.and
+              (i32.add
+               (i32.load
+                (local.get $2)
+               )
+               (i32.const 3)
+              )
+              (i32.const -4)
+             )
+            )
+           )
+          )
+          (i32.store
+           (local.get $2)
+           (i32.add
+            (local.get $1)
+            (i32.const 4)
+           )
+          )
+          (i64.store
+           (local.get $0)
+           (i64.extend_i32_s
+            (i32.shr_s
+             (i32.shl
+              (i32.and
+               (local.get $3)
+               (i32.const 65535)
+              )
+              (i32.const 16)
+             )
+             (i32.const 16)
+            )
+           )
+          )
+          (br $label$1)
+         )
+         (local.set $3
+          (i32.load
+           (local.tee $1
+            (i32.and
+             (i32.add
+              (i32.load
+               (local.get $2)
+              )
+              (i32.const 3)
+             )
+             (i32.const -4)
+            )
+           )
+          )
+         )
+         (i32.store
+          (local.get $2)
+          (i32.add
+           (local.get $1)
+           (i32.const 4)
+          )
+         )
+         (i64.store
+          (local.get $0)
+          (i64.extend_i32_u
+           (i32.and
+            (local.get $3)
+            (i32.const 65535)
+           )
+          )
+         )
+         (br $label$1)
+        )
+        (local.set $3
+         (i32.load
+          (local.tee $1
+           (i32.and
+            (i32.add
+             (i32.load
+              (local.get $2)
+             )
+             (i32.const 3)
+            )
+            (i32.const -4)
+           )
+          )
+         )
+        )
+        (i32.store
+         (local.get $2)
+         (i32.add
+          (local.get $1)
+          (i32.const 4)
+         )
+        )
+        (i64.store
+         (local.get $0)
+         (i64.extend_i32_s
+          (i32.shr_s
+           (i32.shl
+            (i32.and
+             (local.get $3)
+             (i32.const 255)
+            )
+            (i32.const 24)
+           )
+           (i32.const 24)
+          )
+         )
+        )
+        (br $label$1)
+       )
+       (local.set $3
+        (i32.load
+         (local.tee $1
+          (i32.and
+           (i32.add
+            (i32.load
+             (local.get $2)
+            )
+            (i32.const 3)
+           )
+           (i32.const -4)
+          )
+         )
+        )
+       )
+       (i32.store
+        (local.get $2)
+        (i32.add
+         (local.get $1)
+         (i32.const 4)
+        )
+       )
+       (i64.store
+        (local.get $0)
+        (i64.extend_i32_u
+         (i32.and
+          (local.get $3)
+          (i32.const 255)
+         )
+        )
+       )
+       (br $label$1)
+      )
+      (local.set $5
+       (f64.load
+        (local.tee $1
+         (i32.and
+          (i32.add
+           (i32.load
+            (local.get $2)
+           )
+           (i32.const 7)
+          )
+          (i32.const -8)
+         )
+        )
+       )
+      )
+      (i32.store
+       (local.get $2)
+       (i32.add
+        (local.get $1)
+        (i32.const 8)
+       )
+      )
+      (f64.store
+       (local.get $0)
+       (local.get $5)
+      )
+      (br $label$1)
+     )
+     (local.set $5
+      (f64.load
+       (local.tee $1
+        (i32.and
+         (i32.add
+          (i32.load
+           (local.get $2)
+          )
+          (i32.const 7)
+         )
+         (i32.const -8)
+        )
+       )
+      )
+     )
+     (i32.store
+      (local.get $2)
+      (i32.add
+       (local.get $1)
+       (i32.const 8)
+      )
+     )
+     (f64.store
+      (local.get $0)
+      (local.get $5)
+     )
+    )
+   )
+  )
+ )
+ (func $23 (; 36 ;) (type $9) (param $0 i64) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i64)
+  (block $label$1 (result i32)
+   (local.set $2
+    (i32.wrap_i64
+     (local.get $0)
+    )
+   )
+   (if
+    (i64.gt_u
+     (local.get $0)
+     (i64.const 4294967295)
+    )
+    (block
+     (loop $label$3
+      (i64.store8
+       (local.tee $1
+        (i32.add
+         (local.get $1)
+         (i32.const -1)
+        )
+       )
+       (i64.or
+        (i64.rem_u
+         (local.get $0)
+         (i64.const 10)
+        )
+        (i64.const 48)
+       )
+      )
+      (local.set $4
+       (i64.div_u
+        (local.get $0)
+        (i64.const 10)
+       )
+      )
+      (if
+       (i64.gt_u
+        (local.get $0)
+        (i64.const 42949672959)
+       )
+       (block
+        (local.set $0
+         (local.get $4)
+        )
+        (br $label$3)
+       )
+      )
+     )
+     (local.set $2
+      (i32.wrap_i64
+       (local.get $4)
+      )
+     )
+    )
+   )
+   (if
+    (local.get $2)
+    (loop $label$6
+     (i32.store8
+      (local.tee $1
+       (i32.add
+        (local.get $1)
+        (i32.const -1)
+       )
+      )
+      (i32.or
+       (i32.rem_u
+        (local.get $2)
+        (i32.const 10)
+       )
+       (i32.const 48)
+      )
+     )
+     (local.set $3
+      (i32.div_u
+       (local.get $2)
+       (i32.const 10)
+      )
+     )
+     (if
+      (i32.ge_u
+       (local.get $2)
+       (i32.const 10)
+      )
+      (block
+       (local.set $2
+        (local.get $3)
+       )
+       (br $label$6)
+      )
+     )
+    )
+   )
+   (local.get $1)
+  )
+ )
+ (func $24 (; 37 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (i32.const 0)
+   )
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (loop $label$5
+       (br_if $label$4
+        (i32.eq
+         (i32.load8_u
+          (i32.add
+           (local.get $1)
+           (i32.const 1689)
+          )
+         )
+         (local.get $0)
+        )
+       )
+       (br_if $label$5
+        (i32.ne
+         (local.tee $1
+          (i32.add
+           (local.get $1)
+           (i32.const 1)
+          )
+         )
+         (i32.const 87)
+        )
+       )
+       (local.set $1
+        (i32.const 87)
+       )
+       (local.set $0
+        (i32.const 1777)
+       )
+       (br $label$3)
+      )
+     )
+     (if
+      (local.get $1)
+      (block
+       (local.set $0
+        (i32.const 1777)
+       )
+       (br $label$3)
+      )
+      (local.set $0
+       (i32.const 1777)
+      )
+     )
+     (br $label$2)
+    )
+    (loop $label$8
+     (local.set $2
+      (local.get $0)
+     )
+     (loop $label$9
+      (local.set $0
+       (i32.add
+        (local.get $2)
+        (i32.const 1)
+       )
+      )
+      (if
+       (i32.load8_s
+        (local.get $2)
+       )
+       (block
+        (local.set $2
+         (local.get $0)
+        )
+        (br $label$9)
+       )
+      )
+     )
+     (br_if $label$8
+      (local.tee $1
+       (i32.add
+        (local.get $1)
+        (i32.const -1)
+       )
+      )
+     )
+    )
+   )
+   (local.get $0)
+  )
+ )
+ (func $25 (; 38 ;) (type $10) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (param $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (block $label$1
+   (local.set $7
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 256)
+    )
+   )
+   (local.set $6
+    (local.get $7)
+   )
+   (block $label$2
+    (if
+     (i32.and
+      (i32.gt_s
+       (local.get $2)
+       (local.get $3)
+      )
+      (i32.eqz
+       (i32.and
+        (local.get $4)
+        (i32.const 73728)
+       )
+      )
+     )
+     (block
+      (drop
+       (call $41
+        (local.get $6)
+        (local.get $1)
+        (if (result i32)
+         (i32.gt_u
+          (local.tee $5
+           (i32.sub
+            (local.get $2)
+            (local.get $3)
+           )
+          )
+          (i32.const 256)
+         )
+         (i32.const 256)
+         (local.get $5)
+        )
+       )
+      )
+      (local.set $4
+       (i32.eqz
+        (i32.and
+         (local.tee $1
+          (i32.load
+           (local.get $0)
+          )
+         )
+         (i32.const 32)
+        )
+       )
+      )
+      (if
+       (i32.gt_u
+        (local.get $5)
+        (i32.const 255)
+       )
+       (block
+        (loop $label$7
+         (if
+          (local.get $4)
+          (block
+           (drop
+            (call $21
+             (local.get $6)
+             (i32.const 256)
+             (local.get $0)
+            )
+           )
+           (local.set $1
+            (i32.load
+             (local.get $0)
+            )
+           )
+          )
+         )
+         (local.set $4
+          (i32.eqz
+           (i32.and
+            (local.get $1)
+            (i32.const 32)
+           )
+          )
+         )
+         (br_if $label$7
+          (i32.gt_u
+           (local.tee $5
+            (i32.add
+             (local.get $5)
+             (i32.const -256)
+            )
+           )
+           (i32.const 255)
+          )
+         )
+        )
+        (br_if $label$2
+         (i32.eqz
+          (local.get $4)
+         )
+        )
+        (local.set $5
+         (i32.and
+          (i32.sub
+           (local.get $2)
+           (local.get $3)
+          )
+          (i32.const 255)
+         )
+        )
+       )
+       (br_if $label$2
+        (i32.eqz
+         (local.get $4)
+        )
+       )
+      )
+      (drop
+       (call $21
+        (local.get $6)
+        (local.get $5)
+        (local.get $0)
+       )
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $7)
+   )
+  )
+ )
+ (func $26 (; 39 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (if (result i32)
+   (local.get $0)
+   (call $29
+    (local.get $0)
+    (local.get $1)
+    (i32.const 0)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $27 (; 40 ;) (type $11) (param $0 f64) (param $1 i32) (result f64)
+  (call $28
+   (local.get $0)
+   (local.get $1)
+  )
+ )
+ (func $28 (; 41 ;) (type $11) (param $0 f64) (param $1 i32) (result f64)
+  (local $2 i64)
+  (local $3 i64)
+  (block $label$1 (result f64)
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (block $label$5
+       (br_table $label$5 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$4 $label$3
+        (i32.sub
+         (i32.shr_s
+          (i32.shl
+           (i32.and
+            (i32.and
+             (i32.wrap_i64
+              (local.tee $3
+               (i64.shr_u
+                (local.tee $2
+                 (i64.reinterpret_f64
+                  (local.get $0)
+                 )
+                )
+                (i64.const 52)
+               )
+              )
+             )
+             (i32.const 65535)
+            )
+            (i32.const 2047)
+           )
+           (i32.const 16)
+          )
+          (i32.const 16)
+         )
+         (i32.const 0)
+        )
+       )
+      )
+      (i32.store
+       (local.get $1)
+       (if (result i32)
+        (f64.ne
+         (local.get $0)
+         (f64.const 0)
+        )
+        (block (result i32)
+         (local.set $0
+          (call $28
+           (f64.mul
+            (local.get $0)
+            (f64.const 18446744073709551615)
+           )
+           (local.get $1)
+          )
+         )
+         (i32.add
+          (i32.load
+           (local.get $1)
+          )
+          (i32.const -64)
+         )
+        )
+        (i32.const 0)
+       )
+      )
+      (br $label$2)
+     )
+     (br $label$2)
+    )
+    (i32.store
+     (local.get $1)
+     (i32.add
+      (i32.and
+       (i32.wrap_i64
+        (local.get $3)
+       )
+       (i32.const 2047)
+      )
+      (i32.const -1022)
+     )
+    )
+    (local.set $0
+     (f64.reinterpret_i64
+      (i64.or
+       (i64.and
+        (local.get $2)
+        (i64.const -9218868437227405313)
+       )
+       (i64.const 4602678819172646912)
+      )
+     )
+    )
+   )
+   (local.get $0)
+  )
+ )
+ (func $29 (; 42 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (block $label$1 (result i32)
+   (if (result i32)
+    (local.get $0)
+    (block (result i32)
+     (if
+      (i32.lt_u
+       (local.get $1)
+       (i32.const 128)
+      )
+      (block
+       (i32.store8
+        (local.get $0)
+        (local.get $1)
+       )
+       (br $label$1
+        (i32.const 1)
+       )
+      )
+     )
+     (if
+      (i32.lt_u
+       (local.get $1)
+       (i32.const 2048)
+      )
+      (block
+       (i32.store8
+        (local.get $0)
+        (i32.or
+         (i32.shr_u
+          (local.get $1)
+          (i32.const 6)
+         )
+         (i32.const 192)
+        )
+       )
+       (i32.store8 offset=1
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (local.get $1)
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (br $label$1
+        (i32.const 2)
+       )
+      )
+     )
+     (if
+      (i32.or
+       (i32.lt_u
+        (local.get $1)
+        (i32.const 55296)
+       )
+       (i32.eq
+        (i32.and
+         (local.get $1)
+         (i32.const -8192)
+        )
+        (i32.const 57344)
+       )
+      )
+      (block
+       (i32.store8
+        (local.get $0)
+        (i32.or
+         (i32.shr_u
+          (local.get $1)
+          (i32.const 12)
+         )
+         (i32.const 224)
+        )
+       )
+       (i32.store8 offset=1
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (i32.shr_u
+           (local.get $1)
+           (i32.const 6)
+          )
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.store8 offset=2
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (local.get $1)
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (br $label$1
+        (i32.const 3)
+       )
+      )
+     )
+     (if (result i32)
+      (i32.lt_u
+       (i32.add
+        (local.get $1)
+        (i32.const -65536)
+       )
+       (i32.const 1048576)
+      )
+      (block (result i32)
+       (i32.store8
+        (local.get $0)
+        (i32.or
+         (i32.shr_u
+          (local.get $1)
+          (i32.const 18)
+         )
+         (i32.const 240)
+        )
+       )
+       (i32.store8 offset=1
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (i32.shr_u
+           (local.get $1)
+           (i32.const 12)
+          )
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.store8 offset=2
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (i32.shr_u
+           (local.get $1)
+           (i32.const 6)
+          )
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.store8 offset=3
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (local.get $1)
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.const 4)
+      )
+      (block (result i32)
+       (i32.store
+        (call $12)
+        (i32.const 84)
+       )
+       (i32.const -1)
+      )
+     )
+    )
+    (i32.const 1)
+   )
+  )
+ )
+ (func $30 (; 43 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (i32.load8_s
+     (local.tee $2
+      (i32.add
+       (local.get $0)
+       (i32.const 74)
+      )
+     )
+    )
+   )
+   (i32.store8
+    (local.get $2)
+    (i32.or
+     (i32.add
+      (local.get $1)
+      (i32.const 255)
+     )
+     (local.get $1)
+    )
+   )
+   (local.tee $0
+    (if (result i32)
+     (i32.and
+      (local.tee $1
+       (i32.load
+        (local.get $0)
+       )
+      )
+      (i32.const 8)
+     )
+     (block (result i32)
+      (i32.store
+       (local.get $0)
+       (i32.or
+        (local.get $1)
+        (i32.const 32)
+       )
+      )
+      (i32.const -1)
+     )
+     (block (result i32)
+      (i32.store offset=8
+       (local.get $0)
+       (i32.const 0)
+      )
+      (i32.store offset=4
+       (local.get $0)
+       (i32.const 0)
+      )
+      (i32.store offset=28
+       (local.get $0)
+       (local.tee $1
+        (i32.load offset=44
+         (local.get $0)
+        )
+       )
+      )
+      (i32.store offset=20
+       (local.get $0)
+       (local.get $1)
+      )
+      (i32.store offset=16
+       (local.get $0)
+       (i32.add
+        (local.get $1)
+        (i32.load offset=48
+         (local.get $0)
+        )
+       )
+      )
+      (i32.const 0)
+     )
+    )
+   )
+  )
+ )
+ (func $31 (; 44 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (block $label$1 (result i32)
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (i32.eqz
+       (i32.and
+        (local.tee $2
+         (local.get $0)
+        )
+        (i32.const 3)
+       )
+      )
+     )
+     (local.set $1
+      (local.get $2)
+     )
+     (loop $label$4
+      (if
+       (i32.eqz
+        (i32.load8_s
+         (local.get $0)
+        )
+       )
+       (block
+        (local.set $0
+         (local.get $1)
+        )
+        (br $label$2)
+       )
+      )
+      (br_if $label$4
+       (i32.and
+        (local.tee $1
+         (local.tee $0
+          (i32.add
+           (local.get $0)
+           (i32.const 1)
+          )
+         )
+        )
+        (i32.const 3)
+       )
+      )
+      (br $label$3)
+     )
+    )
+    (loop $label$6
+     (local.set $1
+      (i32.add
+       (local.get $0)
+       (i32.const 4)
+      )
+     )
+     (if
+      (i32.eqz
+       (i32.and
+        (i32.xor
+         (i32.and
+          (local.tee $3
+           (i32.load
+            (local.get $0)
+           )
+          )
+          (i32.const -2139062144)
+         )
+         (i32.const -2139062144)
+        )
+        (i32.add
+         (local.get $3)
+         (i32.const -16843009)
+        )
+       )
+      )
+      (block
+       (local.set $0
+        (local.get $1)
+       )
+       (br $label$6)
+      )
+     )
+    )
+    (if
+     (i32.shr_s
+      (i32.shl
+       (i32.and
+        (local.get $3)
+        (i32.const 255)
+       )
+       (i32.const 24)
+      )
+      (i32.const 24)
+     )
+     (loop $label$9
+      (br_if $label$9
+       (i32.load8_s
+        (local.tee $0
+         (i32.add
+          (local.get $0)
+          (i32.const 1)
+         )
+        )
+       )
+      )
+     )
+    )
+   )
+   (i32.sub
+    (local.get $0)
+    (local.get $2)
+   )
+  )
+ )
+ (func $32 (; 45 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (block $label$1 (result i32)
+   (local.set $3
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (i32.store8
+    (local.tee $4
+     (local.get $3)
+    )
+    (local.tee $7
+     (i32.and
+      (local.get $1)
+      (i32.const 255)
+     )
+    )
+   )
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (local.tee $5
+       (i32.load
+        (local.tee $2
+         (i32.add
+          (local.get $0)
+          (i32.const 16)
+         )
+        )
+       )
+      )
+     )
+     (if
+      (call $30
+       (local.get $0)
+      )
+      (local.set $1
+       (i32.const -1)
+      )
+      (block
+       (local.set $5
+        (i32.load
+         (local.get $2)
+        )
+       )
+       (br $label$3)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.lt_u
+      (local.tee $6
+       (i32.load
+        (local.tee $2
+         (i32.add
+          (local.get $0)
+          (i32.const 20)
+         )
+        )
+       )
+      )
+      (local.get $5)
+     )
+     (if
+      (i32.ne
+       (local.tee $1
+        (i32.and
+         (local.get $1)
+         (i32.const 255)
+        )
+       )
+       (i32.load8_s offset=75
+        (local.get $0)
+       )
+      )
+      (block
+       (i32.store
+        (local.get $2)
+        (i32.add
+         (local.get $6)
+         (i32.const 1)
+        )
+       )
+       (i32.store8
+        (local.get $6)
+        (local.get $7)
+       )
+       (br $label$2)
+      )
+     )
+    )
+    (local.set $1
+     (if (result i32)
+      (i32.eq
+       (call_indirect (type $0)
+        (local.get $0)
+        (local.get $4)
+        (i32.const 1)
+        (i32.add
+         (i32.and
+          (i32.load offset=36
+           (local.get $0)
+          )
+          (i32.const 3)
+         )
+         (i32.const 2)
+        )
+       )
+       (i32.const 1)
+      )
+      (i32.load8_u
+       (local.get $4)
+      )
+      (i32.const -1)
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $3)
+   )
+   (local.get $1)
+  )
+ )
+ (func $33 (; 46 ;) (type $12) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (result i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (i32.mul
+     (local.get $2)
+     (local.get $1)
+    )
+   )
+   (if
+    (i32.gt_s
+     (i32.load offset=76
+      (local.get $3)
+     )
+     (i32.const -1)
+    )
+    (block
+     (local.set $5
+      (i32.eqz
+       (call $20
+        (local.get $3)
+       )
+      )
+     )
+     (local.set $0
+      (call $21
+       (local.get $0)
+       (local.get $4)
+       (local.get $3)
+      )
+     )
+     (if
+      (i32.eqz
+       (local.get $5)
+      )
+      (call $13
+       (local.get $3)
+      )
+     )
+    )
+    (local.set $0
+     (call $21
+      (local.get $0)
+      (local.get $4)
+      (local.get $3)
+     )
+    )
+   )
+   (if
+    (i32.ne
+     (local.get $0)
+     (local.get $4)
+    )
+    (local.set $2
+     (i32.div_u
+      (local.get $0)
+      (local.get $1)
+     )
+    )
+   )
+   (local.get $2)
+  )
+ )
+ (func $34 (; 47 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (block $label$1 (result i32)
+   (local.set $2
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (i32.store
+    (local.tee $3
+     (local.get $2)
+    )
+    (local.get $1)
+   )
+   (local.set $0
+    (call $18
+     (i32.load
+      (i32.const 1024)
+     )
+     (local.get $0)
+     (local.get $3)
+    )
+   )
+   (global.set $global$1
+    (local.get $2)
+   )
+   (local.get $0)
+  )
+ )
+ (func $35 (; 48 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (block $label$1 (result i32)
+   (local.set $2
+    (if (result i32)
+     (i32.gt_s
+      (i32.load offset=76
+       (local.tee $1
+        (i32.load
+         (i32.const 1024)
+        )
+       )
+      )
+      (i32.const -1)
+     )
+     (call $20
+      (local.get $1)
+     )
+     (i32.const 0)
+    )
+   )
+   (local.set $0
+    (block $label$4 (result i32)
+     (if (result i32)
+      (i32.lt_s
+       (call $36
+        (local.get $0)
+        (local.get $1)
+       )
+       (i32.const 0)
+      )
+      (i32.const 1)
+      (block (result i32)
+       (if
+        (i32.ne
+         (i32.load8_s offset=75
+          (local.get $1)
+         )
+         (i32.const 10)
+        )
+        (if
+         (i32.lt_u
+          (local.tee $0
+           (i32.load
+            (local.tee $3
+             (i32.add
+              (local.get $1)
+              (i32.const 20)
+             )
+            )
+           )
+          )
+          (i32.load offset=16
+           (local.get $1)
+          )
+         )
+         (block
+          (i32.store
+           (local.get $3)
+           (i32.add
+            (local.get $0)
+            (i32.const 1)
+           )
+          )
+          (i32.store8
+           (local.get $0)
+           (i32.const 10)
+          )
+          (br $label$4
+           (i32.const 0)
+          )
+         )
+        )
+       )
+       (i32.lt_s
+        (call $32
+         (local.get $1)
+         (i32.const 10)
+        )
+        (i32.const 0)
+       )
+      )
+     )
+    )
+   )
+   (if
+    (local.get $2)
+    (call $13
+     (local.get $1)
+    )
+   )
+   (i32.shr_s
+    (i32.shl
+     (local.get $0)
+     (i32.const 31)
+    )
+    (i32.const 31)
+   )
+  )
+ )
+ (func $36 (; 49 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (i32.add
+   (call $33
+    (local.get $0)
+    (call $31
+     (local.get $0)
+    )
+    (i32.const 1)
+    (local.get $1)
+   )
+   (i32.const -1)
+  )
+ )
+ (func $37 (; 50 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (local $16 i32)
+  (local $17 i32)
+  (local $18 i32)
+  (local $19 i32)
+  (local $20 i32)
+  (local $21 i32)
+  (block $label$1 (result i32)
+   (local.set $14
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (local.set $18
+    (local.get $14)
+   )
+   (block $label$2
+    (if
+     (i32.lt_u
+      (local.get $0)
+      (i32.const 245)
+     )
+     (block
+      (local.set $3
+       (i32.and
+        (i32.add
+         (local.get $0)
+         (i32.const 11)
+        )
+        (i32.const -8)
+       )
+      )
+      (if
+       (i32.and
+        (local.tee $0
+         (i32.shr_u
+          (local.tee $8
+           (i32.load
+            (i32.const 3636)
+           )
+          )
+          (local.tee $2
+           (i32.shr_u
+            (if (result i32)
+             (i32.lt_u
+              (local.get $0)
+              (i32.const 11)
+             )
+             (local.tee $3
+              (i32.const 16)
+             )
+             (local.get $3)
+            )
+            (i32.const 3)
+           )
+          )
+         )
+        )
+        (i32.const 3)
+       )
+       (block
+        (local.set $4
+         (i32.load
+          (local.tee $1
+           (i32.add
+            (local.tee $7
+             (i32.load
+              (local.tee $3
+               (i32.add
+                (local.tee $2
+                 (i32.add
+                  (i32.shl
+                   (i32.shl
+                    (local.tee $5
+                     (i32.add
+                      (i32.xor
+                       (i32.and
+                        (local.get $0)
+                        (i32.const 1)
+                       )
+                       (i32.const 1)
+                      )
+                      (local.get $2)
+                     )
+                    )
+                    (i32.const 1)
+                   )
+                   (i32.const 2)
+                  )
+                  (i32.const 3676)
+                 )
+                )
+                (i32.const 8)
+               )
+              )
+             )
+            )
+            (i32.const 8)
+           )
+          )
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $2)
+          (local.get $4)
+         )
+         (i32.store
+          (i32.const 3636)
+          (i32.and
+           (local.get $8)
+           (i32.xor
+            (i32.shl
+             (i32.const 1)
+             (local.get $5)
+            )
+            (i32.const -1)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $4)
+            (i32.load
+             (i32.const 3652)
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $0
+              (i32.add
+               (local.get $4)
+               (i32.const 12)
+              )
+             )
+            )
+            (local.get $7)
+           )
+           (block
+            (i32.store
+             (local.get $0)
+             (local.get $2)
+            )
+            (i32.store
+             (local.get $3)
+             (local.get $4)
+            )
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (i32.store offset=4
+         (local.get $7)
+         (i32.or
+          (local.tee $0
+           (i32.shl
+            (local.get $5)
+            (i32.const 3)
+           )
+          )
+          (i32.const 3)
+         )
+        )
+        (i32.store
+         (local.tee $0
+          (i32.add
+           (i32.add
+            (local.get $7)
+            (local.get $0)
+           )
+           (i32.const 4)
+          )
+         )
+         (i32.or
+          (i32.load
+           (local.get $0)
+          )
+          (i32.const 1)
+         )
+        )
+        (global.set $global$1
+         (local.get $14)
+        )
+        (return
+         (local.get $1)
+        )
+       )
+      )
+      (if
+       (i32.gt_u
+        (local.get $3)
+        (local.tee $16
+         (i32.load
+          (i32.const 3644)
+         )
+        )
+       )
+       (block
+        (if
+         (local.get $0)
+         (block
+          (local.set $5
+           (i32.and
+            (i32.shr_u
+             (local.tee $0
+              (i32.add
+               (i32.and
+                (local.tee $0
+                 (i32.and
+                  (i32.shl
+                   (local.get $0)
+                   (local.get $2)
+                  )
+                  (i32.or
+                   (local.tee $0
+                    (i32.shl
+                     (i32.const 2)
+                     (local.get $2)
+                    )
+                   )
+                   (i32.sub
+                    (i32.const 0)
+                    (local.get $0)
+                   )
+                  )
+                 )
+                )
+                (i32.sub
+                 (i32.const 0)
+                 (local.get $0)
+                )
+               )
+               (i32.const -1)
+              )
+             )
+             (i32.const 12)
+            )
+            (i32.const 16)
+           )
+          )
+          (local.set $12
+           (i32.load
+            (local.tee $5
+             (i32.add
+              (local.tee $9
+               (i32.load
+                (local.tee $2
+                 (i32.add
+                  (local.tee $4
+                   (i32.add
+                    (i32.shl
+                     (i32.shl
+                      (local.tee $11
+                       (i32.add
+                        (i32.or
+                         (i32.or
+                          (i32.or
+                           (i32.or
+                            (local.tee $0
+                             (i32.and
+                              (i32.shr_u
+                               (local.tee $2
+                                (i32.shr_u
+                                 (local.get $0)
+                                 (local.get $5)
+                                )
+                               )
+                               (i32.const 5)
+                              )
+                              (i32.const 8)
+                             )
+                            )
+                            (local.get $5)
+                           )
+                           (local.tee $0
+                            (i32.and
+                             (i32.shr_u
+                              (local.tee $2
+                               (i32.shr_u
+                                (local.get $2)
+                                (local.get $0)
+                               )
+                              )
+                              (i32.const 2)
+                             )
+                             (i32.const 4)
+                            )
+                           )
+                          )
+                          (local.tee $0
+                           (i32.and
+                            (i32.shr_u
+                             (local.tee $2
+                              (i32.shr_u
+                               (local.get $2)
+                               (local.get $0)
+                              )
+                             )
+                             (i32.const 1)
+                            )
+                            (i32.const 2)
+                           )
+                          )
+                         )
+                         (local.tee $0
+                          (i32.and
+                           (i32.shr_u
+                            (local.tee $2
+                             (i32.shr_u
+                              (local.get $2)
+                              (local.get $0)
+                             )
+                            )
+                            (i32.const 1)
+                           )
+                           (i32.const 1)
+                          )
+                         )
+                        )
+                        (i32.shr_u
+                         (local.get $2)
+                         (local.get $0)
+                        )
+                       )
+                      )
+                      (i32.const 1)
+                     )
+                     (i32.const 2)
+                    )
+                    (i32.const 3676)
+                   )
+                  )
+                  (i32.const 8)
+                 )
+                )
+               )
+              )
+              (i32.const 8)
+             )
+            )
+           )
+          )
+          (if
+           (i32.eq
+            (local.get $4)
+            (local.get $12)
+           )
+           (i32.store
+            (i32.const 3636)
+            (local.tee $7
+             (i32.and
+              (local.get $8)
+              (i32.xor
+               (i32.shl
+                (i32.const 1)
+                (local.get $11)
+               )
+               (i32.const -1)
+              )
+             )
+            )
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.get $12)
+              (i32.load
+               (i32.const 3652)
+              )
+             )
+             (call $fimport$10)
+            )
+            (if
+             (i32.eq
+              (i32.load
+               (local.tee $0
+                (i32.add
+                 (local.get $12)
+                 (i32.const 12)
+                )
+               )
+              )
+              (local.get $9)
+             )
+             (block
+              (i32.store
+               (local.get $0)
+               (local.get $4)
+              )
+              (i32.store
+               (local.get $2)
+               (local.get $12)
+              )
+              (local.set $7
+               (local.get $8)
+              )
+             )
+             (call $fimport$10)
+            )
+           )
+          )
+          (i32.store offset=4
+           (local.get $9)
+           (i32.or
+            (local.get $3)
+            (i32.const 3)
+           )
+          )
+          (i32.store offset=4
+           (local.tee $4
+            (i32.add
+             (local.get $9)
+             (local.get $3)
+            )
+           )
+           (i32.or
+            (local.tee $11
+             (i32.sub
+              (i32.shl
+               (local.get $11)
+               (i32.const 3)
+              )
+              (local.get $3)
+             )
+            )
+            (i32.const 1)
+           )
+          )
+          (i32.store
+           (i32.add
+            (local.get $4)
+            (local.get $11)
+           )
+           (local.get $11)
+          )
+          (if
+           (local.get $16)
+           (block
+            (local.set $9
+             (i32.load
+              (i32.const 3656)
+             )
+            )
+            (local.set $2
+             (i32.add
+              (i32.shl
+               (i32.shl
+                (local.tee $0
+                 (i32.shr_u
+                  (local.get $16)
+                  (i32.const 3)
+                 )
+                )
+                (i32.const 1)
+               )
+               (i32.const 2)
+              )
+              (i32.const 3676)
+             )
+            )
+            (if
+             (i32.and
+              (local.get $7)
+              (local.tee $0
+               (i32.shl
+                (i32.const 1)
+                (local.get $0)
+               )
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.tee $0
+                (i32.load
+                 (local.tee $3
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 8)
+                  )
+                 )
+                )
+               )
+               (i32.load
+                (i32.const 3652)
+               )
+              )
+              (call $fimport$10)
+              (block
+               (local.set $6
+                (local.get $3)
+               )
+               (local.set $1
+                (local.get $0)
+               )
+              )
+             )
+             (block
+              (i32.store
+               (i32.const 3636)
+               (i32.or
+                (local.get $7)
+                (local.get $0)
+               )
+              )
+              (local.set $6
+               (i32.add
+                (local.get $2)
+                (i32.const 8)
+               )
+              )
+              (local.set $1
+               (local.get $2)
+              )
+             )
+            )
+            (i32.store
+             (local.get $6)
+             (local.get $9)
+            )
+            (i32.store offset=12
+             (local.get $1)
+             (local.get $9)
+            )
+            (i32.store offset=8
+             (local.get $9)
+             (local.get $1)
+            )
+            (i32.store offset=12
+             (local.get $9)
+             (local.get $2)
+            )
+           )
+          )
+          (i32.store
+           (i32.const 3644)
+           (local.get $11)
+          )
+          (i32.store
+           (i32.const 3656)
+           (local.get $4)
+          )
+          (global.set $global$1
+           (local.get $14)
+          )
+          (return
+           (local.get $5)
+          )
+         )
+        )
+        (if
+         (local.tee $6
+          (i32.load
+           (i32.const 3640)
+          )
+         )
+         (block
+          (local.set $2
+           (i32.and
+            (i32.shr_u
+             (local.tee $0
+              (i32.add
+               (i32.and
+                (local.get $6)
+                (i32.sub
+                 (i32.const 0)
+                 (local.get $6)
+                )
+               )
+               (i32.const -1)
+              )
+             )
+             (i32.const 12)
+            )
+            (i32.const 16)
+           )
+          )
+          (local.set $9
+           (i32.sub
+            (i32.and
+             (i32.load offset=4
+              (local.tee $2
+               (i32.load
+                (i32.add
+                 (i32.shl
+                  (i32.add
+                   (i32.or
+                    (i32.or
+                     (i32.or
+                      (i32.or
+                       (local.tee $0
+                        (i32.and
+                         (i32.shr_u
+                          (local.tee $1
+                           (i32.shr_u
+                            (local.get $0)
+                            (local.get $2)
+                           )
+                          )
+                          (i32.const 5)
+                         )
+                         (i32.const 8)
+                        )
+                       )
+                       (local.get $2)
+                      )
+                      (local.tee $0
+                       (i32.and
+                        (i32.shr_u
+                         (local.tee $1
+                          (i32.shr_u
+                           (local.get $1)
+                           (local.get $0)
+                          )
+                         )
+                         (i32.const 2)
+                        )
+                        (i32.const 4)
+                       )
+                      )
+                     )
+                     (local.tee $0
+                      (i32.and
+                       (i32.shr_u
+                        (local.tee $1
+                         (i32.shr_u
+                          (local.get $1)
+                          (local.get $0)
+                         )
+                        )
+                        (i32.const 1)
+                       )
+                       (i32.const 2)
+                      )
+                     )
+                    )
+                    (local.tee $0
+                     (i32.and
+                      (i32.shr_u
+                       (local.tee $1
+                        (i32.shr_u
+                         (local.get $1)
+                         (local.get $0)
+                        )
+                       )
+                       (i32.const 1)
+                      )
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                   (i32.shr_u
+                    (local.get $1)
+                    (local.get $0)
+                   )
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 3940)
+                )
+               )
+              )
+             )
+             (i32.const -8)
+            )
+            (local.get $3)
+           )
+          )
+          (local.set $1
+           (local.get $2)
+          )
+          (loop $label$25
+           (block $label$26
+            (if
+             (i32.eqz
+              (local.tee $0
+               (i32.load offset=16
+                (local.get $1)
+               )
+              )
+             )
+             (br_if $label$26
+              (i32.eqz
+               (local.tee $0
+                (i32.load offset=20
+                 (local.get $1)
+                )
+               )
+              )
+             )
+            )
+            (if
+             (local.tee $7
+              (i32.lt_u
+               (local.tee $1
+                (i32.sub
+                 (i32.and
+                  (i32.load offset=4
+                   (local.get $0)
+                  )
+                  (i32.const -8)
+                 )
+                 (local.get $3)
+                )
+               )
+               (local.get $9)
+              )
+             )
+             (local.set $9
+              (local.get $1)
+             )
+            )
+            (local.set $1
+             (local.get $0)
+            )
+            (if
+             (local.get $7)
+             (local.set $2
+              (local.get $0)
+             )
+            )
+            (br $label$25)
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.get $2)
+            (local.tee $12
+             (i32.load
+              (i32.const 3652)
+             )
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.ge_u
+            (local.get $2)
+            (local.tee $13
+             (i32.add
+              (local.get $2)
+              (local.get $3)
+             )
+            )
+           )
+           (call $fimport$10)
+          )
+          (local.set $15
+           (i32.load offset=24
+            (local.get $2)
+           )
+          )
+          (block $label$32
+           (if
+            (i32.eq
+             (local.tee $0
+              (i32.load offset=12
+               (local.get $2)
+              )
+             )
+             (local.get $2)
+            )
+            (block
+             (if
+              (i32.eqz
+               (local.tee $0
+                (i32.load
+                 (local.tee $1
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 20)
+                  )
+                 )
+                )
+               )
+              )
+              (if
+               (i32.eqz
+                (local.tee $0
+                 (i32.load
+                  (local.tee $1
+                   (i32.add
+                    (local.get $2)
+                    (i32.const 16)
+                   )
+                  )
+                 )
+                )
+               )
+               (block
+                (local.set $4
+                 (i32.const 0)
+                )
+                (br $label$32)
+               )
+              )
+             )
+             (loop $label$36
+              (if
+               (local.tee $7
+                (i32.load
+                 (local.tee $11
+                  (i32.add
+                   (local.get $0)
+                   (i32.const 20)
+                  )
+                 )
+                )
+               )
+               (block
+                (local.set $0
+                 (local.get $7)
+                )
+                (local.set $1
+                 (local.get $11)
+                )
+                (br $label$36)
+               )
+              )
+              (if
+               (local.tee $7
+                (i32.load
+                 (local.tee $11
+                  (i32.add
+                   (local.get $0)
+                   (i32.const 16)
+                  )
+                 )
+                )
+               )
+               (block
+                (local.set $0
+                 (local.get $7)
+                )
+                (local.set $1
+                 (local.get $11)
+                )
+                (br $label$36)
+               )
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.get $1)
+               (local.get $12)
+              )
+              (call $fimport$10)
+              (block
+               (i32.store
+                (local.get $1)
+                (i32.const 0)
+               )
+               (local.set $4
+                (local.get $0)
+               )
+              )
+             )
+            )
+            (block
+             (if
+              (i32.lt_u
+               (local.tee $11
+                (i32.load offset=8
+                 (local.get $2)
+                )
+               )
+               (local.get $12)
+              )
+              (call $fimport$10)
+             )
+             (if
+              (i32.ne
+               (i32.load
+                (local.tee $7
+                 (i32.add
+                  (local.get $11)
+                  (i32.const 12)
+                 )
+                )
+               )
+               (local.get $2)
+              )
+              (call $fimport$10)
+             )
+             (if
+              (i32.eq
+               (i32.load
+                (local.tee $1
+                 (i32.add
+                  (local.get $0)
+                  (i32.const 8)
+                 )
+                )
+               )
+               (local.get $2)
+              )
+              (block
+               (i32.store
+                (local.get $7)
+                (local.get $0)
+               )
+               (i32.store
+                (local.get $1)
+                (local.get $11)
+               )
+               (local.set $4
+                (local.get $0)
+               )
+              )
+              (call $fimport$10)
+             )
+            )
+           )
+          )
+          (block $label$46
+           (if
+            (local.get $15)
+            (block
+             (if
+              (i32.eq
+               (local.get $2)
+               (i32.load
+                (local.tee $0
+                 (i32.add
+                  (i32.shl
+                   (local.tee $1
+                    (i32.load offset=28
+                     (local.get $2)
+                    )
+                   )
+                   (i32.const 2)
+                  )
+                  (i32.const 3940)
+                 )
+                )
+               )
+              )
+              (block
+               (i32.store
+                (local.get $0)
+                (local.get $4)
+               )
+               (if
+                (i32.eqz
+                 (local.get $4)
+                )
+                (block
+                 (i32.store
+                  (i32.const 3640)
+                  (i32.and
+                   (local.get $6)
+                   (i32.xor
+                    (i32.shl
+                     (i32.const 1)
+                     (local.get $1)
+                    )
+                    (i32.const -1)
+                   )
+                  )
+                 )
+                 (br $label$46)
+                )
+               )
+              )
+              (block
+               (if
+                (i32.lt_u
+                 (local.get $15)
+                 (i32.load
+                  (i32.const 3652)
+                 )
+                )
+                (call $fimport$10)
+               )
+               (if
+                (i32.eq
+                 (i32.load
+                  (local.tee $0
+                   (i32.add
+                    (local.get $15)
+                    (i32.const 16)
+                   )
+                  )
+                 )
+                 (local.get $2)
+                )
+                (i32.store
+                 (local.get $0)
+                 (local.get $4)
+                )
+                (i32.store offset=20
+                 (local.get $15)
+                 (local.get $4)
+                )
+               )
+               (br_if $label$46
+                (i32.eqz
+                 (local.get $4)
+                )
+               )
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.get $4)
+               (local.tee $0
+                (i32.load
+                 (i32.const 3652)
+                )
+               )
+              )
+              (call $fimport$10)
+             )
+             (i32.store offset=24
+              (local.get $4)
+              (local.get $15)
+             )
+             (if
+              (local.tee $1
+               (i32.load offset=16
+                (local.get $2)
+               )
+              )
+              (if
+               (i32.lt_u
+                (local.get $1)
+                (local.get $0)
+               )
+               (call $fimport$10)
+               (block
+                (i32.store offset=16
+                 (local.get $4)
+                 (local.get $1)
+                )
+                (i32.store offset=24
+                 (local.get $1)
+                 (local.get $4)
+                )
+               )
+              )
+             )
+             (if
+              (local.tee $0
+               (i32.load offset=20
+                (local.get $2)
+               )
+              )
+              (if
+               (i32.lt_u
+                (local.get $0)
+                (i32.load
+                 (i32.const 3652)
+                )
+               )
+               (call $fimport$10)
+               (block
+                (i32.store offset=20
+                 (local.get $4)
+                 (local.get $0)
+                )
+                (i32.store offset=24
+                 (local.get $0)
+                 (local.get $4)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.get $9)
+            (i32.const 16)
+           )
+           (block
+            (i32.store offset=4
+             (local.get $2)
+             (i32.or
+              (local.tee $0
+               (i32.add
+                (local.get $9)
+                (local.get $3)
+               )
+              )
+              (i32.const 3)
+             )
+            )
+            (i32.store
+             (local.tee $0
+              (i32.add
+               (i32.add
+                (local.get $2)
+                (local.get $0)
+               )
+               (i32.const 4)
+              )
+             )
+             (i32.or
+              (i32.load
+               (local.get $0)
+              )
+              (i32.const 1)
+             )
+            )
+           )
+           (block
+            (i32.store offset=4
+             (local.get $2)
+             (i32.or
+              (local.get $3)
+              (i32.const 3)
+             )
+            )
+            (i32.store offset=4
+             (local.get $13)
+             (i32.or
+              (local.get $9)
+              (i32.const 1)
+             )
+            )
+            (i32.store
+             (i32.add
+              (local.get $13)
+              (local.get $9)
+             )
+             (local.get $9)
+            )
+            (if
+             (local.get $16)
+             (block
+              (local.set $7
+               (i32.load
+                (i32.const 3656)
+               )
+              )
+              (local.set $3
+               (i32.add
+                (i32.shl
+                 (i32.shl
+                  (local.tee $0
+                   (i32.shr_u
+                    (local.get $16)
+                    (i32.const 3)
+                   )
+                  )
+                  (i32.const 1)
+                 )
+                 (i32.const 2)
+                )
+                (i32.const 3676)
+               )
+              )
+              (if
+               (i32.and
+                (local.get $8)
+                (local.tee $0
+                 (i32.shl
+                  (i32.const 1)
+                  (local.get $0)
+                 )
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.tee $0
+                  (i32.load
+                   (local.tee $1
+                    (i32.add
+                     (local.get $3)
+                     (i32.const 8)
+                    )
+                   )
+                  )
+                 )
+                 (i32.load
+                  (i32.const 3652)
+                 )
+                )
+                (call $fimport$10)
+                (block
+                 (local.set $10
+                  (local.get $1)
+                 )
+                 (local.set $5
+                  (local.get $0)
+                 )
+                )
+               )
+               (block
+                (i32.store
+                 (i32.const 3636)
+                 (i32.or
+                  (local.get $8)
+                  (local.get $0)
+                 )
+                )
+                (local.set $10
+                 (i32.add
+                  (local.get $3)
+                  (i32.const 8)
+                 )
+                )
+                (local.set $5
+                 (local.get $3)
+                )
+               )
+              )
+              (i32.store
+               (local.get $10)
+               (local.get $7)
+              )
+              (i32.store offset=12
+               (local.get $5)
+               (local.get $7)
+              )
+              (i32.store offset=8
+               (local.get $7)
+               (local.get $5)
+              )
+              (i32.store offset=12
+               (local.get $7)
+               (local.get $3)
+              )
+             )
+            )
+            (i32.store
+             (i32.const 3644)
+             (local.get $9)
+            )
+            (i32.store
+             (i32.const 3656)
+             (local.get $13)
+            )
+           )
+          )
+          (global.set $global$1
+           (local.get $14)
+          )
+          (return
+           (i32.add
+            (local.get $2)
+            (i32.const 8)
+           )
+          )
+         )
+         (local.set $0
+          (local.get $3)
+         )
+        )
+       )
+       (local.set $0
+        (local.get $3)
+       )
+      )
+     )
+     (if
+      (i32.gt_u
+       (local.get $0)
+       (i32.const -65)
+      )
+      (local.set $0
+       (i32.const -1)
+      )
+      (block
+       (local.set $7
+        (i32.and
+         (local.tee $0
+          (i32.add
+           (local.get $0)
+           (i32.const 11)
+          )
+         )
+         (i32.const -8)
+        )
+       )
+       (if
+        (local.tee $5
+         (i32.load
+          (i32.const 3640)
+         )
+        )
+        (block
+         (local.set $17
+          (if (result i32)
+           (local.tee $0
+            (i32.shr_u
+             (local.get $0)
+             (i32.const 8)
+            )
+           )
+           (if (result i32)
+            (i32.gt_u
+             (local.get $7)
+             (i32.const 16777215)
+            )
+            (i32.const 31)
+            (i32.or
+             (i32.and
+              (i32.shr_u
+               (local.get $7)
+               (i32.add
+                (local.tee $0
+                 (i32.add
+                  (i32.sub
+                   (i32.const 14)
+                   (i32.or
+                    (i32.or
+                     (local.tee $0
+                      (i32.and
+                       (i32.shr_u
+                        (i32.add
+                         (local.tee $1
+                          (i32.shl
+                           (local.get $0)
+                           (local.tee $3
+                            (i32.and
+                             (i32.shr_u
+                              (i32.add
+                               (local.get $0)
+                               (i32.const 1048320)
+                              )
+                              (i32.const 16)
+                             )
+                             (i32.const 8)
+                            )
+                           )
+                          )
+                         )
+                         (i32.const 520192)
+                        )
+                        (i32.const 16)
+                       )
+                       (i32.const 4)
+                      )
+                     )
+                     (local.get $3)
+                    )
+                    (local.tee $0
+                     (i32.and
+                      (i32.shr_u
+                       (i32.add
+                        (local.tee $1
+                         (i32.shl
+                          (local.get $1)
+                          (local.get $0)
+                         )
+                        )
+                        (i32.const 245760)
+                       )
+                       (i32.const 16)
+                      )
+                      (i32.const 2)
+                     )
+                    )
+                   )
+                  )
+                  (i32.shr_u
+                   (i32.shl
+                    (local.get $1)
+                    (local.get $0)
+                   )
+                   (i32.const 15)
+                  )
+                 )
+                )
+                (i32.const 7)
+               )
+              )
+              (i32.const 1)
+             )
+             (i32.shl
+              (local.get $0)
+              (i32.const 1)
+             )
+            )
+           )
+           (i32.const 0)
+          )
+         )
+         (local.set $3
+          (i32.sub
+           (i32.const 0)
+           (local.get $7)
+          )
+         )
+         (block $label$78
+          (block $label$79
+           (block $label$80
+            (if
+             (local.tee $1
+              (i32.load
+               (i32.add
+                (i32.shl
+                 (local.get $17)
+                 (i32.const 2)
+                )
+                (i32.const 3940)
+               )
+              )
+             )
+             (block
+              (local.set $0
+               (i32.sub
+                (i32.const 25)
+                (i32.shr_u
+                 (local.get $17)
+                 (i32.const 1)
+                )
+               )
+              )
+              (local.set $4
+               (i32.const 0)
+              )
+              (local.set $10
+               (i32.shl
+                (local.get $7)
+                (if (result i32)
+                 (i32.eq
+                  (local.get $17)
+                  (i32.const 31)
+                 )
+                 (i32.const 0)
+                 (local.get $0)
+                )
+               )
+              )
+              (local.set $0
+               (i32.const 0)
+              )
+              (loop $label$84
+               (if
+                (i32.lt_u
+                 (local.tee $6
+                  (i32.sub
+                   (i32.and
+                    (i32.load offset=4
+                     (local.get $1)
+                    )
+                    (i32.const -8)
+                   )
+                   (local.get $7)
+                  )
+                 )
+                 (local.get $3)
+                )
+                (if
+                 (local.get $6)
+                 (block
+                  (local.set $3
+                   (local.get $6)
+                  )
+                  (local.set $0
+                   (local.get $1)
+                  )
+                 )
+                 (block
+                  (local.set $3
+                   (i32.const 0)
+                  )
+                  (local.set $0
+                   (local.get $1)
+                  )
+                  (br $label$79)
+                 )
+                )
+               )
+               (local.set $1
+                (if (result i32)
+                 (i32.or
+                  (i32.eqz
+                   (local.tee $19
+                    (i32.load offset=20
+                     (local.get $1)
+                    )
+                   )
+                  )
+                  (i32.eq
+                   (local.get $19)
+                   (local.tee $6
+                    (i32.load
+                     (i32.add
+                      (i32.add
+                       (local.get $1)
+                       (i32.const 16)
+                      )
+                      (i32.shl
+                       (i32.shr_u
+                        (local.get $10)
+                        (i32.const 31)
+                       )
+                       (i32.const 2)
+                      )
+                     )
+                    )
+                   )
+                  )
+                 )
+                 (local.get $4)
+                 (local.get $19)
+                )
+               )
+               (local.set $10
+                (i32.shl
+                 (local.get $10)
+                 (i32.xor
+                  (i32.and
+                   (local.tee $4
+                    (i32.eqz
+                     (local.get $6)
+                    )
+                   )
+                   (i32.const 1)
+                  )
+                  (i32.const 1)
+                 )
+                )
+               )
+               (if
+                (local.get $4)
+                (block
+                 (local.set $4
+                  (local.get $1)
+                 )
+                 (local.set $1
+                  (local.get $0)
+                 )
+                 (br $label$80)
+                )
+                (block
+                 (local.set $4
+                  (local.get $1)
+                 )
+                 (local.set $1
+                  (local.get $6)
+                 )
+                 (br $label$84)
+                )
+               )
+              )
+             )
+             (block
+              (local.set $4
+               (i32.const 0)
+              )
+              (local.set $1
+               (i32.const 0)
+              )
+             )
+            )
+           )
+           (br_if $label$79
+            (local.tee $0
+             (if (result i32)
+              (i32.and
+               (i32.eqz
+                (local.get $4)
+               )
+               (i32.eqz
+                (local.get $1)
+               )
+              )
+              (block (result i32)
+               (if
+                (i32.eqz
+                 (local.tee $0
+                  (i32.and
+                   (local.get $5)
+                   (i32.or
+                    (local.tee $0
+                     (i32.shl
+                      (i32.const 2)
+                      (local.get $17)
+                     )
+                    )
+                    (i32.sub
+                     (i32.const 0)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                 )
+                )
+                (block
+                 (local.set $0
+                  (local.get $7)
+                 )
+                 (br $label$2)
+                )
+               )
+               (local.set $10
+                (i32.and
+                 (i32.shr_u
+                  (local.tee $0
+                   (i32.add
+                    (i32.and
+                     (local.get $0)
+                     (i32.sub
+                      (i32.const 0)
+                      (local.get $0)
+                     )
+                    )
+                    (i32.const -1)
+                   )
+                  )
+                  (i32.const 12)
+                 )
+                 (i32.const 16)
+                )
+               )
+               (i32.load
+                (i32.add
+                 (i32.shl
+                  (i32.add
+                   (i32.or
+                    (i32.or
+                     (i32.or
+                      (i32.or
+                       (local.tee $0
+                        (i32.and
+                         (i32.shr_u
+                          (local.tee $4
+                           (i32.shr_u
+                            (local.get $0)
+                            (local.get $10)
+                           )
+                          )
+                          (i32.const 5)
+                         )
+                         (i32.const 8)
+                        )
+                       )
+                       (local.get $10)
+                      )
+                      (local.tee $0
+                       (i32.and
+                        (i32.shr_u
+                         (local.tee $4
+                          (i32.shr_u
+                           (local.get $4)
+                           (local.get $0)
+                          )
+                         )
+                         (i32.const 2)
+                        )
+                        (i32.const 4)
+                       )
+                      )
+                     )
+                     (local.tee $0
+                      (i32.and
+                       (i32.shr_u
+                        (local.tee $4
+                         (i32.shr_u
+                          (local.get $4)
+                          (local.get $0)
+                         )
+                        )
+                        (i32.const 1)
+                       )
+                       (i32.const 2)
+                      )
+                     )
+                    )
+                    (local.tee $0
+                     (i32.and
+                      (i32.shr_u
+                       (local.tee $4
+                        (i32.shr_u
+                         (local.get $4)
+                         (local.get $0)
+                        )
+                       )
+                       (i32.const 1)
+                      )
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                   (i32.shr_u
+                    (local.get $4)
+                    (local.get $0)
+                   )
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 3940)
+                )
+               )
+              )
+              (local.get $4)
+             )
+            )
+           )
+           (local.set $4
+            (local.get $1)
+           )
+           (br $label$78)
+          )
+          (loop $label$96
+           (if
+            (local.tee $10
+             (i32.lt_u
+              (local.tee $4
+               (i32.sub
+                (i32.and
+                 (i32.load offset=4
+                  (local.get $0)
+                 )
+                 (i32.const -8)
+                )
+                (local.get $7)
+               )
+              )
+              (local.get $3)
+             )
+            )
+            (local.set $3
+             (local.get $4)
+            )
+           )
+           (if
+            (local.get $10)
+            (local.set $1
+             (local.get $0)
+            )
+           )
+           (if
+            (local.tee $4
+             (i32.load offset=16
+              (local.get $0)
+             )
+            )
+            (block
+             (local.set $0
+              (local.get $4)
+             )
+             (br $label$96)
+            )
+           )
+           (br_if $label$96
+            (local.tee $0
+             (i32.load offset=20
+              (local.get $0)
+             )
+            )
+           )
+           (local.set $4
+            (local.get $1)
+           )
+          )
+         )
+         (if
+          (local.get $4)
+          (if
+           (i32.lt_u
+            (local.get $3)
+            (i32.sub
+             (i32.load
+              (i32.const 3644)
+             )
+             (local.get $7)
+            )
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.get $4)
+              (local.tee $12
+               (i32.load
+                (i32.const 3652)
+               )
+              )
+             )
+             (call $fimport$10)
+            )
+            (if
+             (i32.ge_u
+              (local.get $4)
+              (local.tee $6
+               (i32.add
+                (local.get $4)
+                (local.get $7)
+               )
+              )
+             )
+             (call $fimport$10)
+            )
+            (local.set $10
+             (i32.load offset=24
+              (local.get $4)
+             )
+            )
+            (block $label$104
+             (if
+              (i32.eq
+               (local.tee $0
+                (i32.load offset=12
+                 (local.get $4)
+                )
+               )
+               (local.get $4)
+              )
+              (block
+               (if
+                (i32.eqz
+                 (local.tee $0
+                  (i32.load
+                   (local.tee $1
+                    (i32.add
+                     (local.get $4)
+                     (i32.const 20)
+                    )
+                   )
+                  )
+                 )
+                )
+                (if
+                 (i32.eqz
+                  (local.tee $0
+                   (i32.load
+                    (local.tee $1
+                     (i32.add
+                      (local.get $4)
+                      (i32.const 16)
+                     )
+                    )
+                   )
+                  )
+                 )
+                 (block
+                  (local.set $13
+                   (i32.const 0)
+                  )
+                  (br $label$104)
+                 )
+                )
+               )
+               (loop $label$108
+                (if
+                 (local.tee $11
+                  (i32.load
+                   (local.tee $9
+                    (i32.add
+                     (local.get $0)
+                     (i32.const 20)
+                    )
+                   )
+                  )
+                 )
+                 (block
+                  (local.set $0
+                   (local.get $11)
+                  )
+                  (local.set $1
+                   (local.get $9)
+                  )
+                  (br $label$108)
+                 )
+                )
+                (if
+                 (local.tee $11
+                  (i32.load
+                   (local.tee $9
+                    (i32.add
+                     (local.get $0)
+                     (i32.const 16)
+                    )
+                   )
+                  )
+                 )
+                 (block
+                  (local.set $0
+                   (local.get $11)
+                  )
+                  (local.set $1
+                   (local.get $9)
+                  )
+                  (br $label$108)
+                 )
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $1)
+                 (local.get $12)
+                )
+                (call $fimport$10)
+                (block
+                 (i32.store
+                  (local.get $1)
+                  (i32.const 0)
+                 )
+                 (local.set $13
+                  (local.get $0)
+                 )
+                )
+               )
+              )
+              (block
+               (if
+                (i32.lt_u
+                 (local.tee $9
+                  (i32.load offset=8
+                   (local.get $4)
+                  )
+                 )
+                 (local.get $12)
+                )
+                (call $fimport$10)
+               )
+               (if
+                (i32.ne
+                 (i32.load
+                  (local.tee $11
+                   (i32.add
+                    (local.get $9)
+                    (i32.const 12)
+                   )
+                  )
+                 )
+                 (local.get $4)
+                )
+                (call $fimport$10)
+               )
+               (if
+                (i32.eq
+                 (i32.load
+                  (local.tee $1
+                   (i32.add
+                    (local.get $0)
+                    (i32.const 8)
+                   )
+                  )
+                 )
+                 (local.get $4)
+                )
+                (block
+                 (i32.store
+                  (local.get $11)
+                  (local.get $0)
+                 )
+                 (i32.store
+                  (local.get $1)
+                  (local.get $9)
+                 )
+                 (local.set $13
+                  (local.get $0)
+                 )
+                )
+                (call $fimport$10)
+               )
+              )
+             )
+            )
+            (block $label$118
+             (if
+              (local.get $10)
+              (block
+               (if
+                (i32.eq
+                 (local.get $4)
+                 (i32.load
+                  (local.tee $0
+                   (i32.add
+                    (i32.shl
+                     (local.tee $1
+                      (i32.load offset=28
+                       (local.get $4)
+                      )
+                     )
+                     (i32.const 2)
+                    )
+                    (i32.const 3940)
+                   )
+                  )
+                 )
+                )
+                (block
+                 (i32.store
+                  (local.get $0)
+                  (local.get $13)
+                 )
+                 (if
+                  (i32.eqz
+                   (local.get $13)
+                  )
+                  (block
+                   (i32.store
+                    (i32.const 3640)
+                    (local.tee $2
+                     (i32.and
+                      (local.get $5)
+                      (i32.xor
+                       (i32.shl
+                        (i32.const 1)
+                        (local.get $1)
+                       )
+                       (i32.const -1)
+                      )
+                     )
+                    )
+                   )
+                   (br $label$118)
+                  )
+                 )
+                )
+                (block
+                 (if
+                  (i32.lt_u
+                   (local.get $10)
+                   (i32.load
+                    (i32.const 3652)
+                   )
+                  )
+                  (call $fimport$10)
+                 )
+                 (if
+                  (i32.eq
+                   (i32.load
+                    (local.tee $0
+                     (i32.add
+                      (local.get $10)
+                      (i32.const 16)
+                     )
+                    )
+                   )
+                   (local.get $4)
+                  )
+                  (i32.store
+                   (local.get $0)
+                   (local.get $13)
+                  )
+                  (i32.store offset=20
+                   (local.get $10)
+                   (local.get $13)
+                  )
+                 )
+                 (if
+                  (i32.eqz
+                   (local.get $13)
+                  )
+                  (block
+                   (local.set $2
+                    (local.get $5)
+                   )
+                   (br $label$118)
+                  )
+                 )
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $13)
+                 (local.tee $0
+                  (i32.load
+                   (i32.const 3652)
+                  )
+                 )
+                )
+                (call $fimport$10)
+               )
+               (i32.store offset=24
+                (local.get $13)
+                (local.get $10)
+               )
+               (if
+                (local.tee $1
+                 (i32.load offset=16
+                  (local.get $4)
+                 )
+                )
+                (if
+                 (i32.lt_u
+                  (local.get $1)
+                  (local.get $0)
+                 )
+                 (call $fimport$10)
+                 (block
+                  (i32.store offset=16
+                   (local.get $13)
+                   (local.get $1)
+                  )
+                  (i32.store offset=24
+                   (local.get $1)
+                   (local.get $13)
+                  )
+                 )
+                )
+               )
+               (if
+                (local.tee $0
+                 (i32.load offset=20
+                  (local.get $4)
+                 )
+                )
+                (if
+                 (i32.lt_u
+                  (local.get $0)
+                  (i32.load
+                   (i32.const 3652)
+                  )
+                 )
+                 (call $fimport$10)
+                 (block
+                  (i32.store offset=20
+                   (local.get $13)
+                   (local.get $0)
+                  )
+                  (i32.store offset=24
+                   (local.get $0)
+                   (local.get $13)
+                  )
+                  (local.set $2
+                   (local.get $5)
+                  )
+                 )
+                )
+                (local.set $2
+                 (local.get $5)
+                )
+               )
+              )
+              (local.set $2
+               (local.get $5)
+              )
+             )
+            )
+            (block $label$136
+             (if
+              (i32.lt_u
+               (local.get $3)
+               (i32.const 16)
+              )
+              (block
+               (i32.store offset=4
+                (local.get $4)
+                (i32.or
+                 (local.tee $0
+                  (i32.add
+                   (local.get $3)
+                   (local.get $7)
+                  )
+                 )
+                 (i32.const 3)
+                )
+               )
+               (i32.store
+                (local.tee $0
+                 (i32.add
+                  (i32.add
+                   (local.get $4)
+                   (local.get $0)
+                  )
+                  (i32.const 4)
+                 )
+                )
+                (i32.or
+                 (i32.load
+                  (local.get $0)
+                 )
+                 (i32.const 1)
+                )
+               )
+              )
+              (block
+               (i32.store offset=4
+                (local.get $4)
+                (i32.or
+                 (local.get $7)
+                 (i32.const 3)
+                )
+               )
+               (i32.store offset=4
+                (local.get $6)
+                (i32.or
+                 (local.get $3)
+                 (i32.const 1)
+                )
+               )
+               (i32.store
+                (i32.add
+                 (local.get $6)
+                 (local.get $3)
+                )
+                (local.get $3)
+               )
+               (local.set $0
+                (i32.shr_u
+                 (local.get $3)
+                 (i32.const 3)
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $3)
+                 (i32.const 256)
+                )
+                (block
+                 (local.set $3
+                  (i32.add
+                   (i32.shl
+                    (i32.shl
+                     (local.get $0)
+                     (i32.const 1)
+                    )
+                    (i32.const 2)
+                   )
+                   (i32.const 3676)
+                  )
+                 )
+                 (if
+                  (i32.and
+                   (local.tee $1
+                    (i32.load
+                     (i32.const 3636)
+                    )
+                   )
+                   (local.tee $0
+                    (i32.shl
+                     (i32.const 1)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (if
+                   (i32.lt_u
+                    (local.tee $0
+                     (i32.load
+                      (local.tee $1
+                       (i32.add
+                        (local.get $3)
+                        (i32.const 8)
+                       )
+                      )
+                     )
+                    )
+                    (i32.load
+                     (i32.const 3652)
+                    )
+                   )
+                   (call $fimport$10)
+                   (block
+                    (local.set $16
+                     (local.get $1)
+                    )
+                    (local.set $8
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (block
+                   (i32.store
+                    (i32.const 3636)
+                    (i32.or
+                     (local.get $1)
+                     (local.get $0)
+                    )
+                   )
+                   (local.set $16
+                    (i32.add
+                     (local.get $3)
+                     (i32.const 8)
+                    )
+                   )
+                   (local.set $8
+                    (local.get $3)
+                   )
+                  )
+                 )
+                 (i32.store
+                  (local.get $16)
+                  (local.get $6)
+                 )
+                 (i32.store offset=12
+                  (local.get $8)
+                  (local.get $6)
+                 )
+                 (i32.store offset=8
+                  (local.get $6)
+                  (local.get $8)
+                 )
+                 (i32.store offset=12
+                  (local.get $6)
+                  (local.get $3)
+                 )
+                 (br $label$136)
+                )
+               )
+               (local.set $1
+                (i32.add
+                 (i32.shl
+                  (local.tee $5
+                   (if (result i32)
+                    (local.tee $0
+                     (i32.shr_u
+                      (local.get $3)
+                      (i32.const 8)
+                     )
+                    )
+                    (if (result i32)
+                     (i32.gt_u
+                      (local.get $3)
+                      (i32.const 16777215)
+                     )
+                     (i32.const 31)
+                     (i32.or
+                      (i32.and
+                       (i32.shr_u
+                        (local.get $3)
+                        (i32.add
+                         (local.tee $0
+                          (i32.add
+                           (i32.sub
+                            (i32.const 14)
+                            (i32.or
+                             (i32.or
+                              (local.tee $0
+                               (i32.and
+                                (i32.shr_u
+                                 (i32.add
+                                  (local.tee $1
+                                   (i32.shl
+                                    (local.get $0)
+                                    (local.tee $5
+                                     (i32.and
+                                      (i32.shr_u
+                                       (i32.add
+                                        (local.get $0)
+                                        (i32.const 1048320)
+                                       )
+                                       (i32.const 16)
+                                      )
+                                      (i32.const 8)
+                                     )
+                                    )
+                                   )
+                                  )
+                                  (i32.const 520192)
+                                 )
+                                 (i32.const 16)
+                                )
+                                (i32.const 4)
+                               )
+                              )
+                              (local.get $5)
+                             )
+                             (local.tee $0
+                              (i32.and
+                               (i32.shr_u
+                                (i32.add
+                                 (local.tee $1
+                                  (i32.shl
+                                   (local.get $1)
+                                   (local.get $0)
+                                  )
+                                 )
+                                 (i32.const 245760)
+                                )
+                                (i32.const 16)
+                               )
+                               (i32.const 2)
+                              )
+                             )
+                            )
+                           )
+                           (i32.shr_u
+                            (i32.shl
+                             (local.get $1)
+                             (local.get $0)
+                            )
+                            (i32.const 15)
+                           )
+                          )
+                         )
+                         (i32.const 7)
+                        )
+                       )
+                       (i32.const 1)
+                      )
+                      (i32.shl
+                       (local.get $0)
+                       (i32.const 1)
+                      )
+                     )
+                    )
+                    (i32.const 0)
+                   )
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 3940)
+                )
+               )
+               (i32.store offset=28
+                (local.get $6)
+                (local.get $5)
+               )
+               (i32.store offset=4
+                (local.tee $0
+                 (i32.add
+                  (local.get $6)
+                  (i32.const 16)
+                 )
+                )
+                (i32.const 0)
+               )
+               (i32.store
+                (local.get $0)
+                (i32.const 0)
+               )
+               (if
+                (i32.eqz
+                 (i32.and
+                  (local.get $2)
+                  (local.tee $0
+                   (i32.shl
+                    (i32.const 1)
+                    (local.get $5)
+                   )
+                  )
+                 )
+                )
+                (block
+                 (i32.store
+                  (i32.const 3640)
+                  (i32.or
+                   (local.get $2)
+                   (local.get $0)
+                  )
+                 )
+                 (i32.store
+                  (local.get $1)
+                  (local.get $6)
+                 )
+                 (i32.store offset=24
+                  (local.get $6)
+                  (local.get $1)
+                 )
+                 (i32.store offset=12
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (i32.store offset=8
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (br $label$136)
+                )
+               )
+               (local.set $0
+                (i32.load
+                 (local.get $1)
+                )
+               )
+               (local.set $1
+                (i32.sub
+                 (i32.const 25)
+                 (i32.shr_u
+                  (local.get $5)
+                  (i32.const 1)
+                 )
+                )
+               )
+               (local.set $5
+                (i32.shl
+                 (local.get $3)
+                 (if (result i32)
+                  (i32.eq
+                   (local.get $5)
+                   (i32.const 31)
+                  )
+                  (i32.const 0)
+                  (local.get $1)
+                 )
+                )
+               )
+               (block $label$151
+                (block $label$152
+                 (block $label$153
+                  (loop $label$154
+                   (br_if $label$152
+                    (i32.eq
+                     (i32.and
+                      (i32.load offset=4
+                       (local.get $0)
+                      )
+                      (i32.const -8)
+                     )
+                     (local.get $3)
+                    )
+                   )
+                   (local.set $2
+                    (i32.shl
+                     (local.get $5)
+                     (i32.const 1)
+                    )
+                   )
+                   (br_if $label$153
+                    (i32.eqz
+                     (local.tee $1
+                      (i32.load
+                       (local.tee $5
+                        (i32.add
+                         (i32.add
+                          (local.get $0)
+                          (i32.const 16)
+                         )
+                         (i32.shl
+                          (i32.shr_u
+                           (local.get $5)
+                           (i32.const 31)
+                          )
+                          (i32.const 2)
+                         )
+                        )
+                       )
+                      )
+                     )
+                    )
+                   )
+                   (local.set $5
+                    (local.get $2)
+                   )
+                   (local.set $0
+                    (local.get $1)
+                   )
+                   (br $label$154)
+                  )
+                 )
+                 (if
+                  (i32.lt_u
+                   (local.get $5)
+                   (i32.load
+                    (i32.const 3652)
+                   )
+                  )
+                  (call $fimport$10)
+                  (block
+                   (i32.store
+                    (local.get $5)
+                    (local.get $6)
+                   )
+                   (i32.store offset=24
+                    (local.get $6)
+                    (local.get $0)
+                   )
+                   (i32.store offset=12
+                    (local.get $6)
+                    (local.get $6)
+                   )
+                   (i32.store offset=8
+                    (local.get $6)
+                    (local.get $6)
+                   )
+                   (br $label$136)
+                  )
+                 )
+                 (br $label$151)
+                )
+                (if
+                 (i32.and
+                  (i32.ge_u
+                   (local.tee $2
+                    (i32.load
+                     (local.tee $3
+                      (i32.add
+                       (local.get $0)
+                       (i32.const 8)
+                      )
+                     )
+                    )
+                   )
+                   (local.tee $1
+                    (i32.load
+                     (i32.const 3652)
+                    )
+                   )
+                  )
+                  (i32.ge_u
+                   (local.get $0)
+                   (local.get $1)
+                  )
+                 )
+                 (block
+                  (i32.store offset=12
+                   (local.get $2)
+                   (local.get $6)
+                  )
+                  (i32.store
+                   (local.get $3)
+                   (local.get $6)
+                  )
+                  (i32.store offset=8
+                   (local.get $6)
+                   (local.get $2)
+                  )
+                  (i32.store offset=12
+                   (local.get $6)
+                   (local.get $0)
+                  )
+                  (i32.store offset=24
+                   (local.get $6)
+                   (i32.const 0)
+                  )
+                 )
+                 (call $fimport$10)
+                )
+               )
+              )
+             )
+            )
+            (global.set $global$1
+             (local.get $14)
+            )
+            (return
+             (i32.add
+              (local.get $4)
+              (i32.const 8)
+             )
+            )
+           )
+           (local.set $0
+            (local.get $7)
+           )
+          )
+          (local.set $0
+           (local.get $7)
+          )
+         )
+        )
+        (local.set $0
+         (local.get $7)
+        )
+       )
+      )
+     )
+    )
+   )
+   (if
+    (i32.ge_u
+     (local.tee $1
+      (i32.load
+       (i32.const 3644)
+      )
+     )
+     (local.get $0)
+    )
+    (block
+     (local.set $2
+      (i32.load
+       (i32.const 3656)
+      )
+     )
+     (if
+      (i32.gt_u
+       (local.tee $3
+        (i32.sub
+         (local.get $1)
+         (local.get $0)
+        )
+       )
+       (i32.const 15)
+      )
+      (block
+       (i32.store
+        (i32.const 3656)
+        (local.tee $1
+         (i32.add
+          (local.get $2)
+          (local.get $0)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3644)
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $1)
+        (i32.or
+         (local.get $3)
+         (i32.const 1)
+        )
+       )
+       (i32.store
+        (i32.add
+         (local.get $1)
+         (local.get $3)
+        )
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $2)
+        (i32.or
+         (local.get $0)
+         (i32.const 3)
+        )
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3644)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 3656)
+        (i32.const 0)
+       )
+       (i32.store offset=4
+        (local.get $2)
+        (i32.or
+         (local.get $1)
+         (i32.const 3)
+        )
+       )
+       (i32.store
+        (local.tee $0
+         (i32.add
+          (i32.add
+           (local.get $2)
+           (local.get $1)
+          )
+          (i32.const 4)
+         )
+        )
+        (i32.or
+         (i32.load
+          (local.get $0)
+         )
+         (i32.const 1)
+        )
+       )
+      )
+     )
+     (global.set $global$1
+      (local.get $14)
+     )
+     (return
+      (i32.add
+       (local.get $2)
+       (i32.const 8)
+      )
+     )
+    )
+   )
+   (if
+    (i32.gt_u
+     (local.tee $10
+      (i32.load
+       (i32.const 3648)
+      )
+     )
+     (local.get $0)
+    )
+    (block
+     (i32.store
+      (i32.const 3648)
+      (local.tee $3
+       (i32.sub
+        (local.get $10)
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store
+      (i32.const 3660)
+      (local.tee $1
+       (i32.add
+        (local.tee $2
+         (i32.load
+          (i32.const 3660)
+         )
+        )
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store offset=4
+      (local.get $1)
+      (i32.or
+       (local.get $3)
+       (i32.const 1)
+      )
+     )
+     (i32.store offset=4
+      (local.get $2)
+      (i32.or
+       (local.get $0)
+       (i32.const 3)
+      )
+     )
+     (global.set $global$1
+      (local.get $14)
+     )
+     (return
+      (i32.add
+       (local.get $2)
+       (i32.const 8)
+      )
+     )
+    )
+   )
+   (if
+    (i32.le_u
+     (local.tee $6
+      (i32.and
+       (local.tee $8
+        (i32.add
+         (local.tee $1
+          (if (result i32)
+           (i32.load
+            (i32.const 4108)
+           )
+           (i32.load
+            (i32.const 4116)
+           )
+           (block (result i32)
+            (i32.store
+             (i32.const 4116)
+             (i32.const 4096)
+            )
+            (i32.store
+             (i32.const 4112)
+             (i32.const 4096)
+            )
+            (i32.store
+             (i32.const 4120)
+             (i32.const -1)
+            )
+            (i32.store
+             (i32.const 4124)
+             (i32.const -1)
+            )
+            (i32.store
+             (i32.const 4128)
+             (i32.const 0)
+            )
+            (i32.store
+             (i32.const 4080)
+             (i32.const 0)
+            )
+            (i32.store
+             (local.get $18)
+             (local.tee $1
+              (i32.xor
+               (i32.and
+                (local.get $18)
+                (i32.const -16)
+               )
+               (i32.const 1431655768)
+              )
+             )
+            )
+            (i32.store
+             (i32.const 4108)
+             (local.get $1)
+            )
+            (i32.const 4096)
+           )
+          )
+         )
+         (local.tee $13
+          (i32.add
+           (local.get $0)
+           (i32.const 47)
+          )
+         )
+        )
+       )
+       (local.tee $4
+        (i32.sub
+         (i32.const 0)
+         (local.get $1)
+        )
+       )
+      )
+     )
+     (local.get $0)
+    )
+    (block
+     (global.set $global$1
+      (local.get $14)
+     )
+     (return
+      (i32.const 0)
+     )
+    )
+   )
+   (if
+    (local.tee $2
+     (i32.load
+      (i32.const 4076)
+     )
+    )
+    (if
+     (i32.or
+      (i32.le_u
+       (local.tee $1
+        (i32.add
+         (local.tee $3
+          (i32.load
+           (i32.const 4068)
+          )
+         )
+         (local.get $6)
+        )
+       )
+       (local.get $3)
+      )
+      (i32.gt_u
+       (local.get $1)
+       (local.get $2)
+      )
+     )
+     (block
+      (global.set $global$1
+       (local.get $14)
+      )
+      (return
+       (i32.const 0)
+      )
+     )
+    )
+   )
+   (local.set $7
+    (i32.add
+     (local.get $0)
+     (i32.const 48)
+    )
+   )
+   (block $label$171
+    (block $label$172
+     (if
+      (i32.eqz
+       (i32.and
+        (i32.load
+         (i32.const 4080)
+        )
+        (i32.const 4)
+       )
+      )
+      (block
+       (block $label$174
+        (block $label$175
+         (block $label$176
+          (br_if $label$176
+           (i32.eqz
+            (local.tee $3
+             (i32.load
+              (i32.const 3660)
+             )
+            )
+           )
+          )
+          (local.set $2
+           (i32.const 4084)
+          )
+          (loop $label$177
+           (block $label$178
+            (if
+             (i32.le_u
+              (local.tee $1
+               (i32.load
+                (local.get $2)
+               )
+              )
+              (local.get $3)
+             )
+             (br_if $label$178
+              (i32.gt_u
+               (i32.add
+                (local.get $1)
+                (i32.load
+                 (local.tee $5
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 4)
+                  )
+                 )
+                )
+               )
+               (local.get $3)
+              )
+             )
+            )
+            (br_if $label$176
+             (i32.eqz
+              (local.tee $1
+               (i32.load offset=8
+                (local.get $2)
+               )
+              )
+             )
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (br $label$177)
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.tee $3
+             (i32.and
+              (i32.sub
+               (local.get $8)
+               (local.get $10)
+              )
+              (local.get $4)
+             )
+            )
+            (i32.const 2147483647)
+           )
+           (if
+            (i32.eq
+             (local.tee $1
+              (call $40
+               (local.get $3)
+              )
+             )
+             (i32.add
+              (i32.load
+               (local.get $2)
+              )
+              (i32.load
+               (local.get $5)
+              )
+             )
+            )
+            (br_if $label$172
+             (i32.ne
+              (local.get $1)
+              (i32.const -1)
+             )
+            )
+            (block
+             (local.set $2
+              (local.get $1)
+             )
+             (local.set $1
+              (local.get $3)
+             )
+             (br $label$175)
+            )
+           )
+          )
+          (br $label$174)
+         )
+         (if
+          (i32.ne
+           (local.tee $1
+            (call $40
+             (i32.const 0)
+            )
+           )
+           (i32.const -1)
+          )
+          (block
+           (local.set $2
+            (i32.sub
+             (i32.and
+              (i32.add
+               (local.tee $5
+                (i32.add
+                 (local.tee $2
+                  (i32.load
+                   (i32.const 4112)
+                  )
+                 )
+                 (i32.const -1)
+                )
+               )
+               (local.tee $3
+                (local.get $1)
+               )
+              )
+              (i32.sub
+               (i32.const 0)
+               (local.get $2)
+              )
+             )
+             (local.get $3)
+            )
+           )
+           (local.set $4
+            (i32.add
+             (local.tee $3
+              (i32.add
+               (if (result i32)
+                (i32.and
+                 (local.get $5)
+                 (local.get $3)
+                )
+                (local.get $2)
+                (i32.const 0)
+               )
+               (local.get $6)
+              )
+             )
+             (local.tee $5
+              (i32.load
+               (i32.const 4068)
+              )
+             )
+            )
+           )
+           (if
+            (i32.and
+             (i32.gt_u
+              (local.get $3)
+              (local.get $0)
+             )
+             (i32.lt_u
+              (local.get $3)
+              (i32.const 2147483647)
+             )
+            )
+            (block
+             (if
+              (local.tee $2
+               (i32.load
+                (i32.const 4076)
+               )
+              )
+              (br_if $label$174
+               (i32.or
+                (i32.le_u
+                 (local.get $4)
+                 (local.get $5)
+                )
+                (i32.gt_u
+                 (local.get $4)
+                 (local.get $2)
+                )
+               )
+              )
+             )
+             (br_if $label$172
+              (i32.eq
+               (local.tee $2
+                (call $40
+                 (local.get $3)
+                )
+               )
+               (local.get $1)
+              )
+             )
+             (local.set $1
+              (local.get $3)
+             )
+             (br $label$175)
+            )
+           )
+          )
+         )
+         (br $label$174)
+        )
+        (local.set $5
+         (i32.sub
+          (i32.const 0)
+          (local.get $1)
+         )
+        )
+        (if
+         (i32.and
+          (i32.gt_u
+           (local.get $7)
+           (local.get $1)
+          )
+          (i32.and
+           (i32.lt_u
+            (local.get $1)
+            (i32.const 2147483647)
+           )
+           (i32.ne
+            (local.get $2)
+            (i32.const -1)
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.tee $3
+            (i32.and
+             (i32.add
+              (i32.sub
+               (local.get $13)
+               (local.get $1)
+              )
+              (local.tee $3
+               (i32.load
+                (i32.const 4116)
+               )
+              )
+             )
+             (i32.sub
+              (i32.const 0)
+              (local.get $3)
+             )
+            )
+           )
+           (i32.const 2147483647)
+          )
+          (if
+           (i32.eq
+            (call $40
+             (local.get $3)
+            )
+            (i32.const -1)
+           )
+           (block
+            (drop
+             (call $40
+              (local.get $5)
+             )
+            )
+            (br $label$174)
+           )
+           (local.set $3
+            (i32.add
+             (local.get $3)
+             (local.get $1)
+            )
+           )
+          )
+          (local.set $3
+           (local.get $1)
+          )
+         )
+         (local.set $3
+          (local.get $1)
+         )
+        )
+        (if
+         (i32.ne
+          (local.get $2)
+          (i32.const -1)
+         )
+         (block
+          (local.set $1
+           (local.get $2)
+          )
+          (br $label$172)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 4080)
+        (i32.or
+         (i32.load
+          (i32.const 4080)
+         )
+         (i32.const 4)
+        )
+       )
+      )
+     )
+     (if
+      (i32.lt_u
+       (local.get $6)
+       (i32.const 2147483647)
+      )
+      (if
+       (i32.and
+        (i32.lt_u
+         (local.tee $1
+          (call $40
+           (local.get $6)
+          )
+         )
+         (local.tee $3
+          (call $40
+           (i32.const 0)
+          )
+         )
+        )
+        (i32.and
+         (i32.ne
+          (local.get $1)
+          (i32.const -1)
+         )
+         (i32.ne
+          (local.get $3)
+          (i32.const -1)
+         )
+        )
+       )
+       (br_if $label$172
+        (i32.gt_u
+         (local.tee $3
+          (i32.sub
+           (local.get $3)
+           (local.get $1)
+          )
+         )
+         (i32.add
+          (local.get $0)
+          (i32.const 40)
+         )
+        )
+       )
+      )
+     )
+     (br $label$171)
+    )
+    (i32.store
+     (i32.const 4068)
+     (local.tee $2
+      (i32.add
+       (i32.load
+        (i32.const 4068)
+       )
+       (local.get $3)
+      )
+     )
+    )
+    (if
+     (i32.gt_u
+      (local.get $2)
+      (i32.load
+       (i32.const 4072)
+      )
+     )
+     (i32.store
+      (i32.const 4072)
+      (local.get $2)
+     )
+    )
+    (block $label$198
+     (if
+      (local.tee $8
+       (i32.load
+        (i32.const 3660)
+       )
+      )
+      (block
+       (local.set $2
+        (i32.const 4084)
+       )
+       (block $label$200
+        (block $label$201
+         (loop $label$202
+          (br_if $label$201
+           (i32.eq
+            (local.get $1)
+            (i32.add
+             (local.tee $4
+              (i32.load
+               (local.get $2)
+              )
+             )
+             (local.tee $5
+              (i32.load
+               (local.tee $7
+                (i32.add
+                 (local.get $2)
+                 (i32.const 4)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (br_if $label$202
+           (local.tee $2
+            (i32.load offset=8
+             (local.get $2)
+            )
+           )
+          )
+         )
+         (br $label$200)
+        )
+        (if
+         (i32.eqz
+          (i32.and
+           (i32.load offset=12
+            (local.get $2)
+           )
+           (i32.const 8)
+          )
+         )
+         (if
+          (i32.and
+           (i32.lt_u
+            (local.get $8)
+            (local.get $1)
+           )
+           (i32.ge_u
+            (local.get $8)
+            (local.get $4)
+           )
+          )
+          (block
+           (i32.store
+            (local.get $7)
+            (i32.add
+             (local.get $5)
+             (local.get $3)
+            )
+           )
+           (local.set $5
+            (i32.load
+             (i32.const 3648)
+            )
+           )
+           (local.set $1
+            (i32.and
+             (i32.sub
+              (i32.const 0)
+              (local.tee $2
+               (i32.add
+                (local.get $8)
+                (i32.const 8)
+               )
+              )
+             )
+             (i32.const 7)
+            )
+           )
+           (i32.store
+            (i32.const 3660)
+            (local.tee $2
+             (i32.add
+              (local.get $8)
+              (if (result i32)
+               (i32.and
+                (local.get $2)
+                (i32.const 7)
+               )
+               (local.get $1)
+               (local.tee $1
+                (i32.const 0)
+               )
+              )
+             )
+            )
+           )
+           (i32.store
+            (i32.const 3648)
+            (local.tee $1
+             (i32.add
+              (i32.sub
+               (local.get $3)
+               (local.get $1)
+              )
+              (local.get $5)
+             )
+            )
+           )
+           (i32.store offset=4
+            (local.get $2)
+            (i32.or
+             (local.get $1)
+             (i32.const 1)
+            )
+           )
+           (i32.store offset=4
+            (i32.add
+             (local.get $2)
+             (local.get $1)
+            )
+            (i32.const 40)
+           )
+           (i32.store
+            (i32.const 3664)
+            (i32.load
+             (i32.const 4124)
+            )
+           )
+           (br $label$198)
+          )
+         )
+        )
+       )
+       (if
+        (i32.lt_u
+         (local.get $1)
+         (local.tee $2
+          (i32.load
+           (i32.const 3652)
+          )
+         )
+        )
+        (block
+         (i32.store
+          (i32.const 3652)
+          (local.get $1)
+         )
+         (local.set $2
+          (local.get $1)
+         )
+        )
+       )
+       (local.set $10
+        (i32.add
+         (local.get $1)
+         (local.get $3)
+        )
+       )
+       (local.set $5
+        (i32.const 4084)
+       )
+       (block $label$208
+        (block $label$209
+         (loop $label$210
+          (br_if $label$209
+           (i32.eq
+            (i32.load
+             (local.get $5)
+            )
+            (local.get $10)
+           )
+          )
+          (br_if $label$210
+           (local.tee $5
+            (i32.load offset=8
+             (local.get $5)
+            )
+           )
+          )
+          (local.set $5
+           (i32.const 4084)
+          )
+         )
+         (br $label$208)
+        )
+        (if
+         (i32.and
+          (i32.load offset=12
+           (local.get $5)
+          )
+          (i32.const 8)
+         )
+         (local.set $5
+          (i32.const 4084)
+         )
+         (block
+          (i32.store
+           (local.get $5)
+           (local.get $1)
+          )
+          (i32.store
+           (local.tee $5
+            (i32.add
+             (local.get $5)
+             (i32.const 4)
+            )
+           )
+           (i32.add
+            (i32.load
+             (local.get $5)
+            )
+            (local.get $3)
+           )
+          )
+          (local.set $7
+           (i32.and
+            (i32.sub
+             (i32.const 0)
+             (local.tee $4
+              (i32.add
+               (local.get $1)
+               (i32.const 8)
+              )
+             )
+            )
+            (i32.const 7)
+           )
+          )
+          (local.set $3
+           (i32.and
+            (i32.sub
+             (i32.const 0)
+             (local.tee $5
+              (i32.add
+               (local.get $10)
+               (i32.const 8)
+              )
+             )
+            )
+            (i32.const 7)
+           )
+          )
+          (local.set $6
+           (i32.add
+            (local.tee $13
+             (i32.add
+              (local.get $1)
+              (if (result i32)
+               (i32.and
+                (local.get $4)
+                (i32.const 7)
+               )
+               (local.get $7)
+               (i32.const 0)
+              )
+             )
+            )
+            (local.get $0)
+           )
+          )
+          (local.set $7
+           (i32.sub
+            (i32.sub
+             (local.tee $4
+              (i32.add
+               (local.get $10)
+               (if (result i32)
+                (i32.and
+                 (local.get $5)
+                 (i32.const 7)
+                )
+                (local.get $3)
+                (i32.const 0)
+               )
+              )
+             )
+             (local.get $13)
+            )
+            (local.get $0)
+           )
+          )
+          (i32.store offset=4
+           (local.get $13)
+           (i32.or
+            (local.get $0)
+            (i32.const 3)
+           )
+          )
+          (block $label$217
+           (if
+            (i32.eq
+             (local.get $4)
+             (local.get $8)
+            )
+            (block
+             (i32.store
+              (i32.const 3648)
+              (local.tee $0
+               (i32.add
+                (i32.load
+                 (i32.const 3648)
+                )
+                (local.get $7)
+               )
+              )
+             )
+             (i32.store
+              (i32.const 3660)
+              (local.get $6)
+             )
+             (i32.store offset=4
+              (local.get $6)
+              (i32.or
+               (local.get $0)
+               (i32.const 1)
+              )
+             )
+            )
+            (block
+             (if
+              (i32.eq
+               (local.get $4)
+               (i32.load
+                (i32.const 3656)
+               )
+              )
+              (block
+               (i32.store
+                (i32.const 3644)
+                (local.tee $0
+                 (i32.add
+                  (i32.load
+                   (i32.const 3644)
+                  )
+                  (local.get $7)
+                 )
+                )
+               )
+               (i32.store
+                (i32.const 3656)
+                (local.get $6)
+               )
+               (i32.store offset=4
+                (local.get $6)
+                (i32.or
+                 (local.get $0)
+                 (i32.const 1)
+                )
+               )
+               (i32.store
+                (i32.add
+                 (local.get $6)
+                 (local.get $0)
+                )
+                (local.get $0)
+               )
+               (br $label$217)
+              )
+             )
+             (i32.store
+              (local.tee $0
+               (i32.add
+                (local.tee $0
+                 (if (result i32)
+                  (i32.eq
+                   (i32.and
+                    (local.tee $0
+                     (i32.load offset=4
+                      (local.get $4)
+                     )
+                    )
+                    (i32.const 3)
+                   )
+                   (i32.const 1)
+                  )
+                  (block (result i32)
+                   (local.set $11
+                    (i32.and
+                     (local.get $0)
+                     (i32.const -8)
+                    )
+                   )
+                   (local.set $1
+                    (i32.shr_u
+                     (local.get $0)
+                     (i32.const 3)
+                    )
+                   )
+                   (block $label$222
+                    (if
+                     (i32.lt_u
+                      (local.get $0)
+                      (i32.const 256)
+                     )
+                     (block
+                      (local.set $5
+                       (i32.load offset=12
+                        (local.get $4)
+                       )
+                      )
+                      (block $label$224
+                       (if
+                        (i32.ne
+                         (local.tee $3
+                          (i32.load offset=8
+                           (local.get $4)
+                          )
+                         )
+                         (local.tee $0
+                          (i32.add
+                           (i32.shl
+                            (i32.shl
+                             (local.get $1)
+                             (i32.const 1)
+                            )
+                            (i32.const 2)
+                           )
+                           (i32.const 3676)
+                          )
+                         )
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.get $3)
+                           (local.get $2)
+                          )
+                          (call $fimport$10)
+                         )
+                         (br_if $label$224
+                          (i32.eq
+                           (i32.load offset=12
+                            (local.get $3)
+                           )
+                           (local.get $4)
+                          )
+                         )
+                         (call $fimport$10)
+                        )
+                       )
+                      )
+                      (if
+                       (i32.eq
+                        (local.get $5)
+                        (local.get $3)
+                       )
+                       (block
+                        (i32.store
+                         (i32.const 3636)
+                         (i32.and
+                          (i32.load
+                           (i32.const 3636)
+                          )
+                          (i32.xor
+                           (i32.shl
+                            (i32.const 1)
+                            (local.get $1)
+                           )
+                           (i32.const -1)
+                          )
+                         )
+                        )
+                        (br $label$222)
+                       )
+                      )
+                      (block $label$228
+                       (if
+                        (i32.eq
+                         (local.get $5)
+                         (local.get $0)
+                        )
+                        (local.set $20
+                         (i32.add
+                          (local.get $5)
+                          (i32.const 8)
+                         )
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.get $5)
+                           (local.get $2)
+                          )
+                          (call $fimport$10)
+                         )
+                         (if
+                          (i32.eq
+                           (i32.load
+                            (local.tee $0
+                             (i32.add
+                              (local.get $5)
+                              (i32.const 8)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (block
+                           (local.set $20
+                            (local.get $0)
+                           )
+                           (br $label$228)
+                          )
+                         )
+                         (call $fimport$10)
+                        )
+                       )
+                      )
+                      (i32.store offset=12
+                       (local.get $3)
+                       (local.get $5)
+                      )
+                      (i32.store
+                       (local.get $20)
+                       (local.get $3)
+                      )
+                     )
+                     (block
+                      (local.set $8
+                       (i32.load offset=24
+                        (local.get $4)
+                       )
+                      )
+                      (block $label$234
+                       (if
+                        (i32.eq
+                         (local.tee $0
+                          (i32.load offset=12
+                           (local.get $4)
+                          )
+                         )
+                         (local.get $4)
+                        )
+                        (block
+                         (if
+                          (i32.eqz
+                           (local.tee $0
+                            (i32.load
+                             (local.tee $1
+                              (i32.add
+                               (local.tee $3
+                                (i32.add
+                                 (local.get $4)
+                                 (i32.const 16)
+                                )
+                               )
+                               (i32.const 4)
+                              )
+                             )
+                            )
+                           )
+                          )
+                          (if
+                           (local.tee $0
+                            (i32.load
+                             (local.get $3)
+                            )
+                           )
+                           (local.set $1
+                            (local.get $3)
+                           )
+                           (block
+                            (local.set $12
+                             (i32.const 0)
+                            )
+                            (br $label$234)
+                           )
+                          )
+                         )
+                         (loop $label$239
+                          (if
+                           (local.tee $3
+                            (i32.load
+                             (local.tee $5
+                              (i32.add
+                               (local.get $0)
+                               (i32.const 20)
+                              )
+                             )
+                            )
+                           )
+                           (block
+                            (local.set $0
+                             (local.get $3)
+                            )
+                            (local.set $1
+                             (local.get $5)
+                            )
+                            (br $label$239)
+                           )
+                          )
+                          (if
+                           (local.tee $3
+                            (i32.load
+                             (local.tee $5
+                              (i32.add
+                               (local.get $0)
+                               (i32.const 16)
+                              )
+                             )
+                            )
+                           )
+                           (block
+                            (local.set $0
+                             (local.get $3)
+                            )
+                            (local.set $1
+                             (local.get $5)
+                            )
+                            (br $label$239)
+                           )
+                          )
+                         )
+                         (if
+                          (i32.lt_u
+                           (local.get $1)
+                           (local.get $2)
+                          )
+                          (call $fimport$10)
+                          (block
+                           (i32.store
+                            (local.get $1)
+                            (i32.const 0)
+                           )
+                           (local.set $12
+                            (local.get $0)
+                           )
+                          )
+                         )
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.tee $5
+                            (i32.load offset=8
+                             (local.get $4)
+                            )
+                           )
+                           (local.get $2)
+                          )
+                          (call $fimport$10)
+                         )
+                         (if
+                          (i32.ne
+                           (i32.load
+                            (local.tee $3
+                             (i32.add
+                              (local.get $5)
+                              (i32.const 12)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (call $fimport$10)
+                         )
+                         (if
+                          (i32.eq
+                           (i32.load
+                            (local.tee $1
+                             (i32.add
+                              (local.get $0)
+                              (i32.const 8)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (block
+                           (i32.store
+                            (local.get $3)
+                            (local.get $0)
+                           )
+                           (i32.store
+                            (local.get $1)
+                            (local.get $5)
+                           )
+                           (local.set $12
+                            (local.get $0)
+                           )
+                          )
+                          (call $fimport$10)
+                         )
+                        )
+                       )
+                      )
+                      (br_if $label$222
+                       (i32.eqz
+                        (local.get $8)
+                       )
+                      )
+                      (block $label$249
+                       (if
+                        (i32.eq
+                         (local.get $4)
+                         (i32.load
+                          (local.tee $0
+                           (i32.add
+                            (i32.shl
+                             (local.tee $1
+                              (i32.load offset=28
+                               (local.get $4)
+                              )
+                             )
+                             (i32.const 2)
+                            )
+                            (i32.const 3940)
+                           )
+                          )
+                         )
+                        )
+                        (block
+                         (i32.store
+                          (local.get $0)
+                          (local.get $12)
+                         )
+                         (br_if $label$249
+                          (local.get $12)
+                         )
+                         (i32.store
+                          (i32.const 3640)
+                          (i32.and
+                           (i32.load
+                            (i32.const 3640)
+                           )
+                           (i32.xor
+                            (i32.shl
+                             (i32.const 1)
+                             (local.get $1)
+                            )
+                            (i32.const -1)
+                           )
+                          )
+                         )
+                         (br $label$222)
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.get $8)
+                           (i32.load
+                            (i32.const 3652)
+                           )
+                          )
+                          (call $fimport$10)
+                         )
+                         (if
+                          (i32.eq
+                           (i32.load
+                            (local.tee $0
+                             (i32.add
+                              (local.get $8)
+                              (i32.const 16)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (i32.store
+                           (local.get $0)
+                           (local.get $12)
+                          )
+                          (i32.store offset=20
+                           (local.get $8)
+                           (local.get $12)
+                          )
+                         )
+                         (br_if $label$222
+                          (i32.eqz
+                           (local.get $12)
+                          )
+                         )
+                        )
+                       )
+                      )
+                      (if
+                       (i32.lt_u
+                        (local.get $12)
+                        (local.tee $1
+                         (i32.load
+                          (i32.const 3652)
+                         )
+                        )
+                       )
+                       (call $fimport$10)
+                      )
+                      (i32.store offset=24
+                       (local.get $12)
+                       (local.get $8)
+                      )
+                      (if
+                       (local.tee $3
+                        (i32.load
+                         (local.tee $0
+                          (i32.add
+                           (local.get $4)
+                           (i32.const 16)
+                          )
+                         )
+                        )
+                       )
+                       (if
+                        (i32.lt_u
+                         (local.get $3)
+                         (local.get $1)
+                        )
+                        (call $fimport$10)
+                        (block
+                         (i32.store offset=16
+                          (local.get $12)
+                          (local.get $3)
+                         )
+                         (i32.store offset=24
+                          (local.get $3)
+                          (local.get $12)
+                         )
+                        )
+                       )
+                      )
+                      (br_if $label$222
+                       (i32.eqz
+                        (local.tee $0
+                         (i32.load offset=4
+                          (local.get $0)
+                         )
+                        )
+                       )
+                      )
+                      (if
+                       (i32.lt_u
+                        (local.get $0)
+                        (i32.load
+                         (i32.const 3652)
+                        )
+                       )
+                       (call $fimport$10)
+                       (block
+                        (i32.store offset=20
+                         (local.get $12)
+                         (local.get $0)
+                        )
+                        (i32.store offset=24
+                         (local.get $0)
+                         (local.get $12)
+                        )
+                       )
+                      )
+                     )
+                    )
+                   )
+                   (local.set $7
+                    (i32.add
+                     (local.get $11)
+                     (local.get $7)
+                    )
+                   )
+                   (i32.add
+                    (local.get $4)
+                    (local.get $11)
+                   )
+                  )
+                  (local.get $4)
+                 )
+                )
+                (i32.const 4)
+               )
+              )
+              (i32.and
+               (i32.load
+                (local.get $0)
+               )
+               (i32.const -2)
+              )
+             )
+             (i32.store offset=4
+              (local.get $6)
+              (i32.or
+               (local.get $7)
+               (i32.const 1)
+              )
+             )
+             (i32.store
+              (i32.add
+               (local.get $6)
+               (local.get $7)
+              )
+              (local.get $7)
+             )
+             (local.set $0
+              (i32.shr_u
+               (local.get $7)
+               (i32.const 3)
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.get $7)
+               (i32.const 256)
+              )
+              (block
+               (local.set $3
+                (i32.add
+                 (i32.shl
+                  (i32.shl
+                   (local.get $0)
+                   (i32.const 1)
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 3676)
+                )
+               )
+               (block $label$263
+                (if
+                 (i32.and
+                  (local.tee $1
+                   (i32.load
+                    (i32.const 3636)
+                   )
+                  )
+                  (local.tee $0
+                   (i32.shl
+                    (i32.const 1)
+                    (local.get $0)
+                   )
+                  )
+                 )
+                 (block
+                  (if
+                   (i32.ge_u
+                    (local.tee $0
+                     (i32.load
+                      (local.tee $1
+                       (i32.add
+                        (local.get $3)
+                        (i32.const 8)
+                       )
+                      )
+                     )
+                    )
+                    (i32.load
+                     (i32.const 3652)
+                    )
+                   )
+                   (block
+                    (local.set $21
+                     (local.get $1)
+                    )
+                    (local.set $9
+                     (local.get $0)
+                    )
+                    (br $label$263)
+                   )
+                  )
+                  (call $fimport$10)
+                 )
+                 (block
+                  (i32.store
+                   (i32.const 3636)
+                   (i32.or
+                    (local.get $1)
+                    (local.get $0)
+                   )
+                  )
+                  (local.set $21
+                   (i32.add
+                    (local.get $3)
+                    (i32.const 8)
+                   )
+                  )
+                  (local.set $9
+                   (local.get $3)
+                  )
+                 )
+                )
+               )
+               (i32.store
+                (local.get $21)
+                (local.get $6)
+               )
+               (i32.store offset=12
+                (local.get $9)
+                (local.get $6)
+               )
+               (i32.store offset=8
+                (local.get $6)
+                (local.get $9)
+               )
+               (i32.store offset=12
+                (local.get $6)
+                (local.get $3)
+               )
+               (br $label$217)
+              )
+             )
+             (local.set $3
+              (i32.add
+               (i32.shl
+                (local.tee $2
+                 (block $label$267 (result i32)
+                  (if (result i32)
+                   (local.tee $0
+                    (i32.shr_u
+                     (local.get $7)
+                     (i32.const 8)
+                    )
+                   )
+                   (block (result i32)
+                    (drop
+                     (br_if $label$267
+                      (i32.const 31)
+                      (i32.gt_u
+                       (local.get $7)
+                       (i32.const 16777215)
+                      )
+                     )
+                    )
+                    (i32.or
+                     (i32.and
+                      (i32.shr_u
+                       (local.get $7)
+                       (i32.add
+                        (local.tee $0
+                         (i32.add
+                          (i32.sub
+                           (i32.const 14)
+                           (i32.or
+                            (i32.or
+                             (local.tee $0
+                              (i32.and
+                               (i32.shr_u
+                                (i32.add
+                                 (local.tee $1
+                                  (i32.shl
+                                   (local.get $0)
+                                   (local.tee $3
+                                    (i32.and
+                                     (i32.shr_u
+                                      (i32.add
+                                       (local.get $0)
+                                       (i32.const 1048320)
+                                      )
+                                      (i32.const 16)
+                                     )
+                                     (i32.const 8)
+                                    )
+                                   )
+                                  )
+                                 )
+                                 (i32.const 520192)
+                                )
+                                (i32.const 16)
+                               )
+                               (i32.const 4)
+                              )
+                             )
+                             (local.get $3)
+                            )
+                            (local.tee $0
+                             (i32.and
+                              (i32.shr_u
+                               (i32.add
+                                (local.tee $1
+                                 (i32.shl
+                                  (local.get $1)
+                                  (local.get $0)
+                                 )
+                                )
+                                (i32.const 245760)
+                               )
+                               (i32.const 16)
+                              )
+                              (i32.const 2)
+                             )
+                            )
+                           )
+                          )
+                          (i32.shr_u
+                           (i32.shl
+                            (local.get $1)
+                            (local.get $0)
+                           )
+                           (i32.const 15)
+                          )
+                         )
+                        )
+                        (i32.const 7)
+                       )
+                      )
+                      (i32.const 1)
+                     )
+                     (i32.shl
+                      (local.get $0)
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                   (i32.const 0)
+                  )
+                 )
+                )
+                (i32.const 2)
+               )
+               (i32.const 3940)
+              )
+             )
+             (i32.store offset=28
+              (local.get $6)
+              (local.get $2)
+             )
+             (i32.store offset=4
+              (local.tee $0
+               (i32.add
+                (local.get $6)
+                (i32.const 16)
+               )
+              )
+              (i32.const 0)
+             )
+             (i32.store
+              (local.get $0)
+              (i32.const 0)
+             )
+             (if
+              (i32.eqz
+               (i32.and
+                (local.tee $1
+                 (i32.load
+                  (i32.const 3640)
+                 )
+                )
+                (local.tee $0
+                 (i32.shl
+                  (i32.const 1)
+                  (local.get $2)
+                 )
+                )
+               )
+              )
+              (block
+               (i32.store
+                (i32.const 3640)
+                (i32.or
+                 (local.get $1)
+                 (local.get $0)
+                )
+               )
+               (i32.store
+                (local.get $3)
+                (local.get $6)
+               )
+               (i32.store offset=24
+                (local.get $6)
+                (local.get $3)
+               )
+               (i32.store offset=12
+                (local.get $6)
+                (local.get $6)
+               )
+               (i32.store offset=8
+                (local.get $6)
+                (local.get $6)
+               )
+               (br $label$217)
+              )
+             )
+             (local.set $0
+              (i32.load
+               (local.get $3)
+              )
+             )
+             (local.set $1
+              (i32.sub
+               (i32.const 25)
+               (i32.shr_u
+                (local.get $2)
+                (i32.const 1)
+               )
+              )
+             )
+             (local.set $2
+              (i32.shl
+               (local.get $7)
+               (if (result i32)
+                (i32.eq
+                 (local.get $2)
+                 (i32.const 31)
+                )
+                (i32.const 0)
+                (local.get $1)
+               )
+              )
+             )
+             (block $label$273
+              (block $label$274
+               (block $label$275
+                (loop $label$276
+                 (br_if $label$274
+                  (i32.eq
+                   (i32.and
+                    (i32.load offset=4
+                     (local.get $0)
+                    )
+                    (i32.const -8)
+                   )
+                   (local.get $7)
+                  )
+                 )
+                 (local.set $3
+                  (i32.shl
+                   (local.get $2)
+                   (i32.const 1)
+                  )
+                 )
+                 (br_if $label$275
+                  (i32.eqz
+                   (local.tee $1
+                    (i32.load
+                     (local.tee $2
+                      (i32.add
+                       (i32.add
+                        (local.get $0)
+                        (i32.const 16)
+                       )
+                       (i32.shl
+                        (i32.shr_u
+                         (local.get $2)
+                         (i32.const 31)
+                        )
+                        (i32.const 2)
+                       )
+                      )
+                     )
+                    )
+                   )
+                  )
+                 )
+                 (local.set $2
+                  (local.get $3)
+                 )
+                 (local.set $0
+                  (local.get $1)
+                 )
+                 (br $label$276)
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $2)
+                 (i32.load
+                  (i32.const 3652)
+                 )
+                )
+                (call $fimport$10)
+                (block
+                 (i32.store
+                  (local.get $2)
+                  (local.get $6)
+                 )
+                 (i32.store offset=24
+                  (local.get $6)
+                  (local.get $0)
+                 )
+                 (i32.store offset=12
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (i32.store offset=8
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (br $label$217)
+                )
+               )
+               (br $label$273)
+              )
+              (if
+               (i32.and
+                (i32.ge_u
+                 (local.tee $2
+                  (i32.load
+                   (local.tee $3
+                    (i32.add
+                     (local.get $0)
+                     (i32.const 8)
+                    )
+                   )
+                  )
+                 )
+                 (local.tee $1
+                  (i32.load
+                   (i32.const 3652)
+                  )
+                 )
+                )
+                (i32.ge_u
+                 (local.get $0)
+                 (local.get $1)
+                )
+               )
+               (block
+                (i32.store offset=12
+                 (local.get $2)
+                 (local.get $6)
+                )
+                (i32.store
+                 (local.get $3)
+                 (local.get $6)
+                )
+                (i32.store offset=8
+                 (local.get $6)
+                 (local.get $2)
+                )
+                (i32.store offset=12
+                 (local.get $6)
+                 (local.get $0)
+                )
+                (i32.store offset=24
+                 (local.get $6)
+                 (i32.const 0)
+                )
+               )
+               (call $fimport$10)
+              )
+             )
+            )
+           )
+          )
+          (global.set $global$1
+           (local.get $14)
+          )
+          (return
+           (i32.add
+            (local.get $13)
+            (i32.const 8)
+           )
+          )
+         )
+        )
+       )
+       (loop $label$281
+        (block $label$282
+         (if
+          (i32.le_u
+           (local.tee $2
+            (i32.load
+             (local.get $5)
+            )
+           )
+           (local.get $8)
+          )
+          (br_if $label$282
+           (i32.gt_u
+            (local.tee $13
+             (i32.add
+              (local.get $2)
+              (i32.load offset=4
+               (local.get $5)
+              )
+             )
+            )
+            (local.get $8)
+           )
+          )
+         )
+         (local.set $5
+          (i32.load offset=8
+           (local.get $5)
+          )
+         )
+         (br $label$281)
+        )
+       )
+       (local.set $2
+        (i32.and
+         (i32.sub
+          (i32.const 0)
+          (local.tee $5
+           (i32.add
+            (local.tee $7
+             (i32.add
+              (local.get $13)
+              (i32.const -47)
+             )
+            )
+            (i32.const 8)
+           )
+          )
+         )
+         (i32.const 7)
+        )
+       )
+       (local.set $10
+        (i32.add
+         (local.tee $7
+          (if (result i32)
+           (i32.lt_u
+            (local.tee $2
+             (i32.add
+              (local.get $7)
+              (if (result i32)
+               (i32.and
+                (local.get $5)
+                (i32.const 7)
+               )
+               (local.get $2)
+               (i32.const 0)
+              )
+             )
+            )
+            (local.tee $12
+             (i32.add
+              (local.get $8)
+              (i32.const 16)
+             )
+            )
+           )
+           (local.get $8)
+           (local.get $2)
+          )
+         )
+         (i32.const 8)
+        )
+       )
+       (local.set $5
+        (i32.add
+         (local.get $7)
+         (i32.const 24)
+        )
+       )
+       (local.set $9
+        (i32.add
+         (local.get $3)
+         (i32.const -40)
+        )
+       )
+       (local.set $2
+        (i32.and
+         (i32.sub
+          (i32.const 0)
+          (local.tee $4
+           (i32.add
+            (local.get $1)
+            (i32.const 8)
+           )
+          )
+         )
+         (i32.const 7)
+        )
+       )
+       (i32.store
+        (i32.const 3660)
+        (local.tee $4
+         (i32.add
+          (local.get $1)
+          (if (result i32)
+           (i32.and
+            (local.get $4)
+            (i32.const 7)
+           )
+           (local.get $2)
+           (local.tee $2
+            (i32.const 0)
+           )
+          )
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3648)
+        (local.tee $2
+         (i32.sub
+          (local.get $9)
+          (local.get $2)
+         )
+        )
+       )
+       (i32.store offset=4
+        (local.get $4)
+        (i32.or
+         (local.get $2)
+         (i32.const 1)
+        )
+       )
+       (i32.store offset=4
+        (i32.add
+         (local.get $4)
+         (local.get $2)
+        )
+        (i32.const 40)
+       )
+       (i32.store
+        (i32.const 3664)
+        (i32.load
+         (i32.const 4124)
+        )
+       )
+       (i32.store
+        (local.tee $2
+         (i32.add
+          (local.get $7)
+          (i32.const 4)
+         )
+        )
+        (i32.const 27)
+       )
+       (i64.store align=4
+        (local.get $10)
+        (i64.load align=4
+         (i32.const 4084)
+        )
+       )
+       (i64.store offset=8 align=4
+        (local.get $10)
+        (i64.load align=4
+         (i32.const 4092)
+        )
+       )
+       (i32.store
+        (i32.const 4084)
+        (local.get $1)
+       )
+       (i32.store
+        (i32.const 4088)
+        (local.get $3)
+       )
+       (i32.store
+        (i32.const 4096)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 4092)
+        (local.get $10)
+       )
+       (local.set $1
+        (local.get $5)
+       )
+       (loop $label$290
+        (i32.store
+         (local.tee $1
+          (i32.add
+           (local.get $1)
+           (i32.const 4)
+          )
+         )
+         (i32.const 7)
+        )
+        (br_if $label$290
+         (i32.lt_u
+          (i32.add
+           (local.get $1)
+           (i32.const 4)
+          )
+          (local.get $13)
+         )
+        )
+       )
+       (if
+        (i32.ne
+         (local.get $7)
+         (local.get $8)
+        )
+        (block
+         (i32.store
+          (local.get $2)
+          (i32.and
+           (i32.load
+            (local.get $2)
+           )
+           (i32.const -2)
+          )
+         )
+         (i32.store offset=4
+          (local.get $8)
+          (i32.or
+           (local.tee $4
+            (i32.sub
+             (local.get $7)
+             (local.get $8)
+            )
+           )
+           (i32.const 1)
+          )
+         )
+         (i32.store
+          (local.get $7)
+          (local.get $4)
+         )
+         (local.set $1
+          (i32.shr_u
+           (local.get $4)
+           (i32.const 3)
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $4)
+           (i32.const 256)
+          )
+          (block
+           (local.set $2
+            (i32.add
+             (i32.shl
+              (i32.shl
+               (local.get $1)
+               (i32.const 1)
+              )
+              (i32.const 2)
+             )
+             (i32.const 3676)
+            )
+           )
+           (if
+            (i32.and
+             (local.tee $3
+              (i32.load
+               (i32.const 3636)
+              )
+             )
+             (local.tee $1
+              (i32.shl
+               (i32.const 1)
+               (local.get $1)
+              )
+             )
+            )
+            (if
+             (i32.lt_u
+              (local.tee $1
+               (i32.load
+                (local.tee $3
+                 (i32.add
+                  (local.get $2)
+                  (i32.const 8)
+                 )
+                )
+               )
+              )
+              (i32.load
+               (i32.const 3652)
+              )
+             )
+             (call $fimport$10)
+             (block
+              (local.set $15
+               (local.get $3)
+              )
+              (local.set $11
+               (local.get $1)
+              )
+             )
+            )
+            (block
+             (i32.store
+              (i32.const 3636)
+              (i32.or
+               (local.get $3)
+               (local.get $1)
+              )
+             )
+             (local.set $15
+              (i32.add
+               (local.get $2)
+               (i32.const 8)
+              )
+             )
+             (local.set $11
+              (local.get $2)
+             )
+            )
+           )
+           (i32.store
+            (local.get $15)
+            (local.get $8)
+           )
+           (i32.store offset=12
+            (local.get $11)
+            (local.get $8)
+           )
+           (i32.store offset=8
+            (local.get $8)
+            (local.get $11)
+           )
+           (i32.store offset=12
+            (local.get $8)
+            (local.get $2)
+           )
+           (br $label$198)
+          )
+         )
+         (local.set $2
+          (i32.add
+           (i32.shl
+            (local.tee $5
+             (if (result i32)
+              (local.tee $1
+               (i32.shr_u
+                (local.get $4)
+                (i32.const 8)
+               )
+              )
+              (if (result i32)
+               (i32.gt_u
+                (local.get $4)
+                (i32.const 16777215)
+               )
+               (i32.const 31)
+               (i32.or
+                (i32.and
+                 (i32.shr_u
+                  (local.get $4)
+                  (i32.add
+                   (local.tee $1
+                    (i32.add
+                     (i32.sub
+                      (i32.const 14)
+                      (i32.or
+                       (i32.or
+                        (local.tee $1
+                         (i32.and
+                          (i32.shr_u
+                           (i32.add
+                            (local.tee $3
+                             (i32.shl
+                              (local.get $1)
+                              (local.tee $2
+                               (i32.and
+                                (i32.shr_u
+                                 (i32.add
+                                  (local.get $1)
+                                  (i32.const 1048320)
+                                 )
+                                 (i32.const 16)
+                                )
+                                (i32.const 8)
+                               )
+                              )
+                             )
+                            )
+                            (i32.const 520192)
+                           )
+                           (i32.const 16)
+                          )
+                          (i32.const 4)
+                         )
+                        )
+                        (local.get $2)
+                       )
+                       (local.tee $1
+                        (i32.and
+                         (i32.shr_u
+                          (i32.add
+                           (local.tee $3
+                            (i32.shl
+                             (local.get $3)
+                             (local.get $1)
+                            )
+                           )
+                           (i32.const 245760)
+                          )
+                          (i32.const 16)
+                         )
+                         (i32.const 2)
+                        )
+                       )
+                      )
+                     )
+                     (i32.shr_u
+                      (i32.shl
+                       (local.get $3)
+                       (local.get $1)
+                      )
+                      (i32.const 15)
+                     )
+                    )
+                   )
+                   (i32.const 7)
+                  )
+                 )
+                 (i32.const 1)
+                )
+                (i32.shl
+                 (local.get $1)
+                 (i32.const 1)
+                )
+               )
+              )
+              (i32.const 0)
+             )
+            )
+            (i32.const 2)
+           )
+           (i32.const 3940)
+          )
+         )
+         (i32.store offset=28
+          (local.get $8)
+          (local.get $5)
+         )
+         (i32.store offset=20
+          (local.get $8)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $12)
+          (i32.const 0)
+         )
+         (if
+          (i32.eqz
+           (i32.and
+            (local.tee $3
+             (i32.load
+              (i32.const 3640)
+             )
+            )
+            (local.tee $1
+             (i32.shl
+              (i32.const 1)
+              (local.get $5)
+             )
+            )
+           )
+          )
+          (block
+           (i32.store
+            (i32.const 3640)
+            (i32.or
+             (local.get $3)
+             (local.get $1)
+            )
+           )
+           (i32.store
+            (local.get $2)
+            (local.get $8)
+           )
+           (i32.store offset=24
+            (local.get $8)
+            (local.get $2)
+           )
+           (i32.store offset=12
+            (local.get $8)
+            (local.get $8)
+           )
+           (i32.store offset=8
+            (local.get $8)
+            (local.get $8)
+           )
+           (br $label$198)
+          )
+         )
+         (local.set $1
+          (i32.load
+           (local.get $2)
+          )
+         )
+         (local.set $3
+          (i32.sub
+           (i32.const 25)
+           (i32.shr_u
+            (local.get $5)
+            (i32.const 1)
+           )
+          )
+         )
+         (local.set $5
+          (i32.shl
+           (local.get $4)
+           (if (result i32)
+            (i32.eq
+             (local.get $5)
+             (i32.const 31)
+            )
+            (i32.const 0)
+            (local.get $3)
+           )
+          )
+         )
+         (block $label$304
+          (block $label$305
+           (block $label$306
+            (loop $label$307
+             (br_if $label$305
+              (i32.eq
+               (i32.and
+                (i32.load offset=4
+                 (local.get $1)
+                )
+                (i32.const -8)
+               )
+               (local.get $4)
+              )
+             )
+             (local.set $2
+              (i32.shl
+               (local.get $5)
+               (i32.const 1)
+              )
+             )
+             (br_if $label$306
+              (i32.eqz
+               (local.tee $3
+                (i32.load
+                 (local.tee $5
+                  (i32.add
+                   (i32.add
+                    (local.get $1)
+                    (i32.const 16)
+                   )
+                   (i32.shl
+                    (i32.shr_u
+                     (local.get $5)
+                     (i32.const 31)
+                    )
+                    (i32.const 2)
+                   )
+                  )
+                 )
+                )
+               )
+              )
+             )
+             (local.set $5
+              (local.get $2)
+             )
+             (local.set $1
+              (local.get $3)
+             )
+             (br $label$307)
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $5)
+             (i32.load
+              (i32.const 3652)
+             )
+            )
+            (call $fimport$10)
+            (block
+             (i32.store
+              (local.get $5)
+              (local.get $8)
+             )
+             (i32.store offset=24
+              (local.get $8)
+              (local.get $1)
+             )
+             (i32.store offset=12
+              (local.get $8)
+              (local.get $8)
+             )
+             (i32.store offset=8
+              (local.get $8)
+              (local.get $8)
+             )
+             (br $label$198)
+            )
+           )
+           (br $label$304)
+          )
+          (if
+           (i32.and
+            (i32.ge_u
+             (local.tee $5
+              (i32.load
+               (local.tee $2
+                (i32.add
+                 (local.get $1)
+                 (i32.const 8)
+                )
+               )
+              )
+             )
+             (local.tee $3
+              (i32.load
+               (i32.const 3652)
+              )
+             )
+            )
+            (i32.ge_u
+             (local.get $1)
+             (local.get $3)
+            )
+           )
+           (block
+            (i32.store offset=12
+             (local.get $5)
+             (local.get $8)
+            )
+            (i32.store
+             (local.get $2)
+             (local.get $8)
+            )
+            (i32.store offset=8
+             (local.get $8)
+             (local.get $5)
+            )
+            (i32.store offset=12
+             (local.get $8)
+             (local.get $1)
+            )
+            (i32.store offset=24
+             (local.get $8)
+             (i32.const 0)
+            )
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+       )
+      )
+      (block
+       (if
+        (i32.or
+         (i32.eqz
+          (local.tee $2
+           (i32.load
+            (i32.const 3652)
+           )
+          )
+         )
+         (i32.lt_u
+          (local.get $1)
+          (local.get $2)
+         )
+        )
+        (i32.store
+         (i32.const 3652)
+         (local.get $1)
+        )
+       )
+       (i32.store
+        (i32.const 4084)
+        (local.get $1)
+       )
+       (i32.store
+        (i32.const 4088)
+        (local.get $3)
+       )
+       (i32.store
+        (i32.const 4096)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 3672)
+        (i32.load
+         (i32.const 4108)
+        )
+       )
+       (i32.store
+        (i32.const 3668)
+        (i32.const -1)
+       )
+       (local.set $2
+        (i32.const 0)
+       )
+       (loop $label$314
+        (i32.store offset=12
+         (local.tee $5
+          (i32.add
+           (i32.shl
+            (i32.shl
+             (local.get $2)
+             (i32.const 1)
+            )
+            (i32.const 2)
+           )
+           (i32.const 3676)
+          )
+         )
+         (local.get $5)
+        )
+        (i32.store offset=8
+         (local.get $5)
+         (local.get $5)
+        )
+        (br_if $label$314
+         (i32.ne
+          (local.tee $2
+           (i32.add
+            (local.get $2)
+            (i32.const 1)
+           )
+          )
+          (i32.const 32)
+         )
+        )
+       )
+       (local.set $5
+        (i32.add
+         (local.get $3)
+         (i32.const -40)
+        )
+       )
+       (local.set $3
+        (i32.and
+         (i32.sub
+          (i32.const 0)
+          (local.tee $2
+           (i32.add
+            (local.get $1)
+            (i32.const 8)
+           )
+          )
+         )
+         (i32.const 7)
+        )
+       )
+       (i32.store
+        (i32.const 3660)
+        (local.tee $3
+         (i32.add
+          (local.get $1)
+          (local.tee $1
+           (if (result i32)
+            (i32.and
+             (local.get $2)
+             (i32.const 7)
+            )
+            (local.get $3)
+            (i32.const 0)
+           )
+          )
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3648)
+        (local.tee $1
+         (i32.sub
+          (local.get $5)
+          (local.get $1)
+         )
+        )
+       )
+       (i32.store offset=4
+        (local.get $3)
+        (i32.or
+         (local.get $1)
+         (i32.const 1)
+        )
+       )
+       (i32.store offset=4
+        (i32.add
+         (local.get $3)
+         (local.get $1)
+        )
+        (i32.const 40)
+       )
+       (i32.store
+        (i32.const 3664)
+        (i32.load
+         (i32.const 4124)
+        )
+       )
+      )
+     )
+    )
+    (if
+     (i32.gt_u
+      (local.tee $1
+       (i32.load
+        (i32.const 3648)
+       )
+      )
+      (local.get $0)
+     )
+     (block
+      (i32.store
+       (i32.const 3648)
+       (local.tee $3
+        (i32.sub
+         (local.get $1)
+         (local.get $0)
+        )
+       )
+      )
+      (i32.store
+       (i32.const 3660)
+       (local.tee $1
+        (i32.add
+         (local.tee $2
+          (i32.load
+           (i32.const 3660)
+          )
+         )
+         (local.get $0)
+        )
+       )
+      )
+      (i32.store offset=4
+       (local.get $1)
+       (i32.or
+        (local.get $3)
+        (i32.const 1)
+       )
+      )
+      (i32.store offset=4
+       (local.get $2)
+       (i32.or
+        (local.get $0)
+        (i32.const 3)
+       )
+      )
+      (global.set $global$1
+       (local.get $14)
+      )
+      (return
+       (i32.add
+        (local.get $2)
+        (i32.const 8)
+       )
+      )
+     )
+    )
+   )
+   (i32.store
+    (call $12)
+    (i32.const 12)
+   )
+   (global.set $global$1
+    (local.get $14)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $38 (; 51 ;) (type $2) (param $0 i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (block $label$1
+   (if
+    (i32.eqz
+     (local.get $0)
+    )
+    (return)
+   )
+   (if
+    (i32.lt_u
+     (local.tee $1
+      (i32.add
+       (local.get $0)
+       (i32.const -8)
+      )
+     )
+     (local.tee $11
+      (i32.load
+       (i32.const 3652)
+      )
+     )
+    )
+    (call $fimport$10)
+   )
+   (if
+    (i32.eq
+     (local.tee $8
+      (i32.and
+       (local.tee $0
+        (i32.load
+         (i32.add
+          (local.get $0)
+          (i32.const -4)
+         )
+        )
+       )
+       (i32.const 3)
+      )
+     )
+     (i32.const 1)
+    )
+    (call $fimport$10)
+   )
+   (local.set $6
+    (i32.add
+     (local.get $1)
+     (local.tee $4
+      (i32.and
+       (local.get $0)
+       (i32.const -8)
+      )
+     )
+    )
+   )
+   (block $label$5
+    (if
+     (i32.and
+      (local.get $0)
+      (i32.const 1)
+     )
+     (block
+      (local.set $3
+       (local.get $1)
+      )
+      (local.set $2
+       (local.get $4)
+      )
+     )
+     (block
+      (if
+       (i32.eqz
+        (local.get $8)
+       )
+       (return)
+      )
+      (if
+       (i32.lt_u
+        (local.tee $0
+         (i32.add
+          (local.get $1)
+          (i32.sub
+           (i32.const 0)
+           (local.tee $8
+            (i32.load
+             (local.get $1)
+            )
+           )
+          )
+         )
+        )
+        (local.get $11)
+       )
+       (call $fimport$10)
+      )
+      (local.set $1
+       (i32.add
+        (local.get $8)
+        (local.get $4)
+       )
+      )
+      (if
+       (i32.eq
+        (local.get $0)
+        (i32.load
+         (i32.const 3656)
+        )
+       )
+       (block
+        (if
+         (i32.ne
+          (i32.and
+           (local.tee $3
+            (i32.load
+             (local.tee $2
+              (i32.add
+               (local.get $6)
+               (i32.const 4)
+              )
+             )
+            )
+           )
+           (i32.const 3)
+          )
+          (i32.const 3)
+         )
+         (block
+          (local.set $3
+           (local.get $0)
+          )
+          (local.set $2
+           (local.get $1)
+          )
+          (br $label$5)
+         )
+        )
+        (i32.store
+         (i32.const 3644)
+         (local.get $1)
+        )
+        (i32.store
+         (local.get $2)
+         (i32.and
+          (local.get $3)
+          (i32.const -2)
+         )
+        )
+        (i32.store offset=4
+         (local.get $0)
+         (i32.or
+          (local.get $1)
+          (i32.const 1)
+         )
+        )
+        (i32.store
+         (i32.add
+          (local.get $0)
+          (local.get $1)
+         )
+         (local.get $1)
+        )
+        (return)
+       )
+      )
+      (local.set $10
+       (i32.shr_u
+        (local.get $8)
+        (i32.const 3)
+       )
+      )
+      (if
+       (i32.lt_u
+        (local.get $8)
+        (i32.const 256)
+       )
+       (block
+        (local.set $3
+         (i32.load offset=12
+          (local.get $0)
+         )
+        )
+        (if
+         (i32.ne
+          (local.tee $4
+           (i32.load offset=8
+            (local.get $0)
+           )
+          )
+          (local.tee $2
+           (i32.add
+            (i32.shl
+             (i32.shl
+              (local.get $10)
+              (i32.const 1)
+             )
+             (i32.const 2)
+            )
+            (i32.const 3676)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $4)
+            (local.get $11)
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.ne
+            (i32.load offset=12
+             (local.get $4)
+            )
+            (local.get $0)
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $3)
+          (local.get $4)
+         )
+         (block
+          (i32.store
+           (i32.const 3636)
+           (i32.and
+            (i32.load
+             (i32.const 3636)
+            )
+            (i32.xor
+             (i32.shl
+              (i32.const 1)
+              (local.get $10)
+             )
+             (i32.const -1)
+            )
+           )
+          )
+          (local.set $3
+           (local.get $0)
+          )
+          (local.set $2
+           (local.get $1)
+          )
+          (br $label$5)
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $3)
+          (local.get $2)
+         )
+         (local.set $5
+          (i32.add
+           (local.get $3)
+           (i32.const 8)
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $3)
+            (local.get $11)
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $2
+              (i32.add
+               (local.get $3)
+               (i32.const 8)
+              )
+             )
+            )
+            (local.get $0)
+           )
+           (local.set $5
+            (local.get $2)
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (i32.store offset=12
+         (local.get $4)
+         (local.get $3)
+        )
+        (i32.store
+         (local.get $5)
+         (local.get $4)
+        )
+        (local.set $3
+         (local.get $0)
+        )
+        (local.set $2
+         (local.get $1)
+        )
+        (br $label$5)
+       )
+      )
+      (local.set $12
+       (i32.load offset=24
+        (local.get $0)
+       )
+      )
+      (block $label$22
+       (if
+        (i32.eq
+         (local.tee $4
+          (i32.load offset=12
+           (local.get $0)
+          )
+         )
+         (local.get $0)
+        )
+        (block
+         (if
+          (local.tee $4
+           (i32.load
+            (local.tee $8
+             (i32.add
+              (local.tee $5
+               (i32.add
+                (local.get $0)
+                (i32.const 16)
+               )
+              )
+              (i32.const 4)
+             )
+            )
+           )
+          )
+          (local.set $5
+           (local.get $8)
+          )
+          (if
+           (i32.eqz
+            (local.tee $4
+             (i32.load
+              (local.get $5)
+             )
+            )
+           )
+           (block
+            (local.set $7
+             (i32.const 0)
+            )
+            (br $label$22)
+           )
+          )
+         )
+         (loop $label$27
+          (if
+           (local.tee $10
+            (i32.load
+             (local.tee $8
+              (i32.add
+               (local.get $4)
+               (i32.const 20)
+              )
+             )
+            )
+           )
+           (block
+            (local.set $4
+             (local.get $10)
+            )
+            (local.set $5
+             (local.get $8)
+            )
+            (br $label$27)
+           )
+          )
+          (if
+           (local.tee $10
+            (i32.load
+             (local.tee $8
+              (i32.add
+               (local.get $4)
+               (i32.const 16)
+              )
+             )
+            )
+           )
+           (block
+            (local.set $4
+             (local.get $10)
+            )
+            (local.set $5
+             (local.get $8)
+            )
+            (br $label$27)
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $5)
+           (local.get $11)
+          )
+          (call $fimport$10)
+          (block
+           (i32.store
+            (local.get $5)
+            (i32.const 0)
+           )
+           (local.set $7
+            (local.get $4)
+           )
+          )
+         )
+        )
+        (block
+         (if
+          (i32.lt_u
+           (local.tee $5
+            (i32.load offset=8
+             (local.get $0)
+            )
+           )
+           (local.get $11)
+          )
+          (call $fimport$10)
+         )
+         (if
+          (i32.ne
+           (i32.load
+            (local.tee $8
+             (i32.add
+              (local.get $5)
+              (i32.const 12)
+             )
+            )
+           )
+           (local.get $0)
+          )
+          (call $fimport$10)
+         )
+         (if
+          (i32.eq
+           (i32.load
+            (local.tee $10
+             (i32.add
+              (local.get $4)
+              (i32.const 8)
+             )
+            )
+           )
+           (local.get $0)
+          )
+          (block
+           (i32.store
+            (local.get $8)
+            (local.get $4)
+           )
+           (i32.store
+            (local.get $10)
+            (local.get $5)
+           )
+           (local.set $7
+            (local.get $4)
+           )
+          )
+          (call $fimport$10)
+         )
+        )
+       )
+      )
+      (if
+       (local.get $12)
+       (block
+        (if
+         (i32.eq
+          (local.get $0)
+          (i32.load
+           (local.tee $5
+            (i32.add
+             (i32.shl
+              (local.tee $4
+               (i32.load offset=28
+                (local.get $0)
+               )
+              )
+              (i32.const 2)
+             )
+             (i32.const 3940)
+            )
+           )
+          )
+         )
+         (block
+          (i32.store
+           (local.get $5)
+           (local.get $7)
+          )
+          (if
+           (i32.eqz
+            (local.get $7)
+           )
+           (block
+            (i32.store
+             (i32.const 3640)
+             (i32.and
+              (i32.load
+               (i32.const 3640)
+              )
+              (i32.xor
+               (i32.shl
+                (i32.const 1)
+                (local.get $4)
+               )
+               (i32.const -1)
+              )
+             )
+            )
+            (local.set $3
+             (local.get $0)
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (br $label$5)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $12)
+            (i32.load
+             (i32.const 3652)
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $4
+              (i32.add
+               (local.get $12)
+               (i32.const 16)
+              )
+             )
+            )
+            (local.get $0)
+           )
+           (i32.store
+            (local.get $4)
+            (local.get $7)
+           )
+           (i32.store offset=20
+            (local.get $12)
+            (local.get $7)
+           )
+          )
+          (if
+           (i32.eqz
+            (local.get $7)
+           )
+           (block
+            (local.set $3
+             (local.get $0)
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (br $label$5)
+           )
+          )
+         )
+        )
+        (if
+         (i32.lt_u
+          (local.get $7)
+          (local.tee $5
+           (i32.load
+            (i32.const 3652)
+           )
+          )
+         )
+         (call $fimport$10)
+        )
+        (i32.store offset=24
+         (local.get $7)
+         (local.get $12)
+        )
+        (if
+         (local.tee $4
+          (i32.load
+           (local.tee $8
+            (i32.add
+             (local.get $0)
+             (i32.const 16)
+            )
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $4)
+           (local.get $5)
+          )
+          (call $fimport$10)
+          (block
+           (i32.store offset=16
+            (local.get $7)
+            (local.get $4)
+           )
+           (i32.store offset=24
+            (local.get $4)
+            (local.get $7)
+           )
+          )
+         )
+        )
+        (if
+         (local.tee $4
+          (i32.load offset=4
+           (local.get $8)
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $4)
+           (i32.load
+            (i32.const 3652)
+           )
+          )
+          (call $fimport$10)
+          (block
+           (i32.store offset=20
+            (local.get $7)
+            (local.get $4)
+           )
+           (i32.store offset=24
+            (local.get $4)
+            (local.get $7)
+           )
+           (local.set $3
+            (local.get $0)
+           )
+           (local.set $2
+            (local.get $1)
+           )
+          )
+         )
+         (block
+          (local.set $3
+           (local.get $0)
+          )
+          (local.set $2
+           (local.get $1)
+          )
+         )
+        )
+       )
+       (block
+        (local.set $3
+         (local.get $0)
+        )
+        (local.set $2
+         (local.get $1)
+        )
+       )
+      )
+     )
+    )
+   )
+   (if
+    (i32.ge_u
+     (local.get $3)
+     (local.get $6)
+    )
+    (call $fimport$10)
+   )
+   (if
+    (i32.eqz
+     (i32.and
+      (local.tee $0
+       (i32.load
+        (local.tee $1
+         (i32.add
+          (local.get $6)
+          (i32.const 4)
+         )
+        )
+       )
+      )
+      (i32.const 1)
+     )
+    )
+    (call $fimport$10)
+   )
+   (if
+    (i32.and
+     (local.get $0)
+     (i32.const 2)
+    )
+    (block
+     (i32.store
+      (local.get $1)
+      (i32.and
+       (local.get $0)
+       (i32.const -2)
+      )
+     )
+     (i32.store offset=4
+      (local.get $3)
+      (i32.or
+       (local.get $2)
+       (i32.const 1)
+      )
+     )
+     (i32.store
+      (i32.add
+       (local.get $3)
+       (local.get $2)
+      )
+      (local.get $2)
+     )
+    )
+    (block
+     (if
+      (i32.eq
+       (local.get $6)
+       (i32.load
+        (i32.const 3660)
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3648)
+        (local.tee $0
+         (i32.add
+          (i32.load
+           (i32.const 3648)
+          )
+          (local.get $2)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3660)
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $3)
+        (i32.or
+         (local.get $0)
+         (i32.const 1)
+        )
+       )
+       (if
+        (i32.ne
+         (local.get $3)
+         (i32.load
+          (i32.const 3656)
+         )
+        )
+        (return)
+       )
+       (i32.store
+        (i32.const 3656)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 3644)
+        (i32.const 0)
+       )
+       (return)
+      )
+     )
+     (if
+      (i32.eq
+       (local.get $6)
+       (i32.load
+        (i32.const 3656)
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3644)
+        (local.tee $0
+         (i32.add
+          (i32.load
+           (i32.const 3644)
+          )
+          (local.get $2)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3656)
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $3)
+        (i32.or
+         (local.get $0)
+         (i32.const 1)
+        )
+       )
+       (i32.store
+        (i32.add
+         (local.get $3)
+         (local.get $0)
+        )
+        (local.get $0)
+       )
+       (return)
+      )
+     )
+     (local.set $5
+      (i32.add
+       (i32.and
+        (local.get $0)
+        (i32.const -8)
+       )
+       (local.get $2)
+      )
+     )
+     (local.set $4
+      (i32.shr_u
+       (local.get $0)
+       (i32.const 3)
+      )
+     )
+     (block $label$61
+      (if
+       (i32.lt_u
+        (local.get $0)
+        (i32.const 256)
+       )
+       (block
+        (local.set $2
+         (i32.load offset=12
+          (local.get $6)
+         )
+        )
+        (if
+         (i32.ne
+          (local.tee $1
+           (i32.load offset=8
+            (local.get $6)
+           )
+          )
+          (local.tee $0
+           (i32.add
+            (i32.shl
+             (i32.shl
+              (local.get $4)
+              (i32.const 1)
+             )
+             (i32.const 2)
+            )
+            (i32.const 3676)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $1)
+            (i32.load
+             (i32.const 3652)
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.ne
+            (i32.load offset=12
+             (local.get $1)
+            )
+            (local.get $6)
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $2)
+          (local.get $1)
+         )
+         (block
+          (i32.store
+           (i32.const 3636)
+           (i32.and
+            (i32.load
+             (i32.const 3636)
+            )
+            (i32.xor
+             (i32.shl
+              (i32.const 1)
+              (local.get $4)
+             )
+             (i32.const -1)
+            )
+           )
+          )
+          (br $label$61)
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $2)
+          (local.get $0)
+         )
+         (local.set $14
+          (i32.add
+           (local.get $2)
+           (i32.const 8)
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $2)
+            (i32.load
+             (i32.const 3652)
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $0
+              (i32.add
+               (local.get $2)
+               (i32.const 8)
+              )
+             )
+            )
+            (local.get $6)
+           )
+           (local.set $14
+            (local.get $0)
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (i32.store offset=12
+         (local.get $1)
+         (local.get $2)
+        )
+        (i32.store
+         (local.get $14)
+         (local.get $1)
+        )
+       )
+       (block
+        (local.set $7
+         (i32.load offset=24
+          (local.get $6)
+         )
+        )
+        (block $label$73
+         (if
+          (i32.eq
+           (local.tee $0
+            (i32.load offset=12
+             (local.get $6)
+            )
+           )
+           (local.get $6)
+          )
+          (block
+           (if
+            (local.tee $0
+             (i32.load
+              (local.tee $1
+               (i32.add
+                (local.tee $2
+                 (i32.add
+                  (local.get $6)
+                  (i32.const 16)
+                 )
+                )
+                (i32.const 4)
+               )
+              )
+             )
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (if
+             (i32.eqz
+              (local.tee $0
+               (i32.load
+                (local.get $2)
+               )
+              )
+             )
+             (block
+              (local.set $9
+               (i32.const 0)
+              )
+              (br $label$73)
+             )
+            )
+           )
+           (loop $label$78
+            (if
+             (local.tee $4
+              (i32.load
+               (local.tee $1
+                (i32.add
+                 (local.get $0)
+                 (i32.const 20)
+                )
+               )
+              )
+             )
+             (block
+              (local.set $0
+               (local.get $4)
+              )
+              (local.set $2
+               (local.get $1)
+              )
+              (br $label$78)
+             )
+            )
+            (if
+             (local.tee $4
+              (i32.load
+               (local.tee $1
+                (i32.add
+                 (local.get $0)
+                 (i32.const 16)
+                )
+               )
+              )
+             )
+             (block
+              (local.set $0
+               (local.get $4)
+              )
+              (local.set $2
+               (local.get $1)
+              )
+              (br $label$78)
+             )
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $2)
+             (i32.load
+              (i32.const 3652)
+             )
+            )
+            (call $fimport$10)
+            (block
+             (i32.store
+              (local.get $2)
+              (i32.const 0)
+             )
+             (local.set $9
+              (local.get $0)
+             )
+            )
+           )
+          )
+          (block
+           (if
+            (i32.lt_u
+             (local.tee $2
+              (i32.load offset=8
+               (local.get $6)
+              )
+             )
+             (i32.load
+              (i32.const 3652)
+             )
+            )
+            (call $fimport$10)
+           )
+           (if
+            (i32.ne
+             (i32.load
+              (local.tee $1
+               (i32.add
+                (local.get $2)
+                (i32.const 12)
+               )
+              )
+             )
+             (local.get $6)
+            )
+            (call $fimport$10)
+           )
+           (if
+            (i32.eq
+             (i32.load
+              (local.tee $4
+               (i32.add
+                (local.get $0)
+                (i32.const 8)
+               )
+              )
+             )
+             (local.get $6)
+            )
+            (block
+             (i32.store
+              (local.get $1)
+              (local.get $0)
+             )
+             (i32.store
+              (local.get $4)
+              (local.get $2)
+             )
+             (local.set $9
+              (local.get $0)
+             )
+            )
+            (call $fimport$10)
+           )
+          )
+         )
+        )
+        (if
+         (local.get $7)
+         (block
+          (if
+           (i32.eq
+            (local.get $6)
+            (i32.load
+             (local.tee $2
+              (i32.add
+               (i32.shl
+                (local.tee $0
+                 (i32.load offset=28
+                  (local.get $6)
+                 )
+                )
+                (i32.const 2)
+               )
+               (i32.const 3940)
+              )
+             )
+            )
+           )
+           (block
+            (i32.store
+             (local.get $2)
+             (local.get $9)
+            )
+            (if
+             (i32.eqz
+              (local.get $9)
+             )
+             (block
+              (i32.store
+               (i32.const 3640)
+               (i32.and
+                (i32.load
+                 (i32.const 3640)
+                )
+                (i32.xor
+                 (i32.shl
+                  (i32.const 1)
+                  (local.get $0)
+                 )
+                 (i32.const -1)
+                )
+               )
+              )
+              (br $label$61)
+             )
+            )
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.get $7)
+              (i32.load
+               (i32.const 3652)
+              )
+             )
+             (call $fimport$10)
+            )
+            (if
+             (i32.eq
+              (i32.load
+               (local.tee $0
+                (i32.add
+                 (local.get $7)
+                 (i32.const 16)
+                )
+               )
+              )
+              (local.get $6)
+             )
+             (i32.store
+              (local.get $0)
+              (local.get $9)
+             )
+             (i32.store offset=20
+              (local.get $7)
+              (local.get $9)
+             )
+            )
+            (br_if $label$61
+             (i32.eqz
+              (local.get $9)
+             )
+            )
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.get $9)
+            (local.tee $2
+             (i32.load
+              (i32.const 3652)
+             )
+            )
+           )
+           (call $fimport$10)
+          )
+          (i32.store offset=24
+           (local.get $9)
+           (local.get $7)
+          )
+          (if
+           (local.tee $0
+            (i32.load
+             (local.tee $1
+              (i32.add
+               (local.get $6)
+               (i32.const 16)
+              )
+             )
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $0)
+             (local.get $2)
+            )
+            (call $fimport$10)
+            (block
+             (i32.store offset=16
+              (local.get $9)
+              (local.get $0)
+             )
+             (i32.store offset=24
+              (local.get $0)
+              (local.get $9)
+             )
+            )
+           )
+          )
+          (if
+           (local.tee $0
+            (i32.load offset=4
+             (local.get $1)
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $0)
+             (i32.load
+              (i32.const 3652)
+             )
+            )
+            (call $fimport$10)
+            (block
+             (i32.store offset=20
+              (local.get $9)
+              (local.get $0)
+             )
+             (i32.store offset=24
+              (local.get $0)
+              (local.get $9)
+             )
+            )
+           )
+          )
+         )
+        )
+       )
+      )
+     )
+     (i32.store offset=4
+      (local.get $3)
+      (i32.or
+       (local.get $5)
+       (i32.const 1)
+      )
+     )
+     (i32.store
+      (i32.add
+       (local.get $3)
+       (local.get $5)
+      )
+      (local.get $5)
+     )
+     (if
+      (i32.eq
+       (local.get $3)
+       (i32.load
+        (i32.const 3656)
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3644)
+        (local.get $5)
+       )
+       (return)
+      )
+      (local.set $2
+       (local.get $5)
+      )
+     )
+    )
+   )
+   (local.set $1
+    (i32.shr_u
+     (local.get $2)
+     (i32.const 3)
+    )
+   )
+   (if
+    (i32.lt_u
+     (local.get $2)
+     (i32.const 256)
+    )
+    (block
+     (local.set $0
+      (i32.add
+       (i32.shl
+        (i32.shl
+         (local.get $1)
+         (i32.const 1)
+        )
+        (i32.const 2)
+       )
+       (i32.const 3676)
+      )
+     )
+     (if
+      (i32.and
+       (local.tee $2
+        (i32.load
+         (i32.const 3636)
+        )
+       )
+       (local.tee $1
+        (i32.shl
+         (i32.const 1)
+         (local.get $1)
+        )
+       )
+      )
+      (if
+       (i32.lt_u
+        (local.tee $1
+         (i32.load
+          (local.tee $2
+           (i32.add
+            (local.get $0)
+            (i32.const 8)
+           )
+          )
+         )
+        )
+        (i32.load
+         (i32.const 3652)
+        )
+       )
+       (call $fimport$10)
+       (block
+        (local.set $15
+         (local.get $2)
+        )
+        (local.set $13
+         (local.get $1)
+        )
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3636)
+        (i32.or
+         (local.get $2)
+         (local.get $1)
+        )
+       )
+       (local.set $15
+        (i32.add
+         (local.get $0)
+         (i32.const 8)
+        )
+       )
+       (local.set $13
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store
+      (local.get $15)
+      (local.get $3)
+     )
+     (i32.store offset=12
+      (local.get $13)
+      (local.get $3)
+     )
+     (i32.store offset=8
+      (local.get $3)
+      (local.get $13)
+     )
+     (i32.store offset=12
+      (local.get $3)
+      (local.get $0)
+     )
+     (return)
+    )
+   )
+   (local.set $0
+    (i32.add
+     (i32.shl
+      (local.tee $1
+       (if (result i32)
+        (local.tee $0
+         (i32.shr_u
+          (local.get $2)
+          (i32.const 8)
+         )
+        )
+        (if (result i32)
+         (i32.gt_u
+          (local.get $2)
+          (i32.const 16777215)
+         )
+         (i32.const 31)
+         (i32.or
+          (i32.and
+           (i32.shr_u
+            (local.get $2)
+            (i32.add
+             (local.tee $0
+              (i32.add
+               (i32.sub
+                (i32.const 14)
+                (i32.or
+                 (i32.or
+                  (local.tee $4
+                   (i32.and
+                    (i32.shr_u
+                     (i32.add
+                      (local.tee $1
+                       (i32.shl
+                        (local.get $0)
+                        (local.tee $0
+                         (i32.and
+                          (i32.shr_u
+                           (i32.add
+                            (local.get $0)
+                            (i32.const 1048320)
+                           )
+                           (i32.const 16)
+                          )
+                          (i32.const 8)
+                         )
+                        )
+                       )
+                      )
+                      (i32.const 520192)
+                     )
+                     (i32.const 16)
+                    )
+                    (i32.const 4)
+                   )
+                  )
+                  (local.get $0)
+                 )
+                 (local.tee $1
+                  (i32.and
+                   (i32.shr_u
+                    (i32.add
+                     (local.tee $0
+                      (i32.shl
+                       (local.get $1)
+                       (local.get $4)
+                      )
+                     )
+                     (i32.const 245760)
+                    )
+                    (i32.const 16)
+                   )
+                   (i32.const 2)
+                  )
+                 )
+                )
+               )
+               (i32.shr_u
+                (i32.shl
+                 (local.get $0)
+                 (local.get $1)
+                )
+                (i32.const 15)
+               )
+              )
+             )
+             (i32.const 7)
+            )
+           )
+           (i32.const 1)
+          )
+          (i32.shl
+           (local.get $0)
+           (i32.const 1)
+          )
+         )
+        )
+        (i32.const 0)
+       )
+      )
+      (i32.const 2)
+     )
+     (i32.const 3940)
+    )
+   )
+   (i32.store offset=28
+    (local.get $3)
+    (local.get $1)
+   )
+   (i32.store offset=20
+    (local.get $3)
+    (i32.const 0)
+   )
+   (i32.store offset=16
+    (local.get $3)
+    (i32.const 0)
+   )
+   (block $label$113
+    (if
+     (i32.and
+      (local.tee $4
+       (i32.load
+        (i32.const 3640)
+       )
+      )
+      (local.tee $5
+       (i32.shl
+        (i32.const 1)
+        (local.get $1)
+       )
+      )
+     )
+     (block
+      (local.set $0
+       (i32.load
+        (local.get $0)
+       )
+      )
+      (local.set $4
+       (i32.sub
+        (i32.const 25)
+        (i32.shr_u
+         (local.get $1)
+         (i32.const 1)
+        )
+       )
+      )
+      (local.set $1
+       (i32.shl
+        (local.get $2)
+        (if (result i32)
+         (i32.eq
+          (local.get $1)
+          (i32.const 31)
+         )
+         (i32.const 0)
+         (local.get $4)
+        )
+       )
+      )
+      (block $label$117
+       (block $label$118
+        (block $label$119
+         (loop $label$120
+          (br_if $label$118
+           (i32.eq
+            (i32.and
+             (i32.load offset=4
+              (local.get $0)
+             )
+             (i32.const -8)
+            )
+            (local.get $2)
+           )
+          )
+          (local.set $4
+           (i32.shl
+            (local.get $1)
+            (i32.const 1)
+           )
+          )
+          (br_if $label$119
+           (i32.eqz
+            (local.tee $5
+             (i32.load
+              (local.tee $1
+               (i32.add
+                (i32.add
+                 (local.get $0)
+                 (i32.const 16)
+                )
+                (i32.shl
+                 (i32.shr_u
+                  (local.get $1)
+                  (i32.const 31)
+                 )
+                 (i32.const 2)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (local.set $1
+           (local.get $4)
+          )
+          (local.set $0
+           (local.get $5)
+          )
+          (br $label$120)
+         )
+        )
+        (if
+         (i32.lt_u
+          (local.get $1)
+          (i32.load
+           (i32.const 3652)
+          )
+         )
+         (call $fimport$10)
+         (block
+          (i32.store
+           (local.get $1)
+           (local.get $3)
+          )
+          (i32.store offset=24
+           (local.get $3)
+           (local.get $0)
+          )
+          (i32.store offset=12
+           (local.get $3)
+           (local.get $3)
+          )
+          (i32.store offset=8
+           (local.get $3)
+           (local.get $3)
+          )
+          (br $label$113)
+         )
+        )
+        (br $label$117)
+       )
+       (if
+        (i32.and
+         (i32.ge_u
+          (local.tee $2
+           (i32.load
+            (local.tee $1
+             (i32.add
+              (local.get $0)
+              (i32.const 8)
+             )
+            )
+           )
+          )
+          (local.tee $4
+           (i32.load
+            (i32.const 3652)
+           )
+          )
+         )
+         (i32.ge_u
+          (local.get $0)
+          (local.get $4)
+         )
+        )
+        (block
+         (i32.store offset=12
+          (local.get $2)
+          (local.get $3)
+         )
+         (i32.store
+          (local.get $1)
+          (local.get $3)
+         )
+         (i32.store offset=8
+          (local.get $3)
+          (local.get $2)
+         )
+         (i32.store offset=12
+          (local.get $3)
+          (local.get $0)
+         )
+         (i32.store offset=24
+          (local.get $3)
+          (i32.const 0)
+         )
+        )
+        (call $fimport$10)
+       )
+      )
+     )
+     (block
+      (i32.store
+       (i32.const 3640)
+       (i32.or
+        (local.get $4)
+        (local.get $5)
+       )
+      )
+      (i32.store
+       (local.get $0)
+       (local.get $3)
+      )
+      (i32.store offset=24
+       (local.get $3)
+       (local.get $0)
+      )
+      (i32.store offset=12
+       (local.get $3)
+       (local.get $3)
+      )
+      (i32.store offset=8
+       (local.get $3)
+       (local.get $3)
+      )
+     )
+    )
+   )
+   (i32.store
+    (i32.const 3668)
+    (local.tee $0
+     (i32.add
+      (i32.load
+       (i32.const 3668)
+      )
+      (i32.const -1)
+     )
+    )
+   )
+   (if
+    (local.get $0)
+    (return)
+    (local.set $0
+     (i32.const 4092)
+    )
+   )
+   (loop $label$128
+    (local.set $0
+     (i32.add
+      (local.tee $2
+       (i32.load
+        (local.get $0)
+       )
+      )
+      (i32.const 8)
+     )
+    )
+    (br_if $label$128
+     (local.get $2)
+    )
+   )
+   (i32.store
+    (i32.const 3668)
+    (i32.const -1)
+   )
+  )
+ )
+ (func $39 (; 52 ;) (type $6)
+  (nop)
+ )
+ (func $40 (; 53 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (i32.add
+     (local.tee $2
+      (i32.load
+       (global.get $global$0)
+      )
+     )
+     (local.tee $0
+      (i32.and
+       (i32.add
+        (local.get $0)
+        (i32.const 15)
+       )
+       (i32.const -16)
+      )
+     )
+    )
+   )
+   (if
+    (i32.or
+     (i32.and
+      (i32.gt_s
+       (local.get $0)
+       (i32.const 0)
+      )
+      (i32.lt_s
+       (local.get $1)
+       (local.get $2)
+      )
+     )
+     (i32.lt_s
+      (local.get $1)
+      (i32.const 0)
+     )
+    )
+    (block
+     (drop
+      (call $fimport$6)
+     )
+     (call $fimport$11
+      (i32.const 12)
+     )
+     (return
+      (i32.const -1)
+     )
+    )
+   )
+   (i32.store
+    (global.get $global$0)
+    (local.get $1)
+   )
+   (if
+    (i32.gt_s
+     (local.get $1)
+     (call $fimport$5)
+    )
+    (if
+     (i32.eqz
+      (call $fimport$4)
+     )
+     (block
+      (call $fimport$11
+       (i32.const 12)
+      )
+      (i32.store
+       (global.get $global$0)
+       (local.get $2)
+      )
+      (return
+       (i32.const -1)
+      )
+     )
+    )
+   )
+   (local.get $2)
+  )
+ )
+ (func $41 (; 54 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (i32.add
+     (local.get $0)
+     (local.get $2)
+    )
+   )
+   (if
+    (i32.ge_s
+     (local.get $2)
+     (i32.const 20)
+    )
+    (block
+     (local.set $1
+      (i32.and
+       (local.get $1)
+       (i32.const 255)
+      )
+     )
+     (if
+      (local.tee $3
+       (i32.and
+        (local.get $0)
+        (i32.const 3)
+       )
+      )
+      (block
+       (local.set $3
+        (i32.sub
+         (i32.add
+          (local.get $0)
+          (i32.const 4)
+         )
+         (local.get $3)
+        )
+       )
+       (loop $label$4
+        (if
+         (i32.lt_s
+          (local.get $0)
+          (local.get $3)
+         )
+         (block
+          (i32.store8
+           (local.get $0)
+           (local.get $1)
+          )
+          (local.set $0
+           (i32.add
+            (local.get $0)
+            (i32.const 1)
+           )
+          )
+          (br $label$4)
+         )
+        )
+       )
+      )
+     )
+     (local.set $3
+      (i32.or
+       (i32.or
+        (i32.or
+         (local.get $1)
+         (i32.shl
+          (local.get $1)
+          (i32.const 8)
+         )
+        )
+        (i32.shl
+         (local.get $1)
+         (i32.const 16)
+        )
+       )
+       (i32.shl
+        (local.get $1)
+        (i32.const 24)
+       )
+      )
+     )
+     (local.set $5
+      (i32.and
+       (local.get $4)
+       (i32.const -4)
+      )
+     )
+     (loop $label$6
+      (if
+       (i32.lt_s
+        (local.get $0)
+        (local.get $5)
+       )
+       (block
+        (i32.store
+         (local.get $0)
+         (local.get $3)
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (i32.const 4)
+         )
+        )
+        (br $label$6)
+       )
+      )
+     )
+    )
+   )
+   (loop $label$8
+    (if
+     (i32.lt_s
+      (local.get $0)
+      (local.get $4)
+     )
+     (block
+      (i32.store8
+       (local.get $0)
+       (local.get $1)
+      )
+      (local.set $0
+       (i32.add
+        (local.get $0)
+        (i32.const 1)
+       )
+      )
+      (br $label$8)
+     )
+    )
+   )
+   (i32.sub
+    (local.get $0)
+    (local.get $2)
+   )
+  )
+ )
+ (func $42 (; 55 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (block $label$1 (result i32)
+   (if
+    (i32.ge_s
+     (local.get $2)
+     (i32.const 4096)
+    )
+    (return
+     (call $fimport$12
+      (local.get $0)
+      (local.get $1)
+      (local.get $2)
+     )
+    )
+   )
+   (local.set $3
+    (local.get $0)
+   )
+   (if
+    (i32.eq
+     (i32.and
+      (local.get $0)
+      (i32.const 3)
+     )
+     (i32.and
+      (local.get $1)
+      (i32.const 3)
+     )
+    )
+    (block
+     (loop $label$4
+      (if
+       (i32.and
+        (local.get $0)
+        (i32.const 3)
+       )
+       (block
+        (if
+         (i32.eqz
+          (local.get $2)
+         )
+         (return
+          (local.get $3)
+         )
+        )
+        (i32.store8
+         (local.get $0)
+         (i32.load8_s
+          (local.get $1)
+         )
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (i32.const 1)
+         )
+        )
+        (local.set $1
+         (i32.add
+          (local.get $1)
+          (i32.const 1)
+         )
+        )
+        (local.set $2
+         (i32.sub
+          (local.get $2)
+          (i32.const 1)
+         )
+        )
+        (br $label$4)
+       )
+      )
+     )
+     (loop $label$7
+      (if
+       (i32.ge_s
+        (local.get $2)
+        (i32.const 4)
+       )
+       (block
+        (i32.store
+         (local.get $0)
+         (i32.load
+          (local.get $1)
+         )
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (i32.const 4)
+         )
+        )
+        (local.set $1
+         (i32.add
+          (local.get $1)
+          (i32.const 4)
+         )
+        )
+        (local.set $2
+         (i32.sub
+          (local.get $2)
+          (i32.const 4)
+         )
+        )
+        (br $label$7)
+       )
+      )
+     )
+    )
+   )
+   (loop $label$9
+    (if
+     (i32.gt_s
+      (local.get $2)
+      (i32.const 0)
+     )
+     (block
+      (i32.store8
+       (local.get $0)
+       (i32.load8_s
+        (local.get $1)
+       )
+      )
+      (local.set $0
+       (i32.add
+        (local.get $0)
+        (i32.const 1)
+       )
+      )
+      (local.set $1
+       (i32.add
+        (local.get $1)
+        (i32.const 1)
+       )
+      )
+      (local.set $2
+       (i32.sub
+        (local.get $2)
+        (i32.const 1)
+       )
+      )
+      (br $label$9)
+     )
+    )
+   )
+   (local.get $3)
+  )
+ )
+ (func $43 (; 56 ;) (type $3) (result i32)
+  (i32.const 0)
+ )
+ (func $44 (; 57 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (call_indirect (type $1)
+   (local.get $1)
+   (i32.add
+    (i32.and
+     (local.get $0)
+     (i32.const 1)
+    )
+    (i32.const 0)
+   )
+  )
+ )
+ (func $45 (; 58 ;) (type $12) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (result i32)
+  (call_indirect (type $0)
+   (local.get $1)
+   (local.get $2)
+   (local.get $3)
+   (i32.add
+    (i32.and
+     (local.get $0)
+     (i32.const 3)
+    )
+    (i32.const 2)
+   )
+  )
+ )
+ (func $46 (; 59 ;) (type $5) (param $0 i32) (param $1 i32)
+  (call_indirect (type $2)
+   (local.get $1)
+   (i32.add
+    (i32.and
+     (local.get $0)
+     (i32.const 1)
+    )
+    (i32.const 6)
+   )
+  )
+ )
+ (func $47 (; 60 ;) (type $1) (param $0 i32) (result i32)
+  (block $label$1 (result i32)
+   (call $fimport$3
+    (i32.const 0)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $48 (; 61 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (block $label$1 (result i32)
+   (call $fimport$3
+    (i32.const 1)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $49 (; 62 ;) (type $2) (param $0 i32)
+  (call $fimport$3
+   (i32.const 2)
+  )
+ )
+)
+
diff --git a/cranelift/wasmtests/embenchen_primes.wat b/cranelift/wasmtests/embenchen_primes.wat
new file mode 100644
index 0000000000..4f4603198c
--- /dev/null
+++ b/cranelift/wasmtests/embenchen_primes.wat
@@ -0,0 +1,15334 @@
+(module
+ (type $0 (func (param i32 i32 i32) (result i32)))
+ (type $1 (func (param i32) (result i32)))
+ (type $2 (func (param i32)))
+ (type $3 (func (result i32)))
+ (type $4 (func (param i32 i32) (result i32)))
+ (type $5 (func (param i32 i32)))
+ (type $6 (func))
+ (type $7 (func (param i32 i32 i32 i32 i32) (result i32)))
+ (type $8 (func (param i32 i32 i32)))
+ (type $9 (func (param i64 i32) (result i32)))
+ (type $10 (func (param i32 i32 i32 i32 i32)))
+ (type $11 (func (param f64 i32) (result f64)))
+ (type $12 (func (param i32 i32 i32 i32) (result i32)))
+ (import "env" "memory" (memory $16 2048 2048))
+ (data (i32.const 1024) "\04\04\00\00\05")
+ (data (i32.const 1040) "\01")
+ (data (i32.const 1064) "\01\00\00\00\02\00\00\004\10\00\00\00\04")
+ (data (i32.const 1088) "\01")
+ (data (i32.const 1103) "\n\ff\ff\ff\ff")
+ (data (i32.const 1140) "error: %d\\n\00lastprime: %d.\n\00\11\00\n\00\11\11\11\00\00\00\00\05\00\00\00\00\00\00\t\00\00\00\00\0b")
+ (data (i32.const 1200) "\11\00\0f\n\11\11\11\03\n\07\00\01\13\t\0b\0b\00\00\t\06\0b\00\00\0b\00\06\11\00\00\00\11\11\11")
+ (data (i32.const 1249) "\0b")
+ (data (i32.const 1258) "\11\00\n\n\11\11\11\00\n\00\00\02\00\t\0b\00\00\00\t\00\0b\00\00\0b")
+ (data (i32.const 1307) "\0c")
+ (data (i32.const 1319) "\0c\00\00\00\00\0c\00\00\00\00\t\0c\00\00\00\00\00\0c\00\00\0c")
+ (data (i32.const 1365) "\0e")
+ (data (i32.const 1377) "\0d\00\00\00\04\0d\00\00\00\00\t\0e\00\00\00\00\00\0e\00\00\0e")
+ (data (i32.const 1423) "\10")
+ (data (i32.const 1435) "\0f\00\00\00\00\0f\00\00\00\00\t\10\00\00\00\00\00\10\00\00\10\00\00\12\00\00\00\12\12\12")
+ (data (i32.const 1490) "\12\00\00\00\12\12\12\00\00\00\00\00\00\t")
+ (data (i32.const 1539) "\0b")
+ (data (i32.const 1551) "\n\00\00\00\00\n\00\00\00\00\t\0b\00\00\00\00\00\0b\00\00\0b")
+ (data (i32.const 1597) "\0c")
+ (data (i32.const 1609) "\0c\00\00\00\00\0c\00\00\00\00\t\0c\00\00\00\00\00\0c\00\00\0c\00\000123456789ABCDEF-+   0X0x\00(null)\00-0X+0X 0X-0x+0x 0x\00inf\00INF\00nan\00NAN\00.\00T!\"\19\0d\01\02\03\11K\1c\0c\10\04\0b\1d\12\1e\'hnopqb \05\06\0f\13\14\15\1a\08\16\07($\17\18\t\n\0e\1b\1f%#\83\82}&*+<=>?CGJMXYZ[\\]^_`acdefgijklrstyz{|\00Illegal byte sequence\00Domain error\00Result not representable\00Not a tty\00Permission denied\00Operation not permitted\00No such file or directory\00No such process\00File exists\00Value too large for data type\00No space left on device\00Out of memory\00Resource busy\00Interrupted system call\00Resource temporarily unavailable\00Invalid seek\00Cross-device link\00Read-only file system\00Directory not empty\00Connection reset by peer\00Operation timed out\00Connection refused\00Host is down\00Host is unreachable\00Address in use\00Broken pipe\00I/O error\00No such device or address\00Block device required\00No such device\00Not a directory\00Is a directory\00Text file busy\00Exec format error\00Invalid argument\00Argument list too long\00Symbolic link loop\00Filename too long\00Too many open files in system\00No file descriptors available\00Bad file descriptor\00No child process\00Bad address\00File too large\00Too many links\00No locks available\00Resource deadlock would occur\00State not recoverable\00Previous owner died\00Operation canceled\00Function not implemented\00No message of desired type\00Identifier removed\00Device not a stream\00No data available\00Device timeout\00Out of streams resources\00Link has been severed\00Protocol error\00Bad message\00File descriptor in bad state\00Not a socket\00Destination address required\00Message too large\00Protocol wrong type for socket\00Protocol not available\00Protocol not supported\00Socket type not supported\00Not supported\00Protocol family not supported\00Address family not supported by protocol\00Address not available\00Network is down\00Network unreachable\00Connection reset by network\00Connection aborted\00No buffer space available\00Socket is connected\00Socket not connected\00Cannot send after socket shutdown\00Operation already in progress\00Operation in progress\00Stale file handle\00Remote I/O error\00Quota exceeded\00No medium found\00Wrong medium type\00No error information")
+ (import "env" "table" (table $timport$17 8 8 funcref))
+ (elem (global.get $gimport$19) $41 $8 $42 $13 $9 $14 $43 $15)
+ (import "env" "DYNAMICTOP_PTR" (global $gimport$0 i32))
+ (import "env" "STACKTOP" (global $gimport$1 i32))
+ (import "env" "STACK_MAX" (global $gimport$2 i32))
+ (import "env" "memoryBase" (global $gimport$18 i32))
+ (import "env" "tableBase" (global $gimport$19 i32))
+ (import "env" "abort" (func $fimport$3 (param i32)))
+ (import "env" "enlargeMemory" (func $fimport$4 (result i32)))
+ (import "env" "getTotalMemory" (func $fimport$5 (result i32)))
+ (import "env" "abortOnCannotGrowMemory" (func $fimport$6 (result i32)))
+ (import "env" "_pthread_cleanup_pop" (func $fimport$7 (param i32)))
+ (import "env" "___syscall6" (func $fimport$8 (param i32 i32) (result i32)))
+ (import "env" "_pthread_cleanup_push" (func $fimport$9 (param i32 i32)))
+ (import "env" "_abort" (func $fimport$10))
+ (import "env" "___setErrNo" (func $fimport$11 (param i32)))
+ (import "env" "_emscripten_memcpy_big" (func $fimport$12 (param i32 i32 i32) (result i32)))
+ (import "env" "___syscall54" (func $fimport$13 (param i32 i32) (result i32)))
+ (import "env" "___syscall140" (func $fimport$14 (param i32 i32) (result i32)))
+ (import "env" "___syscall146" (func $fimport$15 (param i32 i32) (result i32)))
+ (global $global$0 (mut i32) (global.get $gimport$0))
+ (global $global$1 (mut i32) (global.get $gimport$1))
+ (global $global$2 (mut i32) (global.get $gimport$2))
+ (global $global$3 (mut i32) (i32.const 0))
+ (global $global$4 (mut i32) (i32.const 0))
+ (global $global$5 (mut i32) (i32.const 0))
+ (export "_sbrk" (func $34))
+ (export "_free" (func $32))
+ (export "_main" (func $7))
+ (export "_pthread_self" (func $37))
+ (export "_memset" (func $35))
+ (export "_malloc" (func $31))
+ (export "_memcpy" (func $36))
+ (export "___errno_location" (func $11))
+ (export "runPostSets" (func $33))
+ (export "stackAlloc" (func $0))
+ (export "stackSave" (func $1))
+ (export "stackRestore" (func $2))
+ (export "establishStackSpace" (func $3))
+ (export "setThrew" (func $4))
+ (export "setTempRet0" (func $5))
+ (export "getTempRet0" (func $6))
+ (export "dynCall_ii" (func $38))
+ (export "dynCall_iiii" (func $39))
+ (export "dynCall_vi" (func $40))
+ (func $0 (; 13 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (local.get $0)
+    )
+   )
+   (global.set $global$1
+    (i32.and
+     (i32.add
+      (global.get $global$1)
+      (i32.const 15)
+     )
+     (i32.const -16)
+    )
+   )
+   (local.get $1)
+  )
+ )
+ (func $1 (; 14 ;) (type $3) (result i32)
+  (global.get $global$1)
+ )
+ (func $2 (; 15 ;) (type $2) (param $0 i32)
+  (global.set $global$1
+   (local.get $0)
+  )
+ )
+ (func $3 (; 16 ;) (type $5) (param $0 i32) (param $1 i32)
+  (block $label$1
+   (global.set $global$1
+    (local.get $0)
+   )
+   (global.set $global$2
+    (local.get $1)
+   )
+  )
+ )
+ (func $4 (; 17 ;) (type $5) (param $0 i32) (param $1 i32)
+  (if
+   (i32.eqz
+    (global.get $global$3)
+   )
+   (block
+    (global.set $global$3
+     (local.get $0)
+    )
+    (global.set $global$4
+     (local.get $1)
+    )
+   )
+  )
+ )
+ (func $5 (; 18 ;) (type $2) (param $0 i32)
+  (global.set $global$5
+   (local.get $0)
+  )
+ )
+ (func $6 (; 19 ;) (type $3) (result i32)
+  (global.get $global$5)
+ )
+ (func $7 (; 20 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 f32)
+  (block $label$1 (result i32)
+   (local.set $3
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (local.set $5
+    (i32.add
+     (local.get $3)
+     (i32.const 8)
+    )
+   )
+   (local.set $2
+    (local.get $3)
+   )
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (i32.le_s
+       (local.get $0)
+       (i32.const 1)
+      )
+     )
+     (block $label$4
+      (block $label$5
+       (block $label$6
+        (block $label$7
+         (block $label$8
+          (block $label$9
+           (block $label$10
+            (br_table $label$5 $label$10 $label$8 $label$9 $label$7 $label$6 $label$4
+             (i32.sub
+              (local.tee $0
+               (i32.load8_s
+                (i32.load offset=4
+                 (local.get $1)
+                )
+               )
+              )
+              (i32.const 48)
+             )
+            )
+           )
+           (local.set $4
+            (i32.const 33000)
+           )
+           (br $label$2)
+          )
+          (br $label$3)
+         )
+         (local.set $4
+          (i32.const 130000)
+         )
+         (br $label$2)
+        )
+        (local.set $4
+         (i32.const 610000)
+        )
+        (br $label$2)
+       )
+       (local.set $4
+        (i32.const 1010000)
+       )
+       (br $label$2)
+      )
+      (global.set $global$1
+       (local.get $3)
+      )
+      (return
+       (i32.const 0)
+      )
+     )
+     (i32.store
+      (local.get $2)
+      (i32.add
+       (local.get $0)
+       (i32.const -48)
+      )
+     )
+     (drop
+      (call $30
+       (i32.const 1140)
+       (local.get $2)
+      )
+     )
+     (global.set $global$1
+      (local.get $3)
+     )
+     (return
+      (i32.const -1)
+     )
+    )
+    (local.set $4
+     (i32.const 220000)
+    )
+   )
+   (local.set $1
+    (i32.const 2)
+   )
+   (local.set $0
+    (i32.const 0)
+   )
+   (loop $label$11
+    (block $label$12
+     (block $label$13
+      (br_if $label$13
+       (i32.eqz
+        (f32.gt
+         (local.tee $6
+          (f32.sqrt
+           (f32.convert_i32_s
+            (local.get $1)
+           )
+          )
+         )
+         (f32.const 2)
+        )
+       )
+      )
+      (local.set $2
+       (i32.const 2)
+      )
+      (loop $label$14
+       (br_if $label$12
+        (i32.eqz
+         (i32.rem_s
+          (local.get $1)
+          (local.get $2)
+         )
+        )
+       )
+       (br_if $label$14
+        (f32.lt
+         (f32.convert_i32_s
+          (local.tee $2
+           (i32.add
+            (local.get $2)
+            (i32.const 1)
+           )
+          )
+         )
+         (local.get $6)
+        )
+       )
+       (br $label$13)
+      )
+     )
+     (local.set $0
+      (i32.add
+       (local.get $0)
+       (i32.const 1)
+      )
+     )
+    )
+    (local.set $2
+     (i32.add
+      (local.get $1)
+      (i32.const 1)
+     )
+    )
+    (if
+     (i32.lt_s
+      (local.get $0)
+      (local.get $4)
+     )
+     (block
+      (local.set $1
+       (local.get $2)
+      )
+      (br $label$11)
+     )
+    )
+   )
+   (i32.store
+    (local.get $5)
+    (local.get $1)
+   )
+   (drop
+    (call $30
+     (i32.const 1152)
+     (local.get $5)
+    )
+   )
+   (global.set $global$1
+    (local.get $3)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $8 (; 21 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (i32.store
+    (local.tee $2
+     (local.get $1)
+    )
+    (i32.load offset=60
+     (local.get $0)
+    )
+   )
+   (local.set $0
+    (call $10
+     (call $fimport$8
+      (i32.const 6)
+      (local.get $2)
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $1)
+   )
+   (local.get $0)
+  )
+ )
+ (func $9 (; 22 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 32)
+    )
+   )
+   (i32.store
+    (local.tee $3
+     (local.get $4)
+    )
+    (i32.load offset=60
+     (local.get $0)
+    )
+   )
+   (i32.store offset=4
+    (local.get $3)
+    (i32.const 0)
+   )
+   (i32.store offset=8
+    (local.get $3)
+    (local.get $1)
+   )
+   (i32.store offset=12
+    (local.get $3)
+    (local.tee $0
+     (i32.add
+      (local.get $4)
+      (i32.const 20)
+     )
+    )
+   )
+   (i32.store offset=16
+    (local.get $3)
+    (local.get $2)
+   )
+   (local.set $0
+    (if (result i32)
+     (i32.lt_s
+      (call $10
+       (call $fimport$14
+        (i32.const 140)
+        (local.get $3)
+       )
+      )
+      (i32.const 0)
+     )
+     (block (result i32)
+      (i32.store
+       (local.get $0)
+       (i32.const -1)
+      )
+      (i32.const -1)
+     )
+     (i32.load
+      (local.get $0)
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $4)
+   )
+   (local.get $0)
+  )
+ )
+ (func $10 (; 23 ;) (type $1) (param $0 i32) (result i32)
+  (if (result i32)
+   (i32.gt_u
+    (local.get $0)
+    (i32.const -4096)
+   )
+   (block (result i32)
+    (i32.store
+     (call $11)
+     (i32.sub
+      (i32.const 0)
+      (local.get $0)
+     )
+    )
+    (i32.const -1)
+   )
+   (local.get $0)
+  )
+ )
+ (func $11 (; 24 ;) (type $3) (result i32)
+  (i32.const 3640)
+ )
+ (func $12 (; 25 ;) (type $2) (param $0 i32)
+  (nop)
+ )
+ (func $13 (; 26 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 80)
+    )
+   )
+   (local.set $3
+    (local.get $4)
+   )
+   (local.set $5
+    (i32.add
+     (local.get $4)
+     (i32.const 12)
+    )
+   )
+   (i32.store offset=36
+    (local.get $0)
+    (i32.const 3)
+   )
+   (if
+    (i32.eqz
+     (i32.and
+      (i32.load
+       (local.get $0)
+      )
+      (i32.const 64)
+     )
+    )
+    (block
+     (i32.store
+      (local.get $3)
+      (i32.load offset=60
+       (local.get $0)
+      )
+     )
+     (i32.store offset=4
+      (local.get $3)
+      (i32.const 21505)
+     )
+     (i32.store offset=8
+      (local.get $3)
+      (local.get $5)
+     )
+     (if
+      (call $fimport$13
+       (i32.const 54)
+       (local.get $3)
+      )
+      (i32.store8 offset=75
+       (local.get $0)
+       (i32.const -1)
+      )
+     )
+    )
+   )
+   (local.set $0
+    (call $14
+     (local.get $0)
+     (local.get $1)
+     (local.get $2)
+    )
+   )
+   (global.set $global$1
+    (local.get $4)
+   )
+   (local.get $0)
+  )
+ )
+ (func $14 (; 27 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (block $label$1 (result i32)
+   (local.set $8
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 48)
+    )
+   )
+   (local.set $9
+    (i32.add
+     (local.get $8)
+     (i32.const 16)
+    )
+   )
+   (local.set $10
+    (local.get $8)
+   )
+   (i32.store
+    (local.tee $3
+     (i32.add
+      (local.get $8)
+      (i32.const 32)
+     )
+    )
+    (local.tee $4
+     (i32.load
+      (local.tee $6
+       (i32.add
+        (local.get $0)
+        (i32.const 28)
+       )
+      )
+     )
+    )
+   )
+   (i32.store offset=4
+    (local.get $3)
+    (local.tee $5
+     (i32.sub
+      (i32.load
+       (local.tee $11
+        (i32.add
+         (local.get $0)
+         (i32.const 20)
+        )
+       )
+      )
+      (local.get $4)
+     )
+    )
+   )
+   (i32.store offset=8
+    (local.get $3)
+    (local.get $1)
+   )
+   (i32.store offset=12
+    (local.get $3)
+    (local.get $2)
+   )
+   (local.set $13
+    (i32.add
+     (local.get $0)
+     (i32.const 60)
+    )
+   )
+   (local.set $14
+    (i32.add
+     (local.get $0)
+     (i32.const 44)
+    )
+   )
+   (local.set $1
+    (local.get $3)
+   )
+   (local.set $4
+    (i32.const 2)
+   )
+   (local.set $12
+    (i32.add
+     (local.get $5)
+     (local.get $2)
+    )
+   )
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (loop $label$5
+       (if
+        (i32.load
+         (i32.const 3596)
+        )
+        (block
+         (call $fimport$9
+          (i32.const 1)
+          (local.get $0)
+         )
+         (i32.store
+          (local.get $10)
+          (i32.load
+           (local.get $13)
+          )
+         )
+         (i32.store offset=4
+          (local.get $10)
+          (local.get $1)
+         )
+         (i32.store offset=8
+          (local.get $10)
+          (local.get $4)
+         )
+         (local.set $3
+          (call $10
+           (call $fimport$15
+            (i32.const 146)
+            (local.get $10)
+           )
+          )
+         )
+         (call $fimport$7
+          (i32.const 0)
+         )
+        )
+        (block
+         (i32.store
+          (local.get $9)
+          (i32.load
+           (local.get $13)
+          )
+         )
+         (i32.store offset=4
+          (local.get $9)
+          (local.get $1)
+         )
+         (i32.store offset=8
+          (local.get $9)
+          (local.get $4)
+         )
+         (local.set $3
+          (call $10
+           (call $fimport$15
+            (i32.const 146)
+            (local.get $9)
+           )
+          )
+         )
+        )
+       )
+       (br_if $label$4
+        (i32.eq
+         (local.get $12)
+         (local.get $3)
+        )
+       )
+       (br_if $label$3
+        (i32.lt_s
+         (local.get $3)
+         (i32.const 0)
+        )
+       )
+       (local.set $5
+        (if (result i32)
+         (i32.gt_u
+          (local.get $3)
+          (local.tee $5
+           (i32.load offset=4
+            (local.get $1)
+           )
+          )
+         )
+         (block (result i32)
+          (i32.store
+           (local.get $6)
+           (local.tee $7
+            (i32.load
+             (local.get $14)
+            )
+           )
+          )
+          (i32.store
+           (local.get $11)
+           (local.get $7)
+          )
+          (local.set $7
+           (i32.load offset=12
+            (local.get $1)
+           )
+          )
+          (local.set $1
+           (i32.add
+            (local.get $1)
+            (i32.const 8)
+           )
+          )
+          (local.set $4
+           (i32.add
+            (local.get $4)
+            (i32.const -1)
+           )
+          )
+          (i32.sub
+           (local.get $3)
+           (local.get $5)
+          )
+         )
+         (if (result i32)
+          (i32.eq
+           (local.get $4)
+           (i32.const 2)
+          )
+          (block (result i32)
+           (i32.store
+            (local.get $6)
+            (i32.add
+             (i32.load
+              (local.get $6)
+             )
+             (local.get $3)
+            )
+           )
+           (local.set $7
+            (local.get $5)
+           )
+           (local.set $4
+            (i32.const 2)
+           )
+           (local.get $3)
+          )
+          (block (result i32)
+           (local.set $7
+            (local.get $5)
+           )
+           (local.get $3)
+          )
+         )
+        )
+       )
+       (i32.store
+        (local.get $1)
+        (i32.add
+         (i32.load
+          (local.get $1)
+         )
+         (local.get $5)
+        )
+       )
+       (i32.store offset=4
+        (local.get $1)
+        (i32.sub
+         (local.get $7)
+         (local.get $5)
+        )
+       )
+       (local.set $12
+        (i32.sub
+         (local.get $12)
+         (local.get $3)
+        )
+       )
+       (br $label$5)
+      )
+     )
+     (i32.store offset=16
+      (local.get $0)
+      (i32.add
+       (local.tee $1
+        (i32.load
+         (local.get $14)
+        )
+       )
+       (i32.load offset=48
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store
+      (local.get $6)
+      (local.get $1)
+     )
+     (i32.store
+      (local.get $11)
+      (local.get $1)
+     )
+     (br $label$2)
+    )
+    (i32.store offset=16
+     (local.get $0)
+     (i32.const 0)
+    )
+    (i32.store
+     (local.get $6)
+     (i32.const 0)
+    )
+    (i32.store
+     (local.get $11)
+     (i32.const 0)
+    )
+    (i32.store
+     (local.get $0)
+     (i32.or
+      (i32.load
+       (local.get $0)
+      )
+      (i32.const 32)
+     )
+    )
+    (local.set $2
+     (if (result i32)
+      (i32.eq
+       (local.get $4)
+       (i32.const 2)
+      )
+      (i32.const 0)
+      (i32.sub
+       (local.get $2)
+       (i32.load offset=4
+        (local.get $1)
+       )
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $8)
+   )
+   (local.get $2)
+  )
+ )
+ (func $15 (; 28 ;) (type $2) (param $0 i32)
+  (if
+   (i32.eqz
+    (i32.load offset=68
+     (local.get $0)
+    )
+   )
+   (call $12
+    (local.get $0)
+   )
+  )
+ )
+ (func $16 (; 29 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $5
+    (i32.and
+     (local.get $1)
+     (i32.const 255)
+    )
+   )
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (if
+       (i32.and
+        (local.tee $4
+         (i32.ne
+          (local.get $2)
+          (i32.const 0)
+         )
+        )
+        (i32.ne
+         (i32.and
+          (local.get $0)
+          (i32.const 3)
+         )
+         (i32.const 0)
+        )
+       )
+       (block
+        (local.set $4
+         (i32.and
+          (local.get $1)
+          (i32.const 255)
+         )
+        )
+        (local.set $3
+         (local.get $2)
+        )
+        (local.set $2
+         (local.get $0)
+        )
+        (loop $label$6
+         (if
+          (i32.eq
+           (i32.load8_s
+            (local.get $2)
+           )
+           (i32.shr_s
+            (i32.shl
+             (local.get $4)
+             (i32.const 24)
+            )
+            (i32.const 24)
+           )
+          )
+          (block
+           (local.set $0
+            (local.get $3)
+           )
+           (br $label$3)
+          )
+         )
+         (br_if $label$6
+          (i32.and
+           (local.tee $0
+            (i32.ne
+             (local.tee $3
+              (i32.add
+               (local.get $3)
+               (i32.const -1)
+              )
+             )
+             (i32.const 0)
+            )
+           )
+           (i32.ne
+            (i32.and
+             (local.tee $2
+              (i32.add
+               (local.get $2)
+               (i32.const 1)
+              )
+             )
+             (i32.const 3)
+            )
+            (i32.const 0)
+           )
+          )
+         )
+         (br $label$4)
+        )
+       )
+       (block
+        (local.set $3
+         (local.get $2)
+        )
+        (local.set $2
+         (local.get $0)
+        )
+        (local.set $0
+         (local.get $4)
+        )
+       )
+      )
+     )
+     (if
+      (local.get $0)
+      (block
+       (local.set $0
+        (local.get $3)
+       )
+       (br $label$3)
+      )
+      (local.set $0
+       (i32.const 0)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.ne
+      (i32.load8_s
+       (local.get $2)
+      )
+      (i32.shr_s
+       (i32.shl
+        (local.tee $1
+         (i32.and
+          (local.get $1)
+          (i32.const 255)
+         )
+        )
+        (i32.const 24)
+       )
+       (i32.const 24)
+      )
+     )
+     (block
+      (local.set $3
+       (i32.mul
+        (local.get $5)
+        (i32.const 16843009)
+       )
+      )
+      (block $label$12
+       (block $label$13
+        (br_if $label$13
+         (i32.le_u
+          (local.get $0)
+          (i32.const 3)
+         )
+        )
+        (loop $label$14
+         (if
+          (i32.eqz
+           (i32.and
+            (i32.xor
+             (i32.and
+              (local.tee $4
+               (i32.xor
+                (i32.load
+                 (local.get $2)
+                )
+                (local.get $3)
+               )
+              )
+              (i32.const -2139062144)
+             )
+             (i32.const -2139062144)
+            )
+            (i32.add
+             (local.get $4)
+             (i32.const -16843009)
+            )
+           )
+          )
+          (block
+           (local.set $2
+            (i32.add
+             (local.get $2)
+             (i32.const 4)
+            )
+           )
+           (br_if $label$14
+            (i32.gt_u
+             (local.tee $0
+              (i32.add
+               (local.get $0)
+               (i32.const -4)
+              )
+             )
+             (i32.const 3)
+            )
+           )
+           (br $label$13)
+          )
+         )
+        )
+        (br $label$12)
+       )
+       (if
+        (i32.eqz
+         (local.get $0)
+        )
+        (block
+         (local.set $0
+          (i32.const 0)
+         )
+         (br $label$2)
+        )
+       )
+      )
+      (loop $label$17
+       (br_if $label$2
+        (i32.eq
+         (i32.load8_s
+          (local.get $2)
+         )
+         (i32.shr_s
+          (i32.shl
+           (local.get $1)
+           (i32.const 24)
+          )
+          (i32.const 24)
+         )
+        )
+       )
+       (local.set $2
+        (i32.add
+         (local.get $2)
+         (i32.const 1)
+        )
+       )
+       (br_if $label$17
+        (local.tee $0
+         (i32.add
+          (local.get $0)
+          (i32.const -1)
+         )
+        )
+       )
+       (local.set $0
+        (i32.const 0)
+       )
+      )
+     )
+    )
+   )
+   (if (result i32)
+    (local.get $0)
+    (local.get $2)
+    (i32.const 0)
+   )
+  )
+ )
+ (func $17 (; 30 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 224)
+    )
+   )
+   (local.set $5
+    (i32.add
+     (local.get $4)
+     (i32.const 136)
+    )
+   )
+   (i64.store align=4
+    (local.tee $3
+     (i32.add
+      (local.get $4)
+      (i32.const 80)
+     )
+    )
+    (i64.const 0)
+   )
+   (i64.store offset=8 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i64.store offset=16 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i64.store offset=24 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i64.store offset=32 align=4
+    (local.get $3)
+    (i64.const 0)
+   )
+   (i32.store
+    (local.tee $6
+     (i32.add
+      (local.get $4)
+      (i32.const 120)
+     )
+    )
+    (i32.load
+     (local.get $2)
+    )
+   )
+   (if
+    (i32.lt_s
+     (call $18
+      (i32.const 0)
+      (local.get $1)
+      (local.get $6)
+      (local.tee $2
+       (local.get $4)
+      )
+      (local.get $3)
+     )
+     (i32.const 0)
+    )
+    (local.set $1
+     (i32.const -1)
+    )
+    (block
+     (local.set $12
+      (if (result i32)
+       (i32.gt_s
+        (i32.load offset=76
+         (local.get $0)
+        )
+        (i32.const -1)
+       )
+       (call $19
+        (local.get $0)
+       )
+       (i32.const 0)
+      )
+     )
+     (local.set $7
+      (i32.load
+       (local.get $0)
+      )
+     )
+     (if
+      (i32.lt_s
+       (i32.load8_s offset=74
+        (local.get $0)
+       )
+       (i32.const 1)
+      )
+      (i32.store
+       (local.get $0)
+       (i32.and
+        (local.get $7)
+        (i32.const -33)
+       )
+      )
+     )
+     (if
+      (i32.load
+       (local.tee $8
+        (i32.add
+         (local.get $0)
+         (i32.const 48)
+        )
+       )
+      )
+      (local.set $1
+       (call $18
+        (local.get $0)
+        (local.get $1)
+        (local.get $6)
+        (local.get $2)
+        (local.get $3)
+       )
+      )
+      (block
+       (local.set $10
+        (i32.load
+         (local.tee $9
+          (i32.add
+           (local.get $0)
+           (i32.const 44)
+          )
+         )
+        )
+       )
+       (i32.store
+        (local.get $9)
+        (local.get $5)
+       )
+       (i32.store
+        (local.tee $13
+         (i32.add
+          (local.get $0)
+          (i32.const 28)
+         )
+        )
+        (local.get $5)
+       )
+       (i32.store
+        (local.tee $11
+         (i32.add
+          (local.get $0)
+          (i32.const 20)
+         )
+        )
+        (local.get $5)
+       )
+       (i32.store
+        (local.get $8)
+        (i32.const 80)
+       )
+       (i32.store
+        (local.tee $14
+         (i32.add
+          (local.get $0)
+          (i32.const 16)
+         )
+        )
+        (i32.add
+         (local.get $5)
+         (i32.const 80)
+        )
+       )
+       (local.set $1
+        (call $18
+         (local.get $0)
+         (local.get $1)
+         (local.get $6)
+         (local.get $2)
+         (local.get $3)
+        )
+       )
+       (if
+        (local.get $10)
+        (block
+         (drop
+          (call_indirect (type $0)
+           (local.get $0)
+           (i32.const 0)
+           (i32.const 0)
+           (i32.add
+            (i32.and
+             (i32.load offset=36
+              (local.get $0)
+             )
+             (i32.const 3)
+            )
+            (i32.const 2)
+           )
+          )
+         )
+         (if
+          (i32.eqz
+           (i32.load
+            (local.get $11)
+           )
+          )
+          (local.set $1
+           (i32.const -1)
+          )
+         )
+         (i32.store
+          (local.get $9)
+          (local.get $10)
+         )
+         (i32.store
+          (local.get $8)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $14)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $13)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $11)
+          (i32.const 0)
+         )
+        )
+       )
+      )
+     )
+     (i32.store
+      (local.get $0)
+      (i32.or
+       (local.tee $2
+        (i32.load
+         (local.get $0)
+        )
+       )
+       (i32.and
+        (local.get $7)
+        (i32.const 32)
+       )
+      )
+     )
+     (if
+      (local.get $12)
+      (call $12
+       (local.get $0)
+      )
+     )
+     (if
+      (i32.and
+       (local.get $2)
+       (i32.const 32)
+      )
+      (local.set $1
+       (i32.const -1)
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $4)
+   )
+   (local.get $1)
+  )
+ )
+ (func $18 (; 31 ;) (type $7) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (param $4 i32) (result i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (local $16 i32)
+  (local $17 i32)
+  (local $18 i32)
+  (local $19 i32)
+  (local $20 i32)
+  (local $21 i32)
+  (local $22 i32)
+  (local $23 i32)
+  (local $24 i32)
+  (local $25 i32)
+  (local $26 i32)
+  (local $27 i32)
+  (local $28 i32)
+  (local $29 i32)
+  (local $30 i32)
+  (local $31 i32)
+  (local $32 i32)
+  (local $33 i32)
+  (local $34 i32)
+  (local $35 i32)
+  (local $36 i32)
+  (local $37 i32)
+  (local $38 i32)
+  (local $39 i32)
+  (local $40 i32)
+  (local $41 i32)
+  (local $42 i32)
+  (local $43 i32)
+  (local $44 i32)
+  (local $45 i32)
+  (local $46 i32)
+  (local $47 i32)
+  (local $48 i32)
+  (local $49 i32)
+  (local $50 i64)
+  (local $51 i64)
+  (local $52 f64)
+  (local $53 f64)
+  (block $label$1 (result i32)
+   (local.set $23
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 624)
+    )
+   )
+   (local.set $20
+    (i32.add
+     (local.get $23)
+     (i32.const 16)
+    )
+   )
+   (local.set $16
+    (local.get $23)
+   )
+   (local.set $36
+    (i32.add
+     (local.get $23)
+     (i32.const 528)
+    )
+   )
+   (local.set $30
+    (i32.ne
+     (local.get $0)
+     (i32.const 0)
+    )
+   )
+   (local.set $38
+    (local.tee $21
+     (i32.add
+      (local.tee $17
+       (i32.add
+        (local.get $23)
+        (i32.const 536)
+       )
+      )
+      (i32.const 40)
+     )
+    )
+   )
+   (local.set $39
+    (i32.add
+     (local.get $17)
+     (i32.const 39)
+    )
+   )
+   (local.set $42
+    (i32.add
+     (local.tee $37
+      (i32.add
+       (local.get $23)
+       (i32.const 8)
+      )
+     )
+     (i32.const 4)
+    )
+   )
+   (local.set $43
+    (i32.sub
+     (i32.const 0)
+     (local.tee $27
+      (local.tee $19
+       (i32.add
+        (local.get $23)
+        (i32.const 588)
+       )
+      )
+     )
+    )
+   )
+   (local.set $33
+    (i32.add
+     (local.tee $17
+      (i32.add
+       (local.get $23)
+       (i32.const 576)
+      )
+     )
+     (i32.const 12)
+    )
+   )
+   (local.set $40
+    (i32.add
+     (local.get $17)
+     (i32.const 11)
+    )
+   )
+   (local.set $44
+    (i32.sub
+     (local.tee $28
+      (local.get $33)
+     )
+     (local.get $27)
+    )
+   )
+   (local.set $45
+    (i32.sub
+     (i32.const -2)
+     (local.get $27)
+    )
+   )
+   (local.set $46
+    (i32.add
+     (local.get $28)
+     (i32.const 2)
+    )
+   )
+   (local.set $48
+    (i32.add
+     (local.tee $47
+      (i32.add
+       (local.get $23)
+       (i32.const 24)
+      )
+     )
+     (i32.const 288)
+    )
+   )
+   (local.set $41
+    (local.tee $31
+     (i32.add
+      (local.get $19)
+      (i32.const 9)
+     )
+    )
+   )
+   (local.set $34
+    (i32.add
+     (local.get $19)
+     (i32.const 8)
+    )
+   )
+   (local.set $15
+    (i32.const 0)
+   )
+   (local.set $10
+    (i32.const 0)
+   )
+   (local.set $17
+    (i32.const 0)
+   )
+   (block $label$2
+    (block $label$3
+     (loop $label$4
+      (block $label$5
+       (if
+        (i32.gt_s
+         (local.get $15)
+         (i32.const -1)
+        )
+        (local.set $15
+         (if (result i32)
+          (i32.gt_s
+           (local.get $10)
+           (i32.sub
+            (i32.const 2147483647)
+            (local.get $15)
+           )
+          )
+          (block (result i32)
+           (i32.store
+            (call $11)
+            (i32.const 75)
+           )
+           (i32.const -1)
+          )
+          (i32.add
+           (local.get $10)
+           (local.get $15)
+          )
+         )
+        )
+       )
+       (br_if $label$3
+        (i32.eqz
+         (i32.shr_s
+          (i32.shl
+           (local.tee $5
+            (i32.load8_s
+             (local.get $1)
+            )
+           )
+           (i32.const 24)
+          )
+          (i32.const 24)
+         )
+        )
+       )
+       (local.set $11
+        (local.get $1)
+       )
+       (block $label$9
+        (block $label$10
+         (loop $label$11
+          (block $label$12
+           (block $label$13
+            (block $label$14
+             (block $label$15
+              (br_table $label$14 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$13 $label$15 $label$13
+               (i32.sub
+                (i32.shr_s
+                 (i32.shl
+                  (local.get $5)
+                  (i32.const 24)
+                 )
+                 (i32.const 24)
+                )
+                (i32.const 0)
+               )
+              )
+             )
+             (local.set $5
+              (local.get $11)
+             )
+             (br $label$10)
+            )
+            (local.set $5
+             (local.get $11)
+            )
+            (br $label$12)
+           )
+           (local.set $5
+            (i32.load8_s
+             (local.tee $11
+              (i32.add
+               (local.get $11)
+               (i32.const 1)
+              )
+             )
+            )
+           )
+           (br $label$11)
+          )
+         )
+         (br $label$9)
+        )
+        (loop $label$16
+         (br_if $label$9
+          (i32.ne
+           (i32.load8_s offset=1
+            (local.get $5)
+           )
+           (i32.const 37)
+          )
+         )
+         (local.set $11
+          (i32.add
+           (local.get $11)
+           (i32.const 1)
+          )
+         )
+         (br_if $label$16
+          (i32.eq
+           (i32.load8_s
+            (local.tee $5
+             (i32.add
+              (local.get $5)
+              (i32.const 2)
+             )
+            )
+           )
+           (i32.const 37)
+          )
+         )
+        )
+       )
+       (local.set $10
+        (i32.sub
+         (local.get $11)
+         (local.get $1)
+        )
+       )
+       (if
+        (local.get $30)
+        (if
+         (i32.eqz
+          (i32.and
+           (i32.load
+            (local.get $0)
+           )
+           (i32.const 32)
+          )
+         )
+         (drop
+          (call $20
+           (local.get $1)
+           (local.get $10)
+           (local.get $0)
+          )
+         )
+        )
+       )
+       (if
+        (local.get $10)
+        (block
+         (local.set $1
+          (local.get $5)
+         )
+         (br $label$4)
+        )
+       )
+       (local.set $10
+        (if (result i32)
+         (i32.lt_u
+          (local.tee $9
+           (i32.add
+            (i32.shr_s
+             (i32.shl
+              (local.tee $10
+               (i32.load8_s
+                (local.tee $11
+                 (i32.add
+                  (local.get $5)
+                  (i32.const 1)
+                 )
+                )
+               )
+              )
+              (i32.const 24)
+             )
+             (i32.const 24)
+            )
+            (i32.const -48)
+           )
+          )
+          (i32.const 10)
+         )
+         (block (result i32)
+          (local.set $10
+           (i32.add
+            (local.get $5)
+            (i32.const 3)
+           )
+          )
+          (if
+           (local.tee $12
+            (i32.eq
+             (i32.load8_s offset=2
+              (local.get $5)
+             )
+             (i32.const 36)
+            )
+           )
+           (local.set $11
+            (local.get $10)
+           )
+          )
+          (if
+           (local.get $12)
+           (local.set $17
+            (i32.const 1)
+           )
+          )
+          (local.set $5
+           (i32.load8_s
+            (local.get $11)
+           )
+          )
+          (if
+           (i32.eqz
+            (local.get $12)
+           )
+           (local.set $9
+            (i32.const -1)
+           )
+          )
+          (local.get $17)
+         )
+         (block (result i32)
+          (local.set $5
+           (local.get $10)
+          )
+          (local.set $9
+           (i32.const -1)
+          )
+          (local.get $17)
+         )
+        )
+       )
+       (block $label$25
+        (if
+         (i32.lt_u
+          (local.tee $12
+           (i32.add
+            (i32.shr_s
+             (i32.shl
+              (local.get $5)
+              (i32.const 24)
+             )
+             (i32.const 24)
+            )
+            (i32.const -32)
+           )
+          )
+          (i32.const 32)
+         )
+         (block
+          (local.set $17
+           (i32.const 0)
+          )
+          (loop $label$27
+           (br_if $label$25
+            (i32.eqz
+             (i32.and
+              (i32.shl
+               (i32.const 1)
+               (local.get $12)
+              )
+              (i32.const 75913)
+             )
+            )
+           )
+           (local.set $17
+            (i32.or
+             (i32.shl
+              (i32.const 1)
+              (i32.add
+               (i32.shr_s
+                (i32.shl
+                 (local.get $5)
+                 (i32.const 24)
+                )
+                (i32.const 24)
+               )
+               (i32.const -32)
+              )
+             )
+             (local.get $17)
+            )
+           )
+           (br_if $label$27
+            (i32.lt_u
+             (local.tee $12
+              (i32.add
+               (i32.shr_s
+                (i32.shl
+                 (local.tee $5
+                  (i32.load8_s
+                   (local.tee $11
+                    (i32.add
+                     (local.get $11)
+                     (i32.const 1)
+                    )
+                   )
+                  )
+                 )
+                 (i32.const 24)
+                )
+                (i32.const 24)
+               )
+               (i32.const -32)
+              )
+             )
+             (i32.const 32)
+            )
+           )
+          )
+         )
+         (local.set $17
+          (i32.const 0)
+         )
+        )
+       )
+       (block $label$29
+        (if
+         (i32.eq
+          (i32.shr_s
+           (i32.shl
+            (local.get $5)
+            (i32.const 24)
+           )
+           (i32.const 24)
+          )
+          (i32.const 42)
+         )
+         (block
+          (local.set $11
+           (block $label$31 (result i32)
+            (block $label$32
+             (br_if $label$32
+              (i32.ge_u
+               (local.tee $12
+                (i32.add
+                 (i32.shr_s
+                  (i32.shl
+                   (local.tee $5
+                    (i32.load8_s
+                     (local.tee $7
+                      (i32.add
+                       (local.get $11)
+                       (i32.const 1)
+                      )
+                     )
+                    )
+                   )
+                   (i32.const 24)
+                  )
+                  (i32.const 24)
+                 )
+                 (i32.const -48)
+                )
+               )
+               (i32.const 10)
+              )
+             )
+             (br_if $label$32
+              (i32.ne
+               (i32.load8_s offset=2
+                (local.get $11)
+               )
+               (i32.const 36)
+              )
+             )
+             (i32.store
+              (i32.add
+               (local.get $4)
+               (i32.shl
+                (local.get $12)
+                (i32.const 2)
+               )
+              )
+              (i32.const 10)
+             )
+             (local.set $8
+              (i32.const 1)
+             )
+             (local.set $10
+              (i32.wrap_i64
+               (i64.load
+                (i32.add
+                 (local.get $3)
+                 (i32.shl
+                  (i32.add
+                   (i32.load8_s
+                    (local.get $7)
+                   )
+                   (i32.const -48)
+                  )
+                  (i32.const 3)
+                 )
+                )
+               )
+              )
+             )
+             (br $label$31
+              (i32.add
+               (local.get $11)
+               (i32.const 3)
+              )
+             )
+            )
+            (if
+             (local.get $10)
+             (block
+              (local.set $15
+               (i32.const -1)
+              )
+              (br $label$5)
+             )
+            )
+            (if
+             (i32.eqz
+              (local.get $30)
+             )
+             (block
+              (local.set $12
+               (local.get $17)
+              )
+              (local.set $17
+               (i32.const 0)
+              )
+              (local.set $11
+               (local.get $7)
+              )
+              (local.set $10
+               (i32.const 0)
+              )
+              (br $label$29)
+             )
+            )
+            (local.set $10
+             (i32.load
+              (local.tee $11
+               (i32.and
+                (i32.add
+                 (i32.load
+                  (local.get $2)
+                 )
+                 (i32.const 3)
+                )
+                (i32.const -4)
+               )
+              )
+             )
+            )
+            (i32.store
+             (local.get $2)
+             (i32.add
+              (local.get $11)
+              (i32.const 4)
+             )
+            )
+            (local.set $8
+             (i32.const 0)
+            )
+            (local.get $7)
+           )
+          )
+          (local.set $12
+           (i32.or
+            (local.get $17)
+            (i32.const 8192)
+           )
+          )
+          (local.set $7
+           (i32.sub
+            (i32.const 0)
+            (local.get $10)
+           )
+          )
+          (local.set $5
+           (i32.load8_s
+            (local.get $11)
+           )
+          )
+          (if
+           (i32.eqz
+            (local.tee $6
+             (i32.lt_s
+              (local.get $10)
+              (i32.const 0)
+             )
+            )
+           )
+           (local.set $12
+            (local.get $17)
+           )
+          )
+          (local.set $17
+           (local.get $8)
+          )
+          (if
+           (local.get $6)
+           (local.set $10
+            (local.get $7)
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.tee $12
+            (i32.add
+             (i32.shr_s
+              (i32.shl
+               (local.get $5)
+               (i32.const 24)
+              )
+              (i32.const 24)
+             )
+             (i32.const -48)
+            )
+           )
+           (i32.const 10)
+          )
+          (block
+           (local.set $7
+            (i32.const 0)
+           )
+           (local.set $5
+            (local.get $12)
+           )
+           (loop $label$39
+            (local.set $7
+             (i32.add
+              (i32.mul
+               (local.get $7)
+               (i32.const 10)
+              )
+              (local.get $5)
+             )
+            )
+            (br_if $label$39
+             (i32.lt_u
+              (local.tee $5
+               (i32.add
+                (i32.shr_s
+                 (i32.shl
+                  (local.tee $12
+                   (i32.load8_s
+                    (local.tee $11
+                     (i32.add
+                      (local.get $11)
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                  )
+                  (i32.const 24)
+                 )
+                 (i32.const 24)
+                )
+                (i32.const -48)
+               )
+              )
+              (i32.const 10)
+             )
+            )
+           )
+           (if
+            (i32.lt_s
+             (local.get $7)
+             (i32.const 0)
+            )
+            (block
+             (local.set $15
+              (i32.const -1)
+             )
+             (br $label$5)
+            )
+            (block
+             (local.set $5
+              (local.get $12)
+             )
+             (local.set $12
+              (local.get $17)
+             )
+             (local.set $17
+              (local.get $10)
+             )
+             (local.set $10
+              (local.get $7)
+             )
+            )
+           )
+          )
+          (block
+           (local.set $12
+            (local.get $17)
+           )
+           (local.set $17
+            (local.get $10)
+           )
+           (local.set $10
+            (i32.const 0)
+           )
+          )
+         )
+        )
+       )
+       (block $label$43
+        (if
+         (i32.eq
+          (i32.shr_s
+           (i32.shl
+            (local.get $5)
+            (i32.const 24)
+           )
+           (i32.const 24)
+          )
+          (i32.const 46)
+         )
+         (block
+          (if
+           (i32.ne
+            (i32.shr_s
+             (i32.shl
+              (local.tee $5
+               (i32.load8_s
+                (local.tee $7
+                 (i32.add
+                  (local.get $11)
+                  (i32.const 1)
+                 )
+                )
+               )
+              )
+              (i32.const 24)
+             )
+             (i32.const 24)
+            )
+            (i32.const 42)
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.tee $5
+               (i32.add
+                (i32.shr_s
+                 (i32.shl
+                  (local.get $5)
+                  (i32.const 24)
+                 )
+                 (i32.const 24)
+                )
+                (i32.const -48)
+               )
+              )
+              (i32.const 10)
+             )
+             (block
+              (local.set $11
+               (local.get $7)
+              )
+              (local.set $7
+               (i32.const 0)
+              )
+             )
+             (block
+              (local.set $5
+               (i32.const 0)
+              )
+              (local.set $11
+               (local.get $7)
+              )
+              (br $label$43)
+             )
+            )
+            (loop $label$48
+             (local.set $5
+              (i32.add
+               (i32.mul
+                (local.get $7)
+                (i32.const 10)
+               )
+               (local.get $5)
+              )
+             )
+             (br_if $label$43
+              (i32.ge_u
+               (local.tee $8
+                (i32.add
+                 (i32.load8_s
+                  (local.tee $11
+                   (i32.add
+                    (local.get $11)
+                    (i32.const 1)
+                   )
+                  )
+                 )
+                 (i32.const -48)
+                )
+               )
+               (i32.const 10)
+              )
+             )
+             (local.set $7
+              (local.get $5)
+             )
+             (local.set $5
+              (local.get $8)
+             )
+             (br $label$48)
+            )
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.tee $5
+             (i32.add
+              (i32.load8_s
+               (local.tee $7
+                (i32.add
+                 (local.get $11)
+                 (i32.const 2)
+                )
+               )
+              )
+              (i32.const -48)
+             )
+            )
+            (i32.const 10)
+           )
+           (if
+            (i32.eq
+             (i32.load8_s offset=3
+              (local.get $11)
+             )
+             (i32.const 36)
+            )
+            (block
+             (i32.store
+              (i32.add
+               (local.get $4)
+               (i32.shl
+                (local.get $5)
+                (i32.const 2)
+               )
+              )
+              (i32.const 10)
+             )
+             (local.set $5
+              (i32.wrap_i64
+               (i64.load
+                (i32.add
+                 (local.get $3)
+                 (i32.shl
+                  (i32.add
+                   (i32.load8_s
+                    (local.get $7)
+                   )
+                   (i32.const -48)
+                  )
+                  (i32.const 3)
+                 )
+                )
+               )
+              )
+             )
+             (local.set $11
+              (i32.add
+               (local.get $11)
+               (i32.const 4)
+              )
+             )
+             (br $label$43)
+            )
+           )
+          )
+          (if
+           (local.get $17)
+           (block
+            (local.set $15
+             (i32.const -1)
+            )
+            (br $label$5)
+           )
+          )
+          (local.set $11
+           (if (result i32)
+            (local.get $30)
+            (block (result i32)
+             (local.set $5
+              (i32.load
+               (local.tee $11
+                (i32.and
+                 (i32.add
+                  (i32.load
+                   (local.get $2)
+                  )
+                  (i32.const 3)
+                 )
+                 (i32.const -4)
+                )
+               )
+              )
+             )
+             (i32.store
+              (local.get $2)
+              (i32.add
+               (local.get $11)
+               (i32.const 4)
+              )
+             )
+             (local.get $7)
+            )
+            (block (result i32)
+             (local.set $5
+              (i32.const 0)
+             )
+             (local.get $7)
+            )
+           )
+          )
+         )
+         (local.set $5
+          (i32.const -1)
+         )
+        )
+       )
+       (local.set $7
+        (local.get $11)
+       )
+       (local.set $8
+        (i32.const 0)
+       )
+       (loop $label$55
+        (if
+         (i32.gt_u
+          (local.tee $6
+           (i32.add
+            (i32.load8_s
+             (local.get $7)
+            )
+            (i32.const -65)
+           )
+          )
+          (i32.const 57)
+         )
+         (block
+          (local.set $15
+           (i32.const -1)
+          )
+          (br $label$5)
+         )
+        )
+        (local.set $11
+         (i32.add
+          (local.get $7)
+          (i32.const 1)
+         )
+        )
+        (if
+         (i32.lt_u
+          (i32.add
+           (local.tee $6
+            (i32.and
+             (local.tee $13
+              (i32.load8_s
+               (i32.add
+                (i32.add
+                 (i32.mul
+                  (local.get $8)
+                  (i32.const 58)
+                 )
+                 (i32.const 1168)
+                )
+                (local.get $6)
+               )
+              )
+             )
+             (i32.const 255)
+            )
+           )
+           (i32.const -1)
+          )
+          (i32.const 8)
+         )
+         (block
+          (local.set $7
+           (local.get $11)
+          )
+          (local.set $8
+           (local.get $6)
+          )
+          (br $label$55)
+         )
+        )
+       )
+       (if
+        (i32.eqz
+         (i32.shr_s
+          (i32.shl
+           (local.get $13)
+           (i32.const 24)
+          )
+          (i32.const 24)
+         )
+        )
+        (block
+         (local.set $15
+          (i32.const -1)
+         )
+         (br $label$5)
+        )
+       )
+       (local.set $14
+        (i32.gt_s
+         (local.get $9)
+         (i32.const -1)
+        )
+       )
+       (block $label$59
+        (block $label$60
+         (if
+          (i32.eq
+           (i32.shr_s
+            (i32.shl
+             (local.get $13)
+             (i32.const 24)
+            )
+            (i32.const 24)
+           )
+           (i32.const 19)
+          )
+          (if
+           (local.get $14)
+           (block
+            (local.set $15
+             (i32.const -1)
+            )
+            (br $label$5)
+           )
+           (br $label$60)
+          )
+          (block
+           (if
+            (local.get $14)
+            (block
+             (i32.store
+              (i32.add
+               (local.get $4)
+               (i32.shl
+                (local.get $9)
+                (i32.const 2)
+               )
+              )
+              (local.get $6)
+             )
+             (i64.store
+              (local.get $16)
+              (i64.load
+               (i32.add
+                (local.get $3)
+                (i32.shl
+                 (local.get $9)
+                 (i32.const 3)
+                )
+               )
+              )
+             )
+             (br $label$60)
+            )
+           )
+           (if
+            (i32.eqz
+             (local.get $30)
+            )
+            (block
+             (local.set $15
+              (i32.const 0)
+             )
+             (br $label$5)
+            )
+           )
+           (call $21
+            (local.get $16)
+            (local.get $6)
+            (local.get $2)
+           )
+          )
+         )
+         (br $label$59)
+        )
+        (if
+         (i32.eqz
+          (local.get $30)
+         )
+         (block
+          (local.set $10
+           (i32.const 0)
+          )
+          (local.set $1
+           (local.get $11)
+          )
+          (br $label$4)
+         )
+        )
+       )
+       (local.set $9
+        (i32.and
+         (local.tee $7
+          (i32.load8_s
+           (local.get $7)
+          )
+         )
+         (i32.const -33)
+        )
+       )
+       (if
+        (i32.eqz
+         (i32.and
+          (i32.ne
+           (local.get $8)
+           (i32.const 0)
+          )
+          (i32.eq
+           (i32.and
+            (local.get $7)
+            (i32.const 15)
+           )
+           (i32.const 3)
+          )
+         )
+        )
+        (local.set $9
+         (local.get $7)
+        )
+       )
+       (local.set $7
+        (i32.and
+         (local.get $12)
+         (i32.const -65537)
+        )
+       )
+       (if
+        (i32.and
+         (local.get $12)
+         (i32.const 8192)
+        )
+        (local.set $12
+         (local.get $7)
+        )
+       )
+       (block $label$70
+        (block $label$71
+         (block $label$72
+          (block $label$73
+           (block $label$74
+            (block $label$75
+             (block $label$76
+              (block $label$77
+               (block $label$78
+                (block $label$79
+                 (block $label$80
+                  (block $label$81
+                   (block $label$82
+                    (block $label$83
+                     (block $label$84
+                      (block $label$85
+                       (block $label$86
+                        (block $label$87
+                         (block $label$88
+                          (block $label$89
+                           (br_table $label$78 $label$77 $label$80 $label$77 $label$78 $label$78 $label$78 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$79 $label$77 $label$77 $label$77 $label$77 $label$87 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$77 $label$78 $label$77 $label$83 $label$85 $label$78 $label$78 $label$78 $label$77 $label$85 $label$77 $label$77 $label$77 $label$82 $label$89 $label$86 $label$88 $label$77 $label$77 $label$81 $label$77 $label$84 $label$77 $label$77 $label$87 $label$77
+                            (i32.sub
+                             (local.get $9)
+                             (i32.const 65)
+                            )
+                           )
+                          )
+                          (block $label$90
+                           (block $label$91
+                            (block $label$92
+                             (block $label$93
+                              (block $label$94
+                               (block $label$95
+                                (block $label$96
+                                 (block $label$97
+                                  (br_table $label$97 $label$96 $label$95 $label$94 $label$93 $label$90 $label$92 $label$91 $label$90
+                                   (i32.sub
+                                    (i32.shr_s
+                                     (i32.shl
+                                      (i32.and
+                                       (local.get $8)
+                                       (i32.const 255)
+                                      )
+                                      (i32.const 24)
+                                     )
+                                     (i32.const 24)
+                                    )
+                                    (i32.const 0)
+                                   )
+                                  )
+                                 )
+                                 (i32.store
+                                  (i32.load
+                                   (local.get $16)
+                                  )
+                                  (local.get $15)
+                                 )
+                                 (local.set $10
+                                  (i32.const 0)
+                                 )
+                                 (local.set $1
+                                  (local.get $11)
+                                 )
+                                 (br $label$4)
+                                )
+                                (i32.store
+                                 (i32.load
+                                  (local.get $16)
+                                 )
+                                 (local.get $15)
+                                )
+                                (local.set $10
+                                 (i32.const 0)
+                                )
+                                (local.set $1
+                                 (local.get $11)
+                                )
+                                (br $label$4)
+                               )
+                               (i64.store
+                                (i32.load
+                                 (local.get $16)
+                                )
+                                (i64.extend_i32_s
+                                 (local.get $15)
+                                )
+                               )
+                               (local.set $10
+                                (i32.const 0)
+                               )
+                               (local.set $1
+                                (local.get $11)
+                               )
+                               (br $label$4)
+                              )
+                              (i32.store16
+                               (i32.load
+                                (local.get $16)
+                               )
+                               (local.get $15)
+                              )
+                              (local.set $10
+                               (i32.const 0)
+                              )
+                              (local.set $1
+                               (local.get $11)
+                              )
+                              (br $label$4)
+                             )
+                             (i32.store8
+                              (i32.load
+                               (local.get $16)
+                              )
+                              (local.get $15)
+                             )
+                             (local.set $10
+                              (i32.const 0)
+                             )
+                             (local.set $1
+                              (local.get $11)
+                             )
+                             (br $label$4)
+                            )
+                            (i32.store
+                             (i32.load
+                              (local.get $16)
+                             )
+                             (local.get $15)
+                            )
+                            (local.set $10
+                             (i32.const 0)
+                            )
+                            (local.set $1
+                             (local.get $11)
+                            )
+                            (br $label$4)
+                           )
+                           (i64.store
+                            (i32.load
+                             (local.get $16)
+                            )
+                            (i64.extend_i32_s
+                             (local.get $15)
+                            )
+                           )
+                           (local.set $10
+                            (i32.const 0)
+                           )
+                           (local.set $1
+                            (local.get $11)
+                           )
+                           (br $label$4)
+                          )
+                          (local.set $10
+                           (i32.const 0)
+                          )
+                          (local.set $1
+                           (local.get $11)
+                          )
+                          (br $label$4)
+                         )
+                         (local.set $12
+                          (i32.or
+                           (local.get $12)
+                           (i32.const 8)
+                          )
+                         )
+                         (if
+                          (i32.le_u
+                           (local.get $5)
+                           (i32.const 8)
+                          )
+                          (local.set $5
+                           (i32.const 8)
+                          )
+                         )
+                         (local.set $9
+                          (i32.const 120)
+                         )
+                         (br $label$76)
+                        )
+                        (br $label$76)
+                       )
+                       (if
+                        (i64.eq
+                         (local.tee $50
+                          (i64.load
+                           (local.get $16)
+                          )
+                         )
+                         (i64.const 0)
+                        )
+                        (local.set $7
+                         (local.get $21)
+                        )
+                        (block
+                         (local.set $1
+                          (local.get $21)
+                         )
+                         (loop $label$101
+                          (i64.store8
+                           (local.tee $1
+                            (i32.add
+                             (local.get $1)
+                             (i32.const -1)
+                            )
+                           )
+                           (i64.or
+                            (i64.and
+                             (local.get $50)
+                             (i64.const 7)
+                            )
+                            (i64.const 48)
+                           )
+                          )
+                          (br_if $label$101
+                           (i64.ne
+                            (local.tee $50
+                             (i64.shr_u
+                              (local.get $50)
+                              (i64.const 3)
+                             )
+                            )
+                            (i64.const 0)
+                           )
+                          )
+                          (local.set $7
+                           (local.get $1)
+                          )
+                         )
+                        )
+                       )
+                       (if
+                        (i32.and
+                         (local.get $12)
+                         (i32.const 8)
+                        )
+                        (block
+                         (local.set $8
+                          (i32.add
+                           (local.tee $1
+                            (i32.sub
+                             (local.get $38)
+                             (local.get $7)
+                            )
+                           )
+                           (i32.const 1)
+                          )
+                         )
+                         (if
+                          (i32.le_s
+                           (local.get $5)
+                           (local.get $1)
+                          )
+                          (local.set $5
+                           (local.get $8)
+                          )
+                         )
+                         (local.set $6
+                          (i32.const 0)
+                         )
+                         (local.set $8
+                          (i32.const 1648)
+                         )
+                         (br $label$71)
+                        )
+                        (block
+                         (local.set $6
+                          (i32.const 0)
+                         )
+                         (local.set $8
+                          (i32.const 1648)
+                         )
+                         (br $label$71)
+                        )
+                       )
+                      )
+                      (if
+                       (i64.lt_s
+                        (local.tee $50
+                         (i64.load
+                          (local.get $16)
+                         )
+                        )
+                        (i64.const 0)
+                       )
+                       (block
+                        (i64.store
+                         (local.get $16)
+                         (local.tee $50
+                          (i64.sub
+                           (i64.const 0)
+                           (local.get $50)
+                          )
+                         )
+                        )
+                        (local.set $6
+                         (i32.const 1)
+                        )
+                        (local.set $8
+                         (i32.const 1648)
+                        )
+                        (br $label$75)
+                       )
+                      )
+                      (if
+                       (i32.and
+                        (local.get $12)
+                        (i32.const 2048)
+                       )
+                       (block
+                        (local.set $6
+                         (i32.const 1)
+                        )
+                        (local.set $8
+                         (i32.const 1649)
+                        )
+                        (br $label$75)
+                       )
+                       (block
+                        (local.set $6
+                         (local.tee $1
+                          (i32.and
+                           (local.get $12)
+                           (i32.const 1)
+                          )
+                         )
+                        )
+                        (local.set $8
+                         (if (result i32)
+                          (local.get $1)
+                          (i32.const 1650)
+                          (i32.const 1648)
+                         )
+                        )
+                        (br $label$75)
+                       )
+                      )
+                     )
+                     (local.set $50
+                      (i64.load
+                       (local.get $16)
+                      )
+                     )
+                     (local.set $6
+                      (i32.const 0)
+                     )
+                     (local.set $8
+                      (i32.const 1648)
+                     )
+                     (br $label$75)
+                    )
+                    (i64.store8
+                     (local.get $39)
+                     (i64.load
+                      (local.get $16)
+                     )
+                    )
+                    (local.set $1
+                     (local.get $39)
+                    )
+                    (local.set $12
+                     (local.get $7)
+                    )
+                    (local.set $7
+                     (i32.const 1)
+                    )
+                    (local.set $6
+                     (i32.const 0)
+                    )
+                    (local.set $8
+                     (i32.const 1648)
+                    )
+                    (local.set $5
+                     (local.get $21)
+                    )
+                    (br $label$70)
+                   )
+                   (local.set $1
+                    (call $23
+                     (i32.load
+                      (call $11)
+                     )
+                    )
+                   )
+                   (br $label$74)
+                  )
+                  (if
+                   (i32.eqz
+                    (local.tee $1
+                     (i32.load
+                      (local.get $16)
+                     )
+                    )
+                   )
+                   (local.set $1
+                    (i32.const 1658)
+                   )
+                  )
+                  (br $label$74)
+                 )
+                 (i64.store32
+                  (local.get $37)
+                  (i64.load
+                   (local.get $16)
+                  )
+                 )
+                 (i32.store
+                  (local.get $42)
+                  (i32.const 0)
+                 )
+                 (i32.store
+                  (local.get $16)
+                  (local.get $37)
+                 )
+                 (local.set $7
+                  (local.get $37)
+                 )
+                 (local.set $6
+                  (i32.const -1)
+                 )
+                 (br $label$73)
+                )
+                (local.set $7
+                 (i32.load
+                  (local.get $16)
+                 )
+                )
+                (if
+                 (local.get $5)
+                 (block
+                  (local.set $6
+                   (local.get $5)
+                  )
+                  (br $label$73)
+                 )
+                 (block
+                  (call $24
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (i32.const 0)
+                   (local.get $12)
+                  )
+                  (local.set $1
+                   (i32.const 0)
+                  )
+                  (br $label$72)
+                 )
+                )
+               )
+               (local.set $52
+                (f64.load
+                 (local.get $16)
+                )
+               )
+               (i32.store
+                (local.get $20)
+                (i32.const 0)
+               )
+               (local.set $26
+                (if (result i32)
+                 (i64.lt_s
+                  (i64.reinterpret_f64
+                   (local.get $52)
+                  )
+                  (i64.const 0)
+                 )
+                 (block (result i32)
+                  (local.set $24
+                   (i32.const 1)
+                  )
+                  (local.set $52
+                   (f64.neg
+                    (local.get $52)
+                   )
+                  )
+                  (i32.const 1665)
+                 )
+                 (block (result i32)
+                  (local.set $1
+                   (i32.and
+                    (local.get $12)
+                    (i32.const 1)
+                   )
+                  )
+                  (if (result i32)
+                   (i32.and
+                    (local.get $12)
+                    (i32.const 2048)
+                   )
+                   (block (result i32)
+                    (local.set $24
+                     (i32.const 1)
+                    )
+                    (i32.const 1668)
+                   )
+                   (block (result i32)
+                    (local.set $24
+                     (local.get $1)
+                    )
+                    (if (result i32)
+                     (local.get $1)
+                     (i32.const 1671)
+                     (i32.const 1666)
+                    )
+                   )
+                  )
+                 )
+                )
+               )
+               (block $label$119
+                (if
+                 (i64.lt_u
+                  (i64.and
+                   (i64.reinterpret_f64
+                    (local.get $52)
+                   )
+                   (i64.const 9218868437227405312)
+                  )
+                  (i64.const 9218868437227405312)
+                 )
+                 (block
+                  (if
+                   (local.tee $1
+                    (f64.ne
+                     (local.tee $52
+                      (f64.mul
+                       (call $26
+                        (local.get $52)
+                        (local.get $20)
+                       )
+                       (f64.const 2)
+                      )
+                     )
+                     (f64.const 0)
+                    )
+                   )
+                   (i32.store
+                    (local.get $20)
+                    (i32.add
+                     (i32.load
+                      (local.get $20)
+                     )
+                     (i32.const -1)
+                    )
+                   )
+                  )
+                  (if
+                   (i32.eq
+                    (local.tee $22
+                     (i32.or
+                      (local.get $9)
+                      (i32.const 32)
+                     )
+                    )
+                    (i32.const 97)
+                   )
+                   (block
+                    (local.set $1
+                     (i32.add
+                      (local.get $26)
+                      (i32.const 9)
+                     )
+                    )
+                    (if
+                     (local.tee $6
+                      (i32.and
+                       (local.get $9)
+                       (i32.const 32)
+                      )
+                     )
+                     (local.set $26
+                      (local.get $1)
+                     )
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.or
+                       (i32.gt_u
+                        (local.get $5)
+                        (i32.const 11)
+                       )
+                       (i32.eqz
+                        (local.tee $1
+                         (i32.sub
+                          (i32.const 12)
+                          (local.get $5)
+                         )
+                        )
+                       )
+                      )
+                     )
+                     (block
+                      (local.set $53
+                       (f64.const 8)
+                      )
+                      (loop $label$125
+                       (local.set $53
+                        (f64.mul
+                         (local.get $53)
+                         (f64.const 16)
+                        )
+                       )
+                       (br_if $label$125
+                        (local.tee $1
+                         (i32.add
+                          (local.get $1)
+                          (i32.const -1)
+                         )
+                        )
+                       )
+                      )
+                      (local.set $52
+                       (if (result f64)
+                        (i32.eq
+                         (i32.load8_s
+                          (local.get $26)
+                         )
+                         (i32.const 45)
+                        )
+                        (f64.neg
+                         (f64.add
+                          (local.get $53)
+                          (f64.sub
+                           (f64.neg
+                            (local.get $52)
+                           )
+                           (local.get $53)
+                          )
+                         )
+                        )
+                        (f64.sub
+                         (f64.add
+                          (local.get $52)
+                          (local.get $53)
+                         )
+                         (local.get $53)
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (local.set $1
+                     (i32.sub
+                      (i32.const 0)
+                      (local.tee $7
+                       (i32.load
+                        (local.get $20)
+                       )
+                      )
+                     )
+                    )
+                    (if
+                     (i32.eq
+                      (local.tee $1
+                       (call $22
+                        (i64.extend_i32_s
+                         (if (result i32)
+                          (i32.lt_s
+                           (local.get $7)
+                           (i32.const 0)
+                          )
+                          (local.get $1)
+                          (local.get $7)
+                         )
+                        )
+                        (local.get $33)
+                       )
+                      )
+                      (local.get $33)
+                     )
+                     (block
+                      (i32.store8
+                       (local.get $40)
+                       (i32.const 48)
+                      )
+                      (local.set $1
+                       (local.get $40)
+                      )
+                     )
+                    )
+                    (local.set $13
+                     (i32.or
+                      (local.get $24)
+                      (i32.const 2)
+                     )
+                    )
+                    (i32.store8
+                     (i32.add
+                      (local.get $1)
+                      (i32.const -1)
+                     )
+                     (i32.add
+                      (i32.and
+                       (i32.shr_s
+                        (local.get $7)
+                        (i32.const 31)
+                       )
+                       (i32.const 2)
+                      )
+                      (i32.const 43)
+                     )
+                    )
+                    (i32.store8
+                     (local.tee $8
+                      (i32.add
+                       (local.get $1)
+                       (i32.const -2)
+                      )
+                     )
+                     (i32.add
+                      (local.get $9)
+                      (i32.const 15)
+                     )
+                    )
+                    (local.set $9
+                     (i32.lt_s
+                      (local.get $5)
+                      (i32.const 1)
+                     )
+                    )
+                    (local.set $14
+                     (i32.eqz
+                      (i32.and
+                       (local.get $12)
+                       (i32.const 8)
+                      )
+                     )
+                    )
+                    (local.set $1
+                     (local.get $19)
+                    )
+                    (loop $label$131
+                     (i32.store8
+                      (local.get $1)
+                      (i32.or
+                       (i32.load8_u
+                        (i32.add
+                         (local.tee $7
+                          (i32.trunc_f64_s
+                           (local.get $52)
+                          )
+                         )
+                         (i32.const 1632)
+                        )
+                       )
+                       (local.get $6)
+                      )
+                     )
+                     (local.set $52
+                      (f64.mul
+                       (f64.sub
+                        (local.get $52)
+                        (f64.convert_i32_s
+                         (local.get $7)
+                        )
+                       )
+                       (f64.const 16)
+                      )
+                     )
+                     (local.set $1
+                      (block $label$132 (result i32)
+                       (if (result i32)
+                        (i32.eq
+                         (i32.sub
+                          (local.tee $7
+                           (i32.add
+                            (local.get $1)
+                            (i32.const 1)
+                           )
+                          )
+                          (local.get $27)
+                         )
+                         (i32.const 1)
+                        )
+                        (block (result i32)
+                         (drop
+                          (br_if $label$132
+                           (local.get $7)
+                           (i32.and
+                            (local.get $14)
+                            (i32.and
+                             (local.get $9)
+                             (f64.eq
+                              (local.get $52)
+                              (f64.const 0)
+                             )
+                            )
+                           )
+                          )
+                         )
+                         (i32.store8
+                          (local.get $7)
+                          (i32.const 46)
+                         )
+                         (i32.add
+                          (local.get $1)
+                          (i32.const 2)
+                         )
+                        )
+                        (local.get $7)
+                       )
+                      )
+                     )
+                     (br_if $label$131
+                      (f64.ne
+                       (local.get $52)
+                       (f64.const 0)
+                      )
+                     )
+                    )
+                    (local.set $6
+                     (i32.sub
+                      (i32.add
+                       (local.get $46)
+                       (local.get $5)
+                      )
+                      (local.tee $7
+                       (local.get $8)
+                      )
+                     )
+                    )
+                    (local.set $9
+                     (i32.add
+                      (i32.sub
+                       (local.get $44)
+                       (local.get $7)
+                      )
+                      (local.get $1)
+                     )
+                    )
+                    (call $24
+                     (local.get $0)
+                     (i32.const 32)
+                     (local.get $10)
+                     (local.tee $5
+                      (i32.add
+                       (if (result i32)
+                        (i32.and
+                         (i32.ne
+                          (local.get $5)
+                          (i32.const 0)
+                         )
+                         (i32.lt_s
+                          (i32.add
+                           (local.get $45)
+                           (local.get $1)
+                          )
+                          (local.get $5)
+                         )
+                        )
+                        (local.get $6)
+                        (local.tee $6
+                         (local.get $9)
+                        )
+                       )
+                       (local.get $13)
+                      )
+                     )
+                     (local.get $12)
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $20
+                       (local.get $26)
+                       (local.get $13)
+                       (local.get $0)
+                      )
+                     )
+                    )
+                    (call $24
+                     (local.get $0)
+                     (i32.const 48)
+                     (local.get $10)
+                     (local.get $5)
+                     (i32.xor
+                      (local.get $12)
+                      (i32.const 65536)
+                     )
+                    )
+                    (local.set $1
+                     (i32.sub
+                      (local.get $1)
+                      (local.get $27)
+                     )
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $20
+                       (local.get $19)
+                       (local.get $1)
+                       (local.get $0)
+                      )
+                     )
+                    )
+                    (call $24
+                     (local.get $0)
+                     (i32.const 48)
+                     (i32.sub
+                      (local.get $6)
+                      (i32.add
+                       (local.get $1)
+                       (local.tee $1
+                        (i32.sub
+                         (local.get $28)
+                         (local.get $7)
+                        )
+                       )
+                      )
+                     )
+                     (i32.const 0)
+                     (i32.const 0)
+                    )
+                    (if
+                     (i32.eqz
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $20
+                       (local.get $8)
+                       (local.get $1)
+                       (local.get $0)
+                      )
+                     )
+                    )
+                    (call $24
+                     (local.get $0)
+                     (i32.const 32)
+                     (local.get $10)
+                     (local.get $5)
+                     (i32.xor
+                      (local.get $12)
+                      (i32.const 8192)
+                     )
+                    )
+                    (if
+                     (i32.ge_s
+                      (local.get $5)
+                      (local.get $10)
+                     )
+                     (local.set $10
+                      (local.get $5)
+                     )
+                    )
+                    (br $label$119)
+                   )
+                  )
+                  (if
+                   (local.get $1)
+                   (block
+                    (i32.store
+                     (local.get $20)
+                     (local.tee $6
+                      (i32.add
+                       (i32.load
+                        (local.get $20)
+                       )
+                       (i32.const -28)
+                      )
+                     )
+                    )
+                    (local.set $52
+                     (f64.mul
+                      (local.get $52)
+                      (f64.const 268435456)
+                     )
+                    )
+                   )
+                   (local.set $6
+                    (i32.load
+                     (local.get $20)
+                    )
+                   )
+                  )
+                  (local.set $8
+                   (local.tee $7
+                    (if (result i32)
+                     (i32.lt_s
+                      (local.get $6)
+                      (i32.const 0)
+                     )
+                     (local.get $47)
+                     (local.get $48)
+                    )
+                   )
+                  )
+                  (loop $label$145
+                   (i32.store
+                    (local.get $8)
+                    (local.tee $1
+                     (i32.trunc_f64_s
+                      (local.get $52)
+                     )
+                    )
+                   )
+                   (local.set $8
+                    (i32.add
+                     (local.get $8)
+                     (i32.const 4)
+                    )
+                   )
+                   (br_if $label$145
+                    (f64.ne
+                     (local.tee $52
+                      (f64.mul
+                       (f64.sub
+                        (local.get $52)
+                        (f64.convert_i32_u
+                         (local.get $1)
+                        )
+                       )
+                       (f64.const 1e9)
+                      )
+                     )
+                     (f64.const 0)
+                    )
+                   )
+                  )
+                  (if
+                   (i32.gt_s
+                    (local.get $6)
+                    (i32.const 0)
+                   )
+                   (block
+                    (local.set $1
+                     (local.get $7)
+                    )
+                    (loop $label$147
+                     (local.set $14
+                      (if (result i32)
+                       (i32.gt_s
+                        (local.get $6)
+                        (i32.const 29)
+                       )
+                       (i32.const 29)
+                       (local.get $6)
+                      )
+                     )
+                     (block $label$150
+                      (if
+                       (i32.ge_u
+                        (local.tee $6
+                         (i32.add
+                          (local.get $8)
+                          (i32.const -4)
+                         )
+                        )
+                        (local.get $1)
+                       )
+                       (block
+                        (local.set $50
+                         (i64.extend_i32_u
+                          (local.get $14)
+                         )
+                        )
+                        (local.set $13
+                         (i32.const 0)
+                        )
+                        (loop $label$152
+                         (i64.store32
+                          (local.get $6)
+                          (i64.rem_u
+                           (local.tee $51
+                            (i64.add
+                             (i64.shl
+                              (i64.extend_i32_u
+                               (i32.load
+                                (local.get $6)
+                               )
+                              )
+                              (local.get $50)
+                             )
+                             (i64.extend_i32_u
+                              (local.get $13)
+                             )
+                            )
+                           )
+                           (i64.const 1000000000)
+                          )
+                         )
+                         (local.set $13
+                          (i32.wrap_i64
+                           (i64.div_u
+                            (local.get $51)
+                            (i64.const 1000000000)
+                           )
+                          )
+                         )
+                         (br_if $label$152
+                          (i32.ge_u
+                           (local.tee $6
+                            (i32.add
+                             (local.get $6)
+                             (i32.const -4)
+                            )
+                           )
+                           (local.get $1)
+                          )
+                         )
+                        )
+                        (br_if $label$150
+                         (i32.eqz
+                          (local.get $13)
+                         )
+                        )
+                        (i32.store
+                         (local.tee $1
+                          (i32.add
+                           (local.get $1)
+                           (i32.const -4)
+                          )
+                         )
+                         (local.get $13)
+                        )
+                       )
+                      )
+                     )
+                     (loop $label$153
+                      (if
+                       (i32.gt_u
+                        (local.get $8)
+                        (local.get $1)
+                       )
+                       (if
+                        (i32.eqz
+                         (i32.load
+                          (local.tee $6
+                           (i32.add
+                            (local.get $8)
+                            (i32.const -4)
+                           )
+                          )
+                         )
+                        )
+                        (block
+                         (local.set $8
+                          (local.get $6)
+                         )
+                         (br $label$153)
+                        )
+                       )
+                      )
+                     )
+                     (i32.store
+                      (local.get $20)
+                      (local.tee $6
+                       (i32.sub
+                        (i32.load
+                         (local.get $20)
+                        )
+                        (local.get $14)
+                       )
+                      )
+                     )
+                     (br_if $label$147
+                      (i32.gt_s
+                       (local.get $6)
+                       (i32.const 0)
+                      )
+                     )
+                    )
+                   )
+                   (local.set $1
+                    (local.get $7)
+                   )
+                  )
+                  (local.set $18
+                   (if (result i32)
+                    (i32.lt_s
+                     (local.get $5)
+                     (i32.const 0)
+                    )
+                    (i32.const 6)
+                    (local.get $5)
+                   )
+                  )
+                  (if
+                   (i32.lt_s
+                    (local.get $6)
+                    (i32.const 0)
+                   )
+                   (block
+                    (local.set $14
+                     (i32.add
+                      (i32.div_s
+                       (i32.add
+                        (local.get $18)
+                        (i32.const 25)
+                       )
+                       (i32.const 9)
+                      )
+                      (i32.const 1)
+                     )
+                    )
+                    (local.set $25
+                     (i32.eq
+                      (local.get $22)
+                      (i32.const 102)
+                     )
+                    )
+                    (local.set $5
+                     (local.get $8)
+                    )
+                    (loop $label$160
+                     (if
+                      (i32.gt_s
+                       (local.tee $13
+                        (i32.sub
+                         (i32.const 0)
+                         (local.get $6)
+                        )
+                       )
+                       (i32.const 9)
+                      )
+                      (local.set $13
+                       (i32.const 9)
+                      )
+                     )
+                     (block $label$162
+                      (if
+                       (i32.lt_u
+                        (local.get $1)
+                        (local.get $5)
+                       )
+                       (block
+                        (local.set $29
+                         (i32.add
+                          (i32.shl
+                           (i32.const 1)
+                           (local.get $13)
+                          )
+                          (i32.const -1)
+                         )
+                        )
+                        (local.set $35
+                         (i32.shr_u
+                          (i32.const 1000000000)
+                          (local.get $13)
+                         )
+                        )
+                        (local.set $6
+                         (i32.const 0)
+                        )
+                        (local.set $8
+                         (local.get $1)
+                        )
+                        (loop $label$164
+                         (i32.store
+                          (local.get $8)
+                          (i32.add
+                           (i32.shr_u
+                            (local.tee $32
+                             (i32.load
+                              (local.get $8)
+                             )
+                            )
+                            (local.get $13)
+                           )
+                           (local.get $6)
+                          )
+                         )
+                         (local.set $6
+                          (i32.mul
+                           (i32.and
+                            (local.get $32)
+                            (local.get $29)
+                           )
+                           (local.get $35)
+                          )
+                         )
+                         (br_if $label$164
+                          (i32.lt_u
+                           (local.tee $8
+                            (i32.add
+                             (local.get $8)
+                             (i32.const 4)
+                            )
+                           )
+                           (local.get $5)
+                          )
+                         )
+                        )
+                        (local.set $8
+                         (i32.add
+                          (local.get $1)
+                          (i32.const 4)
+                         )
+                        )
+                        (if
+                         (i32.eqz
+                          (i32.load
+                           (local.get $1)
+                          )
+                         )
+                         (local.set $1
+                          (local.get $8)
+                         )
+                        )
+                        (br_if $label$162
+                         (i32.eqz
+                          (local.get $6)
+                         )
+                        )
+                        (i32.store
+                         (local.get $5)
+                         (local.get $6)
+                        )
+                        (local.set $5
+                         (i32.add
+                          (local.get $5)
+                          (i32.const 4)
+                         )
+                        )
+                       )
+                       (block
+                        (local.set $8
+                         (i32.add
+                          (local.get $1)
+                          (i32.const 4)
+                         )
+                        )
+                        (if
+                         (i32.eqz
+                          (i32.load
+                           (local.get $1)
+                          )
+                         )
+                         (local.set $1
+                          (local.get $8)
+                         )
+                        )
+                       )
+                      )
+                     )
+                     (local.set $6
+                      (i32.add
+                       (local.tee $8
+                        (if (result i32)
+                         (local.get $25)
+                         (local.get $7)
+                         (local.get $1)
+                        )
+                       )
+                       (i32.shl
+                        (local.get $14)
+                        (i32.const 2)
+                       )
+                      )
+                     )
+                     (if
+                      (i32.gt_s
+                       (i32.shr_s
+                        (i32.sub
+                         (local.get $5)
+                         (local.get $8)
+                        )
+                        (i32.const 2)
+                       )
+                       (local.get $14)
+                      )
+                      (local.set $5
+                       (local.get $6)
+                      )
+                     )
+                     (i32.store
+                      (local.get $20)
+                      (local.tee $6
+                       (i32.add
+                        (i32.load
+                         (local.get $20)
+                        )
+                        (local.get $13)
+                       )
+                      )
+                     )
+                     (br_if $label$160
+                      (i32.lt_s
+                       (local.get $6)
+                       (i32.const 0)
+                      )
+                     )
+                     (local.set $13
+                      (local.get $5)
+                     )
+                    )
+                   )
+                   (local.set $13
+                    (local.get $8)
+                   )
+                  )
+                  (local.set $25
+                   (local.get $7)
+                  )
+                  (block $label$172
+                   (if
+                    (i32.lt_u
+                     (local.get $1)
+                     (local.get $13)
+                    )
+                    (block
+                     (local.set $5
+                      (i32.mul
+                       (i32.shr_s
+                        (i32.sub
+                         (local.get $25)
+                         (local.get $1)
+                        )
+                        (i32.const 2)
+                       )
+                       (i32.const 9)
+                      )
+                     )
+                     (br_if $label$172
+                      (i32.lt_u
+                       (local.tee $6
+                        (i32.load
+                         (local.get $1)
+                        )
+                       )
+                       (i32.const 10)
+                      )
+                     )
+                     (local.set $8
+                      (i32.const 10)
+                     )
+                     (loop $label$174
+                      (local.set $5
+                       (i32.add
+                        (local.get $5)
+                        (i32.const 1)
+                       )
+                      )
+                      (br_if $label$174
+                       (i32.ge_u
+                        (local.get $6)
+                        (local.tee $8
+                         (i32.mul
+                          (local.get $8)
+                          (i32.const 10)
+                         )
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (local.set $5
+                     (i32.const 0)
+                    )
+                   )
+                  )
+                  (local.set $29
+                   (i32.eq
+                    (local.get $22)
+                    (i32.const 103)
+                   )
+                  )
+                  (local.set $35
+                   (i32.ne
+                    (local.get $18)
+                    (i32.const 0)
+                   )
+                  )
+                  (if
+                   (i32.lt_s
+                    (local.tee $8
+                     (i32.add
+                      (i32.sub
+                       (local.get $18)
+                       (if (result i32)
+                        (i32.ne
+                         (local.get $22)
+                         (i32.const 102)
+                        )
+                        (local.get $5)
+                        (i32.const 0)
+                       )
+                      )
+                      (i32.shr_s
+                       (i32.shl
+                        (i32.and
+                         (local.get $35)
+                         (local.get $29)
+                        )
+                        (i32.const 31)
+                       )
+                       (i32.const 31)
+                      )
+                     )
+                    )
+                    (i32.add
+                     (i32.mul
+                      (i32.shr_s
+                       (i32.sub
+                        (local.get $13)
+                        (local.get $25)
+                       )
+                       (i32.const 2)
+                      )
+                      (i32.const 9)
+                     )
+                     (i32.const -9)
+                    )
+                   )
+                   (block
+                    (if
+                     (i32.lt_s
+                      (local.tee $8
+                       (i32.add
+                        (i32.rem_s
+                         (local.tee $14
+                          (i32.add
+                           (local.get $8)
+                           (i32.const 9216)
+                          )
+                         )
+                         (i32.const 9)
+                        )
+                        (i32.const 1)
+                       )
+                      )
+                      (i32.const 9)
+                     )
+                     (block
+                      (local.set $6
+                       (i32.const 10)
+                      )
+                      (loop $label$180
+                       (local.set $6
+                        (i32.mul
+                         (local.get $6)
+                         (i32.const 10)
+                        )
+                       )
+                       (br_if $label$180
+                        (i32.ne
+                         (local.tee $8
+                          (i32.add
+                           (local.get $8)
+                           (i32.const 1)
+                          )
+                         )
+                         (i32.const 9)
+                        )
+                       )
+                      )
+                     )
+                     (local.set $6
+                      (i32.const 10)
+                     )
+                    )
+                    (local.set $14
+                     (i32.rem_u
+                      (local.tee $22
+                       (i32.load
+                        (local.tee $8
+                         (i32.add
+                          (i32.add
+                           (local.get $7)
+                           (i32.const 4)
+                          )
+                          (i32.shl
+                           (i32.add
+                            (i32.div_s
+                             (local.get $14)
+                             (i32.const 9)
+                            )
+                            (i32.const -1024)
+                           )
+                           (i32.const 2)
+                          )
+                         )
+                        )
+                       )
+                      )
+                      (local.get $6)
+                     )
+                    )
+                    (block $label$182
+                     (if
+                      (i32.eqz
+                       (i32.and
+                        (local.tee $32
+                         (i32.eq
+                          (i32.add
+                           (local.get $8)
+                           (i32.const 4)
+                          )
+                          (local.get $13)
+                         )
+                        )
+                        (i32.eqz
+                         (local.get $14)
+                        )
+                       )
+                      )
+                      (block
+                       (local.set $52
+                        (if (result f64)
+                         (i32.lt_u
+                          (local.get $14)
+                          (local.tee $49
+                           (i32.div_s
+                            (local.get $6)
+                            (i32.const 2)
+                           )
+                          )
+                         )
+                         (f64.const 0.5)
+                         (if (result f64)
+                          (i32.and
+                           (local.get $32)
+                           (i32.eq
+                            (local.get $14)
+                            (local.get $49)
+                           )
+                          )
+                          (f64.const 1)
+                          (f64.const 1.5)
+                         )
+                        )
+                       )
+                       (local.set $53
+                        (if (result f64)
+                         (i32.and
+                          (i32.div_u
+                           (local.get $22)
+                           (local.get $6)
+                          )
+                          (i32.const 1)
+                         )
+                         (f64.const 9007199254740994)
+                         (f64.const 9007199254740992)
+                        )
+                       )
+                       (block $label$190
+                        (if
+                         (local.get $24)
+                         (block
+                          (br_if $label$190
+                           (i32.ne
+                            (i32.load8_s
+                             (local.get $26)
+                            )
+                            (i32.const 45)
+                           )
+                          )
+                          (local.set $53
+                           (f64.neg
+                            (local.get $53)
+                           )
+                          )
+                          (local.set $52
+                           (f64.neg
+                            (local.get $52)
+                           )
+                          )
+                         )
+                        )
+                       )
+                       (i32.store
+                        (local.get $8)
+                        (local.tee $14
+                         (i32.sub
+                          (local.get $22)
+                          (local.get $14)
+                         )
+                        )
+                       )
+                       (br_if $label$182
+                        (f64.eq
+                         (f64.add
+                          (local.get $53)
+                          (local.get $52)
+                         )
+                         (local.get $53)
+                        )
+                       )
+                       (i32.store
+                        (local.get $8)
+                        (local.tee $5
+                         (i32.add
+                          (local.get $14)
+                          (local.get $6)
+                         )
+                        )
+                       )
+                       (if
+                        (i32.gt_u
+                         (local.get $5)
+                         (i32.const 999999999)
+                        )
+                        (loop $label$193
+                         (i32.store
+                          (local.get $8)
+                          (i32.const 0)
+                         )
+                         (if
+                          (i32.lt_u
+                           (local.tee $8
+                            (i32.add
+                             (local.get $8)
+                             (i32.const -4)
+                            )
+                           )
+                           (local.get $1)
+                          )
+                          (i32.store
+                           (local.tee $1
+                            (i32.add
+                             (local.get $1)
+                             (i32.const -4)
+                            )
+                           )
+                           (i32.const 0)
+                          )
+                         )
+                         (i32.store
+                          (local.get $8)
+                          (local.tee $5
+                           (i32.add
+                            (i32.load
+                             (local.get $8)
+                            )
+                            (i32.const 1)
+                           )
+                          )
+                         )
+                         (br_if $label$193
+                          (i32.gt_u
+                           (local.get $5)
+                           (i32.const 999999999)
+                          )
+                         )
+                        )
+                       )
+                       (local.set $5
+                        (i32.mul
+                         (i32.shr_s
+                          (i32.sub
+                           (local.get $25)
+                           (local.get $1)
+                          )
+                          (i32.const 2)
+                         )
+                         (i32.const 9)
+                        )
+                       )
+                       (br_if $label$182
+                        (i32.lt_u
+                         (local.tee $14
+                          (i32.load
+                           (local.get $1)
+                          )
+                         )
+                         (i32.const 10)
+                        )
+                       )
+                       (local.set $6
+                        (i32.const 10)
+                       )
+                       (loop $label$195
+                        (local.set $5
+                         (i32.add
+                          (local.get $5)
+                          (i32.const 1)
+                         )
+                        )
+                        (br_if $label$195
+                         (i32.ge_u
+                          (local.get $14)
+                          (local.tee $6
+                           (i32.mul
+                            (local.get $6)
+                            (i32.const 10)
+                           )
+                          )
+                         )
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (local.set $14
+                     (local.get $1)
+                    )
+                    (local.set $6
+                     (local.get $5)
+                    )
+                    (if
+                     (i32.le_u
+                      (local.get $13)
+                      (local.tee $8
+                       (i32.add
+                        (local.get $8)
+                        (i32.const 4)
+                       )
+                      )
+                     )
+                     (local.set $8
+                      (local.get $13)
+                     )
+                    )
+                   )
+                   (block
+                    (local.set $14
+                     (local.get $1)
+                    )
+                    (local.set $6
+                     (local.get $5)
+                    )
+                    (local.set $8
+                     (local.get $13)
+                    )
+                   )
+                  )
+                  (local.set $32
+                   (i32.sub
+                    (i32.const 0)
+                    (local.get $6)
+                   )
+                  )
+                  (loop $label$198
+                   (block $label$199
+                    (if
+                     (i32.le_u
+                      (local.get $8)
+                      (local.get $14)
+                     )
+                     (block
+                      (local.set $22
+                       (i32.const 0)
+                      )
+                      (br $label$199)
+                     )
+                    )
+                    (if
+                     (i32.load
+                      (local.tee $1
+                       (i32.add
+                        (local.get $8)
+                        (i32.const -4)
+                       )
+                      )
+                     )
+                     (local.set $22
+                      (i32.const 1)
+                     )
+                     (block
+                      (local.set $8
+                       (local.get $1)
+                      )
+                      (br $label$198)
+                     )
+                    )
+                   )
+                  )
+                  (block $label$203
+                   (if
+                    (local.get $29)
+                    (block
+                     (local.set $1
+                      (if (result i32)
+                       (i32.and
+                        (i32.gt_s
+                         (local.tee $1
+                          (i32.add
+                           (i32.xor
+                            (i32.and
+                             (local.get $35)
+                             (i32.const 1)
+                            )
+                            (i32.const 1)
+                           )
+                           (local.get $18)
+                          )
+                         )
+                         (local.get $6)
+                        )
+                        (i32.gt_s
+                         (local.get $6)
+                         (i32.const -5)
+                        )
+                       )
+                       (block (result i32)
+                        (local.set $5
+                         (i32.add
+                          (local.get $9)
+                          (i32.const -1)
+                         )
+                        )
+                        (i32.sub
+                         (i32.add
+                          (local.get $1)
+                          (i32.const -1)
+                         )
+                         (local.get $6)
+                        )
+                       )
+                       (block (result i32)
+                        (local.set $5
+                         (i32.add
+                          (local.get $9)
+                          (i32.const -2)
+                         )
+                        )
+                        (i32.add
+                         (local.get $1)
+                         (i32.const -1)
+                        )
+                       )
+                      )
+                     )
+                     (br_if $label$203
+                      (local.tee $13
+                       (i32.and
+                        (local.get $12)
+                        (i32.const 8)
+                       )
+                      )
+                     )
+                     (block $label$207
+                      (if
+                       (local.get $22)
+                       (block
+                        (if
+                         (i32.eqz
+                          (local.tee $18
+                           (i32.load
+                            (i32.add
+                             (local.get $8)
+                             (i32.const -4)
+                            )
+                           )
+                          )
+                         )
+                         (block
+                          (local.set $9
+                           (i32.const 9)
+                          )
+                          (br $label$207)
+                         )
+                        )
+                        (if
+                         (i32.rem_u
+                          (local.get $18)
+                          (i32.const 10)
+                         )
+                         (block
+                          (local.set $9
+                           (i32.const 0)
+                          )
+                          (br $label$207)
+                         )
+                         (block
+                          (local.set $13
+                           (i32.const 10)
+                          )
+                          (local.set $9
+                           (i32.const 0)
+                          )
+                         )
+                        )
+                        (loop $label$212
+                         (local.set $9
+                          (i32.add
+                           (local.get $9)
+                           (i32.const 1)
+                          )
+                         )
+                         (br_if $label$212
+                          (i32.eqz
+                           (i32.rem_u
+                            (local.get $18)
+                            (local.tee $13
+                             (i32.mul
+                              (local.get $13)
+                              (i32.const 10)
+                             )
+                            )
+                           )
+                          )
+                         )
+                        )
+                       )
+                       (local.set $9
+                        (i32.const 9)
+                       )
+                      )
+                     )
+                     (local.set $18
+                      (i32.add
+                       (i32.mul
+                        (i32.shr_s
+                         (i32.sub
+                          (local.get $8)
+                          (local.get $25)
+                         )
+                         (i32.const 2)
+                        )
+                        (i32.const 9)
+                       )
+                       (i32.const -9)
+                      )
+                     )
+                     (if
+                      (i32.eq
+                       (i32.or
+                        (local.get $5)
+                        (i32.const 32)
+                       )
+                       (i32.const 102)
+                      )
+                      (block
+                       (local.set $13
+                        (i32.const 0)
+                       )
+                       (if
+                        (i32.ge_s
+                         (local.get $1)
+                         (if (result i32)
+                          (i32.lt_s
+                           (local.tee $9
+                            (i32.sub
+                             (local.get $18)
+                             (local.get $9)
+                            )
+                           )
+                           (i32.const 0)
+                          )
+                          (local.tee $9
+                           (i32.const 0)
+                          )
+                          (local.get $9)
+                         )
+                        )
+                        (local.set $1
+                         (local.get $9)
+                        )
+                       )
+                      )
+                      (block
+                       (local.set $13
+                        (i32.const 0)
+                       )
+                       (if
+                        (i32.ge_s
+                         (local.get $1)
+                         (if (result i32)
+                          (i32.lt_s
+                           (local.tee $9
+                            (i32.sub
+                             (i32.add
+                              (local.get $18)
+                              (local.get $6)
+                             )
+                             (local.get $9)
+                            )
+                           )
+                           (i32.const 0)
+                          )
+                          (local.tee $9
+                           (i32.const 0)
+                          )
+                          (local.get $9)
+                         )
+                        )
+                        (local.set $1
+                         (local.get $9)
+                        )
+                       )
+                      )
+                     )
+                    )
+                    (block
+                     (local.set $13
+                      (i32.and
+                       (local.get $12)
+                       (i32.const 8)
+                      )
+                     )
+                     (local.set $1
+                      (local.get $18)
+                     )
+                     (local.set $5
+                      (local.get $9)
+                     )
+                    )
+                   )
+                  )
+                  (if
+                   (local.tee $25
+                    (i32.eq
+                     (i32.or
+                      (local.get $5)
+                      (i32.const 32)
+                     )
+                     (i32.const 102)
+                    )
+                   )
+                   (block
+                    (local.set $9
+                     (i32.const 0)
+                    )
+                    (if
+                     (i32.le_s
+                      (local.get $6)
+                      (i32.const 0)
+                     )
+                     (local.set $6
+                      (i32.const 0)
+                     )
+                    )
+                   )
+                   (block
+                    (if
+                     (i32.lt_s
+                      (i32.sub
+                       (local.get $28)
+                       (local.tee $9
+                        (call $22
+                         (i64.extend_i32_s
+                          (if (result i32)
+                           (i32.lt_s
+                            (local.get $6)
+                            (i32.const 0)
+                           )
+                           (local.get $32)
+                           (local.get $6)
+                          )
+                         )
+                         (local.get $33)
+                        )
+                       )
+                      )
+                      (i32.const 2)
+                     )
+                     (loop $label$229
+                      (i32.store8
+                       (local.tee $9
+                        (i32.add
+                         (local.get $9)
+                         (i32.const -1)
+                        )
+                       )
+                       (i32.const 48)
+                      )
+                      (br_if $label$229
+                       (i32.lt_s
+                        (i32.sub
+                         (local.get $28)
+                         (local.get $9)
+                        )
+                        (i32.const 2)
+                       )
+                      )
+                     )
+                    )
+                    (i32.store8
+                     (i32.add
+                      (local.get $9)
+                      (i32.const -1)
+                     )
+                     (i32.add
+                      (i32.and
+                       (i32.shr_s
+                        (local.get $6)
+                        (i32.const 31)
+                       )
+                       (i32.const 2)
+                      )
+                      (i32.const 43)
+                     )
+                    )
+                    (i32.store8
+                     (local.tee $6
+                      (i32.add
+                       (local.get $9)
+                       (i32.const -2)
+                      )
+                     )
+                     (local.get $5)
+                    )
+                    (local.set $9
+                     (local.get $6)
+                    )
+                    (local.set $6
+                     (i32.sub
+                      (local.get $28)
+                      (local.get $6)
+                     )
+                    )
+                   )
+                  )
+                  (call $24
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.tee $18
+                    (i32.add
+                     (i32.add
+                      (i32.add
+                       (i32.add
+                        (local.get $24)
+                        (i32.const 1)
+                       )
+                       (local.get $1)
+                      )
+                      (i32.ne
+                       (local.tee $29
+                        (i32.or
+                         (local.get $1)
+                         (local.get $13)
+                        )
+                       )
+                       (i32.const 0)
+                      )
+                     )
+                     (local.get $6)
+                    )
+                   )
+                   (local.get $12)
+                  )
+                  (if
+                   (i32.eqz
+                    (i32.and
+                     (i32.load
+                      (local.get $0)
+                     )
+                     (i32.const 32)
+                    )
+                   )
+                   (drop
+                    (call $20
+                     (local.get $26)
+                     (local.get $24)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (call $24
+                   (local.get $0)
+                   (i32.const 48)
+                   (local.get $10)
+                   (local.get $18)
+                   (i32.xor
+                    (local.get $12)
+                    (i32.const 65536)
+                   )
+                  )
+                  (block $label$231
+                   (if
+                    (local.get $25)
+                    (block
+                     (local.set $6
+                      (local.tee $9
+                       (if (result i32)
+                        (i32.gt_u
+                         (local.get $14)
+                         (local.get $7)
+                        )
+                        (local.get $7)
+                        (local.get $14)
+                       )
+                      )
+                     )
+                     (loop $label$235
+                      (local.set $5
+                       (call $22
+                        (i64.extend_i32_u
+                         (i32.load
+                          (local.get $6)
+                         )
+                        )
+                        (local.get $31)
+                       )
+                      )
+                      (block $label$236
+                       (if
+                        (i32.eq
+                         (local.get $6)
+                         (local.get $9)
+                        )
+                        (block
+                         (br_if $label$236
+                          (i32.ne
+                           (local.get $5)
+                           (local.get $31)
+                          )
+                         )
+                         (i32.store8
+                          (local.get $34)
+                          (i32.const 48)
+                         )
+                         (local.set $5
+                          (local.get $34)
+                         )
+                        )
+                        (block
+                         (br_if $label$236
+                          (i32.le_u
+                           (local.get $5)
+                           (local.get $19)
+                          )
+                         )
+                         (drop
+                          (call $35
+                           (local.get $19)
+                           (i32.const 48)
+                           (i32.sub
+                            (local.get $5)
+                            (local.get $27)
+                           )
+                          )
+                         )
+                         (loop $label$239
+                          (br_if $label$239
+                           (i32.gt_u
+                            (local.tee $5
+                             (i32.add
+                              (local.get $5)
+                              (i32.const -1)
+                             )
+                            )
+                            (local.get $19)
+                           )
+                          )
+                         )
+                        )
+                       )
+                      )
+                      (if
+                       (i32.eqz
+                        (i32.and
+                         (i32.load
+                          (local.get $0)
+                         )
+                         (i32.const 32)
+                        )
+                       )
+                       (drop
+                        (call $20
+                         (local.get $5)
+                         (i32.sub
+                          (local.get $41)
+                          (local.get $5)
+                         )
+                         (local.get $0)
+                        )
+                       )
+                      )
+                      (if
+                       (i32.le_u
+                        (local.tee $5
+                         (i32.add
+                          (local.get $6)
+                          (i32.const 4)
+                         )
+                        )
+                        (local.get $7)
+                       )
+                       (block
+                        (local.set $6
+                         (local.get $5)
+                        )
+                        (br $label$235)
+                       )
+                      )
+                     )
+                     (block $label$242
+                      (if
+                       (local.get $29)
+                       (block
+                        (br_if $label$242
+                         (i32.and
+                          (i32.load
+                           (local.get $0)
+                          )
+                          (i32.const 32)
+                         )
+                        )
+                        (drop
+                         (call $20
+                          (i32.const 1700)
+                          (i32.const 1)
+                          (local.get $0)
+                         )
+                        )
+                       )
+                      )
+                     )
+                     (if
+                      (i32.and
+                       (i32.gt_s
+                        (local.get $1)
+                        (i32.const 0)
+                       )
+                       (i32.lt_u
+                        (local.get $5)
+                        (local.get $8)
+                       )
+                      )
+                      (loop $label$245
+                       (if
+                        (i32.gt_u
+                         (local.tee $7
+                          (call $22
+                           (i64.extend_i32_u
+                            (i32.load
+                             (local.get $5)
+                            )
+                           )
+                           (local.get $31)
+                          )
+                         )
+                         (local.get $19)
+                        )
+                        (block
+                         (drop
+                          (call $35
+                           (local.get $19)
+                           (i32.const 48)
+                           (i32.sub
+                            (local.get $7)
+                            (local.get $27)
+                           )
+                          )
+                         )
+                         (loop $label$247
+                          (br_if $label$247
+                           (i32.gt_u
+                            (local.tee $7
+                             (i32.add
+                              (local.get $7)
+                              (i32.const -1)
+                             )
+                            )
+                            (local.get $19)
+                           )
+                          )
+                         )
+                        )
+                       )
+                       (if
+                        (i32.eqz
+                         (i32.and
+                          (i32.load
+                           (local.get $0)
+                          )
+                          (i32.const 32)
+                         )
+                        )
+                        (drop
+                         (call $20
+                          (local.get $7)
+                          (if (result i32)
+                           (i32.gt_s
+                            (local.get $1)
+                            (i32.const 9)
+                           )
+                           (i32.const 9)
+                           (local.get $1)
+                          )
+                          (local.get $0)
+                         )
+                        )
+                       )
+                       (local.set $7
+                        (i32.add
+                         (local.get $1)
+                         (i32.const -9)
+                        )
+                       )
+                       (if
+                        (i32.and
+                         (i32.gt_s
+                          (local.get $1)
+                          (i32.const 9)
+                         )
+                         (i32.lt_u
+                          (local.tee $5
+                           (i32.add
+                            (local.get $5)
+                            (i32.const 4)
+                           )
+                          )
+                          (local.get $8)
+                         )
+                        )
+                        (block
+                         (local.set $1
+                          (local.get $7)
+                         )
+                         (br $label$245)
+                        )
+                        (local.set $1
+                         (local.get $7)
+                        )
+                       )
+                      )
+                     )
+                     (call $24
+                      (local.get $0)
+                      (i32.const 48)
+                      (i32.add
+                       (local.get $1)
+                       (i32.const 9)
+                      )
+                      (i32.const 9)
+                      (i32.const 0)
+                     )
+                    )
+                    (block
+                     (local.set $5
+                      (i32.add
+                       (local.get $14)
+                       (i32.const 4)
+                      )
+                     )
+                     (if
+                      (i32.eqz
+                       (local.get $22)
+                      )
+                      (local.set $8
+                       (local.get $5)
+                      )
+                     )
+                     (if
+                      (i32.gt_s
+                       (local.get $1)
+                       (i32.const -1)
+                      )
+                      (block
+                       (local.set $13
+                        (i32.eqz
+                         (local.get $13)
+                        )
+                       )
+                       (local.set $7
+                        (local.get $14)
+                       )
+                       (local.set $5
+                        (local.get $1)
+                       )
+                       (loop $label$256
+                        (if
+                         (i32.eq
+                          (local.tee $1
+                           (call $22
+                            (i64.extend_i32_u
+                             (i32.load
+                              (local.get $7)
+                             )
+                            )
+                            (local.get $31)
+                           )
+                          )
+                          (local.get $31)
+                         )
+                         (block
+                          (i32.store8
+                           (local.get $34)
+                           (i32.const 48)
+                          )
+                          (local.set $1
+                           (local.get $34)
+                          )
+                         )
+                        )
+                        (block $label$258
+                         (if
+                          (i32.eq
+                           (local.get $7)
+                           (local.get $14)
+                          )
+                          (block
+                           (if
+                            (i32.eqz
+                             (i32.and
+                              (i32.load
+                               (local.get $0)
+                              )
+                              (i32.const 32)
+                             )
+                            )
+                            (drop
+                             (call $20
+                              (local.get $1)
+                              (i32.const 1)
+                              (local.get $0)
+                             )
+                            )
+                           )
+                           (local.set $1
+                            (i32.add
+                             (local.get $1)
+                             (i32.const 1)
+                            )
+                           )
+                           (br_if $label$258
+                            (i32.and
+                             (local.get $13)
+                             (i32.lt_s
+                              (local.get $5)
+                              (i32.const 1)
+                             )
+                            )
+                           )
+                           (br_if $label$258
+                            (i32.and
+                             (i32.load
+                              (local.get $0)
+                             )
+                             (i32.const 32)
+                            )
+                           )
+                           (drop
+                            (call $20
+                             (i32.const 1700)
+                             (i32.const 1)
+                             (local.get $0)
+                            )
+                           )
+                          )
+                          (block
+                           (br_if $label$258
+                            (i32.le_u
+                             (local.get $1)
+                             (local.get $19)
+                            )
+                           )
+                           (drop
+                            (call $35
+                             (local.get $19)
+                             (i32.const 48)
+                             (i32.add
+                              (local.get $1)
+                              (local.get $43)
+                             )
+                            )
+                           )
+                           (loop $label$262
+                            (br_if $label$262
+                             (i32.gt_u
+                              (local.tee $1
+                               (i32.add
+                                (local.get $1)
+                                (i32.const -1)
+                               )
+                              )
+                              (local.get $19)
+                             )
+                            )
+                           )
+                          )
+                         )
+                        )
+                        (local.set $6
+                         (i32.sub
+                          (local.get $41)
+                          (local.get $1)
+                         )
+                        )
+                        (if
+                         (i32.eqz
+                          (i32.and
+                           (i32.load
+                            (local.get $0)
+                           )
+                           (i32.const 32)
+                          )
+                         )
+                         (drop
+                          (call $20
+                           (local.get $1)
+                           (if (result i32)
+                            (i32.gt_s
+                             (local.get $5)
+                             (local.get $6)
+                            )
+                            (local.get $6)
+                            (local.get $5)
+                           )
+                           (local.get $0)
+                          )
+                         )
+                        )
+                        (br_if $label$256
+                         (i32.and
+                          (i32.lt_u
+                           (local.tee $7
+                            (i32.add
+                             (local.get $7)
+                             (i32.const 4)
+                            )
+                           )
+                           (local.get $8)
+                          )
+                          (i32.gt_s
+                           (local.tee $5
+                            (i32.sub
+                             (local.get $5)
+                             (local.get $6)
+                            )
+                           )
+                           (i32.const -1)
+                          )
+                         )
+                        )
+                        (local.set $1
+                         (local.get $5)
+                        )
+                       )
+                      )
+                     )
+                     (call $24
+                      (local.get $0)
+                      (i32.const 48)
+                      (i32.add
+                       (local.get $1)
+                       (i32.const 18)
+                      )
+                      (i32.const 18)
+                      (i32.const 0)
+                     )
+                     (br_if $label$231
+                      (i32.and
+                       (i32.load
+                        (local.get $0)
+                       )
+                       (i32.const 32)
+                      )
+                     )
+                     (drop
+                      (call $20
+                       (local.get $9)
+                       (i32.sub
+                        (local.get $28)
+                        (local.get $9)
+                       )
+                       (local.get $0)
+                      )
+                     )
+                    )
+                   )
+                  )
+                  (call $24
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.get $18)
+                   (i32.xor
+                    (local.get $12)
+                    (i32.const 8192)
+                   )
+                  )
+                  (if
+                   (i32.ge_s
+                    (local.get $18)
+                    (local.get $10)
+                   )
+                   (local.set $10
+                    (local.get $18)
+                   )
+                  )
+                 )
+                 (block
+                  (call $24
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.tee $8
+                    (i32.add
+                     (if (result i32)
+                      (local.tee $6
+                       (i32.or
+                        (f64.ne
+                         (local.get $52)
+                         (local.get $52)
+                        )
+                        (i32.const 0)
+                       )
+                      )
+                      (local.tee $24
+                       (i32.const 0)
+                      )
+                      (local.get $24)
+                     )
+                     (i32.const 3)
+                    )
+                   )
+                   (local.get $7)
+                  )
+                  (if
+                   (i32.eqz
+                    (i32.and
+                     (local.tee $1
+                      (i32.load
+                       (local.get $0)
+                      )
+                     )
+                     (i32.const 32)
+                    )
+                   )
+                   (block
+                    (drop
+                     (call $20
+                      (local.get $26)
+                      (local.get $24)
+                      (local.get $0)
+                     )
+                    )
+                    (local.set $1
+                     (i32.load
+                      (local.get $0)
+                     )
+                    )
+                   )
+                  )
+                  (local.set $7
+                   (if (result i32)
+                    (local.tee $5
+                     (i32.ne
+                      (i32.and
+                       (local.get $9)
+                       (i32.const 32)
+                      )
+                      (i32.const 0)
+                     )
+                    )
+                    (i32.const 1684)
+                    (i32.const 1688)
+                   )
+                  )
+                  (local.set $5
+                   (if (result i32)
+                    (local.get $5)
+                    (i32.const 1692)
+                    (i32.const 1696)
+                   )
+                  )
+                  (if
+                   (i32.eqz
+                    (local.get $6)
+                   )
+                   (local.set $5
+                    (local.get $7)
+                   )
+                  )
+                  (if
+                   (i32.eqz
+                    (i32.and
+                     (local.get $1)
+                     (i32.const 32)
+                    )
+                   )
+                   (drop
+                    (call $20
+                     (local.get $5)
+                     (i32.const 3)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (call $24
+                   (local.get $0)
+                   (i32.const 32)
+                   (local.get $10)
+                   (local.get $8)
+                   (i32.xor
+                    (local.get $12)
+                    (i32.const 8192)
+                   )
+                  )
+                  (if
+                   (i32.ge_s
+                    (local.get $8)
+                    (local.get $10)
+                   )
+                   (local.set $10
+                    (local.get $8)
+                   )
+                  )
+                 )
+                )
+               )
+               (local.set $1
+                (local.get $11)
+               )
+               (br $label$4)
+              )
+              (local.set $7
+               (local.get $5)
+              )
+              (local.set $6
+               (i32.const 0)
+              )
+              (local.set $8
+               (i32.const 1648)
+              )
+              (local.set $5
+               (local.get $21)
+              )
+              (br $label$70)
+             )
+             (local.set $7
+              (i32.and
+               (local.get $9)
+               (i32.const 32)
+              )
+             )
+             (local.set $7
+              (if (result i32)
+               (i64.eq
+                (local.tee $50
+                 (i64.load
+                  (local.get $16)
+                 )
+                )
+                (i64.const 0)
+               )
+               (block (result i32)
+                (local.set $50
+                 (i64.const 0)
+                )
+                (local.get $21)
+               )
+               (block (result i32)
+                (local.set $1
+                 (local.get $21)
+                )
+                (loop $label$280
+                 (i32.store8
+                  (local.tee $1
+                   (i32.add
+                    (local.get $1)
+                    (i32.const -1)
+                   )
+                  )
+                  (i32.or
+                   (i32.load8_u
+                    (i32.add
+                     (i32.and
+                      (i32.wrap_i64
+                       (local.get $50)
+                      )
+                      (i32.const 15)
+                     )
+                     (i32.const 1632)
+                    )
+                   )
+                   (local.get $7)
+                  )
+                 )
+                 (br_if $label$280
+                  (i64.ne
+                   (local.tee $50
+                    (i64.shr_u
+                     (local.get $50)
+                     (i64.const 4)
+                    )
+                   )
+                   (i64.const 0)
+                  )
+                 )
+                )
+                (local.set $50
+                 (i64.load
+                  (local.get $16)
+                 )
+                )
+                (local.get $1)
+               )
+              )
+             )
+             (local.set $8
+              (i32.add
+               (i32.shr_s
+                (local.get $9)
+                (i32.const 4)
+               )
+               (i32.const 1648)
+              )
+             )
+             (if
+              (local.tee $1
+               (i32.or
+                (i32.eqz
+                 (i32.and
+                  (local.get $12)
+                  (i32.const 8)
+                 )
+                )
+                (i64.eq
+                 (local.get $50)
+                 (i64.const 0)
+                )
+               )
+              )
+              (local.set $8
+               (i32.const 1648)
+              )
+             )
+             (local.set $6
+              (if (result i32)
+               (local.get $1)
+               (i32.const 0)
+               (i32.const 2)
+              )
+             )
+             (br $label$71)
+            )
+            (local.set $7
+             (call $22
+              (local.get $50)
+              (local.get $21)
+             )
+            )
+            (br $label$71)
+           )
+           (local.set $14
+            (i32.eqz
+             (local.tee $13
+              (call $16
+               (local.get $1)
+               (i32.const 0)
+               (local.get $5)
+              )
+             )
+            )
+           )
+           (local.set $8
+            (i32.sub
+             (local.get $13)
+             (local.get $1)
+            )
+           )
+           (local.set $9
+            (i32.add
+             (local.get $1)
+             (local.get $5)
+            )
+           )
+           (local.set $12
+            (local.get $7)
+           )
+           (local.set $7
+            (if (result i32)
+             (local.get $14)
+             (local.get $5)
+             (local.get $8)
+            )
+           )
+           (local.set $6
+            (i32.const 0)
+           )
+           (local.set $8
+            (i32.const 1648)
+           )
+           (local.set $5
+            (if (result i32)
+             (local.get $14)
+             (local.get $9)
+             (local.get $13)
+            )
+           )
+           (br $label$70)
+          )
+          (local.set $1
+           (i32.const 0)
+          )
+          (local.set $5
+           (i32.const 0)
+          )
+          (local.set $8
+           (local.get $7)
+          )
+          (loop $label$288
+           (block $label$289
+            (br_if $label$289
+             (i32.eqz
+              (local.tee $9
+               (i32.load
+                (local.get $8)
+               )
+              )
+             )
+            )
+            (br_if $label$289
+             (i32.or
+              (i32.lt_s
+               (local.tee $5
+                (call $25
+                 (local.get $36)
+                 (local.get $9)
+                )
+               )
+               (i32.const 0)
+              )
+              (i32.gt_u
+               (local.get $5)
+               (i32.sub
+                (local.get $6)
+                (local.get $1)
+               )
+              )
+             )
+            )
+            (local.set $8
+             (i32.add
+              (local.get $8)
+              (i32.const 4)
+             )
+            )
+            (br_if $label$288
+             (i32.gt_u
+              (local.get $6)
+              (local.tee $1
+               (i32.add
+                (local.get $5)
+                (local.get $1)
+               )
+              )
+             )
+            )
+           )
+          )
+          (if
+           (i32.lt_s
+            (local.get $5)
+            (i32.const 0)
+           )
+           (block
+            (local.set $15
+             (i32.const -1)
+            )
+            (br $label$5)
+           )
+          )
+          (call $24
+           (local.get $0)
+           (i32.const 32)
+           (local.get $10)
+           (local.get $1)
+           (local.get $12)
+          )
+          (if
+           (local.get $1)
+           (block
+            (local.set $5
+             (i32.const 0)
+            )
+            (loop $label$292
+             (br_if $label$72
+              (i32.eqz
+               (local.tee $8
+                (i32.load
+                 (local.get $7)
+                )
+               )
+              )
+             )
+             (br_if $label$72
+              (i32.gt_s
+               (local.tee $5
+                (i32.add
+                 (local.tee $8
+                  (call $25
+                   (local.get $36)
+                   (local.get $8)
+                  )
+                 )
+                 (local.get $5)
+                )
+               )
+               (local.get $1)
+              )
+             )
+             (if
+              (i32.eqz
+               (i32.and
+                (i32.load
+                 (local.get $0)
+                )
+                (i32.const 32)
+               )
+              )
+              (drop
+               (call $20
+                (local.get $36)
+                (local.get $8)
+                (local.get $0)
+               )
+              )
+             )
+             (local.set $7
+              (i32.add
+               (local.get $7)
+               (i32.const 4)
+              )
+             )
+             (br_if $label$292
+              (i32.lt_u
+               (local.get $5)
+               (local.get $1)
+              )
+             )
+             (br $label$72)
+            )
+           )
+           (block
+            (local.set $1
+             (i32.const 0)
+            )
+            (br $label$72)
+           )
+          )
+         )
+         (call $24
+          (local.get $0)
+          (i32.const 32)
+          (local.get $10)
+          (local.get $1)
+          (i32.xor
+           (local.get $12)
+           (i32.const 8192)
+          )
+         )
+         (if
+          (i32.le_s
+           (local.get $10)
+           (local.get $1)
+          )
+          (local.set $10
+           (local.get $1)
+          )
+         )
+         (local.set $1
+          (local.get $11)
+         )
+         (br $label$4)
+        )
+        (local.set $1
+         (i32.and
+          (local.get $12)
+          (i32.const -65537)
+         )
+        )
+        (if
+         (i32.gt_s
+          (local.get $5)
+          (i32.const -1)
+         )
+         (local.set $12
+          (local.get $1)
+         )
+        )
+        (local.set $5
+         (if (result i32)
+          (i32.or
+           (local.get $5)
+           (local.tee $9
+            (i64.ne
+             (i64.load
+              (local.get $16)
+             )
+             (i64.const 0)
+            )
+           )
+          )
+          (block (result i32)
+           (local.set $1
+            (local.get $7)
+           )
+           (if
+            (i32.gt_s
+             (local.get $5)
+             (local.tee $7
+              (i32.add
+               (i32.xor
+                (i32.and
+                 (local.get $9)
+                 (i32.const 1)
+                )
+                (i32.const 1)
+               )
+               (i32.sub
+                (local.get $38)
+                (local.get $7)
+               )
+              )
+             )
+            )
+            (local.set $7
+             (local.get $5)
+            )
+           )
+           (local.get $21)
+          )
+          (block (result i32)
+           (local.set $1
+            (local.get $21)
+           )
+           (local.set $7
+            (i32.const 0)
+           )
+           (local.get $21)
+          )
+         )
+        )
+       )
+       (call $24
+        (local.get $0)
+        (i32.const 32)
+        (if (result i32)
+         (i32.lt_s
+          (local.get $10)
+          (local.tee $5
+           (i32.add
+            (if (result i32)
+             (i32.lt_s
+              (local.get $7)
+              (local.tee $9
+               (i32.sub
+                (local.get $5)
+                (local.get $1)
+               )
+              )
+             )
+             (local.tee $7
+              (local.get $9)
+             )
+             (local.get $7)
+            )
+            (local.get $6)
+           )
+          )
+         )
+         (local.tee $10
+          (local.get $5)
+         )
+         (local.get $10)
+        )
+        (local.get $5)
+        (local.get $12)
+       )
+       (if
+        (i32.eqz
+         (i32.and
+          (i32.load
+           (local.get $0)
+          )
+          (i32.const 32)
+         )
+        )
+        (drop
+         (call $20
+          (local.get $8)
+          (local.get $6)
+          (local.get $0)
+         )
+        )
+       )
+       (call $24
+        (local.get $0)
+        (i32.const 48)
+        (local.get $10)
+        (local.get $5)
+        (i32.xor
+         (local.get $12)
+         (i32.const 65536)
+        )
+       )
+       (call $24
+        (local.get $0)
+        (i32.const 48)
+        (local.get $7)
+        (local.get $9)
+        (i32.const 0)
+       )
+       (if
+        (i32.eqz
+         (i32.and
+          (i32.load
+           (local.get $0)
+          )
+          (i32.const 32)
+         )
+        )
+        (drop
+         (call $20
+          (local.get $1)
+          (local.get $9)
+          (local.get $0)
+         )
+        )
+       )
+       (call $24
+        (local.get $0)
+        (i32.const 32)
+        (local.get $10)
+        (local.get $5)
+        (i32.xor
+         (local.get $12)
+         (i32.const 8192)
+        )
+       )
+       (local.set $1
+        (local.get $11)
+       )
+       (br $label$4)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.eqz
+      (local.get $0)
+     )
+     (if
+      (local.get $17)
+      (block
+       (local.set $0
+        (i32.const 1)
+       )
+       (loop $label$308
+        (if
+         (local.tee $1
+          (i32.load
+           (i32.add
+            (local.get $4)
+            (i32.shl
+             (local.get $0)
+             (i32.const 2)
+            )
+           )
+          )
+         )
+         (block
+          (call $21
+           (i32.add
+            (local.get $3)
+            (i32.shl
+             (local.get $0)
+             (i32.const 3)
+            )
+           )
+           (local.get $1)
+           (local.get $2)
+          )
+          (br_if $label$308
+           (i32.lt_s
+            (local.tee $0
+             (i32.add
+              (local.get $0)
+              (i32.const 1)
+             )
+            )
+            (i32.const 10)
+           )
+          )
+          (local.set $15
+           (i32.const 1)
+          )
+          (br $label$2)
+         )
+        )
+       )
+       (loop $label$310
+        (if
+         (i32.load
+          (i32.add
+           (local.get $4)
+           (i32.shl
+            (local.get $0)
+            (i32.const 2)
+           )
+          )
+         )
+         (block
+          (local.set $15
+           (i32.const -1)
+          )
+          (br $label$2)
+         )
+        )
+        (br_if $label$310
+         (i32.lt_s
+          (local.tee $0
+           (i32.add
+            (local.get $0)
+            (i32.const 1)
+           )
+          )
+          (i32.const 10)
+         )
+        )
+        (local.set $15
+         (i32.const 1)
+        )
+       )
+      )
+      (local.set $15
+       (i32.const 0)
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $23)
+   )
+   (local.get $15)
+  )
+ )
+ (func $19 (; 32 ;) (type $1) (param $0 i32) (result i32)
+  (i32.const 0)
+ )
+ (func $20 (; 33 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (block $label$1 (result i32)
+   (block $label$2
+    (block $label$3
+     (br_if $label$3
+      (local.tee $3
+       (i32.load
+        (local.tee $4
+         (i32.add
+          (local.get $2)
+          (i32.const 16)
+         )
+        )
+       )
+      )
+     )
+     (if
+      (call $29
+       (local.get $2)
+      )
+      (local.set $3
+       (i32.const 0)
+      )
+      (block
+       (local.set $3
+        (i32.load
+         (local.get $4)
+        )
+       )
+       (br $label$3)
+      )
+     )
+     (br $label$2)
+    )
+    (if
+     (i32.lt_u
+      (i32.sub
+       (local.get $3)
+       (local.tee $4
+        (i32.load
+         (local.tee $5
+          (i32.add
+           (local.get $2)
+           (i32.const 20)
+          )
+         )
+        )
+       )
+      )
+      (local.get $1)
+     )
+     (block
+      (local.set $3
+       (call_indirect (type $0)
+        (local.get $2)
+        (local.get $0)
+        (local.get $1)
+        (i32.add
+         (i32.and
+          (i32.load offset=36
+           (local.get $2)
+          )
+          (i32.const 3)
+         )
+         (i32.const 2)
+        )
+       )
+      )
+      (br $label$2)
+     )
+    )
+    (local.set $2
+     (block $label$7 (result i32)
+      (if (result i32)
+       (i32.gt_s
+        (i32.load8_s offset=75
+         (local.get $2)
+        )
+        (i32.const -1)
+       )
+       (block (result i32)
+        (local.set $3
+         (local.get $1)
+        )
+        (loop $label$9
+         (drop
+          (br_if $label$7
+           (i32.const 0)
+           (i32.eqz
+            (local.get $3)
+           )
+          )
+         )
+         (if
+          (i32.ne
+           (i32.load8_s
+            (i32.add
+             (local.get $0)
+             (local.tee $6
+              (i32.add
+               (local.get $3)
+               (i32.const -1)
+              )
+             )
+            )
+           )
+           (i32.const 10)
+          )
+          (block
+           (local.set $3
+            (local.get $6)
+           )
+           (br $label$9)
+          )
+         )
+        )
+        (br_if $label$2
+         (i32.lt_u
+          (call_indirect (type $0)
+           (local.get $2)
+           (local.get $0)
+           (local.get $3)
+           (i32.add
+            (i32.and
+             (i32.load offset=36
+              (local.get $2)
+             )
+             (i32.const 3)
+            )
+            (i32.const 2)
+           )
+          )
+          (local.get $3)
+         )
+        )
+        (local.set $4
+         (i32.load
+          (local.get $5)
+         )
+        )
+        (local.set $1
+         (i32.sub
+          (local.get $1)
+          (local.get $3)
+         )
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (local.get $3)
+         )
+        )
+        (local.get $3)
+       )
+       (i32.const 0)
+      )
+     )
+    )
+    (drop
+     (call $36
+      (local.get $4)
+      (local.get $0)
+      (local.get $1)
+     )
+    )
+    (i32.store
+     (local.get $5)
+     (i32.add
+      (i32.load
+       (local.get $5)
+      )
+      (local.get $1)
+     )
+    )
+    (local.set $3
+     (i32.add
+      (local.get $2)
+      (local.get $1)
+     )
+    )
+   )
+   (local.get $3)
+  )
+ )
+ (func $21 (; 34 ;) (type $8) (param $0 i32) (param $1 i32) (param $2 i32)
+  (local $3 i32)
+  (local $4 i64)
+  (local $5 f64)
+  (block $label$1
+   (if
+    (i32.le_u
+     (local.get $1)
+     (i32.const 20)
+    )
+    (block $label$3
+     (block $label$4
+      (block $label$5
+       (block $label$6
+        (block $label$7
+         (block $label$8
+          (block $label$9
+           (block $label$10
+            (block $label$11
+             (block $label$12
+              (block $label$13
+               (br_table $label$13 $label$12 $label$11 $label$10 $label$9 $label$8 $label$7 $label$6 $label$5 $label$4 $label$3
+                (i32.sub
+                 (local.get $1)
+                 (i32.const 9)
+                )
+               )
+              )
+              (local.set $3
+               (i32.load
+                (local.tee $1
+                 (i32.and
+                  (i32.add
+                   (i32.load
+                    (local.get $2)
+                   )
+                   (i32.const 3)
+                  )
+                  (i32.const -4)
+                 )
+                )
+               )
+              )
+              (i32.store
+               (local.get $2)
+               (i32.add
+                (local.get $1)
+                (i32.const 4)
+               )
+              )
+              (i32.store
+               (local.get $0)
+               (local.get $3)
+              )
+              (br $label$1)
+             )
+             (local.set $3
+              (i32.load
+               (local.tee $1
+                (i32.and
+                 (i32.add
+                  (i32.load
+                   (local.get $2)
+                  )
+                  (i32.const 3)
+                 )
+                 (i32.const -4)
+                )
+               )
+              )
+             )
+             (i32.store
+              (local.get $2)
+              (i32.add
+               (local.get $1)
+               (i32.const 4)
+              )
+             )
+             (i64.store
+              (local.get $0)
+              (i64.extend_i32_s
+               (local.get $3)
+              )
+             )
+             (br $label$1)
+            )
+            (local.set $3
+             (i32.load
+              (local.tee $1
+               (i32.and
+                (i32.add
+                 (i32.load
+                  (local.get $2)
+                 )
+                 (i32.const 3)
+                )
+                (i32.const -4)
+               )
+              )
+             )
+            )
+            (i32.store
+             (local.get $2)
+             (i32.add
+              (local.get $1)
+              (i32.const 4)
+             )
+            )
+            (i64.store
+             (local.get $0)
+             (i64.extend_i32_u
+              (local.get $3)
+             )
+            )
+            (br $label$1)
+           )
+           (local.set $4
+            (i64.load
+             (local.tee $1
+              (i32.and
+               (i32.add
+                (i32.load
+                 (local.get $2)
+                )
+                (i32.const 7)
+               )
+               (i32.const -8)
+              )
+             )
+            )
+           )
+           (i32.store
+            (local.get $2)
+            (i32.add
+             (local.get $1)
+             (i32.const 8)
+            )
+           )
+           (i64.store
+            (local.get $0)
+            (local.get $4)
+           )
+           (br $label$1)
+          )
+          (local.set $3
+           (i32.load
+            (local.tee $1
+             (i32.and
+              (i32.add
+               (i32.load
+                (local.get $2)
+               )
+               (i32.const 3)
+              )
+              (i32.const -4)
+             )
+            )
+           )
+          )
+          (i32.store
+           (local.get $2)
+           (i32.add
+            (local.get $1)
+            (i32.const 4)
+           )
+          )
+          (i64.store
+           (local.get $0)
+           (i64.extend_i32_s
+            (i32.shr_s
+             (i32.shl
+              (i32.and
+               (local.get $3)
+               (i32.const 65535)
+              )
+              (i32.const 16)
+             )
+             (i32.const 16)
+            )
+           )
+          )
+          (br $label$1)
+         )
+         (local.set $3
+          (i32.load
+           (local.tee $1
+            (i32.and
+             (i32.add
+              (i32.load
+               (local.get $2)
+              )
+              (i32.const 3)
+             )
+             (i32.const -4)
+            )
+           )
+          )
+         )
+         (i32.store
+          (local.get $2)
+          (i32.add
+           (local.get $1)
+           (i32.const 4)
+          )
+         )
+         (i64.store
+          (local.get $0)
+          (i64.extend_i32_u
+           (i32.and
+            (local.get $3)
+            (i32.const 65535)
+           )
+          )
+         )
+         (br $label$1)
+        )
+        (local.set $3
+         (i32.load
+          (local.tee $1
+           (i32.and
+            (i32.add
+             (i32.load
+              (local.get $2)
+             )
+             (i32.const 3)
+            )
+            (i32.const -4)
+           )
+          )
+         )
+        )
+        (i32.store
+         (local.get $2)
+         (i32.add
+          (local.get $1)
+          (i32.const 4)
+         )
+        )
+        (i64.store
+         (local.get $0)
+         (i64.extend_i32_s
+          (i32.shr_s
+           (i32.shl
+            (i32.and
+             (local.get $3)
+             (i32.const 255)
+            )
+            (i32.const 24)
+           )
+           (i32.const 24)
+          )
+         )
+        )
+        (br $label$1)
+       )
+       (local.set $3
+        (i32.load
+         (local.tee $1
+          (i32.and
+           (i32.add
+            (i32.load
+             (local.get $2)
+            )
+            (i32.const 3)
+           )
+           (i32.const -4)
+          )
+         )
+        )
+       )
+       (i32.store
+        (local.get $2)
+        (i32.add
+         (local.get $1)
+         (i32.const 4)
+        )
+       )
+       (i64.store
+        (local.get $0)
+        (i64.extend_i32_u
+         (i32.and
+          (local.get $3)
+          (i32.const 255)
+         )
+        )
+       )
+       (br $label$1)
+      )
+      (local.set $5
+       (f64.load
+        (local.tee $1
+         (i32.and
+          (i32.add
+           (i32.load
+            (local.get $2)
+           )
+           (i32.const 7)
+          )
+          (i32.const -8)
+         )
+        )
+       )
+      )
+      (i32.store
+       (local.get $2)
+       (i32.add
+        (local.get $1)
+        (i32.const 8)
+       )
+      )
+      (f64.store
+       (local.get $0)
+       (local.get $5)
+      )
+      (br $label$1)
+     )
+     (local.set $5
+      (f64.load
+       (local.tee $1
+        (i32.and
+         (i32.add
+          (i32.load
+           (local.get $2)
+          )
+          (i32.const 7)
+         )
+         (i32.const -8)
+        )
+       )
+      )
+     )
+     (i32.store
+      (local.get $2)
+      (i32.add
+       (local.get $1)
+       (i32.const 8)
+      )
+     )
+     (f64.store
+      (local.get $0)
+      (local.get $5)
+     )
+    )
+   )
+  )
+ )
+ (func $22 (; 35 ;) (type $9) (param $0 i64) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i64)
+  (block $label$1 (result i32)
+   (local.set $2
+    (i32.wrap_i64
+     (local.get $0)
+    )
+   )
+   (if
+    (i64.gt_u
+     (local.get $0)
+     (i64.const 4294967295)
+    )
+    (block
+     (loop $label$3
+      (i64.store8
+       (local.tee $1
+        (i32.add
+         (local.get $1)
+         (i32.const -1)
+        )
+       )
+       (i64.or
+        (i64.rem_u
+         (local.get $0)
+         (i64.const 10)
+        )
+        (i64.const 48)
+       )
+      )
+      (local.set $4
+       (i64.div_u
+        (local.get $0)
+        (i64.const 10)
+       )
+      )
+      (if
+       (i64.gt_u
+        (local.get $0)
+        (i64.const 42949672959)
+       )
+       (block
+        (local.set $0
+         (local.get $4)
+        )
+        (br $label$3)
+       )
+      )
+     )
+     (local.set $2
+      (i32.wrap_i64
+       (local.get $4)
+      )
+     )
+    )
+   )
+   (if
+    (local.get $2)
+    (loop $label$6
+     (i32.store8
+      (local.tee $1
+       (i32.add
+        (local.get $1)
+        (i32.const -1)
+       )
+      )
+      (i32.or
+       (i32.rem_u
+        (local.get $2)
+        (i32.const 10)
+       )
+       (i32.const 48)
+      )
+     )
+     (local.set $3
+      (i32.div_u
+       (local.get $2)
+       (i32.const 10)
+      )
+     )
+     (if
+      (i32.ge_u
+       (local.get $2)
+       (i32.const 10)
+      )
+      (block
+       (local.set $2
+        (local.get $3)
+       )
+       (br $label$6)
+      )
+     )
+    )
+   )
+   (local.get $1)
+  )
+ )
+ (func $23 (; 36 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (i32.const 0)
+   )
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (loop $label$5
+       (br_if $label$4
+        (i32.eq
+         (i32.load8_u
+          (i32.add
+           (local.get $1)
+           (i32.const 1702)
+          )
+         )
+         (local.get $0)
+        )
+       )
+       (br_if $label$5
+        (i32.ne
+         (local.tee $1
+          (i32.add
+           (local.get $1)
+           (i32.const 1)
+          )
+         )
+         (i32.const 87)
+        )
+       )
+       (local.set $1
+        (i32.const 87)
+       )
+       (local.set $0
+        (i32.const 1790)
+       )
+       (br $label$3)
+      )
+     )
+     (if
+      (local.get $1)
+      (block
+       (local.set $0
+        (i32.const 1790)
+       )
+       (br $label$3)
+      )
+      (local.set $0
+       (i32.const 1790)
+      )
+     )
+     (br $label$2)
+    )
+    (loop $label$8
+     (local.set $2
+      (local.get $0)
+     )
+     (loop $label$9
+      (local.set $0
+       (i32.add
+        (local.get $2)
+        (i32.const 1)
+       )
+      )
+      (if
+       (i32.load8_s
+        (local.get $2)
+       )
+       (block
+        (local.set $2
+         (local.get $0)
+        )
+        (br $label$9)
+       )
+      )
+     )
+     (br_if $label$8
+      (local.tee $1
+       (i32.add
+        (local.get $1)
+        (i32.const -1)
+       )
+      )
+     )
+    )
+   )
+   (local.get $0)
+  )
+ )
+ (func $24 (; 37 ;) (type $10) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (param $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (block $label$1
+   (local.set $7
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 256)
+    )
+   )
+   (local.set $6
+    (local.get $7)
+   )
+   (block $label$2
+    (if
+     (i32.and
+      (i32.gt_s
+       (local.get $2)
+       (local.get $3)
+      )
+      (i32.eqz
+       (i32.and
+        (local.get $4)
+        (i32.const 73728)
+       )
+      )
+     )
+     (block
+      (drop
+       (call $35
+        (local.get $6)
+        (local.get $1)
+        (if (result i32)
+         (i32.gt_u
+          (local.tee $5
+           (i32.sub
+            (local.get $2)
+            (local.get $3)
+           )
+          )
+          (i32.const 256)
+         )
+         (i32.const 256)
+         (local.get $5)
+        )
+       )
+      )
+      (local.set $4
+       (i32.eqz
+        (i32.and
+         (local.tee $1
+          (i32.load
+           (local.get $0)
+          )
+         )
+         (i32.const 32)
+        )
+       )
+      )
+      (if
+       (i32.gt_u
+        (local.get $5)
+        (i32.const 255)
+       )
+       (block
+        (loop $label$7
+         (if
+          (local.get $4)
+          (block
+           (drop
+            (call $20
+             (local.get $6)
+             (i32.const 256)
+             (local.get $0)
+            )
+           )
+           (local.set $1
+            (i32.load
+             (local.get $0)
+            )
+           )
+          )
+         )
+         (local.set $4
+          (i32.eqz
+           (i32.and
+            (local.get $1)
+            (i32.const 32)
+           )
+          )
+         )
+         (br_if $label$7
+          (i32.gt_u
+           (local.tee $5
+            (i32.add
+             (local.get $5)
+             (i32.const -256)
+            )
+           )
+           (i32.const 255)
+          )
+         )
+        )
+        (br_if $label$2
+         (i32.eqz
+          (local.get $4)
+         )
+        )
+        (local.set $5
+         (i32.and
+          (i32.sub
+           (local.get $2)
+           (local.get $3)
+          )
+          (i32.const 255)
+         )
+        )
+       )
+       (br_if $label$2
+        (i32.eqz
+         (local.get $4)
+        )
+       )
+      )
+      (drop
+       (call $20
+        (local.get $6)
+        (local.get $5)
+        (local.get $0)
+       )
+      )
+     )
+    )
+   )
+   (global.set $global$1
+    (local.get $7)
+   )
+  )
+ )
+ (func $25 (; 38 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (if (result i32)
+   (local.get $0)
+   (call $28
+    (local.get $0)
+    (local.get $1)
+    (i32.const 0)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $26 (; 39 ;) (type $11) (param $0 f64) (param $1 i32) (result f64)
+  (call $27
+   (local.get $0)
+   (local.get $1)
+  )
+ )
+ (func $27 (; 40 ;) (type $11) (param $0 f64) (param $1 i32) (result f64)
+  (local $2 i64)
+  (local $3 i64)
+  (block $label$1 (result f64)
+   (block $label$2
+    (block $label$3
+     (block $label$4
+      (block $label$5
+       (br_table $label$5 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$3 $label$4 $label$3
+        (i32.sub
+         (i32.shr_s
+          (i32.shl
+           (i32.and
+            (i32.and
+             (i32.wrap_i64
+              (local.tee $3
+               (i64.shr_u
+                (local.tee $2
+                 (i64.reinterpret_f64
+                  (local.get $0)
+                 )
+                )
+                (i64.const 52)
+               )
+              )
+             )
+             (i32.const 65535)
+            )
+            (i32.const 2047)
+           )
+           (i32.const 16)
+          )
+          (i32.const 16)
+         )
+         (i32.const 0)
+        )
+       )
+      )
+      (i32.store
+       (local.get $1)
+       (if (result i32)
+        (f64.ne
+         (local.get $0)
+         (f64.const 0)
+        )
+        (block (result i32)
+         (local.set $0
+          (call $27
+           (f64.mul
+            (local.get $0)
+            (f64.const 18446744073709551615)
+           )
+           (local.get $1)
+          )
+         )
+         (i32.add
+          (i32.load
+           (local.get $1)
+          )
+          (i32.const -64)
+         )
+        )
+        (i32.const 0)
+       )
+      )
+      (br $label$2)
+     )
+     (br $label$2)
+    )
+    (i32.store
+     (local.get $1)
+     (i32.add
+      (i32.and
+       (i32.wrap_i64
+        (local.get $3)
+       )
+       (i32.const 2047)
+      )
+      (i32.const -1022)
+     )
+    )
+    (local.set $0
+     (f64.reinterpret_i64
+      (i64.or
+       (i64.and
+        (local.get $2)
+        (i64.const -9218868437227405313)
+       )
+       (i64.const 4602678819172646912)
+      )
+     )
+    )
+   )
+   (local.get $0)
+  )
+ )
+ (func $28 (; 41 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (block $label$1 (result i32)
+   (if (result i32)
+    (local.get $0)
+    (block (result i32)
+     (if
+      (i32.lt_u
+       (local.get $1)
+       (i32.const 128)
+      )
+      (block
+       (i32.store8
+        (local.get $0)
+        (local.get $1)
+       )
+       (br $label$1
+        (i32.const 1)
+       )
+      )
+     )
+     (if
+      (i32.lt_u
+       (local.get $1)
+       (i32.const 2048)
+      )
+      (block
+       (i32.store8
+        (local.get $0)
+        (i32.or
+         (i32.shr_u
+          (local.get $1)
+          (i32.const 6)
+         )
+         (i32.const 192)
+        )
+       )
+       (i32.store8 offset=1
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (local.get $1)
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (br $label$1
+        (i32.const 2)
+       )
+      )
+     )
+     (if
+      (i32.or
+       (i32.lt_u
+        (local.get $1)
+        (i32.const 55296)
+       )
+       (i32.eq
+        (i32.and
+         (local.get $1)
+         (i32.const -8192)
+        )
+        (i32.const 57344)
+       )
+      )
+      (block
+       (i32.store8
+        (local.get $0)
+        (i32.or
+         (i32.shr_u
+          (local.get $1)
+          (i32.const 12)
+         )
+         (i32.const 224)
+        )
+       )
+       (i32.store8 offset=1
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (i32.shr_u
+           (local.get $1)
+           (i32.const 6)
+          )
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.store8 offset=2
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (local.get $1)
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (br $label$1
+        (i32.const 3)
+       )
+      )
+     )
+     (if (result i32)
+      (i32.lt_u
+       (i32.add
+        (local.get $1)
+        (i32.const -65536)
+       )
+       (i32.const 1048576)
+      )
+      (block (result i32)
+       (i32.store8
+        (local.get $0)
+        (i32.or
+         (i32.shr_u
+          (local.get $1)
+          (i32.const 18)
+         )
+         (i32.const 240)
+        )
+       )
+       (i32.store8 offset=1
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (i32.shr_u
+           (local.get $1)
+           (i32.const 12)
+          )
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.store8 offset=2
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (i32.shr_u
+           (local.get $1)
+           (i32.const 6)
+          )
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.store8 offset=3
+        (local.get $0)
+        (i32.or
+         (i32.and
+          (local.get $1)
+          (i32.const 63)
+         )
+         (i32.const 128)
+        )
+       )
+       (i32.const 4)
+      )
+      (block (result i32)
+       (i32.store
+        (call $11)
+        (i32.const 84)
+       )
+       (i32.const -1)
+      )
+     )
+    )
+    (i32.const 1)
+   )
+  )
+ )
+ (func $29 (; 42 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (i32.load8_s
+     (local.tee $2
+      (i32.add
+       (local.get $0)
+       (i32.const 74)
+      )
+     )
+    )
+   )
+   (i32.store8
+    (local.get $2)
+    (i32.or
+     (i32.add
+      (local.get $1)
+      (i32.const 255)
+     )
+     (local.get $1)
+    )
+   )
+   (local.tee $0
+    (if (result i32)
+     (i32.and
+      (local.tee $1
+       (i32.load
+        (local.get $0)
+       )
+      )
+      (i32.const 8)
+     )
+     (block (result i32)
+      (i32.store
+       (local.get $0)
+       (i32.or
+        (local.get $1)
+        (i32.const 32)
+       )
+      )
+      (i32.const -1)
+     )
+     (block (result i32)
+      (i32.store offset=8
+       (local.get $0)
+       (i32.const 0)
+      )
+      (i32.store offset=4
+       (local.get $0)
+       (i32.const 0)
+      )
+      (i32.store offset=28
+       (local.get $0)
+       (local.tee $1
+        (i32.load offset=44
+         (local.get $0)
+        )
+       )
+      )
+      (i32.store offset=20
+       (local.get $0)
+       (local.get $1)
+      )
+      (i32.store offset=16
+       (local.get $0)
+       (i32.add
+        (local.get $1)
+        (i32.load offset=48
+         (local.get $0)
+        )
+       )
+      )
+      (i32.const 0)
+     )
+    )
+   )
+  )
+ )
+ (func $30 (; 43 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (block $label$1 (result i32)
+   (local.set $2
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (i32.store
+    (local.tee $3
+     (local.get $2)
+    )
+    (local.get $1)
+   )
+   (local.set $0
+    (call $17
+     (i32.load
+      (i32.const 1024)
+     )
+     (local.get $0)
+     (local.get $3)
+    )
+   )
+   (global.set $global$1
+    (local.get $2)
+   )
+   (local.get $0)
+  )
+ )
+ (func $31 (; 44 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (local $16 i32)
+  (local $17 i32)
+  (local $18 i32)
+  (local $19 i32)
+  (local $20 i32)
+  (local $21 i32)
+  (block $label$1 (result i32)
+   (local.set $14
+    (global.get $global$1)
+   )
+   (global.set $global$1
+    (i32.add
+     (global.get $global$1)
+     (i32.const 16)
+    )
+   )
+   (local.set $18
+    (local.get $14)
+   )
+   (block $label$2
+    (if
+     (i32.lt_u
+      (local.get $0)
+      (i32.const 245)
+     )
+     (block
+      (local.set $3
+       (i32.and
+        (i32.add
+         (local.get $0)
+         (i32.const 11)
+        )
+        (i32.const -8)
+       )
+      )
+      (if
+       (i32.and
+        (local.tee $0
+         (i32.shr_u
+          (local.tee $8
+           (i32.load
+            (i32.const 3644)
+           )
+          )
+          (local.tee $2
+           (i32.shr_u
+            (if (result i32)
+             (i32.lt_u
+              (local.get $0)
+              (i32.const 11)
+             )
+             (local.tee $3
+              (i32.const 16)
+             )
+             (local.get $3)
+            )
+            (i32.const 3)
+           )
+          )
+         )
+        )
+        (i32.const 3)
+       )
+       (block
+        (local.set $4
+         (i32.load
+          (local.tee $1
+           (i32.add
+            (local.tee $7
+             (i32.load
+              (local.tee $3
+               (i32.add
+                (local.tee $2
+                 (i32.add
+                  (i32.shl
+                   (i32.shl
+                    (local.tee $5
+                     (i32.add
+                      (i32.xor
+                       (i32.and
+                        (local.get $0)
+                        (i32.const 1)
+                       )
+                       (i32.const 1)
+                      )
+                      (local.get $2)
+                     )
+                    )
+                    (i32.const 1)
+                   )
+                   (i32.const 2)
+                  )
+                  (i32.const 3684)
+                 )
+                )
+                (i32.const 8)
+               )
+              )
+             )
+            )
+            (i32.const 8)
+           )
+          )
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $2)
+          (local.get $4)
+         )
+         (i32.store
+          (i32.const 3644)
+          (i32.and
+           (local.get $8)
+           (i32.xor
+            (i32.shl
+             (i32.const 1)
+             (local.get $5)
+            )
+            (i32.const -1)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $4)
+            (i32.load
+             (i32.const 3660)
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $0
+              (i32.add
+               (local.get $4)
+               (i32.const 12)
+              )
+             )
+            )
+            (local.get $7)
+           )
+           (block
+            (i32.store
+             (local.get $0)
+             (local.get $2)
+            )
+            (i32.store
+             (local.get $3)
+             (local.get $4)
+            )
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (i32.store offset=4
+         (local.get $7)
+         (i32.or
+          (local.tee $0
+           (i32.shl
+            (local.get $5)
+            (i32.const 3)
+           )
+          )
+          (i32.const 3)
+         )
+        )
+        (i32.store
+         (local.tee $0
+          (i32.add
+           (i32.add
+            (local.get $7)
+            (local.get $0)
+           )
+           (i32.const 4)
+          )
+         )
+         (i32.or
+          (i32.load
+           (local.get $0)
+          )
+          (i32.const 1)
+         )
+        )
+        (global.set $global$1
+         (local.get $14)
+        )
+        (return
+         (local.get $1)
+        )
+       )
+      )
+      (if
+       (i32.gt_u
+        (local.get $3)
+        (local.tee $16
+         (i32.load
+          (i32.const 3652)
+         )
+        )
+       )
+       (block
+        (if
+         (local.get $0)
+         (block
+          (local.set $5
+           (i32.and
+            (i32.shr_u
+             (local.tee $0
+              (i32.add
+               (i32.and
+                (local.tee $0
+                 (i32.and
+                  (i32.shl
+                   (local.get $0)
+                   (local.get $2)
+                  )
+                  (i32.or
+                   (local.tee $0
+                    (i32.shl
+                     (i32.const 2)
+                     (local.get $2)
+                    )
+                   )
+                   (i32.sub
+                    (i32.const 0)
+                    (local.get $0)
+                   )
+                  )
+                 )
+                )
+                (i32.sub
+                 (i32.const 0)
+                 (local.get $0)
+                )
+               )
+               (i32.const -1)
+              )
+             )
+             (i32.const 12)
+            )
+            (i32.const 16)
+           )
+          )
+          (local.set $12
+           (i32.load
+            (local.tee $5
+             (i32.add
+              (local.tee $9
+               (i32.load
+                (local.tee $2
+                 (i32.add
+                  (local.tee $4
+                   (i32.add
+                    (i32.shl
+                     (i32.shl
+                      (local.tee $11
+                       (i32.add
+                        (i32.or
+                         (i32.or
+                          (i32.or
+                           (i32.or
+                            (local.tee $0
+                             (i32.and
+                              (i32.shr_u
+                               (local.tee $2
+                                (i32.shr_u
+                                 (local.get $0)
+                                 (local.get $5)
+                                )
+                               )
+                               (i32.const 5)
+                              )
+                              (i32.const 8)
+                             )
+                            )
+                            (local.get $5)
+                           )
+                           (local.tee $0
+                            (i32.and
+                             (i32.shr_u
+                              (local.tee $2
+                               (i32.shr_u
+                                (local.get $2)
+                                (local.get $0)
+                               )
+                              )
+                              (i32.const 2)
+                             )
+                             (i32.const 4)
+                            )
+                           )
+                          )
+                          (local.tee $0
+                           (i32.and
+                            (i32.shr_u
+                             (local.tee $2
+                              (i32.shr_u
+                               (local.get $2)
+                               (local.get $0)
+                              )
+                             )
+                             (i32.const 1)
+                            )
+                            (i32.const 2)
+                           )
+                          )
+                         )
+                         (local.tee $0
+                          (i32.and
+                           (i32.shr_u
+                            (local.tee $2
+                             (i32.shr_u
+                              (local.get $2)
+                              (local.get $0)
+                             )
+                            )
+                            (i32.const 1)
+                           )
+                           (i32.const 1)
+                          )
+                         )
+                        )
+                        (i32.shr_u
+                         (local.get $2)
+                         (local.get $0)
+                        )
+                       )
+                      )
+                      (i32.const 1)
+                     )
+                     (i32.const 2)
+                    )
+                    (i32.const 3684)
+                   )
+                  )
+                  (i32.const 8)
+                 )
+                )
+               )
+              )
+              (i32.const 8)
+             )
+            )
+           )
+          )
+          (if
+           (i32.eq
+            (local.get $4)
+            (local.get $12)
+           )
+           (i32.store
+            (i32.const 3644)
+            (local.tee $7
+             (i32.and
+              (local.get $8)
+              (i32.xor
+               (i32.shl
+                (i32.const 1)
+                (local.get $11)
+               )
+               (i32.const -1)
+              )
+             )
+            )
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.get $12)
+              (i32.load
+               (i32.const 3660)
+              )
+             )
+             (call $fimport$10)
+            )
+            (if
+             (i32.eq
+              (i32.load
+               (local.tee $0
+                (i32.add
+                 (local.get $12)
+                 (i32.const 12)
+                )
+               )
+              )
+              (local.get $9)
+             )
+             (block
+              (i32.store
+               (local.get $0)
+               (local.get $4)
+              )
+              (i32.store
+               (local.get $2)
+               (local.get $12)
+              )
+              (local.set $7
+               (local.get $8)
+              )
+             )
+             (call $fimport$10)
+            )
+           )
+          )
+          (i32.store offset=4
+           (local.get $9)
+           (i32.or
+            (local.get $3)
+            (i32.const 3)
+           )
+          )
+          (i32.store offset=4
+           (local.tee $4
+            (i32.add
+             (local.get $9)
+             (local.get $3)
+            )
+           )
+           (i32.or
+            (local.tee $11
+             (i32.sub
+              (i32.shl
+               (local.get $11)
+               (i32.const 3)
+              )
+              (local.get $3)
+             )
+            )
+            (i32.const 1)
+           )
+          )
+          (i32.store
+           (i32.add
+            (local.get $4)
+            (local.get $11)
+           )
+           (local.get $11)
+          )
+          (if
+           (local.get $16)
+           (block
+            (local.set $9
+             (i32.load
+              (i32.const 3664)
+             )
+            )
+            (local.set $2
+             (i32.add
+              (i32.shl
+               (i32.shl
+                (local.tee $0
+                 (i32.shr_u
+                  (local.get $16)
+                  (i32.const 3)
+                 )
+                )
+                (i32.const 1)
+               )
+               (i32.const 2)
+              )
+              (i32.const 3684)
+             )
+            )
+            (if
+             (i32.and
+              (local.get $7)
+              (local.tee $0
+               (i32.shl
+                (i32.const 1)
+                (local.get $0)
+               )
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.tee $0
+                (i32.load
+                 (local.tee $3
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 8)
+                  )
+                 )
+                )
+               )
+               (i32.load
+                (i32.const 3660)
+               )
+              )
+              (call $fimport$10)
+              (block
+               (local.set $6
+                (local.get $3)
+               )
+               (local.set $1
+                (local.get $0)
+               )
+              )
+             )
+             (block
+              (i32.store
+               (i32.const 3644)
+               (i32.or
+                (local.get $7)
+                (local.get $0)
+               )
+              )
+              (local.set $6
+               (i32.add
+                (local.get $2)
+                (i32.const 8)
+               )
+              )
+              (local.set $1
+               (local.get $2)
+              )
+             )
+            )
+            (i32.store
+             (local.get $6)
+             (local.get $9)
+            )
+            (i32.store offset=12
+             (local.get $1)
+             (local.get $9)
+            )
+            (i32.store offset=8
+             (local.get $9)
+             (local.get $1)
+            )
+            (i32.store offset=12
+             (local.get $9)
+             (local.get $2)
+            )
+           )
+          )
+          (i32.store
+           (i32.const 3652)
+           (local.get $11)
+          )
+          (i32.store
+           (i32.const 3664)
+           (local.get $4)
+          )
+          (global.set $global$1
+           (local.get $14)
+          )
+          (return
+           (local.get $5)
+          )
+         )
+        )
+        (if
+         (local.tee $6
+          (i32.load
+           (i32.const 3648)
+          )
+         )
+         (block
+          (local.set $2
+           (i32.and
+            (i32.shr_u
+             (local.tee $0
+              (i32.add
+               (i32.and
+                (local.get $6)
+                (i32.sub
+                 (i32.const 0)
+                 (local.get $6)
+                )
+               )
+               (i32.const -1)
+              )
+             )
+             (i32.const 12)
+            )
+            (i32.const 16)
+           )
+          )
+          (local.set $9
+           (i32.sub
+            (i32.and
+             (i32.load offset=4
+              (local.tee $2
+               (i32.load
+                (i32.add
+                 (i32.shl
+                  (i32.add
+                   (i32.or
+                    (i32.or
+                     (i32.or
+                      (i32.or
+                       (local.tee $0
+                        (i32.and
+                         (i32.shr_u
+                          (local.tee $1
+                           (i32.shr_u
+                            (local.get $0)
+                            (local.get $2)
+                           )
+                          )
+                          (i32.const 5)
+                         )
+                         (i32.const 8)
+                        )
+                       )
+                       (local.get $2)
+                      )
+                      (local.tee $0
+                       (i32.and
+                        (i32.shr_u
+                         (local.tee $1
+                          (i32.shr_u
+                           (local.get $1)
+                           (local.get $0)
+                          )
+                         )
+                         (i32.const 2)
+                        )
+                        (i32.const 4)
+                       )
+                      )
+                     )
+                     (local.tee $0
+                      (i32.and
+                       (i32.shr_u
+                        (local.tee $1
+                         (i32.shr_u
+                          (local.get $1)
+                          (local.get $0)
+                         )
+                        )
+                        (i32.const 1)
+                       )
+                       (i32.const 2)
+                      )
+                     )
+                    )
+                    (local.tee $0
+                     (i32.and
+                      (i32.shr_u
+                       (local.tee $1
+                        (i32.shr_u
+                         (local.get $1)
+                         (local.get $0)
+                        )
+                       )
+                       (i32.const 1)
+                      )
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                   (i32.shr_u
+                    (local.get $1)
+                    (local.get $0)
+                   )
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 3948)
+                )
+               )
+              )
+             )
+             (i32.const -8)
+            )
+            (local.get $3)
+           )
+          )
+          (local.set $1
+           (local.get $2)
+          )
+          (loop $label$25
+           (block $label$26
+            (if
+             (i32.eqz
+              (local.tee $0
+               (i32.load offset=16
+                (local.get $1)
+               )
+              )
+             )
+             (br_if $label$26
+              (i32.eqz
+               (local.tee $0
+                (i32.load offset=20
+                 (local.get $1)
+                )
+               )
+              )
+             )
+            )
+            (if
+             (local.tee $7
+              (i32.lt_u
+               (local.tee $1
+                (i32.sub
+                 (i32.and
+                  (i32.load offset=4
+                   (local.get $0)
+                  )
+                  (i32.const -8)
+                 )
+                 (local.get $3)
+                )
+               )
+               (local.get $9)
+              )
+             )
+             (local.set $9
+              (local.get $1)
+             )
+            )
+            (local.set $1
+             (local.get $0)
+            )
+            (if
+             (local.get $7)
+             (local.set $2
+              (local.get $0)
+             )
+            )
+            (br $label$25)
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.get $2)
+            (local.tee $12
+             (i32.load
+              (i32.const 3660)
+             )
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.ge_u
+            (local.get $2)
+            (local.tee $13
+             (i32.add
+              (local.get $2)
+              (local.get $3)
+             )
+            )
+           )
+           (call $fimport$10)
+          )
+          (local.set $15
+           (i32.load offset=24
+            (local.get $2)
+           )
+          )
+          (block $label$32
+           (if
+            (i32.eq
+             (local.tee $0
+              (i32.load offset=12
+               (local.get $2)
+              )
+             )
+             (local.get $2)
+            )
+            (block
+             (if
+              (i32.eqz
+               (local.tee $0
+                (i32.load
+                 (local.tee $1
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 20)
+                  )
+                 )
+                )
+               )
+              )
+              (if
+               (i32.eqz
+                (local.tee $0
+                 (i32.load
+                  (local.tee $1
+                   (i32.add
+                    (local.get $2)
+                    (i32.const 16)
+                   )
+                  )
+                 )
+                )
+               )
+               (block
+                (local.set $4
+                 (i32.const 0)
+                )
+                (br $label$32)
+               )
+              )
+             )
+             (loop $label$36
+              (if
+               (local.tee $7
+                (i32.load
+                 (local.tee $11
+                  (i32.add
+                   (local.get $0)
+                   (i32.const 20)
+                  )
+                 )
+                )
+               )
+               (block
+                (local.set $0
+                 (local.get $7)
+                )
+                (local.set $1
+                 (local.get $11)
+                )
+                (br $label$36)
+               )
+              )
+              (if
+               (local.tee $7
+                (i32.load
+                 (local.tee $11
+                  (i32.add
+                   (local.get $0)
+                   (i32.const 16)
+                  )
+                 )
+                )
+               )
+               (block
+                (local.set $0
+                 (local.get $7)
+                )
+                (local.set $1
+                 (local.get $11)
+                )
+                (br $label$36)
+               )
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.get $1)
+               (local.get $12)
+              )
+              (call $fimport$10)
+              (block
+               (i32.store
+                (local.get $1)
+                (i32.const 0)
+               )
+               (local.set $4
+                (local.get $0)
+               )
+              )
+             )
+            )
+            (block
+             (if
+              (i32.lt_u
+               (local.tee $11
+                (i32.load offset=8
+                 (local.get $2)
+                )
+               )
+               (local.get $12)
+              )
+              (call $fimport$10)
+             )
+             (if
+              (i32.ne
+               (i32.load
+                (local.tee $7
+                 (i32.add
+                  (local.get $11)
+                  (i32.const 12)
+                 )
+                )
+               )
+               (local.get $2)
+              )
+              (call $fimport$10)
+             )
+             (if
+              (i32.eq
+               (i32.load
+                (local.tee $1
+                 (i32.add
+                  (local.get $0)
+                  (i32.const 8)
+                 )
+                )
+               )
+               (local.get $2)
+              )
+              (block
+               (i32.store
+                (local.get $7)
+                (local.get $0)
+               )
+               (i32.store
+                (local.get $1)
+                (local.get $11)
+               )
+               (local.set $4
+                (local.get $0)
+               )
+              )
+              (call $fimport$10)
+             )
+            )
+           )
+          )
+          (block $label$46
+           (if
+            (local.get $15)
+            (block
+             (if
+              (i32.eq
+               (local.get $2)
+               (i32.load
+                (local.tee $0
+                 (i32.add
+                  (i32.shl
+                   (local.tee $1
+                    (i32.load offset=28
+                     (local.get $2)
+                    )
+                   )
+                   (i32.const 2)
+                  )
+                  (i32.const 3948)
+                 )
+                )
+               )
+              )
+              (block
+               (i32.store
+                (local.get $0)
+                (local.get $4)
+               )
+               (if
+                (i32.eqz
+                 (local.get $4)
+                )
+                (block
+                 (i32.store
+                  (i32.const 3648)
+                  (i32.and
+                   (local.get $6)
+                   (i32.xor
+                    (i32.shl
+                     (i32.const 1)
+                     (local.get $1)
+                    )
+                    (i32.const -1)
+                   )
+                  )
+                 )
+                 (br $label$46)
+                )
+               )
+              )
+              (block
+               (if
+                (i32.lt_u
+                 (local.get $15)
+                 (i32.load
+                  (i32.const 3660)
+                 )
+                )
+                (call $fimport$10)
+               )
+               (if
+                (i32.eq
+                 (i32.load
+                  (local.tee $0
+                   (i32.add
+                    (local.get $15)
+                    (i32.const 16)
+                   )
+                  )
+                 )
+                 (local.get $2)
+                )
+                (i32.store
+                 (local.get $0)
+                 (local.get $4)
+                )
+                (i32.store offset=20
+                 (local.get $15)
+                 (local.get $4)
+                )
+               )
+               (br_if $label$46
+                (i32.eqz
+                 (local.get $4)
+                )
+               )
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.get $4)
+               (local.tee $0
+                (i32.load
+                 (i32.const 3660)
+                )
+               )
+              )
+              (call $fimport$10)
+             )
+             (i32.store offset=24
+              (local.get $4)
+              (local.get $15)
+             )
+             (if
+              (local.tee $1
+               (i32.load offset=16
+                (local.get $2)
+               )
+              )
+              (if
+               (i32.lt_u
+                (local.get $1)
+                (local.get $0)
+               )
+               (call $fimport$10)
+               (block
+                (i32.store offset=16
+                 (local.get $4)
+                 (local.get $1)
+                )
+                (i32.store offset=24
+                 (local.get $1)
+                 (local.get $4)
+                )
+               )
+              )
+             )
+             (if
+              (local.tee $0
+               (i32.load offset=20
+                (local.get $2)
+               )
+              )
+              (if
+               (i32.lt_u
+                (local.get $0)
+                (i32.load
+                 (i32.const 3660)
+                )
+               )
+               (call $fimport$10)
+               (block
+                (i32.store offset=20
+                 (local.get $4)
+                 (local.get $0)
+                )
+                (i32.store offset=24
+                 (local.get $0)
+                 (local.get $4)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.get $9)
+            (i32.const 16)
+           )
+           (block
+            (i32.store offset=4
+             (local.get $2)
+             (i32.or
+              (local.tee $0
+               (i32.add
+                (local.get $9)
+                (local.get $3)
+               )
+              )
+              (i32.const 3)
+             )
+            )
+            (i32.store
+             (local.tee $0
+              (i32.add
+               (i32.add
+                (local.get $2)
+                (local.get $0)
+               )
+               (i32.const 4)
+              )
+             )
+             (i32.or
+              (i32.load
+               (local.get $0)
+              )
+              (i32.const 1)
+             )
+            )
+           )
+           (block
+            (i32.store offset=4
+             (local.get $2)
+             (i32.or
+              (local.get $3)
+              (i32.const 3)
+             )
+            )
+            (i32.store offset=4
+             (local.get $13)
+             (i32.or
+              (local.get $9)
+              (i32.const 1)
+             )
+            )
+            (i32.store
+             (i32.add
+              (local.get $13)
+              (local.get $9)
+             )
+             (local.get $9)
+            )
+            (if
+             (local.get $16)
+             (block
+              (local.set $7
+               (i32.load
+                (i32.const 3664)
+               )
+              )
+              (local.set $3
+               (i32.add
+                (i32.shl
+                 (i32.shl
+                  (local.tee $0
+                   (i32.shr_u
+                    (local.get $16)
+                    (i32.const 3)
+                   )
+                  )
+                  (i32.const 1)
+                 )
+                 (i32.const 2)
+                )
+                (i32.const 3684)
+               )
+              )
+              (if
+               (i32.and
+                (local.get $8)
+                (local.tee $0
+                 (i32.shl
+                  (i32.const 1)
+                  (local.get $0)
+                 )
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.tee $0
+                  (i32.load
+                   (local.tee $1
+                    (i32.add
+                     (local.get $3)
+                     (i32.const 8)
+                    )
+                   )
+                  )
+                 )
+                 (i32.load
+                  (i32.const 3660)
+                 )
+                )
+                (call $fimport$10)
+                (block
+                 (local.set $10
+                  (local.get $1)
+                 )
+                 (local.set $5
+                  (local.get $0)
+                 )
+                )
+               )
+               (block
+                (i32.store
+                 (i32.const 3644)
+                 (i32.or
+                  (local.get $8)
+                  (local.get $0)
+                 )
+                )
+                (local.set $10
+                 (i32.add
+                  (local.get $3)
+                  (i32.const 8)
+                 )
+                )
+                (local.set $5
+                 (local.get $3)
+                )
+               )
+              )
+              (i32.store
+               (local.get $10)
+               (local.get $7)
+              )
+              (i32.store offset=12
+               (local.get $5)
+               (local.get $7)
+              )
+              (i32.store offset=8
+               (local.get $7)
+               (local.get $5)
+              )
+              (i32.store offset=12
+               (local.get $7)
+               (local.get $3)
+              )
+             )
+            )
+            (i32.store
+             (i32.const 3652)
+             (local.get $9)
+            )
+            (i32.store
+             (i32.const 3664)
+             (local.get $13)
+            )
+           )
+          )
+          (global.set $global$1
+           (local.get $14)
+          )
+          (return
+           (i32.add
+            (local.get $2)
+            (i32.const 8)
+           )
+          )
+         )
+         (local.set $0
+          (local.get $3)
+         )
+        )
+       )
+       (local.set $0
+        (local.get $3)
+       )
+      )
+     )
+     (if
+      (i32.gt_u
+       (local.get $0)
+       (i32.const -65)
+      )
+      (local.set $0
+       (i32.const -1)
+      )
+      (block
+       (local.set $7
+        (i32.and
+         (local.tee $0
+          (i32.add
+           (local.get $0)
+           (i32.const 11)
+          )
+         )
+         (i32.const -8)
+        )
+       )
+       (if
+        (local.tee $5
+         (i32.load
+          (i32.const 3648)
+         )
+        )
+        (block
+         (local.set $17
+          (if (result i32)
+           (local.tee $0
+            (i32.shr_u
+             (local.get $0)
+             (i32.const 8)
+            )
+           )
+           (if (result i32)
+            (i32.gt_u
+             (local.get $7)
+             (i32.const 16777215)
+            )
+            (i32.const 31)
+            (i32.or
+             (i32.and
+              (i32.shr_u
+               (local.get $7)
+               (i32.add
+                (local.tee $0
+                 (i32.add
+                  (i32.sub
+                   (i32.const 14)
+                   (i32.or
+                    (i32.or
+                     (local.tee $0
+                      (i32.and
+                       (i32.shr_u
+                        (i32.add
+                         (local.tee $1
+                          (i32.shl
+                           (local.get $0)
+                           (local.tee $3
+                            (i32.and
+                             (i32.shr_u
+                              (i32.add
+                               (local.get $0)
+                               (i32.const 1048320)
+                              )
+                              (i32.const 16)
+                             )
+                             (i32.const 8)
+                            )
+                           )
+                          )
+                         )
+                         (i32.const 520192)
+                        )
+                        (i32.const 16)
+                       )
+                       (i32.const 4)
+                      )
+                     )
+                     (local.get $3)
+                    )
+                    (local.tee $0
+                     (i32.and
+                      (i32.shr_u
+                       (i32.add
+                        (local.tee $1
+                         (i32.shl
+                          (local.get $1)
+                          (local.get $0)
+                         )
+                        )
+                        (i32.const 245760)
+                       )
+                       (i32.const 16)
+                      )
+                      (i32.const 2)
+                     )
+                    )
+                   )
+                  )
+                  (i32.shr_u
+                   (i32.shl
+                    (local.get $1)
+                    (local.get $0)
+                   )
+                   (i32.const 15)
+                  )
+                 )
+                )
+                (i32.const 7)
+               )
+              )
+              (i32.const 1)
+             )
+             (i32.shl
+              (local.get $0)
+              (i32.const 1)
+             )
+            )
+           )
+           (i32.const 0)
+          )
+         )
+         (local.set $3
+          (i32.sub
+           (i32.const 0)
+           (local.get $7)
+          )
+         )
+         (block $label$78
+          (block $label$79
+           (block $label$80
+            (if
+             (local.tee $1
+              (i32.load
+               (i32.add
+                (i32.shl
+                 (local.get $17)
+                 (i32.const 2)
+                )
+                (i32.const 3948)
+               )
+              )
+             )
+             (block
+              (local.set $0
+               (i32.sub
+                (i32.const 25)
+                (i32.shr_u
+                 (local.get $17)
+                 (i32.const 1)
+                )
+               )
+              )
+              (local.set $4
+               (i32.const 0)
+              )
+              (local.set $10
+               (i32.shl
+                (local.get $7)
+                (if (result i32)
+                 (i32.eq
+                  (local.get $17)
+                  (i32.const 31)
+                 )
+                 (i32.const 0)
+                 (local.get $0)
+                )
+               )
+              )
+              (local.set $0
+               (i32.const 0)
+              )
+              (loop $label$84
+               (if
+                (i32.lt_u
+                 (local.tee $6
+                  (i32.sub
+                   (i32.and
+                    (i32.load offset=4
+                     (local.get $1)
+                    )
+                    (i32.const -8)
+                   )
+                   (local.get $7)
+                  )
+                 )
+                 (local.get $3)
+                )
+                (if
+                 (local.get $6)
+                 (block
+                  (local.set $3
+                   (local.get $6)
+                  )
+                  (local.set $0
+                   (local.get $1)
+                  )
+                 )
+                 (block
+                  (local.set $3
+                   (i32.const 0)
+                  )
+                  (local.set $0
+                   (local.get $1)
+                  )
+                  (br $label$79)
+                 )
+                )
+               )
+               (local.set $1
+                (if (result i32)
+                 (i32.or
+                  (i32.eqz
+                   (local.tee $19
+                    (i32.load offset=20
+                     (local.get $1)
+                    )
+                   )
+                  )
+                  (i32.eq
+                   (local.get $19)
+                   (local.tee $6
+                    (i32.load
+                     (i32.add
+                      (i32.add
+                       (local.get $1)
+                       (i32.const 16)
+                      )
+                      (i32.shl
+                       (i32.shr_u
+                        (local.get $10)
+                        (i32.const 31)
+                       )
+                       (i32.const 2)
+                      )
+                     )
+                    )
+                   )
+                  )
+                 )
+                 (local.get $4)
+                 (local.get $19)
+                )
+               )
+               (local.set $10
+                (i32.shl
+                 (local.get $10)
+                 (i32.xor
+                  (i32.and
+                   (local.tee $4
+                    (i32.eqz
+                     (local.get $6)
+                    )
+                   )
+                   (i32.const 1)
+                  )
+                  (i32.const 1)
+                 )
+                )
+               )
+               (if
+                (local.get $4)
+                (block
+                 (local.set $4
+                  (local.get $1)
+                 )
+                 (local.set $1
+                  (local.get $0)
+                 )
+                 (br $label$80)
+                )
+                (block
+                 (local.set $4
+                  (local.get $1)
+                 )
+                 (local.set $1
+                  (local.get $6)
+                 )
+                 (br $label$84)
+                )
+               )
+              )
+             )
+             (block
+              (local.set $4
+               (i32.const 0)
+              )
+              (local.set $1
+               (i32.const 0)
+              )
+             )
+            )
+           )
+           (br_if $label$79
+            (local.tee $0
+             (if (result i32)
+              (i32.and
+               (i32.eqz
+                (local.get $4)
+               )
+               (i32.eqz
+                (local.get $1)
+               )
+              )
+              (block (result i32)
+               (if
+                (i32.eqz
+                 (local.tee $0
+                  (i32.and
+                   (local.get $5)
+                   (i32.or
+                    (local.tee $0
+                     (i32.shl
+                      (i32.const 2)
+                      (local.get $17)
+                     )
+                    )
+                    (i32.sub
+                     (i32.const 0)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                 )
+                )
+                (block
+                 (local.set $0
+                  (local.get $7)
+                 )
+                 (br $label$2)
+                )
+               )
+               (local.set $10
+                (i32.and
+                 (i32.shr_u
+                  (local.tee $0
+                   (i32.add
+                    (i32.and
+                     (local.get $0)
+                     (i32.sub
+                      (i32.const 0)
+                      (local.get $0)
+                     )
+                    )
+                    (i32.const -1)
+                   )
+                  )
+                  (i32.const 12)
+                 )
+                 (i32.const 16)
+                )
+               )
+               (i32.load
+                (i32.add
+                 (i32.shl
+                  (i32.add
+                   (i32.or
+                    (i32.or
+                     (i32.or
+                      (i32.or
+                       (local.tee $0
+                        (i32.and
+                         (i32.shr_u
+                          (local.tee $4
+                           (i32.shr_u
+                            (local.get $0)
+                            (local.get $10)
+                           )
+                          )
+                          (i32.const 5)
+                         )
+                         (i32.const 8)
+                        )
+                       )
+                       (local.get $10)
+                      )
+                      (local.tee $0
+                       (i32.and
+                        (i32.shr_u
+                         (local.tee $4
+                          (i32.shr_u
+                           (local.get $4)
+                           (local.get $0)
+                          )
+                         )
+                         (i32.const 2)
+                        )
+                        (i32.const 4)
+                       )
+                      )
+                     )
+                     (local.tee $0
+                      (i32.and
+                       (i32.shr_u
+                        (local.tee $4
+                         (i32.shr_u
+                          (local.get $4)
+                          (local.get $0)
+                         )
+                        )
+                        (i32.const 1)
+                       )
+                       (i32.const 2)
+                      )
+                     )
+                    )
+                    (local.tee $0
+                     (i32.and
+                      (i32.shr_u
+                       (local.tee $4
+                        (i32.shr_u
+                         (local.get $4)
+                         (local.get $0)
+                        )
+                       )
+                       (i32.const 1)
+                      )
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                   (i32.shr_u
+                    (local.get $4)
+                    (local.get $0)
+                   )
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 3948)
+                )
+               )
+              )
+              (local.get $4)
+             )
+            )
+           )
+           (local.set $4
+            (local.get $1)
+           )
+           (br $label$78)
+          )
+          (loop $label$96
+           (if
+            (local.tee $10
+             (i32.lt_u
+              (local.tee $4
+               (i32.sub
+                (i32.and
+                 (i32.load offset=4
+                  (local.get $0)
+                 )
+                 (i32.const -8)
+                )
+                (local.get $7)
+               )
+              )
+              (local.get $3)
+             )
+            )
+            (local.set $3
+             (local.get $4)
+            )
+           )
+           (if
+            (local.get $10)
+            (local.set $1
+             (local.get $0)
+            )
+           )
+           (if
+            (local.tee $4
+             (i32.load offset=16
+              (local.get $0)
+             )
+            )
+            (block
+             (local.set $0
+              (local.get $4)
+             )
+             (br $label$96)
+            )
+           )
+           (br_if $label$96
+            (local.tee $0
+             (i32.load offset=20
+              (local.get $0)
+             )
+            )
+           )
+           (local.set $4
+            (local.get $1)
+           )
+          )
+         )
+         (if
+          (local.get $4)
+          (if
+           (i32.lt_u
+            (local.get $3)
+            (i32.sub
+             (i32.load
+              (i32.const 3652)
+             )
+             (local.get $7)
+            )
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.get $4)
+              (local.tee $12
+               (i32.load
+                (i32.const 3660)
+               )
+              )
+             )
+             (call $fimport$10)
+            )
+            (if
+             (i32.ge_u
+              (local.get $4)
+              (local.tee $6
+               (i32.add
+                (local.get $4)
+                (local.get $7)
+               )
+              )
+             )
+             (call $fimport$10)
+            )
+            (local.set $10
+             (i32.load offset=24
+              (local.get $4)
+             )
+            )
+            (block $label$104
+             (if
+              (i32.eq
+               (local.tee $0
+                (i32.load offset=12
+                 (local.get $4)
+                )
+               )
+               (local.get $4)
+              )
+              (block
+               (if
+                (i32.eqz
+                 (local.tee $0
+                  (i32.load
+                   (local.tee $1
+                    (i32.add
+                     (local.get $4)
+                     (i32.const 20)
+                    )
+                   )
+                  )
+                 )
+                )
+                (if
+                 (i32.eqz
+                  (local.tee $0
+                   (i32.load
+                    (local.tee $1
+                     (i32.add
+                      (local.get $4)
+                      (i32.const 16)
+                     )
+                    )
+                   )
+                  )
+                 )
+                 (block
+                  (local.set $13
+                   (i32.const 0)
+                  )
+                  (br $label$104)
+                 )
+                )
+               )
+               (loop $label$108
+                (if
+                 (local.tee $11
+                  (i32.load
+                   (local.tee $9
+                    (i32.add
+                     (local.get $0)
+                     (i32.const 20)
+                    )
+                   )
+                  )
+                 )
+                 (block
+                  (local.set $0
+                   (local.get $11)
+                  )
+                  (local.set $1
+                   (local.get $9)
+                  )
+                  (br $label$108)
+                 )
+                )
+                (if
+                 (local.tee $11
+                  (i32.load
+                   (local.tee $9
+                    (i32.add
+                     (local.get $0)
+                     (i32.const 16)
+                    )
+                   )
+                  )
+                 )
+                 (block
+                  (local.set $0
+                   (local.get $11)
+                  )
+                  (local.set $1
+                   (local.get $9)
+                  )
+                  (br $label$108)
+                 )
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $1)
+                 (local.get $12)
+                )
+                (call $fimport$10)
+                (block
+                 (i32.store
+                  (local.get $1)
+                  (i32.const 0)
+                 )
+                 (local.set $13
+                  (local.get $0)
+                 )
+                )
+               )
+              )
+              (block
+               (if
+                (i32.lt_u
+                 (local.tee $9
+                  (i32.load offset=8
+                   (local.get $4)
+                  )
+                 )
+                 (local.get $12)
+                )
+                (call $fimport$10)
+               )
+               (if
+                (i32.ne
+                 (i32.load
+                  (local.tee $11
+                   (i32.add
+                    (local.get $9)
+                    (i32.const 12)
+                   )
+                  )
+                 )
+                 (local.get $4)
+                )
+                (call $fimport$10)
+               )
+               (if
+                (i32.eq
+                 (i32.load
+                  (local.tee $1
+                   (i32.add
+                    (local.get $0)
+                    (i32.const 8)
+                   )
+                  )
+                 )
+                 (local.get $4)
+                )
+                (block
+                 (i32.store
+                  (local.get $11)
+                  (local.get $0)
+                 )
+                 (i32.store
+                  (local.get $1)
+                  (local.get $9)
+                 )
+                 (local.set $13
+                  (local.get $0)
+                 )
+                )
+                (call $fimport$10)
+               )
+              )
+             )
+            )
+            (block $label$118
+             (if
+              (local.get $10)
+              (block
+               (if
+                (i32.eq
+                 (local.get $4)
+                 (i32.load
+                  (local.tee $0
+                   (i32.add
+                    (i32.shl
+                     (local.tee $1
+                      (i32.load offset=28
+                       (local.get $4)
+                      )
+                     )
+                     (i32.const 2)
+                    )
+                    (i32.const 3948)
+                   )
+                  )
+                 )
+                )
+                (block
+                 (i32.store
+                  (local.get $0)
+                  (local.get $13)
+                 )
+                 (if
+                  (i32.eqz
+                   (local.get $13)
+                  )
+                  (block
+                   (i32.store
+                    (i32.const 3648)
+                    (local.tee $2
+                     (i32.and
+                      (local.get $5)
+                      (i32.xor
+                       (i32.shl
+                        (i32.const 1)
+                        (local.get $1)
+                       )
+                       (i32.const -1)
+                      )
+                     )
+                    )
+                   )
+                   (br $label$118)
+                  )
+                 )
+                )
+                (block
+                 (if
+                  (i32.lt_u
+                   (local.get $10)
+                   (i32.load
+                    (i32.const 3660)
+                   )
+                  )
+                  (call $fimport$10)
+                 )
+                 (if
+                  (i32.eq
+                   (i32.load
+                    (local.tee $0
+                     (i32.add
+                      (local.get $10)
+                      (i32.const 16)
+                     )
+                    )
+                   )
+                   (local.get $4)
+                  )
+                  (i32.store
+                   (local.get $0)
+                   (local.get $13)
+                  )
+                  (i32.store offset=20
+                   (local.get $10)
+                   (local.get $13)
+                  )
+                 )
+                 (if
+                  (i32.eqz
+                   (local.get $13)
+                  )
+                  (block
+                   (local.set $2
+                    (local.get $5)
+                   )
+                   (br $label$118)
+                  )
+                 )
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $13)
+                 (local.tee $0
+                  (i32.load
+                   (i32.const 3660)
+                  )
+                 )
+                )
+                (call $fimport$10)
+               )
+               (i32.store offset=24
+                (local.get $13)
+                (local.get $10)
+               )
+               (if
+                (local.tee $1
+                 (i32.load offset=16
+                  (local.get $4)
+                 )
+                )
+                (if
+                 (i32.lt_u
+                  (local.get $1)
+                  (local.get $0)
+                 )
+                 (call $fimport$10)
+                 (block
+                  (i32.store offset=16
+                   (local.get $13)
+                   (local.get $1)
+                  )
+                  (i32.store offset=24
+                   (local.get $1)
+                   (local.get $13)
+                  )
+                 )
+                )
+               )
+               (if
+                (local.tee $0
+                 (i32.load offset=20
+                  (local.get $4)
+                 )
+                )
+                (if
+                 (i32.lt_u
+                  (local.get $0)
+                  (i32.load
+                   (i32.const 3660)
+                  )
+                 )
+                 (call $fimport$10)
+                 (block
+                  (i32.store offset=20
+                   (local.get $13)
+                   (local.get $0)
+                  )
+                  (i32.store offset=24
+                   (local.get $0)
+                   (local.get $13)
+                  )
+                  (local.set $2
+                   (local.get $5)
+                  )
+                 )
+                )
+                (local.set $2
+                 (local.get $5)
+                )
+               )
+              )
+              (local.set $2
+               (local.get $5)
+              )
+             )
+            )
+            (block $label$136
+             (if
+              (i32.lt_u
+               (local.get $3)
+               (i32.const 16)
+              )
+              (block
+               (i32.store offset=4
+                (local.get $4)
+                (i32.or
+                 (local.tee $0
+                  (i32.add
+                   (local.get $3)
+                   (local.get $7)
+                  )
+                 )
+                 (i32.const 3)
+                )
+               )
+               (i32.store
+                (local.tee $0
+                 (i32.add
+                  (i32.add
+                   (local.get $4)
+                   (local.get $0)
+                  )
+                  (i32.const 4)
+                 )
+                )
+                (i32.or
+                 (i32.load
+                  (local.get $0)
+                 )
+                 (i32.const 1)
+                )
+               )
+              )
+              (block
+               (i32.store offset=4
+                (local.get $4)
+                (i32.or
+                 (local.get $7)
+                 (i32.const 3)
+                )
+               )
+               (i32.store offset=4
+                (local.get $6)
+                (i32.or
+                 (local.get $3)
+                 (i32.const 1)
+                )
+               )
+               (i32.store
+                (i32.add
+                 (local.get $6)
+                 (local.get $3)
+                )
+                (local.get $3)
+               )
+               (local.set $0
+                (i32.shr_u
+                 (local.get $3)
+                 (i32.const 3)
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $3)
+                 (i32.const 256)
+                )
+                (block
+                 (local.set $3
+                  (i32.add
+                   (i32.shl
+                    (i32.shl
+                     (local.get $0)
+                     (i32.const 1)
+                    )
+                    (i32.const 2)
+                   )
+                   (i32.const 3684)
+                  )
+                 )
+                 (if
+                  (i32.and
+                   (local.tee $1
+                    (i32.load
+                     (i32.const 3644)
+                    )
+                   )
+                   (local.tee $0
+                    (i32.shl
+                     (i32.const 1)
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (if
+                   (i32.lt_u
+                    (local.tee $0
+                     (i32.load
+                      (local.tee $1
+                       (i32.add
+                        (local.get $3)
+                        (i32.const 8)
+                       )
+                      )
+                     )
+                    )
+                    (i32.load
+                     (i32.const 3660)
+                    )
+                   )
+                   (call $fimport$10)
+                   (block
+                    (local.set $16
+                     (local.get $1)
+                    )
+                    (local.set $8
+                     (local.get $0)
+                    )
+                   )
+                  )
+                  (block
+                   (i32.store
+                    (i32.const 3644)
+                    (i32.or
+                     (local.get $1)
+                     (local.get $0)
+                    )
+                   )
+                   (local.set $16
+                    (i32.add
+                     (local.get $3)
+                     (i32.const 8)
+                    )
+                   )
+                   (local.set $8
+                    (local.get $3)
+                   )
+                  )
+                 )
+                 (i32.store
+                  (local.get $16)
+                  (local.get $6)
+                 )
+                 (i32.store offset=12
+                  (local.get $8)
+                  (local.get $6)
+                 )
+                 (i32.store offset=8
+                  (local.get $6)
+                  (local.get $8)
+                 )
+                 (i32.store offset=12
+                  (local.get $6)
+                  (local.get $3)
+                 )
+                 (br $label$136)
+                )
+               )
+               (local.set $1
+                (i32.add
+                 (i32.shl
+                  (local.tee $5
+                   (if (result i32)
+                    (local.tee $0
+                     (i32.shr_u
+                      (local.get $3)
+                      (i32.const 8)
+                     )
+                    )
+                    (if (result i32)
+                     (i32.gt_u
+                      (local.get $3)
+                      (i32.const 16777215)
+                     )
+                     (i32.const 31)
+                     (i32.or
+                      (i32.and
+                       (i32.shr_u
+                        (local.get $3)
+                        (i32.add
+                         (local.tee $0
+                          (i32.add
+                           (i32.sub
+                            (i32.const 14)
+                            (i32.or
+                             (i32.or
+                              (local.tee $0
+                               (i32.and
+                                (i32.shr_u
+                                 (i32.add
+                                  (local.tee $1
+                                   (i32.shl
+                                    (local.get $0)
+                                    (local.tee $5
+                                     (i32.and
+                                      (i32.shr_u
+                                       (i32.add
+                                        (local.get $0)
+                                        (i32.const 1048320)
+                                       )
+                                       (i32.const 16)
+                                      )
+                                      (i32.const 8)
+                                     )
+                                    )
+                                   )
+                                  )
+                                  (i32.const 520192)
+                                 )
+                                 (i32.const 16)
+                                )
+                                (i32.const 4)
+                               )
+                              )
+                              (local.get $5)
+                             )
+                             (local.tee $0
+                              (i32.and
+                               (i32.shr_u
+                                (i32.add
+                                 (local.tee $1
+                                  (i32.shl
+                                   (local.get $1)
+                                   (local.get $0)
+                                  )
+                                 )
+                                 (i32.const 245760)
+                                )
+                                (i32.const 16)
+                               )
+                               (i32.const 2)
+                              )
+                             )
+                            )
+                           )
+                           (i32.shr_u
+                            (i32.shl
+                             (local.get $1)
+                             (local.get $0)
+                            )
+                            (i32.const 15)
+                           )
+                          )
+                         )
+                         (i32.const 7)
+                        )
+                       )
+                       (i32.const 1)
+                      )
+                      (i32.shl
+                       (local.get $0)
+                       (i32.const 1)
+                      )
+                     )
+                    )
+                    (i32.const 0)
+                   )
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 3948)
+                )
+               )
+               (i32.store offset=28
+                (local.get $6)
+                (local.get $5)
+               )
+               (i32.store offset=4
+                (local.tee $0
+                 (i32.add
+                  (local.get $6)
+                  (i32.const 16)
+                 )
+                )
+                (i32.const 0)
+               )
+               (i32.store
+                (local.get $0)
+                (i32.const 0)
+               )
+               (if
+                (i32.eqz
+                 (i32.and
+                  (local.get $2)
+                  (local.tee $0
+                   (i32.shl
+                    (i32.const 1)
+                    (local.get $5)
+                   )
+                  )
+                 )
+                )
+                (block
+                 (i32.store
+                  (i32.const 3648)
+                  (i32.or
+                   (local.get $2)
+                   (local.get $0)
+                  )
+                 )
+                 (i32.store
+                  (local.get $1)
+                  (local.get $6)
+                 )
+                 (i32.store offset=24
+                  (local.get $6)
+                  (local.get $1)
+                 )
+                 (i32.store offset=12
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (i32.store offset=8
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (br $label$136)
+                )
+               )
+               (local.set $0
+                (i32.load
+                 (local.get $1)
+                )
+               )
+               (local.set $1
+                (i32.sub
+                 (i32.const 25)
+                 (i32.shr_u
+                  (local.get $5)
+                  (i32.const 1)
+                 )
+                )
+               )
+               (local.set $5
+                (i32.shl
+                 (local.get $3)
+                 (if (result i32)
+                  (i32.eq
+                   (local.get $5)
+                   (i32.const 31)
+                  )
+                  (i32.const 0)
+                  (local.get $1)
+                 )
+                )
+               )
+               (block $label$151
+                (block $label$152
+                 (block $label$153
+                  (loop $label$154
+                   (br_if $label$152
+                    (i32.eq
+                     (i32.and
+                      (i32.load offset=4
+                       (local.get $0)
+                      )
+                      (i32.const -8)
+                     )
+                     (local.get $3)
+                    )
+                   )
+                   (local.set $2
+                    (i32.shl
+                     (local.get $5)
+                     (i32.const 1)
+                    )
+                   )
+                   (br_if $label$153
+                    (i32.eqz
+                     (local.tee $1
+                      (i32.load
+                       (local.tee $5
+                        (i32.add
+                         (i32.add
+                          (local.get $0)
+                          (i32.const 16)
+                         )
+                         (i32.shl
+                          (i32.shr_u
+                           (local.get $5)
+                           (i32.const 31)
+                          )
+                          (i32.const 2)
+                         )
+                        )
+                       )
+                      )
+                     )
+                    )
+                   )
+                   (local.set $5
+                    (local.get $2)
+                   )
+                   (local.set $0
+                    (local.get $1)
+                   )
+                   (br $label$154)
+                  )
+                 )
+                 (if
+                  (i32.lt_u
+                   (local.get $5)
+                   (i32.load
+                    (i32.const 3660)
+                   )
+                  )
+                  (call $fimport$10)
+                  (block
+                   (i32.store
+                    (local.get $5)
+                    (local.get $6)
+                   )
+                   (i32.store offset=24
+                    (local.get $6)
+                    (local.get $0)
+                   )
+                   (i32.store offset=12
+                    (local.get $6)
+                    (local.get $6)
+                   )
+                   (i32.store offset=8
+                    (local.get $6)
+                    (local.get $6)
+                   )
+                   (br $label$136)
+                  )
+                 )
+                 (br $label$151)
+                )
+                (if
+                 (i32.and
+                  (i32.ge_u
+                   (local.tee $2
+                    (i32.load
+                     (local.tee $3
+                      (i32.add
+                       (local.get $0)
+                       (i32.const 8)
+                      )
+                     )
+                    )
+                   )
+                   (local.tee $1
+                    (i32.load
+                     (i32.const 3660)
+                    )
+                   )
+                  )
+                  (i32.ge_u
+                   (local.get $0)
+                   (local.get $1)
+                  )
+                 )
+                 (block
+                  (i32.store offset=12
+                   (local.get $2)
+                   (local.get $6)
+                  )
+                  (i32.store
+                   (local.get $3)
+                   (local.get $6)
+                  )
+                  (i32.store offset=8
+                   (local.get $6)
+                   (local.get $2)
+                  )
+                  (i32.store offset=12
+                   (local.get $6)
+                   (local.get $0)
+                  )
+                  (i32.store offset=24
+                   (local.get $6)
+                   (i32.const 0)
+                  )
+                 )
+                 (call $fimport$10)
+                )
+               )
+              )
+             )
+            )
+            (global.set $global$1
+             (local.get $14)
+            )
+            (return
+             (i32.add
+              (local.get $4)
+              (i32.const 8)
+             )
+            )
+           )
+           (local.set $0
+            (local.get $7)
+           )
+          )
+          (local.set $0
+           (local.get $7)
+          )
+         )
+        )
+        (local.set $0
+         (local.get $7)
+        )
+       )
+      )
+     )
+    )
+   )
+   (if
+    (i32.ge_u
+     (local.tee $1
+      (i32.load
+       (i32.const 3652)
+      )
+     )
+     (local.get $0)
+    )
+    (block
+     (local.set $2
+      (i32.load
+       (i32.const 3664)
+      )
+     )
+     (if
+      (i32.gt_u
+       (local.tee $3
+        (i32.sub
+         (local.get $1)
+         (local.get $0)
+        )
+       )
+       (i32.const 15)
+      )
+      (block
+       (i32.store
+        (i32.const 3664)
+        (local.tee $1
+         (i32.add
+          (local.get $2)
+          (local.get $0)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3652)
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $1)
+        (i32.or
+         (local.get $3)
+         (i32.const 1)
+        )
+       )
+       (i32.store
+        (i32.add
+         (local.get $1)
+         (local.get $3)
+        )
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $2)
+        (i32.or
+         (local.get $0)
+         (i32.const 3)
+        )
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3652)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 3664)
+        (i32.const 0)
+       )
+       (i32.store offset=4
+        (local.get $2)
+        (i32.or
+         (local.get $1)
+         (i32.const 3)
+        )
+       )
+       (i32.store
+        (local.tee $0
+         (i32.add
+          (i32.add
+           (local.get $2)
+           (local.get $1)
+          )
+          (i32.const 4)
+         )
+        )
+        (i32.or
+         (i32.load
+          (local.get $0)
+         )
+         (i32.const 1)
+        )
+       )
+      )
+     )
+     (global.set $global$1
+      (local.get $14)
+     )
+     (return
+      (i32.add
+       (local.get $2)
+       (i32.const 8)
+      )
+     )
+    )
+   )
+   (if
+    (i32.gt_u
+     (local.tee $10
+      (i32.load
+       (i32.const 3656)
+      )
+     )
+     (local.get $0)
+    )
+    (block
+     (i32.store
+      (i32.const 3656)
+      (local.tee $3
+       (i32.sub
+        (local.get $10)
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store
+      (i32.const 3668)
+      (local.tee $1
+       (i32.add
+        (local.tee $2
+         (i32.load
+          (i32.const 3668)
+         )
+        )
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store offset=4
+      (local.get $1)
+      (i32.or
+       (local.get $3)
+       (i32.const 1)
+      )
+     )
+     (i32.store offset=4
+      (local.get $2)
+      (i32.or
+       (local.get $0)
+       (i32.const 3)
+      )
+     )
+     (global.set $global$1
+      (local.get $14)
+     )
+     (return
+      (i32.add
+       (local.get $2)
+       (i32.const 8)
+      )
+     )
+    )
+   )
+   (if
+    (i32.le_u
+     (local.tee $6
+      (i32.and
+       (local.tee $8
+        (i32.add
+         (local.tee $1
+          (if (result i32)
+           (i32.load
+            (i32.const 4116)
+           )
+           (i32.load
+            (i32.const 4124)
+           )
+           (block (result i32)
+            (i32.store
+             (i32.const 4124)
+             (i32.const 4096)
+            )
+            (i32.store
+             (i32.const 4120)
+             (i32.const 4096)
+            )
+            (i32.store
+             (i32.const 4128)
+             (i32.const -1)
+            )
+            (i32.store
+             (i32.const 4132)
+             (i32.const -1)
+            )
+            (i32.store
+             (i32.const 4136)
+             (i32.const 0)
+            )
+            (i32.store
+             (i32.const 4088)
+             (i32.const 0)
+            )
+            (i32.store
+             (local.get $18)
+             (local.tee $1
+              (i32.xor
+               (i32.and
+                (local.get $18)
+                (i32.const -16)
+               )
+               (i32.const 1431655768)
+              )
+             )
+            )
+            (i32.store
+             (i32.const 4116)
+             (local.get $1)
+            )
+            (i32.const 4096)
+           )
+          )
+         )
+         (local.tee $13
+          (i32.add
+           (local.get $0)
+           (i32.const 47)
+          )
+         )
+        )
+       )
+       (local.tee $4
+        (i32.sub
+         (i32.const 0)
+         (local.get $1)
+        )
+       )
+      )
+     )
+     (local.get $0)
+    )
+    (block
+     (global.set $global$1
+      (local.get $14)
+     )
+     (return
+      (i32.const 0)
+     )
+    )
+   )
+   (if
+    (local.tee $2
+     (i32.load
+      (i32.const 4084)
+     )
+    )
+    (if
+     (i32.or
+      (i32.le_u
+       (local.tee $1
+        (i32.add
+         (local.tee $3
+          (i32.load
+           (i32.const 4076)
+          )
+         )
+         (local.get $6)
+        )
+       )
+       (local.get $3)
+      )
+      (i32.gt_u
+       (local.get $1)
+       (local.get $2)
+      )
+     )
+     (block
+      (global.set $global$1
+       (local.get $14)
+      )
+      (return
+       (i32.const 0)
+      )
+     )
+    )
+   )
+   (local.set $7
+    (i32.add
+     (local.get $0)
+     (i32.const 48)
+    )
+   )
+   (block $label$171
+    (block $label$172
+     (if
+      (i32.eqz
+       (i32.and
+        (i32.load
+         (i32.const 4088)
+        )
+        (i32.const 4)
+       )
+      )
+      (block
+       (block $label$174
+        (block $label$175
+         (block $label$176
+          (br_if $label$176
+           (i32.eqz
+            (local.tee $3
+             (i32.load
+              (i32.const 3668)
+             )
+            )
+           )
+          )
+          (local.set $2
+           (i32.const 4092)
+          )
+          (loop $label$177
+           (block $label$178
+            (if
+             (i32.le_u
+              (local.tee $1
+               (i32.load
+                (local.get $2)
+               )
+              )
+              (local.get $3)
+             )
+             (br_if $label$178
+              (i32.gt_u
+               (i32.add
+                (local.get $1)
+                (i32.load
+                 (local.tee $5
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 4)
+                  )
+                 )
+                )
+               )
+               (local.get $3)
+              )
+             )
+            )
+            (br_if $label$176
+             (i32.eqz
+              (local.tee $1
+               (i32.load offset=8
+                (local.get $2)
+               )
+              )
+             )
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (br $label$177)
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.tee $3
+             (i32.and
+              (i32.sub
+               (local.get $8)
+               (local.get $10)
+              )
+              (local.get $4)
+             )
+            )
+            (i32.const 2147483647)
+           )
+           (if
+            (i32.eq
+             (local.tee $1
+              (call $34
+               (local.get $3)
+              )
+             )
+             (i32.add
+              (i32.load
+               (local.get $2)
+              )
+              (i32.load
+               (local.get $5)
+              )
+             )
+            )
+            (br_if $label$172
+             (i32.ne
+              (local.get $1)
+              (i32.const -1)
+             )
+            )
+            (block
+             (local.set $2
+              (local.get $1)
+             )
+             (local.set $1
+              (local.get $3)
+             )
+             (br $label$175)
+            )
+           )
+          )
+          (br $label$174)
+         )
+         (if
+          (i32.ne
+           (local.tee $1
+            (call $34
+             (i32.const 0)
+            )
+           )
+           (i32.const -1)
+          )
+          (block
+           (local.set $2
+            (i32.sub
+             (i32.and
+              (i32.add
+               (local.tee $5
+                (i32.add
+                 (local.tee $2
+                  (i32.load
+                   (i32.const 4120)
+                  )
+                 )
+                 (i32.const -1)
+                )
+               )
+               (local.tee $3
+                (local.get $1)
+               )
+              )
+              (i32.sub
+               (i32.const 0)
+               (local.get $2)
+              )
+             )
+             (local.get $3)
+            )
+           )
+           (local.set $4
+            (i32.add
+             (local.tee $3
+              (i32.add
+               (if (result i32)
+                (i32.and
+                 (local.get $5)
+                 (local.get $3)
+                )
+                (local.get $2)
+                (i32.const 0)
+               )
+               (local.get $6)
+              )
+             )
+             (local.tee $5
+              (i32.load
+               (i32.const 4076)
+              )
+             )
+            )
+           )
+           (if
+            (i32.and
+             (i32.gt_u
+              (local.get $3)
+              (local.get $0)
+             )
+             (i32.lt_u
+              (local.get $3)
+              (i32.const 2147483647)
+             )
+            )
+            (block
+             (if
+              (local.tee $2
+               (i32.load
+                (i32.const 4084)
+               )
+              )
+              (br_if $label$174
+               (i32.or
+                (i32.le_u
+                 (local.get $4)
+                 (local.get $5)
+                )
+                (i32.gt_u
+                 (local.get $4)
+                 (local.get $2)
+                )
+               )
+              )
+             )
+             (br_if $label$172
+              (i32.eq
+               (local.tee $2
+                (call $34
+                 (local.get $3)
+                )
+               )
+               (local.get $1)
+              )
+             )
+             (local.set $1
+              (local.get $3)
+             )
+             (br $label$175)
+            )
+           )
+          )
+         )
+         (br $label$174)
+        )
+        (local.set $5
+         (i32.sub
+          (i32.const 0)
+          (local.get $1)
+         )
+        )
+        (if
+         (i32.and
+          (i32.gt_u
+           (local.get $7)
+           (local.get $1)
+          )
+          (i32.and
+           (i32.lt_u
+            (local.get $1)
+            (i32.const 2147483647)
+           )
+           (i32.ne
+            (local.get $2)
+            (i32.const -1)
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.tee $3
+            (i32.and
+             (i32.add
+              (i32.sub
+               (local.get $13)
+               (local.get $1)
+              )
+              (local.tee $3
+               (i32.load
+                (i32.const 4124)
+               )
+              )
+             )
+             (i32.sub
+              (i32.const 0)
+              (local.get $3)
+             )
+            )
+           )
+           (i32.const 2147483647)
+          )
+          (if
+           (i32.eq
+            (call $34
+             (local.get $3)
+            )
+            (i32.const -1)
+           )
+           (block
+            (drop
+             (call $34
+              (local.get $5)
+             )
+            )
+            (br $label$174)
+           )
+           (local.set $3
+            (i32.add
+             (local.get $3)
+             (local.get $1)
+            )
+           )
+          )
+          (local.set $3
+           (local.get $1)
+          )
+         )
+         (local.set $3
+          (local.get $1)
+         )
+        )
+        (if
+         (i32.ne
+          (local.get $2)
+          (i32.const -1)
+         )
+         (block
+          (local.set $1
+           (local.get $2)
+          )
+          (br $label$172)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 4088)
+        (i32.or
+         (i32.load
+          (i32.const 4088)
+         )
+         (i32.const 4)
+        )
+       )
+      )
+     )
+     (if
+      (i32.lt_u
+       (local.get $6)
+       (i32.const 2147483647)
+      )
+      (if
+       (i32.and
+        (i32.lt_u
+         (local.tee $1
+          (call $34
+           (local.get $6)
+          )
+         )
+         (local.tee $3
+          (call $34
+           (i32.const 0)
+          )
+         )
+        )
+        (i32.and
+         (i32.ne
+          (local.get $1)
+          (i32.const -1)
+         )
+         (i32.ne
+          (local.get $3)
+          (i32.const -1)
+         )
+        )
+       )
+       (br_if $label$172
+        (i32.gt_u
+         (local.tee $3
+          (i32.sub
+           (local.get $3)
+           (local.get $1)
+          )
+         )
+         (i32.add
+          (local.get $0)
+          (i32.const 40)
+         )
+        )
+       )
+      )
+     )
+     (br $label$171)
+    )
+    (i32.store
+     (i32.const 4076)
+     (local.tee $2
+      (i32.add
+       (i32.load
+        (i32.const 4076)
+       )
+       (local.get $3)
+      )
+     )
+    )
+    (if
+     (i32.gt_u
+      (local.get $2)
+      (i32.load
+       (i32.const 4080)
+      )
+     )
+     (i32.store
+      (i32.const 4080)
+      (local.get $2)
+     )
+    )
+    (block $label$198
+     (if
+      (local.tee $8
+       (i32.load
+        (i32.const 3668)
+       )
+      )
+      (block
+       (local.set $2
+        (i32.const 4092)
+       )
+       (block $label$200
+        (block $label$201
+         (loop $label$202
+          (br_if $label$201
+           (i32.eq
+            (local.get $1)
+            (i32.add
+             (local.tee $4
+              (i32.load
+               (local.get $2)
+              )
+             )
+             (local.tee $5
+              (i32.load
+               (local.tee $7
+                (i32.add
+                 (local.get $2)
+                 (i32.const 4)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (br_if $label$202
+           (local.tee $2
+            (i32.load offset=8
+             (local.get $2)
+            )
+           )
+          )
+         )
+         (br $label$200)
+        )
+        (if
+         (i32.eqz
+          (i32.and
+           (i32.load offset=12
+            (local.get $2)
+           )
+           (i32.const 8)
+          )
+         )
+         (if
+          (i32.and
+           (i32.lt_u
+            (local.get $8)
+            (local.get $1)
+           )
+           (i32.ge_u
+            (local.get $8)
+            (local.get $4)
+           )
+          )
+          (block
+           (i32.store
+            (local.get $7)
+            (i32.add
+             (local.get $5)
+             (local.get $3)
+            )
+           )
+           (local.set $5
+            (i32.load
+             (i32.const 3656)
+            )
+           )
+           (local.set $1
+            (i32.and
+             (i32.sub
+              (i32.const 0)
+              (local.tee $2
+               (i32.add
+                (local.get $8)
+                (i32.const 8)
+               )
+              )
+             )
+             (i32.const 7)
+            )
+           )
+           (i32.store
+            (i32.const 3668)
+            (local.tee $2
+             (i32.add
+              (local.get $8)
+              (if (result i32)
+               (i32.and
+                (local.get $2)
+                (i32.const 7)
+               )
+               (local.get $1)
+               (local.tee $1
+                (i32.const 0)
+               )
+              )
+             )
+            )
+           )
+           (i32.store
+            (i32.const 3656)
+            (local.tee $1
+             (i32.add
+              (i32.sub
+               (local.get $3)
+               (local.get $1)
+              )
+              (local.get $5)
+             )
+            )
+           )
+           (i32.store offset=4
+            (local.get $2)
+            (i32.or
+             (local.get $1)
+             (i32.const 1)
+            )
+           )
+           (i32.store offset=4
+            (i32.add
+             (local.get $2)
+             (local.get $1)
+            )
+            (i32.const 40)
+           )
+           (i32.store
+            (i32.const 3672)
+            (i32.load
+             (i32.const 4132)
+            )
+           )
+           (br $label$198)
+          )
+         )
+        )
+       )
+       (if
+        (i32.lt_u
+         (local.get $1)
+         (local.tee $2
+          (i32.load
+           (i32.const 3660)
+          )
+         )
+        )
+        (block
+         (i32.store
+          (i32.const 3660)
+          (local.get $1)
+         )
+         (local.set $2
+          (local.get $1)
+         )
+        )
+       )
+       (local.set $10
+        (i32.add
+         (local.get $1)
+         (local.get $3)
+        )
+       )
+       (local.set $5
+        (i32.const 4092)
+       )
+       (block $label$208
+        (block $label$209
+         (loop $label$210
+          (br_if $label$209
+           (i32.eq
+            (i32.load
+             (local.get $5)
+            )
+            (local.get $10)
+           )
+          )
+          (br_if $label$210
+           (local.tee $5
+            (i32.load offset=8
+             (local.get $5)
+            )
+           )
+          )
+          (local.set $5
+           (i32.const 4092)
+          )
+         )
+         (br $label$208)
+        )
+        (if
+         (i32.and
+          (i32.load offset=12
+           (local.get $5)
+          )
+          (i32.const 8)
+         )
+         (local.set $5
+          (i32.const 4092)
+         )
+         (block
+          (i32.store
+           (local.get $5)
+           (local.get $1)
+          )
+          (i32.store
+           (local.tee $5
+            (i32.add
+             (local.get $5)
+             (i32.const 4)
+            )
+           )
+           (i32.add
+            (i32.load
+             (local.get $5)
+            )
+            (local.get $3)
+           )
+          )
+          (local.set $7
+           (i32.and
+            (i32.sub
+             (i32.const 0)
+             (local.tee $4
+              (i32.add
+               (local.get $1)
+               (i32.const 8)
+              )
+             )
+            )
+            (i32.const 7)
+           )
+          )
+          (local.set $3
+           (i32.and
+            (i32.sub
+             (i32.const 0)
+             (local.tee $5
+              (i32.add
+               (local.get $10)
+               (i32.const 8)
+              )
+             )
+            )
+            (i32.const 7)
+           )
+          )
+          (local.set $6
+           (i32.add
+            (local.tee $13
+             (i32.add
+              (local.get $1)
+              (if (result i32)
+               (i32.and
+                (local.get $4)
+                (i32.const 7)
+               )
+               (local.get $7)
+               (i32.const 0)
+              )
+             )
+            )
+            (local.get $0)
+           )
+          )
+          (local.set $7
+           (i32.sub
+            (i32.sub
+             (local.tee $4
+              (i32.add
+               (local.get $10)
+               (if (result i32)
+                (i32.and
+                 (local.get $5)
+                 (i32.const 7)
+                )
+                (local.get $3)
+                (i32.const 0)
+               )
+              )
+             )
+             (local.get $13)
+            )
+            (local.get $0)
+           )
+          )
+          (i32.store offset=4
+           (local.get $13)
+           (i32.or
+            (local.get $0)
+            (i32.const 3)
+           )
+          )
+          (block $label$217
+           (if
+            (i32.eq
+             (local.get $4)
+             (local.get $8)
+            )
+            (block
+             (i32.store
+              (i32.const 3656)
+              (local.tee $0
+               (i32.add
+                (i32.load
+                 (i32.const 3656)
+                )
+                (local.get $7)
+               )
+              )
+             )
+             (i32.store
+              (i32.const 3668)
+              (local.get $6)
+             )
+             (i32.store offset=4
+              (local.get $6)
+              (i32.or
+               (local.get $0)
+               (i32.const 1)
+              )
+             )
+            )
+            (block
+             (if
+              (i32.eq
+               (local.get $4)
+               (i32.load
+                (i32.const 3664)
+               )
+              )
+              (block
+               (i32.store
+                (i32.const 3652)
+                (local.tee $0
+                 (i32.add
+                  (i32.load
+                   (i32.const 3652)
+                  )
+                  (local.get $7)
+                 )
+                )
+               )
+               (i32.store
+                (i32.const 3664)
+                (local.get $6)
+               )
+               (i32.store offset=4
+                (local.get $6)
+                (i32.or
+                 (local.get $0)
+                 (i32.const 1)
+                )
+               )
+               (i32.store
+                (i32.add
+                 (local.get $6)
+                 (local.get $0)
+                )
+                (local.get $0)
+               )
+               (br $label$217)
+              )
+             )
+             (i32.store
+              (local.tee $0
+               (i32.add
+                (local.tee $0
+                 (if (result i32)
+                  (i32.eq
+                   (i32.and
+                    (local.tee $0
+                     (i32.load offset=4
+                      (local.get $4)
+                     )
+                    )
+                    (i32.const 3)
+                   )
+                   (i32.const 1)
+                  )
+                  (block (result i32)
+                   (local.set $11
+                    (i32.and
+                     (local.get $0)
+                     (i32.const -8)
+                    )
+                   )
+                   (local.set $1
+                    (i32.shr_u
+                     (local.get $0)
+                     (i32.const 3)
+                    )
+                   )
+                   (block $label$222
+                    (if
+                     (i32.lt_u
+                      (local.get $0)
+                      (i32.const 256)
+                     )
+                     (block
+                      (local.set $5
+                       (i32.load offset=12
+                        (local.get $4)
+                       )
+                      )
+                      (block $label$224
+                       (if
+                        (i32.ne
+                         (local.tee $3
+                          (i32.load offset=8
+                           (local.get $4)
+                          )
+                         )
+                         (local.tee $0
+                          (i32.add
+                           (i32.shl
+                            (i32.shl
+                             (local.get $1)
+                             (i32.const 1)
+                            )
+                            (i32.const 2)
+                           )
+                           (i32.const 3684)
+                          )
+                         )
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.get $3)
+                           (local.get $2)
+                          )
+                          (call $fimport$10)
+                         )
+                         (br_if $label$224
+                          (i32.eq
+                           (i32.load offset=12
+                            (local.get $3)
+                           )
+                           (local.get $4)
+                          )
+                         )
+                         (call $fimport$10)
+                        )
+                       )
+                      )
+                      (if
+                       (i32.eq
+                        (local.get $5)
+                        (local.get $3)
+                       )
+                       (block
+                        (i32.store
+                         (i32.const 3644)
+                         (i32.and
+                          (i32.load
+                           (i32.const 3644)
+                          )
+                          (i32.xor
+                           (i32.shl
+                            (i32.const 1)
+                            (local.get $1)
+                           )
+                           (i32.const -1)
+                          )
+                         )
+                        )
+                        (br $label$222)
+                       )
+                      )
+                      (block $label$228
+                       (if
+                        (i32.eq
+                         (local.get $5)
+                         (local.get $0)
+                        )
+                        (local.set $20
+                         (i32.add
+                          (local.get $5)
+                          (i32.const 8)
+                         )
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.get $5)
+                           (local.get $2)
+                          )
+                          (call $fimport$10)
+                         )
+                         (if
+                          (i32.eq
+                           (i32.load
+                            (local.tee $0
+                             (i32.add
+                              (local.get $5)
+                              (i32.const 8)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (block
+                           (local.set $20
+                            (local.get $0)
+                           )
+                           (br $label$228)
+                          )
+                         )
+                         (call $fimport$10)
+                        )
+                       )
+                      )
+                      (i32.store offset=12
+                       (local.get $3)
+                       (local.get $5)
+                      )
+                      (i32.store
+                       (local.get $20)
+                       (local.get $3)
+                      )
+                     )
+                     (block
+                      (local.set $8
+                       (i32.load offset=24
+                        (local.get $4)
+                       )
+                      )
+                      (block $label$234
+                       (if
+                        (i32.eq
+                         (local.tee $0
+                          (i32.load offset=12
+                           (local.get $4)
+                          )
+                         )
+                         (local.get $4)
+                        )
+                        (block
+                         (if
+                          (i32.eqz
+                           (local.tee $0
+                            (i32.load
+                             (local.tee $1
+                              (i32.add
+                               (local.tee $3
+                                (i32.add
+                                 (local.get $4)
+                                 (i32.const 16)
+                                )
+                               )
+                               (i32.const 4)
+                              )
+                             )
+                            )
+                           )
+                          )
+                          (if
+                           (local.tee $0
+                            (i32.load
+                             (local.get $3)
+                            )
+                           )
+                           (local.set $1
+                            (local.get $3)
+                           )
+                           (block
+                            (local.set $12
+                             (i32.const 0)
+                            )
+                            (br $label$234)
+                           )
+                          )
+                         )
+                         (loop $label$239
+                          (if
+                           (local.tee $3
+                            (i32.load
+                             (local.tee $5
+                              (i32.add
+                               (local.get $0)
+                               (i32.const 20)
+                              )
+                             )
+                            )
+                           )
+                           (block
+                            (local.set $0
+                             (local.get $3)
+                            )
+                            (local.set $1
+                             (local.get $5)
+                            )
+                            (br $label$239)
+                           )
+                          )
+                          (if
+                           (local.tee $3
+                            (i32.load
+                             (local.tee $5
+                              (i32.add
+                               (local.get $0)
+                               (i32.const 16)
+                              )
+                             )
+                            )
+                           )
+                           (block
+                            (local.set $0
+                             (local.get $3)
+                            )
+                            (local.set $1
+                             (local.get $5)
+                            )
+                            (br $label$239)
+                           )
+                          )
+                         )
+                         (if
+                          (i32.lt_u
+                           (local.get $1)
+                           (local.get $2)
+                          )
+                          (call $fimport$10)
+                          (block
+                           (i32.store
+                            (local.get $1)
+                            (i32.const 0)
+                           )
+                           (local.set $12
+                            (local.get $0)
+                           )
+                          )
+                         )
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.tee $5
+                            (i32.load offset=8
+                             (local.get $4)
+                            )
+                           )
+                           (local.get $2)
+                          )
+                          (call $fimport$10)
+                         )
+                         (if
+                          (i32.ne
+                           (i32.load
+                            (local.tee $3
+                             (i32.add
+                              (local.get $5)
+                              (i32.const 12)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (call $fimport$10)
+                         )
+                         (if
+                          (i32.eq
+                           (i32.load
+                            (local.tee $1
+                             (i32.add
+                              (local.get $0)
+                              (i32.const 8)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (block
+                           (i32.store
+                            (local.get $3)
+                            (local.get $0)
+                           )
+                           (i32.store
+                            (local.get $1)
+                            (local.get $5)
+                           )
+                           (local.set $12
+                            (local.get $0)
+                           )
+                          )
+                          (call $fimport$10)
+                         )
+                        )
+                       )
+                      )
+                      (br_if $label$222
+                       (i32.eqz
+                        (local.get $8)
+                       )
+                      )
+                      (block $label$249
+                       (if
+                        (i32.eq
+                         (local.get $4)
+                         (i32.load
+                          (local.tee $0
+                           (i32.add
+                            (i32.shl
+                             (local.tee $1
+                              (i32.load offset=28
+                               (local.get $4)
+                              )
+                             )
+                             (i32.const 2)
+                            )
+                            (i32.const 3948)
+                           )
+                          )
+                         )
+                        )
+                        (block
+                         (i32.store
+                          (local.get $0)
+                          (local.get $12)
+                         )
+                         (br_if $label$249
+                          (local.get $12)
+                         )
+                         (i32.store
+                          (i32.const 3648)
+                          (i32.and
+                           (i32.load
+                            (i32.const 3648)
+                           )
+                           (i32.xor
+                            (i32.shl
+                             (i32.const 1)
+                             (local.get $1)
+                            )
+                            (i32.const -1)
+                           )
+                          )
+                         )
+                         (br $label$222)
+                        )
+                        (block
+                         (if
+                          (i32.lt_u
+                           (local.get $8)
+                           (i32.load
+                            (i32.const 3660)
+                           )
+                          )
+                          (call $fimport$10)
+                         )
+                         (if
+                          (i32.eq
+                           (i32.load
+                            (local.tee $0
+                             (i32.add
+                              (local.get $8)
+                              (i32.const 16)
+                             )
+                            )
+                           )
+                           (local.get $4)
+                          )
+                          (i32.store
+                           (local.get $0)
+                           (local.get $12)
+                          )
+                          (i32.store offset=20
+                           (local.get $8)
+                           (local.get $12)
+                          )
+                         )
+                         (br_if $label$222
+                          (i32.eqz
+                           (local.get $12)
+                          )
+                         )
+                        )
+                       )
+                      )
+                      (if
+                       (i32.lt_u
+                        (local.get $12)
+                        (local.tee $1
+                         (i32.load
+                          (i32.const 3660)
+                         )
+                        )
+                       )
+                       (call $fimport$10)
+                      )
+                      (i32.store offset=24
+                       (local.get $12)
+                       (local.get $8)
+                      )
+                      (if
+                       (local.tee $3
+                        (i32.load
+                         (local.tee $0
+                          (i32.add
+                           (local.get $4)
+                           (i32.const 16)
+                          )
+                         )
+                        )
+                       )
+                       (if
+                        (i32.lt_u
+                         (local.get $3)
+                         (local.get $1)
+                        )
+                        (call $fimport$10)
+                        (block
+                         (i32.store offset=16
+                          (local.get $12)
+                          (local.get $3)
+                         )
+                         (i32.store offset=24
+                          (local.get $3)
+                          (local.get $12)
+                         )
+                        )
+                       )
+                      )
+                      (br_if $label$222
+                       (i32.eqz
+                        (local.tee $0
+                         (i32.load offset=4
+                          (local.get $0)
+                         )
+                        )
+                       )
+                      )
+                      (if
+                       (i32.lt_u
+                        (local.get $0)
+                        (i32.load
+                         (i32.const 3660)
+                        )
+                       )
+                       (call $fimport$10)
+                       (block
+                        (i32.store offset=20
+                         (local.get $12)
+                         (local.get $0)
+                        )
+                        (i32.store offset=24
+                         (local.get $0)
+                         (local.get $12)
+                        )
+                       )
+                      )
+                     )
+                    )
+                   )
+                   (local.set $7
+                    (i32.add
+                     (local.get $11)
+                     (local.get $7)
+                    )
+                   )
+                   (i32.add
+                    (local.get $4)
+                    (local.get $11)
+                   )
+                  )
+                  (local.get $4)
+                 )
+                )
+                (i32.const 4)
+               )
+              )
+              (i32.and
+               (i32.load
+                (local.get $0)
+               )
+               (i32.const -2)
+              )
+             )
+             (i32.store offset=4
+              (local.get $6)
+              (i32.or
+               (local.get $7)
+               (i32.const 1)
+              )
+             )
+             (i32.store
+              (i32.add
+               (local.get $6)
+               (local.get $7)
+              )
+              (local.get $7)
+             )
+             (local.set $0
+              (i32.shr_u
+               (local.get $7)
+               (i32.const 3)
+              )
+             )
+             (if
+              (i32.lt_u
+               (local.get $7)
+               (i32.const 256)
+              )
+              (block
+               (local.set $3
+                (i32.add
+                 (i32.shl
+                  (i32.shl
+                   (local.get $0)
+                   (i32.const 1)
+                  )
+                  (i32.const 2)
+                 )
+                 (i32.const 3684)
+                )
+               )
+               (block $label$263
+                (if
+                 (i32.and
+                  (local.tee $1
+                   (i32.load
+                    (i32.const 3644)
+                   )
+                  )
+                  (local.tee $0
+                   (i32.shl
+                    (i32.const 1)
+                    (local.get $0)
+                   )
+                  )
+                 )
+                 (block
+                  (if
+                   (i32.ge_u
+                    (local.tee $0
+                     (i32.load
+                      (local.tee $1
+                       (i32.add
+                        (local.get $3)
+                        (i32.const 8)
+                       )
+                      )
+                     )
+                    )
+                    (i32.load
+                     (i32.const 3660)
+                    )
+                   )
+                   (block
+                    (local.set $21
+                     (local.get $1)
+                    )
+                    (local.set $9
+                     (local.get $0)
+                    )
+                    (br $label$263)
+                   )
+                  )
+                  (call $fimport$10)
+                 )
+                 (block
+                  (i32.store
+                   (i32.const 3644)
+                   (i32.or
+                    (local.get $1)
+                    (local.get $0)
+                   )
+                  )
+                  (local.set $21
+                   (i32.add
+                    (local.get $3)
+                    (i32.const 8)
+                   )
+                  )
+                  (local.set $9
+                   (local.get $3)
+                  )
+                 )
+                )
+               )
+               (i32.store
+                (local.get $21)
+                (local.get $6)
+               )
+               (i32.store offset=12
+                (local.get $9)
+                (local.get $6)
+               )
+               (i32.store offset=8
+                (local.get $6)
+                (local.get $9)
+               )
+               (i32.store offset=12
+                (local.get $6)
+                (local.get $3)
+               )
+               (br $label$217)
+              )
+             )
+             (local.set $3
+              (i32.add
+               (i32.shl
+                (local.tee $2
+                 (block $label$267 (result i32)
+                  (if (result i32)
+                   (local.tee $0
+                    (i32.shr_u
+                     (local.get $7)
+                     (i32.const 8)
+                    )
+                   )
+                   (block (result i32)
+                    (drop
+                     (br_if $label$267
+                      (i32.const 31)
+                      (i32.gt_u
+                       (local.get $7)
+                       (i32.const 16777215)
+                      )
+                     )
+                    )
+                    (i32.or
+                     (i32.and
+                      (i32.shr_u
+                       (local.get $7)
+                       (i32.add
+                        (local.tee $0
+                         (i32.add
+                          (i32.sub
+                           (i32.const 14)
+                           (i32.or
+                            (i32.or
+                             (local.tee $0
+                              (i32.and
+                               (i32.shr_u
+                                (i32.add
+                                 (local.tee $1
+                                  (i32.shl
+                                   (local.get $0)
+                                   (local.tee $3
+                                    (i32.and
+                                     (i32.shr_u
+                                      (i32.add
+                                       (local.get $0)
+                                       (i32.const 1048320)
+                                      )
+                                      (i32.const 16)
+                                     )
+                                     (i32.const 8)
+                                    )
+                                   )
+                                  )
+                                 )
+                                 (i32.const 520192)
+                                )
+                                (i32.const 16)
+                               )
+                               (i32.const 4)
+                              )
+                             )
+                             (local.get $3)
+                            )
+                            (local.tee $0
+                             (i32.and
+                              (i32.shr_u
+                               (i32.add
+                                (local.tee $1
+                                 (i32.shl
+                                  (local.get $1)
+                                  (local.get $0)
+                                 )
+                                )
+                                (i32.const 245760)
+                               )
+                               (i32.const 16)
+                              )
+                              (i32.const 2)
+                             )
+                            )
+                           )
+                          )
+                          (i32.shr_u
+                           (i32.shl
+                            (local.get $1)
+                            (local.get $0)
+                           )
+                           (i32.const 15)
+                          )
+                         )
+                        )
+                        (i32.const 7)
+                       )
+                      )
+                      (i32.const 1)
+                     )
+                     (i32.shl
+                      (local.get $0)
+                      (i32.const 1)
+                     )
+                    )
+                   )
+                   (i32.const 0)
+                  )
+                 )
+                )
+                (i32.const 2)
+               )
+               (i32.const 3948)
+              )
+             )
+             (i32.store offset=28
+              (local.get $6)
+              (local.get $2)
+             )
+             (i32.store offset=4
+              (local.tee $0
+               (i32.add
+                (local.get $6)
+                (i32.const 16)
+               )
+              )
+              (i32.const 0)
+             )
+             (i32.store
+              (local.get $0)
+              (i32.const 0)
+             )
+             (if
+              (i32.eqz
+               (i32.and
+                (local.tee $1
+                 (i32.load
+                  (i32.const 3648)
+                 )
+                )
+                (local.tee $0
+                 (i32.shl
+                  (i32.const 1)
+                  (local.get $2)
+                 )
+                )
+               )
+              )
+              (block
+               (i32.store
+                (i32.const 3648)
+                (i32.or
+                 (local.get $1)
+                 (local.get $0)
+                )
+               )
+               (i32.store
+                (local.get $3)
+                (local.get $6)
+               )
+               (i32.store offset=24
+                (local.get $6)
+                (local.get $3)
+               )
+               (i32.store offset=12
+                (local.get $6)
+                (local.get $6)
+               )
+               (i32.store offset=8
+                (local.get $6)
+                (local.get $6)
+               )
+               (br $label$217)
+              )
+             )
+             (local.set $0
+              (i32.load
+               (local.get $3)
+              )
+             )
+             (local.set $1
+              (i32.sub
+               (i32.const 25)
+               (i32.shr_u
+                (local.get $2)
+                (i32.const 1)
+               )
+              )
+             )
+             (local.set $2
+              (i32.shl
+               (local.get $7)
+               (if (result i32)
+                (i32.eq
+                 (local.get $2)
+                 (i32.const 31)
+                )
+                (i32.const 0)
+                (local.get $1)
+               )
+              )
+             )
+             (block $label$273
+              (block $label$274
+               (block $label$275
+                (loop $label$276
+                 (br_if $label$274
+                  (i32.eq
+                   (i32.and
+                    (i32.load offset=4
+                     (local.get $0)
+                    )
+                    (i32.const -8)
+                   )
+                   (local.get $7)
+                  )
+                 )
+                 (local.set $3
+                  (i32.shl
+                   (local.get $2)
+                   (i32.const 1)
+                  )
+                 )
+                 (br_if $label$275
+                  (i32.eqz
+                   (local.tee $1
+                    (i32.load
+                     (local.tee $2
+                      (i32.add
+                       (i32.add
+                        (local.get $0)
+                        (i32.const 16)
+                       )
+                       (i32.shl
+                        (i32.shr_u
+                         (local.get $2)
+                         (i32.const 31)
+                        )
+                        (i32.const 2)
+                       )
+                      )
+                     )
+                    )
+                   )
+                  )
+                 )
+                 (local.set $2
+                  (local.get $3)
+                 )
+                 (local.set $0
+                  (local.get $1)
+                 )
+                 (br $label$276)
+                )
+               )
+               (if
+                (i32.lt_u
+                 (local.get $2)
+                 (i32.load
+                  (i32.const 3660)
+                 )
+                )
+                (call $fimport$10)
+                (block
+                 (i32.store
+                  (local.get $2)
+                  (local.get $6)
+                 )
+                 (i32.store offset=24
+                  (local.get $6)
+                  (local.get $0)
+                 )
+                 (i32.store offset=12
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (i32.store offset=8
+                  (local.get $6)
+                  (local.get $6)
+                 )
+                 (br $label$217)
+                )
+               )
+               (br $label$273)
+              )
+              (if
+               (i32.and
+                (i32.ge_u
+                 (local.tee $2
+                  (i32.load
+                   (local.tee $3
+                    (i32.add
+                     (local.get $0)
+                     (i32.const 8)
+                    )
+                   )
+                  )
+                 )
+                 (local.tee $1
+                  (i32.load
+                   (i32.const 3660)
+                  )
+                 )
+                )
+                (i32.ge_u
+                 (local.get $0)
+                 (local.get $1)
+                )
+               )
+               (block
+                (i32.store offset=12
+                 (local.get $2)
+                 (local.get $6)
+                )
+                (i32.store
+                 (local.get $3)
+                 (local.get $6)
+                )
+                (i32.store offset=8
+                 (local.get $6)
+                 (local.get $2)
+                )
+                (i32.store offset=12
+                 (local.get $6)
+                 (local.get $0)
+                )
+                (i32.store offset=24
+                 (local.get $6)
+                 (i32.const 0)
+                )
+               )
+               (call $fimport$10)
+              )
+             )
+            )
+           )
+          )
+          (global.set $global$1
+           (local.get $14)
+          )
+          (return
+           (i32.add
+            (local.get $13)
+            (i32.const 8)
+           )
+          )
+         )
+        )
+       )
+       (loop $label$281
+        (block $label$282
+         (if
+          (i32.le_u
+           (local.tee $2
+            (i32.load
+             (local.get $5)
+            )
+           )
+           (local.get $8)
+          )
+          (br_if $label$282
+           (i32.gt_u
+            (local.tee $13
+             (i32.add
+              (local.get $2)
+              (i32.load offset=4
+               (local.get $5)
+              )
+             )
+            )
+            (local.get $8)
+           )
+          )
+         )
+         (local.set $5
+          (i32.load offset=8
+           (local.get $5)
+          )
+         )
+         (br $label$281)
+        )
+       )
+       (local.set $2
+        (i32.and
+         (i32.sub
+          (i32.const 0)
+          (local.tee $5
+           (i32.add
+            (local.tee $7
+             (i32.add
+              (local.get $13)
+              (i32.const -47)
+             )
+            )
+            (i32.const 8)
+           )
+          )
+         )
+         (i32.const 7)
+        )
+       )
+       (local.set $10
+        (i32.add
+         (local.tee $7
+          (if (result i32)
+           (i32.lt_u
+            (local.tee $2
+             (i32.add
+              (local.get $7)
+              (if (result i32)
+               (i32.and
+                (local.get $5)
+                (i32.const 7)
+               )
+               (local.get $2)
+               (i32.const 0)
+              )
+             )
+            )
+            (local.tee $12
+             (i32.add
+              (local.get $8)
+              (i32.const 16)
+             )
+            )
+           )
+           (local.get $8)
+           (local.get $2)
+          )
+         )
+         (i32.const 8)
+        )
+       )
+       (local.set $5
+        (i32.add
+         (local.get $7)
+         (i32.const 24)
+        )
+       )
+       (local.set $9
+        (i32.add
+         (local.get $3)
+         (i32.const -40)
+        )
+       )
+       (local.set $2
+        (i32.and
+         (i32.sub
+          (i32.const 0)
+          (local.tee $4
+           (i32.add
+            (local.get $1)
+            (i32.const 8)
+           )
+          )
+         )
+         (i32.const 7)
+        )
+       )
+       (i32.store
+        (i32.const 3668)
+        (local.tee $4
+         (i32.add
+          (local.get $1)
+          (if (result i32)
+           (i32.and
+            (local.get $4)
+            (i32.const 7)
+           )
+           (local.get $2)
+           (local.tee $2
+            (i32.const 0)
+           )
+          )
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3656)
+        (local.tee $2
+         (i32.sub
+          (local.get $9)
+          (local.get $2)
+         )
+        )
+       )
+       (i32.store offset=4
+        (local.get $4)
+        (i32.or
+         (local.get $2)
+         (i32.const 1)
+        )
+       )
+       (i32.store offset=4
+        (i32.add
+         (local.get $4)
+         (local.get $2)
+        )
+        (i32.const 40)
+       )
+       (i32.store
+        (i32.const 3672)
+        (i32.load
+         (i32.const 4132)
+        )
+       )
+       (i32.store
+        (local.tee $2
+         (i32.add
+          (local.get $7)
+          (i32.const 4)
+         )
+        )
+        (i32.const 27)
+       )
+       (i64.store align=4
+        (local.get $10)
+        (i64.load align=4
+         (i32.const 4092)
+        )
+       )
+       (i64.store offset=8 align=4
+        (local.get $10)
+        (i64.load align=4
+         (i32.const 4100)
+        )
+       )
+       (i32.store
+        (i32.const 4092)
+        (local.get $1)
+       )
+       (i32.store
+        (i32.const 4096)
+        (local.get $3)
+       )
+       (i32.store
+        (i32.const 4104)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 4100)
+        (local.get $10)
+       )
+       (local.set $1
+        (local.get $5)
+       )
+       (loop $label$290
+        (i32.store
+         (local.tee $1
+          (i32.add
+           (local.get $1)
+           (i32.const 4)
+          )
+         )
+         (i32.const 7)
+        )
+        (br_if $label$290
+         (i32.lt_u
+          (i32.add
+           (local.get $1)
+           (i32.const 4)
+          )
+          (local.get $13)
+         )
+        )
+       )
+       (if
+        (i32.ne
+         (local.get $7)
+         (local.get $8)
+        )
+        (block
+         (i32.store
+          (local.get $2)
+          (i32.and
+           (i32.load
+            (local.get $2)
+           )
+           (i32.const -2)
+          )
+         )
+         (i32.store offset=4
+          (local.get $8)
+          (i32.or
+           (local.tee $4
+            (i32.sub
+             (local.get $7)
+             (local.get $8)
+            )
+           )
+           (i32.const 1)
+          )
+         )
+         (i32.store
+          (local.get $7)
+          (local.get $4)
+         )
+         (local.set $1
+          (i32.shr_u
+           (local.get $4)
+           (i32.const 3)
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $4)
+           (i32.const 256)
+          )
+          (block
+           (local.set $2
+            (i32.add
+             (i32.shl
+              (i32.shl
+               (local.get $1)
+               (i32.const 1)
+              )
+              (i32.const 2)
+             )
+             (i32.const 3684)
+            )
+           )
+           (if
+            (i32.and
+             (local.tee $3
+              (i32.load
+               (i32.const 3644)
+              )
+             )
+             (local.tee $1
+              (i32.shl
+               (i32.const 1)
+               (local.get $1)
+              )
+             )
+            )
+            (if
+             (i32.lt_u
+              (local.tee $1
+               (i32.load
+                (local.tee $3
+                 (i32.add
+                  (local.get $2)
+                  (i32.const 8)
+                 )
+                )
+               )
+              )
+              (i32.load
+               (i32.const 3660)
+              )
+             )
+             (call $fimport$10)
+             (block
+              (local.set $15
+               (local.get $3)
+              )
+              (local.set $11
+               (local.get $1)
+              )
+             )
+            )
+            (block
+             (i32.store
+              (i32.const 3644)
+              (i32.or
+               (local.get $3)
+               (local.get $1)
+              )
+             )
+             (local.set $15
+              (i32.add
+               (local.get $2)
+               (i32.const 8)
+              )
+             )
+             (local.set $11
+              (local.get $2)
+             )
+            )
+           )
+           (i32.store
+            (local.get $15)
+            (local.get $8)
+           )
+           (i32.store offset=12
+            (local.get $11)
+            (local.get $8)
+           )
+           (i32.store offset=8
+            (local.get $8)
+            (local.get $11)
+           )
+           (i32.store offset=12
+            (local.get $8)
+            (local.get $2)
+           )
+           (br $label$198)
+          )
+         )
+         (local.set $2
+          (i32.add
+           (i32.shl
+            (local.tee $5
+             (if (result i32)
+              (local.tee $1
+               (i32.shr_u
+                (local.get $4)
+                (i32.const 8)
+               )
+              )
+              (if (result i32)
+               (i32.gt_u
+                (local.get $4)
+                (i32.const 16777215)
+               )
+               (i32.const 31)
+               (i32.or
+                (i32.and
+                 (i32.shr_u
+                  (local.get $4)
+                  (i32.add
+                   (local.tee $1
+                    (i32.add
+                     (i32.sub
+                      (i32.const 14)
+                      (i32.or
+                       (i32.or
+                        (local.tee $1
+                         (i32.and
+                          (i32.shr_u
+                           (i32.add
+                            (local.tee $3
+                             (i32.shl
+                              (local.get $1)
+                              (local.tee $2
+                               (i32.and
+                                (i32.shr_u
+                                 (i32.add
+                                  (local.get $1)
+                                  (i32.const 1048320)
+                                 )
+                                 (i32.const 16)
+                                )
+                                (i32.const 8)
+                               )
+                              )
+                             )
+                            )
+                            (i32.const 520192)
+                           )
+                           (i32.const 16)
+                          )
+                          (i32.const 4)
+                         )
+                        )
+                        (local.get $2)
+                       )
+                       (local.tee $1
+                        (i32.and
+                         (i32.shr_u
+                          (i32.add
+                           (local.tee $3
+                            (i32.shl
+                             (local.get $3)
+                             (local.get $1)
+                            )
+                           )
+                           (i32.const 245760)
+                          )
+                          (i32.const 16)
+                         )
+                         (i32.const 2)
+                        )
+                       )
+                      )
+                     )
+                     (i32.shr_u
+                      (i32.shl
+                       (local.get $3)
+                       (local.get $1)
+                      )
+                      (i32.const 15)
+                     )
+                    )
+                   )
+                   (i32.const 7)
+                  )
+                 )
+                 (i32.const 1)
+                )
+                (i32.shl
+                 (local.get $1)
+                 (i32.const 1)
+                )
+               )
+              )
+              (i32.const 0)
+             )
+            )
+            (i32.const 2)
+           )
+           (i32.const 3948)
+          )
+         )
+         (i32.store offset=28
+          (local.get $8)
+          (local.get $5)
+         )
+         (i32.store offset=20
+          (local.get $8)
+          (i32.const 0)
+         )
+         (i32.store
+          (local.get $12)
+          (i32.const 0)
+         )
+         (if
+          (i32.eqz
+           (i32.and
+            (local.tee $3
+             (i32.load
+              (i32.const 3648)
+             )
+            )
+            (local.tee $1
+             (i32.shl
+              (i32.const 1)
+              (local.get $5)
+             )
+            )
+           )
+          )
+          (block
+           (i32.store
+            (i32.const 3648)
+            (i32.or
+             (local.get $3)
+             (local.get $1)
+            )
+           )
+           (i32.store
+            (local.get $2)
+            (local.get $8)
+           )
+           (i32.store offset=24
+            (local.get $8)
+            (local.get $2)
+           )
+           (i32.store offset=12
+            (local.get $8)
+            (local.get $8)
+           )
+           (i32.store offset=8
+            (local.get $8)
+            (local.get $8)
+           )
+           (br $label$198)
+          )
+         )
+         (local.set $1
+          (i32.load
+           (local.get $2)
+          )
+         )
+         (local.set $3
+          (i32.sub
+           (i32.const 25)
+           (i32.shr_u
+            (local.get $5)
+            (i32.const 1)
+           )
+          )
+         )
+         (local.set $5
+          (i32.shl
+           (local.get $4)
+           (if (result i32)
+            (i32.eq
+             (local.get $5)
+             (i32.const 31)
+            )
+            (i32.const 0)
+            (local.get $3)
+           )
+          )
+         )
+         (block $label$304
+          (block $label$305
+           (block $label$306
+            (loop $label$307
+             (br_if $label$305
+              (i32.eq
+               (i32.and
+                (i32.load offset=4
+                 (local.get $1)
+                )
+                (i32.const -8)
+               )
+               (local.get $4)
+              )
+             )
+             (local.set $2
+              (i32.shl
+               (local.get $5)
+               (i32.const 1)
+              )
+             )
+             (br_if $label$306
+              (i32.eqz
+               (local.tee $3
+                (i32.load
+                 (local.tee $5
+                  (i32.add
+                   (i32.add
+                    (local.get $1)
+                    (i32.const 16)
+                   )
+                   (i32.shl
+                    (i32.shr_u
+                     (local.get $5)
+                     (i32.const 31)
+                    )
+                    (i32.const 2)
+                   )
+                  )
+                 )
+                )
+               )
+              )
+             )
+             (local.set $5
+              (local.get $2)
+             )
+             (local.set $1
+              (local.get $3)
+             )
+             (br $label$307)
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $5)
+             (i32.load
+              (i32.const 3660)
+             )
+            )
+            (call $fimport$10)
+            (block
+             (i32.store
+              (local.get $5)
+              (local.get $8)
+             )
+             (i32.store offset=24
+              (local.get $8)
+              (local.get $1)
+             )
+             (i32.store offset=12
+              (local.get $8)
+              (local.get $8)
+             )
+             (i32.store offset=8
+              (local.get $8)
+              (local.get $8)
+             )
+             (br $label$198)
+            )
+           )
+           (br $label$304)
+          )
+          (if
+           (i32.and
+            (i32.ge_u
+             (local.tee $5
+              (i32.load
+               (local.tee $2
+                (i32.add
+                 (local.get $1)
+                 (i32.const 8)
+                )
+               )
+              )
+             )
+             (local.tee $3
+              (i32.load
+               (i32.const 3660)
+              )
+             )
+            )
+            (i32.ge_u
+             (local.get $1)
+             (local.get $3)
+            )
+           )
+           (block
+            (i32.store offset=12
+             (local.get $5)
+             (local.get $8)
+            )
+            (i32.store
+             (local.get $2)
+             (local.get $8)
+            )
+            (i32.store offset=8
+             (local.get $8)
+             (local.get $5)
+            )
+            (i32.store offset=12
+             (local.get $8)
+             (local.get $1)
+            )
+            (i32.store offset=24
+             (local.get $8)
+             (i32.const 0)
+            )
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+       )
+      )
+      (block
+       (if
+        (i32.or
+         (i32.eqz
+          (local.tee $2
+           (i32.load
+            (i32.const 3660)
+           )
+          )
+         )
+         (i32.lt_u
+          (local.get $1)
+          (local.get $2)
+         )
+        )
+        (i32.store
+         (i32.const 3660)
+         (local.get $1)
+        )
+       )
+       (i32.store
+        (i32.const 4092)
+        (local.get $1)
+       )
+       (i32.store
+        (i32.const 4096)
+        (local.get $3)
+       )
+       (i32.store
+        (i32.const 4104)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 3680)
+        (i32.load
+         (i32.const 4116)
+        )
+       )
+       (i32.store
+        (i32.const 3676)
+        (i32.const -1)
+       )
+       (local.set $2
+        (i32.const 0)
+       )
+       (loop $label$314
+        (i32.store offset=12
+         (local.tee $5
+          (i32.add
+           (i32.shl
+            (i32.shl
+             (local.get $2)
+             (i32.const 1)
+            )
+            (i32.const 2)
+           )
+           (i32.const 3684)
+          )
+         )
+         (local.get $5)
+        )
+        (i32.store offset=8
+         (local.get $5)
+         (local.get $5)
+        )
+        (br_if $label$314
+         (i32.ne
+          (local.tee $2
+           (i32.add
+            (local.get $2)
+            (i32.const 1)
+           )
+          )
+          (i32.const 32)
+         )
+        )
+       )
+       (local.set $5
+        (i32.add
+         (local.get $3)
+         (i32.const -40)
+        )
+       )
+       (local.set $3
+        (i32.and
+         (i32.sub
+          (i32.const 0)
+          (local.tee $2
+           (i32.add
+            (local.get $1)
+            (i32.const 8)
+           )
+          )
+         )
+         (i32.const 7)
+        )
+       )
+       (i32.store
+        (i32.const 3668)
+        (local.tee $3
+         (i32.add
+          (local.get $1)
+          (local.tee $1
+           (if (result i32)
+            (i32.and
+             (local.get $2)
+             (i32.const 7)
+            )
+            (local.get $3)
+            (i32.const 0)
+           )
+          )
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3656)
+        (local.tee $1
+         (i32.sub
+          (local.get $5)
+          (local.get $1)
+         )
+        )
+       )
+       (i32.store offset=4
+        (local.get $3)
+        (i32.or
+         (local.get $1)
+         (i32.const 1)
+        )
+       )
+       (i32.store offset=4
+        (i32.add
+         (local.get $3)
+         (local.get $1)
+        )
+        (i32.const 40)
+       )
+       (i32.store
+        (i32.const 3672)
+        (i32.load
+         (i32.const 4132)
+        )
+       )
+      )
+     )
+    )
+    (if
+     (i32.gt_u
+      (local.tee $1
+       (i32.load
+        (i32.const 3656)
+       )
+      )
+      (local.get $0)
+     )
+     (block
+      (i32.store
+       (i32.const 3656)
+       (local.tee $3
+        (i32.sub
+         (local.get $1)
+         (local.get $0)
+        )
+       )
+      )
+      (i32.store
+       (i32.const 3668)
+       (local.tee $1
+        (i32.add
+         (local.tee $2
+          (i32.load
+           (i32.const 3668)
+          )
+         )
+         (local.get $0)
+        )
+       )
+      )
+      (i32.store offset=4
+       (local.get $1)
+       (i32.or
+        (local.get $3)
+        (i32.const 1)
+       )
+      )
+      (i32.store offset=4
+       (local.get $2)
+       (i32.or
+        (local.get $0)
+        (i32.const 3)
+       )
+      )
+      (global.set $global$1
+       (local.get $14)
+      )
+      (return
+       (i32.add
+        (local.get $2)
+        (i32.const 8)
+       )
+      )
+     )
+    )
+   )
+   (i32.store
+    (call $11)
+    (i32.const 12)
+   )
+   (global.set $global$1
+    (local.get $14)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $32 (; 45 ;) (type $2) (param $0 i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (block $label$1
+   (if
+    (i32.eqz
+     (local.get $0)
+    )
+    (return)
+   )
+   (if
+    (i32.lt_u
+     (local.tee $1
+      (i32.add
+       (local.get $0)
+       (i32.const -8)
+      )
+     )
+     (local.tee $11
+      (i32.load
+       (i32.const 3660)
+      )
+     )
+    )
+    (call $fimport$10)
+   )
+   (if
+    (i32.eq
+     (local.tee $8
+      (i32.and
+       (local.tee $0
+        (i32.load
+         (i32.add
+          (local.get $0)
+          (i32.const -4)
+         )
+        )
+       )
+       (i32.const 3)
+      )
+     )
+     (i32.const 1)
+    )
+    (call $fimport$10)
+   )
+   (local.set $6
+    (i32.add
+     (local.get $1)
+     (local.tee $4
+      (i32.and
+       (local.get $0)
+       (i32.const -8)
+      )
+     )
+    )
+   )
+   (block $label$5
+    (if
+     (i32.and
+      (local.get $0)
+      (i32.const 1)
+     )
+     (block
+      (local.set $3
+       (local.get $1)
+      )
+      (local.set $2
+       (local.get $4)
+      )
+     )
+     (block
+      (if
+       (i32.eqz
+        (local.get $8)
+       )
+       (return)
+      )
+      (if
+       (i32.lt_u
+        (local.tee $0
+         (i32.add
+          (local.get $1)
+          (i32.sub
+           (i32.const 0)
+           (local.tee $8
+            (i32.load
+             (local.get $1)
+            )
+           )
+          )
+         )
+        )
+        (local.get $11)
+       )
+       (call $fimport$10)
+      )
+      (local.set $1
+       (i32.add
+        (local.get $8)
+        (local.get $4)
+       )
+      )
+      (if
+       (i32.eq
+        (local.get $0)
+        (i32.load
+         (i32.const 3664)
+        )
+       )
+       (block
+        (if
+         (i32.ne
+          (i32.and
+           (local.tee $3
+            (i32.load
+             (local.tee $2
+              (i32.add
+               (local.get $6)
+               (i32.const 4)
+              )
+             )
+            )
+           )
+           (i32.const 3)
+          )
+          (i32.const 3)
+         )
+         (block
+          (local.set $3
+           (local.get $0)
+          )
+          (local.set $2
+           (local.get $1)
+          )
+          (br $label$5)
+         )
+        )
+        (i32.store
+         (i32.const 3652)
+         (local.get $1)
+        )
+        (i32.store
+         (local.get $2)
+         (i32.and
+          (local.get $3)
+          (i32.const -2)
+         )
+        )
+        (i32.store offset=4
+         (local.get $0)
+         (i32.or
+          (local.get $1)
+          (i32.const 1)
+         )
+        )
+        (i32.store
+         (i32.add
+          (local.get $0)
+          (local.get $1)
+         )
+         (local.get $1)
+        )
+        (return)
+       )
+      )
+      (local.set $10
+       (i32.shr_u
+        (local.get $8)
+        (i32.const 3)
+       )
+      )
+      (if
+       (i32.lt_u
+        (local.get $8)
+        (i32.const 256)
+       )
+       (block
+        (local.set $3
+         (i32.load offset=12
+          (local.get $0)
+         )
+        )
+        (if
+         (i32.ne
+          (local.tee $4
+           (i32.load offset=8
+            (local.get $0)
+           )
+          )
+          (local.tee $2
+           (i32.add
+            (i32.shl
+             (i32.shl
+              (local.get $10)
+              (i32.const 1)
+             )
+             (i32.const 2)
+            )
+            (i32.const 3684)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $4)
+            (local.get $11)
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.ne
+            (i32.load offset=12
+             (local.get $4)
+            )
+            (local.get $0)
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $3)
+          (local.get $4)
+         )
+         (block
+          (i32.store
+           (i32.const 3644)
+           (i32.and
+            (i32.load
+             (i32.const 3644)
+            )
+            (i32.xor
+             (i32.shl
+              (i32.const 1)
+              (local.get $10)
+             )
+             (i32.const -1)
+            )
+           )
+          )
+          (local.set $3
+           (local.get $0)
+          )
+          (local.set $2
+           (local.get $1)
+          )
+          (br $label$5)
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $3)
+          (local.get $2)
+         )
+         (local.set $5
+          (i32.add
+           (local.get $3)
+           (i32.const 8)
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $3)
+            (local.get $11)
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $2
+              (i32.add
+               (local.get $3)
+               (i32.const 8)
+              )
+             )
+            )
+            (local.get $0)
+           )
+           (local.set $5
+            (local.get $2)
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (i32.store offset=12
+         (local.get $4)
+         (local.get $3)
+        )
+        (i32.store
+         (local.get $5)
+         (local.get $4)
+        )
+        (local.set $3
+         (local.get $0)
+        )
+        (local.set $2
+         (local.get $1)
+        )
+        (br $label$5)
+       )
+      )
+      (local.set $12
+       (i32.load offset=24
+        (local.get $0)
+       )
+      )
+      (block $label$22
+       (if
+        (i32.eq
+         (local.tee $4
+          (i32.load offset=12
+           (local.get $0)
+          )
+         )
+         (local.get $0)
+        )
+        (block
+         (if
+          (local.tee $4
+           (i32.load
+            (local.tee $8
+             (i32.add
+              (local.tee $5
+               (i32.add
+                (local.get $0)
+                (i32.const 16)
+               )
+              )
+              (i32.const 4)
+             )
+            )
+           )
+          )
+          (local.set $5
+           (local.get $8)
+          )
+          (if
+           (i32.eqz
+            (local.tee $4
+             (i32.load
+              (local.get $5)
+             )
+            )
+           )
+           (block
+            (local.set $7
+             (i32.const 0)
+            )
+            (br $label$22)
+           )
+          )
+         )
+         (loop $label$27
+          (if
+           (local.tee $10
+            (i32.load
+             (local.tee $8
+              (i32.add
+               (local.get $4)
+               (i32.const 20)
+              )
+             )
+            )
+           )
+           (block
+            (local.set $4
+             (local.get $10)
+            )
+            (local.set $5
+             (local.get $8)
+            )
+            (br $label$27)
+           )
+          )
+          (if
+           (local.tee $10
+            (i32.load
+             (local.tee $8
+              (i32.add
+               (local.get $4)
+               (i32.const 16)
+              )
+             )
+            )
+           )
+           (block
+            (local.set $4
+             (local.get $10)
+            )
+            (local.set $5
+             (local.get $8)
+            )
+            (br $label$27)
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $5)
+           (local.get $11)
+          )
+          (call $fimport$10)
+          (block
+           (i32.store
+            (local.get $5)
+            (i32.const 0)
+           )
+           (local.set $7
+            (local.get $4)
+           )
+          )
+         )
+        )
+        (block
+         (if
+          (i32.lt_u
+           (local.tee $5
+            (i32.load offset=8
+             (local.get $0)
+            )
+           )
+           (local.get $11)
+          )
+          (call $fimport$10)
+         )
+         (if
+          (i32.ne
+           (i32.load
+            (local.tee $8
+             (i32.add
+              (local.get $5)
+              (i32.const 12)
+             )
+            )
+           )
+           (local.get $0)
+          )
+          (call $fimport$10)
+         )
+         (if
+          (i32.eq
+           (i32.load
+            (local.tee $10
+             (i32.add
+              (local.get $4)
+              (i32.const 8)
+             )
+            )
+           )
+           (local.get $0)
+          )
+          (block
+           (i32.store
+            (local.get $8)
+            (local.get $4)
+           )
+           (i32.store
+            (local.get $10)
+            (local.get $5)
+           )
+           (local.set $7
+            (local.get $4)
+           )
+          )
+          (call $fimport$10)
+         )
+        )
+       )
+      )
+      (if
+       (local.get $12)
+       (block
+        (if
+         (i32.eq
+          (local.get $0)
+          (i32.load
+           (local.tee $5
+            (i32.add
+             (i32.shl
+              (local.tee $4
+               (i32.load offset=28
+                (local.get $0)
+               )
+              )
+              (i32.const 2)
+             )
+             (i32.const 3948)
+            )
+           )
+          )
+         )
+         (block
+          (i32.store
+           (local.get $5)
+           (local.get $7)
+          )
+          (if
+           (i32.eqz
+            (local.get $7)
+           )
+           (block
+            (i32.store
+             (i32.const 3648)
+             (i32.and
+              (i32.load
+               (i32.const 3648)
+              )
+              (i32.xor
+               (i32.shl
+                (i32.const 1)
+                (local.get $4)
+               )
+               (i32.const -1)
+              )
+             )
+            )
+            (local.set $3
+             (local.get $0)
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (br $label$5)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $12)
+            (i32.load
+             (i32.const 3660)
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $4
+              (i32.add
+               (local.get $12)
+               (i32.const 16)
+              )
+             )
+            )
+            (local.get $0)
+           )
+           (i32.store
+            (local.get $4)
+            (local.get $7)
+           )
+           (i32.store offset=20
+            (local.get $12)
+            (local.get $7)
+           )
+          )
+          (if
+           (i32.eqz
+            (local.get $7)
+           )
+           (block
+            (local.set $3
+             (local.get $0)
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (br $label$5)
+           )
+          )
+         )
+        )
+        (if
+         (i32.lt_u
+          (local.get $7)
+          (local.tee $5
+           (i32.load
+            (i32.const 3660)
+           )
+          )
+         )
+         (call $fimport$10)
+        )
+        (i32.store offset=24
+         (local.get $7)
+         (local.get $12)
+        )
+        (if
+         (local.tee $4
+          (i32.load
+           (local.tee $8
+            (i32.add
+             (local.get $0)
+             (i32.const 16)
+            )
+           )
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $4)
+           (local.get $5)
+          )
+          (call $fimport$10)
+          (block
+           (i32.store offset=16
+            (local.get $7)
+            (local.get $4)
+           )
+           (i32.store offset=24
+            (local.get $4)
+            (local.get $7)
+           )
+          )
+         )
+        )
+        (if
+         (local.tee $4
+          (i32.load offset=4
+           (local.get $8)
+          )
+         )
+         (if
+          (i32.lt_u
+           (local.get $4)
+           (i32.load
+            (i32.const 3660)
+           )
+          )
+          (call $fimport$10)
+          (block
+           (i32.store offset=20
+            (local.get $7)
+            (local.get $4)
+           )
+           (i32.store offset=24
+            (local.get $4)
+            (local.get $7)
+           )
+           (local.set $3
+            (local.get $0)
+           )
+           (local.set $2
+            (local.get $1)
+           )
+          )
+         )
+         (block
+          (local.set $3
+           (local.get $0)
+          )
+          (local.set $2
+           (local.get $1)
+          )
+         )
+        )
+       )
+       (block
+        (local.set $3
+         (local.get $0)
+        )
+        (local.set $2
+         (local.get $1)
+        )
+       )
+      )
+     )
+    )
+   )
+   (if
+    (i32.ge_u
+     (local.get $3)
+     (local.get $6)
+    )
+    (call $fimport$10)
+   )
+   (if
+    (i32.eqz
+     (i32.and
+      (local.tee $0
+       (i32.load
+        (local.tee $1
+         (i32.add
+          (local.get $6)
+          (i32.const 4)
+         )
+        )
+       )
+      )
+      (i32.const 1)
+     )
+    )
+    (call $fimport$10)
+   )
+   (if
+    (i32.and
+     (local.get $0)
+     (i32.const 2)
+    )
+    (block
+     (i32.store
+      (local.get $1)
+      (i32.and
+       (local.get $0)
+       (i32.const -2)
+      )
+     )
+     (i32.store offset=4
+      (local.get $3)
+      (i32.or
+       (local.get $2)
+       (i32.const 1)
+      )
+     )
+     (i32.store
+      (i32.add
+       (local.get $3)
+       (local.get $2)
+      )
+      (local.get $2)
+     )
+    )
+    (block
+     (if
+      (i32.eq
+       (local.get $6)
+       (i32.load
+        (i32.const 3668)
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3656)
+        (local.tee $0
+         (i32.add
+          (i32.load
+           (i32.const 3656)
+          )
+          (local.get $2)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3668)
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $3)
+        (i32.or
+         (local.get $0)
+         (i32.const 1)
+        )
+       )
+       (if
+        (i32.ne
+         (local.get $3)
+         (i32.load
+          (i32.const 3664)
+         )
+        )
+        (return)
+       )
+       (i32.store
+        (i32.const 3664)
+        (i32.const 0)
+       )
+       (i32.store
+        (i32.const 3652)
+        (i32.const 0)
+       )
+       (return)
+      )
+     )
+     (if
+      (i32.eq
+       (local.get $6)
+       (i32.load
+        (i32.const 3664)
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3652)
+        (local.tee $0
+         (i32.add
+          (i32.load
+           (i32.const 3652)
+          )
+          (local.get $2)
+         )
+        )
+       )
+       (i32.store
+        (i32.const 3664)
+        (local.get $3)
+       )
+       (i32.store offset=4
+        (local.get $3)
+        (i32.or
+         (local.get $0)
+         (i32.const 1)
+        )
+       )
+       (i32.store
+        (i32.add
+         (local.get $3)
+         (local.get $0)
+        )
+        (local.get $0)
+       )
+       (return)
+      )
+     )
+     (local.set $5
+      (i32.add
+       (i32.and
+        (local.get $0)
+        (i32.const -8)
+       )
+       (local.get $2)
+      )
+     )
+     (local.set $4
+      (i32.shr_u
+       (local.get $0)
+       (i32.const 3)
+      )
+     )
+     (block $label$61
+      (if
+       (i32.lt_u
+        (local.get $0)
+        (i32.const 256)
+       )
+       (block
+        (local.set $2
+         (i32.load offset=12
+          (local.get $6)
+         )
+        )
+        (if
+         (i32.ne
+          (local.tee $1
+           (i32.load offset=8
+            (local.get $6)
+           )
+          )
+          (local.tee $0
+           (i32.add
+            (i32.shl
+             (i32.shl
+              (local.get $4)
+              (i32.const 1)
+             )
+             (i32.const 2)
+            )
+            (i32.const 3684)
+           )
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $1)
+            (i32.load
+             (i32.const 3660)
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.ne
+            (i32.load offset=12
+             (local.get $1)
+            )
+            (local.get $6)
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $2)
+          (local.get $1)
+         )
+         (block
+          (i32.store
+           (i32.const 3644)
+           (i32.and
+            (i32.load
+             (i32.const 3644)
+            )
+            (i32.xor
+             (i32.shl
+              (i32.const 1)
+              (local.get $4)
+             )
+             (i32.const -1)
+            )
+           )
+          )
+          (br $label$61)
+         )
+        )
+        (if
+         (i32.eq
+          (local.get $2)
+          (local.get $0)
+         )
+         (local.set $14
+          (i32.add
+           (local.get $2)
+           (i32.const 8)
+          )
+         )
+         (block
+          (if
+           (i32.lt_u
+            (local.get $2)
+            (i32.load
+             (i32.const 3660)
+            )
+           )
+           (call $fimport$10)
+          )
+          (if
+           (i32.eq
+            (i32.load
+             (local.tee $0
+              (i32.add
+               (local.get $2)
+               (i32.const 8)
+              )
+             )
+            )
+            (local.get $6)
+           )
+           (local.set $14
+            (local.get $0)
+           )
+           (call $fimport$10)
+          )
+         )
+        )
+        (i32.store offset=12
+         (local.get $1)
+         (local.get $2)
+        )
+        (i32.store
+         (local.get $14)
+         (local.get $1)
+        )
+       )
+       (block
+        (local.set $7
+         (i32.load offset=24
+          (local.get $6)
+         )
+        )
+        (block $label$73
+         (if
+          (i32.eq
+           (local.tee $0
+            (i32.load offset=12
+             (local.get $6)
+            )
+           )
+           (local.get $6)
+          )
+          (block
+           (if
+            (local.tee $0
+             (i32.load
+              (local.tee $1
+               (i32.add
+                (local.tee $2
+                 (i32.add
+                  (local.get $6)
+                  (i32.const 16)
+                 )
+                )
+                (i32.const 4)
+               )
+              )
+             )
+            )
+            (local.set $2
+             (local.get $1)
+            )
+            (if
+             (i32.eqz
+              (local.tee $0
+               (i32.load
+                (local.get $2)
+               )
+              )
+             )
+             (block
+              (local.set $9
+               (i32.const 0)
+              )
+              (br $label$73)
+             )
+            )
+           )
+           (loop $label$78
+            (if
+             (local.tee $4
+              (i32.load
+               (local.tee $1
+                (i32.add
+                 (local.get $0)
+                 (i32.const 20)
+                )
+               )
+              )
+             )
+             (block
+              (local.set $0
+               (local.get $4)
+              )
+              (local.set $2
+               (local.get $1)
+              )
+              (br $label$78)
+             )
+            )
+            (if
+             (local.tee $4
+              (i32.load
+               (local.tee $1
+                (i32.add
+                 (local.get $0)
+                 (i32.const 16)
+                )
+               )
+              )
+             )
+             (block
+              (local.set $0
+               (local.get $4)
+              )
+              (local.set $2
+               (local.get $1)
+              )
+              (br $label$78)
+             )
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $2)
+             (i32.load
+              (i32.const 3660)
+             )
+            )
+            (call $fimport$10)
+            (block
+             (i32.store
+              (local.get $2)
+              (i32.const 0)
+             )
+             (local.set $9
+              (local.get $0)
+             )
+            )
+           )
+          )
+          (block
+           (if
+            (i32.lt_u
+             (local.tee $2
+              (i32.load offset=8
+               (local.get $6)
+              )
+             )
+             (i32.load
+              (i32.const 3660)
+             )
+            )
+            (call $fimport$10)
+           )
+           (if
+            (i32.ne
+             (i32.load
+              (local.tee $1
+               (i32.add
+                (local.get $2)
+                (i32.const 12)
+               )
+              )
+             )
+             (local.get $6)
+            )
+            (call $fimport$10)
+           )
+           (if
+            (i32.eq
+             (i32.load
+              (local.tee $4
+               (i32.add
+                (local.get $0)
+                (i32.const 8)
+               )
+              )
+             )
+             (local.get $6)
+            )
+            (block
+             (i32.store
+              (local.get $1)
+              (local.get $0)
+             )
+             (i32.store
+              (local.get $4)
+              (local.get $2)
+             )
+             (local.set $9
+              (local.get $0)
+             )
+            )
+            (call $fimport$10)
+           )
+          )
+         )
+        )
+        (if
+         (local.get $7)
+         (block
+          (if
+           (i32.eq
+            (local.get $6)
+            (i32.load
+             (local.tee $2
+              (i32.add
+               (i32.shl
+                (local.tee $0
+                 (i32.load offset=28
+                  (local.get $6)
+                 )
+                )
+                (i32.const 2)
+               )
+               (i32.const 3948)
+              )
+             )
+            )
+           )
+           (block
+            (i32.store
+             (local.get $2)
+             (local.get $9)
+            )
+            (if
+             (i32.eqz
+              (local.get $9)
+             )
+             (block
+              (i32.store
+               (i32.const 3648)
+               (i32.and
+                (i32.load
+                 (i32.const 3648)
+                )
+                (i32.xor
+                 (i32.shl
+                  (i32.const 1)
+                  (local.get $0)
+                 )
+                 (i32.const -1)
+                )
+               )
+              )
+              (br $label$61)
+             )
+            )
+           )
+           (block
+            (if
+             (i32.lt_u
+              (local.get $7)
+              (i32.load
+               (i32.const 3660)
+              )
+             )
+             (call $fimport$10)
+            )
+            (if
+             (i32.eq
+              (i32.load
+               (local.tee $0
+                (i32.add
+                 (local.get $7)
+                 (i32.const 16)
+                )
+               )
+              )
+              (local.get $6)
+             )
+             (i32.store
+              (local.get $0)
+              (local.get $9)
+             )
+             (i32.store offset=20
+              (local.get $7)
+              (local.get $9)
+             )
+            )
+            (br_if $label$61
+             (i32.eqz
+              (local.get $9)
+             )
+            )
+           )
+          )
+          (if
+           (i32.lt_u
+            (local.get $9)
+            (local.tee $2
+             (i32.load
+              (i32.const 3660)
+             )
+            )
+           )
+           (call $fimport$10)
+          )
+          (i32.store offset=24
+           (local.get $9)
+           (local.get $7)
+          )
+          (if
+           (local.tee $0
+            (i32.load
+             (local.tee $1
+              (i32.add
+               (local.get $6)
+               (i32.const 16)
+              )
+             )
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $0)
+             (local.get $2)
+            )
+            (call $fimport$10)
+            (block
+             (i32.store offset=16
+              (local.get $9)
+              (local.get $0)
+             )
+             (i32.store offset=24
+              (local.get $0)
+              (local.get $9)
+             )
+            )
+           )
+          )
+          (if
+           (local.tee $0
+            (i32.load offset=4
+             (local.get $1)
+            )
+           )
+           (if
+            (i32.lt_u
+             (local.get $0)
+             (i32.load
+              (i32.const 3660)
+             )
+            )
+            (call $fimport$10)
+            (block
+             (i32.store offset=20
+              (local.get $9)
+              (local.get $0)
+             )
+             (i32.store offset=24
+              (local.get $0)
+              (local.get $9)
+             )
+            )
+           )
+          )
+         )
+        )
+       )
+      )
+     )
+     (i32.store offset=4
+      (local.get $3)
+      (i32.or
+       (local.get $5)
+       (i32.const 1)
+      )
+     )
+     (i32.store
+      (i32.add
+       (local.get $3)
+       (local.get $5)
+      )
+      (local.get $5)
+     )
+     (if
+      (i32.eq
+       (local.get $3)
+       (i32.load
+        (i32.const 3664)
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3652)
+        (local.get $5)
+       )
+       (return)
+      )
+      (local.set $2
+       (local.get $5)
+      )
+     )
+    )
+   )
+   (local.set $1
+    (i32.shr_u
+     (local.get $2)
+     (i32.const 3)
+    )
+   )
+   (if
+    (i32.lt_u
+     (local.get $2)
+     (i32.const 256)
+    )
+    (block
+     (local.set $0
+      (i32.add
+       (i32.shl
+        (i32.shl
+         (local.get $1)
+         (i32.const 1)
+        )
+        (i32.const 2)
+       )
+       (i32.const 3684)
+      )
+     )
+     (if
+      (i32.and
+       (local.tee $2
+        (i32.load
+         (i32.const 3644)
+        )
+       )
+       (local.tee $1
+        (i32.shl
+         (i32.const 1)
+         (local.get $1)
+        )
+       )
+      )
+      (if
+       (i32.lt_u
+        (local.tee $1
+         (i32.load
+          (local.tee $2
+           (i32.add
+            (local.get $0)
+            (i32.const 8)
+           )
+          )
+         )
+        )
+        (i32.load
+         (i32.const 3660)
+        )
+       )
+       (call $fimport$10)
+       (block
+        (local.set $15
+         (local.get $2)
+        )
+        (local.set $13
+         (local.get $1)
+        )
+       )
+      )
+      (block
+       (i32.store
+        (i32.const 3644)
+        (i32.or
+         (local.get $2)
+         (local.get $1)
+        )
+       )
+       (local.set $15
+        (i32.add
+         (local.get $0)
+         (i32.const 8)
+        )
+       )
+       (local.set $13
+        (local.get $0)
+       )
+      )
+     )
+     (i32.store
+      (local.get $15)
+      (local.get $3)
+     )
+     (i32.store offset=12
+      (local.get $13)
+      (local.get $3)
+     )
+     (i32.store offset=8
+      (local.get $3)
+      (local.get $13)
+     )
+     (i32.store offset=12
+      (local.get $3)
+      (local.get $0)
+     )
+     (return)
+    )
+   )
+   (local.set $0
+    (i32.add
+     (i32.shl
+      (local.tee $1
+       (if (result i32)
+        (local.tee $0
+         (i32.shr_u
+          (local.get $2)
+          (i32.const 8)
+         )
+        )
+        (if (result i32)
+         (i32.gt_u
+          (local.get $2)
+          (i32.const 16777215)
+         )
+         (i32.const 31)
+         (i32.or
+          (i32.and
+           (i32.shr_u
+            (local.get $2)
+            (i32.add
+             (local.tee $0
+              (i32.add
+               (i32.sub
+                (i32.const 14)
+                (i32.or
+                 (i32.or
+                  (local.tee $4
+                   (i32.and
+                    (i32.shr_u
+                     (i32.add
+                      (local.tee $1
+                       (i32.shl
+                        (local.get $0)
+                        (local.tee $0
+                         (i32.and
+                          (i32.shr_u
+                           (i32.add
+                            (local.get $0)
+                            (i32.const 1048320)
+                           )
+                           (i32.const 16)
+                          )
+                          (i32.const 8)
+                         )
+                        )
+                       )
+                      )
+                      (i32.const 520192)
+                     )
+                     (i32.const 16)
+                    )
+                    (i32.const 4)
+                   )
+                  )
+                  (local.get $0)
+                 )
+                 (local.tee $1
+                  (i32.and
+                   (i32.shr_u
+                    (i32.add
+                     (local.tee $0
+                      (i32.shl
+                       (local.get $1)
+                       (local.get $4)
+                      )
+                     )
+                     (i32.const 245760)
+                    )
+                    (i32.const 16)
+                   )
+                   (i32.const 2)
+                  )
+                 )
+                )
+               )
+               (i32.shr_u
+                (i32.shl
+                 (local.get $0)
+                 (local.get $1)
+                )
+                (i32.const 15)
+               )
+              )
+             )
+             (i32.const 7)
+            )
+           )
+           (i32.const 1)
+          )
+          (i32.shl
+           (local.get $0)
+           (i32.const 1)
+          )
+         )
+        )
+        (i32.const 0)
+       )
+      )
+      (i32.const 2)
+     )
+     (i32.const 3948)
+    )
+   )
+   (i32.store offset=28
+    (local.get $3)
+    (local.get $1)
+   )
+   (i32.store offset=20
+    (local.get $3)
+    (i32.const 0)
+   )
+   (i32.store offset=16
+    (local.get $3)
+    (i32.const 0)
+   )
+   (block $label$113
+    (if
+     (i32.and
+      (local.tee $4
+       (i32.load
+        (i32.const 3648)
+       )
+      )
+      (local.tee $5
+       (i32.shl
+        (i32.const 1)
+        (local.get $1)
+       )
+      )
+     )
+     (block
+      (local.set $0
+       (i32.load
+        (local.get $0)
+       )
+      )
+      (local.set $4
+       (i32.sub
+        (i32.const 25)
+        (i32.shr_u
+         (local.get $1)
+         (i32.const 1)
+        )
+       )
+      )
+      (local.set $1
+       (i32.shl
+        (local.get $2)
+        (if (result i32)
+         (i32.eq
+          (local.get $1)
+          (i32.const 31)
+         )
+         (i32.const 0)
+         (local.get $4)
+        )
+       )
+      )
+      (block $label$117
+       (block $label$118
+        (block $label$119
+         (loop $label$120
+          (br_if $label$118
+           (i32.eq
+            (i32.and
+             (i32.load offset=4
+              (local.get $0)
+             )
+             (i32.const -8)
+            )
+            (local.get $2)
+           )
+          )
+          (local.set $4
+           (i32.shl
+            (local.get $1)
+            (i32.const 1)
+           )
+          )
+          (br_if $label$119
+           (i32.eqz
+            (local.tee $5
+             (i32.load
+              (local.tee $1
+               (i32.add
+                (i32.add
+                 (local.get $0)
+                 (i32.const 16)
+                )
+                (i32.shl
+                 (i32.shr_u
+                  (local.get $1)
+                  (i32.const 31)
+                 )
+                 (i32.const 2)
+                )
+               )
+              )
+             )
+            )
+           )
+          )
+          (local.set $1
+           (local.get $4)
+          )
+          (local.set $0
+           (local.get $5)
+          )
+          (br $label$120)
+         )
+        )
+        (if
+         (i32.lt_u
+          (local.get $1)
+          (i32.load
+           (i32.const 3660)
+          )
+         )
+         (call $fimport$10)
+         (block
+          (i32.store
+           (local.get $1)
+           (local.get $3)
+          )
+          (i32.store offset=24
+           (local.get $3)
+           (local.get $0)
+          )
+          (i32.store offset=12
+           (local.get $3)
+           (local.get $3)
+          )
+          (i32.store offset=8
+           (local.get $3)
+           (local.get $3)
+          )
+          (br $label$113)
+         )
+        )
+        (br $label$117)
+       )
+       (if
+        (i32.and
+         (i32.ge_u
+          (local.tee $2
+           (i32.load
+            (local.tee $1
+             (i32.add
+              (local.get $0)
+              (i32.const 8)
+             )
+            )
+           )
+          )
+          (local.tee $4
+           (i32.load
+            (i32.const 3660)
+           )
+          )
+         )
+         (i32.ge_u
+          (local.get $0)
+          (local.get $4)
+         )
+        )
+        (block
+         (i32.store offset=12
+          (local.get $2)
+          (local.get $3)
+         )
+         (i32.store
+          (local.get $1)
+          (local.get $3)
+         )
+         (i32.store offset=8
+          (local.get $3)
+          (local.get $2)
+         )
+         (i32.store offset=12
+          (local.get $3)
+          (local.get $0)
+         )
+         (i32.store offset=24
+          (local.get $3)
+          (i32.const 0)
+         )
+        )
+        (call $fimport$10)
+       )
+      )
+     )
+     (block
+      (i32.store
+       (i32.const 3648)
+       (i32.or
+        (local.get $4)
+        (local.get $5)
+       )
+      )
+      (i32.store
+       (local.get $0)
+       (local.get $3)
+      )
+      (i32.store offset=24
+       (local.get $3)
+       (local.get $0)
+      )
+      (i32.store offset=12
+       (local.get $3)
+       (local.get $3)
+      )
+      (i32.store offset=8
+       (local.get $3)
+       (local.get $3)
+      )
+     )
+    )
+   )
+   (i32.store
+    (i32.const 3676)
+    (local.tee $0
+     (i32.add
+      (i32.load
+       (i32.const 3676)
+      )
+      (i32.const -1)
+     )
+    )
+   )
+   (if
+    (local.get $0)
+    (return)
+    (local.set $0
+     (i32.const 4100)
+    )
+   )
+   (loop $label$128
+    (local.set $0
+     (i32.add
+      (local.tee $2
+       (i32.load
+        (local.get $0)
+       )
+      )
+      (i32.const 8)
+     )
+    )
+    (br_if $label$128
+     (local.get $2)
+    )
+   )
+   (i32.store
+    (i32.const 3676)
+    (i32.const -1)
+   )
+  )
+ )
+ (func $33 (; 46 ;) (type $6)
+  (nop)
+ )
+ (func $34 (; 47 ;) (type $1) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (block $label$1 (result i32)
+   (local.set $1
+    (i32.add
+     (local.tee $2
+      (i32.load
+       (global.get $global$0)
+      )
+     )
+     (local.tee $0
+      (i32.and
+       (i32.add
+        (local.get $0)
+        (i32.const 15)
+       )
+       (i32.const -16)
+      )
+     )
+    )
+   )
+   (if
+    (i32.or
+     (i32.and
+      (i32.gt_s
+       (local.get $0)
+       (i32.const 0)
+      )
+      (i32.lt_s
+       (local.get $1)
+       (local.get $2)
+      )
+     )
+     (i32.lt_s
+      (local.get $1)
+      (i32.const 0)
+     )
+    )
+    (block
+     (drop
+      (call $fimport$6)
+     )
+     (call $fimport$11
+      (i32.const 12)
+     )
+     (return
+      (i32.const -1)
+     )
+    )
+   )
+   (i32.store
+    (global.get $global$0)
+    (local.get $1)
+   )
+   (if
+    (i32.gt_s
+     (local.get $1)
+     (call $fimport$5)
+    )
+    (if
+     (i32.eqz
+      (call $fimport$4)
+     )
+     (block
+      (call $fimport$11
+       (i32.const 12)
+      )
+      (i32.store
+       (global.get $global$0)
+       (local.get $2)
+      )
+      (return
+       (i32.const -1)
+      )
+     )
+    )
+   )
+   (local.get $2)
+  )
+ )
+ (func $35 (; 48 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (block $label$1 (result i32)
+   (local.set $4
+    (i32.add
+     (local.get $0)
+     (local.get $2)
+    )
+   )
+   (if
+    (i32.ge_s
+     (local.get $2)
+     (i32.const 20)
+    )
+    (block
+     (local.set $1
+      (i32.and
+       (local.get $1)
+       (i32.const 255)
+      )
+     )
+     (if
+      (local.tee $3
+       (i32.and
+        (local.get $0)
+        (i32.const 3)
+       )
+      )
+      (block
+       (local.set $3
+        (i32.sub
+         (i32.add
+          (local.get $0)
+          (i32.const 4)
+         )
+         (local.get $3)
+        )
+       )
+       (loop $label$4
+        (if
+         (i32.lt_s
+          (local.get $0)
+          (local.get $3)
+         )
+         (block
+          (i32.store8
+           (local.get $0)
+           (local.get $1)
+          )
+          (local.set $0
+           (i32.add
+            (local.get $0)
+            (i32.const 1)
+           )
+          )
+          (br $label$4)
+         )
+        )
+       )
+      )
+     )
+     (local.set $3
+      (i32.or
+       (i32.or
+        (i32.or
+         (local.get $1)
+         (i32.shl
+          (local.get $1)
+          (i32.const 8)
+         )
+        )
+        (i32.shl
+         (local.get $1)
+         (i32.const 16)
+        )
+       )
+       (i32.shl
+        (local.get $1)
+        (i32.const 24)
+       )
+      )
+     )
+     (local.set $5
+      (i32.and
+       (local.get $4)
+       (i32.const -4)
+      )
+     )
+     (loop $label$6
+      (if
+       (i32.lt_s
+        (local.get $0)
+        (local.get $5)
+       )
+       (block
+        (i32.store
+         (local.get $0)
+         (local.get $3)
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (i32.const 4)
+         )
+        )
+        (br $label$6)
+       )
+      )
+     )
+    )
+   )
+   (loop $label$8
+    (if
+     (i32.lt_s
+      (local.get $0)
+      (local.get $4)
+     )
+     (block
+      (i32.store8
+       (local.get $0)
+       (local.get $1)
+      )
+      (local.set $0
+       (i32.add
+        (local.get $0)
+        (i32.const 1)
+       )
+      )
+      (br $label$8)
+     )
+    )
+   )
+   (i32.sub
+    (local.get $0)
+    (local.get $2)
+   )
+  )
+ )
+ (func $36 (; 49 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (local $3 i32)
+  (block $label$1 (result i32)
+   (if
+    (i32.ge_s
+     (local.get $2)
+     (i32.const 4096)
+    )
+    (return
+     (call $fimport$12
+      (local.get $0)
+      (local.get $1)
+      (local.get $2)
+     )
+    )
+   )
+   (local.set $3
+    (local.get $0)
+   )
+   (if
+    (i32.eq
+     (i32.and
+      (local.get $0)
+      (i32.const 3)
+     )
+     (i32.and
+      (local.get $1)
+      (i32.const 3)
+     )
+    )
+    (block
+     (loop $label$4
+      (if
+       (i32.and
+        (local.get $0)
+        (i32.const 3)
+       )
+       (block
+        (if
+         (i32.eqz
+          (local.get $2)
+         )
+         (return
+          (local.get $3)
+         )
+        )
+        (i32.store8
+         (local.get $0)
+         (i32.load8_s
+          (local.get $1)
+         )
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (i32.const 1)
+         )
+        )
+        (local.set $1
+         (i32.add
+          (local.get $1)
+          (i32.const 1)
+         )
+        )
+        (local.set $2
+         (i32.sub
+          (local.get $2)
+          (i32.const 1)
+         )
+        )
+        (br $label$4)
+       )
+      )
+     )
+     (loop $label$7
+      (if
+       (i32.ge_s
+        (local.get $2)
+        (i32.const 4)
+       )
+       (block
+        (i32.store
+         (local.get $0)
+         (i32.load
+          (local.get $1)
+         )
+        )
+        (local.set $0
+         (i32.add
+          (local.get $0)
+          (i32.const 4)
+         )
+        )
+        (local.set $1
+         (i32.add
+          (local.get $1)
+          (i32.const 4)
+         )
+        )
+        (local.set $2
+         (i32.sub
+          (local.get $2)
+          (i32.const 4)
+         )
+        )
+        (br $label$7)
+       )
+      )
+     )
+    )
+   )
+   (loop $label$9
+    (if
+     (i32.gt_s
+      (local.get $2)
+      (i32.const 0)
+     )
+     (block
+      (i32.store8
+       (local.get $0)
+       (i32.load8_s
+        (local.get $1)
+       )
+      )
+      (local.set $0
+       (i32.add
+        (local.get $0)
+        (i32.const 1)
+       )
+      )
+      (local.set $1
+       (i32.add
+        (local.get $1)
+        (i32.const 1)
+       )
+      )
+      (local.set $2
+       (i32.sub
+        (local.get $2)
+        (i32.const 1)
+       )
+      )
+      (br $label$9)
+     )
+    )
+   )
+   (local.get $3)
+  )
+ )
+ (func $37 (; 50 ;) (type $3) (result i32)
+  (i32.const 0)
+ )
+ (func $38 (; 51 ;) (type $4) (param $0 i32) (param $1 i32) (result i32)
+  (call_indirect (type $1)
+   (local.get $1)
+   (i32.add
+    (i32.and
+     (local.get $0)
+     (i32.const 1)
+    )
+    (i32.const 0)
+   )
+  )
+ )
+ (func $39 (; 52 ;) (type $12) (param $0 i32) (param $1 i32) (param $2 i32) (param $3 i32) (result i32)
+  (call_indirect (type $0)
+   (local.get $1)
+   (local.get $2)
+   (local.get $3)
+   (i32.add
+    (i32.and
+     (local.get $0)
+     (i32.const 3)
+    )
+    (i32.const 2)
+   )
+  )
+ )
+ (func $40 (; 53 ;) (type $5) (param $0 i32) (param $1 i32)
+  (call_indirect (type $2)
+   (local.get $1)
+   (i32.add
+    (i32.and
+     (local.get $0)
+     (i32.const 1)
+    )
+    (i32.const 6)
+   )
+  )
+ )
+ (func $41 (; 54 ;) (type $1) (param $0 i32) (result i32)
+  (block $label$1 (result i32)
+   (call $fimport$3
+    (i32.const 0)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $42 (; 55 ;) (type $0) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (block $label$1 (result i32)
+   (call $fimport$3
+    (i32.const 1)
+   )
+   (i32.const 0)
+  )
+ )
+ (func $43 (; 56 ;) (type $2) (param $0 i32)
+  (call $fimport$3
+   (i32.const 2)
+  )
+ )
+)
+
diff --git a/cranelift/wasmtests/rust_fannkuch.wat b/cranelift/wasmtests/rust_fannkuch.wat
new file mode 100644
index 0000000000..cd47f661c8
--- /dev/null
+++ b/cranelift/wasmtests/rust_fannkuch.wat
@@ -0,0 +1,2511 @@
+(module
+ (type $0 (func (param i32 i32 i32) (result i32)))
+ (type $1 (func (param i32 i32) (result i32)))
+ (type $2 (func (param i32)))
+ (type $3 (func (param i32) (result i32)))
+ (type $4 (func (param i32 i32)))
+ (type $5 (func (param i64 i32) (result i32)))
+ (type $6 (func (param i32) (result i64)))
+ (type $7 (func))
+ (type $8 (func (param i32 i32)))
+ (type $9 (func (param i32 i32 i32) (result i32)))
+ (memory $0 17)
+ (data (i32.const 1048576) "src/lib.rs\00\00\00\00\00\00attempt to divide by zero\00\00\00\00\00\00\00attempt to divide with overflow\00index out of bounds: the len is  but the index is 00010203040506070809101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899called `Option::unwrap()` on a `None` valuesrc/libcore/option.rssrc/lib.rs")
+ (data (i32.const 1048982) "\10\00\n\00\00\00%\00\00\00\1d\00\00\00\10\00\10\00\19\00\00\00\00\00\10\00\n\00\00\00&\00\00\00\15\00\00\000\00\10\00\1f\00\00\00\00\00\10\00\n\00\00\00&\00\00\00\15\00\00\00\00\00\10\00\n\00\00\00.\00\00\00\15\00\00\00\00\00\10\00\n\00\00\000\00\00\00\15\00\00\00\00\00\10\00\n\00\00\00-\00\00\00\11\00\00\00\00\00\10\00\n\00\00\00E\00\00\00\17\00\00\00\00\00\10\00\n\00\00\00q\00\00\00\"\00\00\00\00\00\10\00\n\00\00\00s\00\00\00\11\00\00\00P\00\10\00 \00\00\00p\00\10\00\12\00\00\00\02\00\00\00\00\00\00\00\01\00\00\00\03\00\00\00J\01\10\00+\00\00\00u\01\10\00\15\00\00\00Y\01\00\00\15\00\00\00\8a\01\10\00\n\00\00\00\08\00\00\00\t\00\00\00\8a\01\10\00\n\00\00\00\n\00\00\00\14")
+ (table $0 4 4 funcref)
+ (elem (i32.const 1) $4 $7 $8)
+ (global $global$0 (mut i32) (i32.const 1048576))
+ (global $global$1 i32 (i32.const 1049244))
+ (global $global$2 i32 (i32.const 1049244))
+ (export "memory" (memory $0))
+ (export "__heap_base" (global $global$1))
+ (export "__data_end" (global $global$2))
+ (export "run_fannkuch" (func $10))
+ (func $0 (; 0 ;) (type $7)
+  (local $0 i32)
+  (local $1 i32)
+  (local.set $0
+   (i32.const 1)
+  )
+  (block $label$1
+   (block $label$2
+    (block $label$3
+     (if
+      (i32.eq
+       (i32.load
+        (i32.const 1049232)
+       )
+       (i32.const 1)
+      )
+      (block
+       (i32.store
+        (i32.const 1049236)
+        (local.tee $0
+         (i32.add
+          (i32.load
+           (i32.const 1049236)
+          )
+          (i32.const 1)
+         )
+        )
+       )
+       (br_if $label$3
+        (i32.lt_u
+         (local.get $0)
+         (i32.const 3)
+        )
+       )
+       (br $label$2)
+      )
+     )
+     (i64.store
+      (i32.const 1049232)
+      (i64.const 4294967297)
+     )
+    )
+    (br_if $label$2
+     (i32.le_s
+      (local.tee $1
+       (i32.load
+        (i32.const 1049240)
+       )
+      )
+      (i32.const -1)
+     )
+    )
+    (i32.store
+     (i32.const 1049240)
+     (local.get $1)
+    )
+    (br_if $label$1
+     (i32.lt_u
+      (local.get $0)
+      (i32.const 2)
+     )
+    )
+   )
+   (unreachable)
+  )
+  (unreachable)
+ )
+ (func $1 (; 1 ;) (type $2) (param $0 i32)
+  (local $1 i32)
+  (global.set $global$0
+   (local.tee $1
+    (i32.sub
+     (global.get $global$0)
+     (i32.const 16)
+    )
+   )
+  )
+  (if
+   (i32.eqz
+    (i32.load offset=8
+     (local.get $0)
+    )
+   )
+   (block
+    (call $2
+     (i32.const 1049172)
+    )
+    (unreachable)
+   )
+  )
+  (i64.store offset=8
+   (local.get $1)
+   (i64.load align=4
+    (i32.add
+     (local.get $0)
+     (i32.const 20)
+    )
+   )
+  )
+  (i64.store
+   (local.get $1)
+   (i64.load offset=12 align=4
+    (local.get $0)
+   )
+  )
+  (call $0)
+  (unreachable)
+ )
+ (func $2 (; 2 ;) (type $2) (param $0 i32)
+  (local $1 i32)
+  (local $2 i64)
+  (local $3 i64)
+  (local $4 i64)
+  (global.set $global$0
+   (local.tee $1
+    (i32.sub
+     (global.get $global$0)
+     (i32.const 48)
+    )
+   )
+  )
+  (local.set $2
+   (i64.load offset=8 align=4
+    (local.get $0)
+   )
+  )
+  (local.set $3
+   (i64.load offset=16 align=4
+    (local.get $0)
+   )
+  )
+  (local.set $4
+   (i64.load align=4
+    (local.get $0)
+   )
+  )
+  (i32.store
+   (i32.add
+    (local.get $1)
+    (i32.const 20)
+   )
+   (i32.const 0)
+  )
+  (i64.store offset=24
+   (local.get $1)
+   (local.get $4)
+  )
+  (i32.store offset=16
+   (local.get $1)
+   (i32.const 1048656)
+  )
+  (i64.store offset=4 align=4
+   (local.get $1)
+   (i64.const 1)
+  )
+  (i32.store
+   (local.get $1)
+   (i32.add
+    (local.get $1)
+    (i32.const 24)
+   )
+  )
+  (i64.store offset=40
+   (local.get $1)
+   (local.get $3)
+  )
+  (i64.store offset=32
+   (local.get $1)
+   (local.get $2)
+  )
+  (call $5
+   (local.get $1)
+   (i32.add
+    (local.get $1)
+    (i32.const 32)
+   )
+  )
+  (unreachable)
+ )
+ (func $3 (; 3 ;) (type $8) (param $0 i32) (param $1 i32)
+  (local $2 i32)
+  (global.set $global$0
+   (local.tee $2
+    (i32.sub
+     (global.get $global$0)
+     (i32.const 48)
+    )
+   )
+  )
+  (i32.store offset=4
+   (local.get $2)
+   (i32.const 16)
+  )
+  (i32.store
+   (local.get $2)
+   (local.get $1)
+  )
+  (i32.store
+   (i32.add
+    (local.get $2)
+    (i32.const 44)
+   )
+   (i32.const 1)
+  )
+  (i32.store
+   (i32.add
+    (local.get $2)
+    (i32.const 28)
+   )
+   (i32.const 2)
+  )
+  (i32.store offset=36
+   (local.get $2)
+   (i32.const 1)
+  )
+  (i64.store offset=12 align=4
+   (local.get $2)
+   (i64.const 2)
+  )
+  (i32.store offset=8
+   (local.get $2)
+   (i32.const 1049140)
+  )
+  (i32.store offset=40
+   (local.get $2)
+   (local.get $2)
+  )
+  (i32.store offset=32
+   (local.get $2)
+   (i32.add
+    (local.get $2)
+    (i32.const 4)
+   )
+  )
+  (i32.store offset=24
+   (local.get $2)
+   (i32.add
+    (local.get $2)
+    (i32.const 32)
+   )
+  )
+  (call $5
+   (i32.add
+    (local.get $2)
+    (i32.const 8)
+   )
+   (local.get $0)
+  )
+  (unreachable)
+ )
+ (func $4 (; 4 ;) (type $1) (param $0 i32) (param $1 i32) (result i32)
+  (call $6
+   (i64.load32_u
+    (local.get $0)
+   )
+   (local.get $1)
+  )
+ )
+ (func $5 (; 5 ;) (type $4) (param $0 i32) (param $1 i32)
+  (local $2 i32)
+  (local $3 i64)
+  (global.set $global$0
+   (local.tee $2
+    (i32.sub
+     (global.get $global$0)
+     (i32.const 32)
+    )
+   )
+  )
+  (local.set $3
+   (i64.load align=4
+    (local.get $1)
+   )
+  )
+  (i64.store align=4
+   (i32.add
+    (local.get $2)
+    (i32.const 20)
+   )
+   (i64.load offset=8 align=4
+    (local.get $1)
+   )
+  )
+  (i64.store offset=12 align=4
+   (local.get $2)
+   (local.get $3)
+  )
+  (i32.store offset=8
+   (local.get $2)
+   (local.get $0)
+  )
+  (i32.store offset=4
+   (local.get $2)
+   (i32.const 1049156)
+  )
+  (i32.store
+   (local.get $2)
+   (i32.const 1048656)
+  )
+  (call $1
+   (local.get $2)
+  )
+  (unreachable)
+ )
+ (func $6 (; 6 ;) (type $5) (param $0 i64) (param $1 i32) (result i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i64)
+  (local $14 i32)
+  (local $15 i32)
+  (global.set $global$0
+   (local.tee $6
+    (i32.sub
+     (global.get $global$0)
+     (i32.const 48)
+    )
+   )
+  )
+  (local.set $2
+   (i32.const 39)
+  )
+  (block $label$1
+   (block $label$2
+    (if
+     (i64.ge_u
+      (local.get $0)
+      (i64.const 10000)
+     )
+     (block
+      (loop $label$4
+       (i32.store16 align=1
+        (i32.add
+         (local.tee $3
+          (i32.add
+           (i32.add
+            (local.get $6)
+            (i32.const 9)
+           )
+           (local.get $2)
+          )
+         )
+         (i32.const -4)
+        )
+        (i32.load16_u align=1
+         (i32.add
+          (i32.shl
+           (local.tee $5
+            (i32.div_u
+             (local.tee $4
+              (i32.wrap_i64
+               (i64.add
+                (local.get $0)
+                (i64.mul
+                 (local.tee $13
+                  (i64.div_u
+                   (local.get $0)
+                   (i64.const 10000)
+                  )
+                 )
+                 (i64.const -10000)
+                )
+               )
+              )
+             )
+             (i32.const 100)
+            )
+           )
+           (i32.const 1)
+          )
+          (i32.const 1048706)
+         )
+        )
+       )
+       (i32.store16 align=1
+        (i32.add
+         (local.get $3)
+         (i32.const -2)
+        )
+        (i32.load16_u align=1
+         (i32.add
+          (i32.shl
+           (i32.add
+            (i32.mul
+             (local.get $5)
+             (i32.const -100)
+            )
+            (local.get $4)
+           )
+           (i32.const 1)
+          )
+          (i32.const 1048706)
+         )
+        )
+       )
+       (local.set $2
+        (i32.add
+         (local.get $2)
+         (i32.const -4)
+        )
+       )
+       (br_if $label$4
+        (block (result i32)
+         (local.set $14
+          (i64.gt_u
+           (local.get $0)
+           (i64.const 99999999)
+          )
+         )
+         (local.set $0
+          (local.get $13)
+         )
+         (local.get $14)
+        )
+       )
+      )
+      (br_if $label$1
+       (i32.le_s
+        (local.tee $3
+         (i32.wrap_i64
+          (local.get $13)
+         )
+        )
+        (i32.const 99)
+       )
+      )
+      (br $label$2)
+     )
+    )
+    (br_if $label$1
+     (i32.le_s
+      (local.tee $3
+       (i32.wrap_i64
+        (local.tee $13
+         (local.get $0)
+        )
+       )
+      )
+      (i32.const 99)
+     )
+    )
+   )
+   (i32.store16 align=1
+    (i32.add
+     (local.tee $2
+      (i32.add
+       (local.get $2)
+       (i32.const -2)
+      )
+     )
+     (i32.add
+      (local.get $6)
+      (i32.const 9)
+     )
+    )
+    (i32.load16_u align=1
+     (i32.add
+      (i32.shl
+       (i32.and
+        (i32.add
+         (i32.mul
+          (local.tee $3
+           (i32.div_u
+            (i32.and
+             (local.tee $4
+              (i32.wrap_i64
+               (local.get $13)
+              )
+             )
+             (i32.const 65535)
+            )
+            (i32.const 100)
+           )
+          )
+          (i32.const -100)
+         )
+         (local.get $4)
+        )
+        (i32.const 65535)
+       )
+       (i32.const 1)
+      )
+      (i32.const 1048706)
+     )
+    )
+   )
+  )
+  (block $label$5
+   (if
+    (i32.le_s
+     (local.get $3)
+     (i32.const 9)
+    )
+    (block
+     (i32.store8
+      (i32.add
+       (local.tee $2
+        (i32.add
+         (local.get $2)
+         (i32.const -1)
+        )
+       )
+       (i32.add
+        (local.get $6)
+        (i32.const 9)
+       )
+      )
+      (i32.add
+       (local.get $3)
+       (i32.const 48)
+      )
+     )
+     (br $label$5)
+    )
+   )
+   (i32.store16 align=1
+    (i32.add
+     (local.tee $2
+      (i32.add
+       (local.get $2)
+       (i32.const -2)
+      )
+     )
+     (i32.add
+      (local.get $6)
+      (i32.const 9)
+     )
+    )
+    (i32.load16_u align=1
+     (i32.add
+      (i32.shl
+       (local.get $3)
+       (i32.const 1)
+      )
+      (i32.const 1048706)
+     )
+    )
+   )
+  )
+  (local.set $7
+   (i32.sub
+    (i32.const 39)
+    (local.get $2)
+   )
+  )
+  (local.set $3
+   (i32.const 1)
+  )
+  (local.set $8
+   (select
+    (i32.const 43)
+    (i32.const 1114112)
+    (local.tee $11
+     (i32.and
+      (local.tee $4
+       (i32.load
+        (local.get $1)
+       )
+      )
+      (i32.const 1)
+     )
+    )
+   )
+  )
+  (local.set $9
+   (i32.and
+    (i32.shr_s
+     (i32.shl
+      (local.get $4)
+      (i32.const 29)
+     )
+     (i32.const 31)
+    )
+    (i32.const 1048656)
+   )
+  )
+  (local.set $10
+   (i32.add
+    (i32.add
+     (local.get $6)
+     (i32.const 9)
+    )
+    (local.get $2)
+   )
+  )
+  (block $label$7
+   (block $label$8
+    (block $label$9
+     (block $label$10
+      (block $label$11
+       (block $label$12
+        (block $label$13
+         (block $label$14
+          (local.set $3
+           (block $label$15 (result i32)
+            (block $label$16
+             (block $label$17
+              (block $label$18
+               (block $label$19
+                (if
+                 (i32.eq
+                  (i32.load offset=8
+                   (local.get $1)
+                  )
+                  (i32.const 1)
+                 )
+                 (block
+                  (br_if $label$19
+                   (i32.le_u
+                    (local.tee $5
+                     (i32.load
+                      (i32.add
+                       (local.get $1)
+                       (i32.const 12)
+                      )
+                     )
+                    )
+                    (local.tee $2
+                     (i32.add
+                      (local.get $7)
+                      (local.get $11)
+                     )
+                    )
+                   )
+                  )
+                  (br_if $label$18
+                   (i32.and
+                    (local.get $4)
+                    (i32.const 8)
+                   )
+                  )
+                  (local.set $4
+                   (i32.sub
+                    (local.get $5)
+                    (local.get $2)
+                   )
+                  )
+                  (br_if $label$17
+                   (i32.eqz
+                    (i32.and
+                     (local.tee $3
+                      (select
+                       (i32.const 1)
+                       (local.tee $3
+                        (i32.load8_u offset=48
+                         (local.get $1)
+                        )
+                       )
+                       (i32.eq
+                        (local.get $3)
+                        (i32.const 3)
+                       )
+                      )
+                     )
+                     (i32.const 3)
+                    )
+                   )
+                  )
+                  (br_if $label$16
+                   (i32.eq
+                    (local.get $3)
+                    (i32.const 2)
+                   )
+                  )
+                  (local.set $5
+                   (i32.const 0)
+                  )
+                  (br $label$15
+                   (local.get $4)
+                  )
+                 )
+                )
+                (br_if $label$9
+                 (call $9
+                  (local.get $1)
+                  (local.get $8)
+                  (local.get $9)
+                 )
+                )
+                (br $label$8)
+               )
+               (br_if $label$9
+                (call $9
+                 (local.get $1)
+                 (local.get $8)
+                 (local.get $9)
+                )
+               )
+               (br $label$8)
+              )
+              (i32.store8 offset=48
+               (local.get $1)
+               (i32.const 1)
+              )
+              (i32.store offset=4
+               (local.get $1)
+               (i32.const 48)
+              )
+              (br_if $label$9
+               (call $9
+                (local.get $1)
+                (local.get $8)
+                (local.get $9)
+               )
+              )
+              (local.set $3
+               (i32.sub
+                (local.get $5)
+                (local.get $2)
+               )
+              )
+              (br_if $label$14
+               (i32.eqz
+                (i32.and
+                 (local.tee $4
+                  (select
+                   (i32.const 1)
+                   (local.tee $4
+                    (i32.load8_u
+                     (i32.add
+                      (local.get $1)
+                      (i32.const 48)
+                     )
+                    )
+                   )
+                   (i32.eq
+                    (local.get $4)
+                    (i32.const 3)
+                   )
+                  )
+                 )
+                 (i32.const 3)
+                )
+               )
+              )
+              (br_if $label$13
+               (i32.eq
+                (local.get $4)
+                (i32.const 2)
+               )
+              )
+              (local.set $4
+               (i32.const 0)
+              )
+              (br $label$12)
+             )
+             (local.set $5
+              (local.get $4)
+             )
+             (br $label$15
+              (i32.const 0)
+             )
+            )
+            (local.set $5
+             (i32.shr_u
+              (i32.add
+               (local.get $4)
+               (i32.const 1)
+              )
+              (i32.const 1)
+             )
+            )
+            (i32.shr_u
+             (local.get $4)
+             (i32.const 1)
+            )
+           )
+          )
+          (local.set $2
+           (i32.const -1)
+          )
+          (local.set $4
+           (i32.add
+            (local.get $1)
+            (i32.const 4)
+           )
+          )
+          (local.set $11
+           (i32.add
+            (local.get $1)
+            (i32.const 24)
+           )
+          )
+          (local.set $12
+           (i32.add
+            (local.get $1)
+            (i32.const 28)
+           )
+          )
+          (block $label$21
+           (loop $label$22
+            (br_if $label$21
+             (i32.ge_u
+              (local.tee $2
+               (i32.add
+                (local.get $2)
+                (i32.const 1)
+               )
+              )
+              (local.get $3)
+             )
+            )
+            (br_if $label$22
+             (i32.eqz
+              (call_indirect (type $1)
+               (i32.load
+                (local.get $11)
+               )
+               (i32.load
+                (local.get $4)
+               )
+               (i32.load offset=16
+                (i32.load
+                 (local.get $12)
+                )
+               )
+              )
+             )
+            )
+           )
+           (br $label$7)
+          )
+          (local.set $4
+           (i32.load
+            (i32.add
+             (local.get $1)
+             (i32.const 4)
+            )
+           )
+          )
+          (local.set $3
+           (i32.const 1)
+          )
+          (br_if $label$9
+           (call $9
+            (local.get $1)
+            (local.get $8)
+            (local.get $9)
+           )
+          )
+          (br_if $label$9
+           (call_indirect (type $0)
+            (i32.load
+             (local.tee $2
+              (i32.add
+               (local.get $1)
+               (i32.const 24)
+              )
+             )
+            )
+            (local.get $10)
+            (local.get $7)
+            (i32.load offset=12
+             (i32.load
+              (local.tee $1
+               (i32.add
+                (local.get $1)
+                (i32.const 28)
+               )
+              )
+             )
+            )
+           )
+          )
+          (local.set $7
+           (i32.load
+            (local.get $2)
+           )
+          )
+          (local.set $2
+           (i32.const -1)
+          )
+          (local.set $1
+           (i32.add
+            (i32.load
+             (local.get $1)
+            )
+            (i32.const 16)
+           )
+          )
+          (loop $label$23
+           (br_if $label$11
+            (i32.ge_u
+             (local.tee $2
+              (i32.add
+               (local.get $2)
+               (i32.const 1)
+              )
+             )
+             (local.get $5)
+            )
+           )
+           (br_if $label$23
+            (i32.eqz
+             (call_indirect (type $1)
+              (local.get $7)
+              (local.get $4)
+              (i32.load
+               (local.get $1)
+              )
+             )
+            )
+           )
+          )
+          (br $label$9)
+         )
+         (local.set $4
+          (local.get $3)
+         )
+         (local.set $3
+          (i32.const 0)
+         )
+         (br $label$12)
+        )
+        (local.set $4
+         (i32.shr_u
+          (i32.add
+           (local.get $3)
+           (i32.const 1)
+          )
+          (i32.const 1)
+         )
+        )
+        (local.set $3
+         (i32.shr_u
+          (local.get $3)
+          (i32.const 1)
+         )
+        )
+       )
+       (local.set $2
+        (i32.const -1)
+       )
+       (local.set $5
+        (i32.add
+         (local.get $1)
+         (i32.const 4)
+        )
+       )
+       (local.set $8
+        (i32.add
+         (local.get $1)
+         (i32.const 24)
+        )
+       )
+       (local.set $9
+        (i32.add
+         (local.get $1)
+         (i32.const 28)
+        )
+       )
+       (block $label$24
+        (loop $label$25
+         (br_if $label$24
+          (i32.ge_u
+           (local.tee $2
+            (i32.add
+             (local.get $2)
+             (i32.const 1)
+            )
+           )
+           (local.get $3)
+          )
+         )
+         (br_if $label$25
+          (i32.eqz
+           (call_indirect (type $1)
+            (i32.load
+             (local.get $8)
+            )
+            (i32.load
+             (local.get $5)
+            )
+            (i32.load offset=16
+             (i32.load
+              (local.get $9)
+             )
+            )
+           )
+          )
+         )
+        )
+        (br $label$7)
+       )
+       (local.set $5
+        (i32.load
+         (i32.add
+          (local.get $1)
+          (i32.const 4)
+         )
+        )
+       )
+       (local.set $3
+        (i32.const 1)
+       )
+       (br_if $label$9
+        (call_indirect (type $0)
+         (i32.load
+          (local.tee $2
+           (i32.add
+            (local.get $1)
+            (i32.const 24)
+           )
+          )
+         )
+         (local.get $10)
+         (local.get $7)
+         (i32.load offset=12
+          (i32.load
+           (local.tee $1
+            (i32.add
+             (local.get $1)
+             (i32.const 28)
+            )
+           )
+          )
+         )
+        )
+       )
+       (local.set $7
+        (i32.load
+         (local.get $2)
+        )
+       )
+       (local.set $2
+        (i32.const -1)
+       )
+       (local.set $1
+        (i32.add
+         (i32.load
+          (local.get $1)
+         )
+         (i32.const 16)
+        )
+       )
+       (loop $label$26
+        (br_if $label$10
+         (i32.ge_u
+          (local.tee $2
+           (i32.add
+            (local.get $2)
+            (i32.const 1)
+           )
+          )
+          (local.get $4)
+         )
+        )
+        (br_if $label$26
+         (i32.eqz
+          (call_indirect (type $1)
+           (local.get $7)
+           (local.get $5)
+           (i32.load
+            (local.get $1)
+           )
+          )
+         )
+        )
+       )
+       (br $label$9)
+      )
+      (global.set $global$0
+       (i32.add
+        (local.get $6)
+        (i32.const 48)
+       )
+      )
+      (return
+       (i32.const 0)
+      )
+     )
+     (local.set $3
+      (i32.const 0)
+     )
+    )
+    (global.set $global$0
+     (i32.add
+      (local.get $6)
+      (i32.const 48)
+     )
+    )
+    (return
+     (local.get $3)
+    )
+   )
+   (return
+    (block (result i32)
+     (local.set $15
+      (call_indirect (type $0)
+       (i32.load offset=24
+        (local.get $1)
+       )
+       (local.get $10)
+       (local.get $7)
+       (i32.load offset=12
+        (i32.load
+         (i32.add
+          (local.get $1)
+          (i32.const 28)
+         )
+        )
+       )
+      )
+     )
+     (global.set $global$0
+      (i32.add
+       (local.get $6)
+       (i32.const 48)
+      )
+     )
+     (local.get $15)
+    )
+   )
+  )
+  (global.set $global$0
+   (i32.add
+    (local.get $6)
+    (i32.const 48)
+   )
+  )
+  (i32.const 1)
+ )
+ (func $7 (; 7 ;) (type $2) (param $0 i32)
+  (nop)
+ )
+ (func $8 (; 8 ;) (type $6) (param $0 i32) (result i64)
+  (i64.const -2357177763932378009)
+ )
+ (func $9 (; 9 ;) (type $9) (param $0 i32) (param $1 i32) (param $2 i32) (result i32)
+  (block $label$1
+   (return
+    (block $label$2 (result i32)
+     (if
+      (i32.ne
+       (local.get $1)
+       (i32.const 1114112)
+      )
+      (drop
+       (br_if $label$2
+        (i32.const 1)
+        (call_indirect (type $1)
+         (i32.load offset=24
+          (local.get $0)
+         )
+         (local.get $1)
+         (i32.load offset=16
+          (i32.load
+           (i32.add
+            (local.get $0)
+            (i32.const 28)
+           )
+          )
+         )
+        )
+       )
+      )
+     )
+     (br_if $label$1
+      (i32.eqz
+       (local.get $2)
+      )
+     )
+     (call_indirect (type $0)
+      (i32.load offset=24
+       (local.get $0)
+      )
+      (local.get $2)
+      (i32.const 0)
+      (i32.load offset=12
+       (i32.load
+        (i32.add
+         (local.get $0)
+         (i32.const 28)
+        )
+       )
+      )
+     )
+    )
+   )
+  )
+  (i32.const 0)
+ )
+ (func $10 (; 10 ;) (type $3) (param $0 i32) (result i32)
+  (local $1 i32)
+  (local $2 i32)
+  (local $3 i32)
+  (local $4 i32)
+  (local $5 i32)
+  (local $6 i32)
+  (local $7 i32)
+  (local $8 i32)
+  (local $9 i32)
+  (local $10 i32)
+  (local $11 i32)
+  (local $12 i32)
+  (local $13 i32)
+  (local $14 i32)
+  (local $15 i32)
+  (local $16 i32)
+  (local $17 i32)
+  (local $18 i32)
+  (local $19 i32)
+  (local $20 i32)
+  (local $21 i32)
+  (local $22 i32)
+  (local $23 i32)
+  (local $24 i32)
+  (local $25 i32)
+  (local $26 i32)
+  (local $27 i32)
+  (local $28 i32)
+  (local $29 i32)
+  (local $30 i32)
+  (local $31 i32)
+  (local $32 i32)
+  (local $33 i32)
+  (local $34 i32)
+  (local $35 i32)
+  (local $36 i32)
+  (local $37 i32)
+  (local $38 i32)
+  (local $39 i32)
+  (local $40 i32)
+  (local $41 i32)
+  (local $42 i32)
+  (local $43 i32)
+  (local $44 i32)
+  (local $45 i32)
+  (local $46 i32)
+  (global.set $global$0
+   (local.tee $1
+    (i32.sub
+     (global.get $global$0)
+     (i32.const 256)
+    )
+   )
+  )
+  (i64.store offset=56 align=4
+   (local.get $1)
+   (i64.const 4294967297)
+  )
+  (i64.store offset=48 align=4
+   (local.get $1)
+   (i64.const 4294967297)
+  )
+  (i64.store offset=40 align=4
+   (local.get $1)
+   (i64.const 4294967297)
+  )
+  (i64.store offset=32 align=4
+   (local.get $1)
+   (i64.const 4294967297)
+  )
+  (i64.store offset=24 align=4
+   (local.get $1)
+   (i64.const 4294967297)
+  )
+  (i64.store offset=16 align=4
+   (local.get $1)
+   (i64.const 4294967297)
+  )
+  (i64.store offset=8 align=4
+   (local.get $1)
+   (i64.const 4294967297)
+  )
+  (i64.store align=4
+   (local.get $1)
+   (i64.const 4294967297)
+  )
+  (block $label$1
+   (if
+    (i32.ge_u
+     (local.tee $11
+      (i32.add
+       (local.get $0)
+       (i32.const 1)
+      )
+     )
+     (i32.const 2)
+    )
+    (block
+     (local.set $3
+      (local.get $1)
+     )
+     (local.set $2
+      (i32.const 1)
+     )
+     (loop $label$3
+      (br_if $label$1
+       (i32.ge_u
+        (local.get $2)
+        (i32.const 16)
+       )
+      )
+      (i32.store
+       (local.tee $4
+        (i32.add
+         (local.get $3)
+         (i32.const 4)
+        )
+       )
+       (i32.mul
+        (i32.load
+         (local.get $3)
+        )
+        (local.get $2)
+       )
+      )
+      (local.set $3
+       (local.get $4)
+      )
+      (local.set $2
+       (local.tee $4
+        (i32.add
+         (local.get $2)
+         (i32.const 1)
+        )
+       )
+      )
+      (br_if $label$3
+       (i32.lt_u
+        (local.get $4)
+        (local.get $11)
+       )
+      )
+     )
+    )
+   )
+   (if
+    (i32.lt_u
+     (local.get $0)
+     (i32.const 16)
+    )
+    (block
+     (local.set $20
+      (i32.const 1)
+     )
+     (local.set $21
+      (local.tee $9
+       (i32.load
+        (i32.add
+         (local.get $1)
+         (i32.shl
+          (local.get $0)
+          (i32.const 2)
+         )
+        )
+       )
+      )
+     )
+     (if
+      (i32.ge_u
+       (local.get $9)
+       (i32.const 24)
+      )
+      (local.set $20
+       (select
+        (i32.const 24)
+        (i32.const 25)
+        (i32.eq
+         (local.get $9)
+         (i32.mul
+          (local.tee $21
+           (i32.div_u
+            (local.get $9)
+            (i32.const 24)
+           )
+          )
+          (i32.const 24)
+         )
+        )
+       )
+      )
+     )
+     (local.set $40
+      (i32.sub
+       (i32.const 0)
+       (local.get $0)
+      )
+     )
+     (local.set $12
+      (i32.add
+       (local.get $1)
+       (i32.const 196)
+      )
+     )
+     (local.set $41
+      (i32.add
+       (local.get $1)
+       (i32.const 132)
+      )
+     )
+     (local.set $42
+      (i32.add
+       (local.get $1)
+       (i32.const 124)
+      )
+     )
+     (local.set $11
+      (i32.add
+       (local.get $1)
+       (i32.const 68)
+      )
+     )
+     (local.set $43
+      (i32.lt_u
+       (local.get $0)
+       (i32.const 2)
+      )
+     )
+     (loop $label$6
+      (i64.store
+       (i32.add
+        (local.get $1)
+        (i32.const 120)
+       )
+       (i64.const 0)
+      )
+      (i64.store
+       (i32.add
+        (local.get $1)
+        (i32.const 112)
+       )
+       (i64.const 0)
+      )
+      (i64.store
+       (i32.add
+        (local.get $1)
+        (i32.const 104)
+       )
+       (i64.const 0)
+      )
+      (i64.store
+       (i32.add
+        (local.get $1)
+        (i32.const 96)
+       )
+       (i64.const 0)
+      )
+      (i64.store
+       (i32.add
+        (local.get $1)
+        (i32.const 88)
+       )
+       (i64.const 0)
+      )
+      (i64.store
+       (i32.add
+        (local.get $1)
+        (i32.const 80)
+       )
+       (i64.const 0)
+      )
+      (i64.store
+       (i32.add
+        (local.get $1)
+        (i32.const 72)
+       )
+       (i64.const 0)
+      )
+      (i64.store offset=64
+       (local.get $1)
+       (i64.const 0)
+      )
+      (i64.store
+       (local.tee $26
+        (i32.add
+         (local.get $1)
+         (i32.const 184)
+        )
+       )
+       (i64.const 0)
+      )
+      (i64.store
+       (local.tee $27
+        (i32.add
+         (local.get $1)
+         (i32.const 176)
+        )
+       )
+       (i64.const 0)
+      )
+      (i64.store
+       (local.tee $28
+        (i32.add
+         (local.get $1)
+         (i32.const 168)
+        )
+       )
+       (i64.const 0)
+      )
+      (i64.store
+       (local.tee $29
+        (i32.add
+         (local.get $1)
+         (i32.const 160)
+        )
+       )
+       (i64.const 0)
+      )
+      (i64.store
+       (local.tee $30
+        (i32.add
+         (local.get $1)
+         (i32.const 152)
+        )
+       )
+       (i64.const 0)
+      )
+      (i64.store
+       (local.tee $31
+        (i32.add
+         (local.get $1)
+         (i32.const 144)
+        )
+       )
+       (i64.const 0)
+      )
+      (i64.store
+       (local.tee $32
+        (i32.add
+         (local.get $1)
+         (i32.const 136)
+        )
+       )
+       (i64.const 0)
+      )
+      (i64.store offset=128
+       (local.get $1)
+       (i64.const 0)
+      )
+      (i64.store align=4
+       (local.tee $33
+        (i32.add
+         (local.get $1)
+         (i32.const 248)
+        )
+       )
+       (i64.const 64424509454)
+      )
+      (i64.store align=4
+       (local.tee $34
+        (i32.add
+         (local.get $1)
+         (i32.const 240)
+        )
+       )
+       (i64.const 55834574860)
+      )
+      (i64.store align=4
+       (local.tee $35
+        (i32.add
+         (local.get $1)
+         (i32.const 232)
+        )
+       )
+       (i64.const 47244640266)
+      )
+      (i64.store align=4
+       (local.tee $36
+        (i32.add
+         (local.get $1)
+         (i32.const 224)
+        )
+       )
+       (i64.const 38654705672)
+      )
+      (i64.store align=4
+       (local.tee $37
+        (i32.add
+         (local.get $1)
+         (i32.const 216)
+        )
+       )
+       (i64.const 30064771078)
+      )
+      (i64.store align=4
+       (local.tee $38
+        (i32.add
+         (local.get $1)
+         (i32.const 208)
+        )
+       )
+       (i64.const 21474836484)
+      )
+      (i64.store align=4
+       (local.tee $39
+        (i32.add
+         (local.get $1)
+         (i32.const 200)
+        )
+       )
+       (i64.const 12884901890)
+      )
+      (i64.store offset=192 align=4
+       (local.get $1)
+       (i64.const 4294967296)
+      )
+      (local.set $7
+       (i32.mul
+        (local.get $13)
+        (local.get $21)
+       )
+      )
+      (local.set $2
+       (block $label$7 (result i32)
+        (block $label$8
+         (if
+          (i32.eqz
+           (local.get $43)
+          )
+          (block
+           (local.set $23
+            (local.get $40)
+           )
+           (local.set $14
+            (local.get $7)
+           )
+           (local.set $15
+            (local.get $0)
+           )
+           (local.set $5
+            (i32.const 0)
+           )
+           (br $label$8)
+          )
+         )
+         (br $label$7
+          (i32.const 0)
+         )
+        )
+        (i32.const 1)
+       )
+      )
+      (loop $label$10
+       (block $label$11
+        (block $label$12
+         (local.set $2
+          (block $label$13 (result i32)
+           (block $label$14
+            (block $label$15
+             (block $label$16
+              (block $label$17
+               (block $label$18
+                (block $label$19
+                 (if
+                  (i32.eqz
+                   (local.get $2)
+                  )
+                  (block
+                   (local.set $13
+                    (i32.add
+                     (local.get $13)
+                     (i32.const 1)
+                    )
+                   )
+                   (local.set $44
+                    (i32.add
+                     (select
+                      (local.get $9)
+                      (local.tee $3
+                       (i32.add
+                        (local.get $7)
+                        (local.get $21)
+                       )
+                      )
+                      (i32.gt_u
+                       (local.get $3)
+                       (local.get $9)
+                      )
+                     )
+                     (i32.const -1)
+                    )
+                   )
+                   (local.set $24
+                    (i32.const 0)
+                   )
+                   (br_if $label$19
+                    (i32.ge_s
+                     (local.tee $6
+                      (i32.load offset=192
+                       (local.get $1)
+                      )
+                     )
+                     (i32.const 1)
+                    )
+                   )
+                   (br $label$18)
+                  )
+                 )
+                 (block $label$21
+                  (block $label$22
+                   (block $label$23
+                    (block $label$24
+                     (block $label$25
+                      (block $label$26
+                       (block $label$27
+                        (block $label$28
+                         (block $label$29
+                          (block $label$30
+                           (block $label$31
+                            (br_table $label$31 $label$30 $label$29
+                             (local.get $5)
+                            )
+                           )
+                           (br_if $label$24
+                            (i32.ge_u
+                             (local.tee $4
+                              (i32.add
+                               (local.get $15)
+                               (i32.const -1)
+                              )
+                             )
+                             (i32.const 16)
+                            )
+                           )
+                           (br_if $label$23
+                            (i32.eqz
+                             (local.tee $3
+                              (i32.load
+                               (i32.add
+                                (local.get $1)
+                                (local.tee $2
+                                 (i32.shl
+                                  (local.get $4)
+                                  (i32.const 2)
+                                 )
+                                )
+                               )
+                              )
+                             )
+                            )
+                           )
+                           (if
+                            (i32.eq
+                             (local.get $14)
+                             (i32.const -2147483648)
+                            )
+                            (br_if $label$22
+                             (i32.eq
+                              (local.get $3)
+                              (i32.const -1)
+                             )
+                            )
+                           )
+                           (i32.store
+                            (i32.add
+                             (i32.sub
+                              (local.get $1)
+                              (i32.const -64)
+                             )
+                             (local.get $2)
+                            )
+                            (local.tee $16
+                             (i32.div_s
+                              (local.get $14)
+                              (local.get $3)
+                             )
+                            )
+                           )
+                           (i64.store
+                            (local.get $32)
+                            (i64.load align=4
+                             (local.get $39)
+                            )
+                           )
+                           (i64.store
+                            (local.get $31)
+                            (i64.load align=4
+                             (local.get $38)
+                            )
+                           )
+                           (i64.store
+                            (local.get $30)
+                            (i64.load align=4
+                             (local.get $37)
+                            )
+                           )
+                           (i64.store
+                            (local.get $29)
+                            (i64.load align=4
+                             (local.get $36)
+                            )
+                           )
+                           (i64.store
+                            (local.get $28)
+                            (i64.load align=4
+                             (local.get $35)
+                            )
+                           )
+                           (i64.store
+                            (local.get $27)
+                            (i64.load align=4
+                             (local.get $34)
+                            )
+                           )
+                           (i64.store
+                            (local.get $26)
+                            (i64.load align=4
+                             (local.get $33)
+                            )
+                           )
+                           (i64.store offset=128
+                            (local.get $1)
+                            (i64.load offset=192 align=4
+                             (local.get $1)
+                            )
+                           )
+                           (local.set $45
+                            (i32.add
+                             (local.get $16)
+                             (local.get $23)
+                            )
+                           )
+                           (local.set $14
+                            (i32.sub
+                             (local.get $14)
+                             (i32.mul
+                              (local.get $3)
+                              (local.get $16)
+                             )
+                            )
+                           )
+                           (local.set $2
+                            (i32.const 0)
+                           )
+                           (local.set $8
+                            (i32.add
+                             (local.get $1)
+                             (i32.const 192)
+                            )
+                           )
+                           (loop $label$33
+                            (block $label$34
+                             (if
+                              (i32.gt_u
+                               (local.tee $3
+                                (i32.add
+                                 (local.get $2)
+                                 (local.get $16)
+                                )
+                               )
+                               (local.get $4)
+                              )
+                              (block
+                               (br_if $label$27
+                                (i32.gt_u
+                                 (local.tee $46
+                                  (i32.add
+                                   (local.get $2)
+                                   (local.get $45)
+                                  )
+                                 )
+                                 (i32.const 15)
+                                )
+                               )
+                               (local.set $3
+                                (i32.sub
+                                 (local.get $3)
+                                 (local.get $15)
+                                )
+                               )
+                               (br_if $label$34
+                                (i32.le_u
+                                 (local.get $2)
+                                 (i32.const 15)
+                                )
+                               )
+                               (br $label$28)
+                              )
+                             )
+                             (br_if $label$26
+                              (i32.ge_u
+                               (local.get $3)
+                               (i32.const 16)
+                              )
+                             )
+                             (br_if $label$28
+                              (i32.gt_u
+                               (local.get $2)
+                               (i32.const 15)
+                              )
+                             )
+                            )
+                            (i32.store
+                             (local.get $8)
+                             (i32.load
+                              (i32.add
+                               (i32.add
+                                (local.get $1)
+                                (i32.const 128)
+                               )
+                               (i32.shl
+                                (local.get $3)
+                                (i32.const 2)
+                               )
+                              )
+                             )
+                            )
+                            (local.set $8
+                             (i32.add
+                              (local.get $8)
+                              (i32.const 4)
+                             )
+                            )
+                            (br_if $label$33
+                             (i32.lt_u
+                              (local.tee $2
+                               (i32.add
+                                (local.get $2)
+                                (i32.const 1)
+                               )
+                              )
+                              (local.get $15)
+                             )
+                            )
+                           )
+                           (local.set $23
+                            (i32.add
+                             (local.get $23)
+                             (i32.const 1)
+                            )
+                           )
+                           (br_if $label$21
+                            (i32.gt_u
+                             (local.tee $15
+                              (local.get $4)
+                             )
+                             (i32.const 1)
+                            )
+                           )
+                           (local.set $2
+                            (i32.const 0)
+                           )
+                           (br $label$10)
+                          )
+                          (i64.store
+                           (local.get $26)
+                           (i64.load align=4
+                            (local.get $33)
+                           )
+                          )
+                          (i64.store
+                           (local.get $27)
+                           (i64.load align=4
+                            (local.get $34)
+                           )
+                          )
+                          (i64.store
+                           (local.get $28)
+                           (i64.load align=4
+                            (local.get $35)
+                           )
+                          )
+                          (i64.store
+                           (local.get $29)
+                           (i64.load align=4
+                            (local.get $36)
+                           )
+                          )
+                          (i64.store
+                           (local.get $30)
+                           (i64.load align=4
+                            (local.get $37)
+                           )
+                          )
+                          (i64.store
+                           (local.get $31)
+                           (i64.load align=4
+                            (local.get $38)
+                           )
+                          )
+                          (i64.store
+                           (local.get $32)
+                           (i64.load align=4
+                            (local.get $39)
+                           )
+                          )
+                          (i64.store offset=128
+                           (local.get $1)
+                           (i64.load offset=192 align=4
+                            (local.get $1)
+                           )
+                          )
+                          (br_if $label$25
+                           (i32.gt_u
+                            (local.get $6)
+                            (i32.const 15)
+                           )
+                          )
+                          (local.set $17
+                           (i32.const 1)
+                          )
+                          (local.set $10
+                           (local.get $6)
+                          )
+                          (br $label$13
+                           (i32.const 0)
+                          )
+                         )
+                         (if
+                          (i32.lt_u
+                           (local.get $7)
+                           (local.get $44)
+                          )
+                          (block
+                           (local.set $25
+                            (i32.load
+                             (local.get $12)
+                            )
+                           )
+                           (i32.store
+                            (local.get $12)
+                            (local.get $6)
+                           )
+                           (i32.store offset=192
+                            (local.get $1)
+                            (local.get $25)
+                           )
+                           (local.set $18
+                            (local.get $11)
+                           )
+                           (br_if $label$11
+                            (i32.lt_s
+                             (local.tee $2
+                              (i32.load offset=68
+                               (local.get $1)
+                              )
+                             )
+                             (i32.const 1)
+                            )
+                           )
+                           (local.set $19
+                            (i32.const 1)
+                           )
+                           (br $label$14)
+                          )
+                         )
+                         (local.set $22
+                          (i32.add
+                           (local.get $22)
+                           (local.get $24)
+                          )
+                         )
+                         (br_if $label$6
+                          (i32.lt_u
+                           (local.get $13)
+                           (local.get $20)
+                          )
+                         )
+                         (global.set $global$0
+                          (i32.add
+                           (local.get $1)
+                           (i32.const 256)
+                          )
+                         )
+                         (return
+                          (local.get $22)
+                         )
+                        )
+                        (call $3
+                         (i32.const 1049076)
+                         (local.get $2)
+                        )
+                        (unreachable)
+                       )
+                       (call $3
+                        (i32.const 1049060)
+                        (local.get $46)
+                       )
+                       (unreachable)
+                      )
+                      (call $3
+                       (i32.const 1049044)
+                       (i32.add
+                        (local.get $2)
+                        (local.get $16)
+                       )
+                      )
+                      (unreachable)
+                     )
+                     (local.set $10
+                      (local.get $6)
+                     )
+                     (br $label$12)
+                    )
+                    (call $3
+                     (i32.const 1048980)
+                     (local.get $4)
+                    )
+                    (unreachable)
+                   )
+                   (call $2
+                    (i32.const 1048996)
+                   )
+                   (unreachable)
+                  )
+                  (call $2
+                   (i32.const 1049020)
+                  )
+                  (unreachable)
+                 )
+                 (local.set $5
+                  (i32.const 0)
+                 )
+                 (br $label$17)
+                )
+                (local.set $5
+                 (i32.const 1)
+                )
+                (br $label$16)
+               )
+               (local.set $5
+                (i32.const 2)
+               )
+               (br $label$15)
+              )
+              (local.set $2
+               (i32.const 1)
+              )
+              (br $label$10)
+             )
+             (local.set $2
+              (i32.const 1)
+             )
+             (br $label$10)
+            )
+            (local.set $2
+             (i32.const 1)
+            )
+            (br $label$10)
+           )
+           (i32.const 1)
+          )
+         )
+         (loop $label$37
+          (block $label$38
+           (block $label$39
+            (if
+             (i32.eqz
+              (local.get $2)
+             )
+             (block
+              (if
+               (local.tee $10
+                (i32.load
+                 (local.tee $5
+                  (i32.add
+                   (local.tee $4
+                    (i32.shl
+                     (local.tee $3
+                      (local.get $10)
+                     )
+                     (i32.const 2)
+                    )
+                   )
+                   (i32.add
+                    (local.get $1)
+                    (i32.const 128)
+                   )
+                  )
+                 )
+                )
+               )
+               (block
+                (i32.store
+                 (local.get $5)
+                 (local.get $3)
+                )
+                (block $label$42
+                 (br_if $label$42
+                  (i32.lt_u
+                   (local.get $3)
+                   (i32.const 3)
+                  )
+                 )
+                 (br_if $label$42
+                  (i32.eqz
+                   (local.tee $8
+                    (i32.shr_u
+                     (i32.add
+                      (local.get $3)
+                      (i32.const -1)
+                     )
+                     (i32.const 1)
+                    )
+                   )
+                  )
+                 )
+                 (local.set $2
+                  (i32.add
+                   (local.get $4)
+                   (local.get $42)
+                  )
+                 )
+                 (local.set $3
+                  (local.get $41)
+                 )
+                 (loop $label$43
+                  (local.set $4
+                   (i32.load
+                    (local.get $3)
+                   )
+                  )
+                  (i32.store
+                   (local.get $3)
+                   (i32.load
+                    (local.get $2)
+                   )
+                  )
+                  (i32.store
+                   (local.get $2)
+                   (local.get $4)
+                  )
+                  (local.set $3
+                   (i32.add
+                    (local.get $3)
+                    (i32.const 4)
+                   )
+                  )
+                  (local.set $2
+                   (i32.add
+                    (local.get $2)
+                    (i32.const -4)
+                   )
+                  )
+                  (br_if $label$43
+                   (local.tee $8
+                    (i32.add
+                     (local.get $8)
+                     (i32.const -1)
+                    )
+                   )
+                  )
+                 )
+                )
+                (local.set $17
+                 (i32.add
+                  (local.get $17)
+                  (i32.const 1)
+                 )
+                )
+                (br_if $label$38
+                 (i32.lt_u
+                  (local.get $10)
+                  (i32.const 16)
+                 )
+                )
+                (br $label$12)
+               )
+              )
+              (local.set $24
+               (i32.add
+                (select
+                 (i32.sub
+                  (i32.const 0)
+                  (local.get $17)
+                 )
+                 (local.get $17)
+                 (i32.and
+                  (local.get $7)
+                  (i32.const 1)
+                 )
+                )
+                (local.get $24)
+               )
+              )
+              (local.set $5
+               (i32.const 2)
+              )
+              (br $label$39)
+             )
+            )
+            (local.set $2
+             (i32.const 0)
+            )
+            (i32.store
+             (local.get $18)
+             (i32.const 0)
+            )
+            (i32.store offset=192
+             (local.get $1)
+             (local.tee $4
+              (local.get $6)
+             )
+            )
+            (local.set $5
+             (i32.add
+              (local.get $19)
+              (i32.const 1)
+             )
+            )
+            (local.set $3
+             (local.get $12)
+            )
+            (block $label$44
+             (block $label$45
+              (loop $label$46
+               (br_if $label$45
+                (i32.ge_u
+                 (i32.add
+                  (local.get $2)
+                  (i32.const 2)
+                 )
+                 (i32.const 16)
+                )
+               )
+               (i32.store
+                (local.get $3)
+                (i32.load
+                 (local.tee $3
+                  (i32.add
+                   (local.get $3)
+                   (i32.const 4)
+                  )
+                 )
+                )
+               )
+               (br_if $label$46
+                (i32.lt_u
+                 (local.tee $2
+                  (i32.add
+                   (local.get $2)
+                   (i32.const 1)
+                  )
+                 )
+                 (local.get $19)
+                )
+               )
+              )
+              (br_if $label$44
+               (i32.ge_u
+                (local.get $5)
+                (i32.const 16)
+               )
+              )
+              (i32.store
+               (i32.add
+                (local.tee $3
+                 (i32.shl
+                  (local.get $5)
+                  (i32.const 2)
+                 )
+                )
+                (i32.add
+                 (local.get $1)
+                 (i32.const 192)
+                )
+               )
+               (local.get $25)
+              )
+              (br_if $label$11
+               (i32.le_s
+                (local.tee $2
+                 (i32.load
+                  (local.tee $18
+                   (i32.add
+                    (i32.sub
+                     (local.get $1)
+                     (i32.const -64)
+                    )
+                    (local.get $3)
+                   )
+                  )
+                 )
+                )
+                (local.get $19)
+               )
+              )
+              (local.set $6
+               (i32.load
+                (local.get $12)
+               )
+              )
+              (local.set $19
+               (local.get $5)
+              )
+              (local.set $25
+               (local.get $4)
+              )
+              (local.set $2
+               (i32.const 1)
+              )
+              (br $label$37)
+             )
+             (call $3
+              (i32.const 1049108)
+              (i32.add
+               (local.get $2)
+               (i32.const 2)
+              )
+             )
+             (unreachable)
+            )
+            (call $3
+             (i32.const 1049124)
+             (local.get $5)
+            )
+            (unreachable)
+           )
+           (local.set $2
+            (i32.const 1)
+           )
+           (br $label$10)
+          )
+          (local.set $2
+           (i32.const 0)
+          )
+          (br $label$37)
+         )
+        )
+        (call $3
+         (i32.const 1049092)
+         (local.get $10)
+        )
+        (unreachable)
+       )
+       (local.set $7
+        (i32.add
+         (local.get $7)
+         (i32.const 1)
+        )
+       )
+       (i32.store
+        (local.get $18)
+        (i32.add
+         (local.get $2)
+         (i32.const 1)
+        )
+       )
+       (block $label$47
+        (block $label$48
+         (if
+          (i32.ge_s
+           (local.tee $6
+            (i32.load offset=192
+             (local.get $1)
+            )
+           )
+           (i32.const 1)
+          )
+          (block
+           (local.set $5
+            (i32.const 1)
+           )
+           (br $label$48)
+          )
+         )
+         (local.set $5
+          (i32.const 2)
+         )
+         (br $label$47)
+        )
+        (local.set $2
+         (i32.const 1)
+        )
+        (br $label$10)
+       )
+       (local.set $2
+        (i32.const 1)
+       )
+       (br $label$10)
+      )
+     )
+    )
+   )
+   (call $3
+    (i32.const 1049212)
+    (local.get $0)
+   )
+   (unreachable)
+  )
+  (call $3
+   (i32.const 1049196)
+   (local.get $2)
+  )
+  (unreachable)
+ )
+)
+

From 1176e4f17820736eee180639f53eedf6d4063672 Mon Sep 17 00:00:00 2001
From: yjh 
Date: Thu, 24 Oct 2019 23:54:31 +0800
Subject: [PATCH 2876/3084] Fix clippy warnings (#1168)

---
 cranelift/codegen/meta/src/cdsl/ast.rs |  2 +-
 cranelift/src/bugpoint.rs              | 59 +++++++++++---------------
 cranelift/src/cat.rs                   |  2 +-
 cranelift/src/disasm.rs                | 21 ++++-----
 cranelift/src/print_cfg.rs             |  2 +-
 cranelift/src/run.rs                   |  4 +-
 cranelift/src/utils.rs                 |  4 +-
 7 files changed, 40 insertions(+), 54 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 3a994f27ed..644e601924 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -581,7 +581,7 @@ impl Apply {
         pred
     }
 
-    pub fn rust_builder(&self, defined_vars: &Vec, var_pool: &VarPool) -> String {
+    pub fn rust_builder(&self, defined_vars: &[VarIndex], var_pool: &VarPool) -> String {
         let mut args = self
             .args
             .iter()
diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index 2d753548b8..43ba5e2fda 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -202,13 +202,7 @@ impl Mutator for ReplaceInstWithConst {
 
             // Copy result SSA names into our own vector; otherwise we couldn't mutably borrow pos
             // in the loop below.
-            let results = pos
-                .func
-                .dfg
-                .inst_results(prev_inst)
-                .iter()
-                .cloned()
-                .collect::>();
+            let results = pos.func.dfg.inst_results(prev_inst).to_vec();
 
             // Detach results from the previous instruction, since we're going to reuse them.
             pos.func.dfg.clear_results(prev_inst);
@@ -390,15 +384,12 @@ impl Mutator for RemoveUnusedEntities {
                 let mut signatures_usage_map = HashMap::new();
                 for ebb in func.layout.ebbs() {
                     for inst in func.layout.ebb_insts(ebb) {
-                        match func.dfg[inst] {
-                            // Add new cases when there are new instruction formats taking a `SigRef`.
-                            InstructionData::CallIndirect { sig_ref, .. } => {
-                                signatures_usage_map
-                                    .entry(sig_ref)
-                                    .or_insert_with(Vec::new)
-                                    .push(SigRefUser::Instruction(inst));
-                            }
-                            _ => {}
+                        // Add new cases when there are new instruction formats taking a `SigRef`.
+                        if let InstructionData::CallIndirect { sig_ref, .. } = func.dfg[inst] {
+                            signatures_usage_map
+                                .entry(sig_ref)
+                                .or_insert_with(Vec::new)
+                                .push(SigRefUser::Instruction(inst));
                         }
                     }
                 }
@@ -500,15 +491,14 @@ impl Mutator for RemoveUnusedEntities {
                 let mut global_value_usage_map = HashMap::new();
                 for ebb in func.layout.ebbs() {
                     for inst in func.layout.ebb_insts(ebb) {
-                        match func.dfg[inst] {
-                            // Add new cases when there are new instruction formats taking a `GlobalValue`.
-                            InstructionData::UnaryGlobalValue { global_value, .. } => {
-                                global_value_usage_map
-                                    .entry(global_value)
-                                    .or_insert_with(Vec::new)
-                                    .push(inst);
-                            }
-                            _ => {}
+                        // Add new cases when there are new instruction formats taking a `GlobalValue`.
+                        if let InstructionData::UnaryGlobalValue { global_value, .. } =
+                            func.dfg[inst]
+                        {
+                            global_value_usage_map
+                                .entry(global_value)
+                                .or_insert_with(Vec::new)
+                                .push(inst);
                         }
                     }
                 }
@@ -519,8 +509,9 @@ impl Mutator for RemoveUnusedEntities {
                         // These can create cyclic references, which cause complications. Just skip
                         // the global value removal for now.
                         // FIXME Handle them in a better way.
-                        GlobalValueData::Load { base: _, .. }
-                        | GlobalValueData::IAddImm { base: _, .. } => return None,
+                        GlobalValueData::Load { .. } | GlobalValueData::IAddImm { .. } => {
+                            return None
+                        }
                     }
                 }
 
@@ -647,11 +638,11 @@ impl Mutator for MergeBlocks {
         // we'll start back to this EBB.
         self.prev_ebb = Some(pred.ebb);
 
-        return Some((
+        Some((
             func,
             format!("merged {} and {}", pred.ebb, ebb),
             ProgressStatus::ExpandedOrShrinked,
-        ));
+        ))
     }
 
     fn did_crash(&mut self) {
@@ -701,9 +692,9 @@ fn reduce(
 
     match context.check_for_crash(&func) {
         CheckResult::Succeed => {
-            return Err(format!(
-                "Given function compiled successfully or gave a verifier error."
-            ));
+            return Err(
+                "Given function compiled successfully or gave a verifier error.".to_string(),
+            );
         }
         CheckResult::Crash(_) => {}
     }
@@ -926,8 +917,8 @@ mod tests {
 
     #[test]
     fn test_reduce() {
-        const TEST: &'static str = include_str!("../tests/bugpoint_test.clif");
-        const EXPECTED: &'static str = include_str!("../tests/bugpoint_test_expected.clif");
+        const TEST: &str = include_str!("../tests/bugpoint_test.clif");
+        const EXPECTED: &str = include_str!("../tests/bugpoint_test_expected.clif");
 
         let test_file = parse_test(TEST, ParseOptions::default()).unwrap();
 
diff --git a/cranelift/src/cat.rs b/cranelift/src/cat.rs
index a98e509698..4477f10222 100644
--- a/cranelift/src/cat.rs
+++ b/cranelift/src/cat.rs
@@ -8,7 +8,7 @@ use crate::CommandResult;
 use cranelift_reader::parse_functions;
 
 pub fn run(files: &[String]) -> CommandResult {
-    for (i, f) in files.into_iter().enumerate() {
+    for (i, f) in files.iter().enumerate() {
         if i != 0 {
             println!();
         }
diff --git a/cranelift/src/disasm.rs b/cranelift/src/disasm.rs
index 78bf92999e..5e6d3bbe33 100644
--- a/cranelift/src/disasm.rs
+++ b/cranelift/src/disasm.rs
@@ -25,12 +25,7 @@ impl binemit::RelocSink for PrintRelocs {
         offset: binemit::CodeOffset,
     ) {
         if self.flag_print {
-            write!(
-                &mut self.text,
-                "reloc_ebb: {} {} at {}\n",
-                r, offset, where_
-            )
-            .unwrap();
+            writeln!(&mut self.text, "reloc_ebb: {} {} at {}", r, offset, where_).unwrap();
         }
     }
 
@@ -42,9 +37,9 @@ impl binemit::RelocSink for PrintRelocs {
         addend: binemit::Addend,
     ) {
         if self.flag_print {
-            write!(
+            writeln!(
                 &mut self.text,
-                "reloc_external: {} {} {} at {}\n",
+                "reloc_external: {} {} {} at {}",
                 r, name, addend, where_
             )
             .unwrap();
@@ -53,7 +48,7 @@ impl binemit::RelocSink for PrintRelocs {
 
     fn reloc_jt(&mut self, where_: binemit::CodeOffset, r: binemit::Reloc, jt: ir::JumpTable) {
         if self.flag_print {
-            write!(&mut self.text, "reloc_jt: {} {} at {}\n", r, jt, where_).unwrap();
+            writeln!(&mut self.text, "reloc_jt: {} {} at {}", r, jt, where_).unwrap();
         }
     }
 
@@ -64,9 +59,9 @@ impl binemit::RelocSink for PrintRelocs {
         constant: ir::ConstantOffset,
     ) {
         if self.flag_print {
-            write!(
+            writeln!(
                 &mut self.text,
-                "reloc_constant: {} {} at {}\n",
+                "reloc_constant: {} {} at {}",
                 reloc, constant, code_offset
             )
             .unwrap();
@@ -91,7 +86,7 @@ impl PrintTraps {
 impl binemit::TrapSink for PrintTraps {
     fn trap(&mut self, offset: binemit::CodeOffset, _srcloc: ir::SourceLoc, code: ir::TrapCode) {
         if self.flag_print {
-            write!(&mut self.text, "trap: {} at {}\n", code, offset).unwrap();
+            writeln!(&mut self.text, "trap: {} at {}", code, offset).unwrap();
         }
     }
 }
@@ -113,7 +108,7 @@ impl PrintStackmaps {
 impl binemit::StackmapSink for PrintStackmaps {
     fn add_stackmap(&mut self, offset: binemit::CodeOffset, _: binemit::Stackmap) {
         if self.flag_print {
-            write!(&mut self.text, "add_stackmap at {}\n", offset).unwrap();
+            writeln!(&mut self.text, "add_stackmap at {}", offset).unwrap();
         }
     }
 }
diff --git a/cranelift/src/print_cfg.rs b/cranelift/src/print_cfg.rs
index 30d91b5862..2f739f9f9d 100644
--- a/cranelift/src/print_cfg.rs
+++ b/cranelift/src/print_cfg.rs
@@ -9,7 +9,7 @@ use cranelift_codegen::cfg_printer::CFGPrinter;
 use cranelift_reader::parse_functions;
 
 pub fn run(files: &[String]) -> CommandResult {
-    for (i, f) in files.into_iter().enumerate() {
+    for (i, f) in files.iter().enumerate() {
         if i != 0 {
             println!();
         }
diff --git a/cranelift/src/run.rs b/cranelift/src/run.rs
index a3cd94b25d..701104fcf5 100644
--- a/cranelift/src/run.rs
+++ b/cranelift/src/run.rs
@@ -52,7 +52,7 @@ fn iterate_files(files: Vec) -> impl Iterator {
         .filter(|f| match f {
             Ok(d) => {
                 // filter out hidden files (starting with .)
-                !d.file_name().to_str().map_or(false, |s| s.starts_with("."))
+                !d.file_name().to_str().map_or(false, |s| s.starts_with('.'))
                     // filter out directories
                     && !d.file_type().is_dir()
             }
@@ -96,7 +96,7 @@ fn create_target_isa(isa_spec: &IsaSpec) -> Result, String> {
         let builder = host_isa_builder()?;
         Ok(builder.finish(flags.clone()))
     } else {
-        Err(String::from("A target ISA was specified in the file but should not have been--only the host ISA can be used for running CLIF files"))?
+        Err(String::from("A target ISA was specified in the file but should not have been--only the host ISA can be used for running CLIF files"))
     }
 }
 
diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs
index 908b310007..f520aaa000 100644
--- a/cranelift/src/utils.rs
+++ b/cranelift/src/utils.rs
@@ -49,8 +49,8 @@ impl OwnedFlagsOrIsa {
     /// Produce a FlagsOrIsa reference.
     pub fn as_fisa(&self) -> FlagsOrIsa {
         match *self {
-            OwnedFlagsOrIsa::Flags(ref flags) => FlagsOrIsa::from(flags),
-            OwnedFlagsOrIsa::Isa(ref isa) => FlagsOrIsa::from(&**isa),
+            Self::Flags(ref flags) => FlagsOrIsa::from(flags),
+            Self::Isa(ref isa) => FlagsOrIsa::from(&**isa),
         }
     }
 }

From 9f506692c2ce421120af8c9148377b672fb08a59 Mon Sep 17 00:00:00 2001
From: Peter Huene 
Date: Wed, 23 Oct 2019 23:15:42 -0700
Subject: [PATCH 2877/3084] Fix clippy warnings.

This commit fixes the current set of (stable) clippy warnings in the repo.
---
 cranelift/bforest/src/node.rs                 | 61 +++++++++---------
 cranelift/codegen/meta/src/cdsl/ast.rs        | 10 +--
 cranelift/codegen/meta/src/cdsl/cpu_modes.rs  |  2 +-
 .../codegen/meta/src/cdsl/instructions.rs     | 42 ++++++++----
 cranelift/codegen/meta/src/cdsl/recipes.rs    |  8 +--
 cranelift/codegen/meta/src/cdsl/regs.rs       | 11 ++--
 cranelift/codegen/meta/src/cdsl/settings.rs   |  8 +--
 .../codegen/meta/src/cdsl/type_inference.rs   | 17 ++---
 cranelift/codegen/meta/src/cdsl/types.rs      | 14 ++--
 cranelift/codegen/meta/src/cdsl/typevar.rs    | 56 ++++++++--------
 cranelift/codegen/meta/src/cdsl/xform.rs      |  1 +
 cranelift/codegen/meta/src/default_map.rs     |  2 +-
 cranelift/codegen/meta/src/gen_binemit.rs     |  4 +-
 cranelift/codegen/meta/src/gen_encodings.rs   | 12 ++--
 cranelift/codegen/meta/src/gen_inst.rs        | 51 +++++++--------
 cranelift/codegen/meta/src/gen_legalizer.rs   | 10 +--
 cranelift/codegen/meta/src/gen_registers.rs   |  2 +-
 cranelift/codegen/meta/src/gen_settings.rs    |  6 +-
 cranelift/codegen/meta/src/isa/mod.rs         |  5 +-
 .../codegen/meta/src/isa/riscv/encodings.rs   | 36 +++++------
 .../codegen/meta/src/isa/riscv/recipes.rs     | 11 ++--
 .../codegen/meta/src/isa/x86/encodings.rs     |  9 +--
 .../codegen/meta/src/isa/x86/instructions.rs  |  1 +
 .../codegen/meta/src/isa/x86/legalize.rs      | 13 ++--
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 23 ++++---
 cranelift/codegen/meta/src/lib.rs             |  2 +-
 .../codegen/meta/src/shared/instructions.rs   |  1 +
 cranelift/codegen/meta/src/shared/legalize.rs | 41 ++++++------
 cranelift/codegen/meta/src/srcgen.rs          |  4 +-
 cranelift/codegen/meta/src/unique_table.rs    | 12 ++--
 cranelift/codegen/shared/src/constant_hash.rs |  1 +
 cranelift/codegen/src/abi.rs                  | 14 ++--
 cranelift/codegen/src/binemit/mod.rs          | 16 ++---
 cranelift/codegen/src/binemit/relaxation.rs   |  4 +-
 cranelift/codegen/src/binemit/stackmap.rs     |  2 +-
 cranelift/codegen/src/bitset.rs               |  2 +-
 cranelift/codegen/src/cursor.rs               | 34 +++++-----
 .../codegen/src/divconst_magic_numbers.rs     |  4 +-
 cranelift/codegen/src/ir/constant.rs          | 31 ++++-----
 cranelift/codegen/src/ir/dfg.rs               |  6 +-
 cranelift/codegen/src/ir/entities.rs          | 64 +++++++++----------
 cranelift/codegen/src/ir/extfunc.rs           | 16 ++---
 cranelift/codegen/src/ir/extname.rs           | 12 ++--
 cranelift/codegen/src/ir/globalvalue.rs       | 17 ++---
 cranelift/codegen/src/ir/immediates.rs        | 48 +++++++-------
 cranelift/codegen/src/ir/instructions.rs      | 44 ++++++-------
 cranelift/codegen/src/ir/libcall.rs           | 40 ++++++------
 cranelift/codegen/src/ir/progpoint.rs         | 20 +++---
 cranelift/codegen/src/ir/sourceloc.rs         |  4 +-
 cranelift/codegen/src/ir/types.rs             |  6 +-
 cranelift/codegen/src/ir/valueloc.rs          | 16 ++---
 cranelift/codegen/src/isa/call_conv.rs        | 54 ++++++++--------
 cranelift/codegen/src/isa/registers.rs        |  4 +-
 cranelift/codegen/src/legalizer/mod.rs        |  9 ++-
 cranelift/codegen/src/lib.rs                  |  4 +-
 cranelift/codegen/src/licm.rs                 |  2 +-
 .../codegen/src/redundant_reload_remover.rs   | 19 ++----
 cranelift/codegen/src/regalloc/affinity.rs    | 26 ++++----
 .../codegen/src/regalloc/branch_splitting.rs  |  9 +--
 cranelift/codegen/src/regalloc/coloring.rs    |  2 +-
 cranelift/codegen/src/regalloc/diversion.rs   |  6 +-
 cranelift/codegen/src/regalloc/virtregs.rs    |  4 +-
 cranelift/codegen/src/result.rs               |  2 +-
 cranelift/codegen/src/settings.rs             |  2 +-
 cranelift/codegen/src/simple_preopt.rs        | 54 +++++++---------
 cranelift/codegen/src/timing.rs               |  2 +-
 cranelift/codegen/src/value_label.rs          | 24 +++----
 cranelift/codegen/src/verifier/mod.rs         |  4 +-
 cranelift/codegen/src/write.rs                | 35 +++++-----
 cranelift/entity/src/lib.rs                   |  5 +-
 cranelift/entity/src/packed_option.rs         |  4 +-
 cranelift/entity/src/set.rs                   |  3 +-
 cranelift/faerie/src/backend.rs               |  2 +-
 cranelift/faerie/src/lib.rs                   |  5 +-
 cranelift/filetests/src/function_runner.rs    | 13 ++--
 cranelift/filetests/src/runone.rs             |  4 +-
 cranelift/frontend/src/ssa.rs                 | 12 ++--
 cranelift/frontend/src/variable.rs            |  4 +-
 cranelift/module/src/data_context.rs          |  6 +-
 cranelift/module/src/module.rs                | 30 ++++-----
 cranelift/object/src/backend.rs               |  4 +-
 cranelift/object/src/lib.rs                   |  5 +-
 cranelift/preopt/src/constant_folding.rs      | 13 ++--
 cranelift/reader/src/isaspec.rs               |  2 +-
 cranelift/reader/src/lexer.rs                 |  1 +
 cranelift/reader/src/parser.rs                |  5 +-
 cranelift/src/bugpoint.rs                     |  1 +
 cranelift/src/wasm.rs                         |  2 +-
 cranelift/wasm/src/code_translator.rs         |  5 +-
 cranelift/wasm/src/environ/spec.rs            |  5 +-
 cranelift/wasm/src/sections_translator.rs     |  6 +-
 cranelift/wasm/src/state/func_state.rs        | 45 +++++++------
 cranelift/wasm/src/state/module_state.rs      |  9 ++-
 93 files changed, 667 insertions(+), 662 deletions(-)

diff --git a/cranelift/bforest/src/node.rs b/cranelift/bforest/src/node.rs
index 589bc2b5e0..53a0dca386 100644
--- a/cranelift/bforest/src/node.rs
+++ b/cranelift/bforest/src/node.rs
@@ -13,6 +13,7 @@ use core::fmt;
 ///
 /// An inner node contains at least M/2 node references unless it is the right-most node at its
 /// level. A leaf node contains at least N/2 keys unless it is the right-most leaf.
+#[allow(dead_code)] // workaround for https://github.com/rust-lang/rust/issues/64362
 pub(super) enum NodeData {
     Inner {
         /// The number of keys in this node.
@@ -55,7 +56,7 @@ impl NodeData {
     /// Is this a free/unused node?
     pub fn is_free(&self) -> bool {
         match *self {
-            NodeData::Free { .. } => true,
+            Self::Free { .. } => true,
             _ => false,
         }
     }
@@ -66,9 +67,9 @@ impl NodeData {
     /// a leaf node.
     pub fn entries(&self) -> usize {
         match *self {
-            NodeData::Inner { size, .. } => usize::from(size) + 1,
-            NodeData::Leaf { size, .. } => usize::from(size),
-            NodeData::Free { .. } => panic!("freed node"),
+            Self::Inner { size, .. } => usize::from(size) + 1,
+            Self::Leaf { size, .. } => usize::from(size),
+            Self::Free { .. } => panic!("freed node"),
         }
     }
 
@@ -78,7 +79,7 @@ impl NodeData {
         // Saves us from inventing a default/reserved value.
         let mut tree = [right; INNER_SIZE];
         tree[0] = left;
-        NodeData::Inner {
+        Self::Inner {
             size: 1,
             keys: [key; INNER_SIZE - 1],
             tree,
@@ -87,7 +88,7 @@ impl NodeData {
 
     /// Create a leaf node with a single key-value pair.
     pub fn leaf(key: F::Key, value: F::Value) -> Self {
-        NodeData::Leaf {
+        Self::Leaf {
             size: 1,
             keys: F::splat_key(key),
             vals: F::splat_value(value),
@@ -97,7 +98,7 @@ impl NodeData {
     /// Unwrap an inner node into two slices (keys, trees).
     pub fn unwrap_inner(&self) -> (&[F::Key], &[Node]) {
         match *self {
-            NodeData::Inner {
+            Self::Inner {
                 size,
                 ref keys,
                 ref tree,
@@ -114,7 +115,7 @@ impl NodeData {
     /// Unwrap a leaf node into two slices (keys, values) of the same length.
     pub fn unwrap_leaf(&self) -> (&[F::Key], &[F::Value]) {
         match *self {
-            NodeData::Leaf {
+            Self::Leaf {
                 size,
                 ref keys,
                 ref vals,
@@ -133,7 +134,7 @@ impl NodeData {
     /// Unwrap a mutable leaf node into two slices (keys, values) of the same length.
     pub fn unwrap_leaf_mut(&mut self) -> (&mut [F::Key], &mut [F::Value]) {
         match *self {
-            NodeData::Leaf {
+            Self::Leaf {
                 size,
                 ref mut keys,
                 ref mut vals,
@@ -153,7 +154,7 @@ impl NodeData {
     /// This is simply the first key.
     pub fn leaf_crit_key(&self) -> F::Key {
         match *self {
-            NodeData::Leaf { size, ref keys, .. } => {
+            Self::Leaf { size, ref keys, .. } => {
                 debug_assert!(size > 0, "Empty leaf node");
                 keys.borrow()[0]
             }
@@ -166,7 +167,7 @@ impl NodeData {
     /// If the node is full, this leaves the node unchanged and returns false.
     pub fn try_inner_insert(&mut self, index: usize, key: F::Key, node: Node) -> bool {
         match *self {
-            NodeData::Inner {
+            Self::Inner {
                 ref mut size,
                 ref mut keys,
                 ref mut tree,
@@ -192,7 +193,7 @@ impl NodeData {
     /// is full.
     pub fn try_leaf_insert(&mut self, index: usize, key: F::Key, value: F::Value) -> bool {
         match *self {
-            NodeData::Leaf {
+            Self::Leaf {
                 ref mut size,
                 ref mut keys,
                 ref mut vals,
@@ -223,7 +224,7 @@ impl NodeData {
     /// node will be split in half with a bias towards an even split after the insertion is retried.
     pub fn split(&mut self, insert_index: usize) -> SplitOff {
         match *self {
-            NodeData::Inner {
+            Self::Inner {
                 ref mut size,
                 ref keys,
                 ref tree,
@@ -255,14 +256,14 @@ impl NodeData {
                     lhs_entries: l_ents,
                     rhs_entries: r_ents,
                     crit_key: keys[l_ents - 1],
-                    rhs_data: NodeData::Inner {
+                    rhs_data: Self::Inner {
                         size: (r_ents - 1) as u8,
                         keys: r_keys,
                         tree: r_tree,
                     },
                 }
             }
-            NodeData::Leaf {
+            Self::Leaf {
                 ref mut size,
                 ref keys,
                 ref vals,
@@ -288,7 +289,7 @@ impl NodeData {
                     lhs_entries: l_size,
                     rhs_entries: r_size,
                     crit_key: o_keys[l_size],
-                    rhs_data: NodeData::Leaf {
+                    rhs_data: Self::Leaf {
                         size: r_size as u8,
                         keys: r_keys,
                         vals: r_vals,
@@ -308,7 +309,7 @@ impl NodeData {
     /// Return an indication of the node's health (i.e. below half capacity).
     pub fn inner_remove(&mut self, index: usize) -> Removed {
         match *self {
-            NodeData::Inner {
+            Self::Inner {
                 ref mut size,
                 ref mut keys,
                 ref mut tree,
@@ -333,7 +334,7 @@ impl NodeData {
     /// Return an indication of the node's health (i.e. below half capacity).
     pub fn leaf_remove(&mut self, index: usize) -> Removed {
         match *self {
-            NodeData::Leaf {
+            Self::Leaf {
                 ref mut size,
                 ref mut keys,
                 ref mut vals,
@@ -363,12 +364,12 @@ impl NodeData {
     pub fn balance(&mut self, crit_key: F::Key, rhs: &mut Self) -> Option {
         match (self, rhs) {
             (
-                &mut NodeData::Inner {
+                &mut Self::Inner {
                     size: ref mut l_size,
                     keys: ref mut l_keys,
                     tree: ref mut l_tree,
                 },
-                &mut NodeData::Inner {
+                &mut Self::Inner {
                     size: ref mut r_size,
                     keys: ref mut r_keys,
                     tree: ref mut r_tree,
@@ -411,12 +412,12 @@ impl NodeData {
                 }
             }
             (
-                &mut NodeData::Leaf {
+                &mut Self::Leaf {
                     size: ref mut l_size,
                     keys: ref mut l_keys,
                     vals: ref mut l_vals,
                 },
-                &mut NodeData::Leaf {
+                &mut Self::Leaf {
                     size: ref mut r_size,
                     keys: ref mut r_keys,
                     vals: ref mut r_vals,
@@ -521,14 +522,14 @@ impl Removed {
     fn new(removed: usize, new_size: usize, capacity: usize) -> Self {
         if 2 * new_size >= capacity {
             if removed == new_size {
-                Removed::Rightmost
+                Self::Rightmost
             } else {
-                Removed::Healthy
+                Self::Healthy
             }
         } else if new_size > 0 {
-            Removed::Underflow
+            Self::Underflow
         } else {
-            Removed::Empty
+            Self::Empty
         }
     }
 }
@@ -558,14 +559,14 @@ where
 {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
-            NodeData::Inner { size, keys, tree } => {
+            Self::Inner { size, keys, tree } => {
                 write!(f, "[ {}", tree[0])?;
                 for i in 0..usize::from(size) {
                     write!(f, " {} {}", keys[i], tree[i + 1])?;
                 }
                 write!(f, " ]")
             }
-            NodeData::Leaf { size, keys, vals } => {
+            Self::Leaf { size, keys, vals } => {
                 let keys = keys.borrow();
                 let vals = vals.borrow();
                 write!(f, "[")?;
@@ -575,8 +576,8 @@ where
                 }
                 write!(f, " ]")
             }
-            NodeData::Free { next: Some(n) } => write!(f, "[ free -> {} ]", n),
-            NodeData::Free { next: None } => write!(f, "[ free ]"),
+            Self::Free { next: Some(n) } => write!(f, "[ free -> {} ]", n),
+            Self::Free { next: None } => write!(f, "[ free ]"),
         }
     }
 }
diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 644e601924..47dd457e23 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -168,10 +168,12 @@ pub(crate) enum Literal {
 impl Literal {
     pub fn enumerator_for(kind: &OperandKind, value: &'static str) -> Self {
         let value = match &kind.fields {
-            OperandKindFields::ImmEnum(values) => values.get(value).expect(&format!(
-                "nonexistent value '{}' in enumeration '{}'",
-                value, kind.name
-            )),
+            OperandKindFields::ImmEnum(values) => values.get(value).unwrap_or_else(|| {
+                panic!(
+                    "nonexistent value '{}' in enumeration '{}'",
+                    value, kind.name
+                )
+            }),
             _ => panic!("enumerator is for enum values"),
         };
         Literal::Enumerator {
diff --git a/cranelift/codegen/meta/src/cdsl/cpu_modes.rs b/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
index 6436d2e4a2..e62ba72295 100644
--- a/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
+++ b/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
@@ -53,7 +53,7 @@ impl CpuMode {
             Some(typ) => self
                 .typed_legalize
                 .get(typ)
-                .map(|x| *x)
+                .copied()
                 .unwrap_or_else(|| self.get_default_legalize_code()),
             None => self
                 .monomorphic_legalize
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 7663d5e22a..824d028741 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -61,8 +61,8 @@ impl InstructionGroup {
     pub fn by_name(&self, name: &'static str) -> &Instruction {
         self.instructions
             .iter()
-            .find(|inst| &inst.name == name)
-            .expect(&format!("unexisting instruction with name {}", name))
+            .find(|inst| inst.name == name)
+            .unwrap_or_else(|| panic!("unexisting instruction with name {}", name))
     }
 }
 
@@ -167,7 +167,7 @@ impl Bindable for Instruction {
 
 impl fmt::Display for InstructionContent {
     fn fmt(&self, fmt: &mut fmt::Formatter) -> Result<(), fmt::Error> {
-        if self.operands_out.len() > 0 {
+        if !self.operands_out.is_empty() {
             let operands_out = self
                 .operands_out
                 .iter()
@@ -180,7 +180,7 @@ impl fmt::Display for InstructionContent {
 
         fmt.write_str(&self.name)?;
 
-        if self.operands_in.len() > 0 {
+        if !self.operands_in.is_empty() {
             let operands_in = self
                 .operands_in
                 .iter()
@@ -244,53 +244,70 @@ impl InstructionBuilder {
         self.operands_in = Some(operands.iter().map(|x| (*x).clone()).collect());
         self
     }
+
     pub fn operands_out(mut self, operands: Vec<&Operand>) -> Self {
         assert!(self.operands_out.is_none());
         self.operands_out = Some(operands.iter().map(|x| (*x).clone()).collect());
         self
     }
+
     pub fn constraints(mut self, constraints: Vec) -> Self {
         assert!(self.constraints.is_none());
         self.constraints = Some(constraints);
         self
     }
 
+    #[allow(clippy::wrong_self_convention)]
     pub fn is_terminator(mut self, val: bool) -> Self {
         self.is_terminator = val;
         self
     }
+
+    #[allow(clippy::wrong_self_convention)]
     pub fn is_branch(mut self, val: bool) -> Self {
         self.is_branch = val;
         self
     }
+
+    #[allow(clippy::wrong_self_convention)]
     pub fn is_indirect_branch(mut self, val: bool) -> Self {
         self.is_indirect_branch = val;
         self
     }
+
+    #[allow(clippy::wrong_self_convention)]
     pub fn is_call(mut self, val: bool) -> Self {
         self.is_call = val;
         self
     }
+
+    #[allow(clippy::wrong_self_convention)]
     pub fn is_return(mut self, val: bool) -> Self {
         self.is_return = val;
         self
     }
+
+    #[allow(clippy::wrong_self_convention)]
     pub fn is_ghost(mut self, val: bool) -> Self {
         self.is_ghost = val;
         self
     }
+
     pub fn can_load(mut self, val: bool) -> Self {
         self.can_load = val;
         self
     }
+
     pub fn can_store(mut self, val: bool) -> Self {
         self.can_store = val;
         self
     }
+
     pub fn can_trap(mut self, val: bool) -> Self {
         self.can_trap = val;
         self
     }
+
     pub fn other_side_effects(mut self, val: bool) -> Self {
         self.other_side_effects = val;
         self
@@ -534,7 +551,7 @@ impl Bindable for BoundInstruction {
 }
 
 /// Checks that the input operands actually match the given format.
-fn verify_format(inst_name: &str, operands_in: &Vec, format: &InstructionFormat) {
+fn verify_format(inst_name: &str, operands_in: &[Operand], format: &InstructionFormat) {
     // A format is defined by:
     // - its number of input value operands,
     // - its number and names of input immediate operands,
@@ -586,10 +603,10 @@ fn verify_format(inst_name: &str, operands_in: &Vec, format: &Instructi
 
 /// Check if this instruction is polymorphic, and verify its use of type variables.
 fn verify_polymorphic(
-    operands_in: &Vec,
-    operands_out: &Vec,
+    operands_in: &[Operand],
+    operands_out: &[Operand],
     format: &InstructionFormat,
-    value_opnums: &Vec,
+    value_opnums: &[usize],
 ) -> Option {
     // The instruction is polymorphic if it has one free input or output operand.
     let is_polymorphic = operands_in
@@ -633,7 +650,7 @@ fn verify_polymorphic(
     // If we reached here, it means the type variable indicated as the typevar operand couldn't
     // control every other input and output type variable. We need to look at the result type
     // variables.
-    if operands_out.len() == 0 {
+    if operands_out.is_empty() {
         // No result means no other possible type variable, so it's a type inference failure.
         match maybe_error_message {
             Some(msg) => panic!(msg),
@@ -670,8 +687,8 @@ fn verify_polymorphic(
 /// Return a vector of other type variables used, or a string explaining what went wrong.
 fn is_ctrl_typevar_candidate(
     ctrl_typevar: &TypeVar,
-    operands_in: &Vec,
-    operands_out: &Vec,
+    operands_in: &[Operand],
+    operands_out: &[Operand],
 ) -> Result, String> {
     let mut other_typevars = Vec::new();
 
@@ -995,8 +1012,7 @@ impl InstructionPredicate {
             .value_opnums
             .iter()
             .enumerate()
-            .filter(|(_, &op_num)| inst.operands_in[op_num].type_var().unwrap() == type_var)
-            .next()
+            .find(|(_, &op_num)| inst.operands_in[op_num].type_var().unwrap() == type_var)
             .unwrap()
             .0;
         InstructionPredicateNode::TypePredicate(TypePredicateNode::TypeVarCheck(
diff --git a/cranelift/codegen/meta/src/cdsl/recipes.rs b/cranelift/codegen/meta/src/cdsl/recipes.rs
index 20b5c077cb..4aad4debe2 100644
--- a/cranelift/codegen/meta/src/cdsl/recipes.rs
+++ b/cranelift/codegen/meta/src/cdsl/recipes.rs
@@ -42,7 +42,7 @@ impl Stack {
     pub fn new(regclass: RegClassIndex) -> Self {
         Self { regclass }
     }
-    pub fn stack_base_mask(&self) -> &'static str {
+    pub fn stack_base_mask(self) -> &'static str {
         // TODO: Make this configurable instead of just using the SP.
         "StackBaseMask(1)"
     }
@@ -253,8 +253,8 @@ impl EncodingRecipeBuilder {
     }
 
     pub fn build(self) -> EncodingRecipe {
-        let operands_in = self.operands_in.unwrap_or(Vec::new());
-        let operands_out = self.operands_out.unwrap_or(Vec::new());
+        let operands_in = self.operands_in.unwrap_or_default();
+        let operands_out = self.operands_out.unwrap_or_default();
 
         // The number of input constraints must match the number of format input operands.
         if !self.format.has_value_list {
@@ -282,7 +282,7 @@ impl EncodingRecipeBuilder {
         let clobbers_flags = self.clobbers_flags.unwrap_or(true);
 
         EncodingRecipe {
-            name: self.name.into(),
+            name: self.name,
             format: self.format,
             base_size: self.base_size,
             operands_in,
diff --git a/cranelift/codegen/meta/src/cdsl/regs.rs b/cranelift/codegen/meta/src/cdsl/regs.rs
index d60835ab3f..69efe6b660 100644
--- a/cranelift/codegen/meta/src/cdsl/regs.rs
+++ b/cranelift/codegen/meta/src/cdsl/regs.rs
@@ -164,7 +164,7 @@ impl RegClassBuilder {
             name,
             width: 0,
             count: stop - start,
-            start: start,
+            start,
             proto: RegClassProto::SubClass(parent_index),
         }
     }
@@ -214,7 +214,7 @@ impl RegBankBuilder {
         self
     }
     pub fn pinned_reg(mut self, unit: u16) -> Self {
-        assert!(unit < (self.units as u16));
+        assert!(unit < u16::from(self.units));
         self.pinned_reg = Some(unit);
         self
     }
@@ -234,7 +234,7 @@ impl IsaRegsBuilder {
     }
 
     pub fn add_bank(&mut self, builder: RegBankBuilder) -> RegBankIndex {
-        let first_unit = if self.banks.len() == 0 {
+        let first_unit = if self.banks.is_empty() {
             0
         } else {
             let last = &self.banks.last().unwrap();
@@ -358,8 +358,7 @@ impl IsaRegsBuilder {
                             .unwrap()
                             .subclasses
                             .iter()
-                            .find(|x| **x == *i2)
-                            .is_some());
+                            .any(|x| *x == *i2));
                     }
                 }
             }
@@ -402,7 +401,7 @@ impl IsaRegs {
         self.classes
             .values()
             .find(|&class| class.name == name)
-            .expect(&format!("register class {} not found", name))
+            .unwrap_or_else(|| panic!("register class {} not found", name))
             .index
     }
 
diff --git a/cranelift/codegen/meta/src/cdsl/settings.rs b/cranelift/codegen/meta/src/cdsl/settings.rs
index 5a45d9fb9b..3d56fa9839 100644
--- a/cranelift/codegen/meta/src/cdsl/settings.rs
+++ b/cranelift/codegen/meta/src/cdsl/settings.rs
@@ -268,7 +268,7 @@ impl SettingGroupBuilder {
         default: bool,
     ) -> BoolSettingIndex {
         assert!(
-            self.predicates.len() == 0,
+            self.predicates.is_empty(),
             "predicates must be added after the boolean settings"
         );
         self.add_setting(name, comment, ProtoSpecificSetting::Bool(default));
@@ -379,7 +379,7 @@ impl SettingGroupBuilder {
         }
 
         assert!(
-            group.predicates.len() == 0,
+            group.predicates.is_empty(),
             "settings_size is the byte size before adding predicates"
         );
         group.settings_size = group.byte_size();
@@ -393,11 +393,11 @@ impl SettingGroupBuilder {
             .extend(predicates.into_iter().map(|predicate| {
                 let number = predicate_number;
                 predicate_number += 1;
-                return Predicate {
+                Predicate {
                     name: predicate.name,
                     node: predicate.node,
                     number,
-                };
+                }
             }));
 
         group.presets.extend(self.presets);
diff --git a/cranelift/codegen/meta/src/cdsl/type_inference.rs b/cranelift/codegen/meta/src/cdsl/type_inference.rs
index 9cb894d911..9852dd1e7b 100644
--- a/cranelift/codegen/meta/src/cdsl/type_inference.rs
+++ b/cranelift/codegen/meta/src/cdsl/type_inference.rs
@@ -74,7 +74,7 @@ impl Constraint {
                 }
 
                 // Trivially false.
-                if (&ts1.lanes & &ts2.lanes).len() == 0 {
+                if (&ts1.lanes & &ts2.lanes).is_empty() {
                     return true;
                 }
 
@@ -155,12 +155,7 @@ impl TypeEnvironment {
     }
 
     fn add_constraint(&mut self, constraint: Constraint) {
-        if self
-            .constraints
-            .iter()
-            .find(|&item| item == &constraint)
-            .is_some()
-        {
+        if self.constraints.iter().any(|item| *item == constraint) {
             return;
         }
 
@@ -257,7 +252,7 @@ impl TypeEnvironment {
                 .map(|tv| self.get_equivalent(tv).free_typevar())
                 .filter(|opt_tv| {
                     // Filter out singleton types.
-                    return opt_tv.is_some();
+                    opt_tv.is_some()
                 })
                 .map(|tv| tv.unwrap()),
         );
@@ -306,7 +301,7 @@ impl TypeEnvironment {
 
             children
                 .entry(parent_tv)
-                .or_insert(HashSet::new())
+                .or_insert_with(HashSet::new)
                 .insert(type_var.clone());
         }
 
@@ -314,7 +309,7 @@ impl TypeEnvironment {
         for (equivalent_tv, canon_tv) in self.equivalency_map.iter() {
             children
                 .entry(canon_tv.clone())
-                .or_insert(HashSet::new())
+                .or_insert_with(HashSet::new)
                 .insert(equivalent_tv.clone());
         }
 
@@ -604,7 +599,7 @@ fn infer_definition(
 /// Perform type inference on an transformation. Return an updated type environment or error.
 pub(crate) fn infer_transform(
     src: DefIndex,
-    dst: &Vec,
+    dst: &[DefIndex],
     def_pool: &DefPool,
     var_pool: &mut VarPool,
 ) -> Result {
diff --git a/cranelift/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
index 33ce0acfc1..83faa538c3 100644
--- a/cranelift/codegen/meta/src/cdsl/types.rs
+++ b/cranelift/codegen/meta/src/cdsl/types.rs
@@ -6,7 +6,7 @@ use crate::shared::types as shared_types;
 use cranelift_codegen_shared::constants;
 
 // Rust name prefix used for the `rust_name` method.
-static _RUST_NAME_PREFIX: &'static str = "ir::types::";
+static _RUST_NAME_PREFIX: &str = "ir::types::";
 
 // ValueType variants (i8, i32, ...) are provided in `shared::types.rs`.
 
@@ -242,29 +242,29 @@ impl LaneType {
         })
     }
 
-    pub fn by(&self, lanes: u16) -> ValueType {
+    pub fn by(self, lanes: u16) -> ValueType {
         if lanes == 1 {
-            (*self).into()
+            self.into()
         } else {
-            ValueType::Vector(VectorType::new(*self, lanes.into()))
+            ValueType::Vector(VectorType::new(self, lanes.into()))
         }
     }
 
-    pub fn is_float(&self) -> bool {
+    pub fn is_float(self) -> bool {
         match self {
             LaneType::FloatType(_) => true,
             _ => false,
         }
     }
 
-    pub fn is_int(&self) -> bool {
+    pub fn is_int(self) -> bool {
         match self {
             LaneType::IntType(_) => true,
             _ => false,
         }
     }
 
-    pub fn is_bool(&self) -> bool {
+    pub fn is_bool(self) -> bool {
         match self {
             LaneType::BoolType(_) => true,
             _ => false,
diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 78e702cacf..947d0dd82d 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -156,33 +156,33 @@ impl TypeVar {
         let ts = self.get_typeset();
 
         // Safety checks to avoid over/underflows.
-        assert!(ts.specials.len() == 0, "can't derive from special types");
+        assert!(ts.specials.is_empty(), "can't derive from special types");
         match derived_func {
             DerivedFunc::HalfWidth => {
                 assert!(
-                    ts.ints.len() == 0 || *ts.ints.iter().min().unwrap() > 8,
+                    ts.ints.is_empty() || *ts.ints.iter().min().unwrap() > 8,
                     "can't halve all integer types"
                 );
                 assert!(
-                    ts.floats.len() == 0 || *ts.floats.iter().min().unwrap() > 32,
+                    ts.floats.is_empty() || *ts.floats.iter().min().unwrap() > 32,
                     "can't halve all float types"
                 );
                 assert!(
-                    ts.bools.len() == 0 || *ts.bools.iter().min().unwrap() > 8,
+                    ts.bools.is_empty() || *ts.bools.iter().min().unwrap() > 8,
                     "can't halve all boolean types"
                 );
             }
             DerivedFunc::DoubleWidth => {
                 assert!(
-                    ts.ints.len() == 0 || *ts.ints.iter().max().unwrap() < MAX_BITS,
+                    ts.ints.is_empty() || *ts.ints.iter().max().unwrap() < MAX_BITS,
                     "can't double all integer types"
                 );
                 assert!(
-                    ts.floats.len() == 0 || *ts.floats.iter().max().unwrap() < MAX_FLOAT_BITS,
+                    ts.floats.is_empty() || *ts.floats.iter().max().unwrap() < MAX_FLOAT_BITS,
                     "can't double all float types"
                 );
                 assert!(
-                    ts.bools.len() == 0 || *ts.bools.iter().max().unwrap() < MAX_BITS,
+                    ts.bools.is_empty() || *ts.bools.iter().max().unwrap() < MAX_BITS,
                     "can't double all boolean types"
                 );
             }
@@ -203,7 +203,7 @@ impl TypeVar {
             }
         }
 
-        return TypeVar {
+        TypeVar {
             content: Rc::new(RefCell::new(TypeVarContent {
                 name: format!("{}({})", derived_func.name(), self.name),
                 doc: "".into(),
@@ -213,29 +213,29 @@ impl TypeVar {
                     derived_func,
                 }),
             })),
-        };
+        }
     }
 
     pub fn lane_of(&self) -> TypeVar {
-        return self.derived(DerivedFunc::LaneOf);
+        self.derived(DerivedFunc::LaneOf)
     }
     pub fn as_bool(&self) -> TypeVar {
-        return self.derived(DerivedFunc::AsBool);
+        self.derived(DerivedFunc::AsBool)
     }
     pub fn half_width(&self) -> TypeVar {
-        return self.derived(DerivedFunc::HalfWidth);
+        self.derived(DerivedFunc::HalfWidth)
     }
     pub fn double_width(&self) -> TypeVar {
-        return self.derived(DerivedFunc::DoubleWidth);
+        self.derived(DerivedFunc::DoubleWidth)
     }
     pub fn half_vector(&self) -> TypeVar {
-        return self.derived(DerivedFunc::HalfVector);
+        self.derived(DerivedFunc::HalfVector)
     }
     pub fn double_vector(&self) -> TypeVar {
-        return self.derived(DerivedFunc::DoubleVector);
+        self.derived(DerivedFunc::DoubleVector)
     }
     pub fn to_bitvec(&self) -> TypeVar {
-        return self.derived(DerivedFunc::ToBitVec);
+        self.derived(DerivedFunc::ToBitVec)
     }
 
     /// Constrain the range of types this variable can assume to a subset of those in the typeset
@@ -347,7 +347,7 @@ pub enum DerivedFunc {
 }
 
 impl DerivedFunc {
-    pub fn name(&self) -> &'static str {
+    pub fn name(self) -> &'static str {
         match self {
             DerivedFunc::LaneOf => "lane_of",
             DerivedFunc::AsBool => "as_bool",
@@ -360,7 +360,7 @@ impl DerivedFunc {
     }
 
     /// Returns the inverse function of this one, if it is a bijection.
-    pub fn inverse(&self) -> Option {
+    pub fn inverse(self) -> Option {
         match self {
             DerivedFunc::HalfWidth => Some(DerivedFunc::DoubleWidth),
             DerivedFunc::DoubleWidth => Some(DerivedFunc::HalfWidth),
@@ -476,7 +476,7 @@ impl TypeSet {
         copy.floats = NumSet::new();
         copy.refs = NumSet::new();
         copy.bitvecs = NumSet::new();
-        if (&self.lanes - &num_set![1]).len() > 0 {
+        if !(&self.lanes - &num_set![1]).is_empty() {
             copy.bools = &self.ints | &self.floats;
             copy.bools = ©.bools | &self.bools;
         }
@@ -512,7 +512,7 @@ impl TypeSet {
                 .iter()
                 .filter(|&&x| x < MAX_BITS)
                 .map(|&x| x * 2)
-                .filter(legal_bool),
+                .filter(|x| legal_bool(*x)),
         );
         copy.bitvecs = NumSet::from_iter(
             self.bitvecs
@@ -600,7 +600,7 @@ impl TypeSet {
     fn get_singleton(&self) -> ValueType {
         let mut types = self.concrete_types();
         assert_eq!(types.len(), 1);
-        return types.remove(0);
+        types.remove(0)
     }
 
     /// Return the inverse image of self across the derived function func.
@@ -615,7 +615,7 @@ impl TypeSet {
                 let mut copy = self.clone();
                 copy.bitvecs = NumSet::new();
                 copy.lanes =
-                    NumSet::from_iter((0..MAX_LANES.trailing_zeros() + 1).map(|i| u16::pow(2, i)));
+                    NumSet::from_iter((0..=MAX_LANES.trailing_zeros()).map(|i| u16::pow(2, i)));
                 copy
             }
             DerivedFunc::AsBool => {
@@ -724,11 +724,11 @@ impl TypeSet {
 }
 
 fn set_wider_or_equal(s1: &NumSet, s2: &NumSet) -> bool {
-    s1.len() > 0 && s2.len() > 0 && s1.iter().min() >= s2.iter().max()
+    !s1.is_empty() && !s2.is_empty() && s1.iter().min() >= s2.iter().max()
 }
 
 fn set_narrower(s1: &NumSet, s2: &NumSet) -> bool {
-    s1.len() > 0 && s2.len() > 0 && s1.iter().min() < s2.iter().max()
+    !s1.is_empty() && !s2.is_empty() && s1.iter().min() < s2.iter().max()
 }
 
 impl fmt::Debug for TypeSet {
@@ -854,7 +854,7 @@ impl TypeSetBuilder {
 
         let bools = range_to_set(self.bools.to_range(1..MAX_BITS, None))
             .into_iter()
-            .filter(legal_bool)
+            .filter(|x| legal_bool(*x))
             .collect();
 
         TypeSet::new(
@@ -921,9 +921,9 @@ impl Into for Range {
     }
 }
 
-fn legal_bool(bits: &RangeBound) -> bool {
+fn legal_bool(bits: RangeBound) -> bool {
     // Only allow legal bit widths for bool types.
-    *bits == 1 || (*bits >= 8 && *bits <= MAX_BITS && bits.is_power_of_two())
+    bits == 1 || (bits >= 8 && bits <= MAX_BITS && bits.is_power_of_two())
 }
 
 /// Generates a set with all the powers of two included in the range.
@@ -939,7 +939,7 @@ fn range_to_set(range: Option) -> NumSet {
     assert!(high.is_power_of_two());
     assert!(low <= high);
 
-    for i in low.trailing_zeros()..high.trailing_zeros() + 1 {
+    for i in low.trailing_zeros()..=high.trailing_zeros() {
         assert!(1 << i <= RangeBound::max_value());
         set.insert(1 << i);
     }
diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index e9798c404a..56f6baf949 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -255,6 +255,7 @@ fn rewrite_expr(
     Apply::new(apply_target, args)
 }
 
+#[allow(clippy::too_many_arguments)]
 fn rewrite_def_list(
     position: PatternPosition,
     dummy_defs: Vec,
diff --git a/cranelift/codegen/meta/src/default_map.rs b/cranelift/codegen/meta/src/default_map.rs
index 0a2f7fc41a..951a9b2491 100644
--- a/cranelift/codegen/meta/src/default_map.rs
+++ b/cranelift/codegen/meta/src/default_map.rs
@@ -7,7 +7,7 @@ pub(crate) trait MapWithDefault {
 
 impl MapWithDefault for HashMap {
     fn get_or_default(&mut self, k: K) -> &mut V {
-        self.entry(k).or_insert_with(|| V::default())
+        self.entry(k).or_insert_with(V::default)
     }
 }
 
diff --git a/cranelift/codegen/meta/src/gen_binemit.rs b/cranelift/codegen/meta/src/gen_binemit.rs
index 3d7b08a227..f67aa9b5a9 100644
--- a/cranelift/codegen/meta/src/gen_binemit.rs
+++ b/cranelift/codegen/meta/src/gen_binemit.rs
@@ -34,7 +34,7 @@ fn gen_recipe(recipe: &EncodingRecipe, fmt: &mut Formatter) {
     let is_regmove = ["RegMove", "RegSpill", "RegFill"].contains(&inst_format.name);
 
     // Unpack the instruction data.
-    fmtln!(fmt, "if let &InstructionData::{} {{", inst_format.name);
+    fmtln!(fmt, "if let InstructionData::{} {{", inst_format.name);
     fmt.indent(|fmt| {
         fmt.line("opcode,");
         for f in &inst_format.imm_fields {
@@ -49,7 +49,7 @@ fn gen_recipe(recipe: &EncodingRecipe, fmt: &mut Formatter) {
         }
         fmt.line("..");
 
-        fmt.outdented_line("} = inst_data {");
+        fmt.outdented_line("} = *inst_data {");
 
         // Pass recipe arguments in this order: inputs, imm_fields, outputs.
         let mut args = String::new();
diff --git a/cranelift/codegen/meta/src/gen_encodings.rs b/cranelift/codegen/meta/src/gen_encodings.rs
index 86046cfde1..d79dc66340 100644
--- a/cranelift/codegen/meta/src/gen_encodings.rs
+++ b/cranelift/codegen/meta/src/gen_encodings.rs
@@ -158,8 +158,8 @@ fn emit_recipe_predicates(isa: &TargetIsa, fmt: &mut Formatter) {
             fmt,
             "fn {}({}: crate::settings::PredicateView, {}: &ir::InstructionData) -> bool {{",
             func_name,
-            if let Some(_) = isap { "isap" } else { "_" },
-            if let Some(_) = instp { "inst" } else { "_" }
+            if isap.is_some() { "isap" } else { "_" },
+            if instp.is_some() { "inst" } else { "_" }
         );
         fmt.indent(|fmt| {
             match (isap, instp) {
@@ -263,13 +263,13 @@ fn emit_recipe_names(isa: &TargetIsa, fmt: &mut Formatter) {
 }
 
 /// Returns a set of all the registers involved in fixed register constraints.
-fn get_fixed_registers(operands_in: &Vec) -> HashSet {
+fn get_fixed_registers(operands_in: &[OperandConstraint]) -> HashSet {
     HashSet::from_iter(
         operands_in
             .iter()
             .map(|constraint| {
                 if let OperandConstraint::FixedReg(reg) = &constraint {
-                    Some(reg.clone())
+                    Some(*reg)
                 } else {
                     None
                 }
@@ -286,13 +286,13 @@ fn get_fixed_registers(operands_in: &Vec) -> HashSet,
+    constraints: &[OperandConstraint],
     field_name: &'static str,
     tied_operands: &HashMap,
     fixed_registers: &HashSet,
     fmt: &mut Formatter,
 ) {
-    if constraints.len() == 0 {
+    if constraints.is_empty() {
         fmtln!(fmt, "{}: &[],", field_name);
         return;
     }
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index c1a6277b95..39309e05c0 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -17,7 +17,7 @@ use crate::unique_table::{UniqueSeqTable, UniqueTable};
 const TYPESET_LIMIT: usize = 0xff;
 
 /// Generate an instruction format enumeration.
-fn gen_formats(formats: &Vec<&InstructionFormat>, fmt: &mut Formatter) {
+fn gen_formats(formats: &[&InstructionFormat], fmt: &mut Formatter) {
     fmt.doc_comment(
         r#"
         An instruction format
@@ -49,7 +49,7 @@ fn gen_formats(formats: &Vec<&InstructionFormat>, fmt: &mut Formatter) {
                 m.arm(
                     format!("InstructionData::{}", format.name),
                     vec![".."],
-                    format!("InstructionFormat::{}", format.name),
+                    format!("Self::{}", format.name),
                 );
             }
             fmt.add_match(m);
@@ -65,7 +65,7 @@ fn gen_formats(formats: &Vec<&InstructionFormat>, fmt: &mut Formatter) {
 /// Every variant must contain an `opcode` field. The size of `InstructionData` should be kept at
 /// 16 bytes on 64-bit architectures. If more space is needed to represent an instruction, use a
 /// `ValueList` to store the additional information out of line.
-fn gen_instruction_data(formats: &Vec<&InstructionFormat>, fmt: &mut Formatter) {
+fn gen_instruction_data(formats: &[&InstructionFormat], fmt: &mut Formatter) {
     fmt.line("#[derive(Clone, Debug)]");
     fmt.line("#[allow(missing_docs)]");
     fmt.line("pub enum InstructionData {");
@@ -93,7 +93,7 @@ fn gen_instruction_data(formats: &Vec<&InstructionFormat>, fmt: &mut Formatter)
     fmt.line("}");
 }
 
-fn gen_arguments_method(formats: &Vec<&InstructionFormat>, fmt: &mut Formatter, is_mut: bool) {
+fn gen_arguments_method(formats: &[&InstructionFormat], fmt: &mut Formatter, is_mut: bool) {
     let (method, mut_, rslice, as_slice) = if is_mut {
         (
             "arguments_mut",
@@ -116,7 +116,7 @@ fn gen_arguments_method(formats: &Vec<&InstructionFormat>, fmt: &mut Formatter,
     fmt.indent(|fmt| {
         let mut m = Match::new("*self");
         for format in formats {
-            let name = format!("InstructionData::{}", format.name);
+            let name = format!("Self::{}", format.name);
 
             // Formats with a value list put all of their arguments in the list. We don't split
             // them up, just return it all as variable arguments. (I expect the distinction to go
@@ -163,7 +163,7 @@ fn gen_arguments_method(formats: &Vec<&InstructionFormat>, fmt: &mut Formatter,
 /// - `pub fn put_value_list(&mut self, args: ir::ValueList>`
 /// - `pub fn eq(&self, &other: Self, &pool) -> bool`
 /// - `pub fn hash(&self, state: &mut H, &pool)`
-fn gen_instruction_data_impl(formats: &Vec<&InstructionFormat>, fmt: &mut Formatter) {
+fn gen_instruction_data_impl(formats: &[&InstructionFormat], fmt: &mut Formatter) {
     fmt.line("impl InstructionData {");
     fmt.indent(|fmt| {
         fmt.doc_comment("Get the opcode of this instruction.");
@@ -171,7 +171,7 @@ fn gen_instruction_data_impl(formats: &Vec<&InstructionFormat>, fmt: &mut Format
         fmt.indent(|fmt| {
             let mut m = Match::new("*self");
             for format in formats {
-                m.arm(format!("InstructionData::{}", format.name), vec!["opcode", ".."],
+                m.arm(format!("Self::{}", format.name), vec!["opcode", ".."],
                       "opcode".to_string());
             }
             fmt.add_match(m);
@@ -184,7 +184,7 @@ fn gen_instruction_data_impl(formats: &Vec<&InstructionFormat>, fmt: &mut Format
         fmt.indent(|fmt| {
             let mut m = Match::new("*self");
             for format in formats {
-                let name = format!("InstructionData::{}", format.name);
+                let name = format!("Self::{}", format.name);
                 if format.typevar_operand.is_none() {
                     m.arm(name, vec![".."], "None".to_string());
                 } else if format.has_value_list {
@@ -227,7 +227,7 @@ fn gen_instruction_data_impl(formats: &Vec<&InstructionFormat>, fmt: &mut Format
 
             for format in formats {
                 if format.has_value_list {
-                    m.arm(format!("InstructionData::{}", format.name),
+                    m.arm(format!("Self::{}", format.name),
                     vec!["ref mut args", ".."],
                     "Some(args.take())".to_string());
                 }
@@ -254,7 +254,7 @@ fn gen_instruction_data_impl(formats: &Vec<&InstructionFormat>, fmt: &mut Format
             fmt.indent(|fmt| {
                 for format in formats {
                     if format.has_value_list {
-                        fmtln!(fmt, "InstructionData::{} {{ ref mut args, .. }} => args,", format.name);
+                        fmtln!(fmt, "Self::{} {{ ref mut args, .. }} => args,", format.name);
                     }
                 }
                 fmt.line("_ => panic!(\"No value list: {:?}\", self),");
@@ -283,7 +283,7 @@ fn gen_instruction_data_impl(formats: &Vec<&InstructionFormat>, fmt: &mut Format
             fmt.line("match (self, other) {");
             fmt.indent(|fmt| {
                 for format in formats {
-                    let name = format!("&InstructionData::{}", format.name);
+                    let name = format!("&Self::{}", format.name);
                     let mut members = vec!["opcode"];
 
                     let args_eq = if format.typevar_operand.is_none() {
@@ -335,7 +335,7 @@ fn gen_instruction_data_impl(formats: &Vec<&InstructionFormat>, fmt: &mut Format
             fmt.line("match *self {");
             fmt.indent(|fmt| {
                 for format in formats {
-                    let name = format!("InstructionData::{}", format.name);
+                    let name = format!("Self::{}", format.name);
                     let mut members = vec!["opcode"];
 
                     let args = if format.typevar_operand.is_none() {
@@ -388,7 +388,7 @@ fn gen_bool_accessor bool>(
         let mut m = Match::new("self");
         for inst in all_inst.values() {
             if get_attr(inst) {
-                m.arm_no_fields(format!("Opcode::{}", inst.camel_name), "true");
+                m.arm_no_fields(format!("Self::{}", inst.camel_name), "true");
             }
         }
         m.arm_no_fields("_", "false");
@@ -398,7 +398,7 @@ fn gen_bool_accessor bool>(
     fmt.empty_line();
 }
 
-fn gen_opcodes<'a>(all_inst: &AllInstructions, fmt: &mut Formatter) {
+fn gen_opcodes(all_inst: &AllInstructions, fmt: &mut Formatter) {
     fmt.doc_comment(
         r#"
         An instruction opcode.
@@ -613,7 +613,7 @@ fn get_constraint<'entries, 'table>(
     }
 
     assert!(type_var == ctrl_typevar.unwrap());
-    return "Same".into();
+    "Same".into()
 }
 
 fn gen_bitset<'a, T: IntoIterator>(
@@ -641,22 +641,22 @@ fn iterable_to_string>(iterable: T) -
 
 fn typeset_to_string(ts: &TypeSet) -> String {
     let mut result = format!("TypeSet(lanes={}", iterable_to_string(&ts.lanes));
-    if ts.ints.len() > 0 {
+    if !ts.ints.is_empty() {
         result += &format!(", ints={}", iterable_to_string(&ts.ints));
     }
-    if ts.floats.len() > 0 {
+    if !ts.floats.is_empty() {
         result += &format!(", floats={}", iterable_to_string(&ts.floats));
     }
-    if ts.bools.len() > 0 {
+    if !ts.bools.is_empty() {
         result += &format!(", bools={}", iterable_to_string(&ts.bools));
     }
-    if ts.bitvecs.len() > 0 {
+    if !ts.bitvecs.is_empty() {
         result += &format!(", bitvecs={}", iterable_to_string(&ts.bitvecs));
     }
-    if ts.specials.len() > 0 {
+    if !ts.specials.is_empty() {
         result += &format!(", specials=[{}]", iterable_to_string(&ts.specials));
     }
-    if ts.refs.len() > 0 {
+    if !ts.refs.is_empty() {
         result += &format!(", refs={}", iterable_to_string(&ts.refs));
     }
     result += ")";
@@ -680,7 +680,7 @@ pub fn gen_typesets_table(type_sets: &UniqueTable, fmt: &mut Formatter)
         for ts in type_sets.iter() {
             fmt.line("ir::instructions::ValueTypeSet {");
             fmt.indent(|fmt| {
-                assert!(ts.bitvecs.len() == 0, "Bitvector types are not emittable.");
+                assert!(ts.bitvecs.is_empty(), "Bitvector types are not emittable.");
                 fmt.comment(typeset_to_string(ts));
                 gen_bitset(&ts.lanes, "lanes", 16, fmt);
                 gen_bitset(&ts.ints, "ints", 8, fmt);
@@ -710,6 +710,7 @@ fn gen_type_constraints(all_inst: &AllInstructions, fmt: &mut Formatter) {
     let mut operand_seqs = UniqueSeqTable::new();
 
     // Preload table with constraints for typical binops.
+    #[allow(clippy::useless_vec)]
     operand_seqs.add(&vec!["Same".to_string(); 3]);
 
     fmt.comment("Table of opcode constraints.");
@@ -926,7 +927,7 @@ fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Fo
         _ => format!("({})", vec!["Value"; inst.value_results.len()].join(", ")),
     };
 
-    let tmpl = if tmpl_types.len() > 0 {
+    let tmpl = if !tmpl_types.is_empty() {
         format!("<{}>", tmpl_types.join(", "))
     } else {
         "".into()
@@ -1005,7 +1006,7 @@ fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Fo
         // Call to the format constructor,
         let fcall = format!("self.{}({})", format.name, args.join(", "));
 
-        if inst.value_results.len() == 0 {
+        if inst.value_results.is_empty() {
             fmtln!(fmt, "{}.0", fcall);
             return;
         }
@@ -1037,7 +1038,7 @@ fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Fo
 /// Generate a Builder trait with methods for all instructions.
 fn gen_builder(
     instructions: &AllInstructions,
-    formats: &Vec<&InstructionFormat>,
+    formats: &[&InstructionFormat],
     fmt: &mut Formatter,
 ) {
     fmt.doc_comment(
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index c93262701d..6ff2016eaa 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -187,7 +187,7 @@ fn unwrap_inst(transform: &Transform, fmt: &mut Formatter) -> bool {
 
     // If the definition creates results, detach the values and place them in locals.
     let mut replace_inst = false;
-    if def.defined_vars.len() > 0 {
+    if !def.defined_vars.is_empty() {
         if def.defined_vars
             == def_pool
                 .get(var_pool.get(def.defined_vars[0]).dst_def.unwrap())
@@ -266,7 +266,7 @@ fn build_derived_expr(tv: &TypeVar) -> String {
 ///
 /// The emitted code is a statement redefining the `predicate` variable like this:
 ///     let predicate = predicate && ...
-fn emit_runtime_typecheck<'a, 'b>(
+fn emit_runtime_typecheck<'a>(
     constraint: &'a Constraint,
     type_sets: &mut UniqueTable<'a, TypeSet>,
     fmt: &mut Formatter,
@@ -586,7 +586,7 @@ fn gen_transform_group<'a>(
             let inst = &transform.def_pool.get(def_index).apply.inst;
             inst_to_transforms
                 .entry(inst.camel_name.clone())
-                .or_insert(Vec::new())
+                .or_insert_with(Vec::new)
                 .push(transform);
         }
 
@@ -606,7 +606,7 @@ fn gen_transform_group<'a>(
                         let replace_inst = unwrap_inst(&transforms[0], fmt);
                         fmt.empty_line();
 
-                        for (i, transform) in transforms.into_iter().enumerate() {
+                        for (i, transform) in transforms.iter().enumerate() {
                             if i > 0 {
                                 fmt.empty_line();
                             }
@@ -697,7 +697,7 @@ fn gen_isa(
 
 /// Generate the legalizer files.
 pub(crate) fn generate(
-    isas: &Vec,
+    isas: &[TargetIsa],
     transform_groups: &TransformGroups,
     filename_prefix: &str,
     out_dir: &str,
diff --git a/cranelift/codegen/meta/src/gen_registers.rs b/cranelift/codegen/meta/src/gen_registers.rs
index 8f778f8601..df34a20899 100644
--- a/cranelift/codegen/meta/src/gen_registers.rs
+++ b/cranelift/codegen/meta/src/gen_registers.rs
@@ -5,7 +5,7 @@ use crate::srcgen::Formatter;
 use cranelift_entity::EntityRef;
 
 fn gen_regbank(fmt: &mut Formatter, reg_bank: &RegBank) {
-    let names = if reg_bank.names.len() > 0 {
+    let names = if !reg_bank.names.is_empty() {
         format!(r#""{}""#, reg_bank.names.join(r#"", ""#))
     } else {
         "".to_string()
diff --git a/cranelift/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
index f17a7ea0eb..f8b5792ef1 100644
--- a/cranelift/codegen/meta/src/gen_settings.rs
+++ b/cranelift/codegen/meta/src/gen_settings.rs
@@ -81,7 +81,7 @@ fn gen_to_and_from_str(name: &str, values: &[&'static str], fmt: &mut Formatter)
             fmtln!(fmt, "f.write_str(match *self {");
             fmt.indent(|fmt| {
                 for v in values.iter() {
-                    fmtln!(fmt, "{}::{} => \"{}\",", name, camel_case(v), v);
+                    fmtln!(fmt, "Self::{} => \"{}\",", camel_case(v), v);
                 }
             });
             fmtln!(fmt, "})");
@@ -98,7 +98,7 @@ fn gen_to_and_from_str(name: &str, values: &[&'static str], fmt: &mut Formatter)
             fmtln!(fmt, "match s {");
             fmt.indent(|fmt| {
                 for v in values.iter() {
-                    fmtln!(fmt, "\"{}\" => Ok({}::{}),", v, name, camel_case(v));
+                    fmtln!(fmt, "\"{}\" => Ok(Self::{}),", v, camel_case(v));
                 }
                 fmtln!(fmt, "_ => Err(()),");
             });
@@ -198,7 +198,7 @@ fn gen_getters(group: &SettingGroup, fmt: &mut Formatter) {
         });
         fmtln!(fmt, "}");
 
-        if group.settings.len() > 0 {
+        if !group.settings.is_empty() {
             fmt.doc_comment("Dynamic numbered predicate getter.");
             fmtln!(fmt, "fn numbered_predicate(&self, p: usize) -> bool {");
             fmt.indent(|fmt| {
diff --git a/cranelift/codegen/meta/src/isa/mod.rs b/cranelift/codegen/meta/src/isa/mod.rs
index 3b90f7fe13..45b68e7c45 100644
--- a/cranelift/codegen/meta/src/isa/mod.rs
+++ b/cranelift/codegen/meta/src/isa/mod.rs
@@ -22,8 +22,7 @@ impl Isa {
         Isa::all()
             .iter()
             .cloned()
-            .filter(|isa| isa.to_string() == name)
-            .next()
+            .find(|isa| isa.to_string() == name)
     }
 
     /// Creates isa target from arch.
@@ -55,7 +54,7 @@ impl fmt::Display for Isa {
     }
 }
 
-pub(crate) fn define(isas: &Vec, shared_defs: &mut SharedDefinitions) -> Vec {
+pub(crate) fn define(isas: &[Isa], shared_defs: &mut SharedDefinitions) -> Vec {
     isas.iter()
         .map(|isa| match isa {
             Isa::Riscv => riscv::define(shared_defs),
diff --git a/cranelift/codegen/meta/src/isa/riscv/encodings.rs b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
index 936579b2bc..c255ddb483 100644
--- a/cranelift/codegen/meta/src/isa/riscv/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/encodings.rs
@@ -56,7 +56,7 @@ impl<'defs> PerCpuModeEncodings<'defs> {
 
 fn load_bits(funct3: u16) -> u16 {
     assert!(funct3 <= 0b111);
-    0b00000 | (funct3 << 5)
+    funct3 << 5
 }
 
 fn store_bits(funct3: u16) -> u16 {
@@ -91,13 +91,13 @@ fn opimm32_bits(funct3: u16, funct7: u16) -> u16 {
 
 fn op_bits(funct3: u16, funct7: u16) -> u16 {
     assert!(funct3 <= 0b111);
-    assert!(funct7 <= 0b1111111);
+    assert!(funct7 <= 0b111_1111);
     0b01100 | (funct3 << 5) | (funct7 << 8)
 }
 
 fn op32_bits(funct3: u16, funct7: u16) -> u16 {
     assert!(funct3 <= 0b111);
-    assert!(funct7 <= 0b1111111);
+    assert!(funct7 <= 0b111_1111);
     0b01110 | (funct3 << 5) | (funct7 << 8)
 }
 
@@ -177,11 +177,11 @@ pub(crate) fn define<'defs>(
 
     // Basic arithmetic binary instructions are encoded in an R-type instruction.
     for &(inst, inst_imm, f3, f7) in &[
-        (iadd, Some(iadd_imm), 0b000, 0b0000000),
-        (isub, None, 0b000, 0b0100000),
-        (bxor, Some(bxor_imm), 0b100, 0b0000000),
-        (bor, Some(bor_imm), 0b110, 0b0000000),
-        (band, Some(band_imm), 0b111, 0b0000000),
+        (iadd, Some(iadd_imm), 0b000, 0b000_0000),
+        (isub, None, 0b000, 0b010_0000),
+        (bxor, Some(bxor_imm), 0b100, 0b000_0000),
+        (bor, Some(bor_imm), 0b110, 0b000_0000),
+        (band, Some(band_imm), 0b111, 0b000_0000),
     ] {
         e.add32(e.enc(inst.bind(I32), r_r, op_bits(f3, f7)));
         e.add64(e.enc(inst.bind(I64), r_r, op_bits(f3, f7)));
@@ -194,8 +194,8 @@ pub(crate) fn define<'defs>(
     }
 
     // 32-bit ops in RV64.
-    e.add64(e.enc(iadd.bind(I32), r_r, op32_bits(0b000, 0b0000000)));
-    e.add64(e.enc(isub.bind(I32), r_r, op32_bits(0b000, 0b0100000)));
+    e.add64(e.enc(iadd.bind(I32), r_r, op32_bits(0b000, 0b000_0000)));
+    e.add64(e.enc(isub.bind(I32), r_r, op32_bits(0b000, 0b010_0000)));
     // There are no andiw/oriw/xoriw variations.
     e.add64(e.enc(iadd_imm.bind(I32), r_ii, opimm32_bits(0b000, 0)));
 
@@ -208,7 +208,7 @@ pub(crate) fn define<'defs>(
     for &(inst, inst_imm, f3, f7) in &[
         (ishl, ishl_imm, 0b1, 0b0),
         (ushr, ushr_imm, 0b101, 0b0),
-        (sshr, sshr_imm, 0b101, 0b100000),
+        (sshr, sshr_imm, 0b101, 0b10_0000),
     ] {
         e.add32(e.enc(inst.bind(I32).bind(I32), r_r, op_bits(f3, f7)));
         e.add64(e.enc(inst.bind(I64).bind(I64), r_r, op_bits(f3, f7)));
@@ -246,20 +246,20 @@ pub(crate) fn define<'defs>(
         let icmp_i32 = icmp.bind(I32);
         let icmp_i64 = icmp.bind(I64);
         e.add32(
-            e.enc(icmp_i32.clone(), r_ricmp, op_bits(0b010, 0b0000000))
+            e.enc(icmp_i32.clone(), r_ricmp, op_bits(0b010, 0b000_0000))
                 .inst_predicate(icmp_instp(&icmp_i32, "slt")),
         );
         e.add64(
-            e.enc(icmp_i64.clone(), r_ricmp, op_bits(0b010, 0b0000000))
+            e.enc(icmp_i64.clone(), r_ricmp, op_bits(0b010, 0b000_0000))
                 .inst_predicate(icmp_instp(&icmp_i64, "slt")),
         );
 
         e.add32(
-            e.enc(icmp_i32.clone(), r_ricmp, op_bits(0b011, 0b0000000))
+            e.enc(icmp_i32.clone(), r_ricmp, op_bits(0b011, 0b000_0000))
                 .inst_predicate(icmp_instp(&icmp_i32, "ult")),
         );
         e.add64(
-            e.enc(icmp_i64.clone(), r_ricmp, op_bits(0b011, 0b0000000))
+            e.enc(icmp_i64.clone(), r_ricmp, op_bits(0b011, 0b000_0000))
                 .inst_predicate(icmp_instp(&icmp_i64, "ult")),
         );
 
@@ -293,15 +293,15 @@ pub(crate) fn define<'defs>(
     // "M" Standard Extension for Integer Multiplication and Division.
     // Gated by the `use_m` flag.
     e.add32(
-        e.enc(imul.bind(I32), r_r, op_bits(0b000, 0b00000001))
+        e.enc(imul.bind(I32), r_r, op_bits(0b000, 0b0000_0001))
             .isa_predicate(use_m),
     );
     e.add64(
-        e.enc(imul.bind(I64), r_r, op_bits(0b000, 0b00000001))
+        e.enc(imul.bind(I64), r_r, op_bits(0b000, 0b0000_0001))
             .isa_predicate(use_m),
     );
     e.add64(
-        e.enc(imul.bind(I32), r_r, op32_bits(0b000, 0b00000001))
+        e.enc(imul.bind(I32), r_r, op32_bits(0b000, 0b0000_0001))
             .isa_predicate(use_m),
     );
 
diff --git a/cranelift/codegen/meta/src/isa/riscv/recipes.rs b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
index 9f68ed79bd..a75e42c236 100644
--- a/cranelift/codegen/meta/src/isa/riscv/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/riscv/recipes.rs
@@ -33,11 +33,10 @@ impl RecipeGroup {
     }
 
     pub fn by_name(&self, name: &str) -> EncodingRecipeNumber {
-        let number = *self
+        *self
             .name_to_recipe
             .get(name)
-            .expect(&format!("unknown riscv recipe name {}", name));
-        number
+            .unwrap_or_else(|| panic!("unknown riscv recipe name {}", name))
     }
 
     pub fn collect(self) -> Recipes {
@@ -97,7 +96,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
         EncodingRecipeBuilder::new("Iz", &formats.unary_imm, 4)
             .operands_out(vec![gpr])
             .inst_predicate(InstructionPredicate::new_is_signed_int(
-                &*&formats.unary_imm,
+                &formats.unary_imm,
                 "imm",
                 12,
                 0,
@@ -111,7 +110,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
             .operands_in(vec![gpr])
             .operands_out(vec![gpr])
             .inst_predicate(InstructionPredicate::new_is_signed_int(
-                &*&formats.int_compare_imm,
+                &formats.int_compare_imm,
                 "imm",
                 12,
                 0,
@@ -183,7 +182,7 @@ pub(crate) fn define(shared_defs: &SharedDefinitions, regs: &IsaRegs) -> RecipeG
         EncodingRecipeBuilder::new("U", &formats.unary_imm, 4)
             .operands_out(vec![gpr])
             .inst_predicate(InstructionPredicate::new_is_signed_int(
-                &*&formats.unary_imm,
+                &formats.unary_imm,
                 "imm",
                 32,
                 12,
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 3a8f8b696c..6e2755ba92 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -70,7 +70,7 @@ impl PerCpuModeEncodings {
     {
         let (recipe, bits) = template.build();
         let recipe_number = self.add_recipe(recipe);
-        let builder = EncodingBuilder::new(inst.into(), recipe_number, bits);
+        let builder = EncodingBuilder::new(inst, recipe_number, bits);
         builder_closure(builder).build(&self.recipes, &mut self.inst_pred_reg)
     }
 
@@ -367,6 +367,7 @@ impl PerCpuModeEncodings {
 
 // Definitions.
 
+#[allow(clippy::cognitive_complexity)]
 pub(crate) fn define(
     shared_defs: &SharedDefinitions,
     settings: &SettingGroup,
@@ -1916,7 +1917,7 @@ pub(crate) fn define(
 
     // SIMD integer addition
     for (ty, opcodes) in &[(I8, &PADDB), (I16, &PADDW), (I32, &PADDD), (I64, &PADDQ)] {
-        let iadd = iadd.bind(vector(ty.clone(), sse_vector_size));
+        let iadd = iadd.bind(vector(*ty, sse_vector_size));
         e.enc_32_64(iadd, rec_fa.opcodes(*opcodes));
     }
 
@@ -1940,7 +1941,7 @@ pub(crate) fn define(
 
     // SIMD integer subtraction
     for (ty, opcodes) in &[(I8, &PSUBB), (I16, &PSUBW), (I32, &PSUBD), (I64, &PSUBQ)] {
-        let isub = isub.bind(vector(ty.clone(), sse_vector_size));
+        let isub = isub.bind(vector(*ty, sse_vector_size));
         e.enc_32_64(isub, rec_fa.opcodes(*opcodes));
     }
 
@@ -1968,7 +1969,7 @@ pub(crate) fn define(
         (I16, &PMULLW[..], None),
         (I32, &PMULLD[..], Some(use_sse41_simd)),
     ] {
-        let imul = imul.bind(vector(ty.clone(), sse_vector_size));
+        let imul = imul.bind(vector(*ty, sse_vector_size));
         e.enc_32_64_maybe_isap(imul, rec_fa.opcodes(opcodes), *isap);
     }
 
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 366df755b9..2ed57e6e67 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -11,6 +11,7 @@ use crate::shared::formats::Formats;
 use crate::shared::immediates::Immediates;
 use crate::shared::types;
 
+#[allow(clippy::many_single_char_names)]
 pub(crate) fn define(
     mut all_instructions: &mut AllInstructions,
     formats: &Formats,
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 68ca67685f..e43e3c3e37 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -6,6 +6,7 @@ use crate::shared::types::Float::F64;
 use crate::shared::types::Int::{I16, I32, I64};
 use crate::shared::Definitions as SharedDefinitions;
 
+#[allow(clippy::many_single_char_names)]
 pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &InstructionGroup) {
     let mut group = TransformGroupBuilder::new(
         "x86_expand",
@@ -253,7 +254,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         def!(r = popcnt.I64(x)),
         vec![
             def!(qv3 = ushr_imm(x, imm64_1)),
-            def!(qc77 = iconst(Literal::constant(&imm.imm64, 0x7777777777777777))),
+            def!(qc77 = iconst(Literal::constant(&imm.imm64, 0x7777_7777_7777_7777))),
             def!(qv4 = band(qv3, qc77)),
             def!(qv5 = isub(x, qv4)),
             def!(qv6 = ushr_imm(qv4, imm64_1)),
@@ -264,9 +265,9 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
             def!(qv11 = isub(qv8, qv10)),
             def!(qv12 = ushr_imm(qv11, imm64_4)),
             def!(qv13 = iadd(qv11, qv12)),
-            def!(qc0F = iconst(Literal::constant(&imm.imm64, 0x0F0F0F0F0F0F0F0F))),
+            def!(qc0F = iconst(Literal::constant(&imm.imm64, 0x0F0F_0F0F_0F0F_0F0F))),
             def!(qv14 = band(qv13, qc0F)),
-            def!(qc01 = iconst(Literal::constant(&imm.imm64, 0x0101010101010101))),
+            def!(qc01 = iconst(Literal::constant(&imm.imm64, 0x0101_0101_0101_0101))),
             def!(qv15 = imul(qv14, qc01)),
             def!(r = ushr_imm(qv15, Literal::constant(&imm.imm64, 56))),
         ],
@@ -294,7 +295,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         def!(r = popcnt.I32(x)),
         vec![
             def!(lv3 = ushr_imm(x, imm64_1)),
-            def!(lc77 = iconst(Literal::constant(&imm.imm64, 0x77777777))),
+            def!(lc77 = iconst(Literal::constant(&imm.imm64, 0x7777_7777))),
             def!(lv4 = band(lv3, lc77)),
             def!(lv5 = isub(x, lv4)),
             def!(lv6 = ushr_imm(lv4, imm64_1)),
@@ -305,9 +306,9 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
             def!(lv11 = isub(lv8, lv10)),
             def!(lv12 = ushr_imm(lv11, imm64_4)),
             def!(lv13 = iadd(lv11, lv12)),
-            def!(lc0F = iconst(Literal::constant(&imm.imm64, 0x0F0F0F0F))),
+            def!(lc0F = iconst(Literal::constant(&imm.imm64, 0x0F0F_0F0F))),
             def!(lv14 = band(lv13, lc0F)),
-            def!(lc01 = iconst(Literal::constant(&imm.imm64, 0x01010101))),
+            def!(lc01 = iconst(Literal::constant(&imm.imm64, 0x0101_0101))),
             def!(lv15 = imul(lv14, lc01)),
             def!(r = ushr_imm(lv15, Literal::constant(&imm.imm64, 24))),
         ],
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index c1fd7a2c17..83d30a37c2 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -51,14 +51,14 @@ impl<'builder> RecipeGroup<'builder> {
     pub fn recipe(&self, name: &str) -> &EncodingRecipe {
         self.recipes
             .iter()
-            .find(|recipe| &recipe.name == name)
-            .expect(&format!("unknown recipe name: {}. Try template?", name))
+            .find(|recipe| recipe.name == name)
+            .unwrap_or_else(|| panic!("unknown recipe name: {}. Try template?", name))
     }
     pub fn template(&self, name: &str) -> &Template {
         self.templates
             .iter()
             .find(|recipe| recipe.name() == name)
-            .expect(&format!("unknown tail recipe name: {}. Try recipe?", name))
+            .unwrap_or_else(|| panic!("unknown tail recipe name: {}. Try recipe?", name))
     }
 }
 
@@ -96,7 +96,7 @@ impl<'builder> RecipeGroup<'builder> {
 
 /// Given a sequence of opcode bytes, compute the recipe name prefix and encoding bits.
 fn decode_opcodes(op_bytes: &[u8], rrr: u16, w: u16) -> (&'static str, u16) {
-    assert!(op_bytes.len() >= 1, "at least one opcode byte");
+    assert!(!op_bytes.is_empty(), "at least one opcode byte");
 
     let prefix_bytes = &op_bytes[..op_bytes.len() - 1];
     let (name, mmpp) = match prefix_bytes {
@@ -121,7 +121,7 @@ fn decode_opcodes(op_bytes: &[u8], rrr: u16, w: u16) -> (&'static str, u16) {
         }
     };
 
-    let opcode_byte = op_bytes[op_bytes.len() - 1] as u16;
+    let opcode_byte = u16::from(op_bytes[op_bytes.len() - 1]);
     (name, opcode_byte | (mmpp << 8) | (rrr << 12) | w << 15)
 }
 
@@ -243,7 +243,7 @@ impl<'builder> Template<'builder> {
         if let Some(prefixed) = &self.when_prefixed {
             let mut ret = prefixed.rex();
             // Forward specialized parameters.
-            ret.op_bytes = self.op_bytes.clone();
+            ret.op_bytes = self.op_bytes;
             ret.w_bit = self.w_bit;
             ret.rrr_bits = self.rrr_bits;
             return ret;
@@ -266,18 +266,17 @@ impl<'builder> Template<'builder> {
         self.recipe.base_size += size_addendum;
 
         // Branch ranges are relative to the end of the instruction.
-        self.recipe
-            .branch_range
-            .as_mut()
-            .map(|range| range.inst_size += size_addendum);
+        if let Some(range) = self.recipe.branch_range.as_mut() {
+            range.inst_size += size_addendum;
+        }
 
         self.recipe.emit = replace_put_op(self.recipe.emit, &name);
         self.recipe.name = name + &self.recipe.name;
 
         if !self.rex {
-            let operands_in = self.recipe.operands_in.unwrap_or(Vec::new());
+            let operands_in = self.recipe.operands_in.unwrap_or_default();
             self.recipe.operands_in = Some(replace_nonrex_constraints(self.regs, operands_in));
-            let operands_out = self.recipe.operands_out.unwrap_or(Vec::new());
+            let operands_out = self.recipe.operands_out.unwrap_or_default();
             self.recipe.operands_out = Some(replace_nonrex_constraints(self.regs, operands_out));
         }
 
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index 00fae4b455..56417002c7 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -22,7 +22,7 @@ pub fn isa_from_arch(arch: &str) -> Result {
 }
 
 /// Generates all the Rust source files used in Cranelift from the meta-language.
-pub fn generate(isas: &Vec, out_dir: &str) -> Result<(), error::Error> {
+pub fn generate(isas: &[isa::Isa], out_dir: &str) -> Result<(), error::Error> {
     // Create all the definitions:
     // - common definitions.
     let mut shared_defs = shared::define();
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 532fee6c87..967b71aa98 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -11,6 +11,7 @@ use crate::shared::formats::Formats;
 use crate::shared::types;
 use crate::shared::{entities::EntityRefs, immediates::Immediates};
 
+#[allow(clippy::many_single_char_names)]
 pub(crate) fn define(
     all_instructions: &mut AllInstructions,
     formats: &Formats,
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 2606dfb60d..05ffc99dfb 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -7,6 +7,7 @@ use crate::shared::types::Float::{F32, F64};
 use crate::shared::types::Int::{I128, I16, I32, I64, I8};
 use cranelift_codegen_shared::condcodes::{CondCode, IntCC};
 
+#[allow(clippy::many_single_char_names, clippy::cognitive_complexity)]
 pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGroups {
     let mut narrow = TransformGroupBuilder::new(
         "narrow",
@@ -766,24 +767,24 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     expand.legalize(
         def!(a = bitrev.I32(x)),
         vec![
-            def!(a1 = band_imm(x, Literal::constant(&imm.imm64, 0xaaaaaaaa))),
+            def!(a1 = band_imm(x, Literal::constant(&imm.imm64, 0xaaaa_aaaa))),
             def!(a2 = ushr_imm(a1, imm64_1)),
-            def!(a3 = band_imm(x, Literal::constant(&imm.imm64, 0x55555555))),
+            def!(a3 = band_imm(x, Literal::constant(&imm.imm64, 0x5555_5555))),
             def!(a4 = ishl_imm(a3, imm64_1)),
             def!(b = bor(a2, a4)),
-            def!(b1 = band_imm(b, Literal::constant(&imm.imm64, 0xcccccccc))),
+            def!(b1 = band_imm(b, Literal::constant(&imm.imm64, 0xcccc_cccc))),
             def!(b2 = ushr_imm(b1, imm64_2)),
-            def!(b3 = band_imm(b, Literal::constant(&imm.imm64, 0x33333333))),
+            def!(b3 = band_imm(b, Literal::constant(&imm.imm64, 0x3333_3333))),
             def!(b4 = ishl_imm(b3, imm64_2)),
             def!(c = bor(b2, b4)),
-            def!(c1 = band_imm(c, Literal::constant(&imm.imm64, 0xf0f0f0f0))),
+            def!(c1 = band_imm(c, Literal::constant(&imm.imm64, 0xf0f0_f0f0))),
             def!(c2 = ushr_imm(c1, imm64_4)),
-            def!(c3 = band_imm(c, Literal::constant(&imm.imm64, 0x0f0f0f0f))),
+            def!(c3 = band_imm(c, Literal::constant(&imm.imm64, 0x0f0f_0f0f))),
             def!(c4 = ishl_imm(c3, imm64_4)),
             def!(d = bor(c2, c4)),
-            def!(d1 = band_imm(d, Literal::constant(&imm.imm64, 0xff00ff00))),
+            def!(d1 = band_imm(d, Literal::constant(&imm.imm64, 0xff00_ff00))),
             def!(d2 = ushr_imm(d1, imm64_8)),
-            def!(d3 = band_imm(d, Literal::constant(&imm.imm64, 0x00ff00ff))),
+            def!(d3 = band_imm(d, Literal::constant(&imm.imm64, 0x00ff_00ff))),
             def!(d4 = ishl_imm(d3, imm64_8)),
             def!(e = bor(d2, d4)),
             def!(e1 = ushr_imm(e, imm64_16)),
@@ -793,20 +794,20 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     );
 
     #[allow(overflowing_literals)]
-    let imm64_0xaaaaaaaaaaaaaaaa = Literal::constant(&imm.imm64, 0xaaaaaaaaaaaaaaaa);
-    let imm64_0x5555555555555555 = Literal::constant(&imm.imm64, 0x5555555555555555);
+    let imm64_0xaaaaaaaaaaaaaaaa = Literal::constant(&imm.imm64, 0xaaaa_aaaa_aaaa_aaaa);
+    let imm64_0x5555555555555555 = Literal::constant(&imm.imm64, 0x5555_5555_5555_5555);
     #[allow(overflowing_literals)]
-    let imm64_0xcccccccccccccccc = Literal::constant(&imm.imm64, 0xcccccccccccccccc);
-    let imm64_0x3333333333333333 = Literal::constant(&imm.imm64, 0x3333333333333333);
+    let imm64_0xcccccccccccccccc = Literal::constant(&imm.imm64, 0xcccc_cccc_cccc_cccc);
+    let imm64_0x3333333333333333 = Literal::constant(&imm.imm64, 0x3333_3333_3333_3333);
     #[allow(overflowing_literals)]
-    let imm64_0xf0f0f0f0f0f0f0f0 = Literal::constant(&imm.imm64, 0xf0f0f0f0f0f0f0f0);
-    let imm64_0x0f0f0f0f0f0f0f0f = Literal::constant(&imm.imm64, 0x0f0f0f0f0f0f0f0f);
+    let imm64_0xf0f0f0f0f0f0f0f0 = Literal::constant(&imm.imm64, 0xf0f0_f0f0_f0f0_f0f0);
+    let imm64_0x0f0f0f0f0f0f0f0f = Literal::constant(&imm.imm64, 0x0f0f_0f0f_0f0f_0f0f);
     #[allow(overflowing_literals)]
-    let imm64_0xff00ff00ff00ff00 = Literal::constant(&imm.imm64, 0xff00ff00ff00ff00);
-    let imm64_0x00ff00ff00ff00ff = Literal::constant(&imm.imm64, 0x00ff00ff00ff00ff);
+    let imm64_0xff00ff00ff00ff00 = Literal::constant(&imm.imm64, 0xff00_ff00_ff00_ff00);
+    let imm64_0x00ff00ff00ff00ff = Literal::constant(&imm.imm64, 0x00ff_00ff_00ff_00ff);
     #[allow(overflowing_literals)]
-    let imm64_0xffff0000ffff0000 = Literal::constant(&imm.imm64, 0xffff0000ffff0000);
-    let imm64_0x0000ffff0000ffff = Literal::constant(&imm.imm64, 0x0000ffff0000ffff);
+    let imm64_0xffff0000ffff0000 = Literal::constant(&imm.imm64, 0xffff_0000_ffff_0000);
+    let imm64_0x0000ffff0000ffff = Literal::constant(&imm.imm64, 0x0000_ffff_0000_ffff);
     let imm64_32 = Literal::constant(&imm.imm64, 32);
 
     expand.legalize(
@@ -845,11 +846,11 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
 
     // Floating-point sign manipulations.
     for &(ty, const_inst, minus_zero) in &[
-        (F32, f32const, &Literal::bits(&imm.ieee32, 0x80000000)),
+        (F32, f32const, &Literal::bits(&imm.ieee32, 0x8000_0000)),
         (
             F64,
             f64const,
-            &Literal::bits(&imm.ieee64, 0x8000000000000000),
+            &Literal::bits(&imm.ieee64, 0x8000_0000_0000_0000),
         ),
     ] {
         expand.legalize(
diff --git a/cranelift/codegen/meta/src/srcgen.rs b/cranelift/codegen/meta/src/srcgen.rs
index 0a1d21c7b9..5a11d1d785 100644
--- a/cranelift/codegen/meta/src/srcgen.rs
+++ b/cranelift/codegen/meta/src/srcgen.rs
@@ -139,7 +139,7 @@ impl Formatter {
         parse_multiline(contents.as_ref())
             .iter()
             .map(|l| {
-                if l.len() == 0 {
+                if l.is_empty() {
                     "///".into()
                 } else {
                     format!("/// {}", l)
@@ -157,7 +157,7 @@ impl Formatter {
                 let conditions = names
                     .iter()
                     .map(|name| {
-                        if fields.len() > 0 {
+                        if !fields.is_empty() {
                             format!("{} {{ {} }}", name, fields.join(", "))
                         } else {
                             name.clone()
diff --git a/cranelift/codegen/meta/src/unique_table.rs b/cranelift/codegen/meta/src/unique_table.rs
index e50a64f437..7bc694fe26 100644
--- a/cranelift/codegen/meta/src/unique_table.rs
+++ b/cranelift/codegen/meta/src/unique_table.rs
@@ -48,8 +48,8 @@ impl UniqueSeqTable {
     pub fn new() -> Self {
         Self { table: Vec::new() }
     }
-    pub fn add(&mut self, values: &Vec) -> usize {
-        if values.len() == 0 {
+    pub fn add(&mut self, values: &[T]) -> usize {
+        if values.is_empty() {
             return 0;
         }
         if let Some(offset) = find_subsequence(values, &self.table) {
@@ -87,19 +87,19 @@ impl UniqueSeqTable {
 /// Try to find the subsequence `sub` in the `whole` sequence. Returns None if
 /// it's not been found, or Some(index) if it has been. Naive implementation
 /// until proven we need something better.
-fn find_subsequence(sub: &Vec, whole: &Vec) -> Option {
-    assert!(sub.len() > 0);
+fn find_subsequence(sub: &[T], whole: &[T]) -> Option {
+    assert!(!sub.is_empty());
     // We want i + sub.len() <= whole.len(), i.e. i < whole.len() + 1 - sub.len().
     if whole.len() < sub.len() {
         return None;
     }
     let max = whole.len() - sub.len();
-    for i in 0..max + 1 {
+    for i in 0..=max {
         if whole[i..i + sub.len()] == sub[..] {
             return Some(i);
         }
     }
-    return None;
+    None
 }
 
 #[test]
diff --git a/cranelift/codegen/shared/src/constant_hash.rs b/cranelift/codegen/shared/src/constant_hash.rs
index e16d58f5bc..ceac8e2722 100644
--- a/cranelift/codegen/shared/src/constant_hash.rs
+++ b/cranelift/codegen/shared/src/constant_hash.rs
@@ -24,6 +24,7 @@ pub fn simple_hash(s: &str) -> usize {
 /// Compute an open addressed, quadratically probed hash table containing
 /// `items`. The returned table is a list containing the elements of the
 /// iterable `items` and `None` in unused slots.
+#[allow(clippy::float_arithmetic)]
 pub fn generate_table<'cont, T, I: iter::Iterator, H: Fn(&T) -> usize>(
     items: I,
     num_items: usize,
diff --git a/cranelift/codegen/src/abi.rs b/cranelift/codegen/src/abi.rs
index e63f21b8d3..0dc571bba7 100644
--- a/cranelift/codegen/src/abi.rs
+++ b/cranelift/codegen/src/abi.rs
@@ -26,13 +26,13 @@ pub enum ArgAction {
 
 impl From for ArgAction {
     fn from(x: ArgumentLoc) -> Self {
-        ArgAction::Assign(x)
+        Self::Assign(x)
     }
 }
 
 impl From for ArgAction {
     fn from(x: ValueConversion) -> Self {
-        ArgAction::Convert(x)
+        Self::Convert(x)
     }
 }
 
@@ -59,17 +59,17 @@ impl ValueConversion {
     /// Apply this conversion to a type, return the converted type.
     pub fn apply(self, ty: Type) -> Type {
         match self {
-            ValueConversion::IntSplit => ty.half_width().expect("Integer type too small to split"),
-            ValueConversion::VectorSplit => ty.half_vector().expect("Not a vector"),
-            ValueConversion::IntBits => Type::int(ty.bits()).expect("Bad integer size"),
-            ValueConversion::Sext(nty) | ValueConversion::Uext(nty) => nty,
+            Self::IntSplit => ty.half_width().expect("Integer type too small to split"),
+            Self::VectorSplit => ty.half_vector().expect("Not a vector"),
+            Self::IntBits => Type::int(ty.bits()).expect("Bad integer size"),
+            Self::Sext(nty) | Self::Uext(nty) => nty,
         }
     }
 
     /// Is this a split conversion that results in two arguments?
     pub fn is_split(self) -> bool {
         match self {
-            ValueConversion::IntSplit | ValueConversion::VectorSplit => true,
+            Self::IntSplit | Self::VectorSplit => true,
             _ => false,
         }
     }
diff --git a/cranelift/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
index 0123e4dbd3..f7bdf01a37 100644
--- a/cranelift/codegen/src/binemit/mod.rs
+++ b/cranelift/codegen/src/binemit/mod.rs
@@ -63,14 +63,14 @@ impl fmt::Display for Reloc {
     /// already unambiguous, e.g. clif syntax with isa specified. In other contexts, use Debug.
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
-            Reloc::Abs4 => write!(f, "Abs4"),
-            Reloc::Abs8 => write!(f, "Abs8"),
-            Reloc::X86PCRel4 => write!(f, "PCRel4"),
-            Reloc::X86PCRelRodata4 => write!(f, "PCRelRodata4"),
-            Reloc::X86CallPCRel4 => write!(f, "CallPCRel4"),
-            Reloc::X86CallPLTRel4 => write!(f, "CallPLTRel4"),
-            Reloc::X86GOTPCRel4 => write!(f, "GOTPCRel4"),
-            Reloc::Arm32Call | Reloc::Arm64Call | Reloc::RiscvCall => write!(f, "Call"),
+            Self::Abs4 => write!(f, "Abs4"),
+            Self::Abs8 => write!(f, "Abs8"),
+            Self::X86PCRel4 => write!(f, "PCRel4"),
+            Self::X86PCRelRodata4 => write!(f, "PCRelRodata4"),
+            Self::X86CallPCRel4 => write!(f, "CallPCRel4"),
+            Self::X86CallPLTRel4 => write!(f, "CallPLTRel4"),
+            Self::X86GOTPCRel4 => write!(f, "GOTPCRel4"),
+            Self::Arm32Call | Self::Arm64Call | Self::RiscvCall => write!(f, "Call"),
         }
     }
 }
diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index 806129e9af..20e955cf99 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -230,7 +230,7 @@ fn try_fold_redundant_jump(
     let arguments_vec: alloc::vec::Vec<_> = first_args
         .iter()
         .chain(second_params.iter())
-        .map(|x| *x)
+        .copied()
         .collect();
     let value_list = ValueList::from_slice(&arguments_vec, &mut func.dfg.value_lists);
 
@@ -255,7 +255,7 @@ fn try_fold_redundant_jump(
         func.layout.remove_ebb(first_dest); // ...from the layout.
     }
 
-    return true;
+    true
 }
 
 /// Redirects `jump` instructions that point to other `jump` instructions to the final destination.
diff --git a/cranelift/codegen/src/binemit/stackmap.rs b/cranelift/codegen/src/binemit/stackmap.rs
index 4d513f7322..cb5739670f 100644
--- a/cranelift/codegen/src/binemit/stackmap.rs
+++ b/cranelift/codegen/src/binemit/stackmap.rs
@@ -55,7 +55,7 @@ impl Stackmap {
             }
         }
 
-        Stackmap::from_slice(&vec)
+        Self::from_slice(&vec)
     }
 
     /// Create a vec of Bitsets from a slice of bools.
diff --git a/cranelift/codegen/src/bitset.rs b/cranelift/codegen/src/bitset.rs
index ccbe413175..8035d80b96 100644
--- a/cranelift/codegen/src/bitset.rs
+++ b/cranelift/codegen/src/bitset.rs
@@ -74,7 +74,7 @@ where
 
         let lo_rng = (one << lo) - one;
 
-        BitSet(hi_rng - lo_rng)
+        Self(hi_rng - lo_rng)
     }
 }
 
diff --git a/cranelift/codegen/src/cursor.rs b/cranelift/codegen/src/cursor.rs
index e688a90b3c..9af12133d7 100644
--- a/cranelift/codegen/src/cursor.rs
+++ b/cranelift/codegen/src/cursor.rs
@@ -645,13 +645,14 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut FuncCursor<'f> {
                         let prev_op = self.data_flow_graph()[prev].opcode();
                         let inst_op = self.data_flow_graph()[inst].opcode();
                         let curr_op = self.data_flow_graph()[curr].opcode();
-                        if prev_op.is_branch() && !prev_op.is_terminator() {
-                            if !inst_op.is_terminator() {
-                                panic!(
-                                    "Inserting instruction {} after {}, and before {}",
-                                    inst_op, prev_op, curr_op
-                                )
-                            };
+                        if prev_op.is_branch()
+                            && !prev_op.is_terminator()
+                            && !inst_op.is_terminator()
+                        {
+                            panic!(
+                                "Inserting instruction {} after {}, and before {}",
+                                inst_op, prev_op, curr_op
+                            )
                         }
                     };
                 };
@@ -773,15 +774,16 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut EncCursor<'f> {
                     if let Some(prev) = self.layout().prev_inst(curr) {
                         let prev_op = self.data_flow_graph()[prev].opcode();
                         let inst_op = self.data_flow_graph()[inst].opcode();
-                        if prev_op.is_branch() && !prev_op.is_terminator() {
-                            if !inst_op.is_terminator() {
-                                panic!(
-                                    "Inserting instruction {} after {} and before {}",
-                                    self.display_inst(inst),
-                                    self.display_inst(prev),
-                                    self.display_inst(curr)
-                                )
-                            };
+                        if prev_op.is_branch()
+                            && !prev_op.is_terminator()
+                            && !inst_op.is_terminator()
+                        {
+                            panic!(
+                                "Inserting instruction {} after {} and before {}",
+                                self.display_inst(inst),
+                                self.display_inst(prev),
+                                self.display_inst(curr)
+                            )
                         }
                     };
                 };
diff --git a/cranelift/codegen/src/divconst_magic_numbers.rs b/cranelift/codegen/src/divconst_magic_numbers.rs
index 7b302e9b1a..af45444c40 100644
--- a/cranelift/codegen/src/divconst_magic_numbers.rs
+++ b/cranelift/codegen/src/divconst_magic_numbers.rs
@@ -79,7 +79,7 @@ pub fn magic_u32(d: u32) -> MU32 {
 
     MU32 {
         mul_by: q2 + 1,
-        do_add: do_add,
+        do_add,
         shift_by: p - 32,
     }
 }
@@ -125,7 +125,7 @@ pub fn magic_u64(d: u64) -> MU64 {
 
     MU64 {
         mul_by: q2 + 1,
-        do_add: do_add,
+        do_add,
         shift_by: p - 64,
     }
 }
diff --git a/cranelift/codegen/src/ir/constant.rs b/cranelift/codegen/src/ir/constant.rs
index 1bd9fc8b71..edd939f90d 100644
--- a/cranelift/codegen/src/ir/constant.rs
+++ b/cranelift/codegen/src/ir/constant.rs
@@ -29,7 +29,7 @@ pub struct ConstantData(Vec);
 impl FromIterator for ConstantData {
     fn from_iter>(iter: T) -> Self {
         let v = iter.into_iter().collect();
-        ConstantData(v)
+        Self(v)
     }
 }
 
@@ -58,7 +58,7 @@ impl ConstantData {
     }
 
     /// Convert the data to a vector.
-    pub fn to_vec(self) -> Vec {
+    pub fn into_vec(self) -> Vec {
         self.0
     }
 
@@ -122,7 +122,7 @@ impl FromStr for ConstantData {
     /// ```
     /// use cranelift_codegen::ir::ConstantData;
     /// let c: ConstantData = "0x000102".parse().unwrap();
-    /// assert_eq!(c.to_vec(), [2, 1, 0]);
+    /// assert_eq!(c.into_vec(), [2, 1, 0]);
     /// ```
     fn from_str(s: &str) -> Result {
         if s.len() <= 2 || &s[0..2] != "0x" {
@@ -137,7 +137,7 @@ impl FromStr for ConstantData {
             .cloned()
             .collect(); // remove 0x prefix and any intervening _ characters
 
-        if cleaned.len() == 0 {
+        if cleaned.is_empty() {
             Err("Hexadecimal string must have some digits")
         } else if cleaned.len() % 2 != 0 {
             Err("Hexadecimal string must have an even number of digits")
@@ -152,7 +152,7 @@ impl FromStr for ConstantData {
                     .or_else(|_| Err("Unable to parse as hexadecimal"))?;
                 buffer.insert(0, byte);
             }
-            Ok(ConstantData(buffer))
+            Ok(Self(buffer))
         }
     }
 }
@@ -221,15 +221,13 @@ impl ConstantPool {
     /// returned.
     pub fn insert(&mut self, constant_value: ConstantData) -> Constant {
         if self.values_to_handles.contains_key(&constant_value) {
-            self.values_to_handles.get(&constant_value).unwrap().clone()
+            *self.values_to_handles.get(&constant_value).unwrap()
         } else {
             let constant_handle = Constant::new(self.len());
             self.values_to_handles
-                .insert(constant_value.clone(), constant_handle.clone());
-            self.handles_to_values.insert(
-                constant_handle.clone(),
-                ConstantPoolEntry::new(constant_value),
-            );
+                .insert(constant_value.clone(), constant_handle);
+            self.handles_to_values
+                .insert(constant_handle, ConstantPoolEntry::new(constant_value));
             constant_handle
         }
     }
@@ -402,13 +400,13 @@ mod tests {
     fn add_to_constant_data() {
         let d = ConstantData::from([1, 2].as_ref());
         let e = d.append(i16::from(3u8));
-        assert_eq!(e.to_vec(), vec![1, 2, 3, 0])
+        assert_eq!(e.into_vec(), vec![1, 2, 3, 0])
     }
 
     #[test]
     fn extend_constant_data() {
         let d = ConstantData::from([1, 2].as_ref());
-        assert_eq!(d.expand_to(4).to_vec(), vec![1, 2, 0, 0])
+        assert_eq!(d.expand_to(4).into_vec(), vec![1, 2, 0, 0])
     }
 
     #[test]
@@ -460,7 +458,7 @@ mod tests {
 
     #[test]
     fn verify_stored_bytes_in_constant_data() {
-        assert_eq!("0x01".parse::().unwrap().to_vec(), [1]);
+        assert_eq!("0x01".parse::().unwrap().into_vec(), [1]);
         assert_eq!(ConstantData::from([1, 0].as_ref()).0, [1, 0]);
         assert_eq!(ConstantData::from(vec![1, 0, 0, 0]).0, [1, 0, 0, 0]);
     }
@@ -468,7 +466,10 @@ mod tests {
     #[test]
     fn check_constant_data_endianness_as_uimm128() {
         fn parse_to_uimm128(from: &str) -> Vec {
-            from.parse::().unwrap().expand_to(16).to_vec()
+            from.parse::()
+                .unwrap()
+                .expand_to(16)
+                .into_vec()
         }
 
         assert_eq!(
diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index b41f035822..831ffca575 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -378,7 +378,7 @@ impl ValueDef {
     /// Unwrap the instruction where the value was defined, or panic.
     pub fn unwrap_inst(&self) -> Inst {
         match *self {
-            ValueDef::Result(inst, _) => inst,
+            Self::Result(inst, _) => inst,
             _ => panic!("Value is not an instruction result"),
         }
     }
@@ -386,7 +386,7 @@ impl ValueDef {
     /// Unwrap the EBB there the parameter is defined, or panic.
     pub fn unwrap_ebb(&self) -> Ebb {
         match *self {
-            ValueDef::Param(ebb, _) => ebb,
+            Self::Param(ebb, _) => ebb,
             _ => panic!("Value is not an EBB parameter"),
         }
     }
@@ -402,7 +402,7 @@ impl ValueDef {
     /// this value.
     pub fn num(self) -> usize {
         match self {
-            ValueDef::Result(_, n) | ValueDef::Param(_, n) => n,
+            Self::Result(_, n) | Self::Param(_, n) => n,
         }
     }
 }
diff --git a/cranelift/codegen/src/ir/entities.rs b/cranelift/codegen/src/ir/entities.rs
index 1883fea258..02fa9cbc08 100644
--- a/cranelift/codegen/src/ir/entities.rs
+++ b/cranelift/codegen/src/ir/entities.rs
@@ -41,7 +41,7 @@ impl Ebb {
     /// This method is for use by the parser.
     pub fn with_number(n: u32) -> Option {
         if n < u32::MAX {
-            Some(Ebb(n))
+            Some(Self(n))
         } else {
             None
         }
@@ -72,7 +72,7 @@ impl Value {
     /// This method is for use by the parser.
     pub fn with_number(n: u32) -> Option {
         if n < u32::MAX / 2 {
-            Some(Value(n))
+            Some(Self(n))
         } else {
             None
         }
@@ -118,7 +118,7 @@ impl StackSlot {
     /// This method is for use by the parser.
     pub fn with_number(n: u32) -> Option {
         if n < u32::MAX {
-            Some(StackSlot(n))
+            Some(Self(n))
         } else {
             None
         }
@@ -152,7 +152,7 @@ impl GlobalValue {
     /// This method is for use by the parser.
     pub fn with_number(n: u32) -> Option {
         if n < u32::MAX {
-            Some(GlobalValue(n))
+            Some(Self(n))
         } else {
             None
         }
@@ -174,7 +174,7 @@ impl Constant {
     /// This method is for use by the parser.
     pub fn with_number(n: u32) -> Option {
         if n < u32::MAX {
-            Some(Constant(n))
+            Some(Self(n))
         } else {
             None
         }
@@ -197,7 +197,7 @@ impl Immediate {
     /// This method is for use by the parser.
     pub fn with_number(n: u32) -> Option {
         if n < u32::MAX {
-            Some(Immediate(n))
+            Some(Self(n))
         } else {
             None
         }
@@ -225,7 +225,7 @@ impl JumpTable {
     /// This method is for use by the parser.
     pub fn with_number(n: u32) -> Option {
         if n < u32::MAX {
-            Some(JumpTable(n))
+            Some(Self(n))
         } else {
             None
         }
@@ -258,7 +258,7 @@ impl FuncRef {
     /// This method is for use by the parser.
     pub fn with_number(n: u32) -> Option {
         if n < u32::MAX {
-            Some(FuncRef(n))
+            Some(Self(n))
         } else {
             None
         }
@@ -287,7 +287,7 @@ impl SigRef {
     /// This method is for use by the parser.
     pub fn with_number(n: u32) -> Option {
         if n < u32::MAX {
-            Some(SigRef(n))
+            Some(Self(n))
         } else {
             None
         }
@@ -310,7 +310,7 @@ impl Heap {
     /// This method is for use by the parser.
     pub fn with_number(n: u32) -> Option {
         if n < u32::MAX {
-            Some(Heap(n))
+            Some(Self(n))
         } else {
             None
         }
@@ -334,7 +334,7 @@ impl Table {
     /// This method is for use by the parser.
     pub fn with_number(n: u32) -> Option {
         if n < u32::MAX {
-            Some(Table(n))
+            Some(Self(n))
         } else {
             None
         }
@@ -371,17 +371,17 @@ pub enum AnyEntity {
 impl fmt::Display for AnyEntity {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
-            AnyEntity::Function => write!(f, "function"),
-            AnyEntity::Ebb(r) => r.fmt(f),
-            AnyEntity::Inst(r) => r.fmt(f),
-            AnyEntity::Value(r) => r.fmt(f),
-            AnyEntity::StackSlot(r) => r.fmt(f),
-            AnyEntity::GlobalValue(r) => r.fmt(f),
-            AnyEntity::JumpTable(r) => r.fmt(f),
-            AnyEntity::FuncRef(r) => r.fmt(f),
-            AnyEntity::SigRef(r) => r.fmt(f),
-            AnyEntity::Heap(r) => r.fmt(f),
-            AnyEntity::Table(r) => r.fmt(f),
+            Self::Function => write!(f, "function"),
+            Self::Ebb(r) => r.fmt(f),
+            Self::Inst(r) => r.fmt(f),
+            Self::Value(r) => r.fmt(f),
+            Self::StackSlot(r) => r.fmt(f),
+            Self::GlobalValue(r) => r.fmt(f),
+            Self::JumpTable(r) => r.fmt(f),
+            Self::FuncRef(r) => r.fmt(f),
+            Self::SigRef(r) => r.fmt(f),
+            Self::Heap(r) => r.fmt(f),
+            Self::Table(r) => r.fmt(f),
         }
     }
 }
@@ -394,61 +394,61 @@ impl fmt::Debug for AnyEntity {
 
 impl From for AnyEntity {
     fn from(r: Ebb) -> Self {
-        AnyEntity::Ebb(r)
+        Self::Ebb(r)
     }
 }
 
 impl From for AnyEntity {
     fn from(r: Inst) -> Self {
-        AnyEntity::Inst(r)
+        Self::Inst(r)
     }
 }
 
 impl From for AnyEntity {
     fn from(r: Value) -> Self {
-        AnyEntity::Value(r)
+        Self::Value(r)
     }
 }
 
 impl From for AnyEntity {
     fn from(r: StackSlot) -> Self {
-        AnyEntity::StackSlot(r)
+        Self::StackSlot(r)
     }
 }
 
 impl From for AnyEntity {
     fn from(r: GlobalValue) -> Self {
-        AnyEntity::GlobalValue(r)
+        Self::GlobalValue(r)
     }
 }
 
 impl From for AnyEntity {
     fn from(r: JumpTable) -> Self {
-        AnyEntity::JumpTable(r)
+        Self::JumpTable(r)
     }
 }
 
 impl From for AnyEntity {
     fn from(r: FuncRef) -> Self {
-        AnyEntity::FuncRef(r)
+        Self::FuncRef(r)
     }
 }
 
 impl From for AnyEntity {
     fn from(r: SigRef) -> Self {
-        AnyEntity::SigRef(r)
+        Self::SigRef(r)
     }
 }
 
 impl From for AnyEntity {
     fn from(r: Heap) -> Self {
-        AnyEntity::Heap(r)
+        Self::Heap(r)
     }
 }
 
 impl From
 for AnyEntity {
     fn from(r: Table) -> Self {
-        AnyEntity::Table(r)
+        Self::Table(r)
     }
 }
 
diff --git a/cranelift/codegen/src/ir/extfunc.rs b/cranelift/codegen/src/ir/extfunc.rs
index 112fb31b7b..76b249da4a 100644
--- a/cranelift/codegen/src/ir/extfunc.rs
+++ b/cranelift/codegen/src/ir/extfunc.rs
@@ -294,14 +294,14 @@ impl FromStr for ArgumentPurpose {
     type Err = ();
     fn from_str(s: &str) -> Result {
         match s {
-            "normal" => Ok(ArgumentPurpose::Normal),
-            "sret" => Ok(ArgumentPurpose::StructReturn),
-            "link" => Ok(ArgumentPurpose::Link),
-            "fp" => Ok(ArgumentPurpose::FramePointer),
-            "csr" => Ok(ArgumentPurpose::CalleeSaved),
-            "vmctx" => Ok(ArgumentPurpose::VMContext),
-            "sigid" => Ok(ArgumentPurpose::SignatureId),
-            "stack_limit" => Ok(ArgumentPurpose::StackLimit),
+            "normal" => Ok(Self::Normal),
+            "sret" => Ok(Self::StructReturn),
+            "link" => Ok(Self::Link),
+            "fp" => Ok(Self::FramePointer),
+            "csr" => Ok(Self::CalleeSaved),
+            "vmctx" => Ok(Self::VMContext),
+            "sigid" => Ok(Self::SignatureId),
+            "stack_limit" => Ok(Self::StackLimit),
             _ => Err(()),
         }
     }
diff --git a/cranelift/codegen/src/ir/extname.rs b/cranelift/codegen/src/ir/extname.rs
index 3a47699acc..c0a9865373 100644
--- a/cranelift/codegen/src/ir/extname.rs
+++ b/cranelift/codegen/src/ir/extname.rs
@@ -62,7 +62,7 @@ impl ExternalName {
         let mut bytes = [0u8; TESTCASE_NAME_LENGTH];
         bytes[0..len].copy_from_slice(&vec[0..len]);
 
-        ExternalName::TestCase {
+        Self::TestCase {
             length: len as u8,
             ascii: bytes,
         }
@@ -78,7 +78,7 @@ impl ExternalName {
     /// assert_eq!(name.to_string(), "u123:456");
     /// ```
     pub fn user(namespace: u32, index: u32) -> Self {
-        ExternalName::User { namespace, index }
+        Self::User { namespace, index }
     }
 }
 
@@ -91,15 +91,15 @@ impl Default for ExternalName {
 impl fmt::Display for ExternalName {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
-            ExternalName::User { namespace, index } => write!(f, "u{}:{}", namespace, index),
-            ExternalName::TestCase { length, ascii } => {
+            Self::User { namespace, index } => write!(f, "u{}:{}", namespace, index),
+            Self::TestCase { length, ascii } => {
                 f.write_char('%')?;
                 for byte in ascii.iter().take(length as usize) {
                     f.write_char(*byte as char)?;
                 }
                 Ok(())
             }
-            ExternalName::LibCall(lc) => write!(f, "%{}", lc),
+            Self::LibCall(lc) => write!(f, "%{}", lc),
         }
     }
 }
@@ -110,7 +110,7 @@ impl FromStr for ExternalName {
     fn from_str(s: &str) -> Result {
         // Try to parse as a libcall name, otherwise it's a test case.
         match s.parse() {
-            Ok(lc) => Ok(ExternalName::LibCall(lc)),
+            Ok(lc) => Ok(Self::LibCall(lc)),
             Err(_) => Ok(Self::testcase(s.as_bytes())),
         }
     }
diff --git a/cranelift/codegen/src/ir/globalvalue.rs b/cranelift/codegen/src/ir/globalvalue.rs
index 6c7b62aab9..dbba5aa676 100644
--- a/cranelift/codegen/src/ir/globalvalue.rs
+++ b/cranelift/codegen/src/ir/globalvalue.rs
@@ -70,7 +70,7 @@ impl GlobalValueData {
     /// Assume that `self` is an `GlobalValueData::Symbol` and return its name.
     pub fn symbol_name(&self) -> &ExternalName {
         match *self {
-            GlobalValueData::Symbol { ref name, .. } => name,
+            Self::Symbol { ref name, .. } => name,
             _ => panic!("only symbols have names"),
         }
     }
@@ -78,11 +78,8 @@ impl GlobalValueData {
     /// Return the type of this global.
     pub fn global_type(&self, isa: &dyn TargetIsa) -> Type {
         match *self {
-            GlobalValueData::VMContext { .. } | GlobalValueData::Symbol { .. } => {
-                isa.pointer_type()
-            }
-            GlobalValueData::IAddImm { global_type, .. }
-            | GlobalValueData::Load { global_type, .. } => global_type,
+            Self::VMContext { .. } | Self::Symbol { .. } => isa.pointer_type(),
+            Self::IAddImm { global_type, .. } | Self::Load { global_type, .. } => global_type,
         }
     }
 }
@@ -90,8 +87,8 @@ impl GlobalValueData {
 impl fmt::Display for GlobalValueData {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
-            GlobalValueData::VMContext => write!(f, "vmctx"),
-            GlobalValueData::Load {
+            Self::VMContext => write!(f, "vmctx"),
+            Self::Load {
                 base,
                 offset,
                 global_type,
@@ -104,12 +101,12 @@ impl fmt::Display for GlobalValueData {
                 base,
                 offset
             ),
-            GlobalValueData::IAddImm {
+            Self::IAddImm {
                 global_type,
                 base,
                 offset,
             } => write!(f, "iadd_imm.{} {}, {}", global_type, base, offset),
-            GlobalValueData::Symbol {
+            Self::Symbol {
                 ref name,
                 offset,
                 colocated,
diff --git a/cranelift/codegen/src/ir/immediates.rs b/cranelift/codegen/src/ir/immediates.rs
index 8222f612a4..b1d142bd9e 100644
--- a/cranelift/codegen/src/ir/immediates.rs
+++ b/cranelift/codegen/src/ir/immediates.rs
@@ -50,12 +50,12 @@ pub struct Imm64(i64);
 impl Imm64 {
     /// Create a new `Imm64` representing the signed number `x`.
     pub fn new(x: i64) -> Self {
-        Imm64(x)
+        Self(x)
     }
 
     /// Return self negated.
     pub fn wrapping_neg(self) -> Self {
-        Imm64(self.0.wrapping_neg())
+        Self(self.0.wrapping_neg())
     }
 }
 
@@ -73,7 +73,7 @@ impl IntoBytes for Imm64 {
 
 impl From for Imm64 {
     fn from(x: i64) -> Self {
-        Imm64(x)
+        Self(x)
     }
 }
 
@@ -130,12 +130,12 @@ pub struct Uimm64(u64);
 impl Uimm64 {
     /// Create a new `Uimm64` representing the unsigned number `x`.
     pub fn new(x: u64) -> Self {
-        Uimm64(x)
+        Self(x)
     }
 
     /// Return self negated.
     pub fn wrapping_neg(self) -> Self {
-        Uimm64(self.0.wrapping_neg())
+        Self(self.0.wrapping_neg())
     }
 }
 
@@ -147,7 +147,7 @@ impl Into for Uimm64 {
 
 impl From for Uimm64 {
     fn from(x: u64) -> Self {
-        Uimm64(x)
+        Self(x)
     }
 }
 
@@ -272,7 +272,7 @@ impl Into for Uimm32 {
 
 impl From for Uimm32 {
     fn from(x: u32) -> Self {
-        Uimm32(x)
+        Self(x)
     }
 }
 
@@ -293,7 +293,7 @@ impl FromStr for Uimm32 {
     fn from_str(s: &str) -> Result {
         parse_i64(s).and_then(|x| {
             if 0 <= x && x <= i64::from(u32::MAX) {
-                Ok(Uimm32(x as u32))
+                Ok(Self(x as u32))
             } else {
                 Err("Uimm32 out of range")
             }
@@ -329,7 +329,7 @@ impl From<&[u8]> for V128Imm {
         assert_eq!(slice.len(), 16);
         let mut buffer = [0; 16];
         buffer.copy_from_slice(slice);
-        V128Imm(buffer)
+        Self(buffer)
     }
 }
 
@@ -343,7 +343,7 @@ pub struct Offset32(i32);
 impl Offset32 {
     /// Create a new `Offset32` representing the signed number `x`.
     pub fn new(x: i32) -> Self {
-        Offset32(x)
+        Self(x)
     }
 
     /// Create a new `Offset32` representing the signed number `x` if possible.
@@ -381,7 +381,7 @@ impl Into for Offset32 {
 
 impl From for Offset32 {
     fn from(x: i32) -> Self {
-        Offset32(x)
+        Self(x)
     }
 }
 
@@ -676,7 +676,7 @@ fn parse_float(s: &str, w: u8, t: u8) -> Result {
 impl Ieee32 {
     /// Create a new `Ieee32` containing the bits of `x`.
     pub fn with_bits(x: u32) -> Self {
-        Ieee32(x)
+        Self(x)
     }
 
     /// Create an `Ieee32` number representing `2.0^n`.
@@ -688,7 +688,7 @@ impl Ieee32 {
         let exponent = (n + bias) as u32;
         assert!(exponent > 0, "Underflow n={}", n);
         assert!(exponent < (1 << w) + 1, "Overflow n={}", n);
-        Ieee32(exponent << t)
+        Self(exponent << t)
     }
 
     /// Create an `Ieee32` number representing the greatest negative value
@@ -702,12 +702,12 @@ impl Ieee32 {
 
     /// Return self negated.
     pub fn neg(self) -> Self {
-        Ieee32(self.0 ^ (1 << 31))
+        Self(self.0 ^ (1 << 31))
     }
 
     /// Create a new `Ieee32` representing the number `x`.
     pub fn with_float(x: f32) -> Self {
-        Ieee32(x.to_bits())
+        Self(x.to_bits())
     }
 
     /// Get the bitwise representation.
@@ -728,7 +728,7 @@ impl FromStr for Ieee32 {
 
     fn from_str(s: &str) -> Result {
         match parse_float(s, 8, 23) {
-            Ok(b) => Ok(Ieee32(b as u32)),
+            Ok(b) => Ok(Self(b as u32)),
             Err(s) => Err(s),
         }
     }
@@ -736,7 +736,7 @@ impl FromStr for Ieee32 {
 
 impl From for Ieee32 {
     fn from(x: f32) -> Self {
-        Ieee32::with_float(x)
+        Self::with_float(x)
     }
 }
 
@@ -749,7 +749,7 @@ impl IntoBytes for Ieee32 {
 impl Ieee64 {
     /// Create a new `Ieee64` containing the bits of `x`.
     pub fn with_bits(x: u64) -> Self {
-        Ieee64(x)
+        Self(x)
     }
 
     /// Create an `Ieee64` number representing `2.0^n`.
@@ -761,7 +761,7 @@ impl Ieee64 {
         let exponent = (n + bias) as u64;
         assert!(exponent > 0, "Underflow n={}", n);
         assert!(exponent < (1 << w) + 1, "Overflow n={}", n);
-        Ieee64(exponent << t)
+        Self(exponent << t)
     }
 
     /// Create an `Ieee64` number representing the greatest negative value
@@ -775,12 +775,12 @@ impl Ieee64 {
 
     /// Return self negated.
     pub fn neg(self) -> Self {
-        Ieee64(self.0 ^ (1 << 63))
+        Self(self.0 ^ (1 << 63))
     }
 
     /// Create a new `Ieee64` representing the number `x`.
     pub fn with_float(x: f64) -> Self {
-        Ieee64(x.to_bits())
+        Self(x.to_bits())
     }
 
     /// Get the bitwise representation.
@@ -801,7 +801,7 @@ impl FromStr for Ieee64 {
 
     fn from_str(s: &str) -> Result {
         match parse_float(s, 11, 52) {
-            Ok(b) => Ok(Ieee64(b)),
+            Ok(b) => Ok(Self(b)),
             Err(s) => Err(s),
         }
     }
@@ -809,13 +809,13 @@ impl FromStr for Ieee64 {
 
 impl From for Ieee64 {
     fn from(x: f64) -> Self {
-        Ieee64::with_float(x)
+        Self::with_float(x)
     }
 }
 
 impl From for Ieee64 {
     fn from(x: u64) -> Self {
-        Ieee64::with_float(f64::from_bits(x))
+        Self::with_float(f64::from_bits(x))
     }
 }
 
diff --git a/cranelift/codegen/src/ir/instructions.rs b/cranelift/codegen/src/ir/instructions.rs
index 7eb504483f..c194505deb 100644
--- a/cranelift/codegen/src/ir/instructions.rs
+++ b/cranelift/codegen/src/ir/instructions.rs
@@ -101,7 +101,7 @@ pub struct VariableArgs(Vec);
 impl VariableArgs {
     /// Create an empty argument list.
     pub fn new() -> Self {
-        VariableArgs(Vec::new())
+        Self(Vec::new())
     }
 
     /// Add an argument to the end.
@@ -168,35 +168,35 @@ impl InstructionData {
     /// here.
     pub fn analyze_branch<'a>(&'a self, pool: &'a ValueListPool) -> BranchInfo<'a> {
         match *self {
-            InstructionData::Jump {
+            Self::Jump {
                 destination,
                 ref args,
                 ..
             } => BranchInfo::SingleDest(destination, args.as_slice(pool)),
-            InstructionData::BranchInt {
+            Self::BranchInt {
                 destination,
                 ref args,
                 ..
             }
-            | InstructionData::BranchFloat {
+            | Self::BranchFloat {
                 destination,
                 ref args,
                 ..
             }
-            | InstructionData::Branch {
+            | Self::Branch {
                 destination,
                 ref args,
                 ..
             } => BranchInfo::SingleDest(destination, &args.as_slice(pool)[1..]),
-            InstructionData::BranchIcmp {
+            Self::BranchIcmp {
                 destination,
                 ref args,
                 ..
             } => BranchInfo::SingleDest(destination, &args.as_slice(pool)[2..]),
-            InstructionData::BranchTable {
+            Self::BranchTable {
                 table, destination, ..
             } => BranchInfo::Table(table, Some(destination)),
-            InstructionData::IndirectJump { table, .. } => BranchInfo::Table(table, None),
+            Self::IndirectJump { table, .. } => BranchInfo::Table(table, None),
             _ => {
                 debug_assert!(!self.opcode().is_branch());
                 BranchInfo::NotABranch
@@ -210,12 +210,12 @@ impl InstructionData {
     /// Multi-destination branches like `br_table` return `None`.
     pub fn branch_destination(&self) -> Option {
         match *self {
-            InstructionData::Jump { destination, .. }
-            | InstructionData::Branch { destination, .. }
-            | InstructionData::BranchInt { destination, .. }
-            | InstructionData::BranchFloat { destination, .. }
-            | InstructionData::BranchIcmp { destination, .. } => Some(destination),
-            InstructionData::BranchTable { .. } | InstructionData::IndirectJump { .. } => None,
+            Self::Jump { destination, .. }
+            | Self::Branch { destination, .. }
+            | Self::BranchInt { destination, .. }
+            | Self::BranchFloat { destination, .. }
+            | Self::BranchIcmp { destination, .. } => Some(destination),
+            Self::BranchTable { .. } | Self::IndirectJump { .. } => None,
             _ => {
                 debug_assert!(!self.opcode().is_branch());
                 None
@@ -229,27 +229,27 @@ impl InstructionData {
     /// Multi-destination branches like `br_table` return `None`.
     pub fn branch_destination_mut(&mut self) -> Option<&mut Ebb> {
         match *self {
-            InstructionData::Jump {
+            Self::Jump {
                 ref mut destination,
                 ..
             }
-            | InstructionData::Branch {
+            | Self::Branch {
                 ref mut destination,
                 ..
             }
-            | InstructionData::BranchInt {
+            | Self::BranchInt {
                 ref mut destination,
                 ..
             }
-            | InstructionData::BranchFloat {
+            | Self::BranchFloat {
                 ref mut destination,
                 ..
             }
-            | InstructionData::BranchIcmp {
+            | Self::BranchIcmp {
                 ref mut destination,
                 ..
             } => Some(destination),
-            InstructionData::BranchTable { .. } => None,
+            Self::BranchTable { .. } => None,
             _ => {
                 debug_assert!(!self.opcode().is_branch());
                 None
@@ -262,10 +262,10 @@ impl InstructionData {
     /// Any instruction that can call another function reveals its call signature here.
     pub fn analyze_call<'a>(&'a self, pool: &'a ValueListPool) -> CallInfo<'a> {
         match *self {
-            InstructionData::Call {
+            Self::Call {
                 func_ref, ref args, ..
             } => CallInfo::Direct(func_ref, args.as_slice(pool)),
-            InstructionData::CallIndirect {
+            Self::CallIndirect {
                 sig_ref, ref args, ..
             } => CallInfo::Indirect(sig_ref, &args.as_slice(pool)[1..]),
             _ => {
diff --git a/cranelift/codegen/src/ir/libcall.rs b/cranelift/codegen/src/ir/libcall.rs
index 52c4c40dad..88752bb3ec 100644
--- a/cranelift/codegen/src/ir/libcall.rs
+++ b/cranelift/codegen/src/ir/libcall.rs
@@ -59,18 +59,18 @@ impl FromStr for LibCall {
 
     fn from_str(s: &str) -> Result {
         match s {
-            "Probestack" => Ok(LibCall::Probestack),
-            "CeilF32" => Ok(LibCall::CeilF32),
-            "CeilF64" => Ok(LibCall::CeilF64),
-            "FloorF32" => Ok(LibCall::FloorF32),
-            "FloorF64" => Ok(LibCall::FloorF64),
-            "TruncF32" => Ok(LibCall::TruncF32),
-            "TruncF64" => Ok(LibCall::TruncF64),
-            "NearestF32" => Ok(LibCall::NearestF32),
-            "NearestF64" => Ok(LibCall::NearestF64),
-            "Memcpy" => Ok(LibCall::Memcpy),
-            "Memset" => Ok(LibCall::Memset),
-            "Memmove" => Ok(LibCall::Memmove),
+            "Probestack" => Ok(Self::Probestack),
+            "CeilF32" => Ok(Self::CeilF32),
+            "CeilF64" => Ok(Self::CeilF64),
+            "FloorF32" => Ok(Self::FloorF32),
+            "FloorF64" => Ok(Self::FloorF64),
+            "TruncF32" => Ok(Self::TruncF32),
+            "TruncF64" => Ok(Self::TruncF64),
+            "NearestF32" => Ok(Self::NearestF32),
+            "NearestF64" => Ok(Self::NearestF64),
+            "Memcpy" => Ok(Self::Memcpy),
+            "Memset" => Ok(Self::Memset),
+            "Memmove" => Ok(Self::Memmove),
             _ => Err(()),
         }
     }
@@ -84,17 +84,17 @@ impl LibCall {
     pub fn for_inst(opcode: Opcode, ctrl_type: Type) -> Option {
         Some(match ctrl_type {
             types::F32 => match opcode {
-                Opcode::Ceil => LibCall::CeilF32,
-                Opcode::Floor => LibCall::FloorF32,
-                Opcode::Trunc => LibCall::TruncF32,
-                Opcode::Nearest => LibCall::NearestF32,
+                Opcode::Ceil => Self::CeilF32,
+                Opcode::Floor => Self::FloorF32,
+                Opcode::Trunc => Self::TruncF32,
+                Opcode::Nearest => Self::NearestF32,
                 _ => return None,
             },
             types::F64 => match opcode {
-                Opcode::Ceil => LibCall::CeilF64,
-                Opcode::Floor => LibCall::FloorF64,
-                Opcode::Trunc => LibCall::TruncF64,
-                Opcode::Nearest => LibCall::NearestF64,
+                Opcode::Ceil => Self::CeilF64,
+                Opcode::Floor => Self::FloorF64,
+                Opcode::Trunc => Self::TruncF64,
+                Opcode::Nearest => Self::NearestF64,
                 _ => return None,
             },
             _ => return None,
diff --git a/cranelift/codegen/src/ir/progpoint.rs b/cranelift/codegen/src/ir/progpoint.rs
index 32c349aa00..4bfa2c39e7 100644
--- a/cranelift/codegen/src/ir/progpoint.rs
+++ b/cranelift/codegen/src/ir/progpoint.rs
@@ -20,7 +20,7 @@ impl From for ProgramPoint {
     fn from(inst: Inst) -> Self {
         let idx = inst.index();
         debug_assert!(idx < (u32::MAX / 2) as usize);
-        ProgramPoint((idx * 2) as u32)
+        Self((idx * 2) as u32)
     }
 }
 
@@ -28,7 +28,7 @@ impl From for ProgramPoint {
     fn from(ebb: Ebb) -> Self {
         let idx = ebb.index();
         debug_assert!(idx < (u32::MAX / 2) as usize);
-        ProgramPoint((idx * 2 + 1) as u32)
+        Self((idx * 2 + 1) as u32)
     }
 }
 
@@ -55,21 +55,21 @@ impl ExpandedProgramPoint {
     /// Get the instruction we know is inside.
     pub fn unwrap_inst(self) -> Inst {
         match self {
-            ExpandedProgramPoint::Inst(x) => x,
-            ExpandedProgramPoint::Ebb(x) => panic!("expected inst: {}", x),
+            Self::Inst(x) => x,
+            Self::Ebb(x) => panic!("expected inst: {}", x),
         }
     }
 }
 
 impl From for ExpandedProgramPoint {
     fn from(inst: Inst) -> Self {
-        ExpandedProgramPoint::Inst(inst)
+        Self::Inst(inst)
     }
 }
 
 impl From for ExpandedProgramPoint {
     fn from(ebb: Ebb) -> Self {
-        ExpandedProgramPoint::Ebb(ebb)
+        Self::Ebb(ebb)
     }
 }
 
@@ -85,9 +85,9 @@ impl From for ExpandedProgramPoint {
 impl From for ExpandedProgramPoint {
     fn from(pp: ProgramPoint) -> Self {
         if pp.0 & 1 == 0 {
-            ExpandedProgramPoint::Inst(Inst::from_u32(pp.0 / 2))
+            Self::Inst(Inst::from_u32(pp.0 / 2))
         } else {
-            ExpandedProgramPoint::Ebb(Ebb::from_u32(pp.0 / 2))
+            Self::Ebb(Ebb::from_u32(pp.0 / 2))
         }
     }
 }
@@ -95,8 +95,8 @@ impl From for ExpandedProgramPoint {
 impl fmt::Display for ExpandedProgramPoint {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
-            ExpandedProgramPoint::Inst(x) => write!(f, "{}", x),
-            ExpandedProgramPoint::Ebb(x) => write!(f, "{}", x),
+            Self::Inst(x) => write!(f, "{}", x),
+            Self::Ebb(x) => write!(f, "{}", x),
         }
     }
 }
diff --git a/cranelift/codegen/src/ir/sourceloc.rs b/cranelift/codegen/src/ir/sourceloc.rs
index b8f162366f..ccab62f89b 100644
--- a/cranelift/codegen/src/ir/sourceloc.rs
+++ b/cranelift/codegen/src/ir/sourceloc.rs
@@ -21,7 +21,7 @@ pub struct SourceLoc(u32);
 impl SourceLoc {
     /// Create a new source location with the given bits.
     pub fn new(bits: u32) -> Self {
-        SourceLoc(bits)
+        Self(bits)
     }
 
     /// Is this the default source location?
@@ -37,7 +37,7 @@ impl SourceLoc {
 
 impl Default for SourceLoc {
     fn default() -> Self {
-        SourceLoc(!0)
+        Self(!0)
     }
 }
 
diff --git a/cranelift/codegen/src/ir/types.rs b/cranelift/codegen/src/ir/types.rs
index 4bdec45268..1baca567e1 100644
--- a/cranelift/codegen/src/ir/types.rs
+++ b/cranelift/codegen/src/ir/types.rs
@@ -91,7 +91,7 @@ impl Type {
     /// Get a type with the same number of lanes as `self`, but using `lane` as the lane type.
     fn replace_lanes(self, lane: Self) -> Self {
         debug_assert!(lane.is_lane() && !self.is_special());
-        Type((lane.0 & 0x0f) | (self.0 & 0xf0))
+        Self((lane.0 & 0x0f) | (self.0 & 0xf0))
     }
 
     /// Get a type with the same number of lanes as this type, but with the lanes replaced by
@@ -262,7 +262,7 @@ impl Type {
         let log2_lanes: u32 = n.trailing_zeros();
         let new_type = u32::from(self.0) + (log2_lanes << 4);
         if new_type < 0x100 {
-            Some(Type(new_type as u8))
+            Some(Self(new_type as u8))
         } else {
             None
         }
@@ -273,7 +273,7 @@ impl Type {
     /// There is no `double_vector()` method. Use `t.by(2)` instead.
     pub fn half_vector(self) -> Option {
         if self.is_vector() {
-            Some(Type(self.0 - 0x10))
+            Some(Self(self.0 - 0x10))
         } else {
             None
         }
diff --git a/cranelift/codegen/src/ir/valueloc.rs b/cranelift/codegen/src/ir/valueloc.rs
index f3ba44896e..9d81a55381 100644
--- a/cranelift/codegen/src/ir/valueloc.rs
+++ b/cranelift/codegen/src/ir/valueloc.rs
@@ -24,7 +24,7 @@ pub enum ValueLoc {
 
 impl Default for ValueLoc {
     fn default() -> Self {
-        ValueLoc::Unassigned
+        Self::Unassigned
     }
 }
 
@@ -32,7 +32,7 @@ impl ValueLoc {
     /// Is this an assigned location? (That is, not `Unassigned`).
     pub fn is_assigned(self) -> bool {
         match self {
-            ValueLoc::Unassigned => false,
+            Self::Unassigned => false,
             _ => true,
         }
     }
@@ -40,7 +40,7 @@ impl ValueLoc {
     /// Get the register unit of this location, or panic.
     pub fn unwrap_reg(self) -> RegUnit {
         match self {
-            ValueLoc::Reg(ru) => ru,
+            Self::Reg(ru) => ru,
             _ => panic!("Expected register: {:?}", self),
         }
     }
@@ -48,7 +48,7 @@ impl ValueLoc {
     /// Get the stack slot of this location, or panic.
     pub fn unwrap_stack(self) -> StackSlot {
         match self {
-            ValueLoc::Stack(ss) => ss,
+            Self::Stack(ss) => ss,
             _ => panic!("Expected stack slot: {:?}", self),
         }
     }
@@ -109,7 +109,7 @@ pub enum ArgumentLoc {
 
 impl Default for ArgumentLoc {
     fn default() -> Self {
-        ArgumentLoc::Unassigned
+        Self::Unassigned
     }
 }
 
@@ -117,7 +117,7 @@ impl ArgumentLoc {
     /// Is this an assigned location? (That is, not `Unassigned`).
     pub fn is_assigned(self) -> bool {
         match self {
-            ArgumentLoc::Unassigned => false,
+            Self::Unassigned => false,
             _ => true,
         }
     }
@@ -125,7 +125,7 @@ impl ArgumentLoc {
     /// Is this a register location?
     pub fn is_reg(self) -> bool {
         match self {
-            ArgumentLoc::Reg(_) => true,
+            Self::Reg(_) => true,
             _ => false,
         }
     }
@@ -133,7 +133,7 @@ impl ArgumentLoc {
     /// Is this a stack location?
     pub fn is_stack(self) -> bool {
         match self {
-            ArgumentLoc::Stack(_) => true,
+            Self::Stack(_) => true,
             _ => false,
         }
     }
diff --git a/cranelift/codegen/src/isa/call_conv.rs b/cranelift/codegen/src/isa/call_conv.rs
index e834ad2515..2b3b2a5f35 100644
--- a/cranelift/codegen/src/isa/call_conv.rs
+++ b/cranelift/codegen/src/isa/call_conv.rs
@@ -29,8 +29,8 @@ impl CallConv {
         match triple.default_calling_convention() {
             // Default to System V for unknown targets because most everything
             // uses System V.
-            Ok(CallingConvention::SystemV) | Err(()) => CallConv::SystemV,
-            Ok(CallingConvention::WindowsFastcall) => CallConv::WindowsFastcall,
+            Ok(CallingConvention::SystemV) | Err(()) => Self::SystemV,
+            Ok(CallingConvention::WindowsFastcall) => Self::WindowsFastcall,
             Ok(unimp) => unimplemented!("calling convention: {:?}", unimp),
         }
     }
@@ -39,28 +39,28 @@ impl CallConv {
     pub fn for_libcall(isa: &dyn TargetIsa) -> Self {
         match isa.flags().libcall_call_conv() {
             LibcallCallConv::IsaDefault => isa.default_call_conv(),
-            LibcallCallConv::Fast => CallConv::Fast,
-            LibcallCallConv::Cold => CallConv::Cold,
-            LibcallCallConv::SystemV => CallConv::SystemV,
-            LibcallCallConv::WindowsFastcall => CallConv::WindowsFastcall,
-            LibcallCallConv::BaldrdashSystemV => CallConv::BaldrdashSystemV,
-            LibcallCallConv::BaldrdashWindows => CallConv::BaldrdashWindows,
-            LibcallCallConv::Probestack => CallConv::Probestack,
+            LibcallCallConv::Fast => Self::Fast,
+            LibcallCallConv::Cold => Self::Cold,
+            LibcallCallConv::SystemV => Self::SystemV,
+            LibcallCallConv::WindowsFastcall => Self::WindowsFastcall,
+            LibcallCallConv::BaldrdashSystemV => Self::BaldrdashSystemV,
+            LibcallCallConv::BaldrdashWindows => Self::BaldrdashWindows,
+            LibcallCallConv::Probestack => Self::Probestack,
         }
     }
 
     /// Is the calling convention extending the Windows Fastcall ABI?
-    pub fn extends_windows_fastcall(&self) -> bool {
+    pub fn extends_windows_fastcall(self) -> bool {
         match self {
-            CallConv::WindowsFastcall | CallConv::BaldrdashWindows => true,
+            Self::WindowsFastcall | Self::BaldrdashWindows => true,
             _ => false,
         }
     }
 
     /// Is the calling convention extending the Baldrdash ABI?
-    pub fn extends_baldrdash(&self) -> bool {
+    pub fn extends_baldrdash(self) -> bool {
         match self {
-            CallConv::BaldrdashSystemV | CallConv::BaldrdashWindows => true,
+            Self::BaldrdashSystemV | Self::BaldrdashWindows => true,
             _ => false,
         }
     }
@@ -69,13 +69,13 @@ impl CallConv {
 impl fmt::Display for CallConv {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         f.write_str(match *self {
-            CallConv::Fast => "fast",
-            CallConv::Cold => "cold",
-            CallConv::SystemV => "system_v",
-            CallConv::WindowsFastcall => "windows_fastcall",
-            CallConv::BaldrdashSystemV => "baldrdash_system_v",
-            CallConv::BaldrdashWindows => "baldrdash_windows",
-            CallConv::Probestack => "probestack",
+            Self::Fast => "fast",
+            Self::Cold => "cold",
+            Self::SystemV => "system_v",
+            Self::WindowsFastcall => "windows_fastcall",
+            Self::BaldrdashSystemV => "baldrdash_system_v",
+            Self::BaldrdashWindows => "baldrdash_windows",
+            Self::Probestack => "probestack",
         })
     }
 }
@@ -84,13 +84,13 @@ impl str::FromStr for CallConv {
     type Err = ();
     fn from_str(s: &str) -> Result {
         match s {
-            "fast" => Ok(CallConv::Fast),
-            "cold" => Ok(CallConv::Cold),
-            "system_v" => Ok(CallConv::SystemV),
-            "windows_fastcall" => Ok(CallConv::WindowsFastcall),
-            "baldrdash_system_v" => Ok(CallConv::BaldrdashSystemV),
-            "baldrdash_windows" => Ok(CallConv::BaldrdashWindows),
-            "probestack" => Ok(CallConv::Probestack),
+            "fast" => Ok(Self::Fast),
+            "cold" => Ok(Self::Cold),
+            "system_v" => Ok(Self::SystemV),
+            "windows_fastcall" => Ok(Self::WindowsFastcall),
+            "baldrdash_system_v" => Ok(Self::BaldrdashSystemV),
+            "baldrdash_windows" => Ok(Self::BaldrdashWindows),
+            "probestack" => Ok(Self::Probestack),
             _ => Err(()),
         }
     }
diff --git a/cranelift/codegen/src/isa/registers.rs b/cranelift/codegen/src/isa/registers.rs
index d3544904e0..6bb9e9cf4c 100644
--- a/cranelift/codegen/src/isa/registers.rs
+++ b/cranelift/codegen/src/isa/registers.rs
@@ -237,7 +237,7 @@ pub struct RegClassIndex(u8);
 
 impl EntityRef for RegClassIndex {
     fn new(idx: usize) -> Self {
-        RegClassIndex(idx as u8)
+        Self(idx as u8)
     }
 
     fn index(self) -> usize {
@@ -247,7 +247,7 @@ impl EntityRef for RegClassIndex {
 
 impl From for RegClassIndex {
     fn from(rc: RegClass) -> Self {
-        RegClassIndex(rc.index)
+        Self(rc.index)
     }
 }
 
diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 77c67fb2ae..0d6a6c0bb9 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -389,14 +389,17 @@ fn expand_br_table_conds(
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
 
+    // Ignore the lint for this loop as the range needs to be 0 to table_size
+    #[allow(clippy::needless_range_loop)]
     for i in 0..table_size {
         let dest = pos.func.jump_tables[table].as_slice()[i];
         let t = pos.ins().icmp_imm(IntCC::Equal, arg, i as i64);
         pos.ins().brnz(t, dest, &[]);
         // Jump to the next case.
         if i < table_size - 1 {
-            pos.ins().jump(cond_failed_ebb[i], &[]);
-            pos.insert_ebb(cond_failed_ebb[i]);
+            let ebb = cond_failed_ebb[i];
+            pos.ins().jump(ebb, &[]);
+            pos.insert_ebb(ebb);
         }
     }
 
@@ -699,7 +702,7 @@ fn narrow_icmp_imm(
 
     let imm_low = pos
         .ins()
-        .iconst(ty_half, imm & (1u128 << ty_half.bits() - 1) as i64);
+        .iconst(ty_half, imm & (1u128 << (ty_half.bits() - 1)) as i64);
     let imm_high = pos
         .ins()
         .iconst(ty_half, imm.wrapping_shr(ty_half.bits().into()));
diff --git a/cranelift/codegen/src/lib.rs b/cranelift/codegen/src/lib.rs
index 4b6cfc33ca..772562b916 100644
--- a/cranelift/codegen/src/lib.rs
+++ b/cranelift/codegen/src/lib.rs
@@ -18,13 +18,12 @@
                 clippy::assign_op_pattern,
                 clippy::empty_line_after_outer_attr,
 // Hard to avoid in generated code:
-                clippy::cyclomatic_complexity,
+                clippy::cognitive_complexity,
                 clippy::too_many_arguments,
 // Code generator doesn't have a way to collapse identical arms:
                 clippy::match_same_arms,
 // These are relatively minor style issues, but would be easy to fix:
                 clippy::new_without_default,
-                clippy::new_without_default_derive,
                 clippy::should_implement_trait,
                 clippy::len_without_is_empty))]
 #![cfg_attr(
@@ -35,7 +34,6 @@
         clippy::nonminimal_bool,
         clippy::option_map_unwrap_or,
         clippy::option_map_unwrap_or_else,
-        clippy::print_allocout,
         clippy::unicode_not_nfc,
         clippy::use_self
     )
diff --git a/cranelift/codegen/src/licm.rs b/cranelift/codegen/src/licm.rs
index 284d50c5ea..35fd2c0a80 100644
--- a/cranelift/codegen/src/licm.rs
+++ b/cranelift/codegen/src/licm.rs
@@ -71,7 +71,7 @@ fn create_pre_header(
     domtree: &DominatorTree,
 ) -> Ebb {
     let pool = &mut ListPool::::new();
-    let header_args_values: Vec = func.dfg.ebb_params(header).into_iter().cloned().collect();
+    let header_args_values = func.dfg.ebb_params(header).to_vec();
     let header_args_types: Vec = header_args_values
         .clone()
         .into_iter()
diff --git a/cranelift/codegen/src/redundant_reload_remover.rs b/cranelift/codegen/src/redundant_reload_remover.rs
index b1928f03b8..207cec2286 100644
--- a/cranelift/codegen/src/redundant_reload_remover.rs
+++ b/cranelift/codegen/src/redundant_reload_remover.rs
@@ -348,7 +348,7 @@ impl RedundantReloadRemover {
 impl AvailEnv {
     // Create a new one.
     fn new(size: usize) -> Self {
-        let mut env = AvailEnv {
+        let mut env = Self {
             map: Vec::>::new(),
         };
         env.map.resize(size, None);
@@ -494,9 +494,9 @@ impl RedundantReloadRemover {
         debug_assert!(!self.processing_stack.is_empty());
         let ProcessingStackElem {
             avail_env,
-            cursor: _,
             diversions,
-        } = &mut self.processing_stack.last_mut().unwrap();
+            ..
+        } = self.processing_stack.last_mut().unwrap();
 
         #[cfg(debug_assertions)]
         debug_assert!(
@@ -588,12 +588,7 @@ impl RedundantReloadRemover {
                     invalidate_regs_written_by_inst(locations, diversions, dfg, avail_env, inst);
                 }
             }
-            InstructionData::RegMove {
-                opcode: _,
-                arg: _,
-                src,
-                dst,
-            } => {
+            InstructionData::RegMove { src, dst, .. } => {
                 // These happen relatively rarely, but just frequently enough that it's worth
                 // tracking the copy (at the machine level, it's really a copy) in `avail_env`.
                 avail_env.copy_reg(*src, *dst);
@@ -817,9 +812,9 @@ impl RedundantReloadRemover {
         for i in 0..num_ebbs {
             let mut pi = cfg.pred_iter(Ebb::from_u32(i));
             let mut n_pi = ZeroOneOrMany::Zero;
-            if let Some(_) = pi.next() {
+            if pi.next().is_some() {
                 n_pi = ZeroOneOrMany::One;
-                if let Some(_) = pi.next() {
+                if pi.next().is_some() {
                     n_pi = ZeroOneOrMany::Many;
                     // We don't care if there are more than two preds, so stop counting now.
                 }
@@ -886,7 +881,7 @@ impl RedundantReloadRemover {
         let ctx = Context {
             cur: EncCursor::new(func, isa),
             reginfo: isa.register_info(),
-            cfg: cfg,
+            cfg,
             state: self,
         };
         let mut total_regunits = 0;
diff --git a/cranelift/codegen/src/regalloc/affinity.rs b/cranelift/codegen/src/regalloc/affinity.rs
index 7eea98ab8a..b7fa3090ca 100644
--- a/cranelift/codegen/src/regalloc/affinity.rs
+++ b/cranelift/codegen/src/regalloc/affinity.rs
@@ -30,7 +30,7 @@ pub enum Affinity {
 
 impl Default for Affinity {
     fn default() -> Self {
-        Affinity::Unassigned
+        Self::Unassigned
     }
 }
 
@@ -41,25 +41,25 @@ impl Affinity {
     /// Use the `Default` implementation for that.
     pub fn new(constraint: &OperandConstraint) -> Self {
         if constraint.kind == ConstraintKind::Stack {
-            Affinity::Stack
+            Self::Stack
         } else {
-            Affinity::Reg(constraint.regclass.into())
+            Self::Reg(constraint.regclass.into())
         }
     }
 
     /// Create an affinity that matches an ABI argument for `isa`.
     pub fn abi(arg: &AbiParam, isa: &dyn TargetIsa) -> Self {
         match arg.location {
-            ArgumentLoc::Unassigned => Affinity::Unassigned,
-            ArgumentLoc::Reg(_) => Affinity::Reg(isa.regclass_for_abi_type(arg.value_type).into()),
-            ArgumentLoc::Stack(_) => Affinity::Stack,
+            ArgumentLoc::Unassigned => Self::Unassigned,
+            ArgumentLoc::Reg(_) => Self::Reg(isa.regclass_for_abi_type(arg.value_type).into()),
+            ArgumentLoc::Stack(_) => Self::Stack,
         }
     }
 
     /// Is this the `Unassigned` affinity?
     pub fn is_unassigned(self) -> bool {
         match self {
-            Affinity::Unassigned => true,
+            Self::Unassigned => true,
             _ => false,
         }
     }
@@ -67,7 +67,7 @@ impl Affinity {
     /// Is this the `Reg` affinity?
     pub fn is_reg(self) -> bool {
         match self {
-            Affinity::Reg(_) => true,
+            Self::Reg(_) => true,
             _ => false,
         }
     }
@@ -75,7 +75,7 @@ impl Affinity {
     /// Is this the `Stack` affinity?
     pub fn is_stack(self) -> bool {
         match self {
-            Affinity::Stack => true,
+            Self::Stack => true,
             _ => false,
         }
     }
@@ -86,8 +86,8 @@ impl Affinity {
     /// satisfies the constraint.
     pub fn merge(&mut self, constraint: &OperandConstraint, reginfo: &RegInfo) {
         match *self {
-            Affinity::Unassigned => *self = Self::new(constraint),
-            Affinity::Reg(rc) => {
+            Self::Unassigned => *self = Self::new(constraint),
+            Self::Reg(rc) => {
                 // If the preferred register class is a subclass of the constraint, there's no need
                 // to change anything.
                 if constraint.kind != ConstraintKind::Stack && !constraint.regclass.has_subclass(rc)
@@ -96,11 +96,11 @@ impl Affinity {
                     // we just keep our previous affinity.
                     if let Some(subclass) = constraint.regclass.intersect_index(reginfo.rc(rc)) {
                         // This constraint shrinks our preferred register class.
-                        *self = Affinity::Reg(subclass);
+                        *self = Self::Reg(subclass);
                     }
                 }
             }
-            Affinity::Stack => {}
+            Self::Stack => {}
         }
     }
 
diff --git a/cranelift/codegen/src/regalloc/branch_splitting.rs b/cranelift/codegen/src/regalloc/branch_splitting.rs
index 3a9e6c87bc..39901a5588 100644
--- a/cranelift/codegen/src/regalloc/branch_splitting.rs
+++ b/cranelift/codegen/src/regalloc/branch_splitting.rs
@@ -96,7 +96,7 @@ impl<'a> Context<'a> {
             let dfg = &mut self.cur.func.dfg;
             let old_args: Vec<_> = {
                 let args = dfg[branch].take_value_list().expect("ebb parameters");
-                args.as_slice(&dfg.value_lists).iter().map(|x| *x).collect()
+                args.as_slice(&dfg.value_lists).iter().copied().collect()
             };
             let (branch_args, ebb_params) = old_args.split_at(num_fixed);
 
@@ -159,16 +159,13 @@ impl<'a> Context<'a> {
     /// Returns whether we should introduce a new branch.
     fn should_split_edge(&self, target: Ebb) -> bool {
         // We should split the edge if the target has any parameters.
-        if self.cur.func.dfg.ebb_params(target).len() > 0 {
+        if !self.cur.func.dfg.ebb_params(target).is_empty() {
             return true;
         };
 
         // Or, if the target has more than one block reaching it.
         debug_assert!(self.cfg.pred_iter(target).next() != None);
-        if let Some(_) = self.cfg.pred_iter(target).skip(1).next() {
-            return true;
-        };
 
-        false
+        self.cfg.pred_iter(target).nth(1).is_some()
     }
 }
diff --git a/cranelift/codegen/src/regalloc/coloring.rs b/cranelift/codegen/src/regalloc/coloring.rs
index c6c48bf78c..cef27214ed 100644
--- a/cranelift/codegen/src/regalloc/coloring.rs
+++ b/cranelift/codegen/src/regalloc/coloring.rs
@@ -551,7 +551,7 @@ impl<'a> Context<'a> {
         let is_reload = match &self.cur.func.dfg[inst] {
             InstructionData::Unary {
                 opcode: Opcode::Fill,
-                arg: _,
+                ..
             } => true,
             _ => false,
         };
diff --git a/cranelift/codegen/src/regalloc/diversion.rs b/cranelift/codegen/src/regalloc/diversion.rs
index 12461c5cea..feb9c6f0ef 100644
--- a/cranelift/codegen/src/regalloc/diversion.rs
+++ b/cranelift/codegen/src/regalloc/diversion.rs
@@ -200,7 +200,7 @@ impl RegDiversions {
         }
         debug_assert!(!entry_diversions.map.contains_key(target));
         let iter = self.current.iter();
-        let mut entry_divert = RegDiversions::new();
+        let mut entry_divert = Self::new();
         entry_divert.current.extend(iter);
         entry_diversions.map.insert(EntryRegDiversionsValue {
             key: target,
@@ -225,7 +225,7 @@ impl RegDiversions {
                 return false;
             }
         }
-        return true;
+        true
     }
 
     /// Return an object that can display the diversions.
@@ -237,7 +237,7 @@ impl RegDiversions {
 impl EntryRegDiversions {
     /// Create a new empty entry diversion, to associate diversions to each EBB entry.
     pub fn new() -> Self {
-        EntryRegDiversions {
+        Self {
             map: SparseMap::new(),
         }
     }
diff --git a/cranelift/codegen/src/regalloc/virtregs.rs b/cranelift/codegen/src/regalloc/virtregs.rs
index 628c49c286..8e1f4a0276 100644
--- a/cranelift/codegen/src/regalloc/virtregs.rs
+++ b/cranelift/codegen/src/regalloc/virtregs.rs
@@ -259,9 +259,9 @@ impl UFEntry {
     /// Decode a table entry.
     fn decode(x: i32) -> Self {
         if x < 0 {
-            UFEntry::Link(Value::from_u32((!x) as u32))
+            Self::Link(Value::from_u32((!x) as u32))
         } else {
-            UFEntry::Rank(x as u32)
+            Self::Rank(x as u32)
         }
     }
 
diff --git a/cranelift/codegen/src/result.rs b/cranelift/codegen/src/result.rs
index 5beb416a94..b0af78a3d6 100644
--- a/cranelift/codegen/src/result.rs
+++ b/cranelift/codegen/src/result.rs
@@ -37,6 +37,6 @@ pub type CodegenResult = Result;
 
 impl From for CodegenError {
     fn from(e: VerifierErrors) -> Self {
-        CodegenError::Verifier(e)
+        Self::Verifier(e)
     }
 }
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index 44fafd0b64..a20f0662f1 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -324,7 +324,7 @@ pub mod detail {
         /// offset field has a different meaning when the detail is a preset.
         pub fn is_preset(self) -> bool {
             match self {
-                Detail::Preset => true,
+                Self::Preset => true,
                 _ => false,
             }
         }
diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index 73e22451c9..fb91417131 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -597,15 +597,14 @@ fn simplify(pos: &mut FuncCursor, inst: Inst, native_word_width: u32) {
             }
         }
 
-        InstructionData::Unary { opcode, arg } => match opcode {
-            Opcode::AdjustSpDown => {
+        InstructionData::Unary { opcode, arg } => {
+            if let Opcode::AdjustSpDown = opcode {
                 if let Some(imm) = resolve_imm64_value(&pos.func.dfg, arg) {
                     // Note this works for both positive and negative immediate values.
                     pos.func.dfg.replace(inst).adjust_sp_down_imm(imm);
                 }
             }
-            _ => {}
-        },
+        }
 
         InstructionData::BinaryImm { opcode, arg, imm } => {
             let ty = pos.func.dfg.ctrl_typevar(inst);
@@ -626,27 +625,25 @@ fn simplify(pos: &mut FuncCursor, inst: Inst, native_word_width: u32) {
                             imm: prev_imm,
                         } = &pos.func.dfg[arg_inst]
                         {
-                            if opcode == *prev_opcode {
-                                if ty == pos.func.dfg.ctrl_typevar(arg_inst) {
-                                    let lhs: i64 = imm.into();
-                                    let rhs: i64 = (*prev_imm).into();
-                                    let new_imm = match opcode {
-                                        Opcode::BorImm => lhs | rhs,
-                                        Opcode::BandImm => lhs & rhs,
-                                        Opcode::BxorImm => lhs ^ rhs,
-                                        Opcode::IaddImm => lhs.wrapping_add(rhs),
-                                        Opcode::ImulImm => lhs.wrapping_mul(rhs),
-                                        _ => panic!("can't happen"),
-                                    };
-                                    let new_imm = immediates::Imm64::from(new_imm);
-                                    let new_arg = *prev_arg;
-                                    pos.func
-                                        .dfg
-                                        .replace(inst)
-                                        .BinaryImm(opcode, ty, new_imm, new_arg);
-                                    imm = new_imm;
-                                    arg = new_arg;
-                                }
+                            if opcode == *prev_opcode && ty == pos.func.dfg.ctrl_typevar(arg_inst) {
+                                let lhs: i64 = imm.into();
+                                let rhs: i64 = (*prev_imm).into();
+                                let new_imm = match opcode {
+                                    Opcode::BorImm => lhs | rhs,
+                                    Opcode::BandImm => lhs & rhs,
+                                    Opcode::BxorImm => lhs ^ rhs,
+                                    Opcode::IaddImm => lhs.wrapping_add(rhs),
+                                    Opcode::ImulImm => lhs.wrapping_mul(rhs),
+                                    _ => panic!("can't happen"),
+                                };
+                                let new_imm = immediates::Imm64::from(new_imm);
+                                let new_arg = *prev_arg;
+                                pos.func
+                                    .dfg
+                                    .replace(inst)
+                                    .BinaryImm(opcode, ty, new_imm, new_arg);
+                                imm = new_imm;
+                                arg = new_arg;
                             }
                         }
                     }
@@ -679,17 +676,14 @@ fn simplify(pos: &mut FuncCursor, inst: Inst, native_word_width: u32) {
                 | (Opcode::SshrImm, 0) => {
                     // Alias the result value with the original argument.
                     replace_single_result_with_alias(&mut pos.func.dfg, inst, arg);
-                    return;
                 }
                 (Opcode::ImulImm, 0) | (Opcode::BandImm, 0) => {
                     // Replace by zero.
                     pos.func.dfg.replace(inst).iconst(ty, 0);
-                    return;
                 }
                 (Opcode::BorImm, -1) => {
                     // Replace by minus one.
                     pos.func.dfg.replace(inst).iconst(ty, -1);
-                    return;
                 }
                 _ => {}
             }
@@ -789,9 +783,9 @@ fn branch_opt(pos: &mut FuncCursor, inst: Inst) {
 
             BranchOptInfo {
                 br_inst: inst,
-                cmp_arg: cmp_arg,
+                cmp_arg,
                 args: br_args.clone(),
-                new_opcode: new_opcode,
+                new_opcode,
             }
         } else {
             return;
diff --git a/cranelift/codegen/src/timing.rs b/cranelift/codegen/src/timing.rs
index 34ba801593..3cfc67f79d 100644
--- a/cranelift/codegen/src/timing.rs
+++ b/cranelift/codegen/src/timing.rs
@@ -141,7 +141,7 @@ mod details {
 
     impl Default for PassTimes {
         fn default() -> Self {
-            PassTimes {
+            Self {
                 pass: [Default::default(); NUM_PASSES],
             }
         }
diff --git a/cranelift/codegen/src/value_label.rs b/cranelift/codegen/src/value_label.rs
index b597fdd585..aab8792536 100644
--- a/cranelift/codegen/src/value_label.rs
+++ b/cranelift/codegen/src/value_label.rs
@@ -104,14 +104,14 @@ where
         if range.0 >= range.1 || !loc.is_assigned() {
             return;
         }
-        if !ranges.contains_key(&label) {
-            ranges.insert(label, Vec::new());
-        }
-        ranges.get_mut(&label).unwrap().push(ValueLocRange {
-            loc,
-            start: range.0,
-            end: range.1,
-        });
+        ranges
+            .entry(label)
+            .or_insert_with(Vec::new)
+            .push(ValueLocRange {
+                loc,
+                start: range.0,
+                end: range.1,
+            });
     };
 
     let mut end_offset = 0;
@@ -130,7 +130,7 @@ where
                     add_range(*label, (*start_offset, end_offset), *last_loc);
                     return false;
                 }
-                return true;
+                true
             });
 
             let srcloc = func.srclocs[inst];
@@ -152,7 +152,7 @@ where
             }
 
             // New source locations range started: abandon all tracked values.
-            if last_srcloc.is_some() && last_srcloc.as_ref().unwrap() > &srcloc {
+            if last_srcloc.is_some() && last_srcloc.unwrap() > srcloc {
                 for (_, label, start_offset, last_loc) in &tracked_values {
                     add_range(*label, (*start_offset, end_offset), *last_loc);
                 }
@@ -193,7 +193,7 @@ where
 
     // Optimize ranges in-place
     for (_, label_ranges) in ranges.iter_mut() {
-        assert!(label_ranges.len() > 0);
+        assert!(!label_ranges.is_empty());
         label_ranges.sort_by(|a, b| a.start.cmp(&b.start).then_with(|| a.end.cmp(&b.end)));
 
         // Merge ranges
@@ -245,7 +245,7 @@ pub struct ComparableSourceLoc(SourceLoc);
 
 impl From for ComparableSourceLoc {
     fn from(s: SourceLoc) -> Self {
-        ComparableSourceLoc(s)
+        Self(s)
     }
 }
 
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 7fc33dfd67..b57add8de6 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -167,7 +167,7 @@ impl VerifierErrors {
     /// Return a new `VerifierErrors` struct.
     #[inline]
     pub fn new() -> Self {
-        VerifierErrors(Vec::new())
+        Self(Vec::new())
     }
 
     /// Return whether no errors were reported.
@@ -196,7 +196,7 @@ impl VerifierErrors {
 
 impl From> for VerifierErrors {
     fn from(v: Vec) -> Self {
-        VerifierErrors(v)
+        Self(v)
     }
 }
 
diff --git a/cranelift/codegen/src/write.rs b/cranelift/codegen/src/write.rs
index f2758e0c30..507b2f9f6c 100644
--- a/cranelift/codegen/src/write.rs
+++ b/cranelift/codegen/src/write.rs
@@ -268,9 +268,9 @@ pub fn write_ebb_header(
     writeln!(w, "):")
 }
 
-fn write_valueloc(w: &mut dyn Write, loc: &ValueLoc, regs: &RegInfo) -> fmt::Result {
+fn write_valueloc(w: &mut dyn Write, loc: ValueLoc, regs: &RegInfo) -> fmt::Result {
     match loc {
-        ValueLoc::Reg(r) => write!(w, "{}", regs.display_regunit(*r)),
+        ValueLoc::Reg(r) => write!(w, "{}", regs.display_regunit(r)),
         ValueLoc::Stack(ss) => write!(w, "{}", ss),
         ValueLoc::Unassigned => write!(w, "?"),
     }
@@ -289,7 +289,7 @@ fn write_value_range_markers(
         for i in (0..rng.len()).rev() {
             if rng[i].start == offset {
                 write!(&mut result, " {}@", val)?;
-                write_valueloc(&mut result, &rng[i].loc, regs)?;
+                write_valueloc(&mut result, rng[i].loc, regs)?;
                 shown.insert(val);
                 break;
             }
@@ -303,7 +303,7 @@ fn write_value_range_markers(
             }
         }
     }
-    if result.len() > 0 {
+    if !result.is_empty() {
         writeln!(w, ";{1:0$}; {2}", indent + 24, "", result)?;
     }
     Ok(())
@@ -330,21 +330,24 @@ fn decorate_ebb(
         write_value_aliases(w, aliases, a, indent)?;
     }
 
-    if isa.is_some() && !func.offsets.is_empty() {
-        let encinfo = isa.unwrap().encoding_info();
-        let regs = &isa.unwrap().register_info();
-        for (offset, inst, size) in func.inst_offsets(ebb, &encinfo) {
-            func_w.write_instruction(w, func, aliases, isa, inst, indent)?;
-            if size > 0 {
-                if let Some(val_ranges) = annotations.value_ranges {
-                    write_value_range_markers(w, val_ranges, regs, offset + size, indent)?;
+    if let Some(isa) = isa {
+        if !func.offsets.is_empty() {
+            let encinfo = isa.encoding_info();
+            let regs = &isa.register_info();
+            for (offset, inst, size) in func.inst_offsets(ebb, &encinfo) {
+                func_w.write_instruction(w, func, aliases, Some(isa), inst, indent)?;
+                if size > 0 {
+                    if let Some(val_ranges) = annotations.value_ranges {
+                        write_value_range_markers(w, val_ranges, regs, offset + size, indent)?;
+                    }
                 }
             }
+            return Ok(());
         }
-    } else {
-        for inst in func.layout.ebb_insts(ebb) {
-            func_w.write_instruction(w, func, aliases, isa, inst, indent)?;
-        }
+    }
+
+    for inst in func.layout.ebb_insts(ebb) {
+        func_w.write_instruction(w, func, aliases, isa, inst, indent)?;
     }
 
     Ok(())
diff --git a/cranelift/entity/src/lib.rs b/cranelift/entity/src/lib.rs
index 793a8af27a..4f007ed34c 100644
--- a/cranelift/entity/src/lib.rs
+++ b/cranelift/entity/src/lib.rs
@@ -32,10 +32,7 @@
 #![deny(missing_docs, trivial_numeric_casts, unused_extern_crates)]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(
-    feature = "cargo-clippy",
-    allow(clippy::new_without_default, clippy::new_without_default_derive)
-)]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/cranelift/entity/src/packed_option.rs b/cranelift/entity/src/packed_option.rs
index 0757e9e19b..63764406c0 100644
--- a/cranelift/entity/src/packed_option.rs
+++ b/cranelift/entity/src/packed_option.rs
@@ -67,7 +67,7 @@ impl PackedOption {
 impl Default for PackedOption {
     /// Create a default packed option representing `None`.
     fn default() -> Self {
-        PackedOption(T::reserved_value())
+        Self(T::reserved_value())
     }
 }
 
@@ -78,7 +78,7 @@ impl From for PackedOption {
             t != T::reserved_value(),
             "Can't make a PackedOption from the reserved value."
         );
-        PackedOption(t)
+        Self(t)
     }
 }
 
diff --git a/cranelift/entity/src/set.rs b/cranelift/entity/src/set.rs
index c54ae4b31d..c5ba346c63 100644
--- a/cranelift/entity/src/set.rs
+++ b/cranelift/entity/src/set.rs
@@ -126,8 +126,7 @@ where
             // `(i + 1) * 8` = Last bit in byte.
             // `last - byte.leading_zeros()` = last set bit in byte.
             // `as usize` won't ever truncate as the potential range is `0..=8`.
-            .map(|(i, byte)| ((i + 1) * 8) - byte.leading_zeros() as usize)
-            .unwrap_or(0);
+            .map_or(0, |(i, byte)| ((i + 1) * 8) - byte.leading_zeros() as usize);
 
         Some(K::new(last_index))
     }
diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index 585866ca98..db173b929b 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -357,7 +357,7 @@ fn translate_function_linkage(linkage: Linkage) -> faerie::Decl {
 }
 
 fn translate_data_linkage(linkage: Linkage, writable: bool, align: Option) -> faerie::Decl {
-    let align = align.map(|align| u64::from(align));
+    let align = align.map(u64::from);
     match linkage {
         Linkage::Import => faerie::Decl::data_import().into(),
         Linkage::Local => faerie::Decl::data()
diff --git a/cranelift/faerie/src/lib.rs b/cranelift/faerie/src/lib.rs
index 2bf2f2b599..a6098ff8f7 100644
--- a/cranelift/faerie/src/lib.rs
+++ b/cranelift/faerie/src/lib.rs
@@ -10,10 +10,7 @@
 )]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(
-    feature = "cargo-clippy",
-    allow(clippy::new_without_default, clippy::new_without_default_derive)
-)]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/cranelift/filetests/src/function_runner.rs b/cranelift/filetests/src/function_runner.rs
index 6f2cd4a83a..32c2bf5f50 100644
--- a/cranelift/filetests/src/function_runner.rs
+++ b/cranelift/filetests/src/function_runner.rs
@@ -15,20 +15,20 @@ pub struct FunctionRunner {
 impl FunctionRunner {
     /// Build a function runner from a function and the ISA to run on (must be the host machine's ISA)
     pub fn new(function: Function, isa: Box) -> Self {
-        FunctionRunner { function, isa }
+        Self { function, isa }
     }
 
     /// Build a function runner using the host machine's ISA and the passed flags
     pub fn with_host_isa(function: Function, flags: settings::Flags) -> Self {
         let builder = host_isa_builder().expect("Unable to build a TargetIsa for the current host");
         let isa = builder.finish(flags);
-        FunctionRunner::new(function, isa)
+        Self::new(function, isa)
     }
 
     /// Build a function runner using the host machine's ISA and the default flags for this ISA
     pub fn with_default_host_isa(function: Function) -> Self {
         let flags = settings::Flags::new(settings::builder());
-        FunctionRunner::with_host_isa(function, flags)
+        Self::with_host_isa(function, flags)
     }
 
     /// Compile and execute a single function, expecting a boolean to be returned; a 'true' value is
@@ -79,9 +79,10 @@ impl FunctionRunner {
         let callable_fn: fn() -> bool = unsafe { mem::transmute(code_page.as_ptr()) };
 
         // execute
-        match callable_fn() {
-            true => Ok(()),
-            false => Err(format!("Failed: {}", context.func.name.to_string())),
+        if callable_fn() {
+            Ok(())
+        } else {
+            Err(format!("Failed: {}", context.func.name.to_string()))
         }
     }
 }
diff --git a/cranelift/filetests/src/runone.rs b/cranelift/filetests/src/runone.rs
index 1bb7d1c7f2..3cf399abbe 100644
--- a/cranelift/filetests/src/runone.rs
+++ b/cranelift/filetests/src/runone.rs
@@ -57,10 +57,10 @@ pub fn run(path: &Path, passes: Option<&[String]>, target: Option<&str>) -> Test
             Feature::With(name) => (name, true),
             Feature::Without(name) => (name, false),
         };
-        let cranelift_has = match flag {
+        let cranelift_has = match *flag {
             // Add any cranelift feature flag here, and make sure that it is forwarded to the
             // cranelift-filetest crate in the top-level Cargo.toml.
-            &"basic-blocks" => cfg!(feature = "basic-blocks"),
+            "basic-blocks" => cfg!(feature = "basic-blocks"),
             _ => {
                 return Err(format!(
                     r#"{:?}: Unknown feature flag named "{}""#,
diff --git a/cranelift/frontend/src/ssa.rs b/cranelift/frontend/src/ssa.rs
index 7a1f440a1e..c24e373677 100644
--- a/cranelift/frontend/src/ssa.rs
+++ b/cranelift/frontend/src/ssa.rs
@@ -95,8 +95,8 @@ enum BlockData {
 impl BlockData {
     fn add_predecessor(&mut self, pred: Block, inst: Inst) {
         match *self {
-            BlockData::EbbBody { .. } => panic!("you can't add a predecessor to a body block"),
-            BlockData::EbbHeader(ref mut data) => {
+            Self::EbbBody { .. } => panic!("you can't add a predecessor to a body block"),
+            Self::EbbHeader(ref mut data) => {
                 debug_assert!(!data.sealed, "sealed blocks cannot accept new predecessors");
                 data.predecessors.push(PredBlock::new(pred, inst));
             }
@@ -104,8 +104,8 @@ impl BlockData {
     }
     fn remove_predecessor(&mut self, inst: Inst) -> Block {
         match *self {
-            BlockData::EbbBody { .. } => panic!("should not happen"),
-            BlockData::EbbHeader(ref mut data) => {
+            Self::EbbBody { .. } => panic!("should not happen"),
+            Self::EbbHeader(ref mut data) => {
                 // This a linear complexity operation but the number of predecessors is low
                 // in all non-pathological cases
                 let pred: usize = data
@@ -149,7 +149,7 @@ pub struct Block(u32);
 impl EntityRef for Block {
     fn new(index: usize) -> Self {
         debug_assert!(index < (u32::MAX as usize));
-        Block(index as u32)
+        Self(index as u32)
     }
 
     fn index(self) -> usize {
@@ -159,7 +159,7 @@ impl EntityRef for Block {
 
 impl ReservedValue for Block {
     fn reserved_value() -> Self {
-        Block(u32::MAX)
+        Self(u32::MAX)
     }
 }
 
diff --git a/cranelift/frontend/src/variable.rs b/cranelift/frontend/src/variable.rs
index dddcd7490b..9e7c260248 100644
--- a/cranelift/frontend/src/variable.rs
+++ b/cranelift/frontend/src/variable.rs
@@ -16,14 +16,14 @@ impl Variable {
     /// Create a new Variable with the given index.
     pub fn with_u32(index: u32) -> Self {
         debug_assert!(index < u32::MAX);
-        Variable(index)
+        Self(index)
     }
 }
 
 impl EntityRef for Variable {
     fn new(index: usize) -> Self {
         debug_assert!(index < (u32::MAX as usize));
-        Variable(index as u32)
+        Self(index as u32)
     }
 
     fn index(self) -> usize {
diff --git a/cranelift/module/src/data_context.rs b/cranelift/module/src/data_context.rs
index 335dd9a18d..94dbeb0b98 100644
--- a/cranelift/module/src/data_context.rs
+++ b/cranelift/module/src/data_context.rs
@@ -27,9 +27,9 @@ impl Init {
     /// Return the size of the data to be initialized.
     pub fn size(&self) -> usize {
         match *self {
-            Init::Uninitialized => panic!("data size not initialized yet"),
-            Init::Zeros { size } => size,
-            Init::Bytes { ref contents } => contents.len(),
+            Self::Uninitialized => panic!("data size not initialized yet"),
+            Self::Zeros { size } => size,
+            Self::Bytes { ref contents } => contents.len(),
         }
     }
 }
diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index f19dcfab21..847710ed1e 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -25,7 +25,7 @@ entity_impl!(FuncId, "funcid");
 /// Function identifiers are namespace 0 in `ir::ExternalName`
 impl From for ir::ExternalName {
     fn from(id: FuncId) -> Self {
-        ir::ExternalName::User {
+        Self::User {
             namespace: 0,
             index: id.0,
         }
@@ -40,7 +40,7 @@ entity_impl!(DataId, "dataid");
 /// Data identifiers are namespace 1 in `ir::ExternalName`
 impl From for ir::ExternalName {
     fn from(id: DataId) -> Self {
-        ir::ExternalName::User {
+        Self::User {
             namespace: 1,
             index: id.0,
         }
@@ -63,33 +63,33 @@ pub enum Linkage {
 impl Linkage {
     fn merge(a: Self, b: Self) -> Self {
         match a {
-            Linkage::Export => Linkage::Export,
-            Linkage::Preemptible => match b {
-                Linkage::Export => Linkage::Export,
-                _ => Linkage::Preemptible,
+            Self::Export => Self::Export,
+            Self::Preemptible => match b {
+                Self::Export => Self::Export,
+                _ => Self::Preemptible,
             },
-            Linkage::Local => match b {
-                Linkage::Export => Linkage::Export,
-                Linkage::Preemptible => Linkage::Preemptible,
-                _ => Linkage::Local,
+            Self::Local => match b {
+                Self::Export => Self::Export,
+                Self::Preemptible => Self::Preemptible,
+                _ => Self::Local,
             },
-            Linkage::Import => b,
+            Self::Import => b,
         }
     }
 
     /// Test whether this linkage can have a definition.
     pub fn is_definable(self) -> bool {
         match self {
-            Linkage::Import => false,
-            Linkage::Local | Linkage::Preemptible | Linkage::Export => true,
+            Self::Import => false,
+            Self::Local | Self::Preemptible | Self::Export => true,
         }
     }
 
     /// Test whether this linkage will have a definition that cannot be preempted.
     pub fn is_final(self) -> bool {
         match self {
-            Linkage::Import | Linkage::Preemptible => false,
-            Linkage::Local | Linkage::Export => true,
+            Self::Import | Self::Preemptible => false,
+            Self::Local | Self::Export => true,
         }
     }
 }
diff --git a/cranelift/object/src/backend.rs b/cranelift/object/src/backend.rs
index 09f209bb3f..37b4c6bff2 100644
--- a/cranelift/object/src/backend.rs
+++ b/cranelift/object/src/backend.rs
@@ -327,7 +327,7 @@ impl Backend for ObjectBackend {
             addend,
         } in &func.relocs
         {
-            let offset = func.offset + offset as u64;
+            let offset = func.offset + u64::from(offset);
             let symbol = self.get_symbol(namespace, name);
             self.object
                 .add_relocation(
@@ -364,7 +364,7 @@ impl Backend for ObjectBackend {
             addend,
         } in &data.relocs
         {
-            let offset = data.offset + offset as u64;
+            let offset = data.offset + u64::from(offset);
             let symbol = self.get_symbol(namespace, name);
             self.object
                 .add_relocation(
diff --git a/cranelift/object/src/lib.rs b/cranelift/object/src/lib.rs
index 162a0b47aa..241dcb6316 100644
--- a/cranelift/object/src/lib.rs
+++ b/cranelift/object/src/lib.rs
@@ -10,10 +10,7 @@
 )]
 #![warn(unused_import_braces)]
 #![cfg_attr(feature = "clippy", plugin(clippy(conf_file = "../../clippy.toml")))]
-#![cfg_attr(
-    feature = "cargo-clippy",
-    allow(clippy::new_without_default, clippy::new_without_default_derive)
-)]
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::new_without_default))]
 #![cfg_attr(
     feature = "cargo-clippy",
     warn(
diff --git a/cranelift/preopt/src/constant_folding.rs b/cranelift/preopt/src/constant_folding.rs
index cf4dba2815..b87db2d98f 100644
--- a/cranelift/preopt/src/constant_folding.rs
+++ b/cranelift/preopt/src/constant_folding.rs
@@ -1,4 +1,5 @@
 //! Fold operations on constants at compile time.
+#![allow(clippy::float_arithmetic)]
 
 use cranelift_codegen::{
     cursor::{Cursor, FuncCursor},
@@ -18,7 +19,7 @@ enum ConstImm {
 
 impl ConstImm {
     fn unwrap_i64(self) -> i64 {
-        if let ConstImm::I64(imm) = self {
+        if let Self::I64(imm) = self {
             imm
         } else {
             panic!("self did not contain an `i64`.")
@@ -27,8 +28,8 @@ impl ConstImm {
 
     fn evaluate_truthiness(self) -> bool {
         match self {
-            ConstImm::Bool(b) => b,
-            ConstImm::I64(imm) => imm != 0,
+            Self::Bool(b) => b,
+            Self::I64(imm) => imm != 0,
             _ => panic!(
                 "Only a `ConstImm::Bool` and `ConstImm::I64` can be evaluated for \"truthiness\""
             ),
@@ -53,11 +54,7 @@ pub fn fold_constants(func: &mut ir::Function) {
                 Unary { opcode, arg } => {
                     fold_unary(&mut pos.func.dfg, inst, opcode, arg);
                 }
-                Branch {
-                    opcode,
-                    args: _,
-                    destination: _,
-                } => {
+                Branch { opcode, .. } => {
                     fold_branch(&mut pos, inst, opcode);
                 }
                 _ => {}
diff --git a/cranelift/reader/src/isaspec.rs b/cranelift/reader/src/isaspec.rs
index 140eb8324b..ed16fd4416 100644
--- a/cranelift/reader/src/isaspec.rs
+++ b/cranelift/reader/src/isaspec.rs
@@ -25,7 +25,7 @@ pub enum IsaSpec {
 impl IsaSpec {
     /// If the `IsaSpec` contains exactly 1 `TargetIsa` we return a reference to it
     pub fn unique_isa(&self) -> Option<&dyn TargetIsa> {
-        if let IsaSpec::Some(ref isa_vec) = *self {
+        if let Self::Some(ref isa_vec) = *self {
             if isa_vec.len() == 1 {
                 return Some(&*isa_vec[0]);
             }
diff --git a/cranelift/reader/src/lexer.rs b/cranelift/reader/src/lexer.rs
index 2432ab0551..0168d2027a 100644
--- a/cranelift/reader/src/lexer.rs
+++ b/cranelift/reader/src/lexer.rs
@@ -461,6 +461,7 @@ impl<'a> Lexer<'a> {
     /// Get the next token or a lexical error.
     ///
     /// Return None when the end of the source is encountered.
+    #[allow(clippy::cognitive_complexity)]
     pub fn next(&mut self) -> Option, LocatedError>> {
         loop {
             let loc = self.loc();
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 1b1698caf4..24f8b5a2e5 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -3291,6 +3291,9 @@ mod tests {
         let c = Parser::new("true false true false")
             .parse_literals_to_constant_data(B32X4)
             .unwrap();
-        assert_eq!(c.to_vec(), [1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0])
+        assert_eq!(
+            c.into_vec(),
+            [1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]
+        )
     }
 }
diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index 43ba5e2fda..ac1ccecc2d 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -328,6 +328,7 @@ impl Mutator for RemoveUnusedEntities {
         4
     }
 
+    #[allow(clippy::cognitive_complexity)]
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, ProgressStatus)> {
         let name = match self.kind {
             0 => {
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index d2752bdb36..a4ee4c8666 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -4,7 +4,7 @@
 //! Reads Wasm binary/text files, translates the functions' code to Cranelift IR.
 #![cfg_attr(
     feature = "cargo-clippy",
-    allow(clippy::too_many_arguments, clippy::cyclomatic_complexity)
+    allow(clippy::too_many_arguments, clippy::cognitive_complexity)
 )]
 
 use crate::disasm::{print_all, PrintRelocs, PrintStackmaps, PrintTraps};
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 709d40894f..6fda78be27 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -41,7 +41,10 @@ use cranelift_frontend::{FunctionBuilder, Variable};
 use wasmparser::{MemoryImmediate, Operator};
 
 // Clippy warns about "flags: _" but its important to document that the flags field is ignored
-#[cfg_attr(feature = "cargo-clippy", allow(clippy::unneeded_field_pattern))]
+#[cfg_attr(
+    feature = "cargo-clippy",
+    allow(clippy::unneeded_field_pattern, clippy::cognitive_complexity)
+)]
 /// Translates wasm operators into Cranelift IR instructions. Returns `true` if it inserted
 /// a return.
 pub fn translate_operator(
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 6e44f22c74..46a823fc80 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -87,7 +87,7 @@ impl From for WasmError {
     /// Convert from a `BinaryReaderError` to a `WasmError`.
     fn from(e: BinaryReaderError) -> Self {
         let BinaryReaderError { message, offset } = e;
-        WasmError::InvalidWebAssembly { message, offset }
+        Self::InvalidWebAssembly { message, offset }
     }
 }
 
@@ -478,8 +478,7 @@ pub trait ModuleEnvironment<'data> {
     }
 
     /// Indicates that a custom section has been found in the wasm file
-    fn custom_section(&mut self, name: &'data str, data: &'data [u8]) -> WasmResult<()> {
-        drop((name, data));
+    fn custom_section(&mut self, _name: &'data str, _data: &'data [u8]) -> WasmResult<()> {
         Ok(())
     }
 }
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index cc479c8a3e..0a7c678f4c 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -409,8 +409,8 @@ pub fn parse_name_section<'data>(
     Ok(())
 }
 
-fn parse_function_name_subsection<'data>(
-    mut naming_reader: NamingReader<'data>,
+fn parse_function_name_subsection(
+    mut naming_reader: NamingReader<'_>,
 ) -> Option> {
     let mut function_names = HashMap::new();
     for _ in 0..naming_reader.get_count() {
@@ -425,5 +425,5 @@ fn parse_function_name_subsection<'data>(
             return None;
         }
     }
-    return Some(function_names);
+    Some(function_names)
 }
diff --git a/cranelift/wasm/src/state/func_state.rs b/cranelift/wasm/src/state/func_state.rs
index 88d83ab846..c768248ddd 100644
--- a/cranelift/wasm/src/state/func_state.rs
+++ b/cranelift/wasm/src/state/func_state.rs
@@ -91,55 +91,54 @@ pub enum ControlStackFrame {
 impl ControlStackFrame {
     pub fn num_return_values(&self) -> usize {
         match *self {
-            ControlStackFrame::If {
+            Self::If {
                 num_return_values, ..
             }
-            | ControlStackFrame::Block {
+            | Self::Block {
                 num_return_values, ..
             }
-            | ControlStackFrame::Loop {
+            | Self::Loop {
                 num_return_values, ..
             } => num_return_values,
         }
     }
     pub fn num_param_values(&self) -> usize {
         match *self {
-            ControlStackFrame::If {
+            Self::If {
                 num_param_values, ..
             }
-            | ControlStackFrame::Block {
+            | Self::Block {
                 num_param_values, ..
             }
-            | ControlStackFrame::Loop {
+            | Self::Loop {
                 num_param_values, ..
             } => num_param_values,
         }
     }
     pub fn following_code(&self) -> Ebb {
         match *self {
-            ControlStackFrame::If { destination, .. }
-            | ControlStackFrame::Block { destination, .. }
-            | ControlStackFrame::Loop { destination, .. } => destination,
+            Self::If { destination, .. }
+            | Self::Block { destination, .. }
+            | Self::Loop { destination, .. } => destination,
         }
     }
     pub fn br_destination(&self) -> Ebb {
         match *self {
-            ControlStackFrame::If { destination, .. }
-            | ControlStackFrame::Block { destination, .. } => destination,
-            ControlStackFrame::Loop { header, .. } => header,
+            Self::If { destination, .. } | Self::Block { destination, .. } => destination,
+            Self::Loop { header, .. } => header,
         }
     }
     pub fn original_stack_size(&self) -> usize {
         match *self {
-            ControlStackFrame::If {
+            Self::If {
                 original_stack_size,
                 ..
             }
-            | ControlStackFrame::Block {
+            | Self::Block {
                 original_stack_size,
                 ..
             }
-            | ControlStackFrame::Loop {
+            | Self::Loop {
                 original_stack_size,
                 ..
             } => original_stack_size,
@@ -147,36 +146,36 @@ impl ControlStackFrame {
     }
     pub fn is_loop(&self) -> bool {
         match *self {
-            ControlStackFrame::If { .. } | ControlStackFrame::Block { .. } => false,
-            ControlStackFrame::Loop { .. } => true,
+            Self::If { .. } | Self::Block { .. } => false,
+            Self::Loop { .. } => true,
         }
     }
 
     pub fn exit_is_branched_to(&self) -> bool {
         match *self {
-            ControlStackFrame::If {
+            Self::If {
                 exit_is_branched_to,
                 ..
             }
-            | ControlStackFrame::Block {
+            | Self::Block {
                 exit_is_branched_to,
                 ..
             } => exit_is_branched_to,
-            ControlStackFrame::Loop { .. } => false,
+            Self::Loop { .. } => false,
         }
     }
 
     pub fn set_branched_to_exit(&mut self) {
         match *self {
-            ControlStackFrame::If {
+            Self::If {
                 ref mut exit_is_branched_to,
                 ..
             }
-            | ControlStackFrame::Block {
+            | Self::Block {
                 ref mut exit_is_branched_to,
                 ..
             } => *exit_is_branched_to = true,
-            ControlStackFrame::Loop { .. } => {}
+            Self::Loop { .. } => {}
         }
     }
 }
diff --git a/cranelift/wasm/src/state/module_state.rs b/cranelift/wasm/src/state/module_state.rs
index 306b17a068..e997305858 100644
--- a/cranelift/wasm/src/state/module_state.rs
+++ b/cranelift/wasm/src/state/module_state.rs
@@ -2,6 +2,10 @@ use crate::translation_utils::SignatureIndex;
 use cranelift_entity::PrimaryMap;
 use std::boxed::Box;
 
+/// Map of signatures to a function's parameter and return types.
+pub(crate) type WasmTypes =
+    PrimaryMap, Box<[wasmparser::Type]>)>;
+
 /// Contains information decoded from the Wasm module that must be referenced
 /// during each Wasm function's translation.
 ///
@@ -14,14 +18,13 @@ pub struct ModuleTranslationState {
     ///
     /// This is used for translating multi-value Wasm blocks inside functions,
     /// which are encoded to refer to their type signature via index.
-    pub(crate) wasm_types:
-        PrimaryMap, Box<[wasmparser::Type]>)>,
+    pub(crate) wasm_types: WasmTypes,
 }
 
 impl ModuleTranslationState {
     /// Creates a new empty ModuleTranslationState.
     pub fn new() -> Self {
-        ModuleTranslationState {
+        Self {
             wasm_types: PrimaryMap::new(),
         }
     }

From e045a6df2765dcbd0b84dc4a67f1a22c388bf5b5 Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Sun, 27 Oct 2019 17:07:59 -0400
Subject: [PATCH 2878/3084] implement Debug for Linkage

this would have been useful while debugging something in my own project
---
 cranelift/module/src/module.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 847710ed1e..808dd88164 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -48,7 +48,7 @@ impl From for ir::ExternalName {
 }
 
 /// Linkage refers to where an entity is defined and who can see it.
-#[derive(Copy, Clone, PartialEq, Eq)]
+#[derive(Copy, Clone, Debug, PartialEq, Eq)]
 pub enum Linkage {
     /// Defined outside of a module.
     Import,

From f37d1c7ecca703ea07d2af60c992abae201a8639 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 23 Oct 2019 14:52:44 -0700
Subject: [PATCH 2879/3084] Simplify binding of IntCC::Equals to SIMD `icmp`;
 fixes #1150

---
 cranelift/codegen/meta/src/cdsl/encodings.rs  |  5 ++-
 .../codegen/meta/src/cdsl/instructions.rs     | 44 +++++--------------
 .../codegen/meta/src/isa/x86/encodings.rs     | 20 +++------
 3 files changed, 21 insertions(+), 48 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/encodings.rs b/cranelift/codegen/meta/src/cdsl/encodings.rs
index 6797672a87..f79d3b9430 100644
--- a/cranelift/codegen/meta/src/cdsl/encodings.rs
+++ b/cranelift/codegen/meta/src/cdsl/encodings.rs
@@ -93,7 +93,10 @@ impl EncodingBuilder {
                 {
                     let immediate_predicate = InstructionPredicate::new_is_field_equal(
                         &inst.inst.format,
-                        immediate_operand.name,
+                        immediate_operand
+                            .kind
+                            .default_member
+                            .expect("Immediates must always have a default member name set."),
                         immediate_value.to_string(),
                     );
                     inst_predicate = if let Some(type_predicate) = inst_predicate {
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 824d028741..6013724135 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -443,17 +443,15 @@ impl From for BindParameter {
 }
 
 #[derive(Clone)]
-#[allow(dead_code)] // TODO(#1150): remove this once we use it in legalization patterns.
 pub enum Immediate {
-    UInt8(u8),
-    UInt128(u128),
+    // When needed, this enum should be expanded to include other immediate types (e.g. u8, u128).
+    IntCC(IntCC),
 }
 
 impl Display for Immediate {
     fn fmt(&self, f: &mut Formatter) -> Result<(), Error> {
         match self {
-            Immediate::UInt8(x) => write!(f, "{}", x),
-            Immediate::UInt128(x) => write!(f, "{}", x),
+            Immediate::IntCC(x) => write!(f, "IntCC::{:?}", x),
         }
     }
 }
@@ -790,9 +788,6 @@ pub enum FormatPredicateKind {
 
     /// Is the referenced data object colocated?
     IsColocatedData,
-
-    /// Does the operation have a specific condition code?
-    HasConditionCode(IntCC),
 }
 
 #[derive(Clone, Hash, PartialEq, Eq)]
@@ -879,10 +874,6 @@ impl FormatPredicateNode {
             FormatPredicateKind::IsColocatedData => {
                 format!("predicates::is_colocated_data({}, func)", self.member_name)
             }
-            FormatPredicateKind::HasConditionCode(code) => format!(
-                "predicates::is_equal({}, IntCC::{:?})",
-                self.member_name, code
-            ),
         }
     }
 }
@@ -1165,18 +1156,6 @@ impl InstructionPredicate {
         ))
     }
 
-    pub fn new_has_condition_code(
-        format: &InstructionFormat,
-        condition_code: IntCC,
-        field_name: &'static str,
-    ) -> InstructionPredicateNode {
-        InstructionPredicateNode::FormatPredicate(FormatPredicateNode::new(
-            format,
-            field_name,
-            FormatPredicateKind::HasConditionCode(condition_code),
-        ))
-    }
-
     pub fn and(mut self, new_node: InstructionPredicateNode) -> Self {
         let node = self.node;
         let mut and_nodes = match node {
@@ -1318,7 +1297,7 @@ mod test {
     use crate::shared::types::Int::{I32, I64};
 
     fn field_to_operand(index: usize, field: OperandKindFields) -> Operand {
-        // pretend the index string is &'static
+        // Pretend the index string is &'static.
         let name = Box::leak(index.to_string().into_boxed_str());
         let kind = OperandKindBuilder::new(name, field).build();
         let operand = OperandBuilder::new(name, kind).build();
@@ -1337,7 +1316,7 @@ mod test {
         inputs: Vec,
         outputs: Vec,
     ) -> Instruction {
-        // setup a format from the input operands
+        // Setup a format from the input operands.
         let mut format = InstructionFormatBuilder::new("fake");
         for (i, f) in inputs.iter().enumerate() {
             match f {
@@ -1350,7 +1329,7 @@ mod test {
         }
         let format = format.build();
 
-        // create the fake instruction
+        // Create the fake instruction.
         InstructionBuilder::new("fake", "A fake instruction for testing.", &format)
             .operands_in(field_to_operands(inputs).iter().collect())
             .operands_out(field_to_operands(outputs).iter().collect())
@@ -1368,7 +1347,7 @@ mod test {
     #[test]
     fn ensure_bound_instructions_can_bind_immediates() {
         let inst = build_fake_instruction(vec![OperandKindFields::ImmValue], vec![]);
-        let bound_inst = inst.bind(Immediate::UInt8(42));
+        let bound_inst = inst.bind(Immediate::IntCC(IntCC::Equal));
         assert!(bound_inst.verify_bindings().is_ok());
     }
 
@@ -1377,7 +1356,7 @@ mod test {
     fn ensure_instructions_fail_to_bind() {
         let inst = build_fake_instruction(vec![], vec![]);
         inst.bind(BindParameter::Lane(LaneType::IntType(I32)));
-        // trying to bind to an instruction with no inputs should fail
+        // Trying to bind to an instruction with no inputs should fail.
     }
 
     #[test]
@@ -1394,8 +1373,9 @@ mod test {
     #[should_panic]
     fn ensure_instructions_fail_to_bind_too_many_immediates() {
         let inst = build_fake_instruction(vec![OperandKindFields::ImmValue], vec![]);
-        inst.bind(BindParameter::Immediate(Immediate::UInt8(0)))
-            .bind(BindParameter::Immediate(Immediate::UInt8(1)));
-        // trying to bind too many immediates to an instruction  should fail
+        inst.bind(BindParameter::Immediate(Immediate::IntCC(IntCC::Equal)))
+            .bind(BindParameter::Immediate(Immediate::IntCC(IntCC::Equal)));
+        // Trying to bind too many immediates to an instruction should fail; note that the immediate
+        // values are nonsensical but irrelevant to the purpose of this test.
     }
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 6e2755ba92..ad497188ec 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -5,7 +5,7 @@ use std::collections::HashMap;
 
 use crate::cdsl::encodings::{Encoding, EncodingBuilder};
 use crate::cdsl::instructions::{
-    vector, Bindable, InstSpec, Instruction, InstructionGroup, InstructionPredicate,
+    vector, Bindable, Immediate, InstSpec, Instruction, InstructionGroup, InstructionPredicate,
     InstructionPredicateNode, InstructionPredicateRegistry,
 };
 use crate::cdsl::recipes::{EncodingRecipe, EncodingRecipeNumber, Recipes};
@@ -2038,21 +2038,11 @@ pub(crate) fn define(
             _ => panic!("invalid size for SIMD icmp"),
         };
 
-        let instruction = icmp.bind(vector(ty, sse_vector_size));
-        let has_eq_condition_code = InstructionPredicate::new_has_condition_code(
-            &*formats.int_compare,
-            IntCC::Equal,
-            "cond",
-        );
+        let instruction = icmp
+            .bind(Immediate::IntCC(IntCC::Equal))
+            .bind(vector(ty, sse_vector_size));
         let template = rec_icscc_fpr.nonrex().opcodes(opcodes);
-        e.enc_32_64_func(instruction, template, |builder| {
-            let builder = builder.inst_predicate(has_eq_condition_code);
-            if let Some(p) = isa_predicate {
-                builder.isa_predicate(p)
-            } else {
-                builder
-            }
-        });
+        e.enc_32_64_maybe_isap(instruction, template, isa_predicate);
     }
 
     // Reference type instructions

From b657aa57f69c231cf7fc49aed7fea31eb2a4a27d Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 28 Oct 2019 17:45:40 +0100
Subject: [PATCH 2880/3084] [meta] Rename Operand::is_immediate to
 is_immediate_or_entityref

---
 cranelift/codegen/meta/src/cdsl/encodings.rs    | 11 ++++++-----
 cranelift/codegen/meta/src/cdsl/instructions.rs |  4 ++--
 cranelift/codegen/meta/src/cdsl/operands.rs     |  3 +--
 cranelift/codegen/meta/src/gen_legalizer.rs     |  4 ++--
 4 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/encodings.rs b/cranelift/codegen/meta/src/cdsl/encodings.rs
index f79d3b9430..cdeac64489 100644
--- a/cranelift/codegen/meta/src/cdsl/encodings.rs
+++ b/cranelift/codegen/meta/src/cdsl/encodings.rs
@@ -86,11 +86,12 @@ impl EncodingBuilder {
                 }
 
                 // Add immediate value predicates
-                for (immediate_value, immediate_operand) in inst
-                    .immediate_values
-                    .iter()
-                    .zip(inst.inst.operands_in.iter().filter(|o| o.is_immediate()))
-                {
+                for (immediate_value, immediate_operand) in inst.immediate_values.iter().zip(
+                    inst.inst
+                        .operands_in
+                        .iter()
+                        .filter(|o| o.is_pure_immediate()),
+                ) {
                     let immediate_predicate = InstructionPredicate::new_is_field_equal(
                         &inst.inst.format,
                         immediate_operand
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 6013724135..416eaebaff 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -322,7 +322,7 @@ impl InstructionBuilder {
         for (i, op) in operands_in.iter().enumerate() {
             if op.is_value() {
                 value_opnums.push(i);
-            } else if op.is_immediate() {
+            } else if op.is_immediate_or_entityref() {
                 imm_opnums.push(i);
             } else {
                 assert!(op.is_varargs());
@@ -500,7 +500,7 @@ impl BoundInstruction {
             .inst
             .operands_in
             .iter()
-            .filter(|o| o.is_immediate())
+            .filter(|o| o.is_immediate_or_entityref())
             .count();
         if self.immediate_values.len() > immediate_count {
             return Err(format!(
diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
index 631f0807aa..e87e9f9afd 100644
--- a/cranelift/codegen/meta/src/cdsl/operands.rs
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -46,8 +46,7 @@ impl Operand {
     }
 
     /// Returns true if the operand has an immediate kind or is an EntityRef.
-    // TODO inherited name from the python, rename to is_immediate_or_entityref later.
-    pub fn is_immediate(&self) -> bool {
+    pub fn is_immediate_or_entityref(&self) -> bool {
         match self.kind.fields {
             OperandKindFields::ImmEnum(_)
             | OperandKindFields::ImmValue
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index 6ff2016eaa..b73b67b754 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -66,7 +66,7 @@ fn unwrap_inst(transform: &Transform, fmt: &mut Formatter) -> bool {
         let mut imm_and_varargs = inst
             .operands_in
             .iter()
-            .filter(|op| op.is_immediate())
+            .filter(|op| op.is_immediate_or_entityref())
             .count();
         if iform.has_value_list {
             imm_and_varargs += 1;
@@ -115,7 +115,7 @@ fn unwrap_inst(transform: &Transform, fmt: &mut Formatter) -> bool {
         let emit_one_value =
             |fmt: &mut Formatter, needs_comma: bool, op_num: usize, op: &Operand| {
                 let comma = if needs_comma { "," } else { "" };
-                if op.is_immediate() {
+                if op.is_immediate_or_entityref() {
                     let n = inst.imm_opnums.iter().position(|&i| i == op_num).unwrap();
                     fmtln!(fmt, "{}{}", iform.imm_fields[n].member, comma);
                 } else if op.is_value() {

From 06b1817d89a4ec501ecc35e165b19b4da0e7818c Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 28 Oct 2019 17:46:26 +0100
Subject: [PATCH 2881/3084] [meta] Rename Operand::is_pure_immediate into
 is_immediate;

---
 cranelift/codegen/meta/src/cdsl/encodings.rs | 11 +++++------
 cranelift/codegen/meta/src/cdsl/operands.rs  |  2 +-
 cranelift/codegen/meta/src/gen_inst.rs       |  2 +-
 3 files changed, 7 insertions(+), 8 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/encodings.rs b/cranelift/codegen/meta/src/cdsl/encodings.rs
index cdeac64489..f79d3b9430 100644
--- a/cranelift/codegen/meta/src/cdsl/encodings.rs
+++ b/cranelift/codegen/meta/src/cdsl/encodings.rs
@@ -86,12 +86,11 @@ impl EncodingBuilder {
                 }
 
                 // Add immediate value predicates
-                for (immediate_value, immediate_operand) in inst.immediate_values.iter().zip(
-                    inst.inst
-                        .operands_in
-                        .iter()
-                        .filter(|o| o.is_pure_immediate()),
-                ) {
+                for (immediate_value, immediate_operand) in inst
+                    .immediate_values
+                    .iter()
+                    .zip(inst.inst.operands_in.iter().filter(|o| o.is_immediate()))
+                {
                     let immediate_predicate = InstructionPredicate::new_is_field_equal(
                         &inst.inst.format,
                         immediate_operand
diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
index e87e9f9afd..6283b08736 100644
--- a/cranelift/codegen/meta/src/cdsl/operands.rs
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -56,7 +56,7 @@ impl Operand {
     }
 
     /// Returns true if the operand has an immediate kind.
-    pub fn is_pure_immediate(&self) -> bool {
+    pub fn is_immediate(&self) -> bool {
         match self.kind.fields {
             OperandKindFields::ImmEnum(_) | OperandKindFields::ImmValue => true,
             _ => false,
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index 39309e05c0..b5d2454881 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -910,7 +910,7 @@ fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Fo
     let mut tmpl_types = Vec::new();
     let mut into_args = Vec::new();
     for op in &inst.operands_in {
-        let t = if op.is_pure_immediate() {
+        let t = if op.is_immediate() {
             let t = format!("T{}{}", tmpl_types.len() + 1, op.kind.name);
             tmpl_types.push(format!("{}: Into<{}>", t, op.kind.rust_type));
             into_args.push(op.name);

From 5889dd2c641ac32c6afff7a8f14a2b7d7a89dde9 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 28 Oct 2019 18:13:25 +0100
Subject: [PATCH 2882/3084] [meta] Add more pub(crate) definitions.

---
 cranelift/codegen/meta/src/cdsl/formats.rs    |  2 +-
 .../codegen/meta/src/cdsl/instructions.rs     | 14 +++++-----
 cranelift/codegen/meta/src/cdsl/operands.rs   | 21 +++++----------
 cranelift/codegen/meta/src/cdsl/recipes.rs    |  8 +++---
 cranelift/codegen/meta/src/cdsl/regs.rs       | 18 ++++++-------
 cranelift/codegen/meta/src/cdsl/settings.rs   | 26 +++++++++----------
 cranelift/codegen/meta/src/cdsl/types.rs      | 18 ++++++-------
 cranelift/codegen/meta/src/cdsl/typevar.rs    | 14 +++++-----
 cranelift/codegen/meta/src/gen_inst.rs        |  2 +-
 cranelift/codegen/meta/src/gen_settings.rs    |  2 +-
 .../codegen/meta/src/isa/x86/registers.rs     |  2 +-
 .../codegen/meta/src/isa/x86/settings.rs      |  2 +-
 cranelift/codegen/meta/src/shared/entities.rs | 20 +++++++-------
 cranelift/codegen/meta/src/shared/settings.rs |  2 +-
 cranelift/codegen/meta/src/shared/types.rs    | 20 +++++++-------
 cranelift/codegen/meta/src/srcgen.rs          |  4 +--
 cranelift/codegen/meta/src/unique_table.rs    |  4 +--
 17 files changed, 86 insertions(+), 93 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
index d03929b5f6..bcfc6f5650 100644
--- a/cranelift/codegen/meta/src/cdsl/formats.rs
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -7,7 +7,7 @@ use std::rc::Rc;
 /// This corresponds to a single member of a variant of the `InstructionData`
 /// data type.
 #[derive(Debug)]
-pub struct FormatField {
+pub(crate) struct FormatField {
     /// Immediate operand kind.
     pub kind: OperandKind,
 
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 416eaebaff..a00d20a386 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -393,7 +393,7 @@ impl ValueTypeOrAny {
 type VectorBitWidth = u64;
 
 /// An parameter used for binding instructions to specific types or values
-pub enum BindParameter {
+pub(crate) enum BindParameter {
     Any,
     Lane(LaneType),
     Vector(LaneType, VectorBitWidth),
@@ -402,7 +402,7 @@ pub enum BindParameter {
 }
 
 /// Constructor for more easily building vector parameters from any lane type
-pub fn vector(parameter: impl Into, vector_size: VectorBitWidth) -> BindParameter {
+pub(crate) fn vector(parameter: impl Into, vector_size: VectorBitWidth) -> BindParameter {
     BindParameter::Vector(parameter.into(), vector_size)
 }
 
@@ -443,7 +443,7 @@ impl From for BindParameter {
 }
 
 #[derive(Clone)]
-pub enum Immediate {
+pub(crate) enum Immediate {
     // When needed, this enum should be expanded to include other immediate types (e.g. u8, u128).
     IntCC(IntCC),
 }
@@ -750,7 +750,7 @@ fn is_ctrl_typevar_candidate(
 }
 
 #[derive(Clone, Hash, PartialEq, Eq)]
-pub enum FormatPredicateKind {
+pub(crate) enum FormatPredicateKind {
     /// Is the field member equal to the expected value (stored here)?
     IsEqual(String),
 
@@ -791,7 +791,7 @@ pub enum FormatPredicateKind {
 }
 
 #[derive(Clone, Hash, PartialEq, Eq)]
-pub struct FormatPredicateNode {
+pub(crate) struct FormatPredicateNode {
     format_name: &'static str,
     member_name: &'static str,
     kind: FormatPredicateKind,
@@ -879,7 +879,7 @@ impl FormatPredicateNode {
 }
 
 #[derive(Clone, Hash, PartialEq, Eq)]
-pub enum TypePredicateNode {
+pub(crate) enum TypePredicateNode {
     /// Is the value argument (at the index designated by the first member) the same type as the
     /// type name (second member)?
     TypeVarCheck(usize, String),
@@ -905,7 +905,7 @@ impl TypePredicateNode {
 
 /// A basic node in an instruction predicate: either an atom, or an AND of two conditions.
 #[derive(Clone, Hash, PartialEq, Eq)]
-pub enum InstructionPredicateNode {
+pub(crate) enum InstructionPredicateNode {
     FormatPredicate(FormatPredicateNode),
 
     TypePredicate(TypePredicateNode),
diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
index 6283b08736..9f821a7a6d 100644
--- a/cranelift/codegen/meta/src/cdsl/operands.rs
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -17,7 +17,7 @@ use crate::cdsl::typevar::TypeVar;
 /// 4. An `EntityRefKind` instance indicates an operand that references another entity in the
 ///    function, typically something declared in the function preamble.
 #[derive(Clone, Debug)]
-pub struct Operand {
+pub(crate) struct Operand {
     pub name: &'static str,
     pub doc: Option,
     pub kind: OperandKind,
@@ -75,7 +75,7 @@ impl Operand {
     }
 }
 
-pub struct OperandBuilder {
+pub(crate) struct OperandBuilder {
     name: &'static str,
     doc: Option,
     kind: OperandKind,
@@ -114,7 +114,7 @@ impl OperandBuilder {
 type EnumValues = HashMap<&'static str, &'static str>;
 
 #[derive(Clone, Debug)]
-pub enum OperandKindFields {
+pub(crate) enum OperandKindFields {
     EntityRef,
     VariableArgs,
     ImmValue,
@@ -123,7 +123,7 @@ pub enum OperandKindFields {
 }
 
 #[derive(Clone, Debug)]
-pub struct OperandKind {
+pub(crate) struct OperandKind {
     pub name: &'static str,
 
     doc: Option,
@@ -145,16 +145,9 @@ impl OperandKind {
             _ => None,
         }
     }
-
-    pub fn type_var(&self) -> TypeVar {
-        match &self.fields {
-            OperandKindFields::TypeVar(tvar) => tvar.clone(),
-            _ => panic!("not a typevar"),
-        }
-    }
 }
 
-pub struct OperandKindBuilder {
+pub(crate) struct OperandKindBuilder {
     name: &'static str,
 
     doc: Option,
@@ -270,12 +263,12 @@ impl Into for &OperandKind {
 }
 
 /// Helper to create an operand in definitions files.
-pub fn create_operand(name: &'static str, kind: impl Into) -> Operand {
+pub(crate) fn create_operand(name: &'static str, kind: impl Into) -> Operand {
     OperandBuilder::new(name, kind.into()).build()
 }
 
 /// Helper to create an operand with a documentation in definitions files.
-pub fn create_operand_doc(
+pub(crate) fn create_operand_doc(
     name: &'static str,
     kind: impl Into,
     doc: &'static str,
diff --git a/cranelift/codegen/meta/src/cdsl/recipes.rs b/cranelift/codegen/meta/src/cdsl/recipes.rs
index 4aad4debe2..9bb966f605 100644
--- a/cranelift/codegen/meta/src/cdsl/recipes.rs
+++ b/cranelift/codegen/meta/src/cdsl/recipes.rs
@@ -18,7 +18,7 @@ use crate::cdsl::settings::SettingPredicateNumber;
 /// Register instances can be created with the constructor, or accessed as
 /// attributes on the register class: `GPR.rcx`.
 #[derive(Copy, Clone, Hash, PartialEq, Eq)]
-pub struct Register {
+pub(crate) struct Register {
     pub regclass: RegClassIndex,
     pub unit: u8,
 }
@@ -34,7 +34,7 @@ impl Register {
 /// A `Stack` object can be used to indicate an operand constraint for a value
 /// operand that must live in a stack slot.
 #[derive(Copy, Clone, Hash, PartialEq)]
-pub struct Stack {
+pub(crate) struct Stack {
     pub regclass: RegClassIndex,
 }
 
@@ -49,13 +49,13 @@ impl Stack {
 }
 
 #[derive(Clone, Hash, PartialEq)]
-pub struct BranchRange {
+pub(crate) struct BranchRange {
     pub inst_size: u64,
     pub range: u64,
 }
 
 #[derive(Copy, Clone, Hash, PartialEq)]
-pub enum OperandConstraint {
+pub(crate) enum OperandConstraint {
     RegClass(RegClassIndex),
     FixedReg(Register),
     TiedInput(usize),
diff --git a/cranelift/codegen/meta/src/cdsl/regs.rs b/cranelift/codegen/meta/src/cdsl/regs.rs
index 69efe6b660..98a5751f2e 100644
--- a/cranelift/codegen/meta/src/cdsl/regs.rs
+++ b/cranelift/codegen/meta/src/cdsl/regs.rs
@@ -2,10 +2,10 @@ use cranelift_codegen_shared::constants;
 use cranelift_entity::{entity_impl, EntityRef, PrimaryMap};
 
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
-pub struct RegBankIndex(u32);
+pub(crate) struct RegBankIndex(u32);
 entity_impl!(RegBankIndex);
 
-pub struct RegBank {
+pub(crate) struct RegBank {
     pub name: &'static str,
     pub first_unit: u8,
     pub units: u8,
@@ -73,10 +73,10 @@ impl RegBank {
 }
 
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
-pub struct RegClassIndex(u32);
+pub(crate) struct RegClassIndex(u32);
 entity_impl!(RegClassIndex);
 
-pub struct RegClass {
+pub(crate) struct RegClass {
     pub name: &'static str,
     pub index: RegClassIndex,
     pub width: u8,
@@ -130,12 +130,12 @@ impl RegClass {
     }
 }
 
-pub enum RegClassProto {
+pub(crate) enum RegClassProto {
     TopLevel(RegBankIndex),
     SubClass(RegClassIndex),
 }
 
-pub struct RegClassBuilder {
+pub(crate) struct RegClassBuilder {
     pub name: &'static str,
     pub width: u8,
     pub count: u8,
@@ -181,7 +181,7 @@ impl RegClassBuilder {
     }
 }
 
-pub struct RegBankBuilder {
+pub(crate) struct RegBankBuilder {
     pub name: &'static str,
     pub units: u8,
     pub names: Vec<&'static str>,
@@ -220,7 +220,7 @@ impl RegBankBuilder {
     }
 }
 
-pub struct IsaRegsBuilder {
+pub(crate) struct IsaRegsBuilder {
     pub banks: PrimaryMap,
     pub classes: PrimaryMap,
 }
@@ -384,7 +384,7 @@ impl IsaRegsBuilder {
     }
 }
 
-pub struct IsaRegs {
+pub(crate) struct IsaRegs {
     pub banks: PrimaryMap,
     pub classes: PrimaryMap,
 }
diff --git a/cranelift/codegen/meta/src/cdsl/settings.rs b/cranelift/codegen/meta/src/cdsl/settings.rs
index 3d56fa9839..217bad9955 100644
--- a/cranelift/codegen/meta/src/cdsl/settings.rs
+++ b/cranelift/codegen/meta/src/cdsl/settings.rs
@@ -1,24 +1,24 @@
 use std::iter;
 
 #[derive(Clone, Copy, Hash, PartialEq, Eq)]
-pub struct BoolSettingIndex(usize);
+pub(crate) struct BoolSettingIndex(usize);
 
 #[derive(Hash, PartialEq, Eq)]
-pub struct BoolSetting {
+pub(crate) struct BoolSetting {
     pub default: bool,
     pub bit_offset: u8,
     pub predicate_number: u8,
 }
 
 #[derive(Hash, PartialEq, Eq)]
-pub enum SpecificSetting {
+pub(crate) enum SpecificSetting {
     Bool(BoolSetting),
     Enum(Vec<&'static str>),
     Num(u8),
 }
 
 #[derive(Hash, PartialEq, Eq)]
-pub struct Setting {
+pub(crate) struct Setting {
     pub name: &'static str,
     pub comment: &'static str,
     pub specific: SpecificSetting,
@@ -66,10 +66,10 @@ impl Setting {
 }
 
 #[derive(Hash, PartialEq, Eq)]
-pub struct PresetIndex(usize);
+pub(crate) struct PresetIndex(usize);
 
 #[derive(Hash, PartialEq, Eq)]
-pub enum PresetType {
+pub(crate) enum PresetType {
     BoolSetting(BoolSettingIndex),
     OtherPreset(PresetIndex),
 }
@@ -86,7 +86,7 @@ impl Into for PresetIndex {
 }
 
 #[derive(Hash, PartialEq, Eq)]
-pub struct Preset {
+pub(crate) struct Preset {
     pub name: &'static str,
     values: Vec,
 }
@@ -110,7 +110,7 @@ impl Preset {
     }
 }
 
-pub struct SettingGroup {
+pub(crate) struct SettingGroup {
     pub name: &'static str,
     pub settings: Vec,
     pub bool_start_byte_offset: u8,
@@ -160,7 +160,7 @@ impl SettingGroup {
 
 /// This is the basic information needed to track the specific parts of a setting when building
 /// them.
-pub enum ProtoSpecificSetting {
+pub(crate) enum ProtoSpecificSetting {
     Bool(bool),
     Enum(Vec<&'static str>),
     Num(u8),
@@ -174,7 +174,7 @@ struct ProtoSetting {
 }
 
 #[derive(Hash, PartialEq, Eq)]
-pub enum PredicateNode {
+pub(crate) enum PredicateNode {
     OwnedBool(BoolSettingIndex),
     SharedBool(&'static str, &'static str),
     Not(Box),
@@ -217,9 +217,9 @@ struct ProtoPredicate {
     node: PredicateNode,
 }
 
-pub type SettingPredicateNumber = u8;
+pub(crate) type SettingPredicateNumber = u8;
 
-pub struct Predicate {
+pub(crate) struct Predicate {
     pub name: &'static str,
     node: PredicateNode,
     pub number: SettingPredicateNumber,
@@ -231,7 +231,7 @@ impl Predicate {
     }
 }
 
-pub struct SettingGroupBuilder {
+pub(crate) struct SettingGroupBuilder {
     name: &'static str,
     settings: Vec,
     presets: Vec,
diff --git a/cranelift/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
index 83faa538c3..9e0556a850 100644
--- a/cranelift/codegen/meta/src/cdsl/types.rs
+++ b/cranelift/codegen/meta/src/cdsl/types.rs
@@ -15,7 +15,7 @@ static _RUST_NAME_PREFIX: &str = "ir::types::";
 /// All SSA values have a type that is described by an instance of `ValueType`
 /// or one of its subclasses.
 #[derive(Clone, Debug, PartialEq, Eq, Hash)]
-pub enum ValueType {
+pub(crate) enum ValueType {
     BV(BVType),
     Lane(LaneType),
     Reference(ReferenceType),
@@ -151,7 +151,7 @@ impl From for ValueType {
 
 /// A concrete scalar type that can appear as a vector lane too.
 #[derive(Clone, Copy, PartialEq, Eq, Hash)]
-pub enum LaneType {
+pub(crate) enum LaneType {
     BoolType(shared_types::Bool),
     FloatType(shared_types::Float),
     IntType(shared_types::Int),
@@ -319,7 +319,7 @@ impl From for LaneType {
 }
 
 /// An iterator for different lane types.
-pub struct LaneTypeIterator {
+pub(crate) struct LaneTypeIterator {
     bool_iter: shared_types::BoolIterator,
     int_iter: shared_types::IntIterator,
     float_iter: shared_types::FloatIterator,
@@ -356,7 +356,7 @@ impl Iterator for LaneTypeIterator {
 /// A vector type has a lane type which is an instance of `LaneType`,
 /// and a positive number of lanes.
 #[derive(Clone, PartialEq, Eq, Hash)]
-pub struct VectorType {
+pub(crate) struct VectorType {
     base: LaneType,
     lanes: u64,
 }
@@ -422,7 +422,7 @@ impl fmt::Debug for VectorType {
 
 /// A flat bitvector type. Used for semantics description only.
 #[derive(Clone, PartialEq, Eq, Hash)]
-pub struct BVType {
+pub(crate) struct BVType {
     bits: u64,
 }
 
@@ -459,7 +459,7 @@ impl fmt::Debug for BVType {
 ///
 /// Special types cannot be used to form vectors.
 #[derive(Clone, Copy, PartialEq, Eq, Hash)]
-pub enum SpecialType {
+pub(crate) enum SpecialType {
     Flag(shared_types::Flag),
 }
 
@@ -521,7 +521,7 @@ impl From for SpecialType {
     }
 }
 
-pub struct SpecialTypeIterator {
+pub(crate) struct SpecialTypeIterator {
     flag_iter: shared_types::FlagIterator,
 }
 
@@ -546,7 +546,7 @@ impl Iterator for SpecialTypeIterator {
 
 /// Reference type is scalar type, but not lane type.
 #[derive(Clone, Copy, PartialEq, Eq, Hash)]
-pub struct ReferenceType(pub shared_types::Reference);
+pub(crate) struct ReferenceType(pub shared_types::Reference);
 
 impl ReferenceType {
     /// Return a string containing the documentation comment for this reference type.
@@ -600,7 +600,7 @@ impl From for ReferenceType {
 }
 
 /// An iterator for different reference types.
-pub struct ReferenceTypeIterator {
+pub(crate) struct ReferenceTypeIterator {
     reference_iter: shared_types::ReferenceIterator,
 }
 
diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 947d0dd82d..03fb52a4ad 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -21,7 +21,7 @@ const MAX_BITVEC: u16 = MAX_BITS * MAX_LANES;
 /// types and whether the type variable can assume scalar or vector types, or
 /// both.
 #[derive(Debug)]
-pub struct TypeVarContent {
+pub(crate) struct TypeVarContent {
     /// Short name of type variable used in instruction descriptions.
     pub name: String,
 
@@ -37,7 +37,7 @@ pub struct TypeVarContent {
 }
 
 #[derive(Clone, Debug)]
-pub struct TypeVar {
+pub(crate) struct TypeVar {
     content: Rc>,
 }
 
@@ -336,7 +336,7 @@ impl ops::Deref for TypeVar {
 }
 
 #[derive(Clone, Copy, Debug, Hash, PartialEq)]
-pub enum DerivedFunc {
+pub(crate) enum DerivedFunc {
     LaneOf,
     AsBool,
     HalfWidth,
@@ -372,7 +372,7 @@ impl DerivedFunc {
 }
 
 #[derive(Debug, Hash)]
-pub struct TypeVarParent {
+pub(crate) struct TypeVarParent {
     pub type_var: TypeVar,
     pub derived_func: DerivedFunc,
 }
@@ -406,7 +406,7 @@ macro_rules! num_set {
 }
 
 #[derive(Clone, PartialEq, Eq, Hash)]
-pub struct TypeSet {
+pub(crate) struct TypeSet {
     pub lanes: NumSet,
     pub ints: NumSet,
     pub floats: NumSet,
@@ -784,7 +784,7 @@ impl fmt::Debug for TypeSet {
     }
 }
 
-pub struct TypeSetBuilder {
+pub(crate) struct TypeSetBuilder {
     ints: Interval,
     floats: Interval,
     bools: Interval,
@@ -883,7 +883,7 @@ impl TypeSetBuilder {
 }
 
 #[derive(PartialEq)]
-pub enum Interval {
+pub(crate) enum Interval {
     None,
     All,
     Range(Range),
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index b5d2454881..adf5589227 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -664,7 +664,7 @@ fn typeset_to_string(ts: &TypeSet) -> String {
 }
 
 /// Generate the table of ValueTypeSets described by type_sets.
-pub fn gen_typesets_table(type_sets: &UniqueTable, fmt: &mut Formatter) {
+pub(crate) fn gen_typesets_table(type_sets: &UniqueTable, fmt: &mut Formatter) {
     if type_sets.len() == 0 {
         return;
     }
diff --git a/cranelift/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
index f8b5792ef1..a40afbdf4a 100644
--- a/cranelift/codegen/meta/src/gen_settings.rs
+++ b/cranelift/codegen/meta/src/gen_settings.rs
@@ -10,7 +10,7 @@ use crate::error;
 use crate::srcgen::{Formatter, Match};
 use crate::unique_table::UniqueSeqTable;
 
-pub enum ParentGroup {
+pub(crate) enum ParentGroup {
     None,
     Shared,
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/registers.rs b/cranelift/codegen/meta/src/isa/x86/registers.rs
index 4157084c15..5c31401a3c 100644
--- a/cranelift/codegen/meta/src/isa/x86/registers.rs
+++ b/cranelift/codegen/meta/src/isa/x86/registers.rs
@@ -1,6 +1,6 @@
 use crate::cdsl::regs::{IsaRegs, IsaRegsBuilder, RegBankBuilder, RegClassBuilder};
 
-pub fn define() -> IsaRegs {
+pub(crate) fn define() -> IsaRegs {
     let mut regs = IsaRegsBuilder::new();
 
     let builder = RegBankBuilder::new("IntRegs", "r")
diff --git a/cranelift/codegen/meta/src/isa/x86/settings.rs b/cranelift/codegen/meta/src/isa/x86/settings.rs
index 10cb516207..bef8fc3f81 100644
--- a/cranelift/codegen/meta/src/isa/x86/settings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/settings.rs
@@ -1,6 +1,6 @@
 use crate::cdsl::settings::{PredicateNode, SettingGroup, SettingGroupBuilder};
 
-pub fn define(shared: &SettingGroup) -> SettingGroup {
+pub(crate) fn define(shared: &SettingGroup) -> SettingGroup {
     let mut settings = SettingGroupBuilder::new("x86");
 
     // CPUID.01H:ECX
diff --git a/cranelift/codegen/meta/src/shared/entities.rs b/cranelift/codegen/meta/src/shared/entities.rs
index 910722789b..631e0855d2 100644
--- a/cranelift/codegen/meta/src/shared/entities.rs
+++ b/cranelift/codegen/meta/src/shared/entities.rs
@@ -1,35 +1,35 @@
 use crate::cdsl::operands::{OperandKind, OperandKindBuilder as Builder, OperandKindFields};
 
-pub struct EntityRefs {
+pub(crate) struct EntityRefs {
     /// A reference to an extended basic block in the same function.
     /// This is primarliy used in control flow instructions.
-    pub ebb: OperandKind,
+    pub(crate) ebb: OperandKind,
 
     /// A reference to a stack slot declared in the function preamble.
-    pub stack_slot: OperandKind,
+    pub(crate) stack_slot: OperandKind,
 
     /// A reference to a global value.
-    pub global_value: OperandKind,
+    pub(crate) global_value: OperandKind,
 
     /// A reference to a function signature declared in the function preamble.
     /// This is used to provide the call signature in a call_indirect instruction.
-    pub sig_ref: OperandKind,
+    pub(crate) sig_ref: OperandKind,
 
     /// A reference to an external function declared in the function preamble.
     /// This is used to provide the callee and signature in a call instruction.
-    pub func_ref: OperandKind,
+    pub(crate) func_ref: OperandKind,
 
     /// A reference to a jump table declared in the function preamble.
-    pub jump_table: OperandKind,
+    pub(crate) jump_table: OperandKind,
 
     /// A reference to a heap declared in the function preamble.
-    pub heap: OperandKind,
+    pub(crate) heap: OperandKind,
 
     /// A reference to a table declared in the function preamble.
-    pub table: OperandKind,
+    pub(crate) table: OperandKind,
 
     /// A variable-sized list of value operands. Use for Ebb and function call arguments.
-    pub varargs: OperandKind,
+    pub(crate) varargs: OperandKind,
 }
 
 impl EntityRefs {
diff --git a/cranelift/codegen/meta/src/shared/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
index 43fe357ab7..e375a8e66e 100644
--- a/cranelift/codegen/meta/src/shared/settings.rs
+++ b/cranelift/codegen/meta/src/shared/settings.rs
@@ -1,6 +1,6 @@
 use crate::cdsl::settings::{SettingGroup, SettingGroupBuilder};
 
-pub fn define() -> SettingGroup {
+pub(crate) fn define() -> SettingGroup {
     let mut settings = SettingGroupBuilder::new("shared");
 
     settings.add_enum(
diff --git a/cranelift/codegen/meta/src/shared/types.rs b/cranelift/codegen/meta/src/shared/types.rs
index 52fa9545c6..631e5433e9 100644
--- a/cranelift/codegen/meta/src/shared/types.rs
+++ b/cranelift/codegen/meta/src/shared/types.rs
@@ -1,7 +1,7 @@
 //! This module predefines all the Cranelift scalar types.
 
 #[derive(Debug, Clone, Copy, Eq, PartialEq, Hash)]
-pub enum Bool {
+pub(crate) enum Bool {
     /// 1-bit bool.
     B1 = 1,
     /// 8-bit bool.
@@ -17,7 +17,7 @@ pub enum Bool {
 }
 
 /// This provides an iterator through all of the supported bool variants.
-pub struct BoolIterator {
+pub(crate) struct BoolIterator {
     index: u8,
 }
 
@@ -45,7 +45,7 @@ impl Iterator for BoolIterator {
 }
 
 #[derive(Debug, Clone, Copy, Eq, PartialEq, Hash)]
-pub enum Int {
+pub(crate) enum Int {
     /// 8-bit int.
     I8 = 8,
     /// 16-bit int.
@@ -59,7 +59,7 @@ pub enum Int {
 }
 
 /// This provides an iterator through all of the supported int variants.
-pub struct IntIterator {
+pub(crate) struct IntIterator {
     index: u8,
 }
 
@@ -86,13 +86,13 @@ impl Iterator for IntIterator {
 }
 
 #[derive(Debug, Clone, Copy, Eq, PartialEq, Hash)]
-pub enum Float {
+pub(crate) enum Float {
     F32 = 32,
     F64 = 64,
 }
 
 /// Iterator through the variants of the Float enum.
-pub struct FloatIterator {
+pub(crate) struct FloatIterator {
     index: u8,
 }
 
@@ -120,7 +120,7 @@ impl Iterator for FloatIterator {
 ///
 /// Flags can't be stored in memory.
 #[derive(Debug, Clone, Copy, Eq, PartialEq, Hash)]
-pub enum Flag {
+pub(crate) enum Flag {
     /// CPU flags from an integer comparison.
     IFlags,
     /// CPU flags from a floating point comparison.
@@ -128,7 +128,7 @@ pub enum Flag {
 }
 
 /// Iterator through the variants of the Flag enum.
-pub struct FlagIterator {
+pub(crate) struct FlagIterator {
     index: u8,
 }
 
@@ -152,7 +152,7 @@ impl Iterator for FlagIterator {
 }
 
 #[derive(Debug, Clone, Copy, Eq, PartialEq, Hash)]
-pub enum Reference {
+pub(crate) enum Reference {
     /// 32-bit reference.
     R32 = 32,
     /// 64-bit reference.
@@ -160,7 +160,7 @@ pub enum Reference {
 }
 
 /// This provides an iterator through all of the supported reference variants.
-pub struct ReferenceIterator {
+pub(crate) struct ReferenceIterator {
     index: u8,
 }
 
diff --git a/cranelift/codegen/meta/src/srcgen.rs b/cranelift/codegen/meta/src/srcgen.rs
index 5a11d1d785..ad8db175d7 100644
--- a/cranelift/codegen/meta/src/srcgen.rs
+++ b/cranelift/codegen/meta/src/srcgen.rs
@@ -35,7 +35,7 @@ macro_rules! fmtln {
     };
 }
 
-pub struct Formatter {
+pub(crate) struct Formatter {
     indent: usize,
     lines: Vec,
 }
@@ -260,7 +260,7 @@ fn parse_multiline(s: &str) -> Vec {
 /// Note that this class is ignorant of Rust types, and considers two fields
 /// with the same name to be equivalent. BTreeMap/BTreeSet are used to
 /// represent the arms in order to make the order deterministic.
-pub struct Match {
+pub(crate) struct Match {
     expr: String,
     arms: BTreeMap<(Vec, String), BTreeSet>,
     /// The clause for the placeholder pattern _.
diff --git a/cranelift/codegen/meta/src/unique_table.rs b/cranelift/codegen/meta/src/unique_table.rs
index 7bc694fe26..949aa6b9d3 100644
--- a/cranelift/codegen/meta/src/unique_table.rs
+++ b/cranelift/codegen/meta/src/unique_table.rs
@@ -3,7 +3,7 @@ use std::hash::Hash;
 use std::slice;
 
 /// Collect items into the `table` list, removing duplicates.
-pub struct UniqueTable<'entries, T: Eq + Hash> {
+pub(crate) struct UniqueTable<'entries, T: Eq + Hash> {
     table: Vec<&'entries T>,
     map: HashMap<&'entries T, usize>,
 }
@@ -40,7 +40,7 @@ impl<'entries, T: Eq + Hash> UniqueTable<'entries, T> {
 }
 
 /// A table of sequences which tries to avoid common subsequences.
-pub struct UniqueSeqTable {
+pub(crate) struct UniqueSeqTable {
     table: Vec,
 }
 

From e8c03fbd09d0a726022d85e32fbdcf957765fddf Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 28 Oct 2019 18:28:31 +0100
Subject: [PATCH 2883/3084] [meta] Remove unused bit-vector functionalty;

It can be resurrected if needed in the future. It was used only for the
semantics descriptions, which went away with the transition of the
meta-language to Rust.
---
 cranelift/codegen/meta/src/cdsl/types.rs   |  54 ---------
 cranelift/codegen/meta/src/cdsl/typevar.rs | 128 +--------------------
 cranelift/codegen/meta/src/gen_inst.rs     |   4 -
 3 files changed, 3 insertions(+), 183 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
index 9e0556a850..ffa0383693 100644
--- a/cranelift/codegen/meta/src/cdsl/types.rs
+++ b/cranelift/codegen/meta/src/cdsl/types.rs
@@ -16,7 +16,6 @@ static _RUST_NAME_PREFIX: &str = "ir::types::";
 /// or one of its subclasses.
 #[derive(Clone, Debug, PartialEq, Eq, Hash)]
 pub(crate) enum ValueType {
-    BV(BVType),
     Lane(LaneType),
     Reference(ReferenceType),
     Special(SpecialType),
@@ -41,7 +40,6 @@ impl ValueType {
     /// Return a string containing the documentation comment for this type.
     pub fn doc(&self) -> String {
         match *self {
-            ValueType::BV(ref b) => b.doc(),
             ValueType::Lane(l) => l.doc(),
             ValueType::Reference(r) => r.doc(),
             ValueType::Special(s) => s.doc(),
@@ -52,7 +50,6 @@ impl ValueType {
     /// Return the number of bits in a lane.
     pub fn lane_bits(&self) -> u64 {
         match *self {
-            ValueType::BV(ref b) => b.lane_bits(),
             ValueType::Lane(l) => l.lane_bits(),
             ValueType::Reference(r) => r.lane_bits(),
             ValueType::Special(s) => s.lane_bits(),
@@ -76,7 +73,6 @@ impl ValueType {
     /// Find the unique number associated with this type.
     pub fn number(&self) -> Option {
         match *self {
-            ValueType::BV(_) => None,
             ValueType::Lane(l) => Some(l.number()),
             ValueType::Reference(r) => Some(r.number()),
             ValueType::Special(s) => Some(s.number()),
@@ -105,7 +101,6 @@ impl ValueType {
 impl fmt::Display for ValueType {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
-            ValueType::BV(ref b) => b.fmt(f),
             ValueType::Lane(l) => l.fmt(f),
             ValueType::Reference(r) => r.fmt(f),
             ValueType::Special(s) => s.fmt(f),
@@ -114,13 +109,6 @@ impl fmt::Display for ValueType {
     }
 }
 
-/// Create a ValueType from a given bitvector type.
-impl From for ValueType {
-    fn from(bv: BVType) -> Self {
-        ValueType::BV(bv)
-    }
-}
-
 /// Create a ValueType from a given lane type.
 impl From for ValueType {
     fn from(lane: LaneType) -> Self {
@@ -263,13 +251,6 @@ impl LaneType {
             _ => false,
         }
     }
-
-    pub fn is_bool(self) -> bool {
-        match self {
-            LaneType::BoolType(_) => true,
-            _ => false,
-        }
-    }
 }
 
 impl fmt::Display for LaneType {
@@ -420,41 +401,6 @@ impl fmt::Debug for VectorType {
     }
 }
 
-/// A flat bitvector type. Used for semantics description only.
-#[derive(Clone, PartialEq, Eq, Hash)]
-pub(crate) struct BVType {
-    bits: u64,
-}
-
-impl BVType {
-    /// Initialize a new bitvector type with `n` bits.
-    pub fn new(bits: u16) -> Self {
-        Self { bits: bits.into() }
-    }
-
-    /// Return a string containing the documentation comment for this bitvector type.
-    pub fn doc(&self) -> String {
-        format!("A bitvector type with {} bits.", self.bits)
-    }
-
-    /// Return the number of bits in a lane.
-    pub fn lane_bits(&self) -> u64 {
-        self.bits
-    }
-}
-
-impl fmt::Display for BVType {
-    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        write!(f, "bv{}", self.bits)
-    }
-}
-
-impl fmt::Debug for BVType {
-    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        write!(f, "BVType(bits={})", self.lane_bits())
-    }
-}
-
 /// A concrete scalar type that is neither a vector nor a lane type.
 ///
 /// Special types cannot be used to form vectors.
diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 03fb52a4ad..3748656da3 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -6,12 +6,11 @@ use std::iter::FromIterator;
 use std::ops;
 use std::rc::Rc;
 
-use crate::cdsl::types::{BVType, LaneType, ReferenceType, SpecialType, ValueType};
+use crate::cdsl::types::{LaneType, ReferenceType, SpecialType, ValueType};
 
 const MAX_LANES: u16 = 256;
 const MAX_BITS: u16 = 128;
 const MAX_FLOAT_BITS: u16 = 64;
-const MAX_BITVEC: u16 = MAX_BITS * MAX_LANES;
 
 /// Type variables can be used in place of concrete types when defining
 /// instructions. This makes the instructions *polymorphic*.
@@ -58,10 +57,6 @@ impl TypeVar {
         let mut builder = TypeSetBuilder::new();
 
         let (scalar_type, num_lanes) = match value_type {
-            ValueType::BV(bitvec_type) => {
-                let bits = bitvec_type.lane_bits() as RangeBound;
-                return TypeVar::new(name, doc, builder.bitvecs(bits..bits).build());
-            }
             ValueType::Special(special_type) => {
                 return TypeVar::new(name, doc, builder.specials(vec![special_type]).build());
             }
@@ -198,9 +193,7 @@ impl TypeVar {
                     "can't double 256 lanes"
                 );
             }
-            DerivedFunc::LaneOf | DerivedFunc::ToBitVec | DerivedFunc::AsBool => {
-                /* no particular assertions */
-            }
+            DerivedFunc::LaneOf | DerivedFunc::AsBool => { /* no particular assertions */ }
         }
 
         TypeVar {
@@ -234,9 +227,6 @@ impl TypeVar {
     pub fn double_vector(&self) -> TypeVar {
         self.derived(DerivedFunc::DoubleVector)
     }
-    pub fn to_bitvec(&self) -> TypeVar {
-        self.derived(DerivedFunc::ToBitVec)
-    }
 
     /// Constrain the range of types this variable can assume to a subset of those in the typeset
     /// ts.
@@ -343,7 +333,6 @@ pub(crate) enum DerivedFunc {
     DoubleWidth,
     HalfVector,
     DoubleVector,
-    ToBitVec,
 }
 
 impl DerivedFunc {
@@ -355,7 +344,6 @@ impl DerivedFunc {
             DerivedFunc::DoubleWidth => "double_width",
             DerivedFunc::HalfVector => "half_vector",
             DerivedFunc::DoubleVector => "double_vector",
-            DerivedFunc::ToBitVec => "to_bitvec",
         }
     }
 
@@ -412,7 +400,6 @@ pub(crate) struct TypeSet {
     pub floats: NumSet,
     pub bools: NumSet,
     pub refs: NumSet,
-    pub bitvecs: NumSet,
     pub specials: Vec,
 }
 
@@ -423,7 +410,6 @@ impl TypeSet {
         floats: NumSet,
         bools: NumSet,
         refs: NumSet,
-        bitvecs: NumSet,
         specials: Vec,
     ) -> Self {
         Self {
@@ -432,7 +418,6 @@ impl TypeSet {
             floats,
             bools,
             refs,
-            bitvecs,
             specials,
         }
     }
@@ -440,11 +425,7 @@ impl TypeSet {
     /// Return the number of concrete types represented by this typeset.
     pub fn size(&self) -> usize {
         self.lanes.len()
-            * (self.ints.len()
-                + self.floats.len()
-                + self.bools.len()
-                + self.refs.len()
-                + self.bitvecs.len())
+            * (self.ints.len() + self.floats.len() + self.bools.len() + self.refs.len())
             + self.specials.len()
     }
 
@@ -457,7 +438,6 @@ impl TypeSet {
             DerivedFunc::DoubleWidth => self.double_width(),
             DerivedFunc::HalfVector => self.half_vector(),
             DerivedFunc::DoubleVector => self.double_vector(),
-            DerivedFunc::ToBitVec => self.to_bitvec(),
         }
     }
 
@@ -465,7 +445,6 @@ impl TypeSet {
     fn lane_of(&self) -> TypeSet {
         let mut copy = self.clone();
         copy.lanes = num_set![1];
-        copy.bitvecs = NumSet::new();
         copy
     }
 
@@ -475,7 +454,6 @@ impl TypeSet {
         copy.ints = NumSet::new();
         copy.floats = NumSet::new();
         copy.refs = NumSet::new();
-        copy.bitvecs = NumSet::new();
         if !(&self.lanes - &num_set![1]).is_empty() {
             copy.bools = &self.ints | &self.floats;
             copy.bools = ©.bools | &self.bools;
@@ -492,7 +470,6 @@ impl TypeSet {
         copy.ints = NumSet::from_iter(self.ints.iter().filter(|&&x| x > 8).map(|&x| x / 2));
         copy.floats = NumSet::from_iter(self.floats.iter().filter(|&&x| x > 32).map(|&x| x / 2));
         copy.bools = NumSet::from_iter(self.bools.iter().filter(|&&x| x > 8).map(|&x| x / 2));
-        copy.bitvecs = NumSet::from_iter(self.bitvecs.iter().filter(|&&x| x > 1).map(|&x| x / 2));
         copy.specials = Vec::new();
         copy
     }
@@ -514,12 +491,6 @@ impl TypeSet {
                 .map(|&x| x * 2)
                 .filter(|x| legal_bool(*x)),
         );
-        copy.bitvecs = NumSet::from_iter(
-            self.bitvecs
-                .iter()
-                .filter(|&&x| x < MAX_BITVEC)
-                .map(|&x| x * 2),
-        );
         copy.specials = Vec::new();
         copy
     }
@@ -527,7 +498,6 @@ impl TypeSet {
     /// Return a TypeSet describing the image of self across halfvector.
     fn half_vector(&self) -> TypeSet {
         let mut copy = self.clone();
-        copy.bitvecs = NumSet::new();
         copy.lanes = NumSet::from_iter(self.lanes.iter().filter(|&&x| x > 1).map(|&x| x / 2));
         copy.specials = Vec::new();
         copy
@@ -536,7 +506,6 @@ impl TypeSet {
     /// Return a TypeSet describing the image of self across doublevector.
     fn double_vector(&self) -> TypeSet {
         let mut copy = self.clone();
-        copy.bitvecs = NumSet::new();
         copy.lanes = NumSet::from_iter(
             self.lanes
                 .iter()
@@ -547,29 +516,6 @@ impl TypeSet {
         copy
     }
 
-    /// Return a TypeSet describing the image of self across to_bitvec.
-    fn to_bitvec(&self) -> TypeSet {
-        assert!(self.bitvecs.is_empty());
-        let all_scalars = &(&self.ints | &self.floats) | &self.bools;
-
-        let mut copy = self.clone();
-        copy.lanes = num_set![1];
-        copy.ints = NumSet::new();
-        copy.bools = NumSet::new();
-        copy.floats = NumSet::new();
-        copy.refs = NumSet::new();
-        copy.bitvecs = self
-            .lanes
-            .iter()
-            .cycle()
-            .zip(all_scalars.iter())
-            .map(|(num_lanes, lane_width)| num_lanes * lane_width)
-            .collect();
-
-        copy.specials = Vec::new();
-        copy
-    }
-
     fn concrete_types(&self) -> Vec {
         let mut ret = Vec::new();
         for &num_lanes in &self.lanes {
@@ -585,10 +531,6 @@ impl TypeSet {
             for &bits in &self.refs {
                 ret.push(ReferenceType::ref_from_bits(bits).into());
             }
-            for &bits in &self.bitvecs {
-                assert_eq!(num_lanes, 1);
-                ret.push(BVType::new(bits).into());
-            }
         }
         for &special in &self.specials {
             ret.push(special.into());
@@ -613,14 +555,12 @@ impl TypeSet {
         match func {
             DerivedFunc::LaneOf => {
                 let mut copy = self.clone();
-                copy.bitvecs = NumSet::new();
                 copy.lanes =
                     NumSet::from_iter((0..=MAX_LANES.trailing_zeros()).map(|i| u16::pow(2, i)));
                 copy
             }
             DerivedFunc::AsBool => {
                 let mut copy = self.clone();
-                copy.bitvecs = NumSet::new();
                 if self.bools.contains(&1) {
                     copy.ints = NumSet::from_iter(vec![8, 16, 32, 64, 128]);
                     copy.floats = NumSet::from_iter(vec![32, 64]);
@@ -637,43 +577,6 @@ impl TypeSet {
             DerivedFunc::DoubleWidth => self.half_width(),
             DerivedFunc::HalfVector => self.double_vector(),
             DerivedFunc::DoubleVector => self.half_vector(),
-            DerivedFunc::ToBitVec => {
-                let all_lanes = range_to_set(Some(1..MAX_LANES));
-                let all_ints = range_to_set(Some(8..MAX_BITS));
-                let all_floats = range_to_set(Some(32..64));
-                let all_bools = range_to_set(Some(1..MAX_BITS));
-
-                let mut lanes = range_to_set(Some(1..MAX_LANES));
-                let mut ints = range_to_set(Some(8..MAX_BITS));
-                let mut floats = range_to_set(Some(32..64));
-                let mut bools = range_to_set(Some(1..MAX_BITS));
-                let refs = range_to_set(Some(32..64));
-
-                for &l in &all_lanes {
-                    for &i in &all_ints {
-                        if self.bitvecs.contains(&(i * l)) {
-                            lanes.insert(l);
-                            ints.insert(i);
-                        }
-                    }
-                    for &f in &all_floats {
-                        if self.bitvecs.contains(&(f * l)) {
-                            lanes.insert(l);
-                            floats.insert(f);
-                        }
-                    }
-                    for &b in &all_bools {
-                        if self.bitvecs.contains(&(b * l)) {
-                            lanes.insert(l);
-                            bools.insert(b);
-                        }
-                    }
-                }
-
-                let bitvecs = NumSet::new();
-                let specials = Vec::new();
-                TypeSet::new(lanes, ints, floats, bools, refs, bitvecs, specials)
-            }
         }
     }
 
@@ -683,7 +586,6 @@ impl TypeSet {
         self.floats = &self.floats & &other.floats;
         self.bools = &self.bools & &other.bools;
         self.refs = &self.refs & &other.refs;
-        self.bitvecs = &self.bitvecs & &other.bitvecs;
 
         let mut new_specials = Vec::new();
         for spec in &self.specials {
@@ -700,7 +602,6 @@ impl TypeSet {
             && self.floats.is_subset(&other.floats)
             && self.bools.is_subset(&other.bools)
             && self.refs.is_subset(&other.refs)
-            && self.bitvecs.is_subset(&other.bitvecs)
             && {
                 let specials: HashSet = HashSet::from_iter(self.specials.clone());
                 let other_specials = HashSet::from_iter(other.specials.clone());
@@ -766,12 +667,6 @@ impl fmt::Debug for TypeSet {
                 Vec::from_iter(self.refs.iter().map(|x| x.to_string())).join(", ")
             ));
         }
-        if !self.bitvecs.is_empty() {
-            subsets.push(format!(
-                "bitvecs={{{}}}",
-                Vec::from_iter(self.bitvecs.iter().map(|x| x.to_string())).join(", ")
-            ));
-        }
         if !self.specials.is_empty() {
             subsets.push(format!(
                 "specials={{{}}}",
@@ -789,7 +684,6 @@ pub(crate) struct TypeSetBuilder {
     floats: Interval,
     bools: Interval,
     refs: Interval,
-    bitvecs: Interval,
     includes_scalars: bool,
     simd_lanes: Interval,
     specials: Vec,
@@ -802,7 +696,6 @@ impl TypeSetBuilder {
             floats: Interval::None,
             bools: Interval::None,
             refs: Interval::None,
-            bitvecs: Interval::None,
             includes_scalars: true,
             simd_lanes: Interval::None,
             specials: Vec::new(),
@@ -838,11 +731,6 @@ impl TypeSetBuilder {
         self.simd_lanes = interval.into();
         self
     }
-    pub fn bitvecs(mut self, interval: impl Into) -> Self {
-        assert!(self.bitvecs == Interval::None);
-        self.bitvecs = interval.into();
-        self
-    }
     pub fn specials(mut self, specials: Vec) -> Self {
         assert!(self.specials.is_empty());
         self.specials = specials;
@@ -863,7 +751,6 @@ impl TypeSetBuilder {
             range_to_set(self.floats.to_range(32..64, None)),
             bools,
             range_to_set(self.refs.to_range(32..64, None)),
-            range_to_set(self.bitvecs.to_range(1..MAX_BITVEC, None)),
             self.specials,
         )
     }
@@ -875,7 +762,6 @@ impl TypeSetBuilder {
             .bools(Interval::All)
             .refs(Interval::All)
             .simd_lanes(Interval::All)
-            .bitvecs(Interval::All)
             .specials(ValueType::all_special_types().collect())
             .includes_scalars(true)
             .build()
@@ -953,7 +839,6 @@ fn test_typevar_builder() {
     assert!(type_set.floats.is_empty());
     assert_eq!(type_set.ints, num_set![8, 16, 32, 64, 128]);
     assert!(type_set.bools.is_empty());
-    assert!(type_set.bitvecs.is_empty());
     assert!(type_set.specials.is_empty());
 
     let type_set = TypeSetBuilder::new().bools(Interval::All).build();
@@ -961,7 +846,6 @@ fn test_typevar_builder() {
     assert!(type_set.floats.is_empty());
     assert!(type_set.ints.is_empty());
     assert_eq!(type_set.bools, num_set![1, 8, 16, 32, 64, 128]);
-    assert!(type_set.bitvecs.is_empty());
     assert!(type_set.specials.is_empty());
 
     let type_set = TypeSetBuilder::new().floats(Interval::All).build();
@@ -969,7 +853,6 @@ fn test_typevar_builder() {
     assert_eq!(type_set.floats, num_set![32, 64]);
     assert!(type_set.ints.is_empty());
     assert!(type_set.bools.is_empty());
-    assert!(type_set.bitvecs.is_empty());
     assert!(type_set.specials.is_empty());
 
     let type_set = TypeSetBuilder::new()
@@ -981,7 +864,6 @@ fn test_typevar_builder() {
     assert_eq!(type_set.floats, num_set![32, 64]);
     assert!(type_set.ints.is_empty());
     assert!(type_set.bools.is_empty());
-    assert!(type_set.bitvecs.is_empty());
     assert!(type_set.specials.is_empty());
 
     let type_set = TypeSetBuilder::new()
@@ -993,7 +875,6 @@ fn test_typevar_builder() {
     assert_eq!(type_set.floats, num_set![32, 64]);
     assert!(type_set.ints.is_empty());
     assert!(type_set.bools.is_empty());
-    assert!(type_set.bitvecs.is_empty());
     assert!(type_set.specials.is_empty());
 
     let type_set = TypeSetBuilder::new().ints(16..64).build();
@@ -1001,7 +882,6 @@ fn test_typevar_builder() {
     assert_eq!(type_set.ints, num_set![16, 32, 64]);
     assert!(type_set.floats.is_empty());
     assert!(type_set.bools.is_empty());
-    assert!(type_set.bitvecs.is_empty());
     assert!(type_set.specials.is_empty());
 }
 
@@ -1326,7 +1206,6 @@ fn test_typevar_singleton() {
     assert_eq!(typevar.type_set.ints, num_set![32]);
     assert!(typevar.type_set.floats.is_empty());
     assert!(typevar.type_set.bools.is_empty());
-    assert!(typevar.type_set.bitvecs.is_empty());
     assert!(typevar.type_set.specials.is_empty());
     assert_eq!(typevar.type_set.lanes, num_set![1]);
 
@@ -1340,6 +1219,5 @@ fn test_typevar_singleton() {
     assert_eq!(typevar.type_set.floats, num_set![32]);
     assert_eq!(typevar.type_set.lanes, num_set![4]);
     assert!(typevar.type_set.bools.is_empty());
-    assert!(typevar.type_set.bitvecs.is_empty());
     assert!(typevar.type_set.specials.is_empty());
 }
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index adf5589227..85f8063473 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -650,9 +650,6 @@ fn typeset_to_string(ts: &TypeSet) -> String {
     if !ts.bools.is_empty() {
         result += &format!(", bools={}", iterable_to_string(&ts.bools));
     }
-    if !ts.bitvecs.is_empty() {
-        result += &format!(", bitvecs={}", iterable_to_string(&ts.bitvecs));
-    }
     if !ts.specials.is_empty() {
         result += &format!(", specials=[{}]", iterable_to_string(&ts.specials));
     }
@@ -680,7 +677,6 @@ pub(crate) fn gen_typesets_table(type_sets: &UniqueTable, fmt: &mut For
         for ts in type_sets.iter() {
             fmt.line("ir::instructions::ValueTypeSet {");
             fmt.indent(|fmt| {
-                assert!(ts.bitvecs.is_empty(), "Bitvector types are not emittable.");
                 fmt.comment(typeset_to_string(ts));
                 gen_bitset(&ts.lanes, "lanes", 16, fmt);
                 gen_bitset(&ts.ints, "ints", 8, fmt);

From 7e5c33a29ee8730cec36345402f95e5b3b1b9f3e Mon Sep 17 00:00:00 2001
From: lzutao 
Date: Wed, 30 Oct 2019 15:06:54 +0700
Subject: [PATCH 2884/3084] Use array::iter

---
 cranelift/codegen/build.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/build.rs b/cranelift/codegen/build.rs
index ef89d48ffe..f2afc404e6 100644
--- a/cranelift/codegen/build.rs
+++ b/cranelift/codegen/build.rs
@@ -28,7 +28,7 @@ fn main() {
 
     // Configure isa targets cfg.
     let isa_targets = meta::isa::Isa::all()
-        .into_iter()
+        .iter()
         .cloned()
         .filter(|isa| {
             let env_key = format!("CARGO_FEATURE_{}", isa.to_string().to_uppercase());

From 689771caf3a28f3ff67e7cccc185e8d6afbb1d22 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Tue, 29 Oct 2019 20:36:03 +0100
Subject: [PATCH 2885/3084] Don't force enable basic-blocks feature

Fixes #1179
---
 cranelift/faerie/Cargo.toml    | 7 ++++++-
 cranelift/object/Cargo.toml    | 7 ++++++-
 cranelift/simplejit/Cargo.toml | 9 +++++++--
 3 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index c5b471da1e..db77e2e0e5 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -10,13 +10,18 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1" }
 cranelift-module = { path = "../cranelift-module", version = "0.46.1" }
 faerie = "0.11.0"
 goblin = "0.0.24"
 failure = "0.1.2"
 target-lexicon = "0.8.1"
 
+[dependencies.cranelift-codegen]
+path = "../cranelift-codegen"
+version = "0.46.1"
+default-features = false
+features = ["std"]
+
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 8ad641984f..8577a0bfa7 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -10,11 +10,16 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1" }
 cranelift-module = { path = "../cranelift-module", version = "0.46.1" }
 object = { version = "0.14.0", default-features = false, features = ["write"] }
 target-lexicon = "0.8.1"
 
+[dependencies.cranelift-codegen]
+path = "../cranelift-codegen"
+version = "0.46.1"
+default-features = false
+features = ["std"]
+
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 637a08cb90..0ff896d3bc 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -10,14 +10,19 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1" }
 cranelift-module = { path = "../cranelift-module", version = "0.46.1" }
 cranelift-native = { path = "../cranelift-native", version = "0.46.1" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
 target-lexicon = "0.8.1"
-memmap = { version = "0.7.0", optional = true } 
+memmap = { version = "0.7.0", optional = true }
+
+[dependencies.cranelift-codegen]
+path = "../cranelift-codegen"
+version = "0.46.1"
+default-features = false
+features = ["std"]
 
 [target.'cfg(target_os = "windows")'.dependencies]
 winapi = { version = "0.3", features = ["winbase", "memoryapi"] }

From fcf0ad1d5d8368cac7fd5fda56967181708658c4 Mon Sep 17 00:00:00 2001
From: Josh Triplett 
Date: Wed, 30 Oct 2019 07:35:55 -0700
Subject: [PATCH 2886/3084] Log function definition unconditionally

Don't log inside the `.map_err` call, which resulted in only logging
function definitions on codegen errors.
---
 cranelift/module/src/module.rs | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 808dd88164..f45db2e79e 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -547,14 +547,14 @@ where
         func: FuncId,
         ctx: &mut Context,
     ) -> ModuleResult {
-        let CodeInfo { total_size, .. } = ctx.compile(self.backend.isa()).map_err(|e| {
-            info!(
-                "defining function {}: {}",
-                func,
-                ctx.func.display(self.backend.isa())
-            );
-            ModuleError::Compilation(e)
-        })?;
+        info!(
+            "defining function {}: {}",
+            func,
+            ctx.func.display(self.backend.isa())
+        );
+        let CodeInfo { total_size, .. } = ctx
+            .compile(self.backend.isa())
+            .map_err(ModuleError::Compilation)?;
         let info = &self.contents.functions[func];
         if info.compiled.is_some() {
             return Err(ModuleError::DuplicateDefinition(info.decl.name.clone()));

From 0b8a579943b781c9d9ab1e49edce6d7a7ed6eee6 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 29 Oct 2019 14:31:56 +0100
Subject: [PATCH 2887/3084] Fixes #851: Document instructions' input operands
 in InstBuilder;

---
 cranelift/codegen/meta/src/gen_inst.rs | 47 +++++++++++++++++++++++++-
 1 file changed, 46 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index 85f8063473..7abca6a188 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -895,11 +895,18 @@ fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Fo
     }
     .to_string()];
 
+    let mut args_doc = Vec::new();
+    let mut rets_doc = Vec::new();
+
     // The controlling type variable will be inferred from the input values if
     // possible. Otherwise, it is the first method argument.
     if let Some(poly) = &inst.polymorphic_info {
         if !poly.use_typevar_operand {
             args.push(format!("{}: crate::ir::Type", poly.ctrl_typevar.name));
+            args_doc.push(format!(
+                "- {} (controlling type variable): {}",
+                poly.ctrl_typevar.name, poly.ctrl_typevar.doc
+            ));
         }
     }
 
@@ -915,6 +922,23 @@ fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Fo
             op.kind.rust_type.clone()
         };
         args.push(format!("{}: {}", op.name, t));
+        args_doc.push(format!(
+            "- {}: {}",
+            op.name,
+            op.doc
+                .as_ref()
+                .expect("every instruction's input operand must be documented")
+        ));
+    }
+
+    for op in &inst.operands_out {
+        rets_doc.push(format!(
+            "- {}: {}",
+            op.name,
+            op.doc
+                .as_ref()
+                .expect("every instruction's output operand must be documented")
+        ));
     }
 
     let rtype = match inst.value_results.len() {
@@ -938,6 +962,23 @@ fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Fo
     );
 
     fmt.doc_comment(&inst.doc);
+    if !args_doc.is_empty() {
+        fmt.line("///");
+        fmt.doc_comment("Inputs:");
+        fmt.line("///");
+        for doc_line in args_doc {
+            fmt.doc_comment(doc_line);
+        }
+    }
+    if !rets_doc.is_empty() {
+        fmt.line("///");
+        fmt.doc_comment("Outputs:");
+        fmt.line("///");
+        for doc_line in rets_doc {
+            fmt.doc_comment(doc_line);
+        }
+    }
+
     fmt.line("#[allow(non_snake_case)]");
     fmtln!(fmt, "fn {} {{", proto);
     fmt.indent(|fmt| {
@@ -1058,9 +1099,13 @@ fn gen_builder(
     fmt.indent(|fmt| {
         for inst in instructions.values() {
             gen_inst_builder(inst, &*inst.format, fmt);
+            fmt.empty_line();
         }
-        for format in formats {
+        for (i, format) in formats.iter().enumerate() {
             gen_format_constructor(format, fmt);
+            if i + 1 != formats.len() {
+                fmt.empty_line();
+            }
         }
     });
     fmt.line("}");

From 2bebc40c1634b15327d23dd77fed93c5224d9282 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 29 Oct 2019 11:50:34 +0100
Subject: [PATCH 2888/3084] [meta] Move the doc() default values in the
 Operand/OperandKind;

---
 cranelift/codegen/meta/src/cdsl/operands.rs | 48 +++++++++++----------
 cranelift/codegen/meta/src/gen_inst.rs      |  6 +--
 2 files changed, 28 insertions(+), 26 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
index 9f821a7a6d..57968326c6 100644
--- a/cranelift/codegen/meta/src/cdsl/operands.rs
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -19,11 +19,21 @@ use crate::cdsl::typevar::TypeVar;
 #[derive(Clone, Debug)]
 pub(crate) struct Operand {
     pub name: &'static str,
-    pub doc: Option,
+    doc: Option,
     pub kind: OperandKind,
 }
 
 impl Operand {
+    pub fn doc(&self) -> Option<&str> {
+        match &self.doc {
+            Some(doc) => Some(doc),
+            None => match &self.kind.fields {
+                OperandKindFields::TypeVar(tvar) => Some(&tvar.doc),
+                _ => self.kind.doc(),
+            },
+        }
+    }
+
     pub fn is_value(&self) -> bool {
         match self.kind.fields {
             OperandKindFields::TypeVar(_) => true,
@@ -95,17 +105,9 @@ impl OperandBuilder {
         self
     }
     pub fn build(self) -> Operand {
-        let doc = match self.doc {
-            Some(doc) => Some(doc),
-            None => match &self.kind.fields {
-                OperandKindFields::TypeVar(tvar) => Some(tvar.doc.clone()),
-                _ => self.kind.doc.clone(),
-            },
-        };
-
         Operand {
             name: self.name,
-            doc,
+            doc: self.doc,
             kind: self.kind,
         }
     }
@@ -137,6 +139,19 @@ pub(crate) struct OperandKind {
 }
 
 impl OperandKind {
+    fn doc(&self) -> Option<&str> {
+        match &self.doc {
+            Some(doc) => Some(&doc),
+            None => match &self.fields {
+                OperandKindFields::TypeVar(type_var) => Some(&type_var.doc),
+                OperandKindFields::ImmEnum(_)
+                | OperandKindFields::ImmValue
+                | OperandKindFields::EntityRef
+                | OperandKindFields::VariableArgs => None,
+            },
+        }
+    }
+
     pub fn imm_name(&self) -> Option<&str> {
         match self.fields {
             OperandKindFields::ImmEnum(_)
@@ -230,20 +245,9 @@ impl OperandKindBuilder {
             },
         };
 
-        let doc = match self.doc {
-            Some(doc) => Some(doc),
-            None => match &self.fields {
-                OperandKindFields::TypeVar(type_var) => Some(type_var.doc.clone()),
-                OperandKindFields::ImmEnum(_)
-                | OperandKindFields::ImmValue
-                | OperandKindFields::EntityRef
-                | OperandKindFields::VariableArgs => None,
-            },
-        };
-
         OperandKind {
             name: self.name,
-            doc,
+            doc: self.doc,
             default_member,
             rust_type,
             fields: self.fields,
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index 7abca6a188..c32b21b738 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -925,8 +925,7 @@ fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Fo
         args_doc.push(format!(
             "- {}: {}",
             op.name,
-            op.doc
-                .as_ref()
+            op.doc()
                 .expect("every instruction's input operand must be documented")
         ));
     }
@@ -935,8 +934,7 @@ fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Fo
         rets_doc.push(format!(
             "- {}: {}",
             op.name,
-            op.doc
-                .as_ref()
+            op.doc()
                 .expect("every instruction's output operand must be documented")
         ));
     }

From 4632d35196381b2ff56517f74aea1e996e57b12a Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 29 Oct 2019 14:44:25 +0100
Subject: [PATCH 2889/3084] [meta] Remove the OperandBuilder, replace it with
 Operand ctors;

---
 .../codegen/meta/src/cdsl/instructions.rs     |   4 +-
 cranelift/codegen/meta/src/cdsl/operands.rs   |  86 ++--
 .../codegen/meta/src/isa/x86/instructions.rs  |  84 ++--
 .../codegen/meta/src/shared/instructions.rs   | 399 +++++++++---------
 4 files changed, 263 insertions(+), 310 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index a00d20a386..a804efca7c 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -1292,7 +1292,7 @@ impl Into for BoundInstruction {
 mod test {
     use super::*;
     use crate::cdsl::formats::InstructionFormatBuilder;
-    use crate::cdsl::operands::{OperandBuilder, OperandKindBuilder, OperandKindFields};
+    use crate::cdsl::operands::{OperandKindBuilder, OperandKindFields};
     use crate::cdsl::typevar::TypeSetBuilder;
     use crate::shared::types::Int::{I32, I64};
 
@@ -1300,7 +1300,7 @@ mod test {
         // Pretend the index string is &'static.
         let name = Box::leak(index.to_string().into_boxed_str());
         let kind = OperandKindBuilder::new(name, field).build();
-        let operand = OperandBuilder::new(name, kind).build();
+        let operand = Operand::new(name, kind);
         operand
     }
 
diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
index 57968326c6..f8403bf910 100644
--- a/cranelift/codegen/meta/src/cdsl/operands.rs
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -19,18 +19,30 @@ use crate::cdsl::typevar::TypeVar;
 #[derive(Clone, Debug)]
 pub(crate) struct Operand {
     pub name: &'static str,
-    doc: Option,
+    doc: Option<&'static str>,
     pub kind: OperandKind,
 }
 
 impl Operand {
+    pub fn new(name: &'static str, kind: impl Into) -> Self {
+        Self {
+            name,
+            doc: None,
+            kind: kind.into(),
+        }
+    }
+    pub fn with_doc(mut self, doc: &'static str) -> Self {
+        self.doc = Some(doc);
+        self
+    }
+
     pub fn doc(&self) -> Option<&str> {
-        match &self.doc {
-            Some(doc) => Some(doc),
-            None => match &self.kind.fields {
-                OperandKindFields::TypeVar(tvar) => Some(&tvar.doc),
-                _ => self.kind.doc(),
-            },
+        if let Some(doc) = &self.doc {
+            return Some(doc);
+        }
+        match &self.kind.fields {
+            OperandKindFields::TypeVar(tvar) => Some(&tvar.doc),
+            _ => self.kind.doc(),
         }
     }
 
@@ -85,34 +97,6 @@ impl Operand {
     }
 }
 
-pub(crate) struct OperandBuilder {
-    name: &'static str,
-    doc: Option,
-    kind: OperandKind,
-}
-
-impl OperandBuilder {
-    pub fn new(name: &'static str, kind: OperandKind) -> Self {
-        Self {
-            name,
-            doc: None,
-            kind,
-        }
-    }
-    pub fn doc(mut self, doc: impl Into) -> Self {
-        assert!(self.doc.is_none());
-        self.doc = Some(doc.into());
-        self
-    }
-    pub fn build(self) -> Operand {
-        Operand {
-            name: self.name,
-            doc: self.doc,
-            kind: self.kind,
-        }
-    }
-}
-
 type EnumValues = HashMap<&'static str, &'static str>;
 
 #[derive(Clone, Debug)]
@@ -140,15 +124,15 @@ pub(crate) struct OperandKind {
 
 impl OperandKind {
     fn doc(&self) -> Option<&str> {
-        match &self.doc {
-            Some(doc) => Some(&doc),
-            None => match &self.fields {
-                OperandKindFields::TypeVar(type_var) => Some(&type_var.doc),
-                OperandKindFields::ImmEnum(_)
-                | OperandKindFields::ImmValue
-                | OperandKindFields::EntityRef
-                | OperandKindFields::VariableArgs => None,
-            },
+        if let Some(doc) = &self.doc {
+            return Some(doc);
+        }
+        match &self.fields {
+            OperandKindFields::TypeVar(type_var) => Some(&type_var.doc),
+            OperandKindFields::ImmEnum(_)
+            | OperandKindFields::ImmValue
+            | OperandKindFields::EntityRef
+            | OperandKindFields::VariableArgs => None,
         }
     }
 
@@ -265,17 +249,3 @@ impl Into for &OperandKind {
         self.clone()
     }
 }
-
-/// Helper to create an operand in definitions files.
-pub(crate) fn create_operand(name: &'static str, kind: impl Into) -> Operand {
-    OperandBuilder::new(name, kind.into()).build()
-}
-
-/// Helper to create an operand with a documentation in definitions files.
-pub(crate) fn create_operand_doc(
-    name: &'static str,
-    kind: impl Into,
-    doc: &'static str,
-) -> Operand {
-    OperandBuilder::new(name, kind.into()).doc(doc).build()
-}
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 2ed57e6e67..7b77ceaf42 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -3,7 +3,7 @@
 use crate::cdsl::instructions::{
     AllInstructions, InstructionBuilder as Inst, InstructionGroup, InstructionGroupBuilder,
 };
-use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
+use crate::cdsl::operands::Operand;
 use crate::cdsl::types::ValueType;
 use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
 
@@ -26,11 +26,11 @@ pub(crate) fn define(
         "A scalar integer machine word",
         TypeSetBuilder::new().ints(32..64).build(),
     );
-    let nlo = &operand_doc("nlo", iWord, "Low part of numerator");
-    let nhi = &operand_doc("nhi", iWord, "High part of numerator");
-    let d = &operand_doc("d", iWord, "Denominator");
-    let q = &operand_doc("q", iWord, "Quotient");
-    let r = &operand_doc("r", iWord, "Remainder");
+    let nlo = &Operand::new("nlo", iWord).with_doc("Low part of numerator");
+    let nhi = &Operand::new("nhi", iWord).with_doc("High part of numerator");
+    let d = &Operand::new("d", iWord).with_doc("Denominator");
+    let q = &Operand::new("q", iWord).with_doc("Quotient");
+    let r = &Operand::new("r", iWord).with_doc("Remainder");
 
     ig.push(
         Inst::new(
@@ -72,10 +72,10 @@ pub(crate) fn define(
         .can_trap(true),
     );
 
-    let argL = &operand("argL", iWord);
-    let argR = &operand("argR", iWord);
-    let resLo = &operand("resLo", iWord);
-    let resHi = &operand("resHi", iWord);
+    let argL = &Operand::new("argL", iWord);
+    let argR = &Operand::new("argR", iWord);
+    let resLo = &Operand::new("resLo", iWord);
+    let resHi = &Operand::new("resHi", iWord);
 
     ig.push(
         Inst::new(
@@ -123,8 +123,8 @@ pub(crate) fn define(
             .simd_lanes(Interval::All)
             .build(),
     );
-    let x = &operand("x", Float);
-    let a = &operand("a", IntTo);
+    let x = &Operand::new("x", Float);
+    let a = &Operand::new("a", IntTo);
 
     ig.push(
         Inst::new(
@@ -144,9 +144,9 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let x = &operand("x", Float);
-    let a = &operand("a", Float);
-    let y = &operand("y", Float);
+    let x = &Operand::new("x", Float);
+    let a = &Operand::new("a", Float);
+    let y = &Operand::new("y", Float);
 
     ig.push(
         Inst::new(
@@ -186,7 +186,7 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let x = &operand("x", iWord);
+    let x = &Operand::new("x", iWord);
 
     ig.push(
         Inst::new(
@@ -225,8 +225,8 @@ pub(crate) fn define(
         .can_load(true),
     );
 
-    let y = &operand("y", iWord);
-    let rflags = &operand("rflags", iflags);
+    let y = &Operand::new("y", iWord);
+    let rflags = &Operand::new("rflags", iflags);
 
     ig.push(
         Inst::new(
@@ -271,9 +271,9 @@ pub(crate) fn define(
             .includes_scalars(false)
             .build(),
     );
-    let a = &operand_doc("a", TxN, "A vector value (i.e. held in an XMM register)");
-    let b = &operand_doc("b", TxN, "A vector value (i.e. held in an XMM register)");
-    let i = &operand_doc("i", uimm8, "An ordering operand controlling the copying of data from the source to the destination; see PSHUFD in Intel manual for details");
+    let a = &Operand::new("a", TxN).with_doc("A vector value (i.e. held in an XMM register)");
+    let b = &Operand::new("b", TxN).with_doc("A vector value (i.e. held in an XMM register)");
+    let i = &Operand::new("i", uimm8,).with_doc( "An ordering operand controlling the copying of data from the source to the destination; see PSHUFD in Intel manual for details");
 
     ig.push(
         Inst::new(
@@ -301,9 +301,9 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let Idx = &operand_doc("Idx", uimm8, "Lane index");
-    let x = &operand("x", TxN);
-    let a = &operand("a", &TxN.lane_of());
+    let Idx = &Operand::new("Idx", uimm8).with_doc("Lane index");
+    let x = &Operand::new("x", TxN);
+    let a = &Operand::new("a", &TxN.lane_of());
 
     ig.push(
         Inst::new(
@@ -329,9 +329,9 @@ pub(crate) fn define(
             .includes_scalars(false)
             .build(),
     );
-    let x = &operand("x", IBxN);
-    let y = &operand_doc("y", &IBxN.lane_of(), "New lane value");
-    let a = &operand("a", IBxN);
+    let x = &Operand::new("x", IBxN);
+    let y = &Operand::new("y", &IBxN.lane_of()).with_doc("New lane value");
+    let a = &Operand::new("a", IBxN);
 
     ig.push(
         Inst::new(
@@ -356,9 +356,9 @@ pub(crate) fn define(
             .includes_scalars(false)
             .build(),
     );
-    let x = &operand("x", FxN);
-    let y = &operand_doc("y", &FxN.lane_of(), "New lane value");
-    let a = &operand("a", FxN);
+    let x = &Operand::new("x", FxN);
+    let y = &Operand::new("y", &FxN.lane_of()).with_doc("New lane value");
+    let a = &Operand::new("a", FxN);
 
     ig.push(
         Inst::new(
@@ -374,9 +374,9 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let x = &operand("x", FxN);
-    let y = &operand("y", FxN);
-    let a = &operand("a", FxN);
+    let x = &Operand::new("x", FxN);
+    let y = &Operand::new("y", FxN);
+    let a = &Operand::new("a", FxN);
 
     ig.push(
         Inst::new(
@@ -422,9 +422,9 @@ pub(crate) fn define(
             .build(),
     );
 
-    let x = &operand_doc("x", IxN, "Vector value to shift");
-    let y = &operand_doc("y", I64x2, "Number of bits to shift");
-    let a = &operand("a", IxN);
+    let x = &Operand::new("x", IxN).with_doc("Vector value to shift");
+    let y = &Operand::new("y", I64x2).with_doc("Number of bits to shift");
+    let a = &Operand::new("a", IxN);
 
     ig.push(
         Inst::new(
@@ -468,16 +468,16 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let x = &operand("x", TxN);
-    let y = &operand("y", TxN);
-    let f = &operand("f", iflags);
+    let x = &Operand::new("x", TxN);
+    let y = &Operand::new("y", TxN);
+    let f = &Operand::new("f", iflags);
     ig.push(
         Inst::new(
             "x86_ptest",
             r#"
-        Logical Compare -- PTEST will set the ZF flag if all bits in the result are 0 of the 
-        bitwise AND of the first source operand (first operand) and the second source operand 
-        (second operand). PTEST sets the CF flag if all bits in the result are 0 of the bitwise  
+        Logical Compare -- PTEST will set the ZF flag if all bits in the result are 0 of the
+        bitwise AND of the first source operand (first operand) and the second source operand
+        (second operand). PTEST sets the CF flag if all bits in the result are 0 of the bitwise
         AND of the second source operand (second operand) and the logical NOT of the destination
         operand (first operand).
         "#,
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 967b71aa98..388e2cefad 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -3,7 +3,7 @@
 use crate::cdsl::instructions::{
     AllInstructions, InstructionBuilder as Inst, InstructionGroup, InstructionGroupBuilder,
 };
-use crate::cdsl::operands::{create_operand as operand, create_operand_doc as operand_doc};
+use crate::cdsl::operands::Operand;
 use crate::cdsl::type_inference::Constraint::WiderOrEq;
 use crate::cdsl::types::{LaneType, ValueType};
 use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
@@ -113,13 +113,13 @@ pub(crate) fn define(
 
     let MemTo = &TypeVar::copy_from(Mem, "MemTo".to_string());
 
-    let addr = &operand("addr", iAddr);
-    let c = &operand_doc("c", Testable, "Controlling value to test");
-    let Cond = &operand("Cond", &imm.intcc);
-    let x = &operand("x", iB);
-    let y = &operand("y", iB);
-    let EBB = &operand_doc("EBB", &entities.ebb, "Destination extended basic block");
-    let args = &operand_doc("args", &entities.varargs, "EBB arguments");
+    let addr = &Operand::new("addr", iAddr);
+    let c = &Operand::new("c", Testable).with_doc("Controlling value to test");
+    let Cond = &Operand::new("Cond", &imm.intcc);
+    let x = &Operand::new("x", iB);
+    let y = &Operand::new("y", iB);
+    let EBB = &Operand::new("EBB", &entities.ebb).with_doc("Destination extended basic block");
+    let args = &Operand::new("args", &entities.varargs).with_doc("EBB arguments");
 
     ig.push(
         Inst::new(
@@ -218,7 +218,7 @@ pub(crate) fn define(
         .is_branch(true),
     );
 
-    let f = &operand("f", iflags);
+    let f = &Operand::new("f", iflags);
 
     ig.push(
         Inst::new(
@@ -232,8 +232,8 @@ pub(crate) fn define(
         .is_branch(true),
     );
 
-    let Cond = &operand("Cond", &imm.floatcc);
-    let f = &operand("f", fflags);
+    let Cond = &Operand::new("Cond", &imm.floatcc);
+    let f = &Operand::new("f", fflags);
 
     ig.push(
         Inst::new(
@@ -248,9 +248,9 @@ pub(crate) fn define(
     );
 
     // The index into the br_table can be any type; legalizer will convert it to the right type.
-    let x = &operand_doc("x", iB, "index into jump table");
-    let entry = &operand_doc("entry", iAddr, "entry of jump table");
-    let JT = &operand("JT", &entities.jump_table);
+    let x = &Operand::new("x", iB).with_doc("index into jump table");
+    let entry = &Operand::new("entry", iAddr).with_doc("entry of jump table");
+    let JT = &Operand::new("JT", &entities.jump_table);
 
     ig.push(
         Inst::new(
@@ -280,8 +280,8 @@ pub(crate) fn define(
 
     // These are the instructions which br_table legalizes to: they perform address computations,
     // using pointer-sized integers, so their type variables are more constrained.
-    let x = &operand_doc("x", iAddr, "index into jump table");
-    let Size = &operand_doc("Size", &imm.uimm8, "Size in bytes");
+    let x = &Operand::new("x", iAddr).with_doc("index into jump table");
+    let Size = &Operand::new("Size", &imm.uimm8).with_doc("Size in bytes");
 
     ig.push(
         Inst::new(
@@ -350,7 +350,7 @@ pub(crate) fn define(
         .can_store(true),
     );
 
-    let code = &operand("code", &imm.trapcode);
+    let code = &Operand::new("code", &imm.trapcode);
 
     ig.push(
         Inst::new(
@@ -407,8 +407,8 @@ pub(crate) fn define(
         .can_trap(true),
     );
 
-    let Cond = &operand("Cond", &imm.intcc);
-    let f = &operand("f", iflags);
+    let Cond = &Operand::new("Cond", &imm.intcc);
+    let f = &Operand::new("f", iflags);
 
     ig.push(
         Inst::new(
@@ -422,8 +422,8 @@ pub(crate) fn define(
         .can_trap(true),
     );
 
-    let Cond = &operand("Cond", &imm.floatcc);
-    let f = &operand("f", fflags);
+    let Cond = &Operand::new("Cond", &imm.floatcc);
+    let f = &Operand::new("f", fflags);
 
     ig.push(
         Inst::new(
@@ -437,7 +437,7 @@ pub(crate) fn define(
         .can_trap(true),
     );
 
-    let rvals = &operand_doc("rvals", &entities.varargs, "return values");
+    let rvals = &Operand::new("rvals", &entities.varargs).with_doc("return values");
 
     ig.push(
         Inst::new(
@@ -473,12 +473,9 @@ pub(crate) fn define(
         .is_terminator(true),
     );
 
-    let FN = &operand_doc(
-        "FN",
-        &entities.func_ref,
-        "function to call, declared by `function`",
-    );
-    let args = &operand_doc("args", &entities.varargs, "call arguments");
+    let FN = &Operand::new("FN", &entities.func_ref)
+        .with_doc("function to call, declared by `function`");
+    let args = &Operand::new("args", &entities.varargs).with_doc("call arguments");
 
     ig.push(
         Inst::new(
@@ -496,8 +493,8 @@ pub(crate) fn define(
         .is_call(true),
     );
 
-    let SIG = &operand_doc("SIG", &entities.sig_ref, "function signature");
-    let callee = &operand_doc("callee", iAddr, "address of function to call");
+    let SIG = &Operand::new("SIG", &entities.sig_ref).with_doc("function signature");
+    let callee = &Operand::new("callee", iAddr).with_doc("address of function to call");
 
     ig.push(
         Inst::new(
@@ -538,13 +535,13 @@ pub(crate) fn define(
         .operands_out(vec![addr]),
     );
 
-    let SS = &operand("SS", &entities.stack_slot);
-    let Offset = &operand_doc("Offset", &imm.offset32, "Byte offset from base address");
-    let x = &operand_doc("x", Mem, "Value to be stored");
-    let a = &operand_doc("a", Mem, "Value loaded");
-    let p = &operand("p", iAddr);
-    let MemFlags = &operand("MemFlags", &imm.memflags);
-    let args = &operand_doc("args", &entities.varargs, "Address arguments");
+    let SS = &Operand::new("SS", &entities.stack_slot);
+    let Offset = &Operand::new("Offset", &imm.offset32).with_doc("Byte offset from base address");
+    let x = &Operand::new("x", Mem).with_doc("Value to be stored");
+    let a = &Operand::new("a", Mem).with_doc("Value loaded");
+    let p = &Operand::new("p", iAddr);
+    let MemFlags = &Operand::new("MemFlags", &imm.memflags);
+    let args = &Operand::new("args", &entities.varargs).with_doc("Address arguments");
 
     ig.push(
         Inst::new(
@@ -613,8 +610,8 @@ pub(crate) fn define(
         "An integer type with more than 8 bits",
         TypeSetBuilder::new().ints(16..64).build(),
     );
-    let x = &operand("x", iExt8);
-    let a = &operand("a", iExt8);
+    let x = &Operand::new("x", iExt8);
+    let a = &Operand::new("a", iExt8);
 
     ig.push(
         Inst::new(
@@ -709,8 +706,8 @@ pub(crate) fn define(
         "An integer type with more than 16 bits",
         TypeSetBuilder::new().ints(32..64).build(),
     );
-    let x = &operand("x", iExt16);
-    let a = &operand("a", iExt16);
+    let x = &Operand::new("x", iExt16);
+    let a = &Operand::new("a", iExt16);
 
     ig.push(
         Inst::new(
@@ -805,8 +802,8 @@ pub(crate) fn define(
         "An integer type with more than 32 bits",
         TypeSetBuilder::new().ints(64..64).build(),
     );
-    let x = &operand("x", iExt32);
-    let a = &operand("a", iExt32);
+    let x = &Operand::new("x", iExt32);
+    let a = &Operand::new("a", iExt32);
 
     ig.push(
         Inst::new(
@@ -896,9 +893,10 @@ pub(crate) fn define(
         .can_store(true),
     );
 
-    let x = &operand_doc("x", Mem, "Value to be stored");
-    let a = &operand_doc("a", Mem, "Value loaded");
-    let Offset = &operand_doc("Offset", &imm.offset32, "In-bounds offset into stack slot");
+    let x = &Operand::new("x", Mem).with_doc("Value to be stored");
+    let a = &Operand::new("a", Mem).with_doc("Value loaded");
+    let Offset =
+        &Operand::new("Offset", &imm.offset32).with_doc("In-bounds offset into stack slot");
 
     ig.push(
         Inst::new(
@@ -955,7 +953,7 @@ pub(crate) fn define(
         .operands_out(vec![addr]),
     );
 
-    let GV = &operand("GV", &entities.global_value);
+    let GV = &Operand::new("GV", &entities.global_value);
 
     ig.push(
         Inst::new(
@@ -987,9 +985,9 @@ pub(crate) fn define(
         TypeSetBuilder::new().ints(32..64).build(),
     );
 
-    let H = &operand("H", &entities.heap);
-    let p = &operand("p", HeapOffset);
-    let Size = &operand_doc("Size", &imm.uimm32, "Size in bytes");
+    let H = &Operand::new("H", &entities.heap);
+    let p = &Operand::new("p", HeapOffset);
+    let Size = &Operand::new("Size", &imm.uimm32).with_doc("Size in bytes");
 
     ig.push(
         Inst::new(
@@ -1046,9 +1044,10 @@ pub(crate) fn define(
         "An unsigned table offset",
         TypeSetBuilder::new().ints(32..64).build(),
     );
-    let T = &operand("T", &entities.table);
-    let p = &operand("p", TableOffset);
-    let Offset = &operand_doc("Offset", &imm.offset32, "Byte offset from element address");
+    let T = &Operand::new("T", &entities.table);
+    let p = &Operand::new("p", TableOffset);
+    let Offset =
+        &Operand::new("Offset", &imm.offset32).with_doc("Byte offset from element address");
 
     ig.push(
         Inst::new(
@@ -1072,8 +1071,8 @@ pub(crate) fn define(
         .operands_out(vec![addr]),
     );
 
-    let N = &operand("N", &imm.imm64);
-    let a = &operand_doc("a", Int, "A constant integer scalar or vector value");
+    let N = &Operand::new("N", &imm.imm64);
+    let a = &Operand::new("a", Int).with_doc("A constant integer scalar or vector value");
 
     ig.push(
         Inst::new(
@@ -1090,8 +1089,8 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let N = &operand("N", &imm.ieee32);
-    let a = &operand_doc("a", f32_, "A constant f32 scalar value");
+    let N = &Operand::new("N", &imm.ieee32);
+    let a = &Operand::new("a", f32_).with_doc("A constant f32 scalar value");
 
     ig.push(
         Inst::new(
@@ -1107,8 +1106,8 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let N = &operand("N", &imm.ieee64);
-    let a = &operand_doc("a", f64_, "A constant f64 scalar value");
+    let N = &Operand::new("N", &imm.ieee64);
+    let a = &Operand::new("a", f64_).with_doc("A constant f64 scalar value");
 
     ig.push(
         Inst::new(
@@ -1124,8 +1123,8 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let N = &operand("N", &imm.boolean);
-    let a = &operand_doc("a", Bool, "A constant boolean scalar or vector value");
+    let N = &Operand::new("N", &imm.boolean);
+    let a = &Operand::new("a", Bool).with_doc("A constant boolean scalar or vector value");
 
     ig.push(
         Inst::new(
@@ -1142,12 +1141,9 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let N = &operand_doc(
-        "N",
-        &imm.pool_constant,
-        "The 16 immediate bytes of a 128-bit vector",
-    );
-    let a = &operand_doc("a", TxN, "A constant vector value");
+    let N = &Operand::new("N", &imm.pool_constant)
+        .with_doc("The 16 immediate bytes of a 128-bit vector");
+    let a = &Operand::new("a", TxN).with_doc("A constant vector value");
 
     ig.push(
         Inst::new(
@@ -1163,11 +1159,8 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let mask = &operand_doc(
-        "mask",
-        &imm.uimm128,
-        "The 16 immediate bytes used for selecting the elements to shuffle",
-    );
+    let mask = &Operand::new("mask", &imm.uimm128)
+        .with_doc("The 16 immediate bytes used for selecting the elements to shuffle");
     let Tx16 = &TypeVar::new(
         "Tx16",
         "A SIMD vector with exactly 16 lanes of 8-bit values; eventually this may support other \
@@ -1179,8 +1172,8 @@ pub(crate) fn define(
             .includes_scalars(false)
             .build(),
     );
-    let a = &operand_doc("a", Tx16, "A vector value");
-    let b = &operand_doc("b", Tx16, "A vector value");
+    let a = &Operand::new("a", Tx16).with_doc("A vector value");
+    let b = &Operand::new("b", Tx16).with_doc("A vector value");
 
     ig.push(
         Inst::new(
@@ -1199,7 +1192,7 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let a = &operand_doc("a", Ref, "A constant reference null value");
+    let a = &Operand::new("a", Ref).with_doc("A constant reference null value");
 
     ig.push(
         Inst::new(
@@ -1224,10 +1217,10 @@ pub(crate) fn define(
         &formats.nullary,
     ));
 
-    let c = &operand_doc("c", Testable, "Controlling value to test");
-    let x = &operand_doc("x", Any, "Value to use when `c` is true");
-    let y = &operand_doc("y", Any, "Value to use when `c` is false");
-    let a = &operand("a", Any);
+    let c = &Operand::new("c", Testable).with_doc("Controlling value to test");
+    let x = &Operand::new("x", Any).with_doc("Value to use when `c` is true");
+    let y = &Operand::new("y", Any).with_doc("Value to use when `c` is false");
+    let a = &Operand::new("a", Any);
 
     ig.push(
         Inst::new(
@@ -1244,8 +1237,8 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let cc = &operand_doc("cc", &imm.intcc, "Controlling condition code");
-    let flags = &operand_doc("flags", iflags, "The machine's flag register");
+    let cc = &Operand::new("cc", &imm.intcc).with_doc("Controlling condition code");
+    let flags = &Operand::new("flags", iflags).with_doc("The machine's flag register");
 
     ig.push(
         Inst::new(
@@ -1259,7 +1252,7 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let c = &operand_doc("c", Any, "Controlling value to test");
+    let c = &Operand::new("c", Any).with_doc("Controlling value to test");
     ig.push(
         Inst::new(
             "bitselect",
@@ -1276,7 +1269,7 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let x = &operand("x", Any);
+    let x = &Operand::new("x", Any);
 
     ig.push(
         Inst::new(
@@ -1346,8 +1339,8 @@ pub(crate) fn define(
         .can_load(true),
     );
 
-    let src = &operand("src", &imm.regunit);
-    let dst = &operand("dst", &imm.regunit);
+    let src = &Operand::new("src", &imm.regunit);
+    let dst = &Operand::new("dst", &imm.regunit);
 
     ig.push(
         Inst::new(
@@ -1420,7 +1413,7 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let delta = &operand("delta", Int);
+    let delta = &Operand::new("delta", Int);
 
     ig.push(
         Inst::new(
@@ -1436,7 +1429,7 @@ pub(crate) fn define(
         .other_side_effects(true),
     );
 
-    let Offset = &operand_doc("Offset", &imm.imm64, "Offset from current stack pointer");
+    let Offset = &Operand::new("Offset", &imm.imm64).with_doc("Offset from current stack pointer");
 
     ig.push(
         Inst::new(
@@ -1454,7 +1447,7 @@ pub(crate) fn define(
         .other_side_effects(true),
     );
 
-    let Offset = &operand_doc("Offset", &imm.imm64, "Offset from current stack pointer");
+    let Offset = &Operand::new("Offset", &imm.imm64).with_doc("Offset from current stack pointer");
 
     ig.push(
         Inst::new(
@@ -1473,7 +1466,7 @@ pub(crate) fn define(
         .other_side_effects(true),
     );
 
-    let f = &operand("f", iflags);
+    let f = &Operand::new("f", iflags);
 
     ig.push(
         Inst::new(
@@ -1528,11 +1521,8 @@ pub(crate) fn define(
         .other_side_effects(true),
     );
 
-    let N = &operand_doc(
-        "args",
-        &entities.varargs,
-        "Variable number of args for Stackmap",
-    );
+    let N =
+        &Operand::new("args", &entities.varargs).with_doc("Variable number of args for Stackmap");
 
     ig.push(
         Inst::new(
@@ -1547,9 +1537,9 @@ pub(crate) fn define(
         .other_side_effects(true),
     );
 
-    let x = &operand_doc("x", TxN, "Vector to split");
-    let lo = &operand_doc("lo", &TxN.half_vector(), "Low-numbered lanes of `x`");
-    let hi = &operand_doc("hi", &TxN.half_vector(), "High-numbered lanes of `x`");
+    let x = &Operand::new("x", TxN).with_doc("Vector to split");
+    let lo = &Operand::new("lo", &TxN.half_vector()).with_doc("Low-numbered lanes of `x`");
+    let hi = &Operand::new("hi", &TxN.half_vector()).with_doc("High-numbered lanes of `x`");
 
     ig.push(
         Inst::new(
@@ -1580,9 +1570,9 @@ pub(crate) fn define(
             .build(),
     );
 
-    let x = &operand_doc("x", Any128, "Low-numbered lanes");
-    let y = &operand_doc("y", Any128, "High-numbered lanes");
-    let a = &operand_doc("a", &Any128.double_vector(), "Concatenation of `x` and `y`");
+    let x = &Operand::new("x", Any128).with_doc("Low-numbered lanes");
+    let y = &Operand::new("y", Any128).with_doc("High-numbered lanes");
+    let a = &Operand::new("a", &Any128.double_vector()).with_doc("Concatenation of `x` and `y`");
 
     ig.push(
         Inst::new(
@@ -1604,10 +1594,10 @@ pub(crate) fn define(
         .is_ghost(true),
     );
 
-    let c = &operand_doc("c", &TxN.as_bool(), "Controlling vector");
-    let x = &operand_doc("x", TxN, "Value to use where `c` is true");
-    let y = &operand_doc("y", TxN, "Value to use where `c` is false");
-    let a = &operand("a", TxN);
+    let c = &Operand::new("c", &TxN.as_bool()).with_doc("Controlling vector");
+    let x = &Operand::new("x", TxN).with_doc("Value to use where `c` is true");
+    let y = &Operand::new("y", TxN).with_doc("Value to use where `c` is false");
+    let a = &Operand::new("a", TxN);
 
     ig.push(
         Inst::new(
@@ -1624,7 +1614,7 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let s = &operand("s", b1);
+    let s = &Operand::new("s", b1);
 
     ig.push(
         Inst::new(
@@ -1654,7 +1644,7 @@ pub(crate) fn define(
         .operands_out(vec![s]),
     );
 
-    let x = &operand("x", &TxN.lane_of());
+    let x = &Operand::new("x", &TxN.lane_of());
 
     ig.push(
         Inst::new(
@@ -1670,9 +1660,9 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let x = &operand_doc("x", TxN, "SIMD vector to modify");
-    let y = &operand_doc("y", &TxN.lane_of(), "New lane value");
-    let Idx = &operand_doc("Idx", &imm.uimm8, "Lane index");
+    let x = &Operand::new("x", TxN).with_doc("SIMD vector to modify");
+    let y = &Operand::new("y", &TxN.lane_of()).with_doc("New lane value");
+    let Idx = &Operand::new("Idx", &imm.uimm8).with_doc("Lane index");
 
     ig.push(
         Inst::new(
@@ -1689,8 +1679,8 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let x = &operand("x", TxN);
-    let a = &operand("a", &TxN.lane_of());
+    let x = &Operand::new("x", TxN);
+    let a = &Operand::new("a", &TxN.lane_of());
 
     ig.push(
         Inst::new(
@@ -1709,10 +1699,10 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let a = &operand("a", &Int.as_bool());
-    let Cond = &operand("Cond", &imm.intcc);
-    let x = &operand("x", Int);
-    let y = &operand("y", Int);
+    let a = &Operand::new("a", &Int.as_bool());
+    let Cond = &Operand::new("Cond", &imm.intcc);
+    let x = &Operand::new("x", Int);
+    let y = &Operand::new("y", Int);
 
     ig.push(
         Inst::new(
@@ -1748,9 +1738,9 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let a = &operand("a", b1);
-    let x = &operand("x", iB);
-    let Y = &operand("Y", &imm.imm64);
+    let a = &Operand::new("a", b1);
+    let x = &Operand::new("x", iB);
+    let Y = &Operand::new("Y", &imm.imm64);
 
     ig.push(
         Inst::new(
@@ -1770,9 +1760,9 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let f = &operand("f", iflags);
-    let x = &operand("x", iB);
-    let y = &operand("y", iB);
+    let f = &Operand::new("f", iflags);
+    let x = &Operand::new("x", iB);
+    let y = &Operand::new("y", iB);
 
     ig.push(
         Inst::new(
@@ -1804,9 +1794,9 @@ pub(crate) fn define(
         .operands_out(vec![f]),
     );
 
-    let a = &operand("a", Int);
-    let x = &operand("x", Int);
-    let y = &operand("y", Int);
+    let a = &Operand::new("a", Int);
+    let x = &Operand::new("x", Int);
+    let y = &Operand::new("y", Int);
 
     ig.push(
         Inst::new(
@@ -2028,9 +2018,9 @@ pub(crate) fn define(
         .can_trap(true),
     );
 
-    let a = &operand("a", iB);
-    let x = &operand("x", iB);
-    let Y = &operand("Y", &imm.imm64);
+    let a = &Operand::new("a", iB);
+    let x = &Operand::new("x", iB);
+    let Y = &Operand::new("Y", &imm.imm64);
 
     ig.push(
         Inst::new(
@@ -2141,19 +2131,19 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let a = &operand("a", iB);
-    let x = &operand("x", iB);
-    let y = &operand("y", iB);
+    let a = &Operand::new("a", iB);
+    let x = &Operand::new("x", iB);
+    let y = &Operand::new("y", iB);
 
-    let c_in = &operand_doc("c_in", b1, "Input carry flag");
-    let c_out = &operand_doc("c_out", b1, "Output carry flag");
-    let b_in = &operand_doc("b_in", b1, "Input borrow flag");
-    let b_out = &operand_doc("b_out", b1, "Output borrow flag");
+    let c_in = &Operand::new("c_in", b1).with_doc("Input carry flag");
+    let c_out = &Operand::new("c_out", b1).with_doc("Output carry flag");
+    let b_in = &Operand::new("b_in", b1).with_doc("Input borrow flag");
+    let b_out = &Operand::new("b_out", b1).with_doc("Output borrow flag");
 
-    let c_if_in = &operand("c_in", iflags);
-    let c_if_out = &operand("c_out", iflags);
-    let b_if_in = &operand("b_in", iflags);
-    let b_if_out = &operand("b_out", iflags);
+    let c_if_in = &Operand::new("c_in", iflags);
+    let c_if_out = &Operand::new("c_out", iflags);
+    let b_if_in = &Operand::new("b_in", iflags);
+    let b_if_out = &Operand::new("b_out", iflags);
 
     ig.push(
         Inst::new(
@@ -2426,9 +2416,9 @@ pub(crate) fn define(
             .includes_scalars(true)
             .build(),
     );
-    let x = &operand("x", bits);
-    let y = &operand("y", bits);
-    let a = &operand("a", bits);
+    let x = &Operand::new("x", bits);
+    let y = &Operand::new("y", bits);
+    let a = &Operand::new("a", bits);
 
     ig.push(
         Inst::new(
@@ -2520,9 +2510,9 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let x = &operand("x", iB);
-    let Y = &operand("Y", &imm.imm64);
-    let a = &operand("a", iB);
+    let x = &Operand::new("x", iB);
+    let Y = &Operand::new("Y", &imm.imm64);
+    let a = &Operand::new("a", iB);
 
     ig.push(
         Inst::new(
@@ -2575,10 +2565,10 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let x = &operand_doc("x", Int, "Scalar or vector value to shift");
-    let y = &operand_doc("y", iB, "Number of bits to shift");
-    let Y = &operand("Y", &imm.imm64);
-    let a = &operand("a", Int);
+    let x = &Operand::new("x", Int).with_doc("Scalar or vector value to shift");
+    let y = &Operand::new("y", iB).with_doc("Number of bits to shift");
+    let Y = &Operand::new("Y", &imm.imm64);
+    let a = &Operand::new("a", Int);
 
     ig.push(
         Inst::new(
@@ -2735,8 +2725,8 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let x = &operand("x", iB);
-    let a = &operand("a", iB);
+    let x = &Operand::new("x", iB);
+    let a = &Operand::new("a", iB);
 
     ig.push(
         Inst::new(
@@ -2822,10 +2812,10 @@ pub(crate) fn define(
             .simd_lanes(Interval::All)
             .build(),
     );
-    let Cond = &operand("Cond", &imm.floatcc);
-    let x = &operand("x", Float);
-    let y = &operand("y", Float);
-    let a = &operand("a", &Float.as_bool());
+    let Cond = &Operand::new("Cond", &imm.floatcc);
+    let x = &Operand::new("x", Float);
+    let y = &Operand::new("y", Float);
+    let a = &Operand::new("a", &Float.as_bool());
 
     ig.push(
         Inst::new(
@@ -2897,7 +2887,7 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let f = &operand("f", fflags);
+    let f = &Operand::new("f", fflags);
 
     ig.push(
         Inst::new(
@@ -2914,10 +2904,10 @@ pub(crate) fn define(
         .operands_out(vec![f]),
     );
 
-    let x = &operand("x", Float);
-    let y = &operand("y", Float);
-    let z = &operand("z", Float);
-    let a = &operand_doc("a", Float, "Result of applying operator to each lane");
+    let x = &Operand::new("x", Float);
+    let y = &Operand::new("y", Float);
+    let z = &Operand::new("z", Float);
+    let a = &Operand::new("a", Float).with_doc("Result of applying operator to each lane");
 
     ig.push(
         Inst::new(
@@ -2998,7 +2988,7 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let a = &operand_doc("a", Float, "``x`` with its sign bit inverted");
+    let a = &Operand::new("a", Float).with_doc("``x`` with its sign bit inverted");
 
     ig.push(
         Inst::new(
@@ -3014,7 +3004,7 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let a = &operand_doc("a", Float, "``x`` with its sign bit cleared");
+    let a = &Operand::new("a", Float).with_doc("``x`` with its sign bit cleared");
 
     ig.push(
         Inst::new(
@@ -3030,11 +3020,7 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let a = &operand_doc(
-        "a",
-        Float,
-        "``x`` with its sign bit changed to that of ``y``",
-    );
+    let a = &Operand::new("a", Float).with_doc("``x`` with its sign bit changed to that of ``y``");
 
     ig.push(
         Inst::new(
@@ -3051,7 +3037,7 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let a = &operand_doc("a", Float, "The smaller of ``x`` and ``y``");
+    let a = &Operand::new("a", Float).with_doc("The smaller of ``x`` and ``y``");
 
     ig.push(
         Inst::new(
@@ -3067,7 +3053,7 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let a = &operand_doc("a", Float, "The larger of ``x`` and ``y``");
+    let a = &Operand::new("a", Float).with_doc("The larger of ``x`` and ``y``");
 
     ig.push(
         Inst::new(
@@ -3083,7 +3069,7 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let a = &operand_doc("a", Float, "``x`` rounded to integral value");
+    let a = &Operand::new("a", Float).with_doc("``x`` rounded to integral value");
 
     ig.push(
         Inst::new(
@@ -3134,8 +3120,8 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let a = &operand("a", b1);
-    let x = &operand("x", Ref);
+    let a = &Operand::new("a", b1);
+    let x = &Operand::new("x", Ref);
 
     ig.push(
         Inst::new(
@@ -3152,9 +3138,9 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let Cond = &operand("Cond", &imm.intcc);
-    let f = &operand("f", iflags);
-    let a = &operand("a", b1);
+    let Cond = &Operand::new("Cond", &imm.intcc);
+    let f = &Operand::new("f", iflags);
+    let a = &Operand::new("a", b1);
 
     ig.push(
         Inst::new(
@@ -3171,8 +3157,8 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let Cond = &operand("Cond", &imm.floatcc);
-    let f = &operand("f", fflags);
+    let Cond = &Operand::new("Cond", &imm.floatcc);
+    let f = &Operand::new("f", fflags);
 
     ig.push(
         Inst::new(
@@ -3189,8 +3175,8 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let x = &operand("x", Mem);
-    let a = &operand_doc("a", MemTo, "Bits of `x` reinterpreted");
+    let x = &Operand::new("x", Mem);
+    let a = &Operand::new("a", MemTo).with_doc("Bits of `x` reinterpreted");
 
     ig.push(
         Inst::new(
@@ -3208,8 +3194,8 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let x = &operand("x", Any);
-    let a = &operand_doc("a", AnyTo, "Bits of `x` reinterpreted");
+    let x = &Operand::new("x", Any);
+    let a = &Operand::new("a", AnyTo).with_doc("Bits of `x` reinterpreted");
 
     ig.push(
         Inst::new(
@@ -3231,8 +3217,8 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let a = &operand_doc("a", TxN, "A vector value");
-    let s = &operand_doc("s", &TxN.lane_of(), "A scalar value");
+    let a = &Operand::new("a", TxN).with_doc("A vector value");
+    let s = &Operand::new("s", &TxN.lane_of()).with_doc("A scalar value");
 
     ig.push(
         Inst::new(
@@ -3268,8 +3254,8 @@ pub(crate) fn define(
             .build(),
     );
 
-    let x = &operand("x", Bool);
-    let a = &operand("a", BoolTo);
+    let x = &Operand::new("x", Bool);
+    let a = &Operand::new("a", BoolTo);
 
     ig.push(
         Inst::new(
@@ -3296,8 +3282,8 @@ pub(crate) fn define(
             .simd_lanes(Interval::All)
             .build(),
     );
-    let x = &operand("x", Bool);
-    let a = &operand("a", BoolTo);
+    let x = &Operand::new("x", Bool);
+    let a = &Operand::new("a", BoolTo);
 
     ig.push(
         Inst::new(
@@ -3324,8 +3310,8 @@ pub(crate) fn define(
             .simd_lanes(Interval::All)
             .build(),
     );
-    let x = &operand("x", Bool);
-    let a = &operand("a", IntTo);
+    let x = &Operand::new("x", Bool);
+    let a = &Operand::new("a", IntTo);
 
     ig.push(
         Inst::new(
@@ -3374,8 +3360,8 @@ pub(crate) fn define(
             .simd_lanes(Interval::All)
             .build(),
     );
-    let x = &operand("x", Int);
-    let a = &operand("a", IntTo);
+    let x = &Operand::new("x", Int);
+    let a = &Operand::new("a", IntTo);
 
     ig.push(
         Inst::new(
@@ -3406,8 +3392,8 @@ pub(crate) fn define(
             .simd_lanes(Interval::All)
             .build(),
     );
-    let x = &operand("x", Int);
-    let a = &operand("a", IntTo);
+    let x = &Operand::new("x", Int);
+    let a = &Operand::new("a", IntTo);
 
     ig.push(
         Inst::new(
@@ -3459,8 +3445,8 @@ pub(crate) fn define(
             .simd_lanes(Interval::All)
             .build(),
     );
-    let x = &operand("x", Float);
-    let a = &operand("a", FloatTo);
+    let x = &Operand::new("x", Float);
+    let a = &Operand::new("a", FloatTo);
 
     ig.push(
         Inst::new(
@@ -3508,8 +3494,8 @@ pub(crate) fn define(
         .constraints(vec![WiderOrEq(Float.clone(), FloatTo.clone())]),
     );
 
-    let x = &operand("x", Float);
-    let a = &operand("a", IntTo);
+    let x = &Operand::new("x", Float);
+    let a = &Operand::new("a", IntTo);
 
     ig.push(
         Inst::new(
@@ -3576,8 +3562,8 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
-    let x = &operand("x", Int);
-    let a = &operand("a", FloatTo);
+    let x = &Operand::new("x", Int);
+    let a = &Operand::new("a", FloatTo);
 
     ig.push(
         Inst::new(
@@ -3621,9 +3607,9 @@ pub(crate) fn define(
             .simd_lanes(Interval::All)
             .build(),
     );
-    let x = &operand("x", WideInt);
-    let lo = &operand_doc("lo", &WideInt.half_width(), "The low bits of `x`");
-    let hi = &operand_doc("hi", &WideInt.half_width(), "The high bits of `x`");
+    let x = &Operand::new("x", WideInt);
+    let lo = &Operand::new("lo", &WideInt.half_width()).with_doc("The low bits of `x`");
+    let hi = &Operand::new("hi", &WideInt.half_width()).with_doc("The high bits of `x`");
 
     ig.push(
         Inst::new(
@@ -3653,13 +3639,10 @@ pub(crate) fn define(
             .build(),
     );
 
-    let lo = &operand("lo", NarrowInt);
-    let hi = &operand("hi", NarrowInt);
-    let a = &operand_doc(
-        "a",
-        &NarrowInt.double_width(),
-        "The concatenation of `lo` and `hi`",
-    );
+    let lo = &Operand::new("lo", NarrowInt);
+    let hi = &Operand::new("hi", NarrowInt);
+    let a = &Operand::new("a", &NarrowInt.double_width())
+        .with_doc("The concatenation of `lo` and `hi`");
 
     ig.push(
         Inst::new(

From ae3ea47dbd836e6d1cc5c27d4a02989816c8b9fc Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 29 Oct 2019 15:18:45 +0100
Subject: [PATCH 2890/3084] [meta] Delegate finding the default value to
 OperandKind instead of its builder.

This applies both to the default_member value (which is now determined at
runtime, instead of pre-computed) and the rust_type value (which is
determined in the Operand's ctor, instead of the builder).
---
 cranelift/codegen/meta/src/cdsl/encodings.rs |   2 +-
 cranelift/codegen/meta/src/cdsl/formats.rs   |   2 +-
 cranelift/codegen/meta/src/cdsl/operands.rs  | 124 ++++++++++---------
 3 files changed, 67 insertions(+), 61 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/encodings.rs b/cranelift/codegen/meta/src/cdsl/encodings.rs
index f79d3b9430..af6dd296bc 100644
--- a/cranelift/codegen/meta/src/cdsl/encodings.rs
+++ b/cranelift/codegen/meta/src/cdsl/encodings.rs
@@ -95,7 +95,7 @@ impl EncodingBuilder {
                         &inst.inst.format,
                         immediate_operand
                             .kind
-                            .default_member
+                            .default_member()
                             .expect("Immediates must always have a default member name set."),
                         immediate_value.to_string(),
                     );
diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
index bcfc6f5650..40b1d63958 100644
--- a/cranelift/codegen/meta/src/cdsl/formats.rs
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -127,7 +127,7 @@ impl InstructionFormatBuilder {
     pub fn imm(mut self, operand_kind: &OperandKind) -> Self {
         let field = FormatField {
             kind: operand_kind.clone(),
-            member: operand_kind.default_member.unwrap(),
+            member: operand_kind.default_member().unwrap(),
         };
         self.imm_fields.push(field);
         self
diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
index f8403bf910..414a381c03 100644
--- a/cranelift/codegen/meta/src/cdsl/operands.rs
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -111,18 +111,55 @@ pub(crate) enum OperandKindFields {
 #[derive(Clone, Debug)]
 pub(crate) struct OperandKind {
     pub name: &'static str,
-
-    doc: Option,
-
-    pub default_member: Option<&'static str>,
-
+    doc: Option<&'static str>,
+    default_member: Option<&'static str>,
     /// The camel-cased name of an operand kind is also the Rust type used to represent it.
     pub rust_type: String,
-
     pub fields: OperandKindFields,
 }
 
 impl OperandKind {
+    fn new(
+        name: &'static str,
+        doc: Option<&'static str>,
+        default_member: Option<&'static str>,
+        rust_type: Option<&'static str>,
+        fields: OperandKindFields,
+    ) -> Self {
+        // Compute the default rust_type value, if it wasn't provided.
+        let rust_type = match rust_type {
+            Some(rust_type) => rust_type.to_string(),
+            None => match &fields {
+                OperandKindFields::ImmEnum(_) | OperandKindFields::ImmValue => {
+                    format!("ir::immediates::{}", camel_case(name))
+                }
+                OperandKindFields::VariableArgs => "&[Value]".to_string(),
+                OperandKindFields::TypeVar(_) | OperandKindFields::EntityRef => {
+                    format!("ir::{}", camel_case(name))
+                }
+            },
+        };
+        Self {
+            name,
+            doc,
+            default_member,
+            rust_type,
+            fields,
+        }
+    }
+
+    /// Name of this OperandKind in the format's member field.
+    pub fn default_member(&self) -> Option<&'static str> {
+        if let Some(member) = &self.default_member {
+            return Some(member);
+        }
+        match &self.fields {
+            OperandKindFields::ImmEnum(_) | OperandKindFields::ImmValue => Some("imm"),
+            OperandKindFields::TypeVar(_) | OperandKindFields::EntityRef => Some(self.name),
+            OperandKindFields::VariableArgs => None,
+        }
+    }
+
     fn doc(&self) -> Option<&str> {
         if let Some(doc) = &self.doc {
             return Some(doc);
@@ -146,16 +183,22 @@ impl OperandKind {
     }
 }
 
+impl Into for &TypeVar {
+    fn into(self) -> OperandKind {
+        OperandKindBuilder::new("value", OperandKindFields::TypeVar(self.into())).build()
+    }
+}
+impl Into for &OperandKind {
+    fn into(self) -> OperandKind {
+        self.clone()
+    }
+}
+
 pub(crate) struct OperandKindBuilder {
     name: &'static str,
-
-    doc: Option,
-
+    doc: Option<&'static str>,
     default_member: Option<&'static str>,
-
-    /// The camel-cased name of an operand kind is also the Rust type used to represent it.
-    rust_type: Option,
-
+    rust_type: Option<&'static str>,
     fields: OperandKindFields,
 }
 
@@ -169,7 +212,6 @@ impl OperandKindBuilder {
             fields,
         }
     }
-
     pub fn new_imm(name: &'static str) -> Self {
         Self {
             name,
@@ -179,7 +221,6 @@ impl OperandKindBuilder {
             fields: OperandKindFields::ImmValue,
         }
     }
-
     pub fn new_enum(name: &'static str, values: EnumValues) -> Self {
         Self {
             name,
@@ -189,10 +230,9 @@ impl OperandKindBuilder {
             fields: OperandKindFields::ImmEnum(values),
         }
     }
-
     pub fn doc(mut self, doc: &'static str) -> Self {
         assert!(self.doc.is_none());
-        self.doc = Some(doc.to_string());
+        self.doc = Some(doc);
         self
     }
     pub fn default_member(mut self, default_member: &'static str) -> Self {
@@ -202,50 +242,16 @@ impl OperandKindBuilder {
     }
     pub fn rust_type(mut self, rust_type: &'static str) -> Self {
         assert!(self.rust_type.is_none());
-        self.rust_type = Some(rust_type.to_string());
+        self.rust_type = Some(rust_type);
         self
     }
-
     pub fn build(self) -> OperandKind {
-        let default_member = match self.default_member {
-            Some(default_member) => Some(default_member),
-            None => match &self.fields {
-                OperandKindFields::ImmEnum(_) | OperandKindFields::ImmValue => Some("imm"),
-                OperandKindFields::TypeVar(_) | OperandKindFields::EntityRef => Some(self.name),
-                OperandKindFields::VariableArgs => None,
-            },
-        };
-
-        let rust_type = match self.rust_type {
-            Some(rust_type) => rust_type.to_string(),
-            None => match &self.fields {
-                OperandKindFields::ImmEnum(_) | OperandKindFields::ImmValue => {
-                    format!("ir::immediates::{}", camel_case(self.name))
-                }
-                OperandKindFields::VariableArgs => "&[Value]".to_string(),
-                OperandKindFields::TypeVar(_) | OperandKindFields::EntityRef => {
-                    format!("ir::{}", camel_case(self.name))
-                }
-            },
-        };
-
-        OperandKind {
-            name: self.name,
-            doc: self.doc,
-            default_member,
-            rust_type,
-            fields: self.fields,
-        }
-    }
-}
-
-impl Into for &TypeVar {
-    fn into(self) -> OperandKind {
-        OperandKindBuilder::new("value", OperandKindFields::TypeVar(self.into())).build()
-    }
-}
-impl Into for &OperandKind {
-    fn into(self) -> OperandKind {
-        self.clone()
+        OperandKind::new(
+            self.name,
+            self.doc,
+            self.default_member,
+            self.rust_type,
+            self.fields,
+        )
     }
 }

From 4f5b0689f3df1727a54d7d48957aa0118c89d83c Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 29 Oct 2019 15:24:52 +0100
Subject: [PATCH 2891/3084] [meta] Remove OperandKind::imm_name method;

---
 cranelift/codegen/meta/src/cdsl/instructions.rs | 14 +++++++-------
 cranelift/codegen/meta/src/cdsl/operands.rs     |  9 ---------
 2 files changed, 7 insertions(+), 16 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index a804efca7c..32d6a7d66b 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -555,7 +555,7 @@ fn verify_format(inst_name: &str, operands_in: &[Operand], format: &InstructionF
     // - its number and names of input immediate operands,
     // - whether it has a value list or not.
     let mut num_values = 0;
-    let mut imm_index = 0;
+    let mut num_immediates = 0;
 
     for operand in operands_in.iter() {
         if operand.is_varargs() {
@@ -569,15 +569,15 @@ fn verify_format(inst_name: &str, operands_in: &[Operand], format: &InstructionF
         if operand.is_value() {
             num_values += 1;
         }
-        if let Some(imm_name) = operand.kind.imm_name() {
-            if let Some(format_field) = format.imm_fields.get(imm_index) {
+        if operand.is_immediate_or_entityref() {
+            if let Some(format_field) = format.imm_fields.get(num_immediates) {
                 assert_eq!(
-                    format_field.kind.name, imm_name,
+                    format_field.kind.name, operand.kind.name,
                     "{}th operand of {} should be {} (according to format), not {} (according to \
                      inst definition). You may need to use a different format.",
-                    imm_index, inst_name, format_field.kind.name, imm_name
+                    num_immediates, inst_name, format_field.kind.name, operand.kind.name
                 );
-                imm_index += 1;
+                num_immediates += 1;
             }
         }
     }
@@ -590,7 +590,7 @@ fn verify_format(inst_name: &str, operands_in: &[Operand], format: &InstructionF
     );
 
     assert_eq!(
-        imm_index,
+        num_immediates,
         format.imm_fields.len(),
         "inst {} doesn't have as many immediate input \
          operands as its format {} declares; you may need to use a different format.",
diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
index 414a381c03..83bbfa6696 100644
--- a/cranelift/codegen/meta/src/cdsl/operands.rs
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -172,15 +172,6 @@ impl OperandKind {
             | OperandKindFields::VariableArgs => None,
         }
     }
-
-    pub fn imm_name(&self) -> Option<&str> {
-        match self.fields {
-            OperandKindFields::ImmEnum(_)
-            | OperandKindFields::ImmValue
-            | OperandKindFields::EntityRef => Some(&self.name),
-            _ => None,
-        }
-    }
 }
 
 impl Into for &TypeVar {

From 0eb2dfc4a39f4441ca1b6e12a58c53425f95e6cb Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 29 Oct 2019 15:41:13 +0100
Subject: [PATCH 2892/3084] [meta] Rename OperandKind::default_member to
 format_field_name;

---
 cranelift/codegen/meta/src/cdsl/encodings.rs  |  4 +--
 cranelift/codegen/meta/src/cdsl/formats.rs    |  2 +-
 cranelift/codegen/meta/src/cdsl/operands.rs   | 26 +++++++++----------
 cranelift/codegen/meta/src/shared/entities.rs |  4 +--
 .../codegen/meta/src/shared/immediates.rs     | 12 ++++-----
 5 files changed, 24 insertions(+), 24 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/encodings.rs b/cranelift/codegen/meta/src/cdsl/encodings.rs
index af6dd296bc..730e9e75df 100644
--- a/cranelift/codegen/meta/src/cdsl/encodings.rs
+++ b/cranelift/codegen/meta/src/cdsl/encodings.rs
@@ -95,8 +95,8 @@ impl EncodingBuilder {
                         &inst.inst.format,
                         immediate_operand
                             .kind
-                            .default_member()
-                            .expect("Immediates must always have a default member name set."),
+                            .rust_field_name()
+                            .expect("Immediates must always have a field name."),
                         immediate_value.to_string(),
                     );
                     inst_predicate = if let Some(type_predicate) = inst_predicate {
diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
index 40b1d63958..ce63e9fe15 100644
--- a/cranelift/codegen/meta/src/cdsl/formats.rs
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -127,7 +127,7 @@ impl InstructionFormatBuilder {
     pub fn imm(mut self, operand_kind: &OperandKind) -> Self {
         let field = FormatField {
             kind: operand_kind.clone(),
-            member: operand_kind.default_member().unwrap(),
+            member: operand_kind.rust_field_name().unwrap(),
         };
         self.imm_fields.push(field);
         self
diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
index 83bbfa6696..5195f98891 100644
--- a/cranelift/codegen/meta/src/cdsl/operands.rs
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -112,7 +112,7 @@ pub(crate) enum OperandKindFields {
 pub(crate) struct OperandKind {
     pub name: &'static str,
     doc: Option<&'static str>,
-    default_member: Option<&'static str>,
+    rust_field_name: Option<&'static str>,
     /// The camel-cased name of an operand kind is also the Rust type used to represent it.
     pub rust_type: String,
     pub fields: OperandKindFields,
@@ -122,7 +122,7 @@ impl OperandKind {
     fn new(
         name: &'static str,
         doc: Option<&'static str>,
-        default_member: Option<&'static str>,
+        rust_field_name: Option<&'static str>,
         rust_type: Option<&'static str>,
         fields: OperandKindFields,
     ) -> Self {
@@ -142,15 +142,15 @@ impl OperandKind {
         Self {
             name,
             doc,
-            default_member,
+            rust_field_name,
             rust_type,
             fields,
         }
     }
 
     /// Name of this OperandKind in the format's member field.
-    pub fn default_member(&self) -> Option<&'static str> {
-        if let Some(member) = &self.default_member {
+    pub fn rust_field_name(&self) -> Option<&'static str> {
+        if let Some(member) = &self.rust_field_name {
             return Some(member);
         }
         match &self.fields {
@@ -188,7 +188,7 @@ impl Into for &OperandKind {
 pub(crate) struct OperandKindBuilder {
     name: &'static str,
     doc: Option<&'static str>,
-    default_member: Option<&'static str>,
+    rust_field_name: Option<&'static str>,
     rust_type: Option<&'static str>,
     fields: OperandKindFields,
 }
@@ -198,7 +198,7 @@ impl OperandKindBuilder {
         Self {
             name,
             doc: None,
-            default_member: None,
+            rust_field_name: None,
             rust_type: None,
             fields,
         }
@@ -207,7 +207,7 @@ impl OperandKindBuilder {
         Self {
             name,
             doc: None,
-            default_member: None,
+            rust_field_name: None,
             rust_type: None,
             fields: OperandKindFields::ImmValue,
         }
@@ -216,7 +216,7 @@ impl OperandKindBuilder {
         Self {
             name,
             doc: None,
-            default_member: None,
+            rust_field_name: None,
             rust_type: None,
             fields: OperandKindFields::ImmEnum(values),
         }
@@ -226,9 +226,9 @@ impl OperandKindBuilder {
         self.doc = Some(doc);
         self
     }
-    pub fn default_member(mut self, default_member: &'static str) -> Self {
-        assert!(self.default_member.is_none());
-        self.default_member = Some(default_member);
+    pub fn rust_field_name(mut self, rust_field_name: &'static str) -> Self {
+        assert!(self.rust_field_name.is_none());
+        self.rust_field_name = Some(rust_field_name);
         self
     }
     pub fn rust_type(mut self, rust_type: &'static str) -> Self {
@@ -240,7 +240,7 @@ impl OperandKindBuilder {
         OperandKind::new(
             self.name,
             self.doc,
-            self.default_member,
+            self.rust_field_name,
             self.rust_type,
             self.fields,
         )
diff --git a/cranelift/codegen/meta/src/shared/entities.rs b/cranelift/codegen/meta/src/shared/entities.rs
index 631e0855d2..48c3ce27b4 100644
--- a/cranelift/codegen/meta/src/shared/entities.rs
+++ b/cranelift/codegen/meta/src/shared/entities.rs
@@ -36,7 +36,7 @@ impl EntityRefs {
     pub fn new() -> Self {
         Self {
             ebb: create("ebb", "An extended basic block in the same function.")
-                .default_member("destination")
+                .rust_field_name("destination")
                 .build(),
 
             stack_slot: create("stack_slot", "A stack slot").build(),
@@ -48,7 +48,7 @@ impl EntityRefs {
             func_ref: create("func_ref", "An external function.").build(),
 
             jump_table: create("jump_table", "A jump table.")
-                .default_member("table")
+                .rust_field_name("table")
                 .build(),
 
             heap: create("heap", "A heap.").build(),
diff --git a/cranelift/codegen/meta/src/shared/immediates.rs b/cranelift/codegen/meta/src/shared/immediates.rs
index f9c114a6a0..182bd1a4b6 100644
--- a/cranelift/codegen/meta/src/shared/immediates.rs
+++ b/cranelift/codegen/meta/src/shared/immediates.rs
@@ -95,13 +95,13 @@ impl Immediates {
 
             pool_constant: Builder::new_imm("poolConstant")
                 .doc("A constant stored in the constant pool.")
-                .default_member("constant_handle")
+                .rust_field_name("constant_handle")
                 .rust_type("ir::Constant")
                 .build(),
 
             offset32: Builder::new_imm("offset32")
                 .doc("A 32-bit immediate signed offset.")
-                .default_member("offset")
+                .rust_field_name("offset")
                 .build(),
 
             ieee32: Builder::new_imm("ieee32")
@@ -133,7 +133,7 @@ impl Immediates {
                 intcc_values.insert("nof", "NotOverflow");
                 Builder::new_enum("intcc", intcc_values)
                     .doc("An integer comparison condition code.")
-                    .default_member("cond")
+                    .rust_field_name("cond")
                     .rust_type("ir::condcodes::IntCC")
                     .build()
             },
@@ -156,14 +156,14 @@ impl Immediates {
                 floatcc_values.insert("uge", "UnorderedOrGreaterThanOrEqual");
                 Builder::new_enum("floatcc", floatcc_values)
                     .doc("A floating point comparison condition code")
-                    .default_member("cond")
+                    .rust_field_name("cond")
                     .rust_type("ir::condcodes::FloatCC")
                     .build()
             },
 
             memflags: Builder::new_imm("memflags")
                 .doc("Memory operation flags")
-                .default_member("flags")
+                .rust_field_name("flags")
                 .rust_type("ir::MemFlags")
                 .build(),
 
@@ -180,7 +180,7 @@ impl Immediates {
                 trapcode_values.insert("int_divz", "IntegerDivisionByZero");
                 Builder::new_enum("trapcode", trapcode_values)
                     .doc("A trap reason code.")
-                    .default_member("code")
+                    .rust_field_name("code")
                     .rust_type("ir::TrapCode")
                     .build()
             },

From d5e990220e97242bd5a54c1b7914aaf0b7a198af Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 29 Oct 2019 15:46:06 +0100
Subject: [PATCH 2893/3084] [meta] Remove OperandKind::name field and
 explicitly pass rust_field_name/rust_type; (fixes #1177)

---
 cranelift/codegen/meta/src/cdsl/ast.rs        |  18 +--
 cranelift/codegen/meta/src/cdsl/encodings.rs  |   5 +-
 cranelift/codegen/meta/src/cdsl/formats.rs    |   8 +-
 .../codegen/meta/src/cdsl/instructions.rs     |  11 +-
 cranelift/codegen/meta/src/cdsl/operands.rs   | 142 +++++++-----------
 cranelift/codegen/meta/src/cdsl/xform.rs      |   2 +-
 cranelift/codegen/meta/src/gen_inst.rs        |   4 +-
 cranelift/codegen/meta/src/shared/entities.rs |  33 ++--
 .../codegen/meta/src/shared/immediates.rs     |  70 ++++-----
 9 files changed, 123 insertions(+), 170 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index 47dd457e23..e2e661de8b 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -151,12 +151,12 @@ pub(crate) enum Literal {
     /// corresponding to a Rust enum type. An `Enumerator` object is an AST leaf node representing one
     /// of the values.
     Enumerator {
-        rust_type: String,
+        rust_type: &'static str,
         value: &'static str,
     },
 
     /// A bitwise value of an immediate operand, used for bitwise exact floating point constants.
-    Bits { rust_type: String, value: u64 },
+    Bits { rust_type: &'static str, value: u64 },
 
     /// A value of an integer immediate operand.
     Int(i64),
@@ -171,13 +171,13 @@ impl Literal {
             OperandKindFields::ImmEnum(values) => values.get(value).unwrap_or_else(|| {
                 panic!(
                     "nonexistent value '{}' in enumeration '{}'",
-                    value, kind.name
+                    value, kind.rust_type
                 )
             }),
             _ => panic!("enumerator is for enum values"),
         };
         Literal::Enumerator {
-            rust_type: kind.rust_type.clone(),
+            rust_type: kind.rust_type,
             value,
         }
     }
@@ -188,7 +188,7 @@ impl Literal {
             _ => panic!("bits_of is for immediate scalar types"),
         }
         Literal::Bits {
-            rust_type: kind.rust_type.clone(),
+            rust_type: kind.rust_type,
             value: bits,
         }
     }
@@ -475,12 +475,12 @@ impl Apply {
                                 "Nonexistent enum value '{}' passed to field of kind '{}' -- \
                                  did you use the right enum?",
                                 value,
-                                op.kind.name
+                                op.kind.rust_type
                             );
                         } else {
                             panic!(
                                 "Passed non-enum field value {:?} to field of kind {}",
-                                literal, op.kind.name
+                                literal, op.kind.rust_type
                             );
                         }
                     }
@@ -488,14 +488,14 @@ impl Apply {
                         Literal::Enumerator { value, .. } => panic!(
                             "Expected immediate value in immediate field of kind '{}', \
                              obtained enum value '{}'",
-                            op.kind.name, value
+                            op.kind.rust_type, value
                         ),
                         Literal::Bits { .. } | Literal::Int(_) | Literal::EmptyVarArgs => {}
                     },
                     _ => {
                         panic!(
                             "Literal passed to non-literal field of kind {}",
-                            op.kind.name
+                            op.kind.rust_type
                         );
                     }
                 }
diff --git a/cranelift/codegen/meta/src/cdsl/encodings.rs b/cranelift/codegen/meta/src/cdsl/encodings.rs
index 730e9e75df..f66746f92f 100644
--- a/cranelift/codegen/meta/src/cdsl/encodings.rs
+++ b/cranelift/codegen/meta/src/cdsl/encodings.rs
@@ -93,10 +93,7 @@ impl EncodingBuilder {
                 {
                     let immediate_predicate = InstructionPredicate::new_is_field_equal(
                         &inst.inst.format,
-                        immediate_operand
-                            .kind
-                            .rust_field_name()
-                            .expect("Immediates must always have a field name."),
+                        immediate_operand.kind.rust_field_name,
                         immediate_value.to_string(),
                     );
                     inst_predicate = if let Some(type_predicate) = inst_predicate {
diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
index ce63e9fe15..0978d61faf 100644
--- a/cranelift/codegen/meta/src/cdsl/formats.rs
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -49,7 +49,7 @@ pub(crate) struct InstructionFormat {
 pub(crate) struct FormatStructure {
     pub num_value_operands: usize,
     pub has_value_list: bool,
-    pub imm_field_names: Vec<&'static str>,
+    pub imm_field_names: Vec<(&'static str, &'static str)>,
 }
 
 impl fmt::Display for InstructionFormat {
@@ -57,7 +57,7 @@ impl fmt::Display for InstructionFormat {
         let imm_args = self
             .imm_fields
             .iter()
-            .map(|field| format!("{}: {}", field.member, field.kind.name))
+            .map(|field| format!("{}: {}", field.member, field.kind.rust_type))
             .collect::>()
             .join(", ");
         fmt.write_fmt(format_args!(
@@ -89,7 +89,7 @@ impl InstructionFormat {
             imm_field_names: self
                 .imm_fields
                 .iter()
-                .map(|field| field.kind.name)
+                .map(|field| (field.kind.rust_field_name, field.kind.rust_type))
                 .collect::>(),
         }
     }
@@ -127,7 +127,7 @@ impl InstructionFormatBuilder {
     pub fn imm(mut self, operand_kind: &OperandKind) -> Self {
         let field = FormatField {
             kind: operand_kind.clone(),
-            member: operand_kind.rust_field_name().unwrap(),
+            member: operand_kind.rust_field_name,
         };
         self.imm_fields.push(field);
         self
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 32d6a7d66b..6b21bfaf10 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -572,10 +572,14 @@ fn verify_format(inst_name: &str, operands_in: &[Operand], format: &InstructionF
         if operand.is_immediate_or_entityref() {
             if let Some(format_field) = format.imm_fields.get(num_immediates) {
                 assert_eq!(
-                    format_field.kind.name, operand.kind.name,
+                    format_field.kind.rust_field_name,
+                    operand.kind.rust_field_name,
                     "{}th operand of {} should be {} (according to format), not {} (according to \
                      inst definition). You may need to use a different format.",
-                    num_immediates, inst_name, format_field.kind.name, operand.kind.name
+                    num_immediates,
+                    inst_name,
+                    format_field.kind.rust_field_name,
+                    operand.kind.rust_field_name
                 );
                 num_immediates += 1;
             }
@@ -1299,7 +1303,8 @@ mod test {
     fn field_to_operand(index: usize, field: OperandKindFields) -> Operand {
         // Pretend the index string is &'static.
         let name = Box::leak(index.to_string().into_boxed_str());
-        let kind = OperandKindBuilder::new(name, field).build();
+        // Format's name / rust_type don't matter here.
+        let kind = OperandKindBuilder::new(name, name, field).build();
         let operand = Operand::new(name, kind);
         operand
     }
diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
index 5195f98891..d4c1a334c7 100644
--- a/cranelift/codegen/meta/src/cdsl/operands.rs
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -1,6 +1,5 @@
 use std::collections::HashMap;
 
-use crate::cdsl::camel_case;
 use crate::cdsl::typevar::TypeVar;
 
 /// An instruction operand can be an *immediate*, an *SSA value*, or an *entity reference*. The
@@ -18,9 +17,13 @@ use crate::cdsl::typevar::TypeVar;
 ///    function, typically something declared in the function preamble.
 #[derive(Clone, Debug)]
 pub(crate) struct Operand {
+    /// Name of the operand variable, as it appears in function parameters, legalizations, etc.
     pub name: &'static str,
-    doc: Option<&'static str>,
+
+    /// Type of the operand.
     pub kind: OperandKind,
+
+    doc: Option<&'static str>,
 }
 
 impl Operand {
@@ -110,56 +113,19 @@ pub(crate) enum OperandKindFields {
 
 #[derive(Clone, Debug)]
 pub(crate) struct OperandKind {
-    pub name: &'static str,
-    doc: Option<&'static str>,
-    rust_field_name: Option<&'static str>,
-    /// The camel-cased name of an operand kind is also the Rust type used to represent it.
-    pub rust_type: String,
+    /// String representation of the Rust type mapping to this OperandKind.
+    pub rust_type: &'static str,
+
+    /// Name of this OperandKind in the format's member field.
+    pub rust_field_name: &'static str,
+
+    /// Type-specific fields for this OperandKind.
     pub fields: OperandKindFields,
+
+    doc: Option<&'static str>,
 }
 
 impl OperandKind {
-    fn new(
-        name: &'static str,
-        doc: Option<&'static str>,
-        rust_field_name: Option<&'static str>,
-        rust_type: Option<&'static str>,
-        fields: OperandKindFields,
-    ) -> Self {
-        // Compute the default rust_type value, if it wasn't provided.
-        let rust_type = match rust_type {
-            Some(rust_type) => rust_type.to_string(),
-            None => match &fields {
-                OperandKindFields::ImmEnum(_) | OperandKindFields::ImmValue => {
-                    format!("ir::immediates::{}", camel_case(name))
-                }
-                OperandKindFields::VariableArgs => "&[Value]".to_string(),
-                OperandKindFields::TypeVar(_) | OperandKindFields::EntityRef => {
-                    format!("ir::{}", camel_case(name))
-                }
-            },
-        };
-        Self {
-            name,
-            doc,
-            rust_field_name,
-            rust_type,
-            fields,
-        }
-    }
-
-    /// Name of this OperandKind in the format's member field.
-    pub fn rust_field_name(&self) -> Option<&'static str> {
-        if let Some(member) = &self.rust_field_name {
-            return Some(member);
-        }
-        match &self.fields {
-            OperandKindFields::ImmEnum(_) | OperandKindFields::ImmValue => Some("imm"),
-            OperandKindFields::TypeVar(_) | OperandKindFields::EntityRef => Some(self.name),
-            OperandKindFields::VariableArgs => None,
-        }
-    }
-
     fn doc(&self) -> Option<&str> {
         if let Some(doc) = &self.doc {
             return Some(doc);
@@ -176,7 +142,12 @@ impl OperandKind {
 
 impl Into for &TypeVar {
     fn into(self) -> OperandKind {
-        OperandKindBuilder::new("value", OperandKindFields::TypeVar(self.into())).build()
+        OperandKindBuilder::new(
+            "value",
+            "ir::Value",
+            OperandKindFields::TypeVar(self.into()),
+        )
+        .build()
     }
 }
 impl Into for &OperandKind {
@@ -186,63 +157,56 @@ impl Into for &OperandKind {
 }
 
 pub(crate) struct OperandKindBuilder {
-    name: &'static str,
-    doc: Option<&'static str>,
-    rust_field_name: Option<&'static str>,
-    rust_type: Option<&'static str>,
+    rust_field_name: &'static str,
+    rust_type: &'static str,
     fields: OperandKindFields,
+    doc: Option<&'static str>,
 }
 
 impl OperandKindBuilder {
-    pub fn new(name: &'static str, fields: OperandKindFields) -> Self {
+    pub fn new(
+        rust_field_name: &'static str,
+        rust_type: &'static str,
+        fields: OperandKindFields,
+    ) -> Self {
         Self {
-            name,
-            doc: None,
-            rust_field_name: None,
-            rust_type: None,
+            rust_field_name,
+            rust_type,
             fields,
+            doc: None,
         }
     }
-    pub fn new_imm(name: &'static str) -> Self {
+    pub fn new_imm(rust_field_name: &'static str, rust_type: &'static str) -> Self {
         Self {
-            name,
-            doc: None,
-            rust_field_name: None,
-            rust_type: None,
+            rust_field_name,
+            rust_type,
             fields: OperandKindFields::ImmValue,
-        }
-    }
-    pub fn new_enum(name: &'static str, values: EnumValues) -> Self {
-        Self {
-            name,
             doc: None,
-            rust_field_name: None,
-            rust_type: None,
-            fields: OperandKindFields::ImmEnum(values),
         }
     }
-    pub fn doc(mut self, doc: &'static str) -> Self {
+    pub fn new_enum(
+        rust_field_name: &'static str,
+        rust_type: &'static str,
+        values: EnumValues,
+    ) -> Self {
+        Self {
+            rust_field_name,
+            rust_type,
+            fields: OperandKindFields::ImmEnum(values),
+            doc: None,
+        }
+    }
+    pub fn with_doc(mut self, doc: &'static str) -> Self {
         assert!(self.doc.is_none());
         self.doc = Some(doc);
         self
     }
-    pub fn rust_field_name(mut self, rust_field_name: &'static str) -> Self {
-        assert!(self.rust_field_name.is_none());
-        self.rust_field_name = Some(rust_field_name);
-        self
-    }
-    pub fn rust_type(mut self, rust_type: &'static str) -> Self {
-        assert!(self.rust_type.is_none());
-        self.rust_type = Some(rust_type);
-        self
-    }
     pub fn build(self) -> OperandKind {
-        OperandKind::new(
-            self.name,
-            self.doc,
-            self.rust_field_name,
-            self.rust_type,
-            self.fields,
-        )
+        OperandKind {
+            rust_type: self.rust_type,
+            fields: self.fields,
+            rust_field_name: self.rust_field_name,
+            doc: self.doc,
+        }
     }
 }
diff --git a/cranelift/codegen/meta/src/cdsl/xform.rs b/cranelift/codegen/meta/src/cdsl/xform.rs
index 56f6baf949..d21e93128d 100644
--- a/cranelift/codegen/meta/src/cdsl/xform.rs
+++ b/cranelift/codegen/meta/src/cdsl/xform.rs
@@ -214,7 +214,7 @@ fn rewrite_expr(
             .inst()
             .operands_in
             .iter()
-            .map(|operand| format!("{}: {}", operand.name, operand.kind.name))
+            .map(|operand| format!("{}: {}", operand.name, operand.kind.rust_type))
             .collect::>(),
     );
 
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index c32b21b738..7904b1ef28 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -914,12 +914,12 @@ fn gen_inst_builder(inst: &Instruction, format: &InstructionFormat, fmt: &mut Fo
     let mut into_args = Vec::new();
     for op in &inst.operands_in {
         let t = if op.is_immediate() {
-            let t = format!("T{}{}", tmpl_types.len() + 1, op.kind.name);
+            let t = format!("T{}", tmpl_types.len() + 1);
             tmpl_types.push(format!("{}: Into<{}>", t, op.kind.rust_type));
             into_args.push(op.name);
             t
         } else {
-            op.kind.rust_type.clone()
+            op.kind.rust_type.to_string()
         };
         args.push(format!("{}: {}", op.name, t));
         args_doc.push(format!(
diff --git a/cranelift/codegen/meta/src/shared/entities.rs b/cranelift/codegen/meta/src/shared/entities.rs
index 48c3ce27b4..d4c6d6bfd7 100644
--- a/cranelift/codegen/meta/src/shared/entities.rs
+++ b/cranelift/codegen/meta/src/shared/entities.rs
@@ -35,28 +35,29 @@ pub(crate) struct EntityRefs {
 impl EntityRefs {
     pub fn new() -> Self {
         Self {
-            ebb: create("ebb", "An extended basic block in the same function.")
-                .rust_field_name("destination")
-                .build(),
+            ebb: create(
+                "destination",
+                "ir::Ebb",
+                "An extended basic block in the same function.",
+            )
+            .build(),
 
-            stack_slot: create("stack_slot", "A stack slot").build(),
+            stack_slot: create("stack_slot", "ir::StackSlot", "A stack slot").build(),
 
-            global_value: create("global_value", "A global value.").build(),
+            global_value: create("global_value", "ir::GlobalValue", "A global value.").build(),
 
-            sig_ref: create("sig_ref", "A function signature.").build(),
+            sig_ref: create("sig_ref", "ir::SigRef", "A function signature.").build(),
 
-            func_ref: create("func_ref", "An external function.").build(),
+            func_ref: create("func_ref", "ir::FuncRef", "An external function.").build(),
 
-            jump_table: create("jump_table", "A jump table.")
-                .rust_field_name("table")
-                .build(),
+            jump_table: create("table", "ir::JumpTable", "A jump table.").build(),
 
-            heap: create("heap", "A heap.").build(),
+            heap: create("heap", "ir::Heap", "A heap.").build(),
 
-            table: create("table", "A table.").build(),
+            table: create("table", "ir::Table", "A table.").build(),
 
-            varargs: Builder::new("variable_args", OperandKindFields::VariableArgs)
-                .doc(
+            varargs: Builder::new("", "&[Value]", OperandKindFields::VariableArgs)
+                .with_doc(
                     r#"
                         A variable size list of `value` operands.
 
@@ -71,6 +72,6 @@ impl EntityRefs {
 }
 
 /// Small helper to initialize an OperandBuilder with the right kind, for a given name and doc.
-fn create(name: &'static str, doc: &'static str) -> Builder {
-    Builder::new(name, OperandKindFields::EntityRef).doc(doc)
+fn create(format_field_name: &'static str, rust_type: &'static str, doc: &'static str) -> Builder {
+    Builder::new(format_field_name, rust_type, OperandKindFields::EntityRef).with_doc(doc)
 }
diff --git a/cranelift/codegen/meta/src/shared/immediates.rs b/cranelift/codegen/meta/src/shared/immediates.rs
index 182bd1a4b6..71e24beed8 100644
--- a/cranelift/codegen/meta/src/shared/immediates.rs
+++ b/cranelift/codegen/meta/src/shared/immediates.rs
@@ -76,45 +76,40 @@ pub(crate) struct Immediates {
 impl Immediates {
     pub fn new() -> Self {
         Self {
-            imm64: Builder::new_imm("imm64")
-                .doc("A 64-bit immediate integer.")
+            imm64: Builder::new_imm("imm", "ir::immediates::Imm64")
+                .with_doc("A 64-bit immediate integer.")
                 .build(),
 
-            uimm8: Builder::new_imm("uimm8")
-                .doc("An 8-bit immediate unsigned integer.")
+            uimm8: Builder::new_imm("imm", "ir::immediates::Uimm8")
+                .with_doc("An 8-bit immediate unsigned integer.")
                 .build(),
 
-            uimm32: Builder::new_imm("uimm32")
-                .doc("A 32-bit immediate unsigned integer.")
+            uimm32: Builder::new_imm("imm", "ir::immediates::Uimm32")
+                .with_doc("A 32-bit immediate unsigned integer.")
                 .build(),
 
-            uimm128: Builder::new_imm("uimm128")
-                .doc("A 128-bit immediate unsigned integer.")
-                .rust_type("ir::Immediate")
+            uimm128: Builder::new_imm("imm", "ir::Immediate")
+                .with_doc("A 128-bit immediate unsigned integer.")
                 .build(),
 
-            pool_constant: Builder::new_imm("poolConstant")
-                .doc("A constant stored in the constant pool.")
-                .rust_field_name("constant_handle")
-                .rust_type("ir::Constant")
+            pool_constant: Builder::new_imm("constant_handle", "ir::Constant")
+                .with_doc("A constant stored in the constant pool.")
                 .build(),
 
-            offset32: Builder::new_imm("offset32")
-                .doc("A 32-bit immediate signed offset.")
-                .rust_field_name("offset")
+            offset32: Builder::new_imm("offset", "ir::immediates::Offset32")
+                .with_doc("A 32-bit immediate signed offset.")
                 .build(),
 
-            ieee32: Builder::new_imm("ieee32")
-                .doc("A 32-bit immediate floating point number.")
+            ieee32: Builder::new_imm("imm", "ir::immediates::Ieee32")
+                .with_doc("A 32-bit immediate floating point number.")
                 .build(),
 
-            ieee64: Builder::new_imm("ieee64")
-                .doc("A 64-bit immediate floating point number.")
+            ieee64: Builder::new_imm("imm", "ir::immediates::Ieee64")
+                .with_doc("A 64-bit immediate floating point number.")
                 .build(),
 
-            boolean: Builder::new_imm("boolean")
-                .doc("An immediate boolean.")
-                .rust_type("bool")
+            boolean: Builder::new_imm("imm", "bool")
+                .with_doc("An immediate boolean.")
                 .build(),
 
             intcc: {
@@ -131,10 +126,8 @@ impl Immediates {
                 intcc_values.insert("ult", "UnsignedLessThan");
                 intcc_values.insert("of", "Overflow");
                 intcc_values.insert("nof", "NotOverflow");
-                Builder::new_enum("intcc", intcc_values)
-                    .doc("An integer comparison condition code.")
-                    .rust_field_name("cond")
-                    .rust_type("ir::condcodes::IntCC")
+                Builder::new_enum("cond", "ir::condcodes::IntCC", intcc_values)
+                    .with_doc("An integer comparison condition code.")
                     .build()
             },
 
@@ -154,22 +147,17 @@ impl Immediates {
                 floatcc_values.insert("ule", "UnorderedOrLessThanOrEqual");
                 floatcc_values.insert("ugt", "UnorderedOrGreaterThan");
                 floatcc_values.insert("uge", "UnorderedOrGreaterThanOrEqual");
-                Builder::new_enum("floatcc", floatcc_values)
-                    .doc("A floating point comparison condition code")
-                    .rust_field_name("cond")
-                    .rust_type("ir::condcodes::FloatCC")
+                Builder::new_enum("cond", "ir::condcodes::FloatCC", floatcc_values)
+                    .with_doc("A floating point comparison condition code")
                     .build()
             },
 
-            memflags: Builder::new_imm("memflags")
-                .doc("Memory operation flags")
-                .rust_field_name("flags")
-                .rust_type("ir::MemFlags")
+            memflags: Builder::new_imm("flags", "ir::MemFlags")
+                .with_doc("Memory operation flags")
                 .build(),
 
-            regunit: Builder::new_imm("regunit")
-                .doc("A register unit in the target ISA")
-                .rust_type("isa::RegUnit")
+            regunit: Builder::new_imm("regunit", "isa::RegUnit")
+                .with_doc("A register unit in the target ISA")
                 .build(),
 
             trapcode: {
@@ -178,10 +166,8 @@ impl Immediates {
                 trapcode_values.insert("heap_oob", "HeapOutOfBounds");
                 trapcode_values.insert("int_ovf", "IntegerOverflow");
                 trapcode_values.insert("int_divz", "IntegerDivisionByZero");
-                Builder::new_enum("trapcode", trapcode_values)
-                    .doc("A trap reason code.")
-                    .rust_field_name("code")
-                    .rust_type("ir::TrapCode")
+                Builder::new_enum("code", "ir::TrapCode", trapcode_values)
+                    .with_doc("A trap reason code.")
                     .build()
             },
         }

From d8b840d2f5ea565dbf378faf938e6886dfef2466 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 29 Oct 2019 16:57:11 +0100
Subject: [PATCH 2894/3084] [meta] Remove the OperandKindBuilder;

And replace it by constructors in OperandKind. There's a single optional
parameter function `set_doc` that remains, and didn't justify the whole
OperandKindBuilder concept to exist.
---
 cranelift/codegen/meta/src/cdsl/formats.rs    |  1 +
 .../codegen/meta/src/cdsl/instructions.rs     |  4 +-
 cranelift/codegen/meta/src/cdsl/operands.rs   | 77 ++++------------
 cranelift/codegen/meta/src/shared/entities.rs | 40 ++++-----
 .../codegen/meta/src/shared/immediates.rs     | 90 ++++++++-----------
 5 files changed, 78 insertions(+), 134 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/formats.rs b/cranelift/codegen/meta/src/cdsl/formats.rs
index 0978d61faf..e713a8bccb 100644
--- a/cranelift/codegen/meta/src/cdsl/formats.rs
+++ b/cranelift/codegen/meta/src/cdsl/formats.rs
@@ -49,6 +49,7 @@ pub(crate) struct InstructionFormat {
 pub(crate) struct FormatStructure {
     pub num_value_operands: usize,
     pub has_value_list: bool,
+    /// Tuples of (Rust field name / Rust type) for each immediate field.
     pub imm_field_names: Vec<(&'static str, &'static str)>,
 }
 
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 6b21bfaf10..86ef1e2a8a 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -1296,7 +1296,7 @@ impl Into for BoundInstruction {
 mod test {
     use super::*;
     use crate::cdsl::formats::InstructionFormatBuilder;
-    use crate::cdsl::operands::{OperandKindBuilder, OperandKindFields};
+    use crate::cdsl::operands::{OperandKind, OperandKindFields};
     use crate::cdsl::typevar::TypeSetBuilder;
     use crate::shared::types::Int::{I32, I64};
 
@@ -1304,7 +1304,7 @@ mod test {
         // Pretend the index string is &'static.
         let name = Box::leak(index.to_string().into_boxed_str());
         // Format's name / rust_type don't matter here.
-        let kind = OperandKindBuilder::new(name, name, field).build();
+        let kind = OperandKind::new(name, name, field);
         let operand = Operand::new(name, kind);
         operand
     }
diff --git a/cranelift/codegen/meta/src/cdsl/operands.rs b/cranelift/codegen/meta/src/cdsl/operands.rs
index d4c1a334c7..605df24862 100644
--- a/cranelift/codegen/meta/src/cdsl/operands.rs
+++ b/cranelift/codegen/meta/src/cdsl/operands.rs
@@ -100,7 +100,7 @@ impl Operand {
     }
 }
 
-type EnumValues = HashMap<&'static str, &'static str>;
+pub type EnumValues = HashMap<&'static str, &'static str>;
 
 #[derive(Clone, Debug)]
 pub(crate) enum OperandKindFields {
@@ -126,6 +126,23 @@ pub(crate) struct OperandKind {
 }
 
 impl OperandKind {
+    pub fn new(
+        rust_field_name: &'static str,
+        rust_type: &'static str,
+        fields: OperandKindFields,
+    ) -> Self {
+        Self {
+            rust_field_name,
+            rust_type,
+            fields,
+            doc: None,
+        }
+    }
+    pub fn with_doc(mut self, doc: &'static str) -> Self {
+        assert!(self.doc.is_none());
+        self.doc = Some(doc);
+        self
+    }
     fn doc(&self) -> Option<&str> {
         if let Some(doc) = &self.doc {
             return Some(doc);
@@ -142,12 +159,11 @@ impl OperandKind {
 
 impl Into for &TypeVar {
     fn into(self) -> OperandKind {
-        OperandKindBuilder::new(
+        OperandKind::new(
             "value",
             "ir::Value",
             OperandKindFields::TypeVar(self.into()),
         )
-        .build()
     }
 }
 impl Into for &OperandKind {
@@ -155,58 +171,3 @@ impl Into for &OperandKind {
         self.clone()
     }
 }
-
-pub(crate) struct OperandKindBuilder {
-    rust_field_name: &'static str,
-    rust_type: &'static str,
-    fields: OperandKindFields,
-    doc: Option<&'static str>,
-}
-
-impl OperandKindBuilder {
-    pub fn new(
-        rust_field_name: &'static str,
-        rust_type: &'static str,
-        fields: OperandKindFields,
-    ) -> Self {
-        Self {
-            rust_field_name,
-            rust_type,
-            fields,
-            doc: None,
-        }
-    }
-    pub fn new_imm(rust_field_name: &'static str, rust_type: &'static str) -> Self {
-        Self {
-            rust_field_name,
-            rust_type,
-            fields: OperandKindFields::ImmValue,
-            doc: None,
-        }
-    }
-    pub fn new_enum(
-        rust_field_name: &'static str,
-        rust_type: &'static str,
-        values: EnumValues,
-    ) -> Self {
-        Self {
-            rust_field_name,
-            rust_type,
-            fields: OperandKindFields::ImmEnum(values),
-            doc: None,
-        }
-    }
-    pub fn with_doc(mut self, doc: &'static str) -> Self {
-        assert!(self.doc.is_none());
-        self.doc = Some(doc);
-        self
-    }
-    pub fn build(self) -> OperandKind {
-        OperandKind {
-            rust_type: self.rust_type,
-            fields: self.fields,
-            rust_field_name: self.rust_field_name,
-            doc: self.doc,
-        }
-    }
-}
diff --git a/cranelift/codegen/meta/src/shared/entities.rs b/cranelift/codegen/meta/src/shared/entities.rs
index d4c6d6bfd7..068987c344 100644
--- a/cranelift/codegen/meta/src/shared/entities.rs
+++ b/cranelift/codegen/meta/src/shared/entities.rs
@@ -1,4 +1,9 @@
-use crate::cdsl::operands::{OperandKind, OperandKindBuilder as Builder, OperandKindFields};
+use crate::cdsl::operands::{OperandKind, OperandKindFields};
+
+/// Small helper to initialize an OperandBuilder with the right kind, for a given name and doc.
+fn new(format_field_name: &'static str, rust_type: &'static str, doc: &'static str) -> OperandKind {
+    OperandKind::new(format_field_name, rust_type, OperandKindFields::EntityRef).with_doc(doc)
+}
 
 pub(crate) struct EntityRefs {
     /// A reference to an extended basic block in the same function.
@@ -35,43 +40,34 @@ pub(crate) struct EntityRefs {
 impl EntityRefs {
     pub fn new() -> Self {
         Self {
-            ebb: create(
+            ebb: new(
                 "destination",
                 "ir::Ebb",
                 "An extended basic block in the same function.",
-            )
-            .build(),
+            ),
+            stack_slot: new("stack_slot", "ir::StackSlot", "A stack slot"),
 
-            stack_slot: create("stack_slot", "ir::StackSlot", "A stack slot").build(),
+            global_value: new("global_value", "ir::GlobalValue", "A global value."),
 
-            global_value: create("global_value", "ir::GlobalValue", "A global value.").build(),
+            sig_ref: new("sig_ref", "ir::SigRef", "A function signature."),
 
-            sig_ref: create("sig_ref", "ir::SigRef", "A function signature.").build(),
+            func_ref: new("func_ref", "ir::FuncRef", "An external function."),
 
-            func_ref: create("func_ref", "ir::FuncRef", "An external function.").build(),
+            jump_table: new("table", "ir::JumpTable", "A jump table."),
 
-            jump_table: create("table", "ir::JumpTable", "A jump table.").build(),
+            heap: new("heap", "ir::Heap", "A heap."),
 
-            heap: create("heap", "ir::Heap", "A heap.").build(),
+            table: new("table", "ir::Table", "A table."),
 
-            table: create("table", "ir::Table", "A table.").build(),
-
-            varargs: Builder::new("", "&[Value]", OperandKindFields::VariableArgs)
-                .with_doc(
-                    r#"
+            varargs: OperandKind::new("", "&[Value]", OperandKindFields::VariableArgs).with_doc(
+                r#"
                         A variable size list of `value` operands.
 
                         Use this to represent arguments passed to a function call, arguments
                         passed to an extended basic block, or a variable number of results
                         returned from an instruction.
                     "#,
-                )
-                .build(),
+            ),
         }
     }
 }
-
-/// Small helper to initialize an OperandBuilder with the right kind, for a given name and doc.
-fn create(format_field_name: &'static str, rust_type: &'static str, doc: &'static str) -> Builder {
-    Builder::new(format_field_name, rust_type, OperandKindFields::EntityRef).with_doc(doc)
-}
diff --git a/cranelift/codegen/meta/src/shared/immediates.rs b/cranelift/codegen/meta/src/shared/immediates.rs
index 71e24beed8..d8382e4067 100644
--- a/cranelift/codegen/meta/src/shared/immediates.rs
+++ b/cranelift/codegen/meta/src/shared/immediates.rs
@@ -1,4 +1,4 @@
-use crate::cdsl::operands::{OperandKind, OperandKindBuilder as Builder};
+use crate::cdsl::operands::{EnumValues, OperandKind, OperandKindFields};
 
 use std::collections::HashMap;
 
@@ -73,45 +73,40 @@ pub(crate) struct Immediates {
     pub trapcode: OperandKind,
 }
 
+fn new_imm(format_field_name: &'static str, rust_type: &'static str) -> OperandKind {
+    OperandKind::new(format_field_name, rust_type, OperandKindFields::ImmValue)
+}
+fn new_enum(
+    format_field_name: &'static str,
+    rust_type: &'static str,
+    values: EnumValues,
+) -> OperandKind {
+    OperandKind::new(
+        format_field_name,
+        rust_type,
+        OperandKindFields::ImmEnum(values),
+    )
+}
+
 impl Immediates {
     pub fn new() -> Self {
         Self {
-            imm64: Builder::new_imm("imm", "ir::immediates::Imm64")
-                .with_doc("A 64-bit immediate integer.")
-                .build(),
-
-            uimm8: Builder::new_imm("imm", "ir::immediates::Uimm8")
-                .with_doc("An 8-bit immediate unsigned integer.")
-                .build(),
-
-            uimm32: Builder::new_imm("imm", "ir::immediates::Uimm32")
-                .with_doc("A 32-bit immediate unsigned integer.")
-                .build(),
-
-            uimm128: Builder::new_imm("imm", "ir::Immediate")
-                .with_doc("A 128-bit immediate unsigned integer.")
-                .build(),
-
-            pool_constant: Builder::new_imm("constant_handle", "ir::Constant")
-                .with_doc("A constant stored in the constant pool.")
-                .build(),
-
-            offset32: Builder::new_imm("offset", "ir::immediates::Offset32")
-                .with_doc("A 32-bit immediate signed offset.")
-                .build(),
-
-            ieee32: Builder::new_imm("imm", "ir::immediates::Ieee32")
-                .with_doc("A 32-bit immediate floating point number.")
-                .build(),
-
-            ieee64: Builder::new_imm("imm", "ir::immediates::Ieee64")
-                .with_doc("A 64-bit immediate floating point number.")
-                .build(),
-
-            boolean: Builder::new_imm("imm", "bool")
-                .with_doc("An immediate boolean.")
-                .build(),
-
+            imm64: new_imm("imm", "ir::immediates::Imm64").with_doc("A 64-bit immediate integer."),
+            uimm8: new_imm("imm", "ir::immediates::Uimm8")
+                .with_doc("An 8-bit immediate unsigned integer."),
+            uimm32: new_imm("imm", "ir::immediates::Uimm32")
+                .with_doc("A 32-bit immediate unsigned integer."),
+            uimm128: new_imm("imm", "ir::Immediate")
+                .with_doc("A 128-bit immediate unsigned integer."),
+            pool_constant: new_imm("constant_handle", "ir::Constant")
+                .with_doc("A constant stored in the constant pool."),
+            offset32: new_imm("offset", "ir::immediates::Offset32")
+                .with_doc("A 32-bit immediate signed offset."),
+            ieee32: new_imm("imm", "ir::immediates::Ieee32")
+                .with_doc("A 32-bit immediate floating point number."),
+            ieee64: new_imm("imm", "ir::immediates::Ieee64")
+                .with_doc("A 64-bit immediate floating point number."),
+            boolean: new_imm("imm", "bool").with_doc("An immediate boolean."),
             intcc: {
                 let mut intcc_values = HashMap::new();
                 intcc_values.insert("eq", "Equal");
@@ -126,9 +121,8 @@ impl Immediates {
                 intcc_values.insert("ult", "UnsignedLessThan");
                 intcc_values.insert("of", "Overflow");
                 intcc_values.insert("nof", "NotOverflow");
-                Builder::new_enum("cond", "ir::condcodes::IntCC", intcc_values)
+                new_enum("cond", "ir::condcodes::IntCC", intcc_values)
                     .with_doc("An integer comparison condition code.")
-                    .build()
             },
 
             floatcc: {
@@ -147,28 +141,20 @@ impl Immediates {
                 floatcc_values.insert("ule", "UnorderedOrLessThanOrEqual");
                 floatcc_values.insert("ugt", "UnorderedOrGreaterThan");
                 floatcc_values.insert("uge", "UnorderedOrGreaterThanOrEqual");
-                Builder::new_enum("cond", "ir::condcodes::FloatCC", floatcc_values)
+                new_enum("cond", "ir::condcodes::FloatCC", floatcc_values)
                     .with_doc("A floating point comparison condition code")
-                    .build()
             },
 
-            memflags: Builder::new_imm("flags", "ir::MemFlags")
-                .with_doc("Memory operation flags")
-                .build(),
-
-            regunit: Builder::new_imm("regunit", "isa::RegUnit")
-                .with_doc("A register unit in the target ISA")
-                .build(),
-
+            memflags: new_imm("flags", "ir::MemFlags").with_doc("Memory operation flags"),
+            regunit: new_imm("regunit", "isa::RegUnit")
+                .with_doc("A register unit in the target ISA"),
             trapcode: {
                 let mut trapcode_values = HashMap::new();
                 trapcode_values.insert("stk_ovf", "StackOverflow");
                 trapcode_values.insert("heap_oob", "HeapOutOfBounds");
                 trapcode_values.insert("int_ovf", "IntegerOverflow");
                 trapcode_values.insert("int_divz", "IntegerDivisionByZero");
-                Builder::new_enum("code", "ir::TrapCode", trapcode_values)
-                    .with_doc("A trap reason code.")
-                    .build()
+                new_enum("code", "ir::TrapCode", trapcode_values).with_doc("A trap reason code.")
             },
         }
     }

From 52fb94d24c91576eee722996ed2a3b9e8b1967c9 Mon Sep 17 00:00:00 2001
From: "dependabot-preview[bot]"
 <27856297+dependabot-preview[bot]@users.noreply.github.com>
Date: Wed, 30 Oct 2019 13:54:10 +0000
Subject: [PATCH 2895/3084] Update object requirement from 0.14.0 to 0.15.0

Updates the requirements on [object](https://github.com/gimli-rs/object) to permit the latest version.
- [Release notes](https://github.com/gimli-rs/object/releases)
- [Commits](https://github.com/gimli-rs/object/compare/0.14.0...0.15.0)

Signed-off-by: dependabot-preview[bot] 
---
 cranelift/object/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 8577a0bfa7..402ecd3f91 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -11,7 +11,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-module = { path = "../cranelift-module", version = "0.46.1" }
-object = { version = "0.14.0", default-features = false, features = ["write"] }
+object = { version = "0.15.0", default-features = false, features = ["write"] }
 target-lexicon = "0.8.1"
 
 [dependencies.cranelift-codegen]

From 6de45ff8fc035feac723b01e2a30e933e023555d Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 22 Oct 2019 15:34:42 +0200
Subject: [PATCH 2896/3084] Update publish-all so it commits on a different
 branch.

This allows to open a PR for the changes to be submitted on the
repository, instead of silently pushing to master.
---
 cranelift/publish-all.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index b5f832b3a0..7d94ce2aac 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -33,9 +33,11 @@ cargo update
 #
 # Note that libraries need to be published in topological order.
 
+echo git checkout -b bump-version-to-$version
 echo git commit -a -m "\"Bump version to $version"\"
 echo git tag v$version
-echo git push
+echo git push origin bump-version-to-$version
+echo "# Don't forget to click the above link to open a pull-request!"
 echo git push origin v$version
 for crate in \
     entity bforest codegen/shared codegen/meta codegen frontend native \

From 7e725cf880dd5151e746083729a3c749453b0491 Mon Sep 17 00:00:00 2001
From: Josh Triplett 
Date: Wed, 30 Oct 2019 07:30:20 -0700
Subject: [PATCH 2897/3084] Migrate from failure to thiserror

The failure crate invents its own traits that don't use
std::error::Error (because failure predates certain features added to
Error); this prevents using ? on an error from failure in a function
using Error. The thiserror crate integrates with the standard Error
trait instead.
---
 cranelift/codegen/Cargo.toml          |  3 +--
 cranelift/codegen/src/isa/mod.rs      |  8 ++++----
 cranelift/codegen/src/result.rs       | 18 ++++++------------
 cranelift/codegen/src/settings.rs     | 10 +++++-----
 cranelift/codegen/src/verifier/mod.rs | 13 ++++---------
 cranelift/module/Cargo.toml           |  2 +-
 cranelift/module/src/module.rs        | 21 +++++++++------------
 cranelift/wasm/Cargo.toml             |  3 +--
 cranelift/wasm/src/environ/spec.rs    | 12 ++++++------
 9 files changed, 37 insertions(+), 53 deletions(-)

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index fc7a9023a8..5149c68ee4 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -16,13 +16,12 @@ edition = "2018"
 cranelift-codegen-shared = { path = "./shared", version = "0.46.1" }
 cranelift-entity = { path = "../cranelift-entity", version = "0.46.1" }
 cranelift-bforest = { path = "../cranelift-bforest", version = "0.46.1" }
-failure = { version = "0.1.1", default-features = false, features = ["derive"] }
-failure_derive = { version = "0.1.1", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
 smallvec = { version = "0.6.10" }
+thiserror = "1.0.4"
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
 # machine code. Integration tests that need external dependencies can be
diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index 067dff318f..66dde9621a 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -65,8 +65,8 @@ use crate::settings::SetResult;
 use crate::timing;
 use alloc::boxed::Box;
 use core::fmt;
-use failure_derive::Fail;
 use target_lexicon::{triple, Architecture, PointerWidth, Triple};
+use thiserror::Error;
 
 #[cfg(feature = "riscv")]
 mod riscv;
@@ -124,14 +124,14 @@ pub fn lookup_by_name(name: &str) -> Result {
 }
 
 /// Describes reason for target lookup failure
-#[derive(Fail, PartialEq, Eq, Copy, Clone, Debug)]
+#[derive(Error, PartialEq, Eq, Copy, Clone, Debug)]
 pub enum LookupError {
     /// Support for this target was disabled in the current build.
-    #[fail(display = "Support for this target is disabled")]
+    #[error("Support for this target is disabled")]
     SupportDisabled,
 
     /// Support for this target has not yet been implemented.
-    #[fail(display = "Support for this target has not been implemented yet")]
+    #[error("Support for this target has not been implemented yet")]
     Unsupported,
 }
 
diff --git a/cranelift/codegen/src/result.rs b/cranelift/codegen/src/result.rs
index b0af78a3d6..7a577d09f2 100644
--- a/cranelift/codegen/src/result.rs
+++ b/cranelift/codegen/src/result.rs
@@ -1,19 +1,19 @@
 //! Result and error types representing the outcome of compiling a function.
 
 use crate::verifier::VerifierErrors;
-use failure_derive::Fail;
+use thiserror::Error;
 
 /// A compilation error.
 ///
 /// When Cranelift fails to compile a function, it will return one of these error codes.
-#[derive(Fail, Debug, PartialEq, Eq)]
+#[derive(Error, Debug, PartialEq, Eq)]
 pub enum CodegenError {
     /// A list of IR verifier errors.
     ///
     /// This always represents a bug, either in the code that generated IR for Cranelift, or a bug
     /// in Cranelift itself.
-    #[fail(display = "Verifier errors:\n{}", _0)]
-    Verifier(#[cause] VerifierErrors),
+    #[error("Verifier errors")]
+    Verifier(#[from] VerifierErrors),
 
     /// An implementation limit was exceeded.
     ///
@@ -21,22 +21,16 @@ pub enum CodegenError {
     /// limits][limits] that cause compilation to fail when they are exceeded.
     ///
     /// [limits]: https://cranelift.readthedocs.io/en/latest/ir.html#implementation-limits
-    #[fail(display = "Implementation limit exceeded")]
+    #[error("Implementation limit exceeded")]
     ImplLimitExceeded,
 
     /// The code size for the function is too large.
     ///
     /// Different target ISAs may impose a limit on the size of a compiled function. If that limit
     /// is exceeded, compilation fails.
-    #[fail(display = "Code for function is too large")]
+    #[error("Code for function is too large")]
     CodeTooLarge,
 }
 
 /// A convenient alias for a `Result` that uses `CodegenError` as the error type.
 pub type CodegenResult = Result;
-
-impl From for CodegenError {
-    fn from(e: VerifierErrors) -> Self {
-        Self::Verifier(e)
-    }
-}
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index a20f0662f1..bb0e78a1af 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -26,7 +26,7 @@ use alloc::boxed::Box;
 use alloc::string::{String, ToString};
 use core::fmt;
 use core::str;
-use failure_derive::Fail;
+use thiserror::Error;
 
 /// A string-based configurator for settings groups.
 ///
@@ -165,18 +165,18 @@ impl Configurable for Builder {
 }
 
 /// An error produced when changing a setting.
-#[derive(Fail, Debug, PartialEq, Eq)]
+#[derive(Error, Debug, PartialEq, Eq)]
 pub enum SetError {
     /// No setting by this name exists.
-    #[fail(display = "No existing setting named '{}'", _0)]
+    #[error("No existing setting named '{0}'")]
     BadName(String),
 
     /// Type mismatch for setting (e.g., setting an enum setting as a bool).
-    #[fail(display = "Trying to set a setting with the wrong type")]
+    #[error("Trying to set a setting with the wrong type")]
     BadType,
 
     /// This is not a valid value for this setting.
-    #[fail(display = "Unexpected value for a setting, expected {}", _0)]
+    #[error("Unexpected value for a setting, expected {0}")]
     BadValue(String),
 }
 
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index b57add8de6..f31b4e2340 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -77,7 +77,7 @@ use alloc::string::String;
 use alloc::vec::Vec;
 use core::cmp::Ordering;
 use core::fmt::{self, Display, Formatter, Write};
-use failure_derive::Fail;
+use thiserror::Error;
 
 pub use self::cssa::verify_cssa;
 pub use self::liveness::verify_liveness;
@@ -127,7 +127,8 @@ mod liveness;
 mod locations;
 
 /// A verifier error.
-#[derive(Fail, Debug, PartialEq, Eq)]
+#[derive(Error, Debug, PartialEq, Eq)]
+#[error("{location}: {message}")]
 pub struct VerifierError {
     /// The entity causing the verifier error.
     pub location: AnyEntity,
@@ -135,12 +136,6 @@ pub struct VerifierError {
     pub message: String,
 }
 
-impl Display for VerifierError {
-    fn fmt(&self, f: &mut Formatter) -> fmt::Result {
-        write!(f, "{}: {}", self.location, self.message)
-    }
-}
-
 /// Result of a step in the verification process.
 ///
 /// Functions that return `VerifierStepResult<()>` should also take a
@@ -160,7 +155,7 @@ pub type VerifierStepResult = Result;
 pub type VerifierResult = Result;
 
 /// List of verifier errors.
-#[derive(Fail, Debug, Default, PartialEq, Eq)]
+#[derive(Error, Debug, Default, PartialEq, Eq)]
 pub struct VerifierErrors(pub Vec);
 
 impl VerifierErrors {
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 5923cd792f..9c2e162b5e 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -14,8 +14,8 @@ edition = "2018"
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.46.1" }
 hashbrown = { version = "0.6", optional = true }
-failure = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
+thiserror = "1.0.4"
 
 [features]
 default = ["std"]
diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index f45db2e79e..6b682101cb 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -11,11 +11,11 @@ use crate::Backend;
 use cranelift_codegen::binemit::{self, CodeInfo};
 use cranelift_codegen::entity::{entity_impl, PrimaryMap};
 use cranelift_codegen::{ir, isa, CodegenError, Context};
-use failure::Fail;
 use log::info;
 use std::borrow::ToOwned;
 use std::string::String;
 use std::vec::Vec;
+use thiserror::Error;
 
 /// A function identifier for use in the `Module` interface.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
@@ -121,32 +121,29 @@ pub struct FunctionDeclaration {
 }
 
 /// Error messages for all `Module` and `Backend` methods
-#[derive(Fail, Debug)]
+#[derive(Error, Debug)]
 pub enum ModuleError {
     /// Indicates an identifier was used before it was declared
-    #[fail(display = "Undeclared identifier: {}", _0)]
+    #[error("Undeclared identifier: {0}")]
     Undeclared(String),
     /// Indicates an identifier was used as data/function first, but then used as the other
-    #[fail(display = "Incompatible declaration of identifier: {}", _0)]
+    #[error("Incompatible declaration of identifier: {0}")]
     IncompatibleDeclaration(String),
     /// Indicates a function identifier was declared with a
     /// different signature than declared previously
-    #[fail(
-        display = "Function {} signature {:?} is incompatible with previous declaration {:?}",
-        _0, _2, _1
-    )]
+    #[error("Function {0} signature {2:?} is incompatible with previous declaration {1:?}")]
     IncompatibleSignature(String, ir::Signature, ir::Signature),
     /// Indicates an identifier was defined more than once
-    #[fail(display = "Duplicate definition of identifier: {}", _0)]
+    #[error("Duplicate definition of identifier: {0}")]
     DuplicateDefinition(String),
     /// Indicates an identifier was defined, but was declared as an import
-    #[fail(display = "Invalid to define identifier declared as an import: {}", _0)]
+    #[error("Invalid to define identifier declared as an import: {0}")]
     InvalidImportDefinition(String),
     /// Wraps a `cranelift-codegen` error
-    #[fail(display = "Compilation error: {}", _0)]
+    #[error("Compilation error: {0}")]
     Compilation(CodegenError),
     /// Wraps a generic error from a backend
-    #[fail(display = "Backend error: {}", _0)]
+    #[error("Backend error: {0}")]
     Backend(String),
 }
 
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 80feba60c6..6ea44409f9 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -16,10 +16,9 @@ cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default
 cranelift-entity = { path = "../cranelift-entity", version = "0.46.1" }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.46.1", default-features = false }
 hashbrown = { version = "0.6", optional = true }
-failure = { version = "0.1.1", default-features = false, features = ["derive"] }
-failure_derive = { version = "0.1.1", default-features = false }
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
+thiserror = "1.0.4"
 
 [dev-dependencies]
 wabt = "0.9.1"
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 46a823fc80..ad0a93236e 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -16,8 +16,8 @@ use cranelift_codegen::ir::immediates::Offset32;
 use cranelift_codegen::ir::{self, InstBuilder};
 use cranelift_codegen::isa::TargetFrontendConfig;
 use cranelift_frontend::FunctionBuilder;
-use failure_derive::Fail;
 use std::boxed::Box;
+use thiserror::Error;
 use wasmparser::BinaryReaderError;
 use wasmparser::Operator;
 
@@ -42,13 +42,13 @@ pub enum GlobalVariable {
 ///
 /// When a WebAssembly function can't be translated, one of these error codes will be returned
 /// to describe the failure.
-#[derive(Fail, Debug)]
+#[derive(Error, Debug)]
 pub enum WasmError {
     /// The input WebAssembly code is invalid.
     ///
     /// This error code is used by a WebAssembly translator when it encounters invalid WebAssembly
     /// code. This should never happen for validated WebAssembly code.
-    #[fail(display = "Invalid input WebAssembly code at offset {}: {}", _1, _0)]
+    #[error("Invalid input WebAssembly code at offset {offset}: {message}")]
     InvalidWebAssembly {
         /// A string describing the validation error.
         message: &'static str,
@@ -59,7 +59,7 @@ pub enum WasmError {
     /// A feature used by the WebAssembly code is not supported by the embedding environment.
     ///
     /// Embedding environments may have their own limitations and feature restrictions.
-    #[fail(display = "Unsupported feature: {}", _0)]
+    #[error("Unsupported feature: {0}")]
     Unsupported(std::string::String),
 
     /// An implementation limit was exceeded.
@@ -68,11 +68,11 @@ pub enum WasmError {
     /// limits][limits] that cause compilation to fail when they are exceeded.
     ///
     /// [limits]: https://cranelift.readthedocs.io/en/latest/ir.html#implementation-limits
-    #[fail(display = "Implementation limit exceeded")]
+    #[error("Implementation limit exceeded")]
     ImplLimitExceeded,
 
     /// Any user-defined error.
-    #[fail(display = "User error: {}", _0)]
+    #[error("User error: {0}")]
     User(std::string::String),
 }
 

From d6b3ca28b487f35234b3c3b1f95761a8ca3dabca Mon Sep 17 00:00:00 2001
From: Josh Triplett 
Date: Wed, 30 Oct 2019 08:02:16 -0700
Subject: [PATCH 2898/3084] Add and use a From impl from CodegenError to
 ModuleError

This will also improve reporting of the chain of errors.
---
 cranelift/module/src/module.rs | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 6b682101cb..34964a7b61 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -141,7 +141,7 @@ pub enum ModuleError {
     InvalidImportDefinition(String),
     /// Wraps a `cranelift-codegen` error
     #[error("Compilation error: {0}")]
-    Compilation(CodegenError),
+    Compilation(#[from] CodegenError),
     /// Wraps a generic error from a backend
     #[error("Backend error: {0}")]
     Backend(String),
@@ -549,9 +549,7 @@ where
             func,
             ctx.func.display(self.backend.isa())
         );
-        let CodeInfo { total_size, .. } = ctx
-            .compile(self.backend.isa())
-            .map_err(ModuleError::Compilation)?;
+        let CodeInfo { total_size, .. } = ctx.compile(self.backend.isa())?;
         let info = &self.contents.functions[func];
         if info.compiled.is_some() {
             return Err(ModuleError::DuplicateDefinition(info.decl.name.clone()));

From 8c2d9fd32f9bc44d412bb32741161f32e578e40b Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 29 Oct 2019 18:20:42 +0100
Subject: [PATCH 2899/3084] Tweak comments in regalloc code.

---
 cranelift/codegen/src/regalloc/affinity.rs | 4 +---
 cranelift/codegen/src/regalloc/liveness.rs | 9 +++------
 2 files changed, 4 insertions(+), 9 deletions(-)

diff --git a/cranelift/codegen/src/regalloc/affinity.rs b/cranelift/codegen/src/regalloc/affinity.rs
index b7fa3090ca..efcc4dabfa 100644
--- a/cranelift/codegen/src/regalloc/affinity.rs
+++ b/cranelift/codegen/src/regalloc/affinity.rs
@@ -92,10 +92,8 @@ impl Affinity {
                 // to change anything.
                 if constraint.kind != ConstraintKind::Stack && !constraint.regclass.has_subclass(rc)
                 {
-                    // If the register classes don't overlap, `intersect` returns `Unassigned`, and
-                    // we just keep our previous affinity.
+                    // If the register classes overlap, try to shrink our preferred register class.
                     if let Some(subclass) = constraint.regclass.intersect_index(reginfo.rc(rc)) {
-                        // This constraint shrinks our preferred register class.
                         *self = Self::Reg(subclass);
                     }
                 }
diff --git a/cranelift/codegen/src/regalloc/liveness.rs b/cranelift/codegen/src/regalloc/liveness.rs
index 1b9784a198..f195645809 100644
--- a/cranelift/codegen/src/regalloc/liveness.rs
+++ b/cranelift/codegen/src/regalloc/liveness.rs
@@ -389,7 +389,6 @@ impl Liveness {
         // The liveness computation needs to visit all uses, but the order doesn't matter.
         // TODO: Perhaps this traversal of the function could be combined with a dead code
         // elimination pass if we visit a post-order of the dominator tree?
-        // TODO: Resolve value aliases while we're visiting instructions?
         for ebb in func.layout.ebbs() {
             // Make sure we have created live ranges for dead EBB parameters.
             // TODO: If these parameters are really dead, we could remove them, except for the
@@ -436,11 +435,9 @@ impl Liveness {
 
 impl Index for Liveness {
     type Output = LiveRange;
-
     fn index(&self, index: Value) -> &LiveRange {
-        match self.ranges.get(index) {
-            Some(lr) => lr,
-            None => panic!("{} has no live range", index),
-        }
+        self.ranges
+            .get(index)
+            .unwrap_or_else(|| panic!("{} has no live range", index))
     }
 }

From e45c2831942fc071e624bda8f827d0593d49ac2d Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 30 Oct 2019 17:53:31 -0700
Subject: [PATCH 2900/3084] Enable documentation of private items in
 cranelift-codegen/meta; fixes #832

---
 cranelift/codegen/meta/Cargo.toml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index c11a7333db..23507c7874 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -15,3 +15,6 @@ cranelift-entity = { path = "../../cranelift-entity", version = "0.46.1" }
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "CraneStation/cranelift" }
+
+[package.metadata.docs.rs]
+rustdoc-args = [ "--document-private-items" ]

From f19456640cce8d370ce181c5a433db6942293ae3 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 30 Oct 2019 17:54:04 -0700
Subject: [PATCH 2901/3084] Add documentation for top-level items in
 cranelift-codegen/meta

---
 cranelift/codegen/meta/src/cdsl/mod.rs      | 1 +
 cranelift/codegen/meta/src/default_map.rs   | 1 +
 cranelift/codegen/meta/src/error.rs         | 1 +
 cranelift/codegen/meta/src/gen_inst.rs      | 1 +
 cranelift/codegen/meta/src/gen_legalizer.rs | 1 +
 cranelift/codegen/meta/src/gen_registers.rs | 1 +
 cranelift/codegen/meta/src/gen_settings.rs  | 1 +
 cranelift/codegen/meta/src/isa/mod.rs       | 1 +
 cranelift/codegen/meta/src/lib.rs           | 3 +++
 cranelift/codegen/meta/src/unique_table.rs  | 1 +
 10 files changed, 12 insertions(+)

diff --git a/cranelift/codegen/meta/src/cdsl/mod.rs b/cranelift/codegen/meta/src/cdsl/mod.rs
index 370b442ae4..698b64dff3 100644
--- a/cranelift/codegen/meta/src/cdsl/mod.rs
+++ b/cranelift/codegen/meta/src/cdsl/mod.rs
@@ -39,6 +39,7 @@ macro_rules! predicate {
     };
 }
 
+/// A macro that joins boolean settings into a list (e.g. `preset!(feature_a && feature_b)`).
 #[macro_export]
 macro_rules! preset {
     () => {
diff --git a/cranelift/codegen/meta/src/default_map.rs b/cranelift/codegen/meta/src/default_map.rs
index 951a9b2491..3a2be05dac 100644
--- a/cranelift/codegen/meta/src/default_map.rs
+++ b/cranelift/codegen/meta/src/default_map.rs
@@ -1,3 +1,4 @@
+//! Trait for extending `HashMap` with `get_or_default`.
 use std::collections::HashMap;
 use std::hash::Hash;
 
diff --git a/cranelift/codegen/meta/src/error.rs b/cranelift/codegen/meta/src/error.rs
index 316e69bbc5..4cbf3d8285 100644
--- a/cranelift/codegen/meta/src/error.rs
+++ b/cranelift/codegen/meta/src/error.rs
@@ -1,3 +1,4 @@
+//! Error returned during meta code-generation.
 use std::fmt;
 use std::io;
 
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index 7904b1ef28..ffadd22d57 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -1,3 +1,4 @@
+//! Generate instruction data (including opcodes, formats, builders, etc.).
 use std::fmt;
 
 use cranelift_codegen_shared::constant_hash;
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index b73b67b754..ede525e8bc 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -1,3 +1,4 @@
+//! Generate transformations to legalize instructions without encodings.
 use crate::cdsl::ast::{Def, DefPool, Expr, VarPool};
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::operands::Operand;
diff --git a/cranelift/codegen/meta/src/gen_registers.rs b/cranelift/codegen/meta/src/gen_registers.rs
index df34a20899..fbb61beb37 100644
--- a/cranelift/codegen/meta/src/gen_registers.rs
+++ b/cranelift/codegen/meta/src/gen_registers.rs
@@ -1,3 +1,4 @@
+//! Generate the ISA-specific registers.
 use crate::cdsl::isa::TargetIsa;
 use crate::cdsl::regs::{RegBank, RegClass};
 use crate::error;
diff --git a/cranelift/codegen/meta/src/gen_settings.rs b/cranelift/codegen/meta/src/gen_settings.rs
index a40afbdf4a..2ed5941b80 100644
--- a/cranelift/codegen/meta/src/gen_settings.rs
+++ b/cranelift/codegen/meta/src/gen_settings.rs
@@ -1,3 +1,4 @@
+//! Generate the ISA-specific settings.
 use std::collections::HashMap;
 
 use cranelift_codegen_shared::constant_hash::{generate_table, simple_hash};
diff --git a/cranelift/codegen/meta/src/isa/mod.rs b/cranelift/codegen/meta/src/isa/mod.rs
index 45b68e7c45..39cd913300 100644
--- a/cranelift/codegen/meta/src/isa/mod.rs
+++ b/cranelift/codegen/meta/src/isa/mod.rs
@@ -1,3 +1,4 @@
+//! Define supported ISAs; includes ISA-specific instructions, encodings, registers, settings, etc.
 use crate::cdsl::isa::TargetIsa;
 use crate::shared::Definitions as SharedDefinitions;
 use std::fmt;
diff --git a/cranelift/codegen/meta/src/lib.rs b/cranelift/codegen/meta/src/lib.rs
index 56417002c7..796e2a110d 100644
--- a/cranelift/codegen/meta/src/lib.rs
+++ b/cranelift/codegen/meta/src/lib.rs
@@ -1,3 +1,5 @@
+//! This crate generates Rust sources for use by
+//! [`cranelift_codegen`](../cranelift_codegen/index.html).
 #[macro_use]
 mod cdsl;
 mod srcgen;
@@ -17,6 +19,7 @@ mod default_map;
 mod shared;
 mod unique_table;
 
+/// Generate an ISA from an architecture string (e.g. "x86_64").
 pub fn isa_from_arch(arch: &str) -> Result {
     isa::Isa::from_arch(arch).ok_or_else(|| format!("no supported isa found for arch `{}`", arch))
 }
diff --git a/cranelift/codegen/meta/src/unique_table.rs b/cranelift/codegen/meta/src/unique_table.rs
index 949aa6b9d3..65ef7e8b4a 100644
--- a/cranelift/codegen/meta/src/unique_table.rs
+++ b/cranelift/codegen/meta/src/unique_table.rs
@@ -1,3 +1,4 @@
+//! An index-accessed table implementation that avoids duplicate entries.
 use std::collections::HashMap;
 use std::hash::Hash;
 use std::slice;

From 96f5a6b56141a55e606903afb0e5c3bd4089890b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 31 Oct 2019 09:17:27 -0700
Subject: [PATCH 2902/3084] Fix documentation link issues

---
 cranelift/codegen/meta/src/cdsl/ast.rs | 10 ++++++----
 cranelift/codegen/src/ir/constant.rs   |  8 ++++----
 2 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index e2e661de8b..e4eb202be3 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -390,10 +390,12 @@ impl VarPool {
     }
 }
 
-/// Contains constants created in the AST that must be inserted into the true [ConstantPool]
-/// (cranelift_codegen::ir::ConstantPool) when the legalizer code is generated. The constant data
-/// is named in the order it is inserted; inserting data using [insert]
-/// (cranelift_codegen_meta::cdsl::ast::insert) will avoid duplicates.
+/// Contains constants created in the AST that must be inserted into the true [ConstantPool] when
+/// the legalizer code is generated. The constant data is named in the order it is inserted;
+/// inserting data using [insert] will avoid duplicates.
+///
+/// [ConstantPool]: ../../../cranelift_codegen/ir/constant/struct.ConstantPool.html
+/// [insert]: ConstPool::insert
 pub(crate) struct ConstPool {
     pool: Vec>,
 }
diff --git a/cranelift/codegen/src/ir/constant.rs b/cranelift/codegen/src/ir/constant.rs
index edd939f90d..2dc95544c8 100644
--- a/cranelift/codegen/src/ir/constant.rs
+++ b/cranelift/codegen/src/ir/constant.rs
@@ -21,8 +21,9 @@ use cranelift_entity::EntityRef;
 
 /// This type describes the actual constant data. Note that the bytes stored in this structure are
 /// expected to be in little-endian order; this is due to ease-of-use when interacting with
-/// WebAssembly values, which are [little-endian by design]
-/// (https://github.com/WebAssembly/design/blob/master/Portability.md).
+/// WebAssembly values, which are [little-endian by design].
+///
+/// [little-endian by design]: https://github.com/WebAssembly/design/blob/master/Portability.md
 #[derive(Clone, Hash, Eq, PartialEq, Debug, Default)]
 pub struct ConstantData(Vec);
 
@@ -116,8 +117,7 @@ impl fmt::Display for ConstantData {
 impl FromStr for ConstantData {
     type Err = &'static str;
 
-    /// Parse a hexadecimal string to `ConstantData`. This is the inverse of [ConstantData::fmt]
-    /// (cranelift_codegen::ir::ConstantData::fmt).
+    /// Parse a hexadecimal string to `ConstantData`. This is the inverse of `Display::fmt`.
     ///
     /// ```
     /// use cranelift_codegen::ir::ConstantData;

From 8307ad3f3b206fde6beccfc706c588e20f02bcc1 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 31 Oct 2019 08:50:53 -0700
Subject: [PATCH 2903/3084] Build documentation with private items in
 cranelift-codegen/meta

---
 .azure-pipelines.yml  | 4 +++-
 cranelift/test-all.sh | 6 ++++--
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml
index f3935c5af8..2fb328db32 100644
--- a/.azure-pipelines.yml
+++ b/.azure-pipelines.yml
@@ -33,7 +33,9 @@ jobs:
   - checkout: self
     submodules: true
   - template: ci/azure-install-rust.yml
-  - script: cargo doc
+  - script: |
+      cargo doc --all --exclude cranelift-codegen-meta
+      cargo doc --package cranelift-codegen-meta --document-private-items
     displayName: Build documentation
   - script: cargo install cargo-deadlinks
     displayName: Install cargo-deadlinks
diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index d3ad831274..fb9f6578d7 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -69,14 +69,16 @@ ensure_installed() {
 # Make sure the documentation builds.
 banner "Rust documentation: $topdir/target/doc/cranelift/index.html"
 if has_toolchain nightly; then
-    cargo +nightly doc
+    cargo +nightly doc --all --exclude cranelift-codegen-meta
+    cargo +nightly doc --package cranelift-codegen-meta --document-private-items
 
     # Make sure the documentation doesn't have broken links.
     banner "Rust documentation link test"
     ensure_installed cargo-deadlinks
     find ./target/doc -maxdepth 1 -type d -name "cranelift*" | xargs -I{} cargo deadlinks --dir {}
 else
-    cargo doc
+    cargo doc --all --exclude cranelift-codegen-meta
+    cargo doc --package cranelift-codegen-meta --document-private-items
     echo "nightly toolchain not found, some documentation links will not work"
 fi
 

From d1c53a18dfe9dc469c94f729ab431ae4a809af3e Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Thu, 31 Oct 2019 10:05:10 -0700
Subject: [PATCH 2904/3084] Document the `test run` filetest mode (#1189)

* Document the `test run` filetest mode

* Clarify that `test run` functions can return any boolean
---
 cranelift/docs/testing.rst | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/cranelift/docs/testing.rst b/cranelift/docs/testing.rst
index af2819cfc3..b0d3361dc8 100644
--- a/cranelift/docs/testing.rst
+++ b/cranelift/docs/testing.rst
@@ -376,3 +376,24 @@ which is normally used to compile code. This type of test often depends
 on assertions or verifier errors, but it is also possible to use
 filecheck directives which will be matched against the final form of the
 Cranelift IR right before binary machine code emission.
+
+`test run`
+----------
+
+Compile and execute a function.
+
+Add a ``; run`` directive after each function that should be executed. These
+functions must have the signature ``() -> bNN`` where ``bNN`` is some sort of
+boolean, e.g. ``b1`` or ``b32``. A ``true`` value is interpreted as a successful
+test execution, whereas a ``false`` value is interpreted as a failed test.
+
+Example::
+
+    test run
+
+    function %trivial_test() -> b1 {
+    ebb0:
+        v0 = bconst.b1 true
+        return v0
+    }
+    ; run

From d8fa5dcb293dd2bcda6eb7c33d378196502051ca Mon Sep 17 00:00:00 2001
From: "Oliver Scherer @ Cosmian"
 <52913081+oli-cosmian@users.noreply.github.com>
Date: Tue, 22 Oct 2019 13:32:29 +0200
Subject: [PATCH 2905/3084] Update outdated documentation

---
 cranelift/frontend/src/variable.rs | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/cranelift/frontend/src/variable.rs b/cranelift/frontend/src/variable.rs
index 9e7c260248..e96fe7d949 100644
--- a/cranelift/frontend/src/variable.rs
+++ b/cranelift/frontend/src/variable.rs
@@ -1,9 +1,7 @@
 //! A basic `Variable` implementation.
-//!
-//! `FunctionBuilderContext`, `FunctionBuilder`, and related types have a `Variable`
-//! type parameter, to allow frontends that identify variables with
-//! their own index types to use them directly. Frontends which don't
-//! can use the `Variable` defined here.
+//! 
+//! Frontends can use any indexing scheme they see fit and
+//! generate the appropriate `Variable` instances.
 
 use core::u32;
 use cranelift_codegen::entity::EntityRef;

From 9903c75f82637f0990ef2222c80ee76d0dcb16b5 Mon Sep 17 00:00:00 2001
From: "Oliver Scherer @ Cosmian"
 <52913081+oli-cosmian@users.noreply.github.com>
Date: Tue, 22 Oct 2019 14:53:10 +0200
Subject: [PATCH 2906/3084] Update variable.rs

---
 cranelift/frontend/src/variable.rs | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/cranelift/frontend/src/variable.rs b/cranelift/frontend/src/variable.rs
index e96fe7d949..f83c5ad6a3 100644
--- a/cranelift/frontend/src/variable.rs
+++ b/cranelift/frontend/src/variable.rs
@@ -1,7 +1,12 @@
 //! A basic `Variable` implementation.
-//! 
+//!
 //! Frontends can use any indexing scheme they see fit and
 //! generate the appropriate `Variable` instances.
+//!
+//! Note: The `Variable` is used by Cranelift to index an array containing
+//! information about your mutable variables. Thus, when you create a new
+//! `Variable` you should make sure that the index is provided by a counter
+//! incremented by 1 each time you encounter a new mutable variable.
 
 use core::u32;
 use cranelift_codegen::entity::EntityRef;

From e1b0f1f9904aa5d77ef597c2e17e66dbfac12ecb Mon Sep 17 00:00:00 2001
From: Oliver Scherer 
Date: Mon, 4 Nov 2019 12:11:59 +0100
Subject: [PATCH 2907/3084] Use less vague language

---
 cranelift/frontend/src/variable.rs | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/cranelift/frontend/src/variable.rs b/cranelift/frontend/src/variable.rs
index f83c5ad6a3..9a40b9dfe9 100644
--- a/cranelift/frontend/src/variable.rs
+++ b/cranelift/frontend/src/variable.rs
@@ -3,10 +3,10 @@
 //! Frontends can use any indexing scheme they see fit and
 //! generate the appropriate `Variable` instances.
 //!
-//! Note: The `Variable` is used by Cranelift to index an array containing
-//! information about your mutable variables. Thus, when you create a new
-//! `Variable` you should make sure that the index is provided by a counter
-//! incremented by 1 each time you encounter a new mutable variable.
+//! Note: The `Variable` is used by Cranelift to index into densely allocated
+//! arrays containing information about your mutable variables
+//! Thus, make sure that Variable's indexes are allocated contiguously and
+//! starting at `0`.
 
 use core::u32;
 use cranelift_codegen::entity::EntityRef;

From b4d92f0978154cf36d25774bc4359e02bfaeb83e Mon Sep 17 00:00:00 2001
From: "dependabot-preview[bot]"
 <27856297+dependabot-preview[bot]@users.noreply.github.com>
Date: Mon, 4 Nov 2019 13:16:02 +0000
Subject: [PATCH 2908/3084] Update indicatif requirement from 0.12.0 to 0.13.0

Updates the requirements on [indicatif](https://github.com/mitsuhiko/indicatif) to permit the latest version.
- [Release notes](https://github.com/mitsuhiko/indicatif/releases)
- [Commits](https://github.com/mitsuhiko/indicatif/compare/0.12.0...0.13.0)

Signed-off-by: dependabot-preview[bot] 
---
 cranelift/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 667104bcd4..be106c78e3 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -42,7 +42,7 @@ wabt = { version = "0.9.1", optional = true }
 target-lexicon = "0.8.1"
 pretty_env_logger = "0.3.0"
 file-per-thread-logger = "0.1.2"
-indicatif = "0.12.0"
+indicatif = "0.13.0"
 walkdir = "2.2"
 
 [features]

From 811b5e0a7bb75f1674f6222250e4faeb4efb92d3 Mon Sep 17 00:00:00 2001
From: "dependabot-preview[bot]"
 <27856297+dependabot-preview[bot]@users.noreply.github.com>
Date: Mon, 4 Nov 2019 13:18:21 +0000
Subject: [PATCH 2909/3084] Update smallvec requirement from 0.6.10 to 1.0.0

Updates the requirements on [smallvec](https://github.com/servo/rust-smallvec) to permit the latest version.
- [Release notes](https://github.com/servo/rust-smallvec/releases)
- [Commits](https://github.com/servo/rust-smallvec/compare/v0.6.10...v1.0.0)

Signed-off-by: dependabot-preview[bot] 
---
 cranelift/codegen/Cargo.toml  | 2 +-
 cranelift/frontend/Cargo.toml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 5149c68ee4..22d2b9b295 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -20,7 +20,7 @@ hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
-smallvec = { version = "0.6.10" }
+smallvec = { version = "1.0.0" }
 thiserror = "1.0.4"
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 02a61441af..f7c003b96a 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -15,7 +15,7 @@ cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
-smallvec = { version = "0.6.10" }
+smallvec = { version = "1.0.0" }
 
 [features]
 default = ["std", "basic-blocks"]

From a9868de3d8b81a6c80dd0aa526e938b7129a842d Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 4 Nov 2019 15:30:17 -0800
Subject: [PATCH 2910/3084] Bump version to 0.47.0

---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  2 +-
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 19 files changed, 68 insertions(+), 68 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index be106c78e3..d97571d817 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.46.1"
+version = "0.47.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.46.1" }
-cranelift-entity = { path = "cranelift-entity", version = "0.46.1" }
-cranelift-reader = { path = "cranelift-reader", version = "0.46.1" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.46.1" }
-cranelift-serde = { path = "cranelift-serde", version = "0.46.1", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.46.1", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.46.1" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.46.1" }
-cranelift-module = { path = "cranelift-module", version = "0.46.1" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.46.1" }
-cranelift-object = { path = "cranelift-object", version = "0.46.1" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.46.1" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.46.1" }
-cranelift = { path = "cranelift-umbrella", version = "0.46.1" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.47.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.47.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.47.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.47.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.47.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.47.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.47.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.47.0" }
+cranelift-module = { path = "cranelift-module", version = "0.47.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.47.0" }
+cranelift-object = { path = "cranelift-object", version = "0.47.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.47.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.47.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.47.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 9abe721b71..af3bddd6c5 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.46.1"
+version = "0.47.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.46.1", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.47.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 22d2b9b295..3ae2e23053 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.46.1"
+version = "0.47.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.46.1" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.46.1" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.46.1" }
+cranelift-codegen-shared = { path = "./shared", version = "0.47.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.47.0" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.47.0" }
 hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
@@ -28,7 +28,7 @@ thiserror = "1.0.4"
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.46.1" }
+cranelift-codegen-meta = { path = "meta", version = "0.47.0" }
 
 [features]
 default = ["std", "basic-blocks"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 23507c7874..0dad16727d 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.46.1"
+version = "0.47.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.46.1" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.46.1" }
+cranelift-codegen-shared = { path = "../shared", version = "0.47.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.47.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 098ed8b6cc..472e670af0 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.46.1"
+version = "0.47.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 152520be38..67c11e8986 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.46.1"
+version = "0.47.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index db77e2e0e5..2b4984a4a0 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.46.1"
+version = "0.47.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.46.1" }
+cranelift-module = { path = "../cranelift-module", version = "0.47.0" }
 faerie = "0.11.0"
 goblin = "0.0.24"
 failure = "0.1.2"
@@ -18,7 +18,7 @@ target-lexicon = "0.8.1"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.46.1"
+version = "0.47.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 5a359c8e22..c76ee4d478 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.46.1"
+version = "0.47.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.46.1" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.46.1" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.46.1" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.47.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.47.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.47.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index f7c003b96a..6d4e74c464 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.46.1"
+version = "0.47.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", default-features = false }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 9c2e162b5e..bcd6ad4132 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.46.1"
+version = "0.47.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.46.1" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.47.0" }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 thiserror = "1.0.4"
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 10399d046f..954562b25c 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.46.1"
+version = "0.47.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", default-features = false }
 target-lexicon = "0.8.1"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 402ecd3f91..61401a3d65 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.46.1"
+version = "0.47.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,13 +10,13 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.46.1" }
+cranelift-module = { path = "../cranelift-module", version = "0.47.0" }
 object = { version = "0.15.0", default-features = false, features = ["write"] }
 target-lexicon = "0.8.1"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.46.1"
+version = "0.47.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index fae98ff23f..2d11963306 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.46.1"
+version = "0.47.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.46.1" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.47.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 7d94ce2aac..66fe655c08 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.46.1"
+version="0.47.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 14feaee8cb..b1a56abe3b 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.46.1"
+version = "0.47.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0" }
 target-lexicon = "0.8.1"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 348b24d65c..c0e5252d48 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.46.1"
+version = "0.47.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.46.1" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.47.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 0ff896d3bc..16fddb6845 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.46.1"
+version = "0.47.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.46.1" }
-cranelift-native = { path = "../cranelift-native", version = "0.46.1" }
+cranelift-module = { path = "../cranelift-module", version = "0.47.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.47.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -20,7 +20,7 @@ memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.46.1"
+version = "0.47.0"
 default-features = false
 features = ["std"]
 
@@ -32,9 +32,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.46.1" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.46.1" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.46.1" }
+cranelift = { path = "../cranelift-umbrella", version = "0.47.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.47.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.47.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 912ec3cf0d..c8fc3f42ed 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.46.1"
+version = "0.47.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.46.1", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.47.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 6ea44409f9..cf0675b0cf 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.46.1"
+version = "0.47.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.39.2", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.46.1", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.46.1" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.46.1", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.47.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.47.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }

From 2eea366530b2139422a39944c0b1a65d4ecb5cee Mon Sep 17 00:00:00 2001
From: "dependabot-preview[bot]"
 <27856297+dependabot-preview[bot]@users.noreply.github.com>
Date: Tue, 5 Nov 2019 07:48:14 +0000
Subject: [PATCH 2911/3084] Update goblin requirement from 0.0.24 to 0.1.0

Updates the requirements on [goblin](https://github.com/m4b/goblin) to permit the latest version.
- [Release notes](https://github.com/m4b/goblin/releases)
- [Changelog](https://github.com/m4b/goblin/blob/master/CHANGELOG.md)
- [Commits](https://github.com/m4b/goblin/commits)

Signed-off-by: dependabot-preview[bot] 
---
 cranelift/faerie/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 2b4984a4a0..6bd9f120f5 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -12,7 +12,7 @@ edition = "2018"
 [dependencies]
 cranelift-module = { path = "../cranelift-module", version = "0.47.0" }
 faerie = "0.11.0"
-goblin = "0.0.24"
+goblin = "0.1.0"
 failure = "0.1.2"
 target-lexicon = "0.8.1"
 

From 8923bac7e87672724e07cab9b7810815053ab506 Mon Sep 17 00:00:00 2001
From: Peter Huene 
Date: Tue, 5 Nov 2019 13:14:30 -0800
Subject: [PATCH 2912/3084] Implement emitting Windows unwind information for
 fastcall functions. (#1155)

* Implement emitting Windows unwind information for fastcall functions.

This commit implements emitting Windows unwind information for x64 fastcall
calling convention functions.

The unwind information can be used to construct a Windows function table at
runtime for JIT'd code, enabling stack walking and unwinding by the operating
system.

* Address code review feedback.

This commit addresses code review feedback:

* Remove unnecessary unsafe code.
* Emit the unwind information always as little endian.
* Fix comments.

A dependency from cranelift-codegen to the byteorder crate was added.
The byteorder crate is a no-dependencies crate with a reasonable
abstraction for writing binary data for a specific endianness.

* Address code review feedback.

* Disable default features for the `byteorder` crate.
* Add a comment regarding the Windows ABI unwind code numerical values.
* Panic if we encounter a Windows function with a prologue greater than 256
  bytes in size.
---
 cranelift/codegen/Cargo.toml                  |   1 +
 cranelift/codegen/src/context.rs              |  10 +
 cranelift/codegen/src/ir/function.rs          |   7 +
 cranelift/codegen/src/isa/mod.rs              |   8 +
 cranelift/codegen/src/isa/x86/abi.rs          |  18 +-
 cranelift/codegen/src/isa/x86/mod.rs          |   9 +
 cranelift/codegen/src/isa/x86/unwind.rs       | 508 ++++++++++++++++++
 cranelift/filetests/Cargo.toml                |   1 +
 .../isa/x86/windows_fastcall_x64_unwind.clif  | 221 ++++++++
 cranelift/filetests/src/lib.rs                |   2 +
 cranelift/filetests/src/test_unwind.rs        | 198 +++++++
 11 files changed, 979 insertions(+), 4 deletions(-)
 create mode 100644 cranelift/codegen/src/isa/x86/unwind.rs
 create mode 100644 cranelift/filetests/filetests/isa/x86/windows_fastcall_x64_unwind.clif
 create mode 100644 cranelift/filetests/src/test_unwind.rs

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 3ae2e23053..1d45b5b50a 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -22,6 +22,7 @@ log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
 smallvec = { version = "1.0.0" }
 thiserror = "1.0.4"
+byteorder = { version = "1.3.2", default-features = false }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
 # machine code. Integration tests that need external dependencies can be
diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index ccf216d214..353b469bb6 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -193,6 +193,16 @@ impl Context {
         sink.info
     }
 
+    /// Emit unwind information.
+    ///
+    /// Requires that the function layout be calculated (see `relax_branches`).
+    ///
+    /// Only some calling conventions (e.g. Windows fastcall) will have unwind information.
+    /// This is a no-op if the function has no unwind information.
+    pub fn emit_unwind_info(&self, isa: &dyn TargetIsa, mem: &mut Vec) {
+        isa.emit_unwind_info(&self.func, mem);
+    }
+
     /// Run the verifier on the function.
     ///
     /// Also check that the dominator tree and control flow graph are consistent with the function.
diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index 00827240d9..913c2d7be4 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -83,6 +83,11 @@ pub struct Function {
     /// Track the original source location for each instruction. The source locations are not
     /// interpreted by Cranelift, only preserved.
     pub srclocs: SourceLocs,
+
+    /// Instruction that marks the end (inclusive) of the function's prologue.
+    ///
+    /// This is used for some calling conventions to track the end of unwind information.
+    pub prologue_end: Option,
 }
 
 impl Function {
@@ -104,6 +109,7 @@ impl Function {
             offsets: SecondaryMap::new(),
             jt_offsets: SecondaryMap::new(),
             srclocs: SecondaryMap::new(),
+            prologue_end: None,
         }
     }
 
@@ -123,6 +129,7 @@ impl Function {
         self.offsets.clear();
         self.jt_offsets.clear();
         self.srclocs.clear();
+        self.prologue_end = None;
     }
 
     /// Create a new empty, anonymous function with a Fast calling convention.
diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index 66dde9621a..15f9d59c60 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -64,6 +64,7 @@ use crate::settings;
 use crate::settings::SetResult;
 use crate::timing;
 use alloc::boxed::Box;
+use alloc::vec::Vec;
 use core::fmt;
 use target_lexicon::{triple, Architecture, PointerWidth, Triple};
 use thiserror::Error;
@@ -377,4 +378,11 @@ pub trait TargetIsa: fmt::Display + Sync {
 
     /// IntCC condition for Unsigned Subtraction Overflow (Borrow/Carry).
     fn unsigned_sub_overflow_condition(&self) -> ir::condcodes::IntCC;
+
+    /// Emit unwind information for the given function.
+    ///
+    /// Only some calling conventions (e.g. Windows fastcall) will have unwind information.
+    fn emit_unwind_info(&self, _func: &ir::Function, _mem: &mut Vec) {
+        // No-op by default
+    }
 }
diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 6bd766724e..d78af9b448 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -3,6 +3,7 @@
 use super::super::settings as shared_settings;
 use super::registers::{FPR, GPR, RU};
 use super::settings as isa_settings;
+use super::unwind::UnwindInfo;
 use crate::abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion};
 use crate::cursor::{Cursor, CursorPosition, EncCursor};
 use crate::ir;
@@ -16,6 +17,7 @@ use crate::isa::{CallConv, RegClass, RegUnit, TargetIsa};
 use crate::regalloc::RegisterSet;
 use crate::result::CodegenResult;
 use crate::stack_layout::layout_stack;
+use alloc::vec::Vec;
 use core::i32;
 use target_lexicon::{PointerWidth, Triple};
 
@@ -269,7 +271,7 @@ fn callee_saved_gprs(isa: &dyn TargetIsa, call_conv: CallConv) -> &'static [RU]
             if call_conv.extends_windows_fastcall() {
                 // "registers RBX, RBP, RDI, RSI, RSP, R12, R13, R14, R15 are considered nonvolatile
                 //  and must be saved and restored by a function that uses them."
-                // as per https://msdn.microsoft.com/en-us/library/6t169e9c.aspx
+                // as per https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention
                 // RSP & RSB are not listed below, since they are restored automatically during
                 // a function call. If that wasn't the case, function calls (RET) would not work.
                 &[
@@ -372,7 +374,7 @@ fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) ->
 }
 
 /// Implementation of the fastcall-based Win64 calling convention described at [1]
-/// [1] https://msdn.microsoft.com/en-us/library/ms235286.aspx
+/// [1] https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention
 fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> CodegenResult<()> {
     if isa.triple().pointer_width().unwrap() != PointerWidth::U64 {
         panic!("TODO: windows-fastcall: x86-32 not implemented yet");
@@ -580,11 +582,11 @@ fn insert_common_prologue(
             if !isa.flags().probestack_func_adjusts_sp() {
                 let result = pos.func.dfg.inst_results(call)[0];
                 pos.func.locations[result] = rax_val;
-                pos.ins().adjust_sp_down(result);
+                pos.func.prologue_end = Some(pos.ins().adjust_sp_down(result));
             }
         } else {
             // Simply decrement the stack pointer.
-            pos.ins().adjust_sp_down_imm(Imm64::new(stack_size));
+            pos.func.prologue_end = Some(pos.ins().adjust_sp_down_imm(Imm64::new(stack_size)));
         }
     }
 }
@@ -658,3 +660,11 @@ fn insert_common_epilogue(
         pos.func.dfg.append_inst_arg(inst, csr_ret);
     }
 }
+
+pub fn emit_unwind_info(func: &ir::Function, isa: &dyn TargetIsa, mem: &mut Vec) {
+    // Assumption: RBP is being used as the frame pointer
+    // In the future, Windows fastcall codegen should usually omit the frame pointer
+    if let Some(info) = UnwindInfo::try_from_func(func, isa, Some(RU::rbp.into())) {
+        info.emit(mem).expect("failed to emit unwind information");
+    }
+}
diff --git a/cranelift/codegen/src/isa/x86/mod.rs b/cranelift/codegen/src/isa/x86/mod.rs
index 19cc41c99d..cfba3fef48 100644
--- a/cranelift/codegen/src/isa/x86/mod.rs
+++ b/cranelift/codegen/src/isa/x86/mod.rs
@@ -5,6 +5,7 @@ mod binemit;
 mod enc_tables;
 mod registers;
 pub mod settings;
+mod unwind;
 
 use super::super::settings as shared_settings;
 #[cfg(feature = "testing_hooks")]
@@ -18,6 +19,7 @@ use crate::regalloc;
 use crate::result::CodegenResult;
 use crate::timing;
 use alloc::boxed::Box;
+use alloc::vec::Vec;
 use core::fmt;
 use target_lexicon::{PointerWidth, Triple};
 
@@ -150,6 +152,13 @@ impl TargetIsa for Isa {
     fn unsigned_sub_overflow_condition(&self) -> ir::condcodes::IntCC {
         ir::condcodes::IntCC::UnsignedLessThan
     }
+
+    /// Emit unwind information for the given function.
+    ///
+    /// Only some calling conventions (e.g. Windows fastcall) will have unwind information.
+    fn emit_unwind_info(&self, func: &ir::Function, mem: &mut Vec) {
+        abi::emit_unwind_info(func, self, mem);
+    }
 }
 
 impl fmt::Display for Isa {
diff --git a/cranelift/codegen/src/isa/x86/unwind.rs b/cranelift/codegen/src/isa/x86/unwind.rs
new file mode 100644
index 0000000000..40caf44b63
--- /dev/null
+++ b/cranelift/codegen/src/isa/x86/unwind.rs
@@ -0,0 +1,508 @@
+//! Unwind information for x64 Windows.
+
+use super::registers::RU;
+use crate::ir::{Function, InstructionData, Opcode};
+use crate::isa::{CallConv, RegUnit, TargetIsa};
+use alloc::vec::Vec;
+use byteorder::{LittleEndian, WriteBytesExt};
+
+/// Maximum (inclusive) size of a "small" stack allocation
+const SMALL_ALLOC_MAX_SIZE: u32 = 128;
+/// Maximum (inclusive) size of a "large" stack allocation that can represented in 16-bits
+const LARGE_ALLOC_16BIT_MAX_SIZE: u32 = 524280;
+
+/// The supported unwind codes for the x64 Windows ABI.
+///
+/// See: https://docs.microsoft.com/en-us/cpp/build/exception-handling-x64
+/// Only what is needed to describe the prologues generated by the Cranelift x86 ISA are represented here.
+/// Note: the Cranelift x86 ISA RU enum matches the Windows unwind GPR encoding values.
+#[derive(Debug, PartialEq, Eq)]
+enum UnwindCode {
+    PushRegister { offset: u8, reg: RegUnit },
+    StackAlloc { offset: u8, size: u32 },
+    SetFramePointer { offset: u8, sp_offset: u8 },
+}
+
+impl UnwindCode {
+    fn emit(&self, mem: &mut Vec) -> std::io::Result<()> {
+        enum UnwindOperation {
+            PushNonvolatileRegister,
+            LargeStackAlloc,
+            SmallStackAlloc,
+            SetFramePointer,
+        }
+
+        match self {
+            Self::PushRegister { offset, reg } => {
+                mem.write_u8(*offset)?;
+                mem.write_u8(
+                    ((*reg as u8) << 4) | (UnwindOperation::PushNonvolatileRegister as u8),
+                )?;
+            }
+            Self::StackAlloc { offset, size } => {
+                // Stack allocations on Windows must be a multiple of 8 and be at least 1 slot
+                assert!(*size >= 8);
+                assert!((*size % 8) == 0);
+
+                mem.write_u8(*offset)?;
+                if *size <= SMALL_ALLOC_MAX_SIZE {
+                    mem.write_u8(
+                        ((((*size - 8) / 8) as u8) << 4) | UnwindOperation::SmallStackAlloc as u8,
+                    )?;
+                } else if *size <= LARGE_ALLOC_16BIT_MAX_SIZE {
+                    mem.write_u8(UnwindOperation::LargeStackAlloc as u8)?;
+                    mem.write_u16::((*size / 8) as u16)?;
+                } else {
+                    mem.write_u8((1 << 4) | (UnwindOperation::LargeStackAlloc as u8))?;
+                    mem.write_u32::(*size)?;
+                }
+            }
+            Self::SetFramePointer { offset, sp_offset } => {
+                mem.write_u8(*offset)?;
+                mem.write_u8((*sp_offset << 4) | (UnwindOperation::SetFramePointer as u8))?;
+            }
+        };
+
+        Ok(())
+    }
+
+    fn node_count(&self) -> usize {
+        match self {
+            Self::StackAlloc { size, .. } => {
+                if *size <= SMALL_ALLOC_MAX_SIZE {
+                    1
+                } else if *size <= LARGE_ALLOC_16BIT_MAX_SIZE {
+                    2
+                } else {
+                    3
+                }
+            }
+            _ => 1,
+        }
+    }
+}
+
+/// Represents Windows x64 unwind information.
+///
+/// For information about Windows x64 unwind info, see:
+/// https://docs.microsoft.com/en-us/cpp/build/exception-handling-x64
+#[derive(Debug, PartialEq, Eq)]
+pub struct UnwindInfo {
+    flags: u8,
+    prologue_size: u8,
+    frame_register: Option,
+    frame_register_offset: u8,
+    unwind_codes: Vec,
+}
+
+impl UnwindInfo {
+    pub fn try_from_func(
+        func: &Function,
+        isa: &dyn TargetIsa,
+        frame_register: Option,
+    ) -> Option {
+        // Only Windows fastcall is supported for unwind information
+        if func.signature.call_conv != CallConv::WindowsFastcall || func.prologue_end.is_none() {
+            return None;
+        }
+
+        let prologue_end = func.prologue_end.unwrap();
+        let entry_block = func.layout.ebbs().nth(0).expect("missing entry block");
+
+        // Stores the stack size when SP is not adjusted via an immediate value
+        let mut stack_size = None;
+        let mut prologue_size = 0;
+        let mut unwind_codes = Vec::new();
+        let mut found_end = false;
+
+        for (offset, inst, size) in func.inst_offsets(entry_block, &isa.encoding_info()) {
+            // x64 ABI prologues cannot exceed 255 bytes in length
+            if (offset + size) > 255 {
+                panic!("function prologues cannot exceed 255 bytes in size for Windows x64");
+            }
+
+            prologue_size += size;
+
+            let unwind_offset = (offset + size) as u8;
+
+            match func.dfg[inst] {
+                InstructionData::Unary { opcode, arg } => {
+                    match opcode {
+                        Opcode::X86Push => {
+                            unwind_codes.push(UnwindCode::PushRegister {
+                                offset: unwind_offset,
+                                reg: func.locations[arg].unwrap_reg(),
+                            });
+                        }
+                        Opcode::AdjustSpDown => {
+                            // This is used when calling a stack check function
+                            // We need to track the assignment to RAX which has the size of the stack
+                            unwind_codes.push(UnwindCode::StackAlloc {
+                                offset: unwind_offset,
+                                size: stack_size
+                                    .expect("expected a previous stack size instruction"),
+                            });
+                        }
+                        _ => {}
+                    }
+                }
+                InstructionData::CopySpecial { src, dst, .. } => {
+                    if let Some(frame_register) = frame_register {
+                        if src == (RU::rsp as RegUnit) && dst == frame_register {
+                            unwind_codes.push(UnwindCode::SetFramePointer {
+                                offset: unwind_offset,
+                                sp_offset: 0,
+                            });
+                        }
+                    }
+                }
+                InstructionData::UnaryImm { opcode, imm } => {
+                    match opcode {
+                        Opcode::Iconst => {
+                            let imm: i64 = imm.into();
+                            assert!(imm <= std::u32::MAX as i64);
+                            assert!(stack_size.is_none());
+
+                            // This instruction should only appear in a prologue to pass an
+                            // argument of the stack size to a stack check function.
+                            // Record the stack size so we know what it is when we encounter the adjustment
+                            // instruction (which will adjust via the register assigned to this instruction).
+                            stack_size = Some(imm as u32);
+                        }
+                        Opcode::AdjustSpDownImm => {
+                            let imm: i64 = imm.into();
+                            assert!(imm <= std::u32::MAX as i64);
+
+                            unwind_codes.push(UnwindCode::StackAlloc {
+                                offset: unwind_offset,
+                                size: imm as u32,
+                            });
+                        }
+                        _ => {}
+                    }
+                }
+                _ => {}
+            };
+
+            if inst == prologue_end {
+                found_end = true;
+                break;
+            }
+        }
+
+        if !found_end {
+            return None;
+        }
+
+        Some(Self {
+            flags: 0, // this assumes cranelift functions have no SEH handlers
+            prologue_size: prologue_size as u8,
+            frame_register,
+            frame_register_offset: 0,
+            unwind_codes,
+        })
+    }
+
+    pub fn size(&self) -> usize {
+        let node_count = self.node_count();
+
+        // Calculation of the size requires no SEH handler or chained info
+        assert!(self.flags == 0);
+
+        // Size of fixed part of UNWIND_INFO is 4 bytes
+        // Then comes the UNWIND_CODE nodes (2 bytes each)
+        // Then comes 2 bytes of padding for the unwind codes if necessary
+        // Next would come the SEH data, but we assert above that the function doesn't have SEH data
+
+        4 + (node_count * 2) + if (node_count & 1) == 1 { 2 } else { 0 }
+    }
+
+    pub fn node_count(&self) -> usize {
+        self.unwind_codes
+            .iter()
+            .fold(0, |nodes, c| nodes + c.node_count())
+    }
+
+    pub fn emit(&self, mem: &mut Vec) -> std::io::Result<()> {
+        const UNWIND_INFO_VERSION: u8 = 1;
+
+        let size = self.size();
+        let offset = mem.len();
+
+        // Ensure the memory is 32-bit aligned
+        assert_eq!(offset % 4, 0);
+
+        mem.reserve(offset + size);
+
+        let node_count = self.node_count();
+        assert!(node_count <= 256);
+
+        mem.write_u8((self.flags << 3) | UNWIND_INFO_VERSION)?;
+        mem.write_u8(self.prologue_size)?;
+        mem.write_u8(node_count as u8)?;
+
+        if let Some(reg) = self.frame_register {
+            mem.write_u8((self.frame_register_offset << 4) | reg as u8)?;
+        } else {
+            mem.write_u8(0)?;
+        }
+
+        // Unwind codes are written in reverse order (prologue offset descending)
+        for code in self.unwind_codes.iter().rev() {
+            code.emit(mem)?;
+        }
+
+        // To keep a 32-bit alignment, emit 2 bytes of padding if there's an odd number of 16-bit nodes
+        if (node_count & 1) == 1 {
+            mem.write_u16::(0)?;
+        }
+
+        // Ensure the correct number of bytes was emitted
+        assert_eq!(mem.len() - offset, size);
+
+        Ok(())
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::cursor::{Cursor, FuncCursor};
+    use crate::ir::{ExternalName, InstBuilder, Signature, StackSlotData, StackSlotKind};
+    use crate::isa::{lookup, CallConv};
+    use crate::settings::{builder, Flags};
+    use crate::Context;
+    use std::str::FromStr;
+    use target_lexicon::triple;
+
+    #[test]
+    fn test_wrong_calling_convention() {
+        let isa = lookup(triple!("x86_64"))
+            .expect("expect x86 ISA")
+            .finish(Flags::new(builder()));
+
+        let mut context = Context::for_function(create_function(CallConv::SystemV, None));
+
+        context.compile(&*isa).expect("expected compilation");
+
+        assert_eq!(UnwindInfo::try_from_func(&context.func, &*isa, None), None);
+    }
+
+    #[test]
+    fn test_small_alloc() {
+        let isa = lookup(triple!("x86_64"))
+            .expect("expect x86 ISA")
+            .finish(Flags::new(builder()));
+
+        let mut context = Context::for_function(create_function(
+            CallConv::WindowsFastcall,
+            Some(StackSlotData::new(StackSlotKind::ExplicitSlot, 64)),
+        ));
+
+        context.compile(&*isa).expect("expected compilation");
+
+        let unwind = UnwindInfo::try_from_func(&context.func, &*isa, Some(RU::rbp.into()))
+            .expect("expected unwind info");
+
+        assert_eq!(
+            unwind,
+            UnwindInfo {
+                flags: 0,
+                prologue_size: 9,
+                frame_register: Some(RU::rbp.into()),
+                frame_register_offset: 0,
+                unwind_codes: vec![
+                    UnwindCode::PushRegister {
+                        offset: 2,
+                        reg: RU::rbp.into()
+                    },
+                    UnwindCode::SetFramePointer {
+                        offset: 5,
+                        sp_offset: 0
+                    },
+                    UnwindCode::StackAlloc {
+                        offset: 9,
+                        size: 64 + 32
+                    }
+                ]
+            }
+        );
+
+        assert_eq!(unwind.size(), 12);
+
+        let mut mem = Vec::new();
+        unwind
+            .emit(&mut mem)
+            .expect("failed to emit unwind information");
+
+        assert_eq!(
+            mem,
+            [
+                0x01, // Version and flags (version 1, no flags)
+                0x09, // Prologue size
+                0x03, // Unwind code count (1 for stack alloc, 1 for save frame reg, 1 for push reg)
+                0x05, // Frame register + offset (RBP with 0 offset)
+                0x09, // Prolog offset
+                0xB2, // Operation 2 (small stack alloc), size = 0xB slots (e.g. (0xB * 8) + 8 = 96 (64 + 32) bytes)
+                0x05, // Prolog offset
+                0x03, // Operation 3 (save frame register), stack pointer offset = 0
+                0x02, // Prolog offset
+                0x50, // Operation 0 (save nonvolatile register), reg = 5 (RBP)
+                0x00, // Padding byte
+                0x00, // Padding byte
+            ]
+        );
+    }
+
+    #[test]
+    fn test_medium_alloc() {
+        let isa = lookup(triple!("x86_64"))
+            .expect("expect x86 ISA")
+            .finish(Flags::new(builder()));
+
+        let mut context = Context::for_function(create_function(
+            CallConv::WindowsFastcall,
+            Some(StackSlotData::new(StackSlotKind::ExplicitSlot, 10000)),
+        ));
+
+        context.compile(&*isa).expect("expected compilation");
+
+        let unwind = UnwindInfo::try_from_func(&context.func, &*isa, Some(RU::rbp.into()))
+            .expect("expected unwind info");
+
+        assert_eq!(
+            unwind,
+            UnwindInfo {
+                flags: 0,
+                prologue_size: 27,
+                frame_register: Some(RU::rbp.into()),
+                frame_register_offset: 0,
+                unwind_codes: vec![
+                    UnwindCode::PushRegister {
+                        offset: 2,
+                        reg: RU::rbp.into()
+                    },
+                    UnwindCode::SetFramePointer {
+                        offset: 5,
+                        sp_offset: 0
+                    },
+                    UnwindCode::StackAlloc {
+                        offset: 27,
+                        size: 10000 + 32
+                    }
+                ]
+            }
+        );
+
+        assert_eq!(unwind.size(), 12);
+
+        let mut mem = Vec::new();
+        unwind
+            .emit(&mut mem)
+            .expect("failed to emit unwind information");
+
+        assert_eq!(
+            mem,
+            [
+                0x01, // Version and flags (version 1, no flags)
+                0x1B, // Prologue size
+                0x04, // Unwind code count (2 for stack alloc, 1 for save frame reg, 1 for push reg)
+                0x05, // Frame register + offset (RBP with 0 offset)
+                0x1B, // Prolog offset
+                0x01, // Operation 1 (large stack alloc), size is scaled 16-bits (info = 0)
+                0xE6, // Low size byte
+                0x04, // High size byte (e.g. 0x04E6 * 8 = 100032 (10000 + 32) bytes)
+                0x05, // Prolog offset
+                0x03, // Operation 3 (save frame register), stack pointer offset = 0
+                0x02, // Prolog offset
+                0x50, // Operation 0 (push nonvolatile register), reg = 5 (RBP)
+            ]
+        );
+    }
+
+    #[test]
+    fn test_large_alloc() {
+        let isa = lookup(triple!("x86_64"))
+            .expect("expect x86 ISA")
+            .finish(Flags::new(builder()));
+
+        let mut context = Context::for_function(create_function(
+            CallConv::WindowsFastcall,
+            Some(StackSlotData::new(StackSlotKind::ExplicitSlot, 1000000)),
+        ));
+
+        context.compile(&*isa).expect("expected compilation");
+
+        let unwind = UnwindInfo::try_from_func(&context.func, &*isa, Some(RU::rbp.into()))
+            .expect("expected unwind info");
+
+        assert_eq!(
+            unwind,
+            UnwindInfo {
+                flags: 0,
+                prologue_size: 27,
+                frame_register: Some(RU::rbp.into()),
+                frame_register_offset: 0,
+                unwind_codes: vec![
+                    UnwindCode::PushRegister {
+                        offset: 2,
+                        reg: RU::rbp.into()
+                    },
+                    UnwindCode::SetFramePointer {
+                        offset: 5,
+                        sp_offset: 0
+                    },
+                    UnwindCode::StackAlloc {
+                        offset: 27,
+                        size: 1000000 + 32
+                    }
+                ]
+            }
+        );
+
+        assert_eq!(unwind.size(), 16);
+
+        let mut mem = Vec::new();
+        unwind
+            .emit(&mut mem)
+            .expect("failed to emit unwind information");
+
+        assert_eq!(
+            mem,
+            [
+                0x01, // Version and flags (version 1, no flags)
+                0x1B, // Prologue size
+                0x05, // Unwind code count (3 for stack alloc, 1 for save frame reg, 1 for push reg)
+                0x05, // Frame register + offset (RBP with 0 offset)
+                0x1B, // Prolog offset
+                0x11, // Operation 1 (large stack alloc), size is unscaled 32-bits (info = 1)
+                0x60, // Byte 1 of size
+                0x42, // Byte 2 of size
+                0x0F, // Byte 3 of size
+                0x00, // Byte 4 of size (size is 0xF4260 = 1000032 (1000000 + 32) bytes)
+                0x05, // Prolog offset
+                0x03, // Operation 3 (save frame register), stack pointer offset = 0
+                0x02, // Prolog offset
+                0x50, // Operation 0 (push nonvolatile register), reg = 5 (RBP)
+                0x00, // Padding byte
+                0x00, // Padding byte
+            ]
+        );
+    }
+
+    fn create_function(call_conv: CallConv, stack_slot: Option) -> Function {
+        let mut func =
+            Function::with_name_signature(ExternalName::user(0, 0), Signature::new(call_conv));
+
+        let ebb0 = func.dfg.make_ebb();
+        let mut pos = FuncCursor::new(&mut func);
+        pos.insert_ebb(ebb0);
+        pos.ins().return_(&[]);
+
+        if let Some(stack_slot) = stack_slot {
+            func.stack_slots.push(stack_slot);
+        }
+
+        func
+    }
+}
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index c76ee4d478..2f59146613 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -20,6 +20,7 @@ log = "0.4.6"
 memmap = "0.7.0"
 num_cpus = "1.8.0"
 region = "2.1.2"
+byteorder = { version = "1.3.2", default-features = false }
 
 [features]
 basic-blocks = []
diff --git a/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64_unwind.clif b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64_unwind.clif
new file mode 100644
index 0000000000..6782d8cdce
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64_unwind.clif
@@ -0,0 +1,221 @@
+test unwind
+set opt_level=speed_and_size
+set is_pic
+target x86_64 haswell
+
+; check that there is no unwind information for a system_v function
+function %not_fastcall() system_v {
+ebb0:
+    return
+}
+; sameln: No unwind information.
+
+; check the unwind information with a function with no args
+function %no_args() windows_fastcall {
+ebb0:
+    return
+}
+; sameln: UnwindInfo {
+; nextln:     version: 1,
+; nextln:     flags: 0,
+; nextln:     prologue_size: 8,
+; nextln:     unwind_code_count_raw: 3,
+; nextln:     frame_register: 5,
+; nextln:     frame_register_offset: 0,
+; nextln:     unwind_codes: [
+; nextln:         UnwindCode {
+; nextln:             offset: 8,
+; nextln:             op: SmallStackAlloc,
+; nextln:             info: 3,
+; nextln:             value: None,
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 4,
+; nextln:             op: SetFramePointer,
+; nextln:             info: 0,
+; nextln:             value: None,
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 1,
+; nextln:             op: PushNonvolatileRegister,
+; nextln:             info: 5,
+; nextln:             value: None,
+; nextln:         },
+; nextln:     ],
+; nextln: }
+
+; check a function with medium-sized stack alloc
+function %medium_stack() windows_fastcall {
+    ss0 = explicit_slot 100000
+ebb0:
+    return
+}
+; sameln: UnwindInfo {
+; nextln:     version: 1,
+; nextln:     flags: 0,
+; nextln:     prologue_size: 17,
+; nextln:     unwind_code_count_raw: 4,
+; nextln:     frame_register: 5,
+; nextln:     frame_register_offset: 0,
+; nextln:     unwind_codes: [
+; nextln:         UnwindCode {
+; nextln:             offset: 17,
+; nextln:             op: LargeStackAlloc,
+; nextln:             info: 0,
+; nextln:             value: U16(
+; nextln:                 12504,
+; nextln:             ),
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 4,
+; nextln:             op: SetFramePointer,
+; nextln:             info: 0,
+; nextln:             value: None,
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 1,
+; nextln:             op: PushNonvolatileRegister,
+; nextln:             info: 5,
+; nextln:             value: None,
+; nextln:         },
+; nextln:     ],
+; nextln: }
+
+; check a function with large-sized stack alloc
+function %large_stack() windows_fastcall {
+    ss0 = explicit_slot 524288
+ebb0:
+    return
+}
+; sameln: UnwindInfo {
+; nextln:     version: 1,
+; nextln:     flags: 0,
+; nextln:     prologue_size: 17,
+; nextln:     unwind_code_count_raw: 5,
+; nextln:     frame_register: 5,
+; nextln:     frame_register_offset: 0,
+; nextln:     unwind_codes: [
+; nextln:         UnwindCode {
+; nextln:             offset: 17,
+; nextln:             op: LargeStackAlloc,
+; nextln:             info: 1,
+; nextln:             value: U32(
+; nextln:                 524320,
+; nextln:             ),
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 4,
+; nextln:             op: SetFramePointer,
+; nextln:             info: 0,
+; nextln:             value: None,
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 1,
+; nextln:             op: PushNonvolatileRegister,
+; nextln:             info: 5,
+; nextln:             value: None,
+; nextln:         },
+; nextln:     ],
+; nextln: }
+
+; check a function that has CSRs
+function %lots_of_registers(i64, i64) windows_fastcall {
+ebb0(v0: i64, v1: i64):
+    v2 = load.i32 v0+0
+    v3 = load.i32 v0+8
+    v4 = load.i32 v0+16
+    v5 = load.i32 v0+24
+    v6 = load.i32 v0+32
+    v7 = load.i32 v0+40
+    v8 = load.i32 v0+48
+    v9 = load.i32 v0+56
+    v10 = load.i32 v0+64
+    v11 = load.i32 v0+72
+    v12 = load.i32 v0+80
+    v13 = load.i32 v0+88
+    v14 = load.i32 v0+96
+    store.i32 v2, v1+0
+    store.i32 v3, v1+8
+    store.i32 v4, v1+16
+    store.i32 v5, v1+24
+    store.i32 v6, v1+32
+    store.i32 v7, v1+40
+    store.i32 v8, v1+48
+    store.i32 v9, v1+56
+    store.i32 v10, v1+64
+    store.i32 v11, v1+72
+    store.i32 v12, v1+80
+    store.i32 v13, v1+88
+    store.i32 v14, v1+96
+    return
+}
+; sameln: UnwindInfo {
+; nextln:     version: 1,
+; nextln:     flags: 0,
+; nextln:     prologue_size: 19,
+; nextln:     unwind_code_count_raw: 10,
+; nextln:     frame_register: 5,
+; nextln:     frame_register_offset: 0,
+; nextln:     unwind_codes: [
+; nextln:         UnwindCode {
+; nextln:             offset: 19,
+; nextln:             op: SmallStackAlloc,
+; nextln:             info: 4,
+; nextln:             value: None,
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 15,
+; nextln:             op: PushNonvolatileRegister,
+; nextln:             info: 15,
+; nextln:             value: None,
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 13,
+; nextln:             op: PushNonvolatileRegister,
+; nextln:             info: 14,
+; nextln:             value: None,
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 11,
+; nextln:             op: PushNonvolatileRegister,
+; nextln:             info: 13,
+; nextln:             value: None,
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 9,
+; nextln:             op: PushNonvolatileRegister,
+; nextln:             info: 12,
+; nextln:             value: None,
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 7,
+; nextln:             op: PushNonvolatileRegister,
+; nextln:             info: 7,
+; nextln:             value: None,
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 6,
+; nextln:             op: PushNonvolatileRegister,
+; nextln:             info: 6,
+; nextln:             value: None,
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 5,
+; nextln:             op: PushNonvolatileRegister,
+; nextln:             info: 3,
+; nextln:             value: None,
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 4,
+; nextln:             op: SetFramePointer,
+; nextln:             info: 0,
+; nextln:             value: None,
+; nextln:         },
+; nextln:         UnwindCode {
+; nextln:             offset: 1,
+; nextln:             op: PushNonvolatileRegister,
+; nextln:             info: 5,
+; nextln:             value: None,
+; nextln:         },
+; nextln:     ],
+; nextln: }
diff --git a/cranelift/filetests/src/lib.rs b/cranelift/filetests/src/lib.rs
index 073fc24492..ded3acd16c 100644
--- a/cranelift/filetests/src/lib.rs
+++ b/cranelift/filetests/src/lib.rs
@@ -54,6 +54,7 @@ mod test_safepoint;
 mod test_shrink;
 mod test_simple_gvn;
 mod test_simple_preopt;
+mod test_unwind;
 mod test_verifier;
 
 /// The result of running the test in a file.
@@ -135,6 +136,7 @@ fn new_subtest(parsed: &TestCommand) -> subtest::SubtestResult test_verifier::subtest(parsed),
         "preopt" => test_preopt::subtest(parsed),
         "safepoint" => test_safepoint::subtest(parsed),
+        "unwind" => test_unwind::subtest(parsed),
         _ => Err(format!("unknown test command '{}'", parsed.command)),
     }
 }
diff --git a/cranelift/filetests/src/test_unwind.rs b/cranelift/filetests/src/test_unwind.rs
new file mode 100644
index 0000000000..55fcbcdd9c
--- /dev/null
+++ b/cranelift/filetests/src/test_unwind.rs
@@ -0,0 +1,198 @@
+//! Test command for verifying the unwind emitted for each function.
+//!
+//! The `unwind` test command runs each function through the full code generator pipeline.
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
+
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
+use byteorder::{LittleEndian, ReadBytesExt};
+use cranelift_codegen;
+use cranelift_codegen::ir;
+use cranelift_reader::TestCommand;
+use std::borrow::Cow;
+use std::fmt::Write;
+use std::io::Cursor;
+
+struct TestUnwind;
+
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+    assert_eq!(parsed.command, "unwind");
+    if !parsed.options.is_empty() {
+        Err(format!("No options allowed on {}", parsed))
+    } else {
+        Ok(Box::new(TestUnwind))
+    }
+}
+
+impl SubTest for TestUnwind {
+    fn name(&self) -> &'static str {
+        "unwind"
+    }
+
+    fn is_mutating(&self) -> bool {
+        false
+    }
+
+    fn needs_isa(&self) -> bool {
+        true
+    }
+
+    fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> {
+        let isa = context.isa.expect("unwind needs an ISA");
+        let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned());
+
+        comp_ctx.compile(isa).expect("failed to compile function");
+
+        let mut mem = Vec::new();
+        comp_ctx.emit_unwind_info(isa, &mut mem);
+
+        let mut text = String::new();
+        if mem.is_empty() {
+            writeln!(text, "No unwind information.").unwrap();
+        } else {
+            print_unwind_info(&mut text, &mem);
+        }
+
+        run_filecheck(&text, context)
+    }
+}
+
+fn print_unwind_info(text: &mut String, mem: &[u8]) {
+    let info = UnwindInfo::from_cursor(&mut Cursor::new(mem)).expect("failed to read unwind info");
+
+    // Assert correct alignment and padding of the unwind information
+    assert!(mem.len() % 4 == 0);
+    assert_eq!(
+        mem.len(),
+        4 + ((info.unwind_code_count_raw as usize) * 2)
+            + if (info.unwind_code_count_raw & 1) == 1 {
+                2
+            } else {
+                0
+            }
+    );
+
+    writeln!(text, "{:#?}", info).unwrap();
+}
+
+#[derive(Debug)]
+struct UnwindInfo {
+    pub version: u8,
+    pub flags: u8,
+    pub prologue_size: u8,
+    pub unwind_code_count_raw: u8,
+    pub frame_register: u8,
+    pub frame_register_offset: u8,
+    pub unwind_codes: Vec,
+}
+
+impl UnwindInfo {
+    fn from_cursor(cursor: &mut Cursor<&[u8]>) -> std::io::Result {
+        let version_and_flags = cursor.read_u8()?;
+        let prologue_size = cursor.read_u8()?;
+        let unwind_code_count_raw = cursor.read_u8()?;
+        let frame_register_and_offset = cursor.read_u8()?;
+        let mut unwind_codes = Vec::new();
+
+        let mut i = 0;
+        while i < unwind_code_count_raw {
+            let code = UnwindCode::from_cursor(cursor)?;
+
+            i += match &code.value {
+                UnwindValue::None => 1,
+                UnwindValue::U16(_) => 2,
+                UnwindValue::U32(_) => 3,
+            };
+
+            unwind_codes.push(code);
+        }
+
+        Ok(Self {
+            version: version_and_flags & 0x3,
+            flags: (version_and_flags & 0xF8) >> 3,
+            prologue_size,
+            unwind_code_count_raw,
+            frame_register: frame_register_and_offset & 0xF,
+            frame_register_offset: (frame_register_and_offset & 0xF0) >> 4,
+            unwind_codes,
+        })
+    }
+}
+
+#[derive(Debug)]
+struct UnwindCode {
+    pub offset: u8,
+    pub op: UnwindOperation,
+    pub info: u8,
+    pub value: UnwindValue,
+}
+
+impl UnwindCode {
+    fn from_cursor(cursor: &mut Cursor<&[u8]>) -> std::io::Result {
+        let offset = cursor.read_u8()?;
+        let op_and_info = cursor.read_u8()?;
+        let op = UnwindOperation::from(op_and_info & 0xF);
+        let info = (op_and_info & 0xF0) >> 4;
+
+        let value = match op {
+            UnwindOperation::LargeStackAlloc => match info {
+                0 => UnwindValue::U16(cursor.read_u16::()?),
+                1 => UnwindValue::U32(cursor.read_u32::()?),
+                _ => panic!("unexpected stack alloc info value"),
+            },
+            UnwindOperation::SaveNonvolatileRegister => {
+                UnwindValue::U16(cursor.read_u16::()?)
+            }
+            UnwindOperation::SaveNonvolatileRegisterFar => {
+                UnwindValue::U32(cursor.read_u32::()?)
+            }
+            UnwindOperation::SaveXmm128 => UnwindValue::U16(cursor.read_u16::()?),
+            UnwindOperation::SaveXmm128Far => UnwindValue::U32(cursor.read_u32::()?),
+            _ => UnwindValue::None,
+        };
+
+        Ok(Self {
+            offset,
+            op,
+            info,
+            value,
+        })
+    }
+}
+
+#[derive(Debug)]
+enum UnwindOperation {
+    PushNonvolatileRegister,
+    LargeStackAlloc,
+    SmallStackAlloc,
+    SetFramePointer,
+    SaveNonvolatileRegister,
+    SaveNonvolatileRegisterFar,
+    SaveXmm128,
+    SaveXmm128Far,
+    PushMachineFrame,
+}
+
+impl From for UnwindOperation {
+    fn from(value: u8) -> Self {
+        // The numerical value is specified as part of the Windows x64 ABI
+        match value {
+            0 => Self::PushNonvolatileRegister,
+            1 => Self::LargeStackAlloc,
+            2 => Self::SmallStackAlloc,
+            3 => Self::SetFramePointer,
+            4 => Self::SaveNonvolatileRegister,
+            5 => Self::SaveNonvolatileRegisterFar,
+            6 => Self::SaveXmm128,
+            7 => Self::SaveXmm128Far,
+            8 => Self::PushMachineFrame,
+            _ => panic!("unsupported unwind operation"),
+        }
+    }
+}
+
+#[derive(Debug)]
+enum UnwindValue {
+    None,
+    U16(u16),
+    U32(u32),
+}

From 45fb377457ca17fdfea9aca794b690d43c3ef7cc Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 5 Nov 2019 13:59:19 -0800
Subject: [PATCH 2913/3084] Bump version to 0.48.0 (#1202)

* Bump version to 0.48.0

* Re-enable `byteorder`'s default features.

The code uses `WriteBytesExt` which depends on the `std` feature being
enabled. So for now, just enable `std`.
---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 12 ++++++------
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  2 +-
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 19 files changed, 69 insertions(+), 69 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index d97571d817..5df578aa20 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.47.0"
+version = "0.48.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.47.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.47.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.47.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.47.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.47.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.47.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.47.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.47.0" }
-cranelift-module = { path = "cranelift-module", version = "0.47.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.47.0" }
-cranelift-object = { path = "cranelift-object", version = "0.47.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.47.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.47.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.47.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.48.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.48.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.48.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.48.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.48.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.48.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.48.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.48.0" }
+cranelift-module = { path = "cranelift-module", version = "0.48.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.48.0" }
+cranelift-object = { path = "cranelift-object", version = "0.48.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.48.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.48.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.48.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index af3bddd6c5..0f782c89cf 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.47.0"
+version = "0.48.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.47.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.48.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 1d45b5b50a..66176bb418 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.47.0"
+version = "0.48.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,23 +13,23 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.47.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.47.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.47.0" }
+cranelift-codegen-shared = { path = "./shared", version = "0.48.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.48.0" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.48.0" }
 hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
 smallvec = { version = "1.0.0" }
 thiserror = "1.0.4"
-byteorder = { version = "1.3.2", default-features = false }
+byteorder = { version = "1.3.2" }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
 # machine code. Integration tests that need external dependencies can be
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.47.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.48.0" }
 
 [features]
 default = ["std", "basic-blocks"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 0dad16727d..6022e8ae8a 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.47.0"
+version = "0.48.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.47.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.47.0" }
+cranelift-codegen-shared = { path = "../shared", version = "0.48.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.48.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 472e670af0..83b56b4106 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.47.0"
+version = "0.48.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 67c11e8986..09d4ce10a4 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.47.0"
+version = "0.48.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 6bd9f120f5..8d636cdc7c 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.47.0"
+version = "0.48.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.47.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.48.0" }
 faerie = "0.11.0"
 goblin = "0.1.0"
 failure = "0.1.2"
@@ -18,7 +18,7 @@ target-lexicon = "0.8.1"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.47.0"
+version = "0.48.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 2f59146613..bea35779a1 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.47.0"
+version = "0.48.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.47.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.47.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.47.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.48.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.48.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.48.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 6d4e74c464..526dbdd1bc 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.47.0"
+version = "0.48.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
 target-lexicon = "0.8.1"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index bcd6ad4132..dbc4f413d0 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.47.0"
+version = "0.48.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.47.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.48.0" }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 thiserror = "1.0.4"
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 954562b25c..19b952747b 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.47.0"
+version = "0.48.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
 target-lexicon = "0.8.1"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 61401a3d65..9f1a4be124 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.47.0"
+version = "0.48.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,13 +10,13 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.47.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.48.0" }
 object = { version = "0.15.0", default-features = false, features = ["write"] }
 target-lexicon = "0.8.1"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.47.0"
+version = "0.48.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 2d11963306..bd0cb7c3a4 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.47.0"
+version = "0.48.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.47.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.48.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 66fe655c08..6f5e45666c 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.47.0"
+version="0.48.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index b1a56abe3b..fa16526792 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.47.0"
+version = "0.48.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0" }
 target-lexicon = "0.8.1"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index c0e5252d48..da4852585e 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.47.0"
+version = "0.48.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.47.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.48.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 16fddb6845..401587bf56 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.47.0"
+version = "0.48.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.47.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.47.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.48.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.48.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -20,7 +20,7 @@ memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.47.0"
+version = "0.48.0"
 default-features = false
 features = ["std"]
 
@@ -32,9 +32,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.47.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.47.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.47.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.48.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.48.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.48.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index c8fc3f42ed..ee7cb7ceca 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.47.0"
+version = "0.48.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.47.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.48.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index cf0675b0cf..1a6e5ffec4 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.47.0"
+version = "0.48.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.39.2", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.47.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.47.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.47.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.48.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.48.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }

From a49483408c4f2a667bbc55bed48f713c139db1ac Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Tue, 5 Nov 2019 14:36:03 -0800
Subject: [PATCH 2914/3084] Many multi-value returns (#1147)

* Add x86 encodings for `bint` converting to `i8` and `i16`

* Introduce tests for many multi-value returns

* Support arbitrary numbers of return values

This commit implements support for returning an arbitrary number of return
values from a function. During legalization we transform multi-value signatures
to take a struct return ("sret") return pointer, instead of returning its values
in registers. Callers allocate the sret space in their stack frame and pass a
pointer to it into the caller, and once the caller returns to them, they load
the return values back out of the sret stack slot. The callee's return
operations are legalized to store the return values through the given sret
pointer.

* Keep track of old, pre-legalized signatures

When legalizing a call or return for its new legalized signature, we may need to
look at the old signature in order to figure out how to legalize the call or
return.

* Add test for multi-value returns and `call_indirect`

* Encode bool -> int x86 instructions in a loop

* Rename `Signature::uses_sret` to `Signature::uses_struct_return_param`

* Rename `p` to `param`

* Add a clarifiying comment in `num_registers_required`

* Rename `num_registers_required` to `num_return_registers_required`

* Re-add newline

* Handle already-assigned parameters in `num_return_registers_required`

* Document what some debug assertions are checking for

* Make "illegalizing" closure's control flow simpler

* Add unit tests and comments for our rounding-up-to-the-next-multiple-of-a-power-of-2 function

* Use `append_isnt_arg` instead of doing the same thing  manually

* Fix grammar in comment

* Add `Signature::uses_special_{param,return}` helper functions

* Inline the definition of `legalize_type_for_sret_load` for readability

* Move sret legalization debug assertions out into their own function

* Add `round_up_to_multiple_of_type_align` helper for readability

* Add a debug assertion that we aren't removing the wrong return value

* Rename `RetPtr` stack slots to `StructReturnSlot`

* Make `legalize_type_for_sret_store` more symmetrical to `legalized_type_for_sret`

* rustfmt

* Remove unnecessary loop labels

* Do not pre-assign offsets to struct return stack slots

Instead, let the existing frame layout algorithm decide where they should go.

* Expand "sret" into explicit "struct return" in doc comment

* typo: "than" -> "then" in comment

* Fold test's debug message into the assertion itself
---
 .../codegen/meta/src/isa/x86/encodings.rs     |   39 +-
 cranelift/codegen/src/abi.rs                  |   16 +-
 cranelift/codegen/src/ir/dfg.rs               |    4 +
 cranelift/codegen/src/ir/extfunc.rs           |   47 +
 cranelift/codegen/src/ir/function.rs          |    5 +
 cranelift/codegen/src/ir/stackslot.rs         |   11 +
 cranelift/codegen/src/isa/arm32/abi.rs        |    7 +-
 cranelift/codegen/src/isa/arm32/mod.rs        |    3 +-
 cranelift/codegen/src/isa/arm64/abi.rs        |    3 +-
 cranelift/codegen/src/isa/arm64/mod.rs        |    3 +-
 cranelift/codegen/src/isa/mod.rs              |    3 +-
 cranelift/codegen/src/isa/riscv/abi.rs        |   15 +-
 cranelift/codegen/src/isa/riscv/mod.rs        |    3 +-
 cranelift/codegen/src/isa/x86/abi.rs          |  185 +-
 cranelift/codegen/src/isa/x86/mod.rs          |    3 +-
 cranelift/codegen/src/legalizer/boundary.rs   |  382 ++-
 cranelift/codegen/src/legalizer/mod.rs        |    2 +-
 cranelift/codegen/src/stack_layout.rs         |   12 +-
 .../filetests/wasm/multi-val-b1.clif          |   68 +
 .../wasm/multi-val-call-indirect.clif         |   26 +
 .../filetests/wasm/multi-val-f32.clif         |   44 +
 .../filetests/wasm/multi-val-f64.clif         |   44 +
 .../filetests/wasm/multi-val-i32.clif         |   44 +
 .../filetests/wasm/multi-val-i64.clif         |   44 +
 .../filetests/wasm/multi-val-mixed.clif       | 2098 +++++++++++++++++
 .../multi-val-reuse-ret-ptr-stack-slot.clif   |   61 +
 .../wasm/multi-val-sret-slot-alignment.clif   |   51 +
 .../multi-val-take-many-and-return-many.clif  |   18 +
 .../wasm/multi-val-tons-of-results.clif       |   34 +
 29 files changed, 3206 insertions(+), 69 deletions(-)
 create mode 100644 cranelift/filetests/filetests/wasm/multi-val-b1.clif
 create mode 100644 cranelift/filetests/filetests/wasm/multi-val-call-indirect.clif
 create mode 100644 cranelift/filetests/filetests/wasm/multi-val-f32.clif
 create mode 100644 cranelift/filetests/filetests/wasm/multi-val-f64.clif
 create mode 100644 cranelift/filetests/filetests/wasm/multi-val-i32.clif
 create mode 100644 cranelift/filetests/filetests/wasm/multi-val-i64.clif
 create mode 100644 cranelift/filetests/filetests/wasm/multi-val-mixed.clif
 create mode 100644 cranelift/filetests/filetests/wasm/multi-val-reuse-ret-ptr-stack-slot.clif
 create mode 100644 cranelift/filetests/filetests/wasm/multi-val-sret-slot-alignment.clif
 create mode 100644 cranelift/filetests/filetests/wasm/multi-val-take-many-and-return-many.clif
 create mode 100644 cranelift/filetests/filetests/wasm/multi-val-tons-of-results.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index ad497188ec..6148b2ffad 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1420,27 +1420,24 @@ pub(crate) fn define(
     // or 1.
     //
     // Encode movzbq as movzbl, because it's equivalent and shorter.
-    e.enc32(
-        bint.bind(I32).bind(B1),
-        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
-    );
-
-    e.enc64(
-        bint.bind(I64).bind(B1),
-        rec_urm_noflags.opcodes(&MOVZX_BYTE).rex(),
-    );
-    e.enc64(
-        bint.bind(I64).bind(B1),
-        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
-    );
-    e.enc64(
-        bint.bind(I32).bind(B1),
-        rec_urm_noflags.opcodes(&MOVZX_BYTE).rex(),
-    );
-    e.enc64(
-        bint.bind(I32).bind(B1),
-        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
-    );
+    for &to in &[I8, I16, I32, I64] {
+        for &from in &[B1, B8] {
+            e.enc64(
+                bint.bind(to).bind(from),
+                rec_urm_noflags.opcodes(&MOVZX_BYTE).rex(),
+            );
+            e.enc64(
+                bint.bind(to).bind(from),
+                rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
+            );
+            if to != I64 {
+                e.enc32(
+                    bint.bind(to).bind(from),
+                    rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
+                );
+            }
+        }
+    }
 
     // Numerical conversions.
 
diff --git a/cranelift/codegen/src/abi.rs b/cranelift/codegen/src/abi.rs
index 0dc571bba7..f3591c1730 100644
--- a/cranelift/codegen/src/abi.rs
+++ b/cranelift/codegen/src/abi.rs
@@ -4,6 +4,7 @@
 //! `TargetIsa::legalize_signature()` method.
 
 use crate::ir::{AbiParam, ArgumentExtension, ArgumentLoc, Type};
+use alloc::borrow::Cow;
 use alloc::vec::Vec;
 use core::cmp::Ordering;
 
@@ -86,7 +87,9 @@ pub trait ArgAssigner {
 /// Legalize the arguments in `args` using the given argument assigner.
 ///
 /// This function can be used for both arguments and return values.
-pub fn legalize_args(args: &mut Vec, aa: &mut AA) {
+pub fn legalize_args(args: &[AbiParam], aa: &mut AA) -> Option> {
+    let mut args = Cow::Borrowed(args);
+
     // Iterate over the arguments.
     // We may need to mutate the vector in place, so don't use a normal iterator, and clone the
     // argument to avoid holding a reference.
@@ -102,20 +105,25 @@ pub fn legalize_args(args: &mut Vec, aa: &mut AA) {
         match aa.assign(&arg) {
             // Assign argument to a location and move on to the next one.
             ArgAction::Assign(loc) => {
-                args[argno].location = loc;
+                args.to_mut()[argno].location = loc;
                 argno += 1;
             }
             // Split this argument into two smaller ones. Then revisit both.
             ArgAction::Convert(conv) => {
                 let value_type = conv.apply(arg.value_type);
                 let new_arg = AbiParam { value_type, ..arg };
-                args[argno].value_type = value_type;
+                args.to_mut()[argno].value_type = value_type;
                 if conv.is_split() {
-                    args.insert(argno + 1, new_arg);
+                    args.to_mut().insert(argno + 1, new_arg);
                 }
             }
         }
     }
+
+    match args {
+        Cow::Borrowed(_) => None,
+        Cow::Owned(a) => Some(a),
+    }
 }
 
 /// Determine the right action to take when passing a `have` value type to a call signature where
diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index 831ffca575..49f6fdbd73 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -62,6 +62,9 @@ pub struct DataFlowGraph {
     /// well as the external function references.
     pub signatures: PrimaryMap,
 
+    /// The pre-legalization signature for each entry in `signatures`, if any.
+    pub old_signatures: SecondaryMap>,
+
     /// External function references. These are functions that can be called directly.
     pub ext_funcs: PrimaryMap,
 
@@ -85,6 +88,7 @@ impl DataFlowGraph {
             value_lists: ValueListPool::new(),
             values: PrimaryMap::new(),
             signatures: PrimaryMap::new(),
+            old_signatures: SecondaryMap::new(),
             ext_funcs: PrimaryMap::new(),
             values_labels: None,
             constants: ConstantPool::new(),
diff --git a/cranelift/codegen/src/ir/extfunc.rs b/cranelift/codegen/src/ir/extfunc.rs
index 76b249da4a..9274efe9b9 100644
--- a/cranelift/codegen/src/ir/extfunc.rs
+++ b/cranelift/codegen/src/ir/extfunc.rs
@@ -55,6 +55,53 @@ impl Signature {
     pub fn special_param_index(&self, purpose: ArgumentPurpose) -> Option {
         self.params.iter().rposition(|arg| arg.purpose == purpose)
     }
+
+    /// Find the index of a presumed unique special-purpose parameter.
+    pub fn special_return_index(&self, purpose: ArgumentPurpose) -> Option {
+        self.returns.iter().rposition(|arg| arg.purpose == purpose)
+    }
+
+    /// Does this signature have a parameter whose `ArgumentPurpose` is
+    /// `purpose`?
+    pub fn uses_special_param(&self, purpose: ArgumentPurpose) -> bool {
+        self.special_param_index(purpose).is_some()
+    }
+
+    /// Does this signature have a return whose `ArgumentPurpose` is `purpose`?
+    pub fn uses_special_return(&self, purpose: ArgumentPurpose) -> bool {
+        self.special_return_index(purpose).is_some()
+    }
+
+    /// How many special parameters does this function have?
+    pub fn num_special_params(&self) -> usize {
+        self.params
+            .iter()
+            .filter(|p| p.purpose != ArgumentPurpose::Normal)
+            .count()
+    }
+
+    /// How many special returns does this function have?
+    pub fn num_special_returns(&self) -> usize {
+        self.returns
+            .iter()
+            .filter(|r| r.purpose != ArgumentPurpose::Normal)
+            .count()
+    }
+
+    /// Does this signature take an struct return pointer parameter?
+    pub fn uses_struct_return_param(&self) -> bool {
+        self.uses_special_param(ArgumentPurpose::StructReturn)
+    }
+
+    /// Does this return more than one normal value? (Pre-struct return
+    /// legalization)
+    pub fn is_multi_return(&self) -> bool {
+        self.returns
+            .iter()
+            .filter(|r| r.purpose == ArgumentPurpose::Normal)
+            .count()
+            > 1
+    }
 }
 
 /// Wrapper type capable of displaying a `Signature` with correct register names.
diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index 913c2d7be4..5318871689 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -34,6 +34,10 @@ pub struct Function {
     /// Signature of this function.
     pub signature: Signature,
 
+    /// The old signature of this function, before the most recent legalization,
+    /// if any.
+    pub old_signature: Option,
+
     /// Stack slots allocated in this function.
     pub stack_slots: StackSlots,
 
@@ -96,6 +100,7 @@ impl Function {
         Self {
             name,
             signature: sig,
+            old_signature: None,
             stack_slots: StackSlots::new(),
             global_values: PrimaryMap::new(),
             heaps: PrimaryMap::new(),
diff --git a/cranelift/codegen/src/ir/stackslot.rs b/cranelift/codegen/src/ir/stackslot.rs
index 00dee43af4..6a4edd0da6 100644
--- a/cranelift/codegen/src/ir/stackslot.rs
+++ b/cranelift/codegen/src/ir/stackslot.rs
@@ -64,6 +64,15 @@ pub enum StackSlotKind {
     /// stack slots are only valid while setting up a call.
     OutgoingArg,
 
+    /// Space allocated in the caller's frame for the callee's return values
+    /// that are passed out via return pointer.
+    ///
+    /// If there are more return values than registers available for the callee's calling
+    /// convention, or the return value is larger than the available registers' space, then we
+    /// allocate stack space in this frame and pass a pointer to the callee, which then writes its
+    /// return values into this space.
+    StructReturnSlot,
+
     /// An emergency spill slot.
     ///
     /// Emergency slots are allocated late when the register's constraint solver needs extra space
@@ -81,6 +90,7 @@ impl FromStr for StackSlotKind {
             "spill_slot" => Ok(SpillSlot),
             "incoming_arg" => Ok(IncomingArg),
             "outgoing_arg" => Ok(OutgoingArg),
+            "sret_slot" => Ok(StructReturnSlot),
             "emergency_slot" => Ok(EmergencySlot),
             _ => Err(()),
         }
@@ -95,6 +105,7 @@ impl fmt::Display for StackSlotKind {
             SpillSlot => "spill_slot",
             IncomingArg => "incoming_arg",
             OutgoingArg => "outgoing_arg",
+            StructReturnSlot => "sret_slot",
             EmergencySlot => "emergency_slot",
         })
     }
diff --git a/cranelift/codegen/src/isa/arm32/abi.rs b/cranelift/codegen/src/isa/arm32/abi.rs
index 894c67ecb5..85dc8d8f43 100644
--- a/cranelift/codegen/src/isa/arm32/abi.rs
+++ b/cranelift/codegen/src/isa/arm32/abi.rs
@@ -6,6 +6,7 @@ use crate::abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion};
 use crate::ir::{self, AbiParam, ArgumentExtension, ArgumentLoc, Type};
 use crate::isa::RegClass;
 use crate::regalloc::RegisterSet;
+use alloc::borrow::Cow;
 use core::i32;
 use target_lexicon::Triple;
 
@@ -78,11 +79,13 @@ impl ArgAssigner for Args {
 }
 
 /// Legalize `sig`.
-pub fn legalize_signature(sig: &mut ir::Signature, triple: &Triple, _current: bool) {
+pub fn legalize_signature(sig: &mut Cow, triple: &Triple, _current: bool) {
     let bits = triple.pointer_width().unwrap().bits();
 
     let mut args = Args::new(bits);
-    legalize_args(&mut sig.params, &mut args);
+    if let Some(new_params) = legalize_args(&sig.params, &mut args) {
+        sig.to_mut().params = new_params;
+    }
 }
 
 /// Get register class for a type appearing in a legalized signature.
diff --git a/cranelift/codegen/src/isa/arm32/mod.rs b/cranelift/codegen/src/isa/arm32/mod.rs
index 6c00ef9089..0358a70113 100644
--- a/cranelift/codegen/src/isa/arm32/mod.rs
+++ b/cranelift/codegen/src/isa/arm32/mod.rs
@@ -15,6 +15,7 @@ use crate::isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encoding
 use crate::isa::Builder as IsaBuilder;
 use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
+use alloc::borrow::Cow;
 use alloc::boxed::Box;
 use core::fmt;
 use target_lexicon::{Architecture, Triple};
@@ -100,7 +101,7 @@ impl TargetIsa for Isa {
         )
     }
 
-    fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) {
+    fn legalize_signature(&self, sig: &mut Cow, current: bool) {
         abi::legalize_signature(sig, &self.triple, current)
     }
 
diff --git a/cranelift/codegen/src/isa/arm64/abi.rs b/cranelift/codegen/src/isa/arm64/abi.rs
index efc1f6125d..8d486d4193 100644
--- a/cranelift/codegen/src/isa/arm64/abi.rs
+++ b/cranelift/codegen/src/isa/arm64/abi.rs
@@ -5,10 +5,11 @@ use crate::ir;
 use crate::isa::RegClass;
 use crate::regalloc::RegisterSet;
 use crate::settings as shared_settings;
+use alloc::borrow::Cow;
 
 /// Legalize `sig`.
 pub fn legalize_signature(
-    _sig: &mut ir::Signature,
+    _sig: &mut Cow,
     _flags: &shared_settings::Flags,
     _current: bool,
 ) {
diff --git a/cranelift/codegen/src/isa/arm64/mod.rs b/cranelift/codegen/src/isa/arm64/mod.rs
index 26c28329bc..f00062b2af 100644
--- a/cranelift/codegen/src/isa/arm64/mod.rs
+++ b/cranelift/codegen/src/isa/arm64/mod.rs
@@ -15,6 +15,7 @@ use crate::isa::enc_tables::{lookup_enclist, Encodings};
 use crate::isa::Builder as IsaBuilder;
 use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
+use alloc::borrow::Cow;
 use alloc::boxed::Box;
 use core::fmt;
 use target_lexicon::Triple;
@@ -88,7 +89,7 @@ impl TargetIsa for Isa {
         )
     }
 
-    fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) {
+    fn legalize_signature(&self, sig: &mut Cow, current: bool) {
         abi::legalize_signature(sig, &self.shared_flags, current)
     }
 
diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index 15f9d59c60..aaa802b935 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -63,6 +63,7 @@ use crate::result::CodegenResult;
 use crate::settings;
 use crate::settings::SetResult;
 use crate::timing;
+use alloc::borrow::Cow;
 use alloc::boxed::Box;
 use alloc::vec::Vec;
 use core::fmt;
@@ -315,7 +316,7 @@ pub trait TargetIsa: fmt::Display + Sync {
     /// Arguments and return values for the caller's frame pointer and other callee-saved registers
     /// should not be added by this function. These arguments are not added until after register
     /// allocation.
-    fn legalize_signature(&self, sig: &mut ir::Signature, current: bool);
+    fn legalize_signature(&self, sig: &mut Cow, current: bool);
 
     /// Get the register class that should be used to represent an ABI argument or return value of
     /// type `ty`. This should be the top-level register class that contains the argument
diff --git a/cranelift/codegen/src/isa/riscv/abi.rs b/cranelift/codegen/src/isa/riscv/abi.rs
index 59b266f7ca..44c5f36afe 100644
--- a/cranelift/codegen/src/isa/riscv/abi.rs
+++ b/cranelift/codegen/src/isa/riscv/abi.rs
@@ -11,6 +11,7 @@ use crate::abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion};
 use crate::ir::{self, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, Type};
 use crate::isa::RegClass;
 use crate::regalloc::RegisterSet;
+use alloc::borrow::Cow;
 use core::i32;
 use target_lexicon::Triple;
 
@@ -88,7 +89,7 @@ impl ArgAssigner for Args {
 
 /// Legalize `sig` for RISC-V.
 pub fn legalize_signature(
-    sig: &mut ir::Signature,
+    sig: &mut Cow,
     triple: &Triple,
     isa_flags: &settings::Flags,
     current: bool,
@@ -96,10 +97,14 @@ pub fn legalize_signature(
     let bits = triple.pointer_width().unwrap().bits();
 
     let mut args = Args::new(bits, isa_flags.enable_e());
-    legalize_args(&mut sig.params, &mut args);
+    if let Some(new_params) = legalize_args(&sig.params, &mut args) {
+        sig.to_mut().params = new_params;
+    }
 
     let mut rets = Args::new(bits, isa_flags.enable_e());
-    legalize_args(&mut sig.returns, &mut rets);
+    if let Some(new_returns) = legalize_args(&sig.returns, &mut rets) {
+        sig.to_mut().returns = new_returns;
+    }
 
     if current {
         let ptr = Type::int(u16::from(bits)).unwrap();
@@ -110,8 +115,8 @@ pub fn legalize_signature(
         // in any register, but a micro-architecture with a return address predictor will only
         // recognize it as a return if the address is in `x1`.
         let link = AbiParam::special_reg(ptr, ArgumentPurpose::Link, GPR.unit(1));
-        sig.params.push(link);
-        sig.returns.push(link);
+        sig.to_mut().params.push(link);
+        sig.to_mut().returns.push(link);
     }
 }
 
diff --git a/cranelift/codegen/src/isa/riscv/mod.rs b/cranelift/codegen/src/isa/riscv/mod.rs
index 79aaeaddf9..25244fab81 100644
--- a/cranelift/codegen/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/src/isa/riscv/mod.rs
@@ -15,6 +15,7 @@ use crate::isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encoding
 use crate::isa::Builder as IsaBuilder;
 use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
+use alloc::borrow::Cow;
 use alloc::boxed::Box;
 use core::fmt;
 use target_lexicon::{PointerWidth, Triple};
@@ -95,7 +96,7 @@ impl TargetIsa for Isa {
         )
     }
 
-    fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) {
+    fn legalize_signature(&self, sig: &mut Cow, current: bool) {
         abi::legalize_signature(sig, &self.triple, &self.isa_flags, current)
     }
 
diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index d78af9b448..6d09635949 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -17,6 +17,7 @@ use crate::isa::{CallConv, RegClass, RegUnit, TargetIsa};
 use crate::regalloc::RegisterSet;
 use crate::result::CodegenResult;
 use crate::stack_layout::layout_stack;
+use alloc::borrow::Cow;
 use alloc::vec::Vec;
 use core::i32;
 use target_lexicon::{PointerWidth, Triple};
@@ -166,9 +167,117 @@ impl ArgAssigner for Args {
     }
 }
 
+/// Get the number of general-purpose and floating-point registers required to
+/// hold the given `AbiParam` returns.
+fn num_return_registers_required<'a>(
+    word_bit_size: u8,
+    call_conv: CallConv,
+    shared_flags: &shared_settings::Flags,
+    isa_flags: &isa_settings::Flags,
+    return_params: impl IntoIterator,
+) -> (usize, usize) {
+    // Pretend we have "infinite" registers to give out, since we aren't
+    // actually assigning `AbiParam`s to registers yet, just seeing how many
+    // registers we would need in order to fit all the `AbiParam`s in registers.
+    let gprs = &[RU::rax; 128];
+    let fpr_limit = std::usize::MAX;
+
+    let mut assigner = Args::new(
+        word_bit_size,
+        gprs,
+        fpr_limit,
+        call_conv,
+        shared_flags,
+        isa_flags,
+    );
+
+    let mut gprs_required = 0;
+    let mut fprs_required = 0;
+
+    for param in return_params {
+        match param.location {
+            ArgumentLoc::Unassigned => {
+                // Let this fall through so that we assign it a location and
+                // account for how many registers it ends up requiring below...
+            }
+            ArgumentLoc::Reg(_) => {
+                // This is already assigned to a register. Count it.
+                if param.value_type.is_float() {
+                    fprs_required += 1;
+                } else {
+                    gprs_required += 1;
+                }
+                continue;
+            }
+            _ => {
+                // It is already assigned, but not to a register. Skip it.
+                continue;
+            }
+        }
+
+        // We're going to mutate the type as it gets converted, so make our own
+        // copy that isn't visible to the outside world.
+        let mut param = param.clone();
+
+        let mut split_factor = 1;
+
+        loop {
+            match assigner.assign(¶m) {
+                ArgAction::Convert(ValueConversion::IntSplit) => {
+                    split_factor *= 2;
+                    param.value_type = param.value_type.half_width().unwrap();
+                }
+                ArgAction::Convert(ValueConversion::VectorSplit) => {
+                    split_factor *= 2;
+                    param.value_type = param.value_type.half_vector().unwrap();
+                }
+                ArgAction::Assign(ArgumentLoc::Reg(_))
+                | ArgAction::Convert(ValueConversion::IntBits)
+                | ArgAction::Convert(ValueConversion::Sext(_))
+                | ArgAction::Convert(ValueConversion::Uext(_)) => {
+                    // Ok! We can fit this (potentially split) value into a
+                    // register! Add the number of params we split the parameter
+                    // into to our current counts.
+                    if param.value_type.is_float() {
+                        fprs_required += split_factor;
+                    } else {
+                        gprs_required += split_factor;
+                    }
+
+                    // But we also have to call `assign` once for each split value, to
+                    // update `assigner`'s internal state.
+                    for _ in 1..split_factor {
+                        match assigner.assign(¶m) {
+                            ArgAction::Assign(_)
+                            | ArgAction::Convert(ValueConversion::IntBits)
+                            | ArgAction::Convert(ValueConversion::Sext(_))
+                            | ArgAction::Convert(ValueConversion::Uext(_)) => {
+                                continue;
+                            }
+                            otherwise => panic!(
+                                "unexpected action after first split succeeded: {:?}",
+                                otherwise
+                            ),
+                        }
+                    }
+
+                    // Continue to the next param.
+                    break;
+                }
+                ArgAction::Assign(loc) => panic!(
+                    "unexpected location assignment, should have had enough registers: {:?}",
+                    loc
+                ),
+            }
+        }
+    }
+
+    (gprs_required, fprs_required)
+}
+
 /// Legalize `sig`.
 pub fn legalize_signature(
-    sig: &mut ir::Signature,
+    sig: &mut Cow,
     triple: &Triple,
     _current: bool,
     shared_flags: &shared_settings::Flags,
@@ -207,9 +316,7 @@ pub fn legalize_signature(
         }
     }
 
-    legalize_args(&mut sig.params, &mut args);
-
-    let (regs, fpr_limit) = if sig.call_conv.extends_windows_fastcall() {
+    let (ret_regs, ret_fpr_limit) = if sig.call_conv.extends_windows_fastcall() {
         // windows-x64 calling convention only uses XMM0 or RAX for return values
         (&RET_GPRS_WIN_FASTCALL_X64[..], 1)
     } else {
@@ -218,13 +325,77 @@ pub fn legalize_signature(
 
     let mut rets = Args::new(
         bits,
-        regs,
-        fpr_limit,
+        ret_regs,
+        ret_fpr_limit,
         sig.call_conv,
         shared_flags,
         isa_flags,
     );
-    legalize_args(&mut sig.returns, &mut rets);
+
+    if sig.is_multi_return() && {
+        // Even if it is multi-return, see if the return values will fit into
+        // our available return registers.
+        let (gprs_required, fprs_required) = num_return_registers_required(
+            bits,
+            sig.call_conv,
+            shared_flags,
+            isa_flags,
+            &sig.returns,
+        );
+        gprs_required > ret_regs.len() || fprs_required > ret_fpr_limit
+    } {
+        debug_assert!(!sig.uses_struct_return_param());
+
+        // We're using the first register for the return pointer parameter.
+        let mut ret_ptr_param = AbiParam {
+            value_type: args.pointer_type,
+            purpose: ArgumentPurpose::StructReturn,
+            extension: ArgumentExtension::None,
+            location: ArgumentLoc::Unassigned,
+        };
+        match args.assign(&ret_ptr_param) {
+            ArgAction::Assign(ArgumentLoc::Reg(reg)) => {
+                ret_ptr_param.location = ArgumentLoc::Reg(reg);
+                sig.to_mut().params.push(ret_ptr_param);
+            }
+            _ => unreachable!("return pointer should always get a register assignment"),
+        }
+
+        // We're using the first return register for the return pointer (like
+        // sys v does).
+        let mut ret_ptr_return = AbiParam {
+            value_type: args.pointer_type,
+            purpose: ArgumentPurpose::StructReturn,
+            extension: ArgumentExtension::None,
+            location: ArgumentLoc::Unassigned,
+        };
+        match rets.assign(&ret_ptr_return) {
+            ArgAction::Assign(ArgumentLoc::Reg(reg)) => {
+                ret_ptr_return.location = ArgumentLoc::Reg(reg);
+                sig.to_mut().returns.push(ret_ptr_return);
+            }
+            _ => unreachable!("return pointer should always get a register assignment"),
+        }
+
+        sig.to_mut().returns.retain(|ret| {
+            // Either this is the return pointer, in which case we want to keep
+            // it, or else assume that it is assigned for a reason and doesn't
+            // conflict with our return pointering legalization.
+            debug_assert_eq!(
+                ret.location.is_assigned(),
+                ret.purpose != ArgumentPurpose::Normal
+            );
+            ret.location.is_assigned()
+        });
+    }
+
+    if let Some(new_params) = legalize_args(&sig.params, &mut args) {
+        sig.to_mut().params = new_params;
+    }
+
+    if let Some(new_returns) = legalize_args(&sig.returns, &mut rets) {
+        sig.to_mut().returns = new_returns;
+    }
 }
 
 /// Get register class for a type appearing in a legalized signature.
diff --git a/cranelift/codegen/src/isa/x86/mod.rs b/cranelift/codegen/src/isa/x86/mod.rs
index cfba3fef48..e2e785675f 100644
--- a/cranelift/codegen/src/isa/x86/mod.rs
+++ b/cranelift/codegen/src/isa/x86/mod.rs
@@ -18,6 +18,7 @@ use crate::isa::{EncInfo, RegClass, RegInfo, TargetIsa};
 use crate::regalloc;
 use crate::result::CodegenResult;
 use crate::timing;
+use alloc::borrow::Cow;
 use alloc::boxed::Box;
 use alloc::vec::Vec;
 use core::fmt;
@@ -107,7 +108,7 @@ impl TargetIsa for Isa {
         )
     }
 
-    fn legalize_signature(&self, sig: &mut ir::Signature, current: bool) {
+    fn legalize_signature(&self, sig: &mut Cow, current: bool) {
         abi::legalize_signature(
             sig,
             &self.triple,
diff --git a/cranelift/codegen/src/legalizer/boundary.rs b/cranelift/codegen/src/legalizer/boundary.rs
index 479208751f..5063b6d910 100644
--- a/cranelift/codegen/src/legalizer/boundary.rs
+++ b/cranelift/codegen/src/legalizer/boundary.rs
@@ -23,11 +23,14 @@ use crate::flowgraph::ControlFlowGraph;
 use crate::ir::instructions::CallInfo;
 use crate::ir::{
     AbiParam, ArgumentLoc, ArgumentPurpose, DataFlowGraph, Ebb, Function, Inst, InstBuilder,
-    SigRef, Signature, Type, Value, ValueLoc,
+    MemFlags, SigRef, Signature, StackSlotData, StackSlotKind, Type, Value, ValueLoc,
 };
 use crate::isa::TargetIsa;
 use crate::legalizer::split::{isplit, vsplit};
+use alloc::borrow::Cow;
 use alloc::vec::Vec;
+use core::mem;
+use cranelift_entity::EntityList;
 use log::debug;
 
 /// Legalize all the function signatures in `func`.
@@ -36,9 +39,16 @@ use log::debug;
 /// change the entry block arguments, calls, or return instructions, so this can leave the function
 /// in a state with type discrepancies.
 pub fn legalize_signatures(func: &mut Function, isa: &dyn TargetIsa) {
-    legalize_signature(&mut func.signature, true, isa);
-    for sig_data in func.dfg.signatures.values_mut() {
-        legalize_signature(sig_data, false, isa);
+    if let Some(new) = legalize_signature(&func.signature, true, isa) {
+        let old = mem::replace(&mut func.signature, new);
+        func.old_signature = Some(old);
+    }
+
+    for (sig_ref, sig_data) in func.dfg.signatures.iter_mut() {
+        if let Some(new) = legalize_signature(sig_data, false, isa) {
+            let old = mem::replace(sig_data, new);
+            func.dfg.old_signatures[sig_ref] = Some(old);
+        }
     }
 
     if let Some(entry) = func.layout.entry_block() {
@@ -50,14 +60,25 @@ pub fn legalize_signatures(func: &mut Function, isa: &dyn TargetIsa) {
 /// Legalize the libcall signature, which we may generate on the fly after
 /// `legalize_signatures` has been called.
 pub fn legalize_libcall_signature(signature: &mut Signature, isa: &dyn TargetIsa) {
-    legalize_signature(signature, false, isa);
+    if let Some(s) = legalize_signature(signature, false, isa) {
+        *signature = s;
+    }
 }
 
 /// Legalize the given signature.
 ///
 /// `current` is true if this is the signature for the current function.
-fn legalize_signature(signature: &mut Signature, current: bool, isa: &dyn TargetIsa) {
-    isa.legalize_signature(signature, current);
+fn legalize_signature(
+    signature: &Signature,
+    current: bool,
+    isa: &dyn TargetIsa,
+) -> Option {
+    let mut cow = Cow::Borrowed(signature);
+    isa.legalize_signature(&mut cow, current);
+    match cow {
+        Cow::Borrowed(_) => None,
+        Cow::Owned(s) => Some(s),
+    }
 }
 
 /// Legalize the entry block parameters after `func`'s signature has been legalized.
@@ -245,6 +266,166 @@ where
     call
 }
 
+fn assert_is_valid_sret_legalization(
+    old_ret_list: &EntityList,
+    old_sig: &Signature,
+    new_sig: &Signature,
+    pos: &FuncCursor,
+) {
+    debug_assert_eq!(
+        old_sig.returns.len(),
+        old_ret_list.len(&pos.func.dfg.value_lists)
+    );
+
+    // Assert that the only difference in special parameters is that there
+    // is an appended struct return pointer parameter.
+    let old_special_params: Vec<_> = old_sig
+        .params
+        .iter()
+        .filter(|r| r.purpose != ArgumentPurpose::Normal)
+        .collect();
+    let new_special_params: Vec<_> = new_sig
+        .params
+        .iter()
+        .filter(|r| r.purpose != ArgumentPurpose::Normal)
+        .collect();
+    debug_assert_eq!(old_special_params.len() + 1, new_special_params.len());
+    debug_assert!(old_special_params
+        .iter()
+        .zip(&new_special_params)
+        .all(|(old, new)| old.purpose == new.purpose));
+    debug_assert_eq!(
+        new_special_params.last().unwrap().purpose,
+        ArgumentPurpose::StructReturn
+    );
+
+    // If the special returns have changed at all, then the only change
+    // should be that the struct return pointer is returned back out of the
+    // function, so that callers don't have to load its stack address again.
+    let old_special_returns: Vec<_> = old_sig
+        .returns
+        .iter()
+        .filter(|r| r.purpose != ArgumentPurpose::Normal)
+        .collect();
+    let new_special_returns: Vec<_> = new_sig
+        .returns
+        .iter()
+        .filter(|r| r.purpose != ArgumentPurpose::Normal)
+        .collect();
+    debug_assert!(old_special_returns
+        .iter()
+        .zip(&new_special_returns)
+        .all(|(old, new)| old.purpose == new.purpose));
+    debug_assert!(
+        old_special_returns.len() == new_special_returns.len()
+            || (old_special_returns.len() + 1 == new_special_returns.len()
+                && new_special_returns.last().unwrap().purpose == ArgumentPurpose::StructReturn)
+    );
+}
+
+fn legalize_sret_call(isa: &dyn TargetIsa, pos: &mut FuncCursor, sig_ref: SigRef, call: Inst) {
+    let old_ret_list = pos.func.dfg.detach_results(call);
+    let old_sig = pos.func.dfg.old_signatures[sig_ref]
+        .take()
+        .expect("must have an old signature when using an `sret` parameter");
+
+    // We make a bunch of assumptions about the shape of the old, multi-return
+    // signature and the new, sret-using signature in this legalization
+    // function. Assert that these assumptions hold true in debug mode.
+    if cfg!(debug_assertions) {
+        assert_is_valid_sret_legalization(
+            &old_ret_list,
+            &old_sig,
+            &pos.func.dfg.signatures[sig_ref],
+            &pos,
+        );
+    }
+
+    // Go through and remove all normal return values from the `call`
+    // instruction's returns list. These will be stored into the stack slot that
+    // the sret points to. At the same time, calculate the size of the sret
+    // stack slot.
+    let mut sret_slot_size = 0;
+    for (i, ret) in old_sig.returns.iter().enumerate() {
+        let v = old_ret_list.get(i, &pos.func.dfg.value_lists).unwrap();
+        let ty = pos.func.dfg.value_type(v);
+        if ret.purpose == ArgumentPurpose::Normal {
+            debug_assert_eq!(ret.location, ArgumentLoc::Unassigned);
+            let ty = legalized_type_for_sret(ty);
+            let size = ty.bytes();
+            sret_slot_size = round_up_to_multiple_of_type_align(sret_slot_size, ty) + size;
+        } else {
+            let new_v = pos.func.dfg.append_result(call, ty);
+            pos.func.dfg.change_to_alias(v, new_v);
+        }
+    }
+
+    let stack_slot = pos.func.stack_slots.push(StackSlotData {
+        kind: StackSlotKind::StructReturnSlot,
+        size: sret_slot_size,
+        offset: None,
+    });
+
+    // Append the sret pointer to the `call` instruction's arguments.
+    let ptr_type = Type::triple_pointer_type(isa.triple());
+    let sret_arg = pos.ins().stack_addr(ptr_type, stack_slot, 0);
+    pos.func.dfg.append_inst_arg(call, sret_arg);
+
+    // The sret pointer might be returned by the signature as well. If so, we
+    // need to add it to the `call` instruction's results list.
+    //
+    // Additionally, when the sret is explicitly returned in this calling
+    // convention, then use it when loading the sret returns back into ssa
+    // values to avoid keeping the original `sret_arg` live and potentially
+    // having to do spills and fills.
+    let sret =
+        if pos.func.dfg.signatures[sig_ref].uses_special_return(ArgumentPurpose::StructReturn) {
+            pos.func.dfg.append_result(call, ptr_type)
+        } else {
+            sret_arg
+        };
+
+    // Finally, load each of the call's return values out of the sret stack
+    // slot.
+    pos.goto_after_inst(call);
+    let mut offset = 0;
+    for i in 0..old_ret_list.len(&pos.func.dfg.value_lists) {
+        if old_sig.returns[i].purpose != ArgumentPurpose::Normal {
+            continue;
+        }
+
+        let old_v = old_ret_list.get(i, &pos.func.dfg.value_lists).unwrap();
+        let ty = pos.func.dfg.value_type(old_v);
+        let mut legalized_ty = legalized_type_for_sret(ty);
+
+        offset = round_up_to_multiple_of_type_align(offset, legalized_ty);
+
+        let new_legalized_v =
+            pos.ins()
+                .load(legalized_ty, MemFlags::trusted(), sret, offset as i32);
+
+        // "Illegalize" the loaded value from the legalized type back to its
+        // original `ty`. This is basically the opposite of
+        // `legalize_type_for_sret_store`.
+        let mut new_v = new_legalized_v;
+        if ty.is_bool() {
+            legalized_ty = legalized_ty.as_bool_pedantic();
+            new_v = pos.ins().raw_bitcast(legalized_ty, new_v);
+
+            if ty.bits() < legalized_ty.bits() {
+                legalized_ty = ty;
+                new_v = pos.ins().breduce(legalized_ty, new_v);
+            }
+        }
+
+        pos.func.dfg.change_to_alias(old_v, new_v);
+
+        offset += legalized_ty.bytes();
+    }
+
+    pos.func.dfg.old_signatures[sig_ref] = Some(old_sig);
+}
+
 /// Compute original value of type `ty` from the legalized ABI arguments.
 ///
 /// The conversion is recursive, controlled by the `get_arg` closure which is called to retrieve an
@@ -452,6 +633,13 @@ fn legalize_inst_arguments(
         .constraints()
         .num_fixed_value_arguments();
     let have_args = vlist.len(&pos.func.dfg.value_lists) - num_fixed_values;
+    if abi_args < have_args {
+        // This happens with multiple return values after we've legalized the
+        // signature but haven't legalized the return instruction yet. This
+        // legalization is handled in `handle_return_abi`.
+        pos.func.dfg[inst].put_value_list(vlist);
+        return;
+    }
 
     // Grow the value list to the right size and shift all the existing arguments to the right.
     // This lets us write the new argument values into the list without overwriting the old
@@ -508,6 +696,32 @@ fn legalize_inst_arguments(
     pos.func.dfg[inst].put_value_list(vlist);
 }
 
+/// Ensure that the `ty` being returned is a type that can be loaded and stored
+/// (potentially after another narrowing legalization) from memory, since it
+/// will go into the `sret` space.
+fn legalized_type_for_sret(ty: Type) -> Type {
+    if ty.is_bool() {
+        let bits = std::cmp::max(8, ty.bits());
+        Type::int(bits).unwrap()
+    } else {
+        ty
+    }
+}
+
+/// Insert any legalization code required to ensure that `val` can be stored
+/// into the `sret` memory. Returns the (potentially new, potentially
+/// unmodified) legalized value and its type.
+fn legalize_type_for_sret_store(pos: &mut FuncCursor, val: Value, ty: Type) -> (Value, Type) {
+    if ty.is_bool() {
+        let bits = std::cmp::max(8, ty.bits());
+        let ty = Type::int(bits).unwrap();
+        let val = pos.ins().bint(ty, val);
+        (val, ty)
+    } else {
+        (val, ty)
+    }
+}
+
 /// Insert ABI conversion code before and after the call instruction at `pos`.
 ///
 /// Instructions inserted before the call will compute the appropriate ABI values for the
@@ -518,7 +732,12 @@ fn legalize_inst_arguments(
 /// original return values. The call's result values will be adapted to match the new signature.
 ///
 /// Returns `true` if any instructions were inserted.
-pub fn handle_call_abi(mut inst: Inst, func: &mut Function, cfg: &ControlFlowGraph) -> bool {
+pub fn handle_call_abi(
+    isa: &dyn TargetIsa,
+    mut inst: Inst,
+    func: &mut Function,
+    cfg: &ControlFlowGraph,
+) -> bool {
     let pos = &mut FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
 
@@ -528,16 +747,27 @@ pub fn handle_call_abi(mut inst: Inst, func: &mut Function, cfg: &ControlFlowGra
         Err(s) => s,
     };
 
-    // OK, we need to fix the call arguments to match the ABI signature.
-    let abi_args = pos.func.dfg.signatures[sig_ref].params.len();
-    legalize_inst_arguments(pos, cfg, abi_args, |func, abi_arg| {
-        func.dfg.signatures[sig_ref].params[abi_arg]
-    });
+    let sig = &pos.func.dfg.signatures[sig_ref];
+    let old_sig = &pos.func.dfg.old_signatures[sig_ref];
 
-    if !pos.func.dfg.signatures[sig_ref].returns.is_empty() {
-        inst = legalize_inst_results(pos, |func, abi_res| {
-            func.dfg.signatures[sig_ref].returns[abi_res]
+    if sig.uses_struct_return_param()
+        && old_sig
+            .as_ref()
+            .map_or(false, |s| !s.uses_struct_return_param())
+    {
+        legalize_sret_call(isa, pos, sig_ref, inst);
+    } else {
+        // OK, we need to fix the call arguments to match the ABI signature.
+        let abi_args = pos.func.dfg.signatures[sig_ref].params.len();
+        legalize_inst_arguments(pos, cfg, abi_args, |func, abi_arg| {
+            func.dfg.signatures[sig_ref].params[abi_arg]
         });
+
+        if !pos.func.dfg.signatures[sig_ref].returns.is_empty() {
+            inst = legalize_inst_results(pos, |func, abi_res| {
+                func.dfg.signatures[sig_ref].returns[abi_res]
+            });
+        }
     }
 
     debug_assert!(
@@ -586,8 +816,6 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph
     legalize_inst_arguments(pos, cfg, abi_args, |func, abi_arg| {
         func.signature.returns[abi_arg]
     });
-    debug_assert_eq!(pos.func.dfg.inst_variable_args(inst).len(), abi_args);
-
     // Append special return arguments for any `sret`, `link`, and `vmctx` return values added to
     // the legalized signature. These values should simply be propagated from the entry block
     // arguments.
@@ -598,6 +826,8 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph
             pos.func.dfg.display_inst(inst, None)
         );
         let mut vlist = pos.func.dfg[inst].take_value_list().unwrap();
+        let mut sret = None;
+
         for arg in &pos.func.signature.returns[abi_args..] {
             match arg.purpose {
                 ArgumentPurpose::Link
@@ -624,10 +854,45 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph
                 .ebb_params(pos.func.layout.entry_block().unwrap())[idx];
             debug_assert_eq!(pos.func.dfg.value_type(val), arg.value_type);
             vlist.push(val, &mut pos.func.dfg.value_lists);
+
+            if let ArgumentPurpose::StructReturn = arg.purpose {
+                sret = Some(val);
+            }
+        }
+
+        // Store all the regular returns into the retptr space and remove them
+        // from the `return` instruction's value list.
+        if let Some(sret) = sret {
+            let mut offset = 0;
+            let num_regular_rets = vlist.len(&pos.func.dfg.value_lists) - special_args;
+            for i in 0..num_regular_rets {
+                debug_assert_eq!(
+                    pos.func.old_signature.as_ref().unwrap().returns[i].purpose,
+                    ArgumentPurpose::Normal,
+                );
+
+                // The next return value to process is always at `0`, since the
+                // list is emptied as we iterate.
+                let v = vlist.get(0, &pos.func.dfg.value_lists).unwrap();
+                let ty = pos.func.dfg.value_type(v);
+                let (v, ty) = legalize_type_for_sret_store(pos, v, ty);
+
+                let size = ty.bytes();
+                offset = round_up_to_multiple_of_type_align(offset, ty);
+
+                pos.ins().store(MemFlags::trusted(), v, sret, offset as i32);
+                vlist.remove(0, &mut pos.func.dfg.value_lists);
+
+                offset += size;
+            }
         }
         pos.func.dfg[inst].put_value_list(vlist);
     }
 
+    debug_assert_eq!(
+        pos.func.dfg.inst_variable_args(inst).len(),
+        abi_args + special_args
+    );
     debug_assert!(
         check_return_signature(&pos.func.dfg, inst, &pos.func.signature),
         "Signature still wrong: {} / signature {}",
@@ -639,6 +904,56 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph
     true
 }
 
+fn round_up_to_multiple_of_type_align(bytes: u32, ty: Type) -> u32 {
+    // We don't have a dedicated alignment for types, so assume they are
+    // size-aligned.
+    let align = ty.bytes();
+    round_up_to_multiple_of_pow2(bytes, align)
+}
+
+/// Round `n` up to the next multiple of `to` that is greater than or equal to
+/// `n`.
+///
+/// `to` must be a power of two and greater than zero.
+///
+/// This is useful for rounding an offset or pointer up to some type's required
+/// alignment.
+fn round_up_to_multiple_of_pow2(n: u32, to: u32) -> u32 {
+    debug_assert!(to > 0);
+    debug_assert!(to.is_power_of_two());
+
+    // The simple version of this function is
+    //
+    //     (n + to - 1) / to * to
+    //
+    // Consider the numerator: `n + to - 1`. This is ensuring that if there is
+    // any remainder for `n / to`, then the result of the division is one
+    // greater than `n / to`, and that otherwise we get exactly the same result
+    // as `n / to` due to integer division rounding off the remainder. In other
+    // words, we only round up if `n` is not aligned to `to`.
+    //
+    // However, we know `to` is a power of two, and therefore `anything / to` is
+    // equivalent to `anything >> log2(to)` and `anything * to` is equivalent to
+    // `anything << log2(to)`. We can therefore rewrite our simplified function
+    // into the following:
+    //
+    //     (n + to - 1) >> log2(to) << log2(to)
+    //
+    // But shifting a value right by some number of bits `b` and then shifting
+    // it left by that same number of bits `b` is equivalent to clearing the
+    // bottom `b` bits of the number. We can clear the bottom `b` bits of a
+    // number by bit-wise and'ing the number with the bit-wise not of `2^b - 1`.
+    // Plugging this into our function and simplifying, we get:
+    //
+    //       (n + to - 1) >> log2(to) << log2(to)
+    //     = (n + to - 1) & !(2^log2(to) - 1)
+    //     = (n + to - 1) & !(to - 1)
+    //
+    // And now we have the final version of this function!
+
+    (n + to - 1) & !(to - 1)
+}
+
 /// Assign stack slots to incoming function parameters on the stack.
 ///
 /// Values that are passed into the function on the stack must be assigned to an `IncomingArg`
@@ -714,3 +1029,34 @@ fn spill_call_arguments(pos: &mut FuncCursor) -> bool {
     // We changed stuff.
     true
 }
+
+#[cfg(test)]
+mod tests {
+    use super::round_up_to_multiple_of_pow2;
+
+    #[test]
+    fn round_up_to_multiple_of_pow2_works() {
+        for (n, to, expected) in vec![
+            (0, 1, 0),
+            (1, 1, 1),
+            (2, 1, 2),
+            (0, 2, 0),
+            (1, 2, 2),
+            (2, 2, 2),
+            (3, 2, 4),
+            (0, 4, 0),
+            (1, 4, 4),
+            (2, 4, 4),
+            (3, 4, 4),
+            (4, 4, 4),
+            (5, 4, 8),
+        ] {
+            let actual = round_up_to_multiple_of_pow2(n, to);
+            assert_eq!(
+                actual, expected,
+                "round_up_to_multiple_of_pow2(n = {}, to = {}) = {} (expected {})",
+                n, to, actual, expected
+            );
+        }
+    }
+}
diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 0d6a6c0bb9..dcebff7ce4 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -55,7 +55,7 @@ fn legalize_inst(
 
     // Check for ABI boundaries that need to be converted to the legalized signature.
     if opcode.is_call() {
-        if boundary::handle_call_abi(inst, pos.func, cfg) {
+        if boundary::handle_call_abi(isa, inst, pos.func, cfg) {
             return LegalizeInstResult::Legalized;
         }
     } else if opcode.is_return() {
diff --git a/cranelift/codegen/src/stack_layout.rs b/cranelift/codegen/src/stack_layout.rs
index 0f5c8ae639..732f9365c0 100644
--- a/cranelift/codegen/src/stack_layout.rs
+++ b/cranelift/codegen/src/stack_layout.rs
@@ -25,7 +25,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CodegenResu
     // stack layout from high to low addresses will be:
     //
     // 1. incoming arguments.
-    // 2. spills + explicits.
+    // 2. spills + explicits + struct returns.
     // 3. outgoing arguments.
     //
     // The incoming arguments can have both positive and negative offsets. A negative offset
@@ -56,7 +56,8 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CodegenResu
                     .ok_or(CodegenError::ImplLimitExceeded)?;
                 outgoing_max = max(outgoing_max, offset);
             }
-            StackSlotKind::SpillSlot
+            StackSlotKind::StructReturnSlot
+            | StackSlotKind::SpillSlot
             | StackSlotKind::ExplicitSlot
             | StackSlotKind::EmergencySlot => {
                 // Determine the smallest alignment of any explicit or spill slot.
@@ -65,9 +66,9 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CodegenResu
         }
     }
 
-    // Lay out spill slots and explicit slots below the incoming arguments.
-    // The offset is negative, growing downwards.
-    // Start with the smallest alignments for better packing.
+    // Lay out spill slots, struct return slots, and explicit slots below the
+    // incoming arguments. The offset is negative, growing downwards. Start with
+    // the smallest alignments for better packing.
     let mut offset = incoming_min;
     debug_assert!(min_align.is_power_of_two());
     while min_align <= alignment {
@@ -75,6 +76,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CodegenResu
             // Pick out explicit and spill slots with exact alignment `min_align`.
             match slot.kind {
                 StackSlotKind::SpillSlot
+                | StackSlotKind::StructReturnSlot
                 | StackSlotKind::ExplicitSlot
                 | StackSlotKind::EmergencySlot => {
                     if slot.alignment(alignment) != min_align {
diff --git a/cranelift/filetests/filetests/wasm/multi-val-b1.clif b/cranelift/filetests/filetests/wasm/multi-val-b1.clif
new file mode 100644
index 0000000000..582403dcfb
--- /dev/null
+++ b/cranelift/filetests/filetests/wasm/multi-val-b1.clif
@@ -0,0 +1,68 @@
+test compile
+target x86_64 haswell
+
+;; `b1` return values need to be legalized into bytes so that they can be stored
+;; in memory.
+
+function %return_4_b1s(b1, b1, b1, b1) -> b1, b1, b1, b1 {
+;; check: function %return_4_b1s(b1 [%rsi], b1 [%rdx], b1 [%rcx], b1 [%r8], i64 sret [%rdi], i64 fp [%rbp]) -> i64 sret [%rax], i64 fp [%rbp] fast {
+
+ebb0(v0: b1, v1: b1, v2: b1, v3: b1):
+; check: ebb0(v0: b1 [%rsi], v1: b1 [%rdx], v2: b1 [%rcx], v3: b1 [%r8], v4: i64 [%rdi], v13: i64 [%rbp]):
+
+    return v0, v1, v2, v3
+    ; check:  v5 = bint.i8 v0
+    ; nextln: v9 = uextend.i32 v5
+    ; nextln: istore8 notrap aligned v9, v4
+    ; nextln: v6 = bint.i8 v1
+    ; nextln: v10 = uextend.i32 v6
+    ; nextln: istore8 notrap aligned v10, v4+1
+    ; nextln: v7 = bint.i8 v2
+    ; nextln: v11 = uextend.i32 v7
+    ; nextln: istore8 notrap aligned v11, v4+2
+    ; nextln: v8 = bint.i8 v3
+    ; nextln: v12 = uextend.i32 v8
+    ; nextln: istore8 notrap aligned v12, v4+3
+}
+
+function %call_4_b1s() {
+; check: function %call_4_b1s(i64 fp [%rbp], i64 csr [%rbx]) -> i64 fp [%rbp], i64 csr [%rbx] fast {
+; nextln:    ss0 = sret_slot 4, offset -28
+
+    fn0 = colocated %return_4_b1s(b1, b1, b1, b1) -> b1, b1, b1, b1
+    ; check: sig0 = (b1 [%rsi], b1 [%rdx], b1 [%rcx], b1 [%r8], i64 sret [%rdi]) -> i64 sret [%rax] fast
+
+ebb0:
+; check: ebb0(v26: i64 [%rbp], v27: i64 [%rbx]):
+
+    v0 = bconst.b1 true
+    v1 = bconst.b1 false
+    v2 = bconst.b1 true
+    v3 = bconst.b1 false
+
+    ; check: v8 = stack_addr.i64 ss0
+    v4, v5, v6, v7 = call fn0(v0, v1, v2, v3)
+    ; check:  v9 = call fn0(v0, v1, v2, v3, v8)
+    ; nextln: v22 = uload8.i32 notrap aligned v9
+    ; nextln: v10 = ireduce.i8 v22
+    ; nextln: v11 = raw_bitcast.b8 v10
+    ; nextln: v12 = breduce.b1 v11
+    ; nextln: v4 -> v12
+    ; nextln: v23 = uload8.i32 notrap aligned v9+1
+    ; nextln: v13 = ireduce.i8 v23
+    ; nextln: v14 = raw_bitcast.b8 v13
+    ; nextln: v15 = breduce.b1 v14
+    ; nextln: v5 -> v15
+    ; nextln: v24 = uload8.i32 notrap aligned v9+2
+    ; nextln: v16 = ireduce.i8 v24
+    ; nextln: v17 = raw_bitcast.b8 v16
+    ; nextln: v18 = breduce.b1 v17
+    ; nextln: v6 -> v18
+    ; nextln: v25 = uload8.i32 notrap aligned v9+3
+    ; nextln: v19 = ireduce.i8 v25
+    ; nextln: v20 = raw_bitcast.b8 v19
+    ; nextln: v21 = breduce.b1 v20
+    ; nextln: v7 -> v21
+
+    return
+}
diff --git a/cranelift/filetests/filetests/wasm/multi-val-call-indirect.clif b/cranelift/filetests/filetests/wasm/multi-val-call-indirect.clif
new file mode 100644
index 0000000000..b102d652cf
--- /dev/null
+++ b/cranelift/filetests/filetests/wasm/multi-val-call-indirect.clif
@@ -0,0 +1,26 @@
+test legalizer
+target x86_64 haswell
+
+;; Indirect calls with many returns.
+
+function %call_indirect_many_rets(i64) {
+    ; check: ss0 = sret_slot 32
+
+    sig0 = () -> i64, i64, i64, i64
+    ; check: sig0 = (i64 sret [%rdi]) -> i64 sret [%rax] fast
+
+ebb0(v0: i64):
+    v1, v2, v3, v4 = call_indirect sig0, v0()
+    ; check:  v5 = stack_addr.i64 ss0
+    ; nextln: v6 = call_indirect sig0, v0(v5)
+    ; nextln: v7 = load.i64 notrap aligned v6
+    ; nextln: v1 -> v7
+    ; nextln: v8 = load.i64 notrap aligned v6+8
+    ; nextln: v2 -> v8
+    ; nextln: v9 = load.i64 notrap aligned v6+16
+    ; nextln: v3 -> v9
+    ; nextln: v10 = load.i64 notrap aligned v6+24
+    ; nextln: v4 -> v10
+
+    return
+}
diff --git a/cranelift/filetests/filetests/wasm/multi-val-f32.clif b/cranelift/filetests/filetests/wasm/multi-val-f32.clif
new file mode 100644
index 0000000000..9f3d0047cd
--- /dev/null
+++ b/cranelift/filetests/filetests/wasm/multi-val-f32.clif
@@ -0,0 +1,44 @@
+test compile
+target x86_64 haswell
+
+;; Returning many f32s
+
+function %return_2_f32s() -> f32, f32 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = f32const 0x1.0
+    return v0, v1
+}
+
+function %return_3_f32s() -> f32, f32, f32 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = f32const 0x1.0
+    v2 = f32const 0x2.0
+    return v0, v1, v2
+}
+
+function %return_4_f32s() -> f32, f32, f32, f32 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = f32const 0x1.0
+    v2 = f32const 0x2.0
+    v3 = f32const 0x3.0
+    return v0, v1, v2, v3
+}
+
+;; Calling functions that return many f32s
+
+function %call() -> f32 {
+    fn0 = %a() -> f32, f32
+    fn1 = %b(f32, f32) -> f32, f32, f32
+    fn2 = %c(f32, f32, f32) -> f32, f32, f32, f32
+ebb0:
+    v0, v1 = call fn0()
+    v2, v3, v4 = call fn1(v0, v1)
+    v5, v6, v7, v8 = call fn2(v2, v3, v4)
+    v9 = fadd v5, v6
+    v10 = fadd v7, v8
+    v11 = fadd v9, v10
+    return v11
+}
diff --git a/cranelift/filetests/filetests/wasm/multi-val-f64.clif b/cranelift/filetests/filetests/wasm/multi-val-f64.clif
new file mode 100644
index 0000000000..aa7e263eba
--- /dev/null
+++ b/cranelift/filetests/filetests/wasm/multi-val-f64.clif
@@ -0,0 +1,44 @@
+test compile
+target x86_64 haswell
+
+;; Returning many f64s
+
+function %return_2_f64s() -> f64, f64 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = f64const 0x1.0
+    return v0, v1
+}
+
+function %return_3_f64s() -> f64, f64, f64 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = f64const 0x1.0
+    v2 = f64const 0x2.0
+    return v0, v1, v2
+}
+
+function %return_4_f64s() -> f64, f64, f64, f64 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = f64const 0x1.0
+    v2 = f64const 0x2.0
+    v3 = f64const 0x3.0
+    return v0, v1, v2, v3
+}
+
+;; Calling functions that return many f64s
+
+function %call() -> f64 {
+    fn0 = %a() -> f64, f64
+    fn1 = %b(f64, f64) -> f64, f64, f64
+    fn2 = %c(f64, f64, f64) -> f64, f64, f64, f64
+ebb0:
+    v0, v1 = call fn0()
+    v2, v3, v4 = call fn1(v0, v1)
+    v5, v6, v7, v8 = call fn2(v2, v3, v4)
+    v9 = fadd v5, v6
+    v10 = fadd v7, v8
+    v11 = fadd v9, v10
+    return v11
+}
diff --git a/cranelift/filetests/filetests/wasm/multi-val-i32.clif b/cranelift/filetests/filetests/wasm/multi-val-i32.clif
new file mode 100644
index 0000000000..924fcb4bc6
--- /dev/null
+++ b/cranelift/filetests/filetests/wasm/multi-val-i32.clif
@@ -0,0 +1,44 @@
+test compile
+target x86_64 haswell
+
+;; Returning many i32s
+
+function %return_2_i32s() -> i32, i32 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = iconst.i32 1
+    return v0, v1
+}
+
+function %return_3_i32s() -> i32, i32, i32 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = iconst.i32 1
+    v2 = iconst.i32 2
+    return v0, v1, v2
+}
+
+function %return_4_i32s() -> i32, i32, i32, i32 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = iconst.i32 1
+    v2 = iconst.i32 2
+    v3 = iconst.i32 3
+    return v0, v1, v2, v3
+}
+
+;; Calling functions that return many i32s
+
+function %call() -> i32 {
+    fn0 = %a() -> i32, i32
+    fn1 = %b(i32, i32) -> i32, i32, i32
+    fn2 = %c(i32, i32, i32) -> i32, i32, i32, i32
+ebb0:
+    v0, v1 = call fn0()
+    v2, v3, v4 = call fn1(v0, v1)
+    v5, v6, v7, v8 = call fn2(v2, v3, v4)
+    v9 = iadd v5, v6
+    v10 = iadd v7, v8
+    v11 = iadd v9, v10
+    return v11
+}
diff --git a/cranelift/filetests/filetests/wasm/multi-val-i64.clif b/cranelift/filetests/filetests/wasm/multi-val-i64.clif
new file mode 100644
index 0000000000..f5ab392693
--- /dev/null
+++ b/cranelift/filetests/filetests/wasm/multi-val-i64.clif
@@ -0,0 +1,44 @@
+test compile
+target x86_64 haswell
+
+;; Returning many i64s
+
+function %return_2_i64s() -> i64, i64 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = iconst.i64 1
+    return v0, v1
+}
+
+function %return_3_i64s() -> i64, i64, i64 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = iconst.i64 1
+    v2 = iconst.i64 2
+    return v0, v1, v2
+}
+
+function %return_4_i64s() -> i64, i64, i64, i64 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = iconst.i64 1
+    v2 = iconst.i64 2
+    v3 = iconst.i64 3
+    return v0, v1, v2, v3
+}
+
+;; Calling functions that return many i64s
+
+function %call() -> i64 {
+    fn0 = %a() -> i64, i64
+    fn1 = %b(i64, i64) -> i64, i64, i64
+    fn2 = %c(i64, i64, i64) -> i64, i64, i64, i64
+ebb0:
+    v0, v1 = call fn0()
+    v2, v3, v4 = call fn1(v0, v1)
+    v5, v6, v7, v8 = call fn2(v2, v3, v4)
+    v9 = iadd v5, v6
+    v10 = iadd v7, v8
+    v11 = iadd v9, v10
+    return v11
+}
diff --git a/cranelift/filetests/filetests/wasm/multi-val-mixed.clif b/cranelift/filetests/filetests/wasm/multi-val-mixed.clif
new file mode 100644
index 0000000000..db66d202ff
--- /dev/null
+++ b/cranelift/filetests/filetests/wasm/multi-val-mixed.clif
@@ -0,0 +1,2098 @@
+test compile
+target x86_64 haswell
+
+;; Returning many mixed values.
+;;
+;; This test was generated programmatically with this python script:
+;;
+;; ```
+;; from itertools import permutations
+;;
+;; def make_val(i, r):
+;;     val = None
+;;     op = None
+;;     if r == "f32":
+;;         val = "0x0.0"
+;;         op = "f32const"
+;;     elif r == "f64":
+;;         val = "0x0.0"
+;;         op = "f64const"
+;;     elif r == "i32":
+;;         val = "0"
+;;         op = "iconst.i32"
+;;     elif r == "i64":
+;;         val = "0"
+;;         op = "iconst.i64"
+;;     elif r == "b1":
+;;         val = "true"
+;;         op = "bconst.b1"
+;;     else:
+;;         raise Exception("bad r = " + str(r))
+;;     return "    v" + str(i) + " = " + op + " " + val
+;;
+;; def make_returner(results):
+;;     results = list(results)
+;;     head = "function %return_" + "_".join(results) + "() -> " + ", ".join(results) + " {\n"
+;;     ebb = "ebb0:\n"
+;;     vals = [make_val(i, r) for i, r in enumerate(results)]
+;;     ret = "    return " + ", ".join(("v" + str(i) for i in range(0, len(results))))
+;;     return head + ebb + "\n".join(vals) + "\n" + ret + "\n}\n"
+;;
+;; def make_caller(results):
+;;     results = list(results)
+;;     head = "function %call_" + "_".join(results) + "() {\n"
+;;     fn_decl = "    fn0 = %foo() -> " + ",".join(results) + "\n"
+;;     ebb = "ebb0:\n"
+;;     ret_vars = ["v" + str(i) for i, r in enumerate(results)]
+;;     call = "    " + ",".join(ret_vars) + " = call fn0()\n"
+;;     ret = "    return\n"
+;;     tail = "}\n"
+;;     return head + fn_decl + ebb + call + ret + tail
+;;
+;; for results in permutations(["i32", "i64", "f32", "f64", "b1"]):
+;;     print make_returner(results)
+;;     print make_caller(results)
+;; ```
+;;
+;; If you're modifying this test, it is likely easier to modify the script and
+;; regenerate the test.
+
+function %return_i32_i64_f32_f64_b1() -> i32, i64, f32, f64, b1 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = iconst.i64 0
+    v2 = f32const 0x0.0
+    v3 = f64const 0x0.0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_i64_f32_f64_b1() {
+    fn0 = %foo() -> i32,i64,f32,f64,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_i64_f32_b1_f64() -> i32, i64, f32, b1, f64 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = iconst.i64 0
+    v2 = f32const 0x0.0
+    v3 = bconst.b1 true
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_i64_f32_b1_f64() {
+    fn0 = %foo() -> i32,i64,f32,b1,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_i64_f64_f32_b1() -> i32, i64, f64, f32, b1 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = iconst.i64 0
+    v2 = f64const 0x0.0
+    v3 = f32const 0x0.0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_i64_f64_f32_b1() {
+    fn0 = %foo() -> i32,i64,f64,f32,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_i64_f64_b1_f32() -> i32, i64, f64, b1, f32 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = iconst.i64 0
+    v2 = f64const 0x0.0
+    v3 = bconst.b1 true
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_i64_f64_b1_f32() {
+    fn0 = %foo() -> i32,i64,f64,b1,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_i64_b1_f32_f64() -> i32, i64, b1, f32, f64 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = iconst.i64 0
+    v2 = bconst.b1 true
+    v3 = f32const 0x0.0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_i64_b1_f32_f64() {
+    fn0 = %foo() -> i32,i64,b1,f32,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_i64_b1_f64_f32() -> i32, i64, b1, f64, f32 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = iconst.i64 0
+    v2 = bconst.b1 true
+    v3 = f64const 0x0.0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_i64_b1_f64_f32() {
+    fn0 = %foo() -> i32,i64,b1,f64,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_f32_i64_f64_b1() -> i32, f32, i64, f64, b1 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = f32const 0x0.0
+    v2 = iconst.i64 0
+    v3 = f64const 0x0.0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_f32_i64_f64_b1() {
+    fn0 = %foo() -> i32,f32,i64,f64,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_f32_i64_b1_f64() -> i32, f32, i64, b1, f64 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = f32const 0x0.0
+    v2 = iconst.i64 0
+    v3 = bconst.b1 true
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_f32_i64_b1_f64() {
+    fn0 = %foo() -> i32,f32,i64,b1,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_f32_f64_i64_b1() -> i32, f32, f64, i64, b1 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = f32const 0x0.0
+    v2 = f64const 0x0.0
+    v3 = iconst.i64 0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_f32_f64_i64_b1() {
+    fn0 = %foo() -> i32,f32,f64,i64,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_f32_f64_b1_i64() -> i32, f32, f64, b1, i64 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = f32const 0x0.0
+    v2 = f64const 0x0.0
+    v3 = bconst.b1 true
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_f32_f64_b1_i64() {
+    fn0 = %foo() -> i32,f32,f64,b1,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_f32_b1_i64_f64() -> i32, f32, b1, i64, f64 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = f32const 0x0.0
+    v2 = bconst.b1 true
+    v3 = iconst.i64 0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_f32_b1_i64_f64() {
+    fn0 = %foo() -> i32,f32,b1,i64,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_f32_b1_f64_i64() -> i32, f32, b1, f64, i64 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = f32const 0x0.0
+    v2 = bconst.b1 true
+    v3 = f64const 0x0.0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_f32_b1_f64_i64() {
+    fn0 = %foo() -> i32,f32,b1,f64,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_f64_i64_f32_b1() -> i32, f64, i64, f32, b1 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = f64const 0x0.0
+    v2 = iconst.i64 0
+    v3 = f32const 0x0.0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_f64_i64_f32_b1() {
+    fn0 = %foo() -> i32,f64,i64,f32,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_f64_i64_b1_f32() -> i32, f64, i64, b1, f32 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = f64const 0x0.0
+    v2 = iconst.i64 0
+    v3 = bconst.b1 true
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_f64_i64_b1_f32() {
+    fn0 = %foo() -> i32,f64,i64,b1,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_f64_f32_i64_b1() -> i32, f64, f32, i64, b1 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = f64const 0x0.0
+    v2 = f32const 0x0.0
+    v3 = iconst.i64 0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_f64_f32_i64_b1() {
+    fn0 = %foo() -> i32,f64,f32,i64,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_f64_f32_b1_i64() -> i32, f64, f32, b1, i64 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = f64const 0x0.0
+    v2 = f32const 0x0.0
+    v3 = bconst.b1 true
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_f64_f32_b1_i64() {
+    fn0 = %foo() -> i32,f64,f32,b1,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_f64_b1_i64_f32() -> i32, f64, b1, i64, f32 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = f64const 0x0.0
+    v2 = bconst.b1 true
+    v3 = iconst.i64 0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_f64_b1_i64_f32() {
+    fn0 = %foo() -> i32,f64,b1,i64,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_f64_b1_f32_i64() -> i32, f64, b1, f32, i64 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = f64const 0x0.0
+    v2 = bconst.b1 true
+    v3 = f32const 0x0.0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_f64_b1_f32_i64() {
+    fn0 = %foo() -> i32,f64,b1,f32,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_b1_i64_f32_f64() -> i32, b1, i64, f32, f64 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = bconst.b1 true
+    v2 = iconst.i64 0
+    v3 = f32const 0x0.0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_b1_i64_f32_f64() {
+    fn0 = %foo() -> i32,b1,i64,f32,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_b1_i64_f64_f32() -> i32, b1, i64, f64, f32 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = bconst.b1 true
+    v2 = iconst.i64 0
+    v3 = f64const 0x0.0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_b1_i64_f64_f32() {
+    fn0 = %foo() -> i32,b1,i64,f64,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_b1_f32_i64_f64() -> i32, b1, f32, i64, f64 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = bconst.b1 true
+    v2 = f32const 0x0.0
+    v3 = iconst.i64 0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_b1_f32_i64_f64() {
+    fn0 = %foo() -> i32,b1,f32,i64,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_b1_f32_f64_i64() -> i32, b1, f32, f64, i64 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = bconst.b1 true
+    v2 = f32const 0x0.0
+    v3 = f64const 0x0.0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_b1_f32_f64_i64() {
+    fn0 = %foo() -> i32,b1,f32,f64,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_b1_f64_i64_f32() -> i32, b1, f64, i64, f32 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = bconst.b1 true
+    v2 = f64const 0x0.0
+    v3 = iconst.i64 0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_b1_f64_i64_f32() {
+    fn0 = %foo() -> i32,b1,f64,i64,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i32_b1_f64_f32_i64() -> i32, b1, f64, f32, i64 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = bconst.b1 true
+    v2 = f64const 0x0.0
+    v3 = f32const 0x0.0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i32_b1_f64_f32_i64() {
+    fn0 = %foo() -> i32,b1,f64,f32,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_i32_f32_f64_b1() -> i64, i32, f32, f64, b1 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = iconst.i32 0
+    v2 = f32const 0x0.0
+    v3 = f64const 0x0.0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_i32_f32_f64_b1() {
+    fn0 = %foo() -> i64,i32,f32,f64,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_i32_f32_b1_f64() -> i64, i32, f32, b1, f64 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = iconst.i32 0
+    v2 = f32const 0x0.0
+    v3 = bconst.b1 true
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_i32_f32_b1_f64() {
+    fn0 = %foo() -> i64,i32,f32,b1,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_i32_f64_f32_b1() -> i64, i32, f64, f32, b1 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = iconst.i32 0
+    v2 = f64const 0x0.0
+    v3 = f32const 0x0.0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_i32_f64_f32_b1() {
+    fn0 = %foo() -> i64,i32,f64,f32,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_i32_f64_b1_f32() -> i64, i32, f64, b1, f32 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = iconst.i32 0
+    v2 = f64const 0x0.0
+    v3 = bconst.b1 true
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_i32_f64_b1_f32() {
+    fn0 = %foo() -> i64,i32,f64,b1,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_i32_b1_f32_f64() -> i64, i32, b1, f32, f64 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = iconst.i32 0
+    v2 = bconst.b1 true
+    v3 = f32const 0x0.0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_i32_b1_f32_f64() {
+    fn0 = %foo() -> i64,i32,b1,f32,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_i32_b1_f64_f32() -> i64, i32, b1, f64, f32 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = iconst.i32 0
+    v2 = bconst.b1 true
+    v3 = f64const 0x0.0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_i32_b1_f64_f32() {
+    fn0 = %foo() -> i64,i32,b1,f64,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_f32_i32_f64_b1() -> i64, f32, i32, f64, b1 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = f32const 0x0.0
+    v2 = iconst.i32 0
+    v3 = f64const 0x0.0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_f32_i32_f64_b1() {
+    fn0 = %foo() -> i64,f32,i32,f64,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_f32_i32_b1_f64() -> i64, f32, i32, b1, f64 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = f32const 0x0.0
+    v2 = iconst.i32 0
+    v3 = bconst.b1 true
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_f32_i32_b1_f64() {
+    fn0 = %foo() -> i64,f32,i32,b1,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_f32_f64_i32_b1() -> i64, f32, f64, i32, b1 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = f32const 0x0.0
+    v2 = f64const 0x0.0
+    v3 = iconst.i32 0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_f32_f64_i32_b1() {
+    fn0 = %foo() -> i64,f32,f64,i32,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_f32_f64_b1_i32() -> i64, f32, f64, b1, i32 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = f32const 0x0.0
+    v2 = f64const 0x0.0
+    v3 = bconst.b1 true
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_f32_f64_b1_i32() {
+    fn0 = %foo() -> i64,f32,f64,b1,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_f32_b1_i32_f64() -> i64, f32, b1, i32, f64 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = f32const 0x0.0
+    v2 = bconst.b1 true
+    v3 = iconst.i32 0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_f32_b1_i32_f64() {
+    fn0 = %foo() -> i64,f32,b1,i32,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_f32_b1_f64_i32() -> i64, f32, b1, f64, i32 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = f32const 0x0.0
+    v2 = bconst.b1 true
+    v3 = f64const 0x0.0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_f32_b1_f64_i32() {
+    fn0 = %foo() -> i64,f32,b1,f64,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_f64_i32_f32_b1() -> i64, f64, i32, f32, b1 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = f64const 0x0.0
+    v2 = iconst.i32 0
+    v3 = f32const 0x0.0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_f64_i32_f32_b1() {
+    fn0 = %foo() -> i64,f64,i32,f32,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_f64_i32_b1_f32() -> i64, f64, i32, b1, f32 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = f64const 0x0.0
+    v2 = iconst.i32 0
+    v3 = bconst.b1 true
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_f64_i32_b1_f32() {
+    fn0 = %foo() -> i64,f64,i32,b1,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_f64_f32_i32_b1() -> i64, f64, f32, i32, b1 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = f64const 0x0.0
+    v2 = f32const 0x0.0
+    v3 = iconst.i32 0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_f64_f32_i32_b1() {
+    fn0 = %foo() -> i64,f64,f32,i32,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_f64_f32_b1_i32() -> i64, f64, f32, b1, i32 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = f64const 0x0.0
+    v2 = f32const 0x0.0
+    v3 = bconst.b1 true
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_f64_f32_b1_i32() {
+    fn0 = %foo() -> i64,f64,f32,b1,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_f64_b1_i32_f32() -> i64, f64, b1, i32, f32 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = f64const 0x0.0
+    v2 = bconst.b1 true
+    v3 = iconst.i32 0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_f64_b1_i32_f32() {
+    fn0 = %foo() -> i64,f64,b1,i32,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_f64_b1_f32_i32() -> i64, f64, b1, f32, i32 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = f64const 0x0.0
+    v2 = bconst.b1 true
+    v3 = f32const 0x0.0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_f64_b1_f32_i32() {
+    fn0 = %foo() -> i64,f64,b1,f32,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_b1_i32_f32_f64() -> i64, b1, i32, f32, f64 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = bconst.b1 true
+    v2 = iconst.i32 0
+    v3 = f32const 0x0.0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_b1_i32_f32_f64() {
+    fn0 = %foo() -> i64,b1,i32,f32,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_b1_i32_f64_f32() -> i64, b1, i32, f64, f32 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = bconst.b1 true
+    v2 = iconst.i32 0
+    v3 = f64const 0x0.0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_b1_i32_f64_f32() {
+    fn0 = %foo() -> i64,b1,i32,f64,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_b1_f32_i32_f64() -> i64, b1, f32, i32, f64 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = bconst.b1 true
+    v2 = f32const 0x0.0
+    v3 = iconst.i32 0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_b1_f32_i32_f64() {
+    fn0 = %foo() -> i64,b1,f32,i32,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_b1_f32_f64_i32() -> i64, b1, f32, f64, i32 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = bconst.b1 true
+    v2 = f32const 0x0.0
+    v3 = f64const 0x0.0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_b1_f32_f64_i32() {
+    fn0 = %foo() -> i64,b1,f32,f64,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_b1_f64_i32_f32() -> i64, b1, f64, i32, f32 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = bconst.b1 true
+    v2 = f64const 0x0.0
+    v3 = iconst.i32 0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_b1_f64_i32_f32() {
+    fn0 = %foo() -> i64,b1,f64,i32,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_i64_b1_f64_f32_i32() -> i64, b1, f64, f32, i32 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = bconst.b1 true
+    v2 = f64const 0x0.0
+    v3 = f32const 0x0.0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_i64_b1_f64_f32_i32() {
+    fn0 = %foo() -> i64,b1,f64,f32,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_i32_i64_f64_b1() -> f32, i32, i64, f64, b1 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = iconst.i32 0
+    v2 = iconst.i64 0
+    v3 = f64const 0x0.0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_i32_i64_f64_b1() {
+    fn0 = %foo() -> f32,i32,i64,f64,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_i32_i64_b1_f64() -> f32, i32, i64, b1, f64 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = iconst.i32 0
+    v2 = iconst.i64 0
+    v3 = bconst.b1 true
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_i32_i64_b1_f64() {
+    fn0 = %foo() -> f32,i32,i64,b1,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_i32_f64_i64_b1() -> f32, i32, f64, i64, b1 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = iconst.i32 0
+    v2 = f64const 0x0.0
+    v3 = iconst.i64 0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_i32_f64_i64_b1() {
+    fn0 = %foo() -> f32,i32,f64,i64,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_i32_f64_b1_i64() -> f32, i32, f64, b1, i64 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = iconst.i32 0
+    v2 = f64const 0x0.0
+    v3 = bconst.b1 true
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_i32_f64_b1_i64() {
+    fn0 = %foo() -> f32,i32,f64,b1,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_i32_b1_i64_f64() -> f32, i32, b1, i64, f64 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = iconst.i32 0
+    v2 = bconst.b1 true
+    v3 = iconst.i64 0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_i32_b1_i64_f64() {
+    fn0 = %foo() -> f32,i32,b1,i64,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_i32_b1_f64_i64() -> f32, i32, b1, f64, i64 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = iconst.i32 0
+    v2 = bconst.b1 true
+    v3 = f64const 0x0.0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_i32_b1_f64_i64() {
+    fn0 = %foo() -> f32,i32,b1,f64,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_i64_i32_f64_b1() -> f32, i64, i32, f64, b1 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = iconst.i64 0
+    v2 = iconst.i32 0
+    v3 = f64const 0x0.0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_i64_i32_f64_b1() {
+    fn0 = %foo() -> f32,i64,i32,f64,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_i64_i32_b1_f64() -> f32, i64, i32, b1, f64 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = iconst.i64 0
+    v2 = iconst.i32 0
+    v3 = bconst.b1 true
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_i64_i32_b1_f64() {
+    fn0 = %foo() -> f32,i64,i32,b1,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_i64_f64_i32_b1() -> f32, i64, f64, i32, b1 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = iconst.i64 0
+    v2 = f64const 0x0.0
+    v3 = iconst.i32 0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_i64_f64_i32_b1() {
+    fn0 = %foo() -> f32,i64,f64,i32,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_i64_f64_b1_i32() -> f32, i64, f64, b1, i32 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = iconst.i64 0
+    v2 = f64const 0x0.0
+    v3 = bconst.b1 true
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_i64_f64_b1_i32() {
+    fn0 = %foo() -> f32,i64,f64,b1,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_i64_b1_i32_f64() -> f32, i64, b1, i32, f64 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = iconst.i64 0
+    v2 = bconst.b1 true
+    v3 = iconst.i32 0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_i64_b1_i32_f64() {
+    fn0 = %foo() -> f32,i64,b1,i32,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_i64_b1_f64_i32() -> f32, i64, b1, f64, i32 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = iconst.i64 0
+    v2 = bconst.b1 true
+    v3 = f64const 0x0.0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_i64_b1_f64_i32() {
+    fn0 = %foo() -> f32,i64,b1,f64,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_f64_i32_i64_b1() -> f32, f64, i32, i64, b1 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = f64const 0x0.0
+    v2 = iconst.i32 0
+    v3 = iconst.i64 0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_f64_i32_i64_b1() {
+    fn0 = %foo() -> f32,f64,i32,i64,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_f64_i32_b1_i64() -> f32, f64, i32, b1, i64 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = f64const 0x0.0
+    v2 = iconst.i32 0
+    v3 = bconst.b1 true
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_f64_i32_b1_i64() {
+    fn0 = %foo() -> f32,f64,i32,b1,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_f64_i64_i32_b1() -> f32, f64, i64, i32, b1 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = f64const 0x0.0
+    v2 = iconst.i64 0
+    v3 = iconst.i32 0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_f64_i64_i32_b1() {
+    fn0 = %foo() -> f32,f64,i64,i32,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_f64_i64_b1_i32() -> f32, f64, i64, b1, i32 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = f64const 0x0.0
+    v2 = iconst.i64 0
+    v3 = bconst.b1 true
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_f64_i64_b1_i32() {
+    fn0 = %foo() -> f32,f64,i64,b1,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_f64_b1_i32_i64() -> f32, f64, b1, i32, i64 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = f64const 0x0.0
+    v2 = bconst.b1 true
+    v3 = iconst.i32 0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_f64_b1_i32_i64() {
+    fn0 = %foo() -> f32,f64,b1,i32,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_f64_b1_i64_i32() -> f32, f64, b1, i64, i32 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = f64const 0x0.0
+    v2 = bconst.b1 true
+    v3 = iconst.i64 0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_f64_b1_i64_i32() {
+    fn0 = %foo() -> f32,f64,b1,i64,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_b1_i32_i64_f64() -> f32, b1, i32, i64, f64 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = bconst.b1 true
+    v2 = iconst.i32 0
+    v3 = iconst.i64 0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_b1_i32_i64_f64() {
+    fn0 = %foo() -> f32,b1,i32,i64,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_b1_i32_f64_i64() -> f32, b1, i32, f64, i64 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = bconst.b1 true
+    v2 = iconst.i32 0
+    v3 = f64const 0x0.0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_b1_i32_f64_i64() {
+    fn0 = %foo() -> f32,b1,i32,f64,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_b1_i64_i32_f64() -> f32, b1, i64, i32, f64 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = bconst.b1 true
+    v2 = iconst.i64 0
+    v3 = iconst.i32 0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_b1_i64_i32_f64() {
+    fn0 = %foo() -> f32,b1,i64,i32,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_b1_i64_f64_i32() -> f32, b1, i64, f64, i32 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = bconst.b1 true
+    v2 = iconst.i64 0
+    v3 = f64const 0x0.0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_b1_i64_f64_i32() {
+    fn0 = %foo() -> f32,b1,i64,f64,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_b1_f64_i32_i64() -> f32, b1, f64, i32, i64 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = bconst.b1 true
+    v2 = f64const 0x0.0
+    v3 = iconst.i32 0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_b1_f64_i32_i64() {
+    fn0 = %foo() -> f32,b1,f64,i32,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f32_b1_f64_i64_i32() -> f32, b1, f64, i64, i32 {
+ebb0:
+    v0 = f32const 0x0.0
+    v1 = bconst.b1 true
+    v2 = f64const 0x0.0
+    v3 = iconst.i64 0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f32_b1_f64_i64_i32() {
+    fn0 = %foo() -> f32,b1,f64,i64,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_i32_i64_f32_b1() -> f64, i32, i64, f32, b1 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = iconst.i32 0
+    v2 = iconst.i64 0
+    v3 = f32const 0x0.0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_i32_i64_f32_b1() {
+    fn0 = %foo() -> f64,i32,i64,f32,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_i32_i64_b1_f32() -> f64, i32, i64, b1, f32 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = iconst.i32 0
+    v2 = iconst.i64 0
+    v3 = bconst.b1 true
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_i32_i64_b1_f32() {
+    fn0 = %foo() -> f64,i32,i64,b1,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_i32_f32_i64_b1() -> f64, i32, f32, i64, b1 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = iconst.i32 0
+    v2 = f32const 0x0.0
+    v3 = iconst.i64 0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_i32_f32_i64_b1() {
+    fn0 = %foo() -> f64,i32,f32,i64,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_i32_f32_b1_i64() -> f64, i32, f32, b1, i64 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = iconst.i32 0
+    v2 = f32const 0x0.0
+    v3 = bconst.b1 true
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_i32_f32_b1_i64() {
+    fn0 = %foo() -> f64,i32,f32,b1,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_i32_b1_i64_f32() -> f64, i32, b1, i64, f32 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = iconst.i32 0
+    v2 = bconst.b1 true
+    v3 = iconst.i64 0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_i32_b1_i64_f32() {
+    fn0 = %foo() -> f64,i32,b1,i64,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_i32_b1_f32_i64() -> f64, i32, b1, f32, i64 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = iconst.i32 0
+    v2 = bconst.b1 true
+    v3 = f32const 0x0.0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_i32_b1_f32_i64() {
+    fn0 = %foo() -> f64,i32,b1,f32,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_i64_i32_f32_b1() -> f64, i64, i32, f32, b1 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = iconst.i64 0
+    v2 = iconst.i32 0
+    v3 = f32const 0x0.0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_i64_i32_f32_b1() {
+    fn0 = %foo() -> f64,i64,i32,f32,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_i64_i32_b1_f32() -> f64, i64, i32, b1, f32 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = iconst.i64 0
+    v2 = iconst.i32 0
+    v3 = bconst.b1 true
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_i64_i32_b1_f32() {
+    fn0 = %foo() -> f64,i64,i32,b1,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_i64_f32_i32_b1() -> f64, i64, f32, i32, b1 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = iconst.i64 0
+    v2 = f32const 0x0.0
+    v3 = iconst.i32 0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_i64_f32_i32_b1() {
+    fn0 = %foo() -> f64,i64,f32,i32,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_i64_f32_b1_i32() -> f64, i64, f32, b1, i32 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = iconst.i64 0
+    v2 = f32const 0x0.0
+    v3 = bconst.b1 true
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_i64_f32_b1_i32() {
+    fn0 = %foo() -> f64,i64,f32,b1,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_i64_b1_i32_f32() -> f64, i64, b1, i32, f32 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = iconst.i64 0
+    v2 = bconst.b1 true
+    v3 = iconst.i32 0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_i64_b1_i32_f32() {
+    fn0 = %foo() -> f64,i64,b1,i32,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_i64_b1_f32_i32() -> f64, i64, b1, f32, i32 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = iconst.i64 0
+    v2 = bconst.b1 true
+    v3 = f32const 0x0.0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_i64_b1_f32_i32() {
+    fn0 = %foo() -> f64,i64,b1,f32,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_f32_i32_i64_b1() -> f64, f32, i32, i64, b1 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = f32const 0x0.0
+    v2 = iconst.i32 0
+    v3 = iconst.i64 0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_f32_i32_i64_b1() {
+    fn0 = %foo() -> f64,f32,i32,i64,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_f32_i32_b1_i64() -> f64, f32, i32, b1, i64 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = f32const 0x0.0
+    v2 = iconst.i32 0
+    v3 = bconst.b1 true
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_f32_i32_b1_i64() {
+    fn0 = %foo() -> f64,f32,i32,b1,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_f32_i64_i32_b1() -> f64, f32, i64, i32, b1 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = f32const 0x0.0
+    v2 = iconst.i64 0
+    v3 = iconst.i32 0
+    v4 = bconst.b1 true
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_f32_i64_i32_b1() {
+    fn0 = %foo() -> f64,f32,i64,i32,b1
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_f32_i64_b1_i32() -> f64, f32, i64, b1, i32 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = f32const 0x0.0
+    v2 = iconst.i64 0
+    v3 = bconst.b1 true
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_f32_i64_b1_i32() {
+    fn0 = %foo() -> f64,f32,i64,b1,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_f32_b1_i32_i64() -> f64, f32, b1, i32, i64 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = f32const 0x0.0
+    v2 = bconst.b1 true
+    v3 = iconst.i32 0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_f32_b1_i32_i64() {
+    fn0 = %foo() -> f64,f32,b1,i32,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_f32_b1_i64_i32() -> f64, f32, b1, i64, i32 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = f32const 0x0.0
+    v2 = bconst.b1 true
+    v3 = iconst.i64 0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_f32_b1_i64_i32() {
+    fn0 = %foo() -> f64,f32,b1,i64,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_b1_i32_i64_f32() -> f64, b1, i32, i64, f32 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = bconst.b1 true
+    v2 = iconst.i32 0
+    v3 = iconst.i64 0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_b1_i32_i64_f32() {
+    fn0 = %foo() -> f64,b1,i32,i64,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_b1_i32_f32_i64() -> f64, b1, i32, f32, i64 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = bconst.b1 true
+    v2 = iconst.i32 0
+    v3 = f32const 0x0.0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_b1_i32_f32_i64() {
+    fn0 = %foo() -> f64,b1,i32,f32,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_b1_i64_i32_f32() -> f64, b1, i64, i32, f32 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = bconst.b1 true
+    v2 = iconst.i64 0
+    v3 = iconst.i32 0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_b1_i64_i32_f32() {
+    fn0 = %foo() -> f64,b1,i64,i32,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_b1_i64_f32_i32() -> f64, b1, i64, f32, i32 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = bconst.b1 true
+    v2 = iconst.i64 0
+    v3 = f32const 0x0.0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_b1_i64_f32_i32() {
+    fn0 = %foo() -> f64,b1,i64,f32,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_b1_f32_i32_i64() -> f64, b1, f32, i32, i64 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = bconst.b1 true
+    v2 = f32const 0x0.0
+    v3 = iconst.i32 0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_b1_f32_i32_i64() {
+    fn0 = %foo() -> f64,b1,f32,i32,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_f64_b1_f32_i64_i32() -> f64, b1, f32, i64, i32 {
+ebb0:
+    v0 = f64const 0x0.0
+    v1 = bconst.b1 true
+    v2 = f32const 0x0.0
+    v3 = iconst.i64 0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_f64_b1_f32_i64_i32() {
+    fn0 = %foo() -> f64,b1,f32,i64,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_i32_i64_f32_f64() -> b1, i32, i64, f32, f64 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = iconst.i32 0
+    v2 = iconst.i64 0
+    v3 = f32const 0x0.0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_i32_i64_f32_f64() {
+    fn0 = %foo() -> b1,i32,i64,f32,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_i32_i64_f64_f32() -> b1, i32, i64, f64, f32 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = iconst.i32 0
+    v2 = iconst.i64 0
+    v3 = f64const 0x0.0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_i32_i64_f64_f32() {
+    fn0 = %foo() -> b1,i32,i64,f64,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_i32_f32_i64_f64() -> b1, i32, f32, i64, f64 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = iconst.i32 0
+    v2 = f32const 0x0.0
+    v3 = iconst.i64 0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_i32_f32_i64_f64() {
+    fn0 = %foo() -> b1,i32,f32,i64,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_i32_f32_f64_i64() -> b1, i32, f32, f64, i64 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = iconst.i32 0
+    v2 = f32const 0x0.0
+    v3 = f64const 0x0.0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_i32_f32_f64_i64() {
+    fn0 = %foo() -> b1,i32,f32,f64,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_i32_f64_i64_f32() -> b1, i32, f64, i64, f32 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = iconst.i32 0
+    v2 = f64const 0x0.0
+    v3 = iconst.i64 0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_i32_f64_i64_f32() {
+    fn0 = %foo() -> b1,i32,f64,i64,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_i32_f64_f32_i64() -> b1, i32, f64, f32, i64 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = iconst.i32 0
+    v2 = f64const 0x0.0
+    v3 = f32const 0x0.0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_i32_f64_f32_i64() {
+    fn0 = %foo() -> b1,i32,f64,f32,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_i64_i32_f32_f64() -> b1, i64, i32, f32, f64 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = iconst.i64 0
+    v2 = iconst.i32 0
+    v3 = f32const 0x0.0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_i64_i32_f32_f64() {
+    fn0 = %foo() -> b1,i64,i32,f32,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_i64_i32_f64_f32() -> b1, i64, i32, f64, f32 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = iconst.i64 0
+    v2 = iconst.i32 0
+    v3 = f64const 0x0.0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_i64_i32_f64_f32() {
+    fn0 = %foo() -> b1,i64,i32,f64,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_i64_f32_i32_f64() -> b1, i64, f32, i32, f64 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = iconst.i64 0
+    v2 = f32const 0x0.0
+    v3 = iconst.i32 0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_i64_f32_i32_f64() {
+    fn0 = %foo() -> b1,i64,f32,i32,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_i64_f32_f64_i32() -> b1, i64, f32, f64, i32 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = iconst.i64 0
+    v2 = f32const 0x0.0
+    v3 = f64const 0x0.0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_i64_f32_f64_i32() {
+    fn0 = %foo() -> b1,i64,f32,f64,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_i64_f64_i32_f32() -> b1, i64, f64, i32, f32 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = iconst.i64 0
+    v2 = f64const 0x0.0
+    v3 = iconst.i32 0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_i64_f64_i32_f32() {
+    fn0 = %foo() -> b1,i64,f64,i32,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_i64_f64_f32_i32() -> b1, i64, f64, f32, i32 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = iconst.i64 0
+    v2 = f64const 0x0.0
+    v3 = f32const 0x0.0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_i64_f64_f32_i32() {
+    fn0 = %foo() -> b1,i64,f64,f32,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_f32_i32_i64_f64() -> b1, f32, i32, i64, f64 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = f32const 0x0.0
+    v2 = iconst.i32 0
+    v3 = iconst.i64 0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_f32_i32_i64_f64() {
+    fn0 = %foo() -> b1,f32,i32,i64,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_f32_i32_f64_i64() -> b1, f32, i32, f64, i64 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = f32const 0x0.0
+    v2 = iconst.i32 0
+    v3 = f64const 0x0.0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_f32_i32_f64_i64() {
+    fn0 = %foo() -> b1,f32,i32,f64,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_f32_i64_i32_f64() -> b1, f32, i64, i32, f64 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = f32const 0x0.0
+    v2 = iconst.i64 0
+    v3 = iconst.i32 0
+    v4 = f64const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_f32_i64_i32_f64() {
+    fn0 = %foo() -> b1,f32,i64,i32,f64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_f32_i64_f64_i32() -> b1, f32, i64, f64, i32 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = f32const 0x0.0
+    v2 = iconst.i64 0
+    v3 = f64const 0x0.0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_f32_i64_f64_i32() {
+    fn0 = %foo() -> b1,f32,i64,f64,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_f32_f64_i32_i64() -> b1, f32, f64, i32, i64 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = f32const 0x0.0
+    v2 = f64const 0x0.0
+    v3 = iconst.i32 0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_f32_f64_i32_i64() {
+    fn0 = %foo() -> b1,f32,f64,i32,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_f32_f64_i64_i32() -> b1, f32, f64, i64, i32 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = f32const 0x0.0
+    v2 = f64const 0x0.0
+    v3 = iconst.i64 0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_f32_f64_i64_i32() {
+    fn0 = %foo() -> b1,f32,f64,i64,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_f64_i32_i64_f32() -> b1, f64, i32, i64, f32 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = f64const 0x0.0
+    v2 = iconst.i32 0
+    v3 = iconst.i64 0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_f64_i32_i64_f32() {
+    fn0 = %foo() -> b1,f64,i32,i64,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_f64_i32_f32_i64() -> b1, f64, i32, f32, i64 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = f64const 0x0.0
+    v2 = iconst.i32 0
+    v3 = f32const 0x0.0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_f64_i32_f32_i64() {
+    fn0 = %foo() -> b1,f64,i32,f32,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_f64_i64_i32_f32() -> b1, f64, i64, i32, f32 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = f64const 0x0.0
+    v2 = iconst.i64 0
+    v3 = iconst.i32 0
+    v4 = f32const 0x0.0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_f64_i64_i32_f32() {
+    fn0 = %foo() -> b1,f64,i64,i32,f32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_f64_i64_f32_i32() -> b1, f64, i64, f32, i32 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = f64const 0x0.0
+    v2 = iconst.i64 0
+    v3 = f32const 0x0.0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_f64_i64_f32_i32() {
+    fn0 = %foo() -> b1,f64,i64,f32,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_f64_f32_i32_i64() -> b1, f64, f32, i32, i64 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = f64const 0x0.0
+    v2 = f32const 0x0.0
+    v3 = iconst.i32 0
+    v4 = iconst.i64 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_f64_f32_i32_i64() {
+    fn0 = %foo() -> b1,f64,f32,i32,i64
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
+
+function %return_b1_f64_f32_i64_i32() -> b1, f64, f32, i64, i32 {
+ebb0:
+    v0 = bconst.b1 true
+    v1 = f64const 0x0.0
+    v2 = f32const 0x0.0
+    v3 = iconst.i64 0
+    v4 = iconst.i32 0
+    return v0, v1, v2, v3, v4
+}
+
+function %call_b1_f64_f32_i64_i32() {
+    fn0 = %foo() -> b1,f64,f32,i64,i32
+ebb0:
+    v0,v1,v2,v3,v4 = call fn0()
+    return
+}
diff --git a/cranelift/filetests/filetests/wasm/multi-val-reuse-ret-ptr-stack-slot.clif b/cranelift/filetests/filetests/wasm/multi-val-reuse-ret-ptr-stack-slot.clif
new file mode 100644
index 0000000000..f7d0bf846c
--- /dev/null
+++ b/cranelift/filetests/filetests/wasm/multi-val-reuse-ret-ptr-stack-slot.clif
@@ -0,0 +1,61 @@
+test legalizer
+target x86_64 haswell
+
+;; Test that we don't reuse `sret` stack slots for multiple calls. We could do
+;; this one day, but it would require some care to ensure that we don't have
+;; subsequent calls overwrite the results of previous calls.
+
+function %foo() -> i32, f32 {
+    ; check:  ss0 = sret_slot 20
+    ; nextln: ss1 = sret_slot 20
+
+    fn0 = %f() -> i32, i32, i32, i32, i32
+    fn1 = %g() -> f32, f32, f32, f32, f32
+    ; check:  sig0 = (i64 sret [%rdi]) -> i64 sret [%rax] fast
+    ; nextln: sig1 = (i64 sret [%rdi]) -> i64 sret [%rax] fast
+    ; nextln: fn0 = %f sig0
+    ; nextln: fn1 = %g sig1
+
+ebb0:
+    v0, v1, v2, v3, v4 = call fn0()
+    ; check:  v18 = stack_addr.i64 ss0
+    ; nextln: v25 = func_addr.i64 fn0
+    ; nextln: v19 = call_indirect sig0, v25(v18)
+    ; nextln: v20 = load.i32 notrap aligned v19
+    ; nextln: v0 -> v20
+    ; nextln: v21 = load.i32 notrap aligned v19+4
+    ; nextln: v1 -> v21
+    ; nextln: v22 = load.i32 notrap aligned v19+8
+    ; nextln: v2 -> v22
+    ; nextln: v23 = load.i32 notrap aligned v19+12
+    ; nextln: v3 -> v23
+    ; nextln: v24 = load.i32 notrap aligned v19+16
+    ; nextln: v4 -> v24
+
+    v5, v6, v7, v8, v9 = call fn1()
+    ; check:  v26 = stack_addr.i64 ss1
+    ; nextln: v33 = func_addr.i64 fn1
+    ; nextln: v27 = call_indirect sig1, v33(v26)
+    ; nextln: v28 = load.f32 notrap aligned v27
+    ; nextln: v5 -> v28
+    ; nextln: v29 = load.f32 notrap aligned v27+4
+    ; nextln: v6 -> v29
+    ; nextln: v30 = load.f32 notrap aligned v27+8
+    ; nextln: v7 -> v30
+    ; nextln: v31 = load.f32 notrap aligned v27+12
+    ; nextln: v8 -> v31
+    ; nextln: v32 = load.f32 notrap aligned v27+16
+    ; nextln: v9 -> v32
+
+    v10 = iadd v0, v1
+    v11 = iadd v2, v3
+    v12 = iadd v10, v11
+    v13 = iadd v12, v4
+
+    v14 = fadd v5, v6
+    v15 = fadd v7, v8
+    v16 = fadd v14, v15
+    v17 = fadd v16, v9
+
+    return v13, v17
+}
diff --git a/cranelift/filetests/filetests/wasm/multi-val-sret-slot-alignment.clif b/cranelift/filetests/filetests/wasm/multi-val-sret-slot-alignment.clif
new file mode 100644
index 0000000000..b6c74e314e
--- /dev/null
+++ b/cranelift/filetests/filetests/wasm/multi-val-sret-slot-alignment.clif
@@ -0,0 +1,51 @@
+test legalizer
+target x86_64 haswell
+
+;; Need to insert padding after the `i8`s so that the `i32` and `i64` are
+;; aligned.
+
+function %returner() -> i8, i32, i8, i64 {
+; check: function %returner(i64 sret [%rdi]) -> i64 sret [%rax] fast {
+
+ebb0:
+; check: ebb0(v4: i64):
+
+    v0 = iconst.i8 0
+    v1 = iconst.i32 1
+    v2 = iconst.i8 2
+    v3 = iconst.i64 3
+    return v0, v1, v2, v3
+    ; check:  v6 = uextend.i32 v0
+    ; nextln: istore8 notrap aligned v6, v4
+    ; nextln: store notrap aligned v1, v4+4
+    ; nextln: v7 = uextend.i32 v2
+    ; nextln: istore8 notrap aligned v7, v4+8
+    ; nextln: store notrap aligned v3, v4+16
+    ; nextln: return v4
+}
+
+function %caller() {
+    ; check:  ss0 = sret_slot 24
+
+    fn0 = %returner() -> i8, i32, i8, i64
+    ; check:  sig0 = (i64 sret [%rdi]) -> i64 sret [%rax] fast
+    ; nextln: fn0 = %returner sig0
+
+ebb0:
+    v0, v1, v2, v3 = call fn0()
+    ; check:  v4 = stack_addr.i64 ss0
+    ; nextln: v10 = func_addr.i64 fn0
+    ; nextln: v5 = call_indirect sig0, v10(v4)
+    ; nextln: v11 = uload8.i32 notrap aligned v5
+    ; nextln: v6 = ireduce.i8 v11
+    ; nextln: v0 -> v6
+    ; nextln: v7 = load.i32 notrap aligned v5+4
+    ; nextln: v1 -> v7
+    ; nextln: v12 = uload8.i32 notrap aligned v5+8
+    ; nextln: v8 = ireduce.i8 v12
+    ; nextln: v2 -> v8
+    ; nextln: v9 = load.i64 notrap aligned v5+16
+    ; nextln: v3 -> v9
+
+    return
+}
diff --git a/cranelift/filetests/filetests/wasm/multi-val-take-many-and-return-many.clif b/cranelift/filetests/filetests/wasm/multi-val-take-many-and-return-many.clif
new file mode 100644
index 0000000000..385cc9d27c
--- /dev/null
+++ b/cranelift/filetests/filetests/wasm/multi-val-take-many-and-return-many.clif
@@ -0,0 +1,18 @@
+test compile
+target x86_64 haswell
+
+function %returner(i32, i64, f32, f64) -> i32, i64, f32, f64 {
+ebb0(v0: i32, v1: i64, v2: f32, v3: f64):
+    return v0, v1, v2, v3
+}
+
+function %caller() {
+    fn0 = %returner(i32, i64, f32, f64) -> i32, i64, f32, f64
+ebb0:
+    v0 = iconst.i32 0
+    v1 = iconst.i64 1
+    v2 = f32const 0x2.0
+    v3 = f64const 0x3.0
+    v4, v5, v6, v7 = call fn0(v0, v1, v2, v3)
+    return
+}
diff --git a/cranelift/filetests/filetests/wasm/multi-val-tons-of-results.clif b/cranelift/filetests/filetests/wasm/multi-val-tons-of-results.clif
new file mode 100644
index 0000000000..f19b1bcbf0
--- /dev/null
+++ b/cranelift/filetests/filetests/wasm/multi-val-tons-of-results.clif
@@ -0,0 +1,34 @@
+test compile
+target x86_64 haswell
+
+function %return_20_i32s() -> i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 {
+ebb0:
+    v0 = iconst.i32 0
+    v1 = iconst.i32 1
+    v2 = iconst.i32 2
+    v3 = iconst.i32 3
+    v4 = iconst.i32 4
+    v5 = iconst.i32 5
+    v6 = iconst.i32 6
+    v7 = iconst.i32 7
+    v8 = iconst.i32 8
+    v9 = iconst.i32 9
+    v10 = iconst.i32 10
+    v11 = iconst.i32 11
+    v12 = iconst.i32 12
+    v13 = iconst.i32 13
+    v14 = iconst.i32 14
+    v15 = iconst.i32 15
+    v16 = iconst.i32 16
+    v17 = iconst.i32 17
+    v18 = iconst.i32 18
+    v19 = iconst.i32 19
+    return v0, v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16, v17, v18, v19
+}
+
+function %call_20_i32s() {
+    fn0 = %return_20_i32s() -> i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32
+ebb0:
+    v0, v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16, v17, v18, v19 = call fn0()
+    return
+}

From f0e90df9ac145ca6e44bf35d251ed9c6a15b2bc0 Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 5 Nov 2019 10:48:09 -0800
Subject: [PATCH 2915/3084] Migrate CI to GitHub Actions

This commit migrates from Azure Pipelines to Github Actions for CI for
cranelift. The CI configuration was relatively straightforward, and the
intention here is not to change what's actually being done on CI, just
change where it's being done. The previous CI configuration had build
targets for producing releases, but these weren't actually applicable
for cranelift itself (mostly just copied from wasmtime), so they've been
folded into the main "test everything" matrix which now includes
`--release` mode items.

See cranestation/wasmtime#474 for some more context as well in terms of
benefits of Github Actions vs Azure Pipelines.
---
 .azure-pipelines.yml                 | 184 ---------------------------
 cranelift/.github/workflows/main.yml | 136 ++++++++++++++++++++
 cranelift/ci/azure-build-release.yml |  66 ----------
 cranelift/ci/azure-install-rust.yml  |  33 -----
 4 files changed, 136 insertions(+), 283 deletions(-)
 delete mode 100644 .azure-pipelines.yml
 create mode 100644 cranelift/.github/workflows/main.yml
 delete mode 100644 cranelift/ci/azure-build-release.yml
 delete mode 100644 cranelift/ci/azure-install-rust.yml

diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml
deleted file mode 100644
index 2fb328db32..0000000000
--- a/.azure-pipelines.yml
+++ /dev/null
@@ -1,184 +0,0 @@
-name: $(Build.SourceBranch)-$(date:yyyy-MM-dd)$(rev:.r)
-trigger:
-  branches:
-    include:
-    - 'master'
-  tags:
-    include:
-    - '*'
-    exclude:
-      - 'dev'
-
-jobs:
-- job: rustfmt
-  pool:
-    vmImage: 'macos-10.14'
-  steps:
-  - checkout: self
-    submodules: true
-  - template: ci/azure-install-rust.yml
-  - script: rustup component add rustfmt
-    displayName: Add rustfmt
-  - script: cargo fmt --all -- --check
-    displayName: Check formatting
-  variables:
-    toolchain: stable
-
-# Smoke test to build docs on one builder, using OSX for now since it's the
-# fastest
-- job: docs
-  pool:
-    vmImage: 'macos-10.14'
-  steps:
-  - checkout: self
-    submodules: true
-  - template: ci/azure-install-rust.yml
-  - script: |
-      cargo doc --all --exclude cranelift-codegen-meta
-      cargo doc --package cranelift-codegen-meta --document-private-items
-    displayName: Build documentation
-  - script: cargo install cargo-deadlinks
-    displayName: Install cargo-deadlinks
-  - bash: |
-      find ./target/doc -maxdepth 1 -type d -name "cranelift*" | xargs -I{} cargo deadlinks --dir {}
-    displayName: Run cargo-deadlinks
-  variables:
-    toolchain: nightly
-
-- job: Test
-  strategy:
-    matrix:
-      windows-earliest:
-        imageName: 'vs2017-win2016'
-        toolchain: '1.37.0'
-      linux-earliest:
-        imageName: 'ubuntu-16.04'
-        toolchain: '1.37.0'
-      mac-earliest:
-        imageName: 'macos-10.14'
-        toolchain: '1.37.0'
-      mac-stable:
-        imageName: 'macos-10.14'
-        toolchain: stable
-      mac-beta:
-        imageName: 'macos-10.14'
-        toolchain: beta
-      mac-nightly:
-        imageName: 'macos-10.14'
-        toolchain: nightly
-
-  pool:
-    vmImage: $(imageName)
-
-  steps:
-    - checkout: self
-      submodules: true
-    - template: ci/azure-install-rust.yml
-
-    - script: cargo fetch
-      displayName: Fetch cargo dependencies
-
-    - script: cargo build
-      displayName: Cargo build
-
-    - bash: cargo test --all
-      displayName: Cargo test
-      env:
-        RUST_BACKTRACE: 1
-
-    # Ensure fuzzer works by running it with a single input
-    - bash: cargo install cargo-fuzz
-      displayName: Install cargo-fuzz
-      condition: and(succeeded(), eq(variables['toolchain'], 'nightly'))
-
-    - bash: |
-        fuzz_module="ffaefab69523eb11935a9b420d58826c8ea65c4c"
-        cargo fuzz run fuzz_translate_module \
-        "fuzz/corpus/fuzz_translate_module/$fuzz_module"
-      displayName: Run cargo-fuzz
-      env:
-        RUST_BACKTRACE: 1
-      condition: and(succeeded(), eq(variables['toolchain'], 'nightly'))
-      continueOnError: true
-
-- job: Fuzz_regression
-  displayName: Fuzz regression
-  pool:
-    vmImage: "ubuntu-16.04"
-  variables:
-    toolchain: nightly
-  steps:
-    - template: ci/azure-install-rust.yml
-    - bash: cargo install cargo-fuzz
-    - bash: ci/fuzzit.sh local-regression
-
-- job: Fuzz
-  condition: ne(variables['Build.Reason'], 'PullRequest')
-  pool:
-    vmImage: "ubuntu-16.04"
-  variables:
-    toolchain: nightly
-  steps:
-    - template: ci/azure-install-rust.yml
-    - bash: cargo install cargo-fuzz
-    - bash: ci/fuzzit.sh fuzzing
-      env:
-        FUZZIT_API_KEY: $(FUZZIT_API_KEY)
-
-- job: Build
-  strategy:
-    matrix:
-      windows:
-        imageName: 'vs2017-win2016'
-        # Statically link against msvcrt to produce slightly more portable
-        # binaries on Windows by reducing our binary compatibility requirements.
-        RUSTFLAGS: -Ctarget-feature=+crt-static
-      mac:
-        imageName: 'macos-10.14'
-        # Lower the deployment target from our build image in an attempt to
-        # build more portable binaries that run on older releases. Note that
-        # 10.9 here is arbitrarily chosen and just happens to be the lowest that
-        # works at this time. Raising this is probably fine.
-        MACOSX_DEPLOYMENT_TARGET: 10.9
-  variables:
-    toolchain: '1.37.0'
-  pool:
-    vmImage: $(imageName)
-  # We try to be compatible with beta and nightly, but they occasionally
-  # fail, so we don't allow them to hold up people using stable.
-  continueOnError: $[ne(variables['toolchain'], 'stable')]
-  steps:
-  - template: ci/azure-build-release.yml
-
-# Build the Linux release binary in an older Linux container (in this case
-# Centos 6)
-- job: Build_linux
-  variables:
-    toolchain: '1.37.0'
-  container:
-    image: centos:6
-    options: "--name ci-container -v /usr/bin/docker:/tmp/docker:ro"
-  steps:
-  # We're executing in the container as non-root but `yum` requires root. We
-  # need to install `sudo` but to do that we need `sudo`. Do a bit of a weird
-  # hack where we use the host `docker` executable to re-execute in our own
-  # container with the root user to install `sudo`
-  - bash: /tmp/docker exec -t -u 0 ci-container sh -c "yum install -y sudo"
-    displayName: Configure sudo
-
-  # See https://edwards.sdsu.edu/research/c11-on-centos-6/ for where these
-  # various commands came from.
-  - bash: |
-      set -e
-      sudo yum install -y centos-release-scl cmake xz
-      sudo yum install -y devtoolset-8-gcc devtoolset-8-binutils devtoolset-8-gcc-c++
-      echo "##vso[task.prependpath]/opt/rh/devtoolset-8/root/usr/bin"
-    displayName: Install system dependencies
-
-  # Delete `libstdc++.so` to force gcc to link against `libstdc++.a` instead.
-  # This is a hack and not the right way to do this, but it ends up doing the
-  # right thing for now.
-  - bash: sudo rm -f /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/libstdc++.so
-    displayName: Force a static libstdc++
-
-  - template: ci/azure-build-release.yml
diff --git a/cranelift/.github/workflows/main.yml b/cranelift/.github/workflows/main.yml
new file mode 100644
index 0000000000..d39e690d6d
--- /dev/null
+++ b/cranelift/.github/workflows/main.yml
@@ -0,0 +1,136 @@
+name: CI
+on:
+  push:
+    branches: [master]
+  pull_request:
+    branches: [master]
+
+jobs:
+  rustfmt:
+    name: Rustfmt
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@master
+      with:
+        submodules: true
+    - name: Install Rust
+      run: rustup update stable && rustup default stable && rustup component add rustfmt
+    - run: cargo fmt -- --check
+
+  docs:
+    name: Build API Docs
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@master
+      with:
+        submodules: true
+    - name: Install Rust
+      run: rustup update nightly && rustup default nightly
+    - run: cargo doc --all --exclude cranelift-codegen-meta
+    - run: cargo doc --package cranelift-codegen-meta --document-private-items
+    - run: cargo install cargo-deadlinks
+    - run: find ./target/doc -maxdepth 1 -type d -name "cranelift*" | xargs -I{} cargo deadlinks --dir {}
+      name: Run cargo-deadlinks
+
+  test:
+    name: Test
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        # first a list of everything we test...
+        name:
+        - windows-earliest
+        - linux-earliest
+        - mac-earliest
+        - stable
+        - beta
+        - nightly
+        - windows-release
+        - linux-release
+        - mac-release
+        # .. and then the actual configuration for each thing we test
+        include:
+        - name: windows-earliest
+          os: ubuntu-latest
+          rust: 1.37.0
+        - name: linux-earliest
+          os: ubuntu-16.04
+          rust: 1.37.0
+        - name: mac-earliest
+          os: macos-10.14
+          rust: 1.37.0
+        - name: stable
+          os: macos-latest
+          rust: stable
+        - name: beta
+          os: macos-latest
+          rust: beta
+        - name: nightly
+          os: macos-latest
+          rust: nightly
+        - name: mac-release
+          os: macos-latest
+          rust: stable
+          release: --release
+        - name: linux-release
+          os: ubuntu-latest
+          rust: stable
+          release: --release
+        - name: windows-release
+          os: windows-latest
+          rust: stable
+          release: --release
+    steps:
+    - uses: actions/checkout@master
+      with:
+        submodules: true
+
+    - name: Install Rust (rustup)
+      run: rustup update ${{ matrix.rust }} --no-self-update && rustup default ${{ matrix.rust }}
+      if: "!startsWith(matrix.os, 'macos-')"
+      shell: bash
+    - name: Install Rust (macos)
+      run: |
+        curl https://sh.rustup.rs | sh -s -- -y --default-toolchain=${{ matrix.rust }}
+        echo ::add-path::$HOME/.cargo/bin
+      if: startsWith(matrix.os, 'macos-')
+
+    - run: cargo fetch
+    - run: cargo build ${{ matrix.release }}
+    - run: cargo test --all ${{ matrix.release }}
+      env:
+        RUST_BACKTRACE: 1
+
+    # Ensure fuzzer works by running it with a single input
+    - run: cargo install cargo-fuzz
+      if: matrix.rust == 'nightly'
+    - run: |
+        fuzz_module="ffaefab69523eb11935a9b420d58826c8ea65c4c"
+        cargo fuzz run fuzz_translate_module fuzz/corpus/fuzz_translate_module/$fuzz_module
+      env:
+        RUST_BACKTRACE: 1
+      if: matrix.rust == 'nightly'
+      continue-on-error: true
+
+  fuzz:
+    name: Fuzz Regression
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@master
+    - name: Install Rust
+      run: rustup update nightly && rustup default nightly
+    - run: cargo install cargo-fuzz
+    - run: ci/fuzzit.sh local-regression
+
+  fuzz_push:
+    name: Fuzz (push)
+    runs-on: ubuntu-latest
+    if: github.event_name == 'push'
+    steps:
+    - uses: actions/checkout@master
+    - name: Install Rust
+      run: rustup update nightly && rustup default nightly
+    - run: cargo install cargo-fuzz
+    - run: ci/fuzzit.sh fuzzing
+      env:
+        FUZZIT_API_KEY: ${{ secrets.FUZZIT_API_KEY }}
diff --git a/cranelift/ci/azure-build-release.yml b/cranelift/ci/azure-build-release.yml
deleted file mode 100644
index 6400eed0eb..0000000000
--- a/cranelift/ci/azure-build-release.yml
+++ /dev/null
@@ -1,66 +0,0 @@
-steps:
-- checkout: self
-  submodules: true
-
-- template: azure-install-rust.yml
-
-- bash: echo "##vso[task.setvariable variable=RUSTC_VERSION;]`rustc --version`"
-  displayName: Set rustc version string for caching
-
-- bash: cargo build --release
-  displayName: Cargo build
-
-# Test what we're about to release in release mode itself.
-- bash: cargo test --release --all
-  displayName: Cargo test
-  env:
-    RUST_BACKTRACE: 1
-
-- bash: |
-    echo "##vso[task.setvariable variable=tagName;]`echo $BUILD_SOURCEBRANCH | sed -e 's|refs/tags/||'`"
-  displayName: Set tag name
-  condition: startsWith(variables['Build.SourceBranch'], 'refs/tags/')
-- bash: |
-    echo "##vso[task.setvariable variable=tagName;]dev"
-  displayName: Set tag name to "dev"
-  condition: not(startsWith(variables['Build.SourceBranch'], 'refs/tags/'))
-
-- bash: echo "##vso[task.setvariable variable=basename;]cranelift-$(tagName)-x86_64-windows"
-  displayName: Configure basename var
-  condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
-- bash: echo "##vso[task.setvariable variable=basename;]cranelift-$(tagName)-x86_64-macos"
-  displayName: Configure basename var
-  condition: and(succeeded(), eq(variables['Agent.OS'], 'Darwin'))
-- bash: echo "##vso[task.setvariable variable=basename;]cranelift-$(tagName)-x86_64-linux"
-  displayName: Configure basename var
-  condition: and(succeeded(), eq( variables['Agent.OS'], 'Linux' ))
-
-- bash: |
-    set -e
-    mkdir -p $BUILD_BINARIESDIRECTORY/$BASENAME
-    if [ "$AGENT_OS" = "Windows_NT" ]; then
-      ext=.exe
-    fi
-    cp LICENSE README.md target/release/clif-util$ext $BUILD_BINARIESDIRECTORY/$BASENAME
-  displayName: Copy binaries
-
-- task: ArchiveFiles@2
-  inputs:
-    rootFolderOrFile: $(Build.BinariesDirectory)/$(basename)
-    archiveType: 'zip'
-    archiveFile: '$(Build.ArtifactStagingDirectory)/$(basename).zip'
-  displayName: Archive files (Win)
-  condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
-- task: ArchiveFiles@2
-  inputs:
-    rootFolderOrFile: $(Build.BinariesDirectory)/$(basename)
-    archiveType: 'tar'
-    tarCompression: 'xz'
-    archiveFile: '$(Build.ArtifactStagingDirectory)/$(basename).tar.xz'
-  displayName: Archive files (Unix)
-  condition: and(succeeded(), ne(variables['Agent.OS'], 'Windows_NT'))
-- task: PublishPipelineArtifact@1
-  inputs:
-    path: $(Build.ArtifactStagingDirectory)/
-    artifactName: 'bundle-$(Agent.OS)'
-
diff --git a/cranelift/ci/azure-install-rust.yml b/cranelift/ci/azure-install-rust.yml
deleted file mode 100644
index 64e502b775..0000000000
--- a/cranelift/ci/azure-install-rust.yml
+++ /dev/null
@@ -1,33 +0,0 @@
-steps:
-  # Rustup is currently installed on Windows and Linux, but not macOS.
-  # It is installed in /usr/local/cargo/bin/ or C:\Program Files\Rust\.cargo\bin\
-  # This steps ensures that rustup is installed, mainly for macOS, or if the
-  # azure image changes in the future.
-  - bash: |
-      set -ex
-      if [ -x "`command -v rustup`" ]; then
-        echo `command -v rustup` `rustup -V` already installed
-        rustup self update
-      else
-        if [ "$AGENT_OS" = "Windows_NT" ]; then
-          curl -sSf -o rustup-init.exe https://win.rustup.rs
-          ./rustup-init.exe -y --default-toolchain $TOOLCHAIN
-          echo "##vso[task.prependpath]$USERPROFILE/.cargo/bin"
-        else
-          curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain $TOOLCHAIN
-          echo "##vso[task.prependpath]$HOME/.cargo/bin"
-        fi
-      fi
-    displayName: Install rustup
-
-  - bash: |
-      set -ex
-      rustup update $TOOLCHAIN
-      rustup default $TOOLCHAIN
-    displayName: Install rust
-
-  - bash: |
-      set -ex
-      rustc -Vv
-      cargo -V
-    displayName: Query rust and cargo versions

From f8c3dfeb17e8a15b1a046802e7c7db53b83c3d28 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 23 Oct 2019 15:05:36 -0700
Subject: [PATCH 2916/3084] Translate WASM iTxN.eq to CLIF

---
 cranelift/wasm/src/code_translator.rs | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 6fda78be27..77d35373a4 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1164,8 +1164,13 @@ pub fn translate_operator(
             let bool_result = builder.ins().vall_true(state.pop1());
             state.push1(builder.ins().bint(I32, bool_result))
         }
-        Operator::I8x16Eq
-        | Operator::I8x16Ne
+        Operator::I8x16Eq | Operator::I16x8Eq | Operator::I32x4Eq => {
+            let (a, b) = state.pop2();
+            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
+            let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
+            state.push1(builder.ins().icmp(IntCC::Equal, bitcast_a, bitcast_b))
+        }
+        Operator::I8x16Ne
         | Operator::I8x16LtS
         | Operator::I8x16LtU
         | Operator::I8x16GtS
@@ -1174,7 +1179,6 @@ pub fn translate_operator(
         | Operator::I8x16LeU
         | Operator::I8x16GeS
         | Operator::I8x16GeU
-        | Operator::I16x8Eq
         | Operator::I16x8Ne
         | Operator::I16x8LtS
         | Operator::I16x8LtU
@@ -1184,7 +1188,6 @@ pub fn translate_operator(
         | Operator::I16x8LeU
         | Operator::I16x8GeS
         | Operator::I16x8GeU
-        | Operator::I32x4Eq
         | Operator::I32x4Ne
         | Operator::I32x4LtS
         | Operator::I32x4LtU

From e3a20d67b2dd6b26662ad4743412b6af6e14c948 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 23 Oct 2019 15:40:46 -0700
Subject: [PATCH 2917/3084] Add x86 SIMD legalization of `icmp ne`

---
 .../codegen/meta/src/isa/x86/legalize.rs      | 10 +++++
 .../isa/x86/simd-comparison-legalize.clif     | 12 +++++
 .../isa/x86/simd-comparison-run.clif          | 45 +++++++++++++++++++
 3 files changed, 67 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index e43e3c3e37..1aa76798ea 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -496,6 +496,16 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         }
     }
 
+    // SIMD icmp ne
+    let ne = Literal::enumerator_for(&imm.intcc, "ne");
+    for ty in ValueType::all_lane_types().filter(|ty| allowed_simd_type(ty) && ty.is_int()) {
+        let icmp_ = icmp.bind(vector(ty, sse_vector_size));
+        narrow.legalize(
+            def!(c = icmp_(ne, a, b)),
+            vec![def!(x = icmp(eq, a, b)), def!(c = bnot(x))],
+        );
+    }
+
     narrow.custom_legalize(shuffle, "convert_shuffle");
     narrow.custom_legalize(extractlane, "convert_extractlane");
     narrow.custom_legalize(insertlane, "convert_insertlane");
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif
new file mode 100644
index 0000000000..4555a25b42
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif
@@ -0,0 +1,12 @@
+test legalizer
+set enable_simd
+target x86_64 skylake
+
+function %icmp_ne_32x4(i32x4, i32x4) -> b32x4 {
+ebb0(v0: i32x4, v1: i32x4):
+    v2 = icmp ne v0, v1
+    ; check: v3 = icmp eq v0, v1
+    ; nextln: v4 = vconst.b32x4 0xffffffffffffffffffffffffffffffff
+    ; nextln: v2 = bxor v4, v3
+    return v2
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
new file mode 100644
index 0000000000..6afbf418d6
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
@@ -0,0 +1,45 @@
+test run
+set enable_simd
+target x86_64 skylake
+
+function %icmp_eq_i8x16() -> b8 {
+ebb0:
+    v0 = vconst.i8x16 0x00
+    v1 = vconst.i8x16 0x00
+    v2 = icmp eq v0, v1
+    v3 = extractlane v2, 0
+    return v3
+}
+; run
+
+function %icmp_eq_i64x2() -> b64 {
+ebb0:
+    v0 = vconst.i64x2 0xffffffffffffffffffffffffffffffff
+    v1 = vconst.i64x2 0xffffffffffffffffffffffffffffffff
+    v2 = icmp eq v0, v1
+    v3 = extractlane v2, 1
+    return v3
+}
+; run
+
+function %icmp_ne_i32x4() -> b1 {
+ebb0:
+    v0 = vconst.i32x4 [0 1 2 3]
+    v1 = vconst.i32x4 [7 7 7 7]
+    v2 = icmp ne v0, v1
+    v3 = vall_true v2
+    return v3
+}
+; run
+
+function %icmp_ne_i16x8() -> b1 {
+ebb0:
+    v0 = vconst.i16x8 [0 1 2 3 4 5 6 7]
+    v1 = vconst.i16x8 [0 1 2 3 4 5 6 7]
+    v2 = icmp ne v0, v1
+    v3 = vall_true v2
+    v4 = bint.i32 v3
+    v5 = icmp_imm eq v4, 0
+    return v5
+}
+; run

From 3a94e8d46aa129fe474753f039bf640724f8390c Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 23 Oct 2019 15:42:26 -0700
Subject: [PATCH 2918/3084] Translate WASM iTxN.ne to CLIF

---
 cranelift/wasm/src/code_translator.rs | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 77d35373a4..df1912e231 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1170,8 +1170,13 @@ pub fn translate_operator(
             let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
             state.push1(builder.ins().icmp(IntCC::Equal, bitcast_a, bitcast_b))
         }
-        Operator::I8x16Ne
-        | Operator::I8x16LtS
+        Operator::I8x16Ne | Operator::I16x8Ne | Operator::I32x4Ne => {
+            let (a, b) = state.pop2();
+            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
+            let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
+            state.push1(builder.ins().icmp(IntCC::NotEqual, bitcast_a, bitcast_b))
+        }
+        Operator::I8x16LtS
         | Operator::I8x16LtU
         | Operator::I8x16GtS
         | Operator::I8x16GtU
@@ -1179,7 +1184,6 @@ pub fn translate_operator(
         | Operator::I8x16LeU
         | Operator::I8x16GeS
         | Operator::I8x16GeU
-        | Operator::I16x8Ne
         | Operator::I16x8LtS
         | Operator::I16x8LtU
         | Operator::I16x8GtS
@@ -1188,7 +1192,6 @@ pub fn translate_operator(
         | Operator::I16x8LeU
         | Operator::I16x8GeS
         | Operator::I16x8GeU
-        | Operator::I32x4Ne
         | Operator::I32x4LtS
         | Operator::I32x4LtU
         | Operator::I32x4GtS

From 6f35273055fec0b2f0be82697519b5a44c2f7098 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 24 Oct 2019 17:05:56 -0700
Subject: [PATCH 2919/3084] Add hexadecimal parsing for uimm8

This is to make it more clear in signed contexts that the uimm8 (e.g. 255) is in fact being used for its bits (e.g. 0xff).
---
 cranelift/reader/src/parser.rs | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 24f8b5a2e5..01b10cac97 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -665,9 +665,15 @@ impl<'a> Parser<'a> {
         if let Some(Token::Integer(text)) = self.token() {
             self.consume();
             // Lexer just gives us raw text that looks like an integer.
-            // Parse it as a u8 to check for overflow and other issues.
-            text.parse()
-                .map_err(|_| self.error("expected u8 decimal immediate"))
+            if text.starts_with("0x") {
+                // Parse it as a u8 in hexadecimal form.
+                u8::from_str_radix(&text[2..], 16)
+                    .map_err(|_| self.error("unable to parse u8 as a hexadecimal immediate"))
+            } else {
+                // Parse it as a u8 to check for overflow and other issues.
+                text.parse()
+                    .map_err(|_| self.error("expected u8 decimal immediate"))
+            }
         } else {
             err!(self.loc, err_msg)
         }
@@ -3255,6 +3261,18 @@ mod tests {
         );
     }
 
+    #[test]
+    fn u8_as_hex() {
+        fn parse_as_uimm8(text: &str) -> ParseResult {
+            Parser::new(text).match_uimm8("unable to parse u8")
+        }
+
+        assert_eq!(parse_as_uimm8("0").unwrap(), 0);
+        assert_eq!(parse_as_uimm8("0xff").unwrap(), 255);
+        assert!(parse_as_uimm8("-1").is_err());
+        assert!(parse_as_uimm8("0xffa").is_err());
+    }
+
     #[test]
     fn uimm128() {
         macro_rules! parse_as_constant_data {

From c454c3c771951cff2a1a8dbff5ccb19651c93f18 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 24 Oct 2019 17:07:42 -0700
Subject: [PATCH 2920/3084] Add x86 SIMD encoding for `icmp sgt`

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 35 +++++++++++--------
 cranelift/codegen/meta/src/isa/x86/opcodes.rs | 12 +++++++
 .../isa/x86/simd-comparison-binemit.clif      | 27 ++++++++++++++
 .../isa/x86/simd-comparison-run.clif          | 23 ++++++++++++
 4 files changed, 82 insertions(+), 15 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 6148b2ffad..2f21ddf044 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -685,6 +685,7 @@ pub(crate) fn define(
     let use_sse41 = settings.predicate_by_name("use_sse41");
     let use_ssse3_simd = settings.predicate_by_name("use_ssse3_simd");
     let use_sse41_simd = settings.predicate_by_name("use_sse41_simd");
+    let use_sse42_simd = settings.predicate_by_name("use_sse42_simd");
 
     // Definitions.
     let mut e = PerCpuModeEncodings::new();
@@ -2025,21 +2026,25 @@ pub(crate) fn define(
         e.enc_32_64(x86_psra, rec_fa.opcodes(*opcodes));
     }
 
-    // SIMD icmp using PCMPEQ*
-    for ty in ValueType::all_lane_types().filter(|t| t.is_int() && allowed_simd_type(t)) {
-        let (opcodes, isa_predicate): (&[_], _) = match ty.lane_bits() {
-            8 => (&PCMPEQB, None),
-            16 => (&PCMPEQW, None),
-            32 => (&PCMPEQD, None),
-            64 => (&PCMPEQQ, Some(use_sse41_simd)),
-            _ => panic!("invalid size for SIMD icmp"),
-        };
-
-        let instruction = icmp
-            .bind(Immediate::IntCC(IntCC::Equal))
-            .bind(vector(ty, sse_vector_size));
-        let template = rec_icscc_fpr.nonrex().opcodes(opcodes);
-        e.enc_32_64_maybe_isap(instruction, template, isa_predicate);
+    // SIMD integer comparisons
+    {
+        use IntCC::*;
+        for (ty, cc, opcodes, isa_predicate) in &[
+            (I8, Equal, &PCMPEQB[..], None),
+            (I16, Equal, &PCMPEQW[..], None),
+            (I32, Equal, &PCMPEQD[..], None),
+            (I64, Equal, &PCMPEQQ[..], Some(use_sse41_simd)),
+            (I8, SignedGreaterThan, &PCMPGTB[..], None),
+            (I16, SignedGreaterThan, &PCMPGTW[..], None),
+            (I32, SignedGreaterThan, &PCMPGTD[..], None),
+            (I64, SignedGreaterThan, &PCMPGTQ, Some(use_sse42_simd)),
+        ] {
+            let instruction = icmp
+                .bind(Immediate::IntCC(*cc))
+                .bind(vector(*ty, sse_vector_size));
+            let template = rec_icscc_fpr.nonrex().opcodes(opcodes);
+            e.enc_32_64_maybe_isap(instruction, template, *isa_predicate);
+        }
     }
 
     // Reference type instructions
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index 0491028810..ecc64f560d 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -281,6 +281,18 @@ pub static PCMPEQQ: [u8; 4] = [0x66, 0x0f, 0x38, 0x29];
 /// Compare packed data for equal (SSE2).
 pub static PCMPEQW: [u8; 3] = [0x66, 0x0f, 0x75];
 
+/// Compare packed signed byte integers for greater than (SSE2).
+pub static PCMPGTB: [u8; 3] = [0x66, 0x0f, 0x64];
+
+/// Compare packed signed doubleword integers for greater than (SSE2).
+pub static PCMPGTD: [u8; 3] = [0x66, 0x0f, 0x66];
+
+/// Compare packed signed quadword integers for greater than (SSE4.2).
+pub static PCMPGTQ: [u8; 4] = [0x66, 0x0f, 0x38, 0x37];
+
+/// Compare packed signed word integers for greater than (SSE2).
+pub static PCMPGTW: [u8; 3] = [0x66, 0x0f, 0x65];
+
 /// Extract doubleword or quadword, depending on REX.W (SSE4.1).
 pub static PEXTR: [u8; 4] = [0x66, 0x0f, 0x3a, 0x16];
 
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
new file mode 100644
index 0000000000..a60b0eaf4c
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
@@ -0,0 +1,27 @@
+test binemit
+set enable_simd
+target x86_64 skylake
+
+function %icmp_sgt_i8x16(i8x16, i8x16) -> b8x16 {
+ebb0(v0: i8x16 [%xmm2], v1: i8x16 [%xmm1]):
+[-, %xmm2]  v2 = icmp sgt v0, v1     ; bin: 66 0f 64 d1
+            return v2
+}
+
+function %icmp_sgt_i16x8(i16x8, i16x8) -> b16x8 {
+ebb0(v0: i16x8 [%xmm4], v1: i16x8 [%xmm3]):
+[-, %xmm4]  v2 = icmp sgt v0, v1     ; bin: 66 0f 65 e3
+            return v2
+}
+
+function %icmp_sgt_i32x4(i32x4, i32x4) -> b32x4 {
+ebb0(v0: i32x4 [%xmm6], v1: i32x4 [%xmm5]):
+[-, %xmm6]  v2 = icmp sgt v0, v1     ; bin: 66 0f 66 f5
+            return v2
+}
+
+function %icmp_sgt_i64x2(i64x2, i64x2) -> b64x2 {
+ebb0(v0: i64x2 [%xmm0], v1: i64x2 [%xmm7]):
+[-, %xmm0]  v2 = icmp sgt v0, v1     ; bin: 66 0f 38 37 c7
+            return v2
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
index 6afbf418d6..d3f2abe304 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
@@ -43,3 +43,26 @@ ebb0:
     return v5
 }
 ; run
+
+function %icmp_sgt_i8x16() -> b1 {
+ebb0:
+    v0 = vconst.i8x16 [0 1 2 0 0 0 0 0 0 0 0 0 0 0 0 0]
+    v1 = vconst.i8x16 [1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0xff]
+    v2 = icmp sgt v0, v1
+    v3 = raw_bitcast.i8x16 v2
+    v4 = vconst.i8x16 [0 0 0xff 0 0 0 0 0 0 0 0 0 0 0 0 0xff]
+    v7 = icmp eq v3, v4
+    v8 = vall_true v7
+    return v8
+}
+; run
+
+function %icmp_sgt_i64x2() -> b1 {
+ebb0:
+    v0 = vconst.i64x2 [0 -42]
+    v1 = vconst.i64x2 [-1 -43]
+    v2 = icmp sgt v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run

From f053595748753e1b11789fe120ee12dd0492a630 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 24 Oct 2019 17:13:29 -0700
Subject: [PATCH 2921/3084] Translate WASM iTxN.gt_s to CLIF

---
 cranelift/wasm/src/code_translator.rs | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index df1912e231..f5729b59be 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1176,9 +1176,18 @@ pub fn translate_operator(
             let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
             state.push1(builder.ins().icmp(IntCC::NotEqual, bitcast_a, bitcast_b))
         }
+        Operator::I8x16GtS | Operator::I16x8GtS | Operator::I32x4GtS => {
+            let (a, b) = state.pop2();
+            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
+            let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
+            state.push1(
+                builder
+                    .ins()
+                    .icmp(IntCC::SignedGreaterThan, bitcast_a, bitcast_b),
+            )
+        }
         Operator::I8x16LtS
         | Operator::I8x16LtU
-        | Operator::I8x16GtS
         | Operator::I8x16GtU
         | Operator::I8x16LeS
         | Operator::I8x16LeU
@@ -1186,7 +1195,6 @@ pub fn translate_operator(
         | Operator::I8x16GeU
         | Operator::I16x8LtS
         | Operator::I16x8LtU
-        | Operator::I16x8GtS
         | Operator::I16x8GtU
         | Operator::I16x8LeS
         | Operator::I16x8LeU
@@ -1194,7 +1202,6 @@ pub fn translate_operator(
         | Operator::I16x8GeU
         | Operator::I32x4LtS
         | Operator::I32x4LtU
-        | Operator::I32x4GtS
         | Operator::I32x4GtU
         | Operator::I32x4LeS
         | Operator::I32x4LeU

From 0ab5760fd77795cd6f75872a712920f95e892dc9 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 25 Oct 2019 10:12:35 -0700
Subject: [PATCH 2922/3084] Add x86 SIMD instructions for min and max

Only the I8, I16, and I32 versions are included since Cranelift lacks support for AVX.
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 23 ++++++++
 .../codegen/meta/src/isa/x86/instructions.rs  | 55 +++++++++++++++++++
 cranelift/codegen/meta/src/isa/x86/opcodes.rs | 48 ++++++++++++++++
 .../isa/x86/simd-comparison-binemit.clif      | 27 +++++++++
 .../isa/x86/simd-comparison-run.clif          | 41 ++++++++++++++
 5 files changed, 194 insertions(+)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 2f21ddf044..f9e140eead 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -517,6 +517,10 @@ pub(crate) fn define(
     let x86_pop = x86.by_name("x86_pop");
     let x86_pextr = x86.by_name("x86_pextr");
     let x86_pinsr = x86.by_name("x86_pinsr");
+    let x86_pmaxs = x86.by_name("x86_pmaxs");
+    let x86_pmaxu = x86.by_name("x86_pmaxu");
+    let x86_pmins = x86.by_name("x86_pmins");
+    let x86_pminu = x86.by_name("x86_pminu");
     let x86_pshufd = x86.by_name("x86_pshufd");
     let x86_pshufb = x86.by_name("x86_pshufb");
     let x86_psll = x86.by_name("x86_psll");
@@ -2047,6 +2051,25 @@ pub(crate) fn define(
         }
     }
 
+    // SIMD min/max
+    for (ty, inst, opcodes, isa_predicate) in &[
+        (I8, x86_pmaxs, &PMAXSB[..], Some(use_sse41_simd)),
+        (I16, x86_pmaxs, &PMAXSW[..], None),
+        (I32, x86_pmaxs, &PMAXSD[..], Some(use_sse41_simd)),
+        (I8, x86_pmaxu, &PMAXUB[..], None),
+        (I16, x86_pmaxu, &PMAXUW[..], Some(use_sse41_simd)),
+        (I32, x86_pmaxu, &PMAXUD[..], Some(use_sse41_simd)),
+        (I8, x86_pmins, &PMINSB[..], Some(use_sse41_simd)),
+        (I16, x86_pmins, &PMINSW[..], None),
+        (I32, x86_pmins, &PMINSD[..], Some(use_sse41_simd)),
+        (I8, x86_pminu, &PMINUB[..], None),
+        (I16, x86_pminu, &PMINUW[..], Some(use_sse41_simd)),
+        (I32, x86_pminu, &PMINUD[..], Some(use_sse41_simd)),
+    ] {
+        let inst_ = inst.bind(vector(*ty, sse_vector_size));
+        e.enc_32_64_maybe_isap(inst_, rec_fa.opcodes(opcodes), *isa_predicate);
+    }
+
     // Reference type instructions
 
     // Null references implemented as iconst 0.
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 7b77ceaf42..04dc6cfe12 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -487,5 +487,60 @@ pub(crate) fn define(
         .operands_out(vec![f]),
     );
 
+    let x = &Operand::new("x", IxN);
+    let y = &Operand::new("y", IxN);
+    let a = &Operand::new("a", IxN);
+    ig.push(
+        Inst::new(
+            "x86_pmaxs",
+            r#"
+        Maximum of Packed Signed Integers -- Compare signed integers in the first and second
+        operand and return the maximum values.
+        "#,
+            &formats.binary,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
+    ig.push(
+        Inst::new(
+            "x86_pmaxu",
+            r#"
+        Maximum of Packed Unsigned Integers -- Compare unsigned integers in the first and second
+        operand and return the maximum values.
+        "#,
+            &formats.binary,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
+    ig.push(
+        Inst::new(
+            "x86_pmins",
+            r#"
+        Minimum of Packed Signed Integers -- Compare signed integers in the first and second
+        operand and return the minimum values.
+        "#,
+            &formats.binary,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
+    ig.push(
+        Inst::new(
+            "x86_pminu",
+            r#"
+        Minimum of Packed Unsigned Integers -- Compare unsigned integers in the first and second
+        operand and return the minimum values.
+        "#,
+            &formats.binary,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
     ig.build()
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index ecc64f560d..8187283778 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -311,6 +311,54 @@ pub static PINSRB: [u8; 4] = [0x66, 0x0f, 0x3a, 0x20];
 /// Insert word (SSE2).
 pub static PINSRW: [u8; 3] = [0x66, 0x0f, 0xc4];
 
+/// Compare packed signed byte integers in xmm1 and xmm2/m128 and store packed maximum values in
+/// xmm1 (SSE4.1).
+pub static PMAXSB: [u8; 4] = [0x66, 0x0f, 0x38, 0x3c];
+
+/// Compare packed signed doubleword integers in xmm1 and xmm2/m128 and store packed maximum
+/// values in xmm1 (SSE4.1).
+pub static PMAXSD: [u8; 4] = [0x66, 0x0f, 0x38, 0x3d];
+
+/// Compare packed signed word integers in xmm1 and xmm2/m128 and store packed maximum values in
+/// xmm1 (SSE2).
+pub static PMAXSW: [u8; 3] = [0x66, 0x0f, 0xee];
+
+/// Compare packed unsigned byte integers in xmm1 and xmm2/m128 and store packed maximum values in
+/// xmm1 (SSE2).
+pub static PMAXUB: [u8; 3] = [0x66, 0x0f, 0xde];
+
+/// Compare packed unsigned doubleword integers in xmm1 and xmm2/m128 and store packed maximum
+/// values in xmm1 (SSE4.1).
+pub static PMAXUD: [u8; 4] = [0x66, 0x0f, 0x38, 0x3f];
+
+/// Compare packed unsigned word integers in xmm1 and xmm2/m128 and store packed maximum values in
+/// xmm1 (SSE4.1).
+pub static PMAXUW: [u8; 4] = [0x66, 0x0f, 0x38, 0x3e];
+
+/// Compare packed signed byte integers in xmm1 and xmm2/m128 and store packed minimum values in
+/// xmm1 (SSE4.1).
+pub static PMINSB: [u8; 4] = [0x66, 0x0f, 0x38, 0x38];
+
+/// Compare packed signed doubleword integers in xmm1 and xmm2/m128 and store packed minimum
+/// values in xmm1 (SSE4.1).
+pub static PMINSD: [u8; 4] = [0x66, 0x0f, 0x38, 0x39];
+
+/// Compare packed signed word integers in xmm1 and xmm2/m128 and store packed minimum values in
+/// xmm1 (SSE2).
+pub static PMINSW: [u8; 3] = [0x66, 0x0f, 0xea];
+
+/// Compare packed unsigned byte integers in xmm1 and xmm2/m128 and store packed minimum values in
+/// xmm1 (SSE2).
+pub static PMINUB: [u8; 3] = [0x66, 0x0f, 0xda];
+
+/// Compare packed unsigned doubleword integers in xmm1 and xmm2/m128 and store packed minimum
+/// values in xmm1 (SSE4.1).
+pub static PMINUD: [u8; 4] = [0x66, 0x0f, 0x38, 0x3b];
+
+/// Compare packed unsigned word integers in xmm1 and xmm2/m128 and store packed minimum values in
+/// xmm1 (SSE4.1).
+pub static PMINUW: [u8; 4] = [0x66, 0x0f, 0x38, 0x3a];
+
 /// Multiply the packed signed word integers in xmm1 and xmm2/m128, and store the low 16 bits of
 /// the results in xmm1 (SSE2).
 pub static PMULLW: [u8; 3] = [0x66, 0x0f, 0xd5];
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
index a60b0eaf4c..aecfbe1ad7 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
@@ -25,3 +25,30 @@ ebb0(v0: i64x2 [%xmm0], v1: i64x2 [%xmm7]):
 [-, %xmm0]  v2 = icmp sgt v0, v1     ; bin: 66 0f 38 37 c7
             return v2
 }
+
+function %min_max_i8x16(i8x16, i8x16) {
+ebb0(v0: i8x16 [%xmm3], v1: i8x16 [%xmm1]):
+[-, %xmm3]  v2 = x86_pmaxs v0, v1     ; bin: 66 0f 38 3c d9
+[-, %xmm3]  v3 = x86_pmaxu v0, v1     ; bin: 66 0f de d9
+[-, %xmm3]  v4 = x86_pmins v0, v1     ; bin: 66 0f 38 38 d9
+[-, %xmm3]  v5 = x86_pminu v0, v1     ; bin: 66 0f da d9
+            return
+}
+
+function %min_max_i16x8(i16x8, i16x8) {
+ebb0(v0: i16x8 [%xmm2], v1: i16x8 [%xmm5]):
+[-, %xmm2]  v2 = x86_pmaxs v0, v1     ; bin: 66 0f ee d5
+[-, %xmm2]  v3 = x86_pmaxu v0, v1     ; bin: 66 0f 38 3e d5
+[-, %xmm2]  v4 = x86_pmins v0, v1     ; bin: 66 0f ea d5
+[-, %xmm2]  v5 = x86_pminu v0, v1     ; bin: 66 0f 38 3a d5
+            return
+}
+
+function %min_max_i32x4(i32x4, i32x4) {
+ebb0(v0: i32x4 [%xmm2], v1: i32x4 [%xmm4]):
+[-, %xmm2]  v2 = x86_pmaxs v0, v1     ; bin: 66 0f 38 3d d4
+[-, %xmm2]  v3 = x86_pmaxu v0, v1     ; bin: 66 0f 38 3f d4
+[-, %xmm2]  v4 = x86_pmins v0, v1     ; bin: 66 0f 38 39 d4
+[-, %xmm2]  v5 = x86_pminu v0, v1     ; bin: 66 0f 38 3b d4
+            return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
index d3f2abe304..ab3a525243 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
@@ -66,3 +66,44 @@ ebb0:
     return v8
 }
 ; run
+
+function %maxs_i8x16() -> b1 {
+ebb0:
+    v0 = vconst.i8x16 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1] ; 1 will be greater than -1 == 0xff with
+    ; signed max
+    v1 = vconst.i8x16 [0xff 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
+    v2 = x86_pmaxs v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run
+
+function %maxu_i16x8() -> b1 {
+ebb0:
+    v0 = vconst.i16x8 [0 1 1 1 1 1 1 1]
+    v1 = vconst.i16x8 [-1 1 1 1 1 1 1 1] ; -1 == 0xff will be greater with unsigned max
+    v2 = x86_pmaxu v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run
+
+function %mins_i32x4() -> b1 {
+ebb0:
+    v0 = vconst.i32x4 [0 1 1 1]
+    v1 = vconst.i32x4 [-1 1 1 1] ; -1 == 0xff will be less with signed min
+    v2 = x86_pmins v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run
+
+function %minu_i8x16() -> b1 {
+ebb0:
+    v0 = vconst.i8x16 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1] ; 1 < 2 with unsiged min
+    v1 = vconst.i8x16 [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2]
+    v2 = x86_pminu v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run

From feffed85d238f7653fb626ccc4124dece7b2bacf Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 25 Oct 2019 10:43:21 -0700
Subject: [PATCH 2923/3084] Add x86 SIMD legalizations for integer greater-than

This includes `icmp ugt`, `icmp sge`, and `icmp uge` for vectors with lanes of I8, I16, and I32.
---
 .../codegen/meta/src/isa/x86/legalize.rs      | 35 ++++++++++++++++++-
 .../isa/x86/simd-comparison-legalize.clif     | 24 +++++++++++++
 .../isa/x86/simd-comparison-run.clif          | 30 ++++++++++++++++
 3 files changed, 88 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 1aa76798ea..e30ea592a9 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -3,7 +3,7 @@ use crate::cdsl::instructions::{vector, Bindable, InstructionGroup};
 use crate::cdsl::types::{LaneType, ValueType};
 use crate::cdsl::xform::TransformGroupBuilder;
 use crate::shared::types::Float::F64;
-use crate::shared::types::Int::{I16, I32, I64};
+use crate::shared::types::Int::{I16, I32, I64, I8};
 use crate::shared::Definitions as SharedDefinitions;
 
 #[allow(clippy::many_single_char_names)]
@@ -69,6 +69,9 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
 
     let x86_bsf = x86_instructions.by_name("x86_bsf");
     let x86_bsr = x86_instructions.by_name("x86_bsr");
+    let x86_pmaxu = x86_instructions.by_name("x86_pmaxu");
+    let x86_pmins = x86_instructions.by_name("x86_pmins");
+    let x86_pminu = x86_instructions.by_name("x86_pminu");
     let x86_pshufb = x86_instructions.by_name("x86_pshufb");
     let x86_pshufd = x86_instructions.by_name("x86_pshufd");
     let x86_psll = x86_instructions.by_name("x86_psll");
@@ -506,6 +509,36 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         );
     }
 
+    // SIMD icmp ugt
+    let ugt = Literal::enumerator_for(&imm.intcc, "ugt");
+    for ty in &[I8, I16, I32] {
+        let icmp_ = icmp.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(
+            def!(c = icmp_(ugt, a, b)),
+            vec![def!(x = x86_pmaxu(a, b)), def!(c = icmp(eq, a, x))],
+        );
+    }
+
+    // SIMD icmp sge
+    let sge = Literal::enumerator_for(&imm.intcc, "sge");
+    for ty in &[I8, I16, I32] {
+        let icmp_ = icmp.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(
+            def!(c = icmp_(sge, a, b)),
+            vec![def!(x = x86_pmins(a, b)), def!(c = icmp(eq, x, b))],
+        );
+    }
+
+    // SIMD icmp uge
+    let uge = Literal::enumerator_for(&imm.intcc, "uge");
+    for ty in &[I8, I16, I32] {
+        let icmp_ = icmp.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(
+            def!(c = icmp_(uge, a, b)),
+            vec![def!(x = x86_pminu(a, b)), def!(c = icmp(eq, x, b))],
+        );
+    }
+
     narrow.custom_legalize(shuffle, "convert_shuffle");
     narrow.custom_legalize(extractlane, "convert_extractlane");
     narrow.custom_legalize(insertlane, "convert_insertlane");
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif
index 4555a25b42..b4d9681285 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif
@@ -10,3 +10,27 @@ ebb0(v0: i32x4, v1: i32x4):
     ; nextln: v2 = bxor v4, v3
     return v2
 }
+
+function %icmp_ugt_i32x4(i32x4, i32x4) -> b32x4 {
+ebb0(v0: i32x4, v1: i32x4):
+    v2 = icmp ugt v0, v1
+    ; check: v3 = x86_pmaxu v0, v1
+    ; nextln: v2 = icmp eq v0, v3
+    return v2
+}
+
+function %icmp_sge_i16x8(i16x8, i16x8) -> b16x8 {
+ebb0(v0: i16x8, v1: i16x8):
+    v2 = icmp sge v0, v1
+    ; check: v3 = x86_pmins v0, v1
+    ; nextln: v2 = icmp eq v3, v1
+    return v2
+}
+
+function %icmp_uge_i8x16(i8x16, i8x16) -> b8x16 {
+ebb0(v0: i8x16, v1: i8x16):
+    v2 = icmp uge v0, v1
+    ; check: v3 = x86_pminu v0, v1
+    ; nextln: v2 = icmp eq v3, v1
+    return v2
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
index ab3a525243..0cffbc6708 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
@@ -107,3 +107,33 @@ ebb0:
     return v8
 }
 ; run
+
+function %icmp_ugt_i8x16() -> b1 {
+ebb0:
+    v0 = vconst.i8x16 [1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16]
+    v1 = vconst.i8x16 [0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
+    v2 = icmp ugt v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run
+
+function %icmp_sge_i16x8() -> b1 {
+ebb0:
+    v0 = vconst.i16x8 [-1 1 2 3 4 5 6 7]
+    v1 = vconst.i16x8 [-1 1 1 1 1 1 1 1]
+    v2 = icmp sge v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run
+
+function %icmp_uge_i32x4() -> b1 {
+ebb0:
+    v0 = vconst.i32x4 [1 2 3 4]
+    v1 = vconst.i32x4 [1 1 1 1]
+    v2 = icmp uge v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run

From 7715c5e07a8e1f72b2ff24299746e86436db8377 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 25 Oct 2019 10:51:16 -0700
Subject: [PATCH 2924/3084] Translate WASM integer greater-than to CLIF

---
 cranelift/wasm/src/code_translator.rs | 39 ++++++++++++++++++++-------
 1 file changed, 30 insertions(+), 9 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index f5729b59be..fae0efe9e8 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1186,27 +1186,48 @@ pub fn translate_operator(
                     .icmp(IntCC::SignedGreaterThan, bitcast_a, bitcast_b),
             )
         }
+        Operator::I8x16GtU | Operator::I16x8GtU | Operator::I32x4GtU => {
+            let (a, b) = state.pop2();
+            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
+            let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
+            state.push1(
+                builder
+                    .ins()
+                    .icmp(IntCC::UnsignedGreaterThan, bitcast_a, bitcast_b),
+            )
+        }
+        Operator::I8x16GeS | Operator::I16x8GeS | Operator::I32x4GeS => {
+            let (a, b) = state.pop2();
+            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
+            let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
+            state.push1(
+                builder
+                    .ins()
+                    .icmp(IntCC::SignedGreaterThanOrEqual, bitcast_a, bitcast_b),
+            )
+        }
+        Operator::I8x16GeU | Operator::I16x8GeU | Operator::I32x4GeU => {
+            let (a, b) = state.pop2();
+            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
+            let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
+            state.push1(
+                builder
+                    .ins()
+                    .icmp(IntCC::UnsignedGreaterThanOrEqual, bitcast_a, bitcast_b),
+            )
+        }
         Operator::I8x16LtS
         | Operator::I8x16LtU
-        | Operator::I8x16GtU
         | Operator::I8x16LeS
         | Operator::I8x16LeU
-        | Operator::I8x16GeS
-        | Operator::I8x16GeU
         | Operator::I16x8LtS
         | Operator::I16x8LtU
-        | Operator::I16x8GtU
         | Operator::I16x8LeS
         | Operator::I16x8LeU
-        | Operator::I16x8GeS
-        | Operator::I16x8GeU
         | Operator::I32x4LtS
         | Operator::I32x4LtU
-        | Operator::I32x4GtU
         | Operator::I32x4LeS
         | Operator::I32x4LeU
-        | Operator::I32x4GeS
-        | Operator::I32x4GeU
         | Operator::F32x4Eq
         | Operator::F32x4Ne
         | Operator::F32x4Lt

From ce67ea5d5817efd8caac819f1e22c5c4c8db733f Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 25 Oct 2019 10:56:40 -0700
Subject: [PATCH 2925/3084] Factor out common translations for SIMD comparisons

---
 cranelift/wasm/src/code_translator.rs | 65 ++++++++++-----------------
 1 file changed, 23 insertions(+), 42 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index fae0efe9e8..ccc5591a0c 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1165,57 +1165,26 @@ pub fn translate_operator(
             state.push1(builder.ins().bint(I32, bool_result))
         }
         Operator::I8x16Eq | Operator::I16x8Eq | Operator::I32x4Eq => {
-            let (a, b) = state.pop2();
-            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
-            let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
-            state.push1(builder.ins().icmp(IntCC::Equal, bitcast_a, bitcast_b))
+            translate_vector_icmp(IntCC::Equal, type_of(op), builder, state)
         }
         Operator::I8x16Ne | Operator::I16x8Ne | Operator::I32x4Ne => {
-            let (a, b) = state.pop2();
-            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
-            let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
-            state.push1(builder.ins().icmp(IntCC::NotEqual, bitcast_a, bitcast_b))
+            translate_vector_icmp(IntCC::NotEqual, type_of(op), builder, state)
         }
         Operator::I8x16GtS | Operator::I16x8GtS | Operator::I32x4GtS => {
-            let (a, b) = state.pop2();
-            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
-            let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
-            state.push1(
-                builder
-                    .ins()
-                    .icmp(IntCC::SignedGreaterThan, bitcast_a, bitcast_b),
-            )
+            translate_vector_icmp(IntCC::SignedGreaterThan, type_of(op), builder, state)
         }
         Operator::I8x16GtU | Operator::I16x8GtU | Operator::I32x4GtU => {
-            let (a, b) = state.pop2();
-            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
-            let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
-            state.push1(
-                builder
-                    .ins()
-                    .icmp(IntCC::UnsignedGreaterThan, bitcast_a, bitcast_b),
-            )
+            translate_vector_icmp(IntCC::UnsignedGreaterThan, type_of(op), builder, state)
         }
         Operator::I8x16GeS | Operator::I16x8GeS | Operator::I32x4GeS => {
-            let (a, b) = state.pop2();
-            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
-            let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
-            state.push1(
-                builder
-                    .ins()
-                    .icmp(IntCC::SignedGreaterThanOrEqual, bitcast_a, bitcast_b),
-            )
-        }
-        Operator::I8x16GeU | Operator::I16x8GeU | Operator::I32x4GeU => {
-            let (a, b) = state.pop2();
-            let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);
-            let bitcast_b = optionally_bitcast_vector(b, type_of(op), builder);
-            state.push1(
-                builder
-                    .ins()
-                    .icmp(IntCC::UnsignedGreaterThanOrEqual, bitcast_a, bitcast_b),
-            )
+            translate_vector_icmp(IntCC::SignedGreaterThanOrEqual, type_of(op), builder, state)
         }
+        Operator::I8x16GeU | Operator::I16x8GeU | Operator::I32x4GeU => translate_vector_icmp(
+            IntCC::UnsignedGreaterThanOrEqual,
+            type_of(op),
+            builder,
+            state,
+        ),
         Operator::I8x16LtS
         | Operator::I8x16LtU
         | Operator::I8x16LeS
@@ -1498,6 +1467,18 @@ fn translate_icmp(cc: IntCC, builder: &mut FunctionBuilder, state: &mut FuncTran
     state.push1(builder.ins().bint(I32, val));
 }
 
+fn translate_vector_icmp(
+    cc: IntCC,
+    needed_type: Type,
+    builder: &mut FunctionBuilder,
+    state: &mut FuncTranslationState,
+) {
+    let (a, b) = state.pop2();
+    let bitcast_a = optionally_bitcast_vector(a, needed_type, builder);
+    let bitcast_b = optionally_bitcast_vector(b, needed_type, builder);
+    state.push1(builder.ins().icmp(cc, bitcast_a, bitcast_b))
+}
+
 fn translate_fcmp(cc: FloatCC, builder: &mut FunctionBuilder, state: &mut FuncTranslationState) {
     let (arg0, arg1) = state.pop2();
     let val = builder.ins().fcmp(cc, arg0, arg1);

From af4637aff659381ce914da5f7368c4424d12764b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 25 Oct 2019 11:18:26 -0700
Subject: [PATCH 2926/3084] Add x86 SIMD legalizations for icmp less-than

---
 .../codegen/meta/src/isa/x86/legalize.rs      | 30 +++++++++-----
 .../isa/x86/simd-comparison-run.clif          | 40 +++++++++++++++++++
 2 files changed, 59 insertions(+), 11 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index e30ea592a9..f45edd513b 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -509,34 +509,42 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         );
     }
 
-    // SIMD icmp ugt
+    // SIMD icmp greater-/less-than
+    let sgt = Literal::enumerator_for(&imm.intcc, "sgt");
     let ugt = Literal::enumerator_for(&imm.intcc, "ugt");
+    let sge = Literal::enumerator_for(&imm.intcc, "sge");
+    let uge = Literal::enumerator_for(&imm.intcc, "uge");
+    let slt = Literal::enumerator_for(&imm.intcc, "slt");
+    let ult = Literal::enumerator_for(&imm.intcc, "ult");
+    let sle = Literal::enumerator_for(&imm.intcc, "sle");
+    let ule = Literal::enumerator_for(&imm.intcc, "ule");
     for ty in &[I8, I16, I32] {
+        // greater-than
         let icmp_ = icmp.bind(vector(*ty, sse_vector_size));
         narrow.legalize(
             def!(c = icmp_(ugt, a, b)),
             vec![def!(x = x86_pmaxu(a, b)), def!(c = icmp(eq, a, x))],
         );
-    }
-
-    // SIMD icmp sge
-    let sge = Literal::enumerator_for(&imm.intcc, "sge");
-    for ty in &[I8, I16, I32] {
         let icmp_ = icmp.bind(vector(*ty, sse_vector_size));
         narrow.legalize(
             def!(c = icmp_(sge, a, b)),
             vec![def!(x = x86_pmins(a, b)), def!(c = icmp(eq, x, b))],
         );
-    }
-
-    // SIMD icmp uge
-    let uge = Literal::enumerator_for(&imm.intcc, "uge");
-    for ty in &[I8, I16, I32] {
         let icmp_ = icmp.bind(vector(*ty, sse_vector_size));
         narrow.legalize(
             def!(c = icmp_(uge, a, b)),
             vec![def!(x = x86_pminu(a, b)), def!(c = icmp(eq, x, b))],
         );
+
+        // less-than
+        let icmp_ = icmp.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(def!(c = icmp_(slt, a, b)), vec![def!(c = icmp(sgt, b, a))]);
+        let icmp_ = icmp.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(def!(c = icmp_(ult, a, b)), vec![def!(c = icmp(ugt, b, a))]);
+        let icmp_ = icmp.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(def!(c = icmp_(sle, a, b)), vec![def!(c = icmp(sge, b, a))]);
+        let icmp_ = icmp.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(def!(c = icmp_(ule, a, b)), vec![def!(c = icmp(uge, b, a))]);
     }
 
     narrow.custom_legalize(shuffle, "convert_shuffle");
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
index 0cffbc6708..5d96585be0 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
@@ -137,3 +137,43 @@ ebb0:
     return v8
 }
 ; run
+
+function %icmp_slt_i32x4() -> b1 {
+ebb0:
+    v0 = vconst.i32x4 [-1 1 1 1]
+    v1 = vconst.i32x4 [1 2 3 4]
+    v2 = icmp slt v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run
+
+function %icmp_ult_i32x4() -> b1 {
+ebb0:
+    v0 = vconst.i32x4 [1 1 1 1]
+    v1 = vconst.i32x4 [-1 2 3 4] ; -1 = 0xffff... will be greater than 1 when unsigned
+    v2 = icmp ult v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run
+
+function %icmp_sle_i16x8() -> b1 {
+ebb0:
+    v0 = vconst.i16x8 [-1 -1 0 0 0 0 0 0]
+    v1 = vconst.i16x8 [-1  0 0 0 0 0 0 0]
+    v2 = icmp sle v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run
+
+function %icmp_ule_i16x8() -> b1 {
+ebb0:
+    v0 = vconst.i16x8 [-1  0 0 0 0 0 0 0]
+    v1 = vconst.i16x8 [-1 -1 0 0 0 0 0 0]
+    v2 = icmp ule v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run

From 1417215532046af54ed82a594398f3bfd3d8074c Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 25 Oct 2019 11:22:25 -0700
Subject: [PATCH 2927/3084] Translate WASM integer less-than to CLIF

---
 cranelift/wasm/src/code_translator.rs | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index ccc5591a0c..cab5405335 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1173,31 +1173,31 @@ pub fn translate_operator(
         Operator::I8x16GtS | Operator::I16x8GtS | Operator::I32x4GtS => {
             translate_vector_icmp(IntCC::SignedGreaterThan, type_of(op), builder, state)
         }
+        Operator::I8x16LtS | Operator::I16x8LtS | Operator::I32x4LtS => {
+            translate_vector_icmp(IntCC::SignedLessThan, type_of(op), builder, state)
+        }
         Operator::I8x16GtU | Operator::I16x8GtU | Operator::I32x4GtU => {
             translate_vector_icmp(IntCC::UnsignedGreaterThan, type_of(op), builder, state)
         }
+        Operator::I8x16LtU | Operator::I16x8LtU | Operator::I32x4LtU => {
+            translate_vector_icmp(IntCC::UnsignedLessThan, type_of(op), builder, state)
+        }
         Operator::I8x16GeS | Operator::I16x8GeS | Operator::I32x4GeS => {
             translate_vector_icmp(IntCC::SignedGreaterThanOrEqual, type_of(op), builder, state)
         }
+        Operator::I8x16LeS | Operator::I16x8LeS | Operator::I32x4LeS => {
+            translate_vector_icmp(IntCC::SignedLessThanOrEqual, type_of(op), builder, state)
+        }
         Operator::I8x16GeU | Operator::I16x8GeU | Operator::I32x4GeU => translate_vector_icmp(
             IntCC::UnsignedGreaterThanOrEqual,
             type_of(op),
             builder,
             state,
         ),
-        Operator::I8x16LtS
-        | Operator::I8x16LtU
-        | Operator::I8x16LeS
-        | Operator::I8x16LeU
-        | Operator::I16x8LtS
-        | Operator::I16x8LtU
-        | Operator::I16x8LeS
-        | Operator::I16x8LeU
-        | Operator::I32x4LtS
-        | Operator::I32x4LtU
-        | Operator::I32x4LeS
-        | Operator::I32x4LeU
-        | Operator::F32x4Eq
+        Operator::I8x16LeU | Operator::I16x8LeU | Operator::I32x4LeU => {
+            translate_vector_icmp(IntCC::UnsignedLessThanOrEqual, type_of(op), builder, state)
+        }
+        Operator::F32x4Eq
         | Operator::F32x4Ne
         | Operator::F32x4Lt
         | Operator::F32x4Gt

From 16441ed70a7cea55085bc8202c18f9dbbcde8e9a Mon Sep 17 00:00:00 2001
From: Daniel 
Date: Wed, 6 Nov 2019 05:12:34 +0100
Subject: [PATCH 2928/3084] SimpleJIT - Add a Drop impl to PtrLen (#1001)

* Implement SimpleJIT deallocation for non-windows targets.

* Remove custom Drop from feature(selinux-fix).

* Fix typo in unprotect error message.

* Drop SimpleJIT memory by default and add free_memory method instead.

* Move free_memory to a handle returned by Module::finish.

* Reduce memory handle content to necessary fields only.

* Use lower overhead method for leaking.
---
 cranelift/simplejit/src/backend.rs | 72 +++++++++++++++++++++++-------
 cranelift/simplejit/src/memory.rs  | 35 ++++++++++++++-
 2 files changed, 89 insertions(+), 18 deletions(-)

diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index 68d603af30..919a4ffe3f 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -118,9 +118,7 @@ pub struct SimpleJITBackend {
     isa: Box,
     symbols: HashMap,
     libcall_names: Box String>,
-    code_memory: Memory,
-    readonly_memory: Memory,
-    writable_memory: Memory,
+    memory: SimpleJITMemoryHandle,
 }
 
 /// A record of a relocation to perform.
@@ -150,6 +148,13 @@ pub struct SimpleJITCompiledData {
     relocs: Vec,
 }
 
+/// A handle to allow freeing memory allocated by the `Backend`.
+pub struct SimpleJITMemoryHandle {
+    code: Memory,
+    readonly: Memory,
+    writable: Memory,
+}
+
 impl SimpleJITBackend {
     fn lookup_symbol(&self, name: &str) -> *const u8 {
         match self.symbols.get(name) {
@@ -199,23 +204,32 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
     type CompiledData = SimpleJITCompiledData;
 
     /// SimpleJIT emits code and data into memory, and provides raw pointers
-    /// to them.
+    /// to them. They are valid for the remainder of the program's life, unless
+    /// [`free_memory`] is used.
+    ///
+    /// [`free_memory`]: #method.free_memory
     type FinalizedFunction = *const u8;
     type FinalizedData = (*mut u8, usize);
 
     /// SimpleJIT emits code and data into memory as it processes them, so it
     /// doesn't need to provide anything after the `Module` is complete.
-    type Product = ();
+    /// The handle object that is returned can optionally be used to free
+    /// allocated memory if required.
+    type Product = SimpleJITMemoryHandle;
 
     /// Create a new `SimpleJITBackend`.
     fn new(builder: SimpleJITBuilder) -> Self {
+        let memory = SimpleJITMemoryHandle {
+            code: Memory::new(),
+            readonly: Memory::new(),
+            writable: Memory::new(),
+        };
+
         Self {
             isa: builder.isa,
             symbols: builder.symbols,
             libcall_names: builder.libcall_names,
-            code_memory: Memory::new(),
-            readonly_memory: Memory::new(),
-            writable_memory: Memory::new(),
+            memory,
         }
     }
 
@@ -248,7 +262,8 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
     ) -> ModuleResult {
         let size = code_size as usize;
         let ptr = self
-            .code_memory
+            .memory
+            .code
             .allocate(size, EXECUTABLE_DATA_ALIGNMENT)
             .expect("TODO: handle OOM etc.");
 
@@ -303,11 +318,13 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
 
         let size = init.size();
         let storage = if writable {
-            self.writable_memory
+            self.memory
+                .writable
                 .allocate(size, align.unwrap_or(WRITABLE_DATA_ALIGNMENT))
                 .expect("TODO: handle OOM etc.")
         } else {
-            self.readonly_memory
+            self.memory
+                .readonly
                 .allocate(size, align.unwrap_or(READONLY_DATA_ALIGNMENT))
                 .expect("TODO: handle OOM etc.")
         };
@@ -479,13 +496,20 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
 
     fn publish(&mut self) {
         // Now that we're done patching, prepare the memory for execution!
-        self.readonly_memory.set_readonly();
-        self.code_memory.set_readable_and_executable();
+        self.memory.readonly.set_readonly();
+        self.memory.code.set_readable_and_executable();
     }
 
-    /// SimpleJIT emits code and data into memory as it processes them, so it
-    /// doesn't need to provide anything after the `Module` is complete.
-    fn finish(self) {}
+    /// SimpleJIT emits code and data into memory as it processes them. This
+    /// method performs no additional processing, but returns a handle which
+    /// allows freeing the allocated memory. Otherwise said memory is leaked
+    /// to enable safe handling of the resulting pointers.
+    ///
+    /// This method does not need to be called when access to the memory
+    /// handle is not required.
+    fn finish(self) -> Self::Product {
+        self.memory
+    }
 }
 
 #[cfg(not(windows))]
@@ -531,6 +555,22 @@ fn lookup_with_dlsym(name: &str) -> *const u8 {
     }
 }
 
+impl SimpleJITMemoryHandle {
+    /// Free memory allocated for code and data segments of compiled functions.
+    ///
+    /// # Safety
+    ///
+    /// Because this function invalidates any pointers retrived from the
+    /// corresponding module, it should only be used when none of the functions
+    /// from that module are currently executing and none of the`fn` pointers
+    /// are called afterwards.
+    pub unsafe fn free_memory(&mut self) {
+        self.code.free_memory();
+        self.readonly.free_memory();
+        self.writable.free_memory();
+    }
+}
+
 struct SimpleJITRelocSink {
     pub relocs: Vec,
 }
diff --git a/cranelift/simplejit/src/memory.rs b/cranelift/simplejit/src/memory.rs
index c50bb480dc..68c3fd768c 100644
--- a/cranelift/simplejit/src/memory.rs
+++ b/cranelift/simplejit/src/memory.rs
@@ -105,8 +105,26 @@ impl PtrLen {
     }
 }
 
+// `MMapMut` from `cfg(feature = "selinux-fix")` already deallocates properly.
+#[cfg(all(not(target_os = "windows"), not(feature = "selinux-fix")))]
+impl Drop for PtrLen {
+    fn drop(&mut self) {
+        if !self.ptr.is_null() {
+            unsafe {
+                region::protect(self.ptr, self.len, region::Protection::ReadWrite)
+                    .expect("unable to unprotect memory");
+                libc::free(self.ptr as _);
+            }
+        }
+    }
+}
+
+// TODO: add a `Drop` impl for `cfg(target_os = "windows")`
+
 /// JIT memory manager. This manages pages of suitably aligned and
-/// accessible memory.
+/// accessible memory. Memory will be leaked by default to have
+/// function pointers remain valid for the remainder of the
+/// program's life.
 pub struct Memory {
     allocations: Vec,
     executable: usize,
@@ -209,9 +227,22 @@ impl Memory {
             }
         }
     }
+
+    /// Frees all allocated memory regions that would be leaked otherwise.
+    /// Likely to invalidate existing function pointers, causing unsafety.
+    pub unsafe fn free_memory(&mut self) {
+        self.allocations.clear();
+    }
 }
 
-// TODO: Implement Drop to unprotect and deallocate the memory?
+impl Drop for Memory {
+    fn drop(&mut self) {
+        // leak memory to guarantee validity of function pointers
+        mem::replace(&mut self.allocations, Vec::new())
+            .into_iter()
+            .for_each(mem::forget);
+    }
+}
 
 #[cfg(test)]
 mod tests {

From d5fb4825240cdbd0b630477ccdbde0b830e1d92e Mon Sep 17 00:00:00 2001
From: m4b 
Date: Tue, 5 Nov 2019 21:33:30 -0800
Subject: [PATCH 2929/3084] build: update everything to target-lexicon 0.9;

---
 cranelift/Cargo.toml           | 2 +-
 cranelift/codegen/Cargo.toml   | 2 +-
 cranelift/faerie/Cargo.toml    | 4 ++--
 cranelift/frontend/Cargo.toml  | 2 +-
 cranelift/fuzz/Cargo.toml      | 2 +-
 cranelift/native/Cargo.toml    | 2 +-
 cranelift/object/Cargo.toml    | 4 ++--
 cranelift/reader/Cargo.toml    | 2 +-
 cranelift/simplejit/Cargo.toml | 2 +-
 cranelift/wasm/Cargo.toml      | 2 +-
 10 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 5df578aa20..fb3606c092 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -39,7 +39,7 @@ serde = "1.0.8"
 term = "0.6.1"
 capstone = { version = "0.6.0", optional = true }
 wabt = { version = "0.9.1", optional = true }
-target-lexicon = "0.8.1"
+target-lexicon = "0.9"
 pretty_env_logger = "0.3.0"
 file-per-thread-logger = "0.1.2"
 indicatif = "0.13.0"
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 66176bb418..794f560ac1 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -17,7 +17,7 @@ cranelift-codegen-shared = { path = "./shared", version = "0.48.0" }
 cranelift-entity = { path = "../cranelift-entity", version = "0.48.0" }
 cranelift-bforest = { path = "../cranelift-bforest", version = "0.48.0" }
 hashbrown = { version = "0.6", optional = true }
-target-lexicon = "0.8.1"
+target-lexicon = "0.9"
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
 smallvec = { version = "1.0.0" }
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 8d636cdc7c..a886efa69f 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -11,10 +11,10 @@ edition = "2018"
 
 [dependencies]
 cranelift-module = { path = "../cranelift-module", version = "0.48.0" }
-faerie = "0.11.0"
+faerie = "0.11.1"
 goblin = "0.1.0"
 failure = "0.1.2"
-target-lexicon = "0.8.1"
+target-lexicon = "0.9"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 526dbdd1bc..8fe7f0aebb 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -12,7 +12,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
-target-lexicon = "0.8.1"
+target-lexicon = "0.9"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 smallvec = { version = "1.0.0" }
diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
index f164491ef8..8d9cdc0ffa 100644
--- a/cranelift/fuzz/Cargo.toml
+++ b/cranelift/fuzz/Cargo.toml
@@ -15,7 +15,7 @@ libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" }
 cranelift-codegen = { path = "../cranelift-codegen" }
 cranelift-wasm = { path = "../cranelift-wasm" }
 cranelift-reader = { path = "../cranelift-reader" }
-target-lexicon = "0.8.1"
+target-lexicon = "0.9"
 
 # Prevent this from interfering with workspaces
 [workspace]
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 19b952747b..96d4395ac3 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -11,7 +11,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
-target-lexicon = "0.8.1"
+target-lexicon = "0.9"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
 raw-cpuid = "7.0.3"
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 9f1a4be124..2df95b41f8 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -11,8 +11,8 @@ edition = "2018"
 
 [dependencies]
 cranelift-module = { path = "../cranelift-module", version = "0.48.0" }
-object = { version = "0.15.0", default-features = false, features = ["write"] }
-target-lexicon = "0.8.1"
+object = { git = "https://github.com/gimli-rs/object", rev = "cba3ed4932e4c594c5eab4f5ef6c51838f4a5056", default-features = false, features = ["write"] }
+target-lexicon = "0.9"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index fa16526792..e5f30be572 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -11,7 +11,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0" }
-target-lexicon = "0.8.1"
+target-lexicon = "0.9"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 401587bf56..17c9858a61 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -15,7 +15,7 @@ cranelift-native = { path = "../cranelift-native", version = "0.48.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
-target-lexicon = "0.8.1"
+target-lexicon = "0.9"
 memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 1a6e5ffec4..b9b89ef497 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -22,7 +22,7 @@ thiserror = "1.0.4"
 
 [dev-dependencies]
 wabt = "0.9.1"
-target-lexicon = "0.8.1"
+target-lexicon = "0.9"
 
 [features]
 default = ["std", "basic-blocks"]

From cf82863ea92ae8a3bd82a9e717fb89949fba770b Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 6 Nov 2019 14:38:46 -0800
Subject: [PATCH 2930/3084] Bump version to 0.49.0 (#1208)

---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  8 ++++----
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  2 +-
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 19 files changed, 69 insertions(+), 69 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index fb3606c092..521037ce53 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.48.0"
+version = "0.49.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.48.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.48.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.48.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.48.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.48.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.48.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.48.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.48.0" }
-cranelift-module = { path = "cranelift-module", version = "0.48.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.48.0" }
-cranelift-object = { path = "cranelift-object", version = "0.48.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.48.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.48.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.48.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.49.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.49.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.49.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.49.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.49.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.49.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.49.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.49.0" }
+cranelift-module = { path = "cranelift-module", version = "0.49.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.49.0" }
+cranelift-object = { path = "cranelift-object", version = "0.49.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.49.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.49.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.49.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 0f782c89cf..185972c9f0 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.48.0"
+version = "0.49.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.48.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.49.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 794f560ac1..29a07cda3b 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.48.0"
+version = "0.49.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.48.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.48.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.48.0" }
+cranelift-codegen-shared = { path = "./shared", version = "0.49.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.49.0" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.49.0" }
 hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.9"
 log = { version = "0.4.6", default-features = false }
@@ -29,7 +29,7 @@ byteorder = { version = "1.3.2" }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.48.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.49.0" }
 
 [features]
 default = ["std", "basic-blocks"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 6022e8ae8a..5d2aa01350 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.48.0"
+version = "0.49.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.48.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.48.0" }
+cranelift-codegen-shared = { path = "../shared", version = "0.49.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.49.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 83b56b4106..9237b6812f 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.48.0"
+version = "0.49.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/CraneStation/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 09d4ce10a4..9803503ffe 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.48.0"
+version = "0.49.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index a886efa69f..66b6a4bd92 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.48.0"
+version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,15 +10,15 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.48.0" }
-faerie = "0.11.1"
+cranelift-module = { path = "../cranelift-module", version = "0.49.0" }
+faerie = "0.12.0"
 goblin = "0.1.0"
 failure = "0.1.2"
 target-lexicon = "0.9"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.48.0"
+version = "0.49.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index bea35779a1..b65a1c36d9 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.48.0"
+version = "0.49.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.48.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.48.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.48.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.49.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.49.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.49.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 8fe7f0aebb..63bdf8ec62 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.48.0"
+version = "0.49.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", default-features = false }
 target-lexicon = "0.9"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index dbc4f413d0..89764df3fa 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.48.0"
+version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.48.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.49.0" }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 thiserror = "1.0.4"
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 96d4395ac3..7d62dfe95b 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.48.0"
+version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", default-features = false }
 target-lexicon = "0.9"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 2df95b41f8..7693cde0c5 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.48.0"
+version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,13 +10,13 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.48.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.49.0" }
 object = { git = "https://github.com/gimli-rs/object", rev = "cba3ed4932e4c594c5eab4f5ef6c51838f4a5056", default-features = false, features = ["write"] }
 target-lexicon = "0.9"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.48.0"
+version = "0.49.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index bd0cb7c3a4..54c891ded4 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.48.0"
+version = "0.49.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.48.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.49.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 6f5e45666c..0dd18f3616 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.48.0"
+version="0.49.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index e5f30be572..8083666125 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.48.0"
+version = "0.49.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0" }
 target-lexicon = "0.9"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index da4852585e..a56b81a9d8 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.48.0"
+version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.48.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.49.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 17c9858a61..6889acbaac 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.48.0"
+version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/CraneStation/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.48.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.48.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.49.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.49.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -20,7 +20,7 @@ memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.48.0"
+version = "0.49.0"
 default-features = false
 features = ["std"]
 
@@ -32,9 +32,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.48.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.48.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.48.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.49.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.49.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.49.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index ee7cb7ceca..e6749eb395 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.48.0"
+version = "0.49.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.48.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.49.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index b9b89ef497..2274699820 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.48.0"
+version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/CraneStation/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.39.2", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.48.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.48.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.48.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.49.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.49.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }

From 4ccf0fdfa3fa8655679c6dfdb526f2fffeff49f0 Mon Sep 17 00:00:00 2001
From: Peter Huene 
Date: Thu, 7 Nov 2019 16:41:32 -0800
Subject: [PATCH 2931/3084] Fix build errors in x86 unwind info when building
 no_std. (#1214)

This commit fixes the build errors in the unwind info implementation for
the x86 ABI by changing `byteorder` to build `no_std`.

This copies two simple functions from the `WriteBytesExt` trait so that
we can easily write to a `Vec` with a particular endianness.

Fixes #1203.
---
 cranelift/codegen/Cargo.toml            |  2 +-
 cranelift/codegen/src/isa/x86/abi.rs    |  2 +-
 cranelift/codegen/src/isa/x86/unwind.rs | 76 ++++++++++++-------------
 3 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 29a07cda3b..3fca049803 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -22,7 +22,7 @@ log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
 smallvec = { version = "1.0.0" }
 thiserror = "1.0.4"
-byteorder = { version = "1.3.2" }
+byteorder = { version = "1.3.2", default-features = false }
 # It is a goal of the cranelift-codegen crate to have minimal external dependencies.
 # Please don't add any unless they are essential to the task of creating binary
 # machine code. Integration tests that need external dependencies can be
diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 6d09635949..ef4b9f693e 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -836,6 +836,6 @@ pub fn emit_unwind_info(func: &ir::Function, isa: &dyn TargetIsa, mem: &mut Vec<
     // Assumption: RBP is being used as the frame pointer
     // In the future, Windows fastcall codegen should usually omit the frame pointer
     if let Some(info) = UnwindInfo::try_from_func(func, isa, Some(RU::rbp.into())) {
-        info.emit(mem).expect("failed to emit unwind information");
+        info.emit(mem);
     }
 }
diff --git a/cranelift/codegen/src/isa/x86/unwind.rs b/cranelift/codegen/src/isa/x86/unwind.rs
index 40caf44b63..2cddf40782 100644
--- a/cranelift/codegen/src/isa/x86/unwind.rs
+++ b/cranelift/codegen/src/isa/x86/unwind.rs
@@ -4,13 +4,25 @@ use super::registers::RU;
 use crate::ir::{Function, InstructionData, Opcode};
 use crate::isa::{CallConv, RegUnit, TargetIsa};
 use alloc::vec::Vec;
-use byteorder::{LittleEndian, WriteBytesExt};
+use byteorder::{ByteOrder, LittleEndian};
 
 /// Maximum (inclusive) size of a "small" stack allocation
 const SMALL_ALLOC_MAX_SIZE: u32 = 128;
 /// Maximum (inclusive) size of a "large" stack allocation that can represented in 16-bits
 const LARGE_ALLOC_16BIT_MAX_SIZE: u32 = 524280;
 
+fn write_u16(mem: &mut Vec, v: u16) {
+    let mut buf = [0; 2];
+    T::write_u16(&mut buf, v);
+    mem.extend(buf.iter());
+}
+
+fn write_u32(mem: &mut Vec, v: u32) {
+    let mut buf = [0; 4];
+    T::write_u32(&mut buf, v);
+    mem.extend(buf.iter());
+}
+
 /// The supported unwind codes for the x64 Windows ABI.
 ///
 /// See: https://docs.microsoft.com/en-us/cpp/build/exception-handling-x64
@@ -24,7 +36,7 @@ enum UnwindCode {
 }
 
 impl UnwindCode {
-    fn emit(&self, mem: &mut Vec) -> std::io::Result<()> {
+    fn emit(&self, mem: &mut Vec) {
         enum UnwindOperation {
             PushNonvolatileRegister,
             LargeStackAlloc,
@@ -34,36 +46,32 @@ impl UnwindCode {
 
         match self {
             Self::PushRegister { offset, reg } => {
-                mem.write_u8(*offset)?;
-                mem.write_u8(
-                    ((*reg as u8) << 4) | (UnwindOperation::PushNonvolatileRegister as u8),
-                )?;
+                mem.push(*offset);
+                mem.push(((*reg as u8) << 4) | (UnwindOperation::PushNonvolatileRegister as u8));
             }
             Self::StackAlloc { offset, size } => {
                 // Stack allocations on Windows must be a multiple of 8 and be at least 1 slot
                 assert!(*size >= 8);
                 assert!((*size % 8) == 0);
 
-                mem.write_u8(*offset)?;
+                mem.push(*offset);
                 if *size <= SMALL_ALLOC_MAX_SIZE {
-                    mem.write_u8(
+                    mem.push(
                         ((((*size - 8) / 8) as u8) << 4) | UnwindOperation::SmallStackAlloc as u8,
-                    )?;
+                    );
                 } else if *size <= LARGE_ALLOC_16BIT_MAX_SIZE {
-                    mem.write_u8(UnwindOperation::LargeStackAlloc as u8)?;
-                    mem.write_u16::((*size / 8) as u16)?;
+                    mem.push(UnwindOperation::LargeStackAlloc as u8);
+                    write_u16::(mem, (*size / 8) as u16);
                 } else {
-                    mem.write_u8((1 << 4) | (UnwindOperation::LargeStackAlloc as u8))?;
-                    mem.write_u32::(*size)?;
+                    mem.push((1 << 4) | (UnwindOperation::LargeStackAlloc as u8));
+                    write_u32::(mem, *size);
                 }
             }
             Self::SetFramePointer { offset, sp_offset } => {
-                mem.write_u8(*offset)?;
-                mem.write_u8((*sp_offset << 4) | (UnwindOperation::SetFramePointer as u8))?;
+                mem.push(*offset);
+                mem.push((*sp_offset << 4) | (UnwindOperation::SetFramePointer as u8));
             }
         };
-
-        Ok(())
     }
 
     fn node_count(&self) -> usize {
@@ -160,7 +168,7 @@ impl UnwindInfo {
                     match opcode {
                         Opcode::Iconst => {
                             let imm: i64 = imm.into();
-                            assert!(imm <= std::u32::MAX as i64);
+                            assert!(imm <= core::u32::MAX as i64);
                             assert!(stack_size.is_none());
 
                             // This instruction should only appear in a prologue to pass an
@@ -171,7 +179,7 @@ impl UnwindInfo {
                         }
                         Opcode::AdjustSpDownImm => {
                             let imm: i64 = imm.into();
-                            assert!(imm <= std::u32::MAX as i64);
+                            assert!(imm <= core::u32::MAX as i64);
 
                             unwind_codes.push(UnwindCode::StackAlloc {
                                 offset: unwind_offset,
@@ -223,7 +231,7 @@ impl UnwindInfo {
             .fold(0, |nodes, c| nodes + c.node_count())
     }
 
-    pub fn emit(&self, mem: &mut Vec) -> std::io::Result<()> {
+    pub fn emit(&self, mem: &mut Vec) {
         const UNWIND_INFO_VERSION: u8 = 1;
 
         let size = self.size();
@@ -237,30 +245,28 @@ impl UnwindInfo {
         let node_count = self.node_count();
         assert!(node_count <= 256);
 
-        mem.write_u8((self.flags << 3) | UNWIND_INFO_VERSION)?;
-        mem.write_u8(self.prologue_size)?;
-        mem.write_u8(node_count as u8)?;
+        mem.push((self.flags << 3) | UNWIND_INFO_VERSION);
+        mem.push(self.prologue_size);
+        mem.push(node_count as u8);
 
         if let Some(reg) = self.frame_register {
-            mem.write_u8((self.frame_register_offset << 4) | reg as u8)?;
+            mem.push((self.frame_register_offset << 4) | reg as u8);
         } else {
-            mem.write_u8(0)?;
+            mem.push(0);
         }
 
         // Unwind codes are written in reverse order (prologue offset descending)
         for code in self.unwind_codes.iter().rev() {
-            code.emit(mem)?;
+            code.emit(mem);
         }
 
         // To keep a 32-bit alignment, emit 2 bytes of padding if there's an odd number of 16-bit nodes
         if (node_count & 1) == 1 {
-            mem.write_u16::(0)?;
+            write_u16::(mem, 0);
         }
 
         // Ensure the correct number of bytes was emitted
         assert_eq!(mem.len() - offset, size);
-
-        Ok(())
     }
 }
 
@@ -331,9 +337,7 @@ mod tests {
         assert_eq!(unwind.size(), 12);
 
         let mut mem = Vec::new();
-        unwind
-            .emit(&mut mem)
-            .expect("failed to emit unwind information");
+        unwind.emit(&mut mem);
 
         assert_eq!(
             mem,
@@ -397,9 +401,7 @@ mod tests {
         assert_eq!(unwind.size(), 12);
 
         let mut mem = Vec::new();
-        unwind
-            .emit(&mut mem)
-            .expect("failed to emit unwind information");
+        unwind.emit(&mut mem);
 
         assert_eq!(
             mem,
@@ -463,9 +465,7 @@ mod tests {
         assert_eq!(unwind.size(), 16);
 
         let mut mem = Vec::new();
-        unwind
-            .emit(&mut mem)
-            .expect("failed to emit unwind information");
+        unwind.emit(&mut mem);
 
         assert_eq!(
             mem,

From 1074c7675e19a62930d3fa629dab05143bae8afa Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 6 Nov 2019 16:48:21 +0100
Subject: [PATCH 2932/3084] Clear the old_signatures between functions'
 compilations.

---
 cranelift/codegen/src/ir/dfg.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index 49f6fdbd73..9e8634b741 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -104,6 +104,7 @@ impl DataFlowGraph {
         self.value_lists.clear();
         self.values.clear();
         self.signatures.clear();
+        self.old_signatures.clear();
         self.ext_funcs.clear();
         self.values_labels = None;
         self.constants.clear();

From 143cb0148929222fad28d6bc437ad97522695cd5 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 6 Nov 2019 16:48:34 +0100
Subject: [PATCH 2933/3084] Do not align the stack frame for leaf functions not
 using the stack.

---
 cranelift/codegen/src/ir/function.rs          |  8 +++
 cranelift/codegen/src/isa/mod.rs              |  3 +-
 cranelift/codegen/src/isa/x86/abi.rs          |  9 ++--
 cranelift/codegen/src/stack_layout.rs         | 54 +++++++++++++------
 .../filetests/isa/x86/prologue-epilogue.clif  |  2 -
 .../isa/x86/windows_fastcall_x64_unwind.clif  |  2 +-
 6 files changed, 54 insertions(+), 24 deletions(-)

diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index 5318871689..f226eea98f 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -279,6 +279,14 @@ impl Function {
 
         Ok(())
     }
+
+    /// Returns true if the function is function that doesn't call any other functions. This is not
+    /// to be confused with a "leaf function" in Windows terminology.
+    pub fn is_leaf(&self) -> bool {
+        // Conservative result: if there's at least one function signature referenced in this
+        // function, assume it may call.
+        !self.dfg.signatures.is_empty()
+    }
 }
 
 /// Additional annotations for function display.
diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index aaa802b935..35f3581d7e 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -351,7 +351,8 @@ pub trait TargetIsa: fmt::Display + Sync {
             func.stack_slots.push(ss);
         }
 
-        layout_stack(&mut func.stack_slots, word_size)?;
+        let is_leaf = func.is_leaf();
+        layout_stack(&mut func.stack_slots, is_leaf, word_size)?;
         Ok(())
     }
 
diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index ef4b9f693e..fb4f9e53d5 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -540,7 +540,8 @@ fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) ->
     ss.offset = Some(-(bytes as StackOffset));
     func.stack_slots.push(ss);
 
-    layout_stack(&mut func.stack_slots, stack_align)?;
+    let is_leaf = func.is_leaf();
+    layout_stack(&mut func.stack_slots, is_leaf, stack_align)?;
     Ok(())
 }
 
@@ -587,7 +588,8 @@ fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> C
         offset: Some(-(SHADOW_STORE_SIZE + csr_stack_size)),
     });
 
-    let total_stack_size = layout_stack(&mut func.stack_slots, stack_align)? as i32;
+    let is_leaf = func.is_leaf();
+    let total_stack_size = layout_stack(&mut func.stack_slots, is_leaf, stack_align)? as i32;
     let local_stack_size = i64::from(total_stack_size - csr_stack_size);
 
     // Add CSRs to function signature
@@ -642,7 +644,8 @@ fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> C
         offset: Some(-csr_stack_size),
     });
 
-    let total_stack_size = layout_stack(&mut func.stack_slots, stack_align)? as i32;
+    let is_leaf = func.is_leaf();
+    let total_stack_size = layout_stack(&mut func.stack_slots, is_leaf, stack_align)? as i32;
     let local_stack_size = i64::from(total_stack_size - csr_stack_size);
 
     // Add CSRs to function signature
diff --git a/cranelift/codegen/src/stack_layout.rs b/cranelift/codegen/src/stack_layout.rs
index 732f9365c0..c335b844af 100644
--- a/cranelift/codegen/src/stack_layout.rs
+++ b/cranelift/codegen/src/stack_layout.rs
@@ -7,15 +7,20 @@ use core::cmp::{max, min};
 
 /// Compute the stack frame layout.
 ///
-/// Determine the total size of this stack frame and assign offsets to all `Spill` and
-/// `Explicit` stack slots.
+/// Determine the total size of this stack frame and assign offsets to all `Spill` and `Explicit`
+/// stack slots.
 ///
-/// The total frame size will be a multiple of `alignment` which must be a power of two.
+/// The total frame size will be a multiple of `alignment` which must be a power of two, unless the
+/// function doesn't perform any call.
 ///
 /// Returns the total stack frame size which is also saved in `frame.frame_size`.
 ///
 /// If the stack frame is too big, returns an `ImplLimitExceeded` error.
-pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CodegenResult {
+pub fn layout_stack(
+    frame: &mut StackSlots,
+    is_leaf: bool,
+    alignment: StackSize,
+) -> CodegenResult {
     // Each object and the whole stack frame must fit in 2 GB such that any relative offset within
     // the frame fits in a `StackOffset`.
     let max_size = StackOffset::max_value() as StackSize;
@@ -34,10 +39,14 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CodegenResu
     //
     // Both incoming and outgoing argument slots have fixed offsets that are treated as
     // reserved zones by the layout algorithm.
+    //
+    // If a function only has incoming arguments and does not perform any calls, then it doesn't
+    // require the stack to be aligned.
 
     let mut incoming_min = 0;
     let mut outgoing_max = 0;
     let mut min_align = alignment;
+    let mut must_align = is_leaf;
 
     for slot in frame.values() {
         if slot.size > max_size {
@@ -55,6 +64,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CodegenResu
                     .checked_add(slot.size as StackOffset)
                     .ok_or(CodegenError::ImplLimitExceeded)?;
                 outgoing_max = max(outgoing_max, offset);
+                must_align = true;
             }
             StackSlotKind::StructReturnSlot
             | StackSlotKind::SpillSlot
@@ -62,6 +72,7 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CodegenResu
             | StackSlotKind::EmergencySlot => {
                 // Determine the smallest alignment of any explicit or spill slot.
                 min_align = slot.alignment(min_align);
+                must_align = true;
             }
         }
     }
@@ -103,7 +114,10 @@ pub fn layout_stack(frame: &mut StackSlots, alignment: StackSize) -> CodegenResu
     offset = offset
         .checked_sub(outgoing_max)
         .ok_or(CodegenError::ImplLimitExceeded)?;
-    offset &= -(alignment as StackOffset);
+
+    if must_align {
+        offset &= -(alignment as StackOffset);
+    }
 
     let frame_size = (offset as StackSize).wrapping_neg();
     frame.frame_size = Some(frame_size);
@@ -122,16 +136,19 @@ mod tests {
     fn layout() {
         let sss = &mut StackSlots::new();
 
+        // For all these test cases, assume it will call.
+        let is_leaf = true;
+
         // An empty layout should have 0-sized stack frame.
-        assert_eq!(layout_stack(sss, 1), Ok(0));
-        assert_eq!(layout_stack(sss, 16), Ok(0));
+        assert_eq!(layout_stack(sss, is_leaf, 1), Ok(0));
+        assert_eq!(layout_stack(sss, is_leaf, 16), Ok(0));
 
         // Same for incoming arguments with non-negative offsets.
         let in0 = sss.make_incoming_arg(types::I64, 0);
         let in1 = sss.make_incoming_arg(types::I64, 8);
 
-        assert_eq!(layout_stack(sss, 1), Ok(0));
-        assert_eq!(layout_stack(sss, 16), Ok(0));
+        assert_eq!(layout_stack(sss, is_leaf, 1), Ok(0));
+        assert_eq!(layout_stack(sss, is_leaf, 16), Ok(0));
         assert_eq!(sss[in0].offset, Some(0));
         assert_eq!(sss[in1].offset, Some(8));
 
@@ -139,13 +156,13 @@ mod tests {
         let ss0 = sss.make_spill_slot(types::I64);
         let ss1 = sss.make_spill_slot(types::I32);
 
-        assert_eq!(layout_stack(sss, 1), Ok(12));
+        assert_eq!(layout_stack(sss, is_leaf, 1), Ok(12));
         assert_eq!(sss[in0].offset, Some(0));
         assert_eq!(sss[in1].offset, Some(8));
         assert_eq!(sss[ss0].offset, Some(-8));
         assert_eq!(sss[ss1].offset, Some(-12));
 
-        assert_eq!(layout_stack(sss, 16), Ok(16));
+        assert_eq!(layout_stack(sss, is_leaf, 16), Ok(16));
         assert_eq!(sss[in0].offset, Some(0));
         assert_eq!(sss[in1].offset, Some(8));
         assert_eq!(sss[ss0].offset, Some(-16));
@@ -155,14 +172,14 @@ mod tests {
         // should still pack nicely with the spill slots.
         let in2 = sss.make_incoming_arg(types::I32, -4);
 
-        assert_eq!(layout_stack(sss, 1), Ok(16));
+        assert_eq!(layout_stack(sss, is_leaf, 1), Ok(16));
         assert_eq!(sss[in0].offset, Some(0));
         assert_eq!(sss[in1].offset, Some(8));
         assert_eq!(sss[in2].offset, Some(-4));
         assert_eq!(sss[ss0].offset, Some(-12));
         assert_eq!(sss[ss1].offset, Some(-16));
 
-        assert_eq!(layout_stack(sss, 16), Ok(16));
+        assert_eq!(layout_stack(sss, is_leaf, 16), Ok(16));
         assert_eq!(sss[in0].offset, Some(0));
         assert_eq!(sss[in1].offset, Some(8));
         assert_eq!(sss[in2].offset, Some(-4));
@@ -172,7 +189,7 @@ mod tests {
         // Finally, make sure there is room for the outgoing args.
         let out0 = sss.get_outgoing_arg(types::I32, 0);
 
-        assert_eq!(layout_stack(sss, 1), Ok(20));
+        assert_eq!(layout_stack(sss, is_leaf, 1), Ok(20));
         assert_eq!(sss[in0].offset, Some(0));
         assert_eq!(sss[in1].offset, Some(8));
         assert_eq!(sss[in2].offset, Some(-4));
@@ -180,7 +197,7 @@ mod tests {
         assert_eq!(sss[ss1].offset, Some(-16));
         assert_eq!(sss[out0].offset, Some(0));
 
-        assert_eq!(layout_stack(sss, 16), Ok(32));
+        assert_eq!(layout_stack(sss, is_leaf, 16), Ok(32));
         assert_eq!(sss[in0].offset, Some(0));
         assert_eq!(sss[in1].offset, Some(8));
         assert_eq!(sss[in2].offset, Some(-4));
@@ -190,7 +207,10 @@ mod tests {
 
         // Also test that an unsupported offset is rejected.
         sss.get_outgoing_arg(types::I8, StackOffset::max_value() - 1);
-        assert_eq!(layout_stack(sss, 1), Err(CodegenError::ImplLimitExceeded));
+        assert_eq!(
+            layout_stack(sss, is_leaf, 1),
+            Err(CodegenError::ImplLimitExceeded)
+        );
     }
 
     #[test]
@@ -205,7 +225,7 @@ mod tests {
         ));
         let ss2 = sss.get_emergency_slot(types::I32, &[]);
 
-        assert_eq!(layout_stack(sss, 1), Ok(12));
+        assert_eq!(layout_stack(sss, true, 1), Ok(12));
         assert_eq!(sss[ss0].offset, Some(-4));
         assert_eq!(sss[ss1].offset, Some(-8));
         assert_eq!(sss[ss2].offset, Some(-12));
diff --git a/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif b/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
index f2fd3c68ee..f8a0c0146c 100644
--- a/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
+++ b/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
@@ -108,7 +108,6 @@ ebb0(v0: i64, v1: i64):
 ; nextln:     x86_push v18
 ; nextln:     x86_push v19
 ; nextln:     x86_push v20
-; nextln:     adjust_sp_down_imm 8
 ; nextln:     v2 = load.i32 v0
 ; nextln:     v3 = load.i32 v0+8
 ; nextln:     v4 = load.i32 v0+16
@@ -135,7 +134,6 @@ ebb0(v0: i64, v1: i64):
 ; nextln:     store v12, v1+80
 ; nextln:     store v13, v1+88
 ; nextln:     store v14, v1+96
-; nextln:     adjust_sp_up_imm 8
 ; nextln:     v26 = x86_pop.i64
 ; nextln:     v25 = x86_pop.i64
 ; nextln:     v24 = x86_pop.i64
diff --git a/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64_unwind.clif b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64_unwind.clif
index 6782d8cdce..7dc024f33c 100644
--- a/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64_unwind.clif
+++ b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64_unwind.clif
@@ -160,7 +160,7 @@ ebb0(v0: i64, v1: i64):
 ; nextln:         UnwindCode {
 ; nextln:             offset: 19,
 ; nextln:             op: SmallStackAlloc,
-; nextln:             info: 4,
+; nextln:             info: 3,
 ; nextln:             value: None,
 ; nextln:         },
 ; nextln:         UnwindCode {

From a06f2c87c29ebbf53ee1f22cd023e4c71e878a58 Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Thu, 7 Nov 2019 11:43:08 -0700
Subject: [PATCH 2934/3084] Pass Encoding to compute_size() for runtime
 Encoding inspection. #1156

In some cases, compute_size() is used to choose between various different Encodings
before one has been assigned to an instruction. For x86, the REX.W bit is stored
in the Encoding. To share recipes between REX/non-REX, that bit must be inspected
by compute_size().
---
 cranelift/codegen/src/isa/encoding.rs       | 20 +++++++++++++-------
 cranelift/codegen/src/isa/x86/enc_tables.rs |  8 +++++++-
 2 files changed, 20 insertions(+), 8 deletions(-)

diff --git a/cranelift/codegen/src/isa/encoding.rs b/cranelift/codegen/src/isa/encoding.rs
index 118bfa2006..99894cab2c 100644
--- a/cranelift/codegen/src/isa/encoding.rs
+++ b/cranelift/codegen/src/isa/encoding.rs
@@ -80,23 +80,29 @@ impl fmt::Display for DisplayEncoding {
     }
 }
 
-type SizeCalculatorFn = fn(&RecipeSizing, Inst, &RegDiversions, &Function) -> u8;
+type SizeCalculatorFn = fn(&RecipeSizing, Encoding, Inst, &RegDiversions, &Function) -> u8;
 
 /// Returns the base size of the Recipe, assuming it's fixed. This is the default for most
 /// encodings; others can be variable and longer than this base size, depending on the registers
 /// they're using and use a different function, specific per platform.
-pub fn base_size(sizing: &RecipeSizing, _: Inst, _: &RegDiversions, _: &Function) -> u8 {
+pub fn base_size(
+    sizing: &RecipeSizing,
+    _: Encoding,
+    _: Inst,
+    _: &RegDiversions,
+    _: &Function,
+) -> u8 {
     sizing.base_size
 }
 
 /// Code size information for an encoding recipe.
 ///
-/// All encoding recipes correspond to an exact instruction size.
+/// Encoding recipes may have runtime-determined instruction size.
 pub struct RecipeSizing {
-    /// Size in bytes of instructions encoded with this recipe.
+    /// Minimum size in bytes of instructions encoded with this recipe.
     pub base_size: u8,
 
-    /// Method computing the real instruction's size, given inputs and outputs.
+    /// Method computing the instruction's real size, given inputs and outputs.
     pub compute_size: SizeCalculatorFn,
 
     /// Allowed branch range in this recipe, if any.
@@ -132,7 +138,7 @@ impl EncInfo {
         }
     }
 
-    /// Get the precise size in bytes of instructions encoded with `enc`.
+    /// Get the size in bytes of `inst`, if it were encoded with `enc`.
     ///
     /// Returns 0 for illegal encodings.
     pub fn byte_size(
@@ -144,7 +150,7 @@ impl EncInfo {
     ) -> CodeOffset {
         self.sizing.get(enc.recipe()).map_or(0, |s| {
             let compute_size = s.compute_size;
-            CodeOffset::from(compute_size(&s, inst, divert, func))
+            CodeOffset::from(compute_size(&s, enc, inst, divert, func))
         })
     }
 
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 13e9d4ecef..fd95a50a01 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -10,7 +10,7 @@ use crate::ir::{self, Function, Inst, InstBuilder};
 use crate::isa::constraints::*;
 use crate::isa::enc_tables::*;
 use crate::isa::encoding::base_size;
-use crate::isa::encoding::RecipeSizing;
+use crate::isa::encoding::{Encoding, RecipeSizing};
 use crate::isa::RegUnit;
 use crate::isa::{self, TargetIsa};
 use crate::predicates;
@@ -46,6 +46,7 @@ fn additional_size_if(
 
 fn size_plus_maybe_offset_for_in_reg_0(
     sizing: &RecipeSizing,
+    _enc: Encoding,
     inst: Inst,
     divert: &RegDiversions,
     func: &Function,
@@ -54,6 +55,7 @@ fn size_plus_maybe_offset_for_in_reg_0(
 }
 fn size_plus_maybe_offset_for_in_reg_1(
     sizing: &RecipeSizing,
+    _enc: Encoding,
     inst: Inst,
     divert: &RegDiversions,
     func: &Function,
@@ -62,6 +64,7 @@ fn size_plus_maybe_offset_for_in_reg_1(
 }
 fn size_plus_maybe_sib_for_in_reg_0(
     sizing: &RecipeSizing,
+    _enc: Encoding,
     inst: Inst,
     divert: &RegDiversions,
     func: &Function,
@@ -70,6 +73,7 @@ fn size_plus_maybe_sib_for_in_reg_0(
 }
 fn size_plus_maybe_sib_for_in_reg_1(
     sizing: &RecipeSizing,
+    _enc: Encoding,
     inst: Inst,
     divert: &RegDiversions,
     func: &Function,
@@ -78,6 +82,7 @@ fn size_plus_maybe_sib_for_in_reg_1(
 }
 fn size_plus_maybe_sib_or_offset_for_in_reg_0(
     sizing: &RecipeSizing,
+    _enc: Encoding,
     inst: Inst,
     divert: &RegDiversions,
     func: &Function,
@@ -86,6 +91,7 @@ fn size_plus_maybe_sib_or_offset_for_in_reg_0(
 }
 fn size_plus_maybe_sib_or_offset_for_in_reg_1(
     sizing: &RecipeSizing,
+    _enc: Encoding,
     inst: Inst,
     divert: &RegDiversions,
     func: &Function,

From 7e32fa2731878ca9a1fd5351958c9eca7d8d4627 Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Fri, 8 Nov 2019 09:51:57 -0800
Subject: [PATCH 2935/3084] Try and assign directly to return registers;
 backtrack to use struct-return param (#1213)

* Try and assign directly to return registers; backtrack to use struct-return param

Rather than trying to count number of return registers that would be used by a
given set of return values, optimistically assign the return values to
registers. If we later find that we can't fit them all in registers, then
backtrack and introduce the use of a struct-return pointer parameter.

* Rename `rets2` and wrap it in an option so we avoid the clone for non-multi-value
---
 cranelift/codegen/src/isa/x86/abi.rs | 231 ++++++++-------------------
 1 file changed, 69 insertions(+), 162 deletions(-)

diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index fb4f9e53d5..e316d90683 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -34,6 +34,7 @@ static ARG_GPRS_WIN_FASTCALL_X64: [RU; 4] = [RU::rcx, RU::rdx, RU::r8, RU::r9];
 /// Return value registers for x86-64, when using windows fastcall
 static RET_GPRS_WIN_FASTCALL_X64: [RU; 1] = [RU::rax];
 
+#[derive(Clone)]
 struct Args {
     pointer_bytes: u8,
     pointer_bits: u8,
@@ -167,114 +168,6 @@ impl ArgAssigner for Args {
     }
 }
 
-/// Get the number of general-purpose and floating-point registers required to
-/// hold the given `AbiParam` returns.
-fn num_return_registers_required<'a>(
-    word_bit_size: u8,
-    call_conv: CallConv,
-    shared_flags: &shared_settings::Flags,
-    isa_flags: &isa_settings::Flags,
-    return_params: impl IntoIterator,
-) -> (usize, usize) {
-    // Pretend we have "infinite" registers to give out, since we aren't
-    // actually assigning `AbiParam`s to registers yet, just seeing how many
-    // registers we would need in order to fit all the `AbiParam`s in registers.
-    let gprs = &[RU::rax; 128];
-    let fpr_limit = std::usize::MAX;
-
-    let mut assigner = Args::new(
-        word_bit_size,
-        gprs,
-        fpr_limit,
-        call_conv,
-        shared_flags,
-        isa_flags,
-    );
-
-    let mut gprs_required = 0;
-    let mut fprs_required = 0;
-
-    for param in return_params {
-        match param.location {
-            ArgumentLoc::Unassigned => {
-                // Let this fall through so that we assign it a location and
-                // account for how many registers it ends up requiring below...
-            }
-            ArgumentLoc::Reg(_) => {
-                // This is already assigned to a register. Count it.
-                if param.value_type.is_float() {
-                    fprs_required += 1;
-                } else {
-                    gprs_required += 1;
-                }
-                continue;
-            }
-            _ => {
-                // It is already assigned, but not to a register. Skip it.
-                continue;
-            }
-        }
-
-        // We're going to mutate the type as it gets converted, so make our own
-        // copy that isn't visible to the outside world.
-        let mut param = param.clone();
-
-        let mut split_factor = 1;
-
-        loop {
-            match assigner.assign(¶m) {
-                ArgAction::Convert(ValueConversion::IntSplit) => {
-                    split_factor *= 2;
-                    param.value_type = param.value_type.half_width().unwrap();
-                }
-                ArgAction::Convert(ValueConversion::VectorSplit) => {
-                    split_factor *= 2;
-                    param.value_type = param.value_type.half_vector().unwrap();
-                }
-                ArgAction::Assign(ArgumentLoc::Reg(_))
-                | ArgAction::Convert(ValueConversion::IntBits)
-                | ArgAction::Convert(ValueConversion::Sext(_))
-                | ArgAction::Convert(ValueConversion::Uext(_)) => {
-                    // Ok! We can fit this (potentially split) value into a
-                    // register! Add the number of params we split the parameter
-                    // into to our current counts.
-                    if param.value_type.is_float() {
-                        fprs_required += split_factor;
-                    } else {
-                        gprs_required += split_factor;
-                    }
-
-                    // But we also have to call `assign` once for each split value, to
-                    // update `assigner`'s internal state.
-                    for _ in 1..split_factor {
-                        match assigner.assign(¶m) {
-                            ArgAction::Assign(_)
-                            | ArgAction::Convert(ValueConversion::IntBits)
-                            | ArgAction::Convert(ValueConversion::Sext(_))
-                            | ArgAction::Convert(ValueConversion::Uext(_)) => {
-                                continue;
-                            }
-                            otherwise => panic!(
-                                "unexpected action after first split succeeded: {:?}",
-                                otherwise
-                            ),
-                        }
-                    }
-
-                    // Continue to the next param.
-                    break;
-                }
-                ArgAction::Assign(loc) => panic!(
-                    "unexpected location assignment, should have had enough registers: {:?}",
-                    loc
-                ),
-            }
-        }
-    }
-
-    (gprs_required, fprs_required)
-}
-
 /// Legalize `sig`.
 pub fn legalize_signature(
     sig: &mut Cow,
@@ -332,70 +225,84 @@ pub fn legalize_signature(
         isa_flags,
     );
 
-    if sig.is_multi_return() && {
-        // Even if it is multi-return, see if the return values will fit into
-        // our available return registers.
-        let (gprs_required, fprs_required) = num_return_registers_required(
-            bits,
-            sig.call_conv,
-            shared_flags,
-            isa_flags,
-            &sig.returns,
-        );
-        gprs_required > ret_regs.len() || fprs_required > ret_fpr_limit
-    } {
-        debug_assert!(!sig.uses_struct_return_param());
+    let sig_is_multi_return = sig.is_multi_return();
 
-        // We're using the first register for the return pointer parameter.
-        let mut ret_ptr_param = AbiParam {
-            value_type: args.pointer_type,
-            purpose: ArgumentPurpose::StructReturn,
-            extension: ArgumentExtension::None,
-            location: ArgumentLoc::Unassigned,
-        };
-        match args.assign(&ret_ptr_param) {
-            ArgAction::Assign(ArgumentLoc::Reg(reg)) => {
-                ret_ptr_param.location = ArgumentLoc::Reg(reg);
-                sig.to_mut().params.push(ret_ptr_param);
+    // If this is a multi-value return and we don't have enough available return
+    // registers to fit all of the return values, we need to backtrack and start
+    // assigning locations all over again with a different strategy. In order to
+    // do that, we need a copy of the original assigner for the returns.
+    let backup_rets_for_struct_return = if sig_is_multi_return {
+        Some(rets.clone())
+    } else {
+        None
+    };
+
+    if let Some(new_returns) = legalize_args(&sig.returns, &mut rets) {
+        if sig.is_multi_return()
+            && new_returns
+                .iter()
+                .filter(|r| r.purpose == ArgumentPurpose::Normal)
+                .any(|r| !r.location.is_reg())
+        {
+            // The return values couldn't all fit into available return
+            // registers. Introduce the use of a struct-return parameter.
+            debug_assert!(!sig.uses_struct_return_param());
+
+            // We're using the first register for the return pointer parameter.
+            let mut ret_ptr_param = AbiParam {
+                value_type: args.pointer_type,
+                purpose: ArgumentPurpose::StructReturn,
+                extension: ArgumentExtension::None,
+                location: ArgumentLoc::Unassigned,
+            };
+            match args.assign(&ret_ptr_param) {
+                ArgAction::Assign(ArgumentLoc::Reg(reg)) => {
+                    ret_ptr_param.location = ArgumentLoc::Reg(reg);
+                    sig.to_mut().params.push(ret_ptr_param);
+                }
+                _ => unreachable!("return pointer should always get a register assignment"),
             }
-            _ => unreachable!("return pointer should always get a register assignment"),
-        }
 
-        // We're using the first return register for the return pointer (like
-        // sys v does).
-        let mut ret_ptr_return = AbiParam {
-            value_type: args.pointer_type,
-            purpose: ArgumentPurpose::StructReturn,
-            extension: ArgumentExtension::None,
-            location: ArgumentLoc::Unassigned,
-        };
-        match rets.assign(&ret_ptr_return) {
-            ArgAction::Assign(ArgumentLoc::Reg(reg)) => {
-                ret_ptr_return.location = ArgumentLoc::Reg(reg);
-                sig.to_mut().returns.push(ret_ptr_return);
+            let mut backup_rets = backup_rets_for_struct_return.unwrap();
+
+            // We're using the first return register for the return pointer (like
+            // sys v does).
+            let mut ret_ptr_return = AbiParam {
+                value_type: args.pointer_type,
+                purpose: ArgumentPurpose::StructReturn,
+                extension: ArgumentExtension::None,
+                location: ArgumentLoc::Unassigned,
+            };
+            match backup_rets.assign(&ret_ptr_return) {
+                ArgAction::Assign(ArgumentLoc::Reg(reg)) => {
+                    ret_ptr_return.location = ArgumentLoc::Reg(reg);
+                    sig.to_mut().returns.push(ret_ptr_return);
+                }
+                _ => unreachable!("return pointer should always get a register assignment"),
             }
-            _ => unreachable!("return pointer should always get a register assignment"),
-        }
 
-        sig.to_mut().returns.retain(|ret| {
-            // Either this is the return pointer, in which case we want to keep
-            // it, or else assume that it is assigned for a reason and doesn't
-            // conflict with our return pointering legalization.
-            debug_assert_eq!(
-                ret.location.is_assigned(),
-                ret.purpose != ArgumentPurpose::Normal
-            );
-            ret.location.is_assigned()
-        });
+            sig.to_mut().returns.retain(|ret| {
+                // Either this is the return pointer, in which case we want to keep
+                // it, or else assume that it is assigned for a reason and doesn't
+                // conflict with our return pointering legalization.
+                debug_assert_eq!(
+                    ret.location.is_assigned(),
+                    ret.purpose != ArgumentPurpose::Normal
+                );
+                ret.location.is_assigned()
+            });
+
+            if let Some(new_returns) = legalize_args(&sig.returns, &mut backup_rets) {
+                sig.to_mut().returns = new_returns;
+            }
+        } else {
+            sig.to_mut().returns = new_returns;
+        }
     }
 
     if let Some(new_params) = legalize_args(&sig.params, &mut args) {
         sig.to_mut().params = new_params;
     }
-
-    if let Some(new_returns) = legalize_args(&sig.returns, &mut rets) {
-        sig.to_mut().returns = new_returns;
-    }
 }
 
 /// Get register class for a type appearing in a legalized signature.

From e5a36e2c611935d352e3d4349f9de691660dc590 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 28 Oct 2019 13:21:43 -0700
Subject: [PATCH 2936/3084] Allow CLIF lexer to properly identify `NaN` and
 `Inf` keywords

---
 cranelift/reader/src/lexer.rs  | 11 +++++++++--
 cranelift/reader/src/parser.rs |  2 +-
 2 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/cranelift/reader/src/lexer.rs b/cranelift/reader/src/lexer.rs
index 0168d2027a..9838227e86 100644
--- a/cranelift/reader/src/lexer.rs
+++ b/cranelift/reader/src/lexer.rs
@@ -488,7 +488,13 @@ impl<'a> Lexer<'a> {
                     }
                 }
                 Some(ch) if ch.is_digit(10) => Some(self.scan_number()),
-                Some(ch) if ch.is_alphabetic() => Some(self.scan_word()),
+                Some(ch) if ch.is_alphabetic() => {
+                    if self.looking_at("NaN") || self.looking_at("Inf") {
+                        Some(self.scan_number())
+                    } else {
+                        Some(self.scan_word())
+                    }
+                }
                 Some('%') => Some(self.scan_name()),
                 Some('"') => Some(self.scan_string()),
                 Some('#') => Some(self.scan_hex_sequence()),
@@ -593,7 +599,7 @@ mod tests {
 
     #[test]
     fn lex_numbers() {
-        let mut lex = Lexer::new(" 0 2_000 -1,0xf -0x0 0.0 0x0.4p-34 +5");
+        let mut lex = Lexer::new(" 0 2_000 -1,0xf -0x0 0.0 0x0.4p-34 NaN +5");
         assert_eq!(lex.next(), token(Token::Integer("0"), 1));
         assert_eq!(lex.next(), token(Token::Integer("2_000"), 1));
         assert_eq!(lex.next(), token(Token::Integer("-1"), 1));
@@ -602,6 +608,7 @@ mod tests {
         assert_eq!(lex.next(), token(Token::Integer("-0x0"), 1));
         assert_eq!(lex.next(), token(Token::Float("0.0"), 1));
         assert_eq!(lex.next(), token(Token::Float("0x0.4p-34"), 1));
+        assert_eq!(lex.next(), token(Token::Float("NaN"), 1));
         assert_eq!(lex.next(), token(Token::Integer("+5"), 1));
         assert_eq!(lex.next(), None);
     }
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 01b10cac97..8b603c7870 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -896,7 +896,7 @@ impl<'a> Parser<'a> {
                 I16 => consume!(ty, self.match_imm16("Expected a 16-bit integer")?),
                 I32 => consume!(ty, self.match_imm32("Expected a 32-bit integer")?),
                 I64 => consume!(ty, self.match_imm64("Expected a 64-bit integer")?),
-                F32 => consume!(ty, self.match_ieee32("Expected a 32-bit float...")?),
+                F32 => consume!(ty, self.match_ieee32("Expected a 32-bit float")?),
                 F64 => consume!(ty, self.match_ieee64("Expected a 64-bit float")?),
                 b if b.is_bool() => consume!(
                     ty,

From d32301854ddcc4eb1812657fbbf0dadfd40e6aa5 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 28 Oct 2019 13:26:21 -0700
Subject: [PATCH 2937/3084] Add x86 SIMD implementation of float comparison

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 11 +++++
 cranelift/codegen/meta/src/isa/x86/opcodes.rs |  8 ++++
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 37 +++++++++++++++++
 .../isa/x86/simd-comparison-binemit.clif      | 26 ++++++++++++
 .../isa/x86/simd-comparison-run.clif          | 40 +++++++++++++++++++
 5 files changed, 122 insertions(+)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index f9e140eead..3d57444cce 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -610,6 +610,7 @@ pub(crate) fn define(
     let rec_null_fpr = r.recipe("null_fpr");
     let rec_pcrel_fnaddr8 = r.template("pcrel_fnaddr8");
     let rec_pcrel_gvaddr8 = r.template("pcrel_gvaddr8");
+    let rec_pfcmp = r.template("pfcmp");
     let rec_popq = r.template("popq");
     let rec_pu_id = r.template("pu_id");
     let rec_pu_id_bool = r.template("pu_id_bool");
@@ -2070,6 +2071,16 @@ pub(crate) fn define(
         e.enc_32_64_maybe_isap(inst_, rec_fa.opcodes(opcodes), *isa_predicate);
     }
 
+    // SIMD float comparisons
+    e.enc_both(
+        fcmp.bind(vector(F32, sse_vector_size)),
+        rec_pfcmp.opcodes(&CMPPS),
+    );
+    e.enc_both(
+        fcmp.bind(vector(F64, sse_vector_size)),
+        rec_pfcmp.opcodes(&CMPPD),
+    );
+
     // Reference type instructions
 
     // Null references implemented as iconst 0.
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index 8187283778..fde15899e7 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -61,6 +61,14 @@ pub static CMP_IMM8: [u8; 1] = [0x83];
 /// Compare r{16,32,64} with r/m of the same size.
 pub static CMP_REG: [u8; 1] = [0x39];
 
+/// Compare packed double-precision floating-point value in xmm2/m32 and xmm1 using bits 2:0 of
+/// imm8 as comparison predicate (SSE2).
+pub static CMPPD: [u8; 3] = [0x66, 0x0f, 0xc2];
+
+/// Compare packed single-precision floating-point value in xmm2/m32 and xmm1 using bits 2:0 of
+/// imm8 as comparison predicate (SSE).
+pub static CMPPS: [u8; 2] = [0x0f, 0xc2];
+
 /// Convert scalar double-precision floating-point value to scalar single-precision
 /// floating-point value.
 pub static CVTSD2SS: [u8; 3] = [0xf2, 0x0f, 0x5a];
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 83d30a37c2..6b5367a50e 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -3015,6 +3015,43 @@ pub(crate) fn define<'shared>(
             ),
     );
 
+    {
+        let supported_floatccs: Vec = ["eq", "lt", "le", "uno", "ne", "gt", "ge", "ord"]
+            .iter()
+            .map(|name| Literal::enumerator_for(floatcc, name))
+            .collect();
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("pfcmp", &formats.float_compare, 2)
+                .operands_in(vec![fpr, fpr])
+                .operands_out(vec![0])
+                .inst_predicate(supported_floatccs_predicate(
+                    &supported_floatccs[..],
+                    &*formats.float_compare,
+                ))
+                .emit(
+                    r#"
+                    // Comparison instruction.
+                    {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                    modrm_rr(in_reg1, in_reg0, sink);
+                    // Add immediate byte indicating what type of comparison.
+                    use crate::ir::condcodes::FloatCC::*;
+                    let imm = match cond {
+                        Equal               => 0x00,
+                        LessThan            => 0x01,
+                        LessThanOrEqual     => 0x02,
+                        Unordered           => 0x03,
+                        NotEqual            => 0x04,
+                        GreaterThanOrEqual  => 0x05,
+                        GreaterThan         => 0x06,
+                        Ordered             => 0x07,
+                        _ => panic!("{} not supported by pfcmp", cond),
+                    };
+                    sink.put1(imm);
+                "#,
+                ),
+        );
+    }
+
     recipes.add_template_recipe(
         EncodingRecipeBuilder::new("is_zero", &formats.unary, 2 + 2)
             .operands_in(vec![gpr])
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
index aecfbe1ad7..be8e7d4e8e 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
@@ -52,3 +52,29 @@ ebb0(v0: i32x4 [%xmm2], v1: i32x4 [%xmm4]):
 [-, %xmm2]  v5 = x86_pminu v0, v1     ; bin: 66 0f 38 3b d4
             return
 }
+
+function %fcmp_f32x4(f32x4, f32x4) {
+ebb0(v0: f32x4 [%xmm2], v1: f32x4 [%xmm4]):
+[-, %xmm2]  v2 = fcmp eq v0, v1     ; bin: 40 0f c2 d4 00
+[-, %xmm2]  v3 = fcmp lt v0, v1     ; bin: 40 0f c2 d4 01
+[-, %xmm2]  v4 = fcmp le v0, v1     ; bin: 40 0f c2 d4 02
+[-, %xmm2]  v5 = fcmp uno v0, v1    ; bin: 40 0f c2 d4 03
+[-, %xmm2]  v6 = fcmp ne v0, v1     ; bin: 40 0f c2 d4 04
+[-, %xmm2]  v7 = fcmp ge v0, v1     ; bin: 40 0f c2 d4 05
+[-, %xmm2]  v8 = fcmp gt v0, v1     ; bin: 40 0f c2 d4 06
+[-, %xmm2]  v9 = fcmp ord v0, v1    ; bin: 40 0f c2 d4 07
+            return
+}
+
+function %fcmp_f64x2(f64x2, f64x2) {
+ebb0(v0: f64x2 [%xmm2], v1: f64x2 [%xmm0]):
+[-, %xmm2]  v2 = fcmp eq v0, v1     ; bin: 66 40 0f c2 d0 00
+[-, %xmm2]  v3 = fcmp lt v0, v1     ; bin: 66 40 0f c2 d0 01
+[-, %xmm2]  v4 = fcmp le v0, v1     ; bin: 66 40 0f c2 d0 02
+[-, %xmm2]  v5 = fcmp uno v0, v1    ; bin: 66 40 0f c2 d0 03
+[-, %xmm2]  v6 = fcmp ne v0, v1     ; bin: 66 40 0f c2 d0 04
+[-, %xmm2]  v7 = fcmp ge v0, v1     ; bin: 66 40 0f c2 d0 05
+[-, %xmm2]  v8 = fcmp gt v0, v1     ; bin: 66 40 0f c2 d0 06
+[-, %xmm2]  v9 = fcmp ord v0, v1    ; bin: 66 40 0f c2 d0 07
+            return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
index 5d96585be0..4b9da6e4a2 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
@@ -177,3 +177,43 @@ ebb0:
     return v8
 }
 ; run
+
+function %fcmp_eq_f32x4() -> b1 {
+ebb0:
+    v0 = vconst.f32x4 [0.0 -0x4.2 0x0.33333 -0.0]
+    v1 = vconst.f32x4 [0.0 -0x4.2 0x0.33333 -0.0]
+    v2 = fcmp eq v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run
+
+function %fcmp_lt_f32x4() -> b1 {
+ebb0:
+    v0 = vconst.f32x4 [0.0      -0x4.2  0x0.0       -0.0]
+    v1 = vconst.f32x4 [0x0.001  0x4.2   0x0.33333   0x1.0]
+    v2 = fcmp lt v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run
+
+function %fcmp_ge_f64x2() -> b1 {
+ebb0:
+    v0 = vconst.f64x2 [0x0.0  0x4.2]
+    v1 = vconst.f64x2 [0.0    0x4.1]
+    v2 = fcmp ge v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run
+
+function %fcmp_uno_f64x2() -> b1 {
+ebb0:
+    v0 = vconst.f64x2 [0.0  NaN]
+    v1 = vconst.f64x2 [NaN  0x4.1]
+    v2 = fcmp uno v0, v1
+    v8 = vall_true v2
+    return v8
+}
+; run

From 81df93e6a05ffd130fb391687e44e4e5a9da189b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 28 Oct 2019 13:31:05 -0700
Subject: [PATCH 2938/3084] Translate WASM float comparisons to CLIF

---
 cranelift/wasm/src/code_translator.rs | 44 +++++++++++++++++++--------
 1 file changed, 31 insertions(+), 13 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index cab5405335..5e9b76fcdb 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1197,19 +1197,25 @@ pub fn translate_operator(
         Operator::I8x16LeU | Operator::I16x8LeU | Operator::I32x4LeU => {
             translate_vector_icmp(IntCC::UnsignedLessThanOrEqual, type_of(op), builder, state)
         }
-        Operator::F32x4Eq
-        | Operator::F32x4Ne
-        | Operator::F32x4Lt
-        | Operator::F32x4Gt
-        | Operator::F32x4Le
-        | Operator::F32x4Ge
-        | Operator::F64x2Eq
-        | Operator::F64x2Ne
-        | Operator::F64x2Lt
-        | Operator::F64x2Gt
-        | Operator::F64x2Le
-        | Operator::F64x2Ge
-        | Operator::I8x16Shl
+        Operator::F32x4Eq | Operator::F64x2Eq => {
+            translate_vector_fcmp(FloatCC::Equal, type_of(op), builder, state)
+        }
+        Operator::F32x4Ne | Operator::F64x2Ne => {
+            translate_vector_fcmp(FloatCC::NotEqual, type_of(op), builder, state)
+        }
+        Operator::F32x4Lt | Operator::F64x2Lt => {
+            translate_vector_fcmp(FloatCC::LessThan, type_of(op), builder, state)
+        }
+        Operator::F32x4Gt | Operator::F64x2Gt => {
+            translate_vector_fcmp(FloatCC::GreaterThan, type_of(op), builder, state)
+        }
+        Operator::F32x4Le | Operator::F64x2Le => {
+            translate_vector_fcmp(FloatCC::LessThanOrEqual, type_of(op), builder, state)
+        }
+        Operator::F32x4Ge | Operator::F64x2Ge => {
+            translate_vector_fcmp(FloatCC::GreaterThanOrEqual, type_of(op), builder, state)
+        }
+        Operator::I8x16Shl
         | Operator::I8x16ShrS
         | Operator::I8x16ShrU
         | Operator::I8x16Mul
@@ -1485,6 +1491,18 @@ fn translate_fcmp(cc: FloatCC, builder: &mut FunctionBuilder, state: &mut FuncTr
     state.push1(builder.ins().bint(I32, val));
 }
 
+fn translate_vector_fcmp(
+    cc: FloatCC,
+    needed_type: Type,
+    builder: &mut FunctionBuilder,
+    state: &mut FuncTranslationState,
+) {
+    let (a, b) = state.pop2();
+    let bitcast_a = optionally_bitcast_vector(a, needed_type, builder);
+    let bitcast_b = optionally_bitcast_vector(b, needed_type, builder);
+    state.push1(builder.ins().fcmp(cc, bitcast_a, bitcast_b))
+}
+
 fn translate_br_if(
     relative_depth: u32,
     builder: &mut FunctionBuilder,

From f4b021c072465a437cdab5cf97570d051da36e94 Mon Sep 17 00:00:00 2001
From: Till Schneidereit 
Date: Tue, 12 Nov 2019 16:52:48 +0100
Subject: [PATCH 2939/3084] Make it a Bytecode Alliance project (#1220)

---
 cranelift/CODE_OF_CONDUCT.md     |  11 ++-
 cranelift/CONTRIBUTING.md        |   6 +-
 cranelift/ORG_CODE_OF_CONDUCT.md | 143 +++++++++++++++++++++++++++++++
 cranelift/README.md              |   3 +
 cranelift/SECURITY.md            |  29 +++++++
 5 files changed, 186 insertions(+), 6 deletions(-)
 create mode 100644 cranelift/ORG_CODE_OF_CONDUCT.md
 create mode 100644 cranelift/SECURITY.md

diff --git a/cranelift/CODE_OF_CONDUCT.md b/cranelift/CODE_OF_CONDUCT.md
index 38ef2efa50..5c5ebdd259 100644
--- a/cranelift/CODE_OF_CONDUCT.md
+++ b/cranelift/CODE_OF_CONDUCT.md
@@ -1,5 +1,7 @@
 # Contributor Covenant Code of Conduct
 
+*Note*: this Code of Conduct pertains to individuals' behavior. Please also see the [Organizational Code of Conduct][OCoC].
+
 ## Our Pledge
 
 In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
@@ -34,13 +36,14 @@ This Code of Conduct applies both within project spaces and in public spaces whe
 
 ## Enforcement
 
-Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at either sunfish@mozilla.com or tyler@fastly.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the Bytecode Alliance CoC team at [report@bytecodealliance.org](mailto:report@bytecodealliance.org). The CoC team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The CoC team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
 
-Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the Bytecode Alliance's leadership.
 
 ## Attribution
 
 This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
 
-[homepage]: http://contributor-covenant.org
-[version]: http://contributor-covenant.org/version/1/4/
+[OCoC]: ORG_CODE_OF_CONDUCT.md
+[homepage]: https://www.contributor-covenant.org
+[version]: https://www.contributor-covenant.org/version/1/4/
diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
index e5e5b9410f..7013874fe2 100644
--- a/cranelift/CONTRIBUTING.md
+++ b/cranelift/CONTRIBUTING.md
@@ -48,9 +48,11 @@ things set in stone yet.
 
 ### Code of Conduct
 
-We abide by our [Code of Conduct] and ask that you do as well.
+Cranelift is a [Bytecode Alliance] project, and follows the Bytecode Alliance's [Code of Conduct] and [Organizational Code of Conduct].
 
-[Code of Conduct]: CODE_OF_CONDUCT.md
+[Bytecode Alliance] : https://bytecodealliance.org/
+[Code of Conduct] : CODE_OF_CONDUCT.md
+[Organizational Code of Conduct] : ORG_CODE_OF_CONDUCT.md
 
 ## Coding Guidelines
 
diff --git a/cranelift/ORG_CODE_OF_CONDUCT.md b/cranelift/ORG_CODE_OF_CONDUCT.md
new file mode 100644
index 0000000000..6f4fb3f537
--- /dev/null
+++ b/cranelift/ORG_CODE_OF_CONDUCT.md
@@ -0,0 +1,143 @@
+# Bytecode Alliance Organizational Code of Conduct (OCoC)
+
+*Note*: this Code of Conduct pertains to organizations' behavior. Please also see the [Individual Code of Conduct](CODE_OF_CONDUCT.md).
+
+## Preamble
+
+The Bytecode Alliance (BA) welcomes involvement from organizations,
+including commercial organizations.  This document is an
+*organizational* code of conduct, intended particularly to provide
+guidance to commercial organizations.  It is distinct from the
+[Individual Code of Conduct (ICoC)](CODE_OF_CONDUCT.md), and does not 
+replace the ICoC. This OCoC applies to any group of people acting in 
+concert as a BA member or as a participant in BA activities, whether 
+or not that group is formally incorporated in some jurisdiction.
+
+The code of conduct described below is not a set of rigid rules, and
+we did not write it to encompass every conceivable scenario that might
+arise.  For example, it is theoretically possible there would be times
+when asserting patents is in the best interest of the BA community as
+a whole.  In such instances, consult with the BA, strive for
+consensus, and interpret these rules with an intent that is generous
+to the community the BA serves.
+
+While we may revise these guidelines from time to time based on
+real-world experience, overall they are based on a simple principle:
+
+*Bytecode Alliance members should observe the distinction between
+ public community functions and private functions — especially
+ commercial ones — and should ensure that the latter support, or at
+ least do not harm, the former.*
+
+## Guidelines
+
+ * **Do not cause confusion about Wasm standards or interoperability.** 
+ 
+   Having an interoperable WebAssembly core is a high priority for
+   the BA, and members should strive to preserve that core.  It is fine
+   to develop additional non-standard features or APIs, but they
+   should always be clearly distinguished from the core interoperable
+   Wasm.
+ 
+   Treat the WebAssembly name and any BA-associated names with
+   respect, and follow BA trademark and branding guidelines.  If you
+   distribute a customized version of software originally produced by
+   the BA, or if you build a product or service using BA-derived
+   software, use names that clearly distinguish your work from the
+   original.  (You should still provide proper attribution to the
+   original, of course, wherever such attribution would normally be
+   given.)
+     
+   Further, do not use the WebAssembly name or BA-associated names in
+   other public namespaces in ways that could cause confusion, e.g.,
+   in company names, names of commercial service offerings, domain
+   names, publicly-visible social media accounts or online service
+   accounts, etc.  It may sometimes be reasonable, however, to
+   register such a name in a new namespace and then immediately donate
+   control of that account to the BA, because that would help the project
+   maintain its identity.
+
+   For further guidance, see the BA Trademark and Branding Policy
+   [TODO: create policy, then insert link].
+     
+ * **Do not restrict contributors.** If your company requires
+   employees or contractors to sign non-compete agreements, those
+   agreements must not prevent people from participating in the BA or
+   contributing to related projects.
+
+   This does not mean that all non-compete agreements are incompatible
+   with this code of conduct.  For example, a company may restrict an
+   employee's ability to solicit the company's customers.  However, an
+   agreement must not block any form of technical or social
+   participation in BA activities, including but not limited to the
+   implementation of particular features.
+
+   The accumulation of experience and expertise in individual persons,
+   who are ultimately free to direct their energy and attention as
+   they decide, is one of the most important drivers of progress in
+   open source projects.  A company that limits this freedom may hinder
+   the success of the BA's efforts.
+
+ * **Do not use patents as offensive weapons.** If any BA participant
+   prevents the adoption or development of BA technologies by
+   asserting its patents, that undermines the purpose of the
+   coalition.  The collaboration fostered by the BA cannot include
+   members who act to undermine its work.
+ 
+ * **Practice responsible disclosure** for security vulnerabilities.
+   Use designated, non-public reporting channels to disclose technical
+   vulnerabilities, and give the project a reasonable period to
+   respond, remediate, and patch.  [TODO: optionally include the
+   security vulnerability reporting URL here.]
+
+   Vulnerability reporters may patch their company's own offerings, as
+   long as that patching does not significantly delay the reporting of
+   the vulnerability.  Vulnerability information should never be used
+   for unilateral commercial advantage.  Vendors may legitimately
+   compete on the speed and reliability with which they deploy
+   security fixes, but withholding vulnerability information damages
+   everyone in the long run by risking harm to the BA project's
+   reputation and to the security of all users.
+
+ * **Respect the letter and spirit of open source practice.** While
+     there is not space to list here all possible aspects of standard
+     open source practice, some examples will help show what we mean:
+
+   * Abide by all applicable open source license terms.  Do not engage
+     in copyright violation or misattribution of any kind.
+
+   * Do not claim others' ideas or designs as your own.
+
+   * When others engage in publicly visible work (e.g., an upcoming
+     demo that is coordinated in a public issue tracker), do not
+     unilaterally announce early releases or early demonstrations of
+     that work ahead of their schedule in order to secure private
+     advantage (such as marketplace advantage) for yourself.
+
+   The BA reserves the right to determine what constitutes good open
+   source practices and to take action as it deems appropriate to
+   encourage, and if necessary enforce, such practices.
+
+## Enforcement
+
+Instances of organizational behavior in violation of the OCoC may 
+be reported by contacting the Bytecode Alliance CoC team at 
+[report@bytecodealliance.org](mailto:report@bytecodealliance.org). The 
+CoC team will review and investigate all complaints, and will respond 
+in a way that it deems appropriate to the circumstances. The CoC team 
+is obligated to maintain confidentiality with regard to the reporter of 
+an incident. Further details of specific enforcement policies may be 
+posted separately.
+
+When the BA deems an organization in violation of this OCoC, the BA
+will, at its sole discretion, determine what action to take.  The BA
+will decide what type, degree, and duration of corrective action is
+needed, if any, before a violating organization can be considered for
+membership (if it was not already a member) or can have its membership
+reinstated (if it was a member and the BA canceled its membership due
+to the violation).
+
+In practice, the BA's first approach will be to start a conversation,
+with punitive enforcement used only as a last resort.  Violations
+often turn out to be unintentional and swiftly correctable with all
+parties acting in good faith.
diff --git a/cranelift/README.md b/cranelift/README.md
index 171055d030..fe4e833a92 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -1,11 +1,14 @@
 Cranelift Code Generator
 ========================
 
+**A [Bytecode Alliance][BA] project**
+
 Cranelift is a low-level retargetable code generator. It translates a
 [target-independent intermediate
 representation](https://cranelift.readthedocs.io/en/latest/ir.html)
 into executable machine code.
 
+[BA]: https://bytecodealliance.org/
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
 [![Travis Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
 [![Fuzzit Status](https://app.fuzzit.dev/badge?org_id=CraneStation)](https://app.fuzzit.dev/orgs/CraneStation/dashboard)
diff --git a/cranelift/SECURITY.md b/cranelift/SECURITY.md
new file mode 100644
index 0000000000..3513b9cb35
--- /dev/null
+++ b/cranelift/SECURITY.md
@@ -0,0 +1,29 @@
+# Security Policy
+
+Building secure foundations for software development is at the core of what we do in the Bytecode Alliance. Contributions of external security researchers are a vital part of that.
+
+## Scope
+
+If you believe you've found a security issue in any website, service, or software owned or operated by the Bytecode Alliance, we encourage you to notify us.
+
+## How to Submit a Report
+
+To submit a vulnerability report to the Bytecode Alliance, please contact us at [security@bytecodealliance.org](mailto:security@bytecodealliance.org). Your submission will be reviewed and validated by a member of our security team.
+
+## Safe Harbor
+
+The Bytecode Alliance supports safe harbor for security researchers who:
+
+*   Make a good faith effort to avoid privacy violations, destruction of data, and interruption or degradation of our services.
+*   Only interact with accounts you own or with explicit permission of the account holder. If you do encounter Personally Identifiable Information (PII) contact us immediately, do not proceed with access, and immediately purge any local information.
+*   Provide us with a reasonable amount of time to resolve vulnerabilities prior to any disclosure to the public or a third-party.
+
+We will consider activities conducted consistent with this policy to constitute "authorized" conduct and will not pursue civil action or initiate a complaint to law enforcement. We will help to the extent we can if legal action is initiated by a third party against you.
+
+Please submit a report to us before engaging in conduct that may be inconsistent with or unaddressed by this policy.
+
+## Preferences
+
+*   Please provide detailed reports with reproducible steps and a clearly defined impact.
+*   Submit one vulnerability per report.
+*   Social engineering (e.g. phishing, vishing, smishing) is prohibited.

From 9080a02e10ab3c6328ed0b99fa846ea7693c65f5 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 12 Nov 2019 19:09:31 +0100
Subject: [PATCH 2940/3084] Replace CraneStation by bytecodealliance
 everywhere; (#1221)

---
 cranelift/CONTRIBUTING.md                      | 18 +++++++++---------
 cranelift/Cargo.toml                           |  2 +-
 cranelift/README.md                            | 14 +++++++-------
 cranelift/bforest/Cargo.toml                   |  4 ++--
 cranelift/codegen/Cargo.toml                   |  4 ++--
 cranelift/codegen/meta/Cargo.toml              |  4 ++--
 .../codegen/meta/src/isa/x86/encodings.rs      |  4 ++--
 cranelift/codegen/meta/src/isa/x86/legalize.rs |  2 +-
 cranelift/codegen/shared/Cargo.toml            |  2 +-
 cranelift/codegen/src/regalloc/liverange.rs    |  4 ++--
 cranelift/entity/Cargo.toml                    |  4 ++--
 cranelift/faerie/Cargo.toml                    |  4 ++--
 cranelift/filetests/Cargo.toml                 |  2 +-
 .../filetests/isa/x86/binary64-run.clif        |  2 +-
 .../filetests/regalloc/coalescing-207.clif     |  2 +-
 .../filetests/regalloc/coalescing-216.clif     |  2 +-
 .../filetests/regalloc/reload-208-bb.clif      |  2 +-
 .../filetests/regalloc/reload-208.clif         |  2 +-
 .../filetests/regalloc/reload-779.clif         |  2 +-
 .../regalloc/solver-fixedconflict-var.clif     |  2 +-
 cranelift/frontend/Cargo.toml                  |  4 ++--
 cranelift/module/Cargo.toml                    |  4 ++--
 cranelift/native/Cargo.toml                    |  4 ++--
 cranelift/object/Cargo.toml                    |  4 ++--
 cranelift/preopt/Cargo.toml                    |  4 ++--
 cranelift/reader/Cargo.toml                    |  4 ++--
 cranelift/rustc.md                             |  2 +-
 cranelift/serde/Cargo.toml                     |  4 ++--
 cranelift/simplejit/Cargo.toml                 |  4 ++--
 cranelift/simplejit/README.md                  |  2 +-
 cranelift/umbrella/Cargo.toml                  |  4 ++--
 cranelift/wasm/Cargo.toml                      |  4 ++--
 cranelift/wasm/README.md                       |  2 +-
 cranelift/wasm/src/environ/dummy.rs            |  2 +-
 cranelift/wasm/src/environ/spec.rs             |  2 +-
 35 files changed, 66 insertions(+), 66 deletions(-)

diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
index 7013874fe2..d2da273183 100644
--- a/cranelift/CONTRIBUTING.md
+++ b/cranelift/CONTRIBUTING.md
@@ -13,12 +13,12 @@ is reflected in the code or documentation yet. If you see things that seem
 missing or that don't make sense, or even that just don't work the way you
 expect them to, we're interested to hear about it!
 
-We have a [CraneStation chat on Gitter], and questions are also welcome as issues
+We have a [chat room on Gitter], and questions are also welcome as issues
 in the [Cranelift issue tracker]. Some folks also hang out in the #cranelift
 IRC channel on [irc.mozilla.org].
 
-[CraneStation chat on Gitter]: https://gitter.im/CraneStation/Lobby
-[Cranelift issue tracker]: https://github.com/CraneStation/cranelift/issues/new
+[chat room on Gitter]: https://gitter.im/CraneStation/Lobby
+[Cranelift issue tracker]: https://github.com/bytecodealliance/cranelift/issues/new
 [irc.mozilla.org]: https://wiki.mozilla.org/IRC
 
 ### Mentoring
@@ -41,10 +41,10 @@ interested in. This a good time to get involved, as there aren't a lot of
 things set in stone yet.
 
 [Rust's issue tags]: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#issue-triage
-[E-easy]: https://github.com/CraneStation/cranelift/labels/E-easy
-[E-rust]: https://github.com/CraneStation/cranelift/labels/E-rust
-[E-compiler-easy]: https://github.com/CraneStation/cranelift/labels/E-compiler-easy
-[full list of labels]: https://github.com/CraneStation/cranelift/labels
+[E-easy]: https://github.com/bytecodealliance/cranelift/labels/E-easy
+[E-rust]: https://github.com/bytecodealliance/cranelift/labels/E-rust
+[E-compiler-easy]: https://github.com/bytecodealliance/cranelift/labels/E-compiler-easy
+[full list of labels]: https://github.com/bytecodealliance/cranelift/labels
 
 ### Code of Conduct
 
@@ -73,7 +73,7 @@ which may be convenient when there are multiple versions installed.
 
 [rustfmt-preview]: https://github.com/rust-lang/rustfmt
 [rustfmt quickstart]: https://github.com/rust-lang/rustfmt#quick-start
-[format-all.sh]: https://github.com/CraneStation/cranelift/blob/master/format-all.sh
+[format-all.sh]: https://github.com/bytecodealliance/cranelift/blob/master/format-all.sh
 
 ### Rustc version support
 
@@ -150,4 +150,4 @@ review, although it is always preferred to have one.
 [issues]: https://guides.github.com/features/issues/
 [pull requests]: https://help.github.com/articles/about-pull-requests/
 [issue keywords]: https://help.github.com/articles/closing-issues-using-keywords/
-[Core Team]: https://github.com/orgs/CraneStation/people/
+[Core Team]: https://github.com/orgs/bytecodealliance/people/
diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 521037ce53..0f4c7656fc 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.49.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 publish = false
 edition = "2018"
 
diff --git a/cranelift/README.md b/cranelift/README.md
index fe4e833a92..23bf4c353e 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -10,9 +10,9 @@ into executable machine code.
 
 [BA]: https://bytecodealliance.org/
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
-[![Travis Status](https://travis-ci.org/CraneStation/cranelift.svg?branch=master)](https://travis-ci.org/CraneStation/cranelift)
-[![Fuzzit Status](https://app.fuzzit.dev/badge?org_id=CraneStation)](https://app.fuzzit.dev/orgs/CraneStation/dashboard)
-[![Gitter chat](https://badges.gitter.im/CraneStation/CraneStation.svg)](https://gitter.im/CraneStation/Lobby)
+[![Travis Status](https://travis-ci.org/bytecodealliance/cranelift.svg?branch=master)](https://travis-ci.org/bytecodealliance/cranelift)
+[![Fuzzit Status](https://app.fuzzit.dev/badge?org_id=bytecodealliance)](https://app.fuzzit.dev/orgs/bytecodealliance/dashboard)
+[![Gitter chat](https://badges.gitter.im/bytecodealliance/bytecodealliance.svg)](https://gitter.im/CraneStation/Lobby)
 ![Minimum rustc 1.37](https://img.shields.io/badge/rustc-1.37+-green.svg)
 
 For more information, see [the
@@ -21,12 +21,12 @@ documentation](https://cranelift.readthedocs.io/en/latest/?badge=latest).
 For an example of how to use the JIT, see the [SimpleJIT Demo], which
 implements a toy language.
 
-[SimpleJIT Demo]: https://github.com/CraneStation/simplejit-demo
+[SimpleJIT Demo]: https://github.com/bytecodealliance/simplejit-demo
 
 For an example of how to use Cranelift to run WebAssembly code, see
 [Wasmtime], which implements a standalone, embeddable, VM using Cranelift.
 
-[Wasmtime]: https://github.com/CraneStation/wasmtime
+[Wasmtime]: https://github.com/bytecodealliance/wasmtime
 
 Status
 ------
@@ -70,7 +70,7 @@ affected its design are:
  - [Backend for the IonMonkey JavaScript JIT compiler in
     Firefox](spidermonkey.md#phase-2-ionmonkey).
  - [Debug build backend for the Rust compiler](rustc.md).
- - [Wasmtime non-Web wasm engine](https://github.com/CraneStation/wasmtime).
+ - [Wasmtime non-Web wasm engine](https://github.com/bytecodealliance/wasmtime).
 
 Building Cranelift
 ------------------
@@ -173,4 +173,4 @@ Editor Support
 
 Editor support for working with Cranelift IR (clif) files:
 
- - Vim: https://github.com/CraneStation/cranelift.vim
+ - Vim: https://github.com/bytecodealliance/cranelift.vim
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 185972c9f0..d612572949 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.49.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 categories = ["no-std"]
 readme = "README.md"
 keywords = ["btree", "forest", "set", "map"]
@@ -16,4 +16,4 @@ cranelift-entity = { path = "../cranelift-entity", version = "0.49.0", default-f
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 3fca049803..52e4acfb4c 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.49.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 categories = ["no-std"]
 readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
@@ -70,4 +70,4 @@ basic-blocks = []
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 5d2aa01350..754c8791da 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -4,7 +4,7 @@ authors = ["The Cranelift Project Developers"]
 version = "0.49.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 readme = "README.md"
 edition = "2018"
 
@@ -14,7 +14,7 @@ cranelift-entity = { path = "../../cranelift-entity", version = "0.49.0" }
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
 
 [package.metadata.docs.rs]
 rustdoc-args = [ "--document-private-items" ]
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 3d57444cce..e2f6246ed6 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1879,7 +1879,7 @@ pub(crate) fn define(
 
     // SIMD register movement: store, load, spill, fill, regmove. All of these use encodings of
     // MOVUPS and MOVAPS from SSE (TODO ideally all of these would either use MOVAPS when we have
-    // alignment or type-specific encodings, see https://github.com/CraneStation/cranelift/issues/1039).
+    // alignment or type-specific encodings, see https://github.com/bytecodealliance/cranelift/issues/1039).
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         // Store
         let bound_store = store.bind(vector(ty, sse_vector_size)).bind(Any);
@@ -2003,7 +2003,7 @@ pub(crate) fn define(
     // allows SIMD shifts to be legalized more easily. TODO ideally this would be typed as an
     // I128x1 but restrictions on the type builder prevent this; the general idea here is that
     // the upper bits are all zeroed and do not form parts of any separate lane. See
-    // https://github.com/CraneStation/cranelift/issues/1146.
+    // https://github.com/bytecodealliance/cranelift/issues/1146.
     e.enc_both(
         bitcast.bind(vector(I64, sse_vector_size)).bind(I32),
         rec_frurm.opcodes(&MOVD_LOAD_XMM),
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index f45edd513b..07b06beba4 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -349,7 +349,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
             def!(y = splat_any8x16(x)),
             vec![
                 def!(a = scalar_to_vector(x)), // move into the lowest 8 bits of an XMM register
-                // TODO replace the following two instructions with `vconst(0)` when this is possible; see https://github.com/CraneStation/cranelift/issues/1052
+                // TODO replace the following two instructions with `vconst(0)` when this is possible; see https://github.com/bytecodealliance/cranelift/issues/1052
                 def!(b = f64const(ieee64_zero)), // zero out a different XMM register; the shuffle mask for moving the lowest byte to all other byte lanes is 0x0
                 def!(c = bitcast_f64_to_any8x16(b)), // no instruction emitted; informs the SSA that the 0 in b can be used as a vector of this type
                 def!(y = x86_pshufb(a, c)), // PSHUFB takes two XMM operands, one of which is a shuffle mask (i.e. b)
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 9237b6812f..09d2849785 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -4,7 +4,7 @@ name = "cranelift-codegen-shared"
 version = "0.49.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 readme = "README.md"
 edition = "2018"
 
diff --git a/cranelift/codegen/src/regalloc/liverange.rs b/cranelift/codegen/src/regalloc/liverange.rs
index 266c4ca09e..f49cbcc682 100644
--- a/cranelift/codegen/src/regalloc/liverange.rs
+++ b/cranelift/codegen/src/regalloc/liverange.rs
@@ -66,7 +66,7 @@
 //! Our current implementation uses a sorted array of compressed intervals, represented by their
 //! boundaries (Ebb, Inst), sorted by Ebb. This is a simple data structure, enables coalescing of
 //! intervals easily, and shows some nice performance behavior. See
-//! https://github.com/CraneStation/cranelift/issues/1084 for benchmarks against using a
+//! https://github.com/bytecodealliance/cranelift/issues/1084 for benchmarks against using a
 //! bforest::Map.
 //!
 //! ## EBB ordering
@@ -112,7 +112,7 @@
 //! the necessary API to make coalescing easy, nor does it optimize for our types' sizes.
 //!
 //! Even the specialized `bforest::Map` implementation is slower than a plain sorted
-//! array, see https://github.com/CraneStation/cranelift/issues/1084 for details.
+//! array, see https://github.com/bytecodealliance/cranelift/issues/1084 for details.
 
 use crate::entity::SparseMapValue;
 use crate::ir::{Ebb, ExpandedProgramPoint, Inst, Layout, ProgramOrder, ProgramPoint, Value};
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 9803503ffe..e402daf59d 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.49.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 categories = ["no-std"]
 readme = "README.md"
 keywords = ["entity", "set", "map"]
@@ -19,4 +19,4 @@ enable-serde = ["serde"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 66b6a4bd92..cf2783e53f 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -3,7 +3,7 @@ name = "cranelift-faerie"
 version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 documentation = "https://cranelift.readthedocs.io/"
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
@@ -24,4 +24,4 @@ features = ["std"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index b65a1c36d9..dfa66df951 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.49.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 publish = false
 edition = "2018"
 
diff --git a/cranelift/filetests/filetests/isa/x86/binary64-run.clif b/cranelift/filetests/filetests/isa/x86/binary64-run.clif
index b255770c1e..41db7e1b5a 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64-run.clif
@@ -1,7 +1,7 @@
 test run
 target x86_64
 
-; this verifies that returning b64 immediates does not result in a segmentation fault, see https://github.com/CraneStation/cranelift/issues/911
+; this verifies that returning b64 immediates does not result in a segmentation fault, see https://github.com/bytecodealliance/cranelift/issues/911
 function %test_b64() -> b64 {
 ebb0:
 [-, %r10]   v0 = bconst.b64 true
diff --git a/cranelift/filetests/filetests/regalloc/coalescing-207.clif b/cranelift/filetests/filetests/regalloc/coalescing-207.clif
index 17d33d33b3..6af9fcd144 100644
--- a/cranelift/filetests/filetests/regalloc/coalescing-207.clif
+++ b/cranelift/filetests/filetests/regalloc/coalescing-207.clif
@@ -1,7 +1,7 @@
 test regalloc
 target x86_64 haswell
 
-; Reported as https://github.com/CraneStation/cranelift/issues/207
+; Reported as https://github.com/bytecodealliance/cranelift/issues/207
 ;
 ; The coalescer creates a virtual register with two interfering values.
 function %pr207(i64 vmctx, i32, i32) -> i32 system_v {
diff --git a/cranelift/filetests/filetests/regalloc/coalescing-216.clif b/cranelift/filetests/filetests/regalloc/coalescing-216.clif
index a6a73cd2e5..b8de70160a 100644
--- a/cranelift/filetests/filetests/regalloc/coalescing-216.clif
+++ b/cranelift/filetests/filetests/regalloc/coalescing-216.clif
@@ -1,7 +1,7 @@
 test regalloc
 target x86_64 haswell
 
-; Reported as https://github.com/CraneStation/cranelift/issues/216 from the Binaryen fuzzer.
+; Reported as https://github.com/bytecodealliance/cranelift/issues/216 from the Binaryen fuzzer.
 ;
 ; The (old) coalescer creates a virtual register with two identical values.
 function %pr216(i32 [%rdi], i64 vmctx [%rsi]) -> i64 [%rax] system_v {
diff --git a/cranelift/filetests/filetests/regalloc/reload-208-bb.clif b/cranelift/filetests/filetests/regalloc/reload-208-bb.clif
index d77e4d8ebc..a93e4f9c34 100644
--- a/cranelift/filetests/filetests/regalloc/reload-208-bb.clif
+++ b/cranelift/filetests/filetests/regalloc/reload-208-bb.clif
@@ -5,7 +5,7 @@ feature "basic-blocks"
 ; regex: V=v\d+
 ; regex: EBB=ebb\d+
 
-; Filed as https://github.com/CraneStation/cranelift/issues/208
+; Filed as https://github.com/bytecodealliance/cranelift/issues/208
 ;
 ; The verifier complains about a branch argument that is not in the same virtual register as the
 ; corresponding EBB argument.
diff --git a/cranelift/filetests/filetests/regalloc/reload-208.clif b/cranelift/filetests/filetests/regalloc/reload-208.clif
index c767670252..115f67e806 100644
--- a/cranelift/filetests/filetests/regalloc/reload-208.clif
+++ b/cranelift/filetests/filetests/regalloc/reload-208.clif
@@ -4,7 +4,7 @@ feature !"basic-blocks"
 
 ; regex: V=v\d+
 
-; Filed as https://github.com/CraneStation/cranelift/issues/208
+; Filed as https://github.com/bytecodealliance/cranelift/issues/208
 ;
 ; The verifier complains about a branch argument that is not in the same virtual register as the
 ; corresponding EBB argument.
diff --git a/cranelift/filetests/filetests/regalloc/reload-779.clif b/cranelift/filetests/filetests/regalloc/reload-779.clif
index c02464a11f..f4e8e3ecff 100644
--- a/cranelift/filetests/filetests/regalloc/reload-779.clif
+++ b/cranelift/filetests/filetests/regalloc/reload-779.clif
@@ -1,7 +1,7 @@
 test compile
 target x86_64
 
-; Filed as https://github.com/CraneStation/cranelift/issues/779
+; Filed as https://github.com/bytecodealliance/cranelift/issues/779
 ;
 ; The copy_nop optimisation to reload (see Issue 773) was creating
 ; copy_nop instructions for types for which there were no encoding.
diff --git a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
index 25bcd405f8..2756ada054 100644
--- a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
+++ b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
@@ -4,7 +4,7 @@ set enable_pinned_reg=true
 target x86_64 haswell
 feature !"basic-blocks"
 
-;; Test for the issue #1123; https://github.com/CraneStation/cranelift/issues/1123
+;; Test for the issue #1123; https://github.com/bytecodealliance/cranelift/issues/1123
 
 function u0:0(i32, i32, i32, i64 vmctx) -> i64 uext system_v {
 ebb0(v0: i32, v1: i32, v2: i32, v3: i64):
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 63bdf8ec62..6ffc15d283 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -6,7 +6,7 @@ description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
 categories = ["no-std"]
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 readme = "README.md"
 edition = "2018"
 
@@ -27,4 +27,4 @@ basic-blocks = ["cranelift-codegen/basic-blocks"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 89764df3fa..31075aa494 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -3,7 +3,7 @@ name = "cranelift-module"
 version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 documentation = "https://cranelift.readthedocs.io/"
 categories = ["no-std"]
 license = "Apache-2.0 WITH LLVM-exception"
@@ -24,4 +24,4 @@ core = ["hashbrown", "cranelift-codegen/core"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 7d62dfe95b..4497e2d692 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -3,7 +3,7 @@ name = "cranelift-native"
 version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 categories = ["no-std"]
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
@@ -26,4 +26,4 @@ core = ["cranelift-codegen/core", "raw-cpuid/nightly"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 7693cde0c5..2bcf9a9940 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -3,7 +3,7 @@ name = "cranelift-object"
 version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 documentation = "https://cranelift.readthedocs.io/"
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
@@ -22,4 +22,4 @@ features = ["std"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 54c891ded4..020b644cfa 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.49.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 categories = ["no-std"]
 readme = "README.md"
 keywords = ["optimize", "compile", "compiler", "jit"]
@@ -25,4 +25,4 @@ core = ["cranelift-codegen/core"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 8083666125..9609b92407 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.49.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 readme = "README.md"
 edition = "2018"
 
@@ -15,4 +15,4 @@ target-lexicon = "0.9"
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/rustc.md b/cranelift/rustc.md
index 1541940817..a887ca5b80 100644
--- a/cranelift/rustc.md
+++ b/cranelift/rustc.md
@@ -22,7 +22,7 @@ There's plenty of work to do to achieve these goals, and if we achieve
 them, we'll have enabled a Rust compiler written entirely in Rust, and
 enabled faster Rust compile times for important use cases.
 
-See [issues tagged "rustc"](https://github.com/CraneStation/cranelift/labels/goal%3Arustc)
+See [issues tagged "rustc"](https://github.com/bytecodealliance/cranelift/labels/goal%3Arustc)
 for a list of some of the things that will be needed.
 
 With all that said, there is a potential goal beyond that, which is to
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index a56b81a9d8..7a413535d0 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -3,7 +3,7 @@ name = "cranelift-serde"
 version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
 keywords = ["webassembly", "serde"]
@@ -23,4 +23,4 @@ cranelift-reader = { path = "../cranelift-reader", version = "0.49.0" }
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 6889acbaac..2e824342ea 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -3,7 +3,7 @@ name = "cranelift-simplejit"
 version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 documentation = "https://cranelift.readthedocs.io/"
 license = "Apache-2.0 WITH LLVM-exception"
 readme = "README.md"
@@ -38,4 +38,4 @@ cranelift-entity = { path = "../cranelift-entity", version = "0.49.0" }
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/simplejit/README.md b/cranelift/simplejit/README.md
index f37d2f0e14..d4f393cb31 100644
--- a/cranelift/simplejit/README.md
+++ b/cranelift/simplejit/README.md
@@ -5,4 +5,4 @@ This crate is extremely experimental.
 
 See the [example program] for a brief overview of how to use this.
 
-[example program]: https://github.com/CraneStation/cranelift/tree/master/cranelift-simplejit/examples/simplejit-minimal.rs
+[example program]: https://github.com/bytecodealliance/cranelift/tree/master/cranelift-simplejit/examples/simplejit-minimal.rs
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index e6749eb395..2e0ef777f0 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -5,7 +5,7 @@ version = "0.49.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 categories = ["no-std"]
 readme = "README.md"
 keywords = ["compile", "compiler", "jit"]
@@ -22,4 +22,4 @@ core = ["cranelift-codegen/core", "cranelift-frontend/core"]
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 2274699820..2f0add8856 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -3,7 +3,7 @@ name = "cranelift-wasm"
 version = "0.49.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
-repository = "https://github.com/CraneStation/cranelift"
+repository = "https://github.com/bytecodealliance/cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 categories = ["no-std", "wasm"]
 readme = "README.md"
@@ -35,4 +35,4 @@ basic-blocks = ["cranelift-codegen/basic-blocks", "cranelift-frontend/basic-bloc
 
 [badges]
 maintenance = { status = "experimental" }
-travis-ci = { repository = "CraneStation/cranelift" }
+travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/wasm/README.md b/cranelift/wasm/README.md
index a554f3ac2a..7ea2f293d4 100644
--- a/cranelift/wasm/README.md
+++ b/cranelift/wasm/README.md
@@ -4,5 +4,5 @@ in-memory form of the [Cranelift IR].
 If you're looking for a complete WebAssembly implementation that uses this
 library, see [Wasmtime].
 
-[Wasmtime]: https://github.com/CraneStation/wasmtime
+[Wasmtime]: https://github.com/bytecodealliance/wasmtime
 [Cranelift IR]: https://cranelift.readthedocs.io/en/latest/ir.html
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index 2d3137f250..57668d8f48 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -3,7 +3,7 @@
 //! `FuncEnvironment`, see [wasmtime-environ] in [Wasmtime].
 //!
 //! [wasmtime-environ]: https://crates.io/crates/wasmtime-environ
-//! [Wasmtime]: https://github.com/CraneStation/wasmtime
+//! [Wasmtime]: https://github.com/bytecodealliance/wasmtime
 
 use crate::environ::{FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmResult};
 use crate::func_translator::FuncTranslator;
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index ad0a93236e..9acaa2aa47 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -4,7 +4,7 @@
 //! There are skeleton implementations of these traits in the `dummy` module, and complete
 //! implementations in [Wasmtime].
 //!
-//! [Wasmtime]: https://github.com/CraneStation/wasmtime
+//! [Wasmtime]: https://github.com/bytecodealliance/wasmtime
 
 use crate::state::{FuncTranslationState, ModuleTranslationState};
 use crate::translation_utils::{

From 04db2a9f392add1bc24af43886fd3917fa38825b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 12 Nov 2019 15:57:59 -0800
Subject: [PATCH 2941/3084] Bind constant vectors to vconst; fixes #1052
 (#1217)

---
 cranelift/codegen/meta/src/isa/x86/legalize.rs | 18 +++++++-----------
 .../filetests/isa/x86/legalize-splat.clif      |  5 ++---
 2 files changed, 9 insertions(+), 14 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 07b06beba4..bb5566dcbe 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -2,7 +2,6 @@ use crate::cdsl::ast::{constant, var, ExprBuilder, Literal};
 use crate::cdsl::instructions::{vector, Bindable, InstructionGroup};
 use crate::cdsl::types::{LaneType, ValueType};
 use crate::cdsl::xform::TransformGroupBuilder;
-use crate::shared::types::Float::F64;
 use crate::shared::types::Int::{I16, I32, I64, I8};
 use crate::shared::Definitions as SharedDefinitions;
 
@@ -30,7 +29,6 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let clz = insts.by_name("clz");
     let ctz = insts.by_name("ctz");
     let extractlane = insts.by_name("extractlane");
-    let f64const = insts.by_name("f64const");
     let fcmp = insts.by_name("fcmp");
     let fcvt_from_uint = insts.by_name("fcvt_from_uint");
     let fcvt_to_sint = insts.by_name("fcvt_to_sint");
@@ -332,7 +330,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     // SIMD
     let uimm8_zero = Literal::constant(&imm.uimm8, 0x00);
     let uimm8_one = Literal::constant(&imm.uimm8, 0x01);
-    let ieee64_zero = Literal::constant(&imm.ieee64, 0x00);
+    let u128_zeroes = constant(vec![0x00; 16]);
     let b = var("b");
     let c = var("c");
     let d = var("d");
@@ -344,15 +342,14 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     // SIMD splat: 8-bits
     for ty in ValueType::all_lane_types().filter(|t| t.lane_bits() == 8) {
         let splat_any8x16 = splat.bind(vector(ty, sse_vector_size));
-        let bitcast_f64_to_any8x16 = raw_bitcast.bind(vector(ty, sse_vector_size)).bind(F64);
         narrow.legalize(
             def!(y = splat_any8x16(x)),
             vec![
                 def!(a = scalar_to_vector(x)), // move into the lowest 8 bits of an XMM register
-                // TODO replace the following two instructions with `vconst(0)` when this is possible; see https://github.com/bytecodealliance/cranelift/issues/1052
-                def!(b = f64const(ieee64_zero)), // zero out a different XMM register; the shuffle mask for moving the lowest byte to all other byte lanes is 0x0
-                def!(c = bitcast_f64_to_any8x16(b)), // no instruction emitted; informs the SSA that the 0 in b can be used as a vector of this type
-                def!(y = x86_pshufb(a, c)), // PSHUFB takes two XMM operands, one of which is a shuffle mask (i.e. b)
+                def!(b = vconst(u128_zeroes)), // zero out a different XMM register; the shuffle mask
+                // for moving the lowest byte to all other byte lanes is 0x0
+                def!(y = x86_pshufb(a, b)), // PSHUFB takes two XMM operands, one of which is a
+                                            // shuffle mask (i.e. b)
             ],
         );
     }
@@ -466,7 +463,6 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     }
 
     // SIMD vall_true
-    let zeroes = constant(vec![0x00; 16]);
     let eq = Literal::enumerator_for(&imm.intcc, "eq");
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         let vall_true = vall_true.bind(vector(ty, sse_vector_size));
@@ -475,7 +471,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
             narrow.legalize(
                 def!(y = vall_true(x)),
                 vec![
-                    def!(a = vconst(zeroes)),
+                    def!(a = vconst(u128_zeroes)),
                     def!(c = icmp(eq, x, a)),
                     def!(d = x86_ptest(c, c)),
                     def!(y = trueif(eq, d)),
@@ -489,7 +485,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
             narrow.legalize(
                 def!(y = vall_true(x)),
                 vec![
-                    def!(a = vconst(zeroes)),
+                    def!(a = vconst(u128_zeroes)),
                     def!(b = raw_bitcast_to_int(x)),
                     def!(c = icmp(eq, b, a)),
                     def!(d = x86_ptest(c, c)),
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-splat.clif b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
index 19d61d529c..980b4b2c65 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
@@ -67,7 +67,6 @@ ebb0:
 ; nextln:     v2 = iconst.i32 42
 ; nextln:     v0 = ireduce.i8 v2
 ; nextln:     v3 = scalar_to_vector.i8x16 v0
-; nextln:     v4 = f64const 0.0
-; nextln:     v5 = raw_bitcast.i8x16 v4
-; nextln:     v1 = x86_pshufb v3, v5
+; nextln:     v4 = vconst.i8x16 0x00
+; nextln:     v1 = x86_pshufb v3, v4
 ; nextln:     return v1

From c8eb4e961297ad7557ed0365baf5194d07a74a2a Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 29 Oct 2019 09:32:19 -0700
Subject: [PATCH 2942/3084] Add x86 SIMD floating-point arithmetic

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 25 +++++++
 cranelift/codegen/meta/src/isa/x86/opcodes.rs | 56 ++++++++++++++
 .../isa/x86/simd-arithmetic-binemit.clif      | 24 ++++++
 .../isa/x86/simd-arithmetic-run.clif          | 74 +++++++++++++++++++
 4 files changed, 179 insertions(+)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index e2f6246ed6..bc0e26b87e 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -417,6 +417,8 @@ pub(crate) fn define(
     let fill = shared.by_name("fill");
     let fill_nop = shared.by_name("fill_nop");
     let floor = shared.by_name("floor");
+    let fmax = shared.by_name("fmax");
+    let fmin = shared.by_name("fmin");
     let fmul = shared.by_name("fmul");
     let fpromote = shared.by_name("fpromote");
     let fsub = shared.by_name("fsub");
@@ -2081,6 +2083,29 @@ pub(crate) fn define(
         rec_pfcmp.opcodes(&CMPPD),
     );
 
+    // SIMD float arithmetic
+    for (ty, inst, opcodes) in &[
+        (F32, fadd, &ADDPS[..]),
+        (F64, fadd, &ADDPD[..]),
+        (F32, fsub, &SUBPS[..]),
+        (F64, fsub, &SUBPD[..]),
+        (F32, fmul, &MULPS[..]),
+        (F64, fmul, &MULPD[..]),
+        (F32, fdiv, &DIVPS[..]),
+        (F64, fdiv, &DIVPD[..]),
+        (F32, fmin, &MINPS[..]),
+        (F64, fmin, &MINPD[..]),
+        (F32, fmax, &MAXPS[..]),
+        (F64, fmax, &MAXPD[..]),
+    ] {
+        let inst_ = inst.bind(vector(*ty, sse_vector_size));
+        e.enc_both(inst_, rec_fa.opcodes(opcodes));
+    }
+    for (ty, inst, opcodes) in &[(F32, sqrt, &SQRTPS[..]), (F64, sqrt, &SQRTPD[..])] {
+        let inst_ = inst.bind(vector(*ty, sse_vector_size));
+        e.enc_both(inst_, rec_furm.opcodes(opcodes));
+    }
+
     // Reference type instructions
 
     // Null references implemented as iconst 0.
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index fde15899e7..a0d9c8d9c6 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -15,6 +15,14 @@ pub static ADD_IMM: [u8; 1] = [0x81];
 /// Add sign-extended imm8 to r/m{16,32,64}.
 pub static ADD_IMM8_SIGN_EXTEND: [u8; 1] = [0x83];
 
+/// Add packed double-precision floating-point values from xmm2/mem to xmm1 and store result in  
+/// xmm1 (SSE2).
+pub static ADDPD: [u8; 3] = [0x66, 0x0f, 0x58];
+
+/// Add packed single-precision floating-point values from xmm2/mem to xmm1 and store result in  
+/// xmm1 (SSE).
+pub static ADDPS: [u8; 2] = [0x0f, 0x58];
+
 /// Add the low double-precision floating-point value from xmm2/mem to xmm1
 /// and store the result in xmm1.
 pub static ADDSD: [u8; 3] = [0xf2, 0x0f, 0x58];
@@ -93,6 +101,14 @@ pub static CVTTSS2SI: [u8; 3] = [0xf3, 0x0f, 0x2c];
 /// Unsigned divide for {16,32,64}-bit.
 pub static DIV: [u8; 1] = [0xf7];
 
+/// Divide packed double-precision floating-point values in xmm1 by packed double-precision
+/// floating-point values in xmm2/mem (SSE2).
+pub static DIVPD: [u8; 3] = [0x66, 0x0f, 0x5e];
+
+/// Divide packed single-precision floating-point values in xmm1 by packed single-precision
+/// floating-point values in xmm2/mem (SSE).
+pub static DIVPS: [u8; 2] = [0x0f, 0x5e];
+
 /// Divide low double-precision floating-point value in xmm1 by low double-precision
 /// floating-point value in xmm2/m64.
 pub static DIVSD: [u8; 3] = [0xf2, 0x0f, 0x5e];
@@ -142,6 +158,14 @@ pub static LEA: [u8; 1] = [0x8d];
 /// Count the number of leading zero bits.
 pub static LZCNT: [u8; 3] = [0xf3, 0x0f, 0xbd];
 
+/// Return the maximum packed double-precision floating-point values between xmm1 and xmm2/m128
+/// (SSE2).
+pub static MAXPD: [u8; 3] = [0x66, 0x0f, 0x5f];
+
+/// Return the maximum packed single-precision floating-point values between  xmm1 and xmm2/m128
+/// (SSE).
+pub static MAXPS: [u8; 2] = [0x0f, 0x5f];
+
 /// Return the maximum scalar double-precision floating-point value between
 /// xmm2/m64 and xmm1.
 pub static MAXSD: [u8; 3] = [0xf2, 0x0f, 0x5f];
@@ -150,6 +174,14 @@ pub static MAXSD: [u8; 3] = [0xf2, 0x0f, 0x5f];
 /// xmm2/m32 and xmm1.
 pub static MAXSS: [u8; 3] = [0xf3, 0x0f, 0x5f];
 
+/// Return the minimum packed double-precision floating-point values between xmm1 and xmm2/m128
+/// (SSE2).
+pub static MINPD: [u8; 3] = [0x66, 0x0f, 0x5d];
+
+/// Return the minimum packed single-precision floating-point values between xmm1 and xmm2/m128
+/// (SSE).
+pub static MINPS: [u8; 2] = [0x0f, 0x5d];
+
 /// Return the minimum scalar double-precision floating-point value between
 /// xmm2/m64 and xmm1.
 pub static MINSD: [u8; 3] = [0xf2, 0x0f, 0x5d];
@@ -224,6 +256,14 @@ pub static MOVZX_WORD: [u8; 2] = [0x0f, 0xb7];
 /// Unsigned multiply for {16,32,64}-bit.
 pub static MUL: [u8; 1] = [0xf7];
 
+/// Multiply packed double-precision floating-point values from xmm2/mem to xmm1 and store result
+/// in xmm1 (SSE2).
+pub static MULPD: [u8; 3] = [0x66, 0x0f, 0x59];
+
+/// Multiply packed single-precision floating-point values from xmm2/mem to xmm1 and store result
+/// in xmm1 (SSE).
+pub static MULPS: [u8; 2] = [0x0f, 0x59];
+
 /// Multiply the low double-precision floating-point value in xmm2/m64 by the
 /// low double-precision floating-point value in xmm1.
 pub static MULSD: [u8; 3] = [0xf2, 0x0f, 0x59];
@@ -474,6 +514,14 @@ pub static SBB: [u8; 1] = [0x19];
 /// Set byte if overflow (OF=1).
 pub static SET_BYTE_IF_OVERFLOW: [u8; 2] = [0x0f, 0x90];
 
+/// Compute the square root of the packed double-precision floating-point values and store the
+/// result in xmm1 (SSE2).
+pub static SQRTPD: [u8; 3] = [0x66, 0x0f, 0x51];
+
+/// Compute the square root of the packed double-precision floating-point values and store the
+/// result in xmm1 (SSE).
+pub static SQRTPS: [u8; 2] = [0x0f, 0x51];
+
 /// Compute square root of scalar double-precision floating-point value.
 pub static SQRTSD: [u8; 3] = [0xf2, 0x0f, 0x51];
 
@@ -483,6 +531,14 @@ pub static SQRTSS: [u8; 3] = [0xf3, 0x0f, 0x51];
 /// Subtract r{16,32,64} from r/m of same size.
 pub static SUB: [u8; 1] = [0x29];
 
+/// Subtract packed double-precision floating-point values in xmm2/mem from xmm1 and store result
+/// in xmm1 (SSE2).
+pub static SUBPD: [u8; 3] = [0x66, 0x0f, 0x5c];
+
+/// Subtract packed single-precision floating-point values in xmm2/mem from xmm1 and store result
+/// in xmm1 (SSE).
+pub static SUBPS: [u8; 2] = [0x0f, 0x5c];
+
 /// Subtract the low double-precision floating-point value in xmm2/m64 from xmm1
 /// and store the result in xmm1.
 pub static SUBSD: [u8; 3] = [0xf2, 0x0f, 0x5c];
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif
index cc2d7f03e1..2994d36146 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif
@@ -173,3 +173,27 @@ ebb0:
 [-, %xmm3]    v3 = usub_sat v0, v1 ; bin: 66 0f d9 dd
     return
 }
+
+function %float_arithmetic_f32x4(f32x4, f32x4) {
+ebb0(v0: f32x4 [%xmm3], v1: f32x4 [%xmm5]):
+[-, %xmm3]    v2 = fadd v0, v1      ; bin: 40 0f 58 dd
+[-, %xmm3]    v3 = fsub v0, v1      ; bin: 40 0f 5c dd
+[-, %xmm3]    v4 = fmul v0, v1      ; bin: 40 0f 59 dd
+[-, %xmm3]    v5 = fdiv v0, v1      ; bin: 40 0f 5e dd
+[-, %xmm3]    v6 = fmin v0, v1      ; bin: 40 0f 5d dd
+[-, %xmm3]    v7 = fmax v0, v1      ; bin: 40 0f 5f dd
+[-, %xmm3]    v8 = sqrt v0          ; bin: 40 0f 51 db
+    return
+}
+
+function %float_arithmetic_f64x2(f64x2, f64x2) {
+ebb0(v0: f64x2 [%xmm3], v1: f64x2 [%xmm5]):
+[-, %xmm3]    v2 = fadd v0, v1      ; bin: 66 40 0f 58 dd
+[-, %xmm3]    v3 = fsub v0, v1      ; bin: 66 40 0f 5c dd
+[-, %xmm3]    v4 = fmul v0, v1      ; bin: 66 40 0f 59 dd
+[-, %xmm3]    v5 = fdiv v0, v1      ; bin: 66 40 0f 5e dd
+[-, %xmm3]    v6 = fmin v0, v1      ; bin: 66 40 0f 5d dd
+[-, %xmm3]    v7 = fmax v0, v1      ; bin: 66 40 0f 5f dd
+[-, %xmm3]    v8 = sqrt v0          ; bin: 66 40 0f 51 db
+    return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
index 22bcf11bdd..9fa569ac28 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
@@ -153,3 +153,77 @@ ebb0:
     return v8
 }
 ; run
+
+function %add_sub_f32x4() -> b1 {
+ebb0:
+    v0 = vconst.f32x4 [0x4.2 0.0 0.0 0.0]
+    v1 = vconst.f32x4 [0x1.0 0x1.0 0x1.0 0x1.0]
+    v2 = vconst.f32x4 [0x5.2 0x1.0 0x1.0 0x1.0]
+
+    v3 = fadd v0, v1
+    v4 = fcmp eq v3, v2
+
+    v6 = fsub v2, v1
+    v7 = fcmp eq v6, v0
+
+    v8 = band v4, v7
+    v9 = vall_true v8
+    return v9
+}
+; run
+
+function %mul_div_f32x4() -> b1 {
+ebb0:
+    v0 = vconst.f32x4 [0x4.2 -0x2.1 0x2.0 0.0]
+    v1 = vconst.f32x4 [0x3.4 0x6.7 0x8.9 0xa.b]
+    v2 = vconst.f32x4 [0xd.68 -0xd.47 0x11.2 0x0.0]
+
+    v3 = fmul v0, v1
+    v4 = fcmp eq v3, v2
+
+    v6 = fdiv v2, v1
+    v7 = fcmp eq v6, v0
+
+    v8 = band v4, v7
+    v9 = vall_true v8
+    return v9
+}
+; run
+
+function %sqrt_f64x2() -> b1 {
+ebb0:
+    v0 = vconst.f64x2 [0x9.0 0x1.0]
+    v1 = sqrt v0
+    v2 = vconst.f64x2 [0x3.0 0x1.0]
+    v3 = fcmp eq v2, v1
+    v4 = vall_true v3
+    return v4
+}
+; run
+
+function %fmax_f64x2() -> b1 {
+ebb0:
+    v0 = vconst.f64x2 [-0.0 -0x1.0]
+    v1 = vconst.f64x2 [+0.0 +0x1.0]
+
+    v2 = fmax v0, v1
+    v3 = fcmp eq v2, v1
+    v4 = vall_true v3
+
+    return v4
+}
+; run
+
+
+function %fmin_f64x2() -> b1 {
+ebb0:
+    v0 = vconst.f64x2 [-0x1.0 -0x1.0]
+    v1 = vconst.f64x2 [+0.0 +0x1.0]
+
+    v2 = fmin v0, v1
+    v3 = fcmp eq v2, v0
+    v4 = vall_true v3
+
+    return v4
+}
+; run

From b425ddc52dd2bfa9a72c59fc94e5e6bdbb78608b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 29 Oct 2019 09:49:31 -0700
Subject: [PATCH 2943/3084] Translate WASM floating-point arithmetic to CLIF

---
 cranelift/wasm/src/code_translator.rs | 69 +++++++++++++++++++++------
 1 file changed, 54 insertions(+), 15 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 5e9b76fcdb..c97826a16f 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1215,6 +1215,34 @@ pub fn translate_operator(
         Operator::F32x4Ge | Operator::F64x2Ge => {
             translate_vector_fcmp(FloatCC::GreaterThanOrEqual, type_of(op), builder, state)
         }
+        Operator::F32x4Add | Operator::F64x2Add => {
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().fadd(a, b))
+        }
+        Operator::F32x4Sub | Operator::F64x2Sub => {
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().fsub(a, b))
+        }
+        Operator::F32x4Mul | Operator::F64x2Mul => {
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().fmul(a, b))
+        }
+        Operator::F32x4Div | Operator::F64x2Div => {
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().fdiv(a, b))
+        }
+        Operator::F32x4Max | Operator::F64x2Max => {
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().fmax(a, b))
+        }
+        Operator::F32x4Min | Operator::F64x2Min => {
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().fmin(a, b))
+        }
+        Operator::F32x4Sqrt | Operator::F64x2Sqrt => {
+            let a = pop1_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().sqrt(a))
+        }
         Operator::I8x16Shl
         | Operator::I8x16ShrS
         | Operator::I8x16ShrU
@@ -1222,22 +1250,8 @@ pub fn translate_operator(
         | Operator::I64x2ShrS
         | Operator::F32x4Abs
         | Operator::F32x4Neg
-        | Operator::F32x4Sqrt
-        | Operator::F32x4Add
-        | Operator::F32x4Sub
-        | Operator::F32x4Mul
-        | Operator::F32x4Div
-        | Operator::F32x4Min
-        | Operator::F32x4Max
         | Operator::F64x2Abs
         | Operator::F64x2Neg
-        | Operator::F64x2Sqrt
-        | Operator::F64x2Add
-        | Operator::F64x2Sub
-        | Operator::F64x2Mul
-        | Operator::F64x2Div
-        | Operator::F64x2Min
-        | Operator::F64x2Max
         | Operator::I32x4TruncSF32x4Sat
         | Operator::I32x4TruncUF32x4Sat
         | Operator::I64x2TruncSF64x2Sat
@@ -1700,7 +1714,7 @@ fn type_of(operator: &Operator) -> Type {
 }
 
 /// Some SIMD operations only operate on I8X16 in CLIF; this will convert them to that type by
-/// adding a raw_bitcast if necessary
+/// adding a raw_bitcast if necessary.
 fn optionally_bitcast_vector(
     value: Value,
     needed_type: Type,
@@ -1712,3 +1726,28 @@ fn optionally_bitcast_vector(
         value
     }
 }
+
+/// A helper for popping and bitcasting a single value; since SIMD values can lose their type by
+/// using v128 (i.e. CLIF's I8x16) we must re-type the values using a bitcast to avoid CLIF
+/// typing issues.
+fn pop1_with_bitcast(
+    state: &mut FuncTranslationState,
+    needed_type: Type,
+    builder: &mut FunctionBuilder,
+) -> Value {
+    optionally_bitcast_vector(state.pop1(), needed_type, builder)
+}
+
+/// A helper for popping and bitcasting two values; since SIMD values can lose their type by
+/// using v128 (i.e. CLIF's I8x16) we must re-type the values using a bitcast to avoid CLIF
+/// typing issues.
+fn pop2_with_bitcast(
+    state: &mut FuncTranslationState,
+    needed_type: Type,
+    builder: &mut FunctionBuilder,
+) -> (Value, Value) {
+    let (a, b) = state.pop2();
+    let bitcast_a = optionally_bitcast_vector(a, needed_type, builder);
+    let bitcast_b = optionally_bitcast_vector(b, needed_type, builder);
+    (bitcast_a, bitcast_b)
+}

From a2b28f947272233728a16603c3b0c85e1ef5b8d4 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 29 Oct 2019 09:59:35 -0700
Subject: [PATCH 2944/3084] Ensure vectors are bitcast to the correct type

Due to SIMD's v128 operations, vectors that may have had an explicit type (e.g. f32x4) before a v128 operation will subsequently have CLIF's v128 stand-in type: i8x16. In order for follow-on operations (that may be more stricly typed, e.g. f32x4.add) to avoid CLIF errors, we must bitcast them back to the operation type. The raw_bitcast operation used to do this emits no machine code but does incur some small compile-time cost; it would be nice to avoid this in the future.
---
 cranelift/wasm/src/code_translator.rs | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index c97826a16f..3d46aa6959 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1019,12 +1019,12 @@ pub fn translate_operator(
             state.push1(splatted)
         }
         Operator::I8x16ExtractLaneS { lane } | Operator::I16x8ExtractLaneS { lane } => {
-            let vector = optionally_bitcast_vector(state.pop1(), type_of(op), builder);
+            let vector = pop1_with_bitcast(state, type_of(op), builder);
             let extracted = builder.ins().extractlane(vector, lane.clone());
             state.push1(builder.ins().sextend(I32, extracted))
         }
         Operator::I8x16ExtractLaneU { lane } | Operator::I16x8ExtractLaneU { lane } => {
-            let vector = optionally_bitcast_vector(state.pop1(), type_of(op), builder);
+            let vector = pop1_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().extractlane(vector, lane.clone()));
             // on x86, PEXTRB zeroes the upper bits of the destination register of extractlane so uextend is elided; of course, this depends on extractlane being legalized to a PEXTRB
         }
@@ -1032,7 +1032,7 @@ pub fn translate_operator(
         | Operator::I64x2ExtractLane { lane }
         | Operator::F32x4ExtractLane { lane }
         | Operator::F64x2ExtractLane { lane } => {
-            let vector = optionally_bitcast_vector(state.pop1(), type_of(op), builder);
+            let vector = pop1_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().extractlane(vector, lane.clone()))
         }
         Operator::I8x16ReplaceLane { lane }
@@ -1054,9 +1054,7 @@ pub fn translate_operator(
             ))
         }
         Operator::V8x16Shuffle { lanes, .. } => {
-            let (vector_a, vector_b) = state.pop2();
-            let a = optionally_bitcast_vector(vector_a, I8X16, builder);
-            let b = optionally_bitcast_vector(vector_b, I8X16, builder);
+            let (a, b) = pop2_with_bitcast(state, I8X16, builder);
             let lanes = ConstantData::from(lanes.as_ref());
             let mask = builder.func.dfg.immediates.push(lanes);
             let shuffled = builder.ins().shuffle(a, b, mask);
@@ -1067,35 +1065,35 @@ pub fn translate_operator(
             // types (e.g. i8x16) for others.
         }
         Operator::I8x16Add | Operator::I16x8Add | Operator::I32x4Add | Operator::I64x2Add => {
-            let (a, b) = state.pop2();
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().iadd(a, b))
         }
         Operator::I8x16AddSaturateS | Operator::I16x8AddSaturateS => {
-            let (a, b) = state.pop2();
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().sadd_sat(a, b))
         }
         Operator::I8x16AddSaturateU | Operator::I16x8AddSaturateU => {
-            let (a, b) = state.pop2();
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().uadd_sat(a, b))
         }
         Operator::I8x16Sub | Operator::I16x8Sub | Operator::I32x4Sub | Operator::I64x2Sub => {
-            let (a, b) = state.pop2();
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().isub(a, b))
         }
         Operator::I8x16SubSaturateS | Operator::I16x8SubSaturateS => {
-            let (a, b) = state.pop2();
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().ssub_sat(a, b))
         }
         Operator::I8x16SubSaturateU | Operator::I16x8SubSaturateU => {
-            let (a, b) = state.pop2();
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().usub_sat(a, b))
         }
         Operator::I8x16Neg | Operator::I16x8Neg | Operator::I32x4Neg | Operator::I64x2Neg => {
-            let a = state.pop1();
+            let a = pop1_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().ineg(a))
         }
         Operator::I16x8Mul | Operator::I32x4Mul => {
-            let (a, b) = state.pop2();
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().imul(a, b))
         }
         Operator::V128Not => {

From 215884e9071b47b78323ce320c33df42e0c30708 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 12 Nov 2019 15:48:55 -0800
Subject: [PATCH 2945/3084] Simplify variable name: change `inst_` to `inst`

---
 cranelift/codegen/meta/src/isa/x86/encodings.rs | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index bc0e26b87e..0f380ccbfe 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -2069,8 +2069,8 @@ pub(crate) fn define(
         (I16, x86_pminu, &PMINUW[..], Some(use_sse41_simd)),
         (I32, x86_pminu, &PMINUD[..], Some(use_sse41_simd)),
     ] {
-        let inst_ = inst.bind(vector(*ty, sse_vector_size));
-        e.enc_32_64_maybe_isap(inst_, rec_fa.opcodes(opcodes), *isa_predicate);
+        let inst = inst.bind(vector(*ty, sse_vector_size));
+        e.enc_32_64_maybe_isap(inst, rec_fa.opcodes(opcodes), *isa_predicate);
     }
 
     // SIMD float comparisons
@@ -2098,12 +2098,12 @@ pub(crate) fn define(
         (F32, fmax, &MAXPS[..]),
         (F64, fmax, &MAXPD[..]),
     ] {
-        let inst_ = inst.bind(vector(*ty, sse_vector_size));
-        e.enc_both(inst_, rec_fa.opcodes(opcodes));
+        let inst = inst.bind(vector(*ty, sse_vector_size));
+        e.enc_both(inst, rec_fa.opcodes(opcodes));
     }
     for (ty, inst, opcodes) in &[(F32, sqrt, &SQRTPS[..]), (F64, sqrt, &SQRTPD[..])] {
-        let inst_ = inst.bind(vector(*ty, sse_vector_size));
-        e.enc_both(inst_, rec_furm.opcodes(opcodes));
+        let inst = inst.bind(vector(*ty, sse_vector_size));
+        e.enc_both(inst, rec_furm.opcodes(opcodes));
     }
 
     // Reference type instructions

From 37c70995a48ba77ba82a57acc38ae9a73504eee7 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 13 Nov 2019 14:42:45 -0800
Subject: [PATCH 2946/3084] Temporarily disable fuzzing until #1216 is resolved
 (#1224)

---
 cranelift/.github/workflows/main.yml | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/cranelift/.github/workflows/main.yml b/cranelift/.github/workflows/main.yml
index d39e690d6d..8a82fa594c 100644
--- a/cranelift/.github/workflows/main.yml
+++ b/cranelift/.github/workflows/main.yml
@@ -109,12 +109,13 @@ jobs:
         cargo fuzz run fuzz_translate_module fuzz/corpus/fuzz_translate_module/$fuzz_module
       env:
         RUST_BACKTRACE: 1
-      if: matrix.rust == 'nightly'
+      if: false && matrix.rust == 'nightly' # Temporarily disable fuzz tests until https://github.com/bytecodealliance/cranelift/issues/1216 is resolved
       continue-on-error: true
 
   fuzz:
     name: Fuzz Regression
     runs-on: ubuntu-latest
+    if: false # Temporarily disable fuzz tests until https://github.com/bytecodealliance/cranelift/issues/1216 is resolved
     steps:
     - uses: actions/checkout@master
     - name: Install Rust
@@ -125,7 +126,7 @@ jobs:
   fuzz_push:
     name: Fuzz (push)
     runs-on: ubuntu-latest
-    if: github.event_name == 'push'
+    if: false && github.event_name == 'push' # Temporarily disable fuzz tests until https://github.com/bytecodealliance/cranelift/issues/1216 is resolved
     steps:
     - uses: actions/checkout@master
     - name: Install Rust

From b578fd5396b900ae8e150ca4c09ad8bb15a09216 Mon Sep 17 00:00:00 2001
From: Peter Huene 
Date: Wed, 13 Nov 2019 16:22:26 -0800
Subject: [PATCH 2947/3084] Fix the cranelift-filetests build. (#1227)

This commit removes the dependency on `byteorder::ReadBytesExt`, which can't be
used from `no_std`, which is how we're building the `byteorder` crate.

The fix is to simply offset into the slice as needed rather than using a
`std::io::Cursor`.

Fixes #1225.
---
 cranelift/filetests/src/test_unwind.rs | 43 +++++++++++++-------------
 1 file changed, 21 insertions(+), 22 deletions(-)

diff --git a/cranelift/filetests/src/test_unwind.rs b/cranelift/filetests/src/test_unwind.rs
index 55fcbcdd9c..7b67ee7313 100644
--- a/cranelift/filetests/src/test_unwind.rs
+++ b/cranelift/filetests/src/test_unwind.rs
@@ -4,13 +4,12 @@
 #![cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
 
 use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
-use byteorder::{LittleEndian, ReadBytesExt};
+use byteorder::{ByteOrder, LittleEndian};
 use cranelift_codegen;
 use cranelift_codegen::ir;
 use cranelift_reader::TestCommand;
 use std::borrow::Cow;
 use std::fmt::Write;
-use std::io::Cursor;
 
 struct TestUnwind;
 
@@ -57,7 +56,7 @@ impl SubTest for TestUnwind {
 }
 
 fn print_unwind_info(text: &mut String, mem: &[u8]) {
-    let info = UnwindInfo::from_cursor(&mut Cursor::new(mem)).expect("failed to read unwind info");
+    let info = UnwindInfo::from_slice(mem);
 
     // Assert correct alignment and padding of the unwind information
     assert!(mem.len() % 4 == 0);
@@ -86,16 +85,16 @@ struct UnwindInfo {
 }
 
 impl UnwindInfo {
-    fn from_cursor(cursor: &mut Cursor<&[u8]>) -> std::io::Result {
-        let version_and_flags = cursor.read_u8()?;
-        let prologue_size = cursor.read_u8()?;
-        let unwind_code_count_raw = cursor.read_u8()?;
-        let frame_register_and_offset = cursor.read_u8()?;
+    fn from_slice(mem: &[u8]) -> Self {
+        let version_and_flags = mem[0];
+        let prologue_size = mem[1];
+        let unwind_code_count_raw = mem[2];
+        let frame_register_and_offset = mem[3];
         let mut unwind_codes = Vec::new();
 
         let mut i = 0;
         while i < unwind_code_count_raw {
-            let code = UnwindCode::from_cursor(cursor)?;
+            let code = UnwindCode::from_slice(&mem[(4 + (i * 2) as usize)..]);
 
             i += match &code.value {
                 UnwindValue::None => 1,
@@ -106,7 +105,7 @@ impl UnwindInfo {
             unwind_codes.push(code);
         }
 
-        Ok(Self {
+        Self {
             version: version_and_flags & 0x3,
             flags: (version_and_flags & 0xF8) >> 3,
             prologue_size,
@@ -114,7 +113,7 @@ impl UnwindInfo {
             frame_register: frame_register_and_offset & 0xF,
             frame_register_offset: (frame_register_and_offset & 0xF0) >> 4,
             unwind_codes,
-        })
+        }
     }
 }
 
@@ -127,35 +126,35 @@ struct UnwindCode {
 }
 
 impl UnwindCode {
-    fn from_cursor(cursor: &mut Cursor<&[u8]>) -> std::io::Result {
-        let offset = cursor.read_u8()?;
-        let op_and_info = cursor.read_u8()?;
+    fn from_slice(mem: &[u8]) -> Self {
+        let offset = mem[0];
+        let op_and_info = mem[1];
         let op = UnwindOperation::from(op_and_info & 0xF);
         let info = (op_and_info & 0xF0) >> 4;
 
         let value = match op {
             UnwindOperation::LargeStackAlloc => match info {
-                0 => UnwindValue::U16(cursor.read_u16::()?),
-                1 => UnwindValue::U32(cursor.read_u32::()?),
+                0 => UnwindValue::U16(LittleEndian::read_u16(&mem[2..])),
+                1 => UnwindValue::U32(LittleEndian::read_u32(&mem[2..])),
                 _ => panic!("unexpected stack alloc info value"),
             },
             UnwindOperation::SaveNonvolatileRegister => {
-                UnwindValue::U16(cursor.read_u16::()?)
+                UnwindValue::U16(LittleEndian::read_u16(&mem[2..]))
             }
             UnwindOperation::SaveNonvolatileRegisterFar => {
-                UnwindValue::U32(cursor.read_u32::()?)
+                UnwindValue::U32(LittleEndian::read_u32(&mem[2..]))
             }
-            UnwindOperation::SaveXmm128 => UnwindValue::U16(cursor.read_u16::()?),
-            UnwindOperation::SaveXmm128Far => UnwindValue::U32(cursor.read_u32::()?),
+            UnwindOperation::SaveXmm128 => UnwindValue::U16(LittleEndian::read_u16(&mem[2..])),
+            UnwindOperation::SaveXmm128Far => UnwindValue::U32(LittleEndian::read_u32(&mem[2..])),
             _ => UnwindValue::None,
         };
 
-        Ok(Self {
+        Self {
             offset,
             op,
             info,
             value,
-        })
+        }
     }
 }
 

From f8ae62200303b56c346a00f27d63b44d557e8dc2 Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Wed, 13 Nov 2019 18:01:13 -0700
Subject: [PATCH 2948/3084] Use a struct interface for creating and reading
 encoding bits on x86. #1156 (#1212)

---
 cranelift/codegen/meta/src/isa/x86/recipes.rs |  32 +--
 cranelift/codegen/shared/Cargo.toml           |   2 +
 cranelift/codegen/shared/src/isa/mod.rs       |   3 +
 .../shared/src/isa/x86/encoding_bits.rs       | 236 ++++++++++++++++++
 cranelift/codegen/shared/src/isa/x86/mod.rs   |   4 +
 cranelift/codegen/shared/src/lib.rs           |   5 +
 cranelift/codegen/src/isa/x86/binemit.rs      |  36 +--
 7 files changed, 273 insertions(+), 45 deletions(-)
 create mode 100644 cranelift/codegen/shared/src/isa/mod.rs
 create mode 100644 cranelift/codegen/shared/src/isa/x86/encoding_bits.rs
 create mode 100644 cranelift/codegen/shared/src/isa/x86/mod.rs

diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 6b5367a50e..6fcd0b4564 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -1,5 +1,8 @@
+//! Encoding recipes for x86/x86_64.
 use std::rc::Rc;
 
+use cranelift_codegen_shared::isa::x86::EncodingBits;
+
 use crate::cdsl::ast::Literal;
 use crate::cdsl::formats::InstructionFormat;
 use crate::cdsl::instructions::InstructionPredicate;
@@ -96,33 +99,8 @@ impl<'builder> RecipeGroup<'builder> {
 
 /// Given a sequence of opcode bytes, compute the recipe name prefix and encoding bits.
 fn decode_opcodes(op_bytes: &[u8], rrr: u16, w: u16) -> (&'static str, u16) {
-    assert!(!op_bytes.is_empty(), "at least one opcode byte");
-
-    let prefix_bytes = &op_bytes[..op_bytes.len() - 1];
-    let (name, mmpp) = match prefix_bytes {
-        [] => ("Op1", 0b000),
-        [0x66] => ("Mp1", 0b0001),
-        [0xf3] => ("Mp1", 0b0010),
-        [0xf2] => ("Mp1", 0b0011),
-        [0x0f] => ("Op2", 0b0100),
-        [0x66, 0x0f] => ("Mp2", 0b0101),
-        [0xf3, 0x0f] => ("Mp2", 0b0110),
-        [0xf2, 0x0f] => ("Mp2", 0b0111),
-        [0x0f, 0x38] => ("Op3", 0b1000),
-        [0x66, 0x0f, 0x38] => ("Mp3", 0b1001),
-        [0xf3, 0x0f, 0x38] => ("Mp3", 0b1010),
-        [0xf2, 0x0f, 0x38] => ("Mp3", 0b1011),
-        [0x0f, 0x3a] => ("Op3", 0b1100),
-        [0x66, 0x0f, 0x3a] => ("Mp3", 0b1101),
-        [0xf3, 0x0f, 0x3a] => ("Mp3", 0b1110),
-        [0xf2, 0x0f, 0x3a] => ("Mp3", 0b1111),
-        _ => {
-            panic!("unexpected opcode sequence: {:?}", op_bytes);
-        }
-    };
-
-    let opcode_byte = u16::from(op_bytes[op_bytes.len() - 1]);
-    (name, opcode_byte | (mmpp << 8) | (rrr << 12) | w << 15)
+    let enc = EncodingBits::new(op_bytes, rrr, w);
+    (enc.prefix.recipe_name_prefix(), enc.bits())
 }
 
 /// Given a snippet of Rust code (or None), replace the `PUT_OP` macro with the
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 09d2849785..2ab2bfc492 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -9,3 +9,5 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
+packed_struct = "0.3"
+packed_struct_codegen = "0.3"
diff --git a/cranelift/codegen/shared/src/isa/mod.rs b/cranelift/codegen/shared/src/isa/mod.rs
new file mode 100644
index 0000000000..4d8e485f6c
--- /dev/null
+++ b/cranelift/codegen/shared/src/isa/mod.rs
@@ -0,0 +1,3 @@
+//! Shared ISA-specific definitions.
+
+pub mod x86;
diff --git a/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs b/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs
new file mode 100644
index 0000000000..cd83e99a32
--- /dev/null
+++ b/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs
@@ -0,0 +1,236 @@
+//! Provides a named interface to the `u16` Encoding bits.
+
+use packed_struct::prelude::*;
+
+/// Named interface to the `u16` Encoding bits, representing an opcode.
+///
+/// Cranelift requires each recipe to have a single encoding size in bytes.
+/// X86 opcodes are variable length, so we use separate recipes for different
+/// styles of opcodes and prefixes. The opcode format is indicated by the
+/// recipe name prefix.
+///
+/// VEX/XOP and EVEX prefixes are not yet supported.
+/// Encodings using any of these prefixes are represented by separate recipes.
+///
+/// The encoding bits are:
+///
+/// 0-7:   The opcode byte .
+/// 8-9:   pp, mandatory prefix:
+///        00: none (Op*)
+///        01: 66   (Mp*)
+///        10: F3   (Mp*)
+///        11: F2   (Mp*)
+/// 10-11: mm, opcode map:
+///        00:         (Op1/Mp1)
+///        01: 0F      (Op2/Mp2)
+///        10: 0F 38   (Op3/Mp3)
+///        11: 0F 3A   (Op3/Mp3)
+/// 12-14  rrr, opcode bits for the ModR/M byte for certain opcodes.
+/// 15:    REX.W bit (or VEX.W/E)
+#[derive(Copy, Clone, PartialEq, PackedStruct)]
+#[packed_struct(size_bytes = "2", bit_numbering = "lsb0")]
+pub struct EncodingBits {
+    /// Instruction opcode byte, without the prefix.
+    #[packed_field(bits = "0:7")]
+    pub opcode_byte: u8,
+
+    /// Prefix kind for the instruction, as an enum.
+    #[packed_field(bits = "8:11", ty = "enum")]
+    pub prefix: OpcodePrefix,
+
+    /// Bits for the ModR/M byte for certain opcodes.
+    #[packed_field(bits = "12:14")]
+    pub rrr: Integer,
+
+    /// REX.W bit (or VEX.W/E).
+    #[packed_field(bits = "15")]
+    pub rex_w: Integer,
+}
+
+impl From for EncodingBits {
+    fn from(bits: u16) -> EncodingBits {
+        let bytes: [u8; 2] = [((bits >> 8) & 0xff) as u8, (bits & 0xff) as u8];
+        EncodingBits::unpack(&bytes).expect("failed creating EncodingBits")
+    }
+}
+
+impl EncodingBits {
+    /// Constructs a new EncodingBits from parts.
+    pub fn new(op_bytes: &[u8], rrr: u16, rex_w: u16) -> Self {
+        EncodingBits {
+            opcode_byte: op_bytes[op_bytes.len() - 1],
+            prefix: OpcodePrefix::from_opcode(op_bytes),
+            rrr: (rrr as u8).into(),
+            rex_w: (rex_w as u8).into(),
+        }
+    }
+
+    /// Returns the raw bits.
+    #[inline]
+    pub fn bits(self) -> u16 {
+        let bytes: [u8; 2] = self.pack();
+        ((bytes[0] as u16) << 8) | (bytes[1] as u16)
+    }
+
+    /// Extracts the PP bits of the OpcodePrefix.
+    #[inline]
+    pub fn pp(self) -> u8 {
+        self.prefix.to_primitive() & 0x3
+    }
+
+    /// Extracts the MM bits of the OpcodePrefix.
+    #[inline]
+    pub fn mm(self) -> u8 {
+        (self.prefix.to_primitive() >> 2) & 0x3
+    }
+}
+
+/// Opcode prefix representation.
+///
+/// The prefix type occupies four of the EncodingBits.
+#[allow(non_camel_case_types)]
+#[allow(missing_docs)]
+#[derive(Copy, Clone, Debug, Eq, PartialEq, PrimitiveEnum_u8)]
+pub enum OpcodePrefix {
+    Op1 = 0b0000,
+    Mp1_66 = 0b0001,
+    Mp1_f3 = 0b0010,
+    Mp1_f2 = 0b0011,
+    Op2_0f = 0b0100,
+    Mp2_66_0f = 0b0101,
+    Mp2_f3_0f = 0b0110,
+    Mp2_f2_0f = 0b0111,
+    Op3_0f_38 = 0b1000,
+    Mp3_66_0f_38 = 0b1001,
+    Mp3_f3_0f_38 = 0b1010,
+    Mp3_f2_0f_38 = 0b1011,
+    Op3_0f_3a = 0b1100,
+    Mp3_66_0f_3a = 0b1101,
+    Mp3_f3_0f_3a = 0b1110,
+    Mp3_f2_0f_3a = 0b1111,
+}
+
+impl From for OpcodePrefix {
+    fn from(n: u8) -> OpcodePrefix {
+        OpcodePrefix::from_primitive(n).expect("invalid OpcodePrefix")
+    }
+}
+
+impl OpcodePrefix {
+    /// Extracts the OpcodePrefix from the opcode.
+    pub fn from_opcode(op_bytes: &[u8]) -> OpcodePrefix {
+        assert!(!op_bytes.is_empty(), "at least one opcode byte");
+
+        let prefix_bytes = &op_bytes[..op_bytes.len() - 1];
+        match prefix_bytes {
+            [] => OpcodePrefix::Op1,
+            [0x66] => OpcodePrefix::Mp1_66,
+            [0xf3] => OpcodePrefix::Mp1_f3,
+            [0xf2] => OpcodePrefix::Mp1_f2,
+            [0x0f] => OpcodePrefix::Op2_0f,
+            [0x66, 0x0f] => OpcodePrefix::Mp2_66_0f,
+            [0xf3, 0x0f] => OpcodePrefix::Mp2_f3_0f,
+            [0xf2, 0x0f] => OpcodePrefix::Mp2_f2_0f,
+            [0x0f, 0x38] => OpcodePrefix::Op3_0f_38,
+            [0x66, 0x0f, 0x38] => OpcodePrefix::Mp3_66_0f_38,
+            [0xf3, 0x0f, 0x38] => OpcodePrefix::Mp3_f3_0f_38,
+            [0xf2, 0x0f, 0x38] => OpcodePrefix::Mp3_f2_0f_38,
+            [0x0f, 0x3a] => OpcodePrefix::Op3_0f_3a,
+            [0x66, 0x0f, 0x3a] => OpcodePrefix::Mp3_66_0f_3a,
+            [0xf3, 0x0f, 0x3a] => OpcodePrefix::Mp3_f3_0f_3a,
+            [0xf2, 0x0f, 0x3a] => OpcodePrefix::Mp3_f2_0f_3a,
+            _ => {
+                panic!("unexpected opcode sequence: {:?}", op_bytes);
+            }
+        }
+    }
+
+    /// Returns the recipe name prefix.
+    ///
+    /// At the moment, each similar OpcodePrefix group is given its own Recipe.
+    /// In order to distinguish them, this string is prefixed.
+    pub fn recipe_name_prefix(self) -> &'static str {
+        use OpcodePrefix::*;
+        match self {
+            Op1 => "Op1",
+            Op2_0f => "Op2",
+            Op3_0f_38 | Op3_0f_3a => "Op3",
+            Mp1_66 | Mp1_f3 | Mp1_f2 => "Mp1",
+            Mp2_66_0f | Mp2_f3_0f | Mp2_f2_0f => "Mp2",
+            Mp3_66_0f_38 | Mp3_f3_0f_38 | Mp3_f2_0f_38 => "Mp3",
+            Mp3_66_0f_3a | Mp3_f3_0f_3a | Mp3_f2_0f_3a => "Mp3",
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// Helper function for prefix_roundtrip() to avoid long lines.
+    fn test_roundtrip(p: OpcodePrefix) {
+        assert_eq!(p, OpcodePrefix::from(p.to_primitive()));
+    }
+
+    /// Tests that to/from each opcode matches.
+    #[test]
+    fn prefix_roundtrip() {
+        test_roundtrip(OpcodePrefix::Op1);
+        test_roundtrip(OpcodePrefix::Mp1_66);
+        test_roundtrip(OpcodePrefix::Mp1_f3);
+        test_roundtrip(OpcodePrefix::Mp1_f2);
+        test_roundtrip(OpcodePrefix::Op2_0f);
+        test_roundtrip(OpcodePrefix::Mp2_66_0f);
+        test_roundtrip(OpcodePrefix::Mp2_f3_0f);
+        test_roundtrip(OpcodePrefix::Mp2_f2_0f);
+        test_roundtrip(OpcodePrefix::Op3_0f_38);
+        test_roundtrip(OpcodePrefix::Mp3_66_0f_38);
+        test_roundtrip(OpcodePrefix::Mp3_f3_0f_38);
+        test_roundtrip(OpcodePrefix::Mp3_f2_0f_38);
+        test_roundtrip(OpcodePrefix::Op3_0f_3a);
+        test_roundtrip(OpcodePrefix::Mp3_66_0f_3a);
+        test_roundtrip(OpcodePrefix::Mp3_f3_0f_3a);
+        test_roundtrip(OpcodePrefix::Mp3_f2_0f_3a);
+    }
+
+    /// Tests that the opcode_byte is the lower of the EncodingBits.
+    #[test]
+    fn encodingbits_opcode_byte() {
+        let enc = EncodingBits::from(0x00ff);
+        assert_eq!(enc.opcode_byte, 0xff);
+        assert_eq!(enc.prefix.to_primitive(), 0x0);
+        assert_eq!(u8::from(enc.rrr), 0x0);
+        assert_eq!(u8::from(enc.rex_w), 0x0);
+
+        let enc = EncodingBits::from(0x00cd);
+        assert_eq!(enc.opcode_byte, 0xcd);
+    }
+
+    /// Tests that the OpcodePrefix is encoded correctly.
+    #[test]
+    fn encodingbits_prefix() {
+        let enc = EncodingBits::from(0x0c00);
+        assert_eq!(enc.opcode_byte, 0x00);
+        assert_eq!(enc.prefix.to_primitive(), 0xc);
+        assert_eq!(enc.prefix, OpcodePrefix::Op3_0f_3a);
+        assert_eq!(u8::from(enc.rrr), 0x0);
+        assert_eq!(u8::from(enc.rex_w), 0x0);
+    }
+
+    /// Tests that the REX.W bit is encoded correctly.
+    #[test]
+    fn encodingbits_rex_w() {
+        let enc = EncodingBits::from(0x8000);
+        assert_eq!(enc.opcode_byte, 0x00);
+        assert_eq!(enc.prefix.to_primitive(), 0x0);
+        assert_eq!(u8::from(enc.rrr), 0x0);
+        assert_eq!(u8::from(enc.rex_w), 0x1);
+    }
+
+    /// Tests a round-trip of EncodingBits from/to a u16 (hardcoded endianness).
+    #[test]
+    fn encodingbits_roundtrip() {
+        let bits: u16 = 0x1234;
+        assert_eq!(EncodingBits::from(bits).bits(), bits);
+    }
+}
diff --git a/cranelift/codegen/shared/src/isa/x86/mod.rs b/cranelift/codegen/shared/src/isa/x86/mod.rs
new file mode 100644
index 0000000000..fb45ae56c3
--- /dev/null
+++ b/cranelift/codegen/shared/src/isa/x86/mod.rs
@@ -0,0 +1,4 @@
+//! Shared x86-specific definitions.
+
+mod encoding_bits;
+pub use encoding_bits::*;
diff --git a/cranelift/codegen/shared/src/lib.rs b/cranelift/codegen/shared/src/lib.rs
index a2c5f3df33..771f5dfd9d 100644
--- a/cranelift/codegen/shared/src/lib.rs
+++ b/cranelift/codegen/shared/src/lib.rs
@@ -20,9 +20,14 @@
     )
 )]
 
+use packed_struct;
+#[macro_use]
+extern crate packed_struct_codegen;
+
 pub mod condcodes;
 pub mod constant_hash;
 pub mod constants;
+pub mod isa;
 
 /// Version number of this crate.
 pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/cranelift/codegen/src/isa/x86/binemit.rs b/cranelift/codegen/src/isa/x86/binemit.rs
index cda6ae5807..626301d0e2 100644
--- a/cranelift/codegen/src/isa/x86/binemit.rs
+++ b/cranelift/codegen/src/isa/x86/binemit.rs
@@ -8,6 +8,8 @@ use crate::ir::{Constant, Ebb, Function, Inst, InstructionData, JumpTable, Opcod
 use crate::isa::{RegUnit, StackBase, StackBaseMask, StackRef, TargetIsa};
 use crate::regalloc::RegDiversions;
 
+use cranelift_codegen_shared::isa::x86::EncodingBits;
+
 include!(concat!(env!("OUT_DIR"), "/binemit-x86.rs"));
 
 // Convert a stack base to the corresponding register.
@@ -65,8 +67,8 @@ fn rex3(rm: RegUnit, reg: RegUnit, index: RegUnit) -> u8 {
 // extracted from `bits`.
 fn rex_prefix(bits: u16, rex: u8, sink: &mut CS) {
     debug_assert_eq!(rex & 0xf8, BASE_REX);
-    let w = ((bits >> 15) & 1) as u8;
-    sink.put1(rex | (w << 3));
+    let w = EncodingBits::from(bits).rex_w;
+    sink.put1(rex | (u8::from(w) << 3));
 }
 
 // Emit a single-byte opcode with no REX prefix.
@@ -102,8 +104,8 @@ fn put_rexop2(bits: u16, rex: u8, sink: &mut CS) {
 // Emit single-byte opcode with mandatory prefix.
 fn put_mp1(bits: u16, rex: u8, sink: &mut CS) {
     debug_assert_eq!(bits & 0x8c00, 0, "Invalid encoding bits for Mp1*");
-    let pp = (bits >> 8) & 3;
-    sink.put1(PREFIX[(pp - 1) as usize]);
+    let enc = EncodingBits::from(bits);
+    sink.put1(PREFIX[(enc.pp() - 1) as usize]);
     debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less Mp1 encoding");
     sink.put1(bits as u8);
 }
@@ -111,8 +113,8 @@ fn put_mp1(bits: u16, rex: u8, sink: &mut CS) {
 // Emit single-byte opcode with mandatory prefix and REX.
 fn put_rexmp1(bits: u16, rex: u8, sink: &mut CS) {
     debug_assert_eq!(bits & 0x0c00, 0, "Invalid encoding bits for Mp1*");
-    let pp = (bits >> 8) & 3;
-    sink.put1(PREFIX[(pp - 1) as usize]);
+    let enc = EncodingBits::from(bits);
+    sink.put1(PREFIX[(enc.pp() - 1) as usize]);
     rex_prefix(bits, rex, sink);
     sink.put1(bits as u8);
 }
@@ -120,8 +122,8 @@ fn put_rexmp1(bits: u16, rex: u8, sink: &mut CS) {
 // Emit two-byte opcode (0F XX) with mandatory prefix.
 fn put_mp2(bits: u16, rex: u8, sink: &mut CS) {
     debug_assert_eq!(bits & 0x8c00, 0x0400, "Invalid encoding bits for Mp2*");
-    let pp = (bits >> 8) & 3;
-    sink.put1(PREFIX[(pp - 1) as usize]);
+    let enc = EncodingBits::from(bits);
+    sink.put1(PREFIX[(enc.pp() - 1) as usize]);
     debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less Mp2 encoding");
     sink.put1(0x0f);
     sink.put1(bits as u8);
@@ -130,8 +132,8 @@ fn put_mp2(bits: u16, rex: u8, sink: &mut CS) {
 // Emit two-byte opcode (0F XX) with mandatory prefix and REX.
 fn put_rexmp2(bits: u16, rex: u8, sink: &mut CS) {
     debug_assert_eq!(bits & 0x0c00, 0x0400, "Invalid encoding bits for Mp2*");
-    let pp = (bits >> 8) & 3;
-    sink.put1(PREFIX[(pp - 1) as usize]);
+    let enc = EncodingBits::from(bits);
+    sink.put1(PREFIX[(enc.pp() - 1) as usize]);
     rex_prefix(bits, rex, sink);
     sink.put1(0x0f);
     sink.put1(bits as u8);
@@ -140,24 +142,22 @@ fn put_rexmp2(bits: u16, rex: u8, sink: &mut CS) {
 // Emit three-byte opcode (0F 3[8A] XX) with mandatory prefix.
 fn put_mp3(bits: u16, rex: u8, sink: &mut CS) {
     debug_assert_eq!(bits & 0x8800, 0x0800, "Invalid encoding bits for Mp3*");
-    let pp = (bits >> 8) & 3;
-    sink.put1(PREFIX[(pp - 1) as usize]);
     debug_assert_eq!(rex, BASE_REX, "Invalid registers for REX-less Mp3 encoding");
-    let mm = (bits >> 10) & 3;
+    let enc = EncodingBits::from(bits);
+    sink.put1(PREFIX[(enc.pp() - 1) as usize]);
     sink.put1(0x0f);
-    sink.put1(OP3_BYTE2[(mm - 2) as usize]);
+    sink.put1(OP3_BYTE2[(enc.mm() - 2) as usize]);
     sink.put1(bits as u8);
 }
 
 // Emit three-byte opcode (0F 3[8A] XX) with mandatory prefix and REX
 fn put_rexmp3(bits: u16, rex: u8, sink: &mut CS) {
     debug_assert_eq!(bits & 0x0800, 0x0800, "Invalid encoding bits for Mp3*");
-    let pp = (bits >> 8) & 3;
-    sink.put1(PREFIX[(pp - 1) as usize]);
+    let enc = EncodingBits::from(bits);
+    sink.put1(PREFIX[(enc.pp() - 1) as usize]);
     rex_prefix(bits, rex, sink);
-    let mm = (bits >> 10) & 3;
     sink.put1(0x0f);
-    sink.put1(OP3_BYTE2[(mm - 2) as usize]);
+    sink.put1(OP3_BYTE2[(enc.mm() - 2) as usize]);
     sink.put1(bits as u8);
 }
 

From 569a57fa7de40fb3f4f7d4cee3fd8c20072fd5f1 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 8 Nov 2019 17:20:52 +0100
Subject: [PATCH 2949/3084] Hoist the stack alignment and Windows64 fastcall
 shadow stack space constants.

---
 cranelift/codegen/src/isa/x86/abi.rs | 64 ++++++++++++++--------------
 1 file changed, 33 insertions(+), 31 deletions(-)

diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index e316d90683..5201a11f93 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -34,6 +34,29 @@ static ARG_GPRS_WIN_FASTCALL_X64: [RU; 4] = [RU::rcx, RU::rdx, RU::r8, RU::r9];
 /// Return value registers for x86-64, when using windows fastcall
 static RET_GPRS_WIN_FASTCALL_X64: [RU; 1] = [RU::rax];
 
+/// The win64 fastcall ABI uses some shadow stack space, allocated by the caller, that can be used
+/// by the callee for temporary values.
+///
+/// [1] "Space is allocated on the call stack as a shadow store for callees to save" This shadow
+/// store contains the parameters which are passed through registers (ARG_GPRS) and is eventually
+/// used by the callee to save & restore the values of the arguments.
+///
+/// [2] https://blogs.msdn.microsoft.com/oldnewthing/20110302-00/?p=11333 "Although the x64 calling
+/// convention reserves spill space for parameters, you don’t have to use them as such"
+const WIN_SHADOW_STACK_SPACE: i32 = 32;
+
+/// Stack alignment requirement for functions.
+///
+/// 16 bytes is the perfect stack alignment, because:
+///
+/// - On Win64, "The primary exceptions are the stack pointer and malloc or alloca memory, which
+/// are aligned to 16 bytes in order to aid performance".
+/// - The original 32-bit x86 ELF ABI had a 4-byte aligned stack pointer, but newer versions use a
+/// 16-byte aligned stack pointer.
+/// - This allows using aligned loads and stores on SIMD vectors of 16 bytes that are located
+/// higher up in the stack.
+const STACK_ALIGNMENT: u32 = 16;
+
 #[derive(Clone)]
 struct Args {
     pointer_bytes: u8,
@@ -60,12 +83,10 @@ impl Args {
         isa_flags: &isa_settings::Flags,
     ) -> Self {
         let offset = if call_conv.extends_windows_fastcall() {
-            // [1] "The caller is responsible for allocating space for parameters to the callee,
-            // and must always allocate sufficient space to store four register parameters"
-            32
+            WIN_SHADOW_STACK_SPACE
         } else {
             0
-        };
+        } as u32;
 
         Self {
             pointer_bytes: bits / 8,
@@ -431,11 +452,9 @@ fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) ->
         "baldrdash does not expect cranelift to emit stack probes"
     );
 
-    // Baldrdash on 32-bit x86 always aligns its stack pointer to 16 bytes.
-    let stack_align = 16;
     let word_size = StackSize::from(isa.pointer_bytes());
     let shadow_store_size = if func.signature.call_conv.extends_windows_fastcall() {
-        32
+        WIN_SHADOW_STACK_SPACE as u32
     } else {
         0
     };
@@ -448,7 +467,7 @@ fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) ->
     func.stack_slots.push(ss);
 
     let is_leaf = func.is_leaf();
-    layout_stack(&mut func.stack_slots, is_leaf, stack_align)?;
+    layout_stack(&mut func.stack_slots, is_leaf, STACK_ALIGNMENT)?;
     Ok(())
 }
 
@@ -459,23 +478,8 @@ fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> C
         panic!("TODO: windows-fastcall: x86-32 not implemented yet");
     }
 
-    // [1] "The primary exceptions are the stack pointer and malloc or alloca memory,
-    // which are aligned to 16 bytes in order to aid performance"
-    let stack_align = 16;
-
-    let word_size = isa.pointer_bytes() as usize;
-    let reg_type = isa.pointer_type();
-
     let csrs = callee_saved_gprs_used(isa, func);
 
-    // [1] "Space is allocated on the call stack as a shadow store for callees to save"
-    // This shadow store contains the parameters which are passed through registers (ARG_GPRS)
-    // and is eventually used by the callee to save & restore the values of the arguments.
-    //
-    // [2] https://blogs.msdn.microsoft.com/oldnewthing/20110302-00/?p=11333
-    // "Although the x64 calling convention reserves spill space for parameters,
-    //  you don’t have to use them as such"
-    //
     // The reserved stack area is composed of:
     //   return address + frame pointer + all callee-saved registers + shadow space
     //
@@ -483,7 +487,7 @@ fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> C
     // instruction. Each of the others we will then push explicitly. Then we
     // will adjust the stack pointer to make room for the rest of the required
     // space for this frame.
-    const SHADOW_STORE_SIZE: i32 = 32;
+    let word_size = isa.pointer_bytes() as usize;
     let csr_stack_size = ((csrs.iter(GPR).len() + 2) * word_size) as i32;
 
     // TODO: eventually use the 32 bytes (shadow store) as spill slot. This currently doesn't work
@@ -492,14 +496,15 @@ fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> C
     func.create_stack_slot(ir::StackSlotData {
         kind: ir::StackSlotKind::IncomingArg,
         size: csr_stack_size as u32,
-        offset: Some(-(SHADOW_STORE_SIZE + csr_stack_size)),
+        offset: Some(-(WIN_SHADOW_STACK_SPACE + csr_stack_size)),
     });
 
     let is_leaf = func.is_leaf();
-    let total_stack_size = layout_stack(&mut func.stack_slots, is_leaf, stack_align)? as i32;
+    let total_stack_size = layout_stack(&mut func.stack_slots, is_leaf, STACK_ALIGNMENT)? as i32;
     let local_stack_size = i64::from(total_stack_size - csr_stack_size);
 
     // Add CSRs to function signature
+    let reg_type = isa.pointer_type();
     let fp_arg = ir::AbiParam::special_reg(
         reg_type,
         ir::ArgumentPurpose::FramePointer,
@@ -528,12 +533,8 @@ fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> C
 
 /// Insert a System V-compatible prologue and epilogue.
 fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> CodegenResult<()> {
-    // The original 32-bit x86 ELF ABI had a 4-byte aligned stack pointer, but
-    // newer versions use a 16-byte aligned stack pointer.
-    let stack_align = 16;
     let pointer_width = isa.triple().pointer_width().unwrap();
     let word_size = pointer_width.bytes() as usize;
-    let reg_type = ir::Type::int(u16::from(pointer_width.bits())).unwrap();
 
     let csrs = callee_saved_gprs_used(isa, func);
 
@@ -552,10 +553,11 @@ fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> C
     });
 
     let is_leaf = func.is_leaf();
-    let total_stack_size = layout_stack(&mut func.stack_slots, is_leaf, stack_align)? as i32;
+    let total_stack_size = layout_stack(&mut func.stack_slots, is_leaf, STACK_ALIGNMENT)? as i32;
     let local_stack_size = i64::from(total_stack_size - csr_stack_size);
 
     // Add CSRs to function signature
+    let reg_type = ir::Type::int(u16::from(pointer_width.bits())).unwrap();
     let fp_arg = ir::AbiParam::special_reg(
         reg_type,
         ir::ArgumentPurpose::FramePointer,

From f7c7245b062be9aab40b36bd8d1f090abde45807 Mon Sep 17 00:00:00 2001
From: krk 
Date: Wed, 13 Nov 2019 20:59:33 +0100
Subject: [PATCH 2950/3084] Add legalization for popcnt.i128 via narrowing,
 fixes #1116

---
 cranelift/codegen/meta/src/shared/legalize.rs | 11 +++++++
 .../filetests/legalizer/popcnt-i128.clif      | 31 +++++++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100644 cranelift/filetests/filetests/legalizer/popcnt-i128.clif

diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 05ffc99dfb..2f20e5a613 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -275,6 +275,17 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
         ],
     );
 
+    narrow.legalize(
+        def!(a = popcnt.I128(x)),
+        vec![
+            def!((xl, xh) = isplit(x)),
+            def!(e1 = popcnt(xl)),
+            def!(e2 = popcnt(xh)),
+            def!(e3 = iadd(e1, e2)),
+            def!(a = uextend(e3)),
+        ],
+    );
+
     // TODO(ryzokuken): benchmark this and decide if branching is a faster
     // approach than evaluating boolean expressions.
 
diff --git a/cranelift/filetests/filetests/legalizer/popcnt-i128.clif b/cranelift/filetests/filetests/legalizer/popcnt-i128.clif
new file mode 100644
index 0000000000..b241c640a9
--- /dev/null
+++ b/cranelift/filetests/filetests/legalizer/popcnt-i128.clif
@@ -0,0 +1,31 @@
+test legalizer
+target i686
+
+function %foo() -> i128 {
+ebb0:
+    v1 = iconst.i64 0x6400000042
+    v2 = iconst.i64 0x7F10100042
+    v3 = iconcat v1, v2
+    v4 = popcnt.i128 v3
+    return v4
+}
+
+; check: v5 = iconst.i32 66
+; check: v6 = iconst.i32 100
+; check: v1 = iconcat v5, v6
+; check: v7 = iconst.i32 0x1010_0042
+; check: v8 = iconst.i32 127
+; check: v2 = iconcat v7, v8
+; check: v3 = iconcat v1, v2
+; check: v9 = popcnt v1
+; check: v10 = popcnt v2
+; check: v12, v13 = isplit v9
+; check: v14, v15 = isplit v10
+; check: v16, v17 = iadd_ifcout v12, v14
+; check: v18 = iadd_ifcin v13, v15, v17
+; check: v11 = iconcat v16, v18
+; check: v4 = uextend.i128 v11
+; check: v19, v20 = isplit v4
+; check: v21, v22 = isplit v19
+; check: v23, v24 = isplit v20
+; check: return v21, v22, v23, v24

From 4ccacde3d44ebfa5f134a2394dcf6304c6ea115d Mon Sep 17 00:00:00 2001
From: XAMPPRocky <4464295+XAMPPRocky@users.noreply.github.com>
Date: Fri, 15 Nov 2019 14:43:20 +0100
Subject: [PATCH 2951/3084] Fix rendered links in CONTRIBUTING.md

---
 cranelift/CONTRIBUTING.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
index d2da273183..668e12dd05 100644
--- a/cranelift/CONTRIBUTING.md
+++ b/cranelift/CONTRIBUTING.md
@@ -50,9 +50,9 @@ things set in stone yet.
 
 Cranelift is a [Bytecode Alliance] project, and follows the Bytecode Alliance's [Code of Conduct] and [Organizational Code of Conduct].
 
-[Bytecode Alliance] : https://bytecodealliance.org/
-[Code of Conduct] : CODE_OF_CONDUCT.md
-[Organizational Code of Conduct] : ORG_CODE_OF_CONDUCT.md
+[Bytecode Alliance]: https://bytecodealliance.org/
+[Code of Conduct]: CODE_OF_CONDUCT.md
+[Organizational Code of Conduct]: ORG_CODE_OF_CONDUCT.md
 
 ## Coding Guidelines
 

From cd4d7ca0a1a59ad4922eebf7e702500503338c51 Mon Sep 17 00:00:00 2001
From: Projjal Chanda 
Date: Fri, 15 Nov 2019 19:02:24 +0530
Subject: [PATCH 2952/3084] Merged scalar and simd logical operator

---
 cranelift/wasm/src/code_translator.rs | 18 +++---------------
 1 file changed, 3 insertions(+), 15 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 3d46aa6959..b871c9f23a 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -786,15 +786,15 @@ pub fn translate_operator(
             let (arg1, arg2) = state.pop2();
             state.push1(builder.ins().iadd(arg1, arg2));
         }
-        Operator::I32And | Operator::I64And => {
+        Operator::I32And | Operator::I64And | Operator::V128And => {
             let (arg1, arg2) = state.pop2();
             state.push1(builder.ins().band(arg1, arg2));
         }
-        Operator::I32Or | Operator::I64Or => {
+        Operator::I32Or | Operator::I64Or | Operator::V128Or => {
             let (arg1, arg2) = state.pop2();
             state.push1(builder.ins().bor(arg1, arg2));
         }
-        Operator::I32Xor | Operator::I64Xor => {
+        Operator::I32Xor | Operator::I64Xor | Operator::V128Xor => {
             let (arg1, arg2) = state.pop2();
             state.push1(builder.ins().bxor(arg1, arg2));
         }
@@ -1100,18 +1100,6 @@ pub fn translate_operator(
             let a = state.pop1();
             state.push1(builder.ins().bnot(a));
         }
-        Operator::V128And => {
-            let (a, b) = state.pop2();
-            state.push1(builder.ins().band(a, b));
-        }
-        Operator::V128Or => {
-            let (a, b) = state.pop2();
-            state.push1(builder.ins().bor(a, b));
-        }
-        Operator::V128Xor => {
-            let (a, b) = state.pop2();
-            state.push1(builder.ins().bxor(a, b));
-        }
         Operator::I16x8Shl | Operator::I32x4Shl | Operator::I64x2Shl => {
             let (a, b) = state.pop2();
             let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder);

From 6519a43b0861d5aa3898c80df0ffece39cc8e785 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 11 Nov 2019 11:01:35 -0800
Subject: [PATCH 2953/3084] Add x86 SIMD floating-point negation

---
 cranelift/codegen/meta/src/isa/x86/legalize.rs    | 11 +++++++++++
 .../isa/x86/simd-arithmetic-legalize.clif         | 15 +++++++++++++++
 .../filetests/isa/x86/simd-arithmetic-run.clif    | 14 +++++++++++++-
 3 files changed, 39 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index bb5566dcbe..753ee10b21 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -2,6 +2,7 @@ use crate::cdsl::ast::{constant, var, ExprBuilder, Literal};
 use crate::cdsl::instructions::{vector, Bindable, InstructionGroup};
 use crate::cdsl::types::{LaneType, ValueType};
 use crate::cdsl::xform::TransformGroupBuilder;
+use crate::shared::types::Float::{F32, F64};
 use crate::shared::types::Int::{I16, I32, I64, I8};
 use crate::shared::Definitions as SharedDefinitions;
 
@@ -37,6 +38,8 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let fcvt_to_uint_sat = insts.by_name("fcvt_to_uint_sat");
     let fmax = insts.by_name("fmax");
     let fmin = insts.by_name("fmin");
+    let fneg = insts.by_name("fneg");
+    let fsub = insts.by_name("fsub");
     let iadd = insts.by_name("iadd");
     let icmp = insts.by_name("icmp");
     let iconst = insts.by_name("iconst");
@@ -543,6 +546,14 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         narrow.legalize(def!(c = icmp_(ule, a, b)), vec![def!(c = icmp(uge, b, a))]);
     }
 
+    for ty in &[F32, F64] {
+        let fneg = fneg.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(
+            def!(b = fneg(a)),
+            vec![def!(c = vconst(u128_zeroes)), def!(b = fsub(c, a))],
+        );
+    }
+
     narrow.custom_legalize(shuffle, "convert_shuffle");
     narrow.custom_legalize(extractlane, "convert_extractlane");
     narrow.custom_legalize(insertlane, "convert_insertlane");
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
index 6155204899..3a2ae10ab9 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
@@ -34,3 +34,18 @@ ebb0:
 
     return
 }
+
+function %fneg_legalized() {
+ebb0:
+    v0 = vconst.f32x4 [0x1.0 0x2.0 0x3.0 0x4.0]
+    v1 = fneg v0
+    ; check: v4 = vconst.f32x4 0x00
+    ; nextln: v1 = fsub v4, v0
+
+    v2 = vconst.f64x2 [0x1.0 0x2.0]
+    v3 = fneg v2
+    ; check: v5 = vconst.f64x2 0x00
+    ; nextln: v3 = fsub v5, v2
+
+    return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
index 9fa569ac28..fd52a5a52b 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
@@ -214,7 +214,6 @@ ebb0:
 }
 ; run
 
-
 function %fmin_f64x2() -> b1 {
 ebb0:
     v0 = vconst.f64x2 [-0x1.0 -0x1.0]
@@ -227,3 +226,16 @@ ebb0:
     return v4
 }
 ; run
+
+function %fneg_f64x2() -> b1 {
+ebb0:
+    v0 = vconst.f64x2 [0x1.0 -0x1.0]
+    v1 = fneg v0
+
+    v2 = vconst.f64x2 [-0x1.0 0x1.0]
+    v3 = fcmp eq v1, v2
+    v4 = vall_true v3
+
+    return v4
+}
+; run

From 1f17e35e9529254040af7508640605f11a5fa9d6 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 11 Nov 2019 11:55:04 -0800
Subject: [PATCH 2954/3084] Add x86 SIMD immediate shifts

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 13 +++++
 cranelift/codegen/meta/src/isa/x86/opcodes.rs | 12 +++++
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 20 +++++++
 .../isa/x86/simd-bitwise-binemit.clif         | 54 +++++++++++++++++++
 .../filetests/isa/x86/simd-bitwise-run.clif   | 36 +++++++++++++
 5 files changed, 135 insertions(+)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 0f380ccbfe..002963035f 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -553,6 +553,7 @@ pub(crate) fn define(
     let rec_copysp = r.template("copysp");
     let rec_div = r.template("div");
     let rec_debugtrap = r.recipe("debugtrap");
+    let rec_f_ib = r.template("f_ib");
     let rec_f32imm_z = r.template("f32imm_z");
     let rec_f64imm_z = r.template("f64imm_z");
     let rec_fa = r.template("fa");
@@ -2033,6 +2034,18 @@ pub(crate) fn define(
         e.enc_32_64(x86_psra, rec_fa.opcodes(*opcodes));
     }
 
+    // SIMD immediate shift
+    for (ty, opcodes) in &[(I16, &PS_W_IMM), (I32, &PS_D_IMM), (I64, &PS_Q_IMM)] {
+        let ishl_imm = ishl_imm.bind(vector(*ty, sse_vector_size));
+        e.enc_32_64(ishl_imm, rec_f_ib.opcodes(*opcodes).rrr(6));
+
+        let ushr_imm = ushr_imm.bind(vector(*ty, sse_vector_size));
+        e.enc_32_64(ushr_imm, rec_f_ib.opcodes(*opcodes).rrr(2));
+
+        let sshr_imm = sshr_imm.bind(vector(*ty, sse_vector_size));
+        e.enc_32_64(sshr_imm, rec_f_ib.opcodes(*opcodes).rrr(4));
+    }
+
     // SIMD integer comparisons
     {
         use IntCC::*;
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index a0d9c8d9c6..9006ce92cf 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -431,6 +431,18 @@ pub static PSHUFB: [u8; 4] = [0x66, 0x0f, 0x38, 0x00];
 /// store the result in xmm1 (SSE2).
 pub static PSHUFD: [u8; 3] = [0x66, 0x0f, 0x70];
 
+/// Shift words in xmm1 by imm8; the direction and sign-bit behavior is controlled by the RRR
+/// digit used in the ModR/M byte (SSE2).
+pub static PS_W_IMM: [u8; 3] = [0x66, 0x0f, 0x71];
+
+/// Shift doublewords in xmm1 by imm8; the direction and sign-bit behavior is controlled by the RRR
+/// digit used in the ModR/M byte (SSE2).
+pub static PS_D_IMM: [u8; 3] = [0x66, 0x0f, 0x72];
+
+/// Shift quadwords in xmm1 by imm8; the direction and sign-bit behavior is controlled by the RRR
+/// digit used in the ModR/M byte (SSE2).
+pub static PS_Q_IMM: [u8; 3] = [0x66, 0x0f, 0x73];
+
 /// Shift words in xmm1 left by xmm2/m128 while shifting in 0s (SSE2).
 pub static PSLLW: [u8; 3] = [0x66, 0x0f, 0xf1];
 
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 6fcd0b4564..8d271238ea 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -792,6 +792,26 @@ pub(crate) fn define<'shared>(
                 ),
         );
 
+        recipes.add_template_recipe(
+            EncodingRecipeBuilder::new("f_ib", &formats.binary_imm, 2)
+                .operands_in(vec![fpr])
+                .operands_out(vec![0])
+                .inst_predicate(InstructionPredicate::new_is_signed_int(
+                    &*formats.binary_imm,
+                    "imm",
+                    8,
+                    0,
+                ))
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                        modrm_r_bits(in_reg0, bits, sink);
+                        let imm: i64 = imm.into();
+                        sink.put1(imm as u8);
+                    "#,
+                ),
+        );
+
         // XX /n id with 32-bit immediate sign-extended.
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("r_id", &formats.binary_imm, 5)
diff --git a/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif
index 2a6530f7b5..af8796863c 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif
@@ -49,3 +49,57 @@ ebb0(v0: i32x4 [%xmm4], v1: i64x2 [%xmm0]):
 [-, %xmm4]  v2 = x86_psra v0, v1      ; bin: 66 0f e2 e0
             return v2
 }
+
+function %ishl_imm_i16x8(i16x8) -> i16x8 {
+ebb0(v0: i16x8 [%xmm2]):
+[-, %xmm2]  v2 = ishl_imm v0, 3      ; bin: 66 0f 71 f2 03
+            return v2
+}
+
+function %ishl_imm_i32x4(i32x4) -> i32x4 {
+ebb0(v0: i32x4 [%xmm4]):
+[-, %xmm4]  v2 = ishl_imm v0, 10     ; bin: 66 0f 72 f4 0a
+            return v2
+}
+
+function %ishl_imm_i64x2(i64x2) -> i64x2 {
+ebb0(v0: i64x2 [%xmm6]):
+[-, %xmm6]  v2 = ishl_imm v0, 42     ; bin: 66 0f 73 f6 2a
+            return v2
+}
+
+function %ushr_imm_i16x8(i16x8) -> i16x8 {
+ebb0(v0: i16x8 [%xmm2]):
+[-, %xmm2]  v2 = ushr_imm v0, 3      ; bin: 66 0f 71 d2 03
+            return v2
+}
+
+function %ushr_imm_i32x4(i32x4) -> i32x4 {
+ebb0(v0: i32x4 [%xmm4]):
+[-, %xmm4]  v2 = ushr_imm v0, 10     ; bin: 66 0f 72 d4 0a
+            return v2
+}
+
+function %ushr_imm_i64x2(i64x2) -> i64x2 {
+ebb0(v0: i64x2 [%xmm6]):
+[-, %xmm6]  v2 = ushr_imm v0, 42     ; bin: 66 0f 73 d6 2a
+            return v2
+}
+
+function %sshr_imm_i16x8(i16x8) -> i16x8 {
+ebb0(v0: i16x8 [%xmm2]):
+[-, %xmm2]  v2 = sshr_imm v0, 3      ; bin: 66 0f 71 e2 03
+            return v2
+}
+
+function %sshr_imm_i32x4(i32x4) -> i32x4 {
+ebb0(v0: i32x4 [%xmm4]):
+[-, %xmm4]  v2 = sshr_imm v0, 10     ; bin: 66 0f 72 e4 0a
+            return v2
+}
+
+function %sshr_imm_i64x2(i64x2) -> i64x2 {
+ebb0(v0: i64x2 [%xmm6]):
+[-, %xmm6]  v2 = sshr_imm v0, 42     ; bin: 66 0f 73 e6 2a
+            return v2
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif b/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
index 0c6eac6a10..8ca92a756f 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
@@ -127,3 +127,39 @@ ebb0:
     return v11
 }
 ; run
+
+function %sshr_imm_i32x4() -> b1 {
+ebb0:
+    v1 = vconst.i32x4 [1 2 4 -8]
+    v2 = sshr_imm v1, 1
+
+    v3 = vconst.i32x4 [0 1 2 -4]
+    v4 = icmp eq v2, v3
+    v5 = vall_true v4
+    return v5
+}
+; run
+
+function %sshr_imm_i16x8() -> b1 {
+ebb0:
+    v1 = vconst.i16x8 [1 2 4 -8 0 0 0 0]
+    v2 = ushr_imm v1, 1
+
+    v3 = vconst.i16x8 [0 1 2 32764 0 0 0 0] ; -4 with MSB unset == 32764
+    v4 = icmp eq v2, v3
+    v5 = vall_true v4
+    return v5
+}
+; run
+
+function %ishl_imm_i64x2() -> b1 {
+ebb0:
+    v1 = vconst.i64x2 [1 0]
+    v2 = ishl_imm v1, 1
+
+    v3 = vconst.i64x2 [2 0]
+    v4 = icmp eq v2, v3
+    v5 = vall_true v4
+    return v5
+}
+; run

From 91d29c09d06565849842ae193b6841b099a1d830 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 11 Nov 2019 12:47:01 -0800
Subject: [PATCH 2955/3084] Add x86 SIMD floating-point absolute value

---
 .../codegen/meta/src/isa/x86/legalize.rs      | 19 +++++++++++++++++++
 .../isa/x86/simd-arithmetic-legalize.clif     | 11 +++++++++++
 .../isa/x86/simd-arithmetic-run.clif          | 13 +++++++++++++
 3 files changed, 43 insertions(+)

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 753ee10b21..2728cad5cf 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -36,6 +36,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let fcvt_to_uint = insts.by_name("fcvt_to_uint");
     let fcvt_to_sint_sat = insts.by_name("fcvt_to_sint_sat");
     let fcvt_to_uint_sat = insts.by_name("fcvt_to_uint_sat");
+    let fabs = insts.by_name("fabs");
     let fmax = insts.by_name("fmax");
     let fmin = insts.by_name("fmin");
     let fneg = insts.by_name("fneg");
@@ -337,6 +338,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let b = var("b");
     let c = var("c");
     let d = var("d");
+    let e = var("e");
 
     // SIMD vector size: eventually multiple vector sizes may be supported but for now only SSE-sized vectors are available
     let sse_vector_size: u64 = 128;
@@ -554,6 +556,23 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         );
     }
 
+    // SIMD fabs
+    for ty in &[F32, F64] {
+        let fabs = fabs.bind(vector(*ty, sse_vector_size));
+        let lane_type_as_int = LaneType::int_from_bits(LaneType::from(*ty).lane_bits() as u16);
+        let vconst = vconst.bind(vector(lane_type_as_int, sse_vector_size));
+        let bitcast_to_float = raw_bitcast.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(
+            def!(b = fabs(a)),
+            vec![
+                def!(c = vconst(ones)),
+                def!(d = ushr_imm(c, uimm8_one)), // Create a mask of all 1s except the MSB.
+                def!(e = bitcast_to_float(d)),    // Cast mask to the floating-point type.
+                def!(b = band(a, e)),             // Unset the MSB.
+            ],
+        );
+    }
+
     narrow.custom_legalize(shuffle, "convert_shuffle");
     narrow.custom_legalize(extractlane, "convert_extractlane");
     narrow.custom_legalize(insertlane, "convert_insertlane");
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
index 3a2ae10ab9..324027741b 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
@@ -49,3 +49,14 @@ ebb0:
 
     return
 }
+
+function %fabs_legalized() {
+ebb0:
+    v0 = vconst.f64x2 [0x1.0 -0x2.0]
+    v1 = fabs v0
+    ; check: v2 = vconst.i64x2 0xffffffffffffffffffffffffffffffff
+    ; nextln: v3 = ushr_imm v2, 1
+    ; nextln: v4 = raw_bitcast.f64x2 v3
+    ; nextln: v1 = band v0, v4
+    return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
index fd52a5a52b..04facb0078 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
@@ -239,3 +239,16 @@ ebb0:
     return v4
 }
 ; run
+
+function %fabs_f32x4() -> b1 {
+ebb0:
+    v0 = vconst.f32x4 [0x0.0 -0x1.0 0x2.0 -0x3.0]
+    v1 = fabs v0
+
+    v2 = vconst.f32x4 [0x0.0 0x1.0 0x2.0 0x3.0]
+    v3 = fcmp eq v1, v2
+    v4 = vall_true v3
+
+    return v4
+}
+; run

From f72a5d9caac94b2e87c5346153f7f0575b250545 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 11 Nov 2019 12:50:06 -0800
Subject: [PATCH 2956/3084] Translate WASM floating-point `abs` and `neg` to
 CLIF

---
 cranelift/wasm/src/code_translator.rs | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index b871c9f23a..2f645f5009 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1229,15 +1229,19 @@ pub fn translate_operator(
             let a = pop1_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().sqrt(a))
         }
+        Operator::F32x4Neg | Operator::F64x2Neg => {
+            let a = pop1_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().fneg(a))
+        }
+        Operator::F32x4Abs | Operator::F64x2Abs => {
+            let a = pop1_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().fabs(a))
+        }
         Operator::I8x16Shl
         | Operator::I8x16ShrS
         | Operator::I8x16ShrU
         | Operator::I8x16Mul
         | Operator::I64x2ShrS
-        | Operator::F32x4Abs
-        | Operator::F32x4Neg
-        | Operator::F64x2Abs
-        | Operator::F64x2Neg
         | Operator::I32x4TruncSF32x4Sat
         | Operator::I32x4TruncUF32x4Sat
         | Operator::I64x2TruncSF64x2Sat

From d804ab8b92e9f7fe992dc41edc3a542b096f9b36 Mon Sep 17 00:00:00 2001
From: iximeow 
Date: Mon, 18 Nov 2019 10:18:38 -0800
Subject: [PATCH 2957/3084] Track frame layout changes. (#1204)

* Track frame layout changes.
---
 cranelift/codegen/src/ir/framelayout.rs |  70 ++++++++
 cranelift/codegen/src/ir/function.rs    |  12 +-
 cranelift/codegen/src/ir/mod.rs         |   2 +
 cranelift/codegen/src/isa/x86/abi.rs    | 227 ++++++++++++++++++++++--
 4 files changed, 298 insertions(+), 13 deletions(-)
 create mode 100644 cranelift/codegen/src/ir/framelayout.rs

diff --git a/cranelift/codegen/src/ir/framelayout.rs b/cranelift/codegen/src/ir/framelayout.rs
new file mode 100644
index 0000000000..1b914a54f0
--- /dev/null
+++ b/cranelift/codegen/src/ir/framelayout.rs
@@ -0,0 +1,70 @@
+//! Frame layout item changes.
+
+use crate::ir::entities::Inst;
+use crate::isa::RegUnit;
+use std::boxed::Box;
+
+use crate::HashMap;
+
+/// Change in the frame layout information.
+#[derive(Clone, Copy, Debug, Eq, PartialEq, Hash)]
+pub enum FrameLayoutChange {
+    /// Base CallFrameAddress (CFA) pointer moved to different register/offset.
+    CallFrameAddressAt {
+        /// CFA register.
+        reg: RegUnit,
+        /// CFA offset.
+        offset: isize,
+    },
+    /// Register saved at.
+    RegAt {
+        /// Saved register.
+        reg: RegUnit,
+        /// Offset in the frame (offset from CFA).
+        cfa_offset: isize,
+    },
+    /// Return address saved at.
+    ReturnAddressAt {
+        /// Offset in the frame (offset from CFA).
+        cfa_offset: isize,
+    },
+    /// The entire frame layout must be preserved somewhere to be restored at a corresponding
+    /// `Restore` change.
+    ///
+    /// This likely maps to the DWARF call frame instruction `.cfa_remember_state`.
+    Preserve,
+    /// Restore the entire frame layout from a corresponding prior `Preserve` frame change.
+    ///
+    /// This likely maps to the DWARF call frame instruction `.cfa_restore_state`.
+    Restore,
+}
+
+/// Set of frame layout changes.
+pub type FrameLayoutChanges = Box<[FrameLayoutChange]>;
+
+/// Frame items layout for (prologue/epilogue) instructions.
+#[derive(Debug, Clone)]
+pub struct FrameLayout {
+    /// Initial frame layout.
+    pub initial: FrameLayoutChanges,
+
+    /// Instruction frame layout (changes). Because the map will not be dense,
+    /// a HashMap is used instead of a SecondaryMap.
+    pub instructions: HashMap,
+}
+
+impl FrameLayout {
+    /// Create instance of FrameLayout.
+    pub fn new() -> Self {
+        FrameLayout {
+            initial: vec![].into_boxed_slice(),
+            instructions: HashMap::new(),
+        }
+    }
+
+    /// Clear the structure.
+    pub fn clear(&mut self) {
+        self.initial = vec![].into_boxed_slice();
+        self.instructions.clear();
+    }
+}
diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index f226eea98f..48690c429f 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -11,7 +11,7 @@ use crate::ir::{
     Ebb, ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, Inst, JumpTable,
     JumpTableData, SigRef, StackSlot, StackSlotData, Table, TableData,
 };
-use crate::ir::{EbbOffsets, InstEncodings, SourceLocs, StackSlots, ValueLocations};
+use crate::ir::{EbbOffsets, FrameLayout, InstEncodings, SourceLocs, StackSlots, ValueLocations};
 use crate::ir::{JumpTableOffsets, JumpTables};
 use crate::isa::{CallConv, EncInfo, Encoding, Legalize, TargetIsa};
 use crate::regalloc::{EntryRegDiversions, RegDiversions};
@@ -92,6 +92,13 @@ pub struct Function {
     ///
     /// This is used for some calling conventions to track the end of unwind information.
     pub prologue_end: Option,
+
+    /// Frame layout for the instructions.
+    ///
+    /// The stack unwinding requires to have information about which registers and where they
+    /// are saved in the frame. This information is created during the prologue and epilogue
+    /// passes.
+    pub frame_layout: Option,
 }
 
 impl Function {
@@ -115,6 +122,7 @@ impl Function {
             jt_offsets: SecondaryMap::new(),
             srclocs: SecondaryMap::new(),
             prologue_end: None,
+            frame_layout: None,
         }
     }
 
@@ -135,6 +143,7 @@ impl Function {
         self.jt_offsets.clear();
         self.srclocs.clear();
         self.prologue_end = None;
+        self.frame_layout = None;
     }
 
     /// Create a new empty, anonymous function with a Fast calling convention.
@@ -244,6 +253,7 @@ impl Function {
     /// Starts collection of debug information.
     pub fn collect_debug_info(&mut self) {
         self.dfg.collect_debug_info();
+        self.frame_layout = Some(FrameLayout::new());
     }
 
     /// Changes the destination of a jump or branch instruction.
diff --git a/cranelift/codegen/src/ir/mod.rs b/cranelift/codegen/src/ir/mod.rs
index 8b89227ff2..88dfb8eb8d 100644
--- a/cranelift/codegen/src/ir/mod.rs
+++ b/cranelift/codegen/src/ir/mod.rs
@@ -6,6 +6,7 @@ pub mod dfg;
 pub mod entities;
 mod extfunc;
 mod extname;
+mod framelayout;
 pub mod function;
 mod globalvalue;
 mod heap;
@@ -39,6 +40,7 @@ pub use crate::ir::extfunc::{
     AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData, Signature,
 };
 pub use crate::ir::extname::ExternalName;
+pub use crate::ir::framelayout::{FrameLayout, FrameLayoutChange, FrameLayoutChanges};
 pub use crate::ir::function::{DisplayFunctionAnnotations, Function};
 pub use crate::ir::globalvalue::GlobalValueData;
 pub use crate::ir::heap::{HeapData, HeapStyle};
diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 5201a11f93..1ee48ddfcb 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -10,8 +10,8 @@ use crate::ir;
 use crate::ir::immediates::Imm64;
 use crate::ir::stackslot::{StackOffset, StackSize};
 use crate::ir::{
-    get_probestack_funcref, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose, InstBuilder,
-    ValueLoc,
+    get_probestack_funcref, AbiParam, ArgumentExtension, ArgumentLoc, ArgumentPurpose,
+    FrameLayoutChange, InstBuilder, ValueLoc,
 };
 use crate::isa::{CallConv, RegClass, RegUnit, TargetIsa};
 use crate::regalloc::RegisterSet;
@@ -20,6 +20,7 @@ use crate::stack_layout::layout_stack;
 use alloc::borrow::Cow;
 use alloc::vec::Vec;
 use core::i32;
+use std::boxed::Box;
 use target_lexicon::{PointerWidth, Triple};
 
 /// Argument registers for x86-64
@@ -471,6 +472,32 @@ fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) ->
     Ok(())
 }
 
+/// CFAState is cranelift's model of the call frame layout at any particular point in a function.
+/// It describes the call frame's layout in terms of a call frame address, where it is with respect
+/// to the start of the call frame, and the where the top of the stack is with respect to it.
+///
+/// Changes in this layout are used to derive appropriate `ir::FrameLayoutChange` to record for
+/// relevant instructions.
+#[derive(Clone)]
+struct CFAState {
+    /// The register from which we can derive the call frame address. On x86_64, this is typically
+    /// `rbp`, but at function entry and exit may be `rsp` while the call frame is being
+    /// established.
+    cf_ptr_reg: RegUnit,
+    /// Given that `cf_ptr_reg` is a register containing a pointer to some memory, `cf_ptr_offset`
+    /// is the offset from that pointer to the address of the start of this function's call frame.
+    ///
+    /// For a concrete x86_64 example, we will start this at 8 - the call frame begins immediately
+    /// before the return address. This will typically then be set to 16, after pushing `rbp` to
+    /// preserve the parent call frame. It is very unlikely the offset should be anything other
+    /// than one or two pointer widths.
+    cf_ptr_offset: isize,
+    /// The offset between the start of the call frame and the current stack pointer. This is
+    /// primarily useful to point to where on the stack preserved registers are, but is maintained
+    /// through the whole function for consistency.
+    current_depth: isize,
+}
+
 /// Implementation of the fastcall-based Win64 calling convention described at [1]
 /// [1] https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention
 fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> CodegenResult<()> {
@@ -522,11 +549,19 @@ fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> C
     // Set up the cursor and insert the prologue
     let entry_ebb = func.layout.entry_block().expect("missing entry block");
     let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_ebb);
-    insert_common_prologue(&mut pos, local_stack_size, reg_type, &csrs, isa);
+    let prologue_cfa_state =
+        insert_common_prologue(&mut pos, local_stack_size, reg_type, &csrs, isa);
 
     // Reset the cursor and insert the epilogue
     let mut pos = pos.at_position(CursorPosition::Nowhere);
-    insert_common_epilogues(&mut pos, local_stack_size, reg_type, &csrs);
+    insert_common_epilogues(
+        &mut pos,
+        local_stack_size,
+        reg_type,
+        &csrs,
+        isa,
+        prologue_cfa_state,
+    );
 
     Ok(())
 }
@@ -575,11 +610,19 @@ fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> C
     // Set up the cursor and insert the prologue
     let entry_ebb = func.layout.entry_block().expect("missing entry block");
     let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_ebb);
-    insert_common_prologue(&mut pos, local_stack_size, reg_type, &csrs, isa);
+    let prologue_cfa_state =
+        insert_common_prologue(&mut pos, local_stack_size, reg_type, &csrs, isa);
 
     // Reset the cursor and insert the epilogue
     let mut pos = pos.at_position(CursorPosition::Nowhere);
-    insert_common_epilogues(&mut pos, local_stack_size, reg_type, &csrs);
+    insert_common_epilogues(
+        &mut pos,
+        local_stack_size,
+        reg_type,
+        &csrs,
+        isa,
+        prologue_cfa_state,
+    );
 
     Ok(())
 }
@@ -592,7 +635,8 @@ fn insert_common_prologue(
     reg_type: ir::types::Type,
     csrs: &RegisterSet,
     isa: &dyn TargetIsa,
-) {
+) -> Option {
+    let word_size = isa.pointer_bytes() as isize;
     if stack_size > 0 {
         // Check if there is a special stack limit parameter. If so insert stack check.
         if let Some(stack_limit_arg) = pos.func.special_param(ArgumentPurpose::StackLimit) {
@@ -601,22 +645,83 @@ fn insert_common_prologue(
             // Also, the size of a return address, implicitly pushed by a x86 `call` instruction,
             // also should be accounted for.
             // TODO: Check if the function body actually contains a `call` instruction.
-            let word_size = isa.pointer_bytes();
             let total_stack_size = (csrs.iter(GPR).len() + 1 + 1) as i64 * word_size as i64;
 
             insert_stack_check(pos, total_stack_size, stack_limit_arg);
         }
     }
 
+    let mut cfa_state = if let Some(ref mut frame_layout) = pos.func.frame_layout {
+        let cfa_state = CFAState {
+            cf_ptr_reg: RU::rsp as RegUnit,
+            cf_ptr_offset: word_size,
+            current_depth: -word_size,
+        };
+
+        frame_layout.initial = vec![
+            FrameLayoutChange::CallFrameAddressAt {
+                reg: cfa_state.cf_ptr_reg,
+                offset: cfa_state.cf_ptr_offset,
+            },
+            FrameLayoutChange::ReturnAddressAt {
+                cfa_offset: cfa_state.current_depth,
+            },
+        ]
+        .into_boxed_slice();
+
+        Some(cfa_state)
+    } else {
+        None
+    };
+
     // Append param to entry EBB
     let ebb = pos.current_ebb().expect("missing ebb under cursor");
     let fp = pos.func.dfg.append_ebb_param(ebb, reg_type);
     pos.func.locations[fp] = ir::ValueLoc::Reg(RU::rbp as RegUnit);
 
-    pos.ins().x86_push(fp);
-    pos.ins()
+    let push_fp_inst = pos.ins().x86_push(fp);
+
+    if let Some(ref mut frame_layout) = pos.func.frame_layout {
+        let cfa_state = cfa_state
+            .as_mut()
+            .expect("cfa state exists when recording frame layout");
+        cfa_state.current_depth -= word_size;
+        cfa_state.cf_ptr_offset += word_size;
+        frame_layout.instructions.insert(
+            push_fp_inst,
+            vec![
+                FrameLayoutChange::CallFrameAddressAt {
+                    reg: cfa_state.cf_ptr_reg,
+                    offset: cfa_state.cf_ptr_offset,
+                },
+                FrameLayoutChange::RegAt {
+                    reg: RU::rbp as RegUnit,
+                    cfa_offset: cfa_state.current_depth,
+                },
+            ]
+            .into_boxed_slice(),
+        );
+    }
+
+    let mov_sp_inst = pos
+        .ins()
         .copy_special(RU::rsp as RegUnit, RU::rbp as RegUnit);
 
+    if let Some(ref mut frame_layout) = pos.func.frame_layout {
+        let mut cfa_state = cfa_state
+            .as_mut()
+            .expect("cfa state exists when recording frame layout");
+        cfa_state.cf_ptr_reg = RU::rbp as RegUnit;
+        frame_layout.instructions.insert(
+            mov_sp_inst,
+            vec![FrameLayoutChange::CallFrameAddressAt {
+                reg: cfa_state.cf_ptr_reg,
+                offset: cfa_state.cf_ptr_offset,
+            }]
+            .into_boxed_slice(),
+        );
+    }
+
     for reg in csrs.iter(GPR) {
         // Append param to entry EBB
         let csr_arg = pos.func.dfg.append_ebb_param(ebb, reg_type);
@@ -625,7 +730,22 @@ fn insert_common_prologue(
         pos.func.locations[csr_arg] = ir::ValueLoc::Reg(reg);
 
         // Remember it so we can push it momentarily
-        pos.ins().x86_push(csr_arg);
+        let reg_push_inst = pos.ins().x86_push(csr_arg);
+
+        if let Some(ref mut frame_layout) = pos.func.frame_layout {
+            let mut cfa_state = cfa_state
+                .as_mut()
+                .expect("cfa state exists when recording frame layout");
+            cfa_state.current_depth -= word_size;
+            frame_layout.instructions.insert(
+                reg_push_inst,
+                vec![FrameLayoutChange::RegAt {
+                    reg,
+                    cfa_offset: cfa_state.current_depth,
+                }]
+                .into_boxed_slice(),
+            );
+        }
     }
 
     // Allocate stack frame storage.
@@ -672,6 +792,8 @@ fn insert_common_prologue(
             pos.func.prologue_end = Some(pos.ins().adjust_sp_down_imm(Imm64::new(stack_size)));
         }
     }
+
+    cfa_state
 }
 
 /// Insert a check that generates a trap if the stack pointer goes
@@ -703,12 +825,41 @@ fn insert_common_epilogues(
     stack_size: i64,
     reg_type: ir::types::Type,
     csrs: &RegisterSet,
+    isa: &dyn TargetIsa,
+    cfa_state: Option,
 ) {
     while let Some(ebb) = pos.next_ebb() {
         pos.goto_last_inst(ebb);
         if let Some(inst) = pos.current_inst() {
             if pos.func.dfg[inst].opcode().is_return() {
-                insert_common_epilogue(inst, stack_size, pos, reg_type, csrs);
+                if let (Some(ref mut frame_layout), ref func_layout) =
+                    (pos.func.frame_layout.as_mut(), &pos.func.layout)
+                {
+                    // Figure out if we need to insert end-of-function-aware frame layout information.
+                    let following_inst = func_layout
+                        .next_ebb(ebb)
+                        .and_then(|next_ebb| func_layout.first_inst(next_ebb));
+
+                    if let Some(following_inst) = following_inst {
+                        frame_layout
+                            .instructions
+                            .insert(inst, vec![FrameLayoutChange::Preserve].into_boxed_slice());
+                        frame_layout.instructions.insert(
+                            following_inst,
+                            vec![FrameLayoutChange::Restore].into_boxed_slice(),
+                        );
+                    }
+                }
+
+                insert_common_epilogue(
+                    inst,
+                    stack_size,
+                    pos,
+                    reg_type,
+                    csrs,
+                    isa,
+                    cfa_state.clone(),
+                );
             }
         }
     }
@@ -722,7 +873,10 @@ fn insert_common_epilogue(
     pos: &mut EncCursor,
     reg_type: ir::types::Type,
     csrs: &RegisterSet,
+    isa: &dyn TargetIsa,
+    mut cfa_state: Option,
 ) {
+    let word_size = isa.pointer_bytes() as isize;
     if stack_size > 0 {
         pos.ins().adjust_sp_up_imm(Imm64::new(stack_size));
     }
@@ -730,6 +884,18 @@ fn insert_common_epilogue(
     // Pop all the callee-saved registers, stepping backward each time to
     // preserve the correct order.
     let fp_ret = pos.ins().x86_pop(reg_type);
+    let fp_pop_inst = pos.built_inst();
+
+    if let Some(ref mut cfa_state) = cfa_state.as_mut() {
+        // Account for CFA state in the reverse of `insert_common_prologue`.
+        cfa_state.current_depth += word_size;
+        cfa_state.cf_ptr_offset -= word_size;
+        // And now that we're going to overwrite `rbp`, `rsp` is the only way to get to the call frame.
+        // We don't apply a frame layout change *yet* because we check that at return the depth is
+        // exactly one `word_size`.
+        cfa_state.cf_ptr_reg = RU::rsp as RegUnit;
+    }
+
     pos.prev_inst();
 
     pos.func.locations[fp_ret] = ir::ValueLoc::Reg(RU::rbp as RegUnit);
@@ -737,11 +903,48 @@ fn insert_common_epilogue(
 
     for reg in csrs.iter(GPR) {
         let csr_ret = pos.ins().x86_pop(reg_type);
+        if let Some(ref mut cfa_state) = cfa_state.as_mut() {
+            // Note: don't bother recording a frame layout change because the popped value is
+            // still correct in memory, and won't be overwritten until we've returned where the
+            // current frame's layout would no longer matter. Only adjust `current_depth` for a
+            // consistency check later.
+            cfa_state.current_depth += word_size;
+        }
         pos.prev_inst();
 
         pos.func.locations[csr_ret] = ir::ValueLoc::Reg(reg);
         pos.func.dfg.append_inst_arg(inst, csr_ret);
     }
+
+    if let Some(ref mut frame_layout) = pos.func.frame_layout {
+        let cfa_state = cfa_state
+            .as_mut()
+            .expect("cfa state exists when recording frame layout");
+        // Validity checks - if we accounted correctly, CFA state at a return will match CFA state
+        // at the entry of a function.
+        //
+        // Current_depth starts assuming a return address is pushed, and cf_ptr_offset is one
+        // pointer below current_depth.
+        assert_eq!(cfa_state.current_depth, -word_size);
+        assert_eq!(cfa_state.cf_ptr_offset, word_size);
+
+        let new_cfa = FrameLayoutChange::CallFrameAddressAt {
+            reg: cfa_state.cf_ptr_reg,
+            offset: cfa_state.cf_ptr_offset,
+        };
+
+        frame_layout
+            .instructions
+            .entry(fp_pop_inst)
+            .and_modify(|insts| {
+                *insts = insts
+                    .into_iter()
+                    .cloned()
+                    .chain(std::iter::once(new_cfa))
+                    .collect::>();
+            })
+            .or_insert_with(|| Box::new([new_cfa]));
+    }
 }
 
 pub fn emit_unwind_info(func: &ir::Function, isa: &dyn TargetIsa, mem: &mut Vec) {

From 0b7e7916bdc690021d910a777fac3e640387a78f Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 19 Nov 2019 13:49:50 +0100
Subject: [PATCH 2958/3084] Fixes #1228: add a link to the contributing.md file
 from the readme. (#1245)

---
 cranelift/README.md | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/cranelift/README.md b/cranelift/README.md
index 23bf4c353e..e2833df622 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -58,6 +58,13 @@ Cranelift's APIs are not yet stable.
 
 Cranelift currently requires Rust 1.37 or later to build.
 
+Contributing
+------------
+
+If you're interested in contributing to Cranelift: thank you! We have a
+[contributing guide](CONTRIBUTING.md) which will help you getting involved in
+the Cranelift project.
+
 Planned uses
 ------------
 

From c5e74986e1d55532c10fa9f655e9d741b317db73 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Sat, 16 Nov 2019 16:25:57 +0100
Subject: [PATCH 2959/3084] Legalize uextend and sextend to 128bit ints

---
 .../codegen/meta/src/cdsl/type_inference.rs   | 12 +++++-
 cranelift/codegen/meta/src/shared/legalize.rs | 22 +++++++++++
 .../filetests/isa/x86/extend-i128-run.clif    | 26 +++++++++++++
 .../filetests/isa/x86/extend-i128.clif        | 37 +++++++++++++++++++
 .../filetests/legalizer/popcnt-i128.clif      | 10 ++---
 5 files changed, 100 insertions(+), 7 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/extend-i128-run.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/extend-i128.clif

diff --git a/cranelift/codegen/meta/src/cdsl/type_inference.rs b/cranelift/codegen/meta/src/cdsl/type_inference.rs
index 9852dd1e7b..0313345175 100644
--- a/cranelift/codegen/meta/src/cdsl/type_inference.rs
+++ b/cranelift/codegen/meta/src/cdsl/type_inference.rs
@@ -161,8 +161,16 @@ impl TypeEnvironment {
 
         // Check extra conditions for InTypeset constraints.
         if let Constraint::InTypeset(tv, _) = &constraint {
-            assert!(tv.base.is_none());
-            assert!(tv.name.starts_with("typeof_"));
+            assert!(
+                tv.base.is_none(),
+                "type variable is {:?}, while expecting none",
+                tv
+            );
+            assert!(
+                tv.name.starts_with("typeof_"),
+                "Name \"{}\" should start with \"typeof_\"",
+                tv.name
+            );
         }
 
         self.constraints.push(constraint);
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 2f20e5a613..7c7b121ab4 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -207,6 +207,28 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     // embedded as part of arguments), so use a custom legalization for now.
     narrow.custom_legalize(iconst, "narrow_iconst");
 
+    {
+        let inst = uextend.bind(I128).bind(I64);
+        narrow.legalize(
+            def!(a = inst(x)),
+            vec![
+                def!(ah = iconst(Literal::constant(&imm.imm64, 0))),
+                def!(a = iconcat(x, ah)),
+            ],
+        );
+    }
+
+    {
+        let inst = sextend.bind(I128).bind(I64);
+        narrow.legalize(
+            def!(a = inst(x)),
+            vec![
+                def!(ah = sshr_imm(x, Literal::constant(&imm.imm64, 63))), // splat sign bit to whole number
+                def!(a = iconcat(x, ah)),
+            ],
+        );
+    }
+
     for &bin_op in &[band, bor, bxor, band_not, bor_not, bxor_not] {
         narrow.legalize(
             def!(a = bin_op(x, y)),
diff --git a/cranelift/filetests/filetests/isa/x86/extend-i128-run.clif b/cranelift/filetests/filetests/isa/x86/extend-i128-run.clif
new file mode 100644
index 0000000000..c2550b8f4e
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/extend-i128-run.clif
@@ -0,0 +1,26 @@
+test run
+target x86_64
+
+function u0:0() -> b1 {
+ebb0:
+    v0 = iconst.i64 0xffff_ffff_eeee_0000
+    v1 = uextend.i128 v0
+    v2, v3 = isplit v1
+    v4 = icmp_imm eq v2, 0xffff_ffff_eeee_0000
+    v5 = icmp_imm eq v3, 0
+    v6 = band v4, v5
+    return v6
+}
+; run
+
+function u0:1() -> b1 {
+ebb0:
+    v0 = iconst.i64 0xffff_ffff_eeee_0000
+    v1 = sextend.i128 v0
+    v2, v3 = isplit v1
+    v4 = icmp_imm eq v2, 0xffff_ffff_eeee_0000
+    v5 = icmp_imm eq v3, 0xffff_ffff_ffff_ffff
+    v6 = band v4, v5
+    return v6
+}
+; run
diff --git a/cranelift/filetests/filetests/isa/x86/extend-i128.clif b/cranelift/filetests/filetests/isa/x86/extend-i128.clif
new file mode 100644
index 0000000000..0d9e4c8aa9
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/extend-i128.clif
@@ -0,0 +1,37 @@
+test compile
+target x86_64
+
+function u0:0() -> b1 {
+ebb0:
+    v0 = iconst.i64 0xffff_ffff_eeee_0000
+    ; check: v0 = iconst.i64 0xffff_ffff_eeee_0000
+    ; nextln: v2 -> v0
+    v1 = uextend.i128 v0
+    ; nextln: v7 = iconst.i64 0
+    ; nextln: v3 -> v7
+    ; nextln: v1 = iconcat v0, v7
+
+    v2, v3 = isplit v1
+    v4 = icmp_imm eq v2, 0xffff_ffff_eeee_0000
+    v5 = icmp_imm eq v3, 0
+
+    v6 = band v4, v5
+    return v6
+}
+
+function u0:1() -> b1 {
+ebb0:
+    v0 = iconst.i64 0xffff_ffff_eeee_0000
+    ; check: v0 = iconst.i64 0xffff_ffff_eeee_0000
+    ; nextln: v2 -> v0
+    v1 = sextend.i128 v0
+    ; nextln: v8 = copy v0
+    ; nextln: v7 = sshr_imm v8, 63
+    ; nextln: v3 -> v7
+
+    v2, v3 = isplit v1
+    v4 = icmp_imm eq v2, 0xffff_ffff_eeee_0000
+    v5 = icmp_imm eq v3, 0xffff_ffff_ffff_ffff
+    v6 = band v4, v5
+    return v6
+}
diff --git a/cranelift/filetests/filetests/legalizer/popcnt-i128.clif b/cranelift/filetests/filetests/legalizer/popcnt-i128.clif
index b241c640a9..0ecf7f74c5 100644
--- a/cranelift/filetests/filetests/legalizer/popcnt-i128.clif
+++ b/cranelift/filetests/filetests/legalizer/popcnt-i128.clif
@@ -24,8 +24,8 @@ ebb0:
 ; check: v16, v17 = iadd_ifcout v12, v14
 ; check: v18 = iadd_ifcin v13, v15, v17
 ; check: v11 = iconcat v16, v18
-; check: v4 = uextend.i128 v11
-; check: v19, v20 = isplit v4
-; check: v21, v22 = isplit v19
-; check: v23, v24 = isplit v20
-; check: return v21, v22, v23, v24
+; check: v20 = iconst.i32 0
+; check: v21 = iconst.i32 0
+; check: v19 = iconcat v20, v21
+; check: v4 = iconcat v11, v19
+; check: return v16, v18, v20, v21

From 838f2f4646958e9a2e607e2e7be91ea8f4a9b5ab Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 20 Nov 2019 13:29:53 -0800
Subject: [PATCH 2960/3084] Merge pull request #1226 from
 abrown/log-verifier-errors

Log verifier errors
---
 cranelift/codegen/src/verifier/mod.rs | 403 +++++++++++++++++---------
 1 file changed, 264 insertions(+), 139 deletions(-)

diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index f31b4e2340..61b653e469 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -70,13 +70,15 @@ use crate::ir::{
 };
 use crate::isa::TargetIsa;
 use crate::iterators::IteratorExtras;
+use crate::print_errors::pretty_verifier_error;
 use crate::settings::FlagsOrIsa;
 use crate::timing;
 use alloc::collections::BTreeSet;
-use alloc::string::String;
+use alloc::string::{String, ToString};
 use alloc::vec::Vec;
 use core::cmp::Ordering;
 use core::fmt::{self, Display, Formatter, Write};
+use log::debug;
 use thiserror::Error;
 
 pub use self::cssa::verify_cssa;
@@ -93,6 +95,7 @@ macro_rules! report {
     ( $errors: expr, $loc: expr, $msg: tt ) => {
         $errors.0.push(crate::verifier::VerifierError {
             location: $loc.into(),
+            context: None,
             message: String::from($msg),
         })
     };
@@ -100,6 +103,7 @@ macro_rules! report {
     ( $errors: expr, $loc: expr, $fmt: tt, $( $arg: expr ),+ ) => {
         $errors.0.push(crate::verifier::VerifierError {
             location: $loc.into(),
+            context: None,
             message: format!( $fmt, $( $arg ),+ ),
         })
     };
@@ -127,15 +131,26 @@ mod liveness;
 mod locations;
 
 /// A verifier error.
-#[derive(Error, Debug, PartialEq, Eq)]
-#[error("{location}: {message}")]
+#[derive(Error, Debug, PartialEq, Eq, Clone)]
+#[error("{}{}: {}", .location, format_context(.context), .message)]
 pub struct VerifierError {
     /// The entity causing the verifier error.
     pub location: AnyEntity,
+    /// Optionally provide some context for the given location; e.g., for `inst42` provide
+    /// `Some("v3 = iconst.i32 0")` for more comprehensible errors.
+    pub context: Option,
     /// The error message.
     pub message: String,
 }
 
+/// Helper for formatting Verifier::Error context.
+fn format_context(context: &Option) -> String {
+    match context {
+        None => "".to_string(),
+        Some(c) => format!(" ({})", c).to_string(),
+    }
+}
+
 /// Result of a step in the verification process.
 ///
 /// Functions that return `VerifierStepResult<()>` should also take a
@@ -155,7 +170,7 @@ pub type VerifierStepResult = Result;
 pub type VerifierResult = Result;
 
 /// List of verifier errors.
-#[derive(Error, Debug, Default, PartialEq, Eq)]
+#[derive(Error, Debug, Default, PartialEq, Eq, Clone)]
 pub struct VerifierErrors(pub Vec);
 
 impl VerifierErrors {
@@ -276,6 +291,41 @@ impl<'a> Verifier<'a> {
         }
     }
 
+    /// Helper for appending a contextual error to the verifier error list.
+    #[inline]
+    fn append_error(&self, inst: Inst, errors: &mut VerifierErrors, message: impl Into) {
+        let location = inst.into();
+        let context = Some(self.func.dfg.display_inst(inst, self.isa).to_string());
+        let message = message.into();
+        errors.0.push(VerifierError {
+            location,
+            context,
+            message,
+        });
+    }
+
+    /// Diagnose a fatal error, and return `Err`.
+    fn fatal(
+        &self,
+        errors: &mut VerifierErrors,
+        inst: Inst,
+        message: impl Into,
+    ) -> VerifierStepResult<()> {
+        self.append_error(inst, errors, message);
+        Err(())
+    }
+
+    /// Diagnose a non-fatal error, and return `Ok`.
+    fn nonfatal(
+        &self,
+        errors: &mut VerifierErrors,
+        inst: Inst,
+        message: impl Into,
+    ) -> VerifierStepResult<()> {
+        self.append_error(inst, errors, message);
+        Ok(())
+    }
+
     // Check for:
     //  - cycles in the global value declarations.
     //  - use of 'vmctx' when no special parameter declares it.
@@ -469,7 +519,7 @@ impl<'a> Verifier<'a> {
     fn encodable_as_bb(&self, ebb: Ebb, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         match self.func.is_ebb_basic(ebb) {
             Ok(()) => Ok(()),
-            Err((inst, message)) => fatal!(errors, inst, message),
+            Err((inst, message)) => self.fatal(errors, inst, message),
         }
     }
 
@@ -484,11 +534,13 @@ impl<'a> Verifier<'a> {
 
         if is_terminator && !is_last_inst {
             // Terminating instructions only occur at the end of blocks.
-            return fatal!(
+            return self.fatal(
                 errors,
                 inst,
-                "a terminator instruction was encountered before the end of {}",
-                ebb
+                format!(
+                    "a terminator instruction was encountered before the end of {}",
+                    ebb
+                ),
             );
         }
         if is_last_inst && !is_terminator {
@@ -502,7 +554,11 @@ impl<'a> Verifier<'a> {
         // Instructions belong to the correct ebb.
         let inst_ebb = self.func.layout.inst_ebb(inst);
         if inst_ebb != Some(ebb) {
-            return fatal!(errors, inst, "should belong to {} not {:?}", ebb, inst_ebb);
+            return self.fatal(
+                errors,
+                inst,
+                format!("should belong to {} not {:?}", ebb, inst_ebb),
+            );
         }
 
         // Parameters belong to the correct ebb.
@@ -532,10 +588,10 @@ impl<'a> Verifier<'a> {
 
         // The instruction format matches the opcode
         if inst_data.opcode().format() != InstructionFormat::from(inst_data) {
-            return fatal!(
+            return self.fatal(
                 errors,
                 inst,
-                "instruction opcode doesn't match instruction format"
+                "instruction opcode doesn't match instruction format",
             );
         }
 
@@ -549,12 +605,13 @@ impl<'a> Verifier<'a> {
         // All result values for multi-valued instructions are created
         let got_results = dfg.inst_results(inst).len();
         if got_results != total_results {
-            return fatal!(
+            return self.fatal(
                 errors,
                 inst,
-                "expected {} result values, found {}",
-                total_results,
-                got_results
+                format!(
+                    "expected {} result values, found {}",
+                    total_results, got_results,
+                ),
             );
         }
 
@@ -680,14 +737,14 @@ impl<'a> Verifier<'a> {
             } => {
                 if let Some(isa) = &self.isa {
                     if !isa.flags().enable_pinned_reg() {
-                        return fatal!(
+                        return self.fatal(
                             errors,
                             inst,
-                            "GetPinnedReg/SetPinnedReg cannot be used without enable_pinned_reg"
+                            "GetPinnedReg/SetPinnedReg cannot be used without enable_pinned_reg",
                         );
                     }
                 } else {
-                    return fatal!(errors, inst, "GetPinnedReg/SetPinnedReg need an ISA!");
+                    return self.fatal(errors, inst, "GetPinnedReg/SetPinnedReg need an ISA!");
                 }
             }
 
@@ -756,7 +813,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.dfg.signatures.is_valid(s) {
-            fatal!(errors, inst, "invalid signature reference {}", s)
+            self.fatal(errors, inst, format!("invalid signature reference {}", s))
         } else {
             Ok(())
         }
@@ -769,7 +826,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.dfg.ext_funcs.is_valid(f) {
-            nonfatal!(errors, inst, "invalid function reference {}", f)
+            self.nonfatal(errors, inst, format!("invalid function reference {}", f))
         } else {
             Ok(())
         }
@@ -782,7 +839,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.stack_slots.is_valid(ss) {
-            nonfatal!(errors, inst, "invalid stack slot {}", ss)
+            self.nonfatal(errors, inst, format!("invalid stack slot {}", ss))
         } else {
             Ok(())
         }
@@ -795,7 +852,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.global_values.is_valid(gv) {
-            nonfatal!(errors, inst, "invalid global value {}", gv)
+            self.nonfatal(errors, inst, format!("invalid global value {}", gv))
         } else {
             Ok(())
         }
@@ -808,7 +865,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.heaps.is_valid(heap) {
-            nonfatal!(errors, inst, "invalid heap {}", heap)
+            self.nonfatal(errors, inst, format!("invalid heap {}", heap))
         } else {
             Ok(())
         }
@@ -821,7 +878,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.tables.is_valid(table) {
-            nonfatal!(errors, inst, "invalid table {}", table)
+            self.nonfatal(errors, inst, format!("invalid table {}", table))
         } else {
             Ok(())
         }
@@ -834,7 +891,11 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !l.is_valid(&self.func.dfg.value_lists) {
-            nonfatal!(errors, inst, "invalid value list reference {:?}", l)
+            self.nonfatal(
+                errors,
+                inst,
+                format!("invalid value list reference {:?}", l),
+            )
         } else {
             Ok(())
         }
@@ -847,7 +908,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.jump_tables.is_valid(j) {
-            nonfatal!(errors, inst, "invalid jump table reference {}", j)
+            self.nonfatal(errors, inst, format!("invalid jump table reference {}", j))
         } else {
             Ok(())
         }
@@ -861,7 +922,7 @@ impl<'a> Verifier<'a> {
     ) -> VerifierStepResult<()> {
         let dfg = &self.func.dfg;
         if !dfg.value_is_valid(v) {
-            nonfatal!(errors, loc_inst, "invalid value reference {}", v)
+            self.nonfatal(errors, loc_inst, format!("invalid value reference {}", v))
         } else {
             Ok(())
         }
@@ -917,23 +978,29 @@ impl<'a> Verifier<'a> {
                         );
                     }
                     if def_inst == loc_inst {
-                        return fatal!(errors, loc_inst, "uses value {} from itself", v);
+                        return self.fatal(
+                            errors,
+                            loc_inst,
+                            format!("uses value {} from itself", v),
+                        );
                     }
                 }
             }
             ValueDef::Param(ebb, _) => {
                 // Value is defined by an existing EBB.
                 if !dfg.ebb_is_valid(ebb) {
-                    return fatal!(errors, loc_inst, "{} is defined by invalid EBB {}", v, ebb);
+                    return self.fatal(
+                        errors,
+                        loc_inst,
+                        format!("{} is defined by invalid EBB {}", v, ebb),
+                    );
                 }
                 // Defining EBB is inserted in the layout
                 if !self.func.layout.is_ebb_inserted(ebb) {
-                    return fatal!(
+                    return self.fatal(
                         errors,
                         loc_inst,
-                        "{} is defined by {} which is not in the layout",
-                        v,
-                        ebb
+                        format!("{} is defined by {} which is not in the layout", v, ebb),
                     );
                 }
                 // The defining EBB dominates the instruction using this value.
@@ -942,11 +1009,10 @@ impl<'a> Verifier<'a> {
                         .expected_domtree
                         .dominates(ebb, loc_inst, &self.func.layout)
                 {
-                    return fatal!(
+                    return self.fatal(
                         errors,
                         loc_inst,
-                        "uses value arg from non-dominating {}",
-                        ebb
+                        format!("uses value arg from non-dominating {}", ebb),
                     );
                 }
             }
@@ -994,12 +1060,14 @@ impl<'a> Verifier<'a> {
         let value_type = self.func.dfg.value_type(arg);
 
         if typ.lane_bits() < value_type.lane_bits() {
-            fatal!(
+            self.fatal(
                 errors,
                 inst,
-                "The bitcast argument {} doesn't fit in a type of {} bits",
-                arg,
-                typ.lane_bits()
+                format!(
+                    "The bitcast argument {} doesn't fit in a type of {} bits",
+                    arg,
+                    typ.lane_bits(),
+                ),
             )
         } else {
             Ok(())
@@ -1165,14 +1233,14 @@ impl<'a> Verifier<'a> {
                     );
                 }
             } else {
-                return nonfatal!(errors, inst, "has more result values than expected");
+                return self.nonfatal(errors, inst, "has more result values than expected");
             }
             i += 1;
         }
 
         // There aren't any more result types left.
         if self.func.dfg.compute_result_type(inst, i, ctrl_type) != None {
-            return nonfatal!(errors, inst, "has fewer result values than expected");
+            return self.nonfatal(errors, inst, "has fewer result values than expected");
         }
         Ok(())
     }
@@ -1238,24 +1306,26 @@ impl<'a> Verifier<'a> {
                 if let Some(ebb) = ebb {
                     let arg_count = self.func.dfg.num_ebb_params(ebb);
                     if arg_count != 0 {
-                        return nonfatal!(
+                        return self.nonfatal(
                             errors,
                             inst,
-                            "takes no arguments, but had target {} with {} arguments",
-                            ebb,
-                            arg_count
+                            format!(
+                                "takes no arguments, but had target {} with {} arguments",
+                                ebb, arg_count,
+                            ),
                         );
                     }
                 }
                 for ebb in self.func.jump_tables[table].iter() {
                     let arg_count = self.func.dfg.num_ebb_params(*ebb);
                     if arg_count != 0 {
-                        return nonfatal!(
+                        return self.nonfatal(
                             errors,
                             inst,
-                            "takes no arguments, but had target {} with {} arguments",
-                            ebb,
-                            arg_count
+                            format!(
+                                "takes no arguments, but had target {} with {} arguments",
+                                ebb, arg_count,
+                            ),
                         );
                     }
                 }
@@ -1317,13 +1387,15 @@ impl<'a> Verifier<'a> {
             i += 1;
         }
         if i != variable_args.len() {
-            return nonfatal!(
+            return self.nonfatal(
                 errors,
                 inst,
-                "mismatched argument count for `{}`: got {}, expected {}",
-                self.func.dfg.display_inst(inst, None),
-                variable_args.len(),
-                i
+                format!(
+                    "mismatched argument count for `{}`: got {}, expected {}",
+                    self.func.dfg.display_inst(inst, None),
+                    variable_args.len(),
+                    i,
+                ),
             );
         }
         Ok(())
@@ -1353,45 +1425,45 @@ impl<'a> Verifier<'a> {
                     self.verify_stack_slot(inst, ss, errors)?;
                     let slot = &self.func.stack_slots[ss];
                     if slot.kind != StackSlotKind::OutgoingArg {
-                        return fatal!(
+                        return self.fatal(
                             errors,
                             inst,
-                            "Outgoing stack argument {} in wrong stack slot: {} = {}",
-                            arg,
-                            ss,
-                            slot
+                            format!(
+                                "Outgoing stack argument {} in wrong stack slot: {} = {}",
+                                arg, ss, slot,
+                            ),
                         );
                     }
                     if slot.offset != Some(offset) {
-                        return fatal!(
+                        return self.fatal(
                             errors,
                             inst,
-                            "Outgoing stack argument {} should have offset {}: {} = {}",
-                            arg,
-                            offset,
-                            ss,
-                            slot
+                            format!(
+                                "Outgoing stack argument {} should have offset {}: {} = {}",
+                                arg, offset, ss, slot,
+                            ),
                         );
                     }
                     if slot.size != abi.value_type.bytes() {
-                        return fatal!(
+                        return self.fatal(
                             errors,
                             inst,
-                            "Outgoing stack argument {} wrong size for {}: {} = {}",
-                            arg,
-                            abi.value_type,
-                            ss,
-                            slot
+                            format!(
+                                "Outgoing stack argument {} wrong size for {}: {} = {}",
+                                arg, abi.value_type, ss, slot,
+                            ),
                         );
                     }
                 } else {
                     let reginfo = self.isa.map(|i| i.register_info());
-                    return fatal!(
+                    return self.fatal(
                         errors,
                         inst,
-                        "Outgoing stack argument {} in wrong location: {}",
-                        arg,
-                        arg_loc.display(reginfo.as_ref())
+                        format!(
+                            "Outgoing stack argument {} in wrong location: {}",
+                            arg,
+                            arg_loc.display(reginfo.as_ref()),
+                        ),
                     );
                 }
             }
@@ -1404,10 +1476,10 @@ impl<'a> Verifier<'a> {
             let args = self.func.dfg.inst_variable_args(inst);
             let expected_types = &self.func.signature.returns;
             if args.len() != expected_types.len() {
-                return nonfatal!(
+                return self.nonfatal(
                     errors,
                     inst,
-                    "arguments of return must match function signature"
+                    "arguments of return must match function signature",
                 );
             }
             for (i, (&arg, &expected_type)) in args.iter().zip(expected_types).enumerate() {
@@ -1442,41 +1514,45 @@ impl<'a> Verifier<'a> {
                 match opcode {
                     Opcode::Bextend | Opcode::Uextend | Opcode::Sextend | Opcode::Fpromote => {
                         if arg_type.lane_count() != ctrl_type.lane_count() {
-                            return nonfatal!(
+                            return self.nonfatal(
                                 errors,
                                 inst,
-                                "input {} and output {} must have same number of lanes",
-                                arg_type,
-                                ctrl_type
+                                format!(
+                                    "input {} and output {} must have same number of lanes",
+                                    arg_type, ctrl_type,
+                                ),
                             );
                         }
                         if arg_type.lane_bits() >= ctrl_type.lane_bits() {
-                            return nonfatal!(
+                            return self.nonfatal(
                                 errors,
                                 inst,
-                                "input {} must be smaller than output {}",
-                                arg_type,
-                                ctrl_type
+                                format!(
+                                    "input {} must be smaller than output {}",
+                                    arg_type, ctrl_type,
+                                ),
                             );
                         }
                     }
                     Opcode::Breduce | Opcode::Ireduce | Opcode::Fdemote => {
                         if arg_type.lane_count() != ctrl_type.lane_count() {
-                            return nonfatal!(
+                            return self.nonfatal(
                                 errors,
                                 inst,
-                                "input {} and output {} must have same number of lanes",
-                                arg_type,
-                                ctrl_type
+                                format!(
+                                    "input {} and output {} must have same number of lanes",
+                                    arg_type, ctrl_type,
+                                ),
                             );
                         }
                         if arg_type.lane_bits() <= ctrl_type.lane_bits() {
-                            return nonfatal!(
+                            return self.nonfatal(
                                 errors,
                                 inst,
-                                "input {} must be larger than output {}",
-                                arg_type,
-                                ctrl_type
+                                format!(
+                                    "input {} must be larger than output {}",
+                                    arg_type, ctrl_type,
+                                ),
                             );
                         }
                     }
@@ -1487,12 +1563,13 @@ impl<'a> Verifier<'a> {
                 let index_type = self.func.dfg.value_type(arg);
                 let heap_index_type = self.func.heaps[heap].index_type;
                 if index_type != heap_index_type {
-                    return nonfatal!(
+                    return self.nonfatal(
                         errors,
                         inst,
-                        "index type {} differs from heap index type {}",
-                        index_type,
-                        heap_index_type
+                        format!(
+                            "index type {} differs from heap index type {}",
+                            index_type, heap_index_type,
+                        ),
                     );
                 }
             }
@@ -1500,12 +1577,13 @@ impl<'a> Verifier<'a> {
                 let index_type = self.func.dfg.value_type(arg);
                 let table_index_type = self.func.tables[table].index_type;
                 if index_type != table_index_type {
-                    return nonfatal!(
+                    return self.nonfatal(
                         errors,
                         inst,
-                        "index type {} differs from table index type {}",
-                        index_type,
-                        table_index_type
+                        format!(
+                            "index type {} differs from table index type {}",
+                            index_type, table_index_type,
+                        ),
                     );
                 }
             }
@@ -1514,13 +1592,14 @@ impl<'a> Verifier<'a> {
                     let inst_type = self.func.dfg.value_type(self.func.dfg.first_result(inst));
                     let global_type = self.func.global_values[global_value].global_type(isa);
                     if inst_type != global_type {
-                        return nonfatal!(
-                        errors,
-                        inst,
-                        "global_value instruction with type {} references global value with type {}",
-                        inst_type,
-                        global_type
-                    );
+                        return self.nonfatal(
+                            errors,
+                            inst,
+                            format!(
+                                "global_value instruction with type {} references global value with type {}",
+                                inst_type, global_type
+                            ),
+                        );
                     }
                 }
             }
@@ -1541,11 +1620,11 @@ impl<'a> Verifier<'a> {
         {
             let dst_vals = self.func.dfg.inst_results(inst);
             if dst_vals.len() != 1 {
-                return fatal!(errors, inst, "copy_nop must produce exactly one result");
+                return self.fatal(errors, inst, "copy_nop must produce exactly one result");
             }
             let dst_val = dst_vals[0];
             if self.func.dfg.value_type(dst_val) != self.func.dfg.value_type(arg) {
-                return fatal!(errors, inst, "copy_nop src and dst types must be the same");
+                return self.fatal(errors, inst, "copy_nop src and dst types must be the same");
             }
             let src_loc = self.func.locations[arg];
             let dst_loc = self.func.locations[dst_val];
@@ -1554,12 +1633,13 @@ impl<'a> Verifier<'a> {
                 _ => false,
             };
             if !locs_ok {
-                return fatal!(
+                return self.fatal(
                     errors,
                     inst,
-                    "copy_nop must refer to identical stack slots, but found {:?} vs {:?}",
-                    src_loc,
-                    dst_loc
+                    format!(
+                        "copy_nop must refer to identical stack slots, but found {:?} vs {:?}",
+                        src_loc, dst_loc,
+                    ),
                 );
             }
         }
@@ -1658,11 +1738,13 @@ impl<'a> Verifier<'a> {
         let encoding = self.func.encodings[inst];
         if encoding.is_legal() {
             if self.func.dfg[inst].opcode().is_ghost() {
-                return nonfatal!(
+                return self.nonfatal(
                     errors,
                     inst,
-                    "Ghost instruction has an encoding: {}",
-                    isa.encoding_info().display(encoding)
+                    format!(
+                        "Ghost instruction has an encoding: {}",
+                        isa.encoding_info().display(encoding),
+                    ),
                 );
             }
 
@@ -1675,11 +1757,13 @@ impl<'a> Verifier<'a> {
                 .peekable();
 
             if encodings.peek().is_none() {
-                return nonfatal!(
+                return self.nonfatal(
                     errors,
                     inst,
-                    "Instruction failed to re-encode {}",
-                    isa.encoding_info().display(encoding)
+                    format!(
+                        "Instruction failed to re-encode {}",
+                        isa.encoding_info().display(encoding),
+                    ),
                 );
             }
 
@@ -1703,13 +1787,15 @@ impl<'a> Verifier<'a> {
                         .unwrap();
                 }
 
-                return nonfatal!(
+                return self.nonfatal(
                     errors,
                     inst,
-                    "encoding {} should be {}{}",
-                    isa.encoding_info().display(encoding),
-                    if multiple_encodings { "one of: " } else { "" },
-                    possible_encodings
+                    format!(
+                        "encoding {} should be {}{}",
+                        isa.encoding_info().display(encoding),
+                        if multiple_encodings { "one of: " } else { "" },
+                        possible_encodings,
+                    ),
                 );
             }
             return Ok(());
@@ -1749,15 +1835,19 @@ impl<'a> Verifier<'a> {
             // Provide the ISA default encoding as a hint.
             match self.func.encode(inst, isa) {
                 Ok(enc) => {
-                    return nonfatal!(
+                    return self.nonfatal(
                         errors,
                         inst,
-                        "{} must have an encoding (e.g., {})",
-                        text,
-                        isa.encoding_info().display(enc)
+                        format!(
+                            "{} must have an encoding (e.g., {})",
+                            text,
+                            isa.encoding_info().display(enc),
+                        ),
                     );
                 }
-                Err(_) => return nonfatal!(errors, inst, "{} must have an encoding", text),
+                Err(_) => {
+                    return self.nonfatal(errors, inst, format!("{} must have an encoding", text))
+                }
             }
         }
 
@@ -1775,10 +1865,10 @@ impl<'a> Verifier<'a> {
             ir::InstructionData::Store { flags, .. }
             | ir::InstructionData::StoreComplex { flags, .. } => {
                 if flags.readonly() {
-                    fatal!(
+                    self.fatal(
                         errors,
                         inst,
-                        "A store instruction cannot have the `readonly` MemFlag"
+                        "A store instruction cannot have the `readonly` MemFlag",
                     )
                 } else {
                     Ok(())
@@ -1800,12 +1890,10 @@ impl<'a> Verifier<'a> {
                 // the ExtractLane/InsertLane formats.
                 let ty = self.func.dfg.value_type(arg);
                 if u16::from(lane) >= ty.lane_count() {
-                    fatal!(
+                    self.fatal(
                         errors,
                         inst,
-                        "The lane {} does not index into the type {}",
-                        lane,
-                        ty
+                        format!("The lane {} does not index into the type {}", lane, ty,),
                     )
                 } else {
                     Ok(())
@@ -1823,10 +1911,10 @@ impl<'a> Verifier<'a> {
         if let Some(isa) = self.isa {
             if !isa.flags().enable_safepoints() && self.func.dfg[inst].opcode() == Opcode::Safepoint
             {
-                return fatal!(
+                return self.fatal(
                     errors,
                     inst,
-                    "safepoint instruction cannot be used when it is not enabled."
+                    "safepoint instruction cannot be used when it is not enabled.",
                 );
             }
         }
@@ -1895,6 +1983,13 @@ impl<'a> Verifier<'a> {
 
         verify_flags(self.func, &self.expected_cfg, self.isa, errors)?;
 
+        if !errors.is_empty() {
+            debug!(
+                "Found verifier errors in function:\n{}",
+                pretty_verifier_error(self.func, None, None, errors.clone())
+            );
+        }
+
         Ok(())
     }
 }
@@ -1989,4 +2084,34 @@ mod tests {
         let _ = verifier.typecheck_function_signature(&mut errors);
         assert_err_with_msg!(errors, "Return value at position 0 has an invalid type");
     }
+
+    #[test]
+    fn test_printing_contextual_errors() {
+        // Build function.
+        let mut func = Function::new();
+        let ebb0 = func.dfg.make_ebb();
+        func.layout.append_ebb(ebb0);
+
+        // Build instruction: v0, v1 = iconst 42
+        let inst = func.dfg.make_inst(InstructionData::UnaryImm {
+            opcode: Opcode::Iconst,
+            imm: 42.into(),
+        });
+        func.dfg.append_result(inst, types::I32);
+        func.dfg.append_result(inst, types::I32);
+        func.layout.append_inst(inst, ebb0);
+
+        // Setup verifier.
+        let mut errors = VerifierErrors::default();
+        let flags = &settings::Flags::new(settings::builder());
+        let verifier = Verifier::new(&func, flags.into());
+
+        // Now the error message, when printed, should contain the instruction sequence causing the
+        // error (i.e. v0, v1 = iconst.i32 42) and not only its entity value (i.e. inst0)
+        let _ = verifier.typecheck_results(inst, types::I32, &mut errors);
+        assert_eq!(
+            format!("{}", errors.0[0]),
+            "inst0 (v0, v1 = iconst.i32 42): has more result values than expected"
+        )
+    }
 }

From b4528beaf58d1bfc6ec85dac4a085862ee6c64ee Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Wed, 20 Nov 2019 22:40:55 -0800
Subject: [PATCH 2961/3084] Bump version to 0.51.0 (#1250)

---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  2 +-
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 19 files changed, 68 insertions(+), 68 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 0f4c7656fc..f33dec2024 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.49.0"
+version = "0.51.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.49.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.49.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.49.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.49.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.49.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.49.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.49.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.49.0" }
-cranelift-module = { path = "cranelift-module", version = "0.49.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.49.0" }
-cranelift-object = { path = "cranelift-object", version = "0.49.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.49.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.49.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.49.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.51.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.51.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.51.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.51.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.51.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.51.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.51.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.51.0" }
+cranelift-module = { path = "cranelift-module", version = "0.51.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.51.0" }
+cranelift-object = { path = "cranelift-object", version = "0.51.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.51.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.51.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.51.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index d612572949..29f3426f10 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.49.0"
+version = "0.51.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.49.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.51.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 52e4acfb4c..58e6abaae5 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.49.0"
+version = "0.51.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.49.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.49.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.49.0" }
+cranelift-codegen-shared = { path = "./shared", version = "0.51.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.51.0" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.51.0" }
 hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.9"
 log = { version = "0.4.6", default-features = false }
@@ -29,7 +29,7 @@ byteorder = { version = "1.3.2", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.49.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.51.0" }
 
 [features]
 default = ["std", "basic-blocks"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 754c8791da..aefcc7d612 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.49.0"
+version = "0.51.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.49.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.49.0" }
+cranelift-codegen-shared = { path = "../shared", version = "0.51.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.51.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 2ab2bfc492..9ab0cd75ab 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.49.0"
+version = "0.51.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index e402daf59d..324fcffb17 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.49.0"
+version = "0.51.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index cf2783e53f..ec5e399097 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.49.0"
+version = "0.51.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.49.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.51.0" }
 faerie = "0.12.0"
 goblin = "0.1.0"
 failure = "0.1.2"
@@ -18,7 +18,7 @@ target-lexicon = "0.9"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.49.0"
+version = "0.51.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index dfa66df951..0495218e81 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.49.0"
+version = "0.51.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.49.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.49.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.49.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.51.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.51.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.51.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 6ffc15d283..bdfab545a5 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.49.0"
+version = "0.51.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", default-features = false }
 target-lexicon = "0.9"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 31075aa494..98c5ae7315 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.49.0"
+version = "0.51.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.49.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.51.0" }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 thiserror = "1.0.4"
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 4497e2d692..abc5e88922 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.49.0"
+version = "0.51.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", default-features = false }
 target-lexicon = "0.9"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 2bcf9a9940..0a16f50cd9 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.49.0"
+version = "0.51.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,13 +10,13 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.49.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.51.0" }
 object = { git = "https://github.com/gimli-rs/object", rev = "cba3ed4932e4c594c5eab4f5ef6c51838f4a5056", default-features = false, features = ["write"] }
 target-lexicon = "0.9"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.49.0"
+version = "0.51.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 020b644cfa..b912f4fb4e 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.49.0"
+version = "0.51.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.49.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.51.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 0dd18f3616..413127731d 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.49.0"
+version="0.51.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 9609b92407..64fec46428 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.49.0"
+version = "0.51.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0" }
 target-lexicon = "0.9"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 7a413535d0..7a56da3549 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.49.0"
+version = "0.51.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.49.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.51.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 2e824342ea..552d6c7e48 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.49.0"
+version = "0.51.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.49.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.49.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.51.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.51.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -20,7 +20,7 @@ memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.49.0"
+version = "0.51.0"
 default-features = false
 features = ["std"]
 
@@ -32,9 +32,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.49.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.49.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.49.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.51.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.51.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.51.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 2e0ef777f0..431087d264 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.49.0"
+version = "0.51.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.49.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.51.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 2f0add8856..57230b4882 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.49.0"
+version = "0.51.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.39.2", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.49.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.49.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.49.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.51.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.51.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }

From bc9f05e5e289dcf303c227db0eb0e307314ff654 Mon Sep 17 00:00:00 2001
From: krk 
Date: Fri, 22 Nov 2019 12:38:04 +0100
Subject: [PATCH 2962/3084] Add legalization for bitrev.i128 via narrowing,
 fixes #1116 (#1229)

---
 cranelift/codegen/meta/src/shared/legalize.rs | 10 +++
 .../filetests/isa/x86/bitrev-i128-run.clif    | 46 ++++++++++
 .../filetests/legalizer/bitrev-i128.clif      | 89 +++++++++++++++++++
 3 files changed, 145 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/bitrev-i128-run.clif
 create mode 100644 cranelift/filetests/filetests/legalizer/bitrev-i128.clif

diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 7c7b121ab4..041937b8f8 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -877,6 +877,16 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
         ],
     );
 
+    narrow.legalize(
+        def!(a = bitrev.I128(x)),
+        vec![
+            def!((xl, xh) = isplit(x)),
+            def!(yh = bitrev(xl)),
+            def!(yl = bitrev(xh)),
+            def!(a = iconcat(yl, yh)),
+        ],
+    );
+
     // Floating-point sign manipulations.
     for &(ty, const_inst, minus_zero) in &[
         (F32, f32const, &Literal::bits(&imm.ieee32, 0x8000_0000)),
diff --git a/cranelift/filetests/filetests/isa/x86/bitrev-i128-run.clif b/cranelift/filetests/filetests/isa/x86/bitrev-i128-run.clif
new file mode 100644
index 0000000000..effe502d9e
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/bitrev-i128-run.clif
@@ -0,0 +1,46 @@
+test run
+target x86_64
+
+function %reverse_bits_zero() -> b1 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = iconcat v0, v0
+    v2 = bitrev.i128 v1
+    v3 = icmp eq v2, v1
+    return v3
+}
+; run
+
+function %reverse_bits_one() -> b1 {
+ebb0:
+    v0 = iconst.i64 0
+    v1 = iconst.i64 1
+    v2 = iconcat v0, v1
+
+    v3 = bitrev.i128 v2
+
+    v4 = iconst.i64 0x8000_0000_0000_0000
+    v5 = iconst.i64 0
+    v6 = iconcat v4, v5
+
+    v7 = icmp eq v3, v6
+    return v7
+}
+; run
+
+function %reverse_bits() -> b1 {
+ebb0:
+    v0 = iconst.i64 0x06AD_8667_69EC_41BA
+    v1 = iconst.i64 0x6C83_D81A_6E28_83AB
+    v2 = iconcat v0, v1
+
+    v3 = bitrev.i128 v2
+
+    v4 = iconst.i64 0xD5C11476581BC136
+    v5 = iconst.i64 0x5D823796E661B560
+    v6 = iconcat v4, v5
+
+    v7 = icmp eq v3, v6
+    return v7
+}
+; run
diff --git a/cranelift/filetests/filetests/legalizer/bitrev-i128.clif b/cranelift/filetests/filetests/legalizer/bitrev-i128.clif
new file mode 100644
index 0000000000..5a539d0e89
--- /dev/null
+++ b/cranelift/filetests/filetests/legalizer/bitrev-i128.clif
@@ -0,0 +1,89 @@
+test legalizer
+target x86_64
+
+function %reverse_bits(i128) -> i128 {
+ebb0(v0: i128):
+    v1 = bitrev.i128 v0
+    return v1
+}
+
+; check: ebb0(v2: i64, v3: i64):
+; check: v0 = iconcat v2, v3
+; check: v33 = iconst.i64 0xaaaa_aaaa_aaaa_aaaa
+; check: v6 = band v2, v33
+; check: v7 = ushr_imm v6, 1
+; check: v34 = iconst.i64 0x5555_5555_5555_5555
+; check: v8 = band v2, v34
+; check: v9 = ishl_imm v8, 1
+; check: v10 = bor v7, v9
+; check: v35 = iconst.i64 0xcccc_cccc_cccc_cccc
+; check: v11 = band v10, v35
+; check: v12 = ushr_imm v11, 2
+; check: v36 = iconst.i64 0x3333_3333_3333_3333
+; check: v13 = band v10, v36
+; check: v14 = ishl_imm v13, 2
+; check: v15 = bor v12, v14
+; check: v37 = iconst.i64 0xf0f0_f0f0_f0f0_f0f0
+; check: v16 = band v15, v37
+; check: v17 = ushr_imm v16, 4
+; check: v38 = iconst.i64 0x0f0f_0f0f_0f0f_0f0f
+; check: v18 = band v15, v38
+; check: v19 = ishl_imm v18, 4
+; check: v20 = bor v17, v19
+; check: v39 = iconst.i64 0xff00_ff00_ff00_ff00
+; check: v21 = band v20, v39
+; check: v22 = ushr_imm v21, 8
+; check: v40 = iconst.i64 0x00ff_00ff_00ff_00ff
+; check: v23 = band v20, v40
+; check: v24 = ishl_imm v23, 8
+; check: v25 = bor v22, v24
+; check: v41 = iconst.i64 0xffff_0000_ffff_0000
+; check: v26 = band v25, v41
+; check: v27 = ushr_imm v26, 16
+; check: v42 = iconst.i64 0xffff_0000_ffff
+; check: v28 = band v25, v42
+; check: v29 = ishl_imm v28, 16
+; check: v30 = bor v27, v29
+; check: v31 = ushr_imm v30, 32
+; check: v32 = ishl_imm v30, 32
+; check: v4 = bor v31, v32
+; check: v70 = iconst.i64 0xaaaa_aaaa_aaaa_aaaa
+; check: v43 = band v3, v70
+; check: v44 = ushr_imm v43, 1
+; check: v71 = iconst.i64 0x5555_5555_5555_5555
+; check: v45 = band v3, v71
+; check: v46 = ishl_imm v45, 1
+; check: v47 = bor v44, v46
+; check: v72 = iconst.i64 0xcccc_cccc_cccc_cccc
+; check: v48 = band v47, v72
+; check: v49 = ushr_imm v48, 2
+; check: v73 = iconst.i64 0x3333_3333_3333_3333
+; check: v50 = band v47, v73
+; check: v51 = ishl_imm v50, 2
+; check: v52 = bor v49, v51
+; check: v74 = iconst.i64 0xf0f0_f0f0_f0f0_f0f0
+; check: v53 = band v52, v74
+; check: v54 = ushr_imm v53, 4
+; check: v75 = iconst.i64 0x0f0f_0f0f_0f0f_0f0f
+; check: v55 = band v52, v75
+; check: v56 = ishl_imm v55, 4
+; check: v57 = bor v54, v56
+; check: v76 = iconst.i64 0xff00_ff00_ff00_ff00
+; check: v58 = band v57, v76
+; check: v59 = ushr_imm v58, 8
+; check: v77 = iconst.i64 0x00ff_00ff_00ff_00ff
+; check: v60 = band v57, v77
+; check: v61 = ishl_imm v60, 8
+; check: v62 = bor v59, v61
+; check: v78 = iconst.i64 0xffff_0000_ffff_0000
+; check: v63 = band v62, v78
+; check: v64 = ushr_imm v63, 16
+; check: v79 = iconst.i64 0xffff_0000_ffff
+; check: v65 = band v62, v79
+; check: v66 = ishl_imm v65, 16
+; check: v67 = bor v64, v66
+; check: v68 = ushr_imm v67, 32
+; check: v69 = ishl_imm v67, 32
+; check: v5 = bor v68, v69
+; check: v1 = iconcat v5, v4
+; check: return v5, v4

From b9296b792d41856bb65b29e08272b3eae7d6ca95 Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Mon, 25 Nov 2019 07:46:15 -0500
Subject: [PATCH 2963/3084] Derive Debug for FaerieProduct (#1252)

This allows using Result::unwrap_err for `Result`s with FaerieProducts.
---
 cranelift/faerie/src/backend.rs | 1 +
 cranelift/faerie/src/traps.rs   | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index db173b929b..14d8245370 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -322,6 +322,7 @@ impl Backend for FaerieBackend {
 /// This is the output of `Module`'s
 /// [`finish`](../cranelift_module/struct.Module.html#method.finish) function.
 /// It provides functions for writing out the object file to memory or a file.
+#[derive(Debug)]
 pub struct FaerieProduct {
     /// Faerie artifact with all functions, data, and links from the module defined
     pub artifact: faerie::Artifact,
diff --git a/cranelift/faerie/src/traps.rs b/cranelift/faerie/src/traps.rs
index 0a414d7981..d01619967b 100644
--- a/cranelift/faerie/src/traps.rs
+++ b/cranelift/faerie/src/traps.rs
@@ -4,6 +4,7 @@
 use cranelift_codegen::{binemit, ir};
 
 /// Record of the arguments cranelift passes to `TrapSink::trap`
+#[derive(Debug)]
 pub struct FaerieTrapSite {
     /// Offset into function
     pub offset: binemit::CodeOffset,
@@ -14,6 +15,7 @@ pub struct FaerieTrapSite {
 }
 
 /// Record of the trap sites for a given function
+#[derive(Debug)]
 pub struct FaerieTrapSink {
     /// Name of function
     pub name: String,
@@ -45,6 +47,7 @@ impl binemit::TrapSink for FaerieTrapSink {
 }
 
 /// Collection of all `FaerieTrapSink`s for the module
+#[derive(Debug)]
 pub struct FaerieTrapManifest {
     /// All `FaerieTrapSink` for the module
     pub sinks: Vec,

From dbdeb788f0ac9482e589c53e7b86f054a763ed2b Mon Sep 17 00:00:00 2001
From: data-pup 
Date: Wed, 27 Nov 2019 18:41:37 -0500
Subject: [PATCH 2964/3084] Fix jump table comment grammar (#1259)

---
 cranelift/codegen/src/ir/jumptable.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/ir/jumptable.rs b/cranelift/codegen/src/ir/jumptable.rs
index b412e807ac..372ad09837 100644
--- a/cranelift/codegen/src/ir/jumptable.rs
+++ b/cranelift/codegen/src/ir/jumptable.rs
@@ -10,7 +10,7 @@ use core::slice::{Iter, IterMut};
 
 /// Contents of a jump table.
 ///
-/// All jump tables use 0-based indexing and densely populated.
+/// All jump tables use 0-based indexing and are densely populated.
 #[derive(Clone)]
 pub struct JumpTableData {
     // Table entries.

From eac124970c7d3795c3c7a7c49e1bdc71448e2d8c Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Sun, 24 Nov 2019 19:04:10 -0500
Subject: [PATCH 2965/3084] Use a published version of `object`

This allows publishing `cranelift-object` since it no longer depends on
a git version of a dependency.

See https://github.com/bytecodealliance/cranelift/issues/1254 for
context.
---
 cranelift/object/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 0a16f50cd9..fd79e74790 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -11,7 +11,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-module = { path = "../cranelift-module", version = "0.51.0" }
-object = { git = "https://github.com/gimli-rs/object", rev = "cba3ed4932e4c594c5eab4f5ef6c51838f4a5056", default-features = false, features = ["write"] }
+object = { version = "0.16", default-features = false, features = ["write"] }
 target-lexicon = "0.9"
 
 [dependencies.cranelift-codegen]

From 9775c4be1d649537c14906b75c997f7dfbbda695 Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Sun, 24 Nov 2019 19:50:14 -0500
Subject: [PATCH 2966/3084] publish cranelift-object to crates.io

---
 cranelift/publish-all.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 413127731d..47bbd468d9 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -43,7 +43,7 @@ for crate in \
     entity bforest codegen/shared codegen/meta codegen frontend native \
     preopt \
     reader wasm module \
-    faerie umbrella simplejit
+    faerie umbrella simplejit object
 do
     echo cargo publish --manifest-path "cranelift-$crate/Cargo.toml"
 

From e1e4e0bfc63e6c706f3533280992927a6a0ffe46 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 2 Dec 2019 12:33:11 -0800
Subject: [PATCH 2967/3084] Update build badge (#1264)

---
 cranelift/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/README.md b/cranelift/README.md
index e2833df622..1242a1c8e7 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -10,7 +10,7 @@ into executable machine code.
 
 [BA]: https://bytecodealliance.org/
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
-[![Travis Status](https://travis-ci.org/bytecodealliance/cranelift.svg?branch=master)](https://travis-ci.org/bytecodealliance/cranelift)
+[![Build Status](https://github.com/bytecodealliance/cranelift/workflows/CI/badge.svg)](https://github.com/bytecodealliance/cranelift/actions)
 [![Fuzzit Status](https://app.fuzzit.dev/badge?org_id=bytecodealliance)](https://app.fuzzit.dev/orgs/bytecodealliance/dashboard)
 [![Gitter chat](https://badges.gitter.im/bytecodealliance/bytecodealliance.svg)](https://gitter.im/CraneStation/Lobby)
 ![Minimum rustc 1.37](https://img.shields.io/badge/rustc-1.37+-green.svg)

From 3e5f039333f0206d16cfea3f568d95ce8f9416f7 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 20 Nov 2019 14:17:04 -0800
Subject: [PATCH 2968/3084] Remove macros from verifier; fixes #1248

This removes `report!`, `fatal!`, and `nonfatal!` from the verifier code and replaces them with methods on `VerifierErrors`. In order to maintain similar ease-of-use, `VerifierError` is expanded with several `From` implementations that convert a tuple to a verifier error.
---
 cranelift/codegen/src/verifier/cssa.rs      |  65 +-
 cranelift/codegen/src/verifier/flags.rs     |  24 +-
 cranelift/codegen/src/verifier/liveness.rs  | 111 ++-
 cranelift/codegen/src/verifier/locations.rs | 158 ++--
 cranelift/codegen/src/verifier/mod.rs       | 803 ++++++++++----------
 5 files changed, 585 insertions(+), 576 deletions(-)

diff --git a/cranelift/codegen/src/verifier/cssa.rs b/cranelift/codegen/src/verifier/cssa.rs
index 85c6a638c2..54e88dccf3 100644
--- a/cranelift/codegen/src/verifier/cssa.rs
+++ b/cranelift/codegen/src/verifier/cssa.rs
@@ -65,13 +65,13 @@ impl<'a> CssaVerifier<'a> {
 
             for (idx, &val) in values.iter().enumerate() {
                 if !self.func.dfg.value_is_valid(val) {
-                    return fatal!(errors, val, "Invalid value in {}", vreg);
+                    return errors.fatal((val, format!("Invalid value in {}", vreg)));
                 }
                 if !self.func.dfg.value_is_attached(val) {
-                    return fatal!(errors, val, "Detached value in {}", vreg);
+                    return errors.fatal((val, format!("Detached value in {}", vreg)));
                 }
                 if self.liveness.get(val).is_none() {
-                    return fatal!(errors, val, "Value in {} has no live range", vreg);
+                    return errors.fatal((val, format!("Value in {} has no live range", vreg)));
                 };
 
                 // Check topological ordering with the previous values in the virtual register.
@@ -82,29 +82,31 @@ impl<'a> CssaVerifier<'a> {
                     let prev_ebb = self.func.layout.pp_ebb(prev_def);
 
                     if prev_def == def {
-                        return fatal!(
-                            errors,
+                        return errors.fatal((
                             val,
-                            "Values {} and {} in {} = {} defined at the same program point",
-                            prev_val,
-                            val,
-                            vreg,
-                            DisplayList(values)
-                        );
+                            format!(
+                                "Values {} and {} in {} = {} defined at the same program point",
+                                prev_val,
+                                val,
+                                vreg,
+                                DisplayList(values)
+                            ),
+                        ));
                     }
 
                     // Enforce topological ordering of defs in the virtual register.
                     if self.preorder.dominates(def_ebb, prev_ebb)
                         && self.domtree.dominates(def, prev_def, &self.func.layout)
                     {
-                        return fatal!(
-                            errors,
+                        return errors.fatal((
                             val,
-                            "Value in {} = {} def dominates previous {}",
-                            vreg,
-                            DisplayList(values),
-                            prev_val
-                        );
+                            format!(
+                                "Value in {} = {} def dominates previous {}",
+                                vreg,
+                                DisplayList(values),
+                                prev_val
+                            ),
+                        ));
                     }
                 }
 
@@ -119,14 +121,15 @@ impl<'a> CssaVerifier<'a> {
                         && self.domtree.dominates(prev_def, def, &self.func.layout)
                     {
                         if self.liveness[prev_val].overlaps_def(def, def_ebb, &self.func.layout) {
-                            return fatal!(
-                                errors,
+                            return errors.fatal((
                                 val,
-                                "Value def in {} = {} interferes with {}",
-                                vreg,
-                                DisplayList(values),
-                                prev_val
-                            );
+                                format!(
+                                    "Value def in {} = {} interferes with {}",
+                                    vreg,
+                                    DisplayList(values),
+                                    prev_val
+                                ),
+                            ));
                         } else {
                             break;
                         }
@@ -152,13 +155,13 @@ impl<'a> CssaVerifier<'a> {
 
                 for (&ebb_param, &pred_arg) in ebb_params.iter().zip(pred_args) {
                     if !self.virtregs.same_class(ebb_param, pred_arg) {
-                        return fatal!(
-                            errors,
+                        return errors.fatal((
                             pred,
-                            "{} and {} must be in the same virtual register",
-                            ebb_param,
-                            pred_arg
-                        );
+                            format!(
+                                "{} and {} must be in the same virtual register",
+                                ebb_param, pred_arg
+                            ),
+                        ));
                     }
                 }
             }
diff --git a/cranelift/codegen/src/verifier/flags.rs b/cranelift/codegen/src/verifier/flags.rs
index d39b6e0487..1748fccf71 100644
--- a/cranelift/codegen/src/verifier/flags.rs
+++ b/cranelift/codegen/src/verifier/flags.rs
@@ -67,13 +67,10 @@ impl<'a> FlagsVerifier<'a> {
                         }
                     }
                     Some(old) if old != value => {
-                        return fatal!(
-                            errors,
+                        return errors.fatal((
                             ebb,
-                            "conflicting live-in CPU flags: {} and {}",
-                            old,
-                            value
-                        );
+                            format!("conflicting live-in CPU flags: {} and {}", old, value),
+                        ));
                     }
                     x => assert_eq!(x, Some(value)),
                 }
@@ -104,7 +101,9 @@ impl<'a> FlagsVerifier<'a> {
                         // We've reached the def of `live_flags`, so it is no longer live above.
                         live_val = None;
                     } else if self.func.dfg.value_type(res).is_flags() {
-                        return fatal!(errors, inst, "{} clobbers live CPU flags in {}", res, live);
+                        errors
+                            .report((inst, format!("{} clobbers live CPU flags in {}", res, live)));
+                        return Err(());
                     }
                 }
 
@@ -116,7 +115,11 @@ impl<'a> FlagsVerifier<'a> {
                     .map_or(false, |c| c.clobbers_flags)
                     && live_val.is_some()
                 {
-                    return fatal!(errors, inst, "encoding clobbers live CPU flags in {}", live);
+                    errors.report((
+                        inst,
+                        format!("encoding clobbers live CPU flags in {}", live),
+                    ));
+                    return Err(());
                 }
             }
 
@@ -164,7 +167,10 @@ fn merge(
 ) -> VerifierStepResult<()> {
     if let Some(va) = *a {
         if b != va {
-            return fatal!(errors, inst, "conflicting live CPU flags: {} and {}", va, b);
+            return errors.fatal((
+                inst,
+                format!("conflicting live CPU flags: {} and {}", va, b),
+            ));
         }
     } else {
         *a = Some(b);
diff --git a/cranelift/codegen/src/verifier/liveness.rs b/cranelift/codegen/src/verifier/liveness.rs
index 218bb60fdc..4c4940f356 100644
--- a/cranelift/codegen/src/verifier/liveness.rs
+++ b/cranelift/codegen/src/verifier/liveness.rs
@@ -54,7 +54,9 @@ impl<'a> LivenessVerifier<'a> {
             for &val in self.func.dfg.ebb_params(ebb) {
                 let lr = match self.liveness.get(val) {
                     Some(lr) => lr,
-                    None => return fatal!(errors, ebb, "EBB arg {} has no live range", val),
+                    None => {
+                        return errors.fatal((ebb, format!("EBB arg {} has no live range", val)))
+                    }
                 };
                 self.check_lr(ebb.into(), val, lr, errors)?;
             }
@@ -72,30 +74,32 @@ impl<'a> LivenessVerifier<'a> {
                 for &val in self.func.dfg.inst_results(inst) {
                     let lr = match self.liveness.get(val) {
                         Some(lr) => lr,
-                        None => return fatal!(errors, inst, "{} has no live range", val),
+                        None => return errors.fatal((inst, format!("{} has no live range", val))),
                     };
                     self.check_lr(inst.into(), val, lr, errors)?;
 
                     if encoding.is_legal() {
                         // A legal instruction is not allowed to define ghost values.
                         if lr.affinity.is_unassigned() {
-                            return fatal!(
-                                errors,
+                            return errors.fatal((
                                 inst,
-                                "{} is a ghost value defined by a real [{}] instruction",
-                                val,
-                                self.isa.encoding_info().display(encoding)
-                            );
+                                format!(
+                                    "{} is a ghost value defined by a real [{}] instruction",
+                                    val,
+                                    self.isa.encoding_info().display(encoding)
+                                ),
+                            ));
                         }
                     } else if !lr.affinity.is_unassigned() {
                         // A non-encoded instruction can only define ghost values.
-                        return fatal!(
-                            errors,
+                        return errors.fatal((
                             inst,
-                            "{} is a real {} value defined by a ghost instruction",
-                            val,
-                            lr.affinity.display(&self.isa.register_info())
-                        );
+                            format!(
+                                "{} is a real {} value defined by a ghost instruction",
+                                val,
+                                lr.affinity.display(&self.isa.register_info())
+                            ),
+                        ));
                     }
                 }
 
@@ -103,23 +107,24 @@ impl<'a> LivenessVerifier<'a> {
                 for &val in self.func.dfg.inst_args(inst) {
                     let lr = match self.liveness.get(val) {
                         Some(lr) => lr,
-                        None => return fatal!(errors, inst, "{} has no live range", val),
+                        None => return errors.fatal((inst, format!("{} has no live range", val))),
                     };
 
                     debug_assert!(self.func.layout.inst_ebb(inst).unwrap() == ebb);
                     if !lr.reaches_use(inst, ebb, &self.func.layout) {
-                        return fatal!(errors, inst, "{} is not live at this use", val);
+                        return errors.fatal((inst, format!("{} is not live at this use", val)));
                     }
 
                     // A legal instruction is not allowed to depend on ghost values.
                     if encoding.is_legal() && lr.affinity.is_unassigned() {
-                        return fatal!(
-                            errors,
+                        return errors.fatal((
                             inst,
-                            "{} is a ghost value used by a real [{}] instruction",
-                            val,
-                            self.isa.encoding_info().display(encoding)
-                        );
+                            format!(
+                                "{} is a ghost value used by a real [{}] instruction",
+                                val,
+                                self.isa.encoding_info().display(encoding),
+                            ),
+                        ));
                     }
                 }
             }
@@ -142,17 +147,14 @@ impl<'a> LivenessVerifier<'a> {
             ExpandedProgramPoint::Inst(i) => i.into(),
         };
         if lr.def() != def {
-            return fatal!(
-                errors,
+            return errors.fatal((
                 loc,
-                "Wrong live range def ({}) for {}",
-                lr.def(),
-                val
-            );
+                format!("Wrong live range def ({}) for {}", lr.def(), val),
+            ));
         }
         if lr.is_dead() {
             if !lr.is_local() {
-                return fatal!(errors, loc, "Dead live range {} should be local", val);
+                return errors.fatal((loc, format!("Dead live range {} should be local", val)));
             } else {
                 return Ok(());
             }
@@ -163,17 +165,14 @@ impl<'a> LivenessVerifier<'a> {
         };
         match lr.def_local_end().into() {
             ExpandedProgramPoint::Ebb(e) => {
-                return fatal!(
-                    errors,
+                return errors.fatal((
                     loc,
-                    "Def local range for {} can't end at {}",
-                    val,
-                    e
-                );
+                    format!("Def local range for {} can't end at {}", val, e),
+                ));
             }
             ExpandedProgramPoint::Inst(i) => {
                 if self.func.layout.inst_ebb(i) != Some(def_ebb) {
-                    return fatal!(errors, loc, "Def local end for {} in wrong ebb", val);
+                    return errors.fatal((loc, format!("Def local end for {} in wrong ebb", val)));
                 }
             }
         }
@@ -181,25 +180,21 @@ impl<'a> LivenessVerifier<'a> {
         // Now check the live-in intervals against the CFG.
         for (mut ebb, end) in lr.liveins() {
             if !l.is_ebb_inserted(ebb) {
-                return fatal!(
-                    errors,
+                return errors.fatal((
                     loc,
-                    "{} livein at {} which is not in the layout",
-                    val,
-                    ebb
-                );
+                    format!("{} livein at {} which is not in the layout", val, ebb),
+                ));
             }
             let end_ebb = match l.inst_ebb(end) {
                 Some(e) => e,
                 None => {
-                    return fatal!(
-                        errors,
+                    return errors.fatal((
                         loc,
-                        "{} livein for {} ends at {} which is not in the layout",
-                        val,
-                        ebb,
-                        end
-                    );
+                        format!(
+                            "{} livein for {} ends at {} which is not in the layout",
+                            val, ebb, end
+                        ),
+                    ));
                 }
             };
 
@@ -208,13 +203,10 @@ impl<'a> LivenessVerifier<'a> {
                 // If `val` is live-in at `ebb`, it must be live at all the predecessors.
                 for BasicBlock { inst: pred, ebb } in self.cfg.pred_iter(ebb) {
                     if !lr.reaches_use(pred, ebb, &self.func.layout) {
-                        return fatal!(
-                            errors,
+                        return errors.fatal((
                             pred,
-                            "{} is live in to {} but not live at predecessor",
-                            val,
-                            ebb
-                        );
+                            format!("{} is live in to {} but not live at predecessor", val, ebb),
+                        ));
                     }
                 }
 
@@ -224,13 +216,10 @@ impl<'a> LivenessVerifier<'a> {
                 ebb = match l.next_ebb(ebb) {
                     Some(e) => e,
                     None => {
-                        return fatal!(
-                            errors,
+                        return errors.fatal((
                             loc,
-                            "end of {} livein ({}) never reached",
-                            val,
-                            end_ebb
-                        );
+                            format!("end of {} livein ({}) never reached", val, end_ebb),
+                        ));
                     }
                 };
             }
diff --git a/cranelift/codegen/src/verifier/locations.rs b/cranelift/codegen/src/verifier/locations.rs
index 6a32cec0f6..fe180b8e81 100644
--- a/cranelift/codegen/src/verifier/locations.rs
+++ b/cranelift/codegen/src/verifier/locations.rs
@@ -104,14 +104,15 @@ impl<'a> LocationVerifier<'a> {
         }
 
         // TODO: We could give a better error message here.
-        fatal!(
-            errors,
+        errors.fatal((
             inst,
-            "{} constraints not satisfied in: {}\n{}",
-            self.encinfo.display(enc),
-            self.func.dfg.display_inst(inst, self.isa),
-            self.func.display(self.isa)
-        )
+            format!(
+                "{} constraints not satisfied in: {}\n{}",
+                self.encinfo.display(enc),
+                self.func.dfg.display_inst(inst, self.isa),
+                self.func.display(self.isa),
+            ),
+        ))
     }
 
     /// Check that the result values produced by a ghost instruction are not assigned a value
@@ -126,13 +127,14 @@ impl<'a> LocationVerifier<'a> {
         for &res in results {
             let loc = self.func.locations[res];
             if loc.is_assigned() {
-                return fatal!(
-                    errors,
+                return errors.fatal((
                     inst,
-                    "ghost result {} value must not have a location ({}).",
-                    res,
-                    loc.display(&self.reginfo)
-                );
+                    format!(
+                        "ghost result {} value must not have a location ({}).",
+                        res,
+                        loc.display(&self.reginfo)
+                    ),
+                ));
             }
         }
 
@@ -214,50 +216,51 @@ impl<'a> LocationVerifier<'a> {
             ir::ArgumentLoc::Unassigned => {}
             ir::ArgumentLoc::Reg(reg) => {
                 if loc != ir::ValueLoc::Reg(reg) {
-                    return fatal!(
-                        errors,
+                    return errors.fatal((
                         inst,
-                        "ABI expects {} in {}, got {}",
-                        value,
-                        abi.location.display(&self.reginfo),
-                        loc.display(&self.reginfo)
-                    );
+                        format!(
+                            "ABI expects {} in {}, got {}",
+                            value,
+                            abi.location.display(&self.reginfo),
+                            loc.display(&self.reginfo),
+                        ),
+                    ));
                 }
             }
             ir::ArgumentLoc::Stack(offset) => {
                 if let ir::ValueLoc::Stack(ss) = loc {
                     let slot = &self.func.stack_slots[ss];
                     if slot.kind != want_kind {
-                        return fatal!(
-                            errors,
+                        return errors.fatal((
                             inst,
-                            "call argument {} should be in a {} slot, but {} is {}",
-                            value,
-                            want_kind,
-                            ss,
-                            slot.kind
-                        );
+                            format!(
+                                "call argument {} should be in a {} slot, but {} is {}",
+                                value, want_kind, ss, slot.kind
+                            ),
+                        ));
                     }
                     if slot.offset.unwrap() != offset {
-                        return fatal!(
-                            errors,
+                        return errors.fatal((
                             inst,
-                            "ABI expects {} at stack offset {}, but {} is at {}",
-                            value,
-                            offset,
-                            ss,
-                            slot.offset.unwrap()
-                        );
+                            format!(
+                                "ABI expects {} at stack offset {}, but {} is at {}",
+                                value,
+                                offset,
+                                ss,
+                                slot.offset.unwrap()
+                            ),
+                        ));
                     }
                 } else {
-                    return fatal!(
-                        errors,
+                    return errors.fatal((
                         inst,
-                        "ABI expects {} at stack offset {}, got {}",
-                        value,
-                        offset,
-                        loc.display(&self.reginfo)
-                    );
+                        format!(
+                            "ABI expects {} at stack offset {}, got {}",
+                            value,
+                            offset,
+                            loc.display(&self.reginfo)
+                        ),
+                    ));
                 }
             }
         }
@@ -281,21 +284,23 @@ impl<'a> LocationVerifier<'a> {
 
         if let Some(d) = divert.diversion(arg) {
             if d.to != src {
-                return fatal!(
-                    errors,
+                return errors.fatal((
                     inst,
-                    "inconsistent with current diversion to {}",
-                    d.to.display(&self.reginfo)
-                );
+                    format!(
+                        "inconsistent with current diversion to {}",
+                        d.to.display(&self.reginfo)
+                    ),
+                ));
             }
         } else if self.func.locations[arg] != src {
-            return fatal!(
-                errors,
+            return errors.fatal((
                 inst,
-                "inconsistent with global location {} ({})",
-                self.func.locations[arg].display(&self.reginfo),
-                self.func.dfg.display_inst(inst, None)
-            );
+                format!(
+                    "inconsistent with global location {} ({})",
+                    self.func.locations[arg].display(&self.reginfo),
+                    self.func.dfg.display_inst(inst, None)
+                ),
+            ));
         }
 
         divert.apply(&self.func.dfg[inst]);
@@ -338,14 +343,15 @@ impl<'a> LocationVerifier<'a> {
                             val_to_remove.push(value)
                         }
                     } else if lr.is_livein(ebb, &self.func.layout) {
-                        return fatal!(
-                            errors,
+                        return errors.fatal((
                             inst,
-                            "SingleDest: {} is diverted to {} and live in to {}",
-                            value,
-                            d.to.display(&self.reginfo),
-                            ebb
-                        );
+                            format!(
+                                "SingleDest: {} is diverted to {} and live in to {}",
+                                value,
+                                d.to.display(&self.reginfo),
+                                ebb,
+                            ),
+                        ));
                     }
                 }
                 if is_after_branch && unique_predecessor {
@@ -360,26 +366,28 @@ impl<'a> LocationVerifier<'a> {
                     let lr = &liveness[value];
                     if let Some(ebb) = ebb {
                         if lr.is_livein(ebb, &self.func.layout) {
-                            return fatal!(
-                                errors,
+                            return errors.fatal((
                                 inst,
-                                "Table.default: {} is diverted to {} and live in to {}",
-                                value,
-                                d.to.display(&self.reginfo),
-                                ebb
-                            );
+                                format!(
+                                    "Table.default: {} is diverted to {} and live in to {}",
+                                    value,
+                                    d.to.display(&self.reginfo),
+                                    ebb,
+                                ),
+                            ));
                         }
                     }
                     for ebb in self.func.jump_tables[jt].iter() {
                         if lr.is_livein(*ebb, &self.func.layout) {
-                            return fatal!(
-                                errors,
+                            return errors.fatal((
                                 inst,
-                                "Table.case: {} is diverted to {} and live in to {}",
-                                value,
-                                d.to.display(&self.reginfo),
-                                ebb
-                            );
+                                format!(
+                                    "Table.case: {} is diverted to {} and live in to {}",
+                                    value,
+                                    d.to.display(&self.reginfo),
+                                    ebb,
+                                ),
+                            ));
                         }
                     }
                 }
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 61b653e469..30990dbf9b 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -85,46 +85,6 @@ pub use self::cssa::verify_cssa;
 pub use self::liveness::verify_liveness;
 pub use self::locations::verify_locations;
 
-/// Report an error.
-///
-/// The first argument must be a `&mut VerifierErrors` reference, and the following
-/// argument defines the location of the error and must implement `Into`.
-/// Finally, subsequent arguments will be formatted using `format!()` and set
-/// as the error message.
-macro_rules! report {
-    ( $errors: expr, $loc: expr, $msg: tt ) => {
-        $errors.0.push(crate::verifier::VerifierError {
-            location: $loc.into(),
-            context: None,
-            message: String::from($msg),
-        })
-    };
-
-    ( $errors: expr, $loc: expr, $fmt: tt, $( $arg: expr ),+ ) => {
-        $errors.0.push(crate::verifier::VerifierError {
-            location: $loc.into(),
-            context: None,
-            message: format!( $fmt, $( $arg ),+ ),
-        })
-    };
-}
-
-/// Diagnose a fatal error, and return `Err`.
-macro_rules! fatal {
-    ( $( $arg: expr ),+ ) => ({
-        report!( $( $arg ),+ );
-        Err(())
-    });
-}
-
-/// Diagnose a non-fatal error, and return `Ok`.
-macro_rules! nonfatal {
-    ( $( $arg: expr ),+ ) => ({
-        report!( $( $arg ),+ );
-        Ok(())
-    });
-}
-
 mod cssa;
 mod flags;
 mod liveness;
@@ -151,6 +111,50 @@ fn format_context(context: &Option) -> String {
     }
 }
 
+/// Convenience converter for making error-reporting less verbose.
+///
+/// Converts a tuple of `(location, context, message)` to a `VerifierError`.
+/// ```
+/// use cranelift_codegen::verifier::VerifierErrors;
+/// use cranelift_codegen::ir::Inst;
+/// let mut errors = VerifierErrors::new();
+/// errors.report((Inst::from_u32(42), "v3 = iadd v1, v2", "iadd cannot be used with values of this type"));
+/// // note the double parenthenses to use this syntax
+/// ```
+impl From<(L, C, M)> for VerifierError
+where
+    L: Into,
+    C: Into,
+    M: Into,
+{
+    fn from(items: (L, C, M)) -> Self {
+        let (location, context, message) = items;
+        Self {
+            location: location.into(),
+            context: Some(context.into()),
+            message: message.into(),
+        }
+    }
+}
+
+/// Convenience converter for making error-reporting less verbose.
+///
+/// Same as above but without `context`.
+impl From<(L, M)> for VerifierError
+where
+    L: Into,
+    M: Into,
+{
+    fn from(items: (L, M)) -> Self {
+        let (location, message) = items;
+        Self {
+            location: location.into(),
+            context: None,
+            message: message.into(),
+        }
+    }
+}
+
 /// Result of a step in the verification process.
 ///
 /// Functions that return `VerifierStepResult<()>` should also take a
@@ -202,6 +206,23 @@ impl VerifierErrors {
             Err(())
         }
     }
+
+    /// Report an error, adding it to the list of errors.
+    pub fn report(&mut self, error: impl Into) {
+        self.0.push(error.into());
+    }
+
+    /// Report a fatal error and return `Err`.
+    pub fn fatal(&mut self, error: impl Into) -> VerifierStepResult<()> {
+        self.report(error);
+        Err(())
+    }
+
+    /// Report a non-fatal error and return `Ok`.
+    pub fn nonfatal(&mut self, error: impl Into) -> VerifierStepResult<()> {
+        self.report(error);
+        Ok(())
+    }
 }
 
 impl From> for VerifierErrors {
@@ -291,39 +312,10 @@ impl<'a> Verifier<'a> {
         }
     }
 
-    /// Helper for appending a contextual error to the verifier error list.
+    /// Determine a contextual error string for an instruction.
     #[inline]
-    fn append_error(&self, inst: Inst, errors: &mut VerifierErrors, message: impl Into) {
-        let location = inst.into();
-        let context = Some(self.func.dfg.display_inst(inst, self.isa).to_string());
-        let message = message.into();
-        errors.0.push(VerifierError {
-            location,
-            context,
-            message,
-        });
-    }
-
-    /// Diagnose a fatal error, and return `Err`.
-    fn fatal(
-        &self,
-        errors: &mut VerifierErrors,
-        inst: Inst,
-        message: impl Into,
-    ) -> VerifierStepResult<()> {
-        self.append_error(inst, errors, message);
-        Err(())
-    }
-
-    /// Diagnose a non-fatal error, and return `Ok`.
-    fn nonfatal(
-        &self,
-        errors: &mut VerifierErrors,
-        inst: Inst,
-        message: impl Into,
-    ) -> VerifierStepResult<()> {
-        self.append_error(inst, errors, message);
-        Ok(())
+    fn context(&self, inst: Inst) -> String {
+        self.func.dfg.display_inst(inst, self.isa).to_string()
     }
 
     // Check for:
@@ -344,12 +336,10 @@ impl<'a> Verifier<'a> {
                     | ir::GlobalValueData::IAddImm { base, .. } => {
                         if seen.insert(base).is_some() {
                             if !cycle_seen {
-                                report!(
-                                    errors,
+                                errors.report((
                                     gv,
-                                    "global value cycle: {}",
-                                    DisplayList(seen.as_slice())
-                                );
+                                    format!("global value cycle: {}", DisplayList(seen.as_slice())),
+                                ));
                                 // ensures we don't report the cycle multiple times
                                 cycle_seen = true;
                             }
@@ -369,29 +359,27 @@ impl<'a> Verifier<'a> {
                         .special_param(ir::ArgumentPurpose::VMContext)
                         .is_none()
                     {
-                        report!(errors, gv, "undeclared vmctx reference {}", gv);
+                        errors.report((gv, format!("undeclared vmctx reference {}", gv)));
                     }
                 }
                 ir::GlobalValueData::IAddImm {
                     base, global_type, ..
                 } => {
                     if !global_type.is_int() {
-                        report!(
-                            errors,
+                        errors.report((
                             gv,
-                            "iadd_imm global value with non-int type {}",
-                            global_type
-                        );
+                            format!("iadd_imm global value with non-int type {}", global_type),
+                        ));
                     } else if let Some(isa) = self.isa {
                         let base_type = self.func.global_values[base].global_type(isa);
                         if global_type != base_type {
-                            report!(
-                                errors,
+                            errors.report((
                                 gv,
-                                "iadd_imm type {} differs from operand type {}",
-                                global_type,
-                                base_type
-                            );
+                                format!(
+                                    "iadd_imm type {} differs from operand type {}",
+                                    global_type, base_type
+                                ),
+                            ));
                         }
                     }
                 }
@@ -400,14 +388,13 @@ impl<'a> Verifier<'a> {
                         let base_type = self.func.global_values[base].global_type(isa);
                         let pointer_type = isa.pointer_type();
                         if base_type != pointer_type {
-                            report!(
-                                errors,
+                            errors.report((
                                 gv,
-                                "base {} has type {}, which is not the pointer type {}",
-                                base,
-                                base_type,
-                                pointer_type
-                            );
+                                format!(
+                                    "base {} has type {}, which is not the pointer type {}",
+                                    base, base_type, pointer_type
+                                ),
+                            ));
                         }
                     }
                 }
@@ -424,36 +411,37 @@ impl<'a> Verifier<'a> {
             for (heap, heap_data) in &self.func.heaps {
                 let base = heap_data.base;
                 if !self.func.global_values.is_valid(base) {
-                    return nonfatal!(errors, heap, "invalid base global value {}", base);
+                    return errors.nonfatal((heap, format!("invalid base global value {}", base)));
                 }
 
                 let pointer_type = isa.pointer_type();
                 let base_type = self.func.global_values[base].global_type(isa);
                 if base_type != pointer_type {
-                    report!(
-                        errors,
+                    errors.report((
                         heap,
-                        "heap base has type {}, which is not the pointer type {}",
-                        base_type,
-                        pointer_type
-                    );
+                        format!(
+                            "heap base has type {}, which is not the pointer type {}",
+                            base_type, pointer_type
+                        ),
+                    ));
                 }
 
                 if let ir::HeapStyle::Dynamic { bound_gv, .. } = heap_data.style {
                     if !self.func.global_values.is_valid(bound_gv) {
-                        return nonfatal!(errors, heap, "invalid bound global value {}", bound_gv);
+                        return errors
+                            .nonfatal((heap, format!("invalid bound global value {}", bound_gv)));
                     }
 
                     let index_type = heap_data.index_type;
                     let bound_type = self.func.global_values[bound_gv].global_type(isa);
                     if index_type != bound_type {
-                        report!(
-                            errors,
+                        errors.report((
                             heap,
-                            "heap index type {} differs from the type of its bound, {}",
-                            index_type,
-                            bound_type
-                        );
+                            format!(
+                                "heap index type {} differs from the type of its bound, {}",
+                                index_type, bound_type
+                            ),
+                        ));
                     }
                 }
             }
@@ -467,36 +455,37 @@ impl<'a> Verifier<'a> {
             for (table, table_data) in &self.func.tables {
                 let base = table_data.base_gv;
                 if !self.func.global_values.is_valid(base) {
-                    return nonfatal!(errors, table, "invalid base global value {}", base);
+                    return errors.nonfatal((table, format!("invalid base global value {}", base)));
                 }
 
                 let pointer_type = isa.pointer_type();
                 let base_type = self.func.global_values[base].global_type(isa);
                 if base_type != pointer_type {
-                    report!(
-                        errors,
+                    errors.report((
                         table,
-                        "table base has type {}, which is not the pointer type {}",
-                        base_type,
-                        pointer_type
-                    );
+                        format!(
+                            "table base has type {}, which is not the pointer type {}",
+                            base_type, pointer_type
+                        ),
+                    ));
                 }
 
                 let bound_gv = table_data.bound_gv;
                 if !self.func.global_values.is_valid(bound_gv) {
-                    return nonfatal!(errors, table, "invalid bound global value {}", bound_gv);
+                    return errors
+                        .nonfatal((table, format!("invalid bound global value {}", bound_gv)));
                 }
 
                 let index_type = table_data.index_type;
                 let bound_type = self.func.global_values[bound_gv].global_type(isa);
                 if index_type != bound_type {
-                    report!(
-                        errors,
+                    errors.report((
                         table,
-                        "table index type {} differs from the type of its bound, {}",
-                        index_type,
-                        bound_type
-                    );
+                        format!(
+                            "table index type {} differs from the type of its bound, {}",
+                            index_type, bound_type
+                        ),
+                    ));
                 }
             }
         }
@@ -519,7 +508,7 @@ impl<'a> Verifier<'a> {
     fn encodable_as_bb(&self, ebb: Ebb, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         match self.func.is_ebb_basic(ebb) {
             Ok(()) => Ok(()),
-            Err((inst, message)) => self.fatal(errors, inst, message),
+            Err((inst, message)) => errors.fatal((inst, self.context(inst), message)),
         }
     }
 
@@ -534,31 +523,27 @@ impl<'a> Verifier<'a> {
 
         if is_terminator && !is_last_inst {
             // Terminating instructions only occur at the end of blocks.
-            return self.fatal(
-                errors,
+            return errors.fatal((
                 inst,
+                self.context(inst),
                 format!(
                     "a terminator instruction was encountered before the end of {}",
                     ebb
                 ),
-            );
+            ));
         }
         if is_last_inst && !is_terminator {
-            return fatal!(
-                errors,
-                ebb,
-                "block does not end in a terminator instruction"
-            );
+            return errors.fatal((ebb, "block does not end in a terminator instruction"));
         }
 
         // Instructions belong to the correct ebb.
         let inst_ebb = self.func.layout.inst_ebb(inst);
         if inst_ebb != Some(ebb) {
-            return self.fatal(
-                errors,
+            return errors.fatal((
                 inst,
+                self.context(inst),
                 format!("should belong to {} not {:?}", ebb, inst_ebb),
-            );
+            ));
         }
 
         // Parameters belong to the correct ebb.
@@ -566,11 +551,11 @@ impl<'a> Verifier<'a> {
             match self.func.dfg.value_def(arg) {
                 ValueDef::Param(arg_ebb, _) => {
                     if ebb != arg_ebb {
-                        return fatal!(errors, arg, "does not belong to {}", ebb);
+                        return errors.fatal((arg, format!("does not belong to {}", ebb)));
                     }
                 }
                 _ => {
-                    return fatal!(errors, arg, "expected an argument, found a result");
+                    return errors.fatal((arg, "expected an argument, found a result"));
                 }
             }
         }
@@ -588,11 +573,11 @@ impl<'a> Verifier<'a> {
 
         // The instruction format matches the opcode
         if inst_data.opcode().format() != InstructionFormat::from(inst_data) {
-            return self.fatal(
-                errors,
+            return errors.fatal((
                 inst,
+                self.context(inst),
                 "instruction opcode doesn't match instruction format",
-            );
+            ));
         }
 
         let num_fixed_results = inst_data.opcode().constraints().num_fixed_results();
@@ -605,14 +590,14 @@ impl<'a> Verifier<'a> {
         // All result values for multi-valued instructions are created
         let got_results = dfg.inst_results(inst).len();
         if got_results != total_results {
-            return self.fatal(
-                errors,
+            return errors.fatal((
                 inst,
+                self.context(inst),
                 format!(
                     "expected {} result values, found {}",
                     total_results, got_results,
                 ),
-            );
+            ));
         }
 
         self.verify_entity_references(inst, errors)
@@ -631,13 +616,11 @@ impl<'a> Verifier<'a> {
             // All used values must be attached to something.
             let original = self.func.dfg.resolve_aliases(arg);
             if !self.func.dfg.value_is_attached(original) {
-                report!(
-                    errors,
+                errors.report((
                     inst,
-                    "argument {} -> {} is not attached",
-                    arg,
-                    original
-                );
+                    self.context(inst),
+                    format!("argument {} -> {} is not attached", arg, original),
+                ));
             }
         }
 
@@ -737,14 +720,18 @@ impl<'a> Verifier<'a> {
             } => {
                 if let Some(isa) = &self.isa {
                     if !isa.flags().enable_pinned_reg() {
-                        return self.fatal(
-                            errors,
+                        return errors.fatal((
                             inst,
+                            self.context(inst),
                             "GetPinnedReg/SetPinnedReg cannot be used without enable_pinned_reg",
-                        );
+                        ));
                     }
                 } else {
-                    return self.fatal(errors, inst, "GetPinnedReg/SetPinnedReg need an ISA!");
+                    return errors.fatal((
+                        inst,
+                        self.context(inst),
+                        "GetPinnedReg/SetPinnedReg need an ISA!",
+                    ));
                 }
             }
 
@@ -796,11 +783,11 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.dfg.ebb_is_valid(e) || !self.func.layout.is_ebb_inserted(e) {
-            return fatal!(errors, loc, "invalid ebb reference {}", e);
+            return errors.fatal((loc, format!("invalid ebb reference {}", e)));
         }
         if let Some(entry_block) = self.func.layout.entry_block() {
             if e == entry_block {
-                return fatal!(errors, loc, "invalid reference to entry ebb {}", e);
+                return errors.fatal((loc, format!("invalid reference to entry ebb {}", e)));
             }
         }
         Ok(())
@@ -813,7 +800,11 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.dfg.signatures.is_valid(s) {
-            self.fatal(errors, inst, format!("invalid signature reference {}", s))
+            errors.fatal((
+                inst,
+                self.context(inst),
+                format!("invalid signature reference {}", s),
+            ))
         } else {
             Ok(())
         }
@@ -826,7 +817,11 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.dfg.ext_funcs.is_valid(f) {
-            self.nonfatal(errors, inst, format!("invalid function reference {}", f))
+            errors.nonfatal((
+                inst,
+                self.context(inst),
+                format!("invalid function reference {}", f),
+            ))
         } else {
             Ok(())
         }
@@ -839,7 +834,11 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.stack_slots.is_valid(ss) {
-            self.nonfatal(errors, inst, format!("invalid stack slot {}", ss))
+            errors.nonfatal((
+                inst,
+                self.context(inst),
+                format!("invalid stack slot {}", ss),
+            ))
         } else {
             Ok(())
         }
@@ -852,7 +851,11 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.global_values.is_valid(gv) {
-            self.nonfatal(errors, inst, format!("invalid global value {}", gv))
+            errors.nonfatal((
+                inst,
+                self.context(inst),
+                format!("invalid global value {}", gv),
+            ))
         } else {
             Ok(())
         }
@@ -865,7 +868,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.heaps.is_valid(heap) {
-            self.nonfatal(errors, inst, format!("invalid heap {}", heap))
+            errors.nonfatal((inst, self.context(inst), format!("invalid heap {}", heap)))
         } else {
             Ok(())
         }
@@ -878,7 +881,7 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.tables.is_valid(table) {
-            self.nonfatal(errors, inst, format!("invalid table {}", table))
+            errors.nonfatal((inst, self.context(inst), format!("invalid table {}", table)))
         } else {
             Ok(())
         }
@@ -891,11 +894,11 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !l.is_valid(&self.func.dfg.value_lists) {
-            self.nonfatal(
-                errors,
+            errors.nonfatal((
                 inst,
+                self.context(inst),
                 format!("invalid value list reference {:?}", l),
-            )
+            ))
         } else {
             Ok(())
         }
@@ -908,7 +911,11 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         if !self.func.jump_tables.is_valid(j) {
-            self.nonfatal(errors, inst, format!("invalid jump table reference {}", j))
+            errors.nonfatal((
+                inst,
+                self.context(inst),
+                format!("invalid jump table reference {}", j),
+            ))
         } else {
             Ok(())
         }
@@ -922,7 +929,11 @@ impl<'a> Verifier<'a> {
     ) -> VerifierStepResult<()> {
         let dfg = &self.func.dfg;
         if !dfg.value_is_valid(v) {
-            self.nonfatal(errors, loc_inst, format!("invalid value reference {}", v))
+            errors.nonfatal((
+                loc_inst,
+                self.context(loc_inst),
+                format!("invalid value reference {}", v),
+            ))
         } else {
             Ok(())
         }
@@ -945,23 +956,19 @@ impl<'a> Verifier<'a> {
             ValueDef::Result(def_inst, _) => {
                 // Value is defined by an instruction that exists.
                 if !dfg.inst_is_valid(def_inst) {
-                    return fatal!(
-                        errors,
+                    return errors.fatal((
                         loc_inst,
-                        "{} is defined by invalid instruction {}",
-                        v,
-                        def_inst
-                    );
+                        self.context(loc_inst),
+                        format!("{} is defined by invalid instruction {}", v, def_inst),
+                    ));
                 }
                 // Defining instruction is inserted in an EBB.
                 if self.func.layout.inst_ebb(def_inst) == None {
-                    return fatal!(
-                        errors,
+                    return errors.fatal((
                         loc_inst,
-                        "{} is defined by {} which has no EBB",
-                        v,
-                        def_inst
-                    );
+                        self.context(loc_inst),
+                        format!("{} is defined by {} which has no EBB", v, def_inst),
+                    ));
                 }
                 // Defining instruction dominates the instruction that uses the value.
                 if is_reachable {
@@ -969,39 +976,37 @@ impl<'a> Verifier<'a> {
                         .expected_domtree
                         .dominates(def_inst, loc_inst, &self.func.layout)
                     {
-                        return fatal!(
-                            errors,
+                        return errors.fatal((
                             loc_inst,
-                            "uses value {} from non-dominating {}",
-                            v,
-                            def_inst
-                        );
+                            self.context(loc_inst),
+                            format!("uses value {} from non-dominating {}", v, def_inst),
+                        ));
                     }
                     if def_inst == loc_inst {
-                        return self.fatal(
-                            errors,
+                        return errors.fatal((
                             loc_inst,
+                            self.context(loc_inst),
                             format!("uses value {} from itself", v),
-                        );
+                        ));
                     }
                 }
             }
             ValueDef::Param(ebb, _) => {
                 // Value is defined by an existing EBB.
                 if !dfg.ebb_is_valid(ebb) {
-                    return self.fatal(
-                        errors,
+                    return errors.fatal((
                         loc_inst,
+                        self.context(loc_inst),
                         format!("{} is defined by invalid EBB {}", v, ebb),
-                    );
+                    ));
                 }
                 // Defining EBB is inserted in the layout
                 if !self.func.layout.is_ebb_inserted(ebb) {
-                    return self.fatal(
-                        errors,
+                    return errors.fatal((
                         loc_inst,
+                        self.context(loc_inst),
                         format!("{} is defined by {} which is not in the layout", v, ebb),
-                    );
+                    ));
                 }
                 // The defining EBB dominates the instruction using this value.
                 if is_reachable
@@ -1009,11 +1014,11 @@ impl<'a> Verifier<'a> {
                         .expected_domtree
                         .dominates(ebb, loc_inst, &self.func.layout)
                 {
-                    return self.fatal(
-                        errors,
+                    return errors.fatal((
                         loc_inst,
+                        self.context(loc_inst),
                         format!("uses value arg from non-dominating {}", ebb),
-                    );
+                    ));
                 }
             }
         }
@@ -1031,22 +1036,20 @@ impl<'a> Verifier<'a> {
         match self.func.dfg.value_def(v) {
             ValueDef::Result(def_inst, _) => {
                 if def_inst != loc_inst {
-                    fatal!(
-                        errors,
+                    errors.fatal((
                         loc_inst,
-                        "instruction result {} is not defined by the instruction",
-                        v
-                    )
+                        self.context(loc_inst),
+                        format!("instruction result {} is not defined by the instruction", v),
+                    ))
                 } else {
                     Ok(())
                 }
             }
-            ValueDef::Param(_, _) => fatal!(
-                errors,
+            ValueDef::Param(_, _) => errors.fatal((
                 loc_inst,
-                "instruction result {} is not defined by the instruction",
-                v
-            ),
+                self.context(loc_inst),
+                format!("instruction result {} is not defined by the instruction", v),
+            )),
         }
     }
 
@@ -1060,15 +1063,14 @@ impl<'a> Verifier<'a> {
         let value_type = self.func.dfg.value_type(arg);
 
         if typ.lane_bits() < value_type.lane_bits() {
-            self.fatal(
-                errors,
+            errors.fatal((
                 inst,
                 format!(
                     "The bitcast argument {} doesn't fit in a type of {} bits",
                     arg,
-                    typ.lane_bits(),
+                    typ.lane_bits()
                 ),
-            )
+            ))
         } else {
             Ok(())
         }
@@ -1086,23 +1088,21 @@ impl<'a> Verifier<'a> {
             let expected = self.expected_domtree.idom(ebb);
             let got = domtree.idom(ebb);
             if got != expected {
-                return fatal!(
-                    errors,
+                return errors.fatal((
                     ebb,
-                    "invalid domtree, expected idom({}) = {:?}, got {:?}",
-                    ebb,
-                    expected,
-                    got
-                );
+                    format!(
+                        "invalid domtree, expected idom({}) = {:?}, got {:?}",
+                        ebb, expected, got
+                    ),
+                ));
             }
         }
         // We also verify if the postorder defined by `DominatorTree` is sane
         if domtree.cfg_postorder().len() != self.expected_domtree.cfg_postorder().len() {
-            return fatal!(
-                errors,
+            return errors.fatal((
                 AnyEntity::Function,
-                "incorrect number of Ebbs in postorder traversal"
-            );
+                "incorrect number of Ebbs in postorder traversal",
+            ));
         }
         for (index, (&test_ebb, &true_ebb)) in domtree
             .cfg_postorder()
@@ -1111,14 +1111,13 @@ impl<'a> Verifier<'a> {
             .enumerate()
         {
             if test_ebb != true_ebb {
-                return fatal!(
-                    errors,
+                return errors.fatal((
                     test_ebb,
-                    "invalid domtree, postorder ebb number {} should be {}, got {}",
-                    index,
-                    true_ebb,
-                    test_ebb
-                );
+                    format!(
+                        "invalid domtree, postorder ebb number {} should be {}, got {}",
+                        index, true_ebb, test_ebb
+                    ),
+                ));
             }
         }
         // We verify rpo_cmp on pairs of adjacent ebbs in the postorder
@@ -1128,13 +1127,13 @@ impl<'a> Verifier<'a> {
                 .rpo_cmp(prev_ebb, next_ebb, &self.func.layout)
                 != Ordering::Greater
             {
-                return fatal!(
-                    errors,
+                return errors.fatal((
                     next_ebb,
-                    "invalid domtree, rpo_cmp does not says {} is greater than {}",
-                    prev_ebb,
-                    next_ebb
-                );
+                    format!(
+                        "invalid domtree, rpo_cmp does not says {} is greater than {}",
+                        prev_ebb, next_ebb
+                    ),
+                ));
             }
         }
         Ok(())
@@ -1146,26 +1145,26 @@ impl<'a> Verifier<'a> {
             let ebb_param_count = self.func.dfg.num_ebb_params(ebb);
 
             if ebb_param_count != expected_types.len() {
-                return fatal!(
-                    errors,
+                return errors.fatal((
                     ebb,
-                    "entry block parameters ({}) must match function signature ({})",
-                    ebb_param_count,
-                    expected_types.len()
-                );
+                    format!(
+                        "entry block parameters ({}) must match function signature ({})",
+                        ebb_param_count,
+                        expected_types.len()
+                    ),
+                ));
             }
 
             for (i, &arg) in self.func.dfg.ebb_params(ebb).iter().enumerate() {
                 let arg_type = self.func.dfg.value_type(arg);
                 if arg_type != expected_types[i].value_type {
-                    report!(
-                        errors,
+                    errors.report((
                         ebb,
-                        "entry block parameter {} expected to have type {}, got {}",
-                        i,
-                        expected_types[i],
-                        arg_type
-                    );
+                        format!(
+                            "entry block parameter {} expected to have type {}, got {}",
+                            i, expected_types[i], arg_type
+                        ),
+                    ));
                 }
             }
         }
@@ -1182,12 +1181,11 @@ impl<'a> Verifier<'a> {
             let ctrl_type = self.func.dfg.ctrl_typevar(inst);
 
             if !value_typeset.contains(ctrl_type) {
-                report!(
-                    errors,
+                errors.report((
                     inst,
-                    "has an invalid controlling type {}",
-                    ctrl_type
-                );
+                    self.context(inst),
+                    format!("has an invalid controlling type {}", ctrl_type),
+                ));
             }
 
             ctrl_type
@@ -1222,25 +1220,32 @@ impl<'a> Verifier<'a> {
             let expected_type = self.func.dfg.compute_result_type(inst, i, ctrl_type);
             if let Some(expected_type) = expected_type {
                 if result_type != expected_type {
-                    report!(
-                        errors,
+                    errors.report((
                         inst,
-                        "expected result {} ({}) to have type {}, found {}",
-                        i,
-                        result,
-                        expected_type,
-                        result_type
-                    );
+                        self.context(inst),
+                        format!(
+                            "expected result {} ({}) to have type {}, found {}",
+                            i, result, expected_type, result_type
+                        ),
+                    ));
                 }
             } else {
-                return self.nonfatal(errors, inst, "has more result values than expected");
+                return errors.nonfatal((
+                    inst,
+                    self.context(inst),
+                    "has more result values than expected",
+                ));
             }
             i += 1;
         }
 
         // There aren't any more result types left.
         if self.func.dfg.compute_result_type(inst, i, ctrl_type) != None {
-            return self.nonfatal(errors, inst, "has fewer result values than expected");
+            return errors.nonfatal((
+                inst,
+                self.context(inst),
+                "has fewer result values than expected",
+            ));
         }
         Ok(())
     }
@@ -1258,28 +1263,26 @@ impl<'a> Verifier<'a> {
             match constraints.value_argument_constraint(i, ctrl_type) {
                 ResolvedConstraint::Bound(expected_type) => {
                     if arg_type != expected_type {
-                        report!(
-                            errors,
+                        errors.report((
                             inst,
-                            "arg {} ({}) has type {}, expected {}",
-                            i,
-                            arg,
-                            arg_type,
-                            expected_type
-                        );
+                            self.context(inst),
+                            format!(
+                                "arg {} ({}) has type {}, expected {}",
+                                i, arg, arg_type, expected_type
+                            ),
+                        ));
                     }
                 }
                 ResolvedConstraint::Free(type_set) => {
                     if !type_set.contains(arg_type) {
-                        report!(
-                            errors,
+                        errors.report((
                             inst,
-                            "arg {} ({}) with type {} failed to satisfy type set {:?}",
-                            i,
-                            arg,
-                            arg_type,
-                            type_set
-                        );
+                            self.context(inst),
+                            format!(
+                                "arg {} ({}) with type {} failed to satisfy type set {:?}",
+                                i, arg, arg_type, type_set
+                            ),
+                        ));
                     }
                 }
             }
@@ -1306,27 +1309,27 @@ impl<'a> Verifier<'a> {
                 if let Some(ebb) = ebb {
                     let arg_count = self.func.dfg.num_ebb_params(ebb);
                     if arg_count != 0 {
-                        return self.nonfatal(
-                            errors,
+                        return errors.nonfatal((
                             inst,
+                            self.context(inst),
                             format!(
                                 "takes no arguments, but had target {} with {} arguments",
                                 ebb, arg_count,
                             ),
-                        );
+                        ));
                     }
                 }
                 for ebb in self.func.jump_tables[table].iter() {
                     let arg_count = self.func.dfg.num_ebb_params(*ebb);
                     if arg_count != 0 {
-                        return self.nonfatal(
-                            errors,
+                        return errors.nonfatal((
                             inst,
+                            self.context(inst),
                             format!(
                                 "takes no arguments, but had target {} with {} arguments",
                                 ebb, arg_count,
                             ),
-                        );
+                        ));
                     }
                 }
             }
@@ -1374,29 +1377,28 @@ impl<'a> Verifier<'a> {
             let arg = variable_args[i];
             let arg_type = self.func.dfg.value_type(arg);
             if expected_type != arg_type {
-                report!(
-                    errors,
+                errors.report((
                     inst,
-                    "arg {} ({}) has type {}, expected {}",
-                    i,
-                    variable_args[i],
-                    arg_type,
-                    expected_type
-                );
+                    self.context(inst),
+                    format!(
+                        "arg {} ({}) has type {}, expected {}",
+                        i, variable_args[i], arg_type, expected_type
+                    ),
+                ));
             }
             i += 1;
         }
         if i != variable_args.len() {
-            return self.nonfatal(
-                errors,
+            return errors.nonfatal((
                 inst,
+                self.context(inst),
                 format!(
                     "mismatched argument count for `{}`: got {}, expected {}",
                     self.func.dfg.display_inst(inst, None),
                     variable_args.len(),
                     i,
                 ),
-            );
+            ));
         }
         Ok(())
     }
@@ -1425,46 +1427,46 @@ impl<'a> Verifier<'a> {
                     self.verify_stack_slot(inst, ss, errors)?;
                     let slot = &self.func.stack_slots[ss];
                     if slot.kind != StackSlotKind::OutgoingArg {
-                        return self.fatal(
-                            errors,
+                        return errors.fatal((
                             inst,
+                            self.context(inst),
                             format!(
                                 "Outgoing stack argument {} in wrong stack slot: {} = {}",
                                 arg, ss, slot,
                             ),
-                        );
+                        ));
                     }
                     if slot.offset != Some(offset) {
-                        return self.fatal(
-                            errors,
+                        return errors.fatal((
                             inst,
+                            self.context(inst),
                             format!(
                                 "Outgoing stack argument {} should have offset {}: {} = {}",
                                 arg, offset, ss, slot,
                             ),
-                        );
+                        ));
                     }
                     if slot.size != abi.value_type.bytes() {
-                        return self.fatal(
-                            errors,
+                        return errors.fatal((
                             inst,
+                            self.context(inst),
                             format!(
                                 "Outgoing stack argument {} wrong size for {}: {} = {}",
                                 arg, abi.value_type, ss, slot,
                             ),
-                        );
+                        ));
                     }
                 } else {
                     let reginfo = self.isa.map(|i| i.register_info());
-                    return self.fatal(
-                        errors,
+                    return errors.fatal((
                         inst,
+                        self.context(inst),
                         format!(
                             "Outgoing stack argument {} in wrong location: {}",
                             arg,
-                            arg_loc.display(reginfo.as_ref()),
+                            arg_loc.display(reginfo.as_ref())
                         ),
-                    );
+                    ));
                 }
             }
         }
@@ -1476,24 +1478,23 @@ impl<'a> Verifier<'a> {
             let args = self.func.dfg.inst_variable_args(inst);
             let expected_types = &self.func.signature.returns;
             if args.len() != expected_types.len() {
-                return self.nonfatal(
-                    errors,
+                return errors.nonfatal((
                     inst,
+                    self.context(inst),
                     "arguments of return must match function signature",
-                );
+                ));
             }
             for (i, (&arg, &expected_type)) in args.iter().zip(expected_types).enumerate() {
                 let arg_type = self.func.dfg.value_type(arg);
                 if arg_type != expected_type.value_type {
-                    report!(
-                        errors,
+                    errors.report((
                         inst,
-                        "arg {} ({}) has type {}, must match function signature of {}",
-                        i,
-                        arg,
-                        arg_type,
-                        expected_type
-                    );
+                        self.context(inst),
+                        format!(
+                            "arg {} ({}) has type {}, must match function signature of {}",
+                            i, arg, arg_type, expected_type
+                        ),
+                    ));
                 }
             }
         }
@@ -1514,46 +1515,46 @@ impl<'a> Verifier<'a> {
                 match opcode {
                     Opcode::Bextend | Opcode::Uextend | Opcode::Sextend | Opcode::Fpromote => {
                         if arg_type.lane_count() != ctrl_type.lane_count() {
-                            return self.nonfatal(
-                                errors,
+                            return errors.nonfatal((
                                 inst,
+                                self.context(inst),
                                 format!(
                                     "input {} and output {} must have same number of lanes",
                                     arg_type, ctrl_type,
                                 ),
-                            );
+                            ));
                         }
                         if arg_type.lane_bits() >= ctrl_type.lane_bits() {
-                            return self.nonfatal(
-                                errors,
+                            return errors.nonfatal((
                                 inst,
+                                self.context(inst),
                                 format!(
                                     "input {} must be smaller than output {}",
                                     arg_type, ctrl_type,
                                 ),
-                            );
+                            ));
                         }
                     }
                     Opcode::Breduce | Opcode::Ireduce | Opcode::Fdemote => {
                         if arg_type.lane_count() != ctrl_type.lane_count() {
-                            return self.nonfatal(
-                                errors,
+                            return errors.nonfatal((
                                 inst,
+                                self.context(inst),
                                 format!(
                                     "input {} and output {} must have same number of lanes",
                                     arg_type, ctrl_type,
                                 ),
-                            );
+                            ));
                         }
                         if arg_type.lane_bits() <= ctrl_type.lane_bits() {
-                            return self.nonfatal(
-                                errors,
+                            return errors.nonfatal((
                                 inst,
+                                self.context(inst),
                                 format!(
                                     "input {} must be larger than output {}",
                                     arg_type, ctrl_type,
                                 ),
-                            );
+                            ));
                         }
                     }
                     _ => {}
@@ -1563,28 +1564,28 @@ impl<'a> Verifier<'a> {
                 let index_type = self.func.dfg.value_type(arg);
                 let heap_index_type = self.func.heaps[heap].index_type;
                 if index_type != heap_index_type {
-                    return self.nonfatal(
-                        errors,
+                    return errors.nonfatal((
                         inst,
+                        self.context(inst),
                         format!(
                             "index type {} differs from heap index type {}",
                             index_type, heap_index_type,
                         ),
-                    );
+                    ));
                 }
             }
             ir::InstructionData::TableAddr { table, arg, .. } => {
                 let index_type = self.func.dfg.value_type(arg);
                 let table_index_type = self.func.tables[table].index_type;
                 if index_type != table_index_type {
-                    return self.nonfatal(
-                        errors,
+                    return errors.nonfatal((
                         inst,
+                        self.context(inst),
                         format!(
                             "index type {} differs from table index type {}",
                             index_type, table_index_type,
                         ),
-                    );
+                    ));
                 }
             }
             ir::InstructionData::UnaryGlobalValue { global_value, .. } => {
@@ -1592,13 +1593,12 @@ impl<'a> Verifier<'a> {
                     let inst_type = self.func.dfg.value_type(self.func.dfg.first_result(inst));
                     let global_type = self.func.global_values[global_value].global_type(isa);
                     if inst_type != global_type {
-                        return self.nonfatal(
-                            errors,
-                            inst,
+                        return errors.nonfatal((
+                            inst, self.context(inst),
                             format!(
                                 "global_value instruction with type {} references global value with type {}",
                                 inst_type, global_type
-                            ),
+                            )),
                         );
                     }
                 }
@@ -1620,11 +1620,19 @@ impl<'a> Verifier<'a> {
         {
             let dst_vals = self.func.dfg.inst_results(inst);
             if dst_vals.len() != 1 {
-                return self.fatal(errors, inst, "copy_nop must produce exactly one result");
+                return errors.fatal((
+                    inst,
+                    self.context(inst),
+                    "copy_nop must produce exactly one result",
+                ));
             }
             let dst_val = dst_vals[0];
             if self.func.dfg.value_type(dst_val) != self.func.dfg.value_type(arg) {
-                return self.fatal(errors, inst, "copy_nop src and dst types must be the same");
+                return errors.fatal((
+                    inst,
+                    self.context(inst),
+                    "copy_nop src and dst types must be the same",
+                ));
             }
             let src_loc = self.func.locations[arg];
             let dst_loc = self.func.locations[dst_val];
@@ -1633,14 +1641,14 @@ impl<'a> Verifier<'a> {
                 _ => false,
             };
             if !locs_ok {
-                return self.fatal(
-                    errors,
+                return errors.fatal((
                     inst,
+                    self.context(inst),
                     format!(
                         "copy_nop must refer to identical stack slots, but found {:?} vs {:?}",
                         src_loc, dst_loc,
                     ),
-                );
+                ));
             }
         }
         Ok(())
@@ -1662,23 +1670,19 @@ impl<'a> Verifier<'a> {
 
             let missing_succs: Vec = expected_succs.difference(&got_succs).cloned().collect();
             if !missing_succs.is_empty() {
-                report!(
-                    errors,
+                errors.report((
                     ebb,
-                    "cfg lacked the following successor(s) {:?}",
-                    missing_succs
-                );
+                    format!("cfg lacked the following successor(s) {:?}", missing_succs),
+                ));
                 continue;
             }
 
             let excess_succs: Vec = got_succs.difference(&expected_succs).cloned().collect();
             if !excess_succs.is_empty() {
-                report!(
-                    errors,
+                errors.report((
                     ebb,
-                    "cfg had unexpected successor(s) {:?}",
-                    excess_succs
-                );
+                    format!("cfg had unexpected successor(s) {:?}", excess_succs),
+                ));
                 continue;
             }
 
@@ -1691,23 +1695,22 @@ impl<'a> Verifier<'a> {
 
             let missing_preds: Vec = expected_preds.difference(&got_preds).cloned().collect();
             if !missing_preds.is_empty() {
-                report!(
-                    errors,
+                errors.report((
                     ebb,
-                    "cfg lacked the following predecessor(s) {:?}",
-                    missing_preds
-                );
+                    format!(
+                        "cfg lacked the following predecessor(s) {:?}",
+                        missing_preds
+                    ),
+                ));
                 continue;
             }
 
             let excess_preds: Vec = got_preds.difference(&expected_preds).cloned().collect();
             if !excess_preds.is_empty() {
-                report!(
-                    errors,
+                errors.report((
                     ebb,
-                    "cfg had unexpected predecessor(s) {:?}",
-                    excess_preds
-                );
+                    format!("cfg had unexpected predecessor(s) {:?}", excess_preds),
+                ));
                 continue;
             }
 
@@ -1738,14 +1741,14 @@ impl<'a> Verifier<'a> {
         let encoding = self.func.encodings[inst];
         if encoding.is_legal() {
             if self.func.dfg[inst].opcode().is_ghost() {
-                return self.nonfatal(
-                    errors,
+                return errors.nonfatal((
                     inst,
+                    self.context(inst),
                     format!(
                         "Ghost instruction has an encoding: {}",
                         isa.encoding_info().display(encoding),
                     ),
-                );
+                ));
             }
 
             let mut encodings = isa
@@ -1757,14 +1760,14 @@ impl<'a> Verifier<'a> {
                 .peekable();
 
             if encodings.peek().is_none() {
-                return self.nonfatal(
-                    errors,
+                return errors.nonfatal((
                     inst,
+                    self.context(inst),
                     format!(
                         "Instruction failed to re-encode {}",
                         isa.encoding_info().display(encoding),
                     ),
-                );
+                ));
             }
 
             let has_valid_encoding = encodings.any(|possible_enc| encoding == possible_enc);
@@ -1787,16 +1790,16 @@ impl<'a> Verifier<'a> {
                         .unwrap();
                 }
 
-                return self.nonfatal(
-                    errors,
+                return errors.nonfatal((
                     inst,
+                    self.context(inst),
                     format!(
                         "encoding {} should be {}{}",
                         isa.encoding_info().display(encoding),
                         if multiple_encodings { "one of: " } else { "" },
                         possible_encodings,
                     ),
-                );
+                ));
             }
             return Ok(());
         }
@@ -1835,18 +1838,22 @@ impl<'a> Verifier<'a> {
             // Provide the ISA default encoding as a hint.
             match self.func.encode(inst, isa) {
                 Ok(enc) => {
-                    return self.nonfatal(
-                        errors,
+                    return errors.nonfatal((
                         inst,
+                        self.context(inst),
                         format!(
-                            "{} must have an encoding (e.g., {})",
+                            "{} must have an encoding (e.g., {})))",
                             text,
                             isa.encoding_info().display(enc),
                         ),
-                    );
+                    ));
                 }
                 Err(_) => {
-                    return self.nonfatal(errors, inst, format!("{} must have an encoding", text))
+                    return errors.nonfatal((
+                        inst,
+                        self.context(inst),
+                        format!("{} must have an encoding", text),
+                    ))
                 }
             }
         }
@@ -1865,11 +1872,11 @@ impl<'a> Verifier<'a> {
             ir::InstructionData::Store { flags, .. }
             | ir::InstructionData::StoreComplex { flags, .. } => {
                 if flags.readonly() {
-                    self.fatal(
-                        errors,
+                    errors.fatal((
                         inst,
+                        self.context(inst),
                         "A store instruction cannot have the `readonly` MemFlag",
-                    )
+                    ))
                 } else {
                     Ok(())
                 }
@@ -1890,11 +1897,11 @@ impl<'a> Verifier<'a> {
                 // the ExtractLane/InsertLane formats.
                 let ty = self.func.dfg.value_type(arg);
                 if u16::from(lane) >= ty.lane_count() {
-                    self.fatal(
-                        errors,
+                    errors.fatal((
                         inst,
+                        self.context(inst),
                         format!("The lane {} does not index into the type {}", lane, ty,),
-                    )
+                    ))
                 } else {
                     Ok(())
                 }
@@ -1911,11 +1918,11 @@ impl<'a> Verifier<'a> {
         if let Some(isa) = self.isa {
             if !isa.flags().enable_safepoints() && self.func.dfg[inst].opcode() == Opcode::Safepoint
             {
-                return self.fatal(
-                    errors,
+                return errors.fatal((
                     inst,
+                    self.context(inst),
                     "safepoint instruction cannot be used when it is not enabled.",
-                );
+                ));
             }
         }
         Ok(())
@@ -1929,12 +1936,10 @@ impl<'a> Verifier<'a> {
             .enumerate()
             .filter(|(_, ¶m)| param.value_type == types::INVALID)
             .for_each(|(i, _)| {
-                report!(
-                    errors,
+                errors.report((
                     AnyEntity::Function,
-                    "Parameter at position {} has an invalid type",
-                    i
-                );
+                    format!("Parameter at position {} has an invalid type", i),
+                ));
             });
 
         self.func
@@ -1944,12 +1949,10 @@ impl<'a> Verifier<'a> {
             .enumerate()
             .filter(|(_, &ret)| ret.value_type == types::INVALID)
             .for_each(|(i, _)| {
-                report!(
-                    errors,
+                errors.report((
                     AnyEntity::Function,
-                    "Return value at position {} has an invalid type",
-                    i
-                )
+                    format!("Return value at position {} has an invalid type", i),
+                ))
             });
 
         if errors.has_error() {

From de7c0dddc673950b90d92f762ffe263a28de0da5 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Tue, 3 Dec 2019 11:38:44 +0100
Subject: [PATCH 2969/3084] Fix #1247: Tweak enable_verifier doc comment so
 it's true

---
 cranelift/codegen/meta/src/shared/settings.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/meta/src/shared/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
index e375a8e66e..b039c7bc21 100644
--- a/cranelift/codegen/meta/src/shared/settings.rs
+++ b/cranelift/codegen/meta/src/shared/settings.rs
@@ -21,8 +21,8 @@ pub(crate) fn define() -> SettingGroup {
         r#"
         Run the Cranelift IR verifier at strategic times during compilation.
 
-        This makes compilation slower but catches many bugs. The verifier is
-        disabled by default, except when reading Cranelift IR from a text file.
+        This makes compilation slower but catches many bugs. The verifier is always enabled by
+        default, which is useful during development.
         "#,
         true,
     );

From 31cc184c347cb81b1ef12528acbe9ed92031ec1e Mon Sep 17 00:00:00 2001
From: Icy Defiance 
Date: Tue, 3 Dec 2019 18:50:52 -0500
Subject: [PATCH 2970/3084] Pin actions/checkout to v1 because v2 doesn't
 support submodules yet

Fixes error: "The input 'submodules' is not supported in actions/checkout@v2"
---
 cranelift/.github/workflows/main.yml | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/cranelift/.github/workflows/main.yml b/cranelift/.github/workflows/main.yml
index 8a82fa594c..9c7f5a39c9 100644
--- a/cranelift/.github/workflows/main.yml
+++ b/cranelift/.github/workflows/main.yml
@@ -10,7 +10,7 @@ jobs:
     name: Rustfmt
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@master
+    - uses: actions/checkout@v1
       with:
         submodules: true
     - name: Install Rust
@@ -21,7 +21,7 @@ jobs:
     name: Build API Docs
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@master
+    - uses: actions/checkout@v1
       with:
         submodules: true
     - name: Install Rust
@@ -81,7 +81,7 @@ jobs:
           rust: stable
           release: --release
     steps:
-    - uses: actions/checkout@master
+    - uses: actions/checkout@v1
       with:
         submodules: true
 
@@ -117,7 +117,7 @@ jobs:
     runs-on: ubuntu-latest
     if: false # Temporarily disable fuzz tests until https://github.com/bytecodealliance/cranelift/issues/1216 is resolved
     steps:
-    - uses: actions/checkout@master
+    - uses: actions/checkout@v1
     - name: Install Rust
       run: rustup update nightly && rustup default nightly
     - run: cargo install cargo-fuzz
@@ -128,7 +128,7 @@ jobs:
     runs-on: ubuntu-latest
     if: false && github.event_name == 'push' # Temporarily disable fuzz tests until https://github.com/bytecodealliance/cranelift/issues/1216 is resolved
     steps:
-    - uses: actions/checkout@master
+    - uses: actions/checkout@v1
     - name: Install Rust
       run: rustup update nightly && rustup default nightly
     - run: cargo install cargo-fuzz

From b342cbdd643477598b4f79ab48b2d9f21fe713c9 Mon Sep 17 00:00:00 2001
From: Mike Hommey 
Date: Wed, 4 Dec 2019 16:23:43 +0900
Subject: [PATCH 2971/3084] Sort the list of instruction formats by name

This makes opcodes.rs and inst_builders.rs deterministic.
---
 cranelift/codegen/meta/src/shared/mod.rs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
index 808b6bbc49..121e26b10a 100644
--- a/cranelift/codegen/meta/src/shared/mod.rs
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -92,6 +92,8 @@ impl Definitions {
             }
         }
 
-        Vec::from_iter(format_structures.into_iter().map(|(_, v)| v))
+        let mut result = Vec::from_iter(format_structures.into_iter().map(|(_, v)| v));
+        result.sort_by_key(|format| format.name);
+        result
     }
 }

From 4cc0241f37a6a3b75d88c395928aab83553a91f2 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 4 Dec 2019 18:16:22 +0100
Subject: [PATCH 2972/3084] More i8 legalizations (#1253)

* Legalize stack_{load,store}.i8, fixes #433
* Legalize select.i8, cc: #466
* Legalize brz.i8 and brnz.i8, cc: #1117
---
 cranelift/codegen/meta/src/shared/legalize.rs | 17 +++++++++
 .../filetests/isa/x86/brz-i8-run.clif         | 34 +++++++++++++++++
 .../filetests/filetests/isa/x86/brz-i8.clif   | 38 +++++++++++++++++++
 .../filetests/isa/x86/select-i8.clif          |  8 ++++
 .../filetests/isa/x86/stack-load-store8.clif  | 19 ++++++++++
 5 files changed, 116 insertions(+)
 create mode 100644 cranelift/filetests/filetests/isa/x86/brz-i8-run.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/brz-i8.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/select-i8.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/stack-load-store8.clif

diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 041937b8f8..482f5c69c6 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -139,6 +139,7 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     expand.custom_legalize(trapnz, "expand_cond_trap");
     expand.custom_legalize(br_table, "expand_br_table");
     expand.custom_legalize(select, "expand_select");
+    widen.custom_legalize(select, "expand_select"); // small ints
 
     // Custom expansions for floating point constants.
     // These expansions require bit-casting or creating constant pool entries.
@@ -149,6 +150,10 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     expand.custom_legalize(insts.by_name("stack_load"), "expand_stack_load");
     expand.custom_legalize(insts.by_name("stack_store"), "expand_stack_store");
 
+    // Custom expansions for small stack memory acccess.
+    widen.custom_legalize(insts.by_name("stack_load"), "expand_stack_load");
+    widen.custom_legalize(insts.by_name("stack_store"), "expand_stack_store");
+
     // List of variables to reuse in patterns.
     let x = var("x");
     let y = var("y");
@@ -612,6 +617,18 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
         }
     }
 
+    for &ty in &[I8, I16] {
+        widen.legalize(
+            def!(brz.ty(x, ebb, vararg)),
+            vec![def!(a = uextend.I32(x)), def!(brz(a, ebb, vararg))],
+        );
+
+        widen.legalize(
+            def!(brnz.ty(x, ebb, vararg)),
+            vec![def!(a = uextend.I32(x)), def!(brnz(a, ebb, vararg))],
+        );
+    }
+
     // Expand integer operations with carry for RISC architectures that don't have
     // the flags.
     let intcc_ult = Literal::enumerator_for(&imm.intcc, "ult");
diff --git a/cranelift/filetests/filetests/isa/x86/brz-i8-run.clif b/cranelift/filetests/filetests/isa/x86/brz-i8-run.clif
new file mode 100644
index 0000000000..c050a91425
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/brz-i8-run.clif
@@ -0,0 +1,34 @@
+test run
+target x86_64
+
+function u0:0() -> b1 {
+ebb0:
+    v0 = iconst.i8 0
+    brz v0, ebb1
+    jump ebb2
+
+ebb1:
+    v1 = bconst.b1 true
+    return v1
+
+ebb2:
+    v2 = bconst.b1 false
+    return v2
+}
+; run
+
+function u0:1() -> b1 {
+ebb0:
+    v0 = iconst.i8 0
+    brnz v0, ebb1
+    jump ebb2
+
+ebb1:
+    v1 = bconst.b1 false
+    return v1
+
+ebb2:
+    v2 = bconst.b1 true
+    return v2
+}
+; run
diff --git a/cranelift/filetests/filetests/isa/x86/brz-i8.clif b/cranelift/filetests/filetests/isa/x86/brz-i8.clif
new file mode 100644
index 0000000000..a5cc03c985
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/brz-i8.clif
@@ -0,0 +1,38 @@
+test compile
+target x86_64
+
+function u0:0() -> b1 {
+ebb0:
+    v0 = iconst.i8 0
+    ; check: v0 = iconst.i8 0
+    brz v0, ebb1
+    ; nextln: v3 = uextend.i32 v0
+    ; nextln: brz v3, ebb1
+    jump ebb2
+
+ebb1:
+    v1 = bconst.b1 true
+    return v1
+
+ebb2:
+    v2 = bconst.b1 false
+    return v2
+}
+
+function u0:1() -> b1 {
+ebb0:
+    v0 = iconst.i8 0
+    ; check: v0 = iconst.i8 0
+    brnz v0, ebb1
+    ; nextln: v3 = uextend.i32 v0
+    ; nextln: brnz v3, ebb1
+    jump ebb2
+
+ebb1:
+    v1 = bconst.b1 false
+    return v1
+
+ebb2:
+    v2 = bconst.b1 true
+    return v2
+}
diff --git a/cranelift/filetests/filetests/isa/x86/select-i8.clif b/cranelift/filetests/filetests/isa/x86/select-i8.clif
new file mode 100644
index 0000000000..aac59c1e9c
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/select-i8.clif
@@ -0,0 +1,8 @@
+test compile
+target x86_64
+
+function u0:0(b1, i8, i8) -> i8 {
+ebb0(v0: b1, v1: i8, v2: i8):
+    v3 = select v0, v1, v2
+    return v3
+}
diff --git a/cranelift/filetests/filetests/isa/x86/stack-load-store8.clif b/cranelift/filetests/filetests/isa/x86/stack-load-store8.clif
new file mode 100644
index 0000000000..2c368f6dfc
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/stack-load-store8.clif
@@ -0,0 +1,19 @@
+test compile
+target x86_64
+
+function u0:0(i8) -> i8 {
+    ss0 = explicit_slot 1
+
+ebb0(v0: i8):
+    stack_store v0, ss0
+    ; check: v2 = stack_addr.i64 ss0
+    ; nextln: v3 = uextend.i32 v0
+    ; nextln: istore8 notrap aligned v3, v2
+
+    v1 = stack_load.i8 ss0
+    ; check: v4 = stack_addr.i64 ss0
+    ; nextln: v5 = uload8.i32 notrap aligned v4
+    ; nextln: v1 = ireduce.i8 v5
+
+    return v1
+}

From 81782f8efa02023c466d4b673b3cc14cebef81c8 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 4 Dec 2019 10:36:20 +0100
Subject: [PATCH 2973/3084] Ensure builds are deterministic in CI.

This introduces a script that has a high probability of failing if the
Rust source code generated by the meta crate is not consistent accross
build script runs.

It also adds a new CI job to run it on each push, on a single platform.
---
 cranelift/.github/workflows/main.yml       | 12 +++++++
 cranelift/ci/ensure_deterministic_build.sh | 41 ++++++++++++++++++++++
 2 files changed, 53 insertions(+)
 create mode 100755 cranelift/ci/ensure_deterministic_build.sh

diff --git a/cranelift/.github/workflows/main.yml b/cranelift/.github/workflows/main.yml
index 9c7f5a39c9..e8bb3fb41f 100644
--- a/cranelift/.github/workflows/main.yml
+++ b/cranelift/.github/workflows/main.yml
@@ -112,6 +112,18 @@ jobs:
       if: false && matrix.rust == 'nightly' # Temporarily disable fuzz tests until https://github.com/bytecodealliance/cranelift/issues/1216 is resolved
       continue-on-error: true
 
+  meta_determinist_check:
+    name: Meta deterministic check
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v1
+      with:
+        submodules: true
+    - name: Install Rust
+      run: rustup update stable && rustup default stable
+    - run: cargo build
+    - run: ci/ensure_deterministic_build.sh
+
   fuzz:
     name: Fuzz Regression
     runs-on: ubuntu-latest
diff --git a/cranelift/ci/ensure_deterministic_build.sh b/cranelift/ci/ensure_deterministic_build.sh
new file mode 100755
index 0000000000..ebed6cd1c7
--- /dev/null
+++ b/cranelift/ci/ensure_deterministic_build.sh
@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# This script makes sure that the meta crate deterministically generate files
+# with a high probability.
+# The current directory must be set to the repository's root.
+
+set -e
+
+BUILD_SCRIPT=$(find -wholename "./target/debug/build/cranelift-codegen-*/build-script-build")
+
+# First, run the script to generate a reference comparison.
+rm -rf /tmp/reference
+mkdir /tmp/reference
+OUT_DIR=/tmp/reference TARGET=x86_64 $BUILD_SCRIPT
+
+# To make sure the build script doesn't depend on the current directory, we'll
+# change the current working directory on every iteration. Make this easy to
+# reproduce this locally by first copying the target/ directory into an initial
+# temporary directory (and not move and lose the local clone's content).
+rm -rf /tmp/src0
+mkdir /tmp/src0
+
+echo Copying target directory...
+cp -r ./target /tmp/src0/target
+cd /tmp/src0
+echo "Done, starting loop."
+
+# Then, repeatedly make sure that the output is the same.
+for i in {1..20}
+do
+    # Move to a different directory, as explained above.
+    rm -rf /tmp/src$i
+    mkdir /tmp/src$i
+    mv ./* /tmp/src$i
+    cd /tmp/src$i
+
+    rm -rf /tmp/try
+    mkdir /tmp/try
+    OUT_DIR=/tmp/try TARGET=x86_64 $BUILD_SCRIPT
+    diff -qr /tmp/reference /tmp/try
+done

From aabf6559a0434ebaf5ebf4c34f9670cffc03f349 Mon Sep 17 00:00:00 2001
From: Ryan Hunt 
Date: Fri, 6 Dec 2019 09:13:53 -0600
Subject: [PATCH 2974/3084] Add hooks for implementing bulk-memory-operations
 (#1258)

---
 cranelift/wasm/src/code_translator.rs | 110 +++++++++++++++++++++++---
 cranelift/wasm/src/environ/dummy.rs   |  81 +++++++++++++++++++
 cranelift/wasm/src/environ/spec.rs    |  83 +++++++++++++++++++
 3 files changed, 262 insertions(+), 12 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 2f645f5009..9bb4b03792 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -987,18 +987,104 @@ pub fn translate_operator(
         | Operator::Fence { .. } => {
             return Err(wasm_unsupported!("proposed thread operator {:?}", op));
         }
-        Operator::MemoryInit { .. }
-        | Operator::DataDrop { .. }
-        | Operator::MemoryCopy
-        | Operator::MemoryFill
-        | Operator::TableInit { .. }
-        | Operator::ElemDrop { .. }
-        | Operator::TableCopy
-        | Operator::TableGet { .. }
-        | Operator::TableSet { .. }
-        | Operator::TableGrow { .. }
-        | Operator::TableSize { .. } => {
-            return Err(wasm_unsupported!("proposed bulk memory operator {:?}", op));
+        Operator::MemoryCopy => {
+            // The WebAssembly MVP only supports one linear memory and
+            // wasmparser will ensure that the memory indices specified are
+            // zero.
+            let heap_index = MemoryIndex::from_u32(0);
+            let heap = state.get_heap(builder.func, 0, environ)?;
+            let len = state.pop1();
+            let src = state.pop1();
+            let dest = state.pop1();
+            environ.translate_memory_copy(builder.cursor(), heap_index, heap, dest, src, len)?;
+        }
+        Operator::MemoryFill => {
+            // The WebAssembly MVP only supports one linear memory and
+            // wasmparser will ensure that the memory index specified is
+            // zero.
+            let heap_index = MemoryIndex::from_u32(0);
+            let heap = state.get_heap(builder.func, 0, environ)?;
+            let len = state.pop1();
+            let val = state.pop1();
+            let dest = state.pop1();
+            environ.translate_memory_fill(builder.cursor(), heap_index, heap, dest, val, len)?;
+        }
+        Operator::MemoryInit { segment } => {
+            // The WebAssembly MVP only supports one linear memory and
+            // wasmparser will ensure that the memory index specified is
+            // zero.
+            let heap_index = MemoryIndex::from_u32(0);
+            let heap = state.get_heap(builder.func, 0, environ)?;
+            let len = state.pop1();
+            let src = state.pop1();
+            let dest = state.pop1();
+            environ.translate_memory_init(
+                builder.cursor(),
+                heap_index,
+                heap,
+                *segment,
+                dest,
+                src,
+                len,
+            )?;
+        }
+        Operator::DataDrop { segment } => {
+            environ.translate_data_drop(builder.cursor(), *segment)?;
+        }
+        Operator::TableSize { table: index } => {
+            let table = state.get_table(builder.func, *index, environ)?;
+            state.push1(environ.translate_table_size(
+                builder.cursor(),
+                TableIndex::from_u32(*index),
+                table,
+            )?);
+        }
+        Operator::TableCopy => {
+            // The WebAssembly MVP only supports one table and wasmparser will
+            // ensure that the table index specified is zero.
+            let dst_table_index = 0;
+            let dst_table = state.get_table(builder.func, dst_table_index, environ)?;
+            let src_table_index = 0;
+            let src_table = state.get_table(builder.func, src_table_index, environ)?;
+            let len = state.pop1();
+            let src = state.pop1();
+            let dest = state.pop1();
+            environ.translate_table_copy(
+                builder.cursor(),
+                TableIndex::from_u32(dst_table_index),
+                dst_table,
+                TableIndex::from_u32(src_table_index),
+                src_table,
+                dest,
+                src,
+                len,
+            )?;
+        }
+        Operator::TableInit { segment } => {
+            // The WebAssembly MVP only supports one table and we assume it here.
+            let table_index = 0;
+            let table = state.get_table(builder.func, table_index, environ)?;
+            let len = state.pop1();
+            let src = state.pop1();
+            let dest = state.pop1();
+            environ.translate_table_init(
+                builder.cursor(),
+                *segment,
+                TableIndex::from_u32(table_index),
+                table,
+                dest,
+                src,
+                len,
+            )?;
+        }
+        Operator::ElemDrop { segment } => {
+            environ.translate_elem_drop(builder.cursor(), *segment)?;
+        }
+        Operator::TableGet { .. } | Operator::TableSet { .. } | Operator::TableGrow { .. } => {
+            return Err(wasm_unsupported!(
+                "proposed reference types operator {:?}",
+                op
+            ));
         }
         Operator::V128Const { value } => {
             let data = value.bytes().to_vec().into();
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index 57668d8f48..64ca5e7caf 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -371,6 +371,87 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
     ) -> WasmResult {
         Ok(pos.ins().iconst(I32, -1))
     }
+
+    fn translate_memory_copy(
+        &mut self,
+        _pos: FuncCursor,
+        _index: MemoryIndex,
+        _heap: ir::Heap,
+        _dst: ir::Value,
+        _src: ir::Value,
+        _len: ir::Value,
+    ) -> WasmResult<()> {
+        Ok(())
+    }
+
+    fn translate_memory_fill(
+        &mut self,
+        _pos: FuncCursor,
+        _index: MemoryIndex,
+        _heap: ir::Heap,
+        _dst: ir::Value,
+        _val: ir::Value,
+        _len: ir::Value,
+    ) -> WasmResult<()> {
+        Ok(())
+    }
+
+    fn translate_memory_init(
+        &mut self,
+        _pos: FuncCursor,
+        _index: MemoryIndex,
+        _heap: ir::Heap,
+        _seg_index: u32,
+        _dst: ir::Value,
+        _src: ir::Value,
+        _len: ir::Value,
+    ) -> WasmResult<()> {
+        Ok(())
+    }
+
+    fn translate_data_drop(&mut self, _pos: FuncCursor, _seg_index: u32) -> WasmResult<()> {
+        Ok(())
+    }
+
+    fn translate_table_size(
+        &mut self,
+        mut pos: FuncCursor,
+        _index: TableIndex,
+        _table: ir::Table,
+    ) -> WasmResult {
+        Ok(pos.ins().iconst(I32, -1))
+    }
+
+    fn translate_table_copy(
+        &mut self,
+        _pos: FuncCursor,
+        _dst_index: TableIndex,
+        _dst_table: ir::Table,
+        _src_index: TableIndex,
+        _src_table: ir::Table,
+        _dst: ir::Value,
+        _src: ir::Value,
+        _len: ir::Value,
+    ) -> WasmResult<()> {
+        Ok(())
+    }
+
+    fn translate_table_init(
+        &mut self,
+        _pos: FuncCursor,
+        _seg_index: u32,
+        _table_index: TableIndex,
+        _table: ir::Table,
+        _dst: ir::Value,
+        _src: ir::Value,
+        _len: ir::Value,
+    ) -> WasmResult<()> {
+        Ok(())
+    }
+
+    fn translate_elem_drop(&mut self, _pos: FuncCursor, _seg_index: u32) -> WasmResult<()> {
+        Ok(())
+    }
 }
 
 impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 9acaa2aa47..77fc6e2d1c 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -266,6 +266,89 @@ pub trait FuncEnvironment {
         heap: ir::Heap,
     ) -> WasmResult;
 
+    /// Translate a `memory.copy` WebAssembly instruction.
+    ///
+    /// The `index` provided identifies the linear memory to query, and `heap` is the heap reference
+    /// returned by `make_heap` for the same index.
+    fn translate_memory_copy(
+        &mut self,
+        pos: FuncCursor,
+        index: MemoryIndex,
+        heap: ir::Heap,
+        dst: ir::Value,
+        src: ir::Value,
+        len: ir::Value,
+    ) -> WasmResult<()>;
+
+    /// Translate a `memory.fill` WebAssembly instruction.
+    ///
+    /// The `index` provided identifies the linear memory to query, and `heap` is the heap reference
+    /// returned by `make_heap` for the same index.
+    fn translate_memory_fill(
+        &mut self,
+        pos: FuncCursor,
+        index: MemoryIndex,
+        heap: ir::Heap,
+        dst: ir::Value,
+        val: ir::Value,
+        len: ir::Value,
+    ) -> WasmResult<()>;
+
+    /// Translate a `memory.init` WebAssembly instruction.
+    ///
+    /// The `index` provided identifies the linear memory to query, and `heap` is the heap reference
+    /// returned by `make_heap` for the same index. `seg_index` is the index of the segment to copy
+    /// from.
+    fn translate_memory_init(
+        &mut self,
+        pos: FuncCursor,
+        index: MemoryIndex,
+        heap: ir::Heap,
+        seg_index: u32,
+        dst: ir::Value,
+        src: ir::Value,
+        len: ir::Value,
+    ) -> WasmResult<()>;
+
+    /// Translate a `data.drop` WebAssembly instruction.
+    fn translate_data_drop(&mut self, pos: FuncCursor, seg_index: u32) -> WasmResult<()>;
+
+    /// Translate a `table.size` WebAssembly instruction.
+    fn translate_table_size(
+        &mut self,
+        pos: FuncCursor,
+        index: TableIndex,
+        table: ir::Table,
+    ) -> WasmResult;
+
+    /// Translate a `table.copy` WebAssembly instruction.
+    fn translate_table_copy(
+        &mut self,
+        pos: FuncCursor,
+        dst_table_index: TableIndex,
+        dst_table: ir::Table,
+        src_table_index: TableIndex,
+        src_table: ir::Table,
+        dst: ir::Value,
+        src: ir::Value,
+        len: ir::Value,
+    ) -> WasmResult<()>;
+
+    /// Translate a `table.init` WebAssembly instruction.
+    fn translate_table_init(
+        &mut self,
+        pos: FuncCursor,
+        seg_index: u32,
+        table_index: TableIndex,
+        table: ir::Table,
+        dst: ir::Value,
+        src: ir::Value,
+        len: ir::Value,
+    ) -> WasmResult<()>;
+
+    /// Translate a `elem.drop` WebAssembly instruction.
+    fn translate_elem_drop(&mut self, pos: FuncCursor, seg_index: u32) -> WasmResult<()>;
+
     /// Emit code at the beginning of every wasm loop.
     ///
     /// This can be used to insert explicit interrupt or safepoint checking at

From 2c51341888e917c64fdbc0e98fa7997fede5fcfe Mon Sep 17 00:00:00 2001
From: Yury Delendik 
Date: Fri, 6 Dec 2019 17:46:03 -0600
Subject: [PATCH 2975/3084] Add wasm reference/pointers translation. (#1073)

---
 cranelift/src/clif-util.rs                |  8 ++++++
 cranelift/src/wasm.rs                     |  6 ++++
 cranelift/wasm/src/code_translator.rs     | 21 +++++++-------
 cranelift/wasm/src/environ/dummy.rs       | 12 ++++++--
 cranelift/wasm/src/environ/mod.rs         |  3 +-
 cranelift/wasm/src/environ/spec.rs        | 34 +++++++++++------------
 cranelift/wasm/src/func_translator.rs     |  1 +
 cranelift/wasm/src/lib.rs                 |  4 +--
 cranelift/wasm/src/sections_translator.rs | 16 ++++++-----
 cranelift/wasm/src/translation_utils.rs   | 24 +++++++++++++---
 10 files changed, 85 insertions(+), 44 deletions(-)

diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index 0c6110f024..b40f4d26e9 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -119,6 +119,12 @@ fn add_enable_multi_value<'a>() -> clap::Arg<'a, 'a> {
         .help("Enable WASM's multi-value support")
 }
 
+fn add_enable_reference_types_flag<'a>() -> clap::Arg<'a, 'a> {
+    Arg::with_name("enable-reference-types")
+        .long("enable-reference-types")
+        .help("Enable WASM's reference types operations")
+}
+
 fn add_just_decode_flag<'a>() -> clap::Arg<'a, 'a> {
     Arg::with_name("just-decode")
         .short("t")
@@ -163,6 +169,7 @@ fn add_wasm_or_compile<'a>(cmd: &str) -> clap::App<'a, 'a> {
         .arg(add_debug_flag())
         .arg(add_enable_simd_flag())
         .arg(add_enable_multi_value())
+        .arg(add_enable_reference_types_flag())
         .arg(add_just_decode_flag())
         .arg(add_check_translation_flag())
 }
@@ -316,6 +323,7 @@ fn main() {
                     rest_cmd.is_present("value-ranges"),
                     rest_cmd.is_present("enable-simd"),
                     rest_cmd.is_present("enable-multi-value"),
+                    rest_cmd.is_present("enable-reference-types"),
                 )
             };
 
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index a4ee4c8666..041dd792b3 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -51,6 +51,7 @@ pub fn run(
     flag_calc_value_ranges: bool,
     flag_enable_simd: bool,
     flag_enable_multi_value: bool,
+    flag_enable_reference_types: bool,
 ) -> Result<(), String> {
     let parsed = parse_sets_and_triple(flag_set, flag_triple)?;
 
@@ -68,6 +69,7 @@ pub fn run(
             flag_calc_value_ranges,
             flag_enable_simd,
             flag_enable_multi_value,
+            flag_enable_reference_types,
             &path.to_path_buf(),
             &name,
             parsed.as_fisa(),
@@ -87,6 +89,7 @@ fn handle_module(
     flag_calc_value_ranges: bool,
     flag_enable_simd: bool,
     flag_enable_multi_value: bool,
+    flag_enable_reference_types: bool,
     path: &PathBuf,
     name: &str,
     fisa: FlagsOrIsa,
@@ -110,6 +113,9 @@ fn handle_module(
         if flag_enable_multi_value {
             features.enable_multi_value();
         }
+        if flag_enable_reference_types {
+            features.enable_reference_types();
+        }
 
         module_binary = match wat2wasm_with_features(&module_binary, features) {
             Ok(data) => data,
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 9bb4b03792..c0d1218ccf 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -55,7 +55,7 @@ pub fn translate_operator(
     environ: &mut FE,
 ) -> WasmResult<()> {
     if !state.reachable {
-        translate_unreachable_operator(module_translation_state, &op, builder, state)?;
+        translate_unreachable_operator(module_translation_state, &op, builder, state, environ)?;
         return Ok(());
     }
 
@@ -139,13 +139,13 @@ pub fn translate_operator(
          ***********************************************************************************/
         Operator::Block { ty } => {
             let (params, results) = blocktype_params_results(module_translation_state, *ty)?;
-            let next = ebb_with_params(builder, results)?;
+            let next = ebb_with_params(builder, results, environ)?;
             state.push_block(next, params.len(), results.len());
         }
         Operator::Loop { ty } => {
             let (params, results) = blocktype_params_results(module_translation_state, *ty)?;
-            let loop_body = ebb_with_params(builder, params)?;
-            let next = ebb_with_params(builder, results)?;
+            let loop_body = ebb_with_params(builder, params, environ)?;
+            let next = ebb_with_params(builder, results, environ)?;
             builder.ins().jump(loop_body, state.peekn(params.len()));
             state.push_loop(loop_body, next, params.len(), results.len());
 
@@ -168,7 +168,7 @@ pub fn translate_operator(
                 // destination ebb following the whole `if...end`. If we do end
                 // up discovering an `else`, then we will allocate an ebb for it
                 // and go back and patch the jump.
-                let destination = ebb_with_params(builder, results)?;
+                let destination = ebb_with_params(builder, results, environ)?;
                 let branch_inst = builder
                     .ins()
                     .brz(val, destination, state.peekn(params.len()));
@@ -176,8 +176,8 @@ pub fn translate_operator(
             } else {
                 // The `if` type signature is not valid without an `else` block,
                 // so we eagerly allocate the `else` block here.
-                let destination = ebb_with_params(builder, results)?;
-                let else_block = ebb_with_params(builder, params)?;
+                let destination = ebb_with_params(builder, results, environ)?;
+                let else_block = ebb_with_params(builder, params, environ)?;
                 builder
                     .ins()
                     .brz(val, else_block, state.peekn(params.len()));
@@ -229,7 +229,7 @@ pub fn translate_operator(
                                 let (params, _results) =
                                     blocktype_params_results(module_translation_state, blocktype)?;
                                 debug_assert_eq!(params.len(), num_return_values);
-                                let else_ebb = ebb_with_params(builder, params)?;
+                                let else_ebb = ebb_with_params(builder, params, environ)?;
                                 builder.ins().jump(destination, state.peekn(params.len()));
                                 state.popn(params.len());
 
@@ -1352,11 +1352,12 @@ pub fn translate_operator(
 /// Deals with a Wasm instruction located in an unreachable portion of the code. Most of them
 /// are dropped but special ones like `End` or `Else` signal the potential end of the unreachable
 /// portion so the translation state must be updated accordingly.
-fn translate_unreachable_operator(
+fn translate_unreachable_operator(
     module_translation_state: &ModuleTranslationState,
     op: &Operator,
     builder: &mut FunctionBuilder,
     state: &mut FuncTranslationState,
+    environ: &mut FE,
 ) -> WasmResult<()> {
     debug_assert!(!state.reachable);
     match *op {
@@ -1397,7 +1398,7 @@ fn translate_unreachable_operator(
                             ElseData::NoElse { branch_inst } => {
                                 let (params, _results) =
                                     blocktype_params_results(module_translation_state, blocktype)?;
-                                let else_ebb = ebb_with_params(builder, params)?;
+                                let else_ebb = ebb_with_params(builder, params, environ)?;
 
                                 // We change the target of the branch instruction.
                                 builder.change_jump_destination(branch_inst, else_ebb);
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index 64ca5e7caf..6b6f210166 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -5,7 +5,9 @@
 //! [wasmtime-environ]: https://crates.io/crates/wasmtime-environ
 //! [Wasmtime]: https://github.com/bytecodealliance/wasmtime
 
-use crate::environ::{FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmResult};
+use crate::environ::{
+    FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, TargetEnvironment, WasmResult,
+};
 use crate::func_translator::FuncTranslator;
 use crate::state::ModuleTranslationState;
 use crate::translation_utils::{
@@ -192,11 +194,13 @@ impl<'dummy_environment> DummyFuncEnvironment<'dummy_environment> {
     }
 }
 
-impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environment> {
+impl<'dummy_environment> TargetEnvironment for DummyFuncEnvironment<'dummy_environment> {
     fn target_config(&self) -> TargetFrontendConfig {
         self.mod_info.config
     }
+}
 
+impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environment> {
     fn return_mode(&self) -> ReturnMode {
         self.return_mode
     }
@@ -454,11 +458,13 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
     }
 }
 
-impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
+impl TargetEnvironment for DummyEnvironment {
     fn target_config(&self) -> TargetFrontendConfig {
         self.info.config
     }
+}
 
+impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
     fn declare_signature(&mut self, sig: ir::Signature) -> WasmResult<()> {
         self.info.signatures.push(sig);
         Ok(())
diff --git a/cranelift/wasm/src/environ/mod.rs b/cranelift/wasm/src/environ/mod.rs
index 4b7405ea7b..1cdb0b292a 100644
--- a/cranelift/wasm/src/environ/mod.rs
+++ b/cranelift/wasm/src/environ/mod.rs
@@ -6,5 +6,6 @@ mod spec;
 
 pub use crate::environ::dummy::DummyEnvironment;
 pub use crate::environ::spec::{
-    FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmError, WasmResult,
+    FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, TargetEnvironment, WasmError,
+    WasmResult,
 };
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 77fc6e2d1c..5a38d7da62 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -103,12 +103,8 @@ pub enum ReturnMode {
     FallthroughReturn,
 }
 
-/// Environment affecting the translation of a single WebAssembly function.
-///
-/// A `FuncEnvironment` trait object is required to translate a WebAssembly function to Cranelift
-/// IR. The function environment provides information about the WebAssembly module as well as the
-/// runtime environment.
-pub trait FuncEnvironment {
+/// Environment affecting the translation of a WebAssembly.
+pub trait TargetEnvironment {
     /// Get the information needed to produce Cranelift IR for the given target.
     fn target_config(&self) -> TargetFrontendConfig;
 
@@ -124,13 +120,6 @@ pub trait FuncEnvironment {
         self.target_config().pointer_bytes()
     }
 
-    /// Should the code be structured to use a single `fallthrough_return` instruction at the end
-    /// of the function body, rather than `return` instructions as needed? This is used by VMs
-    /// to append custom epilogues.
-    fn return_mode(&self) -> ReturnMode {
-        ReturnMode::NormalReturns
-    }
-
     /// Get the Cranelift reference type to use for native references.
     ///
     /// This returns `R64` for 64-bit architectures and `R32` for 32-bit architectures.
@@ -141,6 +130,20 @@ pub trait FuncEnvironment {
             _ => panic!("unsupported pointer type"),
         }
     }
+}
+
+/// Environment affecting the translation of a single WebAssembly function.
+///
+/// A `FuncEnvironment` trait object is required to translate a WebAssembly function to Cranelift
+/// IR. The function environment provides information about the WebAssembly module as well as the
+/// runtime environment.
+pub trait FuncEnvironment: TargetEnvironment {
+    /// Should the code be structured to use a single `fallthrough_return` instruction at the end
+    /// of the function body, rather than `return` instructions as needed? This is used by VMs
+    /// to append custom epilogues.
+    fn return_mode(&self) -> ReturnMode {
+        ReturnMode::NormalReturns
+    }
 
     /// Set up the necessary preamble definitions in `func` to access the global variable
     /// identified by `index`.
@@ -384,10 +387,7 @@ pub trait FuncEnvironment {
 /// An object satisfying the `ModuleEnvironment` trait can be passed as argument to the
 /// [`translate_module`](fn.translate_module.html) function. These methods should not be called
 /// by the user, they are only for `cranelift-wasm` internal use.
-pub trait ModuleEnvironment<'data> {
-    /// Get the information needed to produce Cranelift IR for the current target.
-    fn target_config(&self) -> TargetFrontendConfig;
-
+pub trait ModuleEnvironment<'data>: TargetEnvironment {
     /// Provides the number of signatures up front. By default this does nothing, but
     /// implementations can use this to preallocate memory if desired.
     fn reserve_signatures(&mut self, _num: u32) -> WasmResult<()> {
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index 08d248157f..8f5d49bcdf 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -193,6 +193,7 @@ fn declare_locals(
             builder.ins().vconst(ir::types::I8X16, constant_handle)
         }
         AnyRef => builder.ins().null(environ.reference_type()),
+        AnyFunc => builder.ins().null(environ.reference_type()),
         ty => return Err(wasm_unsupported!("unsupported local type {:?}", ty)),
     };
 
diff --git a/cranelift/wasm/src/lib.rs b/cranelift/wasm/src/lib.rs
index a50bd79320..28e4361126 100644
--- a/cranelift/wasm/src/lib.rs
+++ b/cranelift/wasm/src/lib.rs
@@ -58,8 +58,8 @@ mod state;
 mod translation_utils;
 
 pub use crate::environ::{
-    DummyEnvironment, FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode, WasmError,
-    WasmResult,
+    DummyEnvironment, FuncEnvironment, GlobalVariable, ModuleEnvironment, ReturnMode,
+    TargetEnvironment, WasmError, WasmResult,
 };
 pub use crate::func_translator::FuncTranslator;
 pub use crate::module_translator::translate_module;
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index 0a7c678f4c..ae980cfcfc 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -44,14 +44,15 @@ pub fn parse_type_section(
                 params,
                 returns,
             } => {
-                let mut sig = Signature::new(environ.target_config().default_call_conv);
+                let mut sig =
+                    Signature::new(ModuleEnvironment::target_config(environ).default_call_conv);
                 sig.params.extend(params.iter().map(|ty| {
-                    let cret_arg: ir::Type = type_to_type(*ty)
+                    let cret_arg: ir::Type = type_to_type(*ty, environ)
                         .expect("only numeric types are supported in function signatures");
                     AbiParam::new(cret_arg)
                 }));
                 sig.returns.extend(returns.iter().map(|ty| {
-                    let cret_arg: ir::Type = type_to_type(*ty)
+                    let cret_arg: ir::Type = type_to_type(*ty, environ)
                         .expect("only numeric types are supported in function signatures");
                     AbiParam::new(cret_arg)
                 }));
@@ -106,7 +107,7 @@ pub fn parse_import_section<'data>(
             ImportSectionEntryType::Global(ref ty) => {
                 environ.declare_global_import(
                     Global {
-                        ty: type_to_type(ty.content_type).unwrap(),
+                        ty: type_to_type(ty.content_type, environ).unwrap(),
                         mutability: ty.mutable,
                         initializer: GlobalInit::Import,
                     },
@@ -117,7 +118,7 @@ pub fn parse_import_section<'data>(
             ImportSectionEntryType::Table(ref tab) => {
                 environ.declare_table_import(
                     Table {
-                        ty: match tabletype_to_type(tab.element_type)? {
+                        ty: match tabletype_to_type(tab.element_type, environ)? {
                             Some(t) => TableElementType::Val(t),
                             None => TableElementType::Func,
                         },
@@ -160,7 +161,7 @@ pub fn parse_table_section(
     for entry in tables {
         let table = entry?;
         environ.declare_table(Table {
-            ty: match tabletype_to_type(table.element_type)? {
+            ty: match tabletype_to_type(table.element_type, environ)? {
                 Some(t) => TableElementType::Val(t),
                 None => TableElementType::Func,
             },
@@ -215,6 +216,7 @@ pub fn parse_global_section(
             Operator::V128Const { value } => {
                 GlobalInit::V128Const(V128Imm::from(value.bytes().to_vec().as_slice()))
             }
+            Operator::RefNull => GlobalInit::RefNullConst,
             Operator::GetGlobal { global_index } => {
                 GlobalInit::GetGlobal(GlobalIndex::from_u32(global_index))
             }
@@ -226,7 +228,7 @@ pub fn parse_global_section(
             }
         };
         let global = Global {
-            ty: type_to_type(content_type).unwrap(),
+            ty: type_to_type(content_type, environ).unwrap(),
             mutability: mutable,
             initializer,
         };
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index 1e5e963b4a..9d38d9dbb6 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -1,5 +1,5 @@
 //! Helper functions and structures for the translation.
-use crate::environ::WasmResult;
+use crate::environ::{TargetEnvironment, WasmResult};
 use crate::state::ModuleTranslationState;
 use crate::wasm_unsupported;
 use core::u32;
@@ -83,6 +83,8 @@ pub enum GlobalInit {
     V128Const(V128Imm),
     /// A `get_global` of another global.
     GetGlobal(GlobalIndex),
+    /// A `ref.null`.
+    RefNullConst,
     ///< The global is imported from, and thus initialized by, a different module.
     Import,
 }
@@ -119,26 +121,34 @@ pub struct Memory {
 }
 
 /// Helper function translating wasmparser types to Cranelift types when possible.
-pub fn type_to_type(ty: wasmparser::Type) -> WasmResult {
+pub fn type_to_type(
+    ty: wasmparser::Type,
+    environ: &PE,
+) -> WasmResult {
     match ty {
         wasmparser::Type::I32 => Ok(ir::types::I32),
         wasmparser::Type::I64 => Ok(ir::types::I64),
         wasmparser::Type::F32 => Ok(ir::types::F32),
         wasmparser::Type::F64 => Ok(ir::types::F64),
         wasmparser::Type::V128 => Ok(ir::types::I8X16),
+        wasmparser::Type::AnyRef | wasmparser::Type::AnyFunc => Ok(environ.reference_type()),
         ty => Err(wasm_unsupported!("type_to_type: wasm type {:?}", ty)),
     }
 }
 
 /// Helper function translating wasmparser possible table types to Cranelift types when possible,
 /// or None for Func tables.
-pub fn tabletype_to_type(ty: wasmparser::Type) -> WasmResult> {
+pub fn tabletype_to_type(
+    ty: wasmparser::Type,
+    environ: &PE,
+) -> WasmResult> {
     match ty {
         wasmparser::Type::I32 => Ok(Some(ir::types::I32)),
         wasmparser::Type::I64 => Ok(Some(ir::types::I64)),
         wasmparser::Type::F32 => Ok(Some(ir::types::F32)),
         wasmparser::Type::F64 => Ok(Some(ir::types::F64)),
         wasmparser::Type::V128 => Ok(Some(ir::types::I8X16)),
+        wasmparser::Type::AnyRef => Ok(Some(environ.reference_type())),
         wasmparser::Type::AnyFunc => Ok(None),
         ty => Err(wasm_unsupported!(
             "tabletype_to_type: table wasm type {:?}",
@@ -159,6 +169,8 @@ pub fn blocktype_params_results(
             wasmparser::Type::F32 => (&[], &[wasmparser::Type::F32]),
             wasmparser::Type::F64 => (&[], &[wasmparser::Type::F64]),
             wasmparser::Type::V128 => (&[], &[wasmparser::Type::V128]),
+            wasmparser::Type::AnyRef => (&[], &[wasmparser::Type::AnyRef]),
+            wasmparser::Type::AnyFunc => (&[], &[wasmparser::Type::AnyFunc]),
             wasmparser::Type::EmptyBlockType => (&[], &[]),
             ty => return Err(wasm_unsupported!("blocktype_params_results: type {:?}", ty)),
         },
@@ -171,9 +183,10 @@ pub fn blocktype_params_results(
 }
 
 /// Create an `Ebb` with the given Wasm parameters.
-pub fn ebb_with_params(
+pub fn ebb_with_params(
     builder: &mut FunctionBuilder,
     params: &[wasmparser::Type],
+    environ: &PE,
 ) -> WasmResult {
     let ebb = builder.create_ebb();
     for ty in params.iter() {
@@ -190,6 +203,9 @@ pub fn ebb_with_params(
             wasmparser::Type::F64 => {
                 builder.append_ebb_param(ebb, ir::types::F64);
             }
+            wasmparser::Type::AnyRef | wasmparser::Type::AnyFunc => {
+                builder.append_ebb_param(ebb, environ.reference_type());
+            }
             wasmparser::Type::V128 => {
                 builder.append_ebb_param(ebb, ir::types::I8X16);
             }

From 0d8f8bc71fb35b217bdddced2a48c4e2205c5287 Mon Sep 17 00:00:00 2001
From: llogiq 
Date: Sat, 7 Dec 2019 18:47:43 +0100
Subject: [PATCH 2976/3084] Fix some clippy warnings (#1277)

---
 cranelift/codegen/build.rs                    |  2 +-
 .../codegen/meta/src/cdsl/type_inference.rs   |  4 ++--
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 22 +++++++++----------
 cranelift/codegen/src/binemit/memorysink.rs   |  2 ++
 cranelift/codegen/src/context.rs              |  2 ++
 cranelift/codegen/src/isa/mod.rs              |  2 --
 cranelift/codegen/src/isa/x86/abi.rs          |  2 +-
 cranelift/codegen/src/licm.rs                 |  1 -
 cranelift/codegen/src/verifier/mod.rs         |  2 +-
 cranelift/entity/src/boxed_slice.rs           |  4 ++++
 10 files changed, 24 insertions(+), 19 deletions(-)

diff --git a/cranelift/codegen/build.rs b/cranelift/codegen/build.rs
index f2afc404e6..bb14364050 100644
--- a/cranelift/codegen/build.rs
+++ b/cranelift/codegen/build.rs
@@ -61,7 +61,7 @@ fn main() {
         process::exit(1);
     }
 
-    if let Ok(_) = env::var("CRANELIFT_VERBOSE") {
+    if env::var("CRANELIFT_VERBOSE").is_ok() {
         for isa in &isas {
             println!("cargo:warning=Includes support for {} ISA", isa.to_string());
         }
diff --git a/cranelift/codegen/meta/src/cdsl/type_inference.rs b/cranelift/codegen/meta/src/cdsl/type_inference.rs
index 0313345175..25a07a9b84 100644
--- a/cranelift/codegen/meta/src/cdsl/type_inference.rs
+++ b/cranelift/codegen/meta/src/cdsl/type_inference.rs
@@ -459,7 +459,7 @@ fn constrain_fixpoint(tv1: &TypeVar, tv2: &TypeVar) {
         }
     }
 
-    let old_tv2_ts = tv2.get_typeset().clone();
+    let old_tv2_ts = tv2.get_typeset();
     tv1.constrain_types(tv2.clone());
     // The above loop should ensure that all reference cycles have been handled.
     assert!(old_tv2_ts == tv2.get_typeset());
@@ -627,7 +627,7 @@ pub(crate) fn infer_transform(
         .map(|&var_index| {
             let var = var_pool.get_mut(var_index);
             let tv = type_env.get_equivalent(&var.get_or_create_typevar());
-            (var_index, tv.get_typeset().clone())
+            (var_index, tv.get_typeset())
         })
         .collect::>();
 
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 8d271238ea..92d339e6bd 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -1381,7 +1381,7 @@ pub(crate) fn define<'shared>(
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("fst", &formats.store, 1)
                 .operands_in(vec![fpr, gpr])
-                .inst_predicate(has_no_offset.clone())
+                .inst_predicate(has_no_offset)
                 .clobbers_flags(false)
                 .compute_size("size_plus_maybe_sib_or_offset_for_in_reg_1")
                 .emit(
@@ -1465,7 +1465,7 @@ pub(crate) fn define<'shared>(
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("fstDisp8", &formats.store, 2)
                 .operands_in(vec![fpr, gpr])
-                .inst_predicate(has_small_offset.clone())
+                .inst_predicate(has_small_offset)
                 .clobbers_flags(false)
                 .compute_size("size_plus_maybe_sib_for_in_reg_1")
                 .emit(
@@ -1628,7 +1628,7 @@ pub(crate) fn define<'shared>(
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("fstWithIndex", &formats.store_complex, 2)
                 .operands_in(vec![fpr, gpr, gpr])
-                .inst_predicate(has_no_offset.clone())
+                .inst_predicate(has_no_offset)
                 .clobbers_flags(false)
                 .compute_size("size_plus_maybe_offset_for_in_reg_1")
                 .emit(
@@ -1698,7 +1698,7 @@ pub(crate) fn define<'shared>(
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("fstWithIndexDisp8", &formats.store_complex, 3)
                 .operands_in(vec![fpr, gpr, gpr])
-                .inst_predicate(has_small_offset.clone())
+                .inst_predicate(has_small_offset)
                 .clobbers_flags(false)
                 .emit(
                     r#"
@@ -1762,7 +1762,7 @@ pub(crate) fn define<'shared>(
         recipes.add_template_recipe(
             EncodingRecipeBuilder::new("fstWithIndexDisp32", &formats.store_complex, 6)
                 .operands_in(vec![fpr, gpr, gpr])
-                .inst_predicate(has_big_offset.clone())
+                .inst_predicate(has_big_offset)
                 .clobbers_flags(false)
                 .emit(
                     r#"
@@ -1892,7 +1892,7 @@ pub(crate) fn define<'shared>(
             EncodingRecipeBuilder::new("fld", &formats.load, 1)
                 .operands_in(vec![gpr])
                 .operands_out(vec![fpr])
-                .inst_predicate(has_no_offset.clone())
+                .inst_predicate(has_no_offset)
                 .clobbers_flags(false)
                 .compute_size("size_plus_maybe_sib_or_offset_for_in_reg_0")
                 .emit(
@@ -1948,7 +1948,7 @@ pub(crate) fn define<'shared>(
             EncodingRecipeBuilder::new("fldDisp8", &formats.load, 2)
                 .operands_in(vec![gpr])
                 .operands_out(vec![fpr])
-                .inst_predicate(has_small_offset.clone())
+                .inst_predicate(has_small_offset)
                 .clobbers_flags(false)
                 .compute_size("size_plus_maybe_sib_for_in_reg_0")
                 .emit(
@@ -2003,7 +2003,7 @@ pub(crate) fn define<'shared>(
             EncodingRecipeBuilder::new("fldDisp32", &formats.load, 5)
                 .operands_in(vec![gpr])
                 .operands_out(vec![fpr])
-                .inst_predicate(has_big_offset.clone())
+                .inst_predicate(has_big_offset)
                 .clobbers_flags(false)
                 .compute_size("size_plus_maybe_sib_for_in_reg_0")
                 .emit(
@@ -2064,7 +2064,7 @@ pub(crate) fn define<'shared>(
             EncodingRecipeBuilder::new("fldWithIndex", &formats.load_complex, 2)
                 .operands_in(vec![gpr, gpr])
                 .operands_out(vec![fpr])
-                .inst_predicate(has_no_offset.clone())
+                .inst_predicate(has_no_offset)
                 .clobbers_flags(false)
                 .compute_size("size_plus_maybe_offset_for_in_reg_0")
                 .emit(
@@ -2115,7 +2115,7 @@ pub(crate) fn define<'shared>(
             EncodingRecipeBuilder::new("fldWithIndexDisp8", &formats.load_complex, 3)
                 .operands_in(vec![gpr, gpr])
                 .operands_out(vec![fpr])
-                .inst_predicate(has_small_offset.clone())
+                .inst_predicate(has_small_offset)
                 .clobbers_flags(false)
                 .emit(
                     r#"
@@ -2160,7 +2160,7 @@ pub(crate) fn define<'shared>(
             EncodingRecipeBuilder::new("fldWithIndexDisp32", &formats.load_complex, 6)
                 .operands_in(vec![gpr, gpr])
                 .operands_out(vec![fpr])
-                .inst_predicate(has_big_offset.clone())
+                .inst_predicate(has_big_offset)
                 .clobbers_flags(false)
                 .emit(
                     r#"
diff --git a/cranelift/codegen/src/binemit/memorysink.rs b/cranelift/codegen/src/binemit/memorysink.rs
index ac3e82bea2..7e1cdf57a2 100644
--- a/cranelift/codegen/src/binemit/memorysink.rs
+++ b/cranelift/codegen/src/binemit/memorysink.rs
@@ -46,6 +46,8 @@ pub struct MemoryCodeSink<'a> {
 impl<'a> MemoryCodeSink<'a> {
     /// Create a new memory code sink that writes a function to the memory pointed to by `data`.
     ///
+    /// # Safety
+    ///
     /// This function is unsafe since `MemoryCodeSink` does not perform bounds checking on the
     /// memory buffer, and it can't guarantee that the `data` pointer is valid.
     pub unsafe fn new(
diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index 353b469bb6..c9fd1e6bcd 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -175,6 +175,8 @@ impl Context {
     ///
     /// The machine code is not relocated. Instead, any relocations are emitted into `relocs`.
     ///
+    /// # Safety
+    ///
     /// This function is unsafe since it does not perform bounds checking on the memory buffer,
     /// and it can't guarantee that the `mem` pointer is valid.
     ///
diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index 35f3581d7e..5679333394 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -22,7 +22,6 @@
 //! ```
 //! # extern crate cranelift_codegen;
 //! # #[macro_use] extern crate target_lexicon;
-//! # fn main() {
 //! use cranelift_codegen::isa;
 //! use cranelift_codegen::settings::{self, Configurable};
 //! use std::str::FromStr;
@@ -40,7 +39,6 @@
 //!         let isa = isa_builder.finish(shared_flags);
 //!     }
 //! }
-//! # }
 //! ```
 //!
 //! The configured target ISA trait object is a `Box` which can be used for multiple
diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 1ee48ddfcb..78f3288fd5 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -938,7 +938,7 @@ fn insert_common_epilogue(
             .entry(fp_pop_inst)
             .and_modify(|insts| {
                 *insts = insts
-                    .into_iter()
+                    .iter()
                     .cloned()
                     .chain(std::iter::once(new_cfa))
                     .collect::>();
diff --git a/cranelift/codegen/src/licm.rs b/cranelift/codegen/src/licm.rs
index 35fd2c0a80..4fa5d314f6 100644
--- a/cranelift/codegen/src/licm.rs
+++ b/cranelift/codegen/src/licm.rs
@@ -73,7 +73,6 @@ fn create_pre_header(
     let pool = &mut ListPool::::new();
     let header_args_values = func.dfg.ebb_params(header).to_vec();
     let header_args_types: Vec = header_args_values
-        .clone()
         .into_iter()
         .map(|val| func.dfg.value_type(val))
         .collect();
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 30990dbf9b..2e58ec4140 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -107,7 +107,7 @@ pub struct VerifierError {
 fn format_context(context: &Option) -> String {
     match context {
         None => "".to_string(),
-        Some(c) => format!(" ({})", c).to_string(),
+        Some(c) => format!(" ({})", c),
     }
 }
 
diff --git a/cranelift/entity/src/boxed_slice.rs b/cranelift/entity/src/boxed_slice.rs
index 8d01b60b7f..3b3b39155b 100644
--- a/cranelift/entity/src/boxed_slice.rs
+++ b/cranelift/entity/src/boxed_slice.rs
@@ -27,6 +27,10 @@ where
 {
     /// Create a new slice from a raw pointer. A safer way to create slices is
     /// to use `PrimaryMap::into_boxed_slice()`.
+    ///
+    /// # Safety
+    ///
+    /// This relies on `raw` pointing to a valid slice of `V`s.
     pub unsafe fn from_raw(raw: *mut [V]) -> Self {
         Self {
             elems: Box::from_raw(raw),

From 396eba523580ee75d72df042cb64f8f840da61aa Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 9 Dec 2019 10:05:47 -0800
Subject: [PATCH 2977/3084] Bitcast values incoming to `vall_true` and
 `vany_true` before use in Wasm translation (#1275)

---
 cranelift/wasm/src/code_translator.rs | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index c0d1218ccf..41f5726cd6 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1226,14 +1226,16 @@ pub fn translate_operator(
         | Operator::I16x8AnyTrue
         | Operator::I32x4AnyTrue
         | Operator::I64x2AnyTrue => {
-            let bool_result = builder.ins().vany_true(state.pop1());
+            let a = pop1_with_bitcast(state, type_of(op), builder);
+            let bool_result = builder.ins().vany_true(a);
             state.push1(builder.ins().bint(I32, bool_result))
         }
         Operator::I8x16AllTrue
         | Operator::I16x8AllTrue
         | Operator::I32x4AllTrue
         | Operator::I64x2AllTrue => {
-            let bool_result = builder.ins().vall_true(state.pop1());
+            let a = pop1_with_bitcast(state, type_of(op), builder);
+            let bool_result = builder.ins().vall_true(a);
             state.push1(builder.ins().bint(I32, bool_result))
         }
         Operator::I8x16Eq | Operator::I16x8Eq | Operator::I32x4Eq => {

From af64187ec7eb4d146bb70e2a07f0d343943eb6f5 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 12 Dec 2019 13:52:26 -0800
Subject: [PATCH 2978/3084] Truncate over-large parameters to splat (#1276)

See https://github.com/WebAssembly/simd/issues/149 for associated discussion on this.
---
 cranelift/wasm/src/code_translator.rs | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 41f5726cd6..efd4d22561 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1093,15 +1093,16 @@ pub fn translate_operator(
             // the v128.const is typed in CLIF as a I8x16 but raw_bitcast to a different type before use
             state.push1(value)
         }
-        Operator::I8x16Splat
-        | Operator::I16x8Splat
-        | Operator::I32x4Splat
+        Operator::I8x16Splat | Operator::I16x8Splat => {
+            let reduced = builder.ins().ireduce(type_of(op).lane_type(), state.pop1());
+            let splatted = builder.ins().splat(type_of(op), reduced);
+            state.push1(splatted)
+        }
+        Operator::I32x4Splat
         | Operator::I64x2Splat
         | Operator::F32x4Splat
         | Operator::F64x2Splat => {
-            let value_to_splat = state.pop1();
-            let ty = type_of(op);
-            let splatted = builder.ins().splat(ty, value_to_splat);
+            let splatted = builder.ins().splat(type_of(op), state.pop1());
             state.push1(splatted)
         }
         Operator::I8x16ExtractLaneS { lane } | Operator::I16x8ExtractLaneS { lane } => {

From d4df756acf7f163612d3b1b000e9f7e52207fda6 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 12 Dec 2019 17:01:31 -0800
Subject: [PATCH 2979/3084] Remove packed_struct dependency; closes #1271 and
 #1284 (#1282)

---
 cranelift/codegen/meta/src/isa/x86/recipes.rs |   2 +-
 cranelift/codegen/shared/Cargo.toml           |   4 +-
 .../shared/src/isa/x86/encoding_bits.rs       | 333 +++++++++++++-----
 cranelift/codegen/shared/src/lib.rs           |   4 -
 cranelift/codegen/src/isa/x86/binemit.rs      |   2 +-
 5 files changed, 253 insertions(+), 92 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 92d339e6bd..9451460482 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -100,7 +100,7 @@ impl<'builder> RecipeGroup<'builder> {
 /// Given a sequence of opcode bytes, compute the recipe name prefix and encoding bits.
 fn decode_opcodes(op_bytes: &[u8], rrr: u16, w: u16) -> (&'static str, u16) {
     let enc = EncodingBits::new(op_bytes, rrr, w);
-    (enc.prefix.recipe_name_prefix(), enc.bits())
+    (enc.prefix().recipe_name_prefix(), enc.bits())
 }
 
 /// Given a snippet of Rust code (or None), replace the `PUT_OP` macro with the
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 9ab0cd75ab..fe1bc5c0ef 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -8,6 +8,4 @@ repository = "https://github.com/bytecodealliance/cranelift"
 readme = "README.md"
 edition = "2018"
 
-[dependencies]
-packed_struct = "0.3"
-packed_struct_codegen = "0.3"
+# Since this is a shared dependency of several packages, please strive to keep this dependency-free.
diff --git a/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs b/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs
index cd83e99a32..5195c24c5c 100644
--- a/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs
+++ b/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs
@@ -1,6 +1,6 @@
 //! Provides a named interface to the `u16` Encoding bits.
 
-use packed_struct::prelude::*;
+use std::ops::RangeInclusive;
 
 /// Named interface to the `u16` Encoding bits, representing an opcode.
 ///
@@ -27,61 +27,103 @@ use packed_struct::prelude::*;
 ///        11: 0F 3A   (Op3/Mp3)
 /// 12-14  rrr, opcode bits for the ModR/M byte for certain opcodes.
 /// 15:    REX.W bit (or VEX.W/E)
-#[derive(Copy, Clone, PartialEq, PackedStruct)]
-#[packed_struct(size_bytes = "2", bit_numbering = "lsb0")]
-pub struct EncodingBits {
-    /// Instruction opcode byte, without the prefix.
-    #[packed_field(bits = "0:7")]
-    pub opcode_byte: u8,
-
-    /// Prefix kind for the instruction, as an enum.
-    #[packed_field(bits = "8:11", ty = "enum")]
-    pub prefix: OpcodePrefix,
-
-    /// Bits for the ModR/M byte for certain opcodes.
-    #[packed_field(bits = "12:14")]
-    pub rrr: Integer,
-
-    /// REX.W bit (or VEX.W/E).
-    #[packed_field(bits = "15")]
-    pub rex_w: Integer,
-}
+#[derive(Copy, Clone, PartialEq)]
+pub struct EncodingBits(u16);
+const OPCODE: RangeInclusive = 0..=7;
+const OPCODE_PREFIX: RangeInclusive = 8..=11; // Includes pp and mm.
+const RRR: RangeInclusive = 12..=14;
+const REX_W: RangeInclusive = 15..=15;
 
 impl From for EncodingBits {
-    fn from(bits: u16) -> EncodingBits {
-        let bytes: [u8; 2] = [((bits >> 8) & 0xff) as u8, (bits & 0xff) as u8];
-        EncodingBits::unpack(&bytes).expect("failed creating EncodingBits")
+    fn from(bits: u16) -> Self {
+        Self(bits)
     }
 }
 
 impl EncodingBits {
     /// Constructs a new EncodingBits from parts.
     pub fn new(op_bytes: &[u8], rrr: u16, rex_w: u16) -> Self {
-        EncodingBits {
-            opcode_byte: op_bytes[op_bytes.len() - 1],
-            prefix: OpcodePrefix::from_opcode(op_bytes),
-            rrr: (rrr as u8).into(),
-            rex_w: (rex_w as u8).into(),
-        }
+        assert!(
+            !op_bytes.is_empty(),
+            "op_bytes must include at least one opcode byte"
+        );
+        let mut new = Self::from(0);
+        let last_byte = op_bytes[op_bytes.len() - 1];
+        new.write(OPCODE, last_byte as u16);
+        let prefix: u8 = OpcodePrefix::from_opcode(op_bytes).into();
+        new.write(OPCODE_PREFIX, prefix as u16);
+        new.write(RRR, rrr);
+        new.write(REX_W, rex_w);
+        new
     }
 
     /// Returns the raw bits.
     #[inline]
     pub fn bits(self) -> u16 {
-        let bytes: [u8; 2] = self.pack();
-        ((bytes[0] as u16) << 8) | (bytes[1] as u16)
+        self.0
+    }
+
+    /// Convenience method for writing bits to specific range.
+    #[inline]
+    fn write(&mut self, range: RangeInclusive, value: u16) {
+        assert!(ExactSizeIterator::len(&range) > 0);
+        let size = range.end() - range.start() + 1; // Calculate the number of bits in the range.
+        let mask = (1 << size) - 1; // Generate a bit mask.
+        debug_assert!(
+            value <= mask,
+            "The written value should have fewer than {} bits.",
+            size
+        );
+        let mask_complement = !(mask << *range.start()); // Create the bitwise complement for the clear mask.
+        self.0 &= mask_complement; // Clear the bits in `range`.
+        let value = (value & mask) << *range.start(); // Place the value in the correct location.
+        self.0 |= value; // Modify the bits in `range`.
+    }
+
+    /// Convenience method for reading bits from a specific range.
+    #[inline]
+    fn read(self, range: RangeInclusive) -> u8 {
+        assert!(ExactSizeIterator::len(&range) > 0);
+        let size = range.end() - range.start() + 1; // Calculate the number of bits in the range.
+        debug_assert!(size <= 8, "This structure expects ranges of at most 8 bits");
+        let mask = (1 << size) - 1; // Generate a bit mask.
+        ((self.0 >> *range.start()) & mask) as u8
+    }
+
+    /// Instruction opcode byte, without the prefix.
+    #[inline]
+    pub fn opcode_byte(self) -> u8 {
+        self.read(OPCODE)
+    }
+
+    /// Prefix kind for the instruction, as an enum.
+    #[inline]
+    pub fn prefix(self) -> OpcodePrefix {
+        OpcodePrefix::from(self.read(OPCODE_PREFIX))
     }
 
     /// Extracts the PP bits of the OpcodePrefix.
     #[inline]
     pub fn pp(self) -> u8 {
-        self.prefix.to_primitive() & 0x3
+        self.prefix().to_primitive() & 0x3
     }
 
     /// Extracts the MM bits of the OpcodePrefix.
     #[inline]
     pub fn mm(self) -> u8 {
-        (self.prefix.to_primitive() >> 2) & 0x3
+        (self.prefix().to_primitive() >> 2) & 0x3
+    }
+
+    /// Bits for the ModR/M byte for certain opcodes.
+    #[inline]
+    pub fn rrr(self) -> u8 {
+        self.read(RRR)
+    }
+
+    /// REX.W bit (or VEX.W/E).
+    #[inline]
+    pub fn rex_w(self) -> u8 {
+        self.read(REX_W)
     }
 }
 
@@ -90,55 +132,103 @@ impl EncodingBits {
 /// The prefix type occupies four of the EncodingBits.
 #[allow(non_camel_case_types)]
 #[allow(missing_docs)]
-#[derive(Copy, Clone, Debug, Eq, PartialEq, PrimitiveEnum_u8)]
+#[derive(Copy, Clone, Debug, Eq, PartialEq)]
 pub enum OpcodePrefix {
-    Op1 = 0b0000,
-    Mp1_66 = 0b0001,
-    Mp1_f3 = 0b0010,
-    Mp1_f2 = 0b0011,
-    Op2_0f = 0b0100,
-    Mp2_66_0f = 0b0101,
-    Mp2_f3_0f = 0b0110,
-    Mp2_f2_0f = 0b0111,
-    Op3_0f_38 = 0b1000,
-    Mp3_66_0f_38 = 0b1001,
-    Mp3_f3_0f_38 = 0b1010,
-    Mp3_f2_0f_38 = 0b1011,
-    Op3_0f_3a = 0b1100,
-    Mp3_66_0f_3a = 0b1101,
-    Mp3_f3_0f_3a = 0b1110,
-    Mp3_f2_0f_3a = 0b1111,
+    Op1,
+    Mp1_66,
+    Mp1_f3,
+    Mp1_f2,
+    Op2_0f,
+    Mp2_66_0f,
+    Mp2_f3_0f,
+    Mp2_f2_0f,
+    Op3_0f_38,
+    Mp3_66_0f_38,
+    Mp3_f3_0f_38,
+    Mp3_f2_0f_38,
+    Op3_0f_3a,
+    Mp3_66_0f_3a,
+    Mp3_f3_0f_3a,
+    Mp3_f2_0f_3a,
 }
 
 impl From for OpcodePrefix {
-    fn from(n: u8) -> OpcodePrefix {
-        OpcodePrefix::from_primitive(n).expect("invalid OpcodePrefix")
+    fn from(n: u8) -> Self {
+        use OpcodePrefix::*;
+        match n {
+            0b0000 => Op1,
+            0b0001 => Mp1_66,
+            0b0010 => Mp1_f3,
+            0b0011 => Mp1_f2,
+            0b0100 => Op2_0f,
+            0b0101 => Mp2_66_0f,
+            0b0110 => Mp2_f3_0f,
+            0b0111 => Mp2_f2_0f,
+            0b1000 => Op3_0f_38,
+            0b1001 => Mp3_66_0f_38,
+            0b1010 => Mp3_f3_0f_38,
+            0b1011 => Mp3_f2_0f_38,
+            0b1100 => Op3_0f_3a,
+            0b1101 => Mp3_66_0f_3a,
+            0b1110 => Mp3_f3_0f_3a,
+            0b1111 => Mp3_f2_0f_3a,
+            _ => panic!("invalid opcode prefix"),
+        }
+    }
+}
+
+impl Into for OpcodePrefix {
+    fn into(self) -> u8 {
+        use OpcodePrefix::*;
+        match self {
+            Op1 => 0b0000,
+            Mp1_66 => 0b0001,
+            Mp1_f3 => 0b0010,
+            Mp1_f2 => 0b0011,
+            Op2_0f => 0b0100,
+            Mp2_66_0f => 0b0101,
+            Mp2_f3_0f => 0b0110,
+            Mp2_f2_0f => 0b0111,
+            Op3_0f_38 => 0b1000,
+            Mp3_66_0f_38 => 0b1001,
+            Mp3_f3_0f_38 => 0b1010,
+            Mp3_f2_0f_38 => 0b1011,
+            Op3_0f_3a => 0b1100,
+            Mp3_66_0f_3a => 0b1101,
+            Mp3_f3_0f_3a => 0b1110,
+            Mp3_f2_0f_3a => 0b1111,
+        }
     }
 }
 
 impl OpcodePrefix {
+    /// Convert an opcode prefix to a `u8`; this is a convenience proxy for `Into`.
+    fn to_primitive(self) -> u8 {
+        self.into()
+    }
+
     /// Extracts the OpcodePrefix from the opcode.
-    pub fn from_opcode(op_bytes: &[u8]) -> OpcodePrefix {
+    pub fn from_opcode(op_bytes: &[u8]) -> Self {
         assert!(!op_bytes.is_empty(), "at least one opcode byte");
 
         let prefix_bytes = &op_bytes[..op_bytes.len() - 1];
         match prefix_bytes {
-            [] => OpcodePrefix::Op1,
-            [0x66] => OpcodePrefix::Mp1_66,
-            [0xf3] => OpcodePrefix::Mp1_f3,
-            [0xf2] => OpcodePrefix::Mp1_f2,
-            [0x0f] => OpcodePrefix::Op2_0f,
-            [0x66, 0x0f] => OpcodePrefix::Mp2_66_0f,
-            [0xf3, 0x0f] => OpcodePrefix::Mp2_f3_0f,
-            [0xf2, 0x0f] => OpcodePrefix::Mp2_f2_0f,
-            [0x0f, 0x38] => OpcodePrefix::Op3_0f_38,
-            [0x66, 0x0f, 0x38] => OpcodePrefix::Mp3_66_0f_38,
-            [0xf3, 0x0f, 0x38] => OpcodePrefix::Mp3_f3_0f_38,
-            [0xf2, 0x0f, 0x38] => OpcodePrefix::Mp3_f2_0f_38,
-            [0x0f, 0x3a] => OpcodePrefix::Op3_0f_3a,
-            [0x66, 0x0f, 0x3a] => OpcodePrefix::Mp3_66_0f_3a,
-            [0xf3, 0x0f, 0x3a] => OpcodePrefix::Mp3_f3_0f_3a,
-            [0xf2, 0x0f, 0x3a] => OpcodePrefix::Mp3_f2_0f_3a,
+            [] => Self::Op1,
+            [0x66] => Self::Mp1_66,
+            [0xf3] => Self::Mp1_f3,
+            [0xf2] => Self::Mp1_f2,
+            [0x0f] => Self::Op2_0f,
+            [0x66, 0x0f] => Self::Mp2_66_0f,
+            [0xf3, 0x0f] => Self::Mp2_f3_0f,
+            [0xf2, 0x0f] => Self::Mp2_f2_0f,
+            [0x0f, 0x38] => Self::Op3_0f_38,
+            [0x66, 0x0f, 0x38] => Self::Mp3_66_0f_38,
+            [0xf3, 0x0f, 0x38] => Self::Mp3_f3_0f_38,
+            [0xf2, 0x0f, 0x38] => Self::Mp3_f2_0f_38,
+            [0x0f, 0x3a] => Self::Op3_0f_3a,
+            [0x66, 0x0f, 0x3a] => Self::Mp3_66_0f_3a,
+            [0xf3, 0x0f, 0x3a] => Self::Mp3_f3_0f_3a,
+            [0xf2, 0x0f, 0x3a] => Self::Mp3_f2_0f_3a,
             _ => {
                 panic!("unexpected opcode sequence: {:?}", op_bytes);
             }
@@ -193,38 +283,93 @@ mod tests {
         test_roundtrip(OpcodePrefix::Mp3_f2_0f_3a);
     }
 
+    #[test]
+    fn prefix_to_name() {
+        assert_eq!(OpcodePrefix::Op1.recipe_name_prefix(), "Op1");
+        assert_eq!(OpcodePrefix::Op2_0f.recipe_name_prefix(), "Op2");
+        assert_eq!(OpcodePrefix::Op3_0f_38.recipe_name_prefix(), "Op3");
+        assert_eq!(OpcodePrefix::Mp1_66.recipe_name_prefix(), "Mp1");
+        assert_eq!(OpcodePrefix::Mp2_66_0f.recipe_name_prefix(), "Mp2");
+        assert_eq!(OpcodePrefix::Mp3_66_0f_3a.recipe_name_prefix(), "Mp3");
+    }
+
     /// Tests that the opcode_byte is the lower of the EncodingBits.
     #[test]
     fn encodingbits_opcode_byte() {
         let enc = EncodingBits::from(0x00ff);
-        assert_eq!(enc.opcode_byte, 0xff);
-        assert_eq!(enc.prefix.to_primitive(), 0x0);
-        assert_eq!(u8::from(enc.rrr), 0x0);
-        assert_eq!(u8::from(enc.rex_w), 0x0);
+        assert_eq!(enc.opcode_byte(), 0xff);
+        assert_eq!(enc.prefix().to_primitive(), 0x0);
+        assert_eq!(enc.rrr(), 0x0);
+        assert_eq!(enc.rex_w(), 0x0);
 
         let enc = EncodingBits::from(0x00cd);
-        assert_eq!(enc.opcode_byte, 0xcd);
+        assert_eq!(enc.opcode_byte(), 0xcd);
     }
 
     /// Tests that the OpcodePrefix is encoded correctly.
     #[test]
     fn encodingbits_prefix() {
         let enc = EncodingBits::from(0x0c00);
-        assert_eq!(enc.opcode_byte, 0x00);
-        assert_eq!(enc.prefix.to_primitive(), 0xc);
-        assert_eq!(enc.prefix, OpcodePrefix::Op3_0f_3a);
-        assert_eq!(u8::from(enc.rrr), 0x0);
-        assert_eq!(u8::from(enc.rex_w), 0x0);
+        assert_eq!(enc.opcode_byte(), 0x00);
+        assert_eq!(enc.prefix().to_primitive(), 0xc);
+        assert_eq!(enc.prefix(), OpcodePrefix::Op3_0f_3a);
+        assert_eq!(enc.rrr(), 0x0);
+        assert_eq!(enc.rex_w(), 0x0);
+    }
+
+    /// Tests that the PP bits are encoded correctly.
+    #[test]
+    fn encodingbits_pp() {
+        let enc = EncodingBits::from(0x0300);
+        assert_eq!(enc.opcode_byte(), 0x0);
+        assert_eq!(enc.pp(), 0x3);
+        assert_eq!(enc.mm(), 0x0);
+        assert_eq!(enc.rrr(), 0x0);
+        assert_eq!(enc.rex_w(), 0x0);
+    }
+
+    /// Tests that the MM bits are encoded correctly.
+    #[test]
+    fn encodingbits_mm() {
+        let enc = EncodingBits::from(0x0c00);
+        assert_eq!(enc.opcode_byte(), 0x0);
+        assert_eq!(enc.pp(), 0x00);
+        assert_eq!(enc.mm(), 0x3);
+        assert_eq!(enc.rrr(), 0x0);
+        assert_eq!(enc.rex_w(), 0x0);
+    }
+
+    /// Tests that the ModR/M bits are encoded correctly.
+    #[test]
+    fn encodingbits_rrr() {
+        let enc = EncodingBits::from(0x5000);
+        assert_eq!(enc.opcode_byte(), 0x0);
+        assert_eq!(enc.prefix().to_primitive(), 0x0);
+        assert_eq!(enc.rrr(), 0x5);
+        assert_eq!(enc.rex_w(), 0x0);
     }
 
     /// Tests that the REX.W bit is encoded correctly.
     #[test]
     fn encodingbits_rex_w() {
         let enc = EncodingBits::from(0x8000);
-        assert_eq!(enc.opcode_byte, 0x00);
-        assert_eq!(enc.prefix.to_primitive(), 0x0);
-        assert_eq!(u8::from(enc.rrr), 0x0);
-        assert_eq!(u8::from(enc.rex_w), 0x1);
+        assert_eq!(enc.opcode_byte(), 0x00);
+        assert_eq!(enc.prefix().to_primitive(), 0x0);
+        assert_eq!(enc.rrr(), 0x0);
+        assert_eq!(enc.rex_w(), 0x1);
+    }
+
+    /// Tests setting and unsetting a bit using EncodingBits::write.
+    #[test]
+    fn encodingbits_flip() {
+        let mut bits = EncodingBits::from(0);
+        let range = 2..=2;
+
+        bits.write(range.clone(), 1);
+        assert_eq!(bits.bits(), 0b100);
+
+        bits.write(range, 0);
+        assert_eq!(bits.bits(), 0b000);
     }
 
     /// Tests a round-trip of EncodingBits from/to a u16 (hardcoded endianness).
@@ -233,4 +378,26 @@ mod tests {
         let bits: u16 = 0x1234;
         assert_eq!(EncodingBits::from(bits).bits(), bits);
     }
+
+    #[test]
+    // I purposely want to divide the bits using the ranges defined above.
+    #[allow(clippy::inconsistent_digit_grouping)]
+    fn encodingbits_construction() {
+        assert_eq!(
+            EncodingBits::new(&[0x66, 0x40], 5, 1).bits(),
+            0b1_101_0001_01000000 // 1 = rex_w, 101 = rrr, 0001 = prefix, 01000000 = opcode
+        );
+    }
+
+    #[test]
+    #[should_panic]
+    fn encodingbits_panics_at_write_to_invalid_range() {
+        EncodingBits::from(0).write(1..=0, 42);
+    }
+
+    #[test]
+    #[should_panic]
+    fn encodingbits_panics_at_read_to_invalid_range() {
+        EncodingBits::from(0).read(1..=0);
+    }
 }
diff --git a/cranelift/codegen/shared/src/lib.rs b/cranelift/codegen/shared/src/lib.rs
index 771f5dfd9d..4a0cf24873 100644
--- a/cranelift/codegen/shared/src/lib.rs
+++ b/cranelift/codegen/shared/src/lib.rs
@@ -20,10 +20,6 @@
     )
 )]
 
-use packed_struct;
-#[macro_use]
-extern crate packed_struct_codegen;
-
 pub mod condcodes;
 pub mod constant_hash;
 pub mod constants;
diff --git a/cranelift/codegen/src/isa/x86/binemit.rs b/cranelift/codegen/src/isa/x86/binemit.rs
index 626301d0e2..4fc074abe5 100644
--- a/cranelift/codegen/src/isa/x86/binemit.rs
+++ b/cranelift/codegen/src/isa/x86/binemit.rs
@@ -67,7 +67,7 @@ fn rex3(rm: RegUnit, reg: RegUnit, index: RegUnit) -> u8 {
 // extracted from `bits`.
 fn rex_prefix(bits: u16, rex: u8, sink: &mut CS) {
     debug_assert_eq!(rex & 0xf8, BASE_REX);
-    let w = EncodingBits::from(bits).rex_w;
+    let w = EncodingBits::from(bits).rex_w();
     sink.put1(rex | (u8::from(w) << 3));
 }
 

From fcb05937964186c6fa7fa0f8e411c4bff59d3c62 Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Sat, 14 Dec 2019 14:33:36 -0500
Subject: [PATCH 2980/3084] [module] Finalize definitions for the end-user

Closes https://github.com/bytecodealliance/cranelift/issues/1288 by
calling `module.finalize_definitions` whenever `module.finish` is
called.
---
 cranelift/module/src/module.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 34964a7b61..56a94b89db 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -714,7 +714,8 @@ where
     /// Consume the module and return the resulting `Product`. Some `Backend`
     /// implementations may provide additional functionality available after
     /// a `Module` is complete.
-    pub fn finish(self) -> B::Product {
+    pub fn finish(mut self) -> B::Product {
+        self.finalize_definitions();
         self.backend.finish()
     }
 }

From 0604ec480c44ee97aff82490a713b2bff23718c3 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 16 Dec 2019 10:17:08 -0800
Subject: [PATCH 2981/3084] Fix `scalar_to_vector`: move not wide enough for
 64-bit values (#1287)

Previously, the use of `enc_x86_64` emitted two 64-bit mode encodings for `scalar_to_vector.i64`, neither of which contained the REX.W bit telling `MOVD/MOVQ` to move 64 bits of data instead of 32 bits. Now, `scalar_to_vector.i64` will always use a sole 64-bit mode REX.W encoding and `scalar_to_vector` with other widths will have three encodings: a 32-bit mode move, a 64-bit mode move with no REX, and a 64-bit mode move with REX (but not REX.W).
---
 cranelift/codegen/meta/src/isa/x86/encodings.rs    |  7 +++++--
 .../isa/x86/scalar_to_vector-binemit.clif          |  2 +-
 .../filetests/isa/x86/simd-construction-run.clif   | 14 ++++++++++++++
 3 files changed, 20 insertions(+), 3 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/simd-construction-run.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 002963035f..26b5334c26 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1746,10 +1746,13 @@ pub(crate) fn define(
         } else {
             let template = rec_frurm.opcodes(&MOVD_LOAD_XMM);
             if ty.lane_bits() < 64 {
-                // no 32-bit encodings for 64-bit widths
                 e.enc32(instruction.clone(), template.clone());
+                e.enc_x86_64(instruction, template);
+            } else {
+                // No 32-bit encodings for 64-bit widths.
+                assert_eq!(ty.lane_bits(), 64);
+                e.enc64(instruction, template.rex().w());
             }
-            e.enc_x86_64(instruction, template);
         }
     }
 
diff --git a/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif
index 149d54b03f..112939cd7d 100644
--- a/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif
@@ -27,6 +27,6 @@ ebb0:
 function %test_scalar_to_vector_i64() {
 ebb0:
 [-, %rdx]   v0 = iconst.i64 42
-[-, %xmm7]  v1 = scalar_to_vector.i64x2 v0    ; bin: 66 0f 6e fa
+[-, %xmm7]  v1 = scalar_to_vector.i64x2 v0    ; bin: 66 48 0f 6e fa
             return
 }
diff --git a/cranelift/filetests/filetests/isa/x86/simd-construction-run.clif b/cranelift/filetests/filetests/isa/x86/simd-construction-run.clif
new file mode 100644
index 0000000000..eecd82e024
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/simd-construction-run.clif
@@ -0,0 +1,14 @@
+test run
+set enable_simd
+target x86_64 skylake
+
+function %splat_i64x2() -> b1 {
+ebb0:
+    v0 = iconst.i64 -1
+    v1 = splat.i64x2 v0
+    v2 = vconst.i64x2 [-1 -1]
+    v3 = icmp eq v1, v2
+    v8 = vall_true v3
+    return v8
+}
+; run

From 6181f20326b73be84fc9b6170a3fac9772b64cf2 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 16 Dec 2019 10:32:08 -0800
Subject: [PATCH 2982/3084] Fix legalization of SIMD `fneg` (#1286)

Previously `fsub` was used but this fails when negating -0.0 and +0.0 in the SIMD spec tests; using more instructions, this change uses shifts to create a constant for flipping the most significant bit of each lane with `bxor`.
---
 cranelift/codegen/meta/src/isa/x86/legalize.rs      | 13 +++++++++++--
 .../filetests/isa/x86/simd-arithmetic-legalize.clif | 12 ++++++++----
 .../filetests/isa/x86/simd-arithmetic-run.clif      | 13 +++++++++++++
 3 files changed, 32 insertions(+), 6 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 2728cad5cf..6644bae161 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -40,7 +40,6 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let fmax = insts.by_name("fmax");
     let fmin = insts.by_name("fmin");
     let fneg = insts.by_name("fneg");
-    let fsub = insts.by_name("fsub");
     let iadd = insts.by_name("iadd");
     let icmp = insts.by_name("icmp");
     let iconst = insts.by_name("iconst");
@@ -48,6 +47,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let ineg = insts.by_name("ineg");
     let insertlane = insts.by_name("insertlane");
     let ishl = insts.by_name("ishl");
+    let ishl_imm = insts.by_name("ishl_imm");
     let isub = insts.by_name("isub");
     let popcnt = insts.by_name("popcnt");
     let raw_bitcast = insts.by_name("raw_bitcast");
@@ -550,9 +550,18 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
 
     for ty in &[F32, F64] {
         let fneg = fneg.bind(vector(*ty, sse_vector_size));
+        let lane_type_as_int = LaneType::int_from_bits(LaneType::from(*ty).lane_bits() as u16);
+        let uimm8_shift = Literal::constant(&imm.uimm8, lane_type_as_int.lane_bits() as i64 - 1);
+        let vconst = vconst.bind(vector(lane_type_as_int, sse_vector_size));
+        let bitcast_to_float = raw_bitcast.bind(vector(*ty, sse_vector_size));
         narrow.legalize(
             def!(b = fneg(a)),
-            vec![def!(c = vconst(u128_zeroes)), def!(b = fsub(c, a))],
+            vec![
+                def!(c = vconst(ones)),
+                def!(d = ishl_imm(c, uimm8_shift)), // Create a mask of all 0s except the MSB.
+                def!(e = bitcast_to_float(d)),      // Cast mask to the floating-point type.
+                def!(b = bxor(a, e)),               // Flip the MSB.
+            ],
         );
     }
 
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
index 324027741b..b6b033833e 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
@@ -39,13 +39,17 @@ function %fneg_legalized() {
 ebb0:
     v0 = vconst.f32x4 [0x1.0 0x2.0 0x3.0 0x4.0]
     v1 = fneg v0
-    ; check: v4 = vconst.f32x4 0x00
-    ; nextln: v1 = fsub v4, v0
+    ; check: v4 = vconst.i32x4 0xffffffffffffffffffffffffffffffff
+    ; nextln: v5 = ishl_imm v4, 31
+    ; nextln: v6 = raw_bitcast.f32x4 v5
+    ; nextln: v1 = bxor v0, v6
 
     v2 = vconst.f64x2 [0x1.0 0x2.0]
     v3 = fneg v2
-    ; check: v5 = vconst.f64x2 0x00
-    ; nextln: v3 = fsub v5, v2
+    ; check: v7 = vconst.i64x2 0xffffffffffffffffffffffffffffffff
+    ; nextln: v8 = ishl_imm v7, 63
+    ; nextln: v9 = raw_bitcast.f64x2 v8
+    ; nextln: v3 = bxor v2, v9
 
     return
 }
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
index 04facb0078..429928b213 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
@@ -240,6 +240,19 @@ ebb0:
 }
 ; run
 
+function %fneg_f32x4() -> b1 {
+ebb0:
+    v0 = vconst.f32x4 [0x0.0 -0x0.0 -Inf Inf]
+    v1 = fneg v0
+
+    v2 = vconst.f32x4 [-0x0.0 0x0.0 Inf -Inf]
+    v3 = fcmp eq v1, v2
+    v4 = vall_true v3
+
+    return v4
+}
+; run
+
 function %fabs_f32x4() -> b1 {
 ebb0:
     v0 = vconst.f32x4 [0x0.0 -0x1.0 0x2.0 -0x3.0]

From 4433ad28580a13eac9e3d6aad9f99d22ccbe5427 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 16 Dec 2019 13:14:51 -0800
Subject: [PATCH 2983/3084] Fix legalization of `icmp ugt` (#1278)

Previously, the same pattern (pmax + pcmpeq) as `uge` was used but this logic was incorrect for operands with equal values.
---
 cranelift/codegen/meta/src/isa/x86/legalize.rs     | 12 ++++++++----
 .../isa/x86/simd-comparison-legalize.clif          |  4 +++-
 .../filetests/isa/x86/simd-comparison-run.clif     | 14 ++++++++++++++
 3 files changed, 25 insertions(+), 5 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 6644bae161..8e7c79d814 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -335,6 +335,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let uimm8_zero = Literal::constant(&imm.uimm8, 0x00);
     let uimm8_one = Literal::constant(&imm.uimm8, 0x01);
     let u128_zeroes = constant(vec![0x00; 16]);
+    let u128_ones = constant(vec![0xff; 16]);
     let b = var("b");
     let c = var("c");
     let d = var("d");
@@ -405,12 +406,11 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     }
 
     // SIMD bnot
-    let ones = constant(vec![0xff; 16]);
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         let bnot = bnot.bind(vector(ty, sse_vector_size));
         narrow.legalize(
             def!(y = bnot(x)),
-            vec![def!(a = vconst(ones)), def!(y = bxor(a, x))],
+            vec![def!(a = vconst(u128_ones)), def!(y = bxor(a, x))],
         );
     }
 
@@ -524,7 +524,11 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         let icmp_ = icmp.bind(vector(*ty, sse_vector_size));
         narrow.legalize(
             def!(c = icmp_(ugt, a, b)),
-            vec![def!(x = x86_pmaxu(a, b)), def!(c = icmp(eq, a, x))],
+            vec![
+                def!(x = x86_pmaxu(a, b)),
+                def!(y = icmp(eq, x, b)),
+                def!(c = bnot(y)),
+            ],
         );
         let icmp_ = icmp.bind(vector(*ty, sse_vector_size));
         narrow.legalize(
@@ -574,7 +578,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         narrow.legalize(
             def!(b = fabs(a)),
             vec![
-                def!(c = vconst(ones)),
+                def!(c = vconst(u128_ones)),
                 def!(d = ushr_imm(c, uimm8_one)), // Create a mask of all 1s except the MSB.
                 def!(e = bitcast_to_float(d)),    // Cast mask to the floating-point type.
                 def!(b = band(a, e)),             // Unset the MSB.
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif
index b4d9681285..acbff943eb 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif
@@ -15,7 +15,9 @@ function %icmp_ugt_i32x4(i32x4, i32x4) -> b32x4 {
 ebb0(v0: i32x4, v1: i32x4):
     v2 = icmp ugt v0, v1
     ; check: v3 = x86_pmaxu v0, v1
-    ; nextln: v2 = icmp eq v0, v3
+    ; nextln: v4 = icmp eq v3, v1
+    ; nextln: v5 = vconst.b32x4 0xffffffffffffffffffffffffffffffff
+    ; nextln: v2 = bxor v5, v4
     return v2
 }
 
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
index 4b9da6e4a2..e4a5e6fea7 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
@@ -158,6 +158,20 @@ ebb0:
 }
 ; run
 
+
+function %icmp_ult_i16x8() -> b1 {
+ebb0:
+    v0 = vconst.i16x8 [-1 -1 -1 -1 -1 -1 -1 -1]
+    v1 = vconst.i16x8 [-1 -1 -1 -1 -1 -1 -1 -1]
+    v2 = icmp ult v0, v1
+    v3 = vconst.i16x8 0x00
+    v4 = raw_bitcast.i16x8 v2
+    v5 = icmp eq v3, v4
+    v8 = vall_true v5
+    return v8
+}
+; run
+
 function %icmp_sle_i16x8() -> b1 {
 ebb0:
     v0 = vconst.i16x8 [-1 -1 0 0 0 0 0 0]

From 86b66e8edefb9a99e8e2b29150ce060c44328e1f Mon Sep 17 00:00:00 2001
From: Philip Craig 
Date: Tue, 17 Dec 2019 13:38:09 +1000
Subject: [PATCH 2984/3084] Fix build failure in cranelift-codegen (#1294)

error[E0425]: cannot find value `ones` in this scope
   --> cranelift-codegen/meta/src/isa/x86/legalize.rs:564:33
    |
564 |                 def!(c = vconst(ones)),
    |                                 ^^^^ not found in this scope
---
 cranelift/codegen/meta/src/isa/x86/legalize.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 8e7c79d814..f62019367b 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -561,7 +561,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         narrow.legalize(
             def!(b = fneg(a)),
             vec![
-                def!(c = vconst(ones)),
+                def!(c = vconst(u128_ones)),
                 def!(d = ishl_imm(c, uimm8_shift)), // Create a mask of all 0s except the MSB.
                 def!(e = bitcast_to_float(d)),      // Cast mask to the floating-point type.
                 def!(b = bxor(a, e)),               // Flip the MSB.

From 8db734971274fc3427552a306532b2711f38b100 Mon Sep 17 00:00:00 2001
From: Y-Nak 
Date: Tue, 17 Dec 2019 19:28:45 +0900
Subject: [PATCH 2985/3084] Report when output register annotation is missing
 (#1289)

---
 cranelift/filetests/src/test_binemit.rs | 54 +++++++++++++------------
 1 file changed, 29 insertions(+), 25 deletions(-)

diff --git a/cranelift/filetests/src/test_binemit.rs b/cranelift/filetests/src/test_binemit.rs
index b291f61446..200251bfa4 100644
--- a/cranelift/filetests/src/test_binemit.rs
+++ b/cranelift/filetests/src/test_binemit.rs
@@ -231,18 +231,8 @@ impl SubTest for TestBinEmit {
                 // Send legal encodings into the emitter.
                 if enc.is_legal() {
                     // Generate a better error message if output locations are not specified.
-                    if let Some(&v) = func
-                        .dfg
-                        .inst_results(inst)
-                        .iter()
-                        .find(|&&v| !func.locations[v].is_assigned())
-                    {
-                        return Err(format!(
-                            "Missing register/stack slot for {} in {}",
-                            v,
-                            func.dfg.display_inst(inst, isa)
-                        ));
-                    }
+                    validate_location_annotations(&func, inst, isa, false)?;
+
                     let before = sink.offset;
                     isa.emit_inst(&func, inst, &mut divert, &mut sink);
                     let emitted = sink.offset - before;
@@ -260,19 +250,9 @@ impl SubTest for TestBinEmit {
                 if let Some(want) = bins.remove(&inst) {
                     if !enc.is_legal() {
                         // A possible cause of an unencoded instruction is a missing location for
-                        // one of the input operands.
-                        if let Some(&v) = func
-                            .dfg
-                            .inst_args(inst)
-                            .iter()
-                            .find(|&&v| !func.locations[v].is_assigned())
-                        {
-                            return Err(format!(
-                                "Missing register/stack slot for {} in {}",
-                                v,
-                                func.dfg.display_inst(inst, isa)
-                            ));
-                        }
+                        // one of the input/output operands.
+                        validate_location_annotations(&func, inst, isa, true)?;
+                        validate_location_annotations(&func, inst, isa, false)?;
 
                         // Do any encodings exist?
                         let encodings = isa
@@ -337,3 +317,27 @@ impl SubTest for TestBinEmit {
         Ok(())
     }
 }
+
+/// Validate registers/stack slots are correctly annotated.
+fn validate_location_annotations(
+    func: &ir::Function,
+    inst: ir::Inst,
+    isa: &dyn isa::TargetIsa,
+    validate_inputs: bool,
+) -> SubtestResult<()> {
+    let values = if validate_inputs {
+        func.dfg.inst_args(inst)
+    } else {
+        func.dfg.inst_results(inst)
+    };
+
+    if let Some(&v) = values.iter().find(|&&v| !func.locations[v].is_assigned()) {
+        Err(format!(
+            "Need register/stack slot annotation for {} in {}",
+            v,
+            func.dfg.display_inst(inst, isa)
+        ))
+    } else {
+        Ok(())
+    }
+}

From 887f897c9a148e87a265d16e81c596efe5091c8d Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 17 Dec 2019 12:07:11 -0800
Subject: [PATCH 2986/3084] Update wasmparser to 0.45.0 (#1295)

Adds many new operators and a few API changes.
---
 cranelift/wasm/Cargo.toml                 |   6 +-
 cranelift/wasm/src/code_translator.rs     | 188 +++++++++++++---------
 cranelift/wasm/src/sections_translator.rs |  14 +-
 3 files changed, 117 insertions(+), 91 deletions(-)

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 57230b4882..a7476558cb 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -11,7 +11,7 @@ keywords = ["webassembly", "wasm"]
 edition = "2018"
 
 [dependencies]
-wasmparser = { version = "0.39.2", default-features = false }
+wasmparser = { version = "0.45.0", default-features = false }
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.51.0" }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.51.0", default-features = false }
@@ -26,8 +26,8 @@ target-lexicon = "0.9"
 
 [features]
 default = ["std", "basic-blocks"]
-std = ["cranelift-codegen/std", "cranelift-frontend/std", "wasmparser/std"]
-core = ["hashbrown", "cranelift-codegen/core", "cranelift-frontend/core", "wasmparser/core"]
+std = ["cranelift-codegen/std", "cranelift-frontend/std"]
+core = ["hashbrown", "cranelift-codegen/core", "cranelift-frontend/core"]
 enable-serde = ["serde"]
 
 # Temporary feature that enforces basic block semantics.
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index efd4d22561..e16b187cb5 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -65,19 +65,19 @@ pub fn translate_operator(
          *  `get_local` and `set_local` are treated as non-SSA variables and will completely
          *  disappear in the Cranelift Code
          ***********************************************************************************/
-        Operator::GetLocal { local_index } => {
+        Operator::LocalGet { local_index } => {
             let val = builder.use_var(Variable::with_u32(*local_index));
             state.push1(val);
             let label = ValueLabel::from_u32(*local_index);
             builder.set_val_label(val, label);
         }
-        Operator::SetLocal { local_index } => {
+        Operator::LocalSet { local_index } => {
             let val = state.pop1();
             builder.def_var(Variable::with_u32(*local_index), val);
             let label = ValueLabel::from_u32(*local_index);
             builder.set_val_label(val, label);
         }
-        Operator::TeeLocal { local_index } => {
+        Operator::LocalTee { local_index } => {
             let val = state.peek1();
             builder.def_var(Variable::with_u32(*local_index), val);
             let label = ValueLabel::from_u32(*local_index);
@@ -86,7 +86,7 @@ pub fn translate_operator(
         /********************************** Globals ****************************************
          *  `get_global` and `set_global` are handled by the environment.
          ***********************************************************************************/
-        Operator::GetGlobal { global_index } => {
+        Operator::GlobalGet { global_index } => {
             let val = match state.get_global(builder.func, *global_index, environ)? {
                 GlobalVariable::Const(val) => val,
                 GlobalVariable::Memory { gv, offset, ty } => {
@@ -97,7 +97,7 @@ pub fn translate_operator(
             };
             state.push1(val);
         }
-        Operator::SetGlobal { global_index } => {
+        Operator::GlobalSet { global_index } => {
             match state.get_global(builder.func, *global_index, environ)? {
                 GlobalVariable::Const(_) => panic!("global #{} is a constant", *global_index),
                 GlobalVariable::Memory { gv, offset, ty } => {
@@ -639,11 +639,11 @@ pub fn translate_operator(
             let arg = state.pop1();
             state.push1(builder.ins().popcnt(arg));
         }
-        Operator::I64ExtendSI32 => {
+        Operator::I64ExtendI32S => {
             let val = state.pop1();
             state.push1(builder.ins().sextend(I64, val));
         }
-        Operator::I64ExtendUI32 => {
+        Operator::I64ExtendI32U => {
             let val = state.pop1();
             state.push1(builder.ins().uextend(I64, val));
         }
@@ -679,19 +679,19 @@ pub fn translate_operator(
             let arg = state.pop1();
             state.push1(builder.ins().fneg(arg));
         }
-        Operator::F64ConvertUI64 | Operator::F64ConvertUI32 => {
+        Operator::F64ConvertI64U | Operator::F64ConvertI32U => {
             let val = state.pop1();
             state.push1(builder.ins().fcvt_from_uint(F64, val));
         }
-        Operator::F64ConvertSI64 | Operator::F64ConvertSI32 => {
+        Operator::F64ConvertI64S | Operator::F64ConvertI32S => {
             let val = state.pop1();
             state.push1(builder.ins().fcvt_from_sint(F64, val));
         }
-        Operator::F32ConvertSI64 | Operator::F32ConvertSI32 => {
+        Operator::F32ConvertI64S | Operator::F32ConvertI32S => {
             let val = state.pop1();
             state.push1(builder.ins().fcvt_from_sint(F32, val));
         }
-        Operator::F32ConvertUI64 | Operator::F32ConvertUI32 => {
+        Operator::F32ConvertI64U | Operator::F32ConvertI32U => {
             let val = state.pop1();
             state.push1(builder.ins().fcvt_from_uint(F32, val));
         }
@@ -703,35 +703,35 @@ pub fn translate_operator(
             let val = state.pop1();
             state.push1(builder.ins().fdemote(F32, val));
         }
-        Operator::I64TruncSF64 | Operator::I64TruncSF32 => {
+        Operator::I64TruncF64S | Operator::I64TruncF32S => {
             let val = state.pop1();
             state.push1(builder.ins().fcvt_to_sint(I64, val));
         }
-        Operator::I32TruncSF64 | Operator::I32TruncSF32 => {
+        Operator::I32TruncF64S | Operator::I32TruncF32S => {
             let val = state.pop1();
             state.push1(builder.ins().fcvt_to_sint(I32, val));
         }
-        Operator::I64TruncUF64 | Operator::I64TruncUF32 => {
+        Operator::I64TruncF64U | Operator::I64TruncF32U => {
             let val = state.pop1();
             state.push1(builder.ins().fcvt_to_uint(I64, val));
         }
-        Operator::I32TruncUF64 | Operator::I32TruncUF32 => {
+        Operator::I32TruncF64U | Operator::I32TruncF32U => {
             let val = state.pop1();
             state.push1(builder.ins().fcvt_to_uint(I32, val));
         }
-        Operator::I64TruncSSatF64 | Operator::I64TruncSSatF32 => {
+        Operator::I64TruncSatF64S | Operator::I64TruncSatF32S => {
             let val = state.pop1();
             state.push1(builder.ins().fcvt_to_sint_sat(I64, val));
         }
-        Operator::I32TruncSSatF64 | Operator::I32TruncSSatF32 => {
+        Operator::I32TruncSatF64S | Operator::I32TruncSatF32S => {
             let val = state.pop1();
             state.push1(builder.ins().fcvt_to_sint_sat(I32, val));
         }
-        Operator::I64TruncUSatF64 | Operator::I64TruncUSatF32 => {
+        Operator::I64TruncSatF64U | Operator::I64TruncSatF32U => {
             let val = state.pop1();
             state.push1(builder.ins().fcvt_to_uint_sat(I64, val));
         }
-        Operator::I32TruncUSatF64 | Operator::I32TruncUSatF32 => {
+        Operator::I32TruncSatF64U | Operator::I32TruncSatF32U => {
             let val = state.pop1();
             state.push1(builder.ins().fcvt_to_uint_sat(I32, val));
         }
@@ -918,9 +918,12 @@ pub fn translate_operator(
             let val = builder.ins().is_null(arg);
             state.push1(val);
         }
-        Operator::Wake { .. }
-        | Operator::I32Wait { .. }
-        | Operator::I64Wait { .. }
+        Operator::RefFunc { .. } => {
+            return Err(wasm_unsupported!("proposed ref operator {:?}", op))
+        }
+        Operator::AtomicNotify { .. }
+        | Operator::I32AtomicWait { .. }
+        | Operator::I64AtomicWait { .. }
         | Operator::I32AtomicLoad { .. }
         | Operator::I64AtomicLoad { .. }
         | Operator::I32AtomicLoad8U { .. }
@@ -937,54 +940,54 @@ pub fn translate_operator(
         | Operator::I64AtomicStore32 { .. }
         | Operator::I32AtomicRmwAdd { .. }
         | Operator::I64AtomicRmwAdd { .. }
-        | Operator::I32AtomicRmw8UAdd { .. }
-        | Operator::I32AtomicRmw16UAdd { .. }
-        | Operator::I64AtomicRmw8UAdd { .. }
-        | Operator::I64AtomicRmw16UAdd { .. }
-        | Operator::I64AtomicRmw32UAdd { .. }
+        | Operator::I32AtomicRmw8AddU { .. }
+        | Operator::I32AtomicRmw16AddU { .. }
+        | Operator::I64AtomicRmw8AddU { .. }
+        | Operator::I64AtomicRmw16AddU { .. }
+        | Operator::I64AtomicRmw32AddU { .. }
         | Operator::I32AtomicRmwSub { .. }
         | Operator::I64AtomicRmwSub { .. }
-        | Operator::I32AtomicRmw8USub { .. }
-        | Operator::I32AtomicRmw16USub { .. }
-        | Operator::I64AtomicRmw8USub { .. }
-        | Operator::I64AtomicRmw16USub { .. }
-        | Operator::I64AtomicRmw32USub { .. }
+        | Operator::I32AtomicRmw8SubU { .. }
+        | Operator::I32AtomicRmw16SubU { .. }
+        | Operator::I64AtomicRmw8SubU { .. }
+        | Operator::I64AtomicRmw16SubU { .. }
+        | Operator::I64AtomicRmw32SubU { .. }
         | Operator::I32AtomicRmwAnd { .. }
         | Operator::I64AtomicRmwAnd { .. }
-        | Operator::I32AtomicRmw8UAnd { .. }
-        | Operator::I32AtomicRmw16UAnd { .. }
-        | Operator::I64AtomicRmw8UAnd { .. }
-        | Operator::I64AtomicRmw16UAnd { .. }
-        | Operator::I64AtomicRmw32UAnd { .. }
+        | Operator::I32AtomicRmw8AndU { .. }
+        | Operator::I32AtomicRmw16AndU { .. }
+        | Operator::I64AtomicRmw8AndU { .. }
+        | Operator::I64AtomicRmw16AndU { .. }
+        | Operator::I64AtomicRmw32AndU { .. }
         | Operator::I32AtomicRmwOr { .. }
         | Operator::I64AtomicRmwOr { .. }
-        | Operator::I32AtomicRmw8UOr { .. }
-        | Operator::I32AtomicRmw16UOr { .. }
-        | Operator::I64AtomicRmw8UOr { .. }
-        | Operator::I64AtomicRmw16UOr { .. }
-        | Operator::I64AtomicRmw32UOr { .. }
+        | Operator::I32AtomicRmw8OrU { .. }
+        | Operator::I32AtomicRmw16OrU { .. }
+        | Operator::I64AtomicRmw8OrU { .. }
+        | Operator::I64AtomicRmw16OrU { .. }
+        | Operator::I64AtomicRmw32OrU { .. }
         | Operator::I32AtomicRmwXor { .. }
         | Operator::I64AtomicRmwXor { .. }
-        | Operator::I32AtomicRmw8UXor { .. }
-        | Operator::I32AtomicRmw16UXor { .. }
-        | Operator::I64AtomicRmw8UXor { .. }
-        | Operator::I64AtomicRmw16UXor { .. }
-        | Operator::I64AtomicRmw32UXor { .. }
+        | Operator::I32AtomicRmw8XorU { .. }
+        | Operator::I32AtomicRmw16XorU { .. }
+        | Operator::I64AtomicRmw8XorU { .. }
+        | Operator::I64AtomicRmw16XorU { .. }
+        | Operator::I64AtomicRmw32XorU { .. }
         | Operator::I32AtomicRmwXchg { .. }
         | Operator::I64AtomicRmwXchg { .. }
-        | Operator::I32AtomicRmw8UXchg { .. }
-        | Operator::I32AtomicRmw16UXchg { .. }
-        | Operator::I64AtomicRmw8UXchg { .. }
-        | Operator::I64AtomicRmw16UXchg { .. }
-        | Operator::I64AtomicRmw32UXchg { .. }
+        | Operator::I32AtomicRmw8XchgU { .. }
+        | Operator::I32AtomicRmw16XchgU { .. }
+        | Operator::I64AtomicRmw8XchgU { .. }
+        | Operator::I64AtomicRmw16XchgU { .. }
+        | Operator::I64AtomicRmw32XchgU { .. }
         | Operator::I32AtomicRmwCmpxchg { .. }
         | Operator::I64AtomicRmwCmpxchg { .. }
-        | Operator::I32AtomicRmw8UCmpxchg { .. }
-        | Operator::I32AtomicRmw16UCmpxchg { .. }
-        | Operator::I64AtomicRmw8UCmpxchg { .. }
-        | Operator::I64AtomicRmw16UCmpxchg { .. }
-        | Operator::I64AtomicRmw32UCmpxchg { .. }
-        | Operator::Fence { .. } => {
+        | Operator::I32AtomicRmw8CmpxchgU { .. }
+        | Operator::I32AtomicRmw16CmpxchgU { .. }
+        | Operator::I64AtomicRmw8CmpxchgU { .. }
+        | Operator::I64AtomicRmw16CmpxchgU { .. }
+        | Operator::I64AtomicRmw32CmpxchgU { .. }
+        | Operator::AtomicFence { .. } => {
             return Err(wasm_unsupported!("proposed thread operator {:?}", op));
         }
         Operator::MemoryCopy => {
@@ -1039,7 +1042,7 @@ pub fn translate_operator(
                 table,
             )?);
         }
-        Operator::TableCopy => {
+        Operator::TableCopy { .. } => {
             // The WebAssembly MVP only supports one table and wasmparser will
             // ensure that the table index specified is zero.
             let dst_table_index = 0;
@@ -1060,7 +1063,7 @@ pub fn translate_operator(
                 len,
             )?;
         }
-        Operator::TableInit { segment } => {
+        Operator::TableInit { segment, table: _ } => {
             // The WebAssembly MVP only supports one table and we assume it here.
             let table_index = 0;
             let table = state.get_table(builder.func, table_index, environ)?;
@@ -1077,6 +1080,9 @@ pub fn translate_operator(
                 len,
             )?;
         }
+        Operator::TableFill { .. } => {
+            return Err(wasm_unsupported!("proposed table operator {:?}", op));
+        }
         Operator::ElemDrop { segment } => {
             environ.translate_elem_drop(builder.cursor(), *segment)?;
         }
@@ -1330,20 +1336,42 @@ pub fn translate_operator(
         | Operator::I8x16ShrS
         | Operator::I8x16ShrU
         | Operator::I8x16Mul
+        | Operator::I64x2Mul
         | Operator::I64x2ShrS
-        | Operator::I32x4TruncSF32x4Sat
-        | Operator::I32x4TruncUF32x4Sat
-        | Operator::I64x2TruncSF64x2Sat
-        | Operator::I64x2TruncUF64x2Sat
-        | Operator::F32x4ConvertSI32x4
-        | Operator::F32x4ConvertUI32x4
-        | Operator::F64x2ConvertSI64x2
-        | Operator::F64x2ConvertUI64x2 { .. }
+        | Operator::I32x4TruncSatF32x4S
+        | Operator::I32x4TruncSatF32x4U
+        | Operator::I64x2TruncSatF64x2S
+        | Operator::I64x2TruncSatF64x2U
+        | Operator::F32x4ConvertI32x4S
+        | Operator::F32x4ConvertI32x4U
+        | Operator::F64x2ConvertI64x2S
+        | Operator::F64x2ConvertI64x2U { .. }
+        | Operator::I8x16NarrowI16x8S { .. }
+        | Operator::I8x16NarrowI16x8U { .. }
+        | Operator::I16x8NarrowI32x4S { .. }
+        | Operator::I16x8NarrowI32x4U { .. }
+        | Operator::I16x8WidenLowI8x16S { .. }
+        | Operator::I16x8WidenHighI8x16S { .. }
+        | Operator::I16x8WidenLowI8x16U { .. }
+        | Operator::I16x8WidenHighI8x16U { .. }
+        | Operator::I32x4WidenLowI16x8S { .. }
+        | Operator::I32x4WidenHighI16x8S { .. }
+        | Operator::I32x4WidenLowI16x8U { .. }
+        | Operator::I32x4WidenHighI16x8U { .. }
         | Operator::V8x16Swizzle
-        | Operator::I8x16LoadSplat { .. }
-        | Operator::I16x8LoadSplat { .. }
-        | Operator::I32x4LoadSplat { .. }
-        | Operator::I64x2LoadSplat { .. } => {
+        | Operator::V8x16LoadSplat { .. }
+        | Operator::V16x8LoadSplat { .. }
+        | Operator::V32x4LoadSplat { .. }
+        | Operator::V64x2LoadSplat { .. }
+        | Operator::I16x8Load8x8S { .. }
+        | Operator::I16x8Load8x8U { .. }
+        | Operator::I32x4Load16x4S { .. }
+        | Operator::I32x4Load16x4U { .. }
+        | Operator::I64x2Load32x2S { .. }
+        | Operator::I64x2Load32x2U { .. }
+        | Operator::I8x16RoundingAverageU { .. }
+        | Operator::I16x8RoundingAverageU { .. }
+        | Operator::V128AndNot { .. } => {
             return Err(wasm_unsupported!("proposed SIMD operator {:?}", op));
         }
     };
@@ -1727,8 +1755,8 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::I32x4Add
         | Operator::I32x4Sub
         | Operator::I32x4Mul
-        | Operator::F32x4ConvertSI32x4
-        | Operator::F32x4ConvertUI32x4 => I32X4,
+        | Operator::F32x4ConvertI32x4S
+        | Operator::F32x4ConvertI32x4U => I32X4,
 
         Operator::I64x2Splat
         | Operator::I64x2ExtractLane { .. }
@@ -1741,8 +1769,8 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::I64x2ShrU
         | Operator::I64x2Add
         | Operator::I64x2Sub
-        | Operator::F64x2ConvertSI64x2
-        | Operator::F64x2ConvertUI64x2 => I64X2,
+        | Operator::F64x2ConvertI64x2S
+        | Operator::F64x2ConvertI64x2U => I64X2,
 
         Operator::F32x4Splat
         | Operator::F32x4ExtractLane { .. }
@@ -1762,8 +1790,8 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::F32x4Div
         | Operator::F32x4Min
         | Operator::F32x4Max
-        | Operator::I32x4TruncSF32x4Sat
-        | Operator::I32x4TruncUF32x4Sat => F32X4,
+        | Operator::I32x4TruncSatF32x4S
+        | Operator::I32x4TruncSatF32x4U => F32X4,
 
         Operator::F64x2Splat
         | Operator::F64x2ExtractLane { .. }
@@ -1783,8 +1811,8 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::F64x2Div
         | Operator::F64x2Min
         | Operator::F64x2Max
-        | Operator::I64x2TruncSF64x2Sat
-        | Operator::I64x2TruncUF64x2Sat => F64X2,
+        | Operator::I64x2TruncSatF64x2S
+        | Operator::I64x2TruncSatF64x2U => F64X2,
 
         _ => unimplemented!(
             "Currently only the SIMD instructions are translated to their return type: {:?}",
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index ae980cfcfc..03444a547f 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -217,7 +217,7 @@ pub fn parse_global_section(
                 GlobalInit::V128Const(V128Imm::from(value.bytes().to_vec().as_slice()))
             }
             Operator::RefNull => GlobalInit::RefNullConst,
-            Operator::GetGlobal { global_index } => {
+            Operator::GlobalGet { global_index } => {
                 GlobalInit::GetGlobal(GlobalIndex::from_u32(global_index))
             }
             ref s => {
@@ -286,8 +286,9 @@ pub fn parse_element_section<'data>(
     environ.reserve_table_elements(elements.get_count())?;
 
     for entry in elements {
-        let Element { kind, items } = entry?;
+        let Element { kind } = entry?;
         if let ElementKind::Active {
+            items,
             table_index,
             init_expr,
         } = kind
@@ -295,7 +296,7 @@ pub fn parse_element_section<'data>(
             let mut init_expr_reader = init_expr.get_binary_reader();
             let (base, offset) = match init_expr_reader.read_operator()? {
                 Operator::I32Const { value } => (None, value as u32 as usize),
-                Operator::GetGlobal { global_index } => {
+                Operator::GlobalGet { global_index } => {
                     (Some(GlobalIndex::from_u32(global_index)), 0)
                 }
                 ref s => {
@@ -318,10 +319,7 @@ pub fn parse_element_section<'data>(
                 elems.into_boxed_slice(),
             )?
         } else {
-            return Err(wasm_unsupported!(
-                "unsupported passive elements section: {:?}",
-                kind
-            ));
+            return Err(wasm_unsupported!("unsupported passive elements section",));
         }
     }
     Ok(())
@@ -359,7 +357,7 @@ pub fn parse_data_section<'data>(
             let mut init_expr_reader = init_expr.get_binary_reader();
             let (base, offset) = match init_expr_reader.read_operator()? {
                 Operator::I32Const { value } => (None, value as u32 as usize),
-                Operator::GetGlobal { global_index } => {
+                Operator::GlobalGet { global_index } => {
                     (Some(GlobalIndex::from_u32(global_index)), 0)
                 }
                 ref s => {

From ac8a952a6bd0503144bda8c5fd0fa581f3da58a6 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 18 Dec 2019 12:07:33 +0100
Subject: [PATCH 2987/3084] Bump version to 0.52.0

---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  2 +-
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/test-all.sh               |  4 ++--
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 20 files changed, 70 insertions(+), 70 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index f33dec2024..7f78b3ddd2 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.51.0"
+version = "0.52.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.51.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.51.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.51.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.51.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.51.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.51.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.51.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.51.0" }
-cranelift-module = { path = "cranelift-module", version = "0.51.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.51.0" }
-cranelift-object = { path = "cranelift-object", version = "0.51.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.51.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.51.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.51.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.52.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.52.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.52.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.52.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.52.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.52.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.52.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.52.0" }
+cranelift-module = { path = "cranelift-module", version = "0.52.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.52.0" }
+cranelift-object = { path = "cranelift-object", version = "0.52.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.52.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.52.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.52.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 29f3426f10..463fc12363 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.51.0"
+version = "0.52.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.51.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.52.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 58e6abaae5..bf85cf02cf 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.51.0"
+version = "0.52.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.51.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.51.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.51.0" }
+cranelift-codegen-shared = { path = "./shared", version = "0.52.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.52.0" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.52.0" }
 hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.9"
 log = { version = "0.4.6", default-features = false }
@@ -29,7 +29,7 @@ byteorder = { version = "1.3.2", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.51.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.52.0" }
 
 [features]
 default = ["std", "basic-blocks"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index aefcc7d612..e804727ae1 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.51.0"
+version = "0.52.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.51.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.51.0" }
+cranelift-codegen-shared = { path = "../shared", version = "0.52.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.52.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index fe1bc5c0ef..6ffdef7abd 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.51.0"
+version = "0.52.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 324fcffb17..2027c02b1f 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.51.0"
+version = "0.52.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index ec5e399097..c7321c3e8c 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.51.0"
+version = "0.52.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.51.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.52.0" }
 faerie = "0.12.0"
 goblin = "0.1.0"
 failure = "0.1.2"
@@ -18,7 +18,7 @@ target-lexicon = "0.9"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.51.0"
+version = "0.52.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 0495218e81..89a63c1a97 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.51.0"
+version = "0.52.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.51.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.51.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.51.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.52.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.52.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.52.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index bdfab545a5..f8e4a0b66a 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.51.0"
+version = "0.52.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
 target-lexicon = "0.9"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 98c5ae7315..cb0f7cde78 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.51.0"
+version = "0.52.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.51.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.52.0" }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 thiserror = "1.0.4"
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index abc5e88922..5bc12aa0f3 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.51.0"
+version = "0.52.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
 target-lexicon = "0.9"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index fd79e74790..95f3aa195c 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.51.0"
+version = "0.52.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,13 +10,13 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.51.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.52.0" }
 object = { version = "0.16", default-features = false, features = ["write"] }
 target-lexicon = "0.9"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.51.0"
+version = "0.52.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index b912f4fb4e..6006c77453 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.51.0"
+version = "0.52.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.51.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.52.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 47bbd468d9..cb7200483d 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.51.0"
+version="0.52.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 64fec46428..cf489e19d3 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.51.0"
+version = "0.52.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0" }
 target-lexicon = "0.9"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 7a56da3549..289ba9f268 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.51.0"
+version = "0.52.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.51.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.52.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 552d6c7e48..613a116bee 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.51.0"
+version = "0.52.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.51.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.51.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.52.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.52.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -20,7 +20,7 @@ memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.51.0"
+version = "0.52.0"
 default-features = false
 features = ["std"]
 
@@ -32,9 +32,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.51.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.51.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.51.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.52.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.52.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.52.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
index fb9f6578d7..646119b4f0 100755
--- a/cranelift/test-all.sh
+++ b/cranelift/test-all.sh
@@ -82,8 +82,8 @@ else
     echo "nightly toolchain not found, some documentation links will not work"
 fi
 
-# Ensure fuzzer works by running it with a single input
-# Note LSAN is disabled due to https://github.com/google/sanitizers/issues/764
+# Ensure fuzzer works by running it with a single input.
+# Note LSAN is disabled due to https://github.com/google/sanitizers/issues/764.
 banner "cargo fuzz check"
 
 if ensure_installed cargo-fuzz nightly; then
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 431087d264..30a46c6e63 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.51.0"
+version = "0.52.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.51.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.52.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index a7476558cb..6c9b77ecb6 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.51.0"
+version = "0.52.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.45.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.51.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.51.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.51.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.52.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.52.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }

From b486289ab8e1a304cc585f7f94b3375e08ae4665 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 18 Dec 2019 12:50:12 -0800
Subject: [PATCH 2988/3084] Translate Wasm's V128AndNot to CLIF's band_not
 (#1297)

The CLIF is already implemented for x86 SIMD.
---
 cranelift/wasm/src/code_translator.rs | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index e16b187cb5..e155594093 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1189,6 +1189,10 @@ pub fn translate_operator(
             let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().imul(a, b))
         }
+        Operator::V128AndNot => {
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().band_not(a, b))
+        }
         Operator::V128Not => {
             let a = state.pop1();
             state.push1(builder.ins().bnot(a));
@@ -1370,8 +1374,7 @@ pub fn translate_operator(
         | Operator::I64x2Load32x2S { .. }
         | Operator::I64x2Load32x2U { .. }
         | Operator::I8x16RoundingAverageU { .. }
-        | Operator::I16x8RoundingAverageU { .. }
-        | Operator::V128AndNot { .. } => {
+        | Operator::I16x8RoundingAverageU { .. } => {
             return Err(wasm_unsupported!("proposed SIMD operator {:?}", op));
         }
     };

From cf9e762f165a355023880cd07f3c974c0807da3d Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Thu, 19 Dec 2019 15:49:34 -0700
Subject: [PATCH 2989/3084] Add a DynRex recipe type for x86, decreasing the
 number of recipes (#1298)

This patch adds a third mode for templates: REX inference is requestable
at template instantiation time. This reduces the number of recipes
by removing rex()/nonrex() redundancy for many instructions.
---
 cranelift/codegen/meta/src/cdsl/recipes.rs    |   2 +-
 .../codegen/meta/src/isa/x86/encodings.rs     |  59 +-
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 887 ++++++++++--------
 .../shared/src/isa/x86/encoding_bits.rs       |  18 +
 cranelift/codegen/src/isa/x86/binemit.rs      |  53 +-
 cranelift/codegen/src/isa/x86/enc_tables.rs   | 162 +++-
 .../filetests/isa/x86/legalize-br-icmp.clif   |   4 +-
 .../filetests/isa/x86/relax_branch.clif       | 122 +--
 .../isa/x86/shrink-multiple-uses.clif         |   2 +-
 .../filetests/filetests/postopt/basic.clif    |  18 +-
 .../filetests/postopt/complex_memory_ops.clif |   8 +-
 .../filetests/regalloc/coloring-227.clif      |  38 +-
 .../filetests/filetests/verifier/flags.clif   |  16 +-
 13 files changed, 875 insertions(+), 514 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/recipes.rs b/cranelift/codegen/meta/src/cdsl/recipes.rs
index 9bb966f605..dfe4cd67a5 100644
--- a/cranelift/codegen/meta/src/cdsl/recipes.rs
+++ b/cranelift/codegen/meta/src/cdsl/recipes.rs
@@ -172,7 +172,7 @@ pub(crate) struct EncodingRecipeBuilder {
     pub base_size: u64,
     pub operands_in: Option>,
     pub operands_out: Option>,
-    compute_size: Option<&'static str>,
+    pub compute_size: Option<&'static str>,
     pub branch_range: Option,
     pub emit: Option,
     clobbers_flags: Option,
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 26b5334c26..4fe2232508 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -140,32 +140,59 @@ impl PerCpuModeEncodings {
         self.enc64.push(encoding);
     }
 
+    /// Adds I32/I64 encodings as appropriate for a typed instruction.
+    /// The REX prefix is always inferred at runtime.
+    ///
     /// Add encodings for `inst.i32` to X86_32.
-    /// Add encodings for `inst.i32` to X86_64 with and without REX.
+    /// Add encodings for `inst.i32` to X86_64 with optional, inferred REX.
     /// Add encodings for `inst.i64` to X86_64 with a REX.W prefix.
     fn enc_i32_i64(&mut self, inst: impl Into, template: Template) {
         let inst: InstSpec = inst.into();
+
+        // I32 on x86: no REX prefix.
+        self.enc32(inst.bind(I32), template.infer_rex());
+
+        // I32 on x86_64: REX.W unset; REX.RXB determined at runtime from registers.
+        self.enc64(inst.bind(I32), template.infer_rex());
+
+        // I64 on x86_64: REX.W set; REX.RXB determined at runtime from registers.
+        self.enc64(inst.bind(I64), template.infer_rex().w());
+    }
+
+    /// Adds I32/I64 encodings as appropriate for a typed instruction.
+    /// All variants of REX prefix are explicitly emitted, not inferred.
+    ///
+    /// Add encodings for `inst.i32` to X86_32.
+    /// Add encodings for `inst.i32` to X86_64 with and without REX.
+    /// Add encodings for `inst.i64` to X86_64 with and without REX.
+    fn enc_i32_i64_explicit_rex(&mut self, inst: impl Into, template: Template) {
+        let inst: InstSpec = inst.into();
         self.enc32(inst.bind(I32), template.nonrex());
 
-        // REX-less encoding must come after REX encoding so we don't use it by default. Otherwise
-        // reg-alloc would never use r8 and up.
+        // REX-less encoding must come after REX encoding so we don't use it by default.
+        // Otherwise reg-alloc would never use r8 and up.
         self.enc64(inst.bind(I32), template.rex());
         self.enc64(inst.bind(I32), template.nonrex());
         self.enc64(inst.bind(I64), template.rex().w());
     }
 
-    /// Add encodings for `inst.b32` to X86_32.
-    /// Add encodings for `inst.b32` to X86_64 with and without REX.
-    /// Add encodings for `inst.b64` to X86_64 with a REX.W prefix.
+    /// Adds B32/B64 encodings as appropriate for a typed instruction.
+    /// The REX prefix is always inferred at runtime.
+    ///
+    /// Adds encoding for `inst.b32` to X86_32.
+    /// Adds encoding for `inst.b32` to X86_64 with optional, inferred REX.
+    /// Adds encoding for `inst.b64` to X86_64 with a REX.W prefix.
     fn enc_b32_b64(&mut self, inst: impl Into, template: Template) {
         let inst: InstSpec = inst.into();
-        self.enc32(inst.bind(B32), template.nonrex());
 
-        // REX-less encoding must come after REX encoding so we don't use it by default. Otherwise
-        // reg-alloc would never use r8 and up.
-        self.enc64(inst.bind(B32), template.rex());
-        self.enc64(inst.bind(B32), template.nonrex());
-        self.enc64(inst.bind(B64), template.rex().w());
+        // B32 on x86: no REX prefix.
+        self.enc32(inst.bind(B32), template.infer_rex());
+
+        // B32 on x86_64: REX.W unset; REX.RXB determined at runtime from registers.
+        self.enc64(inst.bind(B32), template.infer_rex());
+
+        // B64 on x86_64: REX.W set; REX.RXB determined at runtime from registers.
+        self.enc64(inst.bind(B64), template.infer_rex().w());
     }
 
     /// Add encodings for `inst.i32` to X86_32.
@@ -994,8 +1021,8 @@ pub(crate) fn define(
         e.enc_x86_64(istore8.bind(I64).bind(Any), recipe.opcodes(&MOV_BYTE_STORE));
     }
 
-    e.enc_i32_i64(spill, rec_spillSib32.opcodes(&MOV_STORE));
-    e.enc_i32_i64(regspill, rec_regspill32.opcodes(&MOV_STORE));
+    e.enc_i32_i64_explicit_rex(spill, rec_spillSib32.opcodes(&MOV_STORE));
+    e.enc_i32_i64_explicit_rex(regspill, rec_regspill32.opcodes(&MOV_STORE));
     e.enc_r32_r64_rex_only(spill, rec_spillSib32.opcodes(&MOV_STORE));
     e.enc_r32_r64_rex_only(regspill, rec_regspill32.opcodes(&MOV_STORE));
 
@@ -1020,8 +1047,8 @@ pub(crate) fn define(
         e.enc_i32_i64_ld_st(sload8, true, recipe.opcodes(&MOVSX_BYTE));
     }
 
-    e.enc_i32_i64(fill, rec_fillSib32.opcodes(&MOV_LOAD));
-    e.enc_i32_i64(regfill, rec_regfill32.opcodes(&MOV_LOAD));
+    e.enc_i32_i64_explicit_rex(fill, rec_fillSib32.opcodes(&MOV_LOAD));
+    e.enc_i32_i64_explicit_rex(regfill, rec_regfill32.opcodes(&MOV_LOAD));
     e.enc_r32_r64_rex_only(fill, rec_fillSib32.opcodes(&MOV_LOAD));
     e.enc_r32_r64_rex_only(regfill, rec_regfill32.opcodes(&MOV_LOAD));
 
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 9451460482..521248082b 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -61,7 +61,7 @@ impl<'builder> RecipeGroup<'builder> {
         self.templates
             .iter()
             .find(|recipe| recipe.name() == name)
-            .unwrap_or_else(|| panic!("unknown tail recipe name: {}. Try recipe?", name))
+            .unwrap_or_else(|| panic!("unknown template name: {}. Try recipe?", name))
     }
 }
 
@@ -132,6 +132,33 @@ fn replace_nonrex_constraints(
         .collect()
 }
 
+/// Specifies how the REX prefix is emitted by a Recipe.
+#[derive(Copy, Clone, PartialEq)]
+pub enum RexRecipeKind {
+    /// The REX emission behavior is not hardcoded for the Recipe
+    /// and may be overridden when using the Template.
+    Unspecified,
+
+    /// The Recipe must hardcode the non-emission of the REX prefix.
+    NeverEmitRex,
+
+    /// The Recipe must hardcode the emission of the REX prefix.
+    AlwaysEmitRex,
+
+    /// The Recipe should infer the emission of the REX.RXB bits from registers,
+    /// and the REX.W bit from the EncodingBits.
+    ///
+    /// Because such a Recipe has a non-constant instruction size, it must have
+    /// a special `compute_size` handler for the inferrable-REX case.
+    InferRex,
+}
+
+impl Default for RexRecipeKind {
+    fn default() -> Self {
+        Self::Unspecified
+    }
+}
+
 /// Previously called a TailRecipe in the Python meta language, this allows to create multiple
 /// variants of a single base EncodingRecipe (rex prefix, specialized w/rrr bits, different
 /// opcodes). It serves as a prototype of an EncodingRecipe, which is then used when actually creating
@@ -145,16 +172,17 @@ pub(crate) struct Template<'builder> {
     /// The recipe template, which is to be specialized (by copy).
     recipe: EncodingRecipeBuilder,
 
-    /// Does this recipe requires a REX prefix?
-    requires_prefix: bool,
+    /// How is the REX prefix emitted?
+    rex_kind: RexRecipeKind,
+
+    /// Function for `compute_size()` when REX is inferrable.
+    inferred_rex_compute_size: Option<&'static str>,
 
     /// Other recipe to use when REX-prefixed.
     when_prefixed: Option>>,
 
-    // Specialized parameters.
-    /// Should we include the REX prefix?
-    rex: bool,
-    /// Value of the W bit (0 or 1).
+    // Parameters passed in the EncodingBits.
+    /// Value of the W bit (0 or 1), stored in the EncodingBits.
     w_bit: u16,
     /// Value of the RRR bits (between 0 and 0b111).
     rrr_bits: u16,
@@ -167,9 +195,9 @@ impl<'builder> Template<'builder> {
         Self {
             regs,
             recipe,
-            requires_prefix: false,
+            rex_kind: RexRecipeKind::default(),
+            inferred_rex_compute_size: None,
             when_prefixed: None,
-            rex: false,
             w_bit: 0,
             rrr_bits: 0,
             op_bytes: &opcodes::EMPTY,
@@ -179,9 +207,15 @@ impl<'builder> Template<'builder> {
     fn name(&self) -> &str {
         &self.recipe.name
     }
-    fn requires_prefix(self, value: bool) -> Self {
+    fn rex_kind(self, kind: RexRecipeKind) -> Self {
         Self {
-            requires_prefix: value,
+            rex_kind: kind,
+            ..self
+        }
+    }
+    fn inferred_rex_compute_size(self, function: &'static str) -> Self {
+        Self {
+            inferred_rex_compute_size: Some(function),
             ..self
         }
     }
@@ -212,12 +246,19 @@ impl<'builder> Template<'builder> {
         copy
     }
     pub fn nonrex(&self) -> Self {
-        assert!(!self.requires_prefix, "Tail recipe requires REX prefix.");
+        assert!(
+            self.rex_kind != RexRecipeKind::AlwaysEmitRex,
+            "Template requires REX prefix."
+        );
         let mut copy = self.clone();
-        copy.rex = false;
+        copy.rex_kind = RexRecipeKind::NeverEmitRex;
         copy
     }
     pub fn rex(&self) -> Self {
+        assert!(
+            self.rex_kind != RexRecipeKind::NeverEmitRex,
+            "Template requires no REX prefix."
+        );
         if let Some(prefixed) = &self.when_prefixed {
             let mut ret = prefixed.rex();
             // Forward specialized parameters.
@@ -227,36 +268,62 @@ impl<'builder> Template<'builder> {
             return ret;
         }
         let mut copy = self.clone();
-        copy.rex = true;
+        copy.rex_kind = RexRecipeKind::AlwaysEmitRex;
+        copy
+    }
+    pub fn infer_rex(&self) -> Self {
+        assert!(
+            self.rex_kind != RexRecipeKind::NeverEmitRex,
+            "Template requires no REX prefix."
+        );
+        assert!(
+            self.when_prefixed.is_none(),
+            "infer_rex used with when_prefixed()."
+        );
+        let mut copy = self.clone();
+        copy.rex_kind = RexRecipeKind::InferRex;
         copy
     }
 
     pub fn build(mut self) -> (EncodingRecipe, u16) {
-        let (name, bits) = decode_opcodes(&self.op_bytes, self.rrr_bits, self.w_bit);
+        let (opcode, bits) = decode_opcodes(&self.op_bytes, self.rrr_bits, self.w_bit);
 
-        let (name, rex_prefix_size) = if self.rex {
-            ("Rex".to_string() + name, 1)
-        } else {
-            (name.into(), 0)
+        let (recipe_name, rex_prefix_size) = match self.rex_kind {
+            RexRecipeKind::Unspecified | RexRecipeKind::NeverEmitRex => {
+                // Ensure the operands are limited to non-REX constraints.
+                let operands_in = self.recipe.operands_in.unwrap_or_default();
+                self.recipe.operands_in = Some(replace_nonrex_constraints(self.regs, operands_in));
+                let operands_out = self.recipe.operands_out.unwrap_or_default();
+                self.recipe.operands_out =
+                    Some(replace_nonrex_constraints(self.regs, operands_out));
+
+                (opcode.into(), 0)
+            }
+            RexRecipeKind::AlwaysEmitRex => ("Rex".to_string() + opcode, 1),
+            RexRecipeKind::InferRex => {
+                // Hook up the right function for inferred compute_size().
+                assert!(
+                    self.inferred_rex_compute_size.is_some(),
+                    "InferRex recipe '{}' needs an inferred_rex_compute_size function.",
+                    &self.recipe.name
+                );
+                self.recipe.compute_size = self.inferred_rex_compute_size;
+
+                ("DynRex".to_string() + opcode, 0)
+            }
         };
 
         let size_addendum = self.op_bytes.len() as u64 + rex_prefix_size;
         self.recipe.base_size += size_addendum;
 
         // Branch ranges are relative to the end of the instruction.
+        // For InferRex, the range should be the minimum, assuming no REX.
         if let Some(range) = self.recipe.branch_range.as_mut() {
             range.inst_size += size_addendum;
         }
 
-        self.recipe.emit = replace_put_op(self.recipe.emit, &name);
-        self.recipe.name = name + &self.recipe.name;
-
-        if !self.rex {
-            let operands_in = self.recipe.operands_in.unwrap_or_default();
-            self.recipe.operands_in = Some(replace_nonrex_constraints(self.regs, operands_in));
-            let operands_out = self.recipe.operands_out.unwrap_or_default();
-            self.recipe.operands_out = Some(replace_nonrex_constraints(self.regs, operands_out));
-        }
+        self.recipe.emit = replace_put_op(self.recipe.emit, &recipe_name);
+        self.recipe.name = recipe_name + &self.recipe.name;
 
         (self.recipe.build(), bits)
     }
@@ -438,29 +505,37 @@ pub(crate) fn define<'shared>(
     );
 
     // XX /r
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rr", &formats.binary, 1)
-            .operands_in(vec![gpr, gpr])
-            .operands_out(vec![0])
-            .emit(
-                r#"
-                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
-                    modrm_rr(in_reg0, in_reg1, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("rr", &formats.binary, 1)
+                .operands_in(vec![gpr, gpr])
+                .operands_out(vec![0])
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
+                        modrm_rr(in_reg0, in_reg1, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0_inreg1"),
     );
 
     // XX /r with operands swapped. (RM form).
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rrx", &formats.binary, 1)
-            .operands_in(vec![gpr, gpr])
-            .operands_out(vec![0])
-            .emit(
-                r#"
-                    {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
-                    modrm_rr(in_reg1, in_reg0, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("rrx", &formats.binary, 1)
+                .operands_in(vec![gpr, gpr])
+                .operands_out(vec![0])
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex2(in_reg1, in_reg0), sink);
+                        modrm_rr(in_reg1, in_reg0, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0_inreg1"),
     );
 
     // XX /r with FPR ins and outs. A form.
@@ -513,31 +588,39 @@ pub(crate) fn define<'shared>(
     }
 
     // XX /n for a unary operation with extension bits.
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("ur", &formats.unary, 1)
-            .operands_in(vec![gpr])
-            .operands_out(vec![0])
-            .emit(
-                r#"
-                    {{PUT_OP}}(bits, rex1(in_reg0), sink);
-                    modrm_r_bits(in_reg0, bits, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("ur", &formats.unary, 1)
+                .operands_in(vec![gpr])
+                .operands_out(vec![0])
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                        modrm_r_bits(in_reg0, bits, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0"),
     );
 
     // XX /r, but for a unary operator with separate input/output register, like
     // copies. MR form, preserving flags.
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("umr", &formats.unary, 1)
-            .operands_in(vec![gpr])
-            .operands_out(vec![gpr])
-            .clobbers_flags(false)
-            .emit(
-                r#"
-                    {{PUT_OP}}(bits, rex2(out_reg0, in_reg0), sink);
-                    modrm_rr(out_reg0, in_reg0, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("umr", &formats.unary, 1)
+                .operands_in(vec![gpr])
+                .operands_out(vec![gpr])
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex2(out_reg0, in_reg0), sink);
+                        modrm_rr(out_reg0, in_reg0, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0_outreg0"),
     );
 
     // Same as umr, but with FPR -> GPR registers.
@@ -643,17 +726,21 @@ pub(crate) fn define<'shared>(
     );
 
     // XX /r, RM form, GPR -> FPR.
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("frurm", &formats.unary, 1)
-            .operands_in(vec![gpr])
-            .operands_out(vec![fpr])
-            .clobbers_flags(false)
-            .emit(
-                r#"
-                    {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
-                    modrm_rr(in_reg0, out_reg0, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("frurm", &formats.unary, 1)
+                .operands_in(vec![gpr])
+                .operands_out(vec![fpr])
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                        modrm_rr(in_reg0, out_reg0, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0_outreg0"),
     );
 
     // XX /r, RM form, FPR -> GPR.
@@ -734,62 +821,74 @@ pub(crate) fn define<'shared>(
     );
 
     // XX /n for division: inputs in %rax, %rdx, r. Outputs in %rax, %rdx.
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("div", &formats.ternary, 1)
-            .operands_in(vec![
-                OperandConstraint::FixedReg(reg_rax),
-                OperandConstraint::FixedReg(reg_rdx),
-                OperandConstraint::RegClass(gpr),
-            ])
-            .operands_out(vec![reg_rax, reg_rdx])
-            .emit(
-                r#"
-                    sink.trap(TrapCode::IntegerDivisionByZero, func.srclocs[inst]);
-                    {{PUT_OP}}(bits, rex1(in_reg2), sink);
-                    modrm_r_bits(in_reg2, bits, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("div", &formats.ternary, 1)
+                .operands_in(vec![
+                    OperandConstraint::FixedReg(reg_rax),
+                    OperandConstraint::FixedReg(reg_rdx),
+                    OperandConstraint::RegClass(gpr),
+                ])
+                .operands_out(vec![reg_rax, reg_rdx])
+                .emit(
+                    r#"
+                        sink.trap(TrapCode::IntegerDivisionByZero, func.srclocs[inst]);
+                        {{PUT_OP}}(bits, rex1(in_reg2), sink);
+                        modrm_r_bits(in_reg2, bits, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg2"),
     );
 
     // XX /n for {s,u}mulx: inputs in %rax, r. Outputs in %rdx(hi):%rax(lo)
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("mulx", &formats.binary, 1)
-            .operands_in(vec![
-                OperandConstraint::FixedReg(reg_rax),
-                OperandConstraint::RegClass(gpr),
-            ])
-            .operands_out(vec![
-                OperandConstraint::FixedReg(reg_rax),
-                OperandConstraint::FixedReg(reg_rdx),
-            ])
-            .emit(
-                r#"
-                    {{PUT_OP}}(bits, rex1(in_reg1), sink);
-                    modrm_r_bits(in_reg1, bits, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("mulx", &formats.binary, 1)
+                .operands_in(vec![
+                    OperandConstraint::FixedReg(reg_rax),
+                    OperandConstraint::RegClass(gpr),
+                ])
+                .operands_out(vec![
+                    OperandConstraint::FixedReg(reg_rax),
+                    OperandConstraint::FixedReg(reg_rdx),
+                ])
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex1(in_reg1), sink);
+                        modrm_r_bits(in_reg1, bits, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg1"),
     );
 
     // XX /n ib with 8-bit immediate sign-extended.
     {
-        recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("r_ib", &formats.binary_imm, 2)
-                .operands_in(vec![gpr])
-                .operands_out(vec![0])
-                .inst_predicate(InstructionPredicate::new_is_signed_int(
-                    &*formats.binary_imm,
-                    "imm",
-                    8,
-                    0,
-                ))
-                .emit(
-                    r#"
-                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
-                        modrm_r_bits(in_reg0, bits, sink);
-                        let imm: i64 = imm.into();
-                        sink.put1(imm as u8);
-                    "#,
-                ),
+        recipes.add_template(
+            Template::new(
+                EncodingRecipeBuilder::new("r_ib", &formats.binary_imm, 2)
+                    .operands_in(vec![gpr])
+                    .operands_out(vec![0])
+                    .inst_predicate(InstructionPredicate::new_is_signed_int(
+                        &*formats.binary_imm,
+                        "imm",
+                        8,
+                        0,
+                    ))
+                    .emit(
+                        r#"
+                            {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                            modrm_r_bits(in_reg0, bits, sink);
+                            let imm: i64 = imm.into();
+                            sink.put1(imm as u8);
+                        "#,
+                    ),
+                regs,
+            )
+            .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0"),
         );
 
         recipes.add_template_recipe(
@@ -813,24 +912,28 @@ pub(crate) fn define<'shared>(
         );
 
         // XX /n id with 32-bit immediate sign-extended.
-        recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("r_id", &formats.binary_imm, 5)
-                .operands_in(vec![gpr])
-                .operands_out(vec![0])
-                .inst_predicate(InstructionPredicate::new_is_signed_int(
-                    &*formats.binary_imm,
-                    "imm",
-                    32,
-                    0,
-                ))
-                .emit(
-                    r#"
-                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
-                        modrm_r_bits(in_reg0, bits, sink);
-                        let imm: i64 = imm.into();
-                        sink.put4(imm as u32);
-                    "#,
-                ),
+        recipes.add_template(
+            Template::new(
+                EncodingRecipeBuilder::new("r_id", &formats.binary_imm, 5)
+                    .operands_in(vec![gpr])
+                    .operands_out(vec![0])
+                    .inst_predicate(InstructionPredicate::new_is_signed_int(
+                        &*formats.binary_imm,
+                        "imm",
+                        32,
+                        0,
+                    ))
+                    .emit(
+                        r#"
+                            {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                            modrm_r_bits(in_reg0, bits, sink);
+                            let imm: i64 = imm.into();
+                            sink.put4(imm as u32);
+                        "#,
+                    ),
+                regs,
+            )
+            .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0"),
         );
     }
 
@@ -1326,7 +1429,7 @@ pub(crate) fn define<'shared>(
                 .operands_in(vec![gpr, gpr])
                 .inst_predicate(has_no_offset.clone())
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_sib_or_offset_for_in_reg_1")
+                .compute_size("size_plus_maybe_sib_or_offset_for_inreg_1")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -1354,7 +1457,7 @@ pub(crate) fn define<'shared>(
                     .operands_in(vec![abcd, gpr])
                     .inst_predicate(has_no_offset.clone())
                     .clobbers_flags(false)
-                    .compute_size("size_plus_maybe_sib_or_offset_for_in_reg_1")
+                    .compute_size("size_plus_maybe_sib_or_offset_for_inreg_1")
                     .emit(
                         r#"
                         if !flags.notrap() {
@@ -1383,7 +1486,7 @@ pub(crate) fn define<'shared>(
                 .operands_in(vec![fpr, gpr])
                 .inst_predicate(has_no_offset)
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_sib_or_offset_for_in_reg_1")
+                .compute_size("size_plus_maybe_sib_or_offset_for_inreg_1")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -1412,7 +1515,7 @@ pub(crate) fn define<'shared>(
                 .operands_in(vec![gpr, gpr])
                 .inst_predicate(has_small_offset.clone())
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_sib_for_in_reg_1")
+                .compute_size("size_plus_maybe_sib_for_inreg_1")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -1439,7 +1542,7 @@ pub(crate) fn define<'shared>(
                     .operands_in(vec![abcd, gpr])
                     .inst_predicate(has_small_offset.clone())
                     .clobbers_flags(false)
-                    .compute_size("size_plus_maybe_sib_for_in_reg_1")
+                    .compute_size("size_plus_maybe_sib_for_inreg_1")
                     .emit(
                         r#"
                         if !flags.notrap() {
@@ -1467,7 +1570,7 @@ pub(crate) fn define<'shared>(
                 .operands_in(vec![fpr, gpr])
                 .inst_predicate(has_small_offset)
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_sib_for_in_reg_1")
+                .compute_size("size_plus_maybe_sib_for_inreg_1")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -1491,7 +1594,7 @@ pub(crate) fn define<'shared>(
             EncodingRecipeBuilder::new("stDisp32", &formats.store, 5)
                 .operands_in(vec![gpr, gpr])
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_sib_for_in_reg_1")
+                .compute_size("size_plus_maybe_sib_for_inreg_1")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -1517,7 +1620,7 @@ pub(crate) fn define<'shared>(
                 EncodingRecipeBuilder::new("stDisp32_abcd", &formats.store, 5)
                     .operands_in(vec![abcd, gpr])
                     .clobbers_flags(false)
-                    .compute_size("size_plus_maybe_sib_for_in_reg_1")
+                    .compute_size("size_plus_maybe_sib_for_inreg_1")
                     .emit(
                         r#"
                         if !flags.notrap() {
@@ -1544,7 +1647,7 @@ pub(crate) fn define<'shared>(
             EncodingRecipeBuilder::new("fstDisp32", &formats.store, 5)
                 .operands_in(vec![fpr, gpr])
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_sib_for_in_reg_1")
+                .compute_size("size_plus_maybe_sib_for_inreg_1")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -1577,7 +1680,7 @@ pub(crate) fn define<'shared>(
                 .operands_in(vec![gpr, gpr, gpr])
                 .inst_predicate(has_no_offset.clone())
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_offset_for_in_reg_1")
+                .compute_size("size_plus_maybe_offset_for_inreg_1")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -1604,7 +1707,7 @@ pub(crate) fn define<'shared>(
                 .operands_in(vec![abcd, gpr, gpr])
                 .inst_predicate(has_no_offset.clone())
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_offset_for_in_reg_1")
+                .compute_size("size_plus_maybe_offset_for_inreg_1")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -1630,7 +1733,7 @@ pub(crate) fn define<'shared>(
                 .operands_in(vec![fpr, gpr, gpr])
                 .inst_predicate(has_no_offset)
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_offset_for_in_reg_1")
+                .compute_size("size_plus_maybe_offset_for_inreg_1")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -1867,7 +1970,7 @@ pub(crate) fn define<'shared>(
                 .operands_out(vec![gpr])
                 .inst_predicate(has_no_offset.clone())
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_sib_or_offset_for_in_reg_0")
+                .compute_size("size_plus_maybe_sib_or_offset_for_inreg_0")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -1894,7 +1997,7 @@ pub(crate) fn define<'shared>(
                 .operands_out(vec![fpr])
                 .inst_predicate(has_no_offset)
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_sib_or_offset_for_in_reg_0")
+                .compute_size("size_plus_maybe_sib_or_offset_for_inreg_0")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -1924,7 +2027,7 @@ pub(crate) fn define<'shared>(
                 .operands_out(vec![gpr])
                 .inst_predicate(has_small_offset.clone())
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_sib_for_in_reg_0")
+                .compute_size("size_plus_maybe_sib_for_inreg_0")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -1950,7 +2053,7 @@ pub(crate) fn define<'shared>(
                 .operands_out(vec![fpr])
                 .inst_predicate(has_small_offset)
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_sib_for_in_reg_0")
+                .compute_size("size_plus_maybe_sib_for_inreg_0")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -1979,7 +2082,7 @@ pub(crate) fn define<'shared>(
                 .operands_out(vec![gpr])
                 .inst_predicate(has_big_offset.clone())
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_sib_for_in_reg_0")
+                .compute_size("size_plus_maybe_sib_for_inreg_0")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -2005,7 +2108,7 @@ pub(crate) fn define<'shared>(
                 .operands_out(vec![fpr])
                 .inst_predicate(has_big_offset)
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_sib_for_in_reg_0")
+                .compute_size("size_plus_maybe_sib_for_inreg_0")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -2039,7 +2142,7 @@ pub(crate) fn define<'shared>(
                 .operands_out(vec![gpr])
                 .inst_predicate(has_no_offset.clone())
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_offset_for_in_reg_0")
+                .compute_size("size_plus_maybe_offset_for_inreg_0")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -2066,7 +2169,7 @@ pub(crate) fn define<'shared>(
                 .operands_out(vec![fpr])
                 .inst_predicate(has_no_offset)
                 .clobbers_flags(false)
-                .compute_size("size_plus_maybe_offset_for_in_reg_0")
+                .compute_size("size_plus_maybe_offset_for_inreg_0")
                 .emit(
                     r#"
                         if !flags.notrap() {
@@ -2396,7 +2499,7 @@ pub(crate) fn define<'shared>(
             .operands_out(vec![gpr])
             .clobbers_flags(false)
             .inst_predicate(valid_scale(&*formats.branch_table_entry))
-            .compute_size("size_plus_maybe_offset_for_in_reg_1")
+            .compute_size("size_plus_maybe_offset_for_inreg_1")
             .emit(
                 r#"
                     {{PUT_OP}}(bits, rex3(in_reg1, out_reg0, in_reg0), sink);
@@ -2475,7 +2578,7 @@ pub(crate) fn define<'shared>(
                 ),
             regs,
         )
-        .requires_prefix(true),
+        .rex_kind(RexRecipeKind::AlwaysEmitRex),
     );
 
     recipes.add_template(
@@ -2509,7 +2612,7 @@ pub(crate) fn define<'shared>(
                 ),
             regs,
         )
-        .requires_prefix(true),
+        .rex_kind(RexRecipeKind::AlwaysEmitRex),
     );
 
     recipes.add_template(
@@ -2532,110 +2635,134 @@ pub(crate) fn define<'shared>(
     // Conditional move (a.k.a integer select)
     // (maybe-REX.W) 0F 4x modrm(r,r)
     // 1 byte, modrm(r,r), is after the opcode
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("cmov", &formats.int_select, 1)
-            .operands_in(vec![
-                OperandConstraint::FixedReg(reg_rflags),
-                OperandConstraint::RegClass(gpr),
-                OperandConstraint::RegClass(gpr),
-            ])
-            .operands_out(vec![2])
-            .clobbers_flags(false)
-            .emit(
-                r#"
-                    {{PUT_OP}}(bits | icc2opc(cond), rex2(in_reg1, in_reg2), sink);
-                    modrm_rr(in_reg1, in_reg2, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("cmov", &formats.int_select, 1)
+                .operands_in(vec![
+                    OperandConstraint::FixedReg(reg_rflags),
+                    OperandConstraint::RegClass(gpr),
+                    OperandConstraint::RegClass(gpr),
+                ])
+                .operands_out(vec![2])
+                .clobbers_flags(false)
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits | icc2opc(cond), rex2(in_reg1, in_reg2), sink);
+                        modrm_rr(in_reg1, in_reg2, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_cmov"),
     );
 
     // Bit scan forwards and reverse
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("bsf_and_bsr", &formats.unary, 1)
-            .operands_in(vec![gpr])
-            .operands_out(vec![
-                OperandConstraint::RegClass(gpr),
-                OperandConstraint::FixedReg(reg_rflags),
-            ])
-            .emit(
-                r#"
-                    {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
-                    modrm_rr(in_reg0, out_reg0, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("bsf_and_bsr", &formats.unary, 1)
+                .operands_in(vec![gpr])
+                .operands_out(vec![
+                    OperandConstraint::RegClass(gpr),
+                    OperandConstraint::FixedReg(reg_rflags),
+                ])
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex2(in_reg0, out_reg0), sink);
+                        modrm_rr(in_reg0, out_reg0, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0_outreg0"),
     );
 
     // Arithematic with flag I/O.
 
     // XX /r, MR form. Add two GPR registers and set carry flag.
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rout", &formats.binary, 1)
-            .operands_in(vec![gpr, gpr])
-            .operands_out(vec![
-                OperandConstraint::TiedInput(0),
-                OperandConstraint::FixedReg(reg_rflags),
-            ])
-            .clobbers_flags(true)
-            .emit(
-                r#"
-                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
-                    modrm_rr(in_reg0, in_reg1, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("rout", &formats.binary, 1)
+                .operands_in(vec![gpr, gpr])
+                .operands_out(vec![
+                    OperandConstraint::TiedInput(0),
+                    OperandConstraint::FixedReg(reg_rflags),
+                ])
+                .clobbers_flags(true)
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
+                        modrm_rr(in_reg0, in_reg1, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0_inreg1"),
     );
 
     // XX /r, MR form. Add two GPR registers and get carry flag.
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rin", &formats.ternary, 1)
-            .operands_in(vec![
-                OperandConstraint::RegClass(gpr),
-                OperandConstraint::RegClass(gpr),
-                OperandConstraint::FixedReg(reg_rflags),
-            ])
-            .operands_out(vec![0])
-            .clobbers_flags(true)
-            .emit(
-                r#"
-                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
-                    modrm_rr(in_reg0, in_reg1, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("rin", &formats.ternary, 1)
+                .operands_in(vec![
+                    OperandConstraint::RegClass(gpr),
+                    OperandConstraint::RegClass(gpr),
+                    OperandConstraint::FixedReg(reg_rflags),
+                ])
+                .operands_out(vec![0])
+                .clobbers_flags(true)
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
+                        modrm_rr(in_reg0, in_reg1, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0_inreg1"),
     );
 
     // XX /r, MR form. Add two GPR registers with carry flag.
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rio", &formats.ternary, 1)
-            .operands_in(vec![
-                OperandConstraint::RegClass(gpr),
-                OperandConstraint::RegClass(gpr),
-                OperandConstraint::FixedReg(reg_rflags),
-            ])
-            .operands_out(vec![
-                OperandConstraint::TiedInput(0),
-                OperandConstraint::FixedReg(reg_rflags),
-            ])
-            .clobbers_flags(true)
-            .emit(
-                r#"
-                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
-                    modrm_rr(in_reg0, in_reg1, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("rio", &formats.ternary, 1)
+                .operands_in(vec![
+                    OperandConstraint::RegClass(gpr),
+                    OperandConstraint::RegClass(gpr),
+                    OperandConstraint::FixedReg(reg_rflags),
+                ])
+                .operands_out(vec![
+                    OperandConstraint::TiedInput(0),
+                    OperandConstraint::FixedReg(reg_rflags),
+                ])
+                .clobbers_flags(true)
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
+                        modrm_rr(in_reg0, in_reg1, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0_inreg1"),
     );
 
     // Compare and set flags.
 
     // XX /r, MR form. Compare two GPR registers and set flags.
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("rcmp", &formats.binary, 1)
-            .operands_in(vec![gpr, gpr])
-            .operands_out(vec![reg_rflags])
-            .emit(
-                r#"
-                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
-                    modrm_rr(in_reg0, in_reg1, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("rcmp", &formats.binary, 1)
+                .operands_in(vec![gpr, gpr])
+                .operands_out(vec![reg_rflags])
+                .emit(
+                    r#"
+                        {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
+                        modrm_rr(in_reg0, in_reg1, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0_inreg1"),
     );
 
     // Same as rcmp, but second operand is the stack pointer.
@@ -2669,38 +2796,46 @@ pub(crate) fn define<'shared>(
             InstructionPredicate::new_is_signed_int(&*formats.binary_imm, "imm", 8, 0);
 
         // XX /n, MI form with imm8.
-        recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("rcmp_ib", &formats.binary_imm, 2)
-                .operands_in(vec![gpr])
-                .operands_out(vec![reg_rflags])
-                .inst_predicate(has_small_offset)
-                .emit(
-                    r#"
-                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
-                        modrm_r_bits(in_reg0, bits, sink);
-                        let imm: i64 = imm.into();
-                        sink.put1(imm as u8);
-                    "#,
-                ),
+        recipes.add_template(
+            Template::new(
+                EncodingRecipeBuilder::new("rcmp_ib", &formats.binary_imm, 2)
+                    .operands_in(vec![gpr])
+                    .operands_out(vec![reg_rflags])
+                    .inst_predicate(has_small_offset)
+                    .emit(
+                        r#"
+                            {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                            modrm_r_bits(in_reg0, bits, sink);
+                            let imm: i64 = imm.into();
+                            sink.put1(imm as u8);
+                        "#,
+                    ),
+                regs,
+            )
+            .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0"),
         );
 
         let has_big_offset =
             InstructionPredicate::new_is_signed_int(&*formats.binary_imm, "imm", 32, 0);
 
         // XX /n, MI form with imm32.
-        recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("rcmp_id", &formats.binary_imm, 5)
-                .operands_in(vec![gpr])
-                .operands_out(vec![reg_rflags])
-                .inst_predicate(has_big_offset)
-                .emit(
-                    r#"
-                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
-                        modrm_r_bits(in_reg0, bits, sink);
-                        let imm: i64 = imm.into();
-                        sink.put4(imm as u32);
-                    "#,
-                ),
+        recipes.add_template(
+            Template::new(
+                EncodingRecipeBuilder::new("rcmp_id", &formats.binary_imm, 5)
+                    .operands_in(vec![gpr])
+                    .operands_out(vec![reg_rflags])
+                    .inst_predicate(has_big_offset)
+                    .emit(
+                        r#"
+                            {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                            modrm_r_bits(in_reg0, bits, sink);
+                            let imm: i64 = imm.into();
+                            sink.put4(imm as u32);
+                        "#,
+                    ),
+                regs,
+            )
+            .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0"),
         );
     }
 
@@ -2718,37 +2853,45 @@ pub(crate) fn define<'shared>(
     // Bits 0-7 are the Jcc opcode.
     // Bits 8-15 control the test instruction which always has opcode byte 0x85.
 
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("tjccb", &formats.branch, 1 + 2)
-            .operands_in(vec![gpr])
-            .branch_range((3, 8))
-            .emit(
-                r#"
-                    // test r, r.
-                    {{PUT_OP}}((bits & 0xff00) | 0x85, rex2(in_reg0, in_reg0), sink);
-                    modrm_rr(in_reg0, in_reg0, sink);
-                    // Jcc instruction.
-                    sink.put1(bits as u8);
-                    disp1(destination, func, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("tjccb", &formats.branch, 1 + 2)
+                .operands_in(vec![gpr])
+                .branch_range((3, 8))
+                .emit(
+                    r#"
+                        // test r, r.
+                        {{PUT_OP}}((bits & 0xff00) | 0x85, rex2(in_reg0, in_reg0), sink);
+                        modrm_rr(in_reg0, in_reg0, sink);
+                        // Jcc instruction.
+                        sink.put1(bits as u8);
+                        disp1(destination, func, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0"),
     );
 
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("tjccd", &formats.branch, 1 + 6)
-            .operands_in(vec![gpr])
-            .branch_range((7, 32))
-            .emit(
-                r#"
-                    // test r, r.
-                    {{PUT_OP}}((bits & 0xff00) | 0x85, rex2(in_reg0, in_reg0), sink);
-                    modrm_rr(in_reg0, in_reg0, sink);
-                    // Jcc instruction.
-                    sink.put1(0x0f);
-                    sink.put1(bits as u8);
-                    disp4(destination, func, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("tjccd", &formats.branch, 1 + 6)
+                .operands_in(vec![gpr])
+                .branch_range((7, 32))
+                .emit(
+                    r#"
+                        // test r, r.
+                        {{PUT_OP}}((bits & 0xff00) | 0x85, rex2(in_reg0, in_reg0), sink);
+                        modrm_rr(in_reg0, in_reg0, sink);
+                        // Jcc instruction.
+                        sink.put1(0x0f);
+                        sink.put1(bits as u8);
+                        disp4(destination, func, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0"),
     );
 
     // 8-bit test-and-branch.
@@ -2770,7 +2913,7 @@ pub(crate) fn define<'shared>(
                 ),
             regs,
         )
-        .requires_prefix(true),
+        .rex_kind(RexRecipeKind::AlwaysEmitRex),
     );
 
     recipes.add_template(
@@ -2811,7 +2954,7 @@ pub(crate) fn define<'shared>(
                 ),
             regs,
         )
-        .requires_prefix(true),
+        .rex_kind(RexRecipeKind::AlwaysEmitRex),
     );
 
     recipes.add_template(
@@ -2878,22 +3021,26 @@ pub(crate) fn define<'shared>(
     // instruction, so it is limited to the `ABCD` register class for booleans.
     // The omission of a `when_prefixed` alternative is deliberate here.
 
-    recipes.add_template_recipe(
-        EncodingRecipeBuilder::new("icscc", &formats.int_compare, 1 + 3)
-            .operands_in(vec![gpr, gpr])
-            .operands_out(vec![abcd])
-            .emit(
-                r#"
-                    // Comparison instruction.
-                    {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
-                    modrm_rr(in_reg0, in_reg1, sink);
-                    // `setCC` instruction, no REX.
-                    let setcc = 0x90 | icc2opc(cond);
-                    sink.put1(0x0f);
-                    sink.put1(setcc as u8);
-                    modrm_rr(out_reg0, 0, sink);
-                "#,
-            ),
+    recipes.add_template(
+        Template::new(
+            EncodingRecipeBuilder::new("icscc", &formats.int_compare, 1 + 3)
+                .operands_in(vec![gpr, gpr])
+                .operands_out(vec![abcd])
+                .emit(
+                    r#"
+                        // Comparison instruction.
+                        {{PUT_OP}}(bits, rex2(in_reg0, in_reg1), sink);
+                        modrm_rr(in_reg0, in_reg1, sink);
+                        // `setCC` instruction, no REX.
+                        let setcc = 0x90 | icc2opc(cond);
+                        sink.put1(0x0f);
+                        sink.put1(setcc as u8);
+                        modrm_rr(out_reg0, 0, sink);
+                    "#,
+                ),
+            regs,
+        )
+        .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0_inreg1"),
     );
 
     recipes.add_template_recipe(
@@ -2913,49 +3060,57 @@ pub(crate) fn define<'shared>(
         let is_small_imm =
             InstructionPredicate::new_is_signed_int(&*formats.int_compare_imm, "imm", 8, 0);
 
-        recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("icscc_ib", &formats.int_compare_imm, 2 + 3)
-                .operands_in(vec![gpr])
-                .operands_out(vec![abcd])
-                .inst_predicate(is_small_imm)
-                .emit(
-                    r#"
-                        // Comparison instruction.
-                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
-                        modrm_r_bits(in_reg0, bits, sink);
-                        let imm: i64 = imm.into();
-                        sink.put1(imm as u8);
-                        // `setCC` instruction, no REX.
-                        let setcc = 0x90 | icc2opc(cond);
-                        sink.put1(0x0f);
-                        sink.put1(setcc as u8);
-                        modrm_rr(out_reg0, 0, sink);
-                    "#,
-                ),
+        recipes.add_template(
+            Template::new(
+                EncodingRecipeBuilder::new("icscc_ib", &formats.int_compare_imm, 2 + 3)
+                    .operands_in(vec![gpr])
+                    .operands_out(vec![abcd])
+                    .inst_predicate(is_small_imm)
+                    .emit(
+                        r#"
+                            // Comparison instruction.
+                            {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                            modrm_r_bits(in_reg0, bits, sink);
+                            let imm: i64 = imm.into();
+                            sink.put1(imm as u8);
+                            // `setCC` instruction, no REX.
+                            let setcc = 0x90 | icc2opc(cond);
+                            sink.put1(0x0f);
+                            sink.put1(setcc as u8);
+                            modrm_rr(out_reg0, 0, sink);
+                        "#,
+                    ),
+                regs,
+            )
+            .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0"),
         );
 
         let is_big_imm =
             InstructionPredicate::new_is_signed_int(&*formats.int_compare_imm, "imm", 32, 0);
 
-        recipes.add_template_recipe(
-            EncodingRecipeBuilder::new("icscc_id", &formats.int_compare_imm, 5 + 3)
-                .operands_in(vec![gpr])
-                .operands_out(vec![abcd])
-                .inst_predicate(is_big_imm)
-                .emit(
-                    r#"
-                        // Comparison instruction.
-                        {{PUT_OP}}(bits, rex1(in_reg0), sink);
-                        modrm_r_bits(in_reg0, bits, sink);
-                        let imm: i64 = imm.into();
-                        sink.put4(imm as u32);
-                        // `setCC` instruction, no REX.
-                        let setcc = 0x90 | icc2opc(cond);
-                        sink.put1(0x0f);
-                        sink.put1(setcc as u8);
-                        modrm_rr(out_reg0, 0, sink);
-                    "#,
-                ),
+        recipes.add_template(
+            Template::new(
+                EncodingRecipeBuilder::new("icscc_id", &formats.int_compare_imm, 5 + 3)
+                    .operands_in(vec![gpr])
+                    .operands_out(vec![abcd])
+                    .inst_predicate(is_big_imm)
+                    .emit(
+                        r#"
+                            // Comparison instruction.
+                            {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                            modrm_r_bits(in_reg0, bits, sink);
+                            let imm: i64 = imm.into();
+                            sink.put4(imm as u32);
+                            // `setCC` instruction, no REX.
+                            let setcc = 0x90 | icc2opc(cond);
+                            sink.put1(0x0f);
+                            sink.put1(setcc as u8);
+                            modrm_rr(out_reg0, 0, sink);
+                        "#,
+                    ),
+                regs,
+            )
+            .inferred_rex_compute_size("size_with_inferred_rex_for_inreg0"),
         );
     }
 
diff --git a/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs b/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs
index 5195c24c5c..ba6418f7cf 100644
--- a/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs
+++ b/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs
@@ -57,6 +57,24 @@ impl EncodingBits {
         new
     }
 
+    /// Returns a copy of the EncodingBits with the RRR bits set.
+    #[inline]
+    pub fn with_rrr(self, rrr: u8) -> Self {
+        debug_assert_eq!(u8::from(self.rrr()), 0);
+        let mut enc = self.clone();
+        enc.write(RRR, rrr.into());
+        enc
+    }
+
+    /// Returns a copy of the EncodingBits with the REX.W bit set.
+    #[inline]
+    pub fn with_rex_w(self) -> Self {
+        debug_assert_eq!(self.rex_w(), 0);
+        let mut enc = self.clone();
+        enc.write(REX_W, 1);
+        enc
+    }
+
     /// Returns the raw bits.
     #[inline]
     pub fn bits(self) -> u16 {
diff --git a/cranelift/codegen/src/isa/x86/binemit.rs b/cranelift/codegen/src/isa/x86/binemit.rs
index 4fc074abe5..fa67e5efff 100644
--- a/cranelift/codegen/src/isa/x86/binemit.rs
+++ b/cranelift/codegen/src/isa/x86/binemit.rs
@@ -61,6 +61,12 @@ fn rex3(rm: RegUnit, reg: RegUnit, index: RegUnit) -> u8 {
     BASE_REX | b | (x << 1) | (r << 2)
 }
 
+/// Determines whether a REX prefix should be emitted.
+#[inline]
+fn needs_rex(bits: u16, rex: u8) -> bool {
+    rex != BASE_REX || u8::from(EncodingBits::from(bits).rex_w()) == 1
+}
+
 // Emit a REX prefix.
 //
 // The R, X, and B bits are computed from registers using the functions above. The W bit is
@@ -80,11 +86,20 @@ fn put_op1(bits: u16, rex: u8, sink: &mut CS) {
 
 // Emit a single-byte opcode with REX prefix.
 fn put_rexop1(bits: u16, rex: u8, sink: &mut CS) {
-    debug_assert_eq!(bits & 0x0f00, 0, "Invalid encoding bits for Op1*");
+    debug_assert_eq!(bits & 0x0f00, 0, "Invalid encoding bits for RexOp1*");
     rex_prefix(bits, rex, sink);
     sink.put1(bits as u8);
 }
 
+/// Emit a single-byte opcode with inferred REX prefix.
+fn put_dynrexop1(bits: u16, rex: u8, sink: &mut CS) {
+    debug_assert_eq!(bits & 0x0f00, 0, "Invalid encoding bits for DynRexOp1*");
+    if needs_rex(bits, rex) {
+        rex_prefix(bits, rex, sink);
+    }
+    sink.put1(bits as u8);
+}
+
 // Emit two-byte opcode: 0F XX
 fn put_op2(bits: u16, rex: u8, sink: &mut CS) {
     debug_assert_eq!(bits & 0x8f00, 0x0400, "Invalid encoding bits for Op2*");
@@ -101,6 +116,20 @@ fn put_rexop2(bits: u16, rex: u8, sink: &mut CS) {
     sink.put1(bits as u8);
 }
 
+/// Emit two-byte opcode: 0F XX with inferred REX prefix.
+fn put_dynrexop2(bits: u16, rex: u8, sink: &mut CS) {
+    debug_assert_eq!(
+        bits & 0x0f00,
+        0x0400,
+        "Invalid encoding bits for DynRexOp2*"
+    );
+    if needs_rex(bits, rex) {
+        rex_prefix(bits, rex, sink);
+    }
+    sink.put1(0x0f);
+    sink.put1(bits as u8);
+}
+
 // Emit single-byte opcode with mandatory prefix.
 fn put_mp1(bits: u16, rex: u8, sink: &mut CS) {
     debug_assert_eq!(bits & 0x8c00, 0, "Invalid encoding bits for Mp1*");
@@ -112,7 +141,7 @@ fn put_mp1(bits: u16, rex: u8, sink: &mut CS) {
 
 // Emit single-byte opcode with mandatory prefix and REX.
 fn put_rexmp1(bits: u16, rex: u8, sink: &mut CS) {
-    debug_assert_eq!(bits & 0x0c00, 0, "Invalid encoding bits for Mp1*");
+    debug_assert_eq!(bits & 0x0c00, 0, "Invalid encoding bits for RexMp1*");
     let enc = EncodingBits::from(bits);
     sink.put1(PREFIX[(enc.pp() - 1) as usize]);
     rex_prefix(bits, rex, sink);
@@ -131,7 +160,7 @@ fn put_mp2(bits: u16, rex: u8, sink: &mut CS) {
 
 // Emit two-byte opcode (0F XX) with mandatory prefix and REX.
 fn put_rexmp2(bits: u16, rex: u8, sink: &mut CS) {
-    debug_assert_eq!(bits & 0x0c00, 0x0400, "Invalid encoding bits for Mp2*");
+    debug_assert_eq!(bits & 0x0c00, 0x0400, "Invalid encoding bits for RexMp2*");
     let enc = EncodingBits::from(bits);
     sink.put1(PREFIX[(enc.pp() - 1) as usize]);
     rex_prefix(bits, rex, sink);
@@ -139,6 +168,22 @@ fn put_rexmp2(bits: u16, rex: u8, sink: &mut CS) {
     sink.put1(bits as u8);
 }
 
+/// Emit two-byte opcode (0F XX) with mandatory prefix and inferred REX.
+fn put_dynrexmp2(bits: u16, rex: u8, sink: &mut CS) {
+    debug_assert_eq!(
+        bits & 0x0c00,
+        0x0400,
+        "Invalid encoding bits for DynRexMp2*"
+    );
+    let enc = EncodingBits::from(bits);
+    sink.put1(PREFIX[(enc.pp() - 1) as usize]);
+    if needs_rex(bits, rex) {
+        rex_prefix(bits, rex, sink);
+    }
+    sink.put1(0x0f);
+    sink.put1(bits as u8);
+}
+
 // Emit three-byte opcode (0F 3[8A] XX) with mandatory prefix.
 fn put_mp3(bits: u16, rex: u8, sink: &mut CS) {
     debug_assert_eq!(bits & 0x8800, 0x0800, "Invalid encoding bits for Mp3*");
@@ -152,7 +197,7 @@ fn put_mp3(bits: u16, rex: u8, sink: &mut CS) {
 
 // Emit three-byte opcode (0F 3[8A] XX) with mandatory prefix and REX
 fn put_rexmp3(bits: u16, rex: u8, sink: &mut CS) {
-    debug_assert_eq!(bits & 0x0800, 0x0800, "Invalid encoding bits for Mp3*");
+    debug_assert_eq!(bits & 0x0800, 0x0800, "Invalid encoding bits for RexMp3*");
     let enc = EncodingBits::from(bits);
     sink.put1(PREFIX[(enc.pp() - 1) as usize]);
     rex_prefix(bits, rex, sink);
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index fd95a50a01..d45f1314aa 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -16,9 +16,20 @@ use crate::isa::{self, TargetIsa};
 use crate::predicates;
 use crate::regalloc::RegDiversions;
 
+use cranelift_codegen_shared::isa::x86::EncodingBits;
+
 include!(concat!(env!("OUT_DIR"), "/encoding-x86.rs"));
 include!(concat!(env!("OUT_DIR"), "/legalize-x86.rs"));
 
+/// Whether the REX prefix is needed for encoding extended registers (via REX.RXB).
+///
+/// Normal x86 instructions have only 3 bits for encoding a register.
+/// The REX prefix adds REX.R, REX,X, and REX.B bits, interpreted as fourth bits.
+pub fn is_extended_reg(reg: RegUnit) -> bool {
+    // Extended registers have the fourth bit set.
+    reg as u8 & 0b1000 != 0
+}
+
 pub fn needs_sib_byte(reg: RegUnit) -> bool {
     reg == RU::r12 as RegUnit || reg == RU::rsp as RegUnit
 }
@@ -29,74 +40,179 @@ pub fn needs_sib_byte_or_offset(reg: RegUnit) -> bool {
     needs_sib_byte(reg) || needs_offset(reg)
 }
 
-fn additional_size_if(
+fn test_input(
     op_index: usize,
     inst: Inst,
     divert: &RegDiversions,
     func: &Function,
     condition_func: fn(RegUnit) -> bool,
-) -> u8 {
-    let addr_reg = divert.reg(func.dfg.inst_args(inst)[op_index], &func.locations);
-    if condition_func(addr_reg) {
-        1
-    } else {
-        0
-    }
+) -> bool {
+    let in_reg = divert.reg(func.dfg.inst_args(inst)[op_index], &func.locations);
+    condition_func(in_reg)
 }
 
-fn size_plus_maybe_offset_for_in_reg_0(
-    sizing: &RecipeSizing,
-    _enc: Encoding,
+fn test_result(
+    result_index: usize,
     inst: Inst,
     divert: &RegDiversions,
     func: &Function,
-) -> u8 {
-    sizing.base_size + additional_size_if(0, inst, divert, func, needs_offset)
+    condition_func: fn(RegUnit) -> bool,
+) -> bool {
+    let out_reg = divert.reg(func.dfg.inst_results(inst)[result_index], &func.locations);
+    condition_func(out_reg)
 }
-fn size_plus_maybe_offset_for_in_reg_1(
+
+fn size_plus_maybe_offset_for_inreg_0(
     sizing: &RecipeSizing,
     _enc: Encoding,
     inst: Inst,
     divert: &RegDiversions,
     func: &Function,
 ) -> u8 {
-    sizing.base_size + additional_size_if(1, inst, divert, func, needs_offset)
+    let needs_offset = test_input(0, inst, divert, func, needs_offset);
+    sizing.base_size + if needs_offset { 1 } else { 0 }
 }
-fn size_plus_maybe_sib_for_in_reg_0(
+fn size_plus_maybe_offset_for_inreg_1(
     sizing: &RecipeSizing,
     _enc: Encoding,
     inst: Inst,
     divert: &RegDiversions,
     func: &Function,
 ) -> u8 {
-    sizing.base_size + additional_size_if(0, inst, divert, func, needs_sib_byte)
+    let needs_offset = test_input(1, inst, divert, func, needs_offset);
+    sizing.base_size + if needs_offset { 1 } else { 0 }
 }
-fn size_plus_maybe_sib_for_in_reg_1(
+fn size_plus_maybe_sib_for_inreg_0(
     sizing: &RecipeSizing,
     _enc: Encoding,
     inst: Inst,
     divert: &RegDiversions,
     func: &Function,
 ) -> u8 {
-    sizing.base_size + additional_size_if(1, inst, divert, func, needs_sib_byte)
+    let needs_sib = test_input(0, inst, divert, func, needs_sib_byte);
+    sizing.base_size + if needs_sib { 1 } else { 0 }
 }
-fn size_plus_maybe_sib_or_offset_for_in_reg_0(
+fn size_plus_maybe_sib_for_inreg_1(
     sizing: &RecipeSizing,
     _enc: Encoding,
     inst: Inst,
     divert: &RegDiversions,
     func: &Function,
 ) -> u8 {
-    sizing.base_size + additional_size_if(0, inst, divert, func, needs_sib_byte_or_offset)
+    let needs_sib = test_input(1, inst, divert, func, needs_sib_byte);
+    sizing.base_size + if needs_sib { 1 } else { 0 }
 }
-fn size_plus_maybe_sib_or_offset_for_in_reg_1(
+fn size_plus_maybe_sib_or_offset_for_inreg_0(
     sizing: &RecipeSizing,
     _enc: Encoding,
     inst: Inst,
     divert: &RegDiversions,
     func: &Function,
 ) -> u8 {
-    sizing.base_size + additional_size_if(1, inst, divert, func, needs_sib_byte_or_offset)
+    let needs_sib_or_offset = test_input(0, inst, divert, func, needs_sib_byte_or_offset);
+    sizing.base_size + if needs_sib_or_offset { 1 } else { 0 }
+}
+fn size_plus_maybe_sib_or_offset_for_inreg_1(
+    sizing: &RecipeSizing,
+    _enc: Encoding,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    let needs_sib_or_offset = test_input(1, inst, divert, func, needs_sib_byte_or_offset);
+    sizing.base_size + if needs_sib_or_offset { 1 } else { 0 }
+}
+
+/// Infers whether a dynamic REX prefix will be emitted, for use with one input reg.
+///
+/// A REX prefix is known to be emitted if either:
+///  1. The EncodingBits specify that REX.W is to be set.
+///  2. Registers are used that require REX.R or REX.B bits for encoding.
+fn size_with_inferred_rex_for_inreg0(
+    sizing: &RecipeSizing,
+    enc: Encoding,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    let needs_rex = (EncodingBits::from(enc.bits()).rex_w() != 0)
+        || test_input(0, inst, divert, func, is_extended_reg);
+    sizing.base_size + if needs_rex { 1 } else { 0 }
+}
+
+/// Infers whether a dynamic REX prefix will be emitted, based on the second operand.
+fn size_with_inferred_rex_for_inreg1(
+    sizing: &RecipeSizing,
+    enc: Encoding,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    let needs_rex = (EncodingBits::from(enc.bits()).rex_w() != 0)
+        || test_input(1, inst, divert, func, is_extended_reg);
+    sizing.base_size + if needs_rex { 1 } else { 0 }
+}
+
+/// Infers whether a dynamic REX prefix will be emitted, based on the third operand.
+fn size_with_inferred_rex_for_inreg2(
+    sizing: &RecipeSizing,
+    enc: Encoding,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    let needs_rex = (EncodingBits::from(enc.bits()).rex_w() != 0)
+        || test_input(2, inst, divert, func, is_extended_reg);
+    sizing.base_size + if needs_rex { 1 } else { 0 }
+}
+
+/// Infers whether a dynamic REX prefix will be emitted, for use with two input registers.
+///
+/// A REX prefix is known to be emitted if either:
+///  1. The EncodingBits specify that REX.W is to be set.
+///  2. Registers are used that require REX.R or REX.B bits for encoding.
+fn size_with_inferred_rex_for_inreg0_inreg1(
+    sizing: &RecipeSizing,
+    enc: Encoding,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    let needs_rex = (EncodingBits::from(enc.bits()).rex_w() != 0)
+        || test_input(0, inst, divert, func, is_extended_reg)
+        || test_input(1, inst, divert, func, is_extended_reg);
+    sizing.base_size + if needs_rex { 1 } else { 0 }
+}
+
+/// Infers whether a dynamic REX prefix will be emitted, based on a single
+/// input register and a single output register.
+fn size_with_inferred_rex_for_inreg0_outreg0(
+    sizing: &RecipeSizing,
+    enc: Encoding,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    let needs_rex = (EncodingBits::from(enc.bits()).rex_w() != 0)
+        || test_input(0, inst, divert, func, is_extended_reg)
+        || test_result(0, inst, divert, func, is_extended_reg);
+    sizing.base_size + if needs_rex { 1 } else { 0 }
+}
+
+/// Infers whether a dynamic REX prefix will be emitted, for use with CMOV.
+///
+/// CMOV uses 3 inputs, with the REX is inferred from reg1 and reg2.
+fn size_with_inferred_rex_for_cmov(
+    sizing: &RecipeSizing,
+    enc: Encoding,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    let needs_rex = (EncodingBits::from(enc.bits()).rex_w() != 0)
+        || test_input(1, inst, divert, func, is_extended_reg)
+        || test_input(2, inst, divert, func, is_extended_reg);
+    sizing.base_size + if needs_rex { 1 } else { 0 }
 }
 
 /// If the value's definition is a constant immediate, returns its unpacked value, or None
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif
index e7abc4a273..2f8ce5d78a 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif
@@ -15,7 +15,7 @@ ebb1:
 ; sameln: function %br_icmp(i64 [%rdi]) fast {
 ; nextln:                                 ebb0(v0: i64):
 ; nextln: [RexOp1pu_id#b8]                    v1 = iconst.i64 0
-; nextln: [RexOp1icscc#8039]                  v2 = icmp eq v0, v1
+; nextln: [DynRexOp1icscc#8039]               v2 = icmp eq v0, v1
 ; nextln: [RexOp1t8jccb#75]                   brnz v2, ebb1
 ; nextln: [Op1jmpb#eb]                        jump ebb1
 ; nextln: 
@@ -37,7 +37,7 @@ ebb1(v2: i64):
 ; sameln: function %br_icmp_ebb_args(i64 [%rdi]) fast {
 ; nextln:                                 ebb0(v0: i64):
 ; nextln: [RexOp1pu_id#b8]                    v1 = iconst.i64 0
-; nextln: [RexOp1icscc#8039]                  v3 = icmp eq v0, v1
+; nextln: [DynRexOp1icscc#8039]               v3 = icmp eq v0, v1
 ; nextln: [RexOp1t8jccb#75]                   brnz v3, ebb1(v0)
 ; nextln: [Op1jmpb#eb]                        jump ebb1(v0)
 ; nextln: 
diff --git a/cranelift/filetests/filetests/isa/x86/relax_branch.clif b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
index 15c7e876a3..b97ca82f13 100644
--- a/cranelift/filetests/filetests/isa/x86/relax_branch.clif
+++ b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
@@ -21,95 +21,95 @@ function u0:2691(i32 [%rdi], i32 [%rsi], i64 vmctx [%r14]) -> i64 uext [%rax] ba
 @0005 [-]                           fallthrough ebb3(v0, v1)
 
                                 ebb3(v8: i32 [%rdi], v19: i32 [%rsi]):
-@0005 [RexOp1ldDisp8#808b,%rax]     v7 = load.i64 v2+48
-@0005 [RexOp1rcmp_ib#f083,%rflags]  v91 = ifcmp_imm v7, 0
-@0005 [trapif#00]                   trapif ne v91, interrupt
-[Op1umr#89,%rax]                    v105 = copy v8
-@000b [Op1r_ib#83,%rax]             v10 = iadd_imm v105, 1
-                                    v80 -> v10
-@0010 [Op1umr#89,%rcx]              v92 = uextend.i64 v8
-@0010 [RexOp1ld#808b,%rdx]          v93 = load.i64 notrap aligned readonly v2
-                                    v95 -> v93
-@0010 [Op2ldWithIndex#4be,%rcx]     v12 = sload8_complex.i32 v93+v92
-[Op1umr#89,%rbx]                    v106 = copy v12
-@0017 [Op1r_ib#40c1,%rbx]           v14 = ishl_imm v106, 24
-@001a [Op1r_ib#70c1,%rbx]           v16 = sshr_imm v14, 24
-[Op1umr#89,%rdi]                    v107 = copy v16
-@001f [Op1r_ib#83,%rdi]             v18 = iadd_imm v107, 32
-[RexOp1umr#89,%r8]                  v108 = copy v19
-@0026 [RexOp1r_ib#83,%r8]           v21 = iadd_imm v108, 1
-                                    v82 -> v21
-@002b [Op1umr#89,%rsi]              v94 = uextend.i64 v19
-@002b [Op2ldWithIndex#4be,%rdx]     v23 = sload8_complex.i32 v93+v94
-                                    v55 -> v23
-[Op1umr#89,%rsi]                    v109 = copy v23
-@0032 [Op1r_ib#40c1,%rsi]           v25 = ishl_imm v109, 24
-@0035 [Op1r_ib#70c1,%rsi]           v27 = sshr_imm v25, 24
-                                    v69 -> v27
-[RexOp1umr#89,%r9]                  v110 = copy v27
-@003a [RexOp1r_ib#83,%r9]           v29 = iadd_imm v110, 32
-                                    v68 -> v29
-@0042 [Op1r_ib#83,%rcx]             v31 = iadd_imm v12, -65
-@0045 [Op1r_ib#40c1,%rcx]           v33 = ishl_imm v31, 24
-@0048 [Op1r_ib#70c1,%rcx]           v35 = sshr_imm v33, 24
-@004c [Op1r_id#4081,%rcx]           v37 = band_imm v35, 255
-[Op1rcmp_ib#7083,%rflags]           v97 = ifcmp_imm v37, 26
-@0050 [Op1brib#70]                  brif sge v97, ebb6
-@0050 [-]                           fallthrough ebb10
+@0005 [RexOp1ldDisp8#808b,%rax]        v7 = load.i64 v2+48
+@0005 [DynRexOp1rcmp_ib#f083,%rflags]  v91 = ifcmp_imm v7, 0
+@0005 [trapif#00]                      trapif ne v91, interrupt
+[DynRexOp1umr#89,%rax]                 v105 = copy v8
+@000b [DynRexOp1r_ib#83,%rax]          v10 = iadd_imm v105, 1
+                                       v80 -> v10
+@0010 [Op1umr#89,%rcx]                 v92 = uextend.i64 v8
+@0010 [RexOp1ld#808b,%rdx]             v93 = load.i64 notrap aligned readonly v2
+                                       v95 -> v93
+@0010 [Op2ldWithIndex#4be,%rcx]        v12 = sload8_complex.i32 v93+v92
+[DynRexOp1umr#89,%rbx]                 v106 = copy v12
+@0017 [DynRexOp1r_ib#40c1,%rbx]        v14 = ishl_imm v106, 24
+@001a [DynRexOp1r_ib#70c1,%rbx]        v16 = sshr_imm v14, 24
+[DynRexOp1umr#89,%rdi]                 v107 = copy v16
+@001f [DynRexOp1r_ib#83,%rdi]          v18 = iadd_imm v107, 32
+[DynRexOp1umr#89,%r8]                  v108 = copy v19
+@0026 [DynRexOp1r_ib#83,%r8]           v21 = iadd_imm v108, 1
+                                       v82 -> v21
+@002b [Op1umr#89,%rsi]                 v94 = uextend.i64 v19
+@002b [Op2ldWithIndex#4be,%rdx]        v23 = sload8_complex.i32 v93+v94
+                                       v55 -> v23
+[DynRexOp1umr#89,%rsi]                 v109 = copy v23
+@0032 [DynRexOp1r_ib#40c1,%rsi]        v25 = ishl_imm v109, 24
+@0035 [DynRexOp1r_ib#70c1,%rsi]        v27 = sshr_imm v25, 24
+                                       v69 -> v27
+[DynRexOp1umr#89,%r9]                  v110 = copy v27
+@003a [DynRexOp1r_ib#83,%r9]           v29 = iadd_imm v110, 32
+                                       v68 -> v29
+@0042 [DynRexOp1r_ib#83,%rcx]          v31 = iadd_imm v12, -65
+@0045 [DynRexOp1r_ib#40c1,%rcx]        v33 = ishl_imm v31, 24
+@0048 [DynRexOp1r_ib#70c1,%rcx]        v35 = sshr_imm v33, 24
+@004c [DynRexOp1r_id#4081,%rcx]        v37 = band_imm v35, 255
+[DynRexOp1rcmp_ib#7083,%rflags]        v97 = ifcmp_imm v37, 26
+@0050 [Op1brib#70]                     brif sge v97, ebb6
+@0050 [-]                              fallthrough ebb10
 
                                 ebb10:
-[Op1umr#89,%rcx]                    v101 = copy v18
+[DynRexOp1umr#89,%rcx]              v101 = copy v18
 @0054 [Op1jmpb#eb]                  jump ebb5(v18, v101)
 
                                 ebb6:
-[Op1umr#89,%rcx]                    v102 = copy.i32 v16
+[DynRexOp1umr#89,%rcx]              v102 = copy.i32 v16
 @0059 [RexOp1rmov#89]               regmove v102, %rcx -> %rdi
 @0059 [RexOp1rmov#89]               regmove.i32 v16, %rbx -> %rcx
 @0059 [-]                           fallthrough ebb5(v102, v16)
 
                                 ebb5(v41: i32 [%rdi], v84: i32 [%rcx]):
                                     v83 -> v84
-@005d [Op1r_id#4081,%rdi]           v43 = band_imm v41, 255
-@0062 [Op1r_ib#40c1,%rdi]           v45 = ishl_imm v43, 24
+@005d [DynRexOp1r_id#4081,%rdi]     v43 = band_imm v41, 255
+@0062 [DynRexOp1r_ib#40c1,%rdi]     v45 = ishl_imm v43, 24
                                     v52 -> v45
 @0065 [RexOp1rmov#89]               regmove v45, %rdi -> %rbx
-@0065 [Op1r_ib#70c1,%rbx]           v47 = sshr_imm v45, 24
+@0065 [DynRexOp1r_ib#70c1,%rbx]     v47 = sshr_imm v45, 24
                                     v54 -> v47
 @0068 [RexOp1rmov#89]               regmove v47, %rbx -> %rdi
-@0068 [Op1icscc_ib#7083,%rbx]       v49 = icmp_imm ne v47, 0
+@0068 [DynRexOp1icscc_ib#7083,%rbx] v49 = icmp_imm ne v47, 0
 @0068 [RexOp2urm_noflags#4b6,%r10]  v50 = bint.i32 v49
-@0076 [Op1r_ib#83,%rdx]             v57 = iadd_imm.i32 v23, -65
-@0079 [Op1r_ib#40c1,%rdx]           v59 = ishl_imm v57, 24
-@007c [Op1r_ib#70c1,%rdx]           v61 = sshr_imm v59, 24
-@0080 [Op1r_id#4081,%rdx]           v63 = band_imm v61, 255
-[Op1rcmp_ib#7083,%rflags]           v98 = ifcmp_imm v63, 26
+@0076 [DynRexOp1r_ib#83,%rdx]       v57 = iadd_imm.i32 v23, -65
+@0079 [DynRexOp1r_ib#40c1,%rdx]     v59 = ishl_imm v57, 24
+@007c [DynRexOp1r_ib#70c1,%rdx]     v61 = sshr_imm v59, 24
+@0080 [DynRexOp1r_id#4081,%rdx]     v63 = band_imm v61, 255
+[DynRexOp1rcmp_ib#7083,%rflags]     v98 = ifcmp_imm v63, 26
 @0084 [RexOp1rmov#89]               regmove v47, %rdi -> %rbx
 @0084 [Op1brib#70]                  brif sge v98, ebb8
 @0084 [-]                           fallthrough ebb11
 
                                 ebb11:
-[RexOp1umr#89,%rdx]                 v103 = copy.i32 v29
+[DynRexOp1umr#89,%rdx]              v103 = copy.i32 v29
 @0088 [Op1jmpb#eb]                  jump ebb7(v29, v10, v21, v103)
 
                                 ebb8:
-[Op1umr#89,%rdx]                    v104 = copy.i32 v27
+[DynRexOp1umr#89,%rdx]              v104 = copy.i32 v27
 @008d [RexOp1rmov#89]               regmove v104, %rdx -> %r9
 @008d [RexOp1rmov#89]               regmove.i32 v27, %rsi -> %rdx
 @008d [-]                           fallthrough ebb7(v104, v10, v21, v27)
 
                                 ebb7(v67: i32 [%r9], v79: i32 [%rax], v81: i32 [%r8], v87: i32 [%rdx]):
-@0091 [RexOp1r_id#4081,%r9]         v71 = band_imm v67, 255
-@0094 [RexOp1r_ib#40c1,%r9]         v73 = ishl_imm v71, 24
-@0097 [RexOp1r_ib#70c1,%r9]         v75 = sshr_imm v73, 24
-@0098 [RexOp1icscc#39,%rbx]         v76 = icmp.i32 eq v47, v75
-@0098 [Op2urm_noflags_abcd#4b6,%rbx] v77 = bint.i32 v76
-@0099 [RexOp1rr#21,%r10]            v78 = band.i32 v50, v77
-@009a [RexOp1tjccb#74]              brz v78, ebb9
-@009a [-]                           fallthrough ebb12
+@0091 [DynRexOp1r_id#4081,%r9]           v71 = band_imm v67, 255
+@0094 [DynRexOp1r_ib#40c1,%r9]           v73 = ishl_imm v71, 24
+@0097 [DynRexOp1r_ib#70c1,%r9]           v75 = sshr_imm v73, 24
+@0098 [DynRexOp1icscc#39,%rbx]           v76 = icmp.i32 eq v47, v75
+@0098 [Op2urm_noflags_abcd#4b6,%rbx]     v77 = bint.i32 v76
+@0099 [DynRexOp1rr#21,%r10]              v78 = band.i32 v50, v77
+@009a [DynRexOp1tjccb#74]                brz v78, ebb9
+@009a [-]                                fallthrough ebb12
 
                                 ebb12:
-[RexOp1umr#89,%rcx]                 v99 = copy v81
-[Op1umr#89,%rdx]                    v100 = copy v79
+[DynRexOp1umr#89,%rcx]              v99 = copy v81
+[DynRexOp1umr#89,%rdx]              v100 = copy v79
 @00a4 [RexOp1rmov#89]               regmove v100, %rdx -> %rdi
 @00a4 [RexOp1rmov#89]               regmove v99, %rcx -> %rsi
 @00a4 [Op1jmpd#e9]                  jump ebb3(v100, v99); bin: e9 ffffff2d
@@ -118,9 +118,9 @@ function u0:2691(i32 [%rdi], i32 [%rsi], i64 vmctx [%r14]) -> i64 uext [%rax] ba
 @00a7 [-]                           fallthrough ebb4
 
                                 ebb4:
-@00ad [Op1r_id#4081,%rcx]           v86 = band_imm.i32 v84, 255
-@00b3 [Op1r_id#4081,%rdx]           v89 = band_imm.i32 v87, 255
-@00b4 [Op1rr#29,%rcx]               v90 = isub v86, v89
+@00ad [DynRexOp1r_id#4081,%rcx]     v86 = band_imm.i32 v84, 255
+@00b3 [DynRexOp1r_id#4081,%rdx]     v89 = band_imm.i32 v87, 255
+@00b4 [DynRexOp1rr#29,%rcx]         v90 = isub v86, v89
 @00b5 [-]                           fallthrough ebb2(v90)
 
                                 ebb2(v5: i32 [%rcx]):
diff --git a/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif b/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
index 358d098a6a..50e2389feb 100644
--- a/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
+++ b/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
@@ -4,7 +4,7 @@ target x86_64
 
 function %test_multiple_uses(i32 [%rdi]) -> i32 {
 ebb0(v0: i32 [%rdi]):
-[Op1rcmp_ib#7083,%rflags]           v3 = ifcmp_imm v0, 0
+[DynRexOp1rcmp_ib#7083,%rflags]     v3 = ifcmp_imm v0, 0
 [Op2seti_abcd#490,%rax]             v1 = trueif eq v3
 [RexOp2urm_noflags#4b6,%rax]        v2 = bint.i32 v1
 [Op1brib#70]                        brif eq v3, ebb1
diff --git a/cranelift/filetests/filetests/postopt/basic.clif b/cranelift/filetests/filetests/postopt/basic.clif
index 442d47de89..c38065f947 100644
--- a/cranelift/filetests/filetests/postopt/basic.clif
+++ b/cranelift/filetests/filetests/postopt/basic.clif
@@ -5,9 +5,9 @@ target i686
 
 function %br_icmp(i32, i32) -> i32 {
 ebb0(v0: i32, v1: i32):
-[Op1icscc#39,%rdx]  v2 = icmp slt v0, v1
-[Op1t8jccd_long#85] brnz v2, ebb1
-[Op1jmpb#eb]        jump ebb2
+[DynRexOp1icscc#39,%rdx]  v2 = icmp slt v0, v1
+[Op1t8jccd_long#85]       brnz v2, ebb1
+[Op1jmpb#eb]              jump ebb2
 
 ebb2:
 [Op1ret#c3]         return v1
@@ -35,9 +35,9 @@ ebb1:
 
 function %br_icmp_inverse(i32, i32) -> i32 {
 ebb0(v0: i32, v1: i32):
-[Op1icscc#39,%rdx]  v2 = icmp slt v0, v1
-[Op1t8jccd_long#84] brz v2, ebb1
-[Op1jmpb#eb]        jump ebb2
+[DynRexOp1icscc#39,%rdx]  v2 = icmp slt v0, v1
+[Op1t8jccd_long#84]       brz v2, ebb1
+[Op1jmpb#eb]              jump ebb2
 
 ebb2:
 [Op1ret#c3]         return v1
@@ -65,9 +65,9 @@ ebb1:
 
 function %br_icmp_imm(i32, i32) -> i32 {
 ebb0(v0: i32, v1: i32):
-[Op1icscc_ib#7083]  v2 = icmp_imm slt v0, 2
-[Op1t8jccd_long#84] brz v2, ebb1
-[Op1jmpb#eb]        jump ebb2
+[DynRexOp1icscc_ib#7083]  v2 = icmp_imm slt v0, 2
+[Op1t8jccd_long#84]       brz v2, ebb1
+[Op1jmpb#eb]              jump ebb2
 
 ebb2:
 [Op1ret#c3]         return v1
diff --git a/cranelift/filetests/filetests/postopt/complex_memory_ops.clif b/cranelift/filetests/filetests/postopt/complex_memory_ops.clif
index 0977fa0d7d..bae58cd8bb 100644
--- a/cranelift/filetests/filetests/postopt/complex_memory_ops.clif
+++ b/cranelift/filetests/filetests/postopt/complex_memory_ops.clif
@@ -3,7 +3,7 @@ target x86_64
 
 function %dual_loads(i64, i64) -> i64 {
 ebb0(v0: i64, v1: i64):
-[RexOp1rr#8001]    v3 = iadd v0, v1
+[DynRexOp1rr#8001] v3 = iadd v0, v1
                    v4 = load.i64 v3
                    v5 = uload8.i64 v3
                    v6 = sload8.i64 v3
@@ -29,7 +29,7 @@ ebb0(v0: i64, v1: i64):
 
 function %dual_loads2(i64, i64) -> i64 {
 ebb0(v0: i64, v1: i64):
-[RexOp1rr#8001]    v3 = iadd v0, v1
+[DynRexOp1rr#8001] v3 = iadd v0, v1
                    v4 = load.i64 v3+1
                    v5 = uload8.i64 v3+1
                    v6 = sload8.i64 v3+1
@@ -55,7 +55,7 @@ ebb0(v0: i64, v1: i64):
 
 function %dual_stores(i64, i64, i64) {
 ebb0(v0: i64, v1: i64, v2: i64):
-[RexOp1rr#8001]    v3 = iadd v0, v1
+[DynRexOp1rr#8001] v3 = iadd v0, v1
 [RexOp1st#8089]    store.i64 v2, v3
 [RexOp1st#88]      istore8.i64 v2, v3
 [RexMp1st#189]     istore16.i64 v2, v3
@@ -75,7 +75,7 @@ ebb0(v0: i64, v1: i64, v2: i64):
 
 function %dual_stores2(i64, i64, i64) {
 ebb0(v0: i64, v1: i64, v2: i64):
-[RexOp1rr#8001]         v3 = iadd v0, v1
+[DynRexOp1rr#8001] v3 = iadd v0, v1
 [RexOp1stDisp8#8089]    store.i64 v2, v3+1
 [RexOp1stDisp8#88]      istore8.i64 v2, v3+1
 [RexMp1stDisp8#189]     istore16.i64 v2, v3+1
diff --git a/cranelift/filetests/filetests/regalloc/coloring-227.clif b/cranelift/filetests/filetests/regalloc/coloring-227.clif
index a469230b51..1d4860815a 100644
--- a/cranelift/filetests/filetests/regalloc/coloring-227.clif
+++ b/cranelift/filetests/filetests/regalloc/coloring-227.clif
@@ -8,7 +8,7 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
                           ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i64):
 [RexOp1pu_id#b8]              v5 = iconst.i32 0
 [RexOp1pu_id#b8]              v6 = iconst.i32 0
-[RexOp1tjccb#74]              brz v6, ebb10
+[DynRexOp1tjccb#74]           brz v6, ebb10
 [Op1jmpb#eb]                  jump ebb3(v5, v5, v5, v5, v5, v5, v0, v1, v2, v3)
 
                           ebb3(v15: i32, v17: i32, v25: i32, v31: i32, v40: i32, v47: i32, v54: i32, v61: i32, v68: i32, v75: i32):
@@ -16,33 +16,33 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 
                           ebb6:
 [RexOp1pu_id#b8]              v8 = iconst.i32 0
-[RexOp1tjccb#75]              brnz v8, ebb5
+[DynRexOp1tjccb#75]           brnz v8, ebb5
 [Op1jmpb#eb]                  jump ebb20
 
                           ebb20:
 [RexOp1pu_id#b8]              v9 = iconst.i32 0
 [RexOp1pu_id#b8]              v11 = iconst.i32 0
-[RexOp1icscc#39]              v12 = icmp.i32 eq v15, v11
+[DynRexOp1icscc#39]           v12 = icmp.i32 eq v15, v11
 [RexOp2urm_noflags#4b6]       v13 = bint.i32 v12
-[RexOp1rr#21]                 v14 = band v9, v13
-[RexOp1tjccb#75]              brnz v14, ebb6
+[DynRexOp1rr#21]              v14 = band v9, v13
+[DynRexOp1tjccb#75]           brnz v14, ebb6
 [Op1jmpb#eb]                  jump ebb7
 
                           ebb7:
-[RexOp1tjccb#74]              brz.i32 v17, ebb8
+[DynRexOp1tjccb#74]           brz.i32 v17, ebb8
 [Op1jmpb#eb]                  jump ebb17
 
                           ebb17:
 [RexOp1pu_id#b8]              v18 = iconst.i32 0
-[RexOp1tjccb#74]              brz v18, ebb9
+[DynRexOp1tjccb#74]           brz v18, ebb9
 [Op1jmpb#eb]                  jump ebb16
 
                           ebb16:
 [RexOp1pu_id#b8]              v21 = iconst.i32 0
 [RexOp1umr#89]                v79 = uextend.i64 v5
-[RexOp1r_ib#8083]             v80 = iadd_imm.i64 v4, 0
+[DynRexOp1r_ib#8083]          v80 = iadd_imm.i64 v4, 0
 [RexOp1ld#808b]               v81 = load.i64 v80
-[RexOp1rr#8001]               v22 = iadd v81, v79
+[DynRexOp1rr#8001]            v22 = iadd v81, v79
 [RexMp1st#189]                istore16 v21, v22
 [Op1jmpb#eb]                  jump ebb9
 
@@ -52,8 +52,8 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
                           ebb8:
 [RexOp1pu_id#b8]              v27 = iconst.i32 3
 [RexOp1pu_id#b8]              v28 = iconst.i32 4
-[RexOp1rr#09]                 v35 = bor.i32 v31, v13
-[RexOp1tjccb#75]              brnz v35, ebb15(v27)
+[DynRexOp1rr#09]              v35 = bor.i32 v31, v13
+[DynRexOp1tjccb#75]           brnz v35, ebb15(v27)
 [Op1jmpb#eb]                  jump ebb15(v28)
 
                           ebb15(v36: i32):
@@ -71,24 +71,24 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 
                           ebb2(v7: i32, v45: i32, v52: i32, v59: i32, v66: i32, v73: i32):
 [RexOp1pu_id#b8]              v44 = iconst.i32 0
-[RexOp1tjccb#74]              brz v44, ebb12
+[DynRexOp1tjccb#74]           brz v44, ebb12
 [Op1jmpb#eb]                  jump ebb18
 
                           ebb18:
 [RexOp1pu_id#b8]              v50 = iconst.i32 11
-[RexOp1tjccb#74]              brz v50, ebb14
+[DynRexOp1tjccb#74]           brz v50, ebb14
 [Op1jmpb#eb]                  jump ebb19
 
                           ebb19:
 [RexOp1umr#89]                v82 = uextend.i64 v52
-[RexOp1r_ib#8083]             v83 = iadd_imm.i64 v4, 0
+[DynRexOp1r_ib#8083]          v83 = iadd_imm.i64 v4, 0
 [RexOp1ld#808b]               v84 = load.i64 v83
-[RexOp1rr#8001]               v57 = iadd v84, v82
+[DynRexOp1rr#8001]            v57 = iadd v84, v82
 [RexOp1ld#8b]                 v58 = load.i32 v57
 [RexOp1umr#89]                v85 = uextend.i64 v58
-[RexOp1r_ib#8083]             v86 = iadd_imm.i64 v4, 0
+[DynRexOp1r_ib#8083]          v86 = iadd_imm.i64 v4, 0
 [RexOp1ld#808b]               v87 = load.i64 v86
-[RexOp1rr#8001]               v64 = iadd v87, v85
+[DynRexOp1rr#8001]            v64 = iadd v87, v85
 [RexOp1st#88]                 istore8 v59, v64
 [RexOp1pu_id#b8]              v65 = iconst.i32 0
 [Op1jmpb#eb]                  jump ebb13(v65)
@@ -98,9 +98,9 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 
                           ebb13(v51: i32):
 [RexOp1umr#89]                v88 = uextend.i64 v45
-[RexOp1r_ib#8083]             v89 = iadd_imm.i64 v4, 0
+[DynRexOp1r_ib#8083]          v89 = iadd_imm.i64 v4, 0
 [RexOp1ld#808b]               v90 = load.i64 v89
-[RexOp1rr#8001]               v71 = iadd v90, v88
+[DynRexOp1rr#8001]            v71 = iadd v90, v88
 [RexOp1st#89]                 store v51, v71
 [Op1jmpb#eb]                  jump ebb12
 
diff --git a/cranelift/filetests/filetests/verifier/flags.clif b/cranelift/filetests/filetests/verifier/flags.clif
index 8a6abe8b4d..8273160510 100644
--- a/cranelift/filetests/filetests/verifier/flags.clif
+++ b/cranelift/filetests/filetests/verifier/flags.clif
@@ -4,7 +4,7 @@ target i686
 ; Simple, correct use of CPU flags.
 function %simple(i32) -> i32 {
                     ebb0(v0: i32):
-    [Op1rcmp#39]              v1 = ifcmp v0, v0
+    [DynRexOp1rcmp#39]        v1 = ifcmp v0, v0
     [Op2seti_abcd#490]        v2 = trueif ugt v1
     [Op2urm_noflags_abcd#4b6] v3 = bint.i32 v2
     [Op1ret#c3]               return v3
@@ -13,7 +13,7 @@ function %simple(i32) -> i32 {
 ; Overlapping flag values of different types.
 function %overlap(i32, f32) -> i32 {
                     ebb0(v0: i32, v1: f32):
-    [Op1rcmp#39]              v2 = ifcmp v0, v0
+    [DynRexOp1rcmp#39]        v2 = ifcmp v0, v0
     [Op2fcmp#42e]             v3 = ffcmp v1, v1
     [Op2setf_abcd#490]        v4 = trueff gt v3 ; error: conflicting live CPU flags: v2 and v3
     [Op2seti_abcd#490]        v5 = trueif ugt v2
@@ -25,8 +25,8 @@ function %overlap(i32, f32) -> i32 {
 ; CPU flags clobbered by arithmetic.
 function %clobbered(i32) -> i32 {
                     ebb0(v0: i32):
-    [Op1rcmp#39]              v1 = ifcmp v0, v0
-    [Op1rr#01]                v2 = iadd v0, v0 ; error: encoding clobbers live CPU flags in v1
+    [DynRexOp1rcmp#39]        v1 = ifcmp v0, v0
+    [DynRexOp1rr#01]          v2 = iadd v0, v0 ; error: encoding clobbers live CPU flags in v1
     [Op2seti_abcd#490]        v3 = trueif ugt v1
     [Op2urm_noflags_abcd#4b6] v4 = bint.i32 v3
     [Op1ret#c3]               return v4
@@ -35,7 +35,7 @@ function %clobbered(i32) -> i32 {
 ; CPU flags not clobbered by load.
 function %live_across_load(i32) -> i32 {
                     ebb0(v0: i32):
-    [Op1rcmp#39]              v1 = ifcmp v0, v0
+    [DynRexOp1rcmp#39]        v1 = ifcmp v0, v0
     [Op1ld#8b]                v2 = load.i32 v0
     [Op2seti_abcd#490]        v3 = trueif ugt v1
     [Op2urm_noflags_abcd#4b6] v4 = bint.i32 v3
@@ -45,7 +45,7 @@ function %live_across_load(i32) -> i32 {
 ; Correct use of CPU flags across EBB.
 function %live_across_ebb(i32) -> i32 {
                           ebb0(v0: i32):
-    [Op1rcmp#39]              v1 = ifcmp v0, v0
+    [DynRexOp1rcmp#39]        v1 = ifcmp v0, v0
     [Op1jmpb#eb]              jump ebb1
                           ebb1:
     [Op2seti_abcd#490]        v2 = trueif ugt v1
@@ -61,14 +61,14 @@ function %live_across_ebb_backwards(i32) -> i32 {
     [Op2urm_noflags_abcd#4b6] v3 = bint.i32 v2
     [Op1ret#c3]               return v3
                           ebb2:
-    [Op1rcmp#39]              v1 = ifcmp v0, v0
+    [DynRexOp1rcmp#39]        v1 = ifcmp v0, v0
     [Op1jmpb#eb]              jump ebb1
 }
 
 ; Flags live into loop.
 function %live_into_loop(i32) -> i32 {
                     ebb0(v0: i32):
-    [Op1rcmp#39]        v1 = ifcmp v0, v0
+    [DynRexOp1rcmp#39]  v1 = ifcmp v0, v0
     [Op1jmpb#eb]        jump ebb1
                     ebb1:
     [Op2seti_abcd#490]  v2 = trueif ugt v1

From c1c55607e106288a1822587a05c283c77528f152 Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Sat, 21 Dec 2019 13:37:42 -0800
Subject: [PATCH 2990/3084] cranelift-wasm: Check for `u32::MAX` function
 indices (#1307)

As an implementation-specific limit, we do not allow the full index space of
`0..=2^32 - 1` because we reserve index `2^32 - 1` for ourselves in
`cranelift-entity`.

Fixes #1306
---
 cranelift/wasm/src/sections_translator.rs        |  15 +++++++++++++--
 ...name-section-with-u32-max-function-index.wasm | Bin 0 -> 381 bytes
 2 files changed, 13 insertions(+), 2 deletions(-)
 create mode 100644 cranelift/wasmtests/issue-1306-name-section-with-u32-max-function-index.wasm

diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index 03444a547f..503ae7226e 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -7,7 +7,7 @@
 //! The special case of the initialize expressions for table elements offsets or global variables
 //! is handled, according to the semantics of WebAssembly, to only specific expressions that are
 //! interpreted on the fly.
-use crate::environ::{ModuleEnvironment, WasmResult};
+use crate::environ::{ModuleEnvironment, WasmError, WasmResult};
 use crate::state::ModuleTranslationState;
 use crate::translation_utils::{
     tabletype_to_type, type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory,
@@ -141,7 +141,13 @@ pub fn parse_function_section(
     functions: FunctionSectionReader,
     environ: &mut dyn ModuleEnvironment,
 ) -> WasmResult<()> {
-    environ.reserve_func_types(functions.get_count())?;
+    let num_functions = functions.get_count();
+    if num_functions == std::u32::MAX {
+        // We reserve `u32::MAX` for our own use in cranelift-entity.
+        return Err(WasmError::ImplLimitExceeded);
+    }
+
+    environ.reserve_func_types(num_functions)?;
 
     for entry in functions {
         let sigindex = entry?;
@@ -415,6 +421,11 @@ fn parse_function_name_subsection(
     let mut function_names = HashMap::new();
     for _ in 0..naming_reader.get_count() {
         let Naming { index, name } = naming_reader.read().ok()?;
+        if index == std::u32::MAX {
+            // We reserve `u32::MAX` for our own use in cranelift-entity.
+            return None;
+        }
+
         if function_names
             .insert(FuncIndex::from_u32(index), name)
             .is_some()
diff --git a/cranelift/wasmtests/issue-1306-name-section-with-u32-max-function-index.wasm b/cranelift/wasmtests/issue-1306-name-section-with-u32-max-function-index.wasm
new file mode 100644
index 0000000000000000000000000000000000000000..79d8dc33881dcfaa6d7467b3508f9da0cd629a46
GIT binary patch
literal 381
zcmZQbEY4+Q0D@0UEP08!sf@3f*ctTWqx?*g^NmuC3rdO%jZ=#93*s~L;tO&TlT!`N
zGt4awk_?iKObpG8Qlyz80YAGMGh1$IZhlcEliD$c$3M1)ZeFgv!vCqOuD;gp
z)R-2HsnT;r${X`vZs|+pPuk|mRm1zMa?+I^mXs4O2{0O^jSwnbY#~*@TNq
zi%a5DGvW(Ui;DB}5_2+3DkVglnYwc_W0ewew&vMBRypV>+2>TczUxJ9c}T0w=_|RW
z;)%+doIlQUF)Xdq6I5F}J>z3#v{&4eqUPFnLEQ)HA85#JTvDg1GAAKi;OCr^YJo*_
v(l|rzvIu
Date: Sun, 22 Dec 2019 07:10:36 +0100
Subject: [PATCH 2991/3084] Use explicit rex for brz and brnz encodings (#1308)

Fixes #1305. This papers over the problem to prevent crashes while we investigate the cause.
---
 cranelift/codegen/meta/src/isa/x86/encodings.rs | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 4fe2232508..68cc77be0e 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1368,10 +1368,10 @@ pub(crate) fn define(
     e.enc_both(brff, rec_brfd.opcodes(&JUMP_NEAR_IF_OVERFLOW));
 
     // Note that the tjccd opcode will be prefixed with 0x0f.
-    e.enc_i32_i64(brz, rec_tjccb.opcodes(&JUMP_SHORT_IF_EQUAL));
-    e.enc_i32_i64(brz, rec_tjccd.opcodes(&TEST_BYTE_REG));
-    e.enc_i32_i64(brnz, rec_tjccb.opcodes(&JUMP_SHORT_IF_NOT_EQUAL));
-    e.enc_i32_i64(brnz, rec_tjccd.opcodes(&TEST_REG));
+    e.enc_i32_i64_explicit_rex(brz, rec_tjccb.opcodes(&JUMP_SHORT_IF_EQUAL));
+    e.enc_i32_i64_explicit_rex(brz, rec_tjccd.opcodes(&TEST_BYTE_REG));
+    e.enc_i32_i64_explicit_rex(brnz, rec_tjccb.opcodes(&JUMP_SHORT_IF_NOT_EQUAL));
+    e.enc_i32_i64_explicit_rex(brnz, rec_tjccd.opcodes(&TEST_REG));
 
     // Branch on a b1 value in a register only looks at the low 8 bits. See also
     // bint encodings below.

From 9bbe378d4156126a92354840ac86e2d8541ece3f Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Mon, 6 Jan 2020 14:39:04 +0100
Subject: [PATCH 2992/3084] Fix master tests (#1316)

Co-authored-by: Benjamin Bouvier 
---
 .../filetests/isa/x86/relax_branch.clif          |  2 +-
 .../filetests/regalloc/coloring-227.clif         | 16 ++++++++--------
 2 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/cranelift/filetests/filetests/isa/x86/relax_branch.clif b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
index b97ca82f13..ae59a32b0d 100644
--- a/cranelift/filetests/filetests/isa/x86/relax_branch.clif
+++ b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
@@ -104,7 +104,7 @@ function u0:2691(i32 [%rdi], i32 [%rsi], i64 vmctx [%r14]) -> i64 uext [%rax] ba
 @0098 [DynRexOp1icscc#39,%rbx]           v76 = icmp.i32 eq v47, v75
 @0098 [Op2urm_noflags_abcd#4b6,%rbx]     v77 = bint.i32 v76
 @0099 [DynRexOp1rr#21,%r10]              v78 = band.i32 v50, v77
-@009a [DynRexOp1tjccb#74]                brz v78, ebb9
+@009a [RexOp1tjccb#74]                   brz v78, ebb9
 @009a [-]                                fallthrough ebb12
 
                                 ebb12:
diff --git a/cranelift/filetests/filetests/regalloc/coloring-227.clif b/cranelift/filetests/filetests/regalloc/coloring-227.clif
index 1d4860815a..0418924dfb 100644
--- a/cranelift/filetests/filetests/regalloc/coloring-227.clif
+++ b/cranelift/filetests/filetests/regalloc/coloring-227.clif
@@ -8,7 +8,7 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
                           ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i64):
 [RexOp1pu_id#b8]              v5 = iconst.i32 0
 [RexOp1pu_id#b8]              v6 = iconst.i32 0
-[DynRexOp1tjccb#74]           brz v6, ebb10
+[RexOp1tjccb#74]              brz v6, ebb10
 [Op1jmpb#eb]                  jump ebb3(v5, v5, v5, v5, v5, v5, v0, v1, v2, v3)
 
                           ebb3(v15: i32, v17: i32, v25: i32, v31: i32, v40: i32, v47: i32, v54: i32, v61: i32, v68: i32, v75: i32):
@@ -16,7 +16,7 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 
                           ebb6:
 [RexOp1pu_id#b8]              v8 = iconst.i32 0
-[DynRexOp1tjccb#75]           brnz v8, ebb5
+[RexOp1tjccb#75]              brnz v8, ebb5
 [Op1jmpb#eb]                  jump ebb20
 
                           ebb20:
@@ -25,16 +25,16 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 [DynRexOp1icscc#39]           v12 = icmp.i32 eq v15, v11
 [RexOp2urm_noflags#4b6]       v13 = bint.i32 v12
 [DynRexOp1rr#21]              v14 = band v9, v13
-[DynRexOp1tjccb#75]           brnz v14, ebb6
+[RexOp1tjccb#75]              brnz v14, ebb6
 [Op1jmpb#eb]                  jump ebb7
 
                           ebb7:
-[DynRexOp1tjccb#74]           brz.i32 v17, ebb8
+[RexOp1tjccb#74]              brz.i32 v17, ebb8
 [Op1jmpb#eb]                  jump ebb17
 
                           ebb17:
 [RexOp1pu_id#b8]              v18 = iconst.i32 0
-[DynRexOp1tjccb#74]           brz v18, ebb9
+[RexOp1tjccb#74]              brz v18, ebb9
 [Op1jmpb#eb]                  jump ebb16
 
                           ebb16:
@@ -53,7 +53,7 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 [RexOp1pu_id#b8]              v27 = iconst.i32 3
 [RexOp1pu_id#b8]              v28 = iconst.i32 4
 [DynRexOp1rr#09]              v35 = bor.i32 v31, v13
-[DynRexOp1tjccb#75]           brnz v35, ebb15(v27)
+[RexOp1tjccb#75]              brnz v35, ebb15(v27)
 [Op1jmpb#eb]                  jump ebb15(v28)
 
                           ebb15(v36: i32):
@@ -71,12 +71,12 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 
                           ebb2(v7: i32, v45: i32, v52: i32, v59: i32, v66: i32, v73: i32):
 [RexOp1pu_id#b8]              v44 = iconst.i32 0
-[DynRexOp1tjccb#74]           brz v44, ebb12
+[RexOp1tjccb#74]              brz v44, ebb12
 [Op1jmpb#eb]                  jump ebb18
 
                           ebb18:
 [RexOp1pu_id#b8]              v50 = iconst.i32 11
-[DynRexOp1tjccb#74]           brz v50, ebb14
+[RexOp1tjccb#74]              brz v50, ebb14
 [Op1jmpb#eb]                  jump ebb19
 
                           ebb19:

From 46e58fbaaa9c9635f830f59440f7428d602872ba Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Mon, 6 Jan 2020 15:33:22 -0800
Subject: [PATCH 2993/3084] Bitcasting at control flow exits (#1272)

* Bitcast vectors immediately before a return

* Bitcast vectors immediately before a block end

* Use helper function for bitcasting arguments

* Add FuncTranslationState::peekn_mut; allows mutating of peeked values

* Bitcast values in place, avoiding an allocation

Also, retrieves the correct EBB header types for bitcasting on Operator::End.

* Bitcast values of a function with no explicit Wasm return instruction

* Add Signature::return_types method

This eliminates some duplicate code and avoids extra `use`s of `Vec`.

* Add Signature::param_types method; only collect normal parameters in both this and Signature::return_types

* Move normal_args to Signature::num_normal_params method

This matches the organization of the other Signature::num_*_params methods.

* Bitcast values of Operator::Call and Operator::CallIndirect

* Add DataFlowGraph::ebb_param_types

* Bitcast values of Operator::Br and Operator::BrIf

* Bitcast values of Operator::BrTable
---
 cranelift/codegen/src/ir/dfg.rs        |   9 +++
 cranelift/codegen/src/ir/extfunc.rs    |  28 +++++++
 cranelift/wasm/src/code_translator.rs  | 103 ++++++++++++++++++++-----
 cranelift/wasm/src/func_translator.rs  |   8 +-
 cranelift/wasm/src/state/func_state.rs |  46 +++++------
 cranelift/wasmtests/call-simd.wat      |  14 ++++
 cranelift/wasmtests/icall-simd.wat     |   7 ++
 7 files changed, 171 insertions(+), 44 deletions(-)
 create mode 100644 cranelift/wasmtests/call-simd.wat
 create mode 100644 cranelift/wasmtests/icall-simd.wat

diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index 9e8634b741..7d453836f5 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -14,6 +14,7 @@ use crate::isa::TargetIsa;
 use crate::packed_option::ReservedValue;
 use crate::write::write_operands;
 use crate::HashMap;
+use alloc::vec::Vec;
 use core::fmt;
 use core::iter;
 use core::mem;
@@ -776,6 +777,14 @@ impl DataFlowGraph {
         self.ebbs[ebb].params.as_slice(&self.value_lists)
     }
 
+    /// Get the types of the parameters on `ebb`.
+    pub fn ebb_param_types(&self, ebb: Ebb) -> Vec {
+        self.ebb_params(ebb)
+            .iter()
+            .map(|&v| self.value_type(v))
+            .collect()
+    }
+
     /// Append a parameter with type `ty` to `ebb`.
     pub fn append_ebb_param(&mut self, ebb: Ebb, ty: Type) -> Value {
         let param = self.values.next_key();
diff --git a/cranelift/codegen/src/ir/extfunc.rs b/cranelift/codegen/src/ir/extfunc.rs
index 9274efe9b9..ab79bb5ea0 100644
--- a/cranelift/codegen/src/ir/extfunc.rs
+++ b/cranelift/codegen/src/ir/extfunc.rs
@@ -88,6 +88,16 @@ impl Signature {
             .count()
     }
 
+    /// Count the number of normal parameters in a signature.
+    /// Exclude special-purpose parameters that represent runtime stuff and not WebAssembly
+    /// arguments.
+    pub fn num_normal_params(&self) -> usize {
+        self.params
+            .iter()
+            .filter(|arg| arg.purpose == ArgumentPurpose::Normal)
+            .count()
+    }
+
     /// Does this signature take an struct return pointer parameter?
     pub fn uses_struct_return_param(&self) -> bool {
         self.uses_special_param(ArgumentPurpose::StructReturn)
@@ -102,6 +112,24 @@ impl Signature {
             .count()
             > 1
     }
+
+    /// Collect the normal parameter types of the signature; see `[ArgumentPurpose::Normal]`.
+    pub fn param_types(&self) -> Vec {
+        self.params
+            .iter()
+            .filter(|ap| ap.purpose == ArgumentPurpose::Normal)
+            .map(|ap| ap.value_type)
+            .collect()
+    }
+
+    /// Collect the normal return types of the signature; see `[ArgumentPurpose::Normal]`.
+    pub fn return_types(&self) -> Vec {
+        self.returns
+            .iter()
+            .filter(|ap| ap.purpose == ArgumentPurpose::Normal)
+            .map(|ap| ap.value_type)
+            .collect()
+    }
 }
 
 /// Wrapper type capable of displaying a `Signature` with correct register names.
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index e155594093..68f9cac1ba 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -267,12 +267,14 @@ pub fn translate_operator(
         }
         Operator::End => {
             let frame = state.control_stack.pop().unwrap();
+            let next_ebb = frame.following_code();
 
             if !builder.is_unreachable() || !builder.is_pristine() {
                 let return_count = frame.num_return_values();
-                builder
-                    .ins()
-                    .jump(frame.following_code(), state.peekn(return_count));
+                let return_args = state.peekn_mut(return_count);
+                let next_ebb_types = builder.func.dfg.ebb_param_types(next_ebb);
+                bitcast_arguments(return_args, &next_ebb_types, builder);
+                builder.ins().jump(frame.following_code(), return_args);
                 // You might expect that if we just finished an `if` block that
                 // didn't have a corresponding `else` block, then we would clean
                 // up our duplicate set of parameters that we pushed earlier
@@ -280,16 +282,14 @@ pub fn translate_operator(
                 // since we truncate the stack back to the original height
                 // below.
             }
-            builder.switch_to_block(frame.following_code());
-            builder.seal_block(frame.following_code());
+            builder.switch_to_block(next_ebb);
+            builder.seal_block(next_ebb);
             // If it is a loop we also have to seal the body loop block
             if let ControlStackFrame::Loop { header, .. } = frame {
                 builder.seal_block(header)
             }
             state.stack.truncate(frame.original_stack_size());
-            state
-                .stack
-                .extend_from_slice(builder.ebb_params(frame.following_code()));
+            state.stack.extend_from_slice(builder.ebb_params(next_ebb));
         }
         /**************************** Branch instructions *********************************
          * The branch instructions all have as arguments a target nesting level, which
@@ -325,9 +325,17 @@ pub fn translate_operator(
                 };
                 (return_count, frame.br_destination())
             };
-            builder
-                .ins()
-                .jump(br_destination, state.peekn(return_count));
+
+            // Bitcast any vector arguments to their default type, I8X16, before jumping.
+            let destination_args = state.peekn_mut(return_count);
+            let destination_types = builder.func.dfg.ebb_param_types(br_destination);
+            bitcast_arguments(
+                destination_args,
+                &destination_types[..return_count],
+                builder,
+            );
+
+            builder.ins().jump(br_destination, destination_args);
             state.popn(return_count);
             state.reachable = false;
         }
@@ -406,7 +414,17 @@ pub fn translate_operator(
                         frame.set_branched_to_exit();
                         frame.br_destination()
                     };
-                    builder.ins().jump(real_dest_ebb, state.peekn(return_count));
+
+                    // Bitcast any vector arguments to their default type, I8X16, before jumping.
+                    let destination_args = state.peekn_mut(return_count);
+                    let destination_types = builder.func.dfg.ebb_param_types(real_dest_ebb);
+                    bitcast_arguments(
+                        destination_args,
+                        &destination_types[..return_count],
+                        builder,
+                    );
+
+                    builder.ins().jump(real_dest_ebb, destination_args);
                 }
                 state.popn(return_count);
             }
@@ -420,10 +438,14 @@ pub fn translate_operator(
                 (return_count, frame.br_destination())
             };
             {
-                let args = state.peekn(return_count);
+                let return_args = state.peekn_mut(return_count);
+                let return_types = &builder.func.signature.return_types();
+                bitcast_arguments(return_args, &return_types, builder);
                 match environ.return_mode() {
-                    ReturnMode::NormalReturns => builder.ins().return_(args),
-                    ReturnMode::FallthroughReturn => builder.ins().jump(br_destination, args),
+                    ReturnMode::NormalReturns => builder.ins().return_(return_args),
+                    ReturnMode::FallthroughReturn => {
+                        builder.ins().jump(br_destination, return_args)
+                    }
                 };
             }
             state.popn(return_count);
@@ -436,11 +458,18 @@ pub fn translate_operator(
          ************************************************************************************/
         Operator::Call { function_index } => {
             let (fref, num_args) = state.get_direct_func(builder.func, *function_index, environ)?;
+
+            // Bitcast any vector arguments to their default type, I8X16, before calling.
+            let callee_signature =
+                &builder.func.dfg.signatures[builder.func.dfg.ext_funcs[fref].signature];
+            let args = state.peekn_mut(num_args);
+            bitcast_arguments(args, &callee_signature.param_types(), builder);
+
             let call = environ.translate_call(
                 builder.cursor(),
                 FuncIndex::from_u32(*function_index),
                 fref,
-                state.peekn(num_args),
+                args,
             )?;
             let inst_results = builder.inst_results(call);
             debug_assert_eq!(
@@ -459,6 +488,12 @@ pub fn translate_operator(
             let (sigref, num_args) = state.get_indirect_sig(builder.func, *index, environ)?;
             let table = state.get_table(builder.func, *table_index, environ)?;
             let callee = state.pop1();
+
+            // Bitcast any vector arguments to their default type, I8X16, before calling.
+            let callee_signature = &builder.func.dfg.signatures[sigref];
+            let args = state.peekn_mut(num_args);
+            bitcast_arguments(args, &callee_signature.param_types(), builder);
+
             let call = environ.translate_call_indirect(
                 builder.cursor(),
                 TableIndex::from_u32(*table_index),
@@ -1635,6 +1670,11 @@ fn translate_br_if(
 ) {
     let val = state.pop1();
     let (br_destination, inputs) = translate_br_if_args(relative_depth, state);
+
+    // Bitcast any vector arguments to their default type, I8X16, before jumping.
+    let destination_types = builder.func.dfg.ebb_param_types(br_destination);
+    bitcast_arguments(inputs, &destination_types[..inputs.len()], builder);
+
     builder.ins().brnz(val, br_destination, inputs);
 
     #[cfg(feature = "basic-blocks")]
@@ -1649,7 +1689,7 @@ fn translate_br_if(
 fn translate_br_if_args(
     relative_depth: u32,
     state: &mut FuncTranslationState,
-) -> (ir::Ebb, &[ir::Value]) {
+) -> (ir::Ebb, &mut [ir::Value]) {
     let i = state.control_stack.len() - 1 - (relative_depth as usize);
     let (return_count, br_destination) = {
         let frame = &mut state.control_stack[i];
@@ -1663,7 +1703,7 @@ fn translate_br_if_args(
         };
         (return_count, frame.br_destination())
     };
-    let inputs = state.peekn(return_count);
+    let inputs = state.peekn_mut(return_count);
     (br_destination, inputs)
 }
 
@@ -1826,7 +1866,7 @@ fn type_of(operator: &Operator) -> Type {
 
 /// Some SIMD operations only operate on I8X16 in CLIF; this will convert them to that type by
 /// adding a raw_bitcast if necessary.
-fn optionally_bitcast_vector(
+pub fn optionally_bitcast_vector(
     value: Value,
     needed_type: Type,
     builder: &mut FunctionBuilder,
@@ -1862,3 +1902,28 @@ fn pop2_with_bitcast(
     let bitcast_b = optionally_bitcast_vector(b, needed_type, builder);
     (bitcast_a, bitcast_b)
 }
+
+/// A helper for bitcasting a sequence of values (e.g. function arguments). If a value is a
+/// vector type that does not match its expected type, this will modify the value in place to point
+/// to the result of a `raw_bitcast`. This conversion is necessary to translate Wasm code that
+/// uses `V128` as function parameters (or implicitly in EBB parameters) and still use specific
+/// CLIF types (e.g. `I32X4`) in the function body.
+pub fn bitcast_arguments(
+    arguments: &mut [Value],
+    expected_types: &[Type],
+    builder: &mut FunctionBuilder,
+) {
+    assert_eq!(arguments.len(), expected_types.len());
+    for (i, t) in expected_types.iter().enumerate() {
+        if t.is_vector() {
+            assert!(
+                builder.func.dfg.value_type(arguments[i]).is_vector(),
+                "unexpected type mismatch: expected {}, argument {} was actually of type {}",
+                t,
+                arguments[i],
+                builder.func.dfg.value_type(arguments[i])
+            );
+            arguments[i] = optionally_bitcast_vector(arguments[i], *t, builder)
+        }
+    }
+}
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index 8f5d49bcdf..b6a46ecb01 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -4,7 +4,7 @@
 //! function to Cranelift IR guided by a `FuncEnvironment` which provides information about the
 //! WebAssembly module and the runtime environment.
 
-use crate::code_translator::translate_operator;
+use crate::code_translator::{bitcast_arguments, translate_operator};
 use crate::environ::{FuncEnvironment, ReturnMode, WasmResult};
 use crate::state::{FuncTranslationState, ModuleTranslationState};
 use crate::translation_utils::get_vmctx_value_label;
@@ -240,7 +240,11 @@ fn parse_function_body(
         debug_assert!(builder.is_pristine());
         if !builder.is_unreachable() {
             match environ.return_mode() {
-                ReturnMode::NormalReturns => builder.ins().return_(&state.stack),
+                ReturnMode::NormalReturns => {
+                    let return_types = &builder.func.signature.return_types();
+                    bitcast_arguments(&mut state.stack, &return_types, builder);
+                    builder.ins().return_(&state.stack)
+                }
                 ReturnMode::FallthroughReturn => builder.ins().fallthrough_return(&state.stack),
             };
         }
diff --git a/cranelift/wasm/src/state/func_state.rs b/cranelift/wasm/src/state/func_state.rs
index c768248ddd..98f75e6bc9 100644
--- a/cranelift/wasm/src/state/func_state.rs
+++ b/cranelift/wasm/src/state/func_state.rs
@@ -306,31 +306,40 @@ impl FuncTranslationState {
         (v1, v2, v3)
     }
 
+    /// Helper to ensure the the stack size is at least as big as `n`; note that due to
+    /// `debug_assert` this will not execute in non-optimized builds.
+    #[inline]
+    fn ensure_length_is_at_least(&self, n: usize) {
+        debug_assert!(
+            n <= self.stack.len(),
+            "attempted to access {} values but stack only has {} values",
+            n,
+            self.stack.len()
+        )
+    }
+
     /// Pop the top `n` values on the stack.
     ///
     /// The popped values are not returned. Use `peekn` to look at them before popping.
     pub(crate) fn popn(&mut self, n: usize) {
-        debug_assert!(
-            n <= self.stack.len(),
-            "popn({}) but stack only has {} values",
-            n,
-            self.stack.len()
-        );
+        self.ensure_length_is_at_least(n);
         let new_len = self.stack.len() - n;
         self.stack.truncate(new_len);
     }
 
     /// Peek at the top `n` values on the stack in the order they were pushed.
     pub(crate) fn peekn(&self, n: usize) -> &[Value] {
-        debug_assert!(
-            n <= self.stack.len(),
-            "peekn({}) but stack only has {} values",
-            n,
-            self.stack.len()
-        );
+        self.ensure_length_is_at_least(n);
         &self.stack[self.stack.len() - n..]
     }
 
+    /// Peek at the top `n` values on the stack in the order they were pushed.
+    pub(crate) fn peekn_mut(&mut self, n: usize) -> &mut [Value] {
+        self.ensure_length_is_at_least(n);
+        let len = self.stack.len();
+        &mut self.stack[len - n..]
+    }
+
     /// Push a block on the control stack.
     pub(crate) fn push_block(
         &mut self,
@@ -465,7 +474,7 @@ impl FuncTranslationState {
             Occupied(entry) => Ok(*entry.get()),
             Vacant(entry) => {
                 let sig = environ.make_indirect_sig(func, index)?;
-                Ok(*entry.insert((sig, normal_args(&func.dfg.signatures[sig]))))
+                Ok(*entry.insert((sig, func.dfg.signatures[sig].num_normal_params())))
             }
         }
     }
@@ -486,17 +495,8 @@ impl FuncTranslationState {
             Vacant(entry) => {
                 let fref = environ.make_direct_func(func, index)?;
                 let sig = func.dfg.ext_funcs[fref].signature;
-                Ok(*entry.insert((fref, normal_args(&func.dfg.signatures[sig]))))
+                Ok(*entry.insert((fref, func.dfg.signatures[sig].num_normal_params())))
             }
         }
     }
 }
-
-/// Count the number of normal parameters in a signature.
-/// Exclude special-purpose parameters that represent runtime stuff and not WebAssembly arguments.
-fn normal_args(sig: &ir::Signature) -> usize {
-    sig.params
-        .iter()
-        .filter(|arg| arg.purpose == ir::ArgumentPurpose::Normal)
-        .count()
-}
diff --git a/cranelift/wasmtests/call-simd.wat b/cranelift/wasmtests/call-simd.wat
new file mode 100644
index 0000000000..61834d86bd
--- /dev/null
+++ b/cranelift/wasmtests/call-simd.wat
@@ -0,0 +1,14 @@
+(module
+  (func $main
+    (v128.const i32x4 1 2 3 4)
+    (v128.const i32x4 1 2 3 4)
+    (call $add)
+    drop
+  )
+  (func $add (param $a v128) (param $b v128) (result v128)
+    (local.get $a)
+    (local.get $b)
+    (i32x4.add)
+  )
+  (start $main)
+)
diff --git a/cranelift/wasmtests/icall-simd.wat b/cranelift/wasmtests/icall-simd.wat
new file mode 100644
index 0000000000..d656b265b9
--- /dev/null
+++ b/cranelift/wasmtests/icall-simd.wat
@@ -0,0 +1,7 @@
+(module
+  (type $ft (func (param v128) (result v128)))
+  (func $foo (export "foo") (param i32) (param v128) (result v128)
+    (call_indirect (type $ft) (local.get 1) (local.get 0))
+  )
+  (table (;0;) 23 23 anyfunc)
+)

From 8ff6d640d6d7bf64f1f3936c4df4e995db67e711 Mon Sep 17 00:00:00 2001
From: C Burgos 
Date: Tue, 7 Jan 2020 13:09:34 -0500
Subject: [PATCH 2994/3084] Remove unused function and associated tests (#1312)

* Remove unused function and associated tests
* Removed unused imports
---
 cranelift/codegen/src/dominator_tree.rs | 117 ------------------------
 1 file changed, 117 deletions(-)

diff --git a/cranelift/codegen/src/dominator_tree.rs b/cranelift/codegen/src/dominator_tree.rs
index 67446b21f6..5251121cdc 100644
--- a/cranelift/codegen/src/dominator_tree.rs
+++ b/cranelift/codegen/src/dominator_tree.rs
@@ -448,61 +448,6 @@ impl DominatorTree {
     }
 }
 
-impl DominatorTree {
-    /// When splitting an `Ebb` using `Layout::split_ebb`, you can use this method to update
-    /// the dominator tree locally rather than recomputing it.
-    ///
-    /// `old_ebb` is the `Ebb` before splitting, and `new_ebb` is the `Ebb` which now contains
-    /// the second half of `old_ebb`. `split_jump_inst` is the terminator jump instruction of
-    /// `old_ebb` that points to `new_ebb`.
-    pub fn recompute_split_ebb(&mut self, old_ebb: Ebb, new_ebb: Ebb, split_jump_inst: Inst) {
-        if !self.is_reachable(old_ebb) {
-            // old_ebb is unreachable, it stays so and new_ebb is unreachable too
-            self.nodes[new_ebb] = Default::default();
-            return;
-        }
-        // We use the RPO comparison on the postorder list so we invert the operands of the
-        // comparison
-        let old_ebb_postorder_index = self
-            .postorder
-            .as_slice()
-            .binary_search_by(|probe| self.rpo_cmp_ebb(old_ebb, *probe))
-            .expect("the old ebb is not declared to the dominator tree");
-        let new_ebb_rpo = self.insert_after_rpo(old_ebb, old_ebb_postorder_index, new_ebb);
-        self.nodes[new_ebb] = DomNode {
-            rpo_number: new_ebb_rpo,
-            idom: Some(split_jump_inst).into(),
-        };
-    }
-
-    // Insert new_ebb just after ebb in the RPO. This function checks
-    // if there is a gap in rpo numbers; if yes it returns the number in the gap and if
-    // not it renumbers.
-    fn insert_after_rpo(&mut self, ebb: Ebb, ebb_postorder_index: usize, new_ebb: Ebb) -> u32 {
-        let ebb_rpo_number = self.nodes[ebb].rpo_number;
-        let inserted_rpo_number = ebb_rpo_number + 1;
-        // If there is no gaps in RPo numbers to insert this new number, we iterate
-        // forward in RPO numbers and backwards in the postorder list of EBBs, renumbering the Ebbs
-        // until we find a gap
-        for (¤t_ebb, current_rpo) in self.postorder[0..ebb_postorder_index]
-            .iter()
-            .rev()
-            .zip(inserted_rpo_number + 1..)
-        {
-            if self.nodes[current_ebb].rpo_number < current_rpo {
-                // There is no gap, we renumber
-                self.nodes[current_ebb].rpo_number = current_rpo;
-            } else {
-                // There is a gap, we stop the renumbering and exit
-                break;
-            }
-        }
-        // TODO: insert in constant time?
-        self.postorder.insert(ebb_postorder_index, new_ebb);
-        inserted_rpo_number
-    }
-}
-
 /// Optional pre-order information that can be computed for a dominator tree.
 ///
 /// This data structure is computed from a `DominatorTree` and provides:
@@ -681,8 +626,6 @@ mod tests {
     use crate::flowgraph::ControlFlowGraph;
     use crate::ir::types::*;
     use crate::ir::{Function, InstBuilder, TrapCode};
-    use crate::settings;
-    use crate::verifier::{verify_context, VerifierErrors};
 
     #[test]
     fn empty() {
@@ -886,64 +829,4 @@ mod tests {
         assert!(!dt.dominates(jmp21, ebb2, &cur.func.layout));
         assert!(dt.dominates(jmp21, jmp21, &cur.func.layout));
     }
-
-    #[test]
-    fn renumbering() {
-        let mut func = Function::new();
-        let entry = func.dfg.make_ebb();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb100 = func.dfg.make_ebb();
-
-        let mut cur = FuncCursor::new(&mut func);
-
-        cur.insert_ebb(entry);
-        cur.ins().jump(ebb0, &[]);
-
-        cur.insert_ebb(ebb0);
-        let cond = cur.ins().iconst(I32, 0);
-        let inst2 = cur.ins().brz(cond, ebb0, &[]);
-        let inst3 = cur.ins().brz(cond, ebb0, &[]);
-        let inst4 = cur.ins().brz(cond, ebb0, &[]);
-        let inst5 = cur.ins().brz(cond, ebb0, &[]);
-        cur.ins().jump(ebb100, &[]);
-        cur.insert_ebb(ebb100);
-        cur.ins().return_(&[]);
-
-        let mut cfg = ControlFlowGraph::with_function(cur.func);
-        let mut dt = DominatorTree::with_function(cur.func, &cfg);
-
-        let ebb1 = cur.func.dfg.make_ebb();
-        cur.func.layout.split_ebb(ebb1, inst2);
-        cur.goto_bottom(ebb0);
-        let middle_jump_inst = cur.ins().jump(ebb1, &[]);
-
-        dt.recompute_split_ebb(ebb0, ebb1, middle_jump_inst);
-
-        let ebb2 = cur.func.dfg.make_ebb();
-        cur.func.layout.split_ebb(ebb2, inst3);
-        cur.goto_bottom(ebb1);
-        let middle_jump_inst = cur.ins().jump(ebb2, &[]);
-        dt.recompute_split_ebb(ebb1, ebb2, middle_jump_inst);
-
-        let ebb3 = cur.func.dfg.make_ebb();
-        cur.func.layout.split_ebb(ebb3, inst4);
-        cur.goto_bottom(ebb2);
-        let middle_jump_inst = cur.ins().jump(ebb3, &[]);
-        dt.recompute_split_ebb(ebb2, ebb3, middle_jump_inst);
-
-        let ebb4 = cur.func.dfg.make_ebb();
-        cur.func.layout.split_ebb(ebb4, inst5);
-        cur.goto_bottom(ebb3);
-        let middle_jump_inst = cur.ins().jump(ebb4, &[]);
-        dt.recompute_split_ebb(ebb3, ebb4, middle_jump_inst);
-
-        cfg.compute(cur.func);
-
-        let flags = settings::Flags::new(settings::builder());
-        let mut errors = VerifierErrors::default();
-
-        verify_context(cur.func, &cfg, &dt, &flags, &mut errors).unwrap();
-
-        assert!(errors.0.is_empty());
-    }
 }

From 6fe86bcb610bf0979017186ace0557974916df6b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 8 Jan 2020 09:28:05 -0800
Subject: [PATCH 2995/3084] Fix SIMD float comparison encoding (#1285)

The Intel manual uses `CMPNLT` and `CMPNLE` to denote not-less-than and not-less-than-or-equals. These were translated previously to `FloatCC::GreaterThan` and `FloatCC::GreaterThanOrEqual` but should be correctly translated to `FloatCC::UnorderedOrGreaterThanOrEqual` and `FloatCC::UnorderedOrGreaterThan`. This change adds the necessary legalizations to make use of these new encodings.
---
 .../codegen/meta/src/isa/x86/legalize.rs      | 20 +++++++++++++++++++
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 18 ++++++++---------
 .../isa/x86/simd-comparison-binemit.clif      |  8 ++++----
 .../isa/x86/simd-comparison-run.clif          | 14 +++++++++++++
 4 files changed, 47 insertions(+), 13 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index f62019367b..c6161cf43b 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -552,6 +552,26 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         narrow.legalize(def!(c = icmp_(ule, a, b)), vec![def!(c = icmp(uge, b, a))]);
     }
 
+    // SIMD fcmp greater-/less-than
+    let gt = Literal::enumerator_for(&imm.floatcc, "gt");
+    let lt = Literal::enumerator_for(&imm.floatcc, "lt");
+    let ge = Literal::enumerator_for(&imm.floatcc, "ge");
+    let le = Literal::enumerator_for(&imm.floatcc, "le");
+    let ugt = Literal::enumerator_for(&imm.floatcc, "ugt");
+    let ult = Literal::enumerator_for(&imm.floatcc, "ult");
+    let uge = Literal::enumerator_for(&imm.floatcc, "uge");
+    let ule = Literal::enumerator_for(&imm.floatcc, "ule");
+    for ty in &[F32, F64] {
+        let fcmp_ = fcmp.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(def!(c = fcmp_(gt, a, b)), vec![def!(c = fcmp(lt, b, a))]);
+        let fcmp_ = fcmp.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(def!(c = fcmp_(ge, a, b)), vec![def!(c = fcmp(le, b, a))]);
+        let fcmp_ = fcmp.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(def!(c = fcmp_(ult, a, b)), vec![def!(c = fcmp(ugt, b, a))]);
+        let fcmp_ = fcmp.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(def!(c = fcmp_(ule, a, b)), vec![def!(c = fcmp(uge, b, a))]);
+    }
+
     for ty in &[F32, F64] {
         let fneg = fneg.bind(vector(*ty, sse_vector_size));
         let lane_type_as_int = LaneType::int_from_bits(LaneType::from(*ty).lane_bits() as u16);
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 521248082b..b2ce9e628e 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -3169,7 +3169,7 @@ pub(crate) fn define<'shared>(
     );
 
     {
-        let supported_floatccs: Vec = ["eq", "lt", "le", "uno", "ne", "gt", "ge", "ord"]
+        let supported_floatccs: Vec = ["eq", "lt", "le", "uno", "ne", "uge", "ugt", "ord"]
             .iter()
             .map(|name| Literal::enumerator_for(floatcc, name))
             .collect();
@@ -3189,14 +3189,14 @@ pub(crate) fn define<'shared>(
                     // Add immediate byte indicating what type of comparison.
                     use crate::ir::condcodes::FloatCC::*;
                     let imm = match cond {
-                        Equal               => 0x00,
-                        LessThan            => 0x01,
-                        LessThanOrEqual     => 0x02,
-                        Unordered           => 0x03,
-                        NotEqual            => 0x04,
-                        GreaterThanOrEqual  => 0x05,
-                        GreaterThan         => 0x06,
-                        Ordered             => 0x07,
+                        Equal                      => 0x00,
+                        LessThan                   => 0x01,
+                        LessThanOrEqual            => 0x02,
+                        Unordered                  => 0x03,
+                        NotEqual                   => 0x04,
+                        UnorderedOrGreaterThanOrEqual => 0x05,
+                        UnorderedOrGreaterThan => 0x06,
+                        Ordered                    => 0x07,
                         _ => panic!("{} not supported by pfcmp", cond),
                     };
                     sink.put1(imm);
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
index be8e7d4e8e..722e705a85 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
@@ -60,8 +60,8 @@ ebb0(v0: f32x4 [%xmm2], v1: f32x4 [%xmm4]):
 [-, %xmm2]  v4 = fcmp le v0, v1     ; bin: 40 0f c2 d4 02
 [-, %xmm2]  v5 = fcmp uno v0, v1    ; bin: 40 0f c2 d4 03
 [-, %xmm2]  v6 = fcmp ne v0, v1     ; bin: 40 0f c2 d4 04
-[-, %xmm2]  v7 = fcmp ge v0, v1     ; bin: 40 0f c2 d4 05
-[-, %xmm2]  v8 = fcmp gt v0, v1     ; bin: 40 0f c2 d4 06
+[-, %xmm2]  v7 = fcmp uge v0, v1    ; bin: 40 0f c2 d4 05
+[-, %xmm2]  v8 = fcmp ugt v0, v1    ; bin: 40 0f c2 d4 06
 [-, %xmm2]  v9 = fcmp ord v0, v1    ; bin: 40 0f c2 d4 07
             return
 }
@@ -73,8 +73,8 @@ ebb0(v0: f64x2 [%xmm2], v1: f64x2 [%xmm0]):
 [-, %xmm2]  v4 = fcmp le v0, v1     ; bin: 66 40 0f c2 d0 02
 [-, %xmm2]  v5 = fcmp uno v0, v1    ; bin: 66 40 0f c2 d0 03
 [-, %xmm2]  v6 = fcmp ne v0, v1     ; bin: 66 40 0f c2 d0 04
-[-, %xmm2]  v7 = fcmp ge v0, v1     ; bin: 66 40 0f c2 d0 05
-[-, %xmm2]  v8 = fcmp gt v0, v1     ; bin: 66 40 0f c2 d0 06
+[-, %xmm2]  v7 = fcmp uge v0, v1    ; bin: 66 40 0f c2 d0 05
+[-, %xmm2]  v8 = fcmp ugt v0, v1    ; bin: 66 40 0f c2 d0 06
 [-, %xmm2]  v9 = fcmp ord v0, v1    ; bin: 66 40 0f c2 d0 07
             return
 }
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
index e4a5e6fea7..444d4e28bd 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
@@ -231,3 +231,17 @@ ebb0:
     return v8
 }
 ; run
+
+function %fcmp_gt_nans_f32x4() -> b1 {
+ebb0:
+    v0 = vconst.f32x4 [NaN 0x42.0 -NaN NaN]
+    v1 = vconst.f32x4 [NaN NaN 0x42.0 Inf]
+    v2 = fcmp gt v0, v1
+    ; now check that the result v2 is all zeroes
+    v3 = vconst.i32x4 0x00
+    v4 = raw_bitcast.i32x4 v2
+    v5 = icmp eq v3, v4
+    v8 = vall_true v5
+    return v8
+}
+; run

From 3a4b1cc98995689962d5997c21e6875903323b70 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 8 Jan 2020 18:38:40 +0100
Subject: [PATCH 2996/3084] Split define encodings + start splitting
 instruction definitions (#1322)

* [meta] Split the x86 encodings define function into smaller ones;
* [meta] Start splitting instruction definitions into smaller functions;
---
 .../codegen/meta/src/isa/x86/encodings.rs     | 1824 +++++++++--------
 .../codegen/meta/src/shared/instructions.rs   |  892 ++++----
 2 files changed, 1487 insertions(+), 1229 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 68cc77be0e..ad4934ee97 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -394,336 +394,41 @@ impl PerCpuModeEncodings {
 
 // Definitions.
 
-#[allow(clippy::cognitive_complexity)]
-pub(crate) fn define(
-    shared_defs: &SharedDefinitions,
-    settings: &SettingGroup,
-    x86: &InstructionGroup,
-    r: &RecipeGroup,
-) -> PerCpuModeEncodings {
+#[inline(never)]
+fn define_moves(e: &mut PerCpuModeEncodings, shared_defs: &SharedDefinitions, r: &RecipeGroup) {
     let shared = &shared_defs.instructions;
     let formats = &shared_defs.formats;
 
     // Shorthands for instructions.
-    let adjust_sp_down = shared.by_name("adjust_sp_down");
-    let adjust_sp_down_imm = shared.by_name("adjust_sp_down_imm");
-    let adjust_sp_up_imm = shared.by_name("adjust_sp_up_imm");
-    let band = shared.by_name("band");
-    let band_imm = shared.by_name("band_imm");
-    let band_not = shared.by_name("band_not");
     let bconst = shared.by_name("bconst");
     let bint = shared.by_name("bint");
-    let bitcast = shared.by_name("bitcast");
-    let bnot = shared.by_name("bnot");
-    let bor = shared.by_name("bor");
-    let bor_imm = shared.by_name("bor_imm");
-    let brff = shared.by_name("brff");
-    let brif = shared.by_name("brif");
-    let brnz = shared.by_name("brnz");
-    let brz = shared.by_name("brz");
-    let bxor = shared.by_name("bxor");
-    let bxor_imm = shared.by_name("bxor_imm");
-    let call = shared.by_name("call");
-    let call_indirect = shared.by_name("call_indirect");
-    let ceil = shared.by_name("ceil");
-    let clz = shared.by_name("clz");
     let copy = shared.by_name("copy");
-    let copy_nop = shared.by_name("copy_nop");
     let copy_special = shared.by_name("copy_special");
     let copy_to_ssa = shared.by_name("copy_to_ssa");
-    let ctz = shared.by_name("ctz");
-    let debugtrap = shared.by_name("debugtrap");
-    let f32const = shared.by_name("f32const");
-    let f64const = shared.by_name("f64const");
-    let fadd = shared.by_name("fadd");
-    let fcmp = shared.by_name("fcmp");
-    let fcvt_from_sint = shared.by_name("fcvt_from_sint");
-    let fdemote = shared.by_name("fdemote");
-    let fdiv = shared.by_name("fdiv");
-    let ffcmp = shared.by_name("ffcmp");
-    let fill = shared.by_name("fill");
-    let fill_nop = shared.by_name("fill_nop");
-    let floor = shared.by_name("floor");
-    let fmax = shared.by_name("fmax");
-    let fmin = shared.by_name("fmin");
-    let fmul = shared.by_name("fmul");
-    let fpromote = shared.by_name("fpromote");
-    let fsub = shared.by_name("fsub");
-    let func_addr = shared.by_name("func_addr");
     let get_pinned_reg = shared.by_name("get_pinned_reg");
-    let iadd = shared.by_name("iadd");
-    let iadd_ifcout = shared.by_name("iadd_ifcout");
-    let iadd_ifcin = shared.by_name("iadd_ifcin");
-    let iadd_ifcarry = shared.by_name("iadd_ifcarry");
-    let iadd_imm = shared.by_name("iadd_imm");
-    let icmp = shared.by_name("icmp");
-    let icmp_imm = shared.by_name("icmp_imm");
     let iconst = shared.by_name("iconst");
-    let ifcmp = shared.by_name("ifcmp");
-    let ifcmp_imm = shared.by_name("ifcmp_imm");
-    let ifcmp_sp = shared.by_name("ifcmp_sp");
-    let imul = shared.by_name("imul");
-    let indirect_jump_table_br = shared.by_name("indirect_jump_table_br");
     let ireduce = shared.by_name("ireduce");
-    let ishl = shared.by_name("ishl");
-    let ishl_imm = shared.by_name("ishl_imm");
-    let is_null = shared.by_name("is_null");
-    let istore16 = shared.by_name("istore16");
-    let istore16_complex = shared.by_name("istore16_complex");
-    let istore32 = shared.by_name("istore32");
-    let istore32_complex = shared.by_name("istore32_complex");
-    let istore8 = shared.by_name("istore8");
-    let istore8_complex = shared.by_name("istore8_complex");
-    let isub = shared.by_name("isub");
-    let isub_ifbout = shared.by_name("isub_ifbout");
-    let isub_ifbin = shared.by_name("isub_ifbin");
-    let isub_ifborrow = shared.by_name("isub_ifborrow");
-    let jump = shared.by_name("jump");
-    let jump_table_base = shared.by_name("jump_table_base");
-    let jump_table_entry = shared.by_name("jump_table_entry");
-    let load = shared.by_name("load");
-    let load_complex = shared.by_name("load_complex");
-    let nearest = shared.by_name("nearest");
-    let null = shared.by_name("null");
-    let popcnt = shared.by_name("popcnt");
-    let raw_bitcast = shared.by_name("raw_bitcast");
-    let regfill = shared.by_name("regfill");
     let regmove = shared.by_name("regmove");
-    let regspill = shared.by_name("regspill");
-    let return_ = shared.by_name("return");
-    let rotl = shared.by_name("rotl");
-    let rotl_imm = shared.by_name("rotl_imm");
-    let rotr = shared.by_name("rotr");
-    let rotr_imm = shared.by_name("rotr_imm");
-    let sadd_sat = shared.by_name("sadd_sat");
-    let safepoint = shared.by_name("safepoint");
-    let scalar_to_vector = shared.by_name("scalar_to_vector");
-    let selectif = shared.by_name("selectif");
     let sextend = shared.by_name("sextend");
     let set_pinned_reg = shared.by_name("set_pinned_reg");
-    let sload16 = shared.by_name("sload16");
-    let sload16_complex = shared.by_name("sload16_complex");
-    let sload32 = shared.by_name("sload32");
-    let sload32_complex = shared.by_name("sload32_complex");
-    let sload8 = shared.by_name("sload8");
-    let sload8_complex = shared.by_name("sload8_complex");
-    let spill = shared.by_name("spill");
-    let sqrt = shared.by_name("sqrt");
-    let sshr = shared.by_name("sshr");
-    let sshr_imm = shared.by_name("sshr_imm");
-    let ssub_sat = shared.by_name("ssub_sat");
-    let stack_addr = shared.by_name("stack_addr");
-    let store = shared.by_name("store");
-    let store_complex = shared.by_name("store_complex");
-    let symbol_value = shared.by_name("symbol_value");
-    let trap = shared.by_name("trap");
-    let trapff = shared.by_name("trapff");
-    let trapif = shared.by_name("trapif");
-    let resumable_trap = shared.by_name("resumable_trap");
-    let trueff = shared.by_name("trueff");
-    let trueif = shared.by_name("trueif");
-    let trunc = shared.by_name("trunc");
-    let uadd_sat = shared.by_name("uadd_sat");
     let uextend = shared.by_name("uextend");
-    let uload16 = shared.by_name("uload16");
-    let uload16_complex = shared.by_name("uload16_complex");
-    let uload32 = shared.by_name("uload32");
-    let uload32_complex = shared.by_name("uload32_complex");
-    let uload8 = shared.by_name("uload8");
-    let uload8_complex = shared.by_name("uload8_complex");
-    let ushr = shared.by_name("ushr");
-    let ushr_imm = shared.by_name("ushr_imm");
-    let usub_sat = shared.by_name("usub_sat");
-    let vconst = shared.by_name("vconst");
-    let x86_bsf = x86.by_name("x86_bsf");
-    let x86_bsr = x86.by_name("x86_bsr");
-    let x86_cvtt2si = x86.by_name("x86_cvtt2si");
-    let x86_fmax = x86.by_name("x86_fmax");
-    let x86_fmin = x86.by_name("x86_fmin");
-    let x86_insertps = x86.by_name("x86_insertps");
-    let x86_movlhps = x86.by_name("x86_movlhps");
-    let x86_movsd = x86.by_name("x86_movsd");
-    let x86_pop = x86.by_name("x86_pop");
-    let x86_pextr = x86.by_name("x86_pextr");
-    let x86_pinsr = x86.by_name("x86_pinsr");
-    let x86_pmaxs = x86.by_name("x86_pmaxs");
-    let x86_pmaxu = x86.by_name("x86_pmaxu");
-    let x86_pmins = x86.by_name("x86_pmins");
-    let x86_pminu = x86.by_name("x86_pminu");
-    let x86_pshufd = x86.by_name("x86_pshufd");
-    let x86_pshufb = x86.by_name("x86_pshufb");
-    let x86_psll = x86.by_name("x86_psll");
-    let x86_psra = x86.by_name("x86_psra");
-    let x86_psrl = x86.by_name("x86_psrl");
-    let x86_ptest = x86.by_name("x86_ptest");
-    let x86_push = x86.by_name("x86_push");
-    let x86_sdivmodx = x86.by_name("x86_sdivmodx");
-    let x86_smulx = x86.by_name("x86_smulx");
-    let x86_udivmodx = x86.by_name("x86_udivmodx");
-    let x86_umulx = x86.by_name("x86_umulx");
 
     // Shorthands for recipes.
-    let rec_adjustsp = r.template("adjustsp");
-    let rec_adjustsp_ib = r.template("adjustsp_ib");
-    let rec_adjustsp_id = r.template("adjustsp_id");
-    let rec_allones_fnaddr4 = r.template("allones_fnaddr4");
-    let rec_allones_fnaddr8 = r.template("allones_fnaddr8");
-    let rec_brfb = r.template("brfb");
-    let rec_brfd = r.template("brfd");
-    let rec_brib = r.template("brib");
-    let rec_brid = r.template("brid");
-    let rec_bsf_and_bsr = r.template("bsf_and_bsr");
-    let rec_call_id = r.template("call_id");
-    let rec_call_plt_id = r.template("call_plt_id");
-    let rec_call_r = r.template("call_r");
-    let rec_cmov = r.template("cmov");
     let rec_copysp = r.template("copysp");
-    let rec_div = r.template("div");
-    let rec_debugtrap = r.recipe("debugtrap");
-    let rec_f_ib = r.template("f_ib");
-    let rec_f32imm_z = r.template("f32imm_z");
-    let rec_f64imm_z = r.template("f64imm_z");
-    let rec_fa = r.template("fa");
-    let rec_fax = r.template("fax");
-    let rec_fa_ib = r.template("fa_ib");
-    let rec_fcmp = r.template("fcmp");
-    let rec_fcscc = r.template("fcscc");
-    let rec_ffillnull = r.recipe("ffillnull");
-    let rec_ffillSib32 = r.template("ffillSib32");
-    let rec_fillnull = r.recipe("fillnull");
-    let rec_fillSib32 = r.template("fillSib32");
-    let rec_fld = r.template("fld");
-    let rec_fldDisp32 = r.template("fldDisp32");
-    let rec_fldDisp8 = r.template("fldDisp8");
-    let rec_fldWithIndex = r.template("fldWithIndex");
-    let rec_fldWithIndexDisp32 = r.template("fldWithIndexDisp32");
-    let rec_fldWithIndexDisp8 = r.template("fldWithIndexDisp8");
-    let rec_fnaddr4 = r.template("fnaddr4");
-    let rec_fnaddr8 = r.template("fnaddr8");
-    let rec_fregfill32 = r.template("fregfill32");
-    let rec_fregspill32 = r.template("fregspill32");
-    let rec_frmov = r.template("frmov");
-    let rec_frurm = r.template("frurm");
-    let rec_fspillSib32 = r.template("fspillSib32");
-    let rec_fst = r.template("fst");
-    let rec_fstDisp32 = r.template("fstDisp32");
-    let rec_fstDisp8 = r.template("fstDisp8");
-    let rec_fstWithIndex = r.template("fstWithIndex");
-    let rec_fstWithIndexDisp32 = r.template("fstWithIndexDisp32");
-    let rec_fstWithIndexDisp8 = r.template("fstWithIndexDisp8");
-    let rec_furm = r.template("furm");
     let rec_furm_reg_to_ssa = r.template("furm_reg_to_ssa");
-    let rec_furmi_rnd = r.template("furmi_rnd");
     let rec_get_pinned_reg = r.recipe("get_pinned_reg");
-    let rec_got_fnaddr8 = r.template("got_fnaddr8");
-    let rec_got_gvaddr8 = r.template("got_gvaddr8");
-    let rec_gvaddr4 = r.template("gvaddr4");
-    let rec_gvaddr8 = r.template("gvaddr8");
-    let rec_icscc = r.template("icscc");
-    let rec_icscc_fpr = r.template("icscc_fpr");
-    let rec_icscc_ib = r.template("icscc_ib");
-    let rec_icscc_id = r.template("icscc_id");
-    let rec_indirect_jmp = r.template("indirect_jmp");
-    let rec_is_zero = r.template("is_zero");
-    let rec_jmpb = r.template("jmpb");
-    let rec_jmpd = r.template("jmpd");
-    let rec_jt_base = r.template("jt_base");
-    let rec_jt_entry = r.template("jt_entry");
-    let rec_ld = r.template("ld");
-    let rec_ldDisp32 = r.template("ldDisp32");
-    let rec_ldDisp8 = r.template("ldDisp8");
-    let rec_ldWithIndex = r.template("ldWithIndex");
-    let rec_ldWithIndexDisp32 = r.template("ldWithIndexDisp32");
-    let rec_ldWithIndexDisp8 = r.template("ldWithIndexDisp8");
-    let rec_mulx = r.template("mulx");
     let rec_null = r.recipe("null");
-    let rec_null_fpr = r.recipe("null_fpr");
-    let rec_pcrel_fnaddr8 = r.template("pcrel_fnaddr8");
-    let rec_pcrel_gvaddr8 = r.template("pcrel_gvaddr8");
-    let rec_pfcmp = r.template("pfcmp");
-    let rec_popq = r.template("popq");
     let rec_pu_id = r.template("pu_id");
     let rec_pu_id_bool = r.template("pu_id_bool");
-    let rec_pu_id_ref = r.template("pu_id_ref");
     let rec_pu_iq = r.template("pu_iq");
-    let rec_pushq = r.template("pushq");
-    let rec_ret = r.template("ret");
-    let rec_r_ib = r.template("r_ib");
-    let rec_r_ib_unsigned_gpr = r.template("r_ib_unsigned_gpr");
-    let rec_r_ib_unsigned_fpr = r.template("r_ib_unsigned_fpr");
-    let rec_r_ib_unsigned_r = r.template("r_ib_unsigned_r");
-    let rec_r_id = r.template("r_id");
-    let rec_rcmp = r.template("rcmp");
-    let rec_rcmp_ib = r.template("rcmp_ib");
-    let rec_rcmp_id = r.template("rcmp_id");
-    let rec_rcmp_sp = r.template("rcmp_sp");
-    let rec_regfill32 = r.template("regfill32");
-    let rec_regspill32 = r.template("regspill32");
-    let rec_rc = r.template("rc");
-    let rec_rfumr = r.template("rfumr");
-    let rec_rfurm = r.template("rfurm");
     let rec_rmov = r.template("rmov");
-    let rec_rr = r.template("rr");
-    let rec_rout = r.template("rout");
-    let rec_rin = r.template("rin");
-    let rec_rio = r.template("rio");
-    let rec_rrx = r.template("rrx");
-    let rec_safepoint = r.recipe("safepoint");
-    let rec_setf_abcd = r.template("setf_abcd");
-    let rec_seti_abcd = r.template("seti_abcd");
     let rec_set_pinned_reg = r.template("set_pinned_reg");
-    let rec_spaddr4_id = r.template("spaddr4_id");
-    let rec_spaddr8_id = r.template("spaddr8_id");
-    let rec_spillSib32 = r.template("spillSib32");
-    let rec_st = r.template("st");
-    let rec_stacknull = r.recipe("stacknull");
-    let rec_stDisp32 = r.template("stDisp32");
-    let rec_stDisp32_abcd = r.template("stDisp32_abcd");
-    let rec_stDisp8 = r.template("stDisp8");
-    let rec_stDisp8_abcd = r.template("stDisp8_abcd");
-    let rec_stWithIndex = r.template("stWithIndex");
-    let rec_stWithIndexDisp32 = r.template("stWithIndexDisp32");
-    let rec_stWithIndexDisp32_abcd = r.template("stWithIndexDisp32_abcd");
-    let rec_stWithIndexDisp8 = r.template("stWithIndexDisp8");
-    let rec_stWithIndexDisp8_abcd = r.template("stWithIndexDisp8_abcd");
-    let rec_stWithIndex_abcd = r.template("stWithIndex_abcd");
-    let rec_st_abcd = r.template("st_abcd");
-    let rec_t8jccb_abcd = r.template("t8jccb_abcd");
-    let rec_t8jccd_abcd = r.template("t8jccd_abcd");
-    let rec_t8jccd_long = r.template("t8jccd_long");
-    let rec_tjccb = r.template("tjccb");
-    let rec_tjccd = r.template("tjccd");
-    let rec_trap = r.template("trap");
-    let rec_trapif = r.recipe("trapif");
-    let rec_trapff = r.recipe("trapff");
     let rec_u_id = r.template("u_id");
     let rec_u_id_z = r.template("u_id_z");
     let rec_umr = r.template("umr");
     let rec_umr_reg_to_ssa = r.template("umr_reg_to_ssa");
-    let rec_ur = r.template("ur");
-    let rec_urm = r.template("urm");
     let rec_urm_noflags = r.template("urm_noflags");
     let rec_urm_noflags_abcd = r.template("urm_noflags_abcd");
-    let rec_vconst = r.template("vconst");
-    let rec_vconst_optimized = r.template("vconst_optimized");
-
-    // Predicates shorthands.
-    let all_ones_funcaddrs_and_not_is_pic =
-        settings.predicate_by_name("all_ones_funcaddrs_and_not_is_pic");
-    let is_pic = settings.predicate_by_name("is_pic");
-    let not_all_ones_funcaddrs_and_not_is_pic =
-        settings.predicate_by_name("not_all_ones_funcaddrs_and_not_is_pic");
-    let not_is_pic = settings.predicate_by_name("not_is_pic");
-    let use_popcnt = settings.predicate_by_name("use_popcnt");
-    let use_lzcnt = settings.predicate_by_name("use_lzcnt");
-    let use_bmi1 = settings.predicate_by_name("use_bmi1");
-    let use_sse41 = settings.predicate_by_name("use_sse41");
-    let use_ssse3_simd = settings.predicate_by_name("use_ssse3_simd");
-    let use_sse41_simd = settings.predicate_by_name("use_sse41_simd");
-    let use_sse42_simd = settings.predicate_by_name("use_sse42_simd");
-
-    // Definitions.
-    let mut e = PerCpuModeEncodings::new();
 
     // The pinned reg is fixed to a certain value entirely user-controlled, so it generates nothing!
     e.enc64_rec(get_pinned_reg.bind(I64), rec_get_pinned_reg, 0);
@@ -732,41 +437,6 @@ pub(crate) fn define(
         rec_set_pinned_reg.opcodes(&MOV_STORE).rex().w(),
     );
 
-    e.enc_i32_i64(iadd, rec_rr.opcodes(&ADD));
-    e.enc_i32_i64(iadd_ifcout, rec_rout.opcodes(&ADD));
-    e.enc_i32_i64(iadd_ifcin, rec_rin.opcodes(&ADC));
-    e.enc_i32_i64(iadd_ifcarry, rec_rio.opcodes(&ADC));
-
-    e.enc_i32_i64(isub, rec_rr.opcodes(&SUB));
-    e.enc_i32_i64(isub_ifbout, rec_rout.opcodes(&SUB));
-    e.enc_i32_i64(isub_ifbin, rec_rin.opcodes(&SBB));
-    e.enc_i32_i64(isub_ifborrow, rec_rio.opcodes(&SBB));
-
-    e.enc_i32_i64(band, rec_rr.opcodes(&AND));
-    e.enc_b32_b64(band, rec_rr.opcodes(&AND));
-    e.enc_i32_i64(bor, rec_rr.opcodes(&OR));
-    e.enc_b32_b64(bor, rec_rr.opcodes(&OR));
-    e.enc_i32_i64(bxor, rec_rr.opcodes(&XOR));
-    e.enc_b32_b64(bxor, rec_rr.opcodes(&XOR));
-
-    // x86 has a bitwise not instruction NOT.
-    e.enc_i32_i64(bnot, rec_ur.opcodes(&NOT).rrr(2));
-    e.enc_b32_b64(bnot, rec_ur.opcodes(&NOT).rrr(2));
-
-    // Also add a `b1` encodings for the logic instructions.
-    // TODO: Should this be done with 8-bit instructions? It would improve partial register
-    // dependencies.
-    e.enc_both(band.bind(B1), rec_rr.opcodes(&AND));
-    e.enc_both(bor.bind(B1), rec_rr.opcodes(&OR));
-    e.enc_both(bxor.bind(B1), rec_rr.opcodes(&XOR));
-
-    e.enc_i32_i64(imul, rec_rrx.opcodes(&IMUL));
-    e.enc_i32_i64(x86_sdivmodx, rec_div.opcodes(&IDIV).rrr(7));
-    e.enc_i32_i64(x86_udivmodx, rec_div.opcodes(&DIV).rrr(6));
-
-    e.enc_i32_i64(x86_smulx, rec_mulx.opcodes(&IMUL_RDX_RAX).rrr(5));
-    e.enc_i32_i64(x86_umulx, rec_mulx.opcodes(&MUL).rrr(4));
-
     e.enc_i32_i64(copy, rec_umr.opcodes(&MOV_STORE));
     e.enc_r32_r64_rex_only(copy, rec_umr.opcodes(&MOV_STORE));
     e.enc_both(copy.bind(B1), rec_umr.opcodes(&MOV_STORE));
@@ -790,21 +460,6 @@ pub(crate) fn define(
     e.enc64(regmove.bind(R32), rec_rmov.opcodes(&MOV_STORE).rex());
     e.enc64(regmove.bind(R64), rec_rmov.opcodes(&MOV_STORE).rex().w());
 
-    e.enc_i32_i64(iadd_imm, rec_r_ib.opcodes(&ADD_IMM8_SIGN_EXTEND).rrr(0));
-    e.enc_i32_i64(iadd_imm, rec_r_id.opcodes(&ADD_IMM).rrr(0));
-
-    e.enc_i32_i64(band_imm, rec_r_ib.opcodes(&AND_IMM8_SIGN_EXTEND).rrr(4));
-    e.enc_i32_i64(band_imm, rec_r_id.opcodes(&AND_IMM).rrr(4));
-
-    e.enc_i32_i64(bor_imm, rec_r_ib.opcodes(&OR_IMM8_SIGN_EXTEND).rrr(1));
-    e.enc_i32_i64(bor_imm, rec_r_id.opcodes(&OR_IMM).rrr(1));
-
-    e.enc_i32_i64(bxor_imm, rec_r_ib.opcodes(&XOR_IMM8_SIGN_EXTEND).rrr(6));
-    e.enc_i32_i64(bxor_imm, rec_r_id.opcodes(&XOR_IMM).rrr(6));
-
-    // TODO: band_imm.i64 with an unsigned 32-bit immediate can be encoded as band_imm.i32. Can
-    // even use the single-byte immediate for 0xffff_ffXX masks.
-
     // Immediate constants.
     e.enc32(iconst.bind(I32), rec_pu_id.opcodes(&MOV_IMM));
 
@@ -845,6 +500,7 @@ pub(crate) fn define(
         rec_u_id_z.opcodes(&XORB),
         is_zero_int.clone(),
     );
+
     // You may expect that i16 encodings would have an 0x66 prefix on the opcode to indicate that
     // encodings should be on 16-bit operands (f.ex, "xor %ax, %ax"). Cranelift currently does not
     // know that it can drop the 0x66 prefix and clear the upper half of a 32-bit register in these
@@ -864,58 +520,252 @@ pub(crate) fn define(
     );
     e.enc_x86_64_instp(iconst.bind(I64), rec_u_id_z.opcodes(&XOR), is_zero_int);
 
-    // Shifts and rotates.
-    // Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit
-    // and 16-bit shifts would need explicit masking.
+    // Numerical conversions.
 
-    for &(inst, rrr) in &[(rotl, 0), (rotr, 1), (ishl, 4), (ushr, 5), (sshr, 7)] {
-        // Cannot use enc_i32_i64 for this pattern because instructions require
-        // to bind any.
-        e.enc32(
-            inst.bind(I32).bind(Any),
-            rec_rc.opcodes(&ROTATE_CL).rrr(rrr),
-        );
-        e.enc64(
-            inst.bind(I64).bind(Any),
-            rec_rc.opcodes(&ROTATE_CL).rrr(rrr).rex().w(),
-        );
-        e.enc64(
-            inst.bind(I32).bind(Any),
-            rec_rc.opcodes(&ROTATE_CL).rrr(rrr).rex(),
-        );
-        e.enc64(
-            inst.bind(I32).bind(Any),
-            rec_rc.opcodes(&ROTATE_CL).rrr(rrr),
-        );
+    // Reducing an integer is a no-op.
+    e.enc32_rec(ireduce.bind(I8).bind(I16), rec_null, 0);
+    e.enc32_rec(ireduce.bind(I8).bind(I32), rec_null, 0);
+    e.enc32_rec(ireduce.bind(I16).bind(I32), rec_null, 0);
+
+    e.enc64_rec(ireduce.bind(I8).bind(I16), rec_null, 0);
+    e.enc64_rec(ireduce.bind(I8).bind(I32), rec_null, 0);
+    e.enc64_rec(ireduce.bind(I16).bind(I32), rec_null, 0);
+    e.enc64_rec(ireduce.bind(I8).bind(I64), rec_null, 0);
+    e.enc64_rec(ireduce.bind(I16).bind(I64), rec_null, 0);
+    e.enc64_rec(ireduce.bind(I32).bind(I64), rec_null, 0);
+
+    // TODO: Add encodings for cbw, cwde, cdqe, which are sign-extending
+    // instructions for %al/%ax/%eax to %ax/%eax/%rax.
+
+    // movsbl
+    e.enc32(
+        sextend.bind(I32).bind(I8),
+        rec_urm_noflags_abcd.opcodes(&MOVSX_BYTE),
+    );
+    e.enc64(
+        sextend.bind(I32).bind(I8),
+        rec_urm_noflags.opcodes(&MOVSX_BYTE).rex(),
+    );
+    e.enc64(
+        sextend.bind(I32).bind(I8),
+        rec_urm_noflags_abcd.opcodes(&MOVSX_BYTE),
+    );
+
+    // movswl
+    e.enc32(
+        sextend.bind(I32).bind(I16),
+        rec_urm_noflags.opcodes(&MOVSX_WORD),
+    );
+    e.enc64(
+        sextend.bind(I32).bind(I16),
+        rec_urm_noflags.opcodes(&MOVSX_WORD).rex(),
+    );
+    e.enc64(
+        sextend.bind(I32).bind(I16),
+        rec_urm_noflags.opcodes(&MOVSX_WORD),
+    );
+
+    // movsbq
+    e.enc64(
+        sextend.bind(I64).bind(I8),
+        rec_urm_noflags.opcodes(&MOVSX_BYTE).rex().w(),
+    );
+
+    // movswq
+    e.enc64(
+        sextend.bind(I64).bind(I16),
+        rec_urm_noflags.opcodes(&MOVSX_WORD).rex().w(),
+    );
+
+    // movslq
+    e.enc64(
+        sextend.bind(I64).bind(I32),
+        rec_urm_noflags.opcodes(&MOVSXD).rex().w(),
+    );
+
+    // movzbl
+    e.enc32(
+        uextend.bind(I32).bind(I8),
+        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
+    );
+    e.enc64(
+        uextend.bind(I32).bind(I8),
+        rec_urm_noflags.opcodes(&MOVZX_BYTE).rex(),
+    );
+    e.enc64(
+        uextend.bind(I32).bind(I8),
+        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
+    );
+
+    // movzwl
+    e.enc32(
+        uextend.bind(I32).bind(I16),
+        rec_urm_noflags.opcodes(&MOVZX_WORD),
+    );
+    e.enc64(
+        uextend.bind(I32).bind(I16),
+        rec_urm_noflags.opcodes(&MOVZX_WORD).rex(),
+    );
+    e.enc64(
+        uextend.bind(I32).bind(I16),
+        rec_urm_noflags.opcodes(&MOVZX_WORD),
+    );
+
+    // movzbq, encoded as movzbl because it's equivalent and shorter.
+    e.enc64(
+        uextend.bind(I64).bind(I8),
+        rec_urm_noflags.opcodes(&MOVZX_BYTE).rex(),
+    );
+    e.enc64(
+        uextend.bind(I64).bind(I8),
+        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
+    );
+
+    // movzwq, encoded as movzwl because it's equivalent and shorter
+    e.enc64(
+        uextend.bind(I64).bind(I16),
+        rec_urm_noflags.opcodes(&MOVZX_WORD).rex(),
+    );
+    e.enc64(
+        uextend.bind(I64).bind(I16),
+        rec_urm_noflags.opcodes(&MOVZX_WORD),
+    );
+
+    // A 32-bit register copy clears the high 32 bits.
+    e.enc64(
+        uextend.bind(I64).bind(I32),
+        rec_umr.opcodes(&MOV_STORE).rex(),
+    );
+    e.enc64(uextend.bind(I64).bind(I32), rec_umr.opcodes(&MOV_STORE));
+
+    // Convert bool to int.
+    //
+    // This assumes that b1 is represented as an 8-bit low register with the value 0
+    // or 1.
+    //
+    // Encode movzbq as movzbl, because it's equivalent and shorter.
+    for &to in &[I8, I16, I32, I64] {
+        for &from in &[B1, B8] {
+            e.enc64(
+                bint.bind(to).bind(from),
+                rec_urm_noflags.opcodes(&MOVZX_BYTE).rex(),
+            );
+            e.enc64(
+                bint.bind(to).bind(from),
+                rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
+            );
+            if to != I64 {
+                e.enc32(
+                    bint.bind(to).bind(from),
+                    rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
+                );
+            }
+        }
     }
 
-    e.enc_i32_i64(rotl_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(0));
-    e.enc_i32_i64(rotr_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(1));
-    e.enc_i32_i64(ishl_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(4));
-    e.enc_i32_i64(ushr_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(5));
-    e.enc_i32_i64(sshr_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(7));
+    // Copy Special
+    // For x86-64, only define REX forms for now, since we can't describe the
+    // special regunit immediate operands with the current constraint language.
+    e.enc64(copy_special, rec_copysp.opcodes(&MOV_STORE).rex().w());
+    e.enc32(copy_special, rec_copysp.opcodes(&MOV_STORE));
 
-    // Population count.
-    e.enc32_isap(popcnt.bind(I32), rec_urm.opcodes(&POPCNT), use_popcnt);
-    e.enc64_isap(
-        popcnt.bind(I64),
-        rec_urm.opcodes(&POPCNT).rex().w(),
-        use_popcnt,
+    // Copy to SSA.  These have to be done with special _rex_only encoders, because the standard
+    // machinery for deciding whether a REX.{RXB} prefix is needed doesn't take into account
+    // the source register, which is specified directly in the instruction.
+    e.enc_i32_i64_rex_only(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(&MOV_STORE));
+    e.enc_r32_r64_rex_only(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(&MOV_STORE));
+    e.enc_both_rex_only(copy_to_ssa.bind(B1), rec_umr_reg_to_ssa.opcodes(&MOV_STORE));
+    e.enc_both_rex_only(copy_to_ssa.bind(I8), rec_umr_reg_to_ssa.opcodes(&MOV_STORE));
+    e.enc_both_rex_only(
+        copy_to_ssa.bind(I16),
+        rec_umr_reg_to_ssa.opcodes(&MOV_STORE),
     );
-    e.enc64_isap(popcnt.bind(I32), rec_urm.opcodes(&POPCNT).rex(), use_popcnt);
-    e.enc64_isap(popcnt.bind(I32), rec_urm.opcodes(&POPCNT), use_popcnt);
+    e.enc_both_rex_only(
+        copy_to_ssa.bind(F64),
+        rec_furm_reg_to_ssa.opcodes(&MOVSD_LOAD),
+    );
+    e.enc_both_rex_only(
+        copy_to_ssa.bind(F32),
+        rec_furm_reg_to_ssa.opcodes(&MOVSS_LOAD),
+    );
+}
 
-    // Count leading zero bits.
-    e.enc32_isap(clz.bind(I32), rec_urm.opcodes(&LZCNT), use_lzcnt);
-    e.enc64_isap(clz.bind(I64), rec_urm.opcodes(&LZCNT).rex().w(), use_lzcnt);
-    e.enc64_isap(clz.bind(I32), rec_urm.opcodes(&LZCNT).rex(), use_lzcnt);
-    e.enc64_isap(clz.bind(I32), rec_urm.opcodes(&LZCNT), use_lzcnt);
+#[inline(never)]
+fn define_memory(
+    e: &mut PerCpuModeEncodings,
+    shared_defs: &SharedDefinitions,
+    x86: &InstructionGroup,
+    r: &RecipeGroup,
+) {
+    let shared = &shared_defs.instructions;
+    let formats = &shared_defs.formats;
 
-    // Count trailing zero bits.
-    e.enc32_isap(ctz.bind(I32), rec_urm.opcodes(&TZCNT), use_bmi1);
-    e.enc64_isap(ctz.bind(I64), rec_urm.opcodes(&TZCNT).rex().w(), use_bmi1);
-    e.enc64_isap(ctz.bind(I32), rec_urm.opcodes(&TZCNT).rex(), use_bmi1);
-    e.enc64_isap(ctz.bind(I32), rec_urm.opcodes(&TZCNT), use_bmi1);
+    // Shorthands for instructions.
+    let adjust_sp_down = shared.by_name("adjust_sp_down");
+    let adjust_sp_down_imm = shared.by_name("adjust_sp_down_imm");
+    let adjust_sp_up_imm = shared.by_name("adjust_sp_up_imm");
+    let copy_nop = shared.by_name("copy_nop");
+    let fill = shared.by_name("fill");
+    let fill_nop = shared.by_name("fill_nop");
+    let istore16 = shared.by_name("istore16");
+    let istore16_complex = shared.by_name("istore16_complex");
+    let istore32 = shared.by_name("istore32");
+    let istore32_complex = shared.by_name("istore32_complex");
+    let istore8 = shared.by_name("istore8");
+    let istore8_complex = shared.by_name("istore8_complex");
+    let load = shared.by_name("load");
+    let load_complex = shared.by_name("load_complex");
+    let regfill = shared.by_name("regfill");
+    let regspill = shared.by_name("regspill");
+    let sload16 = shared.by_name("sload16");
+    let sload16_complex = shared.by_name("sload16_complex");
+    let sload32 = shared.by_name("sload32");
+    let sload32_complex = shared.by_name("sload32_complex");
+    let sload8 = shared.by_name("sload8");
+    let sload8_complex = shared.by_name("sload8_complex");
+    let spill = shared.by_name("spill");
+    let store = shared.by_name("store");
+    let store_complex = shared.by_name("store_complex");
+    let uload16 = shared.by_name("uload16");
+    let uload16_complex = shared.by_name("uload16_complex");
+    let uload32 = shared.by_name("uload32");
+    let uload32_complex = shared.by_name("uload32_complex");
+    let uload8 = shared.by_name("uload8");
+    let uload8_complex = shared.by_name("uload8_complex");
+    let x86_pop = x86.by_name("x86_pop");
+    let x86_push = x86.by_name("x86_push");
+
+    // Shorthands for recipes.
+    let rec_adjustsp = r.template("adjustsp");
+    let rec_adjustsp_ib = r.template("adjustsp_ib");
+    let rec_adjustsp_id = r.template("adjustsp_id");
+    let rec_ffillnull = r.recipe("ffillnull");
+    let rec_fillnull = r.recipe("fillnull");
+    let rec_fillSib32 = r.template("fillSib32");
+    let rec_ld = r.template("ld");
+    let rec_ldDisp32 = r.template("ldDisp32");
+    let rec_ldDisp8 = r.template("ldDisp8");
+    let rec_ldWithIndex = r.template("ldWithIndex");
+    let rec_ldWithIndexDisp32 = r.template("ldWithIndexDisp32");
+    let rec_ldWithIndexDisp8 = r.template("ldWithIndexDisp8");
+    let rec_popq = r.template("popq");
+    let rec_pushq = r.template("pushq");
+    let rec_regfill32 = r.template("regfill32");
+    let rec_regspill32 = r.template("regspill32");
+    let rec_spillSib32 = r.template("spillSib32");
+    let rec_st = r.template("st");
+    let rec_stacknull = r.recipe("stacknull");
+    let rec_stDisp32 = r.template("stDisp32");
+    let rec_stDisp32_abcd = r.template("stDisp32_abcd");
+    let rec_stDisp8 = r.template("stDisp8");
+    let rec_stDisp8_abcd = r.template("stDisp8_abcd");
+    let rec_stWithIndex = r.template("stWithIndex");
+    let rec_stWithIndexDisp32 = r.template("stWithIndexDisp32");
+    let rec_stWithIndexDisp32_abcd = r.template("stWithIndexDisp32_abcd");
+    let rec_stWithIndexDisp8 = r.template("stWithIndexDisp8");
+    let rec_stWithIndexDisp8_abcd = r.template("stWithIndexDisp8_abcd");
+    let rec_stWithIndex_abcd = r.template("stWithIndex_abcd");
+    let rec_st_abcd = r.template("st_abcd");
 
     // Loads and stores.
     let is_load_complex_length_two =
@@ -1080,32 +930,6 @@ pub(crate) fn define(
     e.enc32(x86_pop.bind(I32), rec_popq.opcodes(&POP_REG));
     e.enc_x86_64(x86_pop.bind(I64), rec_popq.opcodes(&POP_REG));
 
-    // Copy Special
-    // For x86-64, only define REX forms for now, since we can't describe the
-    // special regunit immediate operands with the current constraint language.
-    e.enc64(copy_special, rec_copysp.opcodes(&MOV_STORE).rex().w());
-    e.enc32(copy_special, rec_copysp.opcodes(&MOV_STORE));
-
-    // Copy to SSA.  These have to be done with special _rex_only encoders, because the standard
-    // machinery for deciding whether a REX.{RXB} prefix is needed doesn't take into account
-    // the source register, which is specified directly in the instruction.
-    e.enc_i32_i64_rex_only(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(&MOV_STORE));
-    e.enc_r32_r64_rex_only(copy_to_ssa, rec_umr_reg_to_ssa.opcodes(&MOV_STORE));
-    e.enc_both_rex_only(copy_to_ssa.bind(B1), rec_umr_reg_to_ssa.opcodes(&MOV_STORE));
-    e.enc_both_rex_only(copy_to_ssa.bind(I8), rec_umr_reg_to_ssa.opcodes(&MOV_STORE));
-    e.enc_both_rex_only(
-        copy_to_ssa.bind(I16),
-        rec_umr_reg_to_ssa.opcodes(&MOV_STORE),
-    );
-    e.enc_both_rex_only(
-        copy_to_ssa.bind(F64),
-        rec_furm_reg_to_ssa.opcodes(&MOVSD_LOAD),
-    );
-    e.enc_both_rex_only(
-        copy_to_ssa.bind(F32),
-        rec_furm_reg_to_ssa.opcodes(&MOVSS_LOAD),
-    );
-
     // Stack-slot-to-the-same-stack-slot copy, which is guaranteed to turn
     // into a no-op.
     // The same encoding is generated for both the 64- and 32-bit architectures.
@@ -1151,6 +975,94 @@ pub(crate) fn define(
         adjust_sp_down_imm,
         rec_adjustsp_id.opcodes(&CMP_IMM).rrr(5).rex().w(),
     );
+}
+
+#[inline(never)]
+fn define_fpu_moves(e: &mut PerCpuModeEncodings, shared_defs: &SharedDefinitions, r: &RecipeGroup) {
+    let shared = &shared_defs.instructions;
+
+    // Shorthands for instructions.
+    let bitcast = shared.by_name("bitcast");
+    let copy = shared.by_name("copy");
+    let regmove = shared.by_name("regmove");
+
+    // Shorthands for recipes.
+    let rec_frmov = r.template("frmov");
+    let rec_frurm = r.template("frurm");
+    let rec_furm = r.template("furm");
+    let rec_rfumr = r.template("rfumr");
+
+    // Floating-point moves.
+    // movd
+    e.enc_both(
+        bitcast.bind(F32).bind(I32),
+        rec_frurm.opcodes(&MOVD_LOAD_XMM),
+    );
+    e.enc_both(
+        bitcast.bind(I32).bind(F32),
+        rec_rfumr.opcodes(&MOVD_STORE_XMM),
+    );
+
+    // movq
+    e.enc64(
+        bitcast.bind(F64).bind(I64),
+        rec_frurm.opcodes(&MOVD_LOAD_XMM).rex().w(),
+    );
+    e.enc64(
+        bitcast.bind(I64).bind(F64),
+        rec_rfumr.opcodes(&MOVD_STORE_XMM).rex().w(),
+    );
+
+    // movaps
+    e.enc_both(copy.bind(F32), rec_furm.opcodes(&MOVAPS_LOAD));
+    e.enc_both(copy.bind(F64), rec_furm.opcodes(&MOVAPS_LOAD));
+
+    // TODO For x86-64, only define REX forms for now, since we can't describe the special regunit
+    // immediate operands with the current constraint language.
+    e.enc32(regmove.bind(F32), rec_frmov.opcodes(&MOVAPS_LOAD));
+    e.enc64(regmove.bind(F32), rec_frmov.opcodes(&MOVAPS_LOAD).rex());
+
+    // TODO For x86-64, only define REX forms for now, since we can't describe the special regunit
+    // immediate operands with the current constraint language.
+    e.enc32(regmove.bind(F64), rec_frmov.opcodes(&MOVAPS_LOAD));
+    e.enc64(regmove.bind(F64), rec_frmov.opcodes(&MOVAPS_LOAD).rex());
+}
+
+#[inline(never)]
+fn define_fpu_memory(
+    e: &mut PerCpuModeEncodings,
+    shared_defs: &SharedDefinitions,
+    r: &RecipeGroup,
+) {
+    let shared = &shared_defs.instructions;
+
+    // Shorthands for instructions.
+    let fill = shared.by_name("fill");
+    let load = shared.by_name("load");
+    let load_complex = shared.by_name("load_complex");
+    let regfill = shared.by_name("regfill");
+    let regspill = shared.by_name("regspill");
+    let spill = shared.by_name("spill");
+    let store = shared.by_name("store");
+    let store_complex = shared.by_name("store_complex");
+
+    // Shorthands for recipes.
+    let rec_ffillSib32 = r.template("ffillSib32");
+    let rec_fld = r.template("fld");
+    let rec_fldDisp32 = r.template("fldDisp32");
+    let rec_fldDisp8 = r.template("fldDisp8");
+    let rec_fldWithIndex = r.template("fldWithIndex");
+    let rec_fldWithIndexDisp32 = r.template("fldWithIndexDisp32");
+    let rec_fldWithIndexDisp8 = r.template("fldWithIndexDisp8");
+    let rec_fregfill32 = r.template("fregfill32");
+    let rec_fregspill32 = r.template("fregspill32");
+    let rec_fspillSib32 = r.template("fspillSib32");
+    let rec_fst = r.template("fst");
+    let rec_fstDisp32 = r.template("fstDisp32");
+    let rec_fstDisp8 = r.template("fstDisp8");
+    let rec_fstWithIndex = r.template("fstWithIndex");
+    let rec_fstWithIndexDisp32 = r.template("fstWithIndexDisp32");
+    let rec_fstWithIndexDisp8 = r.template("fstWithIndexDisp8");
 
     // Float loads and stores.
     e.enc_both(load.bind(F32).bind(Any), rec_fld.opcodes(&MOVSS_LOAD));
@@ -1242,358 +1154,53 @@ pub(crate) fn define(
     e.enc_both(regspill.bind(F32), rec_fregspill32.opcodes(&MOVSS_STORE));
     e.enc_both(spill.bind(F64), rec_fspillSib32.opcodes(&MOVSD_STORE));
     e.enc_both(regspill.bind(F64), rec_fregspill32.opcodes(&MOVSD_STORE));
+}
 
-    // Function addresses.
+#[inline(never)]
+fn define_fpu_ops(
+    e: &mut PerCpuModeEncodings,
+    shared_defs: &SharedDefinitions,
+    settings: &SettingGroup,
+    x86: &InstructionGroup,
+    r: &RecipeGroup,
+) {
+    let shared = &shared_defs.instructions;
+    let formats = &shared_defs.formats;
 
-    // Non-PIC, all-ones funcaddresses.
-    e.enc32_isap(
-        func_addr.bind(I32),
-        rec_fnaddr4.opcodes(&MOV_IMM),
-        not_all_ones_funcaddrs_and_not_is_pic,
-    );
-    e.enc64_isap(
-        func_addr.bind(I64),
-        rec_fnaddr8.opcodes(&MOV_IMM).rex().w(),
-        not_all_ones_funcaddrs_and_not_is_pic,
-    );
+    // Shorthands for instructions.
+    let ceil = shared.by_name("ceil");
+    let f32const = shared.by_name("f32const");
+    let f64const = shared.by_name("f64const");
+    let fadd = shared.by_name("fadd");
+    let fcmp = shared.by_name("fcmp");
+    let fcvt_from_sint = shared.by_name("fcvt_from_sint");
+    let fdemote = shared.by_name("fdemote");
+    let fdiv = shared.by_name("fdiv");
+    let ffcmp = shared.by_name("ffcmp");
+    let floor = shared.by_name("floor");
+    let fmul = shared.by_name("fmul");
+    let fpromote = shared.by_name("fpromote");
+    let fsub = shared.by_name("fsub");
+    let nearest = shared.by_name("nearest");
+    let sqrt = shared.by_name("sqrt");
+    let trunc = shared.by_name("trunc");
+    let x86_cvtt2si = x86.by_name("x86_cvtt2si");
+    let x86_fmax = x86.by_name("x86_fmax");
+    let x86_fmin = x86.by_name("x86_fmin");
 
-    // Non-PIC, all-zeros funcaddresses.
-    e.enc32_isap(
-        func_addr.bind(I32),
-        rec_allones_fnaddr4.opcodes(&MOV_IMM),
-        all_ones_funcaddrs_and_not_is_pic,
-    );
-    e.enc64_isap(
-        func_addr.bind(I64),
-        rec_allones_fnaddr8.opcodes(&MOV_IMM).rex().w(),
-        all_ones_funcaddrs_and_not_is_pic,
-    );
+    // Shorthands for recipes.
+    let rec_f32imm_z = r.template("f32imm_z");
+    let rec_f64imm_z = r.template("f64imm_z");
+    let rec_fa = r.template("fa");
+    let rec_fcmp = r.template("fcmp");
+    let rec_fcscc = r.template("fcscc");
+    let rec_frurm = r.template("frurm");
+    let rec_furm = r.template("furm");
+    let rec_furmi_rnd = r.template("furmi_rnd");
+    let rec_rfurm = r.template("rfurm");
 
-    // 64-bit, colocated, both PIC and non-PIC. Use the lea instruction's pc-relative field.
-    let is_colocated_func =
-        InstructionPredicate::new_is_colocated_func(&*formats.func_addr, "func_ref");
-    e.enc64_instp(
-        func_addr.bind(I64),
-        rec_pcrel_fnaddr8.opcodes(&LEA).rex().w(),
-        is_colocated_func,
-    );
-
-    // 64-bit, non-colocated, PIC.
-    e.enc64_isap(
-        func_addr.bind(I64),
-        rec_got_fnaddr8.opcodes(&MOV_LOAD).rex().w(),
-        is_pic,
-    );
-
-    // Global addresses.
-
-    // Non-PIC.
-    e.enc32_isap(
-        symbol_value.bind(I32),
-        rec_gvaddr4.opcodes(&MOV_IMM),
-        not_is_pic,
-    );
-    e.enc64_isap(
-        symbol_value.bind(I64),
-        rec_gvaddr8.opcodes(&MOV_IMM).rex().w(),
-        not_is_pic,
-    );
-
-    // PIC, colocated.
-    e.enc64_func(
-        symbol_value.bind(I64),
-        rec_pcrel_gvaddr8.opcodes(&LEA).rex().w(),
-        |encoding| {
-            encoding
-                .isa_predicate(is_pic)
-                .inst_predicate(InstructionPredicate::new_is_colocated_data(formats))
-        },
-    );
-
-    // PIC, non-colocated.
-    e.enc64_isap(
-        symbol_value.bind(I64),
-        rec_got_gvaddr8.opcodes(&MOV_LOAD).rex().w(),
-        is_pic,
-    );
-
-    // Stack addresses.
-    //
-    // TODO: Add encoding rules for stack_load and stack_store, so that they
-    // don't get legalized to stack_addr + load/store.
-    e.enc32(stack_addr.bind(I32), rec_spaddr4_id.opcodes(&LEA));
-    e.enc64(stack_addr.bind(I64), rec_spaddr8_id.opcodes(&LEA).rex().w());
-
-    // Call/return
-
-    // 32-bit, both PIC and non-PIC.
-    e.enc32(call, rec_call_id.opcodes(&CALL_RELATIVE));
-
-    // 64-bit, colocated, both PIC and non-PIC. Use the call instruction's pc-relative field.
-    let is_colocated_func = InstructionPredicate::new_is_colocated_func(&*formats.call, "func_ref");
-    e.enc64_instp(call, rec_call_id.opcodes(&CALL_RELATIVE), is_colocated_func);
-
-    // 64-bit, non-colocated, PIC. There is no 64-bit non-colocated non-PIC version, since non-PIC
-    // is currently using the large model, which requires calls be lowered to
-    // func_addr+call_indirect.
-    e.enc64_isap(call, rec_call_plt_id.opcodes(&CALL_RELATIVE), is_pic);
-
-    e.enc32(
-        call_indirect.bind(I32),
-        rec_call_r.opcodes(&JUMP_ABSOLUTE).rrr(2),
-    );
-    e.enc64(
-        call_indirect.bind(I64),
-        rec_call_r.opcodes(&JUMP_ABSOLUTE).rrr(2).rex(),
-    );
-    e.enc64(
-        call_indirect.bind(I64),
-        rec_call_r.opcodes(&JUMP_ABSOLUTE).rrr(2),
-    );
-
-    e.enc32(return_, rec_ret.opcodes(&RET_NEAR));
-    e.enc64(return_, rec_ret.opcodes(&RET_NEAR));
-
-    // Branches.
-    e.enc32(jump, rec_jmpb.opcodes(&JUMP_SHORT));
-    e.enc64(jump, rec_jmpb.opcodes(&JUMP_SHORT));
-    e.enc32(jump, rec_jmpd.opcodes(&JUMP_NEAR_RELATIVE));
-    e.enc64(jump, rec_jmpd.opcodes(&JUMP_NEAR_RELATIVE));
-
-    e.enc_both(brif, rec_brib.opcodes(&JUMP_SHORT_IF_OVERFLOW));
-    e.enc_both(brif, rec_brid.opcodes(&JUMP_NEAR_IF_OVERFLOW));
-
-    // Not all float condition codes are legal, see `supported_floatccs`.
-    e.enc_both(brff, rec_brfb.opcodes(&JUMP_SHORT_IF_OVERFLOW));
-    e.enc_both(brff, rec_brfd.opcodes(&JUMP_NEAR_IF_OVERFLOW));
-
-    // Note that the tjccd opcode will be prefixed with 0x0f.
-    e.enc_i32_i64_explicit_rex(brz, rec_tjccb.opcodes(&JUMP_SHORT_IF_EQUAL));
-    e.enc_i32_i64_explicit_rex(brz, rec_tjccd.opcodes(&TEST_BYTE_REG));
-    e.enc_i32_i64_explicit_rex(brnz, rec_tjccb.opcodes(&JUMP_SHORT_IF_NOT_EQUAL));
-    e.enc_i32_i64_explicit_rex(brnz, rec_tjccd.opcodes(&TEST_REG));
-
-    // Branch on a b1 value in a register only looks at the low 8 bits. See also
-    // bint encodings below.
-    //
-    // Start with the worst-case encoding for X86_32 only. The register allocator
-    // can't handle a branch with an ABCD-constrained operand.
-    e.enc32(brz.bind(B1), rec_t8jccd_long.opcodes(&TEST_BYTE_REG));
-    e.enc32(brnz.bind(B1), rec_t8jccd_long.opcodes(&TEST_REG));
-
-    e.enc_both(brz.bind(B1), rec_t8jccb_abcd.opcodes(&JUMP_SHORT_IF_EQUAL));
-    e.enc_both(brz.bind(B1), rec_t8jccd_abcd.opcodes(&TEST_BYTE_REG));
-    e.enc_both(
-        brnz.bind(B1),
-        rec_t8jccb_abcd.opcodes(&JUMP_SHORT_IF_NOT_EQUAL),
-    );
-    e.enc_both(brnz.bind(B1), rec_t8jccd_abcd.opcodes(&TEST_REG));
-
-    // Jump tables.
-    e.enc64(
-        jump_table_entry.bind(I64),
-        rec_jt_entry.opcodes(&MOVSXD).rex().w(),
-    );
-    e.enc32(jump_table_entry.bind(I32), rec_jt_entry.opcodes(&MOV_LOAD));
-
-    e.enc64(
-        jump_table_base.bind(I64),
-        rec_jt_base.opcodes(&LEA).rex().w(),
-    );
-    e.enc32(jump_table_base.bind(I32), rec_jt_base.opcodes(&LEA));
-
-    e.enc_x86_64(
-        indirect_jump_table_br.bind(I64),
-        rec_indirect_jmp.opcodes(&JUMP_ABSOLUTE).rrr(4),
-    );
-    e.enc32(
-        indirect_jump_table_br.bind(I32),
-        rec_indirect_jmp.opcodes(&JUMP_ABSOLUTE).rrr(4),
-    );
-
-    // Trap as ud2
-    e.enc32(trap, rec_trap.opcodes(&UNDEFINED2));
-    e.enc64(trap, rec_trap.opcodes(&UNDEFINED2));
-    e.enc32(resumable_trap, rec_trap.opcodes(&UNDEFINED2));
-    e.enc64(resumable_trap, rec_trap.opcodes(&UNDEFINED2));
-
-    // Debug trap as int3
-    e.enc32_rec(debugtrap, rec_debugtrap, 0);
-    e.enc64_rec(debugtrap, rec_debugtrap, 0);
-
-    e.enc32_rec(trapif, rec_trapif, 0);
-    e.enc64_rec(trapif, rec_trapif, 0);
-    e.enc32_rec(trapff, rec_trapff, 0);
-    e.enc64_rec(trapff, rec_trapff, 0);
-
-    // Comparisons
-    e.enc_i32_i64(icmp, rec_icscc.opcodes(&CMP_REG));
-    e.enc_i32_i64(icmp_imm, rec_icscc_ib.opcodes(&CMP_IMM8).rrr(7));
-    e.enc_i32_i64(icmp_imm, rec_icscc_id.opcodes(&CMP_IMM).rrr(7));
-    e.enc_i32_i64(ifcmp, rec_rcmp.opcodes(&CMP_REG));
-    e.enc_i32_i64(ifcmp_imm, rec_rcmp_ib.opcodes(&CMP_IMM8).rrr(7));
-    e.enc_i32_i64(ifcmp_imm, rec_rcmp_id.opcodes(&CMP_IMM).rrr(7));
-    // TODO: We could special-case ifcmp_imm(x, 0) to TEST(x, x).
-
-    e.enc32(ifcmp_sp.bind(I32), rec_rcmp_sp.opcodes(&CMP_REG));
-    e.enc64(ifcmp_sp.bind(I64), rec_rcmp_sp.opcodes(&CMP_REG).rex().w());
-
-    // Convert flags to bool.
-    // This encodes `b1` as an 8-bit low register with the value 0 or 1.
-    e.enc_both(trueif, rec_seti_abcd.opcodes(&SET_BYTE_IF_OVERFLOW));
-    e.enc_both(trueff, rec_setf_abcd.opcodes(&SET_BYTE_IF_OVERFLOW));
-
-    // Conditional move (a.k.a integer select).
-    e.enc_i32_i64(selectif, rec_cmov.opcodes(&CMOV_OVERFLOW));
-
-    // Bit scan forwards and reverse
-    e.enc_i32_i64(x86_bsf, rec_bsf_and_bsr.opcodes(&BIT_SCAN_FORWARD));
-    e.enc_i32_i64(x86_bsr, rec_bsf_and_bsr.opcodes(&BIT_SCAN_REVERSE));
-
-    // Convert bool to int.
-    //
-    // This assumes that b1 is represented as an 8-bit low register with the value 0
-    // or 1.
-    //
-    // Encode movzbq as movzbl, because it's equivalent and shorter.
-    for &to in &[I8, I16, I32, I64] {
-        for &from in &[B1, B8] {
-            e.enc64(
-                bint.bind(to).bind(from),
-                rec_urm_noflags.opcodes(&MOVZX_BYTE).rex(),
-            );
-            e.enc64(
-                bint.bind(to).bind(from),
-                rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
-            );
-            if to != I64 {
-                e.enc32(
-                    bint.bind(to).bind(from),
-                    rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
-                );
-            }
-        }
-    }
-
-    // Numerical conversions.
-
-    // Reducing an integer is a no-op.
-    e.enc32_rec(ireduce.bind(I8).bind(I16), rec_null, 0);
-    e.enc32_rec(ireduce.bind(I8).bind(I32), rec_null, 0);
-    e.enc32_rec(ireduce.bind(I16).bind(I32), rec_null, 0);
-
-    e.enc64_rec(ireduce.bind(I8).bind(I16), rec_null, 0);
-    e.enc64_rec(ireduce.bind(I8).bind(I32), rec_null, 0);
-    e.enc64_rec(ireduce.bind(I16).bind(I32), rec_null, 0);
-    e.enc64_rec(ireduce.bind(I8).bind(I64), rec_null, 0);
-    e.enc64_rec(ireduce.bind(I16).bind(I64), rec_null, 0);
-    e.enc64_rec(ireduce.bind(I32).bind(I64), rec_null, 0);
-
-    // TODO: Add encodings for cbw, cwde, cdqe, which are sign-extending
-    // instructions for %al/%ax/%eax to %ax/%eax/%rax.
-
-    // movsbl
-    e.enc32(
-        sextend.bind(I32).bind(I8),
-        rec_urm_noflags_abcd.opcodes(&MOVSX_BYTE),
-    );
-    e.enc64(
-        sextend.bind(I32).bind(I8),
-        rec_urm_noflags.opcodes(&MOVSX_BYTE).rex(),
-    );
-    e.enc64(
-        sextend.bind(I32).bind(I8),
-        rec_urm_noflags_abcd.opcodes(&MOVSX_BYTE),
-    );
-
-    // movswl
-    e.enc32(
-        sextend.bind(I32).bind(I16),
-        rec_urm_noflags.opcodes(&MOVSX_WORD),
-    );
-    e.enc64(
-        sextend.bind(I32).bind(I16),
-        rec_urm_noflags.opcodes(&MOVSX_WORD).rex(),
-    );
-    e.enc64(
-        sextend.bind(I32).bind(I16),
-        rec_urm_noflags.opcodes(&MOVSX_WORD),
-    );
-
-    // movsbq
-    e.enc64(
-        sextend.bind(I64).bind(I8),
-        rec_urm_noflags.opcodes(&MOVSX_BYTE).rex().w(),
-    );
-
-    // movswq
-    e.enc64(
-        sextend.bind(I64).bind(I16),
-        rec_urm_noflags.opcodes(&MOVSX_WORD).rex().w(),
-    );
-
-    // movslq
-    e.enc64(
-        sextend.bind(I64).bind(I32),
-        rec_urm_noflags.opcodes(&MOVSXD).rex().w(),
-    );
-
-    // movzbl
-    e.enc32(
-        uextend.bind(I32).bind(I8),
-        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
-    );
-    e.enc64(
-        uextend.bind(I32).bind(I8),
-        rec_urm_noflags.opcodes(&MOVZX_BYTE).rex(),
-    );
-    e.enc64(
-        uextend.bind(I32).bind(I8),
-        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
-    );
-
-    // movzwl
-    e.enc32(
-        uextend.bind(I32).bind(I16),
-        rec_urm_noflags.opcodes(&MOVZX_WORD),
-    );
-    e.enc64(
-        uextend.bind(I32).bind(I16),
-        rec_urm_noflags.opcodes(&MOVZX_WORD).rex(),
-    );
-    e.enc64(
-        uextend.bind(I32).bind(I16),
-        rec_urm_noflags.opcodes(&MOVZX_WORD),
-    );
-
-    // movzbq, encoded as movzbl because it's equivalent and shorter.
-    e.enc64(
-        uextend.bind(I64).bind(I8),
-        rec_urm_noflags.opcodes(&MOVZX_BYTE).rex(),
-    );
-    e.enc64(
-        uextend.bind(I64).bind(I8),
-        rec_urm_noflags_abcd.opcodes(&MOVZX_BYTE),
-    );
-
-    // movzwq, encoded as movzwl because it's equivalent and shorter
-    e.enc64(
-        uextend.bind(I64).bind(I16),
-        rec_urm_noflags.opcodes(&MOVZX_WORD).rex(),
-    );
-    e.enc64(
-        uextend.bind(I64).bind(I16),
-        rec_urm_noflags.opcodes(&MOVZX_WORD),
-    );
-
-    // A 32-bit register copy clears the high 32 bits.
-    e.enc64(
-        uextend.bind(I64).bind(I32),
-        rec_umr.opcodes(&MOV_STORE).rex(),
-    );
-    e.enc64(uextend.bind(I64).bind(I32), rec_umr.opcodes(&MOV_STORE));
-
-    // Floating point
+    // Predicates shorthands.
+    let use_sse41 = settings.predicate_by_name("use_sse41");
 
     // Floating-point constants equal to 0.0 can be encoded using either `xorps` or `xorpd`, for
     // 32-bit and 64-bit floats respectively.
@@ -1616,40 +1223,6 @@ pub(crate) fn define(
     e.enc_x86_64_instp(f32const, rec_f32imm_z.opcodes(&XORPS), is_zero_32_bit_float);
     e.enc_x86_64_instp(f64const, rec_f64imm_z.opcodes(&XORPD), is_zero_64_bit_float);
 
-    // movd
-    e.enc_both(
-        bitcast.bind(F32).bind(I32),
-        rec_frurm.opcodes(&MOVD_LOAD_XMM),
-    );
-    e.enc_both(
-        bitcast.bind(I32).bind(F32),
-        rec_rfumr.opcodes(&MOVD_STORE_XMM),
-    );
-
-    // movq
-    e.enc64(
-        bitcast.bind(F64).bind(I64),
-        rec_frurm.opcodes(&MOVD_LOAD_XMM).rex().w(),
-    );
-    e.enc64(
-        bitcast.bind(I64).bind(F64),
-        rec_rfumr.opcodes(&MOVD_STORE_XMM).rex().w(),
-    );
-
-    // movaps
-    e.enc_both(copy.bind(F32), rec_furm.opcodes(&MOVAPS_LOAD));
-    e.enc_both(copy.bind(F64), rec_furm.opcodes(&MOVAPS_LOAD));
-
-    // TODO For x86-64, only define REX forms for now, since we can't describe the special regunit
-    // immediate operands with the current constraint language.
-    e.enc32(regmove.bind(F32), rec_frmov.opcodes(&MOVAPS_LOAD));
-    e.enc64(regmove.bind(F32), rec_frmov.opcodes(&MOVAPS_LOAD).rex());
-
-    // TODO For x86-64, only define REX forms for now, since we can't describe the special regunit
-    // immediate operands with the current constraint language.
-    e.enc32(regmove.bind(F64), rec_frmov.opcodes(&MOVAPS_LOAD));
-    e.enc64(regmove.bind(F64), rec_frmov.opcodes(&MOVAPS_LOAD).rex());
-
     // cvtsi2ss
     e.enc_i32_i64(fcvt_from_sint.bind(F32), rec_frurm.opcodes(&CVTSI2SS));
 
@@ -1711,6 +1284,156 @@ pub(crate) fn define(
     e.enc_both(x86_fmax.bind(F32), rec_fa.opcodes(&MAXSS));
     e.enc_both(x86_fmax.bind(F64), rec_fa.opcodes(&MAXSD));
 
+    // Comparisons.
+    //
+    // This only covers the condition codes in `supported_floatccs`, the rest are
+    // handled by legalization patterns.
+    e.enc_both(fcmp.bind(F32), rec_fcscc.opcodes(&UCOMISS));
+    e.enc_both(fcmp.bind(F64), rec_fcscc.opcodes(&UCOMISD));
+    e.enc_both(ffcmp.bind(F32), rec_fcmp.opcodes(&UCOMISS));
+    e.enc_both(ffcmp.bind(F64), rec_fcmp.opcodes(&UCOMISD));
+}
+
+#[inline(never)]
+fn define_alu(
+    e: &mut PerCpuModeEncodings,
+    shared_defs: &SharedDefinitions,
+    settings: &SettingGroup,
+    x86: &InstructionGroup,
+    r: &RecipeGroup,
+) {
+    let shared = &shared_defs.instructions;
+
+    // Shorthands for instructions.
+    let clz = shared.by_name("clz");
+    let ctz = shared.by_name("ctz");
+    let icmp = shared.by_name("icmp");
+    let icmp_imm = shared.by_name("icmp_imm");
+    let ifcmp = shared.by_name("ifcmp");
+    let ifcmp_imm = shared.by_name("ifcmp_imm");
+    let ifcmp_sp = shared.by_name("ifcmp_sp");
+    let ishl = shared.by_name("ishl");
+    let ishl_imm = shared.by_name("ishl_imm");
+    let popcnt = shared.by_name("popcnt");
+    let rotl = shared.by_name("rotl");
+    let rotl_imm = shared.by_name("rotl_imm");
+    let rotr = shared.by_name("rotr");
+    let rotr_imm = shared.by_name("rotr_imm");
+    let selectif = shared.by_name("selectif");
+    let sshr = shared.by_name("sshr");
+    let sshr_imm = shared.by_name("sshr_imm");
+    let trueff = shared.by_name("trueff");
+    let trueif = shared.by_name("trueif");
+    let ushr = shared.by_name("ushr");
+    let ushr_imm = shared.by_name("ushr_imm");
+    let x86_bsf = x86.by_name("x86_bsf");
+    let x86_bsr = x86.by_name("x86_bsr");
+
+    // Shorthands for recipes.
+    let rec_bsf_and_bsr = r.template("bsf_and_bsr");
+    let rec_cmov = r.template("cmov");
+    let rec_icscc = r.template("icscc");
+    let rec_icscc_ib = r.template("icscc_ib");
+    let rec_icscc_id = r.template("icscc_id");
+    let rec_rcmp = r.template("rcmp");
+    let rec_rcmp_ib = r.template("rcmp_ib");
+    let rec_rcmp_id = r.template("rcmp_id");
+    let rec_rcmp_sp = r.template("rcmp_sp");
+    let rec_rc = r.template("rc");
+    let rec_setf_abcd = r.template("setf_abcd");
+    let rec_seti_abcd = r.template("seti_abcd");
+    let rec_urm = r.template("urm");
+
+    // Predicates shorthands.
+    let use_popcnt = settings.predicate_by_name("use_popcnt");
+    let use_lzcnt = settings.predicate_by_name("use_lzcnt");
+    let use_bmi1 = settings.predicate_by_name("use_bmi1");
+
+    let band = shared.by_name("band");
+    let band_imm = shared.by_name("band_imm");
+    let band_not = shared.by_name("band_not");
+    let bnot = shared.by_name("bnot");
+    let bor = shared.by_name("bor");
+    let bor_imm = shared.by_name("bor_imm");
+    let bxor = shared.by_name("bxor");
+    let bxor_imm = shared.by_name("bxor_imm");
+    let iadd = shared.by_name("iadd");
+    let iadd_ifcarry = shared.by_name("iadd_ifcarry");
+    let iadd_ifcin = shared.by_name("iadd_ifcin");
+    let iadd_ifcout = shared.by_name("iadd_ifcout");
+    let iadd_imm = shared.by_name("iadd_imm");
+    let imul = shared.by_name("imul");
+    let isub = shared.by_name("isub");
+    let isub_ifbin = shared.by_name("isub_ifbin");
+    let isub_ifborrow = shared.by_name("isub_ifborrow");
+    let isub_ifbout = shared.by_name("isub_ifbout");
+    let x86_sdivmodx = x86.by_name("x86_sdivmodx");
+    let x86_smulx = x86.by_name("x86_smulx");
+    let x86_udivmodx = x86.by_name("x86_udivmodx");
+    let x86_umulx = x86.by_name("x86_umulx");
+
+    let rec_div = r.template("div");
+    let rec_fa = r.template("fa");
+    let rec_fax = r.template("fax");
+    let rec_mulx = r.template("mulx");
+    let rec_r_ib = r.template("r_ib");
+    let rec_r_id = r.template("r_id");
+    let rec_rin = r.template("rin");
+    let rec_rio = r.template("rio");
+    let rec_rout = r.template("rout");
+    let rec_rr = r.template("rr");
+    let rec_rrx = r.template("rrx");
+    let rec_ur = r.template("ur");
+
+    e.enc_i32_i64(iadd, rec_rr.opcodes(&ADD));
+    e.enc_i32_i64(iadd_ifcout, rec_rout.opcodes(&ADD));
+    e.enc_i32_i64(iadd_ifcin, rec_rin.opcodes(&ADC));
+    e.enc_i32_i64(iadd_ifcarry, rec_rio.opcodes(&ADC));
+    e.enc_i32_i64(iadd_imm, rec_r_ib.opcodes(&ADD_IMM8_SIGN_EXTEND).rrr(0));
+    e.enc_i32_i64(iadd_imm, rec_r_id.opcodes(&ADD_IMM).rrr(0));
+
+    e.enc_i32_i64(isub, rec_rr.opcodes(&SUB));
+    e.enc_i32_i64(isub_ifbout, rec_rout.opcodes(&SUB));
+    e.enc_i32_i64(isub_ifbin, rec_rin.opcodes(&SBB));
+    e.enc_i32_i64(isub_ifborrow, rec_rio.opcodes(&SBB));
+
+    e.enc_i32_i64(band, rec_rr.opcodes(&AND));
+    e.enc_b32_b64(band, rec_rr.opcodes(&AND));
+
+    // TODO: band_imm.i64 with an unsigned 32-bit immediate can be encoded as band_imm.i32. Can
+    // even use the single-byte immediate for 0xffff_ffXX masks.
+
+    e.enc_i32_i64(band_imm, rec_r_ib.opcodes(&AND_IMM8_SIGN_EXTEND).rrr(4));
+    e.enc_i32_i64(band_imm, rec_r_id.opcodes(&AND_IMM).rrr(4));
+
+    e.enc_i32_i64(bor, rec_rr.opcodes(&OR));
+    e.enc_b32_b64(bor, rec_rr.opcodes(&OR));
+    e.enc_i32_i64(bor_imm, rec_r_ib.opcodes(&OR_IMM8_SIGN_EXTEND).rrr(1));
+    e.enc_i32_i64(bor_imm, rec_r_id.opcodes(&OR_IMM).rrr(1));
+
+    e.enc_i32_i64(bxor, rec_rr.opcodes(&XOR));
+    e.enc_b32_b64(bxor, rec_rr.opcodes(&XOR));
+    e.enc_i32_i64(bxor_imm, rec_r_ib.opcodes(&XOR_IMM8_SIGN_EXTEND).rrr(6));
+    e.enc_i32_i64(bxor_imm, rec_r_id.opcodes(&XOR_IMM).rrr(6));
+
+    // x86 has a bitwise not instruction NOT.
+    e.enc_i32_i64(bnot, rec_ur.opcodes(&NOT).rrr(2));
+    e.enc_b32_b64(bnot, rec_ur.opcodes(&NOT).rrr(2));
+
+    // Also add a `b1` encodings for the logic instructions.
+    // TODO: Should this be done with 8-bit instructions? It would improve partial register
+    // dependencies.
+    e.enc_both(band.bind(B1), rec_rr.opcodes(&AND));
+    e.enc_both(bor.bind(B1), rec_rr.opcodes(&OR));
+    e.enc_both(bxor.bind(B1), rec_rr.opcodes(&XOR));
+
+    e.enc_i32_i64(imul, rec_rrx.opcodes(&IMUL));
+    e.enc_i32_i64(x86_sdivmodx, rec_div.opcodes(&IDIV).rrr(7));
+    e.enc_i32_i64(x86_udivmodx, rec_div.opcodes(&DIV).rrr(6));
+
+    e.enc_i32_i64(x86_smulx, rec_mulx.opcodes(&IMUL_RDX_RAX).rrr(5));
+    e.enc_i32_i64(x86_umulx, rec_mulx.opcodes(&MUL).rrr(4));
+
     // Binary bitwise ops.
     //
     // The F64 version is intentionally encoded using the single-precision opcode:
@@ -1728,14 +1451,182 @@ pub(crate) fn define(
     e.enc_both(band_not.bind(F32), rec_fax.opcodes(&ANDNPS));
     e.enc_both(band_not.bind(F64), rec_fax.opcodes(&ANDNPS));
 
-    // Comparisons.
-    //
-    // This only covers the condition codes in `supported_floatccs`, the rest are
-    // handled by legalization patterns.
-    e.enc_both(fcmp.bind(F32), rec_fcscc.opcodes(&UCOMISS));
-    e.enc_both(fcmp.bind(F64), rec_fcscc.opcodes(&UCOMISD));
-    e.enc_both(ffcmp.bind(F32), rec_fcmp.opcodes(&UCOMISS));
-    e.enc_both(ffcmp.bind(F64), rec_fcmp.opcodes(&UCOMISD));
+    // Shifts and rotates.
+    // Note that the dynamic shift amount is only masked by 5 or 6 bits; the 8-bit
+    // and 16-bit shifts would need explicit masking.
+
+    for &(inst, rrr) in &[(rotl, 0), (rotr, 1), (ishl, 4), (ushr, 5), (sshr, 7)] {
+        // Cannot use enc_i32_i64 for this pattern because instructions require
+        // to bind any.
+        e.enc32(
+            inst.bind(I32).bind(Any),
+            rec_rc.opcodes(&ROTATE_CL).rrr(rrr),
+        );
+        e.enc64(
+            inst.bind(I64).bind(Any),
+            rec_rc.opcodes(&ROTATE_CL).rrr(rrr).rex().w(),
+        );
+        e.enc64(
+            inst.bind(I32).bind(Any),
+            rec_rc.opcodes(&ROTATE_CL).rrr(rrr).rex(),
+        );
+        e.enc64(
+            inst.bind(I32).bind(Any),
+            rec_rc.opcodes(&ROTATE_CL).rrr(rrr),
+        );
+    }
+
+    e.enc_i32_i64(rotl_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(0));
+    e.enc_i32_i64(rotr_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(1));
+    e.enc_i32_i64(ishl_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(4));
+    e.enc_i32_i64(ushr_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(5));
+    e.enc_i32_i64(sshr_imm, rec_r_ib.opcodes(&ROTATE_IMM8).rrr(7));
+
+    // Population count.
+    e.enc32_isap(popcnt.bind(I32), rec_urm.opcodes(&POPCNT), use_popcnt);
+    e.enc64_isap(
+        popcnt.bind(I64),
+        rec_urm.opcodes(&POPCNT).rex().w(),
+        use_popcnt,
+    );
+    e.enc64_isap(popcnt.bind(I32), rec_urm.opcodes(&POPCNT).rex(), use_popcnt);
+    e.enc64_isap(popcnt.bind(I32), rec_urm.opcodes(&POPCNT), use_popcnt);
+
+    // Count leading zero bits.
+    e.enc32_isap(clz.bind(I32), rec_urm.opcodes(&LZCNT), use_lzcnt);
+    e.enc64_isap(clz.bind(I64), rec_urm.opcodes(&LZCNT).rex().w(), use_lzcnt);
+    e.enc64_isap(clz.bind(I32), rec_urm.opcodes(&LZCNT).rex(), use_lzcnt);
+    e.enc64_isap(clz.bind(I32), rec_urm.opcodes(&LZCNT), use_lzcnt);
+
+    // Count trailing zero bits.
+    e.enc32_isap(ctz.bind(I32), rec_urm.opcodes(&TZCNT), use_bmi1);
+    e.enc64_isap(ctz.bind(I64), rec_urm.opcodes(&TZCNT).rex().w(), use_bmi1);
+    e.enc64_isap(ctz.bind(I32), rec_urm.opcodes(&TZCNT).rex(), use_bmi1);
+    e.enc64_isap(ctz.bind(I32), rec_urm.opcodes(&TZCNT), use_bmi1);
+
+    // Bit scan forwards and reverse
+    e.enc_i32_i64(x86_bsf, rec_bsf_and_bsr.opcodes(&BIT_SCAN_FORWARD));
+    e.enc_i32_i64(x86_bsr, rec_bsf_and_bsr.opcodes(&BIT_SCAN_REVERSE));
+
+    // Comparisons
+    e.enc_i32_i64(icmp, rec_icscc.opcodes(&CMP_REG));
+    e.enc_i32_i64(icmp_imm, rec_icscc_ib.opcodes(&CMP_IMM8).rrr(7));
+    e.enc_i32_i64(icmp_imm, rec_icscc_id.opcodes(&CMP_IMM).rrr(7));
+    e.enc_i32_i64(ifcmp, rec_rcmp.opcodes(&CMP_REG));
+    e.enc_i32_i64(ifcmp_imm, rec_rcmp_ib.opcodes(&CMP_IMM8).rrr(7));
+    e.enc_i32_i64(ifcmp_imm, rec_rcmp_id.opcodes(&CMP_IMM).rrr(7));
+    // TODO: We could special-case ifcmp_imm(x, 0) to TEST(x, x).
+
+    e.enc32(ifcmp_sp.bind(I32), rec_rcmp_sp.opcodes(&CMP_REG));
+    e.enc64(ifcmp_sp.bind(I64), rec_rcmp_sp.opcodes(&CMP_REG).rex().w());
+
+    // Convert flags to bool.
+    // This encodes `b1` as an 8-bit low register with the value 0 or 1.
+    e.enc_both(trueif, rec_seti_abcd.opcodes(&SET_BYTE_IF_OVERFLOW));
+    e.enc_both(trueff, rec_setf_abcd.opcodes(&SET_BYTE_IF_OVERFLOW));
+
+    // Conditional move (a.k.a integer select).
+    e.enc_i32_i64(selectif, rec_cmov.opcodes(&CMOV_OVERFLOW));
+}
+
+#[inline(never)]
+fn define_simd(
+    e: &mut PerCpuModeEncodings,
+    shared_defs: &SharedDefinitions,
+    settings: &SettingGroup,
+    x86: &InstructionGroup,
+    r: &RecipeGroup,
+) {
+    let shared = &shared_defs.instructions;
+    let formats = &shared_defs.formats;
+
+    // Shorthands for instructions.
+    let bitcast = shared.by_name("bitcast");
+    let bor = shared.by_name("bor");
+    let bxor = shared.by_name("bxor");
+    let copy = shared.by_name("copy");
+    let copy_nop = shared.by_name("copy_nop");
+    let fadd = shared.by_name("fadd");
+    let fcmp = shared.by_name("fcmp");
+    let fdiv = shared.by_name("fdiv");
+    let fill = shared.by_name("fill");
+    let fill_nop = shared.by_name("fill_nop");
+    let fmax = shared.by_name("fmax");
+    let fmin = shared.by_name("fmin");
+    let fmul = shared.by_name("fmul");
+    let fsub = shared.by_name("fsub");
+    let iadd = shared.by_name("iadd");
+    let icmp = shared.by_name("icmp");
+    let imul = shared.by_name("imul");
+    let ishl_imm = shared.by_name("ishl_imm");
+    let load = shared.by_name("load");
+    let raw_bitcast = shared.by_name("raw_bitcast");
+    let regfill = shared.by_name("regfill");
+    let regmove = shared.by_name("regmove");
+    let regspill = shared.by_name("regspill");
+    let sadd_sat = shared.by_name("sadd_sat");
+    let scalar_to_vector = shared.by_name("scalar_to_vector");
+    let spill = shared.by_name("spill");
+    let sqrt = shared.by_name("sqrt");
+    let sshr_imm = shared.by_name("sshr_imm");
+    let ssub_sat = shared.by_name("ssub_sat");
+    let store = shared.by_name("store");
+    let uadd_sat = shared.by_name("uadd_sat");
+    let ushr_imm = shared.by_name("ushr_imm");
+    let usub_sat = shared.by_name("usub_sat");
+    let vconst = shared.by_name("vconst");
+    let x86_insertps = x86.by_name("x86_insertps");
+    let x86_movlhps = x86.by_name("x86_movlhps");
+    let x86_movsd = x86.by_name("x86_movsd");
+    let x86_pextr = x86.by_name("x86_pextr");
+    let x86_pinsr = x86.by_name("x86_pinsr");
+    let x86_pmaxs = x86.by_name("x86_pmaxs");
+    let x86_pmaxu = x86.by_name("x86_pmaxu");
+    let x86_pmins = x86.by_name("x86_pmins");
+    let x86_pminu = x86.by_name("x86_pminu");
+    let x86_pshufb = x86.by_name("x86_pshufb");
+    let x86_pshufd = x86.by_name("x86_pshufd");
+    let x86_psll = x86.by_name("x86_psll");
+    let x86_psra = x86.by_name("x86_psra");
+    let x86_psrl = x86.by_name("x86_psrl");
+    let x86_ptest = x86.by_name("x86_ptest");
+
+    // Shorthands for recipes.
+    let rec_f_ib = r.template("f_ib");
+    let rec_fa = r.template("fa");
+    let rec_fa_ib = r.template("fa_ib");
+    let rec_fax = r.template("fax");
+    let rec_fcmp = r.template("fcmp");
+    let rec_ffillSib32 = r.template("ffillSib32");
+    let rec_ffillnull = r.recipe("ffillnull");
+    let rec_fld = r.template("fld");
+    let rec_fldDisp32 = r.template("fldDisp32");
+    let rec_fldDisp8 = r.template("fldDisp8");
+    let rec_fregfill32 = r.template("fregfill32");
+    let rec_fregspill32 = r.template("fregspill32");
+    let rec_frmov = r.template("frmov");
+    let rec_frurm = r.template("frurm");
+    let rec_fspillSib32 = r.template("fspillSib32");
+    let rec_fst = r.template("fst");
+    let rec_fstDisp32 = r.template("fstDisp32");
+    let rec_fstDisp8 = r.template("fstDisp8");
+    let rec_furm = r.template("furm");
+    let rec_icscc_fpr = r.template("icscc_fpr");
+    let rec_null_fpr = r.recipe("null_fpr");
+    let rec_pfcmp = r.template("pfcmp");
+    let rec_r_ib_unsigned_fpr = r.template("r_ib_unsigned_fpr");
+    let rec_r_ib_unsigned_gpr = r.template("r_ib_unsigned_gpr");
+    let rec_r_ib_unsigned_r = r.template("r_ib_unsigned_r");
+    let rec_stacknull = r.recipe("stacknull");
+    let rec_vconst = r.template("vconst");
+    let rec_vconst_optimized = r.template("vconst_optimized");
+
+    // Predicates shorthands.
+    settings.predicate_by_name("all_ones_funcaddrs_and_not_is_pic");
+    settings.predicate_by_name("not_all_ones_funcaddrs_and_not_is_pic");
+    let use_ssse3_simd = settings.predicate_by_name("use_ssse3_simd");
+    let use_sse41_simd = settings.predicate_by_name("use_sse41_simd");
+    let use_sse42_simd = settings.predicate_by_name("use_sse42_simd");
 
     // SIMD vector size: eventually multiple vector sizes may be supported but for now only
     // SSE-sized vectors are available.
@@ -1976,6 +1867,7 @@ pub(crate) fn define(
     );
 
     // SIMD integer subtraction
+    let isub = shared.by_name("isub");
     for (ty, opcodes) in &[(I8, &PSUBB), (I16, &PSUBW), (I32, &PSUBD), (I64, &PSUBQ)] {
         let isub = isub.bind(vector(*ty, sse_vector_size));
         e.enc_32_64(isub, rec_fa.opcodes(*opcodes));
@@ -2010,6 +1902,8 @@ pub(crate) fn define(
     }
 
     // SIMD logical operations
+    let band = shared.by_name("band");
+    let band_not = shared.by_name("band_not");
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         // and
         let band = band.bind(vector(ty, sse_vector_size));
@@ -2148,8 +2042,298 @@ pub(crate) fn define(
         let inst = inst.bind(vector(*ty, sse_vector_size));
         e.enc_both(inst, rec_furm.opcodes(opcodes));
     }
+}
 
-    // Reference type instructions
+#[inline(never)]
+fn define_entity_ref(
+    e: &mut PerCpuModeEncodings,
+    shared_defs: &SharedDefinitions,
+    settings: &SettingGroup,
+    r: &RecipeGroup,
+) {
+    let shared = &shared_defs.instructions;
+    let formats = &shared_defs.formats;
+
+    // Shorthands for instructions.
+    let func_addr = shared.by_name("func_addr");
+    let stack_addr = shared.by_name("stack_addr");
+    let symbol_value = shared.by_name("symbol_value");
+
+    // Shorthands for recipes.
+    let rec_allones_fnaddr4 = r.template("allones_fnaddr4");
+    let rec_allones_fnaddr8 = r.template("allones_fnaddr8");
+    let rec_fnaddr4 = r.template("fnaddr4");
+    let rec_fnaddr8 = r.template("fnaddr8");
+    let rec_got_fnaddr8 = r.template("got_fnaddr8");
+    let rec_got_gvaddr8 = r.template("got_gvaddr8");
+    let rec_gvaddr4 = r.template("gvaddr4");
+    let rec_gvaddr8 = r.template("gvaddr8");
+    let rec_pcrel_fnaddr8 = r.template("pcrel_fnaddr8");
+    let rec_pcrel_gvaddr8 = r.template("pcrel_gvaddr8");
+    let rec_spaddr4_id = r.template("spaddr4_id");
+    let rec_spaddr8_id = r.template("spaddr8_id");
+
+    // Predicates shorthands.
+    let all_ones_funcaddrs_and_not_is_pic =
+        settings.predicate_by_name("all_ones_funcaddrs_and_not_is_pic");
+    let is_pic = settings.predicate_by_name("is_pic");
+    let not_all_ones_funcaddrs_and_not_is_pic =
+        settings.predicate_by_name("not_all_ones_funcaddrs_and_not_is_pic");
+    let not_is_pic = settings.predicate_by_name("not_is_pic");
+
+    // Function addresses.
+
+    // Non-PIC, all-ones funcaddresses.
+    e.enc32_isap(
+        func_addr.bind(I32),
+        rec_fnaddr4.opcodes(&MOV_IMM),
+        not_all_ones_funcaddrs_and_not_is_pic,
+    );
+    e.enc64_isap(
+        func_addr.bind(I64),
+        rec_fnaddr8.opcodes(&MOV_IMM).rex().w(),
+        not_all_ones_funcaddrs_and_not_is_pic,
+    );
+
+    // Non-PIC, all-zeros funcaddresses.
+    e.enc32_isap(
+        func_addr.bind(I32),
+        rec_allones_fnaddr4.opcodes(&MOV_IMM),
+        all_ones_funcaddrs_and_not_is_pic,
+    );
+    e.enc64_isap(
+        func_addr.bind(I64),
+        rec_allones_fnaddr8.opcodes(&MOV_IMM).rex().w(),
+        all_ones_funcaddrs_and_not_is_pic,
+    );
+
+    // 64-bit, colocated, both PIC and non-PIC. Use the lea instruction's pc-relative field.
+    let is_colocated_func =
+        InstructionPredicate::new_is_colocated_func(&*formats.func_addr, "func_ref");
+    e.enc64_instp(
+        func_addr.bind(I64),
+        rec_pcrel_fnaddr8.opcodes(&LEA).rex().w(),
+        is_colocated_func,
+    );
+
+    // 64-bit, non-colocated, PIC.
+    e.enc64_isap(
+        func_addr.bind(I64),
+        rec_got_fnaddr8.opcodes(&MOV_LOAD).rex().w(),
+        is_pic,
+    );
+
+    // Global addresses.
+
+    // Non-PIC.
+    e.enc32_isap(
+        symbol_value.bind(I32),
+        rec_gvaddr4.opcodes(&MOV_IMM),
+        not_is_pic,
+    );
+    e.enc64_isap(
+        symbol_value.bind(I64),
+        rec_gvaddr8.opcodes(&MOV_IMM).rex().w(),
+        not_is_pic,
+    );
+
+    // PIC, colocated.
+    e.enc64_func(
+        symbol_value.bind(I64),
+        rec_pcrel_gvaddr8.opcodes(&LEA).rex().w(),
+        |encoding| {
+            encoding
+                .isa_predicate(is_pic)
+                .inst_predicate(InstructionPredicate::new_is_colocated_data(formats))
+        },
+    );
+
+    // PIC, non-colocated.
+    e.enc64_isap(
+        symbol_value.bind(I64),
+        rec_got_gvaddr8.opcodes(&MOV_LOAD).rex().w(),
+        is_pic,
+    );
+
+    // Stack addresses.
+    //
+    // TODO: Add encoding rules for stack_load and stack_store, so that they
+    // don't get legalized to stack_addr + load/store.
+    e.enc32(stack_addr.bind(I32), rec_spaddr4_id.opcodes(&LEA));
+    e.enc64(stack_addr.bind(I64), rec_spaddr8_id.opcodes(&LEA).rex().w());
+}
+
+/// Control flow opcodes.
+#[inline(never)]
+fn define_control_flow(
+    e: &mut PerCpuModeEncodings,
+    shared_defs: &SharedDefinitions,
+    settings: &SettingGroup,
+    r: &RecipeGroup,
+) {
+    let shared = &shared_defs.instructions;
+    let formats = &shared_defs.formats;
+
+    // Shorthands for instructions.
+    let brff = shared.by_name("brff");
+    let brif = shared.by_name("brif");
+    let brnz = shared.by_name("brnz");
+    let brz = shared.by_name("brz");
+    let call = shared.by_name("call");
+    let call_indirect = shared.by_name("call_indirect");
+    let debugtrap = shared.by_name("debugtrap");
+    let indirect_jump_table_br = shared.by_name("indirect_jump_table_br");
+    let jump = shared.by_name("jump");
+    let jump_table_base = shared.by_name("jump_table_base");
+    let jump_table_entry = shared.by_name("jump_table_entry");
+    let return_ = shared.by_name("return");
+    let trap = shared.by_name("trap");
+    let trapff = shared.by_name("trapff");
+    let trapif = shared.by_name("trapif");
+    let resumable_trap = shared.by_name("resumable_trap");
+
+    // Shorthands for recipes.
+    let rec_brfb = r.template("brfb");
+    let rec_brfd = r.template("brfd");
+    let rec_brib = r.template("brib");
+    let rec_brid = r.template("brid");
+    let rec_call_id = r.template("call_id");
+    let rec_call_plt_id = r.template("call_plt_id");
+    let rec_call_r = r.template("call_r");
+    let rec_debugtrap = r.recipe("debugtrap");
+    let rec_indirect_jmp = r.template("indirect_jmp");
+    let rec_jmpb = r.template("jmpb");
+    let rec_jmpd = r.template("jmpd");
+    let rec_jt_base = r.template("jt_base");
+    let rec_jt_entry = r.template("jt_entry");
+    let rec_ret = r.template("ret");
+    let rec_t8jccb_abcd = r.template("t8jccb_abcd");
+    let rec_t8jccd_abcd = r.template("t8jccd_abcd");
+    let rec_t8jccd_long = r.template("t8jccd_long");
+    let rec_tjccb = r.template("tjccb");
+    let rec_tjccd = r.template("tjccd");
+    let rec_trap = r.template("trap");
+    let rec_trapif = r.recipe("trapif");
+    let rec_trapff = r.recipe("trapff");
+
+    // Predicates shorthands.
+    let is_pic = settings.predicate_by_name("is_pic");
+
+    // Call/return
+
+    // 32-bit, both PIC and non-PIC.
+    e.enc32(call, rec_call_id.opcodes(&CALL_RELATIVE));
+
+    // 64-bit, colocated, both PIC and non-PIC. Use the call instruction's pc-relative field.
+    let is_colocated_func = InstructionPredicate::new_is_colocated_func(&*formats.call, "func_ref");
+    e.enc64_instp(call, rec_call_id.opcodes(&CALL_RELATIVE), is_colocated_func);
+
+    // 64-bit, non-colocated, PIC. There is no 64-bit non-colocated non-PIC version, since non-PIC
+    // is currently using the large model, which requires calls be lowered to
+    // func_addr+call_indirect.
+    e.enc64_isap(call, rec_call_plt_id.opcodes(&CALL_RELATIVE), is_pic);
+
+    e.enc32(
+        call_indirect.bind(I32),
+        rec_call_r.opcodes(&JUMP_ABSOLUTE).rrr(2),
+    );
+    e.enc64(
+        call_indirect.bind(I64),
+        rec_call_r.opcodes(&JUMP_ABSOLUTE).rrr(2).rex(),
+    );
+    e.enc64(
+        call_indirect.bind(I64),
+        rec_call_r.opcodes(&JUMP_ABSOLUTE).rrr(2),
+    );
+
+    e.enc32(return_, rec_ret.opcodes(&RET_NEAR));
+    e.enc64(return_, rec_ret.opcodes(&RET_NEAR));
+
+    // Branches.
+    e.enc32(jump, rec_jmpb.opcodes(&JUMP_SHORT));
+    e.enc64(jump, rec_jmpb.opcodes(&JUMP_SHORT));
+    e.enc32(jump, rec_jmpd.opcodes(&JUMP_NEAR_RELATIVE));
+    e.enc64(jump, rec_jmpd.opcodes(&JUMP_NEAR_RELATIVE));
+
+    e.enc_both(brif, rec_brib.opcodes(&JUMP_SHORT_IF_OVERFLOW));
+    e.enc_both(brif, rec_brid.opcodes(&JUMP_NEAR_IF_OVERFLOW));
+
+    // Not all float condition codes are legal, see `supported_floatccs`.
+    e.enc_both(brff, rec_brfb.opcodes(&JUMP_SHORT_IF_OVERFLOW));
+    e.enc_both(brff, rec_brfd.opcodes(&JUMP_NEAR_IF_OVERFLOW));
+
+    // Note that the tjccd opcode will be prefixed with 0x0f.
+    e.enc_i32_i64_explicit_rex(brz, rec_tjccb.opcodes(&JUMP_SHORT_IF_EQUAL));
+    e.enc_i32_i64_explicit_rex(brz, rec_tjccd.opcodes(&TEST_BYTE_REG));
+    e.enc_i32_i64_explicit_rex(brnz, rec_tjccb.opcodes(&JUMP_SHORT_IF_NOT_EQUAL));
+    e.enc_i32_i64_explicit_rex(brnz, rec_tjccd.opcodes(&TEST_REG));
+
+    // Branch on a b1 value in a register only looks at the low 8 bits. See also
+    // bint encodings below.
+    //
+    // Start with the worst-case encoding for X86_32 only. The register allocator
+    // can't handle a branch with an ABCD-constrained operand.
+    e.enc32(brz.bind(B1), rec_t8jccd_long.opcodes(&TEST_BYTE_REG));
+    e.enc32(brnz.bind(B1), rec_t8jccd_long.opcodes(&TEST_REG));
+
+    e.enc_both(brz.bind(B1), rec_t8jccb_abcd.opcodes(&JUMP_SHORT_IF_EQUAL));
+    e.enc_both(brz.bind(B1), rec_t8jccd_abcd.opcodes(&TEST_BYTE_REG));
+    e.enc_both(
+        brnz.bind(B1),
+        rec_t8jccb_abcd.opcodes(&JUMP_SHORT_IF_NOT_EQUAL),
+    );
+    e.enc_both(brnz.bind(B1), rec_t8jccd_abcd.opcodes(&TEST_REG));
+
+    // Jump tables.
+    e.enc64(
+        jump_table_entry.bind(I64),
+        rec_jt_entry.opcodes(&MOVSXD).rex().w(),
+    );
+    e.enc32(jump_table_entry.bind(I32), rec_jt_entry.opcodes(&MOV_LOAD));
+
+    e.enc64(
+        jump_table_base.bind(I64),
+        rec_jt_base.opcodes(&LEA).rex().w(),
+    );
+    e.enc32(jump_table_base.bind(I32), rec_jt_base.opcodes(&LEA));
+
+    e.enc_x86_64(
+        indirect_jump_table_br.bind(I64),
+        rec_indirect_jmp.opcodes(&JUMP_ABSOLUTE).rrr(4),
+    );
+    e.enc32(
+        indirect_jump_table_br.bind(I32),
+        rec_indirect_jmp.opcodes(&JUMP_ABSOLUTE).rrr(4),
+    );
+
+    // Trap as ud2
+    e.enc32(trap, rec_trap.opcodes(&UNDEFINED2));
+    e.enc64(trap, rec_trap.opcodes(&UNDEFINED2));
+    e.enc32(resumable_trap, rec_trap.opcodes(&UNDEFINED2));
+    e.enc64(resumable_trap, rec_trap.opcodes(&UNDEFINED2));
+
+    // Debug trap as int3
+    e.enc32_rec(debugtrap, rec_debugtrap, 0);
+    e.enc64_rec(debugtrap, rec_debugtrap, 0);
+
+    e.enc32_rec(trapif, rec_trapif, 0);
+    e.enc64_rec(trapif, rec_trapif, 0);
+    e.enc32_rec(trapff, rec_trapff, 0);
+    e.enc64_rec(trapff, rec_trapff, 0);
+}
+
+/// Reference type instructions.
+#[inline(never)]
+fn define_reftypes(e: &mut PerCpuModeEncodings, shared_defs: &SharedDefinitions, r: &RecipeGroup) {
+    let shared = &shared_defs.instructions;
+
+    let is_null = shared.by_name("is_null");
+    let null = shared.by_name("null");
+    let safepoint = shared.by_name("safepoint");
+
+    let rec_is_zero = r.template("is_zero");
+    let rec_pu_id_ref = r.template("pu_id_ref");
+    let rec_safepoint = r.recipe("safepoint");
 
     // Null references implemented as iconst 0.
     e.enc32(null.bind(R32), rec_pu_id_ref.opcodes(&MOV_IMM));
@@ -2163,6 +2347,28 @@ pub(crate) fn define(
     // safepoint instruction calls sink, no actual encoding.
     e.enc32_rec(safepoint, rec_safepoint, 0);
     e.enc64_rec(safepoint, rec_safepoint, 0);
+}
+
+#[allow(clippy::cognitive_complexity)]
+pub(crate) fn define(
+    shared_defs: &SharedDefinitions,
+    settings: &SettingGroup,
+    x86: &InstructionGroup,
+    r: &RecipeGroup,
+) -> PerCpuModeEncodings {
+    // Definitions.
+    let mut e = PerCpuModeEncodings::new();
+
+    define_moves(&mut e, shared_defs, r);
+    define_memory(&mut e, shared_defs, x86, r);
+    define_fpu_moves(&mut e, shared_defs, r);
+    define_fpu_memory(&mut e, shared_defs, r);
+    define_fpu_ops(&mut e, shared_defs, settings, x86, r);
+    define_alu(&mut e, shared_defs, settings, x86, r);
+    define_simd(&mut e, shared_defs, settings, x86, r);
+    define_entity_ref(&mut e, shared_defs, settings, r);
+    define_control_flow(&mut e, shared_defs, settings, r);
+    define_reftypes(&mut e, shared_defs, r);
 
     e
 }
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 388e2cefad..5f9a525353 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -11,6 +11,476 @@ use crate::shared::formats::Formats;
 use crate::shared::types;
 use crate::shared::{entities::EntityRefs, immediates::Immediates};
 
+#[inline(never)]
+fn define_control_flow(
+    ig: &mut InstructionGroupBuilder,
+    formats: &Formats,
+    imm: &Immediates,
+    entities: &EntityRefs,
+) {
+    let EBB = &Operand::new("EBB", &entities.ebb).with_doc("Destination extended basic block");
+    let args = &Operand::new("args", &entities.varargs).with_doc("EBB arguments");
+
+    ig.push(
+        Inst::new(
+            "jump",
+            r#"
+        Jump.
+
+        Unconditionally jump to an extended basic block, passing the specified
+        EBB arguments. The number and types of arguments must match the
+        destination EBB.
+        "#,
+            &formats.jump,
+        )
+        .operands_in(vec![EBB, args])
+        .is_terminator(true)
+        .is_branch(true),
+    );
+
+    ig.push(
+        Inst::new(
+            "fallthrough",
+            r#"
+        Fall through to the next EBB.
+
+        This is the same as `jump`, except the destination EBB must be
+        the next one in the layout.
+
+        Jumps are turned into fall-through instructions by the branch
+        relaxation pass. There is no reason to use this instruction outside
+        that pass.
+        "#,
+            &formats.jump,
+        )
+        .operands_in(vec![EBB, args])
+        .is_terminator(true)
+        .is_branch(true),
+    );
+
+    let Testable = &TypeVar::new(
+        "Testable",
+        "A scalar boolean or integer type",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .bools(Interval::All)
+            .build(),
+    );
+
+    {
+        let c = &Operand::new("c", Testable).with_doc("Controlling value to test");
+
+        ig.push(
+            Inst::new(
+                "brz",
+                r#"
+        Branch when zero.
+
+        If ``c`` is a `b1` value, take the branch when ``c`` is false. If
+        ``c`` is an integer value, take the branch when ``c = 0``.
+        "#,
+                &formats.branch,
+            )
+            .operands_in(vec![c, EBB, args])
+            .is_branch(true),
+        );
+
+        ig.push(
+            Inst::new(
+                "brnz",
+                r#"
+        Branch when non-zero.
+
+        If ``c`` is a `b1` value, take the branch when ``c`` is true. If
+        ``c`` is an integer value, take the branch when ``c != 0``.
+        "#,
+                &formats.branch,
+            )
+            .operands_in(vec![c, EBB, args])
+            .is_branch(true),
+        );
+    }
+
+    let iB = &TypeVar::new(
+        "iB",
+        "A scalar integer type",
+        TypeSetBuilder::new().ints(Interval::All).build(),
+    );
+    let iflags: &TypeVar = &ValueType::Special(types::Flag::IFlags.into()).into();
+    let fflags: &TypeVar = &ValueType::Special(types::Flag::FFlags.into()).into();
+
+    {
+        let Cond = &Operand::new("Cond", &imm.intcc);
+        let x = &Operand::new("x", iB);
+        let y = &Operand::new("y", iB);
+
+        ig.push(
+            Inst::new(
+                "br_icmp",
+                r#"
+        Compare scalar integers and branch.
+
+        Compare ``x`` and ``y`` in the same way as the `icmp` instruction
+        and take the branch if the condition is true:
+
+        ```text
+            br_icmp ugt v1, v2, ebb4(v5, v6)
+        ```
+
+        is semantically equivalent to:
+
+        ```text
+            v10 = icmp ugt, v1, v2
+            brnz v10, ebb4(v5, v6)
+        ```
+
+        Some RISC architectures like MIPS and RISC-V provide instructions that
+        implement all or some of the condition codes. The instruction can also
+        be used to represent *macro-op fusion* on architectures like Intel's.
+        "#,
+                &formats.branch_icmp,
+            )
+            .operands_in(vec![Cond, x, y, EBB, args])
+            .is_branch(true),
+        );
+
+        let f = &Operand::new("f", iflags);
+
+        ig.push(
+            Inst::new(
+                "brif",
+                r#"
+        Branch when condition is true in integer CPU flags.
+        "#,
+                &formats.branch_int,
+            )
+            .operands_in(vec![Cond, f, EBB, args])
+            .is_branch(true),
+        );
+    }
+
+    {
+        let Cond = &Operand::new("Cond", &imm.floatcc);
+
+        let f = &Operand::new("f", fflags);
+
+        ig.push(
+            Inst::new(
+                "brff",
+                r#"
+        Branch when condition is true in floating point CPU flags.
+        "#,
+                &formats.branch_float,
+            )
+            .operands_in(vec![Cond, f, EBB, args])
+            .is_branch(true),
+        );
+    }
+
+    {
+        let x = &Operand::new("x", iB).with_doc("index into jump table");
+        let JT = &Operand::new("JT", &entities.jump_table);
+
+        ig.push(
+            Inst::new(
+                "br_table",
+                r#"
+        Indirect branch via jump table.
+
+        Use ``x`` as an unsigned index into the jump table ``JT``. If a jump
+        table entry is found, branch to the corresponding EBB. If no entry was
+        found or the index is out-of-bounds, branch to the given default EBB.
+
+        Note that this branch instruction can't pass arguments to the targeted
+        blocks. Split critical edges as needed to work around this.
+
+        Do not confuse this with "tables" in WebAssembly. ``br_table`` is for
+        jump tables with destinations within the current function only -- think
+        of a ``match`` in Rust or a ``switch`` in C.  If you want to call a
+        function in a dynamic library, that will typically use
+        ``call_indirect``.
+        "#,
+                &formats.branch_table,
+            )
+            .operands_in(vec![x, EBB, JT])
+            .is_terminator(true)
+            .is_branch(true),
+        );
+    }
+
+    let iAddr = &TypeVar::new(
+        "iAddr",
+        "An integer address type",
+        TypeSetBuilder::new().ints(32..64).build(),
+    );
+
+    {
+        let x = &Operand::new("x", iAddr).with_doc("index into jump table");
+        let addr = &Operand::new("addr", iAddr);
+        let Size = &Operand::new("Size", &imm.uimm8).with_doc("Size in bytes");
+        let JT = &Operand::new("JT", &entities.jump_table);
+        let entry = &Operand::new("entry", iAddr).with_doc("entry of jump table");
+
+        ig.push(
+            Inst::new(
+                "jump_table_entry",
+                r#"
+    Get an entry from a jump table.
+
+    Load a serialized ``entry`` from a jump table ``JT`` at a given index
+    ``addr`` with a specific ``Size``. The retrieved entry may need to be
+    decoded after loading, depending upon the jump table type used.
+
+    Currently, the only type supported is entries which are relative to the
+    base of the jump table.
+    "#,
+                &formats.branch_table_entry,
+            )
+            .operands_in(vec![x, addr, Size, JT])
+            .operands_out(vec![entry])
+            .can_load(true),
+        );
+
+        ig.push(
+            Inst::new(
+                "jump_table_base",
+                r#"
+    Get the absolute base address of a jump table.
+
+    This is used for jump tables wherein the entries are stored relative to
+    the base of jump table. In order to use these, generated code should first
+    load an entry using ``jump_table_entry``, then use this instruction to add
+    the relative base back to it.
+    "#,
+                &formats.branch_table_base,
+            )
+            .operands_in(vec![JT])
+            .operands_out(vec![addr]),
+        );
+
+        ig.push(
+            Inst::new(
+                "indirect_jump_table_br",
+                r#"
+    Branch indirectly via a jump table entry.
+
+    Unconditionally jump via a jump table entry that was previously loaded
+    with the ``jump_table_entry`` instruction.
+    "#,
+                &formats.indirect_jump,
+            )
+            .operands_in(vec![addr, JT])
+            .is_indirect_branch(true)
+            .is_terminator(true)
+            .is_branch(true),
+        );
+    }
+
+    ig.push(
+        Inst::new(
+            "debugtrap",
+            r#"
+    Encodes an assembly debug trap.
+    "#,
+            &formats.nullary,
+        )
+        .other_side_effects(true)
+        .can_load(true)
+        .can_store(true),
+    );
+
+    {
+        let code = &Operand::new("code", &imm.trapcode);
+        ig.push(
+            Inst::new(
+                "trap",
+                r#"
+        Terminate execution unconditionally.
+        "#,
+                &formats.trap,
+            )
+            .operands_in(vec![code])
+            .can_trap(true)
+            .is_terminator(true),
+        );
+
+        let c = &Operand::new("c", Testable).with_doc("Controlling value to test");
+        ig.push(
+            Inst::new(
+                "trapz",
+                r#"
+        Trap when zero.
+
+        if ``c`` is non-zero, execution continues at the following instruction.
+        "#,
+                &formats.cond_trap,
+            )
+            .operands_in(vec![c, code])
+            .can_trap(true),
+        );
+
+        ig.push(
+            Inst::new(
+                "resumable_trap",
+                r#"
+        A resumable trap.
+
+        This instruction allows non-conditional traps to be used as non-terminal instructions.
+        "#,
+                &formats.trap,
+            )
+            .operands_in(vec![code])
+            .can_trap(true),
+        );
+
+        let c = &Operand::new("c", Testable).with_doc("Controlling value to test");
+        ig.push(
+            Inst::new(
+                "trapnz",
+                r#"
+        Trap when non-zero.
+
+        if ``c`` is zero, execution continues at the following instruction.
+        "#,
+                &formats.cond_trap,
+            )
+            .operands_in(vec![c, code])
+            .can_trap(true),
+        );
+
+        let Cond = &Operand::new("Cond", &imm.intcc);
+        let f = &Operand::new("f", iflags);
+        ig.push(
+            Inst::new(
+                "trapif",
+                r#"
+        Trap when condition is true in integer CPU flags.
+        "#,
+                &formats.int_cond_trap,
+            )
+            .operands_in(vec![Cond, f, code])
+            .can_trap(true),
+        );
+
+        let Cond = &Operand::new("Cond", &imm.floatcc);
+        let f = &Operand::new("f", fflags);
+        let code = &Operand::new("code", &imm.trapcode);
+        ig.push(
+            Inst::new(
+                "trapff",
+                r#"
+        Trap when condition is true in floating point CPU flags.
+        "#,
+                &formats.float_cond_trap,
+            )
+            .operands_in(vec![Cond, f, code])
+            .can_trap(true),
+        );
+    }
+
+    let rvals = &Operand::new("rvals", &entities.varargs).with_doc("return values");
+    ig.push(
+        Inst::new(
+            "return",
+            r#"
+        Return from the function.
+
+        Unconditionally transfer control to the calling function, passing the
+        provided return values. The list of return values must match the
+        function signature's return types.
+        "#,
+            &formats.multiary,
+        )
+        .operands_in(vec![rvals])
+        .is_return(true)
+        .is_terminator(true),
+    );
+
+    let rvals = &Operand::new("rvals", &entities.varargs).with_doc("return values");
+    ig.push(
+        Inst::new(
+            "fallthrough_return",
+            r#"
+        Return from the function by fallthrough.
+
+        This is a specialized instruction for use where one wants to append
+        a custom epilogue, which will then perform the real return. This
+        instruction has no encoding.
+        "#,
+            &formats.multiary,
+        )
+        .operands_in(vec![rvals])
+        .is_return(true)
+        .is_terminator(true),
+    );
+
+    let FN = &Operand::new("FN", &entities.func_ref)
+        .with_doc("function to call, declared by `function`");
+    let args = &Operand::new("args", &entities.varargs).with_doc("call arguments");
+    let rvals = &Operand::new("rvals", &entities.varargs).with_doc("return values");
+    ig.push(
+        Inst::new(
+            "call",
+            r#"
+        Direct function call.
+
+        Call a function which has been declared in the preamble. The argument
+        types must match the function's signature.
+        "#,
+            &formats.call,
+        )
+        .operands_in(vec![FN, args])
+        .operands_out(vec![rvals])
+        .is_call(true),
+    );
+
+    let SIG = &Operand::new("SIG", &entities.sig_ref).with_doc("function signature");
+    let callee = &Operand::new("callee", iAddr).with_doc("address of function to call");
+    let args = &Operand::new("args", &entities.varargs).with_doc("call arguments");
+    let rvals = &Operand::new("rvals", &entities.varargs).with_doc("return values");
+    ig.push(
+        Inst::new(
+            "call_indirect",
+            r#"
+        Indirect function call.
+
+        Call the function pointed to by `callee` with the given arguments. The
+        called function must match the specified signature.
+
+        Note that this is different from WebAssembly's ``call_indirect``; the
+        callee is a native address, rather than a table index. For WebAssembly,
+        `table_addr` and `load` are used to obtain a native address
+        from a table.
+        "#,
+            &formats.call_indirect,
+        )
+        .operands_in(vec![SIG, callee, args])
+        .operands_out(vec![rvals])
+        .is_call(true),
+    );
+
+    let FN = &Operand::new("FN", &entities.func_ref)
+        .with_doc("function to call, declared by `function`");
+    let addr = &Operand::new("addr", iAddr);
+    ig.push(
+        Inst::new(
+            "func_addr",
+            r#"
+        Get the address of a function.
+
+        Compute the absolute address of a function declared in the preamble.
+        The returned address can be used as a ``callee`` argument to
+        `call_indirect`. This is also a method for calling functions that
+        are too far away to be addressable by a direct `call`
+        instruction.
+        "#,
+            &formats.func_addr,
+        )
+        .operands_in(vec![FN])
+        .operands_out(vec![addr]),
+    );
+}
+
 #[allow(clippy::many_single_char_names)]
 pub(crate) fn define(
     all_instructions: &mut AllInstructions,
@@ -20,6 +490,8 @@ pub(crate) fn define(
 ) -> InstructionGroup {
     let mut ig = InstructionGroupBuilder::new(all_instructions);
 
+    define_control_flow(&mut ig, formats, imm, entities);
+
     // Operand kind shorthands.
     let iflags: &TypeVar = &ValueType::Special(types::Flag::IFlags.into()).into();
     let fflags: &TypeVar = &ValueType::Special(types::Flag::FFlags.into()).into();
@@ -114,426 +586,6 @@ pub(crate) fn define(
     let MemTo = &TypeVar::copy_from(Mem, "MemTo".to_string());
 
     let addr = &Operand::new("addr", iAddr);
-    let c = &Operand::new("c", Testable).with_doc("Controlling value to test");
-    let Cond = &Operand::new("Cond", &imm.intcc);
-    let x = &Operand::new("x", iB);
-    let y = &Operand::new("y", iB);
-    let EBB = &Operand::new("EBB", &entities.ebb).with_doc("Destination extended basic block");
-    let args = &Operand::new("args", &entities.varargs).with_doc("EBB arguments");
-
-    ig.push(
-        Inst::new(
-            "jump",
-            r#"
-        Jump.
-
-        Unconditionally jump to an extended basic block, passing the specified
-        EBB arguments. The number and types of arguments must match the
-        destination EBB.
-        "#,
-            &formats.jump,
-        )
-        .operands_in(vec![EBB, args])
-        .is_terminator(true)
-        .is_branch(true),
-    );
-
-    ig.push(
-        Inst::new(
-            "fallthrough",
-            r#"
-        Fall through to the next EBB.
-
-        This is the same as `jump`, except the destination EBB must be
-        the next one in the layout.
-
-        Jumps are turned into fall-through instructions by the branch
-        relaxation pass. There is no reason to use this instruction outside
-        that pass.
-        "#,
-            &formats.jump,
-        )
-        .operands_in(vec![EBB, args])
-        .is_terminator(true)
-        .is_branch(true),
-    );
-
-    ig.push(
-        Inst::new(
-            "brz",
-            r#"
-        Branch when zero.
-
-        If ``c`` is a `b1` value, take the branch when ``c`` is false. If
-        ``c`` is an integer value, take the branch when ``c = 0``.
-        "#,
-            &formats.branch,
-        )
-        .operands_in(vec![c, EBB, args])
-        .is_branch(true),
-    );
-
-    ig.push(
-        Inst::new(
-            "brnz",
-            r#"
-        Branch when non-zero.
-
-        If ``c`` is a `b1` value, take the branch when ``c`` is true. If
-        ``c`` is an integer value, take the branch when ``c != 0``.
-        "#,
-            &formats.branch,
-        )
-        .operands_in(vec![c, EBB, args])
-        .is_branch(true),
-    );
-
-    ig.push(
-        Inst::new(
-            "br_icmp",
-            r#"
-        Compare scalar integers and branch.
-
-        Compare ``x`` and ``y`` in the same way as the `icmp` instruction
-        and take the branch if the condition is true:
-
-        ```text
-            br_icmp ugt v1, v2, ebb4(v5, v6)
-        ```
-
-        is semantically equivalent to:
-
-        ```text
-            v10 = icmp ugt, v1, v2
-            brnz v10, ebb4(v5, v6)
-        ```
-
-        Some RISC architectures like MIPS and RISC-V provide instructions that
-        implement all or some of the condition codes. The instruction can also
-        be used to represent *macro-op fusion* on architectures like Intel's.
-        "#,
-            &formats.branch_icmp,
-        )
-        .operands_in(vec![Cond, x, y, EBB, args])
-        .is_branch(true),
-    );
-
-    let f = &Operand::new("f", iflags);
-
-    ig.push(
-        Inst::new(
-            "brif",
-            r#"
-        Branch when condition is true in integer CPU flags.
-        "#,
-            &formats.branch_int,
-        )
-        .operands_in(vec![Cond, f, EBB, args])
-        .is_branch(true),
-    );
-
-    let Cond = &Operand::new("Cond", &imm.floatcc);
-    let f = &Operand::new("f", fflags);
-
-    ig.push(
-        Inst::new(
-            "brff",
-            r#"
-        Branch when condition is true in floating point CPU flags.
-        "#,
-            &formats.branch_float,
-        )
-        .operands_in(vec![Cond, f, EBB, args])
-        .is_branch(true),
-    );
-
-    // The index into the br_table can be any type; legalizer will convert it to the right type.
-    let x = &Operand::new("x", iB).with_doc("index into jump table");
-    let entry = &Operand::new("entry", iAddr).with_doc("entry of jump table");
-    let JT = &Operand::new("JT", &entities.jump_table);
-
-    ig.push(
-        Inst::new(
-            "br_table",
-            r#"
-        Indirect branch via jump table.
-
-        Use ``x`` as an unsigned index into the jump table ``JT``. If a jump
-        table entry is found, branch to the corresponding EBB. If no entry was
-        found or the index is out-of-bounds, branch to the given default EBB.
-
-        Note that this branch instruction can't pass arguments to the targeted
-        blocks. Split critical edges as needed to work around this.
-
-        Do not confuse this with "tables" in WebAssembly. ``br_table`` is for
-        jump tables with destinations within the current function only -- think
-        of a ``match`` in Rust or a ``switch`` in C.  If you want to call a
-        function in a dynamic library, that will typically use
-        ``call_indirect``.
-        "#,
-            &formats.branch_table,
-        )
-        .operands_in(vec![x, EBB, JT])
-        .is_terminator(true)
-        .is_branch(true),
-    );
-
-    // These are the instructions which br_table legalizes to: they perform address computations,
-    // using pointer-sized integers, so their type variables are more constrained.
-    let x = &Operand::new("x", iAddr).with_doc("index into jump table");
-    let Size = &Operand::new("Size", &imm.uimm8).with_doc("Size in bytes");
-
-    ig.push(
-        Inst::new(
-            "jump_table_entry",
-            r#"
-    Get an entry from a jump table.
-
-    Load a serialized ``entry`` from a jump table ``JT`` at a given index
-    ``addr`` with a specific ``Size``. The retrieved entry may need to be
-    decoded after loading, depending upon the jump table type used.
-
-    Currently, the only type supported is entries which are relative to the
-    base of the jump table.
-    "#,
-            &formats.branch_table_entry,
-        )
-        .operands_in(vec![x, addr, Size, JT])
-        .operands_out(vec![entry])
-        .can_load(true),
-    );
-
-    ig.push(
-        Inst::new(
-            "jump_table_base",
-            r#"
-    Get the absolute base address of a jump table.
-
-    This is used for jump tables wherein the entries are stored relative to
-    the base of jump table. In order to use these, generated code should first
-    load an entry using ``jump_table_entry``, then use this instruction to add
-    the relative base back to it.
-    "#,
-            &formats.branch_table_base,
-        )
-        .operands_in(vec![JT])
-        .operands_out(vec![addr]),
-    );
-
-    ig.push(
-        Inst::new(
-            "indirect_jump_table_br",
-            r#"
-    Branch indirectly via a jump table entry.
-
-    Unconditionally jump via a jump table entry that was previously loaded
-    with the ``jump_table_entry`` instruction.
-    "#,
-            &formats.indirect_jump,
-        )
-        .operands_in(vec![addr, JT])
-        .is_indirect_branch(true)
-        .is_terminator(true)
-        .is_branch(true),
-    );
-
-    ig.push(
-        Inst::new(
-            "debugtrap",
-            r#"
-    Encodes an assembly debug trap.
-    "#,
-            &formats.nullary,
-        )
-        .other_side_effects(true)
-        .can_load(true)
-        .can_store(true),
-    );
-
-    let code = &Operand::new("code", &imm.trapcode);
-
-    ig.push(
-        Inst::new(
-            "trap",
-            r#"
-        Terminate execution unconditionally.
-        "#,
-            &formats.trap,
-        )
-        .operands_in(vec![code])
-        .can_trap(true)
-        .is_terminator(true),
-    );
-
-    ig.push(
-        Inst::new(
-            "trapz",
-            r#"
-        Trap when zero.
-
-        if ``c`` is non-zero, execution continues at the following instruction.
-        "#,
-            &formats.cond_trap,
-        )
-        .operands_in(vec![c, code])
-        .can_trap(true),
-    );
-
-    ig.push(
-        Inst::new(
-            "resumable_trap",
-            r#"
-        A resumable trap.
-
-        This instruction allows non-conditional traps to be used as non-terminal instructions.
-        "#,
-            &formats.trap,
-        )
-        .operands_in(vec![code])
-        .can_trap(true),
-    );
-
-    ig.push(
-        Inst::new(
-            "trapnz",
-            r#"
-        Trap when non-zero.
-
-        if ``c`` is zero, execution continues at the following instruction.
-        "#,
-            &formats.cond_trap,
-        )
-        .operands_in(vec![c, code])
-        .can_trap(true),
-    );
-
-    let Cond = &Operand::new("Cond", &imm.intcc);
-    let f = &Operand::new("f", iflags);
-
-    ig.push(
-        Inst::new(
-            "trapif",
-            r#"
-        Trap when condition is true in integer CPU flags.
-        "#,
-            &formats.int_cond_trap,
-        )
-        .operands_in(vec![Cond, f, code])
-        .can_trap(true),
-    );
-
-    let Cond = &Operand::new("Cond", &imm.floatcc);
-    let f = &Operand::new("f", fflags);
-
-    ig.push(
-        Inst::new(
-            "trapff",
-            r#"
-        Trap when condition is true in floating point CPU flags.
-        "#,
-            &formats.float_cond_trap,
-        )
-        .operands_in(vec![Cond, f, code])
-        .can_trap(true),
-    );
-
-    let rvals = &Operand::new("rvals", &entities.varargs).with_doc("return values");
-
-    ig.push(
-        Inst::new(
-            "return",
-            r#"
-        Return from the function.
-
-        Unconditionally transfer control to the calling function, passing the
-        provided return values. The list of return values must match the
-        function signature's return types.
-        "#,
-            &formats.multiary,
-        )
-        .operands_in(vec![rvals])
-        .is_return(true)
-        .is_terminator(true),
-    );
-
-    ig.push(
-        Inst::new(
-            "fallthrough_return",
-            r#"
-        Return from the function by fallthrough.
-
-        This is a specialized instruction for use where one wants to append
-        a custom epilogue, which will then perform the real return. This
-        instruction has no encoding.
-        "#,
-            &formats.multiary,
-        )
-        .operands_in(vec![rvals])
-        .is_return(true)
-        .is_terminator(true),
-    );
-
-    let FN = &Operand::new("FN", &entities.func_ref)
-        .with_doc("function to call, declared by `function`");
-    let args = &Operand::new("args", &entities.varargs).with_doc("call arguments");
-
-    ig.push(
-        Inst::new(
-            "call",
-            r#"
-        Direct function call.
-
-        Call a function which has been declared in the preamble. The argument
-        types must match the function's signature.
-        "#,
-            &formats.call,
-        )
-        .operands_in(vec![FN, args])
-        .operands_out(vec![rvals])
-        .is_call(true),
-    );
-
-    let SIG = &Operand::new("SIG", &entities.sig_ref).with_doc("function signature");
-    let callee = &Operand::new("callee", iAddr).with_doc("address of function to call");
-
-    ig.push(
-        Inst::new(
-            "call_indirect",
-            r#"
-        Indirect function call.
-
-        Call the function pointed to by `callee` with the given arguments. The
-        called function must match the specified signature.
-
-        Note that this is different from WebAssembly's ``call_indirect``; the
-        callee is a native address, rather than a table index. For WebAssembly,
-        `table_addr` and `load` are used to obtain a native address
-        from a table.
-        "#,
-            &formats.call_indirect,
-        )
-        .operands_in(vec![SIG, callee, args])
-        .operands_out(vec![rvals])
-        .is_call(true),
-    );
-
-    ig.push(
-        Inst::new(
-            "func_addr",
-            r#"
-        Get the address of a function.
-
-        Compute the absolute address of a function declared in the preamble.
-        The returned address can be used as a ``callee`` argument to
-        `call_indirect`. This is also a method for calling functions that
-        are too far away to be addressable by a direct `call`
-        instruction.
-        "#,
-            &formats.func_addr,
-        )
-        .operands_in(vec![FN])
-        .operands_out(vec![addr]),
-    );
 
     let SS = &Operand::new("SS", &entities.stack_slot);
     let Offset = &Operand::new("Offset", &imm.offset32).with_doc("Byte offset from base address");

From 04d233301c56d15b476f19488475847f56fa32af Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Wed, 8 Jan 2020 09:10:53 -0800
Subject: [PATCH 2997/3084] Require the `Send` trait for `TargetIsa`

The `TargetIsa` trait already requires that the implementor is `Sync` to
be shared across threads, and this commit adds in an additional
restriction of `Send` to ensure that the type can be sent-by-value
across threads as well.

This is part of an effort to make various data structures in `wasmtime`
sendable/shareable across threads.
---
 cranelift/codegen/src/isa/mod.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index 5679333394..4b0a3e7de7 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -198,7 +198,7 @@ impl TargetFrontendConfig {
 
 /// Methods that are specialized to a target ISA. Implies a Display trait that shows the
 /// shared flags, as well as any isa-specific flags.
-pub trait TargetIsa: fmt::Display + Sync {
+pub trait TargetIsa: fmt::Display + Send + Sync {
     /// Get the name of this ISA.
     fn name(&self) -> &'static str;
 

From 43f1e05156bce744ec3d4e57e53497c9bc3bf7af Mon Sep 17 00:00:00 2001
From: Philip Craig 
Date: Mon, 30 Dec 2019 19:31:47 +1000
Subject: [PATCH 2998/3084] Update target-lexicon to 0.10

---
 cranelift/Cargo.toml            |  2 +-
 cranelift/codegen/Cargo.toml    |  2 +-
 cranelift/faerie/Cargo.toml     |  6 +++---
 cranelift/faerie/src/backend.rs |  2 +-
 cranelift/frontend/Cargo.toml   |  2 +-
 cranelift/fuzz/Cargo.toml       |  2 +-
 cranelift/native/Cargo.toml     |  2 +-
 cranelift/object/Cargo.toml     |  4 ++--
 cranelift/object/src/backend.rs | 15 ++++++++++-----
 cranelift/reader/Cargo.toml     |  2 +-
 cranelift/simplejit/Cargo.toml  |  2 +-
 cranelift/wasm/Cargo.toml       |  2 +-
 12 files changed, 24 insertions(+), 19 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 7f78b3ddd2..d446b97f23 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -39,7 +39,7 @@ serde = "1.0.8"
 term = "0.6.1"
 capstone = { version = "0.6.0", optional = true }
 wabt = { version = "0.9.1", optional = true }
-target-lexicon = "0.9"
+target-lexicon = "0.10"
 pretty_env_logger = "0.3.0"
 file-per-thread-logger = "0.1.2"
 indicatif = "0.13.0"
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index bf85cf02cf..4c501d8302 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -17,7 +17,7 @@ cranelift-codegen-shared = { path = "./shared", version = "0.52.0" }
 cranelift-entity = { path = "../cranelift-entity", version = "0.52.0" }
 cranelift-bforest = { path = "../cranelift-bforest", version = "0.52.0" }
 hashbrown = { version = "0.6", optional = true }
-target-lexicon = "0.9"
+target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
 smallvec = { version = "1.0.0" }
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index c7321c3e8c..198676d057 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -11,10 +11,10 @@ edition = "2018"
 
 [dependencies]
 cranelift-module = { path = "../cranelift-module", version = "0.52.0" }
-faerie = "0.12.0"
+faerie = "0.14.0"
 goblin = "0.1.0"
-failure = "0.1.2"
-target-lexicon = "0.9"
+anyhow = "1.0"
+target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index 14d8245370..aca9196bc5 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -2,6 +2,7 @@
 
 use crate::container;
 use crate::traps::{FaerieTrapManifest, FaerieTrapSink};
+use anyhow::Error;
 use cranelift_codegen::binemit::{
     Addend, CodeOffset, NullStackmapSink, NullTrapSink, Reloc, RelocSink, Stackmap, StackmapSink,
 };
@@ -12,7 +13,6 @@ use cranelift_module::{
     ModuleNamespace, ModuleResult,
 };
 use faerie;
-use failure::Error;
 use std::fs::File;
 use target_lexicon::Triple;
 
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index f8e4a0b66a..544eb0a553 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -12,7 +12,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
-target-lexicon = "0.9"
+target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 smallvec = { version = "1.0.0" }
diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
index 8d9cdc0ffa..d2dca75c64 100644
--- a/cranelift/fuzz/Cargo.toml
+++ b/cranelift/fuzz/Cargo.toml
@@ -15,7 +15,7 @@ libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" }
 cranelift-codegen = { path = "../cranelift-codegen" }
 cranelift-wasm = { path = "../cranelift-wasm" }
 cranelift-reader = { path = "../cranelift-reader" }
-target-lexicon = "0.9"
+target-lexicon = "0.10"
 
 # Prevent this from interfering with workspaces
 [workspace]
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 5bc12aa0f3..ee954f21f1 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -11,7 +11,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
-target-lexicon = "0.9"
+target-lexicon = "0.10"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
 raw-cpuid = "7.0.3"
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 95f3aa195c..917490cc70 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -11,8 +11,8 @@ edition = "2018"
 
 [dependencies]
 cranelift-module = { path = "../cranelift-module", version = "0.52.0" }
-object = { version = "0.16", default-features = false, features = ["write"] }
-target-lexicon = "0.9"
+object = { version = "0.17", default-features = false, features = ["write"] }
+target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
diff --git a/cranelift/object/src/backend.rs b/cranelift/object/src/backend.rs
index 37b4c6bff2..28874cb521 100644
--- a/cranelift/object/src/backend.rs
+++ b/cranelift/object/src/backend.rs
@@ -11,8 +11,10 @@ use cranelift_module::{
     Backend, DataContext, DataDescription, DataId, FuncId, Init, Linkage, ModuleNamespace,
     ModuleResult,
 };
-use object::write::{Object, Relocation, SectionId, StandardSection, Symbol, SymbolId};
-use object::{RelocationEncoding, RelocationKind, SymbolKind, SymbolScope};
+use object::write::{
+    Object, Relocation, SectionId, StandardSection, Symbol, SymbolId, SymbolSection,
+};
+use object::{RelocationEncoding, RelocationKind, SymbolFlags, SymbolKind, SymbolScope};
 use std::collections::HashMap;
 use target_lexicon::PointerWidth;
 
@@ -133,7 +135,8 @@ impl Backend for ObjectBackend {
                 kind: SymbolKind::Text,
                 scope,
                 weak,
-                section: None,
+                section: SymbolSection::Undefined,
+                flags: SymbolFlags::None,
             });
             self.functions[id] = Some(symbol_id);
         }
@@ -161,7 +164,8 @@ impl Backend for ObjectBackend {
                 kind: SymbolKind::Data,
                 scope,
                 weak,
-                section: None,
+                section: SymbolSection::Undefined,
+                flags: SymbolFlags::None,
             });
             self.data_objects[id] = Some(symbol_id);
         }
@@ -432,7 +436,8 @@ impl ObjectBackend {
                         kind: SymbolKind::Text,
                         scope: SymbolScope::Unknown,
                         weak: false,
-                        section: None,
+                        section: SymbolSection::Undefined,
+                        flags: SymbolFlags::None,
                     });
                     self.libcalls.insert(*libcall, symbol);
                     symbol
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index cf489e19d3..d212016408 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -11,7 +11,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0" }
-target-lexicon = "0.9"
+target-lexicon = "0.10"
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 613a116bee..0a277e9d9b 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -15,7 +15,7 @@ cranelift-native = { path = "../cranelift-native", version = "0.52.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
-target-lexicon = "0.9"
+target-lexicon = "0.10"
 memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 6c9b77ecb6..11ac403f0f 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -22,7 +22,7 @@ thiserror = "1.0.4"
 
 [dev-dependencies]
 wabt = "0.9.1"
-target-lexicon = "0.9"
+target-lexicon = "0.10"
 
 [features]
 default = ["std", "basic-blocks"]

From d765677fcc501da10b702f62dae02195fb100f44 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 10 Jan 2020 04:40:25 -0800
Subject: [PATCH 2999/3084] Add a `is_wasm_parameter` method to the wasm
 `FuncEnvironment`. (#1329)

This provides a more flexible way to allow embedding to tell
cranelift-wasm which function parameters are hidden, and which should be
translated as wasm user variables.

This replaces https://github.com/bytecodealliance/cranelift/pull/1086.
---
 cranelift/wasm/src/environ/dummy.rs   |  4 ++++
 cranelift/wasm/src/environ/spec.rs    |  4 ++++
 cranelift/wasm/src/func_translator.rs | 12 ++++++++----
 3 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index 6b6f210166..cb6d78e9f9 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -201,6 +201,10 @@ impl<'dummy_environment> TargetEnvironment for DummyFuncEnvironment<'dummy_envir
 }
 
 impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environment> {
+    fn is_wasm_parameter(&self, func: &ir::Function, index: usize) -> bool {
+        func.signature.params[index].purpose == ir::ArgumentPurpose::Normal
+    }
+
     fn return_mode(&self) -> ReturnMode {
         self.return_mode
     }
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 5a38d7da62..446a637020 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -138,6 +138,10 @@ pub trait TargetEnvironment {
 /// IR. The function environment provides information about the WebAssembly module as well as the
 /// runtime environment.
 pub trait FuncEnvironment: TargetEnvironment {
+    /// Is the given parameter of the given function a wasm-level parameter, as opposed to a hidden
+    /// parameter added for use by the implementation?
+    fn is_wasm_parameter(&self, func: &ir::Function, index: usize) -> bool;
+
     /// Should the code be structured to use a single `fallthrough_return` instruction at the end
     /// of the function body, rather than `return` instructions as needed? This is used by VMs
     /// to append custom epilogues.
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index b6a46ecb01..2e7c1bc766 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -99,7 +99,7 @@ impl FuncTranslator {
         // `environ`. The callback functions may need to insert things in the entry block.
         builder.ensure_inserted_ebb();
 
-        let num_params = declare_wasm_parameters(&mut builder, entry_block);
+        let num_params = declare_wasm_parameters(&mut builder, entry_block, environ);
 
         // Set up the translation state with a single pushed control block representing the whole
         // function and its return values.
@@ -124,14 +124,18 @@ impl FuncTranslator {
 /// Declare local variables for the signature parameters that correspond to WebAssembly locals.
 ///
 /// Return the number of local variables declared.
-fn declare_wasm_parameters(builder: &mut FunctionBuilder, entry_block: Ebb) -> usize {
+fn declare_wasm_parameters(
+    builder: &mut FunctionBuilder,
+    entry_block: Ebb,
+    environ: &FE,
+) -> usize {
     let sig_len = builder.func.signature.params.len();
     let mut next_local = 0;
     for i in 0..sig_len {
         let param_type = builder.func.signature.params[i];
-        // There may be additional special-purpose parameters following the normal WebAssembly
+        // There may be additional special-purpose parameters in addition to the normal WebAssembly
         // signature parameters. For example, a `vmctx` pointer.
-        if param_type.purpose == ir::ArgumentPurpose::Normal {
+        if environ.is_wasm_parameter(&builder.func, i) {
             // This is a normal WebAssembly signature parameter, so create a local for it.
             let local = Variable::new(next_local);
             builder.declare_var(local, param_type.value_type);

From 71914c7668f05531767aec361d077fc7c90ac00a Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 8 Jan 2020 10:12:41 -0800
Subject: [PATCH 3000/3084] Fix clippy warnings in EncodingBits

---
 .../codegen/shared/src/isa/x86/encoding_bits.rs  | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)

diff --git a/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs b/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs
index ba6418f7cf..9edb2a6e6f 100644
--- a/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs
+++ b/cranelift/codegen/shared/src/isa/x86/encoding_bits.rs
@@ -59,20 +59,18 @@ impl EncodingBits {
 
     /// Returns a copy of the EncodingBits with the RRR bits set.
     #[inline]
-    pub fn with_rrr(self, rrr: u8) -> Self {
-        debug_assert_eq!(u8::from(self.rrr()), 0);
-        let mut enc = self.clone();
-        enc.write(RRR, rrr.into());
-        enc
+    pub fn with_rrr(mut self, rrr: u8) -> Self {
+        debug_assert_eq!(self.rrr(), 0);
+        self.write(RRR, rrr.into());
+        self
     }
 
     /// Returns a copy of the EncodingBits with the REX.W bit set.
     #[inline]
-    pub fn with_rex_w(self) -> Self {
+    pub fn with_rex_w(mut self) -> Self {
         debug_assert_eq!(self.rex_w(), 0);
-        let mut enc = self.clone();
-        enc.write(REX_W, 1);
-        enc
+        self.write(REX_W, 1);
+        self
     }
 
     /// Returns the raw bits.

From e8c3302bc5b31316b638531386233ae30035a0e0 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 9 Jan 2020 11:46:21 -0800
Subject: [PATCH 3001/3084] Fix some additional clippy warnings

---
 cranelift/codegen/src/ir/framelayout.rs  | 2 +-
 cranelift/codegen/src/isa/x86/binemit.rs | 4 ++--
 cranelift/codegen/src/simple_gvn.rs      | 3 ++-
 cranelift/wasm/src/environ/spec.rs       | 3 +++
 4 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/src/ir/framelayout.rs b/cranelift/codegen/src/ir/framelayout.rs
index 1b914a54f0..983b209279 100644
--- a/cranelift/codegen/src/ir/framelayout.rs
+++ b/cranelift/codegen/src/ir/framelayout.rs
@@ -56,7 +56,7 @@ pub struct FrameLayout {
 impl FrameLayout {
     /// Create instance of FrameLayout.
     pub fn new() -> Self {
-        FrameLayout {
+        Self {
             initial: vec![].into_boxed_slice(),
             instructions: HashMap::new(),
         }
diff --git a/cranelift/codegen/src/isa/x86/binemit.rs b/cranelift/codegen/src/isa/x86/binemit.rs
index fa67e5efff..5a373e6f96 100644
--- a/cranelift/codegen/src/isa/x86/binemit.rs
+++ b/cranelift/codegen/src/isa/x86/binemit.rs
@@ -64,7 +64,7 @@ fn rex3(rm: RegUnit, reg: RegUnit, index: RegUnit) -> u8 {
 /// Determines whether a REX prefix should be emitted.
 #[inline]
 fn needs_rex(bits: u16, rex: u8) -> bool {
-    rex != BASE_REX || u8::from(EncodingBits::from(bits).rex_w()) == 1
+    rex != BASE_REX || EncodingBits::from(bits).rex_w() == 1
 }
 
 // Emit a REX prefix.
@@ -74,7 +74,7 @@ fn needs_rex(bits: u16, rex: u8) -> bool {
 fn rex_prefix(bits: u16, rex: u8, sink: &mut CS) {
     debug_assert_eq!(rex & 0xf8, BASE_REX);
     let w = EncodingBits::from(bits).rex_w();
-    sink.put1(rex | (u8::from(w) << 3));
+    sink.put1(rex | (w << 3));
 }
 
 // Emit a single-byte opcode with no REX prefix.
diff --git a/cranelift/codegen/src/simple_gvn.rs b/cranelift/codegen/src/simple_gvn.rs
index 2130634c47..ff8696aee6 100644
--- a/cranelift/codegen/src/simple_gvn.rs
+++ b/cranelift/codegen/src/simple_gvn.rs
@@ -124,7 +124,8 @@ pub fn do_simple_gvn(func: &mut Function, domtree: &mut DominatorTree) {
             use crate::scoped_hash_map::Entry::*;
             match visible_values.entry(key) {
                 Occupied(entry) => {
-                    debug_assert!(domtree.dominates(*entry.get(), inst, &func.layout));
+                    let layout = &func.layout;
+                    debug_assert!(domtree.dominates(*entry.get(), inst, layout));
                     // If the redundant instruction is representing the current
                     // scope, pick a new representative.
                     let old = scope_stack.last_mut().unwrap();
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 446a637020..71f22c8b39 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -306,6 +306,7 @@ pub trait FuncEnvironment: TargetEnvironment {
     /// The `index` provided identifies the linear memory to query, and `heap` is the heap reference
     /// returned by `make_heap` for the same index. `seg_index` is the index of the segment to copy
     /// from.
+    #[allow(clippy::too_many_arguments)]
     fn translate_memory_init(
         &mut self,
         pos: FuncCursor,
@@ -329,6 +330,7 @@ pub trait FuncEnvironment: TargetEnvironment {
     ) -> WasmResult;
 
     /// Translate a `table.copy` WebAssembly instruction.
+    #[allow(clippy::too_many_arguments)]
     fn translate_table_copy(
         &mut self,
         pos: FuncCursor,
@@ -342,6 +344,7 @@ pub trait FuncEnvironment: TargetEnvironment {
     ) -> WasmResult<()>;
 
     /// Translate a `table.init` WebAssembly instruction.
+    #[allow(clippy::too_many_arguments)]
     fn translate_table_init(
         &mut self,
         pos: FuncCursor,

From c7b0d495506976f989becd3f5d3ffc29af89d6c3 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 10 Jan 2020 08:41:11 -0800
Subject: [PATCH 3002/3084] Fix V128AndNot type mapping (#1319)

Prior to this fix, `type_of` could not determine the Cranelift type for the parameters to `V128AndNot`.
---
 cranelift/wasm/src/code_translator.rs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 68f9cac1ba..839a218dae 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1715,6 +1715,7 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::V128Const { .. }
         | Operator::V128Not
         | Operator::V128And
+        | Operator::V128AndNot
         | Operator::V128Or
         | Operator::V128Xor
         | Operator::V128Bitselect => I8X16, // default type representing V128
@@ -1858,7 +1859,8 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::I64x2TruncSatF64x2U => F64X2,
 
         _ => unimplemented!(
-            "Currently only the SIMD instructions are translated to their return type: {:?}",
+            "Currently only SIMD instructions are mapped to their return type; the \
+             following instruction is not mapped: {:?}",
             operator
         ),
     }

From 250ea0e5dbfbd5cde668e93bcf695bfb5b658a91 Mon Sep 17 00:00:00 2001
From: data-pup 
Date: Fri, 10 Jan 2020 13:40:27 -0500
Subject: [PATCH 3003/3084] Verify that EBBs are non-empty (#1330)

Co-authored-by: Joshua Nelson 
---
 cranelift/codegen/src/verifier/mod.rs | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 2e58ec4140..6d97d9af4e 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -1971,6 +1971,9 @@ impl<'a> Verifier<'a> {
         self.typecheck_function_signature(errors)?;
 
         for ebb in self.func.layout.ebbs() {
+            if self.func.layout.first_inst(ebb).is_none() {
+                return errors.fatal((ebb, format!("{} cannot be empty", ebb)));
+            }
             for inst in self.func.layout.ebb_insts(ebb) {
                 self.ebb_integrity(ebb, inst, errors)?;
                 self.instruction_integrity(inst, errors)?;
@@ -2117,4 +2120,18 @@ mod tests {
             "inst0 (v0, v1 = iconst.i32 42): has more result values than expected"
         )
     }
+
+    #[test]
+    fn test_empty_ebb() {
+        let mut func = Function::new();
+        let ebb0 = func.dfg.make_ebb();
+        func.layout.append_ebb(ebb0);
+
+        let flags = &settings::Flags::new(settings::builder());
+        let verifier = Verifier::new(&func, flags.into());
+        let mut errors = VerifierErrors::default();
+        let _ = verifier.run(&mut errors);
+
+        assert_err_with_msg!(errors, "ebb0 cannot be empty");
+    }
 }

From 582e7942f8a0021914bab8b0415463dd44fe4a01 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 10 Jan 2020 14:08:07 -0800
Subject: [PATCH 3004/3084] Bump version to 0.54.0 (#1333)

---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  2 +-
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 19 files changed, 68 insertions(+), 68 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index d446b97f23..23d846ea05 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.52.0"
+version = "0.54.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.52.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.52.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.52.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.52.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.52.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.52.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.52.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.52.0" }
-cranelift-module = { path = "cranelift-module", version = "0.52.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.52.0" }
-cranelift-object = { path = "cranelift-object", version = "0.52.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.52.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.52.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.52.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.54.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.54.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.54.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.54.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.54.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.54.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.54.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.54.0" }
+cranelift-module = { path = "cranelift-module", version = "0.54.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.54.0" }
+cranelift-object = { path = "cranelift-object", version = "0.54.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.54.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.54.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.54.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 463fc12363..5f78dc0d1c 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.52.0"
+version = "0.54.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.52.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.54.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 4c501d8302..831c24f109 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.52.0"
+version = "0.54.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.52.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.52.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.52.0" }
+cranelift-codegen-shared = { path = "./shared", version = "0.54.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.54.0" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.54.0" }
 hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
@@ -29,7 +29,7 @@ byteorder = { version = "1.3.2", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.52.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.54.0" }
 
 [features]
 default = ["std", "basic-blocks"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index e804727ae1..42e1ffad0a 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.52.0"
+version = "0.54.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.52.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.52.0" }
+cranelift-codegen-shared = { path = "../shared", version = "0.54.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.54.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 6ffdef7abd..24a2f67acd 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.52.0"
+version = "0.54.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 2027c02b1f..f8d1bc6500 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.52.0"
+version = "0.54.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 198676d057..c919b63543 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.52.0"
+version = "0.54.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.52.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.54.0" }
 faerie = "0.14.0"
 goblin = "0.1.0"
 anyhow = "1.0"
@@ -18,7 +18,7 @@ target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.52.0"
+version = "0.54.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 89a63c1a97..da2a92814b 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.52.0"
+version = "0.54.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.52.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.52.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.52.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.54.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.54.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.54.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 log = "0.4.6"
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 544eb0a553..8fbc6a66e0 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.52.0"
+version = "0.54.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", default-features = false }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index cb0f7cde78..7665c74c94 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.52.0"
+version = "0.54.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.52.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.54.0" }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 thiserror = "1.0.4"
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index ee954f21f1..1e6b08014d 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.52.0"
+version = "0.54.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", default-features = false }
 target-lexicon = "0.10"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 917490cc70..8fd512411f 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.52.0"
+version = "0.54.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,13 +10,13 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.52.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.54.0" }
 object = { version = "0.17", default-features = false, features = ["write"] }
 target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.52.0"
+version = "0.54.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 6006c77453..f7e0462825 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.52.0"
+version = "0.54.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.52.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.54.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index cb7200483d..363824b60c 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.52.0"
+version="0.54.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index d212016408..b37b5d0cc2 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.52.0"
+version = "0.54.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0" }
 target-lexicon = "0.10"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 289ba9f268..15ff9bafd4 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.52.0"
+version = "0.54.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.52.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.54.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 0a277e9d9b..4bf97d44f8 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.52.0"
+version = "0.54.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.52.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.52.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.54.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.54.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -20,7 +20,7 @@ memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.52.0"
+version = "0.54.0"
 default-features = false
 features = ["std"]
 
@@ -32,9 +32,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.52.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.52.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.52.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.54.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.54.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.54.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 30a46c6e63..127a315803 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.52.0"
+version = "0.54.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.52.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.54.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 11ac403f0f..e40977f882 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.52.0"
+version = "0.54.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.45.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.52.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.52.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.52.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.54.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.54.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }

From 3a13f79b663f00d00af05c5fdf0983d3993c478d Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Fri, 10 Jan 2020 16:32:16 -0600
Subject: [PATCH 3005/3084] Try to reduce CI times with a Rust `*.wat` parser
 (#1332)

This commit moves the cranelift tests and tools from the `wabt` crate on
crates.io (which compiles the wabt C++ codebase) to the `wat` crate on
crates.io which is a Rust parser for the `*.wat` format. This was
motivated by me noticing that release builds on Windows are ~5 minutes
longer than Linux builds, and local timing graphs showed that `wabt-sys`
was by far the longest build step in the build process.

This commit changes the `clif-util` binary where the `--enable-simd`
flag is no longer respected with the text format as input, since the
`wat` crate has no feature gating. This was already sort of not
respected, though, since `--enable-simd` wasn't consulted for binary
inputs which `clif-util` supports as well. If this isn't ok though then
it should be ok to close this PR!
---
 cranelift/Cargo.toml                   |  4 +--
 cranelift/src/clif-util.rs             | 24 --------------
 cranelift/src/utils.rs                 | 15 ---------
 cranelift/src/wasm.rs                  | 45 +++++++++-----------------
 cranelift/wasm/Cargo.toml              |  2 +-
 cranelift/wasm/tests/wasm_testsuite.rs | 24 ++++----------
 6 files changed, 24 insertions(+), 90 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 23d846ea05..b16b00876d 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -38,7 +38,7 @@ clap = "2.32.0"
 serde = "1.0.8"
 term = "0.6.1"
 capstone = { version = "0.6.0", optional = true }
-wabt = { version = "0.9.1", optional = true }
+wat = { version = "1.0.7", optional = true }
 target-lexicon = "0.10"
 pretty_env_logger = "0.3.0"
 file-per-thread-logger = "0.1.2"
@@ -48,7 +48,7 @@ walkdir = "2.2"
 [features]
 default = ["disas", "wasm", "cranelift-codegen/all-arch", "basic-blocks"]
 disas = ["capstone"]
-wasm = ["wabt", "cranelift-wasm"]
+wasm = ["wat", "cranelift-wasm"]
 basic-blocks = ["cranelift-codegen/basic-blocks", "cranelift-frontend/basic-blocks",
 "cranelift-wasm/basic-blocks", "cranelift-filetests/basic-blocks"]
 
diff --git a/cranelift/src/clif-util.rs b/cranelift/src/clif-util.rs
index b40f4d26e9..4066ef0fda 100755
--- a/cranelift/src/clif-util.rs
+++ b/cranelift/src/clif-util.rs
@@ -107,24 +107,6 @@ fn add_debug_flag<'a>() -> clap::Arg<'a, 'a> {
         .help("Enable debug output on stderr/stdout")
 }
 
-fn add_enable_simd_flag<'a>() -> clap::Arg<'a, 'a> {
-    Arg::with_name("enable-simd")
-        .long("enable-simd")
-        .help("Enable WASM's SIMD operations")
-}
-
-fn add_enable_multi_value<'a>() -> clap::Arg<'a, 'a> {
-    Arg::with_name("enable-multi-value")
-        .long("enable-multi-value")
-        .help("Enable WASM's multi-value support")
-}
-
-fn add_enable_reference_types_flag<'a>() -> clap::Arg<'a, 'a> {
-    Arg::with_name("enable-reference-types")
-        .long("enable-reference-types")
-        .help("Enable WASM's reference types operations")
-}
-
 fn add_just_decode_flag<'a>() -> clap::Arg<'a, 'a> {
     Arg::with_name("just-decode")
         .short("t")
@@ -167,9 +149,6 @@ fn add_wasm_or_compile<'a>(cmd: &str) -> clap::App<'a, 'a> {
         .arg(add_target_flag())
         .arg(add_input_file_arg())
         .arg(add_debug_flag())
-        .arg(add_enable_simd_flag())
-        .arg(add_enable_multi_value())
-        .arg(add_enable_reference_types_flag())
         .arg(add_just_decode_flag())
         .arg(add_check_translation_flag())
 }
@@ -321,9 +300,6 @@ fn main() {
                     rest_cmd.is_present("print-size"),
                     rest_cmd.is_present("time-passes"),
                     rest_cmd.is_present("value-ranges"),
-                    rest_cmd.is_present("enable-simd"),
-                    rest_cmd.is_present("enable-multi-value"),
-                    rest_cmd.is_present("enable-reference-types"),
                 )
             };
 
diff --git a/cranelift/src/utils.rs b/cranelift/src/utils.rs
index f520aaa000..bb5a2ac485 100644
--- a/cranelift/src/utils.rs
+++ b/cranelift/src/utils.rs
@@ -24,21 +24,6 @@ pub fn read_to_string>(path: P) -> io::Result {
     Ok(buffer)
 }
 
-/// Read an entire file into a vector of bytes.
-#[cfg(feature = "wasm")]
-pub fn read_to_end>(path: P) -> io::Result> {
-    let mut buffer = Vec::new();
-    if path.as_ref() == Path::new("-") {
-        let stdin = io::stdin();
-        let mut stdin = stdin.lock();
-        stdin.read_to_end(&mut buffer)?;
-    } else {
-        let mut file = File::open(path)?;
-        file.read_to_end(&mut buffer)?;
-    }
-    Ok(buffer)
-}
-
 /// Like `FlagsOrIsa`, but holds ownership.
 pub enum OwnedFlagsOrIsa {
     Flags(settings::Flags),
diff --git a/cranelift/src/wasm.rs b/cranelift/src/wasm.rs
index 041dd792b3..f93471393e 100644
--- a/cranelift/src/wasm.rs
+++ b/cranelift/src/wasm.rs
@@ -8,7 +8,7 @@
 )]
 
 use crate::disasm::{print_all, PrintRelocs, PrintStackmaps, PrintTraps};
-use crate::utils::{parse_sets_and_triple, read_to_end};
+use crate::utils::parse_sets_and_triple;
 use cranelift_codegen::ir::DisplayFunctionAnnotations;
 use cranelift_codegen::print_errors::{pretty_error, pretty_verifier_error};
 use cranelift_codegen::settings::FlagsOrIsa;
@@ -16,10 +16,10 @@ use cranelift_codegen::timing;
 use cranelift_codegen::Context;
 use cranelift_entity::EntityRef;
 use cranelift_wasm::{translate_module, DummyEnvironment, FuncIndex, ReturnMode};
+use std::io::Read;
 use std::path::Path;
 use std::path::PathBuf;
 use term;
-use wabt::{wat2wasm_with_features, Features};
 
 macro_rules! vprintln {
     ($x: expr, $($tts:tt)*) => {
@@ -49,9 +49,6 @@ pub fn run(
     flag_print_size: bool,
     flag_report_times: bool,
     flag_calc_value_ranges: bool,
-    flag_enable_simd: bool,
-    flag_enable_multi_value: bool,
-    flag_enable_reference_types: bool,
 ) -> Result<(), String> {
     let parsed = parse_sets_and_triple(flag_set, flag_triple)?;
 
@@ -67,9 +64,6 @@ pub fn run(
             flag_print_disasm,
             flag_report_times,
             flag_calc_value_ranges,
-            flag_enable_simd,
-            flag_enable_multi_value,
-            flag_enable_reference_types,
             &path.to_path_buf(),
             &name,
             parsed.as_fisa(),
@@ -87,9 +81,6 @@ fn handle_module(
     flag_print_disasm: bool,
     flag_report_times: bool,
     flag_calc_value_ranges: bool,
-    flag_enable_simd: bool,
-    flag_enable_multi_value: bool,
-    flag_enable_reference_types: bool,
     path: &PathBuf,
     name: &str,
     fisa: FlagsOrIsa,
@@ -103,25 +94,19 @@ fn handle_module(
     vprint!(flag_verbose, "Translating... ");
     let _ = terminal.reset();
 
-    let mut module_binary = read_to_end(path.clone()).map_err(|err| err.to_string())?;
-
-    if !module_binary.starts_with(&[b'\0', b'a', b's', b'm']) {
-        let mut features = Features::new();
-        if flag_enable_simd {
-            features.enable_simd();
-        }
-        if flag_enable_multi_value {
-            features.enable_multi_value();
-        }
-        if flag_enable_reference_types {
-            features.enable_reference_types();
-        }
-
-        module_binary = match wat2wasm_with_features(&module_binary, features) {
-            Ok(data) => data,
-            Err(e) => return Err(e.to_string()),
-        };
-    }
+    let module_binary = if path.to_str() == Some("-") {
+        let stdin = std::io::stdin();
+        let mut buf = Vec::new();
+        stdin
+            .lock()
+            .read_to_end(&mut buf)
+            .map_err(|e| e.to_string())?;
+        wat::parse_bytes(&buf)
+            .map_err(|err| format!("{:?}", err))?
+            .into()
+    } else {
+        wat::parse_file(path).map_err(|err| format!("{:?}", err))?
+    };
 
     let isa = match fisa.isa {
         Some(isa) => isa,
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index e40977f882..1dbebdce73 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -21,7 +21,7 @@ serde = { version = "1.0.94", features = ["derive"], optional = true }
 thiserror = "1.0.4"
 
 [dev-dependencies]
-wabt = "0.9.1"
+wat = "1.0.7"
 target-lexicon = "0.10"
 
 [features]
diff --git a/cranelift/wasm/tests/wasm_testsuite.rs b/cranelift/wasm/tests/wasm_testsuite.rs
index 8c0d8e510b..e371a84882 100644
--- a/cranelift/wasm/tests/wasm_testsuite.rs
+++ b/cranelift/wasm/tests/wasm_testsuite.rs
@@ -10,7 +10,6 @@ use std::io::prelude::*;
 use std::path::Path;
 use std::str::FromStr;
 use target_lexicon::triple;
-use wabt::{wat2wasm_with_features, Features, Wat2Wasm};
 
 #[test]
 fn testsuite() {
@@ -47,15 +46,14 @@ fn use_fallthrough_return() {
 
 #[test]
 fn use_name_section() {
-    let wat = r#"
+    let data = wat::parse_str(
+        r#"
         (module $module_name
             (func $func_name (local $loc_name i32)
             )
-        )"#;
-    let data = Wat2Wasm::new()
-        .write_debug_names(true)
-        .convert(wat)
-        .unwrap_or_else(|e| panic!("error converting wat to wasm: {:?}", e));
+        )"#,
+    )
+    .unwrap();
 
     let flags = Flags::new(settings::builder());
     let triple = triple!("riscv64");
@@ -79,23 +77,13 @@ fn read_file(path: &Path) -> io::Result> {
 }
 
 fn read_module(path: &Path) -> Vec {
-    let mut features = Features::new();
-    features.enable_all();
     match path.extension() {
         None => {
             panic!("the file extension is not wasm or wat");
         }
         Some(ext) => match ext.to_str() {
             Some("wasm") => read_file(path).expect("error reading wasm file"),
-            Some("wat") => {
-                let wat = read_file(path).expect("error reading wat file");
-                match wat2wasm_with_features(&wat, features) {
-                    Ok(wasm) => wasm,
-                    Err(e) => {
-                        panic!("error converting wat to wasm: {:?}", e);
-                    }
-                }
-            }
+            Some("wat") => wat::parse_file(path).expect("failed to parse wat"),
             None | Some(&_) => panic!("the file extension for {:?} is not wasm or wat", path),
         },
     }

From 8bec6fe869550db8dc816f7b19c8703515b7b357 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 10 Jan 2020 15:31:37 -0800
Subject: [PATCH 3006/3084] Update to wasmparser 0.47. (#1331)

Co-authored-by: Yury Delendik 
---
 cranelift/wasm/Cargo.toml                 |  2 +-
 cranelift/wasm/src/code_translator.rs     |  3 +++
 cranelift/wasm/src/sections_translator.rs | 21 +++++++++++++++------
 3 files changed, 19 insertions(+), 7 deletions(-)

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 1dbebdce73..6521d71577 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -11,7 +11,7 @@ keywords = ["webassembly", "wasm"]
 edition = "2018"
 
 [dependencies]
-wasmparser = { version = "0.45.0", default-features = false }
+wasmparser = { version = "0.47.0", default-features = false }
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.54.0" }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.54.0", default-features = false }
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 839a218dae..5e230f37d4 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -947,6 +947,9 @@ pub fn translate_operator(
         Operator::F32Le | Operator::F64Le => {
             translate_fcmp(FloatCC::LessThanOrEqual, builder, state)
         }
+        Operator::TypedSelect { .. } => {
+            return Err(wasm_unsupported!("proposed typed select operator {:?}", op))
+        }
         Operator::RefNull => state.push1(builder.ins().null(environ.reference_type())),
         Operator::RefIsNull => {
             let arg = state.pop1();
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index 503ae7226e..008b9b7134 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -17,14 +17,15 @@ use crate::{wasm_unsupported, HashMap};
 use core::convert::TryFrom;
 use cranelift_codegen::ir::immediates::V128Imm;
 use cranelift_codegen::ir::{self, AbiParam, Signature};
+use cranelift_entity::packed_option::ReservedValue;
 use cranelift_entity::EntityRef;
 use std::vec::Vec;
 use wasmparser::{
-    self, CodeSectionReader, Data, DataKind, DataSectionReader, Element, ElementKind,
+    self, CodeSectionReader, Data, DataKind, DataSectionReader, Element, ElementItem, ElementKind,
     ElementSectionReader, Export, ExportSectionReader, ExternalKind, FuncType,
     FunctionSectionReader, GlobalSectionReader, GlobalType, ImportSectionEntryType,
     ImportSectionReader, MemorySectionReader, MemoryType, NameSectionReader, Naming, NamingReader,
-    Operator, TableSectionReader, TypeSectionReader,
+    Operator, TableSectionReader, Type, TypeSectionReader,
 };
 
 /// Parses the Type section of the wasm module.
@@ -292,9 +293,14 @@ pub fn parse_element_section<'data>(
     environ.reserve_table_elements(elements.get_count())?;
 
     for entry in elements {
-        let Element { kind } = entry?;
+        let Element { kind, items, ty } = entry?;
+        if ty != Type::AnyFunc {
+            return Err(wasm_unsupported!(
+                "unsupported table element type: {:?}",
+                ty
+            ));
+        }
         if let ElementKind::Active {
-            items,
             table_index,
             init_expr,
         } = kind
@@ -315,8 +321,11 @@ pub fn parse_element_section<'data>(
             let items_reader = items.get_items_reader()?;
             let mut elems = Vec::with_capacity(usize::try_from(items_reader.get_count()).unwrap());
             for item in items_reader {
-                let x = item?;
-                elems.push(FuncIndex::from_u32(x));
+                let elem = match item? {
+                    ElementItem::Null => FuncIndex::reserved_value(),
+                    ElementItem::Func(index) => FuncIndex::from_u32(index),
+                };
+                elems.push(elem);
             }
             environ.declare_table_elements(
                 TableIndex::from_u32(table_index),

From 6080eeb5442edafe7c880f031db08293beb6a466 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 10 Jan 2020 14:13:22 -0800
Subject: [PATCH 3007/3084] Add a reminder to create the PR to the end of the
 publish script.

With all the sleeps between commands, the publish script takes a few
miniutes to run, and it can be easy to forget to create the PR when it
finishes. Add a reminder to be displayed at the end of the script.
---
 cranelift/publish-all.sh | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 363824b60c..bd92e935ca 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -51,3 +51,7 @@ do
     # https://internals.rust-lang.org/t/changes-to-how-crates-io-handles-index-updates/9608
     echo sleep 10
 done
+echo
+echo "echo \"#\""
+echo "echo \"# Don't forget to click the above link to open a pull-request!\""
+echo "echo \"#\""

From bd88155483dc8fe588396c4e4491718a47ec7e4a Mon Sep 17 00:00:00 2001
From: Yury Delendik 
Date: Mon, 13 Jan 2020 10:32:55 -0600
Subject: [PATCH 3008/3084] Refactor unwind; add FDE support. (#1320)

* Refactor unwind

* add FDE support

* use sink directly in emit functions

* pref off all unwinding generation with feature
---
 cranelift/codegen/Cargo.toml                  |   6 +-
 cranelift/codegen/src/binemit/mod.rs          |  30 ++
 cranelift/codegen/src/context.rs              |  13 +-
 cranelift/codegen/src/ir/function.rs          |   5 +
 cranelift/codegen/src/isa/mod.rs              |   8 +-
 cranelift/codegen/src/isa/x86/abi.rs          |  31 +-
 cranelift/codegen/src/isa/x86/fde.rs          | 354 +++++++++++++++
 cranelift/codegen/src/isa/x86/mod.rs          |  16 +-
 cranelift/codegen/src/isa/x86/unwind.rs       |  99 +++--
 cranelift/filetests/Cargo.toml                |   1 +
 .../isa/x86/windows_systemv_x64_fde.clif      |  54 +++
 cranelift/filetests/src/lib.rs                |   2 +
 cranelift/filetests/src/test_fde.rs           | 415 ++++++++++++++++++
 cranelift/filetests/src/test_unwind.rs        |  25 +-
 14 files changed, 1002 insertions(+), 57 deletions(-)
 create mode 100644 cranelift/codegen/src/isa/x86/fde.rs
 create mode 100644 cranelift/filetests/filetests/isa/x86/windows_systemv_x64_fde.clif
 create mode 100644 cranelift/filetests/src/test_fde.rs

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 831c24f109..1a7b19564d 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -20,6 +20,7 @@ hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
+gimli = { version = "0.19.0", default-features = false, features = ["write"], optional = true }
 smallvec = { version = "1.0.0" }
 thiserror = "1.0.4"
 byteorder = { version = "1.3.2", default-features = false }
@@ -32,7 +33,7 @@ byteorder = { version = "1.3.2", default-features = false }
 cranelift-codegen-meta = { path = "meta", version = "0.54.0" }
 
 [features]
-default = ["std", "basic-blocks"]
+default = ["std", "basic-blocks", "unwind"]
 
 # The "std" feature enables use of libstd. The "core" feature enables use
 # of some minimal std-like replacement libraries. At least one of these two
@@ -47,6 +48,9 @@ core = ["hashbrown"]
 # can significantly increase the size of the library.
 testing_hooks = []
 
+# This enables unwind info generation functionality.
+unwind = ["gimli"]
+
 # ISA targets for which we should build.
 # If no ISA targets are explicitly enabled, the ISA target for the host machine is enabled.
 x86 = []
diff --git a/cranelift/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
index f7bdf01a37..450bcd33d6 100644
--- a/cranelift/codegen/src/binemit/mod.rs
+++ b/cranelift/codegen/src/binemit/mod.rs
@@ -155,6 +155,36 @@ pub trait CodeSink {
     fn add_stackmap(&mut self, _: &[Value], _: &Function, _: &dyn TargetIsa);
 }
 
+/// Type of the frame unwind information.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub enum FrameUnwindKind {
+    /// Windows fastcall unwinding (as in .pdata).
+    Fastcall,
+    /// FDE entry for libunwind (similar to .eh_frame format).
+    Libunwind,
+}
+
+/// Offset in frame unwind information buffer.
+pub type FrameUnwindOffset = usize;
+
+/// Sink for frame unwind information.
+pub trait FrameUnwindSink {
+    /// Get the current position.
+    fn len(&self) -> FrameUnwindOffset;
+
+    /// Add bytes to the code section.
+    fn bytes(&mut self, _: &[u8]);
+
+    /// Reserves bytes in the buffer.
+    fn reserve(&mut self, _len: usize) {}
+
+    /// Add a relocation entry.
+    fn reloc(&mut self, _: Reloc, _: FrameUnwindOffset);
+
+    /// Specified offset to main structure.
+    fn set_entry_offset(&mut self, _: FrameUnwindOffset);
+}
+
 /// Report a bad encoding error.
 #[cold]
 pub fn bad_encoding(func: &Function, inst: Inst) -> ! {
diff --git a/cranelift/codegen/src/context.rs b/cranelift/codegen/src/context.rs
index c9fd1e6bcd..ca70293c05 100644
--- a/cranelift/codegen/src/context.rs
+++ b/cranelift/codegen/src/context.rs
@@ -10,8 +10,8 @@
 //! single ISA instance.
 
 use crate::binemit::{
-    relax_branches, shrink_instructions, CodeInfo, MemoryCodeSink, RelocSink, StackmapSink,
-    TrapSink,
+    relax_branches, shrink_instructions, CodeInfo, FrameUnwindKind, FrameUnwindSink,
+    MemoryCodeSink, RelocSink, StackmapSink, TrapSink,
 };
 use crate::dce::do_dce;
 use crate::dominator_tree::DominatorTree;
@@ -201,8 +201,13 @@ impl Context {
     ///
     /// Only some calling conventions (e.g. Windows fastcall) will have unwind information.
     /// This is a no-op if the function has no unwind information.
-    pub fn emit_unwind_info(&self, isa: &dyn TargetIsa, mem: &mut Vec) {
-        isa.emit_unwind_info(&self.func, mem);
+    pub fn emit_unwind_info(
+        &self,
+        isa: &dyn TargetIsa,
+        kind: FrameUnwindKind,
+        sink: &mut dyn FrameUnwindSink,
+    ) {
+        isa.emit_unwind_info(&self.func, kind, sink);
     }
 
     /// Run the verifier on the function.
diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index 48690c429f..10536cd234 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -253,6 +253,11 @@ impl Function {
     /// Starts collection of debug information.
     pub fn collect_debug_info(&mut self) {
         self.dfg.collect_debug_info();
+        self.collect_frame_layout_info();
+    }
+
+    /// Starts collection of frame layout information.
+    pub fn collect_frame_layout_info(&mut self) {
         self.frame_layout = Some(FrameLayout::new());
     }
 
diff --git a/cranelift/codegen/src/isa/mod.rs b/cranelift/codegen/src/isa/mod.rs
index 4b0a3e7de7..590f59a293 100644
--- a/cranelift/codegen/src/isa/mod.rs
+++ b/cranelift/codegen/src/isa/mod.rs
@@ -63,7 +63,6 @@ use crate::settings::SetResult;
 use crate::timing;
 use alloc::borrow::Cow;
 use alloc::boxed::Box;
-use alloc::vec::Vec;
 use core::fmt;
 use target_lexicon::{triple, Architecture, PointerWidth, Triple};
 use thiserror::Error;
@@ -382,7 +381,12 @@ pub trait TargetIsa: fmt::Display + Send + Sync {
     /// Emit unwind information for the given function.
     ///
     /// Only some calling conventions (e.g. Windows fastcall) will have unwind information.
-    fn emit_unwind_info(&self, _func: &ir::Function, _mem: &mut Vec) {
+    fn emit_unwind_info(
+        &self,
+        _func: &ir::Function,
+        _kind: binemit::FrameUnwindKind,
+        _sink: &mut dyn binemit::FrameUnwindSink,
+    ) {
         // No-op by default
     }
 }
diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 78f3288fd5..011548a5b8 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -1,10 +1,15 @@
 //! x86 ABI implementation.
 
 use super::super::settings as shared_settings;
+#[cfg(feature = "unwind")]
+use super::fde::emit_fde;
 use super::registers::{FPR, GPR, RU};
 use super::settings as isa_settings;
+#[cfg(feature = "unwind")]
 use super::unwind::UnwindInfo;
 use crate::abi::{legalize_args, ArgAction, ArgAssigner, ValueConversion};
+#[cfg(feature = "unwind")]
+use crate::binemit::{FrameUnwindKind, FrameUnwindSink};
 use crate::cursor::{Cursor, CursorPosition, EncCursor};
 use crate::ir;
 use crate::ir::immediates::Imm64;
@@ -18,7 +23,6 @@ use crate::regalloc::RegisterSet;
 use crate::result::CodegenResult;
 use crate::stack_layout::layout_stack;
 use alloc::borrow::Cow;
-use alloc::vec::Vec;
 use core::i32;
 use std::boxed::Box;
 use target_lexicon::{PointerWidth, Triple};
@@ -947,10 +951,25 @@ fn insert_common_epilogue(
     }
 }
 
-pub fn emit_unwind_info(func: &ir::Function, isa: &dyn TargetIsa, mem: &mut Vec) {
-    // Assumption: RBP is being used as the frame pointer
-    // In the future, Windows fastcall codegen should usually omit the frame pointer
-    if let Some(info) = UnwindInfo::try_from_func(func, isa, Some(RU::rbp.into())) {
-        info.emit(mem);
+#[cfg(feature = "unwind")]
+pub fn emit_unwind_info(
+    func: &ir::Function,
+    isa: &dyn TargetIsa,
+    kind: FrameUnwindKind,
+    sink: &mut dyn FrameUnwindSink,
+) {
+    match kind {
+        FrameUnwindKind::Fastcall => {
+            // Assumption: RBP is being used as the frame pointer
+            // In the future, Windows fastcall codegen should usually omit the frame pointer
+            if let Some(info) = UnwindInfo::try_from_func(func, isa, Some(RU::rbp.into())) {
+                info.emit(sink);
+            }
+        }
+        FrameUnwindKind::Libunwind => {
+            if func.frame_layout.is_some() {
+                emit_fde(func, isa, sink);
+            }
+        }
     }
 }
diff --git a/cranelift/codegen/src/isa/x86/fde.rs b/cranelift/codegen/src/isa/x86/fde.rs
new file mode 100644
index 0000000000..6572c650b7
--- /dev/null
+++ b/cranelift/codegen/src/isa/x86/fde.rs
@@ -0,0 +1,354 @@
+//! Support for FDE data generation.
+
+use crate::binemit::{FrameUnwindOffset, FrameUnwindSink, Reloc};
+use crate::ir::{FrameLayoutChange, Function};
+use crate::isa::{CallConv, RegUnit, TargetIsa};
+use alloc::vec::Vec;
+use core::convert::TryInto;
+use gimli::write::{
+    Address, CallFrameInstruction, CommonInformationEntry, EhFrame, EndianVec,
+    FrameDescriptionEntry, FrameTable, Result, Writer,
+};
+use gimli::{Encoding, Format, LittleEndian, Register, X86_64};
+use std::ptr;
+
+pub type FDERelocEntry = (FrameUnwindOffset, Reloc);
+
+const FUNCTION_ENTRY_ADDRESS: Address = Address::Symbol {
+    symbol: 0,
+    addend: 0,
+};
+
+#[derive(Clone)]
+struct FDEWriter {
+    vec: EndianVec,
+    relocs: Vec,
+}
+
+impl FDEWriter {
+    fn new() -> Self {
+        Self {
+            vec: EndianVec::new(LittleEndian),
+            relocs: Vec::new(),
+        }
+    }
+    fn into_vec_and_relocs(self) -> (Vec, Vec) {
+        (self.vec.into_vec(), self.relocs)
+    }
+}
+
+impl Writer for FDEWriter {
+    type Endian = LittleEndian;
+    fn endian(&self) -> Self::Endian {
+        LittleEndian
+    }
+    fn len(&self) -> usize {
+        self.vec.len()
+    }
+    fn write(&mut self, bytes: &[u8]) -> Result<()> {
+        self.vec.write(bytes)
+    }
+    fn write_at(&mut self, offset: usize, bytes: &[u8]) -> Result<()> {
+        self.vec.write_at(offset, bytes)
+    }
+    fn write_address(&mut self, address: Address, size: u8) -> Result<()> {
+        match address {
+            Address::Constant(_) => self.vec.write_address(address, size),
+            Address::Symbol { .. } => {
+                assert_eq!(address, FUNCTION_ENTRY_ADDRESS);
+                let rt = match size {
+                    4 => Reloc::Abs4,
+                    8 => Reloc::Abs8,
+                    _ => {
+                        panic!("Unexpected address size at FDEWriter::write_address");
+                    }
+                };
+                self.relocs.push((self.vec.len().try_into().unwrap(), rt));
+                self.vec.write_udata(0, size)
+            }
+        }
+    }
+}
+
+fn return_address_reg(isa: &dyn TargetIsa) -> Register {
+    assert!(isa.name() == "x86" && isa.pointer_bits() == 64);
+    X86_64::RA
+}
+
+fn map_reg(isa: &dyn TargetIsa, reg: RegUnit) -> Register {
+    assert!(isa.name() == "x86" && isa.pointer_bits() == 64);
+    // Mapping from https://github.com/bytecodealliance/cranelift/pull/902 by @iximeow
+    const X86_GP_REG_MAP: [gimli::Register; 16] = [
+        X86_64::RAX,
+        X86_64::RCX,
+        X86_64::RDX,
+        X86_64::RBX,
+        X86_64::RSP,
+        X86_64::RBP,
+        X86_64::RSI,
+        X86_64::RDI,
+        X86_64::R8,
+        X86_64::R9,
+        X86_64::R10,
+        X86_64::R11,
+        X86_64::R12,
+        X86_64::R13,
+        X86_64::R14,
+        X86_64::R15,
+    ];
+    const X86_XMM_REG_MAP: [gimli::Register; 16] = [
+        X86_64::XMM0,
+        X86_64::XMM1,
+        X86_64::XMM2,
+        X86_64::XMM3,
+        X86_64::XMM4,
+        X86_64::XMM5,
+        X86_64::XMM6,
+        X86_64::XMM7,
+        X86_64::XMM8,
+        X86_64::XMM9,
+        X86_64::XMM10,
+        X86_64::XMM11,
+        X86_64::XMM12,
+        X86_64::XMM13,
+        X86_64::XMM14,
+        X86_64::XMM15,
+    ];
+    let reg_info = isa.register_info();
+    let bank = reg_info.bank_containing_regunit(reg).unwrap();
+    match bank.name {
+        "IntRegs" => {
+            // x86 GP registers have a weird mapping to DWARF registers, so we use a
+            // lookup table.
+            X86_GP_REG_MAP[(reg - bank.first_unit) as usize]
+        }
+        "FloatRegs" => X86_XMM_REG_MAP[(reg - bank.first_unit) as usize],
+        _ => {
+            panic!("unsupported register bank: {}", bank.name);
+        }
+    }
+}
+
+fn to_cfi(
+    isa: &dyn TargetIsa,
+    change: &FrameLayoutChange,
+    cfa_def_reg: &mut Register,
+    cfa_def_offset: &mut i32,
+) -> Option {
+    Some(match change {
+        FrameLayoutChange::CallFrameAddressAt { reg, offset } => {
+            let mapped = map_reg(isa, *reg);
+            let offset = (*offset) as i32;
+            if mapped != *cfa_def_reg && offset != *cfa_def_offset {
+                *cfa_def_reg = mapped;
+                *cfa_def_offset = offset;
+                CallFrameInstruction::Cfa(mapped, offset)
+            } else if offset != *cfa_def_offset {
+                *cfa_def_offset = offset;
+                CallFrameInstruction::CfaOffset(offset)
+            } else if mapped != *cfa_def_reg {
+                *cfa_def_reg = mapped;
+                CallFrameInstruction::CfaRegister(mapped)
+            } else {
+                return None;
+            }
+        }
+        FrameLayoutChange::RegAt { reg, cfa_offset } => {
+            assert!(cfa_offset % -8 == 0);
+            let cfa_offset = *cfa_offset as i32;
+            let mapped = map_reg(isa, *reg);
+            CallFrameInstruction::Offset(mapped, cfa_offset)
+        }
+        FrameLayoutChange::ReturnAddressAt { cfa_offset } => {
+            assert!(cfa_offset % -8 == 0);
+            let cfa_offset = *cfa_offset as i32;
+            CallFrameInstruction::Offset(X86_64::RA, cfa_offset)
+        }
+        _ => {
+            return None;
+        }
+    })
+}
+
+/// Creates FDE structure from FrameLayout.
+pub fn emit_fde(func: &Function, isa: &dyn TargetIsa, sink: &mut dyn FrameUnwindSink) {
+    assert!(isa.name() == "x86");
+
+    // Expecting function with System V prologue
+    assert!(
+        func.signature.call_conv == CallConv::Fast
+            || func.signature.call_conv == CallConv::Cold
+            || func.signature.call_conv == CallConv::SystemV
+    );
+
+    assert!(func.frame_layout.is_some(), "expected func.frame_layout");
+    let frame_layout = func.frame_layout.as_ref().unwrap();
+
+    let mut ebbs = func.layout.ebbs().collect::>();
+    ebbs.sort_by_key(|ebb| func.offsets[*ebb]); // Ensure inst offsets always increase
+
+    let encinfo = isa.encoding_info();
+    let mut last_offset = 0;
+    let mut changes = Vec::new();
+    for ebb in ebbs {
+        for (offset, inst, size) in func.inst_offsets(ebb, &encinfo) {
+            let address_offset = (offset + size) as usize;
+            assert!(last_offset <= address_offset);
+            if let Some(cmds) = frame_layout.instructions.get(&inst) {
+                for cmd in cmds.iter() {
+                    changes.push((address_offset, cmd.clone()));
+                }
+            }
+            last_offset = address_offset;
+        }
+    }
+
+    let len = last_offset as u32;
+
+    let word_size = isa.pointer_bytes() as i32;
+
+    let encoding = Encoding {
+        format: Format::Dwarf32,
+        version: 1,
+        address_size: word_size as u8,
+    };
+    let mut frames = FrameTable::default();
+
+    let mut cfa_def_reg = return_address_reg(isa);
+    let mut cfa_def_offset = 0i32;
+
+    let mut cie = CommonInformationEntry::new(
+        encoding,
+        /* code_alignment_factor = */ 1,
+        /* data_alignment_factor = */ -word_size as i8,
+        return_address_reg(isa),
+    );
+    for ch in frame_layout.initial.iter() {
+        if let Some(cfi) = to_cfi(isa, ch, &mut cfa_def_reg, &mut cfa_def_offset) {
+            cie.add_instruction(cfi);
+        }
+    }
+
+    let cie_id = frames.add_cie(cie);
+
+    let mut fde = FrameDescriptionEntry::new(FUNCTION_ENTRY_ADDRESS, len);
+
+    for (addr, ch) in changes.iter() {
+        if let Some(cfi) = to_cfi(isa, ch, &mut cfa_def_reg, &mut cfa_def_offset) {
+            fde.add_instruction((*addr) as u32, cfi);
+        }
+    }
+
+    frames.add_fde(cie_id, fde);
+
+    let mut eh_frame = EhFrame::from(FDEWriter::new());
+    frames.write_eh_frame(&mut eh_frame).unwrap();
+
+    let (bytes, relocs) = eh_frame.clone().into_vec_and_relocs();
+
+    let unwind_start = sink.len();
+    sink.bytes(&bytes);
+
+    for (off, r) in relocs {
+        sink.reloc(r, off + unwind_start);
+    }
+    let fde_offset = unsafe { ptr::read::(bytes.as_ptr() as *const u32) } as usize + 4;
+    sink.set_entry_offset(unwind_start + fde_offset);
+
+    // Need 0 marker for GCC unwind to end FDE "list".
+    sink.bytes(&[0, 0, 0, 0]);
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::binemit::{FrameUnwindOffset, Reloc};
+    use crate::cursor::{Cursor, FuncCursor};
+    use crate::ir::{ExternalName, InstBuilder, Signature, StackSlotData, StackSlotKind};
+    use crate::isa::{lookup, CallConv};
+    use crate::settings::{builder, Flags};
+    use crate::Context;
+    use std::str::FromStr;
+    use target_lexicon::triple;
+
+    struct SimpleUnwindSink(pub Vec, pub usize, pub Vec<(Reloc, usize)>);
+    impl FrameUnwindSink for SimpleUnwindSink {
+        fn len(&self) -> FrameUnwindOffset {
+            self.0.len()
+        }
+        fn bytes(&mut self, b: &[u8]) {
+            self.0.extend_from_slice(b);
+        }
+        fn reloc(&mut self, r: Reloc, off: FrameUnwindOffset) {
+            self.2.push((r, off));
+        }
+        fn set_entry_offset(&mut self, off: FrameUnwindOffset) {
+            self.1 = off;
+        }
+    }
+
+    #[test]
+    fn test_simple_func() {
+        let isa = lookup(triple!("x86_64"))
+            .expect("expect x86 ISA")
+            .finish(Flags::new(builder()));
+
+        let mut context = Context::for_function(create_function(
+            CallConv::SystemV,
+            Some(StackSlotData::new(StackSlotKind::ExplicitSlot, 64)),
+        ));
+        context.func.collect_frame_layout_info();
+
+        context.compile(&*isa).expect("expected compilation");
+
+        let mut sink = SimpleUnwindSink(Vec::new(), 0, Vec::new());
+        emit_fde(&context.func, &*isa, &mut sink);
+
+        assert_eq!(
+            sink.0,
+            vec![
+                20, 0, 0, 0, // CIE len
+                0, 0, 0, 0,   // CIE marker
+                1,   // version
+                0,   // augmentation string
+                1,   // code aligment = 1
+                120, // data alignment = -8
+                16,  // RA = r16
+                0x0c, 0x07, 0x08, // DW_CFA_def_cfa r7, 8
+                0x90, 0x01, //  DW_CFA_offset r16, -8 * 1
+                0, 0, 0, 0, 0, 0, // padding
+                36, 0, 0, 0, // FDE len
+                28, 0, 0, 0, // CIE offset
+                0, 0, 0, 0, 0, 0, 0, 0, // addr reloc
+                16, 0, 0, 0, 0, 0, 0, 0,    // function length
+                0x42, // DW_CFA_advance_loc 2
+                0x0e, 0x10, // DW_CFA_def_cfa_offset 16
+                0x86, 0x02, // DW_CFA_offset r6, -8 * 2
+                0x43, // DW_CFA_advance_loc 3
+                0x0d, 0x06, // DW_CFA_def_cfa_register
+                0x4a, // DW_CFA_advance_loc 10
+                0x0c, 0x07, 0x08, // DW_CFA_def_cfa r7, 8
+                0, 0, 0, 0, // padding
+                0, 0, 0, 0, // End of FDEs
+            ]
+        );
+        assert_eq!(sink.1, 24);
+        assert_eq!(sink.2.len(), 1);
+    }
+
+    fn create_function(call_conv: CallConv, stack_slot: Option) -> Function {
+        let mut func =
+            Function::with_name_signature(ExternalName::user(0, 0), Signature::new(call_conv));
+
+        let ebb0 = func.dfg.make_ebb();
+        let mut pos = FuncCursor::new(&mut func);
+        pos.insert_ebb(ebb0);
+        pos.ins().return_(&[]);
+
+        if let Some(stack_slot) = stack_slot {
+            func.stack_slots.push(stack_slot);
+        }
+
+        func
+    }
+}
diff --git a/cranelift/codegen/src/isa/x86/mod.rs b/cranelift/codegen/src/isa/x86/mod.rs
index e2e785675f..c3b871ab42 100644
--- a/cranelift/codegen/src/isa/x86/mod.rs
+++ b/cranelift/codegen/src/isa/x86/mod.rs
@@ -3,14 +3,19 @@
 mod abi;
 mod binemit;
 mod enc_tables;
+#[cfg(feature = "unwind")]
+mod fde;
 mod registers;
 pub mod settings;
+#[cfg(feature = "unwind")]
 mod unwind;
 
 use super::super::settings as shared_settings;
 #[cfg(feature = "testing_hooks")]
 use crate::binemit::CodeSink;
 use crate::binemit::{emit_function, MemoryCodeSink};
+#[cfg(feature = "unwind")]
+use crate::binemit::{FrameUnwindKind, FrameUnwindSink};
 use crate::ir;
 use crate::isa::enc_tables::{self as shared_enc_tables, lookup_enclist, Encodings};
 use crate::isa::Builder as IsaBuilder;
@@ -20,7 +25,6 @@ use crate::result::CodegenResult;
 use crate::timing;
 use alloc::borrow::Cow;
 use alloc::boxed::Box;
-use alloc::vec::Vec;
 use core::fmt;
 use target_lexicon::{PointerWidth, Triple};
 
@@ -157,8 +161,14 @@ impl TargetIsa for Isa {
     /// Emit unwind information for the given function.
     ///
     /// Only some calling conventions (e.g. Windows fastcall) will have unwind information.
-    fn emit_unwind_info(&self, func: &ir::Function, mem: &mut Vec) {
-        abi::emit_unwind_info(func, self, mem);
+    #[cfg(feature = "unwind")]
+    fn emit_unwind_info(
+        &self,
+        func: &ir::Function,
+        kind: FrameUnwindKind,
+        sink: &mut dyn FrameUnwindSink,
+    ) {
+        abi::emit_unwind_info(func, self, kind, sink);
     }
 }
 
diff --git a/cranelift/codegen/src/isa/x86/unwind.rs b/cranelift/codegen/src/isa/x86/unwind.rs
index 2cddf40782..ce05f3afab 100644
--- a/cranelift/codegen/src/isa/x86/unwind.rs
+++ b/cranelift/codegen/src/isa/x86/unwind.rs
@@ -1,6 +1,7 @@
 //! Unwind information for x64 Windows.
 
 use super::registers::RU;
+use crate::binemit::FrameUnwindSink;
 use crate::ir::{Function, InstructionData, Opcode};
 use crate::isa::{CallConv, RegUnit, TargetIsa};
 use alloc::vec::Vec;
@@ -11,16 +12,20 @@ const SMALL_ALLOC_MAX_SIZE: u32 = 128;
 /// Maximum (inclusive) size of a "large" stack allocation that can represented in 16-bits
 const LARGE_ALLOC_16BIT_MAX_SIZE: u32 = 524280;
 
-fn write_u16(mem: &mut Vec, v: u16) {
-    let mut buf = [0; 2];
-    T::write_u16(&mut buf, v);
-    mem.extend(buf.iter());
+fn write_u8(sink: &mut dyn FrameUnwindSink, v: u8) {
+    sink.bytes(&[v]);
 }
 
-fn write_u32(mem: &mut Vec, v: u32) {
+fn write_u16(sink: &mut dyn FrameUnwindSink, v: u16) {
+    let mut buf = [0; 2];
+    T::write_u16(&mut buf, v);
+    sink.bytes(&buf);
+}
+
+fn write_u32(sink: &mut dyn FrameUnwindSink, v: u32) {
     let mut buf = [0; 4];
     T::write_u32(&mut buf, v);
-    mem.extend(buf.iter());
+    sink.bytes(&buf);
 }
 
 /// The supported unwind codes for the x64 Windows ABI.
@@ -36,7 +41,7 @@ enum UnwindCode {
 }
 
 impl UnwindCode {
-    fn emit(&self, mem: &mut Vec) {
+    fn emit(&self, sink: &mut dyn FrameUnwindSink) {
         enum UnwindOperation {
             PushNonvolatileRegister,
             LargeStackAlloc,
@@ -46,30 +51,37 @@ impl UnwindCode {
 
         match self {
             Self::PushRegister { offset, reg } => {
-                mem.push(*offset);
-                mem.push(((*reg as u8) << 4) | (UnwindOperation::PushNonvolatileRegister as u8));
+                write_u8(sink, *offset);
+                write_u8(
+                    sink,
+                    ((*reg as u8) << 4) | (UnwindOperation::PushNonvolatileRegister as u8),
+                );
             }
             Self::StackAlloc { offset, size } => {
                 // Stack allocations on Windows must be a multiple of 8 and be at least 1 slot
                 assert!(*size >= 8);
                 assert!((*size % 8) == 0);
 
-                mem.push(*offset);
+                write_u8(sink, *offset);
                 if *size <= SMALL_ALLOC_MAX_SIZE {
-                    mem.push(
+                    write_u8(
+                        sink,
                         ((((*size - 8) / 8) as u8) << 4) | UnwindOperation::SmallStackAlloc as u8,
                     );
                 } else if *size <= LARGE_ALLOC_16BIT_MAX_SIZE {
-                    mem.push(UnwindOperation::LargeStackAlloc as u8);
-                    write_u16::(mem, (*size / 8) as u16);
+                    write_u8(sink, UnwindOperation::LargeStackAlloc as u8);
+                    write_u16::(sink, (*size / 8) as u16);
                 } else {
-                    mem.push((1 << 4) | (UnwindOperation::LargeStackAlloc as u8));
-                    write_u32::(mem, *size);
+                    write_u8(sink, (1 << 4) | (UnwindOperation::LargeStackAlloc as u8));
+                    write_u32::(sink, *size);
                 }
             }
             Self::SetFramePointer { offset, sp_offset } => {
-                mem.push(*offset);
-                mem.push((*sp_offset << 4) | (UnwindOperation::SetFramePointer as u8));
+                write_u8(sink, *offset);
+                write_u8(
+                    sink,
+                    (*sp_offset << 4) | (UnwindOperation::SetFramePointer as u8),
+                );
             }
         };
     }
@@ -231,48 +243,49 @@ impl UnwindInfo {
             .fold(0, |nodes, c| nodes + c.node_count())
     }
 
-    pub fn emit(&self, mem: &mut Vec) {
+    pub fn emit(&self, sink: &mut dyn FrameUnwindSink) {
         const UNWIND_INFO_VERSION: u8 = 1;
 
         let size = self.size();
-        let offset = mem.len();
+        let offset = sink.len();
 
         // Ensure the memory is 32-bit aligned
         assert_eq!(offset % 4, 0);
 
-        mem.reserve(offset + size);
+        sink.reserve(offset + size);
 
         let node_count = self.node_count();
         assert!(node_count <= 256);
 
-        mem.push((self.flags << 3) | UNWIND_INFO_VERSION);
-        mem.push(self.prologue_size);
-        mem.push(node_count as u8);
+        write_u8(sink, (self.flags << 3) | UNWIND_INFO_VERSION);
+        write_u8(sink, self.prologue_size);
+        write_u8(sink, node_count as u8);
 
         if let Some(reg) = self.frame_register {
-            mem.push((self.frame_register_offset << 4) | reg as u8);
+            write_u8(sink, (self.frame_register_offset << 4) | reg as u8);
         } else {
-            mem.push(0);
+            write_u8(sink, 0);
         }
 
         // Unwind codes are written in reverse order (prologue offset descending)
         for code in self.unwind_codes.iter().rev() {
-            code.emit(mem);
+            code.emit(sink);
         }
 
         // To keep a 32-bit alignment, emit 2 bytes of padding if there's an odd number of 16-bit nodes
         if (node_count & 1) == 1 {
-            write_u16::(mem, 0);
+            write_u16::(sink, 0);
         }
 
         // Ensure the correct number of bytes was emitted
-        assert_eq!(mem.len() - offset, size);
+        assert_eq!(sink.len() - offset, size);
     }
 }
 
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::binemit::{FrameUnwindOffset, Reloc};
     use crate::cursor::{Cursor, FuncCursor};
     use crate::ir::{ExternalName, InstBuilder, Signature, StackSlotData, StackSlotKind};
     use crate::isa::{lookup, CallConv};
@@ -281,6 +294,18 @@ mod tests {
     use std::str::FromStr;
     use target_lexicon::triple;
 
+    struct SimpleUnwindSink(pub Vec);
+    impl FrameUnwindSink for SimpleUnwindSink {
+        fn len(&self) -> FrameUnwindOffset {
+            self.0.len()
+        }
+        fn bytes(&mut self, b: &[u8]) {
+            self.0.extend_from_slice(b);
+        }
+        fn reloc(&mut self, _: Reloc, _: FrameUnwindOffset) {}
+        fn set_entry_offset(&mut self, _: FrameUnwindOffset) {}
+    }
+
     #[test]
     fn test_wrong_calling_convention() {
         let isa = lookup(triple!("x86_64"))
@@ -336,11 +361,11 @@ mod tests {
 
         assert_eq!(unwind.size(), 12);
 
-        let mut mem = Vec::new();
-        unwind.emit(&mut mem);
+        let mut sink = SimpleUnwindSink(Vec::new());
+        unwind.emit(&mut sink);
 
         assert_eq!(
-            mem,
+            sink.0,
             [
                 0x01, // Version and flags (version 1, no flags)
                 0x09, // Prologue size
@@ -400,11 +425,11 @@ mod tests {
 
         assert_eq!(unwind.size(), 12);
 
-        let mut mem = Vec::new();
-        unwind.emit(&mut mem);
+        let mut sink = SimpleUnwindSink(Vec::new());
+        unwind.emit(&mut sink);
 
         assert_eq!(
-            mem,
+            sink.0,
             [
                 0x01, // Version and flags (version 1, no flags)
                 0x1B, // Prologue size
@@ -464,11 +489,11 @@ mod tests {
 
         assert_eq!(unwind.size(), 16);
 
-        let mut mem = Vec::new();
-        unwind.emit(&mut mem);
+        let mut sink = SimpleUnwindSink(Vec::new());
+        unwind.emit(&mut sink);
 
         assert_eq!(
-            mem,
+            sink.0,
             [
                 0x01, // Version and flags (version 1, no flags)
                 0x1B, // Prologue size
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index da2a92814b..def330776c 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -16,6 +16,7 @@ cranelift-reader = { path = "../cranelift-reader", version = "0.54.0" }
 cranelift-preopt = { path = "../cranelift-preopt", version = "0.54.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
+gimli = { version = "0.19.0", default-features = false, features = ["read"] }
 log = "0.4.6"
 memmap = "0.7.0"
 num_cpus = "1.8.0"
diff --git a/cranelift/filetests/filetests/isa/x86/windows_systemv_x64_fde.clif b/cranelift/filetests/filetests/isa/x86/windows_systemv_x64_fde.clif
new file mode 100644
index 0000000000..1444359de7
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/windows_systemv_x64_fde.clif
@@ -0,0 +1,54 @@
+test fde
+set opt_level=speed_and_size
+set is_pic
+target x86_64 haswell
+
+; check that there is no libunwind information for a windows_fastcall function
+function %not_fastcall() windows_fastcall {
+ebb0:
+    return
+}
+; sameln: No unwind information.
+
+; check the libunwind information with a function with no args
+function %no_args() system_v {
+ebb0:
+    return
+}
+; sameln: 0x00000000: CIE
+; nextln:         length: 0x00000014
+; nextln:        version: 0x01
+; nextln:     code_align: 1
+; nextln:     data_align: -8
+; nextln:    ra_register: 0x10
+; nextln:                 DW_CFA_def_cfa (r7, 8)
+; nextln:                 DW_CFA_offset (r16, 1)
+; nextln:                 DW_CFA_nop
+; nextln:                 DW_CFA_nop
+; nextln:                 DW_CFA_nop
+; nextln:                 DW_CFA_nop
+; nextln:                 DW_CFA_nop
+; nextln:                 DW_CFA_nop
+; nextln:   Instructions: Init State:
+; nextln: 
+; nextln: 
+; nextln: 0x00000018: FDE
+; nextln:         length: 0x00000024
+; nextln:    CIE_pointer: 0x00000000
+; nextln:     start_addr: 0x0000000000000000
+; nextln:     range_size: 0x0000000000000006 (end_addr = 0x0000000000000006)
+; nextln:   Instructions:
+; nextln:                 DW_CFA_advance_loc (1)
+; nextln:                 DW_CFA_def_cfa_offset (16)
+; nextln:                 DW_CFA_offset (r6, 2)
+; nextln:                 DW_CFA_advance_loc (3)
+; nextln:                 DW_CFA_def_cfa_register (r6)
+; nextln:                 DW_CFA_advance_loc (1)
+; nextln:                 DW_CFA_def_cfa (r7, 8)
+; nextln:                 DW_CFA_nop
+; nextln:                 DW_CFA_nop
+; nextln:                 DW_CFA_nop
+; nextln:                 DW_CFA_nop
+; nextln: 
+; nextln: Entry: 24
+; nextln: Relocs: [(Abs8, 32)]
diff --git a/cranelift/filetests/src/lib.rs b/cranelift/filetests/src/lib.rs
index ded3acd16c..0d3b12e458 100644
--- a/cranelift/filetests/src/lib.rs
+++ b/cranelift/filetests/src/lib.rs
@@ -42,6 +42,7 @@ mod test_cat;
 mod test_compile;
 mod test_dce;
 mod test_domtree;
+mod test_fde;
 mod test_legalizer;
 mod test_licm;
 mod test_postopt;
@@ -137,6 +138,7 @@ fn new_subtest(parsed: &TestCommand) -> subtest::SubtestResult test_preopt::subtest(parsed),
         "safepoint" => test_safepoint::subtest(parsed),
         "unwind" => test_unwind::subtest(parsed),
+        "fde" => test_fde::subtest(parsed),
         _ => Err(format!("unknown test command '{}'", parsed.command)),
     }
 }
diff --git a/cranelift/filetests/src/test_fde.rs b/cranelift/filetests/src/test_fde.rs
new file mode 100644
index 0000000000..1a9e3250d7
--- /dev/null
+++ b/cranelift/filetests/src/test_fde.rs
@@ -0,0 +1,415 @@
+//! Test command for verifying the unwind emitted for each function.
+//!
+//! The `unwind` test command runs each function through the full code generator pipeline.
+#![cfg_attr(feature = "cargo-clippy", allow(clippy::cast_ptr_alignment))]
+
+use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
+use cranelift_codegen;
+use cranelift_codegen::binemit::{FrameUnwindKind, FrameUnwindOffset, FrameUnwindSink, Reloc};
+use cranelift_codegen::ir;
+use cranelift_reader::TestCommand;
+use std::borrow::Cow;
+use std::fmt::Write;
+
+struct TestUnwind;
+
+pub fn subtest(parsed: &TestCommand) -> SubtestResult> {
+    assert_eq!(parsed.command, "fde");
+    if !parsed.options.is_empty() {
+        Err(format!("No options allowed on {}", parsed))
+    } else {
+        Ok(Box::new(TestUnwind))
+    }
+}
+
+impl SubTest for TestUnwind {
+    fn name(&self) -> &'static str {
+        "fde"
+    }
+
+    fn is_mutating(&self) -> bool {
+        false
+    }
+
+    fn needs_isa(&self) -> bool {
+        true
+    }
+
+    fn run(&self, func: Cow, context: &Context) -> SubtestResult<()> {
+        let isa = context.isa.expect("unwind needs an ISA");
+
+        if func.signature.call_conv != cranelift_codegen::isa::CallConv::SystemV {
+            return run_filecheck(&"No unwind information.", context);
+        }
+
+        let mut comp_ctx = cranelift_codegen::Context::for_function(func.into_owned());
+        comp_ctx.func.collect_frame_layout_info();
+
+        comp_ctx.compile(isa).expect("failed to compile function");
+
+        struct SimpleUnwindSink(pub Vec, pub usize, pub Vec<(Reloc, usize)>);
+        impl FrameUnwindSink for SimpleUnwindSink {
+            fn len(&self) -> FrameUnwindOffset {
+                self.0.len()
+            }
+            fn bytes(&mut self, b: &[u8]) {
+                self.0.extend_from_slice(b);
+            }
+            fn reloc(&mut self, r: Reloc, off: FrameUnwindOffset) {
+                self.2.push((r, off));
+            }
+            fn set_entry_offset(&mut self, off: FrameUnwindOffset) {
+                self.1 = off;
+            }
+        }
+
+        let mut sink = SimpleUnwindSink(Vec::new(), 0, Vec::new());
+        comp_ctx.emit_unwind_info(isa, FrameUnwindKind::Libunwind, &mut sink);
+
+        let mut text = String::new();
+        if sink.0.is_empty() {
+            writeln!(text, "No unwind information.").unwrap();
+        } else {
+            print_unwind_info(&mut text, &sink.0, isa.pointer_bytes());
+            writeln!(text, "Entry: {}", sink.1).unwrap();
+            writeln!(text, "Relocs: {:?}", sink.2).unwrap();
+        }
+
+        run_filecheck(&text, context)
+    }
+}
+
+fn register_name<'a>(register: gimli::Register) -> std::borrow::Cow<'a, str> {
+    Cow::Owned(format!("r{}", register.0))
+}
+
+fn print_unwind_info(text: &mut String, mem: &[u8], address_size: u8) {
+    let mut eh_frame = gimli::EhFrame::new(mem, gimli::LittleEndian);
+    eh_frame.set_address_size(address_size);
+    let bases = gimli::BaseAddresses::default();
+    dwarfdump::dump_eh_frame(text, &eh_frame, &bases, ®ister_name).unwrap();
+}
+
+mod dwarfdump {
+    // Copied from https://github.com/gimli-rs/gimli/blob/1e49ffc9af4ec64a1b7316924d73c933dd7157c5/examples/dwarfdump.rs
+    use gimli::UnwindSection;
+    use std::borrow::Cow;
+    use std::collections::HashMap;
+    use std::fmt::{self, Debug, Write};
+    use std::result;
+
+    #[derive(Debug, Clone, Copy, PartialEq, Eq)]
+    pub(super) enum Error {
+        GimliError(gimli::Error),
+        IoError,
+    }
+
+    impl fmt::Display for Error {
+        #[inline]
+        fn fmt(&self, f: &mut fmt::Formatter) -> ::std::result::Result<(), fmt::Error> {
+            Debug::fmt(self, f)
+        }
+    }
+
+    impl From for Error {
+        fn from(err: gimli::Error) -> Self {
+            Error::GimliError(err)
+        }
+    }
+
+    impl From for Error {
+        fn from(_: fmt::Error) -> Self {
+            Error::IoError
+        }
+    }
+
+    pub(super) type Result = result::Result;
+
+    pub(super) trait Reader: gimli::Reader + Send + Sync {}
+
+    impl<'input, Endian> Reader for gimli::EndianSlice<'input, Endian> where
+        Endian: gimli::Endianity + Send + Sync
+    {
+    }
+
+    pub(super) fn dump_eh_frame(
+        w: &mut W,
+        eh_frame: &gimli::EhFrame,
+        bases: &gimli::BaseAddresses,
+        register_name: &dyn Fn(gimli::Register) -> Cow<'static, str>,
+    ) -> Result<()> {
+        let mut cies = HashMap::new();
+
+        let mut entries = eh_frame.entries(bases);
+        loop {
+            match entries.next()? {
+                None => return Ok(()),
+                Some(gimli::CieOrFde::Cie(cie)) => {
+                    writeln!(w, "{:#010x}: CIE", cie.offset())?;
+                    writeln!(w, "        length: {:#010x}", cie.entry_len())?;
+                    // TODO: CIE_id
+                    writeln!(w, "       version: {:#04x}", cie.version())?;
+                    // TODO: augmentation
+                    writeln!(w, "    code_align: {}", cie.code_alignment_factor())?;
+                    writeln!(w, "    data_align: {}", cie.data_alignment_factor())?;
+                    writeln!(w, "   ra_register: {:#x}", cie.return_address_register().0)?;
+                    if let Some(encoding) = cie.lsda_encoding() {
+                        writeln!(w, " lsda_encoding: {:#02x}", encoding.0)?;
+                    }
+                    if let Some((encoding, personality)) = cie.personality_with_encoding() {
+                        write!(w, "   personality: {:#02x} ", encoding.0)?;
+                        dump_pointer(w, personality)?;
+                        writeln!(w)?;
+                    }
+                    if let Some(encoding) = cie.fde_address_encoding() {
+                        writeln!(w, "  fde_encoding: {:#02x}", encoding.0)?;
+                    }
+                    dump_cfi_instructions(
+                        w,
+                        cie.instructions(eh_frame, bases),
+                        true,
+                        register_name,
+                    )?;
+                    writeln!(w)?;
+                }
+                Some(gimli::CieOrFde::Fde(partial)) => {
+                    let mut offset = None;
+                    let fde = partial.parse(|_, bases, o| {
+                        offset = Some(o);
+                        cies.entry(o)
+                            .or_insert_with(|| eh_frame.cie_from_offset(bases, o))
+                            .clone()
+                    })?;
+
+                    writeln!(w)?;
+                    writeln!(w, "{:#010x}: FDE", fde.offset())?;
+                    writeln!(w, "        length: {:#010x}", fde.entry_len())?;
+                    writeln!(w, "   CIE_pointer: {:#010x}", offset.unwrap().0)?;
+                    // TODO: symbolicate the start address like the canonical dwarfdump does.
+                    writeln!(w, "    start_addr: {:#018x}", fde.initial_address())?;
+                    writeln!(
+                        w,
+                        "    range_size: {:#018x} (end_addr = {:#018x})",
+                        fde.len(),
+                        fde.initial_address() + fde.len()
+                    )?;
+                    if let Some(lsda) = fde.lsda() {
+                        write!(w, "          lsda: ")?;
+                        dump_pointer(w, lsda)?;
+                        writeln!(w)?;
+                    }
+                    dump_cfi_instructions(
+                        w,
+                        fde.instructions(eh_frame, bases),
+                        false,
+                        register_name,
+                    )?;
+                    writeln!(w)?;
+                }
+            }
+        }
+    }
+
+    fn dump_pointer(w: &mut W, p: gimli::Pointer) -> Result<()> {
+        match p {
+            gimli::Pointer::Direct(p) => {
+                write!(w, "{:#018x}", p)?;
+            }
+            gimli::Pointer::Indirect(p) => {
+                write!(w, "({:#018x})", p)?;
+            }
+        }
+        Ok(())
+    }
+
+    #[allow(clippy::unneeded_field_pattern)]
+    fn dump_cfi_instructions(
+        w: &mut W,
+        mut insns: gimli::CallFrameInstructionIter,
+        is_initial: bool,
+        register_name: &dyn Fn(gimli::Register) -> Cow<'static, str>,
+    ) -> Result<()> {
+        use gimli::CallFrameInstruction::*;
+
+        // TODO: we need to actually evaluate these instructions as we iterate them
+        // so we can print the initialized state for CIEs, and each unwind row's
+        // registers for FDEs.
+        //
+        // TODO: We should print DWARF expressions for the CFI instructions that
+        // embed DWARF expressions within themselves.
+
+        if !is_initial {
+            writeln!(w, "  Instructions:")?;
+        }
+
+        loop {
+            match insns.next() {
+                Err(e) => {
+                    writeln!(w, "Failed to decode CFI instruction: {}", e)?;
+                    return Ok(());
+                }
+                Ok(None) => {
+                    if is_initial {
+                        writeln!(w, "  Instructions: Init State:")?;
+                    }
+                    return Ok(());
+                }
+                Ok(Some(op)) => match op {
+                    SetLoc { address } => {
+                        writeln!(w, "                DW_CFA_set_loc ({:#x})", address)?;
+                    }
+                    AdvanceLoc { delta } => {
+                        writeln!(w, "                DW_CFA_advance_loc ({})", delta)?;
+                    }
+                    DefCfa { register, offset } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_def_cfa ({}, {})",
+                            register_name(register),
+                            offset
+                        )?;
+                    }
+                    DefCfaSf {
+                        register,
+                        factored_offset,
+                    } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_def_cfa_sf ({}, {})",
+                            register_name(register),
+                            factored_offset
+                        )?;
+                    }
+                    DefCfaRegister { register } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_def_cfa_register ({})",
+                            register_name(register)
+                        )?;
+                    }
+                    DefCfaOffset { offset } => {
+                        writeln!(w, "                DW_CFA_def_cfa_offset ({})", offset)?;
+                    }
+                    DefCfaOffsetSf { factored_offset } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_def_cfa_offset_sf ({})",
+                            factored_offset
+                        )?;
+                    }
+                    DefCfaExpression { expression: _ } => {
+                        writeln!(w, "                DW_CFA_def_cfa_expression (...)")?;
+                    }
+                    Undefined { register } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_undefined ({})",
+                            register_name(register)
+                        )?;
+                    }
+                    SameValue { register } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_same_value ({})",
+                            register_name(register)
+                        )?;
+                    }
+                    Offset {
+                        register,
+                        factored_offset,
+                    } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_offset ({}, {})",
+                            register_name(register),
+                            factored_offset
+                        )?;
+                    }
+                    OffsetExtendedSf {
+                        register,
+                        factored_offset,
+                    } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_offset_extended_sf ({}, {})",
+                            register_name(register),
+                            factored_offset
+                        )?;
+                    }
+                    ValOffset {
+                        register,
+                        factored_offset,
+                    } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_val_offset ({}, {})",
+                            register_name(register),
+                            factored_offset
+                        )?;
+                    }
+                    ValOffsetSf {
+                        register,
+                        factored_offset,
+                    } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_val_offset_sf ({}, {})",
+                            register_name(register),
+                            factored_offset
+                        )?;
+                    }
+                    Register {
+                        dest_register,
+                        src_register,
+                    } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_register ({}, {})",
+                            register_name(dest_register),
+                            register_name(src_register)
+                        )?;
+                    }
+                    Expression {
+                        register,
+                        expression: _,
+                    } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_expression ({}, ...)",
+                            register_name(register)
+                        )?;
+                    }
+                    ValExpression {
+                        register,
+                        expression: _,
+                    } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_val_expression ({}, ...)",
+                            register_name(register)
+                        )?;
+                    }
+                    Restore { register } => {
+                        writeln!(
+                            w,
+                            "                DW_CFA_restore ({})",
+                            register_name(register)
+                        )?;
+                    }
+                    RememberState => {
+                        writeln!(w, "                DW_CFA_remember_state")?;
+                    }
+                    RestoreState => {
+                        writeln!(w, "                DW_CFA_restore_state")?;
+                    }
+                    ArgsSize { size } => {
+                        writeln!(w, "                DW_CFA_GNU_args_size ({})", size)?;
+                    }
+                    Nop => {
+                        writeln!(w, "                DW_CFA_nop")?;
+                    }
+                },
+            }
+        }
+    }
+}
diff --git a/cranelift/filetests/src/test_unwind.rs b/cranelift/filetests/src/test_unwind.rs
index 7b67ee7313..3db1cbf829 100644
--- a/cranelift/filetests/src/test_unwind.rs
+++ b/cranelift/filetests/src/test_unwind.rs
@@ -6,6 +6,7 @@
 use crate::subtest::{run_filecheck, Context, SubTest, SubtestResult};
 use byteorder::{ByteOrder, LittleEndian};
 use cranelift_codegen;
+use cranelift_codegen::binemit::{FrameUnwindKind, FrameUnwindOffset, FrameUnwindSink, Reloc};
 use cranelift_codegen::ir;
 use cranelift_reader::TestCommand;
 use std::borrow::Cow;
@@ -41,14 +42,30 @@ impl SubTest for TestUnwind {
 
         comp_ctx.compile(isa).expect("failed to compile function");
 
-        let mut mem = Vec::new();
-        comp_ctx.emit_unwind_info(isa, &mut mem);
+        struct Sink(Vec);
+        impl FrameUnwindSink for Sink {
+            fn len(&self) -> FrameUnwindOffset {
+                self.0.len()
+            }
+            fn bytes(&mut self, b: &[u8]) {
+                self.0.extend_from_slice(b);
+            }
+            fn reloc(&mut self, _: Reloc, _: FrameUnwindOffset) {
+                unimplemented!();
+            }
+            fn set_entry_offset(&mut self, _: FrameUnwindOffset) {
+                unimplemented!();
+            }
+        }
+
+        let mut sink = Sink(Vec::new());
+        comp_ctx.emit_unwind_info(isa, FrameUnwindKind::Fastcall, &mut sink);
 
         let mut text = String::new();
-        if mem.is_empty() {
+        if sink.0.is_empty() {
             writeln!(text, "No unwind information.").unwrap();
         } else {
-            print_unwind_info(&mut text, &mem);
+            print_unwind_info(&mut text, &sink.0);
         }
 
         run_filecheck(&text, context)

From dd497c19e1203973bde10a94337a8a4a1fb1df8f Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Mon, 13 Jan 2020 22:42:49 +0100
Subject: [PATCH 3009/3084] =?UTF-8?q?Renames=20Settings=20=E2=9A=A0?=
 =?UTF-8?q?=EF=B8=8F=20=20(fixes=20#976)=20(#1321)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This is a breaking API change: the following settings have been renamed:

- jump_tables_enabled -> enable_jump_tables
- colocated_libcalls -> use_colocated_libcalls
- probestack_enabled -> enable_probestack
- allones_funcaddrs -> emit_all_ones_funcaddrs
---
 cranelift/codegen/meta/src/isa/x86/settings.rs            | 6 +++---
 cranelift/codegen/meta/src/shared/settings.rs             | 8 ++++----
 cranelift/codegen/src/ir/libcall.rs                       | 4 ++--
 cranelift/codegen/src/isa/x86/abi.rs                      | 5 ++---
 cranelift/codegen/src/legalizer/mod.rs                    | 4 ++--
 cranelift/codegen/src/settings.rs                         | 8 ++++----
 .../filetests/filetests/isa/x86/allones_funcaddrs32.clif  | 4 ++--
 .../filetests/filetests/isa/x86/allones_funcaddrs64.clif  | 4 ++--
 .../filetests/isa/x86/baldrdash-table-sig-reg.clif        | 2 +-
 cranelift/filetests/filetests/isa/x86/compile-vconst.clif | 2 +-
 cranelift/filetests/filetests/isa/x86/legalize-splat.clif | 2 +-
 .../filetests/isa/x86/probestack-adjusts-sp.clif          | 2 +-
 .../filetests/filetests/isa/x86/probestack-disabled.clif  | 4 ++--
 .../filetests/filetests/isa/x86/probestack-size.clif      | 2 +-
 cranelift/filetests/filetests/isa/x86/probestack.clif     | 2 +-
 cranelift/filetests/filetests/isa/x86/relax_branch.clif   | 4 ++--
 .../filetests/isa/x86/scalar_to_vector-compile.clif       | 2 +-
 .../filetests/filetests/legalizer/br_table_cond.clif      | 4 ++--
 .../filetests/filetests/legalizer/empty_br_table.clif     | 4 ++--
 .../filetests/filetests/regalloc/unreachable_code.clif    | 2 +-
 20 files changed, 37 insertions(+), 38 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/settings.rs b/cranelift/codegen/meta/src/isa/x86/settings.rs
index bef8fc3f81..e39384a5c3 100644
--- a/cranelift/codegen/meta/src/isa/x86/settings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/settings.rs
@@ -59,16 +59,16 @@ pub(crate) fn define(shared: &SettingGroup) -> SettingGroup {
     // back in the shared SettingGroup, and use it in x86 instruction predicates.
 
     let is_pic = shared.get_bool("is_pic");
-    let allones_funcaddrs = shared.get_bool("allones_funcaddrs");
+    let emit_all_ones_funcaddrs = shared.get_bool("emit_all_ones_funcaddrs");
     settings.add_predicate("is_pic", predicate!(is_pic));
     settings.add_predicate("not_is_pic", predicate!(!is_pic));
     settings.add_predicate(
         "all_ones_funcaddrs_and_not_is_pic",
-        predicate!(allones_funcaddrs && !is_pic),
+        predicate!(emit_all_ones_funcaddrs && !is_pic),
     );
     settings.add_predicate(
         "not_all_ones_funcaddrs_and_not_is_pic",
-        predicate!(!allones_funcaddrs && !is_pic),
+        predicate!(!emit_all_ones_funcaddrs && !is_pic),
     );
 
     // Presets corresponding to x86 CPUs.
diff --git a/cranelift/codegen/meta/src/shared/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
index b039c7bc21..545c2732b3 100644
--- a/cranelift/codegen/meta/src/shared/settings.rs
+++ b/cranelift/codegen/meta/src/shared/settings.rs
@@ -37,7 +37,7 @@ pub(crate) fn define() -> SettingGroup {
     );
 
     settings.add_bool(
-        "colocated_libcalls",
+        "use_colocated_libcalls",
         r#"
             Use colocated libcalls.
 
@@ -177,7 +177,7 @@ pub(crate) fn define() -> SettingGroup {
     // BaldrMonkey requires that not-yet-relocated function addresses be encoded
     // as all-ones bitpatterns.
     settings.add_bool(
-        "allones_funcaddrs",
+        "emit_all_ones_funcaddrs",
         "Emit not-yet-relocated function addresses as all-ones bit patterns.",
         false,
     );
@@ -185,7 +185,7 @@ pub(crate) fn define() -> SettingGroup {
     // Stack probing options.
 
     settings.add_bool(
-        "probestack_enabled",
+        "enable_probestack",
         r#"
             Enable the use of stack probes, for calling conventions which support this
             functionality.
@@ -218,7 +218,7 @@ pub(crate) fn define() -> SettingGroup {
     // Jump table options.
 
     settings.add_bool(
-        "jump_tables_enabled",
+        "enable_jump_tables",
         "Enable the use of jump tables in generated machine code.",
         true,
     );
diff --git a/cranelift/codegen/src/ir/libcall.rs b/cranelift/codegen/src/ir/libcall.rs
index 88752bb3ec..cffbfb5a90 100644
--- a/cranelift/codegen/src/ir/libcall.rs
+++ b/cranelift/codegen/src/ir/libcall.rs
@@ -22,7 +22,7 @@ use serde::{Deserialize, Serialize};
 #[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub enum LibCall {
     /// probe for stack overflow. These are emitted for functions which need
-    /// when the `probestack_enabled` setting is true.
+    /// when the `enable_probestack` setting is true.
     Probestack,
     /// ceil.f32
     CeilF32,
@@ -202,7 +202,7 @@ fn make_funcref(
     func.import_function(ExtFuncData {
         name: ExternalName::LibCall(libcall),
         signature: sigref,
-        colocated: isa.flags().colocated_libcalls(),
+        colocated: isa.flags().use_colocated_libcalls(),
     })
 }
 
diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 011548a5b8..4c1925708d 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -453,7 +453,7 @@ pub fn prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> Codege
 
 fn baldrdash_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> CodegenResult<()> {
     debug_assert!(
-        !isa.flags().probestack_enabled(),
+        !isa.flags().enable_probestack(),
         "baldrdash does not expect cranelift to emit stack probes"
     );
 
@@ -754,8 +754,7 @@ fn insert_common_prologue(
 
     // Allocate stack frame storage.
     if stack_size > 0 {
-        if isa.flags().probestack_enabled()
-            && stack_size > (1 << isa.flags().probestack_size_log2())
+        if isa.flags().enable_probestack() && stack_size > (1 << isa.flags().probestack_size_log2())
         {
             // Emit a stack probe.
             let rax = RU::rax as RegUnit;
diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index dcebff7ce4..5186576ec3 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -191,7 +191,7 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
     }
 
     // Now that we've lowered all br_tables, we don't need the jump tables anymore.
-    if !isa.flags().jump_tables_enabled() {
+    if !isa.flags().enable_jump_tables() {
         pos.func.jump_tables.clear();
     }
 }
@@ -276,7 +276,7 @@ fn expand_br_table(
     cfg: &mut ControlFlowGraph,
     isa: &dyn TargetIsa,
 ) {
-    if isa.flags().jump_tables_enabled() {
+    if isa.flags().enable_jump_tables() {
         expand_br_table_jt(inst, func, cfg, isa);
     } else {
         expand_br_table_conds(inst, func, cfg, isa);
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index bb0e78a1af..b6e55b6147 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -384,7 +384,7 @@ mod tests {
              probestack_size_log2 = 12\n\
              enable_verifier = true\n\
              is_pic = false\n\
-             colocated_libcalls = false\n\
+             use_colocated_libcalls = false\n\
              avoid_div_traps = false\n\
              enable_float = true\n\
              enable_nan_canonicalization = false\n\
@@ -393,10 +393,10 @@ mod tests {
              enable_simd = false\n\
              enable_atomics = true\n\
              enable_safepoints = false\n\
-             allones_funcaddrs = false\n\
-             probestack_enabled = true\n\
+             emit_all_ones_funcaddrs = false\n\
+             enable_probestack = true\n\
              probestack_func_adjusts_sp = false\n\
-             jump_tables_enabled = true\n"
+             enable_jump_tables = true\n"
         );
         assert_eq!(f.opt_level(), super::OptLevel::None);
         assert_eq!(f.enable_simd(), false);
diff --git a/cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif
index 5a6a5b9708..124803af6c 100644
--- a/cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif
+++ b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif
@@ -1,7 +1,7 @@
 ; binary emission of 32-bit code.
 test binemit
 set opt_level=speed_and_size
-set allones_funcaddrs
+set emit_all_ones_funcaddrs
 target i686 haswell
 
 ; The binary encodings can be verified with the command:
@@ -9,7 +9,7 @@ target i686 haswell
 ;   sed -ne 's/^ *; asm: *//p' filetests/isa/x86/allones_funcaddrs32.clif | llvm-mc -show-encoding -triple=i386
 ;
 
-; Tests from binary32.clif affected by allones_funcaddrs.
+; Tests from binary32.clif affected by emit_all_ones_funcaddrs.
 function %I32() {
     sig0 = ()
     fn0 = %foo()
diff --git a/cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif
index 617db5a445..f4dd9f2a6e 100644
--- a/cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif
+++ b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif
@@ -1,7 +1,7 @@
 ; binary emission of 64-bit code.
 test binemit
 set opt_level=speed_and_size
-set allones_funcaddrs
+set emit_all_ones_funcaddrs
 target x86_64 haswell
 
 ; The binary encodings can be verified with the command:
@@ -9,7 +9,7 @@ target x86_64 haswell
 ;   sed -ne 's/^ *; asm: *//p' filetests/isa/x86/allones_funcaddrs64.clif | llvm-mc -show-encoding -triple=x86_64
 ;
 
-; Tests from binary64.clif affected by allones_funcaddrs.
+; Tests from binary64.clif affected by emit_all_ones_funcaddrs.
 function %I64() {
     sig0 = ()
     fn0 = %foo()
diff --git a/cranelift/filetests/filetests/isa/x86/baldrdash-table-sig-reg.clif b/cranelift/filetests/filetests/isa/x86/baldrdash-table-sig-reg.clif
index 73aaaeb283..6d1f72203d 100644
--- a/cranelift/filetests/filetests/isa/x86/baldrdash-table-sig-reg.clif
+++ b/cranelift/filetests/filetests/isa/x86/baldrdash-table-sig-reg.clif
@@ -1,5 +1,5 @@
 test compile
-set probestack_enabled=false
+set enable_probestack=false
 target i686
 
 function u0:0(i32 vmctx) baldrdash_system_v {
diff --git a/cranelift/filetests/filetests/isa/x86/compile-vconst.clif b/cranelift/filetests/filetests/isa/x86/compile-vconst.clif
index c64c9fc503..ee6dff07db 100644
--- a/cranelift/filetests/filetests/isa/x86/compile-vconst.clif
+++ b/cranelift/filetests/filetests/isa/x86/compile-vconst.clif
@@ -1,6 +1,6 @@
 test compile
 set enable_simd=true
-set probestack_enabled=false
+set enable_probestack=false
 target x86_64 haswell
 
 ; use baldrdash calling convention here for simplicity (avoids prologue, epilogue)
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-splat.clif b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
index 980b4b2c65..38731c778f 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
@@ -1,6 +1,6 @@
 test compile
 set enable_simd=true
-set probestack_enabled=false
+set enable_probestack=false
 target x86_64 haswell
 
 ; use baldrdash_system_v calling convention here for simplicity (avoids prologue, epilogue)
diff --git a/cranelift/filetests/filetests/isa/x86/probestack-adjusts-sp.clif b/cranelift/filetests/filetests/isa/x86/probestack-adjusts-sp.clif
index 656bd776ec..934e308bc3 100644
--- a/cranelift/filetests/filetests/isa/x86/probestack-adjusts-sp.clif
+++ b/cranelift/filetests/filetests/isa/x86/probestack-adjusts-sp.clif
@@ -1,5 +1,5 @@
 test compile
-set colocated_libcalls=1
+set use_colocated_libcalls=1
 set probestack_func_adjusts_sp=1
 target x86_64
 
diff --git a/cranelift/filetests/filetests/isa/x86/probestack-disabled.clif b/cranelift/filetests/filetests/isa/x86/probestack-disabled.clif
index 267085b0c5..f548e1a11f 100644
--- a/cranelift/filetests/filetests/isa/x86/probestack-disabled.clif
+++ b/cranelift/filetests/filetests/isa/x86/probestack-disabled.clif
@@ -1,6 +1,6 @@
 test compile
-set colocated_libcalls=1
-set probestack_enabled=0
+set use_colocated_libcalls=1
+set enable_probestack=0
 target x86_64
 
 ; Like %big in probestack.clif, but with probes disabled.
diff --git a/cranelift/filetests/filetests/isa/x86/probestack-size.clif b/cranelift/filetests/filetests/isa/x86/probestack-size.clif
index 4c147b9e24..6b01b5cd92 100644
--- a/cranelift/filetests/filetests/isa/x86/probestack-size.clif
+++ b/cranelift/filetests/filetests/isa/x86/probestack-size.clif
@@ -1,5 +1,5 @@
 test compile
-set colocated_libcalls=1
+set use_colocated_libcalls=1
 set probestack_size_log2=13
 target x86_64
 
diff --git a/cranelift/filetests/filetests/isa/x86/probestack.clif b/cranelift/filetests/filetests/isa/x86/probestack.clif
index fbfd656309..0fae5f33e2 100644
--- a/cranelift/filetests/filetests/isa/x86/probestack.clif
+++ b/cranelift/filetests/filetests/isa/x86/probestack.clif
@@ -1,5 +1,5 @@
 test compile
-set colocated_libcalls=1
+set use_colocated_libcalls=1
 target x86_64
 
 ; A function with a big stack frame. This should have a stack probe.
diff --git a/cranelift/filetests/filetests/isa/x86/relax_branch.clif b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
index ae59a32b0d..7fb8a85167 100644
--- a/cranelift/filetests/filetests/isa/x86/relax_branch.clif
+++ b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
@@ -2,8 +2,8 @@ test binemit
 set opt_level=speed_and_size
 set avoid_div_traps
 set baldrdash_prologue_words=3
-set allones_funcaddrs
-set probestack_enabled=false
+set emit_all_ones_funcaddrs
+set enable_probestack=false
 target x86_64 haswell
 
 ; This checks that a branch that is too far away is getting relaxed. In
diff --git a/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif
index 6db3d12f40..7fa27eecc6 100644
--- a/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif
+++ b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif
@@ -1,6 +1,6 @@
 test compile
 set opt_level=speed_and_size
-set probestack_enabled=false
+set enable_probestack=false
 set enable_simd
 target x86_64
 
diff --git a/cranelift/filetests/filetests/legalizer/br_table_cond.clif b/cranelift/filetests/filetests/legalizer/br_table_cond.clif
index bd823d3cad..dc0bd6473c 100644
--- a/cranelift/filetests/filetests/legalizer/br_table_cond.clif
+++ b/cranelift/filetests/filetests/legalizer/br_table_cond.clif
@@ -1,6 +1,6 @@
 test legalizer
-set probestack_enabled=false
-set jump_tables_enabled=false
+set enable_probestack=false
+set enable_jump_tables=false
 target x86_64
 
 ; Test that when jump_tables_enables is false, all jump tables are eliminated.
diff --git a/cranelift/filetests/filetests/legalizer/empty_br_table.clif b/cranelift/filetests/filetests/legalizer/empty_br_table.clif
index 6dfceb5a6b..b043733fba 100644
--- a/cranelift/filetests/filetests/legalizer/empty_br_table.clif
+++ b/cranelift/filetests/filetests/legalizer/empty_br_table.clif
@@ -1,6 +1,6 @@
 test legalizer
-set probestack_enabled=false
-set jump_tables_enabled=false
+set enable_probestack=false
+set enable_jump_tables=false
 target x86_64
 
 function u0:0(i64) {
diff --git a/cranelift/filetests/filetests/regalloc/unreachable_code.clif b/cranelift/filetests/filetests/regalloc/unreachable_code.clif
index 79e95ab9b0..0eac70fc6f 100644
--- a/cranelift/filetests/filetests/regalloc/unreachable_code.clif
+++ b/cranelift/filetests/filetests/regalloc/unreachable_code.clif
@@ -1,7 +1,7 @@
 ; Use "test compile" here otherwise the dead blocks won't be eliminated.
 test compile
 
-set probestack_enabled=0
+set enable_probestack=0
 target x86_64 haswell
 
 ; This function contains unreachable blocks which trip up the register

From 1d504ecf6dc5554f4a9ff364a74f6413488aa0cf Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 14 Jan 2020 11:42:22 -0800
Subject: [PATCH 3010/3084] Correctly count the number of wasm parameters.
 (#1337)

* Correctly count the number of wasm parameters.

Following up on #1329, this further replaces `num_normal_params` with a function
which calls `is_wasm_parameter` to correctly count the number of wasm
parameters a function has.

* Move is_wasm_parameter's implementation into the trait.
---
 cranelift/codegen/src/ir/extfunc.rs    | 10 ----------
 cranelift/wasm/src/environ/dummy.rs    |  4 ----
 cranelift/wasm/src/environ/spec.rs     |  4 +++-
 cranelift/wasm/src/func_translator.rs  |  2 +-
 cranelift/wasm/src/state/func_state.rs | 16 ++++++++++++++--
 5 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/cranelift/codegen/src/ir/extfunc.rs b/cranelift/codegen/src/ir/extfunc.rs
index ab79bb5ea0..42030b2e9d 100644
--- a/cranelift/codegen/src/ir/extfunc.rs
+++ b/cranelift/codegen/src/ir/extfunc.rs
@@ -88,16 +88,6 @@ impl Signature {
             .count()
     }
 
-    /// Count the number of normal parameters in a signature.
-    /// Exclude special-purpose parameters that represent runtime stuff and not WebAssembly
-    /// arguments.
-    pub fn num_normal_params(&self) -> usize {
-        self.params
-            .iter()
-            .filter(|arg| arg.purpose == ArgumentPurpose::Normal)
-            .count()
-    }
-
     /// Does this signature take an struct return pointer parameter?
     pub fn uses_struct_return_param(&self) -> bool {
         self.uses_special_param(ArgumentPurpose::StructReturn)
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index cb6d78e9f9..6b6f210166 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -201,10 +201,6 @@ impl<'dummy_environment> TargetEnvironment for DummyFuncEnvironment<'dummy_envir
 }
 
 impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environment> {
-    fn is_wasm_parameter(&self, func: &ir::Function, index: usize) -> bool {
-        func.signature.params[index].purpose == ir::ArgumentPurpose::Normal
-    }
-
     fn return_mode(&self) -> ReturnMode {
         self.return_mode
     }
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 71f22c8b39..74dd1e8441 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -140,7 +140,9 @@ pub trait TargetEnvironment {
 pub trait FuncEnvironment: TargetEnvironment {
     /// Is the given parameter of the given function a wasm-level parameter, as opposed to a hidden
     /// parameter added for use by the implementation?
-    fn is_wasm_parameter(&self, func: &ir::Function, index: usize) -> bool;
+    fn is_wasm_parameter(&self, signature: &ir::Signature, index: usize) -> bool {
+        signature.params[index].purpose == ir::ArgumentPurpose::Normal
+    }
 
     /// Should the code be structured to use a single `fallthrough_return` instruction at the end
     /// of the function body, rather than `return` instructions as needed? This is used by VMs
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index 2e7c1bc766..f7ac10cfc3 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -135,7 +135,7 @@ fn declare_wasm_parameters(
         let param_type = builder.func.signature.params[i];
         // There may be additional special-purpose parameters in addition to the normal WebAssembly
         // signature parameters. For example, a `vmctx` pointer.
-        if environ.is_wasm_parameter(&builder.func, i) {
+        if environ.is_wasm_parameter(&builder.func.signature, i) {
             // This is a normal WebAssembly signature parameter, so create a local for it.
             let local = Variable::new(next_local);
             builder.declare_var(local, param_type.value_type);
diff --git a/cranelift/wasm/src/state/func_state.rs b/cranelift/wasm/src/state/func_state.rs
index 98f75e6bc9..0ea0e13ea9 100644
--- a/cranelift/wasm/src/state/func_state.rs
+++ b/cranelift/wasm/src/state/func_state.rs
@@ -474,7 +474,7 @@ impl FuncTranslationState {
             Occupied(entry) => Ok(*entry.get()),
             Vacant(entry) => {
                 let sig = environ.make_indirect_sig(func, index)?;
-                Ok(*entry.insert((sig, func.dfg.signatures[sig].num_normal_params())))
+                Ok(*entry.insert((sig, num_wasm_parameters(environ, &func.dfg.signatures[sig]))))
             }
         }
     }
@@ -495,8 +495,20 @@ impl FuncTranslationState {
             Vacant(entry) => {
                 let fref = environ.make_direct_func(func, index)?;
                 let sig = func.dfg.ext_funcs[fref].signature;
-                Ok(*entry.insert((fref, func.dfg.signatures[sig].num_normal_params())))
+                Ok(*entry.insert((
+                    fref,
+                    num_wasm_parameters(environ, &func.dfg.signatures[sig]),
+                )))
             }
         }
     }
 }
+
+fn num_wasm_parameters(
+    environ: &FE,
+    signature: &ir::Signature,
+) -> usize {
+    (0..signature.params.len())
+        .filter(|index| environ.is_wasm_parameter(signature, *index))
+        .count()
+}

From b00a1818249a5dabc19017aa2eca5dde91850723 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Tue, 14 Jan 2020 13:04:11 -0800
Subject: [PATCH 3011/3084] Bump version to 0.55.0 (#1345)

---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  4 ++--
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 19 files changed, 69 insertions(+), 69 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index b16b00876d..e914bce391 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.54.0"
+version = "0.55.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.54.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.54.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.54.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.54.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.54.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.54.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.54.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.54.0" }
-cranelift-module = { path = "cranelift-module", version = "0.54.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.54.0" }
-cranelift-object = { path = "cranelift-object", version = "0.54.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.54.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.54.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.54.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.55.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.55.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.55.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.55.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.55.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.55.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.55.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.55.0" }
+cranelift-module = { path = "cranelift-module", version = "0.55.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.55.0" }
+cranelift-object = { path = "cranelift-object", version = "0.55.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.55.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.55.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.55.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 5f78dc0d1c..e006c847ec 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.54.0"
+version = "0.55.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.54.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.55.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 1a7b19564d..6692f989e6 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.54.0"
+version = "0.55.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.54.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.54.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.54.0" }
+cranelift-codegen-shared = { path = "./shared", version = "0.55.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.55.0" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.55.0" }
 hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
@@ -30,7 +30,7 @@ byteorder = { version = "1.3.2", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.54.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.55.0" }
 
 [features]
 default = ["std", "basic-blocks", "unwind"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 42e1ffad0a..a1892e14cd 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.54.0"
+version = "0.55.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.54.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.54.0" }
+cranelift-codegen-shared = { path = "../shared", version = "0.55.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.55.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 24a2f67acd..aa43150cc9 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.54.0"
+version = "0.55.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index f8d1bc6500..d5f8de42a9 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.54.0"
+version = "0.55.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index c919b63543..55238ff820 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.54.0"
+version = "0.55.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.54.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.55.0" }
 faerie = "0.14.0"
 goblin = "0.1.0"
 anyhow = "1.0"
@@ -18,7 +18,7 @@ target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.54.0"
+version = "0.55.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index def330776c..8552f650ac 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.54.0"
+version = "0.55.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.54.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.54.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.54.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.55.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.55.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.55.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 gimli = { version = "0.19.0", default-features = false, features = ["read"] }
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 8fbc6a66e0..1955211e7e 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.54.0"
+version = "0.55.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", default-features = false }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 7665c74c94..7649b23995 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.54.0"
+version = "0.55.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.54.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.55.0" }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 thiserror = "1.0.4"
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 1e6b08014d..13b36a4c52 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.54.0"
+version = "0.55.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", default-features = false }
 target-lexicon = "0.10"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 8fd512411f..9198534ab0 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.54.0"
+version = "0.55.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,13 +10,13 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.54.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.55.0" }
 object = { version = "0.17", default-features = false, features = ["write"] }
 target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.54.0"
+version = "0.55.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index f7e0462825..b222310de9 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.54.0"
+version = "0.55.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.54.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.55.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index bd92e935ca..e9972cb898 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.54.0"
+version="0.55.0"
 
 # Update all of the Cargo.toml files.
 #
@@ -27,7 +27,7 @@ done
 
 # Update our local Cargo.lock (not checked in).
 cargo update
-./test-all.sh
+#./test-all.sh
 
 # Commands needed to publish.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index b37b5d0cc2..f40f868743 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.54.0"
+version = "0.55.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0" }
 target-lexicon = "0.10"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 15ff9bafd4..28947777f8 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.54.0"
+version = "0.55.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.54.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.55.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 4bf97d44f8..e356b8cc81 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.54.0"
+version = "0.55.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.54.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.54.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.55.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.55.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -20,7 +20,7 @@ memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.54.0"
+version = "0.55.0"
 default-features = false
 features = ["std"]
 
@@ -32,9 +32,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.54.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.54.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.54.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.55.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.55.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.55.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 127a315803..ce9dd0e875 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.54.0"
+version = "0.55.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.54.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.55.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 6521d71577..6af1eba677 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.54.0"
+version = "0.55.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.47.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.54.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.54.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.54.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.55.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.55.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }

From 435fc71d68d4a4365311c1edf0d04c234263f3ed Mon Sep 17 00:00:00 2001
From: Aleksey Kuznetsov 
Date: Fri, 17 Jan 2020 22:40:56 +0500
Subject: [PATCH 3012/3084] Allow 'clif-util run' to read stdin as intended,
 closes #1004 (#1335)

---
 cranelift/src/run.rs | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/cranelift/src/run.rs b/cranelift/src/run.rs
index 701104fcf5..7825b1852d 100644
--- a/cranelift/src/run.rs
+++ b/cranelift/src/run.rs
@@ -10,9 +10,19 @@ use target_lexicon::Triple;
 use walkdir::WalkDir;
 
 pub fn run(files: Vec, flag_print: bool) -> Result<(), String> {
+    let stdin_exist = files.iter().find(|file| *file == "-").is_some();
+    let filtered_files = files
+        .iter()
+        .filter(|file| *file != "-")
+        .map(|file| file.to_string())
+        .collect::>();
     let mut total = 0;
     let mut errors = 0;
-    for file in iterate_files(files) {
+    let mut special_files: Vec = vec![];
+    if stdin_exist {
+        special_files.push("-".into());
+    }
+    for file in iterate_files(filtered_files).chain(special_files) {
         total += 1;
         match run_single_file(&file) {
             Ok(_) => {

From e1d513ab4b37f2e31847945addb1f8b8fe372fdb Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 17 Jan 2020 12:03:30 -0800
Subject: [PATCH 3013/3084] Fix remaining clippy warnings (#1340)

* clippy: allow complex encoding function

* clippy: remove unnecessary main() function in doctest

* clippy: remove redundant `Type` suffix on LaneType enum variants

* clippy: ignore incorrect debug_assert_with_mut_call warning

* clippy: fix FDE clippy warnings
---
 .../codegen/meta/src/cdsl/instructions.rs     |   7 +-
 cranelift/codegen/meta/src/cdsl/types.rs      |  76 ++++-----
 cranelift/codegen/meta/src/cdsl/typevar.rs    |  11 +-
 .../codegen/meta/src/isa/x86/encodings.rs     |   1 +
 cranelift/codegen/src/isa/x86/fde.rs          |   7 +-
 cranelift/codegen/src/simple_gvn.rs           |   9 +-
 cranelift/filetests/src/test_fde.rs           |   4 +-
 cranelift/frontend/src/lib.rs                 | 158 +++++++++---------
 8 files changed, 138 insertions(+), 135 deletions(-)

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 86ef1e2a8a..10c7f54a7d 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -1346,7 +1346,7 @@ mod test {
         let type1 = TypeSetBuilder::new().ints(8..64).build();
         let in1 = OperandKindFields::TypeVar(TypeVar::new("a", "...", type1));
         let inst = build_fake_instruction(vec![in1], vec![]);
-        inst.bind(LaneType::IntType(I32));
+        inst.bind(LaneType::Int(I32));
     }
 
     #[test]
@@ -1360,7 +1360,7 @@ mod test {
     #[should_panic]
     fn ensure_instructions_fail_to_bind() {
         let inst = build_fake_instruction(vec![], vec![]);
-        inst.bind(BindParameter::Lane(LaneType::IntType(I32)));
+        inst.bind(BindParameter::Lane(LaneType::Int(I32)));
         // Trying to bind to an instruction with no inputs should fail.
     }
 
@@ -1370,8 +1370,7 @@ mod test {
         let type1 = TypeSetBuilder::new().ints(8..64).build();
         let in1 = OperandKindFields::TypeVar(TypeVar::new("a", "...", type1));
         let inst = build_fake_instruction(vec![in1], vec![]);
-        inst.bind(LaneType::IntType(I32))
-            .bind(LaneType::IntType(I64));
+        inst.bind(LaneType::Int(I32)).bind(LaneType::Int(I64));
     }
 
     #[test]
diff --git a/cranelift/codegen/meta/src/cdsl/types.rs b/cranelift/codegen/meta/src/cdsl/types.rs
index ffa0383693..d971f45c61 100644
--- a/cranelift/codegen/meta/src/cdsl/types.rs
+++ b/cranelift/codegen/meta/src/cdsl/types.rs
@@ -140,42 +140,42 @@ impl From for ValueType {
 /// A concrete scalar type that can appear as a vector lane too.
 #[derive(Clone, Copy, PartialEq, Eq, Hash)]
 pub(crate) enum LaneType {
-    BoolType(shared_types::Bool),
-    FloatType(shared_types::Float),
-    IntType(shared_types::Int),
+    Bool(shared_types::Bool),
+    Float(shared_types::Float),
+    Int(shared_types::Int),
 }
 
 impl LaneType {
     /// Return a string containing the documentation comment for this lane type.
     pub fn doc(self) -> String {
         match self {
-            LaneType::BoolType(_) => format!("A boolean type with {} bits.", self.lane_bits()),
-            LaneType::FloatType(shared_types::Float::F32) => String::from(
+            LaneType::Bool(_) => format!("A boolean type with {} bits.", self.lane_bits()),
+            LaneType::Float(shared_types::Float::F32) => String::from(
                 "A 32-bit floating point type represented in the IEEE 754-2008
                 *binary32* interchange format. This corresponds to the :c:type:`float`
                 type in most C implementations.",
             ),
-            LaneType::FloatType(shared_types::Float::F64) => String::from(
+            LaneType::Float(shared_types::Float::F64) => String::from(
                 "A 64-bit floating point type represented in the IEEE 754-2008
                 *binary64* interchange format. This corresponds to the :c:type:`double`
                 type in most C implementations.",
             ),
-            LaneType::IntType(_) if self.lane_bits() < 32 => format!(
+            LaneType::Int(_) if self.lane_bits() < 32 => format!(
                 "An integer type with {} bits.
                 WARNING: arithmetic on {}bit integers is incomplete",
                 self.lane_bits(),
                 self.lane_bits()
             ),
-            LaneType::IntType(_) => format!("An integer type with {} bits.", self.lane_bits()),
+            LaneType::Int(_) => format!("An integer type with {} bits.", self.lane_bits()),
         }
     }
 
     /// Return the number of bits in a lane.
     pub fn lane_bits(self) -> u64 {
         match self {
-            LaneType::BoolType(ref b) => *b as u64,
-            LaneType::FloatType(ref f) => *f as u64,
-            LaneType::IntType(ref i) => *i as u64,
+            LaneType::Bool(ref b) => *b as u64,
+            LaneType::Float(ref f) => *f as u64,
+            LaneType::Int(ref i) => *i as u64,
         }
     }
 
@@ -183,24 +183,24 @@ impl LaneType {
     pub fn number(self) -> u8 {
         constants::LANE_BASE
             + match self {
-                LaneType::BoolType(shared_types::Bool::B1) => 0,
-                LaneType::BoolType(shared_types::Bool::B8) => 1,
-                LaneType::BoolType(shared_types::Bool::B16) => 2,
-                LaneType::BoolType(shared_types::Bool::B32) => 3,
-                LaneType::BoolType(shared_types::Bool::B64) => 4,
-                LaneType::BoolType(shared_types::Bool::B128) => 5,
-                LaneType::IntType(shared_types::Int::I8) => 6,
-                LaneType::IntType(shared_types::Int::I16) => 7,
-                LaneType::IntType(shared_types::Int::I32) => 8,
-                LaneType::IntType(shared_types::Int::I64) => 9,
-                LaneType::IntType(shared_types::Int::I128) => 10,
-                LaneType::FloatType(shared_types::Float::F32) => 11,
-                LaneType::FloatType(shared_types::Float::F64) => 12,
+                LaneType::Bool(shared_types::Bool::B1) => 0,
+                LaneType::Bool(shared_types::Bool::B8) => 1,
+                LaneType::Bool(shared_types::Bool::B16) => 2,
+                LaneType::Bool(shared_types::Bool::B32) => 3,
+                LaneType::Bool(shared_types::Bool::B64) => 4,
+                LaneType::Bool(shared_types::Bool::B128) => 5,
+                LaneType::Int(shared_types::Int::I8) => 6,
+                LaneType::Int(shared_types::Int::I16) => 7,
+                LaneType::Int(shared_types::Int::I32) => 8,
+                LaneType::Int(shared_types::Int::I64) => 9,
+                LaneType::Int(shared_types::Int::I128) => 10,
+                LaneType::Float(shared_types::Float::F32) => 11,
+                LaneType::Float(shared_types::Float::F64) => 12,
             }
     }
 
     pub fn bool_from_bits(num_bits: u16) -> LaneType {
-        LaneType::BoolType(match num_bits {
+        LaneType::Bool(match num_bits {
             1 => shared_types::Bool::B1,
             8 => shared_types::Bool::B8,
             16 => shared_types::Bool::B16,
@@ -212,7 +212,7 @@ impl LaneType {
     }
 
     pub fn int_from_bits(num_bits: u16) -> LaneType {
-        LaneType::IntType(match num_bits {
+        LaneType::Int(match num_bits {
             8 => shared_types::Int::I8,
             16 => shared_types::Int::I16,
             32 => shared_types::Int::I32,
@@ -223,7 +223,7 @@ impl LaneType {
     }
 
     pub fn float_from_bits(num_bits: u16) -> LaneType {
-        LaneType::FloatType(match num_bits {
+        LaneType::Float(match num_bits {
             32 => shared_types::Float::F32,
             64 => shared_types::Float::F64,
             _ => unreachable!("unxpected num bits for float"),
@@ -240,14 +240,14 @@ impl LaneType {
 
     pub fn is_float(self) -> bool {
         match self {
-            LaneType::FloatType(_) => true,
+            LaneType::Float(_) => true,
             _ => false,
         }
     }
 
     pub fn is_int(self) -> bool {
         match self {
-            LaneType::IntType(_) => true,
+            LaneType::Int(_) => true,
             _ => false,
         }
     }
@@ -256,9 +256,9 @@ impl LaneType {
 impl fmt::Display for LaneType {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
-            LaneType::BoolType(_) => write!(f, "b{}", self.lane_bits()),
-            LaneType::FloatType(_) => write!(f, "f{}", self.lane_bits()),
-            LaneType::IntType(_) => write!(f, "i{}", self.lane_bits()),
+            LaneType::Bool(_) => write!(f, "b{}", self.lane_bits()),
+            LaneType::Float(_) => write!(f, "f{}", self.lane_bits()),
+            LaneType::Int(_) => write!(f, "i{}", self.lane_bits()),
         }
     }
 }
@@ -270,9 +270,9 @@ impl fmt::Debug for LaneType {
             f,
             "{}",
             match *self {
-                LaneType::BoolType(_) => format!("BoolType({})", inner_msg),
-                LaneType::FloatType(_) => format!("FloatType({})", inner_msg),
-                LaneType::IntType(_) => format!("IntType({})", inner_msg),
+                LaneType::Bool(_) => format!("BoolType({})", inner_msg),
+                LaneType::Float(_) => format!("FloatType({})", inner_msg),
+                LaneType::Int(_) => format!("IntType({})", inner_msg),
             }
         )
     }
@@ -281,21 +281,21 @@ impl fmt::Debug for LaneType {
 /// Create a LaneType from a given bool variant.
 impl From for LaneType {
     fn from(b: shared_types::Bool) -> Self {
-        LaneType::BoolType(b)
+        LaneType::Bool(b)
     }
 }
 
 /// Create a LaneType from a given float variant.
 impl From for LaneType {
     fn from(f: shared_types::Float) -> Self {
-        LaneType::FloatType(f)
+        LaneType::Float(f)
     }
 }
 
 /// Create a LaneType from a given int variant.
 impl From for LaneType {
     fn from(i: shared_types::Int) -> Self {
-        LaneType::IntType(i)
+        LaneType::Int(i)
     }
 }
 
diff --git a/cranelift/codegen/meta/src/cdsl/typevar.rs b/cranelift/codegen/meta/src/cdsl/typevar.rs
index 3748656da3..302da4561f 100644
--- a/cranelift/codegen/meta/src/cdsl/typevar.rs
+++ b/cranelift/codegen/meta/src/cdsl/typevar.rs
@@ -73,15 +73,15 @@ impl TypeVar {
         builder = builder.simd_lanes(num_lanes..num_lanes);
 
         let builder = match scalar_type {
-            LaneType::IntType(int_type) => {
+            LaneType::Int(int_type) => {
                 let bits = int_type as RangeBound;
                 builder.ints(bits..bits)
             }
-            LaneType::FloatType(float_type) => {
+            LaneType::Float(float_type) => {
                 let bits = float_type as RangeBound;
                 builder.floats(bits..bits)
             }
-            LaneType::BoolType(bool_type) => {
+            LaneType::Bool(bool_type) => {
                 let bits = bool_type as RangeBound;
                 builder.bools(bits..bits)
             }
@@ -1200,8 +1200,7 @@ fn test_typevar_singleton() {
     use crate::shared::types as shared_types;
 
     // Test i32.
-    let typevar =
-        TypeVar::new_singleton(ValueType::Lane(LaneType::IntType(shared_types::Int::I32)));
+    let typevar = TypeVar::new_singleton(ValueType::Lane(LaneType::Int(shared_types::Int::I32)));
     assert_eq!(typevar.name, "i32");
     assert_eq!(typevar.type_set.ints, num_set![32]);
     assert!(typevar.type_set.floats.is_empty());
@@ -1211,7 +1210,7 @@ fn test_typevar_singleton() {
 
     // Test f32x4.
     let typevar = TypeVar::new_singleton(ValueType::Vector(VectorType::new(
-        LaneType::FloatType(shared_types::Float::F32),
+        LaneType::Float(shared_types::Float::F32),
         4,
     )));
     assert_eq!(typevar.name, "f32x4");
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index ad4934ee97..8baa109e5c 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1530,6 +1530,7 @@ fn define_alu(
 }
 
 #[inline(never)]
+#[allow(clippy::cognitive_complexity)]
 fn define_simd(
     e: &mut PerCpuModeEncodings,
     shared_defs: &SharedDefinitions,
diff --git a/cranelift/codegen/src/isa/x86/fde.rs b/cranelift/codegen/src/isa/x86/fde.rs
index 6572c650b7..3e04c93af2 100644
--- a/cranelift/codegen/src/isa/x86/fde.rs
+++ b/cranelift/codegen/src/isa/x86/fde.rs
@@ -10,7 +10,6 @@ use gimli::write::{
     FrameDescriptionEntry, FrameTable, Result, Writer,
 };
 use gimli::{Encoding, Format, LittleEndian, Register, X86_64};
-use std::ptr;
 
 pub type FDERelocEntry = (FrameUnwindOffset, Reloc);
 
@@ -196,7 +195,7 @@ pub fn emit_fde(func: &Function, isa: &dyn TargetIsa, sink: &mut dyn FrameUnwind
             assert!(last_offset <= address_offset);
             if let Some(cmds) = frame_layout.instructions.get(&inst) {
                 for cmd in cmds.iter() {
-                    changes.push((address_offset, cmd.clone()));
+                    changes.push((address_offset, *cmd));
                 }
             }
             last_offset = address_offset;
@@ -252,7 +251,9 @@ pub fn emit_fde(func: &Function, isa: &dyn TargetIsa, sink: &mut dyn FrameUnwind
     for (off, r) in relocs {
         sink.reloc(r, off + unwind_start);
     }
-    let fde_offset = unsafe { ptr::read::(bytes.as_ptr() as *const u32) } as usize + 4;
+
+    let cie_len = u32::from_le_bytes(bytes.as_slice()[..4].try_into().unwrap());
+    let fde_offset = cie_len as usize + 4;
     sink.set_entry_offset(unwind_start + fde_offset);
 
     // Need 0 marker for GCC unwind to end FDE "list".
diff --git a/cranelift/codegen/src/simple_gvn.rs b/cranelift/codegen/src/simple_gvn.rs
index ff8696aee6..60771d4cd8 100644
--- a/cranelift/codegen/src/simple_gvn.rs
+++ b/cranelift/codegen/src/simple_gvn.rs
@@ -124,8 +124,13 @@ pub fn do_simple_gvn(func: &mut Function, domtree: &mut DominatorTree) {
             use crate::scoped_hash_map::Entry::*;
             match visible_values.entry(key) {
                 Occupied(entry) => {
-                    let layout = &func.layout;
-                    debug_assert!(domtree.dominates(*entry.get(), inst, layout));
+                    #[allow(clippy::debug_assert_with_mut_call)]
+                    {
+                        // Clippy incorrectly believes `&func.layout` should not be used here:
+                        // https://github.com/rust-lang/rust-clippy/issues/4737
+                        debug_assert!(domtree.dominates(*entry.get(), inst, &func.layout));
+                    }
+
                     // If the redundant instruction is representing the current
                     // scope, pick a new representative.
                     let old = scope_stack.last_mut().unwrap();
diff --git a/cranelift/filetests/src/test_fde.rs b/cranelift/filetests/src/test_fde.rs
index 1a9e3250d7..3e3747fdde 100644
--- a/cranelift/filetests/src/test_fde.rs
+++ b/cranelift/filetests/src/test_fde.rs
@@ -113,13 +113,13 @@ mod dwarfdump {
 
     impl From for Error {
         fn from(err: gimli::Error) -> Self {
-            Error::GimliError(err)
+            Self::GimliError(err)
         }
     }
 
     impl From for Error {
         fn from(_: fmt::Error) -> Self {
-            Error::IoError
+            Self::IoError
         }
     }
 
diff --git a/cranelift/frontend/src/lib.rs b/cranelift/frontend/src/lib.rs
index 0801be740d..d6d63381ce 100644
--- a/cranelift/frontend/src/lib.rs
+++ b/cranelift/frontend/src/lib.rs
@@ -75,92 +75,90 @@
 //! use cranelift_codegen::verifier::verify_function;
 //! use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
 //!
-//! fn main() {
-//!     let mut sig = Signature::new(CallConv::SystemV);
-//!     sig.returns.push(AbiParam::new(I32));
-//!     sig.params.push(AbiParam::new(I32));
-//!     let mut fn_builder_ctx = FunctionBuilderContext::new();
-//!     let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
+//! let mut sig = Signature::new(CallConv::SystemV);
+//! sig.returns.push(AbiParam::new(I32));
+//! sig.params.push(AbiParam::new(I32));
+//! let mut fn_builder_ctx = FunctionBuilderContext::new();
+//! let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
+//! {
+//!     let mut builder = FunctionBuilder::new(&mut func, &mut fn_builder_ctx);
+//!
+//!     let block0 = builder.create_ebb();
+//!     let block1 = builder.create_ebb();
+//!     let block2 = builder.create_ebb();
+//!     let block3 = builder.create_ebb();
+//!     let x = Variable::new(0);
+//!     let y = Variable::new(1);
+//!     let z = Variable::new(2);
+//!     builder.declare_var(x, I32);
+//!     builder.declare_var(y, I32);
+//!     builder.declare_var(z, I32);
+//!     builder.append_ebb_params_for_function_params(block0);
+//!
+//!     builder.switch_to_block(block0);
+//!     builder.seal_block(block0);
 //!     {
-//!         let mut builder = FunctionBuilder::new(&mut func, &mut fn_builder_ctx);
+//!         let tmp = builder.ebb_params(block0)[0]; // the first function parameter
+//!         builder.def_var(x, tmp);
+//!     }
+//!     {
+//!         let tmp = builder.ins().iconst(I32, 2);
+//!         builder.def_var(y, tmp);
+//!     }
+//!     {
+//!         let arg1 = builder.use_var(x);
+//!         let arg2 = builder.use_var(y);
+//!         let tmp = builder.ins().iadd(arg1, arg2);
+//!         builder.def_var(z, tmp);
+//!     }
+//!     builder.ins().jump(block1, &[]);
 //!
-//!         let block0 = builder.create_ebb();
-//!         let block1 = builder.create_ebb();
-//!         let block2 = builder.create_ebb();
-//!         let block3 = builder.create_ebb();
-//!         let x = Variable::new(0);
-//!         let y = Variable::new(1);
-//!         let z = Variable::new(2);
-//!         builder.declare_var(x, I32);
-//!         builder.declare_var(y, I32);
-//!         builder.declare_var(z, I32);
-//!         builder.append_ebb_params_for_function_params(block0);
+//!     builder.switch_to_block(block1);
+//!     {
+//!         let arg1 = builder.use_var(y);
+//!         let arg2 = builder.use_var(z);
+//!         let tmp = builder.ins().iadd(arg1, arg2);
+//!         builder.def_var(z, tmp);
+//!     }
+//!     {
+//!         let arg = builder.use_var(y);
+//!         builder.ins().brnz(arg, block3, &[]);
+//!     }
+//!     builder.ins().jump(block2, &[]);
 //!
-//!         builder.switch_to_block(block0);
-//!         builder.seal_block(block0);
-//!         {
-//!             let tmp = builder.ebb_params(block0)[0]; // the first function parameter
-//!             builder.def_var(x, tmp);
-//!         }
-//!         {
-//!             let tmp = builder.ins().iconst(I32, 2);
-//!             builder.def_var(y, tmp);
-//!         }
-//!         {
-//!             let arg1 = builder.use_var(x);
-//!             let arg2 = builder.use_var(y);
-//!             let tmp = builder.ins().iadd(arg1, arg2);
-//!             builder.def_var(z, tmp);
-//!         }
-//!         builder.ins().jump(block1, &[]);
-//!
-//!         builder.switch_to_block(block1);
-//!         {
-//!             let arg1 = builder.use_var(y);
-//!             let arg2 = builder.use_var(z);
-//!             let tmp = builder.ins().iadd(arg1, arg2);
-//!             builder.def_var(z, tmp);
-//!         }
-//!         {
-//!             let arg = builder.use_var(y);
-//!             builder.ins().brnz(arg, block3, &[]);
-//!         }
-//!         builder.ins().jump(block2, &[]);
-//!
-//!         builder.switch_to_block(block2);
-//!         builder.seal_block(block2);
-//!         {
-//!             let arg1 = builder.use_var(z);
-//!             let arg2 = builder.use_var(x);
-//!             let tmp = builder.ins().isub(arg1, arg2);
-//!             builder.def_var(z, tmp);
-//!         }
-//!         {
-//!             let arg = builder.use_var(y);
-//!             builder.ins().return_(&[arg]);
-//!         }
-//!
-//!         builder.switch_to_block(block3);
-//!         builder.seal_block(block3);
-//!
-//!         {
-//!             let arg1 = builder.use_var(y);
-//!             let arg2 = builder.use_var(x);
-//!             let tmp = builder.ins().isub(arg1, arg2);
-//!             builder.def_var(y, tmp);
-//!         }
-//!         builder.ins().jump(block1, &[]);
-//!         builder.seal_block(block1);
-//!
-//!         builder.finalize();
+//!     builder.switch_to_block(block2);
+//!     builder.seal_block(block2);
+//!     {
+//!         let arg1 = builder.use_var(z);
+//!         let arg2 = builder.use_var(x);
+//!         let tmp = builder.ins().isub(arg1, arg2);
+//!         builder.def_var(z, tmp);
+//!     }
+//!     {
+//!         let arg = builder.use_var(y);
+//!         builder.ins().return_(&[arg]);
 //!     }
 //!
-//!     let flags = settings::Flags::new(settings::builder());
-//!     let res = verify_function(&func, &flags);
-//!     println!("{}", func.display(None));
-//!     if let Err(errors) = res {
-//!         panic!("{}", errors);
+//!     builder.switch_to_block(block3);
+//!     builder.seal_block(block3);
+//!
+//!     {
+//!         let arg1 = builder.use_var(y);
+//!         let arg2 = builder.use_var(x);
+//!         let tmp = builder.ins().isub(arg1, arg2);
+//!         builder.def_var(y, tmp);
 //!     }
+//!     builder.ins().jump(block1, &[]);
+//!     builder.seal_block(block1);
+//!
+//!     builder.finalize();
+//! }
+//!
+//! let flags = settings::Flags::new(settings::builder());
+//! let res = verify_function(&func, &flags);
+//! println!("{}", func.display(None));
+//! if let Err(errors) = res {
+//!     panic!("{}", errors);
 //! }
 //! ```
 

From 1266b68f9aaa454072c0ede353ea5e1f6c3d1236 Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Fri, 17 Jan 2020 14:11:54 -0600
Subject: [PATCH 3014/3084] Use `is_wasm_parameter` in translating wasm calls
 (#1352)

* Use `is_wasm_parameter` in translating wasm calls

Added in #1329 it's now possible for multiple parameters to be non-wasm
parameters, so the previous `param_types` method is no longer suitable
for acquiring all wasm-related parameters, rather then `FuncEnvironment`
must be consulted. This removes usage of `param_types()` as a method
from the wasm translation and instead adds a custom method inline for
filtering the parameters based on `is_wasm_parameter`.

* Apply feedback

* Run rustfmt

* Don't require `mut`

* Run rustfmt
---
 cranelift/codegen/src/ir/extfunc.rs   | 18 -----------------
 cranelift/wasm/src/code_translator.rs | 28 ++++++++++++++++++++++++---
 cranelift/wasm/src/environ/spec.rs    |  6 ++++++
 cranelift/wasm/src/func_translator.rs |  6 ++++--
 4 files changed, 35 insertions(+), 23 deletions(-)

diff --git a/cranelift/codegen/src/ir/extfunc.rs b/cranelift/codegen/src/ir/extfunc.rs
index 42030b2e9d..9274efe9b9 100644
--- a/cranelift/codegen/src/ir/extfunc.rs
+++ b/cranelift/codegen/src/ir/extfunc.rs
@@ -102,24 +102,6 @@ impl Signature {
             .count()
             > 1
     }
-
-    /// Collect the normal parameter types of the signature; see `[ArgumentPurpose::Normal]`.
-    pub fn param_types(&self) -> Vec {
-        self.params
-            .iter()
-            .filter(|ap| ap.purpose == ArgumentPurpose::Normal)
-            .map(|ap| ap.value_type)
-            .collect()
-    }
-
-    /// Collect the normal return types of the signature; see `[ArgumentPurpose::Normal]`.
-    pub fn return_types(&self) -> Vec {
-        self.returns
-            .iter()
-            .filter(|ap| ap.purpose == ArgumentPurpose::Normal)
-            .map(|ap| ap.value_type)
-            .collect()
-    }
 }
 
 /// Wrapper type capable of displaying a `Signature` with correct register names.
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 5e230f37d4..c2caa38a13 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -38,6 +38,7 @@ use cranelift_codegen::ir::{
 };
 use cranelift_codegen::packed_option::ReservedValue;
 use cranelift_frontend::{FunctionBuilder, Variable};
+use std::vec::Vec;
 use wasmparser::{MemoryImmediate, Operator};
 
 // Clippy warns about "flags: _" but its important to document that the flags field is ignored
@@ -439,7 +440,9 @@ pub fn translate_operator(
             };
             {
                 let return_args = state.peekn_mut(return_count);
-                let return_types = &builder.func.signature.return_types();
+                let return_types = wasm_param_types(&builder.func.signature.returns, |i| {
+                    environ.is_wasm_return(&builder.func.signature, i)
+                });
                 bitcast_arguments(return_args, &return_types, builder);
                 match environ.return_mode() {
                     ReturnMode::NormalReturns => builder.ins().return_(return_args),
@@ -463,7 +466,10 @@ pub fn translate_operator(
             let callee_signature =
                 &builder.func.dfg.signatures[builder.func.dfg.ext_funcs[fref].signature];
             let args = state.peekn_mut(num_args);
-            bitcast_arguments(args, &callee_signature.param_types(), builder);
+            let types = wasm_param_types(&callee_signature.params, |i| {
+                environ.is_wasm_parameter(&callee_signature, i)
+            });
+            bitcast_arguments(args, &types, builder);
 
             let call = environ.translate_call(
                 builder.cursor(),
@@ -492,7 +498,10 @@ pub fn translate_operator(
             // Bitcast any vector arguments to their default type, I8X16, before calling.
             let callee_signature = &builder.func.dfg.signatures[sigref];
             let args = state.peekn_mut(num_args);
-            bitcast_arguments(args, &callee_signature.param_types(), builder);
+            let types = wasm_param_types(&callee_signature.params, |i| {
+                environ.is_wasm_parameter(&callee_signature, i)
+            });
+            bitcast_arguments(args, &types, builder);
 
             let call = environ.translate_call_indirect(
                 builder.cursor(),
@@ -1932,3 +1941,16 @@ pub fn bitcast_arguments(
         }
     }
 }
+
+/// A helper to extract all the `Type` listings of each variable in `params`
+/// for only parameters the return true for `is_wasm`, typically paired with
+/// `is_wasm_return` or `is_wasm_parameter`.
+pub fn wasm_param_types(params: &[ir::AbiParam], is_wasm: impl Fn(usize) -> bool) -> Vec {
+    let mut ret = Vec::with_capacity(params.len());
+    for (i, param) in params.iter().enumerate() {
+        if is_wasm(i) {
+            ret.push(param.value_type);
+        }
+    }
+    return ret;
+}
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 74dd1e8441..c6928a491f 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -144,6 +144,12 @@ pub trait FuncEnvironment: TargetEnvironment {
         signature.params[index].purpose == ir::ArgumentPurpose::Normal
     }
 
+    /// Is the given return of the given function a wasm-level parameter, as
+    /// opposed to a hidden parameter added for use by the implementation?
+    fn is_wasm_return(&self, signature: &ir::Signature, index: usize) -> bool {
+        signature.returns[index].purpose == ir::ArgumentPurpose::Normal
+    }
+
     /// Should the code be structured to use a single `fallthrough_return` instruction at the end
     /// of the function body, rather than `return` instructions as needed? This is used by VMs
     /// to append custom epilogues.
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index f7ac10cfc3..a7dbf0820a 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -4,7 +4,7 @@
 //! function to Cranelift IR guided by a `FuncEnvironment` which provides information about the
 //! WebAssembly module and the runtime environment.
 
-use crate::code_translator::{bitcast_arguments, translate_operator};
+use crate::code_translator::{bitcast_arguments, translate_operator, wasm_param_types};
 use crate::environ::{FuncEnvironment, ReturnMode, WasmResult};
 use crate::state::{FuncTranslationState, ModuleTranslationState};
 use crate::translation_utils::get_vmctx_value_label;
@@ -245,7 +245,9 @@ fn parse_function_body(
         if !builder.is_unreachable() {
             match environ.return_mode() {
                 ReturnMode::NormalReturns => {
-                    let return_types = &builder.func.signature.return_types();
+                    let return_types = wasm_param_types(&builder.func.signature.returns, |i| {
+                        environ.is_wasm_return(&builder.func.signature, i)
+                    });
                     bitcast_arguments(&mut state.stack, &return_types, builder);
                     builder.ins().return_(&state.stack)
                 }

From 80d11e3f8d5b28a461d9a3a7956f6bda48155d82 Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 17 Jan 2020 14:33:52 -0800
Subject: [PATCH 3015/3084] Bump version to 0.56.0 (#1356)

---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  4 ++--
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 19 files changed, 69 insertions(+), 69 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index e914bce391..f064d470b1 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.55.0"
+version = "0.56.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.55.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.55.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.55.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.55.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.55.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.55.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.55.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.55.0" }
-cranelift-module = { path = "cranelift-module", version = "0.55.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.55.0" }
-cranelift-object = { path = "cranelift-object", version = "0.55.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.55.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.55.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.55.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.56.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.56.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.56.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.56.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.56.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.56.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.56.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.56.0" }
+cranelift-module = { path = "cranelift-module", version = "0.56.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.56.0" }
+cranelift-object = { path = "cranelift-object", version = "0.56.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.56.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.56.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.56.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index e006c847ec..0fc288dab4 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.55.0"
+version = "0.56.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.55.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.56.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 6692f989e6..188ecf5575 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.55.0"
+version = "0.56.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.55.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.55.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.55.0" }
+cranelift-codegen-shared = { path = "./shared", version = "0.56.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.56.0" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.56.0" }
 hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
@@ -30,7 +30,7 @@ byteorder = { version = "1.3.2", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.55.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.56.0" }
 
 [features]
 default = ["std", "basic-blocks", "unwind"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index a1892e14cd..85f07548eb 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.55.0"
+version = "0.56.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.55.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.55.0" }
+cranelift-codegen-shared = { path = "../shared", version = "0.56.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.56.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index aa43150cc9..81a4ed62e8 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.55.0"
+version = "0.56.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index d5f8de42a9..a9bd2ff323 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.55.0"
+version = "0.56.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 55238ff820..9b632ae9f9 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.55.0"
+version = "0.56.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.55.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.56.0" }
 faerie = "0.14.0"
 goblin = "0.1.0"
 anyhow = "1.0"
@@ -18,7 +18,7 @@ target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.55.0"
+version = "0.56.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 8552f650ac..ae0af21f68 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.55.0"
+version = "0.56.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.55.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.55.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.55.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.56.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.56.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.56.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 gimli = { version = "0.19.0", default-features = false, features = ["read"] }
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 1955211e7e..9e1c930e94 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.55.0"
+version = "0.56.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", default-features = false }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 7649b23995..8bf838d341 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.55.0"
+version = "0.56.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.55.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.56.0" }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 thiserror = "1.0.4"
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 13b36a4c52..cbbe4b1c06 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.55.0"
+version = "0.56.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", default-features = false }
 target-lexicon = "0.10"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 9198534ab0..f65db0affc 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.55.0"
+version = "0.56.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,13 +10,13 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.55.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.56.0" }
 object = { version = "0.17", default-features = false, features = ["write"] }
 target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.55.0"
+version = "0.56.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index b222310de9..f9c483befc 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.55.0"
+version = "0.56.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.55.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.56.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index e9972cb898..34cb615ab4 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.55.0"
+version="0.56.0"
 
 # Update all of the Cargo.toml files.
 #
@@ -27,7 +27,7 @@ done
 
 # Update our local Cargo.lock (not checked in).
 cargo update
-#./test-all.sh
+./test-all.sh
 
 # Commands needed to publish.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index f40f868743..06b8772deb 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.55.0"
+version = "0.56.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0" }
 target-lexicon = "0.10"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 28947777f8..caba72db8c 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.55.0"
+version = "0.56.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.55.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.56.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index e356b8cc81..c9b052ac52 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.55.0"
+version = "0.56.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.55.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.55.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.56.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.56.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -20,7 +20,7 @@ memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.55.0"
+version = "0.56.0"
 default-features = false
 features = ["std"]
 
@@ -32,9 +32,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.55.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.55.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.55.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.56.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.56.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.56.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index ce9dd0e875..07d99f9c74 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.55.0"
+version = "0.56.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.55.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.56.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 6af1eba677..1b58a6c611 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.55.0"
+version = "0.56.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.47.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.55.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.55.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.55.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.56.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.56.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }

From fd04ea2b0615ecc602d835fbd57783b072155ed3 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 17 Jan 2020 14:39:31 -0800
Subject: [PATCH 3016/3084] Fix incorrect assertion for `insertlane` (#1355)

Previously, the assertion checked for `lane > 0` when it should have been `lane >= 0`; since lane is unsigned, this half of the assertion can be entirely removed.
---
 cranelift/codegen/src/isa/x86/enc_tables.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index d45f1314aa..9a5481e228 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -1173,7 +1173,7 @@ fn convert_insertlane(
             // Floats are already in XMM registers and can stay there.
             match value_type {
                 F32X4 => {
-                    assert!(lane > 0 && lane <= 3);
+                    assert!(lane <= 3);
                     let immediate = 0b00_00_00_00 | lane << 4;
                     // Insert 32-bits from replacement (at index 00, bits 7:8) to vector (lane
                     // shifted into bits 5:6).

From 175b9641e03fcbc09b993ec2e51723e7bac35731 Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Mon, 20 Jan 2020 08:29:05 -0500
Subject: [PATCH 3017/3084] frontend: Add `entries()` API for Switch (#1358)

---
 cranelift/frontend/src/switch.rs | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/cranelift/frontend/src/switch.rs b/cranelift/frontend/src/switch.rs
index 25cba38ac3..e4c147d16b 100644
--- a/cranelift/frontend/src/switch.rs
+++ b/cranelift/frontend/src/switch.rs
@@ -62,6 +62,11 @@ impl Switch {
         );
     }
 
+    /// Get a reference to all existing entries
+    pub fn entries(&self) -> &HashMap {
+        &self.cases
+    }
+
     /// Turn the `cases` `HashMap` into a list of `ContiguousCaseRange`s.
     ///
     /// # Postconditions

From ae6ba1e58c64cbeaf217e0e4fcbf3ba2dbeb118c Mon Sep 17 00:00:00 2001
From: jmkrauz <45767732+jmkrauz@users.noreply.github.com>
Date: Tue, 21 Jan 2020 15:20:44 +0100
Subject: [PATCH 3018/3084] Fix narrow_icmp_imm (#1343)

---
 cranelift/codegen/src/legalizer/mod.rs        |  2 +-
 .../isa/riscv/legalize-icmp_imm-i64.clif      | 55 +++++++++++++++++++
 2 files changed, 56 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/filetests/filetests/isa/riscv/legalize-icmp_imm-i64.clif

diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 5186576ec3..3257949955 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -702,7 +702,7 @@ fn narrow_icmp_imm(
 
     let imm_low = pos
         .ins()
-        .iconst(ty_half, imm & (1u128 << (ty_half.bits() - 1)) as i64);
+        .iconst(ty_half, imm & ((1u128 << ty_half.bits()) - 1) as i64);
     let imm_high = pos
         .ins()
         .iconst(ty_half, imm.wrapping_shr(ty_half.bits().into()));
diff --git a/cranelift/filetests/filetests/isa/riscv/legalize-icmp_imm-i64.clif b/cranelift/filetests/filetests/isa/riscv/legalize-icmp_imm-i64.clif
new file mode 100644
index 0000000000..3dd674a5d3
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/riscv/legalize-icmp_imm-i64.clif
@@ -0,0 +1,55 @@
+test legalizer
+target riscv32
+
+; regex: V=v\d+
+
+function %icmp_imm_eq(i64) -> b1 {
+ebb0(v0: i64):
+    v1 = icmp_imm eq v0, 0x20202020_10101010
+    return v1
+}
+; check: ebb0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32):
+; nextln: $(v2l=$V) -> $(v0l)
+; nextln: $(v2h=$V) -> $(v0h)
+; nextln: v0 = iconcat $(v0l), $(v0h)
+; nextln: $(imm_low=$V) = iconst.i32 0x1010_1010
+; nextln: $(imm_high=$V) = iconst.i32 0x2020_2020
+; nextln: $(v3=$V) = icmp eq $(v2l), $(imm_low)
+; nextln: $(v4=$V) = icmp eq $(v2h), $(imm_high)
+; nextln: v1 = band $(v3), $(v4)
+; nextln: return v1, $(link)
+
+function %icmp_imm_ne(i64) -> b1 {
+ebb0(v0: i64):
+    v1 = icmp_imm ne v0, 0x33333333_44444444
+    return v1
+}
+; check: ebb0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32):
+; nextln: $(v2l=$V) -> $(v0l)
+; nextln: $(v2h=$V) -> $(v0h)
+; nextln: v0 = iconcat $(v0l), $(v0h)
+; nextln: $(imm_low=$V) = iconst.i32 0x4444_4444
+; nextln: $(imm_high=$V) = iconst.i32 0x3333_3333
+; nextln: $(v3=$V) = icmp ne $(v2l), $(imm_low)
+; nextln: $(v4=$V) = icmp ne $(v2h), $(imm_high)
+; nextln: v1 = bor $(v3), $(v4)
+; nextln: return v1, $(link)
+
+function %icmp_imm_sge(i64) -> b1 {
+ebb0(v0: i64):
+    v1 = icmp_imm sge v0, 0x01020304_05060708
+    return v1
+}
+; check: ebb0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32):
+; nextln: $(v2l=$V) -> $(v0l)
+; nextln: $(v2h=$V) -> $(v0h)
+; nextln: v0 = iconcat $(v0l), $(v0h)
+; nextln: $(imm_low=$V) = iconst.i32 0x0506_0708
+; nextln: $(imm_high=$V) = iconst.i32 0x0102_0304
+; nextln: $(v3=$V) = icmp sgt $(v2h), $(imm_high)
+; nextln: $(v4=$V) = icmp slt $(v2h), $(imm_high)
+; nextln: $(v5=$V) = icmp uge $(v2l), $(imm_low)
+; nextln: $(v6=$V) = bnot $v4
+; nextln: $(v7=$V) = band $v6, $v5
+; nextln: v1 = bor $(v3), $(v7)
+; nextln: return v1, $(link)

From dfc9e195ee9359c06ed98ef17295089084554f1f Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 21 Jan 2020 08:18:52 -0800
Subject: [PATCH 3019/3084] Add temporary translation of Wasm's `load_splat`
 using Cranelift's `load` and `splat` instructions (#1347)

If/when Cranelift gains a `load_splat` instruction, the `load + splat` could be replaced with a single Cranelift `load_splat`. This change allows the `simd_load_splat.wast` spec test to pass.
---
 cranelift/wasm/src/code_translator.rs | 34 +++++++++++++++++++++++----
 1 file changed, 30 insertions(+), 4 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index c2caa38a13..e6055cea17 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1158,6 +1158,32 @@ pub fn translate_operator(
             let splatted = builder.ins().splat(type_of(op), state.pop1());
             state.push1(splatted)
         }
+        Operator::V8x16LoadSplat {
+            memarg: MemoryImmediate { flags: _, offset },
+        }
+        | Operator::V16x8LoadSplat {
+            memarg: MemoryImmediate { flags: _, offset },
+        }
+        | Operator::V32x4LoadSplat {
+            memarg: MemoryImmediate { flags: _, offset },
+        }
+        | Operator::V64x2LoadSplat {
+            memarg: MemoryImmediate { flags: _, offset },
+        } => {
+            // TODO: For spec compliance, this is initially implemented as a combination of `load +
+            // splat` but could be implemented eventually as a single instruction (`load_splat`).
+            // See https://github.com/bytecodealliance/cranelift/issues/1348.
+            translate_load(
+                *offset,
+                ir::Opcode::Load,
+                type_of(op).lane_type(),
+                builder,
+                state,
+                environ,
+            )?;
+            let splatted = builder.ins().splat(type_of(op), state.pop1());
+            state.push1(splatted)
+        }
         Operator::I8x16ExtractLaneS { lane } | Operator::I16x8ExtractLaneS { lane } => {
             let vector = pop1_with_bitcast(state, type_of(op), builder);
             let extracted = builder.ins().extractlane(vector, lane.clone());
@@ -1410,10 +1436,6 @@ pub fn translate_operator(
         | Operator::I32x4WidenLowI16x8U { .. }
         | Operator::I32x4WidenHighI16x8U { .. }
         | Operator::V8x16Swizzle
-        | Operator::V8x16LoadSplat { .. }
-        | Operator::V16x8LoadSplat { .. }
-        | Operator::V32x4LoadSplat { .. }
-        | Operator::V64x2LoadSplat { .. }
         | Operator::I16x8Load8x8S { .. }
         | Operator::I16x8Load8x8U { .. }
         | Operator::I32x4Load16x4S { .. }
@@ -1734,6 +1756,7 @@ fn type_of(operator: &Operator) -> Type {
 
         Operator::V8x16Shuffle { .. }
         | Operator::I8x16Splat
+        | Operator::V8x16LoadSplat { .. }
         | Operator::I8x16ExtractLaneS { .. }
         | Operator::I8x16ExtractLaneU { .. }
         | Operator::I8x16ReplaceLane { .. }
@@ -1762,6 +1785,7 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::I8x16Mul => I8X16,
 
         Operator::I16x8Splat
+        | Operator::V16x8LoadSplat { .. }
         | Operator::I16x8ExtractLaneS { .. }
         | Operator::I16x8ExtractLaneU { .. }
         | Operator::I16x8ReplaceLane { .. }
@@ -1790,6 +1814,7 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::I16x8Mul => I16X8,
 
         Operator::I32x4Splat
+        | Operator::V32x4LoadSplat { .. }
         | Operator::I32x4ExtractLane { .. }
         | Operator::I32x4ReplaceLane { .. }
         | Operator::I32x4Eq
@@ -1815,6 +1840,7 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::F32x4ConvertI32x4U => I32X4,
 
         Operator::I64x2Splat
+        | Operator::V64x2LoadSplat { .. }
         | Operator::I64x2ExtractLane { .. }
         | Operator::I64x2ReplaceLane { .. }
         | Operator::I64x2Neg

From d6134a6f3a3d71b870a24ee9d152ffdecbea66ba Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 22 Jan 2020 01:48:24 -0800
Subject: [PATCH 3020/3084] Improve issue template: make questions invisible,
 add headings, add space for writing (#1349)

---
 .../.github/ISSUE_TEMPLATE/improvement.md     | 28 +++++++++++++------
 1 file changed, 20 insertions(+), 8 deletions(-)

diff --git a/cranelift/.github/ISSUE_TEMPLATE/improvement.md b/cranelift/.github/ISSUE_TEMPLATE/improvement.md
index 16e082466e..2cc9857213 100644
--- a/cranelift/.github/ISSUE_TEMPLATE/improvement.md
+++ b/cranelift/.github/ISSUE_TEMPLATE/improvement.md
@@ -3,13 +3,25 @@ name: "Improvement"
 about: "A feature request or code improvement."
 ---
 
-Please try to describe precisely what you would like to do in Cranelift and/or
+
 
-- What is the feature or code improvement you would like to do in Cranelift?
-- What is the value of adding this in Cranelift?
-- Do you have an implementation plan, and/or ideas for data structures or
-  algorithms to use?
-- Have you considered alternative implementations? If so, how are they better
-  or worse than your proposal?
+#### Feature
+
+
+
+
+#### Benefit
+
+
+
+
+#### Implementation
+
+
+
+
+#### Alternatives
+
+

From b4c6bfd371fe6bdead48ce80bd135233f96243f6 Mon Sep 17 00:00:00 2001
From: Sean Stangl 
Date: Wed, 22 Jan 2020 09:14:41 -0700
Subject: [PATCH 3021/3084] When splitting a const, insert prior to the
 terminal branch group. (#1325)

* When splitting a const, insert prior to the terminal branch group. Closes #1159

Given code like the following, on x86_64, which does not have i128 registers:

    ebb0(v0: i64):
        v1 = iconst.i128 0
        v2 = icmp_imm eq v0, 1
        brnz v2, ebb1
        jump ebb2(v1)

It would be split to:

    ebb0(v0: i64):
        v1 = iconst.i128 0
        v2 = icmp_imm eq v0, 1
        brnz v2, ebb1
        v3, v4 = isplit.i128 v1
        jump ebb2(v3, v4)

But that fails basic-block invariants. This patch changes that to:

    ebb0(v0: i64):
        v1 = iconst.i128 0
        v2 = icmp_imm eq v0, 1
        v3, v4 = isplit.i128 v1
        brnz v2, ebb1
        jump ebb2(v3, v4)

* Add isplit-bb.clif testcase
---
 cranelift/codegen/src/ir/layout.rs            | 14 ++++++++++
 cranelift/codegen/src/legalizer/split.rs      | 15 ++++++++--
 .../filetests/verifier/isplit-bb.clif         | 28 +++++++++++++++++++
 3 files changed, 55 insertions(+), 2 deletions(-)
 create mode 100644 cranelift/filetests/filetests/verifier/isplit-bb.clif

diff --git a/cranelift/codegen/src/ir/layout.rs b/cranelift/codegen/src/ir/layout.rs
index 7b624a92d2..b090b8d819 100644
--- a/cranelift/codegen/src/ir/layout.rs
+++ b/cranelift/codegen/src/ir/layout.rs
@@ -4,6 +4,7 @@
 //! determined by the `Layout` data structure defined in this module.
 
 use crate::entity::SecondaryMap;
+use crate::ir::dfg::DataFlowGraph;
 use crate::ir::progpoint::{ExpandedProgramPoint, ProgramOrder};
 use crate::ir::{Ebb, Inst};
 use crate::packed_option::PackedOption;
@@ -573,6 +574,19 @@ impl Layout {
         self.insts[inst].prev.expand()
     }
 
+    /// Fetch the first instruction in an ebb's terminal branch group.
+    pub fn canonical_branch_inst(&self, dfg: &DataFlowGraph, ebb: Ebb) -> Option {
+        // Basic blocks permit at most two terminal branch instructions.
+        // If two, the former is conditional and the latter is unconditional.
+        let last = self.last_inst(ebb)?;
+        if let Some(prev) = self.prev_inst(last) {
+            if dfg[prev].opcode().is_branch() {
+                return Some(prev);
+            }
+        }
+        Some(last)
+    }
+
     /// Insert `inst` before the instruction `before` in the same EBB.
     pub fn insert_inst(&mut self, inst: Inst, before: Inst) {
         debug_assert_eq!(self.inst_ebb(inst), None);
diff --git a/cranelift/codegen/src/legalizer/split.rs b/cranelift/codegen/src/legalizer/split.rs
index a3fa29f267..405b18e4f7 100644
--- a/cranelift/codegen/src/legalizer/split.rs
+++ b/cranelift/codegen/src/legalizer/split.rs
@@ -189,6 +189,17 @@ fn perform_repairs(pos: &mut FuncCursor, cfg: &ControlFlowGraph, mut repairs: Ve
 
             // Split the old argument, possibly causing more repairs to be scheduled.
             pos.goto_inst(inst);
+            #[cfg(feature = "basic-blocks")]
+            {
+                let inst_ebb = pos.func.layout.inst_ebb(inst).expect("inst in ebb");
+
+                // Insert split values prior to the terminal branch group.
+                let dfg = &pos.func.dfg;
+                let canonical = pos.func.layout.canonical_branch_inst(dfg, inst_ebb);
+                if let Some(first_branch) = canonical {
+                    pos.goto_inst(first_branch);
+                }
+            }
             let (lo, hi) = split_value(pos, old_arg, repair.concat, &mut repairs);
 
             // The `lo` part replaces the original argument.
@@ -248,8 +259,8 @@ fn split_value(
             }
         }
         ValueDef::Param(ebb, num) => {
-            // This is an EBB parameter. We can split the parameter value unless this is the entry
-            // block.
+            // This is an EBB parameter.
+            // We can split the parameter value unless this is the entry block.
             if pos.func.layout.entry_block() != Some(ebb) {
                 reuse = Some(split_ebb_param(pos, ebb, num, value, concat, repairs));
             }
diff --git a/cranelift/filetests/filetests/verifier/isplit-bb.clif b/cranelift/filetests/filetests/verifier/isplit-bb.clif
new file mode 100644
index 0000000000..ba789706a1
--- /dev/null
+++ b/cranelift/filetests/filetests/verifier/isplit-bb.clif
@@ -0,0 +1,28 @@
+test compile
+target x86_64
+
+function u0:0(i128, i128, i64) -> i128 system_v {
+    ebb0(v0: i128, v1: i128, v2: i64):
+        trap user0
+
+    ebb1:
+        v10 = iconst.i64 0
+        v11 = iconst.i64 0
+        v17 = iconcat v10, v11
+        v12 = iconst.i64 0
+        v13 = iconst.i64 0 
+        v20 = iconcat v12, v13
+        trap user0
+
+    ebb79:
+        v425 = iconst.i64 0
+        v426 = icmp_imm eq v425, 1
+        brnz v426, ebb80
+        jump ebb85(v20, v17)
+
+    ebb80:
+        trap user0
+
+    ebb85(v462: i128, v874: i128):
+        trap user0
+}

From e1446cff8d5b0ad91b5edaf4d7b44d55bdfb6457 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 22 Jan 2020 18:18:23 +0100
Subject: [PATCH 3022/3084] Derive Ord for all entities (#1313)

---
 cranelift/codegen/src/ir/entities.rs | 43 ++++++++++++++++++++++------
 1 file changed, 34 insertions(+), 9 deletions(-)

diff --git a/cranelift/codegen/src/ir/entities.rs b/cranelift/codegen/src/ir/entities.rs
index 02fa9cbc08..6673c71a17 100644
--- a/cranelift/codegen/src/ir/entities.rs
+++ b/cranelift/codegen/src/ir/entities.rs
@@ -31,6 +31,8 @@ use serde::{Deserialize, Serialize};
 ///
 /// You can get an `Ebb` using
 /// [`FunctionBuilder::create_ebb`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.create_ebb)
+///
+/// While the order is stable, it is arbitrary and does not necessarily resemble the layout order.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct Ebb(u32);
 entity_impl!(Ebb, "ebb");
@@ -61,6 +63,8 @@ impl Ebb {
 /// - [`null`](super::InstBuilder::null) for null reference constants
 ///
 /// Any `InstBuilder` instruction that has an output will also return a `Value`.
+///
+/// While the order is stable, it is arbitrary.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct Value(u32);
 entity_impl!(Value, "v");
@@ -91,6 +95,8 @@ impl Value {
 /// on the type of instruction.
 ///
 /// [inst_comment]: https://github.com/bjorn3/rustc_codegen_cranelift/blob/0f8814fd6da3d436a90549d4bb19b94034f2b19c/src/pretty_clif.rs
+///
+/// While the order is stable, it is arbitrary and does not necessarily resemble the layout order.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct Inst(u32);
 entity_impl!(Inst, "inst");
@@ -107,7 +113,9 @@ entity_impl!(Inst, "inst");
 /// [`stack_addr`](super::InstBuilder::stack_addr),
 /// [`stack_load`](super::InstBuilder::stack_load), and
 /// [`stack_store`](super::InstBuilder::stack_store).
-#[derive(Copy, Clone, PartialEq, Eq, Hash)]
+///
+/// While the order is stable, it is arbitrary and does not necessarily resemble the stack order.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 #[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct StackSlot(u32);
 entity_impl!(StackSlot, "ss");
@@ -142,7 +150,9 @@ impl StackSlot {
 ///
 /// `GlobalValue`s can be retrieved with
 /// [`InstBuilder:global_value`](super::InstBuilder::global_value).
-#[derive(Copy, Clone, PartialEq, Eq, Hash)]
+///
+/// While the order is stable, it is arbitrary.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct GlobalValue(u32);
 entity_impl!(GlobalValue, "gv");
 
@@ -164,6 +174,9 @@ impl GlobalValue {
 /// You can store [`ConstantData`](super::ConstantData) in a
 /// [`ConstantPool`](super::ConstantPool) for efficient storage and retrieval.
 /// See [`ConstantPool::insert`](super::ConstantPool::insert).
+///
+/// While the order is stable, it is arbitrary and does not necessarily resemble the order in which
+/// the constants are written in the constant pool.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, Ord, PartialOrd)]
 pub struct Constant(u32);
 entity_impl!(Constant, "const");
@@ -187,7 +200,9 @@ impl Constant {
 /// [`InstructionData`](super::instructions::InstructionData) struct and therefore must be
 /// tracked separately in [`DataFlowGraph::immediates`](super::dfg::DataFlowGraph). `Immediate`
 /// provides a way to reference values stored there.
-#[derive(Copy, Clone, PartialEq, Eq, Hash)]
+///
+/// While the order is stable, it is arbitrary.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct Immediate(u32);
 entity_impl!(Immediate, "imm");
 
@@ -214,7 +229,9 @@ impl Immediate {
 ///
 /// `JumpTable`s can be created with
 /// [`create_jump_table`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.create_jump_table).
-#[derive(Copy, Clone, PartialEq, Eq, Hash)]
+///
+/// While the order is stable, it is arbitrary.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 #[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
 pub struct JumpTable(u32);
 entity_impl!(JumpTable, "jt");
@@ -248,7 +265,9 @@ impl JumpTable {
 /// - [`FuncEnvironment::make_direct_func`](https://docs.rs/cranelift-wasm/*/cranelift_wasm/trait.FuncEnvironment.html#tymethod.make_direct_func)
 /// for functions declared in the same WebAssembly
 /// [`FuncEnvironment`](https://docs.rs/cranelift-wasm/*/cranelift_wasm/trait.FuncEnvironment.html#tymethod.make_direct_func)
-#[derive(Copy, Clone, PartialEq, Eq, Hash)]
+///
+/// While the order is stable, it is arbitrary.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct FuncRef(u32);
 entity_impl!(FuncRef, "fn");
 
@@ -277,7 +296,9 @@ impl FuncRef {
 /// You can retrieve the [`Signature`](super::Signature) that was used to create a `SigRef` with
 /// [`FunctionBuilder::signature`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.signature) or
 /// [`func.dfg.signatures`](super::dfg::DataFlowGraph::signatures).
-#[derive(Copy, Clone, PartialEq, Eq, Hash)]
+///
+/// While the order is stable, it is arbitrary.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct SigRef(u32);
 entity_impl!(SigRef, "sig");
 
@@ -300,7 +321,9 @@ impl SigRef {
 /// [`heap_addr`](super::InstBuilder::heap_addr).
 ///
 /// To create a heap, use [`FunctionBuilder::create_heap`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.create_heap).
-#[derive(Copy, Clone, PartialEq, Eq, Hash)]
+///
+/// While the order is stable, it is arbitrary.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct Heap(u32);
 entity_impl!(Heap, "heap");
 
@@ -324,7 +347,9 @@ impl Heap {
 /// They can be created with [`FuncEnvironment::make_table`](https://docs.rs/cranelift-wasm/*/cranelift_wasm/trait.FuncEnvironment.html#tymethod.make_table).
 /// They can be used with
 /// [`FuncEnvironment::translate_call_indirect`](https://docs.rs/cranelift-wasm/*/cranelift_wasm/trait.FuncEnvironment.html#tymethod.translate_call_indirect).
-#[derive(Copy, Clone, PartialEq, Eq, Hash)]
+///
+/// While the order is stable, it is arbitrary.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub struct Table(u32);
 entity_impl!(Table, "table");
 
@@ -342,7 +367,7 @@ impl Table {
 }
 
 /// An opaque reference to any of the entities defined in this module that can appear in CLIF IR.
-#[derive(Copy, Clone, PartialEq, Eq, Hash)]
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
 pub enum AnyEntity {
     /// The whole function.
     Function,

From cc50e65f311e6260dee9dd8522376260c009276b Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 22 Jan 2020 20:25:35 +0100
Subject: [PATCH 3023/3084] Update gimli to 0.20 (#1361)

---
 cranelift/codegen/Cargo.toml   | 2 +-
 cranelift/filetests/Cargo.toml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 188ecf5575..e135b166a2 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -20,7 +20,7 @@ hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
-gimli = { version = "0.19.0", default-features = false, features = ["write"], optional = true }
+gimli = { version = "0.20.0", default-features = false, features = ["write"], optional = true }
 smallvec = { version = "1.0.0" }
 thiserror = "1.0.4"
 byteorder = { version = "1.3.2", default-features = false }
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index ae0af21f68..c5ce847000 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -16,7 +16,7 @@ cranelift-reader = { path = "../cranelift-reader", version = "0.56.0" }
 cranelift-preopt = { path = "../cranelift-preopt", version = "0.56.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
-gimli = { version = "0.19.0", default-features = false, features = ["read"] }
+gimli = { version = "0.20.0", default-features = false, features = ["read"] }
 log = "0.4.6"
 memmap = "0.7.0"
 num_cpus = "1.8.0"

From 3125431ece5f87576d06d341d9eadbb68b16aac4 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Wed, 22 Jan 2020 17:25:52 +0100
Subject: [PATCH 3024/3084] Address nits from #1325

---
 cranelift/codegen/src/legalizer/split.rs      |  6 ++--
 .../filetests/legalizer/isplit-bb.clif        | 24 ++++++++++++++++
 .../filetests/verifier/isplit-bb.clif         | 28 -------------------
 3 files changed, 28 insertions(+), 30 deletions(-)
 create mode 100644 cranelift/filetests/filetests/legalizer/isplit-bb.clif
 delete mode 100644 cranelift/filetests/filetests/verifier/isplit-bb.clif

diff --git a/cranelift/codegen/src/legalizer/split.rs b/cranelift/codegen/src/legalizer/split.rs
index 405b18e4f7..727c766d66 100644
--- a/cranelift/codegen/src/legalizer/split.rs
+++ b/cranelift/codegen/src/legalizer/split.rs
@@ -194,8 +194,10 @@ fn perform_repairs(pos: &mut FuncCursor, cfg: &ControlFlowGraph, mut repairs: Ve
                 let inst_ebb = pos.func.layout.inst_ebb(inst).expect("inst in ebb");
 
                 // Insert split values prior to the terminal branch group.
-                let dfg = &pos.func.dfg;
-                let canonical = pos.func.layout.canonical_branch_inst(dfg, inst_ebb);
+                let canonical = pos
+                    .func
+                    .layout
+                    .canonical_branch_inst(&pos.func.dfg, inst_ebb);
                 if let Some(first_branch) = canonical {
                     pos.goto_inst(first_branch);
                 }
diff --git a/cranelift/filetests/filetests/legalizer/isplit-bb.clif b/cranelift/filetests/filetests/legalizer/isplit-bb.clif
new file mode 100644
index 0000000000..10ab41c440
--- /dev/null
+++ b/cranelift/filetests/filetests/legalizer/isplit-bb.clif
@@ -0,0 +1,24 @@
+test legalizer
+target x86_64
+
+function u0:0(i128, i128, i64) -> i128 system_v {
+ebb0(v0: i128, v1: i128, v2: i64):
+    jump ebb1
+
+ebb1:
+    v17 = iadd v0, v1
+    v20 = iadd v1, v17
+    jump ebb79
+
+ebb79:
+    v425 = iconst.i64 0
+    v426 = icmp_imm eq v425, 1
+    brnz v426, ebb80
+    jump ebb85(v20, v17)
+
+ebb80:
+    trap user0
+
+ebb85(v462: i128, v874: i128):
+    trap user0
+}
diff --git a/cranelift/filetests/filetests/verifier/isplit-bb.clif b/cranelift/filetests/filetests/verifier/isplit-bb.clif
deleted file mode 100644
index ba789706a1..0000000000
--- a/cranelift/filetests/filetests/verifier/isplit-bb.clif
+++ /dev/null
@@ -1,28 +0,0 @@
-test compile
-target x86_64
-
-function u0:0(i128, i128, i64) -> i128 system_v {
-    ebb0(v0: i128, v1: i128, v2: i64):
-        trap user0
-
-    ebb1:
-        v10 = iconst.i64 0
-        v11 = iconst.i64 0
-        v17 = iconcat v10, v11
-        v12 = iconst.i64 0
-        v13 = iconst.i64 0 
-        v20 = iconcat v12, v13
-        trap user0
-
-    ebb79:
-        v425 = iconst.i64 0
-        v426 = icmp_imm eq v425, 1
-        brnz v426, ebb80
-        jump ebb85(v20, v17)
-
-    ebb80:
-        trap user0
-
-    ebb85(v462: i128, v874: i128):
-        trap user0
-}

From f41bf5ecca81f72268d68d5aa77270b2fd7ca621 Mon Sep 17 00:00:00 2001
From: Ryan Hunt 
Date: Mon, 6 Jan 2020 15:40:15 -0600
Subject: [PATCH 3025/3084] Wasm: Use environment to translate reference types
 instructions and add support for multiple tables

This commit introduces environment functions to handle the translation of
reference type instructions, analogous to how bulk-memory was implemented.

Additionally, the bulk-memory instructions that operate on tables are extended
to support multiple table indices.
---
 cranelift/wasm/src/code_translator.rs | 65 +++++++++++++++++----------
 cranelift/wasm/src/environ/dummy.rs   | 48 ++++++++++++++++++++
 cranelift/wasm/src/environ/spec.rs    | 39 ++++++++++++++++
 3 files changed, 128 insertions(+), 24 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index e6055cea17..c50b00b39d 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -965,8 +965,8 @@ pub fn translate_operator(
             let val = builder.ins().is_null(arg);
             state.push1(val);
         }
-        Operator::RefFunc { .. } => {
-            return Err(wasm_unsupported!("proposed ref operator {:?}", op))
+        Operator::RefFunc { function_index } => {
+            state.push1(environ.translate_ref_func(builder.cursor(), *function_index)?);
         }
         Operator::AtomicNotify { .. }
         | Operator::I32AtomicWait { .. }
@@ -1089,56 +1089,73 @@ pub fn translate_operator(
                 table,
             )?);
         }
-        Operator::TableCopy { .. } => {
-            // The WebAssembly MVP only supports one table and wasmparser will
-            // ensure that the table index specified is zero.
-            let dst_table_index = 0;
-            let dst_table = state.get_table(builder.func, dst_table_index, environ)?;
-            let src_table_index = 0;
-            let src_table = state.get_table(builder.func, src_table_index, environ)?;
+        Operator::TableGrow { table } => {
+            let delta = state.pop1();
+            let init_value = state.pop1();
+            state.push1(environ.translate_table_grow(
+                builder.cursor(),
+                *table,
+                delta,
+                init_value,
+            )?);
+        }
+        Operator::TableGet { table } => {
+            let index = state.pop1();
+            state.push1(environ.translate_table_get(builder.cursor(), *table, index)?);
+        }
+        Operator::TableSet { table } => {
+            let value = state.pop1();
+            let index = state.pop1();
+            environ.translate_table_set(builder.cursor(), *table, value, index)?;
+        }
+        Operator::TableCopy {
+            dst_table: dst_table_index,
+            src_table: src_table_index,
+        } => {
+            let dst_table = state.get_table(builder.func, *dst_table_index, environ)?;
+            let src_table = state.get_table(builder.func, *src_table_index, environ)?;
             let len = state.pop1();
             let src = state.pop1();
             let dest = state.pop1();
             environ.translate_table_copy(
                 builder.cursor(),
-                TableIndex::from_u32(dst_table_index),
+                TableIndex::from_u32(*dst_table_index),
                 dst_table,
-                TableIndex::from_u32(src_table_index),
+                TableIndex::from_u32(*src_table_index),
                 src_table,
                 dest,
                 src,
                 len,
             )?;
         }
-        Operator::TableInit { segment, table: _ } => {
+        Operator::TableFill { table } => {
+            let len = state.pop1();
+            let val = state.pop1();
+            let dest = state.pop1();
+            environ.translate_table_fill(builder.cursor(), *table, dest, val, len)?;
+        }
+        Operator::TableInit {
+            segment,
+            table: table_index,
+        } => {
             // The WebAssembly MVP only supports one table and we assume it here.
-            let table_index = 0;
-            let table = state.get_table(builder.func, table_index, environ)?;
+            let table = state.get_table(builder.func, *table_index, environ)?;
             let len = state.pop1();
             let src = state.pop1();
             let dest = state.pop1();
             environ.translate_table_init(
                 builder.cursor(),
                 *segment,
-                TableIndex::from_u32(table_index),
+                TableIndex::from_u32(*table_index),
                 table,
                 dest,
                 src,
                 len,
             )?;
         }
-        Operator::TableFill { .. } => {
-            return Err(wasm_unsupported!("proposed table operator {:?}", op));
-        }
         Operator::ElemDrop { segment } => {
             environ.translate_elem_drop(builder.cursor(), *segment)?;
         }
-        Operator::TableGet { .. } | Operator::TableSet { .. } | Operator::TableGrow { .. } => {
-            return Err(wasm_unsupported!(
-                "proposed reference types operator {:?}",
-                op
-            ));
-        }
         Operator::V128Const { value } => {
             let data = value.bytes().to_vec().into();
             let handle = builder.func.dfg.constants.insert(data);
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index 6b6f210166..1fc11cf3e7 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -426,6 +426,35 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         Ok(pos.ins().iconst(I32, -1))
     }
 
+    fn translate_table_grow(
+        &mut self,
+        mut pos: FuncCursor,
+        _table_index: u32,
+        _delta: ir::Value,
+        _init_value: ir::Value,
+    ) -> WasmResult {
+        Ok(pos.ins().iconst(I32, -1))
+    }
+
+    fn translate_table_get(
+        &mut self,
+        mut pos: FuncCursor,
+        _table_index: u32,
+        _index: ir::Value,
+    ) -> WasmResult {
+        Ok(pos.ins().null(self.reference_type()))
+    }
+
+    fn translate_table_set(
+        &mut self,
+        _pos: FuncCursor,
+        _table_index: u32,
+        _value: ir::Value,
+        _index: ir::Value,
+    ) -> WasmResult<()> {
+        Ok(())
+    }
+
     fn translate_table_copy(
         &mut self,
         _pos: FuncCursor,
@@ -440,6 +469,17 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
         Ok(())
     }
 
+    fn translate_table_fill(
+        &mut self,
+        _pos: FuncCursor,
+        _table_index: u32,
+        _dst: ir::Value,
+        _val: ir::Value,
+        _len: ir::Value,
+    ) -> WasmResult<()> {
+        Ok(())
+    }
+
     fn translate_table_init(
         &mut self,
         _pos: FuncCursor,
@@ -456,6 +496,14 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
     fn translate_elem_drop(&mut self, _pos: FuncCursor, _seg_index: u32) -> WasmResult<()> {
         Ok(())
     }
+
+    fn translate_ref_func(
+        &mut self,
+        mut pos: FuncCursor,
+        _func_index: u32,
+    ) -> WasmResult {
+        Ok(pos.ins().null(self.reference_type()))
+    }
 }
 
 impl TargetEnvironment for DummyEnvironment {
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index c6928a491f..dad2c7f50f 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -337,6 +337,32 @@ pub trait FuncEnvironment: TargetEnvironment {
         table: ir::Table,
     ) -> WasmResult;
 
+    /// Translate a `table.grow` WebAssembly instruction.
+    fn translate_table_grow(
+        &mut self,
+        pos: FuncCursor,
+        table_index: u32,
+        delta: ir::Value,
+        init_value: ir::Value,
+    ) -> WasmResult;
+
+    /// Translate a `table.get` WebAssembly instruction.
+    fn translate_table_get(
+        &mut self,
+        pos: FuncCursor,
+        table_index: u32,
+        index: ir::Value,
+    ) -> WasmResult;
+
+    /// Translate a `table.set` WebAssembly instruction.
+    fn translate_table_set(
+        &mut self,
+        pos: FuncCursor,
+        table_index: u32,
+        value: ir::Value,
+        index: ir::Value,
+    ) -> WasmResult<()>;
+
     /// Translate a `table.copy` WebAssembly instruction.
     #[allow(clippy::too_many_arguments)]
     fn translate_table_copy(
@@ -351,6 +377,16 @@ pub trait FuncEnvironment: TargetEnvironment {
         len: ir::Value,
     ) -> WasmResult<()>;
 
+    /// Translate a `table.fill` WebAssembly instruction.
+    fn translate_table_fill(
+        &mut self,
+        pos: FuncCursor,
+        table_index: u32,
+        dst: ir::Value,
+        val: ir::Value,
+        len: ir::Value,
+    ) -> WasmResult<()>;
+
     /// Translate a `table.init` WebAssembly instruction.
     #[allow(clippy::too_many_arguments)]
     fn translate_table_init(
@@ -367,6 +403,9 @@ pub trait FuncEnvironment: TargetEnvironment {
     /// Translate a `elem.drop` WebAssembly instruction.
     fn translate_elem_drop(&mut self, pos: FuncCursor, seg_index: u32) -> WasmResult<()>;
 
+    /// Translate a `ref.func` WebAssembly instruction.
+    fn translate_ref_func(&mut self, pos: FuncCursor, func_index: u32) -> WasmResult;
+
     /// Emit code at the beginning of every wasm loop.
     ///
     /// This can be used to insert explicit interrupt or safepoint checking at

From 41f225804bd881bab570f58895bcbdf3f6b104b1 Mon Sep 17 00:00:00 2001
From: Ryan Hunt 
Date: Mon, 6 Jan 2020 15:41:02 -0600
Subject: [PATCH 3026/3084] Wasm: Allow environment to translate some
 global.set/get operations

Spidermonkey will need to emit pre/post barriers for global.set/get to a
reference type. #1176 and #1299 plan to add a template concept that could
be used to implement this. Once that has been stabilized, we should be able
to remove this code in favor of templates easily.
---
 cranelift/wasm/src/code_translator.rs | 14 +++++++++++++-
 cranelift/wasm/src/environ/dummy.rs   | 17 +++++++++++++++++
 cranelift/wasm/src/environ/spec.rs    | 20 ++++++++++++++++++++
 3 files changed, 50 insertions(+), 1 deletion(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index c50b00b39d..bb92e8e85e 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -28,7 +28,7 @@ use crate::state::{ControlStackFrame, ElseData, FuncTranslationState, ModuleTran
 use crate::translation_utils::{
     blocktype_params_results, ebb_with_params, f32_translation, f64_translation,
 };
-use crate::translation_utils::{FuncIndex, MemoryIndex, SignatureIndex, TableIndex};
+use crate::translation_utils::{FuncIndex, GlobalIndex, MemoryIndex, SignatureIndex, TableIndex};
 use crate::wasm_unsupported;
 use core::{i32, u32};
 use cranelift_codegen::ir::condcodes::{FloatCC, IntCC};
@@ -95,6 +95,10 @@ pub fn translate_operator(
                     let flags = ir::MemFlags::trusted();
                     builder.ins().load(ty, flags, addr, offset)
                 }
+                GlobalVariable::Custom => environ.translate_custom_global_get(
+                    builder.cursor(),
+                    GlobalIndex::from_u32(*global_index),
+                )?,
             };
             state.push1(val);
         }
@@ -108,6 +112,14 @@ pub fn translate_operator(
                     debug_assert_eq!(ty, builder.func.dfg.value_type(val));
                     builder.ins().store(flags, val, addr, offset);
                 }
+                GlobalVariable::Custom => {
+                    let val = state.pop1();
+                    environ.translate_custom_global_set(
+                        builder.cursor(),
+                        GlobalIndex::from_u32(*global_index),
+                        val,
+                    )?;
+                }
             }
         }
         /********************************* Stack misc ***************************************
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index 1fc11cf3e7..a36a0bce70 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -504,6 +504,23 @@ impl<'dummy_environment> FuncEnvironment for DummyFuncEnvironment<'dummy_environ
     ) -> WasmResult {
         Ok(pos.ins().null(self.reference_type()))
     }
+
+    fn translate_custom_global_get(
+        &mut self,
+        mut pos: FuncCursor,
+        _global_index: GlobalIndex,
+    ) -> WasmResult {
+        Ok(pos.ins().iconst(I32, -1))
+    }
+
+    fn translate_custom_global_set(
+        &mut self,
+        _pos: FuncCursor,
+        _global_index: GlobalIndex,
+        _val: ir::Value,
+    ) -> WasmResult<()> {
+        Ok(())
+    }
 }
 
 impl TargetEnvironment for DummyEnvironment {
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index dad2c7f50f..123a8d0c09 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -36,6 +36,9 @@ pub enum GlobalVariable {
         /// The global variable's type.
         ty: ir::Type,
     },
+
+    /// This is a global variable that needs to be handled by the environment.
+    Custom,
 }
 
 /// A WebAssembly translation error.
@@ -406,6 +409,23 @@ pub trait FuncEnvironment: TargetEnvironment {
     /// Translate a `ref.func` WebAssembly instruction.
     fn translate_ref_func(&mut self, pos: FuncCursor, func_index: u32) -> WasmResult;
 
+    /// Translate a `global.get` WebAssembly instruction at `pos` for a global
+    /// that is custom.
+    fn translate_custom_global_get(
+        &mut self,
+        pos: FuncCursor,
+        global_index: GlobalIndex,
+    ) -> WasmResult;
+
+    /// Translate a `global.set` WebAssembly instruction at `pos` for a global
+    /// that is custom.
+    fn translate_custom_global_set(
+        &mut self,
+        pos: FuncCursor,
+        global_index: GlobalIndex,
+        val: ir::Value,
+    ) -> WasmResult<()>;
+
     /// Emit code at the beginning of every wasm loop.
     ///
     /// This can be used to insert explicit interrupt or safepoint checking at

From 3b1dda8e92af79d1895528cc4e24182d81e66814 Mon Sep 17 00:00:00 2001
From: Ryan Hunt 
Date: Wed, 27 Nov 2019 14:18:51 -0600
Subject: [PATCH 3027/3084] Wasm: Ensure result of `ref.is_null` is I32

The (r32|r64).is_null instruction yields a boolean type, so we must
convert a Wasm `ref.is_null` to an integer so we don't get verifier
errors.
---
 cranelift/wasm/src/code_translator.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index bb92e8e85e..baaa335be4 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -975,7 +975,8 @@ pub fn translate_operator(
         Operator::RefIsNull => {
             let arg = state.pop1();
             let val = builder.ins().is_null(arg);
-            state.push1(val);
+            let val_int = builder.ins().bint(I32, val);
+            state.push1(val_int);
         }
         Operator::RefFunc { function_index } => {
             state.push1(environ.translate_ref_func(builder.cursor(), *function_index)?);

From a15bb9cfcbf3f4c992c00b7a8271889bee3d2b5e Mon Sep 17 00:00:00 2001
From: Ryan Hunt 
Date: Wed, 27 Nov 2019 14:16:36 -0600
Subject: [PATCH 3028/3084] Codegen: Use GPR regclass for reference types on
 x86

---
 cranelift/codegen/src/isa/x86/abi.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 4c1925708d..6a1a3cea11 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -333,7 +333,7 @@ pub fn legalize_signature(
 
 /// Get register class for a type appearing in a legalized signature.
 pub fn regclass_for_abi_type(ty: ir::Type) -> RegClass {
-    if ty.is_int() || ty.is_bool() {
+    if ty.is_int() || ty.is_bool() || ty.is_ref() {
         GPR
     } else {
         FPR

From 848baa0aa7486ea2ad1a8541f86c5032fcdf6139 Mon Sep 17 00:00:00 2001
From: Ryan Hunt 
Date: Wed, 27 Nov 2019 14:16:09 -0600
Subject: [PATCH 3029/3084] Codegen: Add ref.is_invalid instruction

Spidermonkey returns a sentinel ref value of '-1' from some VM functions
to indicate failure. This commit adds an instruction analagous to ref.is_null
that checks for this value.
---
 .../codegen/meta/src/isa/x86/encodings.rs     |  5 +++++
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 20 +++++++++++++++++++
 .../codegen/meta/src/shared/instructions.rs   | 18 +++++++++++++++++
 3 files changed, 43 insertions(+)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 8baa109e5c..bd86715c6a 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -2329,10 +2329,12 @@ fn define_reftypes(e: &mut PerCpuModeEncodings, shared_defs: &SharedDefinitions,
     let shared = &shared_defs.instructions;
 
     let is_null = shared.by_name("is_null");
+    let is_invalid = shared.by_name("is_invalid");
     let null = shared.by_name("null");
     let safepoint = shared.by_name("safepoint");
 
     let rec_is_zero = r.template("is_zero");
+    let rec_is_invalid = r.template("is_invalid");
     let rec_pu_id_ref = r.template("pu_id_ref");
     let rec_safepoint = r.recipe("safepoint");
 
@@ -2345,6 +2347,9 @@ fn define_reftypes(e: &mut PerCpuModeEncodings, shared_defs: &SharedDefinitions,
     // is_null, implemented by testing whether the value is 0.
     e.enc_r32_r64_rex_only(is_null, rec_is_zero.opcodes(&TEST_REG));
 
+    // is_invalid, implemented by testing whether the value is -1.
+    e.enc_r32_r64_rex_only(is_invalid, rec_is_invalid.opcodes(&CMP_IMM8).rrr(7));
+
     // safepoint instruction calls sink, no actual encoding.
     e.enc32_rec(safepoint, rec_safepoint, 0);
     e.enc64_rec(safepoint, rec_safepoint, 0);
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index b2ce9e628e..c512e98467 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -3222,6 +3222,26 @@ pub(crate) fn define<'shared>(
             ),
     );
 
+    recipes.add_template_recipe(
+        EncodingRecipeBuilder::new("is_invalid", &formats.unary, 2 + 3)
+            .operands_in(vec![gpr])
+            .operands_out(vec![abcd])
+            .emit(
+                r#"
+                    // Comparison instruction.
+                    {{PUT_OP}}(bits, rex1(in_reg0), sink);
+                    modrm_r_bits(in_reg0, bits, sink);
+                    sink.put1(0xff);
+                    // `setCC` instruction, no REX.
+                    use crate::ir::condcodes::IntCC::*;
+                    let setcc = 0x90 | icc2opc(Equal);
+                    sink.put1(0x0f);
+                    sink.put1(setcc as u8);
+                    modrm_rr(out_reg0, 0, sink);
+                "#,
+            ),
+    );
+
     recipes.add_recipe(
         EncodingRecipeBuilder::new("safepoint", &formats.multiary, 0).emit(
             r#"
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 5f9a525353..0161096f64 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -3190,6 +3190,24 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    let a = &Operand::new("a", b1);
+    let x = &Operand::new("x", Ref);
+
+    ig.push(
+        Inst::new(
+            "is_invalid",
+            r#"
+        Reference verification.
+
+        The condition code determines if the reference type in question is
+        invalid or not.
+        "#,
+            &formats.unary,
+        )
+        .operands_in(vec![x])
+        .operands_out(vec![a]),
+    );
+
     let Cond = &Operand::new("Cond", &imm.intcc);
     let f = &Operand::new("f", iflags);
     let a = &Operand::new("a", b1);

From bbc0a328c74ead5a36a4baede0084e68d5635312 Mon Sep 17 00:00:00 2001
From: Ryan Hunt 
Date: Mon, 6 Jan 2020 15:49:06 -0600
Subject: [PATCH 3030/3084] Codegen: Allow encoding of (r32|r64).(load|store)

Accessing Wasm reference globals that are reference types will
want to use the plain load/store instructions. This commit adds
encodings for these instructions to match loading a i32/i64.
Producers of IR are required to insert the appropriate barriers
around the loads/stores.
---
 cranelift/codegen/meta/src/cdsl/cpu_modes.rs   |  6 ++++++
 .../codegen/meta/src/isa/x86/encodings.rs      | 18 ++++++++++++++++++
 cranelift/codegen/meta/src/isa/x86/mod.rs      |  4 ++++
 .../codegen/meta/src/shared/instructions.rs    |  1 +
 4 files changed, 29 insertions(+)

diff --git a/cranelift/codegen/meta/src/cdsl/cpu_modes.rs b/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
index e62ba72295..7d119b00ce 100644
--- a/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
+++ b/cranelift/codegen/meta/src/cdsl/cpu_modes.rs
@@ -37,6 +37,12 @@ impl CpuMode {
         assert!(self.default_legalize.is_none());
         self.default_legalize = Some(group.id);
     }
+    pub fn legalize_value_type(&mut self, lane_type: impl Into, group: &TransformGroup) {
+        assert!(self
+            .typed_legalize
+            .insert(lane_type.into(), group.id)
+            .is_none());
+    }
     pub fn legalize_type(&mut self, lane_type: impl Into, group: &TransformGroup) {
         assert!(self
             .typed_legalize
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index bd86715c6a..fd160e3e21 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -239,6 +239,22 @@ impl PerCpuModeEncodings {
         self.enc64(inst.bind(R64), template.rex().w());
     }
 
+    fn enc_r32_r64_ld_st(&mut self, inst: &Instruction, w_bit: bool, template: Template) {
+        self.enc32(inst.clone().bind(R32).bind(Any), template.clone());
+
+        // REX-less encoding must come after REX encoding so we don't use it by
+        // default. Otherwise reg-alloc would never use r8 and up.
+        self.enc64(inst.clone().bind(R32).bind(Any), template.clone().rex());
+        self.enc64(inst.clone().bind(R32).bind(Any), template.clone());
+
+        if w_bit {
+            self.enc64(inst.clone().bind(R64).bind(Any), template.rex().w());
+        } else {
+            self.enc64(inst.clone().bind(R64).bind(Any), template.clone().rex());
+            self.enc64(inst.clone().bind(R64).bind(Any), template);
+        }
+    }
+
     /// Add encodings for `inst` to X86_64 with and without a REX prefix.
     fn enc_x86_64(&mut self, inst: impl Into + Clone, template: Template) {
         // See above comment about the ordering of rex vs non-rex encodings.
@@ -858,6 +874,7 @@ fn define_memory(
 
     for recipe in &[rec_st, rec_stDisp8, rec_stDisp32] {
         e.enc_i32_i64_ld_st(store, true, recipe.opcodes(&MOV_STORE));
+        e.enc_r32_r64_ld_st(store, true, recipe.opcodes(&MOV_STORE));
         e.enc_x86_64(istore32.bind(I64).bind(Any), recipe.opcodes(&MOV_STORE));
         e.enc_i32_i64_ld_st(istore16, false, recipe.opcodes(&MOV_STORE_16));
     }
@@ -889,6 +906,7 @@ fn define_memory(
 
     for recipe in &[rec_ld, rec_ldDisp8, rec_ldDisp32] {
         e.enc_i32_i64_ld_st(load, true, recipe.opcodes(&MOV_LOAD));
+        e.enc_r32_r64_ld_st(load, true, recipe.opcodes(&MOV_LOAD));
         e.enc_x86_64(uload32.bind(I64), recipe.opcodes(&MOV_LOAD));
         e.enc64(sload32.bind(I64), recipe.opcodes(&MOVSXD).rex().w());
         e.enc_i32_i64_ld_st(uload16, true, recipe.opcodes(&MOVZX_WORD));
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index 1322368265..ab1b5b4053 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -1,9 +1,11 @@
 use crate::cdsl::cpu_modes::CpuMode;
 use crate::cdsl::isa::TargetIsa;
+use crate::cdsl::types::ReferenceType;
 
 use crate::shared::types::Bool::B1;
 use crate::shared::types::Float::{F32, F64};
 use crate::shared::types::Int::{I16, I32, I64, I8};
+use crate::shared::types::Reference::{R32, R64};
 use crate::shared::Definitions as SharedDefinitions;
 
 mod encodings;
@@ -41,6 +43,7 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     x86_32.legalize_type(I8, widen);
     x86_32.legalize_type(I16, widen);
     x86_32.legalize_type(I32, x86_expand);
+    x86_32.legalize_value_type(ReferenceType(R32), x86_expand);
     x86_32.legalize_type(F32, x86_expand);
     x86_32.legalize_type(F64, x86_expand);
 
@@ -51,6 +54,7 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     x86_64.legalize_type(I16, widen);
     x86_64.legalize_type(I32, x86_expand);
     x86_64.legalize_type(I64, x86_expand);
+    x86_64.legalize_value_type(ReferenceType(R64), x86_expand);
     x86_64.legalize_type(F32, x86_expand);
     x86_64.legalize_type(F64, x86_expand);
 
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 0161096f64..3207dad96e 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -580,6 +580,7 @@ pub(crate) fn define(
             .ints(Interval::All)
             .floats(Interval::All)
             .simd_lanes(Interval::All)
+            .refs(Interval::All)
             .build(),
     );
 

From 946251e6555413b06ba51b31960dd5d6a59f1e62 Mon Sep 17 00:00:00 2001
From: Ryan Hunt 
Date: Mon, 6 Jan 2020 15:48:12 -0600
Subject: [PATCH 3031/3084] Codegen: Align representation of stackmap with
 SpiderMonkey

This commit aligns the representation of stackmaps to be the same
as Spidermonkey's by:
 * Reversing the order of the bitmap from low addresses to high addresses
 * Including incoming stack arguments
 * Excluding outgoing stack arguments

Additionally, some accessor functions were added to allow Spidermonkey
to access the internals of the bitmap.
---
 cranelift/codegen/src/binemit/stackmap.rs | 65 ++++++++++++++++-------
 cranelift/codegen/src/ir/mod.rs           |  2 +-
 cranelift/codegen/src/ir/stackslot.rs     | 31 +++++++----
 cranelift/codegen/src/isa/stack.rs        |  5 +-
 cranelift/codegen/src/stack_layout.rs     | 12 ++++-
 cranelift/filetests/src/test_binemit.rs   |  5 +-
 cranelift/wasm/src/code_translator.rs     |  1 -
 7 files changed, 85 insertions(+), 36 deletions(-)

diff --git a/cranelift/codegen/src/binemit/stackmap.rs b/cranelift/codegen/src/binemit/stackmap.rs
index cb5739670f..10ae96a7cb 100644
--- a/cranelift/codegen/src/binemit/stackmap.rs
+++ b/cranelift/codegen/src/binemit/stackmap.rs
@@ -6,10 +6,19 @@ use alloc::vec::Vec;
 type Num = u32;
 const NUM_BITS: usize = core::mem::size_of::() * 8;
 
-/// Wrapper class for longer bit vectors that cannot be represented by a single BitSet.
+/// A stack map is a bitmap with one bit per machine word on the stack. Stack
+/// maps are created at `safepoint` instructions and record all live reference
+/// values that are on the stack. All slot kinds, except `OutgoingArg` are
+/// captured in a stack map. The `OutgoingArg`'s will be captured in the callee
+/// function as `IncomingArg`'s.
+///
+/// The first value in the bitmap is of the lowest addressed slot on the stack.
+/// As all stacks in Isa's supported by Cranelift grow down, this means that
+/// first value is of the top of the stack and values proceed down the stack.
 #[derive(Clone, Debug)]
 pub struct Stackmap {
     bitmap: Vec>,
+    mapped_words: u32,
 }
 
 impl Stackmap {
@@ -27,32 +36,37 @@ impl Stackmap {
         for val in args {
             if let Some(value_loc) = loc.get(*val) {
                 match *value_loc {
-                    ir::ValueLoc::Stack(stack_slot) => live_ref_in_stack_slot.insert(stack_slot),
-                    _ => false,
-                };
+                    ir::ValueLoc::Stack(stack_slot) => {
+                        live_ref_in_stack_slot.insert(stack_slot);
+                    }
+                    _ => {}
+                }
             }
         }
 
-        // SpiderMonkey stackmap structure:
-        //  +  +  + 
-        // Bit vector goes from lower addresses to higher addresses.
-
-        // TODO: Get trap register layout from Spidermonkey and prepend to bitvector below.
         let stack = &func.stack_slots;
-        let frame_size = stack.frame_size.unwrap();
-        let word_size = ir::stackslot::StackSize::from(isa.pointer_bytes());
-        let num_words = (frame_size / word_size) as usize;
-        let mut vec = alloc::vec::Vec::with_capacity(num_words);
+        let info = func.stack_slots.layout_info.unwrap();
 
+        // Refer to the doc comment for `Stackmap` above to understand the
+        // bitmap representation used here.
+        let map_size = (info.frame_size + info.inbound_args_size) as usize;
+        let word_size = isa.pointer_bytes() as usize;
+        let num_words = map_size / word_size;
+
+        let mut vec = alloc::vec::Vec::with_capacity(num_words);
         vec.resize(num_words, false);
 
-        // Frame (includes spills and inbound args).
         for (ss, ssd) in stack.iter() {
-            if live_ref_in_stack_slot.contains(&ss) {
-                // Assumption: greater magnitude of offset imply higher address.
-                let index = (((ssd.offset.unwrap().abs() as u32) - ssd.size) / word_size) as usize;
-                vec[index] = true;
+            if !live_ref_in_stack_slot.contains(&ss)
+                || ssd.kind == ir::stackslot::StackSlotKind::OutgoingArg
+            {
+                continue;
             }
+
+            debug_assert!(ssd.size as usize == word_size);
+            let bytes_from_bottom = info.frame_size as i32 + ssd.offset.unwrap();
+            let words_from_bottom = (bytes_from_bottom as usize) / word_size;
+            vec[words_from_bottom] = true;
         }
 
         Self::from_slice(&vec)
@@ -73,7 +87,10 @@ impl Stackmap {
             }
             bitmap.push(BitSet(curr_word));
         }
-        Self { bitmap }
+        Self {
+            mapped_words: len as u32,
+            bitmap,
+        }
     }
 
     /// Returns a specified bit.
@@ -83,6 +100,16 @@ impl Stackmap {
         let word_offset = (bit_index % NUM_BITS) as u8;
         self.bitmap[word_index].contains(word_offset)
     }
+
+    /// Returns the raw bitmap that represents this stack map.
+    pub fn as_slice(&self) -> &[BitSet] {
+        &self.bitmap
+    }
+
+    /// Returns the number of words represented by this stack map.
+    pub fn mapped_words(&self) -> u32 {
+        self.mapped_words
+    }
 }
 
 #[cfg(test)]
diff --git a/cranelift/codegen/src/ir/mod.rs b/cranelift/codegen/src/ir/mod.rs
index 88dfb8eb8d..b37812e9d3 100644
--- a/cranelift/codegen/src/ir/mod.rs
+++ b/cranelift/codegen/src/ir/mod.rs
@@ -53,7 +53,7 @@ pub use crate::ir::libcall::{get_libcall_funcref, get_probestack_funcref, LibCal
 pub use crate::ir::memflags::MemFlags;
 pub use crate::ir::progpoint::{ExpandedProgramPoint, ProgramOrder, ProgramPoint};
 pub use crate::ir::sourceloc::SourceLoc;
-pub use crate::ir::stackslot::{StackSlotData, StackSlotKind, StackSlots};
+pub use crate::ir::stackslot::{StackLayoutInfo, StackSlotData, StackSlotKind, StackSlots};
 pub use crate::ir::table::TableData;
 pub use crate::ir::trapcode::TrapCode;
 pub use crate::ir::types::Type;
diff --git a/cranelift/codegen/src/ir/stackslot.rs b/cranelift/codegen/src/ir/stackslot.rs
index 6a4edd0da6..5bb70d1f0e 100644
--- a/cranelift/codegen/src/ir/stackslot.rs
+++ b/cranelift/codegen/src/ir/stackslot.rs
@@ -162,6 +162,23 @@ impl fmt::Display for StackSlotData {
     }
 }
 
+/// Stack frame layout information.
+///
+/// This is computed by the `layout_stack()` method.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
+pub struct StackLayoutInfo {
+    /// The total size of the stack frame.
+    ///
+    /// This is the distance from the stack pointer in the current function to the stack pointer in
+    /// the calling function, so it includes a pushed return address as well as space for outgoing
+    /// call arguments.
+    pub frame_size: StackSize,
+
+    /// The total size of the stack frame for inbound arguments pushed by the caller.
+    pub inbound_args_size: StackSize,
+}
+
 /// Stack frame manager.
 ///
 /// Keep track of all the stack slots used by a function.
@@ -177,14 +194,8 @@ pub struct StackSlots {
     /// All the emergency slots.
     emergency: Vec,
 
-    /// The total size of the stack frame.
-    ///
-    /// This is the distance from the stack pointer in the current function to the stack pointer in
-    /// the calling function, so it includes a pushed return address as well as space for outgoing
-    /// call arguments.
-    ///
-    /// This is computed by the `layout()` method.
-    pub frame_size: Option,
+    /// Layout information computed from `layout_stack`.
+    pub layout_info: Option,
 }
 
 /// Stack slot manager functions that behave mostly like an entity map.
@@ -195,7 +206,7 @@ impl StackSlots {
             slots: PrimaryMap::new(),
             outgoing: Vec::new(),
             emergency: Vec::new(),
-            frame_size: None,
+            layout_info: None,
         }
     }
 
@@ -204,7 +215,7 @@ impl StackSlots {
         self.slots.clear();
         self.outgoing.clear();
         self.emergency.clear();
-        self.frame_size = None;
+        self.layout_info = None;
     }
 
     /// Allocate a new stack slot.
diff --git a/cranelift/codegen/src/isa/stack.rs b/cranelift/codegen/src/isa/stack.rs
index 852aedddd8..ae093bed28 100644
--- a/cranelift/codegen/src/isa/stack.rs
+++ b/cranelift/codegen/src/isa/stack.rs
@@ -36,8 +36,9 @@ impl StackRef {
     /// Get a reference to `ss` using the stack pointer as a base.
     pub fn sp(ss: StackSlot, frame: &StackSlots) -> Self {
         let size = frame
-            .frame_size
-            .expect("Stack layout must be computed before referencing stack slots");
+            .layout_info
+            .expect("Stack layout must be computed before referencing stack slots")
+            .frame_size;
         let slot = &frame[ss];
         let offset = if slot.kind == StackSlotKind::OutgoingArg {
             // Outgoing argument slots have offsets relative to our stack pointer.
diff --git a/cranelift/codegen/src/stack_layout.rs b/cranelift/codegen/src/stack_layout.rs
index c335b844af..d04137ff10 100644
--- a/cranelift/codegen/src/stack_layout.rs
+++ b/cranelift/codegen/src/stack_layout.rs
@@ -1,7 +1,7 @@
 //! Computing stack layout.
 
 use crate::ir::stackslot::{StackOffset, StackSize, StackSlotKind};
-use crate::ir::StackSlots;
+use crate::ir::{StackLayoutInfo, StackSlots};
 use crate::result::{CodegenError, CodegenResult};
 use core::cmp::{max, min};
 
@@ -44,6 +44,7 @@ pub fn layout_stack(
     // require the stack to be aligned.
 
     let mut incoming_min = 0;
+    let mut incoming_max = 0;
     let mut outgoing_max = 0;
     let mut min_align = alignment;
     let mut must_align = is_leaf;
@@ -56,6 +57,7 @@ pub fn layout_stack(
         match slot.kind {
             StackSlotKind::IncomingArg => {
                 incoming_min = min(incoming_min, slot.offset.unwrap());
+                incoming_max = max(incoming_max, slot.offset.unwrap() + slot.size as i32);
             }
             StackSlotKind::OutgoingArg => {
                 let offset = slot
@@ -119,8 +121,14 @@ pub fn layout_stack(
         offset &= -(alignment as StackOffset);
     }
 
+    // Set the computed layout information for the frame
     let frame_size = (offset as StackSize).wrapping_neg();
-    frame.frame_size = Some(frame_size);
+    let inbound_args_size = incoming_max as u32;
+    frame.layout_info = Some(StackLayoutInfo {
+        frame_size,
+        inbound_args_size,
+    });
+
     Ok(frame_size)
 }
 
diff --git a/cranelift/filetests/src/test_binemit.rs b/cranelift/filetests/src/test_binemit.rs
index 200251bfa4..1a6bd60e99 100644
--- a/cranelift/filetests/src/test_binemit.rs
+++ b/cranelift/filetests/src/test_binemit.rs
@@ -142,7 +142,10 @@ impl SubTest for TestBinEmit {
             .values()
             .map(|slot| slot.offset.unwrap())
             .min();
-        func.stack_slots.frame_size = min_offset.map(|off| (-off) as u32);
+        func.stack_slots.layout_info = min_offset.map(|off| ir::StackLayoutInfo {
+            frame_size: (-off) as u32,
+            inbound_args_size: 0,
+        });
 
         let opt_level = isa.flags().opt_level();
 
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index baaa335be4..58da8536ae 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1151,7 +1151,6 @@ pub fn translate_operator(
             segment,
             table: table_index,
         } => {
-            // The WebAssembly MVP only supports one table and we assume it here.
             let table = state.get_table(builder.func, *table_index, environ)?;
             let len = state.pop1();
             let src = state.pop1();

From fc58dd6aff5844b96b91a55e846eb2dd0448da7e Mon Sep 17 00:00:00 2001
From: Ryan Hunt 
Date: Thu, 23 Jan 2020 09:30:28 -0600
Subject: [PATCH 3032/3084] Tests: Add basic test for r32, r64

This commit adds a basic test for reference types on 32/64bit systems.
 * Storing a ref type in a table
 * Loading a ref type from a table
 * Passing ref types to a function
 * Returning ref types from a function
 * `is_null` instruction
 * `is_invalid` instruction
---
 cranelift/filetests/filetests/wasm/r32.clif | 75 +++++++++++++++++++++
 cranelift/filetests/filetests/wasm/r64.clif | 75 +++++++++++++++++++++
 2 files changed, 150 insertions(+)
 create mode 100644 cranelift/filetests/filetests/wasm/r32.clif
 create mode 100644 cranelift/filetests/filetests/wasm/r64.clif

diff --git a/cranelift/filetests/filetests/wasm/r32.clif b/cranelift/filetests/filetests/wasm/r32.clif
new file mode 100644
index 0000000000..a60dd4d29b
--- /dev/null
+++ b/cranelift/filetests/filetests/wasm/r32.clif
@@ -0,0 +1,75 @@
+; Test basic code generation for 32-bit reftypes
+; This test is the 32-bit version of r64.clif. If you change this test you
+; should most likely update that test as well.
+test compile
+set enable_safepoints=true
+
+target i686 haswell
+
+function %select_ref(i32, r32, r32) -> r32 {
+ebb0(v0: i32, v1: r32, v2: r32):
+    brz v0, ebb1(v2)
+    jump ebb1(v1)
+
+ebb1(v3: r32):
+    return v3
+}
+
+function %table_set(i32, r32, i32 vmctx) {
+    gv0 = vmctx
+    gv1 = load.i32 notrap aligned gv0
+    gv2 = load.i32 notrap aligned gv0 +4
+    table0 = dynamic gv1, element_size 1, bound gv2, index_type i32
+
+ebb0(v0: i32, v1: r32, v2: i32):
+    v3 = table_addr.i32 table0, v0, +0;
+    store.r32 notrap aligned v1, v3
+    return
+}
+
+function %table_get(i32, i32 vmctx) -> r32 {
+    gv0 = vmctx
+    gv1 = load.i32 notrap aligned gv0
+    gv2 = load.i32 notrap aligned gv0 +4
+    table0 = dynamic gv1, element_size 1, bound gv2, index_type i32
+
+ebb0(v0: i32, v1: i32):
+    v2 = table_addr.i32 table0, v0, +0;
+    v3 = load.r32 notrap aligned v2
+    return v3
+}
+
+function %test_refs(r32, r32, r32, i32 vmctx) {
+    fn0 = %select_ref(i32, r32, r32) -> r32
+    fn1 = %table_set(i32, r32, i32 vmctx)
+    fn2 = %table_get(i32, i32 vmctx) -> r32
+
+ebb0(v0: r32, v1: r32, v2: r32, v3: i32):
+    v4 = iconst.i32 0
+    v5 = iconst.i32 1
+    v8 = iconst.i32 2
+
+    ; Shuffle around the first two refs
+    v6 = call fn0(v4, v0, v1)
+    v7 = call fn0(v5, v0, v1)
+
+    ; Store in the table
+    call fn1(v4, v6, v3)
+    call fn1(v5, v7, v3)
+    call fn1(v8, v2, v3)
+
+    ; Load from the table
+    v9 = call fn2(v4, v3)
+    v10 = call fn2(v5, v3)
+    v11 = call fn2(v8, v3)
+
+    ; Compare the results
+    v12 = is_null v9
+    trapnz v12, user0
+    v13 = is_null v10
+    trapnz v13, user0
+    v14 = is_invalid v11
+    trapnz v14, user0
+
+    return
+}
diff --git a/cranelift/filetests/filetests/wasm/r64.clif b/cranelift/filetests/filetests/wasm/r64.clif
new file mode 100644
index 0000000000..5637028103
--- /dev/null
+++ b/cranelift/filetests/filetests/wasm/r64.clif
@@ -0,0 +1,75 @@
+; Test basic code generation for 64-bit reftypes
+; This test is the 64-bit version of r32.clif. If you change this test you
+; should most likely update that test as well.
+test compile
+set enable_safepoints=true
+
+target x86_64 haswell
+
+function %select_ref(i32, r64, r64) -> r64 {
+ebb0(v0: i32, v1: r64, v2: r64):
+    brz v0, ebb1(v2)
+    jump ebb1(v1)
+
+ebb1(v3: r64):
+    return v3
+}
+
+function %table_set(i32, r64, i64 vmctx) {
+    gv0 = vmctx
+    gv1 = load.i64 notrap aligned gv0
+    gv2 = load.i32 notrap aligned gv0 +8
+    table0 = dynamic gv1, element_size 1, bound gv2, index_type i32
+
+ebb0(v0: i32, v1: r64, v2: i64):
+    v3 = table_addr.i64 table0, v0, +0;
+    store.r64 notrap aligned v1, v3
+    return
+}
+
+function %table_get(i32, i64 vmctx) -> r64 {
+    gv0 = vmctx
+    gv1 = load.i64 notrap aligned gv0
+    gv2 = load.i32 notrap aligned gv0 +8
+    table0 = dynamic gv1, element_size 1, bound gv2, index_type i32
+
+ebb0(v0: i32, v1: i64):
+    v2 = table_addr.i64 table0, v0, +0;
+    v3 = load.r64 notrap aligned v2
+    return v3
+}
+
+function %test_refs(r64, r64, r64, i64 vmctx) {
+    fn0 = %select_ref(i32, r64, r64) -> r64
+    fn1 = %table_set(i32, r64, i64 vmctx)
+    fn2 = %table_get(i32, i64 vmctx) -> r64
+
+ebb0(v0: r64, v1: r64, v2: r64, v3: i64):
+    v4 = iconst.i32 0
+    v5 = iconst.i32 1
+    v8 = iconst.i32 2
+
+    ; Shuffle around the first two refs
+    v6 = call fn0(v4, v0, v1)
+    v7 = call fn0(v5, v0, v1)
+
+    ; Store in the table
+    call fn1(v4, v6, v3)
+    call fn1(v5, v7, v3)
+    call fn1(v8, v2, v3)
+
+    ; Load from the table
+    v9 = call fn2(v4, v3)
+    v10 = call fn2(v5, v3)
+    v11 = call fn2(v8, v3)
+
+    ; Compare the results
+    v12 = is_null v9
+    trapnz v12, user0
+    v13 = is_null v10
+    trapnz v13, user0
+    v14 = is_invalid v11
+    trapnz v14, user0
+
+    return
+}

From 32a95a89eba6138cdf84030c19c205816f1d8c0a Mon Sep 17 00:00:00 2001
From: Ryan Hunt 
Date: Mon, 6 Jan 2020 15:36:06 -0600
Subject: [PATCH 3033/3084] Wasm: Add support for typed select instruction

Reference types introduces a typed select operation. It has identical execution
semantics so no codegen change is needed.
---
 cranelift/wasm/src/code_translator.rs | 10 +++++++---
 cranelift/wasmtests/select.wat        | 19 +++++++++++++++++++
 2 files changed, 26 insertions(+), 3 deletions(-)
 create mode 100644 cranelift/wasmtests/select.wat

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 58da8536ae..dcbfe8eeb9 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -132,6 +132,13 @@ pub fn translate_operator(
             let (arg1, arg2, cond) = state.pop3();
             state.push1(builder.ins().select(cond, arg1, arg2));
         }
+        Operator::TypedSelect { ty: _ } => {
+            // We ignore the explicit type parameter as it is only needed for
+            // validation, which we require to have been performed before
+            // translation.
+            let (arg1, arg2, cond) = state.pop3();
+            state.push1(builder.ins().select(cond, arg1, arg2));
+        }
         Operator::Nop => {
             // We do nothing
         }
@@ -968,9 +975,6 @@ pub fn translate_operator(
         Operator::F32Le | Operator::F64Le => {
             translate_fcmp(FloatCC::LessThanOrEqual, builder, state)
         }
-        Operator::TypedSelect { .. } => {
-            return Err(wasm_unsupported!("proposed typed select operator {:?}", op))
-        }
         Operator::RefNull => state.push1(builder.ins().null(environ.reference_type())),
         Operator::RefIsNull => {
             let arg = state.pop1();
diff --git a/cranelift/wasmtests/select.wat b/cranelift/wasmtests/select.wat
new file mode 100644
index 0000000000..45ef241833
--- /dev/null
+++ b/cranelift/wasmtests/select.wat
@@ -0,0 +1,19 @@
+(module
+  (func $untyped-select (result i32)
+  	i32.const 42
+  	i32.const 24
+  	i32.const 1
+  	select)
+
+  (func $typed-select-1 (result anyref)
+  	ref.null
+  	ref.null
+  	i32.const 1
+  	select (result anyref))
+
+  (func $typed-select-2 (param anyref) (result anyref)
+    ref.null
+    local.get 0
+    i32.const 1
+    select (result anyref))
+)

From 710182ad26ce0155b71f20cbb8dc6ade45919802 Mon Sep 17 00:00:00 2001
From: Ryan Hunt 
Date: Mon, 6 Jan 2020 15:36:46 -0600
Subject: [PATCH 3034/3084] Wasm: Add support for nullref type

Reference types now supports proper a nullref type. This commit changes
the Wasm translator to support this.
---
 cranelift/wasm/src/func_translator.rs   |  1 +
 cranelift/wasm/src/translation_utils.rs |  7 +++++--
 cranelift/wasmtests/nullref.wat         | 11 +++++++++++
 3 files changed, 17 insertions(+), 2 deletions(-)
 create mode 100644 cranelift/wasmtests/nullref.wat

diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index a7dbf0820a..cc61b28c9b 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -196,6 +196,7 @@ fn declare_locals(
             let constant_handle = builder.func.dfg.constants.insert([0; 16].to_vec().into());
             builder.ins().vconst(ir::types::I8X16, constant_handle)
         }
+        NullRef => builder.ins().null(environ.reference_type()),
         AnyRef => builder.ins().null(environ.reference_type()),
         AnyFunc => builder.ins().null(environ.reference_type()),
         ty => return Err(wasm_unsupported!("unsupported local type {:?}", ty)),
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index 9d38d9dbb6..d431c13f29 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -131,7 +131,9 @@ pub fn type_to_type(
         wasmparser::Type::F32 => Ok(ir::types::F32),
         wasmparser::Type::F64 => Ok(ir::types::F64),
         wasmparser::Type::V128 => Ok(ir::types::I8X16),
-        wasmparser::Type::AnyRef | wasmparser::Type::AnyFunc => Ok(environ.reference_type()),
+        wasmparser::Type::AnyRef | wasmparser::Type::AnyFunc | wasmparser::Type::NullRef => {
+            Ok(environ.reference_type())
+        }
         ty => Err(wasm_unsupported!("type_to_type: wasm type {:?}", ty)),
     }
 }
@@ -171,6 +173,7 @@ pub fn blocktype_params_results(
             wasmparser::Type::V128 => (&[], &[wasmparser::Type::V128]),
             wasmparser::Type::AnyRef => (&[], &[wasmparser::Type::AnyRef]),
             wasmparser::Type::AnyFunc => (&[], &[wasmparser::Type::AnyFunc]),
+            wasmparser::Type::NullRef => (&[], &[wasmparser::Type::NullRef]),
             wasmparser::Type::EmptyBlockType => (&[], &[]),
             ty => return Err(wasm_unsupported!("blocktype_params_results: type {:?}", ty)),
         },
@@ -203,7 +206,7 @@ pub fn ebb_with_params(
             wasmparser::Type::F64 => {
                 builder.append_ebb_param(ebb, ir::types::F64);
             }
-            wasmparser::Type::AnyRef | wasmparser::Type::AnyFunc => {
+            wasmparser::Type::AnyRef | wasmparser::Type::AnyFunc | wasmparser::Type::NullRef => {
                 builder.append_ebb_param(ebb, environ.reference_type());
             }
             wasmparser::Type::V128 => {
diff --git a/cranelift/wasmtests/nullref.wat b/cranelift/wasmtests/nullref.wat
new file mode 100644
index 0000000000..86714a2b6e
--- /dev/null
+++ b/cranelift/wasmtests/nullref.wat
@@ -0,0 +1,11 @@
+(module
+	(func (result nullref)
+		ref.null
+	)
+
+	(func (result nullref)
+		(block (result nullref)
+			ref.null
+		)
+	)
+)

From c360007b19bab82049a9142bbe62601968d03d89 Mon Sep 17 00:00:00 2001
From: Ryan Hunt 
Date: Thu, 23 Jan 2020 23:36:06 -0600
Subject: [PATCH 3035/3084] Drop 'basic-blocks' feature (#1363)

* All: Drop 'basic-blocks' feature

This makes it so that 'basic-blocks' cannot be disabled and we can
start assuming it everywhere.

* Tests: Replace non-bb filetests with bb version

* Tests: Adapt solver-fixedconflict filetests to use basic blocks
---
 cranelift/Cargo.toml                          |   4 +-
 cranelift/codegen/Cargo.toml                  |   5 +-
 cranelift/codegen/src/binemit/relaxation.rs   |   8 +-
 cranelift/codegen/src/cursor.rs               |   2 -
 cranelift/codegen/src/ir/function.rs          |   6 +-
 cranelift/codegen/src/legalizer/split.rs      |  21 ++-
 .../codegen/src/regalloc/branch_splitting.rs  |   2 -
 cranelift/codegen/src/regalloc/coloring.rs    |   2 +-
 cranelift/codegen/src/regalloc/context.rs     |   6 +-
 cranelift/codegen/src/verifier/mod.rs         |   2 -
 cranelift/filetests/Cargo.toml                |   3 -
 .../isa/x86/legalize-br-table-bb.clif         |  32 ----
 .../filetests/isa/x86/legalize-br-table.clif  |   4 +-
 .../filetests/regalloc/coalesce-bb.clif       | 158 ------------------
 .../filetests/regalloc/coalesce.clif          |  39 +++--
 .../filetests/regalloc/reload-208-bb.clif     | 113 -------------
 .../filetests/regalloc/reload-208.clif        |   9 +-
 .../regalloc/solver-fixedconflict-var-3.clif  |  35 ++--
 .../regalloc/solver-fixedconflict-var.clif    |  33 ++--
 .../filetests/regalloc/x86-regres-bb.clif     |  50 ------
 .../filetests/regalloc/x86-regres.clif        |   9 +-
 .../filetests/safepoint/basic-bb.clif         |  72 --------
 .../filetests/filetests/safepoint/basic.clif  |  42 +++--
 cranelift/filetests/src/runone.rs             |  25 ---
 cranelift/frontend/Cargo.toml                 |   5 +-
 cranelift/frontend/src/frontend.rs            |   1 -
 cranelift/src/bugpoint.rs                     |  23 ++-
 cranelift/wasm/Cargo.toml                     |   5 +-
 cranelift/wasm/src/code_translator.rs         |  22 +--
 29 files changed, 139 insertions(+), 599 deletions(-)
 delete mode 100644 cranelift/filetests/filetests/isa/x86/legalize-br-table-bb.clif
 delete mode 100644 cranelift/filetests/filetests/regalloc/coalesce-bb.clif
 delete mode 100644 cranelift/filetests/filetests/regalloc/reload-208-bb.clif
 delete mode 100644 cranelift/filetests/filetests/regalloc/x86-regres-bb.clif
 delete mode 100644 cranelift/filetests/filetests/safepoint/basic-bb.clif

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index f064d470b1..30a0c434be 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -46,11 +46,9 @@ indicatif = "0.13.0"
 walkdir = "2.2"
 
 [features]
-default = ["disas", "wasm", "cranelift-codegen/all-arch", "basic-blocks"]
+default = ["disas", "wasm", "cranelift-codegen/all-arch"]
 disas = ["capstone"]
 wasm = ["wat", "cranelift-wasm"]
-basic-blocks = ["cranelift-codegen/basic-blocks", "cranelift-frontend/basic-blocks",
-"cranelift-wasm/basic-blocks", "cranelift-filetests/basic-blocks"]
 
 # We want debug symbols on release binaries by default since it allows profiling
 # tools to give more accurate information. We can always strip them out later if
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index e135b166a2..6d5c571261 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -33,7 +33,7 @@ byteorder = { version = "1.3.2", default-features = false }
 cranelift-codegen-meta = { path = "meta", version = "0.56.0" }
 
 [features]
-default = ["std", "basic-blocks", "unwind"]
+default = ["std", "unwind"]
 
 # The "std" feature enables use of libstd. The "core" feature enables use
 # of some minimal std-like replacement libraries. At least one of these two
@@ -69,9 +69,6 @@ all-arch = [
 # For dependent crates that want to serialize some parts of cranelift
 enable-serde = ["serde"]
 
-# Temporary feature that enforces basic block semantics.
-basic-blocks = []
-
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index 20e955cf99..3cd8b68de0 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -31,7 +31,7 @@ use crate::binemit::{CodeInfo, CodeOffset};
 use crate::cursor::{Cursor, FuncCursor};
 use crate::dominator_tree::DominatorTree;
 use crate::flowgraph::ControlFlowGraph;
-use crate::ir::{Function, InstructionData, Opcode};
+use crate::ir::{Ebb, Function, Inst, InstructionData, Opcode, Value, ValueList};
 use crate::isa::{EncInfo, TargetIsa};
 use crate::iterators::IteratorExtras;
 use crate::regalloc::RegDiversions;
@@ -40,9 +40,6 @@ use crate::CodegenResult;
 use core::convert::TryFrom;
 use log::debug;
 
-#[cfg(feature = "basic-blocks")]
-use crate::ir::{Ebb, Inst, Value, ValueList};
-
 /// Relax branches and compute the final layout of EBB headers in `func`.
 ///
 /// Fill in the `func.offsets` table so the function is ready for binary emission.
@@ -61,7 +58,6 @@ pub fn relax_branches(
     func.offsets.resize(func.dfg.num_ebbs());
 
     // Start by removing redundant jumps.
-    #[cfg(feature = "basic-blocks")]
     fold_redundant_jumps(func, _cfg, _domtree);
 
     // Convert jumps to fallthrough instructions where possible.
@@ -154,7 +150,6 @@ pub fn relax_branches(
 
 /// Folds an instruction if it is a redundant jump.
 /// Returns whether folding was performed (which invalidates the CFG).
-#[cfg(feature = "basic-blocks")]
 fn try_fold_redundant_jump(
     func: &mut Function,
     cfg: &mut ControlFlowGraph,
@@ -260,7 +255,6 @@ fn try_fold_redundant_jump(
 
 /// Redirects `jump` instructions that point to other `jump` instructions to the final destination.
 /// This transformation may orphan some blocks.
-#[cfg(feature = "basic-blocks")]
 fn fold_redundant_jumps(
     func: &mut Function,
     cfg: &mut ControlFlowGraph,
diff --git a/cranelift/codegen/src/cursor.rs b/cranelift/codegen/src/cursor.rs
index 9af12133d7..e2ab8e5e5f 100644
--- a/cranelift/codegen/src/cursor.rs
+++ b/cranelift/codegen/src/cursor.rs
@@ -636,7 +636,6 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut FuncCursor<'f> {
 
     fn insert_built_inst(self, inst: ir::Inst, _: ir::Type) -> &'c mut ir::DataFlowGraph {
         // TODO: Remove this assertion once #796 is fixed.
-        #[cfg(feature = "basic-blocks")]
         #[cfg(debug_assertions)]
         {
             if let CursorPosition::At(_) = self.position() {
@@ -766,7 +765,6 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut EncCursor<'f> {
         ctrl_typevar: ir::Type,
     ) -> &'c mut ir::DataFlowGraph {
         // TODO: Remove this assertion once #796 is fixed.
-        #[cfg(feature = "basic-blocks")]
         #[cfg(debug_assertions)]
         {
             if let CursorPosition::At(_) = self.position() {
diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index 10536cd234..d3ff59690a 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -9,7 +9,7 @@ use crate::ir;
 use crate::ir::{DataFlowGraph, ExternalName, Layout, Signature};
 use crate::ir::{
     Ebb, ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, Inst, JumpTable,
-    JumpTableData, SigRef, StackSlot, StackSlotData, Table, TableData,
+    JumpTableData, Opcode, SigRef, StackSlot, StackSlotData, Table, TableData,
 };
 use crate::ir::{EbbOffsets, FrameLayout, InstEncodings, SourceLocs, StackSlots, ValueLocations};
 use crate::ir::{JumpTableOffsets, JumpTables};
@@ -19,9 +19,6 @@ use crate::value_label::ValueLabelsRanges;
 use crate::write::write_function;
 use core::fmt;
 
-#[cfg(feature = "basic-blocks")]
-use crate::ir::Opcode;
-
 /// A function.
 ///
 /// Functions can be cloned, but it is not a very fast operation.
@@ -273,7 +270,6 @@ impl Function {
     /// Checks that the specified EBB can be encoded as a basic block.
     ///
     /// On error, returns the first invalid instruction and an error message.
-    #[cfg(feature = "basic-blocks")]
     pub fn is_ebb_basic(&self, ebb: Ebb) -> Result<(), (Inst, &'static str)> {
         let dfg = &self.dfg;
         let inst_iter = self.layout.ebb_insts(ebb);
diff --git a/cranelift/codegen/src/legalizer/split.rs b/cranelift/codegen/src/legalizer/split.rs
index 727c766d66..62f89b3975 100644
--- a/cranelift/codegen/src/legalizer/split.rs
+++ b/cranelift/codegen/src/legalizer/split.rs
@@ -189,19 +189,18 @@ fn perform_repairs(pos: &mut FuncCursor, cfg: &ControlFlowGraph, mut repairs: Ve
 
             // Split the old argument, possibly causing more repairs to be scheduled.
             pos.goto_inst(inst);
-            #[cfg(feature = "basic-blocks")]
-            {
-                let inst_ebb = pos.func.layout.inst_ebb(inst).expect("inst in ebb");
 
-                // Insert split values prior to the terminal branch group.
-                let canonical = pos
-                    .func
-                    .layout
-                    .canonical_branch_inst(&pos.func.dfg, inst_ebb);
-                if let Some(first_branch) = canonical {
-                    pos.goto_inst(first_branch);
-                }
+            let inst_ebb = pos.func.layout.inst_ebb(inst).expect("inst in ebb");
+
+            // Insert split values prior to the terminal branch group.
+            let canonical = pos
+                .func
+                .layout
+                .canonical_branch_inst(&pos.func.dfg, inst_ebb);
+            if let Some(first_branch) = canonical {
+                pos.goto_inst(first_branch);
             }
+
             let (lo, hi) = split_value(pos, old_arg, repair.concat, &mut repairs);
 
             // The `lo` part replaces the original argument.
diff --git a/cranelift/codegen/src/regalloc/branch_splitting.rs b/cranelift/codegen/src/regalloc/branch_splitting.rs
index 39901a5588..35291e5213 100644
--- a/cranelift/codegen/src/regalloc/branch_splitting.rs
+++ b/cranelift/codegen/src/regalloc/branch_splitting.rs
@@ -2,8 +2,6 @@
 //!
 //! One of the reason for splitting edges is to be able to insert `copy` and `regmove` instructions
 //! between a conditional branch and the following terminator.
-#![cfg(feature = "basic-blocks")]
-
 use alloc::vec::Vec;
 
 use crate::cursor::{Cursor, EncCursor};
diff --git a/cranelift/codegen/src/regalloc/coloring.rs b/cranelift/codegen/src/regalloc/coloring.rs
index cef27214ed..347aec9ade 100644
--- a/cranelift/codegen/src/regalloc/coloring.rs
+++ b/cranelift/codegen/src/regalloc/coloring.rs
@@ -206,7 +206,7 @@ impl<'a> Context<'a> {
             // We are not able to insert any regmove for diversion or un-diversion after the first
             // branch. Instead, we record the diversion to be restored at the entry of the next EBB,
             // which should have a single predecessor.
-            if opcode.is_branch() && cfg!(feature = "basic-blocks") {
+            if opcode.is_branch() {
                 // The next instruction is necessarily an unconditional branch.
                 if let Some(branch) = self.cur.next_inst() {
                     debug!(
diff --git a/cranelift/codegen/src/regalloc/context.rs b/cranelift/codegen/src/regalloc/context.rs
index c5f5da4ecb..dfbec985eb 100644
--- a/cranelift/codegen/src/regalloc/context.rs
+++ b/cranelift/codegen/src/regalloc/context.rs
@@ -8,7 +8,6 @@ use crate::dominator_tree::DominatorTree;
 use crate::flowgraph::ControlFlowGraph;
 use crate::ir::Function;
 use crate::isa::TargetIsa;
-#[cfg(feature = "basic-blocks")]
 use crate::regalloc::branch_splitting;
 use crate::regalloc::coalescing::Coalescing;
 use crate::regalloc::coloring::Coloring;
@@ -96,10 +95,7 @@ impl Context {
         self.tracker.clear();
 
         // Pass: Split branches, add space where to add copy & regmove instructions.
-        #[cfg(feature = "basic-blocks")]
-        {
-            branch_splitting::run(isa, func, cfg, domtree, &mut self.topo);
-        }
+        branch_splitting::run(isa, func, cfg, domtree, &mut self.topo);
 
         // Pass: Liveness analysis.
         self.liveness.compute(isa, func, cfg);
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 6d97d9af4e..9833a8c0f5 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -504,7 +504,6 @@ impl<'a> Verifier<'a> {
 
     /// Check that the given EBB can be encoded as a BB, by checking that only
     /// branching instructions are ending the EBB.
-    #[cfg(feature = "basic-blocks")]
     fn encodable_as_bb(&self, ebb: Ebb, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         match self.func.is_ebb_basic(ebb) {
             Ok(()) => Ok(()),
@@ -1983,7 +1982,6 @@ impl<'a> Verifier<'a> {
                 self.immediate_constraints(inst, errors)?;
             }
 
-            #[cfg(feature = "basic-blocks")]
             self.encodable_as_bb(ebb, errors)?;
         }
 
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index c5ce847000..04d4db4bb3 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -22,6 +22,3 @@ memmap = "0.7.0"
 num_cpus = "1.8.0"
 region = "2.1.2"
 byteorder = { version = "1.3.2", default-features = false }
-
-[features]
-basic-blocks = []
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-table-bb.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-table-bb.clif
deleted file mode 100644
index 78d66d43af..0000000000
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-table-bb.clif
+++ /dev/null
@@ -1,32 +0,0 @@
-test compile
-set opt_level=speed_and_size
-target x86_64
-feature "basic-blocks"
-; regex: V=v\d+
-; regex: EBB=ebb\d+
-
-function u0:0(i64) system_v {
-    ss0 = explicit_slot 1
-    jt0 = jump_table [ebb1]
-
-ebb0(v0: i64):
-    v1 = stack_addr.i64 ss0
-    v2 = load.i8 v1
-    br_table v2, ebb2, jt0
-; check:     $(oob=$V) = ifcmp_imm $(idx=$V), 1
-; ebb2 is replaced by ebb1 by fold_redundant_jump
-; nextln:    brif uge $oob, ebb1
-; nextln:    fallthrough $(inb=$EBB)
-; check:   $inb:
-; nextln:    $(final_idx=$V) = uextend.i64 $idx
-; nextln:    $(base=$V) = jump_table_base.i64 jt0
-; nextln:    $(rel_addr=$V) = jump_table_entry $final_idx, $base, 4, jt0
-; nextln:    $(addr=$V) = iadd $base, $rel_addr
-; nextln:    indirect_jump_table_br $addr, jt0
-
-ebb2:
-    jump ebb1
-
-ebb1:
-    return
-}
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
index 51ab2d08d5..e9464e6219 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
@@ -1,7 +1,6 @@
 test compile
 set opt_level=speed_and_size
 target x86_64
-feature !"basic-blocks"
 ; regex: V=v\d+
 ; regex: EBB=ebb\d+
 
@@ -14,7 +13,8 @@ ebb0(v0: i64):
     v2 = load.i8 v1
     br_table v2, ebb2, jt0
 ; check:     $(oob=$V) = ifcmp_imm $(idx=$V), 1
-; nextln:    brif uge $oob, ebb2
+; ebb2 is replaced by ebb1 by fold_redundant_jump
+; nextln:    brif uge $oob, ebb1
 ; nextln:    fallthrough $(inb=$EBB)
 ; check:   $inb:
 ; nextln:    $(final_idx=$V) = uextend.i64 $idx
diff --git a/cranelift/filetests/filetests/regalloc/coalesce-bb.clif b/cranelift/filetests/filetests/regalloc/coalesce-bb.clif
deleted file mode 100644
index 7718df7d17..0000000000
--- a/cranelift/filetests/filetests/regalloc/coalesce-bb.clif
+++ /dev/null
@@ -1,158 +0,0 @@
-test regalloc
-target riscv32
-feature "basic-blocks"
-
-; Test the coalescer.
-; regex: V=v\d+
-; regex: WS=\s+
-; regex: LOC=%\w+
-; regex: EBB=ebb\d+
-
-; This function is already CSSA, so no copies should be inserted.
-function %cssa(i32) -> i32 {
-ebb0(v0: i32):
-    ; not: copy
-    ; v0 is used by the branch and passed as an arg - that's no conflict.
-    brnz v0, ebb1(v0)
-    jump ebb2
-
-ebb2:
-    ; v0 is live across the branch above. That's no conflict.
-    v1 = iadd_imm v0, 7
-    jump ebb1(v1)
-
-ebb1(v10: i32):
-    v11 = iadd_imm v10, 7
-    return v11
-}
-
-function %trivial(i32) -> i32 {
-ebb0(v0: i32):
-    ; check:    brnz v0, $(splitEdge=$EBB)
-    brnz v0, ebb1(v0)
-    jump ebb2
-
-ebb2:
-    ; not: copy
-    v1 = iadd_imm v0, 7
-    jump ebb1(v1)
-
-    ; check:  $splitEdge:
-    ; nextln:   $(cp1=$V) = copy.i32 v0
-    ; nextln:   jump ebb1($cp1)
-
-ebb1(v10: i32):
-    ; Use v0 in the destination EBB causes a conflict.
-    v11 = iadd v10, v0
-    return v11
-}
-
-; A value is used as an SSA argument twice in the same branch.
-function %dualuse(i32) -> i32 {
-ebb0(v0: i32):
-    ; check:  brnz v0, $(splitEdge=$EBB)
-    brnz v0, ebb1(v0, v0)
-    jump ebb2
-
-ebb2:
-    v1 = iadd_imm v0, 7
-    v2 = iadd_imm v1, 56
-    jump ebb1(v1, v2)
-
-    ; check:  $splitEdge:
-    ; check:    $(cp1=$V) = copy.i32 v0
-    ; nextln:   jump ebb1($cp1, v0)
-
-ebb1(v10: i32, v11: i32):
-    v12 = iadd v10, v11
-    return v12
-}
-
-; Interference away from the branch
-; The interference can be broken with a copy at either branch.
-function %interference(i32) -> i32 {
-ebb0(v0: i32):
-    ; not:    copy
-    ; check:  brnz v0, $(splitEdge=$EBB)
-    ; not:    copy
-    brnz v0, ebb1(v0)
-    jump ebb2
-
-ebb2:
-    v1 = iadd_imm v0, 7
-    ; v1 and v0 interfere here:
-    v2 = iadd_imm v0, 8
-    ; check: $(cp0=$V) = copy v1
-    ; check: jump ebb1($cp0)
-    jump ebb1(v1)
-
-    ; check:  $splitEdge:
-    ; not:      copy
-    ; nextln:   jump ebb1(v0)
-
-ebb1(v10: i32):
-    ; not: copy
-    v11 = iadd_imm v10, 7
-    return v11
-}
-
-; A loop where one induction variable is used as a backedge argument.
-function %fibonacci(i32) -> i32 {
-ebb0(v0: i32):
-    v1 = iconst.i32 1
-    v2 = iconst.i32 2
-    jump ebb1(v1, v2)
-
-    ; check:  $(splitEdge=$EBB):
-    ; check:    $(nv11b=$V) = copy.i32 v11
-    ; not:      copy
-    ; check:    jump ebb1($nv11b, v12)
-
-ebb1(v10: i32, v11: i32):
-    ; v11 needs to be isolated because it interferes with v10.
-    ; check: ebb1(v10: i32 [$LOC], $(nv11a=$V): i32 [$LOC])
-    ; check: v11 = copy $nv11a
-    v12 = iadd v10, v11
-    v13 = icmp ult v12, v0
-    ; check:  brnz v13, $splitEdge
-    brnz v13, ebb1(v11, v12)
-    jump ebb2
-
-ebb2:
-    return v12
-}
-
-; Function arguments passed on the stack aren't allowed to be part of a virtual
-; register, at least for now. This is because the other values in the virtual
-; register would need to be spilled to the incoming_arg stack slot which we treat
-; as belonging to the caller.
-function %stackarg(i32, i32, i32, i32, i32, i32, i32, i32, i32) -> i32 {
-; check: ss0 = incoming_arg 4
-; not: incoming_arg
-ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32, v7: i32, v8: i32):
-    ; check: fill v8
-    ; not: v8
-    jump ebb1(v8)
-
-ebb1(v10: i32):
-    v11 = iadd_imm v10, 1
-    return v11
-}
-
-function %gvn_unremovable_phi(i32) system_v {
-ebb0(v0: i32):
-    v2 = iconst.i32 0
-    jump ebb2(v2, v0)
-
-ebb2(v3: i32, v4: i32):
-    brnz v3, ebb2(v3, v4)
-    jump ebb3
-
-ebb3:
-    v5 = iconst.i32 1
-    brnz v3, ebb2(v2, v5)
-    jump ebb4
-
-ebb4:
-    return
-}
diff --git a/cranelift/filetests/filetests/regalloc/coalesce.clif b/cranelift/filetests/filetests/regalloc/coalesce.clif
index c78219ea53..3c139e73f6 100644
--- a/cranelift/filetests/filetests/regalloc/coalesce.clif
+++ b/cranelift/filetests/filetests/regalloc/coalesce.clif
@@ -1,11 +1,11 @@
 test regalloc
 target riscv32
-feature !"basic-blocks"
 
 ; Test the coalescer.
 ; regex: V=v\d+
 ; regex: WS=\s+
 ; regex: LOC=%\w+
+; regex: EBB=ebb\d+
 
 ; This function is already CSSA, so no copies should be inserted.
 function %cssa(i32) -> i32 {
@@ -27,8 +27,7 @@ ebb1(v10: i32):
 
 function %trivial(i32) -> i32 {
 ebb0(v0: i32):
-    ; check: $(cp1=$V) = copy v0
-    ; nextln: brnz v0, ebb1($cp1)
+    ; check:    brnz v0, $(splitEdge=$EBB)
     brnz v0, ebb1(v0)
     jump ebb2
 
@@ -37,6 +36,10 @@ ebb2:
     v1 = iadd_imm v0, 7
     jump ebb1(v1)
 
+    ; check:  $splitEdge:
+    ; nextln:   $(cp1=$V) = copy.i32 v0
+    ; nextln:   jump ebb1($cp1)
+
 ebb1(v10: i32):
     ; Use v0 in the destination EBB causes a conflict.
     v11 = iadd v10, v0
@@ -46,8 +49,7 @@ ebb1(v10: i32):
 ; A value is used as an SSA argument twice in the same branch.
 function %dualuse(i32) -> i32 {
 ebb0(v0: i32):
-    ; check: $(cp1=$V) = copy v0
-    ; nextln: brnz v0, ebb1($cp1, v0)
+    ; check:  brnz v0, $(splitEdge=$EBB)
     brnz v0, ebb1(v0, v0)
     jump ebb2
 
@@ -56,6 +58,10 @@ ebb2:
     v2 = iadd_imm v1, 56
     jump ebb1(v1, v2)
 
+    ; check:  $splitEdge:
+    ; check:    $(cp1=$V) = copy.i32 v0
+    ; nextln:   jump ebb1($cp1, v0)
+
 ebb1(v10: i32, v11: i32):
     v12 = iadd v10, v11
     return v12
@@ -65,9 +71,9 @@ ebb1(v10: i32, v11: i32):
 ; The interference can be broken with a copy at either branch.
 function %interference(i32) -> i32 {
 ebb0(v0: i32):
-    ; check: $(cp0=$V) = copy v0
-    ; not: copy
-    ; check: brnz v0, ebb1($cp0)
+    ; not:    copy
+    ; check:  brnz v0, $(splitEdge=$EBB)
+    ; not:    copy
     brnz v0, ebb1(v0)
     jump ebb2
 
@@ -75,10 +81,14 @@ ebb2:
     v1 = iadd_imm v0, 7
     ; v1 and v0 interfere here:
     v2 = iadd_imm v0, 8
-    ; not: copy
-    ; check: jump ebb1(v1)
+    ; check: $(cp0=$V) = copy v1
+    ; check: jump ebb1($cp0)
     jump ebb1(v1)
 
+    ; check:  $splitEdge:
+    ; not:      copy
+    ; nextln:   jump ebb1(v0)
+
 ebb1(v10: i32):
     ; not: copy
     v11 = iadd_imm v10, 7
@@ -92,15 +102,18 @@ ebb0(v0: i32):
     v2 = iconst.i32 2
     jump ebb1(v1, v2)
 
+    ; check:  $(splitEdge=$EBB):
+    ; check:    $(nv11b=$V) = copy.i32 v11
+    ; not:      copy
+    ; check:    jump ebb1($nv11b, v12)
+
 ebb1(v10: i32, v11: i32):
     ; v11 needs to be isolated because it interferes with v10.
     ; check: ebb1(v10: i32 [$LOC], $(nv11a=$V): i32 [$LOC])
     ; check: v11 = copy $nv11a
     v12 = iadd v10, v11
     v13 = icmp ult v12, v0
-    ; check: $(nv11b=$V) = copy v11
-    ; not: copy
-    ; check: brnz v13, ebb1($nv11b, v12)
+    ; check:  brnz v13, $splitEdge
     brnz v13, ebb1(v11, v12)
     jump ebb2
 
diff --git a/cranelift/filetests/filetests/regalloc/reload-208-bb.clif b/cranelift/filetests/filetests/regalloc/reload-208-bb.clif
deleted file mode 100644
index a93e4f9c34..0000000000
--- a/cranelift/filetests/filetests/regalloc/reload-208-bb.clif
+++ /dev/null
@@ -1,113 +0,0 @@
-test regalloc
-target x86_64 haswell
-feature "basic-blocks"
-
-; regex: V=v\d+
-; regex: EBB=ebb\d+
-
-; Filed as https://github.com/bytecodealliance/cranelift/issues/208
-;
-; The verifier complains about a branch argument that is not in the same virtual register as the
-; corresponding EBB argument.
-;
-; The problem was the reload pass rewriting EBB arguments on "brnz v9, ebb3(v9)"
-
-function %pr208(i64 vmctx [%rdi]) system_v {
-    gv1 = vmctx
-    gv0 = iadd_imm.i64 gv1, -8
-    heap0 = static gv0, min 0, bound 0x5000, offset_guard 0x0040_0000
-    sig0 = (i64 vmctx [%rdi]) -> i32 [%rax] system_v
-    sig1 = (i64 vmctx [%rdi], i32 [%rsi]) system_v
-    fn0 = u0:1 sig0
-    fn1 = u0:3 sig1
-
-ebb0(v0: i64):
-    v1 = iconst.i32 0
-    v2 = call fn0(v0)
-    v20 = iconst.i32 0x4ffe
-    v16 = icmp uge v2, v20
-    brz v16, ebb5
-    jump ebb9
-
-ebb9:
-    trap heap_oob
-
-ebb5:
-    v17 = uextend.i64 v2
-    v18 = iadd_imm.i64 v0, -8
-    v19 = load.i64 v18
-    v3 = iadd v19, v17
-    v4 = load.i32 v3
-    v21 = iconst.i32 0
-    v5 = icmp eq v4, v21
-    v6 = bint.i32 v5
-    brnz v6, ebb2
-    jump ebb3(v4)
-
-    ; check: ebb5:
-    ; check:   jump ebb3(v4)
-    ; check: $(splitEdge=$EBB):
-    ; nextln:  jump ebb3(v9)
-
-ebb3(v7: i32):
-    call fn1(v0, v7)
-    v26 = iconst.i32 0x4ffe
-    v22 = icmp uge v7, v26
-    brz v22, ebb6
-    jump ebb10
-
-ebb10:
-    trap heap_oob
-
-ebb6:
-    v23 = uextend.i64 v7
-    v24 = iadd_imm.i64 v0, -8
-    v25 = load.i64 v24
-    v8 = iadd v25, v23
-    v9 = load.i32 v8+56
-    ; check: v9 = spill
-    ; check: brnz $V, $splitEdge
-    brnz v9, ebb3(v9)
-    jump ebb4
-
-ebb4:
-    jump ebb2
-
-ebb2:
-    v10 = iconst.i32 0
-    v31 = iconst.i32 0x4ffe
-    v27 = icmp uge v10, v31
-    brz v27, ebb7
-    jump ebb11
-
-ebb11:
-    trap heap_oob
-
-ebb7:
-    v28 = uextend.i64 v10
-    v29 = iadd_imm.i64 v0, -8
-    v30 = load.i64 v29
-    v11 = iadd v30, v28
-    v12 = load.i32 v11+12
-    call fn1(v0, v12)
-    v13 = iconst.i32 0
-    v36 = iconst.i32 0x4ffe
-    v32 = icmp uge v13, v36
-    brz v32, ebb8
-    jump ebb12
-
-ebb12:
-    trap heap_oob
-
-ebb8:
-    v33 = uextend.i64 v13
-    v34 = iadd_imm.i64 v0, -8
-    v35 = load.i64 v34
-    v14 = iadd v35, v33
-    v15 = load.i32 v14+12
-    call fn1(v0, v15)
-    jump ebb1
-
-ebb1:
-    return
-}
diff --git a/cranelift/filetests/filetests/regalloc/reload-208.clif b/cranelift/filetests/filetests/regalloc/reload-208.clif
index 115f67e806..23783def04 100644
--- a/cranelift/filetests/filetests/regalloc/reload-208.clif
+++ b/cranelift/filetests/filetests/regalloc/reload-208.clif
@@ -1,8 +1,8 @@
 test regalloc
 target x86_64 haswell
-feature !"basic-blocks"
 
 ; regex: V=v\d+
+; regex: EBB=ebb\d+
 
 ; Filed as https://github.com/bytecodealliance/cranelift/issues/208
 ;
@@ -43,6 +43,11 @@ ebb5:
     brnz v6, ebb2
     jump ebb3(v4)
 
+    ; check: ebb5:
+    ; check:   jump ebb3(v4)
+    ; check: $(splitEdge=$EBB):
+    ; nextln:  jump ebb3(v9)
+
 ebb3(v7: i32):
     call fn1(v0, v7)
     v26 = iconst.i32 0x4ffe
@@ -60,7 +65,7 @@ ebb6:
     v8 = iadd v25, v23
     v9 = load.i32 v8+56
     ; check: v9 = spill
-    ; check: brnz $V, ebb3(v9)
+    ; check: brnz $V, $splitEdge
     brnz v9, ebb3(v9)
     jump ebb4
 
diff --git a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif
index c06c22f6b3..771957c44e 100644
--- a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif
+++ b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif
@@ -2,7 +2,6 @@ test compile
 set opt_level=speed
 set enable_pinned_reg=true
 target x86_64 haswell
-feature !"basic-blocks"
 
 function u0:0(i32, i32, i32, i64 vmctx) -> i64 uext system_v {
 ebb0(v0: i32, v1: i32, v2: i32, v3: i64):
@@ -11,8 +10,8 @@ ebb0(v0: i32, v1: i32, v2: i32, v3: i64):
     v16 = iconst.i32 -8
     v17 = popcnt v16
     v192 = ifcmp_imm v17, -1
-    brif eq v192, ebb12
-    trap user0
+    trapif ne v192, user0
+    jump ebb12
 
 ebb12:
     v122 = iconst.i32 0
@@ -23,8 +22,8 @@ ebb12:
     v204 = iconst.i32 0
     v31 -> v204
     v210 = ifcmp_imm v31, -1
-    brif eq v210, ebb18
-    trap user0
+    trapif ne v210, user0
+    jump ebb18
 
 ebb18:
     v215 = iconst.i32 0
@@ -33,8 +32,8 @@ ebb18:
 ebb19(v32: i32):
     v35 = iconst.i32 0
     v218 = ifcmp_imm v35, -1
-    brif eq v218, ebb21
-    trap user0
+    trapif ne v218, user0
+    jump ebb21
 
 ebb21:
     v223 = iconst.i32 0
@@ -44,8 +43,8 @@ ebb22(v36: i32):
     v136 = iconst.i32 0
     v40 -> v136
     v227 = ifcmp_imm v136, -1
-    brif eq v227, ebb24
-    trap user0
+    trapif ne v227, user0
+    jump ebb24
 
 ebb24:
     v232 = iconst.i32 0
@@ -55,8 +54,8 @@ ebb25(v41: i32):
     v142 = iconst.i32 0
     v45 -> v142
     v236 = ifcmp_imm v142, -1
-    brif eq v236, ebb27
-    trap user0
+    trapif ne v236, user0
+    jump ebb27
 
 ebb27:
     v241 = iconst.i32 0
@@ -65,8 +64,8 @@ ebb27:
 ebb28(v46: i32):
     v49 = iconst.i32 0
     v244 = ifcmp_imm v49, -1
-    brif eq v244, ebb30
-    trap user0
+    trapif ne v244, user0
+    jump ebb30
 
 ebb30:
     v254 = iconst.i32 0
@@ -92,9 +91,9 @@ ebb35:
     v60 -> v271
     v61 = f32const 0.0
     v280 = iconst.i32 0
-    v281 = fcmp uno v61, v61
-    brnz v281, ebb41(v280)
-    trap user0
+    v281 = ffcmp v61, v61
+    trapff ord v281, user0
+    jump ebb41(v280)
 
 ebb41(v62: i32):
     v157 = iconst.i32 0
@@ -103,8 +102,8 @@ ebb41(v62: i32):
     v160 = iadd v158, v159
     v75 -> v160
     v308 = ifcmp_imm v160, -1
-    brif eq v308, ebb52
-    trap user0
+    trapif ne v308, user0
+    jump ebb52
 
 ebb52:
     v87 = iconst.i32 -23
diff --git a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
index 2756ada054..7182bea43a 100644
--- a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
+++ b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
@@ -2,7 +2,6 @@ test compile
 set opt_level=speed
 set enable_pinned_reg=true
 target x86_64 haswell
-feature !"basic-blocks"
 
 ;; Test for the issue #1123; https://github.com/bytecodealliance/cranelift/issues/1123
 
@@ -24,26 +23,26 @@ ebb0(v0: i32, v1: i32, v2: i32, v3: i64):
     v39 -> v533
     v53 = iconst.i32 0
     v547 = ifcmp_imm v53, -1
-    brif eq v547, ebb30
-    trap user0
+    trapif ne v547, user0
+    jump ebb30
 
 ebb30:
     v75 = iconst.i32 0
     v581 = ifcmp_imm v75, -1
-    brif eq v581, ebb42
-    trap user0
+    trapif ne v581, user0
+    jump ebb42
 
 ebb42:
     v136 = iconst.i32 0
     v691 = ifcmp_imm v136, -1
-    brif eq v691, ebb81
-    trap user0
+    trapif ne v691, user0
+    jump ebb81
 
 ebb81:
     v158 = iconst.i32 0
     v725 = ifcmp_imm v158, -1
-    brif eq v725, ebb93
-    trap user0
+    trapif ne v725, user0
+    jump ebb93
 
 ebb93:
     v760 = iconst.i32 0
@@ -54,8 +53,8 @@ ebb106(v175: i32):
     v180 = icmp_imm eq v179, 0
     v183 = iconst.i32 0
     v766 = ifcmp_imm v183, -1
-    brif eq v766, ebb108
-    trap user0
+    trapif ne v766, user0
+    jump ebb108
 
 ebb108:
     v771 = iconst.i32 0
@@ -65,8 +64,8 @@ ebb109(v184: i32):
     v785 = iconst.i32 0
     v193 -> v785
     v791 = ifcmp_imm v193, -1
-    brif eq v791, ebb117
-    trap user0
+    trapif ne v791, user0
+    jump ebb117
 
 ebb117:
     v796 = iconst.i32 0
@@ -77,14 +76,14 @@ ebb118(v194: i32):
     v809 = iconst.i32 0
     v207 -> v809
     v815 = ifcmp_imm v207, -1
-    brif eq v815, ebb126
-    trap user0
+    trapif ne v815, user0
+    jump ebb126
 
 ebb126:
     v209 = iconst.i32 0
     v823 = ifcmp_imm v209, -1
-    brif eq v823, ebb129
-    trap user0
+    trapif ne v823, user0
+    jump ebb129
 
 ebb129:
     v213 = iconst.i32 -23
diff --git a/cranelift/filetests/filetests/regalloc/x86-regres-bb.clif b/cranelift/filetests/filetests/regalloc/x86-regres-bb.clif
deleted file mode 100644
index 060164034a..0000000000
--- a/cranelift/filetests/filetests/regalloc/x86-regres-bb.clif
+++ /dev/null
@@ -1,50 +0,0 @@
-test regalloc
-target i686
-feature "basic-blocks"
-
-; regex: V=v\d+
-; regex: EBB=ebb\d+
-
-; The value v9 appears both as the branch control and one of the EBB arguments
-; in the brnz instruction in ebb2. It also happens that v7 and v9 are assigned
-; to the same register, so v9 doesn't need to be moved before the brnz.
-;
-; This ended up confusong the constraint solver which had not made a record of
-; the fixed register assignment for v9 since it was already in the correct
-; register.
-function %pr147(i32) -> i32 system_v {
-ebb0(v0: i32):
-    v1 = iconst.i32 0
-    v2 = iconst.i32 1
-    v3 = iconst.i32 0
-    jump ebb2(v3, v2, v0)
-
-    ; check: $(splitEdge=$EBB):
-    ; check:   jump ebb2($V, $V, v9)
-
-ebb2(v4: i32, v5: i32, v7: i32):
-    ; check: ebb2
-    v6 = iadd v4, v5
-    v8 = iconst.i32 -1
-    ; v7 is killed here and v9 gets the same register.
-    v9 = iadd v7, v8
-    ; check:   v9 = iadd v7, v8
-    ; Here v9 the brnz control appears to interfere with v9 the EBB argument,
-    ; so divert_fixed_input_conflicts() calls add_var(v9), which is ok. The
-    ; add_var sanity checks got confused when no fixed assignment could be
-    ; found for v9.
-    ;
-    ; We should be able to handle this situation without making copies of v9.
-    brnz v9, ebb2(v5, v6, v9)
-    ; check:   brnz v9, $splitEdge
-    jump ebb3
-
-ebb3:
-    return v5
-}
-
-function %select_i64(i64, i64, i32) -> i64 {
-ebb0(v0: i64, v1: i64, v2: i32):
-    v3 = select v2, v0, v1
-    return v3
-}
diff --git a/cranelift/filetests/filetests/regalloc/x86-regres.clif b/cranelift/filetests/filetests/regalloc/x86-regres.clif
index ac6e82d66f..0b9bf12736 100644
--- a/cranelift/filetests/filetests/regalloc/x86-regres.clif
+++ b/cranelift/filetests/filetests/regalloc/x86-regres.clif
@@ -1,8 +1,8 @@
 test regalloc
 target i686
-feature !"basic-blocks"
 
 ; regex: V=v\d+
+; regex: EBB=ebb\d+
 
 ; The value v9 appears both as the branch control and one of the EBB arguments
 ; in the brnz instruction in ebb2. It also happens that v7 and v9 are assigned
@@ -18,13 +18,16 @@ ebb0(v0: i32):
     v3 = iconst.i32 0
     jump ebb2(v3, v2, v0)
 
+    ; check: $(splitEdge=$EBB):
+    ; check:   jump ebb2($V, $V, v9)
+
 ebb2(v4: i32, v5: i32, v7: i32):
     ; check: ebb2
     v6 = iadd v4, v5
     v8 = iconst.i32 -1
     ; v7 is killed here and v9 gets the same register.
     v9 = iadd v7, v8
-    ; check: v9 = iadd v7, v8
+    ; check:   v9 = iadd v7, v8
     ; Here v9 the brnz control appears to interfere with v9 the EBB argument,
     ; so divert_fixed_input_conflicts() calls add_var(v9), which is ok. The
     ; add_var sanity checks got confused when no fixed assignment could be
@@ -32,7 +35,7 @@ ebb2(v4: i32, v5: i32, v7: i32):
     ;
     ; We should be able to handle this situation without making copies of v9.
     brnz v9, ebb2(v5, v6, v9)
-    ; check: brnz v9, ebb2($V, $V, v9)
+    ; check:   brnz v9, $splitEdge
     jump ebb3
 
 ebb3:
diff --git a/cranelift/filetests/filetests/safepoint/basic-bb.clif b/cranelift/filetests/filetests/safepoint/basic-bb.clif
deleted file mode 100644
index 14a43d09a6..0000000000
--- a/cranelift/filetests/filetests/safepoint/basic-bb.clif
+++ /dev/null
@@ -1,72 +0,0 @@
-test safepoint
-set enable_safepoints=true
-target x86_64
-feature "basic-blocks"
-
-function %test(i32, r64, r64) -> r64 {
-    ebb0(v0: i32, v1:r64, v2:r64):
-        jump ebb1(v0)
-    ebb1(v3: i32):
-        v4 = irsub_imm v3, 1
-        jump ebb2(v4)
-    ebb2(v5: i32):
-        resumable_trap interrupt
-        brz v5, ebb1(v5)
-        jump ebb3
-    ebb3:
-        v6 = null.r64
-        v7 = is_null v6
-        brnz v7, ebb2(v0)
-        jump ebb4
-    ebb4:
-        brnz v0, ebb5
-        jump ebb6
-    ebb5:
-        return v1
-    ebb6:
-        return v2
-}
-
-; sameln: function %test(i32 [%rdi], r64 [%rsi], r64 [%rdx]) -> r64 [%rax] fast {
-; nextln: ebb0(v0: i32 [%rdi], v1: r64 [%rsi], v2: r64 [%rdx]):
-; nextln:   v10 = copy v0
-; nextln:   jump ebb1(v10)
-; nextln: 
-; nextln: ebb7:
-; nextln:   regmove.i32 v5, %rcx -> %rax
-; nextln:   jump ebb1(v5)
-; nextln: 
-; nextln: ebb1(v3: i32 [%rax]):
-; nextln:   v8 = iconst.i32 1
-; nextln:   v4 = isub v8, v3
-; nextln:   jump ebb2(v4)
-; nextln: 
-; nextln: ebb8:
-; nextln:   v9 = copy.i32 v0
-; nextln:   regmove v9, %rax -> %rcx
-; nextln:   jump ebb2(v9)
-; nextln: 
-; nextln: ebb2(v5: i32 [%rcx]):
-; nextln:   safepoint v1, v2
-; nextln:   resumable_trap interrupt
-; nextln:   brz v5, ebb7
-; nextln:   jump ebb3
-; nextln: 
-; nextln: ebb3:
-; nextln:   v6 = null.r64 
-; nextln:   v7 = is_null v6
-; nextln:   brnz v7, ebb8
-; nextln:   jump ebb4
-; nextln: 
-; nextln: ebb4:
-; nextln:   brnz.i32 v0, ebb5
-; nextln:   jump ebb6
-; nextln: 
-; nextln: ebb5:
-; nextln:   regmove.r64 v1, %rsi -> %rax
-; nextln:   return v1
-; nextln: 
-; nextln: ebb6:
-; nextln:   regmove.r64 v2, %rdx -> %rax
-; nextln:   return v2
-; nextln: }
diff --git a/cranelift/filetests/filetests/safepoint/basic.clif b/cranelift/filetests/filetests/safepoint/basic.clif
index 820594e85b..cb52fbf66b 100644
--- a/cranelift/filetests/filetests/safepoint/basic.clif
+++ b/cranelift/filetests/filetests/safepoint/basic.clif
@@ -1,7 +1,6 @@
 test safepoint
 set enable_safepoints=true
 target x86_64
-feature !"basic-blocks"
 
 function %test(i32, r64, r64) -> r64 {
     ebb0(v0: i32, v1:r64, v2:r64):
@@ -12,14 +11,18 @@ function %test(i32, r64, r64) -> r64 {
     ebb2(v5: i32):
         resumable_trap interrupt
         brz v5, ebb1(v5)
+        jump ebb3
+    ebb3:
         v6 = null.r64
         v7 = is_null v6
         brnz v7, ebb2(v0)
-        brnz v0, ebb3
         jump ebb4
-    ebb3:
-        return v1
     ebb4:
+        brnz v0, ebb5
+        jump ebb6
+    ebb5:
+        return v1
+    ebb6:
         return v2
 }
 
@@ -28,28 +31,41 @@ function %test(i32, r64, r64) -> r64 {
 ; nextln:   v10 = copy v0
 ; nextln:   jump ebb1(v10)
 ; nextln: 
+; nextln: ebb7:
+; nextln:   regmove.i32 v5, %rcx -> %rax
+; nextln:   jump ebb1(v5)
+; nextln: 
 ; nextln: ebb1(v3: i32 [%rax]):
 ; nextln:   v8 = iconst.i32 1
 ; nextln:   v4 = isub v8, v3
 ; nextln:   jump ebb2(v4)
 ; nextln: 
+; nextln: ebb8:
+; nextln:   v9 = copy.i32 v0
+; nextln:   regmove v9, %rax -> %rcx
+; nextln:   jump ebb2(v9)
+; nextln: 
 ; nextln: ebb2(v5: i32 [%rcx]):
 ; nextln:   safepoint v1, v2
 ; nextln:   resumable_trap interrupt
-; nextln:   regmove v5, %rcx -> %rax
-; nextln:   brz v5, ebb1(v5)
-; nextln:   v6 = null.r64 
-; nextln:   v7 = is_null v6
-; nextln:   v9 = copy.i32 v0
-; nextln:   brnz v7, ebb2(v9)
-; nextln:   brnz.i32 v0, ebb3
-; nextln:   jump ebb4
+; nextln:   brz v5, ebb7
+; nextln:   jump ebb3
 ; nextln: 
 ; nextln: ebb3:
+; nextln:   v6 = null.r64 
+; nextln:   v7 = is_null v6
+; nextln:   brnz v7, ebb8
+; nextln:   jump ebb4
+; nextln: 
+; nextln: ebb4:
+; nextln:   brnz.i32 v0, ebb5
+; nextln:   jump ebb6
+; nextln: 
+; nextln: ebb5:
 ; nextln:   regmove.r64 v1, %rsi -> %rax
 ; nextln:   return v1
 ; nextln: 
-; nextln: ebb4:
+; nextln: ebb6:
 ; nextln:   regmove.r64 v2, %rdx -> %rax
 ; nextln:   return v2
 ; nextln: }
diff --git a/cranelift/filetests/src/runone.rs b/cranelift/filetests/src/runone.rs
index 3cf399abbe..7dbedf8c87 100644
--- a/cranelift/filetests/src/runone.rs
+++ b/cranelift/filetests/src/runone.rs
@@ -52,31 +52,6 @@ pub fn run(path: &Path, passes: Option<&[String]>, target: Option<&str>) -> Test
         }
     };
 
-    for feature in testfile.features.iter() {
-        let (flag, test_expect) = match feature {
-            Feature::With(name) => (name, true),
-            Feature::Without(name) => (name, false),
-        };
-        let cranelift_has = match *flag {
-            // Add any cranelift feature flag here, and make sure that it is forwarded to the
-            // cranelift-filetest crate in the top-level Cargo.toml.
-            "basic-blocks" => cfg!(feature = "basic-blocks"),
-            _ => {
-                return Err(format!(
-                    r#"{:?}: Unknown feature flag named "{}""#,
-                    path, flag
-                ))
-            }
-        };
-        if cranelift_has != test_expect {
-            println!(
-                r#"skipping test {:?}: non-matching feature flag "{}""#,
-                path, flag
-            );
-            return Ok(started.elapsed());
-        }
-    }
-
     if testfile.functions.is_empty() {
         return Err("no functions found".to_string());
     }
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 9e1c930e94..3a711f5b10 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -18,13 +18,10 @@ hashbrown = { version = "0.6", optional = true }
 smallvec = { version = "1.0.0" }
 
 [features]
-default = ["std", "basic-blocks"]
+default = ["std"]
 std = ["cranelift-codegen/std"]
 core = ["hashbrown", "cranelift-codegen/core"]
 
-# Temporary feature that enforces basic block semantics.
-basic-blocks = ["cranelift-codegen/basic-blocks"]
-
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index 85a7725930..e1be5e774e 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -480,7 +480,6 @@ impl<'a> FunctionBuilder<'a> {
         );
 
         // In debug mode, check that all blocks are valid basic blocks.
-        #[cfg(feature = "basic-blocks")]
         #[cfg(debug_assertions)]
         {
             // Iterate manually to provide more helpful error messages.
diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index ac1ccecc2d..5d075e0001 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -589,19 +589,16 @@ impl Mutator for MergeBlocks {
 
         let pred = cfg.pred_iter(ebb).next().unwrap();
 
-        #[cfg(feature = "basic-blocks")]
-        {
-            // If the branch instruction that lead us to this block is preceded by another branch
-            // instruction, then we have a conditional jump sequence that we should not break by
-            // replacing the second instruction by more of them.
-            if let Some(pred_pred_inst) = func.layout.prev_inst(pred.inst) {
-                if func.dfg[pred_pred_inst].opcode().is_branch() {
-                    return Some((
-                        func,
-                        format!("did nothing for {}", ebb),
-                        ProgressStatus::Skip,
-                    ));
-                }
+        // If the branch instruction that lead us to this block is preceded by another branch
+        // instruction, then we have a conditional jump sequence that we should not break by
+        // replacing the second instruction by more of them.
+        if let Some(pred_pred_inst) = func.layout.prev_inst(pred.inst) {
+            if func.dfg[pred_pred_inst].opcode().is_branch() {
+                return Some((
+                    func,
+                    format!("did nothing for {}", ebb),
+                    ProgressStatus::Skip,
+                ));
             }
         }
 
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 1b58a6c611..bb0887f287 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -25,14 +25,11 @@ wat = "1.0.7"
 target-lexicon = "0.10"
 
 [features]
-default = ["std", "basic-blocks"]
+default = ["std"]
 std = ["cranelift-codegen/std", "cranelift-frontend/std"]
 core = ["hashbrown", "cranelift-codegen/core", "cranelift-frontend/core"]
 enable-serde = ["serde"]
 
-# Temporary feature that enforces basic block semantics.
-basic-blocks = ["cranelift-codegen/basic-blocks", "cranelift-frontend/basic-blocks"]
-
 [badges]
 maintenance = { status = "experimental" }
 travis-ci = { repository = "bytecodealliance/cranelift" }
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index dcbfe8eeb9..a767f093db 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -205,13 +205,10 @@ pub fn translate_operator(
                 (destination, ElseData::WithElse { else_block })
             };
 
-            #[cfg(feature = "basic-blocks")]
-            {
-                let next_ebb = builder.create_ebb();
-                builder.ins().jump(next_ebb, &[]);
-                builder.seal_block(next_ebb); // Only predecessor is the current block.
-                builder.switch_to_block(next_ebb);
-            }
+            let next_ebb = builder.create_ebb();
+            builder.ins().jump(next_ebb, &[]);
+            builder.seal_block(next_ebb); // Only predecessor is the current block.
+            builder.switch_to_block(next_ebb);
 
             // Here we append an argument to an Ebb targeted by an argumentless jump instruction
             // But in fact there are two cases:
@@ -1744,13 +1741,10 @@ fn translate_br_if(
 
     builder.ins().brnz(val, br_destination, inputs);
 
-    #[cfg(feature = "basic-blocks")]
-    {
-        let next_ebb = builder.create_ebb();
-        builder.ins().jump(next_ebb, &[]);
-        builder.seal_block(next_ebb); // The only predecessor is the current block.
-        builder.switch_to_block(next_ebb);
-    }
+    let next_ebb = builder.create_ebb();
+    builder.ins().jump(next_ebb, &[]);
+    builder.seal_block(next_ebb); // The only predecessor is the current block.
+    builder.switch_to_block(next_ebb);
 }
 
 fn translate_br_if_args(

From 5edf015ada5c7cdc85369443b063ea0162ba6591 Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Fri, 24 Jan 2020 10:43:44 -0500
Subject: [PATCH 3036/3084] Make `get_libcall_funcref` `pub(crate)` (#1291)

* Make `get_libcall_funcref` `pub(crate)`

Closes https://github.com/bytecodealliance/cranelift/issues/1273.

Since get_libcall_funcref is only used internally by the verifier,
it doesn't make sense to have it be public. This will encourage users to
look elsewhere for `memcpy` (they should be looking at
https://docs.rs/cranelift-frontend/0.51.0/cranelift_frontend/struct.FunctionBuilder.html#method.emit_small_memcpy)
---
 cranelift/codegen/src/ir/libcall.rs        | 2 +-
 cranelift/codegen/src/ir/mod.rs            | 4 ++--
 cranelift/codegen/src/legalizer/libcall.rs | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/src/ir/libcall.rs b/cranelift/codegen/src/ir/libcall.rs
index cffbfb5a90..33d3fa6060 100644
--- a/cranelift/codegen/src/ir/libcall.rs
+++ b/cranelift/codegen/src/ir/libcall.rs
@@ -106,7 +106,7 @@ impl LibCall {
 /// for `inst`.
 ///
 /// If there is an existing reference, use it, otherwise make a new one.
-pub fn get_libcall_funcref(
+pub(crate) fn get_libcall_funcref(
     libcall: LibCall,
     call_conv: CallConv,
     func: &mut Function,
diff --git a/cranelift/codegen/src/ir/mod.rs b/cranelift/codegen/src/ir/mod.rs
index b37812e9d3..096e372db0 100644
--- a/cranelift/codegen/src/ir/mod.rs
+++ b/cranelift/codegen/src/ir/mod.rs
@@ -14,7 +14,7 @@ pub mod immediates;
 pub mod instructions;
 pub mod jumptable;
 pub mod layout;
-mod libcall;
+pub(crate) mod libcall;
 mod memflags;
 mod progpoint;
 mod sourceloc;
@@ -49,7 +49,7 @@ pub use crate::ir::instructions::{
 };
 pub use crate::ir::jumptable::JumpTableData;
 pub use crate::ir::layout::Layout;
-pub use crate::ir::libcall::{get_libcall_funcref, get_probestack_funcref, LibCall};
+pub use crate::ir::libcall::{get_probestack_funcref, LibCall};
 pub use crate::ir::memflags::MemFlags;
 pub use crate::ir::progpoint::{ExpandedProgramPoint, ProgramOrder, ProgramPoint};
 pub use crate::ir::sourceloc::SourceLoc;
diff --git a/cranelift/codegen/src/legalizer/libcall.rs b/cranelift/codegen/src/legalizer/libcall.rs
index b34a90b519..2a4aeb4912 100644
--- a/cranelift/codegen/src/legalizer/libcall.rs
+++ b/cranelift/codegen/src/legalizer/libcall.rs
@@ -1,7 +1,7 @@
 //! Expanding instructions as runtime library calls.
 
 use crate::ir;
-use crate::ir::{get_libcall_funcref, InstBuilder};
+use crate::ir::{libcall::get_libcall_funcref, InstBuilder};
 use crate::isa::{CallConv, TargetIsa};
 use crate::legalizer::boundary::legalize_libcall_signature;
 use alloc::vec::Vec;

From 3502cd3cd18311d5afe2506728f961d5817d98b4 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 28 Jan 2020 10:11:37 -0800
Subject: [PATCH 3037/3084] Remove unused import; fixes #1367 (#1368)

---
 cranelift/filetests/src/runone.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/filetests/src/runone.rs b/cranelift/filetests/src/runone.rs
index 7dbedf8c87..8d1c36dff8 100644
--- a/cranelift/filetests/src/runone.rs
+++ b/cranelift/filetests/src/runone.rs
@@ -8,7 +8,7 @@ use cranelift_codegen::print_errors::pretty_verifier_error;
 use cranelift_codegen::settings::Flags;
 use cranelift_codegen::timing;
 use cranelift_codegen::verify_function;
-use cranelift_reader::{parse_test, Feature, IsaSpec, ParseOptions};
+use cranelift_reader::{parse_test, IsaSpec, ParseOptions};
 use log::info;
 use std::borrow::Cow;
 use std::fs;

From 169dbef784742331de4fed466f7159b530383954 Mon Sep 17 00:00:00 2001
From: Yury Delendik 
Date: Mon, 3 Feb 2020 16:08:40 -0600
Subject: [PATCH 3038/3084] Properly preserve and restore CFA state in FDE
 (#1373)

* Properly preserve and restore CFA state in FDE
---
 cranelift/codegen/src/isa/x86/abi.rs | 39 ++++++-------
 cranelift/codegen/src/isa/x86/fde.rs | 85 ++++++++++++++++++++++++++--
 2 files changed, 99 insertions(+), 25 deletions(-)

diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 6a1a3cea11..3d160a2cf5 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -835,25 +835,7 @@ fn insert_common_epilogues(
         pos.goto_last_inst(ebb);
         if let Some(inst) = pos.current_inst() {
             if pos.func.dfg[inst].opcode().is_return() {
-                if let (Some(ref mut frame_layout), ref func_layout) =
-                    (pos.func.frame_layout.as_mut(), &pos.func.layout)
-                {
-                    // Figure out if we need to insert end-of-function-aware frame layout information.
-                    let following_inst = func_layout
-                        .next_ebb(ebb)
-                        .and_then(|next_ebb| func_layout.first_inst(next_ebb));
-
-                    if let Some(following_inst) = following_inst {
-                        frame_layout
-                            .instructions
-                            .insert(inst, vec![FrameLayoutChange::Preserve].into_boxed_slice());
-                        frame_layout.instructions.insert(
-                            following_inst,
-                            vec![FrameLayoutChange::Restore].into_boxed_slice(),
-                        );
-                    }
-                }
-
+                let is_last = pos.func.layout.last_ebb() == Some(ebb);
                 insert_common_epilogue(
                     inst,
                     stack_size,
@@ -861,6 +843,7 @@ fn insert_common_epilogues(
                     reg_type,
                     csrs,
                     isa,
+                    is_last,
                     cfa_state.clone(),
                 );
             }
@@ -877,6 +860,7 @@ fn insert_common_epilogue(
     reg_type: ir::types::Type,
     csrs: &RegisterSet,
     isa: &dyn TargetIsa,
+    is_last: bool,
     mut cfa_state: Option,
 ) {
     let word_size = isa.pointer_bytes() as isize;
@@ -931,10 +915,16 @@ fn insert_common_epilogue(
         assert_eq!(cfa_state.current_depth, -word_size);
         assert_eq!(cfa_state.cf_ptr_offset, word_size);
 
+        // Inserting preserve CFA state operation after FP pop instructions.
         let new_cfa = FrameLayoutChange::CallFrameAddressAt {
             reg: cfa_state.cf_ptr_reg,
             offset: cfa_state.cf_ptr_offset,
         };
+        let new_cfa = if is_last {
+            vec![new_cfa]
+        } else {
+            vec![FrameLayoutChange::Preserve, new_cfa]
+        };
 
         frame_layout
             .instructions
@@ -943,10 +933,17 @@ fn insert_common_epilogue(
                 *insts = insts
                     .iter()
                     .cloned()
-                    .chain(std::iter::once(new_cfa))
+                    .chain(new_cfa.clone().into_iter())
                     .collect::>();
             })
-            .or_insert_with(|| Box::new([new_cfa]));
+            .or_insert_with(|| new_cfa.into_boxed_slice());
+
+        if !is_last {
+            // Inserting restore CFA state operation after each return.
+            frame_layout
+                .instructions
+                .insert(inst, vec![FrameLayoutChange::Restore].into_boxed_slice());
+        }
     }
 }
 
diff --git a/cranelift/codegen/src/isa/x86/fde.rs b/cranelift/codegen/src/isa/x86/fde.rs
index 3e04c93af2..6a56a09451 100644
--- a/cranelift/codegen/src/isa/x86/fde.rs
+++ b/cranelift/codegen/src/isa/x86/fde.rs
@@ -163,9 +163,8 @@ fn to_cfi(
             let cfa_offset = *cfa_offset as i32;
             CallFrameInstruction::Offset(X86_64::RA, cfa_offset)
         }
-        _ => {
-            return None;
-        }
+        FrameLayoutChange::Preserve => CallFrameInstruction::RememberState,
+        FrameLayoutChange::Restore => CallFrameInstruction::RestoreState,
     })
 }
 
@@ -265,7 +264,10 @@ mod tests {
     use super::*;
     use crate::binemit::{FrameUnwindOffset, Reloc};
     use crate::cursor::{Cursor, FuncCursor};
-    use crate::ir::{ExternalName, InstBuilder, Signature, StackSlotData, StackSlotKind};
+    use crate::ir::{
+        types, AbiParam, ExternalName, InstBuilder, Signature, StackSlotData, StackSlotKind,
+        TrapCode,
+    };
     use crate::isa::{lookup, CallConv};
     use crate::settings::{builder, Flags};
     use crate::Context;
@@ -352,4 +354,79 @@ mod tests {
 
         func
     }
+
+    #[test]
+    fn test_multi_return_func() {
+        let isa = lookup(triple!("x86_64"))
+            .expect("expect x86 ISA")
+            .finish(Flags::new(builder()));
+
+        let mut context = Context::for_function(create_multi_return_function(CallConv::SystemV));
+        context.func.collect_frame_layout_info();
+
+        context.compile(&*isa).expect("expected compilation");
+
+        let mut sink = SimpleUnwindSink(Vec::new(), 0, Vec::new());
+        emit_fde(&context.func, &*isa, &mut sink);
+
+        assert_eq!(
+            sink.0,
+            vec![
+                20, 0, 0, 0, // CIE len
+                0, 0, 0, 0,   // CIE marker
+                1,   // version
+                0,   // augmentation string
+                1,   // code aligment = 1
+                120, // data alignment = -8
+                16,  // RA = r16
+                0x0c, 0x07, 0x08, // DW_CFA_def_cfa r7, 8
+                0x90, 0x01, //  DW_CFA_offset r16, -8 * 1
+                0, 0, 0, 0, 0, 0, // padding
+                36, 0, 0, 0, // FDE len
+                28, 0, 0, 0, // CIE offset
+                0, 0, 0, 0, 0, 0, 0, 0, // addr reloc
+                15, 0, 0, 0, 0, 0, 0, 0,    // function length
+                0x42, // DW_CFA_advance_loc 2
+                0x0e, 0x10, // DW_CFA_def_cfa_offset 16
+                0x86, 0x02, // DW_CFA_offset r6, -8 * 2
+                0x43, // DW_CFA_advance_loc 3
+                0x0d, 0x06, // DW_CFA_def_cfa_register
+                0x47, // DW_CFA_advance_loc 10
+                0x0a, // DW_CFA_preserve_state
+                0x0c, 0x07, 0x08, // DW_CFA_def_cfa r7, 8
+                0x41, // DW_CFA_advance_loc 1
+                0x0b, // DW_CFA_restore_state
+                // NOTE: no additional CFA directives -- DW_CFA_restore_state
+                // is done before trap and it is last instruction in the function.
+                0, // padding
+                0, 0, 0, 0, // End of FDEs
+            ]
+        );
+        assert_eq!(sink.1, 24);
+        assert_eq!(sink.2.len(), 1);
+    }
+
+    fn create_multi_return_function(call_conv: CallConv) -> Function {
+        let mut sig = Signature::new(call_conv);
+        sig.params.push(AbiParam::new(types::I32));
+        let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
+
+        let ebb0 = func.dfg.make_ebb();
+        let v0 = func.dfg.append_ebb_param(ebb0, types::I32);
+        let ebb1 = func.dfg.make_ebb();
+        let ebb2 = func.dfg.make_ebb();
+
+        let mut pos = FuncCursor::new(&mut func);
+        pos.insert_ebb(ebb0);
+        pos.ins().brnz(v0, ebb2, &[]);
+        pos.ins().jump(ebb1, &[]);
+
+        pos.insert_ebb(ebb1);
+        pos.ins().return_(&[]);
+
+        pos.insert_ebb(ebb2);
+        pos.ins().trap(TrapCode::User(0));
+
+        func
+    }
 }

From 19a188789bf089f342a82078897f0b381b907860 Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Tue, 4 Feb 2020 16:18:59 +0100
Subject: [PATCH 3039/3084] Bump to 0.57.0 (#1375)

* Update wasmparser to 0.48.2

* Bump to version 0.57.0
---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           | 10 +++++-----
 18 files changed, 68 insertions(+), 68 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 30a0c434be..a6f57398f9 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.56.0"
+version = "0.57.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.56.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.56.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.56.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.56.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.56.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.56.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.56.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.56.0" }
-cranelift-module = { path = "cranelift-module", version = "0.56.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.56.0" }
-cranelift-object = { path = "cranelift-object", version = "0.56.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.56.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.56.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.56.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.57.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.57.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.57.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.57.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.57.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.57.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.57.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.57.0" }
+cranelift-module = { path = "cranelift-module", version = "0.57.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.57.0" }
+cranelift-object = { path = "cranelift-object", version = "0.57.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.57.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.57.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.57.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 0fc288dab4..899f33e246 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.56.0"
+version = "0.57.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.56.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.57.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 6d5c571261..1df76bb1d8 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.56.0"
+version = "0.57.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.56.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.56.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.56.0" }
+cranelift-codegen-shared = { path = "./shared", version = "0.57.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.57.0" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.57.0" }
 hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
@@ -30,7 +30,7 @@ byteorder = { version = "1.3.2", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.56.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.57.0" }
 
 [features]
 default = ["std", "unwind"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 85f07548eb..42f1d3e445 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.56.0"
+version = "0.57.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.56.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.56.0" }
+cranelift-codegen-shared = { path = "../shared", version = "0.57.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.57.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 81a4ed62e8..eb050e5780 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.56.0"
+version = "0.57.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index a9bd2ff323..b4a0c72931 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.56.0"
+version = "0.57.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 9b632ae9f9..878d2d266c 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.56.0"
+version = "0.57.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.56.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.57.0" }
 faerie = "0.14.0"
 goblin = "0.1.0"
 anyhow = "1.0"
@@ -18,7 +18,7 @@ target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.56.0"
+version = "0.57.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 04d4db4bb3..a66bf68dea 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.56.0"
+version = "0.57.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.56.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.56.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.56.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.57.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.57.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.57.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 gimli = { version = "0.20.0", default-features = false, features = ["read"] }
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 3a711f5b10..7cca019ed5 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.56.0"
+version = "0.57.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", default-features = false }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 8bf838d341..35de56226a 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.56.0"
+version = "0.57.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.56.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.57.0" }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 thiserror = "1.0.4"
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index cbbe4b1c06..c34ab1064a 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.56.0"
+version = "0.57.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", default-features = false }
 target-lexicon = "0.10"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index f65db0affc..6434a0f514 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.56.0"
+version = "0.57.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,13 +10,13 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.56.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.57.0" }
 object = { version = "0.17", default-features = false, features = ["write"] }
 target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.56.0"
+version = "0.57.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index f9c483befc..a714585fa0 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.56.0"
+version = "0.57.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.56.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.57.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 06b8772deb..00f129336c 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.56.0"
+version = "0.57.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0" }
 target-lexicon = "0.10"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index caba72db8c..7c277d063c 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.56.0"
+version = "0.57.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.56.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.57.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index c9b052ac52..6183d17a06 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.56.0"
+version = "0.57.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.56.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.56.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.57.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.57.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -20,7 +20,7 @@ memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.56.0"
+version = "0.57.0"
 default-features = false
 features = ["std"]
 
@@ -32,9 +32,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.56.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.56.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.56.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.57.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.57.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.57.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 07d99f9c74..fc8eecc3c3 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.56.0"
+version = "0.57.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.56.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.57.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index bb0887f287..6b89bfae1a 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.56.0"
+version = "0.57.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -11,10 +11,10 @@ keywords = ["webassembly", "wasm"]
 edition = "2018"
 
 [dependencies]
-wasmparser = { version = "0.47.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.56.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.56.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.56.0", default-features = false }
+wasmparser = { version = "0.48.2", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.57.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.57.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }

From a136d1cb001d9e3b95ff13547564134a80343e9b Mon Sep 17 00:00:00 2001
From: Nathan Froyd 
Date: Tue, 4 Feb 2020 14:44:21 -0500
Subject: [PATCH 3040/3084] add documentation mentions of cranelift-object
 where appropriate

This change makes it slightly more obvious that `cranelift-object` can
be used in lieu of `cranelift-faerie`.
---
 cranelift/docs/index.rst        | 5 +++++
 cranelift/module/README.md      | 3 +++
 cranelift/module/src/backend.rs | 5 ++++-
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/cranelift/docs/index.rst b/cranelift/docs/index.rst
index 9951f6d9e9..457cd3bacf 100644
--- a/cranelift/docs/index.rst
+++ b/cranelift/docs/index.rst
@@ -46,6 +46,11 @@ Rust Crate Documentation
     This crate manages compiling multiple functions and data objects
     together.
 
+`cranelift-object `_
+    This crate provides a object-based backend for `cranelift-module`, which
+    emits native object files using the
+    `object `_ library.
+
 `cranelift-faerie `_
     This crate provides a faerie-based backend for `cranelift-module`, which
     emits native object files using the
diff --git a/cranelift/module/README.md b/cranelift/module/README.md
index 35f973e916..07e7bd998d 100644
--- a/cranelift/module/README.md
+++ b/cranelift/module/README.md
@@ -13,8 +13,11 @@ following `Backend` implementations:
 
  - `SimpleJITBackend`, provided by [cranelift-simplejit], which JITs
    code to memory for direct execution.
+ - `ObjectBackend`, provided by [cranelift-object], which emits native
+   object files.
  - `FaerieBackend`, provided by [cranelift-faerie], which emits native
    object files.
 
 [cranelift-simplejit]: https://crates.io/crates/cranelift-simplejit
+[cranelift-object]: https://crates.io/crates/cranelift-object
 [cranelift-faerie]: https://crates.io/crates/cranelift-faerie
diff --git a/cranelift/module/src/backend.rs b/cranelift/module/src/backend.rs
index 89b610bdba..27472d042b 100644
--- a/cranelift/module/src/backend.rs
+++ b/cranelift/module/src/backend.rs
@@ -17,13 +17,16 @@ use std::string::String;
 
 /// A `Backend` implements the functionality needed to support a `Module`.
 ///
-/// Two notable implementations of this trait are:
+/// Three notable implementations of this trait are:
 ///  - `SimpleJITBackend`, defined in [cranelift-simplejit], which JITs
 ///    the contents of a `Module` to memory which can be directly executed.
+///  - `ObjectBackend`, defined in [cranelift-object], which writes the
+///    contents of a `Module` out as a native object file.
 ///  - `FaerieBackend`, defined in [cranelift-faerie], which writes the
 ///    contents of a `Module` out as a native object file.
 ///
 /// [cranelift-simplejit]: https://docs.rs/cranelift-simplejit/
+/// [cranelift-object]: https://docs.rs/cranelift-object/
 /// [cranelift-faerie]: https://docs.rs/cranelift-faerie/
 pub trait Backend
 where

From 832666c45ead73fd833ea86da94cacb2f30a530c Mon Sep 17 00:00:00 2001
From: Ryan Hunt 
Date: Fri, 7 Feb 2020 10:46:47 -0600
Subject: [PATCH 3041/3084] Mass rename Ebb and relatives to Block (#1365)

* Manually rename BasicBlock to BlockPredecessor

BasicBlock is a pair of (Ebb, Inst) that is used to represent the
basic block subcomponent of an Ebb that is a predecessor to an Ebb.

Eventually we will be able to remove this struct, but for now it
makes sense to give it a non-conflicting name so that we can start
to transition Ebb to represent a basic block.

I have not updated any comments that refer to BasicBlock, as
eventually we will remove BlockPredecessor and replace with Block,
which is a basic block, so the comments will become correct.

* Manually rename SSABuilder block types to avoid conflict

SSABuilder has its own Block and BlockData types. These along with
associated identifier will cause conflicts in a later commit, so
they are renamed to be more verbose here.

* Automatically rename 'Ebb' to 'Block' in *.rs

* Automatically rename 'EBB' to 'block' in *.rs

* Automatically rename 'ebb' to 'block' in *.rs

* Automatically rename 'extended basic block' to 'basic block' in *.rs

* Automatically rename 'an basic block' to 'a basic block' in *.rs

* Manually update comment for `Block`

`Block`'s wikipedia article required an update.

* Automatically rename 'an `Block`' to 'a `Block`' in *.rs

* Automatically rename 'extended_basic_block' to 'basic_block' in *.rs

* Automatically rename 'ebb' to 'block' in *.clif

* Manually rename clif constant that contains 'ebb' as substring to avoid conflict

* Automatically rename filecheck uses of 'EBB' to 'BB'

'regex: EBB' -> 'regex: BB'
'$EBB' -> '$BB'

* Automatically rename 'EBB' 'Ebb' to 'block' in *.clif

* Automatically rename 'an block' to 'a block' in *.clif

* Fix broken testcase when function name length increases

Test function names are limited to 16 characters. This causes
the new longer name to be truncated and fail a filecheck test. An
outdated comment was also fixed.
---
 cranelift/bforest/src/lib.rs                  |  22 +-
 cranelift/codegen/meta/src/cdsl/ast.rs        |   4 +-
 .../codegen/meta/src/cdsl/instructions.rs     |   2 +-
 cranelift/codegen/meta/src/gen_inst.rs        |   2 +-
 cranelift/codegen/meta/src/gen_legalizer.rs   |  14 +-
 cranelift/codegen/meta/src/shared/entities.rs |  14 +-
 cranelift/codegen/meta/src/shared/formats.rs  |  12 +-
 .../codegen/meta/src/shared/instructions.rs   |  40 +-
 cranelift/codegen/meta/src/shared/legalize.rs |  28 +-
 cranelift/codegen/src/abi.rs                  |   2 +-
 cranelift/codegen/src/binemit/memorysink.rs   |  10 +-
 cranelift/codegen/src/binemit/mod.rs          |  16 +-
 cranelift/codegen/src/binemit/relaxation.rs   | 104 +-
 cranelift/codegen/src/binemit/shrink.rs       |   6 +-
 cranelift/codegen/src/cfg_printer.rs          |  24 +-
 cranelift/codegen/src/cursor.rs               | 274 ++---
 cranelift/codegen/src/dce.rs                  |   4 +-
 cranelift/codegen/src/dominator_tree.rs       | 507 ++++-----
 cranelift/codegen/src/flowgraph.rs            | 260 ++---
 cranelift/codegen/src/ir/builder.rs           |  12 +-
 cranelift/codegen/src/ir/dfg.rs               | 315 +++---
 cranelift/codegen/src/ir/entities.rs          |  29 +-
 cranelift/codegen/src/ir/extname.rs           |   4 +-
 cranelift/codegen/src/ir/function.rs          |  36 +-
 cranelift/codegen/src/ir/instructions.rs      |  18 +-
 cranelift/codegen/src/ir/jumptable.rs         |  32 +-
 cranelift/codegen/src/ir/layout.rs            | 753 +++++++-------
 cranelift/codegen/src/ir/mod.rs               |   6 +-
 cranelift/codegen/src/ir/progpoint.rs         |  44 +-
 cranelift/codegen/src/isa/constraints.rs      |   4 +-
 cranelift/codegen/src/isa/riscv/mod.rs        |  16 +-
 cranelift/codegen/src/isa/x86/abi.rs          |  28 +-
 cranelift/codegen/src/isa/x86/binemit.rs      |   6 +-
 cranelift/codegen/src/isa/x86/enc_tables.rs   | 290 +++---
 cranelift/codegen/src/isa/x86/fde.rs          |  30 +-
 cranelift/codegen/src/isa/x86/unwind.rs       |   6 +-
 cranelift/codegen/src/legalizer/boundary.rs   |  37 +-
 cranelift/codegen/src/legalizer/heap.rs       |  12 +-
 cranelift/codegen/src/legalizer/mod.rs        | 126 +--
 cranelift/codegen/src/legalizer/split.rs      |  84 +-
 cranelift/codegen/src/licm.rs                 |  54 +-
 cranelift/codegen/src/loop_analysis.rs        | 184 ++--
 cranelift/codegen/src/nan_canonicalization.rs |   4 +-
 cranelift/codegen/src/postopt.rs              |   6 +-
 cranelift/codegen/src/print_errors.rs         |  18 +-
 .../codegen/src/redundant_reload_remover.rs   | 119 +--
 .../codegen/src/regalloc/branch_splitting.rs  |  44 +-
 cranelift/codegen/src/regalloc/coalescing.rs  | 288 ++---
 cranelift/codegen/src/regalloc/coloring.rs    | 112 +-
 cranelift/codegen/src/regalloc/diversion.rs   |  40 +-
 .../src/regalloc/live_value_tracker.rs        |  68 +-
 cranelift/codegen/src/regalloc/liveness.rs    |  88 +-
 cranelift/codegen/src/regalloc/liverange.rs   | 288 ++---
 cranelift/codegen/src/regalloc/reload.rs      |  70 +-
 cranelift/codegen/src/regalloc/safepoint.rs   |  12 +-
 cranelift/codegen/src/regalloc/solver.rs      |  10 +-
 cranelift/codegen/src/regalloc/spilling.rs    |  63 +-
 cranelift/codegen/src/regalloc/virtregs.rs    |   4 +-
 cranelift/codegen/src/simple_gvn.rs           |  10 +-
 cranelift/codegen/src/simple_preopt.rs        |  22 +-
 cranelift/codegen/src/topo_order.rs           |  78 +-
 cranelift/codegen/src/unreachable_code.rs     |  24 +-
 cranelift/codegen/src/value_label.rs          |  14 +-
 cranelift/codegen/src/verifier/cssa.rs        |  30 +-
 cranelift/codegen/src/verifier/flags.rs       |  42 +-
 cranelift/codegen/src/verifier/liveness.rs    |  71 +-
 cranelift/codegen/src/verifier/locations.rs   |  34 +-
 cranelift/codegen/src/verifier/mod.rs         | 244 ++---
 cranelift/codegen/src/write.rs                | 106 +-
 cranelift/docs/callex.clif                    |  10 +-
 cranelift/docs/example.clif                   |  20 +-
 cranelift/docs/heapex-dyn.clif                |   2 +-
 cranelift/docs/heapex-sm32.clif               |   2 +-
 cranelift/docs/heapex-sm64.clif               |   2 +-
 cranelift/entity/src/lib.rs                   |   2 +-
 cranelift/entity/src/set.rs                   |  12 +-
 cranelift/faerie/src/backend.rs               |   2 +-
 cranelift/filetests/filetests/cfg/loop.clif   |  40 +-
 .../filetests/filetests/cfg/traps_early.clif  |  12 +-
 .../filetests/filetests/cfg/unused_node.clif  |  20 +-
 cranelift/filetests/filetests/dce/basic.clif  |  24 +-
 .../filetests/filetests/domtree/basic.clif    |  32 +-
 .../filetests/filetests/domtree/loops.clif    | 182 ++--
 .../filetests/filetests/domtree/loops2.clif   | 148 +--
 .../filetests/domtree/tall-tree.clif          |  88 +-
 .../filetests/domtree/wide-tree.clif          | 128 +--
 .../filetests/filetests/isa/riscv/abi-e.clif  |   2 +-
 .../filetests/filetests/isa/riscv/abi.clif    |   2 +-
 .../filetests/isa/riscv/binary32.clif         | 106 +-
 .../filetests/isa/riscv/encoding.clif         |   2 +-
 .../filetests/isa/riscv/expand-i32.clif       |   6 +-
 .../filetests/isa/riscv/legalize-abi.clif     |  56 +-
 .../filetests/isa/riscv/legalize-i64.clif     |  16 +-
 .../isa/riscv/legalize-icmp_imm-i64.clif      |  12 +-
 .../filetests/isa/riscv/parse-encoding.clif   |   2 +-
 .../filetests/isa/riscv/regmove.clif          |   2 +-
 .../filetests/isa/riscv/split-args.clif       |  50 +-
 .../filetests/isa/riscv/verify-encoding.clif  |   4 +-
 .../filetests/filetests/isa/x86/abcd.clif     |   2 +-
 .../filetests/filetests/isa/x86/abi-bool.clif |  12 +-
 .../filetests/filetests/isa/x86/abi32.clif    |   2 +-
 .../filetests/filetests/isa/x86/abi64.clif    |   8 +-
 .../isa/x86/allones_funcaddrs32.clif          |   2 +-
 .../isa/x86/allones_funcaddrs64.clif          |   2 +-
 .../isa/x86/baldrdash-table-sig-reg.clif      |   2 +-
 .../isa/x86/baseline_clz_ctz_popcount.clif    |  12 +-
 .../baseline_clz_ctz_popcount_encoding.clif   |   2 +-
 .../filetests/isa/x86/binary32-float.clif     |  74 +-
 .../filetests/filetests/isa/x86/binary32.clif | 168 +--
 .../filetests/isa/x86/binary64-float.clif     |  74 +-
 .../filetests/isa/x86/binary64-pic.clif       |   2 +-
 .../filetests/isa/x86/binary64-run.clif       |   2 +-
 .../filetests/filetests/isa/x86/binary64.clif | 244 ++---
 .../filetests/isa/x86/bitrev-i128-run.clif    |   6 +-
 .../filetests/isa/x86/br-i128-run.clif        |  20 +-
 .../filetests/filetests/isa/x86/br-i128.clif  |  32 +-
 .../filetests/isa/x86/brz-i8-run.clif         |  20 +-
 .../filetests/filetests/isa/x86/brz-i8.clif   |  24 +-
 .../filetests/isa/x86/compile-vconst.clif     |   4 +-
 .../filetests/isa/x86/extend-i128-run.clif    |   4 +-
 .../filetests/isa/x86/extend-i128.clif        |   4 +-
 .../isa/x86/extractlane-binemit.clif          |   8 +-
 .../filetests/isa/x86/extractlane-run.clif    |  10 +-
 .../floating-point-zero-constants-32bit.clif  |   4 +-
 .../x86/floating-point-zero-constants.clif    |   8 +-
 .../isa/x86/i128-isplit-forward-jump.clif     |  18 +-
 .../filetests/filetests/isa/x86/i128.clif     |  14 +-
 .../filetests/isa/x86/icmp-compile.clif       |   8 +-
 .../filetests/isa/x86/icmp-i128.clif          |   8 +-
 .../filetests/filetests/isa/x86/icmp-run.clif |   4 +-
 .../filetests/isa/x86/imul-i128.clif          |   2 +-
 .../filetests/isa/x86/insertlane-binemit.clif |   8 +-
 .../filetests/isa/x86/insertlane-run.clif     |   8 +-
 .../filetests/isa/x86/ireduce-i16-to-i8.clif  |   2 +-
 .../isa/x86/isplit-not-legalized-twice.clif   |   6 +-
 .../filetests/isa/x86/isub_imm-i8.clif        |   2 +-
 .../isa/x86/jump_i128_param_unused.clif       |   6 +-
 .../filetests/isa/x86/legalize-bint-i8.clif   |   2 +-
 .../filetests/isa/x86/legalize-bnot.clif      |   6 +-
 .../filetests/isa/x86/legalize-br-icmp.clif   |  36 +-
 .../filetests/isa/x86/legalize-br-table.clif  |  20 +-
 .../isa/x86/legalize-byte-ops-i8.clif         |   2 +-
 .../filetests/isa/x86/legalize-call.clif      |   2 +-
 .../isa/x86/legalize-clz-ctz-i8.clif          |   2 +-
 .../filetests/isa/x86/legalize-custom.clif    |  44 +-
 .../filetests/isa/x86/legalize-div-traps.clif |  70 +-
 .../filetests/isa/x86/legalize-div.clif       |  24 +-
 .../isa/x86/legalize-f64const-x64.clif        |   2 +-
 .../isa/x86/legalize-fcvt_from_usint-i16.clif |   4 +-
 .../filetests/isa/x86/legalize-heaps.clif     |  32 +-
 .../filetests/isa/x86/legalize-i128.clif      |   2 +-
 .../filetests/isa/x86/legalize-i64.clif       |  46 +-
 .../filetests/isa/x86/legalize-icmp-i8.clif   |   2 +-
 .../filetests/isa/x86/legalize-iconst-i8.clif |   6 +-
 .../filetests/isa/x86/legalize-imul-i8.clif   |   2 +-
 .../isa/x86/legalize-imul-imm-i8.clif         |   2 +-
 .../isa/x86/legalize-isplit-backwards.clif    |  12 +-
 .../filetests/isa/x86/legalize-libcall.clif   |   2 +-
 .../isa/x86/legalize-load-store-i8.clif       |   6 +-
 .../filetests/isa/x86/legalize-memory.clif    |  30 +-
 .../filetests/isa/x86/legalize-mulhi.clif     |   8 +-
 .../filetests/isa/x86/legalize-popcnt-i8.clif |   2 +-
 .../isa/x86/legalize-regmove-i8.clif          |  10 +-
 .../filetests/isa/x86/legalize-rotate.clif    |   8 +-
 .../filetests/isa/x86/legalize-shlr-i8.clif   |   2 +-
 .../filetests/isa/x86/legalize-splat.clif     |  16 +-
 .../filetests/isa/x86/legalize-tables.clif    |  20 +-
 .../filetests/isa/x86/legalize-urem-i8.clif   |   2 +-
 .../filetests/isa/x86/load-store-narrow.clif  |   4 +-
 .../filetests/filetests/isa/x86/nop.clif      |   2 +-
 .../x86/optimized-zero-constants-32bit.clif   |  10 +-
 .../isa/x86/optimized-zero-constants.clif     |  14 +-
 .../filetests/isa/x86/pinned-reg.clif         |   8 +-
 .../isa/x86/probestack-adjusts-sp.clif        |   4 +-
 .../isa/x86/probestack-disabled.clif          |   4 +-
 .../isa/x86/probestack-noncolocated.clif      |   4 +-
 .../filetests/isa/x86/probestack-size.clif    |  12 +-
 .../filetests/isa/x86/probestack.clif         |   8 +-
 .../filetests/isa/x86/prologue-epilogue.clif  |  40 +-
 .../filetests/filetests/isa/x86/pshufb.clif   |   2 +-
 .../filetests/filetests/isa/x86/pshufd.clif   |   2 +-
 .../filetests/isa/x86/raw_bitcast.clif        |   2 +-
 .../filetests/isa/x86/relax_branch.clif       |  56 +-
 .../filetests/isa/x86/run-const.clif          |   2 +-
 .../filetests/filetests/isa/x86/run-i64.clif  |   4 +-
 .../isa/x86/saturating-float-cast.clif        |   2 +-
 .../isa/x86/scalar_to_vector-binemit.clif     |   8 +-
 .../isa/x86/scalar_to_vector-compile.clif     |   4 +-
 .../filetests/isa/x86/select-i8.clif          |   2 +-
 .../isa/x86/shrink-multiple-uses.clif         |  10 +-
 .../filetests/filetests/isa/x86/shrink.clif   |   6 +-
 .../filetests/isa/x86/shuffle-legalize.clif   |   4 +-
 .../filetests/isa/x86/shuffle-run.clif        |   8 +-
 .../isa/x86/simd-arithmetic-binemit.clif      |  32 +-
 .../isa/x86/simd-arithmetic-legalize.clif     |   8 +-
 .../isa/x86/simd-arithmetic-run.clif          |  34 +-
 .../isa/x86/simd-bitwise-binemit.clif         |  34 +-
 .../isa/x86/simd-bitwise-legalize.clif        |   8 +-
 .../filetests/isa/x86/simd-bitwise-run.clif   |  20 +-
 .../isa/x86/simd-comparison-binemit.clif      |  18 +-
 .../isa/x86/simd-comparison-legalize.clif     |   8 +-
 .../isa/x86/simd-comparison-run.clif          |  46 +-
 .../isa/x86/simd-construction-run.clif        |   2 +-
 .../isa/x86/simd-logical-binemit.clif         |  10 +-
 .../isa/x86/simd-logical-legalize.clif        |   6 +-
 .../isa/x86/simd-logical-rodata.clif          |   2 +-
 .../filetests/isa/x86/simd-logical-run.clif   |  12 +-
 .../filetests/isa/x86/stack-addr64.clif       |   2 +-
 .../filetests/isa/x86/stack-load-store64.clif |   2 +-
 .../filetests/isa/x86/stack-load-store8.clif  |   2 +-
 .../filetests/isa/x86/uextend-i8-to-i16.clif  |   4 +-
 .../filetests/isa/x86/vconst-binemit.clif     |   2 +-
 .../filetests/isa/x86/vconst-opt-run.clif     |   4 +-
 .../filetests/isa/x86/vconst-opt.clif         |   2 +-
 .../filetests/isa/x86/vconst-rodata.clif      |   4 +-
 .../filetests/isa/x86/vconst-run.clif         |   2 +-
 .../isa/x86/windows_fastcall_x64.clif         |  12 +-
 .../isa/x86/windows_fastcall_x64_unwind.clif  |  10 +-
 .../isa/x86/windows_systemv_x64_fde.clif      |   4 +-
 .../filetests/legalizer/bitrev-i128.clif      |   4 +-
 .../filetests/filetests/legalizer/bitrev.clif |   8 +-
 .../filetests/legalizer/br_table_cond.clif    |  56 +-
 .../filetests/legalizer/bxor_imm.clif         |   2 +-
 .../filetests/legalizer/empty_br_table.clif   |  10 +-
 .../filetests/legalizer/iconst-i64.clif       |   2 +-
 .../filetests/legalizer/isplit-bb.clif        |  18 +-
 .../filetests/legalizer/popcnt-i128.clif      |   2 +-
 cranelift/filetests/filetests/licm/basic.clif |  32 +-
 .../filetests/filetests/licm/complex.clif     |  84 +-
 .../filetests/licm/critical-edge.clif         |  44 +-
 .../filetests/filetests/licm/encoding.clif    |  32 +-
 .../filetests/licm/jump-table-entry.clif      |  28 +-
 .../filetests/licm/load_readonly_notrap.clif  |  32 +-
 .../filetests/licm/multiple-blocks.clif       |  52 +-
 .../filetests/licm/nested_loops.clif          |  52 +-
 .../filetests/filetests/licm/reject.clif      |  72 +-
 .../filetests/licm/reject_load_notrap.clif    |  32 +-
 .../filetests/licm/reject_load_readonly.clif  |  32 +-
 .../filetests/filetests/parser/alias.clif     |   6 +-
 .../filetests/filetests/parser/branch.clif    | 108 +-
 .../filetests/filetests/parser/call.clif      |  18 +-
 .../filetests/filetests/parser/flags.clif     |  56 +-
 .../parser/instruction_encoding.clif          |   4 +-
 .../filetests/filetests/parser/memory.clif    |  12 +-
 .../filetests/filetests/parser/rewrite.clif   |  14 +-
 .../filetests/filetests/parser/ternary.clif   |   4 +-
 .../filetests/filetests/parser/tiny.clif      |  60 +-
 .../filetests/filetests/postopt/basic.clif    |  80 +-
 .../filetests/postopt/complex_memory_ops.clif |  16 +-
 .../postopt/fold_offset_into_address.clif     |   8 +-
 .../filetests/filetests/preopt/branch.clif    |  56 +-
 .../filetests/preopt/constant_fold.clif       |   4 +-
 .../filetests/filetests/preopt/numerical.clif |   8 +-
 .../filetests/filetests/regalloc/aliases.clif |  20 +-
 .../filetests/filetests/regalloc/basic.clif   |  48 +-
 .../filetests/regalloc/coalesce.clif          | 108 +-
 .../filetests/regalloc/coalescing-207.clif    | 894 ++++++++--------
 .../filetests/regalloc/coalescing-216.clif    |  94 +-
 .../filetests/regalloc/coloring-227.clif      |  98 +-
 .../filetests/regalloc/constraints.clif       |  10 +-
 .../regalloc/fallthrough-return.clif          |   4 +-
 .../filetests/regalloc/ghost-param.clif       |  40 +-
 .../regalloc/global-constraints.clif          |  10 +-
 .../filetests/regalloc/global-fixed.clif      |  12 +-
 .../regalloc/gpr-deref-safe-335.clif          |  30 +-
 .../regalloc/infinite-interference.clif       |  24 +-
 .../filetests/filetests/regalloc/iterate.clif |  96 +-
 .../filetests/regalloc/multi-constraints.clif |  14 +-
 .../filetests/regalloc/multiple-returns.clif  |   4 +-
 .../regalloc/output-interference.clif         |   2 +-
 .../filetests/regalloc/reload-208.clif        |  68 +-
 .../filetests/regalloc/reload-779.clif        |   8 +-
 .../filetests/filetests/regalloc/reload.clif  |   6 +-
 .../filetests/regalloc/schedule-moves.clif    |   4 +-
 .../regalloc/solver-fixedconflict-var-2.clif  |  22 +-
 .../regalloc/solver-fixedconflict-var-3.clif  |  54 +-
 .../regalloc/solver-fixedconflict-var.clif    |  50 +-
 .../filetests/regalloc/spill-noregs.clif      | 162 +--
 .../filetests/filetests/regalloc/spill.clif   |  40 +-
 .../filetests/regalloc/unreachable_code.clif  |  38 +-
 .../filetests/regalloc/x86-regres.clif        |  28 +-
 .../regress/allow-relaxation-shrink.clif      |  24 +-
 .../filetests/filetests/safepoint/basic.clif  |  68 +-
 .../filetests/filetests/safepoint/call.clif   |  20 +-
 .../filetests/filetests/simple_gvn/basic.clif |  24 +-
 .../filetests/simple_gvn/readonly.clif        |   2 +-
 .../filetests/simple_gvn/reject.clif          |   8 +-
 .../filetests/simple_gvn/scopes.clif          |  30 +-
 .../filetests/simple_preopt/branch.clif       |  60 +-
 .../simple_preopt/div_by_const_indirect.clif  |   8 +-
 .../div_by_const_non_power_of_2.clif          |  40 +-
 .../div_by_const_power_of_2.clif              |  56 +-
 .../fold-extended-move-wraparound.clif        |   2 +-
 .../rem_by_const_non_power_of_2.clif          |  40 +-
 .../rem_by_const_power_of_2.clif              |  56 +-
 .../filetests/simple_preopt/simplify32.clif   |  16 +-
 .../filetests/simple_preopt/simplify64.clif   |  82 +-
 .../filetests/verifier/bad_layout.clif        |  16 +-
 .../filetests/filetests/verifier/bitcast.clif |   6 +-
 .../verifier/defs_dominates_uses.clif         |   4 +-
 .../filetests/filetests/verifier/flags.clif   |  38 +-
 .../filetests/filetests/verifier/globals.clif |   4 +-
 .../filetests/filetests/verifier/heap.clif    |  10 +-
 .../filetests/verifier/jump_table.clif        |  16 +-
 .../filetests/filetests/verifier/memory.clif  |   4 +-
 .../filetests/verifier/scalar-to-vector.clif  |   2 +-
 .../filetests/verifier/simd-lane-index.clif   |  10 +-
 .../filetests/filetests/verifier/table.clif   |  10 +-
 .../filetests/verifier/type_check.clif        |  50 +-
 .../filetests/verifier/undeclared_vmctx.clif  |   4 +-
 .../filetests/verifier/unreachable_code.clif  |  28 +-
 .../filetests/filetests/wasm/control.clif     |  48 +-
 .../filetests/filetests/wasm/conversions.clif |  66 +-
 .../filetests/filetests/wasm/f32-arith.clif   |  30 +-
 .../filetests/wasm/f32-compares.clif          |  12 +-
 .../filetests/wasm/f32-memory64.clif          |   4 +-
 .../filetests/filetests/wasm/f64-arith.clif   |  30 +-
 .../filetests/wasm/f64-compares.clif          |  12 +-
 .../filetests/wasm/f64-memory64.clif          |   4 +-
 .../filetests/filetests/wasm/i32-arith.clif   |  38 +-
 .../filetests/wasm/i32-compares.clif          |  22 +-
 .../filetests/wasm/i32-memory64.clif          |  16 +-
 .../filetests/filetests/wasm/i64-arith.clif   |  38 +-
 .../filetests/wasm/i64-compares.clif          |  22 +-
 .../filetests/wasm/i64-memory64.clif          |  22 +-
 .../filetests/wasm/multi-val-b1.clif          |   8 +-
 .../wasm/multi-val-call-indirect.clif         |   2 +-
 .../filetests/wasm/multi-val-f32.clif         |   8 +-
 .../filetests/wasm/multi-val-f64.clif         |   8 +-
 .../filetests/wasm/multi-val-i32.clif         |   8 +-
 .../filetests/wasm/multi-val-i64.clif         |   8 +-
 .../filetests/wasm/multi-val-mixed.clif       | 488 ++++-----
 .../multi-val-reuse-ret-ptr-stack-slot.clif   |   2 +-
 .../wasm/multi-val-sret-slot-alignment.clif   |   6 +-
 .../multi-val-take-many-and-return-many.clif  |   4 +-
 .../wasm/multi-val-tons-of-results.clif       |   4 +-
 cranelift/filetests/filetests/wasm/r32.clif   |  14 +-
 cranelift/filetests/filetests/wasm/r64.clif   |  14 +-
 .../filetests/filetests/wasm/select.clif      |   8 +-
 cranelift/filetests/src/function_runner.rs    |   2 +-
 cranelift/filetests/src/match_directive.rs    |   2 +-
 cranelift/filetests/src/test_binemit.rs       |  22 +-
 cranelift/filetests/src/test_compile.rs       |   2 +-
 cranelift/filetests/src/test_domtree.rs       |  46 +-
 cranelift/filetests/src/test_rodata.rs        |   2 +-
 cranelift/filetests/src/test_verifier.rs      |   4 +-
 cranelift/frontend/src/frontend.rs            | 275 +++--
 cranelift/frontend/src/lib.rs                 |  12 +-
 cranelift/frontend/src/ssa.rs                 | 983 +++++++++---------
 cranelift/frontend/src/switch.rs              | 243 ++---
 cranelift/object/src/backend.rs               |   2 +-
 cranelift/preopt/src/constant_folding.rs      |   8 +-
 cranelift/reader/src/lexer.rs                 |  20 +-
 cranelift/reader/src/parser.rs                | 228 ++--
 cranelift/reader/src/sourcemap.rs             |  26 +-
 cranelift/reader/src/testfile.rs              |   2 +-
 cranelift/serde/src/serde_clif_json.rs        |  46 +-
 .../simplejit/examples/simplejit-minimal.rs   |  12 +-
 cranelift/simplejit/src/backend.rs            |   2 +-
 cranelift/simplejit/tests/basic.rs            |  20 +-
 cranelift/src/bugpoint.rs                     | 326 +++---
 cranelift/src/disasm.rs                       |   9 +-
 cranelift/src/run.rs                          |   2 +-
 cranelift/tests/bugpoint_test.clif            | 672 ++++++------
 cranelift/tests/bugpoint_test_expected.clif   |  14 +-
 cranelift/umbrella/src/lib.rs                 |   2 +-
 cranelift/wasm/src/code_translator.rs         | 160 +--
 cranelift/wasm/src/func_translator.rs         |  20 +-
 cranelift/wasm/src/state/func_state.rs        |  30 +-
 cranelift/wasm/src/translation_utils.rs       |  24 +-
 370 files changed, 8090 insertions(+), 7988 deletions(-)

diff --git a/cranelift/bforest/src/lib.rs b/cranelift/bforest/src/lib.rs
index 26e5b8efcf..bc79ffc7d0 100644
--- a/cranelift/bforest/src/lib.rs
+++ b/cranelift/bforest/src/lib.rs
@@ -148,22 +148,22 @@ mod tests {
     use super::*;
     use crate::entity::EntityRef;
 
-    /// An opaque reference to an extended basic block in a function.
+    /// An opaque reference to a basic block in a function.
     #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
-    pub struct Ebb(u32);
-    entity_impl!(Ebb, "ebb");
+    pub struct Block(u32);
+    entity_impl!(Block, "block");
 
     #[test]
     fn comparator() {
-        let ebb1 = Ebb::new(1);
-        let ebb2 = Ebb::new(2);
-        let ebb3 = Ebb::new(3);
-        let ebb4 = Ebb::new(4);
-        let vals = [ebb1, ebb2, ebb4];
+        let block1 = Block::new(1);
+        let block2 = Block::new(2);
+        let block3 = Block::new(3);
+        let block4 = Block::new(4);
+        let vals = [block1, block2, block4];
         let comp = ();
-        assert_eq!(comp.search(ebb1, &vals), Ok(0));
-        assert_eq!(comp.search(ebb3, &vals), Err(2));
-        assert_eq!(comp.search(ebb4, &vals), Ok(2));
+        assert_eq!(comp.search(block1, &vals), Ok(0));
+        assert_eq!(comp.search(block3, &vals), Err(2));
+        assert_eq!(comp.search(block4, &vals), Ok(2));
     }
 
     #[test]
diff --git a/cranelift/codegen/meta/src/cdsl/ast.rs b/cranelift/codegen/meta/src/cdsl/ast.rs
index e4eb202be3..82cdbad762 100644
--- a/cranelift/codegen/meta/src/cdsl/ast.rs
+++ b/cranelift/codegen/meta/src/cdsl/ast.rs
@@ -708,8 +708,8 @@ macro_rules! def {
 }
 
 // Helper macro to define legalization recipes.
-macro_rules! ebb {
-    // An basic block definition, splitting the current block in 2.
+macro_rules! block {
+    // a basic block definition, splitting the current block in 2.
     ($block: ident) => {
         ExprBuilder::block($block).assign_to(Vec::new())
     };
diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index 10c7f54a7d..f17202eb1c 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -112,7 +112,7 @@ pub(crate) struct InstructionContent {
     /// Indices in operands_out of output operands that are values.
     pub value_results: Vec,
 
-    /// True for instructions that terminate the EBB.
+    /// True for instructions that terminate the block.
     pub is_terminator: bool,
     /// True for all branch or jump instructions.
     pub is_branch: bool,
diff --git a/cranelift/codegen/meta/src/gen_inst.rs b/cranelift/codegen/meta/src/gen_inst.rs
index ffadd22d57..af54257fea 100644
--- a/cranelift/codegen/meta/src/gen_inst.rs
+++ b/cranelift/codegen/meta/src/gen_inst.rs
@@ -450,7 +450,7 @@ fn gen_opcodes(all_inst: &AllInstructions, fmt: &mut Formatter) {
             all_inst,
             |inst| inst.is_terminator,
             "is_terminator",
-            "True for instructions that terminate the EBB",
+            "True for instructions that terminate the block",
             fmt,
         );
         gen_bool_accessor(
diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index ede525e8bc..c872ba32d6 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -474,7 +474,7 @@ fn gen_transform<'a>(
         // If we are adding some blocks, we need to recall the original block, such that we can
         // recompute it.
         if !transform.block_pool.is_empty() {
-            fmt.line("let orig_ebb = pos.current_ebb().unwrap();");
+            fmt.line("let orig_block = pos.current_block().unwrap();");
         }
 
         // If we're going to delete `inst`, we need to detach its results first so they can be
@@ -486,14 +486,14 @@ fn gen_transform<'a>(
         // Emit new block creation.
         for block in &transform.block_pool {
             let var = transform.var_pool.get(block.name);
-            fmtln!(fmt, "let {} = pos.func.dfg.make_ebb();", var.name);
+            fmtln!(fmt, "let {} = pos.func.dfg.make_block();", var.name);
         }
 
         // Emit the destination pattern.
         for &def_index in &transform.dst {
             if let Some(block) = transform.block_pool.get(def_index) {
                 let var = transform.var_pool.get(block.name);
-                fmtln!(fmt, "pos.insert_ebb({});", var.name);
+                fmtln!(fmt, "pos.insert_block({});", var.name);
             }
             emit_dst_inst(
                 transform.def_pool.get(def_index),
@@ -507,7 +507,7 @@ fn gen_transform<'a>(
         let def_next_index = transform.def_pool.next_index();
         if let Some(block) = transform.block_pool.get(def_next_index) {
             let var = transform.var_pool.get(block.name);
-            fmtln!(fmt, "pos.insert_ebb({});", var.name);
+            fmtln!(fmt, "pos.insert_block({});", var.name);
         }
 
         // Delete the original instruction if we didn't have an opportunity to replace it.
@@ -520,14 +520,14 @@ fn gen_transform<'a>(
             if transform.def_pool.get(transform.src).apply.inst.is_branch {
                 // A branch might have been legalized into multiple branches, so we need to recompute
                 // the cfg.
-                fmt.line("cfg.recompute_ebb(pos.func, pos.current_ebb().unwrap());");
+                fmt.line("cfg.recompute_block(pos.func, pos.current_block().unwrap());");
             }
         } else {
             // Update CFG for the new blocks.
-            fmt.line("cfg.recompute_ebb(pos.func, orig_ebb);");
+            fmt.line("cfg.recompute_block(pos.func, orig_block);");
             for block in &transform.block_pool {
                 let var = transform.var_pool.get(block.name);
-                fmtln!(fmt, "cfg.recompute_ebb(pos.func, {});", var.name);
+                fmtln!(fmt, "cfg.recompute_block(pos.func, {});", var.name);
             }
         }
 
diff --git a/cranelift/codegen/meta/src/shared/entities.rs b/cranelift/codegen/meta/src/shared/entities.rs
index 068987c344..c3f2bc0387 100644
--- a/cranelift/codegen/meta/src/shared/entities.rs
+++ b/cranelift/codegen/meta/src/shared/entities.rs
@@ -6,9 +6,9 @@ fn new(format_field_name: &'static str, rust_type: &'static str, doc: &'static s
 }
 
 pub(crate) struct EntityRefs {
-    /// A reference to an extended basic block in the same function.
+    /// A reference to a basic block in the same function.
     /// This is primarliy used in control flow instructions.
-    pub(crate) ebb: OperandKind,
+    pub(crate) block: OperandKind,
 
     /// A reference to a stack slot declared in the function preamble.
     pub(crate) stack_slot: OperandKind,
@@ -33,17 +33,17 @@ pub(crate) struct EntityRefs {
     /// A reference to a table declared in the function preamble.
     pub(crate) table: OperandKind,
 
-    /// A variable-sized list of value operands. Use for Ebb and function call arguments.
+    /// A variable-sized list of value operands. Use for Block and function call arguments.
     pub(crate) varargs: OperandKind,
 }
 
 impl EntityRefs {
     pub fn new() -> Self {
         Self {
-            ebb: new(
+            block: new(
                 "destination",
-                "ir::Ebb",
-                "An extended basic block in the same function.",
+                "ir::Block",
+                "a basic block in the same function.",
             ),
             stack_slot: new("stack_slot", "ir::StackSlot", "A stack slot"),
 
@@ -64,7 +64,7 @@ impl EntityRefs {
                         A variable size list of `value` operands.
 
                         Use this to represent arguments passed to a function call, arguments
-                        passed to an extended basic block, or a variable number of results
+                        passed to a basic block, or a variable number of results
                         returned from an instruction.
                     "#,
             ),
diff --git a/cranelift/codegen/meta/src/shared/formats.rs b/cranelift/codegen/meta/src/shared/formats.rs
index 3cc3d343d7..03c09e2e2b 100644
--- a/cranelift/codegen/meta/src/shared/formats.rs
+++ b/cranelift/codegen/meta/src/shared/formats.rs
@@ -140,25 +140,25 @@ impl Formats {
                 .value()
                 .build(),
 
-            jump: Builder::new("Jump").imm(&entities.ebb).varargs().build(),
+            jump: Builder::new("Jump").imm(&entities.block).varargs().build(),
 
             branch: Builder::new("Branch")
                 .value()
-                .imm(&entities.ebb)
+                .imm(&entities.block)
                 .varargs()
                 .build(),
 
             branch_int: Builder::new("BranchInt")
                 .imm(&imm.intcc)
                 .value()
-                .imm(&entities.ebb)
+                .imm(&entities.block)
                 .varargs()
                 .build(),
 
             branch_float: Builder::new("BranchFloat")
                 .imm(&imm.floatcc)
                 .value()
-                .imm(&entities.ebb)
+                .imm(&entities.block)
                 .varargs()
                 .build(),
 
@@ -166,13 +166,13 @@ impl Formats {
                 .imm(&imm.intcc)
                 .value()
                 .value()
-                .imm(&entities.ebb)
+                .imm(&entities.block)
                 .varargs()
                 .build(),
 
             branch_table: Builder::new("BranchTable")
                 .value()
-                .imm(&entities.ebb)
+                .imm(&entities.block)
                 .imm(&entities.jump_table)
                 .build(),
 
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 3207dad96e..08b4f66a7a 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -18,8 +18,8 @@ fn define_control_flow(
     imm: &Immediates,
     entities: &EntityRefs,
 ) {
-    let EBB = &Operand::new("EBB", &entities.ebb).with_doc("Destination extended basic block");
-    let args = &Operand::new("args", &entities.varargs).with_doc("EBB arguments");
+    let block = &Operand::new("block", &entities.block).with_doc("Destination basic block");
+    let args = &Operand::new("args", &entities.varargs).with_doc("block arguments");
 
     ig.push(
         Inst::new(
@@ -27,13 +27,13 @@ fn define_control_flow(
             r#"
         Jump.
 
-        Unconditionally jump to an extended basic block, passing the specified
-        EBB arguments. The number and types of arguments must match the
-        destination EBB.
+        Unconditionally jump to a basic block, passing the specified
+        block arguments. The number and types of arguments must match the
+        destination block.
         "#,
             &formats.jump,
         )
-        .operands_in(vec![EBB, args])
+        .operands_in(vec![block, args])
         .is_terminator(true)
         .is_branch(true),
     );
@@ -42,9 +42,9 @@ fn define_control_flow(
         Inst::new(
             "fallthrough",
             r#"
-        Fall through to the next EBB.
+        Fall through to the next block.
 
-        This is the same as `jump`, except the destination EBB must be
+        This is the same as `jump`, except the destination block must be
         the next one in the layout.
 
         Jumps are turned into fall-through instructions by the branch
@@ -53,7 +53,7 @@ fn define_control_flow(
         "#,
             &formats.jump,
         )
-        .operands_in(vec![EBB, args])
+        .operands_in(vec![block, args])
         .is_terminator(true)
         .is_branch(true),
     );
@@ -81,7 +81,7 @@ fn define_control_flow(
         "#,
                 &formats.branch,
             )
-            .operands_in(vec![c, EBB, args])
+            .operands_in(vec![c, block, args])
             .is_branch(true),
         );
 
@@ -96,7 +96,7 @@ fn define_control_flow(
         "#,
                 &formats.branch,
             )
-            .operands_in(vec![c, EBB, args])
+            .operands_in(vec![c, block, args])
             .is_branch(true),
         );
     }
@@ -124,14 +124,14 @@ fn define_control_flow(
         and take the branch if the condition is true:
 
         ```text
-            br_icmp ugt v1, v2, ebb4(v5, v6)
+            br_icmp ugt v1, v2, block4(v5, v6)
         ```
 
         is semantically equivalent to:
 
         ```text
             v10 = icmp ugt, v1, v2
-            brnz v10, ebb4(v5, v6)
+            brnz v10, block4(v5, v6)
         ```
 
         Some RISC architectures like MIPS and RISC-V provide instructions that
@@ -140,7 +140,7 @@ fn define_control_flow(
         "#,
                 &formats.branch_icmp,
             )
-            .operands_in(vec![Cond, x, y, EBB, args])
+            .operands_in(vec![Cond, x, y, block, args])
             .is_branch(true),
         );
 
@@ -154,7 +154,7 @@ fn define_control_flow(
         "#,
                 &formats.branch_int,
             )
-            .operands_in(vec![Cond, f, EBB, args])
+            .operands_in(vec![Cond, f, block, args])
             .is_branch(true),
         );
     }
@@ -172,7 +172,7 @@ fn define_control_flow(
         "#,
                 &formats.branch_float,
             )
-            .operands_in(vec![Cond, f, EBB, args])
+            .operands_in(vec![Cond, f, block, args])
             .is_branch(true),
         );
     }
@@ -188,8 +188,8 @@ fn define_control_flow(
         Indirect branch via jump table.
 
         Use ``x`` as an unsigned index into the jump table ``JT``. If a jump
-        table entry is found, branch to the corresponding EBB. If no entry was
-        found or the index is out-of-bounds, branch to the given default EBB.
+        table entry is found, branch to the corresponding block. If no entry was
+        found or the index is out-of-bounds, branch to the given default block.
 
         Note that this branch instruction can't pass arguments to the targeted
         blocks. Split critical edges as needed to work around this.
@@ -202,7 +202,7 @@ fn define_control_flow(
         "#,
                 &formats.branch_table,
             )
-            .operands_in(vec![x, EBB, JT])
+            .operands_in(vec![x, block, JT])
             .is_terminator(true)
             .is_branch(true),
         );
@@ -1407,7 +1407,7 @@ pub(crate) fn define(
         satisfy instruction constraints.
 
         The register diversions created by this instruction must be undone
-        before the value leaves the EBB. At the entry to a new EBB, all live
+        before the value leaves the block. At the entry to a new block, all live
         values must be in their originally assigned registers.
         "#,
             &formats.reg_move,
diff --git a/cranelift/codegen/meta/src/shared/legalize.rs b/cranelift/codegen/meta/src/shared/legalize.rs
index 482f5c69c6..1b37f9661b 100644
--- a/cranelift/codegen/meta/src/shared/legalize.rs
+++ b/cranelift/codegen/meta/src/shared/legalize.rs
@@ -197,9 +197,9 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     let al = var("al");
     let ah = var("ah");
     let cc = var("cc");
-    let ebb = var("ebb");
-    let ebb1 = var("ebb1");
-    let ebb2 = var("ebb2");
+    let block = var("block");
+    let block1 = var("block1");
+    let block2 = var("block2");
     let ptr = var("ptr");
     let flags = var("flags");
     let offset = var("off");
@@ -269,7 +269,7 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
     );
 
     narrow.legalize(
-        def!(brz.I128(x, ebb, vararg)),
+        def!(brz.I128(x, block, vararg)),
         vec![
             def!((xl, xh) = isplit(x)),
             def!(
@@ -287,18 +287,18 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
                 )
             ),
             def!(c = band(a, b)),
-            def!(brnz(c, ebb, vararg)),
+            def!(brnz(c, block, vararg)),
         ],
     );
 
     narrow.legalize(
-        def!(brnz.I128(x, ebb1, vararg)),
+        def!(brnz.I128(x, block1, vararg)),
         vec![
             def!((xl, xh) = isplit(x)),
-            def!(brnz(xl, ebb1, vararg)),
-            def!(jump(ebb2, Literal::empty_vararg())),
-            ebb!(ebb2),
-            def!(brnz(xh, ebb1, vararg)),
+            def!(brnz(xl, block1, vararg)),
+            def!(jump(block2, Literal::empty_vararg())),
+            block!(block2),
+            def!(brnz(xh, block1, vararg)),
         ],
     );
 
@@ -619,13 +619,13 @@ pub(crate) fn define(insts: &InstructionGroup, imm: &Immediates) -> TransformGro
 
     for &ty in &[I8, I16] {
         widen.legalize(
-            def!(brz.ty(x, ebb, vararg)),
-            vec![def!(a = uextend.I32(x)), def!(brz(a, ebb, vararg))],
+            def!(brz.ty(x, block, vararg)),
+            vec![def!(a = uextend.I32(x)), def!(brz(a, block, vararg))],
         );
 
         widen.legalize(
-            def!(brnz.ty(x, ebb, vararg)),
-            vec![def!(a = uextend.I32(x)), def!(brnz(a, ebb, vararg))],
+            def!(brnz.ty(x, block, vararg)),
+            vec![def!(a = uextend.I32(x)), def!(brnz(a, block, vararg))],
         );
     }
 
diff --git a/cranelift/codegen/src/abi.rs b/cranelift/codegen/src/abi.rs
index f3591c1730..3a3ed7a53b 100644
--- a/cranelift/codegen/src/abi.rs
+++ b/cranelift/codegen/src/abi.rs
@@ -135,7 +135,7 @@ pub fn legalize_args(args: &[AbiParam], aa: &mut AA) -> Option<
 ///
 /// The legalizer needs to repair the values at all ABI boundaries:
 ///
-/// - Incoming function arguments to the entry EBB.
+/// - Incoming function arguments to the entry block.
 /// - Function arguments passed to a call.
 /// - Return values from a call.
 /// - Return values passed to a return instruction.
diff --git a/cranelift/codegen/src/binemit/memorysink.rs b/cranelift/codegen/src/binemit/memorysink.rs
index 7e1cdf57a2..60c7fe251e 100644
--- a/cranelift/codegen/src/binemit/memorysink.rs
+++ b/cranelift/codegen/src/binemit/memorysink.rs
@@ -74,8 +74,8 @@ impl<'a> MemoryCodeSink<'a> {
 
 /// A trait for receiving relocations for code that is emitted directly into memory.
 pub trait RelocSink {
-    /// Add a relocation referencing an EBB at the current offset.
-    fn reloc_ebb(&mut self, _: CodeOffset, _: Reloc, _: CodeOffset);
+    /// Add a relocation referencing an block at the current offset.
+    fn reloc_block(&mut self, _: CodeOffset, _: Reloc, _: CodeOffset);
 
     /// Add a relocation referencing an external symbol at the current offset.
     fn reloc_external(&mut self, _: CodeOffset, _: Reloc, _: &ExternalName, _: Addend);
@@ -127,9 +127,9 @@ impl<'a> CodeSink for MemoryCodeSink<'a> {
         self.write(x);
     }
 
-    fn reloc_ebb(&mut self, rel: Reloc, ebb_offset: CodeOffset) {
+    fn reloc_block(&mut self, rel: Reloc, block_offset: CodeOffset) {
         let ofs = self.offset();
-        self.relocs.reloc_ebb(ofs, rel, ebb_offset);
+        self.relocs.reloc_block(ofs, rel, block_offset);
     }
 
     fn reloc_external(&mut self, rel: Reloc, name: &ExternalName, addend: Addend) {
@@ -177,7 +177,7 @@ impl<'a> CodeSink for MemoryCodeSink<'a> {
 pub struct NullRelocSink {}
 
 impl RelocSink for NullRelocSink {
-    fn reloc_ebb(&mut self, _: u32, _: Reloc, _: u32) {}
+    fn reloc_block(&mut self, _: u32, _: Reloc, _: u32) {}
     fn reloc_external(&mut self, _: u32, _: Reloc, _: &ExternalName, _: i64) {}
     fn reloc_constant(&mut self, _: CodeOffset, _: Reloc, _: ConstantOffset) {}
     fn reloc_jt(&mut self, _: u32, _: Reloc, _: JumpTable) {}
diff --git a/cranelift/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
index 450bcd33d6..7a781e2b56 100644
--- a/cranelift/codegen/src/binemit/mod.rs
+++ b/cranelift/codegen/src/binemit/mod.rs
@@ -127,8 +127,8 @@ pub trait CodeSink {
     /// Add 8 bytes to the code section.
     fn put8(&mut self, _: u64);
 
-    /// Add a relocation referencing an EBB at the current offset.
-    fn reloc_ebb(&mut self, _: Reloc, _: CodeOffset);
+    /// Add a relocation referencing an block at the current offset.
+    fn reloc_block(&mut self, _: Reloc, _: CodeOffset);
 
     /// Add a relocation referencing an external symbol plus the addend at the current offset.
     fn reloc_external(&mut self, _: Reloc, _: &ExternalName, _: Addend);
@@ -205,10 +205,10 @@ where
     EI: Fn(&Function, Inst, &mut RegDiversions, &mut CS, &dyn TargetIsa),
 {
     let mut divert = RegDiversions::new();
-    for ebb in func.layout.ebbs() {
-        divert.at_ebb(&func.entry_diversions, ebb);
-        debug_assert_eq!(func.offsets[ebb], sink.offset());
-        for inst in func.layout.ebb_insts(ebb) {
+    for block in func.layout.blocks() {
+        divert.at_block(&func.entry_diversions, block);
+        debug_assert_eq!(func.offsets[block], sink.offset());
+        for inst in func.layout.block_insts(block) {
             emit_inst(func, inst, &mut divert, sink, isa);
         }
     }
@@ -218,8 +218,8 @@ where
     // Output jump tables.
     for (jt, jt_data) in func.jump_tables.iter() {
         let jt_offset = func.jt_offsets[jt];
-        for ebb in jt_data.iter() {
-            let rel_offset: i32 = func.offsets[*ebb] as i32 - jt_offset as i32;
+        for block in jt_data.iter() {
+            let rel_offset: i32 = func.offsets[*block] as i32 - jt_offset as i32;
             sink.put4(rel_offset as u32)
         }
     }
diff --git a/cranelift/codegen/src/binemit/relaxation.rs b/cranelift/codegen/src/binemit/relaxation.rs
index 3cd8b68de0..abdd778aaf 100644
--- a/cranelift/codegen/src/binemit/relaxation.rs
+++ b/cranelift/codegen/src/binemit/relaxation.rs
@@ -1,9 +1,9 @@
 //! Branch relaxation and offset computation.
 //!
-//! # EBB header offsets
+//! # block header offsets
 //!
 //! Before we can generate binary machine code for branch instructions, we need to know the final
-//! offsets of all the EBB headers in the function. This information is encoded in the
+//! offsets of all the block headers in the function. This information is encoded in the
 //! `func.offsets` table.
 //!
 //! # Branch relaxation
@@ -16,22 +16,22 @@
 //! unconditional branches:
 //!
 //! ```clif
-//!     brz v1, ebb17
+//!     brz v1, block17
 //! ```
 //!
 //! can be transformed into:
 //!
 //! ```clif
-//!     brnz v1, ebb23
-//!     jump ebb17
-//! ebb23:
+//!     brnz v1, block23
+//!     jump block17
+//! block23:
 //! ```
 
 use crate::binemit::{CodeInfo, CodeOffset};
 use crate::cursor::{Cursor, FuncCursor};
 use crate::dominator_tree::DominatorTree;
 use crate::flowgraph::ControlFlowGraph;
-use crate::ir::{Ebb, Function, Inst, InstructionData, Opcode, Value, ValueList};
+use crate::ir::{Block, Function, Inst, InstructionData, Opcode, Value, ValueList};
 use crate::isa::{EncInfo, TargetIsa};
 use crate::iterators::IteratorExtras;
 use crate::regalloc::RegDiversions;
@@ -40,7 +40,7 @@ use crate::CodegenResult;
 use core::convert::TryFrom;
 use log::debug;
 
-/// Relax branches and compute the final layout of EBB headers in `func`.
+/// Relax branches and compute the final layout of block headers in `func`.
 ///
 /// Fill in the `func.offsets` table so the function is ready for binary emission.
 pub fn relax_branches(
@@ -53,9 +53,9 @@ pub fn relax_branches(
 
     let encinfo = isa.encoding_info();
 
-    // Clear all offsets so we can recognize EBBs that haven't been visited yet.
+    // Clear all offsets so we can recognize blocks that haven't been visited yet.
     func.offsets.clear();
-    func.offsets.resize(func.dfg.num_ebbs());
+    func.offsets.resize(func.dfg.num_blocks());
 
     // Start by removing redundant jumps.
     fold_redundant_jumps(func, _cfg, _domtree);
@@ -66,12 +66,12 @@ pub fn relax_branches(
     let mut offset = 0;
     let mut divert = RegDiversions::new();
 
-    // First, compute initial offsets for every EBB.
+    // First, compute initial offsets for every block.
     {
         let mut cur = FuncCursor::new(func);
-        while let Some(ebb) = cur.next_ebb() {
-            divert.at_ebb(&cur.func.entry_diversions, ebb);
-            cur.func.offsets[ebb] = offset;
+        while let Some(block) = cur.next_block() {
+            divert.at_block(&cur.func.entry_diversions, block);
+            cur.func.offsets[block] = offset;
             while let Some(inst) = cur.next_inst() {
                 divert.apply(&cur.func.dfg[inst]);
                 let enc = cur.func.encodings[inst];
@@ -88,12 +88,12 @@ pub fn relax_branches(
 
         // Visit all instructions in layout order.
         let mut cur = FuncCursor::new(func);
-        while let Some(ebb) = cur.next_ebb() {
-            divert.at_ebb(&cur.func.entry_diversions, ebb);
+        while let Some(block) = cur.next_block() {
+            divert.at_block(&cur.func.entry_diversions, block);
 
-            // Record the offset for `ebb` and make sure we iterate until offsets are stable.
-            if cur.func.offsets[ebb] != offset {
-                cur.func.offsets[ebb] = offset;
+            // Record the offset for `block` and make sure we iterate until offsets are stable.
+            if cur.func.offsets[block] != offset {
+                cur.func.offsets[block] = offset;
                 go_again = true;
             }
 
@@ -153,21 +153,21 @@ pub fn relax_branches(
 fn try_fold_redundant_jump(
     func: &mut Function,
     cfg: &mut ControlFlowGraph,
-    ebb: Ebb,
+    block: Block,
     first_inst: Inst,
 ) -> bool {
     let first_dest = match func.dfg[first_inst].branch_destination() {
-        Some(ebb) => ebb, // The instruction was a single-target branch.
+        Some(block) => block, // The instruction was a single-target branch.
         None => {
             return false; // The instruction was either multi-target or not a branch.
         }
     };
 
-    // For the moment, only attempt to fold a branch to an ebb that is parameterless.
+    // For the moment, only attempt to fold a branch to an block that is parameterless.
     // These blocks are mainly produced by critical edge splitting.
     //
     // TODO: Allow folding blocks that define SSA values and function as phi nodes.
-    if func.dfg.num_ebb_params(first_dest) != 0 {
+    if func.dfg.num_block_params(first_dest) != 0 {
         return false;
     }
 
@@ -178,7 +178,7 @@ fn try_fold_redundant_jump(
         return false;
     }
 
-    // Now we need to fix up first_inst's ebb parameters to match second_inst's,
+    // Now we need to fix up first_inst's block parameters to match second_inst's,
     // without changing the branch-specific arguments.
     //
     // The intermediary block is allowed to reference any SSA value that dominates it,
@@ -208,14 +208,14 @@ fn try_fold_redundant_jump(
     // was a block parameter, rewrite it to refer to the value that the first jump
     // passed in its parameters. Otherwise, make sure it dominates first_inst.
     //
-    // For example: if we `ebb0: jump ebb1(v1)` to `ebb1(v2): jump ebb2(v2)`,
-    // we want to rewrite the original jump to `jump ebb2(v1)`.
-    let ebb_params: &[Value] = func.dfg.ebb_params(first_dest);
-    debug_assert!(ebb_params.len() == first_params.len());
+    // For example: if we `block0: jump block1(v1)` to `block1(v2): jump block2(v2)`,
+    // we want to rewrite the original jump to `jump block2(v1)`.
+    let block_params: &[Value] = func.dfg.block_params(first_dest);
+    debug_assert!(block_params.len() == first_params.len());
 
     for value in second_params.iter_mut() {
-        if let Some((n, _)) = ebb_params.iter().enumerate().find(|(_, &p)| p == *value) {
-            // This value was the Nth parameter passed to the second_inst's ebb.
+        if let Some((n, _)) = block_params.iter().enumerate().find(|(_, &p)| p == *value) {
+            // This value was the Nth parameter passed to the second_inst's block.
             // Rewrite it as the Nth parameter passed by first_inst.
             *value = first_params[n];
         }
@@ -233,21 +233,21 @@ fn try_fold_redundant_jump(
     func.dfg[first_inst].put_value_list(value_list); // Put the new list.
 
     // Bypass the second jump.
-    // This can disconnect the Ebb containing `second_inst`, to be cleaned up later.
+    // This can disconnect the Block containing `second_inst`, to be cleaned up later.
     let second_dest = func.dfg[second_inst].branch_destination().expect("Dest");
     func.change_branch_destination(first_inst, second_dest);
-    cfg.recompute_ebb(func, ebb);
+    cfg.recompute_block(func, block);
 
-    // The previously-intermediary Ebb may now be unreachable. Update CFG.
+    // The previously-intermediary Block may now be unreachable. Update CFG.
     if cfg.pred_iter(first_dest).count() == 0 {
-        // Remove all instructions from that ebb.
+        // Remove all instructions from that block.
         while let Some(inst) = func.layout.first_inst(first_dest) {
             func.layout.remove_inst(inst);
         }
 
         // Remove the block...
-        cfg.recompute_ebb(func, first_dest); // ...from predecessor lists.
-        func.layout.remove_ebb(first_dest); // ...from the layout.
+        cfg.recompute_block(func, first_dest); // ...from predecessor lists.
+        func.layout.remove_block(first_dest); // ...from the layout.
     }
 
     true
@@ -264,14 +264,17 @@ fn fold_redundant_jumps(
 
     // Postorder iteration guarantees that a chain of jumps is visited from
     // the end of the chain to the start of the chain.
-    for &ebb in domtree.cfg_postorder() {
+    for &block in domtree.cfg_postorder() {
         // Only proceed if the first terminator instruction is a single-target branch.
-        let first_inst = func.layout.last_inst(ebb).expect("Ebb has no terminator");
-        folded |= try_fold_redundant_jump(func, cfg, ebb, first_inst);
+        let first_inst = func
+            .layout
+            .last_inst(block)
+            .expect("Block has no terminator");
+        folded |= try_fold_redundant_jump(func, cfg, block, first_inst);
 
         // Also try the previous instruction.
         if let Some(prev_inst) = func.layout.prev_inst(first_inst) {
-            folded |= try_fold_redundant_jump(func, cfg, ebb, prev_inst);
+            folded |= try_fold_redundant_jump(func, cfg, block, prev_inst);
         }
     }
 
@@ -284,8 +287,11 @@ fn fold_redundant_jumps(
 /// Convert `jump` instructions to `fallthrough` instructions where possible and verify that any
 /// existing `fallthrough` instructions are correct.
 fn fallthroughs(func: &mut Function) {
-    for (ebb, succ) in func.layout.ebbs().adjacent_pairs() {
-        let term = func.layout.last_inst(ebb).expect("EBB has no terminator.");
+    for (block, succ) in func.layout.blocks().adjacent_pairs() {
+        let term = func
+            .layout
+            .last_inst(block)
+            .expect("block has no terminator.");
         if let InstructionData::Jump {
             ref mut opcode,
             destination,
@@ -296,10 +302,10 @@ fn fallthroughs(func: &mut Function) {
                 Opcode::Fallthrough => {
                     // Somebody used a fall-through instruction before the branch relaxation pass.
                     // Make sure it is correct, i.e. the destination is the layout successor.
-                    debug_assert_eq!(destination, succ, "Illegal fall-through in {}", ebb)
+                    debug_assert_eq!(destination, succ, "Illegal fall-through in {}", block)
                 }
                 Opcode::Jump => {
-                    // If this is a jump to the successor EBB, change it to a fall-through.
+                    // If this is a jump to the successor block, change it to a fall-through.
                     if destination == succ {
                         *opcode = Opcode::Fallthrough;
                         func.encodings[term] = Default::default();
@@ -368,18 +374,18 @@ fn relax_branch(
     // branches, so one way of extending the range of a conditional branch is to invert its
     // condition and make it branch over an unconditional jump which has the larger range.
     //
-    // Splitting the EBB is problematic this late because there may be register diversions in
+    // Splitting the block is problematic this late because there may be register diversions in
     // effect across the conditional branch, and they can't survive the control flow edge to a new
-    // EBB. We have two options for handling that:
+    // block. We have two options for handling that:
     //
-    // 1. Set a flag on the new EBB that indicates it wants the preserve the register diversions of
+    // 1. Set a flag on the new block that indicates it wants the preserve the register diversions of
     //    its layout predecessor, or
-    // 2. Use an encoding macro for the branch-over-jump pattern so we don't need to split the EBB.
+    // 2. Use an encoding macro for the branch-over-jump pattern so we don't need to split the block.
     //
     // It seems that 1. would allow us to share code among RISC ISAs that need this.
     //
     // We can't allow register diversions to survive from the layout predecessor because the layout
-    // predecessor could contain kill points for some values that are live in this EBB, and
+    // predecessor could contain kill points for some values that are live in this block, and
     // diversions are not automatically cancelled when the live range of a value ends.
 
     // This assumes solution 2. above:
diff --git a/cranelift/codegen/src/binemit/shrink.rs b/cranelift/codegen/src/binemit/shrink.rs
index 084ed2bc3d..f6fa43e062 100644
--- a/cranelift/codegen/src/binemit/shrink.rs
+++ b/cranelift/codegen/src/binemit/shrink.rs
@@ -19,11 +19,11 @@ pub fn shrink_instructions(func: &mut Function, isa: &dyn TargetIsa) {
     let encinfo = isa.encoding_info();
     let mut divert = RegDiversions::new();
 
-    for ebb in func.layout.ebbs() {
+    for block in func.layout.blocks() {
         // Load diversions from predecessors.
-        divert.at_ebb(&func.entry_diversions, ebb);
+        divert.at_block(&func.entry_diversions, block);
 
-        for inst in func.layout.ebb_insts(ebb) {
+        for inst in func.layout.block_insts(block) {
             let enc = func.encodings[inst];
             if enc.is_legal() {
                 // regmove/regfill/regspill are special instructions with register immediates
diff --git a/cranelift/codegen/src/cfg_printer.rs b/cranelift/codegen/src/cfg_printer.rs
index ed47475295..364b2985fe 100644
--- a/cranelift/codegen/src/cfg_printer.rs
+++ b/cranelift/codegen/src/cfg_printer.rs
@@ -4,7 +4,7 @@ use alloc::vec::Vec;
 use core::fmt::{Display, Formatter, Result, Write};
 
 use crate::entity::SecondaryMap;
-use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::flowgraph::{BlockPredecessor, ControlFlowGraph};
 use crate::ir::Function;
 use crate::write::{FuncWriter, PlainWriter};
 
@@ -27,7 +27,7 @@ impl<'a> CFGPrinter<'a> {
     /// Write the CFG for this function to `w`.
     pub fn write(&self, w: &mut dyn Write) -> Result {
         self.header(w)?;
-        self.ebb_nodes(w)?;
+        self.block_nodes(w)?;
         self.cfg_connections(w)?;
         writeln!(w, "}}")
     }
@@ -40,7 +40,7 @@ impl<'a> CFGPrinter<'a> {
         Ok(())
     }
 
-    fn ebb_nodes(&self, w: &mut dyn Write) -> Result {
+    fn block_nodes(&self, w: &mut dyn Write) -> Result {
         let mut aliases = SecondaryMap::<_, Vec<_>>::new();
         for v in self.func.dfg.values() {
             // VADFS returns the immediate target of an alias
@@ -49,11 +49,11 @@ impl<'a> CFGPrinter<'a> {
             }
         }
 
-        for ebb in &self.func.layout {
-            write!(w, "    {} [shape=record, label=\"{{", ebb)?;
-            crate::write::write_ebb_header(w, self.func, None, ebb, 4)?;
+        for block in &self.func.layout {
+            write!(w, "    {} [shape=record, label=\"{{", block)?;
+            crate::write::write_block_header(w, self.func, None, block, 4)?;
             // Add all outgoing branch instructions to the label.
-            for inst in self.func.layout.ebb_insts(ebb) {
+            for inst in self.func.layout.block_insts(block) {
                 write!(w, " | <{}>", inst)?;
                 PlainWriter.write_instruction(w, self.func, &aliases, None, inst, 0)?;
             }
@@ -63,9 +63,13 @@ impl<'a> CFGPrinter<'a> {
     }
 
     fn cfg_connections(&self, w: &mut dyn Write) -> Result {
-        for ebb in &self.func.layout {
-            for BasicBlock { ebb: parent, inst } in self.cfg.pred_iter(ebb) {
-                writeln!(w, "    {}:{} -> {}", parent, inst, ebb)?;
+        for block in &self.func.layout {
+            for BlockPredecessor {
+                block: parent,
+                inst,
+            } in self.cfg.pred_iter(block)
+            {
+                writeln!(w, "    {}:{} -> {}", parent, inst, block)?;
             }
         }
         Ok(())
diff --git a/cranelift/codegen/src/cursor.rs b/cranelift/codegen/src/cursor.rs
index e2ab8e5e5f..51345cde47 100644
--- a/cranelift/codegen/src/cursor.rs
+++ b/cranelift/codegen/src/cursor.rs
@@ -13,12 +13,12 @@ pub enum CursorPosition {
     /// Cursor is pointing at an existing instruction.
     /// New instructions will be inserted *before* the current instruction.
     At(ir::Inst),
-    /// Cursor is before the beginning of an EBB. No instructions can be inserted. Calling
-    /// `next_inst()` will move to the first instruction in the EBB.
-    Before(ir::Ebb),
-    /// Cursor is pointing after the end of an EBB.
-    /// New instructions will be appended to the EBB.
-    After(ir::Ebb),
+    /// Cursor is before the beginning of an block. No instructions can be inserted. Calling
+    /// `next_inst()` will move to the first instruction in the block.
+    Before(ir::Block),
+    /// Cursor is pointing after the end of an block.
+    /// New instructions will be appended to the block.
+    After(ir::Block),
 }
 
 /// All cursor types implement the `Cursor` which provides common navigation operations.
@@ -46,7 +46,7 @@ pub trait Cursor {
     /// This is intended to be used as a builder method:
     ///
     /// ```
-    /// # use cranelift_codegen::ir::{Function, Ebb, SourceLoc};
+    /// # use cranelift_codegen::ir::{Function, Block, SourceLoc};
     /// # use cranelift_codegen::cursor::{Cursor, FuncCursor};
     /// fn edit_func(func: &mut Function, srcloc: SourceLoc) {
     ///     let mut pos = FuncCursor::new(func).with_srcloc(srcloc);
@@ -76,7 +76,7 @@ pub trait Cursor {
     /// This is intended to be used as a builder method:
     ///
     /// ```
-    /// # use cranelift_codegen::ir::{Function, Ebb, Inst};
+    /// # use cranelift_codegen::ir::{Function, Block, Inst};
     /// # use cranelift_codegen::cursor::{Cursor, FuncCursor};
     /// fn edit_func(func: &mut Function, inst: Inst) {
     ///     let mut pos = FuncCursor::new(func).at_inst(inst);
@@ -92,68 +92,68 @@ pub trait Cursor {
         self
     }
 
-    /// Rebuild this cursor positioned at the first insertion point for `ebb`.
+    /// Rebuild this cursor positioned at the first insertion point for `block`.
     /// This differs from `at_first_inst` in that it doesn't assume that any
-    /// instructions have been inserted into `ebb` yet.
+    /// instructions have been inserted into `block` yet.
     ///
     /// This is intended to be used as a builder method:
     ///
     /// ```
-    /// # use cranelift_codegen::ir::{Function, Ebb, Inst};
+    /// # use cranelift_codegen::ir::{Function, Block, Inst};
     /// # use cranelift_codegen::cursor::{Cursor, FuncCursor};
-    /// fn edit_func(func: &mut Function, ebb: Ebb) {
-    ///     let mut pos = FuncCursor::new(func).at_first_insertion_point(ebb);
+    /// fn edit_func(func: &mut Function, block: Block) {
+    ///     let mut pos = FuncCursor::new(func).at_first_insertion_point(block);
     ///
     ///     // Use `pos`...
     /// }
     /// ```
-    fn at_first_insertion_point(mut self, ebb: ir::Ebb) -> Self
+    fn at_first_insertion_point(mut self, block: ir::Block) -> Self
     where
         Self: Sized,
     {
-        self.goto_first_insertion_point(ebb);
+        self.goto_first_insertion_point(block);
         self
     }
 
-    /// Rebuild this cursor positioned at the first instruction in `ebb`.
+    /// Rebuild this cursor positioned at the first instruction in `block`.
     ///
     /// This is intended to be used as a builder method:
     ///
     /// ```
-    /// # use cranelift_codegen::ir::{Function, Ebb, Inst};
+    /// # use cranelift_codegen::ir::{Function, Block, Inst};
     /// # use cranelift_codegen::cursor::{Cursor, FuncCursor};
-    /// fn edit_func(func: &mut Function, ebb: Ebb) {
-    ///     let mut pos = FuncCursor::new(func).at_first_inst(ebb);
+    /// fn edit_func(func: &mut Function, block: Block) {
+    ///     let mut pos = FuncCursor::new(func).at_first_inst(block);
     ///
     ///     // Use `pos`...
     /// }
     /// ```
-    fn at_first_inst(mut self, ebb: ir::Ebb) -> Self
+    fn at_first_inst(mut self, block: ir::Block) -> Self
     where
         Self: Sized,
     {
-        self.goto_first_inst(ebb);
+        self.goto_first_inst(block);
         self
     }
 
-    /// Rebuild this cursor positioned at the last instruction in `ebb`.
+    /// Rebuild this cursor positioned at the last instruction in `block`.
     ///
     /// This is intended to be used as a builder method:
     ///
     /// ```
-    /// # use cranelift_codegen::ir::{Function, Ebb, Inst};
+    /// # use cranelift_codegen::ir::{Function, Block, Inst};
     /// # use cranelift_codegen::cursor::{Cursor, FuncCursor};
-    /// fn edit_func(func: &mut Function, ebb: Ebb) {
-    ///     let mut pos = FuncCursor::new(func).at_last_inst(ebb);
+    /// fn edit_func(func: &mut Function, block: Block) {
+    ///     let mut pos = FuncCursor::new(func).at_last_inst(block);
     ///
     ///     // Use `pos`...
     /// }
     /// ```
-    fn at_last_inst(mut self, ebb: ir::Ebb) -> Self
+    fn at_last_inst(mut self, block: ir::Block) -> Self
     where
         Self: Sized,
     {
-        self.goto_last_inst(ebb);
+        self.goto_last_inst(block);
         self
     }
 
@@ -162,7 +162,7 @@ pub trait Cursor {
     /// This is intended to be used as a builder method:
     ///
     /// ```
-    /// # use cranelift_codegen::ir::{Function, Ebb, Inst};
+    /// # use cranelift_codegen::ir::{Function, Block, Inst};
     /// # use cranelift_codegen::cursor::{Cursor, FuncCursor};
     /// fn edit_func(func: &mut Function, inst: Inst) {
     ///     let mut pos = FuncCursor::new(func).after_inst(inst);
@@ -178,55 +178,55 @@ pub trait Cursor {
         self
     }
 
-    /// Rebuild this cursor positioned at the top of `ebb`.
+    /// Rebuild this cursor positioned at the top of `block`.
     ///
     /// This is intended to be used as a builder method:
     ///
     /// ```
-    /// # use cranelift_codegen::ir::{Function, Ebb, Inst};
+    /// # use cranelift_codegen::ir::{Function, Block, Inst};
     /// # use cranelift_codegen::cursor::{Cursor, FuncCursor};
-    /// fn edit_func(func: &mut Function, ebb: Ebb) {
-    ///     let mut pos = FuncCursor::new(func).at_top(ebb);
+    /// fn edit_func(func: &mut Function, block: Block) {
+    ///     let mut pos = FuncCursor::new(func).at_top(block);
     ///
     ///     // Use `pos`...
     /// }
     /// ```
-    fn at_top(mut self, ebb: ir::Ebb) -> Self
+    fn at_top(mut self, block: ir::Block) -> Self
     where
         Self: Sized,
     {
-        self.goto_top(ebb);
+        self.goto_top(block);
         self
     }
 
-    /// Rebuild this cursor positioned at the bottom of `ebb`.
+    /// Rebuild this cursor positioned at the bottom of `block`.
     ///
     /// This is intended to be used as a builder method:
     ///
     /// ```
-    /// # use cranelift_codegen::ir::{Function, Ebb, Inst};
+    /// # use cranelift_codegen::ir::{Function, Block, Inst};
     /// # use cranelift_codegen::cursor::{Cursor, FuncCursor};
-    /// fn edit_func(func: &mut Function, ebb: Ebb) {
-    ///     let mut pos = FuncCursor::new(func).at_bottom(ebb);
+    /// fn edit_func(func: &mut Function, block: Block) {
+    ///     let mut pos = FuncCursor::new(func).at_bottom(block);
     ///
     ///     // Use `pos`...
     /// }
     /// ```
-    fn at_bottom(mut self, ebb: ir::Ebb) -> Self
+    fn at_bottom(mut self, block: ir::Block) -> Self
     where
         Self: Sized,
     {
-        self.goto_bottom(ebb);
+        self.goto_bottom(block);
         self
     }
 
-    /// Get the EBB corresponding to the current position.
-    fn current_ebb(&self) -> Option {
+    /// Get the block corresponding to the current position.
+    fn current_block(&self) -> Option {
         use self::CursorPosition::*;
         match self.position() {
             Nowhere => None,
-            At(inst) => self.layout().inst_ebb(inst),
-            Before(ebb) | After(ebb) => Some(ebb),
+            At(inst) => self.layout().inst_block(inst),
+            Before(block) | After(block) => Some(block),
         }
     }
 
@@ -242,13 +242,13 @@ pub trait Cursor {
     /// Go to the position after a specific instruction, which must be inserted
     /// in the layout. New instructions will be inserted after `inst`.
     fn goto_after_inst(&mut self, inst: ir::Inst) {
-        debug_assert!(self.layout().inst_ebb(inst).is_some());
+        debug_assert!(self.layout().inst_block(inst).is_some());
         let new_pos = if let Some(next) = self.layout().next_inst(inst) {
             CursorPosition::At(next)
         } else {
             CursorPosition::After(
                 self.layout()
-                    .inst_ebb(inst)
+                    .inst_block(inst)
                     .expect("current instruction removed?"),
             )
         };
@@ -258,133 +258,133 @@ pub trait Cursor {
     /// Go to a specific instruction which must be inserted in the layout.
     /// New instructions will be inserted before `inst`.
     fn goto_inst(&mut self, inst: ir::Inst) {
-        debug_assert!(self.layout().inst_ebb(inst).is_some());
+        debug_assert!(self.layout().inst_block(inst).is_some());
         self.set_position(CursorPosition::At(inst));
     }
 
-    /// Go to the position for inserting instructions at the beginning of `ebb`,
+    /// Go to the position for inserting instructions at the beginning of `block`,
     /// which unlike `goto_first_inst` doesn't assume that any instructions have
-    /// been inserted into `ebb` yet.
-    fn goto_first_insertion_point(&mut self, ebb: ir::Ebb) {
-        if let Some(inst) = self.layout().first_inst(ebb) {
+    /// been inserted into `block` yet.
+    fn goto_first_insertion_point(&mut self, block: ir::Block) {
+        if let Some(inst) = self.layout().first_inst(block) {
             self.goto_inst(inst);
         } else {
-            self.goto_bottom(ebb);
+            self.goto_bottom(block);
         }
     }
 
-    /// Go to the first instruction in `ebb`.
-    fn goto_first_inst(&mut self, ebb: ir::Ebb) {
-        let inst = self.layout().first_inst(ebb).expect("Empty EBB");
+    /// Go to the first instruction in `block`.
+    fn goto_first_inst(&mut self, block: ir::Block) {
+        let inst = self.layout().first_inst(block).expect("Empty block");
         self.goto_inst(inst);
     }
 
-    /// Go to the last instruction in `ebb`.
-    fn goto_last_inst(&mut self, ebb: ir::Ebb) {
-        let inst = self.layout().last_inst(ebb).expect("Empty EBB");
+    /// Go to the last instruction in `block`.
+    fn goto_last_inst(&mut self, block: ir::Block) {
+        let inst = self.layout().last_inst(block).expect("Empty block");
         self.goto_inst(inst);
     }
 
-    /// Go to the top of `ebb` which must be inserted into the layout.
+    /// Go to the top of `block` which must be inserted into the layout.
     /// At this position, instructions cannot be inserted, but `next_inst()` will move to the first
-    /// instruction in `ebb`.
-    fn goto_top(&mut self, ebb: ir::Ebb) {
-        debug_assert!(self.layout().is_ebb_inserted(ebb));
-        self.set_position(CursorPosition::Before(ebb));
+    /// instruction in `block`.
+    fn goto_top(&mut self, block: ir::Block) {
+        debug_assert!(self.layout().is_block_inserted(block));
+        self.set_position(CursorPosition::Before(block));
     }
 
-    /// Go to the bottom of `ebb` which must be inserted into the layout.
-    /// At this position, inserted instructions will be appended to `ebb`.
-    fn goto_bottom(&mut self, ebb: ir::Ebb) {
-        debug_assert!(self.layout().is_ebb_inserted(ebb));
-        self.set_position(CursorPosition::After(ebb));
+    /// Go to the bottom of `block` which must be inserted into the layout.
+    /// At this position, inserted instructions will be appended to `block`.
+    fn goto_bottom(&mut self, block: ir::Block) {
+        debug_assert!(self.layout().is_block_inserted(block));
+        self.set_position(CursorPosition::After(block));
     }
 
-    /// Go to the top of the next EBB in layout order and return it.
+    /// Go to the top of the next block in layout order and return it.
     ///
-    /// - If the cursor wasn't pointing at anything, go to the top of the first EBB in the
+    /// - If the cursor wasn't pointing at anything, go to the top of the first block in the
     ///   function.
-    /// - If there are no more EBBs, leave the cursor pointing at nothing and return `None`.
+    /// - If there are no more blocks, leave the cursor pointing at nothing and return `None`.
     ///
     /// # Examples
     ///
-    /// The `next_ebb()` method is intended for iterating over the EBBs in layout order:
+    /// The `next_block()` method is intended for iterating over the blocks in layout order:
     ///
     /// ```
-    /// # use cranelift_codegen::ir::{Function, Ebb};
+    /// # use cranelift_codegen::ir::{Function, Block};
     /// # use cranelift_codegen::cursor::{Cursor, FuncCursor};
     /// fn edit_func(func: &mut Function) {
     ///     let mut cursor = FuncCursor::new(func);
-    ///     while let Some(ebb) = cursor.next_ebb() {
-    ///         // Edit ebb.
+    ///     while let Some(block) = cursor.next_block() {
+    ///         // Edit block.
     ///     }
     /// }
     /// ```
-    fn next_ebb(&mut self) -> Option {
-        let next = if let Some(ebb) = self.current_ebb() {
-            self.layout().next_ebb(ebb)
+    fn next_block(&mut self) -> Option {
+        let next = if let Some(block) = self.current_block() {
+            self.layout().next_block(block)
         } else {
             self.layout().entry_block()
         };
         self.set_position(match next {
-            Some(ebb) => CursorPosition::Before(ebb),
+            Some(block) => CursorPosition::Before(block),
             None => CursorPosition::Nowhere,
         });
         next
     }
 
-    /// Go to the bottom of the previous EBB in layout order and return it.
+    /// Go to the bottom of the previous block in layout order and return it.
     ///
-    /// - If the cursor wasn't pointing at anything, go to the bottom of the last EBB in the
+    /// - If the cursor wasn't pointing at anything, go to the bottom of the last block in the
     ///   function.
-    /// - If there are no more EBBs, leave the cursor pointing at nothing and return `None`.
+    /// - If there are no more blocks, leave the cursor pointing at nothing and return `None`.
     ///
     /// # Examples
     ///
-    /// The `prev_ebb()` method is intended for iterating over the EBBs in backwards layout order:
+    /// The `prev_block()` method is intended for iterating over the blocks in backwards layout order:
     ///
     /// ```
-    /// # use cranelift_codegen::ir::{Function, Ebb};
+    /// # use cranelift_codegen::ir::{Function, Block};
     /// # use cranelift_codegen::cursor::{Cursor, FuncCursor};
     /// fn edit_func(func: &mut Function) {
     ///     let mut cursor = FuncCursor::new(func);
-    ///     while let Some(ebb) = cursor.prev_ebb() {
-    ///         // Edit ebb.
+    ///     while let Some(block) = cursor.prev_block() {
+    ///         // Edit block.
     ///     }
     /// }
     /// ```
-    fn prev_ebb(&mut self) -> Option {
-        let prev = if let Some(ebb) = self.current_ebb() {
-            self.layout().prev_ebb(ebb)
+    fn prev_block(&mut self) -> Option {
+        let prev = if let Some(block) = self.current_block() {
+            self.layout().prev_block(block)
         } else {
-            self.layout().last_ebb()
+            self.layout().last_block()
         };
         self.set_position(match prev {
-            Some(ebb) => CursorPosition::After(ebb),
+            Some(block) => CursorPosition::After(block),
             None => CursorPosition::Nowhere,
         });
         prev
     }
 
-    /// Move to the next instruction in the same EBB and return it.
+    /// Move to the next instruction in the same block and return it.
     ///
-    /// - If the cursor was positioned before an EBB, go to the first instruction in that EBB.
-    /// - If there are no more instructions in the EBB, go to the `After(ebb)` position and return
+    /// - If the cursor was positioned before an block, go to the first instruction in that block.
+    /// - If there are no more instructions in the block, go to the `After(block)` position and return
     ///   `None`.
     /// - If the cursor wasn't pointing anywhere, keep doing that.
     ///
-    /// This method will never move the cursor to a different EBB.
+    /// This method will never move the cursor to a different block.
     ///
     /// # Examples
     ///
-    /// The `next_inst()` method is intended for iterating over the instructions in an EBB like
+    /// The `next_inst()` method is intended for iterating over the instructions in an block like
     /// this:
     ///
     /// ```
-    /// # use cranelift_codegen::ir::{Function, Ebb};
+    /// # use cranelift_codegen::ir::{Function, Block};
     /// # use cranelift_codegen::cursor::{Cursor, FuncCursor};
-    /// fn edit_ebb(func: &mut Function, ebb: Ebb) {
-    ///     let mut cursor = FuncCursor::new(func).at_top(ebb);
+    /// fn edit_block(func: &mut Function, block: Block) {
+    ///     let mut cursor = FuncCursor::new(func).at_top(block);
     ///     while let Some(inst) = cursor.next_inst() {
     ///         // Edit instructions...
     ///     }
@@ -395,11 +395,11 @@ pub trait Cursor {
     /// Iterating over all the instructions in a function looks like this:
     ///
     /// ```
-    /// # use cranelift_codegen::ir::{Function, Ebb};
+    /// # use cranelift_codegen::ir::{Function, Block};
     /// # use cranelift_codegen::cursor::{Cursor, FuncCursor};
     /// fn edit_func(func: &mut Function) {
     ///     let mut cursor = FuncCursor::new(func);
-    ///     while let Some(ebb) = cursor.next_ebb() {
+    ///     while let Some(block) = cursor.next_block() {
     ///         while let Some(inst) = cursor.next_inst() {
     ///             // Edit instructions...
     ///         }
@@ -417,44 +417,44 @@ pub trait Cursor {
                 } else {
                     let pos = After(
                         self.layout()
-                            .inst_ebb(inst)
+                            .inst_block(inst)
                             .expect("current instruction removed?"),
                     );
                     self.set_position(pos);
                     None
                 }
             }
-            Before(ebb) => {
-                if let Some(next) = self.layout().first_inst(ebb) {
+            Before(block) => {
+                if let Some(next) = self.layout().first_inst(block) {
                     self.set_position(At(next));
                     Some(next)
                 } else {
-                    self.set_position(After(ebb));
+                    self.set_position(After(block));
                     None
                 }
             }
         }
     }
 
-    /// Move to the previous instruction in the same EBB and return it.
+    /// Move to the previous instruction in the same block and return it.
     ///
-    /// - If the cursor was positioned after an EBB, go to the last instruction in that EBB.
-    /// - If there are no more instructions in the EBB, go to the `Before(ebb)` position and return
+    /// - If the cursor was positioned after an block, go to the last instruction in that block.
+    /// - If there are no more instructions in the block, go to the `Before(block)` position and return
     ///   `None`.
     /// - If the cursor wasn't pointing anywhere, keep doing that.
     ///
-    /// This method will never move the cursor to a different EBB.
+    /// This method will never move the cursor to a different block.
     ///
     /// # Examples
     ///
     /// The `prev_inst()` method is intended for iterating backwards over the instructions in an
-    /// EBB like this:
+    /// block like this:
     ///
     /// ```
-    /// # use cranelift_codegen::ir::{Function, Ebb};
+    /// # use cranelift_codegen::ir::{Function, Block};
     /// # use cranelift_codegen::cursor::{Cursor, FuncCursor};
-    /// fn edit_ebb(func: &mut Function, ebb: Ebb) {
-    ///     let mut cursor = FuncCursor::new(func).at_bottom(ebb);
+    /// fn edit_block(func: &mut Function, block: Block) {
+    ///     let mut cursor = FuncCursor::new(func).at_bottom(block);
     ///     while let Some(inst) = cursor.prev_inst() {
     ///         // Edit instructions...
     ///     }
@@ -471,19 +471,19 @@ pub trait Cursor {
                 } else {
                     let pos = Before(
                         self.layout()
-                            .inst_ebb(inst)
+                            .inst_block(inst)
                             .expect("current instruction removed?"),
                     );
                     self.set_position(pos);
                     None
                 }
             }
-            After(ebb) => {
-                if let Some(prev) = self.layout().last_inst(ebb) {
+            After(block) => {
+                if let Some(prev) = self.layout().last_inst(block) {
                     self.set_position(At(prev));
                     Some(prev)
                 } else {
-                    self.set_position(Before(ebb));
+                    self.set_position(Before(block));
                     None
                 }
             }
@@ -494,17 +494,17 @@ pub trait Cursor {
     ///
     /// - If pointing at an instruction, the new instruction is inserted before the current
     ///   instruction.
-    /// - If pointing at the bottom of an EBB, the new instruction is appended to the EBB.
+    /// - If pointing at the bottom of an block, the new instruction is appended to the block.
     /// - Otherwise panic.
     ///
     /// In either case, the cursor is not moved, such that repeated calls to `insert_inst()` causes
-    /// instructions to appear in insertion order in the EBB.
+    /// instructions to appear in insertion order in the block.
     fn insert_inst(&mut self, inst: ir::Inst) {
         use self::CursorPosition::*;
         match self.position() {
             Nowhere | Before(..) => panic!("Invalid insert_inst position"),
             At(cur) => self.layout_mut().insert_inst(inst, cur),
-            After(ebb) => self.layout_mut().append_inst(inst, ebb),
+            After(block) => self.layout_mut().append_inst(inst, block),
         }
     }
 
@@ -532,34 +532,34 @@ pub trait Cursor {
         inst
     }
 
-    /// Insert an EBB at the current position and switch to it.
+    /// Insert an block at the current position and switch to it.
     ///
-    /// As far as possible, this method behaves as if the EBB header were an instruction inserted
+    /// As far as possible, this method behaves as if the block header were an instruction inserted
     /// at the current position.
     ///
-    /// - If the cursor is pointing at an existing instruction, *the current EBB is split in two*
-    ///   and the current instruction becomes the first instruction in the inserted EBB.
-    /// - If the cursor points at the bottom of an EBB, the new EBB is inserted after the current
-    ///   one, and moved to the bottom of the new EBB where instructions can be appended.
-    /// - If the cursor points to the top of an EBB, the new EBB is inserted above the current one.
-    /// - If the cursor is not pointing at anything, the new EBB is placed last in the layout.
+    /// - If the cursor is pointing at an existing instruction, *the current block is split in two*
+    ///   and the current instruction becomes the first instruction in the inserted block.
+    /// - If the cursor points at the bottom of an block, the new block is inserted after the current
+    ///   one, and moved to the bottom of the new block where instructions can be appended.
+    /// - If the cursor points to the top of an block, the new block is inserted above the current one.
+    /// - If the cursor is not pointing at anything, the new block is placed last in the layout.
     ///
     /// This means that it is always valid to call this method, and it always leaves the cursor in
-    /// a state that will insert instructions into the new EBB.
-    fn insert_ebb(&mut self, new_ebb: ir::Ebb) {
+    /// a state that will insert instructions into the new block.
+    fn insert_block(&mut self, new_block: ir::Block) {
         use self::CursorPosition::*;
         match self.position() {
             At(inst) => {
-                self.layout_mut().split_ebb(new_ebb, inst);
-                // All other cases move to `After(ebb)`, but in this case we'll stay `At(inst)`.
+                self.layout_mut().split_block(new_block, inst);
+                // All other cases move to `After(block)`, but in this case we'll stay `At(inst)`.
                 return;
             }
-            Nowhere => self.layout_mut().append_ebb(new_ebb),
-            Before(ebb) => self.layout_mut().insert_ebb(new_ebb, ebb),
-            After(ebb) => self.layout_mut().insert_ebb_after(new_ebb, ebb),
+            Nowhere => self.layout_mut().append_block(new_block),
+            Before(block) => self.layout_mut().insert_block(new_block, block),
+            After(block) => self.layout_mut().insert_block_after(new_block, block),
         }
-        // For everything but `At(inst)` we end up appending to the new EBB.
-        self.set_position(After(new_ebb));
+        // For everything but `At(inst)` we end up appending to the new block.
+        self.set_position(After(new_block));
     }
 }
 
diff --git a/cranelift/codegen/src/dce.rs b/cranelift/codegen/src/dce.rs
index 0aa9adc15b..b217534c3e 100644
--- a/cranelift/codegen/src/dce.rs
+++ b/cranelift/codegen/src/dce.rs
@@ -46,8 +46,8 @@ pub fn do_dce(func: &mut Function, domtree: &mut DominatorTree) {
     debug_assert!(domtree.is_valid());
 
     let mut live = vec![false; func.dfg.num_values()];
-    for &ebb in domtree.cfg_postorder() {
-        let mut pos = FuncCursor::new(func).at_bottom(ebb);
+    for &block in domtree.cfg_postorder() {
+        let mut pos = FuncCursor::new(func).at_bottom(block);
         while let Some(inst) = pos.prev_inst() {
             {
                 let data = &pos.func.dfg[inst];
diff --git a/cranelift/codegen/src/dominator_tree.rs b/cranelift/codegen/src/dominator_tree.rs
index 5251121cdc..d397fc7183 100644
--- a/cranelift/codegen/src/dominator_tree.rs
+++ b/cranelift/codegen/src/dominator_tree.rs
@@ -1,9 +1,9 @@
-//! A Dominator Tree represented as mappings of Ebbs to their immediate dominator.
+//! A Dominator Tree represented as mappings of Blocks to their immediate dominator.
 
 use crate::entity::SecondaryMap;
-use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::flowgraph::{BlockPredecessor, ControlFlowGraph};
 use crate::ir::instructions::BranchInfo;
-use crate::ir::{Ebb, ExpandedProgramPoint, Function, Inst, Layout, ProgramOrder, Value};
+use crate::ir::{Block, ExpandedProgramPoint, Function, Inst, Layout, ProgramOrder, Value};
 use crate::packed_option::PackedOption;
 use crate::timing;
 use alloc::vec::Vec;
@@ -19,7 +19,7 @@ const STRIDE: u32 = 4;
 const DONE: u32 = 1;
 const SEEN: u32 = 2;
 
-/// Dominator tree node. We keep one of these per EBB.
+/// Dominator tree node. We keep one of these per block.
 #[derive(Clone, Default)]
 struct DomNode {
     /// Number of this node in a reverse post-order traversal of the CFG, starting from 1.
@@ -28,7 +28,7 @@ struct DomNode {
     /// Unreachable nodes get number 0, all others are positive.
     rpo_number: u32,
 
-    /// The immediate dominator of this EBB, represented as the branch or jump instruction at the
+    /// The immediate dominator of this block, represented as the branch or jump instruction at the
     /// end of the dominating basic block.
     ///
     /// This is `None` for unreachable blocks and the entry block which doesn't have an immediate
@@ -38,53 +38,53 @@ struct DomNode {
 
 /// The dominator tree for a single function.
 pub struct DominatorTree {
-    nodes: SecondaryMap,
+    nodes: SecondaryMap,
 
-    /// CFG post-order of all reachable EBBs.
-    postorder: Vec,
+    /// CFG post-order of all reachable blocks.
+    postorder: Vec,
 
     /// Scratch memory used by `compute_postorder()`.
-    stack: Vec,
+    stack: Vec,
 
     valid: bool,
 }
 
 /// Methods for querying the dominator tree.
 impl DominatorTree {
-    /// Is `ebb` reachable from the entry block?
-    pub fn is_reachable(&self, ebb: Ebb) -> bool {
-        self.nodes[ebb].rpo_number != 0
+    /// Is `block` reachable from the entry block?
+    pub fn is_reachable(&self, block: Block) -> bool {
+        self.nodes[block].rpo_number != 0
     }
 
-    /// Get the CFG post-order of EBBs that was used to compute the dominator tree.
+    /// Get the CFG post-order of blocks that was used to compute the dominator tree.
     ///
     /// Note that this post-order is not updated automatically when the CFG is modified. It is
     /// computed from scratch and cached by `compute()`.
-    pub fn cfg_postorder(&self) -> &[Ebb] {
+    pub fn cfg_postorder(&self) -> &[Block] {
         debug_assert!(self.is_valid());
         &self.postorder
     }
 
-    /// Returns the immediate dominator of `ebb`.
+    /// Returns the immediate dominator of `block`.
     ///
-    /// The immediate dominator of an extended basic block is a basic block which we represent by
+    /// The immediate dominator of a basic block is a basic block which we represent by
     /// the branch or jump instruction at the end of the basic block. This does not have to be the
-    /// terminator of its EBB.
+    /// terminator of its block.
     ///
-    /// A branch or jump is said to *dominate* `ebb` if all control flow paths from the function
-    /// entry to `ebb` must go through the branch.
+    /// A branch or jump is said to *dominate* `block` if all control flow paths from the function
+    /// entry to `block` must go through the branch.
     ///
-    /// The *immediate dominator* is the dominator that is closest to `ebb`. All other dominators
+    /// The *immediate dominator* is the dominator that is closest to `block`. All other dominators
     /// also dominate the immediate dominator.
     ///
-    /// This returns `None` if `ebb` is not reachable from the entry EBB, or if it is the entry EBB
+    /// This returns `None` if `block` is not reachable from the entry block, or if it is the entry block
     /// which has no dominators.
-    pub fn idom(&self, ebb: Ebb) -> Option {
-        self.nodes[ebb].idom.into()
+    pub fn idom(&self, block: Block) -> Option {
+        self.nodes[block].idom.into()
     }
 
-    /// Compare two EBBs relative to the reverse post-order.
-    fn rpo_cmp_ebb(&self, a: Ebb, b: Ebb) -> Ordering {
+    /// Compare two blocks relative to the reverse post-order.
+    fn rpo_cmp_block(&self, a: Block, b: Block) -> Ordering {
         self.nodes[a].rpo_number.cmp(&self.nodes[b].rpo_number)
     }
 
@@ -93,7 +93,7 @@ impl DominatorTree {
     ///
     /// Return `Ordering::Less` if `a` comes before `b` in the RPO.
     ///
-    /// If `a` and `b` belong to the same EBB, compare their relative position in the EBB.
+    /// If `a` and `b` belong to the same block, compare their relative position in the block.
     pub fn rpo_cmp(&self, a: A, b: B, layout: &Layout) -> Ordering
     where
         A: Into,
@@ -101,7 +101,7 @@ impl DominatorTree {
     {
         let a = a.into();
         let b = b.into();
-        self.rpo_cmp_ebb(layout.pp_ebb(a), layout.pp_ebb(b))
+        self.rpo_cmp_block(layout.pp_block(a), layout.pp_block(b))
             .then(layout.cmp(a, b))
     }
 
@@ -110,7 +110,7 @@ impl DominatorTree {
     /// This means that every control-flow path from the function entry to `b` must go through `a`.
     ///
     /// Dominance is ill defined for unreachable blocks. This function can always determine
-    /// dominance for instructions in the same EBB, but otherwise returns `false` if either block
+    /// dominance for instructions in the same block, but otherwise returns `false` if either block
     /// is unreachable.
     ///
     /// An instruction is considered to dominate itself.
@@ -122,12 +122,14 @@ impl DominatorTree {
         let a = a.into();
         let b = b.into();
         match a {
-            ExpandedProgramPoint::Ebb(ebb_a) => {
-                a == b || self.last_dominator(ebb_a, b, layout).is_some()
+            ExpandedProgramPoint::Block(block_a) => {
+                a == b || self.last_dominator(block_a, b, layout).is_some()
             }
             ExpandedProgramPoint::Inst(inst_a) => {
-                let ebb_a = layout.inst_ebb(inst_a).expect("Instruction not in layout.");
-                match self.last_dominator(ebb_a, b, layout) {
+                let block_a = layout
+                    .inst_block(inst_a)
+                    .expect("Instruction not in layout.");
+                match self.last_dominator(block_a, b, layout) {
                     Some(last) => layout.cmp(inst_a, last) != Ordering::Greater,
                     None => false,
                 }
@@ -137,14 +139,14 @@ impl DominatorTree {
 
     /// Find the last instruction in `a` that dominates `b`.
     /// If no instructions in `a` dominate `b`, return `None`.
-    pub fn last_dominator(&self, a: Ebb, b: B, layout: &Layout) -> Option
+    pub fn last_dominator(&self, a: Block, b: B, layout: &Layout) -> Option
     where
         B: Into,
     {
-        let (mut ebb_b, mut inst_b) = match b.into() {
-            ExpandedProgramPoint::Ebb(ebb) => (ebb, None),
+        let (mut block_b, mut inst_b) = match b.into() {
+            ExpandedProgramPoint::Block(block) => (block, None),
             ExpandedProgramPoint::Inst(inst) => (
-                layout.inst_ebb(inst).expect("Instruction not in layout."),
+                layout.inst_block(inst).expect("Instruction not in layout."),
                 Some(inst),
             ),
         };
@@ -152,15 +154,15 @@ impl DominatorTree {
 
         // Run a finger up the dominator tree from b until we see a.
         // Do nothing if b is unreachable.
-        while rpo_a < self.nodes[ebb_b].rpo_number {
-            let idom = match self.idom(ebb_b) {
+        while rpo_a < self.nodes[block_b].rpo_number {
+            let idom = match self.idom(block_b) {
                 Some(idom) => idom,
                 None => return None, // a is unreachable, so we climbed past the entry
             };
-            ebb_b = layout.inst_ebb(idom).expect("Dominator got removed.");
+            block_b = layout.inst_block(idom).expect("Dominator got removed.");
             inst_b = Some(idom);
         }
-        if a == ebb_b {
+        if a == block_b {
             inst_b
         } else {
             None
@@ -172,25 +174,25 @@ impl DominatorTree {
     /// Both basic blocks are assumed to be reachable.
     pub fn common_dominator(
         &self,
-        mut a: BasicBlock,
-        mut b: BasicBlock,
+        mut a: BlockPredecessor,
+        mut b: BlockPredecessor,
         layout: &Layout,
-    ) -> BasicBlock {
+    ) -> BlockPredecessor {
         loop {
-            match self.rpo_cmp_ebb(a.ebb, b.ebb) {
+            match self.rpo_cmp_block(a.block, b.block) {
                 Ordering::Less => {
                     // `a` comes before `b` in the RPO. Move `b` up.
-                    let idom = self.nodes[b.ebb].idom.expect("Unreachable basic block?");
-                    b = BasicBlock::new(
-                        layout.inst_ebb(idom).expect("Dangling idom instruction"),
+                    let idom = self.nodes[b.block].idom.expect("Unreachable basic block?");
+                    b = BlockPredecessor::new(
+                        layout.inst_block(idom).expect("Dangling idom instruction"),
                         idom,
                     );
                 }
                 Ordering::Greater => {
                     // `b` comes before `a` in the RPO. Move `a` up.
-                    let idom = self.nodes[a.ebb].idom.expect("Unreachable basic block?");
-                    a = BasicBlock::new(
-                        layout.inst_ebb(idom).expect("Dangling idom instruction"),
+                    let idom = self.nodes[a.block].idom.expect("Unreachable basic block?");
+                    a = BlockPredecessor::new(
+                        layout.inst_block(idom).expect("Dangling idom instruction"),
                         idom,
                     );
                 }
@@ -199,11 +201,11 @@ impl DominatorTree {
         }
 
         debug_assert_eq!(
-            a.ebb, b.ebb,
+            a.block, b.block,
             "Unreachable block passed to common_dominator?"
         );
 
-        // We're in the same EBB. The common dominator is the earlier instruction.
+        // We're in the same block. The common dominator is the earlier instruction.
         if layout.cmp(a.inst, b.inst) == Ordering::Less {
             a
         } else {
@@ -226,10 +228,10 @@ impl DominatorTree {
 
     /// Allocate and compute a dominator tree.
     pub fn with_function(func: &Function, cfg: &ControlFlowGraph) -> Self {
-        let ebb_capacity = func.layout.ebb_capacity();
+        let block_capacity = func.layout.block_capacity();
         let mut domtree = Self {
-            nodes: SecondaryMap::with_capacity(ebb_capacity),
-            postorder: Vec::with_capacity(ebb_capacity),
+            nodes: SecondaryMap::with_capacity(block_capacity),
+            postorder: Vec::with_capacity(block_capacity),
             stack: Vec::new(),
             valid: false,
         };
@@ -266,13 +268,13 @@ impl DominatorTree {
 
     /// Reset all internal data structures and compute a post-order of the control flow graph.
     ///
-    /// This leaves `rpo_number == 1` for all reachable EBBs, 0 for unreachable ones.
+    /// This leaves `rpo_number == 1` for all reachable blocks, 0 for unreachable ones.
     fn compute_postorder(&mut self, func: &Function) {
         self.clear();
-        self.nodes.resize(func.dfg.num_ebbs());
+        self.nodes.resize(func.dfg.num_blocks());
 
         // This algorithm is a depth first traversal (DFT) of the control flow graph, computing a
-        // post-order of the EBBs that are reachable form the entry block. A DFT post-order is not
+        // post-order of the blocks that are reachable form the entry block. A DFT post-order is not
         // unique. The specific order we get is controlled by two factors:
         //
         // 1. The order each node's children are visited, and
@@ -280,76 +282,76 @@ impl DominatorTree {
         //
         // There are two ways of viewing the CFG as a graph:
         //
-        // 1. Each EBB is a node, with outgoing edges for all the branches in the EBB.
+        // 1. Each block is a node, with outgoing edges for all the branches in the block.
         // 2. Each basic block is a node, with outgoing edges for the single branch at the end of
-        //    the BB. (An EBB is a linear sequence of basic blocks).
+        //    the BB. (An block is a linear sequence of basic blocks).
         //
-        // The first graph is a contraction of the second one. We want to compute an EBB post-order
+        // The first graph is a contraction of the second one. We want to compute an block post-order
         // that is compatible both graph interpretations. That is, if you compute a BB post-order
-        // and then remove those BBs that do not correspond to EBB headers, you get a post-order of
-        // the EBB graph.
+        // and then remove those BBs that do not correspond to block headers, you get a post-order of
+        // the block graph.
         //
         // Node child order:
         //
         //     In the BB graph, we always go down the fall-through path first and follow the branch
         //     destination second.
         //
-        //     In the EBB graph, this is equivalent to visiting EBB successors in a bottom-up
-        //     order, starting from the destination of the EBB's terminating jump, ending at the
-        //     destination of the first branch in the EBB.
+        //     In the block graph, this is equivalent to visiting block successors in a bottom-up
+        //     order, starting from the destination of the block's terminating jump, ending at the
+        //     destination of the first branch in the block.
         //
         // Edge pruning:
         //
-        //     In the BB graph, we keep an edge to an EBB the first time we visit the *source* side
-        //     of the edge. Any subsequent edges to the same EBB are pruned.
+        //     In the BB graph, we keep an edge to an block the first time we visit the *source* side
+        //     of the edge. Any subsequent edges to the same block are pruned.
         //
-        //     The equivalent tree is reached in the EBB graph by keeping the first edge to an EBB
+        //     The equivalent tree is reached in the block graph by keeping the first edge to an block
         //     in a top-down traversal of the successors. (And then visiting edges in a bottom-up
         //     order).
         //
         // This pruning method makes it possible to compute the DFT without storing lots of
-        // information about the progress through an EBB.
+        // information about the progress through an block.
 
         // During this algorithm only, use `rpo_number` to hold the following state:
         //
-        //   0:    EBB has not yet been reached in the pre-order.
-        //   SEEN: EBB has been pushed on the stack but successors not yet pushed.
+        //   0:    block has not yet been reached in the pre-order.
+        //   SEEN: block has been pushed on the stack but successors not yet pushed.
         //   DONE: Successors pushed.
 
         match func.layout.entry_block() {
-            Some(ebb) => {
-                self.stack.push(ebb);
-                self.nodes[ebb].rpo_number = SEEN;
+            Some(block) => {
+                self.stack.push(block);
+                self.nodes[block].rpo_number = SEEN;
             }
             None => return,
         }
 
-        while let Some(ebb) = self.stack.pop() {
-            match self.nodes[ebb].rpo_number {
+        while let Some(block) = self.stack.pop() {
+            match self.nodes[block].rpo_number {
                 SEEN => {
-                    // This is the first time we pop the EBB, so we need to scan its successors and
+                    // This is the first time we pop the block, so we need to scan its successors and
                     // then revisit it.
-                    self.nodes[ebb].rpo_number = DONE;
-                    self.stack.push(ebb);
-                    self.push_successors(func, ebb);
+                    self.nodes[block].rpo_number = DONE;
+                    self.stack.push(block);
+                    self.push_successors(func, block);
                 }
                 DONE => {
-                    // This is the second time we pop the EBB, so all successors have been
+                    // This is the second time we pop the block, so all successors have been
                     // processed.
-                    self.postorder.push(ebb);
+                    self.postorder.push(block);
                 }
                 _ => unreachable!(),
             }
         }
     }
 
-    /// Push `ebb` successors onto `self.stack`, filtering out those that have already been seen.
+    /// Push `block` successors onto `self.stack`, filtering out those that have already been seen.
     ///
     /// The successors are pushed in program order which is important to get a split-invariant
-    /// post-order. Split-invariant means that if an EBB is split in two, we get the same
-    /// post-order except for the insertion of the new EBB header at the split point.
-    fn push_successors(&mut self, func: &Function, ebb: Ebb) {
-        for inst in func.layout.ebb_insts(ebb) {
+    /// post-order. Split-invariant means that if an block is split in two, we get the same
+    /// post-order except for the insertion of the new block header at the split point.
+    fn push_successors(&mut self, func: &Function, block: Block) {
+        for inst in func.layout.block_insts(block) {
             match func.dfg.analyze_branch(inst) {
                 BranchInfo::SingleDest(succ, _) => self.push_if_unseen(succ),
                 BranchInfo::Table(jt, dest) => {
@@ -365,11 +367,11 @@ impl DominatorTree {
         }
     }
 
-    /// Push `ebb` onto `self.stack` if it has not already been seen.
-    fn push_if_unseen(&mut self, ebb: Ebb) {
-        if self.nodes[ebb].rpo_number == 0 {
-            self.nodes[ebb].rpo_number = SEEN;
-            self.stack.push(ebb);
+    /// Push `block` onto `self.stack` if it has not already been seen.
+    fn push_if_unseen(&mut self, block: Block) {
+        if self.nodes[block].rpo_number == 0 {
+            self.nodes[block].rpo_number = SEEN;
+            self.stack.push(block);
         }
     }
 
@@ -378,10 +380,10 @@ impl DominatorTree {
     fn compute_domtree(&mut self, func: &Function, cfg: &ControlFlowGraph) {
         // During this algorithm, `rpo_number` has the following values:
         //
-        // 0: EBB is not reachable.
-        // 1: EBB is reachable, but has not yet been visited during the first pass. This is set by
+        // 0: block is not reachable.
+        // 1: block is reachable, but has not yet been visited during the first pass. This is set by
         // `compute_postorder`.
-        // 2+: EBB is reachable and has an assigned RPO number.
+        // 2+: block is reachable and has an assigned RPO number.
 
         // We'll be iterating over a reverse post-order of the CFG, skipping the entry block.
         let (entry_block, postorder) = match self.postorder.as_slice().split_last() {
@@ -392,7 +394,7 @@ impl DominatorTree {
 
         // Do a first pass where we assign RPO numbers to all reachable nodes.
         self.nodes[entry_block].rpo_number = 2 * STRIDE;
-        for (rpo_idx, &ebb) in postorder.iter().rev().enumerate() {
+        for (rpo_idx, &block) in postorder.iter().rev().enumerate() {
             // Update the current node and give it an RPO number.
             // The entry block got 2, the rest start at 3 by multiples of STRIDE to leave
             // room for future dominator tree modifications.
@@ -402,8 +404,8 @@ impl DominatorTree {
             //
             // Due to the nature of the post-order traversal, every node we visit will have at
             // least one predecessor that has previously been visited during this RPO.
-            self.nodes[ebb] = DomNode {
-                idom: self.compute_idom(ebb, cfg, &func.layout).into(),
+            self.nodes[block] = DomNode {
+                idom: self.compute_idom(block, cfg, &func.layout).into(),
                 rpo_number: (rpo_idx as u32 + 3) * STRIDE,
             }
         }
@@ -415,30 +417,30 @@ impl DominatorTree {
         let mut changed = true;
         while changed {
             changed = false;
-            for &ebb in postorder.iter().rev() {
-                let idom = self.compute_idom(ebb, cfg, &func.layout).into();
-                if self.nodes[ebb].idom != idom {
-                    self.nodes[ebb].idom = idom;
+            for &block in postorder.iter().rev() {
+                let idom = self.compute_idom(block, cfg, &func.layout).into();
+                if self.nodes[block].idom != idom {
+                    self.nodes[block].idom = idom;
                     changed = true;
                 }
             }
         }
     }
 
-    // Compute the immediate dominator for `ebb` using the current `idom` states for the reachable
+    // Compute the immediate dominator for `block` using the current `idom` states for the reachable
     // nodes.
-    fn compute_idom(&self, ebb: Ebb, cfg: &ControlFlowGraph, layout: &Layout) -> Inst {
-        // Get an iterator with just the reachable, already visited predecessors to `ebb`.
+    fn compute_idom(&self, block: Block, cfg: &ControlFlowGraph, layout: &Layout) -> Inst {
+        // Get an iterator with just the reachable, already visited predecessors to `block`.
         // Note that during the first pass, `rpo_number` is 1 for reachable blocks that haven't
         // been visited yet, 0 for unreachable blocks.
         let mut reachable_preds = cfg
-            .pred_iter(ebb)
-            .filter(|&BasicBlock { ebb: pred, .. }| self.nodes[pred].rpo_number > 1);
+            .pred_iter(block)
+            .filter(|&BlockPredecessor { block: pred, .. }| self.nodes[pred].rpo_number > 1);
 
         // The RPO must visit at least one predecessor before this node.
         let mut idom = reachable_preds
             .next()
-            .expect("EBB node must have one reachable predecessor");
+            .expect("block node must have one reachable predecessor");
 
         for pred in reachable_preds {
             idom = self.common_dominator(idom, pred, layout);
@@ -453,25 +455,25 @@ impl DominatorTree {
 /// This data structure is computed from a `DominatorTree` and provides:
 ///
 /// - A forward traversable dominator tree through the `children()` iterator.
-/// - An ordering of EBBs according to a dominator tree pre-order.
-/// - Constant time dominance checks at the EBB granularity.
+/// - An ordering of blocks according to a dominator tree pre-order.
+/// - Constant time dominance checks at the block granularity.
 ///
 /// The information in this auxiliary data structure is not easy to update when the control flow
 /// graph changes, which is why it is kept separate.
 pub struct DominatorTreePreorder {
-    nodes: SecondaryMap,
+    nodes: SecondaryMap,
 
     // Scratch memory used by `compute_postorder()`.
-    stack: Vec,
+    stack: Vec,
 }
 
 #[derive(Default, Clone)]
 struct ExtraNode {
     /// First child node in the domtree.
-    child: PackedOption,
+    child: PackedOption,
 
     /// Next sibling node in the domtree. This linked list is ordered according to the CFG RPO.
-    sibling: PackedOption,
+    sibling: PackedOption,
 
     /// Sequence number for this node in a pre-order traversal of the dominator tree.
     /// Unreachable blocks have number 0, the entry block is 1.
@@ -501,23 +503,23 @@ impl DominatorTreePreorder {
         //
         // By following the CFG post-order and pushing to the front of the lists, we make sure that
         // sibling lists are ordered according to the CFG reverse post-order.
-        for &ebb in domtree.cfg_postorder() {
-            if let Some(idom_inst) = domtree.idom(ebb) {
-                let idom = layout.pp_ebb(idom_inst);
-                let sib = mem::replace(&mut self.nodes[idom].child, ebb.into());
-                self.nodes[ebb].sibling = sib;
+        for &block in domtree.cfg_postorder() {
+            if let Some(idom_inst) = domtree.idom(block) {
+                let idom = layout.pp_block(idom_inst);
+                let sib = mem::replace(&mut self.nodes[idom].child, block.into());
+                self.nodes[block].sibling = sib;
             } else {
-                // The only EBB without an immediate dominator is the entry.
-                self.stack.push(ebb);
+                // The only block without an immediate dominator is the entry.
+                self.stack.push(block);
             }
         }
 
         // Step 2. Assign pre-order numbers from a DFS of the dominator tree.
         debug_assert!(self.stack.len() <= 1);
         let mut n = 0;
-        while let Some(ebb) = self.stack.pop() {
+        while let Some(block) = self.stack.pop() {
             n += 1;
-            let node = &mut self.nodes[ebb];
+            let node = &mut self.nodes[block];
             node.pre_number = n;
             node.pre_max = n;
             if let Some(n) = node.sibling.expand() {
@@ -531,29 +533,29 @@ impl DominatorTreePreorder {
         // Step 3. Propagate the `pre_max` numbers up the tree.
         // The CFG post-order is topologically ordered w.r.t. dominance so a node comes after all
         // its dominator tree children.
-        for &ebb in domtree.cfg_postorder() {
-            if let Some(idom_inst) = domtree.idom(ebb) {
-                let idom = layout.pp_ebb(idom_inst);
-                let pre_max = cmp::max(self.nodes[ebb].pre_max, self.nodes[idom].pre_max);
+        for &block in domtree.cfg_postorder() {
+            if let Some(idom_inst) = domtree.idom(block) {
+                let idom = layout.pp_block(idom_inst);
+                let pre_max = cmp::max(self.nodes[block].pre_max, self.nodes[idom].pre_max);
                 self.nodes[idom].pre_max = pre_max;
             }
         }
     }
 }
 
-/// An iterator that enumerates the direct children of an EBB in the dominator tree.
+/// An iterator that enumerates the direct children of an block in the dominator tree.
 pub struct ChildIter<'a> {
     dtpo: &'a DominatorTreePreorder,
-    next: PackedOption,
+    next: PackedOption,
 }
 
 impl<'a> Iterator for ChildIter<'a> {
-    type Item = Ebb;
+    type Item = Block;
 
-    fn next(&mut self) -> Option {
+    fn next(&mut self) -> Option {
         let n = self.next.expand();
-        if let Some(ebb) = n {
-            self.next = self.dtpo.nodes[ebb].sibling;
+        if let Some(block) = n {
+            self.next = self.dtpo.nodes[block].sibling;
         }
         n
     }
@@ -561,32 +563,32 @@ impl<'a> Iterator for ChildIter<'a> {
 
 /// Query interface for the dominator tree pre-order.
 impl DominatorTreePreorder {
-    /// Get an iterator over the direct children of `ebb` in the dominator tree.
+    /// Get an iterator over the direct children of `block` in the dominator tree.
     ///
-    /// These are the EBB's whose immediate dominator is an instruction in `ebb`, ordered according
+    /// These are the block's whose immediate dominator is an instruction in `block`, ordered according
     /// to the CFG reverse post-order.
-    pub fn children(&self, ebb: Ebb) -> ChildIter {
+    pub fn children(&self, block: Block) -> ChildIter {
         ChildIter {
             dtpo: self,
-            next: self.nodes[ebb].child,
+            next: self.nodes[block].child,
         }
     }
 
-    /// Fast, constant time dominance check with EBB granularity.
+    /// Fast, constant time dominance check with block granularity.
     ///
     /// This computes the same result as `domtree.dominates(a, b)`, but in guaranteed fast constant
-    /// time. This is less general than the `DominatorTree` method because it only works with EBB
+    /// time. This is less general than the `DominatorTree` method because it only works with block
     /// program points.
     ///
-    /// An EBB is considered to dominate itself.
-    pub fn dominates(&self, a: Ebb, b: Ebb) -> bool {
+    /// An block is considered to dominate itself.
+    pub fn dominates(&self, a: Block, b: Block) -> bool {
         let na = &self.nodes[a];
         let nb = &self.nodes[b];
         na.pre_number <= nb.pre_number && na.pre_max >= nb.pre_max
     }
 
-    /// Compare two EBBs according to the dominator pre-order.
-    pub fn pre_cmp_ebb(&self, a: Ebb, b: Ebb) -> Ordering {
+    /// Compare two blocks according to the dominator pre-order.
+    pub fn pre_cmp_block(&self, a: Block, b: Block) -> Ordering {
         self.nodes[a].pre_number.cmp(&self.nodes[b].pre_number)
     }
 
@@ -601,7 +603,7 @@ impl DominatorTreePreorder {
     {
         let a = a.into();
         let b = b.into();
-        self.pre_cmp_ebb(layout.pp_ebb(a), layout.pp_ebb(b))
+        self.pre_cmp_block(layout.pp_block(a), layout.pp_block(b))
             .then(layout.cmp(a, b))
     }
 
@@ -643,23 +645,23 @@ mod tests {
     #[test]
     fn unreachable_node() {
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
-        let v0 = func.dfg.append_ebb_param(ebb0, I32);
-        let ebb1 = func.dfg.make_ebb();
-        let ebb2 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
+        let v0 = func.dfg.append_block_param(block0, I32);
+        let block1 = func.dfg.make_block();
+        let block2 = func.dfg.make_block();
 
         let mut cur = FuncCursor::new(&mut func);
 
-        cur.insert_ebb(ebb0);
-        cur.ins().brnz(v0, ebb2, &[]);
+        cur.insert_block(block0);
+        cur.ins().brnz(v0, block2, &[]);
         cur.ins().trap(TrapCode::User(0));
 
-        cur.insert_ebb(ebb1);
+        cur.insert_block(block1);
         let v1 = cur.ins().iconst(I32, 1);
         let v2 = cur.ins().iadd(v0, v1);
-        cur.ins().jump(ebb0, &[v2]);
+        cur.ins().jump(block0, &[v2]);
 
-        cur.insert_ebb(ebb2);
+        cur.insert_block(block2);
         cur.ins().return_(&[v0]);
 
         let cfg = ControlFlowGraph::with_function(cur.func);
@@ -667,96 +669,99 @@ mod tests {
 
         // Fall-through-first, prune-at-source DFT:
         //
-        // ebb0 {
-        //   brnz ebb2 {
+        // block0 {
+        //   brnz block2 {
         //     trap
-        //     ebb2 {
+        //     block2 {
         //       return
-        //     } ebb2
-        // } ebb0
-        assert_eq!(dt.cfg_postorder(), &[ebb2, ebb0]);
+        //     } block2
+        // } block0
+        assert_eq!(dt.cfg_postorder(), &[block2, block0]);
 
         let v2_def = cur.func.dfg.value_def(v2).unwrap_inst();
-        assert!(!dt.dominates(v2_def, ebb0, &cur.func.layout));
-        assert!(!dt.dominates(ebb0, v2_def, &cur.func.layout));
+        assert!(!dt.dominates(v2_def, block0, &cur.func.layout));
+        assert!(!dt.dominates(block0, v2_def, &cur.func.layout));
 
         let mut dtpo = DominatorTreePreorder::new();
         dtpo.compute(&dt, &cur.func.layout);
-        assert!(dtpo.dominates(ebb0, ebb0));
-        assert!(!dtpo.dominates(ebb0, ebb1));
-        assert!(dtpo.dominates(ebb0, ebb2));
-        assert!(!dtpo.dominates(ebb1, ebb0));
-        assert!(dtpo.dominates(ebb1, ebb1));
-        assert!(!dtpo.dominates(ebb1, ebb2));
-        assert!(!dtpo.dominates(ebb2, ebb0));
-        assert!(!dtpo.dominates(ebb2, ebb1));
-        assert!(dtpo.dominates(ebb2, ebb2));
+        assert!(dtpo.dominates(block0, block0));
+        assert!(!dtpo.dominates(block0, block1));
+        assert!(dtpo.dominates(block0, block2));
+        assert!(!dtpo.dominates(block1, block0));
+        assert!(dtpo.dominates(block1, block1));
+        assert!(!dtpo.dominates(block1, block2));
+        assert!(!dtpo.dominates(block2, block0));
+        assert!(!dtpo.dominates(block2, block1));
+        assert!(dtpo.dominates(block2, block2));
     }
 
     #[test]
     fn non_zero_entry_block() {
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb1 = func.dfg.make_ebb();
-        let ebb2 = func.dfg.make_ebb();
-        let ebb3 = func.dfg.make_ebb();
-        let cond = func.dfg.append_ebb_param(ebb3, I32);
+        let block0 = func.dfg.make_block();
+        let block1 = func.dfg.make_block();
+        let block2 = func.dfg.make_block();
+        let block3 = func.dfg.make_block();
+        let cond = func.dfg.append_block_param(block3, I32);
 
         let mut cur = FuncCursor::new(&mut func);
 
-        cur.insert_ebb(ebb3);
-        let jmp_ebb3_ebb1 = cur.ins().jump(ebb1, &[]);
+        cur.insert_block(block3);
+        let jmp_block3_block1 = cur.ins().jump(block1, &[]);
 
-        cur.insert_ebb(ebb1);
-        let br_ebb1_ebb0 = cur.ins().brnz(cond, ebb0, &[]);
-        let jmp_ebb1_ebb2 = cur.ins().jump(ebb2, &[]);
+        cur.insert_block(block1);
+        let br_block1_block0 = cur.ins().brnz(cond, block0, &[]);
+        let jmp_block1_block2 = cur.ins().jump(block2, &[]);
 
-        cur.insert_ebb(ebb2);
-        cur.ins().jump(ebb0, &[]);
+        cur.insert_block(block2);
+        cur.ins().jump(block0, &[]);
 
-        cur.insert_ebb(ebb0);
+        cur.insert_block(block0);
 
         let cfg = ControlFlowGraph::with_function(cur.func);
         let dt = DominatorTree::with_function(cur.func, &cfg);
 
         // Fall-through-first, prune-at-source DFT:
         //
-        // ebb3 {
-        //   ebb3:jump ebb1 {
-        //     ebb1 {
-        //       ebb1:brnz ebb0 {
-        //         ebb1:jump ebb2 {
-        //           ebb2 {
-        //             ebb2:jump ebb0 (seen)
-        //           } ebb2
-        //         } ebb1:jump ebb2
-        //         ebb0 {
-        //         } ebb0
-        //       } ebb1:brnz ebb0
-        //     } ebb1
-        //   } ebb3:jump ebb1
-        // } ebb3
+        // block3 {
+        //   block3:jump block1 {
+        //     block1 {
+        //       block1:brnz block0 {
+        //         block1:jump block2 {
+        //           block2 {
+        //             block2:jump block0 (seen)
+        //           } block2
+        //         } block1:jump block2
+        //         block0 {
+        //         } block0
+        //       } block1:brnz block0
+        //     } block1
+        //   } block3:jump block1
+        // } block3
 
-        assert_eq!(dt.cfg_postorder(), &[ebb2, ebb0, ebb1, ebb3]);
+        assert_eq!(dt.cfg_postorder(), &[block2, block0, block1, block3]);
 
-        assert_eq!(cur.func.layout.entry_block().unwrap(), ebb3);
-        assert_eq!(dt.idom(ebb3), None);
-        assert_eq!(dt.idom(ebb1).unwrap(), jmp_ebb3_ebb1);
-        assert_eq!(dt.idom(ebb2).unwrap(), jmp_ebb1_ebb2);
-        assert_eq!(dt.idom(ebb0).unwrap(), br_ebb1_ebb0);
+        assert_eq!(cur.func.layout.entry_block().unwrap(), block3);
+        assert_eq!(dt.idom(block3), None);
+        assert_eq!(dt.idom(block1).unwrap(), jmp_block3_block1);
+        assert_eq!(dt.idom(block2).unwrap(), jmp_block1_block2);
+        assert_eq!(dt.idom(block0).unwrap(), br_block1_block0);
 
-        assert!(dt.dominates(br_ebb1_ebb0, br_ebb1_ebb0, &cur.func.layout));
-        assert!(!dt.dominates(br_ebb1_ebb0, jmp_ebb3_ebb1, &cur.func.layout));
-        assert!(dt.dominates(jmp_ebb3_ebb1, br_ebb1_ebb0, &cur.func.layout));
+        assert!(dt.dominates(br_block1_block0, br_block1_block0, &cur.func.layout));
+        assert!(!dt.dominates(br_block1_block0, jmp_block3_block1, &cur.func.layout));
+        assert!(dt.dominates(jmp_block3_block1, br_block1_block0, &cur.func.layout));
 
-        assert_eq!(dt.rpo_cmp(ebb3, ebb3, &cur.func.layout), Ordering::Equal);
-        assert_eq!(dt.rpo_cmp(ebb3, ebb1, &cur.func.layout), Ordering::Less);
         assert_eq!(
-            dt.rpo_cmp(ebb3, jmp_ebb3_ebb1, &cur.func.layout),
+            dt.rpo_cmp(block3, block3, &cur.func.layout),
+            Ordering::Equal
+        );
+        assert_eq!(dt.rpo_cmp(block3, block1, &cur.func.layout), Ordering::Less);
+        assert_eq!(
+            dt.rpo_cmp(block3, jmp_block3_block1, &cur.func.layout),
             Ordering::Less
         );
         assert_eq!(
-            dt.rpo_cmp(jmp_ebb3_ebb1, jmp_ebb1_ebb2, &cur.func.layout),
+            dt.rpo_cmp(jmp_block3_block1, jmp_block1_block2, &cur.func.layout),
             Ordering::Less
         );
     }
@@ -764,69 +769,69 @@ mod tests {
     #[test]
     fn backwards_layout() {
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb1 = func.dfg.make_ebb();
-        let ebb2 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
+        let block1 = func.dfg.make_block();
+        let block2 = func.dfg.make_block();
 
         let mut cur = FuncCursor::new(&mut func);
 
-        cur.insert_ebb(ebb0);
-        let jmp02 = cur.ins().jump(ebb2, &[]);
+        cur.insert_block(block0);
+        let jmp02 = cur.ins().jump(block2, &[]);
 
-        cur.insert_ebb(ebb1);
+        cur.insert_block(block1);
         let trap = cur.ins().trap(TrapCode::User(5));
 
-        cur.insert_ebb(ebb2);
-        let jmp21 = cur.ins().jump(ebb1, &[]);
+        cur.insert_block(block2);
+        let jmp21 = cur.ins().jump(block1, &[]);
 
         let cfg = ControlFlowGraph::with_function(cur.func);
         let dt = DominatorTree::with_function(cur.func, &cfg);
 
-        assert_eq!(cur.func.layout.entry_block(), Some(ebb0));
-        assert_eq!(dt.idom(ebb0), None);
-        assert_eq!(dt.idom(ebb1), Some(jmp21));
-        assert_eq!(dt.idom(ebb2), Some(jmp02));
+        assert_eq!(cur.func.layout.entry_block(), Some(block0));
+        assert_eq!(dt.idom(block0), None);
+        assert_eq!(dt.idom(block1), Some(jmp21));
+        assert_eq!(dt.idom(block2), Some(jmp02));
 
-        assert!(dt.dominates(ebb0, ebb0, &cur.func.layout));
-        assert!(dt.dominates(ebb0, jmp02, &cur.func.layout));
-        assert!(dt.dominates(ebb0, ebb1, &cur.func.layout));
-        assert!(dt.dominates(ebb0, trap, &cur.func.layout));
-        assert!(dt.dominates(ebb0, ebb2, &cur.func.layout));
-        assert!(dt.dominates(ebb0, jmp21, &cur.func.layout));
+        assert!(dt.dominates(block0, block0, &cur.func.layout));
+        assert!(dt.dominates(block0, jmp02, &cur.func.layout));
+        assert!(dt.dominates(block0, block1, &cur.func.layout));
+        assert!(dt.dominates(block0, trap, &cur.func.layout));
+        assert!(dt.dominates(block0, block2, &cur.func.layout));
+        assert!(dt.dominates(block0, jmp21, &cur.func.layout));
 
-        assert!(!dt.dominates(jmp02, ebb0, &cur.func.layout));
+        assert!(!dt.dominates(jmp02, block0, &cur.func.layout));
         assert!(dt.dominates(jmp02, jmp02, &cur.func.layout));
-        assert!(dt.dominates(jmp02, ebb1, &cur.func.layout));
+        assert!(dt.dominates(jmp02, block1, &cur.func.layout));
         assert!(dt.dominates(jmp02, trap, &cur.func.layout));
-        assert!(dt.dominates(jmp02, ebb2, &cur.func.layout));
+        assert!(dt.dominates(jmp02, block2, &cur.func.layout));
         assert!(dt.dominates(jmp02, jmp21, &cur.func.layout));
 
-        assert!(!dt.dominates(ebb1, ebb0, &cur.func.layout));
-        assert!(!dt.dominates(ebb1, jmp02, &cur.func.layout));
-        assert!(dt.dominates(ebb1, ebb1, &cur.func.layout));
-        assert!(dt.dominates(ebb1, trap, &cur.func.layout));
-        assert!(!dt.dominates(ebb1, ebb2, &cur.func.layout));
-        assert!(!dt.dominates(ebb1, jmp21, &cur.func.layout));
+        assert!(!dt.dominates(block1, block0, &cur.func.layout));
+        assert!(!dt.dominates(block1, jmp02, &cur.func.layout));
+        assert!(dt.dominates(block1, block1, &cur.func.layout));
+        assert!(dt.dominates(block1, trap, &cur.func.layout));
+        assert!(!dt.dominates(block1, block2, &cur.func.layout));
+        assert!(!dt.dominates(block1, jmp21, &cur.func.layout));
 
-        assert!(!dt.dominates(trap, ebb0, &cur.func.layout));
+        assert!(!dt.dominates(trap, block0, &cur.func.layout));
         assert!(!dt.dominates(trap, jmp02, &cur.func.layout));
-        assert!(!dt.dominates(trap, ebb1, &cur.func.layout));
+        assert!(!dt.dominates(trap, block1, &cur.func.layout));
         assert!(dt.dominates(trap, trap, &cur.func.layout));
-        assert!(!dt.dominates(trap, ebb2, &cur.func.layout));
+        assert!(!dt.dominates(trap, block2, &cur.func.layout));
         assert!(!dt.dominates(trap, jmp21, &cur.func.layout));
 
-        assert!(!dt.dominates(ebb2, ebb0, &cur.func.layout));
-        assert!(!dt.dominates(ebb2, jmp02, &cur.func.layout));
-        assert!(dt.dominates(ebb2, ebb1, &cur.func.layout));
-        assert!(dt.dominates(ebb2, trap, &cur.func.layout));
-        assert!(dt.dominates(ebb2, ebb2, &cur.func.layout));
-        assert!(dt.dominates(ebb2, jmp21, &cur.func.layout));
+        assert!(!dt.dominates(block2, block0, &cur.func.layout));
+        assert!(!dt.dominates(block2, jmp02, &cur.func.layout));
+        assert!(dt.dominates(block2, block1, &cur.func.layout));
+        assert!(dt.dominates(block2, trap, &cur.func.layout));
+        assert!(dt.dominates(block2, block2, &cur.func.layout));
+        assert!(dt.dominates(block2, jmp21, &cur.func.layout));
 
-        assert!(!dt.dominates(jmp21, ebb0, &cur.func.layout));
+        assert!(!dt.dominates(jmp21, block0, &cur.func.layout));
         assert!(!dt.dominates(jmp21, jmp02, &cur.func.layout));
-        assert!(dt.dominates(jmp21, ebb1, &cur.func.layout));
+        assert!(dt.dominates(jmp21, block1, &cur.func.layout));
         assert!(dt.dominates(jmp21, trap, &cur.func.layout));
-        assert!(!dt.dominates(jmp21, ebb2, &cur.func.layout));
+        assert!(!dt.dominates(jmp21, block2, &cur.func.layout));
         assert!(dt.dominates(jmp21, jmp21, &cur.func.layout));
     }
 }
diff --git a/cranelift/codegen/src/flowgraph.rs b/cranelift/codegen/src/flowgraph.rs
index f53cfa6bff..37245da912 100644
--- a/cranelift/codegen/src/flowgraph.rs
+++ b/cranelift/codegen/src/flowgraph.rs
@@ -1,80 +1,80 @@
-//! A control flow graph represented as mappings of extended basic blocks to their predecessors
+//! A control flow graph represented as mappings of basic blocks to their predecessors
 //! and successors.
 //!
-//! Successors are represented as extended basic blocks while predecessors are represented by basic
-//! blocks. Basic blocks are denoted by tuples of EBB and branch/jump instructions. Each
+//! Successors are represented as basic blocks while predecessors are represented by basic
+//! blocks. Basic blocks are denoted by tuples of block and branch/jump instructions. Each
 //! predecessor tuple corresponds to the end of a basic block.
 //!
 //! ```c
-//!     Ebb0:
+//!     Block0:
 //!         ...          ; beginning of basic block
 //!
 //!         ...
 //!
-//!         brz vx, Ebb1 ; end of basic block
+//!         brz vx, Block1 ; end of basic block
 //!
 //!         ...          ; beginning of basic block
 //!
 //!         ...
 //!
-//!         jmp Ebb2     ; end of basic block
+//!         jmp Block2     ; end of basic block
 //! ```
 //!
-//! Here `Ebb1` and `Ebb2` would each have a single predecessor denoted as `(Ebb0, brz)`
-//! and `(Ebb0, jmp Ebb2)` respectively.
+//! Here `Block1` and `Block2` would each have a single predecessor denoted as `(Block0, brz)`
+//! and `(Block0, jmp Block2)` respectively.
 
 use crate::bforest;
 use crate::entity::SecondaryMap;
 use crate::ir::instructions::BranchInfo;
-use crate::ir::{Ebb, Function, Inst};
+use crate::ir::{Block, Function, Inst};
 use crate::timing;
 use core::mem;
 
-/// A basic block denoted by its enclosing Ebb and last instruction.
+/// A basic block denoted by its enclosing Block and last instruction.
 #[derive(Debug, PartialEq, Eq)]
-pub struct BasicBlock {
-    /// Enclosing Ebb key.
-    pub ebb: Ebb,
+pub struct BlockPredecessor {
+    /// Enclosing Block key.
+    pub block: Block,
     /// Last instruction in the basic block.
     pub inst: Inst,
 }
 
-impl BasicBlock {
-    /// Convenient method to construct new BasicBlock.
-    pub fn new(ebb: Ebb, inst: Inst) -> Self {
-        Self { ebb, inst }
+impl BlockPredecessor {
+    /// Convenient method to construct new BlockPredecessor.
+    pub fn new(block: Block, inst: Inst) -> Self {
+        Self { block, inst }
     }
 }
 
-/// A container for the successors and predecessors of some Ebb.
+/// A container for the successors and predecessors of some Block.
 #[derive(Clone, Default)]
 struct CFGNode {
-    /// Instructions that can branch or jump to this EBB.
+    /// Instructions that can branch or jump to this block.
     ///
-    /// This maps branch instruction -> predecessor EBB which is redundant since the EBB containing
-    /// the branch instruction is available from the `layout.inst_ebb()` method. We store the
+    /// This maps branch instruction -> predecessor block which is redundant since the block containing
+    /// the branch instruction is available from the `layout.inst_block()` method. We store the
     /// redundant information because:
     ///
-    /// 1. Many `pred_iter()` consumers want the EBB anyway, so it is handily available.
-    /// 2. The `invalidate_ebb_successors()` may be called *after* branches have been removed from
-    ///    their EBB, but we still need to remove them form the old EBB predecessor map.
+    /// 1. Many `pred_iter()` consumers want the block anyway, so it is handily available.
+    /// 2. The `invalidate_block_successors()` may be called *after* branches have been removed from
+    ///    their block, but we still need to remove them form the old block predecessor map.
     ///
-    /// The redundant EBB stored here is always consistent with the CFG successor lists, even after
+    /// The redundant block stored here is always consistent with the CFG successor lists, even after
     /// the IR has been edited.
-    pub predecessors: bforest::Map,
+    pub predecessors: bforest::Map,
 
-    /// Set of EBBs that are the targets of branches and jumps in this EBB.
-    /// The set is ordered by EBB number, indicated by the `()` comparator type.
-    pub successors: bforest::Set,
+    /// Set of blocks that are the targets of branches and jumps in this block.
+    /// The set is ordered by block number, indicated by the `()` comparator type.
+    pub successors: bforest::Set,
 }
 
-/// The Control Flow Graph maintains a mapping of ebbs to their predecessors
+/// The Control Flow Graph maintains a mapping of blocks to their predecessors
 /// and successors where predecessors are basic blocks and successors are
-/// extended basic blocks.
+/// basic blocks.
 pub struct ControlFlowGraph {
-    data: SecondaryMap,
-    pred_forest: bforest::MapForest,
-    succ_forest: bforest::SetForest,
+    data: SecondaryMap,
+    pred_forest: bforest::MapForest,
+    succ_forest: bforest::SetForest,
     valid: bool,
 }
 
@@ -110,27 +110,27 @@ impl ControlFlowGraph {
     pub fn compute(&mut self, func: &Function) {
         let _tt = timing::flowgraph();
         self.clear();
-        self.data.resize(func.dfg.num_ebbs());
+        self.data.resize(func.dfg.num_blocks());
 
-        for ebb in &func.layout {
-            self.compute_ebb(func, ebb);
+        for block in &func.layout {
+            self.compute_block(func, block);
         }
 
         self.valid = true;
     }
 
-    fn compute_ebb(&mut self, func: &Function, ebb: Ebb) {
-        for inst in func.layout.ebb_insts(ebb) {
+    fn compute_block(&mut self, func: &Function, block: Block) {
+        for inst in func.layout.block_insts(block) {
             match func.dfg.analyze_branch(inst) {
                 BranchInfo::SingleDest(dest, _) => {
-                    self.add_edge(ebb, inst, dest);
+                    self.add_edge(block, inst, dest);
                 }
                 BranchInfo::Table(jt, dest) => {
                     if let Some(dest) = dest {
-                        self.add_edge(ebb, inst, dest);
+                        self.add_edge(block, inst, dest);
                     }
                     for dest in func.jump_tables[jt].iter() {
-                        self.add_edge(ebb, inst, *dest);
+                        self.add_edge(block, inst, *dest);
                     }
                 }
                 BranchInfo::NotABranch => {}
@@ -138,32 +138,32 @@ impl ControlFlowGraph {
         }
     }
 
-    fn invalidate_ebb_successors(&mut self, ebb: Ebb) {
+    fn invalidate_block_successors(&mut self, block: Block) {
         // Temporarily take ownership because we need mutable access to self.data inside the loop.
         // Unfortunately borrowck cannot see that our mut accesses to predecessors don't alias
         // our iteration over successors.
-        let mut successors = mem::replace(&mut self.data[ebb].successors, Default::default());
+        let mut successors = mem::replace(&mut self.data[block].successors, Default::default());
         for succ in successors.iter(&self.succ_forest) {
             self.data[succ]
                 .predecessors
-                .retain(&mut self.pred_forest, |_, &mut e| e != ebb);
+                .retain(&mut self.pred_forest, |_, &mut e| e != block);
         }
         successors.clear(&mut self.succ_forest);
     }
 
-    /// Recompute the control flow graph of `ebb`.
+    /// Recompute the control flow graph of `block`.
     ///
-    /// This is for use after modifying instructions within a specific EBB. It recomputes all edges
-    /// from `ebb` while leaving edges to `ebb` intact. Its functionality a subset of that of the
+    /// This is for use after modifying instructions within a specific block. It recomputes all edges
+    /// from `block` while leaving edges to `block` intact. Its functionality a subset of that of the
     /// more expensive `compute`, and should be used when we know we don't need to recompute the CFG
-    /// from scratch, but rather that our changes have been restricted to specific EBBs.
-    pub fn recompute_ebb(&mut self, func: &Function, ebb: Ebb) {
+    /// from scratch, but rather that our changes have been restricted to specific blocks.
+    pub fn recompute_block(&mut self, func: &Function, block: Block) {
         debug_assert!(self.is_valid());
-        self.invalidate_ebb_successors(ebb);
-        self.compute_ebb(func, ebb);
+        self.invalidate_block_successors(block);
+        self.compute_block(func, block);
     }
 
-    fn add_edge(&mut self, from: Ebb, from_inst: Inst, to: Ebb) {
+    fn add_edge(&mut self, from: Block, from_inst: Inst, to: Block) {
         self.data[from]
             .successors
             .insert(to, &mut self.succ_forest, &());
@@ -172,15 +172,15 @@ impl ControlFlowGraph {
             .insert(from_inst, from, &mut self.pred_forest, &());
     }
 
-    /// Get an iterator over the CFG predecessors to `ebb`.
-    pub fn pred_iter(&self, ebb: Ebb) -> PredIter {
-        PredIter(self.data[ebb].predecessors.iter(&self.pred_forest))
+    /// Get an iterator over the CFG predecessors to `block`.
+    pub fn pred_iter(&self, block: Block) -> PredIter {
+        PredIter(self.data[block].predecessors.iter(&self.pred_forest))
     }
 
-    /// Get an iterator over the CFG successors to `ebb`.
-    pub fn succ_iter(&self, ebb: Ebb) -> SuccIter {
+    /// Get an iterator over the CFG successors to `block`.
+    pub fn succ_iter(&self, block: Block) -> SuccIter {
         debug_assert!(self.is_valid());
-        self.data[ebb].successors.iter(&self.succ_forest)
+        self.data[block].successors.iter(&self.succ_forest)
     }
 
     /// Check if the CFG is in a valid state.
@@ -193,21 +193,21 @@ impl ControlFlowGraph {
     }
 }
 
-/// An iterator over EBB predecessors. The iterator type is `BasicBlock`.
+/// An iterator over block predecessors. The iterator type is `BlockPredecessor`.
 ///
-/// Each predecessor is an instruction that branches to the EBB.
-pub struct PredIter<'a>(bforest::MapIter<'a, Inst, Ebb>);
+/// Each predecessor is an instruction that branches to the block.
+pub struct PredIter<'a>(bforest::MapIter<'a, Inst, Block>);
 
 impl<'a> Iterator for PredIter<'a> {
-    type Item = BasicBlock;
+    type Item = BlockPredecessor;
 
-    fn next(&mut self) -> Option {
-        self.0.next().map(|(i, e)| BasicBlock::new(e, i))
+    fn next(&mut self) -> Option {
+        self.0.next().map(|(i, e)| BlockPredecessor::new(e, i))
     }
 }
 
-/// An iterator over EBB successors. The iterator type is `Ebb`.
-pub type SuccIter<'a> = bforest::SetIter<'a, Ebb>;
+/// An iterator over block successors. The iterator type is `Block`.
+pub type SuccIter<'a> = bforest::SetIter<'a, Block>;
 
 #[cfg(test)]
 mod tests {
@@ -225,126 +225,126 @@ mod tests {
     #[test]
     fn no_predecessors() {
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb1 = func.dfg.make_ebb();
-        let ebb2 = func.dfg.make_ebb();
-        func.layout.append_ebb(ebb0);
-        func.layout.append_ebb(ebb1);
-        func.layout.append_ebb(ebb2);
+        let block0 = func.dfg.make_block();
+        let block1 = func.dfg.make_block();
+        let block2 = func.dfg.make_block();
+        func.layout.append_block(block0);
+        func.layout.append_block(block1);
+        func.layout.append_block(block2);
 
         let cfg = ControlFlowGraph::with_function(&func);
 
-        let mut fun_ebbs = func.layout.ebbs();
-        for ebb in func.layout.ebbs() {
-            assert_eq!(ebb, fun_ebbs.next().unwrap());
-            assert_eq!(cfg.pred_iter(ebb).count(), 0);
-            assert_eq!(cfg.succ_iter(ebb).count(), 0);
+        let mut fun_blocks = func.layout.blocks();
+        for block in func.layout.blocks() {
+            assert_eq!(block, fun_blocks.next().unwrap());
+            assert_eq!(cfg.pred_iter(block).count(), 0);
+            assert_eq!(cfg.succ_iter(block).count(), 0);
         }
     }
 
     #[test]
     fn branches_and_jumps() {
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
-        let cond = func.dfg.append_ebb_param(ebb0, types::I32);
-        let ebb1 = func.dfg.make_ebb();
-        let ebb2 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
+        let cond = func.dfg.append_block_param(block0, types::I32);
+        let block1 = func.dfg.make_block();
+        let block2 = func.dfg.make_block();
 
-        let br_ebb0_ebb2;
-        let br_ebb1_ebb1;
-        let jmp_ebb0_ebb1;
-        let jmp_ebb1_ebb2;
+        let br_block0_block2;
+        let br_block1_block1;
+        let jmp_block0_block1;
+        let jmp_block1_block2;
 
         {
             let mut cur = FuncCursor::new(&mut func);
 
-            cur.insert_ebb(ebb0);
-            br_ebb0_ebb2 = cur.ins().brnz(cond, ebb2, &[]);
-            jmp_ebb0_ebb1 = cur.ins().jump(ebb1, &[]);
+            cur.insert_block(block0);
+            br_block0_block2 = cur.ins().brnz(cond, block2, &[]);
+            jmp_block0_block1 = cur.ins().jump(block1, &[]);
 
-            cur.insert_ebb(ebb1);
-            br_ebb1_ebb1 = cur.ins().brnz(cond, ebb1, &[]);
-            jmp_ebb1_ebb2 = cur.ins().jump(ebb2, &[]);
+            cur.insert_block(block1);
+            br_block1_block1 = cur.ins().brnz(cond, block1, &[]);
+            jmp_block1_block2 = cur.ins().jump(block2, &[]);
 
-            cur.insert_ebb(ebb2);
+            cur.insert_block(block2);
         }
 
         let mut cfg = ControlFlowGraph::with_function(&func);
 
         {
-            let ebb0_predecessors = cfg.pred_iter(ebb0).collect::>();
-            let ebb1_predecessors = cfg.pred_iter(ebb1).collect::>();
-            let ebb2_predecessors = cfg.pred_iter(ebb2).collect::>();
+            let block0_predecessors = cfg.pred_iter(block0).collect::>();
+            let block1_predecessors = cfg.pred_iter(block1).collect::>();
+            let block2_predecessors = cfg.pred_iter(block2).collect::>();
 
-            let ebb0_successors = cfg.succ_iter(ebb0).collect::>();
-            let ebb1_successors = cfg.succ_iter(ebb1).collect::>();
-            let ebb2_successors = cfg.succ_iter(ebb2).collect::>();
+            let block0_successors = cfg.succ_iter(block0).collect::>();
+            let block1_successors = cfg.succ_iter(block1).collect::>();
+            let block2_successors = cfg.succ_iter(block2).collect::>();
 
-            assert_eq!(ebb0_predecessors.len(), 0);
-            assert_eq!(ebb1_predecessors.len(), 2);
-            assert_eq!(ebb2_predecessors.len(), 2);
+            assert_eq!(block0_predecessors.len(), 0);
+            assert_eq!(block1_predecessors.len(), 2);
+            assert_eq!(block2_predecessors.len(), 2);
 
             assert_eq!(
-                ebb1_predecessors.contains(&BasicBlock::new(ebb0, jmp_ebb0_ebb1)),
+                block1_predecessors.contains(&BlockPredecessor::new(block0, jmp_block0_block1)),
                 true
             );
             assert_eq!(
-                ebb1_predecessors.contains(&BasicBlock::new(ebb1, br_ebb1_ebb1)),
+                block1_predecessors.contains(&BlockPredecessor::new(block1, br_block1_block1)),
                 true
             );
             assert_eq!(
-                ebb2_predecessors.contains(&BasicBlock::new(ebb0, br_ebb0_ebb2)),
+                block2_predecessors.contains(&BlockPredecessor::new(block0, br_block0_block2)),
                 true
             );
             assert_eq!(
-                ebb2_predecessors.contains(&BasicBlock::new(ebb1, jmp_ebb1_ebb2)),
+                block2_predecessors.contains(&BlockPredecessor::new(block1, jmp_block1_block2)),
                 true
             );
 
-            assert_eq!(ebb0_successors, [ebb1, ebb2]);
-            assert_eq!(ebb1_successors, [ebb1, ebb2]);
-            assert_eq!(ebb2_successors, []);
+            assert_eq!(block0_successors, [block1, block2]);
+            assert_eq!(block1_successors, [block1, block2]);
+            assert_eq!(block2_successors, []);
         }
 
-        // Change some instructions and recompute ebb0
-        func.dfg.replace(br_ebb0_ebb2).brnz(cond, ebb1, &[]);
-        func.dfg.replace(jmp_ebb0_ebb1).return_(&[]);
-        cfg.recompute_ebb(&mut func, ebb0);
-        let br_ebb0_ebb1 = br_ebb0_ebb2;
+        // Change some instructions and recompute block0
+        func.dfg.replace(br_block0_block2).brnz(cond, block1, &[]);
+        func.dfg.replace(jmp_block0_block1).return_(&[]);
+        cfg.recompute_block(&mut func, block0);
+        let br_block0_block1 = br_block0_block2;
 
         {
-            let ebb0_predecessors = cfg.pred_iter(ebb0).collect::>();
-            let ebb1_predecessors = cfg.pred_iter(ebb1).collect::>();
-            let ebb2_predecessors = cfg.pred_iter(ebb2).collect::>();
+            let block0_predecessors = cfg.pred_iter(block0).collect::>();
+            let block1_predecessors = cfg.pred_iter(block1).collect::>();
+            let block2_predecessors = cfg.pred_iter(block2).collect::>();
 
-            let ebb0_successors = cfg.succ_iter(ebb0);
-            let ebb1_successors = cfg.succ_iter(ebb1);
-            let ebb2_successors = cfg.succ_iter(ebb2);
+            let block0_successors = cfg.succ_iter(block0);
+            let block1_successors = cfg.succ_iter(block1);
+            let block2_successors = cfg.succ_iter(block2);
 
-            assert_eq!(ebb0_predecessors.len(), 0);
-            assert_eq!(ebb1_predecessors.len(), 2);
-            assert_eq!(ebb2_predecessors.len(), 1);
+            assert_eq!(block0_predecessors.len(), 0);
+            assert_eq!(block1_predecessors.len(), 2);
+            assert_eq!(block2_predecessors.len(), 1);
 
             assert_eq!(
-                ebb1_predecessors.contains(&BasicBlock::new(ebb0, br_ebb0_ebb1)),
+                block1_predecessors.contains(&BlockPredecessor::new(block0, br_block0_block1)),
                 true
             );
             assert_eq!(
-                ebb1_predecessors.contains(&BasicBlock::new(ebb1, br_ebb1_ebb1)),
+                block1_predecessors.contains(&BlockPredecessor::new(block1, br_block1_block1)),
                 true
             );
             assert_eq!(
-                ebb2_predecessors.contains(&BasicBlock::new(ebb0, br_ebb0_ebb2)),
+                block2_predecessors.contains(&BlockPredecessor::new(block0, br_block0_block2)),
                 false
             );
             assert_eq!(
-                ebb2_predecessors.contains(&BasicBlock::new(ebb1, jmp_ebb1_ebb2)),
+                block2_predecessors.contains(&BlockPredecessor::new(block1, jmp_block1_block2)),
                 true
             );
 
-            assert_eq!(ebb0_successors.collect::>(), [ebb1]);
-            assert_eq!(ebb1_successors.collect::>(), [ebb1, ebb2]);
-            assert_eq!(ebb2_successors.collect::>(), []);
+            assert_eq!(block0_successors.collect::>(), [block1]);
+            assert_eq!(block1_successors.collect::>(), [block1, block2]);
+            assert_eq!(block2_successors.collect::>(), []);
         }
     }
 }
diff --git a/cranelift/codegen/src/ir/builder.rs b/cranelift/codegen/src/ir/builder.rs
index 13202870f3..63054928f2 100644
--- a/cranelift/codegen/src/ir/builder.rs
+++ b/cranelift/codegen/src/ir/builder.rs
@@ -223,10 +223,10 @@ mod tests {
     #[test]
     fn types() {
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
-        let arg0 = func.dfg.append_ebb_param(ebb0, I32);
+        let block0 = func.dfg.make_block();
+        let arg0 = func.dfg.append_block_param(block0, I32);
         let mut pos = FuncCursor::new(&mut func);
-        pos.insert_ebb(ebb0);
+        pos.insert_block(block0);
 
         // Explicit types.
         let v0 = pos.ins().iconst(I32, 3);
@@ -244,10 +244,10 @@ mod tests {
     #[test]
     fn reuse_results() {
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
-        let arg0 = func.dfg.append_ebb_param(ebb0, I32);
+        let block0 = func.dfg.make_block();
+        let arg0 = func.dfg.append_block_param(block0, I32);
         let mut pos = FuncCursor::new(&mut func);
-        pos.insert_ebb(ebb0);
+        pos.insert_block(block0);
 
         let v0 = pos.ins().iadd_imm(arg0, 17);
         assert_eq!(pos.func.dfg.value_type(v0), I32);
diff --git a/cranelift/codegen/src/ir/dfg.rs b/cranelift/codegen/src/ir/dfg.rs
index 7d453836f5..479aec1cfb 100644
--- a/cranelift/codegen/src/ir/dfg.rs
+++ b/cranelift/codegen/src/ir/dfg.rs
@@ -1,4 +1,4 @@
-//! Data flow graph tracking Instructions, Values, and EBBs.
+//! Data flow graph tracking Instructions, Values, and blocks.
 
 use crate::entity::{self, PrimaryMap, SecondaryMap};
 use crate::ir;
@@ -7,7 +7,7 @@ use crate::ir::extfunc::ExtFuncData;
 use crate::ir::instructions::{BranchInfo, CallInfo, InstructionData};
 use crate::ir::{types, ConstantData, ConstantPool, Immediate};
 use crate::ir::{
-    Ebb, FuncRef, Inst, SigRef, Signature, Type, Value, ValueLabelAssignments, ValueList,
+    Block, FuncRef, Inst, SigRef, Signature, Type, Value, ValueLabelAssignments, ValueList,
     ValueListPool,
 };
 use crate::isa::TargetIsa;
@@ -21,18 +21,18 @@ use core::mem;
 use core::ops::{Index, IndexMut};
 use core::u16;
 
-/// A data flow graph defines all instructions and extended basic blocks in a function as well as
+/// A data flow graph defines all instructions and basic blocks in a function as well as
 /// the data flow dependencies between them. The DFG also tracks values which can be either
-/// instruction results or EBB parameters.
+/// instruction results or block parameters.
 ///
-/// The layout of EBBs in the function and of instructions in each EBB is recorded by the
+/// The layout of blocks in the function and of instructions in each block is recorded by the
 /// `Layout` data structure which forms the other half of the function representation.
 ///
 #[derive(Clone)]
 pub struct DataFlowGraph {
     /// Data about all of the instructions in the function, including opcodes and operands.
     /// The instructions in this map are not in program order. That is tracked by `Layout`, along
-    /// with the EBB containing each instruction.
+    /// with the block containing each instruction.
     insts: PrimaryMap,
 
     /// List of result values for each instruction.
@@ -41,11 +41,11 @@ pub struct DataFlowGraph {
     /// primary `insts` map.
     results: SecondaryMap,
 
-    /// Extended basic blocks in the function and their parameters.
+    /// basic blocks in the function and their parameters.
     ///
     /// This map is not in program order. That is handled by `Layout`, and so is the sequence of
-    /// instructions contained in each EBB.
-    ebbs: PrimaryMap,
+    /// instructions contained in each block.
+    blocks: PrimaryMap,
 
     /// Memory pool of value lists.
     ///
@@ -53,7 +53,7 @@ pub struct DataFlowGraph {
     ///
     /// - Instructions in `insts` that don't have room for their entire argument list inline.
     /// - Instruction result values in `results`.
-    /// - EBB parameters in `ebbs`.
+    /// - block parameters in `blocks`.
     pub value_lists: ValueListPool,
 
     /// Primary value table with entries for all values.
@@ -85,7 +85,7 @@ impl DataFlowGraph {
         Self {
             insts: PrimaryMap::new(),
             results: SecondaryMap::new(),
-            ebbs: PrimaryMap::new(),
+            blocks: PrimaryMap::new(),
             value_lists: ValueListPool::new(),
             values: PrimaryMap::new(),
             signatures: PrimaryMap::new(),
@@ -101,7 +101,7 @@ impl DataFlowGraph {
     pub fn clear(&mut self) {
         self.insts.clear();
         self.results.clear();
-        self.ebbs.clear();
+        self.blocks.clear();
         self.value_lists.clear();
         self.values.clear();
         self.signatures.clear();
@@ -125,17 +125,17 @@ impl DataFlowGraph {
         self.insts.is_valid(inst)
     }
 
-    /// Get the total number of extended basic blocks created in this function, whether they are
+    /// Get the total number of basic blocks created in this function, whether they are
     /// currently inserted in the layout or not.
     ///
     /// This is intended for use with `SecondaryMap::with_capacity`.
-    pub fn num_ebbs(&self) -> usize {
-        self.ebbs.len()
+    pub fn num_blocks(&self) -> usize {
+        self.blocks.len()
     }
 
-    /// Returns `true` if the given ebb reference is valid.
-    pub fn ebb_is_valid(&self, ebb: Ebb) -> bool {
-        self.ebbs.is_valid(ebb)
+    /// Returns `true` if the given block reference is valid.
+    pub fn block_is_valid(&self, block: Block) -> bool {
+        self.blocks.is_valid(block)
     }
 
     /// Get the total number of values.
@@ -213,7 +213,7 @@ impl<'a> Iterator for Values<'a> {
 
 /// Handling values.
 ///
-/// Values are either EBB parameters or instruction results.
+/// Values are either block parameters or instruction results.
 impl DataFlowGraph {
     /// Allocate an extended value entry.
     fn make_value(&mut self, data: ValueData) -> Value {
@@ -243,12 +243,12 @@ impl DataFlowGraph {
 
     /// Get the definition of a value.
     ///
-    /// This is either the instruction that defined it or the Ebb that has the value as an
+    /// This is either the instruction that defined it or the Block that has the value as an
     /// parameter.
     pub fn value_def(&self, v: Value) -> ValueDef {
         match self.values[v] {
             ValueData::Inst { inst, num, .. } => ValueDef::Result(inst, num as usize),
-            ValueData::Param { ebb, num, .. } => ValueDef::Param(ebb, num as usize),
+            ValueData::Param { block, num, .. } => ValueDef::Param(block, num as usize),
             ValueData::Alias { original, .. } => {
                 // Make sure we only recurse one level. `resolve_aliases` has safeguards to
                 // detect alias loops without overrunning the stack.
@@ -257,7 +257,7 @@ impl DataFlowGraph {
         }
     }
 
-    /// Determine if `v` is an attached instruction result / EBB parameter.
+    /// Determine if `v` is an attached instruction result / block parameter.
     ///
     /// An attached value can't be attached to something else without first being detached.
     ///
@@ -267,7 +267,7 @@ impl DataFlowGraph {
         use self::ValueData::*;
         match self.values[v] {
             Inst { inst, num, .. } => Some(&v) == self.inst_results(inst).get(num as usize),
-            Param { ebb, num, .. } => Some(&v) == self.ebb_params(ebb).get(num as usize),
+            Param { block, num, .. } => Some(&v) == self.block_params(block).get(num as usize),
             Alias { .. } => false,
         }
     }
@@ -297,7 +297,7 @@ impl DataFlowGraph {
     /// Change the `dest` value to behave as an alias of `src`. This means that all uses of `dest`
     /// will behave as if they used that value `src`.
     ///
-    /// The `dest` value can't be attached to an instruction or EBB.
+    /// The `dest` value can't be attached to an instruction or block.
     pub fn change_to_alias(&mut self, dest: Value, src: Value) {
         debug_assert!(!self.value_is_attached(dest));
         // Try to create short alias chains by finding the original source value.
@@ -376,8 +376,8 @@ impl DataFlowGraph {
 pub enum ValueDef {
     /// Value is the n'th result of an instruction.
     Result(Inst, usize),
-    /// Value is the n'th parameter to an EBB.
-    Param(Ebb, usize),
+    /// Value is the n'th parameter to an block.
+    Param(Block, usize),
 }
 
 impl ValueDef {
@@ -389,11 +389,11 @@ impl ValueDef {
         }
     }
 
-    /// Unwrap the EBB there the parameter is defined, or panic.
-    pub fn unwrap_ebb(&self) -> Ebb {
+    /// Unwrap the block there the parameter is defined, or panic.
+    pub fn unwrap_block(&self) -> Block {
         match *self {
-            Self::Param(ebb, _) => ebb,
-            _ => panic!("Value is not an EBB parameter"),
+            Self::Param(block, _) => block,
+            _ => panic!("Value is not an block parameter"),
         }
     }
 
@@ -419,12 +419,12 @@ enum ValueData {
     /// Value is defined by an instruction.
     Inst { ty: Type, num: u16, inst: Inst },
 
-    /// Value is an EBB parameter.
-    Param { ty: Type, num: u16, ebb: Ebb },
+    /// Value is an block parameter.
+    Param { ty: Type, num: u16, block: Block },
 
     /// Value is an alias of another value.
-    /// An alias value can't be linked as an instruction result or EBB parameter. It is used as a
-    /// placeholder when the original instruction or EBB has been rewritten or modified.
+    /// An alias value can't be linked as an instruction result or block parameter. It is used as a
+    /// placeholder when the original instruction or block has been rewritten or modified.
     Alias { ty: Type, original: Value },
 }
 
@@ -760,61 +760,64 @@ impl IndexMut for DataFlowGraph {
     }
 }
 
-/// Extended basic blocks.
+/// basic blocks.
 impl DataFlowGraph {
     /// Create a new basic block.
-    pub fn make_ebb(&mut self) -> Ebb {
-        self.ebbs.push(EbbData::new())
+    pub fn make_block(&mut self) -> Block {
+        self.blocks.push(BlockData::new())
     }
 
-    /// Get the number of parameters on `ebb`.
-    pub fn num_ebb_params(&self, ebb: Ebb) -> usize {
-        self.ebbs[ebb].params.len(&self.value_lists)
+    /// Get the number of parameters on `block`.
+    pub fn num_block_params(&self, block: Block) -> usize {
+        self.blocks[block].params.len(&self.value_lists)
     }
 
-    /// Get the parameters on `ebb`.
-    pub fn ebb_params(&self, ebb: Ebb) -> &[Value] {
-        self.ebbs[ebb].params.as_slice(&self.value_lists)
+    /// Get the parameters on `block`.
+    pub fn block_params(&self, block: Block) -> &[Value] {
+        self.blocks[block].params.as_slice(&self.value_lists)
     }
 
-    /// Get the types of the parameters on `ebb`.
-    pub fn ebb_param_types(&self, ebb: Ebb) -> Vec {
-        self.ebb_params(ebb)
+    /// Get the types of the parameters on `block`.
+    pub fn block_param_types(&self, block: Block) -> Vec {
+        self.block_params(block)
             .iter()
             .map(|&v| self.value_type(v))
             .collect()
     }
 
-    /// Append a parameter with type `ty` to `ebb`.
-    pub fn append_ebb_param(&mut self, ebb: Ebb, ty: Type) -> Value {
+    /// Append a parameter with type `ty` to `block`.
+    pub fn append_block_param(&mut self, block: Block, ty: Type) -> Value {
         let param = self.values.next_key();
-        let num = self.ebbs[ebb].params.push(param, &mut self.value_lists);
-        debug_assert!(num <= u16::MAX as usize, "Too many parameters on EBB");
+        let num = self.blocks[block].params.push(param, &mut self.value_lists);
+        debug_assert!(num <= u16::MAX as usize, "Too many parameters on block");
         self.make_value(ValueData::Param {
             ty,
             num: num as u16,
-            ebb,
+            block,
         })
     }
 
-    /// Removes `val` from `ebb`'s parameters by swapping it with the last parameter on `ebb`.
+    /// Removes `val` from `block`'s parameters by swapping it with the last parameter on `block`.
     /// Returns the position of `val` before removal.
     ///
     /// *Important*: to ensure O(1) deletion, this method swaps the removed parameter with the
-    /// last `ebb` parameter. This can disrupt all the branch instructions jumping to this
-    /// `ebb` for which you have to change the branch argument order if necessary.
+    /// last `block` parameter. This can disrupt all the branch instructions jumping to this
+    /// `block` for which you have to change the branch argument order if necessary.
     ///
-    /// Panics if `val` is not an EBB parameter.
-    pub fn swap_remove_ebb_param(&mut self, val: Value) -> usize {
-        let (ebb, num) = if let ValueData::Param { num, ebb, .. } = self.values[val] {
-            (ebb, num)
+    /// Panics if `val` is not an block parameter.
+    pub fn swap_remove_block_param(&mut self, val: Value) -> usize {
+        let (block, num) = if let ValueData::Param { num, block, .. } = self.values[val] {
+            (block, num)
         } else {
-            panic!("{} must be an EBB parameter", val);
+            panic!("{} must be an block parameter", val);
         };
-        self.ebbs[ebb]
+        self.blocks[block]
             .params
             .swap_remove(num as usize, &mut self.value_lists);
-        if let Some(last_arg_val) = self.ebbs[ebb].params.get(num as usize, &self.value_lists) {
+        if let Some(last_arg_val) = self.blocks[block]
+            .params
+            .get(num as usize, &self.value_lists)
+        {
             // We update the position of the old last arg.
             if let ValueData::Param {
                 num: ref mut old_num,
@@ -823,25 +826,25 @@ impl DataFlowGraph {
             {
                 *old_num = num;
             } else {
-                panic!("{} should be an Ebb parameter", last_arg_val);
+                panic!("{} should be an Block parameter", last_arg_val);
             }
         }
         num as usize
     }
 
-    /// Removes `val` from `ebb`'s parameters by a standard linear time list removal which
+    /// Removes `val` from `block`'s parameters by a standard linear time list removal which
     /// preserves ordering. Also updates the values' data.
-    pub fn remove_ebb_param(&mut self, val: Value) {
-        let (ebb, num) = if let ValueData::Param { num, ebb, .. } = self.values[val] {
-            (ebb, num)
+    pub fn remove_block_param(&mut self, val: Value) {
+        let (block, num) = if let ValueData::Param { num, block, .. } = self.values[val] {
+            (block, num)
         } else {
-            panic!("{} must be an EBB parameter", val);
+            panic!("{} must be an block parameter", val);
         };
-        self.ebbs[ebb]
+        self.blocks[block]
             .params
             .remove(num as usize, &mut self.value_lists);
-        for index in num..(self.num_ebb_params(ebb) as u16) {
-            match self.values[self.ebbs[ebb]
+        for index in num..(self.num_block_params(block) as u16) {
+            match self.values[self.blocks[block]
                 .params
                 .get(index as usize, &self.value_lists)
                 .unwrap()]
@@ -850,8 +853,8 @@ impl DataFlowGraph {
                     *num -= 1;
                 }
                 _ => panic!(
-                    "{} must be an EBB parameter",
-                    self.ebbs[ebb]
+                    "{} must be an block parameter",
+                    self.blocks[block]
                         .params
                         .get(index as usize, &self.value_lists)
                         .unwrap()
@@ -860,71 +863,73 @@ impl DataFlowGraph {
         }
     }
 
-    /// Append an existing value to `ebb`'s parameters.
+    /// Append an existing value to `block`'s parameters.
     ///
     /// The appended value can't already be attached to something else.
     ///
-    /// In almost all cases, you should be using `append_ebb_param()` instead of this method.
-    pub fn attach_ebb_param(&mut self, ebb: Ebb, param: Value) {
+    /// In almost all cases, you should be using `append_block_param()` instead of this method.
+    pub fn attach_block_param(&mut self, block: Block, param: Value) {
         debug_assert!(!self.value_is_attached(param));
-        let num = self.ebbs[ebb].params.push(param, &mut self.value_lists);
-        debug_assert!(num <= u16::MAX as usize, "Too many parameters on EBB");
+        let num = self.blocks[block].params.push(param, &mut self.value_lists);
+        debug_assert!(num <= u16::MAX as usize, "Too many parameters on block");
         let ty = self.value_type(param);
         self.values[param] = ValueData::Param {
             ty,
             num: num as u16,
-            ebb,
+            block,
         };
     }
 
-    /// Replace an EBB parameter with a new value of type `ty`.
+    /// Replace an block parameter with a new value of type `ty`.
     ///
-    /// The `old_value` must be an attached EBB parameter. It is removed from its place in the list
+    /// The `old_value` must be an attached block parameter. It is removed from its place in the list
     /// of parameters and replaced by a new value of type `new_type`. The new value gets the same
     /// position in the list, and other parameters are not disturbed.
     ///
     /// The old value is left detached, so it should probably be changed into something else.
     ///
     /// Returns the new value.
-    pub fn replace_ebb_param(&mut self, old_value: Value, new_type: Type) -> Value {
+    pub fn replace_block_param(&mut self, old_value: Value, new_type: Type) -> Value {
         // Create new value identical to the old one except for the type.
-        let (ebb, num) = if let ValueData::Param { num, ebb, .. } = self.values[old_value] {
-            (ebb, num)
+        let (block, num) = if let ValueData::Param { num, block, .. } = self.values[old_value] {
+            (block, num)
         } else {
-            panic!("{} must be an EBB parameter", old_value);
+            panic!("{} must be an block parameter", old_value);
         };
         let new_arg = self.make_value(ValueData::Param {
             ty: new_type,
             num,
-            ebb,
+            block,
         });
 
-        self.ebbs[ebb].params.as_mut_slice(&mut self.value_lists)[num as usize] = new_arg;
+        self.blocks[block]
+            .params
+            .as_mut_slice(&mut self.value_lists)[num as usize] = new_arg;
         new_arg
     }
 
-    /// Detach all the parameters from `ebb` and return them as a `ValueList`.
+    /// Detach all the parameters from `block` and return them as a `ValueList`.
     ///
-    /// This is a quite low-level operation. Sensible things to do with the detached EBB parameters
-    /// is to put them back on the same EBB with `attach_ebb_param()` or change them into aliases
+    /// This is a quite low-level operation. Sensible things to do with the detached block parameters
+    /// is to put them back on the same block with `attach_block_param()` or change them into aliases
     /// with `change_to_alias()`.
-    pub fn detach_ebb_params(&mut self, ebb: Ebb) -> ValueList {
-        self.ebbs[ebb].params.take()
+    pub fn detach_block_params(&mut self, block: Block) -> ValueList {
+        self.blocks[block].params.take()
     }
 }
 
-/// Contents of an extended basic block.
+/// Contents of a basic block.
 ///
-/// Parameters on an extended basic block are values that dominate everything in the EBB. All
-/// branches to this EBB must provide matching arguments, and the arguments to the entry EBB must
+/// Parameters on a basic block are values that dominate everything in the block. All
+/// branches to this block must provide matching arguments, and the arguments to the entry block must
 /// match the function arguments.
 #[derive(Clone)]
-struct EbbData {
-    /// List of parameters to this EBB.
+struct BlockData {
+    /// List of parameters to this block.
     params: ValueList,
 }
 
-impl EbbData {
+impl BlockData {
     fn new() -> Self {
         Self {
             params: ValueList::new(),
@@ -1012,17 +1017,17 @@ impl DataFlowGraph {
         self.make_inst_results_reusing(inst, ctrl_typevar, reuse.iter().map(|x| Some(*x)))
     }
 
-    /// Similar to `append_ebb_param`, append a parameter with type `ty` to
-    /// `ebb`, but using value `val`. This is only for use by the parser to
+    /// Similar to `append_block_param`, append a parameter with type `ty` to
+    /// `block`, but using value `val`. This is only for use by the parser to
     /// create parameters with specific values.
     #[cold]
-    pub fn append_ebb_param_for_parser(&mut self, ebb: Ebb, ty: Type, val: Value) {
-        let num = self.ebbs[ebb].params.push(val, &mut self.value_lists);
-        assert!(num <= u16::MAX as usize, "Too many parameters on EBB");
+    pub fn append_block_param_for_parser(&mut self, block: Block, ty: Type, val: Value) {
+        let num = self.blocks[block].params.push(val, &mut self.value_lists);
+        assert!(num <= u16::MAX as usize, "Too many parameters on block");
         self.values[val] = ValueData::Param {
             ty,
             num: num as u16,
-            ebb,
+            block,
         };
     }
 
@@ -1165,95 +1170,95 @@ mod tests {
     }
 
     #[test]
-    fn ebb() {
+    fn block() {
         let mut dfg = DataFlowGraph::new();
 
-        let ebb = dfg.make_ebb();
-        assert_eq!(ebb.to_string(), "ebb0");
-        assert_eq!(dfg.num_ebb_params(ebb), 0);
-        assert_eq!(dfg.ebb_params(ebb), &[]);
-        assert!(dfg.detach_ebb_params(ebb).is_empty());
-        assert_eq!(dfg.num_ebb_params(ebb), 0);
-        assert_eq!(dfg.ebb_params(ebb), &[]);
+        let block = dfg.make_block();
+        assert_eq!(block.to_string(), "block0");
+        assert_eq!(dfg.num_block_params(block), 0);
+        assert_eq!(dfg.block_params(block), &[]);
+        assert!(dfg.detach_block_params(block).is_empty());
+        assert_eq!(dfg.num_block_params(block), 0);
+        assert_eq!(dfg.block_params(block), &[]);
 
-        let arg1 = dfg.append_ebb_param(ebb, types::F32);
+        let arg1 = dfg.append_block_param(block, types::F32);
         assert_eq!(arg1.to_string(), "v0");
-        assert_eq!(dfg.num_ebb_params(ebb), 1);
-        assert_eq!(dfg.ebb_params(ebb), &[arg1]);
+        assert_eq!(dfg.num_block_params(block), 1);
+        assert_eq!(dfg.block_params(block), &[arg1]);
 
-        let arg2 = dfg.append_ebb_param(ebb, types::I16);
+        let arg2 = dfg.append_block_param(block, types::I16);
         assert_eq!(arg2.to_string(), "v1");
-        assert_eq!(dfg.num_ebb_params(ebb), 2);
-        assert_eq!(dfg.ebb_params(ebb), &[arg1, arg2]);
+        assert_eq!(dfg.num_block_params(block), 2);
+        assert_eq!(dfg.block_params(block), &[arg1, arg2]);
 
-        assert_eq!(dfg.value_def(arg1), ValueDef::Param(ebb, 0));
-        assert_eq!(dfg.value_def(arg2), ValueDef::Param(ebb, 1));
+        assert_eq!(dfg.value_def(arg1), ValueDef::Param(block, 0));
+        assert_eq!(dfg.value_def(arg2), ValueDef::Param(block, 1));
         assert_eq!(dfg.value_type(arg1), types::F32);
         assert_eq!(dfg.value_type(arg2), types::I16);
 
-        // Swap the two EBB parameters.
-        let vlist = dfg.detach_ebb_params(ebb);
-        assert_eq!(dfg.num_ebb_params(ebb), 0);
-        assert_eq!(dfg.ebb_params(ebb), &[]);
+        // Swap the two block parameters.
+        let vlist = dfg.detach_block_params(block);
+        assert_eq!(dfg.num_block_params(block), 0);
+        assert_eq!(dfg.block_params(block), &[]);
         assert_eq!(vlist.as_slice(&dfg.value_lists), &[arg1, arg2]);
-        dfg.attach_ebb_param(ebb, arg2);
-        let arg3 = dfg.append_ebb_param(ebb, types::I32);
-        dfg.attach_ebb_param(ebb, arg1);
-        assert_eq!(dfg.ebb_params(ebb), &[arg2, arg3, arg1]);
+        dfg.attach_block_param(block, arg2);
+        let arg3 = dfg.append_block_param(block, types::I32);
+        dfg.attach_block_param(block, arg1);
+        assert_eq!(dfg.block_params(block), &[arg2, arg3, arg1]);
     }
 
     #[test]
-    fn replace_ebb_params() {
+    fn replace_block_params() {
         let mut dfg = DataFlowGraph::new();
 
-        let ebb = dfg.make_ebb();
-        let arg1 = dfg.append_ebb_param(ebb, types::F32);
+        let block = dfg.make_block();
+        let arg1 = dfg.append_block_param(block, types::F32);
 
-        let new1 = dfg.replace_ebb_param(arg1, types::I64);
+        let new1 = dfg.replace_block_param(arg1, types::I64);
         assert_eq!(dfg.value_type(arg1), types::F32);
         assert_eq!(dfg.value_type(new1), types::I64);
-        assert_eq!(dfg.ebb_params(ebb), &[new1]);
+        assert_eq!(dfg.block_params(block), &[new1]);
 
-        dfg.attach_ebb_param(ebb, arg1);
-        assert_eq!(dfg.ebb_params(ebb), &[new1, arg1]);
+        dfg.attach_block_param(block, arg1);
+        assert_eq!(dfg.block_params(block), &[new1, arg1]);
 
-        let new2 = dfg.replace_ebb_param(arg1, types::I8);
+        let new2 = dfg.replace_block_param(arg1, types::I8);
         assert_eq!(dfg.value_type(arg1), types::F32);
         assert_eq!(dfg.value_type(new2), types::I8);
-        assert_eq!(dfg.ebb_params(ebb), &[new1, new2]);
+        assert_eq!(dfg.block_params(block), &[new1, new2]);
 
-        dfg.attach_ebb_param(ebb, arg1);
-        assert_eq!(dfg.ebb_params(ebb), &[new1, new2, arg1]);
+        dfg.attach_block_param(block, arg1);
+        assert_eq!(dfg.block_params(block), &[new1, new2, arg1]);
 
-        let new3 = dfg.replace_ebb_param(new2, types::I16);
+        let new3 = dfg.replace_block_param(new2, types::I16);
         assert_eq!(dfg.value_type(new1), types::I64);
         assert_eq!(dfg.value_type(new2), types::I8);
         assert_eq!(dfg.value_type(new3), types::I16);
-        assert_eq!(dfg.ebb_params(ebb), &[new1, new3, arg1]);
+        assert_eq!(dfg.block_params(block), &[new1, new3, arg1]);
     }
 
     #[test]
-    fn swap_remove_ebb_params() {
+    fn swap_remove_block_params() {
         let mut dfg = DataFlowGraph::new();
 
-        let ebb = dfg.make_ebb();
-        let arg1 = dfg.append_ebb_param(ebb, types::F32);
-        let arg2 = dfg.append_ebb_param(ebb, types::F32);
-        let arg3 = dfg.append_ebb_param(ebb, types::F32);
-        assert_eq!(dfg.ebb_params(ebb), &[arg1, arg2, arg3]);
+        let block = dfg.make_block();
+        let arg1 = dfg.append_block_param(block, types::F32);
+        let arg2 = dfg.append_block_param(block, types::F32);
+        let arg3 = dfg.append_block_param(block, types::F32);
+        assert_eq!(dfg.block_params(block), &[arg1, arg2, arg3]);
 
-        dfg.swap_remove_ebb_param(arg1);
+        dfg.swap_remove_block_param(arg1);
         assert_eq!(dfg.value_is_attached(arg1), false);
         assert_eq!(dfg.value_is_attached(arg2), true);
         assert_eq!(dfg.value_is_attached(arg3), true);
-        assert_eq!(dfg.ebb_params(ebb), &[arg3, arg2]);
-        dfg.swap_remove_ebb_param(arg2);
+        assert_eq!(dfg.block_params(block), &[arg3, arg2]);
+        dfg.swap_remove_block_param(arg2);
         assert_eq!(dfg.value_is_attached(arg2), false);
         assert_eq!(dfg.value_is_attached(arg3), true);
-        assert_eq!(dfg.ebb_params(ebb), &[arg3]);
-        dfg.swap_remove_ebb_param(arg3);
+        assert_eq!(dfg.block_params(block), &[arg3]);
+        dfg.swap_remove_block_param(arg3);
         assert_eq!(dfg.value_is_attached(arg3), false);
-        assert_eq!(dfg.ebb_params(ebb), &[]);
+        assert_eq!(dfg.block_params(block), &[]);
     }
 
     #[test]
@@ -1261,9 +1266,9 @@ mod tests {
         use crate::ir::InstBuilder;
 
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
         let mut pos = FuncCursor::new(&mut func);
-        pos.insert_ebb(ebb0);
+        pos.insert_block(block0);
 
         // Build a little test program.
         let v1 = pos.ins().iconst(types::I32, 42);
@@ -1271,7 +1276,7 @@ mod tests {
         // Make sure we can resolve value aliases even when values is empty.
         assert_eq!(pos.func.dfg.resolve_aliases(v1), v1);
 
-        let arg0 = pos.func.dfg.append_ebb_param(ebb0, types::I32);
+        let arg0 = pos.func.dfg.append_block_param(block0, types::I32);
         let (s, c) = pos.ins().iadd_ifcout(v1, arg0);
         let iadd = match pos.func.dfg.value_def(s) {
             ValueDef::Result(i, 0) => i,
diff --git a/cranelift/codegen/src/ir/entities.rs b/cranelift/codegen/src/ir/entities.rs
index 6673c71a17..57906ab63a 100644
--- a/cranelift/codegen/src/ir/entities.rs
+++ b/cranelift/codegen/src/ir/entities.rs
@@ -1,7 +1,7 @@
 //! Cranelift IR entity references.
 //!
 //! Instructions in Cranelift IR need to reference other entities in the function. This can be other
-//! parts of the function like extended basic blocks or stack slots, or it can be external entities
+//! parts of the function like basic blocks or stack slots, or it can be external entities
 //! that are declared in the function preamble in the text format.
 //!
 //! These entity references in instruction operands are not implemented as Rust references both
@@ -25,20 +25,19 @@ use core::u32;
 #[cfg(feature = "enable-serde")]
 use serde::{Deserialize, Serialize};
 
-/// An opaque reference to an [extended basic
-/// block](https://en.wikipedia.org/wiki/Extended_basic_block) in a
+/// An opaque reference to a [basic block](https://en.wikipedia.org/wiki/Basic_block) in a
 /// [`Function`](super::function::Function).
 ///
-/// You can get an `Ebb` using
-/// [`FunctionBuilder::create_ebb`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.create_ebb)
+/// You can get a `Block` using
+/// [`FunctionBuilder::create_block`](https://docs.rs/cranelift-frontend/*/cranelift_frontend/struct.FunctionBuilder.html#method.create_block)
 ///
 /// While the order is stable, it is arbitrary and does not necessarily resemble the layout order.
 #[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
-pub struct Ebb(u32);
-entity_impl!(Ebb, "ebb");
+pub struct Block(u32);
+entity_impl!(Block, "block");
 
-impl Ebb {
-    /// Create a new EBB reference from its number. This corresponds to the `ebbNN` representation.
+impl Block {
+    /// Create a new block reference from its number. This corresponds to the `blockNN` representation.
     ///
     /// This method is for use by the parser.
     pub fn with_number(n: u32) -> Option {
@@ -371,8 +370,8 @@ impl Table {
 pub enum AnyEntity {
     /// The whole function.
     Function,
-    /// An extended basic block.
-    Ebb(Ebb),
+    /// a basic block.
+    Block(Block),
     /// An instruction.
     Inst(Inst),
     /// An SSA value.
@@ -397,7 +396,7 @@ impl fmt::Display for AnyEntity {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
             Self::Function => write!(f, "function"),
-            Self::Ebb(r) => r.fmt(f),
+            Self::Block(r) => r.fmt(f),
             Self::Inst(r) => r.fmt(f),
             Self::Value(r) => r.fmt(f),
             Self::StackSlot(r) => r.fmt(f),
@@ -417,9 +416,9 @@ impl fmt::Debug for AnyEntity {
     }
 }
 
-impl From for AnyEntity {
-    fn from(r: Ebb) -> Self {
-        Self::Ebb(r)
+impl From for AnyEntity {
+    fn from(r: Block) -> Self {
+        Self::Block(r)
     }
 }
 
diff --git a/cranelift/codegen/src/ir/extname.rs b/cranelift/codegen/src/ir/extname.rs
index c0a9865373..c12a873d26 100644
--- a/cranelift/codegen/src/ir/extname.rs
+++ b/cranelift/codegen/src/ir/extname.rs
@@ -32,8 +32,8 @@ pub enum ExternalName {
         /// Arbitrary.
         index: u32,
     },
-    /// A test case function name of up to 10 ascii characters. This is
-    /// not intended to be used outside test cases.
+    /// A test case function name of up to a hardcoded amount of ascii
+    /// characters. This is not intended to be used outside test cases.
     TestCase {
         /// How many of the bytes in `ascii` are valid?
         length: u8,
diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index d3ff59690a..77c06964bf 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -1,17 +1,17 @@
 //! Intermediate representation of a function.
 //!
-//! The `Function` struct defined in this module owns all of its extended basic blocks and
+//! The `Function` struct defined in this module owns all of its basic blocks and
 //! instructions.
 
 use crate::binemit::CodeOffset;
 use crate::entity::{PrimaryMap, SecondaryMap};
 use crate::ir;
-use crate::ir::{DataFlowGraph, ExternalName, Layout, Signature};
 use crate::ir::{
-    Ebb, ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, Inst, JumpTable,
+    Block, ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, Inst, JumpTable,
     JumpTableData, Opcode, SigRef, StackSlot, StackSlotData, Table, TableData,
 };
-use crate::ir::{EbbOffsets, FrameLayout, InstEncodings, SourceLocs, StackSlots, ValueLocations};
+use crate::ir::{BlockOffsets, FrameLayout, InstEncodings, SourceLocs, StackSlots, ValueLocations};
+use crate::ir::{DataFlowGraph, ExternalName, Layout, Signature};
 use crate::ir::{JumpTableOffsets, JumpTables};
 use crate::isa::{CallConv, EncInfo, Encoding, Legalize, TargetIsa};
 use crate::regalloc::{EntryRegDiversions, RegDiversions};
@@ -50,10 +50,10 @@ pub struct Function {
     /// Jump tables used in this function.
     pub jump_tables: JumpTables,
 
-    /// Data flow graph containing the primary definition of all instructions, EBBs and values.
+    /// Data flow graph containing the primary definition of all instructions, blocks and values.
     pub dfg: DataFlowGraph,
 
-    /// Layout of EBBs and instructions in the function body.
+    /// Layout of blocks and instructions in the function body.
     pub layout: Layout,
 
     /// Encoding recipe and bits for the legal instructions.
@@ -69,12 +69,12 @@ pub struct Function {
     /// ValueLocation. This field records these register-to-register moves as Diversions.
     pub entry_diversions: EntryRegDiversions,
 
-    /// Code offsets of the EBB headers.
+    /// Code offsets of the block headers.
     ///
     /// This information is only transiently available after the `binemit::relax_branches` function
     /// computes it, and it can easily be recomputed by calling that function. It is not included
     /// in the textual IR format.
-    pub offsets: EbbOffsets,
+    pub offsets: BlockOffsets,
 
     /// Code offsets of Jump Table headers.
     pub jt_offsets: JumpTableOffsets,
@@ -207,10 +207,10 @@ impl Function {
         let entry = self.layout.entry_block().expect("Function is empty");
         self.signature
             .special_param_index(purpose)
-            .map(|i| self.dfg.ebb_params(entry)[i])
+            .map(|i| self.dfg.block_params(entry)[i])
     }
 
-    /// Get an iterator over the instructions in `ebb`, including offsets and encoded instruction
+    /// Get an iterator over the instructions in `block`, including offsets and encoded instruction
     /// sizes.
     ///
     /// The iterator returns `(offset, inst, size)` tuples, where `offset` if the offset in bytes
@@ -219,20 +219,20 @@ impl Function {
     ///
     /// This function can only be used after the code layout has been computed by the
     /// `binemit::relax_branches()` function.
-    pub fn inst_offsets<'a>(&'a self, ebb: Ebb, encinfo: &EncInfo) -> InstOffsetIter<'a> {
+    pub fn inst_offsets<'a>(&'a self, block: Block, encinfo: &EncInfo) -> InstOffsetIter<'a> {
         assert!(
             !self.offsets.is_empty(),
             "Code layout must be computed first"
         );
         let mut divert = RegDiversions::new();
-        divert.at_ebb(&self.entry_diversions, ebb);
+        divert.at_block(&self.entry_diversions, block);
         InstOffsetIter {
             encinfo: encinfo.clone(),
             func: self,
             divert,
             encodings: &self.encodings,
-            offset: self.offsets[ebb],
-            iter: self.layout.ebb_insts(ebb),
+            offset: self.offsets[block],
+            iter: self.layout.block_insts(block),
         }
     }
 
@@ -260,19 +260,19 @@ impl Function {
 
     /// Changes the destination of a jump or branch instruction.
     /// Does nothing if called with a non-jump or non-branch instruction.
-    pub fn change_branch_destination(&mut self, inst: Inst, new_dest: Ebb) {
+    pub fn change_branch_destination(&mut self, inst: Inst, new_dest: Block) {
         match self.dfg[inst].branch_destination_mut() {
             None => (),
             Some(inst_dest) => *inst_dest = new_dest,
         }
     }
 
-    /// Checks that the specified EBB can be encoded as a basic block.
+    /// Checks that the specified block can be encoded as a basic block.
     ///
     /// On error, returns the first invalid instruction and an error message.
-    pub fn is_ebb_basic(&self, ebb: Ebb) -> Result<(), (Inst, &'static str)> {
+    pub fn is_block_basic(&self, block: Block) -> Result<(), (Inst, &'static str)> {
         let dfg = &self.dfg;
-        let inst_iter = self.layout.ebb_insts(ebb);
+        let inst_iter = self.layout.block_insts(block);
 
         // Ignore all instructions prior to the first branch.
         let mut inst_iter = inst_iter.skip_while(|&inst| !dfg[inst].opcode().is_branch());
diff --git a/cranelift/codegen/src/ir/instructions.rs b/cranelift/codegen/src/ir/instructions.rs
index c194505deb..afe0266fe4 100644
--- a/cranelift/codegen/src/ir/instructions.rs
+++ b/cranelift/codegen/src/ir/instructions.rs
@@ -13,7 +13,7 @@ use core::str::FromStr;
 
 use crate::ir;
 use crate::ir::types;
-use crate::ir::{Ebb, FuncRef, JumpTable, SigRef, Type, Value};
+use crate::ir::{Block, FuncRef, JumpTable, SigRef, Type, Value};
 use crate::isa;
 
 use crate::bitset::BitSet;
@@ -164,7 +164,7 @@ impl Default for VariableArgs {
 impl InstructionData {
     /// Return information about the destination of a branch or jump instruction.
     ///
-    /// Any instruction that can transfer control to another EBB reveals its possible destinations
+    /// Any instruction that can transfer control to another block reveals its possible destinations
     /// here.
     pub fn analyze_branch<'a>(&'a self, pool: &'a ValueListPool) -> BranchInfo<'a> {
         match *self {
@@ -208,7 +208,7 @@ impl InstructionData {
     /// branch or jump.
     ///
     /// Multi-destination branches like `br_table` return `None`.
-    pub fn branch_destination(&self) -> Option {
+    pub fn branch_destination(&self) -> Option {
         match *self {
             Self::Jump { destination, .. }
             | Self::Branch { destination, .. }
@@ -227,7 +227,7 @@ impl InstructionData {
     /// single destination branch or jump.
     ///
     /// Multi-destination branches like `br_table` return `None`.
-    pub fn branch_destination_mut(&mut self) -> Option<&mut Ebb> {
+    pub fn branch_destination_mut(&mut self) -> Option<&mut Block> {
         match *self {
             Self::Jump {
                 ref mut destination,
@@ -279,15 +279,15 @@ impl InstructionData {
 /// Information about branch and jump instructions.
 pub enum BranchInfo<'a> {
     /// This is not a branch or jump instruction.
-    /// This instruction will not transfer control to another EBB in the function, but it may still
+    /// This instruction will not transfer control to another block in the function, but it may still
     /// affect control flow by returning or trapping.
     NotABranch,
 
-    /// This is a branch or jump to a single destination EBB, possibly taking value arguments.
-    SingleDest(Ebb, &'a [Value]),
+    /// This is a branch or jump to a single destination block, possibly taking value arguments.
+    SingleDest(Block, &'a [Value]),
 
-    /// This is a jump table branch which can have many destination EBBs and maybe one default EBB.
-    Table(JumpTable, Option),
+    /// This is a jump table branch which can have many destination blocks and maybe one default block.
+    Table(JumpTable, Option),
 }
 
 /// Information about call instructions.
diff --git a/cranelift/codegen/src/ir/jumptable.rs b/cranelift/codegen/src/ir/jumptable.rs
index 372ad09837..a0596728a3 100644
--- a/cranelift/codegen/src/ir/jumptable.rs
+++ b/cranelift/codegen/src/ir/jumptable.rs
@@ -3,7 +3,7 @@
 //! Jump tables are declared in the preamble and assigned an `ir::entities::JumpTable` reference.
 //! The actual table of destinations is stored in a `JumpTableData` struct defined in this module.
 
-use crate::ir::entities::Ebb;
+use crate::ir::entities::Block;
 use alloc::vec::Vec;
 use core::fmt::{self, Display, Formatter};
 use core::slice::{Iter, IterMut};
@@ -14,7 +14,7 @@ use core::slice::{Iter, IterMut};
 #[derive(Clone)]
 pub struct JumpTableData {
     // Table entries.
-    table: Vec,
+    table: Vec,
 }
 
 impl JumpTableData {
@@ -36,32 +36,32 @@ impl JumpTableData {
     }
 
     /// Append a table entry.
-    pub fn push_entry(&mut self, dest: Ebb) {
+    pub fn push_entry(&mut self, dest: Block) {
         self.table.push(dest)
     }
 
-    /// Checks if any of the entries branch to `ebb`.
-    pub fn branches_to(&self, ebb: Ebb) -> bool {
-        self.table.iter().any(|target_ebb| *target_ebb == ebb)
+    /// Checks if any of the entries branch to `block`.
+    pub fn branches_to(&self, block: Block) -> bool {
+        self.table.iter().any(|target_block| *target_block == block)
     }
 
     /// Access the whole table as a slice.
-    pub fn as_slice(&self) -> &[Ebb] {
+    pub fn as_slice(&self) -> &[Block] {
         self.table.as_slice()
     }
 
     /// Access the whole table as a mutable slice.
-    pub fn as_mut_slice(&mut self) -> &mut [Ebb] {
+    pub fn as_mut_slice(&mut self) -> &mut [Block] {
         self.table.as_mut_slice()
     }
 
     /// Returns an iterator over the table.
-    pub fn iter(&self) -> Iter {
+    pub fn iter(&self) -> Iter {
         self.table.iter()
     }
 
     /// Returns an iterator that allows modifying each value.
-    pub fn iter_mut(&mut self) -> IterMut {
+    pub fn iter_mut(&mut self) -> IterMut {
         self.table.iter_mut()
     }
 }
@@ -73,8 +73,8 @@ impl Display for JumpTableData {
             None => (),
             Some(first) => write!(fmt, "{}", first)?,
         }
-        for ebb in self.table.iter().skip(1) {
-            write!(fmt, ", {}", ebb)?;
+        for block in self.table.iter().skip(1) {
+            write!(fmt, ", {}", block)?;
         }
         write!(fmt, "]")
     }
@@ -84,7 +84,7 @@ impl Display for JumpTableData {
 mod tests {
     use super::JumpTableData;
     use crate::entity::EntityRef;
-    use crate::ir::Ebb;
+    use crate::ir::Block;
     use alloc::string::ToString;
 
     #[test]
@@ -102,8 +102,8 @@ mod tests {
 
     #[test]
     fn insert() {
-        let e1 = Ebb::new(1);
-        let e2 = Ebb::new(2);
+        let e1 = Block::new(1);
+        let e2 = Block::new(2);
 
         let mut jt = JumpTableData::new();
 
@@ -111,7 +111,7 @@ mod tests {
         jt.push_entry(e2);
         jt.push_entry(e1);
 
-        assert_eq!(jt.to_string(), "jump_table [ebb1, ebb2, ebb1]");
+        assert_eq!(jt.to_string(), "jump_table [block1, block2, block1]");
 
         let v = jt.as_slice();
         assert_eq!(v, [e1, e2, e1]);
diff --git a/cranelift/codegen/src/ir/layout.rs b/cranelift/codegen/src/ir/layout.rs
index b090b8d819..567a92514c 100644
--- a/cranelift/codegen/src/ir/layout.rs
+++ b/cranelift/codegen/src/ir/layout.rs
@@ -1,82 +1,82 @@
 //! Function layout.
 //!
-//! The order of extended basic blocks in a function and the order of instructions in an EBB is
+//! The order of basic blocks in a function and the order of instructions in an block is
 //! determined by the `Layout` data structure defined in this module.
 
 use crate::entity::SecondaryMap;
 use crate::ir::dfg::DataFlowGraph;
 use crate::ir::progpoint::{ExpandedProgramPoint, ProgramOrder};
-use crate::ir::{Ebb, Inst};
+use crate::ir::{Block, Inst};
 use crate::packed_option::PackedOption;
 use crate::timing;
 use core::cmp;
 use core::iter::{IntoIterator, Iterator};
 use log::debug;
 
-/// The `Layout` struct determines the layout of EBBs and instructions in a function. It does not
-/// contain definitions of instructions or EBBs, but depends on `Inst` and `Ebb` entity references
+/// The `Layout` struct determines the layout of blocks and instructions in a function. It does not
+/// contain definitions of instructions or blocks, but depends on `Inst` and `Block` entity references
 /// being defined elsewhere.
 ///
 /// This data structure determines:
 ///
-/// - The order of EBBs in the function.
-/// - Which EBB contains a given instruction.
-/// - The order of instructions with an EBB.
+/// - The order of blocks in the function.
+/// - Which block contains a given instruction.
+/// - The order of instructions with an block.
 ///
 /// While data dependencies are not recorded, instruction ordering does affect control
 /// dependencies, so part of the semantics of the program are determined by the layout.
 ///
 #[derive(Clone)]
 pub struct Layout {
-    /// Linked list nodes for the layout order of EBBs Forms a doubly linked list, terminated in
+    /// Linked list nodes for the layout order of blocks Forms a doubly linked list, terminated in
     /// both ends by `None`.
-    ebbs: SecondaryMap,
+    blocks: SecondaryMap,
 
-    /// Linked list nodes for the layout order of instructions. Forms a double linked list per EBB,
+    /// Linked list nodes for the layout order of instructions. Forms a double linked list per block,
     /// terminated in both ends by `None`.
     insts: SecondaryMap,
 
-    /// First EBB in the layout order, or `None` when no EBBs have been laid out.
-    first_ebb: Option,
+    /// First block in the layout order, or `None` when no blocks have been laid out.
+    first_block: Option,
 
-    /// Last EBB in the layout order, or `None` when no EBBs have been laid out.
-    last_ebb: Option,
+    /// Last block in the layout order, or `None` when no blocks have been laid out.
+    last_block: Option,
 }
 
 impl Layout {
     /// Create a new empty `Layout`.
     pub fn new() -> Self {
         Self {
-            ebbs: SecondaryMap::new(),
+            blocks: SecondaryMap::new(),
             insts: SecondaryMap::new(),
-            first_ebb: None,
-            last_ebb: None,
+            first_block: None,
+            last_block: None,
         }
     }
 
     /// Clear the layout.
     pub fn clear(&mut self) {
-        self.ebbs.clear();
+        self.blocks.clear();
         self.insts.clear();
-        self.first_ebb = None;
-        self.last_ebb = None;
+        self.first_block = None;
+        self.last_block = None;
     }
 
-    /// Returns the capacity of the `EbbData` map.
-    pub fn ebb_capacity(&self) -> usize {
-        self.ebbs.capacity()
+    /// Returns the capacity of the `BlockData` map.
+    pub fn block_capacity(&self) -> usize {
+        self.blocks.capacity()
     }
 }
 
 /// Sequence numbers.
 ///
-/// All instructions and EBBs are given a sequence number that can be used to quickly determine
+/// All instructions and blocks are given a sequence number that can be used to quickly determine
 /// their relative position in the layout. The sequence numbers are not contiguous, but are assigned
 /// like line numbers in BASIC: 10, 20, 30, ...
 ///
-/// The EBB sequence numbers are strictly increasing, and so are the instruction sequence numbers
-/// within an EBB. The instruction sequence numbers are all between the sequence number of their
-/// containing EBB and the following EBB.
+/// The block sequence numbers are strictly increasing, and so are the instruction sequence numbers
+/// within an block. The instruction sequence numbers are all between the sequence number of their
+/// containing block and the following block.
 ///
 /// The result is that sequence numbers work like BASIC line numbers for the textual form of the IR.
 type SequenceNumber = u32;
@@ -127,11 +127,11 @@ impl ProgramOrder for Layout {
         a_seq.cmp(&b_seq)
     }
 
-    fn is_ebb_gap(&self, inst: Inst, ebb: Ebb) -> bool {
+    fn is_block_gap(&self, inst: Inst, block: Block) -> bool {
         let i = &self.insts[inst];
-        let e = &self.ebbs[ebb];
+        let e = &self.blocks[block];
 
-        i.next.is_none() && i.ebb == e.prev
+        i.next.is_none() && i.block == e.prev
     }
 }
 
@@ -139,71 +139,71 @@ impl ProgramOrder for Layout {
 impl Layout {
     /// Get the sequence number of a program point that must correspond to an entity in the layout.
     fn seq>(&self, pp: PP) -> SequenceNumber {
-        // When `PP = Inst` or `PP = Ebb`, we expect this dynamic type check to be optimized out.
+        // When `PP = Inst` or `PP = Block`, we expect this dynamic type check to be optimized out.
         match pp.into() {
-            ExpandedProgramPoint::Ebb(ebb) => self.ebbs[ebb].seq,
+            ExpandedProgramPoint::Block(block) => self.blocks[block].seq,
             ExpandedProgramPoint::Inst(inst) => self.insts[inst].seq,
         }
     }
 
-    /// Get the last sequence number in `ebb`.
-    fn last_ebb_seq(&self, ebb: Ebb) -> SequenceNumber {
-        // Get the seq of the last instruction if it exists, otherwise use the EBB header seq.
-        self.ebbs[ebb]
+    /// Get the last sequence number in `block`.
+    fn last_block_seq(&self, block: Block) -> SequenceNumber {
+        // Get the seq of the last instruction if it exists, otherwise use the block header seq.
+        self.blocks[block]
             .last_inst
             .map(|inst| self.insts[inst].seq)
-            .unwrap_or(self.ebbs[ebb].seq)
+            .unwrap_or(self.blocks[block].seq)
     }
 
-    /// Assign a valid sequence number to `ebb` such that the numbers are still monotonic. This may
+    /// Assign a valid sequence number to `block` such that the numbers are still monotonic. This may
     /// require renumbering.
-    fn assign_ebb_seq(&mut self, ebb: Ebb) {
-        debug_assert!(self.is_ebb_inserted(ebb));
+    fn assign_block_seq(&mut self, block: Block) {
+        debug_assert!(self.is_block_inserted(block));
 
-        // Get the sequence number immediately before `ebb`, or 0.
-        let prev_seq = self.ebbs[ebb]
+        // Get the sequence number immediately before `block`, or 0.
+        let prev_seq = self.blocks[block]
             .prev
-            .map(|prev_ebb| self.last_ebb_seq(prev_ebb))
+            .map(|prev_block| self.last_block_seq(prev_block))
             .unwrap_or(0);
 
-        // Get the sequence number immediately following `ebb`.
-        let next_seq = if let Some(inst) = self.ebbs[ebb].first_inst.expand() {
+        // Get the sequence number immediately following `block`.
+        let next_seq = if let Some(inst) = self.blocks[block].first_inst.expand() {
             self.insts[inst].seq
-        } else if let Some(next_ebb) = self.ebbs[ebb].next.expand() {
-            self.ebbs[next_ebb].seq
+        } else if let Some(next_block) = self.blocks[block].next.expand() {
+            self.blocks[next_block].seq
         } else {
-            // There is nothing after `ebb`. We can just use a major stride.
-            self.ebbs[ebb].seq = prev_seq + MAJOR_STRIDE;
+            // There is nothing after `block`. We can just use a major stride.
+            self.blocks[block].seq = prev_seq + MAJOR_STRIDE;
             return;
         };
 
         // Check if there is room between these sequence numbers.
         if let Some(seq) = midpoint(prev_seq, next_seq) {
-            self.ebbs[ebb].seq = seq;
+            self.blocks[block].seq = seq;
         } else {
             // No available integers between `prev_seq` and `next_seq`. We have to renumber.
-            self.renumber_from_ebb(ebb, prev_seq + MINOR_STRIDE, prev_seq + LOCAL_LIMIT);
+            self.renumber_from_block(block, prev_seq + MINOR_STRIDE, prev_seq + LOCAL_LIMIT);
         }
     }
 
     /// Assign a valid sequence number to `inst` such that the numbers are still monotonic. This may
     /// require renumbering.
     fn assign_inst_seq(&mut self, inst: Inst) {
-        let ebb = self
-            .inst_ebb(inst)
+        let block = self
+            .inst_block(inst)
             .expect("inst must be inserted before assigning an seq");
 
         // Get the sequence number immediately before `inst`.
         let prev_seq = match self.insts[inst].prev.expand() {
             Some(prev_inst) => self.insts[prev_inst].seq,
-            None => self.ebbs[ebb].seq,
+            None => self.blocks[block].seq,
         };
 
         // Get the sequence number immediately following `inst`.
         let next_seq = if let Some(next_inst) = self.insts[inst].next.expand() {
             self.insts[next_inst].seq
-        } else if let Some(next_ebb) = self.ebbs[ebb].next.expand() {
-            self.ebbs[next_ebb].seq
+        } else if let Some(next_block) = self.blocks[block].next.expand() {
+            self.blocks[next_block].seq
         } else {
             // There is nothing after `inst`. We can just use a major stride.
             self.insts[inst].seq = prev_seq + MAJOR_STRIDE;
@@ -219,7 +219,7 @@ impl Layout {
         }
     }
 
-    /// Renumber instructions starting from `inst` until the end of the EBB or until numbers catch
+    /// Renumber instructions starting from `inst` until the end of the block or until numbers catch
     /// up.
     ///
     /// Return `None` if renumbering has caught up and the sequence is monotonic again. Otherwise
@@ -260,31 +260,36 @@ impl Layout {
         }
     }
 
-    /// Renumber starting from `ebb` to `seq` and continuing until the sequence numbers are
+    /// Renumber starting from `block` to `seq` and continuing until the sequence numbers are
     /// monotonic again.
-    fn renumber_from_ebb(&mut self, ebb: Ebb, first_seq: SequenceNumber, limit: SequenceNumber) {
-        let mut ebb = ebb;
+    fn renumber_from_block(
+        &mut self,
+        block: Block,
+        first_seq: SequenceNumber,
+        limit: SequenceNumber,
+    ) {
+        let mut block = block;
         let mut seq = first_seq;
 
         loop {
-            self.ebbs[ebb].seq = seq;
+            self.blocks[block].seq = seq;
 
-            // Renumber instructions in `ebb`. Stop when the numbers catch up.
-            if let Some(inst) = self.ebbs[ebb].first_inst.expand() {
+            // Renumber instructions in `block`. Stop when the numbers catch up.
+            if let Some(inst) = self.blocks[block].first_inst.expand() {
                 seq = match self.renumber_insts(inst, seq + MINOR_STRIDE, limit) {
                     Some(s) => s,
                     None => return,
                 }
             }
 
-            // Advance to the next EBB.
-            ebb = match self.ebbs[ebb].next.expand() {
+            // Advance to the next block.
+            block = match self.blocks[block].next.expand() {
                 Some(next) => next,
                 None => return,
             };
 
             // Stop renumbering once the numbers catch up.
-            if seq < self.ebbs[ebb].seq {
+            if seq < self.blocks[block].seq {
                 return;
             }
 
@@ -296,27 +301,27 @@ impl Layout {
     /// monotonic again.
     fn renumber_from_inst(&mut self, inst: Inst, first_seq: SequenceNumber, limit: SequenceNumber) {
         if let Some(seq) = self.renumber_insts(inst, first_seq, limit) {
-            // Renumbering spills over into next EBB.
-            if let Some(next_ebb) = self.ebbs[self.inst_ebb(inst).unwrap()].next.expand() {
-                self.renumber_from_ebb(next_ebb, seq + MINOR_STRIDE, limit);
+            // Renumbering spills over into next block.
+            if let Some(next_block) = self.blocks[self.inst_block(inst).unwrap()].next.expand() {
+                self.renumber_from_block(next_block, seq + MINOR_STRIDE, limit);
             }
         }
     }
 
-    /// Renumber all EBBs and instructions in the layout.
+    /// Renumber all blocks and instructions in the layout.
     ///
     /// This doesn't affect the position of anything, but it gives more room in the internal
     /// sequence numbers for inserting instructions later.
     fn full_renumber(&mut self) {
         let _tt = timing::layout_renumber();
         let mut seq = 0;
-        let mut next_ebb = self.first_ebb;
-        while let Some(ebb) = next_ebb {
-            self.ebbs[ebb].seq = seq;
+        let mut next_block = self.first_block;
+        while let Some(block) = next_block {
+            self.blocks[block].seq = seq;
             seq += MAJOR_STRIDE;
-            next_ebb = self.ebbs[ebb].next.expand();
+            next_block = self.blocks[block].next.expand();
 
-            let mut next_inst = self.ebbs[ebb].first_inst.expand();
+            let mut next_inst = self.blocks[block].first_inst.expand();
             while let Some(inst) = next_inst {
                 self.insts[inst].seq = seq;
                 seq += MAJOR_STRIDE;
@@ -327,169 +332,169 @@ impl Layout {
     }
 }
 
-/// Methods for laying out EBBs.
+/// Methods for laying out blocks.
 ///
-/// An unknown EBB starts out as *not inserted* in the EBB layout. The layout is a linear order of
-/// inserted EBBs. Once an EBB has been inserted in the layout, instructions can be added. An EBB
+/// An unknown block starts out as *not inserted* in the block layout. The layout is a linear order of
+/// inserted blocks. Once an block has been inserted in the layout, instructions can be added. An block
 /// can only be removed from the layout when it is empty.
 ///
-/// Since every EBB must end with a terminator instruction which cannot fall through, the layout of
-/// EBBs do not affect the semantics of the program.
+/// Since every block must end with a terminator instruction which cannot fall through, the layout of
+/// blocks do not affect the semantics of the program.
 ///
 impl Layout {
-    /// Is `ebb` currently part of the layout?
-    pub fn is_ebb_inserted(&self, ebb: Ebb) -> bool {
-        Some(ebb) == self.first_ebb || self.ebbs[ebb].prev.is_some()
+    /// Is `block` currently part of the layout?
+    pub fn is_block_inserted(&self, block: Block) -> bool {
+        Some(block) == self.first_block || self.blocks[block].prev.is_some()
     }
 
-    /// Insert `ebb` as the last EBB in the layout.
-    pub fn append_ebb(&mut self, ebb: Ebb) {
+    /// Insert `block` as the last block in the layout.
+    pub fn append_block(&mut self, block: Block) {
         debug_assert!(
-            !self.is_ebb_inserted(ebb),
-            "Cannot append EBB that is already in the layout"
+            !self.is_block_inserted(block),
+            "Cannot append block that is already in the layout"
         );
         {
-            let node = &mut self.ebbs[ebb];
+            let node = &mut self.blocks[block];
             debug_assert!(node.first_inst.is_none() && node.last_inst.is_none());
-            node.prev = self.last_ebb.into();
+            node.prev = self.last_block.into();
             node.next = None.into();
         }
-        if let Some(last) = self.last_ebb {
-            self.ebbs[last].next = ebb.into();
+        if let Some(last) = self.last_block {
+            self.blocks[last].next = block.into();
         } else {
-            self.first_ebb = Some(ebb);
+            self.first_block = Some(block);
         }
-        self.last_ebb = Some(ebb);
-        self.assign_ebb_seq(ebb);
+        self.last_block = Some(block);
+        self.assign_block_seq(block);
     }
 
-    /// Insert `ebb` in the layout before the existing EBB `before`.
-    pub fn insert_ebb(&mut self, ebb: Ebb, before: Ebb) {
+    /// Insert `block` in the layout before the existing block `before`.
+    pub fn insert_block(&mut self, block: Block, before: Block) {
         debug_assert!(
-            !self.is_ebb_inserted(ebb),
-            "Cannot insert EBB that is already in the layout"
+            !self.is_block_inserted(block),
+            "Cannot insert block that is already in the layout"
         );
         debug_assert!(
-            self.is_ebb_inserted(before),
-            "EBB Insertion point not in the layout"
+            self.is_block_inserted(before),
+            "block Insertion point not in the layout"
         );
-        let after = self.ebbs[before].prev;
+        let after = self.blocks[before].prev;
         {
-            let node = &mut self.ebbs[ebb];
+            let node = &mut self.blocks[block];
             node.next = before.into();
             node.prev = after;
         }
-        self.ebbs[before].prev = ebb.into();
+        self.blocks[before].prev = block.into();
         match after.expand() {
-            None => self.first_ebb = Some(ebb),
-            Some(a) => self.ebbs[a].next = ebb.into(),
+            None => self.first_block = Some(block),
+            Some(a) => self.blocks[a].next = block.into(),
         }
-        self.assign_ebb_seq(ebb);
+        self.assign_block_seq(block);
     }
 
-    /// Insert `ebb` in the layout *after* the existing EBB `after`.
-    pub fn insert_ebb_after(&mut self, ebb: Ebb, after: Ebb) {
+    /// Insert `block` in the layout *after* the existing block `after`.
+    pub fn insert_block_after(&mut self, block: Block, after: Block) {
         debug_assert!(
-            !self.is_ebb_inserted(ebb),
-            "Cannot insert EBB that is already in the layout"
+            !self.is_block_inserted(block),
+            "Cannot insert block that is already in the layout"
         );
         debug_assert!(
-            self.is_ebb_inserted(after),
-            "EBB Insertion point not in the layout"
+            self.is_block_inserted(after),
+            "block Insertion point not in the layout"
         );
-        let before = self.ebbs[after].next;
+        let before = self.blocks[after].next;
         {
-            let node = &mut self.ebbs[ebb];
+            let node = &mut self.blocks[block];
             node.next = before;
             node.prev = after.into();
         }
-        self.ebbs[after].next = ebb.into();
+        self.blocks[after].next = block.into();
         match before.expand() {
-            None => self.last_ebb = Some(ebb),
-            Some(b) => self.ebbs[b].prev = ebb.into(),
+            None => self.last_block = Some(block),
+            Some(b) => self.blocks[b].prev = block.into(),
         }
-        self.assign_ebb_seq(ebb);
+        self.assign_block_seq(block);
     }
 
-    /// Remove `ebb` from the layout.
-    pub fn remove_ebb(&mut self, ebb: Ebb) {
-        debug_assert!(self.is_ebb_inserted(ebb), "EBB not in the layout");
-        debug_assert!(self.first_inst(ebb).is_none(), "EBB must be empty.");
+    /// Remove `block` from the layout.
+    pub fn remove_block(&mut self, block: Block) {
+        debug_assert!(self.is_block_inserted(block), "block not in the layout");
+        debug_assert!(self.first_inst(block).is_none(), "block must be empty.");
 
-        // Clear the `ebb` node and extract links.
+        // Clear the `block` node and extract links.
         let prev;
         let next;
         {
-            let n = &mut self.ebbs[ebb];
+            let n = &mut self.blocks[block];
             prev = n.prev;
             next = n.next;
             n.prev = None.into();
             n.next = None.into();
         }
-        // Fix up links to `ebb`.
+        // Fix up links to `block`.
         match prev.expand() {
-            None => self.first_ebb = next.expand(),
-            Some(p) => self.ebbs[p].next = next,
+            None => self.first_block = next.expand(),
+            Some(p) => self.blocks[p].next = next,
         }
         match next.expand() {
-            None => self.last_ebb = prev.expand(),
-            Some(n) => self.ebbs[n].prev = prev,
+            None => self.last_block = prev.expand(),
+            Some(n) => self.blocks[n].prev = prev,
         }
     }
 
-    /// Return an iterator over all EBBs in layout order.
-    pub fn ebbs(&self) -> Ebbs {
-        Ebbs {
+    /// Return an iterator over all blocks in layout order.
+    pub fn blocks(&self) -> Blocks {
+        Blocks {
             layout: self,
-            next: self.first_ebb,
+            next: self.first_block,
         }
     }
 
     /// Get the function's entry block.
-    /// This is simply the first EBB in the layout order.
-    pub fn entry_block(&self) -> Option {
-        self.first_ebb
+    /// This is simply the first block in the layout order.
+    pub fn entry_block(&self) -> Option {
+        self.first_block
     }
 
-    /// Get the last EBB in the layout.
-    pub fn last_ebb(&self) -> Option {
-        self.last_ebb
+    /// Get the last block in the layout.
+    pub fn last_block(&self) -> Option {
+        self.last_block
     }
 
-    /// Get the block preceding `ebb` in the layout order.
-    pub fn prev_ebb(&self, ebb: Ebb) -> Option {
-        self.ebbs[ebb].prev.expand()
+    /// Get the block preceding `block` in the layout order.
+    pub fn prev_block(&self, block: Block) -> Option {
+        self.blocks[block].prev.expand()
     }
 
-    /// Get the block following `ebb` in the layout order.
-    pub fn next_ebb(&self, ebb: Ebb) -> Option {
-        self.ebbs[ebb].next.expand()
+    /// Get the block following `block` in the layout order.
+    pub fn next_block(&self, block: Block) -> Option {
+        self.blocks[block].next.expand()
     }
 }
 
 #[derive(Clone, Debug, Default)]
-struct EbbNode {
-    prev: PackedOption,
-    next: PackedOption,
+struct BlockNode {
+    prev: PackedOption,
+    next: PackedOption,
     first_inst: PackedOption,
     last_inst: PackedOption,
     seq: SequenceNumber,
 }
 
-/// Iterate over EBBs in layout order. See `Layout::ebbs()`.
-pub struct Ebbs<'f> {
+/// Iterate over blocks in layout order. See `Layout::blocks()`.
+pub struct Blocks<'f> {
     layout: &'f Layout,
-    next: Option,
+    next: Option,
 }
 
-impl<'f> Iterator for Ebbs<'f> {
-    type Item = Ebb;
+impl<'f> Iterator for Blocks<'f> {
+    type Item = Block;
 
-    fn next(&mut self) -> Option {
+    fn next(&mut self) -> Option {
         match self.next {
-            Some(ebb) => {
-                self.next = self.layout.next_ebb(ebb);
-                Some(ebb)
+            Some(block) => {
+                self.next = self.layout.next_block(block);
+                Some(block)
             }
             None => None,
         }
@@ -498,70 +503,70 @@ impl<'f> Iterator for Ebbs<'f> {
 
 /// Use a layout reference in a for loop.
 impl<'f> IntoIterator for &'f Layout {
-    type Item = Ebb;
-    type IntoIter = Ebbs<'f>;
+    type Item = Block;
+    type IntoIter = Blocks<'f>;
 
-    fn into_iter(self) -> Ebbs<'f> {
-        self.ebbs()
+    fn into_iter(self) -> Blocks<'f> {
+        self.blocks()
     }
 }
 
 /// Methods for arranging instructions.
 ///
 /// An instruction starts out as *not inserted* in the layout. An instruction can be inserted into
-/// an EBB at a given position.
+/// an block at a given position.
 impl Layout {
-    /// Get the EBB containing `inst`, or `None` if `inst` is not inserted in the layout.
-    pub fn inst_ebb(&self, inst: Inst) -> Option {
-        self.insts[inst].ebb.into()
+    /// Get the block containing `inst`, or `None` if `inst` is not inserted in the layout.
+    pub fn inst_block(&self, inst: Inst) -> Option {
+        self.insts[inst].block.into()
     }
 
-    /// Get the EBB containing the program point `pp`. Panic if `pp` is not in the layout.
-    pub fn pp_ebb(&self, pp: PP) -> Ebb
+    /// Get the block containing the program point `pp`. Panic if `pp` is not in the layout.
+    pub fn pp_block(&self, pp: PP) -> Block
     where
         PP: Into,
     {
         match pp.into() {
-            ExpandedProgramPoint::Ebb(ebb) => ebb,
+            ExpandedProgramPoint::Block(block) => block,
             ExpandedProgramPoint::Inst(inst) => {
-                self.inst_ebb(inst).expect("Program point not in layout")
+                self.inst_block(inst).expect("Program point not in layout")
             }
         }
     }
 
-    /// Append `inst` to the end of `ebb`.
-    pub fn append_inst(&mut self, inst: Inst, ebb: Ebb) {
-        debug_assert_eq!(self.inst_ebb(inst), None);
+    /// Append `inst` to the end of `block`.
+    pub fn append_inst(&mut self, inst: Inst, block: Block) {
+        debug_assert_eq!(self.inst_block(inst), None);
         debug_assert!(
-            self.is_ebb_inserted(ebb),
-            "Cannot append instructions to EBB not in layout"
+            self.is_block_inserted(block),
+            "Cannot append instructions to block not in layout"
         );
         {
-            let ebb_node = &mut self.ebbs[ebb];
+            let block_node = &mut self.blocks[block];
             {
                 let inst_node = &mut self.insts[inst];
-                inst_node.ebb = ebb.into();
-                inst_node.prev = ebb_node.last_inst;
+                inst_node.block = block.into();
+                inst_node.prev = block_node.last_inst;
                 debug_assert!(inst_node.next.is_none());
             }
-            if ebb_node.first_inst.is_none() {
-                ebb_node.first_inst = inst.into();
+            if block_node.first_inst.is_none() {
+                block_node.first_inst = inst.into();
             } else {
-                self.insts[ebb_node.last_inst.unwrap()].next = inst.into();
+                self.insts[block_node.last_inst.unwrap()].next = inst.into();
             }
-            ebb_node.last_inst = inst.into();
+            block_node.last_inst = inst.into();
         }
         self.assign_inst_seq(inst);
     }
 
-    /// Fetch an ebb's first instruction.
-    pub fn first_inst(&self, ebb: Ebb) -> Option {
-        self.ebbs[ebb].first_inst.into()
+    /// Fetch an block's first instruction.
+    pub fn first_inst(&self, block: Block) -> Option {
+        self.blocks[block].first_inst.into()
     }
 
-    /// Fetch an ebb's last instruction.
-    pub fn last_inst(&self, ebb: Ebb) -> Option {
-        self.ebbs[ebb].last_inst.into()
+    /// Fetch an block's last instruction.
+    pub fn last_inst(&self, block: Block) -> Option {
+        self.blocks[block].last_inst.into()
     }
 
     /// Fetch the instruction following `inst`.
@@ -574,11 +579,11 @@ impl Layout {
         self.insts[inst].prev.expand()
     }
 
-    /// Fetch the first instruction in an ebb's terminal branch group.
-    pub fn canonical_branch_inst(&self, dfg: &DataFlowGraph, ebb: Ebb) -> Option {
+    /// Fetch the first instruction in an block's terminal branch group.
+    pub fn canonical_branch_inst(&self, dfg: &DataFlowGraph, block: Block) -> Option {
         // Basic blocks permit at most two terminal branch instructions.
         // If two, the former is conditional and the latter is unconditional.
-        let last = self.last_inst(ebb)?;
+        let last = self.last_inst(block)?;
         if let Some(prev) = self.prev_inst(last) {
             if dfg[prev].opcode().is_branch() {
                 return Some(prev);
@@ -587,22 +592,22 @@ impl Layout {
         Some(last)
     }
 
-    /// Insert `inst` before the instruction `before` in the same EBB.
+    /// Insert `inst` before the instruction `before` in the same block.
     pub fn insert_inst(&mut self, inst: Inst, before: Inst) {
-        debug_assert_eq!(self.inst_ebb(inst), None);
-        let ebb = self
-            .inst_ebb(before)
+        debug_assert_eq!(self.inst_block(inst), None);
+        let block = self
+            .inst_block(before)
             .expect("Instruction before insertion point not in the layout");
         let after = self.insts[before].prev;
         {
             let inst_node = &mut self.insts[inst];
-            inst_node.ebb = ebb.into();
+            inst_node.block = block.into();
             inst_node.next = before.into();
             inst_node.prev = after;
         }
         self.insts[before].prev = inst.into();
         match after.expand() {
-            None => self.ebbs[ebb].first_inst = inst.into(),
+            None => self.blocks[block].first_inst = inst.into(),
             Some(a) => self.insts[a].next = inst.into(),
         }
         self.assign_inst_seq(inst);
@@ -610,7 +615,7 @@ impl Layout {
 
     /// Remove `inst` from the layout.
     pub fn remove_inst(&mut self, inst: Inst) {
-        let ebb = self.inst_ebb(inst).expect("Instruction already removed.");
+        let block = self.inst_block(inst).expect("Instruction already removed.");
         // Clear the `inst` node and extract links.
         let prev;
         let next;
@@ -618,37 +623,37 @@ impl Layout {
             let n = &mut self.insts[inst];
             prev = n.prev;
             next = n.next;
-            n.ebb = None.into();
+            n.block = None.into();
             n.prev = None.into();
             n.next = None.into();
         }
         // Fix up links to `inst`.
         match prev.expand() {
-            None => self.ebbs[ebb].first_inst = next,
+            None => self.blocks[block].first_inst = next,
             Some(p) => self.insts[p].next = next,
         }
         match next.expand() {
-            None => self.ebbs[ebb].last_inst = prev,
+            None => self.blocks[block].last_inst = prev,
             Some(n) => self.insts[n].prev = prev,
         }
     }
 
-    /// Iterate over the instructions in `ebb` in layout order.
-    pub fn ebb_insts(&self, ebb: Ebb) -> Insts {
+    /// Iterate over the instructions in `block` in layout order.
+    pub fn block_insts(&self, block: Block) -> Insts {
         Insts {
             layout: self,
-            head: self.ebbs[ebb].first_inst.into(),
-            tail: self.ebbs[ebb].last_inst.into(),
+            head: self.blocks[block].first_inst.into(),
+            tail: self.blocks[block].last_inst.into(),
         }
     }
 
-    /// Split the EBB containing `before` in two.
+    /// Split the block containing `before` in two.
     ///
-    /// Insert `new_ebb` after the old EBB and move `before` and the following instructions to
-    /// `new_ebb`:
+    /// Insert `new_block` after the old block and move `before` and the following instructions to
+    /// `new_block`:
     ///
     /// ```text
-    /// old_ebb:
+    /// old_block:
     ///     i1
     ///     i2
     ///     i3 << before
@@ -657,69 +662,69 @@ impl Layout {
     /// becomes:
     ///
     /// ```text
-    /// old_ebb:
+    /// old_block:
     ///     i1
     ///     i2
-    /// new_ebb:
+    /// new_block:
     ///     i3 << before
     ///     i4
     /// ```
-    pub fn split_ebb(&mut self, new_ebb: Ebb, before: Inst) {
-        let old_ebb = self
-            .inst_ebb(before)
+    pub fn split_block(&mut self, new_block: Block, before: Inst) {
+        let old_block = self
+            .inst_block(before)
             .expect("The `before` instruction must be in the layout");
-        debug_assert!(!self.is_ebb_inserted(new_ebb));
+        debug_assert!(!self.is_block_inserted(new_block));
 
-        // Insert new_ebb after old_ebb.
-        let next_ebb = self.ebbs[old_ebb].next;
-        let last_inst = self.ebbs[old_ebb].last_inst;
+        // Insert new_block after old_block.
+        let next_block = self.blocks[old_block].next;
+        let last_inst = self.blocks[old_block].last_inst;
         {
-            let node = &mut self.ebbs[new_ebb];
-            node.prev = old_ebb.into();
-            node.next = next_ebb;
+            let node = &mut self.blocks[new_block];
+            node.prev = old_block.into();
+            node.next = next_block;
             node.first_inst = before.into();
             node.last_inst = last_inst;
         }
-        self.ebbs[old_ebb].next = new_ebb.into();
+        self.blocks[old_block].next = new_block.into();
 
         // Fix backwards link.
-        if Some(old_ebb) == self.last_ebb {
-            self.last_ebb = Some(new_ebb);
+        if Some(old_block) == self.last_block {
+            self.last_block = Some(new_block);
         } else {
-            self.ebbs[next_ebb.unwrap()].prev = new_ebb.into();
+            self.blocks[next_block.unwrap()].prev = new_block.into();
         }
 
         // Disconnect the instruction links.
         let prev_inst = self.insts[before].prev;
         self.insts[before].prev = None.into();
-        self.ebbs[old_ebb].last_inst = prev_inst;
+        self.blocks[old_block].last_inst = prev_inst;
         match prev_inst.expand() {
-            None => self.ebbs[old_ebb].first_inst = None.into(),
+            None => self.blocks[old_block].first_inst = None.into(),
             Some(pi) => self.insts[pi].next = None.into(),
         }
 
-        // Fix the instruction -> ebb pointers.
+        // Fix the instruction -> block pointers.
         let mut opt_i = Some(before);
         while let Some(i) = opt_i {
-            debug_assert_eq!(self.insts[i].ebb.expand(), Some(old_ebb));
-            self.insts[i].ebb = new_ebb.into();
+            debug_assert_eq!(self.insts[i].block.expand(), Some(old_block));
+            self.insts[i].block = new_block.into();
             opt_i = self.insts[i].next.into();
         }
 
-        self.assign_ebb_seq(new_ebb);
+        self.assign_block_seq(new_block);
     }
 }
 
 #[derive(Clone, Debug, Default)]
 struct InstNode {
-    /// The Ebb containing this instruction, or `None` if the instruction is not yet inserted.
-    ebb: PackedOption,
+    /// The Block containing this instruction, or `None` if the instruction is not yet inserted.
+    block: PackedOption,
     prev: PackedOption,
     next: PackedOption,
     seq: SequenceNumber,
 }
 
-/// Iterate over instructions in an EBB in layout order. See `Layout::ebb_insts()`.
+/// Iterate over instructions in an block in layout order. See `Layout::block_insts()`.
 pub struct Insts<'f> {
     layout: &'f Layout,
     head: Option,
@@ -763,7 +768,7 @@ mod tests {
     use super::Layout;
     use crate::cursor::{Cursor, CursorPosition};
     use crate::entity::EntityRef;
-    use crate::ir::{Ebb, Inst, ProgramOrder, SourceLoc};
+    use crate::ir::{Block, Inst, ProgramOrder, SourceLoc};
     use alloc::vec::Vec;
     use core::cmp::Ordering;
 
@@ -810,76 +815,76 @@ mod tests {
         }
     }
 
-    fn verify(layout: &mut Layout, ebbs: &[(Ebb, &[Inst])]) {
-        // Check that EBBs are inserted and instructions belong the right places.
+    fn verify(layout: &mut Layout, blocks: &[(Block, &[Inst])]) {
+        // Check that blocks are inserted and instructions belong the right places.
         // Check forward linkage with iterators.
         // Check that layout sequence numbers are strictly monotonic.
         {
             let mut seq = 0;
-            let mut ebb_iter = layout.ebbs();
-            for &(ebb, insts) in ebbs {
-                assert!(layout.is_ebb_inserted(ebb));
-                assert_eq!(ebb_iter.next(), Some(ebb));
-                assert!(layout.ebbs[ebb].seq > seq);
-                seq = layout.ebbs[ebb].seq;
+            let mut block_iter = layout.blocks();
+            for &(block, insts) in blocks {
+                assert!(layout.is_block_inserted(block));
+                assert_eq!(block_iter.next(), Some(block));
+                assert!(layout.blocks[block].seq > seq);
+                seq = layout.blocks[block].seq;
 
-                let mut inst_iter = layout.ebb_insts(ebb);
+                let mut inst_iter = layout.block_insts(block);
                 for &inst in insts {
-                    assert_eq!(layout.inst_ebb(inst), Some(ebb));
+                    assert_eq!(layout.inst_block(inst), Some(block));
                     assert_eq!(inst_iter.next(), Some(inst));
                     assert!(layout.insts[inst].seq > seq);
                     seq = layout.insts[inst].seq;
                 }
                 assert_eq!(inst_iter.next(), None);
             }
-            assert_eq!(ebb_iter.next(), None);
+            assert_eq!(block_iter.next(), None);
         }
 
         // Check backwards linkage with a cursor.
         let mut cur = LayoutCursor::new(layout);
-        for &(ebb, insts) in ebbs.into_iter().rev() {
-            assert_eq!(cur.prev_ebb(), Some(ebb));
+        for &(block, insts) in blocks.into_iter().rev() {
+            assert_eq!(cur.prev_block(), Some(block));
             for &inst in insts.into_iter().rev() {
                 assert_eq!(cur.prev_inst(), Some(inst));
             }
             assert_eq!(cur.prev_inst(), None);
         }
-        assert_eq!(cur.prev_ebb(), None);
+        assert_eq!(cur.prev_block(), None);
     }
 
     #[test]
-    fn append_ebb() {
+    fn append_block() {
         let mut layout = Layout::new();
-        let e0 = Ebb::new(0);
-        let e1 = Ebb::new(1);
-        let e2 = Ebb::new(2);
+        let e0 = Block::new(0);
+        let e1 = Block::new(1);
+        let e2 = Block::new(2);
 
         {
             let imm = &layout;
-            assert!(!imm.is_ebb_inserted(e0));
-            assert!(!imm.is_ebb_inserted(e1));
+            assert!(!imm.is_block_inserted(e0));
+            assert!(!imm.is_block_inserted(e1));
         }
         verify(&mut layout, &[]);
 
-        layout.append_ebb(e1);
-        assert!(!layout.is_ebb_inserted(e0));
-        assert!(layout.is_ebb_inserted(e1));
-        assert!(!layout.is_ebb_inserted(e2));
-        let v: Vec = layout.ebbs().collect();
+        layout.append_block(e1);
+        assert!(!layout.is_block_inserted(e0));
+        assert!(layout.is_block_inserted(e1));
+        assert!(!layout.is_block_inserted(e2));
+        let v: Vec = layout.blocks().collect();
         assert_eq!(v, [e1]);
 
-        layout.append_ebb(e2);
-        assert!(!layout.is_ebb_inserted(e0));
-        assert!(layout.is_ebb_inserted(e1));
-        assert!(layout.is_ebb_inserted(e2));
-        let v: Vec = layout.ebbs().collect();
+        layout.append_block(e2);
+        assert!(!layout.is_block_inserted(e0));
+        assert!(layout.is_block_inserted(e1));
+        assert!(layout.is_block_inserted(e2));
+        let v: Vec = layout.blocks().collect();
         assert_eq!(v, [e1, e2]);
 
-        layout.append_ebb(e0);
-        assert!(layout.is_ebb_inserted(e0));
-        assert!(layout.is_ebb_inserted(e1));
-        assert!(layout.is_ebb_inserted(e2));
-        let v: Vec = layout.ebbs().collect();
+        layout.append_block(e0);
+        assert!(layout.is_block_inserted(e0));
+        assert!(layout.is_block_inserted(e1));
+        assert!(layout.is_block_inserted(e2));
+        let v: Vec = layout.blocks().collect();
         assert_eq!(v, [e1, e2, e0]);
 
         {
@@ -899,111 +904,111 @@ mod tests {
         assert_eq!(cur.prev_inst(), None);
         assert_eq!(cur.position(), CursorPosition::Nowhere);
 
-        assert_eq!(cur.next_ebb(), Some(e1));
+        assert_eq!(cur.next_block(), Some(e1));
         assert_eq!(cur.position(), CursorPosition::Before(e1));
         assert_eq!(cur.next_inst(), None);
         assert_eq!(cur.position(), CursorPosition::After(e1));
         assert_eq!(cur.next_inst(), None);
         assert_eq!(cur.position(), CursorPosition::After(e1));
-        assert_eq!(cur.next_ebb(), Some(e2));
+        assert_eq!(cur.next_block(), Some(e2));
         assert_eq!(cur.prev_inst(), None);
         assert_eq!(cur.position(), CursorPosition::Before(e2));
-        assert_eq!(cur.next_ebb(), Some(e0));
-        assert_eq!(cur.next_ebb(), None);
+        assert_eq!(cur.next_block(), Some(e0));
+        assert_eq!(cur.next_block(), None);
         assert_eq!(cur.position(), CursorPosition::Nowhere);
 
-        // Backwards through the EBBs.
-        assert_eq!(cur.prev_ebb(), Some(e0));
+        // Backwards through the blocks.
+        assert_eq!(cur.prev_block(), Some(e0));
         assert_eq!(cur.position(), CursorPosition::After(e0));
-        assert_eq!(cur.prev_ebb(), Some(e2));
-        assert_eq!(cur.prev_ebb(), Some(e1));
-        assert_eq!(cur.prev_ebb(), None);
+        assert_eq!(cur.prev_block(), Some(e2));
+        assert_eq!(cur.prev_block(), Some(e1));
+        assert_eq!(cur.prev_block(), None);
         assert_eq!(cur.position(), CursorPosition::Nowhere);
     }
 
     #[test]
-    fn insert_ebb() {
+    fn insert_block() {
         let mut layout = Layout::new();
-        let e0 = Ebb::new(0);
-        let e1 = Ebb::new(1);
-        let e2 = Ebb::new(2);
+        let e0 = Block::new(0);
+        let e1 = Block::new(1);
+        let e2 = Block::new(2);
 
         {
             let imm = &layout;
-            assert!(!imm.is_ebb_inserted(e0));
-            assert!(!imm.is_ebb_inserted(e1));
+            assert!(!imm.is_block_inserted(e0));
+            assert!(!imm.is_block_inserted(e1));
 
-            let v: Vec = layout.ebbs().collect();
+            let v: Vec = layout.blocks().collect();
             assert_eq!(v, []);
         }
 
-        layout.append_ebb(e1);
-        assert!(!layout.is_ebb_inserted(e0));
-        assert!(layout.is_ebb_inserted(e1));
-        assert!(!layout.is_ebb_inserted(e2));
+        layout.append_block(e1);
+        assert!(!layout.is_block_inserted(e0));
+        assert!(layout.is_block_inserted(e1));
+        assert!(!layout.is_block_inserted(e2));
         verify(&mut layout, &[(e1, &[])]);
 
-        layout.insert_ebb(e2, e1);
-        assert!(!layout.is_ebb_inserted(e0));
-        assert!(layout.is_ebb_inserted(e1));
-        assert!(layout.is_ebb_inserted(e2));
+        layout.insert_block(e2, e1);
+        assert!(!layout.is_block_inserted(e0));
+        assert!(layout.is_block_inserted(e1));
+        assert!(layout.is_block_inserted(e2));
         verify(&mut layout, &[(e2, &[]), (e1, &[])]);
 
-        layout.insert_ebb(e0, e1);
-        assert!(layout.is_ebb_inserted(e0));
-        assert!(layout.is_ebb_inserted(e1));
-        assert!(layout.is_ebb_inserted(e2));
+        layout.insert_block(e0, e1);
+        assert!(layout.is_block_inserted(e0));
+        assert!(layout.is_block_inserted(e1));
+        assert!(layout.is_block_inserted(e2));
         verify(&mut layout, &[(e2, &[]), (e0, &[]), (e1, &[])]);
     }
 
     #[test]
-    fn insert_ebb_after() {
+    fn insert_block_after() {
         let mut layout = Layout::new();
-        let e0 = Ebb::new(0);
-        let e1 = Ebb::new(1);
-        let e2 = Ebb::new(2);
+        let e0 = Block::new(0);
+        let e1 = Block::new(1);
+        let e2 = Block::new(2);
 
-        layout.append_ebb(e1);
-        layout.insert_ebb_after(e2, e1);
+        layout.append_block(e1);
+        layout.insert_block_after(e2, e1);
         verify(&mut layout, &[(e1, &[]), (e2, &[])]);
 
-        layout.insert_ebb_after(e0, e1);
+        layout.insert_block_after(e0, e1);
         verify(&mut layout, &[(e1, &[]), (e0, &[]), (e2, &[])]);
     }
 
     #[test]
     fn append_inst() {
         let mut layout = Layout::new();
-        let e1 = Ebb::new(1);
+        let e1 = Block::new(1);
 
-        layout.append_ebb(e1);
-        let v: Vec = layout.ebb_insts(e1).collect();
+        layout.append_block(e1);
+        let v: Vec = layout.block_insts(e1).collect();
         assert_eq!(v, []);
 
         let i0 = Inst::new(0);
         let i1 = Inst::new(1);
         let i2 = Inst::new(2);
 
-        assert_eq!(layout.inst_ebb(i0), None);
-        assert_eq!(layout.inst_ebb(i1), None);
-        assert_eq!(layout.inst_ebb(i2), None);
+        assert_eq!(layout.inst_block(i0), None);
+        assert_eq!(layout.inst_block(i1), None);
+        assert_eq!(layout.inst_block(i2), None);
 
         layout.append_inst(i1, e1);
-        assert_eq!(layout.inst_ebb(i0), None);
-        assert_eq!(layout.inst_ebb(i1), Some(e1));
-        assert_eq!(layout.inst_ebb(i2), None);
-        let v: Vec = layout.ebb_insts(e1).collect();
+        assert_eq!(layout.inst_block(i0), None);
+        assert_eq!(layout.inst_block(i1), Some(e1));
+        assert_eq!(layout.inst_block(i2), None);
+        let v: Vec = layout.block_insts(e1).collect();
         assert_eq!(v, [i1]);
 
         layout.append_inst(i2, e1);
-        assert_eq!(layout.inst_ebb(i0), None);
-        assert_eq!(layout.inst_ebb(i1), Some(e1));
-        assert_eq!(layout.inst_ebb(i2), Some(e1));
-        let v: Vec = layout.ebb_insts(e1).collect();
+        assert_eq!(layout.inst_block(i0), None);
+        assert_eq!(layout.inst_block(i1), Some(e1));
+        assert_eq!(layout.inst_block(i2), Some(e1));
+        let v: Vec = layout.block_insts(e1).collect();
         assert_eq!(v, [i1, i2]);
 
         // Test double-ended instruction iterator.
-        let v: Vec = layout.ebb_insts(e1).rev().collect();
+        let v: Vec = layout.block_insts(e1).rev().collect();
         assert_eq!(v, [i2, i1]);
 
         layout.append_inst(i0, e1);
@@ -1036,45 +1041,45 @@ mod tests {
         cur.goto_inst(i2);
         assert_eq!(cur.remove_inst(), i2);
         verify(cur.layout, &[(e1, &[i1, i0])]);
-        assert_eq!(cur.layout.inst_ebb(i2), None);
+        assert_eq!(cur.layout.inst_block(i2), None);
         assert_eq!(cur.remove_inst(), i0);
         verify(cur.layout, &[(e1, &[i1])]);
-        assert_eq!(cur.layout.inst_ebb(i0), None);
+        assert_eq!(cur.layout.inst_block(i0), None);
         assert_eq!(cur.position(), CursorPosition::After(e1));
         cur.layout.remove_inst(i1);
         verify(cur.layout, &[(e1, &[])]);
-        assert_eq!(cur.layout.inst_ebb(i1), None);
+        assert_eq!(cur.layout.inst_block(i1), None);
     }
 
     #[test]
     fn insert_inst() {
         let mut layout = Layout::new();
-        let e1 = Ebb::new(1);
+        let e1 = Block::new(1);
 
-        layout.append_ebb(e1);
-        let v: Vec = layout.ebb_insts(e1).collect();
+        layout.append_block(e1);
+        let v: Vec = layout.block_insts(e1).collect();
         assert_eq!(v, []);
 
         let i0 = Inst::new(0);
         let i1 = Inst::new(1);
         let i2 = Inst::new(2);
 
-        assert_eq!(layout.inst_ebb(i0), None);
-        assert_eq!(layout.inst_ebb(i1), None);
-        assert_eq!(layout.inst_ebb(i2), None);
+        assert_eq!(layout.inst_block(i0), None);
+        assert_eq!(layout.inst_block(i1), None);
+        assert_eq!(layout.inst_block(i2), None);
 
         layout.append_inst(i1, e1);
-        assert_eq!(layout.inst_ebb(i0), None);
-        assert_eq!(layout.inst_ebb(i1), Some(e1));
-        assert_eq!(layout.inst_ebb(i2), None);
-        let v: Vec = layout.ebb_insts(e1).collect();
+        assert_eq!(layout.inst_block(i0), None);
+        assert_eq!(layout.inst_block(i1), Some(e1));
+        assert_eq!(layout.inst_block(i2), None);
+        let v: Vec = layout.block_insts(e1).collect();
         assert_eq!(v, [i1]);
 
         layout.insert_inst(i2, i1);
-        assert_eq!(layout.inst_ebb(i0), None);
-        assert_eq!(layout.inst_ebb(i1), Some(e1));
-        assert_eq!(layout.inst_ebb(i2), Some(e1));
-        let v: Vec = layout.ebb_insts(e1).collect();
+        assert_eq!(layout.inst_block(i0), None);
+        assert_eq!(layout.inst_block(i1), Some(e1));
+        assert_eq!(layout.inst_block(i2), Some(e1));
+        let v: Vec = layout.block_insts(e1).collect();
         assert_eq!(v, [i2, i1]);
 
         layout.insert_inst(i0, i1);
@@ -1082,16 +1087,16 @@ mod tests {
     }
 
     #[test]
-    fn multiple_ebbs() {
+    fn multiple_blocks() {
         let mut layout = Layout::new();
 
-        let e0 = Ebb::new(0);
-        let e1 = Ebb::new(1);
+        let e0 = Block::new(0);
+        let e1 = Block::new(1);
 
         assert_eq!(layout.entry_block(), None);
-        layout.append_ebb(e0);
+        layout.append_block(e0);
         assert_eq!(layout.entry_block(), Some(e0));
-        layout.append_ebb(e1);
+        layout.append_block(e1);
         assert_eq!(layout.entry_block(), Some(e0));
 
         let i0 = Inst::new(0);
@@ -1104,84 +1109,84 @@ mod tests {
         layout.append_inst(i2, e1);
         layout.append_inst(i3, e1);
 
-        let v0: Vec = layout.ebb_insts(e0).collect();
-        let v1: Vec = layout.ebb_insts(e1).collect();
+        let v0: Vec = layout.block_insts(e0).collect();
+        let v1: Vec = layout.block_insts(e1).collect();
         assert_eq!(v0, [i0, i1]);
         assert_eq!(v1, [i2, i3]);
     }
 
     #[test]
-    fn split_ebb() {
+    fn split_block() {
         let mut layout = Layout::new();
 
-        let e0 = Ebb::new(0);
-        let e1 = Ebb::new(1);
-        let e2 = Ebb::new(2);
+        let e0 = Block::new(0);
+        let e1 = Block::new(1);
+        let e2 = Block::new(2);
 
         let i0 = Inst::new(0);
         let i1 = Inst::new(1);
         let i2 = Inst::new(2);
         let i3 = Inst::new(3);
 
-        layout.append_ebb(e0);
+        layout.append_block(e0);
         layout.append_inst(i0, e0);
-        assert_eq!(layout.inst_ebb(i0), Some(e0));
-        layout.split_ebb(e1, i0);
-        assert_eq!(layout.inst_ebb(i0), Some(e1));
+        assert_eq!(layout.inst_block(i0), Some(e0));
+        layout.split_block(e1, i0);
+        assert_eq!(layout.inst_block(i0), Some(e1));
 
         {
             let mut cur = LayoutCursor::new(&mut layout);
-            assert_eq!(cur.next_ebb(), Some(e0));
+            assert_eq!(cur.next_block(), Some(e0));
             assert_eq!(cur.next_inst(), None);
-            assert_eq!(cur.next_ebb(), Some(e1));
+            assert_eq!(cur.next_block(), Some(e1));
             assert_eq!(cur.next_inst(), Some(i0));
             assert_eq!(cur.next_inst(), None);
-            assert_eq!(cur.next_ebb(), None);
+            assert_eq!(cur.next_block(), None);
 
             // Check backwards links.
-            assert_eq!(cur.prev_ebb(), Some(e1));
+            assert_eq!(cur.prev_block(), Some(e1));
             assert_eq!(cur.prev_inst(), Some(i0));
             assert_eq!(cur.prev_inst(), None);
-            assert_eq!(cur.prev_ebb(), Some(e0));
+            assert_eq!(cur.prev_block(), Some(e0));
             assert_eq!(cur.prev_inst(), None);
-            assert_eq!(cur.prev_ebb(), None);
+            assert_eq!(cur.prev_block(), None);
         }
 
         layout.append_inst(i1, e0);
         layout.append_inst(i2, e0);
         layout.append_inst(i3, e0);
-        layout.split_ebb(e2, i2);
+        layout.split_block(e2, i2);
 
-        assert_eq!(layout.inst_ebb(i0), Some(e1));
-        assert_eq!(layout.inst_ebb(i1), Some(e0));
-        assert_eq!(layout.inst_ebb(i2), Some(e2));
-        assert_eq!(layout.inst_ebb(i3), Some(e2));
+        assert_eq!(layout.inst_block(i0), Some(e1));
+        assert_eq!(layout.inst_block(i1), Some(e0));
+        assert_eq!(layout.inst_block(i2), Some(e2));
+        assert_eq!(layout.inst_block(i3), Some(e2));
 
         {
             let mut cur = LayoutCursor::new(&mut layout);
-            assert_eq!(cur.next_ebb(), Some(e0));
+            assert_eq!(cur.next_block(), Some(e0));
             assert_eq!(cur.next_inst(), Some(i1));
             assert_eq!(cur.next_inst(), None);
-            assert_eq!(cur.next_ebb(), Some(e2));
+            assert_eq!(cur.next_block(), Some(e2));
             assert_eq!(cur.next_inst(), Some(i2));
             assert_eq!(cur.next_inst(), Some(i3));
             assert_eq!(cur.next_inst(), None);
-            assert_eq!(cur.next_ebb(), Some(e1));
+            assert_eq!(cur.next_block(), Some(e1));
             assert_eq!(cur.next_inst(), Some(i0));
             assert_eq!(cur.next_inst(), None);
-            assert_eq!(cur.next_ebb(), None);
+            assert_eq!(cur.next_block(), None);
 
-            assert_eq!(cur.prev_ebb(), Some(e1));
+            assert_eq!(cur.prev_block(), Some(e1));
             assert_eq!(cur.prev_inst(), Some(i0));
             assert_eq!(cur.prev_inst(), None);
-            assert_eq!(cur.prev_ebb(), Some(e2));
+            assert_eq!(cur.prev_block(), Some(e2));
             assert_eq!(cur.prev_inst(), Some(i3));
             assert_eq!(cur.prev_inst(), Some(i2));
             assert_eq!(cur.prev_inst(), None);
-            assert_eq!(cur.prev_ebb(), Some(e0));
+            assert_eq!(cur.prev_block(), Some(e0));
             assert_eq!(cur.prev_inst(), Some(i1));
             assert_eq!(cur.prev_inst(), None);
-            assert_eq!(cur.prev_ebb(), None);
+            assert_eq!(cur.prev_block(), None);
         }
 
         // Check `ProgramOrder`.
@@ -1189,9 +1194,9 @@ mod tests {
         assert_eq!(layout.cmp(e2, i2), Ordering::Less);
         assert_eq!(layout.cmp(i3, i2), Ordering::Greater);
 
-        assert_eq!(layout.is_ebb_gap(i1, e2), true);
-        assert_eq!(layout.is_ebb_gap(i3, e1), true);
-        assert_eq!(layout.is_ebb_gap(i1, e1), false);
-        assert_eq!(layout.is_ebb_gap(i2, e1), false);
+        assert_eq!(layout.is_block_gap(i1, e2), true);
+        assert_eq!(layout.is_block_gap(i3, e1), true);
+        assert_eq!(layout.is_block_gap(i1, e1), false);
+        assert_eq!(layout.is_block_gap(i2, e1), false);
     }
 }
diff --git a/cranelift/codegen/src/ir/mod.rs b/cranelift/codegen/src/ir/mod.rs
index 096e372db0..3c222ca9f5 100644
--- a/cranelift/codegen/src/ir/mod.rs
+++ b/cranelift/codegen/src/ir/mod.rs
@@ -33,7 +33,7 @@ pub use crate::ir::builder::{
 pub use crate::ir::constant::{ConstantData, ConstantOffset, ConstantPool};
 pub use crate::ir::dfg::{DataFlowGraph, ValueDef};
 pub use crate::ir::entities::{
-    Constant, Ebb, FuncRef, GlobalValue, Heap, Immediate, Inst, JumpTable, SigRef, StackSlot,
+    Block, Constant, FuncRef, GlobalValue, Heap, Immediate, Inst, JumpTable, SigRef, StackSlot,
     Table, Value,
 };
 pub use crate::ir::extfunc::{
@@ -73,8 +73,8 @@ pub type JumpTables = PrimaryMap;
 /// Map of instruction encodings.
 pub type InstEncodings = SecondaryMap;
 
-/// Code offsets for EBBs.
-pub type EbbOffsets = SecondaryMap;
+/// Code offsets for blocks.
+pub type BlockOffsets = SecondaryMap;
 
 /// Code offsets for Jump Tables.
 pub type JumpTableOffsets = SecondaryMap;
diff --git a/cranelift/codegen/src/ir/progpoint.rs b/cranelift/codegen/src/ir/progpoint.rs
index 4bfa2c39e7..df1a7d14b3 100644
--- a/cranelift/codegen/src/ir/progpoint.rs
+++ b/cranelift/codegen/src/ir/progpoint.rs
@@ -1,7 +1,7 @@
 //! Program points.
 
 use crate::entity::EntityRef;
-use crate::ir::{Ebb, Inst, ValueDef};
+use crate::ir::{Block, Inst, ValueDef};
 use core::cmp;
 use core::fmt;
 use core::u32;
@@ -10,7 +10,7 @@ use core::u32;
 /// begin or end. It can be either:
 ///
 /// 1. An instruction or
-/// 2. An EBB header.
+/// 2. An block header.
 ///
 /// This corresponds more or less to the lines in the textual form of Cranelift IR.
 #[derive(PartialEq, Eq, Clone, Copy)]
@@ -24,9 +24,9 @@ impl From for ProgramPoint {
     }
 }
 
-impl From for ProgramPoint {
-    fn from(ebb: Ebb) -> Self {
-        let idx = ebb.index();
+impl From for ProgramPoint {
+    fn from(block: Block) -> Self {
+        let idx = block.index();
         debug_assert!(idx < (u32::MAX / 2) as usize);
         Self((idx * 2 + 1) as u32)
     }
@@ -36,7 +36,7 @@ impl From for ProgramPoint {
     fn from(def: ValueDef) -> Self {
         match def {
             ValueDef::Result(inst, _) => inst.into(),
-            ValueDef::Param(ebb, _) => ebb.into(),
+            ValueDef::Param(block, _) => block.into(),
         }
     }
 }
@@ -47,8 +47,8 @@ impl From for ProgramPoint {
 pub enum ExpandedProgramPoint {
     /// An instruction in the function.
     Inst(Inst),
-    /// An EBB header.
-    Ebb(Ebb),
+    /// An block header.
+    Block(Block),
 }
 
 impl ExpandedProgramPoint {
@@ -56,7 +56,7 @@ impl ExpandedProgramPoint {
     pub fn unwrap_inst(self) -> Inst {
         match self {
             Self::Inst(x) => x,
-            Self::Ebb(x) => panic!("expected inst: {}", x),
+            Self::Block(x) => panic!("expected inst: {}", x),
         }
     }
 }
@@ -67,9 +67,9 @@ impl From for ExpandedProgramPoint {
     }
 }
 
-impl From for ExpandedProgramPoint {
-    fn from(ebb: Ebb) -> Self {
-        Self::Ebb(ebb)
+impl From for ExpandedProgramPoint {
+    fn from(block: Block) -> Self {
+        Self::Block(block)
     }
 }
 
@@ -77,7 +77,7 @@ impl From for ExpandedProgramPoint {
     fn from(def: ValueDef) -> Self {
         match def {
             ValueDef::Result(inst, _) => inst.into(),
-            ValueDef::Param(ebb, _) => ebb.into(),
+            ValueDef::Param(block, _) => block.into(),
         }
     }
 }
@@ -87,7 +87,7 @@ impl From for ExpandedProgramPoint {
         if pp.0 & 1 == 0 {
             Self::Inst(Inst::from_u32(pp.0 / 2))
         } else {
-            Self::Ebb(Ebb::from_u32(pp.0 / 2))
+            Self::Block(Block::from_u32(pp.0 / 2))
         }
     }
 }
@@ -96,7 +96,7 @@ impl fmt::Display for ExpandedProgramPoint {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match *self {
             Self::Inst(x) => write!(f, "{}", x),
-            Self::Ebb(x) => write!(f, "{}", x),
+            Self::Block(x) => write!(f, "{}", x),
         }
     }
 }
@@ -129,7 +129,7 @@ pub trait ProgramOrder {
     ///
     /// Return `Less` if `a` appears in the program before `b`.
     ///
-    /// This is declared as a generic such that it can be called with `Inst` and `Ebb` arguments
+    /// This is declared as a generic such that it can be called with `Inst` and `Block` arguments
     /// directly. Depending on the implementation, there is a good chance performance will be
     /// improved for those cases where the type of either argument is known statically.
     fn cmp(&self, a: A, b: B) -> cmp::Ordering
@@ -137,28 +137,28 @@ pub trait ProgramOrder {
         A: Into,
         B: Into;
 
-    /// Is the range from `inst` to `ebb` just the gap between consecutive EBBs?
+    /// Is the range from `inst` to `block` just the gap between consecutive blocks?
     ///
-    /// This returns true if `inst` is the terminator in the EBB immediately before `ebb`.
-    fn is_ebb_gap(&self, inst: Inst, ebb: Ebb) -> bool;
+    /// This returns true if `inst` is the terminator in the block immediately before `block`.
+    fn is_block_gap(&self, inst: Inst, block: Block) -> bool;
 }
 
 #[cfg(test)]
 mod tests {
     use super::*;
     use crate::entity::EntityRef;
-    use crate::ir::{Ebb, Inst};
+    use crate::ir::{Block, Inst};
     use alloc::string::ToString;
 
     #[test]
     fn convert() {
         let i5 = Inst::new(5);
-        let b3 = Ebb::new(3);
+        let b3 = Block::new(3);
 
         let pp1: ProgramPoint = i5.into();
         let pp2: ProgramPoint = b3.into();
 
         assert_eq!(pp1.to_string(), "inst5");
-        assert_eq!(pp2.to_string(), "ebb3");
+        assert_eq!(pp2.to_string(), "block3");
     }
 }
diff --git a/cranelift/codegen/src/isa/constraints.rs b/cranelift/codegen/src/isa/constraints.rs
index ce1d2a5fe1..c87c3bd9d4 100644
--- a/cranelift/codegen/src/isa/constraints.rs
+++ b/cranelift/codegen/src/isa/constraints.rs
@@ -95,7 +95,7 @@ pub struct RecipeConstraints {
     /// If the instruction takes a variable number of operands, the register constraints for those
     /// operands must be computed dynamically.
     ///
-    /// - For branches and jumps, EBB arguments must match the expectations of the destination EBB.
+    /// - For branches and jumps, block arguments must match the expectations of the destination block.
     /// - For calls and returns, the calling convention ABI specifies constraints.
     pub ins: &'static [OperandConstraint],
 
@@ -173,7 +173,7 @@ pub struct BranchRange {
 impl BranchRange {
     /// Determine if this branch range can represent the range from `branch` to `dest`, where
     /// `branch` is the code offset of the branch instruction itself and `dest` is the code offset
-    /// of the destination EBB header.
+    /// of the destination block header.
     ///
     /// This method does not detect if the range is larger than 2 GB.
     pub fn contains(self, branch: CodeOffset, dest: CodeOffset) -> bool {
diff --git a/cranelift/codegen/src/isa/riscv/mod.rs b/cranelift/codegen/src/isa/riscv/mod.rs
index 25244fab81..8aa264f34f 100644
--- a/cranelift/codegen/src/isa/riscv/mod.rs
+++ b/cranelift/codegen/src/isa/riscv/mod.rs
@@ -158,9 +158,9 @@ mod tests {
             .finish(shared_flags);
 
         let mut func = Function::new();
-        let ebb = func.dfg.make_ebb();
-        let arg64 = func.dfg.append_ebb_param(ebb, types::I64);
-        let arg32 = func.dfg.append_ebb_param(ebb, types::I32);
+        let block = func.dfg.make_block();
+        let arg64 = func.dfg.append_block_param(block, types::I64);
+        let arg32 = func.dfg.append_block_param(block, types::I32);
 
         // Try to encode iadd_imm.i64 v1, -10.
         let inst64 = InstructionData::BinaryImm {
@@ -209,9 +209,9 @@ mod tests {
             .finish(shared_flags);
 
         let mut func = Function::new();
-        let ebb = func.dfg.make_ebb();
-        let arg64 = func.dfg.append_ebb_param(ebb, types::I64);
-        let arg32 = func.dfg.append_ebb_param(ebb, types::I32);
+        let block = func.dfg.make_block();
+        let arg64 = func.dfg.append_block_param(block, types::I64);
+        let arg32 = func.dfg.append_block_param(block, types::I32);
 
         // Try to encode iadd_imm.i64 v1, -10.
         let inst64 = InstructionData::BinaryImm {
@@ -268,8 +268,8 @@ mod tests {
         let isa = isa_builder.finish(shared_flags);
 
         let mut func = Function::new();
-        let ebb = func.dfg.make_ebb();
-        let arg32 = func.dfg.append_ebb_param(ebb, types::I32);
+        let block = func.dfg.make_block();
+        let arg32 = func.dfg.append_block_param(block, types::I32);
 
         // Create an imul.i32 which is encodable in RV32M.
         let mul32 = InstructionData::Binary {
diff --git a/cranelift/codegen/src/isa/x86/abi.rs b/cranelift/codegen/src/isa/x86/abi.rs
index 3d160a2cf5..db67457a6c 100644
--- a/cranelift/codegen/src/isa/x86/abi.rs
+++ b/cranelift/codegen/src/isa/x86/abi.rs
@@ -419,8 +419,8 @@ fn callee_saved_gprs_used(isa: &dyn TargetIsa, func: &ir::Function) -> RegisterS
     //
     // TODO: Consider re-evaluating how regmove/regfill/regspill work and whether it's possible
     // to avoid this step.
-    for ebb in &func.layout {
-        for inst in func.layout.ebb_insts(ebb) {
+    for block in &func.layout {
+        for inst in func.layout.block_insts(block) {
             match func.dfg[inst] {
                 ir::instructions::InstructionData::RegMove { dst, .. }
                 | ir::instructions::InstructionData::RegFill { dst, .. } => {
@@ -551,8 +551,8 @@ fn fastcall_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> C
     }
 
     // Set up the cursor and insert the prologue
-    let entry_ebb = func.layout.entry_block().expect("missing entry block");
-    let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_ebb);
+    let entry_block = func.layout.entry_block().expect("missing entry block");
+    let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_block);
     let prologue_cfa_state =
         insert_common_prologue(&mut pos, local_stack_size, reg_type, &csrs, isa);
 
@@ -612,8 +612,8 @@ fn system_v_prologue_epilogue(func: &mut ir::Function, isa: &dyn TargetIsa) -> C
     }
 
     // Set up the cursor and insert the prologue
-    let entry_ebb = func.layout.entry_block().expect("missing entry block");
-    let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_ebb);
+    let entry_block = func.layout.entry_block().expect("missing entry block");
+    let mut pos = EncCursor::new(func, isa).at_first_insertion_point(entry_block);
     let prologue_cfa_state =
         insert_common_prologue(&mut pos, local_stack_size, reg_type, &csrs, isa);
 
@@ -678,9 +678,9 @@ fn insert_common_prologue(
         None
     };
 
-    // Append param to entry EBB
-    let ebb = pos.current_ebb().expect("missing ebb under cursor");
-    let fp = pos.func.dfg.append_ebb_param(ebb, reg_type);
+    // Append param to entry block
+    let block = pos.current_block().expect("missing block under cursor");
+    let fp = pos.func.dfg.append_block_param(block, reg_type);
     pos.func.locations[fp] = ir::ValueLoc::Reg(RU::rbp as RegUnit);
 
     let push_fp_inst = pos.ins().x86_push(fp);
@@ -727,8 +727,8 @@ fn insert_common_prologue(
     }
 
     for reg in csrs.iter(GPR) {
-        // Append param to entry EBB
-        let csr_arg = pos.func.dfg.append_ebb_param(ebb, reg_type);
+        // Append param to entry block
+        let csr_arg = pos.func.dfg.append_block_param(block, reg_type);
 
         // Assign it a location
         pos.func.locations[csr_arg] = ir::ValueLoc::Reg(reg);
@@ -831,11 +831,11 @@ fn insert_common_epilogues(
     isa: &dyn TargetIsa,
     cfa_state: Option,
 ) {
-    while let Some(ebb) = pos.next_ebb() {
-        pos.goto_last_inst(ebb);
+    while let Some(block) = pos.next_block() {
+        pos.goto_last_inst(block);
         if let Some(inst) = pos.current_inst() {
             if pos.func.dfg[inst].opcode().is_return() {
-                let is_last = pos.func.layout.last_ebb() == Some(ebb);
+                let is_last = pos.func.layout.last_block() == Some(block);
                 insert_common_epilogue(
                     inst,
                     stack_size,
diff --git a/cranelift/codegen/src/isa/x86/binemit.rs b/cranelift/codegen/src/isa/x86/binemit.rs
index 5a373e6f96..44c497e547 100644
--- a/cranelift/codegen/src/isa/x86/binemit.rs
+++ b/cranelift/codegen/src/isa/x86/binemit.rs
@@ -4,7 +4,7 @@ use super::enc_tables::{needs_offset, needs_sib_byte};
 use super::registers::RU;
 use crate::binemit::{bad_encoding, CodeSink, Reloc};
 use crate::ir::condcodes::{CondCode, FloatCC, IntCC};
-use crate::ir::{Constant, Ebb, Function, Inst, InstructionData, JumpTable, Opcode, TrapCode};
+use crate::ir::{Block, Constant, Function, Inst, InstructionData, JumpTable, Opcode, TrapCode};
 use crate::isa::{RegUnit, StackBase, StackBaseMask, StackRef, TargetIsa};
 use crate::regalloc::RegDiversions;
 
@@ -369,13 +369,13 @@ fn fcc2opc(cond: FloatCC) -> u16 {
 }
 
 /// Emit a single-byte branch displacement to `destination`.
-fn disp1(destination: Ebb, func: &Function, sink: &mut CS) {
+fn disp1(destination: Block, func: &Function, sink: &mut CS) {
     let delta = func.offsets[destination].wrapping_sub(sink.offset() + 1);
     sink.put1(delta as u8);
 }
 
 /// Emit a four-byte branch displacement to `destination`.
-fn disp4(destination: Ebb, func: &Function, sink: &mut CS) {
+fn disp4(destination: Block, func: &Function, sink: &mut CS) {
     let delta = func.offsets[destination].wrapping_sub(sink.offset() + 4);
     sink.put4(delta);
 }
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 9a5481e228..947b315cd2 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -253,7 +253,7 @@ fn expand_sdivrem(
         _ => panic!("Need sdiv/srem: {}", func.dfg.display_inst(inst, None)),
     };
 
-    let old_ebb = func.layout.pp_ebb(inst);
+    let old_block = func.layout.pp_block(inst);
     let result = func.dfg.first_result(inst);
     let ty = func.dfg.value_type(result);
 
@@ -297,17 +297,17 @@ fn expand_sdivrem(
         return;
     }
 
-    // EBB handling the nominal case.
-    let nominal = pos.func.dfg.make_ebb();
+    // block handling the nominal case.
+    let nominal = pos.func.dfg.make_block();
 
-    // EBB handling the -1 divisor case.
-    let minus_one = pos.func.dfg.make_ebb();
+    // block handling the -1 divisor case.
+    let minus_one = pos.func.dfg.make_block();
 
-    // Final EBB with one argument representing the final result value.
-    let done = pos.func.dfg.make_ebb();
+    // Final block with one argument representing the final result value.
+    let done = pos.func.dfg.make_block();
 
-    // Move the `inst` result value onto the `done` EBB.
-    pos.func.dfg.attach_ebb_param(done, result);
+    // Move the `inst` result value onto the `done` block.
+    pos.func.dfg.attach_block_param(done, result);
 
     // Start by checking for a -1 divisor which needs to be handled specially.
     let is_m1 = pos.ins().ifcmp_imm(y, -1);
@@ -316,14 +316,14 @@ fn expand_sdivrem(
 
     // Now it is safe to execute the `x86_sdivmodx` instruction which will still trap on division
     // by zero.
-    pos.insert_ebb(nominal);
+    pos.insert_block(nominal);
     let xhi = pos.ins().sshr_imm(x, i64::from(ty.lane_bits()) - 1);
     let (quot, rem) = pos.ins().x86_sdivmodx(x, xhi, y);
     let divres = if is_srem { rem } else { quot };
     pos.ins().jump(done, &[divres]);
 
     // Now deal with the -1 divisor case.
-    pos.insert_ebb(minus_one);
+    pos.insert_block(minus_one);
     let m1_result = if is_srem {
         // x % -1 = 0.
         pos.ins().iconst(ty, 0)
@@ -342,12 +342,12 @@ fn expand_sdivrem(
 
     // Finally insert a label for the completion.
     pos.next_inst();
-    pos.insert_ebb(done);
+    pos.insert_block(done);
 
-    cfg.recompute_ebb(pos.func, old_ebb);
-    cfg.recompute_ebb(pos.func, nominal);
-    cfg.recompute_ebb(pos.func, minus_one);
-    cfg.recompute_ebb(pos.func, done);
+    cfg.recompute_block(pos.func, old_block);
+    cfg.recompute_block(pos.func, nominal);
+    cfg.recompute_block(pos.func, minus_one);
+    cfg.recompute_block(pos.func, done);
 }
 
 /// Expand the `udiv` and `urem` instructions using `x86_udivmodx`.
@@ -421,7 +421,7 @@ fn expand_minmax(
         } => (args[0], args[1], ir::Opcode::X86Fmax, ir::Opcode::Band),
         _ => panic!("Expected fmin/fmax: {}", func.dfg.display_inst(inst, None)),
     };
-    let old_ebb = func.layout.pp_ebb(inst);
+    let old_block = func.layout.pp_block(inst);
 
     // We need to handle the following conditions, depending on how x and y compare:
     //
@@ -430,20 +430,20 @@ fn expand_minmax(
     //    fmin(0.0, -0.0) -> -0.0 and fmax(0.0, -0.0) -> 0.0.
     // 3. UN: We need to produce a quiet NaN that is canonical if the inputs are canonical.
 
-    // EBB handling case 1) where operands are ordered but not equal.
-    let one_ebb = func.dfg.make_ebb();
+    // block handling case 1) where operands are ordered but not equal.
+    let one_block = func.dfg.make_block();
 
-    // EBB handling case 3) where one operand is NaN.
-    let uno_ebb = func.dfg.make_ebb();
+    // block handling case 3) where one operand is NaN.
+    let uno_block = func.dfg.make_block();
 
-    // EBB that handles the unordered or equal cases 2) and 3).
-    let ueq_ebb = func.dfg.make_ebb();
+    // block that handles the unordered or equal cases 2) and 3).
+    let ueq_block = func.dfg.make_block();
 
-    // EBB handling case 2) where operands are ordered and equal.
-    let eq_ebb = func.dfg.make_ebb();
+    // block handling case 2) where operands are ordered and equal.
+    let eq_block = func.dfg.make_block();
 
-    // Final EBB with one argument representing the final result value.
-    let done = func.dfg.make_ebb();
+    // Final block with one argument representing the final result value.
+    let done = func.dfg.make_block();
 
     // The basic blocks are laid out to minimize branching for the common cases:
     //
@@ -451,21 +451,21 @@ fn expand_minmax(
     // 2) One branch taken.
     // 3) Two branches taken, one jump.
 
-    // Move the `inst` result value onto the `done` EBB.
+    // Move the `inst` result value onto the `done` block.
     let result = func.dfg.first_result(inst);
     let ty = func.dfg.value_type(result);
     func.dfg.clear_results(inst);
-    func.dfg.attach_ebb_param(done, result);
+    func.dfg.attach_block_param(done, result);
 
     // Test for case 1) ordered and not equal.
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
     let cmp_ueq = pos.ins().fcmp(FloatCC::UnorderedOrEqual, x, y);
-    pos.ins().brnz(cmp_ueq, ueq_ebb, &[]);
-    pos.ins().jump(one_ebb, &[]);
+    pos.ins().brnz(cmp_ueq, ueq_block, &[]);
+    pos.ins().jump(one_block, &[]);
 
     // Handle the common ordered, not equal (LT|GT) case.
-    pos.insert_ebb(one_ebb);
+    pos.insert_block(one_block);
     let one_inst = pos.ins().Binary(x86_opc, ty, x, y).0;
     let one_result = pos.func.dfg.first_result(one_inst);
     pos.ins().jump(done, &[one_result]);
@@ -473,21 +473,21 @@ fn expand_minmax(
     // Case 3) Unordered.
     // We know that at least one operand is a NaN that needs to be propagated. We simply use an
     // `fadd` instruction which has the same NaN propagation semantics.
-    pos.insert_ebb(uno_ebb);
+    pos.insert_block(uno_block);
     let uno_result = pos.ins().fadd(x, y);
     pos.ins().jump(done, &[uno_result]);
 
     // Case 2) or 3).
-    pos.insert_ebb(ueq_ebb);
+    pos.insert_block(ueq_block);
     // Test for case 3) (UN) one value is NaN.
     // TODO: When we get support for flag values, we can reuse the above comparison.
     let cmp_uno = pos.ins().fcmp(FloatCC::Unordered, x, y);
-    pos.ins().brnz(cmp_uno, uno_ebb, &[]);
-    pos.ins().jump(eq_ebb, &[]);
+    pos.ins().brnz(cmp_uno, uno_block, &[]);
+    pos.ins().jump(eq_block, &[]);
 
     // We are now in case 2) where x and y compare EQ.
     // We need a bitwise operation to get the sign right.
-    pos.insert_ebb(eq_ebb);
+    pos.insert_block(eq_block);
     let bw_inst = pos.ins().Binary(bitwise_opc, ty, x, y).0;
     let bw_result = pos.func.dfg.first_result(bw_inst);
     // This should become a fall-through for this second most common case.
@@ -496,14 +496,14 @@ fn expand_minmax(
 
     // Finally insert a label for the completion.
     pos.next_inst();
-    pos.insert_ebb(done);
+    pos.insert_block(done);
 
-    cfg.recompute_ebb(pos.func, old_ebb);
-    cfg.recompute_ebb(pos.func, one_ebb);
-    cfg.recompute_ebb(pos.func, uno_ebb);
-    cfg.recompute_ebb(pos.func, ueq_ebb);
-    cfg.recompute_ebb(pos.func, eq_ebb);
-    cfg.recompute_ebb(pos.func, done);
+    cfg.recompute_block(pos.func, old_block);
+    cfg.recompute_block(pos.func, one_block);
+    cfg.recompute_block(pos.func, uno_block);
+    cfg.recompute_block(pos.func, ueq_block);
+    cfg.recompute_block(pos.func, eq_block);
+    cfg.recompute_block(pos.func, done);
 }
 
 /// x86 has no unsigned-to-float conversions. We handle the easy case of zero-extending i32 to
@@ -540,33 +540,33 @@ fn expand_fcvt_from_uint(
         _ => unimplemented!(),
     }
 
-    let old_ebb = pos.func.layout.pp_ebb(inst);
+    let old_block = pos.func.layout.pp_block(inst);
 
-    // EBB handling the case where x >= 0.
-    let poszero_ebb = pos.func.dfg.make_ebb();
+    // block handling the case where x >= 0.
+    let poszero_block = pos.func.dfg.make_block();
 
-    // EBB handling the case where x < 0.
-    let neg_ebb = pos.func.dfg.make_ebb();
+    // block handling the case where x < 0.
+    let neg_block = pos.func.dfg.make_block();
 
-    // Final EBB with one argument representing the final result value.
-    let done = pos.func.dfg.make_ebb();
+    // Final block with one argument representing the final result value.
+    let done = pos.func.dfg.make_block();
 
-    // Move the `inst` result value onto the `done` EBB.
+    // Move the `inst` result value onto the `done` block.
     pos.func.dfg.clear_results(inst);
-    pos.func.dfg.attach_ebb_param(done, result);
+    pos.func.dfg.attach_block_param(done, result);
 
     // If x as a signed int is not negative, we can use the existing `fcvt_from_sint` instruction.
     let is_neg = pos.ins().icmp_imm(IntCC::SignedLessThan, x, 0);
-    pos.ins().brnz(is_neg, neg_ebb, &[]);
-    pos.ins().jump(poszero_ebb, &[]);
+    pos.ins().brnz(is_neg, neg_block, &[]);
+    pos.ins().jump(poszero_block, &[]);
 
     // Easy case: just use a signed conversion.
-    pos.insert_ebb(poszero_ebb);
+    pos.insert_block(poszero_block);
     let posres = pos.ins().fcvt_from_sint(ty, x);
     pos.ins().jump(done, &[posres]);
 
     // Now handle the negative case.
-    pos.insert_ebb(neg_ebb);
+    pos.insert_block(neg_block);
 
     // Divide x by two to get it in range for the signed conversion, keep the LSB, and scale it
     // back up on the FP side.
@@ -581,12 +581,12 @@ fn expand_fcvt_from_uint(
 
     // Finally insert a label for the completion.
     pos.next_inst();
-    pos.insert_ebb(done);
+    pos.insert_block(done);
 
-    cfg.recompute_ebb(pos.func, old_ebb);
-    cfg.recompute_ebb(pos.func, poszero_ebb);
-    cfg.recompute_ebb(pos.func, neg_ebb);
-    cfg.recompute_ebb(pos.func, done);
+    cfg.recompute_block(pos.func, old_block);
+    cfg.recompute_block(pos.func, poszero_block);
+    cfg.recompute_block(pos.func, neg_block);
+    cfg.recompute_block(pos.func, done);
 }
 
 fn expand_fcvt_to_sint(
@@ -604,16 +604,16 @@ fn expand_fcvt_to_sint(
         } => arg,
         _ => panic!("Need fcvt_to_sint: {}", func.dfg.display_inst(inst, None)),
     };
-    let old_ebb = func.layout.pp_ebb(inst);
+    let old_block = func.layout.pp_block(inst);
     let xty = func.dfg.value_type(x);
     let result = func.dfg.first_result(inst);
     let ty = func.dfg.value_type(result);
 
-    // Final EBB after the bad value checks.
-    let done = func.dfg.make_ebb();
+    // Final block after the bad value checks.
+    let done = func.dfg.make_block();
 
-    // EBB for checking failure cases.
-    let maybe_trap_ebb = func.dfg.make_ebb();
+    // block for checking failure cases.
+    let maybe_trap_block = func.dfg.make_block();
 
     // The `x86_cvtt2si` performs the desired conversion, but it doesn't trap on NaN or overflow.
     // It produces an INT_MIN result instead.
@@ -626,7 +626,7 @@ fn expand_fcvt_to_sint(
         .ins()
         .icmp_imm(IntCC::NotEqual, result, 1 << (ty.lane_bits() - 1));
     pos.ins().brnz(is_done, done, &[]);
-    pos.ins().jump(maybe_trap_ebb, &[]);
+    pos.ins().jump(maybe_trap_block, &[]);
 
     // We now have the following possibilities:
     //
@@ -634,7 +634,7 @@ fn expand_fcvt_to_sint(
     // 2. The input was NaN -> trap bad_toint
     // 3. The input was out of range -> trap int_ovf
     //
-    pos.insert_ebb(maybe_trap_ebb);
+    pos.insert_block(maybe_trap_block);
 
     // Check for NaN.
     let is_nan = pos.ins().fcmp(FloatCC::Unordered, x, x);
@@ -683,11 +683,11 @@ fn expand_fcvt_to_sint(
     pos.ins().trapnz(overflow, ir::TrapCode::IntegerOverflow);
 
     pos.ins().jump(done, &[]);
-    pos.insert_ebb(done);
+    pos.insert_block(done);
 
-    cfg.recompute_ebb(pos.func, old_ebb);
-    cfg.recompute_ebb(pos.func, maybe_trap_ebb);
-    cfg.recompute_ebb(pos.func, done);
+    cfg.recompute_block(pos.func, old_block);
+    cfg.recompute_block(pos.func, maybe_trap_block);
+    cfg.recompute_block(pos.func, done);
 }
 
 fn expand_fcvt_to_sint_sat(
@@ -709,18 +709,18 @@ fn expand_fcvt_to_sint_sat(
         ),
     };
 
-    let old_ebb = func.layout.pp_ebb(inst);
+    let old_block = func.layout.pp_block(inst);
     let xty = func.dfg.value_type(x);
     let result = func.dfg.first_result(inst);
     let ty = func.dfg.value_type(result);
 
-    // Final EBB after the bad value checks.
-    let done_ebb = func.dfg.make_ebb();
-    let intmin_ebb = func.dfg.make_ebb();
-    let minsat_ebb = func.dfg.make_ebb();
-    let maxsat_ebb = func.dfg.make_ebb();
+    // Final block after the bad value checks.
+    let done_block = func.dfg.make_block();
+    let intmin_block = func.dfg.make_block();
+    let minsat_block = func.dfg.make_block();
+    let maxsat_block = func.dfg.make_block();
     func.dfg.clear_results(inst);
-    func.dfg.attach_ebb_param(done_ebb, result);
+    func.dfg.attach_block_param(done_block, result);
 
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
@@ -732,25 +732,25 @@ fn expand_fcvt_to_sint_sat(
     let is_done = pos
         .ins()
         .icmp_imm(IntCC::NotEqual, cvtt2si, 1 << (ty.lane_bits() - 1));
-    pos.ins().brnz(is_done, done_ebb, &[cvtt2si]);
-    pos.ins().jump(intmin_ebb, &[]);
+    pos.ins().brnz(is_done, done_block, &[cvtt2si]);
+    pos.ins().jump(intmin_block, &[]);
 
     // We now have the following possibilities:
     //
     // 1. INT_MIN was actually the correct conversion result.
     // 2. The input was NaN -> replace the result value with 0.
     // 3. The input was out of range -> saturate the result to the min/max value.
-    pos.insert_ebb(intmin_ebb);
+    pos.insert_block(intmin_block);
 
     // Check for NaN, which is truncated to 0.
     let zero = pos.ins().iconst(ty, 0);
     let is_nan = pos.ins().fcmp(FloatCC::Unordered, x, x);
-    pos.ins().brnz(is_nan, done_ebb, &[zero]);
-    pos.ins().jump(minsat_ebb, &[]);
+    pos.ins().brnz(is_nan, done_block, &[zero]);
+    pos.ins().jump(minsat_block, &[]);
 
     // Check for case 1: INT_MIN is the correct result.
     // Determine the smallest floating point number that would convert to INT_MIN.
-    pos.insert_ebb(minsat_ebb);
+    pos.insert_block(minsat_block);
     let mut overflow_cc = FloatCC::LessThan;
     let output_bits = ty.lane_bits();
     let flimit = match xty {
@@ -786,11 +786,11 @@ fn expand_fcvt_to_sint_sat(
         _ => panic!("Don't know the min value for {}", ty),
     };
     let min_value = pos.ins().iconst(ty, min_imm);
-    pos.ins().brnz(overflow, done_ebb, &[min_value]);
-    pos.ins().jump(maxsat_ebb, &[]);
+    pos.ins().brnz(overflow, done_block, &[min_value]);
+    pos.ins().jump(maxsat_block, &[]);
 
     // Finally, we could have a positive value that is too large.
-    pos.insert_ebb(maxsat_ebb);
+    pos.insert_block(maxsat_block);
     let fzero = match xty {
         ir::types::F32 => pos.ins().f32const(Ieee32::with_bits(0)),
         ir::types::F64 => pos.ins().f64const(Ieee64::with_bits(0)),
@@ -805,20 +805,20 @@ fn expand_fcvt_to_sint_sat(
     let max_value = pos.ins().iconst(ty, max_imm);
 
     let overflow = pos.ins().fcmp(FloatCC::GreaterThanOrEqual, x, fzero);
-    pos.ins().brnz(overflow, done_ebb, &[max_value]);
+    pos.ins().brnz(overflow, done_block, &[max_value]);
 
     // Recycle the original instruction.
-    pos.func.dfg.replace(inst).jump(done_ebb, &[cvtt2si]);
+    pos.func.dfg.replace(inst).jump(done_block, &[cvtt2si]);
 
     // Finally insert a label for the completion.
     pos.next_inst();
-    pos.insert_ebb(done_ebb);
+    pos.insert_block(done_block);
 
-    cfg.recompute_ebb(pos.func, old_ebb);
-    cfg.recompute_ebb(pos.func, intmin_ebb);
-    cfg.recompute_ebb(pos.func, minsat_ebb);
-    cfg.recompute_ebb(pos.func, maxsat_ebb);
-    cfg.recompute_ebb(pos.func, done_ebb);
+    cfg.recompute_block(pos.func, old_block);
+    cfg.recompute_block(pos.func, intmin_block);
+    cfg.recompute_block(pos.func, minsat_block);
+    cfg.recompute_block(pos.func, maxsat_block);
+    cfg.recompute_block(pos.func, done_block);
 }
 
 fn expand_fcvt_to_uint(
@@ -837,26 +837,26 @@ fn expand_fcvt_to_uint(
         _ => panic!("Need fcvt_to_uint: {}", func.dfg.display_inst(inst, None)),
     };
 
-    let old_ebb = func.layout.pp_ebb(inst);
+    let old_block = func.layout.pp_block(inst);
     let xty = func.dfg.value_type(x);
     let result = func.dfg.first_result(inst);
     let ty = func.dfg.value_type(result);
 
-    // EBB handle numbers < 2^(N-1).
-    let below_uint_max_ebb = func.dfg.make_ebb();
+    // block handle numbers < 2^(N-1).
+    let below_uint_max_block = func.dfg.make_block();
 
-    // EBB handle numbers < 0.
-    let below_zero_ebb = func.dfg.make_ebb();
+    // block handle numbers < 0.
+    let below_zero_block = func.dfg.make_block();
 
-    // EBB handling numbers >= 2^(N-1).
-    let large = func.dfg.make_ebb();
+    // block handling numbers >= 2^(N-1).
+    let large = func.dfg.make_block();
 
-    // Final EBB after the bad value checks.
-    let done = func.dfg.make_ebb();
+    // Final block after the bad value checks.
+    let done = func.dfg.make_block();
 
-    // Move the `inst` result value onto the `done` EBB.
+    // Move the `inst` result value onto the `done` block.
     func.dfg.clear_results(inst);
-    func.dfg.attach_ebb_param(done, result);
+    func.dfg.attach_block_param(done, result);
 
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
@@ -871,11 +871,11 @@ fn expand_fcvt_to_uint(
     let is_large = pos.ins().ffcmp(x, pow2nm1);
     pos.ins()
         .brff(FloatCC::GreaterThanOrEqual, is_large, large, &[]);
-    pos.ins().jump(below_uint_max_ebb, &[]);
+    pos.ins().jump(below_uint_max_block, &[]);
 
     // We need to generate a specific trap code when `x` is NaN, so reuse the flags from the
     // previous comparison.
-    pos.insert_ebb(below_uint_max_ebb);
+    pos.insert_block(below_uint_max_block);
     pos.ins().trapff(
         FloatCC::Unordered,
         is_large,
@@ -887,13 +887,13 @@ fn expand_fcvt_to_uint(
     let is_neg = pos.ins().ifcmp_imm(sres, 0);
     pos.ins()
         .brif(IntCC::SignedGreaterThanOrEqual, is_neg, done, &[sres]);
-    pos.ins().jump(below_zero_ebb, &[]);
+    pos.ins().jump(below_zero_block, &[]);
 
-    pos.insert_ebb(below_zero_ebb);
+    pos.insert_block(below_zero_block);
     pos.ins().trap(ir::TrapCode::IntegerOverflow);
 
     // Handle the case where x >= 2^(N-1) and not NaN.
-    pos.insert_ebb(large);
+    pos.insert_block(large);
     let adjx = pos.ins().fsub(x, pow2nm1);
     let lres = pos.ins().x86_cvtt2si(ty, adjx);
     let is_neg = pos.ins().ifcmp_imm(lres, 0);
@@ -906,13 +906,13 @@ fn expand_fcvt_to_uint(
 
     // Finally insert a label for the completion.
     pos.next_inst();
-    pos.insert_ebb(done);
+    pos.insert_block(done);
 
-    cfg.recompute_ebb(pos.func, old_ebb);
-    cfg.recompute_ebb(pos.func, below_uint_max_ebb);
-    cfg.recompute_ebb(pos.func, below_zero_ebb);
-    cfg.recompute_ebb(pos.func, large);
-    cfg.recompute_ebb(pos.func, done);
+    cfg.recompute_block(pos.func, old_block);
+    cfg.recompute_block(pos.func, below_uint_max_block);
+    cfg.recompute_block(pos.func, below_zero_block);
+    cfg.recompute_block(pos.func, large);
+    cfg.recompute_block(pos.func, done);
 }
 
 fn expand_fcvt_to_uint_sat(
@@ -934,27 +934,27 @@ fn expand_fcvt_to_uint_sat(
         ),
     };
 
-    let old_ebb = func.layout.pp_ebb(inst);
+    let old_block = func.layout.pp_block(inst);
     let xty = func.dfg.value_type(x);
     let result = func.dfg.first_result(inst);
     let ty = func.dfg.value_type(result);
 
-    // EBB handle numbers < 2^(N-1).
-    let below_pow2nm1_or_nan_ebb = func.dfg.make_ebb();
-    let below_pow2nm1_ebb = func.dfg.make_ebb();
+    // block handle numbers < 2^(N-1).
+    let below_pow2nm1_or_nan_block = func.dfg.make_block();
+    let below_pow2nm1_block = func.dfg.make_block();
 
-    // EBB handling numbers >= 2^(N-1).
-    let large = func.dfg.make_ebb();
+    // block handling numbers >= 2^(N-1).
+    let large = func.dfg.make_block();
 
-    // EBB handling numbers < 2^N.
-    let uint_large_ebb = func.dfg.make_ebb();
+    // block handling numbers < 2^N.
+    let uint_large_block = func.dfg.make_block();
 
-    // Final EBB after the bad value checks.
-    let done = func.dfg.make_ebb();
+    // Final block after the bad value checks.
+    let done = func.dfg.make_block();
 
-    // Move the `inst` result value onto the `done` EBB.
+    // Move the `inst` result value onto the `done` block.
     func.dfg.clear_results(inst);
-    func.dfg.attach_ebb_param(done, result);
+    func.dfg.attach_block_param(done, result);
 
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
@@ -970,16 +970,16 @@ fn expand_fcvt_to_uint_sat(
     let is_large = pos.ins().ffcmp(x, pow2nm1);
     pos.ins()
         .brff(FloatCC::GreaterThanOrEqual, is_large, large, &[]);
-    pos.ins().jump(below_pow2nm1_or_nan_ebb, &[]);
+    pos.ins().jump(below_pow2nm1_or_nan_block, &[]);
 
     // We need to generate zero when `x` is NaN, so reuse the flags from the previous comparison.
-    pos.insert_ebb(below_pow2nm1_or_nan_ebb);
+    pos.insert_block(below_pow2nm1_or_nan_block);
     pos.ins().brff(FloatCC::Unordered, is_large, done, &[zero]);
-    pos.ins().jump(below_pow2nm1_ebb, &[]);
+    pos.ins().jump(below_pow2nm1_block, &[]);
 
     // Now we know that x < 2^(N-1) and not NaN. If the result of the cvtt2si is positive, we're
     // done; otherwise saturate to the minimum unsigned value, that is 0.
-    pos.insert_ebb(below_pow2nm1_ebb);
+    pos.insert_block(below_pow2nm1_block);
     let sres = pos.ins().x86_cvtt2si(ty, x);
     let is_neg = pos.ins().ifcmp_imm(sres, 0);
     pos.ins()
@@ -987,7 +987,7 @@ fn expand_fcvt_to_uint_sat(
     pos.ins().jump(done, &[zero]);
 
     // Handle the case where x >= 2^(N-1) and not NaN.
-    pos.insert_ebb(large);
+    pos.insert_block(large);
     let adjx = pos.ins().fsub(x, pow2nm1);
     let lres = pos.ins().x86_cvtt2si(ty, adjx);
     let max_value = pos.ins().iconst(
@@ -1001,9 +1001,9 @@ fn expand_fcvt_to_uint_sat(
     let is_neg = pos.ins().ifcmp_imm(lres, 0);
     pos.ins()
         .brif(IntCC::SignedLessThan, is_neg, done, &[max_value]);
-    pos.ins().jump(uint_large_ebb, &[]);
+    pos.ins().jump(uint_large_block, &[]);
 
-    pos.insert_ebb(uint_large_ebb);
+    pos.insert_block(uint_large_block);
     let lfinal = pos.ins().iadd_imm(lres, 1 << (ty.lane_bits() - 1));
 
     // Recycle the original instruction as a jump.
@@ -1011,14 +1011,14 @@ fn expand_fcvt_to_uint_sat(
 
     // Finally insert a label for the completion.
     pos.next_inst();
-    pos.insert_ebb(done);
+    pos.insert_block(done);
 
-    cfg.recompute_ebb(pos.func, old_ebb);
-    cfg.recompute_ebb(pos.func, below_pow2nm1_or_nan_ebb);
-    cfg.recompute_ebb(pos.func, below_pow2nm1_ebb);
-    cfg.recompute_ebb(pos.func, large);
-    cfg.recompute_ebb(pos.func, uint_large_ebb);
-    cfg.recompute_ebb(pos.func, done);
+    cfg.recompute_block(pos.func, old_block);
+    cfg.recompute_block(pos.func, below_pow2nm1_or_nan_block);
+    cfg.recompute_block(pos.func, below_pow2nm1_block);
+    cfg.recompute_block(pos.func, large);
+    cfg.recompute_block(pos.func, uint_large_block);
+    cfg.recompute_block(pos.func, done);
 }
 
 /// Convert shuffle instructions.
diff --git a/cranelift/codegen/src/isa/x86/fde.rs b/cranelift/codegen/src/isa/x86/fde.rs
index 6a56a09451..6687f532b0 100644
--- a/cranelift/codegen/src/isa/x86/fde.rs
+++ b/cranelift/codegen/src/isa/x86/fde.rs
@@ -182,14 +182,14 @@ pub fn emit_fde(func: &Function, isa: &dyn TargetIsa, sink: &mut dyn FrameUnwind
     assert!(func.frame_layout.is_some(), "expected func.frame_layout");
     let frame_layout = func.frame_layout.as_ref().unwrap();
 
-    let mut ebbs = func.layout.ebbs().collect::>();
-    ebbs.sort_by_key(|ebb| func.offsets[*ebb]); // Ensure inst offsets always increase
+    let mut blocks = func.layout.blocks().collect::>();
+    blocks.sort_by_key(|block| func.offsets[*block]); // Ensure inst offsets always increase
 
     let encinfo = isa.encoding_info();
     let mut last_offset = 0;
     let mut changes = Vec::new();
-    for ebb in ebbs {
-        for (offset, inst, size) in func.inst_offsets(ebb, &encinfo) {
+    for block in blocks {
+        for (offset, inst, size) in func.inst_offsets(block, &encinfo) {
             let address_offset = (offset + size) as usize;
             assert!(last_offset <= address_offset);
             if let Some(cmds) = frame_layout.instructions.get(&inst) {
@@ -343,9 +343,9 @@ mod tests {
         let mut func =
             Function::with_name_signature(ExternalName::user(0, 0), Signature::new(call_conv));
 
-        let ebb0 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
         let mut pos = FuncCursor::new(&mut func);
-        pos.insert_ebb(ebb0);
+        pos.insert_block(block0);
         pos.ins().return_(&[]);
 
         if let Some(stack_slot) = stack_slot {
@@ -411,20 +411,20 @@ mod tests {
         sig.params.push(AbiParam::new(types::I32));
         let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
 
-        let ebb0 = func.dfg.make_ebb();
-        let v0 = func.dfg.append_ebb_param(ebb0, types::I32);
-        let ebb1 = func.dfg.make_ebb();
-        let ebb2 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
+        let v0 = func.dfg.append_block_param(block0, types::I32);
+        let block1 = func.dfg.make_block();
+        let block2 = func.dfg.make_block();
 
         let mut pos = FuncCursor::new(&mut func);
-        pos.insert_ebb(ebb0);
-        pos.ins().brnz(v0, ebb2, &[]);
-        pos.ins().jump(ebb1, &[]);
+        pos.insert_block(block0);
+        pos.ins().brnz(v0, block2, &[]);
+        pos.ins().jump(block1, &[]);
 
-        pos.insert_ebb(ebb1);
+        pos.insert_block(block1);
         pos.ins().return_(&[]);
 
-        pos.insert_ebb(ebb2);
+        pos.insert_block(block2);
         pos.ins().trap(TrapCode::User(0));
 
         func
diff --git a/cranelift/codegen/src/isa/x86/unwind.rs b/cranelift/codegen/src/isa/x86/unwind.rs
index ce05f3afab..693693ab37 100644
--- a/cranelift/codegen/src/isa/x86/unwind.rs
+++ b/cranelift/codegen/src/isa/x86/unwind.rs
@@ -127,7 +127,7 @@ impl UnwindInfo {
         }
 
         let prologue_end = func.prologue_end.unwrap();
-        let entry_block = func.layout.ebbs().nth(0).expect("missing entry block");
+        let entry_block = func.layout.blocks().nth(0).expect("missing entry block");
 
         // Stores the stack size when SP is not adjusted via an immediate value
         let mut stack_size = None;
@@ -519,9 +519,9 @@ mod tests {
         let mut func =
             Function::with_name_signature(ExternalName::user(0, 0), Signature::new(call_conv));
 
-        let ebb0 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
         let mut pos = FuncCursor::new(&mut func);
-        pos.insert_ebb(ebb0);
+        pos.insert_block(block0);
         pos.ins().return_(&[]);
 
         if let Some(stack_slot) = stack_slot {
diff --git a/cranelift/codegen/src/legalizer/boundary.rs b/cranelift/codegen/src/legalizer/boundary.rs
index 5063b6d910..7fb977a06a 100644
--- a/cranelift/codegen/src/legalizer/boundary.rs
+++ b/cranelift/codegen/src/legalizer/boundary.rs
@@ -22,7 +22,7 @@ use crate::cursor::{Cursor, FuncCursor};
 use crate::flowgraph::ControlFlowGraph;
 use crate::ir::instructions::CallInfo;
 use crate::ir::{
-    AbiParam, ArgumentLoc, ArgumentPurpose, DataFlowGraph, Ebb, Function, Inst, InstBuilder,
+    AbiParam, ArgumentLoc, ArgumentPurpose, Block, DataFlowGraph, Function, Inst, InstBuilder,
     MemFlags, SigRef, Signature, StackSlotData, StackSlotKind, Type, Value, ValueLoc,
 };
 use crate::isa::TargetIsa;
@@ -84,12 +84,12 @@ fn legalize_signature(
 /// Legalize the entry block parameters after `func`'s signature has been legalized.
 ///
 /// The legalized signature may contain more parameters than the original signature, and the
-/// parameter types have been changed. This function goes through the parameters of the entry EBB
+/// parameter types have been changed. This function goes through the parameters of the entry block
 /// and replaces them with parameters of the right type for the ABI.
 ///
-/// The original entry EBB parameters are computed from the new ABI parameters by code inserted at
+/// The original entry block parameters are computed from the new ABI parameters by code inserted at
 /// the top of the entry block.
-fn legalize_entry_params(func: &mut Function, entry: Ebb) {
+fn legalize_entry_params(func: &mut Function, entry: Block) {
     let mut has_sret = false;
     let mut has_link = false;
     let mut has_vmctx = false;
@@ -104,19 +104,19 @@ fn legalize_entry_params(func: &mut Function, entry: Ebb) {
     // Keep track of the argument types in the ABI-legalized signature.
     let mut abi_arg = 0;
 
-    // Process the EBB parameters one at a time, possibly replacing one argument with multiple new
-    // ones. We do this by detaching the entry EBB parameters first.
-    let ebb_params = pos.func.dfg.detach_ebb_params(entry);
+    // Process the block parameters one at a time, possibly replacing one argument with multiple new
+    // ones. We do this by detaching the entry block parameters first.
+    let block_params = pos.func.dfg.detach_block_params(entry);
     let mut old_arg = 0;
-    while let Some(arg) = ebb_params.get(old_arg, &pos.func.dfg.value_lists) {
+    while let Some(arg) = block_params.get(old_arg, &pos.func.dfg.value_lists) {
         old_arg += 1;
 
         let abi_type = pos.func.signature.params[abi_arg];
         let arg_type = pos.func.dfg.value_type(arg);
         if arg_type == abi_type.value_type {
             // No value translation is necessary, this argument matches the ABI type.
-            // Just use the original EBB argument value. This is the most common case.
-            pos.func.dfg.attach_ebb_param(entry, arg);
+            // Just use the original block argument value. This is the most common case.
+            pos.func.dfg.attach_block_param(entry, arg);
             match abi_type.purpose {
                 ArgumentPurpose::Normal => {}
                 ArgumentPurpose::FramePointer => {}
@@ -151,13 +151,13 @@ fn legalize_entry_params(func: &mut Function, entry: Ebb) {
                 );
                 if ty == abi_type.value_type {
                     abi_arg += 1;
-                    Ok(func.dfg.append_ebb_param(entry, ty))
+                    Ok(func.dfg.append_block_param(entry, ty))
                 } else {
                     Err(abi_type)
                 }
             };
             let converted = convert_from_abi(&mut pos, arg_type, Some(arg), &mut get_arg);
-            // The old `arg` is no longer an attached EBB argument, but there are probably still
+            // The old `arg` is no longer an attached block argument, but there are probably still
             // uses of the value.
             debug_assert_eq!(pos.func.dfg.resolve_aliases(arg), converted);
         }
@@ -201,7 +201,7 @@ fn legalize_entry_params(func: &mut Function, entry: Ebb) {
 
         // Just create entry block values to match here. We will use them in `handle_return_abi()`
         // below.
-        pos.func.dfg.append_ebb_param(entry, arg.value_type);
+        pos.func.dfg.append_block_param(entry, arg.value_type);
     }
 }
 
@@ -851,7 +851,7 @@ pub fn handle_return_abi(inst: Inst, func: &mut Function, cfg: &ControlFlowGraph
             let val = pos
                 .func
                 .dfg
-                .ebb_params(pos.func.layout.entry_block().unwrap())[idx];
+                .block_params(pos.func.layout.entry_block().unwrap())[idx];
             debug_assert_eq!(pos.func.dfg.value_type(val), arg.value_type);
             vlist.push(val, &mut pos.func.dfg.value_lists);
 
@@ -958,8 +958,13 @@ fn round_up_to_multiple_of_pow2(n: u32, to: u32) -> u32 {
 ///
 /// Values that are passed into the function on the stack must be assigned to an `IncomingArg`
 /// stack slot already during legalization.
-fn spill_entry_params(func: &mut Function, entry: Ebb) {
-    for (abi, &arg) in func.signature.params.iter().zip(func.dfg.ebb_params(entry)) {
+fn spill_entry_params(func: &mut Function, entry: Block) {
+    for (abi, &arg) in func
+        .signature
+        .params
+        .iter()
+        .zip(func.dfg.block_params(entry))
+    {
         if let ArgumentLoc::Stack(offset) = abi.location {
             let ss = func.stack_slots.make_incoming_arg(abi.value_type, offset);
             func.locations[arg] = ValueLoc::Stack(ss);
diff --git a/cranelift/codegen/src/legalizer/heap.rs b/cranelift/codegen/src/legalizer/heap.rs
index a6d9d9637d..cc4308c268 100644
--- a/cranelift/codegen/src/legalizer/heap.rs
+++ b/cranelift/codegen/src/legalizer/heap.rs
@@ -120,12 +120,12 @@ fn static_addr(
         pos.ins().trap(ir::TrapCode::HeapOutOfBounds);
         pos.func.dfg.replace(inst).iconst(addr_ty, 0);
 
-        // Split Ebb, as the trap is a terminator instruction.
-        let curr_ebb = pos.current_ebb().expect("Cursor is not in an ebb");
-        let new_ebb = pos.func.dfg.make_ebb();
-        pos.insert_ebb(new_ebb);
-        cfg.recompute_ebb(pos.func, curr_ebb);
-        cfg.recompute_ebb(pos.func, new_ebb);
+        // Split Block, as the trap is a terminator instruction.
+        let curr_block = pos.current_block().expect("Cursor is not in an block");
+        let new_block = pos.func.dfg.make_block();
+        pos.insert_block(new_block);
+        cfg.recompute_block(pos.func, curr_block);
+        cfg.recompute_block(pos.func, new_block);
         return;
     }
 
diff --git a/cranelift/codegen/src/legalizer/mod.rs b/cranelift/codegen/src/legalizer/mod.rs
index 3257949955..781767336a 100644
--- a/cranelift/codegen/src/legalizer/mod.rs
+++ b/cranelift/codegen/src/legalizer/mod.rs
@@ -87,7 +87,7 @@ fn legalize_inst(
                     return LegalizeInstResult::SplitLegalizePending;
                 }
             }
-            ir::ValueDef::Param(_ebb, _num) => {}
+            ir::ValueDef::Param(_block, _num) => {}
         }
 
         let res = pos.func.dfg.inst_results(inst).to_vec();
@@ -148,10 +148,10 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
     let mut pos = FuncCursor::new(func);
     let func_begin = pos.position();
 
-    // Split ebb params before trying to legalize instructions, so that the newly introduced
+    // Split block params before trying to legalize instructions, so that the newly introduced
     // isplit instructions get legalized.
-    while let Some(ebb) = pos.next_ebb() {
-        split::split_ebb_params(pos.func, cfg, ebb);
+    while let Some(block) = pos.next_block() {
+        split::split_block_params(pos.func, cfg, block);
     }
 
     pos.set_position(func_begin);
@@ -159,9 +159,9 @@ pub fn legalize_function(func: &mut ir::Function, cfg: &mut ControlFlowGraph, is
     // This must be a set to prevent trying to legalize `isplit` and `vsplit` twice in certain cases.
     let mut pending_splits = BTreeSet::new();
 
-    // Process EBBs in layout order. Some legalization actions may split the current EBB or append
-    // new ones to the end. We need to make sure we visit those new EBBs too.
-    while let Some(_ebb) = pos.next_ebb() {
+    // Process blocks in layout order. Some legalization actions may split the current block or append
+    // new ones to the end. We need to make sure we visit those new blocks too.
+    while let Some(_block) = pos.next_block() {
         // Keep track of the cursor position before the instruction being processed, so we can
         // double back when replacing instructions.
         let mut prev_pos = pos.position();
@@ -225,48 +225,48 @@ fn expand_cond_trap(
         _ => panic!("Expected cond trap: {}", func.dfg.display_inst(inst, None)),
     };
 
-    // Split the EBB after `inst`:
+    // Split the block after `inst`:
     //
     //     trapnz arg
     //     ..
     //
     // Becomes:
     //
-    //     brz arg, new_ebb_resume
-    //     jump new_ebb_trap
+    //     brz arg, new_block_resume
+    //     jump new_block_trap
     //
-    //   new_ebb_trap:
+    //   new_block_trap:
     //     trap
     //
-    //   new_ebb_resume:
+    //   new_block_resume:
     //     ..
-    let old_ebb = func.layout.pp_ebb(inst);
-    let new_ebb_trap = func.dfg.make_ebb();
-    let new_ebb_resume = func.dfg.make_ebb();
+    let old_block = func.layout.pp_block(inst);
+    let new_block_trap = func.dfg.make_block();
+    let new_block_resume = func.dfg.make_block();
 
     // Replace trap instruction by the inverted condition.
     if trapz {
-        func.dfg.replace(inst).brnz(arg, new_ebb_resume, &[]);
+        func.dfg.replace(inst).brnz(arg, new_block_resume, &[]);
     } else {
-        func.dfg.replace(inst).brz(arg, new_ebb_resume, &[]);
+        func.dfg.replace(inst).brz(arg, new_block_resume, &[]);
     }
 
     // Add jump instruction after the inverted branch.
     let mut pos = FuncCursor::new(func).after_inst(inst);
     pos.use_srcloc(inst);
-    pos.ins().jump(new_ebb_trap, &[]);
+    pos.ins().jump(new_block_trap, &[]);
 
     // Insert the new label and the unconditional trap terminator.
-    pos.insert_ebb(new_ebb_trap);
+    pos.insert_block(new_block_trap);
     pos.ins().trap(code);
 
     // Insert the new label and resume the execution when the trap fails.
-    pos.insert_ebb(new_ebb_resume);
+    pos.insert_block(new_block_resume);
 
     // Finally update the CFG.
-    cfg.recompute_ebb(pos.func, old_ebb);
-    cfg.recompute_ebb(pos.func, new_ebb_resume);
-    cfg.recompute_ebb(pos.func, new_ebb_trap);
+    cfg.recompute_block(pos.func, old_block);
+    cfg.recompute_block(pos.func, new_block_resume);
+    cfg.recompute_block(pos.func, new_block_trap);
 }
 
 /// Jump tables.
@@ -292,7 +292,7 @@ fn expand_br_table_jt(
 ) {
     use crate::ir::condcodes::IntCC;
 
-    let (arg, default_ebb, table) = match func.dfg[inst] {
+    let (arg, default_block, table) = match func.dfg[inst] {
         ir::InstructionData::BranchTable {
             opcode: ir::Opcode::BrTable,
             arg,
@@ -304,22 +304,22 @@ fn expand_br_table_jt(
 
     // Rewrite:
     //
-    //     br_table $idx, default_ebb, $jt
+    //     br_table $idx, default_block, $jt
     //
     // To:
     //
     //     $oob = ifcmp_imm $idx, len($jt)
-    //     brif uge $oob, default_ebb
-    //     jump fallthrough_ebb
+    //     brif uge $oob, default_block
+    //     jump fallthrough_block
     //
-    //   fallthrough_ebb:
+    //   fallthrough_block:
     //     $base = jump_table_base.i64 $jt
     //     $rel_addr = jump_table_entry.i64 $idx, $base, 4, $jt
     //     $addr = iadd $base, $rel_addr
     //     indirect_jump_table_br $addr, $jt
 
-    let ebb = func.layout.pp_ebb(inst);
-    let jump_table_ebb = func.dfg.make_ebb();
+    let block = func.layout.pp_block(inst);
+    let jump_table_block = func.dfg.make_block();
 
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
@@ -330,9 +330,9 @@ fn expand_br_table_jt(
         .ins()
         .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, arg, table_size);
 
-    pos.ins().brnz(oob, default_ebb, &[]);
-    pos.ins().jump(jump_table_ebb, &[]);
-    pos.insert_ebb(jump_table_ebb);
+    pos.ins().brnz(oob, default_block, &[]);
+    pos.ins().jump(jump_table_block, &[]);
+    pos.insert_block(jump_table_block);
 
     let addr_ty = isa.pointer_type();
 
@@ -351,8 +351,8 @@ fn expand_br_table_jt(
     pos.ins().indirect_jump_table_br(addr, table);
 
     pos.remove_inst();
-    cfg.recompute_ebb(pos.func, ebb);
-    cfg.recompute_ebb(pos.func, jump_table_ebb);
+    cfg.recompute_block(pos.func, block);
+    cfg.recompute_block(pos.func, jump_table_block);
 }
 
 /// Expand br_table to series of conditionals.
@@ -364,7 +364,7 @@ fn expand_br_table_conds(
 ) {
     use crate::ir::condcodes::IntCC;
 
-    let (arg, default_ebb, table) = match func.dfg[inst] {
+    let (arg, default_block, table) = match func.dfg[inst] {
         ir::InstructionData::BranchTable {
             opcode: ir::Opcode::BrTable,
             arg,
@@ -374,15 +374,15 @@ fn expand_br_table_conds(
         _ => panic!("Expected br_table: {}", func.dfg.display_inst(inst, None)),
     };
 
-    let ebb = func.layout.pp_ebb(inst);
+    let block = func.layout.pp_block(inst);
 
     // This is a poor man's jump table using just a sequence of conditional branches.
     let table_size = func.jump_tables[table].len();
-    let mut cond_failed_ebb = vec![];
+    let mut cond_failed_block = vec![];
     if table_size >= 1 {
-        cond_failed_ebb = alloc::vec::Vec::with_capacity(table_size - 1);
+        cond_failed_block = alloc::vec::Vec::with_capacity(table_size - 1);
         for _ in 0..table_size - 1 {
-            cond_failed_ebb.push(func.dfg.make_ebb());
+            cond_failed_block.push(func.dfg.make_block());
         }
     }
 
@@ -397,19 +397,19 @@ fn expand_br_table_conds(
         pos.ins().brnz(t, dest, &[]);
         // Jump to the next case.
         if i < table_size - 1 {
-            let ebb = cond_failed_ebb[i];
-            pos.ins().jump(ebb, &[]);
-            pos.insert_ebb(ebb);
+            let block = cond_failed_block[i];
+            pos.ins().jump(block, &[]);
+            pos.insert_block(block);
         }
     }
 
     // `br_table` jumps to the default destination if nothing matches
-    pos.ins().jump(default_ebb, &[]);
+    pos.ins().jump(default_block, &[]);
 
     pos.remove_inst();
-    cfg.recompute_ebb(pos.func, ebb);
-    for failed_ebb in cond_failed_ebb.into_iter() {
-        cfg.recompute_ebb(pos.func, failed_ebb);
+    cfg.recompute_block(pos.func, block);
+    for failed_block in cond_failed_block.into_iter() {
+        cfg.recompute_block(pos.func, failed_block);
     }
 }
 
@@ -433,23 +433,23 @@ fn expand_select(
 
     // Replace `result = select ctrl, tval, fval` with:
     //
-    //   brnz ctrl, new_ebb(tval)
-    //   jump new_ebb(fval)
-    // new_ebb(result):
-    let old_ebb = func.layout.pp_ebb(inst);
+    //   brnz ctrl, new_block(tval)
+    //   jump new_block(fval)
+    // new_block(result):
+    let old_block = func.layout.pp_block(inst);
     let result = func.dfg.first_result(inst);
     func.dfg.clear_results(inst);
-    let new_ebb = func.dfg.make_ebb();
-    func.dfg.attach_ebb_param(new_ebb, result);
+    let new_block = func.dfg.make_block();
+    func.dfg.attach_block_param(new_block, result);
 
-    func.dfg.replace(inst).brnz(ctrl, new_ebb, &[tval]);
+    func.dfg.replace(inst).brnz(ctrl, new_block, &[tval]);
     let mut pos = FuncCursor::new(func).after_inst(inst);
     pos.use_srcloc(inst);
-    pos.ins().jump(new_ebb, &[fval]);
-    pos.insert_ebb(new_ebb);
+    pos.ins().jump(new_block, &[fval]);
+    pos.insert_block(new_block);
 
-    cfg.recompute_ebb(pos.func, new_ebb);
-    cfg.recompute_ebb(pos.func, old_ebb);
+    cfg.recompute_block(pos.func, new_block);
+    cfg.recompute_block(pos.func, old_block);
 }
 
 fn expand_br_icmp(
@@ -458,7 +458,7 @@ fn expand_br_icmp(
     cfg: &mut ControlFlowGraph,
     _isa: &dyn TargetIsa,
 ) {
-    let (cond, a, b, destination, ebb_args) = match func.dfg[inst] {
+    let (cond, a, b, destination, block_args) = match func.dfg[inst] {
         ir::InstructionData::BranchIcmp {
             cond,
             destination,
@@ -474,16 +474,16 @@ fn expand_br_icmp(
         _ => panic!("Expected br_icmp {}", func.dfg.display_inst(inst, None)),
     };
 
-    let old_ebb = func.layout.pp_ebb(inst);
+    let old_block = func.layout.pp_block(inst);
     func.dfg.clear_results(inst);
 
     let icmp_res = func.dfg.replace(inst).icmp(cond, a, b);
     let mut pos = FuncCursor::new(func).after_inst(inst);
     pos.use_srcloc(inst);
-    pos.ins().brnz(icmp_res, destination, &ebb_args);
+    pos.ins().brnz(icmp_res, destination, &block_args);
 
-    cfg.recompute_ebb(pos.func, destination);
-    cfg.recompute_ebb(pos.func, old_ebb);
+    cfg.recompute_block(pos.func, destination);
+    cfg.recompute_block(pos.func, old_block);
 }
 
 /// Expand illegal `f32const` and `f64const` instructions.
diff --git a/cranelift/codegen/src/legalizer/split.rs b/cranelift/codegen/src/legalizer/split.rs
index 62f89b3975..ea4a032163 100644
--- a/cranelift/codegen/src/legalizer/split.rs
+++ b/cranelift/codegen/src/legalizer/split.rs
@@ -54,19 +54,19 @@
 //! This means that the `iconcat` instructions defining `v1` and `v4` end up with no uses, so they
 //! can be trivially deleted by a dead code elimination pass.
 //!
-//! # EBB arguments
+//! # block arguments
 //!
 //! If all instructions that produce an `i64` value are legalized as above, we will eventually end
-//! up with no `i64` values anywhere, except for EBB arguments. We can work around this by
-//! iteratively splitting EBB arguments too. That should leave us with no illegal value types
+//! up with no `i64` values anywhere, except for block arguments. We can work around this by
+//! iteratively splitting block arguments too. That should leave us with no illegal value types
 //! anywhere.
 //!
-//! It is possible to have circular dependencies of EBB arguments that are never used by any real
+//! It is possible to have circular dependencies of block arguments that are never used by any real
 //! instructions. These loops will remain in the program.
 
 use crate::cursor::{Cursor, CursorPosition, FuncCursor};
-use crate::flowgraph::{BasicBlock, ControlFlowGraph};
-use crate::ir::{self, Ebb, Inst, InstBuilder, InstructionData, Opcode, Type, Value, ValueDef};
+use crate::flowgraph::{BlockPredecessor, ControlFlowGraph};
+use crate::ir::{self, Block, Inst, InstBuilder, InstructionData, Opcode, Type, Value, ValueDef};
 use alloc::vec::Vec;
 use core::iter;
 use smallvec::SmallVec;
@@ -95,7 +95,7 @@ pub fn vsplit(
     split_any(func, cfg, pos, srcloc, value, Opcode::Vconcat)
 }
 
-/// After splitting an EBB argument, we need to go back and fix up all of the predecessor
+/// After splitting an block argument, we need to go back and fix up all of the predecessor
 /// instructions. This is potentially a recursive operation, but we don't implement it recursively
 /// since that could use up too muck stack.
 ///
@@ -104,11 +104,11 @@ struct Repair {
     concat: Opcode,
     // The argument type after splitting.
     split_type: Type,
-    // The destination EBB whose arguments have been split.
-    ebb: Ebb,
-    // Number of the original EBB argument which has been replaced by the low part.
+    // The destination block whose arguments have been split.
+    block: Block,
+    // Number of the original block argument which has been replaced by the low part.
     num: usize,
-    // Number of the new EBB argument which represents the high part after the split.
+    // Number of the new block argument which represents the high part after the split.
     hi_num: usize,
 }
 
@@ -130,9 +130,9 @@ fn split_any(
     result
 }
 
-pub fn split_ebb_params(func: &mut ir::Function, cfg: &ControlFlowGraph, ebb: Ebb) {
-    let pos = &mut FuncCursor::new(func).at_top(ebb);
-    let ebb_params = pos.func.dfg.ebb_params(ebb);
+pub fn split_block_params(func: &mut ir::Function, cfg: &ControlFlowGraph, block: Block) {
+    let pos = &mut FuncCursor::new(func).at_top(block);
+    let block_params = pos.func.dfg.block_params(block);
 
     // Add further splittable types here.
     fn type_requires_splitting(ty: Type) -> bool {
@@ -140,31 +140,31 @@ pub fn split_ebb_params(func: &mut ir::Function, cfg: &ControlFlowGraph, ebb: Eb
     }
 
     // A shortcut.  If none of the param types require splitting, exit now.  This helps because
-    // the loop below necessarily has to copy the ebb params into a new vector, so it's better to
+    // the loop below necessarily has to copy the block params into a new vector, so it's better to
     // avoid doing so when possible.
-    if !ebb_params
+    if !block_params
         .iter()
-        .any(|ebb_param| type_requires_splitting(pos.func.dfg.value_type(*ebb_param)))
+        .any(|block_param| type_requires_splitting(pos.func.dfg.value_type(*block_param)))
     {
         return;
     }
 
     let mut repairs = Vec::new();
-    for (num, ebb_param) in ebb_params.to_vec().into_iter().enumerate() {
-        if !type_requires_splitting(pos.func.dfg.value_type(ebb_param)) {
+    for (num, block_param) in block_params.to_vec().into_iter().enumerate() {
+        if !type_requires_splitting(pos.func.dfg.value_type(block_param)) {
             continue;
         }
 
-        split_ebb_param(pos, ebb, num, ebb_param, Opcode::Iconcat, &mut repairs);
+        split_block_param(pos, block, num, block_param, Opcode::Iconcat, &mut repairs);
     }
 
     perform_repairs(pos, cfg, repairs);
 }
 
 fn perform_repairs(pos: &mut FuncCursor, cfg: &ControlFlowGraph, mut repairs: Vec) {
-    // We have split the value requested, and now we may need to fix some EBB predecessors.
+    // We have split the value requested, and now we may need to fix some block predecessors.
     while let Some(repair) = repairs.pop() {
-        for BasicBlock { inst, .. } in cfg.pred_iter(repair.ebb) {
+        for BlockPredecessor { inst, .. } in cfg.pred_iter(repair.block) {
             let branch_opc = pos.func.dfg[inst].opcode();
             debug_assert!(
                 branch_opc.is_branch(),
@@ -176,7 +176,7 @@ fn perform_repairs(pos: &mut FuncCursor, cfg: &ControlFlowGraph, mut repairs: Ve
                 .take_value_list()
                 .expect("Branches must have value lists.");
             let num_args = args.len(&pos.func.dfg.value_lists);
-            // Get the old value passed to the EBB argument we're repairing.
+            // Get the old value passed to the block argument we're repairing.
             let old_arg = args
                 .get(num_fixed_args + repair.num, &pos.func.dfg.value_lists)
                 .expect("Too few branch arguments");
@@ -190,13 +190,13 @@ fn perform_repairs(pos: &mut FuncCursor, cfg: &ControlFlowGraph, mut repairs: Ve
             // Split the old argument, possibly causing more repairs to be scheduled.
             pos.goto_inst(inst);
 
-            let inst_ebb = pos.func.layout.inst_ebb(inst).expect("inst in ebb");
+            let inst_block = pos.func.layout.inst_block(inst).expect("inst in block");
 
             // Insert split values prior to the terminal branch group.
             let canonical = pos
                 .func
                 .layout
-                .canonical_branch_inst(&pos.func.dfg, inst_ebb);
+                .canonical_branch_inst(&pos.func.dfg, inst_block);
             if let Some(first_branch) = canonical {
                 pos.goto_inst(first_branch);
             }
@@ -209,7 +209,7 @@ fn perform_repairs(pos: &mut FuncCursor, cfg: &ControlFlowGraph, mut repairs: Ve
                 .unwrap() = lo;
 
             // The `hi` part goes at the end. Since multiple repairs may have been scheduled to the
-            // same EBB, there could be multiple arguments missing.
+            // same block, there could be multiple arguments missing.
             if num_args > num_fixed_args + repair.hi_num {
                 *args
                     .get_mut(
@@ -259,11 +259,11 @@ fn split_value(
                 }
             }
         }
-        ValueDef::Param(ebb, num) => {
-            // This is an EBB parameter.
+        ValueDef::Param(block, num) => {
+            // This is an block parameter.
             // We can split the parameter value unless this is the entry block.
-            if pos.func.layout.entry_block() != Some(ebb) {
-                reuse = Some(split_ebb_param(pos, ebb, num, value, concat, repairs));
+            if pos.func.layout.entry_block() != Some(block) {
+                reuse = Some(split_block_param(pos, block, num, value, concat, repairs));
             }
         }
     }
@@ -273,7 +273,7 @@ fn split_value(
         pair
     } else {
         // No, we'll just have to insert the requested split instruction at `pos`. Note that `pos`
-        // has not been moved by the EBB argument code above when `reuse` is `None`.
+        // has not been moved by the block argument code above when `reuse` is `None`.
         match concat {
             Opcode::Iconcat => pos.ins().isplit(value),
             Opcode::Vconcat => pos.ins().vsplit(value),
@@ -282,9 +282,9 @@ fn split_value(
     }
 }
 
-fn split_ebb_param(
+fn split_block_param(
     pos: &mut FuncCursor,
-    ebb: Ebb,
+    block: Block,
     param_num: usize,
     value: Value,
     concat: Opcode,
@@ -300,14 +300,14 @@ fn split_ebb_param(
     };
 
     // Since the `repairs` stack potentially contains other parameter numbers for
-    // `ebb`, avoid shifting and renumbering EBB parameters. It could invalidate other
+    // `block`, avoid shifting and renumbering block parameters. It could invalidate other
     // `repairs` entries.
     //
     // Replace the original `value` with the low part, and append the high part at the
     // end of the argument list.
-    let lo = pos.func.dfg.replace_ebb_param(value, split_type);
-    let hi_num = pos.func.dfg.num_ebb_params(ebb);
-    let hi = pos.func.dfg.append_ebb_param(ebb, split_type);
+    let lo = pos.func.dfg.replace_block_param(value, split_type);
+    let hi_num = pos.func.dfg.num_block_params(block);
+    let hi = pos.func.dfg.append_block_param(block, split_type);
 
     // Now the original value is dangling. Insert a concatenation instruction that can
     // compute it from the two new parameters. This also serves as a record of what we
@@ -315,14 +315,14 @@ fn split_ebb_param(
     //
     // Note that it is safe to move `pos` here since `reuse` was set above, so we don't
     // need to insert a split instruction before returning.
-    pos.goto_first_inst(ebb);
+    pos.goto_first_inst(block);
     pos.ins()
         .with_result(value)
         .Binary(concat, split_type, lo, hi);
 
-    // Finally, splitting the EBB parameter is not enough. We also have to repair all
+    // Finally, splitting the block parameter is not enough. We also have to repair all
     // of the predecessor instructions that branch here.
-    add_repair(concat, split_type, ebb, param_num, hi_num, repairs);
+    add_repair(concat, split_type, block, param_num, hi_num, repairs);
 
     (lo, hi)
 }
@@ -331,7 +331,7 @@ fn split_ebb_param(
 fn add_repair(
     concat: Opcode,
     split_type: Type,
-    ebb: Ebb,
+    block: Block,
     num: usize,
     hi_num: usize,
     repairs: &mut Vec,
@@ -339,7 +339,7 @@ fn add_repair(
     repairs.push(Repair {
         concat,
         split_type,
-        ebb,
+        block,
         num,
         hi_num,
     });
diff --git a/cranelift/codegen/src/licm.rs b/cranelift/codegen/src/licm.rs
index 4fa5d314f6..75000b5297 100644
--- a/cranelift/codegen/src/licm.rs
+++ b/cranelift/codegen/src/licm.rs
@@ -3,10 +3,10 @@
 use crate::cursor::{Cursor, EncCursor, FuncCursor};
 use crate::dominator_tree::DominatorTree;
 use crate::entity::{EntityList, ListPool};
-use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::flowgraph::{BlockPredecessor, ControlFlowGraph};
 use crate::fx::FxHashSet;
 use crate::ir::{
-    DataFlowGraph, Ebb, Function, Inst, InstBuilder, InstructionData, Layout, Opcode, Type, Value,
+    Block, DataFlowGraph, Function, Inst, InstBuilder, InstructionData, Layout, Opcode, Type, Value,
 };
 use crate::isa::TargetIsa;
 use crate::loop_analysis::{Loop, LoopAnalysis};
@@ -65,23 +65,23 @@ pub fn do_licm(
 // A jump instruction to the header is placed at the end of the pre-header.
 fn create_pre_header(
     isa: &dyn TargetIsa,
-    header: Ebb,
+    header: Block,
     func: &mut Function,
     cfg: &mut ControlFlowGraph,
     domtree: &DominatorTree,
-) -> Ebb {
+) -> Block {
     let pool = &mut ListPool::::new();
-    let header_args_values = func.dfg.ebb_params(header).to_vec();
+    let header_args_values = func.dfg.block_params(header).to_vec();
     let header_args_types: Vec = header_args_values
         .into_iter()
         .map(|val| func.dfg.value_type(val))
         .collect();
-    let pre_header = func.dfg.make_ebb();
+    let pre_header = func.dfg.make_block();
     let mut pre_header_args_value: EntityList = EntityList::new();
     for typ in header_args_types {
-        pre_header_args_value.push(func.dfg.append_ebb_param(pre_header, typ), pool);
+        pre_header_args_value.push(func.dfg.append_block_param(pre_header, typ), pool);
     }
-    for BasicBlock {
+    for BlockPredecessor {
         inst: last_inst, ..
     } in cfg.pred_iter(header)
     {
@@ -93,7 +93,7 @@ fn create_pre_header(
     {
         let mut pos = EncCursor::new(func, isa).at_top(header);
         // Inserts the pre-header at the right place in the layout.
-        pos.insert_ebb(pre_header);
+        pos.insert_block(pre_header);
         pos.next_inst();
         pos.ins().jump(header, pre_header_args_value.as_slice(pool));
     }
@@ -104,16 +104,16 @@ fn create_pre_header(
 //
 // A loop header has a pre-header if there is only one predecessor that the header doesn't
 // dominate.
-// Returns the pre-header Ebb and the instruction jumping to the header.
+// Returns the pre-header Block and the instruction jumping to the header.
 fn has_pre_header(
     layout: &Layout,
     cfg: &ControlFlowGraph,
     domtree: &DominatorTree,
-    header: Ebb,
-) -> Option<(Ebb, Inst)> {
+    header: Block,
+) -> Option<(Block, Inst)> {
     let mut result = None;
-    for BasicBlock {
-        ebb: pred_ebb,
+    for BlockPredecessor {
+        block: pred_block,
         inst: branch_inst,
     } in cfg.pred_iter(header)
     {
@@ -123,13 +123,13 @@ fn has_pre_header(
                 // We have already found one, there are more than one
                 return None;
             }
-            if branch_inst != layout.last_inst(pred_ebb).unwrap()
-                || cfg.succ_iter(pred_ebb).nth(1).is_some()
+            if branch_inst != layout.last_inst(pred_block).unwrap()
+                || cfg.succ_iter(pred_block).nth(1).is_some()
             {
                 // It's along a critical edge, so don't use it.
                 return None;
             }
-            result = Some((pred_ebb, branch_inst));
+            result = Some((pred_block, branch_inst));
         }
     }
     result
@@ -176,7 +176,7 @@ fn is_loop_invariant(inst: Inst, dfg: &DataFlowGraph, loop_values: &FxHashSet = FxHashSet();
     let mut invariant_insts: Vec = Vec::new();
     let mut pos = FuncCursor::new(func);
-    // We traverse the loop EBB in reverse post-order.
-    for ebb in postorder_ebbs_loop(loop_analysis, cfg, lp).iter().rev() {
-        // Arguments of the EBB are loop values
-        for val in pos.func.dfg.ebb_params(*ebb) {
+    // We traverse the loop block in reverse post-order.
+    for block in postorder_blocks_loop(loop_analysis, cfg, lp).iter().rev() {
+        // Arguments of the block are loop values
+        for val in pos.func.dfg.block_params(*block) {
             loop_values.insert(*val);
         }
-        pos.goto_top(*ebb);
+        pos.goto_top(*block);
         #[cfg_attr(feature = "cargo-clippy", allow(clippy::block_in_if_condition_stmt))]
         while let Some(inst) = pos.next_inst() {
             if is_loop_invariant(inst, &pos.func.dfg, &loop_values) {
@@ -215,8 +215,12 @@ fn remove_loop_invariant_instructions(
     invariant_insts
 }
 
-/// Return ebbs from a loop in post-order, starting from an entry point in the block.
-fn postorder_ebbs_loop(loop_analysis: &LoopAnalysis, cfg: &ControlFlowGraph, lp: Loop) -> Vec {
+/// Return blocks from a loop in post-order, starting from an entry point in the block.
+fn postorder_blocks_loop(
+    loop_analysis: &LoopAnalysis,
+    cfg: &ControlFlowGraph,
+    lp: Loop,
+) -> Vec {
     let mut grey = FxHashSet();
     let mut black = FxHashSet();
     let mut stack = vec![loop_analysis.loop_header(lp)];
diff --git a/cranelift/codegen/src/loop_analysis.rs b/cranelift/codegen/src/loop_analysis.rs
index 40de9afdff..dc659bc5f2 100644
--- a/cranelift/codegen/src/loop_analysis.rs
+++ b/cranelift/codegen/src/loop_analysis.rs
@@ -1,12 +1,12 @@
-//! A loop analysis represented as mappings of loops to their header Ebb
+//! A loop analysis represented as mappings of loops to their header Block
 //! and parent in the loop tree.
 
 use crate::dominator_tree::DominatorTree;
 use crate::entity::entity_impl;
 use crate::entity::SecondaryMap;
 use crate::entity::{Keys, PrimaryMap};
-use crate::flowgraph::{BasicBlock, ControlFlowGraph};
-use crate::ir::{Ebb, Function, Layout};
+use crate::flowgraph::{BlockPredecessor, ControlFlowGraph};
+use crate::ir::{Block, Function, Layout};
 use crate::packed_option::PackedOption;
 use crate::timing;
 use alloc::vec::Vec;
@@ -18,22 +18,22 @@ entity_impl!(Loop, "loop");
 
 /// Loop tree information for a single function.
 ///
-/// Loops are referenced by the Loop object, and for each loop you can access its header EBB,
-/// its eventual parent in the loop tree and all the EBB belonging to the loop.
+/// Loops are referenced by the Loop object, and for each loop you can access its header block,
+/// its eventual parent in the loop tree and all the block belonging to the loop.
 pub struct LoopAnalysis {
     loops: PrimaryMap,
-    ebb_loop_map: SecondaryMap>,
+    block_loop_map: SecondaryMap>,
     valid: bool,
 }
 
 struct LoopData {
-    header: Ebb,
+    header: Block,
     parent: PackedOption,
 }
 
 impl LoopData {
     /// Creates a `LoopData` object with the loop header and its eventual parent in the loop tree.
-    pub fn new(header: Ebb, parent: Option) -> Self {
+    pub fn new(header: Block, parent: Option) -> Self {
         Self {
             header,
             parent: parent.into(),
@@ -49,7 +49,7 @@ impl LoopAnalysis {
         Self {
             valid: false,
             loops: PrimaryMap::new(),
-            ebb_loop_map: SecondaryMap::new(),
+            block_loop_map: SecondaryMap::new(),
         }
     }
 
@@ -58,11 +58,11 @@ impl LoopAnalysis {
         self.loops.keys()
     }
 
-    /// Returns the header EBB of a particular loop.
+    /// Returns the header block of a particular loop.
     ///
     /// The characteristic property of a loop header block is that it dominates some of its
     /// predecessors.
-    pub fn loop_header(&self, lp: Loop) -> Ebb {
+    pub fn loop_header(&self, lp: Loop) -> Block {
         self.loops[lp].header
     }
 
@@ -71,14 +71,14 @@ impl LoopAnalysis {
         self.loops[lp].parent.expand()
     }
 
-    /// Determine if an Ebb belongs to a loop by running a finger along the loop tree.
+    /// Determine if an Block belongs to a loop by running a finger along the loop tree.
     ///
-    /// Returns `true` if `ebb` is in loop `lp`.
-    pub fn is_in_loop(&self, ebb: Ebb, lp: Loop) -> bool {
-        let ebb_loop = self.ebb_loop_map[ebb];
-        match ebb_loop.expand() {
+    /// Returns `true` if `block` is in loop `lp`.
+    pub fn is_in_loop(&self, block: Block, lp: Loop) -> bool {
+        let block_loop = self.block_loop_map[block];
+        match block_loop.expand() {
             None => false,
-            Some(ebb_loop) => self.is_child_loop(ebb_loop, lp),
+            Some(block_loop) => self.is_child_loop(block_loop, lp),
         }
     }
 
@@ -103,8 +103,8 @@ impl LoopAnalysis {
     pub fn compute(&mut self, func: &Function, cfg: &ControlFlowGraph, domtree: &DominatorTree) {
         let _tt = timing::loop_analysis();
         self.loops.clear();
-        self.ebb_loop_map.clear();
-        self.ebb_loop_map.resize(func.dfg.num_ebbs());
+        self.block_loop_map.clear();
+        self.block_loop_map.resize(func.dfg.num_blocks());
         self.find_loop_headers(cfg, domtree, &func.layout);
         self.discover_loop_blocks(cfg, domtree, &func.layout);
         self.valid = true;
@@ -124,11 +124,11 @@ impl LoopAnalysis {
     /// memory be retained.
     pub fn clear(&mut self) {
         self.loops.clear();
-        self.ebb_loop_map.clear();
+        self.block_loop_map.clear();
         self.valid = false;
     }
 
-    // Traverses the CFG in reverse postorder and create a loop object for every EBB having a
+    // Traverses the CFG in reverse postorder and create a loop object for every block having a
     // back edge.
     fn find_loop_headers(
         &mut self,
@@ -137,16 +137,16 @@ impl LoopAnalysis {
         layout: &Layout,
     ) {
         // We traverse the CFG in reverse postorder
-        for &ebb in domtree.cfg_postorder().iter().rev() {
-            for BasicBlock {
+        for &block in domtree.cfg_postorder().iter().rev() {
+            for BlockPredecessor {
                 inst: pred_inst, ..
-            } in cfg.pred_iter(ebb)
+            } in cfg.pred_iter(block)
             {
-                // If the ebb dominates one of its predecessors it is a back edge
-                if domtree.dominates(ebb, pred_inst, layout) {
-                    // This ebb is a loop header, so we create its associated loop
-                    let lp = self.loops.push(LoopData::new(ebb, None));
-                    self.ebb_loop_map[ebb] = lp.into();
+                // If the block dominates one of its predecessors it is a back edge
+                if domtree.dominates(block, pred_inst, layout) {
+                    // This block is a loop header, so we create its associated loop
+                    let lp = self.loops.push(LoopData::new(block, None));
+                    self.block_loop_map[block] = lp.into();
                     break;
                     // We break because we only need one back edge to identify a loop header.
                 }
@@ -155,7 +155,7 @@ impl LoopAnalysis {
     }
 
     // Intended to be called after `find_loop_headers`. For each detected loop header,
-    // discovers all the ebb belonging to the loop and its inner loops. After a call to this
+    // discovers all the block belonging to the loop and its inner loops. After a call to this
     // function, the loop tree is fully constructed.
     fn discover_loop_blocks(
         &mut self,
@@ -163,12 +163,12 @@ impl LoopAnalysis {
         domtree: &DominatorTree,
         layout: &Layout,
     ) {
-        let mut stack: Vec = Vec::new();
+        let mut stack: Vec = Vec::new();
         // We handle each loop header in reverse order, corresponding to a pseudo postorder
         // traversal of the graph.
         for lp in self.loops().rev() {
-            for BasicBlock {
-                ebb: pred,
+            for BlockPredecessor {
+                block: pred,
                 inst: pred_inst,
             } in cfg.pred_iter(self.loops[lp].header)
             {
@@ -178,11 +178,11 @@ impl LoopAnalysis {
                 }
             }
             while let Some(node) = stack.pop() {
-                let continue_dfs: Option;
-                match self.ebb_loop_map[node].expand() {
+                let continue_dfs: Option;
+                match self.block_loop_map[node].expand() {
                     None => {
                         // The node hasn't been visited yet, we tag it as part of the loop
-                        self.ebb_loop_map[node] = PackedOption::from(lp);
+                        self.block_loop_map[node] = PackedOption::from(lp);
                         continue_dfs = Some(node);
                     }
                     Some(node_loop) => {
@@ -221,7 +221,7 @@ impl LoopAnalysis {
                 // Now we have handled the popped node and need to continue the DFS by adding the
                 // predecessors of that node
                 if let Some(continue_dfs) = continue_dfs {
-                    for BasicBlock { ebb: pred, .. } in cfg.pred_iter(continue_dfs) {
+                    for BlockPredecessor { block: pred, .. } in cfg.pred_iter(continue_dfs) {
                         stack.push(pred)
                     }
                 }
@@ -242,27 +242,27 @@ mod tests {
     #[test]
     fn nested_loops_detection() {
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb1 = func.dfg.make_ebb();
-        let ebb2 = func.dfg.make_ebb();
-        let ebb3 = func.dfg.make_ebb();
-        let cond = func.dfg.append_ebb_param(ebb0, types::I32);
+        let block0 = func.dfg.make_block();
+        let block1 = func.dfg.make_block();
+        let block2 = func.dfg.make_block();
+        let block3 = func.dfg.make_block();
+        let cond = func.dfg.append_block_param(block0, types::I32);
 
         {
             let mut cur = FuncCursor::new(&mut func);
 
-            cur.insert_ebb(ebb0);
-            cur.ins().jump(ebb1, &[]);
+            cur.insert_block(block0);
+            cur.ins().jump(block1, &[]);
 
-            cur.insert_ebb(ebb1);
-            cur.ins().jump(ebb2, &[]);
+            cur.insert_block(block1);
+            cur.ins().jump(block2, &[]);
 
-            cur.insert_ebb(ebb2);
-            cur.ins().brnz(cond, ebb1, &[]);
-            cur.ins().jump(ebb3, &[]);
+            cur.insert_block(block2);
+            cur.ins().brnz(cond, block1, &[]);
+            cur.ins().jump(block3, &[]);
 
-            cur.insert_ebb(ebb3);
-            cur.ins().brnz(cond, ebb0, &[]);
+            cur.insert_block(block3);
+            cur.ins().brnz(cond, block0, &[]);
         }
 
         let mut loop_analysis = LoopAnalysis::new();
@@ -274,54 +274,54 @@ mod tests {
 
         let loops = loop_analysis.loops().collect::>();
         assert_eq!(loops.len(), 2);
-        assert_eq!(loop_analysis.loop_header(loops[0]), ebb0);
-        assert_eq!(loop_analysis.loop_header(loops[1]), ebb1);
+        assert_eq!(loop_analysis.loop_header(loops[0]), block0);
+        assert_eq!(loop_analysis.loop_header(loops[1]), block1);
         assert_eq!(loop_analysis.loop_parent(loops[1]), Some(loops[0]));
         assert_eq!(loop_analysis.loop_parent(loops[0]), None);
-        assert_eq!(loop_analysis.is_in_loop(ebb0, loops[0]), true);
-        assert_eq!(loop_analysis.is_in_loop(ebb0, loops[1]), false);
-        assert_eq!(loop_analysis.is_in_loop(ebb1, loops[1]), true);
-        assert_eq!(loop_analysis.is_in_loop(ebb1, loops[0]), true);
-        assert_eq!(loop_analysis.is_in_loop(ebb2, loops[1]), true);
-        assert_eq!(loop_analysis.is_in_loop(ebb2, loops[0]), true);
-        assert_eq!(loop_analysis.is_in_loop(ebb3, loops[0]), true);
-        assert_eq!(loop_analysis.is_in_loop(ebb0, loops[1]), false);
+        assert_eq!(loop_analysis.is_in_loop(block0, loops[0]), true);
+        assert_eq!(loop_analysis.is_in_loop(block0, loops[1]), false);
+        assert_eq!(loop_analysis.is_in_loop(block1, loops[1]), true);
+        assert_eq!(loop_analysis.is_in_loop(block1, loops[0]), true);
+        assert_eq!(loop_analysis.is_in_loop(block2, loops[1]), true);
+        assert_eq!(loop_analysis.is_in_loop(block2, loops[0]), true);
+        assert_eq!(loop_analysis.is_in_loop(block3, loops[0]), true);
+        assert_eq!(loop_analysis.is_in_loop(block0, loops[1]), false);
     }
 
     #[test]
     fn complex_loop_detection() {
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb1 = func.dfg.make_ebb();
-        let ebb2 = func.dfg.make_ebb();
-        let ebb3 = func.dfg.make_ebb();
-        let ebb4 = func.dfg.make_ebb();
-        let ebb5 = func.dfg.make_ebb();
-        let cond = func.dfg.append_ebb_param(ebb0, types::I32);
+        let block0 = func.dfg.make_block();
+        let block1 = func.dfg.make_block();
+        let block2 = func.dfg.make_block();
+        let block3 = func.dfg.make_block();
+        let block4 = func.dfg.make_block();
+        let block5 = func.dfg.make_block();
+        let cond = func.dfg.append_block_param(block0, types::I32);
 
         {
             let mut cur = FuncCursor::new(&mut func);
 
-            cur.insert_ebb(ebb0);
-            cur.ins().brnz(cond, ebb1, &[]);
-            cur.ins().jump(ebb3, &[]);
+            cur.insert_block(block0);
+            cur.ins().brnz(cond, block1, &[]);
+            cur.ins().jump(block3, &[]);
 
-            cur.insert_ebb(ebb1);
-            cur.ins().jump(ebb2, &[]);
+            cur.insert_block(block1);
+            cur.ins().jump(block2, &[]);
 
-            cur.insert_ebb(ebb2);
-            cur.ins().brnz(cond, ebb1, &[]);
-            cur.ins().jump(ebb5, &[]);
+            cur.insert_block(block2);
+            cur.ins().brnz(cond, block1, &[]);
+            cur.ins().jump(block5, &[]);
 
-            cur.insert_ebb(ebb3);
-            cur.ins().jump(ebb4, &[]);
+            cur.insert_block(block3);
+            cur.ins().jump(block4, &[]);
 
-            cur.insert_ebb(ebb4);
-            cur.ins().brnz(cond, ebb3, &[]);
-            cur.ins().jump(ebb5, &[]);
+            cur.insert_block(block4);
+            cur.ins().brnz(cond, block3, &[]);
+            cur.ins().jump(block5, &[]);
 
-            cur.insert_ebb(ebb5);
-            cur.ins().brnz(cond, ebb0, &[]);
+            cur.insert_block(block5);
+            cur.ins().brnz(cond, block0, &[]);
         }
 
         let mut loop_analysis = LoopAnalysis::new();
@@ -333,17 +333,17 @@ mod tests {
 
         let loops = loop_analysis.loops().collect::>();
         assert_eq!(loops.len(), 3);
-        assert_eq!(loop_analysis.loop_header(loops[0]), ebb0);
-        assert_eq!(loop_analysis.loop_header(loops[1]), ebb1);
-        assert_eq!(loop_analysis.loop_header(loops[2]), ebb3);
+        assert_eq!(loop_analysis.loop_header(loops[0]), block0);
+        assert_eq!(loop_analysis.loop_header(loops[1]), block1);
+        assert_eq!(loop_analysis.loop_header(loops[2]), block3);
         assert_eq!(loop_analysis.loop_parent(loops[1]), Some(loops[0]));
         assert_eq!(loop_analysis.loop_parent(loops[2]), Some(loops[0]));
         assert_eq!(loop_analysis.loop_parent(loops[0]), None);
-        assert_eq!(loop_analysis.is_in_loop(ebb0, loops[0]), true);
-        assert_eq!(loop_analysis.is_in_loop(ebb1, loops[1]), true);
-        assert_eq!(loop_analysis.is_in_loop(ebb2, loops[1]), true);
-        assert_eq!(loop_analysis.is_in_loop(ebb3, loops[2]), true);
-        assert_eq!(loop_analysis.is_in_loop(ebb4, loops[2]), true);
-        assert_eq!(loop_analysis.is_in_loop(ebb5, loops[0]), true);
+        assert_eq!(loop_analysis.is_in_loop(block0, loops[0]), true);
+        assert_eq!(loop_analysis.is_in_loop(block1, loops[1]), true);
+        assert_eq!(loop_analysis.is_in_loop(block2, loops[1]), true);
+        assert_eq!(loop_analysis.is_in_loop(block3, loops[2]), true);
+        assert_eq!(loop_analysis.is_in_loop(block4, loops[2]), true);
+        assert_eq!(loop_analysis.is_in_loop(block5, loops[0]), true);
     }
 }
diff --git a/cranelift/codegen/src/nan_canonicalization.rs b/cranelift/codegen/src/nan_canonicalization.rs
index 235203b764..e7c0e53419 100644
--- a/cranelift/codegen/src/nan_canonicalization.rs
+++ b/cranelift/codegen/src/nan_canonicalization.rs
@@ -18,7 +18,7 @@ static CANON_64BIT_NAN: u64 = 0b011111111111100000000000000000000000000000000000
 pub fn do_nan_canonicalization(func: &mut Function) {
     let _tt = timing::canonicalize_nans();
     let mut pos = FuncCursor::new(func);
-    while let Some(_ebb) = pos.next_ebb() {
+    while let Some(_block) = pos.next_block() {
         while let Some(inst) = pos.next_inst() {
             if is_fp_arith(&mut pos, inst) {
                 add_nan_canon_seq(&mut pos, inst);
@@ -59,7 +59,7 @@ fn add_nan_canon_seq(pos: &mut FuncCursor, inst: Inst) {
     let val = pos.func.dfg.first_result(inst);
     let val_type = pos.func.dfg.value_type(val);
     let new_res = pos.func.dfg.replace_result(val, val_type);
-    let _next_inst = pos.next_inst().expect("EBB missing terminator!");
+    let _next_inst = pos.next_inst().expect("block missing terminator!");
 
     // Insert a comparison instruction, to check if `inst_res` is NaN. Select
     // the canonical NaN value if `val` is NaN, assign the result to `inst`.
diff --git a/cranelift/codegen/src/postopt.rs b/cranelift/codegen/src/postopt.rs
index ef027f6f25..42121817d5 100644
--- a/cranelift/codegen/src/postopt.rs
+++ b/cranelift/codegen/src/postopt.rs
@@ -7,7 +7,7 @@ use crate::ir::condcodes::{CondCode, FloatCC, IntCC};
 use crate::ir::dfg::ValueDef;
 use crate::ir::immediates::{Imm64, Offset32};
 use crate::ir::instructions::{Opcode, ValueList};
-use crate::ir::{Ebb, Function, Inst, InstBuilder, InstructionData, MemFlags, Type, Value};
+use crate::ir::{Block, Function, Inst, InstBuilder, InstructionData, MemFlags, Type, Value};
 use crate::isa::TargetIsa;
 use crate::timing;
 
@@ -18,7 +18,7 @@ struct CmpBrInfo {
     /// The icmp, icmp_imm, or fcmp instruction.
     cmp_inst: Inst,
     /// The destination of the branch.
-    destination: Ebb,
+    destination: Block,
     /// The arguments of the branch.
     args: ValueList,
     /// The first argument to the comparison. The second is in the `kind` field.
@@ -360,7 +360,7 @@ fn optimize_complex_addresses(pos: &mut EncCursor, inst: Inst, isa: &dyn TargetI
 pub fn do_postopt(func: &mut Function, isa: &dyn TargetIsa) {
     let _tt = timing::postopt();
     let mut pos = EncCursor::new(func, isa);
-    while let Some(_ebb) = pos.next_ebb() {
+    while let Some(_block) = pos.next_block() {
         let mut last_flags_clobber = None;
         while let Some(inst) = pos.next_inst() {
             if isa.uses_cpu_flags() {
diff --git a/cranelift/codegen/src/print_errors.rs b/cranelift/codegen/src/print_errors.rs
index 78ae325630..e4f6234ebd 100644
--- a/cranelift/codegen/src/print_errors.rs
+++ b/cranelift/codegen/src/print_errors.rs
@@ -2,7 +2,7 @@
 
 use crate::entity::SecondaryMap;
 use crate::ir;
-use crate::ir::entities::{AnyEntity, Ebb, Inst, Value};
+use crate::ir::entities::{AnyEntity, Block, Inst, Value};
 use crate::ir::function::Function;
 use crate::isa::TargetIsa;
 use crate::result::CodegenError;
@@ -47,15 +47,15 @@ pub fn pretty_verifier_error<'a>(
 struct PrettyVerifierError<'a>(Box, &'a mut Vec);
 
 impl<'a> FuncWriter for PrettyVerifierError<'a> {
-    fn write_ebb_header(
+    fn write_block_header(
         &mut self,
         w: &mut dyn Write,
         func: &Function,
         isa: Option<&dyn TargetIsa>,
-        ebb: Ebb,
+        block: Block,
         indent: usize,
     ) -> fmt::Result {
-        pretty_ebb_header_error(w, func, isa, ebb, indent, &mut *self.0, self.1)
+        pretty_block_header_error(w, func, isa, block, indent, &mut *self.0, self.1)
     }
 
     fn write_instruction(
@@ -81,18 +81,18 @@ impl<'a> FuncWriter for PrettyVerifierError<'a> {
     }
 }
 
-/// Pretty-print a function verifier error for a given EBB.
-fn pretty_ebb_header_error(
+/// Pretty-print a function verifier error for a given block.
+fn pretty_block_header_error(
     w: &mut dyn Write,
     func: &Function,
     isa: Option<&dyn TargetIsa>,
-    cur_ebb: Ebb,
+    cur_block: Block,
     indent: usize,
     func_w: &mut dyn FuncWriter,
     errors: &mut Vec,
 ) -> fmt::Result {
     let mut s = String::new();
-    func_w.write_ebb_header(&mut s, func, isa, cur_ebb, indent)?;
+    func_w.write_block_header(&mut s, func, isa, cur_block, indent)?;
     write!(w, "{}", s)?;
 
     // TODO: Use drain_filter here when it gets stabilized
@@ -100,7 +100,7 @@ fn pretty_ebb_header_error(
     let mut printed_error = false;
     while i != errors.len() {
         match errors[i].location {
-            ir::entities::AnyEntity::Ebb(ebb) if ebb == cur_ebb => {
+            ir::entities::AnyEntity::Block(block) if block == cur_block => {
                 if !printed_error {
                     print_arrow(w, &s)?;
                     printed_error = true;
diff --git a/cranelift/codegen/src/redundant_reload_remover.rs b/cranelift/codegen/src/redundant_reload_remover.rs
index 207cec2286..f33eb98fde 100644
--- a/cranelift/codegen/src/redundant_reload_remover.rs
+++ b/cranelift/codegen/src/redundant_reload_remover.rs
@@ -8,7 +8,8 @@ use crate::ir::dfg::DataFlowGraph;
 use crate::ir::instructions::BranchInfo;
 use crate::ir::stackslot::{StackSlotKind, StackSlots};
 use crate::ir::{
-    Ebb, Function, Inst, InstBuilder, InstructionData, Opcode, StackSlotData, Type, Value, ValueLoc,
+    Block, Function, Inst, InstBuilder, InstructionData, Opcode, StackSlotData, Type, Value,
+    ValueLoc,
 };
 use crate::isa::{RegInfo, RegUnit, TargetIsa};
 use crate::regalloc::RegDiversions;
@@ -20,7 +21,7 @@ use cranelift_entity::{PrimaryMap, SecondaryMap};
 // A description of the redundant-fill-removal algorithm
 //
 //
-// The algorithm works forwards through each Ebb.  It carries along and updates a table,
+// The algorithm works forwards through each Block.  It carries along and updates a table,
 // AvailEnv, with which it tracks registers that are known to have the same value as some stack
 // slot.  The actions on encountering an instruction depend on the instruction, as follows:
 //
@@ -68,19 +69,19 @@ use cranelift_entity::{PrimaryMap, SecondaryMap};
 //
 // The overall algorithm, for a function, starts like this:
 //
-// * (once per function): finds Ebbs that have two or more predecessors, since they will be the
-//   roots of Ebb trees.  Also, the entry node for the function is considered to be a root.
+// * (once per function): finds Blocks that have two or more predecessors, since they will be the
+//   roots of Block trees.  Also, the entry node for the function is considered to be a root.
 //
-// It then continues with a loop that first finds a tree of Ebbs ("discovery") and then removes
+// It then continues with a loop that first finds a tree of Blocks ("discovery") and then removes
 // redundant fills as described above ("processing"):
 //
-// * (discovery; once per tree): for each root, performs a depth first search to find all the Ebbs
+// * (discovery; once per tree): for each root, performs a depth first search to find all the Blocks
 //   in the tree, guided by RedundantReloadRemover::discovery_stack.
 //
 // * (processing; once per tree): the just-discovered tree is then processed as described above,
 //   guided by RedundantReloadRemover::processing_stack.
 //
-// In this way, all Ebbs reachable from the function's entry point are eventually processed.  Note
+// In this way, all Blocks reachable from the function's entry point are eventually processed.  Note
 // that each tree is processed as soon as it has been discovered, so the algorithm never creates a
 // list of trees for the function.
 //
@@ -88,7 +89,7 @@ use cranelift_entity::{PrimaryMap, SecondaryMap};
 // reused for multiple functions so as to minimise heap turnover.  The fields are, roughly:
 //
 //   num_regunits -- constant for the whole function; used by the tree processing phase
-//   num_preds_per_ebb -- constant for the whole function; used by the tree discovery process
+//   num_preds_per_block -- constant for the whole function; used by the tree discovery process
 //
 //   discovery_stack -- used to guide the tree discovery process
 //   nodes_in_tree -- the discovered nodes are recorded here
@@ -121,8 +122,8 @@ use cranelift_entity::{PrimaryMap, SecondaryMap};
 // =============================================================================================
 // Data structures used for discovery of trees
 
-// `ZeroOneOrMany` is used to record the number of predecessors an Ebb block has.  The `Zero` case
-// is included so as to cleanly handle the case where the incoming graph has unreachable Ebbs.
+// `ZeroOneOrMany` is used to record the number of predecessors an Block block has.  The `Zero` case
+// is included so as to cleanly handle the case where the incoming graph has unreachable Blocks.
 
 #[derive(Clone, PartialEq)]
 enum ZeroOneOrMany {
@@ -183,23 +184,23 @@ struct AvailEnv {
 }
 
 // `ProcessingStackElem` combines AvailEnv with contextual information needed to "navigate" within
-// an Ebb.
+// an Block.
 //
-// A ProcessingStackElem conceptually has the lifetime of exactly one Ebb: once the current Ebb is
+// A ProcessingStackElem conceptually has the lifetime of exactly one Block: once the current Block is
 // completed, the ProcessingStackElem will be abandoned.  In practice the top level state,
 // RedundantReloadRemover, caches them, so as to avoid heap turnover.
 //
 // Note that ProcessingStackElem must contain a CursorPosition.  The CursorPosition, which
-// indicates where we are in the current Ebb, cannot be implicitly maintained by looping over all
-// the instructions in an Ebb in turn, because we may choose to suspend processing the current Ebb
+// indicates where we are in the current Block, cannot be implicitly maintained by looping over all
+// the instructions in an Block in turn, because we may choose to suspend processing the current Block
 // at a side exit, continue by processing the subtree reached via the side exit, and only later
-// resume the current Ebb.
+// resume the current Block.
 
 struct ProcessingStackElem {
-    /// Indicates the AvailEnv at the current point in the Ebb.
+    /// Indicates the AvailEnv at the current point in the Block.
     avail_env: AvailEnv,
 
-    /// Shows where we currently are inside the Ebb.
+    /// Shows where we currently are inside the Block.
     cursor: CursorPosition,
 
     /// Indicates the currently active register diversions at the current point.
@@ -212,7 +213,7 @@ struct ProcessingStackElem {
 // `RedundantReloadRemover` contains data structures for the two passes: discovery of tree shaped
 // regions, and processing of them.  These are allocated once and stay alive for the entire
 // function, even though they are cleared out for each new tree shaped region.  It also caches
-// `num_regunits` and `num_preds_per_ebb`, which are computed at the start of each function and
+// `num_regunits` and `num_preds_per_block`, which are computed at the start of each function and
 // then remain constant.
 
 /// The redundant reload remover's state.
@@ -222,22 +223,22 @@ pub struct RedundantReloadRemover {
     /// function.
     num_regunits: Option,
 
-    /// This stores, for each Ebb, a characterisation of the number of predecessors it has.
-    num_preds_per_ebb: PrimaryMap,
+    /// This stores, for each Block, a characterisation of the number of predecessors it has.
+    num_preds_per_block: PrimaryMap,
 
     /// The stack used for the first phase (discovery).  There is one element on the discovery
-    /// stack for each currently unexplored Ebb in the tree being searched.
-    discovery_stack: Vec,
+    /// stack for each currently unexplored Block in the tree being searched.
+    discovery_stack: Vec,
 
     /// The nodes in the discovered tree are inserted here.
-    nodes_in_tree: EntitySet,
+    nodes_in_tree: EntitySet,
 
     /// The stack used during the second phase (transformation).  There is one element on the
     /// processing stack for each currently-open node in the tree being transformed.
     processing_stack: Vec,
 
     /// Used in the second phase to avoid visiting nodes more than once.
-    nodes_already_visited: EntitySet,
+    nodes_already_visited: EntitySet,
 }
 
 // =============================================================================================
@@ -301,17 +302,17 @@ fn slot_of_value<'s>(
 
 impl RedundantReloadRemover {
     // A helper for `add_nodes_to_tree` below.
-    fn discovery_stack_push_successors_of(&mut self, cfg: &ControlFlowGraph, node: Ebb) {
+    fn discovery_stack_push_successors_of(&mut self, cfg: &ControlFlowGraph, node: Block) {
         for successor in cfg.succ_iter(node) {
             self.discovery_stack.push(successor);
         }
     }
 
-    // Visit the tree of Ebbs rooted at `starting_point` and add them to `self.nodes_in_tree`.
-    // `self.num_preds_per_ebb` guides the process, ensuring we don't leave the tree-ish region
+    // Visit the tree of Blocks rooted at `starting_point` and add them to `self.nodes_in_tree`.
+    // `self.num_preds_per_block` guides the process, ensuring we don't leave the tree-ish region
     // and indirectly ensuring that the process will terminate in the presence of cycles in the
     // graph.  `self.discovery_stack` holds the search state in this function.
-    fn add_nodes_to_tree(&mut self, cfg: &ControlFlowGraph, starting_point: Ebb) {
+    fn add_nodes_to_tree(&mut self, cfg: &ControlFlowGraph, starting_point: Block) {
         // One might well ask why this doesn't loop forever when it encounters cycles in the
         // control flow graph.  The reason is that any cycle in the graph that is reachable from
         // anywhere outside the cycle -- in particular, that is reachable from the function's
@@ -325,7 +326,7 @@ impl RedundantReloadRemover {
         self.discovery_stack_push_successors_of(cfg, starting_point);
 
         while let Some(node) = self.discovery_stack.pop() {
-            match self.num_preds_per_ebb[node] {
+            match self.num_preds_per_block[node] {
                 // We arrived at a node with multiple predecessors, so it's a new root.  Ignore it.
                 ZeroOneOrMany::Many => {}
                 // This node has just one predecessor, so we should incorporate it in the tree and
@@ -652,8 +653,8 @@ impl RedundantReloadRemover {
 
 impl RedundantReloadRemover {
     // Push a clone of the top-of-stack ProcessingStackElem.  This will be used to process exactly
-    // one Ebb.  The diversions are created new, rather than cloned, to reflect the fact
-    // that diversions are local to each Ebb.
+    // one Block.  The diversions are created new, rather than cloned, to reflect the fact
+    // that diversions are local to each Block.
     fn processing_stack_push(&mut self, cursor: CursorPosition) {
         let avail_env = if let Some(stack_top) = self.processing_stack.last() {
             stack_top.avail_env.clone()
@@ -674,7 +675,7 @@ impl RedundantReloadRemover {
     // This pushes the node `dst` onto the processing stack, and sets up the new
     // ProcessingStackElem accordingly.  But it does all that only if `dst` is part of the current
     // tree *and* we haven't yet visited it.
-    fn processing_stack_maybe_push(&mut self, dst: Ebb) {
+    fn processing_stack_maybe_push(&mut self, dst: Block) {
         if self.nodes_in_tree.contains(dst) && !self.nodes_already_visited.contains(dst) {
             if !self.processing_stack.is_empty() {
                 // If this isn't the outermost node in the tree (that is, the root), then it must
@@ -682,7 +683,7 @@ impl RedundantReloadRemover {
                 // incorporated in any tree.  Nodes with two or more predecessors are the root of
                 // some other tree, and visiting them as if they were part of the current tree
                 // would be a serious error.
-                debug_assert!(self.num_preds_per_ebb[dst] == ZeroOneOrMany::One);
+                debug_assert!(self.num_preds_per_block[dst] == ZeroOneOrMany::One);
             }
             self.processing_stack_push(CursorPosition::Before(dst));
             self.nodes_already_visited.insert(dst);
@@ -697,7 +698,7 @@ impl RedundantReloadRemover {
         func: &mut Function,
         reginfo: &RegInfo,
         isa: &dyn TargetIsa,
-        root: Ebb,
+        root: Block,
     ) {
         debug_assert!(self.nodes_in_tree.contains(root));
         debug_assert!(self.processing_stack.is_empty());
@@ -728,10 +729,10 @@ impl RedundantReloadRemover {
                 // Update diversions after the insn.
                 self.processing_stack[tos].diversions.apply(&func.dfg[inst]);
 
-                // If the insn can branch outside this Ebb, push work items on the stack for all
-                // target Ebbs that are part of the same tree and that we haven't yet visited.
+                // If the insn can branch outside this Block, push work items on the stack for all
+                // target Blocks that are part of the same tree and that we haven't yet visited.
                 // The next iteration of this instruction-processing loop will immediately start
-                // work on the most recently pushed Ebb, and will eventually continue in this Ebb
+                // work on the most recently pushed Block, and will eventually continue in this Block
                 // when those new items have been removed from the stack.
                 match func.dfg.analyze_branch(inst) {
                     BranchInfo::NotABranch => (),
@@ -748,7 +749,7 @@ impl RedundantReloadRemover {
                     }
                 }
             } else {
-                // We've come to the end of the current work-item (Ebb).  We'll already have
+                // We've come to the end of the current work-item (Block).  We'll already have
                 // processed the fallthrough/continuation/whatever for it using the logic above.
                 // Pop it off the stack and resume work on its parent.
                 self.processing_stack.pop();
@@ -765,11 +766,11 @@ impl RedundantReloadRemover {
     pub fn new() -> Self {
         Self {
             num_regunits: None,
-            num_preds_per_ebb: PrimaryMap::::with_capacity(8),
-            discovery_stack: Vec::::with_capacity(16),
-            nodes_in_tree: EntitySet::::new(),
+            num_preds_per_block: PrimaryMap::::with_capacity(8),
+            discovery_stack: Vec::::with_capacity(16),
+            nodes_in_tree: EntitySet::::new(),
             processing_stack: Vec::::with_capacity(8),
-            nodes_already_visited: EntitySet::::new(),
+            nodes_already_visited: EntitySet::::new(),
         }
     }
 
@@ -779,7 +780,7 @@ impl RedundantReloadRemover {
     }
 
     fn clear_for_new_function(&mut self) {
-        self.num_preds_per_ebb.clear();
+        self.num_preds_per_block.clear();
         self.clear_for_new_tree();
     }
 
@@ -798,19 +799,19 @@ impl RedundantReloadRemover {
         isa: &dyn TargetIsa,
         cfg: &ControlFlowGraph,
     ) {
-        // Fail in an obvious way if there are more than (2^32)-1 Ebbs in this function.
-        let num_ebbs: u32 = func.dfg.num_ebbs().try_into().unwrap();
+        // Fail in an obvious way if there are more than (2^32)-1 Blocks in this function.
+        let num_blocks: u32 = func.dfg.num_blocks().try_into().unwrap();
 
         // Clear out per-tree state.
         self.clear_for_new_function();
 
         // Create a PrimaryMap that summarises the number of predecessors for each block, as 0, 1
         // or "many", and that also claims the entry block as having "many" predecessors.
-        self.num_preds_per_ebb.clear();
-        self.num_preds_per_ebb.reserve(num_ebbs as usize);
+        self.num_preds_per_block.clear();
+        self.num_preds_per_block.reserve(num_blocks as usize);
 
-        for i in 0..num_ebbs {
-            let mut pi = cfg.pred_iter(Ebb::from_u32(i));
+        for i in 0..num_blocks {
+            let mut pi = cfg.pred_iter(Block::from_u32(i));
             let mut n_pi = ZeroOneOrMany::Zero;
             if pi.next().is_some() {
                 n_pi = ZeroOneOrMany::One;
@@ -819,24 +820,24 @@ impl RedundantReloadRemover {
                     // We don't care if there are more than two preds, so stop counting now.
                 }
             }
-            self.num_preds_per_ebb.push(n_pi);
+            self.num_preds_per_block.push(n_pi);
         }
-        debug_assert!(self.num_preds_per_ebb.len() == num_ebbs as usize);
+        debug_assert!(self.num_preds_per_block.len() == num_blocks as usize);
 
         // The entry block must be the root of some tree, so set up the state to reflect that.
-        let entry_ebb = func
+        let entry_block = func
             .layout
             .entry_block()
-            .expect("do_redundant_fill_removal_on_function: entry ebb unknown");
-        debug_assert!(self.num_preds_per_ebb[entry_ebb] == ZeroOneOrMany::Zero);
-        self.num_preds_per_ebb[entry_ebb] = ZeroOneOrMany::Many;
+            .expect("do_redundant_fill_removal_on_function: entry block unknown");
+        debug_assert!(self.num_preds_per_block[entry_block] == ZeroOneOrMany::Zero);
+        self.num_preds_per_block[entry_block] = ZeroOneOrMany::Many;
 
         // Now build and process trees.
-        for root_ix in 0..self.num_preds_per_ebb.len() {
-            let root = Ebb::from_u32(root_ix as u32);
+        for root_ix in 0..self.num_preds_per_block.len() {
+            let root = Block::from_u32(root_ix as u32);
 
             // Build a tree for each node that has two or more preds, and ignore all other nodes.
-            if self.num_preds_per_ebb[root] != ZeroOneOrMany::Many {
+            if self.num_preds_per_block[root] != ZeroOneOrMany::Many {
                 continue;
             }
 
@@ -846,7 +847,7 @@ impl RedundantReloadRemover {
             // Discovery phase: build the tree, as `root` and `self.nodes_in_tree`.
             self.add_nodes_to_tree(cfg, root);
             debug_assert!(self.nodes_in_tree.cardinality() > 0);
-            debug_assert!(self.num_preds_per_ebb[root] == ZeroOneOrMany::Many);
+            debug_assert!(self.num_preds_per_block[root] == ZeroOneOrMany::Many);
 
             // Processing phase: do redundant-reload-removal.
             self.process_tree(func, reginfo, isa, root);
diff --git a/cranelift/codegen/src/regalloc/branch_splitting.rs b/cranelift/codegen/src/regalloc/branch_splitting.rs
index 35291e5213..4e9a159f3e 100644
--- a/cranelift/codegen/src/regalloc/branch_splitting.rs
+++ b/cranelift/codegen/src/regalloc/branch_splitting.rs
@@ -7,7 +7,7 @@ use alloc::vec::Vec;
 use crate::cursor::{Cursor, EncCursor};
 use crate::dominator_tree::DominatorTree;
 use crate::flowgraph::ControlFlowGraph;
-use crate::ir::{Ebb, Function, Inst, InstBuilder, InstructionData, Opcode, ValueList};
+use crate::ir::{Block, Function, Inst, InstBuilder, InstructionData, Opcode, ValueList};
 use crate::isa::TargetIsa;
 use crate::topo_order::TopoOrder;
 
@@ -43,12 +43,12 @@ struct Context<'a> {
 
 impl<'a> Context<'a> {
     fn run(&mut self) {
-        // Any ebb order will do.
-        self.topo.reset(self.cur.func.layout.ebbs());
-        while let Some(ebb) = self.topo.next(&self.cur.func.layout, self.domtree) {
+        // Any block order will do.
+        self.topo.reset(self.cur.func.layout.blocks());
+        while let Some(block) = self.topo.next(&self.cur.func.layout, self.domtree) {
             // Branches can only be at the last or second to last position in an extended basic
             // block.
-            self.cur.goto_last_inst(ebb);
+            self.cur.goto_last_inst(block);
             let terminator_inst = self.cur.current_inst().expect("terminator");
             if let Some(inst) = self.cur.prev_inst() {
                 let opcode = self.cur.func.dfg[inst].opcode();
@@ -80,38 +80,38 @@ impl<'a> Context<'a> {
         // If there are any parameters, split the edge.
         if self.should_split_edge(target) {
             // Create the block the branch will jump to.
-            let new_ebb = self.cur.func.dfg.make_ebb();
+            let new_block = self.cur.func.dfg.make_block();
 
             // Insert the new block before the destination, such that it can fallthrough in the
             // target block.
             assert_ne!(Some(target), self.cur.layout().entry_block());
-            self.cur.layout_mut().insert_ebb(new_ebb, target);
+            self.cur.layout_mut().insert_block(new_block, target);
             self.has_new_blocks = true;
 
-            // Extract the arguments of the branch instruction, split the Ebb parameters and the
+            // Extract the arguments of the branch instruction, split the Block parameters and the
             // branch arguments
             let num_fixed = opcode.constraints().num_fixed_value_arguments();
             let dfg = &mut self.cur.func.dfg;
             let old_args: Vec<_> = {
-                let args = dfg[branch].take_value_list().expect("ebb parameters");
+                let args = dfg[branch].take_value_list().expect("block parameters");
                 args.as_slice(&dfg.value_lists).iter().copied().collect()
             };
-            let (branch_args, ebb_params) = old_args.split_at(num_fixed);
+            let (branch_args, block_params) = old_args.split_at(num_fixed);
 
-            // Replace the branch destination by the new Ebb created with no parameters, and restore
-            // the branch arguments, without the original Ebb parameters.
+            // Replace the branch destination by the new Block created with no parameters, and restore
+            // the branch arguments, without the original Block parameters.
             {
                 let branch_args = ValueList::from_slice(branch_args, &mut dfg.value_lists);
                 let data = &mut dfg[branch];
-                *data.branch_destination_mut().expect("branch") = new_ebb;
+                *data.branch_destination_mut().expect("branch") = new_block;
                 data.put_value_list(branch_args);
             }
             let ok = self.cur.func.update_encoding(branch, self.cur.isa).is_ok();
             debug_assert!(ok);
 
             // Insert a jump to the original target with its arguments into the new block.
-            self.cur.goto_first_insertion_point(new_ebb);
-            self.cur.ins().jump(target, ebb_params);
+            self.cur.goto_first_insertion_point(new_block);
+            self.cur.ins().jump(target, block_params);
 
             // Reset the cursor to point to the branch.
             self.cur.goto_inst(branch);
@@ -122,7 +122,7 @@ impl<'a> Context<'a> {
         let inst_data = &self.cur.func.dfg[inst];
         let opcode = inst_data.opcode();
         if opcode != Opcode::Jump && opcode != Opcode::Fallthrough {
-            // This opcode is ignored as it does not have any EBB parameters.
+            // This opcode is ignored as it does not have any block parameters.
             if opcode != Opcode::IndirectJumpTableBr {
                 debug_assert!(!opcode.is_branch())
             }
@@ -141,23 +141,23 @@ impl<'a> Context<'a> {
         // If there are any parameters, split the edge.
         if self.should_split_edge(*target) {
             // Create the block the branch will jump to.
-            let new_ebb = self.cur.func.dfg.make_ebb();
+            let new_block = self.cur.func.dfg.make_block();
             self.has_new_blocks = true;
 
             // Split the current block before its terminator, and insert a new jump instruction to
             // jump to it.
-            let jump = self.cur.ins().jump(new_ebb, &[]);
-            self.cur.insert_ebb(new_ebb);
+            let jump = self.cur.ins().jump(new_block, &[]);
+            self.cur.insert_block(new_block);
 
-            // Reset the cursor to point to new terminator of the old ebb.
+            // Reset the cursor to point to new terminator of the old block.
             self.cur.goto_inst(jump);
         }
     }
 
     /// Returns whether we should introduce a new branch.
-    fn should_split_edge(&self, target: Ebb) -> bool {
+    fn should_split_edge(&self, target: Block) -> bool {
         // We should split the edge if the target has any parameters.
-        if !self.cur.func.dfg.ebb_params(target).is_empty() {
+        if !self.cur.func.dfg.block_params(target).is_empty() {
             return true;
         };
 
diff --git a/cranelift/codegen/src/regalloc/coalescing.rs b/cranelift/codegen/src/regalloc/coalescing.rs
index c408b912fa..4067a950cf 100644
--- a/cranelift/codegen/src/regalloc/coalescing.rs
+++ b/cranelift/codegen/src/regalloc/coalescing.rs
@@ -2,16 +2,16 @@
 //!
 //! Conventional SSA (CSSA) form is a subset of SSA form where any (transitively) phi-related
 //! values do not interfere. We construct CSSA by building virtual registers that are as large as
-//! possible and inserting copies where necessary such that all argument values passed to an EBB
-//! parameter will belong to the same virtual register as the EBB parameter value itself.
+//! possible and inserting copies where necessary such that all argument values passed to an block
+//! parameter will belong to the same virtual register as the block parameter value itself.
 
 use crate::cursor::{Cursor, EncCursor};
 use crate::dbg::DisplayList;
 use crate::dominator_tree::{DominatorTree, DominatorTreePreorder};
-use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::flowgraph::{BlockPredecessor, ControlFlowGraph};
 use crate::fx::FxHashMap;
 use crate::ir::{self, InstBuilder, ProgramOrder};
-use crate::ir::{Ebb, ExpandedProgramPoint, Function, Inst, Value};
+use crate::ir::{Block, ExpandedProgramPoint, Function, Inst, Value};
 use crate::isa::{EncInfo, TargetIsa};
 use crate::regalloc::affinity::Affinity;
 use crate::regalloc::liveness::Liveness;
@@ -40,8 +40,8 @@ use log::debug;
 //
 // Phase 1: Union-find.
 //
-// We use the union-find support in `VirtRegs` to build virtual registers such that EBB parameter
-// values always belong to the same virtual register as their corresponding EBB arguments at the
+// We use the union-find support in `VirtRegs` to build virtual registers such that block parameter
+// values always belong to the same virtual register as their corresponding block arguments at the
 // predecessor branches. Trivial interferences between parameter and argument value live ranges are
 // detected and resolved before unioning congruence classes, but non-trivial interferences between
 // values that end up in the same congruence class are possible.
@@ -135,8 +135,8 @@ impl Coalescing {
         };
 
         // Run phase 1 (union-find) of the coalescing algorithm on the current function.
-        for &ebb in domtree.cfg_postorder() {
-            context.union_find_ebb(ebb);
+        for &block in domtree.cfg_postorder() {
+            context.union_find_block(block);
         }
         context.finish_union_find();
 
@@ -147,114 +147,114 @@ impl Coalescing {
 
 /// Phase 1: Union-find.
 ///
-/// The two entry points for phase 1 are `union_find_ebb()` and `finish_union_find`.
+/// The two entry points for phase 1 are `union_find_block()` and `finish_union_find`.
 impl<'a> Context<'a> {
-    /// Run the union-find algorithm on the parameter values on `ebb`.
+    /// Run the union-find algorithm on the parameter values on `block`.
     ///
-    /// This ensure that all EBB parameters will belong to the same virtual register as their
+    /// This ensure that all block parameters will belong to the same virtual register as their
     /// corresponding arguments at all predecessor branches.
-    pub fn union_find_ebb(&mut self, ebb: Ebb) {
-        let num_params = self.func.dfg.num_ebb_params(ebb);
+    pub fn union_find_block(&mut self, block: Block) {
+        let num_params = self.func.dfg.num_block_params(block);
         if num_params == 0 {
             return;
         }
 
-        self.isolate_conflicting_params(ebb, num_params);
+        self.isolate_conflicting_params(block, num_params);
 
         for i in 0..num_params {
-            self.union_pred_args(ebb, i);
+            self.union_pred_args(block, i);
         }
     }
 
-    // Identify EBB parameter values that are live at one of the predecessor branches.
+    // Identify block parameter values that are live at one of the predecessor branches.
     //
     // Such a parameter value will conflict with any argument value at the predecessor branch, so
     // it must be isolated by inserting a copy.
-    fn isolate_conflicting_params(&mut self, ebb: Ebb, num_params: usize) {
-        debug_assert_eq!(num_params, self.func.dfg.num_ebb_params(ebb));
-        // The only way a parameter value can interfere with a predecessor branch is if the EBB is
+    fn isolate_conflicting_params(&mut self, block: Block, num_params: usize) {
+        debug_assert_eq!(num_params, self.func.dfg.num_block_params(block));
+        // The only way a parameter value can interfere with a predecessor branch is if the block is
         // dominating the predecessor branch. That is, we are looking for loop back-edges.
-        for BasicBlock {
-            ebb: pred_ebb,
+        for BlockPredecessor {
+            block: pred_block,
             inst: pred_inst,
-        } in self.cfg.pred_iter(ebb)
+        } in self.cfg.pred_iter(block)
         {
-            // The quick pre-order dominance check is accurate because the EBB parameter is defined
-            // at the top of the EBB before any branches.
-            if !self.preorder.dominates(ebb, pred_ebb) {
+            // The quick pre-order dominance check is accurate because the block parameter is defined
+            // at the top of the block before any branches.
+            if !self.preorder.dominates(block, pred_block) {
                 continue;
             }
 
             debug!(
                 " - checking {} params at back-edge {}: {}",
                 num_params,
-                pred_ebb,
+                pred_block,
                 self.func.dfg.display_inst(pred_inst, self.isa)
             );
 
             // Now `pred_inst` is known to be a back-edge, so it is possible for parameter values
             // to be live at the use.
             for i in 0..num_params {
-                let param = self.func.dfg.ebb_params(ebb)[i];
-                if self.liveness[param].reaches_use(pred_inst, pred_ebb, &self.func.layout) {
-                    self.isolate_param(ebb, param);
+                let param = self.func.dfg.block_params(block)[i];
+                if self.liveness[param].reaches_use(pred_inst, pred_block, &self.func.layout) {
+                    self.isolate_param(block, param);
                 }
             }
         }
     }
 
-    // Union EBB parameter value `num` with the corresponding EBB arguments on the predecessor
+    // Union block parameter value `num` with the corresponding block arguments on the predecessor
     // branches.
     //
-    // Detect cases where the argument value is live-in to `ebb` so it conflicts with any EBB
+    // Detect cases where the argument value is live-in to `block` so it conflicts with any block
     // parameter. Isolate the argument in those cases before unioning it with the parameter value.
-    fn union_pred_args(&mut self, ebb: Ebb, argnum: usize) {
-        let param = self.func.dfg.ebb_params(ebb)[argnum];
+    fn union_pred_args(&mut self, block: Block, argnum: usize) {
+        let param = self.func.dfg.block_params(block)[argnum];
 
-        for BasicBlock {
-            ebb: pred_ebb,
+        for BlockPredecessor {
+            block: pred_block,
             inst: pred_inst,
-        } in self.cfg.pred_iter(ebb)
+        } in self.cfg.pred_iter(block)
         {
             let arg = self.func.dfg.inst_variable_args(pred_inst)[argnum];
 
             // Never coalesce incoming function parameters on the stack. These parameters are
             // pre-spilled, and the rest of the virtual register would be forced to spill to the
             // `incoming_arg` stack slot too.
-            if let ir::ValueDef::Param(def_ebb, def_num) = self.func.dfg.value_def(arg) {
-                if Some(def_ebb) == self.func.layout.entry_block()
+            if let ir::ValueDef::Param(def_block, def_num) = self.func.dfg.value_def(arg) {
+                if Some(def_block) == self.func.layout.entry_block()
                     && self.func.signature.params[def_num].location.is_stack()
                 {
                     debug!("-> isolating function stack parameter {}", arg);
-                    let new_arg = self.isolate_arg(pred_ebb, pred_inst, argnum, arg);
+                    let new_arg = self.isolate_arg(pred_block, pred_inst, argnum, arg);
                     self.virtregs.union(param, new_arg);
                     continue;
                 }
             }
 
             // Check for basic interference: If `arg` overlaps a value defined at the entry to
-            // `ebb`, it can never be used as an EBB argument.
+            // `block`, it can never be used as an block argument.
             let interference = {
                 let lr = &self.liveness[arg];
 
-                // There are two ways the argument value can interfere with `ebb`:
+                // There are two ways the argument value can interfere with `block`:
                 //
-                // 1. It is defined in a dominating EBB and live-in to `ebb`.
-                // 2. If is itself a parameter value for `ebb`. This case should already have been
+                // 1. It is defined in a dominating block and live-in to `block`.
+                // 2. If is itself a parameter value for `block`. This case should already have been
                 //    eliminated by `isolate_conflicting_params()`.
                 debug_assert!(
-                    lr.def() != ebb.into(),
+                    lr.def() != block.into(),
                     "{} parameter {} was missed by isolate_conflicting_params()",
-                    ebb,
+                    block,
                     arg
                 );
 
-                // The only other possibility is that `arg` is live-in to `ebb`.
-                lr.is_livein(ebb, &self.func.layout)
+                // The only other possibility is that `arg` is live-in to `block`.
+                lr.is_livein(block, &self.func.layout)
             };
 
             if interference {
-                let new_arg = self.isolate_arg(pred_ebb, pred_inst, argnum, arg);
+                let new_arg = self.isolate_arg(pred_block, pred_inst, argnum, arg);
                 self.virtregs.union(param, new_arg);
             } else {
                 self.virtregs.union(param, arg);
@@ -262,31 +262,31 @@ impl<'a> Context<'a> {
         }
     }
 
-    // Isolate EBB parameter value `param` on `ebb`.
+    // Isolate block parameter value `param` on `block`.
     //
     // When `param=v10`:
     //
-    //     ebb1(v10: i32):
+    //     block1(v10: i32):
     //         foo
     //
     // becomes:
     //
-    //     ebb1(v11: i32):
+    //     block1(v11: i32):
     //         v10 = copy v11
     //         foo
     //
     // This function inserts the copy and updates the live ranges of the old and new parameter
     // values. Returns the new parameter value.
-    fn isolate_param(&mut self, ebb: Ebb, param: Value) -> Value {
+    fn isolate_param(&mut self, block: Block, param: Value) -> Value {
         debug_assert_eq!(
             self.func.dfg.value_def(param).pp(),
-            ExpandedProgramPoint::Ebb(ebb)
+            ExpandedProgramPoint::Block(block)
         );
         let ty = self.func.dfg.value_type(param);
-        let new_val = self.func.dfg.replace_ebb_param(param, ty);
+        let new_val = self.func.dfg.replace_block_param(param, ty);
 
-        // Insert a copy instruction at the top of `ebb`.
-        let mut pos = EncCursor::new(self.func, self.isa).at_first_inst(ebb);
+        // Insert a copy instruction at the top of `block`.
+        let mut pos = EncCursor::new(self.func, self.isa).at_first_inst(block);
         if let Some(inst) = pos.current_inst() {
             pos.use_srcloc(inst);
         }
@@ -297,7 +297,7 @@ impl<'a> Context<'a> {
         debug!(
             "-> inserted {}, following {}({}: {})",
             pos.display_inst(inst),
-            ebb,
+            block,
             new_val,
             ty
         );
@@ -311,27 +311,27 @@ impl<'a> Context<'a> {
                 .expect("Bad copy encoding")
                 .outs[0],
         );
-        self.liveness.create_dead(new_val, ebb, affinity);
+        self.liveness.create_dead(new_val, block, affinity);
         self.liveness
-            .extend_locally(new_val, ebb, inst, &pos.func.layout);
+            .extend_locally(new_val, block, inst, &pos.func.layout);
 
         new_val
     }
 
-    // Isolate the EBB argument `pred_val` from the predecessor `(pred_ebb, pred_inst)`.
+    // Isolate the block argument `pred_val` from the predecessor `(pred_block, pred_inst)`.
     //
-    // It is assumed that `pred_inst` is a branch instruction in `pred_ebb` whose `argnum`'th EBB
-    // argument is `pred_val`. Since the argument value interferes with the corresponding EBB
+    // It is assumed that `pred_inst` is a branch instruction in `pred_block` whose `argnum`'th block
+    // argument is `pred_val`. Since the argument value interferes with the corresponding block
     // parameter at the destination, a copy is used instead:
     //
-    //     brnz v1, ebb2(v10)
+    //     brnz v1, block2(v10)
     //
     // Becomes:
     //
     //     v11 = copy v10
-    //     brnz v1, ebb2(v11)
+    //     brnz v1, block2(v11)
     //
-    // This way the interference with the EBB parameter is avoided.
+    // This way the interference with the block parameter is avoided.
     //
     // A live range for the new value is created while the live range for `pred_val` is left
     // unaltered.
@@ -339,7 +339,7 @@ impl<'a> Context<'a> {
     // The new argument value is returned.
     fn isolate_arg(
         &mut self,
-        pred_ebb: Ebb,
+        pred_block: Block,
         pred_inst: Inst,
         argnum: usize,
         pred_val: Value,
@@ -360,14 +360,14 @@ impl<'a> Context<'a> {
         );
         self.liveness.create_dead(copy, inst, affinity);
         self.liveness
-            .extend_locally(copy, pred_ebb, pred_inst, &pos.func.layout);
+            .extend_locally(copy, pred_block, pred_inst, &pos.func.layout);
 
         pos.func.dfg.inst_variable_args_mut(pred_inst)[argnum] = copy;
 
         debug!(
             "-> inserted {}, before {}: {}",
             pos.display_inst(inst),
-            pred_ebb,
+            pred_block,
             pos.display_inst(pred_inst)
         );
 
@@ -377,7 +377,7 @@ impl<'a> Context<'a> {
     /// Finish the union-find part of the coalescing algorithm.
     ///
     /// This builds the initial set of virtual registers as the transitive/reflexive/symmetric
-    /// closure of the relation formed by EBB parameter-argument pairs found by `union_find_ebb()`.
+    /// closure of the relation formed by block parameter-argument pairs found by `union_find_block()`.
     fn finish_union_find(&mut self) {
         self.virtregs.finish_union_find(None);
         debug!("After union-find phase:{}", self.virtregs);
@@ -430,7 +430,7 @@ impl<'a> Context<'a> {
 
             // Check for interference between `parent` and `value`. Since `parent` dominates
             // `value`, we only have to check if it overlaps the definition.
-            if self.liveness[parent.value].overlaps_def(node.def, node.ebb, &self.func.layout) {
+            if self.liveness[parent.value].overlaps_def(node.def, node.block, &self.func.layout) {
                 // The two values are interfering, so they can't be in the same virtual register.
                 debug!("-> interference: {} overlaps def of {}", parent, value);
                 return false;
@@ -470,9 +470,9 @@ impl<'a> Context<'a> {
         }
     }
 
-    /// Merge EBB parameter value `param` with virtual registers at its predecessors.
+    /// Merge block parameter value `param` with virtual registers at its predecessors.
     fn merge_param(&mut self, param: Value) {
-        let (ebb, argnum) = match self.func.dfg.value_def(param) {
+        let (block, argnum) = match self.func.dfg.value_def(param) {
             ir::ValueDef::Param(e, n) => (e, n),
             ir::ValueDef::Result(_, _) => panic!("Expected parameter"),
         };
@@ -493,12 +493,12 @@ impl<'a> Context<'a> {
         // not loop backedges.
         debug_assert!(self.predecessors.is_empty());
         debug_assert!(self.backedges.is_empty());
-        for BasicBlock {
-            ebb: pred_ebb,
+        for BlockPredecessor {
+            block: pred_block,
             inst: pred_inst,
-        } in self.cfg.pred_iter(ebb)
+        } in self.cfg.pred_iter(block)
         {
-            if self.preorder.dominates(ebb, pred_ebb) {
+            if self.preorder.dominates(block, pred_block) {
                 self.backedges.push(pred_inst);
             } else {
                 self.predecessors.push(pred_inst);
@@ -522,8 +522,8 @@ impl<'a> Context<'a> {
             }
 
             // Can't merge because of interference. Insert a copy instead.
-            let pred_ebb = self.func.layout.pp_ebb(pred_inst);
-            let new_arg = self.isolate_arg(pred_ebb, pred_inst, argnum, arg);
+            let pred_block = self.func.layout.pp_block(pred_inst);
+            let new_arg = self.isolate_arg(pred_block, pred_inst, argnum, arg);
             self.virtregs
                 .insert_single(param, new_arg, self.func, self.preorder);
         }
@@ -616,12 +616,12 @@ impl<'a> Context<'a> {
                 // Check if the parent value interferes with the virtual copy.
                 let inst = node.def.unwrap_inst();
                 if node.set_id != parent.set_id
-                    && self.liveness[parent.value].reaches_use(inst, node.ebb, &self.func.layout)
+                    && self.liveness[parent.value].reaches_use(inst, node.block, &self.func.layout)
                 {
                     debug!(
                         " - interference: {} overlaps vcopy at {}:{}",
                         parent,
-                        node.ebb,
+                        node.block,
                         self.func.dfg.display_inst(inst, self.isa)
                     );
                     return false;
@@ -640,7 +640,7 @@ impl<'a> Context<'a> {
             // Both node and parent are values, so check for interference.
             debug_assert!(node.is_value() && parent.is_value());
             if node.set_id != parent.set_id
-                && self.liveness[parent.value].overlaps_def(node.def, node.ebb, &self.func.layout)
+                && self.liveness[parent.value].overlaps_def(node.def, node.block, &self.func.layout)
             {
                 // The two values are interfering.
                 debug!(" - interference: {} overlaps def of {}", parent, node.value);
@@ -663,7 +663,7 @@ impl<'a> Context<'a> {
 ///
 /// The idea of a dominator forest was introduced on the Budimlic paper and the linear stack
 /// representation in the Boissinot paper. Our version of the linear stack is slightly modified
-/// because we have a pre-order of the dominator tree at the EBB granularity, not basic block
+/// because we have a pre-order of the dominator tree at the block granularity, not basic block
 /// granularity.
 ///
 /// Values are pushed in dominator tree pre-order of their definitions, and for each value pushed,
@@ -673,7 +673,7 @@ struct DomForest {
     // Stack representing the rightmost edge of the dominator forest so far, ending in the last
     // element of `values`.
     //
-    // At all times, the EBB of each element in the stack dominates the EBB of the next one.
+    // At all times, the block of each element in the stack dominates the block of the next one.
     stack: Vec,
 }
 
@@ -683,8 +683,8 @@ struct DomForest {
 struct Node {
     /// The program point where the live range is defined.
     def: ExpandedProgramPoint,
-    /// EBB containing `def`.
-    ebb: Ebb,
+    /// block containing `def`.
+    block: Block,
     /// Is this a virtual copy or a value?
     is_vcopy: bool,
     /// Set identifier.
@@ -698,10 +698,10 @@ impl Node {
     /// Create a node representing `value`.
     pub fn value(value: Value, set_id: u8, func: &Function) -> Self {
         let def = func.dfg.value_def(value).pp();
-        let ebb = func.layout.pp_ebb(def);
+        let block = func.layout.pp_block(def);
         Self {
             def,
-            ebb,
+            block,
             is_vcopy: false,
             set_id,
             value,
@@ -711,10 +711,10 @@ impl Node {
     /// Create a node representing a virtual copy.
     pub fn vcopy(branch: Inst, value: Value, set_id: u8, func: &Function) -> Self {
         let def = branch.into();
-        let ebb = func.layout.pp_ebb(def);
+        let block = func.layout.pp_block(def);
         Self {
             def,
-            ebb,
+            block,
             is_vcopy: true,
             set_id,
             value,
@@ -730,9 +730,9 @@ impl Node {
 impl fmt::Display for Node {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         if self.is_vcopy {
-            write!(f, "{}:vcopy({})@{}", self.set_id, self.value, self.ebb)
+            write!(f, "{}:vcopy({})@{}", self.set_id, self.value, self.block)
         } else {
-            write!(f, "{}:{}@{}", self.set_id, self.value, self.ebb)
+            write!(f, "{}:{}@{}", self.set_id, self.value, self.block)
         }
     }
 }
@@ -760,16 +760,16 @@ impl DomForest {
         preorder: &DominatorTreePreorder,
     ) -> Option {
         // The stack contains the current sequence of dominating defs. Pop elements until we
-        // find one whose EBB dominates `node.ebb`.
+        // find one whose block dominates `node.block`.
         while let Some(top) = self.stack.pop() {
-            if preorder.dominates(top.ebb, node.ebb) {
+            if preorder.dominates(top.block, node.block) {
                 // This is the right insertion spot for `node`.
                 self.stack.push(top);
                 self.stack.push(node);
 
-                // We know here that `top.ebb` dominates `node.ebb`, and thus `node.def`. This does
+                // We know here that `top.block` dominates `node.block`, and thus `node.def`. This does
                 // not necessarily mean that `top.def` dominates `node.def`, though. The `top.def`
-                // program point may be below the last branch in `top.ebb` that dominates
+                // program point may be below the last branch in `top.block` that dominates
                 // `node.def`.
                 //
                 // We do know, though, that if there is a nearest value dominating `node.def`, it
@@ -777,16 +777,16 @@ impl DomForest {
                 // dominates.
                 let mut last_dom = node.def;
                 for &n in self.stack.iter().rev().skip(1) {
-                    // If the node is defined at the EBB header, it does in fact dominate
+                    // If the node is defined at the block header, it does in fact dominate
                     // everything else pushed on the stack.
                     let def_inst = match n.def {
-                        ExpandedProgramPoint::Ebb(_) => return Some(n),
+                        ExpandedProgramPoint::Block(_) => return Some(n),
                         ExpandedProgramPoint::Inst(i) => i,
                     };
 
-                    // We need to find the last program point in `n.ebb` to dominate `node.def`.
-                    last_dom = match domtree.last_dominator(n.ebb, last_dom, &func.layout) {
-                        None => n.ebb.into(),
+                    // We need to find the last program point in `n.block` to dominate `node.def`.
+                    last_dom = match domtree.last_dominator(n.block, last_dom, &func.layout) {
+                        None => n.block.into(),
                         Some(inst) => {
                             if func.layout.cmp(def_inst, inst) != cmp::Ordering::Greater {
                                 return Some(n);
@@ -816,18 +816,18 @@ impl DomForest {
 /// When building a full virtual register at once, like phase 1 does with union-find, it is good
 /// enough to check for interference between the values in the full virtual register like
 /// `check_vreg()` does. However, in phase 2 we are doing pairwise merges of partial virtual
-/// registers that don't represent the full transitive closure of the EBB argument-parameter
+/// registers that don't represent the full transitive closure of the block argument-parameter
 /// relation. This means that just checking for interference between values is inadequate.
 ///
 /// Example:
 ///
 ///   v1 = iconst.i32 1
-///   brnz v10, ebb1(v1)
+///   brnz v10, block1(v1)
 ///   v2 = iconst.i32 2
-///   brnz v11, ebb1(v2)
+///   brnz v11, block1(v2)
 ///   return v1
 ///
-/// ebb1(v3: i32):
+/// block1(v3: i32):
 ///   v4 = iadd v3, v1
 ///
 /// With just value interference checking, we could build the virtual register [v3, v1] since those
@@ -835,13 +835,13 @@ impl DomForest {
 /// interfere. However, we can't resolve that interference either by inserting a copy:
 ///
 ///   v1 = iconst.i32 1
-///   brnz v10, ebb1(v1)
+///   brnz v10, block1(v1)
 ///   v2 = iconst.i32 2
 ///   v20 = copy v2          <-- new value
-///   brnz v11, ebb1(v20)
+///   brnz v11, block1(v20)
 ///   return v1
 ///
-/// ebb1(v3: i32):
+/// block1(v3: i32):
 ///   v4 = iadd v3, v1
 ///
 /// The new value v20 still interferes with v1 because v1 is live across the "brnz v11" branch. We
@@ -851,32 +851,32 @@ impl DomForest {
 /// instructions, then attempting to delete the copies. This is quite expensive because it involves
 /// creating a large number of copies and value.
 ///
-/// We'll detect this form of interference with *virtual copies*: Each EBB parameter value that
-/// hasn't yet been fully merged with its EBB argument values is given a set of virtual copies at
+/// We'll detect this form of interference with *virtual copies*: Each block parameter value that
+/// hasn't yet been fully merged with its block argument values is given a set of virtual copies at
 /// the predecessors. Any candidate value to be merged is checked for interference against both the
 /// virtual register and the virtual copies.
 ///
 /// In the general case, we're checking if two virtual registers can be merged, and both can
-/// contain incomplete EBB parameter values with associated virtual copies.
+/// contain incomplete block parameter values with associated virtual copies.
 ///
 /// The `VirtualCopies` struct represents a set of incomplete parameters and their associated
 /// virtual copies. Given two virtual registers, it can produce an ordered sequence of nodes
 /// representing the virtual copies in both vregs.
 struct VirtualCopies {
-    // Incomplete EBB parameters. These don't need to belong to the same virtual register.
+    // Incomplete block parameters. These don't need to belong to the same virtual register.
     params: Vec,
 
-    // Set of `(branch, destination)` pairs. These are all the predecessor branches for the EBBs
+    // Set of `(branch, destination)` pairs. These are all the predecessor branches for the blocks
     // whose parameters can be found in `params`.
     //
     // Ordered by dominator tree pre-order of the branch instructions.
-    branches: Vec<(Inst, Ebb)>,
+    branches: Vec<(Inst, Block)>,
 
     // Filter for the currently active node iterator.
     //
-    // An ebb => (set_id, num) entry means that branches to `ebb` are active in `set_id` with
+    // An block => (set_id, num) entry means that branches to `block` are active in `set_id` with
     // branch argument number `num`.
-    filter: FxHashMap,
+    filter: FxHashMap,
 }
 
 impl VirtualCopies {
@@ -901,7 +901,7 @@ impl VirtualCopies {
     ///
     /// The values are assumed to be in domtree pre-order.
     ///
-    /// This will extract the EBB parameter values and associate virtual copies all of them.
+    /// This will extract the block parameter values and associate virtual copies all of them.
     pub fn initialize(
         &mut self,
         values: &[Value],
@@ -911,29 +911,29 @@ impl VirtualCopies {
     ) {
         self.clear();
 
-        let mut last_ebb = None;
+        let mut last_block = None;
         for &val in values {
-            if let ir::ValueDef::Param(ebb, _) = func.dfg.value_def(val) {
+            if let ir::ValueDef::Param(block, _) = func.dfg.value_def(val) {
                 self.params.push(val);
 
-                // We may have multiple parameters from the same EBB, but we only need to collect
+                // We may have multiple parameters from the same block, but we only need to collect
                 // predecessors once. Also verify the ordering of values.
-                if let Some(last) = last_ebb {
-                    match preorder.pre_cmp_ebb(last, ebb) {
+                if let Some(last) = last_block {
+                    match preorder.pre_cmp_block(last, block) {
                         cmp::Ordering::Less => {}
                         cmp::Ordering::Equal => continue,
                         cmp::Ordering::Greater => panic!("values in wrong order"),
                     }
                 }
 
-                // This EBB hasn't been seen before.
-                for BasicBlock {
+                // This block hasn't been seen before.
+                for BlockPredecessor {
                     inst: pred_inst, ..
-                } in cfg.pred_iter(ebb)
+                } in cfg.pred_iter(block)
                 {
-                    self.branches.push((pred_inst, ebb));
+                    self.branches.push((pred_inst, block));
                 }
-                last_ebb = Some(ebb);
+                last_block = Some(block);
             }
         }
 
@@ -953,7 +953,7 @@ impl VirtualCopies {
         debug_assert_eq!(popped, Some(param));
 
         // The domtree pre-order in `self.params` guarantees that all parameters defined at the
-        // same EBB will be adjacent. This means we can see when all parameters at an EBB have been
+        // same block will be adjacent. This means we can see when all parameters at an block have been
         // merged.
         //
         // We don't care about the last parameter - when that is merged we are done.
@@ -961,16 +961,16 @@ impl VirtualCopies {
             None => return,
             Some(x) => *x,
         };
-        let ebb = func.dfg.value_def(param).unwrap_ebb();
-        if func.dfg.value_def(last).unwrap_ebb() == ebb {
-            // We're not done with `ebb` parameters yet.
+        let block = func.dfg.value_def(param).unwrap_block();
+        if func.dfg.value_def(last).unwrap_block() == block {
+            // We're not done with `block` parameters yet.
             return;
         }
 
-        // Alright, we know there are no remaining `ebb` parameters in `self.params`. This means we
-        // can get rid of the `ebb` predecessors in `self.branches`. We don't have to, the
+        // Alright, we know there are no remaining `block` parameters in `self.params`. This means we
+        // can get rid of the `block` predecessors in `self.branches`. We don't have to, the
         // `VCopyIter` will just skip them, but this reduces its workload.
-        self.branches.retain(|&(_, dest)| dest != ebb);
+        self.branches.retain(|&(_, dest)| dest != block);
     }
 
     /// Set a filter for the virtual copy nodes we're generating.
@@ -991,28 +991,28 @@ impl VirtualCopies {
         // removed from the back once they are fully merged. This means we can stop looking for
         // parameters once we're beyond the last one.
         let last_param = *self.params.last().expect("No more parameters");
-        let limit = func.dfg.value_def(last_param).unwrap_ebb();
+        let limit = func.dfg.value_def(last_param).unwrap_block();
 
         for (set_id, repr) in reprs.iter().enumerate() {
             let set_id = set_id as u8;
             for &value in virtregs.congruence_class(repr) {
-                if let ir::ValueDef::Param(ebb, num) = func.dfg.value_def(value) {
-                    if preorder.pre_cmp_ebb(ebb, limit) == cmp::Ordering::Greater {
+                if let ir::ValueDef::Param(block, num) = func.dfg.value_def(value) {
+                    if preorder.pre_cmp_block(block, limit) == cmp::Ordering::Greater {
                         // Stop once we're outside the bounds of `self.params`.
                         break;
                     }
-                    self.filter.insert(ebb, (set_id, num));
+                    self.filter.insert(block, (set_id, num));
                 }
             }
         }
     }
 
-    /// Look up the set_id and argument number for `ebb` in the current filter.
+    /// Look up the set_id and argument number for `block` in the current filter.
     ///
-    /// Returns `None` if none of the currently active parameters are defined at `ebb`. Otherwise
-    /// returns `(set_id, argnum)` for an active parameter defined at `ebb`.
-    fn lookup(&self, ebb: Ebb) -> Option<(u8, usize)> {
-        self.filter.get(&ebb).cloned()
+    /// Returns `None` if none of the currently active parameters are defined at `block`. Otherwise
+    /// returns `(set_id, argnum)` for an active parameter defined at `block`.
+    fn lookup(&self, block: Block) -> Option<(u8, usize)> {
+        self.filter.get(&block).cloned()
     }
 
     /// Get an iterator of dom-forest nodes corresponding to the current filter.
@@ -1032,7 +1032,7 @@ impl VirtualCopies {
 struct VCopyIter<'a> {
     func: &'a Function,
     vcopies: &'a VirtualCopies,
-    branches: slice::Iter<'a, (Inst, Ebb)>,
+    branches: slice::Iter<'a, (Inst, Block)>,
 }
 
 impl<'a> Iterator for VCopyIter<'a> {
@@ -1090,7 +1090,7 @@ where
             (Some(a), Some(b)) => {
                 let layout = self.layout;
                 self.preorder
-                    .pre_cmp_ebb(a.ebb, b.ebb)
+                    .pre_cmp_block(a.block, b.block)
                     .then_with(|| layout.cmp(a.def, b.def))
             }
             (Some(_), None) => cmp::Ordering::Less,
diff --git a/cranelift/codegen/src/regalloc/coloring.rs b/cranelift/codegen/src/regalloc/coloring.rs
index 347aec9ade..eb3cb513c8 100644
--- a/cranelift/codegen/src/regalloc/coloring.rs
+++ b/cranelift/codegen/src/regalloc/coloring.rs
@@ -24,8 +24,8 @@
 //!    a register.
 //!
 //! 5. The code must be in Conventional SSA form. Among other things, this means that values passed
-//!    as arguments when branching to an EBB must belong to the same virtual register as the
-//!    corresponding EBB argument value.
+//!    as arguments when branching to an block must belong to the same virtual register as the
+//!    corresponding block argument value.
 //!
 //! # Iteration order
 //!
@@ -35,10 +35,10 @@
 //! defined by the instruction and only consider the colors of other values that are live at the
 //! instruction.
 //!
-//! The first time we see a branch to an EBB, the EBB's argument values are colored to match the
+//! The first time we see a branch to an block, the block's argument values are colored to match the
 //! registers currently holding branch argument values passed to the predecessor branch. By
-//! visiting EBBs in a CFG topological order, we guarantee that at least one predecessor branch has
-//! been visited before the destination EBB. Therefore, the EBB's arguments are already colored.
+//! visiting blocks in a CFG topological order, we guarantee that at least one predecessor branch has
+//! been visited before the destination block. Therefore, the block's arguments are already colored.
 //!
 //! The exception is the entry block whose arguments are colored from the ABI requirements.
 
@@ -46,7 +46,7 @@ use crate::cursor::{Cursor, EncCursor};
 use crate::dominator_tree::DominatorTree;
 use crate::flowgraph::ControlFlowGraph;
 use crate::ir::{ArgumentLoc, InstBuilder, ValueDef};
-use crate::ir::{Ebb, Function, Inst, InstructionData, Layout, Opcode, SigRef, Value, ValueLoc};
+use crate::ir::{Block, Function, Inst, InstructionData, Layout, Opcode, SigRef, Value, ValueLoc};
 use crate::isa::{regs_overlap, RegClass, RegInfo, RegUnit};
 use crate::isa::{ConstraintKind, EncInfo, OperandConstraint, RecipeConstraints, TargetIsa};
 use crate::packed_option::PackedOption;
@@ -168,20 +168,20 @@ impl<'a> Context<'a> {
             .resize(self.cur.func.dfg.num_values());
 
         // Visit blocks in reverse post-order. We need to ensure that at least one predecessor has
-        // been visited before each EBB. That guarantees that the EBB arguments have been colored.
-        for &ebb in self.domtree.cfg_postorder().iter().rev() {
-            self.visit_ebb(ebb, tracker);
+        // been visited before each block. That guarantees that the block arguments have been colored.
+        for &block in self.domtree.cfg_postorder().iter().rev() {
+            self.visit_block(block, tracker);
         }
     }
 
-    /// Visit `ebb`, assuming that the immediate dominator has already been visited.
-    fn visit_ebb(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) {
-        debug!("Coloring {}:", ebb);
-        let mut regs = self.visit_ebb_header(ebb, tracker);
+    /// Visit `block`, assuming that the immediate dominator has already been visited.
+    fn visit_block(&mut self, block: Block, tracker: &mut LiveValueTracker) {
+        debug!("Coloring {}:", block);
+        let mut regs = self.visit_block_header(block, tracker);
         tracker.drop_dead_params();
 
-        // Now go through the instructions in `ebb` and color the values they define.
-        self.cur.goto_top(ebb);
+        // Now go through the instructions in `block` and color the values they define.
+        self.cur.goto_top(block);
         while let Some(inst) = self.cur.next_inst() {
             self.cur.use_srcloc(inst);
             let opcode = self.cur.func.dfg[inst].opcode();
@@ -204,7 +204,7 @@ impl<'a> Context<'a> {
             tracker.drop_dead(inst);
 
             // We are not able to insert any regmove for diversion or un-diversion after the first
-            // branch. Instead, we record the diversion to be restored at the entry of the next EBB,
+            // branch. Instead, we record the diversion to be restored at the entry of the next block,
             // which should have a single predecessor.
             if opcode.is_branch() {
                 // The next instruction is necessarily an unconditional branch.
@@ -221,15 +221,15 @@ impl<'a> Context<'a> {
                             "unexpected instruction {} after a conditional branch",
                             self.cur.display_inst(branch)
                         ),
-                        SingleDest(ebb, _) => ebb,
+                        SingleDest(block, _) => block,
                     };
 
-                    // We have a single branch with a single target, and an EBB with a single
-                    // predecessor. Thus we can forward the diversion set to the next EBB.
+                    // We have a single branch with a single target, and an block with a single
+                    // predecessor. Thus we can forward the diversion set to the next block.
                     if self.cfg.pred_iter(target).count() == 1 {
-                        // Transfer the diversion to the next EBB.
+                        // Transfer the diversion to the next block.
                         self.divert
-                            .save_for_ebb(&mut self.cur.func.entry_diversions, target);
+                            .save_for_block(&mut self.cur.func.entry_diversions, target);
                         debug!(
                             "Set entry-diversion for {} to\n      {}",
                             target,
@@ -253,13 +253,17 @@ impl<'a> Context<'a> {
         }
     }
 
-    /// Visit the `ebb` header.
+    /// Visit the `block` header.
     ///
-    /// Initialize the set of live registers and color the arguments to `ebb`.
-    fn visit_ebb_header(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) -> AvailableRegs {
-        // Reposition the live value tracker and deal with the EBB arguments.
-        tracker.ebb_top(
-            ebb,
+    /// Initialize the set of live registers and color the arguments to `block`.
+    fn visit_block_header(
+        &mut self,
+        block: Block,
+        tracker: &mut LiveValueTracker,
+    ) -> AvailableRegs {
+        // Reposition the live value tracker and deal with the block arguments.
+        tracker.block_top(
+            block,
             &self.cur.func.dfg,
             self.liveness,
             &self.cur.func.layout,
@@ -268,18 +272,18 @@ impl<'a> Context<'a> {
 
         // Copy the content of the registered diversions to be reused at the
         // entry of this basic block.
-        self.divert.at_ebb(&self.cur.func.entry_diversions, ebb);
+        self.divert.at_block(&self.cur.func.entry_diversions, block);
         debug!(
             "Start {} with entry-diversion set to\n      {}",
-            ebb,
+            block,
             self.divert.display(&self.reginfo)
         );
 
-        if self.cur.func.layout.entry_block() == Some(ebb) {
+        if self.cur.func.layout.entry_block() == Some(block) {
             // Parameters on the entry block have ABI constraints.
             self.color_entry_params(tracker.live())
         } else {
-            // The live-ins and parameters of a non-entry EBB have already been assigned a register.
+            // The live-ins and parameters of a non-entry block have already been assigned a register.
             // Reconstruct the allocatable set.
             self.livein_regs(tracker.live())
         }
@@ -288,7 +292,7 @@ impl<'a> Context<'a> {
     /// Initialize a set of allocatable registers from the values that are live-in to a block.
     /// These values must already be colored when the dominating blocks were processed.
     ///
-    /// Also process the EBB arguments which were colored when the first predecessor branch was
+    /// Also process the block arguments which were colored when the first predecessor branch was
     /// encountered.
     fn livein_regs(&self, live: &[LiveValue]) -> AvailableRegs {
         // Start from the registers that are actually usable. We don't want to include any reserved
@@ -428,7 +432,7 @@ impl<'a> Context<'a> {
             regs.input.display(&self.reginfo),
         );
 
-        // EBB whose arguments should be colored to match the current branch instruction's
+        // block whose arguments should be colored to match the current branch instruction's
         // arguments.
         let mut color_dest_args = None;
 
@@ -446,10 +450,10 @@ impl<'a> Context<'a> {
             self.program_input_abi(inst, AbiParams::Returns);
         } else if self.cur.func.dfg[inst].opcode().is_branch() {
             // This is a branch, so we need to make sure that globally live values are in their
-            // global registers. For EBBs that take arguments, we also need to place the argument
+            // global registers. For blocks that take arguments, we also need to place the argument
             // values in the expected registers.
             if let Some(dest) = self.cur.func.dfg[inst].branch_destination() {
-                if self.program_ebb_arguments(inst, dest) {
+                if self.program_block_arguments(inst, dest) {
                     color_dest_args = Some(dest);
                 }
             } else {
@@ -458,7 +462,7 @@ impl<'a> Context<'a> {
                 debug_assert_eq!(
                     self.cur.func.dfg.inst_variable_args(inst).len(),
                     0,
-                    "Can't handle EBB arguments: {}",
+                    "Can't handle block arguments: {}",
                     self.cur.display_inst(inst)
                 );
                 self.undivert_regs(|lr, _| !lr.is_local());
@@ -576,7 +580,7 @@ impl<'a> Context<'a> {
         // If this is the first time we branch to `dest`, color its arguments to match the current
         // register state.
         if let Some(dest) = color_dest_args {
-            self.color_ebb_params(inst, dest);
+            self.color_block_params(inst, dest);
         }
 
         // Apply the solution to the defs.
@@ -727,7 +731,7 @@ impl<'a> Context<'a> {
                         // This code runs after calling `solver.inputs_done()` so we must identify
                         // the new variable as killed or live-through.
                         let layout = &self.cur.func.layout;
-                        if self.liveness[arg_val].killed_at(inst, layout.pp_ebb(inst), layout) {
+                        if self.liveness[arg_val].killed_at(inst, layout.pp_block(inst), layout) {
                             self.solver
                                 .add_killed_var(arg_val, constraint.regclass, cur_reg);
                         } else {
@@ -747,12 +751,12 @@ impl<'a> Context<'a> {
     ///
     /// 1. Any values that are live-in to `dest` must be un-diverted so they live in their globally
     ///    assigned register.
-    /// 2. If the `dest` EBB takes arguments, reassign the branch argument values to the matching
+    /// 2. If the `dest` block takes arguments, reassign the branch argument values to the matching
     ///    registers.
     ///
     /// Returns true if this is the first time a branch to `dest` is seen, so the `dest` argument
     /// values should be colored after `shuffle_inputs`.
-    fn program_ebb_arguments(&mut self, inst: Inst, dest: Ebb) -> bool {
+    fn program_block_arguments(&mut self, inst: Inst, dest: Block) -> bool {
         // Find diverted registers that are live-in to `dest` and reassign them to their global
         // home.
         //
@@ -760,9 +764,9 @@ impl<'a> Context<'a> {
         // arguments, so they can't always be un-diverted.
         self.undivert_regs(|lr, layout| lr.is_livein(dest, layout));
 
-        // Now handle the EBB arguments.
+        // Now handle the block arguments.
         let br_args = self.cur.func.dfg.inst_variable_args(inst);
-        let dest_args = self.cur.func.dfg.ebb_params(dest);
+        let dest_args = self.cur.func.dfg.block_params(dest);
         debug_assert_eq!(br_args.len(), dest_args.len());
         for (&dest_arg, &br_arg) in dest_args.iter().zip(br_args) {
             // The first time we encounter a branch to `dest`, we get to pick the location. The
@@ -771,7 +775,7 @@ impl<'a> Context<'a> {
                 ValueLoc::Unassigned => {
                     // This is the first branch to `dest`, so we should color `dest_arg` instead of
                     // `br_arg`. However, we don't know where `br_arg` will end up until
-                    // after `shuffle_inputs`. See `color_ebb_params` below.
+                    // after `shuffle_inputs`. See `color_block_params` below.
                     //
                     // It is possible for `dest_arg` to have no affinity, and then it should simply
                     // be ignored.
@@ -804,10 +808,10 @@ impl<'a> Context<'a> {
     /// Knowing that we've never seen a branch to `dest` before, color its parameters to match our
     /// register state.
     ///
-    /// This function is only called when `program_ebb_arguments()` returned `true`.
-    fn color_ebb_params(&mut self, inst: Inst, dest: Ebb) {
+    /// This function is only called when `program_block_arguments()` returned `true`.
+    fn color_block_params(&mut self, inst: Inst, dest: Block) {
         let br_args = self.cur.func.dfg.inst_variable_args(inst);
-        let dest_args = self.cur.func.dfg.ebb_params(dest);
+        let dest_args = self.cur.func.dfg.block_params(dest);
         debug_assert_eq!(br_args.len(), dest_args.len());
         for (&dest_arg, &br_arg) in dest_args.iter().zip(br_args) {
             match self.cur.func.locations[dest_arg] {
@@ -818,7 +822,7 @@ impl<'a> Context<'a> {
                     }
                 }
                 ValueLoc::Reg(_) => panic!("{} arg {} already colored", dest, dest_arg),
-                // Spilled value consistency is verified by `program_ebb_arguments()` above.
+                // Spilled value consistency is verified by `program_block_arguments()` above.
                 ValueLoc::Stack(_) => {}
             }
         }
@@ -1082,7 +1086,7 @@ impl<'a> Context<'a> {
     /// Determine if `value` is live on a CFG edge from the current instruction.
     ///
     /// This means that the current instruction is a branch and `value` is live in to one of the
-    /// branch destinations. Branch arguments and EBB parameters are not considered live on the
+    /// branch destinations. Branch arguments and block parameters are not considered live on the
     /// edge.
     fn is_live_on_outgoing_edge(&self, value: Value) -> bool {
         use crate::ir::instructions::BranchInfo::*;
@@ -1091,17 +1095,17 @@ impl<'a> Context<'a> {
         let layout = &self.cur.func.layout;
         match self.cur.func.dfg.analyze_branch(inst) {
             NotABranch => false,
-            SingleDest(ebb, _) => {
+            SingleDest(block, _) => {
                 let lr = &self.liveness[value];
-                lr.is_livein(ebb, layout)
+                lr.is_livein(block, layout)
             }
-            Table(jt, ebb) => {
+            Table(jt, block) => {
                 let lr = &self.liveness[value];
                 !lr.is_local()
-                    && (ebb.map_or(false, |ebb| lr.is_livein(ebb, layout))
+                    && (block.map_or(false, |block| lr.is_livein(block, layout))
                         || self.cur.func.jump_tables[jt]
                             .iter()
-                            .any(|ebb| lr.is_livein(*ebb, layout)))
+                            .any(|block| lr.is_livein(*block, layout)))
             }
         }
     }
@@ -1232,7 +1236,7 @@ impl<'a> Context<'a> {
             self.liveness.create_dead(local, inst, lv.affinity);
             self.liveness.extend_locally(
                 local,
-                self.cur.func.layout.pp_ebb(inst),
+                self.cur.func.layout.pp_block(inst),
                 copy,
                 &self.cur.func.layout,
             );
diff --git a/cranelift/codegen/src/regalloc/diversion.rs b/cranelift/codegen/src/regalloc/diversion.rs
index feb9c6f0ef..e3bacbae72 100644
--- a/cranelift/codegen/src/regalloc/diversion.rs
+++ b/cranelift/codegen/src/regalloc/diversion.rs
@@ -4,12 +4,12 @@
 //! Sometimes, it is necessary to move register values to a different register in order to satisfy
 //! instruction constraints.
 //!
-//! These register diversions are local to an EBB. No values can be diverted when entering a new
-//! EBB.
+//! These register diversions are local to an block. No values can be diverted when entering a new
+//! block.
 
 use crate::fx::FxHashMap;
 use crate::hash_map::{Entry, Iter};
-use crate::ir::{Ebb, StackSlot, Value, ValueLoc, ValueLocations};
+use crate::ir::{Block, StackSlot, Value, ValueLoc, ValueLocations};
 use crate::ir::{InstructionData, Opcode};
 use crate::isa::{RegInfo, RegUnit};
 use core::fmt;
@@ -38,22 +38,22 @@ impl Diversion {
     }
 }
 
-/// Keep track of diversions in an EBB.
+/// Keep track of diversions in an block.
 #[derive(Clone)]
 pub struct RegDiversions {
     current: FxHashMap,
 }
 
-/// Keep track of diversions at the entry of EBB.
+/// Keep track of diversions at the entry of block.
 #[derive(Clone)]
 struct EntryRegDiversionsValue {
-    key: Ebb,
+    key: Block,
     divert: RegDiversions,
 }
 
-/// Map EBB to their matching RegDiversions at basic blocks entry.
+/// Map block to their matching RegDiversions at basic blocks entry.
 pub struct EntryRegDiversions {
-    map: SparseMap,
+    map: SparseMap,
 }
 
 impl RegDiversions {
@@ -178,22 +178,22 @@ impl RegDiversions {
     }
 
     /// Resets the state of the current diversions to the recorded diversions at the entry of the
-    /// given `ebb`. The recoded diversions is available after coloring on `func.entry_diversions`
+    /// given `block`. The recoded diversions is available after coloring on `func.entry_diversions`
     /// field.
-    pub fn at_ebb(&mut self, entry_diversions: &EntryRegDiversions, ebb: Ebb) {
+    pub fn at_block(&mut self, entry_diversions: &EntryRegDiversions, block: Block) {
         self.clear();
-        if let Some(entry_divert) = entry_diversions.map.get(ebb) {
+        if let Some(entry_divert) = entry_diversions.map.get(block) {
             let iter = entry_divert.divert.current.iter();
             self.current.extend(iter);
         }
     }
 
-    /// Copy the current state of the diversions, and save it for the entry of the `ebb` given as
+    /// Copy the current state of the diversions, and save it for the entry of the `block` given as
     /// argument.
     ///
-    /// Note: This function can only be called once on an `ebb` with a given `entry_diversions`
+    /// Note: This function can only be called once on a `Block` with a given `entry_diversions`
     /// argument, otherwise it would panic.
-    pub fn save_for_ebb(&mut self, entry_diversions: &mut EntryRegDiversions, target: Ebb) {
+    pub fn save_for_block(&mut self, entry_diversions: &mut EntryRegDiversions, target: Block) {
         // No need to save anything if there is no diversions to be recorded.
         if self.is_empty() {
             return;
@@ -208,9 +208,9 @@ impl RegDiversions {
         });
     }
 
-    /// Check that the recorded entry for a given `ebb` matches what is recorded in the
+    /// Check that the recorded entry for a given `block` matches what is recorded in the
     /// `entry_diversions`.
-    pub fn check_ebb_entry(&self, entry_diversions: &EntryRegDiversions, target: Ebb) -> bool {
+    pub fn check_block_entry(&self, entry_diversions: &EntryRegDiversions, target: Block) -> bool {
         let entry_divert = match entry_diversions.map.get(target) {
             Some(entry_divert) => entry_divert,
             None => return self.is_empty(),
@@ -235,7 +235,7 @@ impl RegDiversions {
 }
 
 impl EntryRegDiversions {
-    /// Create a new empty entry diversion, to associate diversions to each EBB entry.
+    /// Create a new empty entry diversion, to associate diversions to each block entry.
     pub fn new() -> Self {
         Self {
             map: SparseMap::new(),
@@ -259,9 +259,9 @@ impl Clone for EntryRegDiversions {
 }
 
 /// Implement `SparseMapValue`, as required to make use of a `SparseMap` for mapping the entry
-/// diversions for each EBB.
-impl SparseMapValue for EntryRegDiversionsValue {
-    fn key(&self) -> Ebb {
+/// diversions for each block.
+impl SparseMapValue for EntryRegDiversionsValue {
+    fn key(&self) -> Block {
         self.key
     }
 }
diff --git a/cranelift/codegen/src/regalloc/live_value_tracker.rs b/cranelift/codegen/src/regalloc/live_value_tracker.rs
index 7faed970a7..f106f4b39d 100644
--- a/cranelift/codegen/src/regalloc/live_value_tracker.rs
+++ b/cranelift/codegen/src/regalloc/live_value_tracker.rs
@@ -1,13 +1,13 @@
-//! Track which values are live in an EBB with instruction granularity.
+//! Track which values are live in an block with instruction granularity.
 //!
-//! The `LiveValueTracker` keeps track of the set of live SSA values at each instruction in an EBB.
+//! The `LiveValueTracker` keeps track of the set of live SSA values at each instruction in an block.
 //! The sets of live values are computed on the fly as the tracker is moved from instruction to
-//! instruction, starting at the EBB header.
+//! instruction, starting at the block header.
 
 use crate::dominator_tree::DominatorTree;
 use crate::entity::{EntityList, ListPool};
 use crate::fx::FxHashMap;
-use crate::ir::{DataFlowGraph, Ebb, ExpandedProgramPoint, Inst, Layout, Value};
+use crate::ir::{Block, DataFlowGraph, ExpandedProgramPoint, Inst, Layout, Value};
 use crate::partition_slice::partition_slice;
 use crate::regalloc::affinity::Affinity;
 use crate::regalloc::liveness::Liveness;
@@ -16,13 +16,13 @@ use alloc::vec::Vec;
 
 type ValueList = EntityList;
 
-/// Compute and track live values throughout an EBB.
+/// Compute and track live values throughout an block.
 pub struct LiveValueTracker {
     /// The set of values that are live at the current program point.
     live: LiveValueVec,
 
     /// Saved set of live values for every jump and branch that can potentially be an immediate
-    /// dominator of an EBB.
+    /// dominator of an block.
     ///
     /// This is the set of values that are live *before* the branch.
     idom_sets: FxHashMap,
@@ -37,7 +37,7 @@ pub struct LiveValue {
     /// The live value.
     pub value: Value,
 
-    /// The local ending point of the live range in the current EBB, as returned by
+    /// The local ending point of the live range in the current block, as returned by
     /// `LiveRange::def_local_end()` or `LiveRange::livein_local_end()`.
     pub endpoint: Inst,
 
@@ -47,7 +47,7 @@ pub struct LiveValue {
     /// almost all users of `LiveValue` need to look at it.
     pub affinity: Affinity,
 
-    /// The live range for this value never leaves its EBB.
+    /// The live range for this value never leaves its block.
     pub is_local: bool,
 
     /// This value is dead - the live range ends immediately.
@@ -155,75 +155,75 @@ impl LiveValueTracker {
         &mut self.live.values
     }
 
-    /// Move the current position to the top of `ebb`.
+    /// Move the current position to the top of `block`.
     ///
-    /// This depends on the stored live value set at `ebb`'s immediate dominator, so that must have
+    /// This depends on the stored live value set at `block`'s immediate dominator, so that must have
     /// been visited first.
     ///
     /// Returns `(liveins, args)` as a pair of slices. The first slice is the set of live-in values
-    /// from the immediate dominator. The second slice is the set of `ebb` parameters.
+    /// from the immediate dominator. The second slice is the set of `block` parameters.
     ///
     /// Dead parameters with no uses are included in `args`. Call `drop_dead_args()` to remove them.
-    pub fn ebb_top(
+    pub fn block_top(
         &mut self,
-        ebb: Ebb,
+        block: Block,
         dfg: &DataFlowGraph,
         liveness: &Liveness,
         layout: &Layout,
         domtree: &DominatorTree,
     ) -> (&[LiveValue], &[LiveValue]) {
-        // Start over, compute the set of live values at the top of the EBB from two sources:
+        // Start over, compute the set of live values at the top of the block from two sources:
         //
-        // 1. Values that were live before `ebb`'s immediate dominator, filtered for those that are
+        // 1. Values that were live before `block`'s immediate dominator, filtered for those that are
         //    actually live-in.
-        // 2. Arguments to `ebb` that are not dead.
+        // 2. Arguments to `block` that are not dead.
         //
         self.live.clear();
 
         // Compute the live-in values. Start by filtering the set of values that were live before
         // the immediate dominator. Just use the empty set if there's no immediate dominator (i.e.,
         // the entry block or an unreachable block).
-        if let Some(idom) = domtree.idom(ebb) {
+        if let Some(idom) = domtree.idom(block) {
             // If the immediate dominator exits, we must have a stored list for it. This is a
-            // requirement to the order EBBs are visited: All dominators must have been processed
-            // before the current EBB.
+            // requirement to the order blocks are visited: All dominators must have been processed
+            // before the current block.
             let idom_live_list = self
                 .idom_sets
                 .get(&idom)
                 .expect("No stored live set for dominator");
-            // Get just the values that are live-in to `ebb`.
+            // Get just the values that are live-in to `block`.
             for &value in idom_live_list.as_slice(&self.idom_pool) {
                 let lr = liveness
                     .get(value)
                     .expect("Immediate dominator value has no live range");
 
                 // Check if this value is live-in here.
-                if let Some(endpoint) = lr.livein_local_end(ebb, layout) {
+                if let Some(endpoint) = lr.livein_local_end(block, layout) {
                     self.live.push(value, endpoint, lr);
                 }
             }
         }
 
-        // Now add all the live parameters to `ebb`.
+        // Now add all the live parameters to `block`.
         let first_arg = self.live.values.len();
-        for &value in dfg.ebb_params(ebb) {
+        for &value in dfg.block_params(block) {
             let lr = &liveness[value];
-            debug_assert_eq!(lr.def(), ebb.into());
+            debug_assert_eq!(lr.def(), block.into());
             match lr.def_local_end().into() {
                 ExpandedProgramPoint::Inst(endpoint) => {
                     self.live.push(value, endpoint, lr);
                 }
-                ExpandedProgramPoint::Ebb(local_ebb) => {
-                    // This is a dead EBB parameter which is not even live into the first
-                    // instruction in the EBB.
+                ExpandedProgramPoint::Block(local_block) => {
+                    // This is a dead block parameter which is not even live into the first
+                    // instruction in the block.
                     debug_assert_eq!(
-                        local_ebb, ebb,
-                        "EBB parameter live range ends at wrong EBB header"
+                        local_block, block,
+                        "block parameter live range ends at wrong block header"
                     );
-                    // Give this value a fake endpoint that is the first instruction in the EBB.
+                    // Give this value a fake endpoint that is the first instruction in the block.
                     // We expect it to be removed by calling `drop_dead_args()`.
                     self.live
-                        .push(value, layout.first_inst(ebb).expect("Empty EBB"), lr);
+                        .push(value, layout.first_inst(block).expect("Empty block"), lr);
                 }
             }
         }
@@ -274,8 +274,8 @@ impl LiveValueTracker {
                 ExpandedProgramPoint::Inst(endpoint) => {
                     self.live.push(value, endpoint, lr);
                 }
-                ExpandedProgramPoint::Ebb(ebb) => {
-                    panic!("Instruction result live range can't end at {}", ebb);
+                ExpandedProgramPoint::Block(block) => {
+                    panic!("Instruction result live range can't end at {}", block);
                 }
             }
         }
@@ -310,7 +310,7 @@ impl LiveValueTracker {
 
     /// Drop any values that are marked as `is_dead`.
     ///
-    /// Use this after calling `ebb_top` to clean out dead EBB parameters.
+    /// Use this after calling `block_top` to clean out dead block parameters.
     pub fn drop_dead_params(&mut self) {
         self.live.remove_dead_values();
     }
diff --git a/cranelift/codegen/src/regalloc/liveness.rs b/cranelift/codegen/src/regalloc/liveness.rs
index f195645809..88c106cce4 100644
--- a/cranelift/codegen/src/regalloc/liveness.rs
+++ b/cranelift/codegen/src/regalloc/liveness.rs
@@ -7,18 +7,18 @@
 //! # Liveness consumers
 //!
 //! The primary consumer of the liveness analysis is the SSA coloring pass which goes through each
-//! EBB and assigns a register to the defined values. This algorithm needs to maintain a set of the
-//! currently live values as it is iterating down the instructions in the EBB. It asks the
+//! block and assigns a register to the defined values. This algorithm needs to maintain a set of the
+//! currently live values as it is iterating down the instructions in the block. It asks the
 //! following questions:
 //!
-//! - What is the set of live values at the entry to the EBB?
-//! - When moving past a use of a value, is that value still alive in the EBB, or was that the last
+//! - What is the set of live values at the entry to the block?
+//! - When moving past a use of a value, is that value still alive in the block, or was that the last
 //!   use?
 //! - When moving past a branch, which of the live values are still live below the branch?
 //!
 //! The set of `LiveRange` instances can answer these questions through their `def_local_end` and
-//! `livein_local_end` queries. The coloring algorithm visits EBBs in a topological order of the
-//! dominator tree, so it can compute the set of live values at the beginning of an EBB by starting
+//! `livein_local_end` queries. The coloring algorithm visits blocks in a topological order of the
+//! dominator tree, so it can compute the set of live values at the beginning of an block by starting
 //! from the set of live values at the dominating branch instruction and filtering it with
 //! `livein_local_end`. These sets do not need to be stored in the liveness analysis.
 //!
@@ -43,7 +43,7 @@
 //!
 //! - Quadratic memory use. We need a bit per variable per basic block in the function.
 //! - Dense representation of sparse data. In practice, the majority of SSA values never leave
-//!   their basic block, and those that do span basic blocks rarely span a large number of basic
+//!   their basic block, and those that do spa basic blocks rarely span a large number of basic
 //!   blocks. This makes the data stored in the bitvectors quite sparse.
 //! - Traditionally, the data-flow equations were solved for real program *variables* which does
 //!   not include temporaries used in evaluating expressions. We have an SSA form program which
@@ -141,10 +141,10 @@
 //! - The first time a value is encountered, its live range is constructed as a dead live range
 //!   containing only the defining program point.
 //! - The local interval of the value's live range is extended so it reaches the use. This may
-//!   require creating a new live-in local interval for the EBB.
-//! - If the live range became live-in to the EBB, add the EBB to a work-list.
-//! - While the work-list is non-empty pop a live-in EBB and repeat the two steps above, using each
-//!   of the live-in EBB's CFG predecessor instructions as a 'use'.
+//!   require creating a new live-in local interval for the block.
+//! - If the live range became live-in to the block, add the block to a work-list.
+//! - While the work-list is non-empty pop a live-in block and repeat the two steps above, using each
+//!   of the live-in block's CFG predecessor instructions as a 'use'.
 //!
 //! The effect of this algorithm is to extend the live range of each to reach uses as they are
 //! visited. No data about each value beyond the live range is needed between visiting uses, so
@@ -176,9 +176,9 @@
 //! There is some room for improvement.
 
 use crate::entity::SparseMap;
-use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::flowgraph::{BlockPredecessor, ControlFlowGraph};
 use crate::ir::dfg::ValueDef;
-use crate::ir::{Ebb, Function, Inst, Layout, ProgramPoint, Value};
+use crate::ir::{Block, Function, Inst, Layout, ProgramPoint, Value};
 use crate::isa::{EncInfo, OperandConstraint, TargetIsa};
 use crate::regalloc::affinity::Affinity;
 use crate::regalloc::liverange::LiveRange;
@@ -223,14 +223,14 @@ fn get_or_create<'a>(
                     })
                     .unwrap_or_default();
             }
-            ValueDef::Param(ebb, num) => {
-                def = ebb.into();
-                if func.layout.entry_block() == Some(ebb) {
+            ValueDef::Param(block, num) => {
+                def = block.into();
+                if func.layout.entry_block() == Some(block) {
                     // The affinity for entry block parameters can be inferred from the function
                     // signature.
                     affinity = Affinity::abi(&func.signature.params[num], isa);
                 } else {
-                    // Give normal EBB parameters a register affinity matching their type.
+                    // Give normal block parameters a register affinity matching their type.
                     let rc = isa.regclass_for_abi_type(func.dfg.value_type(value));
                     affinity = Affinity::Reg(rc.into());
                 }
@@ -241,43 +241,43 @@ fn get_or_create<'a>(
     lrset.get_mut(value).unwrap()
 }
 
-/// Extend the live range for `value` so it reaches `to` which must live in `ebb`.
+/// Extend the live range for `value` so it reaches `to` which must live in `block`.
 fn extend_to_use(
     lr: &mut LiveRange,
-    ebb: Ebb,
+    block: Block,
     to: Inst,
-    worklist: &mut Vec,
+    worklist: &mut Vec,
     func: &Function,
     cfg: &ControlFlowGraph,
 ) {
     // This is our scratch working space, and we'll leave it empty when we return.
     debug_assert!(worklist.is_empty());
 
-    // Extend the range locally in `ebb`.
+    // Extend the range locally in `block`.
     // If there already was a live interval in that block, we're done.
-    if lr.extend_in_ebb(ebb, to, &func.layout) {
-        worklist.push(ebb);
+    if lr.extend_in_block(block, to, &func.layout) {
+        worklist.push(block);
     }
 
-    // The work list contains those EBBs where we have learned that the value needs to be
+    // The work list contains those blocks where we have learned that the value needs to be
     // live-in.
     //
     // This algorithm becomes a depth-first traversal up the CFG, enumerating all paths through the
-    // CFG from the existing live range to `ebb`.
+    // CFG from the existing live range to `block`.
     //
     // Extend the live range as we go. The live range itself also serves as a visited set since
-    // `extend_in_ebb` will never return true twice for the same EBB.
+    // `extend_in_block` will never return true twice for the same block.
     //
     while let Some(livein) = worklist.pop() {
-        // We've learned that the value needs to be live-in to the `livein` EBB.
+        // We've learned that the value needs to be live-in to the `livein` block.
         // Make sure it is also live at all predecessor branches to `livein`.
-        for BasicBlock {
-            ebb: pred,
+        for BlockPredecessor {
+            block: pred,
             inst: branch,
         } in cfg.pred_iter(livein)
         {
-            if lr.extend_in_ebb(pred, branch, &func.layout) {
-                // This predecessor EBB also became live-in. We need to process it later.
+            if lr.extend_in_block(pred, branch, &func.layout) {
+                // This predecessor block also became live-in. We need to process it later.
                 worklist.push(pred);
             }
         }
@@ -294,7 +294,7 @@ pub struct Liveness {
     /// Working space for the `extend_to_use` algorithm.
     /// This vector is always empty, except for inside that function.
     /// It lives here to avoid repeated allocation of scratch memory.
-    worklist: Vec,
+    worklist: Vec,
 }
 
 impl Liveness {
@@ -342,7 +342,7 @@ impl Liveness {
 
     /// Move the definition of `value` to `def`.
     ///
-    /// The old and new def points must be in the same EBB, and before the end of the live range.
+    /// The old and new def points must be in the same block, and before the end of the live range.
     pub fn move_def_locally(&mut self, value: Value, def: PP)
     where
         PP: Into,
@@ -353,20 +353,20 @@ impl Liveness {
 
     /// Locally extend the live range for `value` to reach `user`.
     ///
-    /// It is assumed the `value` is already live before `user` in `ebb`.
+    /// It is assumed the `value` is already live before `user` in `block`.
     ///
     /// Returns a mutable reference to the value's affinity in case that also needs to be updated.
     pub fn extend_locally(
         &mut self,
         value: Value,
-        ebb: Ebb,
+        block: Block,
         user: Inst,
         layout: &Layout,
     ) -> &mut Affinity {
-        debug_assert_eq!(Some(ebb), layout.inst_ebb(user));
+        debug_assert_eq!(Some(block), layout.inst_block(user));
         let lr = self.ranges.get_mut(value).expect("Value has no live range");
-        let livein = lr.extend_in_ebb(ebb, user, layout);
-        debug_assert!(!livein, "{} should already be live in {}", value, ebb);
+        let livein = lr.extend_in_block(block, user, layout);
+        debug_assert!(!livein, "{} should already be live in {}", value, block);
         &mut lr.affinity
     }
 
@@ -389,15 +389,15 @@ impl Liveness {
         // The liveness computation needs to visit all uses, but the order doesn't matter.
         // TODO: Perhaps this traversal of the function could be combined with a dead code
         // elimination pass if we visit a post-order of the dominator tree?
-        for ebb in func.layout.ebbs() {
-            // Make sure we have created live ranges for dead EBB parameters.
+        for block in func.layout.blocks() {
+            // Make sure we have created live ranges for dead block parameters.
             // TODO: If these parameters are really dead, we could remove them, except for the
             // entry block which must match the function signature.
-            for &arg in func.dfg.ebb_params(ebb) {
+            for &arg in func.dfg.block_params(block) {
                 get_or_create(&mut self.ranges, arg, isa, func, &encinfo);
             }
 
-            for inst in func.layout.ebb_insts(ebb) {
+            for inst in func.layout.block_insts(block) {
                 // Eliminate all value aliases, they would confuse the register allocator.
                 func.dfg.resolve_aliases_in_arguments(inst);
 
@@ -419,11 +419,11 @@ impl Liveness {
                     let lr = get_or_create(&mut self.ranges, arg, isa, func, &encinfo);
 
                     // Extend the live range to reach this use.
-                    extend_to_use(lr, ebb, inst, &mut self.worklist, func, cfg);
+                    extend_to_use(lr, block, inst, &mut self.worklist, func, cfg);
 
                     // Apply operand constraint, ignoring any variable arguments after the fixed
                     // operands described by `operand_constraints`. Variable arguments are either
-                    // EBB arguments or call/return ABI arguments.
+                    // block arguments or call/return ABI arguments.
                     if let Some(constraint) = operand_constraints.next() {
                         lr.affinity.merge(constraint, ®info);
                     }
diff --git a/cranelift/codegen/src/regalloc/liverange.rs b/cranelift/codegen/src/regalloc/liverange.rs
index f49cbcc682..0e2f8385fc 100644
--- a/cranelift/codegen/src/regalloc/liverange.rs
+++ b/cranelift/codegen/src/regalloc/liverange.rs
@@ -6,29 +6,29 @@
 //!
 //! # Local Live Ranges
 //!
-//! Inside a single extended basic block, the live range of a value is always an interval between
-//! two program points (if the value is live in the EBB at all). The starting point is either:
+//! Inside a single basic block, the live range of a value is always an interval between
+//! two program points (if the value is live in the block at all). The starting point is either:
 //!
 //! 1. The instruction that defines the value, or
-//! 2. The EBB header, because the value is an argument to the EBB, or
-//! 3. The EBB header, because the value is defined in another EBB and live-in to this one.
+//! 2. The block header, because the value is an argument to the block, or
+//! 3. The block header, because the value is defined in another block and live-in to this one.
 //!
 //! The ending point of the local live range is the last of the following program points in the
-//! EBB:
+//! block:
 //!
-//! 1. The last use in the EBB, where a *use* is an instruction that has the value as an argument.
-//! 2. The last branch or jump instruction in the EBB that can reach a use.
+//! 1. The last use in the block, where a *use* is an instruction that has the value as an argument.
+//! 2. The last branch or jump instruction in the block that can reach a use.
 //! 3. If the value has no uses anywhere (a *dead value*), the program point that defines it.
 //!
-//! Note that 2. includes loop back-edges to the same EBB. In general, if a value is defined
+//! Note that 2. includes loop back-edges to the same block. In general, if a value is defined
 //! outside a loop and used inside the loop, it will be live in the entire loop.
 //!
 //! # Global Live Ranges
 //!
-//! Values that appear in more than one EBB have a *global live range* which can be seen as the
-//! disjoint union of the per-EBB local intervals for all of the EBBs where the value is live.
-//! Together with a `ProgramOrder` which provides a linear ordering of the EBBs, the global live
-//! range becomes a linear sequence of disjoint intervals, at most one per EBB.
+//! Values that appear in more than one block have a *global live range* which can be seen as the
+//! disjoint union of the per-block local intervals for all of the blocks where the value is live.
+//! Together with a `ProgramOrder` which provides a linear ordering of the blocks, the global live
+//! range becomes a linear sequence of disjoint intervals, at most one per block.
 //!
 //! In the special case of a dead value, the global live range is a single interval where the start
 //! and end points are the same. The global live range of a value is never completely empty.
@@ -64,58 +64,58 @@
 //! ## Current representation
 //!
 //! Our current implementation uses a sorted array of compressed intervals, represented by their
-//! boundaries (Ebb, Inst), sorted by Ebb. This is a simple data structure, enables coalescing of
+//! boundaries (Block, Inst), sorted by Block. This is a simple data structure, enables coalescing of
 //! intervals easily, and shows some nice performance behavior. See
 //! https://github.com/bytecodealliance/cranelift/issues/1084 for benchmarks against using a
-//! bforest::Map.
+//! bforest::Map.
 //!
-//! ## EBB ordering
+//! ## block ordering
 //!
-//! The relative order of EBBs is used to maintain a sorted list of live-in intervals and to
-//! coalesce adjacent live-in intervals when the prior interval covers the whole EBB. This doesn't
+//! The relative order of blocks is used to maintain a sorted list of live-in intervals and to
+//! coalesce adjacent live-in intervals when the prior interval covers the whole block. This doesn't
 //! depend on any property of the program order, so alternative orderings are possible:
 //!
-//! 1. The EBB layout order. This is what we currently use.
+//! 1. The block layout order. This is what we currently use.
 //! 2. A topological order of the dominator tree. All the live-in intervals would come after the
 //!    def interval.
-//! 3. A numerical order by EBB number. Performant because it doesn't need to indirect through the
+//! 3. A numerical order by block number. Performant because it doesn't need to indirect through the
 //!    `ProgramOrder` for comparisons.
 //!
 //! These orderings will cause small differences in coalescing opportunities, but all of them would
 //! do a decent job of compressing a long live range. The numerical order might be preferable
 //! because:
 //!
-//! - It has better performance because EBB numbers can be compared directly without any table
+//! - It has better performance because block numbers can be compared directly without any table
 //!   lookups.
-//! - If EBB numbers are not reused, it is safe to allocate new EBBs without getting spurious
-//!   live-in intervals from any coalesced representations that happen to cross a new EBB.
+//! - If block numbers are not reused, it is safe to allocate new blocks without getting spurious
+//!   live-in intervals from any coalesced representations that happen to cross a new block.
 //!
 //! For comparing instructions, the layout order is always what we want.
 //!
 //! ## Alternative representation
 //!
-//! Since a local live-in interval always begins at its EBB header, it is uniquely described by its
-//! end point instruction alone. We can use the layout to look up the EBB containing the end point.
+//! Since a local live-in interval always begins at its block header, it is uniquely described by its
+//! end point instruction alone. We can use the layout to look up the block containing the end point.
 //! This means that a sorted `Vec` would be enough to represent the set of live-in intervals.
 //!
 //! Coalescing is an important compression technique because some live ranges can span thousands of
-//! EBBs. We can represent that by switching to a sorted `Vec` representation where
-//! an `[Ebb, Inst]` pair represents a coalesced range, while an `Inst` entry without a preceding
-//! `Ebb` entry represents a single live-in interval.
+//! blocks. We can represent that by switching to a sorted `Vec` representation where
+//! an `[Block, Inst]` pair represents a coalesced range, while an `Inst` entry without a preceding
+//! `Block` entry represents a single live-in interval.
 //!
 //! This representation is more compact for a live range with many uncoalesced live-in intervals.
 //! It is more complicated to work with, though, so it is probably not worth it. The performance
-//! benefits of switching to a numerical EBB order only appears if the binary search is doing
-//! EBB-EBB comparisons.
+//! benefits of switching to a numerical block order only appears if the binary search is doing
+//! block-block comparisons.
 //!
-//! A `BTreeMap` could have been used for the live-in intervals, but it doesn't provide
+//! A `BTreeMap` could have been used for the live-in intervals, but it doesn't provide
 //! the necessary API to make coalescing easy, nor does it optimize for our types' sizes.
 //!
-//! Even the specialized `bforest::Map` implementation is slower than a plain sorted
+//! Even the specialized `bforest::Map` implementation is slower than a plain sorted
 //! array, see https://github.com/bytecodealliance/cranelift/issues/1084 for details.
 
 use crate::entity::SparseMapValue;
-use crate::ir::{Ebb, ExpandedProgramPoint, Inst, Layout, ProgramOrder, ProgramPoint, Value};
+use crate::ir::{Block, ExpandedProgramPoint, Inst, Layout, ProgramOrder, ProgramPoint, Value};
 use crate::regalloc::affinity::Affinity;
 use core::cmp::Ordering;
 use core::marker::PhantomData;
@@ -124,14 +124,14 @@ use smallvec::SmallVec;
 /// Global live range of a single SSA value.
 ///
 /// As [explained in the module documentation](index.html#local-live-ranges), the live range of an
-/// SSA value is the disjoint union of a set of intervals, each local to a single EBB, and with at
-/// most one interval per EBB. We further distinguish between:
+/// SSA value is the disjoint union of a set of intervals, each local to a single block, and with at
+/// most one interval per block. We further distinguish between:
 ///
-/// 1. The *def interval* is the local interval in the EBB where the value is defined, and
-/// 2. The *live-in intervals* are the local intervals in the remaining EBBs.
+/// 1. The *def interval* is the local interval in the block where the value is defined, and
+/// 2. The *live-in intervals* are the local intervals in the remaining blocks.
 ///
-/// A live-in interval always begins at the EBB header, while the def interval can begin at the
-/// defining instruction, or at the EBB header for an EBB argument value.
+/// A live-in interval always begins at the block header, while the def interval can begin at the
+/// defining instruction, or at the block header for an block argument value.
 ///
 /// All values have a def interval, but a large proportion of values don't have any live-in
 /// intervals. These are called *local live ranges*.
@@ -139,11 +139,11 @@ use smallvec::SmallVec;
 /// # Program order requirements
 ///
 /// The internal representation of a `LiveRange` depends on a consistent `ProgramOrder` both for
-/// ordering instructions inside an EBB *and* for ordering EBBs. The methods that depend on the
+/// ordering instructions inside an block *and* for ordering blocks. The methods that depend on the
 /// ordering take an explicit `ProgramOrder` object, and it is the caller's responsibility to
 /// ensure that the provided ordering is consistent between calls.
 ///
-/// In particular, changing the order of EBBs or inserting new EBBs will invalidate live ranges.
+/// In particular, changing the order of blocks or inserting new blocks will invalidate live ranges.
 ///
 /// Inserting new instructions in the layout is safe, but removing instructions is not. Besides the
 /// instructions using or defining their value, `LiveRange` structs can contain references to
@@ -152,7 +152,7 @@ pub type LiveRange = GenericLiveRange;
 
 // See comment of liveins below.
 pub struct Interval {
-    begin: Ebb,
+    begin: Block,
     end: Inst,
 }
 
@@ -168,10 +168,10 @@ pub struct GenericLiveRange {
     /// The preferred register allocation for this value.
     pub affinity: Affinity,
 
-    /// The instruction or EBB header where this value is defined.
+    /// The instruction or block header where this value is defined.
     def_begin: ProgramPoint,
 
-    /// The end point of the def interval. This must always belong to the same EBB as `def_begin`.
+    /// The end point of the def interval. This must always belong to the same block as `def_begin`.
     ///
     /// We always have `def_begin <= def_end` with equality implying a dead def live range with no
     /// uses.
@@ -179,12 +179,12 @@ pub struct GenericLiveRange {
 
     /// Additional live-in intervals sorted in program order.
     ///
-    /// This vector is empty for most values which are only used in one EBB.
+    /// This vector is empty for most values which are only used in one block.
     ///
-    /// An entry `ebb -> inst` means that the live range is live-in to `ebb`, continuing up to
-    /// `inst` which may belong to a later EBB in the program order.
+    /// An entry `block -> inst` means that the live range is live-in to `block`, continuing up to
+    /// `inst` which may belong to a later block in the program order.
     ///
-    /// The entries are non-overlapping, and none of them overlap the EBB where the value is
+    /// The entries are non-overlapping, and none of them overlap the block where the value is
     /// defined.
     liveins: SmallVec<[Interval; 2]>,
 
@@ -210,7 +210,7 @@ macro_rules! cmp {
 impl GenericLiveRange {
     /// Create a new live range for `value` defined at `def`.
     ///
-    /// The live range will be created as dead, but it can be extended with `extend_in_ebb()`.
+    /// The live range will be created as dead, but it can be extended with `extend_in_block()`.
     pub fn new(value: Value, def: ProgramPoint, affinity: Affinity) -> Self {
         Self {
             value,
@@ -222,14 +222,14 @@ impl GenericLiveRange {
         }
     }
 
-    /// Finds an entry in the compressed set of live-in intervals that contains `ebb`, or return
+    /// Finds an entry in the compressed set of live-in intervals that contains `block`, or return
     /// the position where to insert such a new entry.
-    fn lookup_entry_containing_ebb(&self, ebb: Ebb, order: &PO) -> Result {
+    fn lookup_entry_containing_block(&self, block: Block, order: &PO) -> Result {
         self.liveins
-            .binary_search_by(|interval| order.cmp(interval.begin, ebb))
+            .binary_search_by(|interval| order.cmp(interval.begin, block))
             .or_else(|n| {
-                // The previous interval's end might cover the searched ebb.
-                if n > 0 && cmp!(order, ebb <= self.liveins[n - 1].end) {
+                // The previous interval's end might cover the searched block.
+                if n > 0 && cmp!(order, block <= self.liveins[n - 1].end) {
                     Ok(n - 1)
                 } else {
                     Err(n)
@@ -237,23 +237,23 @@ impl GenericLiveRange {
             })
     }
 
-    /// Extend the local interval for `ebb` so it reaches `to` which must belong to `ebb`.
+    /// Extend the local interval for `block` so it reaches `to` which must belong to `block`.
     /// Create a live-in interval if necessary.
     ///
-    /// If the live range already has a local interval in `ebb`, extend its end point so it
+    /// If the live range already has a local interval in `block`, extend its end point so it
     /// includes `to`, and return false.
     ///
-    /// If the live range did not previously have a local interval in `ebb`, add one so the value
-    /// is live-in to `ebb`, extending to `to`. Return true.
+    /// If the live range did not previously have a local interval in `block`, add one so the value
+    /// is live-in to `block`, extending to `to`. Return true.
     ///
     /// The return value can be used to detect if we just learned that the value is live-in to
-    /// `ebb`. This can trigger recursive extensions in `ebb`'s CFG predecessor blocks.
-    pub fn extend_in_ebb(&mut self, ebb: Ebb, inst: Inst, order: &PO) -> bool {
+    /// `block`. This can trigger recursive extensions in `block`'s CFG predecessor blocks.
+    pub fn extend_in_block(&mut self, block: Block, inst: Inst, order: &PO) -> bool {
         // First check if we're extending the def interval.
         //
-        // We're assuming here that `inst` never precedes `def_begin` in the same EBB, but we can't
-        // check it without a method for getting `inst`'s EBB.
-        if cmp!(order, ebb <= self.def_end) && cmp!(order, inst >= self.def_begin) {
+        // We're assuming here that `inst` never precedes `def_begin` in the same block, but we can't
+        // check it without a method for getting `inst`'s block.
+        if cmp!(order, block <= self.def_end) && cmp!(order, inst >= self.def_begin) {
             let inst_pp = inst.into();
             debug_assert_ne!(
                 inst_pp, self.def_begin,
@@ -266,7 +266,7 @@ impl GenericLiveRange {
         }
 
         // Now check if we're extending any of the existing live-in intervals.
-        match self.lookup_entry_containing_ebb(ebb, order) {
+        match self.lookup_entry_containing_block(block, order) {
             Ok(n) => {
                 // We found one interval and might need to extend it.
                 if cmp!(order, inst <= self.liveins[n].end) {
@@ -278,7 +278,7 @@ impl GenericLiveRange {
                 // coalesce the two intervals:
                 // [ival.begin; ival.end] + [next.begin; next.end] = [ival.begin; next.end]
                 if let Some(next) = &self.liveins.get(n + 1) {
-                    if order.is_ebb_gap(inst, next.begin) {
+                    if order.is_block_gap(inst, next.begin) {
                         // At this point we can choose to remove the current interval or the next
                         // one; remove the next one to avoid one memory move.
                         let next_end = next.end;
@@ -295,17 +295,17 @@ impl GenericLiveRange {
             }
 
             Err(n) => {
-                // No interval was found containing the current EBB: we need to insert a new one,
+                // No interval was found containing the current block: we need to insert a new one,
                 // unless there's a coalescing opportunity with the previous or next one.
                 let coalesce_next = self
                     .liveins
                     .get(n)
-                    .filter(|next| order.is_ebb_gap(inst, next.begin))
+                    .filter(|next| order.is_block_gap(inst, next.begin))
                     .is_some();
                 let coalesce_prev = self
                     .liveins
                     .get(n.wrapping_sub(1))
-                    .filter(|prev| order.is_ebb_gap(prev.end, ebb))
+                    .filter(|prev| order.is_block_gap(prev.end, block))
                     .is_some();
 
                 match (coalesce_prev, coalesce_next) {
@@ -324,8 +324,8 @@ impl GenericLiveRange {
                         self.liveins[n - 1].end = inst;
                     }
                     (false, true) => {
-                        debug_assert!(cmp!(order, ebb <= self.liveins[n].begin));
-                        self.liveins[n].begin = ebb;
+                        debug_assert!(cmp!(order, block <= self.liveins[n].begin));
+                        self.liveins[n].begin = block;
                     }
 
                     (false, false) => {
@@ -333,7 +333,7 @@ impl GenericLiveRange {
                         self.liveins.insert(
                             n,
                             Interval {
-                                begin: ebb,
+                                begin: block,
                                 end: inst,
                             },
                         );
@@ -355,15 +355,15 @@ impl GenericLiveRange {
 
     /// Is this a local live range?
     ///
-    /// A local live range is only used in the same EBB where it was defined. It is allowed to span
-    /// multiple basic blocks within that EBB.
+    /// A local live range is only used in the same block where it was defined. It is allowed to span
+    /// multiple basic blocks within that block.
     pub fn is_local(&self) -> bool {
         self.liveins.is_empty()
     }
 
     /// Get the program point where this live range is defined.
     ///
-    /// This will be an EBB header when the value is an EBB argument, otherwise it is the defining
+    /// This will be an block header when the value is an block argument, otherwise it is the defining
     /// instruction.
     pub fn def(&self) -> ProgramPoint {
         self.def_begin
@@ -371,33 +371,33 @@ impl GenericLiveRange {
 
     /// Move the definition of this value to a new program point.
     ///
-    /// It is only valid to move the definition within the same EBB, and it can't be moved beyond
+    /// It is only valid to move the definition within the same block, and it can't be moved beyond
     /// `def_local_end()`.
     pub fn move_def_locally(&mut self, def: ProgramPoint) {
         self.def_begin = def;
     }
 
-    /// Get the local end-point of this live range in the EBB where it is defined.
+    /// Get the local end-point of this live range in the block where it is defined.
     ///
-    /// This can be the EBB header itself in the case of a dead EBB argument.
+    /// This can be the block header itself in the case of a dead block argument.
     /// Otherwise, it will be the last local use or branch/jump that can reach a use.
     pub fn def_local_end(&self) -> ProgramPoint {
         self.def_end
     }
 
-    /// Get the local end-point of this live range in an EBB where it is live-in.
+    /// Get the local end-point of this live range in an block where it is live-in.
     ///
-    /// If this live range is not live-in to `ebb`, return `None`. Otherwise, return the end-point
-    /// of this live range's local interval in `ebb`.
+    /// If this live range is not live-in to `block`, return `None`. Otherwise, return the end-point
+    /// of this live range's local interval in `block`.
     ///
-    /// If the live range is live through all of `ebb`, the terminator of `ebb` is a correct
+    /// If the live range is live through all of `block`, the terminator of `block` is a correct
     /// answer, but it is also possible that an even later program point is returned. So don't
-    /// depend on the returned `Inst` to belong to `ebb`.
-    pub fn livein_local_end(&self, ebb: Ebb, order: &PO) -> Option {
-        self.lookup_entry_containing_ebb(ebb, order)
+    /// depend on the returned `Inst` to belong to `block`.
+    pub fn livein_local_end(&self, block: Block, order: &PO) -> Option {
+        self.lookup_entry_containing_block(block, order)
             .and_then(|i| {
                 let inst = self.liveins[i].end;
-                if cmp!(order, ebb < inst) {
+                if cmp!(order, block < inst) {
                     Ok(inst)
                 } else {
                     // Can be any error type, really, since it's discarded by ok().
@@ -407,25 +407,25 @@ impl GenericLiveRange {
             .ok()
     }
 
-    /// Is this value live-in to `ebb`?
+    /// Is this value live-in to `block`?
     ///
-    /// An EBB argument is not considered to be live in.
-    pub fn is_livein(&self, ebb: Ebb, order: &PO) -> bool {
-        self.livein_local_end(ebb, order).is_some()
+    /// An block argument is not considered to be live in.
+    pub fn is_livein(&self, block: Block, order: &PO) -> bool {
+        self.livein_local_end(block, order).is_some()
     }
 
     /// Get all the live-in intervals.
     ///
     /// Note that the intervals are stored in a compressed form so each entry may span multiple
-    /// EBBs where the value is live in.
-    pub fn liveins<'a>(&'a self) -> impl Iterator + 'a {
+    /// blocks where the value is live in.
+    pub fn liveins<'a>(&'a self) -> impl Iterator + 'a {
         self.liveins
             .iter()
             .map(|interval| (interval.begin, interval.end))
     }
 
-    /// Check if this live range overlaps a definition in `ebb`.
-    pub fn overlaps_def(&self, def: ExpandedProgramPoint, ebb: Ebb, order: &PO) -> bool {
+    /// Check if this live range overlaps a definition in `block`.
+    pub fn overlaps_def(&self, def: ExpandedProgramPoint, block: Block, order: &PO) -> bool {
         // Two defs at the same program point always overlap, even if one is dead.
         if def == self.def_begin.into() {
             return true;
@@ -437,29 +437,29 @@ impl GenericLiveRange {
         }
 
         // Check for an overlap with a live-in range.
-        match self.livein_local_end(ebb, order) {
+        match self.livein_local_end(block, order) {
             Some(inst) => cmp!(order, def < inst),
             None => false,
         }
     }
 
-    /// Check if this live range reaches a use at `user` in `ebb`.
-    pub fn reaches_use(&self, user: Inst, ebb: Ebb, order: &PO) -> bool {
+    /// Check if this live range reaches a use at `user` in `block`.
+    pub fn reaches_use(&self, user: Inst, block: Block, order: &PO) -> bool {
         // Check for an overlap with the local range.
         if cmp!(order, user > self.def_begin) && cmp!(order, user <= self.def_end) {
             return true;
         }
 
         // Check for an overlap with a live-in range.
-        match self.livein_local_end(ebb, order) {
+        match self.livein_local_end(block, order) {
             Some(inst) => cmp!(order, user <= inst),
             None => false,
         }
     }
 
-    /// Check if this live range is killed at `user` in `ebb`.
-    pub fn killed_at(&self, user: Inst, ebb: Ebb, order: &PO) -> bool {
-        self.def_local_end() == user.into() || self.livein_local_end(ebb, order) == Some(user)
+    /// Check if this live range is killed at `user` in `block`.
+    pub fn killed_at(&self, user: Inst, block: Block, order: &PO) -> bool {
+        self.def_local_end() == user.into() || self.livein_local_end(block, order) == Some(user)
     }
 }
 
@@ -474,15 +474,15 @@ impl SparseMapValue for GenericLiveRange {
 mod tests {
     use super::{GenericLiveRange, Interval};
     use crate::entity::EntityRef;
-    use crate::ir::{Ebb, Inst, Value};
+    use crate::ir::{Block, Inst, Value};
     use crate::ir::{ExpandedProgramPoint, ProgramOrder};
     use alloc::vec::Vec;
     use core::cmp::Ordering;
 
     // Dummy program order which simply compares indexes.
-    // It is assumed that EBBs have indexes that are multiples of 10, and instructions have indexes
-    // in between. `is_ebb_gap` assumes that terminator instructions have indexes of the form
-    // ebb * 10 + 1. This is used in the coalesce test.
+    // It is assumed that blocks have indexes that are multiples of 10, and instructions have indexes
+    // in between. `is_block_gap` assumes that terminator instructions have indexes of the form
+    // block * 10 + 1. This is used in the coalesce test.
     struct ProgOrder {}
 
     impl ProgramOrder for ProgOrder {
@@ -494,7 +494,7 @@ mod tests {
             fn idx(pp: ExpandedProgramPoint) -> usize {
                 match pp {
                     ExpandedProgramPoint::Inst(i) => i.index(),
-                    ExpandedProgramPoint::Ebb(e) => e.index(),
+                    ExpandedProgramPoint::Block(e) => e.index(),
                 }
             }
 
@@ -503,31 +503,31 @@ mod tests {
             ia.cmp(&ib)
         }
 
-        fn is_ebb_gap(&self, inst: Inst, ebb: Ebb) -> bool {
-            inst.index() % 10 == 1 && ebb.index() / 10 == inst.index() / 10 + 1
+        fn is_block_gap(&self, inst: Inst, block: Block) -> bool {
+            inst.index() % 10 == 1 && block.index() / 10 == inst.index() / 10 + 1
         }
     }
 
     impl ProgOrder {
-        // Get the EBB corresponding to `inst`.
-        fn inst_ebb(&self, inst: Inst) -> Ebb {
+        // Get the block corresponding to `inst`.
+        fn inst_block(&self, inst: Inst) -> Block {
             let i = inst.index();
-            Ebb::new(i - i % 10)
+            Block::new(i - i % 10)
         }
 
-        // Get the EBB of a program point.
-        fn pp_ebb>(&self, pp: PP) -> Ebb {
+        // Get the block of a program point.
+        fn pp_block>(&self, pp: PP) -> Block {
             match pp.into() {
-                ExpandedProgramPoint::Inst(i) => self.inst_ebb(i),
-                ExpandedProgramPoint::Ebb(e) => e,
+                ExpandedProgramPoint::Inst(i) => self.inst_block(i),
+                ExpandedProgramPoint::Block(e) => e,
             }
         }
 
         // Validate the live range invariants.
         fn validate(&self, lr: &GenericLiveRange) {
-            // The def interval must cover a single EBB.
-            let def_ebb = self.pp_ebb(lr.def_begin);
-            assert_eq!(def_ebb, self.pp_ebb(lr.def_end));
+            // The def interval must cover a single block.
+            let def_block = self.pp_block(lr.def_begin);
+            assert_eq!(def_block, self.pp_block(lr.def_end));
 
             // Check that the def interval isn't backwards.
             match self.cmp(lr.def_begin, lr.def_end) {
@@ -552,7 +552,7 @@ mod tests {
                 assert!(
                     self.cmp(lr.def_end, begin) == Ordering::Less
                         || self.cmp(lr.def_begin, end) == Ordering::Greater,
-                    "Interval can't overlap the def EBB"
+                    "Interval can't overlap the def block"
                 );
 
                 // Save for next round.
@@ -567,10 +567,10 @@ mod tests {
     #[test]
     fn dead_def_range() {
         let v0 = Value::new(0);
-        let e0 = Ebb::new(0);
+        let e0 = Block::new(0);
         let i1 = Inst::new(1);
         let i2 = Inst::new(2);
-        let e2 = Ebb::new(2);
+        let e2 = Block::new(2);
         let lr = GenericLiveRange::new(v0, i1.into(), Default::default());
         assert!(lr.is_dead());
         assert!(lr.is_local());
@@ -588,13 +588,13 @@ mod tests {
     #[test]
     fn dead_arg_range() {
         let v0 = Value::new(0);
-        let e2 = Ebb::new(2);
+        let e2 = Block::new(2);
         let lr = GenericLiveRange::new(v0, e2.into(), Default::default());
         assert!(lr.is_dead());
         assert!(lr.is_local());
         assert_eq!(lr.def(), e2.into());
         assert_eq!(lr.def_local_end(), e2.into());
-        // The def interval of an EBB argument does not count as live-in.
+        // The def interval of an block argument does not count as live-in.
         assert_eq!(lr.livein_local_end(e2, PO), None);
         PO.validate(&lr);
     }
@@ -602,13 +602,13 @@ mod tests {
     #[test]
     fn local_def() {
         let v0 = Value::new(0);
-        let e10 = Ebb::new(10);
+        let e10 = Block::new(10);
         let i11 = Inst::new(11);
         let i12 = Inst::new(12);
         let i13 = Inst::new(13);
         let mut lr = GenericLiveRange::new(v0, i11.into(), Default::default());
 
-        assert_eq!(lr.extend_in_ebb(e10, i13, PO), false);
+        assert_eq!(lr.extend_in_block(e10, i13, PO), false);
         PO.validate(&lr);
         assert!(!lr.is_dead());
         assert!(lr.is_local());
@@ -616,7 +616,7 @@ mod tests {
         assert_eq!(lr.def_local_end(), i13.into());
 
         // Extending to an already covered inst should not change anything.
-        assert_eq!(lr.extend_in_ebb(e10, i12, PO), false);
+        assert_eq!(lr.extend_in_block(e10, i12, PO), false);
         PO.validate(&lr);
         assert_eq!(lr.def(), i11.into());
         assert_eq!(lr.def_local_end(), i13.into());
@@ -625,15 +625,15 @@ mod tests {
     #[test]
     fn local_arg() {
         let v0 = Value::new(0);
-        let e10 = Ebb::new(10);
+        let e10 = Block::new(10);
         let i11 = Inst::new(11);
         let i12 = Inst::new(12);
         let i13 = Inst::new(13);
         let mut lr = GenericLiveRange::new(v0, e10.into(), Default::default());
 
-        // Extending a dead EBB argument in its own block should not indicate that a live-in
+        // Extending a dead block argument in its own block should not indicate that a live-in
         // interval was created.
-        assert_eq!(lr.extend_in_ebb(e10, i12, PO), false);
+        assert_eq!(lr.extend_in_block(e10, i12, PO), false);
         PO.validate(&lr);
         assert!(!lr.is_dead());
         assert!(lr.is_local());
@@ -641,13 +641,13 @@ mod tests {
         assert_eq!(lr.def_local_end(), i12.into());
 
         // Extending to an already covered inst should not change anything.
-        assert_eq!(lr.extend_in_ebb(e10, i11, PO), false);
+        assert_eq!(lr.extend_in_block(e10, i11, PO), false);
         PO.validate(&lr);
         assert_eq!(lr.def(), e10.into());
         assert_eq!(lr.def_local_end(), i12.into());
 
         // Extending further.
-        assert_eq!(lr.extend_in_ebb(e10, i13, PO), false);
+        assert_eq!(lr.extend_in_block(e10, i13, PO), false);
         PO.validate(&lr);
         assert_eq!(lr.def(), e10.into());
         assert_eq!(lr.def_local_end(), i13.into());
@@ -656,28 +656,28 @@ mod tests {
     #[test]
     fn global_def() {
         let v0 = Value::new(0);
-        let e10 = Ebb::new(10);
+        let e10 = Block::new(10);
         let i11 = Inst::new(11);
         let i12 = Inst::new(12);
-        let e20 = Ebb::new(20);
+        let e20 = Block::new(20);
         let i21 = Inst::new(21);
         let i22 = Inst::new(22);
         let i23 = Inst::new(23);
         let mut lr = GenericLiveRange::new(v0, i11.into(), Default::default());
 
-        assert_eq!(lr.extend_in_ebb(e10, i12, PO), false);
+        assert_eq!(lr.extend_in_block(e10, i12, PO), false);
 
         // Adding a live-in interval.
-        assert_eq!(lr.extend_in_ebb(e20, i22, PO), true);
+        assert_eq!(lr.extend_in_block(e20, i22, PO), true);
         PO.validate(&lr);
         assert_eq!(lr.livein_local_end(e20, PO), Some(i22));
 
         // Non-extending the live-in.
-        assert_eq!(lr.extend_in_ebb(e20, i21, PO), false);
+        assert_eq!(lr.extend_in_block(e20, i21, PO), false);
         assert_eq!(lr.livein_local_end(e20, PO), Some(i22));
 
         // Extending the existing live-in.
-        assert_eq!(lr.extend_in_ebb(e20, i23, PO), false);
+        assert_eq!(lr.extend_in_block(e20, i23, PO), false);
         PO.validate(&lr);
         assert_eq!(lr.livein_local_end(e20, PO), Some(i23));
     }
@@ -686,35 +686,35 @@ mod tests {
     fn coalesce() {
         let v0 = Value::new(0);
         let i11 = Inst::new(11);
-        let e20 = Ebb::new(20);
+        let e20 = Block::new(20);
         let i21 = Inst::new(21);
-        let e30 = Ebb::new(30);
+        let e30 = Block::new(30);
         let i31 = Inst::new(31);
-        let e40 = Ebb::new(40);
+        let e40 = Block::new(40);
         let i41 = Inst::new(41);
         let mut lr = GenericLiveRange::new(v0, i11.into(), Default::default());
 
-        assert_eq!(lr.extend_in_ebb(e30, i31, PO,), true);
+        assert_eq!(lr.extend_in_block(e30, i31, PO,), true);
         assert_eq!(lr.liveins().collect::>(), [(e30, i31)]);
 
         // Coalesce to previous
-        assert_eq!(lr.extend_in_ebb(e40, i41, PO,), true);
+        assert_eq!(lr.extend_in_block(e40, i41, PO,), true);
         assert_eq!(lr.liveins().collect::>(), [(e30, i41)]);
 
         // Coalesce to next
-        assert_eq!(lr.extend_in_ebb(e20, i21, PO,), true);
+        assert_eq!(lr.extend_in_block(e20, i21, PO,), true);
         assert_eq!(lr.liveins().collect::>(), [(e20, i41)]);
 
         let mut lr = GenericLiveRange::new(v0, i11.into(), Default::default());
 
-        assert_eq!(lr.extend_in_ebb(e40, i41, PO,), true);
+        assert_eq!(lr.extend_in_block(e40, i41, PO,), true);
         assert_eq!(lr.liveins().collect::>(), [(e40, i41)]);
 
-        assert_eq!(lr.extend_in_ebb(e20, i21, PO,), true);
+        assert_eq!(lr.extend_in_block(e20, i21, PO,), true);
         assert_eq!(lr.liveins().collect::>(), [(e20, i21), (e40, i41)]);
 
         // Coalesce to previous and next
-        assert_eq!(lr.extend_in_ebb(e30, i31, PO,), true);
+        assert_eq!(lr.extend_in_block(e30, i31, PO,), true);
         assert_eq!(lr.liveins().collect::>(), [(e20, i41)]);
     }
 }
diff --git a/cranelift/codegen/src/regalloc/reload.rs b/cranelift/codegen/src/regalloc/reload.rs
index 0f4f595346..cdafb68af8 100644
--- a/cranelift/codegen/src/regalloc/reload.rs
+++ b/cranelift/codegen/src/regalloc/reload.rs
@@ -13,7 +13,7 @@ use crate::cursor::{Cursor, EncCursor};
 use crate::dominator_tree::DominatorTree;
 use crate::entity::{SparseMap, SparseMapValue};
 use crate::ir::{AbiParam, ArgumentLoc, InstBuilder};
-use crate::ir::{Ebb, Function, Inst, InstructionData, Opcode, Value, ValueLoc};
+use crate::ir::{Block, Function, Inst, InstructionData, Opcode, Value, ValueLoc};
 use crate::isa::RegClass;
 use crate::isa::{ConstraintKind, EncInfo, Encoding, RecipeConstraints, TargetIsa};
 use crate::regalloc::affinity::Affinity;
@@ -113,24 +113,24 @@ impl SparseMapValue for ReloadedValue {
 
 impl<'a> Context<'a> {
     fn run(&mut self, tracker: &mut LiveValueTracker) {
-        self.topo.reset(self.cur.func.layout.ebbs());
-        while let Some(ebb) = self.topo.next(&self.cur.func.layout, self.domtree) {
-            self.visit_ebb(ebb, tracker);
+        self.topo.reset(self.cur.func.layout.blocks());
+        while let Some(block) = self.topo.next(&self.cur.func.layout, self.domtree) {
+            self.visit_block(block, tracker);
         }
     }
 
-    fn visit_ebb(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) {
-        debug!("Reloading {}:", ebb);
-        self.visit_ebb_header(ebb, tracker);
+    fn visit_block(&mut self, block: Block, tracker: &mut LiveValueTracker) {
+        debug!("Reloading {}:", block);
+        self.visit_block_header(block, tracker);
         tracker.drop_dead_params();
 
-        // visit_ebb_header() places us at the first interesting instruction in the EBB.
+        // visit_block_header() places us at the first interesting instruction in the block.
         while let Some(inst) = self.cur.current_inst() {
             if !self.cur.func.dfg[inst].opcode().is_ghost() {
                 // This instruction either has an encoding or has ABI constraints, so visit it to
                 // insert spills and fills as needed.
                 let encoding = self.cur.func.encodings[inst];
-                self.visit_inst(ebb, inst, encoding, tracker);
+                self.visit_inst(block, inst, encoding, tracker);
                 tracker.drop_dead(inst);
             } else {
                 // This is a ghost instruction with no encoding and no extra constraints, so we can
@@ -140,29 +140,29 @@ impl<'a> Context<'a> {
         }
     }
 
-    /// Process the EBB parameters. Move to the next instruction in the EBB to be processed
-    fn visit_ebb_header(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) {
-        let (liveins, args) = tracker.ebb_top(
-            ebb,
+    /// Process the block parameters. Move to the next instruction in the block to be processed
+    fn visit_block_header(&mut self, block: Block, tracker: &mut LiveValueTracker) {
+        let (liveins, args) = tracker.block_top(
+            block,
             &self.cur.func.dfg,
             self.liveness,
             &self.cur.func.layout,
             self.domtree,
         );
 
-        if self.cur.func.layout.entry_block() == Some(ebb) {
+        if self.cur.func.layout.entry_block() == Some(block) {
             debug_assert_eq!(liveins.len(), 0);
-            self.visit_entry_params(ebb, args);
+            self.visit_entry_params(block, args);
         } else {
-            self.visit_ebb_params(ebb, args);
+            self.visit_block_params(block, args);
         }
     }
 
     /// Visit the parameters on the entry block.
     /// These values have ABI constraints from the function signature.
-    fn visit_entry_params(&mut self, ebb: Ebb, args: &[LiveValue]) {
+    fn visit_entry_params(&mut self, block: Block, args: &[LiveValue]) {
         debug_assert_eq!(self.cur.func.signature.params.len(), args.len());
-        self.cur.goto_first_inst(ebb);
+        self.cur.goto_first_inst(block);
 
         for (arg_idx, arg) in args.iter().enumerate() {
             let abi = self.cur.func.signature.params[arg_idx];
@@ -175,10 +175,10 @@ impl<'a> Context<'a> {
                             .cur
                             .func
                             .dfg
-                            .replace_ebb_param(arg.value, abi.value_type);
+                            .replace_block_param(arg.value, abi.value_type);
                         let affinity = Affinity::abi(&abi, self.cur.isa);
-                        self.liveness.create_dead(reg, ebb, affinity);
-                        self.insert_spill(ebb, arg.value, reg);
+                        self.liveness.create_dead(reg, block, affinity);
+                        self.insert_spill(block, arg.value, reg);
                     }
                 }
                 ArgumentLoc::Stack(_) => {
@@ -189,15 +189,15 @@ impl<'a> Context<'a> {
         }
     }
 
-    fn visit_ebb_params(&mut self, ebb: Ebb, _args: &[LiveValue]) {
-        self.cur.goto_first_inst(ebb);
+    fn visit_block_params(&mut self, block: Block, _args: &[LiveValue]) {
+        self.cur.goto_first_inst(block);
     }
 
     /// Process the instruction pointed to by `pos`, and advance the cursor to the next instruction
     /// that needs processing.
     fn visit_inst(
         &mut self,
-        ebb: Ebb,
+        block: Block,
         inst: Inst,
         encoding: Encoding,
         tracker: &mut LiveValueTracker,
@@ -265,7 +265,7 @@ impl<'a> Context<'a> {
         {
             self.reload_copy_candidates(inst);
         } else {
-            self.reload_inst_candidates(ebb, inst);
+            self.reload_inst_candidates(block, inst);
         }
 
         // TODO: Reuse reloads for future instructions.
@@ -304,7 +304,7 @@ impl<'a> Context<'a> {
                         let value_type = self.cur.func.dfg.value_type(lv.value);
                         let reg = self.cur.func.dfg.replace_result(lv.value, value_type);
                         self.liveness.create_dead(reg, inst, Affinity::new(op));
-                        self.insert_spill(ebb, lv.value, reg);
+                        self.insert_spill(block, lv.value, reg);
                     }
                 }
             }
@@ -333,14 +333,14 @@ impl<'a> Context<'a> {
                     let reg = self.cur.func.dfg.replace_result(lv.value, abi.value_type);
                     self.liveness
                         .create_dead(reg, inst, Affinity::abi(&abi, self.cur.isa));
-                    self.insert_spill(ebb, lv.value, reg);
+                    self.insert_spill(block, lv.value, reg);
                 }
             }
         }
     }
 
     // Reload the current candidates for the given `inst`.
-    fn reload_inst_candidates(&mut self, ebb: Ebb, inst: Inst) {
+    fn reload_inst_candidates(&mut self, block: Block, inst: Inst) {
         // Insert fill instructions before `inst` and replace `cand.value` with the filled value.
         for cand in self.candidates.iter_mut() {
             if let Some(reload) = self.reloads.get(cand.value) {
@@ -361,15 +361,15 @@ impl<'a> Context<'a> {
             let affinity = Affinity::Reg(cand.regclass.into());
             self.liveness.create_dead(reg, fill, affinity);
             self.liveness
-                .extend_locally(reg, ebb, inst, &self.cur.func.layout);
+                .extend_locally(reg, block, inst, &self.cur.func.layout);
         }
 
         // Rewrite instruction arguments.
         //
-        // Only rewrite those arguments that were identified as candidates. This leaves EBB
-        // arguments on branches as-is without rewriting them. A spilled EBB argument needs to stay
-        // spilled because the matching EBB parameter is going to be in the same virtual register
-        // and therefore the same stack slot as the EBB argument value.
+        // Only rewrite those arguments that were identified as candidates. This leaves block
+        // arguments on branches as-is without rewriting them. A spilled block argument needs to stay
+        // spilled because the matching block parameter is going to be in the same virtual register
+        // and therefore the same stack slot as the block argument value.
         if !self.candidates.is_empty() {
             let args = self.cur.func.dfg.inst_args_mut(inst);
             while let Some(cand) = self.candidates.pop() {
@@ -448,14 +448,14 @@ impl<'a> Context<'a> {
     /// - Insert `stack = spill reg` at `pos`, and assign an encoding.
     /// - Move the `stack` live range starting point to the new instruction.
     /// - Extend the `reg` live range to reach the new instruction.
-    fn insert_spill(&mut self, ebb: Ebb, stack: Value, reg: Value) {
+    fn insert_spill(&mut self, block: Block, stack: Value, reg: Value) {
         self.cur.ins().with_result(stack).spill(reg);
         let inst = self.cur.built_inst();
 
         // Update live ranges.
         self.liveness.move_def_locally(stack, inst);
         self.liveness
-            .extend_locally(reg, ebb, inst, &self.cur.func.layout);
+            .extend_locally(reg, block, inst, &self.cur.func.layout);
     }
 }
 
diff --git a/cranelift/codegen/src/regalloc/safepoint.rs b/cranelift/codegen/src/regalloc/safepoint.rs
index ba846190f3..128900d360 100644
--- a/cranelift/codegen/src/regalloc/safepoint.rs
+++ b/cranelift/codegen/src/regalloc/safepoint.rs
@@ -32,7 +32,7 @@ fn insert_and_encode_safepoint<'f>(
 }
 
 // The emit_stackmaps() function analyzes each instruction to retrieve the liveness of
-// the defs and operands by traversing a function's ebbs in layout order.
+// the defs and operands by traversing a function's blocks in layout order.
 pub fn emit_stackmaps(
     func: &mut Function,
     domtree: &DominatorTree,
@@ -42,13 +42,13 @@ pub fn emit_stackmaps(
 ) {
     let mut curr = func.layout.entry_block();
 
-    while let Some(ebb) = curr {
-        tracker.ebb_top(ebb, &func.dfg, liveness, &func.layout, domtree);
+    while let Some(block) = curr {
+        tracker.block_top(block, &func.dfg, liveness, &func.layout, domtree);
         tracker.drop_dead_params();
         let mut pos = FuncCursor::new(func);
 
-        // From the top of the ebb, step through the instructions.
-        pos.goto_top(ebb);
+        // From the top of the block, step through the instructions.
+        pos.goto_top(block);
 
         while let Some(inst) = pos.next_inst() {
             if let InstructionData::Trap {
@@ -67,6 +67,6 @@ pub fn emit_stackmaps(
             tracker.process_inst(inst, &pos.func.dfg, liveness);
             tracker.drop_dead(inst);
         }
-        curr = func.layout.next_ebb(ebb);
+        curr = func.layout.next_block(block);
     }
 }
diff --git a/cranelift/codegen/src/regalloc/solver.rs b/cranelift/codegen/src/regalloc/solver.rs
index 96ce702fdc..7416ec9bc7 100644
--- a/cranelift/codegen/src/regalloc/solver.rs
+++ b/cranelift/codegen/src/regalloc/solver.rs
@@ -34,20 +34,20 @@
 //! # Register diversions and global interference
 //!
 //! We can divert register values temporarily to satisfy constraints, but we need to put the
-//! values back into their originally assigned register locations before leaving the EBB.
-//! Otherwise, values won't be in the right register at the entry point of other EBBs.
+//! values back into their originally assigned register locations before leaving the block.
+//! Otherwise, values won't be in the right register at the entry point of other blocks.
 //!
 //! Some values are *local*, and we don't need to worry about putting those values back since they
-//! are not used in any other EBBs.
+//! are not used in any other blocks.
 //!
 //! When we assign register locations to defines, we are assigning both the register used locally
 //! immediately after the instruction and the register used globally when the defined value is used
-//! in a different EBB. We need to avoid interference both locally at the instruction and globally.
+//! in a different block. We need to avoid interference both locally at the instruction and globally.
 //!
 //! We have multiple mappings of values to registers:
 //!
 //! 1. The initial local mapping before the instruction. This includes any diversions from previous
-//!    instructions in the EBB, but not diversions for the current instruction.
+//!    instructions in the block, but not diversions for the current instruction.
 //! 2. The local mapping after applying the additional reassignments required to satisfy the
 //!    constraints of the current instruction.
 //! 3. The local mapping after the instruction. This excludes values killed by the instruction and
diff --git a/cranelift/codegen/src/regalloc/spilling.rs b/cranelift/codegen/src/regalloc/spilling.rs
index e7127b9606..d27c68ae42 100644
--- a/cranelift/codegen/src/regalloc/spilling.rs
+++ b/cranelift/codegen/src/regalloc/spilling.rs
@@ -17,7 +17,7 @@
 
 use crate::cursor::{Cursor, EncCursor};
 use crate::dominator_tree::DominatorTree;
-use crate::ir::{ArgumentLoc, Ebb, Function, Inst, InstBuilder, SigRef, Value, ValueLoc};
+use crate::ir::{ArgumentLoc, Block, Function, Inst, InstBuilder, SigRef, Value, ValueLoc};
 use crate::isa::registers::{RegClass, RegClassIndex, RegClassMask, RegUnit};
 use crate::isa::{ConstraintKind, EncInfo, RecipeConstraints, RegInfo, TargetIsa};
 use crate::regalloc::affinity::Affinity;
@@ -121,22 +121,22 @@ impl Spilling {
 
 impl<'a> Context<'a> {
     fn run(&mut self, tracker: &mut LiveValueTracker) {
-        self.topo.reset(self.cur.func.layout.ebbs());
-        while let Some(ebb) = self.topo.next(&self.cur.func.layout, self.domtree) {
-            self.visit_ebb(ebb, tracker);
+        self.topo.reset(self.cur.func.layout.blocks());
+        while let Some(block) = self.topo.next(&self.cur.func.layout, self.domtree) {
+            self.visit_block(block, tracker);
         }
     }
 
-    fn visit_ebb(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) {
-        debug!("Spilling {}:", ebb);
-        self.cur.goto_top(ebb);
-        self.visit_ebb_header(ebb, tracker);
+    fn visit_block(&mut self, block: Block, tracker: &mut LiveValueTracker) {
+        debug!("Spilling {}:", block);
+        self.cur.goto_top(block);
+        self.visit_block_header(block, tracker);
         tracker.drop_dead_params();
         self.process_spills(tracker);
 
         while let Some(inst) = self.cur.next_inst() {
             if !self.cur.func.dfg[inst].opcode().is_ghost() {
-                self.visit_inst(inst, ebb, tracker);
+                self.visit_inst(inst, block, tracker);
             } else {
                 let (_throughs, kills) = tracker.process_ghost(inst);
                 self.free_regs(kills);
@@ -185,9 +185,9 @@ impl<'a> Context<'a> {
         }
     }
 
-    fn visit_ebb_header(&mut self, ebb: Ebb, tracker: &mut LiveValueTracker) {
-        let (liveins, params) = tracker.ebb_top(
-            ebb,
+    fn visit_block_header(&mut self, block: Block, tracker: &mut LiveValueTracker) {
+        let (liveins, params) = tracker.block_top(
+            block,
             &self.cur.func.dfg,
             self.liveness,
             &self.cur.func.layout,
@@ -199,26 +199,26 @@ impl<'a> Context<'a> {
         self.pressure.reset();
         self.take_live_regs(liveins);
 
-        // An EBB can have an arbitrary (up to 2^16...) number of parameters, so they are not
+        // An block can have an arbitrary (up to 2^16...) number of parameters, so they are not
         // guaranteed to fit in registers.
         for lv in params {
             if let Affinity::Reg(rci) = lv.affinity {
                 let rc = self.reginfo.rc(rci);
                 'try_take: while let Err(mask) = self.pressure.take_transient(rc) {
-                    debug!("Need {} reg for EBB param {}", rc, lv.value);
+                    debug!("Need {} reg for block param {}", rc, lv.value);
                     match self.spill_candidate(mask, liveins) {
                         Some(cand) => {
                             debug!(
-                                "Spilling live-in {} to make room for {} EBB param {}",
+                                "Spilling live-in {} to make room for {} block param {}",
                                 cand, rc, lv.value
                             );
                             self.spill_reg(cand);
                         }
                         None => {
                             // We can't spill any of the live-in registers, so we have to spill an
-                            // EBB argument. Since the current spill metric would consider all the
-                            // EBB arguments equal, just spill the present register.
-                            debug!("Spilling {} EBB argument {}", rc, lv.value);
+                            // block argument. Since the current spill metric would consider all the
+                            // block arguments equal, just spill the present register.
+                            debug!("Spilling {} block argument {}", rc, lv.value);
 
                             // Since `spill_reg` will free a register, add the current one here.
                             self.pressure.take(rc);
@@ -230,15 +230,15 @@ impl<'a> Context<'a> {
             }
         }
 
-        // The transient pressure counts for the EBB arguments are accurate. Just preserve them.
+        // The transient pressure counts for the block arguments are accurate. Just preserve them.
         self.pressure.preserve_transient();
         self.free_dead_regs(params);
     }
 
-    fn visit_inst(&mut self, inst: Inst, ebb: Ebb, tracker: &mut LiveValueTracker) {
+    fn visit_inst(&mut self, inst: Inst, block: Block, tracker: &mut LiveValueTracker) {
         debug!("Inst {}, {}", self.cur.display_inst(inst), self.pressure);
         debug_assert_eq!(self.cur.current_inst(), Some(inst));
-        debug_assert_eq!(self.cur.current_ebb(), Some(ebb));
+        debug_assert_eq!(self.cur.current_block(), Some(block));
 
         let constraints = self
             .encinfo
@@ -246,7 +246,7 @@ impl<'a> Context<'a> {
 
         // We may need to resolve register constraints if there are any noteworthy uses.
         debug_assert!(self.reg_uses.is_empty());
-        self.collect_reg_uses(inst, ebb, constraints);
+        self.collect_reg_uses(inst, block, constraints);
 
         // Calls usually have fixed register uses.
         let call_sig = self.cur.func.dfg.call_signature(inst);
@@ -313,7 +313,12 @@ impl<'a> Context<'a> {
     // We are assuming here that if a value is used both by a fixed register operand and a register
     // class operand, they two are compatible. We are also assuming that two register class
     // operands are always compatible.
-    fn collect_reg_uses(&mut self, inst: Inst, ebb: Ebb, constraints: Option<&RecipeConstraints>) {
+    fn collect_reg_uses(
+        &mut self,
+        inst: Inst,
+        block: Block,
+        constraints: Option<&RecipeConstraints>,
+    ) {
         let args = self.cur.func.dfg.inst_args(inst);
         let num_fixed_ins = if let Some(constraints) = constraints {
             for (idx, (op, &arg)) in constraints.ins.iter().zip(args).enumerate() {
@@ -324,11 +329,11 @@ impl<'a> Context<'a> {
                     ConstraintKind::FixedReg(_) => reguse.fixed = true,
                     ConstraintKind::Tied(_) => {
                         // A tied operand must kill the used value.
-                        reguse.tied = !lr.killed_at(inst, ebb, &self.cur.func.layout);
+                        reguse.tied = !lr.killed_at(inst, block, &self.cur.func.layout);
                     }
                     ConstraintKind::FixedTied(_) => {
                         reguse.fixed = true;
-                        reguse.tied = !lr.killed_at(inst, ebb, &self.cur.func.layout);
+                        reguse.tied = !lr.killed_at(inst, block, &self.cur.func.layout);
                     }
                     ConstraintKind::Reg => {}
                 }
@@ -450,10 +455,10 @@ impl<'a> Context<'a> {
                     // Spill a live register that is *not* used by the current instruction.
                     // Spilling a use wouldn't help.
                     //
-                    // Do allow spilling of EBB arguments on branches. This is safe since we spill
-                    // the whole virtual register which includes the matching EBB parameter value
+                    // Do allow spilling of block arguments on branches. This is safe since we spill
+                    // the whole virtual register which includes the matching block parameter value
                     // at the branch destination. It is also necessary since there can be
-                    // arbitrarily many EBB arguments.
+                    // arbitrarily many block arguments.
                     match {
                         let args = if self.cur.func.dfg[inst].opcode().is_branch() {
                             self.cur.func.dfg.inst_fixed_args(inst)
@@ -572,7 +577,7 @@ impl<'a> Context<'a> {
         self.liveness.create_dead(copy, inst, Affinity::Reg(rci));
         self.liveness.extend_locally(
             copy,
-            self.cur.func.layout.pp_ebb(inst),
+            self.cur.func.layout.pp_block(inst),
             self.cur.current_inst().expect("must be at an instruction"),
             &self.cur.func.layout,
         );
diff --git a/cranelift/codegen/src/regalloc/virtregs.rs b/cranelift/codegen/src/regalloc/virtregs.rs
index 8e1f4a0276..28af9e22df 100644
--- a/cranelift/codegen/src/regalloc/virtregs.rs
+++ b/cranelift/codegen/src/regalloc/virtregs.rs
@@ -5,11 +5,11 @@
 //! output.
 //!
 //! A virtual register is typically built by merging together SSA values that are "phi-related" -
-//! that is, one value is passed as an EBB argument to a branch and the other is the EBB parameter
+//! that is, one value is passed as an block argument to a branch and the other is the block parameter
 //! value itself.
 //!
 //! If any values in a virtual register are spilled, they will use the same stack slot. This avoids
-//! memory-to-memory copies when a spilled value is passed as an EBB argument.
+//! memory-to-memory copies when a spilled value is passed as an block argument.
 
 use crate::dbg::DisplayList;
 use crate::dominator_tree::DominatorTreePreorder;
diff --git a/cranelift/codegen/src/simple_gvn.rs b/cranelift/codegen/src/simple_gvn.rs
index 60771d4cd8..5351aced43 100644
--- a/cranelift/codegen/src/simple_gvn.rs
+++ b/cranelift/codegen/src/simple_gvn.rs
@@ -59,7 +59,7 @@ pub fn do_simple_gvn(func: &mut Function, domtree: &mut DominatorTree) {
     let _tt = timing::gvn();
     debug_assert!(domtree.is_valid());
 
-    // Visit EBBs in a reverse post-order.
+    // Visit blocks in a reverse post-order.
     //
     // The RefCell here is a bit ugly since the HashKeys in the ScopedHashMap
     // need a reference to the function.
@@ -68,13 +68,13 @@ pub fn do_simple_gvn(func: &mut Function, domtree: &mut DominatorTree) {
     let mut visible_values: ScopedHashMap = ScopedHashMap::new();
     let mut scope_stack: Vec = Vec::new();
 
-    for &ebb in domtree.cfg_postorder().iter().rev() {
+    for &block in domtree.cfg_postorder().iter().rev() {
         {
             // Pop any scopes that we just exited.
             let layout = &pos.borrow().func.layout;
             loop {
                 if let Some(current) = scope_stack.last() {
-                    if domtree.dominates(*current, ebb, layout) {
+                    if domtree.dominates(*current, block, layout) {
                         break;
                     }
                 } else {
@@ -85,11 +85,11 @@ pub fn do_simple_gvn(func: &mut Function, domtree: &mut DominatorTree) {
             }
 
             // Push a scope for the current block.
-            scope_stack.push(layout.first_inst(ebb).unwrap());
+            scope_stack.push(layout.first_inst(block).unwrap());
             visible_values.increment_depth();
         }
 
-        pos.borrow_mut().goto_top(ebb);
+        pos.borrow_mut().goto_top(block);
         while let Some(inst) = {
             let mut pos = pos.borrow_mut();
             pos.next_inst()
diff --git a/cranelift/codegen/src/simple_preopt.rs b/cranelift/codegen/src/simple_preopt.rs
index fb91417131..c5a02caea4 100644
--- a/cranelift/codegen/src/simple_preopt.rs
+++ b/cranelift/codegen/src/simple_preopt.rs
@@ -14,7 +14,7 @@ use crate::ir::{
     immediates,
     instructions::{Opcode, ValueList},
     types::{I16, I32, I64, I8},
-    DataFlowGraph, Ebb, Function, Inst, InstBuilder, InstructionData, Type, Value,
+    Block, DataFlowGraph, Function, Inst, InstBuilder, InstructionData, Type, Value,
 };
 use crate::isa::TargetIsa;
 use crate::timing;
@@ -810,10 +810,10 @@ enum BranchOrderKind {
 
 /// Reorder branches to encourage fallthroughs.
 ///
-/// When an ebb ends with a conditional branch followed by an unconditional
-/// branch, this will reorder them if one of them is branching to the next Ebb
+/// When an block ends with a conditional branch followed by an unconditional
+/// branch, this will reorder them if one of them is branching to the next Block
 /// layout-wise. The unconditional jump can then become a fallthrough.
-fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst: Inst) {
+fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, block: Block, inst: Inst) {
     let (term_inst, term_inst_args, term_dest, cond_inst, cond_inst_args, cond_dest, kind) =
         match pos.func.dfg[inst] {
             InstructionData::Jump {
@@ -821,13 +821,13 @@ fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst
                 destination,
                 ref args,
             } => {
-                let next_ebb = if let Some(next_ebb) = pos.func.layout.next_ebb(ebb) {
-                    next_ebb
+                let next_block = if let Some(next_block) = pos.func.layout.next_block(block) {
+                    next_block
                 } else {
                     return;
                 };
 
-                if destination == next_ebb {
+                if destination == next_block {
                     return;
                 }
 
@@ -840,7 +840,7 @@ fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst
                 let prev_inst_data = &pos.func.dfg[prev_inst];
 
                 if let Some(prev_dest) = prev_inst_data.branch_destination() {
-                    if prev_dest != next_ebb {
+                    if prev_dest != next_block {
                         return;
                     }
                 } else {
@@ -941,7 +941,7 @@ fn branch_order(pos: &mut FuncCursor, cfg: &mut ControlFlowGraph, ebb: Ebb, inst
         }
     }
 
-    cfg.recompute_ebb(pos.func, ebb);
+    cfg.recompute_block(pos.func, block);
 }
 
 /// The main pre-opt pass.
@@ -949,7 +949,7 @@ pub fn do_preopt(func: &mut Function, cfg: &mut ControlFlowGraph, isa: &dyn Targ
     let _tt = timing::preopt();
     let mut pos = FuncCursor::new(func);
     let native_word_width = isa.pointer_bytes();
-    while let Some(ebb) = pos.next_ebb() {
+    while let Some(block) = pos.next_block() {
         while let Some(inst) = pos.next_inst() {
             // Apply basic simplifications.
             simplify(&mut pos, inst, native_word_width as u32);
@@ -961,7 +961,7 @@ pub fn do_preopt(func: &mut Function, cfg: &mut ControlFlowGraph, isa: &dyn Targ
             }
 
             branch_opt(&mut pos, inst);
-            branch_order(&mut pos, cfg, ebb, inst);
+            branch_order(&mut pos, cfg, block, inst);
         }
     }
 }
diff --git a/cranelift/codegen/src/topo_order.rs b/cranelift/codegen/src/topo_order.rs
index 647a8da941..8d38e4f324 100644
--- a/cranelift/codegen/src/topo_order.rs
+++ b/cranelift/codegen/src/topo_order.rs
@@ -1,28 +1,28 @@
-//! Topological order of EBBs, according to the dominator tree.
+//! Topological order of blocks, according to the dominator tree.
 
 use crate::dominator_tree::DominatorTree;
 use crate::entity::EntitySet;
-use crate::ir::{Ebb, Layout};
+use crate::ir::{Block, Layout};
 use alloc::vec::Vec;
 
-/// Present EBBs in a topological order such that all dominating EBBs are guaranteed to be visited
-/// before the current EBB.
+/// Present blocks in a topological order such that all dominating blocks are guaranteed to be visited
+/// before the current block.
 ///
-/// There are many topological orders of the EBBs in a function, so it is possible to provide a
-/// preferred order, and the `TopoOrder` will present EBBs in an order that is as close as possible
+/// There are many topological orders of the blocks in a function, so it is possible to provide a
+/// preferred order, and the `TopoOrder` will present blocks in an order that is as close as possible
 /// to the preferred order.
 pub struct TopoOrder {
-    /// Preferred order of EBBs to visit.
-    preferred: Vec,
+    /// Preferred order of blocks to visit.
+    preferred: Vec,
 
     /// Next entry to get from `preferred`.
     next: usize,
 
-    /// Set of visited EBBs.
-    visited: EntitySet,
+    /// Set of visited blocks.
+    visited: EntitySet,
 
-    /// Stack of EBBs to be visited next, already in `visited`.
-    stack: Vec,
+    /// Stack of blocks to be visited next, already in `visited`.
+    stack: Vec,
 }
 
 impl TopoOrder {
@@ -44,11 +44,11 @@ impl TopoOrder {
         self.stack.clear();
     }
 
-    /// Reset and initialize with a preferred sequence of EBBs. The resulting topological order is
-    /// guaranteed to contain all of the EBBs in `preferred` as well as any dominators.
-    pub fn reset(&mut self, preferred: Ebbs)
+    /// Reset and initialize with a preferred sequence of blocks. The resulting topological order is
+    /// guaranteed to contain all of the blocks in `preferred` as well as any dominators.
+    pub fn reset(&mut self, preferred: Blocks)
     where
-        Ebbs: IntoIterator,
+        Blocks: IntoIterator,
     {
         self.preferred.clear();
         self.preferred.extend(preferred);
@@ -57,27 +57,29 @@ impl TopoOrder {
         self.stack.clear();
     }
 
-    /// Get the next EBB in the topological order.
+    /// Get the next block in the topological order.
     ///
-    /// Two things are guaranteed about the EBBs returned by this function:
+    /// Two things are guaranteed about the blocks returned by this function:
     ///
-    /// - All EBBs in the `preferred` iterator given to `reset` will be returned.
-    /// - All dominators are visited before the EBB returned.
-    pub fn next(&mut self, layout: &Layout, domtree: &DominatorTree) -> Option {
-        self.visited.resize(layout.ebb_capacity());
+    /// - All blocks in the `preferred` iterator given to `reset` will be returned.
+    /// - All dominators are visited before the block returned.
+    pub fn next(&mut self, layout: &Layout, domtree: &DominatorTree) -> Option {
+        self.visited.resize(layout.block_capacity());
         // Any entries in `stack` should be returned immediately. They have already been added to
         // `visited`.
         while self.stack.is_empty() {
             match self.preferred.get(self.next).cloned() {
                 None => return None,
-                Some(mut ebb) => {
-                    // We have the next EBB in the preferred order.
+                Some(mut block) => {
+                    // We have the next block in the preferred order.
                     self.next += 1;
                     // Push it along with any non-visited dominators.
-                    while self.visited.insert(ebb) {
-                        self.stack.push(ebb);
-                        match domtree.idom(ebb) {
-                            Some(idom) => ebb = layout.inst_ebb(idom).expect("idom not in layout"),
+                    while self.visited.insert(block) {
+                        self.stack.push(block);
+                        match domtree.idom(block) {
+                            Some(idom) => {
+                                block = layout.inst_block(idom).expect("idom not in layout")
+                            }
                             None => break,
                         }
                     }
@@ -105,32 +107,32 @@ mod tests {
         let mut topo = TopoOrder::new();
 
         assert_eq!(topo.next(&func.layout, &domtree), None);
-        topo.reset(func.layout.ebbs());
+        topo.reset(func.layout.blocks());
         assert_eq!(topo.next(&func.layout, &domtree), None);
     }
 
     #[test]
     fn simple() {
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb1 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
+        let block1 = func.dfg.make_block();
 
         {
             let mut cur = FuncCursor::new(&mut func);
 
-            cur.insert_ebb(ebb0);
-            cur.ins().jump(ebb1, &[]);
-            cur.insert_ebb(ebb1);
-            cur.ins().jump(ebb1, &[]);
+            cur.insert_block(block0);
+            cur.ins().jump(block1, &[]);
+            cur.insert_block(block1);
+            cur.ins().jump(block1, &[]);
         }
 
         let cfg = ControlFlowGraph::with_function(&func);
         let domtree = DominatorTree::with_function(&func, &cfg);
         let mut topo = TopoOrder::new();
 
-        topo.reset(iter::once(ebb1));
-        assert_eq!(topo.next(&func.layout, &domtree), Some(ebb0));
-        assert_eq!(topo.next(&func.layout, &domtree), Some(ebb1));
+        topo.reset(iter::once(block1));
+        assert_eq!(topo.next(&func.layout, &domtree), Some(block0));
+        assert_eq!(topo.next(&func.layout, &domtree), Some(block1));
         assert_eq!(topo.next(&func.layout, &domtree), None);
     }
 }
diff --git a/cranelift/codegen/src/unreachable_code.rs b/cranelift/codegen/src/unreachable_code.rs
index 95e2955245..63e3e230f8 100644
--- a/cranelift/codegen/src/unreachable_code.rs
+++ b/cranelift/codegen/src/unreachable_code.rs
@@ -9,7 +9,7 @@ use log::debug;
 
 /// Eliminate unreachable code.
 ///
-/// This pass deletes whole EBBs that can't be reached from the entry block. It does not delete
+/// This pass deletes whole blocks that can't be reached from the entry block. It does not delete
 /// individual instructions whose results are unused.
 ///
 /// The reachability analysis is performed by the dominator tree analysis.
@@ -20,27 +20,27 @@ pub fn eliminate_unreachable_code(
 ) {
     let _tt = timing::unreachable_code();
     let mut pos = FuncCursor::new(func);
-    while let Some(ebb) = pos.next_ebb() {
-        if domtree.is_reachable(ebb) {
+    while let Some(block) = pos.next_block() {
+        if domtree.is_reachable(block) {
             continue;
         }
 
-        debug!("Eliminating unreachable {}", ebb);
+        debug!("Eliminating unreachable {}", block);
         // Move the cursor out of the way and make sure the next lop iteration goes to the right
-        // EBB.
-        pos.prev_ebb();
+        // block.
+        pos.prev_block();
 
-        // Remove all instructions from `ebb`.
-        while let Some(inst) = pos.func.layout.first_inst(ebb) {
+        // Remove all instructions from `block`.
+        while let Some(inst) = pos.func.layout.first_inst(block) {
             debug!(" - {}", pos.func.dfg.display_inst(inst, None));
             pos.func.layout.remove_inst(inst);
         }
 
-        // Once the EBB is completely empty, we can update the CFG which removes it from any
+        // Once the block is completely empty, we can update the CFG which removes it from any
         // predecessor lists.
-        cfg.recompute_ebb(pos.func, ebb);
+        cfg.recompute_block(pos.func, block);
 
-        // Finally, remove the EBB from the layout.
-        pos.func.layout.remove_ebb(ebb);
+        // Finally, remove the block from the layout.
+        pos.func.layout.remove_block(block);
     }
 }
diff --git a/cranelift/codegen/src/value_label.rs b/cranelift/codegen/src/value_label.rs
index aab8792536..94e5c58171 100644
--- a/cranelift/codegen/src/value_label.rs
+++ b/cranelift/codegen/src/value_label.rs
@@ -93,8 +93,8 @@ where
 {
     let values_labels = build_value_labels_index::(func);
 
-    let mut ebbs = func.layout.ebbs().collect::>();
-    ebbs.sort_by_key(|ebb| func.offsets[*ebb]); // Ensure inst offsets always increase
+    let mut blocks = func.layout.blocks().collect::>();
+    blocks.sort_by_key(|block| func.offsets[*block]); // Ensure inst offsets always increase
     let encinfo = isa.encoding_info();
     let values_locations = &func.locations;
     let liveness_ranges = regalloc.liveness().ranges();
@@ -117,16 +117,16 @@ where
     let mut end_offset = 0;
     let mut tracked_values: Vec<(Value, ValueLabel, u32, ValueLoc)> = Vec::new();
     let mut divert = RegDiversions::new();
-    for ebb in ebbs {
-        divert.at_ebb(&func.entry_diversions, ebb);
+    for block in blocks {
+        divert.at_block(&func.entry_diversions, block);
         let mut last_srcloc: Option = None;
-        for (offset, inst, size) in func.inst_offsets(ebb, &encinfo) {
+        for (offset, inst, size) in func.inst_offsets(block, &encinfo) {
             divert.apply(&func.dfg[inst]);
             end_offset = offset + size;
             // Remove killed values.
             tracked_values.retain(|(x, label, start_offset, last_loc)| {
                 let range = liveness_ranges.get(*x);
-                if range.expect("value").killed_at(inst, ebb, &func.layout) {
+                if range.expect("value").killed_at(inst, block, &func.layout) {
                     add_range(*label, (*start_offset, end_offset), *last_loc);
                     return false;
                 }
@@ -173,7 +173,7 @@ where
                 // Ignore dead/inactive Values.
                 let range = liveness_ranges.get(*v);
                 match range {
-                    Some(r) => r.reaches_use(inst, ebb, &func.layout),
+                    Some(r) => r.reaches_use(inst, block, &func.layout),
                     None => false,
                 }
             });
diff --git a/cranelift/codegen/src/verifier/cssa.rs b/cranelift/codegen/src/verifier/cssa.rs
index 54e88dccf3..f1ff72597a 100644
--- a/cranelift/codegen/src/verifier/cssa.rs
+++ b/cranelift/codegen/src/verifier/cssa.rs
@@ -2,7 +2,7 @@
 
 use crate::dbg::DisplayList;
 use crate::dominator_tree::{DominatorTree, DominatorTreePreorder};
-use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::flowgraph::{BlockPredecessor, ControlFlowGraph};
 use crate::ir::{ExpandedProgramPoint, Function};
 use crate::regalloc::liveness::Liveness;
 use crate::regalloc::virtregs::VirtRegs;
@@ -13,7 +13,7 @@ use crate::verifier::{VerifierErrors, VerifierStepResult};
 ///
 /// Conventional SSA form is represented in Cranelift with the help of virtual registers:
 ///
-/// - Two values are said to be *PHI-related* if one is an EBB argument and the other is passed as
+/// - Two values are said to be *PHI-related* if one is an block argument and the other is passed as
 ///   a branch argument in a location that matches the first value.
 /// - PHI-related values must belong to the same virtual register.
 /// - Two values in the same virtual register must not have overlapping live ranges.
@@ -76,10 +76,10 @@ impl<'a> CssaVerifier<'a> {
 
                 // Check topological ordering with the previous values in the virtual register.
                 let def: ExpandedProgramPoint = self.func.dfg.value_def(val).into();
-                let def_ebb = self.func.layout.pp_ebb(def);
+                let def_block = self.func.layout.pp_block(def);
                 for &prev_val in &values[0..idx] {
                     let prev_def: ExpandedProgramPoint = self.func.dfg.value_def(prev_val).into();
-                    let prev_ebb = self.func.layout.pp_ebb(prev_def);
+                    let prev_block = self.func.layout.pp_block(prev_def);
 
                     if prev_def == def {
                         return errors.fatal((
@@ -95,7 +95,7 @@ impl<'a> CssaVerifier<'a> {
                     }
 
                     // Enforce topological ordering of defs in the virtual register.
-                    if self.preorder.dominates(def_ebb, prev_ebb)
+                    if self.preorder.dominates(def_block, prev_block)
                         && self.domtree.dominates(def, prev_def, &self.func.layout)
                     {
                         return errors.fatal((
@@ -115,12 +115,12 @@ impl<'a> CssaVerifier<'a> {
                 // We only have to check against the nearest dominating value.
                 for &prev_val in values[0..idx].iter().rev() {
                     let prev_def: ExpandedProgramPoint = self.func.dfg.value_def(prev_val).into();
-                    let prev_ebb = self.func.layout.pp_ebb(prev_def);
+                    let prev_block = self.func.layout.pp_block(prev_def);
 
-                    if self.preorder.dominates(prev_ebb, def_ebb)
+                    if self.preorder.dominates(prev_block, def_block)
                         && self.domtree.dominates(prev_def, def, &self.func.layout)
                     {
-                        if self.liveness[prev_val].overlaps_def(def, def_ebb, &self.func.layout) {
+                        if self.liveness[prev_val].overlaps_def(def, def_block, &self.func.layout) {
                             return errors.fatal((
                                 val,
                                 format!(
@@ -142,24 +142,24 @@ impl<'a> CssaVerifier<'a> {
     }
 
     fn check_cssa(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
-        for ebb in self.func.layout.ebbs() {
-            let ebb_params = self.func.dfg.ebb_params(ebb);
-            for BasicBlock { inst: pred, .. } in self.cfg.pred_iter(ebb) {
+        for block in self.func.layout.blocks() {
+            let block_params = self.func.dfg.block_params(block);
+            for BlockPredecessor { inst: pred, .. } in self.cfg.pred_iter(block) {
                 let pred_args = self.func.dfg.inst_variable_args(pred);
                 // This should have been caught by an earlier verifier pass.
                 assert_eq!(
-                    ebb_params.len(),
+                    block_params.len(),
                     pred_args.len(),
                     "Wrong arguments on branch."
                 );
 
-                for (&ebb_param, &pred_arg) in ebb_params.iter().zip(pred_args) {
-                    if !self.virtregs.same_class(ebb_param, pred_arg) {
+                for (&block_param, &pred_arg) in block_params.iter().zip(pred_args) {
+                    if !self.virtregs.same_class(block_param, pred_arg) {
                         return errors.fatal((
                             pred,
                             format!(
                                 "{} and {} must be in the same virtual register",
-                                ebb_param, pred_arg
+                                block_param, pred_arg
                             ),
                         ));
                     }
diff --git a/cranelift/codegen/src/verifier/flags.rs b/cranelift/codegen/src/verifier/flags.rs
index 1748fccf71..1a20303d20 100644
--- a/cranelift/codegen/src/verifier/flags.rs
+++ b/cranelift/codegen/src/verifier/flags.rs
@@ -1,7 +1,7 @@
 //! Verify CPU flags values.
 
 use crate::entity::{EntitySet, SecondaryMap};
-use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::flowgraph::{BlockPredecessor, ControlFlowGraph};
 use crate::ir;
 use crate::ir::instructions::BranchInfo;
 use crate::isa;
@@ -42,33 +42,33 @@ struct FlagsVerifier<'a> {
     cfg: &'a ControlFlowGraph,
     encinfo: Option,
 
-    /// The single live-in flags value (if any) for each EBB.
-    livein: SecondaryMap>,
+    /// The single live-in flags value (if any) for each block.
+    livein: SecondaryMap>,
 }
 
 impl<'a> FlagsVerifier<'a> {
     fn check(&mut self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
-        // List of EBBs that need to be processed. EBBs may be re-added to this list when we detect
+        // List of blocks that need to be processed. blocks may be re-added to this list when we detect
         // that one of their successor blocks needs a live-in flags value.
-        let mut worklist = EntitySet::with_capacity(self.func.layout.ebb_capacity());
-        for ebb in self.func.layout.ebbs() {
-            worklist.insert(ebb);
+        let mut worklist = EntitySet::with_capacity(self.func.layout.block_capacity());
+        for block in self.func.layout.blocks() {
+            worklist.insert(block);
         }
 
-        while let Some(ebb) = worklist.pop() {
-            if let Some(value) = self.visit_ebb(ebb, errors)? {
-                // The EBB has live-in flags. Check if the value changed.
-                match self.livein[ebb].expand() {
-                    // Revisit any predecessor blocks the first time we see a live-in for `ebb`.
+        while let Some(block) = worklist.pop() {
+            if let Some(value) = self.visit_block(block, errors)? {
+                // The block has live-in flags. Check if the value changed.
+                match self.livein[block].expand() {
+                    // Revisit any predecessor blocks the first time we see a live-in for `block`.
                     None => {
-                        self.livein[ebb] = value.into();
-                        for BasicBlock { ebb: pred, .. } in self.cfg.pred_iter(ebb) {
+                        self.livein[block] = value.into();
+                        for BlockPredecessor { block: pred, .. } in self.cfg.pred_iter(block) {
                             worklist.insert(pred);
                         }
                     }
                     Some(old) if old != value => {
                         return errors.fatal((
-                            ebb,
+                            block,
                             format!("conflicting live-in CPU flags: {} and {}", old, value),
                         ));
                     }
@@ -76,24 +76,24 @@ impl<'a> FlagsVerifier<'a> {
                 }
             } else {
                 // Existing live-in flags should never be able to disappear.
-                assert_eq!(self.livein[ebb].expand(), None);
+                assert_eq!(self.livein[block].expand(), None);
             }
         }
 
         Ok(())
     }
 
-    /// Check flags usage in `ebb` and return the live-in flags value, if any.
-    fn visit_ebb(
+    /// Check flags usage in `block` and return the live-in flags value, if any.
+    fn visit_block(
         &self,
-        ebb: ir::Ebb,
+        block: ir::Block,
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult> {
         // The single currently live flags value.
         let mut live_val = None;
 
         // Visit instructions backwards so we can track liveness accurately.
-        for inst in self.func.layout.ebb_insts(ebb).rev() {
+        for inst in self.func.layout.block_insts(block).rev() {
             // Check if `inst` interferes with existing live flags.
             if let Some(live) = live_val {
                 for &res in self.func.dfg.inst_results(inst) {
@@ -130,7 +130,7 @@ impl<'a> FlagsVerifier<'a> {
                 }
             }
 
-            // Include live-in flags to successor EBBs.
+            // Include live-in flags to successor blocks.
             match self.func.dfg.analyze_branch(inst) {
                 BranchInfo::NotABranch => {}
                 BranchInfo::SingleDest(dest, _) => {
diff --git a/cranelift/codegen/src/verifier/liveness.rs b/cranelift/codegen/src/verifier/liveness.rs
index 4c4940f356..921babc6a0 100644
--- a/cranelift/codegen/src/verifier/liveness.rs
+++ b/cranelift/codegen/src/verifier/liveness.rs
@@ -1,6 +1,6 @@
 //! Liveness verifier.
 
-use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::flowgraph::{BlockPredecessor, ControlFlowGraph};
 use crate::ir::entities::AnyEntity;
 use crate::ir::{ExpandedProgramPoint, Function, ProgramPoint, Value};
 use crate::isa::TargetIsa;
@@ -16,7 +16,7 @@ use crate::verifier::{VerifierErrors, VerifierStepResult};
 /// - All values in the program must have a live range.
 /// - The live range def point must match where the value is defined.
 /// - The live range must reach all uses.
-/// - When a live range is live-in to an EBB, it must be live at all the predecessors.
+/// - When a live range is live-in to an block, it must be live at all the predecessors.
 /// - The live range affinity must be compatible with encoding constraints.
 ///
 /// We don't verify that live ranges are minimal. This would require recomputing live ranges for
@@ -35,7 +35,7 @@ pub fn verify_liveness(
         cfg,
         liveness,
     };
-    verifier.check_ebbs(errors)?;
+    verifier.check_blocks(errors)?;
     verifier.check_insts(errors)?;
     Ok(())
 }
@@ -48,17 +48,18 @@ struct LivenessVerifier<'a> {
 }
 
 impl<'a> LivenessVerifier<'a> {
-    /// Check all EBB arguments.
-    fn check_ebbs(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
-        for ebb in self.func.layout.ebbs() {
-            for &val in self.func.dfg.ebb_params(ebb) {
+    /// Check all block arguments.
+    fn check_blocks(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
+        for block in self.func.layout.blocks() {
+            for &val in self.func.dfg.block_params(block) {
                 let lr = match self.liveness.get(val) {
                     Some(lr) => lr,
                     None => {
-                        return errors.fatal((ebb, format!("EBB arg {} has no live range", val)))
+                        return errors
+                            .fatal((block, format!("block arg {} has no live range", val)))
                     }
                 };
-                self.check_lr(ebb.into(), val, lr, errors)?;
+                self.check_lr(block.into(), val, lr, errors)?;
             }
         }
         Ok(())
@@ -66,8 +67,8 @@ impl<'a> LivenessVerifier<'a> {
 
     /// Check all instructions.
     fn check_insts(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
-        for ebb in self.func.layout.ebbs() {
-            for inst in self.func.layout.ebb_insts(ebb) {
+        for block in self.func.layout.blocks() {
+            for inst in self.func.layout.block_insts(block) {
                 let encoding = self.func.encodings[inst];
 
                 // Check the defs.
@@ -110,8 +111,8 @@ impl<'a> LivenessVerifier<'a> {
                         None => return errors.fatal((inst, format!("{} has no live range", val))),
                     };
 
-                    debug_assert!(self.func.layout.inst_ebb(inst).unwrap() == ebb);
-                    if !lr.reaches_use(inst, ebb, &self.func.layout) {
+                    debug_assert!(self.func.layout.inst_block(inst).unwrap() == block);
+                    if !lr.reaches_use(inst, block, &self.func.layout) {
                         return errors.fatal((inst, format!("{} is not live at this use", val)));
                     }
 
@@ -143,7 +144,7 @@ impl<'a> LivenessVerifier<'a> {
         let l = &self.func.layout;
 
         let loc: AnyEntity = match def.into() {
-            ExpandedProgramPoint::Ebb(e) => e.into(),
+            ExpandedProgramPoint::Block(e) => e.into(),
             ExpandedProgramPoint::Inst(i) => i.into(),
         };
         if lr.def() != def {
@@ -159,66 +160,70 @@ impl<'a> LivenessVerifier<'a> {
                 return Ok(());
             }
         }
-        let def_ebb = match def.into() {
-            ExpandedProgramPoint::Ebb(e) => e,
-            ExpandedProgramPoint::Inst(i) => l.inst_ebb(i).unwrap(),
+        let def_block = match def.into() {
+            ExpandedProgramPoint::Block(e) => e,
+            ExpandedProgramPoint::Inst(i) => l.inst_block(i).unwrap(),
         };
         match lr.def_local_end().into() {
-            ExpandedProgramPoint::Ebb(e) => {
+            ExpandedProgramPoint::Block(e) => {
                 return errors.fatal((
                     loc,
                     format!("Def local range for {} can't end at {}", val, e),
                 ));
             }
             ExpandedProgramPoint::Inst(i) => {
-                if self.func.layout.inst_ebb(i) != Some(def_ebb) {
-                    return errors.fatal((loc, format!("Def local end for {} in wrong ebb", val)));
+                if self.func.layout.inst_block(i) != Some(def_block) {
+                    return errors
+                        .fatal((loc, format!("Def local end for {} in wrong block", val)));
                 }
             }
         }
 
         // Now check the live-in intervals against the CFG.
-        for (mut ebb, end) in lr.liveins() {
-            if !l.is_ebb_inserted(ebb) {
+        for (mut block, end) in lr.liveins() {
+            if !l.is_block_inserted(block) {
                 return errors.fatal((
                     loc,
-                    format!("{} livein at {} which is not in the layout", val, ebb),
+                    format!("{} livein at {} which is not in the layout", val, block),
                 ));
             }
-            let end_ebb = match l.inst_ebb(end) {
+            let end_block = match l.inst_block(end) {
                 Some(e) => e,
                 None => {
                     return errors.fatal((
                         loc,
                         format!(
                             "{} livein for {} ends at {} which is not in the layout",
-                            val, ebb, end
+                            val, block, end
                         ),
                     ));
                 }
             };
 
-            // Check all the EBBs in the interval independently.
+            // Check all the blocks in the interval independently.
             loop {
-                // If `val` is live-in at `ebb`, it must be live at all the predecessors.
-                for BasicBlock { inst: pred, ebb } in self.cfg.pred_iter(ebb) {
-                    if !lr.reaches_use(pred, ebb, &self.func.layout) {
+                // If `val` is live-in at `block`, it must be live at all the predecessors.
+                for BlockPredecessor { inst: pred, block } in self.cfg.pred_iter(block) {
+                    if !lr.reaches_use(pred, block, &self.func.layout) {
                         return errors.fatal((
                             pred,
-                            format!("{} is live in to {} but not live at predecessor", val, ebb),
+                            format!(
+                                "{} is live in to {} but not live at predecessor",
+                                val, block
+                            ),
                         ));
                     }
                 }
 
-                if ebb == end_ebb {
+                if block == end_block {
                     break;
                 }
-                ebb = match l.next_ebb(ebb) {
+                block = match l.next_block(block) {
                     Some(e) => e,
                     None => {
                         return errors.fatal((
                             loc,
-                            format!("end of {} livein ({}) never reached", val, end_ebb),
+                            format!("end of {} livein ({}) never reached", val, end_block),
                         ));
                     }
                 };
diff --git a/cranelift/codegen/src/verifier/locations.rs b/cranelift/codegen/src/verifier/locations.rs
index fe180b8e81..287413a412 100644
--- a/cranelift/codegen/src/verifier/locations.rs
+++ b/cranelift/codegen/src/verifier/locations.rs
@@ -15,7 +15,7 @@ use crate::verifier::{VerifierErrors, VerifierStepResult};
 /// instruction encoding recipes.
 ///
 /// Values can be temporarily diverted to a different location by using the `regmove`, `regspill`,
-/// and `regfill` instructions, but only inside an EBB.
+/// and `regfill` instructions, but only inside an block.
 ///
 /// If a liveness analysis is provided, it is used to verify that there are no active register
 /// diversions across control flow edges.
@@ -54,11 +54,11 @@ impl<'a> LocationVerifier<'a> {
         let dfg = &self.func.dfg;
         let mut divert = RegDiversions::new();
 
-        for ebb in self.func.layout.ebbs() {
-            divert.at_ebb(&self.func.entry_diversions, ebb);
+        for block in self.func.layout.blocks() {
+            divert.at_block(&self.func.entry_diversions, block);
 
             let mut is_after_branch = false;
-            for inst in self.func.layout.ebb_insts(ebb) {
+            for inst in self.func.layout.block_insts(block) {
                 let enc = self.func.encodings[inst];
 
                 if enc.is_legal() {
@@ -332,24 +332,24 @@ impl<'a> LocationVerifier<'a> {
                 "No branch information for {}",
                 dfg.display_inst(inst, self.isa)
             ),
-            SingleDest(ebb, _) => {
-                let unique_predecessor = self.cfg.pred_iter(ebb).count() == 1;
+            SingleDest(block, _) => {
+                let unique_predecessor = self.cfg.pred_iter(block).count() == 1;
                 let mut val_to_remove = vec![];
                 for (&value, d) in divert.iter() {
                     let lr = &liveness[value];
                     if is_after_branch && unique_predecessor {
                         // Forward diversions based on the targeted branch.
-                        if !lr.is_livein(ebb, &self.func.layout) {
+                        if !lr.is_livein(block, &self.func.layout) {
                             val_to_remove.push(value)
                         }
-                    } else if lr.is_livein(ebb, &self.func.layout) {
+                    } else if lr.is_livein(block, &self.func.layout) {
                         return errors.fatal((
                             inst,
                             format!(
                                 "SingleDest: {} is diverted to {} and live in to {}",
                                 value,
                                 d.to.display(&self.reginfo),
-                                ebb,
+                                block,
                             ),
                         ));
                     }
@@ -358,34 +358,34 @@ impl<'a> LocationVerifier<'a> {
                     for val in val_to_remove.into_iter() {
                         divert.remove(val);
                     }
-                    debug_assert!(divert.check_ebb_entry(&self.func.entry_diversions, ebb));
+                    debug_assert!(divert.check_block_entry(&self.func.entry_diversions, block));
                 }
             }
-            Table(jt, ebb) => {
+            Table(jt, block) => {
                 for (&value, d) in divert.iter() {
                     let lr = &liveness[value];
-                    if let Some(ebb) = ebb {
-                        if lr.is_livein(ebb, &self.func.layout) {
+                    if let Some(block) = block {
+                        if lr.is_livein(block, &self.func.layout) {
                             return errors.fatal((
                                 inst,
                                 format!(
                                     "Table.default: {} is diverted to {} and live in to {}",
                                     value,
                                     d.to.display(&self.reginfo),
-                                    ebb,
+                                    block,
                                 ),
                             ));
                         }
                     }
-                    for ebb in self.func.jump_tables[jt].iter() {
-                        if lr.is_livein(*ebb, &self.func.layout) {
+                    for block in self.func.jump_tables[jt].iter() {
+                        if lr.is_livein(*block, &self.func.layout) {
                             return errors.fatal((
                                 inst,
                                 format!(
                                     "Table.case: {} is diverted to {} and live in to {}",
                                     value,
                                     d.to.display(&self.reginfo),
-                                    ebb,
+                                    block,
                                 ),
                             ));
                         }
diff --git a/cranelift/codegen/src/verifier/mod.rs b/cranelift/codegen/src/verifier/mod.rs
index 9833a8c0f5..58dbe259c8 100644
--- a/cranelift/codegen/src/verifier/mod.rs
+++ b/cranelift/codegen/src/verifier/mod.rs
@@ -1,40 +1,40 @@
 //! A verifier for ensuring that functions are well formed.
 //! It verifies:
 //!
-//! EBB integrity
+//! block integrity
 //!
-//! - All instructions reached from the `ebb_insts` iterator must belong to
-//!   the EBB as reported by `inst_ebb()`.
-//! - Every EBB must end in a terminator instruction, and no other instruction
+//! - All instructions reached from the `block_insts` iterator must belong to
+//!   the block as reported by `inst_block()`.
+//! - Every block must end in a terminator instruction, and no other instruction
 //!   can be a terminator.
-//! - Every value in the `ebb_params` iterator belongs to the EBB as reported by `value_ebb`.
+//! - Every value in the `block_params` iterator belongs to the block as reported by `value_block`.
 //!
 //! Instruction integrity
 //!
 //! - The instruction format must match the opcode.
 //! - All result values must be created for multi-valued instructions.
-//! - All referenced entities must exist. (Values, EBBs, stack slots, ...)
+//! - All referenced entities must exist. (Values, blocks, stack slots, ...)
 //! - Instructions must not reference (eg. branch to) the entry block.
 //!
 //! SSA form
 //!
 //! - Values must be defined by an instruction that exists and that is inserted in
-//!   an EBB, or be an argument of an existing EBB.
+//!   an block, or be an argument of an existing block.
 //! - Values used by an instruction must dominate the instruction.
 //!
 //! Control flow graph and dominator tree integrity:
 //!
-//! - All predecessors in the CFG must be branches to the EBB.
-//! - All branches to an EBB must be present in the CFG.
+//! - All predecessors in the CFG must be branches to the block.
+//! - All branches to an block must be present in the CFG.
 //! - A recomputed dominator tree is identical to the existing one.
 //!
 //! Type checking
 //!
 //! - Compare input and output values against the opcode's type constraints.
 //!   For polymorphic opcodes, determine the controlling type variable first.
-//! - Branches and jumps must pass arguments to destination EBBs that match the
+//! - Branches and jumps must pass arguments to destination blocks that match the
 //!   expected types exactly. The number of arguments must match.
-//! - All EBBs in a jump table must take no arguments.
+//! - All blocks in a jump table must take no arguments.
 //! - Function calls are type checked against their signature.
 //! - The entry block must take arguments that match the signature of the current
 //!   function.
@@ -60,12 +60,12 @@ use self::flags::verify_flags;
 use crate::dbg::DisplayList;
 use crate::dominator_tree::DominatorTree;
 use crate::entity::SparseSet;
-use crate::flowgraph::{BasicBlock, ControlFlowGraph};
+use crate::flowgraph::{BlockPredecessor, ControlFlowGraph};
 use crate::ir;
 use crate::ir::entities::AnyEntity;
 use crate::ir::instructions::{BranchInfo, CallInfo, InstructionFormat, ResolvedConstraint};
 use crate::ir::{
-    types, ArgumentLoc, Ebb, FuncRef, Function, GlobalValue, Inst, InstructionData, JumpTable,
+    types, ArgumentLoc, Block, FuncRef, Function, GlobalValue, Inst, InstructionData, JumpTable,
     Opcode, SigRef, StackSlot, StackSlotKind, Type, Value, ValueDef, ValueList, ValueLoc,
 };
 use crate::isa::TargetIsa;
@@ -495,30 +495,30 @@ impl<'a> Verifier<'a> {
 
     fn verify_jump_tables(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
         for (jt, jt_data) in &self.func.jump_tables {
-            for &ebb in jt_data.iter() {
-                self.verify_ebb(jt, ebb, errors)?;
+            for &block in jt_data.iter() {
+                self.verify_block(jt, block, errors)?;
             }
         }
         Ok(())
     }
 
-    /// Check that the given EBB can be encoded as a BB, by checking that only
-    /// branching instructions are ending the EBB.
-    fn encodable_as_bb(&self, ebb: Ebb, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
-        match self.func.is_ebb_basic(ebb) {
+    /// Check that the given block can be encoded as a BB, by checking that only
+    /// branching instructions are ending the block.
+    fn encodable_as_bb(&self, block: Block, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
+        match self.func.is_block_basic(block) {
             Ok(()) => Ok(()),
             Err((inst, message)) => errors.fatal((inst, self.context(inst), message)),
         }
     }
 
-    fn ebb_integrity(
+    fn block_integrity(
         &self,
-        ebb: Ebb,
+        block: Block,
         inst: Inst,
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         let is_terminator = self.func.dfg[inst].opcode().is_terminator();
-        let is_last_inst = self.func.layout.last_inst(ebb) == Some(inst);
+        let is_last_inst = self.func.layout.last_inst(block) == Some(inst);
 
         if is_terminator && !is_last_inst {
             // Terminating instructions only occur at the end of blocks.
@@ -527,30 +527,30 @@ impl<'a> Verifier<'a> {
                 self.context(inst),
                 format!(
                     "a terminator instruction was encountered before the end of {}",
-                    ebb
+                    block
                 ),
             ));
         }
         if is_last_inst && !is_terminator {
-            return errors.fatal((ebb, "block does not end in a terminator instruction"));
+            return errors.fatal((block, "block does not end in a terminator instruction"));
         }
 
-        // Instructions belong to the correct ebb.
-        let inst_ebb = self.func.layout.inst_ebb(inst);
-        if inst_ebb != Some(ebb) {
+        // Instructions belong to the correct block.
+        let inst_block = self.func.layout.inst_block(inst);
+        if inst_block != Some(block) {
             return errors.fatal((
                 inst,
                 self.context(inst),
-                format!("should belong to {} not {:?}", ebb, inst_ebb),
+                format!("should belong to {} not {:?}", block, inst_block),
             ));
         }
 
-        // Parameters belong to the correct ebb.
-        for &arg in self.func.dfg.ebb_params(ebb) {
+        // Parameters belong to the correct block.
+        for &arg in self.func.dfg.block_params(block) {
             match self.func.dfg.value_def(arg) {
-                ValueDef::Param(arg_ebb, _) => {
-                    if ebb != arg_ebb {
-                        return errors.fatal((arg, format!("does not belong to {}", ebb)));
+                ValueDef::Param(arg_block, _) => {
+                    if block != arg_block {
+                        return errors.fatal((arg, format!("does not belong to {}", block)));
                     }
                 }
                 _ => {
@@ -656,13 +656,13 @@ impl<'a> Verifier<'a> {
                 ref args,
                 ..
             } => {
-                self.verify_ebb(inst, destination, errors)?;
+                self.verify_block(inst, destination, errors)?;
                 self.verify_value_list(inst, args, errors)?;
             }
             BranchTable {
                 table, destination, ..
             } => {
-                self.verify_ebb(inst, destination, errors)?;
+                self.verify_block(inst, destination, errors)?;
                 self.verify_jump_table(inst, table, errors)?;
             }
             BranchTableBase { table, .. }
@@ -775,18 +775,18 @@ impl<'a> Verifier<'a> {
         Ok(())
     }
 
-    fn verify_ebb(
+    fn verify_block(
         &self,
         loc: impl Into,
-        e: Ebb,
+        e: Block,
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
-        if !self.func.dfg.ebb_is_valid(e) || !self.func.layout.is_ebb_inserted(e) {
-            return errors.fatal((loc, format!("invalid ebb reference {}", e)));
+        if !self.func.dfg.block_is_valid(e) || !self.func.layout.is_block_inserted(e) {
+            return errors.fatal((loc, format!("invalid block reference {}", e)));
         }
         if let Some(entry_block) = self.func.layout.entry_block() {
             if e == entry_block {
-                return errors.fatal((loc, format!("invalid reference to entry ebb {}", e)));
+                return errors.fatal((loc, format!("invalid reference to entry block {}", e)));
             }
         }
         Ok(())
@@ -947,8 +947,8 @@ impl<'a> Verifier<'a> {
         self.verify_value(loc_inst, v, errors)?;
 
         let dfg = &self.func.dfg;
-        let loc_ebb = self.func.layout.pp_ebb(loc_inst);
-        let is_reachable = self.expected_domtree.is_reachable(loc_ebb);
+        let loc_block = self.func.layout.pp_block(loc_inst);
+        let is_reachable = self.expected_domtree.is_reachable(loc_block);
 
         // SSA form
         match dfg.value_def(v) {
@@ -961,12 +961,12 @@ impl<'a> Verifier<'a> {
                         format!("{} is defined by invalid instruction {}", v, def_inst),
                     ));
                 }
-                // Defining instruction is inserted in an EBB.
-                if self.func.layout.inst_ebb(def_inst) == None {
+                // Defining instruction is inserted in an block.
+                if self.func.layout.inst_block(def_inst) == None {
                     return errors.fatal((
                         loc_inst,
                         self.context(loc_inst),
-                        format!("{} is defined by {} which has no EBB", v, def_inst),
+                        format!("{} is defined by {} which has no block", v, def_inst),
                     ));
                 }
                 // Defining instruction dominates the instruction that uses the value.
@@ -990,33 +990,33 @@ impl<'a> Verifier<'a> {
                     }
                 }
             }
-            ValueDef::Param(ebb, _) => {
-                // Value is defined by an existing EBB.
-                if !dfg.ebb_is_valid(ebb) {
+            ValueDef::Param(block, _) => {
+                // Value is defined by an existing block.
+                if !dfg.block_is_valid(block) {
                     return errors.fatal((
                         loc_inst,
                         self.context(loc_inst),
-                        format!("{} is defined by invalid EBB {}", v, ebb),
+                        format!("{} is defined by invalid block {}", v, block),
                     ));
                 }
-                // Defining EBB is inserted in the layout
-                if !self.func.layout.is_ebb_inserted(ebb) {
+                // Defining block is inserted in the layout
+                if !self.func.layout.is_block_inserted(block) {
                     return errors.fatal((
                         loc_inst,
                         self.context(loc_inst),
-                        format!("{} is defined by {} which is not in the layout", v, ebb),
+                        format!("{} is defined by {} which is not in the layout", v, block),
                     ));
                 }
-                // The defining EBB dominates the instruction using this value.
+                // The defining block dominates the instruction using this value.
                 if is_reachable
                     && !self
                         .expected_domtree
-                        .dominates(ebb, loc_inst, &self.func.layout)
+                        .dominates(block, loc_inst, &self.func.layout)
                 {
                     return errors.fatal((
                         loc_inst,
                         self.context(loc_inst),
-                        format!("uses value arg from non-dominating {}", ebb),
+                        format!("uses value arg from non-dominating {}", block),
                     ));
                 }
             }
@@ -1081,17 +1081,17 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         // We consider two `DominatorTree`s to be equal if they return the same immediate
-        // dominator for each EBB. Therefore the current domtree is valid if it matches the freshly
+        // dominator for each block. Therefore the current domtree is valid if it matches the freshly
         // computed one.
-        for ebb in self.func.layout.ebbs() {
-            let expected = self.expected_domtree.idom(ebb);
-            let got = domtree.idom(ebb);
+        for block in self.func.layout.blocks() {
+            let expected = self.expected_domtree.idom(block);
+            let got = domtree.idom(block);
             if got != expected {
                 return errors.fatal((
-                    ebb,
+                    block,
                     format!(
                         "invalid domtree, expected idom({}) = {:?}, got {:?}",
-                        ebb, expected, got
+                        block, expected, got
                     ),
                 ));
             }
@@ -1100,37 +1100,37 @@ impl<'a> Verifier<'a> {
         if domtree.cfg_postorder().len() != self.expected_domtree.cfg_postorder().len() {
             return errors.fatal((
                 AnyEntity::Function,
-                "incorrect number of Ebbs in postorder traversal",
+                "incorrect number of Blocks in postorder traversal",
             ));
         }
-        for (index, (&test_ebb, &true_ebb)) in domtree
+        for (index, (&test_block, &true_block)) in domtree
             .cfg_postorder()
             .iter()
             .zip(self.expected_domtree.cfg_postorder().iter())
             .enumerate()
         {
-            if test_ebb != true_ebb {
+            if test_block != true_block {
                 return errors.fatal((
-                    test_ebb,
+                    test_block,
                     format!(
-                        "invalid domtree, postorder ebb number {} should be {}, got {}",
-                        index, true_ebb, test_ebb
+                        "invalid domtree, postorder block number {} should be {}, got {}",
+                        index, true_block, test_block
                     ),
                 ));
             }
         }
-        // We verify rpo_cmp on pairs of adjacent ebbs in the postorder
-        for (&prev_ebb, &next_ebb) in domtree.cfg_postorder().iter().adjacent_pairs() {
+        // We verify rpo_cmp on pairs of adjacent blocks in the postorder
+        for (&prev_block, &next_block) in domtree.cfg_postorder().iter().adjacent_pairs() {
             if self
                 .expected_domtree
-                .rpo_cmp(prev_ebb, next_ebb, &self.func.layout)
+                .rpo_cmp(prev_block, next_block, &self.func.layout)
                 != Ordering::Greater
             {
                 return errors.fatal((
-                    next_ebb,
+                    next_block,
                     format!(
                         "invalid domtree, rpo_cmp does not says {} is greater than {}",
-                        prev_ebb, next_ebb
+                        prev_block, next_block
                     ),
                 ));
             }
@@ -1139,26 +1139,26 @@ impl<'a> Verifier<'a> {
     }
 
     fn typecheck_entry_block_params(&self, errors: &mut VerifierErrors) -> VerifierStepResult<()> {
-        if let Some(ebb) = self.func.layout.entry_block() {
+        if let Some(block) = self.func.layout.entry_block() {
             let expected_types = &self.func.signature.params;
-            let ebb_param_count = self.func.dfg.num_ebb_params(ebb);
+            let block_param_count = self.func.dfg.num_block_params(block);
 
-            if ebb_param_count != expected_types.len() {
+            if block_param_count != expected_types.len() {
                 return errors.fatal((
-                    ebb,
+                    block,
                     format!(
                         "entry block parameters ({}) must match function signature ({})",
-                        ebb_param_count,
+                        block_param_count,
                         expected_types.len()
                     ),
                 ));
             }
 
-            for (i, &arg) in self.func.dfg.ebb_params(ebb).iter().enumerate() {
+            for (i, &arg) in self.func.dfg.block_params(block).iter().enumerate() {
                 let arg_type = self.func.dfg.value_type(arg);
                 if arg_type != expected_types[i].value_type {
                     errors.report((
-                        ebb,
+                        block,
                         format!(
                             "entry block parameter {} expected to have type {}, got {}",
                             i, expected_types[i], arg_type
@@ -1295,38 +1295,38 @@ impl<'a> Verifier<'a> {
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
         match self.func.dfg.analyze_branch(inst) {
-            BranchInfo::SingleDest(ebb, _) => {
+            BranchInfo::SingleDest(block, _) => {
                 let iter = self
                     .func
                     .dfg
-                    .ebb_params(ebb)
+                    .block_params(block)
                     .iter()
                     .map(|&v| self.func.dfg.value_type(v));
                 self.typecheck_variable_args_iterator(inst, iter, errors)?;
             }
-            BranchInfo::Table(table, ebb) => {
-                if let Some(ebb) = ebb {
-                    let arg_count = self.func.dfg.num_ebb_params(ebb);
+            BranchInfo::Table(table, block) => {
+                if let Some(block) = block {
+                    let arg_count = self.func.dfg.num_block_params(block);
                     if arg_count != 0 {
                         return errors.nonfatal((
                             inst,
                             self.context(inst),
                             format!(
                                 "takes no arguments, but had target {} with {} arguments",
-                                ebb, arg_count,
+                                block, arg_count,
                             ),
                         ));
                     }
                 }
-                for ebb in self.func.jump_tables[table].iter() {
-                    let arg_count = self.func.dfg.num_ebb_params(*ebb);
+                for block in self.func.jump_tables[table].iter() {
+                    let arg_count = self.func.dfg.num_block_params(*block);
                     if arg_count != 0 {
                         return errors.nonfatal((
                             inst,
                             self.context(inst),
                             format!(
                                 "takes no arguments, but had target {} with {} arguments",
-                                ebb, arg_count,
+                                block, arg_count,
                             ),
                         ));
                     }
@@ -1658,28 +1658,29 @@ impl<'a> Verifier<'a> {
         cfg: &ControlFlowGraph,
         errors: &mut VerifierErrors,
     ) -> VerifierStepResult<()> {
-        let mut expected_succs = BTreeSet::::new();
-        let mut got_succs = BTreeSet::::new();
+        let mut expected_succs = BTreeSet::::new();
+        let mut got_succs = BTreeSet::::new();
         let mut expected_preds = BTreeSet::::new();
         let mut got_preds = BTreeSet::::new();
 
-        for ebb in self.func.layout.ebbs() {
-            expected_succs.extend(self.expected_cfg.succ_iter(ebb));
-            got_succs.extend(cfg.succ_iter(ebb));
+        for block in self.func.layout.blocks() {
+            expected_succs.extend(self.expected_cfg.succ_iter(block));
+            got_succs.extend(cfg.succ_iter(block));
 
-            let missing_succs: Vec = expected_succs.difference(&got_succs).cloned().collect();
+            let missing_succs: Vec =
+                expected_succs.difference(&got_succs).cloned().collect();
             if !missing_succs.is_empty() {
                 errors.report((
-                    ebb,
+                    block,
                     format!("cfg lacked the following successor(s) {:?}", missing_succs),
                 ));
                 continue;
             }
 
-            let excess_succs: Vec = got_succs.difference(&expected_succs).cloned().collect();
+            let excess_succs: Vec = got_succs.difference(&expected_succs).cloned().collect();
             if !excess_succs.is_empty() {
                 errors.report((
-                    ebb,
+                    block,
                     format!("cfg had unexpected successor(s) {:?}", excess_succs),
                 ));
                 continue;
@@ -1687,15 +1688,18 @@ impl<'a> Verifier<'a> {
 
             expected_preds.extend(
                 self.expected_cfg
-                    .pred_iter(ebb)
-                    .map(|BasicBlock { inst, .. }| inst),
+                    .pred_iter(block)
+                    .map(|BlockPredecessor { inst, .. }| inst),
+            );
+            got_preds.extend(
+                cfg.pred_iter(block)
+                    .map(|BlockPredecessor { inst, .. }| inst),
             );
-            got_preds.extend(cfg.pred_iter(ebb).map(|BasicBlock { inst, .. }| inst));
 
             let missing_preds: Vec = expected_preds.difference(&got_preds).cloned().collect();
             if !missing_preds.is_empty() {
                 errors.report((
-                    ebb,
+                    block,
                     format!(
                         "cfg lacked the following predecessor(s) {:?}",
                         missing_preds
@@ -1707,7 +1711,7 @@ impl<'a> Verifier<'a> {
             let excess_preds: Vec = got_preds.difference(&expected_preds).cloned().collect();
             if !excess_preds.is_empty() {
                 errors.report((
-                    ebb,
+                    block,
                     format!("cfg had unexpected predecessor(s) {:?}", excess_preds),
                 ));
                 continue;
@@ -1969,12 +1973,12 @@ impl<'a> Verifier<'a> {
         self.typecheck_entry_block_params(errors)?;
         self.typecheck_function_signature(errors)?;
 
-        for ebb in self.func.layout.ebbs() {
-            if self.func.layout.first_inst(ebb).is_none() {
-                return errors.fatal((ebb, format!("{} cannot be empty", ebb)));
+        for block in self.func.layout.blocks() {
+            if self.func.layout.first_inst(block).is_none() {
+                return errors.fatal((block, format!("{} cannot be empty", block)));
             }
-            for inst in self.func.layout.ebb_insts(ebb) {
-                self.ebb_integrity(ebb, inst, errors)?;
+            for inst in self.func.layout.block_insts(block) {
+                self.block_integrity(block, inst, errors)?;
                 self.instruction_integrity(inst, errors)?;
                 self.verify_safepoint_unused(inst, errors)?;
                 self.typecheck(inst, errors)?;
@@ -1982,7 +1986,7 @@ impl<'a> Verifier<'a> {
                 self.immediate_constraints(inst, errors)?;
             }
 
-            self.encodable_as_bb(ebb, errors)?;
+            self.encodable_as_bb(block, errors)?;
         }
 
         verify_flags(self.func, &self.expected_cfg, self.isa, errors)?;
@@ -2039,20 +2043,20 @@ mod tests {
     #[test]
     fn bad_instruction_format() {
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
-        func.layout.append_ebb(ebb0);
+        let block0 = func.dfg.make_block();
+        func.layout.append_block(block0);
         let nullary_with_bad_opcode = func.dfg.make_inst(InstructionData::UnaryImm {
             opcode: Opcode::F32const,
             imm: 0.into(),
         });
-        func.layout.append_inst(nullary_with_bad_opcode, ebb0);
+        func.layout.append_inst(nullary_with_bad_opcode, block0);
         func.layout.append_inst(
             func.dfg.make_inst(InstructionData::Jump {
                 opcode: Opcode::Jump,
-                destination: ebb0,
+                destination: block0,
                 args: EntityList::default(),
             }),
-            ebb0,
+            block0,
         );
         let flags = &settings::Flags::new(settings::builder());
         let verifier = Verifier::new(&func, flags.into());
@@ -2093,8 +2097,8 @@ mod tests {
     fn test_printing_contextual_errors() {
         // Build function.
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
-        func.layout.append_ebb(ebb0);
+        let block0 = func.dfg.make_block();
+        func.layout.append_block(block0);
 
         // Build instruction: v0, v1 = iconst 42
         let inst = func.dfg.make_inst(InstructionData::UnaryImm {
@@ -2103,7 +2107,7 @@ mod tests {
         });
         func.dfg.append_result(inst, types::I32);
         func.dfg.append_result(inst, types::I32);
-        func.layout.append_inst(inst, ebb0);
+        func.layout.append_inst(inst, block0);
 
         // Setup verifier.
         let mut errors = VerifierErrors::default();
@@ -2120,16 +2124,16 @@ mod tests {
     }
 
     #[test]
-    fn test_empty_ebb() {
+    fn test_empty_block() {
         let mut func = Function::new();
-        let ebb0 = func.dfg.make_ebb();
-        func.layout.append_ebb(ebb0);
+        let block0 = func.dfg.make_block();
+        func.layout.append_block(block0);
 
         let flags = &settings::Flags::new(settings::builder());
         let verifier = Verifier::new(&func, flags.into());
         let mut errors = VerifierErrors::default();
         let _ = verifier.run(&mut errors);
 
-        assert_err_with_msg!(errors, "ebb0 cannot be empty");
+        assert_err_with_msg!(errors, "block0 cannot be empty");
     }
 }
diff --git a/cranelift/codegen/src/write.rs b/cranelift/codegen/src/write.rs
index 507b2f9f6c..6d109f4c04 100644
--- a/cranelift/codegen/src/write.rs
+++ b/cranelift/codegen/src/write.rs
@@ -6,8 +6,8 @@
 use crate::entity::SecondaryMap;
 use crate::ir::entities::AnyEntity;
 use crate::ir::{
-    DataFlowGraph, DisplayFunctionAnnotations, Ebb, Function, Inst, SigRef, Type, Value, ValueDef,
-    ValueLoc,
+    Block, DataFlowGraph, DisplayFunctionAnnotations, Function, Inst, SigRef, Type, Value,
+    ValueDef, ValueLoc,
 };
 use crate::isa::{RegInfo, TargetIsa};
 use crate::packed_option::ReservedValue;
@@ -19,13 +19,13 @@ use core::fmt::{self, Write};
 
 /// A `FuncWriter` used to decorate functions during printing.
 pub trait FuncWriter {
-    /// Write the extended basic block header for the current function.
-    fn write_ebb_header(
+    /// Write the basic block header for the current function.
+    fn write_block_header(
         &mut self,
         w: &mut dyn Write,
         func: &Function,
         isa: Option<&dyn TargetIsa>,
-        ebb: Ebb,
+        block: Block,
         indent: usize,
     ) -> fmt::Result;
 
@@ -145,15 +145,15 @@ impl FuncWriter for PlainWriter {
         write_instruction(w, func, aliases, isa, inst, indent)
     }
 
-    fn write_ebb_header(
+    fn write_block_header(
         &mut self,
         w: &mut dyn Write,
         func: &Function,
         isa: Option<&dyn TargetIsa>,
-        ebb: Ebb,
+        block: Block,
         indent: usize,
     ) -> fmt::Result {
-        write_ebb_header(w, func, isa, ebb, indent)
+        write_block_header(w, func, isa, block, indent)
     }
 }
 
@@ -196,11 +196,11 @@ pub fn decorate_function(
     writeln!(w, " {{")?;
     let aliases = alias_map(func);
     let mut any = func_w.write_preamble(w, func, regs)?;
-    for ebb in &func.layout {
+    for block in &func.layout {
         if any {
             writeln!(w)?;
         }
-        decorate_ebb(func_w, w, func, &aliases, annotations, ebb)?;
+        decorate_block(func_w, w, func, &aliases, annotations, block)?;
         any = true;
     }
     writeln!(w, "}}")
@@ -235,24 +235,24 @@ fn write_arg(
 
 /// Write out the basic block header, outdented:
 ///
-///    ebb1:
-///    ebb1(v1: i32):
-///    ebb10(v4: f64, v5: b1):
+///    block1:
+///    block1(v1: i32):
+///    block10(v4: f64, v5: b1):
 ///
-pub fn write_ebb_header(
+pub fn write_block_header(
     w: &mut dyn Write,
     func: &Function,
     isa: Option<&dyn TargetIsa>,
-    ebb: Ebb,
+    block: Block,
     indent: usize,
 ) -> fmt::Result {
-    // The `indent` is the instruction indentation. EBB headers are 4 spaces out from that.
-    write!(w, "{1:0$}{2}", indent - 4, "", ebb)?;
+    // The `indent` is the instruction indentation. block headers are 4 spaces out from that.
+    write!(w, "{1:0$}{2}", indent - 4, "", block)?;
 
     let regs = isa.map(TargetIsa::register_info);
     let regs = regs.as_ref();
 
-    let mut args = func.dfg.ebb_params(ebb).iter().cloned();
+    let mut args = func.dfg.block_params(block).iter().cloned();
     match args.next() {
         None => return writeln!(w, ":"),
         Some(arg) => {
@@ -309,13 +309,13 @@ fn write_value_range_markers(
     Ok(())
 }
 
-fn decorate_ebb(
+fn decorate_block(
     func_w: &mut FW,
     w: &mut dyn Write,
     func: &Function,
     aliases: &SecondaryMap>,
     annotations: &DisplayFunctionAnnotations,
-    ebb: Ebb,
+    block: Block,
 ) -> fmt::Result {
     // Indent all instructions if any encodings are present.
     let indent = if func.encodings.is_empty() && func.srclocs.is_empty() {
@@ -325,8 +325,8 @@ fn decorate_ebb(
     };
     let isa = annotations.isa;
 
-    func_w.write_ebb_header(w, func, isa, ebb, indent)?;
-    for a in func.dfg.ebb_params(ebb).iter().cloned() {
+    func_w.write_block_header(w, func, isa, block, indent)?;
+    for a in func.dfg.block_params(block).iter().cloned() {
         write_value_aliases(w, aliases, a, indent)?;
     }
 
@@ -334,7 +334,7 @@ fn decorate_ebb(
         if !func.offsets.is_empty() {
             let encinfo = isa.encoding_info();
             let regs = &isa.register_info();
-            for (offset, inst, size) in func.inst_offsets(ebb, &encinfo) {
+            for (offset, inst, size) in func.inst_offsets(block, &encinfo) {
                 func_w.write_instruction(w, func, aliases, Some(isa), inst, indent)?;
                 if size > 0 {
                     if let Some(val_ranges) = annotations.value_ranges {
@@ -346,7 +346,7 @@ fn decorate_ebb(
         }
     }
 
-    for inst in func.layout.ebb_insts(ebb) {
+    for inst in func.layout.block_insts(block) {
         func_w.write_instruction(w, func, aliases, isa, inst, indent)?;
     }
 
@@ -374,11 +374,11 @@ fn type_suffix(func: &Function, inst: Inst) -> Option {
     // operand, we don't need the type suffix.
     if constraints.use_typevar_operand() {
         let ctrl_var = inst_data.typevar_operand(&func.dfg.value_lists).unwrap();
-        let def_ebb = match func.dfg.value_def(ctrl_var) {
-            ValueDef::Result(instr, _) => func.layout.inst_ebb(instr),
-            ValueDef::Param(ebb, _) => Some(ebb),
+        let def_block = match func.dfg.value_def(ctrl_var) {
+            ValueDef::Result(instr, _) => func.layout.inst_block(instr),
+            ValueDef::Param(block, _) => Some(block),
         };
-        if def_ebb.is_some() && def_ebb == func.layout.inst_ebb(inst) {
+        if def_block.is_some() && def_block == func.layout.inst_block(inst) {
             return None;
         }
     }
@@ -533,7 +533,7 @@ pub fn write_operands(
             ..
         } => {
             write!(w, " {}", destination)?;
-            write_ebb_args(w, args.as_slice(pool))
+            write_block_args(w, args.as_slice(pool))
         }
         Branch {
             destination,
@@ -542,7 +542,7 @@ pub fn write_operands(
         } => {
             let args = args.as_slice(pool);
             write!(w, " {}, {}", args[0], destination)?;
-            write_ebb_args(w, &args[1..])
+            write_block_args(w, &args[1..])
         }
         BranchInt {
             cond,
@@ -552,7 +552,7 @@ pub fn write_operands(
         } => {
             let args = args.as_slice(pool);
             write!(w, " {} {}, {}", cond, args[0], destination)?;
-            write_ebb_args(w, &args[1..])
+            write_block_args(w, &args[1..])
         }
         BranchFloat {
             cond,
@@ -562,7 +562,7 @@ pub fn write_operands(
         } => {
             let args = args.as_slice(pool);
             write!(w, " {} {}, {}", cond, args[0], destination)?;
-            write_ebb_args(w, &args[1..])
+            write_block_args(w, &args[1..])
         }
         BranchIcmp {
             cond,
@@ -572,7 +572,7 @@ pub fn write_operands(
         } => {
             let args = args.as_slice(pool);
             write!(w, " {} {}, {}, {}", cond, args[0], args[1], destination)?;
-            write_ebb_args(w, &args[2..])
+            write_block_args(w, &args[2..])
         }
         BranchTable {
             arg,
@@ -714,8 +714,8 @@ pub fn write_operands(
     }
 }
 
-/// Write EBB args using optional parantheses.
-fn write_ebb_args(w: &mut dyn Write, args: &[Value]) -> fmt::Result {
+/// Write block args using optional parantheses.
+fn write_block_args(w: &mut dyn Write, args: &[Value]) -> fmt::Result {
     if args.is_empty() {
         Ok(())
     } else {
@@ -775,33 +775,33 @@ mod tests {
             "function %foo() fast {\n    ss0 = explicit_slot 4\n}\n"
         );
 
-        let ebb = f.dfg.make_ebb();
-        f.layout.append_ebb(ebb);
+        let block = f.dfg.make_block();
+        f.layout.append_block(block);
         assert_eq!(
             f.to_string(),
-            "function %foo() fast {\n    ss0 = explicit_slot 4\n\nebb0:\n}\n"
+            "function %foo() fast {\n    ss0 = explicit_slot 4\n\nblock0:\n}\n"
         );
 
-        f.dfg.append_ebb_param(ebb, types::I8);
+        f.dfg.append_block_param(block, types::I8);
         assert_eq!(
             f.to_string(),
-            "function %foo() fast {\n    ss0 = explicit_slot 4\n\nebb0(v0: i8):\n}\n"
+            "function %foo() fast {\n    ss0 = explicit_slot 4\n\nblock0(v0: i8):\n}\n"
         );
 
-        f.dfg.append_ebb_param(ebb, types::F32.by(4).unwrap());
+        f.dfg.append_block_param(block, types::F32.by(4).unwrap());
         assert_eq!(
             f.to_string(),
-            "function %foo() fast {\n    ss0 = explicit_slot 4\n\nebb0(v0: i8, v1: f32x4):\n}\n"
+            "function %foo() fast {\n    ss0 = explicit_slot 4\n\nblock0(v0: i8, v1: f32x4):\n}\n"
         );
 
         {
             let mut cursor = FuncCursor::new(&mut f);
-            cursor.set_position(CursorPosition::After(ebb));
+            cursor.set_position(CursorPosition::After(block));
             cursor.ins().return_(&[])
         };
         assert_eq!(
             f.to_string(),
-            "function %foo() fast {\n    ss0 = explicit_slot 4\n\nebb0(v0: i8, v1: f32x4):\n    return\n}\n"
+            "function %foo() fast {\n    ss0 = explicit_slot 4\n\nblock0(v0: i8, v1: f32x4):\n    return\n}\n"
         );
     }
 
@@ -811,18 +811,18 @@ mod tests {
 
         let mut func = Function::new();
         {
-            let ebb0 = func.dfg.make_ebb();
+            let block0 = func.dfg.make_block();
             let mut pos = FuncCursor::new(&mut func);
-            pos.insert_ebb(ebb0);
+            pos.insert_block(block0);
 
             // make some detached values for change_to_alias
-            let v0 = pos.func.dfg.append_ebb_param(ebb0, types::I32);
-            let v1 = pos.func.dfg.append_ebb_param(ebb0, types::I32);
-            let v2 = pos.func.dfg.append_ebb_param(ebb0, types::I32);
-            pos.func.dfg.detach_ebb_params(ebb0);
+            let v0 = pos.func.dfg.append_block_param(block0, types::I32);
+            let v1 = pos.func.dfg.append_block_param(block0, types::I32);
+            let v2 = pos.func.dfg.append_block_param(block0, types::I32);
+            pos.func.dfg.detach_block_params(block0);
 
-            // alias to a param--will be printed at beginning of ebb defining param
-            let v3 = pos.func.dfg.append_ebb_param(ebb0, types::I32);
+            // alias to a param--will be printed at beginning of block defining param
+            let v3 = pos.func.dfg.append_block_param(block0, types::I32);
             pos.func.dfg.change_to_alias(v0, v3);
 
             // alias to an alias--should print attached to alias, not ultimate target
@@ -837,7 +837,7 @@ mod tests {
         }
         assert_eq!(
             func.to_string(),
-            "function u0:0() fast {\nebb0(v3: i32):\n    v0 -> v3\n    v2 -> v0\n    v4 = iconst.i32 42\n    v5 = iadd v0, v0\n    v1 -> v5\n    v6 = iconst.i32 23\n    v7 = iadd v1, v1\n}\n"
+            "function u0:0() fast {\nblock0(v3: i32):\n    v0 -> v3\n    v2 -> v0\n    v4 = iconst.i32 42\n    v5 = iadd v0, v0\n    v1 -> v5\n    v6 = iconst.i32 23\n    v7 = iadd v1, v1\n}\n"
         );
     }
 }
diff --git a/cranelift/docs/callex.clif b/cranelift/docs/callex.clif
index 1d93239199..5b3ba2fe3d 100644
--- a/cranelift/docs/callex.clif
+++ b/cranelift/docs/callex.clif
@@ -3,14 +3,14 @@ test verifier
 function %gcd(i32 uext, i32 uext) -> i32 uext system_v {
     fn0 = %divmod(i32 uext, i32 uext) -> i32 uext, i32 uext
 
-ebb1(v0: i32, v1: i32):
-    brz v1, ebb3
-    jump ebb2
+block1(v0: i32, v1: i32):
+    brz v1, block3
+    jump block2
 
-ebb2:
+block2:
     v2, v3 = call fn0(v0, v1)
     return v2
 
-ebb3:
+block3:
     return v0
 }
diff --git a/cranelift/docs/example.clif b/cranelift/docs/example.clif
index b848f7026a..a465c85d0b 100644
--- a/cranelift/docs/example.clif
+++ b/cranelift/docs/example.clif
@@ -3,17 +3,17 @@ test verifier
 function %average(i32, i32) -> f32 system_v {
     ss0 = explicit_slot 8         ; Stack slot for ``sum``.
 
-ebb1(v0: i32, v1: i32):
+block1(v0: i32, v1: i32):
     v2 = f64const 0x0.0
     stack_store v2, ss0
-    brz v1, ebb5                  ; Handle count == 0.
-    jump ebb2
+    brz v1, block5                  ; Handle count == 0.
+    jump block2
 
-ebb2:
+block2:
     v3 = iconst.i32 0
-    jump ebb3(v3)
+    jump block3(v3)
 
-ebb3(v4: i32):
+block3(v4: i32):
     v5 = imul_imm v4, 4
     v6 = iadd v0, v5
     v7 = load.f32 v6              ; array[i]
@@ -23,17 +23,17 @@ ebb3(v4: i32):
     stack_store v10, ss0
     v11 = iadd_imm v4, 1
     v12 = icmp ult v11, v1
-    brnz v12, ebb3(v11)           ; Loop backedge.
-    jump ebb4
+    brnz v12, block3(v11)           ; Loop backedge.
+    jump block4
 
-ebb4:
+block4:
     v13 = stack_load.f64 ss0
     v14 = fcvt_from_uint.f64 v1
     v15 = fdiv v13, v14
     v16 = fdemote.f32 v15
     return v16
 
-ebb5:
+block5:
     v100 = f32const +NaN
     return v100
 }
diff --git a/cranelift/docs/heapex-dyn.clif b/cranelift/docs/heapex-dyn.clif
index 93c40bd29c..161bb4887a 100644
--- a/cranelift/docs/heapex-dyn.clif
+++ b/cranelift/docs/heapex-dyn.clif
@@ -6,7 +6,7 @@ function %add_members(i32, i64 vmctx) -> f32 baldrdash_system_v {
     gv2 = load.i32 notrap aligned gv0+72
     heap0 = dynamic gv1, min 0x1000, bound gv2, offset_guard 0
 
-ebb0(v0: i32, v6: i64):
+block0(v0: i32, v6: i64):
     v1 = heap_addr.i64 heap0, v0, 20
     v2 = load.f32 v1+16
     v3 = heap_addr.i64 heap0, v0, 24
diff --git a/cranelift/docs/heapex-sm32.clif b/cranelift/docs/heapex-sm32.clif
index acd38a6564..ce8fffb914 100644
--- a/cranelift/docs/heapex-sm32.clif
+++ b/cranelift/docs/heapex-sm32.clif
@@ -5,7 +5,7 @@ function %add_members(i32, i32 vmctx) -> f32 baldrdash_system_v {
     gv1 = load.i32 notrap aligned gv0+64
     heap0 = static gv1, min 0x1000, bound 0x10_0000, offset_guard 0x1000
 
-ebb0(v0: i32, v5: i32):
+block0(v0: i32, v5: i32):
     v1 = heap_addr.i32 heap0, v0, 1
     v2 = load.f32 v1+16
     v3 = load.f32 v1+20
diff --git a/cranelift/docs/heapex-sm64.clif b/cranelift/docs/heapex-sm64.clif
index 20934ecba5..89c2df841b 100644
--- a/cranelift/docs/heapex-sm64.clif
+++ b/cranelift/docs/heapex-sm64.clif
@@ -5,7 +5,7 @@ function %add_members(i32, i64 vmctx) -> f32 baldrdash_system_v {
     gv1 = load.i64 notrap aligned gv0+64
     heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v5: i64):
+block0(v0: i32, v5: i64):
     v1 = heap_addr.i64 heap0, v0, 1
     v2 = load.f32 v1+16
     v3 = load.f32 v1+20
diff --git a/cranelift/entity/src/lib.rs b/cranelift/entity/src/lib.rs
index 4f007ed34c..f9062a8c51 100644
--- a/cranelift/entity/src/lib.rs
+++ b/cranelift/entity/src/lib.rs
@@ -104,7 +104,7 @@ macro_rules! entity_impl {
     };
 
     // Include basic `Display` impl using the given display prefix.
-    // Display an `Ebb` reference as "ebb12".
+    // Display a `Block` reference as "block12".
     ($entity:ident, $display_prefix:expr) => {
         entity_impl!($entity);
 
diff --git a/cranelift/entity/src/set.rs b/cranelift/entity/src/set.rs
index c5ba346c63..ac8b156be2 100644
--- a/cranelift/entity/src/set.rs
+++ b/cranelift/entity/src/set.rs
@@ -216,7 +216,7 @@ mod tests {
 
     #[test]
     fn pop_unordered() {
-        let mut ebbs = [
+        let mut blocks = [
             E(0),
             E(1),
             E(6),
@@ -231,14 +231,14 @@ mod tests {
         ];
 
         let mut m = EntitySet::new();
-        for &ebb in &ebbs {
-            m.insert(ebb);
+        for &block in &blocks {
+            m.insert(block);
         }
         assert_eq!(m.len, 13);
-        ebbs.sort();
+        blocks.sort();
 
-        for &ebb in ebbs.iter().rev() {
-            assert_eq!(ebb, m.pop().unwrap());
+        for &block in blocks.iter().rev() {
+            assert_eq!(block, m.pop().unwrap());
         }
 
         assert!(m.is_empty());
diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index aca9196bc5..f2c227a52a 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -387,7 +387,7 @@ struct FaerieRelocSink<'a> {
 }
 
 impl<'a> RelocSink for FaerieRelocSink<'a> {
-    fn reloc_ebb(&mut self, _offset: CodeOffset, _reloc: Reloc, _ebb_offset: CodeOffset) {
+    fn reloc_block(&mut self, _offset: CodeOffset, _reloc: Reloc, _block_offset: CodeOffset) {
         unimplemented!();
     }
 
diff --git a/cranelift/filetests/filetests/cfg/loop.clif b/cranelift/filetests/filetests/cfg/loop.clif
index a52ae09986..a18de9dc31 100644
--- a/cranelift/filetests/filetests/cfg/loop.clif
+++ b/cranelift/filetests/filetests/cfg/loop.clif
@@ -5,33 +5,33 @@ test verifier
 function %nonsense(i32, i32) -> f32 {
 ; regex: I=\binst\d+\b
 ; check: digraph "%nonsense" {
-; check:     ebb0 [shape=record, label="{ebb0(v1: i32, v2: i32):
-; check: | <$(BRZ=$I)>brz v2, ebb2
-; nextln: | <$(JUMP0=$I)>jump ebb3
+; check:     block0 [shape=record, label="{block0(v1: i32, v2: i32):
+; check: | <$(BRZ=$I)>brz v2, block2
+; nextln: | <$(JUMP0=$I)>jump block3
 ; nextln: }"]
-; nextln:     ebb3 [shape=record, label="{ebb3:
-; check: | <$(JUMP3=$I)>jump ebb1(v4)
+; nextln:     block3 [shape=record, label="{block3:
+; check: | <$(JUMP3=$I)>jump block1(v4)
 ; nextln: }"]
-; nextln:     ebb1 [shape=record, label="{ebb1(v5: i32):
-; check:  | <$(BRNZ1=$I)>brnz v13, ebb1(v12)
-; nextln: | <$(JUMP1=$I)>jump ebb4
+; nextln:     block1 [shape=record, label="{block1(v5: i32):
+; check:  | <$(BRNZ1=$I)>brnz v13, block1(v12)
+; nextln: | <$(JUMP1=$I)>jump block4
 ; nextln: }"]
-; nextln:    ebb4 [shape=record, label="{ebb4:
+; nextln:    block4 [shape=record, label="{block4:
 ; check:  | <$I>return v17
 ; nextln: }"]
-; nextln:    ebb2 [shape=record, label="{ebb2:
+; nextln:    block2 [shape=record, label="{block2:
 ; check:  | <$I>return v100
 ; check:}"]
-ebb0(v1: i32, v2: i32):
+block0(v1: i32, v2: i32):
     v3 = f64const 0x0.0
-    brz v2, ebb2            ; unordered: ebb0:$BRZ -> ebb2
-    jump ebb3               ; unordered: ebb0:$JUMP0 -> ebb3
+    brz v2, block2            ; unordered: block0:$BRZ -> block2
+    jump block3               ; unordered: block0:$JUMP0 -> block3
 
-ebb3:
+block3:
     v4 = iconst.i32 0
-    jump ebb1(v4)           ; unordered: ebb3:$JUMP3 -> ebb1
+    jump block1(v4)           ; unordered: block3:$JUMP3 -> block1
 
-ebb1(v5: i32):
+block1(v5: i32):
     v6 = imul_imm v5, 4
     v7 = iadd v1, v6
     v8 = f32const 0.0
@@ -40,17 +40,17 @@ ebb1(v5: i32):
     v11 = fadd v9, v10
     v12 = iadd_imm v5, 1
     v13 = icmp ult v12, v2
-    brnz v13, ebb1(v12)     ; unordered: ebb1:$BRNZ1 -> ebb1
-    jump ebb4               ; unordered: ebb1:$JUMP1 -> ebb4
+    brnz v13, block1(v12)     ; unordered: block1:$BRNZ1 -> block1
+    jump block4               ; unordered: block1:$JUMP1 -> block4
 
-ebb4:
+block4:
     v14 = f64const 0.0
     v15 = f64const 0.0
     v16 = fdiv v14, v15
     v17 = f32const 0.0
     return v17
 
-ebb2:
+block2:
     v100 = f32const 0.0
     return v100
 }
diff --git a/cranelift/filetests/filetests/cfg/traps_early.clif b/cranelift/filetests/filetests/cfg/traps_early.clif
index 36f3016d5c..33de056e4c 100644
--- a/cranelift/filetests/filetests/cfg/traps_early.clif
+++ b/cranelift/filetests/filetests/cfg/traps_early.clif
@@ -6,16 +6,16 @@ test verifier
 function %nonsense(i32) {
 ; check: digraph "%nonsense" {
 
-ebb0(v1: i32):
+block0(v1: i32):
     trap user0      ; error: terminator instruction was encountered before the end
-    brnz v1, ebb2   ; unordered: ebb0:inst1 -> ebb2
-    jump ebb1       ; unordered: ebb0:inst2 -> ebb1
+    brnz v1, block2   ; unordered: block0:inst1 -> block2
+    jump block1       ; unordered: block0:inst2 -> block1
 
-ebb1:
+block1:
     v2 = iconst.i32 0
     v3 = iadd v1, v3
-    jump ebb0(v3)   ; unordered: ebb1:inst5 -> ebb0
+    jump block0(v3)   ; unordered: block1:inst5 -> block0
 
-ebb2:
+block2:
     return v1
 }
diff --git a/cranelift/filetests/filetests/cfg/unused_node.clif b/cranelift/filetests/filetests/cfg/unused_node.clif
index 1a2dd9fb1d..41f98073fd 100644
--- a/cranelift/filetests/filetests/cfg/unused_node.clif
+++ b/cranelift/filetests/filetests/cfg/unused_node.clif
@@ -3,25 +3,25 @@ test print-cfg
 
 function %not_reached(i32) -> i32 {
 ; check: digraph "%not_reached" {
-; check:     ebb0 [shape=record, label="{ebb0(v0: i32):
-; check:  | brnz v0, ebb2
+; check:     block0 [shape=record, label="{block0(v0: i32):
+; check:  | brnz v0, block2
 ; check:  | trap user0
 ; check: }"]
-; check:     ebb1 [shape=record, label="{ebb1:
-; check:  | jump ebb0(v2)
+; check:     block1 [shape=record, label="{block1:
+; check:  | jump block0(v2)
 ; check: }"]
-; check:     ebb2 [shape=record, label="{ebb2:
+; check:     block2 [shape=record, label="{block2:
 ; check:  | return v0
 ; check: }"]
-ebb0(v0: i32):
-    brnz v0, ebb2       ; unordered: ebb0:inst0 -> ebb2
+block0(v0: i32):
+    brnz v0, block2       ; unordered: block0:inst0 -> block2
     trap user0
 
-ebb1:
+block1:
     v1 = iconst.i32 1
     v2 = iadd v0, v1
-    jump ebb0(v2)       ; unordered: ebb1:inst4 -> ebb0
+    jump block0(v2)       ; unordered: block1:inst4 -> block0
 
-ebb2:
+block2:
     return v0
 }
diff --git a/cranelift/filetests/filetests/dce/basic.clif b/cranelift/filetests/filetests/dce/basic.clif
index 436b4aee19..0c94926584 100644
--- a/cranelift/filetests/filetests/dce/basic.clif
+++ b/cranelift/filetests/filetests/dce/basic.clif
@@ -1,46 +1,46 @@
 test dce
 
 function %simple() -> i32 {
-ebb0:
+block0:
     v2 = iconst.i32 2
     v3 = iconst.i32 3
     return v3
 }
 ; sameln: function %simple
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v3 = iconst.i32 3
 ; nextln:     return v3
 ; nextln: }
 
 function %some_branching(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v3 = iconst.i32 70
     v4 = iconst.i32 71
     v5 = iconst.i32 72
     v8 = iconst.i32 73
-    brz v0, ebb1
-    jump ebb2(v8)
+    brz v0, block1
+    jump block2(v8)
 
-ebb1:
+block1:
     v2 = iadd v0, v3
     return v0
 
-ebb2(v9: i32):
+block2(v9: i32):
     v6 = iadd v1, v4
     v7 = iadd v6, v9
     return v7
 }
 ; sameln: function %some_branching
-; nextln: ebb0(v0: i32, v1: i32):
+; nextln: block0(v0: i32, v1: i32):
 ; nextln:     v4 = iconst.i32 71
 ; nextln:     v8 = iconst.i32 73
-; nextln:     brz v0, ebb1
-; nextln:     jump ebb2(v8)
+; nextln:     brz v0, block1
+; nextln:     jump block2(v8)
 ; nextln: 
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:     return v0
 ; nextln: 
-; nextln: ebb2(v9: i32):
+; nextln: block2(v9: i32):
 ; nextln:     v6 = iadd.i32 v1, v4
 ; nextln:     v7 = iadd v6, v9
 ; nextln:     return v7
diff --git a/cranelift/filetests/filetests/domtree/basic.clif b/cranelift/filetests/filetests/domtree/basic.clif
index 37cb20d41d..2960ab0e62 100644
--- a/cranelift/filetests/filetests/domtree/basic.clif
+++ b/cranelift/filetests/filetests/domtree/basic.clif
@@ -1,25 +1,25 @@
 test domtree
 
 function %test(i32) {
-    ebb0(v0: i32):
-        jump ebb1     ; dominates: ebb1
-    ebb1:
-        brz v0, ebb3  ; dominates: ebb3
-        jump ebb2     ; dominates: ebb2
-    ebb2:
-        jump ebb3
-    ebb3:
+    block0(v0: i32):
+        jump block1     ; dominates: block1
+    block1:
+        brz v0, block3  ; dominates: block3
+        jump block2     ; dominates: block2
+    block2:
+        jump block3
+    block3:
         return
 }
 ; check: cfg_postorder:
-; sameln: ebb2
-; sameln: ebb3
-; sameln: ebb1
-; sameln: ebb0
+; sameln: block2
+; sameln: block3
+; sameln: block1
+; sameln: block0
 
 ; check: domtree_preorder {
-; nextln: ebb0: ebb1
-; nextln: ebb1: ebb3 ebb2
-; nextln: ebb3:
-; nextln: ebb2:
+; nextln: block0: block1
+; nextln: block1: block3 block2
+; nextln: block3:
+; nextln: block2:
 ; nextln: }
diff --git a/cranelift/filetests/filetests/domtree/loops.clif b/cranelift/filetests/filetests/domtree/loops.clif
index 3f3fafb01d..a2a334e3fa 100644
--- a/cranelift/filetests/filetests/domtree/loops.clif
+++ b/cranelift/filetests/filetests/domtree/loops.clif
@@ -1,118 +1,118 @@
 test domtree
 
 function %test(i32) {
-    ebb0(v0: i32):
-        brz v0, ebb1  ; dominates: ebb1 ebb3 ebb4 ebb5
-        jump ebb2     ; dominates: ebb2
-    ebb1:
-        jump ebb3
-    ebb2:
-        brz v0, ebb4
-        jump ebb5
-    ebb3:
-        jump ebb4
-    ebb4:
-        brz v0, ebb3
-        jump ebb5
-    ebb5:
-        brz v0, ebb4
-        jump ebb6     ; dominates: ebb6
-    ebb6:
+    block0(v0: i32):
+        brz v0, block1  ; dominates: block1 block3 block4 block5
+        jump block2     ; dominates: block2
+    block1:
+        jump block3
+    block2:
+        brz v0, block4
+        jump block5
+    block3:
+        jump block4
+    block4:
+        brz v0, block3
+        jump block5
+    block5:
+        brz v0, block4
+        jump block6     ; dominates: block6
+    block6:
         return
 }
 ; Fall-through-first, prune-at-source DFT:
 ;
-; ebb0 {
-;     ebb0:brz v0, ebb1 {
-;         ebb0:jump ebb2 {
-;             ebb2 {
-;                 ebb2:brz v2, ebb2 -
-;                 ebb2:brz v3, ebb1 -
-;                 ebb2:brz v4, ebb4 {
-;                     ebb2: jump ebb5 {
-;                         ebb5: jump ebb6 {
-;                             ebb6 {}
+; block0 {
+;     block0:brz v0, block1 {
+;         block0:jump block2 {
+;             block2 {
+;                 block2:brz v2, block2 -
+;                 block2:brz v3, block1 -
+;                 block2:brz v4, block4 {
+;                     block2: jump block5 {
+;                         block5: jump block6 {
+;                             block6 {}
 ;                         }
 ;                     }
-;                     ebb4 {}
+;                     block4 {}
 ;                 }
-;             } ebb2
+;             } block2
 ;         }
-;         ebb1 {
-;             ebb1:jump ebb3 {
-;                 ebb3 {}
+;         block1 {
+;             block1:jump block3 {
+;                 block3 {}
 ;             }
-;         } ebb1
+;         } block1
 ;     }
-; } ebb0
+; } block0
 ;
 ; check: cfg_postorder:
-; sameln: ebb6
-; sameln: ebb5
-; sameln: ebb3
-; sameln: ebb4
-; sameln: ebb2
-; sameln: ebb1
-; sameln: ebb0
+; sameln: block6
+; sameln: block5
+; sameln: block3
+; sameln: block4
+; sameln: block2
+; sameln: block1
+; sameln: block0
 
 ; check: domtree_preorder {
-; nextln: ebb0: ebb1 ebb2 ebb4 ebb3 ebb5
-; nextln: ebb1:
-; nextln: ebb2:
-; nextln: ebb4:
-; nextln: ebb3:
-; nextln: ebb5: ebb6
-; nextln: ebb6:
+; nextln: block0: block1 block2 block4 block3 block5
+; nextln: block1:
+; nextln: block2:
+; nextln: block4:
+; nextln: block3:
+; nextln: block5: block6
+; nextln: block6:
 ; nextln: }
 
 function %loop2(i32) system_v {
-    ebb0(v0: i32):
-        brz v0, ebb1    ; dominates: ebb1 ebb3 ebb4 ebb5
-        jump ebb2       ; dominates: ebb2
-    ebb1:
-        jump ebb3
-    ebb2:
-        brz v0, ebb4
-        jump ebb5
-    ebb3:
-        jump ebb4
-    ebb4:
-        brz v0, ebb3
-        jump ebb8       ; dominates: ebb8
-    ebb8:
-        brnz v0, ebb5
-        jump ebb6       ; dominates: ebb6
-    ebb5:
-        brz v0, ebb4
-        jump ebb9       ; dominates: ebb9
-    ebb9:
+    block0(v0: i32):
+        brz v0, block1    ; dominates: block1 block3 block4 block5
+        jump block2       ; dominates: block2
+    block1:
+        jump block3
+    block2:
+        brz v0, block4
+        jump block5
+    block3:
+        jump block4
+    block4:
+        brz v0, block3
+        jump block8       ; dominates: block8
+    block8:
+        brnz v0, block5
+        jump block6       ; dominates: block6
+    block5:
+        brz v0, block4
+        jump block9       ; dominates: block9
+    block9:
         trap user0
-    ebb6:
-        jump ebb7       ; dominates: ebb7
-    ebb7:
+    block6:
+        jump block7       ; dominates: block7
+    block7:
         return
 }
 ; check: cfg_postorder:
-; sameln: ebb9
-; sameln: ebb5
-; sameln: ebb7
-; sameln: ebb6
-; sameln: ebb8
-; sameln: ebb3
-; sameln: ebb4
-; sameln: ebb2
-; sameln: ebb1
-; sameln: ebb0
+; sameln: block9
+; sameln: block5
+; sameln: block7
+; sameln: block6
+; sameln: block8
+; sameln: block3
+; sameln: block4
+; sameln: block2
+; sameln: block1
+; sameln: block0
 
 ; check: domtree_preorder {
-; nextln: ebb0: ebb1 ebb2 ebb4 ebb3 ebb5
-; nextln: ebb1:
-; nextln: ebb2:
-; nextln: ebb4: ebb8
-; nextln: ebb8: ebb6
-; nextln: ebb6: ebb7
-; nextln: ebb7:
-; nextln: ebb3:
-; nextln: ebb5: ebb9
-; nextln: ebb9:
+; nextln: block0: block1 block2 block4 block3 block5
+; nextln: block1:
+; nextln: block2:
+; nextln: block4: block8
+; nextln: block8: block6
+; nextln: block6: block7
+; nextln: block7:
+; nextln: block3:
+; nextln: block5: block9
+; nextln: block9:
 ; nextln: }
diff --git a/cranelift/filetests/filetests/domtree/loops2.clif b/cranelift/filetests/filetests/domtree/loops2.clif
index 84712c112f..140916bafb 100644
--- a/cranelift/filetests/filetests/domtree/loops2.clif
+++ b/cranelift/filetests/filetests/domtree/loops2.clif
@@ -1,92 +1,92 @@
 test domtree
 
 function %loop1(i32) {
-    ebb0(v0: i32):
-        brz v0, ebb1    ; dominates: ebb1 ebb6
-        jump ebb10      ; dominates: ebb10
-    ebb10:
-        brnz v0, ebb2   ; dominates: ebb2 ebb9
-        jump ebb3       ; dominates: ebb3
-    ebb1:
-        jump ebb6
-    ebb2:
-        brz v0, ebb4    ; dominates: ebb4 ebb7 ebb8
-        jump ebb5       ; dominates: ebb5
-    ebb3:
-        jump ebb9
-    ebb4:
-        brz v0, ebb4
-        jump ebb11      ; dominates: ebb11
-    ebb11:
-        brnz v0, ebb6
-        jump ebb7
-    ebb5:
-        brz v0, ebb7
-        jump ebb12      ; dominates: ebb12
-    ebb12:
-        brnz v0, ebb8
-        jump ebb9
-    ebb6:
+    block0(v0: i32):
+        brz v0, block1    ; dominates: block1 block6
+        jump block10      ; dominates: block10
+    block10:
+        brnz v0, block2   ; dominates: block2 block9
+        jump block3       ; dominates: block3
+    block1:
+        jump block6
+    block2:
+        brz v0, block4    ; dominates: block4 block7 block8
+        jump block5       ; dominates: block5
+    block3:
+        jump block9
+    block4:
+        brz v0, block4
+        jump block11      ; dominates: block11
+    block11:
+        brnz v0, block6
+        jump block7
+    block5:
+        brz v0, block7
+        jump block12      ; dominates: block12
+    block12:
+        brnz v0, block8
+        jump block9
+    block6:
         return
-    ebb7:
-        jump ebb8
-    ebb8:
+    block7:
+        jump block8
+    block8:
         return
-    ebb9:
+    block9:
         return
 }
 
 ; check: domtree_preorder {
-; nextln: ebb0: ebb1 ebb10 ebb6
-; nextln: ebb1:
-; nextln: ebb10: ebb2 ebb3 ebb9
-; nextln: ebb2: ebb4 ebb5 ebb7 ebb8
-; nextln: ebb4: ebb11
-; nextln: ebb11:
-; nextln: ebb5: ebb12
-; nextln: ebb12:
-; nextln: ebb7:
-; nextln: ebb8:
-; nextln: ebb3:
-; nextln: ebb9:
-; nextln: ebb6:
+; nextln: block0: block1 block10 block6
+; nextln: block1:
+; nextln: block10: block2 block3 block9
+; nextln: block2: block4 block5 block7 block8
+; nextln: block4: block11
+; nextln: block11:
+; nextln: block5: block12
+; nextln: block12:
+; nextln: block7:
+; nextln: block8:
+; nextln: block3:
+; nextln: block9:
+; nextln: block6:
 ; nextln: }
 
 function %loop2(i32) system_v {
-    ebb0(v0: i32):
-        brz v0, ebb1    ; dominates: ebb1 ebb3 ebb4 ebb5
-        jump ebb2       ; dominates: ebb2
-    ebb1:
-        jump ebb3
-    ebb2:
-        brz v0, ebb4
-        jump ebb5
-    ebb3:
-        jump ebb4
-    ebb4:
-        brz v0, ebb3
-        jump ebb5
-    ebb5:
-        brz v0, ebb4
-        jump ebb6       ; dominates: ebb6
-    ebb6:
+    block0(v0: i32):
+        brz v0, block1    ; dominates: block1 block3 block4 block5
+        jump block2       ; dominates: block2
+    block1:
+        jump block3
+    block2:
+        brz v0, block4
+        jump block5
+    block3:
+        jump block4
+    block4:
+        brz v0, block3
+        jump block5
+    block5:
+        brz v0, block4
+        jump block6       ; dominates: block6
+    block6:
         return
 }
 ; check: cfg_postorder:
-; sameln: ebb6
-; sameln: ebb5
-; sameln: ebb3
-; sameln: ebb4
-; sameln: ebb2
-; sameln: ebb1
-; sameln: ebb0
+; sameln: block6
+; sameln: block5
+; sameln: block3
+; sameln: block4
+; sameln: block2
+; sameln: block1
+; sameln: block0
 
 ; check: domtree_preorder {
-; nextln: ebb0: ebb1 ebb2 ebb4 ebb3 ebb5
-; nextln: ebb1:
-; nextln: ebb2:
-; nextln: ebb4:
-; nextln: ebb3:
-; nextln: ebb5: ebb6
-; nextln: ebb6:
+; nextln: block0: block1 block2 block4 block3 block5
+; nextln: block1:
+; nextln: block2:
+; nextln: block4:
+; nextln: block3:
+; nextln: block5: block6
+; nextln: block6:
 ; nextln: }
diff --git a/cranelift/filetests/filetests/domtree/tall-tree.clif b/cranelift/filetests/filetests/domtree/tall-tree.clif
index 6f93c023e8..436edc643b 100644
--- a/cranelift/filetests/filetests/domtree/tall-tree.clif
+++ b/cranelift/filetests/filetests/domtree/tall-tree.clif
@@ -1,54 +1,54 @@
 test domtree
 
 function %test(i32) {
-    ebb0(v0: i32):
-        brz v0, ebb1    ; dominates: ebb1
-        jump ebb12      ; dominates: ebb12
-    ebb12:
-        brnz v0, ebb2   ; dominates: ebb2 ebb5
-        jump ebb3       ; dominates: ebb3
-    ebb1:
-        jump ebb4       ; dominates: ebb4
-    ebb2:
-        jump ebb5
-    ebb3:
-        jump ebb5
-    ebb4:
-        brz v0, ebb6    ; dominates: ebb6 ebb10
-        jump ebb7       ; dominates: ebb7
-    ebb5:
+    block0(v0: i32):
+        brz v0, block1    ; dominates: block1
+        jump block12      ; dominates: block12
+    block12:
+        brnz v0, block2   ; dominates: block2 block5
+        jump block3       ; dominates: block3
+    block1:
+        jump block4       ; dominates: block4
+    block2:
+        jump block5
+    block3:
+        jump block5
+    block4:
+        brz v0, block6    ; dominates: block6 block10
+        jump block7       ; dominates: block7
+    block5:
         return
-    ebb6:
-        brz v0, ebb8    ; dominates: ebb11 ebb8
-        jump ebb13      ; dominates: ebb13
-    ebb13:
-        brnz v0, ebb9   ; dominates: ebb9
-        jump ebb10
-    ebb7:
-        jump ebb10
-    ebb8:
-        jump ebb11
-    ebb9:
-        jump ebb11
-    ebb10:
+    block6:
+        brz v0, block8    ; dominates: block11 block8
+        jump block13      ; dominates: block13
+    block13:
+        brnz v0, block9   ; dominates: block9
+        jump block10
+    block7:
+        jump block10
+    block8:
+        jump block11
+    block9:
+        jump block11
+    block10:
         return
-    ebb11:
+    block11:
         return
 }
 
 ; check: domtree_preorder {
-; nextln: ebb0: ebb1 ebb12
-; nextln: ebb1: ebb4
-; nextln: ebb4: ebb6 ebb7 ebb10
-; nextln: ebb6: ebb8 ebb13 ebb11
-; nextln: ebb8:
-; nextln: ebb13: ebb9
-; nextln: ebb9:
-; nextln: ebb11:
-; nextln: ebb7:
-; nextln: ebb10:
-; nextln: ebb12: ebb2 ebb3 ebb5
-; nextln: ebb2:
-; nextln: ebb3:
-; nextln: ebb5:
+; nextln: block0: block1 block12
+; nextln: block1: block4
+; nextln: block4: block6 block7 block10
+; nextln: block6: block8 block13 block11
+; nextln: block8:
+; nextln: block13: block9
+; nextln: block9:
+; nextln: block11:
+; nextln: block7:
+; nextln: block10:
+; nextln: block12: block2 block3 block5
+; nextln: block2:
+; nextln: block3:
+; nextln: block5:
 ; nextln: }
diff --git a/cranelift/filetests/filetests/domtree/wide-tree.clif b/cranelift/filetests/filetests/domtree/wide-tree.clif
index fdfdc169a1..e118e684f0 100644
--- a/cranelift/filetests/filetests/domtree/wide-tree.clif
+++ b/cranelift/filetests/filetests/domtree/wide-tree.clif
@@ -1,73 +1,73 @@
 test domtree
 
 function %test(i32) {
-    ebb0(v0: i32):
-        brz v0, ebb13   ; dominates: ebb13
-        jump ebb1       ; dominates: ebb1
-    ebb1:
-        brz v0, ebb2    ; dominates: ebb2 ebb7
-        jump ebb20      ; dominates: ebb20
-    ebb20:
-        brnz v0, ebb3   ; dominates: ebb3
-        jump ebb21      ; dominates: ebb21
-    ebb21:
-        brz v0, ebb4    ; dominates: ebb4
-        jump ebb22      ; dominates: ebb22
-    ebb22:
-        brnz v0, ebb5   ; dominates: ebb5
-        jump ebb6       ; dominates: ebb6
-    ebb2:
-        jump ebb7
-    ebb3:
-        jump ebb7
-    ebb4:
-        jump ebb7
-    ebb5:
-        jump ebb7
-    ebb6:
-        jump ebb7
-    ebb7:
-        brnz v0, ebb8   ; dominates: ebb8 ebb12
-        jump ebb23      ; dominates: ebb23
-    ebb23:
-        brz v0, ebb9    ; dominates: ebb9
-        jump ebb24      ; dominates: ebb24
-    ebb24:
-        brnz v0, ebb10  ; dominates: ebb10
-        jump ebb11      ; dominates: ebb11
-    ebb8:
-        jump ebb12
-    ebb9:
-        jump ebb12
-    ebb10:
-        brz v0, ebb13
-        jump ebb12
-    ebb11:
-        jump ebb13
-    ebb12:
+    block0(v0: i32):
+        brz v0, block13   ; dominates: block13
+        jump block1       ; dominates: block1
+    block1:
+        brz v0, block2    ; dominates: block2 block7
+        jump block20      ; dominates: block20
+    block20:
+        brnz v0, block3   ; dominates: block3
+        jump block21      ; dominates: block21
+    block21:
+        brz v0, block4    ; dominates: block4
+        jump block22      ; dominates: block22
+    block22:
+        brnz v0, block5   ; dominates: block5
+        jump block6       ; dominates: block6
+    block2:
+        jump block7
+    block3:
+        jump block7
+    block4:
+        jump block7
+    block5:
+        jump block7
+    block6:
+        jump block7
+    block7:
+        brnz v0, block8   ; dominates: block8 block12
+        jump block23      ; dominates: block23
+    block23:
+        brz v0, block9    ; dominates: block9
+        jump block24      ; dominates: block24
+    block24:
+        brnz v0, block10  ; dominates: block10
+        jump block11      ; dominates: block11
+    block8:
+        jump block12
+    block9:
+        jump block12
+    block10:
+        brz v0, block13
+        jump block12
+    block11:
+        jump block13
+    block12:
         return
-    ebb13:
+    block13:
         return
 }
 
 ; check: domtree_preorder {
-; nextln: ebb0: ebb13 ebb1
-; nextln: ebb13:
-; nextln: ebb1: ebb2 ebb20 ebb7
-; nextln: ebb2:
-; nextln: ebb20: ebb3 ebb21
-; nextln: ebb3:
-; nextln: ebb21: ebb4 ebb22
-; nextln: ebb4:
-; nextln: ebb22: ebb5 ebb6
-; nextln: ebb5:
-; nextln: ebb6:
-; nextln: ebb7: ebb8 ebb23 ebb12
-; nextln: ebb8:
-; nextln: ebb23: ebb9 ebb24
-; nextln: ebb9:
-; nextln: ebb24: ebb10 ebb11
-; nextln: ebb10:
-; nextln: ebb11:
-; nextln: ebb12:
+; nextln: block0: block13 block1
+; nextln: block13:
+; nextln: block1: block2 block20 block7
+; nextln: block2:
+; nextln: block20: block3 block21
+; nextln: block3:
+; nextln: block21: block4 block22
+; nextln: block4:
+; nextln: block22: block5 block6
+; nextln: block5:
+; nextln: block6:
+; nextln: block7: block8 block23 block12
+; nextln: block8:
+; nextln: block23: block9 block24
+; nextln: block9:
+; nextln: block24: block10 block11
+; nextln: block10:
+; nextln: block11:
+; nextln: block12:
 ; nextln: }
diff --git a/cranelift/filetests/filetests/isa/riscv/abi-e.clif b/cranelift/filetests/filetests/isa/riscv/abi-e.clif
index 80b275506c..fcd762ee81 100644
--- a/cranelift/filetests/filetests/isa/riscv/abi-e.clif
+++ b/cranelift/filetests/filetests/isa/riscv/abi-e.clif
@@ -9,6 +9,6 @@ function %f() {
     ; available in RV32E.
     sig0 = (i64, i64, i64, i64) -> i64 system_v
     ; check: sig0 = (i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [0], i32 [4]) -> i32 [%x10], i32 [%x11] system_v
-ebb0:
+block0:
     return
 }
diff --git a/cranelift/filetests/filetests/isa/riscv/abi.clif b/cranelift/filetests/filetests/isa/riscv/abi.clif
index b3c9513aef..d9469f490e 100644
--- a/cranelift/filetests/filetests/isa/riscv/abi.clif
+++ b/cranelift/filetests/filetests/isa/riscv/abi.clif
@@ -27,6 +27,6 @@ function %f() {
     sig5 = (i64x4) system_v
     ; check: sig5 = (i32 [%x10], i32 [%x11], i32 [%x12], i32 [%x13], i32 [%x14], i32 [%x15], i32 [%x16], i32 [%x17]) system_v
 
-ebb0:
+block0:
     return
 }
diff --git a/cranelift/filetests/filetests/isa/riscv/binary32.clif b/cranelift/filetests/filetests/isa/riscv/binary32.clif
index b76bfcce69..5a69c4289b 100644
--- a/cranelift/filetests/filetests/isa/riscv/binary32.clif
+++ b/cranelift/filetests/filetests/isa/riscv/binary32.clif
@@ -6,7 +6,7 @@ function %RV32I(i32 link [%x1]) -> i32 link [%x1] {
     sig0 = ()
     fn0 = %foo()
 
-ebb0(v9999: i32):
+block0(v9999: i32):
     [-,%x10]            v1 = iconst.i32 1
     [-,%x21]            v2 = iconst.i32 2
 
@@ -94,96 +94,96 @@ ebb0(v9999: i32):
     call_indirect sig0, v1()                    ; bin: 000500e7
     call_indirect sig0, v2()                    ; bin: 000a80e7
 
-    brz v1, ebb3
-    fallthrough ebb4
+    brz v1, block3
+    fallthrough block4
 
-ebb4:
-    brnz v1, ebb1
-    fallthrough ebb5
+block4:
+    brnz v1, block1
+    fallthrough block5
 
-ebb5:
+block5:
     ; jalr %x0, %x1, 0
     return v9999                        ; bin: 00008067
 
-ebb1:
+block1:
     ; beq 0x000
-    br_icmp eq v1, v2, ebb1             ; bin: 01550063
-    fallthrough ebb100
+    br_icmp eq v1, v2, block1             ; bin: 01550063
+    fallthrough block100
 
-ebb100:
+block100:
     ; bne 0xffc
-    br_icmp ne v1, v2, ebb1             ; bin: ff551ee3
-    fallthrough ebb101
+    br_icmp ne v1, v2, block1             ; bin: ff551ee3
+    fallthrough block101
 
-ebb101:
+block101:
     ; blt 0xff8
-    br_icmp slt v1, v2, ebb1            ; bin: ff554ce3
-    fallthrough ebb102
+    br_icmp slt v1, v2, block1            ; bin: ff554ce3
+    fallthrough block102
 
-ebb102:
+block102:
     ; bge 0xff4
-    br_icmp sge v1, v2, ebb1            ; bin: ff555ae3
-    fallthrough ebb103
+    br_icmp sge v1, v2, block1            ; bin: ff555ae3
+    fallthrough block103
 
-ebb103:
+block103:
     ; bltu 0xff0
-    br_icmp ult v1, v2, ebb1            ; bin: ff5568e3
-    fallthrough ebb104
+    br_icmp ult v1, v2, block1            ; bin: ff5568e3
+    fallthrough block104
 
-ebb104:
+block104:
     ; bgeu 0xfec
-    br_icmp uge v1, v2, ebb1            ; bin: ff5576e3
-    fallthrough ebb105
+    br_icmp uge v1, v2, block1            ; bin: ff5576e3
+    fallthrough block105
 
-ebb105:
+block105:
 
     ; Forward branches.
-    fallthrough ebb106
+    fallthrough block106
 
-ebb106:
+block106:
     ; beq 0x018
-    br_icmp eq v2, v1, ebb2             ; bin: 00aa8c63
-    fallthrough ebb107
+    br_icmp eq v2, v1, block2             ; bin: 00aa8c63
+    fallthrough block107
 
-ebb107:
+block107:
     ; bne 0x014
-    br_icmp ne v2, v1, ebb2             ; bin: 00aa9a63
-    fallthrough ebb108
+    br_icmp ne v2, v1, block2             ; bin: 00aa9a63
+    fallthrough block108
 
-ebb108:
+block108:
     ; blt 0x010
-    br_icmp slt v2, v1, ebb2            ; bin: 00aac863
-    fallthrough ebb109
+    br_icmp slt v2, v1, block2            ; bin: 00aac863
+    fallthrough block109
 
-ebb109:
+block109:
     ; bge 0x00c
-    br_icmp sge v2, v1, ebb2            ; bin: 00aad663
-    fallthrough ebb110
+    br_icmp sge v2, v1, block2            ; bin: 00aad663
+    fallthrough block110
 
-ebb110:
+block110:
     ; bltu 0x008
-    br_icmp ult v2, v1, ebb2            ; bin: 00aae463
-    fallthrough ebb111
+    br_icmp ult v2, v1, block2            ; bin: 00aae463
+    fallthrough block111
 
-ebb111:
+block111:
     ; bgeu 0x004
-    br_icmp uge v2, v1, ebb2            ; bin: 00aaf263
+    br_icmp uge v2, v1, block2            ; bin: 00aaf263
 
-    fallthrough ebb2
+    fallthrough block2
 
-ebb2:
+block2:
     ; jal %x0, 0x00000
-    jump ebb2                           ; bin: 0000006f
+    jump block2                           ; bin: 0000006f
 
-ebb3:
+block3:
     ; beq x, %x0
-    brz v1, ebb3                        ; bin: 00050063
-    fallthrough ebb6
+    brz v1, block3                        ; bin: 00050063
+    fallthrough block6
 
-ebb6:
+block6:
     ; bne x, %x0
-    brnz v1, ebb3                       ; bin: fe051ee3
+    brnz v1, block3                       ; bin: fe051ee3
 
     ; jal %x0, 0x1ffff4
-    jump ebb2                           ; bin: ff5ff06f
+    jump block2                           ; bin: ff5ff06f
 }
diff --git a/cranelift/filetests/filetests/isa/riscv/encoding.clif b/cranelift/filetests/filetests/isa/riscv/encoding.clif
index 98b5f66db6..b8c991f52e 100644
--- a/cranelift/filetests/filetests/isa/riscv/encoding.clif
+++ b/cranelift/filetests/filetests/isa/riscv/encoding.clif
@@ -2,7 +2,7 @@ test legalizer
 target riscv32 supports_m=1
 
 function %int32(i32, i32) {
-ebb0(v1: i32, v2: i32):
+block0(v1: i32, v2: i32):
     v10 = iadd v1, v2
     ; check: [R#0c]
     ; sameln: v10 = iadd
diff --git a/cranelift/filetests/filetests/isa/riscv/expand-i32.clif b/cranelift/filetests/filetests/isa/riscv/expand-i32.clif
index eb63d7cdcd..ee62bc093f 100644
--- a/cranelift/filetests/filetests/isa/riscv/expand-i32.clif
+++ b/cranelift/filetests/filetests/isa/riscv/expand-i32.clif
@@ -8,7 +8,7 @@ target riscv64 supports_m=1
 ; regex: V=v\d+
 
 function %carry_out(i32, i32) -> i32, b1 {
-ebb0(v1: i32, v2: i32):
+block0(v1: i32, v2: i32):
     v3, v4 = iadd_cout v1, v2
     return v3, v4
 }
@@ -19,7 +19,7 @@ ebb0(v1: i32, v2: i32):
 ; Expanding illegal immediate constants.
 ; Note that at some point we'll probably expand the iconst as well.
 function %large_imm(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iadd_imm v0, 1000000000
     return v1
 }
@@ -28,7 +28,7 @@ ebb0(v0: i32):
 ; check: return v1
 
 function %bitclear(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = band_not v0, v1
     ; check: iconst.i32 -1
     ; check: bxor
diff --git a/cranelift/filetests/filetests/isa/riscv/legalize-abi.clif b/cranelift/filetests/filetests/isa/riscv/legalize-abi.clif
index 88c5989db6..0a5fb801a3 100644
--- a/cranelift/filetests/filetests/isa/riscv/legalize-abi.clif
+++ b/cranelift/filetests/filetests/isa/riscv/legalize-abi.clif
@@ -7,8 +7,8 @@ target riscv32
 ; regex: WS=\s+
 
 function %int_split_args(i64) -> i64 {
-ebb0(v0: i64):
-    ; check: ebb0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32):
+block0(v0: i64):
+    ; check: block0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32):
     ; check: v0 = iconcat $v0l, $v0h
     v1 = iadd_imm v0, 1
     ; check: $(v1l=$V), $(v1h=$V) = isplit v1
@@ -19,7 +19,7 @@ ebb0(v0: i64):
 function %split_call_arg(i32) {
     fn1 = %foo(i64)
     fn2 = %foo(i32, i64)
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = uextend.i64 v0
     call fn1(v1)
     ; check: $(v1l=$V), $(v1h=$V) = isplit v1
@@ -31,36 +31,36 @@ ebb0(v0: i32):
 
 function %split_ret_val() {
     fn1 = %foo() -> i64
-ebb0:
+block0:
     v1 = call fn1()
-    ; check: ebb0($(link=$V): i32):
+    ; check: block0($(link=$V): i32):
     ; nextln: $(v1l=$V), $(v1h=$V) = call fn1()
     ; check: v1 = iconcat $v1l, $v1h
-    jump ebb1(v1)
-    ; check: jump ebb1(v1)
+    jump block1(v1)
+    ; check: jump block1(v1)
 
-ebb1(v10: i64):
-    jump ebb1(v10)
+block1(v10: i64):
+    jump block1(v10)
 }
 
 ; First return value is fine, second one is expanded.
 function %split_ret_val2() {
     fn1 = %foo() -> i32, i64
-ebb0:
+block0:
     v1, v2 = call fn1()
-    ; check: ebb0($(link=$V): i32):
+    ; check: block0($(link=$V): i32):
     ; nextln: v1, $(v2l=$V), $(v2h=$V) = call fn1()
     ; check: v2 = iconcat $v2l, $v2h
-    jump ebb1(v1, v2)
-    ; check: jump ebb1(v1, v2)
+    jump block1(v1, v2)
+    ; check: jump block1(v1, v2)
 
-ebb1(v9: i32, v10: i64):
-    jump ebb1(v9, v10)
+block1(v9: i32, v10: i64):
+    jump block1(v9, v10)
 }
 
 function %int_ext(i8, i8 sext, i8 uext) -> i8 uext {
-ebb0(v1: i8, v2: i8, v3: i8):
-    ; check: ebb0(v1: i8, $(v2x=$V): i32, $(v3x=$V): i32, $(link=$V): i32):
+block0(v1: i8, v2: i8, v3: i8):
+    ; check: block0(v1: i8, $(v2x=$V): i32, $(v3x=$V): i32, $(link=$V): i32):
     ; check: v2 = ireduce.i8 $v2x
     ; check: v3 = ireduce.i8 $v3x
     ; check: $(v1x=$V) = uextend.i32 v1
@@ -71,21 +71,21 @@ ebb0(v1: i8, v2: i8, v3: i8):
 ; Function produces single return value, still need to copy.
 function %ext_ret_val() {
     fn1 = %foo() -> i8 sext
-ebb0:
+block0:
     v1 = call fn1()
-    ; check: ebb0($V: i32):
+    ; check: block0($V: i32):
     ; nextln: $(rv=$V) = call fn1()
     ; check: v1 = ireduce.i8 $rv
-    jump ebb1(v1)
-    ; check: jump ebb1(v1)
+    jump block1(v1)
+    ; check: jump block1(v1)
 
-ebb1(v10: i8):
-    jump ebb1(v10)
+block1(v10: i8):
+    jump block1(v10)
 }
 
 function %vector_split_args(i64x4) -> i64x4 {
-ebb0(v0: i64x4):
-    ; check: ebb0($(v0al=$V): i32, $(v0ah=$V): i32, $(v0bl=$V): i32, $(v0bh=$V): i32, $(v0cl=$V): i32, $(v0ch=$V): i32, $(v0dl=$V): i32, $(v0dh=$V): i32, $(link=$V): i32):
+block0(v0: i64x4):
+    ; check: block0($(v0al=$V): i32, $(v0ah=$V): i32, $(v0bl=$V): i32, $(v0bh=$V): i32, $(v0cl=$V): i32, $(v0ch=$V): i32, $(v0dl=$V): i32, $(v0dh=$V): i32, $(link=$V): i32):
     ; check: $(v0a=$V) = iconcat $v0al, $v0ah
     ; check: $(v0b=$V) = iconcat $v0bl, $v0bh
     ; check: $(v0ab=$V) = vconcat $v0a, $v0b
@@ -107,7 +107,7 @@ ebb0(v0: i64x4):
 
 function %indirect(i32) {
     sig1 = () system_v
-ebb0(v0: i32):
+block0(v0: i32):
     call_indirect sig1, v0()
     return
 }
@@ -115,7 +115,7 @@ ebb0(v0: i32):
 ; The first argument to call_indirect doesn't get altered.
 function %indirect_arg(i32, f32x2) {
     sig1 = (f32x2) system_v
-ebb0(v0: i32, v1: f32x2):
+block0(v0: i32, v1: f32x2):
     call_indirect sig1, v0(v1)
     ; check: call_indirect sig1, v0($V, $V)
     return
@@ -125,7 +125,7 @@ ebb0(v0: i32, v1: f32x2):
 function %stack_args(i32) {
     ; check: $(ss0=$SS) = outgoing_arg 4
     fn1 = %foo(i64, i64, i64, i64, i32)
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i64 1
     call fn1(v1, v1, v1, v1, v0)
     ; check: [GPsp#48,$ss0]$WS $(v0s=$V) = spill v0
diff --git a/cranelift/filetests/filetests/isa/riscv/legalize-i64.clif b/cranelift/filetests/filetests/isa/riscv/legalize-i64.clif
index d043337a21..11b31218be 100644
--- a/cranelift/filetests/filetests/isa/riscv/legalize-i64.clif
+++ b/cranelift/filetests/filetests/isa/riscv/legalize-i64.clif
@@ -5,11 +5,11 @@ target riscv32 supports_m=1
 ; regex: V=v\d+
 
 function %bitwise_and(i64, i64) -> i64 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v3 = band v1, v2
     return v3
 }
-; check: ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32):
+; check: block0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32):
 ; check: [R#ec
 ; sameln: $(v3l=$V) = band $v1l, $v2l
 ; check: [R#ec
@@ -18,11 +18,11 @@ ebb0(v1: i64, v2: i64):
 ; check: return $v3l, $v3h, $link
 
 function %bitwise_or(i64, i64) -> i64 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v3 = bor v1, v2
     return v3
 }
-; check: ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32):
+; check: block0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32):
 ; check: [R#cc
 ; sameln: $(v3l=$V) = bor $v1l, $v2l
 ; check: [R#cc
@@ -31,11 +31,11 @@ ebb0(v1: i64, v2: i64):
 ; check: return $v3l, $v3h, $link
 
 function %bitwise_xor(i64, i64) -> i64 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v3 = bxor v1, v2
     return v3
 }
-; check: ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32):
+; check: block0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32):
 ; check: [R#8c
 ; sameln: $(v3l=$V) = bxor $v1l, $v2l
 ; check: [R#8c
@@ -47,11 +47,11 @@ function %arith_add(i64, i64) -> i64 {
 ; Legalizing iadd.i64 requires two steps:
 ; 1. Narrow to iadd_cout.i32, then
 ; 2. Expand iadd_cout.i32 since RISC-V has no carry flag.
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v3 = iadd v1, v2
     return v3
 }
-; check: ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32):
+; check: block0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32):
 ; check: [R#0c
 ; sameln: $(v3l=$V) = iadd $v1l, $v2l
 ; check: $(c=$V) = icmp ult $v3l, $v1l
diff --git a/cranelift/filetests/filetests/isa/riscv/legalize-icmp_imm-i64.clif b/cranelift/filetests/filetests/isa/riscv/legalize-icmp_imm-i64.clif
index 3dd674a5d3..d7250cb3af 100644
--- a/cranelift/filetests/filetests/isa/riscv/legalize-icmp_imm-i64.clif
+++ b/cranelift/filetests/filetests/isa/riscv/legalize-icmp_imm-i64.clif
@@ -4,11 +4,11 @@ target riscv32
 ; regex: V=v\d+
 
 function %icmp_imm_eq(i64) -> b1 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = icmp_imm eq v0, 0x20202020_10101010
     return v1
 }
-; check: ebb0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32):
+; check: block0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32):
 ; nextln: $(v2l=$V) -> $(v0l)
 ; nextln: $(v2h=$V) -> $(v0h)
 ; nextln: v0 = iconcat $(v0l), $(v0h)
@@ -20,11 +20,11 @@ ebb0(v0: i64):
 ; nextln: return v1, $(link)
 
 function %icmp_imm_ne(i64) -> b1 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = icmp_imm ne v0, 0x33333333_44444444
     return v1
 }
-; check: ebb0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32):
+; check: block0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32):
 ; nextln: $(v2l=$V) -> $(v0l)
 ; nextln: $(v2h=$V) -> $(v0h)
 ; nextln: v0 = iconcat $(v0l), $(v0h)
@@ -36,11 +36,11 @@ ebb0(v0: i64):
 ; nextln: return v1, $(link)
 
 function %icmp_imm_sge(i64) -> b1 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = icmp_imm sge v0, 0x01020304_05060708
     return v1
 }
-; check: ebb0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32):
+; check: block0($(v0l=$V): i32, $(v0h=$V): i32, $(link=$V): i32):
 ; nextln: $(v2l=$V) -> $(v0l)
 ; nextln: $(v2h=$V) -> $(v0h)
 ; nextln: v0 = iconcat $(v0l), $(v0h)
diff --git a/cranelift/filetests/filetests/isa/riscv/parse-encoding.clif b/cranelift/filetests/filetests/isa/riscv/parse-encoding.clif
index f79e552e6d..21cd828b8a 100644
--- a/cranelift/filetests/filetests/isa/riscv/parse-encoding.clif
+++ b/cranelift/filetests/filetests/isa/riscv/parse-encoding.clif
@@ -31,6 +31,6 @@ function %parse_encoding(i32 [%x5]) -> i32 [%x10] {
     ; check: sig6 = (i32 [%x10]) -> b1 [%x10] system_v
     ; nextln: fn0 = %bar sig6
 
-ebb0(v0: i32):
+block0(v0: i32):
     return v0
 }
diff --git a/cranelift/filetests/filetests/isa/riscv/regmove.clif b/cranelift/filetests/filetests/isa/riscv/regmove.clif
index 6ec17ef813..f1509e8178 100644
--- a/cranelift/filetests/filetests/isa/riscv/regmove.clif
+++ b/cranelift/filetests/filetests/isa/riscv/regmove.clif
@@ -3,7 +3,7 @@ test binemit
 target riscv32
 
 function %regmoves(i32 link [%x1]) -> i32 link [%x1] {
-ebb0(v9999: i32):
+block0(v9999: i32):
     [-,%x10]            v1 = iconst.i32 1
     [-,%x7]             v2 = iadd_imm v1, 1000          ; bin: 3e850393
                         regmove v1, %x10 -> %x11        ; bin: 00050593
diff --git a/cranelift/filetests/filetests/isa/riscv/split-args.clif b/cranelift/filetests/filetests/isa/riscv/split-args.clif
index dd605de81a..9f4b3e9268 100644
--- a/cranelift/filetests/filetests/isa/riscv/split-args.clif
+++ b/cranelift/filetests/filetests/isa/riscv/split-args.clif
@@ -1,17 +1,17 @@
-; Test the legalization of EBB arguments that are split.
+; Test the legalization of block arguments that are split.
 test legalizer
 target riscv32
 
 ; regex: V=v\d+
 
 function %simple(i64, i64) -> i64 {
-ebb0(v1: i64, v2: i64):
-; check: ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32):
-    jump ebb1(v1)
-    ; check: jump ebb1($v1l, $v1h)
+block0(v1: i64, v2: i64):
+; check: block0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32):
+    jump block1(v1)
+    ; check: jump block1($v1l, $v1h)
 
-ebb1(v3: i64):
-; check: ebb1($(v3l=$V): i32, $(v3h=$V): i32):
+block1(v3: i64):
+; check: block1($(v3l=$V): i32, $(v3h=$V): i32):
     v4 = band v3, v2
     ; check: $(v4l=$V) = band $v3l, $v2l
     ; check: $(v4h=$V) = band $v3h, $v2h
@@ -20,18 +20,18 @@ ebb1(v3: i64):
 }
 
 function %multi(i64) -> i64 {
-ebb1(v1: i64):
-; check: ebb1($(v1l=$V): i32, $(v1h=$V): i32, $(link=$V): i32):
-    jump ebb2(v1, v1)
-    ; check: jump ebb2($v1l, $v1l, $v1h, $v1h)
+block1(v1: i64):
+; check: block1($(v1l=$V): i32, $(v1h=$V): i32, $(link=$V): i32):
+    jump block2(v1, v1)
+    ; check: jump block2($v1l, $v1l, $v1h, $v1h)
 
-ebb2(v2: i64, v3: i64):
-; check: ebb2($(v2l=$V): i32, $(v3l=$V): i32, $(v2h=$V): i32, $(v3h=$V): i32):
-    jump ebb3(v2)
-    ; check: jump ebb3($v2l, $v2h)
+block2(v2: i64, v3: i64):
+; check: block2($(v2l=$V): i32, $(v3l=$V): i32, $(v2h=$V): i32, $(v3h=$V): i32):
+    jump block3(v2)
+    ; check: jump block3($v2l, $v2h)
 
-ebb3(v4: i64):
-; check: ebb3($(v4l=$V): i32, $(v4h=$V): i32):
+block3(v4: i64):
+; check: block3($(v4l=$V): i32, $(v4h=$V): i32):
     v5 = band v4, v3
     ; check: $(v5l=$V) = band $v4l, $v3l
     ; check: $(v5h=$V) = band $v4h, $v3h
@@ -40,16 +40,16 @@ ebb3(v4: i64):
 }
 
 function %loop(i64, i64) -> i64 {
-ebb0(v1: i64, v2: i64):
-; check: ebb0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32):
-    jump ebb1(v1)
-    ; check: jump ebb1($v1l, $v1h)
+block0(v1: i64, v2: i64):
+; check: block0($(v1l=$V): i32, $(v1h=$V): i32, $(v2l=$V): i32, $(v2h=$V): i32, $(link=$V): i32):
+    jump block1(v1)
+    ; check: jump block1($v1l, $v1h)
 
-ebb1(v3: i64):
-; check: ebb1($(v3l=$V): i32, $(v3h=$V): i32):
+block1(v3: i64):
+; check: block1($(v3l=$V): i32, $(v3h=$V): i32):
     v4 = band v3, v2
     ; check: $(v4l=$V) = band $v3l, $v2l
     ; check: $(v4h=$V) = band $v3h, $v2h
-    jump ebb1(v4)
-    ; check: jump ebb1($v4l, $v4h)
+    jump block1(v4)
+    ; check: jump block1($v4l, $v4h)
 }
diff --git a/cranelift/filetests/filetests/isa/riscv/verify-encoding.clif b/cranelift/filetests/filetests/isa/riscv/verify-encoding.clif
index 0ee7eb7a83..1d29b86da9 100644
--- a/cranelift/filetests/filetests/isa/riscv/verify-encoding.clif
+++ b/cranelift/filetests/filetests/isa/riscv/verify-encoding.clif
@@ -4,7 +4,7 @@ target riscv32
 function %RV32I(i32 link [%x1]) -> i32 link [%x1] {
     fn0 = %foo()
 
-ebb0(v9999: i32):
+block0(v9999: i32):
     ; iconst.i32 needs legalizing, so it should throw a
     [R#0,-]         v1 = iconst.i32 0xf0f0f0f0f0 ; error: Instruction failed to re-encode
     [Iret#19]       return v9999
@@ -13,7 +13,7 @@ ebb0(v9999: i32):
 function %RV32I(i32 link [%x1]) -> i32 link [%x1] {
     fn0 = %foo()
 
-ebb0(v9999: i32):
+block0(v9999: i32):
     v1 = iconst.i32 1
     v2 = iconst.i32 2
     [R#0,-]         v3 = iadd v1, v2 ; error: encoding R#00 should be R#0c
diff --git a/cranelift/filetests/filetests/isa/x86/abcd.clif b/cranelift/filetests/filetests/isa/x86/abcd.clif
index 67acac970b..658ba66ca4 100644
--- a/cranelift/filetests/filetests/isa/x86/abcd.clif
+++ b/cranelift/filetests/filetests/isa/x86/abcd.clif
@@ -5,7 +5,7 @@ target i686
 ; allocator can move it to a register that can be.
 
 function %test(i32 [%rdi]) -> i32 system_v {
-ebb0(v0: i32 [%rdi]):
+block0(v0: i32 [%rdi]):
     v1 = ireduce.i8 v0
     v2 = sextend.i32 v1
     return v2
diff --git a/cranelift/filetests/filetests/isa/x86/abi-bool.clif b/cranelift/filetests/filetests/isa/x86/abi-bool.clif
index fdf21ba055..2083250a91 100644
--- a/cranelift/filetests/filetests/isa/x86/abi-bool.clif
+++ b/cranelift/filetests/filetests/isa/x86/abi-bool.clif
@@ -2,18 +2,18 @@ test compile
 target x86_64 haswell
 
 function %foo(i64, i64, i64, i32) -> b1 system_v {
-ebb3(v0: i64, v1: i64, v2: i64, v3: i32):
+block3(v0: i64, v1: i64, v2: i64, v3: i32):
     v5 = icmp ne v2, v2
     v8 = iconst.i64 0
-    jump ebb2(v8, v3, v5)
+    jump block2(v8, v3, v5)
 
-ebb2(v10: i64, v30: i32, v37: b1):
+block2(v10: i64, v30: i32, v37: b1):
     v18 = load.i32 notrap aligned v2
     v27 = iadd.i64 v10, v10
     v31 = icmp eq v30, v30
-    brz v31, ebb2(v27, v30, v37)
-    jump ebb0(v37)
+    brz v31, block2(v27, v30, v37)
+    jump block0(v37)
 
-ebb0(v35: b1):
+block0(v35: b1):
     return v35
 }
diff --git a/cranelift/filetests/filetests/isa/x86/abi32.clif b/cranelift/filetests/filetests/isa/x86/abi32.clif
index 4b9f5fbcd1..155d0efc4c 100644
--- a/cranelift/filetests/filetests/isa/x86/abi32.clif
+++ b/cranelift/filetests/filetests/isa/x86/abi32.clif
@@ -14,7 +14,7 @@ function %f() {
     sig2 = (f32, i64) -> f64 system_v
     ; check: sig2 = (f32 [0], i32 [4], i32 [8]) -> f64 [%xmm0] system_v
 
-ebb0:
+block0:
     return
 }
 
diff --git a/cranelift/filetests/filetests/isa/x86/abi64.clif b/cranelift/filetests/filetests/isa/x86/abi64.clif
index ccea6304e6..9494e78c67 100644
--- a/cranelift/filetests/filetests/isa/x86/abi64.clif
+++ b/cranelift/filetests/filetests/isa/x86/abi64.clif
@@ -14,7 +14,7 @@ function %f() {
     sig2 = (f32, i64) -> f64 system_v
     ; check: sig2 = (f32 [%xmm0], i64 [%rdi]) -> f64 [%xmm0] system_v
 
-ebb0:
+block0:
     return
 }
 
@@ -22,10 +22,10 @@ function %pass_stack_int64(i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64
     sig0 = (i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64 vmctx) baldrdash_system_v
     fn0 = u0:0 sig0
 
-ebb0(v0: i64, v1: i64, v2: i64, v3: i64, v4: i64, v5: i64, v6: i64, v7: i64, v8: i64, v9: i64, v10: i64, v11: i64, v12: i64, v13: i64, v14: i64, v15: i64, v16: i64, v17: i64, v18: i64, v19: i64, v20: i64):
+block0(v0: i64, v1: i64, v2: i64, v3: i64, v4: i64, v5: i64, v6: i64, v7: i64, v8: i64, v9: i64, v10: i64, v11: i64, v12: i64, v13: i64, v14: i64, v15: i64, v16: i64, v17: i64, v18: i64, v19: i64, v20: i64):
     call fn0(v0, v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16, v17, v18, v19, v20)
-    jump ebb1
+    jump block1
 
-ebb1:
+block1:
     return
 }
diff --git a/cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif
index 124803af6c..744b936c83 100644
--- a/cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif
+++ b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs32.clif
@@ -14,7 +14,7 @@ function %I32() {
     sig0 = ()
     fn0 = %foo()
 
-ebb0:
+block0:
 
     ; asm: movl $-1, %ecx
     [-,%rcx]            v400 = func_addr.i32 fn0        ; bin: b9 Abs4(%foo) ffffffff
diff --git a/cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif
index f4dd9f2a6e..623e96c9d3 100644
--- a/cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif
+++ b/cranelift/filetests/filetests/isa/x86/allones_funcaddrs64.clif
@@ -14,7 +14,7 @@ function %I64() {
     sig0 = ()
     fn0 = %foo()
 
-ebb0:
+block0:
 
     ; asm: movabsq $-1, %rcx
     [-,%rcx]            v400 = func_addr.i64 fn0        ; bin: 48 b9 Abs8(%foo) ffffffffffffffff
diff --git a/cranelift/filetests/filetests/isa/x86/baldrdash-table-sig-reg.clif b/cranelift/filetests/filetests/isa/x86/baldrdash-table-sig-reg.clif
index 6d1f72203d..e8dc4393ca 100644
--- a/cranelift/filetests/filetests/isa/x86/baldrdash-table-sig-reg.clif
+++ b/cranelift/filetests/filetests/isa/x86/baldrdash-table-sig-reg.clif
@@ -5,7 +5,7 @@ target i686
 function u0:0(i32 vmctx) baldrdash_system_v {
     sig0 = (i32 vmctx, i32 sigid) baldrdash_system_v
 
-ebb0(v0: i32):
+block0(v0: i32):
     v2 = iconst.i32 0
     v8 = iconst.i32 0
     v9 = iconst.i32 0
diff --git a/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount.clif b/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount.clif
index 595e5e99bb..7aca619d09 100644
--- a/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount.clif
+++ b/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount.clif
@@ -5,7 +5,7 @@ target x86_64 baseline
 ; clz/ctz on 64 bit operands
 
 function %i64_clz(i64) -> i64 {
-ebb0(v10: i64):
+block0(v10: i64):
   v11 = clz v10
   ; check: x86_bsr
   ; check: selectif.i64
@@ -13,7 +13,7 @@ ebb0(v10: i64):
 }
 
 function %i64_ctz(i64) -> i64 {
-ebb1(v20: i64):
+block1(v20: i64):
   v21 = ctz v20
   ; check: x86_bsf
   ; check: selectif.i64
@@ -24,7 +24,7 @@ ebb1(v20: i64):
 ; clz/ctz on 32 bit operands
 
 function %i32_clz(i32) -> i32 {
-ebb0(v10: i32):
+block0(v10: i32):
   v11 = clz v10
   ; check: x86_bsr
   ; check: selectif.i32
@@ -32,7 +32,7 @@ ebb0(v10: i32):
 }
 
 function %i32_ctz(i32) -> i32 {
-ebb1(v20: i32):
+block1(v20: i32):
   v21 = ctz v20
   ; check: x86_bsf
   ; check: selectif.i32
@@ -43,7 +43,7 @@ ebb1(v20: i32):
 ; popcount on 64 bit operands
 
 function %i64_popcount(i64) -> i64 {
-ebb0(v30: i64):
+block0(v30: i64):
   v31 = popcnt v30;
   ; check: ushr_imm
   ; check: iconst.i64
@@ -69,7 +69,7 @@ ebb0(v30: i64):
 ; popcount on 32 bit operands
 
 function %i32_popcount(i32) -> i32 {
-ebb0(v40: i32):
+block0(v40: i32):
   v41 = popcnt v40;
   ; check: ushr_imm
   ; check: iconst.i32
diff --git a/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif b/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif
index 31c4016dc0..cbe18d904c 100644
--- a/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif
+++ b/cranelift/filetests/filetests/isa/x86/baseline_clz_ctz_popcount_encoding.clif
@@ -8,7 +8,7 @@ target x86_64 baseline
 ;
 
 function %Foo() {
-ebb0:
+block0:
     ; 64-bit wide bsf
 
     [-,%r11]                 v10 = iconst.i64 0x1234
diff --git a/cranelift/filetests/filetests/isa/x86/binary32-float.clif b/cranelift/filetests/filetests/isa/x86/binary32-float.clif
index 8a4ae5fe7e..dc65a1f234 100644
--- a/cranelift/filetests/filetests/isa/x86/binary32-float.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary32-float.clif
@@ -13,7 +13,7 @@ function %F32() {
     ss2 = incoming_arg 1024, offset -2048
     ss3 = incoming_arg 8, offset -2056
 
-ebb0:
+block0:
     [-,%rcx]            v0 = iconst.i32 1
     [-,%rsi]            v1 = iconst.i32 2
 
@@ -262,7 +262,7 @@ function %F64() {
     ss2 = incoming_arg 1024, offset -2048
     ss3 = incoming_arg 8, offset -2056
 
-ebb0:
+block0:
     [-,%rcx]            v0 = iconst.i32 1
     [-,%rsi]            v1 = iconst.i32 2
 
@@ -472,53 +472,53 @@ ebb0:
 }
 
 function %cpuflags_float(f32 [%xmm0]) {
-ebb0(v0: f32 [%xmm0]):
+block0(v0: f32 [%xmm0]):
     ; asm: ucomiss %xmm0, %xmm0
     [-,%rflags]         v1 = ffcmp v0, v0                       ; bin: 0f 2e c0
 
-    jump ebb1
+    jump block1
 
-ebb1:
-    ; asm: jnp ebb1
-    brff ord v1, ebb1                                           ; bin: 7b fe
-    jump ebb2
+block1:
+    ; asm: jnp block1
+    brff ord v1, block1                                           ; bin: 7b fe
+    jump block2
 
-ebb2:
-    ; asm: jp ebb1
-    brff uno v1, ebb1                                           ; bin: 7a fc
-    jump ebb3
+block2:
+    ; asm: jp block1
+    brff uno v1, block1                                           ; bin: 7a fc
+    jump block3
 
-ebb3:
-    ; asm: jne ebb1
-    brff one v1, ebb1                                           ; bin: 75 fa
-    jump ebb4
+block3:
+    ; asm: jne block1
+    brff one v1, block1                                           ; bin: 75 fa
+    jump block4
 
-ebb4:
-    ; asm: je ebb1
-    brff ueq v1, ebb1                                           ; bin: 74 f8
-    jump ebb5
+block4:
+    ; asm: je block1
+    brff ueq v1, block1                                           ; bin: 74 f8
+    jump block5
 
-ebb5:
-    ; asm: ja ebb1
-    brff gt v1, ebb1                                            ; bin: 77 f6
-    jump ebb6
+block5:
+    ; asm: ja block1
+    brff gt v1, block1                                            ; bin: 77 f6
+    jump block6
 
-ebb6:
-    ; asm: jae ebb1
-    brff ge v1, ebb1                                            ; bin: 73 f4
-    jump ebb7
+block6:
+    ; asm: jae block1
+    brff ge v1, block1                                            ; bin: 73 f4
+    jump block7
 
-ebb7:
-    ; asm: jb ebb1
-    brff ult v1, ebb1                                           ; bin: 72 f2
-    jump ebb8
+block7:
+    ; asm: jb block1
+    brff ult v1, block1                                           ; bin: 72 f2
+    jump block8
 
-ebb8:
-    ; asm: jbe ebb1
-    brff ule v1, ebb1                                           ; bin: 76 f0
-    jump ebb9
+block8:
+    ; asm: jbe block1
+    brff ule v1, block1                                           ; bin: 76 f0
+    jump block9
 
-ebb9:
+block9:
     ; asm: jp .+4; ud2
     trapff ord v1, user0                                        ; bin: 7a 02 user0 0f 0b
     ; asm: jnp .+4; ud2
diff --git a/cranelift/filetests/filetests/isa/x86/binary32.clif b/cranelift/filetests/filetests/isa/x86/binary32.clif
index 5db78ee2e0..abe99ce0ea 100644
--- a/cranelift/filetests/filetests/isa/x86/binary32.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary32.clif
@@ -19,7 +19,7 @@ function %I32() {
     ss2 = incoming_arg 1024, offset -2048
     ss3 = incoming_arg 8, offset -2056
 
-ebb0:
+block0:
     ; asm: movl $1, %ecx
     [-,%rcx]            v1 = iconst.i32 1        ; bin: b9 00000001
     ; asm: movl $2, %esi
@@ -486,141 +486,141 @@ ebb0:
     [-,%rcx,%rflags] v709, v710 = isub_ifborrow v1, v2, v707 ; bin: 19 f1
 
     ; asm: testl %ecx, %ecx
-    ; asm: je ebb1
-    brz v1, ebb1                                ; bin: 85 c9 74 0e
-    fallthrough ebb3
+    ; asm: je block1
+    brz v1, block1                                ; bin: 85 c9 74 0e
+    fallthrough block3
 
-ebb3:
+block3:
     ; asm: testl %esi, %esi
-    ; asm: je ebb1
-    brz v2, ebb1                                ; bin: 85 f6 74 0a
-    fallthrough ebb4
+    ; asm: je block1
+    brz v2, block1                                ; bin: 85 f6 74 0a
+    fallthrough block4
 
-ebb4:
+block4:
     ; asm: testl %ecx, %ecx
-    ; asm: jne ebb1
-    brnz v1, ebb1                               ; bin: 85 c9 75 06
-    fallthrough ebb5
+    ; asm: jne block1
+    brnz v1, block1                               ; bin: 85 c9 75 06
+    fallthrough block5
 
-ebb5:
+block5:
     ; asm: testl %esi, %esi
-    ; asm: jne ebb1
-    brnz v2, ebb1                               ; bin: 85 f6 75 02
+    ; asm: jne block1
+    brnz v2, block1                               ; bin: 85 f6 75 02
 
-    ; asm: jmp ebb2
-    jump ebb2                                   ; bin: eb 01
+    ; asm: jmp block2
+    jump block2                                   ; bin: eb 01
 
-    ; asm: ebb1:
-ebb1:
+    ; asm: block1:
+block1:
     ; asm: ret
     return                                      ; bin: c3
 
-    ; asm: ebb2:
-ebb2:
+    ; asm: block2:
+block2:
     trap user0                                  ; bin: user0 0f 0b
 }
 
 ; Special branch encodings only for I32 mode.
 function %special_branches() {
-ebb0:
+block0:
     [-,%rcx]            v1 = iconst.i32 1
     [-,%rsi]            v2 = iconst.i32 2
     [-,%rdi]            v3 = icmp eq v1, v2
     [-,%rbx]            v4 = icmp ugt v1, v2
 
     ; asm: testl $0xff, %edi
-    ; asm: je ebb1
-    brz v3, ebb1                                ; bin: f7 c7 000000ff 0f 84 00000015
-    fallthrough ebb2
+    ; asm: je block1
+    brz v3, block1                                ; bin: f7 c7 000000ff 0f 84 00000015
+    fallthrough block2
 
-ebb2:
+block2:
     ; asm: testb %bl, %bl
-    ; asm: je ebb1
-    brz v4, ebb1                                ; bin: 84 db 74 11
-    fallthrough ebb3
+    ; asm: je block1
+    brz v4, block1                                ; bin: 84 db 74 11
+    fallthrough block3
 
-ebb3:
+block3:
     ; asm: testl $0xff, %edi
-    ; asm: jne ebb1
-    brnz v3, ebb1                               ; bin: f7 c7 000000ff 0f 85 00000005
-    fallthrough ebb4
+    ; asm: jne block1
+    brnz v3, block1                               ; bin: f7 c7 000000ff 0f 85 00000005
+    fallthrough block4
 
-ebb4:
+block4:
     ; asm: testb %bl, %bl
-    ; asm: jne ebb1
-    brnz v4, ebb1                               ; bin: 84 db 75 01
-    fallthrough ebb5
+    ; asm: jne block1
+    brnz v4, block1                               ; bin: 84 db 75 01
+    fallthrough block5
 
-ebb5:
+block5:
     return
 
-ebb1:
+block1:
     return
 }
 
 ; CPU flag instructions.
 function %cpu_flags() {
-ebb0:
+block0:
     [-,%rcx]            v1 = iconst.i32 1
     [-,%rsi]            v2 = iconst.i32 2
-    jump ebb1
+    jump block1
 
-ebb1:
+block1:
     ; asm: cmpl %esi, %ecx
     [-,%rflags]         v10 = ifcmp v1, v2      ; bin: 39 f1
     ; asm: cmpl %ecx, %esi
     [-,%rflags]         v11 = ifcmp v2, v1      ; bin: 39 ce
 
-    ; asm: je ebb1
-    brif eq v11, ebb1                           ; bin: 74 fa
-    jump ebb2
+    ; asm: je block1
+    brif eq v11, block1                           ; bin: 74 fa
+    jump block2
 
-ebb2:
-    ; asm: jne ebb1
-    brif ne v11, ebb1                           ; bin: 75 f8
-    jump ebb3
+block2:
+    ; asm: jne block1
+    brif ne v11, block1                           ; bin: 75 f8
+    jump block3
 
-ebb3:
-    ; asm: jl ebb1
-    brif slt v11, ebb1                          ; bin: 7c f6
-    jump ebb4
+block3:
+    ; asm: jl block1
+    brif slt v11, block1                          ; bin: 7c f6
+    jump block4
 
-ebb4:
-    ; asm: jge ebb1
-    brif sge v11, ebb1                          ; bin: 7d f4
-    jump ebb5
+block4:
+    ; asm: jge block1
+    brif sge v11, block1                          ; bin: 7d f4
+    jump block5
 
-ebb5:
-    ; asm: jg ebb1
-    brif sgt v11, ebb1                          ; bin: 7f f2
-    jump ebb6
+block5:
+    ; asm: jg block1
+    brif sgt v11, block1                          ; bin: 7f f2
+    jump block6
 
-ebb6:
-    ; asm: jle ebb1
-    brif sle v11, ebb1                          ; bin: 7e f0
-    jump ebb7
+block6:
+    ; asm: jle block1
+    brif sle v11, block1                          ; bin: 7e f0
+    jump block7
 
-ebb7:
-    ; asm: jb ebb1
-    brif ult v11, ebb1                          ; bin: 72 ee
-    jump ebb8
+block7:
+    ; asm: jb block1
+    brif ult v11, block1                          ; bin: 72 ee
+    jump block8
 
-ebb8:
-    ; asm: jae ebb1
-    brif uge v11, ebb1                          ; bin: 73 ec
-    jump ebb9
+block8:
+    ; asm: jae block1
+    brif uge v11, block1                          ; bin: 73 ec
+    jump block9
 
-ebb9:
-    ; asm: ja ebb1
-    brif ugt v11, ebb1                          ; bin: 77 ea
-    jump ebb10
+block9:
+    ; asm: ja block1
+    brif ugt v11, block1                          ; bin: 77 ea
+    jump block10
 
-ebb10:
-    ; asm: jbe ebb1
-    brif ule v11, ebb1                          ; bin: 76 e8
-    jump ebb11
+block10:
+    ; asm: jbe block1
+    brif ule v11, block1                          ; bin: 76 e8
+    jump block11
 
-ebb11:
+block11:
 
     ; asm: sete %bl
     [-,%rbx]            v20 = trueif eq v11                           ; bin: 0f 94 c3
@@ -690,7 +690,7 @@ ebb11:
 
 ; Tests for i32/i8 conversion instructions.
 function %I32_I8() {
-ebb0:
+block0:
     [-,%rcx]            v1 = iconst.i32 1
 
     [-,%rcx]            v11 = ireduce.i8 v1             ; bin:
@@ -706,7 +706,7 @@ ebb0:
 
 ; Tests for i32/i16 conversion instructions.
 function %I32_I16() {
-ebb0:
+block0:
     [-,%rcx]            v1 = iconst.i32 1
 
     [-,%rcx]            v11 = ireduce.i16 v1            ; bin:
diff --git a/cranelift/filetests/filetests/isa/x86/binary64-float.clif b/cranelift/filetests/filetests/isa/x86/binary64-float.clif
index 171a3db7c9..2ec733b2c0 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64-float.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64-float.clif
@@ -14,7 +14,7 @@ function %F32() {
     ss2 = incoming_arg 1024, offset -2048
     ss3 = incoming_arg 8, offset -2056
 
-ebb0:
+block0:
     [-,%r11]            v0 = iconst.i32 1
     [-,%rsi]            v1 = iconst.i32 2
     [-,%rax]            v2 = iconst.i64 11
@@ -297,7 +297,7 @@ function %F64() {
     ss2 = incoming_arg 1024, offset -2048
     ss3 = incoming_arg 8, offset -2056
 
-ebb0:
+block0:
     [-,%r11]            v0 = iconst.i32 1
     [-,%rsi]            v1 = iconst.i32 2
     [-,%rax]            v2 = iconst.i64 11
@@ -553,53 +553,53 @@ ebb0:
 }
 
 function %cpuflags_float(f32 [%xmm0]) {
-ebb0(v0: f32 [%xmm0]):
+block0(v0: f32 [%xmm0]):
     ; asm: ucomiss %xmm0, %xmm0
     [-,%rflags]         v1 = ffcmp v0, v0                       ; bin: 0f 2e c0
 
-    jump ebb1
+    jump block1
 
-ebb1:
-    ; asm: jnp ebb1
-    brff ord v1, ebb1                                           ; bin: 7b fe
-    jump ebb2
+block1:
+    ; asm: jnp block1
+    brff ord v1, block1                                           ; bin: 7b fe
+    jump block2
 
-ebb2:
-    ; asm: jp ebb1
-    brff uno v1, ebb1                                           ; bin: 7a fc
-    jump ebb3
+block2:
+    ; asm: jp block1
+    brff uno v1, block1                                           ; bin: 7a fc
+    jump block3
 
-ebb3:
-    ; asm: jne ebb1
-    brff one v1, ebb1                                           ; bin: 75 fa
-    jump ebb4
+block3:
+    ; asm: jne block1
+    brff one v1, block1                                           ; bin: 75 fa
+    jump block4
 
-ebb4:
-    ; asm: je ebb1
-    brff ueq v1, ebb1                                           ; bin: 74 f8
-    jump ebb5
+block4:
+    ; asm: je block1
+    brff ueq v1, block1                                           ; bin: 74 f8
+    jump block5
 
-ebb5:
-    ; asm: ja ebb1
-    brff gt v1, ebb1                                            ; bin: 77 f6
-    jump ebb6
+block5:
+    ; asm: ja block1
+    brff gt v1, block1                                            ; bin: 77 f6
+    jump block6
 
-ebb6:
-    ; asm: jae ebb1
-    brff ge v1, ebb1                                            ; bin: 73 f4
-    jump ebb7
+block6:
+    ; asm: jae block1
+    brff ge v1, block1                                            ; bin: 73 f4
+    jump block7
 
-ebb7:
-    ; asm: jb ebb1
-    brff ult v1, ebb1                                           ; bin: 72 f2
-    jump ebb8
+block7:
+    ; asm: jb block1
+    brff ult v1, block1                                           ; bin: 72 f2
+    jump block8
 
-ebb8:
-    ; asm: jbe ebb1
-    brff ule v1, ebb1                                           ; bin: 76 f0
-    jump ebb9
+block8:
+    ; asm: jbe block1
+    brff ule v1, block1                                           ; bin: 76 f0
+    jump block9
 
-ebb9:
+block9:
     ; asm: jp .+4; ud2
     trapff ord v1, user0                                        ; bin: 7a 02 user0 0f 0b
     ; asm: jnp .+4; ud2
diff --git a/cranelift/filetests/filetests/isa/x86/binary64-pic.clif b/cranelift/filetests/filetests/isa/x86/binary64-pic.clif
index 3f3d86288c..5a2443adce 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64-pic.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64-pic.clif
@@ -25,7 +25,7 @@ function %I64() {
     ss2 = incoming_arg 1024, offset -2048
     ss3 = incoming_arg 8, offset -2056
 
-ebb0:
+block0:
 
     ; Colocated functions.
 
diff --git a/cranelift/filetests/filetests/isa/x86/binary64-run.clif b/cranelift/filetests/filetests/isa/x86/binary64-run.clif
index 41db7e1b5a..6e6e1071c5 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64-run.clif
@@ -3,7 +3,7 @@ target x86_64
 
 ; this verifies that returning b64 immediates does not result in a segmentation fault, see https://github.com/bytecodealliance/cranelift/issues/911
 function %test_b64() -> b64 {
-ebb0:
+block0:
 [-, %r10]   v0 = bconst.b64 true
             return v0
 }
diff --git a/cranelift/filetests/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
index a3fcede60d..41290d1462 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64.clif
@@ -23,7 +23,7 @@ function %I64() {
     ss2 = incoming_arg 1024, offset -2048
     ss3 = incoming_arg 8, offset -2056
 
-ebb0:
+block0:
 
     ; Integer Constants.
 
@@ -708,117 +708,117 @@ ebb0:
     istore8_complex v601, v521+v522             ; bin: heap_oob 88 0c 18
 
     ; asm: testq %rcx, %rcx
-    ; asm: je ebb1
-    brz v1, ebb1                                ; bin: 48 85 c9 74 1b
-    fallthrough ebb3
+    ; asm: je block1
+    brz v1, block1                                ; bin: 48 85 c9 74 1b
+    fallthrough block3
 
-ebb3:
+block3:
     ; asm: testq %rsi, %rsi
-    ; asm: je ebb1
-    brz v2, ebb1                                ; bin: 48 85 f6 74 16
-    fallthrough ebb4
+    ; asm: je block1
+    brz v2, block1                                ; bin: 48 85 f6 74 16
+    fallthrough block4
 
-ebb4:
+block4:
     ; asm: testq %r10, %r10
-    ; asm: je ebb1
-    brz v3, ebb1                                ; bin: 4d 85 d2 74 11
-    fallthrough ebb5
+    ; asm: je block1
+    brz v3, block1                                ; bin: 4d 85 d2 74 11
+    fallthrough block5
 
-ebb5:
+block5:
     ; asm: testq %rcx, %rcx
-    ; asm: jne ebb1
-    brnz v1, ebb1                               ; bin: 48 85 c9 75 0c
-    fallthrough ebb6
+    ; asm: jne block1
+    brnz v1, block1                               ; bin: 48 85 c9 75 0c
+    fallthrough block6
 
-ebb6:
+block6:
     ; asm: testq %rsi, %rsi
-    ; asm: jne ebb1
-    brnz v2, ebb1                               ; bin: 48 85 f6 75 07
-    fallthrough ebb7
+    ; asm: jne block1
+    brnz v2, block1                               ; bin: 48 85 f6 75 07
+    fallthrough block7
 
-ebb7:
+block7:
     ; asm: testq %r10, %r10
-    ; asm: jne ebb1
-    brnz v3, ebb1                               ; bin: 4d 85 d2 75 02
+    ; asm: jne block1
+    brnz v3, block1                               ; bin: 4d 85 d2 75 02
 
-    ; asm: jmp ebb2
-    jump ebb2                                   ; bin: eb 01
+    ; asm: jmp block2
+    jump block2                                   ; bin: eb 01
 
-    ; asm: ebb1:
-ebb1:
+    ; asm: block1:
+block1:
     return                                      ; bin: c3
 
-    ; asm: ebb2:
-ebb2:
+    ; asm: block2:
+block2:
     ; Add a no-op instruction to prevent fold_redundant_jump from removing this block.
     ; asm: notq %rcx
     [-,%rcx]             v5000 = bnot v1        ; bin: 48 f7 d1
-    jump ebb1                                   ; bin: eb fa
+    jump block1                                   ; bin: eb fa
 }
 
 ; CPU flag instructions.
 function %cpu_flags_I64() {
-ebb0:
+block0:
     [-,%rcx]            v1 = iconst.i64 1
     [-,%r10]            v2 = iconst.i64 2
-    jump ebb1
+    jump block1
 
-ebb1:
+block1:
     ; asm: cmpq %r10, %rcx
     [-,%rflags]         v10 = ifcmp v1, v2      ; bin: 4c 39 d1
     ; asm: cmpq %rcx, %r10
     [-,%rflags]         v11 = ifcmp v2, v1      ; bin: 49 39 ca
 
-    ; asm: je ebb1
-    brif eq v11, ebb1                           ; bin: 74 f8
-    jump ebb2
+    ; asm: je block1
+    brif eq v11, block1                           ; bin: 74 f8
+    jump block2
 
-ebb2:
-    ; asm: jne ebb1
-    brif ne v11, ebb1                           ; bin: 75 f6
-    jump ebb3
+block2:
+    ; asm: jne block1
+    brif ne v11, block1                           ; bin: 75 f6
+    jump block3
 
-ebb3:
-    ; asm: jl ebb1
-    brif slt v11, ebb1                          ; bin: 7c f4
-    jump ebb4
+block3:
+    ; asm: jl block1
+    brif slt v11, block1                          ; bin: 7c f4
+    jump block4
 
-ebb4:
-    ; asm: jge ebb1
-    brif sge v11, ebb1                          ; bin: 7d f2
-    jump ebb5
+block4:
+    ; asm: jge block1
+    brif sge v11, block1                          ; bin: 7d f2
+    jump block5
 
-ebb5:
-    ; asm: jg ebb1
-    brif sgt v11, ebb1                          ; bin: 7f f0
-    jump ebb6
+block5:
+    ; asm: jg block1
+    brif sgt v11, block1                          ; bin: 7f f0
+    jump block6
 
-ebb6:
-    ; asm: jle ebb1
-    brif sle v11, ebb1                          ; bin: 7e ee
-    jump ebb7
+block6:
+    ; asm: jle block1
+    brif sle v11, block1                          ; bin: 7e ee
+    jump block7
 
-ebb7:
-    ; asm: jb ebb1
-    brif ult v11, ebb1                          ; bin: 72 ec
-    jump ebb8
+block7:
+    ; asm: jb block1
+    brif ult v11, block1                          ; bin: 72 ec
+    jump block8
 
-ebb8:
-    ; asm: jae ebb1
-    brif uge v11, ebb1                          ; bin: 73 ea
-    jump ebb9
+block8:
+    ; asm: jae block1
+    brif uge v11, block1                          ; bin: 73 ea
+    jump block9
 
-ebb9:
-    ; asm: ja ebb1
-    brif ugt v11, ebb1                          ; bin: 77 e8
-    jump ebb10
+block9:
+    ; asm: ja block1
+    brif ugt v11, block1                          ; bin: 77 e8
+    jump block10
 
-ebb10:
-    ; asm: jbe ebb1
-    brif ule v11, ebb1                          ; bin: 76 e6
-    jump ebb11
+block10:
+    ; asm: jbe block1
+    brif ule v11, block1                          ; bin: 76 e6
+    jump block11
 
-ebb11:
+block11:
 
     ; asm: sete %bl
     [-,%rbx]            v20 = trueif eq v11                           ; bin: 0f 94 c3
@@ -896,7 +896,7 @@ function %outargs() {
     ss1 = outgoing_arg 8, offset 8
     ss2 = outgoing_arg 8, offset 0
 
-ebb0:
+block0:
     [-,%rcx]            v1 = iconst.i64 1
 
     ; asm: movq %rcx, 8(%rsp)
@@ -922,7 +922,7 @@ function %I32() {
     ss2 = incoming_arg 1024, offset -2048
     ss3 = incoming_arg 8, offset -2056
 
-ebb0:
+block0:
 
     ; Integer Constants.
 
@@ -1318,58 +1318,58 @@ ebb0:
     [-,%r10]             v533 = ushr_imm v3, 31   ; bin: 41 c1 ea 1f
 
     ; asm: testl %ecx, %ecx
-    ; asm: je ebb1x
-    brz v1, ebb1                                ; bin: 85 c9 74 18
-    fallthrough ebb3
+    ; asm: je block1x
+    brz v1, block1                                ; bin: 85 c9 74 18
+    fallthrough block3
 
-ebb3:
+block3:
     ; asm: testl %esi, %esi
-    ; asm: je ebb1x
-    brz v2, ebb1                                ; bin: 85 f6 74 14
-    fallthrough ebb4
+    ; asm: je block1x
+    brz v2, block1                                ; bin: 85 f6 74 14
+    fallthrough block4
 
-ebb4:
+block4:
     ; asm: testl %r10d, %r10d
-    ; asm: je ebb1x
-    brz v3, ebb1                                ; bin: 45 85 d2 74 0f
-    fallthrough ebb5
+    ; asm: je block1x
+    brz v3, block1                                ; bin: 45 85 d2 74 0f
+    fallthrough block5
 
-ebb5:
+block5:
     ; asm: testl %ecx, %ecx
-    ; asm: jne ebb1x
-    brnz v1, ebb1                               ; bin: 85 c9 75 0b
-    fallthrough ebb6
+    ; asm: jne block1x
+    brnz v1, block1                               ; bin: 85 c9 75 0b
+    fallthrough block6
 
-ebb6:
+block6:
     ; asm: testl %esi, %esi
-    ; asm: jne ebb1x
-    brnz v2, ebb1                               ; bin: 85 f6 75 07
-    fallthrough ebb7
+    ; asm: jne block1x
+    brnz v2, block1                               ; bin: 85 f6 75 07
+    fallthrough block7
 
-ebb7:
+block7:
     ; asm: testl %r10d, %r10d
-    ; asm: jne ebb1x
-    brnz v3, ebb1                               ; bin: 45 85 d2 75 02
+    ; asm: jne block1x
+    brnz v3, block1                               ; bin: 45 85 d2 75 02
 
-    ; asm: jmp ebb2x
-    jump ebb2                                   ; bin: eb 01
+    ; asm: jmp block2x
+    jump block2                                   ; bin: eb 01
 
-    ; asm: ebb1x:
-ebb1:
+    ; asm: block1x:
+block1:
     return                                      ; bin: c3
 
-    ; asm: ebb2x:
-ebb2:
+    ; asm: block2x:
+block2:
     ; Add a no-op instruction to prevent fold_redundant_jump from removing this block.
     ; asm: notl %ecx
     [-,%rcx]             v5000 = bnot v1        ; bin: f7 d1
-    jump ebb1                                   ; bin: eb fb
+    jump block1                                   ; bin: eb fb
 
 }
 
 ; Tests for i32/i8 conversion instructions.
 function %I32_I8() {
-ebb0:
+block0:
     [-,%rcx]            v1 = iconst.i32 1
     [-,%rsi]            v2 = iconst.i32 2
     [-,%r10]            v3 = iconst.i32 3
@@ -1397,7 +1397,7 @@ ebb0:
 
 ; Tests for i32/i16 conversion instructions.
 function %I32_I16() {
-ebb0:
+block0:
     [-,%rcx]            v1 = iconst.i32 1
     [-,%rsi]            v2 = iconst.i32 2
     [-,%r10]            v3 = iconst.i32 3
@@ -1425,7 +1425,7 @@ ebb0:
 
 ; Tests for i64/i8 conversion instructions.
 function %I64_I8() {
-ebb0:
+block0:
     [-,%rcx]            v1 = iconst.i64 1
     [-,%rsi]            v2 = iconst.i64 2
     [-,%r10]            v3 = iconst.i64 3
@@ -1453,7 +1453,7 @@ ebb0:
 
 ; Tests for i64/i16 conversion instructions.
 function %I64_I16() {
-ebb0:
+block0:
     [-,%rcx]            v1 = iconst.i64 1
     [-,%rsi]            v2 = iconst.i64 2
     [-,%r10]            v3 = iconst.i64 3
@@ -1481,7 +1481,7 @@ ebb0:
 
 ; Tests for i64/i32 conversion instructions.
 function %I64_I32() {
-ebb0:
+block0:
     [-,%rcx]            v1 = iconst.i64 1
     [-,%rsi]            v2 = iconst.i64 2
     [-,%r10]            v3 = iconst.i64 3
@@ -1509,9 +1509,9 @@ ebb0:
 
 ; Tests for i64 jump table instructions.
 function %I64_JT(i64 [%rdi]) {
-    jt0 = jump_table [ebb1, ebb2, ebb3]
+    jt0 = jump_table [block1, block2, block3]
 
-ebb0(v0: i64 [%rdi]):
+block0(v0: i64 [%rdi]):
     ; Note: The next two lines will need to change whenever instructions are
     ;        added or removed from this test.
     [-, %rax]           v1 = jump_table_base.i64 jt0    ; bin: 48 8d 05 00000039 PCRelRodata4(jt0)
@@ -1530,23 +1530,23 @@ ebb0(v0: i64 [%rdi]):
     [-, %r10]           v32 = jump_table_entry.i64 v11, v1, 4, jt0      ; bin: 4e 63 14 a8
     [-, %r10]           v33 = jump_table_entry.i64 v11, v2, 4, jt0      ; bin: 4f 63 14 aa
 
-    fallthrough ebb10
+    fallthrough block10
 
-ebb10:
+block10:
     indirect_jump_table_br v10, jt0             ; bin: ff e3
-ebb11:
+block11:
     indirect_jump_table_br v11, jt0             ; bin: 41 ff e5
 
-ebb1:
-    fallthrough ebb2
-ebb2:
-    fallthrough ebb3
-ebb3:
+block1:
+    fallthrough block2
+block2:
+    fallthrough block3
+block3:
     trap user0
 }
 
 function %r12_r13_loads() {
-ebb0:
+block0:
     [-,%r12]            v1 = iconst.i64 0x0123_4567_89ab_cdef
     [-,%r13]            v2 = iconst.i64 0xfedc_ba98_7654_3210
     [-,%rax]            v3 = iconst.i64 0x1
@@ -1603,7 +1603,7 @@ ebb0:
 }
 
 function %r12_r13_stores() {
-ebb0:
+block0:
     [-,%r12]            v1 = iconst.i64 0x0123_4567_89ab_cdef
     [-,%r13]            v2 = iconst.i64 0xfedc_ba98_7654_3210
     [-,%rax]            v3 = iconst.i64 0x1
@@ -1674,7 +1674,7 @@ ebb0:
 }
 
 function %B64() {
-ebb0:
+block0:
     [-, %rax]   v1 = bconst.b64 true    ; bin: 40 b8 00000001
     [-, %r10]   v0 = bconst.b64 true    ; bin: 41 ba 00000001
     return
diff --git a/cranelift/filetests/filetests/isa/x86/bitrev-i128-run.clif b/cranelift/filetests/filetests/isa/x86/bitrev-i128-run.clif
index effe502d9e..4d3fe2ef57 100644
--- a/cranelift/filetests/filetests/isa/x86/bitrev-i128-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/bitrev-i128-run.clif
@@ -2,7 +2,7 @@ test run
 target x86_64
 
 function %reverse_bits_zero() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = iconcat v0, v0
     v2 = bitrev.i128 v1
@@ -12,7 +12,7 @@ ebb0:
 ; run
 
 function %reverse_bits_one() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = iconst.i64 1
     v2 = iconcat v0, v1
@@ -29,7 +29,7 @@ ebb0:
 ; run
 
 function %reverse_bits() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i64 0x06AD_8667_69EC_41BA
     v1 = iconst.i64 0x6C83_D81A_6E28_83AB
     v2 = iconcat v0, v1
diff --git a/cranelift/filetests/filetests/isa/x86/br-i128-run.clif b/cranelift/filetests/filetests/isa/x86/br-i128-run.clif
index bde3238462..95a1de81cf 100644
--- a/cranelift/filetests/filetests/isa/x86/br-i128-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/br-i128-run.clif
@@ -2,36 +2,36 @@ test run
 target x86_64
 
 function %br_false() -> b1 {
-ebb0:
+block0:
     v10 = iconst.i64 0x42
     v11 = iconst.i64 0x00
     v0 = iconcat v10, v11
-    brz v0, ebb2
-    jump ebb1
+    brz v0, block2
+    jump block1
 
-ebb1:
+block1:
     v1 = bconst.b1 true
     return v1
 
-ebb2:
+block2:
     v2 = bconst.b1 false
     return v2
 }
 ; run
 
 function %br_true() -> b1 {
-ebb0:
+block0:
     v10 = iconst.i64 0x00
     v11 = iconst.i64 0x00
     v0 = iconcat v10, v11
-    brz v0, ebb2
-    jump ebb1
+    brz v0, block2
+    jump block1
 
-ebb1:
+block1:
     v1 = bconst.b1 false
     return v1
 
-ebb2:
+block2:
     v2 = bconst.b1 true
     return v2
 }
diff --git a/cranelift/filetests/filetests/isa/x86/br-i128.clif b/cranelift/filetests/filetests/isa/x86/br-i128.clif
index 442be8d6c8..a1778f4cf5 100644
--- a/cranelift/filetests/filetests/isa/x86/br-i128.clif
+++ b/cranelift/filetests/filetests/isa/x86/br-i128.clif
@@ -2,41 +2,41 @@ test compile
 target x86_64
 
 function u0:0(i128) -> i8 fast {
-ebb0(v0: i128):
-    brz v0, ebb2
+block0(v0: i128):
+    brz v0, block2
     ; check: v0 = iconcat v3, v4
     ; nextln: v5 = icmp_imm eq v3, 0
     ; nextln: v6 = icmp_imm eq v4, 0
     ; nextln: v7 = band v5, v6
-    ; nextln: brnz v7, ebb2
-    jump ebb1
+    ; nextln: brnz v7, block2
+    jump block1
 
-ebb1:
+block1:
     v1 = iconst.i8 0
     return v1
 
-ebb2:
+block2:
     v2 = iconst.i8 1
     return v2
 }
 
 function u0:1(i128) -> i8 fast {
-ebb0(v0: i128):
-    brnz v0, ebb2
+block0(v0: i128):
+    brnz v0, block2
     ; check: v0 = iconcat v3, v4
-    ; nextln: brnz v3, ebb2
-    ; nextln: fallthrough ebb3
+    ; nextln: brnz v3, block2
+    ; nextln: fallthrough block3
 
-    ; check: ebb3:
-    ; nextln: brnz.i64 v4, ebb2
-    jump ebb1
-    ; nextln: fallthrough ebb1
+    ; check: block3:
+    ; nextln: brnz.i64 v4, block2
+    jump block1
+    ; nextln: fallthrough block1
 
-ebb1:
+block1:
     v1 = iconst.i8 0
     return v1
 
-ebb2:
+block2:
     v2 = iconst.i8 1
     return v2
 }
diff --git a/cranelift/filetests/filetests/isa/x86/brz-i8-run.clif b/cranelift/filetests/filetests/isa/x86/brz-i8-run.clif
index c050a91425..c8520830e6 100644
--- a/cranelift/filetests/filetests/isa/x86/brz-i8-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/brz-i8-run.clif
@@ -2,32 +2,32 @@ test run
 target x86_64
 
 function u0:0() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i8 0
-    brz v0, ebb1
-    jump ebb2
+    brz v0, block1
+    jump block2
 
-ebb1:
+block1:
     v1 = bconst.b1 true
     return v1
 
-ebb2:
+block2:
     v2 = bconst.b1 false
     return v2
 }
 ; run
 
 function u0:1() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i8 0
-    brnz v0, ebb1
-    jump ebb2
+    brnz v0, block1
+    jump block2
 
-ebb1:
+block1:
     v1 = bconst.b1 false
     return v1
 
-ebb2:
+block2:
     v2 = bconst.b1 true
     return v2
 }
diff --git a/cranelift/filetests/filetests/isa/x86/brz-i8.clif b/cranelift/filetests/filetests/isa/x86/brz-i8.clif
index a5cc03c985..6c2f95c359 100644
--- a/cranelift/filetests/filetests/isa/x86/brz-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/brz-i8.clif
@@ -2,37 +2,37 @@ test compile
 target x86_64
 
 function u0:0() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i8 0
     ; check: v0 = iconst.i8 0
-    brz v0, ebb1
+    brz v0, block1
     ; nextln: v3 = uextend.i32 v0
-    ; nextln: brz v3, ebb1
-    jump ebb2
+    ; nextln: brz v3, block1
+    jump block2
 
-ebb1:
+block1:
     v1 = bconst.b1 true
     return v1
 
-ebb2:
+block2:
     v2 = bconst.b1 false
     return v2
 }
 
 function u0:1() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i8 0
     ; check: v0 = iconst.i8 0
-    brnz v0, ebb1
+    brnz v0, block1
     ; nextln: v3 = uextend.i32 v0
-    ; nextln: brnz v3, ebb1
-    jump ebb2
+    ; nextln: brnz v3, block1
+    jump block2
 
-ebb1:
+block1:
     v1 = bconst.b1 false
     return v1
 
-ebb2:
+block2:
     v2 = bconst.b1 true
     return v2
 }
diff --git a/cranelift/filetests/filetests/isa/x86/compile-vconst.clif b/cranelift/filetests/filetests/isa/x86/compile-vconst.clif
index ee6dff07db..f2cb9259e6 100644
--- a/cranelift/filetests/filetests/isa/x86/compile-vconst.clif
+++ b/cranelift/filetests/filetests/isa/x86/compile-vconst.clif
@@ -5,12 +5,12 @@ target x86_64 haswell
 
 ; use baldrdash calling convention here for simplicity (avoids prologue, epilogue)
 function %test_vconst_i32() -> i32x4 baldrdash_system_v {
-ebb0:
+block0:
     v0 = vconst.i32x4 0x1234
     return v0
 }
 
-; check:   ebb0:
+; check:   block0:
 ; nextln:     v0 = vconst.i32x4 0x1234
 ; nextln:     return v0
 ; nextln: }
diff --git a/cranelift/filetests/filetests/isa/x86/extend-i128-run.clif b/cranelift/filetests/filetests/isa/x86/extend-i128-run.clif
index c2550b8f4e..3626e5ebf4 100644
--- a/cranelift/filetests/filetests/isa/x86/extend-i128-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/extend-i128-run.clif
@@ -2,7 +2,7 @@ test run
 target x86_64
 
 function u0:0() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i64 0xffff_ffff_eeee_0000
     v1 = uextend.i128 v0
     v2, v3 = isplit v1
@@ -14,7 +14,7 @@ ebb0:
 ; run
 
 function u0:1() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i64 0xffff_ffff_eeee_0000
     v1 = sextend.i128 v0
     v2, v3 = isplit v1
diff --git a/cranelift/filetests/filetests/isa/x86/extend-i128.clif b/cranelift/filetests/filetests/isa/x86/extend-i128.clif
index 0d9e4c8aa9..db2b53276a 100644
--- a/cranelift/filetests/filetests/isa/x86/extend-i128.clif
+++ b/cranelift/filetests/filetests/isa/x86/extend-i128.clif
@@ -2,7 +2,7 @@ test compile
 target x86_64
 
 function u0:0() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i64 0xffff_ffff_eeee_0000
     ; check: v0 = iconst.i64 0xffff_ffff_eeee_0000
     ; nextln: v2 -> v0
@@ -20,7 +20,7 @@ ebb0:
 }
 
 function u0:1() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i64 0xffff_ffff_eeee_0000
     ; check: v0 = iconst.i64 0xffff_ffff_eeee_0000
     ; nextln: v2 -> v0
diff --git a/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif b/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif
index d1478b99d0..84140a23bd 100644
--- a/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/extractlane-binemit.clif
@@ -6,7 +6,7 @@ target x86_64 haswell
 ; booleans use x86_pextr which is manually placed in the IR so that it can be binemit-tested
 
 function %test_extractlane_b8() {
-ebb0:
+block0:
 [-, %rax]   v0 = bconst.b8 true
 [-, %xmm0]  v1 = splat.b8x16 v0
 [-, %rax]   v2 = x86_pextr v1, 10    ; bin: 66 0f 3a 14 c0 0a
@@ -14,7 +14,7 @@ ebb0:
 }
 
 function %test_extractlane_i16() {
-ebb0:
+block0:
 [-, %rax]   v0 = iconst.i16 4
 [-, %xmm1]  v1 = splat.i16x8 v0
 [-, %rax]   v2 = x86_pextr v1, 4    ; bin: 66 0f 3a 15 c8 04
@@ -22,7 +22,7 @@ ebb0:
 }
 
 function %test_extractlane_i32() {
-ebb0:
+block0:
 [-, %rax]   v0 = iconst.i32 42
 [-, %xmm4]  v1 = splat.i32x4 v0
 [-, %rcx]   v2 = x86_pextr v1, 2    ; bin: 66 0f 3a 16 e1 02
@@ -30,7 +30,7 @@ ebb0:
 }
 
 function %test_extractlane_b64() {
-ebb0:
+block0:
 [-, %rax]   v0 = bconst.b64 false
 [-, %xmm2]  v1 = splat.b64x2 v0
 [-, %rbx]   v2 = x86_pextr v1, 1    ; bin: 66 48 0f 3a 16 d3 01
diff --git a/cranelift/filetests/filetests/isa/x86/extractlane-run.clif b/cranelift/filetests/filetests/isa/x86/extractlane-run.clif
index adb2e7b8e6..4e1d735bfe 100644
--- a/cranelift/filetests/filetests/isa/x86/extractlane-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/extractlane-run.clif
@@ -2,7 +2,7 @@ test run
 set enable_simd
 
 function %test_extractlane_b8() -> b8 {
-ebb0:
+block0:
     v1 = vconst.b8x16 [false false false false false false false false false false true false false
     false false false]
     v2 = extractlane v1, 10
@@ -11,7 +11,7 @@ ebb0:
 ; run
 
 function %test_extractlane_i16() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i16x8 0x00080007000600050004000300020001
     v1 = extractlane v0, 1
     v2 = icmp_imm eq v1, 2
@@ -20,7 +20,7 @@ ebb0:
 ; run
 
 function %test_extractlane_f32() -> b1 {
-ebb0:
+block0:
     v0 = f32const 0x42.42
     v1 = vconst.f32x4 [0x00.00 0x00.00 0x00.00 0x42.42]
     v2 = extractlane v1, 3
@@ -30,7 +30,7 @@ ebb0:
 ; run
 
 function %test_extractlane_i32_with_vector_reuse() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i32 42
     v1 = iconst.i32 99
 
@@ -49,7 +49,7 @@ ebb0:
 ; run
 
 function %test_extractlane_f32_with_vector_reuse() -> b1 {
-ebb0:
+block0:
     v0 = f32const 0x42.42
     v1 = f32const 0x99.99
 
diff --git a/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants-32bit.clif b/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants-32bit.clif
index 8021375558..4d736287e0 100644
--- a/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants-32bit.clif
+++ b/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants-32bit.clif
@@ -3,14 +3,14 @@ test binemit
 target i686
 
 function %foo() -> f32 fast {
-ebb0:
+block0:
   ; asm: xorps %xmm0, %xmm0
   [-,%xmm0]    v0 = f32const 0.0     ; bin: 0f 57 c0
   return v0
 }
 
 function %bar() -> f64 fast {
-ebb0:
+block0:
   ; asm: xorpd %xmm0, %xmm0
   [-,%xmm0]    v1 = f64const 0.0     ; bin: 66 0f 57 c0
   return v1
diff --git a/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants.clif b/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants.clif
index 049320870e..25cd686996 100644
--- a/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants.clif
+++ b/cranelift/filetests/filetests/isa/x86/floating-point-zero-constants.clif
@@ -3,28 +3,28 @@ test binemit
 target x86_64
 
 function %zero_const_32bit_no_rex() -> f32 fast {
-ebb0:
+block0:
   ; asm: xorps %xmm0, %xmm0
   [-,%xmm0]    v0 = f32const 0.0     ; bin: 40 0f 57 c0
   return v0
 }
 
 function %zero_const_32bit_rex() -> f32 fast {
-ebb0:
+block0:
   ; asm: xorps %xmm8, %xmm8
   [-,%xmm8]    v1 = f32const 0.0     ; bin: 45 0f 57 c0
   return v1
 }
 
 function %zero_const_64bit_no_rex() -> f64 fast {
-ebb0:
+block0:
   ; asm: xorpd %xmm0, %xmm0
   [-,%xmm0]    v0 = f64const 0.0     ; bin: 66 40 0f 57 c0
   return v0
 }
 
 function %zero_const_64bit_rex() -> f64 fast {
-ebb0:
+block0:
   ; asm: xorpd %xmm8, %xmm8
   [-,%xmm8]    v1 = f64const 0.0     ; bin: 66 45 0f 57 c0
   return v1
diff --git a/cranelift/filetests/filetests/isa/x86/i128-isplit-forward-jump.clif b/cranelift/filetests/filetests/isa/x86/i128-isplit-forward-jump.clif
index b5144203ac..493d2e6365 100644
--- a/cranelift/filetests/filetests/isa/x86/i128-isplit-forward-jump.clif
+++ b/cranelift/filetests/filetests/isa/x86/i128-isplit-forward-jump.clif
@@ -2,24 +2,24 @@ test compile
 target x86_64
 
 function u0:0() -> i128 system_v {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = iconst.i64 0
     v2 = iconcat v0, v1
-    jump ebb5
+    jump block5
 
-ebb2:
-    jump ebb4(v27)
+block2:
+    jump block4(v27)
 
-ebb4(v23: i128):
+block4(v23: i128):
     return v23
 
-ebb5:
+block5:
     v27 = bxor.i128 v2, v2
     v32 = iconst.i32 0
-    brz v32, ebb2
-    jump ebb6
+    brz v32, block2
+    jump block6
 
-ebb6:
+block6:
     trap user0
 }
diff --git a/cranelift/filetests/filetests/isa/x86/i128.clif b/cranelift/filetests/filetests/isa/x86/i128.clif
index b710a7430e..028fb6e551 100644
--- a/cranelift/filetests/filetests/isa/x86/i128.clif
+++ b/cranelift/filetests/filetests/isa/x86/i128.clif
@@ -2,8 +2,8 @@ test compile
 target x86_64
 
 function u0:0(i64, i64) -> i128 fast {
-ebb0(v0: i64, v1: i64):
-;check: ebb0(v0: i64 [%rdi], v1: i64 [%rsi], v3: i64 [%rbp]):
+block0(v0: i64, v1: i64):
+;check: block0(v0: i64 [%rdi], v1: i64 [%rsi], v3: i64 [%rbp]):
 
     v2 = iconcat.i64 v0, v1
     ; check: regmove v0, %rdi -> %rax
@@ -15,8 +15,8 @@ ebb0(v0: i64, v1: i64):
 }
 
 function u0:1(i128) -> i64, i64 fast {
-ebb0(v0: i128):
-; check: ebb0(v3: i64 [%rdi], v4: i64 [%rsi], v5: i64 [%rbp]):
+block0(v0: i128):
+; check: block0(v3: i64 [%rdi], v4: i64 [%rsi], v5: i64 [%rbp]):
 
     v1, v2 = isplit v0
     ; check: regmove v3, %rdi -> %rax
@@ -28,8 +28,8 @@ ebb0(v0: i128):
 }
 
 function u0:2(i64, i128) fast {
-; check: ebb0(v0: i64 [%rdi], v2: i64 [%rsi], v3: i64 [%rdx], v6: i64 [%rbp]):
-ebb0(v0: i64, v1: i128):
+; check: block0(v0: i64 [%rdi], v2: i64 [%rsi], v3: i64 [%rdx], v6: i64 [%rbp]):
+block0(v0: i64, v1: i128):
     ; check: store v2, v0+8
     ; check: store v3, v0+16
     store v1, v0+8
@@ -37,7 +37,7 @@ ebb0(v0: i64, v1: i128):
 }
 
 function u0:3(i64) -> i128 fast {
-ebb0(v0: i64):
+block0(v0: i64):
     ; check: v2 = load.i64 v0+8
     ; check: v3 = load.i64 v0+16
     v1 = load.i128 v0+8
diff --git a/cranelift/filetests/filetests/isa/x86/icmp-compile.clif b/cranelift/filetests/filetests/isa/x86/icmp-compile.clif
index cf9cb3ff07..4a4ac0fc59 100644
--- a/cranelift/filetests/filetests/isa/x86/icmp-compile.clif
+++ b/cranelift/filetests/filetests/isa/x86/icmp-compile.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64 skylake
 
 function %icmp_i8x16() {
-ebb0:
+block0:
 [-, %xmm3]  v0 = vconst.i8x16 0x00                                  ; bin: 66 0f ef db
 [-, %xmm4]  v1 = vconst.i8x16 0xffffffffffffffffffffffffffffffff    ; bin: 66 0f 74 e4
 [-, %xmm3]  v2 = icmp eq v0, v1                                     ; bin: 66 0f 74 dc
@@ -11,7 +11,7 @@ ebb0:
 }
 
 function %icmp_i16x8() {
-ebb0:
+block0:
 [-, %xmm0]  v0 = vconst.i16x8 0x00
 [-, %xmm7]  v1 = vconst.i16x8 0xffffffffffffffffffffffffffffffff
 [-, %xmm0]  v2 = icmp eq v0, v1                                     ; bin: 66 0f 75 c7
@@ -19,7 +19,7 @@ ebb0:
 }
 
 function %icmp_i32x4() {
-ebb0:
+block0:
 [-, %xmm0]  v0 = vconst.i32x4 0x00
 [-, %xmm4]  v1 = vconst.i32x4 0xffffffffffffffffffffffffffffffff
 [-, %xmm0]  v2 = icmp eq v0, v1                                     ; bin: 66 0f 76 c4
@@ -27,7 +27,7 @@ ebb0:
 }
 
 function %icmp_i64x2() {
-ebb0:
+block0:
 [-, %xmm0]  v0 = vconst.i64x2 0x00
 [-, %xmm1]  v1 = vconst.i64x2 0xffffffffffffffffffffffffffffffff
 [-, %xmm0]  v2 = icmp eq v0, v1                                     ; bin: 66 0f 38 29 c1
diff --git a/cranelift/filetests/filetests/isa/x86/icmp-i128.clif b/cranelift/filetests/filetests/isa/x86/icmp-i128.clif
index 8816d22d69..dce0e1db87 100644
--- a/cranelift/filetests/filetests/isa/x86/icmp-i128.clif
+++ b/cranelift/filetests/filetests/isa/x86/icmp-i128.clif
@@ -2,7 +2,7 @@ test run
 target x86_64 haswell
 
 function %test_icmp_eq_i128() -> b1 {
-ebb0:
+block0:
     v11 = iconst.i64 0x0
     v12 = iconst.i64 0x0
     v1 = iconcat v11, v12
@@ -16,7 +16,7 @@ ebb0:
 ; run
 
 function %test_icmp_imm_eq_i128() -> b1 {
-ebb0:
+block0:
     v11 = iconst.i64 0x0
     v12 = iconst.i64 0x0
     v1 = iconcat v11, v12
@@ -27,7 +27,7 @@ ebb0:
 ; run
 
 function %test_icmp_ne_i128() -> b1 {
-ebb0:
+block0:
     v11 = iconst.i64 0x0
     v12 = iconst.i64 0x0
     v1 = iconcat v11, v12
@@ -41,7 +41,7 @@ ebb0:
 ; run
 
 function %test_icmp_imm_ne_i128() -> b1 {
-ebb0:
+block0:
     v11 = iconst.i64 0x0
     v12 = iconst.i64 0x0
     v1 = iconcat v11, v12
diff --git a/cranelift/filetests/filetests/isa/x86/icmp-run.clif b/cranelift/filetests/filetests/isa/x86/icmp-run.clif
index c470af662a..0820cac013 100644
--- a/cranelift/filetests/filetests/isa/x86/icmp-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/icmp-run.clif
@@ -2,7 +2,7 @@ test run
 set enable_simd
 
 function %run_icmp_i8x16() -> b8 {
-ebb0:
+block0:
     v0 = vconst.i8x16 0x00
     v1 = vconst.i8x16 0x00
     v2 = icmp eq v0, v1
@@ -13,7 +13,7 @@ ebb0:
 ; run
 
 function %run_icmp_i64x2() -> b64 {
-ebb0:
+block0:
     v0 = vconst.i64x2 0xffffffffffffffffffffffffffffffff
     v1 = vconst.i64x2 0xffffffffffffffffffffffffffffffff
     v2 = icmp eq v0, v1
diff --git a/cranelift/filetests/filetests/isa/x86/imul-i128.clif b/cranelift/filetests/filetests/isa/x86/imul-i128.clif
index 2d683a32dd..65d21463fd 100644
--- a/cranelift/filetests/filetests/isa/x86/imul-i128.clif
+++ b/cranelift/filetests/filetests/isa/x86/imul-i128.clif
@@ -2,7 +2,7 @@ test run
 target x86_64 haswell
 
 function %test_imul_i128() -> b1 {
-ebb0:
+block0:
     v11 = iconst.i64 0xf2347ac4503f1e24
     v12 = iconst.i64 0x0098fe985354ab06
     v1 = iconcat v11, v12
diff --git a/cranelift/filetests/filetests/isa/x86/insertlane-binemit.clif b/cranelift/filetests/filetests/isa/x86/insertlane-binemit.clif
index c388ed6fae..4be35a47b3 100644
--- a/cranelift/filetests/filetests/isa/x86/insertlane-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/insertlane-binemit.clif
@@ -6,7 +6,7 @@ target x86_64 haswell
 ; booleans use x86_pinsr which is manually placed in the IR so that it can be binemit-tested
 
 function %test_insertlane_b8() {
-ebb0:
+block0:
 [-, %rax]   v0 = bconst.b8 true
 [-, %rbx]   v1 = bconst.b8 false
 [-, %xmm0]  v2 = splat.b8x16 v0
@@ -15,7 +15,7 @@ ebb0:
 }
 
 function %test_insertlane_i16() {
-ebb0:
+block0:
 [-, %rax]   v0 = iconst.i16 4
 [-, %rbx]   v1 = iconst.i16 5
 [-, %xmm1]  v2 = splat.i16x8 v0
@@ -24,7 +24,7 @@ ebb0:
 }
 
 function %test_insertlane_i32() {
-ebb0:
+block0:
 [-, %rax]   v0 = iconst.i32 42
 [-, %rbx]   v1 = iconst.i32 99
 [-, %xmm4]  v2 = splat.i32x4 v0
@@ -33,7 +33,7 @@ ebb0:
 }
 
 function %test_insertlane_b64() {
-ebb0:
+block0:
 [-, %rax]   v0 = bconst.b64 true
 [-, %rbx]   v1 = bconst.b64 false
 [-, %xmm2]  v2 = splat.b64x2 v0
diff --git a/cranelift/filetests/filetests/isa/x86/insertlane-run.clif b/cranelift/filetests/filetests/isa/x86/insertlane-run.clif
index 92fb38202e..8f1cd7ef46 100644
--- a/cranelift/filetests/filetests/isa/x86/insertlane-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/insertlane-run.clif
@@ -4,7 +4,7 @@ set enable_simd
 ; TODO once SIMD vector comparison is implemented, remove use of extractlane below
 
 function %test_insertlane_b8() -> b8 {
-ebb0:
+block0:
     v1 = bconst.b8 true
     v2 = vconst.b8x16 [false false false false false false false false false false false false false
      false false false]
@@ -15,7 +15,7 @@ ebb0:
 ; run
 
 function %test_insertlane_f32() -> b1 {
-ebb0:
+block0:
     v0 = f32const 0x42.42
     v1 = vconst.f32x4 0x00
     v2 = insertlane v1, 1, v0
@@ -26,7 +26,7 @@ ebb0:
 ; run
 
 function %test_insertlane_f64_lane1() -> b1 {
-ebb0:
+block0:
     v0 = f64const 0x42.42
     v1 = vconst.f64x2 0x00
     v2 = insertlane v1, 1, v0
@@ -37,7 +37,7 @@ ebb0:
 ; run
 
 function %test_insertlane_f64_lane0() -> b1 {
-ebb0:
+block0:
     v0 = f64const 0x42.42
     v1 = vconst.f64x2 0x00
     v2 = insertlane v1, 0, v0
diff --git a/cranelift/filetests/filetests/isa/x86/ireduce-i16-to-i8.clif b/cranelift/filetests/filetests/isa/x86/ireduce-i16-to-i8.clif
index 0f8303dfc4..2a283af485 100644
--- a/cranelift/filetests/filetests/isa/x86/ireduce-i16-to-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/ireduce-i16-to-i8.clif
@@ -2,7 +2,7 @@ test compile
 target x86_64
 
 function u0:0(i16) -> i8 fast {
-ebb0(v0: i16):
+block0(v0: i16):
     v1 = ireduce.i8 v0
     return v1
 }
diff --git a/cranelift/filetests/filetests/isa/x86/isplit-not-legalized-twice.clif b/cranelift/filetests/filetests/isa/x86/isplit-not-legalized-twice.clif
index 4b81a186da..c3ace05158 100644
--- a/cranelift/filetests/filetests/isa/x86/isplit-not-legalized-twice.clif
+++ b/cranelift/filetests/filetests/isa/x86/isplit-not-legalized-twice.clif
@@ -2,10 +2,10 @@ test compile
 target x86_64
 
 function u0:0(i64, i64) -> i128 system_v {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     trap user0
 
-ebb30:
+block30:
     v245 = iconst.i64 0
     v246 = iconcat v245, v245
     ; The next instruction used to be legalized twice, causing a panic the second time.
@@ -13,7 +13,7 @@ ebb30:
     v252, v253 = isplit v246
     trap user0
 
-ebb45:
+block45:
     v369 = iconst.i64 0
     v370 = load.i128 v369
     trap user0
diff --git a/cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif b/cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif
index 35698c9abc..018ac95fbc 100644
--- a/cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/isub_imm-i8.clif
@@ -3,7 +3,7 @@ set opt_level=speed_and_size
 target x86_64
 
 function u0:0(i8) -> i8 fast {
-ebb0(v0: i8):
+block0(v0: i8):
     v1 = iconst.i8 0
     v2 = isub v1, v0
     ; check: v3 = uextend.i32 v0
diff --git a/cranelift/filetests/filetests/isa/x86/jump_i128_param_unused.clif b/cranelift/filetests/filetests/isa/x86/jump_i128_param_unused.clif
index 9d96fcbe31..19f22c3906 100644
--- a/cranelift/filetests/filetests/isa/x86/jump_i128_param_unused.clif
+++ b/cranelift/filetests/filetests/isa/x86/jump_i128_param_unused.clif
@@ -2,9 +2,9 @@ test compile
 target x86_64
 
 function u0:0(i128) system_v {
-ebb0(v0: i128):
-    jump ebb1(v0)
+block0(v0: i128):
+    jump block1(v0)
 
-ebb1(v1: i128):
+block1(v1: i128):
     return
 }
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-bint-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-bint-i8.clif
index b2684ae105..dec3416a89 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-bint-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-bint-i8.clif
@@ -3,7 +3,7 @@ test compile
 target x86_64
 
 function u0:0() -> i8 fast {
-ebb0:
+block0:
     v14 = bconst.b1 false
     v15 = bint.i8 v14
     return v15
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-bnot.clif b/cranelift/filetests/filetests/isa/x86/legalize-bnot.clif
index 07a41dfdc4..dbd1397e45 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-bnot.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-bnot.clif
@@ -8,15 +8,15 @@ function u0:51(i64, i64) system_v {
     ss2 = explicit_slot 1
     ss3 = explicit_slot 1
 
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = stack_addr.i64 ss1
     v3 = load.i8 v1
     store v3, v2
     v4 = stack_addr.i64 ss2
     v5 = stack_addr.i64 ss3
-    jump ebb1
+    jump block1
 
-ebb1:
+block1:
     v6 = load.i8 v2
     store v6, v5
     v7 = load.i8 v5
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif
index 2f8ce5d78a..5c4004a539 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-icmp.clif
@@ -3,44 +3,44 @@ test legalizer
 target x86_64
 
 function %br_icmp(i64) fast {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = iconst.i64 0
-    br_icmp eq v0, v1, ebb1
-    jump ebb1
+    br_icmp eq v0, v1, block1
+    jump block1
 
-ebb1:
+block1:
     return
 }
 
 ; sameln: function %br_icmp(i64 [%rdi]) fast {
-; nextln:                                 ebb0(v0: i64):
+; nextln:                                 block0(v0: i64):
 ; nextln: [RexOp1pu_id#b8]                    v1 = iconst.i64 0
 ; nextln: [DynRexOp1icscc#8039]               v2 = icmp eq v0, v1
-; nextln: [RexOp1t8jccb#75]                   brnz v2, ebb1
-; nextln: [Op1jmpb#eb]                        jump ebb1
+; nextln: [RexOp1t8jccb#75]                   brnz v2, block1
+; nextln: [Op1jmpb#eb]                        jump block1
 ; nextln: 
-; nextln:                                 ebb1:
+; nextln:                                 block1:
 ; nextln: [Op1ret#c3]                         return
 ; nextln: }
 
 
-function %br_icmp_ebb_args(i64) fast {
-ebb0(v0: i64):
+function %br_icmp_args(i64) fast {
+block0(v0: i64):
     v1 = iconst.i64 0
-    br_icmp eq v0, v1, ebb1(v0)
-    jump ebb1(v0)
+    br_icmp eq v0, v1, block1(v0)
+    jump block1(v0)
 
-ebb1(v2: i64):
+block1(v2: i64):
     return
 }
 
-; sameln: function %br_icmp_ebb_args(i64 [%rdi]) fast {
-; nextln:                                 ebb0(v0: i64):
+; sameln: function %br_icmp_args(i64 [%rdi]) fast {
+; nextln:                                 block0(v0: i64):
 ; nextln: [RexOp1pu_id#b8]                    v1 = iconst.i64 0
 ; nextln: [DynRexOp1icscc#8039]               v3 = icmp eq v0, v1
-; nextln: [RexOp1t8jccb#75]                   brnz v3, ebb1(v0)
-; nextln: [Op1jmpb#eb]                        jump ebb1(v0)
+; nextln: [RexOp1t8jccb#75]                   brnz v3, block1(v0)
+; nextln: [Op1jmpb#eb]                        jump block1(v0)
 ; nextln: 
-; nextln:                                 ebb1(v2: i64):
+; nextln:                                 block1(v2: i64):
 ; nextln: [Op1ret#c3]                         return
 ; nextln: }
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
index e9464e6219..b9ed036755 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-br-table.clif
@@ -2,20 +2,20 @@ test compile
 set opt_level=speed_and_size
 target x86_64
 ; regex: V=v\d+
-; regex: EBB=ebb\d+
+; regex: BB=block\d+
 
 function u0:0(i64) system_v {
     ss0 = explicit_slot 1
-    jt0 = jump_table [ebb1]
+    jt0 = jump_table [block1]
 
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = stack_addr.i64 ss0
     v2 = load.i8 v1
-    br_table v2, ebb2, jt0
+    br_table v2, block2, jt0
 ; check:     $(oob=$V) = ifcmp_imm $(idx=$V), 1
-; ebb2 is replaced by ebb1 by fold_redundant_jump
-; nextln:    brif uge $oob, ebb1
-; nextln:    fallthrough $(inb=$EBB)
+; block2 is replaced by block1 by fold_redundant_jump
+; nextln:    brif uge $oob, block1
+; nextln:    fallthrough $(inb=$BB)
 ; check:   $inb:
 ; nextln:    $(final_idx=$V) = uextend.i64 $idx
 ; nextln:    $(base=$V) = jump_table_base.i64 jt0
@@ -23,9 +23,9 @@ ebb0(v0: i64):
 ; nextln:    $(addr=$V) = iadd $base, $rel_addr
 ; nextln:    indirect_jump_table_br $addr, jt0
 
-ebb2:
-    jump ebb1
+block2:
+    jump block1
 
-ebb1:
+block1:
     return
 }
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-byte-ops-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-byte-ops-i8.clif
index b0a318b8d4..2c8c8612d6 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-byte-ops-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-byte-ops-i8.clif
@@ -7,7 +7,7 @@ function u0:0(i8, i8) fast {
 fn0 = %black_box(i8)
 ss0 = explicit_slot 1 ; black box
 
-ebb0(v0: i8, v1: i8):
+block0(v0: i8, v1: i8):
     v99 = stack_addr.i64 ss0
 
     ; check: istore8 $(V), $(V)
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-call.clif b/cranelift/filetests/filetests/isa/x86/legalize-call.clif
index 240b075374..c761a8d5aa 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-call.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-call.clif
@@ -5,7 +5,7 @@ target x86_64 haswell
 
 function %call() {
     fn0 = %foo()
-ebb0:
+block0:
     call fn0()
     return
 }
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-clz-ctz-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-clz-ctz-i8.clif
index 914bcb0e30..8e63f1e0c6 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-clz-ctz-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-clz-ctz-i8.clif
@@ -4,7 +4,7 @@ target x86_64
 ; regex: V=v\d+
 
 function u0:0(i8) -> i8, i8 fast {
-ebb0(v0: i8):
+block0(v0: i8):
     v1 = clz v0
     ; check: v3 = uextend.i32 v0
     ; nextln: v6 = iconst.i32 -1
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-custom.clif b/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
index 2657bfd497..c2bc6bec19 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
@@ -4,36 +4,36 @@ target i686
 target x86_64
 
 ; regex: V=v\d+
-; regex: EBB=ebb\d+
+; regex: BB=block\d+
 
 function %cond_trap(i32) {
-ebb0(v1: i32):
+block0(v1: i32):
     trapz v1, user67
     return
-    ; check: ebb0(v1: i32
+    ; check: block0(v1: i32
     ; nextln: $(f=$V) = ifcmp_imm v1, 0
     ; nextln: trapif eq $f, user67
     ; nextln: return
 }
 
 function %cond_trap2(i32) {
-ebb0(v1: i32):
+block0(v1: i32):
     trapnz v1, int_ovf
     return
-    ; check: ebb0(v1: i32
+    ; check: block0(v1: i32
     ; nextln: $(f=$V) = ifcmp_imm v1, 0
     ; nextln: trapif ne $f, int_ovf
     ; nextln: return
 }
 
 function %cond_trap_b1(i32) {
-ebb0(v1: i32):
+block0(v1: i32):
     v2 = icmp_imm eq v1, 6
     trapz v2, user7
     return
-    ; check: ebb0(v1: i32
-    ; check: brnz v2, $(new=$EBB)
-    ; check: jump $(trap=$EBB)
+    ; check: block0(v1: i32
+    ; check: brnz v2, $(new=$BB)
+    ; check: jump $(trap=$BB)
     ; check: $trap:
     ; nextln: trap user7
     ; check: $new:
@@ -41,13 +41,13 @@ ebb0(v1: i32):
 }
 
 function %cond_trap2_b1(i32) {
-ebb0(v1: i32):
+block0(v1: i32):
     v2 = icmp_imm eq v1, 6
     trapnz v2, user9
     return
-    ; check: ebb0(v1: i32
-    ; check: brz v2, $(new=$EBB)
-    ; check: jump $(trap=$EBB)
+    ; check: block0(v1: i32
+    ; check: brz v2, $(new=$BB)
+    ; check: jump $(trap=$BB)
     ; check: $trap:
     ; nextln: trap user9
     ; check: $new:
@@ -55,7 +55,7 @@ ebb0(v1: i32):
 }
 
 function %f32const() -> f32 {
-ebb0:
+block0:
     v1 = f32const 0x1.0p1
     ; check: $(tmp=$V) = iconst.i32
     ; check: v1 = bitcast.f32 $tmp
@@ -63,9 +63,9 @@ ebb0:
 }
 
 function %select_f64(f64, f64, i32) -> f64 {
-ebb0(v0: f64, v1: f64, v2: i32):
+block0(v0: f64, v1: f64, v2: i32):
     v3 = select v2, v0, v1
-    ; check:  brnz v2, $(new=$EBB)(v0)
+    ; check:  brnz v2, $(new=$BB)(v0)
     ; nextln: jump $new(v1)
     ; check: $new(v3: f64):
     ; nextln: return v3
@@ -73,19 +73,19 @@ ebb0(v0: f64, v1: f64, v2: i32):
 }
 
 function %f32_min(f32, f32) -> f32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fmin v0, v1
     return v2
     ; check: $(vnat=$V) = x86_fmin.f32 v0, v1
-    ; nextln: jump $(done=$EBB)($vnat)
+    ; nextln: jump $(done=$BB)($vnat)
 
-    ; check: $(uno=$EBB):
+    ; check: $(uno=$BB):
     ; nextln: $(vuno=$V) = fadd.f32 v0, v1
-    ; nextln: jump $(done=$EBB)($vuno)
+    ; nextln: jump $(done=$BB)($vuno)
 
-    ; check: $(ueq=$EBB):
+    ; check: $(ueq=$BB):
     ; check: $(veq=$V) = bor.f32 v0, v1
-    ; nextln: jump $(done=$EBB)($veq)
+    ; nextln: jump $(done=$BB)($veq)
 
     ; check: $done(v2: f32):
     ; nextln: return v2
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-div-traps.clif b/cranelift/filetests/filetests/isa/x86/legalize-div-traps.clif
index 2622ae48f3..1be81ec186 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-div-traps.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-div-traps.clif
@@ -5,11 +5,11 @@ set avoid_div_traps=1
 target x86_64
 
 ; regex: V=v\d+
-; regex: EBB=ebb\d+
+; regex: BB=block\d+
 
 function %udiv(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
-    ; check: ebb0(
+block0(v0: i64, v1: i64):
+    ; check: block0(
     v2 = udiv v0, v1
     ; nextln: $(fz=$V) = ifcmp_imm v1, 0
     ; nextln: trapif eq $fz, int_divz
@@ -20,8 +20,8 @@ ebb0(v0: i64, v1: i64):
 }
 
 function %udiv_0(i64) -> i64 {
-ebb0(v0: i64):
-    ; check: ebb0(
+block0(v0: i64):
+    ; check: block0(
     v1 = iconst.i64 0
     ; nextln: v1 = iconst.i64 0
     v2 = udiv v0, v1
@@ -34,8 +34,8 @@ ebb0(v0: i64):
 }
 
 function %udiv_minus_1(i64) -> i64 {
-ebb0(v0: i64):
-    ; check: ebb0(
+block0(v0: i64):
+    ; check: block0(
     v1 = iconst.i64 -1
     ; nextln: v1 = iconst.i64 -1
     v2 = udiv v0, v1
@@ -46,8 +46,8 @@ ebb0(v0: i64):
 }
 
 function %urem(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
-    ; check: ebb0(
+block0(v0: i64, v1: i64):
+    ; check: block0(
     v2 = urem v0, v1
     ; nextln: $(fz=$V) = ifcmp_imm v1, 0
     ; nextln: trapif eq $fz, int_divz
@@ -58,8 +58,8 @@ ebb0(v0: i64, v1: i64):
 }
 
 function %urem_0(i64) -> i64 {
-ebb0(v0: i64):
-    ; check: ebb0(
+block0(v0: i64):
+    ; check: block0(
     v1 = iconst.i64 0
     ; nextln: v1 = iconst.i64 0
     v2 = urem v0, v1
@@ -72,8 +72,8 @@ ebb0(v0: i64):
 }
 
 function %urem_minus_1(i64) -> i64 {
-ebb0(v0: i64):
-    ; check: ebb0(
+block0(v0: i64):
+    ; check: block0(
     v1 = iconst.i64 -1
     ; nextln: v1 = iconst.i64 -1
     v2 = urem v0, v1
@@ -84,16 +84,16 @@ ebb0(v0: i64):
 }
 
 function %sdiv(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
-    ; check: ebb0(
+block0(v0: i64, v1: i64):
+    ; check: block0(
     v2 = sdiv v0, v1
     ; nextln: $(fz=$V) = ifcmp_imm v1, 0
     ; nextln: trapif eq $fz, int_divz
     ; nextln: $(fm1=$V) = ifcmp_imm v1, -1
-    ; nextln: brif eq $fm1, $(m1=$EBB)
+    ; nextln: brif eq $fm1, $(m1=$BB)
     ; check: $(hi=$V) = sshr_imm
     ; nextln: $(q=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1
-    ; nextln: jump $(done=$EBB)($q)
+    ; nextln: jump $(done=$BB)($q)
     ; check: $m1:
     ; nextln: $(imin=$V) = iconst.i64 0x8000_0000_0000_0000
     ; nextln: $(fm=$V) = ifcmp.i64 v0, $imin
@@ -104,8 +104,8 @@ ebb0(v0: i64, v1: i64):
 }
 
 function %sdiv_0(i64) -> i64 {
-ebb0(v0: i64):
-    ; check: ebb0(
+block0(v0: i64):
+    ; check: block0(
     v1 = iconst.i64 0
     ; nextln: v1 = iconst.i64 0
     v2 = sdiv v0, v1
@@ -118,16 +118,16 @@ ebb0(v0: i64):
 }
 
 function %sdiv_minus_1(i64) -> i64 {
-ebb0(v0: i64):
-    ; check: ebb0(
+block0(v0: i64):
+    ; check: block0(
     v1 = iconst.i64 -1
     ; nextln: v1 = iconst.i64 -1
     v2 = sdiv v0, v1
     ; nextln: $(fm1=$V) = ifcmp_imm v1, -1
-    ; nextln: brif eq $fm1, $(m1=$EBB)
+    ; nextln: brif eq $fm1, $(m1=$BB)
     ; check: $(hi=$V) = sshr_imm
     ; nextln: $(q=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1
-    ; nextln: jump $(done=$EBB)($q)
+    ; nextln: jump $(done=$BB)($q)
     ; check: $m1:
     ; nextln: $(imin=$V) = iconst.i64 0x8000_0000_0000_0000
     ; nextln: $(fm=$V) = ifcmp.i64 v0, $imin
@@ -140,27 +140,27 @@ ebb0(v0: i64):
 ; The srem expansion needs to special-case x % -1 since x86_sdivmodx traps on INT_MIN/-1.
 ; TODO: Add more explicit pattern matching once we've cleaned up the ifcmp+brif pattern.
 function %srem(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
-    ; check: ebb0(
+block0(v0: i64, v1: i64):
+    ; check: block0(
     v2 = srem v0, v1
     ; nextln: $(fz=$V) = ifcmp_imm v1, 0
     ; nextln: trapif eq $fz, int_divz
     ; nextln: $(fm1=$V) = ifcmp_imm v1, -1
-    ; nextln: brif eq $fm1, $(m1=$EBB)
+    ; nextln: brif eq $fm1, $(m1=$BB)
     ; check: $(hi=$V) = sshr_imm
     ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1
-    ; nextln: jump $(done=$EBB)($r)
+    ; nextln: jump $(done=$BB)($r)
     ; check: $m1:
     ; nextln: $(zero=$V) = iconst.i64 0
-    ; nextln: jump $(done=$EBB)($zero)
+    ; nextln: jump $(done=$BB)($zero)
     ; check: $done(v2: i64):
     return v2
     ; nextln: return v2
 }
 
 function %srem_0(i64) -> i64 {
-ebb0(v0: i64):
-    ; check: ebb0(
+block0(v0: i64):
+    ; check: block0(
     v1 = iconst.i64 0
     ; nextln: v1 = iconst.i64 0
     v2 = srem v0, v1
@@ -173,19 +173,19 @@ ebb0(v0: i64):
 }
 
 function %srem_minus_1(i64) -> i64 {
-ebb0(v0: i64):
-    ; check: ebb0(
+block0(v0: i64):
+    ; check: block0(
     v1 = iconst.i64 -1
     ; nextln: v1 = iconst.i64 -1
     v2 = srem v0, v1
     ; nextln: $(fm1=$V) = ifcmp_imm v1, -1
-    ; nextln: brif eq $fm1, $(m1=$EBB)
+    ; nextln: brif eq $fm1, $(m1=$BB)
     ; check: $(hi=$V) = sshr_imm
     ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1
-    ; nextln: jump $(done=$EBB)($r)
+    ; nextln: jump $(done=$BB)($r)
     ; check: $m1:
     ; nextln: $(zero=$V) = iconst.i64 0
-    ; nextln: jump $(done=$EBB)($zero)
+    ; nextln: jump $(done=$BB)($zero)
     ; check: $done(v2: i64):
     return v2
     ; nextln: return v2
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-div.clif b/cranelift/filetests/filetests/isa/x86/legalize-div.clif
index be30accbc6..b9f115b85b 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-div.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-div.clif
@@ -5,11 +5,11 @@ set avoid_div_traps=0
 target x86_64
 
 ; regex: V=v\d+
-; regex: EBB=ebb\d+
+; regex: BB=block\d+
 
 function %udiv(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
-    ; check: ebb0(
+block0(v0: i64, v1: i64):
+    ; check: block0(
     v2 = udiv v0, v1
     ; nextln: $(hi=$V) = iconst.i64 0
     ; nextln: $(d=$V), $(r=$V) = x86_udivmodx v0, $hi, v1
@@ -18,8 +18,8 @@ ebb0(v0: i64, v1: i64):
 }
 
 function %urem(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
-    ; check: ebb0(
+block0(v0: i64, v1: i64):
+    ; check: block0(
     v2 = urem v0, v1
     ; nextln: $(hi=$V) = iconst.i64 0
     ; nextln: $(d=$V), $(r=$V) = x86_udivmodx v0, $hi, v1
@@ -28,8 +28,8 @@ ebb0(v0: i64, v1: i64):
 }
 
 function %sdiv(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
-    ; check: ebb0(
+block0(v0: i64, v1: i64):
+    ; check: block0(
     v2 = sdiv v0, v1
     ; check: $(hi=$V) = sshr_imm
     ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1
@@ -40,17 +40,17 @@ ebb0(v0: i64, v1: i64):
 ; The srem expansion needs to special-case x % -1 since x86_sdivmodx traps on INT_MIN/-1.
 ; TODO: Add more explicit pattern matching once we've cleaned up the ifcmp+brif pattern.
 function %srem(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
-    ; check: ebb0(
+block0(v0: i64, v1: i64):
+    ; check: block0(
     v2 = srem v0, v1
     ; nextln: $(fm1=$V) = ifcmp_imm v1, -1
-    ; nextln: brif eq $fm1, $(m1=$EBB)
+    ; nextln: brif eq $fm1, $(m1=$BB)
     ; check: $(hi=$V) = sshr_imm
     ; nextln: $(d=$V), $(r=$V) = x86_sdivmodx v0, $hi, v1
-    ; nextln: jump $(done=$EBB)($r)
+    ; nextln: jump $(done=$BB)($r)
     ; check: $m1:
     ; nextln: $(zero=$V) = iconst.i64 0
-    ; nextln: jump $(done=$EBB)($zero)
+    ; nextln: jump $(done=$BB)($zero)
     ; check: $done(v2: i64):
     return v2
     ; nextln: return v2
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-f64const-x64.clif b/cranelift/filetests/filetests/isa/x86/legalize-f64const-x64.clif
index addafe90a3..382c6ba80a 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-f64const-x64.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-f64const-x64.clif
@@ -5,7 +5,7 @@ target x86_64
 ; regex: V=v\d+
 
 function %f64const() -> f64 {
-ebb0:
+block0:
     v1 = f64const 0x1.0p1
     ; check: $(tmp=$V) = iconst.i64
     ; check: v1 = bitcast.f64 $tmp
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-fcvt_from_usint-i16.clif b/cranelift/filetests/filetests/isa/x86/legalize-fcvt_from_usint-i16.clif
index b7d4b80977..c11e77f2c7 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-fcvt_from_usint-i16.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-fcvt_from_usint-i16.clif
@@ -2,13 +2,13 @@ test compile
 target x86_64
 
 function u0:0(i16) -> f64 fast {
-ebb0(v0: i16):
+block0(v0: i16):
     v1 = fcvt_from_uint.f64 v0
     return v1
 }
 
 function u0:1(i16) -> f64 fast {
-ebb0(v0: i16):
+block0(v0: i16):
     v1 = fcvt_from_sint.f64 v0
     return v1
 }
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-heaps.clif b/cranelift/filetests/filetests/isa/x86/legalize-heaps.clif
index c2f1ccb2d9..5fb080f8a6 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-heaps.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-heaps.clif
@@ -2,7 +2,7 @@ test legalizer
 target x86_64
 
 ; Test legalization for various forms of heap addresses.
-; regex: EBB=ebb\d+
+; regex: BB=block\d+
 
 function %heap_addrs(i32, i64, i64 vmctx) {
     gv4 = vmctx
@@ -29,7 +29,7 @@ function %heap_addrs(i32, i64, i64 vmctx) {
     ; check: heap6 = dynamic gv1, min 0x0001_0000, bound gv2, offset_guard 0x8000_0000, index_type i64
     ; check: heap7 = dynamic gv1, min 0, bound gv2, offset_guard 4096, index_type i64
 
-ebb0(v0: i32, v1: i64, v3: i64):
+block0(v0: i32, v1: i64, v3: i64):
     ; The fast-path; 32-bit index, static heap with a sufficient bound, no bounds check needed!
     v4 = heap_addr.i64 heap0, v0, 0
     ; check:         v12 = uextend.i64 v0
@@ -38,8 +38,8 @@ ebb0(v0: i32, v1: i64, v3: i64):
 
     v5 = heap_addr.i64 heap1, v0, 0
     ; check:         v14 = icmp_imm ugt v0, 0x0001_0000
-    ; check:         brz v14, $(resume_1=$EBB)
-    ; nextln:        jump $(trap_1=$EBB)
+    ; check:         brz v14, $(resume_1=$BB)
+    ; nextln:        jump $(trap_1=$BB)
     ; check:     $trap_1:
     ; nextln:        trap heap_oob
     ; check:     $resume_1:
@@ -50,8 +50,8 @@ ebb0(v0: i32, v1: i64, v3: i64):
     v6 = heap_addr.i64 heap2, v1, 0
     ; check:         v19 = iconst.i64 0x0001_0000_0000
     ; check:         v17 = icmp.i64 ugt v1, v19
-    ; check:         brz v17, $(resume_2=$EBB)
-    ; nextln:        jump $(trap_2=$EBB)
+    ; check:         brz v17, $(resume_2=$BB)
+    ; nextln:        jump $(trap_2=$BB)
     ; check:     $trap_2:
     ; nextln:        trap heap_oob
     ; check:     $resume_2:
@@ -60,8 +60,8 @@ ebb0(v0: i32, v1: i64, v3: i64):
 
     v7 = heap_addr.i64 heap3, v1, 0
     ; check:         v20 = icmp_imm.i64 ugt v1, 0x0001_0000
-    ; check:         brz v20, $(resume_3=$EBB)
-    ; nextln:        jump $(trap_3=$EBB)
+    ; check:         brz v20, $(resume_3=$BB)
+    ; nextln:        jump $(trap_3=$BB)
     ; check:     $trap_3:
     ; nextln:        trap heap_oob
     ; check:     $resume_3:
@@ -72,8 +72,8 @@ ebb0(v0: i32, v1: i64, v3: i64):
     ; check:         v22 = load.i32 notrap aligned v3+88
     ; check:         v23 = iadd_imm v22, 0
     ; check:         v24 = icmp.i32 ugt v0, v23
-    ; check:         brz v24, $(resume_4=$EBB)
-    ; nextln:        jump $(trap_4=$EBB)
+    ; check:         brz v24, $(resume_4=$BB)
+    ; nextln:        jump $(trap_4=$BB)
     ; check:     $trap_4:
     ; nextln:        trap heap_oob
     ; check:     $resume_4:
@@ -85,8 +85,8 @@ ebb0(v0: i32, v1: i64, v3: i64):
     ; check:         v27 = load.i32 notrap aligned v3+88
     ; check:         v28 = iadd_imm v27, 0
     ; check:         v29 = icmp.i32 ugt v0, v28
-    ; check:         brz v29, $(resume_5=$EBB)
-    ; nextln:        jump $(trap_5=$EBB)
+    ; check:         brz v29, $(resume_5=$BB)
+    ; nextln:        jump $(trap_5=$BB)
     ; check:     $trap_5:
     ; nextln:        trap heap_oob
     ; check:     $resume_5:
@@ -98,8 +98,8 @@ ebb0(v0: i32, v1: i64, v3: i64):
     ; check:         v32 = iadd_imm.i64 v3, 80
     ; check:         v33 = iadd_imm v32, 0
     ; check:         v34 = icmp.i64 ugt v1, v33
-    ; check:         brz v34, $(resume_6=$EBB)
-    ; nextln:        jump $(trap_6=$EBB)
+    ; check:         brz v34, $(resume_6=$BB)
+    ; nextln:        jump $(trap_6=$BB)
     ; check:     $trap_6:
     ; nextln:        trap heap_oob
     ; check:     $resume_6:
@@ -110,8 +110,8 @@ ebb0(v0: i32, v1: i64, v3: i64):
     ; check:         v36 = iadd_imm.i64 v3, 80
     ; check:         v37 = iadd_imm v36, 0
     ; check:         v38 = icmp.i64 ugt v1, v37
-    ; check:         brz v38, $(resume_7=$EBB)
-    ; nextln:        jump $(trap_7=$EBB)
+    ; check:         brz v38, $(resume_7=$BB)
+    ; nextln:        jump $(trap_7=$BB)
     ; check:     $trap_7:
     ; nextln:        trap heap_oob
     ; check:     $resume_7:
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-i128.clif b/cranelift/filetests/filetests/isa/x86/legalize-i128.clif
index db071ba3c7..81a2d1ecdd 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-i128.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-i128.clif
@@ -5,7 +5,7 @@ target x86_64 haswell
 ; regex: V=v\d+
 
 function %imul(i128, i128) -> i128 {
-ebb0(v1: i128, v2: i128):
+block0(v1: i128, v2: i128):
     v10 = imul v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-i64.clif b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
index a484818a34..0d1eea5152 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
@@ -5,7 +5,7 @@ target i686 haswell
 ; regex: V=v\d+
 
 function %iadd(i64, i64) -> i64 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = iadd v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
@@ -16,7 +16,7 @@ ebb0(v1: i64, v2: i64):
 }
 
 function %isub(i64, i64) -> i64 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = isub v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
@@ -27,7 +27,7 @@ ebb0(v1: i64, v2: i64):
 }
 
 function %imul(i64, i64) -> i64 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = imul v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
@@ -42,7 +42,7 @@ ebb0(v1: i64, v2: i64):
 }
 
 function %icmp_eq(i64, i64) -> b1 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = icmp eq v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
@@ -53,7 +53,7 @@ ebb0(v1: i64, v2: i64):
 }
 
 function %icmp_imm_eq(i64) -> b1 {
-ebb0(v1: i64):
+block0(v1: i64):
     v10 = icmp_imm eq v1, 0
     ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
     ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
@@ -67,7 +67,7 @@ ebb0(v1: i64):
 }
 
 function %icmp_ne(i64, i64) -> b1 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = icmp ne v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
@@ -78,7 +78,7 @@ ebb0(v1: i64, v2: i64):
 }
 
 function %icmp_imm_ne(i64) -> b1 {
-ebb0(v1: i64):
+block0(v1: i64):
     v10 = icmp_imm ne v1, 0
     ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
     ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
@@ -92,7 +92,7 @@ ebb0(v1: i64):
 }
 
 function %icmp_sgt(i64, i64) -> b1 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = icmp sgt v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
@@ -106,7 +106,7 @@ ebb0(v1: i64, v2: i64):
 }
 
 function %icmp_imm_sgt(i64) -> b1 {
-ebb0(v1: i64):
+block0(v1: i64):
     v10 = icmp_imm sgt v1, 0
     ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
     ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
@@ -123,7 +123,7 @@ ebb0(v1: i64):
 }
 
 function %icmp_sge(i64, i64) -> b1 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = icmp sge v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
@@ -137,7 +137,7 @@ ebb0(v1: i64, v2: i64):
 }
 
 function %icmp_imm_sge(i64) -> b1 {
-ebb0(v1: i64):
+block0(v1: i64):
     v10 = icmp_imm sge v1, 0
     ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
     ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
@@ -154,7 +154,7 @@ ebb0(v1: i64):
 }
 
 function %icmp_slt(i64, i64) -> b1 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = icmp slt v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
@@ -168,7 +168,7 @@ ebb0(v1: i64, v2: i64):
 }
 
 function %icmp_imm_slt(i64) -> b1 {
-ebb0(v1: i64):
+block0(v1: i64):
     v10 = icmp_imm slt v1, 0
     ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
     ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
@@ -185,7 +185,7 @@ ebb0(v1: i64):
 }
 
 function %icmp_sle(i64, i64) -> b1 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = icmp sle v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
@@ -199,7 +199,7 @@ ebb0(v1: i64, v2: i64):
 }
 
 function %icmp_imm_sle(i64) -> b1 {
-ebb0(v1: i64):
+block0(v1: i64):
     v10 = icmp_imm sle v1, 0
     ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
     ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
@@ -216,7 +216,7 @@ ebb0(v1: i64):
 }
 
 function %icmp_ugt(i64, i64) -> b1 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = icmp ugt v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
@@ -230,7 +230,7 @@ ebb0(v1: i64, v2: i64):
 }
 
 function %icmp_imm_ugt(i64) -> b1 {
-ebb0(v1: i64):
+block0(v1: i64):
     v10 = icmp_imm ugt v1, 0
     ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
     ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
@@ -247,7 +247,7 @@ ebb0(v1: i64):
 }
 
 function %icmp_uge(i64, i64) -> b1 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = icmp uge v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
@@ -261,7 +261,7 @@ ebb0(v1: i64, v2: i64):
 }
 
 function %icmp_imm_uge(i64) -> b1 {
-ebb0(v1: i64):
+block0(v1: i64):
     v10 = icmp_imm uge v1, 0
     ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
     ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
@@ -278,7 +278,7 @@ ebb0(v1: i64):
 }
 
 function %icmp_ult(i64, i64) -> b1 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = icmp ult v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
@@ -292,7 +292,7 @@ ebb0(v1: i64, v2: i64):
 }
 
 function %icmp_imm_ult(i64) -> b1 {
-ebb0(v1: i64):
+block0(v1: i64):
     v10 = icmp_imm ult v1, 0
     ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
     ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
@@ -309,7 +309,7 @@ ebb0(v1: i64):
 }
 
 function %icmp_ule(i64, i64) -> b1 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = icmp ule v1, v2
     ; check: v1 = iconcat $(v1_lsb=$V), $(v1_msb=$V)
     ; nextln: v2 = iconcat $(v2_lsb=$V), $(v2_msb=$V)
@@ -323,7 +323,7 @@ ebb0(v1: i64, v2: i64):
 }
 
 function %icmp_imm_ule(i64) -> b1 {
-ebb0(v1: i64):
+block0(v1: i64):
     v10 = icmp_imm ule v1, 0
     ; check: $(v1_lsb=$V) -> $(v1_lsb_a=$V)
     ; nextln: $(v1_msb=$V) -> $(v1_msb_a=$V)
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-icmp-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-icmp-i8.clif
index 41bd27950f..2519d3b484 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-icmp-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-icmp-i8.clif
@@ -4,7 +4,7 @@ target x86_64
 ; regex: V=v\d+
 
 function u0:0(i8, i8) -> i8 fast {
-ebb0(v0: i8, v1: i8):
+block0(v0: i8, v1: i8):
     v2 = icmp_imm sle v0, 0
     ; check: $(e1=$V) = sextend.i32 v0
     ; nextln: v2 = icmp_imm sle $e1, 0
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-iconst-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-iconst-i8.clif
index 8245ee73ce..39908d1f1d 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-iconst-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-iconst-i8.clif
@@ -5,10 +5,10 @@ target x86_64
 function u0:0(i64) system_v {
     ss0 = explicit_slot 0
 
-ebb0(v0: i64):
-    jump ebb1
+block0(v0: i64):
+    jump block1
 
-ebb1:
+block1:
 ; _0 = const 42u8
     v1 = iconst.i8 42
     store v1, v0
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-imul-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-imul-i8.clif
index d56ff787eb..6902636008 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-imul-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-imul-i8.clif
@@ -4,7 +4,7 @@ target x86_64
 
 function u0:0(i64, i8, i8) system_v {
 
-ebb0(v0: i64, v1: i8, v2: i8):
+block0(v0: i64, v1: i8, v2: i8):
     v11 = imul v1, v2
     store v11, v0
     return
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-imul-imm-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-imul-imm-i8.clif
index 6655c562e7..82d3fa26ce 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-imul-imm-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-imul-imm-i8.clif
@@ -5,7 +5,7 @@ target x86_64
 function u0:0(i64, i8) system_v {
     ss0 = explicit_slot 1
 
-ebb0(v0: i64, v1: i8):
+block0(v0: i64, v1: i8):
     v3 = stack_addr.i64 ss0
     v5 = load.i8 v3
     v6 = iconst.i8 2
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-isplit-backwards.clif b/cranelift/filetests/filetests/isa/x86/legalize-isplit-backwards.clif
index 43881fe09e..5a903350b5 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-isplit-backwards.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-isplit-backwards.clif
@@ -2,11 +2,11 @@ test compile
 target x86_64
 
 function u0:0(i128) -> i64, i64 fast {
-; check: ebb0(v4: i64 [%rdi], v5: i64 [%rsi], v8: i64 [%rbp]):
-ebb0(v0: i128):
-    jump ebb2
+; check: block0(v4: i64 [%rdi], v5: i64 [%rsi], v8: i64 [%rbp]):
+block0(v0: i128):
+    jump block2
 
-ebb1:
+block1:
     ; When this `isplit` is legalized, the bnot below is not yet legalized,
     ; so there isn't a corresponding `iconcat` yet. We should try legalization
     ; for this `isplit` again once all instrucions have been legalized.
@@ -14,11 +14,11 @@ ebb1:
     ; return v6, v7
     return v2, v3
 
-ebb2:
+block2:
     ; check: v6 = bnot.i64 v4
     ; check: v2 -> v6
     ; check: v7 = bnot.i64 v5
     ; check: v3 -> v7
     v1 = bnot.i128 v0
-    jump ebb1
+    jump block1
 }
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-libcall.clif b/cranelift/filetests/filetests/isa/x86/legalize-libcall.clif
index e28bebd668..8ddb0865f8 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-libcall.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-libcall.clif
@@ -5,7 +5,7 @@ set is_pic
 target x86_64
 
 function %floor(f32) -> f32 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = floor v0
     return v1
 }
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-load-store-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-load-store-i8.clif
index cecf0e145f..2fcb086e72 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-load-store-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-load-store-i8.clif
@@ -9,16 +9,16 @@ function u0:0(i64, i8, i8) system_v {
     ss3 = explicit_slot 1
     ss4 = explicit_slot 1
 
-ebb0(v0: i64, v1: i8, v2: i8):
+block0(v0: i64, v1: i8, v2: i8):
     v3 = stack_addr.i64 ss1
     store v1, v3
     v4 = stack_addr.i64 ss2
     store v2, v4
     v5 = stack_addr.i64 ss3
     v6 = stack_addr.i64 ss4
-    jump ebb1
+    jump block1
 
-ebb1:
+block1:
     v7 = load.i8 v3
     store v7, v5
     v8 = load.i8 v4
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-memory.clif b/cranelift/filetests/filetests/isa/x86/legalize-memory.clif
index 348d763716..78d1796d00 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-memory.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-memory.clif
@@ -3,13 +3,13 @@ test legalizer
 target x86_64
 
 ; regex: V=v\d+
-; regex: EBB=ebb\d+
+; regex: BB=block\d+
 
 function %vmctx(i64 vmctx) -> i64 {
     gv0 = vmctx
     gv1 = iadd_imm.i64 gv0, -16
 
-ebb1(v1: i64):
+block1(v1: i64):
     v2 = global_value.i64 gv1
     ; check: v2 = iadd_imm v1, -16
     return v2
@@ -21,7 +21,7 @@ function %load(i64 vmctx) -> i64 {
     gv1 = load.i64 notrap aligned gv0-16
     gv2 = iadd_imm.i64 gv1, 32
 
-ebb1(v1: i64):
+block1(v1: i64):
     v2 = global_value.i64 gv2
     ; check: $(p1=$V) = load.i64 notrap aligned v1-16
     ; check: v2 = iadd_imm $p1, 32
@@ -33,7 +33,7 @@ function %symbol() -> i64 {
     gv0 = symbol %something
     gv1 = symbol u123:456
 
-ebb1:
+block1:
     v0 = global_value.i64 gv0
     ; check: v0 = symbol_value.i64 gv0
     v1 = global_value.i64 gv1
@@ -49,8 +49,8 @@ function %staticheap_sm64(i32, i64 vmctx) -> f32 baldrdash_system_v {
     gv1 = iadd_imm.i64 gv0, 64
     heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v999: i64):
-    ; check: ebb0(
+block0(v0: i32, v999: i64):
+    ; check: block0(
     v1 = heap_addr.i64 heap0, v0, 1
     ; Boundscheck should be eliminated.
     ; Checks here are assuming that no pipehole opts fold the load offsets.
@@ -70,13 +70,13 @@ function %staticheap_static_oob_sm64(i32, i64 vmctx) -> f32 baldrdash_system_v {
     gv1 = iadd_imm.i64 gv0, 64
     heap0 = static gv1, min 0x1000, bound 0x1000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v999: i64):
+block0(v0: i32, v999: i64):
     ; Everything after the obviously OOB access should be eliminated, leaving
-    ; the `trap heap_oob` instruction as the terminator of the Ebb and moving
-    ; the remainder of the instructions into an inaccessible Ebb.
-    ; check: ebb0(
+    ; the `trap heap_oob` instruction as the terminator of the block and moving
+    ; the remainder of the instructions into an inaccessible block.
+    ; check: block0(
     ; nextln:     trap heap_oob
-    ; check: ebb1:
+    ; check: block1:
     ; nextln:     v1 = iconst.i64 0
     ; nextln:     v2 = load.f32 v1+16
     ; nextln:     return v2
@@ -94,13 +94,13 @@ function %staticheap_sm64(i32, i64 vmctx) -> f32 baldrdash_system_v {
     gv1 = iadd_imm.i64 gv0, 64
     heap0 = static gv1, min 0x1000, bound 0x1_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v999: i64):
-    ; check: ebb0(
+block0(v0: i32, v999: i64):
+    ; check: block0(
     v1 = heap_addr.i64 heap0, v0, 0x8000_0000
     ; Boundscheck code
     ; check: $(oob=$V) = icmp
-    ; nextln: brz $oob, $(ok=$EBB)
-    ; nextln: jump $(trap_oob=$EBB)
+    ; nextln: brz $oob, $(ok=$BB)
+    ; nextln: jump $(trap_oob=$BB)
     ; check: $trap_oob:
     ; nextln: trap heap_oob
     ; check: $ok:
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-mulhi.clif b/cranelift/filetests/filetests/isa/x86/legalize-mulhi.clif
index 946bcd8428..375a454c20 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-mulhi.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-mulhi.clif
@@ -4,7 +4,7 @@ target x86_64 baseline
 ; umulhi/smulhi on 64 bit operands
 
 function %i64_umulhi(i64, i64) -> i64 {
-ebb0(v10: i64, v11: i64):
+block0(v10: i64, v11: i64):
   v12 = umulhi v10, v11
   ; check: %rdi -> %rax
   ; check: x86_umulx
@@ -13,7 +13,7 @@ ebb0(v10: i64, v11: i64):
 }
 
 function %i64_smulhi(i64, i64) -> i64 {
-ebb0(v20: i64, v21: i64):
+block0(v20: i64, v21: i64):
   v22 = smulhi v20, v21
   ; check: %rdi -> %rax
   ; check: x86_smulx
@@ -25,7 +25,7 @@ ebb0(v20: i64, v21: i64):
 ; umulhi/smulhi on 32 bit operands
 
 function %i32_umulhi(i32, i32) -> i32 {
-ebb0(v30: i32, v31: i32):
+block0(v30: i32, v31: i32):
   v32 = umulhi v30, v31
   ; check: %rdi -> %rax
   ; check: x86_umulx
@@ -34,7 +34,7 @@ ebb0(v30: i32, v31: i32):
 }
 
 function %i32_smulhi(i32, i32) -> i32 {
-ebb0(v40: i32, v41: i32):
+block0(v40: i32, v41: i32):
   v42 = smulhi v40, v41
   ; check: %rdi -> %rax
   ; check: x86_smulx
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-popcnt-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-popcnt-i8.clif
index e761a2c7ca..c3f89c4807 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-popcnt-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-popcnt-i8.clif
@@ -2,7 +2,7 @@ test compile
 target x86_64
 
 function u0:0(i8) -> i8 fast {
-ebb0(v0: i8):
+block0(v0: i8):
     v1 = popcnt v0
     ; check-not: sextend.i32 v0
     return v1
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-regmove-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-regmove-i8.clif
index 8dc746d701..6f080ca89b 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-regmove-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-regmove-i8.clif
@@ -11,16 +11,16 @@ function u0:0(i64, i64, i64) system_v {
     sig0 = (i64, i16, i64) system_v
     fn0 = colocated u0:11 sig0
 
-ebb0(v0: i64, v1: i64, v2: i64):
+block0(v0: i64, v1: i64, v2: i64):
     v3 = stack_addr.i64 ss1
     store v1, v3
     v4 = stack_addr.i64 ss2
     store v2, v4
     v5 = stack_addr.i64 ss3
     v6 = stack_addr.i64 ss4
-    jump ebb1
+    jump block1
 
-ebb1:
+block1:
     v7 = load.i64 v3
     v8 = load.i16 v7
     store v8, v5
@@ -29,8 +29,8 @@ ebb1:
     v10 = load.i16 v5
     v11 = load.i64 v6
     call fn0(v0, v10, v11)
-    jump ebb2
+    jump block2
 
-ebb2:
+block2:
     return
 }
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-rotate.clif b/cranelift/filetests/filetests/isa/x86/legalize-rotate.clif
index 155b6001b4..78524d2969 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-rotate.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-rotate.clif
@@ -5,7 +5,7 @@ target x86_64
 ; regex: R=%[a-z0-9]+
 
 function %i32_rotr(i32, i32) -> i32 fast {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     ; check: regmove v1, $R -> %rcx
     ; check: v2 = rotr v0, v1
     v2 = rotr v0, v1
@@ -13,14 +13,14 @@ ebb0(v0: i32, v1: i32):
 }
 
 function %i32_rotr_imm_1(i32) -> i32 fast {
-ebb0(v0: i32):
+block0(v0: i32):
     ; check: $V = rotr_imm v0, 1
     v2 = rotr_imm v0, 1
     return v2
 }
 
 function %i32_rotl(i32, i32) -> i32 fast {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     ; check: regmove v1, $R -> %rcx
     ; check: v2 = rotl v0, v1
     v2 = rotl v0, v1
@@ -28,7 +28,7 @@ ebb0(v0: i32, v1: i32):
 }
 
 function %i32_rotl_imm_1(i32) -> i32 fast {
-ebb0(v0: i32):
+block0(v0: i32):
     ; check: $V = rotl_imm v0, 1
     v2 = rotl_imm v0, 1
     return v2
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-shlr-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-shlr-i8.clif
index dbd0da1204..ee6e3e6d11 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-shlr-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-shlr-i8.clif
@@ -4,7 +4,7 @@ target x86_64
 ; regex: V=v\d+
 
 function u0:0(i8, i8) -> i8 fast {
-ebb0(v0: i8, v1: i8):
+block0(v0: i8, v1: i8):
     v2 = ishl v0, v1
     ; check: $(e1=$V) = uextend.i32 v0
     ; check: $(r1=$V) = ishl $e1, v1
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-splat.clif b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
index 38731c778f..2fa6ace7e9 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-splat.clif
@@ -5,7 +5,7 @@ target x86_64 haswell
 
 ; use baldrdash_system_v calling convention here for simplicity (avoids prologue, epilogue)
 function %test_splat_i32() -> i32x4 baldrdash_system_v {
-ebb0:
+block0:
     v0 = iconst.i32 42
     v1 = splat.i32x4 v0
     return v1
@@ -14,7 +14,7 @@ ebb0:
 ; sameln: function %test_splat_i32() -> i32x4 [%xmm0] baldrdash_system_v {
 ; nextln:   ss0 = incoming_arg 0, offset 0
 ; nextln: 
-; nextln:   ebb0:
+; nextln:   block0:
 ; nextln:     v0 = iconst.i32 42
 ; nextln:     v2 = scalar_to_vector.i32x4 v0
 ; nextln:     v1 = x86_pshufd v2, 0
@@ -24,13 +24,13 @@ ebb0:
 
 
 function %test_splat_i64() -> i64x2 baldrdash_system_v {
-ebb0:
+block0:
     v0 = iconst.i64 42
     v1 = splat.i64x2 v0
     return v1
 }
 
-; check:   ebb0:
+; check:   block0:
 ; nextln:     v0 = iconst.i64 42
 ; nextln:     v2 = scalar_to_vector.i64x2 v0
 ; nextln:     v1 = x86_pinsr v2, 1, v0
@@ -39,13 +39,13 @@ ebb0:
 
 
 function %test_splat_b16() -> b16x8 baldrdash_system_v {
-ebb0:
+block0:
     v0 = bconst.b16 true
     v1 = splat.b16x8 v0
     return v1
 }
 
-; check:   ebb0:
+; check:   block0:
 ; nextln:     v0 = bconst.b16 true
 ; nextln:     v2 = scalar_to_vector.b16x8 v0
 ; nextln:     v3 = x86_pinsr v2, 1, v0
@@ -57,13 +57,13 @@ ebb0:
 
 
 function %test_splat_i8() -> i8x16 baldrdash_system_v {
-ebb0:
+block0:
     v0 = iconst.i8 42
     v1 = splat.i8x16 v0
     return v1
 }
 
-; check:   ebb0:
+; check:   block0:
 ; nextln:     v2 = iconst.i32 42
 ; nextln:     v0 = ireduce.i8 v2
 ; nextln:     v3 = scalar_to_vector.i8x16 v0
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-tables.clif b/cranelift/filetests/filetests/isa/x86/legalize-tables.clif
index 762f8a1038..5f4632041d 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-tables.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-tables.clif
@@ -2,7 +2,7 @@ test legalizer
 target x86_64
 
 ; Test legalization for various forms of table addresses.
-; regex: EBB=ebb\d+
+; regex: BB=block\d+
 
 function %table_addrs(i32, i64, i64 vmctx) {
     gv4 = vmctx
@@ -20,12 +20,12 @@ function %table_addrs(i32, i64, i64 vmctx) {
     ; check: table2 = dynamic gv0, min 0x0001_0000, bound gv1, element_size 1, index_type i64
     ; check: table3 = dynamic gv0, min 0, bound gv1, element_size 16, index_type i64
 
-ebb0(v0: i32, v1: i64, v3: i64):
+block0(v0: i32, v1: i64, v3: i64):
     v4 = table_addr.i64 table0, v0, +0
     ; check:         v8 = load.i32 notrap aligned v3+88
     ; check:         v9 = icmp uge v0, v8
-    ; check:         brz v9, $(resume_1=$EBB)
-    ; nextln:        jump $(trap_1=$EBB)
+    ; check:         brz v9, $(resume_1=$BB)
+    ; nextln:        jump $(trap_1=$BB)
     ; check:     $trap_1:
     ; nextln:        trap table_oob
     ; check:     $resume_1:
@@ -36,8 +36,8 @@ ebb0(v0: i32, v1: i64, v3: i64):
     v5 = table_addr.i64 table1, v0, +0
     ; check:         v12 = load.i32 notrap aligned v3+88
     ; check:         v13 = icmp.i32 uge v0, v12
-    ; check:         brz v13, $(resume_2=$EBB)
-    ; nextln:        jump $(trap_2=$EBB)
+    ; check:         brz v13, $(resume_2=$BB)
+    ; nextln:        jump $(trap_2=$BB)
     ; check:     $trap_2:
     ; nextln:        trap table_oob
     ; check:     $resume_2:
@@ -49,8 +49,8 @@ ebb0(v0: i32, v1: i64, v3: i64):
     v6 = table_addr.i64 table2, v1, +0
     ; check:         v17 = iadd_imm.i64 v3, 80
     ; check:         v18 = icmp.i64 uge v1, v17
-    ; check:         brz v18, $(resume_3=$EBB)
-    ; nextln:        jump $(trap_3=$EBB)
+    ; check:         brz v18, $(resume_3=$BB)
+    ; nextln:        jump $(trap_3=$BB)
     ; check:     $trap_3:
     ; nextln:        trap table_oob
     ; check:     $resume_3:
@@ -60,8 +60,8 @@ ebb0(v0: i32, v1: i64, v3: i64):
     v7 = table_addr.i64 table3, v1, +0
     ; check:         v20 = iadd_imm.i64 v3, 80
     ; check:         v21 = icmp.i64 uge v1, v20
-    ; check:         brz v21, $(resume_4=$EBB)
-    ; nextln:        jump $(trap_4=$EBB)
+    ; check:         brz v21, $(resume_4=$BB)
+    ; nextln:        jump $(trap_4=$BB)
     ; check:     $trap_4:
     ; nextln:        trap table_oob
     ; check:     $resume_4:
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-urem-i8.clif b/cranelift/filetests/filetests/isa/x86/legalize-urem-i8.clif
index 0c66a3f580..de193c2abb 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-urem-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-urem-i8.clif
@@ -4,7 +4,7 @@ target x86_64
 ; regex: V=v\d+
 
 function u0:0(i8, i8) -> i8 fast {
-ebb0(v0: i8, v1: i8):
+block0(v0: i8, v1: i8):
     v2 = urem v0, v1
     ; check: $(a=$V) = uextend.i32 v0
     ; nextln: $(b=$V) = uextend.i32 v1
diff --git a/cranelift/filetests/filetests/isa/x86/load-store-narrow.clif b/cranelift/filetests/filetests/isa/x86/load-store-narrow.clif
index 5f95b92fc0..070b7459e2 100644
--- a/cranelift/filetests/filetests/isa/x86/load-store-narrow.clif
+++ b/cranelift/filetests/filetests/isa/x86/load-store-narrow.clif
@@ -2,14 +2,14 @@ test compile
 target i686
 
 function u0:0(i64, i32) system_v {
-ebb0(v0: i64, v1: i32):
+block0(v0: i64, v1: i32):
     v2 = bor v0, v0
     store v2, v1
     return
 }
 
 function u0:1(i32) -> i64 system_v {
-ebb0(v1: i32):
+block0(v1: i32):
     v0 = load.i64 v1
     v2 = bor v0, v0
     return v2
diff --git a/cranelift/filetests/filetests/isa/x86/nop.clif b/cranelift/filetests/filetests/isa/x86/nop.clif
index 2863185e41..08d4fdd7a0 100644
--- a/cranelift/filetests/filetests/isa/x86/nop.clif
+++ b/cranelift/filetests/filetests/isa/x86/nop.clif
@@ -3,7 +3,7 @@ test compile
 target x86_64
 
 function %test(i32) -> i32 system_v {
-ebb0(v0: i32):
+block0(v0: i32):
     nop
     v1 = iconst.i32 42
     return v1
diff --git a/cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif
index 7dbbcc86e0..0f0f06e6f2 100644
--- a/cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif
+++ b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants-32bit.clif
@@ -4,21 +4,21 @@ set opt_level=speed_and_size
 target i686
 
 function %foo() -> f32 fast {
-ebb0:
+block0:
   ; asm: xorps %xmm0, %xmm0
   [-,%xmm0]    v0 = f32const 0.0     ; bin: 0f 57 c0
   return v0
 }
 
 function %bar() -> f64 fast {
-ebb0:
+block0:
   ; asm: xorpd %xmm0, %xmm0
   [-,%xmm0]    v1 = f64const 0.0     ; bin: 66 0f 57 c0
   return v1
 }
 
 function %zero_dword() -> i32 fast {
-ebb0:
+block0:
   ; asm: xor %eax, %eax
   [-,%rax]     v0 = iconst.i32 0     ; bin: 31 c0
   ; asm: xor %edi, %edi
@@ -27,7 +27,7 @@ ebb0:
 }
 
 function %zero_word() -> i16 fast {
-ebb0:
+block0:
   ; while you may expect this to be encoded like 6631c0, aka
   ; xor %ax, %ax, the upper 16 bits of the register used for
   ; i16 are left undefined, so it's not wrong to clear them.
@@ -43,7 +43,7 @@ ebb0:
 }
 
 function %zero_byte() -> i8 fast {
-ebb0:
+block0:
   ; asm: xor %al, %al
   [-,%rax]     v0 = iconst.i8 0     ; bin: 30 c0
   ; asm: xor %dh, %dh
diff --git a/cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif
index 807466e84c..7f5890a1ae 100644
--- a/cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif
+++ b/cranelift/filetests/filetests/isa/x86/optimized-zero-constants.clif
@@ -4,35 +4,35 @@ set opt_level=speed_and_size
 target x86_64
 
 function %zero_const_32bit_no_rex() -> f32 fast {
-ebb0:
+block0:
   ; asm: xorps %xmm0, %xmm0
   [-,%xmm0]    v0 = f32const 0.0     ; bin: 0f 57 c0
   return v0
 }
 
 function %zero_const_32bit_rex() -> f32 fast {
-ebb0:
+block0:
   ; asm: xorps %xmm8, %xmm8
   [-,%xmm8]    v1 = f32const 0.0     ; bin: 45 0f 57 c0
   return v1
 }
 
 function %zero_const_64bit_no_rex() -> f64 fast {
-ebb0:
+block0:
   ; asm: xorpd %xmm0, %xmm0
   [-,%xmm0]    v0 = f64const 0.0     ; bin: 66 0f 57 c0
   return v0
 }
 
 function %zero_const_64bit_rex() -> f64 fast {
-ebb0:
+block0:
   ; asm: xorpd %xmm8, %xmm8
   [-,%xmm8]    v1 = f64const 0.0     ; bin: 66 45 0f 57 c0
   return v1
 }
 
 function %imm_zero_register() -> i64 fast {
-ebb0:
+block0:
   ; asm: xor %eax, %eax
   [-,%rax]     v0 = iconst.i64 0     ; bin: 31 c0
   ; asm: xor %edi, %edi
@@ -45,7 +45,7 @@ ebb0:
 }
 
 function %zero_word() -> i16 fast {
-ebb0:
+block0:
   ; while you may expect this to be encoded like 6631c0, aka
   ; xor %ax, %ax, the upper 16 bits of the register used for
   ; i16 are left undefined, so it's not wrong to clear them.
@@ -61,7 +61,7 @@ ebb0:
 }
 
 function %zero_byte() -> i8 fast {
-ebb0:
+block0:
   ; asm: xor %r8b, %r8b
   [-,%r15]     v0 = iconst.i8 0     ; bin: 45 30 ff
   ; asm: xor %al, %al
diff --git a/cranelift/filetests/filetests/isa/x86/pinned-reg.clif b/cranelift/filetests/filetests/isa/x86/pinned-reg.clif
index 2a447a6d9d..f4bbc2501b 100644
--- a/cranelift/filetests/filetests/isa/x86/pinned-reg.clif
+++ b/cranelift/filetests/filetests/isa/x86/pinned-reg.clif
@@ -11,7 +11,7 @@ target x86_64
 ; r15 is the pinned heap register. It must not be rewritten, so it must not be
 ; used as a tied output register.
 function %tied_input() -> i64 system_v {
-ebb0:
+block0:
     v1 = get_pinned_reg.i64
     v2 = iadd_imm v1, 42
     return v2
@@ -25,7 +25,7 @@ ebb0:
 
 ;; It musn't be used even if this is a tied input used twice.
 function %tied_twice() -> i64 system_v {
-ebb0:
+block0:
     v1 = get_pinned_reg.i64
     v2 = iadd v1, v1
     return v2
@@ -38,7 +38,7 @@ ebb0:
 ; sameln: iadd v1, v1
 
 function %uses() -> i64 system_v {
-ebb0:
+block0:
     v1 = get_pinned_reg.i64
     v2 = iadd_imm v1, 42
     v3 = get_pinned_reg.i64
@@ -62,7 +62,7 @@ function u0:1(i64 vmctx) -> i64 system_v {
     gv0 = vmctx
     heap0 = static gv0, min 0x000a_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000, index_type i32
 
-ebb0(v42: i64):
+block0(v42: i64):
     v5 = iconst.i32 42
     v6 = heap_addr.i64 heap0, v5, 0
     v7 = load.i64 v6
diff --git a/cranelift/filetests/filetests/isa/x86/probestack-adjusts-sp.clif b/cranelift/filetests/filetests/isa/x86/probestack-adjusts-sp.clif
index 934e308bc3..17be399a4e 100644
--- a/cranelift/filetests/filetests/isa/x86/probestack-adjusts-sp.clif
+++ b/cranelift/filetests/filetests/isa/x86/probestack-adjusts-sp.clif
@@ -8,7 +8,7 @@ target x86_64
 
 function %big() system_v {
     ss0 = explicit_slot 300000
-ebb0:
+block0:
     return
 }
 ; check: function %big(i64 fp [%rbp]) -> i64 fp [%rbp] system_v {
@@ -17,7 +17,7 @@ ebb0:
 ; nextln:    sig0 = (i64 [%rax]) probestack
 ; nextln:    fn0 = colocated %Probestack sig0
 ; nextln: 
-; nextln:                                 ebb0(v0: i64 [%rbp]):
+; nextln:                                 block0(v0: i64 [%rbp]):
 ; nextln: [RexOp1pushq#50]                    x86_push v0
 ; nextln: [RexOp1copysp#8089]                 copy_special %rsp -> %rbp
 ; nextln: [RexOp1pu_id#b8,%rax]               v1 = iconst.i64 0x0004_93e0
diff --git a/cranelift/filetests/filetests/isa/x86/probestack-disabled.clif b/cranelift/filetests/filetests/isa/x86/probestack-disabled.clif
index f548e1a11f..433c634cab 100644
--- a/cranelift/filetests/filetests/isa/x86/probestack-disabled.clif
+++ b/cranelift/filetests/filetests/isa/x86/probestack-disabled.clif
@@ -7,14 +7,14 @@ target x86_64
 
 function %big() system_v {
     ss0 = explicit_slot 300000
-ebb0:
+block0:
     return
 }
 ; check: function %big(i64 fp [%rbp]) -> i64 fp [%rbp] system_v {
 ; nextln:    ss0 = explicit_slot 300000, offset -300016
 ; nextln:    ss1 = incoming_arg 16, offset -16
 ; nextln: 
-; nextln:                                 ebb0(v0: i64 [%rbp]):
+; nextln:                                 block0(v0: i64 [%rbp]):
 ; nextln: [RexOp1pushq#50]                    x86_push v0
 ; nextln: [RexOp1copysp#8089]                 copy_special %rsp -> %rbp
 ; nextln: [RexOp1adjustsp_id#d081]            adjust_sp_down_imm 0x0004_93e0
diff --git a/cranelift/filetests/filetests/isa/x86/probestack-noncolocated.clif b/cranelift/filetests/filetests/isa/x86/probestack-noncolocated.clif
index c304945231..9af61f0586 100644
--- a/cranelift/filetests/filetests/isa/x86/probestack-noncolocated.clif
+++ b/cranelift/filetests/filetests/isa/x86/probestack-noncolocated.clif
@@ -5,7 +5,7 @@ target x86_64
 
 function %big() system_v {
     ss0 = explicit_slot 300000
-ebb0:
+block0:
     return
 }
 ; check: function %big(i64 fp [%rbp]) -> i64 fp [%rbp] system_v {
@@ -14,7 +14,7 @@ ebb0:
 ; nextln:    sig0 = (i64 [%rax]) -> i64 [%rax] probestack
 ; nextln:    fn0 = %Probestack sig0
 ; nextln: 
-; nextln:                                 ebb0(v0: i64 [%rbp]):
+; nextln:                                 block0(v0: i64 [%rbp]):
 ; nextln: [RexOp1pushq#50]                    x86_push v0
 ; nextln: [RexOp1copysp#8089]                 copy_special %rsp -> %rbp
 ; nextln: [RexOp1pu_id#b8,%rax]               v1 = iconst.i64 0x0004_93e0
diff --git a/cranelift/filetests/filetests/isa/x86/probestack-size.clif b/cranelift/filetests/filetests/isa/x86/probestack-size.clif
index 6b01b5cd92..8eb934b06c 100644
--- a/cranelift/filetests/filetests/isa/x86/probestack-size.clif
+++ b/cranelift/filetests/filetests/isa/x86/probestack-size.clif
@@ -8,7 +8,7 @@ target x86_64
 
 function %big() system_v {
     ss0 = explicit_slot 4097
-ebb0:
+block0:
     return
 }
 
@@ -16,7 +16,7 @@ ebb0:
 ; nextln:     ss0 = explicit_slot 4097, offset -4113
 ; nextln:     ss1 = incoming_arg 16, offset -16
 ; nextln: 
-; nextln:                                 ebb0(v0: i64 [%rbp]):
+; nextln:                                 block0(v0: i64 [%rbp]):
 ; nextln: [RexOp1pushq#50]                    x86_push v0
 ; nextln: [RexOp1copysp#8089]                 copy_special %rsp -> %rbp
 ; nextln: [RexOp1adjustsp_id#d081]            adjust_sp_down_imm 4112
@@ -30,7 +30,7 @@ ebb0:
 
 function %bigger() system_v {
     ss0 = explicit_slot 8192
-ebb0:
+block0:
     return
 }
 
@@ -38,7 +38,7 @@ ebb0:
 ; nextln:     ss0 = explicit_slot 8192, offset -8208
 ; nextln:     ss1 = incoming_arg 16, offset -16
 ; nextln: 
-; nextln:                                 ebb0(v0: i64 [%rbp]):
+; nextln:                                 block0(v0: i64 [%rbp]):
 ; nextln: [RexOp1pushq#50]                    x86_push v0
 ; nextln: [RexOp1copysp#8089]                 copy_special %rsp -> %rbp
 ; nextln: [RexOp1adjustsp_id#d081]            adjust_sp_down_imm 8192
@@ -52,7 +52,7 @@ ebb0:
 
 function %biggest() system_v {
     ss0 = explicit_slot 8193
-ebb0:
+block0:
     return
 }
 
@@ -62,7 +62,7 @@ ebb0:
 ; nextln:     sig0 = (i64 [%rax]) -> i64 [%rax] probestack
 ; nextln:     fn0 = colocated %Probestack sig0
 ; nextln: 
-; nextln:                                 ebb0(v0: i64 [%rbp]):
+; nextln:                                 block0(v0: i64 [%rbp]):
 ; nextln: [RexOp1pushq#50]                    x86_push v0
 ; nextln: [RexOp1copysp#8089]                 copy_special %rsp -> %rbp
 ; nextln: [RexOp1pu_id#b8,%rax]               v1 = iconst.i64 8208
diff --git a/cranelift/filetests/filetests/isa/x86/probestack.clif b/cranelift/filetests/filetests/isa/x86/probestack.clif
index 0fae5f33e2..d9f29a8681 100644
--- a/cranelift/filetests/filetests/isa/x86/probestack.clif
+++ b/cranelift/filetests/filetests/isa/x86/probestack.clif
@@ -6,7 +6,7 @@ target x86_64
 
 function %big() system_v {
     ss0 = explicit_slot 4097
-ebb0:
+block0:
     return
 }
 ; check: function %big(i64 fp [%rbp]) -> i64 fp [%rbp] system_v {
@@ -15,7 +15,7 @@ ebb0:
 ; nextln:    sig0 = (i64 [%rax]) -> i64 [%rax] probestack
 ; nextln:    fn0 = colocated %Probestack sig0
 ; nextln: 
-; nextln:                                 ebb0(v0: i64 [%rbp]):
+; nextln:                                 block0(v0: i64 [%rbp]):
 ; nextln: [RexOp1pushq#50]                    x86_push v0
 ; nextln: [RexOp1copysp#8089]                 copy_special %rsp -> %rbp
 ; nextln: [RexOp1pu_id#b8,%rax]               v1 = iconst.i64 4112
@@ -31,7 +31,7 @@ ebb0:
 
 function %small() system_v {
     ss0 = explicit_slot 4096
-ebb0:
+block0:
     return
 }
 
@@ -39,7 +39,7 @@ ebb0:
 ; nextln:     ss0 = explicit_slot 4096, offset -4112
 ; nextln:     ss1 = incoming_arg 16, offset -16
 ; nextln: 
-; nextln:                                 ebb0(v0: i64 [%rbp]):
+; nextln:                                 block0(v0: i64 [%rbp]):
 ; nextln: [RexOp1pushq#50]                    x86_push v0
 ; nextln: [RexOp1copysp#8089]                 copy_special %rsp -> %rbp
 ; nextln: [RexOp1adjustsp_id#d081]            adjust_sp_down_imm 4096
diff --git a/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif b/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
index f8a0c0146c..25118ca72b 100644
--- a/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
+++ b/cranelift/filetests/filetests/isa/x86/prologue-epilogue.clif
@@ -6,14 +6,14 @@ target x86_64 haswell
 ; An empty function.
 
 function %empty() {
-ebb0:
+block0:
     return
 }
 
 ; check: function %empty(i64 fp [%rbp]) -> i64 fp [%rbp] fast {
 ; nextln:     ss0 = incoming_arg 16, offset -16
 ; nextln: 
-; nextln: ebb0(v0: i64 [%rbp]):
+; nextln: block0(v0: i64 [%rbp]):
 ; nextln:     x86_push v0
 ; nextln:     copy_special %rsp -> %rbp
 ; nextln:     v1 = x86_pop.i64
@@ -24,7 +24,7 @@ ebb0:
 
 function %one_stack_slot() {
     ss0 = explicit_slot 168
-ebb0:
+block0:
     return
 }
 
@@ -32,7 +32,7 @@ ebb0:
 ; nextln:     ss0 = explicit_slot 168, offset -184
 ; nextln:     ss1 = incoming_arg 16, offset -16
 ; nextln: 
-; nextln: ebb0(v0: i64 [%rbp]):
+; nextln: block0(v0: i64 [%rbp]):
 ; nextln:     x86_push v0
 ; nextln:     copy_special %rsp -> %rbp
 ; nextln:     adjust_sp_down_imm 176
@@ -46,7 +46,7 @@ ebb0:
 function %call() {
     fn0 = %foo()
 
-ebb0:
+block0:
     call fn0()
     return
 }
@@ -56,7 +56,7 @@ ebb0:
 ; nextln:     sig0 = () fast
 ; nextln:     fn0 = %foo sig0
 ; nextln: 
-; nextln: ebb0(v0: i64 [%rbp]):
+; nextln: block0(v0: i64 [%rbp]):
 ; nextln:     x86_push v0
 ; nextln:     copy_special %rsp -> %rbp
 ; nextln:     call fn0()
@@ -67,7 +67,7 @@ ebb0:
 ; A function that uses a lot of registers but doesn't quite need to spill.
 
 function %no_spill(i64, i64) {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = load.i32 v0+0
     v3 = load.i32 v0+8
     v4 = load.i32 v0+16
@@ -100,7 +100,7 @@ ebb0(v0: i64, v1: i64):
 ; check: function %no_spill(i64 [%rdi], i64 [%rsi], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] fast {
 ; nextln:     ss0 = incoming_arg 56, offset -56
 ; nextln: 
-; nextln: ebb0(v0: i64 [%rdi], v1: i64 [%rsi], v15: i64 [%rbp], v16: i64 [%rbx], v17: i64 [%r12], v18: i64 [%r13], v19: i64 [%r14], v20: i64 [%r15]):
+; nextln: block0(v0: i64 [%rdi], v1: i64 [%rsi], v15: i64 [%rbp], v16: i64 [%rbx], v17: i64 [%r12], v18: i64 [%r13], v19: i64 [%r14], v20: i64 [%r15]):
 ; nextln:     x86_push v15
 ; nextln:     copy_special %rsp -> %rbp
 ; nextln:     x86_push v16
@@ -146,7 +146,7 @@ ebb0(v0: i64, v1: i64):
 ; This function requires too many registers and must spill.
 
 function %yes_spill(i64, i64) {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = load.i32 v0+0
     v3 = load.i32 v0+8
     v4 = load.i32 v0+16
@@ -181,7 +181,7 @@ ebb0(v0: i64, v1: i64):
 ; check: function %yes_spill(i64 [%rdi], i64 [%rsi], i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15]) -> i64 fp [%rbp], i64 csr [%rbx], i64 csr [%r12], i64 csr [%r13], i64 csr [%r14], i64 csr [%r15] fast {
 ; check:     ss0 = spill_slot
 
-; check: ebb0(v16: i64 [%rdi], v17: i64 [%rsi], v48: i64 [%rbp], v49: i64 [%rbx], v50: i64 [%r12], v51: i64 [%r13], v52: i64 [%r14], v53: i64 [%r15]):
+; check: block0(v16: i64 [%rdi], v17: i64 [%rsi], v48: i64 [%rbp], v49: i64 [%rbx], v50: i64 [%r12], v51: i64 [%r13], v52: i64 [%r14], v53: i64 [%r15]):
 ; nextln:     x86_push v48
 ; nextln:     copy_special %rsp -> %rbp
 ; nextln:     x86_push v49
@@ -208,21 +208,21 @@ ebb0(v0: i64, v1: i64):
 ; A function which uses diverted registers.
 
 function %divert(i32) -> i32 system_v {
-ebb0(v0: i32):
+block0(v0: i32):
     v2 = iconst.i32 0
     v3 = iconst.i32 1
-    jump ebb1(v0, v3, v2)
+    jump block1(v0, v3, v2)
 
-ebb1(v4: i32, v5: i32, v6: i32):
-    brz v4, ebb3
-    jump ebb2
+block1(v4: i32, v5: i32, v6: i32):
+    brz v4, block3
+    jump block2
 
-ebb2:
+block2:
     v7 = iadd v5, v6
     v8 = iadd_imm v4, -1
-    jump ebb1(v8, v7, v5)
+    jump block1(v8, v7, v5)
 
-ebb3:
+block3:
     return v5
 }
 
@@ -234,7 +234,7 @@ ebb3:
 
 function %stack_limit(i64 stack_limit) {
     ss0 = explicit_slot 168
-ebb0(v0: i64):
+block0(v0: i64):
     return
 }
 
@@ -242,7 +242,7 @@ ebb0(v0: i64):
 ; nextln:     ss0 = explicit_slot 168, offset -184
 ; nextln:     ss1 = incoming_arg 16, offset -16
 ; nextln: 
-; nextln: ebb0(v0: i64 [%rdi], v4: i64 [%rbp]):
+; nextln: block0(v0: i64 [%rdi], v4: i64 [%rbp]):
 ; nextln:     v1 = copy v0
 ; nextln:     v2 = iadd_imm v1, 16
 ; nextln:     v3 = ifcmp_sp v2
diff --git a/cranelift/filetests/filetests/isa/x86/pshufb.clif b/cranelift/filetests/filetests/isa/x86/pshufb.clif
index 6fb31b198c..c9d5d798d9 100644
--- a/cranelift/filetests/filetests/isa/x86/pshufb.clif
+++ b/cranelift/filetests/filetests/isa/x86/pshufb.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64 has_ssse3=true
 
 function %test_pshufb() {
-ebb0:
+block0:
 [-, %rax]   v0 = iconst.i8 42
 [-, %xmm0]  v1 = scalar_to_vector.i8x16 v0   ; bin: 66 40 0f 6e c0
 [-, %rbx]   v2 = iconst.i8 43
diff --git a/cranelift/filetests/filetests/isa/x86/pshufd.clif b/cranelift/filetests/filetests/isa/x86/pshufd.clif
index 6f4896d0d9..69dc3f4ea0 100644
--- a/cranelift/filetests/filetests/isa/x86/pshufd.clif
+++ b/cranelift/filetests/filetests/isa/x86/pshufd.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64
 
 function %test_pshuf() {
-ebb0:
+block0:
 [-, %rax]   v0 = iconst.i32 42
 [-, %xmm0]  v1 = scalar_to_vector.i32x4 v0  ; bin: 66 40 0f 6e c0
 [-, %xmm0]  v2 = x86_pshufd v1, 0                ; bin: 66 0f 70 c0 00
diff --git a/cranelift/filetests/filetests/isa/x86/raw_bitcast.clif b/cranelift/filetests/filetests/isa/x86/raw_bitcast.clif
index 5c1c2ea322..717f655ec6 100644
--- a/cranelift/filetests/filetests/isa/x86/raw_bitcast.clif
+++ b/cranelift/filetests/filetests/isa/x86/raw_bitcast.clif
@@ -2,7 +2,7 @@ test binemit
 target x86_64
 
 function %test_raw_bitcast_i16x8_to_b32x4() {
-ebb0:
+block0:
 [-, %rbx]   v0 = bconst.b16 true
 [-, %xmm2]  v1 = scalar_to_vector.b16x8 v0
 [-, %xmm2]  v2 = raw_bitcast.i32x4 v1       ; bin:
diff --git a/cranelift/filetests/filetests/isa/x86/relax_branch.clif b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
index 7fb8a85167..8b29a057e0 100644
--- a/cranelift/filetests/filetests/isa/x86/relax_branch.clif
+++ b/cranelift/filetests/filetests/isa/x86/relax_branch.clif
@@ -17,10 +17,10 @@ function u0:2691(i32 [%rdi], i32 [%rsi], i64 vmctx [%r14]) -> i64 uext [%rax] ba
     gv2 = load.i64 notrap aligned readonly gv0
     heap0 = static gv2, min 0xd839_6000, bound 0x0001_0000_0000, offset_guard 0x8000_0000, index_type i32
 
-                                ebb0(v0: i32 [%rdi], v1: i32 [%rsi], v2: i64 [%r14]):
-@0005 [-]                           fallthrough ebb3(v0, v1)
+                                block0(v0: i32 [%rdi], v1: i32 [%rsi], v2: i64 [%r14]):
+@0005 [-]                           fallthrough block3(v0, v1)
 
-                                ebb3(v8: i32 [%rdi], v19: i32 [%rsi]):
+                                block3(v8: i32 [%rdi], v19: i32 [%rsi]):
 @0005 [RexOp1ldDisp8#808b,%rax]        v7 = load.i64 v2+48
 @0005 [DynRexOp1rcmp_ib#f083,%rflags]  v91 = ifcmp_imm v7, 0
 @0005 [trapif#00]                      trapif ne v91, interrupt
@@ -54,20 +54,20 @@ function u0:2691(i32 [%rdi], i32 [%rsi], i64 vmctx [%r14]) -> i64 uext [%rax] ba
 @0048 [DynRexOp1r_ib#70c1,%rcx]        v35 = sshr_imm v33, 24
 @004c [DynRexOp1r_id#4081,%rcx]        v37 = band_imm v35, 255
 [DynRexOp1rcmp_ib#7083,%rflags]        v97 = ifcmp_imm v37, 26
-@0050 [Op1brib#70]                     brif sge v97, ebb6
-@0050 [-]                              fallthrough ebb10
+@0050 [Op1brib#70]                     brif sge v97, block6
+@0050 [-]                              fallthrough block10
 
-                                ebb10:
+                                block10:
 [DynRexOp1umr#89,%rcx]              v101 = copy v18
-@0054 [Op1jmpb#eb]                  jump ebb5(v18, v101)
+@0054 [Op1jmpb#eb]                  jump block5(v18, v101)
 
-                                ebb6:
+                                block6:
 [DynRexOp1umr#89,%rcx]              v102 = copy.i32 v16
 @0059 [RexOp1rmov#89]               regmove v102, %rcx -> %rdi
 @0059 [RexOp1rmov#89]               regmove.i32 v16, %rbx -> %rcx
-@0059 [-]                           fallthrough ebb5(v102, v16)
+@0059 [-]                           fallthrough block5(v102, v16)
 
-                                ebb5(v41: i32 [%rdi], v84: i32 [%rcx]):
+                                block5(v41: i32 [%rdi], v84: i32 [%rcx]):
                                     v83 -> v84
 @005d [DynRexOp1r_id#4081,%rdi]     v43 = band_imm v41, 255
 @0062 [DynRexOp1r_ib#40c1,%rdi]     v45 = ishl_imm v43, 24
@@ -84,49 +84,49 @@ function u0:2691(i32 [%rdi], i32 [%rsi], i64 vmctx [%r14]) -> i64 uext [%rax] ba
 @0080 [DynRexOp1r_id#4081,%rdx]     v63 = band_imm v61, 255
 [DynRexOp1rcmp_ib#7083,%rflags]     v98 = ifcmp_imm v63, 26
 @0084 [RexOp1rmov#89]               regmove v47, %rdi -> %rbx
-@0084 [Op1brib#70]                  brif sge v98, ebb8
-@0084 [-]                           fallthrough ebb11
+@0084 [Op1brib#70]                  brif sge v98, block8
+@0084 [-]                           fallthrough block11
 
-                                ebb11:
+                                block11:
 [DynRexOp1umr#89,%rdx]              v103 = copy.i32 v29
-@0088 [Op1jmpb#eb]                  jump ebb7(v29, v10, v21, v103)
+@0088 [Op1jmpb#eb]                  jump block7(v29, v10, v21, v103)
 
-                                ebb8:
+                                block8:
 [DynRexOp1umr#89,%rdx]              v104 = copy.i32 v27
 @008d [RexOp1rmov#89]               regmove v104, %rdx -> %r9
 @008d [RexOp1rmov#89]               regmove.i32 v27, %rsi -> %rdx
-@008d [-]                           fallthrough ebb7(v104, v10, v21, v27)
+@008d [-]                           fallthrough block7(v104, v10, v21, v27)
 
-                                ebb7(v67: i32 [%r9], v79: i32 [%rax], v81: i32 [%r8], v87: i32 [%rdx]):
+                                block7(v67: i32 [%r9], v79: i32 [%rax], v81: i32 [%r8], v87: i32 [%rdx]):
 @0091 [DynRexOp1r_id#4081,%r9]           v71 = band_imm v67, 255
 @0094 [DynRexOp1r_ib#40c1,%r9]           v73 = ishl_imm v71, 24
 @0097 [DynRexOp1r_ib#70c1,%r9]           v75 = sshr_imm v73, 24
 @0098 [DynRexOp1icscc#39,%rbx]           v76 = icmp.i32 eq v47, v75
 @0098 [Op2urm_noflags_abcd#4b6,%rbx]     v77 = bint.i32 v76
 @0099 [DynRexOp1rr#21,%r10]              v78 = band.i32 v50, v77
-@009a [RexOp1tjccb#74]                   brz v78, ebb9
-@009a [-]                                fallthrough ebb12
+@009a [RexOp1tjccb#74]                   brz v78, block9
+@009a [-]                                fallthrough block12
 
-                                ebb12:
+                                block12:
 [DynRexOp1umr#89,%rcx]              v99 = copy v81
 [DynRexOp1umr#89,%rdx]              v100 = copy v79
 @00a4 [RexOp1rmov#89]               regmove v100, %rdx -> %rdi
 @00a4 [RexOp1rmov#89]               regmove v99, %rcx -> %rsi
-@00a4 [Op1jmpd#e9]                  jump ebb3(v100, v99); bin: e9 ffffff2d
+@00a4 [Op1jmpd#e9]                  jump block3(v100, v99); bin: e9 ffffff2d
 
-                                ebb9:
-@00a7 [-]                           fallthrough ebb4
+                                block9:
+@00a7 [-]                           fallthrough block4
 
-                                ebb4:
+                                block4:
 @00ad [DynRexOp1r_id#4081,%rcx]     v86 = band_imm.i32 v84, 255
 @00b3 [DynRexOp1r_id#4081,%rdx]     v89 = band_imm.i32 v87, 255
 @00b4 [DynRexOp1rr#29,%rcx]         v90 = isub v86, v89
-@00b5 [-]                           fallthrough ebb2(v90)
+@00b5 [-]                           fallthrough block2(v90)
 
-                                ebb2(v5: i32 [%rcx]):
-@00b6 [-]                           fallthrough ebb1(v5)
+                                block2(v5: i32 [%rcx]):
+@00b6 [-]                           fallthrough block1(v5)
 
-                                ebb1(v3: i32 [%rcx]):
+                                block1(v3: i32 [%rcx]):
 @00b6 [Op1umr#89,%rax]              v96 = uextend.i64 v3
 @00b6 [-]                           fallthrough_return v96
 }
diff --git a/cranelift/filetests/filetests/isa/x86/run-const.clif b/cranelift/filetests/filetests/isa/x86/run-const.clif
index 1ac5062e49..c39d39adfb 100644
--- a/cranelift/filetests/filetests/isa/x86/run-const.clif
+++ b/cranelift/filetests/filetests/isa/x86/run-const.clif
@@ -1,7 +1,7 @@
 test run
 
 function %test_compare_i32() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i32 42
     v1 = iconst.i32 42
     v2 = icmp eq v0, v1
diff --git a/cranelift/filetests/filetests/isa/x86/run-i64.clif b/cranelift/filetests/filetests/isa/x86/run-i64.clif
index 6fae71966e..ae4a618573 100644
--- a/cranelift/filetests/filetests/isa/x86/run-i64.clif
+++ b/cranelift/filetests/filetests/isa/x86/run-i64.clif
@@ -3,14 +3,14 @@ test compile
 target i686 haswell
 
 function %iadd(i64, i64) -> i64 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = iadd v1, v2
     ; check: iadd_ifcout
     return v10
 }
 
 function %isub(i64, i64) -> i64 {
-ebb0(v1: i64, v2: i64):
+block0(v1: i64, v2: i64):
     v10 = isub v1, v2
     ; check: isub_ifbout
     return v10
diff --git a/cranelift/filetests/filetests/isa/x86/saturating-float-cast.clif b/cranelift/filetests/filetests/isa/x86/saturating-float-cast.clif
index 5986e1f864..36b69ca8b7 100644
--- a/cranelift/filetests/filetests/isa/x86/saturating-float-cast.clif
+++ b/cranelift/filetests/filetests/isa/x86/saturating-float-cast.clif
@@ -2,7 +2,7 @@ test compile
 target x86_64
 
 function u0:0() -> f32 system_v {
-ebb0:
+block0:
     v0 = iconst.i8 255
 ; check: v2 = iconst.i32 255
 ; nextln: v0 = ireduce.i8 v2
diff --git a/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif
index 112939cd7d..4a02e6bac6 100644
--- a/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-binemit.clif
@@ -4,28 +4,28 @@ set enable_simd
 target x86_64
 
 function %test_scalar_to_vector_b8() {
-ebb0:
+block0:
 [-, %rax]   v0 = bconst.b8 true
 [-, %xmm0]  v1 = scalar_to_vector.b8x16 v0    ; bin: 66 0f 6e c0
             return
 }
 
 function %test_scalar_to_vector_i16() {
-ebb0:
+block0:
 [-, %rbx]   v0 = iconst.i16 42
 [-, %xmm2]  v1 = scalar_to_vector.i16x8 v0    ; bin: 66 0f 6e d3
             return
 }
 
 function %test_scalar_to_vector_b32() {
-ebb0:
+block0:
 [-, %rcx]   v0 = bconst.b32 false
 [-, %xmm3]  v1 = scalar_to_vector.b32x4 v0    ; bin: 66 0f 6e d9
             return
 }
 
 function %test_scalar_to_vector_i64() {
-ebb0:
+block0:
 [-, %rdx]   v0 = iconst.i64 42
 [-, %xmm7]  v1 = scalar_to_vector.i64x2 v0    ; bin: 66 48 0f 6e fa
             return
diff --git a/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif
index 7fa27eecc6..8ab62db59d 100644
--- a/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif
+++ b/cranelift/filetests/filetests/isa/x86/scalar_to_vector-compile.clif
@@ -6,13 +6,13 @@ target x86_64
 
 ; ensure that scalar_to_vector emits no instructions for floats (already exist in an XMM register)
 function %test_scalar_to_vector_f32() -> f32x4 baldrdash_system_v {
-ebb0:
+block0:
     v0 = f32const 0x0.42
     v1 = scalar_to_vector.f32x4 v0
     return v1
 }
 
-; check: ebb0
+; check: block0
 ; nextln: v2 = iconst.i32 0x3e84_0000
 ; nextln: v0 = bitcast.f32 v2
 ; nextln: [null_fpr#00,%xmm0]                 v1 = scalar_to_vector.f32x4 v0
diff --git a/cranelift/filetests/filetests/isa/x86/select-i8.clif b/cranelift/filetests/filetests/isa/x86/select-i8.clif
index aac59c1e9c..feec520860 100644
--- a/cranelift/filetests/filetests/isa/x86/select-i8.clif
+++ b/cranelift/filetests/filetests/isa/x86/select-i8.clif
@@ -2,7 +2,7 @@ test compile
 target x86_64
 
 function u0:0(b1, i8, i8) -> i8 {
-ebb0(v0: b1, v1: i8, v2: i8):
+block0(v0: b1, v1: i8, v2: i8):
     v3 = select v0, v1, v2
     return v3
 }
diff --git a/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif b/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
index 50e2389feb..f896d8cc25 100644
--- a/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
+++ b/cranelift/filetests/filetests/isa/x86/shrink-multiple-uses.clif
@@ -3,16 +3,16 @@ set opt_level=speed_and_size
 target x86_64
 
 function %test_multiple_uses(i32 [%rdi]) -> i32 {
-ebb0(v0: i32 [%rdi]):
+block0(v0: i32 [%rdi]):
 [DynRexOp1rcmp_ib#7083,%rflags]     v3 = ifcmp_imm v0, 0
 [Op2seti_abcd#490,%rax]             v1 = trueif eq v3
 [RexOp2urm_noflags#4b6,%rax]        v2 = bint.i32 v1
-[Op1brib#70]                        brif eq v3, ebb1
-[Op1jmpb#eb]                        jump ebb2
+[Op1brib#70]                        brif eq v3, block1
+[Op1jmpb#eb]                        jump block2
 
-ebb2:
+block2:
 [Op1ret#c3]                         return v2
 
-ebb1:
+block1:
 [Op2trap#40b]                       trap user0
 }
diff --git a/cranelift/filetests/filetests/isa/x86/shrink.clif b/cranelift/filetests/filetests/isa/x86/shrink.clif
index b0d3174ece..9b0832b2a7 100644
--- a/cranelift/filetests/filetests/isa/x86/shrink.clif
+++ b/cranelift/filetests/filetests/isa/x86/shrink.clif
@@ -10,7 +10,7 @@ target x86_64
 ;
 
 function %test_shrinking(i32) -> i32 {
-ebb0(v0: i32 [ %rdi ]):
+block0(v0: i32 [ %rdi ]):
            ; asm: movl $0x2,%eax
 [-,%rcx]   v1 = iconst.i32 2     ; bin: b9 00000002
            ; asm: subl %ecx,%edi
@@ -19,7 +19,7 @@ ebb0(v0: i32 [ %rdi ]):
 }
 
 function %test_not_shrinking(i32) -> i32 {
-ebb0(v0: i32 [ %r8 ]):
+block0(v0: i32 [ %r8 ]):
            ; asm: movl $0x2,%eax
 [-,%rcx]   v1 = iconst.i32 2     ; bin: b9 00000002
            ; asm: subl %ecx,%edi
@@ -28,7 +28,7 @@ ebb0(v0: i32 [ %r8 ]):
 }
 
 function %test_not_shrinking_i8() {
-ebb0:
+block0:
 [-,%rsi]   v1 = iconst.i8 1
            ; asm: movsbl %sil,%esi
 [-,%rsi]   v2 = sextend.i32 v1   ; bin: 40 0f be f6
diff --git a/cranelift/filetests/filetests/isa/x86/shuffle-legalize.clif b/cranelift/filetests/filetests/isa/x86/shuffle-legalize.clif
index d192489448..78c6bfef40 100644
--- a/cranelift/filetests/filetests/isa/x86/shuffle-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/shuffle-legalize.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64 skylake
 
 function %test_shuffle_different_ssa_values() -> i8x16 {
-ebb0:
+block0:
     v0 = vconst.i8x16 0x00
     v1 = vconst.i8x16 0x01
     v2 = shuffle v0, v1, 0x11000000000000000000000000000000     ; pick the second lane of v1, the rest use the first lane of v0
@@ -20,7 +20,7 @@ ebb0:
 
 
 function %test_shuffle_same_ssa_value() -> i8x16 {
-ebb0:
+block0:
     v1 = vconst.i8x16 0x01
     v2 = shuffle v1, v1, 0x13000000000000000000000000000000     ; pick the fourth lane of v1 and the rest from the first lane of v1
     return v2
diff --git a/cranelift/filetests/filetests/isa/x86/shuffle-run.clif b/cranelift/filetests/filetests/isa/x86/shuffle-run.clif
index bc9eecb689..44e4998907 100644
--- a/cranelift/filetests/filetests/isa/x86/shuffle-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/shuffle-run.clif
@@ -2,7 +2,7 @@ test run
 set enable_simd
 
 function %test_shuffle_different_ssa_values() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i8x16 0x00
     v1 = vconst.i8x16 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 42]
     v2 = shuffle v0, v1, [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 31]     ; use the first lane of v0 throughout except use the last lane of v1
@@ -15,7 +15,7 @@ ebb0:
 ; run
 
 function %test_shuffle_same_ssa_value() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i8x16 0x01000000_00000000_00000000_00000000     ; note where lane 15 is when written with hexadecimal syntax
     v1 = shuffle v0, v0, 0x0f0f0f0f_0f0f0f0f_0f0f0f0f_0f0f0f0f  ; use the last lane of v0 to fill all lanes
     v2 = extractlane.i8x16 v1, 4
@@ -27,7 +27,7 @@ ebb0:
 ; run
 
 function %compare_shuffle() -> b1 {
-ebb0:
+block0:
     v1 = vconst.i32x4 [0 1 2 3]
     v2 = raw_bitcast.i8x16  v1 ; we have to cast because shuffle is type-limited to Tx16
     ; keep each lane in place from the first vector
@@ -45,7 +45,7 @@ ebb0:
 
 
 function %compare_shuffle() -> b32 {
-ebb0:
+block0:
     v1 = vconst.b32x4 [true false true false]
     v2 = raw_bitcast.b8x16 v1 ; we have to cast because shuffle is type-limited to Tx16
     ; pair up the true values to make the entire vector true
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif
index 2994d36146..9f5b4f0080 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64 skylake
 
 function %iadd_i32x4() -> b1 {
-ebb0:
+block0:
 [-, %xmm0]    v0 = vconst.i32x4 [1 1 1 1]
 [-, %xmm1]    v1 = vconst.i32x4 [1 2 3 4]
 [-, %xmm0]    v2 = iadd v0, v1      ; bin: 66 0f fe c1
@@ -19,7 +19,7 @@ ebb0:
 }
 
 function %iadd_i8x16_with_overflow() -> b1 {
-ebb0:
+block0:
 [-, %xmm0]    v0 = vconst.i8x16 [255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255]
 [-, %xmm7]    v1 = vconst.i8x16 [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2]
 [-, %xmm0]    v2 = iadd v0, v1      ; bin: 66 0f fc c7
@@ -31,19 +31,19 @@ ebb0:
 }
 
 function %iadd_i16x8(i16x8, i16x8) -> i16x8 {
-ebb0(v0: i16x8 [%xmm1], v1: i16x8 [%xmm2]):
+block0(v0: i16x8 [%xmm1], v1: i16x8 [%xmm2]):
 [-, %xmm1]  v2 = iadd v0, v1      ; bin: 66 0f fd ca
             return v2
 }
 
 function %iadd_i64x2(i64x2, i64x2) -> i64x2 {
-ebb0(v0: i64x2 [%xmm3], v1: i64x2 [%xmm4]):
+block0(v0: i64x2 [%xmm3], v1: i64x2 [%xmm4]):
 [-, %xmm3]  v2 = iadd v0, v1      ; bin: 66 0f d4 dc
             return v2
 }
 
 function %isub_i32x4() -> b1 {
-ebb0:
+block0:
 [-, %xmm3]    v0 = vconst.i32x4 [1 1 1 1]
 [-, %xmm5]    v1 = vconst.i32x4 [1 2 3 4]
 [-, %xmm3]    v2 = isub v0, v1  ; bin: 66 0f fa dd
@@ -59,25 +59,25 @@ ebb0:
 }
 
 function %isub_i64x2(i64x2, i64x2) -> i64x2 {
-ebb0(v0: i64x2 [%xmm0], v1: i64x2 [%xmm1]):
+block0(v0: i64x2 [%xmm0], v1: i64x2 [%xmm1]):
 [-, %xmm0]  v2 = isub v0, v1      ; bin: 66 0f fb c1
     return v2
 }
 
 function %isub_i16x8(i16x8, i16x8) -> i16x8 {
-ebb0(v0: i16x8 [%xmm3], v1: i16x8 [%xmm4]):
+block0(v0: i16x8 [%xmm3], v1: i16x8 [%xmm4]):
 [-, %xmm3]  v2 = isub v0, v1      ; bin: 66 0f f9 dc
     return v2
 }
 
 function %isub_i8x16(i8x16, i8x16) -> i8x16 {
-ebb0(v0: i8x16 [%xmm3], v1: i8x16 [%xmm4]):
+block0(v0: i8x16 [%xmm3], v1: i8x16 [%xmm4]):
 [-, %xmm3]  v2 = isub v0, v1      ; bin: 66 0f f8 dc
     return v2
 }
 
 function %imul_i32x4() -> b1 {
-ebb0:
+block0:
 [-, %xmm0]    v0 = vconst.i32x4 [-1 0 1 -2147483647] ; e.g. -2147483647 == 0x80_00_00_01
 [-, %xmm1]    v1 = vconst.i32x4 [2 2 2 2]
 [-, %xmm0]    v2 = imul v0, v1 ; bin: 66 0f 38 40 c1
@@ -98,7 +98,7 @@ ebb0:
 
 
 function %imul_i16x8() -> b1 {
-ebb0:
+block0:
 [-, %xmm1]    v0 = vconst.i16x8 [-1 0 1 32767 0 0 0 0] ; e.g. 32767 == 0x7f_ff
 [-, %xmm2]    v1 = vconst.i16x8 [2 2 2 2 0 0 0 0]
 [-, %xmm1]    v2 = imul v0, v1 ; bin: 66 0f d5 ca
@@ -121,7 +121,7 @@ ebb0:
 
 
 function %sadd_sat_i8x16() -> b1 {
-ebb0:
+block0:
 [-, %xmm2]    v0 = vconst.i8x16 [127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
 [-, %xmm3]    v1 = vconst.i8x16 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
 
@@ -134,7 +134,7 @@ ebb0:
 
 
 function %uadd_sat_i16x8() -> b1 {
-ebb0:
+block0:
 [-, %xmm2]    v0 = vconst.i16x8 [-1 0 0 0 0 0 0 0]
 [-, %xmm3]    v1 = vconst.i16x8 [-1 1 1 1 1 1 1 1]
 
@@ -147,7 +147,7 @@ ebb0:
 
 
 function %sub_sat_i8x16() -> b1 {
-ebb0:
+block0:
 [-, %xmm2]    v0 = vconst.i8x16 [128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] ; 128 == 0x80 == -128
 [-, %xmm3]    v1 = vconst.i8x16 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
 
@@ -166,7 +166,7 @@ ebb0:
 
 
 function %sub_sat_i16x8() {
-ebb0:
+block0:
 [-, %xmm3]    v0 = vconst.i16x8 [0 0 0 0 0 0 0 0]
 [-, %xmm5]    v1 = vconst.i16x8 [1 1 1 1 1 1 1 1]
 [-, %xmm3]    v2 = ssub_sat v0, v1 ; bin: 66 0f e9 dd
@@ -175,7 +175,7 @@ ebb0:
 }
 
 function %float_arithmetic_f32x4(f32x4, f32x4) {
-ebb0(v0: f32x4 [%xmm3], v1: f32x4 [%xmm5]):
+block0(v0: f32x4 [%xmm3], v1: f32x4 [%xmm5]):
 [-, %xmm3]    v2 = fadd v0, v1      ; bin: 40 0f 58 dd
 [-, %xmm3]    v3 = fsub v0, v1      ; bin: 40 0f 5c dd
 [-, %xmm3]    v4 = fmul v0, v1      ; bin: 40 0f 59 dd
@@ -187,7 +187,7 @@ ebb0(v0: f32x4 [%xmm3], v1: f32x4 [%xmm5]):
 }
 
 function %float_arithmetic_f64x2(f64x2, f64x2) {
-ebb0(v0: f64x2 [%xmm3], v1: f64x2 [%xmm5]):
+block0(v0: f64x2 [%xmm3], v1: f64x2 [%xmm5]):
 [-, %xmm3]    v2 = fadd v0, v1      ; bin: 66 40 0f 58 dd
 [-, %xmm3]    v3 = fsub v0, v1      ; bin: 66 40 0f 5c dd
 [-, %xmm3]    v4 = fmul v0, v1      ; bin: 66 40 0f 59 dd
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
index b6b033833e..5211e1d796 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-legalize.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64 skylake
 
 function %ineg_i32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i32x4 [1 1 1 1]
     v2 = ineg v0
     ; check: v5 = vconst.i32x4 0x00
@@ -16,7 +16,7 @@ ebb0:
 }
 
 function %ineg_legalized() {
-ebb0:
+block0:
     v0 = vconst.i8x16 0x00
     v1 = ineg v0
     ; check: v6 = vconst.i8x16 0x00
@@ -36,7 +36,7 @@ ebb0:
 }
 
 function %fneg_legalized() {
-ebb0:
+block0:
     v0 = vconst.f32x4 [0x1.0 0x2.0 0x3.0 0x4.0]
     v1 = fneg v0
     ; check: v4 = vconst.i32x4 0xffffffffffffffffffffffffffffffff
@@ -55,7 +55,7 @@ ebb0:
 }
 
 function %fabs_legalized() {
-ebb0:
+block0:
     v0 = vconst.f64x2 [0x1.0 -0x2.0]
     v1 = fabs v0
     ; check: v2 = vconst.i64x2 0xffffffffffffffffffffffffffffffff
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
index 429928b213..971f5c9bdb 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64 skylake
 
 function %iadd_i32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i32x4 [1 1 1 1]
     v1 = vconst.i32x4 [1 2 3 4]
     v2 = iadd v0, v1
@@ -21,7 +21,7 @@ ebb0:
 ; run
 
 function %iadd_i8x16_with_overflow() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i8x16 [255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255]
     v1 = vconst.i8x16 [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2]
     v2 = iadd v0, v1
@@ -35,7 +35,7 @@ ebb0:
 ; run
 
 function %isub_i32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i32x4 [1 1 1 1]
     v1 = vconst.i32x4 [1 2 3 4]
     v2 = isub v0, v1
@@ -54,7 +54,7 @@ ebb0:
 
 
 function %ineg_i32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i32x4 [1 1 1 1]
     v2 = ineg v0
 
@@ -66,7 +66,7 @@ ebb0:
 ; run
 
 function %imul_i32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i32x4 [-1 0 1 -2147483647] ; e.g. -2147483647 == 0x80_00_00_01
     v1 = vconst.i32x4 [2 2 2 2]
     v2 = imul v0, v1
@@ -87,7 +87,7 @@ ebb0:
 ; run
 
 function %imul_i16x8() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i16x8 [-1 0 1 32767 0 0 0 0] ; e.g. 32767 == 0x7f_ff
     v1 = vconst.i16x8 [2 2 2 2 0 0 0 0]
     v2 = imul v0, v1
@@ -110,7 +110,7 @@ ebb0:
 ; run
 
 function %sadd_sat_i8x16() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i8x16 [127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
     v1 = vconst.i8x16 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
 
@@ -123,7 +123,7 @@ ebb0:
 ; run
 
 function %uadd_sat_i16x8() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i16x8 [-1 0 0 0 0 0 0 0]
     v1 = vconst.i16x8 [-1 1 1 1 1 1 1 1]
 
@@ -136,7 +136,7 @@ ebb0:
 ; run
 
 function %sub_sat_i8x16() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i8x16 [128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] ; 128 == 0x80 == -128
     v1 = vconst.i8x16 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
 
@@ -155,7 +155,7 @@ ebb0:
 ; run
 
 function %add_sub_f32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.f32x4 [0x4.2 0.0 0.0 0.0]
     v1 = vconst.f32x4 [0x1.0 0x1.0 0x1.0 0x1.0]
     v2 = vconst.f32x4 [0x5.2 0x1.0 0x1.0 0x1.0]
@@ -173,7 +173,7 @@ ebb0:
 ; run
 
 function %mul_div_f32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.f32x4 [0x4.2 -0x2.1 0x2.0 0.0]
     v1 = vconst.f32x4 [0x3.4 0x6.7 0x8.9 0xa.b]
     v2 = vconst.f32x4 [0xd.68 -0xd.47 0x11.2 0x0.0]
@@ -191,7 +191,7 @@ ebb0:
 ; run
 
 function %sqrt_f64x2() -> b1 {
-ebb0:
+block0:
     v0 = vconst.f64x2 [0x9.0 0x1.0]
     v1 = sqrt v0
     v2 = vconst.f64x2 [0x3.0 0x1.0]
@@ -202,7 +202,7 @@ ebb0:
 ; run
 
 function %fmax_f64x2() -> b1 {
-ebb0:
+block0:
     v0 = vconst.f64x2 [-0.0 -0x1.0]
     v1 = vconst.f64x2 [+0.0 +0x1.0]
 
@@ -215,7 +215,7 @@ ebb0:
 ; run
 
 function %fmin_f64x2() -> b1 {
-ebb0:
+block0:
     v0 = vconst.f64x2 [-0x1.0 -0x1.0]
     v1 = vconst.f64x2 [+0.0 +0x1.0]
 
@@ -228,7 +228,7 @@ ebb0:
 ; run
 
 function %fneg_f64x2() -> b1 {
-ebb0:
+block0:
     v0 = vconst.f64x2 [0x1.0 -0x1.0]
     v1 = fneg v0
 
@@ -241,7 +241,7 @@ ebb0:
 ; run
 
 function %fneg_f32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.f32x4 [0x0.0 -0x0.0 -Inf Inf]
     v1 = fneg v0
 
@@ -254,7 +254,7 @@ ebb0:
 ; run
 
 function %fabs_f32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.f32x4 [0x0.0 -0x1.0 0x2.0 -0x3.0]
     v1 = fabs v0
 
diff --git a/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif
index af8796863c..599c58fd80 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-bitwise-binemit.clif
@@ -3,103 +3,103 @@ set enable_simd
 target x86_64 skylake
 
 function %ishl_i16x8(i16x8, i64x2) -> i16x8 {
-ebb0(v0: i16x8 [%xmm2], v1: i64x2 [%xmm1]):
+block0(v0: i16x8 [%xmm2], v1: i64x2 [%xmm1]):
 [-, %xmm2]  v2 = x86_psll v0, v1     ; bin: 66 0f f1 d1
             return v2
 }
 
 function %ishl_i32x4(i32x4, i64x2) -> i32x4 {
-ebb0(v0: i32x4 [%xmm4], v1: i64x2 [%xmm0]):
+block0(v0: i32x4 [%xmm4], v1: i64x2 [%xmm0]):
 [-, %xmm4]  v2 = x86_psll v0, v1      ; bin: 66 0f f2 e0
             return v2
 }
 
 function %ishl_i64x2(i64x2, i64x2) -> i64x2 {
-ebb0(v0: i64x2 [%xmm6], v1: i64x2 [%xmm3]):
+block0(v0: i64x2 [%xmm6], v1: i64x2 [%xmm3]):
 [-, %xmm6]  v2 = x86_psll v0, v1      ; bin: 66 0f f3 f3
             return v2
 }
 
 function %ushr_i16x8(i16x8, i64x2) -> i16x8 {
-ebb0(v0: i16x8 [%xmm2], v1: i64x2 [%xmm1]):
+block0(v0: i16x8 [%xmm2], v1: i64x2 [%xmm1]):
 [-, %xmm2]  v2 = x86_psrl v0, v1     ; bin: 66 0f d1 d1
             return v2
 }
 
 function %ushr_i32x4(i32x4, i64x2) -> i32x4 {
-ebb0(v0: i32x4 [%xmm4], v1: i64x2 [%xmm0]):
+block0(v0: i32x4 [%xmm4], v1: i64x2 [%xmm0]):
 [-, %xmm4]  v2 = x86_psrl v0, v1      ; bin: 66 0f d2 e0
             return v2
 }
 
 function %ushr_i64x2(i64x2, i64x2) -> i64x2 {
-ebb0(v0: i64x2 [%xmm6], v1: i64x2 [%xmm3]):
+block0(v0: i64x2 [%xmm6], v1: i64x2 [%xmm3]):
 [-, %xmm6]  v2 = x86_psrl v0, v1      ; bin: 66 0f d3 f3
             return v2
 }
 
 function %sshr_i16x8(i16x8, i64x2) -> i16x8 {
-ebb0(v0: i16x8 [%xmm2], v1: i64x2 [%xmm1]):
+block0(v0: i16x8 [%xmm2], v1: i64x2 [%xmm1]):
 [-, %xmm2]  v2 = x86_psra v0, v1     ; bin: 66 0f e1 d1
             return v2
 }
 
 function %sshr_i32x4(i32x4, i64x2) -> i32x4 {
-ebb0(v0: i32x4 [%xmm4], v1: i64x2 [%xmm0]):
+block0(v0: i32x4 [%xmm4], v1: i64x2 [%xmm0]):
 [-, %xmm4]  v2 = x86_psra v0, v1      ; bin: 66 0f e2 e0
             return v2
 }
 
 function %ishl_imm_i16x8(i16x8) -> i16x8 {
-ebb0(v0: i16x8 [%xmm2]):
+block0(v0: i16x8 [%xmm2]):
 [-, %xmm2]  v2 = ishl_imm v0, 3      ; bin: 66 0f 71 f2 03
             return v2
 }
 
 function %ishl_imm_i32x4(i32x4) -> i32x4 {
-ebb0(v0: i32x4 [%xmm4]):
+block0(v0: i32x4 [%xmm4]):
 [-, %xmm4]  v2 = ishl_imm v0, 10     ; bin: 66 0f 72 f4 0a
             return v2
 }
 
 function %ishl_imm_i64x2(i64x2) -> i64x2 {
-ebb0(v0: i64x2 [%xmm6]):
+block0(v0: i64x2 [%xmm6]):
 [-, %xmm6]  v2 = ishl_imm v0, 42     ; bin: 66 0f 73 f6 2a
             return v2
 }
 
 function %ushr_imm_i16x8(i16x8) -> i16x8 {
-ebb0(v0: i16x8 [%xmm2]):
+block0(v0: i16x8 [%xmm2]):
 [-, %xmm2]  v2 = ushr_imm v0, 3      ; bin: 66 0f 71 d2 03
             return v2
 }
 
 function %ushr_imm_i32x4(i32x4) -> i32x4 {
-ebb0(v0: i32x4 [%xmm4]):
+block0(v0: i32x4 [%xmm4]):
 [-, %xmm4]  v2 = ushr_imm v0, 10     ; bin: 66 0f 72 d4 0a
             return v2
 }
 
 function %ushr_imm_i64x2(i64x2) -> i64x2 {
-ebb0(v0: i64x2 [%xmm6]):
+block0(v0: i64x2 [%xmm6]):
 [-, %xmm6]  v2 = ushr_imm v0, 42     ; bin: 66 0f 73 d6 2a
             return v2
 }
 
 function %sshr_imm_i16x8(i16x8) -> i16x8 {
-ebb0(v0: i16x8 [%xmm2]):
+block0(v0: i16x8 [%xmm2]):
 [-, %xmm2]  v2 = sshr_imm v0, 3      ; bin: 66 0f 71 e2 03
             return v2
 }
 
 function %sshr_imm_i32x4(i32x4) -> i32x4 {
-ebb0(v0: i32x4 [%xmm4]):
+block0(v0: i32x4 [%xmm4]):
 [-, %xmm4]  v2 = sshr_imm v0, 10     ; bin: 66 0f 72 e4 0a
             return v2
 }
 
 function %sshr_imm_i64x2(i64x2) -> i64x2 {
-ebb0(v0: i64x2 [%xmm6]):
+block0(v0: i64x2 [%xmm6]):
 [-, %xmm6]  v2 = sshr_imm v0, 42     ; bin: 66 0f 73 e6 2a
             return v2
 }
diff --git a/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif
index e8391c8a73..af7036b27a 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-bitwise-legalize.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64 skylake
 
 function %ishl_i32x4() -> i32x4 {
-ebb0:
+block0:
     v0 = iconst.i32 1
     v1 = vconst.i32x4 [1 2 4 8]
     v2 = ishl v1, v0
@@ -13,7 +13,7 @@ ebb0:
 }
 
 function %ushr_i64x2() -> i64x2 {
-ebb0:
+block0:
     v0 = iconst.i32 1
     v1 = vconst.i64x2 [1 2]
     v2 = ushr v1, v0
@@ -23,7 +23,7 @@ ebb0:
 }
 
 function %sshr_i16x8() -> i16x8 {
-ebb0:
+block0:
     v0 = iconst.i32 1
     v1 = vconst.i16x8 [1 2 4 8 16 32 64 128]
     v2 = sshr v1, v0
@@ -33,7 +33,7 @@ ebb0:
 }
 
 function %bitselect_i16x8() -> i16x8 {
-ebb0:
+block0:
     v0 = vconst.i16x8 [0 0 0 0 0 0 0 0]
     v1 = vconst.i16x8 [0 0 0 0 0 0 0 0]
     v2 = vconst.i16x8 [0 0 0 0 0 0 0 0]
diff --git a/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif b/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
index 8ca92a756f..670c501c9b 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-bitwise-run.clif
@@ -5,7 +5,7 @@ target x86_64 skylake
 ; TODO: once available, replace all lane extraction with `icmp + all_ones`
 
 function %ishl_i32x4() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i32 1
     v1 = vconst.i32x4 [1 2 4 8]
     v2 = ishl v1, v0
@@ -22,7 +22,7 @@ ebb0:
 ; run
 
 function %ishl_too_large_i16x8() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i32 17 ; note that this will shift off the end of each lane
     v1 = vconst.i16x8 [1 2 4 8 16 32 64 128]
     v2 = ishl v1, v0
@@ -39,7 +39,7 @@ ebb0:
 ; run
 
 function %ushr_i64x2() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i32 1
     v1 = vconst.i64x2 [1 2]
     v2 = ushr v1, v0
@@ -56,7 +56,7 @@ ebb0:
 ; run
 
 function %ushr_too_large_i32x4() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i32 33 ; note that this will shift off the end of each lane
     v1 = vconst.i32x4 [1 2 4 8]
     v2 = ushr v1, v0
@@ -73,7 +73,7 @@ ebb0:
 ; run
 
 function %sshr_i16x8() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i32 1
     v1 = vconst.i16x8 [-1 2 4 8 -16 32 64 128]
     v2 = sshr v1, v0
@@ -90,7 +90,7 @@ ebb0:
 ; run
 
 function %sshr_too_large_i32x4() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i32 33 ; note that this will shift off the end of each lane
     v1 = vconst.i32x4 [1 2 4 -8]
     v2 = sshr v1, v0
@@ -107,7 +107,7 @@ ebb0:
 ; run
 
 function %bitselect_i8x16() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i8x16 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255]  ; the selector vector
     v1 = vconst.i8x16 [127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 42] ; for each 1-bit in v0 the bit of v1 is selected
     v2 = vconst.i8x16 [42 0 0 0 0 0 0 0 0 0 0 0 0 0 0 127] ; for each 0-bit in v0 the bit of v2 is selected
@@ -129,7 +129,7 @@ ebb0:
 ; run
 
 function %sshr_imm_i32x4() -> b1 {
-ebb0:
+block0:
     v1 = vconst.i32x4 [1 2 4 -8]
     v2 = sshr_imm v1, 1
 
@@ -141,7 +141,7 @@ ebb0:
 ; run
 
 function %sshr_imm_i16x8() -> b1 {
-ebb0:
+block0:
     v1 = vconst.i16x8 [1 2 4 -8 0 0 0 0]
     v2 = ushr_imm v1, 1
 
@@ -153,7 +153,7 @@ ebb0:
 ; run
 
 function %ishl_imm_i64x2() -> b1 {
-ebb0:
+block0:
     v1 = vconst.i64x2 [1 0]
     v2 = ishl_imm v1, 1
 
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
index 722e705a85..053b50a9f3 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-binemit.clif
@@ -3,31 +3,31 @@ set enable_simd
 target x86_64 skylake
 
 function %icmp_sgt_i8x16(i8x16, i8x16) -> b8x16 {
-ebb0(v0: i8x16 [%xmm2], v1: i8x16 [%xmm1]):
+block0(v0: i8x16 [%xmm2], v1: i8x16 [%xmm1]):
 [-, %xmm2]  v2 = icmp sgt v0, v1     ; bin: 66 0f 64 d1
             return v2
 }
 
 function %icmp_sgt_i16x8(i16x8, i16x8) -> b16x8 {
-ebb0(v0: i16x8 [%xmm4], v1: i16x8 [%xmm3]):
+block0(v0: i16x8 [%xmm4], v1: i16x8 [%xmm3]):
 [-, %xmm4]  v2 = icmp sgt v0, v1     ; bin: 66 0f 65 e3
             return v2
 }
 
 function %icmp_sgt_i32x4(i32x4, i32x4) -> b32x4 {
-ebb0(v0: i32x4 [%xmm6], v1: i32x4 [%xmm5]):
+block0(v0: i32x4 [%xmm6], v1: i32x4 [%xmm5]):
 [-, %xmm6]  v2 = icmp sgt v0, v1     ; bin: 66 0f 66 f5
             return v2
 }
 
 function %icmp_sgt_i64x2(i64x2, i64x2) -> b64x2 {
-ebb0(v0: i64x2 [%xmm0], v1: i64x2 [%xmm7]):
+block0(v0: i64x2 [%xmm0], v1: i64x2 [%xmm7]):
 [-, %xmm0]  v2 = icmp sgt v0, v1     ; bin: 66 0f 38 37 c7
             return v2
 }
 
 function %min_max_i8x16(i8x16, i8x16) {
-ebb0(v0: i8x16 [%xmm3], v1: i8x16 [%xmm1]):
+block0(v0: i8x16 [%xmm3], v1: i8x16 [%xmm1]):
 [-, %xmm3]  v2 = x86_pmaxs v0, v1     ; bin: 66 0f 38 3c d9
 [-, %xmm3]  v3 = x86_pmaxu v0, v1     ; bin: 66 0f de d9
 [-, %xmm3]  v4 = x86_pmins v0, v1     ; bin: 66 0f 38 38 d9
@@ -36,7 +36,7 @@ ebb0(v0: i8x16 [%xmm3], v1: i8x16 [%xmm1]):
 }
 
 function %min_max_i16x8(i16x8, i16x8) {
-ebb0(v0: i16x8 [%xmm2], v1: i16x8 [%xmm5]):
+block0(v0: i16x8 [%xmm2], v1: i16x8 [%xmm5]):
 [-, %xmm2]  v2 = x86_pmaxs v0, v1     ; bin: 66 0f ee d5
 [-, %xmm2]  v3 = x86_pmaxu v0, v1     ; bin: 66 0f 38 3e d5
 [-, %xmm2]  v4 = x86_pmins v0, v1     ; bin: 66 0f ea d5
@@ -45,7 +45,7 @@ ebb0(v0: i16x8 [%xmm2], v1: i16x8 [%xmm5]):
 }
 
 function %min_max_i32x4(i32x4, i32x4) {
-ebb0(v0: i32x4 [%xmm2], v1: i32x4 [%xmm4]):
+block0(v0: i32x4 [%xmm2], v1: i32x4 [%xmm4]):
 [-, %xmm2]  v2 = x86_pmaxs v0, v1     ; bin: 66 0f 38 3d d4
 [-, %xmm2]  v3 = x86_pmaxu v0, v1     ; bin: 66 0f 38 3f d4
 [-, %xmm2]  v4 = x86_pmins v0, v1     ; bin: 66 0f 38 39 d4
@@ -54,7 +54,7 @@ ebb0(v0: i32x4 [%xmm2], v1: i32x4 [%xmm4]):
 }
 
 function %fcmp_f32x4(f32x4, f32x4) {
-ebb0(v0: f32x4 [%xmm2], v1: f32x4 [%xmm4]):
+block0(v0: f32x4 [%xmm2], v1: f32x4 [%xmm4]):
 [-, %xmm2]  v2 = fcmp eq v0, v1     ; bin: 40 0f c2 d4 00
 [-, %xmm2]  v3 = fcmp lt v0, v1     ; bin: 40 0f c2 d4 01
 [-, %xmm2]  v4 = fcmp le v0, v1     ; bin: 40 0f c2 d4 02
@@ -67,7 +67,7 @@ ebb0(v0: f32x4 [%xmm2], v1: f32x4 [%xmm4]):
 }
 
 function %fcmp_f64x2(f64x2, f64x2) {
-ebb0(v0: f64x2 [%xmm2], v1: f64x2 [%xmm0]):
+block0(v0: f64x2 [%xmm2], v1: f64x2 [%xmm0]):
 [-, %xmm2]  v2 = fcmp eq v0, v1     ; bin: 66 40 0f c2 d0 00
 [-, %xmm2]  v3 = fcmp lt v0, v1     ; bin: 66 40 0f c2 d0 01
 [-, %xmm2]  v4 = fcmp le v0, v1     ; bin: 66 40 0f c2 d0 02
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif
index acbff943eb..61888ccb6d 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-legalize.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64 skylake
 
 function %icmp_ne_32x4(i32x4, i32x4) -> b32x4 {
-ebb0(v0: i32x4, v1: i32x4):
+block0(v0: i32x4, v1: i32x4):
     v2 = icmp ne v0, v1
     ; check: v3 = icmp eq v0, v1
     ; nextln: v4 = vconst.b32x4 0xffffffffffffffffffffffffffffffff
@@ -12,7 +12,7 @@ ebb0(v0: i32x4, v1: i32x4):
 }
 
 function %icmp_ugt_i32x4(i32x4, i32x4) -> b32x4 {
-ebb0(v0: i32x4, v1: i32x4):
+block0(v0: i32x4, v1: i32x4):
     v2 = icmp ugt v0, v1
     ; check: v3 = x86_pmaxu v0, v1
     ; nextln: v4 = icmp eq v3, v1
@@ -22,7 +22,7 @@ ebb0(v0: i32x4, v1: i32x4):
 }
 
 function %icmp_sge_i16x8(i16x8, i16x8) -> b16x8 {
-ebb0(v0: i16x8, v1: i16x8):
+block0(v0: i16x8, v1: i16x8):
     v2 = icmp sge v0, v1
     ; check: v3 = x86_pmins v0, v1
     ; nextln: v2 = icmp eq v3, v1
@@ -30,7 +30,7 @@ ebb0(v0: i16x8, v1: i16x8):
 }
 
 function %icmp_uge_i8x16(i8x16, i8x16) -> b8x16 {
-ebb0(v0: i8x16, v1: i8x16):
+block0(v0: i8x16, v1: i8x16):
     v2 = icmp uge v0, v1
     ; check: v3 = x86_pminu v0, v1
     ; nextln: v2 = icmp eq v3, v1
diff --git a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
index 444d4e28bd..7cbd285860 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-comparison-run.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64 skylake
 
 function %icmp_eq_i8x16() -> b8 {
-ebb0:
+block0:
     v0 = vconst.i8x16 0x00
     v1 = vconst.i8x16 0x00
     v2 = icmp eq v0, v1
@@ -13,7 +13,7 @@ ebb0:
 ; run
 
 function %icmp_eq_i64x2() -> b64 {
-ebb0:
+block0:
     v0 = vconst.i64x2 0xffffffffffffffffffffffffffffffff
     v1 = vconst.i64x2 0xffffffffffffffffffffffffffffffff
     v2 = icmp eq v0, v1
@@ -23,7 +23,7 @@ ebb0:
 ; run
 
 function %icmp_ne_i32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i32x4 [0 1 2 3]
     v1 = vconst.i32x4 [7 7 7 7]
     v2 = icmp ne v0, v1
@@ -33,7 +33,7 @@ ebb0:
 ; run
 
 function %icmp_ne_i16x8() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i16x8 [0 1 2 3 4 5 6 7]
     v1 = vconst.i16x8 [0 1 2 3 4 5 6 7]
     v2 = icmp ne v0, v1
@@ -45,7 +45,7 @@ ebb0:
 ; run
 
 function %icmp_sgt_i8x16() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i8x16 [0 1 2 0 0 0 0 0 0 0 0 0 0 0 0 0]
     v1 = vconst.i8x16 [1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0xff]
     v2 = icmp sgt v0, v1
@@ -58,7 +58,7 @@ ebb0:
 ; run
 
 function %icmp_sgt_i64x2() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i64x2 [0 -42]
     v1 = vconst.i64x2 [-1 -43]
     v2 = icmp sgt v0, v1
@@ -68,7 +68,7 @@ ebb0:
 ; run
 
 function %maxs_i8x16() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i8x16 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1] ; 1 will be greater than -1 == 0xff with
     ; signed max
     v1 = vconst.i8x16 [0xff 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
@@ -79,7 +79,7 @@ ebb0:
 ; run
 
 function %maxu_i16x8() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i16x8 [0 1 1 1 1 1 1 1]
     v1 = vconst.i16x8 [-1 1 1 1 1 1 1 1] ; -1 == 0xff will be greater with unsigned max
     v2 = x86_pmaxu v0, v1
@@ -89,7 +89,7 @@ ebb0:
 ; run
 
 function %mins_i32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i32x4 [0 1 1 1]
     v1 = vconst.i32x4 [-1 1 1 1] ; -1 == 0xff will be less with signed min
     v2 = x86_pmins v0, v1
@@ -99,7 +99,7 @@ ebb0:
 ; run
 
 function %minu_i8x16() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i8x16 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1] ; 1 < 2 with unsiged min
     v1 = vconst.i8x16 [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2]
     v2 = x86_pminu v0, v1
@@ -109,7 +109,7 @@ ebb0:
 ; run
 
 function %icmp_ugt_i8x16() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i8x16 [1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16]
     v1 = vconst.i8x16 [0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
     v2 = icmp ugt v0, v1
@@ -119,7 +119,7 @@ ebb0:
 ; run
 
 function %icmp_sge_i16x8() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i16x8 [-1 1 2 3 4 5 6 7]
     v1 = vconst.i16x8 [-1 1 1 1 1 1 1 1]
     v2 = icmp sge v0, v1
@@ -129,7 +129,7 @@ ebb0:
 ; run
 
 function %icmp_uge_i32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i32x4 [1 2 3 4]
     v1 = vconst.i32x4 [1 1 1 1]
     v2 = icmp uge v0, v1
@@ -139,7 +139,7 @@ ebb0:
 ; run
 
 function %icmp_slt_i32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i32x4 [-1 1 1 1]
     v1 = vconst.i32x4 [1 2 3 4]
     v2 = icmp slt v0, v1
@@ -149,7 +149,7 @@ ebb0:
 ; run
 
 function %icmp_ult_i32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i32x4 [1 1 1 1]
     v1 = vconst.i32x4 [-1 2 3 4] ; -1 = 0xffff... will be greater than 1 when unsigned
     v2 = icmp ult v0, v1
@@ -160,7 +160,7 @@ ebb0:
 
 
 function %icmp_ult_i16x8() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i16x8 [-1 -1 -1 -1 -1 -1 -1 -1]
     v1 = vconst.i16x8 [-1 -1 -1 -1 -1 -1 -1 -1]
     v2 = icmp ult v0, v1
@@ -173,7 +173,7 @@ ebb0:
 ; run
 
 function %icmp_sle_i16x8() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i16x8 [-1 -1 0 0 0 0 0 0]
     v1 = vconst.i16x8 [-1  0 0 0 0 0 0 0]
     v2 = icmp sle v0, v1
@@ -183,7 +183,7 @@ ebb0:
 ; run
 
 function %icmp_ule_i16x8() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i16x8 [-1  0 0 0 0 0 0 0]
     v1 = vconst.i16x8 [-1 -1 0 0 0 0 0 0]
     v2 = icmp ule v0, v1
@@ -193,7 +193,7 @@ ebb0:
 ; run
 
 function %fcmp_eq_f32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.f32x4 [0.0 -0x4.2 0x0.33333 -0.0]
     v1 = vconst.f32x4 [0.0 -0x4.2 0x0.33333 -0.0]
     v2 = fcmp eq v0, v1
@@ -203,7 +203,7 @@ ebb0:
 ; run
 
 function %fcmp_lt_f32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.f32x4 [0.0      -0x4.2  0x0.0       -0.0]
     v1 = vconst.f32x4 [0x0.001  0x4.2   0x0.33333   0x1.0]
     v2 = fcmp lt v0, v1
@@ -213,7 +213,7 @@ ebb0:
 ; run
 
 function %fcmp_ge_f64x2() -> b1 {
-ebb0:
+block0:
     v0 = vconst.f64x2 [0x0.0  0x4.2]
     v1 = vconst.f64x2 [0.0    0x4.1]
     v2 = fcmp ge v0, v1
@@ -223,7 +223,7 @@ ebb0:
 ; run
 
 function %fcmp_uno_f64x2() -> b1 {
-ebb0:
+block0:
     v0 = vconst.f64x2 [0.0  NaN]
     v1 = vconst.f64x2 [NaN  0x4.1]
     v2 = fcmp uno v0, v1
@@ -233,7 +233,7 @@ ebb0:
 ; run
 
 function %fcmp_gt_nans_f32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.f32x4 [NaN 0x42.0 -NaN NaN]
     v1 = vconst.f32x4 [NaN NaN 0x42.0 Inf]
     v2 = fcmp gt v0, v1
diff --git a/cranelift/filetests/filetests/isa/x86/simd-construction-run.clif b/cranelift/filetests/filetests/isa/x86/simd-construction-run.clif
index eecd82e024..ef2aeea26d 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-construction-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-construction-run.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64 skylake
 
 function %splat_i64x2() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i64 -1
     v1 = splat.i64x2 v0
     v2 = vconst.i64x2 [-1 -1]
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
index 6d6a3fac31..2f7c4f5b22 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-binemit.clif
@@ -3,31 +3,31 @@ set enable_simd
 target x86_64 skylake
 
 function %bor_b16x8(b16x8, b16x8) -> b16x8 {
-ebb0(v0: b16x8 [%xmm2], v1: b16x8 [%xmm1]):
+block0(v0: b16x8 [%xmm2], v1: b16x8 [%xmm1]):
 [-, %xmm2]  v2 = bor v0, v1     ; bin: 66 0f eb d1
             return v2
 }
 
 function %band_b64x2(b64x2, b64x2) -> b64x2 {
-ebb0(v0: b64x2 [%xmm6], v1: b64x2 [%xmm3]):
+block0(v0: b64x2 [%xmm6], v1: b64x2 [%xmm3]):
 [-, %xmm6]  v2 = band v0, v1      ; bin: 66 0f db f3
             return v2
 }
 
 function %bxor_b32x4(b32x4, b32x4) -> b32x4 {
-ebb0(v0: b32x4 [%xmm4], v1: b32x4 [%xmm0]):
+block0(v0: b32x4 [%xmm4], v1: b32x4 [%xmm0]):
 [-, %xmm4]  v2 = bxor v0, v1      ; bin: 66 0f ef e0
             return v2
 }
 
 function %band_not_b64x2(b64x2, b64x2) -> b64x2 {
-ebb0(v0: b64x2 [%xmm6], v1: b64x2 [%xmm3]):
+block0(v0: b64x2 [%xmm6], v1: b64x2 [%xmm3]):
 [-, %xmm3]  v2 = band_not v0, v1      ; bin: 66 0f df de
             return v2
 }
 
 function %x86_ptest_f64x2(f64x2, f64x2) {
-ebb0(v0: f64x2 [%xmm0], v1: f64x2 [%xmm2]):
+block0(v0: f64x2 [%xmm0], v1: f64x2 [%xmm2]):
 [-, %rflags]  v2 = x86_ptest v0, v1      ; bin: 66 0f 38 17 c2
             return
 }
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif
index 2e13f79b9b..a1248c8bba 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-legalize.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64 skylake
 
 function %bnot_b32x4(b32x4) -> b32x4 {
-ebb0(v0: b32x4):
+block0(v0: b32x4):
     v1 = bnot v0
     ; check: v2 = vconst.b32x4 0xffffffffffffffffffffffffffffffff
     ; nextln: v1 = bxor v2, v0
@@ -11,7 +11,7 @@ ebb0(v0: b32x4):
 }
 
 function %vany_true_b32x4(b32x4) -> b1 {
-ebb0(v0: b32x4):
+block0(v0: b32x4):
     v1 = vany_true v0
     ; check: v2 = x86_ptest v0, v0
     ; nextln: v1 = trueif ne v2
@@ -19,7 +19,7 @@ ebb0(v0: b32x4):
 }
 
 function %vall_true_i64x2(i64x2) -> b1 {
-ebb0(v0: i64x2):
+block0(v0: i64x2):
     v1 = vall_true v0
     ; check: v2 = vconst.i64x2 0x00
     ; nextln: v3 = icmp eq v0, v2
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-rodata.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-rodata.clif
index 1c5fb89733..619d300bfe 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-logical-rodata.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-rodata.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64 skylake
 
 function %bnot_b32x4(b32x4) -> b32x4 {
-ebb0(v0: b32x4):
+block0(v0: b32x4):
     v1 = bnot v0
     return v1
 }
diff --git a/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif b/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif
index 9b525f2e10..35fc44bc6a 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-logical-run.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64 skylake
 
 function %bnot() -> b32 {
-ebb0:
+block0:
     v0 = vconst.b32x4 [true true true false]
     v1 = bnot v0
     v2 = extractlane v1, 3
@@ -12,7 +12,7 @@ ebb0:
 ; run
 
 function %band_not() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i16x8 [1 0 0 0 0 0 0 0]
     v1 = vconst.i16x8 [0 0 0 0 0 0 0 0]
     v2 = band_not v0, v1
@@ -23,7 +23,7 @@ ebb0:
 ; run
 
 function %vany_true_i16x8() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i16x8 [1 0 0 0 0 0 0 0]
     v1 = vany_true v0
     return v1
@@ -31,7 +31,7 @@ ebb0:
 ; run
 
 function %vany_true_b32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.b32x4 [false false false false]
     v1 = vany_true v0
     v2 = bint.i32 v1
@@ -41,7 +41,7 @@ ebb0:
 ; run
 
 function %vall_true_i16x8() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i16x8 [1 0 0 0 0 0 0 0]
     v1 = vall_true v0
     v2 = bint.i32 v1
@@ -51,7 +51,7 @@ ebb0:
 ; run
 
 function %vall_true_b32x4() -> b1 {
-ebb0:
+block0:
     v0 = vconst.b32x4 [true true true true]
     v1 = vall_true v0
     return v1
diff --git a/cranelift/filetests/filetests/isa/x86/stack-addr64.clif b/cranelift/filetests/filetests/isa/x86/stack-addr64.clif
index c80d190907..bcb441cd6b 100644
--- a/cranelift/filetests/filetests/isa/x86/stack-addr64.clif
+++ b/cranelift/filetests/filetests/isa/x86/stack-addr64.clif
@@ -16,7 +16,7 @@ function %stack_addr() {
            ss4 = explicit_slot 8, offset 0
            ss5 = explicit_slot 8, offset 1024
 
-ebb0:
+block0:
 [-,%rcx]   v0 = stack_addr.i64 ss0                      ; bin: 48 8d 8c 24 00000808
 [-,%rcx]   v1 = stack_addr.i64 ss1                      ; bin: 48 8d 8c 24 00000408
 [-,%rcx]   v2 = stack_addr.i64 ss2                      ; bin: 48 8d 8c 24 00000008
diff --git a/cranelift/filetests/filetests/isa/x86/stack-load-store64.clif b/cranelift/filetests/filetests/isa/x86/stack-load-store64.clif
index 3c0e2c8c0e..a74a1dfc32 100644
--- a/cranelift/filetests/filetests/isa/x86/stack-load-store64.clif
+++ b/cranelift/filetests/filetests/isa/x86/stack-load-store64.clif
@@ -6,7 +6,7 @@ target x86_64 haswell
 function %stack_load_and_store() {
            ss0 = explicit_slot 8, offset 0
 
-ebb0:
+block0:
    v0 = stack_load.i64 ss0
 
 ; check: v1 = stack_addr.i64 ss0
diff --git a/cranelift/filetests/filetests/isa/x86/stack-load-store8.clif b/cranelift/filetests/filetests/isa/x86/stack-load-store8.clif
index 2c368f6dfc..2c5bb1553b 100644
--- a/cranelift/filetests/filetests/isa/x86/stack-load-store8.clif
+++ b/cranelift/filetests/filetests/isa/x86/stack-load-store8.clif
@@ -4,7 +4,7 @@ target x86_64
 function u0:0(i8) -> i8 {
     ss0 = explicit_slot 1
 
-ebb0(v0: i8):
+block0(v0: i8):
     stack_store v0, ss0
     ; check: v2 = stack_addr.i64 ss0
     ; nextln: v3 = uextend.i32 v0
diff --git a/cranelift/filetests/filetests/isa/x86/uextend-i8-to-i16.clif b/cranelift/filetests/filetests/isa/x86/uextend-i8-to-i16.clif
index d92da90343..7d778aa778 100644
--- a/cranelift/filetests/filetests/isa/x86/uextend-i8-to-i16.clif
+++ b/cranelift/filetests/filetests/isa/x86/uextend-i8-to-i16.clif
@@ -2,13 +2,13 @@ test compile
 target x86_64
 
 function u0:0(i8) -> i16 fast {
-ebb0(v0: i8):
+block0(v0: i8):
     v1 = uextend.i16 v0
     return v1
 }
 
 function u0:1(i8) -> i16 fast {
-ebb0(v0: i8):
+block0(v0: i8):
     v1 = sextend.i16 v0
     return v1
 }
diff --git a/cranelift/filetests/filetests/isa/x86/vconst-binemit.clif b/cranelift/filetests/filetests/isa/x86/vconst-binemit.clif
index 2d6f862679..15522e3d38 100644
--- a/cranelift/filetests/filetests/isa/x86/vconst-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/vconst-binemit.clif
@@ -4,7 +4,7 @@ set enable_simd
 target x86_64
 
 function %test_vconst_b8() {
-ebb0:
+block0:
 [-, %xmm2]  v0 = vconst.b8x16 0x01  ; bin: 0f 10 15 00000008 PCRelRodata4(15)
 [-, %xmm3]  v1 = vconst.b8x16 0x02  ; bin: 0f 10 1d 00000011 PCRelRodata4(31)
             return
diff --git a/cranelift/filetests/filetests/isa/x86/vconst-opt-run.clif b/cranelift/filetests/filetests/isa/x86/vconst-opt-run.clif
index 487ff4f844..9294614c96 100644
--- a/cranelift/filetests/filetests/isa/x86/vconst-opt-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/vconst-opt-run.clif
@@ -5,7 +5,7 @@ target x86_64
 ; TODO move to vconst-run.clif
 
 function %test_vconst_zeroes() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i8x16 0x00
     v1 = extractlane v0, 4
     v2 = icmp_imm eq v1, 0
@@ -14,7 +14,7 @@ ebb0:
 ; run
 
 function %test_vconst_ones() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i8x16 0xffffffffffffffffffffffffffffffff
     v1 = extractlane v0, 2
     v2 = icmp_imm eq v1, 0xff
diff --git a/cranelift/filetests/filetests/isa/x86/vconst-opt.clif b/cranelift/filetests/filetests/isa/x86/vconst-opt.clif
index 4daeed8abe..bc444b7784 100644
--- a/cranelift/filetests/filetests/isa/x86/vconst-opt.clif
+++ b/cranelift/filetests/filetests/isa/x86/vconst-opt.clif
@@ -5,7 +5,7 @@ target x86_64
 ; TODO move to vconst-compile.clif or vconst-binemit.clif
 
 function %test_vconst_optimizations() {
-ebb0:
+block0:
 [-, %xmm4]  v0 = vconst.b8x16 0x00                                  ; bin: 66 0f ef e4
 [-, %xmm7]  v1 = vconst.b8x16 0xffffffffffffffffffffffffffffffff    ; bin: 66 0f 74 ff
             return
diff --git a/cranelift/filetests/filetests/isa/x86/vconst-rodata.clif b/cranelift/filetests/filetests/isa/x86/vconst-rodata.clif
index 34c203dce6..0df8493d5d 100644
--- a/cranelift/filetests/filetests/isa/x86/vconst-rodata.clif
+++ b/cranelift/filetests/filetests/isa/x86/vconst-rodata.clif
@@ -3,7 +3,7 @@ set enable_simd=true
 target x86_64 haswell
 
 function %test_vconst_i32() -> i32x4 {
-ebb0:
+block0:
     v0 = vconst.i32x4 0x1234
     return v0
 }
@@ -11,7 +11,7 @@ ebb0:
 ; sameln: [34, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
 function %test_vconst_b16() -> b16x8 {
-ebb0:
+block0:
     v0 = vconst.b16x8 [true false true false true false true true]
     return v0
 }
diff --git a/cranelift/filetests/filetests/isa/x86/vconst-run.clif b/cranelift/filetests/filetests/isa/x86/vconst-run.clif
index 9ec160c2e5..60d94fbccd 100644
--- a/cranelift/filetests/filetests/isa/x86/vconst-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/vconst-run.clif
@@ -2,7 +2,7 @@ test run
 set enable_simd
 
 function %test_vconst_syntax() -> b1 {
-ebb0:
+block0:
     v0 = vconst.i32x4 0x00000004_00000003_00000002_00000001     ; build constant using hexadecimal syntax
     v1 = vconst.i32x4 [1 2 3 4]                                 ; build constant using literal list syntax
 
diff --git a/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
index a621abfe9f..55a6c59bed 100644
--- a/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
+++ b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64.clif
@@ -5,7 +5,7 @@ target x86_64 haswell
 
 ; check if for one arg we use the right register
 function %one_arg(i64) windows_fastcall {
-ebb0(v0: i64):
+block0(v0: i64):
     return
 }
 ; check: function %one_arg(i64 [%rcx], i64 fp [%rbp]) -> i64 fp [%rbp] windows_fastcall {
@@ -13,33 +13,33 @@ ebb0(v0: i64):
 
 ; check if we still use registers for 4 arguments
 function %four_args(i64, i64, i64, i64) windows_fastcall {
-ebb0(v0: i64, v1: i64, v2: i64, v3: i64):
+block0(v0: i64, v1: i64, v2: i64, v3: i64):
     return
 }
 ; check: function %four_args(i64 [%rcx], i64 [%rdx], i64 [%r8], i64 [%r9], i64 fp [%rbp]) -> i64 fp [%rbp] windows_fastcall {
 
 ; check if float arguments are passed through XMM registers
 function %four_float_args(f64, f64, f64, f64) windows_fastcall {
-ebb0(v0: f64, v1: f64, v2: f64, v3: f64):
+block0(v0: f64, v1: f64, v2: f64, v3: f64):
     return
 }
 ; check: function %four_float_args(f64 [%xmm0], f64 [%xmm1], f64 [%xmm2], f64 [%xmm3], i64 fp [%rbp]) -> i64 fp [%rbp] windows_fastcall {
 
 ; check if we use stack space for > 4 arguments
 function %five_args(i64, i64, i64, i64, i64) windows_fastcall {
-ebb0(v0: i64, v1: i64, v2: i64, v3: i64, v4: i64):
+block0(v0: i64, v1: i64, v2: i64, v3: i64, v4: i64):
     return
 }
 ; check: function %five_args(i64 [%rcx], i64 [%rdx], i64 [%r8], i64 [%r9], i64 [32], i64 fp [%rbp]) -> i64 fp [%rbp] windows_fastcall {
 
 function %mixed_int_float(i64, f64, i64, f32) windows_fastcall {
-ebb0(v0: i64, v1: f64, v2: i64, v3: f32):
+block0(v0: i64, v1: f64, v2: i64, v3: f32):
     return
 }
 ; check: function %mixed_int_float(i64 [%rcx], f64 [%xmm1], i64 [%r8], f32 [%xmm3], i64 fp [%rbp]) -> i64 fp [%rbp] windows_fastcall {
 
 function %ret_val_float(f32, f64, i64, i64) -> f64 windows_fastcall {
-ebb0(v0: f32, v1: f64, v2: i64, v3: i64):
+block0(v0: f32, v1: f64, v2: i64, v3: i64):
     return v1
 }
 ; check: function %ret_val_float(f32 [%xmm0], f64 [%xmm1], i64 [%r8], i64 [%r9], i64 fp [%rbp]) -> f64 [%xmm0], i64 fp [%rbp] windows_fastcall {
diff --git a/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64_unwind.clif b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64_unwind.clif
index 7dc024f33c..b146f0ac76 100644
--- a/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64_unwind.clif
+++ b/cranelift/filetests/filetests/isa/x86/windows_fastcall_x64_unwind.clif
@@ -5,14 +5,14 @@ target x86_64 haswell
 
 ; check that there is no unwind information for a system_v function
 function %not_fastcall() system_v {
-ebb0:
+block0:
     return
 }
 ; sameln: No unwind information.
 
 ; check the unwind information with a function with no args
 function %no_args() windows_fastcall {
-ebb0:
+block0:
     return
 }
 ; sameln: UnwindInfo {
@@ -47,7 +47,7 @@ ebb0:
 ; check a function with medium-sized stack alloc
 function %medium_stack() windows_fastcall {
     ss0 = explicit_slot 100000
-ebb0:
+block0:
     return
 }
 ; sameln: UnwindInfo {
@@ -84,7 +84,7 @@ ebb0:
 ; check a function with large-sized stack alloc
 function %large_stack() windows_fastcall {
     ss0 = explicit_slot 524288
-ebb0:
+block0:
     return
 }
 ; sameln: UnwindInfo {
@@ -120,7 +120,7 @@ ebb0:
 
 ; check a function that has CSRs
 function %lots_of_registers(i64, i64) windows_fastcall {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = load.i32 v0+0
     v3 = load.i32 v0+8
     v4 = load.i32 v0+16
diff --git a/cranelift/filetests/filetests/isa/x86/windows_systemv_x64_fde.clif b/cranelift/filetests/filetests/isa/x86/windows_systemv_x64_fde.clif
index 1444359de7..31b75b6c16 100644
--- a/cranelift/filetests/filetests/isa/x86/windows_systemv_x64_fde.clif
+++ b/cranelift/filetests/filetests/isa/x86/windows_systemv_x64_fde.clif
@@ -5,14 +5,14 @@ target x86_64 haswell
 
 ; check that there is no libunwind information for a windows_fastcall function
 function %not_fastcall() windows_fastcall {
-ebb0:
+block0:
     return
 }
 ; sameln: No unwind information.
 
 ; check the libunwind information with a function with no args
 function %no_args() system_v {
-ebb0:
+block0:
     return
 }
 ; sameln: 0x00000000: CIE
diff --git a/cranelift/filetests/filetests/legalizer/bitrev-i128.clif b/cranelift/filetests/filetests/legalizer/bitrev-i128.clif
index 5a539d0e89..fad0f2aace 100644
--- a/cranelift/filetests/filetests/legalizer/bitrev-i128.clif
+++ b/cranelift/filetests/filetests/legalizer/bitrev-i128.clif
@@ -2,12 +2,12 @@ test legalizer
 target x86_64
 
 function %reverse_bits(i128) -> i128 {
-ebb0(v0: i128):
+block0(v0: i128):
     v1 = bitrev.i128 v0
     return v1
 }
 
-; check: ebb0(v2: i64, v3: i64):
+; check: block0(v2: i64, v3: i64):
 ; check: v0 = iconcat v2, v3
 ; check: v33 = iconst.i64 0xaaaa_aaaa_aaaa_aaaa
 ; check: v6 = band v2, v33
diff --git a/cranelift/filetests/filetests/legalizer/bitrev.clif b/cranelift/filetests/filetests/legalizer/bitrev.clif
index b7ee07735e..5651d7a7f3 100644
--- a/cranelift/filetests/filetests/legalizer/bitrev.clif
+++ b/cranelift/filetests/filetests/legalizer/bitrev.clif
@@ -2,7 +2,7 @@ test legalizer
 target x86_64
 
 function %reverse_bits_8(i8) -> i8 {
-ebb0(v0: i8):
+block0(v0: i8):
     v1 = bitrev.i8 v0
     return v1
 }
@@ -57,7 +57,7 @@ ebb0(v0: i8):
 ; check: return v1
 
 function %reverse_bits_16(i16) -> i16 {
-ebb0(v0: i16):
+block0(v0: i16):
     v1 = bitrev.i16 v0
     return v1
 }
@@ -128,7 +128,7 @@ ebb0(v0: i16):
 ; check: return v1
 
 function %reverse_bits_32(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = bitrev.i32 v0
     return v1
 }
@@ -162,7 +162,7 @@ ebb0(v0: i32):
 
 
 function %reverse_bits_64(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = bitrev.i64 v0
     return v1
 }
diff --git a/cranelift/filetests/filetests/legalizer/br_table_cond.clif b/cranelift/filetests/filetests/legalizer/br_table_cond.clif
index dc0bd6473c..9677e2c9f3 100644
--- a/cranelift/filetests/filetests/legalizer/br_table_cond.clif
+++ b/cranelift/filetests/filetests/legalizer/br_table_cond.clif
@@ -5,60 +5,60 @@ target x86_64
 
 ; Test that when jump_tables_enables is false, all jump tables are eliminated.
 ; regex: V=v\d+
-; regex: EBB=ebb\d+
+; regex: BB=block\d+
 
 function u0:0(i64 vmctx) baldrdash_system_v {
     gv0 = vmctx
     gv1 = iadd_imm.i64 gv0, 48
-    jt0 = jump_table [ebb2, ebb2, ebb7]
-    jt1 = jump_table [ebb8, ebb8]
+    jt0 = jump_table [block2, block2, block7]
+    jt1 = jump_table [block8, block8]
 
-ebb0(v0: i64):
-    jump ebb5
+block0(v0: i64):
+    jump block5
 
-ebb5:
+block5:
     v1 = global_value.i64 gv1
     v2 = load.i64 v1
     trapnz v2, interrupt
     v3 = iconst.i32 0
-    br_table v3, ebb3, jt0
-; check:  ebb5:
+    br_table v3, block3, jt0
+; check:  block5:
 ; check:    $(val0=$V) = iconst.i32 0
 ; nextln:   $(cmp0=$V) = icmp_imm eq $val0, 0
-; nextln:   brnz $cmp0, ebb2
-; nextln:   jump $(fail0=$EBB)
+; nextln:   brnz $cmp0, block2
+; nextln:   jump $(fail0=$BB)
 ; check:  $fail0:
 ; nextln:   $(cmp1=$V) = icmp_imm.i32 eq $val0, 1
-; nextln:   brnz $cmp1, ebb2
-; nextln:   jump $(fail1=$EBB)
+; nextln:   brnz $cmp1, block2
+; nextln:   jump $(fail1=$BB)
 ; check:  $fail1:
 ; nextln:   $(cmp2=$V) = icmp_imm.i32 eq $val0, 2
-; nextln:   brnz $cmp2, ebb7
-; nextln:   jump ebb3
+; nextln:   brnz $cmp2, block7
+; nextln:   jump block3
 
-ebb7:
+block7:
     v4 = iconst.i32 0
-    br_table v4, ebb3, jt1
-; check:  ebb7:
+    br_table v4, block3, jt1
+; check:  block7:
 ; check:    $(val1=$V) = iconst.i32 0
 ; nextln:   $(cmp3=$V) = icmp_imm eq $val1, 0
-; nextln:   brnz $cmp3, ebb8
-; nextln:   jump $(fail3=$EBB)
+; nextln:   brnz $cmp3, block8
+; nextln:   jump $(fail3=$BB)
 ; check:  $fail3:
 ; nextln:   $(cmp4=$V) = icmp_imm.i32 eq $val1, 1
-; nextln:   brnz $cmp4, ebb8
-; nextln:   jump ebb3
+; nextln:   brnz $cmp4, block8
+; nextln:   jump block3
 
-ebb8:
-    jump ebb5
+block8:
+    jump block5
 
-ebb3:
-    jump ebb2
+block3:
+    jump block2
 
-ebb2:
-    jump ebb1
+block2:
+    jump block1
 
-ebb1:
+block1:
     fallthrough_return
 }
 ; not: jump_table
diff --git a/cranelift/filetests/filetests/legalizer/bxor_imm.clif b/cranelift/filetests/filetests/legalizer/bxor_imm.clif
index 19372613ff..bf959a7364 100644
--- a/cranelift/filetests/filetests/legalizer/bxor_imm.clif
+++ b/cranelift/filetests/filetests/legalizer/bxor_imm.clif
@@ -2,7 +2,7 @@ test legalizer
 target x86_64
 
 function %foo(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = bxor_imm.i64 v0, 0x100000000
     return v2
 }
diff --git a/cranelift/filetests/filetests/legalizer/empty_br_table.clif b/cranelift/filetests/filetests/legalizer/empty_br_table.clif
index b043733fba..606a07f605 100644
--- a/cranelift/filetests/filetests/legalizer/empty_br_table.clif
+++ b/cranelift/filetests/filetests/legalizer/empty_br_table.clif
@@ -6,12 +6,12 @@ target x86_64
 function u0:0(i64) {
     jt0 = jump_table []
 
-ebb0(v0: i64):
-    br_table v0, ebb1, jt0
-; check:  ebb0(v0: i64):
-; nextln:   jump ebb1
+block0(v0: i64):
+    br_table v0, block1, jt0
+; check:  block0(v0: i64):
+; nextln:   jump block1
 
-ebb1:
+block1:
     return
 }
 ; not: jump_table
diff --git a/cranelift/filetests/filetests/legalizer/iconst-i64.clif b/cranelift/filetests/filetests/legalizer/iconst-i64.clif
index a3c9168416..6aa7361b45 100644
--- a/cranelift/filetests/filetests/legalizer/iconst-i64.clif
+++ b/cranelift/filetests/filetests/legalizer/iconst-i64.clif
@@ -2,7 +2,7 @@ test legalizer
 target i686
 
 function %foo() -> i64 {
-ebb0:
+block0:
     v1 = iconst.i64 0x6400000042
     return v1
 }
diff --git a/cranelift/filetests/filetests/legalizer/isplit-bb.clif b/cranelift/filetests/filetests/legalizer/isplit-bb.clif
index 10ab41c440..7e55eb1eb9 100644
--- a/cranelift/filetests/filetests/legalizer/isplit-bb.clif
+++ b/cranelift/filetests/filetests/legalizer/isplit-bb.clif
@@ -2,23 +2,23 @@ test legalizer
 target x86_64
 
 function u0:0(i128, i128, i64) -> i128 system_v {
-ebb0(v0: i128, v1: i128, v2: i64):
-    jump ebb1
+block0(v0: i128, v1: i128, v2: i64):
+    jump block1
 
-ebb1:
+block1:
     v17 = iadd v0, v1
     v20 = iadd v1, v17
-    jump ebb79
+    jump block79
 
-ebb79:
+block79:
     v425 = iconst.i64 0
     v426 = icmp_imm eq v425, 1
-    brnz v426, ebb80
-    jump ebb85(v20, v17)
+    brnz v426, block80
+    jump block85(v20, v17)
 
-ebb80:
+block80:
     trap user0
 
-ebb85(v462: i128, v874: i128):
+block85(v462: i128, v874: i128):
     trap user0
 }
diff --git a/cranelift/filetests/filetests/legalizer/popcnt-i128.clif b/cranelift/filetests/filetests/legalizer/popcnt-i128.clif
index 0ecf7f74c5..f4919f4781 100644
--- a/cranelift/filetests/filetests/legalizer/popcnt-i128.clif
+++ b/cranelift/filetests/filetests/legalizer/popcnt-i128.clif
@@ -2,7 +2,7 @@ test legalizer
 target i686
 
 function %foo() -> i128 {
-ebb0:
+block0:
     v1 = iconst.i64 0x6400000042
     v2 = iconst.i64 0x7F10100042
     v3 = iconcat v1, v2
diff --git a/cranelift/filetests/filetests/licm/basic.clif b/cranelift/filetests/filetests/licm/basic.clif
index 3f5dfbbe14..b089d0b182 100644
--- a/cranelift/filetests/filetests/licm/basic.clif
+++ b/cranelift/filetests/filetests/licm/basic.clif
@@ -3,39 +3,39 @@ target riscv32
 
 function %simple_loop(i32) -> i32 {
 
-ebb0(v0: i32):
-    jump ebb1(v0)
+block0(v0: i32):
+    jump block1(v0)
 
-ebb1(v1: i32):
+block1(v1: i32):
     v2 = iconst.i32 1
     v3 = iconst.i32 2
     v4 = iadd v2, v3
-    brz v1, ebb3(v1)
-    jump ebb2
+    brz v1, block3(v1)
+    jump block2
 
-ebb2:
+block2:
     v5 = isub v1, v2
-    jump ebb1(v5)
+    jump block1(v5)
 
-ebb3(v6: i32):
+block3(v6: i32):
     return v6
 
 }
 ; sameln: function %simple_loop
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     v2 = iconst.i32 1
 ; nextln:     v3 = iconst.i32 2
 ; nextln:     v4 = iadd v2, v3
-; nextln:     jump ebb1(v0)
+; nextln:     jump block1(v0)
 ; nextln: 
-; nextln: ebb1(v1: i32):
-; nextln:     brz v1, ebb3(v1)
-; nextln:     jump ebb2
+; nextln: block1(v1: i32):
+; nextln:     brz v1, block3(v1)
+; nextln:     jump block2
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:     v5 = isub.i32 v1, v2
-; nextln:     jump ebb1(v5)
+; nextln:     jump block1(v5)
 ; nextln: 
-; nextln: ebb3(v6: i32):
+; nextln: block3(v6: i32):
 ; nextln:     return v6
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/complex.clif b/cranelift/filetests/filetests/licm/complex.clif
index 2774cde5b3..ab9c905e39 100644
--- a/cranelift/filetests/filetests/licm/complex.clif
+++ b/cranelift/filetests/filetests/licm/complex.clif
@@ -2,95 +2,95 @@ test licm
 target riscv32
 
 function %complex(i32) -> i32 system_v {
-ebb0(v0: i32):
-[UJ#1b]    jump ebb1(v0)
+block0(v0: i32):
+[UJ#1b]    jump block1(v0)
 
-         ebb1(v1: i32):
+         block1(v1: i32):
             v2 = iconst.i32 1
             v3 = iconst.i32 4
             v4 = iadd v2, v1
-[SBzero#18] brz v1, ebb2(v2)
-[UJ#1b]     jump ebb4(v4)
+[SBzero#18] brz v1, block2(v2)
+[UJ#1b]     jump block4(v4)
 
-         ebb2(v5: i32):
+         block2(v5: i32):
             v6 = iconst.i32 2
             v7 = iadd v5, v4
             v8 = iadd v6, v1
-[UJ#1b]   jump ebb3(v8)
+[UJ#1b]   jump block3(v8)
 
-         ebb3(v9: i32):
+         block3(v9: i32):
             v10 = iadd v9, v5
             v11 = iadd.i32 v1, v4
-[SBzero#18] brz.i32 v1, ebb2(v9)
-[UJ#1b]   jump ebb6(v10)
+[SBzero#18] brz.i32 v1, block2(v9)
+[UJ#1b]   jump block6(v10)
 
-         ebb4(v12: i32):
+         block4(v12: i32):
             v13 = iconst.i32 3
             v14 = iadd v12, v13
             v15 = iadd.i32 v4, v13
-[UJ#1b]   jump ebb5(v13)
+[UJ#1b]   jump block5(v13)
 
-         ebb5(v16: i32):
+         block5(v16: i32):
             v17 = iadd.i32 v14, v4
-[SBzero#18] brz.i32 v1, ebb4(v16)
-[UJ#1b]   jump ebb6(v16)
+[SBzero#18] brz.i32 v1, block4(v16)
+[UJ#1b]   jump block6(v16)
 
-         ebb6(v18: i32):
+         block6(v18: i32):
             v19 = iadd v18, v2
             v20 = iadd.i32 v2, v3
-[SBzero#18] brz.i32 v1, ebb1(v20)
-[UJ#1b]     jump ebb7
+[SBzero#18] brz.i32 v1, block1(v20)
+[UJ#1b]     jump block7
 
-         ebb7:
+         block7:
 [Iret#19]   return v19
 }
 
 ; sameln: function %complex
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     v2 = iconst.i32 1
 ; nextln:     v3 = iconst.i32 4
 ; nextln:     v6 = iconst.i32 2
 ; nextln:     v13 = iconst.i32 3
 ; nextln:     v20 = iadd v2, v3
-; nextln:     jump ebb1(v0)
+; nextln:     jump block1(v0)
 ; nextln: 
-; nextln: ebb1(v1: i32):
+; nextln: block1(v1: i32):
 ; nextln:     v4 = iadd.i32 v2, v1
-; nextln:     brz v1, ebb8(v2)
-; nextln:     jump ebb9(v4)
+; nextln:     brz v1, block8(v2)
+; nextln:     jump block9(v4)
 ; nextln: 
-; nextln: ebb8(v21: i32):
+; nextln: block8(v21: i32):
 ; nextln:     v8 = iadd.i32 v6, v1
 ; nextln:     v11 = iadd.i32 v1, v4
-; nextln:     jump ebb2(v21)
+; nextln:     jump block2(v21)
 ; nextln: 
-; nextln: ebb2(v5: i32):
+; nextln: block2(v5: i32):
 ; nextln:     v7 = iadd v5, v4
-; nextln:     jump ebb3(v8)
+; nextln:     jump block3(v8)
 ; nextln: 
-; nextln: ebb3(v9: i32):
+; nextln: block3(v9: i32):
 ; nextln:     v10 = iadd v9, v5
-; nextln:     brz.i32 v1, ebb2(v9)
-; nextln:     jump ebb6(v10)
+; nextln:     brz.i32 v1, block2(v9)
+; nextln:     jump block6(v10)
 ; nextln: 
-; nextln: ebb9(v22: i32):
+; nextln: block9(v22: i32):
 ; nextln:     v15 = iadd.i32 v4, v13
-; nextln:     jump ebb4(v22)
+; nextln:     jump block4(v22)
 ; nextln: 
-; nextln: ebb4(v12: i32):
+; nextln: block4(v12: i32):
 ; nextln:     v14 = iadd v12, v13
-; nextln:     jump ebb5(v13)
+; nextln:     jump block5(v13)
 ; nextln: 
-; nextln: ebb5(v16: i32):
+; nextln: block5(v16: i32):
 ; nextln:     v17 = iadd.i32 v14, v4
-; nextln:     brz.i32 v1, ebb4(v16)
-; nextln:     jump ebb6(v16)
+; nextln:     brz.i32 v1, block4(v16)
+; nextln:     jump block6(v16)
 ; nextln: 
-; nextln: ebb6(v18: i32):
+; nextln: block6(v18: i32):
 ; nextln:     v19 = iadd v18, v2
-; nextln:     brz.i32 v1, ebb1(v20)
-; nextln:     jump ebb7
+; nextln:     brz.i32 v1, block1(v20)
+; nextln:     jump block7
 ; nextln: 
-; nextln: ebb7:
+; nextln: block7:
 ; nextln:     return v19
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/critical-edge.clif b/cranelift/filetests/filetests/licm/critical-edge.clif
index 89beb387cc..1940a4ed36 100644
--- a/cranelift/filetests/filetests/licm/critical-edge.clif
+++ b/cranelift/filetests/filetests/licm/critical-edge.clif
@@ -5,50 +5,50 @@ target riscv32
 
 function %critical_edge(i32, i32) -> i32 {
 
-            ebb0(v0: i32, v7: i32):
-[SBzero#38]   brnz v7, ebb2(v0)
-[UJ#1b]       jump ebb1
+            block0(v0: i32, v7: i32):
+[SBzero#38]   brnz v7, block2(v0)
+[UJ#1b]       jump block1
 
-            ebb1:
+            block1:
 [Iret#19]     return v0
 
-            ebb2(v1: i32):
+            block2(v1: i32):
               v2 = iconst.i32 1
               v3 = iconst.i32 2
               v4 = iadd v2, v3
-[SBzero#18]   brz v1, ebb4(v1)
-[UJ#1b]       jump ebb3
+[SBzero#18]   brz v1, block4(v1)
+[UJ#1b]       jump block3
 
-            ebb3:
+            block3:
               v5 = isub v1, v2
-[UJ#1b]       jump ebb2(v5)
+[UJ#1b]       jump block2(v5)
 
-            ebb4(v6: i32):
+            block4(v6: i32):
 [Iret#19]     return v6
 
 }
 ; sameln: function %critical_edge
-; nextln: ebb0(v0: i32, v7: i32):
-; nextln:     brnz v7, ebb5(v0)
-; nextln:     jump ebb1
+; nextln: block0(v0: i32, v7: i32):
+; nextln:     brnz v7, block5(v0)
+; nextln:     jump block1
 ; nextln: 
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:     return v0
 ; nextln: 
-; nextln: ebb5(v8: i32):
+; nextln: block5(v8: i32):
 ; nextln:     v2 = iconst.i32 1
 ; nextln:     v3 = iconst.i32 2
 ; nextln:     v4 = iadd v2, v3
-; nextln:     jump ebb2(v8)
+; nextln:     jump block2(v8)
 ; nextln: 
-; nextln: ebb2(v1: i32):
-; nextln:     brz v1, ebb4(v1)
-; nextln:     jump ebb3
+; nextln: block2(v1: i32):
+; nextln:     brz v1, block4(v1)
+; nextln:     jump block3
 ; nextln: 
-; nextln: ebb3:
+; nextln: block3:
 ; nextln:     v5 = isub.i32 v1, v2
-; nextln:     jump ebb2(v5)
+; nextln:     jump block2(v5)
 ; nextln: 
-; nextln: ebb4(v6: i32):
+; nextln: block4(v6: i32):
 ; nextln:     return v6
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/encoding.clif b/cranelift/filetests/filetests/licm/encoding.clif
index b029a51c18..2b0114d2d0 100644
--- a/cranelift/filetests/filetests/licm/encoding.clif
+++ b/cranelift/filetests/filetests/licm/encoding.clif
@@ -4,39 +4,39 @@ target riscv32
 ; Ensure that instructions emitted by LICM get encodings.
 
 function %simple_loop(i32) -> i32 {
-            ebb0(v0: i32):
-[UJ#1b]       jump ebb1(v0)
+            block0(v0: i32):
+[UJ#1b]       jump block1(v0)
 
-            ebb1(v1: i32):
+            block1(v1: i32):
 [Iz#04,%x0]   v2 = iconst.i32 1
 [Iz#04,%x1]   v3 = iconst.i32 2
 [R#0c,%x2]    v4 = iadd v2, v3
-[SBzero#18]   brz v1, ebb3(v1)
-[UJ#1b]       jump ebb2
+[SBzero#18]   brz v1, block3(v1)
+[UJ#1b]       jump block2
 
-            ebb2:
+            block2:
 [R#200c,%x5]  v5 = isub v1, v2
-[UJ#1b]       jump ebb1(v5)
+[UJ#1b]       jump block1(v5)
 
-            ebb3(v6: i32):
+            block3(v6: i32):
 [Iret#19]     return v6
 }
 
 ; check: function
-; nextln:                                 ebb0(v0: i32):
+; nextln:                                 block0(v0: i32):
 ; nextln: [Iz#04,%x0]                         v2 = iconst.i32 1
 ; nextln: [Iz#04,%x1]                         v3 = iconst.i32 2
 ; nextln: [R#0c,%x2]                          v4 = iadd v2, v3
-; nextln: [UJ#1b]                             jump ebb1(v0)
+; nextln: [UJ#1b]                             jump block1(v0)
 ; nextln: 
-; nextln:                                 ebb1(v1: i32):
-; nextln: [SBzero#18]                         brz v1, ebb3(v1)
-; nextln: [UJ#1b]                             jump ebb2
+; nextln:                                 block1(v1: i32):
+; nextln: [SBzero#18]                         brz v1, block3(v1)
+; nextln: [UJ#1b]                             jump block2
 ; nextln: 
-; nextln:                                 ebb2:
+; nextln:                                 block2:
 ; nextln: [R#200c,%x5]                        v5 = isub.i32 v1, v2
-; nextln: [UJ#1b]                             jump ebb1(v5)
+; nextln: [UJ#1b]                             jump block1(v5)
 ; nextln: 
-; nextln:                                 ebb3(v6: i32):
+; nextln:                                 block3(v6: i32):
 ; nextln: [Iret#19]                           return v6
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/jump-table-entry.clif b/cranelift/filetests/filetests/licm/jump-table-entry.clif
index cbf51cd080..6f754185a5 100644
--- a/cranelift/filetests/filetests/licm/jump-table-entry.clif
+++ b/cranelift/filetests/filetests/licm/jump-table-entry.clif
@@ -2,32 +2,32 @@ test licm
 target x86_64
 
 function %dont_hoist_jump_table_entry_during_licm() {
-    jt0 = jump_table [ebb1, ebb1]
+    jt0 = jump_table [block1, block1]
 
-ebb0:
-    fallthrough ebb1
+block0:
+    fallthrough block1
 
-ebb1: ; the loop!
+block1: ; the loop!
     v2 = iconst.i32 42
     v3 = ifcmp_imm v2, 0
-    brif uge v3, ebb1
-    fallthrough ebb2
+    brif uge v3, block1
+    fallthrough block2
 
-ebb2:
+block2:
     v1 = iconst.i64 -14
     v8 = ifcmp_imm v1, 2
-    brif uge v8, ebb1
-    jump ebb3
+    brif uge v8, block1
+    jump block3
 
-ebb3:
+block3:
     v5 = jump_table_base.i64 jt0
     v6 = jump_table_entry.i64 v1, v5, 4, jt0
     v7 = iadd v5, v6
     indirect_jump_table_br v7, jt0
-; check: ebb2:
+; check: block2:
 ; nextln: v8 = ifcmp_imm.i64 v1, 2
-; nextln: brif uge v8, ebb1
-; nextln: jump ebb3
-; check: ebb3:
+; nextln: brif uge v8, block1
+; nextln: jump block3
+; check: block3:
 ; nextln: jump_table_entry.i64
 }
diff --git a/cranelift/filetests/filetests/licm/load_readonly_notrap.clif b/cranelift/filetests/filetests/licm/load_readonly_notrap.clif
index 4731bd664e..f663646b9e 100644
--- a/cranelift/filetests/filetests/licm/load_readonly_notrap.clif
+++ b/cranelift/filetests/filetests/licm/load_readonly_notrap.clif
@@ -10,22 +10,22 @@ function %hoist_load(i32, i64 vmctx) -> i32 {
     gv1 = load.i64 notrap aligned readonly gv0
     heap0 = static gv1, min 0x1_0000, bound 0x1_0000_0000, offset_guard 0x8000_0000, index_type i32
 
-ebb0(v0: i32, v1: i64):
-    jump ebb1(v0, v1)
+block0(v0: i32, v1: i64):
+    jump block1(v0, v1)
 
-ebb1(v2: i32, v3: i64):
+block1(v2: i32, v3: i64):
     v4 = iconst.i32 1
     v5 = heap_addr.i64 heap0, v4, 1
     v6 = load.i32 notrap aligned readonly v5
     v7 = iadd v2, v6
-    brz v2, ebb3(v2)
-    jump ebb2
+    brz v2, block3(v2)
+    jump block2
 
-ebb2:
+block2:
     v8 = isub v2, v4
-    jump ebb1(v8, v3)
+    jump block1(v8, v3)
 
-ebb3(v9: i32):
+block3(v9: i32):
     return v9
 }
 
@@ -34,21 +34,21 @@ ebb3(v9: i32):
 ; nextln:    gv1 = load.i64 notrap aligned readonly gv0
 ; nextln:    heap0 = static gv1, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000, index_type i32
 ; nextln: 
-; nextln: ebb0(v0: i32, v1: i64):
+; nextln: block0(v0: i32, v1: i64):
 ; nextln:    v4 = iconst.i32 1
 ; nextln:    v5 = heap_addr.i64 heap0, v4, 1
 ; nextln:    v6 = load.i32 notrap aligned readonly v5
-; nextln:    jump ebb1(v0, v1)
+; nextln:    jump block1(v0, v1)
 ; nextln: 
-; nextln: ebb1(v2: i32, v3: i64):
+; nextln: block1(v2: i32, v3: i64):
 ; nextln:    v7 = iadd v2, v6
-; nextln:    brz v2, ebb3(v2)
-; nextln:     jump ebb2
+; nextln:    brz v2, block3(v2)
+; nextln:     jump block2
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:    v8 = isub.i32 v2, v4
-; nextln:    jump ebb1(v8, v3)
+; nextln:    jump block1(v8, v3)
 ; nextln: 
-; nextln: ebb3(v9: i32):
+; nextln: block3(v9: i32):
 ; nextln:    return v9
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/multiple-blocks.clif b/cranelift/filetests/filetests/licm/multiple-blocks.clif
index ea23505ef5..04cfb9d621 100644
--- a/cranelift/filetests/filetests/licm/multiple-blocks.clif
+++ b/cranelift/filetests/filetests/licm/multiple-blocks.clif
@@ -3,57 +3,57 @@ target riscv32
 
 function %multiple_blocks(i32) -> i32 {
 
-ebb0(v0: i32):
-    jump ebb1(v0)
+block0(v0: i32):
+    jump block1(v0)
 
-ebb1(v10: i32):
+block1(v10: i32):
     v11 = iconst.i32 1
     v12 = iconst.i32 2
     v13 = iadd v11, v12
-    brz v10, ebb4(v10)
-    jump ebb2
+    brz v10, block4(v10)
+    jump block2
 
-ebb2:
+block2:
     v15 = isub v10, v11
-    brz v15, ebb5(v15)
-    jump ebb3
+    brz v15, block5(v15)
+    jump block3
 
-ebb3:
+block3:
     v14 = isub v10, v11
-    jump ebb1(v14)
+    jump block1(v14)
 
-ebb4(v20: i32):
+block4(v20: i32):
     return v20
 
-ebb5(v30: i32):
+block5(v30: i32):
     v31 = iadd v11, v13
-    jump ebb1(v30)
+    jump block1(v30)
 
 }
 ; sameln:function %multiple_blocks(i32) -> i32 {
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     v11 = iconst.i32 1
 ; nextln:     v12 = iconst.i32 2
 ; nextln:     v13 = iadd v11, v12
 ; nextln:     v31 = iadd v11, v13
-; nextln:     jump ebb1(v0)
+; nextln:     jump block1(v0)
 ; nextln: 
-; nextln: ebb1(v10: i32):
-; nextln:     brz v10, ebb4(v10)
-; nextln:     jump ebb2
+; nextln: block1(v10: i32):
+; nextln:     brz v10, block4(v10)
+; nextln:     jump block2
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:     v15 = isub.i32 v10, v11
-; nextln:     brz v15, ebb5(v15)
-; nextln:     jump ebb3
+; nextln:     brz v15, block5(v15)
+; nextln:     jump block3
 ; nextln: 
-; nextln: ebb3:
+; nextln: block3:
 ; nextln:     v14 = isub.i32 v10, v11
-; nextln:     jump ebb1(v14)
+; nextln:     jump block1(v14)
 ; nextln: 
-; nextln: ebb4(v20: i32):
+; nextln: block4(v20: i32):
 ; nextln:     return v20
 ; nextln: 
-; nextln: ebb5(v30: i32):
-; nextln:     jump ebb1(v30)
+; nextln: block5(v30: i32):
+; nextln:     jump block1(v30)
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/nested_loops.clif b/cranelift/filetests/filetests/licm/nested_loops.clif
index 423b24d33f..7f9cb928db 100644
--- a/cranelift/filetests/filetests/licm/nested_loops.clif
+++ b/cranelift/filetests/filetests/licm/nested_loops.clif
@@ -3,60 +3,60 @@ target riscv32
 
 function %nested_loops(i32) -> i32 {
 
-ebb0(v0: i32):
-    jump ebb1(v0)
+block0(v0: i32):
+    jump block1(v0)
 
-ebb1(v1: i32):
+block1(v1: i32):
     v2 = iconst.i32 1
     v3 = iconst.i32 2
     v4 = iadd v2, v3
     v5 = isub v1, v2
-    jump ebb2(v5, v5)
+    jump block2(v5, v5)
 
-ebb2(v10: i32, v11: i32):
-    brz v11, ebb4(v10)
-    jump ebb3
+block2(v10: i32, v11: i32):
+    brz v11, block4(v10)
+    jump block3
 
-ebb3:
+block3:
     v12 = iconst.i32 1
     v15 = iadd v12, v5
     v13 = isub v11, v12
-    jump ebb2(v10,v13)
+    jump block2(v10,v13)
 
-ebb4(v20: i32):
-    brz v20, ebb5(v20)
-    jump ebb1(v20)
+block4(v20: i32):
+    brz v20, block5(v20)
+    jump block1(v20)
 
-ebb5(v30: i32):
+block5(v30: i32):
     return v30
 
 }
 
 ; sameln:function %nested_loops(i32) -> i32 {
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     v2 = iconst.i32 1
 ; nextln:     v3 = iconst.i32 2
 ; nextln:     v4 = iadd v2, v3
 ; nextln:     v12 = iconst.i32 1
-; nextln:     jump ebb1(v0)
+; nextln:     jump block1(v0)
 ; nextln: 
-; nextln: ebb1(v1: i32):
+; nextln: block1(v1: i32):
 ; nextln:     v5 = isub v1, v2
 ; nextln:     v15 = iadd.i32 v12, v5
-; nextln:     jump ebb2(v5, v5)
+; nextln:     jump block2(v5, v5)
 ; nextln: 
-; nextln: ebb2(v10: i32, v11: i32):
-; nextln:     brz v11, ebb4(v10)
-; nextln:     jump ebb3
+; nextln: block2(v10: i32, v11: i32):
+; nextln:     brz v11, block4(v10)
+; nextln:     jump block3
 ; nextln: 
-; nextln: ebb3:
+; nextln: block3:
 ; nextln:     v13 = isub.i32 v11, v12
-; nextln:     jump ebb2(v10, v13)
+; nextln:     jump block2(v10, v13)
 ; nextln: 
-; nextln: ebb4(v20: i32):
-; nextln:     brz v20, ebb5(v20)
-; nextln:     jump ebb1(v20)
+; nextln: block4(v20: i32):
+; nextln:     brz v20, block5(v20)
+; nextln:     jump block1(v20)
 ; nextln: 
-; nextln: ebb5(v30: i32):
+; nextln: block5(v30: i32):
 ; nextln:     return v30
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/reject.clif b/cranelift/filetests/filetests/licm/reject.clif
index 43823c1295..eab03760b6 100644
--- a/cranelift/filetests/filetests/licm/reject.clif
+++ b/cranelift/filetests/filetests/licm/reject.clif
@@ -3,92 +3,92 @@ target riscv32
 
 function %other_side_effects(i32) -> i32 {
 
-ebb0(v0: i32):
-    jump ebb1(v0)
+block0(v0: i32):
+    jump block1(v0)
 
-ebb1(v1: i32):
+block1(v1: i32):
     regmove.i32 v0, %x10 -> %x20
-; check: ebb1(v1: i32):
+; check: block1(v1: i32):
 ; check: regmove.i32 v0, %x10 -> %x20
     v2 = iconst.i32 1
-    brz v1, ebb3(v1)
-    jump ebb2
+    brz v1, block3(v1)
+    jump block2
 
-ebb2:
+block2:
     v5 = isub v1, v2
-    jump ebb1(v5)
+    jump block1(v5)
 
-ebb3(v6: i32):
+block3(v6: i32):
     return v6
 
 }
 
 function %cpu_flags(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
-    jump ebb1(v0, v1)
+block0(v0: i32, v1: i32):
+    jump block1(v0, v1)
 
-ebb1(v2: i32, v3: i32):
+block1(v2: i32, v3: i32):
     v4 = ifcmp.i32 v0, v1
     v5 = selectif.i32 eq v4, v2, v3
-; check: ebb1(v2: i32, v3: i32):
+; check: block1(v2: i32, v3: i32):
 ; check: ifcmp.i32 v0, v1
 ; check: v5 = selectif.i32 eq v4, v2, v3
     v8 = iconst.i32 1
-    brz v1, ebb3(v1)
-    jump ebb2
+    brz v1, block3(v1)
+    jump block2
 
-ebb2:
+block2:
     v9 = isub v1, v8
     v10 = iadd v1, v8
-    jump ebb1(v9, v10)
+    jump block1(v9, v10)
 
-ebb3(v6: i32):
+block3(v6: i32):
     return v6
 }
 
 function %spill(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = spill.i32 v0
-    jump ebb1(v0, v1)
+    jump block1(v0, v1)
 
-ebb1(v3: i32, v4: i32):
+block1(v3: i32, v4: i32):
     v5 = spill.i32 v1
     v6 = fill.i32 v2
     v7 = fill.i32 v5
-; check: ebb1(v3: i32, v4: i32):
+; check: block1(v3: i32, v4: i32):
 ; check: v5 = spill.i32 v1
 ; check: v6 = fill.i32 v2
 ; check: v7 = fill v5
-    brz v1, ebb3(v1)
-    jump ebb2
+    brz v1, block3(v1)
+    jump block2
 
-ebb2:
+block2:
     v9 = isub v1, v4
-    jump ebb1(v9, v3)
+    jump block1(v9, v3)
 
-ebb3(v10: i32):
+block3(v10: i32):
     return v10
 }
 
 function %non_invariant_aliases(i32) -> i32 {
 
-ebb0(v0: i32):
-    jump ebb1(v0)
+block0(v0: i32):
+    jump block1(v0)
 
-ebb1(v1: i32):
+block1(v1: i32):
     v8 -> v1
     v9 -> v1
     v2 = iadd v8, v9
-; check: ebb1(v1: i32):
+; check: block1(v1: i32):
 ; check: v2 = iadd v8, v9
-    brz v1, ebb3(v1)
-    jump ebb2
+    brz v1, block3(v1)
+    jump block2
 
-ebb2:
+block2:
     v5 = isub v1, v2
-    jump ebb1(v5)
+    jump block1(v5)
 
-ebb3(v6: i32):
+block3(v6: i32):
     return v6
 
 }
diff --git a/cranelift/filetests/filetests/licm/reject_load_notrap.clif b/cranelift/filetests/filetests/licm/reject_load_notrap.clif
index 71385807e3..58f046357d 100644
--- a/cranelift/filetests/filetests/licm/reject_load_notrap.clif
+++ b/cranelift/filetests/filetests/licm/reject_load_notrap.clif
@@ -11,22 +11,22 @@ function %hoist_load(i32, i64 vmctx) -> i32 {
     gv1 = load.i64 notrap aligned readonly gv0
     heap0 = static gv1, min 0x1_0000, bound 0x1_0000_0000, offset_guard 0x8000_0000, index_type i32
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v4 = iconst.i32 1
     v5 = heap_addr.i64 heap0, v4, 1
-    jump ebb1(v0, v1)
+    jump block1(v0, v1)
 
-ebb1(v2: i32, v3: i64):
+block1(v2: i32, v3: i64):
     v6 = load.i32 notrap aligned v5
     v7 = iadd v2, v6
-    brz v2, ebb3(v2)
-    jump ebb2
+    brz v2, block3(v2)
+    jump block2
 
-ebb2:
+block2:
     v8 = isub v2, v4
-    jump ebb1(v8, v3)
+    jump block1(v8, v3)
 
-ebb3(v9: i32):
+block3(v9: i32):
     return v9
 }
 
@@ -35,21 +35,21 @@ ebb3(v9: i32):
 ; nextln:    gv1 = load.i64 notrap aligned readonly gv0
 ; nextln:    heap0 = static gv1, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000, index_type i32
 ; nextln: 
-; nextln: ebb0(v0: i32, v1: i64):
+; nextln: block0(v0: i32, v1: i64):
 ; nextln:    v4 = iconst.i32 1
 ; nextln:    v5 = heap_addr.i64 heap0, v4, 1
-; nextln:    jump ebb1(v0, v1)
+; nextln:    jump block1(v0, v1)
 ; nextln: 
-; nextln: ebb1(v2: i32, v3: i64):
+; nextln: block1(v2: i32, v3: i64):
 ; nextln:    v6 = load.i32 notrap aligned v5
 ; nextln:    v7 = iadd v2, v6
-; nextln:    brz v2, ebb3(v2)
-; nextln:    jump ebb2
+; nextln:    brz v2, block3(v2)
+; nextln:    jump block2
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:    v8 = isub.i32 v2, v4
-; nextln:    jump ebb1(v8, v3)
+; nextln:    jump block1(v8, v3)
 ; nextln: 
-; nextln: ebb3(v9: i32):
+; nextln: block3(v9: i32):
 ; nextln:    return v9
 ; nextln: }
diff --git a/cranelift/filetests/filetests/licm/reject_load_readonly.clif b/cranelift/filetests/filetests/licm/reject_load_readonly.clif
index ea7b72469e..f794bad6b0 100644
--- a/cranelift/filetests/filetests/licm/reject_load_readonly.clif
+++ b/cranelift/filetests/filetests/licm/reject_load_readonly.clif
@@ -11,22 +11,22 @@ function %hoist_load(i32, i64 vmctx) -> i32 {
     gv1 = load.i64 notrap aligned readonly gv0
     heap0 = static gv1, min 0x1_0000, bound 0x1_0000_0000, offset_guard 0x8000_0000, index_type i32
 
-ebb0(v0: i32, v1: i64):
-    jump ebb1(v0, v1)
+block0(v0: i32, v1: i64):
+    jump block1(v0, v1)
 
-ebb1(v2: i32, v3: i64):
+block1(v2: i32, v3: i64):
     v4 = iconst.i32 1
     v5 = heap_addr.i64 heap0, v4, 1
     v6 = load.i32 aligned readonly v5
     v7 = iadd v2, v6
-    brz v2, ebb3(v2)
-    jump ebb2
+    brz v2, block3(v2)
+    jump block2
 
-ebb2:
+block2:
     v8 = isub v2, v4
-    jump ebb1(v8, v3)
+    jump block1(v8, v3)
 
-ebb3(v9: i32):
+block3(v9: i32):
     return v9
 }
 
@@ -35,21 +35,21 @@ ebb3(v9: i32):
 ; nextln:    gv1 = load.i64 notrap aligned readonly gv0
 ; nextln:    heap0 = static gv1, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000, index_type i32
 ; nextln: 
-; nextln: ebb0(v0: i32, v1: i64):
+; nextln: block0(v0: i32, v1: i64):
 ; nextln:    v4 = iconst.i32 1
 ; nextln:    v5 = heap_addr.i64 heap0, v4, 1
-; nextln:    jump ebb1(v0, v1)
+; nextln:    jump block1(v0, v1)
 ; nextln: 
-; nextln: ebb1(v2: i32, v3: i64):
+; nextln: block1(v2: i32, v3: i64):
 ; nextln:    v6 = load.i32 aligned readonly v5
 ; nextln:    v7 = iadd v2, v6
-; nextln:    brz v2, ebb3(v2)
-; nextln:    jump ebb2
+; nextln:    brz v2, block3(v2)
+; nextln:    jump block2
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:    v8 = isub.i32 v2, v4
-; nextln:    jump ebb1(v8, v3)
+; nextln:    jump block1(v8, v3)
 ; nextln: 
-; nextln: ebb3(v9: i32):
+; nextln: block3(v9: i32):
 ; nextln:    return v9
 ; nextln: }
diff --git a/cranelift/filetests/filetests/parser/alias.clif b/cranelift/filetests/filetests/parser/alias.clif
index 4e253bdb03..6197ae35d1 100644
--- a/cranelift/filetests/filetests/parser/alias.clif
+++ b/cranelift/filetests/filetests/parser/alias.clif
@@ -2,7 +2,7 @@ test cat
 test verifier
 
 function %basic(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 -> v0
     v3 -> v1
     v4 = iadd.i32 v2, v3
@@ -10,7 +10,7 @@ ebb0(v0: i32, v1: i32):
 }
 
 function %transitive() -> i32 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 -> v0
     v2 -> v1
@@ -20,7 +20,7 @@ ebb0:
 }
 
 function %duplicate(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 -> v0
     v2 -> v0
     v2 -> v0
diff --git a/cranelift/filetests/filetests/parser/branch.clif b/cranelift/filetests/filetests/parser/branch.clif
index 4404feba14..c9a71312d9 100644
--- a/cranelift/filetests/filetests/parser/branch.clif
+++ b/cranelift/filetests/filetests/parser/branch.clif
@@ -3,114 +3,114 @@ test cat
 
 ; Jumps with no arguments. The '()' empty argument list is optional.
 function %minimal() {
-ebb0:
-    jump ebb1
+block0:
+    jump block1
 
-ebb1:
-    jump ebb0()
+block1:
+    jump block0()
 }
 ; sameln: function %minimal() fast {
-; nextln: ebb0:
-; nextln:     jump ebb1
+; nextln: block0:
+; nextln:     jump block1
 ; nextln: 
-; nextln: ebb1:
-; nextln:     jump ebb0
+; nextln: block1:
+; nextln:     jump block0
 ; nextln: }
 
 ; Jumps with 1 arg.
 function %onearg(i32) {
-ebb0(v90: i32):
-    jump ebb1(v90)
+block0(v90: i32):
+    jump block1(v90)
 
-ebb1(v91: i32):
-    jump ebb0(v91)
+block1(v91: i32):
+    jump block0(v91)
 }
 ; sameln: function %onearg(i32) fast {
-; nextln: ebb0(v90: i32):
-; nextln:     jump ebb1(v90)
+; nextln: block0(v90: i32):
+; nextln:     jump block1(v90)
 ; nextln: 
-; nextln: ebb1(v91: i32):
-; nextln:     jump ebb0(v91)
+; nextln: block1(v91: i32):
+; nextln:     jump block0(v91)
 ; nextln: }
 
 ; Jumps with 2 args.
 function %twoargs(i32, f32) {
-ebb0(v90: i32, v91: f32):
-    jump ebb1(v90, v91)
+block0(v90: i32, v91: f32):
+    jump block1(v90, v91)
 
-ebb1(v92: i32, v93: f32):
-    jump ebb0(v92, v93)
+block1(v92: i32, v93: f32):
+    jump block0(v92, v93)
 }
 ; sameln: function %twoargs(i32, f32) fast {
-; nextln: ebb0(v90: i32, v91: f32):
-; nextln:     jump ebb1(v90, v91)
+; nextln: block0(v90: i32, v91: f32):
+; nextln:     jump block1(v90, v91)
 ; nextln: 
-; nextln: ebb1(v92: i32, v93: f32):
-; nextln:     jump ebb0(v92, v93)
+; nextln: block1(v92: i32, v93: f32):
+; nextln:     jump block0(v92, v93)
 ; nextln: }
 
 ; Branches with no arguments. The '()' empty argument list is optional.
 function %minimal(i32) {
-ebb0(v90: i32):
-    brz v90, ebb1
+block0(v90: i32):
+    brz v90, block1
 
-ebb1:
-    brnz v90, ebb1()
+block1:
+    brnz v90, block1()
 }
 ; sameln: function %minimal(i32) fast {
-; nextln: ebb0(v90: i32):
-; nextln:     brz v90, ebb1
+; nextln: block0(v90: i32):
+; nextln:     brz v90, block1
 ; nextln: 
-; nextln: ebb1:
-; nextln:     brnz.i32 v90, ebb1
+; nextln: block1:
+; nextln:     brnz.i32 v90, block1
 ; nextln: }
 
 function %twoargs(i32, f32) {
-ebb0(v90: i32, v91: f32):
-    brz v90, ebb1(v90, v91)
+block0(v90: i32, v91: f32):
+    brz v90, block1(v90, v91)
 
-ebb1(v92: i32, v93: f32):
-    brnz v90, ebb0(v92, v93)
+block1(v92: i32, v93: f32):
+    brnz v90, block0(v92, v93)
 }
 ; sameln: function %twoargs(i32, f32) fast {
-; nextln: ebb0(v90: i32, v91: f32):
-; nextln:     brz v90, ebb1(v90, v91)
+; nextln: block0(v90: i32, v91: f32):
+; nextln:     brz v90, block1(v90, v91)
 ; nextln: 
-; nextln: ebb1(v92: i32, v93: f32):
-; nextln:     brnz.i32 v90, ebb0(v92, v93)
+; nextln: block1(v92: i32, v93: f32):
+; nextln:     brnz.i32 v90, block0(v92, v93)
 ; nextln: }
 
 function %jumptable(i32) {
     jt200 = jump_table []
-    jt2 = jump_table [ebb10, ebb40, ebb20, ebb30]
+    jt2 = jump_table [block10, block40, block20, block30]
 
-ebb10(v3: i32):
-    br_table v3, ebb50, jt2
+block10(v3: i32):
+    br_table v3, block50, jt2
 
-ebb20:
+block20:
     trap user2
-ebb30:
+block30:
     trap user3
-ebb40:
+block40:
     trap user4
-ebb50:
+block50:
     trap user1    
 }
 ; sameln: function %jumptable(i32) fast {
-; check:      jt2 = jump_table [ebb10, ebb40, ebb20, ebb30]
+; check:      jt2 = jump_table [block10, block40, block20, block30]
 ; check:      jt200 = jump_table []
-; check:  ebb10(v3: i32):
-; nextln:     br_table v3, ebb50, jt2
+; check:  block10(v3: i32):
+; nextln:     br_table v3, block50, jt2
 ; nextln: 
-; nextln: ebb20:
+; nextln: block20:
 ; nextln:     trap user2
 ; nextln: 
-; nextln: ebb30:
+; nextln: block30:
 ; nextln:     trap user3
 ; nextln: 
-; nextln: ebb40:
+; nextln: block40:
 ; nextln:     trap user4
 ; nextln: 
-; nextln: ebb50:
+; nextln: block50:
 ; nextln:     trap user1
 ; nextln: }
diff --git a/cranelift/filetests/filetests/parser/call.clif b/cranelift/filetests/filetests/parser/call.clif
index 28e3011f33..35e43822d1 100644
--- a/cranelift/filetests/filetests/parser/call.clif
+++ b/cranelift/filetests/filetests/parser/call.clif
@@ -2,22 +2,22 @@
 test cat
 
 function %mini() {
-ebb1:
+block1:
     return
 }
 ; sameln: function %mini() fast {
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:     return
 ; nextln: }
 
 function %r1() -> i32, f32 baldrdash_system_v {
-ebb1:
+block1:
     v1 = iconst.i32 3
     v2 = f32const 0.0
     return v1, v2
 }
 ; sameln: function %r1() -> i32, f32 baldrdash_system_v {
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:     v1 = iconst.i32 3
 ; nextln:     v2 = f32const 0.0
 ; nextln:     return v1, v2
@@ -43,7 +43,7 @@ function %direct() {
     fn1 = %one() -> i32
     fn2 = %two() -> i32, f32
 
-ebb0:
+block0:
     call fn0()
     v1 = call fn1()
     v2, v3 = call fn2()
@@ -59,7 +59,7 @@ function %indirect(i64) {
     sig1 = () -> i32
     sig2 = () -> i32, f32
 
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = call_indirect sig1, v0()
     call_indirect sig0, v1(v0)
     v3, v4 = call_indirect sig2, v1()
@@ -74,7 +74,7 @@ function %long_call() {
     sig0 = ()
     fn0 = %none sig0
 
-ebb0:
+block0:
     v0 = func_addr.i32 fn0
     call_indirect sig0, v0()
     return
@@ -85,10 +85,10 @@ ebb0:
 
 ; Special purpose function arguments
 function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret {
-ebb0(v1: i32, v2: i32, v3: i32, v4: i32):
+block0(v1: i32, v2: i32, v3: i32, v4: i32):
     return v4, v2, v3, v1
 }
 ; check: function %special1(i32 sret, i32 fp, i32 csr, i32 link) -> i32 link, i32 fp, i32 csr, i32 sret fast {
-; check: ebb0(v1: i32, v2: i32, v3: i32, v4: i32):
+; check: block0(v1: i32, v2: i32, v3: i32, v4: i32):
 ; check:     return v4, v2, v3, v1
 ; check: }
diff --git a/cranelift/filetests/filetests/parser/flags.clif b/cranelift/filetests/filetests/parser/flags.clif
index aac8017e85..c8d6e78912 100644
--- a/cranelift/filetests/filetests/parser/flags.clif
+++ b/cranelift/filetests/filetests/parser/flags.clif
@@ -2,63 +2,63 @@ test cat
 test verifier
 
 function %iflags(i32) {
-ebb200(v0: i32):
+block200(v0: i32):
     v1 = ifcmp_imm v0, 17
-    brif eq v1, ebb201
-    jump ebb400
+    brif eq v1, block201
+    jump block400
 
-ebb400:
-    brif ugt v1, ebb202
-    jump ebb401
+block400:
+    brif ugt v1, block202
+    jump block401
 
-ebb401:
+block401:
     v2 = iconst.i32 34
     v3 = ifcmp v0, v2
     v4 = trueif eq v3
-    brnz v4, ebb202
-    jump ebb402
+    brnz v4, block202
+    jump block402
 
-ebb402:
+block402:
     return
 
-ebb201:
+block201:
     return
 
-ebb202:
+block202:
     trap oob
 }
 ; check: v1 = ifcmp_imm v0, 17
-; check: brif eq v1, ebb201
-; check: brif ugt v1, ebb202
+; check: brif eq v1, block201
+; check: brif ugt v1, block202
 ; check: v3 = ifcmp.i32 v0, v2
 ; check: v4 = trueif eq v3
 
 function %fflags(f32) {
-ebb200(v0: f32):
+block200(v0: f32):
     v1 = f32const 0x34.0p0
     v2 = ffcmp v0, v1
-    brff eq v2, ebb201
-    jump ebb400
+    brff eq v2, block201
+    jump block400
 
-ebb400:
-    brff ord v2, ebb202
-    jump ebb401
+block400:
+    brff ord v2, block202
+    jump block401
 
-ebb401:
+block401:
     v3 = trueff gt v2
-    brnz v3, ebb202
-    jump ebb402
+    brnz v3, block202
+    jump block402
 
-ebb402:
+block402:
     return
 
-ebb201:
+block201:
     return
 
-ebb202:
+block202:
     trap oob
 }
 ; check: v2 = ffcmp v0, v1
-; check: brff eq v2, ebb201
-; check: brff ord v2, ebb202
+; check: brff eq v2, block201
+; check: brff ord v2, block202
 ; check: v3 = trueff gt v2
diff --git a/cranelift/filetests/filetests/parser/instruction_encoding.clif b/cranelift/filetests/filetests/parser/instruction_encoding.clif
index 5f7ae26af3..5386808482 100644
--- a/cranelift/filetests/filetests/parser/instruction_encoding.clif
+++ b/cranelift/filetests/filetests/parser/instruction_encoding.clif
@@ -5,7 +5,7 @@ target riscv32
 ; regex: WS=[ \t]*
 
 function %foo(i32, i32) {
-ebb1(v0: i32 [%x8], v1: i32):
+block1(v0: i32 [%x8], v1: i32):
     [-,-] v2 = iadd v0, v1
     [-] trap heap_oob
     [R#1234, %x5, %x11] v6, v7 = iadd_ifcout v2, v0
@@ -14,7 +14,7 @@ ebb1(v0: i32 [%x8], v1: i32):
 @a5 [Iret#5] return v0, v8
 }
 ; sameln: function %foo(i32, i32) fast {
-; nextln: ebb1(v0: i32 [%x8], v1: i32):
+; nextln: block1(v0: i32 [%x8], v1: i32):
 ; nextln:     [-,-]$WS v2 = iadd v0, v1
 ; nextln:     [-]$WS trap heap_oob
 ; nextln:     [R#1234,%x5,%x11]$WS v6, v7 = iadd_ifcout v2, v0
diff --git a/cranelift/filetests/filetests/parser/memory.clif b/cranelift/filetests/filetests/parser/memory.clif
index 4e763f2b4d..ecf872d64f 100644
--- a/cranelift/filetests/filetests/parser/memory.clif
+++ b/cranelift/filetests/filetests/parser/memory.clif
@@ -4,7 +4,7 @@ test verifier
 function %vmglobal(i64 vmctx) -> i32 {
     gv3 = vmctx
     ; check: gv3 = vmctx
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = global_value.i32 gv3
     ; check: v1 = global_value.i32 gv3
     return v1
@@ -17,7 +17,7 @@ function %load_and_add_imm(i64 vmctx) -> i32 {
     ; check: gv2 = vmctx
     ; check: gv3 = load.i32 notrap aligned gv2-72
     ; check: gv4 = iadd_imm.i32 gv3, -32
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = global_value.i32 gv4
     ; check: v1 = global_value.i32 gv4
     return v1
@@ -31,7 +31,7 @@ function %backref(i64 vmctx) -> i32 {
     ; check: gv1 = load.i32 notrap aligned gv2
     gv2 = vmctx
     ; check: gv2 = vmctx
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = global_value.i32 gv1
     return v1
 }
@@ -41,7 +41,7 @@ function %symbol() -> i32 {
     ; check: gv0 = symbol %something
     gv1 = symbol u8:9
     ; check: gv1 = symbol u8:9
-ebb0:
+block0:
     v0 = global_value.i32 gv0
     ; check: v0 = global_value.i32 gv0
     v1 = global_value.i32 gv1
@@ -59,7 +59,7 @@ function %sheap(i32, i64 vmctx) -> i64 {
 
     ; check: heap1 = static gv5, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
     ; check: heap2 = static gv5, min 0, bound 0x0001_0000, offset_guard 4096
-ebb0(v1: i32, v2: i64):
+block0(v1: i32, v2: i64):
     v3 = heap_addr.i64 heap1, v1, 0
     ; check: v3 = heap_addr.i64 heap1, v1, 0
     return v3
@@ -75,7 +75,7 @@ function %dheap(i32, i64 vmctx) -> i64 {
 
     ; check: heap1 = dynamic gv5, min 0x0001_0000, bound gv6, offset_guard 0x8000_0000
     ; check: heap2 = dynamic gv5, min 0, bound gv6, offset_guard 4096
-ebb0(v1: i32, v2: i64):
+block0(v1: i32, v2: i64):
     v3 = heap_addr.i64 heap2, v1, 0
     ; check: v3 = heap_addr.i64 heap2, v1, 0
     return v3
diff --git a/cranelift/filetests/filetests/parser/rewrite.clif b/cranelift/filetests/filetests/parser/rewrite.clif
index c40bd9589b..a0520d25f5 100644
--- a/cranelift/filetests/filetests/parser/rewrite.clif
+++ b/cranelift/filetests/filetests/parser/rewrite.clif
@@ -1,16 +1,16 @@
-; It is possible to refer to instructions and EBBs that have not yet been
+; It is possible to refer to instructions and blocks that have not yet been
 ; defined in the lexical order.
 test cat
 
 ; Defining numbers.
 function %defs() {
-ebb100(v20: i32):
+block100(v20: i32):
     v1000 = iconst.i32x8 5
     v9200 = f64const 0x4.0p0
     trap user4
 }
 ; sameln: function %defs() fast {
-; nextln: ebb100(v20: i32):
+; nextln: block100(v20: i32):
 ; nextln:     v1000 = iconst.i32x8 5
 ; nextln:     v9200 = f64const 0x1.0000000000000p2
 ; nextln:     trap user4
@@ -18,14 +18,14 @@ ebb100(v20: i32):
 
 ; Using values.
 function %use_value() {
-ebb100(v20: i32):
+block100(v20: i32):
     v1000 = iadd_imm v20, 5
     v200 = iadd v20, v1000
-    jump ebb100(v1000)
+    jump block100(v1000)
 }
 ; sameln: function %use_value() fast {
-; nextln: ebb100(v20: i32):
+; nextln: block100(v20: i32):
 ; nextln:     v1000 = iadd_imm v20, 5
 ; nextln:     v200 = iadd v20, v1000
-; nextln:     jump ebb100(v1000)
+; nextln:     jump block100(v1000)
 ; nextln: }
diff --git a/cranelift/filetests/filetests/parser/ternary.clif b/cranelift/filetests/filetests/parser/ternary.clif
index d17e5592c6..b148850198 100644
--- a/cranelift/filetests/filetests/parser/ternary.clif
+++ b/cranelift/filetests/filetests/parser/ternary.clif
@@ -2,7 +2,7 @@ test cat
 test verifier
 
 function %add_i96(i32, i32, i32, i32, i32, i32) -> i32, i32, i32 {
-ebb1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32):
+block1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32):
     v10, v11 = iadd_ifcout v1, v4
     ;check: v10, v11 = iadd_ifcout v1, v4
     v20, v21 = iadd_ifcarry v2, v5, v11
@@ -13,7 +13,7 @@ ebb1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32):
 }
 
 function %sub_i96(i32, i32, i32, i32, i32, i32) -> i32, i32, i32 {
-ebb1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32):
+block1(v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32):
     v10, v11 = isub_ifbout v1, v4
     ;check: v10, v11 = isub_ifbout v1, v4
     v20, v21 = isub_ifborrow v2, v5, v11
diff --git a/cranelift/filetests/filetests/parser/tiny.clif b/cranelift/filetests/filetests/parser/tiny.clif
index 49628a27d6..98f477f808 100644
--- a/cranelift/filetests/filetests/parser/tiny.clif
+++ b/cranelift/filetests/filetests/parser/tiny.clif
@@ -2,24 +2,24 @@ test cat
 
 ; The smallest possible function.
 function %minimal() {
-ebb0:
+block0:
     trap user0
 }
 ; sameln: function %minimal() fast {
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     trap user0
 ; nextln: }
 
 ; Create and use values.
 ; Polymorphic instructions with type suffix.
 function %ivalues() {
-ebb0:
+block0:
     v0 = iconst.i32 2
     v1 = iconst.i8 6
     v2 = ishl v0, v1
 }
 ; sameln: function %ivalues() fast {
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i32 2
 ; nextln:     v1 = iconst.i8 6
 ; nextln:     v2 = ishl v0, v1
@@ -28,14 +28,14 @@ ebb0:
 ; Create and use values.
 ; Polymorphic instructions with type suffix.
 function %bvalues() {
-ebb0:
+block0:
     v0 = bconst.b32 true
     v1 = bconst.b8 false
     v2 = bextend.b32 v1
     v3 = bxor v0, v2
 }
 ; sameln: function %bvalues() fast {
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = bconst.b32 true
 ; nextln:     v1 = bconst.b8 false
 ; nextln:     v2 = bextend.b32 v1
@@ -44,33 +44,33 @@ ebb0:
 
 ; Polymorphic instruction controlled by second operand.
 function %select() {
-ebb0(v90: i32, v91: i32, v92: b1):
+block0(v90: i32, v91: i32, v92: b1):
     v0 = select v92, v90, v91
 }
 ; sameln: function %select() fast {
-; nextln: ebb0(v90: i32, v91: i32, v92: b1):
+; nextln: block0(v90: i32, v91: i32, v92: b1):
 ; nextln:     v0 = select v92, v90, v91
 ; nextln: }
 
 ; Polymorphic instruction controlled by third operand.
 function %selectif() system_v {
-ebb0(v95: i32, v96: i32, v97: b1):
+block0(v95: i32, v96: i32, v97: b1):
     v98 = selectif.i32 eq v97, v95, v96
 }
 ; sameln: function %selectif() system_v {
-; nextln: ebb0(v95: i32, v96: i32, v97: b1):
+; nextln: block0(v95: i32, v96: i32, v97: b1):
 ; nextln: v98 = selectif.i32 eq v97, v95, v96
 ; nextln: }
 
 ; Lane indexes.
 function %lanes() {
-ebb0:
+block0:
     v0 = iconst.i32x4 2
     v1 = extractlane v0, 3
     v2 = insertlane v0, 1, v1
 }
 ; sameln: function %lanes() fast {
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i32x4 2
 ; nextln:     v1 = extractlane v0, 3
 ; nextln:     v2 = insertlane v0, 1, v1
@@ -78,31 +78,31 @@ ebb0:
 
 ; Integer condition codes.
 function %icmp(i32, i32) {
-ebb0(v90: i32, v91: i32):
+block0(v90: i32, v91: i32):
     v0 = icmp eq v90, v91
     v1 = icmp ult v90, v91
     v2 = icmp_imm sge v90, -12
     v3 = irsub_imm v91, 45
-    br_icmp eq v90, v91, ebb0(v91, v90)
+    br_icmp eq v90, v91, block0(v91, v90)
 }
 ; sameln: function %icmp(i32, i32) fast {
-; nextln: ebb0(v90: i32, v91: i32):
+; nextln: block0(v90: i32, v91: i32):
 ; nextln:     v0 = icmp eq v90, v91
 ; nextln:     v1 = icmp ult v90, v91
 ; nextln:     v2 = icmp_imm sge v90, -12
 ; nextln:     v3 = irsub_imm v91, 45
-; nextln:     br_icmp eq v90, v91, ebb0(v91, v90)
+; nextln:     br_icmp eq v90, v91, block0(v91, v90)
 ; nextln: }
 
 ; Floating condition codes.
 function %fcmp(f32, f32) {
-ebb0(v90: f32, v91: f32):
+block0(v90: f32, v91: f32):
     v0 = fcmp eq v90, v91
     v1 = fcmp uno v90, v91
     v2 = fcmp lt v90, v91
 }
 ; sameln: function %fcmp(f32, f32) fast {
-; nextln: ebb0(v90: f32, v91: f32):
+; nextln: block0(v90: f32, v91: f32):
 ; nextln:     v0 = fcmp eq v90, v91
 ; nextln:     v1 = fcmp uno v90, v91
 ; nextln:     v2 = fcmp lt v90, v91
@@ -111,12 +111,12 @@ ebb0(v90: f32, v91: f32):
 ; The bitcast instruction has two type variables: The controlling type variable
 ; controls the outout type, and the input type is a free variable.
 function %bitcast(i32, f32) {
-ebb0(v90: i32, v91: f32):
+block0(v90: i32, v91: f32):
     v0 = bitcast.i8x4 v90
     v1 = bitcast.i32 v91
 }
 ; sameln: function %bitcast(i32, f32) fast {
-; nextln: ebb0(v90: i32, v91: f32):
+; nextln: block0(v90: i32, v91: f32):
 ; nextln:     v0 = bitcast.i8x4 v90
 ; nextln:     v1 = bitcast.i32 v91
 ; nextln: }
@@ -129,7 +129,7 @@ function %stack() {
     ss4 = outgoing_arg 4
     ss5 = emergency_slot 4
 
-ebb0:
+block0:
     v1 = stack_load.i32 ss10
     v2 = stack_load.i32 ss10+4
     stack_store v1, ss10+2
@@ -142,7 +142,7 @@ ebb0:
 ; check:     ss5 = emergency_slot 4
 ; check:     ss10 = spill_slot 8
 
-; check: ebb0:
+; check: block0:
 ; nextln: v1 = stack_load.i32 ss10
 ; nextln: v2 = stack_load.i32 ss10+4
 ; nextln: stack_store v1, ss10+2
@@ -150,7 +150,7 @@ ebb0:
 
 ; Memory access instructions.
 function %memory(i32) {
-ebb0(v1: i32):
+block0(v1: i32):
     v2 = load.i64 v1
     v3 = load.i64 aligned v1
     v4 = load.i64 notrap v1
@@ -167,7 +167,7 @@ ebb0(v1: i32):
     store_complex v3, v1+v2+0x1
 }
 ; sameln: function %memory(i32) fast {
-; nextln: ebb0(v1: i32):
+; nextln: block0(v1: i32):
 ; nextln:     v2 = load.i64 v1
 ; nextln:     v3 = load.i64 aligned v1
 ; nextln:     v4 = load.i64 notrap v1
@@ -188,7 +188,7 @@ ebb0(v1: i32):
 function %diversion(i32) {
     ss0 = spill_slot 4
 
-ebb0(v1: i32):
+block0(v1: i32):
     regmove v1, %10 -> %20
     regmove v1, %20 -> %10
     regspill v1, %10 -> ss0
@@ -197,7 +197,7 @@ ebb0(v1: i32):
 }
 ; sameln: function %diversion(i32) fast {
 ; nextln:     ss0 = spill_slot 4
-; check: ebb0(v1: i32):
+; check: block0(v1: i32):
 ; nextln:     regmove v1, %10 -> %20
 ; nextln:     regmove v1, %20 -> %10
 ; nextln:     regspill v1, %10 -> ss0
@@ -207,20 +207,20 @@ ebb0(v1: i32):
 
 ; Register copies.
 function %copy_special() {
-ebb0:
+block0:
     copy_special %10 -> %20
     copy_special %20 -> %10
     return
 }
 ; sameln: function %copy_special() fast {
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     copy_special %10 -> %20
 ; nextln:     copy_special %20 -> %10
 ; nextln:     return
 ; nextln: }
 
 function %cond_traps(i32) {
-ebb0(v0: i32):
+block0(v0: i32):
     trapz v0, stk_ovf
     v1 = ifcmp_imm v0, 5
     trapif ugt v1, oob
@@ -230,7 +230,7 @@ ebb0(v0: i32):
     return
 }
 ; sameln: function %cond_traps(i32)
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     trapz v0, stk_ovf
 ; nextln:     v1 = ifcmp_imm v0, 5
 ; nextln:     trapif ugt v1, oob
diff --git a/cranelift/filetests/filetests/postopt/basic.clif b/cranelift/filetests/filetests/postopt/basic.clif
index c38065f947..4fb9e9664c 100644
--- a/cranelift/filetests/filetests/postopt/basic.clif
+++ b/cranelift/filetests/filetests/postopt/basic.clif
@@ -4,29 +4,29 @@ target i686
 ; Test that compare+branch sequences are folded effectively on x86.
 
 function %br_icmp(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
 [DynRexOp1icscc#39,%rdx]  v2 = icmp slt v0, v1
-[Op1t8jccd_long#85]       brnz v2, ebb1
-[Op1jmpb#eb]              jump ebb2
+[Op1t8jccd_long#85]       brnz v2, block1
+[Op1jmpb#eb]              jump block2
 
-ebb2:
+block2:
 [Op1ret#c3]         return v1
 
-ebb1:
+block1:
 [Op1pu_id#b8,%rax]  v8 = iconst.i32 3
 [Op1ret#c3]         return v8
 }
 ; sameln: function %br_icmp
-; nextln: ebb0(v0: i32, v1: i32):
+; nextln: block0(v0: i32, v1: i32):
 ; nextln:    v9 = ifcmp v0, v1
 ; nextln:    v2 = trueif slt v9
-; nextln:    brif slt v9, ebb1
-; nextln:    jump ebb2
+; nextln:    brif slt v9, block1
+; nextln:    jump block2
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:    return v1
 ; nextln: 
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:    v8 = iconst.i32 3
 ; nextln:    return v8
 ; nextln: }
@@ -34,29 +34,29 @@ ebb1:
 ; Use brz instead of brnz, so the condition is inverted.
 
 function %br_icmp_inverse(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
 [DynRexOp1icscc#39,%rdx]  v2 = icmp slt v0, v1
-[Op1t8jccd_long#84]       brz v2, ebb1
-[Op1jmpb#eb]              jump ebb2
+[Op1t8jccd_long#84]       brz v2, block1
+[Op1jmpb#eb]              jump block2
 
-ebb2:
+block2:
 [Op1ret#c3]         return v1
 
-ebb1:
+block1:
 [Op1pu_id#b8,%rax]  v8 = iconst.i32 3
 [Op1ret#c3]         return v8
 }
 ; sameln: function %br_icmp_inverse
-; nextln: ebb0(v0: i32, v1: i32):
+; nextln: block0(v0: i32, v1: i32):
 ; nextln:    v9 = ifcmp v0, v1
 ; nextln:    v2 = trueif slt v9
-; nextln:    brif sge v9, ebb1
-; nextln:    jump ebb2
+; nextln:    brif sge v9, block1
+; nextln:    jump block2
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:    return v1
 ; nextln: 
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:    v8 = iconst.i32 3
 ; nextln:    return v8
 ; nextln: }
@@ -64,29 +64,29 @@ ebb1:
 ; Use icmp_imm instead of icmp.
 
 function %br_icmp_imm(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
 [DynRexOp1icscc_ib#7083]  v2 = icmp_imm slt v0, 2
-[Op1t8jccd_long#84]       brz v2, ebb1
-[Op1jmpb#eb]              jump ebb2
+[Op1t8jccd_long#84]       brz v2, block1
+[Op1jmpb#eb]              jump block2
 
-ebb2:
+block2:
 [Op1ret#c3]         return v1
 
-ebb1:
+block1:
 [Op1pu_id#b8,%rax]  v8 = iconst.i32 3
 [Op1ret#c3]         return v8
 }
 ; sameln: function %br_icmp_imm
-; nextln: ebb0(v0: i32, v1: i32):
+; nextln: block0(v0: i32, v1: i32):
 ; nextln:    v9 = ifcmp_imm v0, 2
 ; nextln:    v2 = trueif slt v9
-; nextln:    brif sge v9, ebb1
-; nextln:    jump ebb2
+; nextln:    brif sge v9, block1
+; nextln:    jump block2
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:    return v1
 ; nextln: 
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:    v8 = iconst.i32 3
 ; nextln:    return v8
 ; nextln: }
@@ -94,30 +94,30 @@ ebb1:
 ; Use fcmp instead of icmp.
 
 function %br_fcmp(f32, f32) -> f32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
 [Op2fcscc#42e,%rdx] v2 = fcmp gt v0, v1
-[Op1t8jccd_long#84] brz v2, ebb1
-[Op1jmpb#eb]        jump ebb2
+[Op1t8jccd_long#84] brz v2, block1
+[Op1jmpb#eb]        jump block2
 
-ebb2:
+block2:
 [Op1ret#c3]         return v1
 
-ebb1:
+block1:
 [Op1pu_id#b8,%rax]   v18 = iconst.i32 0x40a8_0000
 [Mp2frurm#56e,%xmm0] v8 = bitcast.f32 v18
 [Op1ret#c3]         return v8
 }
 ; sameln: function %br_fcmp
-; nextln: ebb0(v0: f32, v1: f32):
+; nextln: block0(v0: f32, v1: f32):
 ; nextln:    v19 = ffcmp v0, v1
 ; nextln:    v2 = trueff gt v19
-; nextln:    brff ule v19, ebb1
-; nextln:    jump ebb2
+; nextln:    brff ule v19, block1
+; nextln:    jump block2
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:    return v1
 ; nextln: 
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:    v18 = iconst.i32 0x40a8_0000
 ; nextln:    v8 = bitcast.f32 v18
 ; nextln:    return v8
diff --git a/cranelift/filetests/filetests/postopt/complex_memory_ops.clif b/cranelift/filetests/filetests/postopt/complex_memory_ops.clif
index bae58cd8bb..43206f431c 100644
--- a/cranelift/filetests/filetests/postopt/complex_memory_ops.clif
+++ b/cranelift/filetests/filetests/postopt/complex_memory_ops.clif
@@ -2,7 +2,7 @@ test postopt
 target x86_64
 
 function %dual_loads(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
 [DynRexOp1rr#8001] v3 = iadd v0, v1
                    v4 = load.i64 v3
                    v5 = uload8.i64 v3
@@ -15,7 +15,7 @@ ebb0(v0: i64, v1: i64):
 }
 
 ; sameln: function %dual_loads
-; nextln: ebb0(v0: i64, v1: i64):
+; nextln: block0(v0: i64, v1: i64):
 ; nextln:    v3 = iadd v0, v1
 ; nextln:    v4 = load_complex.i64 v0+v1
 ; nextln:    v5 = uload8_complex.i64 v0+v1
@@ -28,7 +28,7 @@ ebb0(v0: i64, v1: i64):
 ; nextln: }
 
 function %dual_loads2(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
 [DynRexOp1rr#8001] v3 = iadd v0, v1
                    v4 = load.i64 v3+1
                    v5 = uload8.i64 v3+1
@@ -41,7 +41,7 @@ ebb0(v0: i64, v1: i64):
 }
 
 ; sameln: function %dual_loads2
-; nextln: ebb0(v0: i64, v1: i64):
+; nextln: block0(v0: i64, v1: i64):
 ; nextln:    v3 = iadd v0, v1
 ; nextln:    v4 = load_complex.i64 v0+v1+1
 ; nextln:    v5 = uload8_complex.i64 v0+v1+1
@@ -54,7 +54,7 @@ ebb0(v0: i64, v1: i64):
 ; nextln: }
 
 function %dual_stores(i64, i64, i64) {
-ebb0(v0: i64, v1: i64, v2: i64):
+block0(v0: i64, v1: i64, v2: i64):
 [DynRexOp1rr#8001] v3 = iadd v0, v1
 [RexOp1st#8089]    store.i64 v2, v3
 [RexOp1st#88]      istore8.i64 v2, v3
@@ -64,7 +64,7 @@ ebb0(v0: i64, v1: i64, v2: i64):
 }
 
 ; sameln: function %dual_stores
-; nextln: ebb0(v0: i64, v1: i64, v2: i64):
+; nextln: block0(v0: i64, v1: i64, v2: i64):
 ; nextln:    v3 = iadd v0, v1
 ; nextln:    store_complex v2, v0+v1
 ; nextln:    istore8_complex v2, v0+v1
@@ -74,7 +74,7 @@ ebb0(v0: i64, v1: i64, v2: i64):
 ; nextln: }
 
 function %dual_stores2(i64, i64, i64) {
-ebb0(v0: i64, v1: i64, v2: i64):
+block0(v0: i64, v1: i64, v2: i64):
 [DynRexOp1rr#8001] v3 = iadd v0, v1
 [RexOp1stDisp8#8089]    store.i64 v2, v3+1
 [RexOp1stDisp8#88]      istore8.i64 v2, v3+1
@@ -84,7 +84,7 @@ ebb0(v0: i64, v1: i64, v2: i64):
 }
 
 ; sameln: function %dual_stores2
-; nextln: ebb0(v0: i64, v1: i64, v2: i64):
+; nextln: block0(v0: i64, v1: i64, v2: i64):
 ; nextln:    v3 = iadd v0, v1
 ; nextln:    store_complex v2, v0+v1+1
 ; nextln:    istore8_complex v2, v0+v1+1
diff --git a/cranelift/filetests/filetests/postopt/fold_offset_into_address.clif b/cranelift/filetests/filetests/postopt/fold_offset_into_address.clif
index 52379f4a50..1b58caed72 100644
--- a/cranelift/filetests/filetests/postopt/fold_offset_into_address.clif
+++ b/cranelift/filetests/filetests/postopt/fold_offset_into_address.clif
@@ -4,28 +4,28 @@ target x86_64
 ; Fold the immediate of an iadd_imm into an address offset.
 
 function u0:0(i64 vmctx) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
                      v1 = iadd_imm.i64 v0, 16
 [RexOp1ldDisp8#808b] v2 = load.i64 notrap aligned v1
 [Op1ret#c3]          return v2
 }
 
 ; sameln: function u0:0(i64 vmctx) -> i64 fast {
-; nextln: ebb0(v0: i64):
+; nextln: block0(v0: i64):
 ; nextln:                                     v1 = iadd_imm v0, 16
 ; nextln: [RexOp1ldDisp8#808b]                v2 = load.i64 notrap aligned v0+16
 ; nextln: [Op1ret#c3]                         return v2
 ; nextln: }
 
 function u0:1(i64, i64 vmctx) {
-ebb0(v3: i64, v0: i64):
+block0(v3: i64, v0: i64):
                      v1 = iadd_imm.i64 v0, 16
 [RexOp1stDisp8#8089] store.i64 notrap aligned v3, v1
 [Op1ret#c3]          return
 }
 
 ; sameln: function u0:1(i64, i64 vmctx) fast {
-; nextln: ebb0(v3: i64, v0: i64):
+; nextln: block0(v3: i64, v0: i64):
 ; nextln:                                     v1 = iadd_imm v0, 16
 ; nextln: [RexOp1stDisp8#8089]                store notrap aligned v3, v0+16
 ; nextln: [Op1ret#c3]                         return
diff --git a/cranelift/filetests/filetests/preopt/branch.clif b/cranelift/filetests/filetests/preopt/branch.clif
index 8e139952f9..50274c4890 100644
--- a/cranelift/filetests/filetests/preopt/branch.clif
+++ b/cranelift/filetests/filetests/preopt/branch.clif
@@ -2,78 +2,78 @@ test preopt
 target x86_64
 
 function %brz_fold() -> i32 {
-ebb0:
+block0:
     v0 = bconst.b1 false
-    brz v0, ebb2
-    jump ebb1
-ebb1:
+    brz v0, block2
+    jump block1
+block1:
     v1 = iconst.i32 42
     return v1
-ebb2:
+block2:
     v2 = iconst.i32 24
     return v2
 }
 ; sameln: function %brz_fold
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = bconst.b1 false
-; nextln:     jump ebb2
+; nextln:     jump block2
 ; nextln: 
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:     v1 = iconst.i32 42
 ; nextln:     return v1
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:     v2 = iconst.i32 24
 ; nextln:     return v2
 ; nextln: }
 
 function %brnz_fold() -> i32 {
-ebb0:
+block0:
     v0 = bconst.b1 true
-    brnz v0, ebb2
-    jump ebb1
-ebb1:
+    brnz v0, block2
+    jump block1
+block1:
     v1 = iconst.i32 42
     return v1
-ebb2:
+block2:
     v2 = iconst.i32 24
     return v2
 }
 ; sameln: function %brnz_fold
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = bconst.b1 true
-; nextln:     jump ebb2
+; nextln:     jump block2
 ; nextln: 
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:     v1 = iconst.i32 42
 ; nextln:     return v1
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:     v2 = iconst.i32 24
 ; nextln:     return v2
 ; nextln: }
 
 function %brz_fold_param(b1) -> i32 {
-ebb0(v0: b1):
-    brz v0, ebb2
-    jump ebb1
-ebb1:
+block0(v0: b1):
+    brz v0, block2
+    jump block1
+block1:
     v1 = iconst.i32 42
     return v1
-ebb2:
+block2:
     v2 = iconst.i32 24
     return v2
 }
 ; sameln: function %brz_fold_param(b1) -> i32 fast {
-; nextln: ebb0(v0: b1):
-; nextln:     brz v0, ebb2
-; nextln:     jump ebb1
+; nextln: block0(v0: b1):
+; nextln:     brz v0, block2
+; nextln:     jump block1
 ; nextln: 
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:     v1 = iconst.i32 42
 ; nextln:     return v1
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:     v2 = iconst.i32 24
 ; nextln:     return v2
 ; nextln: }
diff --git a/cranelift/filetests/filetests/preopt/constant_fold.clif b/cranelift/filetests/filetests/preopt/constant_fold.clif
index f0ea9539ba..e2cc3e4562 100644
--- a/cranelift/filetests/filetests/preopt/constant_fold.clif
+++ b/cranelift/filetests/filetests/preopt/constant_fold.clif
@@ -2,7 +2,7 @@ test preopt
 target x86_64
 
 function %constant_fold(f64) -> f64 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = f64const 0x1.0000000000000p0
     v2 = f64const 0x1.0000000000000p1
     v3 = fadd v1, v2
@@ -10,7 +10,7 @@ ebb0(v0: f64):
     return v4
 }
 ; sameln: function %constant_fold(f64) -> f64 fast {
-; nextln: ebb0(v0: f64):
+; nextln: block0(v0: f64):
 ; nextln:     v1 = f64const 0x1.0000000000000p0
 ; nextln:     v2 = f64const 0x1.0000000000000p1
 ; nextln:     v3 = f64const 0x1.8000000000000p1
diff --git a/cranelift/filetests/filetests/preopt/numerical.clif b/cranelift/filetests/filetests/preopt/numerical.clif
index 27fdaec0f7..044a3df6a0 100644
--- a/cranelift/filetests/filetests/preopt/numerical.clif
+++ b/cranelift/filetests/filetests/preopt/numerical.clif
@@ -2,7 +2,7 @@ test preopt
 target x86_64
 
 function %iadd_fold() -> i32 {
-ebb0:
+block0:
     v0 = iconst.i32 37
     v1 = iconst.i32 5
     v2 = iadd v0, v1
@@ -11,7 +11,7 @@ ebb0:
     return v4
 }
 ; sameln: function %iadd_fold
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i32 37
 ; nextln:     v1 = iconst.i32 5
 ; nextln:     v2 = iconst.i32 42
@@ -21,14 +21,14 @@ ebb0:
 ; nextln: }
 
 function %isub_fold() -> i32 {
-ebb0:
+block0:
     v0 = iconst.i32 42
     v1 = iconst.i32 1
     v2 = isub v0, v1
     return v2
 }
 ; sameln: function %isub_fold
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i32 42
 ; nextln:     v1 = iconst.i32 1
 ; nextln:     v2 = iconst.i32 41
diff --git a/cranelift/filetests/filetests/regalloc/aliases.clif b/cranelift/filetests/filetests/regalloc/aliases.clif
index 7e6d5c6028..6114298873 100644
--- a/cranelift/filetests/filetests/regalloc/aliases.clif
+++ b/cranelift/filetests/filetests/regalloc/aliases.clif
@@ -5,11 +5,11 @@ function %value_aliases(i32, f32, i64 vmctx) baldrdash_system_v {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: f32, v2: i64):
+block0(v0: i32, v1: f32, v2: i64):
     v3 = iconst.i32 0
-    jump ebb3(v3)
+    jump block3(v3)
 
-ebb3(v4: i32):
+block3(v4: i32):
     v5 = heap_addr.i64 heap0, v4, 1
     v6 = load.f32 v5
     v7 -> v1
@@ -21,15 +21,15 @@ ebb3(v4: i32):
     v12 -> v0
     v13 = icmp ult v11, v12
     v14 = bint.i32 v13
-    brnz v14, ebb3(v11)
-    jump ebb4
+    brnz v14, block3(v11)
+    jump block4
 
-ebb4:
-    jump ebb2
+block4:
+    jump block2
 
-ebb2:
-    jump ebb1
+block2:
+    jump block1
 
-ebb1:
+block1:
     return
 }
diff --git a/cranelift/filetests/filetests/regalloc/basic.clif b/cranelift/filetests/filetests/regalloc/basic.clif
index d10ea726ce..48111253ae 100644
--- a/cranelift/filetests/filetests/regalloc/basic.clif
+++ b/cranelift/filetests/filetests/regalloc/basic.clif
@@ -6,7 +6,7 @@ target riscv32
 ; regex: RX=%x\d+
 
 function %add(i32, i32) {
-ebb0(v1: i32, v2: i32):
+block0(v1: i32, v2: i32):
     v3 = iadd v1, v2
 ; check: [R#0c,%x5]
 ; sameln: iadd
@@ -15,7 +15,7 @@ ebb0(v1: i32, v2: i32):
 
 ; Function with a dead argument.
 function %dead_arg(i32, i32) -> i32{
-ebb0(v1: i32, v2: i32):
+block0(v1: i32, v2: i32):
 ; not: regmove
 ; check: return v1
     return v1
@@ -23,7 +23,7 @@ ebb0(v1: i32, v2: i32):
 
 ; Return a value from a different register.
 function %move1(i32, i32) -> i32 {
-ebb0(v1: i32, v2: i32):
+block0(v1: i32, v2: i32):
 ; not: regmove
 ; check: regmove v2, %x11 -> %x10
 ; nextln: return v2
@@ -32,7 +32,7 @@ ebb0(v1: i32, v2: i32):
 
 ; Swap two registers.
 function %swap(i32, i32) -> i32, i32 {
-ebb0(v1: i32, v2: i32):
+block0(v1: i32, v2: i32):
 ; not: regmove
 ; check: regmove v2, %x11 -> $(tmp=$RX)
 ; nextln: regmove v1, %x10 -> %x11
@@ -41,40 +41,40 @@ ebb0(v1: i32, v2: i32):
         return v2, v1
 }
 
-; Return an EBB argument.
-function %retebb(i32, i32) -> i32 {
-ebb0(v1: i32, v2: i32):
-    brnz v1, ebb1(v1)
-    jump ebb1(v2)
+; Return a block argument.
+function %retblock(i32, i32) -> i32 {
+block0(v1: i32, v2: i32):
+    brnz v1, block1(v1)
+    jump block1(v2)
 
-ebb1(v10: i32):
+block1(v10: i32):
     return v10
 }
 
-; Pass an EBB argument as a function argument.
-function %callebb(i32, i32) -> i32 {
+; Pass a block argument as a function argument.
+function %callblock(i32, i32) -> i32 {
     fn0 = %foo(i32) -> i32
 
-ebb0(v1: i32, v2: i32):
-    brnz v1, ebb1(v1)
-    jump ebb1(v2)
+block0(v1: i32, v2: i32):
+    brnz v1, block1(v1)
+    jump block1(v2)
 
-ebb1(v10: i32):
+block1(v10: i32):
     v11 = call fn0(v10)
     return v11
 }
 
-; Pass an EBB argument as a jump argument.
-function %jumpebb(i32, i32) -> i32 {
+; Pass a block argument as a jump argument.
+function %jumpblock(i32, i32) -> i32 {
     fn0 = %foo(i32) -> i32
 
-ebb0(v1: i32, v2: i32):
-    brnz v1, ebb1(v1, v2)
-    jump ebb1(v2, v1)
+block0(v1: i32, v2: i32):
+    brnz v1, block1(v1, v2)
+    jump block1(v2, v1)
 
-ebb1(v10: i32, v11: i32):
-    jump ebb2(v10, v11)
+block1(v10: i32, v11: i32):
+    jump block2(v10, v11)
 
-ebb2(v20: i32, v21: i32):
+block2(v20: i32, v21: i32):
     return v21
 }
diff --git a/cranelift/filetests/filetests/regalloc/coalesce.clif b/cranelift/filetests/filetests/regalloc/coalesce.clif
index 3c139e73f6..48395da1b3 100644
--- a/cranelift/filetests/filetests/regalloc/coalesce.clif
+++ b/cranelift/filetests/filetests/regalloc/coalesce.clif
@@ -5,64 +5,64 @@ target riscv32
 ; regex: V=v\d+
 ; regex: WS=\s+
 ; regex: LOC=%\w+
-; regex: EBB=ebb\d+
+; regex: BB=block\d+
 
 ; This function is already CSSA, so no copies should be inserted.
 function %cssa(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     ; not: copy
     ; v0 is used by the branch and passed as an arg - that's no conflict.
-    brnz v0, ebb1(v0)
-    jump ebb2
+    brnz v0, block1(v0)
+    jump block2
 
-ebb2:
+block2:
     ; v0 is live across the branch above. That's no conflict.
     v1 = iadd_imm v0, 7
-    jump ebb1(v1)
+    jump block1(v1)
 
-ebb1(v10: i32):
+block1(v10: i32):
     v11 = iadd_imm v10, 7
     return v11
 }
 
 function %trivial(i32) -> i32 {
-ebb0(v0: i32):
-    ; check:    brnz v0, $(splitEdge=$EBB)
-    brnz v0, ebb1(v0)
-    jump ebb2
+block0(v0: i32):
+    ; check:    brnz v0, $(splitEdge=$BB)
+    brnz v0, block1(v0)
+    jump block2
 
-ebb2:
+block2:
     ; not: copy
     v1 = iadd_imm v0, 7
-    jump ebb1(v1)
+    jump block1(v1)
 
     ; check:  $splitEdge:
     ; nextln:   $(cp1=$V) = copy.i32 v0
-    ; nextln:   jump ebb1($cp1)
+    ; nextln:   jump block1($cp1)
 
-ebb1(v10: i32):
-    ; Use v0 in the destination EBB causes a conflict.
+block1(v10: i32):
+    ; Use v0 in the destination block causes a conflict.
     v11 = iadd v10, v0
     return v11
 }
 
 ; A value is used as an SSA argument twice in the same branch.
 function %dualuse(i32) -> i32 {
-ebb0(v0: i32):
-    ; check:  brnz v0, $(splitEdge=$EBB)
-    brnz v0, ebb1(v0, v0)
-    jump ebb2
+block0(v0: i32):
+    ; check:  brnz v0, $(splitEdge=$BB)
+    brnz v0, block1(v0, v0)
+    jump block2
 
-ebb2:
+block2:
     v1 = iadd_imm v0, 7
     v2 = iadd_imm v1, 56
-    jump ebb1(v1, v2)
+    jump block1(v1, v2)
 
     ; check:  $splitEdge:
     ; check:    $(cp1=$V) = copy.i32 v0
-    ; nextln:   jump ebb1($cp1, v0)
+    ; nextln:   jump block1($cp1, v0)
 
-ebb1(v10: i32, v11: i32):
+block1(v10: i32, v11: i32):
     v12 = iadd v10, v11
     return v12
 }
@@ -70,26 +70,26 @@ ebb1(v10: i32, v11: i32):
 ; Interference away from the branch
 ; The interference can be broken with a copy at either branch.
 function %interference(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     ; not:    copy
-    ; check:  brnz v0, $(splitEdge=$EBB)
+    ; check:  brnz v0, $(splitEdge=$BB)
     ; not:    copy
-    brnz v0, ebb1(v0)
-    jump ebb2
+    brnz v0, block1(v0)
+    jump block2
 
-ebb2:
+block2:
     v1 = iadd_imm v0, 7
     ; v1 and v0 interfere here:
     v2 = iadd_imm v0, 8
     ; check: $(cp0=$V) = copy v1
-    ; check: jump ebb1($cp0)
-    jump ebb1(v1)
+    ; check: jump block1($cp0)
+    jump block1(v1)
 
     ; check:  $splitEdge:
     ; not:      copy
-    ; nextln:   jump ebb1(v0)
+    ; nextln:   jump block1(v0)
 
-ebb1(v10: i32):
+block1(v10: i32):
     ; not: copy
     v11 = iadd_imm v10, 7
     return v11
@@ -97,27 +97,27 @@ ebb1(v10: i32):
 
 ; A loop where one induction variable is used as a backedge argument.
 function %fibonacci(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 1
     v2 = iconst.i32 2
-    jump ebb1(v1, v2)
+    jump block1(v1, v2)
 
-    ; check:  $(splitEdge=$EBB):
+    ; check:  $(splitEdge=$BB):
     ; check:    $(nv11b=$V) = copy.i32 v11
     ; not:      copy
-    ; check:    jump ebb1($nv11b, v12)
+    ; check:    jump block1($nv11b, v12)
 
-ebb1(v10: i32, v11: i32):
+block1(v10: i32, v11: i32):
     ; v11 needs to be isolated because it interferes with v10.
-    ; check: ebb1(v10: i32 [$LOC], $(nv11a=$V): i32 [$LOC])
+    ; check: block1(v10: i32 [$LOC], $(nv11a=$V): i32 [$LOC])
     ; check: v11 = copy $nv11a
     v12 = iadd v10, v11
     v13 = icmp ult v12, v0
     ; check:  brnz v13, $splitEdge
-    brnz v13, ebb1(v11, v12)
-    jump ebb2
+    brnz v13, block1(v11, v12)
+    jump block2
 
-ebb2:
+block2:
     return v12
 }
 
@@ -128,30 +128,30 @@ ebb2:
 function %stackarg(i32, i32, i32, i32, i32, i32, i32, i32, i32) -> i32 {
 ; check: ss0 = incoming_arg 4
 ; not: incoming_arg
-ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32, v7: i32, v8: i32):
+block0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32, v7: i32, v8: i32):
     ; check: fill v8
     ; not: v8
-    jump ebb1(v8)
+    jump block1(v8)
 
-ebb1(v10: i32):
+block1(v10: i32):
     v11 = iadd_imm v10, 1
     return v11
 }
 
 function %gvn_unremovable_phi(i32) system_v {
-ebb0(v0: i32):
+block0(v0: i32):
     v2 = iconst.i32 0
-    jump ebb2(v2, v0)
+    jump block2(v2, v0)
 
-ebb2(v3: i32, v4: i32):
-    brnz v3, ebb2(v3, v4)
-    jump ebb3
+block2(v3: i32, v4: i32):
+    brnz v3, block2(v3, v4)
+    jump block3
 
-ebb3:
+block3:
     v5 = iconst.i32 1
-    brnz v3, ebb2(v2, v5)
-    jump ebb4
+    brnz v3, block2(v2, v5)
+    jump block4
 
-ebb4:
+block4:
     return
 }
diff --git a/cranelift/filetests/filetests/regalloc/coalescing-207.clif b/cranelift/filetests/filetests/regalloc/coalescing-207.clif
index 6af9fcd144..39ddf0fa88 100644
--- a/cranelift/filetests/filetests/regalloc/coalescing-207.clif
+++ b/cranelift/filetests/filetests/regalloc/coalescing-207.clif
@@ -15,19 +15,19 @@ function %pr207(i64 vmctx, i32, i32) -> i32 system_v {
     fn1 = u0:0 sig1
     fn2 = u0:1 sig2
 
-ebb0(v0: i64, v1: i32, v2: i32):
+block0(v0: i64, v1: i32, v2: i32):
     v3 = iconst.i32 0
     v4 = iconst.i32 0
     v5 = iconst.i32 0
     v6 = iconst.i32 0x4ffe
     v7 = icmp uge v5, v6
-    brz v7, ebb1
-    jump ebb100
+    brz v7, block1
+    jump block100
 
-ebb100:
+block100:
     trap heap_oob
 
-ebb1:
+block1:
     v8 = uextend.i64 v5
     v9 = iadd_imm.i64 v0, -8
     v10 = load.i64 v9
@@ -37,13 +37,13 @@ ebb1:
     v14 = isub v12, v13
     v15 = iconst.i32 0x4ffe
     v16 = icmp.i32 uge v4, v15
-    brz v16, ebb2
-    jump ebb101
+    brz v16, block2
+    jump block101
 
-ebb101:
+block101:
     trap heap_oob
 
-ebb2:
+block2:
     v17 = uextend.i64 v4
     v18 = iadd_imm.i64 v0, -8
     v19 = load.i64 v18
@@ -51,13 +51,13 @@ ebb2:
     store.i32 v14, v20+4
     v21 = iconst.i32 0x4ffe
     v22 = icmp.i32 uge v2, v21
-    brz v22, ebb3
-    jump ebb102
+    brz v22, block3
+    jump block102
 
-ebb102:
+block102:
     trap heap_oob
 
-ebb3:
+block3:
     v23 = uextend.i64 v2
     v24 = iadd_imm.i64 v0, -8
     v25 = load.i64 v24
@@ -68,28 +68,28 @@ ebb3:
     v30 = iconst.i32 0
     v31 = icmp eq v29, v30
     v32 = bint.i32 v31
-    brnz v32, ebb90(v14, v1)
-    jump ebb103
+    brnz v32, block90(v14, v1)
+    jump block103
 
-ebb103:
+block103:
     v33 = call fn0(v0, v1, v27)
     v34 = iconst.i32 0
     v35 = iconst.i32 0
     v36 = icmp eq v33, v35
     v37 = bint.i32 v36
-    brnz v37, ebb90(v14, v34)
-    jump ebb104
+    brnz v37, block90(v14, v34)
+    jump block104
 
-ebb104:
+block104:
     v38 = iconst.i32 0x4ffe
     v39 = icmp.i32 uge v2, v38
-    brz v39, ebb4
-    jump ebb105
+    brz v39, block4
+    jump block105
 
-ebb105:
+block105:
     trap heap_oob
 
-ebb4:
+block4:
     v40 = uextend.i64 v2
     v41 = iadd_imm.i64 v0, -8
     v42 = load.i64 v41
@@ -98,19 +98,19 @@ ebb4:
     v45 = iconst.i32 0
     v46 = icmp eq v44, v45
     v47 = bint.i32 v46
-    brnz v47, ebb56(v33, v14)
-    jump ebb106
+    brnz v47, block56(v33, v14)
+    jump block106
 
-ebb106:
+block106:
     v48 = iconst.i32 0x4ffe
     v49 = icmp.i32 uge v33, v48
-    brz v49, ebb5
-    jump ebb107
+    brz v49, block5
+    jump block107
 
-ebb107:
+block107:
     trap heap_oob
 
-ebb5:
+block5:
     v50 = uextend.i64 v33
     v51 = iadd_imm.i64 v0, -8
     v52 = load.i64 v51
@@ -119,19 +119,19 @@ ebb5:
     v55 = iconst.i32 0
     v56 = icmp eq v54, v55
     v57 = bint.i32 v56
-    brnz v57, ebb90(v14, v34)
-    jump ebb108
+    brnz v57, block90(v14, v34)
+    jump block108
 
-ebb108:
+block108:
     v58 = iconst.i32 0x4ffe
     v59 = icmp.i32 uge v2, v58
-    brz v59, ebb6
-    jump ebb109
+    brz v59, block6
+    jump block109
 
-ebb109:
+block109:
     trap heap_oob
 
-ebb6:
+block6:
     v60 = uextend.i64 v2
     v61 = iadd_imm.i64 v0, -8
     v62 = load.i64 v61
@@ -140,19 +140,19 @@ ebb6:
     v65 = iconst.i32 0
     v66 = icmp eq v64, v65
     v67 = bint.i32 v66
-    brnz v67, ebb42
-    jump ebb110
+    brnz v67, block42
+    jump block110
 
-ebb110:
+block110:
     v68 = iconst.i32 0x4ffe
     v69 = icmp.i32 uge v33, v68
-    brz v69, ebb7
-    jump ebb111
+    brz v69, block7
+    jump block111
 
-ebb111:
+block111:
     trap heap_oob
 
-ebb7:
+block7:
     v70 = uextend.i64 v33
     v71 = iadd_imm.i64 v0, -8
     v72 = load.i64 v71
@@ -161,19 +161,19 @@ ebb7:
     v75 = iconst.i32 0
     v76 = icmp eq v74, v75
     v77 = bint.i32 v76
-    brnz v77, ebb90(v14, v34)
-    jump ebb112
+    brnz v77, block90(v14, v34)
+    jump block112
 
-ebb112:
+block112:
     v78 = iconst.i32 0x4ffe
     v79 = icmp.i32 uge v2, v78
-    brz v79, ebb8
-    jump ebb113
+    brz v79, block8
+    jump block113
 
-ebb113:
+block113:
     trap heap_oob
 
-ebb8:
+block8:
     v80 = uextend.i64 v2
     v81 = iadd_imm.i64 v0, -8
     v82 = load.i64 v81
@@ -182,19 +182,19 @@ ebb8:
     v85 = iconst.i32 0
     v86 = icmp eq v84, v85
     v87 = bint.i32 v86
-    brnz v87, ebb46
-    jump ebb114
+    brnz v87, block46
+    jump block114
 
-ebb114:
+block114:
     v88 = iconst.i32 0x4ffe
     v89 = icmp.i32 uge v33, v88
-    brz v89, ebb9
-    jump ebb115
+    brz v89, block9
+    jump block115
 
-ebb115:
+block115:
     trap heap_oob
 
-ebb9:
+block9:
     v90 = uextend.i64 v33
     v91 = iadd_imm.i64 v0, -8
     v92 = load.i64 v91
@@ -203,19 +203,19 @@ ebb9:
     v95 = iconst.i32 0
     v96 = icmp eq v94, v95
     v97 = bint.i32 v96
-    brnz v97, ebb90(v14, v34)
-    jump ebb116
+    brnz v97, block90(v14, v34)
+    jump block116
 
-ebb116:
+block116:
     v98 = iconst.i32 0x4ffe
     v99 = icmp.i32 uge v2, v98
-    brz v99, ebb10
-    jump ebb117
+    brz v99, block10
+    jump block117
 
-ebb117:
+block117:
     trap heap_oob
 
-ebb10:
+block10:
     v100 = uextend.i64 v2
     v101 = iadd_imm.i64 v0, -8
     v102 = load.i64 v101
@@ -224,10 +224,10 @@ ebb10:
     v105 = iconst.i32 0
     v106 = icmp eq v104, v105
     v107 = bint.i32 v106
-    brnz v107, ebb54
-    jump ebb118
+    brnz v107, block54
+    jump block118
 
-ebb118:
+block118:
     v108 = iconst.i32 1
     v109 = iadd.i32 v2, v108
     v110 = iconst.i32 1048
@@ -235,13 +235,13 @@ ebb118:
     v112 = iconst.i64 0
     v113 = iconst.i32 0x4ffe
     v114 = icmp uge v111, v113
-    brz v114, ebb11
-    jump ebb119
+    brz v114, block11
+    jump block119
 
-ebb119:
+block119:
     trap heap_oob
 
-ebb11:
+block11:
     v115 = uextend.i64 v111
     v116 = iadd_imm.i64 v0, -8
     v117 = load.i64 v116
@@ -252,13 +252,13 @@ ebb11:
     v121 = iconst.i64 0
     v122 = iconst.i32 0x4ffe
     v123 = icmp uge v120, v122
-    brz v123, ebb12
-    jump ebb120
+    brz v123, block12
+    jump block120
 
-ebb120:
+block120:
     trap heap_oob
 
-ebb12:
+block12:
     v124 = uextend.i64 v120
     v125 = iadd_imm.i64 v0, -8
     v126 = load.i64 v125
@@ -267,13 +267,13 @@ ebb12:
     v128 = iconst.i64 0
     v129 = iconst.i32 0x4ffe
     v130 = icmp.i32 uge v14, v129
-    brz v130, ebb13
-    jump ebb121
+    brz v130, block13
+    jump block121
 
-ebb121:
+block121:
     trap heap_oob
 
-ebb13:
+block13:
     v131 = uextend.i64 v14
     v132 = iadd_imm.i64 v0, -8
     v133 = load.i64 v132
@@ -282,34 +282,34 @@ ebb13:
     v135 = iconst.i64 0
     v136 = iconst.i32 0x4ffe
     v137 = icmp.i32 uge v14, v136
-    brz v137, ebb14
-    jump ebb122
+    brz v137, block14
+    jump block122
 
-ebb122:
+block122:
     trap heap_oob
 
-ebb14:
+block14:
     v138 = uextend.i64 v14
     v139 = iadd_imm.i64 v0, -8
     v140 = load.i64 v139
     v141 = iadd v140, v138
     store.i64 v135, v141+1024
     v142 = iconst.i32 -1
-    jump ebb15(v142, v27)
+    jump block15(v142, v27)
 
-ebb15(v143: i32, v144: i32):
+block15(v143: i32, v144: i32):
     v145 = iadd.i32 v33, v143
     v146 = iconst.i32 1
     v147 = iadd v145, v146
     v148 = iconst.i32 0x4ffe
     v149 = icmp uge v147, v148
-    brz v149, ebb16
-    jump ebb123
+    brz v149, block16
+    jump block123
 
-ebb123:
+block123:
     trap heap_oob
 
-ebb16:
+block16:
     v150 = uextend.i64 v147
     v151 = iadd_imm.i64 v0, -8
     v152 = load.i64 v151
@@ -318,10 +318,10 @@ ebb16:
     v155 = iconst.i32 0
     v156 = icmp eq v154, v155
     v157 = bint.i32 v156
-    brnz v157, ebb89(v14)
-    jump ebb124
+    brnz v157, block89(v14)
+    jump block124
 
-ebb124:
+block124:
     v158 = iconst.i32 255
     v159 = band.i32 v144, v158
     v160 = iconst.i32 2
@@ -331,13 +331,13 @@ ebb124:
     v164 = iadd.i32 v143, v163
     v165 = iconst.i32 0x4ffe
     v166 = icmp uge v162, v165
-    brz v166, ebb17
-    jump ebb125
+    brz v166, block17
+    jump block125
 
-ebb125:
+block125:
     trap heap_oob
 
-ebb17:
+block17:
     v167 = uextend.i64 v162
     v168 = iadd_imm.i64 v0, -8
     v169 = load.i64 v168
@@ -352,13 +352,13 @@ ebb17:
     v177 = iadd v172, v176
     v178 = iconst.i32 0x4ffe
     v179 = icmp uge v177, v178
-    brz v179, ebb18
-    jump ebb126
+    brz v179, block18
+    jump block126
 
-ebb126:
+block126:
     trap heap_oob
 
-ebb18:
+block18:
     v180 = uextend.i64 v177
     v181 = iadd_imm.i64 v0, -8
     v182 = load.i64 v181
@@ -371,13 +371,13 @@ ebb18:
     v189 = bor v184, v188
     v190 = iconst.i32 0x4ffe
     v191 = icmp.i32 uge v177, v190
-    brz v191, ebb19
-    jump ebb127
+    brz v191, block19
+    jump block127
 
-ebb127:
+block127:
     trap heap_oob
 
-ebb19:
+block19:
     v192 = uextend.i64 v177
     v193 = iadd_imm.i64 v0, -8
     v194 = load.i64 v193
@@ -390,22 +390,22 @@ ebb19:
     v200 = iadd v196, v199
     v201 = iconst.i32 0x4ffe
     v202 = icmp uge v200, v201
-    brz v202, ebb20
-    jump ebb128
+    brz v202, block20
+    jump block128
 
-ebb128:
+block128:
     trap heap_oob
 
-ebb20:
+block20:
     v203 = uextend.i64 v200
     v204 = iadd_imm.i64 v0, -8
     v205 = load.i64 v204
     v206 = iadd v205, v203
     v207 = uload8.i32 v206
-    brnz v207, ebb15(v198, v207)
-    jump ebb21
+    brnz v207, block15(v198, v207)
+    jump block21
 
-ebb21:
+block21:
     v208 = iconst.i32 -1
     v209 = iconst.i32 1
     v210 = iconst.i32 -1
@@ -415,31 +415,31 @@ ebb21:
     v214 = iconst.i32 2
     v215 = icmp ult v213, v214
     v216 = bint.i32 v215
-    brnz v216, ebb38(v2, v211, v209, v210, v208, v198, v213, v33, v14)
-    jump ebb129
+    brnz v216, block38(v2, v211, v209, v210, v208, v198, v213, v33, v14)
+    jump block129
 
-ebb129:
+block129:
     v217 = iconst.i32 -1
     v218 = iconst.i32 0
     v219 = iconst.i32 1
     v220 = iconst.i32 1
     v221 = iconst.i32 1
     v222 = copy.i32 v44
-    jump ebb22(v217, v221, v44, v220, v218, v219, v213, v222, v198, v33, v14)
+    jump block22(v217, v221, v44, v220, v218, v219, v213, v222, v198, v33, v14)
 
-ebb22(v223: i32, v224: i32, v225: i32, v226: i32, v227: i32, v228: i32, v229: i32, v230: i32, v231: i32, v232: i32, v233: i32):
+block22(v223: i32, v224: i32, v225: i32, v226: i32, v227: i32, v228: i32, v229: i32, v230: i32, v231: i32, v232: i32, v233: i32):
     v234 = copy v228
     v235 = iadd v223, v224
     v236 = iadd.i32 v2, v235
     v237 = iconst.i32 0x4ffe
     v238 = icmp uge v236, v237
-    brz v238, ebb23
-    jump ebb130
+    brz v238, block23
+    jump block130
 
-ebb130:
+block130:
     trap heap_oob
 
-ebb23:
+block23:
     v239 = uextend.i64 v236
     v240 = iadd_imm.i64 v0, -8
     v241 = load.i64 v240
@@ -449,95 +449,95 @@ ebb23:
     v245 = band.i32 v225, v244
     v246 = icmp ne v243, v245
     v247 = bint.i32 v246
-    brnz v247, ebb24
-    jump ebb131
+    brnz v247, block24
+    jump block131
 
-ebb131:
+block131:
     v248 = icmp.i32 ne v224, v226
     v249 = bint.i32 v248
-    brnz v249, ebb25
-    jump ebb132
+    brnz v249, block25
+    jump block132
 
-ebb132:
+block132:
     v250 = iadd.i32 v227, v226
     v251 = iconst.i32 1
-    jump ebb27(v251, v250, v223, v226)
+    jump block27(v251, v250, v223, v226)
 
-ebb24:
+block24:
     v252 = icmp.i32 ule v243, v245
     v253 = bint.i32 v252
-    brnz v253, ebb26
-    jump ebb133
+    brnz v253, block26
+    jump block133
 
-ebb133:
+block133:
     v254 = isub.i32 v234, v223
     v255 = iconst.i32 1
-    jump ebb27(v255, v234, v223, v254)
+    jump block27(v255, v234, v223, v254)
 
-ebb25:
+block25:
     v256 = iconst.i32 1
     v257 = iadd.i32 v224, v256
     v258 = copy.i32 v227
-    jump ebb27(v257, v258, v223, v226)
+    jump block27(v257, v258, v223, v226)
 
-ebb26:
+block26:
     v259 = iconst.i32 1
     v260 = iconst.i32 1
     v261 = iadd.i32 v227, v260
     v262 = iconst.i32 1
     v263 = copy.i32 v227
-    jump ebb27(v259, v261, v263, v262)
+    jump block27(v259, v261, v263, v262)
 
-ebb27(v264: i32, v265: i32, v266: i32, v267: i32):
+block27(v264: i32, v265: i32, v266: i32, v267: i32):
     v268 = iadd v264, v265
     v269 = icmp uge v268, v229
     v270 = bint.i32 v269
-    brnz v270, ebb29
-    jump ebb134
+    brnz v270, block29
+    jump block134
 
-ebb134:
+block134:
     v271 = iadd.i32 v2, v268
     v272 = iconst.i32 0x4ffe
     v273 = icmp uge v271, v272
-    brz v273, ebb28
-    jump ebb135
+    brz v273, block28
+    jump block135
 
-ebb135:
+block135:
     trap heap_oob
 
-ebb28:
+block28:
     v274 = uextend.i64 v271
     v275 = iadd_imm.i64 v0, -8
     v276 = load.i64 v275
     v277 = iadd v276, v274
     v278 = uload8.i32 v277
     v279 = copy.i32 v265
-    jump ebb22(v266, v264, v278, v267, v279, v268, v229, v230, v231, v232, v233)
+    jump block22(v266, v264, v278, v267, v279, v268, v229, v230, v231, v232, v233)
 
-ebb29:
-    jump ebb30
+block29:
+    jump block30
 
-ebb30:
+block30:
     v280 = iconst.i32 -1
     v281 = iconst.i32 0
     v282 = iconst.i32 1
     v283 = iconst.i32 1
     v284 = iconst.i32 1
-    jump ebb31(v280, v284, v230, v283, v281, v282, v229, v267, v266, v231, v232, v233)
+    jump block31(v280, v284, v230, v283, v281, v282, v229, v267, v266, v231, v232, v233)
 
-ebb31(v285: i32, v286: i32, v287: i32, v288: i32, v289: i32, v290: i32, v291: i32, v292: i32, v293: i32, v294: i32, v295: i32, v296: i32):
+block31(v285: i32, v286: i32, v287: i32, v288: i32, v289: i32, v290: i32, v291: i32, v292: i32, v293: i32, v294: i32, v295: i32, v296: i32):
     v297 = copy v290
     v298 = iadd v285, v286
     v299 = iadd.i32 v2, v298
     v300 = iconst.i32 0x4ffe
     v301 = icmp uge v299, v300
-    brz v301, ebb32
-    jump ebb136
+    brz v301, block32
+    jump block136
 
-ebb136:
+block136:
     trap heap_oob
 
-ebb32:
+block32:
     v302 = uextend.i64 v299
     v303 = iadd_imm.i64 v0, -8
     v304 = load.i64 v303
@@ -547,105 +547,105 @@ ebb32:
     v308 = band.i32 v287, v307
     v309 = icmp ne v306, v308
     v310 = bint.i32 v309
-    brnz v310, ebb33
-    jump ebb137
+    brnz v310, block33
+    jump block137
 
-ebb137:
+block137:
     v311 = icmp.i32 ne v286, v288
     v312 = bint.i32 v311
-    brnz v312, ebb34
-    jump ebb138
+    brnz v312, block34
+    jump block138
 
-ebb138:
+block138:
     v313 = iadd.i32 v289, v288
     v314 = iconst.i32 1
-    jump ebb36(v314, v313, v285, v288)
+    jump block36(v314, v313, v285, v288)
 
-ebb33:
+block33:
     v315 = icmp.i32 uge v306, v308
     v316 = bint.i32 v315
-    brnz v316, ebb35
-    jump ebb139
+    brnz v316, block35
+    jump block139
 
-ebb139:
+block139:
     v317 = isub.i32 v297, v285
     v318 = iconst.i32 1
-    jump ebb36(v318, v297, v285, v317)
+    jump block36(v318, v297, v285, v317)
 
-ebb34:
+block34:
     v319 = iconst.i32 1
     v320 = iadd.i32 v286, v319
     v321 = copy.i32 v289
-    jump ebb36(v320, v321, v285, v288)
+    jump block36(v320, v321, v285, v288)
 
-ebb35:
+block35:
     v322 = iconst.i32 1
     v323 = iconst.i32 1
     v324 = iadd.i32 v289, v323
     v325 = iconst.i32 1
     v326 = copy.i32 v289
-    jump ebb36(v322, v324, v326, v325)
+    jump block36(v322, v324, v326, v325)
 
-ebb36(v327: i32, v328: i32, v329: i32, v330: i32):
+block36(v327: i32, v328: i32, v329: i32, v330: i32):
     v331 = iadd v327, v328
     v332 = icmp uge v331, v291
     v333 = bint.i32 v332
-    brnz v333, ebb38(v2, v330, v292, v329, v293, v294, v291, v295, v296)
-    jump ebb140
+    brnz v333, block38(v2, v330, v292, v329, v293, v294, v291, v295, v296)
+    jump block140
 
-ebb140:
+block140:
     v334 = iadd.i32 v2, v331
     v335 = iconst.i32 0x4ffe
     v336 = icmp uge v334, v335
-    brz v336, ebb37
-    jump ebb141
+    brz v336, block37
+    jump block141
 
-ebb141:
+block141:
     trap heap_oob
 
-ebb37:
+block37:
     v337 = uextend.i64 v334
     v338 = iadd_imm.i64 v0, -8
     v339 = load.i64 v338
     v340 = iadd v339, v337
     v341 = uload8.i32 v340
     v342 = copy.i32 v328
-    jump ebb31(v329, v327, v341, v330, v342, v331, v291, v292, v293, v294, v295, v296)
+    jump block31(v329, v327, v341, v330, v342, v331, v291, v292, v293, v294, v295, v296)
 
-ebb38(v343: i32, v344: i32, v345: i32, v346: i32, v347: i32, v348: i32, v349: i32, v350: i32, v351: i32):
+block38(v343: i32, v344: i32, v345: i32, v346: i32, v347: i32, v348: i32, v349: i32, v350: i32, v351: i32):
     v352 = iconst.i32 1
     v353 = iadd v346, v352
     v354 = iconst.i32 1
     v355 = iadd v347, v354
     v356 = icmp ugt v353, v355
     v357 = bint.i32 v356
-    brnz v357, ebb39(v344)
-    jump ebb142
+    brnz v357, block39(v344)
+    jump block142
 
-ebb142:
+block142:
     v358 = copy v345
-    jump ebb39(v358)
+    jump block39(v358)
 
-ebb39(v359: i32):
+block39(v359: i32):
     v360 = iadd.i32 v343, v359
-    brnz.i32 v357, ebb40(v346)
-    jump ebb143
+    brnz.i32 v357, block40(v346)
+    jump block143
 
-ebb143:
+block143:
     v361 = copy.i32 v347
-    jump ebb40(v361)
+    jump block40(v361)
 
-ebb40(v362: i32):
+block40(v362: i32):
     v363 = iconst.i32 1
     v364 = iadd v362, v363
     v365 = call fn1(v0, v343, v360, v364)
     v366 = iconst.i32 0
     v367 = icmp eq v365, v366
     v368 = bint.i32 v367
-    brnz v368, ebb63
-    jump ebb144
+    brnz v368, block63
+    jump block144
 
-ebb144:
+block144:
     v369 = iconst.i32 1
     v370 = iadd v362, v369
     v371 = isub.i32 v348, v370
@@ -654,40 +654,40 @@ ebb144:
     v374 = icmp ugt v362, v373
     v375 = bint.i32 v374
     v376 = copy v362
-    brnz v375, ebb41(v376)
-    jump ebb145
+    brnz v375, block41(v376)
+    jump block145
 
-ebb145:
+block145:
     v377 = copy v373
-    jump ebb41(v377)
+    jump block41(v377)
 
-ebb41(v378: i32):
+block41(v378: i32):
     v379 = iconst.i32 1
     v380 = iadd v378, v379
     v381 = iconst.i32 0
-    jump ebb64(v380, v381)
+    jump block64(v380, v381)
 
-ebb42:
+block42:
     v382 = iconst.i32 8
     v383 = ishl.i32 v29, v382
     v384 = bor v383, v44
     v385 = iconst.i32 0x4ffe
     v386 = icmp.i32 uge v33, v385
-    brz v386, ebb43
-    jump ebb146
+    brz v386, block43
+    jump block146
 
-ebb146:
+block146:
     trap heap_oob
 
-ebb43:
+block43:
     v387 = uextend.i64 v33
     v388 = iadd_imm.i64 v0, -8
     v389 = load.i64 v388
     v390 = iadd v389, v387
     v391 = uload8.i32 v390
-    jump ebb44(v391, v54, v33)
+    jump block44(v391, v54, v33)
 
-ebb44(v392: i32, v393: i32, v394: i32):
+block44(v392: i32, v393: i32, v394: i32):
     v395 = iconst.i32 8
     v396 = ishl v392, v395
     v397 = iconst.i32 0xff00
@@ -697,32 +697,32 @@ ebb44(v392: i32, v393: i32, v394: i32):
     v401 = bor v398, v400
     v402 = icmp eq v401, v384
     v403 = bint.i32 v402
-    brnz v403, ebb56(v394, v14)
-    jump ebb147
+    brnz v403, block56(v394, v14)
+    jump block147
 
-ebb147:
+block147:
     v404 = iconst.i32 2
     v405 = iadd v394, v404
     v406 = iconst.i32 1
     v407 = iadd v394, v406
     v408 = iconst.i32 0x4ffe
     v409 = icmp uge v405, v408
-    brz v409, ebb45
-    jump ebb148
+    brz v409, block45
+    jump block148
 
-ebb148:
+block148:
     trap heap_oob
 
-ebb45:
+block45:
     v410 = uextend.i64 v405
     v411 = iadd_imm.i64 v0, -8
     v412 = load.i64 v411
     v413 = iadd v412, v410
     v414 = uload8.i32 v413
-    brnz v414, ebb44(v401, v414, v407)
-    jump ebb90(v14, v34)
+    brnz v414, block44(v401, v414, v407)
+    jump block90(v14, v34)
 
-ebb46:
+block46:
     v415 = iconst.i32 8
     v416 = ishl.i32 v74, v415
     v417 = iconst.i32 16
@@ -730,13 +730,13 @@ ebb46:
     v419 = bor v416, v418
     v420 = iconst.i32 0x4ffe
     v421 = icmp.i32 uge v33, v420
-    brz v421, ebb47
-    jump ebb149
+    brz v421, block47
+    jump block149
 
-ebb149:
+block149:
     trap heap_oob
 
-ebb47:
+block47:
     v422 = uextend.i64 v33
     v423 = iadd_imm.i64 v0, -8
     v424 = load.i64 v423
@@ -755,23 +755,23 @@ ebb47:
     v437 = bor v434, v436
     v438 = icmp eq v429, v437
     v439 = bint.i32 v438
-    brnz v439, ebb56(v33, v14)
-    jump ebb48(v33, v429)
+    brnz v439, block56(v33, v14)
+    jump block48(v33, v429)
 
-ebb48(v440: i32, v441: i32):
+block48(v440: i32, v441: i32):
     v442 = iconst.i32 1
     v443 = iadd v440, v442
     v444 = iconst.i32 3
     v445 = iadd v440, v444
     v446 = iconst.i32 0x4ffe
     v447 = icmp uge v445, v446
-    brz v447, ebb49
-    jump ebb150
+    brz v447, block49
+    jump block150
 
-ebb150:
+block150:
     trap heap_oob
 
-ebb49:
+block49:
     v448 = uextend.i64 v445
     v449 = iadd_imm.i64 v0, -8
     v450 = load.i64 v449
@@ -780,52 +780,52 @@ ebb49:
     v453 = iconst.i32 0
     v454 = icmp eq v452, v453
     v455 = bint.i32 v454
-    brnz v455, ebb51(v14)
-    jump ebb151
+    brnz v455, block51(v14)
+    jump block151
 
-ebb151:
+block151:
     v456 = bor.i32 v441, v452
     v457 = iconst.i32 8
     v458 = ishl v456, v457
     v459 = icmp ne v458, v437
     v460 = bint.i32 v459
     v461 = copy.i32 v443
-    brnz v460, ebb48(v461, v458)
-    jump ebb50
+    brnz v460, block48(v461, v458)
+    jump block50
 
-ebb50:
-    jump ebb51(v14)
+block50:
+    jump block51(v14)
 
-ebb51(v462: i32):
+block51(v462: i32):
     v463 = iconst.i32 0
     v464 = iconst.i32 1056
     v465 = iadd v462, v464
     v466 = iconst.i32 0x4ffe
     v467 = icmp uge v463, v466
-    brz v467, ebb52
-    jump ebb152
+    brz v467, block52
+    jump block152
 
-ebb152:
+block152:
     trap heap_oob
 
-ebb52:
+block52:
     v468 = uextend.i64 v463
     v469 = iadd_imm.i64 v0, -8
     v470 = load.i64 v469
     v471 = iadd v470, v468
     store.i32 v465, v471+4
     v472 = iconst.i32 0
-    brnz.i32 v452, ebb53(v443)
-    jump ebb153
+    brnz.i32 v452, block53(v443)
+    jump block153
 
-ebb153:
+block153:
     v473 = copy v472
-    jump ebb53(v473)
+    jump block53(v473)
 
-ebb53(v474: i32):
+block53(v474: i32):
     return v474
 
-ebb54:
+block54:
     v475 = iconst.i32 8
     v476 = ishl.i32 v74, v475
     v477 = iconst.i32 16
@@ -834,13 +834,13 @@ ebb54:
     v480 = bor v479, v94
     v481 = iconst.i32 0x4ffe
     v482 = icmp.i32 uge v33, v481
-    brz v482, ebb55
-    jump ebb154
+    brz v482, block55
+    jump block154
 
-ebb154:
+block154:
     trap heap_oob
 
-ebb55:
+block55:
     v483 = uextend.i64 v33
     v484 = iadd_imm.i64 v0, -8
     v485 = load.i64 v484
@@ -860,30 +860,30 @@ ebb55:
     v499 = bor v498, v84
     v500 = icmp ne v490, v499
     v501 = bint.i32 v500
-    brnz v501, ebb57
-    jump ebb56(v33, v14)
+    brnz v501, block57
+    jump block56(v33, v14)
 
-ebb56(v502: i32, v503: i32):
+block56(v502: i32, v503: i32):
     v504 = copy v502
-    jump ebb90(v503, v504)
+    jump block90(v503, v504)
 
-ebb57:
-    jump ebb58(v33, v490)
+block57:
+    jump block58(v33, v490)
 
-ebb58(v505: i32, v506: i32):
+block58(v505: i32, v506: i32):
     v507 = iconst.i32 4
     v508 = iadd v505, v507
     v509 = iconst.i32 1
     v510 = iadd v505, v509
     v511 = iconst.i32 0x4ffe
     v512 = icmp uge v508, v511
-    brz v512, ebb59
-    jump ebb155
+    brz v512, block59
+    jump block155
 
-ebb155:
+block155:
     trap heap_oob
 
-ebb59:
+block59:
     v513 = uextend.i64 v508
     v514 = iadd_imm.i64 v0, -8
     v515 = load.i64 v514
@@ -892,41 +892,41 @@ ebb59:
     v518 = iconst.i32 0
     v519 = icmp eq v517, v518
     v520 = bint.i32 v519
-    brnz v520, ebb61(v14)
-    jump ebb156
+    brnz v520, block61(v14)
+    jump block156
 
-ebb156:
+block156:
     v521 = iconst.i32 8
     v522 = ishl.i32 v506, v521
     v523 = bor v522, v517
     v524 = icmp ne v523, v499
     v525 = bint.i32 v524
-    brnz v525, ebb58(v510, v523)
-    jump ebb60
+    brnz v525, block58(v510, v523)
+    jump block60
 
-ebb60:
-    jump ebb61(v14)
+block60:
+    jump block61(v14)
 
-ebb61(v526: i32):
+block61(v526: i32):
     v527 = iconst.i32 0
-    brnz.i32 v517, ebb62(v510)
-    jump ebb157
+    brnz.i32 v517, block62(v510)
+    jump block157
 
-ebb157:
+block157:
     v528 = copy v527
-    jump ebb62(v528)
+    jump block62(v528)
 
-ebb62(v529: i32):
+block62(v529: i32):
     v530 = copy v529
-    jump ebb90(v526, v530)
+    jump block90(v526, v530)
 
-ebb63:
+block63:
     v531 = isub.i32 v348, v359
     v532 = iconst.i32 1
     v533 = iadd v531, v532
-    jump ebb64(v359, v533)
+    jump block64(v359, v533)
 
-ebb64(v534: i32, v535: i32):
+block64(v534: i32, v535: i32):
     v536 = iconst.i32 1
     v537 = iadd.i32 v343, v536
     v538 = iconst.i32 0
@@ -938,49 +938,49 @@ ebb64(v534: i32, v535: i32):
     v544 = iadd v542, v543
     v545 = iconst.i32 0
     v546 = copy.i32 v350
-    jump ebb65(v350, v546, v349, v541, v348, v351, v544, v534, v545, v535, v343, v364, v537, v539, v362)
+    jump block65(v350, v546, v349, v541, v348, v351, v544, v534, v545, v535, v343, v364, v537, v539, v362)
 
-ebb65(v547: i32, v548: i32, v549: i32, v550: i32, v551: i32, v552: i32, v553: i32, v554: i32, v555: i32, v556: i32, v557: i32, v558: i32, v559: i32, v560: i32, v561: i32):
+block65(v547: i32, v548: i32, v549: i32, v550: i32, v551: i32, v552: i32, v553: i32, v554: i32, v555: i32, v556: i32, v557: i32, v558: i32, v559: i32, v560: i32, v561: i32):
     v562 = copy v556
     v563 = isub v547, v548
     v564 = icmp uge v563, v549
     v565 = bint.i32 v564
-    brnz v565, ebb67(v547)
-    jump ebb158
+    brnz v565, block67(v547)
+    jump block158
 
-ebb158:
+block158:
     v566 = iconst.i32 0
     v567 = call fn2(v0, v547, v566, v550)
-    brnz v567, ebb66
-    jump ebb159
+    brnz v567, block66
+    jump block159
 
-ebb159:
+block159:
     v568 = iadd v547, v550
-    jump ebb67(v568)
+    jump block67(v568)
 
-ebb66:
+block66:
     v569 = isub.i32 v567, v548
     v570 = icmp ult v569, v549
     v571 = bint.i32 v570
-    brnz v571, ebb89(v552)
-    jump ebb160
+    brnz v571, block89(v552)
+    jump block160
 
-ebb160:
+block160:
     v572 = copy.i32 v567
-    jump ebb67(v572)
+    jump block67(v572)
 
-ebb67(v573: i32):
+block67(v573: i32):
     v574 = iconst.i32 1
     v575 = iadd.i32 v548, v551
     v576 = iconst.i32 0x4ffe
     v577 = icmp uge v575, v576
-    brz v577, ebb68
-    jump ebb161
+    brz v577, block68
+    jump block161
 
-ebb161:
+block161:
     trap heap_oob
 
-ebb68:
+block68:
     v578 = uextend.i64 v575
     v579 = iadd_imm.i64 v0, -8
     v580 = load.i64 v579
@@ -998,13 +998,13 @@ ebb68:
     v592 = iadd v587, v591
     v593 = iconst.i32 0x4ffe
     v594 = icmp uge v592, v593
-    brz v594, ebb69
-    jump ebb162
+    brz v594, block69
+    jump block162
 
-ebb162:
+block162:
     trap heap_oob
 
-ebb69:
+block69:
     v595 = uextend.i64 v592
     v596 = iadd_imm.i64 v0, -8
     v597 = load.i64 v596
@@ -1014,22 +1014,22 @@ ebb69:
     v601 = iconst.i32 0
     v602 = icmp eq v600, v601
     v603 = bint.i32 v602
-    brnz v603, ebb74
-    jump ebb163
+    brnz v603, block74
+    jump block163
 
-ebb163:
+block163:
     v604 = iconst.i32 2
     v605 = ishl.i32 v582, v604
     v606 = iadd.i32 v552, v605
     v607 = iconst.i32 0x4ffe
     v608 = icmp uge v606, v607
-    brz v608, ebb70
-    jump ebb164
+    brz v608, block70
+    jump block164
 
-ebb164:
+block164:
     trap heap_oob
 
-ebb70:
+block70:
     v609 = uextend.i64 v606
     v610 = iadd_imm.i64 v0, -8
     v611 = load.i64 v610
@@ -1039,64 +1039,64 @@ ebb70:
     v615 = iconst.i32 -1
     v616 = icmp eq v614, v615
     v617 = bint.i32 v616
-    brnz v617, ebb75
-    jump ebb165
+    brnz v617, block75
+    jump block165
 
-ebb165:
+block165:
     v618 = iconst.i32 1
     v619 = iadd v614, v618
     v620 = icmp ult v619, v554
     v621 = bint.i32 v620
     v622 = copy.i32 v553
-    brnz v621, ebb71(v622)
-    jump ebb166
+    brnz v621, block71(v622)
+    jump block166
 
-ebb166:
+block166:
     v623 = copy v619
-    jump ebb71(v623)
+    jump block71(v623)
 
-ebb71(v624: i32):
+block71(v624: i32):
     v625 = copy v624
-    brnz.i32 v555, ebb72(v625)
-    jump ebb72(v619)
+    brnz.i32 v555, block72(v625)
+    jump block72(v619)
 
-ebb72(v626: i32):
-    brnz.i32 v562, ebb73(v626)
-    jump ebb73(v619)
+block72(v626: i32):
+    brnz.i32 v562, block73(v626)
+    jump block73(v619)
 
-ebb73(v627: i32):
+block73(v627: i32):
     v628 = copy.i32 v554
     v629 = copy.i32 v562
-    jump ebb87(v548, v627, v573, v549, v550, v551, v552, v553, v628, v629, v557, v558, v559, v560, v561)
+    jump block87(v548, v627, v573, v549, v550, v551, v552, v553, v628, v629, v557, v558, v559, v560, v561)
 
-ebb74:
+block74:
     v630 = copy.i32 v549
     v631 = copy.i32 v554
     v632 = copy.i32 v562
-    jump ebb87(v548, v630, v573, v549, v550, v551, v552, v553, v631, v632, v557, v558, v559, v560, v561)
+    jump block87(v548, v630, v573, v549, v550, v551, v552, v553, v631, v632, v557, v558, v559, v560, v561)
 
-ebb75:
+block75:
     v633 = icmp.i32 ugt v558, v555
     v634 = bint.i32 v633
     v635 = copy.i32 v558
-    brnz v634, ebb76(v635)
-    jump ebb167
+    brnz v634, block76(v635)
+    jump block167
 
-ebb167:
+block167:
     v636 = copy.i32 v555
-    jump ebb76(v636)
+    jump block76(v636)
 
-ebb76(v637: i32):
+block76(v637: i32):
     v638 = iadd.i32 v557, v637
     v639 = iconst.i32 0x4ffe
     v640 = icmp uge v638, v639
-    brz v640, ebb77
-    jump ebb168
+    brz v640, block77
+    jump block168
 
-ebb168:
+block168:
     trap heap_oob
 
-ebb77:
+block77:
     v641 = uextend.i64 v638
     v642 = iadd_imm.i64 v0, -8
     v643 = load.i64 v642
@@ -1105,27 +1105,27 @@ ebb77:
     v646 = iconst.i32 0
     v647 = icmp eq v645, v646
     v648 = bint.i32 v647
-    brnz v648, ebb82(v548, v549, v551, v552)
-    jump ebb169
+    brnz v648, block82(v548, v549, v551, v552)
+    jump block169
 
-ebb169:
+block169:
     v649 = iadd.i32 v548, v637
     v650 = iadd.i32 v559, v637
     v651 = iadd.i32 v560, v637
-    jump ebb78(v645, v649, v651, v650)
+    jump block78(v645, v649, v651, v650)
 
-ebb78(v652: i32, v653: i32, v654: i32, v655: i32):
+block78(v652: i32, v653: i32, v654: i32, v655: i32):
     v656 = iconst.i32 255
     v657 = band v652, v656
     v658 = iconst.i32 0x4ffe
     v659 = icmp uge v653, v658
-    brz v659, ebb79
-    jump ebb170
+    brz v659, block79
+    jump block170
 
-ebb170:
+block170:
     trap heap_oob
 
-ebb79:
+block79:
     v660 = uextend.i64 v653
     v661 = iadd_imm.i64 v0, -8
     v662 = load.i64 v661
@@ -1135,23 +1135,23 @@ ebb79:
     v666 = bint.i32 v665
     v667 = copy.i32 v554
     v668 = copy.i32 v562
-    brnz v666, ebb87(v548, v654, v573, v549, v550, v551, v552, v553, v667, v668, v557, v558, v559, v560, v561)
-    jump ebb171
+    brnz v666, block87(v548, v654, v573, v549, v550, v551, v552, v553, v667, v668, v557, v558, v559, v560, v561)
+    jump block171
 
-ebb171:
+block171:
     v669 = iconst.i32 1
     v670 = iadd.i32 v653, v669
     v671 = iconst.i32 1
     v672 = iadd.i32 v654, v671
     v673 = iconst.i32 0x4ffe
     v674 = icmp.i32 uge v655, v673
-    brz v674, ebb80
-    jump ebb172
+    brz v674, block80
+    jump block172
 
-ebb172:
+block172:
     trap heap_oob
 
-ebb80:
+block80:
     v675 = uextend.i64 v655
     v676 = iadd_imm.i64 v0, -8
     v677 = load.i64 v676
@@ -1159,33 +1159,33 @@ ebb80:
     v679 = uload8.i32 v678
     v680 = iconst.i32 1
     v681 = iadd.i32 v655, v680
-    brnz v679, ebb78(v679, v670, v672, v681)
-    jump ebb81
+    brnz v679, block78(v679, v670, v672, v681)
+    jump block81
 
-ebb81:
-    jump ebb82(v548, v549, v551, v552)
+block81:
+    jump block82(v548, v549, v551, v552)
 
-ebb82(v682: i32, v683: i32, v684: i32, v685: i32):
+block82(v682: i32, v683: i32, v684: i32, v685: i32):
     v686 = icmp.i32 ule v558, v555
     v687 = bint.i32 v686
-    brnz v687, ebb90(v685, v682)
-    jump ebb173
+    brnz v687, block90(v685, v682)
+    jump block173
 
-ebb173:
+block173:
     v688 = copy.i32 v561
-    jump ebb83(v688)
+    jump block83(v688)
 
-ebb83(v689: i32):
+block83(v689: i32):
     v690 = iadd.i32 v557, v689
     v691 = iconst.i32 0x4ffe
     v692 = icmp uge v690, v691
-    brz v692, ebb84
-    jump ebb174
+    brz v692, block84
+    jump block174
 
-ebb174:
+block174:
     trap heap_oob
 
-ebb84:
+block84:
     v693 = uextend.i64 v690
     v694 = iadd_imm.i64 v0, -8
     v695 = load.i64 v694
@@ -1194,13 +1194,13 @@ ebb84:
     v698 = iadd.i32 v682, v689
     v699 = iconst.i32 0x4ffe
     v700 = icmp uge v698, v699
-    brz v700, ebb85
-    jump ebb175
+    brz v700, block85
+    jump block175
 
-ebb175:
+block175:
     trap heap_oob
 
-ebb85:
+block85:
     v701 = uextend.i64 v698
     v702 = iadd_imm.i64 v0, -8
     v703 = load.i64 v702
@@ -1208,10 +1208,10 @@ ebb85:
     v705 = uload8.i32 v704
     v706 = icmp.i32 ne v697, v705
     v707 = bint.i32 v706
-    brnz v707, ebb86
-    jump ebb176
+    brnz v707, block86
+    jump block176
 
-ebb176:
+block176:
     v708 = icmp.i32 ule v689, v555
     v709 = bint.i32 v708
     v710 = iconst.i32 -1
@@ -1219,51 +1219,51 @@ ebb176:
     v712 = iconst.i32 0
     v713 = icmp eq v709, v712
     v714 = bint.i32 v713
-    brnz v714, ebb83(v711)
-    jump ebb90(v685, v682)
+    brnz v714, block83(v711)
+    jump block90(v685, v682)
 
-ebb86:
+block86:
     v715 = copy.i32 v554
     v716 = copy.i32 v562
-    jump ebb88(v682, v554, v573, v683, v550, v684, v685, v553, v715, v562, v716, v557, v558, v559, v560, v561)
+    jump block88(v682, v554, v573, v683, v550, v684, v685, v553, v715, v562, v716, v557, v558, v559, v560, v561)
 
-ebb87(v717: i32, v718: i32, v719: i32, v720: i32, v721: i32, v722: i32, v723: i32, v724: i32, v725: i32, v726: i32, v727: i32, v728: i32, v729: i32, v730: i32, v731: i32):
+block87(v717: i32, v718: i32, v719: i32, v720: i32, v721: i32, v722: i32, v723: i32, v724: i32, v725: i32, v726: i32, v727: i32, v728: i32, v729: i32, v730: i32, v731: i32):
     v732 = copy v718
     v733 = iconst.i32 0
-    jump ebb88(v717, v732, v719, v720, v721, v722, v723, v724, v725, v733, v726, v727, v728, v729, v730, v731)
+    jump block88(v717, v732, v719, v720, v721, v722, v723, v724, v725, v733, v726, v727, v728, v729, v730, v731)
 
-ebb88(v734: i32, v735: i32, v736: i32, v737: i32, v738: i32, v739: i32, v740: i32, v741: i32, v742: i32, v743: i32, v744: i32, v745: i32, v746: i32, v747: i32, v748: i32, v749: i32):
+block88(v734: i32, v735: i32, v736: i32, v737: i32, v738: i32, v739: i32, v740: i32, v741: i32, v742: i32, v743: i32, v744: i32, v745: i32, v746: i32, v747: i32, v748: i32, v749: i32):
     v750 = iadd v734, v735
     v751 = copy v742
     v752 = copy v743
     v753 = copy v744
-    jump ebb65(v736, v750, v737, v738, v739, v740, v741, v751, v752, v753, v745, v746, v747, v748, v749)
+    jump block65(v736, v750, v737, v738, v739, v740, v741, v751, v752, v753, v745, v746, v747, v748, v749)
 
-ebb89(v754: i32):
+block89(v754: i32):
     v755 = iconst.i32 0
-    jump ebb90(v754, v755)
+    jump block90(v754, v755)
 
-ebb90(v756: i32, v757: i32):
+block90(v756: i32, v757: i32):
     v758 = iconst.i32 0
     v759 = iconst.i32 1056
     v760 = iadd v756, v759
     v761 = iconst.i32 0x4ffe
     v762 = icmp uge v758, v761
-    brz v762, ebb91
-    jump ebb177
+    brz v762, block91
+    jump block177
 
-ebb177:
+block177:
     trap heap_oob
 
-ebb91:
+block91:
     v763 = uextend.i64 v758
     v764 = iadd_imm.i64 v0, -8
     v765 = load.i64 v764
     v766 = iadd v765, v763
     store.i32 v760, v766+4
-    jump ebb92(v757)
+    jump block92(v757)
 
-ebb92(v767: i32):
+block92(v767: i32):
     return v767
 }
 
@@ -1274,7 +1274,7 @@ function %musl(f64 [%xmm0], i64 vmctx [%rdi]) -> f64 [%xmm0] system_v {
     sig0 = (f64 [%xmm0], i32 [%rdi], i64 vmctx [%rsi]) -> f64 [%xmm0] system_v
     fn0 = u0:517 sig0
 
-ebb0(v0: f64, v1: i64):
+block0(v0: f64, v1: i64):
     v3 = iconst.i64 0
     v4 = iconst.i32 0
     v131 = iconst.i64 0
@@ -1306,39 +1306,39 @@ ebb0(v0: f64, v1: i64):
     v23 = iconst.i32 0x4086_232b
     v24 = icmp ult v22, v23
     v25 = bint.i32 v24
-    brnz v25, ebb10
-    jump ebb178
+    brnz v25, block10
+    jump block178
 
-ebb178:
+block178:
     v26 = iconst.i64 0x7fff_ffff_ffff_ffff
     v27 = band v14, v26
     v28 = iconst.i64 0x7ff0_0000_0000_0000
     v29 = icmp ule v27, v28
     v30 = bint.i32 v29
-    brnz v30, ebb9
-    jump ebb2(v12, v0)
+    brnz v30, block9
+    jump block2(v12, v0)
 
-ebb10:
+block10:
     v31 = iconst.i32 0x3fd6_2e43
     v32 = icmp.i32 ult v22, v31
     v33 = bint.i32 v32
-    brnz v33, ebb8
-    jump ebb179
+    brnz v33, block8
+    jump block179
 
-ebb179:
+block179:
     v34 = iconst.i32 0x3ff0_a2b2
     v35 = icmp.i32 uge v22, v34
     v36 = bint.i32 v35
-    brnz v36, ebb6
-    jump ebb180
+    brnz v36, block6
+    jump block180
 
-ebb180:
+block180:
     v37 = iconst.i32 1
     v38 = bxor.i32 v17, v37
     v39 = isub v38, v17
-    jump ebb5(v0, v39)
+    jump block5(v0, v39)
 
-ebb9:
+block9:
     v138 = iconst.i64 0x4086_2e42_fefa_39ef
     v40 = bitcast.f64 v138
     v41 = fcmp ge v40, v0
@@ -1348,39 +1348,39 @@ ebb9:
     v43 = bor v139, v140
     v44 = bint.i32 v43
     v45 = bor v42, v44
-    brnz v45, ebb7
-    jump ebb181
+    brnz v45, block7
+    jump block181
 
-ebb181:
+block181:
     v141 = iconst.i64 0x7fe0_0000_0000_0000
     v46 = bitcast.f64 v141
     v47 = fmul.f64 v0, v46
-    jump ebb2(v12, v47)
+    jump block2(v12, v47)
 
-ebb8:
+block8:
     v48 = iconst.i32 0x3e30_0000
     v49 = icmp.i32 ule v22, v48
     v50 = bint.i32 v49
-    brnz v50, ebb3
-    jump ebb182
+    brnz v50, block3
+    jump block182
 
-ebb182:
+block182:
     v51 = iconst.i32 0
     v142 = iconst.i64 0
     v52 = bitcast.f64 v142
     v178 = copy.f64 v0
-    jump ebb4(v0, v178, v52, v51)
+    jump block4(v0, v178, v52, v51)
 
-ebb7:
+block7:
     v143 = iconst.i64 0xc086_232b_dd7a_bcd2
     v53 = bitcast.f64 v143
     v54 = fcmp.f64 ge v0, v53
     v55 = bint.i32 v54
     v56 = bor v55, v44
-    brnz v56, ebb6
-    jump ebb183
+    brnz v56, block6
+    jump block183
 
-ebb183:
+block183:
     v144 = iconst.i64 0xb6a0_0000_0000_0000
     v57 = bitcast.f64 v144
     v58 = fdiv v57, v0
@@ -1396,10 +1396,10 @@ ebb183:
     v62 = bitcast.f64 v149
     v63 = fcmp gt v62, v0
     v64 = bint.i32 v63
-    brnz v64, ebb2(v12, v61)
-    jump ebb6
+    brnz v64, block2(v12, v61)
+    jump block6
 
-ebb6:
+block6:
     v150 = iconst.i64 0x3ff7_1547_652b_82fe
     v66 = bitcast.f64 v150
     v67 = fmul.f64 v0, v66
@@ -1416,34 +1416,34 @@ ebb6:
     v76 = x86_cvtt2si.i32 v75
     v158 = iconst.i32 0x8000_0000
     v154 = icmp ne v76, v158
-    brnz v154, ebb11
-    jump ebb184
+    brnz v154, block11
+    jump block184
 
-ebb184:
+block184:
     v155 = fcmp uno v75, v75
-    brz v155, ebb12
-    jump ebb185
+    brz v155, block12
+    jump block185
 
-ebb185:
+block185:
     trap bad_toint
 
-ebb12:
+block12:
     v159 = iconst.i64 0xc1e0_0000_0020_0000
     v156 = bitcast.f64 v159
     v157 = fcmp ge v156, v75
-    brz v157, ebb13
-    jump ebb186
+    brz v157, block13
+    jump block186
 
-ebb186:
+block186:
     trap int_ovf
 
-ebb13:
-    jump ebb11
+block13:
+    jump block11
 
-ebb11:
-    jump ebb5(v0, v76)
+block11:
+    jump block5(v0, v76)
 
-ebb5(v77: f64, v78: i32):
+block5(v77: f64, v78: i32):
     v79 = fcvt_from_sint.f64 v78
     v160 = iconst.i64 0xbfe6_2e42_fee0_0000
     v80 = bitcast.f64 v160
@@ -1453,14 +1453,14 @@ ebb5(v77: f64, v78: i32):
     v83 = bitcast.f64 v161
     v84 = fmul v79, v83
     v85 = fsub v82, v84
-    jump ebb4(v82, v85, v84, v78)
+    jump block4(v82, v85, v84, v78)
 
-ebb4(v86: f64, v87: f64, v108: f64, v113: i32):
+block4(v86: f64, v87: f64, v108: f64, v113: i32):
     v88 = fmul v87, v87
     v162 = iconst.i64 0x3e66_3769_72be_a4d0
     v89 = bitcast.f64 v162
     v90 = fmul v88, v89
-    v163 = iconst.i64 0xbebb_bd41_c5d2_6bf1
+    v163 = iconst.i64 0xbeeb_bd41_c5d2_6bf1
     v91 = bitcast.f64 v163
     v92 = fadd v90, v91
     v93 = fmul v88, v92
@@ -1490,14 +1490,14 @@ ebb4(v86: f64, v87: f64, v108: f64, v113: i32):
     v169 = iconst.i32 0
     v114 = icmp eq v113, v169
     v115 = bint.i32 v114
-    brnz v115, ebb2(v12, v112)
-    jump ebb187
+    brnz v115, block2(v12, v112)
+    jump block187
 
-ebb187:
+block187:
     v116 = call fn0(v112, v113, v1)
-    jump ebb2(v12, v116)
+    jump block2(v12, v116)
 
-ebb3:
+block3:
     v170 = iconst.i64 0x7fe0_0000_0000_0000
     v117 = bitcast.f64 v170
     v118 = fadd.f64 v0, v117
@@ -1509,9 +1509,9 @@ ebb3:
     v174 = iconst.i64 0x3ff0_0000_0000_0000
     v120 = bitcast.f64 v174
     v121 = fadd.f64 v0, v120
-    jump ebb2(v12, v121)
+    jump block2(v12, v121)
 
-ebb2(v123: i32, v130: f64):
+block2(v123: i32, v130: f64):
     v122 = iconst.i32 0
     v127 = iconst.i32 16
     v128 = iadd v123, v127
@@ -1520,8 +1520,8 @@ ebb2(v123: i32, v130: f64):
     v177 = load.i64 v176
     v129 = iadd v177, v175
     store v128, v129+4
-    jump ebb1(v130)
+    jump block1(v130)
 
-ebb1(v2: f64):
+block1(v2: f64):
     return v2
 }
diff --git a/cranelift/filetests/filetests/regalloc/coalescing-216.clif b/cranelift/filetests/filetests/regalloc/coalescing-216.clif
index b8de70160a..020ced084b 100644
--- a/cranelift/filetests/filetests/regalloc/coalescing-216.clif
+++ b/cranelift/filetests/filetests/regalloc/coalescing-216.clif
@@ -5,83 +5,83 @@ target x86_64 haswell
 ;
 ; The (old) coalescer creates a virtual register with two identical values.
 function %pr216(i32 [%rdi], i64 vmctx [%rsi]) -> i64 [%rax] system_v {
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v3 = iconst.i64 0
     v5 = iconst.i32 0
-    brz v5, ebb3(v3)
-    jump ebb4(v3, v3)
+    brz v5, block3(v3)
+    jump block4(v3, v3)
 
-ebb4(v11: i64, v29: i64):
+block4(v11: i64, v29: i64):
     v6 = iconst.i32 0
-    brz v6, ebb14
-    jump ebb15
+    brz v6, block14
+    jump block15
 
-ebb15:
+block15:
     v9 = iconst.i32 -17
     v12 = iconst.i32 0xffff_ffff_ffff_8000
-    jump ebb9(v12)
+    jump block9(v12)
 
-ebb9(v10: i32):
-    brnz v10, ebb8(v9, v11, v11)
-    jump ebb16
+block9(v10: i32):
+    brnz v10, block8(v9, v11, v11)
+    jump block16
 
-ebb16:
-    brz.i32 v9, ebb13
-    jump ebb17
+block16:
+    brz.i32 v9, block13
+    jump block17
 
-ebb17:
+block17:
     v13 = iconst.i32 0
-    brnz v13, ebb6(v11, v11)
-    jump ebb18
+    brnz v13, block6(v11, v11)
+    jump block18
 
-ebb18:
+block18:
     v14 = iconst.i32 0
-    brz v14, ebb12
-    jump ebb11
+    brz v14, block12
+    jump block11
 
-ebb12:
-    jump ebb4(v11, v11)
+block12:
+    jump block4(v11, v11)
 
-ebb11:
-    jump ebb10(v11)
+block11:
+    jump block10(v11)
 
-ebb13:
+block13:
     v15 = iconst.i64 1
-    jump ebb10(v15)
+    jump block10(v15)
 
-ebb10(v21: i64):
+block10(v21: i64):
     v16 = iconst.i32 0
-    brnz v16, ebb6(v21, v11)
-    jump ebb19
+    brnz v16, block6(v21, v11)
+    jump block19
 
-ebb19:
+block19:
     v17 = iconst.i32 0xffff_ffff_ffff_9f35
-    jump ebb8(v17, v21, v11)
+    jump block8(v17, v21, v11)
 
-ebb8(v8: i32, v23: i64, v28: i64):
-    jump ebb7(v8, v23, v28)
+block8(v8: i32, v23: i64, v28: i64):
+    jump block7(v8, v23, v28)
 
-ebb14:
+block14:
     v18 = iconst.i32 0
-    jump ebb7(v18, v11, v29)
+    jump block7(v18, v11, v29)
 
-ebb7(v7: i32, v22: i64, v27: i64):
-    jump ebb6(v22, v27)
+block7(v7: i32, v22: i64, v27: i64):
+    jump block6(v22, v27)
 
-ebb6(v20: i64, v25: i64):
+block6(v20: i64, v25: i64):
     v19 = iconst.i32 0xffc7
-    brnz v19, ebb4(v20, v25)
-    jump ebb5
+    brnz v19, block4(v20, v25)
+    jump block5
 
-ebb5:
-    jump ebb3(v25)
+block5:
+    jump block3(v25)
 
-ebb3(v24: i64):
-    jump ebb2(v24)
+block3(v24: i64):
+    jump block2(v24)
 
-ebb2(v4: i64):
-    jump ebb1(v4)
+block2(v4: i64):
+    jump block1(v4)
 
-ebb1(v2: i64):
+block1(v2: i64):
     return v2
 }
diff --git a/cranelift/filetests/filetests/regalloc/coloring-227.clif b/cranelift/filetests/filetests/regalloc/coloring-227.clif
index 0418924dfb..52b36d4c70 100644
--- a/cranelift/filetests/filetests/regalloc/coloring-227.clif
+++ b/cranelift/filetests/filetests/regalloc/coloring-227.clif
@@ -5,81 +5,81 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
     gv0 = vmctx
     heap0 = static gv0, min 0, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-                          ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i64):
+                          block0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i64):
 [RexOp1pu_id#b8]              v5 = iconst.i32 0
 [RexOp1pu_id#b8]              v6 = iconst.i32 0
-[RexOp1tjccb#74]              brz v6, ebb10
-[Op1jmpb#eb]                  jump ebb3(v5, v5, v5, v5, v5, v5, v0, v1, v2, v3)
+[RexOp1tjccb#74]              brz v6, block10
+[Op1jmpb#eb]                  jump block3(v5, v5, v5, v5, v5, v5, v0, v1, v2, v3)
 
-                          ebb3(v15: i32, v17: i32, v25: i32, v31: i32, v40: i32, v47: i32, v54: i32, v61: i32, v68: i32, v75: i32):
-[Op1jmpb#eb]                  jump ebb6
+                          block3(v15: i32, v17: i32, v25: i32, v31: i32, v40: i32, v47: i32, v54: i32, v61: i32, v68: i32, v75: i32):
+[Op1jmpb#eb]                  jump block6
 
-                          ebb6:
+                          block6:
 [RexOp1pu_id#b8]              v8 = iconst.i32 0
-[RexOp1tjccb#75]              brnz v8, ebb5
-[Op1jmpb#eb]                  jump ebb20
+[RexOp1tjccb#75]              brnz v8, block5
+[Op1jmpb#eb]                  jump block20
 
-                          ebb20:
+                          block20:
 [RexOp1pu_id#b8]              v9 = iconst.i32 0
 [RexOp1pu_id#b8]              v11 = iconst.i32 0
 [DynRexOp1icscc#39]           v12 = icmp.i32 eq v15, v11
 [RexOp2urm_noflags#4b6]       v13 = bint.i32 v12
 [DynRexOp1rr#21]              v14 = band v9, v13
-[RexOp1tjccb#75]              brnz v14, ebb6
-[Op1jmpb#eb]                  jump ebb7
+[RexOp1tjccb#75]              brnz v14, block6
+[Op1jmpb#eb]                  jump block7
 
-                          ebb7:
-[RexOp1tjccb#74]              brz.i32 v17, ebb8
-[Op1jmpb#eb]                  jump ebb17
+                          block7:
+[RexOp1tjccb#74]              brz.i32 v17, block8
+[Op1jmpb#eb]                  jump block17
 
-                          ebb17:
+                          block17:
 [RexOp1pu_id#b8]              v18 = iconst.i32 0
-[RexOp1tjccb#74]              brz v18, ebb9
-[Op1jmpb#eb]                  jump ebb16
+[RexOp1tjccb#74]              brz v18, block9
+[Op1jmpb#eb]                  jump block16
 
-                          ebb16:
+                          block16:
 [RexOp1pu_id#b8]              v21 = iconst.i32 0
 [RexOp1umr#89]                v79 = uextend.i64 v5
 [DynRexOp1r_ib#8083]          v80 = iadd_imm.i64 v4, 0
 [RexOp1ld#808b]               v81 = load.i64 v80
 [DynRexOp1rr#8001]            v22 = iadd v81, v79
 [RexMp1st#189]                istore16 v21, v22
-[Op1jmpb#eb]                  jump ebb9
+[Op1jmpb#eb]                  jump block9
 
-                          ebb9:
-[Op1jmpb#eb]                  jump ebb8
+                          block9:
+[Op1jmpb#eb]                  jump block8
 
-                          ebb8:
+                          block8:
 [RexOp1pu_id#b8]              v27 = iconst.i32 3
 [RexOp1pu_id#b8]              v28 = iconst.i32 4
 [DynRexOp1rr#09]              v35 = bor.i32 v31, v13
-[RexOp1tjccb#75]              brnz v35, ebb15(v27)
-[Op1jmpb#eb]                  jump ebb15(v28)
+[RexOp1tjccb#75]              brnz v35, block15(v27)
+[Op1jmpb#eb]                  jump block15(v28)
 
-                          ebb15(v36: i32):
-[Op1jmpb#eb]                  jump ebb3(v25, v36, v25, v31, v40, v47, v54, v61, v68, v75)
+                          block15(v36: i32):
+[Op1jmpb#eb]                  jump block3(v25, v36, v25, v31, v40, v47, v54, v61, v68, v75)
 
-                          ebb5:
-[Op1jmpb#eb]                  jump ebb4
+                          block5:
+[Op1jmpb#eb]                  jump block4
 
-                          ebb4:
-[Op1jmpb#eb]                  jump ebb2(v40, v47, v54, v61, v68, v75)
+                          block4:
+[Op1jmpb#eb]                  jump block2(v40, v47, v54, v61, v68, v75)
 
-                          ebb10:
+                          block10:
 [RexOp1pu_id#b8]              v43 = iconst.i32 0
-[Op1jmpb#eb]                  jump ebb2(v43, v5, v0, v1, v2, v3)
+[Op1jmpb#eb]                  jump block2(v43, v5, v0, v1, v2, v3)
 
-                          ebb2(v7: i32, v45: i32, v52: i32, v59: i32, v66: i32, v73: i32):
+                          block2(v7: i32, v45: i32, v52: i32, v59: i32, v66: i32, v73: i32):
 [RexOp1pu_id#b8]              v44 = iconst.i32 0
-[RexOp1tjccb#74]              brz v44, ebb12
-[Op1jmpb#eb]                  jump ebb18
+[RexOp1tjccb#74]              brz v44, block12
+[Op1jmpb#eb]                  jump block18
 
-                          ebb18:
+                          block18:
 [RexOp1pu_id#b8]              v50 = iconst.i32 11
-[RexOp1tjccb#74]              brz v50, ebb14
-[Op1jmpb#eb]                  jump ebb19
+[RexOp1tjccb#74]              brz v50, block14
+[Op1jmpb#eb]                  jump block19
 
-                          ebb19:
+                          block19:
 [RexOp1umr#89]                v82 = uextend.i64 v52
 [DynRexOp1r_ib#8083]          v83 = iadd_imm.i64 v4, 0
 [RexOp1ld#808b]               v84 = load.i64 v83
@@ -91,25 +91,25 @@ function %pr227(i32 [%rdi], i32 [%rsi], i32 [%rdx], i32 [%rcx], i64 vmctx [%r8])
 [DynRexOp1rr#8001]            v64 = iadd v87, v85
 [RexOp1st#88]                 istore8 v59, v64
 [RexOp1pu_id#b8]              v65 = iconst.i32 0
-[Op1jmpb#eb]                  jump ebb13(v65)
+[Op1jmpb#eb]                  jump block13(v65)
 
-                          ebb14:
-[Op1jmpb#eb]                  jump ebb13(v66)
+                          block14:
+[Op1jmpb#eb]                  jump block13(v66)
 
-                          ebb13(v51: i32):
+                          block13(v51: i32):
 [RexOp1umr#89]                v88 = uextend.i64 v45
 [DynRexOp1r_ib#8083]          v89 = iadd_imm.i64 v4, 0
 [RexOp1ld#808b]               v90 = load.i64 v89
 [DynRexOp1rr#8001]            v71 = iadd v90, v88
 [RexOp1st#89]                 store v51, v71
-[Op1jmpb#eb]                  jump ebb12
+[Op1jmpb#eb]                  jump block12
 
-                          ebb12:
-[Op1jmpb#eb]                  jump ebb11
+                          block12:
+[Op1jmpb#eb]                  jump block11
 
-                          ebb11:
-[Op1jmpb#eb]                  jump ebb1
+                          block11:
+[Op1jmpb#eb]                  jump block1
 
-                          ebb1:
+                          block1:
 [Op1ret#c3]                   return
 }
diff --git a/cranelift/filetests/filetests/regalloc/constraints.clif b/cranelift/filetests/filetests/regalloc/constraints.clif
index a0478baae9..60cd731ed8 100644
--- a/cranelift/filetests/filetests/regalloc/constraints.clif
+++ b/cranelift/filetests/filetests/regalloc/constraints.clif
@@ -6,7 +6,7 @@ target i686
 
 ; Tied operands, both are killed at instruction.
 function %tied_easy() -> i32 {
-ebb0:
+block0:
     v0 = iconst.i32 12
     v1 = iconst.i32 13
     ; not: copy
@@ -17,7 +17,7 @@ ebb0:
 
 ; Tied operand is live after instruction.
 function %tied_alive() -> i32 {
-ebb0:
+block0:
     v0 = iconst.i32 12
     v1 = iconst.i32 13
     ; check: $(v0c=$V) = copy v0
@@ -30,7 +30,7 @@ ebb0:
 
 ; Fixed register constraint.
 function %fixed_op() -> i32 {
-ebb0:
+block0:
     ; check: ,%rax]
     ; sameln: v0 = iconst.i32 12
     v0 = iconst.i32 12
@@ -43,7 +43,7 @@ ebb0:
 
 ; Fixed register constraint twice.
 function %fixed_op_twice() -> i32 {
-ebb0:
+block0:
     ; check: ,%rax]
     ; sameln: v0 = iconst.i32 12
     v0 = iconst.i32 12
@@ -60,7 +60,7 @@ ebb0:
 
 ; Tied use of a diverted register.
 function %fixed_op_twice() -> i32 {
-ebb0:
+block0:
     ; check: ,%rax]
     ; sameln: v0 = iconst.i32 12
     v0 = iconst.i32 12
diff --git a/cranelift/filetests/filetests/regalloc/fallthrough-return.clif b/cranelift/filetests/filetests/regalloc/fallthrough-return.clif
index 557710eb5a..58ec61f0d8 100644
--- a/cranelift/filetests/filetests/regalloc/fallthrough-return.clif
+++ b/cranelift/filetests/filetests/regalloc/fallthrough-return.clif
@@ -6,7 +6,7 @@ target x86_64
 function %foo() -> f64 {
   fn0 = %bar()
 
-ebb0:
+block0:
   v0 = f64const 0.0
   call fn0()
   fallthrough_return v0
@@ -16,7 +16,7 @@ ebb0:
 function %foo() -> f64 {
   fn0 = %bar() -> f64, f64
 
-ebb0:
+block0:
   v0, v1 = call fn0()
   fallthrough_return v1
 }
diff --git a/cranelift/filetests/filetests/regalloc/ghost-param.clif b/cranelift/filetests/filetests/regalloc/ghost-param.clif
index f2a1883a0d..d51f4a7f72 100644
--- a/cranelift/filetests/filetests/regalloc/ghost-param.clif
+++ b/cranelift/filetests/filetests/regalloc/ghost-param.clif
@@ -1,45 +1,45 @@
 test regalloc
 target x86_64 haswell
 
-; This test case would create an EBB parameter that was a ghost value.
+; This test case would create a block parameter that was a ghost value.
 ; The coalescer would insert a copy of the ghost value, leading to verifier errors.
 ;
-; We don't allow EBB parameters to be ghost values any longer.
+; We don't allow block parameters to be ghost values any longer.
 ;
 ; Test case by binaryen fuzzer!
 
 function %pr215(i64 vmctx [%rdi]) system_v {
-ebb0(v0: i64):
+block0(v0: i64):
     v10 = iconst.i64 0
     v1 = bitcast.f64 v10
-    jump ebb5(v1)
+    jump block5(v1)
 
-ebb5(v9: f64):
+block5(v9: f64):
     v11 = iconst.i64 0xffff_ffff_ff9a_421a
     v4 = bitcast.f64 v11
     v6 = iconst.i32 0
     v7 = iconst.i32 1
-    brnz v7, ebb4(v6)
-    jump ebb8
+    brnz v7, block4(v6)
+    jump block8
 
-ebb8:
+block8:
     v8 = iconst.i32 0
-    jump ebb7(v8)
+    jump block7(v8)
 
-ebb7(v5: i32):
-    brnz v5, ebb3(v4)
-    jump ebb5(v4)
+block7(v5: i32):
+    brnz v5, block3(v4)
+    jump block5(v4)
 
-ebb4(v3: i32):
-    brnz v3, ebb2
-    jump ebb3(v9)
+block4(v3: i32):
+    brnz v3, block2
+    jump block3(v9)
 
-ebb3(v2: f64):
-    jump ebb2
+block3(v2: f64):
+    jump block2
 
-ebb2:
-    jump ebb1
+block2:
+    jump block1
 
-ebb1:
+block1:
     return
 }
diff --git a/cranelift/filetests/filetests/regalloc/global-constraints.clif b/cranelift/filetests/filetests/regalloc/global-constraints.clif
index 11a3dbef2c..8149b9bae6 100644
--- a/cranelift/filetests/filetests/regalloc/global-constraints.clif
+++ b/cranelift/filetests/filetests/regalloc/global-constraints.clif
@@ -7,19 +7,19 @@ target i686
 ; The icmp_imm instrutions write their b1 result to the ABCD register class on
 ; 32-bit x86. So if we define 5 live values, they can't all fit.
 function %global_constraints(i32) {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = icmp_imm eq v0, 1
     v2 = icmp_imm ugt v0, 2
     v3 = icmp_imm sle v0, 3
     v4 = icmp_imm ne v0, 4
     v5 = icmp_imm sge v0, 5
-    brnz v5, ebb1
-    jump ebb2
+    brnz v5, block1
+    jump block2
 
-ebb2:
+block2:
     return
 
-ebb1:
+block1:
     ; Make sure v1-v5 are live in.
     v10 = band v1, v2
     v11 = bor v3, v4
diff --git a/cranelift/filetests/filetests/regalloc/global-fixed.clif b/cranelift/filetests/filetests/regalloc/global-fixed.clif
index eb8e23d7af..851f012492 100644
--- a/cranelift/filetests/filetests/regalloc/global-fixed.clif
+++ b/cranelift/filetests/filetests/regalloc/global-fixed.clif
@@ -2,15 +2,15 @@ test regalloc
 target x86_64 haswell
 
 function %foo() system_v {
-ebb4:
+block4:
     v3 = iconst.i32 0
-    jump ebb3
+    jump block3
 
-ebb3:
+block3:
     v9 = udiv v3, v3
-    jump ebb1
+    jump block1
 
-ebb1:
+block1:
     v19 = iadd.i32 v9, v9
-    jump ebb3
+    jump block3
 }
diff --git a/cranelift/filetests/filetests/regalloc/gpr-deref-safe-335.clif b/cranelift/filetests/filetests/regalloc/gpr-deref-safe-335.clif
index 8e8b2260cf..04e9cc54fb 100644
--- a/cranelift/filetests/filetests/regalloc/gpr-deref-safe-335.clif
+++ b/cranelift/filetests/filetests/regalloc/gpr-deref-safe-335.clif
@@ -2,43 +2,43 @@ test regalloc
 target x86_64
 
 function u0:587() fast {
-ebb0:
+block0:
     v97 = iconst.i32 0
     v169 = iconst.i32 0
     v1729 = iconst.i32 0
-    jump ebb100(v97, v97, v97, v97, v97)
+    jump block100(v97, v97, v97, v97, v97)
 
-ebb100(v1758: i32, v1784: i32, v1845: i32, v1856: i32, v1870: i32):
+block100(v1758: i32, v1784: i32, v1845: i32, v1856: i32, v1870: i32):
     v1762 = iconst.i32 0
     v1769 = iconst.i32 0
     v1774 = iconst.i32 0
     v1864 = iconst.i32 0
     v1897 = iconst.i32 0
-    jump ebb102(v1774, v1784, v1845, v1856, v1870, v1758, v1762, v169, v1729, v97, v169, v169, v169, v169)
+    jump block102(v1774, v1784, v1845, v1856, v1870, v1758, v1762, v169, v1729, v97, v169, v169, v169, v169)
 
-ebb102(v1785: i32, v1789: i32, v1843: i32, v1854: i32, v1868: i32, v1882: i32, v1890: i32, v1901: i32, v1921: i32, v1933: i32, v2058: i32, v2124: i32, v2236: i32, v2366: i32):
+block102(v1785: i32, v1789: i32, v1843: i32, v1854: i32, v1868: i32, v1882: i32, v1890: i32, v1901: i32, v1921: i32, v1933: i32, v2058: i32, v2124: i32, v2236: i32, v2366: i32):
     v1929 = iconst.i32 0
     v1943 = iconst.i32 0
     v1949 = iconst.i32 0
-    jump ebb123(v1897, v1769)
+    jump block123(v1897, v1769)
 
-ebb123(v1950: i32, v1979: i32):
+block123(v1950: i32, v1979: i32):
     v1955 = iconst.i32 0
-    brz v1955, ebb125
-    jump ebb122(v1929, v1843, v1864, v2058, v1882, v1897, v1943, v1868, v2124, v1901)
+    brz v1955, block125
+    jump block122(v1929, v1843, v1864, v2058, v1882, v1897, v1943, v1868, v2124, v1901)
 
-ebb125:
+block125:
     v1961 = iadd_imm.i32 v1949, 0
     v1952 = iconst.i32 0
     v1962 = iconst.i64 0
     v1963 = load.i32 v1962
-    brz v1963, ebb123(v1952, v1961)
-    jump ebb127
+    brz v1963, block123(v1952, v1961)
+    jump block127
 
-ebb127:
+block127:
     v1966 = iconst.i32 0
-    jump ebb122(v1963, v1966, v1966, v1966, v1966, v1966, v1966, v1966, v1966, v1966)
+    jump block122(v1963, v1966, v1966, v1966, v1966, v1966, v1966, v1966, v1966, v1966)
 
-ebb122(v1967: i32, v1971: i32, v1972: i32, v1978: i32, v2032: i32, v2041: i32, v2053: i32, v2076: i32, v2085: i32, v2096: i32):
+block122(v1967: i32, v1971: i32, v1972: i32, v1978: i32, v2032: i32, v2041: i32, v2053: i32, v2076: i32, v2085: i32, v2096: i32):
     trap user0
 }
diff --git a/cranelift/filetests/filetests/regalloc/infinite-interference.clif b/cranelift/filetests/filetests/regalloc/infinite-interference.clif
index bce607b7f6..b7a7736405 100644
--- a/cranelift/filetests/filetests/regalloc/infinite-interference.clif
+++ b/cranelift/filetests/filetests/regalloc/infinite-interference.clif
@@ -7,31 +7,31 @@ target riscv32
 ; resolve that conflict since v1 will just interfere with the inserted copy too.
 
 ;function %c1(i32) -> i32 {
-;ebb0(v0: i32):
+;block0(v0: i32):
 ;    v1 = iadd_imm v0, 1
 ;    v2 = iconst.i32 1
-;    brz v1, ebb1(v2)
-;    jump ebb2
+;    brz v1, block1(v2)
+;    jump block2
 ;
-;ebb1(v3: i32):
+;block1(v3: i32):
 ;    return v3
 ;
-;ebb2:
-;    jump ebb1(v1)
+;block2:
+;    jump block1(v1)
 ;}
 
 ; Same thing with v1 and v2 swapped to reverse the order of definitions.
 
 function %c2(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iadd_imm v0, 1
     v2 = iconst.i32 1
-    brz v2, ebb1(v1)
-    jump ebb2
+    brz v2, block1(v1)
+    jump block2
 
-ebb1(v3: i32):
+block1(v3: i32):
     return v3
 
-ebb2:
-    jump ebb1(v2)
+block2:
+    jump block1(v2)
 }
diff --git a/cranelift/filetests/filetests/regalloc/iterate.clif b/cranelift/filetests/filetests/regalloc/iterate.clif
index 347dbc5f29..2c7d691765 100644
--- a/cranelift/filetests/filetests/regalloc/iterate.clif
+++ b/cranelift/filetests/filetests/regalloc/iterate.clif
@@ -2,7 +2,7 @@ test regalloc
 target x86_64 haswell
 
 function u0:9(i64 [%rdi], f32 [%xmm0], f64 [%xmm1], i32 [%rsi], i32 [%rdx], i64 vmctx [%r14]) -> i64 [%rax] baldrdash_system_v {
-ebb0(v0: i64, v1: f32, v2: f64, v3: i32, v4: i32, v5: i64):
+block0(v0: i64, v1: f32, v2: f64, v3: i32, v4: i32, v5: i64):
     v32 = iconst.i32 0
     v6 = bitcast.f32 v32
     v7 = iconst.i64 0
@@ -19,23 +19,23 @@ ebb0(v0: i64, v1: f32, v2: f64, v3: i32, v4: i32, v5: i64):
     v14 = bitcast.f64 v36
     v44 = iconst.i64 0
     v37 = icmp slt v0, v44
-    brnz v37, ebb2
-    jump ebb11
+    brnz v37, block2
+    jump block11
 
-ebb11:
+block11:
     v38 = fcvt_from_sint.f64 v0
-    jump ebb3(v38)
+    jump block3(v38)
 
-ebb2:
+block2:
     v45 = iconst.i32 1
     v39 = ushr.i64 v0, v45
     v40 = band_imm.i64 v0, 1
     v41 = bor v39, v40
     v42 = fcvt_from_sint.f64 v41
     v43 = fadd v42, v42
-    jump ebb3(v43)
+    jump block3(v43)
 
-ebb3(v15: f64):
+block3(v15: f64):
     v16 = fpromote.f64 v9
     v46 = uextend.i64 v10
     v17 = fcvt_from_sint.f64 v46
@@ -43,42 +43,42 @@ ebb3(v15: f64):
     v19 = fpromote.f64 v12
     v54 = iconst.i64 0
     v47 = icmp.i64 slt v13, v54
-    brnz v47, ebb4
-    jump ebb12
+    brnz v47, block4
+    jump block12
 
-ebb12:
+block12:
     v48 = fcvt_from_sint.f64 v13
-    jump ebb5(v48)
+    jump block5(v48)
 
-ebb4:
+block4:
     v55 = iconst.i32 1
     v49 = ushr.i64 v13, v55
     v50 = band_imm.i64 v13, 1
     v51 = bor v49, v50
     v52 = fcvt_from_sint.f64 v51
     v53 = fadd v52, v52
-    jump ebb5(v53)
+    jump block5(v53)
 
-ebb5(v20: f64):
+block5(v20: f64):
     v63 = iconst.i64 0
     v56 = icmp.i64 slt v7, v63
-    brnz v56, ebb6
-    jump ebb13
+    brnz v56, block6
+    jump block13
 
-ebb13:
+block13:
     v57 = fcvt_from_sint.f64 v7
-    jump ebb7(v57)
+    jump block7(v57)
 
-ebb6:
+block6:
     v64 = iconst.i32 1
     v58 = ushr.i64 v7, v64
     v59 = band_imm.i64 v7, 1
     v60 = bor v58, v59
     v61 = fcvt_from_sint.f64 v60
     v62 = fadd v61, v61
-    jump ebb7(v62)
+    jump block7(v62)
 
-ebb7(v21: f64):
+block7(v21: f64):
     v22 = fadd v21, v14
     v23 = fadd.f64 v20, v22
     v24 = fadd.f64 v19, v23
@@ -90,34 +90,34 @@ ebb7(v21: f64):
     v30 = x86_cvtt2si.i64 v29
     v69 = iconst.i64 0x8000_0000_0000_0000
     v65 = icmp ne v30, v69
-    brnz v65, ebb8
-    jump ebb15
+    brnz v65, block8
+    jump block15
 
-ebb15:
+block15:
     v66 = fcmp uno v29, v29
-    brz v66, ebb9
-    jump ebb16
+    brz v66, block9
+    jump block16
 
-ebb16:
+block16:
     trap bad_toint
 
-ebb9:
+block9:
     v70 = iconst.i64 0xc3e0_0000_0000_0000
     v67 = bitcast.f64 v70
     v68 = fcmp gt v67, v29
-    brz v68, ebb10
-    jump ebb17
+    brz v68, block10
+    jump block17
 
-ebb17:
+block17:
     trap int_ovf
 
-ebb10:
-    jump ebb8
+block10:
+    jump block8
 
-ebb8:
-    jump ebb1(v30)
+block8:
+    jump block1(v30)
 
-ebb1(v31: i64):
+block1(v31: i64):
     return v31
 }
 
@@ -126,7 +126,7 @@ function u0:26(i64 vmctx [%r14]) -> i64 [%rax] baldrdash_system_v {
     gv0 = iadd_imm.i64 gv1, 48
     sig0 = (i32 [%rdi], i64 [%rsi], i64 vmctx [%r14], i64 sigid [%rbx]) -> i64 [%rax] baldrdash_system_v
 
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = iconst.i32 32
     v2 = iconst.i64 64
     v3 = iconst.i32 9
@@ -135,30 +135,30 @@ ebb0(v0: i64):
     v6 = load.i32 v5
     v7 = icmp uge v3, v6
     ; If we're unlucky, there are no ABCD registers available for v7 at this branch.
-    brz v7, ebb2
-    jump ebb4
+    brz v7, block2
+    jump block4
 
-ebb4:
+block4:
     trap oob
 
-ebb2:
+block2:
     v8 = load.i64 v5+8
     v9 = uextend.i64 v3
     v16 = iconst.i64 16
     v10 = imul v9, v16
     v11 = iadd v8, v10
     v12 = load.i64 v11
-    brnz v12, ebb3
-    jump ebb5
+    brnz v12, block3
+    jump block5
 
-ebb5:
+block5:
     trap icall_null
 
-ebb3:
+block3:
     v13 = load.i64 v11+8
     v14 = call_indirect.i64 sig0, v12(v1, v2, v13, v4)
-    jump ebb1(v14)
+    jump block1(v14)
 
-ebb1(v15: i64):
+block1(v15: i64):
     return v15
 }
diff --git a/cranelift/filetests/filetests/regalloc/multi-constraints.clif b/cranelift/filetests/filetests/regalloc/multi-constraints.clif
index d0b6f7faf0..b01be532f8 100644
--- a/cranelift/filetests/filetests/regalloc/multi-constraints.clif
+++ b/cranelift/filetests/filetests/regalloc/multi-constraints.clif
@@ -14,16 +14,16 @@ target x86_64 haswell
 ;
 ; - The same value used for a tied operand and a fixed operand.
 ; - The common value is already in %rcx.
-; - The tied output value is live outside the EBB.
+; - The tied output value is live outside the block.
 ;
 ; Under these conditions, Solver::add_tied_input() would create a variable for the tied input
 ; without considering the fixed constraint.
 function %pr221(i64 [%rdi], i64 [%rsi], i64 [%rdx], i64 [%rcx]) -> i64 [%rax] {
-ebb0(v0: i64, v1: i64, v2: i64, v3: i64):
+block0(v0: i64, v1: i64, v2: i64, v3: i64):
     v4 = ushr v3, v3
-    jump ebb1
+    jump block1
 
-ebb1:
+block1:
     return v4
 }
 
@@ -37,13 +37,13 @@ ebb1:
 ; Since the ushr x, x result is forced to be placed in %rcx, we must set the replace_global_defines
 ; flag so it can be reassigned to a different global register.
 function %pr218(i64 [%rdi], i64 [%rsi], i64 [%rdx], i64 [%rcx]) -> i64 [%rax] {
-ebb0(v0: i64, v1: i64, v2: i64, v3: i64):
+block0(v0: i64, v1: i64, v2: i64, v3: i64):
     ; check: regmove v3, %rcx ->
     v4 = ushr v0, v0
     ; check: v4 = copy
-    jump ebb1
+    jump block1
 
-ebb1:
+block1:
     ; v3 is globally live in %rcx.
     ; v4 is also globally live. Needs to be assigned something else for the trip across the CFG edge.
     v5 = iadd v3, v4
diff --git a/cranelift/filetests/filetests/regalloc/multiple-returns.clif b/cranelift/filetests/filetests/regalloc/multiple-returns.clif
index 6514e3e030..3481747a60 100644
--- a/cranelift/filetests/filetests/regalloc/multiple-returns.clif
+++ b/cranelift/filetests/filetests/regalloc/multiple-returns.clif
@@ -4,7 +4,7 @@ target x86_64
 ; Return the same value twice. This needs a copy so that each value can be
 ; allocated its own register.
 function %multiple_returns() -> i64, i64 {
-ebb0:
+block0:
     v2 = iconst.i64 0
     return v2, v2
 }
@@ -14,7 +14,7 @@ ebb0:
 
 ; Same thing, now with a fallthrough_return.
 function %multiple_returns() -> i64, i64 {
-ebb0:
+block0:
     v2 = iconst.i64 0
     fallthrough_return v2, v2
 }
diff --git a/cranelift/filetests/filetests/regalloc/output-interference.clif b/cranelift/filetests/filetests/regalloc/output-interference.clif
index ab027a72fb..513c81f4e5 100644
--- a/cranelift/filetests/filetests/regalloc/output-interference.clif
+++ b/cranelift/filetests/filetests/regalloc/output-interference.clif
@@ -2,7 +2,7 @@ test regalloc
 target x86_64 haswell
 
 function %test(i64) -> i64 system_v {
-ebb0(v0: i64):
+block0(v0: i64):
     v2 = iconst.i64 12
     ; This division clobbers two of its fixed input registers on x86.
     ; These are FixedTied constraints that the spiller needs to resolve.
diff --git a/cranelift/filetests/filetests/regalloc/reload-208.clif b/cranelift/filetests/filetests/regalloc/reload-208.clif
index 23783def04..6a723f02f5 100644
--- a/cranelift/filetests/filetests/regalloc/reload-208.clif
+++ b/cranelift/filetests/filetests/regalloc/reload-208.clif
@@ -2,14 +2,14 @@ test regalloc
 target x86_64 haswell
 
 ; regex: V=v\d+
-; regex: EBB=ebb\d+
+; regex: BB=block\d+
 
 ; Filed as https://github.com/bytecodealliance/cranelift/issues/208
 ;
 ; The verifier complains about a branch argument that is not in the same virtual register as the
-; corresponding EBB argument.
+; corresponding block argument.
 ;
-; The problem was the reload pass rewriting EBB arguments on "brnz v9, ebb3(v9)"
+; The problem was the reload pass rewriting block arguments on "brnz v9, block3(v9)"
 
 function %pr208(i64 vmctx [%rdi]) system_v {
     gv1 = vmctx
@@ -20,18 +20,18 @@ function %pr208(i64 vmctx [%rdi]) system_v {
     fn0 = u0:1 sig0
     fn1 = u0:3 sig1
 
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = iconst.i32 0
     v2 = call fn0(v0)
     v20 = iconst.i32 0x4ffe
     v16 = icmp uge v2, v20
-    brz v16, ebb5
-    jump ebb9
+    brz v16, block5
+    jump block9
 
-ebb9:
+block9:
     trap heap_oob
 
-ebb5:
+block5:
     v17 = uextend.i64 v2
     v18 = iadd_imm.i64 v0, -8
     v19 = load.i64 v18
@@ -40,25 +40,25 @@ ebb5:
     v21 = iconst.i32 0
     v5 = icmp eq v4, v21
     v6 = bint.i32 v5
-    brnz v6, ebb2
-    jump ebb3(v4)
+    brnz v6, block2
+    jump block3(v4)
 
-    ; check: ebb5:
-    ; check:   jump ebb3(v4)
-    ; check: $(splitEdge=$EBB):
-    ; nextln:  jump ebb3(v9)
+    ; check: block5:
+    ; check:   jump block3(v4)
+    ; check: $(splitEdge=$BB):
+    ; nextln:  jump block3(v9)
 
-ebb3(v7: i32):
+block3(v7: i32):
     call fn1(v0, v7)
     v26 = iconst.i32 0x4ffe
     v22 = icmp uge v7, v26
-    brz v22, ebb6
-    jump ebb10
+    brz v22, block6
+    jump block10
 
-ebb10:
+block10:
     trap heap_oob
 
-ebb6:
+block6:
     v23 = uextend.i64 v7
     v24 = iadd_imm.i64 v0, -8
     v25 = load.i64 v24
@@ -66,23 +66,23 @@ ebb6:
     v9 = load.i32 v8+56
     ; check: v9 = spill
     ; check: brnz $V, $splitEdge
-    brnz v9, ebb3(v9)
-    jump ebb4
+    brnz v9, block3(v9)
+    jump block4
 
-ebb4:
-    jump ebb2
+block4:
+    jump block2
 
-ebb2:
+block2:
     v10 = iconst.i32 0
     v31 = iconst.i32 0x4ffe
     v27 = icmp uge v10, v31
-    brz v27, ebb7
-    jump ebb11
+    brz v27, block7
+    jump block11
 
-ebb11:
+block11:
     trap heap_oob
 
-ebb7:
+block7:
     v28 = uextend.i64 v10
     v29 = iadd_imm.i64 v0, -8
     v30 = load.i64 v29
@@ -92,21 +92,21 @@ ebb7:
     v13 = iconst.i32 0
     v36 = iconst.i32 0x4ffe
     v32 = icmp uge v13, v36
-    brz v32, ebb8
-    jump ebb12
+    brz v32, block8
+    jump block12
 
-ebb12:
+block12:
     trap heap_oob
 
-ebb8:
+block8:
     v33 = uextend.i64 v13
     v34 = iadd_imm.i64 v0, -8
     v35 = load.i64 v34
     v14 = iadd v35, v33
     v15 = load.i32 v14+12
     call fn1(v0, v15)
-    jump ebb1
+    jump block1
 
-ebb1:
+block1:
     return
 }
diff --git a/cranelift/filetests/filetests/regalloc/reload-779.clif b/cranelift/filetests/filetests/regalloc/reload-779.clif
index f4e8e3ecff..ed6374c9fb 100644
--- a/cranelift/filetests/filetests/regalloc/reload-779.clif
+++ b/cranelift/filetests/filetests/regalloc/reload-779.clif
@@ -12,12 +12,12 @@ function u0:0(i64, i64, i64) system_v {
     fn1 = u0:94 sig0
     fn2 = u0:95 sig1
 
-ebb0(v0: i64, v1: i64, v2: i64):
+block0(v0: i64, v1: i64, v2: i64):
     v3 = iconst.i16 0
-    jump ebb1(v3)
+    jump block1(v3)
 
-ebb1(v4: i16):
+block1(v4: i16):
     call fn1()
     call fn2(v4)
-    jump ebb1(v4)
+    jump block1(v4)
 }
diff --git a/cranelift/filetests/filetests/regalloc/reload.clif b/cranelift/filetests/filetests/regalloc/reload.clif
index 0d39047dfd..88b20c1501 100644
--- a/cranelift/filetests/filetests/regalloc/reload.clif
+++ b/cranelift/filetests/filetests/regalloc/reload.clif
@@ -7,7 +7,7 @@ target riscv32 enable_e
 function %spill_return() -> i32 {
     fn0 = %foo() -> i32 system_v
 
-ebb0:
+block0:
     v0 = call fn0()
     ; check: $(reg=$V) = call fn0
     ; check: v0 = spill $reg
@@ -24,7 +24,7 @@ ebb0:
 ; on the stack.
 function %spilled_copy_arg(i32, i32, i32, i32, i32, i32, i32) -> i32 {
 
-ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32):
+block0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32):
     ; not: copy
     ; check: v10 = fill v6
     v10 = copy v6
@@ -37,7 +37,7 @@ ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32):
 function %spilled_copy_result(i32) -> i32 {
     fn0 = %foo(i32)
 
-ebb0(v0: i32):
+block0(v0: i32):
     ; not: copy
     ; check: v1 = spill v0
     v1 = copy v0
diff --git a/cranelift/filetests/filetests/regalloc/schedule-moves.clif b/cranelift/filetests/filetests/regalloc/schedule-moves.clif
index afd652ece9..f46d8958f7 100644
--- a/cranelift/filetests/filetests/regalloc/schedule-moves.clif
+++ b/cranelift/filetests/filetests/regalloc/schedule-moves.clif
@@ -2,7 +2,7 @@ test regalloc
 target i686 haswell
 
 function %pr165() system_v {
-ebb0:
+block0:
     v0 = iconst.i32 0x0102_0304
     v1 = iconst.i32 0x1102_0304
     v2 = iconst.i32 0x2102_0304
@@ -20,7 +20,7 @@ ebb0:
 ; Same as above, but use so many registers that spilling is required.
 ; Note: This is also a candidate for using xchg instructions.
 function %emergency_spill() system_v {
-ebb0:
+block0:
     v0 = iconst.i32 0x0102_0304
     v1 = iconst.i32 0x1102_0304
     v2 = iconst.i32 0x2102_0304
diff --git a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-2.clif b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-2.clif
index be64db792d..9737d4e163 100644
--- a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-2.clif
+++ b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-2.clif
@@ -4,16 +4,16 @@ set enable_pinned_reg=true
 target x86_64 haswell
 
 function u0:0(i32, i32, i32, i64 vmctx) -> i64 uext system_v {
-ebb0(v0: i32, v1: i32, v2: i32, v3: i64):
+block0(v0: i32, v1: i32, v2: i32, v3: i64):
     v236 = iconst.i32 0x4de9_bd37
     v424 = iconst.i32 0
-    jump ebb37(v424)
+    jump block37(v424)
 
-ebb37(v65: i32):
+block37(v65: i32):
     v433 = iconst.i32 0
-    jump ebb40(v433)
+    jump block40(v433)
 
-ebb40(v70: i32):
+block40(v70: i32):
     v75 = iconst.i32 0
     v259 = iconst.i32 0
     v78 -> v259
@@ -28,17 +28,17 @@ ebb40(v70: i32):
     v272 = iconst.i32 0x4de9_bd37
     v490, v273 = x86_smulx v100, v272
     v493 = iconst.i32 0
-    jump ebb61(v493)
+    jump block61(v493)
 
-ebb61(v103: i32):
+block61(v103: i32):
     v104 = iconst.i32 -23
     v105 = iconst.i32 -23
     v106 = popcnt v105
     v500 = sshr_imm v104, 31
     v501 = iconst.i32 0
-    jump ebb64(v501)
+    jump block64(v501)
 
-ebb64(v107: i32):
+block64(v107: i32):
     v108 = iconst.i32 0
     v109 = iconst.i32 0
     v278 = iconst.i32 0
@@ -49,9 +49,9 @@ ebb64(v107: i32):
     v283 = iadd v281, v282
     v111 -> v283
     v112 = rotr v108, v283
-    jump ebb65
+    jump block65
 
-ebb65:
+block65:
     v509 = iconst.i32 0
     v510, v511 = x86_sdivmodx v107, v509, v112
     v113 -> v510
diff --git a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif
index 771957c44e..8a9a040eb1 100644
--- a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif
+++ b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var-3.clif
@@ -4,16 +4,16 @@ set enable_pinned_reg=true
 target x86_64 haswell
 
 function u0:0(i32, i32, i32, i64 vmctx) -> i64 uext system_v {
-ebb0(v0: i32, v1: i32, v2: i32, v3: i64):
+block0(v0: i32, v1: i32, v2: i32, v3: i64):
     v5 = iconst.i32 -8
     v114 = iconst.i32 0
     v16 = iconst.i32 -8
     v17 = popcnt v16
     v192 = ifcmp_imm v17, -1
     trapif ne v192, user0
-    jump ebb12
+    jump block12
 
-ebb12:
+block12:
     v122 = iconst.i32 0
     v123 = ushr_imm v122, 31
     v124 = iadd v122, v123
@@ -23,51 +23,51 @@ ebb12:
     v31 -> v204
     v210 = ifcmp_imm v31, -1
     trapif ne v210, user0
-    jump ebb18
+    jump block18
 
-ebb18:
+block18:
     v215 = iconst.i32 0
-    jump ebb19(v215)
+    jump block19(v215)
 
-ebb19(v32: i32):
+block19(v32: i32):
     v35 = iconst.i32 0
     v218 = ifcmp_imm v35, -1
     trapif ne v218, user0
-    jump ebb21
+    jump block21
 
-ebb21:
+block21:
     v223 = iconst.i32 0
-    jump ebb22(v223)
+    jump block22(v223)
 
-ebb22(v36: i32):
+block22(v36: i32):
     v136 = iconst.i32 0
     v40 -> v136
     v227 = ifcmp_imm v136, -1
     trapif ne v227, user0
-    jump ebb24
+    jump block24
 
-ebb24:
+block24:
     v232 = iconst.i32 0
-    jump ebb25(v232)
+    jump block25(v232)
 
-ebb25(v41: i32):
+block25(v41: i32):
     v142 = iconst.i32 0
     v45 -> v142
     v236 = ifcmp_imm v142, -1
     trapif ne v236, user0
-    jump ebb27
+    jump block27
 
-ebb27:
+block27:
     v241 = iconst.i32 0
-    jump ebb28(v241)
+    jump block28(v241)
 
-ebb28(v46: i32):
+block28(v46: i32):
     v49 = iconst.i32 0
     v244 = ifcmp_imm v49, -1
     trapif ne v244, user0
-    jump ebb30
+    jump block30
 
-ebb30:
+block30:
     v254 = iconst.i32 0
     v53 -> v254
     v54 = iconst.i32 -23
@@ -80,9 +80,9 @@ ebb30:
     v148 = iadd v146, v147
     v57 -> v148
     v58 = ishl v53, v148
-    jump ebb35
+    jump block35
 
-ebb35:
+block35:
     v262 = iconst.i32 0
     v263, v264 = x86_sdivmodx v46, v262, v58
     v59 -> v263
@@ -93,9 +93,9 @@ ebb35:
     v280 = iconst.i32 0
     v281 = ffcmp v61, v61
     trapff ord v281, user0
-    jump ebb41(v280)
+    jump block41(v280)
 
-ebb41(v62: i32):
+block41(v62: i32):
     v157 = iconst.i32 0
     v158 = sshr_imm v157, 4
     v159 = iconst.i32 0
@@ -103,9 +103,9 @@ ebb41(v62: i32):
     v75 -> v160
     v308 = ifcmp_imm v160, -1
     trapif ne v308, user0
-    jump ebb52
+    jump block52
 
-ebb52:
+block52:
     v87 = iconst.i32 -23
     v88 = iconst.i32 -23
     v89 = popcnt v88
diff --git a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
index 7182bea43a..475bfa0f47 100644
--- a/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
+++ b/cranelift/filetests/filetests/regalloc/solver-fixedconflict-var.clif
@@ -6,7 +6,7 @@ target x86_64 haswell
 ;; Test for the issue #1123; https://github.com/bytecodealliance/cranelift/issues/1123
 
 function u0:0(i32, i32, i32, i64 vmctx) -> i64 uext system_v {
-ebb0(v0: i32, v1: i32, v2: i32, v3: i64):
+block0(v0: i32, v1: i32, v2: i32, v3: i64):
     v351 = iconst.i32 0x4de9_bd37
     v31 = iconst.i32 -23
     v35 = iconst.i32 0
@@ -24,68 +24,68 @@ ebb0(v0: i32, v1: i32, v2: i32, v3: i64):
     v53 = iconst.i32 0
     v547 = ifcmp_imm v53, -1
     trapif ne v547, user0
-    jump ebb30
+    jump block30
 
-ebb30:
+block30:
     v75 = iconst.i32 0
     v581 = ifcmp_imm v75, -1
     trapif ne v581, user0
-    jump ebb42
+    jump block42
 
-ebb42:
+block42:
     v136 = iconst.i32 0
     v691 = ifcmp_imm v136, -1
     trapif ne v691, user0
-    jump ebb81
+    jump block81
 
-ebb81:
+block81:
     v158 = iconst.i32 0
     v725 = ifcmp_imm v158, -1
     trapif ne v725, user0
-    jump ebb93
+    jump block93
 
-ebb93:
+block93:
     v760 = iconst.i32 0
-    jump ebb106(v760)
+    jump block106(v760)
 
-ebb106(v175: i32):
+block106(v175: i32):
     v179 = iconst.i32 0
     v180 = icmp_imm eq v179, 0
     v183 = iconst.i32 0
     v766 = ifcmp_imm v183, -1
     trapif ne v766, user0
-    jump ebb108
+    jump block108
 
-ebb108:
+block108:
     v771 = iconst.i32 0
-    jump ebb109(v771)
+    jump block109(v771)
 
-ebb109(v184: i32):
+block109(v184: i32):
     v785 = iconst.i32 0
     v193 -> v785
     v791 = ifcmp_imm v193, -1
     trapif ne v791, user0
-    jump ebb117
+    jump block117
 
-ebb117:
+block117:
     v796 = iconst.i32 0
-    jump ebb118(v796)
+    jump block118(v796)
 
-ebb118(v194: i32):
+block118(v194: i32):
     v203 = iconst.i32 -63
     v809 = iconst.i32 0
     v207 -> v809
     v815 = ifcmp_imm v207, -1
     trapif ne v815, user0
-    jump ebb126
+    jump block126
 
-ebb126:
+block126:
     v209 = iconst.i32 0
     v823 = ifcmp_imm v209, -1
     trapif ne v823, user0
-    jump ebb129
+    jump block129
 
-ebb129:
+block129:
     v213 = iconst.i32 -23
     v214 = iconst.i32 -19
     v215 = icmp_imm eq v214, 0
@@ -111,9 +111,9 @@ ebb129:
     v858, v859 = x86_sdivmodx v175, v857, v232
     v233 -> v858
     v915 = iconst.i32 0
-    jump ebb163(v915)
+    jump block163(v915)
 
-ebb163(v253: i32):
+block163(v253: i32):
     v255 = iconst.i32 0
     v256 = iconst.i32 -23
     v257 = iconst.i32 -19
diff --git a/cranelift/filetests/filetests/regalloc/spill-noregs.clif b/cranelift/filetests/filetests/regalloc/spill-noregs.clif
index c470b3355b..5acdd45b17 100644
--- a/cranelift/filetests/filetests/regalloc/spill-noregs.clif
+++ b/cranelift/filetests/filetests/regalloc/spill-noregs.clif
@@ -7,20 +7,20 @@ target x86_64
 ; 'Ran out of GPR registers when inserting copy before v68 = icmp.i32 eq v66, v67',
 ; cranelift-codegen/src/regalloc/spilling.rs:425:28 message.
 ;
-; The process_reg_uses() function is trying to insert a copy before the icmp instruction in ebb4
-; and runs out of registers to spill. Note that ebb7 has a lot of dead parameter values.
+; The process_reg_uses() function is trying to insert a copy before the icmp instruction in block4
+; and runs out of registers to spill. Note that block7 has a lot of dead parameter values.
 ;
-; The spiller was not releasing register pressure for dead EBB parameters.
+; The spiller was not releasing register pressure for dead block parameters.
 
 function %pr223(i32 [%rdi], i64 vmctx [%rsi]) -> i64 [%rax] system_v {
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = iconst.i32 0
     v3 = iconst.i64 0
     v4 = iconst.i32 0xffff_ffff_bb3f_4a2c
-    brz v4, ebb5
-    jump ebb1
+    brz v4, block5
+    jump block1
 
-ebb1:
+block1:
     v5 = iconst.i32 0
     v6 = copy.i64 v3
     v7 = copy.i64 v3
@@ -33,10 +33,10 @@ ebb1:
     v14 = copy.i64 v3
     v15 = copy.i64 v3
     v16 = copy.i64 v3
-    brnz v5, ebb4(v2, v3, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16)
-    jump ebb2
+    brnz v5, block4(v2, v3, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16)
+    jump block2
 
-ebb2:
+block2:
     v17 = iconst.i32 0
     v18 = copy.i64 v3
     v19 = copy.i64 v3
@@ -49,19 +49,19 @@ ebb2:
     v26 = copy.i64 v3
     v27 = copy.i64 v3
     v28 = copy.i64 v3
-    brnz v17, ebb4(v2, v3, v18, v19, v20, v21, v22, v23, v24, v25, v26, v27, v28)
-    jump ebb3
+    brnz v17, block4(v2, v3, v18, v19, v20, v21, v22, v23, v24, v25, v26, v27, v28)
+    jump block3
 
-ebb3:
-    jump ebb1
+block3:
+    jump block1
 
-ebb4(v29: i32, v30: i64, v31: i64, v32: i64, v33: i64, v34: i64, v35: i64, v36: i64, v37: i64, v38: i64, v39: i64, v40: i64, v41: i64):
-    jump ebb7(v29, v30, v31, v32, v33, v34, v35, v36, v37, v38, v39, v40, v41)
+block4(v29: i32, v30: i64, v31: i64, v32: i64, v33: i64, v34: i64, v35: i64, v36: i64, v37: i64, v38: i64, v39: i64, v40: i64, v41: i64):
+    jump block7(v29, v30, v31, v32, v33, v34, v35, v36, v37, v38, v39, v40, v41)
 
-ebb5:
-    jump ebb6
+block5:
+    jump block6
 
-ebb6:
+block6:
     v42 = copy.i64 v3
     v43 = copy.i64 v3
     v44 = copy.i64 v3
@@ -73,103 +73,103 @@ ebb6:
     v50 = copy.i64 v3
     v51 = copy.i64 v3
     v52 = copy.i64 v3
-    jump ebb7(v2, v3, v42, v43, v44, v45, v46, v47, v48, v49, v50, v51, v52)
+    jump block7(v2, v3, v42, v43, v44, v45, v46, v47, v48, v49, v50, v51, v52)
 
-ebb7(v53: i32, v54: i64, v55: i64, v56: i64, v57: i64, v58: i64, v59: i64, v60: i64, v61: i64, v62: i64, v63: i64, v64: i64, v65: i64):
+block7(v53: i32, v54: i64, v55: i64, v56: i64, v57: i64, v58: i64, v59: i64, v60: i64, v61: i64, v62: i64, v63: i64, v64: i64, v65: i64):
     v66 = iconst.i32 0
     v67 = iconst.i32 0
     v68 = icmp eq v66, v67
     v69 = bint.i32 v68
-    jump ebb8
+    jump block8
 
-ebb8:
-    jump ebb9
+block8:
+    jump block9
 
-ebb9:
+block9:
     v70 = iconst.i32 0xffff_ffff_ffff_912f
-    brz v70, ebb10
-    jump ebb35
+    brz v70, block10
+    jump block35
 
-ebb10:
+block10:
     v71 = iconst.i32 0
-    brz v71, ebb11
-    jump ebb27
+    brz v71, block11
+    jump block27
 
-ebb11:
-    jump ebb12
+block11:
+    jump block12
 
-ebb12:
-    jump ebb13
+block12:
+    jump block13
 
-ebb13:
-    jump ebb14
+block13:
+    jump block14
 
-ebb14:
-    jump ebb15
+block14:
+    jump block15
 
-ebb15:
-    jump ebb16
+block15:
+    jump block16
 
-ebb16:
-    jump ebb17
+block16:
+    jump block17
 
-ebb17:
-    jump ebb18
+block17:
+    jump block18
 
-ebb18:
-    jump ebb19
+block18:
+    jump block19
 
-ebb19:
-    jump ebb20
+block19:
+    jump block20
 
-ebb20:
-    jump ebb21
+block20:
+    jump block21
 
-ebb21:
-    jump ebb22
+block21:
+    jump block22
 
-ebb22:
-    jump ebb23
+block22:
+    jump block23
 
-ebb23:
-    jump ebb24
+block23:
+    jump block24
 
-ebb24:
-    jump ebb25
+block24:
+    jump block25
 
-ebb25:
-    jump ebb26
+block25:
+    jump block26
 
-ebb26:
-    jump ebb27
+block26:
+    jump block27
 
-ebb27:
-    jump ebb28
+block27:
+    jump block28
 
-ebb28:
-    jump ebb29
+block28:
+    jump block29
 
-ebb29:
-    jump ebb30
+block29:
+    jump block30
 
-ebb30:
-    jump ebb31
+block30:
+    jump block31
 
-ebb31:
-    jump ebb32
+block31:
+    jump block32
 
-ebb32:
-    jump ebb33
+block32:
+    jump block33
 
-ebb33:
-    jump ebb34
+block33:
+    jump block34
 
-ebb34:
-    jump ebb35
+block34:
+    jump block35
 
-ebb35:
-    jump ebb36
+block35:
+    jump block36
 
-ebb36:
+block36:
     trap user0
 }
diff --git a/cranelift/filetests/filetests/regalloc/spill.clif b/cranelift/filetests/filetests/regalloc/spill.clif
index 525921a374..23706cd2cf 100644
--- a/cranelift/filetests/filetests/regalloc/spill.clif
+++ b/cranelift/filetests/filetests/regalloc/spill.clif
@@ -24,8 +24,8 @@ function %pyramid(i32) -> i32 {
 ; check: ss1 = spill_slot 4
 ; check: ss2 = spill_slot 4
 ; not: spill_slot
-ebb0(v1: i32):
-; check: ebb0($(rv1=$V): i32 [%x10], $(rlink=$V): i32 [%x1])
+block0(v1: i32):
+; check: block0($(rv1=$V): i32 [%x10], $(rlink=$V): i32 [%x1])
     ; check: ,ss0]$WS v1 = spill $rv1
     ; nextln: ,ss1]$WS $(link=$V) = spill $rlink
     ; not: spill
@@ -71,7 +71,7 @@ ebb0(v1: i32):
 ; All values live across a call must be spilled
 function %across_call(i32) {
     fn0 = %foo(i32)
-ebb0(v1: i32):
+block0(v1: i32):
     ; check: v1 = spill
     call fn0(v1)
     ; check: call fn0
@@ -84,7 +84,7 @@ ebb0(v1: i32):
 ; The same value used for two function arguments.
 function %doubleuse(i32) {
     fn0 = %xx(i32, i32)
-ebb0(v0: i32):
+block0(v0: i32):
     ; check: $(c=$V) = copy v0
     call fn0(v0, v0)
     ; check: call fn0(v0, $c)
@@ -94,7 +94,7 @@ ebb0(v0: i32):
 ; The same value used as indirect callee and argument.
 function %doubleuse_icall1(i32) {
     sig0 = (i32) system_v
-ebb0(v0: i32):
+block0(v0: i32):
     ; not:copy
     call_indirect sig0, v0(v0)
     return
@@ -103,7 +103,7 @@ ebb0(v0: i32):
 ; The same value used as indirect callee and two arguments.
 function %doubleuse_icall2(i32) {
     sig0 = (i32, i32) system_v
-ebb0(v0: i32):
+block0(v0: i32):
     ; check: $(c=$V) = copy v0
     call_indirect sig0, v0(v0, v0)
     ; check: call_indirect sig0, v0(v0, $c)
@@ -115,21 +115,21 @@ function %stackargs(i32, i32, i32, i32, i32, i32, i32, i32) -> i32 {
 ; check: ss0 = incoming_arg 4
 ; check: ss1 = incoming_arg 4, offset 4
 ; not: incoming_arg
-ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32, v7: i32):
+block0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32, v5: i32, v6: i32, v7: i32):
     ; unordered: fill v6
     ; unordered: fill v7
     v10 = iadd v6, v7
     return v10
 }
 
-; More EBB arguments than registers.
-function %ebbargs(i32) -> i32 {
-ebb0(v1: i32):
+; More block arguments than registers.
+function %blockargs(i32) -> i32 {
+block0(v1: i32):
     ; check: v1 = spill
     v2 = iconst.i32 1
-    jump ebb1(v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2)
+    jump block1(v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2)
 
-ebb1(v10: i32, v11: i32, v12: i32, v13: i32, v14: i32, v15: i32, v16: i32, v17: i32, v18: i32, v19: i32, v20: i32, v21: i32):
+block1(v10: i32, v11: i32, v12: i32, v13: i32, v14: i32, v15: i32, v16: i32, v17: i32, v18: i32, v19: i32, v20: i32, v21: i32):
     v22 = iadd v10, v11
     v23 = iadd v22, v12
     v24 = iadd v23, v13
@@ -145,18 +145,18 @@ ebb1(v10: i32, v11: i32, v12: i32, v13: i32, v14: i32, v15: i32, v16: i32, v17:
     return v33
 }
 
-; Spilling an EBB argument to make room for a branch operand.
+; Spilling a block argument to make room for a branch operand.
 function %brargs(i32) -> i32 {
-ebb0(v1: i32):
+block0(v1: i32):
     ; check: v1 = spill
     v2 = iconst.i32 1
-    brnz v1, ebb1(v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2)
-    jump ebb2
+    brnz v1, block1(v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2, v2)
+    jump block2
 
-ebb2:
+block2:
     return v1
 
-ebb1(v10: i32, v11: i32, v12: i32, v13: i32, v14: i32, v15: i32, v16: i32, v17: i32, v18: i32, v19: i32, v20: i32, v21: i32):
+block1(v10: i32, v11: i32, v12: i32, v13: i32, v14: i32, v15: i32, v16: i32, v17: i32, v18: i32, v19: i32, v20: i32, v21: i32):
     v22 = iadd v10, v11
     v23 = iadd v22, v12
     v24 = iadd v23, v13
@@ -181,8 +181,8 @@ function %use_spilled_value(i32) -> i32 {
 ; check: ss0 = spill_slot 4
 ; check: ss1 = spill_slot 4
 ; check: ss2 = spill_slot 4
-ebb0(v1: i32):
-; check: ebb0($(rv1=$V): i32 [%x10], $(rlink=$V): i32 [%x1])
+block0(v1: i32):
+; check: block0($(rv1=$V): i32 [%x10], $(rlink=$V): i32 [%x1])
     ; check: ,ss0]$WS v1 = spill $rv1
     ; nextln: ,ss1]$WS $(link=$V) = spill $rlink
     ; not: spill
diff --git a/cranelift/filetests/filetests/regalloc/unreachable_code.clif b/cranelift/filetests/filetests/regalloc/unreachable_code.clif
index 0eac70fc6f..4c288a91dd 100644
--- a/cranelift/filetests/filetests/regalloc/unreachable_code.clif
+++ b/cranelift/filetests/filetests/regalloc/unreachable_code.clif
@@ -7,41 +7,41 @@ target x86_64 haswell
 ; This function contains unreachable blocks which trip up the register
 ; allocator if they don't get cleared out.
 function %unreachable_blocks(i64 vmctx) -> i32 baldrdash_system_v {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = iconst.i32 0
     v2 = iconst.i32 0
-    jump ebb2
+    jump block2
 
-ebb2:
-    jump ebb4
+block2:
+    jump block4
 
-ebb4:
-    jump ebb2
+block4:
+    jump block2
 
 ; Everything below this point is unreachable.
 
-ebb3(v3: i32):
+block3(v3: i32):
     v5 = iadd.i32 v2, v3
-    jump ebb6
+    jump block6
 
-ebb6:
-    jump ebb6
+block6:
+    jump block6
 
-ebb7(v6: i32):
+block7(v6: i32):
     v7 = iadd.i32 v5, v6
-    jump ebb8
+    jump block8
 
-ebb8:
-    jump ebb10
+block8:
+    jump block10
 
-ebb10:
-    jump ebb8
+block10:
+    jump block8
 
-ebb9(v8: i32):
+block9(v8: i32):
     v10 = iadd.i32 v7, v8
-    jump ebb1(v10)
+    jump block1(v10)
 
-ebb1(v11: i32):
+block1(v11: i32):
     return v11
 }
 
diff --git a/cranelift/filetests/filetests/regalloc/x86-regres.clif b/cranelift/filetests/filetests/regalloc/x86-regres.clif
index 0b9bf12736..e239d0ad37 100644
--- a/cranelift/filetests/filetests/regalloc/x86-regres.clif
+++ b/cranelift/filetests/filetests/regalloc/x86-regres.clif
@@ -2,48 +2,48 @@ test regalloc
 target i686
 
 ; regex: V=v\d+
-; regex: EBB=ebb\d+
+; regex: BB=block\d+
 
-; The value v9 appears both as the branch control and one of the EBB arguments
-; in the brnz instruction in ebb2. It also happens that v7 and v9 are assigned
+; The value v9 appears both as the branch control and one of the block arguments
+; in the brnz instruction in block2. It also happens that v7 and v9 are assigned
 ; to the same register, so v9 doesn't need to be moved before the brnz.
 ;
 ; This ended up confusong the constraint solver which had not made a record of
 ; the fixed register assignment for v9 since it was already in the correct
 ; register.
 function %pr147(i32) -> i32 system_v {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 0
     v2 = iconst.i32 1
     v3 = iconst.i32 0
-    jump ebb2(v3, v2, v0)
+    jump block2(v3, v2, v0)
 
-    ; check: $(splitEdge=$EBB):
-    ; check:   jump ebb2($V, $V, v9)
+    ; check: $(splitEdge=$BB):
+    ; check:   jump block2($V, $V, v9)
 
-ebb2(v4: i32, v5: i32, v7: i32):
-    ; check: ebb2
+block2(v4: i32, v5: i32, v7: i32):
+    ; check: block2
     v6 = iadd v4, v5
     v8 = iconst.i32 -1
     ; v7 is killed here and v9 gets the same register.
     v9 = iadd v7, v8
     ; check:   v9 = iadd v7, v8
-    ; Here v9 the brnz control appears to interfere with v9 the EBB argument,
+    ; Here v9 the brnz control appears to interfere with v9 the block argument,
     ; so divert_fixed_input_conflicts() calls add_var(v9), which is ok. The
     ; add_var sanity checks got confused when no fixed assignment could be
     ; found for v9.
     ;
     ; We should be able to handle this situation without making copies of v9.
-    brnz v9, ebb2(v5, v6, v9)
+    brnz v9, block2(v5, v6, v9)
     ; check:   brnz v9, $splitEdge
-    jump ebb3
+    jump block3
 
-ebb3:
+block3:
     return v5
 }
 
 function %select_i64(i64, i64, i32) -> i64 {
-ebb0(v0: i64, v1: i64, v2: i32):
+block0(v0: i64, v1: i64, v2: i32):
     v3 = select v2, v0, v1
     return v3
 }
diff --git a/cranelift/filetests/filetests/regress/allow-relaxation-shrink.clif b/cranelift/filetests/filetests/regress/allow-relaxation-shrink.clif
index 3dd8250656..995e7c5f64 100644
--- a/cranelift/filetests/filetests/regress/allow-relaxation-shrink.clif
+++ b/cranelift/filetests/filetests/regress/allow-relaxation-shrink.clif
@@ -9,33 +9,33 @@ function u0:0(i64, i64) -> i64 system_v {
     sig0 = (i64) -> i64 system_v
     fn0 = u0:8 sig0
 
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v3 = stack_addr.i64 ss1
     v5 = call fn0(v1)
     v6 = iconst.i64 0
     v8 = iconst.i64 0
-    jump ebb3(v6, v1, v8)
+    jump block3(v6, v1, v8)
 
-ebb3(v39: i64, v40: i64, v42: i64):
+block3(v39: i64, v40: i64, v42: i64):
     v9 = load.i64 v3
     v11 = icmp_imm ugt v9, 1
     v12 = bint.i8 v11
     v13 = uextend.i32 v12
     v14 = icmp_imm eq v13, 0
-    brnz v14, ebb4
-    jump ebb5
+    brnz v14, block4
+    jump block5
 
-ebb4:
+block4:
     v18 = icmp_imm.i64 eq v40, 0
     v19 = bint.i8 v18
     v20 = uextend.i32 v19
-    brz v20, ebb6
-    jump ebb7
+    brz v20, block6
+    jump block7
 
-ebb7:
+block7:
     trap user0
 
-ebb5:
+block5:
     v22 = iconst.i32 1
     v23 = ishl.i64 v39, v22
     v25 = iconst.i64 1
@@ -47,9 +47,9 @@ ebb5:
     v31 = iconst.i32 1
     v32 = ushr v30, v31
     store v32, v3
-    jump ebb3(v27, v40, v29)
+    jump block3(v27, v40, v29)
 
-ebb6:
+block6:
     v38 = iconst.i64 0
     return v38
 }
diff --git a/cranelift/filetests/filetests/safepoint/basic.clif b/cranelift/filetests/filetests/safepoint/basic.clif
index cb52fbf66b..7e0088b23b 100644
--- a/cranelift/filetests/filetests/safepoint/basic.clif
+++ b/cranelift/filetests/filetests/safepoint/basic.clif
@@ -3,69 +3,69 @@ set enable_safepoints=true
 target x86_64
 
 function %test(i32, r64, r64) -> r64 {
-    ebb0(v0: i32, v1:r64, v2:r64):
-        jump ebb1(v0)
-    ebb1(v3: i32):
+    block0(v0: i32, v1:r64, v2:r64):
+        jump block1(v0)
+    block1(v3: i32):
         v4 = irsub_imm v3, 1
-        jump ebb2(v4)
-    ebb2(v5: i32):
+        jump block2(v4)
+    block2(v5: i32):
         resumable_trap interrupt
-        brz v5, ebb1(v5)
-        jump ebb3
-    ebb3:
+        brz v5, block1(v5)
+        jump block3
+    block3:
         v6 = null.r64
         v7 = is_null v6
-        brnz v7, ebb2(v0)
-        jump ebb4
-    ebb4:
-        brnz v0, ebb5
-        jump ebb6
-    ebb5:
+        brnz v7, block2(v0)
+        jump block4
+    block4:
+        brnz v0, block5
+        jump block6
+    block5:
         return v1
-    ebb6:
+    block6:
         return v2
 }
 
 ; sameln: function %test(i32 [%rdi], r64 [%rsi], r64 [%rdx]) -> r64 [%rax] fast {
-; nextln: ebb0(v0: i32 [%rdi], v1: r64 [%rsi], v2: r64 [%rdx]):
+; nextln: block0(v0: i32 [%rdi], v1: r64 [%rsi], v2: r64 [%rdx]):
 ; nextln:   v10 = copy v0
-; nextln:   jump ebb1(v10)
+; nextln:   jump block1(v10)
 ; nextln: 
-; nextln: ebb7:
+; nextln: block7:
 ; nextln:   regmove.i32 v5, %rcx -> %rax
-; nextln:   jump ebb1(v5)
+; nextln:   jump block1(v5)
 ; nextln: 
-; nextln: ebb1(v3: i32 [%rax]):
+; nextln: block1(v3: i32 [%rax]):
 ; nextln:   v8 = iconst.i32 1
 ; nextln:   v4 = isub v8, v3
-; nextln:   jump ebb2(v4)
+; nextln:   jump block2(v4)
 ; nextln: 
-; nextln: ebb8:
+; nextln: block8:
 ; nextln:   v9 = copy.i32 v0
 ; nextln:   regmove v9, %rax -> %rcx
-; nextln:   jump ebb2(v9)
+; nextln:   jump block2(v9)
 ; nextln: 
-; nextln: ebb2(v5: i32 [%rcx]):
+; nextln: block2(v5: i32 [%rcx]):
 ; nextln:   safepoint v1, v2
 ; nextln:   resumable_trap interrupt
-; nextln:   brz v5, ebb7
-; nextln:   jump ebb3
+; nextln:   brz v5, block7
+; nextln:   jump block3
 ; nextln: 
-; nextln: ebb3:
+; nextln: block3:
 ; nextln:   v6 = null.r64 
 ; nextln:   v7 = is_null v6
-; nextln:   brnz v7, ebb8
-; nextln:   jump ebb4
+; nextln:   brnz v7, block8
+; nextln:   jump block4
 ; nextln: 
-; nextln: ebb4:
-; nextln:   brnz.i32 v0, ebb5
-; nextln:   jump ebb6
+; nextln: block4:
+; nextln:   brnz.i32 v0, block5
+; nextln:   jump block6
 ; nextln: 
-; nextln: ebb5:
+; nextln: block5:
 ; nextln:   regmove.r64 v1, %rsi -> %rax
 ; nextln:   return v1
 ; nextln: 
-; nextln: ebb6:
+; nextln: block6:
 ; nextln:   regmove.r64 v2, %rdx -> %rax
 ; nextln:   return v2
 ; nextln: }
diff --git a/cranelift/filetests/filetests/safepoint/call.clif b/cranelift/filetests/filetests/safepoint/call.clif
index 9e9583093b..53c9246323 100644
--- a/cranelift/filetests/filetests/safepoint/call.clif
+++ b/cranelift/filetests/filetests/safepoint/call.clif
@@ -7,15 +7,15 @@ function %direct() -> r64 {
     fn1 = %one() -> r64
     fn2 = %two() -> i32, r64
 
-ebb0:
+block0:
     call fn0()
     v1 = call fn1()
     v2, v3 = call fn2()
-    brz v2, ebb2
-    jump ebb1
-ebb1:
+    brz v2, block2
+    jump block1
+block1:
     return v1
-ebb2:
+block2:
     v4 = call fn1()
     return v3
 }
@@ -30,7 +30,7 @@ ebb2:
 ; nextln: fn1 = %one sig1
 ; nextln: fn2 = %two sig2
 ; nextln: 
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:   v5 = func_addr.i64 fn0
 ; nextln:   call_indirect sig0, v5()
 ; nextln:   v6 = func_addr.i64 fn1
@@ -40,15 +40,15 @@ ebb2:
 ; nextln:   safepoint v1
 ; nextln:   v2, v10 = call_indirect sig2, v7()
 ; nextln:   v3 = spill v10
-; nextln:   brz v2, ebb2
-; nextln:   jump ebb1
+; nextln:   brz v2, block2
+; nextln:   jump block1
 ; nextln: 
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:   v11 = fill.r64 v1
 ; nextln:   regmove v11, %r15 -> %rax
 ; nextln:   return v11
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:   v8 = func_addr.i64 fn1
 ; nextln:   safepoint v3
 ; nextln:   v4 = call_indirect sig1, v8()
diff --git a/cranelift/filetests/filetests/simple_gvn/basic.clif b/cranelift/filetests/filetests/simple_gvn/basic.clif
index 72155a324c..107c3897d1 100644
--- a/cranelift/filetests/filetests/simple_gvn/basic.clif
+++ b/cranelift/filetests/filetests/simple_gvn/basic.clif
@@ -1,7 +1,7 @@
 test simple-gvn
 
 function %simple_redundancy(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = iadd v0, v1
     v3 = iadd v0, v1
     v4 = imul v2, v3
@@ -10,7 +10,7 @@ ebb0(v0: i32, v1: i32):
 }
 
 function %cascading_redundancy(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = iadd v0, v1
     v3 = iadd v0, v1
     v4 = imul v2, v3
@@ -21,22 +21,22 @@ ebb0(v0: i32, v1: i32):
 }
 
 function %redundancies_on_some_paths(i32, i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32, v2: i32):
+block0(v0: i32, v1: i32, v2: i32):
     v3 = iadd v0, v1
-    brz v3, ebb1
-    jump ebb3
+    brz v3, block1
+    jump block3
 
-ebb3:
+block3:
     v4 = iadd v0, v1
-    jump ebb2(v4)
-; check: jump ebb2(v3)
+    jump block2(v4)
+; check: jump block2(v3)
 
-ebb1:
+block1:
     v5 = iadd v0, v1
-    jump ebb2(v5)
-; check: jump ebb2(v3)
+    jump block2(v5)
+; check: jump block2(v3)
 
-ebb2(v6: i32):
+block2(v6: i32):
     v7 = iadd v0, v1
     v8 = iadd v6, v7
 ; check: v8 = iadd v6, v3
diff --git a/cranelift/filetests/filetests/simple_gvn/readonly.clif b/cranelift/filetests/filetests/simple_gvn/readonly.clif
index 3c01299064..802396f4f8 100644
--- a/cranelift/filetests/filetests/simple_gvn/readonly.clif
+++ b/cranelift/filetests/filetests/simple_gvn/readonly.clif
@@ -7,7 +7,7 @@ function %eliminate_redundant_global_loads(i32, i64 vmctx) {
     gv1 = load.i64 notrap aligned readonly gv0
     heap0 = static gv1, min 0x1_0000, bound 0x1_0000_0000, offset_guard 0x8000_0000, index_type i32
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = heap_addr.i64 heap0, v0, 1
 
diff --git a/cranelift/filetests/filetests/simple_gvn/reject.clif b/cranelift/filetests/filetests/simple_gvn/reject.clif
index 00d88e5e66..c4613af4dc 100644
--- a/cranelift/filetests/filetests/simple_gvn/reject.clif
+++ b/cranelift/filetests/filetests/simple_gvn/reject.clif
@@ -1,7 +1,7 @@
 test simple-gvn
 
 function %other_side_effects(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
    regmove v0, %10 -> %20
    regmove v0, %10 -> %20
    regmove v0, %20 -> %10
@@ -11,7 +11,7 @@ ebb0(v0: i32):
 }
 
 function %differing_typevars() -> i64 {
-ebb0:
+block0:
     v0 = iconst.i32 7
     v1 = iconst.i64 7
     v2 = iconst.i64 8
@@ -25,7 +25,7 @@ ebb0:
 }
 
 function %cpu_flags() -> b1 {
-ebb0:
+block0:
     v0 = iconst.i32 7
     v1 = iconst.i32 8
     v2 = ifcmp v0, v1
@@ -41,7 +41,7 @@ ebb0:
 }
 
 function %spill() -> i32 {
-ebb0:
+block0:
     v0 = iconst.i32 7
     v1 = spill v0
     v2 = fill v1
diff --git a/cranelift/filetests/filetests/simple_gvn/scopes.clif b/cranelift/filetests/filetests/simple_gvn/scopes.clif
index bf4e7fac94..63a425ad3f 100644
--- a/cranelift/filetests/filetests/simple_gvn/scopes.clif
+++ b/cranelift/filetests/filetests/simple_gvn/scopes.clif
@@ -1,29 +1,29 @@
 test simple-gvn
 
 function %two_diamonds(i32, i32, i32, i32, i32) {
-ebb0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32):
+block0(v0: i32, v1: i32, v2: i32, v3: i32, v4: i32):
     v5 = iconst.i32 16
     ; check: v5 = iconst.i32 16
-    brz v0, ebb1
-    jump ebb5
+    brz v0, block1
+    jump block5
 
-ebb5:
+block5:
     v6 = iconst.i32 17
     ; check: v6 = iconst.i32 17
     v7 = iconst.i32 16
     ; not: v7 = iconst.i32 16
-    jump ebb2
+    jump block2
 
-ebb1:
+block1:
     v8 = iconst.i32 18
     ; check: v8 = iconst.i32 18
     v9 = iconst.i32 17
     ; check: v9 = iconst.i32 17
     v10 = iconst.i32 16
     ; not: v10 = iconst.i32 16
-    jump ebb2
+    jump block2
 
-ebb2:
+block2:
     v11 = iconst.i32 19
     ; check: v11 = iconst.i32 19
     v12 = iconst.i32 18
@@ -32,10 +32,10 @@ ebb2:
     ; check: v13 = iconst.i32 17
     v14 = iconst.i32 16
     ; not: v14 = iconst.i32 16
-    brz v1, ebb3
-    jump ebb6
+    brz v1, block3
+    jump block6
 
-ebb6:
+block6:
     v15 = iconst.i32 20
     ; check: v15 = iconst.i32 20
     v16 = iconst.i32 19
@@ -46,9 +46,9 @@ ebb6:
     ; not: v18 = iconst.i32 17
     v19 = iconst.i32 16
     ; not: v19 = iconst.i32 16
-    jump ebb4
+    jump block4
 
-ebb3:
+block3:
     v20 = iconst.i32 21
     ; check: v20 = iconst.i32 21
     v21 = iconst.i32 20
@@ -61,9 +61,9 @@ ebb3:
     ; not: v24 = iconst.i32 17
     v25 = iconst.i32 16
     ; not: v25 = iconst.i32 16
-    jump ebb4
+    jump block4
 
-ebb4:
+block4:
     v26 = iconst.i32 22
     ; check: v26 = iconst.i32 22
     v27 = iconst.i32 21
diff --git a/cranelift/filetests/filetests/simple_preopt/branch.clif b/cranelift/filetests/filetests/simple_preopt/branch.clif
index f3535b1e70..21cc7afda3 100644
--- a/cranelift/filetests/filetests/simple_preopt/branch.clif
+++ b/cranelift/filetests/filetests/simple_preopt/branch.clif
@@ -2,80 +2,80 @@ test simple_preopt
 target x86_64
 
 function %icmp_to_brz_fold(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = icmp_imm eq v0, 0
-    brnz v1, ebb1
-    jump ebb2
-ebb1:
+    brnz v1, block1
+    jump block2
+block1:
     v3 = iconst.i32 1
     return v3
-ebb2:
+block2:
     v4 = iconst.i32 2
     return v4
 }
 ; sameln: function %icmp_to_brz_fold
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     v1 = icmp_imm eq v0, 0
-; nextln:     brnz v0, ebb2
-; nextln:     jump ebb1
+; nextln:     brnz v0, block2
+; nextln:     jump block1
 ; nextln: 
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:     v3 = iconst.i32 1
 ; nextln:     return v3
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:     v4 = iconst.i32 2
 ; nextln:     return v4
 ; nextln: }
 
 function %icmp_to_brz_inverted_fold(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = icmp_imm ne v0, 0
-    brz v1, ebb1
-    jump ebb2
-ebb1:
+    brz v1, block1
+    jump block2
+block1:
     v3 = iconst.i32 1
     return v3
-ebb2:
+block2:
     v4 = iconst.i32 2
     return v4
 }
 ; sameln: function %icmp_to_brz_inve
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     v1 = icmp_imm ne v0, 0
-; nextln:     brnz v0, ebb2
-; nextln:     jump ebb1
+; nextln:     brnz v0, block2
+; nextln:     jump block1
 ; nextln: 
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:     v3 = iconst.i32 1
 ; nextln:     return v3
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:     v4 = iconst.i32 2
 ; nextln:     return v4
 ; nextln: }
 
 function %br_icmp_inversion(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
-    br_icmp ugt v0, v1, ebb1
-    jump ebb2
-ebb1:
+block0(v0: i32, v1: i32):
+    br_icmp ugt v0, v1, block1
+    jump block2
+block1:
     v2 = iconst.i32 1
     return v2
-ebb2:
+block2:
     v3 = iconst.i32 2
     return v3
 }
 ; sameln: function %br_icmp_inversio
-; nextln: ebb0(v0: i32, v1: i32):
-; nextln:     br_icmp ule v0, v1, ebb2
-; nextln:     jump ebb1
+; nextln: block0(v0: i32, v1: i32):
+; nextln:     br_icmp ule v0, v1, block2
+; nextln:     jump block1
 ; nextln: 
-; nextln: ebb1:
+; nextln: block1:
 ; nextln:     v2 = iconst.i32 1
 ; nextln:     return v2
 ; nextln: 
-; nextln: ebb2:
+; nextln: block2:
 ; nextln:     v3 = iconst.i32 2
 ; nextln:     return v3
 ; nextln: }
diff --git a/cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif b/cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif
index c111113197..101e4eb201 100644
--- a/cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif
+++ b/cranelift/filetests/filetests/simple_preopt/div_by_const_indirect.clif
@@ -4,7 +4,7 @@ target x86_64 baseline
 ; Cases where the denominator is created by an iconst
 
 function %indir_udiv32(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 7
     v2 = udiv v0, v1
     ; check: iconst.i32 7
@@ -19,7 +19,7 @@ ebb0(v0: i32):
 }
 
 function %indir_sdiv32(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 -17
     v2 = sdiv v0, v1
     ; check: iconst.i32 -17
@@ -33,7 +33,7 @@ ebb0(v0: i32):
 }
 
 function %indir_udiv64(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = iconst.i64 1337
     v2 = udiv v0, v1
     ; check: iconst.i64 1337
@@ -45,7 +45,7 @@ ebb0(v0: i64):
 }
 
 function %indir_sdiv64(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = iconst.i64 -90210
     v2 = sdiv v0, v1
     ; check: iconst.i64 0xffff_ffff_fffe_9f9e
diff --git a/cranelift/filetests/filetests/simple_preopt/div_by_const_non_power_of_2.clif b/cranelift/filetests/filetests/simple_preopt/div_by_const_non_power_of_2.clif
index 2a16699aae..b1225a28d5 100644
--- a/cranelift/filetests/filetests/simple_preopt/div_by_const_non_power_of_2.clif
+++ b/cranelift/filetests/filetests/simple_preopt/div_by_const_non_power_of_2.clif
@@ -5,7 +5,7 @@ target i686 baseline
 
 ; complex case (mul, sub, shift, add, shift)
 function %t_udiv32_p7(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = udiv_imm v0, 7
     ; check: iconst.i32 0x2492_4925
     ; check: umulhi v0, v2
@@ -19,7 +19,7 @@ ebb0(v0: i32):
 
 ; simple case (mul, shift)
 function %t_udiv32_p125(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = udiv_imm v0, 125
     ; check: iconst.i32 0x1062_4dd3
     ; check: umulhi v0, v2
@@ -30,7 +30,7 @@ ebb0(v0: i32):
 
 ; simple case w/ shift by zero (mul)
 function %t_udiv32_p641(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = udiv_imm v0, 641
     ; check: iconst.i32 0x0066_3d81
     ; check: v3 = umulhi v0, v2
@@ -43,7 +43,7 @@ ebb0(v0: i32):
 
 ; simple case w/ shift by zero (mul, add-sign-bit)
 function %t_sdiv32_n6(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, -6
     ; check: iconst.i32 0xffff_ffff_d555_5555
     ; check: smulhi v0, v2
@@ -55,7 +55,7 @@ ebb0(v0: i32):
 
 ; simple case (mul, shift, add-sign-bit)
 function %t_sdiv32_n5(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, -5
     ; check: iconst.i32 0xffff_ffff_9999_9999
     ; check: smulhi v0, v2
@@ -68,7 +68,7 @@ ebb0(v0: i32):
 
 ; case d < 0 && M > 0 (mul, sub, shift, add-sign-bit)
 function %t_sdiv32_n3(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, -3
     ; check: iconst.i32 0x5555_5555
     ; check: smulhi v0, v2
@@ -82,7 +82,7 @@ ebb0(v0: i32):
 
 ; simple case w/ shift by zero (mul, add-sign-bit)
 function %t_sdiv32_p6(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, 6
     ; check: iconst.i32 0x2aaa_aaab
     ; check: smulhi v0, v2
@@ -94,7 +94,7 @@ ebb0(v0: i32):
 
 ; case d > 0 && M < 0 (mull, add, shift, add-sign-bit)
 function %t_sdiv32_p7(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, 7
     ; check: iconst.i32 0xffff_ffff_9249_2493
     ; check: smulhi v0, v2
@@ -108,7 +108,7 @@ ebb0(v0: i32):
 
 ; simple case (mul, shift, add-sign-bit)
 function %t_sdiv32_p625(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, 625
     ; check: iconst.i32 0x68db_8bad
     ; check: smulhi v0, v2
@@ -124,7 +124,7 @@ ebb0(v0: i32):
 
 ; complex case (mul, sub, shift, add, shift)
 function %t_udiv64_p7(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = udiv_imm v0, 7
     ; check: iconst.i64 0x2492_4924_9249_2493
     ; check: umulhi v0, v2
@@ -138,7 +138,7 @@ ebb0(v0: i64):
 
 ; simple case (mul, shift)
 function %t_udiv64_p9(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = udiv_imm v0, 9
     ; check: iconst.i64 0xe38e_38e3_8e38_e38f
     ; check: umulhi v0, v2
@@ -149,7 +149,7 @@ ebb0(v0: i64):
 
 ; complex case (mul, sub, shift, add, shift)
 function %t_udiv64_p125(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = udiv_imm v0, 125
     ; check: iconst.i64 0x0624_dd2f_1a9f_be77
     ; check: umulhi v0, v2
@@ -163,7 +163,7 @@ ebb0(v0: i64):
 
 ; simple case w/ shift by zero (mul)
 function %t_udiv64_p274177(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = udiv_imm v0, 274177
     ; check: iconst.i64 0x3d30_f19c_d101
     ; check: v3 = umulhi v0, v2
@@ -176,7 +176,7 @@ ebb0(v0: i64):
 
 ; simple case (mul, shift, add-sign-bit)
 function %t_sdiv64_n625(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, -625
     ; check: iconst.i64 0xcb92_3a29_c779_a6b5
     ; check: smulhi v0, v2
@@ -189,7 +189,7 @@ ebb0(v0: i64):
 
 ; simple case w/ zero shift (mul, add-sign-bit)
 function %t_sdiv64_n6(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, -6
     ; check: iconst.i64 0xd555_5555_5555_5555
     ; check: smulhi v0, v2
@@ -201,7 +201,7 @@ ebb0(v0: i64):
 
 ; simple case w/ zero shift (mul, add-sign-bit)
 function %t_sdiv64_n5(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, -5
     ; check: iconst.i64 0x9999_9999_9999_9999
     ; check: smulhi v0, v2
@@ -214,7 +214,7 @@ ebb0(v0: i64):
 
 ; case d < 0 && M > 0 (mul, sub, shift, add-sign-bit)
 function %t_sdiv64_n3(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, -3
     ; check: iconst.i64 0x5555_5555_5555_5555
     ; check: smulhi v0, v2
@@ -228,7 +228,7 @@ ebb0(v0: i64):
 
 ; simple case w/ zero shift (mul, add-sign-bit)
 function %t_sdiv64_p6(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, 6
     ; check: iconst.i64 0x2aaa_aaaa_aaaa_aaab
     ; check: smulhi v0, v2
@@ -240,7 +240,7 @@ ebb0(v0: i64):
 
 ; case d > 0 && M < 0 (mul, add, shift, add-sign-bit)
 function %t_sdiv64_p15(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, 15
     ; check: iconst.i64 0x8888_8888_8888_8889
     ; check: smulhi v0, v2
@@ -254,7 +254,7 @@ ebb0(v0: i64):
 
 ; simple case (mul, shift, add-sign-bit)
 function %t_sdiv64_p625(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, 625
     ; check: iconst.i64 0x346d_c5d6_3886_594b
     ; check: smulhi v0, v2
diff --git a/cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif b/cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif
index fb9c1744f5..83e9f95c8a 100644
--- a/cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif
+++ b/cranelift/filetests/filetests/simple_preopt/div_by_const_power_of_2.clif
@@ -5,7 +5,7 @@ target i686 baseline
 
 ; ignored
 function %t_udiv32_p0(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = udiv_imm v0, 0
     ; check: udiv_imm v0, 0
     return v1
@@ -13,7 +13,7 @@ ebb0(v0: i32):
 
 ; converted to a nop
 function %t_udiv32_p1(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = udiv_imm v0, 1
     ; check: nop
     return v1
@@ -21,7 +21,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_udiv32_p2(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = udiv_imm v0, 2
     ; check: ushr_imm v0, 1
     return v1
@@ -29,7 +29,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_udiv32_p2p31(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = udiv_imm v0, 0x8000_0000
     ; check: ushr_imm v0, 31
     return v1
@@ -40,7 +40,7 @@ ebb0(v0: i32):
 
 ; ignored
 function %t_udiv64_p0(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = udiv_imm v0, 0
     ; check: udiv_imm v0, 0
     return v1
@@ -48,7 +48,7 @@ ebb0(v0: i64):
 
 ; converted to a nop
 function %t_udiv64_p1(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = udiv_imm v0, 1
     ; check: nop
     return v1
@@ -56,7 +56,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_udiv64_p2(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = udiv_imm v0, 2
     ; check: ushr_imm v0, 1
     return v1
@@ -64,7 +64,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_udiv64_p2p63(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = udiv_imm v0, 0x8000_0000_0000_0000
     ; check: ushr_imm v0, 63
     return v1
@@ -75,7 +75,7 @@ ebb0(v0: i64):
 
 ; ignored
 function %t_sdiv32_p0(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, 0
     ; check: sdiv_imm v0, 0
     return v1
@@ -83,7 +83,7 @@ ebb0(v0: i32):
 
 ; converted to a nop
 function %t_sdiv32_p1(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, 1
     ; check: nop
     return v1
@@ -91,7 +91,7 @@ ebb0(v0: i32):
 
 ; ignored
 function %t_sdiv32_n1(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, -1
     ; check: sdiv_imm v0, -1
     return v1
@@ -99,7 +99,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_sdiv32_p2(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, 2
     ; check: ushr_imm v0, 31
     ; check: iadd v0, v2
@@ -110,7 +110,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_sdiv32_n2(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, -2
     ; check: ushr_imm v0, 31
     ; check: iadd v0, v2
@@ -121,7 +121,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_sdiv32_p4(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, 4
     ; check: v2 = sshr_imm v0, 1
     ; check: ushr_imm v2, 30
@@ -134,7 +134,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_sdiv32_n4(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, -4
     ; check: sshr_imm v0, 1
     ; check: ushr_imm v2, 30
@@ -146,7 +146,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_sdiv32_p2p30(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, 0x4000_0000
     ; check: sshr_imm v0, 29
     ; check: ushr_imm v2, 2
@@ -158,7 +158,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_sdiv32_n2p30(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, -0x4000_0000
     ; check: sshr_imm v0, 29
     ; check: ushr_imm v2, 2
@@ -171,7 +171,7 @@ ebb0(v0: i32):
 ; there's no positive version of this, since -(-0x8000_0000) isn't
 ; representable.
 function %t_sdiv32_n2p31(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sdiv_imm v0, -0x8000_0000
     ; check: sshr_imm v0, 30
     ; check: ushr_imm v2, 1
@@ -186,7 +186,7 @@ ebb0(v0: i32):
 
 ; ignored
 function %t_sdiv64_p0(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, 0
     ; check: sdiv_imm v0, 0
     return v1
@@ -194,7 +194,7 @@ ebb0(v0: i64):
 
 ; converted to a nop
 function %t_sdiv64_p1(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, 1
     ; check: nop
     return v1
@@ -202,7 +202,7 @@ ebb0(v0: i64):
 
 ; ignored
 function %t_sdiv64_n1(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, -1
     ; check: sdiv_imm v0, -1
     return v1
@@ -210,7 +210,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_sdiv64_p2(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, 2
     ; check: ushr_imm v0, 63
     ; check: iadd v0, v2
@@ -221,7 +221,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_sdiv64_n2(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, -2
     ; check: ushr_imm v0, 63
     ; check: iadd v0, v2
@@ -232,7 +232,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_sdiv64_p4(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, 4
     ; check: sshr_imm v0, 1
     ; check: ushr_imm v2, 62
@@ -244,7 +244,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_sdiv64_n4(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, -4
     ; check: sshr_imm v0, 1
     ; check: ushr_imm v2, 62
@@ -256,7 +256,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_sdiv64_p2p62(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, 0x4000_0000_0000_0000
     ; check: sshr_imm v0, 61
     ; check: ushr_imm v2, 2
@@ -268,7 +268,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_sdiv64_n2p62(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, -0x4000_0000_0000_0000
     ; check: sshr_imm v0, 61
     ; check: ushr_imm v2, 2
@@ -281,7 +281,7 @@ ebb0(v0: i64):
 ; there's no positive version of this, since -(-0x8000_0000_0000_0000) isn't
 ; representable.
 function %t_sdiv64_n2p63(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = sdiv_imm v0, -0x8000_0000_0000_0000
     ; check: sshr_imm v0, 62
     ; check: ushr_imm v2, 1
diff --git a/cranelift/filetests/filetests/simple_preopt/fold-extended-move-wraparound.clif b/cranelift/filetests/filetests/simple_preopt/fold-extended-move-wraparound.clif
index 074507a786..44342481b8 100644
--- a/cranelift/filetests/filetests/simple_preopt/fold-extended-move-wraparound.clif
+++ b/cranelift/filetests/filetests/simple_preopt/fold-extended-move-wraparound.clif
@@ -5,7 +5,7 @@ function %wraparound(i64 vmctx) -> f32 system_v {
     gv0 = vmctx
     gv1 = iadd_imm.i64 gv0, 48
 
-ebb35(v0: i64):
+block35(v0: i64):
     v88 = iconst.i64 0
     v89 = iconst.i64 0x8000_0000_0000_0000
     v90 = ishl_imm v88, 0x8000_0000_0000_0000
diff --git a/cranelift/filetests/filetests/simple_preopt/rem_by_const_non_power_of_2.clif b/cranelift/filetests/filetests/simple_preopt/rem_by_const_non_power_of_2.clif
index 40c5e1d828..00d0d9f16e 100644
--- a/cranelift/filetests/filetests/simple_preopt/rem_by_const_non_power_of_2.clif
+++ b/cranelift/filetests/filetests/simple_preopt/rem_by_const_non_power_of_2.clif
@@ -5,7 +5,7 @@ target i686 baseline
 
 ; complex case (mul, sub, shift, add, shift)
 function %t_urem32_p7(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = urem_imm v0, 7
     ; check: iconst.i32 0x2492_4925
     ; check: umulhi v0, v2
@@ -20,7 +20,7 @@ ebb0(v0: i32):
 
 ; simple case (mul, shift)
 function %t_urem32_p125(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = urem_imm v0, 125
     ; check: iconst.i32 0x1062_4dd3
     ; check: umulhi v0, v2
@@ -32,7 +32,7 @@ ebb0(v0: i32):
 
 ; simple case w/ shift by zero (mul)
 function %t_urem32_p641(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = urem_imm v0, 641
     ; check: iconst.i32 0x0066_3d81
     ; check: umulhi v0, v2
@@ -46,7 +46,7 @@ ebb0(v0: i32):
 
 ; simple case w/ shift by zero (mul, add-sign-bit)
 function %t_srem32_n6(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, -6
     ; check: iconst.i32 0xffff_ffff_d555_5555
     ; check: smulhi v0, v2
@@ -59,7 +59,7 @@ ebb0(v0: i32):
 
 ; simple case (mul, shift, add-sign-bit)
 function %t_srem32_n5(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, -5
     ; check: iconst.i32 0xffff_ffff_9999_9999
     ; check: smulhi v0, v2
@@ -73,7 +73,7 @@ ebb0(v0: i32):
 
 ; case d < 0 && M > 0 (mul, sub, shift, add-sign-bit)
 function %t_srem32_n3(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, -3
     ; check: iconst.i32 0x5555_5555
     ; check: smulhi v0, v2
@@ -88,7 +88,7 @@ ebb0(v0: i32):
 
 ; simple case w/ shift by zero (mul, add-sign-bit)
 function %t_srem32_p6(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, 6
     ; check: iconst.i32 0x2aaa_aaab
     ; check: smulhi v0, v2
@@ -101,7 +101,7 @@ ebb0(v0: i32):
 
 ; case d > 0 && M < 0 (mull, add, shift, add-sign-bit)
 function %t_srem32_p7(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, 7
     ; check: iconst.i32 0xffff_ffff_9249_2493
     ; check: smulhi v0, v2
@@ -116,7 +116,7 @@ ebb0(v0: i32):
 
 ; simple case (mul, shift, add-sign-bit)
 function %t_srem32_p625(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, 625
     ; check: iconst.i32 0x68db_8bad
     ; check: smulhi v0, v2
@@ -133,7 +133,7 @@ ebb0(v0: i32):
 
 ; complex case (mul, sub, shift, add, shift)
 function %t_urem64_p7(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = urem_imm v0, 7
     ; check: umulhi v0, v2
     ; check: isub v0, v3
@@ -147,7 +147,7 @@ ebb0(v0: i64):
 
 ; simple case (mul, shift)
 function %t_urem64_p9(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = urem_imm v0, 9
     ; check: iconst.i64 0xe38e_38e3_8e38_e38f
     ; check: umulhi v0, v2
@@ -159,7 +159,7 @@ ebb0(v0: i64):
 
 ; complex case (mul, sub, shift, add, shift)
 function %t_urem64_p125(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = urem_imm v0, 125
     ; check: iconst.i64 0x0624_dd2f_1a9f_be77
     ; check: umulhi v0, v2
@@ -174,7 +174,7 @@ ebb0(v0: i64):
 
 ; simple case w/ shift by zero (mul)
 function %t_urem64_p274177(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = urem_imm v0, 274177
     ; check: iconst.i64 0x3d30_f19c_d101
     ; check: umulhi v0, v2
@@ -188,7 +188,7 @@ ebb0(v0: i64):
 
 ; simple case (mul, shift, add-sign-bit)
 function %t_srem64_n625(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, -625
     ; check: iconst.i64 0xcb92_3a29_c779_a6b5
     ; check: smulhi v0, v2
@@ -202,7 +202,7 @@ ebb0(v0: i64):
 
 ; simple case w/ zero shift (mul, add-sign-bit)
 function %t_srem64_n6(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, -6
     ; check: iconst.i64 0xd555_5555_5555_5555
     ; check: smulhi v0, v2
@@ -215,7 +215,7 @@ ebb0(v0: i64):
 
 ; simple case w/ zero shift (mul, add-sign-bit)
 function %t_srem64_n5(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, -5
     ; check: iconst.i64 0x9999_9999_9999_9999
     ; check: smulhi v0, v2
@@ -229,7 +229,7 @@ ebb0(v0: i64):
 
 ; case d < 0 && M > 0 (mul, sub, shift, add-sign-bit)
 function %t_srem64_n3(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, -3
     ; check: iconst.i64 0x5555_5555_5555_5555
     ; check: smulhi v0, v2
@@ -244,7 +244,7 @@ ebb0(v0: i64):
 
 ; simple case w/ zero shift (mul, add-sign-bit)
 function %t_srem64_p6(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, 6
     ; check: iconst.i64 0x2aaa_aaaa_aaaa_aaab
     ; check: smulhi v0, v2
@@ -257,7 +257,7 @@ ebb0(v0: i64):
 
 ; case d > 0 && M < 0 (mul, add, shift, add-sign-bit)
 function %t_srem64_p15(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, 15
     ; check: iconst.i64 0x8888_8888_8888_8889
     ; check: smulhi v0, v2
@@ -272,7 +272,7 @@ ebb0(v0: i64):
 
 ; simple case (mul, shift, add-sign-bit)
 function %t_srem64_p625(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, 625
     ; check: iconst.i64 0x346d_c5d6_3886_594b
     ; check: smulhi v0, v2
diff --git a/cranelift/filetests/filetests/simple_preopt/rem_by_const_power_of_2.clif b/cranelift/filetests/filetests/simple_preopt/rem_by_const_power_of_2.clif
index 09eebfa684..1fe085e37c 100644
--- a/cranelift/filetests/filetests/simple_preopt/rem_by_const_power_of_2.clif
+++ b/cranelift/filetests/filetests/simple_preopt/rem_by_const_power_of_2.clif
@@ -5,7 +5,7 @@ target i686 baseline
 
 ; ignored
 function %t_urem32_p0(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = urem_imm v0, 0
     ; check: urem_imm v0, 0
     return v1
@@ -13,7 +13,7 @@ ebb0(v0: i32):
 
 ; converted to constant zero
 function %t_urem32_p1(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = urem_imm v0, 1
     ; check: iconst.i32 0
     return v1
@@ -21,7 +21,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_urem32_p2(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = urem_imm v0, 2
     ; check: band_imm v0, 1
     return v1
@@ -29,7 +29,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_urem32_p2p31(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = urem_imm v0, 0x8000_0000
     ; check: band_imm v0, 0x7fff_ffff
     return v1
@@ -40,7 +40,7 @@ ebb0(v0: i32):
 
 ; ignored
 function %t_urem64_p0(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = urem_imm v0, 0
     ; check: urem_imm v0, 0
     return v1
@@ -48,7 +48,7 @@ ebb0(v0: i64):
 
 ; converted to constant zero
 function %t_urem64_p1(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = urem_imm v0, 1
     ; check: iconst.i64 0
     return v1
@@ -56,7 +56,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_urem64_p2(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = urem_imm v0, 2
     ; check: band_imm v0, 1
    return v1
@@ -64,7 +64,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_urem64_p2p63(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = urem_imm v0, 0x8000_0000_0000_0000
     ; check: band_imm v0, 0x7fff_ffff_ffff_ffff
     return v1
@@ -75,7 +75,7 @@ ebb0(v0: i64):
 
 ; ignored
 function %t_srem32_n1(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, -1
     ; check: srem_imm v0, -1
     return v1
@@ -83,7 +83,7 @@ ebb0(v0: i32):
 
 ; ignored
 function %t_srem32_p0(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, 0
     ; check: srem_imm v0, 0
     return v1
@@ -91,7 +91,7 @@ ebb0(v0: i32):
 
 ; converted to constant zero
 function %t_srem32_p1(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, 1
     ; check: iconst.i32 0
     return v1
@@ -99,7 +99,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_srem32_p2(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, 2
     ; check: ushr_imm v0, 31
     ; check: iadd v0, v2
@@ -110,7 +110,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_srem32_n2(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, -2
     ; check: ushr_imm v0, 31
     ; check: iadd v0, v2
@@ -121,7 +121,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_srem32_p4(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, 4
     ; check: sshr_imm v0, 1
     ; check: ushr_imm v2, 30
@@ -133,7 +133,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_srem32_n4(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, -4
     ; check: sshr_imm v0, 1
     ; check: ushr_imm v2, 30
@@ -145,7 +145,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_srem32_p2p30(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, 0x4000_0000
     ; check: sshr_imm v0, 29
     ; check: ushr_imm v2, 2
@@ -157,7 +157,7 @@ ebb0(v0: i32):
 
 ; shift
 function %t_srem32_n2p30(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, -0x4000_0000
     ; check: sshr_imm v0, 29
     ; check: ushr_imm v2, 2
@@ -170,7 +170,7 @@ ebb0(v0: i32):
 ; there's no positive version of this, since -(-0x8000_0000) isn't
 ; representable.
 function %t_srem32_n2p31(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = srem_imm v0, -0x8000_0000
     ; check: sshr_imm v0, 30
     ; check: ushr_imm v2, 1
@@ -185,7 +185,7 @@ ebb0(v0: i32):
 
 ; ignored
 function %t_srem64_n1(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, -1
     ; check: srem_imm v0, -1
     return v1
@@ -193,7 +193,7 @@ ebb0(v0: i64):
 
 ; ignored
 function %t_srem64_p0(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, 0
     ; check: srem_imm v0, 0
     return v1
@@ -201,7 +201,7 @@ ebb0(v0: i64):
 
 ; converted to constant zero
 function %t_srem64_p1(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, 1
     ; check: iconst.i64 0
     return v1
@@ -209,7 +209,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_srem64_p2(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, 2
     ; check: ushr_imm v0, 63
     ; check: iadd v0, v2
@@ -220,7 +220,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_srem64_n2(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, -2
     ; check: ushr_imm v0, 63
     ; check: iadd v0, v2
@@ -231,7 +231,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_srem64_p4(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, 4
     ; check: sshr_imm v0, 1
     ; check: ushr_imm v2, 62
@@ -243,7 +243,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_srem64_n4(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, -4
     ; check: sshr_imm v0, 1
     ; check: ushr_imm v2, 62
@@ -255,7 +255,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_srem64_p2p62(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, 0x4000_0000_0000_0000
     ; check: sshr_imm v0, 61
     ; check: ushr_imm v2, 2
@@ -267,7 +267,7 @@ ebb0(v0: i64):
 
 ; shift
 function %t_srem64_n2p62(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, -0x4000_0000_0000_0000
     ; check: sshr_imm v0, 61
     ; check: ushr_imm v2, 2
@@ -280,7 +280,7 @@ ebb0(v0: i64):
 ; there's no positive version of this, since -(-0x8000_0000_0000_0000) isn't
 ; representable.
 function %t_srem64_n2p63(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = srem_imm v0, -0x8000_0000_0000_0000
     ; check: sshr_imm v0, 62
     ; check: ushr_imm v2, 1
diff --git a/cranelift/filetests/filetests/simple_preopt/simplify32.clif b/cranelift/filetests/filetests/simple_preopt/simplify32.clif
index 45add1b7a3..2582fd69aa 100644
--- a/cranelift/filetests/filetests/simple_preopt/simplify32.clif
+++ b/cranelift/filetests/filetests/simple_preopt/simplify32.clif
@@ -4,40 +4,40 @@ target i686
 ;; 32-bits platforms.
 
 function %iadd_imm(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 2
     v2 = iadd v0, v1
     return v2
 }
 ; sameln: function %iadd_imm
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     v1 = iconst.i32 2
 ; nextln:     v2 = iadd_imm v0, 2
 ; nextln:     return v2
 ; nextln: }
 
 function %isub_imm(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 2
     v2 = isub v0, v1
     return v2
 }
 ; sameln: function %isub_imm
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     v1 = iconst.i32 2
 ; nextln:     v2 = iadd_imm v0, -2
 ; nextln:     return v2
 ; nextln: }
 
 function %icmp_imm(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 2
     v2 = icmp slt v0, v1
     v3 = bint.i32 v2
     return v3
 }
 ; sameln: function %icmp_imm
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     v1 = iconst.i32 2
 ; nextln:     v2 = icmp_imm slt v0, 2
 ; nextln:     v3 = bint.i32 v2
@@ -47,13 +47,13 @@ ebb0(v0: i32):
 ;; Don't simplify operations that would get illegal because of lack of native
 ;; support.
 function %iadd_imm(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = iconst.i64 2
     v2 = iadd v0, v1
     return v2
 }
 ; sameln: function %iadd_imm
-; nextln: ebb0(v0: i64):
+; nextln: block0(v0: i64):
 ; nextln:     v1 = iconst.i64 2
 ; nextln:     v2 = iadd v0, v1
 ; nextln:     return v2
diff --git a/cranelift/filetests/filetests/simple_preopt/simplify64.clif b/cranelift/filetests/filetests/simple_preopt/simplify64.clif
index db485ce773..4ceabdc335 100644
--- a/cranelift/filetests/filetests/simple_preopt/simplify64.clif
+++ b/cranelift/filetests/filetests/simple_preopt/simplify64.clif
@@ -4,40 +4,40 @@ target x86_64
 ;; 64-bits platforms.
 
 function %iadd_imm(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 2
     v2 = iadd v0, v1
     return v2
 }
 ; sameln: function %iadd_imm
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     v1 = iconst.i32 2
 ; nextln:     v2 = iadd_imm v0, 2
 ; nextln:     return v2
 ; nextln: }
 
 function %isub_imm(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 2
     v2 = isub v0, v1
     return v2
 }
 ; sameln: function %isub_imm
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     v1 = iconst.i32 2
 ; nextln:     v2 = iadd_imm v0, -2
 ; nextln:     return v2
 ; nextln: }
 
 function %icmp_imm(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 2
     v2 = icmp slt v0, v1
     v3 = bint.i32 v2
     return v3
 }
 ; sameln: function %icmp_imm
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     v1 = iconst.i32 2
 ; nextln:     v2 = icmp_imm slt v0, 2
 ; nextln:     v3 = bint.i32 v2
@@ -45,18 +45,18 @@ ebb0(v0: i32):
 ; nextln: }
 
 function %brz_bint(i32) {
-ebb0(v0: i32):
+block0(v0: i32):
     v3 = icmp_imm slt v0, 0
     v1 = bint.i32 v3
     v2 = select v1, v1, v1
     trapz v1, user0
-    brz v1, ebb1
-    jump ebb2
+    brz v1, block1
+    jump block2
 
-ebb1:
+block1:
     return
 
-ebb2:
+block2:
     return
 }
 ; sameln: function %brz_bint
@@ -65,17 +65,17 @@ ebb2:
 ; nextln:    v1 = bint.i32 v3
 ; nextln:    v2 = select v3, v1, v1
 ; nextln:    trapz v3, user0
-; nextln:    brnz v3, ebb2
-; nextln:    jump ebb1
+; nextln:    brnz v3, block2
+; nextln:    jump block1
 
 function %irsub_imm(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 2
     v2 = isub v1, v0
     return v2
 }
 ; sameln: function %irsub_imm
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:     v1 = iconst.i32 2
 ; nextln:     v2 = irsub_imm v0, 2
 ; nextln:     return v2
@@ -85,14 +85,14 @@ ebb0(v0: i32):
 
 ;; 8 -> 16
 function %uextend_8_16() -> i16 {
-ebb0:
+block0:
     v0 = iconst.i16 37
     v1 = ishl_imm v0, 8
     v2 = ushr_imm v1, 8
     return v2
 }
 ; sameln: function %uextend_8_16
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i16 37
 ; nextln:     v1 = ishl_imm v0, 8
 ; nextln:     v3 = ireduce.i8 v0
@@ -101,14 +101,14 @@ ebb0:
 ; nextln: }
 
 function %sextend_8_16() -> i16 {
-ebb0:
+block0:
     v0 = iconst.i16 37
     v1 = ishl_imm v0, 8
     v2 = sshr_imm v1, 8
     return v2
 }
 ; sameln: function %sextend_8_16
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i16 37
 ; nextln:     v1 = ishl_imm v0, 8
 ; nextln:     v3 = ireduce.i8 v0
@@ -118,14 +118,14 @@ ebb0:
 
 ;; 8 -> 32
 function %uextend_8_32() -> i32 {
-ebb0:
+block0:
     v0 = iconst.i32 37
     v1 = ishl_imm v0, 24
     v2 = ushr_imm v1, 24
     return v2
 }
 ; sameln: function %uextend_8_32
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i32 37
 ; nextln:     v1 = ishl_imm v0, 24
 ; nextln:     v3 = ireduce.i8 v0
@@ -134,14 +134,14 @@ ebb0:
 ; nextln: }
 
 function %sextend_8_32() -> i32 {
-ebb0:
+block0:
     v0 = iconst.i32 37
     v1 = ishl_imm v0, 24
     v2 = sshr_imm v1, 24
     return v2
 }
 ; sameln: function %sextend_8_32
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i32 37
 ; nextln:     v1 = ishl_imm v0, 24
 ; nextln:     v3 = ireduce.i8 v0
@@ -151,14 +151,14 @@ ebb0:
 
 ;; 16 -> 32
 function %uextend_16_32() -> i32 {
-ebb0:
+block0:
     v0 = iconst.i32 37
     v1 = ishl_imm v0, 16
     v2 = ushr_imm v1, 16
     return v2
 }
 ; sameln: function %uextend_16_32
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i32 37
 ; nextln:     v1 = ishl_imm v0, 16
 ; nextln:     v3 = ireduce.i16 v0
@@ -167,14 +167,14 @@ ebb0:
 ; nextln: }
 
 function %sextend_16_32() -> i32 {
-ebb0:
+block0:
     v0 = iconst.i32 37
     v1 = ishl_imm v0, 16
     v2 = sshr_imm v1, 16
     return v2
 }
 ; sameln: function %sextend_16_32
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i32 37
 ; nextln:     v1 = ishl_imm v0, 16
 ; nextln:     v3 = ireduce.i16 v0
@@ -184,14 +184,14 @@ ebb0:
 
 ;; 8 -> 64
 function %uextend_8_64() -> i64 {
-ebb0:
+block0:
     v0 = iconst.i64 37
     v1 = ishl_imm v0, 56
     v2 = ushr_imm v1, 56
     return v2
 }
 ; sameln: function %uextend_8_64
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i64 37
 ; nextln:     v1 = ishl_imm v0, 56
 ; nextln:     v3 = ireduce.i8 v0
@@ -200,14 +200,14 @@ ebb0:
 ; nextln: }
 
 function %sextend_8_64() -> i64 {
-ebb0:
+block0:
     v0 = iconst.i64 37
     v1 = ishl_imm v0, 56
     v2 = sshr_imm v1, 56
     return v2
 }
 ; sameln: function %sextend_8_64
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i64 37
 ; nextln:     v1 = ishl_imm v0, 56
 ; nextln:     v3 = ireduce.i8 v0
@@ -217,14 +217,14 @@ ebb0:
 
 ;; 16 -> 64
 function %uextend_16_64() -> i64 {
-ebb0:
+block0:
     v0 = iconst.i64 37
     v1 = ishl_imm v0, 48
     v2 = ushr_imm v1, 48
     return v2
 }
 ; sameln: function %uextend_16_64
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i64 37
 ; nextln:     v1 = ishl_imm v0, 48
 ; nextln:     v3 = ireduce.i16 v0
@@ -233,14 +233,14 @@ ebb0:
 ; nextln: }
 
 function %sextend_16_64() -> i64 {
-ebb0:
+block0:
     v0 = iconst.i64 37
     v1 = ishl_imm v0, 48
     v2 = sshr_imm v1, 48
     return v2
 }
 ; sameln: function %sextend_16_64
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i64 37
 ; nextln:     v1 = ishl_imm v0, 48
 ; nextln:     v3 = ireduce.i16 v0
@@ -250,14 +250,14 @@ ebb0:
 
 ;; 32 -> 64
 function %uextend_32_64() -> i64 {
-ebb0:
+block0:
     v0 = iconst.i64 37
     v1 = ishl_imm v0, 32
     v2 = ushr_imm v1, 32
     return v2
 }
 ; sameln: function %uextend_32_64
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i64 37
 ; nextln:     v1 = ishl_imm v0, 32
 ; nextln:     v3 = ireduce.i32 v0
@@ -266,14 +266,14 @@ ebb0:
 ; nextln: }
 
 function %sextend_32_64() -> i64 {
-ebb0:
+block0:
     v0 = iconst.i64 37
     v1 = ishl_imm v0, 32
     v2 = sshr_imm v1, 32
     return v2
 }
 ; sameln: function %sextend_32_64
-; nextln: ebb0:
+; nextln: block0:
 ; nextln:     v0 = iconst.i64 37
 ; nextln:     v1 = ishl_imm v0, 32
 ; nextln:     v3 = ireduce.i32 v0
@@ -282,13 +282,13 @@ ebb0:
 ; nextln: }
 
 function %add_imm_fold(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
   v1 = iadd_imm v0, 42
   v2 = iadd_imm v1, -42
   return v2
 }
 ; sameln: function %add_imm_fold(i32)
-; nextln: ebb0(v0: i32):
+; nextln: block0(v0: i32):
 ; nextln:    v2 -> v0
 ; nextln:    v1 = iadd_imm v0, 42
 ; nextln:    nop
diff --git a/cranelift/filetests/filetests/verifier/bad_layout.clif b/cranelift/filetests/filetests/verifier/bad_layout.clif
index 034dd7843f..0cc2d2ed6f 100644
--- a/cranelift/filetests/filetests/verifier/bad_layout.clif
+++ b/cranelift/filetests/filetests/verifier/bad_layout.clif
@@ -1,21 +1,21 @@
 test verifier
 
 function %test_1(i32) {
-    ebb0(v0: i32):
+    block0(v0: i32):
         return          ; error: terminator
         return
 }
 function %test_2(i32) {
-    ebb0(v0: i32):
-        jump ebb2       ; error: a terminator instruction was encountered before the end of ebb0
-        brz v0, ebb3
-    ebb2:
-        jump ebb3
-    ebb3:
+    block0(v0: i32):
+        jump block2       ; error: a terminator instruction was encountered before the end of block0
+        brz v0, block3
+    block2:
+        jump block3
+    block3:
         return
 }
 
 function %test_3(i32) { ; Ok
-    ebb0(v0: i32):
+    block0(v0: i32):
         return
 }
diff --git a/cranelift/filetests/filetests/verifier/bitcast.clif b/cranelift/filetests/filetests/verifier/bitcast.clif
index eb5303cfc0..98ac9c6b35 100644
--- a/cranelift/filetests/filetests/verifier/bitcast.clif
+++ b/cranelift/filetests/filetests/verifier/bitcast.clif
@@ -2,21 +2,21 @@ test verifier
 
 ; bitcast between two types of equal size if ok
 function %valid_bitcast1(i32) -> f32 { ; Ok
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = bitcast.f32 v0
     return v1
 }
 
 ; bitcast to a type larger than the operand is ok
 function %valid_bitcast2(i32) -> i64 { ; Ok
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = bitcast.i64 v0
     return v1
 }
 
 ; bitcast to a smaller type is not ok
 function %bad_bitcast(i64) -> i32 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = bitcast.i32 v0 ; error: The bitcast argument v0 doesn't fit in a type of 32 bits
     return v1
 }
diff --git a/cranelift/filetests/filetests/verifier/defs_dominates_uses.clif b/cranelift/filetests/filetests/verifier/defs_dominates_uses.clif
index fcbeb14816..c7b3b752a8 100644
--- a/cranelift/filetests/filetests/verifier/defs_dominates_uses.clif
+++ b/cranelift/filetests/filetests/verifier/defs_dominates_uses.clif
@@ -3,14 +3,14 @@ test verifier
 ; Test verification that uses properly dominate defs.
 
 function %non_dominating(i32) -> i32 system_v {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iadd.i32 v2, v0   ; error: uses value v2 from non-dominating
     v2 = iadd.i32 v1, v0
     return v2
 }
 
 function %inst_uses_its_own_values(i32) -> i32 system_v {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iadd.i32 v1, v0   ; error: uses value v1 from itself
     return v1
 }
diff --git a/cranelift/filetests/filetests/verifier/flags.clif b/cranelift/filetests/filetests/verifier/flags.clif
index 8273160510..dc370c58cb 100644
--- a/cranelift/filetests/filetests/verifier/flags.clif
+++ b/cranelift/filetests/filetests/verifier/flags.clif
@@ -3,7 +3,7 @@ target i686
 
 ; Simple, correct use of CPU flags.
 function %simple(i32) -> i32 {
-                    ebb0(v0: i32):
+                    block0(v0: i32):
     [DynRexOp1rcmp#39]        v1 = ifcmp v0, v0
     [Op2seti_abcd#490]        v2 = trueif ugt v1
     [Op2urm_noflags_abcd#4b6] v3 = bint.i32 v2
@@ -12,7 +12,7 @@ function %simple(i32) -> i32 {
 
 ; Overlapping flag values of different types.
 function %overlap(i32, f32) -> i32 {
-                    ebb0(v0: i32, v1: f32):
+                    block0(v0: i32, v1: f32):
     [DynRexOp1rcmp#39]        v2 = ifcmp v0, v0
     [Op2fcmp#42e]             v3 = ffcmp v1, v1
     [Op2setf_abcd#490]        v4 = trueff gt v3 ; error: conflicting live CPU flags: v2 and v3
@@ -24,7 +24,7 @@ function %overlap(i32, f32) -> i32 {
 
 ; CPU flags clobbered by arithmetic.
 function %clobbered(i32) -> i32 {
-                    ebb0(v0: i32):
+                    block0(v0: i32):
     [DynRexOp1rcmp#39]        v1 = ifcmp v0, v0
     [DynRexOp1rr#01]          v2 = iadd v0, v0 ; error: encoding clobbers live CPU flags in v1
     [Op2seti_abcd#490]        v3 = trueif ugt v1
@@ -34,7 +34,7 @@ function %clobbered(i32) -> i32 {
 
 ; CPU flags not clobbered by load.
 function %live_across_load(i32) -> i32 {
-                    ebb0(v0: i32):
+                    block0(v0: i32):
     [DynRexOp1rcmp#39]        v1 = ifcmp v0, v0
     [Op1ld#8b]                v2 = load.i32 v0
     [Op2seti_abcd#490]        v3 = trueif ugt v1
@@ -42,35 +42,35 @@ function %live_across_load(i32) -> i32 {
     [Op1ret#c3]               return v4
 }
 
-; Correct use of CPU flags across EBB.
-function %live_across_ebb(i32) -> i32 {
-                          ebb0(v0: i32):
+; Correct use of CPU flags across block.
+function %live_across_block(i32) -> i32 {
+                          block0(v0: i32):
     [DynRexOp1rcmp#39]        v1 = ifcmp v0, v0
-    [Op1jmpb#eb]              jump ebb1
-                          ebb1:
+    [Op1jmpb#eb]              jump block1
+                          block1:
     [Op2seti_abcd#490]        v2 = trueif ugt v1
     [Op2urm_noflags_abcd#4b6] v3 = bint.i32 v2
     [Op1ret#c3]               return v3
 }
 
-function %live_across_ebb_backwards(i32) -> i32 {
-                          ebb0(v0: i32):
-    [Op1jmpb#eb]              jump ebb2
-                          ebb1:
+function %live_across_block_backwards(i32) -> i32 {
+                          block0(v0: i32):
+    [Op1jmpb#eb]              jump block2
+                          block1:
     [Op2seti_abcd#490]        v2 = trueif ugt v1
     [Op2urm_noflags_abcd#4b6] v3 = bint.i32 v2
     [Op1ret#c3]               return v3
-                          ebb2:
+                          block2:
     [DynRexOp1rcmp#39]        v1 = ifcmp v0, v0
-    [Op1jmpb#eb]              jump ebb1
+    [Op1jmpb#eb]              jump block1
 }
 
 ; Flags live into loop.
 function %live_into_loop(i32) -> i32 {
-                    ebb0(v0: i32):
+                    block0(v0: i32):
     [DynRexOp1rcmp#39]  v1 = ifcmp v0, v0
-    [Op1jmpb#eb]        jump ebb1
-                    ebb1:
+    [Op1jmpb#eb]        jump block1
+                    block1:
     [Op2seti_abcd#490]  v2 = trueif ugt v1
-    [Op1jmpb#eb]        jump ebb1
+    [Op1jmpb#eb]        jump block1
 }
diff --git a/cranelift/filetests/filetests/verifier/globals.clif b/cranelift/filetests/filetests/verifier/globals.clif
index 4882cae2ee..1a44cf8001 100644
--- a/cranelift/filetests/filetests/verifier/globals.clif
+++ b/cranelift/filetests/filetests/verifier/globals.clif
@@ -6,14 +6,14 @@ function %load_base_type(i64 vmctx) {
     gv1 = load.i32 notrap aligned gv0
     gv2 = load.i32 notrap aligned gv1 ; error: base gv1 has type i32, which is not the pointer type i64
 
-ebb0(v0: i64):
+block0(v0: i64):
     return
 }
 
 function %global_value_wrong_type(i64 vmctx) {
     gv0 = vmctx
 
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = global_value.i32 gv0 ; error: global_value instruction with type i32 references global value with type i64
     return
 }
diff --git a/cranelift/filetests/filetests/verifier/heap.clif b/cranelift/filetests/filetests/verifier/heap.clif
index 2c73b726ba..ffd6bb7ac4 100644
--- a/cranelift/filetests/filetests/verifier/heap.clif
+++ b/cranelift/filetests/filetests/verifier/heap.clif
@@ -6,7 +6,7 @@ function %heap_base_type(i64 vmctx) {
     gv1 = load.i32 notrap aligned gv0
     heap0 = static gv1, offset_guard 0x1000, bound 0x1_0000, index_type i32 ; error: heap base has type i32, which is not the pointer type i64
 
-ebb0(v0: i64):
+block0(v0: i64):
     return
 }
 
@@ -14,7 +14,7 @@ function %invalid_base(i64 vmctx) {
     gv0 = vmctx
     heap0 = dynamic gv1, bound gv0, offset_guard 0x1000, index_type i64 ; error: invalid base global value gv1
 
-ebb0(v0: i64):
+block0(v0: i64):
     return
 }
 
@@ -22,7 +22,7 @@ function %invalid_bound(i64 vmctx) {
     gv0 = vmctx
     heap0 = dynamic gv0, bound gv1, offset_guard 0x1000, index_type i64 ; error: invalid bound global value gv1
 
-ebb0(v0: i64):
+block0(v0: i64):
     return
 }
 
@@ -31,7 +31,7 @@ function %heap_bound_type(i64 vmctx) {
     gv1 = load.i16 notrap aligned gv0
     heap0 = dynamic gv0, bound gv1, offset_guard 0x1000, index_type i32 ; error: heap index type i32 differs from the type of its bound, i16
 
-ebb0(v0: i64):
+block0(v0: i64):
     return
 }
 
@@ -39,7 +39,7 @@ function %heap_addr_index_type(i64 vmctx, i64) {
     gv0 = vmctx
     heap0 = static gv0, offset_guard 0x1000, bound 0x1_0000, index_type i32
 
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = heap_addr.i64 heap0, v1, 0; error: index type i64 differs from heap index type i32
     return
 }
diff --git a/cranelift/filetests/filetests/verifier/jump_table.clif b/cranelift/filetests/filetests/verifier/jump_table.clif
index bafd751f2e..67cd935320 100644
--- a/cranelift/filetests/filetests/verifier/jump_table.clif
+++ b/cranelift/filetests/filetests/verifier/jump_table.clif
@@ -1,19 +1,19 @@
 test verifier
 
 function %br_invalid_default(i64) {
-    jt0 = jump_table [ebb1, ebb1]
+    jt0 = jump_table [block1, block1]
 
-ebb0(v0: i64):
-    br_table.i64 v0, ebb2, jt0 ; error: invalid ebb reference ebb2
-ebb1:
+block0(v0: i64):
+    br_table.i64 v0, block2, jt0 ; error: invalid block reference block2
+block1:
     return
 }
 
 function %br(i64) {
-    jt0 = jump_table [ebb1, ebb2] ; error: invalid ebb reference ebb2
+    jt0 = jump_table [block1, block2] ; error: invalid block reference block2
 
-ebb0(v0: i64):
-    br_table.i64 v0, ebb1, jt0
-ebb1:
+block0(v0: i64):
+    br_table.i64 v0, block1, jt0
+block1:
     return
 }
diff --git a/cranelift/filetests/filetests/verifier/memory.clif b/cranelift/filetests/filetests/verifier/memory.clif
index cbaddcb13b..496b71c815 100644
--- a/cranelift/filetests/filetests/verifier/memory.clif
+++ b/cranelift/filetests/filetests/verifier/memory.clif
@@ -4,13 +4,13 @@ function %cycle() {
     gv0 = load.i32 notrap aligned gv1 ; error: global value cycle: [gv0, gv1]
     gv1 = load.i32 notrap aligned gv0-32
 
-ebb1:
+block1:
     return
 }
 
 function %self_cycle() {
     gv0 = load.i32 notrap aligned gv0 ; error: global value cycle: [gv0]
 
-ebb1:
+block1:
     return
 }
diff --git a/cranelift/filetests/filetests/verifier/scalar-to-vector.clif b/cranelift/filetests/filetests/verifier/scalar-to-vector.clif
index 927abdafc2..1d04db9957 100644
--- a/cranelift/filetests/filetests/verifier/scalar-to-vector.clif
+++ b/cranelift/filetests/filetests/verifier/scalar-to-vector.clif
@@ -3,7 +3,7 @@ set enable_simd=true
 target x86_64
 
 function %scalar_to_vector() {
-ebb0:
+block0:
     v0 = iconst.i32 42
     v1 = scalar_to_vector.f32x4 v0 ; error: arg 0 (v0) has type i32, expected f32
     return
diff --git a/cranelift/filetests/filetests/verifier/simd-lane-index.clif b/cranelift/filetests/filetests/verifier/simd-lane-index.clif
index 064254c0e2..2f7ca8d095 100644
--- a/cranelift/filetests/filetests/verifier/simd-lane-index.clif
+++ b/cranelift/filetests/filetests/verifier/simd-lane-index.clif
@@ -3,7 +3,7 @@ set enable_simd
 target x86_64
 
 function %insertlane_i32x4() {
-ebb0:
+block0:
     v0 = vconst.i32x4 [0 0 0 0]
     v1 = iconst.i32 42
     v2 = insertlane v0, 4, v1 ; error: The lane 4 does not index into the type i32x4
@@ -11,7 +11,7 @@ ebb0:
 }
 
 function %insertlane_b16x8() {
-ebb0:
+block0:
     v0 = vconst.b16x8 [false false false false false false false false]
     v1 = bconst.b16 true
     v2 = insertlane v0, 8, v1 ; error: The lane 8 does not index into the type b16x8
@@ -19,7 +19,7 @@ ebb0:
 }
 
 function %insertlane_f64x2() {
-ebb0:
+block0:
     v0 = vconst.f64x2 0x00
     v1 = f64const 0x0.1
     v2 = insertlane v0, 2, v1 ; error: The lane 2 does not index into the type f64x2
@@ -27,14 +27,14 @@ ebb0:
 }
 
 function %extractlane_i32x4() {
-ebb0:
+block0:
     v0 = vconst.i32x4 [0 0 0 0]
     v1 = extractlane v0, 4 ; error: The lane 4 does not index into the type i32x4
     return
 }
 
 function %extractlane_b8x16() {
-ebb0:
+block0:
     v0 = vconst.b8x16 0x00
     v1 = extractlane v0, 16 ; error: The lane 16 does not index into the type b8x16
     return
diff --git a/cranelift/filetests/filetests/verifier/table.clif b/cranelift/filetests/filetests/verifier/table.clif
index 7502e00044..204ae5c93a 100644
--- a/cranelift/filetests/filetests/verifier/table.clif
+++ b/cranelift/filetests/filetests/verifier/table.clif
@@ -6,7 +6,7 @@ function %table_base_type(i64 vmctx) {
     gv1 = load.i32 notrap aligned gv0
     table0 = dynamic gv1, element_size 1, bound gv1, index_type i32 ; error: table base has type i32, which is not the pointer type i64
 
-ebb0(v0: i64):
+block0(v0: i64):
     return
 }
 
@@ -14,7 +14,7 @@ function %invalid_base(i64 vmctx) {
     gv0 = vmctx
     table0 = dynamic gv1, bound gv0, element_size 1, index_type i64 ; error: invalid base global value gv1
 
-ebb0(v0: i64):
+block0(v0: i64):
     return
 }
 
@@ -22,7 +22,7 @@ function %invalid_bound(i64 vmctx) {
     gv0 = vmctx
     table0 = dynamic gv0, bound gv1, element_size 1, index_type i64 ; error: invalid bound global value gv1
 
-ebb0(v0: i64):
+block0(v0: i64):
     return
 }
 
@@ -31,7 +31,7 @@ function %table_bound_type(i64 vmctx) {
     gv1 = load.i16 notrap aligned gv0
     table0 = dynamic gv0, bound gv1, element_size 1, index_type i32 ; error: table index type i32 differs from the type of its bound, i16
 
-ebb0(v0: i64):
+block0(v0: i64):
     return
 }
 
@@ -40,7 +40,7 @@ function %table_addr_index_type(i64 vmctx, i64) {
     gv1 = load.i32 notrap aligned gv0
     table0 = dynamic gv0, element_size 1, bound gv1, index_type i32
 
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = table_addr.i64 table0, v1, +0; error: index type i64 differs from table index type i32
     return
 }
diff --git a/cranelift/filetests/filetests/verifier/type_check.clif b/cranelift/filetests/filetests/verifier/type_check.clif
index ce61b81a10..c708ca76ad 100644
--- a/cranelift/filetests/filetests/verifier/type_check.clif
+++ b/cranelift/filetests/filetests/verifier/type_check.clif
@@ -1,40 +1,40 @@
 test verifier
 
 function %entry_block_signature_mismatch(i32) {
-    ebb0: ; error: entry block parameters (0) must match function signature (1)
+    block0: ; error: entry block parameters (0) must match function signature (1)
         return
 }
 
 function %entry_block_arg_type(i32) {
-    ebb0(v0: f32): ; error: entry block parameter 0 expected to have type i32, got f32
+    block0(v0: f32): ; error: entry block parameter 0 expected to have type i32, got f32
         return
 }
 
 function %incorrect_arg_type(i32, b1) -> i32 {
-    ebb0(v0: i32, v1: b1):
+    block0(v0: i32, v1: b1):
         v2 = iadd v0, v1 ; error: arg 1 (v1) has type b1, expected i32
         return v2
 }
 
 function %incorrect_return_type() -> f32 {
-    ebb0:
+    block0:
         v0 = iconst.i32 1
         return v0 ; error: arg 0 (v0) has type i32, must match function signature of f32
 }
 
 function %too_many_return_values() {
-    ebb0:
+    block0:
         v0 = iconst.i32 1
         return v0 ; error: arguments of return must match function signature
 }
 
 function %too_few_return_values() -> f32, i64 {
-    ebb0:
+    block0:
         return ; error: arguments of return must match function signature
 }
 
 function %type_mismatch_controlling_variable() {
-    ebb0:
+    block0:
         v0 = iconst.i32 5
         v1 = iconst.i64 6
         v2 = iadd v0, v1 ; error: arg 1 (v1) has type i64, expected i32
@@ -43,14 +43,14 @@ function %type_mismatch_controlling_variable() {
 
 function %fn_call_too_few_args() {
     fn2 = %great_fn(i32, f32)
-    ebb0:
+    block0:
         call fn2() ; error: mismatched argument count for `call fn2()`: got 0, expected 2
         return
 }
 
 function %fn_call_too_many_args() {
     fn5 = %best_fn()
-    ebb0:
+    block0:
         v0 = iconst.i64 56
         v1 = f32const 0.0
         call fn5(v0, v1) ; error: mismatched argument count for `call fn5(v0, v1)`: got 2, expected 0
@@ -59,56 +59,56 @@ function %fn_call_too_many_args() {
 
 function %fn_call_incorrect_arg_type(i64) {
     sig9 = (f32)
-    ebb0(v0: i64):
+    block0(v0: i64):
         v1 = iconst.i32 56
         call_indirect sig9, v0(v1) ; error: arg 0 (v1) has type i32, expected f32
         return
 }
 
-; TODO: Should we instead just verify that jump tables contain no EBBs that take arguments? This
+; TODO: Should we instead just verify that jump tables contain no blocks that take arguments? This
 ; error doesn't occur if no instruction uses the jump table.
 function %jump_table_args() {
-    jt1 = jump_table [ebb1]
-    ebb0:
+    jt1 = jump_table [block1]
+    block0:
         v0 = iconst.i32 0
-        br_table v0, ebb2, jt1 ; error: takes no arguments, but had target ebb1 with 1 arguments
+        br_table v0, block2, jt1 ; error: takes no arguments, but had target block1 with 1 arguments
 
-    ebb1(v5: i32):
+    block1(v5: i32):
         return
-    ebb2:
+    block2:
         return
 }
 
 function %jump_args() {
-    ebb0:
+    block0:
         v0 = iconst.i16 10
         v3 = iconst.i64 20
-        jump ebb1(v0, v3) ; error: arg 0 (v0) has type i16, expected i64
+        jump block1(v0, v3) ; error: arg 0 (v0) has type i16, expected i64
                           ; error: arg 1 (v3) has type i64, expected i16
-    ebb1(v10: i64, v11: i16):
+    block1(v10: i64, v11: i16):
         return
 }
 
 function %jump_args2() {
-    ebb0:
+    block0:
         v0 = iconst.i16 10
         v3 = iconst.i64 20
-        brz v0, ebb1(v0, v3) ; error: arg 0 (v0) has type i16, expected i64
+        brz v0, block1(v0, v3) ; error: arg 0 (v0) has type i16, expected i64
                              ; error: arg 1 (v3) has type i64, expected i16
-        jump ebb1(v3, v0)
-    ebb1(v10: i64, v11: i16):
+        jump block1(v3, v0)
+    block1(v10: i64, v11: i16):
         return
 }
 
 function %bad_extend() {
-ebb0:
+block0:
     v0 = iconst.i32 10
     v1 = uextend.i16 v0 ; error: input i32 must be smaller than output i16
     return
 }
 
 function %bad_reduce() {
-ebb0:
+block0:
     v0 = iconst.i32 10
     v1 = ireduce.i64 v0 ; error: input i32 must be larger than output i64
     return
diff --git a/cranelift/filetests/filetests/verifier/undeclared_vmctx.clif b/cranelift/filetests/filetests/verifier/undeclared_vmctx.clif
index 1b760b3ff3..a48e7a0ef6 100644
--- a/cranelift/filetests/filetests/verifier/undeclared_vmctx.clif
+++ b/cranelift/filetests/filetests/verifier/undeclared_vmctx.clif
@@ -3,7 +3,7 @@ test verifier
 ; Using a vmctx global value without declaring it first leads to an error.
 function %vmglobal_err(i64) -> i64 {
     gv4 = vmctx ; error: undeclared vmctx reference
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = global_value.i64 gv4
     return v1
 }
@@ -11,7 +11,7 @@ ebb0(v0: i64):
 ; If it is declared, all is fine.
 function %vmglobal_ok(i64 vmctx) -> i64 {
     gv4 = vmctx
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = global_value.i64 gv4
     return v1
 }
diff --git a/cranelift/filetests/filetests/verifier/unreachable_code.clif b/cranelift/filetests/filetests/verifier/unreachable_code.clif
index 7ea7dd49b0..0a12aac8d0 100644
--- a/cranelift/filetests/filetests/verifier/unreachable_code.clif
+++ b/cranelift/filetests/filetests/verifier/unreachable_code.clif
@@ -1,22 +1,22 @@
 test verifier
 
 function %test() -> i32 {   ; Ok
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = iconst.i32 0
-    jump ebb2
+    jump block2
 
-ebb2:
-    jump ebb4
+block2:
+    jump block4
 
-ebb4:
-    jump ebb2
+block4:
+    jump block2
 
-ebb3(v2: i32):
+block3(v2: i32):
     v4 = iadd.i32 v1, v2
-    jump ebb9(v4)
+    jump block9(v4)
 
-ebb9(v7: i32):
+block9(v7: i32):
     v9 = iadd.i32 v2, v7
     return v9
 
@@ -24,22 +24,22 @@ ebb9(v7: i32):
 
 ; Using a function argument in an unreachable block is ok.
 function %arg(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iadd_imm v0, 1
     return v1
 
-ebb1:
+block1:
     v10 = iadd_imm v0, 10
     return v10
 }
 
-; Using an EBB argument from an unreachable block is not ok.
+; Using a block argument from an unreachable block is not ok.
 function %arg2(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iadd v0, v10   ; error: uses value arg from non-dominating
     return v1
 
-ebb1(v10: i32):
+block1(v10: i32):
     v11 = iadd v0, v10
     return v11
 }
diff --git a/cranelift/filetests/filetests/wasm/control.clif b/cranelift/filetests/filetests/wasm/control.clif
index 66c82df937..d00c5b3166 100644
--- a/cranelift/filetests/filetests/wasm/control.clif
+++ b/cranelift/filetests/filetests/wasm/control.clif
@@ -6,60 +6,60 @@ target i686 haswell
 target x86_64 haswell
 
 function %br_if(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 1
-    brz v0, ebb1(v1)
-    jump ebb2
+    brz v0, block1(v1)
+    jump block2
 
-ebb1(v2: i32):
+block1(v2: i32):
     return v2
 
-ebb2:
-    jump ebb1(v0)
+block2:
+    jump block1(v0)
 }
 
 function %br_if_not(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 1
-    brnz v0, ebb1(v0)
-    jump ebb2
+    brnz v0, block1(v0)
+    jump block2
 
-ebb1(v2: i32):
+block1(v2: i32):
     return v2
 
-ebb2:
-    jump ebb1(v0)
+block2:
+    jump block1(v0)
 }
 
 function %br_if_fallthrough(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = iconst.i32 1
-    brz v0, ebb1(v1)
+    brz v0, block1(v1)
     ; This jump gets converted to a fallthrough.
-    jump ebb1(v0)
+    jump block1(v0)
 
-ebb1(v2: i32):
+block1(v2: i32):
     return v2
 }
 
 function %undefined() {
-ebb0:
+block0:
     trap user0
 }
 
 function %br_table(i32) {
-jt0 = jump_table [ebb3, ebb1, ebb2]
+jt0 = jump_table [block3, block1, block2]
 
-ebb0(v0: i32):
-    br_table v0, ebb4, jt0
+block0(v0: i32):
+    br_table v0, block4, jt0
 
-ebb4:
+block4:
     trap oob
 
-ebb1:
+block1:
     return
-ebb2:
+block2:
     return
-ebb3:
+block3:
     return
 }
diff --git a/cranelift/filetests/filetests/wasm/conversions.clif b/cranelift/filetests/filetests/wasm/conversions.clif
index 6784637136..33602166b4 100644
--- a/cranelift/filetests/filetests/wasm/conversions.clif
+++ b/cranelift/filetests/filetests/wasm/conversions.clif
@@ -4,199 +4,199 @@ test compile
 target x86_64 haswell
 
 function %i32_wrap_i64(i64) -> i32 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = ireduce.i32 v0
     return v1
 }
 
 function %i64_extend_s_i32(i32) -> i64 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = sextend.i64 v0
     return v1
 }
 
 function %i64_extend_u_i32(i32) -> i64 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = uextend.i64 v0
     return v1
 }
 
 function %i32_trunc_s_f32(f32) -> i32 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = fcvt_to_sint.i32 v0
     return v1
 }
 
 function %i32_trunc_u_f32(f32) -> i32 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = fcvt_to_uint.i32 v0
     return v1
 }
 
 function %i32_trunc_s_f64(f64) -> i32 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = fcvt_to_sint.i32 v0
     return v1
 }
 
 function %i32_trunc_u_f64(f64) -> i32 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = fcvt_to_uint.i32 v0
     return v1
 }
 
 function %i64_trunc_s_f32(f32) -> i64 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = fcvt_to_sint.i64 v0
     return v1
 }
 
 function %i64_trunc_u_f32(f32) -> i64 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = fcvt_to_uint.i64 v0
     return v1
 }
 
 function %i64_trunc_s_f64(f64) -> i64 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = fcvt_to_sint.i64 v0
     return v1
 }
 
 function %i64_trunc_u_f64(f64) -> i64 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = fcvt_to_uint.i64 v0
     return v1
 }
 
 function %i32_trunc_s_sat_f32(f32) -> i32 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = fcvt_to_sint_sat.i32 v0
     return v1
 }
 
 function %i32_trunc_u_sat_f32(f32) -> i32 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = fcvt_to_uint_sat.i32 v0
     return v1
 }
 
 function %i32_trunc_s_sat_f64(f64) -> i32 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = fcvt_to_sint_sat.i32 v0
     return v1
 }
 
 function %i32_trunc_u_sat_f64(f64) -> i32 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = fcvt_to_uint_sat.i32 v0
     return v1
 }
 
 function %i64_trunc_s_sat_f32(f32) -> i64 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = fcvt_to_sint_sat.i64 v0
     return v1
 }
 
 function %i64_trunc_u_sat_f32(f32) -> i64 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = fcvt_to_uint_sat.i64 v0
     return v1
 }
 
 function %i64_trunc_s_sat_f64(f64) -> i64 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = fcvt_to_sint_sat.i64 v0
     return v1
 }
 
 function %i64_trunc_u_sat_f64(f64) -> i64 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = fcvt_to_uint_sat.i64 v0
     return v1
 }
 
 function %f32_trunc_f64(f64) -> f32 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = fdemote.f32 v0
     return v1
 }
 
 function %f64_promote_f32(f32) -> f64 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = fpromote.f64 v0
     return v1
 }
 
 function %f32_convert_s_i32(i32) -> f32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = fcvt_from_sint.f32 v0
     return v1
 }
 
 function %f32_convert_u_i32(i32) -> f32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = fcvt_from_uint.f32 v0
     return v1
 }
 
 function %f64_convert_s_i32(i32) -> f64 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = fcvt_from_sint.f64 v0
     return v1
 }
 
 function %f64_convert_u_i32(i32) -> f64 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = fcvt_from_uint.f64 v0
     return v1
 }
 
 function %f32_convert_s_i64(i64) -> f32 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = fcvt_from_sint.f32 v0
     return v1
 }
 
 function %f32_convert_u_i64(i64) -> f32 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = fcvt_from_uint.f32 v0
     return v1
 }
 
 function %f64_convert_s_i64(i64) -> f64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = fcvt_from_sint.f64 v0
     return v1
 }
 
 function %f64_convert_u_i64(i64) -> f64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = fcvt_from_uint.f64 v0
     return v1
 }
 
 function %i32_reinterpret_f32(f32) -> i32 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = bitcast.i32 v0
     return v1
 }
 
 function %f32_reinterpret_i32(i32) -> f32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = bitcast.f32 v0
     return v1
 }
 
 function %i64_reinterpret_f64(f64) -> i64 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = bitcast.i64 v0
     return v1
 }
 
 function %f64_reinterpret_i64(i64) -> f64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = bitcast.f64 v0
     return v1
 }
diff --git a/cranelift/filetests/filetests/wasm/f32-arith.clif b/cranelift/filetests/filetests/wasm/f32-arith.clif
index c439d2e798..b7a83f5434 100644
--- a/cranelift/filetests/filetests/wasm/f32-arith.clif
+++ b/cranelift/filetests/filetests/wasm/f32-arith.clif
@@ -9,7 +9,7 @@ target x86_64 baseline
 ; Constants.
 
 function %f32_const() -> f32 {
-ebb0:
+block0:
     v1 = f32const 0x3.0
     return v1
 }
@@ -17,43 +17,43 @@ ebb0:
 ; Unary operations
 
 function %f32_abs(f32) -> f32 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = fabs v0
     return v1
 }
 
 function %f32_neg(f32) -> f32 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = fneg v0
     return v1
 }
 
 function %f32_sqrt(f32) -> f32 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = sqrt v0
     return v1
 }
 
 function %f32_ceil(f32) -> f32 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = ceil v0
     return v1
 }
 
 function %f32_floor(f32) -> f32 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = floor v0
     return v1
 }
 
 function %f32_trunc(f32) -> f32 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = trunc v0
     return v1
 }
 
 function %f32_nearest (f32) -> f32 {
-ebb0(v0: f32):
+block0(v0: f32):
     v1 = nearest v0
     return v1
 }
@@ -61,43 +61,43 @@ ebb0(v0: f32):
 ; Binary Operations
 
 function %f32_add(f32, f32) -> f32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fadd v0, v1
     return v2
 }
 
 function %f32_sub(f32, f32) -> f32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fsub v0, v1
     return v2
 }
 
 function %f32_mul(f32, f32) -> f32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fmul v0, v1
     return v2
 }
 
 function %f32_div(f32, f32) -> f32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fdiv v0, v1
     return v2
 }
 
 function %f32_min(f32, f32) -> f32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fmin v0, v1
     return v2
 }
 
 function %f32_max(f32, f32) -> f32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fmax v0, v1
     return v2
 }
 
 function %f32_copysign(f32, f32) -> f32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fcopysign v0, v1
     return v2
 }
diff --git a/cranelift/filetests/filetests/wasm/f32-compares.clif b/cranelift/filetests/filetests/wasm/f32-compares.clif
index e314646b1c..e569a94821 100644
--- a/cranelift/filetests/filetests/wasm/f32-compares.clif
+++ b/cranelift/filetests/filetests/wasm/f32-compares.clif
@@ -6,42 +6,42 @@ target i686 haswell
 target x86_64 haswell
 
 function %f32_eq(f32, f32) -> i32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fcmp eq v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %f32_ne(f32, f32) -> i32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fcmp ne v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %f32_lt(f32, f32) -> i32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fcmp lt v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %f32_gt(f32, f32) -> i32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fcmp gt v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %f32_le(f32, f32) -> i32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fcmp le v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %f32_ge(f32, f32) -> i32 {
-ebb0(v0: f32, v1: f32):
+block0(v0: f32, v1: f32):
     v2 = fcmp ge v0, v1
     v3 = bint.i32 v2
     return v3
diff --git a/cranelift/filetests/filetests/wasm/f32-memory64.clif b/cranelift/filetests/filetests/wasm/f32-memory64.clif
index edc5c22780..33e3100537 100644
--- a/cranelift/filetests/filetests/wasm/f32-memory64.clif
+++ b/cranelift/filetests/filetests/wasm/f32-memory64.clif
@@ -9,7 +9,7 @@ function %f32_load(i32, i64 vmctx) -> f32 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = load.f32 v2
     return v3
@@ -19,7 +19,7 @@ function %f32_store(f32, i32, i64 vmctx) {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: f32, v1: i32, v2: i64):
+block0(v0: f32, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
     store v0, v3
     return
diff --git a/cranelift/filetests/filetests/wasm/f64-arith.clif b/cranelift/filetests/filetests/wasm/f64-arith.clif
index 7209ed6024..cecd954f90 100644
--- a/cranelift/filetests/filetests/wasm/f64-arith.clif
+++ b/cranelift/filetests/filetests/wasm/f64-arith.clif
@@ -7,7 +7,7 @@ target x86_64 baseline
 ; Constants.
 
 function %f64_const() -> f64 {
-ebb0:
+block0:
     v1 = f64const 0x3.0
     return v1
 }
@@ -15,43 +15,43 @@ ebb0:
 ; Unary operations
 
 function %f64_abs(f64) -> f64 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = fabs v0
     return v1
 }
 
 function %f64_neg(f64) -> f64 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = fneg v0
     return v1
 }
 
 function %f64_sqrt(f64) -> f64 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = sqrt v0
     return v1
 }
 
 function %f64_ceil(f64) -> f64 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = ceil v0
     return v1
 }
 
 function %f64_floor(f64) -> f64 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = floor v0
     return v1
 }
 
 function %f64_trunc(f64) -> f64 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = trunc v0
     return v1
 }
 
 function %f64_nearest (f64) -> f64 {
-ebb0(v0: f64):
+block0(v0: f64):
     v1 = nearest v0
     return v1
 }
@@ -59,43 +59,43 @@ ebb0(v0: f64):
 ; Binary Operations
 
 function %f64_add(f64, f64) -> f64 {
-ebb0(v0: f64, v1: f64):
+block0(v0: f64, v1: f64):
     v2 = fadd v0, v1
     return v2
 }
 
 function %f64_sub(f64, f64) -> f64 {
-ebb0(v0: f64, v1: f64):
+block0(v0: f64, v1: f64):
     v2 = fsub v0, v1
     return v2
 }
 
 function %f64_mul(f64, f64) -> f64 {
-ebb0(v0: f64, v1: f64):
+block0(v0: f64, v1: f64):
     v2 = fmul v0, v1
     return v2
 }
 
 function %f64_div(f64, f64) -> f64 {
-ebb0(v0: f64, v1: f64):
+block0(v0: f64, v1: f64):
     v2 = fdiv v0, v1
     return v2
 }
 
 function %f64_min(f64, f64) -> f64 {
-ebb0(v0: f64, v1: f64):
+block0(v0: f64, v1: f64):
     v2 = fmin v0, v1
     return v2
 }
 
 function %f64_max(f64, f64) -> f64 {
-ebb0(v0: f64, v1: f64):
+block0(v0: f64, v1: f64):
     v2 = fmax v0, v1
     return v2
 }
 
 function %f64_copysign(f64, f64) -> f64 {
-ebb0(v0: f64, v1: f64):
+block0(v0: f64, v1: f64):
     v2 = fcopysign v0, v1
     return v2
 }
diff --git a/cranelift/filetests/filetests/wasm/f64-compares.clif b/cranelift/filetests/filetests/wasm/f64-compares.clif
index 5d51ebb443..b75a7634bf 100644
--- a/cranelift/filetests/filetests/wasm/f64-compares.clif
+++ b/cranelift/filetests/filetests/wasm/f64-compares.clif
@@ -6,42 +6,42 @@ target i686 haswell
 target x86_64 haswell
 
 function %f64_eq(f64, f64) -> i32 {
-ebb0(v0: f64, v1: f64):
+block0(v0: f64, v1: f64):
     v2 = fcmp eq v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %f64_ne(f64, f64) -> i32 {
-ebb0(v0: f64, v1: f64):
+block0(v0: f64, v1: f64):
     v2 = fcmp ne v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %f64_lt(f64, f64) -> i32 {
-ebb0(v0: f64, v1: f64):
+block0(v0: f64, v1: f64):
     v2 = fcmp lt v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %f64_gt(f64, f64) -> i32 {
-ebb0(v0: f64, v1: f64):
+block0(v0: f64, v1: f64):
     v2 = fcmp gt v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %f64_le(f64, f64) -> i32 {
-ebb0(v0: f64, v1: f64):
+block0(v0: f64, v1: f64):
     v2 = fcmp le v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %f64_ge(f64, f64) -> i32 {
-ebb0(v0: f64, v1: f64):
+block0(v0: f64, v1: f64):
     v2 = fcmp ge v0, v1
     v3 = bint.i32 v2
     return v3
diff --git a/cranelift/filetests/filetests/wasm/f64-memory64.clif b/cranelift/filetests/filetests/wasm/f64-memory64.clif
index 85351d9d8d..c0a58de4a1 100644
--- a/cranelift/filetests/filetests/wasm/f64-memory64.clif
+++ b/cranelift/filetests/filetests/wasm/f64-memory64.clif
@@ -9,7 +9,7 @@ function %f64_load(i32, i64 vmctx) -> f64 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = load.f64 v2
     return v3
@@ -19,7 +19,7 @@ function %f64_store(f64, i32, i64 vmctx) {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: f64, v1: i32, v2: i64):
+block0(v0: f64, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
     store v0, v3
     return
diff --git a/cranelift/filetests/filetests/wasm/i32-arith.clif b/cranelift/filetests/filetests/wasm/i32-arith.clif
index 8c92613150..cb9597741b 100644
--- a/cranelift/filetests/filetests/wasm/i32-arith.clif
+++ b/cranelift/filetests/filetests/wasm/i32-arith.clif
@@ -9,7 +9,7 @@ target x86_64 baseline
 ; Constants.
 
 function %i32_const() -> i32 {
-ebb0:
+block0:
     v0 = iconst.i32 0x8765_4321
     return v0
 }
@@ -17,19 +17,19 @@ ebb0:
 ; Unary operations.
 
 function %i32_clz(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = clz v0
     return v1
 }
 
 function %i32_ctz(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = ctz v0
     return v1
 }
 
 function %i32_popcnt(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = popcnt v0
     return v1
 }
@@ -37,91 +37,91 @@ ebb0(v0: i32):
 ; Binary operations.
 
 function %i32_add(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = iadd v0, v1
     return v2
 }
 
 function %i32_sub(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = isub v0, v1
     return v2
 }
 
 function %i32_mul(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = imul v0, v1
     return v2
 }
 
 function %i32_div_s(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = sdiv v0, v1
     return v2
 }
 
 function %i32_div_u(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = udiv v0, v1
     return v2
 }
 
 function %i32_rem_s(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = srem v0, v1
     return v2
 }
 
 function %i32_rem_u(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = urem v0, v1
     return v2
 }
 
 function %i32_and(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = band v0, v1
     return v2
 }
 
 function %i32_or(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = bor v0, v1
     return v2
 }
 
 function %i32_xor(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = bxor v0, v1
     return v2
 }
 
 function %i32_shl(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = ishl v0, v1
     return v2
 }
 
 function %i32_shr_s(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = sshr v0, v1
     return v2
 }
 
 function %i32_shr_u(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = ushr v0, v1
     return v2
 }
 
 function %i32_rotl(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = rotl v0, v1
     return v2
 }
 
 function %i32_rotr(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = rotr v0, v1
     return v2
 }
diff --git a/cranelift/filetests/filetests/wasm/i32-compares.clif b/cranelift/filetests/filetests/wasm/i32-compares.clif
index 1b43c5938a..f5be0a25c1 100644
--- a/cranelift/filetests/filetests/wasm/i32-compares.clif
+++ b/cranelift/filetests/filetests/wasm/i32-compares.clif
@@ -6,77 +6,77 @@ target i686 haswell
 target x86_64 haswell
 
 function %i32_eqz(i32) -> i32 {
-ebb0(v0: i32):
+block0(v0: i32):
     v1 = icmp_imm eq v0, 0
     v2 = bint.i32 v1
     return v2
 }
 
 function %i32_eq(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = icmp eq v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i32_ne(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = icmp ne v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i32_lt_s(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = icmp slt v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i32_lt_u(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = icmp ult v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i32_gt_s(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = icmp sgt v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i32_gt_u(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = icmp ugt v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i32_le_s(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = icmp sle v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i32_le_u(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = icmp ule v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i32_ge_s(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = icmp sge v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i32_ge_u(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = icmp uge v0, v1
     v3 = bint.i32 v2
     return v3
diff --git a/cranelift/filetests/filetests/wasm/i32-memory64.clif b/cranelift/filetests/filetests/wasm/i32-memory64.clif
index 306c4e4aa5..b1418c5ed1 100644
--- a/cranelift/filetests/filetests/wasm/i32-memory64.clif
+++ b/cranelift/filetests/filetests/wasm/i32-memory64.clif
@@ -9,7 +9,7 @@ function %i32_load(i32, i64 vmctx) -> i32 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = load.i32 v2
     return v3
@@ -19,7 +19,7 @@ function %i32_store(i32, i32, i64 vmctx) {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i32, v2: i64):
+block0(v0: i32, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
     store v0, v3
     return
@@ -29,7 +29,7 @@ function %i32_load8_s(i32, i64 vmctx) -> i32 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = sload8.i32 v2
     return v3
@@ -39,7 +39,7 @@ function %i32_load8_u(i32, i64 vmctx) -> i32 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = uload8.i32 v2
     return v3
@@ -49,7 +49,7 @@ function %i32_store8(i32, i32, i64 vmctx) {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i32, v2: i64):
+block0(v0: i32, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
     istore8 v0, v3
     return
@@ -59,7 +59,7 @@ function %i32_load16_s(i32, i64 vmctx) -> i32 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = sload16.i32 v2
     return v3
@@ -69,7 +69,7 @@ function %i32_load16_u(i32, i64 vmctx) -> i32 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = uload16.i32 v2
     return v3
@@ -79,7 +79,7 @@ function %i32_store16(i32, i32, i64 vmctx) {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i32, v2: i64):
+block0(v0: i32, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
     istore16 v0, v3
     return
diff --git a/cranelift/filetests/filetests/wasm/i64-arith.clif b/cranelift/filetests/filetests/wasm/i64-arith.clif
index c4cd4a4507..b457f9942d 100644
--- a/cranelift/filetests/filetests/wasm/i64-arith.clif
+++ b/cranelift/filetests/filetests/wasm/i64-arith.clif
@@ -7,7 +7,7 @@ target x86_64 baseline
 ; Constants.
 
 function %i64_const() -> i64 {
-ebb0:
+block0:
     v0 = iconst.i64 0x8765_4321
     return v0
 }
@@ -15,19 +15,19 @@ ebb0:
 ; Unary operations.
 
 function %i64_clz(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = clz v0
     return v1
 }
 
 function %i64_ctz(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = ctz v0
     return v1
 }
 
 function %i64_popcnt(i64) -> i64 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = popcnt v0
     return v1
 }
@@ -35,91 +35,91 @@ ebb0(v0: i64):
 ; Binary operations.
 
 function %i64_add(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = iadd v0, v1
     return v2
 }
 
 function %i64_sub(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = isub v0, v1
     return v2
 }
 
 function %i64_mul(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = imul v0, v1
     return v2
 }
 
 function %i32_div_s(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = sdiv v0, v1
     return v2
 }
 
 function %i32_div_u(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = udiv v0, v1
     return v2
 }
 
 function %i32_rem_s(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = srem v0, v1
     return v2
 }
 
 function %i32_rem_u(i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = urem v0, v1
     return v2
 }
 
 function %i64_and(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = band v0, v1
     return v2
 }
 
 function %i64_or(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = bor v0, v1
     return v2
 }
 
 function %i64_xor(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = bxor v0, v1
     return v2
 }
 
 function %i64_shl(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = ishl v0, v1
     return v2
 }
 
 function %i64_shr_s(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = sshr v0, v1
     return v2
 }
 
 function %i64_shr_u(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = ushr v0, v1
     return v2
 }
 
 function %i64_rotl(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = rotl v0, v1
     return v2
 }
 
 function %i64_rotr(i64, i64) -> i64 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = rotr v0, v1
     return v2
 }
diff --git a/cranelift/filetests/filetests/wasm/i64-compares.clif b/cranelift/filetests/filetests/wasm/i64-compares.clif
index e137b7e19c..2863efb6c3 100644
--- a/cranelift/filetests/filetests/wasm/i64-compares.clif
+++ b/cranelift/filetests/filetests/wasm/i64-compares.clif
@@ -4,77 +4,77 @@ test compile
 target x86_64 haswell
 
 function %i64_eqz(i64) -> i32 {
-ebb0(v0: i64):
+block0(v0: i64):
     v1 = icmp_imm eq v0, 0
     v2 = bint.i32 v1
     return v2
 }
 
 function %i64_eq(i64, i64) -> i32 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = icmp eq v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i64_ne(i64, i64) -> i32 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = icmp ne v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i64_lt_s(i64, i64) -> i32 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = icmp slt v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i64_lt_u(i64, i64) -> i32 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = icmp ult v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i64_gt_s(i64, i64) -> i32 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = icmp sgt v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i64_gt_u(i64, i64) -> i32 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = icmp ugt v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i64_le_s(i64, i64) -> i32 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = icmp sle v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i64_le_u(i64, i64) -> i32 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = icmp ule v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i64_ge_s(i64, i64) -> i32 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = icmp sge v0, v1
     v3 = bint.i32 v2
     return v3
 }
 
 function %i64_ge_u(i64, i64) -> i32 {
-ebb0(v0: i64, v1: i64):
+block0(v0: i64, v1: i64):
     v2 = icmp uge v0, v1
     v3 = bint.i32 v2
     return v3
diff --git a/cranelift/filetests/filetests/wasm/i64-memory64.clif b/cranelift/filetests/filetests/wasm/i64-memory64.clif
index edea6da503..f2b34fc8b0 100644
--- a/cranelift/filetests/filetests/wasm/i64-memory64.clif
+++ b/cranelift/filetests/filetests/wasm/i64-memory64.clif
@@ -9,7 +9,7 @@ function %i64_load(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = load.i64 v2
     return v3
@@ -19,7 +19,7 @@ function %i64_store(i64, i32, i64 vmctx) {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i64, v1: i32, v2: i64):
+block0(v0: i64, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
     store v0, v3
     return
@@ -29,7 +29,7 @@ function %i64_load8_s(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = sload8.i64 v2
     return v3
@@ -39,7 +39,7 @@ function %i64_load8_u(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = uload8.i64 v2
     return v3
@@ -49,7 +49,7 @@ function %i64_store8(i64, i32, i64 vmctx) {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i64, v1: i32, v2: i64):
+block0(v0: i64, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
     istore8 v0, v3
     return
@@ -59,7 +59,7 @@ function %i64_load16_s(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = sload16.i64 v2
     return v3
@@ -69,7 +69,7 @@ function %i64_load16_u(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = uload16.i64 v2
     return v3
@@ -79,7 +79,7 @@ function %i64_store16(i64, i32, i64 vmctx) {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i64, v1: i32, v2: i64):
+block0(v0: i64, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
     istore16 v0, v3
     return
@@ -89,7 +89,7 @@ function %i64_load32_s(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = sload32.i64 v2
     return v3
@@ -99,7 +99,7 @@ function %i64_load32_u(i32, i64 vmctx) -> i64 {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = heap_addr.i64 heap0, v0, 1
     v3 = uload32.i64 v2
     return v3
@@ -109,7 +109,7 @@ function %i64_store32(i64, i32, i64 vmctx) {
     gv0 = vmctx
     heap0 = static gv0, min 0x0001_0000, bound 0x0001_0000_0000, offset_guard 0x8000_0000
 
-ebb0(v0: i64, v1: i32, v2: i64):
+block0(v0: i64, v1: i32, v2: i64):
     v3 = heap_addr.i64 heap0, v1, 1
     istore32 v0, v3
     return
diff --git a/cranelift/filetests/filetests/wasm/multi-val-b1.clif b/cranelift/filetests/filetests/wasm/multi-val-b1.clif
index 582403dcfb..7a4d4d02b0 100644
--- a/cranelift/filetests/filetests/wasm/multi-val-b1.clif
+++ b/cranelift/filetests/filetests/wasm/multi-val-b1.clif
@@ -7,8 +7,8 @@ target x86_64 haswell
 function %return_4_b1s(b1, b1, b1, b1) -> b1, b1, b1, b1 {
 ;; check: function %return_4_b1s(b1 [%rsi], b1 [%rdx], b1 [%rcx], b1 [%r8], i64 sret [%rdi], i64 fp [%rbp]) -> i64 sret [%rax], i64 fp [%rbp] fast {
 
-ebb0(v0: b1, v1: b1, v2: b1, v3: b1):
-; check: ebb0(v0: b1 [%rsi], v1: b1 [%rdx], v2: b1 [%rcx], v3: b1 [%r8], v4: i64 [%rdi], v13: i64 [%rbp]):
+block0(v0: b1, v1: b1, v2: b1, v3: b1):
+; check: block0(v0: b1 [%rsi], v1: b1 [%rdx], v2: b1 [%rcx], v3: b1 [%r8], v4: i64 [%rdi], v13: i64 [%rbp]):
 
     return v0, v1, v2, v3
     ; check:  v5 = bint.i8 v0
@@ -32,8 +32,8 @@ function %call_4_b1s() {
     fn0 = colocated %return_4_b1s(b1, b1, b1, b1) -> b1, b1, b1, b1
     ; check: sig0 = (b1 [%rsi], b1 [%rdx], b1 [%rcx], b1 [%r8], i64 sret [%rdi]) -> i64 sret [%rax] fast
 
-ebb0:
-; check: ebb0(v26: i64 [%rbp], v27: i64 [%rbx]):
+block0:
+; check: block0(v26: i64 [%rbp], v27: i64 [%rbx]):
 
     v0 = bconst.b1 true
     v1 = bconst.b1 false
diff --git a/cranelift/filetests/filetests/wasm/multi-val-call-indirect.clif b/cranelift/filetests/filetests/wasm/multi-val-call-indirect.clif
index b102d652cf..6f5afd4700 100644
--- a/cranelift/filetests/filetests/wasm/multi-val-call-indirect.clif
+++ b/cranelift/filetests/filetests/wasm/multi-val-call-indirect.clif
@@ -9,7 +9,7 @@ function %call_indirect_many_rets(i64) {
     sig0 = () -> i64, i64, i64, i64
     ; check: sig0 = (i64 sret [%rdi]) -> i64 sret [%rax] fast
 
-ebb0(v0: i64):
+block0(v0: i64):
     v1, v2, v3, v4 = call_indirect sig0, v0()
     ; check:  v5 = stack_addr.i64 ss0
     ; nextln: v6 = call_indirect sig0, v0(v5)
diff --git a/cranelift/filetests/filetests/wasm/multi-val-f32.clif b/cranelift/filetests/filetests/wasm/multi-val-f32.clif
index 9f3d0047cd..b69b71e047 100644
--- a/cranelift/filetests/filetests/wasm/multi-val-f32.clif
+++ b/cranelift/filetests/filetests/wasm/multi-val-f32.clif
@@ -4,14 +4,14 @@ target x86_64 haswell
 ;; Returning many f32s
 
 function %return_2_f32s() -> f32, f32 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = f32const 0x1.0
     return v0, v1
 }
 
 function %return_3_f32s() -> f32, f32, f32 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = f32const 0x1.0
     v2 = f32const 0x2.0
@@ -19,7 +19,7 @@ ebb0:
 }
 
 function %return_4_f32s() -> f32, f32, f32, f32 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = f32const 0x1.0
     v2 = f32const 0x2.0
@@ -33,7 +33,7 @@ function %call() -> f32 {
     fn0 = %a() -> f32, f32
     fn1 = %b(f32, f32) -> f32, f32, f32
     fn2 = %c(f32, f32, f32) -> f32, f32, f32, f32
-ebb0:
+block0:
     v0, v1 = call fn0()
     v2, v3, v4 = call fn1(v0, v1)
     v5, v6, v7, v8 = call fn2(v2, v3, v4)
diff --git a/cranelift/filetests/filetests/wasm/multi-val-f64.clif b/cranelift/filetests/filetests/wasm/multi-val-f64.clif
index aa7e263eba..afb6585efc 100644
--- a/cranelift/filetests/filetests/wasm/multi-val-f64.clif
+++ b/cranelift/filetests/filetests/wasm/multi-val-f64.clif
@@ -4,14 +4,14 @@ target x86_64 haswell
 ;; Returning many f64s
 
 function %return_2_f64s() -> f64, f64 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = f64const 0x1.0
     return v0, v1
 }
 
 function %return_3_f64s() -> f64, f64, f64 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = f64const 0x1.0
     v2 = f64const 0x2.0
@@ -19,7 +19,7 @@ ebb0:
 }
 
 function %return_4_f64s() -> f64, f64, f64, f64 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = f64const 0x1.0
     v2 = f64const 0x2.0
@@ -33,7 +33,7 @@ function %call() -> f64 {
     fn0 = %a() -> f64, f64
     fn1 = %b(f64, f64) -> f64, f64, f64
     fn2 = %c(f64, f64, f64) -> f64, f64, f64, f64
-ebb0:
+block0:
     v0, v1 = call fn0()
     v2, v3, v4 = call fn1(v0, v1)
     v5, v6, v7, v8 = call fn2(v2, v3, v4)
diff --git a/cranelift/filetests/filetests/wasm/multi-val-i32.clif b/cranelift/filetests/filetests/wasm/multi-val-i32.clif
index 924fcb4bc6..035cc2e332 100644
--- a/cranelift/filetests/filetests/wasm/multi-val-i32.clif
+++ b/cranelift/filetests/filetests/wasm/multi-val-i32.clif
@@ -4,14 +4,14 @@ target x86_64 haswell
 ;; Returning many i32s
 
 function %return_2_i32s() -> i32, i32 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = iconst.i32 1
     return v0, v1
 }
 
 function %return_3_i32s() -> i32, i32, i32 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = iconst.i32 1
     v2 = iconst.i32 2
@@ -19,7 +19,7 @@ ebb0:
 }
 
 function %return_4_i32s() -> i32, i32, i32, i32 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = iconst.i32 1
     v2 = iconst.i32 2
@@ -33,7 +33,7 @@ function %call() -> i32 {
     fn0 = %a() -> i32, i32
     fn1 = %b(i32, i32) -> i32, i32, i32
     fn2 = %c(i32, i32, i32) -> i32, i32, i32, i32
-ebb0:
+block0:
     v0, v1 = call fn0()
     v2, v3, v4 = call fn1(v0, v1)
     v5, v6, v7, v8 = call fn2(v2, v3, v4)
diff --git a/cranelift/filetests/filetests/wasm/multi-val-i64.clif b/cranelift/filetests/filetests/wasm/multi-val-i64.clif
index f5ab392693..bacaf8240f 100644
--- a/cranelift/filetests/filetests/wasm/multi-val-i64.clif
+++ b/cranelift/filetests/filetests/wasm/multi-val-i64.clif
@@ -4,14 +4,14 @@ target x86_64 haswell
 ;; Returning many i64s
 
 function %return_2_i64s() -> i64, i64 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = iconst.i64 1
     return v0, v1
 }
 
 function %return_3_i64s() -> i64, i64, i64 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = iconst.i64 1
     v2 = iconst.i64 2
@@ -19,7 +19,7 @@ ebb0:
 }
 
 function %return_4_i64s() -> i64, i64, i64, i64 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = iconst.i64 1
     v2 = iconst.i64 2
@@ -33,7 +33,7 @@ function %call() -> i64 {
     fn0 = %a() -> i64, i64
     fn1 = %b(i64, i64) -> i64, i64, i64
     fn2 = %c(i64, i64, i64) -> i64, i64, i64, i64
-ebb0:
+block0:
     v0, v1 = call fn0()
     v2, v3, v4 = call fn1(v0, v1)
     v5, v6, v7, v8 = call fn2(v2, v3, v4)
diff --git a/cranelift/filetests/filetests/wasm/multi-val-mixed.clif b/cranelift/filetests/filetests/wasm/multi-val-mixed.clif
index db66d202ff..e7289332c7 100644
--- a/cranelift/filetests/filetests/wasm/multi-val-mixed.clif
+++ b/cranelift/filetests/filetests/wasm/multi-val-mixed.clif
@@ -33,21 +33,21 @@ target x86_64 haswell
 ;; def make_returner(results):
 ;;     results = list(results)
 ;;     head = "function %return_" + "_".join(results) + "() -> " + ", ".join(results) + " {\n"
-;;     ebb = "ebb0:\n"
+;;     block = "block0:\n"
 ;;     vals = [make_val(i, r) for i, r in enumerate(results)]
 ;;     ret = "    return " + ", ".join(("v" + str(i) for i in range(0, len(results))))
-;;     return head + ebb + "\n".join(vals) + "\n" + ret + "\n}\n"
+;;     return head + block + "\n".join(vals) + "\n" + ret + "\n}\n"
 ;;
 ;; def make_caller(results):
 ;;     results = list(results)
 ;;     head = "function %call_" + "_".join(results) + "() {\n"
 ;;     fn_decl = "    fn0 = %foo() -> " + ",".join(results) + "\n"
-;;     ebb = "ebb0:\n"
+;;     block = "block0:\n"
 ;;     ret_vars = ["v" + str(i) for i, r in enumerate(results)]
 ;;     call = "    " + ",".join(ret_vars) + " = call fn0()\n"
 ;;     ret = "    return\n"
 ;;     tail = "}\n"
-;;     return head + fn_decl + ebb + call + ret + tail
+;;     return head + fn_decl + block + call + ret + tail
 ;;
 ;; for results in permutations(["i32", "i64", "f32", "f64", "b1"]):
 ;;     print make_returner(results)
@@ -58,7 +58,7 @@ target x86_64 haswell
 ;; regenerate the test.
 
 function %return_i32_i64_f32_f64_b1() -> i32, i64, f32, f64, b1 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = iconst.i64 0
     v2 = f32const 0x0.0
@@ -69,13 +69,13 @@ ebb0:
 
 function %call_i32_i64_f32_f64_b1() {
     fn0 = %foo() -> i32,i64,f32,f64,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_i64_f32_b1_f64() -> i32, i64, f32, b1, f64 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = iconst.i64 0
     v2 = f32const 0x0.0
@@ -86,13 +86,13 @@ ebb0:
 
 function %call_i32_i64_f32_b1_f64() {
     fn0 = %foo() -> i32,i64,f32,b1,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_i64_f64_f32_b1() -> i32, i64, f64, f32, b1 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = iconst.i64 0
     v2 = f64const 0x0.0
@@ -103,13 +103,13 @@ ebb0:
 
 function %call_i32_i64_f64_f32_b1() {
     fn0 = %foo() -> i32,i64,f64,f32,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_i64_f64_b1_f32() -> i32, i64, f64, b1, f32 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = iconst.i64 0
     v2 = f64const 0x0.0
@@ -120,13 +120,13 @@ ebb0:
 
 function %call_i32_i64_f64_b1_f32() {
     fn0 = %foo() -> i32,i64,f64,b1,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_i64_b1_f32_f64() -> i32, i64, b1, f32, f64 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = iconst.i64 0
     v2 = bconst.b1 true
@@ -137,13 +137,13 @@ ebb0:
 
 function %call_i32_i64_b1_f32_f64() {
     fn0 = %foo() -> i32,i64,b1,f32,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_i64_b1_f64_f32() -> i32, i64, b1, f64, f32 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = iconst.i64 0
     v2 = bconst.b1 true
@@ -154,13 +154,13 @@ ebb0:
 
 function %call_i32_i64_b1_f64_f32() {
     fn0 = %foo() -> i32,i64,b1,f64,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_f32_i64_f64_b1() -> i32, f32, i64, f64, b1 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = f32const 0x0.0
     v2 = iconst.i64 0
@@ -171,13 +171,13 @@ ebb0:
 
 function %call_i32_f32_i64_f64_b1() {
     fn0 = %foo() -> i32,f32,i64,f64,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_f32_i64_b1_f64() -> i32, f32, i64, b1, f64 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = f32const 0x0.0
     v2 = iconst.i64 0
@@ -188,13 +188,13 @@ ebb0:
 
 function %call_i32_f32_i64_b1_f64() {
     fn0 = %foo() -> i32,f32,i64,b1,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_f32_f64_i64_b1() -> i32, f32, f64, i64, b1 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = f32const 0x0.0
     v2 = f64const 0x0.0
@@ -205,13 +205,13 @@ ebb0:
 
 function %call_i32_f32_f64_i64_b1() {
     fn0 = %foo() -> i32,f32,f64,i64,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_f32_f64_b1_i64() -> i32, f32, f64, b1, i64 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = f32const 0x0.0
     v2 = f64const 0x0.0
@@ -222,13 +222,13 @@ ebb0:
 
 function %call_i32_f32_f64_b1_i64() {
     fn0 = %foo() -> i32,f32,f64,b1,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_f32_b1_i64_f64() -> i32, f32, b1, i64, f64 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = f32const 0x0.0
     v2 = bconst.b1 true
@@ -239,13 +239,13 @@ ebb0:
 
 function %call_i32_f32_b1_i64_f64() {
     fn0 = %foo() -> i32,f32,b1,i64,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_f32_b1_f64_i64() -> i32, f32, b1, f64, i64 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = f32const 0x0.0
     v2 = bconst.b1 true
@@ -256,13 +256,13 @@ ebb0:
 
 function %call_i32_f32_b1_f64_i64() {
     fn0 = %foo() -> i32,f32,b1,f64,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_f64_i64_f32_b1() -> i32, f64, i64, f32, b1 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = f64const 0x0.0
     v2 = iconst.i64 0
@@ -273,13 +273,13 @@ ebb0:
 
 function %call_i32_f64_i64_f32_b1() {
     fn0 = %foo() -> i32,f64,i64,f32,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_f64_i64_b1_f32() -> i32, f64, i64, b1, f32 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = f64const 0x0.0
     v2 = iconst.i64 0
@@ -290,13 +290,13 @@ ebb0:
 
 function %call_i32_f64_i64_b1_f32() {
     fn0 = %foo() -> i32,f64,i64,b1,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_f64_f32_i64_b1() -> i32, f64, f32, i64, b1 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = f64const 0x0.0
     v2 = f32const 0x0.0
@@ -307,13 +307,13 @@ ebb0:
 
 function %call_i32_f64_f32_i64_b1() {
     fn0 = %foo() -> i32,f64,f32,i64,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_f64_f32_b1_i64() -> i32, f64, f32, b1, i64 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = f64const 0x0.0
     v2 = f32const 0x0.0
@@ -324,13 +324,13 @@ ebb0:
 
 function %call_i32_f64_f32_b1_i64() {
     fn0 = %foo() -> i32,f64,f32,b1,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_f64_b1_i64_f32() -> i32, f64, b1, i64, f32 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = f64const 0x0.0
     v2 = bconst.b1 true
@@ -341,13 +341,13 @@ ebb0:
 
 function %call_i32_f64_b1_i64_f32() {
     fn0 = %foo() -> i32,f64,b1,i64,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_f64_b1_f32_i64() -> i32, f64, b1, f32, i64 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = f64const 0x0.0
     v2 = bconst.b1 true
@@ -358,13 +358,13 @@ ebb0:
 
 function %call_i32_f64_b1_f32_i64() {
     fn0 = %foo() -> i32,f64,b1,f32,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_b1_i64_f32_f64() -> i32, b1, i64, f32, f64 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = bconst.b1 true
     v2 = iconst.i64 0
@@ -375,13 +375,13 @@ ebb0:
 
 function %call_i32_b1_i64_f32_f64() {
     fn0 = %foo() -> i32,b1,i64,f32,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_b1_i64_f64_f32() -> i32, b1, i64, f64, f32 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = bconst.b1 true
     v2 = iconst.i64 0
@@ -392,13 +392,13 @@ ebb0:
 
 function %call_i32_b1_i64_f64_f32() {
     fn0 = %foo() -> i32,b1,i64,f64,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_b1_f32_i64_f64() -> i32, b1, f32, i64, f64 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = bconst.b1 true
     v2 = f32const 0x0.0
@@ -409,13 +409,13 @@ ebb0:
 
 function %call_i32_b1_f32_i64_f64() {
     fn0 = %foo() -> i32,b1,f32,i64,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_b1_f32_f64_i64() -> i32, b1, f32, f64, i64 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = bconst.b1 true
     v2 = f32const 0x0.0
@@ -426,13 +426,13 @@ ebb0:
 
 function %call_i32_b1_f32_f64_i64() {
     fn0 = %foo() -> i32,b1,f32,f64,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_b1_f64_i64_f32() -> i32, b1, f64, i64, f32 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = bconst.b1 true
     v2 = f64const 0x0.0
@@ -443,13 +443,13 @@ ebb0:
 
 function %call_i32_b1_f64_i64_f32() {
     fn0 = %foo() -> i32,b1,f64,i64,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i32_b1_f64_f32_i64() -> i32, b1, f64, f32, i64 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = bconst.b1 true
     v2 = f64const 0x0.0
@@ -460,13 +460,13 @@ ebb0:
 
 function %call_i32_b1_f64_f32_i64() {
     fn0 = %foo() -> i32,b1,f64,f32,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_i32_f32_f64_b1() -> i64, i32, f32, f64, b1 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = iconst.i32 0
     v2 = f32const 0x0.0
@@ -477,13 +477,13 @@ ebb0:
 
 function %call_i64_i32_f32_f64_b1() {
     fn0 = %foo() -> i64,i32,f32,f64,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_i32_f32_b1_f64() -> i64, i32, f32, b1, f64 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = iconst.i32 0
     v2 = f32const 0x0.0
@@ -494,13 +494,13 @@ ebb0:
 
 function %call_i64_i32_f32_b1_f64() {
     fn0 = %foo() -> i64,i32,f32,b1,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_i32_f64_f32_b1() -> i64, i32, f64, f32, b1 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = iconst.i32 0
     v2 = f64const 0x0.0
@@ -511,13 +511,13 @@ ebb0:
 
 function %call_i64_i32_f64_f32_b1() {
     fn0 = %foo() -> i64,i32,f64,f32,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_i32_f64_b1_f32() -> i64, i32, f64, b1, f32 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = iconst.i32 0
     v2 = f64const 0x0.0
@@ -528,13 +528,13 @@ ebb0:
 
 function %call_i64_i32_f64_b1_f32() {
     fn0 = %foo() -> i64,i32,f64,b1,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_i32_b1_f32_f64() -> i64, i32, b1, f32, f64 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = iconst.i32 0
     v2 = bconst.b1 true
@@ -545,13 +545,13 @@ ebb0:
 
 function %call_i64_i32_b1_f32_f64() {
     fn0 = %foo() -> i64,i32,b1,f32,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_i32_b1_f64_f32() -> i64, i32, b1, f64, f32 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = iconst.i32 0
     v2 = bconst.b1 true
@@ -562,13 +562,13 @@ ebb0:
 
 function %call_i64_i32_b1_f64_f32() {
     fn0 = %foo() -> i64,i32,b1,f64,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_f32_i32_f64_b1() -> i64, f32, i32, f64, b1 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = f32const 0x0.0
     v2 = iconst.i32 0
@@ -579,13 +579,13 @@ ebb0:
 
 function %call_i64_f32_i32_f64_b1() {
     fn0 = %foo() -> i64,f32,i32,f64,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_f32_i32_b1_f64() -> i64, f32, i32, b1, f64 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = f32const 0x0.0
     v2 = iconst.i32 0
@@ -596,13 +596,13 @@ ebb0:
 
 function %call_i64_f32_i32_b1_f64() {
     fn0 = %foo() -> i64,f32,i32,b1,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_f32_f64_i32_b1() -> i64, f32, f64, i32, b1 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = f32const 0x0.0
     v2 = f64const 0x0.0
@@ -613,13 +613,13 @@ ebb0:
 
 function %call_i64_f32_f64_i32_b1() {
     fn0 = %foo() -> i64,f32,f64,i32,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_f32_f64_b1_i32() -> i64, f32, f64, b1, i32 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = f32const 0x0.0
     v2 = f64const 0x0.0
@@ -630,13 +630,13 @@ ebb0:
 
 function %call_i64_f32_f64_b1_i32() {
     fn0 = %foo() -> i64,f32,f64,b1,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_f32_b1_i32_f64() -> i64, f32, b1, i32, f64 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = f32const 0x0.0
     v2 = bconst.b1 true
@@ -647,13 +647,13 @@ ebb0:
 
 function %call_i64_f32_b1_i32_f64() {
     fn0 = %foo() -> i64,f32,b1,i32,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_f32_b1_f64_i32() -> i64, f32, b1, f64, i32 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = f32const 0x0.0
     v2 = bconst.b1 true
@@ -664,13 +664,13 @@ ebb0:
 
 function %call_i64_f32_b1_f64_i32() {
     fn0 = %foo() -> i64,f32,b1,f64,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_f64_i32_f32_b1() -> i64, f64, i32, f32, b1 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = f64const 0x0.0
     v2 = iconst.i32 0
@@ -681,13 +681,13 @@ ebb0:
 
 function %call_i64_f64_i32_f32_b1() {
     fn0 = %foo() -> i64,f64,i32,f32,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_f64_i32_b1_f32() -> i64, f64, i32, b1, f32 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = f64const 0x0.0
     v2 = iconst.i32 0
@@ -698,13 +698,13 @@ ebb0:
 
 function %call_i64_f64_i32_b1_f32() {
     fn0 = %foo() -> i64,f64,i32,b1,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_f64_f32_i32_b1() -> i64, f64, f32, i32, b1 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = f64const 0x0.0
     v2 = f32const 0x0.0
@@ -715,13 +715,13 @@ ebb0:
 
 function %call_i64_f64_f32_i32_b1() {
     fn0 = %foo() -> i64,f64,f32,i32,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_f64_f32_b1_i32() -> i64, f64, f32, b1, i32 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = f64const 0x0.0
     v2 = f32const 0x0.0
@@ -732,13 +732,13 @@ ebb0:
 
 function %call_i64_f64_f32_b1_i32() {
     fn0 = %foo() -> i64,f64,f32,b1,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_f64_b1_i32_f32() -> i64, f64, b1, i32, f32 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = f64const 0x0.0
     v2 = bconst.b1 true
@@ -749,13 +749,13 @@ ebb0:
 
 function %call_i64_f64_b1_i32_f32() {
     fn0 = %foo() -> i64,f64,b1,i32,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_f64_b1_f32_i32() -> i64, f64, b1, f32, i32 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = f64const 0x0.0
     v2 = bconst.b1 true
@@ -766,13 +766,13 @@ ebb0:
 
 function %call_i64_f64_b1_f32_i32() {
     fn0 = %foo() -> i64,f64,b1,f32,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_b1_i32_f32_f64() -> i64, b1, i32, f32, f64 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = bconst.b1 true
     v2 = iconst.i32 0
@@ -783,13 +783,13 @@ ebb0:
 
 function %call_i64_b1_i32_f32_f64() {
     fn0 = %foo() -> i64,b1,i32,f32,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_b1_i32_f64_f32() -> i64, b1, i32, f64, f32 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = bconst.b1 true
     v2 = iconst.i32 0
@@ -800,13 +800,13 @@ ebb0:
 
 function %call_i64_b1_i32_f64_f32() {
     fn0 = %foo() -> i64,b1,i32,f64,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_b1_f32_i32_f64() -> i64, b1, f32, i32, f64 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = bconst.b1 true
     v2 = f32const 0x0.0
@@ -817,13 +817,13 @@ ebb0:
 
 function %call_i64_b1_f32_i32_f64() {
     fn0 = %foo() -> i64,b1,f32,i32,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_b1_f32_f64_i32() -> i64, b1, f32, f64, i32 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = bconst.b1 true
     v2 = f32const 0x0.0
@@ -834,13 +834,13 @@ ebb0:
 
 function %call_i64_b1_f32_f64_i32() {
     fn0 = %foo() -> i64,b1,f32,f64,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_b1_f64_i32_f32() -> i64, b1, f64, i32, f32 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = bconst.b1 true
     v2 = f64const 0x0.0
@@ -851,13 +851,13 @@ ebb0:
 
 function %call_i64_b1_f64_i32_f32() {
     fn0 = %foo() -> i64,b1,f64,i32,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_i64_b1_f64_f32_i32() -> i64, b1, f64, f32, i32 {
-ebb0:
+block0:
     v0 = iconst.i64 0
     v1 = bconst.b1 true
     v2 = f64const 0x0.0
@@ -868,13 +868,13 @@ ebb0:
 
 function %call_i64_b1_f64_f32_i32() {
     fn0 = %foo() -> i64,b1,f64,f32,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_i32_i64_f64_b1() -> f32, i32, i64, f64, b1 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = iconst.i32 0
     v2 = iconst.i64 0
@@ -885,13 +885,13 @@ ebb0:
 
 function %call_f32_i32_i64_f64_b1() {
     fn0 = %foo() -> f32,i32,i64,f64,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_i32_i64_b1_f64() -> f32, i32, i64, b1, f64 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = iconst.i32 0
     v2 = iconst.i64 0
@@ -902,13 +902,13 @@ ebb0:
 
 function %call_f32_i32_i64_b1_f64() {
     fn0 = %foo() -> f32,i32,i64,b1,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_i32_f64_i64_b1() -> f32, i32, f64, i64, b1 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = iconst.i32 0
     v2 = f64const 0x0.0
@@ -919,13 +919,13 @@ ebb0:
 
 function %call_f32_i32_f64_i64_b1() {
     fn0 = %foo() -> f32,i32,f64,i64,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_i32_f64_b1_i64() -> f32, i32, f64, b1, i64 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = iconst.i32 0
     v2 = f64const 0x0.0
@@ -936,13 +936,13 @@ ebb0:
 
 function %call_f32_i32_f64_b1_i64() {
     fn0 = %foo() -> f32,i32,f64,b1,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_i32_b1_i64_f64() -> f32, i32, b1, i64, f64 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = iconst.i32 0
     v2 = bconst.b1 true
@@ -953,13 +953,13 @@ ebb0:
 
 function %call_f32_i32_b1_i64_f64() {
     fn0 = %foo() -> f32,i32,b1,i64,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_i32_b1_f64_i64() -> f32, i32, b1, f64, i64 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = iconst.i32 0
     v2 = bconst.b1 true
@@ -970,13 +970,13 @@ ebb0:
 
 function %call_f32_i32_b1_f64_i64() {
     fn0 = %foo() -> f32,i32,b1,f64,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_i64_i32_f64_b1() -> f32, i64, i32, f64, b1 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = iconst.i64 0
     v2 = iconst.i32 0
@@ -987,13 +987,13 @@ ebb0:
 
 function %call_f32_i64_i32_f64_b1() {
     fn0 = %foo() -> f32,i64,i32,f64,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_i64_i32_b1_f64() -> f32, i64, i32, b1, f64 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = iconst.i64 0
     v2 = iconst.i32 0
@@ -1004,13 +1004,13 @@ ebb0:
 
 function %call_f32_i64_i32_b1_f64() {
     fn0 = %foo() -> f32,i64,i32,b1,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_i64_f64_i32_b1() -> f32, i64, f64, i32, b1 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = iconst.i64 0
     v2 = f64const 0x0.0
@@ -1021,13 +1021,13 @@ ebb0:
 
 function %call_f32_i64_f64_i32_b1() {
     fn0 = %foo() -> f32,i64,f64,i32,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_i64_f64_b1_i32() -> f32, i64, f64, b1, i32 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = iconst.i64 0
     v2 = f64const 0x0.0
@@ -1038,13 +1038,13 @@ ebb0:
 
 function %call_f32_i64_f64_b1_i32() {
     fn0 = %foo() -> f32,i64,f64,b1,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_i64_b1_i32_f64() -> f32, i64, b1, i32, f64 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = iconst.i64 0
     v2 = bconst.b1 true
@@ -1055,13 +1055,13 @@ ebb0:
 
 function %call_f32_i64_b1_i32_f64() {
     fn0 = %foo() -> f32,i64,b1,i32,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_i64_b1_f64_i32() -> f32, i64, b1, f64, i32 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = iconst.i64 0
     v2 = bconst.b1 true
@@ -1072,13 +1072,13 @@ ebb0:
 
 function %call_f32_i64_b1_f64_i32() {
     fn0 = %foo() -> f32,i64,b1,f64,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_f64_i32_i64_b1() -> f32, f64, i32, i64, b1 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = f64const 0x0.0
     v2 = iconst.i32 0
@@ -1089,13 +1089,13 @@ ebb0:
 
 function %call_f32_f64_i32_i64_b1() {
     fn0 = %foo() -> f32,f64,i32,i64,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_f64_i32_b1_i64() -> f32, f64, i32, b1, i64 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = f64const 0x0.0
     v2 = iconst.i32 0
@@ -1106,13 +1106,13 @@ ebb0:
 
 function %call_f32_f64_i32_b1_i64() {
     fn0 = %foo() -> f32,f64,i32,b1,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_f64_i64_i32_b1() -> f32, f64, i64, i32, b1 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = f64const 0x0.0
     v2 = iconst.i64 0
@@ -1123,13 +1123,13 @@ ebb0:
 
 function %call_f32_f64_i64_i32_b1() {
     fn0 = %foo() -> f32,f64,i64,i32,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_f64_i64_b1_i32() -> f32, f64, i64, b1, i32 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = f64const 0x0.0
     v2 = iconst.i64 0
@@ -1140,13 +1140,13 @@ ebb0:
 
 function %call_f32_f64_i64_b1_i32() {
     fn0 = %foo() -> f32,f64,i64,b1,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_f64_b1_i32_i64() -> f32, f64, b1, i32, i64 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = f64const 0x0.0
     v2 = bconst.b1 true
@@ -1157,13 +1157,13 @@ ebb0:
 
 function %call_f32_f64_b1_i32_i64() {
     fn0 = %foo() -> f32,f64,b1,i32,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_f64_b1_i64_i32() -> f32, f64, b1, i64, i32 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = f64const 0x0.0
     v2 = bconst.b1 true
@@ -1174,13 +1174,13 @@ ebb0:
 
 function %call_f32_f64_b1_i64_i32() {
     fn0 = %foo() -> f32,f64,b1,i64,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_b1_i32_i64_f64() -> f32, b1, i32, i64, f64 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = bconst.b1 true
     v2 = iconst.i32 0
@@ -1191,13 +1191,13 @@ ebb0:
 
 function %call_f32_b1_i32_i64_f64() {
     fn0 = %foo() -> f32,b1,i32,i64,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_b1_i32_f64_i64() -> f32, b1, i32, f64, i64 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = bconst.b1 true
     v2 = iconst.i32 0
@@ -1208,13 +1208,13 @@ ebb0:
 
 function %call_f32_b1_i32_f64_i64() {
     fn0 = %foo() -> f32,b1,i32,f64,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_b1_i64_i32_f64() -> f32, b1, i64, i32, f64 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = bconst.b1 true
     v2 = iconst.i64 0
@@ -1225,13 +1225,13 @@ ebb0:
 
 function %call_f32_b1_i64_i32_f64() {
     fn0 = %foo() -> f32,b1,i64,i32,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_b1_i64_f64_i32() -> f32, b1, i64, f64, i32 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = bconst.b1 true
     v2 = iconst.i64 0
@@ -1242,13 +1242,13 @@ ebb0:
 
 function %call_f32_b1_i64_f64_i32() {
     fn0 = %foo() -> f32,b1,i64,f64,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_b1_f64_i32_i64() -> f32, b1, f64, i32, i64 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = bconst.b1 true
     v2 = f64const 0x0.0
@@ -1259,13 +1259,13 @@ ebb0:
 
 function %call_f32_b1_f64_i32_i64() {
     fn0 = %foo() -> f32,b1,f64,i32,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f32_b1_f64_i64_i32() -> f32, b1, f64, i64, i32 {
-ebb0:
+block0:
     v0 = f32const 0x0.0
     v1 = bconst.b1 true
     v2 = f64const 0x0.0
@@ -1276,13 +1276,13 @@ ebb0:
 
 function %call_f32_b1_f64_i64_i32() {
     fn0 = %foo() -> f32,b1,f64,i64,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_i32_i64_f32_b1() -> f64, i32, i64, f32, b1 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = iconst.i32 0
     v2 = iconst.i64 0
@@ -1293,13 +1293,13 @@ ebb0:
 
 function %call_f64_i32_i64_f32_b1() {
     fn0 = %foo() -> f64,i32,i64,f32,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_i32_i64_b1_f32() -> f64, i32, i64, b1, f32 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = iconst.i32 0
     v2 = iconst.i64 0
@@ -1310,13 +1310,13 @@ ebb0:
 
 function %call_f64_i32_i64_b1_f32() {
     fn0 = %foo() -> f64,i32,i64,b1,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_i32_f32_i64_b1() -> f64, i32, f32, i64, b1 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = iconst.i32 0
     v2 = f32const 0x0.0
@@ -1327,13 +1327,13 @@ ebb0:
 
 function %call_f64_i32_f32_i64_b1() {
     fn0 = %foo() -> f64,i32,f32,i64,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_i32_f32_b1_i64() -> f64, i32, f32, b1, i64 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = iconst.i32 0
     v2 = f32const 0x0.0
@@ -1344,13 +1344,13 @@ ebb0:
 
 function %call_f64_i32_f32_b1_i64() {
     fn0 = %foo() -> f64,i32,f32,b1,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_i32_b1_i64_f32() -> f64, i32, b1, i64, f32 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = iconst.i32 0
     v2 = bconst.b1 true
@@ -1361,13 +1361,13 @@ ebb0:
 
 function %call_f64_i32_b1_i64_f32() {
     fn0 = %foo() -> f64,i32,b1,i64,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_i32_b1_f32_i64() -> f64, i32, b1, f32, i64 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = iconst.i32 0
     v2 = bconst.b1 true
@@ -1378,13 +1378,13 @@ ebb0:
 
 function %call_f64_i32_b1_f32_i64() {
     fn0 = %foo() -> f64,i32,b1,f32,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_i64_i32_f32_b1() -> f64, i64, i32, f32, b1 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = iconst.i64 0
     v2 = iconst.i32 0
@@ -1395,13 +1395,13 @@ ebb0:
 
 function %call_f64_i64_i32_f32_b1() {
     fn0 = %foo() -> f64,i64,i32,f32,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_i64_i32_b1_f32() -> f64, i64, i32, b1, f32 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = iconst.i64 0
     v2 = iconst.i32 0
@@ -1412,13 +1412,13 @@ ebb0:
 
 function %call_f64_i64_i32_b1_f32() {
     fn0 = %foo() -> f64,i64,i32,b1,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_i64_f32_i32_b1() -> f64, i64, f32, i32, b1 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = iconst.i64 0
     v2 = f32const 0x0.0
@@ -1429,13 +1429,13 @@ ebb0:
 
 function %call_f64_i64_f32_i32_b1() {
     fn0 = %foo() -> f64,i64,f32,i32,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_i64_f32_b1_i32() -> f64, i64, f32, b1, i32 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = iconst.i64 0
     v2 = f32const 0x0.0
@@ -1446,13 +1446,13 @@ ebb0:
 
 function %call_f64_i64_f32_b1_i32() {
     fn0 = %foo() -> f64,i64,f32,b1,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_i64_b1_i32_f32() -> f64, i64, b1, i32, f32 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = iconst.i64 0
     v2 = bconst.b1 true
@@ -1463,13 +1463,13 @@ ebb0:
 
 function %call_f64_i64_b1_i32_f32() {
     fn0 = %foo() -> f64,i64,b1,i32,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_i64_b1_f32_i32() -> f64, i64, b1, f32, i32 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = iconst.i64 0
     v2 = bconst.b1 true
@@ -1480,13 +1480,13 @@ ebb0:
 
 function %call_f64_i64_b1_f32_i32() {
     fn0 = %foo() -> f64,i64,b1,f32,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_f32_i32_i64_b1() -> f64, f32, i32, i64, b1 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = f32const 0x0.0
     v2 = iconst.i32 0
@@ -1497,13 +1497,13 @@ ebb0:
 
 function %call_f64_f32_i32_i64_b1() {
     fn0 = %foo() -> f64,f32,i32,i64,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_f32_i32_b1_i64() -> f64, f32, i32, b1, i64 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = f32const 0x0.0
     v2 = iconst.i32 0
@@ -1514,13 +1514,13 @@ ebb0:
 
 function %call_f64_f32_i32_b1_i64() {
     fn0 = %foo() -> f64,f32,i32,b1,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_f32_i64_i32_b1() -> f64, f32, i64, i32, b1 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = f32const 0x0.0
     v2 = iconst.i64 0
@@ -1531,13 +1531,13 @@ ebb0:
 
 function %call_f64_f32_i64_i32_b1() {
     fn0 = %foo() -> f64,f32,i64,i32,b1
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_f32_i64_b1_i32() -> f64, f32, i64, b1, i32 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = f32const 0x0.0
     v2 = iconst.i64 0
@@ -1548,13 +1548,13 @@ ebb0:
 
 function %call_f64_f32_i64_b1_i32() {
     fn0 = %foo() -> f64,f32,i64,b1,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_f32_b1_i32_i64() -> f64, f32, b1, i32, i64 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = f32const 0x0.0
     v2 = bconst.b1 true
@@ -1565,13 +1565,13 @@ ebb0:
 
 function %call_f64_f32_b1_i32_i64() {
     fn0 = %foo() -> f64,f32,b1,i32,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_f32_b1_i64_i32() -> f64, f32, b1, i64, i32 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = f32const 0x0.0
     v2 = bconst.b1 true
@@ -1582,13 +1582,13 @@ ebb0:
 
 function %call_f64_f32_b1_i64_i32() {
     fn0 = %foo() -> f64,f32,b1,i64,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_b1_i32_i64_f32() -> f64, b1, i32, i64, f32 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = bconst.b1 true
     v2 = iconst.i32 0
@@ -1599,13 +1599,13 @@ ebb0:
 
 function %call_f64_b1_i32_i64_f32() {
     fn0 = %foo() -> f64,b1,i32,i64,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_b1_i32_f32_i64() -> f64, b1, i32, f32, i64 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = bconst.b1 true
     v2 = iconst.i32 0
@@ -1616,13 +1616,13 @@ ebb0:
 
 function %call_f64_b1_i32_f32_i64() {
     fn0 = %foo() -> f64,b1,i32,f32,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_b1_i64_i32_f32() -> f64, b1, i64, i32, f32 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = bconst.b1 true
     v2 = iconst.i64 0
@@ -1633,13 +1633,13 @@ ebb0:
 
 function %call_f64_b1_i64_i32_f32() {
     fn0 = %foo() -> f64,b1,i64,i32,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_b1_i64_f32_i32() -> f64, b1, i64, f32, i32 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = bconst.b1 true
     v2 = iconst.i64 0
@@ -1650,13 +1650,13 @@ ebb0:
 
 function %call_f64_b1_i64_f32_i32() {
     fn0 = %foo() -> f64,b1,i64,f32,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_b1_f32_i32_i64() -> f64, b1, f32, i32, i64 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = bconst.b1 true
     v2 = f32const 0x0.0
@@ -1667,13 +1667,13 @@ ebb0:
 
 function %call_f64_b1_f32_i32_i64() {
     fn0 = %foo() -> f64,b1,f32,i32,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_f64_b1_f32_i64_i32() -> f64, b1, f32, i64, i32 {
-ebb0:
+block0:
     v0 = f64const 0x0.0
     v1 = bconst.b1 true
     v2 = f32const 0x0.0
@@ -1684,13 +1684,13 @@ ebb0:
 
 function %call_f64_b1_f32_i64_i32() {
     fn0 = %foo() -> f64,b1,f32,i64,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_i32_i64_f32_f64() -> b1, i32, i64, f32, f64 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = iconst.i32 0
     v2 = iconst.i64 0
@@ -1701,13 +1701,13 @@ ebb0:
 
 function %call_b1_i32_i64_f32_f64() {
     fn0 = %foo() -> b1,i32,i64,f32,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_i32_i64_f64_f32() -> b1, i32, i64, f64, f32 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = iconst.i32 0
     v2 = iconst.i64 0
@@ -1718,13 +1718,13 @@ ebb0:
 
 function %call_b1_i32_i64_f64_f32() {
     fn0 = %foo() -> b1,i32,i64,f64,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_i32_f32_i64_f64() -> b1, i32, f32, i64, f64 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = iconst.i32 0
     v2 = f32const 0x0.0
@@ -1735,13 +1735,13 @@ ebb0:
 
 function %call_b1_i32_f32_i64_f64() {
     fn0 = %foo() -> b1,i32,f32,i64,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_i32_f32_f64_i64() -> b1, i32, f32, f64, i64 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = iconst.i32 0
     v2 = f32const 0x0.0
@@ -1752,13 +1752,13 @@ ebb0:
 
 function %call_b1_i32_f32_f64_i64() {
     fn0 = %foo() -> b1,i32,f32,f64,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_i32_f64_i64_f32() -> b1, i32, f64, i64, f32 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = iconst.i32 0
     v2 = f64const 0x0.0
@@ -1769,13 +1769,13 @@ ebb0:
 
 function %call_b1_i32_f64_i64_f32() {
     fn0 = %foo() -> b1,i32,f64,i64,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_i32_f64_f32_i64() -> b1, i32, f64, f32, i64 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = iconst.i32 0
     v2 = f64const 0x0.0
@@ -1786,13 +1786,13 @@ ebb0:
 
 function %call_b1_i32_f64_f32_i64() {
     fn0 = %foo() -> b1,i32,f64,f32,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_i64_i32_f32_f64() -> b1, i64, i32, f32, f64 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = iconst.i64 0
     v2 = iconst.i32 0
@@ -1803,13 +1803,13 @@ ebb0:
 
 function %call_b1_i64_i32_f32_f64() {
     fn0 = %foo() -> b1,i64,i32,f32,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_i64_i32_f64_f32() -> b1, i64, i32, f64, f32 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = iconst.i64 0
     v2 = iconst.i32 0
@@ -1820,13 +1820,13 @@ ebb0:
 
 function %call_b1_i64_i32_f64_f32() {
     fn0 = %foo() -> b1,i64,i32,f64,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_i64_f32_i32_f64() -> b1, i64, f32, i32, f64 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = iconst.i64 0
     v2 = f32const 0x0.0
@@ -1837,13 +1837,13 @@ ebb0:
 
 function %call_b1_i64_f32_i32_f64() {
     fn0 = %foo() -> b1,i64,f32,i32,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_i64_f32_f64_i32() -> b1, i64, f32, f64, i32 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = iconst.i64 0
     v2 = f32const 0x0.0
@@ -1854,13 +1854,13 @@ ebb0:
 
 function %call_b1_i64_f32_f64_i32() {
     fn0 = %foo() -> b1,i64,f32,f64,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_i64_f64_i32_f32() -> b1, i64, f64, i32, f32 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = iconst.i64 0
     v2 = f64const 0x0.0
@@ -1871,13 +1871,13 @@ ebb0:
 
 function %call_b1_i64_f64_i32_f32() {
     fn0 = %foo() -> b1,i64,f64,i32,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_i64_f64_f32_i32() -> b1, i64, f64, f32, i32 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = iconst.i64 0
     v2 = f64const 0x0.0
@@ -1888,13 +1888,13 @@ ebb0:
 
 function %call_b1_i64_f64_f32_i32() {
     fn0 = %foo() -> b1,i64,f64,f32,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_f32_i32_i64_f64() -> b1, f32, i32, i64, f64 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = f32const 0x0.0
     v2 = iconst.i32 0
@@ -1905,13 +1905,13 @@ ebb0:
 
 function %call_b1_f32_i32_i64_f64() {
     fn0 = %foo() -> b1,f32,i32,i64,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_f32_i32_f64_i64() -> b1, f32, i32, f64, i64 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = f32const 0x0.0
     v2 = iconst.i32 0
@@ -1922,13 +1922,13 @@ ebb0:
 
 function %call_b1_f32_i32_f64_i64() {
     fn0 = %foo() -> b1,f32,i32,f64,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_f32_i64_i32_f64() -> b1, f32, i64, i32, f64 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = f32const 0x0.0
     v2 = iconst.i64 0
@@ -1939,13 +1939,13 @@ ebb0:
 
 function %call_b1_f32_i64_i32_f64() {
     fn0 = %foo() -> b1,f32,i64,i32,f64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_f32_i64_f64_i32() -> b1, f32, i64, f64, i32 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = f32const 0x0.0
     v2 = iconst.i64 0
@@ -1956,13 +1956,13 @@ ebb0:
 
 function %call_b1_f32_i64_f64_i32() {
     fn0 = %foo() -> b1,f32,i64,f64,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_f32_f64_i32_i64() -> b1, f32, f64, i32, i64 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = f32const 0x0.0
     v2 = f64const 0x0.0
@@ -1973,13 +1973,13 @@ ebb0:
 
 function %call_b1_f32_f64_i32_i64() {
     fn0 = %foo() -> b1,f32,f64,i32,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_f32_f64_i64_i32() -> b1, f32, f64, i64, i32 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = f32const 0x0.0
     v2 = f64const 0x0.0
@@ -1990,13 +1990,13 @@ ebb0:
 
 function %call_b1_f32_f64_i64_i32() {
     fn0 = %foo() -> b1,f32,f64,i64,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_f64_i32_i64_f32() -> b1, f64, i32, i64, f32 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = f64const 0x0.0
     v2 = iconst.i32 0
@@ -2007,13 +2007,13 @@ ebb0:
 
 function %call_b1_f64_i32_i64_f32() {
     fn0 = %foo() -> b1,f64,i32,i64,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_f64_i32_f32_i64() -> b1, f64, i32, f32, i64 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = f64const 0x0.0
     v2 = iconst.i32 0
@@ -2024,13 +2024,13 @@ ebb0:
 
 function %call_b1_f64_i32_f32_i64() {
     fn0 = %foo() -> b1,f64,i32,f32,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_f64_i64_i32_f32() -> b1, f64, i64, i32, f32 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = f64const 0x0.0
     v2 = iconst.i64 0
@@ -2041,13 +2041,13 @@ ebb0:
 
 function %call_b1_f64_i64_i32_f32() {
     fn0 = %foo() -> b1,f64,i64,i32,f32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_f64_i64_f32_i32() -> b1, f64, i64, f32, i32 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = f64const 0x0.0
     v2 = iconst.i64 0
@@ -2058,13 +2058,13 @@ ebb0:
 
 function %call_b1_f64_i64_f32_i32() {
     fn0 = %foo() -> b1,f64,i64,f32,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_f64_f32_i32_i64() -> b1, f64, f32, i32, i64 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = f64const 0x0.0
     v2 = f32const 0x0.0
@@ -2075,13 +2075,13 @@ ebb0:
 
 function %call_b1_f64_f32_i32_i64() {
     fn0 = %foo() -> b1,f64,f32,i32,i64
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
 
 function %return_b1_f64_f32_i64_i32() -> b1, f64, f32, i64, i32 {
-ebb0:
+block0:
     v0 = bconst.b1 true
     v1 = f64const 0x0.0
     v2 = f32const 0x0.0
@@ -2092,7 +2092,7 @@ ebb0:
 
 function %call_b1_f64_f32_i64_i32() {
     fn0 = %foo() -> b1,f64,f32,i64,i32
-ebb0:
+block0:
     v0,v1,v2,v3,v4 = call fn0()
     return
 }
diff --git a/cranelift/filetests/filetests/wasm/multi-val-reuse-ret-ptr-stack-slot.clif b/cranelift/filetests/filetests/wasm/multi-val-reuse-ret-ptr-stack-slot.clif
index f7d0bf846c..d712bf21ce 100644
--- a/cranelift/filetests/filetests/wasm/multi-val-reuse-ret-ptr-stack-slot.clif
+++ b/cranelift/filetests/filetests/wasm/multi-val-reuse-ret-ptr-stack-slot.clif
@@ -16,7 +16,7 @@ function %foo() -> i32, f32 {
     ; nextln: fn0 = %f sig0
     ; nextln: fn1 = %g sig1
 
-ebb0:
+block0:
     v0, v1, v2, v3, v4 = call fn0()
     ; check:  v18 = stack_addr.i64 ss0
     ; nextln: v25 = func_addr.i64 fn0
diff --git a/cranelift/filetests/filetests/wasm/multi-val-sret-slot-alignment.clif b/cranelift/filetests/filetests/wasm/multi-val-sret-slot-alignment.clif
index b6c74e314e..5004ebbe54 100644
--- a/cranelift/filetests/filetests/wasm/multi-val-sret-slot-alignment.clif
+++ b/cranelift/filetests/filetests/wasm/multi-val-sret-slot-alignment.clif
@@ -7,8 +7,8 @@ target x86_64 haswell
 function %returner() -> i8, i32, i8, i64 {
 ; check: function %returner(i64 sret [%rdi]) -> i64 sret [%rax] fast {
 
-ebb0:
-; check: ebb0(v4: i64):
+block0:
+; check: block0(v4: i64):
 
     v0 = iconst.i8 0
     v1 = iconst.i32 1
@@ -31,7 +31,7 @@ function %caller() {
     ; check:  sig0 = (i64 sret [%rdi]) -> i64 sret [%rax] fast
     ; nextln: fn0 = %returner sig0
 
-ebb0:
+block0:
     v0, v1, v2, v3 = call fn0()
     ; check:  v4 = stack_addr.i64 ss0
     ; nextln: v10 = func_addr.i64 fn0
diff --git a/cranelift/filetests/filetests/wasm/multi-val-take-many-and-return-many.clif b/cranelift/filetests/filetests/wasm/multi-val-take-many-and-return-many.clif
index 385cc9d27c..17f2f306d4 100644
--- a/cranelift/filetests/filetests/wasm/multi-val-take-many-and-return-many.clif
+++ b/cranelift/filetests/filetests/wasm/multi-val-take-many-and-return-many.clif
@@ -2,13 +2,13 @@ test compile
 target x86_64 haswell
 
 function %returner(i32, i64, f32, f64) -> i32, i64, f32, f64 {
-ebb0(v0: i32, v1: i64, v2: f32, v3: f64):
+block0(v0: i32, v1: i64, v2: f32, v3: f64):
     return v0, v1, v2, v3
 }
 
 function %caller() {
     fn0 = %returner(i32, i64, f32, f64) -> i32, i64, f32, f64
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = iconst.i64 1
     v2 = f32const 0x2.0
diff --git a/cranelift/filetests/filetests/wasm/multi-val-tons-of-results.clif b/cranelift/filetests/filetests/wasm/multi-val-tons-of-results.clif
index f19b1bcbf0..f394bdd904 100644
--- a/cranelift/filetests/filetests/wasm/multi-val-tons-of-results.clif
+++ b/cranelift/filetests/filetests/wasm/multi-val-tons-of-results.clif
@@ -2,7 +2,7 @@ test compile
 target x86_64 haswell
 
 function %return_20_i32s() -> i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 {
-ebb0:
+block0:
     v0 = iconst.i32 0
     v1 = iconst.i32 1
     v2 = iconst.i32 2
@@ -28,7 +28,7 @@ ebb0:
 
 function %call_20_i32s() {
     fn0 = %return_20_i32s() -> i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32
-ebb0:
+block0:
     v0, v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16, v17, v18, v19 = call fn0()
     return
 }
diff --git a/cranelift/filetests/filetests/wasm/r32.clif b/cranelift/filetests/filetests/wasm/r32.clif
index a60dd4d29b..7e1622246a 100644
--- a/cranelift/filetests/filetests/wasm/r32.clif
+++ b/cranelift/filetests/filetests/wasm/r32.clif
@@ -7,11 +7,11 @@ set enable_safepoints=true
 target i686 haswell
 
 function %select_ref(i32, r32, r32) -> r32 {
-ebb0(v0: i32, v1: r32, v2: r32):
-    brz v0, ebb1(v2)
-    jump ebb1(v1)
+block0(v0: i32, v1: r32, v2: r32):
+    brz v0, block1(v2)
+    jump block1(v1)
 
-ebb1(v3: r32):
+block1(v3: r32):
     return v3
 }
 
@@ -21,7 +21,7 @@ function %table_set(i32, r32, i32 vmctx) {
     gv2 = load.i32 notrap aligned gv0 +4
     table0 = dynamic gv1, element_size 1, bound gv2, index_type i32
 
-ebb0(v0: i32, v1: r32, v2: i32):
+block0(v0: i32, v1: r32, v2: i32):
     v3 = table_addr.i32 table0, v0, +0;
     store.r32 notrap aligned v1, v3
     return
@@ -33,7 +33,7 @@ function %table_get(i32, i32 vmctx) -> r32 {
     gv2 = load.i32 notrap aligned gv0 +4
     table0 = dynamic gv1, element_size 1, bound gv2, index_type i32
 
-ebb0(v0: i32, v1: i32):
+block0(v0: i32, v1: i32):
     v2 = table_addr.i32 table0, v0, +0;
     v3 = load.r32 notrap aligned v2
     return v3
@@ -44,7 +44,7 @@ function %test_refs(r32, r32, r32, i32 vmctx) {
     fn1 = %table_set(i32, r32, i32 vmctx)
     fn2 = %table_get(i32, i32 vmctx) -> r32
 
-ebb0(v0: r32, v1: r32, v2: r32, v3: i32):
+block0(v0: r32, v1: r32, v2: r32, v3: i32):
     v4 = iconst.i32 0
     v5 = iconst.i32 1
     v8 = iconst.i32 2
diff --git a/cranelift/filetests/filetests/wasm/r64.clif b/cranelift/filetests/filetests/wasm/r64.clif
index 5637028103..9fab27fbb5 100644
--- a/cranelift/filetests/filetests/wasm/r64.clif
+++ b/cranelift/filetests/filetests/wasm/r64.clif
@@ -7,11 +7,11 @@ set enable_safepoints=true
 target x86_64 haswell
 
 function %select_ref(i32, r64, r64) -> r64 {
-ebb0(v0: i32, v1: r64, v2: r64):
-    brz v0, ebb1(v2)
-    jump ebb1(v1)
+block0(v0: i32, v1: r64, v2: r64):
+    brz v0, block1(v2)
+    jump block1(v1)
 
-ebb1(v3: r64):
+block1(v3: r64):
     return v3
 }
 
@@ -21,7 +21,7 @@ function %table_set(i32, r64, i64 vmctx) {
     gv2 = load.i32 notrap aligned gv0 +8
     table0 = dynamic gv1, element_size 1, bound gv2, index_type i32
 
-ebb0(v0: i32, v1: r64, v2: i64):
+block0(v0: i32, v1: r64, v2: i64):
     v3 = table_addr.i64 table0, v0, +0;
     store.r64 notrap aligned v1, v3
     return
@@ -33,7 +33,7 @@ function %table_get(i32, i64 vmctx) -> r64 {
     gv2 = load.i32 notrap aligned gv0 +8
     table0 = dynamic gv1, element_size 1, bound gv2, index_type i32
 
-ebb0(v0: i32, v1: i64):
+block0(v0: i32, v1: i64):
     v2 = table_addr.i64 table0, v0, +0;
     v3 = load.r64 notrap aligned v2
     return v3
@@ -44,7 +44,7 @@ function %test_refs(r64, r64, r64, i64 vmctx) {
     fn1 = %table_set(i32, r64, i64 vmctx)
     fn2 = %table_get(i32, i64 vmctx) -> r64
 
-ebb0(v0: r64, v1: r64, v2: r64, v3: i64):
+block0(v0: r64, v1: r64, v2: r64, v3: i64):
     v4 = iconst.i32 0
     v5 = iconst.i32 1
     v8 = iconst.i32 2
diff --git a/cranelift/filetests/filetests/wasm/select.clif b/cranelift/filetests/filetests/wasm/select.clif
index f5f55cda24..b2508ef6e5 100644
--- a/cranelift/filetests/filetests/wasm/select.clif
+++ b/cranelift/filetests/filetests/wasm/select.clif
@@ -6,25 +6,25 @@ target i686 haswell
 target x86_64 haswell
 
 function %select_i32(i32, i32, i32) -> i32 {
-ebb0(v0: i32, v1: i32, v2: i32):
+block0(v0: i32, v1: i32, v2: i32):
     v3 = select v2, v0, v1
     return v3
 }
 
 function %select_i64(i64, i64, i32) -> i64 {
-ebb0(v0: i64, v1: i64, v2: i32):
+block0(v0: i64, v1: i64, v2: i32):
     v3 = select v2, v0, v1
     return v3
 }
 
 function %select_f32(f32, f32, i32) -> f32 {
-ebb0(v0: f32, v1: f32, v2: i32):
+block0(v0: f32, v1: f32, v2: i32):
     v3 = select v2, v0, v1
     return v3
 }
 
 function %select_f64(f64, f64, i32) -> f64 {
-ebb0(v0: f64, v1: f64, v2: i32):
+block0(v0: f64, v1: f64, v2: i32):
     v3 = select v2, v0, v1
     return v3
 }
diff --git a/cranelift/filetests/src/function_runner.rs b/cranelift/filetests/src/function_runner.rs
index 32c2bf5f50..302acf0cb4 100644
--- a/cranelift/filetests/src/function_runner.rs
+++ b/cranelift/filetests/src/function_runner.rs
@@ -98,7 +98,7 @@ mod test {
             "
             test run
             function %test() -> b8 {
-            ebb0:
+            block0:
                 nop
                 v1 = bconst.b8 true
                 return v1
diff --git a/cranelift/filetests/src/match_directive.rs b/cranelift/filetests/src/match_directive.rs
index a3f951f3b4..bb379f25c9 100644
--- a/cranelift/filetests/src/match_directive.rs
+++ b/cranelift/filetests/src/match_directive.rs
@@ -1,7 +1,7 @@
 /// Look for a directive in a comment string.
 /// The directive is of the form "foo:" and should follow the leading `;` in the comment:
 ///
-/// ; dominates: ebb3 ebb4
+/// ; dominates: block3 block4
 ///
 /// Return the comment text following the directive.
 pub fn match_directive<'a>(comment: &'a str, directive: &str) -> Option<&'a str> {
diff --git a/cranelift/filetests/src/test_binemit.rs b/cranelift/filetests/src/test_binemit.rs
index 1a6bd60e99..d2cfdd97ca 100644
--- a/cranelift/filetests/src/test_binemit.rs
+++ b/cranelift/filetests/src/test_binemit.rs
@@ -73,8 +73,8 @@ impl binemit::CodeSink for TextSink {
         self.offset += 8;
     }
 
-    fn reloc_ebb(&mut self, reloc: binemit::Reloc, ebb_offset: binemit::CodeOffset) {
-        write!(self.text, "{}({}) ", reloc, ebb_offset).unwrap();
+    fn reloc_block(&mut self, reloc: binemit::Reloc, block_offset: binemit::CodeOffset) {
+        write!(self.text, "{}({}) ", reloc, block_offset).unwrap();
     }
 
     fn reloc_external(
@@ -151,9 +151,9 @@ impl SubTest for TestBinEmit {
 
         // Give an encoding to any instruction that doesn't already have one.
         let mut divert = RegDiversions::new();
-        for ebb in func.layout.ebbs() {
+        for block in func.layout.blocks() {
             divert.clear();
-            for inst in func.layout.ebb_insts(ebb) {
+            for inst in func.layout.block_insts(block) {
                 if !func.encodings[inst].is_legal() {
                     // Find an encoding that satisfies both immediate field and register
                     // constraints.
@@ -181,7 +181,7 @@ impl SubTest for TestBinEmit {
             }
         }
 
-        // Relax branches and compute EBB offsets based on the encodings.
+        // Relax branches and compute block offsets based on the encodings.
         let mut cfg = ControlFlowGraph::with_function(&func);
         let mut domtree = DominatorTree::with_function(&func, &cfg);
         let CodeInfo { total_size, .. } =
@@ -218,15 +218,15 @@ impl SubTest for TestBinEmit {
 
         // Now emit all instructions.
         let mut sink = TextSink::new();
-        for ebb in func.layout.ebbs() {
+        for block in func.layout.blocks() {
             divert.clear();
             // Correct header offsets should have been computed by `relax_branches()`.
             assert_eq!(
-                sink.offset, func.offsets[ebb],
+                sink.offset, func.offsets[block],
                 "Inconsistent {} header offset",
-                ebb
+                block
             );
-            for (offset, inst, enc_bytes) in func.inst_offsets(ebb, &encinfo) {
+            for (offset, inst, enc_bytes) in func.inst_offsets(block, &encinfo) {
                 assert_eq!(sink.offset, offset);
                 sink.text.clear();
                 let enc = func.encodings[inst];
@@ -293,8 +293,8 @@ impl SubTest for TestBinEmit {
 
         for (jt, jt_data) in func.jump_tables.iter() {
             let jt_offset = func.jt_offsets[jt];
-            for ebb in jt_data.iter() {
-                let rel_offset: i32 = func.offsets[*ebb] as i32 - jt_offset as i32;
+            for block in jt_data.iter() {
+                let rel_offset: i32 = func.offsets[*block] as i32 - jt_offset as i32;
                 sink.put4(rel_offset as u32)
             }
         }
diff --git a/cranelift/filetests/src/test_compile.rs b/cranelift/filetests/src/test_compile.rs
index 10e07440ed..0ac8c48c5e 100644
--- a/cranelift/filetests/src/test_compile.rs
+++ b/cranelift/filetests/src/test_compile.rs
@@ -98,7 +98,7 @@ impl binemit::CodeSink for SizeSink {
         self.offset += 8;
     }
 
-    fn reloc_ebb(&mut self, _reloc: binemit::Reloc, _ebb_offset: binemit::CodeOffset) {}
+    fn reloc_block(&mut self, _reloc: binemit::Reloc, _block_offset: binemit::CodeOffset) {}
     fn reloc_external(
         &mut self,
         _reloc: binemit::Reloc,
diff --git a/cranelift/filetests/src/test_domtree.rs b/cranelift/filetests/src/test_domtree.rs
index 05c7dc31e5..f5f81ed03a 100644
--- a/cranelift/filetests/src/test_domtree.rs
+++ b/cranelift/filetests/src/test_domtree.rs
@@ -3,11 +3,11 @@
 //! The `test domtree` test command looks for annotations on instructions like this:
 //!
 //! ```clif
-//!     jump ebb3 ; dominates: ebb3
+//!     jump block3 ; dominates: block3
 //! ```
 //!
 //! This annotation means that the jump instruction is expected to be the immediate dominator of
-//! `ebb3`.
+//! `block3`.
 //!
 //! We verify that the dominator tree annotations are complete and correct.
 //!
@@ -58,31 +58,31 @@ impl SubTest for TestDomtree {
                         ));
                     }
                 };
-                for src_ebb in tail.split_whitespace() {
-                    let ebb = match context.details.map.lookup_str(src_ebb) {
-                        Some(AnyEntity::Ebb(ebb)) => ebb,
-                        _ => return Err(format!("expected defined EBB, got {}", src_ebb)),
+                for src_block in tail.split_whitespace() {
+                    let block = match context.details.map.lookup_str(src_block) {
+                        Some(AnyEntity::Block(block)) => block,
+                        _ => return Err(format!("expected defined block, got {}", src_block)),
                     };
 
-                    // Annotations say that `inst` is the idom of `ebb`.
-                    if expected.insert(ebb, inst).is_some() {
-                        return Err(format!("multiple dominators for {}", src_ebb));
+                    // Annotations say that `inst` is the idom of `block`.
+                    if expected.insert(block, inst).is_some() {
+                        return Err(format!("multiple dominators for {}", src_block));
                     }
 
                     // Compare to computed domtree.
-                    match domtree.idom(ebb) {
+                    match domtree.idom(block) {
                         Some(got_inst) if got_inst != inst => {
                             return Err(format!(
                                 "mismatching idoms for {}:\n\
                                  want: {}, got: {}",
-                                src_ebb, inst, got_inst
+                                src_block, inst, got_inst
                             ));
                         }
                         None => {
                             return Err(format!(
                                 "mismatching idoms for {}:\n\
                                  want: {}, got: unreachable",
-                                src_ebb, inst
+                                src_block, inst
                             ));
                         }
                         _ => {}
@@ -92,18 +92,18 @@ impl SubTest for TestDomtree {
         }
 
         // Now we know that everything in `expected` is consistent with `domtree`.
-        // All other EBB's should be either unreachable or the entry block.
-        for ebb in func
+        // All other block's should be either unreachable or the entry block.
+        for block in func
             .layout
-            .ebbs()
+            .blocks()
             .skip(1)
-            .filter(|ebb| !expected.contains_key(ebb))
+            .filter(|block| !expected.contains_key(block))
         {
-            if let Some(got_inst) = domtree.idom(ebb) {
+            if let Some(got_inst) = domtree.idom(block) {
                 return Err(format!(
                     "mismatching idoms for renumbered {}:\n\
                      want: unrechable, got: {}",
-                    ebb, got_inst
+                    block, got_inst
                 ));
             }
         }
@@ -118,8 +118,8 @@ fn filecheck_text(func: &Function, domtree: &DominatorTree) -> Result Result,
+    blocks: SecondaryMap,
     types: SecondaryMap,
 }
 
@@ -40,12 +40,12 @@ pub struct FunctionBuilder<'a> {
 }
 
 #[derive(Clone, Default)]
-struct EbbData {
-    /// An Ebb is "pristine" iff no instructions have been added since the last
+struct BlockData {
+    /// An Block is "pristine" iff no instructions have been added since the last
     /// call to `switch_to_block()`.
     pristine: bool,
 
-    /// An Ebb is "filled" iff a terminator instruction has been inserted since
+    /// An Block is "filled" iff a terminator instruction has been inserted since
     /// the last call to `switch_to_block()`.
     ///
     /// A filled block cannot be pristine.
@@ -57,20 +57,20 @@ struct EbbData {
 
 #[derive(Default)]
 struct Position {
-    ebb: PackedOption,
-    basic_block: PackedOption,
+    block: PackedOption,
+    basic_block: PackedOption,
 }
 
 impl Position {
-    fn at(ebb: Ebb, basic_block: Block) -> Self {
+    fn at(block: Block, basic_block: SSABlock) -> Self {
         Self {
-            ebb: PackedOption::from(ebb),
+            block: PackedOption::from(block),
             basic_block: PackedOption::from(basic_block),
         }
     }
 
     fn is_default(&self) -> bool {
-        self.ebb.is_none() && self.basic_block.is_none()
+        self.block.is_none() && self.basic_block.is_none()
     }
 }
 
@@ -80,19 +80,19 @@ impl FunctionBuilderContext {
     pub fn new() -> Self {
         Self {
             ssa: SSABuilder::new(),
-            ebbs: SecondaryMap::new(),
+            blocks: SecondaryMap::new(),
             types: SecondaryMap::new(),
         }
     }
 
     fn clear(&mut self) {
         self.ssa.clear();
-        self.ebbs.clear();
+        self.blocks.clear();
         self.types.clear();
     }
 
     fn is_empty(&self) -> bool {
-        self.ssa.is_empty() && self.ebbs.is_empty() && self.types.is_empty()
+        self.ssa.is_empty() && self.blocks.is_empty() && self.types.is_empty()
     }
 }
 
@@ -100,12 +100,12 @@ impl FunctionBuilderContext {
 /// one convenience method per Cranelift IR instruction.
 pub struct FuncInstBuilder<'short, 'long: 'short> {
     builder: &'short mut FunctionBuilder<'long>,
-    ebb: Ebb,
+    block: Block,
 }
 
 impl<'short, 'long> FuncInstBuilder<'short, 'long> {
-    fn new(builder: &'short mut FunctionBuilder<'long>, ebb: Ebb) -> Self {
-        Self { builder, ebb }
+    fn new(builder: &'short mut FunctionBuilder<'long>, block: Block) -> Self {
+        Self { builder, block }
     }
 }
 
@@ -122,22 +122,22 @@ impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
     // instruction being inserted to add related info to the DFG and the SSA building system,
     // and perform debug sanity checks.
     fn build(self, data: InstructionData, ctrl_typevar: Type) -> (Inst, &'short mut DataFlowGraph) {
-        // We only insert the Ebb in the layout when an instruction is added to it
-        self.builder.ensure_inserted_ebb();
+        // We only insert the Block in the layout when an instruction is added to it
+        self.builder.ensure_inserted_block();
 
         let inst = self.builder.func.dfg.make_inst(data.clone());
         self.builder.func.dfg.make_inst_results(inst, ctrl_typevar);
-        self.builder.func.layout.append_inst(inst, self.ebb);
+        self.builder.func.layout.append_inst(inst, self.block);
         if !self.builder.srcloc.is_default() {
             self.builder.func.srclocs[inst] = self.builder.srcloc;
         }
 
         if data.opcode().is_branch() {
             match data.branch_destination() {
-                Some(dest_ebb) => {
+                Some(dest_block) => {
                     // If the user has supplied jump arguments we must adapt the arguments of
-                    // the destination ebb
-                    self.builder.declare_successor(dest_ebb, inst);
+                    // the destination block
+                    self.builder.declare_successor(dest_block, inst);
                 }
                 None => {
                     // branch_destination() doesn't detect jump_tables
@@ -149,23 +149,23 @@ impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
                         // Unlike all other jumps/branches, jump tables are
                         // capable of having the same successor appear
                         // multiple times, so we must deduplicate.
-                        let mut unique = EntitySet::::new();
-                        for dest_ebb in self
+                        let mut unique = EntitySet::::new();
+                        for dest_block in self
                             .builder
                             .func
                             .jump_tables
                             .get(table)
                             .expect("you are referencing an undeclared jump table")
                             .iter()
-                            .filter(|&dest_ebb| unique.insert(*dest_ebb))
+                            .filter(|&dest_block| unique.insert(*dest_block))
                         {
-                            self.builder.func_ctx.ssa.declare_ebb_predecessor(
-                                *dest_ebb,
+                            self.builder.func_ctx.ssa.declare_block_predecessor(
+                                *dest_block,
                                 self.builder.position.basic_block.unwrap(),
                                 inst,
                             );
                         }
-                        self.builder.func_ctx.ssa.declare_ebb_predecessor(
+                        self.builder.func_ctx.ssa.declare_block_predecessor(
                             destination,
                             self.builder.position.basic_block.unwrap(),
                             inst,
@@ -189,7 +189,7 @@ impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
 /// The module is parametrized by one type which is the representation of variables in your
 /// origin language. It offers a way to conveniently append instruction to your program flow.
 /// You are responsible to split your instruction flow into extended blocks (declared with
-/// `create_ebb`) whose properties are:
+/// `create_block`) whose properties are:
 ///
 /// - branch and jump instructions can only point at the top of extended blocks;
 /// - the last instruction of each block is a terminator instruction which has no natural successor,
@@ -214,7 +214,7 @@ impl<'short, 'long> InstBuilderBase<'short> for FuncInstBuilder<'short, 'long> {
 ///
 /// The functions below will panic in debug mode whenever you try to modify the Cranelift IR
 /// function in a way that violate the coherence of the code. For instance: switching to a new
-/// `Ebb` when you haven't filled the current one with a terminator instruction, inserting a
+/// `Block` when you haven't filled the current one with a terminator instruction, inserting a
 /// return instruction with arguments that don't match the function's signature.
 impl<'a> FunctionBuilder<'a> {
     /// Creates a new FunctionBuilder structure that will operate on a `Function` using a
@@ -234,26 +234,26 @@ impl<'a> FunctionBuilder<'a> {
         self.srcloc = srcloc;
     }
 
-    /// Creates a new `Ebb` and returns its reference.
-    pub fn create_ebb(&mut self) -> Ebb {
-        let ebb = self.func.dfg.make_ebb();
-        self.func_ctx.ssa.declare_ebb_header_block(ebb);
-        self.func_ctx.ebbs[ebb] = EbbData {
+    /// Creates a new `Block` and returns its reference.
+    pub fn create_block(&mut self) -> Block {
+        let block = self.func.dfg.make_block();
+        self.func_ctx.ssa.declare_block_header_block(block);
+        self.func_ctx.blocks[block] = BlockData {
             filled: false,
             pristine: true,
             user_param_count: 0,
         };
-        ebb
+        block
     }
 
     /// After the call to this function, new instructions will be inserted into the designated
-    /// block, in the order they are declared. You must declare the types of the Ebb arguments
+    /// block, in the order they are declared. You must declare the types of the Block arguments
     /// you will use here.
     ///
     /// When inserting the terminator instruction (which doesn't have a fallthrough to its immediate
     /// successor), the block will be declared filled and it will not be possible to append
     /// instructions to it.
-    pub fn switch_to_block(&mut self, ebb: Ebb) {
+    pub fn switch_to_block(&mut self, block: Block) {
         // First we check that the previous block has been filled.
         debug_assert!(
             self.position.is_default()
@@ -264,33 +264,33 @@ impl<'a> FunctionBuilder<'a> {
         );
         // We cannot switch to a filled block
         debug_assert!(
-            !self.func_ctx.ebbs[ebb].filled,
+            !self.func_ctx.blocks[block].filled,
             "you cannot switch to a block which is already filled"
         );
 
-        let basic_block = self.func_ctx.ssa.header_block(ebb);
+        let basic_block = self.func_ctx.ssa.header_block(block);
         // Then we change the cursor position.
-        self.position = Position::at(ebb, basic_block);
+        self.position = Position::at(block, basic_block);
     }
 
     /// Declares that all the predecessors of this block are known.
     ///
-    /// Function to call with `ebb` as soon as the last branch instruction to `ebb` has been
+    /// Function to call with `block` as soon as the last branch instruction to `block` has been
     /// created. Forgetting to call this method on every block will cause inconsistencies in the
     /// produced functions.
-    pub fn seal_block(&mut self, ebb: Ebb) {
-        let side_effects = self.func_ctx.ssa.seal_ebb_header_block(ebb, self.func);
+    pub fn seal_block(&mut self, block: Block) {
+        let side_effects = self.func_ctx.ssa.seal_block_header_block(block, self.func);
         self.handle_ssa_side_effects(side_effects);
     }
 
     /// Effectively calls seal_block on all blocks in the function.
     ///
-    /// It's more efficient to seal `Ebb`s as soon as possible, during
+    /// It's more efficient to seal `Block`s as soon as possible, during
     /// translation, but for frontends where this is impractical to do, this
     /// function can be used at the end of translating all blocks to ensure
     /// that everything is sealed.
     pub fn seal_all_blocks(&mut self) {
-        let side_effects = self.func_ctx.ssa.seal_all_ebb_header_blocks(self.func);
+        let side_effects = self.func_ctx.ssa.seal_all_block_header_blocks(self.func);
         self.handle_ssa_side_effects(side_effects);
     }
 
@@ -392,26 +392,26 @@ impl<'a> FunctionBuilder<'a> {
     }
 
     /// Returns an object with the [`InstBuilder`](cranelift_codegen::ir::InstBuilder)
-    /// trait that allows to conveniently append an instruction to the current `Ebb` being built.
+    /// trait that allows to conveniently append an instruction to the current `Block` being built.
     pub fn ins<'short>(&'short mut self) -> FuncInstBuilder<'short, 'a> {
-        let ebb = self
+        let block = self
             .position
-            .ebb
+            .block
             .expect("Please call switch_to_block before inserting instructions");
-        FuncInstBuilder::new(self, ebb)
+        FuncInstBuilder::new(self, block)
     }
 
-    /// Make sure that the current EBB is inserted in the layout.
-    pub fn ensure_inserted_ebb(&mut self) {
-        let ebb = self.position.ebb.unwrap();
-        if self.func_ctx.ebbs[ebb].pristine {
-            if !self.func.layout.is_ebb_inserted(ebb) {
-                self.func.layout.append_ebb(ebb);
+    /// Make sure that the current block is inserted in the layout.
+    pub fn ensure_inserted_block(&mut self) {
+        let block = self.position.block.unwrap();
+        if self.func_ctx.blocks[block].pristine {
+            if !self.func.layout.is_block_inserted(block) {
+                self.func.layout.append_block(block);
             }
-            self.func_ctx.ebbs[ebb].pristine = false;
+            self.func_ctx.blocks[block].pristine = false;
         } else {
             debug_assert!(
-                !self.func_ctx.ebbs[ebb].filled,
+                !self.func_ctx.blocks[block].filled,
                 "you cannot add an instruction to a block already filled"
             );
         }
@@ -422,40 +422,40 @@ impl<'a> FunctionBuilder<'a> {
     /// This can be used to insert SSA code that doesn't need to access locals and that doesn't
     /// need to know about `FunctionBuilder` at all.
     pub fn cursor(&mut self) -> FuncCursor {
-        self.ensure_inserted_ebb();
+        self.ensure_inserted_block();
         FuncCursor::new(self.func)
             .with_srcloc(self.srcloc)
-            .at_bottom(self.position.ebb.unwrap())
+            .at_bottom(self.position.block.unwrap())
     }
 
-    /// Append parameters to the given `Ebb` corresponding to the function
-    /// parameters. This can be used to set up the ebb parameters for the
+    /// Append parameters to the given `Block` corresponding to the function
+    /// parameters. This can be used to set up the block parameters for the
     /// entry block.
-    pub fn append_ebb_params_for_function_params(&mut self, ebb: Ebb) {
+    pub fn append_block_params_for_function_params(&mut self, block: Block) {
         debug_assert!(
-            !self.func_ctx.ssa.has_any_predecessors(ebb),
-            "ebb parameters for function parameters should only be added to the entry block"
+            !self.func_ctx.ssa.has_any_predecessors(block),
+            "block parameters for function parameters should only be added to the entry block"
         );
 
         // These parameters count as "user" parameters here because they aren't
         // inserted by the SSABuilder.
-        let user_param_count = &mut self.func_ctx.ebbs[ebb].user_param_count;
+        let user_param_count = &mut self.func_ctx.blocks[block].user_param_count;
         for argtyp in &self.func.signature.params {
             *user_param_count += 1;
-            self.func.dfg.append_ebb_param(ebb, argtyp.value_type);
+            self.func.dfg.append_block_param(block, argtyp.value_type);
         }
     }
 
-    /// Append parameters to the given `Ebb` corresponding to the function
-    /// return values. This can be used to set up the ebb parameters for a
+    /// Append parameters to the given `Block` corresponding to the function
+    /// return values. This can be used to set up the block parameters for a
     /// function exit block.
-    pub fn append_ebb_params_for_function_returns(&mut self, ebb: Ebb) {
+    pub fn append_block_params_for_function_returns(&mut self, block: Block) {
         // These parameters count as "user" parameters here because they aren't
         // inserted by the SSABuilder.
-        let user_param_count = &mut self.func_ctx.ebbs[ebb].user_param_count;
+        let user_param_count = &mut self.func_ctx.blocks[block].user_param_count;
         for argtyp in &self.func.signature.returns {
             *user_param_count += 1;
-            self.func.dfg.append_ebb_param(ebb, argtyp.value_type);
+            self.func.dfg.append_block_param(block, argtyp.value_type);
         }
     }
 
@@ -463,19 +463,18 @@ impl<'a> FunctionBuilder<'a> {
     /// resets the state of the `FunctionBuilder` in preparation to be used
     /// for another function.
     pub fn finalize(&mut self) {
-        // Check that all the `Ebb`s are filled and sealed.
+        // Check that all the `Block`s are filled and sealed.
         debug_assert!(
-            self.func_ctx
-                .ebbs
-                .iter()
-                .all(|(ebb, ebb_data)| ebb_data.pristine || self.func_ctx.ssa.is_sealed(ebb)),
+            self.func_ctx.blocks.iter().all(
+                |(block, block_data)| block_data.pristine || self.func_ctx.ssa.is_sealed(block)
+            ),
             "all blocks should be sealed before dropping a FunctionBuilder"
         );
         debug_assert!(
             self.func_ctx
-                .ebbs
+                .blocks
                 .values()
-                .all(|ebb_data| ebb_data.pristine || ebb_data.filled),
+                .all(|block_data| block_data.pristine || block_data.filled),
             "all blocks should be filled before dropping a FunctionBuilder"
         );
 
@@ -483,10 +482,10 @@ impl<'a> FunctionBuilder<'a> {
         #[cfg(debug_assertions)]
         {
             // Iterate manually to provide more helpful error messages.
-            for ebb in self.func_ctx.ebbs.keys() {
-                if let Err((inst, _msg)) = self.func.is_ebb_basic(ebb) {
+            for block in self.func_ctx.blocks.keys() {
+                if let Err((inst, _msg)) = self.func.is_block_basic(block) {
                     let inst_str = self.func.dfg.display_inst(inst, None);
-                    panic!("{} failed basic block invariants on {}", ebb, inst_str);
+                    panic!("{} failed basic block invariants on {}", block, inst_str);
                 }
             }
         }
@@ -507,10 +506,10 @@ impl<'a> FunctionBuilder<'a> {
 /// function. The functions below help you inspect the function you're creating and modify it
 /// in ways that can be unsafe if used incorrectly.
 impl<'a> FunctionBuilder<'a> {
-    /// Retrieves all the parameters for an `Ebb` currently inferred from the jump instructions
+    /// Retrieves all the parameters for a `Block` currently inferred from the jump instructions
     /// inserted that target it and the SSA construction.
-    pub fn ebb_params(&self, ebb: Ebb) -> &[Value] {
-        self.func.dfg.ebb_params(ebb)
+    pub fn block_params(&self, block: Block) -> &[Value] {
+        self.func.dfg.block_params(block)
     }
 
     /// Retrieves the signature with reference `sigref` previously added with `import_signature`.
@@ -518,22 +517,22 @@ impl<'a> FunctionBuilder<'a> {
         self.func.dfg.signatures.get(sigref)
     }
 
-    /// Creates a parameter for a specific `Ebb` by appending it to the list of already existing
+    /// Creates a parameter for a specific `Block` by appending it to the list of already existing
     /// parameters.
     ///
-    /// **Note:** this function has to be called at the creation of the `Ebb` before adding
+    /// **Note:** this function has to be called at the creation of the `Block` before adding
     /// instructions to it, otherwise this could interfere with SSA construction.
-    pub fn append_ebb_param(&mut self, ebb: Ebb, ty: Type) -> Value {
+    pub fn append_block_param(&mut self, block: Block, ty: Type) -> Value {
         debug_assert!(
-            self.func_ctx.ebbs[ebb].pristine,
-            "You can't add EBB parameters after adding any instruction"
+            self.func_ctx.blocks[block].pristine,
+            "You can't add block parameters after adding any instruction"
         );
         debug_assert_eq!(
-            self.func_ctx.ebbs[ebb].user_param_count,
-            self.func.dfg.num_ebb_params(ebb)
+            self.func_ctx.blocks[block].user_param_count,
+            self.func.dfg.num_block_params(block)
         );
-        self.func_ctx.ebbs[ebb].user_param_count += 1;
-        self.func.dfg.append_ebb_param(ebb, ty)
+        self.func_ctx.blocks[block].user_param_count += 1;
+        self.func.dfg.append_block_param(block, ty)
     }
 
     /// Returns the result values of an instruction.
@@ -545,43 +544,43 @@ impl<'a> FunctionBuilder<'a> {
     ///
     /// **Note:** You are responsible for maintaining the coherence with the arguments of
     /// other jump instructions.
-    pub fn change_jump_destination(&mut self, inst: Inst, new_dest: Ebb) {
+    pub fn change_jump_destination(&mut self, inst: Inst, new_dest: Block) {
         let old_dest = self.func.dfg[inst]
             .branch_destination_mut()
             .expect("you want to change the jump destination of a non-jump instruction");
-        let pred = self.func_ctx.ssa.remove_ebb_predecessor(*old_dest, inst);
+        let pred = self.func_ctx.ssa.remove_block_predecessor(*old_dest, inst);
         *old_dest = new_dest;
         self.func_ctx
             .ssa
-            .declare_ebb_predecessor(new_dest, pred, inst);
+            .declare_block_predecessor(new_dest, pred, inst);
     }
 
-    /// Returns `true` if and only if the current `Ebb` is sealed and has no predecessors declared.
+    /// Returns `true` if and only if the current `Block` is sealed and has no predecessors declared.
     ///
     /// The entry block of a function is never unreachable.
     pub fn is_unreachable(&self) -> bool {
         let is_entry = match self.func.layout.entry_block() {
             None => false,
-            Some(entry) => self.position.ebb.unwrap() == entry,
+            Some(entry) => self.position.block.unwrap() == entry,
         };
         !is_entry
-            && self.func_ctx.ssa.is_sealed(self.position.ebb.unwrap())
+            && self.func_ctx.ssa.is_sealed(self.position.block.unwrap())
             && !self
                 .func_ctx
                 .ssa
-                .has_any_predecessors(self.position.ebb.unwrap())
+                .has_any_predecessors(self.position.block.unwrap())
     }
 
     /// Returns `true` if and only if no instructions have been added since the last call to
     /// `switch_to_block`.
     pub fn is_pristine(&self) -> bool {
-        self.func_ctx.ebbs[self.position.ebb.unwrap()].pristine
+        self.func_ctx.blocks[self.position.block.unwrap()].pristine
     }
 
     /// Returns `true` if and only if a terminator instruction has been inserted since the
     /// last call to `switch_to_block`.
     pub fn is_filled(&self) -> bool {
-        self.func_ctx.ebbs[self.position.ebb.unwrap()].filled
+        self.func_ctx.blocks[self.position.block.unwrap()].filled
     }
 
     /// Returns a displayable object for the function as it is.
@@ -860,29 +859,29 @@ impl<'a> FunctionBuilder<'a> {
         self.position.basic_block = PackedOption::from(
             self.func_ctx
                 .ssa
-                .declare_ebb_body_block(self.position.basic_block.unwrap()),
+                .declare_block_body_block(self.position.basic_block.unwrap()),
         );
     }
 
-    /// An Ebb is 'filled' when a terminator instruction is present.
+    /// An Block is 'filled' when a terminator instruction is present.
     fn fill_current_block(&mut self) {
-        self.func_ctx.ebbs[self.position.ebb.unwrap()].filled = true;
+        self.func_ctx.blocks[self.position.block.unwrap()].filled = true;
     }
 
-    fn declare_successor(&mut self, dest_ebb: Ebb, jump_inst: Inst) {
-        self.func_ctx.ssa.declare_ebb_predecessor(
-            dest_ebb,
+    fn declare_successor(&mut self, dest_block: Block, jump_inst: Inst) {
+        self.func_ctx.ssa.declare_block_predecessor(
+            dest_block,
             self.position.basic_block.unwrap(),
             jump_inst,
         );
     }
 
     fn handle_ssa_side_effects(&mut self, side_effects: SideEffects) {
-        for split_ebb in side_effects.split_ebbs_created {
-            self.func_ctx.ebbs[split_ebb].filled = true
+        for split_block in side_effects.split_blocks_created {
+            self.func_ctx.blocks[split_block].filled = true
         }
-        for modified_ebb in side_effects.instructions_added_to_ebbs {
-            self.func_ctx.ebbs[modified_ebb].pristine = false
+        for modified_block in side_effects.instructions_added_to_blocks {
+            self.func_ctx.blocks[modified_block].pristine = false
         }
     }
 }
@@ -910,24 +909,24 @@ mod tests {
         {
             let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
 
-            let block0 = builder.create_ebb();
-            let block1 = builder.create_ebb();
-            let block2 = builder.create_ebb();
-            let block3 = builder.create_ebb();
+            let block0 = builder.create_block();
+            let block1 = builder.create_block();
+            let block2 = builder.create_block();
+            let block3 = builder.create_block();
             let x = Variable::new(0);
             let y = Variable::new(1);
             let z = Variable::new(2);
             builder.declare_var(x, I32);
             builder.declare_var(y, I32);
             builder.declare_var(z, I32);
-            builder.append_ebb_params_for_function_params(block0);
+            builder.append_block_params_for_function_params(block0);
 
             builder.switch_to_block(block0);
             if !lazy_seal {
                 builder.seal_block(block0);
             }
             {
-                let tmp = builder.ebb_params(block0)[0]; // the first function parameter
+                let tmp = builder.block_params(block0)[0]; // the first function parameter
                 builder.def_var(x, tmp);
             }
             {
@@ -1033,14 +1032,14 @@ mod tests {
         {
             let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
 
-            let block0 = builder.create_ebb();
+            let block0 = builder.create_block();
             let x = Variable::new(0);
             let y = Variable::new(1);
             let z = Variable::new(2);
             builder.declare_var(x, target.pointer_type());
             builder.declare_var(y, target.pointer_type());
             builder.declare_var(z, I32);
-            builder.append_ebb_params_for_function_params(block0);
+            builder.append_block_params_for_function_params(block0);
             builder.switch_to_block(block0);
 
             let src = builder.use_var(x);
@@ -1059,7 +1058,7 @@ mod tests {
     sig0 = (i32, i32, i32) system_v
     fn0 = %Memcpy sig0
 
-ebb0:
+block0:
     v3 = iconst.i32 0
     v1 -> v3
     v2 = iconst.i32 0
@@ -1094,12 +1093,12 @@ ebb0:
         {
             let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
 
-            let block0 = builder.create_ebb();
+            let block0 = builder.create_block();
             let x = Variable::new(0);
             let y = Variable::new(16);
             builder.declare_var(x, target.pointer_type());
             builder.declare_var(y, target.pointer_type());
-            builder.append_ebb_params_for_function_params(block0);
+            builder.append_block_params_for_function_params(block0);
             builder.switch_to_block(block0);
 
             let src = builder.use_var(x);
@@ -1115,7 +1114,7 @@ ebb0:
         assert_eq!(
             func.display(None).to_string(),
             "function %sample() -> i32 system_v {
-ebb0:
+block0:
     v4 = iconst.i32 0
     v1 -> v4
     v3 = iconst.i32 0
@@ -1151,12 +1150,12 @@ ebb0:
         {
             let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
 
-            let block0 = builder.create_ebb();
+            let block0 = builder.create_block();
             let x = Variable::new(0);
             let y = Variable::new(16);
             builder.declare_var(x, target.pointer_type());
             builder.declare_var(y, target.pointer_type());
-            builder.append_ebb_params_for_function_params(block0);
+            builder.append_block_params_for_function_params(block0);
             builder.switch_to_block(block0);
 
             let src = builder.use_var(x);
@@ -1175,7 +1174,7 @@ ebb0:
     sig0 = (i32, i32, i32) system_v
     fn0 = %Memcpy sig0
 
-ebb0:
+block0:
     v4 = iconst.i32 0
     v1 -> v4
     v3 = iconst.i32 0
@@ -1211,10 +1210,10 @@ ebb0:
         {
             let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
 
-            let block0 = builder.create_ebb();
+            let block0 = builder.create_block();
             let y = Variable::new(16);
             builder.declare_var(y, target.pointer_type());
-            builder.append_ebb_params_for_function_params(block0);
+            builder.append_block_params_for_function_params(block0);
             builder.switch_to_block(block0);
 
             let dest = builder.use_var(y);
@@ -1229,7 +1228,7 @@ ebb0:
         assert_eq!(
             func.display(None).to_string(),
             "function %sample() -> i32 system_v {
-ebb0:
+block0:
     v2 = iconst.i32 0
     v0 -> v2
     v1 = iconst.i64 0x0001_0001_0101
@@ -1263,10 +1262,10 @@ ebb0:
         {
             let mut builder = FunctionBuilder::new(&mut func, &mut fn_ctx);
 
-            let block0 = builder.create_ebb();
+            let block0 = builder.create_block();
             let y = Variable::new(16);
             builder.declare_var(y, target.pointer_type());
-            builder.append_ebb_params_for_function_params(block0);
+            builder.append_block_params_for_function_params(block0);
             builder.switch_to_block(block0);
 
             let dest = builder.use_var(y);
@@ -1284,7 +1283,7 @@ ebb0:
     sig0 = (i32, i32, i32) system_v
     fn0 = %Memset sig0
 
-ebb0:
+block0:
     v4 = iconst.i32 0
     v0 -> v4
     v1 = iconst.i8 1
diff --git a/cranelift/frontend/src/lib.rs b/cranelift/frontend/src/lib.rs
index d6d63381ce..d28cb53cdf 100644
--- a/cranelift/frontend/src/lib.rs
+++ b/cranelift/frontend/src/lib.rs
@@ -83,22 +83,22 @@
 //! {
 //!     let mut builder = FunctionBuilder::new(&mut func, &mut fn_builder_ctx);
 //!
-//!     let block0 = builder.create_ebb();
-//!     let block1 = builder.create_ebb();
-//!     let block2 = builder.create_ebb();
-//!     let block3 = builder.create_ebb();
+//!     let block0 = builder.create_block();
+//!     let block1 = builder.create_block();
+//!     let block2 = builder.create_block();
+//!     let block3 = builder.create_block();
 //!     let x = Variable::new(0);
 //!     let y = Variable::new(1);
 //!     let z = Variable::new(2);
 //!     builder.declare_var(x, I32);
 //!     builder.declare_var(y, I32);
 //!     builder.declare_var(z, I32);
-//!     builder.append_ebb_params_for_function_params(block0);
+//!     builder.append_block_params_for_function_params(block0);
 //!
 //!     builder.switch_to_block(block0);
 //!     builder.seal_block(block0);
 //!     {
-//!         let tmp = builder.ebb_params(block0)[0]; // the first function parameter
+//!         let tmp = builder.block_params(block0)[0]; // the first function parameter
 //!         builder.def_var(x, tmp);
 //!     }
 //!     {
diff --git a/cranelift/frontend/src/ssa.rs b/cranelift/frontend/src/ssa.rs
index c24e373677..7d36d9b1c7 100644
--- a/cranelift/frontend/src/ssa.rs
+++ b/cranelift/frontend/src/ssa.rs
@@ -16,7 +16,7 @@ use cranelift_codegen::entity::{EntityRef, PrimaryMap, SecondaryMap};
 use cranelift_codegen::ir::immediates::{Ieee32, Ieee64};
 use cranelift_codegen::ir::instructions::BranchInfo;
 use cranelift_codegen::ir::types::{F32, F64};
-use cranelift_codegen::ir::{Ebb, Function, Inst, InstBuilder, InstructionData, Type, Value};
+use cranelift_codegen::ir::{Block, Function, Inst, InstBuilder, InstructionData, Type, Value};
 use cranelift_codegen::packed_option::PackedOption;
 use cranelift_codegen::packed_option::ReservedValue;
 use smallvec::SmallVec;
@@ -27,7 +27,7 @@ use smallvec::SmallVec;
 /// non-SSA language you're translating from.
 ///
 /// The SSA building relies on information about the variables used and defined, as well as
-/// their position relative to basic blocks which are stricter than extended basic blocks since
+/// their position relative to basic blocks which are stricter tha basic blocks since
 /// they don't allow branching in the middle of them.
 ///
 /// This SSA building module allows you to def and use variables on the fly while you are
@@ -40,14 +40,14 @@ pub struct SSABuilder {
     // TODO: Consider a sparse representation rather than SecondaryMap-of-SecondaryMap.
     /// Records for every variable and for every relevant block, the last definition of
     /// the variable in the block.
-    variables: SecondaryMap>>,
+    variables: SecondaryMap>>,
 
     /// Records the position of the basic blocks and the list of values used but not defined in the
     /// block.
-    blocks: PrimaryMap,
+    ssa_blocks: PrimaryMap,
 
-    /// Records the basic blocks at the beginning of the `Ebb`s.
-    ebb_headers: SecondaryMap>,
+    /// Records the basic blocks at the beginning of the `Block`s.
+    block_headers: SecondaryMap>,
 
     /// Call stack for use in the `use_var`/`predecessors_lookup` state machine.
     calls: Vec,
@@ -58,54 +58,54 @@ pub struct SSABuilder {
     side_effects: SideEffects,
 }
 
-/// Side effects of a `use_var` or a `seal_ebb_header_block` method call.
+/// Side effects of a `use_var` or a `seal_block_header_block` method call.
 pub struct SideEffects {
     /// When we want to append jump arguments to a `br_table` instruction, the critical edge is
-    /// splitted and the newly created `Ebb`s are signaled here.
-    pub split_ebbs_created: Vec,
+    /// splitted and the newly created `Block`s are signaled here.
+    pub split_blocks_created: Vec,
     /// When a variable is used but has never been defined before (this happens in the case of
-    /// unreachable code), a placeholder `iconst` or `fconst` value is added to the right `Ebb`.
-    /// This field signals if it is the case and return the `Ebb` to which the initialization has
+    /// unreachable code), a placeholder `iconst` or `fconst` value is added to the right `Block`.
+    /// This field signals if it is the case and return the `Block` to which the initialization has
     /// been added.
-    pub instructions_added_to_ebbs: Vec,
+    pub instructions_added_to_blocks: Vec,
 }
 
 impl SideEffects {
     fn new() -> Self {
         Self {
-            split_ebbs_created: Vec::new(),
-            instructions_added_to_ebbs: Vec::new(),
+            split_blocks_created: Vec::new(),
+            instructions_added_to_blocks: Vec::new(),
         }
     }
 
     fn is_empty(&self) -> bool {
-        self.split_ebbs_created.is_empty() && self.instructions_added_to_ebbs.is_empty()
+        self.split_blocks_created.is_empty() && self.instructions_added_to_blocks.is_empty()
     }
 }
 
 /// Describes the current position of a basic block in the control flow graph.
-enum BlockData {
-    /// A block at the top of an `Ebb`.
-    EbbHeader(EbbHeaderBlockData),
-    /// A block inside an `Ebb` with an unique other block as its predecessor.
+enum SSABlockData {
+    /// A block at the top of a `Block`.
+    BlockHeader(BlockHeaderSSABlockData),
+    /// A block inside a `Block` with an unique other block as its predecessor.
     /// The block is implicitly sealed at creation.
-    EbbBody { predecessor: Block },
+    BlockBody { ssa_pred: SSABlock },
 }
 
-impl BlockData {
-    fn add_predecessor(&mut self, pred: Block, inst: Inst) {
+impl SSABlockData {
+    fn add_predecessor(&mut self, ssa_pred: SSABlock, inst: Inst) {
         match *self {
-            Self::EbbBody { .. } => panic!("you can't add a predecessor to a body block"),
-            Self::EbbHeader(ref mut data) => {
+            Self::BlockBody { .. } => panic!("you can't add a predecessor to a body block"),
+            Self::BlockHeader(ref mut data) => {
                 debug_assert!(!data.sealed, "sealed blocks cannot accept new predecessors");
-                data.predecessors.push(PredBlock::new(pred, inst));
+                data.predecessors.push(PredBlock::new(ssa_pred, inst));
             }
         }
     }
-    fn remove_predecessor(&mut self, inst: Inst) -> Block {
+    fn remove_predecessor(&mut self, inst: Inst) -> SSABlock {
         match *self {
-            Self::EbbBody { .. } => panic!("should not happen"),
-            Self::EbbHeader(ref mut data) => {
+            Self::BlockBody { .. } => panic!("should not happen"),
+            Self::BlockHeader(ref mut data) => {
                 // This a linear complexity operation but the number of predecessors is low
                 // in all non-pathological cases
                 let pred: usize = data
@@ -113,40 +113,40 @@ impl BlockData {
                     .iter()
                     .position(|&PredBlock { branch, .. }| branch == inst)
                     .expect("the predecessor you are trying to remove is not declared");
-                data.predecessors.swap_remove(pred).block
+                data.predecessors.swap_remove(pred).ssa_block
             }
         }
     }
 }
 
 struct PredBlock {
-    block: Block,
+    ssa_block: SSABlock,
     branch: Inst,
 }
 
 impl PredBlock {
-    fn new(block: Block, branch: Inst) -> Self {
-        Self { block, branch }
+    fn new(ssa_block: SSABlock, branch: Inst) -> Self {
+        Self { ssa_block, branch }
     }
 }
 
 type PredBlockSmallVec = SmallVec<[PredBlock; 4]>;
 
-struct EbbHeaderBlockData {
-    // The predecessors of the Ebb header block, with the block and branch instruction.
+struct BlockHeaderSSABlockData {
+    // The predecessors of the Block header block, with the block and branch instruction.
     predecessors: PredBlockSmallVec,
-    // A ebb header block is sealed if all of its predecessors have been declared.
+    // A block header block is sealed if all of its predecessors have been declared.
     sealed: bool,
-    // The ebb which this block is part of.
-    ebb: Ebb,
-    // List of current Ebb arguments for which an earlier def has not been found yet.
+    // The block which this block is part of.
+    block: Block,
+    // List of current Block arguments for which an earlier def has not been found yet.
     undef_variables: Vec<(Variable, Value)>,
 }
 
 /// A opaque reference to a basic block.
 #[derive(Copy, Clone, PartialEq, Eq, Debug)]
-pub struct Block(u32);
-impl EntityRef for Block {
+pub struct SSABlock(u32);
+impl EntityRef for SSABlock {
     fn new(index: usize) -> Self {
         debug_assert!(index < (u32::MAX as usize));
         Self(index as u32)
@@ -157,7 +157,7 @@ impl EntityRef for Block {
     }
 }
 
-impl ReservedValue for Block {
+impl ReservedValue for SSABlock {
     fn reserved_value() -> Self {
         Self(u32::MAX)
     }
@@ -168,8 +168,8 @@ impl SSABuilder {
     pub fn new() -> Self {
         Self {
             variables: SecondaryMap::with_default(SecondaryMap::new()),
-            blocks: PrimaryMap::new(),
-            ebb_headers: SecondaryMap::new(),
+            ssa_blocks: PrimaryMap::new(),
+            block_headers: SecondaryMap::new(),
             calls: Vec::new(),
             results: Vec::new(),
             side_effects: SideEffects::new(),
@@ -180,8 +180,8 @@ impl SSABuilder {
     /// deallocating memory.
     pub fn clear(&mut self) {
         self.variables.clear();
-        self.blocks.clear();
-        self.ebb_headers.clear();
+        self.ssa_blocks.clear();
+        self.block_headers.clear();
         debug_assert!(self.calls.is_empty());
         debug_assert!(self.results.is_empty());
         debug_assert!(self.side_effects.is_empty());
@@ -190,8 +190,8 @@ impl SSABuilder {
     /// Tests whether an `SSABuilder` is in a cleared state.
     pub fn is_empty(&self) -> bool {
         self.variables.is_empty()
-            && self.blocks.is_empty()
-            && self.ebb_headers.is_empty()
+            && self.ssa_blocks.is_empty()
+            && self.block_headers.is_empty()
             && self.calls.is_empty()
             && self.results.is_empty()
             && self.side_effects.is_empty()
@@ -210,15 +210,15 @@ enum ZeroOneOrMore {
 #[derive(Debug)]
 enum UseVarCases {
     Unsealed(Value),
-    SealedOnePredecessor(Block),
-    SealedMultiplePredecessors(Value, Ebb),
+    SealedOnePredecessor(SSABlock),
+    SealedMultiplePredecessors(Value, Block),
 }
 
 /// States for the `use_var`/`predecessors_lookup` state machine.
 enum Call {
-    UseVar(Block),
-    FinishSealedOnePredecessor(Block),
-    FinishPredecessorsLookup(Value, Ebb),
+    UseVar(SSABlock),
+    FinishSealedOnePredecessor(SSABlock),
+    FinishPredecessorsLookup(Value, Block),
 }
 
 /// Emit instructions to produce a zero value in the given type.
@@ -257,7 +257,7 @@ fn emit_zero(ty: Type, mut cur: FuncCursor) -> Value {
 /// translating to Cranelift IR:
 ///
 /// - for each sequence of contiguous instructions (with no branches), create a corresponding
-///   basic block with `declare_ebb_body_block` or `declare_ebb_header_block` depending on the
+///   basic block with `declare_block_body_block` or `declare_block_header_block` depending on the
 ///   position of the basic block;
 ///
 /// - while traversing a basic block and translating instruction, use `def_var` and `use_var`
@@ -265,25 +265,25 @@ fn emit_zero(ty: Type, mut cur: FuncCursor) -> Value {
 ///   SSA values;
 ///
 /// - when all the instructions in a basic block have translated, the block is said _filled_ and
-///   only then you can add it as a predecessor to other blocks with `declare_ebb_predecessor`;
+///   only then you can add it as a predecessor to other blocks with `declare_block_predecessor`;
 ///
-/// - when you have constructed all the predecessor to a basic block at the beginning of an `Ebb`,
-///   call `seal_ebb_header_block` on it with the `Function` that you are building.
+/// - when you have constructed all the predecessor to a basic block at the beginning of a `Block`,
+///   call `seal_block_header_block` on it with the `Function` that you are building.
 ///
 /// This API will give you the correct SSA values to use as arguments of your instructions,
-/// as well as modify the jump instruction and `Ebb` headers parameters to account for the SSA
+/// as well as modify the jump instruction and `Block` headers parameters to account for the SSA
 /// Phi functions.
 ///
 impl SSABuilder {
     /// Declares a new definition of a variable in a given basic block.
     /// The SSA value is passed as an argument because it should be created with
     /// `ir::DataFlowGraph::append_result`.
-    pub fn def_var(&mut self, var: Variable, val: Value, block: Block) {
-        self.variables[var][block] = PackedOption::from(val);
+    pub fn def_var(&mut self, var: Variable, val: Value, ssa_block: SSABlock) {
+        self.variables[var][ssa_block] = PackedOption::from(val);
     }
 
     /// Declares a use of a variable in a given basic block. Returns the SSA value corresponding
-    /// to the current SSA definition of this variable and a list of newly created Ebbs that
+    /// to the current SSA definition of this variable and a list of newly created Blocks that
     /// are the results of critical edge splitting for `br_table` with arguments.
     ///
     /// If the variable has never been defined in this blocks or recursively in its predecessors,
@@ -294,12 +294,12 @@ impl SSABuilder {
         func: &mut Function,
         var: Variable,
         ty: Type,
-        block: Block,
+        ssa_block: SSABlock,
     ) -> (Value, SideEffects) {
         // First, try Local Value Numbering (Algorithm 1 in the paper).
         // If the variable already has a known Value in this block, use that.
         if let Some(var_defs) = self.variables.get(var) {
-            if let Some(val) = var_defs[block].expand() {
+            if let Some(val) = var_defs[ssa_block].expand() {
                 return (val, SideEffects::new());
             }
         }
@@ -311,7 +311,7 @@ impl SSABuilder {
         debug_assert!(self.side_effects.is_empty());
 
         // Prepare the 'calls' and 'results' stacks for the state machine.
-        self.use_var_nonlocal(func, var, ty, block);
+        self.use_var_nonlocal(func, var, ty, ssa_block);
 
         let value = self.run_state_machine(func, var, ty);
         let side_effects = mem::replace(&mut self.side_effects, SideEffects::new());
@@ -322,182 +322,190 @@ impl SSABuilder {
     /// Resolve the minimal SSA Value of `var` in `block` by traversing predecessors.
     ///
     /// This function sets up state for `run_state_machine()` but does not execute it.
-    fn use_var_nonlocal(&mut self, func: &mut Function, var: Variable, ty: Type, block: Block) {
+    fn use_var_nonlocal(
+        &mut self,
+        func: &mut Function,
+        var: Variable,
+        ty: Type,
+        ssa_block: SSABlock,
+    ) {
         // This function is split into two parts to appease the borrow checker.
         // Part 1: With a mutable borrow of self, update the DataFlowGraph if necessary.
-        let case = match self.blocks[block] {
-            BlockData::EbbHeader(ref mut data) => {
-                // The block has multiple predecessors so we append an Ebb parameter that
+        let case = match self.ssa_blocks[ssa_block] {
+            SSABlockData::BlockHeader(ref mut data) => {
+                // The block has multiple predecessors so we append an Block parameter that
                 // will serve as a value.
                 if data.sealed {
                     if data.predecessors.len() == 1 {
                         // Optimize the common case of one predecessor: no param needed.
-                        UseVarCases::SealedOnePredecessor(data.predecessors[0].block)
+                        UseVarCases::SealedOnePredecessor(data.predecessors[0].ssa_block)
                     } else {
                         // Break potential cycles by eagerly adding an operandless param.
-                        let val = func.dfg.append_ebb_param(data.ebb, ty);
-                        UseVarCases::SealedMultiplePredecessors(val, data.ebb)
+                        let val = func.dfg.append_block_param(data.block, ty);
+                        UseVarCases::SealedMultiplePredecessors(val, data.block)
                     }
                 } else {
-                    let val = func.dfg.append_ebb_param(data.ebb, ty);
+                    let val = func.dfg.append_block_param(data.block, ty);
                     data.undef_variables.push((var, val));
                     UseVarCases::Unsealed(val)
                 }
             }
-            BlockData::EbbBody { predecessor: pred } => UseVarCases::SealedOnePredecessor(pred),
+            SSABlockData::BlockBody { ssa_pred } => UseVarCases::SealedOnePredecessor(ssa_pred),
         };
 
         // Part 2: Prepare SSABuilder state for run_state_machine().
         match case {
             UseVarCases::SealedOnePredecessor(pred) => {
                 // Get the Value directly from the single predecessor.
-                self.calls.push(Call::FinishSealedOnePredecessor(block));
+                self.calls.push(Call::FinishSealedOnePredecessor(ssa_block));
                 self.calls.push(Call::UseVar(pred));
             }
             UseVarCases::Unsealed(val) => {
                 // Define the operandless param added above to prevent lookup cycles.
-                self.def_var(var, val, block);
+                self.def_var(var, val, ssa_block);
 
                 // Nothing more can be known at this point.
                 self.results.push(val);
             }
-            UseVarCases::SealedMultiplePredecessors(val, ebb) => {
+            UseVarCases::SealedMultiplePredecessors(val, block) => {
                 // Define the operandless param added above to prevent lookup cycles.
-                self.def_var(var, val, block);
+                self.def_var(var, val, ssa_block);
 
                 // Look up a use_var for each precessor.
-                self.begin_predecessors_lookup(val, ebb);
+                self.begin_predecessors_lookup(val, block);
             }
         }
     }
 
     /// For blocks with a single predecessor, once we've determined the value,
     /// record a local def for it for future queries to find.
-    fn finish_sealed_one_predecessor(&mut self, var: Variable, block: Block) {
+    fn finish_sealed_one_predecessor(&mut self, var: Variable, ssa_block: SSABlock) {
         let val = *self.results.last().unwrap();
-        self.def_var(var, val, block);
+        self.def_var(var, val, ssa_block);
     }
 
-    /// Declares a new basic block belonging to the body of a certain `Ebb` and having `pred`
+    /// Declares a new basic block belonging to the body of a certain `Block` and having `pred`
     /// as a predecessor. `pred` is the only predecessor of the block and the block is sealed
     /// at creation.
     ///
-    /// To declare a `Ebb` header block, see `declare_ebb_header_block`.
-    pub fn declare_ebb_body_block(&mut self, pred: Block) -> Block {
-        self.blocks.push(BlockData::EbbBody { predecessor: pred })
+    /// To declare a `Block` header block, see `declare_block_header_block`.
+    pub fn declare_block_body_block(&mut self, ssa_pred: SSABlock) -> SSABlock {
+        self.ssa_blocks.push(SSABlockData::BlockBody { ssa_pred })
     }
 
-    /// Declares a new basic block at the beginning of an `Ebb`. No predecessors are declared
+    /// Declares a new basic block at the beginning of a `Block`. No predecessors are declared
     /// here and the block is not sealed.
-    /// Predecessors have to be added with `declare_ebb_predecessor`.
-    pub fn declare_ebb_header_block(&mut self, ebb: Ebb) -> Block {
-        let block = self.blocks.push(BlockData::EbbHeader(EbbHeaderBlockData {
-            predecessors: PredBlockSmallVec::new(),
-            sealed: false,
-            ebb,
-            undef_variables: Vec::new(),
-        }));
-        self.ebb_headers[ebb] = block.into();
-        block
+    /// Predecessors have to be added with `declare_block_predecessor`.
+    pub fn declare_block_header_block(&mut self, block: Block) -> SSABlock {
+        let ssa_block = self
+            .ssa_blocks
+            .push(SSABlockData::BlockHeader(BlockHeaderSSABlockData {
+                predecessors: PredBlockSmallVec::new(),
+                sealed: false,
+                block,
+                undef_variables: Vec::new(),
+            }));
+        self.block_headers[block] = ssa_block.into();
+        ssa_block
     }
-    /// Gets the header block corresponding to an Ebb, panics if the Ebb or the header block
+    /// Gets the header block corresponding to an Block, panics if the Block or the header block
     /// isn't declared.
-    pub fn header_block(&self, ebb: Ebb) -> Block {
-        self.ebb_headers
-            .get(ebb)
-            .expect("the ebb has not been declared")
+    pub fn header_block(&self, block: Block) -> SSABlock {
+        self.block_headers
+            .get(block)
+            .expect("the block has not been declared")
             .expand()
             .expect("the header block has not been defined")
     }
 
-    /// Declares a new predecessor for an `Ebb` header block and record the branch instruction
+    /// Declares a new predecessor for a `Block` header block and record the branch instruction
     /// of the predecessor that leads to it.
     ///
-    /// Note that the predecessor is a `Block` and not an `Ebb`. This `Block` must be filled
+    /// Note that the predecessor is a `SSABlock` and not a `Block`. This `SSABlock` must be filled
     /// before added as predecessor. Note that you must provide no jump arguments to the branch
     /// instruction when you create it since `SSABuilder` will fill them for you.
     ///
     /// Callers are expected to avoid adding the same predecessor more than once in the case
     /// of a jump table.
-    pub fn declare_ebb_predecessor(&mut self, ebb: Ebb, pred: Block, inst: Inst) {
-        debug_assert!(!self.is_sealed(ebb));
-        let header_block = self.header_block(ebb);
-        self.blocks[header_block].add_predecessor(pred, inst)
+    pub fn declare_block_predecessor(&mut self, block: Block, ssa_pred: SSABlock, inst: Inst) {
+        debug_assert!(!self.is_sealed(block));
+        let header_block = self.header_block(block);
+        self.ssa_blocks[header_block].add_predecessor(ssa_pred, inst)
     }
 
-    /// Remove a previously declared Ebb predecessor by giving a reference to the jump
+    /// Remove a previously declared Block predecessor by giving a reference to the jump
     /// instruction. Returns the basic block containing the instruction.
     ///
     /// Note: use only when you know what you are doing, this might break the SSA building problem
-    pub fn remove_ebb_predecessor(&mut self, ebb: Ebb, inst: Inst) -> Block {
-        debug_assert!(!self.is_sealed(ebb));
-        let header_block = self.header_block(ebb);
-        self.blocks[header_block].remove_predecessor(inst)
+    pub fn remove_block_predecessor(&mut self, block: Block, inst: Inst) -> SSABlock {
+        debug_assert!(!self.is_sealed(block));
+        let header_block = self.header_block(block);
+        self.ssa_blocks[header_block].remove_predecessor(inst)
     }
 
-    /// Completes the global value numbering for an `Ebb`, all of its predecessors having been
+    /// Completes the global value numbering for a `Block`, all of its predecessors having been
     /// already sealed.
     ///
-    /// This method modifies the function's `Layout` by adding arguments to the `Ebb`s to
+    /// This method modifies the function's `Layout` by adding arguments to the `Block`s to
     /// take into account the Phi function placed by the SSA algorithm.
     ///
-    /// Returns the list of newly created ebbs for critical edge splitting.
-    pub fn seal_ebb_header_block(&mut self, ebb: Ebb, func: &mut Function) -> SideEffects {
-        self.seal_one_ebb_header_block(ebb, func);
+    /// Returns the list of newly created blocks for critical edge splitting.
+    pub fn seal_block_header_block(&mut self, block: Block, func: &mut Function) -> SideEffects {
+        self.seal_one_block_header_block(block, func);
         mem::replace(&mut self.side_effects, SideEffects::new())
     }
 
-    /// Completes the global value numbering for all `Ebb`s in `func`.
+    /// Completes the global value numbering for all `Block`s in `func`.
     ///
-    /// It's more efficient to seal `Ebb`s as soon as possible, during
+    /// It's more efficient to seal `Block`s as soon as possible, during
     /// translation, but for frontends where this is impractical to do, this
     /// function can be used at the end of translating all blocks to ensure
     /// that everything is sealed.
-    pub fn seal_all_ebb_header_blocks(&mut self, func: &mut Function) -> SideEffects {
-        // Seal all `Ebb`s currently in the function. This can entail splitting
+    pub fn seal_all_block_header_blocks(&mut self, func: &mut Function) -> SideEffects {
+        // Seal all `Block`s currently in the function. This can entail splitting
         // and creation of new blocks, however such new blocks are sealed on
         // the fly, so we don't need to account for them here.
-        for ebb in self.ebb_headers.keys() {
-            self.seal_one_ebb_header_block(ebb, func);
+        for block in self.block_headers.keys() {
+            self.seal_one_block_header_block(block, func);
         }
         mem::replace(&mut self.side_effects, SideEffects::new())
     }
 
-    /// Helper function for `seal_ebb_header_block` and
-    /// `seal_all_ebb_header_blocks`.
-    fn seal_one_ebb_header_block(&mut self, ebb: Ebb, func: &mut Function) {
-        let block = self.header_block(ebb);
+    /// Helper function for `seal_block_header_block` and
+    /// `seal_all_block_header_blocks`.
+    fn seal_one_block_header_block(&mut self, block: Block, func: &mut Function) {
+        let ssa_block = self.header_block(block);
 
-        let undef_vars = match self.blocks[block] {
-            BlockData::EbbBody { .. } => panic!("this should not happen"),
-            BlockData::EbbHeader(ref mut data) => {
+        let undef_vars = match self.ssa_blocks[ssa_block] {
+            SSABlockData::BlockBody { .. } => panic!("this should not happen"),
+            SSABlockData::BlockHeader(ref mut data) => {
                 debug_assert!(
                     !data.sealed,
                     "Attempting to seal {} which is already sealed.",
-                    ebb
+                    block
                 );
-                debug_assert_eq!(ebb, data.ebb);
+                debug_assert_eq!(block, data.block);
                 // Extract the undef_variables data from the block so that we
                 // can iterate over it without borrowing the whole builder.
                 mem::replace(&mut data.undef_variables, Vec::new())
             }
         };
 
-        // For each undef var we look up values in the predecessors and create an EBB parameter
+        // For each undef var we look up values in the predecessors and create an block parameter
         // only if necessary.
         for (var, val) in undef_vars {
             let ty = func.dfg.value_type(val);
-            self.predecessors_lookup(func, val, var, ty, ebb);
+            self.predecessors_lookup(func, val, var, ty, block);
         }
-        self.mark_ebb_header_block_sealed(block);
+        self.mark_block_header_block_sealed(ssa_block);
     }
 
     /// Set the `sealed` flag for `block`.
-    fn mark_ebb_header_block_sealed(&mut self, block: Block) {
+    fn mark_block_header_block_sealed(&mut self, ssa_block: SSABlock) {
         // Then we mark the block as sealed.
-        match self.blocks[block] {
-            BlockData::EbbBody { .. } => panic!("this should not happen"),
-            BlockData::EbbHeader(ref mut data) => {
+        match self.ssa_blocks[ssa_block] {
+            SSABlockData::BlockBody { .. } => panic!("this should not happen"),
+            SSABlockData::BlockHeader(ref mut data) => {
                 debug_assert!(!data.sealed);
                 debug_assert!(data.undef_variables.is_empty());
                 data.sealed = true;
@@ -508,21 +516,21 @@ impl SSABuilder {
         }
     }
 
-    /// Given the local SSA Value of a Variable in an Ebb, perform a recursive lookup on
+    /// Given the local SSA Value of a Variable in an Block, perform a recursive lookup on
     /// predecessors to determine if it is redundant with another Value earlier in the CFG.
     ///
     /// If such a Value exists and is redundant, the local Value is replaced by the
-    /// corresponding non-local Value. If the original Value was an Ebb parameter,
+    /// corresponding non-local Value. If the original Value was an Block parameter,
     /// the parameter may be removed if redundant. Parameters are placed eagerly by callers
-    /// to avoid infinite loops when looking up a Value for an Ebb that is in a CFG loop.
+    /// to avoid infinite loops when looking up a Value for an Block that is in a CFG loop.
     ///
-    /// Doing this lookup for each Value in each Ebb preserves SSA form during construction.
+    /// Doing this lookup for each Value in each Block preserves SSA form during construction.
     ///
     /// Returns the chosen Value.
     ///
     /// ## Arguments
     ///
-    /// `sentinel` is a dummy Ebb parameter inserted by `use_var_nonlocal()`.
+    /// `sentinel` is a dummy Block parameter inserted by `use_var_nonlocal()`.
     /// Its purpose is to allow detection of CFG cycles while traversing predecessors.
     ///
     /// The `sentinel: Value` and the `ty: Type` are describing the `var: Variable`
@@ -533,30 +541,29 @@ impl SSABuilder {
         sentinel: Value,
         var: Variable,
         ty: Type,
-        ebb: Ebb,
+        block: Block,
     ) -> Value {
         debug_assert!(self.calls.is_empty());
         debug_assert!(self.results.is_empty());
         // self.side_effects may be non-empty here so that callers can
         // accumulate side effects over multiple calls.
-        self.begin_predecessors_lookup(sentinel, ebb);
+        self.begin_predecessors_lookup(sentinel, block);
         self.run_state_machine(func, var, ty)
     }
 
     /// Set up state for `run_state_machine()` to initiate non-local use lookups
-    /// in all predecessors of `dest_ebb`, and arrange for a call to
+    /// in all predecessors of `dest_block`, and arrange for a call to
     /// `finish_predecessors_lookup` once they complete.
-    fn begin_predecessors_lookup(&mut self, sentinel: Value, dest_ebb: Ebb) {
+    fn begin_predecessors_lookup(&mut self, sentinel: Value, dest_block: Block) {
         self.calls
-            .push(Call::FinishPredecessorsLookup(sentinel, dest_ebb));
+            .push(Call::FinishPredecessorsLookup(sentinel, dest_block));
         // Iterate over the predecessors.
         let mut calls = mem::replace(&mut self.calls, Vec::new());
-        calls.extend(
-            self.predecessors(dest_ebb)
-                .iter()
-                .rev()
-                .map(|&PredBlock { block: pred, .. }| Call::UseVar(pred)),
-        );
+        calls.extend(self.predecessors(dest_block).iter().rev().map(
+            |&PredBlock {
+                 ssa_block: pred, ..
+             }| Call::UseVar(pred),
+        ));
         self.calls = calls;
     }
 
@@ -567,12 +574,12 @@ impl SSABuilder {
         func: &mut Function,
         sentinel: Value,
         var: Variable,
-        dest_ebb: Ebb,
+        dest_block: Block,
     ) {
         let mut pred_values: ZeroOneOrMore = ZeroOneOrMore::Zero;
 
         // Determine how many predecessors are yielding unique, non-temporary Values.
-        let num_predecessors = self.predecessors(dest_ebb).len();
+        let num_predecessors = self.predecessors(dest_block).len();
         for &pred_val in self.results.iter().rev().take(num_predecessors) {
             match pred_values {
                 ZeroOneOrMore::Zero => {
@@ -600,21 +607,23 @@ impl SSABuilder {
                 // The variable is used but never defined before. This is an irregularity in the
                 // code, but rather than throwing an error we silently initialize the variable to
                 // 0. This will have no effect since this situation happens in unreachable code.
-                if !func.layout.is_ebb_inserted(dest_ebb) {
-                    func.layout.append_ebb(dest_ebb);
+                if !func.layout.is_block_inserted(dest_block) {
+                    func.layout.append_block(dest_block);
                 }
-                self.side_effects.instructions_added_to_ebbs.push(dest_ebb);
+                self.side_effects
+                    .instructions_added_to_blocks
+                    .push(dest_block);
                 let zero = emit_zero(
                     func.dfg.value_type(sentinel),
-                    FuncCursor::new(func).at_first_insertion_point(dest_ebb),
+                    FuncCursor::new(func).at_first_insertion_point(dest_block),
                 );
-                func.dfg.remove_ebb_param(sentinel);
+                func.dfg.remove_block_param(sentinel);
                 func.dfg.change_to_alias(sentinel, zero);
                 zero
             }
             ZeroOneOrMore::One(pred_val) => {
                 // Here all the predecessors use a single value to represent our variable
-                // so we don't need to have it as an ebb argument.
+                // so we don't need to have it as an block argument.
                 // We need to replace all the occurrences of val with pred_val but since
                 // we can't afford a re-writing pass right now we just declare an alias.
                 // Resolve aliases eagerly so that we can check for cyclic aliasing,
@@ -624,44 +633,44 @@ impl SSABuilder {
                     // Cycle detected. Break it by creating a zero value.
                     resolved = emit_zero(
                         func.dfg.value_type(sentinel),
-                        FuncCursor::new(func).at_first_insertion_point(dest_ebb),
+                        FuncCursor::new(func).at_first_insertion_point(dest_block),
                     );
                 }
-                func.dfg.remove_ebb_param(sentinel);
+                func.dfg.remove_block_param(sentinel);
                 func.dfg.change_to_alias(sentinel, resolved);
                 resolved
             }
             ZeroOneOrMore::More => {
                 // There is disagreement in the predecessors on which value to use so we have
-                // to keep the ebb argument. To avoid borrowing `self` for the whole loop,
+                // to keep the block argument. To avoid borrowing `self` for the whole loop,
                 // temporarily detach the predecessors list and replace it with an empty list.
                 let mut preds =
-                    mem::replace(self.predecessors_mut(dest_ebb), PredBlockSmallVec::new());
+                    mem::replace(self.predecessors_mut(dest_block), PredBlockSmallVec::new());
                 for &mut PredBlock {
-                    block: ref mut pred_block,
+                    ssa_block: ref mut pred_ssa_block,
                     branch: ref mut last_inst,
                 } in &mut preds
                 {
                     // We already did a full `use_var` above, so we can do just the fast path.
-                    let block_map = self.variables.get(var).unwrap();
-                    let pred_val = block_map.get(*pred_block).unwrap().unwrap();
+                    let ssa_block_map = self.variables.get(var).unwrap();
+                    let pred_val = ssa_block_map.get(*pred_ssa_block).unwrap().unwrap();
                     let jump_arg = self.append_jump_argument(
                         func,
                         *last_inst,
-                        *pred_block,
-                        dest_ebb,
+                        *pred_ssa_block,
+                        dest_block,
                         pred_val,
                         var,
                     );
-                    if let Some((middle_ebb, middle_block, middle_jump_inst)) = jump_arg {
-                        *pred_block = middle_block;
+                    if let Some((middle_block, middle_ssa_block, middle_jump_inst)) = jump_arg {
+                        *pred_ssa_block = middle_ssa_block;
                         *last_inst = middle_jump_inst;
-                        self.side_effects.split_ebbs_created.push(middle_ebb);
+                        self.side_effects.split_blocks_created.push(middle_block);
                     }
                 }
                 // Now that we're done, move the predecessors list back.
-                debug_assert!(self.predecessors(dest_ebb).is_empty());
-                *self.predecessors_mut(dest_ebb) = preds;
+                debug_assert!(self.predecessors(dest_block).is_empty());
+                *self.predecessors_mut(dest_block) = preds;
 
                 sentinel
             }
@@ -670,20 +679,20 @@ impl SSABuilder {
         self.results.push(result_val);
     }
 
-    /// Appends a jump argument to a jump instruction, returns ebb created in case of
+    /// Appends a jump argument to a jump instruction, returns block created in case of
     /// critical edge splitting.
     fn append_jump_argument(
         &mut self,
         func: &mut Function,
         jump_inst: Inst,
-        jump_inst_block: Block,
-        dest_ebb: Ebb,
+        jump_inst_ssa_block: SSABlock,
+        dest_block: Block,
         val: Value,
         var: Variable,
-    ) -> Option<(Ebb, Block, Inst)> {
+    ) -> Option<(Block, SSABlock, Inst)> {
         match func.dfg.analyze_branch(jump_inst) {
             BranchInfo::NotABranch => {
-                panic!("you have declared a non-branch instruction as a predecessor to an ebb");
+                panic!("you have declared a non-branch instruction as a predecessor to an block");
             }
             // For a single destination appending a jump argument to the instruction
             // is sufficient.
@@ -691,24 +700,24 @@ impl SSABuilder {
                 func.dfg.append_inst_arg(jump_inst, val);
                 None
             }
-            BranchInfo::Table(jt, default_ebb) => {
+            BranchInfo::Table(jt, default_block) => {
                 // In the case of a jump table, the situation is tricky because br_table doesn't
                 // support arguments.
                 // We have to split the critical edge
-                let middle_ebb = func.dfg.make_ebb();
-                func.layout.append_ebb(middle_ebb);
-                let middle_block = self.declare_ebb_header_block(middle_ebb);
-                self.blocks[middle_block].add_predecessor(jump_inst_block, jump_inst);
-                self.mark_ebb_header_block_sealed(middle_block);
+                let middle_block = func.dfg.make_block();
+                func.layout.append_block(middle_block);
+                let middle_ssa_block = self.declare_block_header_block(middle_block);
+                self.ssa_blocks[middle_ssa_block].add_predecessor(jump_inst_ssa_block, jump_inst);
+                self.mark_block_header_block_sealed(middle_ssa_block);
 
-                if let Some(default_ebb) = default_ebb {
-                    if dest_ebb == default_ebb {
+                if let Some(default_block) = default_block {
+                    if dest_block == default_block {
                         match func.dfg[jump_inst] {
                             InstructionData::BranchTable {
                                 destination: ref mut dest,
                                 ..
                             } => {
-                                *dest = middle_ebb;
+                                *dest = middle_block;
                             }
                             _ => panic!("should not happen"),
                         }
@@ -716,46 +725,46 @@ impl SSABuilder {
                 }
 
                 for old_dest in func.jump_tables[jt].as_mut_slice() {
-                    if *old_dest == dest_ebb {
-                        *old_dest = middle_ebb;
+                    if *old_dest == dest_block {
+                        *old_dest = middle_block;
                     }
                 }
-                let mut cur = FuncCursor::new(func).at_bottom(middle_ebb);
-                let middle_jump_inst = cur.ins().jump(dest_ebb, &[val]);
-                self.def_var(var, val, middle_block);
-                Some((middle_ebb, middle_block, middle_jump_inst))
+                let mut cur = FuncCursor::new(func).at_bottom(middle_block);
+                let middle_jump_inst = cur.ins().jump(dest_block, &[val]);
+                self.def_var(var, val, middle_ssa_block);
+                Some((middle_block, middle_ssa_block, middle_jump_inst))
             }
         }
     }
 
-    /// Returns the list of `Ebb`s that have been declared as predecessors of the argument.
-    fn predecessors(&self, ebb: Ebb) -> &[PredBlock] {
-        let block = self.header_block(ebb);
-        match self.blocks[block] {
-            BlockData::EbbBody { .. } => panic!("should not happen"),
-            BlockData::EbbHeader(ref data) => &data.predecessors,
+    /// Returns the list of `Block`s that have been declared as predecessors of the argument.
+    fn predecessors(&self, block: Block) -> &[PredBlock] {
+        let ssa_block = self.header_block(block);
+        match self.ssa_blocks[ssa_block] {
+            SSABlockData::BlockBody { .. } => panic!("should not happen"),
+            SSABlockData::BlockHeader(ref data) => &data.predecessors,
         }
     }
 
-    /// Returns whether the given Ebb has any predecessor or not.
-    pub fn has_any_predecessors(&self, ebb: Ebb) -> bool {
-        !self.predecessors(ebb).is_empty()
+    /// Returns whether the given Block has any predecessor or not.
+    pub fn has_any_predecessors(&self, block: Block) -> bool {
+        !self.predecessors(block).is_empty()
     }
 
     /// Same as predecessors, but for &mut.
-    fn predecessors_mut(&mut self, ebb: Ebb) -> &mut PredBlockSmallVec {
-        let block = self.header_block(ebb);
-        match self.blocks[block] {
-            BlockData::EbbBody { .. } => panic!("should not happen"),
-            BlockData::EbbHeader(ref mut data) => &mut data.predecessors,
+    fn predecessors_mut(&mut self, block: Block) -> &mut PredBlockSmallVec {
+        let ssa_block = self.header_block(block);
+        match self.ssa_blocks[ssa_block] {
+            SSABlockData::BlockBody { .. } => panic!("should not happen"),
+            SSABlockData::BlockHeader(ref mut data) => &mut data.predecessors,
         }
     }
 
-    /// Returns `true` if and only if `seal_ebb_header_block` has been called on the argument.
-    pub fn is_sealed(&self, ebb: Ebb) -> bool {
-        match self.blocks[self.header_block(ebb)] {
-            BlockData::EbbBody { .. } => panic!("should not happen"),
-            BlockData::EbbHeader(ref data) => data.sealed,
+    /// Returns `true` if and only if `seal_block_header_block` has been called on the argument.
+    pub fn is_sealed(&self, block: Block) -> bool {
+        match self.ssa_blocks[self.header_block(block)] {
+            SSABlockData::BlockBody { .. } => panic!("should not happen"),
+            SSABlockData::BlockHeader(ref data) => data.sealed,
         }
     }
 
@@ -768,21 +777,21 @@ impl SSABuilder {
         // Process the calls scheduled in `self.calls` until it is empty.
         while let Some(call) = self.calls.pop() {
             match call {
-                Call::UseVar(block) => {
+                Call::UseVar(ssa_block) => {
                     // First we lookup for the current definition of the variable in this block
                     if let Some(var_defs) = self.variables.get(var) {
-                        if let Some(val) = var_defs[block].expand() {
+                        if let Some(val) = var_defs[ssa_block].expand() {
                             self.results.push(val);
                             continue;
                         }
                     }
-                    self.use_var_nonlocal(func, var, ty, block);
+                    self.use_var_nonlocal(func, var, ty, ssa_block);
                 }
-                Call::FinishSealedOnePredecessor(block) => {
-                    self.finish_sealed_one_predecessor(var, block);
+                Call::FinishSealedOnePredecessor(ssa_block) => {
+                    self.finish_sealed_one_predecessor(var, ssa_block);
                 }
-                Call::FinishPredecessorsLookup(sentinel, dest_ebb) => {
-                    self.finish_predecessors_lookup(func, sentinel, var, dest_ebb);
+                Call::FinishPredecessorsLookup(sentinel, dest_block) => {
+                    self.finish_predecessors_lookup(func, sentinel, var, dest_block);
                 }
             }
         }
@@ -807,124 +816,124 @@ mod tests {
     fn simple_block() {
         let mut func = Function::new();
         let mut ssa = SSABuilder::new();
-        let ebb0 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
         // Here is the pseudo-program we want to translate:
         // x = 1;
         // y = 2;
         // z = x + y;
         // z = x + z;
 
-        let block = ssa.declare_ebb_header_block(ebb0);
+        let ssa_block = ssa.declare_block_header_block(block0);
         let x_var = Variable::new(0);
         let x_ssa = {
             let mut cur = FuncCursor::new(&mut func);
-            cur.insert_ebb(ebb0);
+            cur.insert_block(block0);
             cur.ins().iconst(I32, 1)
         };
-        ssa.def_var(x_var, x_ssa, block);
+        ssa.def_var(x_var, x_ssa, ssa_block);
         let y_var = Variable::new(1);
         let y_ssa = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
             cur.ins().iconst(I32, 2)
         };
-        ssa.def_var(y_var, y_ssa, block);
+        ssa.def_var(y_var, y_ssa, ssa_block);
 
-        assert_eq!(ssa.use_var(&mut func, x_var, I32, block).0, x_ssa);
-        assert_eq!(ssa.use_var(&mut func, y_var, I32, block).0, y_ssa);
+        assert_eq!(ssa.use_var(&mut func, x_var, I32, ssa_block).0, x_ssa);
+        assert_eq!(ssa.use_var(&mut func, y_var, I32, ssa_block).0, y_ssa);
         let z_var = Variable::new(2);
-        let x_use1 = ssa.use_var(&mut func, x_var, I32, block).0;
-        let y_use1 = ssa.use_var(&mut func, y_var, I32, block).0;
+        let x_use1 = ssa.use_var(&mut func, x_var, I32, ssa_block).0;
+        let y_use1 = ssa.use_var(&mut func, y_var, I32, ssa_block).0;
         let z1_ssa = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
             cur.ins().iadd(x_use1, y_use1)
         };
-        ssa.def_var(z_var, z1_ssa, block);
-        assert_eq!(ssa.use_var(&mut func, z_var, I32, block).0, z1_ssa);
-        let x_use2 = ssa.use_var(&mut func, x_var, I32, block).0;
-        let z_use1 = ssa.use_var(&mut func, z_var, I32, block).0;
+        ssa.def_var(z_var, z1_ssa, ssa_block);
+        assert_eq!(ssa.use_var(&mut func, z_var, I32, ssa_block).0, z1_ssa);
+        let x_use2 = ssa.use_var(&mut func, x_var, I32, ssa_block).0;
+        let z_use1 = ssa.use_var(&mut func, z_var, I32, ssa_block).0;
         let z2_ssa = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
             cur.ins().iadd(x_use2, z_use1)
         };
-        ssa.def_var(z_var, z2_ssa, block);
-        assert_eq!(ssa.use_var(&mut func, z_var, I32, block).0, z2_ssa);
+        ssa.def_var(z_var, z2_ssa, ssa_block);
+        assert_eq!(ssa.use_var(&mut func, z_var, I32, ssa_block).0, z2_ssa);
     }
 
     #[test]
     fn sequence_of_blocks() {
         let mut func = Function::new();
         let mut ssa = SSABuilder::new();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb1 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
+        let block1 = func.dfg.make_block();
         // Here is the pseudo-program we want to translate:
-        // ebb0:
+        // block0:
         //    x = 1;
         //    y = 2;
         //    z = x + y;
-        //    brnz y, ebb1;
+        //    brnz y, block1;
         //    z = x + z;
-        // ebb1:
+        // block1:
         //    y = x + y;
 
-        let block0 = ssa.declare_ebb_header_block(ebb0);
+        let ssa_block0 = ssa.declare_block_header_block(block0);
         let x_var = Variable::new(0);
         let x_ssa = {
             let mut cur = FuncCursor::new(&mut func);
-            cur.insert_ebb(ebb0);
-            cur.insert_ebb(ebb1);
-            cur.goto_bottom(ebb0);
+            cur.insert_block(block0);
+            cur.insert_block(block1);
+            cur.goto_bottom(block0);
             cur.ins().iconst(I32, 1)
         };
-        ssa.def_var(x_var, x_ssa, block0);
+        ssa.def_var(x_var, x_ssa, ssa_block0);
         let y_var = Variable::new(1);
         let y_ssa = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
             cur.ins().iconst(I32, 2)
         };
-        ssa.def_var(y_var, y_ssa, block0);
-        assert_eq!(ssa.use_var(&mut func, x_var, I32, block0).0, x_ssa);
-        assert_eq!(ssa.use_var(&mut func, y_var, I32, block0).0, y_ssa);
+        ssa.def_var(y_var, y_ssa, ssa_block0);
+        assert_eq!(ssa.use_var(&mut func, x_var, I32, ssa_block0).0, x_ssa);
+        assert_eq!(ssa.use_var(&mut func, y_var, I32, ssa_block0).0, y_ssa);
         let z_var = Variable::new(2);
-        let x_use1 = ssa.use_var(&mut func, x_var, I32, block0).0;
-        let y_use1 = ssa.use_var(&mut func, y_var, I32, block0).0;
+        let x_use1 = ssa.use_var(&mut func, x_var, I32, ssa_block0).0;
+        let y_use1 = ssa.use_var(&mut func, y_var, I32, ssa_block0).0;
         let z1_ssa = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
             cur.ins().iadd(x_use1, y_use1)
         };
-        ssa.def_var(z_var, z1_ssa, block0);
-        assert_eq!(ssa.use_var(&mut func, z_var, I32, block0).0, z1_ssa);
-        let y_use2 = ssa.use_var(&mut func, y_var, I32, block0).0;
+        ssa.def_var(z_var, z1_ssa, ssa_block0);
+        assert_eq!(ssa.use_var(&mut func, z_var, I32, ssa_block0).0, z1_ssa);
+        let y_use2 = ssa.use_var(&mut func, y_var, I32, ssa_block0).0;
         let jump_inst: Inst = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
-            cur.ins().brnz(y_use2, ebb1, &[])
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
+            cur.ins().brnz(y_use2, block1, &[])
         };
-        let block1 = ssa.declare_ebb_body_block(block0);
-        let x_use2 = ssa.use_var(&mut func, x_var, I32, block1).0;
+        let ssa_block1 = ssa.declare_block_body_block(ssa_block0);
+        let x_use2 = ssa.use_var(&mut func, x_var, I32, ssa_block1).0;
         assert_eq!(x_use2, x_ssa);
-        let z_use1 = ssa.use_var(&mut func, z_var, I32, block1).0;
+        let z_use1 = ssa.use_var(&mut func, z_var, I32, ssa_block1).0;
         assert_eq!(z_use1, z1_ssa);
         let z2_ssa = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
             cur.ins().iadd(x_use2, z_use1)
         };
-        ssa.def_var(z_var, z2_ssa, block1);
-        assert_eq!(ssa.use_var(&mut func, z_var, I32, block1).0, z2_ssa);
-        ssa.seal_ebb_header_block(ebb0, &mut func);
-        let block2 = ssa.declare_ebb_header_block(ebb1);
-        ssa.declare_ebb_predecessor(ebb1, block0, jump_inst);
-        ssa.seal_ebb_header_block(ebb1, &mut func);
-        let x_use3 = ssa.use_var(&mut func, x_var, I32, block2).0;
+        ssa.def_var(z_var, z2_ssa, ssa_block1);
+        assert_eq!(ssa.use_var(&mut func, z_var, I32, ssa_block1).0, z2_ssa);
+        ssa.seal_block_header_block(block0, &mut func);
+        let ssa_block2 = ssa.declare_block_header_block(block1);
+        ssa.declare_block_predecessor(block1, ssa_block0, jump_inst);
+        ssa.seal_block_header_block(block1, &mut func);
+        let x_use3 = ssa.use_var(&mut func, x_var, I32, ssa_block2).0;
         assert_eq!(x_ssa, x_use3);
-        let y_use3 = ssa.use_var(&mut func, y_var, I32, block2).0;
+        let y_use3 = ssa.use_var(&mut func, y_var, I32, ssa_block2).0;
         assert_eq!(y_ssa, y_use3);
         let y2_ssa = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
             cur.ins().iadd(x_use3, y_use3)
         };
-        ssa.def_var(y_var, y2_ssa, block2);
+        ssa.def_var(y_var, y2_ssa, ssa_block2);
         match func.dfg.analyze_branch(jump_inst) {
             BranchInfo::SingleDest(dest, jump_args) => {
-                assert_eq!(dest, ebb1);
+                assert_eq!(dest, block1);
                 assert_eq!(jump_args.len(), 0);
             }
             _ => assert!(false),
@@ -935,110 +944,110 @@ mod tests {
     fn program_with_loop() {
         let mut func = Function::new();
         let mut ssa = SSABuilder::new();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb1 = func.dfg.make_ebb();
-        let ebb2 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
+        let block1 = func.dfg.make_block();
+        let block2 = func.dfg.make_block();
         // Here is the pseudo-program we want to translate:
-        // ebb0:
+        // block0:
         //    x = 1;
         //    y = 2;
         //    z = x + y;
-        //    jump ebb1
-        // ebb1:
+        //    jump block1
+        // block1:
         //    z = z + y;
-        //    brnz y, ebb1;
+        //    brnz y, block1;
         //    z = z - x;
         //    return y
-        // ebb2:
+        // block2:
         //    y = y - x
-        //    jump ebb1
+        //    jump block1
 
-        let block0 = ssa.declare_ebb_header_block(ebb0);
-        ssa.seal_ebb_header_block(ebb0, &mut func);
+        let ssa_block0 = ssa.declare_block_header_block(block0);
+        ssa.seal_block_header_block(block0, &mut func);
         let x_var = Variable::new(0);
         let x1 = {
             let mut cur = FuncCursor::new(&mut func);
-            cur.insert_ebb(ebb0);
-            cur.insert_ebb(ebb1);
-            cur.insert_ebb(ebb2);
-            cur.goto_bottom(ebb0);
+            cur.insert_block(block0);
+            cur.insert_block(block1);
+            cur.insert_block(block2);
+            cur.goto_bottom(block0);
             cur.ins().iconst(I32, 1)
         };
-        ssa.def_var(x_var, x1, block0);
-        assert_eq!(ssa.use_var(&mut func, x_var, I32, block0).0, x1);
+        ssa.def_var(x_var, x1, ssa_block0);
+        assert_eq!(ssa.use_var(&mut func, x_var, I32, ssa_block0).0, x1);
         let y_var = Variable::new(1);
         let y1 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
             cur.ins().iconst(I32, 2)
         };
-        ssa.def_var(y_var, y1, block0);
-        assert_eq!(ssa.use_var(&mut func, y_var, I32, block0).0, y1);
+        ssa.def_var(y_var, y1, ssa_block0);
+        assert_eq!(ssa.use_var(&mut func, y_var, I32, ssa_block0).0, y1);
         let z_var = Variable::new(2);
-        let x2 = ssa.use_var(&mut func, x_var, I32, block0).0;
+        let x2 = ssa.use_var(&mut func, x_var, I32, ssa_block0).0;
         assert_eq!(x2, x1);
-        let y2 = ssa.use_var(&mut func, y_var, I32, block0).0;
+        let y2 = ssa.use_var(&mut func, y_var, I32, ssa_block0).0;
         assert_eq!(y2, y1);
         let z1 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
             cur.ins().iadd(x2, y2)
         };
-        ssa.def_var(z_var, z1, block0);
-        let jump_ebb0_ebb1 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
-            cur.ins().jump(ebb1, &[])
+        ssa.def_var(z_var, z1, ssa_block0);
+        let jump_block0_block1 = {
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
+            cur.ins().jump(block1, &[])
         };
-        let block1 = ssa.declare_ebb_header_block(ebb1);
-        ssa.declare_ebb_predecessor(ebb1, block0, jump_ebb0_ebb1);
-        let z2 = ssa.use_var(&mut func, z_var, I32, block1).0;
-        let y3 = ssa.use_var(&mut func, y_var, I32, block1).0;
+        let ssa_block1 = ssa.declare_block_header_block(block1);
+        ssa.declare_block_predecessor(block1, ssa_block0, jump_block0_block1);
+        let z2 = ssa.use_var(&mut func, z_var, I32, ssa_block1).0;
+        let y3 = ssa.use_var(&mut func, y_var, I32, ssa_block1).0;
         let z3 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block1);
             cur.ins().iadd(z2, y3)
         };
-        ssa.def_var(z_var, z3, block1);
-        let y4 = ssa.use_var(&mut func, y_var, I32, block1).0;
+        ssa.def_var(z_var, z3, ssa_block1);
+        let y4 = ssa.use_var(&mut func, y_var, I32, ssa_block1).0;
         assert_eq!(y4, y3);
-        let jump_ebb1_ebb2 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1);
-            cur.ins().brnz(y4, ebb2, &[])
+        let jump_block1_block2 = {
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block1);
+            cur.ins().brnz(y4, block2, &[])
         };
-        let block2 = ssa.declare_ebb_body_block(block1);
-        let z4 = ssa.use_var(&mut func, z_var, I32, block2).0;
+        let ssa_block2 = ssa.declare_block_body_block(ssa_block1);
+        let z4 = ssa.use_var(&mut func, z_var, I32, ssa_block2).0;
         assert_eq!(z4, z3);
-        let x3 = ssa.use_var(&mut func, x_var, I32, block2).0;
+        let x3 = ssa.use_var(&mut func, x_var, I32, ssa_block2).0;
         let z5 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block1);
             cur.ins().isub(z4, x3)
         };
-        ssa.def_var(z_var, z5, block2);
-        let y5 = ssa.use_var(&mut func, y_var, I32, block2).0;
+        ssa.def_var(z_var, z5, ssa_block2);
+        let y5 = ssa.use_var(&mut func, y_var, I32, ssa_block2).0;
         assert_eq!(y5, y3);
         {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block1);
             cur.ins().return_(&[y5])
         };
 
-        let block3 = ssa.declare_ebb_header_block(ebb2);
-        ssa.declare_ebb_predecessor(ebb2, block1, jump_ebb1_ebb2);
-        ssa.seal_ebb_header_block(ebb2, &mut func);
-        let y6 = ssa.use_var(&mut func, y_var, I32, block3).0;
+        let ssa_block3 = ssa.declare_block_header_block(block2);
+        ssa.declare_block_predecessor(block2, ssa_block1, jump_block1_block2);
+        ssa.seal_block_header_block(block2, &mut func);
+        let y6 = ssa.use_var(&mut func, y_var, I32, ssa_block3).0;
         assert_eq!(y6, y3);
-        let x4 = ssa.use_var(&mut func, x_var, I32, block3).0;
+        let x4 = ssa.use_var(&mut func, x_var, I32, ssa_block3).0;
         assert_eq!(x4, x3);
         let y7 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb2);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block2);
             cur.ins().isub(y6, x4)
         };
-        ssa.def_var(y_var, y7, block3);
-        let jump_ebb2_ebb1 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb2);
-            cur.ins().jump(ebb1, &[])
+        ssa.def_var(y_var, y7, ssa_block3);
+        let jump_block2_block1 = {
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block2);
+            cur.ins().jump(block1, &[])
         };
 
-        ssa.declare_ebb_predecessor(ebb1, block3, jump_ebb2_ebb1);
-        ssa.seal_ebb_header_block(ebb1, &mut func);
-        assert_eq!(func.dfg.ebb_params(ebb1)[0], z2);
-        assert_eq!(func.dfg.ebb_params(ebb1)[1], y3);
+        ssa.declare_block_predecessor(block1, ssa_block3, jump_block2_block1);
+        ssa.seal_block_header_block(block1, &mut func);
+        assert_eq!(func.dfg.block_params(block1)[0], z2);
+        assert_eq!(func.dfg.block_params(block1)[1], y3);
         assert_eq!(func.dfg.resolve_aliases(x3), x1);
     }
 
@@ -1049,14 +1058,14 @@ mod tests {
         // Here is the pseudo-program we want to translate:
         //
         // function %f {
-        // jt = jump_table [ebb2, ebb1]
-        // ebb0:
+        // jt = jump_table [block2, block1]
+        // block0:
         //    x = 1;
-        //    br_table x, ebb2, jt
-        // ebb1:
+        //    br_table x, block2, jt
+        // block1:
         //    x = 2
-        //    jump ebb2
-        // ebb2:
+        //    jump block2
+        // block2:
         //    x = x + 1
         //    return
         // }
@@ -1064,70 +1073,70 @@ mod tests {
         let mut func = Function::new();
         let mut ssa = SSABuilder::new();
         let mut jump_table = JumpTableData::new();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb1 = func.dfg.make_ebb();
-        let ebb2 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
+        let block1 = func.dfg.make_block();
+        let block2 = func.dfg.make_block();
 
-        // ebb0:
+        // block0:
         //    x = 1;
-        let block0 = ssa.declare_ebb_header_block(ebb0);
-        ssa.seal_ebb_header_block(ebb0, &mut func);
+        let ssa_block0 = ssa.declare_block_header_block(block0);
+        ssa.seal_block_header_block(block0, &mut func);
         let x_var = Variable::new(0);
         let x1 = {
             let mut cur = FuncCursor::new(&mut func);
-            cur.insert_ebb(ebb0);
-            cur.insert_ebb(ebb1);
-            cur.insert_ebb(ebb2);
-            cur.goto_bottom(ebb0);
+            cur.insert_block(block0);
+            cur.insert_block(block1);
+            cur.insert_block(block2);
+            cur.goto_bottom(block0);
             cur.ins().iconst(I32, 1)
         };
-        ssa.def_var(x_var, x1, block0);
+        ssa.def_var(x_var, x1, ssa_block0);
 
-        // jt = jump_table [ebb2, ebb1]
-        jump_table.push_entry(ebb2);
-        jump_table.push_entry(ebb1);
+        // jt = jump_table [block2, block1]
+        jump_table.push_entry(block2);
+        jump_table.push_entry(block1);
         let jt = func.create_jump_table(jump_table);
 
-        // ebb0:
+        // block0:
         //    ...
-        //    br_table x, ebb2, jt
-        ssa.use_var(&mut func, x_var, I32, block0).0;
+        //    br_table x, block2, jt
+        ssa.use_var(&mut func, x_var, I32, ssa_block0).0;
         let br_table = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
-            cur.ins().br_table(x1, ebb2, jt)
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
+            cur.ins().br_table(x1, block2, jt)
         };
 
-        // ebb1:
+        // block1:
         //    x = 2
-        //    jump ebb2
-        let block1 = ssa.declare_ebb_header_block(ebb1);
-        ssa.seal_ebb_header_block(ebb1, &mut func);
+        //    jump block2
+        let ssa_block1 = ssa.declare_block_header_block(block1);
+        ssa.seal_block_header_block(block1, &mut func);
         let x2 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block1);
             cur.ins().iconst(I32, 2)
         };
-        ssa.def_var(x_var, x2, block1);
+        ssa.def_var(x_var, x2, ssa_block1);
         let jump_inst = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1);
-            cur.ins().jump(ebb2, &[])
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block1);
+            cur.ins().jump(block2, &[])
         };
 
-        // ebb2:
+        // block2:
         //    x = x + 1
         //    return
-        let block3 = ssa.declare_ebb_header_block(ebb2);
-        ssa.declare_ebb_predecessor(ebb2, block1, jump_inst);
-        ssa.declare_ebb_predecessor(ebb2, block0, br_table);
-        ssa.seal_ebb_header_block(ebb2, &mut func);
-        let block4 = ssa.declare_ebb_body_block(block3);
-        let x3 = ssa.use_var(&mut func, x_var, I32, block4).0;
+        let ssa_block3 = ssa.declare_block_header_block(block2);
+        ssa.declare_block_predecessor(block2, ssa_block1, jump_inst);
+        ssa.declare_block_predecessor(block2, ssa_block0, br_table);
+        ssa.seal_block_header_block(block2, &mut func);
+        let ssa_block4 = ssa.declare_block_body_block(ssa_block3);
+        let x3 = ssa.use_var(&mut func, x_var, I32, ssa_block4).0;
         let x4 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb2);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block2);
             cur.ins().iadd_imm(x3, 1)
         };
-        ssa.def_var(x_var, x4, block4);
+        ssa.def_var(x_var, x4, ssa_block4);
         {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb2);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block2);
             cur.ins().return_(&[])
         };
 
@@ -1147,75 +1156,75 @@ mod tests {
     fn undef_values_reordering() {
         let mut func = Function::new();
         let mut ssa = SSABuilder::new();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb1 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
+        let block1 = func.dfg.make_block();
         // Here is the pseudo-program we want to translate:
-        // ebb0:
+        // block0:
         //    x = 0
         //    y = 1
         //    z = 2
-        //    jump ebb1
-        // ebb1:
+        //    jump block1
+        // block1:
         //    x = z + x
         //    y = y - x
-        //    jump ebb1
+        //    jump block1
         //
-        let block0 = ssa.declare_ebb_header_block(ebb0);
+        let ssa_block0 = ssa.declare_block_header_block(block0);
         let x_var = Variable::new(0);
         let y_var = Variable::new(1);
         let z_var = Variable::new(2);
-        ssa.seal_ebb_header_block(ebb0, &mut func);
+        ssa.seal_block_header_block(block0, &mut func);
         let x1 = {
             let mut cur = FuncCursor::new(&mut func);
-            cur.insert_ebb(ebb0);
-            cur.insert_ebb(ebb1);
-            cur.goto_bottom(ebb0);
+            cur.insert_block(block0);
+            cur.insert_block(block1);
+            cur.goto_bottom(block0);
             cur.ins().iconst(I32, 0)
         };
-        ssa.def_var(x_var, x1, block0);
+        ssa.def_var(x_var, x1, ssa_block0);
         let y1 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
             cur.ins().iconst(I32, 1)
         };
-        ssa.def_var(y_var, y1, block0);
+        ssa.def_var(y_var, y1, ssa_block0);
         let z1 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
             cur.ins().iconst(I32, 2)
         };
-        ssa.def_var(z_var, z1, block0);
+        ssa.def_var(z_var, z1, ssa_block0);
         let jump_inst = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb0);
-            cur.ins().jump(ebb1, &[])
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block0);
+            cur.ins().jump(block1, &[])
         };
-        let block1 = ssa.declare_ebb_header_block(ebb1);
-        ssa.declare_ebb_predecessor(ebb1, block0, jump_inst);
-        let z2 = ssa.use_var(&mut func, z_var, I32, block1).0;
-        assert_eq!(func.dfg.ebb_params(ebb1)[0], z2);
-        let x2 = ssa.use_var(&mut func, x_var, I32, block1).0;
-        assert_eq!(func.dfg.ebb_params(ebb1)[1], x2);
+        let ssa_block1 = ssa.declare_block_header_block(block1);
+        ssa.declare_block_predecessor(block1, ssa_block0, jump_inst);
+        let z2 = ssa.use_var(&mut func, z_var, I32, ssa_block1).0;
+        assert_eq!(func.dfg.block_params(block1)[0], z2);
+        let x2 = ssa.use_var(&mut func, x_var, I32, ssa_block1).0;
+        assert_eq!(func.dfg.block_params(block1)[1], x2);
         let x3 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block1);
             cur.ins().iadd(x2, z2)
         };
-        ssa.def_var(x_var, x3, block1);
-        let x4 = ssa.use_var(&mut func, x_var, I32, block1).0;
-        let y3 = ssa.use_var(&mut func, y_var, I32, block1).0;
-        assert_eq!(func.dfg.ebb_params(ebb1)[2], y3);
+        ssa.def_var(x_var, x3, ssa_block1);
+        let x4 = ssa.use_var(&mut func, x_var, I32, ssa_block1).0;
+        let y3 = ssa.use_var(&mut func, y_var, I32, ssa_block1).0;
+        assert_eq!(func.dfg.block_params(block1)[2], y3);
         let y4 = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1);
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block1);
             cur.ins().isub(y3, x4)
         };
-        ssa.def_var(y_var, y4, block1);
+        ssa.def_var(y_var, y4, ssa_block1);
         let jump_inst = {
-            let mut cur = FuncCursor::new(&mut func).at_bottom(ebb1);
-            cur.ins().jump(ebb1, &[])
+            let mut cur = FuncCursor::new(&mut func).at_bottom(block1);
+            cur.ins().jump(block1, &[])
         };
-        ssa.declare_ebb_predecessor(ebb1, block1, jump_inst);
-        ssa.seal_ebb_header_block(ebb1, &mut func);
+        ssa.declare_block_predecessor(block1, ssa_block1, jump_inst);
+        ssa.seal_block_header_block(block1, &mut func);
         // At sealing the "z" argument disappear but the remaining "x" and "y" args have to be
         // in the right order.
-        assert_eq!(func.dfg.ebb_params(ebb1)[1], y3);
-        assert_eq!(func.dfg.ebb_params(ebb1)[0], x2);
+        assert_eq!(func.dfg.block_params(block1)[1], y3);
+        assert_eq!(func.dfg.block_params(block1)[0], x2);
     }
 
     #[test]
@@ -1223,20 +1232,20 @@ mod tests {
         // Use vars of various types which have not been defined.
         let mut func = Function::new();
         let mut ssa = SSABuilder::new();
-        let ebb0 = func.dfg.make_ebb();
-        let block = ssa.declare_ebb_header_block(ebb0);
-        ssa.seal_ebb_header_block(ebb0, &mut func);
+        let block0 = func.dfg.make_block();
+        let ssa_block = ssa.declare_block_header_block(block0);
+        ssa.seal_block_header_block(block0, &mut func);
         let i32_var = Variable::new(0);
         let f32_var = Variable::new(1);
         let f64_var = Variable::new(2);
         let b1_var = Variable::new(3);
         let f32x4_var = Variable::new(4);
-        ssa.use_var(&mut func, i32_var, I32, block);
-        ssa.use_var(&mut func, f32_var, F32, block);
-        ssa.use_var(&mut func, f64_var, F64, block);
-        ssa.use_var(&mut func, b1_var, B1, block);
-        ssa.use_var(&mut func, f32x4_var, F32X4, block);
-        assert_eq!(func.dfg.num_ebb_params(ebb0), 0);
+        ssa.use_var(&mut func, i32_var, I32, ssa_block);
+        ssa.use_var(&mut func, f32_var, F32, ssa_block);
+        ssa.use_var(&mut func, f64_var, F64, ssa_block);
+        ssa.use_var(&mut func, b1_var, B1, ssa_block);
+        ssa.use_var(&mut func, f32x4_var, F32X4, ssa_block);
+        assert_eq!(func.dfg.num_block_params(block0), 0);
     }
 
     #[test]
@@ -1245,15 +1254,15 @@ mod tests {
         // top of the entry block, and then fall back to inserting an iconst.
         let mut func = Function::new();
         let mut ssa = SSABuilder::new();
-        let ebb0 = func.dfg.make_ebb();
-        let block = ssa.declare_ebb_header_block(ebb0);
-        ssa.seal_ebb_header_block(ebb0, &mut func);
+        let block0 = func.dfg.make_block();
+        let ssa_block = ssa.declare_block_header_block(block0);
+        ssa.seal_block_header_block(block0, &mut func);
         let x_var = Variable::new(0);
-        assert_eq!(func.dfg.num_ebb_params(ebb0), 0);
-        ssa.use_var(&mut func, x_var, I32, block);
-        assert_eq!(func.dfg.num_ebb_params(ebb0), 0);
+        assert_eq!(func.dfg.num_block_params(block0), 0);
+        ssa.use_var(&mut func, x_var, I32, ssa_block);
+        assert_eq!(func.dfg.num_block_params(block0), 0);
         assert_eq!(
-            func.dfg[func.layout.first_inst(ebb0).unwrap()].opcode(),
+            func.dfg[func.layout.first_inst(block0).unwrap()].opcode(),
             Opcode::Iconst
         );
     }
@@ -1262,19 +1271,19 @@ mod tests {
     fn undef_in_entry_sealed_after() {
         // Use a var which has not been defined, but the block is not sealed
         // until afterward. Before sealing, the SSA builder should insert an
-        // ebb param; after sealing, it should be removed.
+        // block param; after sealing, it should be removed.
         let mut func = Function::new();
         let mut ssa = SSABuilder::new();
-        let ebb0 = func.dfg.make_ebb();
-        let block = ssa.declare_ebb_header_block(ebb0);
+        let block0 = func.dfg.make_block();
+        let ssa_block = ssa.declare_block_header_block(block0);
         let x_var = Variable::new(0);
-        assert_eq!(func.dfg.num_ebb_params(ebb0), 0);
-        ssa.use_var(&mut func, x_var, I32, block);
-        assert_eq!(func.dfg.num_ebb_params(ebb0), 1);
-        ssa.seal_ebb_header_block(ebb0, &mut func);
-        assert_eq!(func.dfg.num_ebb_params(ebb0), 0);
+        assert_eq!(func.dfg.num_block_params(block0), 0);
+        ssa.use_var(&mut func, x_var, I32, ssa_block);
+        assert_eq!(func.dfg.num_block_params(block0), 1);
+        ssa.seal_block_header_block(block0, &mut func);
+        assert_eq!(func.dfg.num_block_params(block0), 0);
         assert_eq!(
-            func.dfg[func.layout.first_inst(ebb0).unwrap()].opcode(),
+            func.dfg[func.layout.first_inst(block0).unwrap()].opcode(),
             Opcode::Iconst
         );
     }
@@ -1283,33 +1292,33 @@ mod tests {
     fn unreachable_use() {
         let mut func = Function::new();
         let mut ssa = SSABuilder::new();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb1 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
+        let block1 = func.dfg.make_block();
         // Here is the pseudo-program we want to translate:
-        // ebb0:
+        // block0:
         //    return
-        // ebb1:
-        //    brz v1, ebb1
-        //    jump ebb1
-        let _block0 = ssa.declare_ebb_header_block(ebb0);
-        ssa.seal_ebb_header_block(ebb0, &mut func);
-        let block1 = ssa.declare_ebb_header_block(ebb1);
-        let block2 = ssa.declare_ebb_body_block(block1);
+        // block1:
+        //    brz v1, block1
+        //    jump block1
+        let _ssa_block0 = ssa.declare_block_header_block(block0);
+        ssa.seal_block_header_block(block0, &mut func);
+        let ssa_block1 = ssa.declare_block_header_block(block1);
+        let ssa_block2 = ssa.declare_block_body_block(ssa_block1);
         {
             let mut cur = FuncCursor::new(&mut func);
-            cur.insert_ebb(ebb0);
-            cur.insert_ebb(ebb1);
-            cur.goto_bottom(ebb0);
+            cur.insert_block(block0);
+            cur.insert_block(block1);
+            cur.goto_bottom(block0);
             cur.ins().return_(&[]);
             let x_var = Variable::new(0);
-            cur.goto_bottom(ebb1);
-            let val = ssa.use_var(&mut cur.func, x_var, I32, block1).0;
-            let brz = cur.ins().brz(val, ebb1, &[]);
-            ssa.declare_ebb_predecessor(ebb1, block1, brz);
-            let j = cur.ins().jump(ebb1, &[]);
-            ssa.declare_ebb_predecessor(ebb1, block2, j);
+            cur.goto_bottom(block1);
+            let val = ssa.use_var(&mut cur.func, x_var, I32, ssa_block1).0;
+            let brz = cur.ins().brz(val, block1, &[]);
+            ssa.declare_block_predecessor(block1, ssa_block1, brz);
+            let j = cur.ins().jump(block1, &[]);
+            ssa.declare_block_predecessor(block1, ssa_block2, j);
         }
-        ssa.seal_ebb_header_block(ebb1, &mut func);
+        ssa.seal_block_header_block(block1, &mut func);
         let flags = settings::Flags::new(settings::builder());
         match verify_function(&func, &flags) {
             Ok(()) => {}
@@ -1326,41 +1335,41 @@ mod tests {
     fn unreachable_use_with_multiple_preds() {
         let mut func = Function::new();
         let mut ssa = SSABuilder::new();
-        let ebb0 = func.dfg.make_ebb();
-        let ebb1 = func.dfg.make_ebb();
-        let ebb2 = func.dfg.make_ebb();
+        let block0 = func.dfg.make_block();
+        let block1 = func.dfg.make_block();
+        let block2 = func.dfg.make_block();
         // Here is the pseudo-program we want to translate:
-        // ebb0:
+        // block0:
         //    return
-        // ebb1:
-        //    brz v1, ebb2
-        //    jump ebb1
-        // ebb2:
-        //    jump ebb1
-        let _block0 = ssa.declare_ebb_header_block(ebb0);
-        ssa.seal_ebb_header_block(ebb0, &mut func);
-        let block1 = ssa.declare_ebb_header_block(ebb1);
-        let block2 = ssa.declare_ebb_header_block(ebb2);
+        // block1:
+        //    brz v1, block2
+        //    jump block1
+        // block2:
+        //    jump block1
+        let _ssa_block0 = ssa.declare_block_header_block(block0);
+        ssa.seal_block_header_block(block0, &mut func);
+        let ssa_block1 = ssa.declare_block_header_block(block1);
+        let ssa_block2 = ssa.declare_block_header_block(block2);
         {
             let mut cur = FuncCursor::new(&mut func);
             let x_var = Variable::new(0);
-            cur.insert_ebb(ebb0);
-            cur.insert_ebb(ebb1);
-            cur.insert_ebb(ebb2);
-            cur.goto_bottom(ebb0);
+            cur.insert_block(block0);
+            cur.insert_block(block1);
+            cur.insert_block(block2);
+            cur.goto_bottom(block0);
             cur.ins().return_(&[]);
-            cur.goto_bottom(ebb1);
-            let v = ssa.use_var(&mut cur.func, x_var, I32, block1).0;
-            let brz = cur.ins().brz(v, ebb2, &[]);
-            let j0 = cur.ins().jump(ebb1, &[]);
-            cur.goto_bottom(ebb2);
-            let j1 = cur.ins().jump(ebb1, &[]);
-            ssa.declare_ebb_predecessor(ebb1, block2, brz);
-            ssa.declare_ebb_predecessor(ebb1, block1, j0);
-            ssa.declare_ebb_predecessor(ebb2, block1, j1);
+            cur.goto_bottom(block1);
+            let v = ssa.use_var(&mut cur.func, x_var, I32, ssa_block1).0;
+            let brz = cur.ins().brz(v, block2, &[]);
+            let j0 = cur.ins().jump(block1, &[]);
+            cur.goto_bottom(block2);
+            let j1 = cur.ins().jump(block1, &[]);
+            ssa.declare_block_predecessor(block1, ssa_block2, brz);
+            ssa.declare_block_predecessor(block1, ssa_block1, j0);
+            ssa.declare_block_predecessor(block2, ssa_block1, j1);
         }
-        ssa.seal_ebb_header_block(ebb1, &mut func);
-        ssa.seal_ebb_header_block(ebb2, &mut func);
+        ssa.seal_block_header_block(block1, &mut func);
+        ssa.seal_block_header_block(block2, &mut func);
         let flags = settings::Flags::new(settings::builder());
         match verify_function(&func, &flags) {
             Ok(()) => {}
diff --git a/cranelift/frontend/src/switch.rs b/cranelift/frontend/src/switch.rs
index e4c147d16b..f444d9aacc 100644
--- a/cranelift/frontend/src/switch.rs
+++ b/cranelift/frontend/src/switch.rs
@@ -23,13 +23,13 @@ type EntryIndex = u64;
 /// # let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
 /// # let mut builder = FunctionBuilder::new(&mut func, &mut fn_builder_ctx);
 /// #
-/// # let entry = builder.create_ebb();
+/// # let entry = builder.create_block();
 /// # builder.switch_to_block(entry);
 /// #
-/// let block0 = builder.create_ebb();
-/// let block1 = builder.create_ebb();
-/// let block2 = builder.create_ebb();
-/// let fallback = builder.create_ebb();
+/// let block0 = builder.create_block();
+/// let block1 = builder.create_block();
+/// let block2 = builder.create_block();
+/// let fallback = builder.create_block();
 ///
 /// let val = builder.ins().iconst(I32, 1);
 ///
@@ -41,7 +41,7 @@ type EntryIndex = u64;
 /// ```
 #[derive(Debug, Default)]
 pub struct Switch {
-    cases: HashMap,
+    cases: HashMap,
 }
 
 impl Switch {
@@ -53,8 +53,8 @@ impl Switch {
     }
 
     /// Set a switch entry
-    pub fn set_entry(&mut self, index: EntryIndex, ebb: Ebb) {
-        let prev = self.cases.insert(index, ebb);
+    pub fn set_entry(&mut self, index: EntryIndex, block: Block) {
+        let prev = self.cases.insert(index, block);
         assert!(
             prev.is_none(),
             "Tried to set the same entry {} twice",
@@ -63,7 +63,7 @@ impl Switch {
     }
 
     /// Get a reference to all existing entries
-    pub fn entries(&self) -> &HashMap {
+    pub fn entries(&self) -> &HashMap {
         &self.cases
     }
 
@@ -82,7 +82,7 @@ impl Switch {
 
         let mut contiguous_case_ranges: Vec = vec![];
         let mut last_index = None;
-        for (index, ebb) in cases {
+        for (index, block) in cases {
             match last_index {
                 None => contiguous_case_ranges.push(ContiguousCaseRange::new(index)),
                 Some(last_index) => {
@@ -91,7 +91,11 @@ impl Switch {
                     }
                 }
             }
-            contiguous_case_ranges.last_mut().unwrap().ebbs.push(ebb);
+            contiguous_case_ranges
+                .last_mut()
+                .unwrap()
+                .blocks
+                .push(block);
             last_index = Some(index);
         }
 
@@ -107,10 +111,10 @@ impl Switch {
     fn build_search_tree(
         bx: &mut FunctionBuilder,
         val: Value,
-        otherwise: Ebb,
+        otherwise: Block,
         contiguous_case_ranges: Vec,
-    ) -> Vec<(EntryIndex, Ebb, Vec)> {
-        let mut cases_and_jt_ebbs = Vec::new();
+    ) -> Vec<(EntryIndex, Block, Vec)> {
+        let mut cases_and_jt_blocks = Vec::new();
 
         // Avoid allocation in the common case
         if contiguous_case_ranges.len() <= 3 {
@@ -119,17 +123,17 @@ impl Switch {
                 val,
                 otherwise,
                 contiguous_case_ranges,
-                &mut cases_and_jt_ebbs,
+                &mut cases_and_jt_blocks,
             );
-            return cases_and_jt_ebbs;
+            return cases_and_jt_blocks;
         }
 
-        let mut stack: Vec<(Option, Vec)> = Vec::new();
+        let mut stack: Vec<(Option, Vec)> = Vec::new();
         stack.push((None, contiguous_case_ranges));
 
-        while let Some((ebb, contiguous_case_ranges)) = stack.pop() {
-            if let Some(ebb) = ebb {
-                bx.switch_to_block(ebb);
+        while let Some((block, contiguous_case_ranges)) = stack.pop() {
+            if let Some(block) = block {
+                bx.switch_to_block(block);
             }
 
             if contiguous_case_ranges.len() <= 3 {
@@ -138,64 +142,68 @@ impl Switch {
                     val,
                     otherwise,
                     contiguous_case_ranges,
-                    &mut cases_and_jt_ebbs,
+                    &mut cases_and_jt_blocks,
                 );
             } else {
                 let split_point = contiguous_case_ranges.len() / 2;
                 let mut left = contiguous_case_ranges;
                 let right = left.split_off(split_point);
 
-                let left_ebb = bx.create_ebb();
-                let right_ebb = bx.create_ebb();
+                let left_block = bx.create_block();
+                let right_block = bx.create_block();
 
                 let should_take_right_side = bx.ins().icmp_imm(
                     IntCC::UnsignedGreaterThanOrEqual,
                     val,
                     right[0].first_index as i64,
                 );
-                bx.ins().brnz(should_take_right_side, right_ebb, &[]);
-                bx.ins().jump(left_ebb, &[]);
+                bx.ins().brnz(should_take_right_side, right_block, &[]);
+                bx.ins().jump(left_block, &[]);
 
-                stack.push((Some(left_ebb), left));
-                stack.push((Some(right_ebb), right));
+                stack.push((Some(left_block), left));
+                stack.push((Some(right_block), right));
             }
         }
 
-        cases_and_jt_ebbs
+        cases_and_jt_blocks
     }
 
     /// Linear search for the right `ContiguousCaseRange`.
     fn build_search_branches(
         bx: &mut FunctionBuilder,
         val: Value,
-        otherwise: Ebb,
+        otherwise: Block,
         contiguous_case_ranges: Vec,
-        cases_and_jt_ebbs: &mut Vec<(EntryIndex, Ebb, Vec)>,
+        cases_and_jt_blocks: &mut Vec<(EntryIndex, Block, Vec)>,
     ) {
         let mut was_branch = false;
         let ins_fallthrough_jump = |was_branch: bool, bx: &mut FunctionBuilder| {
             if was_branch {
-                let ebb = bx.create_ebb();
-                bx.ins().jump(ebb, &[]);
-                bx.switch_to_block(ebb);
+                let block = bx.create_block();
+                bx.ins().jump(block, &[]);
+                bx.switch_to_block(block);
             }
         };
-        for ContiguousCaseRange { first_index, ebbs } in contiguous_case_ranges.into_iter().rev() {
-            match (ebbs.len(), first_index) {
+        for ContiguousCaseRange {
+            first_index,
+            blocks,
+        } in contiguous_case_ranges.into_iter().rev()
+        {
+            match (blocks.len(), first_index) {
                 (1, 0) => {
                     ins_fallthrough_jump(was_branch, bx);
-                    bx.ins().brz(val, ebbs[0], &[]);
+                    bx.ins().brz(val, blocks[0], &[]);
                 }
                 (1, _) => {
                     ins_fallthrough_jump(was_branch, bx);
                     let is_good_val = bx.ins().icmp_imm(IntCC::Equal, val, first_index as i64);
-                    bx.ins().brnz(is_good_val, ebbs[0], &[]);
+                    bx.ins().brnz(is_good_val, blocks[0], &[]);
                 }
                 (_, 0) => {
                     // if `first_index` is 0, then `icmp_imm uge val, first_index` is trivially true
-                    let jt_ebb = bx.create_ebb();
-                    bx.ins().jump(jt_ebb, &[]);
-                    cases_and_jt_ebbs.push((first_index, jt_ebb, ebbs));
+                    let jt_block = bx.create_block();
+                    bx.ins().jump(jt_block, &[]);
+                    cases_and_jt_blocks.push((first_index, jt_block, blocks));
                     // `jump otherwise` below must not be hit, because the current block has been
                     // filled above. This is the last iteration anyway, as 0 is the smallest
                     // unsigned int, so just return here.
@@ -203,14 +211,14 @@ impl Switch {
                 }
                 (_, _) => {
                     ins_fallthrough_jump(was_branch, bx);
-                    let jt_ebb = bx.create_ebb();
+                    let jt_block = bx.create_block();
                     let is_good_val = bx.ins().icmp_imm(
                         IntCC::UnsignedGreaterThanOrEqual,
                         val,
                         first_index as i64,
                     );
-                    bx.ins().brnz(is_good_val, jt_ebb, &[]);
-                    cases_and_jt_ebbs.push((first_index, jt_ebb, ebbs));
+                    bx.ins().brnz(is_good_val, jt_block, &[]);
+                    cases_and_jt_blocks.push((first_index, jt_block, blocks));
                 }
             }
             was_branch = true;
@@ -219,21 +227,21 @@ impl Switch {
         bx.ins().jump(otherwise, &[]);
     }
 
-    /// For every item in `cases_and_jt_ebbs` this will create a jump table in the specified ebb.
+    /// For every item in `cases_and_jt_blocks` this will create a jump table in the specified block.
     fn build_jump_tables(
         bx: &mut FunctionBuilder,
         val: Value,
-        otherwise: Ebb,
-        cases_and_jt_ebbs: Vec<(EntryIndex, Ebb, Vec)>,
+        otherwise: Block,
+        cases_and_jt_blocks: Vec<(EntryIndex, Block, Vec)>,
     ) {
-        for (first_index, jt_ebb, ebbs) in cases_and_jt_ebbs.into_iter().rev() {
+        for (first_index, jt_block, blocks) in cases_and_jt_blocks.into_iter().rev() {
             let mut jt_data = JumpTableData::new();
-            for ebb in ebbs {
-                jt_data.push_entry(ebb);
+            for block in blocks {
+                jt_data.push_entry(block);
             }
             let jump_table = bx.create_jump_table(jt_data);
 
-            bx.switch_to_block(jt_ebb);
+            bx.switch_to_block(jt_block);
             let discr = if first_index == 0 {
                 val
             } else {
@@ -249,8 +257,8 @@ impl Switch {
     ///
     /// * The function builder to emit to
     /// * The value to switch on
-    /// * The default ebb
-    pub fn emit(self, bx: &mut FunctionBuilder, val: Value, otherwise: Ebb) {
+    /// * The default block
+    pub fn emit(self, bx: &mut FunctionBuilder, val: Value, otherwise: Block) {
         // FIXME icmp(_imm) doesn't have encodings for i8 and i16 on x86(_64) yet
         let val = match bx.func.dfg.value_type(val) {
             types::I8 | types::I16 => bx.ins().uextend(types::I32, val),
@@ -258,19 +266,20 @@ impl Switch {
         };
 
         let contiguous_case_ranges = self.collect_contiguous_case_ranges();
-        let cases_and_jt_ebbs = Self::build_search_tree(bx, val, otherwise, contiguous_case_ranges);
-        Self::build_jump_tables(bx, val, otherwise, cases_and_jt_ebbs);
+        let cases_and_jt_blocks =
+            Self::build_search_tree(bx, val, otherwise, contiguous_case_ranges);
+        Self::build_jump_tables(bx, val, otherwise, cases_and_jt_blocks);
     }
 }
 
 /// This represents a contiguous range of cases to switch on.
 ///
-/// For example 10 => ebb1, 11 => ebb2, 12 => ebb7 will be represented as:
+/// For example 10 => block1, 11 => block2, 12 => block7 will be represented as:
 ///
 /// ```plain
 /// ContiguousCaseRange {
 ///     first_index: 10,
-///     ebbs: vec![Ebb::from_u32(1), Ebb::from_u32(2), Ebb::from_u32(7)]
+///     blocks: vec![Block::from_u32(1), Block::from_u32(2), Block::from_u32(7)]
 /// }
 /// ```
 #[derive(Debug)]
@@ -278,15 +287,15 @@ struct ContiguousCaseRange {
     /// The entry index of the first case. Eg. 10 when the entry indexes are 10, 11, 12 and 13.
     first_index: EntryIndex,
 
-    /// The ebbs to jump to sorted in ascending order of entry index.
-    ebbs: Vec,
+    /// The blocks to jump to sorted in ascending order of entry index.
+    blocks: Vec,
 }
 
 impl ContiguousCaseRange {
     fn new(first_index: EntryIndex) -> Self {
         Self {
             first_index,
-            ebbs: Vec::new(),
+            blocks: Vec::new(),
         }
     }
 }
@@ -304,15 +313,15 @@ mod tests {
             let mut func_ctx = FunctionBuilderContext::new();
             {
                 let mut bx = FunctionBuilder::new(&mut func, &mut func_ctx);
-                let ebb = bx.create_ebb();
-                bx.switch_to_block(ebb);
+                let block = bx.create_block();
+                bx.switch_to_block(block);
                 let val = bx.ins().iconst(types::I8, 0);
                 let mut switch = Switch::new();
                 $(
-                    let ebb = bx.create_ebb();
-                    switch.set_entry($index, ebb);
+                    let block = bx.create_block();
+                    switch.set_entry($index, block);
                 )*
-                switch.emit(&mut bx, val, Ebb::with_number($default).unwrap());
+                switch.emit(&mut bx, val, Block::with_number($default).unwrap());
             }
             func
                 .to_string()
@@ -327,11 +336,11 @@ mod tests {
         let func = setup!(0, [0,]);
         assert_eq!(
             func,
-            "ebb0:
+            "block0:
     v0 = iconst.i8 0
     v1 = uextend.i32 v0
-    brz v1, ebb1
-    jump ebb0"
+    brz v1, block1
+    jump block0"
         );
     }
 
@@ -340,12 +349,12 @@ mod tests {
         let func = setup!(0, [1,]);
         assert_eq!(
             func,
-            "ebb0:
+            "block0:
     v0 = iconst.i8 0
     v1 = uextend.i32 v0
     v2 = icmp_imm eq v1, 1
-    brnz v2, ebb1
-    jump ebb0"
+    brnz v2, block1
+    jump block0"
         );
     }
 
@@ -354,15 +363,15 @@ mod tests {
         let func = setup!(0, [0, 1,]);
         assert_eq!(
             func,
-            "    jt0 = jump_table [ebb1, ebb2]
+            "    jt0 = jump_table [block1, block2]
 
-ebb0:
+block0:
     v0 = iconst.i8 0
     v1 = uextend.i32 v0
-    jump ebb3
+    jump block3
 
-ebb3:
-    br_table.i32 v1, ebb0, jt0"
+block3:
+    br_table.i32 v1, block0, jt0"
         );
     }
 
@@ -371,16 +380,16 @@ ebb3:
         let func = setup!(0, [0, 2,]);
         assert_eq!(
             func,
-            "ebb0:
+            "block0:
     v0 = iconst.i8 0
     v1 = uextend.i32 v0
     v2 = icmp_imm eq v1, 2
-    brnz v2, ebb2
-    jump ebb3
+    brnz v2, block2
+    jump block3
 
-ebb3:
-    brz.i32 v1, ebb1
-    jump ebb0"
+block3:
+    brz.i32 v1, block1
+    jump block0"
         );
     }
 
@@ -389,37 +398,37 @@ ebb3:
         let func = setup!(0, [0, 1, 5, 7, 10, 11, 12,]);
         assert_eq!(
             func,
-            "    jt0 = jump_table [ebb1, ebb2]
-    jt1 = jump_table [ebb5, ebb6, ebb7]
+            "    jt0 = jump_table [block1, block2]
+    jt1 = jump_table [block5, block6, block7]
 
-ebb0:
+block0:
     v0 = iconst.i8 0
     v1 = uextend.i32 v0
     v2 = icmp_imm uge v1, 7
-    brnz v2, ebb9
-    jump ebb8
+    brnz v2, block9
+    jump block8
 
-ebb9:
+block9:
     v3 = icmp_imm.i32 uge v1, 10
-    brnz v3, ebb10
-    jump ebb11
+    brnz v3, block10
+    jump block11
 
-ebb11:
+block11:
     v4 = icmp_imm.i32 eq v1, 7
-    brnz v4, ebb4
-    jump ebb0
+    brnz v4, block4
+    jump block0
 
-ebb8:
+block8:
     v5 = icmp_imm.i32 eq v1, 5
-    brnz v5, ebb3
-    jump ebb12
+    brnz v5, block3
+    jump block12
 
-ebb12:
-    br_table.i32 v1, ebb0, jt0
+block12:
+    br_table.i32 v1, block0, jt0
 
-ebb10:
+block10:
     v6 = iadd_imm.i32 v1, -10
-    br_table v6, ebb0, jt1"
+    br_table v6, block0, jt1"
         );
     }
 
@@ -428,17 +437,17 @@ ebb10:
         let func = setup!(0, [::core::i64::MIN as u64, 1,]);
         assert_eq!(
             func,
-            "ebb0:
+            "block0:
     v0 = iconst.i8 0
     v1 = uextend.i32 v0
     v2 = icmp_imm eq v1, 0x8000_0000_0000_0000
-    brnz v2, ebb1
-    jump ebb3
+    brnz v2, block1
+    jump block3
 
-ebb3:
+block3:
     v3 = icmp_imm.i32 eq v1, 1
-    brnz v3, ebb2
-    jump ebb0"
+    brnz v3, block2
+    jump block0"
         );
     }
 
@@ -447,17 +456,17 @@ ebb3:
         let func = setup!(0, [::core::i64::MAX as u64, 1,]);
         assert_eq!(
             func,
-            "ebb0:
+            "block0:
     v0 = iconst.i8 0
     v1 = uextend.i32 v0
     v2 = icmp_imm eq v1, 0x7fff_ffff_ffff_ffff
-    brnz v2, ebb1
-    jump ebb3
+    brnz v2, block1
+    jump block3
 
-ebb3:
+block3:
     v3 = icmp_imm.i32 eq v1, 1
-    brnz v3, ebb2
-    jump ebb0"
+    brnz v3, block2
+    jump block0"
         )
     }
 
@@ -466,17 +475,17 @@ ebb3:
         let func = setup!(0, [-1i64 as u64, 0, 1,]);
         assert_eq!(
             func,
-            "    jt0 = jump_table [ebb2, ebb3]
+            "    jt0 = jump_table [block2, block3]
 
-ebb0:
+block0:
     v0 = iconst.i8 0
     v1 = uextend.i32 v0
     v2 = icmp_imm eq v1, -1
-    brnz v2, ebb1
-    jump ebb4
+    brnz v2, block1
+    jump block4
 
-ebb4:
-    br_table.i32 v1, ebb0, jt0"
+block4:
+    br_table.i32 v1, block0, jt0"
         );
     }
 }
diff --git a/cranelift/object/src/backend.rs b/cranelift/object/src/backend.rs
index 28874cb521..d6a2d98385 100644
--- a/cranelift/object/src/backend.rs
+++ b/cranelift/object/src/backend.rs
@@ -525,7 +525,7 @@ struct ObjectRelocSink {
 }
 
 impl RelocSink for ObjectRelocSink {
-    fn reloc_ebb(&mut self, _offset: CodeOffset, _reloc: Reloc, _ebb_offset: CodeOffset) {
+    fn reloc_block(&mut self, _offset: CodeOffset, _reloc: Reloc, _block_offset: CodeOffset) {
         unimplemented!();
     }
 
diff --git a/cranelift/preopt/src/constant_folding.rs b/cranelift/preopt/src/constant_folding.rs
index b87db2d98f..40d597eddc 100644
--- a/cranelift/preopt/src/constant_folding.rs
+++ b/cranelift/preopt/src/constant_folding.rs
@@ -44,7 +44,7 @@ impl ConstImm {
 pub fn fold_constants(func: &mut ir::Function) {
     let mut pos = FuncCursor::new(func);
 
-    while let Some(_ebb) = pos.next_ebb() {
+    while let Some(_block) = pos.next_block() {
         while let Some(inst) = pos.next_inst() {
             use self::ir::InstructionData::*;
             match pos.func.dfg[inst] {
@@ -225,7 +225,7 @@ fn fold_unary(dfg: &mut ir::DataFlowGraph, inst: ir::Inst, opcode: ir::Opcode, a
 }
 
 fn fold_branch(pos: &mut FuncCursor, inst: ir::Inst, opcode: ir::Opcode) {
-    let (cond, ebb, args) = {
+    let (cond, block, args) = {
         let values = pos.func.dfg.inst_args(inst);
         let inst_data = &pos.func.dfg[inst];
         (
@@ -246,8 +246,8 @@ fn fold_branch(pos: &mut FuncCursor, inst: ir::Inst, opcode: ir::Opcode) {
     };
 
     if (branch_if_zero && !truthiness) || (!branch_if_zero && truthiness) {
-        pos.func.dfg.replace(inst).jump(ebb, &args);
-        // remove the rest of the ebb to avoid verifier errors
+        pos.func.dfg.replace(inst).jump(block, &args);
+        // remove the rest of the block to avoid verifier errors
         while let Some(next_inst) = pos.func.layout.next_inst(inst) {
             pos.func.layout.remove_inst(next_inst);
         }
diff --git a/cranelift/reader/src/lexer.rs b/cranelift/reader/src/lexer.rs
index 9838227e86..1d2908a92f 100644
--- a/cranelift/reader/src/lexer.rs
+++ b/cranelift/reader/src/lexer.rs
@@ -2,7 +2,7 @@
 
 use crate::error::Location;
 use cranelift_codegen::ir::types;
-use cranelift_codegen::ir::{Ebb, Value};
+use cranelift_codegen::ir::{Block, Value};
 #[allow(unused_imports, deprecated)]
 use std::ascii::AsciiExt;
 use std::str::CharIndices;
@@ -33,7 +33,7 @@ pub enum Token<'a> {
     Integer(&'a str),     // Integer immediate
     Type(types::Type),    // i32, f32, b32x4, ...
     Value(Value),         // v12, v7
-    Ebb(Ebb),             // ebb3
+    Block(Block),         // block3
     StackSlot(u32),       // ss3
     GlobalValue(u32),     // gv3
     Heap(u32),            // heap2
@@ -318,7 +318,7 @@ impl<'a> Lexer<'a> {
         }
         let text = &self.source[begin..self.pos];
 
-        // Look for numbered well-known entities like ebb15, v45, ...
+        // Look for numbered well-known entities like block15, v45, ...
         token(
             split_entity_name(text)
                 .and_then(|(prefix, number)| {
@@ -339,7 +339,7 @@ impl<'a> Lexer<'a> {
     fn numbered_entity(prefix: &str, number: u32) -> Option> {
         match prefix {
             "v" => Value::with_number(number).map(Token::Value),
-            "ebb" => Ebb::with_number(number).map(Token::Ebb),
+            "block" => Block::with_number(number).map(Token::Block),
             "ss" => Some(Token::StackSlot(number)),
             "gv" => Some(Token::GlobalValue(number)),
             "heap" => Some(Token::Heap(number)),
@@ -519,7 +519,7 @@ mod tests {
     use super::*;
     use crate::error::Location;
     use cranelift_codegen::ir::types;
-    use cranelift_codegen::ir::{Ebb, Value};
+    use cranelift_codegen::ir::{Block, Value};
 
     #[test]
     fn digits() {
@@ -616,7 +616,7 @@ mod tests {
     #[test]
     fn lex_identifiers() {
         let mut lex = Lexer::new(
-            "v0 v00 vx01 ebb1234567890 ebb5234567890 v1x vx1 vxvx4 \
+            "v0 v00 vx01 block1234567890 block5234567890 v1x vx1 vxvx4 \
              function0 function b1 i32x4 f32x5 \
              iflags fflags iflagss",
         );
@@ -628,9 +628,9 @@ mod tests {
         assert_eq!(lex.next(), token(Token::Identifier("vx01"), 1));
         assert_eq!(
             lex.next(),
-            token(Token::Ebb(Ebb::with_number(1234567890).unwrap()), 1)
+            token(Token::Block(Block::with_number(1234567890).unwrap()), 1)
         );
-        assert_eq!(lex.next(), token(Token::Identifier("ebb5234567890"), 1));
+        assert_eq!(lex.next(), token(Token::Identifier("block5234567890"), 1));
         assert_eq!(lex.next(), token(Token::Identifier("v1x"), 1));
         assert_eq!(lex.next(), token(Token::Identifier("vx1"), 1));
         assert_eq!(lex.next(), token(Token::Identifier("vxvx4"), 1));
@@ -656,7 +656,7 @@ mod tests {
 
     #[test]
     fn lex_names() {
-        let mut lex = Lexer::new("%0 %x3 %function %123_abc %ss0 %v3 %ebb11 %_");
+        let mut lex = Lexer::new("%0 %x3 %function %123_abc %ss0 %v3 %block11 %_");
 
         assert_eq!(lex.next(), token(Token::Name("0"), 1));
         assert_eq!(lex.next(), token(Token::Name("x3"), 1));
@@ -664,7 +664,7 @@ mod tests {
         assert_eq!(lex.next(), token(Token::Name("123_abc"), 1));
         assert_eq!(lex.next(), token(Token::Name("ss0"), 1));
         assert_eq!(lex.next(), token(Token::Name("v3"), 1));
-        assert_eq!(lex.next(), token(Token::Name("ebb11"), 1));
+        assert_eq!(lex.next(), token(Token::Name("block11"), 1));
         assert_eq!(lex.next(), token(Token::Name("_"), 1));
     }
 
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 8b603c7870..f0b866f472 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -14,7 +14,7 @@ use cranelift_codegen::ir::instructions::{InstructionData, InstructionFormat, Va
 use cranelift_codegen::ir::types::INVALID;
 use cranelift_codegen::ir::types::*;
 use cranelift_codegen::ir::{
-    AbiParam, ArgumentExtension, ArgumentLoc, ConstantData, Ebb, ExtFuncData, ExternalName,
+    AbiParam, ArgumentExtension, ArgumentLoc, Block, ConstantData, ExtFuncData, ExternalName,
     FuncRef, Function, GlobalValue, GlobalValueData, Heap, HeapData, HeapStyle, JumpTable,
     JumpTableData, MemFlags, Opcode, SigRef, Signature, StackSlot, StackSlotData, StackSlotKind,
     Table, TableData, Type, Value, ValueLoc,
@@ -334,14 +334,14 @@ impl<'a> Context<'a> {
         }
     }
 
-    // Allocate a new EBB.
-    fn add_ebb(&mut self, ebb: Ebb, loc: Location) -> ParseResult {
-        self.map.def_ebb(ebb, loc)?;
-        while self.function.dfg.num_ebbs() <= ebb.index() {
-            self.function.dfg.make_ebb();
+    // Allocate a new block.
+    fn add_block(&mut self, block: Block, loc: Location) -> ParseResult {
+        self.map.def_block(block, loc)?;
+        while self.function.dfg.num_blocks() <= block.index() {
+            self.function.dfg.make_block();
         }
-        self.function.layout.append_ebb(ebb);
-        Ok(ebb)
+        self.function.layout.append_block(block);
+        Ok(block)
     }
 }
 
@@ -554,11 +554,11 @@ impl<'a> Parser<'a> {
         err!(self.loc, "expected jump table number: jt«n»")
     }
 
-    // Match and consume an ebb reference.
-    fn match_ebb(&mut self, err_msg: &str) -> ParseResult {
-        if let Some(Token::Ebb(ebb)) = self.token() {
+    // Match and consume an block reference.
+    fn match_block(&mut self, err_msg: &str) -> ParseResult {
+        if let Some(Token::Block(block)) = self.token() {
             self.consume();
-            Ok(ebb)
+            Ok(block)
         } else {
             err!(self.loc, err_msg)
         }
@@ -1686,9 +1686,9 @@ impl<'a> Parser<'a> {
 
         let mut data = JumpTableData::new();
 
-        // jump-table-decl ::= JumpTable(jt) "=" "jump_table" "[" * Ebb(dest) {"," Ebb(dest)} "]"
+        // jump-table-decl ::= JumpTable(jt) "=" "jump_table" "[" * Block(dest) {"," Block(dest)} "]"
         match self.token() {
-            Some(Token::Ebb(dest)) => {
+            Some(Token::Block(dest)) => {
                 self.consume();
                 data.push_entry(dest);
 
@@ -1696,7 +1696,7 @@ impl<'a> Parser<'a> {
                     match self.token() {
                         Some(Token::Comma) => {
                             self.consume();
-                            if let Some(Token::Ebb(dest)) = self.token() {
+                            if let Some(Token::Block(dest)) = self.token() {
                                 self.consume();
                                 data.push_entry(dest);
                             } else {
@@ -1727,13 +1727,13 @@ impl<'a> Parser<'a> {
     //
     fn parse_function_body(&mut self, ctx: &mut Context) -> ParseResult<()> {
         while self.token() != Some(Token::RBrace) {
-            self.parse_extended_basic_block(ctx)?;
+            self.parse_basic_block(ctx)?;
         }
 
         // Now that we've seen all defined values in the function, ensure that
         // all references refer to a definition.
-        for ebb in &ctx.function.layout {
-            for inst in ctx.function.layout.ebb_insts(ebb) {
+        for block in &ctx.function.layout {
+            for inst in ctx.function.layout.block_insts(block) {
                 for value in ctx.function.dfg.inst_args(inst) {
                     if !ctx.map.contains_value(*value) {
                         return err!(
@@ -1756,29 +1756,29 @@ impl<'a> Parser<'a> {
         Ok(())
     }
 
-    // Parse an extended basic block, add contents to `ctx`.
+    // Parse a basic block, add contents to `ctx`.
     //
-    // extended-basic-block ::= * ebb-header { instruction }
-    // ebb-header           ::= Ebb(ebb) [ebb-params] ":"
+    // extended-basic-block ::= * block-header { instruction }
+    // block-header           ::= Block(block) [block-params] ":"
     //
-    fn parse_extended_basic_block(&mut self, ctx: &mut Context) -> ParseResult<()> {
-        // Collect comments for the next ebb.
+    fn parse_basic_block(&mut self, ctx: &mut Context) -> ParseResult<()> {
+        // Collect comments for the next block.
         self.start_gathering_comments();
 
-        let ebb_num = self.match_ebb("expected EBB header")?;
-        let ebb = ctx.add_ebb(ebb_num, self.loc)?;
+        let block_num = self.match_block("expected block header")?;
+        let block = ctx.add_block(block_num, self.loc)?;
 
         if !self.optional(Token::Colon) {
-            // ebb-header ::= Ebb(ebb) [ * ebb-params ] ":"
-            self.parse_ebb_params(ctx, ebb)?;
-            self.match_token(Token::Colon, "expected ':' after EBB parameters")?;
+            // block-header ::= Block(block) [ * block-params ] ":"
+            self.parse_block_params(ctx, block)?;
+            self.match_token(Token::Colon, "expected ':' after block parameters")?;
         }
 
         // Collect any trailing comments.
         self.token();
-        self.claim_gathered_comments(ebb);
+        self.claim_gathered_comments(block);
 
-        // extended-basic-block ::= ebb-header * { instruction }
+        // extended-basic-block ::= block-header * { instruction }
         while match self.token() {
             Some(Token::Value(_))
             | Some(Token::Identifier(_))
@@ -1808,64 +1808,76 @@ impl<'a> Parser<'a> {
                 }
                 Some(Token::Equal) => {
                     self.consume();
-                    self.parse_instruction(&results, srcloc, encoding, result_locations, ctx, ebb)?;
+                    self.parse_instruction(
+                        &results,
+                        srcloc,
+                        encoding,
+                        result_locations,
+                        ctx,
+                        block,
+                    )?;
                 }
                 _ if !results.is_empty() => return err!(self.loc, "expected -> or ="),
-                _ => {
-                    self.parse_instruction(&results, srcloc, encoding, result_locations, ctx, ebb)?
-                }
+                _ => self.parse_instruction(
+                    &results,
+                    srcloc,
+                    encoding,
+                    result_locations,
+                    ctx,
+                    block,
+                )?,
             }
         }
 
         Ok(())
     }
 
-    // Parse parenthesized list of EBB parameters. Returns a vector of (u32, Type) pairs with the
+    // Parse parenthesized list of block parameters. Returns a vector of (u32, Type) pairs with the
     // value numbers of the defined values and the defined types.
     //
-    // ebb-params ::= * "(" ebb-param { "," ebb-param } ")"
-    fn parse_ebb_params(&mut self, ctx: &mut Context, ebb: Ebb) -> ParseResult<()> {
-        // ebb-params ::= * "(" ebb-param { "," ebb-param } ")"
-        self.match_token(Token::LPar, "expected '(' before EBB parameters")?;
+    // block-params ::= * "(" block-param { "," block-param } ")"
+    fn parse_block_params(&mut self, ctx: &mut Context, block: Block) -> ParseResult<()> {
+        // block-params ::= * "(" block-param { "," block-param } ")"
+        self.match_token(Token::LPar, "expected '(' before block parameters")?;
 
-        // ebb-params ::= "(" * ebb-param { "," ebb-param } ")"
-        self.parse_ebb_param(ctx, ebb)?;
+        // block-params ::= "(" * block-param { "," block-param } ")"
+        self.parse_block_param(ctx, block)?;
 
-        // ebb-params ::= "(" ebb-param * { "," ebb-param } ")"
+        // block-params ::= "(" block-param * { "," block-param } ")"
         while self.optional(Token::Comma) {
-            // ebb-params ::= "(" ebb-param { "," * ebb-param } ")"
-            self.parse_ebb_param(ctx, ebb)?;
+            // block-params ::= "(" block-param { "," * block-param } ")"
+            self.parse_block_param(ctx, block)?;
         }
 
-        // ebb-params ::= "(" ebb-param { "," ebb-param } * ")"
-        self.match_token(Token::RPar, "expected ')' after EBB parameters")?;
+        // block-params ::= "(" block-param { "," block-param } * ")"
+        self.match_token(Token::RPar, "expected ')' after block parameters")?;
 
         Ok(())
     }
 
-    // Parse a single EBB parameter declaration, and append it to `ebb`.
+    // Parse a single block parameter declaration, and append it to `block`.
     //
-    // ebb-param ::= * Value(v) ":" Type(t) arg-loc?
+    // block-param ::= * Value(v) ":" Type(t) arg-loc?
     // arg-loc ::= "[" value-location "]"
     //
-    fn parse_ebb_param(&mut self, ctx: &mut Context, ebb: Ebb) -> ParseResult<()> {
-        // ebb-param ::= * Value(v) ":" Type(t) arg-loc?
-        let v = self.match_value("EBB argument must be a value")?;
+    fn parse_block_param(&mut self, ctx: &mut Context, block: Block) -> ParseResult<()> {
+        // block-param ::= * Value(v) ":" Type(t) arg-loc?
+        let v = self.match_value("block argument must be a value")?;
         let v_location = self.loc;
-        // ebb-param ::= Value(v) * ":" Type(t) arg-loc?
-        self.match_token(Token::Colon, "expected ':' after EBB argument")?;
-        // ebb-param ::= Value(v) ":" * Type(t) arg-loc?
+        // block-param ::= Value(v) * ":" Type(t) arg-loc?
+        self.match_token(Token::Colon, "expected ':' after block argument")?;
+        // block-param ::= Value(v) ":" * Type(t) arg-loc?
 
         while ctx.function.dfg.num_values() <= v.index() {
             ctx.function.dfg.make_invalid_value_for_parser();
         }
 
-        let t = self.match_type("expected EBB argument type")?;
-        // Allocate the EBB argument.
-        ctx.function.dfg.append_ebb_param_for_parser(ebb, t, v);
+        let t = self.match_type("expected block argument type")?;
+        // Allocate the block argument.
+        ctx.function.dfg.append_block_param_for_parser(block, t, v);
         ctx.map.def_value(v, v_location)?;
 
-        // ebb-param ::= Value(v) ":" Type(t) * arg-loc?
+        // block-param ::= Value(v) ":" Type(t) * arg-loc?
         if self.optional(Token::LBracket) {
             let loc = self.parse_value_location(ctx)?;
             ctx.function.locations[v] = loc;
@@ -1981,7 +1993,7 @@ impl<'a> Parser<'a> {
         Ok(results)
     }
 
-    // Parse a value alias, and append it to `ebb`.
+    // Parse a value alias, and append it to `block`.
     //
     // value_alias ::= [inst-results] "->" Value(v)
     //
@@ -2022,7 +2034,7 @@ impl<'a> Parser<'a> {
         Ok(())
     }
 
-    // Parse an instruction, append it to `ebb`.
+    // Parse an instruction, append it to `block`.
     //
     // instruction ::= [inst-results "="] Opcode(opc) ["." Type] ...
     //
@@ -2033,7 +2045,7 @@ impl<'a> Parser<'a> {
         encoding: Option,
         result_locations: Option>,
         ctx: &mut Context,
-        ebb: Ebb,
+        block: Block,
     ) -> ParseResult<()> {
         // Define the result values.
         for val in results {
@@ -2077,7 +2089,7 @@ impl<'a> Parser<'a> {
             ctx.function
                 .dfg
                 .make_inst_results_for_parser(inst, ctrl_typevar, results);
-        ctx.function.layout.append_inst(inst, ebb);
+        ctx.function.layout.append_inst(inst, block);
         ctx.map
             .def_entity(inst.into(), opcode_loc)
             .expect("duplicate inst references created");
@@ -2345,23 +2357,23 @@ impl<'a> Parser<'a> {
             }
             InstructionFormat::NullAry => InstructionData::NullAry { opcode },
             InstructionFormat::Jump => {
-                // Parse the destination EBB number.
-                let ebb_num = self.match_ebb("expected jump destination EBB")?;
+                // Parse the destination block number.
+                let block_num = self.match_block("expected jump destination block")?;
                 let args = self.parse_opt_value_list()?;
                 InstructionData::Jump {
                     opcode,
-                    destination: ebb_num,
+                    destination: block_num,
                     args: args.into_value_list(&[], &mut ctx.function.dfg.value_lists),
                 }
             }
             InstructionFormat::Branch => {
                 let ctrl_arg = self.match_value("expected SSA value control operand")?;
                 self.match_token(Token::Comma, "expected ',' between operands")?;
-                let ebb_num = self.match_ebb("expected branch destination EBB")?;
+                let block_num = self.match_block("expected branch destination block")?;
                 let args = self.parse_opt_value_list()?;
                 InstructionData::Branch {
                     opcode,
-                    destination: ebb_num,
+                    destination: block_num,
                     args: args.into_value_list(&[ctrl_arg], &mut ctx.function.dfg.value_lists),
                 }
             }
@@ -2369,12 +2381,12 @@ impl<'a> Parser<'a> {
                 let cond = self.match_enum("expected intcc condition code")?;
                 let arg = self.match_value("expected SSA value first operand")?;
                 self.match_token(Token::Comma, "expected ',' between operands")?;
-                let ebb_num = self.match_ebb("expected branch destination EBB")?;
+                let block_num = self.match_block("expected branch destination block")?;
                 let args = self.parse_opt_value_list()?;
                 InstructionData::BranchInt {
                     opcode,
                     cond,
-                    destination: ebb_num,
+                    destination: block_num,
                     args: args.into_value_list(&[arg], &mut ctx.function.dfg.value_lists),
                 }
             }
@@ -2382,12 +2394,12 @@ impl<'a> Parser<'a> {
                 let cond = self.match_enum("expected floatcc condition code")?;
                 let arg = self.match_value("expected SSA value first operand")?;
                 self.match_token(Token::Comma, "expected ',' between operands")?;
-                let ebb_num = self.match_ebb("expected branch destination EBB")?;
+                let block_num = self.match_block("expected branch destination block")?;
                 let args = self.parse_opt_value_list()?;
                 InstructionData::BranchFloat {
                     opcode,
                     cond,
-                    destination: ebb_num,
+                    destination: block_num,
                     args: args.into_value_list(&[arg], &mut ctx.function.dfg.value_lists),
                 }
             }
@@ -2397,26 +2409,26 @@ impl<'a> Parser<'a> {
                 self.match_token(Token::Comma, "expected ',' between operands")?;
                 let rhs = self.match_value("expected SSA value second operand")?;
                 self.match_token(Token::Comma, "expected ',' between operands")?;
-                let ebb_num = self.match_ebb("expected branch destination EBB")?;
+                let block_num = self.match_block("expected branch destination block")?;
                 let args = self.parse_opt_value_list()?;
                 InstructionData::BranchIcmp {
                     opcode,
                     cond,
-                    destination: ebb_num,
+                    destination: block_num,
                     args: args.into_value_list(&[lhs, rhs], &mut ctx.function.dfg.value_lists),
                 }
             }
             InstructionFormat::BranchTable => {
                 let arg = self.match_value("expected SSA value operand")?;
                 self.match_token(Token::Comma, "expected ',' between operands")?;
-                let ebb_num = self.match_ebb("expected branch destination EBB")?;
+                let block_num = self.match_block("expected branch destination block")?;
                 self.match_token(Token::Comma, "expected ',' between operands")?;
                 let table = self.match_jt()?;
                 ctx.check_jt(table, self.loc)?;
                 InstructionData::BranchTable {
                     opcode,
                     arg,
-                    destination: ebb_num,
+                    destination: block_num,
                     table,
                 }
             }
@@ -2810,7 +2822,7 @@ mod tests {
     fn aliases() {
         let (func, details) = Parser::new(
             "function %qux() system_v {
-                                           ebb0:
+                                           block0:
                                              v4 = iconst.i8 6
                                              v3 -> v4
                                              v1 = iadd_imm v3, 17
@@ -2925,45 +2937,45 @@ mod tests {
     }
 
     #[test]
-    fn ebb_header() {
+    fn block_header() {
         let (func, _) = Parser::new(
-            "function %ebbs() system_v {
-                                     ebb0:
-                                     ebb4(v3: i32):
+            "function %blocks() system_v {
+                                     block0:
+                                     block4(v3: i32):
                                      }",
         )
         .parse_function(None)
         .unwrap();
-        assert_eq!(func.name.to_string(), "%ebbs");
+        assert_eq!(func.name.to_string(), "%blocks");
 
-        let mut ebbs = func.layout.ebbs();
+        let mut blocks = func.layout.blocks();
 
-        let ebb0 = ebbs.next().unwrap();
-        assert_eq!(func.dfg.ebb_params(ebb0), &[]);
+        let block0 = blocks.next().unwrap();
+        assert_eq!(func.dfg.block_params(block0), &[]);
 
-        let ebb4 = ebbs.next().unwrap();
-        let ebb4_args = func.dfg.ebb_params(ebb4);
-        assert_eq!(ebb4_args.len(), 1);
-        assert_eq!(func.dfg.value_type(ebb4_args[0]), types::I32);
+        let block4 = blocks.next().unwrap();
+        let block4_args = func.dfg.block_params(block4);
+        assert_eq!(block4_args.len(), 1);
+        assert_eq!(func.dfg.value_type(block4_args[0]), types::I32);
     }
 
     #[test]
-    fn duplicate_ebb() {
+    fn duplicate_block() {
         let ParseError {
             location,
             message,
             is_warning,
         } = Parser::new(
-            "function %ebbs() system_v {
-                ebb0:
-                ebb0:
+            "function %blocks() system_v {
+                block0:
+                block0:
                     return 2",
         )
         .parse_function(None)
         .unwrap_err();
 
         assert_eq!(location.line_number, 3);
-        assert_eq!(message, "duplicate entity: ebb0");
+        assert_eq!(message, "duplicate entity: block0");
         assert!(!is_warning);
     }
 
@@ -2974,7 +2986,7 @@ mod tests {
             message,
             is_warning,
         } = Parser::new(
-            "function %ebbs() system_v {
+            "function %blocks() system_v {
                 jt0 = jump_table []
                 jt0 = jump_table []",
         )
@@ -2993,7 +3005,7 @@ mod tests {
             message,
             is_warning,
         } = Parser::new(
-            "function %ebbs() system_v {
+            "function %blocks() system_v {
                 ss0 = explicit_slot 8
                 ss0 = explicit_slot 8",
         )
@@ -3012,7 +3024,7 @@ mod tests {
             message,
             is_warning,
         } = Parser::new(
-            "function %ebbs() system_v {
+            "function %blocks() system_v {
                 gv0 = vmctx
                 gv0 = vmctx",
         )
@@ -3031,7 +3043,7 @@ mod tests {
             message,
             is_warning,
         } = Parser::new(
-            "function %ebbs() system_v {
+            "function %blocks() system_v {
                 heap0 = static gv0, min 0x1000, bound 0x10_0000, offset_guard 0x1000
                 heap0 = static gv0, min 0x1000, bound 0x10_0000, offset_guard 0x1000",
         )
@@ -3050,7 +3062,7 @@ mod tests {
             message,
             is_warning,
         } = Parser::new(
-            "function %ebbs() system_v {
+            "function %blocks() system_v {
                 sig0 = ()
                 sig0 = ()",
         )
@@ -3069,7 +3081,7 @@ mod tests {
             message,
             is_warning,
         } = Parser::new(
-            "function %ebbs() system_v {
+            "function %blocks() system_v {
                 sig0 = ()
                 fn0 = %foo sig0
                 fn0 = %foo sig0",
@@ -3089,9 +3101,9 @@ mod tests {
                          function %comment() system_v { ; decl
                             ss10  = outgoing_arg 13 ; stackslot.
                             ; Still stackslot.
-                            jt10 = jump_table [ebb0]
+                            jt10 = jump_table [block0]
                             ; Jumptable
-                         ebb0: ; Basic block
+                         block0: ; Basic block
                          trap user42; Instruction
                          } ; Trailing.
                          ; More trailing.",
@@ -3112,7 +3124,7 @@ mod tests {
         assert_eq!(comments[2].text, "; Still stackslot.");
         assert_eq!(comments[3].entity.to_string(), "jt10");
         assert_eq!(comments[3].text, "; Jumptable");
-        assert_eq!(comments[4].entity.to_string(), "ebb0");
+        assert_eq!(comments[4].entity.to_string(), "block0");
         assert_eq!(comments[4].text, "; Basic block");
 
         assert_eq!(comments[5].entity.to_string(), "inst0");
@@ -3195,7 +3207,7 @@ mod tests {
         // Valid characters in the name:
         let func = Parser::new(
             "function u1:2() system_v {
-                                           ebb0:
+                                           block0:
                                              trap int_divz
                                            }",
         )
@@ -3207,7 +3219,7 @@ mod tests {
         // Invalid characters in the name:
         let mut parser = Parser::new(
             "function u123:abc() system_v {
-                                           ebb0:
+                                           block0:
                                              trap stk_ovf
                                            }",
         );
@@ -3216,7 +3228,7 @@ mod tests {
         // Incomplete function names should not be valid:
         let mut parser = Parser::new(
             "function u() system_v {
-                                           ebb0:
+                                           block0:
                                              trap int_ovf
                                            }",
         );
@@ -3224,7 +3236,7 @@ mod tests {
 
         let mut parser = Parser::new(
             "function u0() system_v {
-                                           ebb0:
+                                           block0:
                                              trap int_ovf
                                            }",
         );
@@ -3232,7 +3244,7 @@ mod tests {
 
         let mut parser = Parser::new(
             "function u0:() system_v {
-                                           ebb0:
+                                           block0:
                                              trap int_ovf
                                            }",
         );
@@ -3242,7 +3254,7 @@ mod tests {
     #[test]
     fn change_default_calling_convention() {
         let code = "function %test() {
-        ebb0:
+        block0:
             return
         }";
 
diff --git a/cranelift/reader/src/sourcemap.rs b/cranelift/reader/src/sourcemap.rs
index 6291d0cd76..126fd219c3 100644
--- a/cranelift/reader/src/sourcemap.rs
+++ b/cranelift/reader/src/sourcemap.rs
@@ -1,7 +1,7 @@
 //! Source map associating entities with their source locations.
 //!
 //! When the parser reads in a source file, it records the locations of the
-//! definitions of entities like instructions, EBBs, and values.
+//! definitions of entities like instructions, blocks, and values.
 //!
 //! The `SourceMap` struct defined in this module makes this mapping available
 //! to parser clients.
@@ -10,7 +10,7 @@ use crate::error::{Location, ParseResult};
 use crate::lexer::split_entity_name;
 use cranelift_codegen::ir::entities::AnyEntity;
 use cranelift_codegen::ir::{
-    Ebb, FuncRef, GlobalValue, Heap, JumpTable, SigRef, StackSlot, Table, Value,
+    Block, FuncRef, GlobalValue, Heap, JumpTable, SigRef, StackSlot, Table, Value,
 };
 use std::collections::HashMap;
 
@@ -28,9 +28,9 @@ impl SourceMap {
         self.locations.contains_key(&v.into())
     }
 
-    /// Look up a EBB entity.
-    pub fn contains_ebb(&self, ebb: Ebb) -> bool {
-        self.locations.contains_key(&ebb.into())
+    /// Look up a block entity.
+    pub fn contains_block(&self, block: Block) -> bool {
+        self.locations.contains_key(&block.into())
     }
 
     /// Look up a stack slot entity.
@@ -79,11 +79,11 @@ impl SourceMap {
                     Some(v.into())
                 }
             }),
-            "ebb" => Ebb::with_number(num).and_then(|ebb| {
-                if !self.contains_ebb(ebb) {
+            "block" => Block::with_number(num).and_then(|block| {
+                if !self.contains_block(block) {
                     None
                 } else {
-                    Some(ebb.into())
+                    Some(block.into())
                 }
             }),
             "ss" => StackSlot::with_number(num).and_then(|ss| {
@@ -158,8 +158,8 @@ impl SourceMap {
         self.def_entity(entity.into(), loc)
     }
 
-    /// Define the ebb `entity`.
-    pub fn def_ebb(&mut self, entity: Ebb, loc: Location) -> ParseResult<()> {
+    /// Define the block `entity`.
+    pub fn def_block(&mut self, entity: Block, loc: Location) -> ParseResult<()> {
         self.def_entity(entity.into(), loc)
     }
 
@@ -218,8 +218,8 @@ mod tests {
         let tf = parse_test(
             "function %detail() {
                                ss10 = incoming_arg 13
-                               jt10 = jump_table [ebb0]
-                             ebb0(v4: i32, v7: i32):
+                               jt10 = jump_table [block0]
+                             block0(v4: i32, v7: i32):
                                v10 = iadd v4, v7
                              }",
             ParseOptions::default(),
@@ -231,7 +231,7 @@ mod tests {
         assert_eq!(map.lookup_str("ss1"), None);
         assert_eq!(map.lookup_str("ss10").unwrap().to_string(), "ss10");
         assert_eq!(map.lookup_str("jt10").unwrap().to_string(), "jt10");
-        assert_eq!(map.lookup_str("ebb0").unwrap().to_string(), "ebb0");
+        assert_eq!(map.lookup_str("block0").unwrap().to_string(), "block0");
         assert_eq!(map.lookup_str("v4").unwrap().to_string(), "v4");
         assert_eq!(map.lookup_str("v7").unwrap().to_string(), "v7");
         assert_eq!(map.lookup_str("v10").unwrap().to_string(), "v10");
diff --git a/cranelift/reader/src/testfile.rs b/cranelift/reader/src/testfile.rs
index 68c7d30a93..4cfdd8f3e5 100644
--- a/cranelift/reader/src/testfile.rs
+++ b/cranelift/reader/src/testfile.rs
@@ -44,7 +44,7 @@ pub struct Details<'a> {
 
 /// A comment in a parsed function.
 ///
-/// The comment belongs to the immediately preceding entity, whether that is an EBB header, and
+/// The comment belongs to the immediately preceding entity, whether that is an block header, and
 /// instruction, or one of the preamble declarations.
 ///
 /// Comments appearing inside the function but before the preamble, as well as comments appearing
diff --git a/cranelift/serde/src/serde_clif_json.rs b/cranelift/serde/src/serde_clif_json.rs
index 0d19ee5fa0..2d950cf3a8 100644
--- a/cranelift/serde/src/serde_clif_json.rs
+++ b/cranelift/serde/src/serde_clif_json.rs
@@ -1,4 +1,4 @@
-use cranelift_codegen::ir::{Ebb, Function, Inst, InstructionData, Signature};
+use cranelift_codegen::ir::{Block, Function, Inst, InstructionData, Signature};
 use serde_derive::{Deserialize, Serialize};
 
 /// Serializable version of the original Cranelift IR
@@ -758,27 +758,27 @@ impl SerInst {
     }
 }
 
-/// Serializable version of Cranelift IR Ebbs.
+/// Serializable version of Cranelift IR Blocks.
 #[derive(Clone, Deserialize, Serialize, Debug)]
-pub struct SerEbb {
-    pub ebb: String,
+pub struct SerBlock {
+    pub block: String,
     pub params: Vec,
     pub insts: Vec,
 }
 
-impl SerEbb {
+impl SerBlock {
     pub fn new(name: String) -> Self {
         Self {
-            ebb: name,
+            block: name,
             params: Vec::new(),
             insts: Vec::new(),
         }
     }
 }
 
-pub fn populate_inst(func: &Function, ebb: Ebb) -> Vec {
+pub fn populate_inst(func: &Function, block: Block) -> Vec {
     let mut ser_vec: Vec = Vec::new();
-    let ret_iter = func.layout.ebb_insts(ebb);
+    let ret_iter = func.layout.block_insts(block);
     for inst in ret_iter {
         let ser_inst: SerInst = SerInst::new(inst, &func);
         ser_vec.push(ser_inst);
@@ -786,10 +786,10 @@ pub fn populate_inst(func: &Function, ebb: Ebb) -> Vec {
     ser_vec
 }
 
-/// Translating Ebb parameters into serializable parameters.
-pub fn populate_params(func: &Function, ebb: Ebb) -> Vec {
+/// Translating Block parameters into serializable parameters.
+pub fn populate_params(func: &Function, block: Block) -> Vec {
     let mut ser_vec: Vec = Vec::new();
-    let parameters = func.dfg.ebb_params(ebb);
+    let parameters = func.dfg.block_params(block);
     for param in parameters {
         ser_vec.push(param.to_string());
     }
@@ -799,27 +799,27 @@ pub fn populate_params(func: &Function, ebb: Ebb) -> Vec {
 /// Serializable Data Flow Graph.
 #[derive(Deserialize, Serialize, Debug)]
 pub struct SerDataFlowGraph {
-    ebbs: Vec,
+    blocks: Vec,
 }
 
-/// Serialize all parts of the Cranelift Ebb data structure, this includes name, parameters, and
+/// Serialize all parts of the Cranelift Block data structure, this includes name, parameters, and
 /// instructions.
-pub fn populate_ebbs(func: &Function) -> Vec {
-    let mut ebb_vec: Vec = Vec::new();
-    for ebb in func.layout.ebbs() {
-        let mut ser_ebb: SerEbb = SerEbb::new(ebb.to_string());
-        ser_ebb.params = populate_params(&func, ebb);
-        ser_ebb.insts = populate_inst(&func, ebb);
-        ebb_vec.push(ser_ebb);
+pub fn populate_blocks(func: &Function) -> Vec {
+    let mut block_vec: Vec = Vec::new();
+    for block in func.layout.blocks() {
+        let mut ser_block: SerBlock = SerBlock::new(block.to_string());
+        ser_block.params = populate_params(&func, block);
+        ser_block.insts = populate_inst(&func, block);
+        block_vec.push(ser_block);
     }
-    ebb_vec
+    block_vec
 }
 
-/// Serializable Cranelift IR data flow graph, including all ebbs.
+/// Serializable Cranelift IR data flow graph, including all blocks.
 impl SerDataFlowGraph {
     pub fn create_new(func: &Function) -> Self {
         Self {
-            ebbs: populate_ebbs(func),
+            blocks: populate_blocks(func),
         }
     }
 
diff --git a/cranelift/simplejit/examples/simplejit-minimal.rs b/cranelift/simplejit/examples/simplejit-minimal.rs
index 3b8e147830..ade2cd3422 100644
--- a/cranelift/simplejit/examples/simplejit-minimal.rs
+++ b/cranelift/simplejit/examples/simplejit-minimal.rs
@@ -27,11 +27,11 @@ fn main() {
     ctx.func.name = ExternalName::user(0, func_a.as_u32());
     {
         let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
-        let ebb = bcx.create_ebb();
+        let block = bcx.create_block();
 
-        bcx.switch_to_block(ebb);
-        bcx.append_ebb_params_for_function_params(ebb);
-        let param = bcx.ebb_params(ebb)[0];
+        bcx.switch_to_block(block);
+        bcx.append_block_params_for_function_params(block);
+        let param = bcx.block_params(block)[0];
         let cst = bcx.ins().iconst(types::I32, 37);
         let add = bcx.ins().iadd(cst, param);
         bcx.ins().return_(&[add]);
@@ -45,9 +45,9 @@ fn main() {
     ctx.func.name = ExternalName::user(0, func_b.as_u32());
     {
         let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
-        let ebb = bcx.create_ebb();
+        let block = bcx.create_block();
 
-        bcx.switch_to_block(ebb);
+        bcx.switch_to_block(block);
         let local_func = module.declare_func_in_func(func_a, &mut bcx.func);
         let arg = bcx.ins().iconst(types::I32, 5);
         let call = bcx.ins().call(local_func, &[arg]);
diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index 919a4ffe3f..45c7d3d260 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -582,7 +582,7 @@ impl SimpleJITRelocSink {
 }
 
 impl RelocSink for SimpleJITRelocSink {
-    fn reloc_ebb(&mut self, _offset: CodeOffset, _reloc: Reloc, _ebb_offset: CodeOffset) {
+    fn reloc_block(&mut self, _offset: CodeOffset, _reloc: Reloc, _block_offset: CodeOffset) {
         unimplemented!();
     }
 
diff --git a/cranelift/simplejit/tests/basic.rs b/cranelift/simplejit/tests/basic.rs
index a3932b1d6a..0e8ea0aa9d 100644
--- a/cranelift/simplejit/tests/basic.rs
+++ b/cranelift/simplejit/tests/basic.rs
@@ -41,8 +41,8 @@ fn define_simple_function(module: &mut Module) -> FuncId {
     let mut func_ctx = FunctionBuilderContext::new();
     {
         let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
-        let ebb = bcx.create_ebb();
-        bcx.switch_to_block(ebb);
+        let block = bcx.create_block();
+        bcx.switch_to_block(block);
         bcx.ins().return_(&[]);
     }
 
@@ -90,16 +90,16 @@ fn switch_error() {
     let mut func_ctx = FunctionBuilderContext::new();
     {
         let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut func, &mut func_ctx);
-        let start = bcx.create_ebb();
-        let bb0 = bcx.create_ebb();
-        let bb1 = bcx.create_ebb();
-        let bb2 = bcx.create_ebb();
-        let bb3 = bcx.create_ebb();
+        let start = bcx.create_block();
+        let bb0 = bcx.create_block();
+        let bb1 = bcx.create_block();
+        let bb2 = bcx.create_block();
+        let bb3 = bcx.create_block();
         println!("{} {} {} {} {}", start, bb0, bb1, bb2, bb3);
 
         bcx.declare_var(Variable::new(0), types::I32);
         bcx.declare_var(Variable::new(1), types::I32);
-        let in_val = bcx.append_ebb_param(start, types::I32);
+        let in_val = bcx.append_block_param(start, types::I32);
         bcx.switch_to_block(start);
         bcx.def_var(Variable::new(0), in_val);
         bcx.ins().jump(bb0, &[]);
@@ -168,8 +168,8 @@ fn libcall_function() {
     let mut func_ctx = FunctionBuilderContext::new();
     {
         let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
-        let ebb = bcx.create_ebb();
-        bcx.switch_to_block(ebb);
+        let block = bcx.create_block();
+        bcx.switch_to_block(block);
 
         let int = module.target_config().pointer_type();
         let zero = bcx.ins().iconst(I16, 0);
diff --git a/cranelift/src/bugpoint.rs b/cranelift/src/bugpoint.rs
index 5d075e0001..b409143be4 100644
--- a/cranelift/src/bugpoint.rs
+++ b/cranelift/src/bugpoint.rs
@@ -6,8 +6,8 @@ use cranelift_codegen::cursor::{Cursor, FuncCursor};
 use cranelift_codegen::flowgraph::ControlFlowGraph;
 use cranelift_codegen::ir::types::{F32, F64};
 use cranelift_codegen::ir::{
-    self, Ebb, FuncRef, Function, GlobalValueData, Inst, InstBuilder, InstructionData, StackSlots,
-    TrapCode,
+    self, Block, FuncRef, Function, GlobalValueData, Inst, InstBuilder, InstructionData,
+    StackSlots, TrapCode,
 };
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::Context;
@@ -46,17 +46,17 @@ pub fn run(
     std::env::set_var("RUST_BACKTRACE", "0"); // Disable backtraces to reduce verbosity
 
     for (func, _) in test_file.functions {
-        let (orig_ebb_count, orig_inst_count) = (ebb_count(&func), inst_count(&func));
+        let (orig_block_count, orig_inst_count) = (block_count(&func), inst_count(&func));
 
         match reduce(isa, func, verbose) {
             Ok((func, crash_msg)) => {
                 println!("Crash message: {}", crash_msg);
                 println!("\n{}", func);
                 println!(
-                    "{} ebbs {} insts -> {} ebbs {} insts",
-                    orig_ebb_count,
+                    "{} blocks {} insts -> {} blocks {} insts",
+                    orig_block_count,
                     orig_inst_count,
-                    ebb_count(&func),
+                    block_count(&func),
                     inst_count(&func)
                 );
             }
@@ -68,7 +68,7 @@ pub fn run(
 }
 
 enum ProgressStatus {
-    /// The mutation raised or reduced the amount of instructions or ebbs.
+    /// The mutation raised or reduced the amount of instructions or blocks.
     ExpandedOrShrinked,
 
     /// The mutation only changed an instruction. Performing another round of mutations may only
@@ -92,16 +92,16 @@ trait Mutator {
 
 /// Try to remove instructions.
 struct RemoveInst {
-    ebb: Ebb,
+    block: Block,
     inst: Inst,
 }
 
 impl RemoveInst {
     fn new(func: &Function) -> Self {
-        let first_ebb = func.layout.entry_block().unwrap();
-        let first_inst = func.layout.first_inst(first_ebb).unwrap();
+        let first_block = func.layout.entry_block().unwrap();
+        let first_inst = func.layout.first_inst(first_block).unwrap();
         Self {
-            ebb: first_ebb,
+            block: first_block,
             inst: first_inst,
         }
     }
@@ -117,12 +117,12 @@ impl Mutator for RemoveInst {
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, ProgressStatus)> {
-        next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst).map(|(prev_ebb, prev_inst)| {
+        next_inst_ret_prev(&func, &mut self.block, &mut self.inst).map(|(prev_block, prev_inst)| {
             func.layout.remove_inst(prev_inst);
-            let msg = if func.layout.ebb_insts(prev_ebb).next().is_none() {
-                // Make sure empty ebbs are removed, as `next_inst_ret_prev` depends on non empty ebbs
-                func.layout.remove_ebb(prev_ebb);
-                format!("Remove inst {} and empty ebb {}", prev_inst, prev_ebb)
+            let msg = if func.layout.block_insts(prev_block).next().is_none() {
+                // Make sure empty blocks are removed, as `next_inst_ret_prev` depends on non empty blocks
+                func.layout.remove_block(prev_block);
+                format!("Remove inst {} and empty block {}", prev_inst, prev_block)
             } else {
                 format!("Remove inst {}", prev_inst)
             };
@@ -133,16 +133,16 @@ impl Mutator for RemoveInst {
 
 /// Try to replace instructions with `iconst` or `fconst`.
 struct ReplaceInstWithConst {
-    ebb: Ebb,
+    block: Block,
     inst: Inst,
 }
 
 impl ReplaceInstWithConst {
     fn new(func: &Function) -> Self {
-        let first_ebb = func.layout.entry_block().unwrap();
-        let first_inst = func.layout.first_inst(first_ebb).unwrap();
+        let first_block = func.layout.entry_block().unwrap();
+        let first_inst = func.layout.first_inst(first_block).unwrap();
         Self {
-            ebb: first_ebb,
+            block: first_block,
             inst: first_inst,
         }
     }
@@ -174,71 +174,73 @@ impl Mutator for ReplaceInstWithConst {
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, ProgressStatus)> {
-        next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst).map(|(_prev_ebb, prev_inst)| {
-            let num_results = func.dfg.inst_results(prev_inst).len();
+        next_inst_ret_prev(&func, &mut self.block, &mut self.inst).map(
+            |(_prev_block, prev_inst)| {
+                let num_results = func.dfg.inst_results(prev_inst).len();
 
-            let opcode = func.dfg[prev_inst].opcode();
-            if num_results == 0
-                || opcode == ir::Opcode::Iconst
-                || opcode == ir::Opcode::F32const
-                || opcode == ir::Opcode::F64const
-            {
-                return (func, format!(""), ProgressStatus::Skip);
-            }
+                let opcode = func.dfg[prev_inst].opcode();
+                if num_results == 0
+                    || opcode == ir::Opcode::Iconst
+                    || opcode == ir::Opcode::F32const
+                    || opcode == ir::Opcode::F64const
+                {
+                    return (func, format!(""), ProgressStatus::Skip);
+                }
 
-            if num_results == 1 {
-                let ty = func.dfg.value_type(func.dfg.first_result(prev_inst));
-                let new_inst_name = Self::const_for_type(func.dfg.replace(prev_inst), ty);
-                return (
+                if num_results == 1 {
+                    let ty = func.dfg.value_type(func.dfg.first_result(prev_inst));
+                    let new_inst_name = Self::const_for_type(func.dfg.replace(prev_inst), ty);
+                    return (
+                        func,
+                        format!("Replace inst {} with {}.", prev_inst, new_inst_name),
+                        ProgressStatus::Changed,
+                    );
+                }
+
+                // At least 2 results. Replace each instruction with as many const instructions as
+                // there are results.
+                let mut pos = FuncCursor::new(&mut func).at_inst(prev_inst);
+
+                // Copy result SSA names into our own vector; otherwise we couldn't mutably borrow pos
+                // in the loop below.
+                let results = pos.func.dfg.inst_results(prev_inst).to_vec();
+
+                // Detach results from the previous instruction, since we're going to reuse them.
+                pos.func.dfg.clear_results(prev_inst);
+
+                let mut inst_names = Vec::new();
+                for r in results {
+                    let ty = pos.func.dfg.value_type(r);
+                    let builder = pos.ins().with_results([Some(r)]);
+                    let new_inst_name = Self::const_for_type(builder, ty);
+                    inst_names.push(new_inst_name);
+                }
+
+                // Remove the instruction.
+                assert_eq!(pos.remove_inst(), prev_inst);
+
+                (
                     func,
-                    format!("Replace inst {} with {}.", prev_inst, new_inst_name),
-                    ProgressStatus::Changed,
-                );
-            }
-
-            // At least 2 results. Replace each instruction with as many const instructions as
-            // there are results.
-            let mut pos = FuncCursor::new(&mut func).at_inst(prev_inst);
-
-            // Copy result SSA names into our own vector; otherwise we couldn't mutably borrow pos
-            // in the loop below.
-            let results = pos.func.dfg.inst_results(prev_inst).to_vec();
-
-            // Detach results from the previous instruction, since we're going to reuse them.
-            pos.func.dfg.clear_results(prev_inst);
-
-            let mut inst_names = Vec::new();
-            for r in results {
-                let ty = pos.func.dfg.value_type(r);
-                let builder = pos.ins().with_results([Some(r)]);
-                let new_inst_name = Self::const_for_type(builder, ty);
-                inst_names.push(new_inst_name);
-            }
-
-            // Remove the instruction.
-            assert_eq!(pos.remove_inst(), prev_inst);
-
-            (
-                func,
-                format!("Replace inst {} with {}", prev_inst, inst_names.join(" / ")),
-                ProgressStatus::ExpandedOrShrinked,
-            )
-        })
+                    format!("Replace inst {} with {}", prev_inst, inst_names.join(" / ")),
+                    ProgressStatus::ExpandedOrShrinked,
+                )
+            },
+        )
     }
 }
 
 /// Try to replace instructions with `trap`.
 struct ReplaceInstWithTrap {
-    ebb: Ebb,
+    block: Block,
     inst: Inst,
 }
 
 impl ReplaceInstWithTrap {
     fn new(func: &Function) -> Self {
-        let first_ebb = func.layout.entry_block().unwrap();
-        let first_inst = func.layout.first_inst(first_ebb).unwrap();
+        let first_block = func.layout.entry_block().unwrap();
+        let first_inst = func.layout.first_inst(first_block).unwrap();
         Self {
-            ebb: first_ebb,
+            block: first_block,
             inst: first_inst,
         }
     }
@@ -254,54 +256,56 @@ impl Mutator for ReplaceInstWithTrap {
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, ProgressStatus)> {
-        next_inst_ret_prev(&func, &mut self.ebb, &mut self.inst).map(|(_prev_ebb, prev_inst)| {
-            let status = if func.dfg[prev_inst].opcode() == ir::Opcode::Trap {
-                ProgressStatus::Skip
-            } else {
-                func.dfg.replace(prev_inst).trap(TrapCode::User(0));
-                ProgressStatus::Changed
-            };
-            (
-                func,
-                format!("Replace inst {} with trap", prev_inst),
-                status,
-            )
-        })
+        next_inst_ret_prev(&func, &mut self.block, &mut self.inst).map(
+            |(_prev_block, prev_inst)| {
+                let status = if func.dfg[prev_inst].opcode() == ir::Opcode::Trap {
+                    ProgressStatus::Skip
+                } else {
+                    func.dfg.replace(prev_inst).trap(TrapCode::User(0));
+                    ProgressStatus::Changed
+                };
+                (
+                    func,
+                    format!("Replace inst {} with trap", prev_inst),
+                    status,
+                )
+            },
+        )
     }
 }
 
-/// Try to remove an ebb.
-struct RemoveEbb {
-    ebb: Ebb,
+/// Try to remove an block.
+struct RemoveBlock {
+    block: Block,
 }
 
-impl RemoveEbb {
+impl RemoveBlock {
     fn new(func: &Function) -> Self {
         Self {
-            ebb: func.layout.entry_block().unwrap(),
+            block: func.layout.entry_block().unwrap(),
         }
     }
 }
 
-impl Mutator for RemoveEbb {
+impl Mutator for RemoveBlock {
     fn name(&self) -> &'static str {
-        "remove ebb"
+        "remove block"
     }
 
     fn mutation_count(&self, func: &Function) -> usize {
-        ebb_count(func)
+        block_count(func)
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, ProgressStatus)> {
-        func.layout.next_ebb(self.ebb).map(|next_ebb| {
-            self.ebb = next_ebb;
-            while let Some(inst) = func.layout.last_inst(self.ebb) {
+        func.layout.next_block(self.block).map(|next_block| {
+            self.block = next_block;
+            while let Some(inst) = func.layout.last_inst(self.block) {
                 func.layout.remove_inst(inst);
             }
-            func.layout.remove_ebb(self.ebb);
+            func.layout.remove_block(self.block);
             (
                 func,
-                format!("Remove ebb {}", next_ebb),
+                format!("Remove block {}", next_block),
                 ProgressStatus::ExpandedOrShrinked,
             )
         })
@@ -333,8 +337,8 @@ impl Mutator for RemoveUnusedEntities {
         let name = match self.kind {
             0 => {
                 let mut ext_func_usage_map = HashMap::new();
-                for ebb in func.layout.ebbs() {
-                    for inst in func.layout.ebb_insts(ebb) {
+                for block in func.layout.blocks() {
+                    for inst in func.layout.block_insts(block) {
                         match func.dfg[inst] {
                             // Add new cases when there are new instruction formats taking a `FuncRef`.
                             InstructionData::Call { func_ref, .. }
@@ -383,8 +387,8 @@ impl Mutator for RemoveUnusedEntities {
                 }
 
                 let mut signatures_usage_map = HashMap::new();
-                for ebb in func.layout.ebbs() {
-                    for inst in func.layout.ebb_insts(ebb) {
+                for block in func.layout.blocks() {
+                    for inst in func.layout.block_insts(block) {
                         // Add new cases when there are new instruction formats taking a `SigRef`.
                         if let InstructionData::CallIndirect { sig_ref, .. } = func.dfg[inst] {
                             signatures_usage_map
@@ -431,8 +435,8 @@ impl Mutator for RemoveUnusedEntities {
             }
             2 => {
                 let mut stack_slot_usage_map = HashMap::new();
-                for ebb in func.layout.ebbs() {
-                    for inst in func.layout.ebb_insts(ebb) {
+                for block in func.layout.blocks() {
+                    for inst in func.layout.block_insts(block) {
                         match func.dfg[inst] {
                             // Add new cases when there are new instruction formats taking a `StackSlot`.
                             InstructionData::StackLoad { stack_slot, .. }
@@ -490,8 +494,8 @@ impl Mutator for RemoveUnusedEntities {
             }
             3 => {
                 let mut global_value_usage_map = HashMap::new();
-                for ebb in func.layout.ebbs() {
-                    for inst in func.layout.ebb_insts(ebb) {
+                for block in func.layout.blocks() {
+                    for inst in func.layout.block_insts(block) {
                         // Add new cases when there are new instruction formats taking a `GlobalValue`.
                         if let InstructionData::UnaryGlobalValue { global_value, .. } =
                             func.dfg[inst]
@@ -545,15 +549,15 @@ impl Mutator for RemoveUnusedEntities {
 }
 
 struct MergeBlocks {
-    ebb: Ebb,
-    prev_ebb: Option,
+    block: Block,
+    prev_block: Option,
 }
 
 impl MergeBlocks {
     fn new(func: &Function) -> Self {
         Self {
-            ebb: func.layout.entry_block().unwrap(),
-            prev_ebb: None,
+            block: func.layout.entry_block().unwrap(),
+            prev_block: None,
         }
     }
 }
@@ -564,30 +568,30 @@ impl Mutator for MergeBlocks {
     }
 
     fn mutation_count(&self, func: &Function) -> usize {
-        // N ebbs may result in at most N-1 merges.
-        ebb_count(func) - 1
+        // N blocks may result in at most N-1 merges.
+        block_count(func) - 1
     }
 
     fn mutate(&mut self, mut func: Function) -> Option<(Function, String, ProgressStatus)> {
-        let ebb = match func.layout.next_ebb(self.ebb) {
-            Some(ebb) => ebb,
+        let block = match func.layout.next_block(self.block) {
+            Some(block) => block,
             None => return None,
         };
 
-        self.ebb = ebb;
+        self.block = block;
 
         let mut cfg = ControlFlowGraph::new();
         cfg.compute(&func);
 
-        if cfg.pred_iter(ebb).count() != 1 {
+        if cfg.pred_iter(block).count() != 1 {
             return Some((
                 func,
-                format!("did nothing for {}", ebb),
+                format!("did nothing for {}", block),
                 ProgressStatus::Skip,
             ));
         }
 
-        let pred = cfg.pred_iter(ebb).next().unwrap();
+        let pred = cfg.pred_iter(block).next().unwrap();
 
         // If the branch instruction that lead us to this block is preceded by another branch
         // instruction, then we have a conditional jump sequence that we should not break by
@@ -596,86 +600,90 @@ impl Mutator for MergeBlocks {
             if func.dfg[pred_pred_inst].opcode().is_branch() {
                 return Some((
                     func,
-                    format!("did nothing for {}", ebb),
+                    format!("did nothing for {}", block),
                     ProgressStatus::Skip,
                 ));
             }
         }
 
-        assert!(func.dfg.ebb_params(ebb).len() == func.dfg.inst_variable_args(pred.inst).len());
+        assert!(func.dfg.block_params(block).len() == func.dfg.inst_variable_args(pred.inst).len());
 
-        // If there were any EBB parameters in ebb, then the last instruction in pred will
-        // fill these parameters. Make the EBB params aliases of the terminator arguments.
-        for (ebb_param, arg) in func
+        // If there were any block parameters in block, then the last instruction in pred will
+        // fill these parameters. Make the block params aliases of the terminator arguments.
+        for (block_param, arg) in func
             .dfg
-            .detach_ebb_params(ebb)
+            .detach_block_params(block)
             .as_slice(&func.dfg.value_lists)
             .iter()
             .cloned()
             .zip(func.dfg.inst_variable_args(pred.inst).iter().cloned())
             .collect::>()
         {
-            if ebb_param != arg {
-                func.dfg.change_to_alias(ebb_param, arg);
+            if block_param != arg {
+                func.dfg.change_to_alias(block_param, arg);
             }
         }
 
-        // Remove the terminator branch to the current EBB.
+        // Remove the terminator branch to the current block.
         func.layout.remove_inst(pred.inst);
 
         // Move all the instructions to the predecessor.
-        while let Some(inst) = func.layout.first_inst(ebb) {
+        while let Some(inst) = func.layout.first_inst(block) {
             func.layout.remove_inst(inst);
-            func.layout.append_inst(inst, pred.ebb);
+            func.layout.append_inst(inst, pred.block);
         }
 
-        // Remove the predecessor EBB.
-        func.layout.remove_ebb(ebb);
+        // Remove the predecessor block.
+        func.layout.remove_block(block);
 
-        // Record the previous EBB: if we caused a crash (as signaled by a call to did_crash), then
-        // we'll start back to this EBB.
-        self.prev_ebb = Some(pred.ebb);
+        // Record the previous block: if we caused a crash (as signaled by a call to did_crash), then
+        // we'll start back to this block.
+        self.prev_block = Some(pred.block);
 
         Some((
             func,
-            format!("merged {} and {}", pred.ebb, ebb),
+            format!("merged {} and {}", pred.block, block),
             ProgressStatus::ExpandedOrShrinked,
         ))
     }
 
     fn did_crash(&mut self) {
-        self.ebb = self.prev_ebb.unwrap();
+        self.block = self.prev_block.unwrap();
     }
 }
 
-fn next_inst_ret_prev(func: &Function, ebb: &mut Ebb, inst: &mut Inst) -> Option<(Ebb, Inst)> {
-    let prev = (*ebb, *inst);
+fn next_inst_ret_prev(
+    func: &Function,
+    block: &mut Block,
+    inst: &mut Inst,
+) -> Option<(Block, Inst)> {
+    let prev = (*block, *inst);
     if let Some(next_inst) = func.layout.next_inst(*inst) {
         *inst = next_inst;
         return Some(prev);
     }
-    if let Some(next_ebb) = func.layout.next_ebb(*ebb) {
-        *ebb = next_ebb;
-        *inst = func.layout.first_inst(*ebb).expect("no inst");
+    if let Some(next_block) = func.layout.next_block(*block) {
+        *block = next_block;
+        *inst = func.layout.first_inst(*block).expect("no inst");
         return Some(prev);
     }
     None
 }
 
-fn ebb_count(func: &Function) -> usize {
-    func.layout.ebbs().count()
+fn block_count(func: &Function) -> usize {
+    func.layout.blocks().count()
 }
 
 fn inst_count(func: &Function) -> usize {
     func.layout
-        .ebbs()
-        .map(|ebb| func.layout.ebb_insts(ebb).count())
+        .blocks()
+        .map(|block| func.layout.block_insts(block).count())
         .sum()
 }
 
 fn resolve_aliases(func: &mut Function) {
-    for ebb in func.layout.ebbs() {
-        for inst in func.layout.ebb_insts(ebb) {
+    for block in func.layout.blocks() {
+        for inst in func.layout.block_insts(block) {
             func.dfg.resolve_aliases_in_arguments(inst);
         }
     }
@@ -713,7 +721,7 @@ fn reduce(
                 0 => Box::new(RemoveInst::new(&func)),
                 1 => Box::new(ReplaceInstWithConst::new(&func)),
                 2 => Box::new(ReplaceInstWithTrap::new(&func)),
-                3 => Box::new(RemoveEbb::new(&func)),
+                3 => Box::new(RemoveBlock::new(&func)),
                 4 => Box::new(RemoveUnusedEntities::new()),
                 5 => Box::new(MergeBlocks::new(&func)),
                 _ => break,
@@ -775,10 +783,10 @@ fn reduce(
         }
 
         progress_bar.println(format!(
-            "After pass {}, remaining insts/ebbs: {}/{} ({})",
+            "After pass {}, remaining insts/blocks: {}/{} ({})",
             pass_idx,
             inst_count(&func),
-            ebb_count(&func),
+            block_count(&func),
             if should_keep_reducing {
                 "will keep reducing"
             } else {
@@ -861,7 +869,7 @@ impl<'a> CrashCheckContext<'a> {
             Ok(None) => {}
             // The verifier panicked. Compiling it will probably give the same panic.
             // We treat it as succeeding to make it possible to reduce for the actual error.
-            // FIXME prevent verifier panic on removing ebb0.
+            // FIXME prevent verifier panic on removing block0.
             Err(_) => return CheckResult::Succeed,
         }
 
@@ -869,11 +877,13 @@ impl<'a> CrashCheckContext<'a> {
         {
             // For testing purposes we emulate a panic caused by the existence of
             // a `call` instruction.
-            let contains_call = func.layout.ebbs().any(|ebb| {
-                func.layout.ebb_insts(ebb).any(|inst| match func.dfg[inst] {
-                    InstructionData::Call { .. } => true,
-                    _ => false,
-                })
+            let contains_call = func.layout.blocks().any(|block| {
+                func.layout
+                    .block_insts(block)
+                    .any(|inst| match func.dfg[inst] {
+                        InstructionData::Call { .. } => true,
+                        _ => false,
+                    })
             });
             if contains_call {
                 return CheckResult::Crash("test crash".to_string());
@@ -934,9 +944,9 @@ mod tests {
             assert_eq!(crash_msg, "test crash");
 
             assert_eq!(
-                ebb_count(&func_reduced_twice),
-                ebb_count(&reduced_func),
-                "reduction wasn't maximal for ebbs"
+                block_count(&func_reduced_twice),
+                block_count(&reduced_func),
+                "reduction wasn't maximal for blocks"
             );
             assert_eq!(
                 inst_count(&func_reduced_twice),
diff --git a/cranelift/src/disasm.rs b/cranelift/src/disasm.rs
index 5e6d3bbe33..ba80e3d57f 100644
--- a/cranelift/src/disasm.rs
+++ b/cranelift/src/disasm.rs
@@ -18,14 +18,19 @@ impl PrintRelocs {
 }
 
 impl binemit::RelocSink for PrintRelocs {
-    fn reloc_ebb(
+    fn reloc_block(
         &mut self,
         where_: binemit::CodeOffset,
         r: binemit::Reloc,
         offset: binemit::CodeOffset,
     ) {
         if self.flag_print {
-            writeln!(&mut self.text, "reloc_ebb: {} {} at {}", r, offset, where_).unwrap();
+            writeln!(
+                &mut self.text,
+                "reloc_block: {} {} at {}",
+                r, offset, where_
+            )
+            .unwrap();
         }
     }
 
diff --git a/cranelift/src/run.rs b/cranelift/src/run.rs
index 7825b1852d..0f553c8702 100644
--- a/cranelift/src/run.rs
+++ b/cranelift/src/run.rs
@@ -119,7 +119,7 @@ mod test {
         let code = String::from(
             "
             function %test() -> b8 {
-            ebb0:
+            block0:
                 nop
                 v1 = bconst.b8 true
                 return v1
diff --git a/cranelift/tests/bugpoint_test.clif b/cranelift/tests/bugpoint_test.clif
index 772b36d58e..b2e9acc37e 100644
--- a/cranelift/tests/bugpoint_test.clif
+++ b/cranelift/tests/bugpoint_test.clif
@@ -288,7 +288,7 @@ function u0:0(i64, i64, i64) system_v {
     fn103 = u0:13 sig103
     fn104 = u0:95 sig104
 
-ebb0(v0: i64, v1: i64, v2: i64):
+block0(v0: i64, v1: i64, v2: i64):
     v113 -> v1
     v124 -> v1
     v136 -> v1
@@ -411,9 +411,9 @@ ebb0(v0: i64, v1: i64, v2: i64):
     v110 = stack_addr.i64 ss105
     v111 = stack_addr.i64 ss106
     v112 = stack_addr.i64 ss107
-    jump ebb1
+    jump block1
 
-ebb1:
+block1:
     v114 = load.i64 v113
     v115 = iconst.i64 0
     v116 = icmp ugt v114, v115
@@ -422,15 +422,15 @@ ebb1:
     v119 = icmp_imm eq v118, 0
     v120 = bint.i8 v119
     v121 = uextend.i32 v120
-    brz v121, ebb3
-    jump ebb2
+    brz v121, block3
+    jump block2
 
-ebb2:
+block2:
     v122 = global_value.i64 gv0
     v123 = global_value.i64 gv1
     trap user65535
 
-ebb3:
+block3:
     v125 = iadd_imm.i64 v124, 8
     v126 = load.i64 v125
     v127 = iconst.i64 0
@@ -440,15 +440,15 @@ ebb3:
     v131 = icmp_imm eq v130, 0
     v132 = bint.i8 v131
     v133 = uextend.i32 v132
-    brz v133, ebb5
-    jump ebb4
+    brz v133, block5
+    jump block4
 
-ebb4:
+block4:
     v134 = global_value.i64 gv2
     v135 = global_value.i64 gv3
     trap user65535
 
-ebb5:
+block5:
     v137 = iadd_imm.i64 v136, 16
     v138 = load.i64 v137+42
     v139 = iconst.i64 0
@@ -458,71 +458,71 @@ ebb5:
     v143 = icmp_imm eq v142, 0
     v144 = bint.i8 v143
     v145 = uextend.i32 v144
-    brz v145, ebb7
-    jump ebb6
+    brz v145, block7
+    jump block6
 
-ebb6:
+block6:
     v146 = global_value.i64 gv4
     v147 = global_value.i64 gv5
     trap user65535
 
-ebb7:
+block7:
     v149 = load.i64 v148
     v150 = iadd_imm.i64 v148, 16
     v151 = load.i64 v150
     call fn6(v7, v149, v151)
-    jump ebb8
+    jump block8
 
-ebb8:
+block8:
     v152 = call fn7(v7)
-    jump ebb9
+    jump block9
 
-ebb9:
+block9:
     v153 = load.i8 v6
     v154 = uextend.i32 v153
     v155 = icmp_imm eq v154, 0
     v156 = bint.i8 v155
     v157 = uextend.i32 v156
-    brz v157, ebb11
-    jump ebb10
+    brz v157, block11
+    jump block10
 
-ebb10:
+block10:
     v158 = global_value.i64 gv6
     v159 = global_value.i64 gv7
     trap user65535
 
-ebb11:
+block11:
     v161 = load.i64 v160
     v162 = iadd_imm.i64 v160, 8
     v163 = load.i64 v162
     call fn10(v9, v161, v163)
-    jump ebb12
+    jump block12
 
-ebb12:
+block12:
     v164 = call fn11(v9)
-    jump ebb13
+    jump block13
 
-ebb13:
+block13:
     v165 = load.i8 v8
     v166 = uextend.i32 v165
     v167 = icmp_imm eq v166, 0
     v168 = bint.i8 v167
     v169 = uextend.i32 v168
-    brz v169, ebb15
-    jump ebb14
+    brz v169, block15
+    jump block14
 
-ebb14:
+block14:
     v170 = global_value.i64 gv8
     v171 = global_value.i64 gv9
     trap user65535
 
-ebb15:
+block15:
     v172 = load.i64 aligned v3
     v173 = load.i64 aligned v3+8
     v174 = call fn14(v11)
-    jump ebb16
+    jump block16
 
-ebb16:
+block16:
     v175 = iconst.i64 17
     v176 = load.i64 v10
     v177 = icmp uge v176, v175
@@ -531,15 +531,15 @@ ebb16:
     v180 = icmp_imm eq v179, 0
     v181 = bint.i8 v180
     v182 = uextend.i32 v181
-    brz v182, ebb18
-    jump ebb17
+    brz v182, block18
+    jump block17
 
-ebb17:
+block17:
     v183 = global_value.i64 gv10
     v184 = global_value.i64 gv11
     trap user65535
 
-ebb18:
+block18:
     v186 = load.i64 v185
     v187 = iadd_imm.i64 v185, 16
     v188 = load.i64 v187
@@ -552,14 +552,14 @@ ebb18:
     v195 = iadd_imm.i64 v12, 8
     v196 = load.i8 v195
     v197 = uextend.i32 v196
-    brz v197, ebb19
-    jump ebb164
+    brz v197, block19
+    jump block164
 
-ebb164:
+block164:
     v198 = global_value.i64 gv12
     trap user0
 
-ebb19:
+block19:
     v199 = load.i64 v12
         v213 -> v199
     v200 = iconst.i64 1
@@ -573,14 +573,14 @@ ebb19:
     v208 = iadd_imm.i64 v13, 8
     v209 = load.i8 v208
     v210 = uextend.i32 v209
-    brz v210, ebb20
-    jump ebb163
+    brz v210, block20
+    jump block163
 
-ebb163:
+block163:
     v211 = global_value.i64 gv13
     trap user0
 
-ebb20:
+block20:
     v212 = load.i64 v13
     v214 = icmp.i64 ult v213, v212
     v215 = bint.i8 v214
@@ -588,15 +588,15 @@ ebb20:
     v217 = icmp_imm eq v216, 0
     v218 = bint.i8 v217
     v219 = uextend.i32 v218
-    brz v219, ebb22
-    jump ebb21
+    brz v219, block22
+    jump block21
 
-ebb21:
+block21:
     v220 = global_value.i64 gv14
     v221 = global_value.i64 gv15
     trap user65535
 
-ebb22:
+block22:
     v223 = load.i64 v222
     v224 = iadd_imm.i64 v222, 16
     v225 = load.i64 v224
@@ -609,22 +609,22 @@ ebb22:
     v232 = iadd_imm.i64 v16, 8
     v233 = load.i8 v232
     v234 = uextend.i32 v233
-    brz v234, ebb23
-    jump ebb162
+    brz v234, block23
+    jump block162
 
-ebb162:
+block162:
     v235 = global_value.i64 gv16
     trap user0
 
-ebb23:
+block23:
     v236 = load.i64 v16
     v238 = iadd_imm.i64 v237, 24
     v239 = load.i16 v238
     v240 = iadd_imm.i64 v15, 8
     call fn22(v14, v15)
-    jump ebb24
+    jump block24
 
-ebb24:
+block24:
     v242 = load.i64 v241
     v243 = iadd_imm.i64 v241, 8
     v244 = load.i64 v243
@@ -637,14 +637,14 @@ ebb24:
     v251 = iadd_imm.i64 v19, 8
     v252 = load.i8 v251
     v253 = uextend.i32 v252
-    brz v253, ebb25
-    jump ebb161
+    brz v253, block25
+    jump block161
 
-ebb161:
+block161:
     v254 = global_value.i64 gv17
     trap user0
 
-ebb25:
+block25:
     v255 = load.i64 v19
     v257 = iadd_imm.i64 v256, 24
     v258 = load.i16 v257
@@ -652,9 +652,9 @@ ebb25:
     v260 = iadd_imm.i64 v14, 8
     v261 = load.i16 v260
     call fn24(v17, v18, v261)
-    jump ebb26
+    jump block26
 
-ebb26:
+block26:
     v263 = load.i64 v262
     v264 = iadd_imm.i64 v262, 24
     v265 = load.i16 v264
@@ -662,9 +662,9 @@ ebb26:
     v267 = iadd_imm.i64 v14, 8
     v268 = load.i16 v267
     call fn25(v20, v21, v268)
-    jump ebb27
+    jump block27
 
-ebb27:
+block27:
     v269 = iadd_imm.i64 v14, 8
     v270 = load.i16 v269
     v271 = iconst.i16 -60
@@ -676,14 +676,14 @@ ebb27:
     v277 = iadd_imm.i64 v24, 2
     v278 = load.i8 v277
     v279 = uextend.i32 v278
-    brz v279, ebb28
-    jump ebb160
+    brz v279, block28
+    jump block160
 
-ebb160:
+block160:
     v280 = global_value.i64 gv18
     trap user0
 
-ebb28:
+block28:
     v281 = load.i16 v24
     v282 = iconst.i16 64
     v283 = isub v281, v282
@@ -694,14 +694,14 @@ ebb28:
     v288 = iadd_imm.i64 v25, 2
     v289 = load.i8 v288
     v290 = uextend.i32 v289
-    brz v290, ebb29
-    jump ebb159
+    brz v290, block29
+    jump block159
 
-ebb159:
+block159:
     v291 = global_value.i64 gv19
     trap user0
 
-ebb29:
+block29:
     v292 = load.i16 v25
         v317 -> v292
     v293 = iadd_imm.i64 v14, 8
@@ -715,14 +715,14 @@ ebb29:
     v301 = iadd_imm.i64 v26, 2
     v302 = load.i8 v301
     v303 = uextend.i32 v302
-    brz v303, ebb30
-    jump ebb158
+    brz v303, block30
+    jump block158
 
-ebb158:
+block158:
     v304 = global_value.i64 gv20
     trap user0
 
-ebb30:
+block30:
     v305 = load.i16 v26
     v306 = iconst.i16 64
     v307 = isub v305, v306
@@ -733,42 +733,42 @@ ebb30:
     v312 = iadd_imm.i64 v27, 2
     v313 = load.i8 v312
     v314 = uextend.i32 v313
-    brz v314, ebb31
-    jump ebb157
+    brz v314, block31
+    jump block157
 
-ebb157:
+block157:
     v315 = global_value.i64 gv21
     trap user0
 
-ebb31:
+block31:
     v316 = load.i16 v27
     call fn30(v23, v317, v316)
-    jump ebb32
+    jump block32
 
-ebb32:
+block32:
     v318 = load.i16 v23
         v1007 -> v318
     v319 = iadd_imm.i64 v23, 8
     v320 = load.i64 aligned v319
     v321 = load.i64 aligned v319+8
     call fn31(v28, v14, v22)
-    jump ebb33
+    jump block33
 
-ebb33:
+block33:
     call fn32(v29, v17, v22)
-    jump ebb34
+    jump block34
 
-ebb34:
+block34:
     call fn33(v30, v20, v22)
-    jump ebb35
+    jump block35
 
-ebb35:
+block35:
     v322 = iconst.i8 1
     v323 = uextend.i32 v322
-    brz v323, ebb42
-    jump ebb36
+    brz v323, block42
+    jump block36
 
-ebb36:
+block36:
     v324 = iadd_imm.i64 v28, 8
     v325 = iadd_imm.i64 v29, 8
     v326 = iadd_imm.i64 v31, 8
@@ -785,10 +785,10 @@ ebb36:
     v335 = icmp_imm eq v334, 0
     v336 = bint.i8 v335
     v337 = uextend.i32 v336
-    brz v337, ebb38
-    jump ebb37
+    brz v337, block38
+    jump block37
 
-ebb37:
+block37:
     v338 = global_value.i64 gv22
     v339 = iconst.i64 3
     v342 = iadd_imm.i64 v36, 8
@@ -798,17 +798,17 @@ ebb37:
         v347 -> v345
     v346 = func_addr.i64 fn34
     call fn35(v39, v343, v346)
-    jump ebb39
+    jump block39
 
-ebb38:
-    jump ebb42
+block38:
+    jump block42
 
-ebb39:
+block39:
     v348 = func_addr.i64 fn36
     call fn37(v40, v347, v348)
-    jump ebb40
+    jump block40
 
-ebb40:
+block40:
     v349 = iconst.i64 0
     v350 = imul_imm v349, 16
     v351 = iadd.i64 v35, v350
@@ -821,21 +821,21 @@ ebb40:
     v358 = load.i64 aligned v40+8
     v359 = iconst.i64 2
     call fn38(v32, v33, v34)
-    jump ebb41
+    jump block41
 
-ebb41:
+block41:
     v360 = global_value.i64 gv23
     call fn39(v32, v360)
     v361 = global_value.i64 gv24
     trap user65535
 
-ebb42:
+block42:
     v362 = iconst.i8 1
     v363 = uextend.i32 v362
-    brz v363, ebb49(v1007)
-    jump ebb43
+    brz v363, block49(v1007)
+    jump block43
 
-ebb43:
+block43:
     v364 = iadd_imm.i64 v28, 8
     v365 = iadd_imm.i64 v30, 8
     v366 = iadd_imm.i64 v41, 8
@@ -852,10 +852,10 @@ ebb43:
     v375 = icmp_imm eq v374, 0
     v376 = bint.i8 v375
     v377 = uextend.i32 v376
-    brz v377, ebb45
-    jump ebb44
+    brz v377, block45
+    jump block44
 
-ebb44:
+block44:
     v378 = global_value.i64 gv25
     v379 = iconst.i64 3
     v382 = iadd_imm.i64 v46, 8
@@ -865,17 +865,17 @@ ebb44:
         v387 -> v385
     v386 = func_addr.i64 fn41
     call fn42(v49, v383, v386)
-    jump ebb46
+    jump block46
 
-ebb45:
-    jump ebb49(v1007)
+block45:
+    jump block49(v1007)
 
-ebb46:
+block46:
     v388 = func_addr.i64 fn43
     call fn44(v50, v387, v388)
-    jump ebb47
+    jump block47
 
-ebb47:
+block47:
     v389 = iconst.i64 0
     v390 = imul_imm v389, 16
     v391 = iadd.i64 v45, v390
@@ -888,15 +888,15 @@ ebb47:
     v398 = load.i64 aligned v50+8
     v399 = iconst.i64 2
     call fn45(v42, v43, v44)
-    jump ebb48
+    jump block48
 
-ebb48:
+block48:
     v400 = global_value.i64 gv26
     call fn46(v42, v400)
     v401 = global_value.i64 gv27
     trap user65535
 
-ebb49(v1006: i16):
+block49(v1006: i16):
         v486 -> v1006
     v402 = load.i64 v28
     v403 = iconst.i64 1
@@ -909,14 +909,14 @@ ebb49(v1006: i16):
     v410 = iadd_imm.i64 v51, 8
     v411 = load.i8 v410
     v412 = uextend.i32 v411
-    brz v412, ebb50
-    jump ebb156
+    brz v412, block50
+    jump block156
 
-ebb156:
+block156:
     v413 = global_value.i64 gv28
     trap user0
 
-ebb50:
+block50:
     v414 = load.i64 v51
         v439 -> v414
         v452 -> v414
@@ -933,14 +933,14 @@ ebb50:
     v423 = iadd_imm.i64 v52, 8
     v424 = load.i8 v423
     v425 = uextend.i32 v424
-    brz v425, ebb51
-    jump ebb155
+    brz v425, block51
+    jump block155
 
-ebb155:
+block155:
     v426 = global_value.i64 gv29
     trap user0
 
-ebb51:
+block51:
     v427 = load.i64 v52
         v509 -> v427
     v428 = iadd_imm.i64 v28, 8
@@ -950,14 +950,14 @@ ebb51:
     v431 = icmp eq v429, v430
     v432 = bint.i8 v431
     v433 = uextend.i32 v432
-    brz v433, ebb52
-    jump ebb154
+    brz v433, block52
+    jump block154
 
-ebb154:
+block154:
     v434 = global_value.i64 gv30
     trap user0
 
-ebb52:
+block52:
     v436 = iconst.i16 0
     v437 = isub v436, v435
     v438 = sextend.i64 v437
@@ -972,14 +972,14 @@ ebb52:
     v446 = iadd_imm.i64 v53, 8
     v447 = load.i8 v446
     v448 = uextend.i32 v447
-    brz v448, ebb53
-    jump ebb153
+    brz v448, block53
+    jump block153
 
-ebb153:
+block153:
     v449 = global_value.i64 gv31
     trap user0
 
-ebb53:
+block53:
     v450 = load.i64 v53
     v451 = ireduce.i32 v450
         v480 -> v451
@@ -994,14 +994,14 @@ ebb53:
     v461 = iadd_imm.i64 v54, 8
     v462 = load.i8 v461
     v463 = uextend.i32 v462
-    brz v463, ebb54
-    jump ebb152
+    brz v463, block54
+    jump block152
 
-ebb152:
+block152:
     v464 = global_value.i64 gv32
     trap user0
 
-ebb54:
+block54:
     v465 = load.i64 v54
     v466 = iconst.i64 1
     v467 = isub v465, v466
@@ -1013,20 +1013,20 @@ ebb54:
     v473 = iadd_imm.i64 v55, 8
     v474 = load.i8 v473
     v475 = uextend.i32 v474
-    brz v475, ebb55
-    jump ebb151
+    brz v475, block55
+    jump block151
 
-ebb151:
+block151:
     v476 = global_value.i64 gv33
     trap user0
 
-ebb55:
+block55:
     v477 = load.i64 v55
     v479 = band.i64 v478, v477
     call fn54(v56, v480)
-    jump ebb56
+    jump block56
 
-ebb56:
+block56:
     v481 = load.i8 v56
         v548 -> v481
     v482 = iadd_imm.i64 v56, 4
@@ -1042,14 +1042,14 @@ ebb56:
     v492 = iadd_imm.i64 v57, 2
     v493 = load.i8 v492
     v494 = uextend.i32 v493
-    brz v494, ebb57
-    jump ebb150
+    brz v494, block57
+    jump block150
 
-ebb150:
+block150:
     v495 = global_value.i64 gv34
     trap user0
 
-ebb57:
+block57:
     v496 = load.i16 v57
     v497 = iconst.i16 1
     v498 = iadd v496, v497
@@ -1060,14 +1060,14 @@ ebb57:
     v503 = iadd_imm.i64 v58, 2
     v504 = load.i8 v503
     v505 = uextend.i32 v504
-    brz v505, ebb58
-    jump ebb149
+    brz v505, block58
+    jump block149
 
-ebb149:
+block149:
     v506 = global_value.i64 gv35
     trap user0
 
-ebb58:
+block58:
     v507 = load.i16 v58
     v510 = isub.i64 v508, v509
     v511 = iconst.i8 0
@@ -1078,14 +1078,14 @@ ebb58:
     v516 = iadd_imm.i64 v59, 8
     v517 = load.i8 v516
     v518 = uextend.i32 v517
-    brz v518, ebb59
-    jump ebb148
+    brz v518, block59
+    jump block148
 
-ebb148:
+block148:
     v519 = global_value.i64 gv36
     trap user0
 
-ebb59:
+block59:
     v520 = load.i64 v59
         v546 -> v520
     v522 = iconst.i64 1
@@ -1098,14 +1098,14 @@ ebb59:
     v529 = iadd_imm.i64 v60, 8
     v530 = load.i8 v529
     v531 = uextend.i32 v530
-    brz v531, ebb60
-    jump ebb147
+    brz v531, block60
+    jump block147
 
-ebb147:
+block147:
     v532 = global_value.i64 gv37
     trap user0
 
-ebb60:
+block60:
     v533 = load.i64 v60
     v534 = iconst.i64 1
     v535 = isub v533, v534
@@ -1117,20 +1117,20 @@ ebb60:
     v541 = iadd_imm.i64 v61, 8
     v542 = load.i8 v541
     v543 = uextend.i32 v542
-    brz v543, ebb61
-    jump ebb146
+    brz v543, block61
+    jump block146
 
-ebb146:
+block146:
     v544 = global_value.i64 gv38
     trap user0
 
-ebb61:
+block61:
     v545 = load.i64 v61
     v547 = band.i64 v546, v545
     v549 = uextend.i16 v548
-    jump ebb62(v551, v484, v521, v479, v520, v507, v508, v548, v547)
+    jump block62(v551, v484, v521, v479, v520, v507, v508, v548, v547)
 
-ebb62(v552: i32, v1009: i64, v1013: i64, v1016: i64, v1019: i64, v1022: i16, v1025: i64, v1028: i8, v1033: i64):
+block62(v552: i32, v1009: i64, v1013: i64, v1016: i64, v1019: i64, v1022: i16, v1025: i64, v1028: i8, v1033: i64):
         v559 -> v552
         v562 -> v552
         v569 -> v552
@@ -1173,14 +1173,14 @@ ebb62(v552: i32, v1009: i64, v1013: i64, v1016: i64, v1019: i64, v1022: i16, v10
     v555 = icmp eq v553, v554
     v556 = bint.i8 v555
     v557 = uextend.i32 v556
-    brz v557, ebb63
-    jump ebb145
+    brz v557, block63
+    jump block145
 
-ebb145:
+block145:
     v558 = global_value.i64 gv39
     trap user0
 
-ebb63:
+block63:
     v561 = udiv.i32 v559, v560
         v574 -> v561
     v563 = load.i32 v63
@@ -1189,24 +1189,24 @@ ebb63:
     v565 = icmp eq v563, v564
     v566 = bint.i8 v565
     v567 = uextend.i32 v566
-    brz v567, ebb64
-    jump ebb144
+    brz v567, block64
+    jump block144
 
-ebb144:
+block144:
     v568 = global_value.i64 gv40
     trap user0
 
-ebb64:
+block64:
     v571 = urem.i32 v569, v570
         v622 -> v571
         v803 -> v571
         v1011 -> v571
     v572 = iconst.i8 1
     v573 = uextend.i32 v572
-    brz v573, ebb68(v561)
-    jump ebb65
+    brz v573, block68(v561)
+    jump block65
 
-ebb65:
+block65:
     v575 = iconst.i32 10
     v576 = icmp.i32 ult v574, v575
     v577 = bint.i8 v576
@@ -1214,18 +1214,18 @@ ebb65:
     v579 = icmp_imm eq v578, 0
     v580 = bint.i8 v579
     v581 = uextend.i32 v580
-    brz v581, ebb67
-    jump ebb66
+    brz v581, block67
+    jump block66
 
-ebb66:
+block66:
     v582 = global_value.i64 gv41
     v583 = global_value.i64 gv42
     trap user65535
 
-ebb67:
-    jump ebb68(v574)
+block67:
+    jump block68(v574)
 
-ebb68(v584: i32):
+block68(v584: i32):
     v585 = ireduce.i8 v584
     v586 = iconst.i8 48
     v587 = iadd v586, v585
@@ -1236,27 +1236,27 @@ ebb68(v584: i32):
     v592 = iadd_imm.i64 v64, 1
     v593 = load.i8 v592
     v594 = uextend.i32 v593
-    brz v594, ebb69
-    jump ebb143
+    brz v594, block69
+    jump block143
 
-ebb143:
+block143:
     v595 = global_value.i64 gv43
     trap user0
 
-ebb69:
+block69:
     v597 = load.i64 v3
     v598 = load.i64 v3+8
     v599 = icmp.i64 ult v596, v598
     v600 = bint.i8 v599
     v601 = uextend.i32 v600
-    brnz v601, ebb70
-    jump ebb142
+    brnz v601, block70
+    jump block142
 
-ebb142:
+block142:
     v602 = global_value.i64 gv44
     trap user0
 
-ebb70:
+block70:
     v603 = load.i64 v3
     v604 = load.i64 v3+8
     v606 = imul_imm.i64 v605, 1
@@ -1272,14 +1272,14 @@ ebb70:
     v617 = iadd_imm.i64 v65, 8
     v618 = load.i8 v617
     v619 = uextend.i32 v618
-    brz v619, ebb71
-    jump ebb141
+    brz v619, block71
+    jump block141
 
-ebb141:
+block141:
     v620 = global_value.i64 gv45
     trap user0
 
-ebb71:
+block71:
     v621 = load.i64 v65
         v668 -> v621
         v695 -> v621
@@ -1295,14 +1295,14 @@ ebb71:
     v631 = iadd_imm.i64 v66, 8
     v632 = load.i8 v631
     v633 = uextend.i32 v632
-    brz v633, ebb72
-    jump ebb140
+    brz v633, block72
+    jump block140
 
-ebb140:
+block140:
     v634 = global_value.i64 gv46
     trap user0
 
-ebb72:
+block72:
     v635 = load.i64 v66
     v637 = iadd v635, v636
     v638 = iconst.i8 0
@@ -1313,24 +1313,24 @@ ebb72:
     v643 = iadd_imm.i64 v67, 8
     v644 = load.i8 v643
     v645 = uextend.i32 v644
-    brz v645, ebb73
-    jump ebb139
+    brz v645, block73
+    jump block139
 
-ebb139:
+block139:
     v646 = global_value.i64 gv47
     trap user0
 
-ebb73:
+block73:
     v647 = load.i64 v67
         v675 -> v647
         v692 -> v647
     v649 = icmp ult v647, v648
     v650 = bint.i8 v649
     v651 = uextend.i32 v650
-    brz v651, ebb80
-    jump ebb74
+    brz v651, block80
+    jump block74
 
-ebb74:
+block74:
     v652 = load.i32 v63
     v653 = uextend.i64 v652
     v655 = ishl v653, v654
@@ -1342,23 +1342,23 @@ ebb74:
     v661 = iadd_imm.i64 v68, 8
     v662 = load.i8 v661
     v663 = uextend.i32 v662
-    brz v663, ebb75
-    jump ebb138
+    brz v663, block75
+    jump block138
 
-ebb138:
+block138:
     v664 = global_value.i64 gv48
     trap user0
 
-ebb75:
+block75:
     v665 = load.i64 v68
         v690 -> v665
     v666 = load.i64 aligned v3
     v667 = load.i64 aligned v3+8
     v669 = load.i64 v73
     call fn70(v71, v72, v669)
-    jump ebb76
+    jump block76
 
-ebb76:
+block76:
     v670 = load.i64 aligned v71
     v671 = load.i64 aligned v71+8
     v672 = load.i64 aligned v70
@@ -1373,40 +1373,40 @@ ebb76:
     v685 = iadd_imm.i64 v74, 8
     v686 = load.i8 v685
     v687 = uextend.i32 v686
-    brz v687, ebb77
-    jump ebb137
+    brz v687, block77
+    jump block137
 
-ebb137:
+block137:
     v688 = global_value.i64 gv49
     trap user0
 
-ebb77:
+block77:
     v689 = load.i64 v74
     v694 = iconst.i64 1
     call fn72(v0, v69, v691, v692, v693, v689, v690, v694)
-    jump ebb78
+    jump block78
 
-ebb78:
-    jump ebb79
+block78:
+    jump block79
 
-ebb79:
+block79:
     return
 
-ebb80:
+block80:
     v697 = uextend.i64 v696
     v698 = icmp.i64 ugt v695, v697
     v699 = bint.i8 v698
     v700 = uextend.i32 v699
-    brz v700, ebb96
-    jump ebb81
+    brz v700, block96
+    jump block81
 
-ebb81:
+block81:
     v701 = iconst.i8 1
     v702 = uextend.i32 v701
-    brz v702, ebb88
-    jump ebb82
+    brz v702, block88
+    jump block82
 
-ebb82:
+block82:
     v703 = global_value.i64 gv50
     v704 = iadd_imm.i64 v75, 8
     v705 = load.i64 v75
@@ -1422,10 +1422,10 @@ ebb82:
     v713 = icmp_imm eq v712, 0
     v714 = bint.i8 v713
     v715 = uextend.i32 v714
-    brz v715, ebb84
-    jump ebb83
+    brz v715, block84
+    jump block83
 
-ebb83:
+block83:
     v716 = global_value.i64 gv51
     v717 = iconst.i64 3
     v720 = iadd_imm.i64 v80, 8
@@ -1435,17 +1435,17 @@ ebb83:
         v725 -> v723
     v724 = func_addr.i64 fn73
     call fn74(v83, v721, v724)
-    jump ebb85
+    jump block85
 
-ebb84:
-    jump ebb88
+block84:
+    jump block88
 
-ebb85:
+block85:
     v726 = func_addr.i64 fn75
     call fn76(v84, v725, v726)
-    jump ebb86
+    jump block86
 
-ebb86:
+block86:
     v727 = iconst.i64 0
     v728 = imul_imm v727, 16
     v729 = iadd.i64 v79, v728
@@ -1458,21 +1458,21 @@ ebb86:
     v736 = load.i64 aligned v84+8
     v737 = iconst.i64 2
     call fn77(v76, v77, v78)
-    jump ebb87
+    jump block87
 
-ebb87:
+block87:
     v738 = global_value.i64 gv52
     call fn78(v76, v738)
     v739 = global_value.i64 gv53
     trap user65535
 
-ebb88:
+block88:
     v740 = iconst.i8 1
     v741 = uextend.i32 v740
-    brz v741, ebb95(v1030, v1031, v1041, v1046, v1054, v1059)
-    jump ebb89
+    brz v741, block95(v1030, v1031, v1041, v1046, v1054, v1059)
+    jump block89
 
-ebb89:
+block89:
     v742 = global_value.i64 gv54
     v743 = iadd_imm.i64 v85, 8
     v744 = load.i64 v85
@@ -1488,10 +1488,10 @@ ebb89:
     v752 = icmp_imm eq v751, 0
     v753 = bint.i8 v752
     v754 = uextend.i32 v753
-    brz v754, ebb91
-    jump ebb90
+    brz v754, block91
+    jump block90
 
-ebb90:
+block90:
     v755 = global_value.i64 gv55
     v756 = iconst.i64 3
     v759 = iadd_imm.i64 v90, 8
@@ -1501,17 +1501,17 @@ ebb90:
         v764 -> v762
     v763 = func_addr.i64 fn80
     call fn81(v93, v760, v763)
-    jump ebb92
+    jump block92
 
-ebb91:
-    jump ebb95(v1030, v1031, v1041, v1046, v1054, v1059)
+block91:
+    jump block95(v1030, v1031, v1041, v1046, v1054, v1059)
 
-ebb92:
+block92:
     v765 = func_addr.i64 fn82
     call fn83(v94, v764, v765)
-    jump ebb93
+    jump block93
 
-ebb93:
+block93:
     v766 = iconst.i64 0
     v767 = imul_imm v766, 16
     v768 = iadd.i64 v89, v767
@@ -1524,19 +1524,19 @@ ebb93:
     v775 = load.i64 aligned v94+8
     v776 = iconst.i64 2
     call fn84(v86, v87, v88)
-    jump ebb94
+    jump block94
 
-ebb94:
+block94:
     v777 = global_value.i64 gv56
     call fn85(v86, v777)
     v778 = global_value.i64 gv57
     trap user65535
 
-ebb95(v779: i64, v780: i64, v1040: i64, v1045: i64, v1053: i16, v1058: i64):
+block95(v779: i64, v780: i64, v1040: i64, v1045: i64, v1053: i16, v1058: i64):
     v781 = iconst.i64 1
-    jump ebb99(v779, v780, v781, v1040, v1045, v1053, v1058)
+    jump block99(v779, v780, v781, v1040, v1045, v1053, v1058)
 
-ebb96:
+block96:
     v782 = iconst.i16 1
     v783 = load.i16 v62
     v784 = isub v783, v782
@@ -1547,34 +1547,34 @@ ebb96:
     v789 = iadd_imm.i64 v95, 2
     v790 = load.i8 v789
     v791 = uextend.i32 v790
-    brz v791, ebb97
-    jump ebb136
+    brz v791, block97
+    jump block136
 
-ebb136:
+block136:
     v792 = global_value.i64 gv58
     trap user0
 
-ebb97:
+block97:
     v793 = load.i16 aligned v95
     v794 = iconst.i32 10
     v795 = iconst.i32 0
     v796 = icmp eq v794, v795
     v797 = bint.i8 v796
     v798 = uextend.i32 v797
-    brz v798, ebb98
-    jump ebb135
+    brz v798, block98
+    jump block135
 
-ebb135:
+block135:
     v799 = global_value.i64 gv59
     trap user0
 
-ebb98:
+block98:
     v800 = iconst.i32 10
     v801 = load.i32 v63
     v802 = udiv v801, v800
-    jump ebb62(v803, v1010, v1014, v1017, v1020, v1023, v1026, v1029, v1034)
+    jump block62(v803, v1010, v1014, v1017, v1020, v1023, v1026, v1029, v1034)
 
-ebb99(v804: i64, v1035: i64, v1037: i64, v1039: i64, v1044: i64, v1052: i16, v1057: i64):
+block99(v804: i64, v1035: i64, v1037: i64, v1039: i64, v1044: i64, v1052: i16, v1057: i64):
         v817 -> v1035
         v830 -> v1037
         v844 -> v1039
@@ -1603,14 +1603,14 @@ ebb99(v804: i64, v1035: i64, v1037: i64, v1039: i64, v1044: i64, v1052: i16, v10
     v812 = iadd_imm.i64 v96, 8
     v813 = load.i8 v812
     v814 = uextend.i32 v813
-    brz v814, ebb100
-    jump ebb134
+    brz v814, block100
+    jump block134
 
-ebb134:
+block134:
     v815 = global_value.i64 gv60
     trap user0
 
-ebb100:
+block100:
     v816 = load.i64 v96
         v843 -> v816
         v856 -> v816
@@ -1625,14 +1625,14 @@ ebb100:
     v825 = iadd_imm.i64 v97, 8
     v826 = load.i8 v825
     v827 = uextend.i32 v826
-    brz v827, ebb101
-    jump ebb133
+    brz v827, block101
+    jump block133
 
-ebb133:
+block133:
     v828 = global_value.i64 gv61
     trap user0
 
-ebb101:
+block101:
     v829 = load.i64 v97
         v935 -> v829
         v962 -> v829
@@ -1649,14 +1649,14 @@ ebb101:
     v838 = iadd_imm.i64 v98, 8
     v839 = load.i8 v838
     v840 = uextend.i32 v839
-    brz v840, ebb102
-    jump ebb132
+    brz v840, block102
+    jump block132
 
-ebb132:
+block132:
     v841 = global_value.i64 gv62
     trap user0
 
-ebb102:
+block102:
     v842 = load.i64 v98
         v976 -> v842
         v989 -> v842
@@ -1671,14 +1671,14 @@ ebb102:
     v851 = iadd_imm.i64 v99, 8
     v852 = load.i8 v851
     v853 = uextend.i32 v852
-    brz v853, ebb103
-    jump ebb131
+    brz v853, block103
+    jump block131
 
-ebb131:
+block131:
     v854 = global_value.i64 gv63
     trap user0
 
-ebb103:
+block103:
     v855 = load.i64 v99
         v886 -> v855
     v858 = iconst.i64 1
@@ -1691,14 +1691,14 @@ ebb103:
     v865 = iadd_imm.i64 v100, 8
     v866 = load.i8 v865
     v867 = uextend.i32 v866
-    brz v867, ebb104
-    jump ebb130
+    brz v867, block104
+    jump block130
 
-ebb130:
+block130:
     v868 = global_value.i64 gv64
     trap user0
 
-ebb104:
+block104:
     v869 = load.i64 v100
     v870 = iconst.i64 1
     v871 = isub v869, v870
@@ -1710,14 +1710,14 @@ ebb104:
     v877 = iadd_imm.i64 v101, 8
     v878 = load.i8 v877
     v879 = uextend.i32 v878
-    brz v879, ebb105
-    jump ebb129
+    brz v879, block105
+    jump block129
 
-ebb129:
+block129:
     v880 = global_value.i64 gv65
     trap user0
 
-ebb105:
+block105:
     v881 = load.i64 v101
     v883 = band.i64 v882, v881
         v934 -> v883
@@ -1727,10 +1727,10 @@ ebb105:
         v1048 -> v883
     v884 = iconst.i8 1
     v885 = uextend.i32 v884
-    brz v885, ebb109(v855)
-    jump ebb106
+    brz v885, block109(v855)
+    jump block106
 
-ebb106:
+block106:
     v887 = iconst.i64 10
     v888 = icmp.i64 ult v886, v887
     v889 = bint.i8 v888
@@ -1738,18 +1738,18 @@ ebb106:
     v891 = icmp_imm eq v890, 0
     v892 = bint.i8 v891
     v893 = uextend.i32 v892
-    brz v893, ebb108
-    jump ebb107
+    brz v893, block108
+    jump block107
 
-ebb107:
+block107:
     v894 = global_value.i64 gv66
     v895 = global_value.i64 gv67
     trap user65535
 
-ebb108:
-    jump ebb109(v886)
+block108:
+    jump block109(v886)
 
-ebb109(v896: i64):
+block109(v896: i64):
     v897 = ireduce.i8 v896
     v898 = iconst.i8 48
     v899 = iadd v898, v897
@@ -1760,27 +1760,27 @@ ebb109(v896: i64):
     v904 = iadd_imm.i64 v102, 1
     v905 = load.i8 v904
     v906 = uextend.i32 v905
-    brz v906, ebb110
-    jump ebb128
+    brz v906, block110
+    jump block128
 
-ebb128:
+block128:
     v907 = global_value.i64 gv68
     trap user0
 
-ebb110:
+block110:
     v909 = load.i64 v3
     v910 = load.i64 v3+8
     v911 = icmp.i64 ult v908, v910
     v912 = bint.i8 v911
     v913 = uextend.i32 v912
-    brnz v913, ebb111
-    jump ebb127
+    brnz v913, block111
+    jump block127
 
-ebb127:
+block127:
     v914 = global_value.i64 gv69
     trap user0
 
-ebb111:
+block111:
     v915 = load.i64 v3
     v916 = load.i64 v3+8
     v918 = imul_imm.i64 v917, 1
@@ -1796,24 +1796,24 @@ ebb111:
     v929 = iadd_imm.i64 v103, 8
     v930 = load.i8 v929
     v931 = uextend.i32 v930
-    brz v931, ebb112
-    jump ebb126
+    brz v931, block112
+    jump block126
 
-ebb126:
+block126:
     v932 = global_value.i64 gv70
     trap user0
 
-ebb112:
+block112:
     v933 = load.i64 v103
         v954 -> v933
         v1047 -> v933
     v936 = icmp.i64 ult v934, v935
     v937 = bint.i8 v936
     v938 = uextend.i32 v937
-    brz v938, ebb119
-    jump ebb113
+    brz v938, block119
+    jump block113
 
-ebb113:
+block113:
     v940 = iconst.i64 1
     v941 = ishl v940, v939
     v942 = iconst.i8 0
@@ -1824,23 +1824,23 @@ ebb113:
     v947 = iadd_imm.i64 v104, 8
     v948 = load.i8 v947
     v949 = uextend.i32 v948
-    brz v949, ebb114
-    jump ebb125
+    brz v949, block114
+    jump block125
 
-ebb125:
+block125:
     v950 = global_value.i64 gv71
     trap user0
 
-ebb114:
+block114:
     v951 = load.i64 v104
         v988 -> v951
     v952 = load.i64 aligned v3
     v953 = load.i64 aligned v3+8
     v955 = load.i64 v109
     call fn101(v107, v108, v955)
-    jump ebb115
+    jump block115
 
-ebb115:
+block115:
     v956 = load.i64 aligned v107
     v957 = load.i64 aligned v107+8
     v958 = load.i64 aligned v106
@@ -1855,14 +1855,14 @@ ebb115:
     v971 = iadd_imm.i64 v110, 8
     v972 = load.i8 v971
     v973 = uextend.i32 v972
-    brz v973, ebb116
-    jump ebb123
+    brz v973, block116
+    jump block123
 
-ebb123:
+block123:
     v974 = global_value.i64 gv72
     trap user0
 
-ebb116:
+block116:
     v975 = load.i64 v110
     v977 = imul v975, v976
     v978 = iconst.i8 0
@@ -1873,22 +1873,22 @@ ebb116:
     v983 = iadd_imm.i64 v111, 8
     v984 = load.i8 v983
     v985 = uextend.i32 v984
-    brz v985, ebb117
-    jump ebb122
+    brz v985, block117
+    jump block122
 
-ebb122:
+block122:
     v986 = global_value.i64 gv73
     trap user0
 
-ebb117:
+block117:
     v987 = load.i64 v111
     call fn104(v0, v105, v990, v991, v992, v987, v988, v989)
-    jump ebb118
+    jump block118
 
-ebb118:
-    jump ebb79
+block118:
+    jump block79
 
-ebb119:
+block119:
     v993 = iconst.i16 1
     v994 = load.i16 v62
     v995 = isub v994, v993
@@ -1899,14 +1899,14 @@ ebb119:
     v1000 = iadd_imm.i64 v112, 2
     v1001 = load.i8 v1000
     v1002 = uextend.i32 v1001
-    brz v1002, ebb120
-    jump ebb121
+    brz v1002, block120
+    jump block121
 
-ebb121:
+block121:
     v1003 = global_value.i64 gv74
     trap user0
 
-ebb120:
+block120:
     v1004 = load.i16 aligned v112
-    jump ebb99(v1005, v1036, v1038, v1042, v1047, v1055, v1060)
+    jump block99(v1005, v1036, v1038, v1042, v1047, v1055, v1060)
 }
diff --git a/cranelift/tests/bugpoint_test_expected.clif b/cranelift/tests/bugpoint_test_expected.clif
index 0382b856f1..b2ca38a064 100644
--- a/cranelift/tests/bugpoint_test_expected.clif
+++ b/cranelift/tests/bugpoint_test_expected.clif
@@ -2,7 +2,7 @@ function u0:0(i64, i64, i64) system_v {
     sig0 = (i64, i64, i16, i64, i64, i64, i64, i64) system_v
     fn0 = u0:95 sig0
 
-ebb0(v0: i64, v1: i64, v2: i64):
+block0(v0: i64, v1: i64, v2: i64):
     v113 -> v1
     v124 -> v1
     v136 -> v1
@@ -17,7 +17,7 @@ ebb0(v0: i64, v1: i64, v2: i64):
     v105 = iconst.i64 0
     trap user0
 
-ebb99(v804: i64, v1035: i64, v1037: i64, v1039: i64, v1044: i64, v1052: i16, v1057: i64):
+block99(v804: i64, v1035: i64, v1037: i64, v1039: i64, v1044: i64, v1052: i16, v1057: i64):
     v817 -> v1035
     v830 -> v1037
     v844 -> v1039
@@ -38,7 +38,7 @@ ebb99(v804: i64, v1035: i64, v1037: i64, v1039: i64, v1044: i64, v1052: i16, v10
     v1060 -> v1057
     trap user0
 
-ebb101:
+block101:
     v829 = iconst.i64 0
     v935 -> v829
     v962 -> v829
@@ -47,7 +47,7 @@ ebb101:
     v1049 -> v829
     trap user0
 
-ebb102:
+block102:
     v842 = iconst.i64 0
     v976 -> v842
     v989 -> v842
@@ -55,7 +55,7 @@ ebb102:
     v1061 -> v842
     trap user0
 
-ebb105:
+block105:
     v883 = iconst.i64 0
     v934 -> v883
     v961 -> v883
@@ -64,12 +64,12 @@ ebb105:
     v1048 -> v883
     trap user0
 
-ebb114:
+block114:
     v951 = iconst.i64 0
     v988 -> v951
     trap user0
 
-ebb117:
+block117:
     v987 = iconst.i64 0
     call fn0(v0, v105, v1052, v883, v829, v987, v951, v842)
     trap user0
diff --git a/cranelift/umbrella/src/lib.rs b/cranelift/umbrella/src/lib.rs
index f97ed443ce..46582c9555 100644
--- a/cranelift/umbrella/src/lib.rs
+++ b/cranelift/umbrella/src/lib.rs
@@ -37,7 +37,7 @@ pub mod prelude {
     pub use crate::codegen::ir::immediates::{Ieee32, Ieee64, Imm64, Uimm64};
     pub use crate::codegen::ir::types;
     pub use crate::codegen::ir::{
-        AbiParam, Ebb, ExtFuncData, ExternalName, GlobalValueData, InstBuilder, JumpTableData,
+        AbiParam, Block, ExtFuncData, ExternalName, GlobalValueData, InstBuilder, JumpTableData,
         MemFlags, Signature, StackSlotData, StackSlotKind, TrapCode, Type, Value,
     };
     pub use crate::codegen::isa;
diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index a767f093db..58b6eb1771 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -26,7 +26,7 @@ use super::{hash_map, HashMap};
 use crate::environ::{FuncEnvironment, GlobalVariable, ReturnMode, WasmResult};
 use crate::state::{ControlStackFrame, ElseData, FuncTranslationState, ModuleTranslationState};
 use crate::translation_utils::{
-    blocktype_params_results, ebb_with_params, f32_translation, f64_translation,
+    block_with_params, blocktype_params_results, f32_translation, f64_translation,
 };
 use crate::translation_utils::{FuncIndex, GlobalIndex, MemoryIndex, SignatureIndex, TableIndex};
 use crate::wasm_unsupported;
@@ -147,32 +147,34 @@ pub fn translate_operator(
             state.reachable = false;
         }
         /***************************** Control flow blocks **********************************
-         *  When starting a control flow block, we create a new `Ebb` that will hold the code
+         *  When starting a control flow block, we create a new `Block` that will hold the code
          *  after the block, and we push a frame on the control stack. Depending on the type
-         *  of block, we create a new `Ebb` for the body of the block with an associated
+         *  of block, we create a new `Block` for the body of the block with an associated
          *  jump instruction.
          *
          *  The `End` instruction pops the last control frame from the control stack, seals
          *  the destination block (since `br` instructions targeting it only appear inside the
          *  block and have already been translated) and modify the value stack to use the
-         *  possible `Ebb`'s arguments values.
+         *  possible `Block`'s arguments values.
          ***********************************************************************************/
         Operator::Block { ty } => {
             let (params, results) = blocktype_params_results(module_translation_state, *ty)?;
-            let next = ebb_with_params(builder, results, environ)?;
+            let next = block_with_params(builder, results, environ)?;
             state.push_block(next, params.len(), results.len());
         }
         Operator::Loop { ty } => {
             let (params, results) = blocktype_params_results(module_translation_state, *ty)?;
-            let loop_body = ebb_with_params(builder, params, environ)?;
-            let next = ebb_with_params(builder, results, environ)?;
+            let loop_body = block_with_params(builder, params, environ)?;
+            let next = block_with_params(builder, results, environ)?;
             builder.ins().jump(loop_body, state.peekn(params.len()));
             state.push_loop(loop_body, next, params.len(), results.len());
 
-            // Pop the initial `Ebb` actuals and replace them with the `Ebb`'s
+            // Pop the initial `Block` actuals and replace them with the `Block`'s
             // params since control flow joins at the top of the loop.
             state.popn(params.len());
-            state.stack.extend_from_slice(builder.ebb_params(loop_body));
+            state
+                .stack
+                .extend_from_slice(builder.block_params(loop_body));
 
             builder.switch_to_block(loop_body);
             environ.translate_loop_header(builder.cursor())?;
@@ -183,12 +185,12 @@ pub fn translate_operator(
             let (params, results) = blocktype_params_results(module_translation_state, *ty)?;
             let (destination, else_data) = if params == results {
                 // It is possible there is no `else` block, so we will only
-                // allocate an ebb for it if/when we find the `else`. For now,
+                // allocate an block for it if/when we find the `else`. For now,
                 // we if the condition isn't true, then we jump directly to the
-                // destination ebb following the whole `if...end`. If we do end
-                // up discovering an `else`, then we will allocate an ebb for it
+                // destination block following the whole `if...end`. If we do end
+                // up discovering an `else`, then we will allocate an block for it
                 // and go back and patch the jump.
-                let destination = ebb_with_params(builder, results, environ)?;
+                let destination = block_with_params(builder, results, environ)?;
                 let branch_inst = builder
                     .ins()
                     .brz(val, destination, state.peekn(params.len()));
@@ -196,8 +198,8 @@ pub fn translate_operator(
             } else {
                 // The `if` type signature is not valid without an `else` block,
                 // so we eagerly allocate the `else` block here.
-                let destination = ebb_with_params(builder, results, environ)?;
-                let else_block = ebb_with_params(builder, params, environ)?;
+                let destination = block_with_params(builder, results, environ)?;
+                let else_block = block_with_params(builder, params, environ)?;
                 builder
                     .ins()
                     .brz(val, else_block, state.peekn(params.len()));
@@ -205,12 +207,12 @@ pub fn translate_operator(
                 (destination, ElseData::WithElse { else_block })
             };
 
-            let next_ebb = builder.create_ebb();
-            builder.ins().jump(next_ebb, &[]);
-            builder.seal_block(next_ebb); // Only predecessor is the current block.
-            builder.switch_to_block(next_ebb);
+            let next_block = builder.create_block();
+            builder.ins().jump(next_block, &[]);
+            builder.seal_block(next_block); // Only predecessor is the current block.
+            builder.switch_to_block(next_block);
 
-            // Here we append an argument to an Ebb targeted by an argumentless jump instruction
+            // Here we append an argument to an Block targeted by an argumentless jump instruction
             // But in fact there are two cases:
             // - either the If does not have a Else clause, in that case ty = EmptyBlock
             //   and we add nothing;
@@ -239,20 +241,20 @@ pub fn translate_operator(
                         // We have a branch from the head of the `if` to the `else`.
                         state.reachable = true;
 
-                        // Ensure we have an ebb for the `else` block (it may have
+                        // Ensure we have an block for the `else` block (it may have
                         // already been pre-allocated, see `ElseData` for details).
-                        let else_ebb = match *else_data {
+                        let else_block = match *else_data {
                             ElseData::NoElse { branch_inst } => {
                                 let (params, _results) =
                                     blocktype_params_results(module_translation_state, blocktype)?;
                                 debug_assert_eq!(params.len(), num_return_values);
-                                let else_ebb = ebb_with_params(builder, params, environ)?;
+                                let else_block = block_with_params(builder, params, environ)?;
                                 builder.ins().jump(destination, state.peekn(params.len()));
                                 state.popn(params.len());
 
-                                builder.change_jump_destination(branch_inst, else_ebb);
-                                builder.seal_block(else_ebb);
-                                else_ebb
+                                builder.change_jump_destination(branch_inst, else_block);
+                                builder.seal_block(else_block);
+                                else_block
                             }
                             ElseData::WithElse { else_block } => {
                                 builder
@@ -273,7 +275,7 @@ pub fn translate_operator(
                         // `if` so that we wouldn't have to save the parameters in the
                         // `ControlStackFrame` as another `Vec` allocation.
 
-                        builder.switch_to_block(else_ebb);
+                        builder.switch_to_block(else_block);
 
                         // We don't bother updating the control frame's `ElseData`
                         // to `WithElse` because nothing else will read it.
@@ -284,13 +286,13 @@ pub fn translate_operator(
         }
         Operator::End => {
             let frame = state.control_stack.pop().unwrap();
-            let next_ebb = frame.following_code();
+            let next_block = frame.following_code();
 
             if !builder.is_unreachable() || !builder.is_pristine() {
                 let return_count = frame.num_return_values();
                 let return_args = state.peekn_mut(return_count);
-                let next_ebb_types = builder.func.dfg.ebb_param_types(next_ebb);
-                bitcast_arguments(return_args, &next_ebb_types, builder);
+                let next_block_types = builder.func.dfg.block_param_types(next_block);
+                bitcast_arguments(return_args, &next_block_types, builder);
                 builder.ins().jump(frame.following_code(), return_args);
                 // You might expect that if we just finished an `if` block that
                 // didn't have a corresponding `else` block, then we would clean
@@ -299,33 +301,35 @@ pub fn translate_operator(
                 // since we truncate the stack back to the original height
                 // below.
             }
-            builder.switch_to_block(next_ebb);
-            builder.seal_block(next_ebb);
+            builder.switch_to_block(next_block);
+            builder.seal_block(next_block);
             // If it is a loop we also have to seal the body loop block
             if let ControlStackFrame::Loop { header, .. } = frame {
                 builder.seal_block(header)
             }
             state.stack.truncate(frame.original_stack_size());
-            state.stack.extend_from_slice(builder.ebb_params(next_ebb));
+            state
+                .stack
+                .extend_from_slice(builder.block_params(next_block));
         }
         /**************************** Branch instructions *********************************
          * The branch instructions all have as arguments a target nesting level, which
          * corresponds to how many control stack frames do we have to pop to get the
-         * destination `Ebb`.
+         * destination `Block`.
          *
-         * Once the destination `Ebb` is found, we sometimes have to declare a certain depth
+         * Once the destination `Block` is found, we sometimes have to declare a certain depth
          * of the stack unreachable, because some branch instructions are terminator.
          *
          * The `br_table` case is much more complicated because Cranelift's `br_table` instruction
          * does not support jump arguments like all the other branch instructions. That is why, in
          * the case where we would use jump arguments for every other branch instructions, we
-         * need to split the critical edges leaving the `br_tables` by creating one `Ebb` per
-         * table destination; the `br_table` will point to these newly created `Ebbs` and these
-         * `Ebb`s contain only a jump instruction pointing to the final destination, this time with
+         * need to split the critical edges leaving the `br_tables` by creating one `Block` per
+         * table destination; the `br_table` will point to these newly created `Blocks` and these
+         * `Block`s contain only a jump instruction pointing to the final destination, this time with
          * jump arguments.
          *
          * This system is also implemented in Cranelift's SSA construction algorithm, because
-         * `use_var` located in a destination `Ebb` of a `br_table` might trigger the addition
+         * `use_var` located in a destination `Block` of a `br_table` might trigger the addition
          * of jump arguments in each predecessor branch instruction, one of which might be a
          * `br_table`.
          ***********************************************************************************/
@@ -345,7 +349,7 @@ pub fn translate_operator(
 
             // Bitcast any vector arguments to their default type, I8X16, before jumping.
             let destination_args = state.peekn_mut(return_count);
-            let destination_types = builder.func.dfg.ebb_param_types(br_destination);
+            let destination_types = builder.func.dfg.block_param_types(br_destination);
             bitcast_arguments(
                 destination_args,
                 &destination_types[..return_count],
@@ -379,53 +383,53 @@ pub fn translate_operator(
             if jump_args_count == 0 {
                 // No jump arguments
                 for depth in &*depths {
-                    let ebb = {
+                    let block = {
                         let i = state.control_stack.len() - 1 - (*depth as usize);
                         let frame = &mut state.control_stack[i];
                         frame.set_branched_to_exit();
                         frame.br_destination()
                     };
-                    data.push_entry(ebb);
+                    data.push_entry(block);
                 }
                 let jt = builder.create_jump_table(data);
-                let ebb = {
+                let block = {
                     let i = state.control_stack.len() - 1 - (default as usize);
                     let frame = &mut state.control_stack[i];
                     frame.set_branched_to_exit();
                     frame.br_destination()
                 };
-                builder.ins().br_table(val, ebb, jt);
+                builder.ins().br_table(val, block, jt);
             } else {
                 // Here we have jump arguments, but Cranelift's br_table doesn't support them
                 // We then proceed to split the edges going out of the br_table
                 let return_count = jump_args_count;
-                let mut dest_ebb_sequence = vec![];
-                let mut dest_ebb_map = HashMap::new();
+                let mut dest_block_sequence = vec![];
+                let mut dest_block_map = HashMap::new();
                 for depth in &*depths {
-                    let branch_ebb = match dest_ebb_map.entry(*depth as usize) {
+                    let branch_block = match dest_block_map.entry(*depth as usize) {
                         hash_map::Entry::Occupied(entry) => *entry.get(),
                         hash_map::Entry::Vacant(entry) => {
-                            let ebb = builder.create_ebb();
-                            dest_ebb_sequence.push((*depth as usize, ebb));
-                            *entry.insert(ebb)
+                            let block = builder.create_block();
+                            dest_block_sequence.push((*depth as usize, block));
+                            *entry.insert(block)
                         }
                     };
-                    data.push_entry(branch_ebb);
+                    data.push_entry(branch_block);
                 }
-                let default_branch_ebb = match dest_ebb_map.entry(default as usize) {
+                let default_branch_block = match dest_block_map.entry(default as usize) {
                     hash_map::Entry::Occupied(entry) => *entry.get(),
                     hash_map::Entry::Vacant(entry) => {
-                        let ebb = builder.create_ebb();
-                        dest_ebb_sequence.push((default as usize, ebb));
-                        *entry.insert(ebb)
+                        let block = builder.create_block();
+                        dest_block_sequence.push((default as usize, block));
+                        *entry.insert(block)
                     }
                 };
                 let jt = builder.create_jump_table(data);
-                builder.ins().br_table(val, default_branch_ebb, jt);
-                for (depth, dest_ebb) in dest_ebb_sequence {
-                    builder.switch_to_block(dest_ebb);
-                    builder.seal_block(dest_ebb);
-                    let real_dest_ebb = {
+                builder.ins().br_table(val, default_branch_block, jt);
+                for (depth, dest_block) in dest_block_sequence {
+                    builder.switch_to_block(dest_block);
+                    builder.seal_block(dest_block);
+                    let real_dest_block = {
                         let i = state.control_stack.len() - 1 - depth;
                         let frame = &mut state.control_stack[i];
                         frame.set_branched_to_exit();
@@ -434,14 +438,14 @@ pub fn translate_operator(
 
                     // Bitcast any vector arguments to their default type, I8X16, before jumping.
                     let destination_args = state.peekn_mut(return_count);
-                    let destination_types = builder.func.dfg.ebb_param_types(real_dest_ebb);
+                    let destination_types = builder.func.dfg.block_param_types(real_dest_block);
                     bitcast_arguments(
                         destination_args,
                         &destination_types[..return_count],
                         builder,
                     );
 
-                    builder.ins().jump(real_dest_ebb, destination_args);
+                    builder.ins().jump(real_dest_block, destination_args);
                 }
                 state.popn(return_count);
             }
@@ -1498,7 +1502,7 @@ fn translate_unreachable_operator(
             // Push a placeholder control stack entry. The if isn't reachable,
             // so we don't have any branches anywhere.
             state.push_if(
-                ir::Ebb::reserved_value(),
+                ir::Block::reserved_value(),
                 ElseData::NoElse {
                     branch_inst: ir::Inst::reserved_value(),
                 },
@@ -1508,7 +1512,7 @@ fn translate_unreachable_operator(
             );
         }
         Operator::Loop { ty: _ } | Operator::Block { ty: _ } => {
-            state.push_block(ir::Ebb::reserved_value(), 0, 0);
+            state.push_block(ir::Block::reserved_value(), 0, 0);
         }
         Operator::Else => {
             let i = state.control_stack.len() - 1;
@@ -1527,21 +1531,21 @@ fn translate_unreachable_operator(
                         // We have a branch from the head of the `if` to the `else`.
                         state.reachable = true;
 
-                        let else_ebb = match *else_data {
+                        let else_block = match *else_data {
                             ElseData::NoElse { branch_inst } => {
                                 let (params, _results) =
                                     blocktype_params_results(module_translation_state, blocktype)?;
-                                let else_ebb = ebb_with_params(builder, params, environ)?;
+                                let else_block = block_with_params(builder, params, environ)?;
 
                                 // We change the target of the branch instruction.
-                                builder.change_jump_destination(branch_inst, else_ebb);
-                                builder.seal_block(else_ebb);
-                                else_ebb
+                                builder.change_jump_destination(branch_inst, else_block);
+                                builder.seal_block(else_block);
+                                else_block
                             }
                             ElseData::WithElse { else_block } => else_block,
                         };
 
-                        builder.switch_to_block(else_ebb);
+                        builder.switch_to_block(else_block);
 
                         // Again, no need to push the parameters for the `else`,
                         // since we already did when we saw the original `if`. See
@@ -1596,7 +1600,7 @@ fn translate_unreachable_operator(
 
                 // And add the return values of the block but only if the next block is reachable
                 // (which corresponds to testing if the stack depth is 1)
-                stack.extend_from_slice(builder.ebb_params(frame.following_code()));
+                stack.extend_from_slice(builder.block_params(frame.following_code()));
                 state.reachable = true;
             }
         }
@@ -1736,21 +1740,21 @@ fn translate_br_if(
     let (br_destination, inputs) = translate_br_if_args(relative_depth, state);
 
     // Bitcast any vector arguments to their default type, I8X16, before jumping.
-    let destination_types = builder.func.dfg.ebb_param_types(br_destination);
+    let destination_types = builder.func.dfg.block_param_types(br_destination);
     bitcast_arguments(inputs, &destination_types[..inputs.len()], builder);
 
     builder.ins().brnz(val, br_destination, inputs);
 
-    let next_ebb = builder.create_ebb();
-    builder.ins().jump(next_ebb, &[]);
-    builder.seal_block(next_ebb); // The only predecessor is the current block.
-    builder.switch_to_block(next_ebb);
+    let next_block = builder.create_block();
+    builder.ins().jump(next_block, &[]);
+    builder.seal_block(next_block); // The only predecessor is the current block.
+    builder.switch_to_block(next_block);
 }
 
 fn translate_br_if_args(
     relative_depth: u32,
     state: &mut FuncTranslationState,
-) -> (ir::Ebb, &mut [ir::Value]) {
+) -> (ir::Block, &mut [ir::Value]) {
     let i = state.control_stack.len() - 1 - (relative_depth as usize);
     let (return_count, br_destination) = {
         let frame = &mut state.control_stack[i];
@@ -1973,7 +1977,7 @@ fn pop2_with_bitcast(
 /// A helper for bitcasting a sequence of values (e.g. function arguments). If a value is a
 /// vector type that does not match its expected type, this will modify the value in place to point
 /// to the result of a `raw_bitcast`. This conversion is necessary to translate Wasm code that
-/// uses `V128` as function parameters (or implicitly in EBB parameters) and still use specific
+/// uses `V128` as function parameters (or implicitly in block parameters) and still use specific
 /// CLIF types (e.g. `I32X4`) in the function body.
 pub fn bitcast_arguments(
     arguments: &mut [Value],
diff --git a/cranelift/wasm/src/func_translator.rs b/cranelift/wasm/src/func_translator.rs
index cc61b28c9b..27431be148 100644
--- a/cranelift/wasm/src/func_translator.rs
+++ b/cranelift/wasm/src/func_translator.rs
@@ -10,7 +10,7 @@ use crate::state::{FuncTranslationState, ModuleTranslationState};
 use crate::translation_utils::get_vmctx_value_label;
 use crate::wasm_unsupported;
 use cranelift_codegen::entity::EntityRef;
-use cranelift_codegen::ir::{self, Ebb, InstBuilder, ValueLabel};
+use cranelift_codegen::ir::{self, Block, InstBuilder, ValueLabel};
 use cranelift_codegen::timing;
 use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable};
 use log::info;
@@ -84,27 +84,27 @@ impl FuncTranslator {
             func.name,
             func.signature
         );
-        debug_assert_eq!(func.dfg.num_ebbs(), 0, "Function must be empty");
+        debug_assert_eq!(func.dfg.num_blocks(), 0, "Function must be empty");
         debug_assert_eq!(func.dfg.num_insts(), 0, "Function must be empty");
 
         // This clears the `FunctionBuilderContext`.
         let mut builder = FunctionBuilder::new(func, &mut self.func_ctx);
         builder.set_srcloc(cur_srcloc(&reader));
-        let entry_block = builder.create_ebb();
-        builder.append_ebb_params_for_function_params(entry_block);
+        let entry_block = builder.create_block();
+        builder.append_block_params_for_function_params(entry_block);
         builder.switch_to_block(entry_block); // This also creates values for the arguments.
         builder.seal_block(entry_block); // Declare all predecessors known.
 
         // Make sure the entry block is inserted in the layout before we make any callbacks to
         // `environ`. The callback functions may need to insert things in the entry block.
-        builder.ensure_inserted_ebb();
+        builder.ensure_inserted_block();
 
         let num_params = declare_wasm_parameters(&mut builder, entry_block, environ);
 
         // Set up the translation state with a single pushed control block representing the whole
         // function and its return values.
-        let exit_block = builder.create_ebb();
-        builder.append_ebb_params_for_function_returns(exit_block);
+        let exit_block = builder.create_block();
+        builder.append_block_params_for_function_returns(exit_block);
         self.state.initialize(&builder.func.signature, exit_block);
 
         parse_local_decls(&mut reader, &mut builder, num_params, environ)?;
@@ -126,7 +126,7 @@ impl FuncTranslator {
 /// Return the number of local variables declared.
 fn declare_wasm_parameters(
     builder: &mut FunctionBuilder,
-    entry_block: Ebb,
+    entry_block: Block,
     environ: &FE,
 ) -> usize {
     let sig_len = builder.func.signature.params.len();
@@ -141,11 +141,11 @@ fn declare_wasm_parameters(
             builder.declare_var(local, param_type.value_type);
             next_local += 1;
 
-            let param_value = builder.ebb_params(entry_block)[i];
+            let param_value = builder.block_params(entry_block)[i];
             builder.def_var(local, param_value);
         }
         if param_type.purpose == ir::ArgumentPurpose::VMContext {
-            let param_value = builder.ebb_params(entry_block)[i];
+            let param_value = builder.block_params(entry_block)[i];
             builder.set_val_label(param_value, get_vmctx_value_label());
         }
     }
diff --git a/cranelift/wasm/src/state/func_state.rs b/cranelift/wasm/src/state/func_state.rs
index 0ea0e13ea9..81a0d35e97 100644
--- a/cranelift/wasm/src/state/func_state.rs
+++ b/cranelift/wasm/src/state/func_state.rs
@@ -9,7 +9,7 @@
 use crate::environ::{FuncEnvironment, GlobalVariable, WasmResult};
 use crate::translation_utils::{FuncIndex, GlobalIndex, MemoryIndex, SignatureIndex, TableIndex};
 use crate::{HashMap, Occupied, Vacant};
-use cranelift_codegen::ir::{self, Ebb, Inst, Value};
+use cranelift_codegen::ir::{self, Block, Inst, Value};
 use std::vec::Vec;
 
 /// Information about the presence of an associated `else` for an `if`, or the
@@ -35,24 +35,24 @@ pub enum ElseData {
     /// these cases, we pre-allocate the `else` block.
     WithElse {
         /// This is the `else` block.
-        else_block: Ebb,
+        else_block: Block,
     },
 }
 
 /// A control stack frame can be an `if`, a `block` or a `loop`, each one having the following
 /// fields:
 ///
-/// - `destination`: reference to the `Ebb` that will hold the code after the control block;
+/// - `destination`: reference to the `Block` that will hold the code after the control block;
 /// - `num_return_values`: number of values returned by the control block;
 /// - `original_stack_size`: size of the value stack at the beginning of the control block.
 ///
 /// Moreover, the `if` frame has the `branch_inst` field that points to the `brz` instruction
 /// separating the `true` and `false` branch. The `loop` frame has a `header` field that references
-/// the `Ebb` that contains the beginning of the body of the loop.
+/// the `Block` that contains the beginning of the body of the loop.
 #[derive(Debug)]
 pub enum ControlStackFrame {
     If {
-        destination: Ebb,
+        destination: Block,
         else_data: ElseData,
         num_param_values: usize,
         num_return_values: usize,
@@ -72,15 +72,15 @@ pub enum ControlStackFrame {
         // `state.reachable` when we hit the `end` in the `if .. else .. end`.
     },
     Block {
-        destination: Ebb,
+        destination: Block,
         num_param_values: usize,
         num_return_values: usize,
         original_stack_size: usize,
         exit_is_branched_to: bool,
     },
     Loop {
-        destination: Ebb,
-        header: Ebb,
+        destination: Block,
+        header: Block,
         num_param_values: usize,
         num_return_values: usize,
         original_stack_size: usize,
@@ -115,14 +115,14 @@ impl ControlStackFrame {
             } => num_param_values,
         }
     }
-    pub fn following_code(&self) -> Ebb {
+    pub fn following_code(&self) -> Block {
         match *self {
             Self::If { destination, .. }
             | Self::Block { destination, .. }
             | Self::Loop { destination, .. } => destination,
         }
     }
-    pub fn br_destination(&self) -> Ebb {
+    pub fn br_destination(&self) -> Block {
         match *self {
             Self::If { destination, .. } | Self::Block { destination, .. } => destination,
             Self::Loop { header, .. } => header,
@@ -254,7 +254,7 @@ impl FuncTranslationState {
     ///
     /// This resets the state to containing only a single block representing the whole function.
     /// The exit block is the last block in the function which will contain the return instruction.
-    pub(crate) fn initialize(&mut self, sig: &ir::Signature, exit_block: Ebb) {
+    pub(crate) fn initialize(&mut self, sig: &ir::Signature, exit_block: Block) {
         self.clear();
         self.push_block(
             exit_block,
@@ -343,7 +343,7 @@ impl FuncTranslationState {
     /// Push a block on the control stack.
     pub(crate) fn push_block(
         &mut self,
-        following_code: Ebb,
+        following_code: Block,
         num_param_types: usize,
         num_result_types: usize,
     ) {
@@ -360,8 +360,8 @@ impl FuncTranslationState {
     /// Push a loop on the control stack.
     pub(crate) fn push_loop(
         &mut self,
-        header: Ebb,
-        following_code: Ebb,
+        header: Block,
+        following_code: Block,
         num_param_types: usize,
         num_result_types: usize,
     ) {
@@ -378,7 +378,7 @@ impl FuncTranslationState {
     /// Push an if on the control stack.
     pub(crate) fn push_if(
         &mut self,
-        destination: Ebb,
+        destination: Block,
         else_data: ElseData,
         num_param_types: usize,
         num_result_types: usize,
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index d431c13f29..796cc5e49b 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -185,42 +185,42 @@ pub fn blocktype_params_results(
     })
 }
 
-/// Create an `Ebb` with the given Wasm parameters.
-pub fn ebb_with_params(
+/// Create a `Block` with the given Wasm parameters.
+pub fn block_with_params(
     builder: &mut FunctionBuilder,
     params: &[wasmparser::Type],
     environ: &PE,
-) -> WasmResult {
-    let ebb = builder.create_ebb();
+) -> WasmResult {
+    let block = builder.create_block();
     for ty in params.iter() {
         match ty {
             wasmparser::Type::I32 => {
-                builder.append_ebb_param(ebb, ir::types::I32);
+                builder.append_block_param(block, ir::types::I32);
             }
             wasmparser::Type::I64 => {
-                builder.append_ebb_param(ebb, ir::types::I64);
+                builder.append_block_param(block, ir::types::I64);
             }
             wasmparser::Type::F32 => {
-                builder.append_ebb_param(ebb, ir::types::F32);
+                builder.append_block_param(block, ir::types::F32);
             }
             wasmparser::Type::F64 => {
-                builder.append_ebb_param(ebb, ir::types::F64);
+                builder.append_block_param(block, ir::types::F64);
             }
             wasmparser::Type::AnyRef | wasmparser::Type::AnyFunc | wasmparser::Type::NullRef => {
-                builder.append_ebb_param(ebb, environ.reference_type());
+                builder.append_block_param(block, environ.reference_type());
             }
             wasmparser::Type::V128 => {
-                builder.append_ebb_param(ebb, ir::types::I8X16);
+                builder.append_block_param(block, ir::types::I8X16);
             }
             ty => {
                 return Err(wasm_unsupported!(
-                    "ebb_with_params: type {:?} in multi-value block's signature",
+                    "block_with_params: type {:?} in multi-value block's signature",
                     ty
                 ))
             }
         }
     }
-    Ok(ebb)
+    Ok(block)
 }
 
 /// Turns a `wasmparser` `f32` into a `Cranelift` one.

From ce1ee2d2f595627b6a63aa9c1bca3a0e5416b4a9 Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Fri, 7 Feb 2020 11:44:07 -0800
Subject: [PATCH 3042/3084] Enable `ref.func` global initializers (#1380)

* Fix comment referencing an outdated instruction name

* cranelift-wasm: Enable `ref.func` global initializers
---
 cranelift/wasm/src/sections_translator.rs |  3 +++
 cranelift/wasm/src/translation_utils.rs   |  4 +++-
 cranelift/wasmtests/ref-func-0.wat        | 12 ++++++++++++
 3 files changed, 18 insertions(+), 1 deletion(-)
 create mode 100644 cranelift/wasmtests/ref-func-0.wat

diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index 008b9b7134..f4c5079a6e 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -224,6 +224,9 @@ pub fn parse_global_section(
                 GlobalInit::V128Const(V128Imm::from(value.bytes().to_vec().as_slice()))
             }
             Operator::RefNull => GlobalInit::RefNullConst,
+            Operator::RefFunc { function_index } => {
+                GlobalInit::RefFunc(FuncIndex::from_u32(function_index))
+            }
             Operator::GlobalGet { global_index } => {
                 GlobalInit::GetGlobal(GlobalIndex::from_u32(global_index))
             }
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index 796cc5e49b..cd19ed820f 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -81,10 +81,12 @@ pub enum GlobalInit {
     F64Const(u64),
     /// A `vconst`.
     V128Const(V128Imm),
-    /// A `get_global` of another global.
+    /// A `global.get` of another global.
     GetGlobal(GlobalIndex),
     /// A `ref.null`.
     RefNullConst,
+    /// A `ref.func `.
+    RefFunc(FuncIndex),
     ///< The global is imported from, and thus initialized by, a different module.
     Import,
 }
diff --git a/cranelift/wasmtests/ref-func-0.wat b/cranelift/wasmtests/ref-func-0.wat
new file mode 100644
index 0000000000..5a6a6a3b68
--- /dev/null
+++ b/cranelift/wasmtests/ref-func-0.wat
@@ -0,0 +1,12 @@
+(module
+  (func $imported (import "env" "f") (param i32) (result i32))
+  (func $local (result anyref anyref funcref funcref)
+    global.get 0
+    global.get 1
+    global.get 2
+    global.get 3)
+
+  (global (export "anyref-imported") anyref (ref.func $imported))
+  (global (export "anyref-local") anyref (ref.func $local))
+  (global (export "funcref-imported") funcref (ref.func $imported))
+  (global (export "funcref-local") funcref (ref.func $local)))

From 950dadac9441d412823554a8321203ba55bb4222 Mon Sep 17 00:00:00 2001
From: Gabor Greif 
Date: Fri, 7 Feb 2020 21:51:28 +0100
Subject: [PATCH 3043/3084] Catch a few typos (#1381)

---
 cranelift/wasm/src/code_translator.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 58b6eb1771..01242348cd 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -15,7 +15,7 @@
 //! are being translated:
 //!
 //! - the loads and stores need the memory base address;
-//! - the `get_global` et `set_global` instructions depends on how the globals are implemented;
+//! - the `get_global` and `set_global` instructions depend on how the globals are implemented;
 //! - `memory.size` and `memory.grow` are runtime functions;
 //! - `call_indirect` has to translate the function index into the address of where this
 //!    is;
@@ -322,7 +322,7 @@ pub fn translate_operator(
          *
          * The `br_table` case is much more complicated because Cranelift's `br_table` instruction
          * does not support jump arguments like all the other branch instructions. That is why, in
-         * the case where we would use jump arguments for every other branch instructions, we
+         * the case where we would use jump arguments for every other branch instruction, we
          * need to split the critical edges leaving the `br_tables` by creating one `Block` per
          * table destination; the `br_table` will point to these newly created `Blocks` and these
          * `Block`s contain only a jump instruction pointing to the final destination, this time with

From 376654bdfc979188e4002b423e2c8761b17f175b Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Fri, 7 Feb 2020 13:58:06 -0800
Subject: [PATCH 3044/3084] Bump version to 0.58.0 (#1382)

---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  2 +-
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 19 files changed, 68 insertions(+), 68 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index a6f57398f9..8e784ff3e2 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.57.0"
+version = "0.58.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.57.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.57.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.57.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.57.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.57.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.57.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.57.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.57.0" }
-cranelift-module = { path = "cranelift-module", version = "0.57.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.57.0" }
-cranelift-object = { path = "cranelift-object", version = "0.57.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.57.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.57.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.57.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.58.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.58.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.58.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.58.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.58.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.58.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.58.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.58.0" }
+cranelift-module = { path = "cranelift-module", version = "0.58.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.58.0" }
+cranelift-object = { path = "cranelift-object", version = "0.58.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.58.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.58.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.58.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 899f33e246..bac7f8d042 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.57.0"
+version = "0.58.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.57.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.58.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 1df76bb1d8..d4bbe8d3aa 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.57.0"
+version = "0.58.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.57.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.57.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.57.0" }
+cranelift-codegen-shared = { path = "./shared", version = "0.58.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.58.0" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.58.0" }
 hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
@@ -30,7 +30,7 @@ byteorder = { version = "1.3.2", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.57.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.58.0" }
 
 [features]
 default = ["std", "unwind"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 42f1d3e445..6bdb6a3463 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.57.0"
+version = "0.58.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.57.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.57.0" }
+cranelift-codegen-shared = { path = "../shared", version = "0.58.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.58.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index eb050e5780..67ffcbec19 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.57.0"
+version = "0.58.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index b4a0c72931..2a553e2345 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.57.0"
+version = "0.58.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 878d2d266c..144a18f769 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.57.0"
+version = "0.58.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.57.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.58.0" }
 faerie = "0.14.0"
 goblin = "0.1.0"
 anyhow = "1.0"
@@ -18,7 +18,7 @@ target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.57.0"
+version = "0.58.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index a66bf68dea..2539ef79c1 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.57.0"
+version = "0.58.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.57.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.57.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.57.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.58.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.58.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.58.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 gimli = { version = "0.20.0", default-features = false, features = ["read"] }
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 7cca019ed5..7b98c0fce9 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.57.0"
+version = "0.58.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 35de56226a..68eb3b5d49 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.57.0"
+version = "0.58.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.57.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.58.0" }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 thiserror = "1.0.4"
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index c34ab1064a..089acc3ccb 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.57.0"
+version = "0.58.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
 target-lexicon = "0.10"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 6434a0f514..f5bbcc2868 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.57.0"
+version = "0.58.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,13 +10,13 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.57.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.58.0" }
 object = { version = "0.17", default-features = false, features = ["write"] }
 target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.57.0"
+version = "0.58.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index a714585fa0..2454dabf20 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.57.0"
+version = "0.58.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.57.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.58.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 34cb615ab4..8f009287b4 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.56.0"
+version="0.58.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 00f129336c..a5645820a9 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.57.0"
+version = "0.58.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0" }
 target-lexicon = "0.10"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 7c277d063c..9fd94c1b56 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.57.0"
+version = "0.58.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.57.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.58.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 6183d17a06..f11946f41f 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.57.0"
+version = "0.58.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.57.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.57.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.58.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.58.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -20,7 +20,7 @@ memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.57.0"
+version = "0.58.0"
 default-features = false
 features = ["std"]
 
@@ -32,9 +32,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.57.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.57.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.57.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.58.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.58.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.58.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index fc8eecc3c3..3196bdfed7 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.57.0"
+version = "0.58.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.57.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.58.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 6b89bfae1a..cc5cc3d90e 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.57.0"
+version = "0.58.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.48.2", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.57.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.57.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.57.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.58.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.58.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }

From 3c15f8f12949dede11badb084dd27fa3386e9d00 Mon Sep 17 00:00:00 2001
From: Philip Craig 
Date: Fri, 24 Jan 2020 14:09:32 +1000
Subject: [PATCH 3045/3084] cranelift-object: move relocation processing to
 `finish`

This removes the need to call `finalize_definitions` for cranelift-object.
`finalize_definitions` is only intended for backends that produce
finalized functions and data objects, which cranelift-object does not.
---
 cranelift/faerie/src/backend.rs    |   2 +-
 cranelift/module/src/backend.rs    |   2 +-
 cranelift/module/src/module.rs     |   7 +-
 cranelift/object/src/backend.rs    | 146 +++++++++++++----------------
 cranelift/simplejit/src/backend.rs |   2 +-
 5 files changed, 72 insertions(+), 87 deletions(-)

diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index f2c227a52a..f10d457778 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -311,7 +311,7 @@ impl Backend for FaerieBackend {
         // Nothing to do.
     }
 
-    fn finish(self) -> FaerieProduct {
+    fn finish(self, _namespace: &ModuleNamespace) -> FaerieProduct {
         FaerieProduct {
             artifact: self.artifact,
             trap_manifest: self.trap_manifest,
diff --git a/cranelift/module/src/backend.rs b/cranelift/module/src/backend.rs
index 27472d042b..9efe17a926 100644
--- a/cranelift/module/src/backend.rs
+++ b/cranelift/module/src/backend.rs
@@ -146,7 +146,7 @@ where
 
     /// Consume this `Backend` and return a result. Some implementations may
     /// provide additional functionality through this result.
-    fn finish(self) -> Self::Product;
+    fn finish(self, namespace: &ModuleNamespace) -> Self::Product;
 }
 
 /// Default names for `ir::LibCall`s. A function by this name is imported into the object as
diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 56a94b89db..c6aa533807 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -714,8 +714,9 @@ where
     /// Consume the module and return the resulting `Product`. Some `Backend`
     /// implementations may provide additional functionality available after
     /// a `Module` is complete.
-    pub fn finish(mut self) -> B::Product {
-        self.finalize_definitions();
-        self.backend.finish()
+    pub fn finish(self) -> B::Product {
+        self.backend.finish(&ModuleNamespace:: {
+            contents: &self.contents,
+        })
     }
 }
diff --git a/cranelift/object/src/backend.rs b/cranelift/object/src/backend.rs
index d6a2d98385..0859046685 100644
--- a/cranelift/object/src/backend.rs
+++ b/cranelift/object/src/backend.rs
@@ -16,6 +16,7 @@ use object::write::{
 };
 use object::{RelocationEncoding, RelocationKind, SymbolFlags, SymbolKind, SymbolScope};
 use std::collections::HashMap;
+use std::mem;
 use target_lexicon::PointerWidth;
 
 #[derive(Debug)]
@@ -79,6 +80,7 @@ pub struct ObjectBackend {
     functions: SecondaryMap>,
     data_objects: SecondaryMap>,
     traps: SecondaryMap>,
+    relocs: Vec,
     libcalls: HashMap,
     libcall_names: Box String>,
     collect_traps: ObjectTrapCollection,
@@ -109,6 +111,7 @@ impl Backend for ObjectBackend {
             functions: SecondaryMap::new(),
             data_objects: SecondaryMap::new(),
             traps: SecondaryMap::new(),
+            relocs: Vec::new(),
             libcalls: HashMap::new(),
             libcall_names: builder.libcall_names,
             collect_traps: builder.collect_traps,
@@ -212,13 +215,15 @@ impl Backend for ObjectBackend {
         let offset = self
             .object
             .add_symbol_data(symbol, section, &code, self.function_alignment);
+        if !reloc_sink.relocs.is_empty() {
+            self.relocs.push(SymbolRelocs {
+                section,
+                offset,
+                relocs: reloc_sink.relocs,
+            });
+        }
         self.traps[func_id] = trap_sink.sites;
-        Ok(ObjectCompiledFunction {
-            offset,
-            size: code_size,
-            section,
-            relocs: reloc_sink.relocs,
-        })
+        Ok(ObjectCompiledFunction)
     }
 
     fn define_data(
@@ -290,11 +295,14 @@ impl Backend for ObjectBackend {
         let offset =
             self.object
                 .add_symbol_data(symbol, section, &data, u64::from(align.unwrap_or(1)));
-        Ok(ObjectCompiledData {
-            offset,
-            section,
-            relocs,
-        })
+        if !relocs.is_empty() {
+            self.relocs.push(SymbolRelocs {
+                section,
+                offset,
+                relocs,
+            });
+        }
+        Ok(ObjectCompiledData)
     }
 
     fn write_data_funcaddr(
@@ -319,34 +327,10 @@ impl Backend for ObjectBackend {
     fn finalize_function(
         &mut self,
         _id: FuncId,
-        func: &ObjectCompiledFunction,
-        namespace: &ModuleNamespace,
+        _func: &ObjectCompiledFunction,
+        _namespace: &ModuleNamespace,
     ) {
-        for &RelocRecord {
-            offset,
-            ref name,
-            kind,
-            encoding,
-            size,
-            addend,
-        } in &func.relocs
-        {
-            let offset = func.offset + u64::from(offset);
-            let symbol = self.get_symbol(namespace, name);
-            self.object
-                .add_relocation(
-                    func.section,
-                    Relocation {
-                        offset,
-                        size,
-                        kind,
-                        encoding,
-                        symbol,
-                        addend,
-                    },
-                )
-                .unwrap();
-        }
+        // Nothing to do.
     }
 
     fn get_finalized_function(&self, _func: &ObjectCompiledFunction) {
@@ -356,34 +340,10 @@ impl Backend for ObjectBackend {
     fn finalize_data(
         &mut self,
         _id: DataId,
-        data: &ObjectCompiledData,
-        namespace: &ModuleNamespace,
+        _data: &ObjectCompiledData,
+        _namespace: &ModuleNamespace,
     ) {
-        for &RelocRecord {
-            offset,
-            ref name,
-            kind,
-            encoding,
-            size,
-            addend,
-        } in &data.relocs
-        {
-            let offset = data.offset + u64::from(offset);
-            let symbol = self.get_symbol(namespace, name);
-            self.object
-                .add_relocation(
-                    data.section,
-                    Relocation {
-                        offset,
-                        size,
-                        kind,
-                        encoding,
-                        symbol,
-                        addend,
-                    },
-                )
-                .unwrap();
-        }
+        // Nothing to do.
     }
 
     fn get_finalized_data(&self, _data: &ObjectCompiledData) {
@@ -394,7 +354,36 @@ impl Backend for ObjectBackend {
         // Nothing to do.
     }
 
-    fn finish(self) -> ObjectProduct {
+    fn finish(mut self, namespace: &ModuleNamespace) -> ObjectProduct {
+        let mut symbol_relocs = Vec::new();
+        mem::swap(&mut symbol_relocs, &mut self.relocs);
+        for symbol in symbol_relocs {
+            for &RelocRecord {
+                offset,
+                ref name,
+                kind,
+                encoding,
+                size,
+                addend,
+            } in &symbol.relocs
+            {
+                let target_symbol = self.get_symbol(namespace, name);
+                self.object
+                    .add_relocation(
+                        symbol.section,
+                        Relocation {
+                            offset: symbol.offset + u64::from(offset),
+                            size,
+                            kind,
+                            encoding,
+                            symbol: target_symbol,
+                            addend,
+                        },
+                    )
+                    .unwrap();
+            }
+        }
+
         ObjectProduct {
             object: self.object,
             functions: self.functions,
@@ -405,7 +394,7 @@ impl Backend for ObjectBackend {
 }
 
 impl ObjectBackend {
-    // This should only be called during finalization because it creates
+    // This should only be called during finish because it creates
     // symbols for missing libcalls.
     fn get_symbol(
         &mut self,
@@ -459,20 +448,8 @@ fn translate_linkage(linkage: Linkage) -> (SymbolScope, bool) {
     (scope, weak)
 }
 
-#[derive(Clone)]
-pub struct ObjectCompiledFunction {
-    offset: u64,
-    size: u32,
-    section: SectionId,
-    relocs: Vec,
-}
-
-#[derive(Clone)]
-pub struct ObjectCompiledData {
-    offset: u64,
-    section: SectionId,
-    relocs: Vec,
-}
+pub struct ObjectCompiledFunction;
+pub struct ObjectCompiledData;
 
 /// This is the output of `Module`'s
 /// [`finish`](../cranelift_module/struct.Module.html#method.finish) function.
@@ -509,6 +486,13 @@ impl ObjectProduct {
     }
 }
 
+#[derive(Clone)]
+struct SymbolRelocs {
+    section: SectionId,
+    offset: u64,
+    relocs: Vec,
+}
+
 #[derive(Clone)]
 struct RelocRecord {
     offset: CodeOffset,
diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index 45c7d3d260..a4b7d1a1da 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -507,7 +507,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
     ///
     /// This method does not need to be called when access to the memory
     /// handle is not required.
-    fn finish(self) -> Self::Product {
+    fn finish(self, _namespace: &ModuleNamespace) -> Self::Product {
         self.memory
     }
 }

From 51229c3f58552ee854e3511d354db245343057e6 Mon Sep 17 00:00:00 2001
From: Philip Craig 
Date: Sat, 25 Jan 2020 13:19:09 +1000
Subject: [PATCH 3046/3084] cranelift-module: document that finalize methods
 may not be relevant

---
 cranelift/module/src/backend.rs | 9 +++++++++
 cranelift/module/src/module.rs  | 3 +++
 2 files changed, 12 insertions(+)

diff --git a/cranelift/module/src/backend.rs b/cranelift/module/src/backend.rs
index 9efe17a926..218070a5c0 100644
--- a/cranelift/module/src/backend.rs
+++ b/cranelift/module/src/backend.rs
@@ -119,6 +119,9 @@ where
 
     /// Perform all outstanding relocations on the given function. This requires all `Local`
     /// and `Export` entities referenced to be defined.
+    ///
+    /// This method is not relevant for `Backend` implementations that do not provide
+    /// `Backend::FinalizedFunction`.
     fn finalize_function(
         &mut self,
         id: FuncId,
@@ -131,6 +134,9 @@ where
 
     /// Perform all outstanding relocations on the given data object. This requires all
     /// `Local` and `Export` entities referenced to be defined.
+    ///
+    /// This method is not relevant for `Backend` implementations that do not provide
+    /// `Backend::FinalizedData`.
     fn finalize_data(
         &mut self,
         id: DataId,
@@ -142,6 +148,9 @@ where
     fn get_finalized_data(&self, data: &Self::CompiledData) -> Self::FinalizedData;
 
     /// "Publish" all finalized functions and data objects to their ultimate destinations.
+    ///
+    /// This method is not relevant for `Backend` implementations that do not provide
+    /// `Backend::FinalizedFunction` or `Backend::FinalizedData`.
     fn publish(&mut self);
 
     /// Consume this `Backend` and return a result. Some implementations may
diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index c6aa533807..6c2f109c20 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -648,6 +648,9 @@ where
     ///
     /// Use `get_finalized_function` and `get_finalized_data` to obtain the final
     /// artifacts.
+    ///
+    /// This method is not relevant for `Backend` implementations that do not provide
+    /// `Backend::FinalizedFunction` or `Backend::FinalizedData`.
     pub fn finalize_definitions(&mut self) {
         for func in self.functions_to_finalize.drain(..) {
             let info = &self.contents.functions[func];

From 58e5a62cde5212d6edacab31428f8336357a8b00 Mon Sep 17 00:00:00 2001
From: Y-Nak 
Date: Thu, 13 Feb 2020 16:56:08 +0900
Subject: [PATCH 3047/3084] Fix inverted result of is_leaf method

---
 cranelift/codegen/src/ir/function.rs  | 4 ++--
 cranelift/codegen/src/stack_layout.rs | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/cranelift/codegen/src/ir/function.rs b/cranelift/codegen/src/ir/function.rs
index 77c06964bf..1e72d2bc48 100644
--- a/cranelift/codegen/src/ir/function.rs
+++ b/cranelift/codegen/src/ir/function.rs
@@ -295,8 +295,8 @@ impl Function {
     /// to be confused with a "leaf function" in Windows terminology.
     pub fn is_leaf(&self) -> bool {
         // Conservative result: if there's at least one function signature referenced in this
-        // function, assume it may call.
-        !self.dfg.signatures.is_empty()
+        // function, assume it is not a leaf.
+        self.dfg.signatures.is_empty()
     }
 }
 
diff --git a/cranelift/codegen/src/stack_layout.rs b/cranelift/codegen/src/stack_layout.rs
index d04137ff10..55f3eb864f 100644
--- a/cranelift/codegen/src/stack_layout.rs
+++ b/cranelift/codegen/src/stack_layout.rs
@@ -47,7 +47,7 @@ pub fn layout_stack(
     let mut incoming_max = 0;
     let mut outgoing_max = 0;
     let mut min_align = alignment;
-    let mut must_align = is_leaf;
+    let mut must_align = !is_leaf;
 
     for slot in frame.values() {
         if slot.size > max_size {
@@ -145,7 +145,7 @@ mod tests {
         let sss = &mut StackSlots::new();
 
         // For all these test cases, assume it will call.
-        let is_leaf = true;
+        let is_leaf = false;
 
         // An empty layout should have 0-sized stack frame.
         assert_eq!(layout_stack(sss, is_leaf, 1), Ok(0));

From 4d8cf563f3c943e10b861c28de7f8f06520cc236 Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Thu, 13 Feb 2020 16:25:41 -0500
Subject: [PATCH 3048/3084] Use zeroinit API for faerie and object (#1209)

* use new zeroinit API for faerie

* use bss for cranelift-object

* don't crash when initializing bss

* fix formatting

* Improve code locality

Co-Authored-By: Philip Craig 

* use `as` instead of try_into() for usize -> u64

* don't allocate unnecessarily in `faerie`

Co-authored-by: Philip Craig 
---
 cranelift/faerie/src/backend.rs | 33 ++++++++++++++---------------
 cranelift/object/src/backend.rs | 37 ++++++++++++++++-----------------
 2 files changed, 34 insertions(+), 36 deletions(-)

diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index f10d457778..c0cb086a6d 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -217,20 +217,6 @@ impl Backend for FaerieBackend {
             ref data_relocs,
         } = data_ctx.description();
 
-        let size = init.size();
-        let mut bytes = Vec::with_capacity(size);
-        match *init {
-            Init::Uninitialized => {
-                panic!("data is not initialized yet");
-            }
-            Init::Zeros { .. } => {
-                bytes.resize(size, 0);
-            }
-            Init::Bytes { ref contents } => {
-                bytes.extend_from_slice(contents);
-            }
-        }
-
         for &(offset, id) in function_relocs {
             let to = &namespace.get_function_decl(&function_decls[id]).name;
             self.artifact
@@ -256,9 +242,22 @@ impl Backend for FaerieBackend {
                 .map_err(|e| ModuleError::Backend(e.to_string()))?;
         }
 
-        self.artifact
-            .define(name, bytes)
-            .expect("inconsistent declaration");
+        match *init {
+            Init::Uninitialized => {
+                panic!("data is not initialized yet");
+            }
+            Init::Zeros { size } => {
+                self.artifact
+                    .define_zero_init(name, size)
+                    .expect("inconsistent declaration");
+            }
+            Init::Bytes { ref contents } => {
+                self.artifact
+                    .define(name, contents.to_vec())
+                    .expect("inconsistent declaration");
+            }
+        }
+
         Ok(FaerieCompiledData {})
     }
 
diff --git a/cranelift/object/src/backend.rs b/cranelift/object/src/backend.rs
index 0859046685..85af3b5a5d 100644
--- a/cranelift/object/src/backend.rs
+++ b/cranelift/object/src/backend.rs
@@ -243,20 +243,6 @@ impl Backend for ObjectBackend {
             ref data_relocs,
         } = data_ctx.description();
 
-        let size = init.size();
-        let mut data = Vec::with_capacity(size);
-        match *init {
-            Init::Uninitialized => {
-                panic!("data is not initialized yet");
-            }
-            Init::Zeros { .. } => {
-                data.resize(size, 0);
-            }
-            Init::Bytes { ref contents } => {
-                data.extend_from_slice(contents);
-            }
-        }
-
         let reloc_size = match self.isa.triple().pointer_width().unwrap() {
             PointerWidth::U16 => 16,
             PointerWidth::U32 => 32,
@@ -285,16 +271,29 @@ impl Backend for ObjectBackend {
         }
 
         let symbol = self.data_objects[data_id].unwrap();
-        let section = self.object.section_id(if writable {
+        let section_kind = if let Init::Zeros { .. } = *init {
+            StandardSection::UninitializedData
+        } else if writable {
             StandardSection::Data
         } else if relocs.is_empty() {
             StandardSection::ReadOnlyData
         } else {
             StandardSection::ReadOnlyDataWithRel
-        });
-        let offset =
-            self.object
-                .add_symbol_data(symbol, section, &data, u64::from(align.unwrap_or(1)));
+        };
+        let section = self.object.section_id(section_kind);
+
+        let align = u64::from(align.unwrap_or(1));
+        let offset = match *init {
+            Init::Uninitialized => {
+                panic!("data is not initialized yet");
+            }
+            Init::Zeros { size } => self
+                .object
+                .add_symbol_bss(symbol, section, size as u64, align),
+            Init::Bytes { ref contents } => self
+                .object
+                .add_symbol_data(symbol, section, &contents, align),
+        };
         if !relocs.is_empty() {
             self.relocs.push(SymbolRelocs {
                 section,

From 18b40d1101b0e059182e6b3ff5a3b8096fc256a3 Mon Sep 17 00:00:00 2001
From: Peter Delevoryas 
Date: Fri, 14 Feb 2020 13:16:02 -0800
Subject: [PATCH 3049/3084] Add ineg legalization for scalar integer types
 (#1385)

---
 .../codegen/meta/src/isa/x86/legalize.rs      | 15 +++++++
 cranelift/codegen/meta/src/isa/x86/mod.rs     | 13 +++---
 cranelift/codegen/src/isa/x86/enc_tables.rs   | 16 +++++---
 .../filetests/isa/x86/legalize-custom.clif    | 41 +++++++++++++++++++
 .../filetests/isa/x86/legalize-i64.clif       | 17 ++++++++
 .../isa/x86/legalize-ineg-x86_64.clif         | 13 ++++++
 6 files changed, 103 insertions(+), 12 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/legalize-ineg-x86_64.clif

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index c6161cf43b..0d0ab76a57 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -319,6 +319,8 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         ],
     );
 
+    group.custom_legalize(ineg, "convert_ineg");
+
     group.build_and_add_to(&mut shared.transform_groups);
 
     let mut narrow = TransformGroupBuilder::new(
@@ -612,4 +614,17 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     narrow.custom_legalize(ineg, "convert_ineg");
 
     narrow.build_and_add_to(&mut shared.transform_groups);
+
+    let mut widen = TransformGroupBuilder::new(
+        "x86_widen",
+        r#"
+    Legalize instructions by widening.
+
+    Use x86-specific instructions if needed."#,
+    )
+    .isa("x86")
+    .chain_with(shared.transform_groups.by_name("widen").id);
+
+    widen.custom_legalize(ineg, "convert_ineg");
+    widen.build_and_add_to(&mut shared.transform_groups);
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index ab1b5b4053..519ad884ff 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -32,16 +32,15 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     let mut x86_32 = CpuMode::new("I32");
 
     let expand_flags = shared_defs.transform_groups.by_name("expand_flags");
-    let narrow_flags = shared_defs.transform_groups.by_name("narrow_flags");
-    let widen = shared_defs.transform_groups.by_name("widen");
+    let x86_widen = shared_defs.transform_groups.by_name("x86_widen");
     let x86_narrow = shared_defs.transform_groups.by_name("x86_narrow");
     let x86_expand = shared_defs.transform_groups.by_name("x86_expand");
 
     x86_32.legalize_monomorphic(expand_flags);
-    x86_32.legalize_default(narrow_flags);
+    x86_32.legalize_default(x86_narrow);
     x86_32.legalize_type(B1, expand_flags);
-    x86_32.legalize_type(I8, widen);
-    x86_32.legalize_type(I16, widen);
+    x86_32.legalize_type(I8, x86_widen);
+    x86_32.legalize_type(I16, x86_widen);
     x86_32.legalize_type(I32, x86_expand);
     x86_32.legalize_value_type(ReferenceType(R32), x86_expand);
     x86_32.legalize_type(F32, x86_expand);
@@ -50,8 +49,8 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
     x86_64.legalize_monomorphic(expand_flags);
     x86_64.legalize_default(x86_narrow);
     x86_64.legalize_type(B1, expand_flags);
-    x86_64.legalize_type(I8, widen);
-    x86_64.legalize_type(I16, widen);
+    x86_64.legalize_type(I8, x86_widen);
+    x86_64.legalize_type(I16, x86_widen);
     x86_64.legalize_type(I32, x86_expand);
     x86_64.legalize_type(I64, x86_expand);
     x86_64.legalize_value_type(ReferenceType(R64), x86_expand);
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 947b315cd2..a93be1d658 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -1213,7 +1213,7 @@ fn convert_insertlane(
     }
 }
 
-/// For SIMD negation, convert an `ineg` to a `vconst + isub`.
+/// For SIMD or scalar integer negation, convert `ineg` to `vconst + isub` or `iconst + isub`.
 fn convert_ineg(
     inst: ir::Inst,
     func: &mut ir::Function,
@@ -1229,10 +1229,16 @@ fn convert_ineg(
     } = pos.func.dfg[inst]
     {
         let value_type = pos.func.dfg.value_type(arg);
-        if value_type.is_vector() && value_type.lane_type().is_int() {
+        let zero_value = if value_type.is_vector() && value_type.lane_type().is_int() {
             let zero_immediate = pos.func.dfg.constants.insert(vec![0; 16].into());
-            let zero_value = pos.ins().vconst(value_type, zero_immediate); // this should be legalized to a PXOR
-            pos.func.dfg.replace(inst).isub(zero_value, arg);
-        }
+            pos.ins().vconst(value_type, zero_immediate) // this should be legalized to a PXOR
+        } else if value_type.is_int() {
+            pos.ins().iconst(value_type, 0)
+        } else {
+            panic!("Can't convert ineg of type {}", value_type)
+        };
+        pos.func.dfg.replace(inst).isub(zero_value, arg);
+    } else {
+        unreachable!()
     }
 }
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-custom.clif b/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
index c2bc6bec19..3ee674d66a 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-custom.clif
@@ -90,3 +90,44 @@ block0(v0: f32, v1: f32):
     ; check: $done(v2: f32):
     ; nextln: return v2
 }
+
+function %ineg_legalized_i8() {
+block0:
+    v0 = iconst.i8 1
+    v1 = ineg v0
+    ; check: v2 = iconst.i32 1
+    ; nextln: v0 = ireduce.i8 v2
+    ; nextln: v3 = iconst.i8 0
+    ; nextln: v4 = uextend.i32 v3
+    ; nextln: v5 = uextend.i32 v0
+    ; nextln: v6 = isub v4, v5
+    ; nextln: v1 = ireduce.i8 v6
+
+    return
+}
+
+function %ineg_legalized_i16() {
+block0:
+    v0 = iconst.i16 1
+    v1 = ineg v0
+    ; check: v2 = iconst.i32 1
+    ; nextln: v0 = ireduce.i16 v2
+    ; nextln: v3 = iconst.i16 0
+    ; nextln: v4 = uextend.i32 v3
+    ; nextln: v5 = uextend.i32 v0
+    ; nextln: v6 = isub v4, v5
+    ; nextln: v1 = ireduce.i16 v6
+
+    return
+}
+
+function %ineg_legalized_i32() {
+block0:
+    v0 = iconst.i32 1
+    v1 = ineg v0
+    ; check: v0 = iconst.i32 1
+    ; nextln: v2 = iconst.i32 0
+    ; nextln: v1 = isub v2, v0
+
+    return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-i64.clif b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
index 0d1eea5152..94fbc82015 100644
--- a/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
+++ b/cranelift/filetests/filetests/isa/x86/legalize-i64.clif
@@ -338,3 +338,20 @@ block0(v1: i64):
     ; nextln: v10 = bor $b1, $c2
     return v10
 }
+
+function %ineg_legalized_i64() {
+block0:
+    v0 = iconst.i64 1
+    v1 = ineg v0
+    ; check: v2 = iconst.i32 1
+    ; nextln: v3 = iconst.i32 0
+    ; nextln: v0 = iconcat v2, v3
+    ; nextln: v5 = iconst.i32 0
+    ; nextln: v6 = iconst.i32 0
+    ; nextln: v4 = iconcat v5, v6
+    ; nextln: v7, v8 = isub_ifbout v5, v2
+    ; nextln: v9 = isub_ifbin v6, v3, v8
+    ; nextln: v1 = iconcat v7, v9
+
+    return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/legalize-ineg-x86_64.clif b/cranelift/filetests/filetests/isa/x86/legalize-ineg-x86_64.clif
new file mode 100644
index 0000000000..fac17d6ff1
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/legalize-ineg-x86_64.clif
@@ -0,0 +1,13 @@
+; Test the custom legalization of ineg.i64 on x86_64.
+test legalizer
+target x86_64
+
+function %ineg_legalized_i64() {
+block0:
+    v0 = iconst.i64 1
+    v1 = ineg v0
+    ; check: v0 = iconst.i64 1
+    ; nextln: v2 = iconst.i64 0
+    ; nextln: v1 = isub v2, v0
+    return
+}

From 45cc95e60ec82dafa32c2f82444131892a3ea280 Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Fri, 14 Feb 2020 23:44:40 +0100
Subject: [PATCH 3050/3084] Merge emit_small_memcpy and emit_small_memmove
 (#1301)

* Merge emit_small_memcpy and emit_small_memmove

* Fix typo
---
 cranelift/frontend/src/frontend.rs | 85 ++++++++++--------------------
 1 file changed, 27 insertions(+), 58 deletions(-)

diff --git a/cranelift/frontend/src/frontend.rs b/cranelift/frontend/src/frontend.rs
index f81ccc88bb..a5803cbcf6 100644
--- a/cranelift/frontend/src/frontend.rs
+++ b/cranelift/frontend/src/frontend.rs
@@ -1,7 +1,6 @@
 //! A frontend for building Cranelift IR from other languages.
 use crate::ssa::{SSABlock, SSABuilder, SideEffects};
 use crate::variable::Variable;
-use alloc::vec::Vec;
 use cranelift_codegen::cursor::{Cursor, FuncCursor};
 use cranelift_codegen::entity::{EntitySet, SecondaryMap};
 use cranelift_codegen::ir;
@@ -626,8 +625,15 @@ impl<'a> FunctionBuilder<'a> {
         self.ins().call(libc_memcpy, &[dest, src, size]);
     }
 
-    /// Optimised memcpy for small copies.
-    pub fn emit_small_memcpy(
+    /// Optimised memcpy or memmove for small copies.
+    ///
+    /// # Codegen safety
+    ///
+    /// The following properties must hold to prevent UB:
+    ///
+    /// * `src_align` and `dest_align` are an upper-bound on the alignment of `src` respectively `dest`.
+    /// * If `non_overlapping` is true, then this must be correct.
+    pub fn emit_small_memory_copy(
         &mut self,
         config: TargetFrontendConfig,
         dest: Value,
@@ -635,6 +641,7 @@ impl<'a> FunctionBuilder<'a> {
         size: u64,
         dest_align: u8,
         src_align: u8,
+        non_overlapping: bool,
     ) {
         // Currently the result of guess work, not actual profiling.
         const THRESHOLD: u64 = 4;
@@ -663,16 +670,27 @@ impl<'a> FunctionBuilder<'a> {
 
         if load_and_store_amount > THRESHOLD {
             let size_value = self.ins().iconst(config.pointer_type(), size as i64);
-            self.call_memcpy(config, dest, src, size_value);
+            if non_overlapping {
+                self.call_memcpy(config, dest, src, size_value);
+            } else {
+                self.call_memmove(config, dest, src, size_value);
+            }
             return;
         }
 
         let mut flags = MemFlags::new();
         flags.set_aligned();
 
-        for i in 0..load_and_store_amount {
-            let offset = (access_size * i) as i32;
-            let value = self.ins().load(int_type, flags, src, offset);
+        // Load all of the memory first. This is necessary in case `dest` overlaps.
+        // It can also improve performance a bit.
+        let registers: smallvec::SmallVec<[_; THRESHOLD as usize]> = (0..load_and_store_amount)
+            .map(|i| {
+                let offset = (access_size * i) as i32;
+                (self.ins().load(int_type, flags, src, offset), offset)
+            })
+            .collect();
+
+        for (value, offset) in registers {
             self.ins().store(flags, value, dest, offset);
         }
     }
@@ -798,55 +816,6 @@ impl<'a> FunctionBuilder<'a> {
 
         self.ins().call(libc_memmove, &[dest, source, size]);
     }
-
-    /// Optimised memmove for small moves.
-    pub fn emit_small_memmove(
-        &mut self,
-        config: TargetFrontendConfig,
-        dest: Value,
-        src: Value,
-        size: u64,
-        dest_align: u8,
-        src_align: u8,
-    ) {
-        // Currently the result of guess work, not actual profiling.
-        const THRESHOLD: u64 = 4;
-
-        let access_size = greatest_divisible_power_of_two(size);
-        assert!(
-            access_size.is_power_of_two(),
-            "`size` is not a power of two"
-        );
-        assert!(
-            access_size >= u64::from(::core::cmp::min(src_align, dest_align)),
-            "`size` is smaller than `dest` and `src`'s alignment value."
-        );
-        let load_and_store_amount = size / access_size;
-
-        if load_and_store_amount > THRESHOLD {
-            let size_value = self.ins().iconst(config.pointer_type(), size as i64);
-            self.call_memmove(config, dest, src, size_value);
-            return;
-        }
-
-        let mut flags = MemFlags::new();
-        flags.set_aligned();
-
-        // Load all of the memory first in case `dest` overlaps.
-        let registers: Vec<_> = (0..load_and_store_amount)
-            .map(|i| {
-                let offset = (access_size * i) as i32;
-                (
-                    self.ins().load(config.pointer_type(), flags, src, offset),
-                    offset,
-                )
-            })
-            .collect();
-
-        for (value, offset) in registers {
-            self.ins().store(flags, value, dest, offset);
-        }
-    }
 }
 
 fn greatest_divisible_power_of_two(size: u64) -> u64 {
@@ -1104,7 +1073,7 @@ block0:
             let src = builder.use_var(x);
             let dest = builder.use_var(y);
             let size = 8;
-            builder.emit_small_memcpy(target.frontend_config(), dest, src, size, 8, 8);
+            builder.emit_small_memory_copy(target.frontend_config(), dest, src, size, 8, 8, true);
             builder.ins().return_(&[dest]);
 
             builder.seal_all_blocks();
@@ -1161,7 +1130,7 @@ block0:
             let src = builder.use_var(x);
             let dest = builder.use_var(y);
             let size = 8192;
-            builder.emit_small_memcpy(target.frontend_config(), dest, src, size, 8, 8);
+            builder.emit_small_memory_copy(target.frontend_config(), dest, src, size, 8, 8, true);
             builder.ins().return_(&[dest]);
 
             builder.seal_all_blocks();

From 9b3ac10ebcc37e2d2d1d14b66763344011c5d777 Mon Sep 17 00:00:00 2001
From: Nick Fitzgerald 
Date: Sat, 15 Feb 2020 14:53:32 -0800
Subject: [PATCH 3051/3084] wasm: Add support for passive data and element
 segments (#1389)

This is part of the bulk memory and reference types proposals.
---
 cranelift/wasm/Cargo.toml                 |   2 +-
 cranelift/wasm/src/environ/dummy.rs       |  20 ++-
 cranelift/wasm/src/environ/spec.rs        |  26 +++-
 cranelift/wasm/src/lib.rs                 |   2 +-
 cranelift/wasm/src/module_translator.rs   |   9 +-
 cranelift/wasm/src/sections_translator.rs | 145 ++++++++++++----------
 cranelift/wasm/src/translation_utils.rs   |  10 ++
 cranelift/wasmtests/passive-data.wat      |  11 ++
 cranelift/wasmtests/table-copy.wat        |  22 ++++
 9 files changed, 169 insertions(+), 78 deletions(-)
 create mode 100644 cranelift/wasmtests/passive-data.wat
 create mode 100644 cranelift/wasmtests/table-copy.wat

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index cc5cc3d90e..f9f2b8b4bd 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -21,7 +21,7 @@ serde = { version = "1.0.94", features = ["derive"], optional = true }
 thiserror = "1.0.4"
 
 [dev-dependencies]
-wat = "1.0.7"
+wat = "1.0.9"
 target-lexicon = "0.10"
 
 [features]
diff --git a/cranelift/wasm/src/environ/dummy.rs b/cranelift/wasm/src/environ/dummy.rs
index a36a0bce70..4ebf0bdc29 100644
--- a/cranelift/wasm/src/environ/dummy.rs
+++ b/cranelift/wasm/src/environ/dummy.rs
@@ -11,8 +11,8 @@ use crate::environ::{
 use crate::func_translator::FuncTranslator;
 use crate::state::ModuleTranslationState;
 use crate::translation_utils::{
-    DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table,
-    TableIndex,
+    DefinedFuncIndex, FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, PassiveDataIndex,
+    PassiveElemIndex, SignatureIndex, Table, TableIndex,
 };
 use core::convert::TryFrom;
 use cranelift_codegen::cursor::FuncCursor;
@@ -605,6 +605,22 @@ impl<'data> ModuleEnvironment<'data> for DummyEnvironment {
         Ok(())
     }
 
+    fn declare_passive_element(
+        &mut self,
+        _elem_index: PassiveElemIndex,
+        _segments: Box<[FuncIndex]>,
+    ) -> WasmResult<()> {
+        Ok(())
+    }
+
+    fn declare_passive_data(
+        &mut self,
+        _elem_index: PassiveDataIndex,
+        _segments: &'data [u8],
+    ) -> WasmResult<()> {
+        Ok(())
+    }
+
     fn declare_memory(&mut self, memory: Memory) -> WasmResult<()> {
         self.info.memories.push(Exportable::new(memory));
         Ok(())
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 123a8d0c09..73a02c6709 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -8,7 +8,8 @@
 
 use crate::state::{FuncTranslationState, ModuleTranslationState};
 use crate::translation_utils::{
-    FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, SignatureIndex, Table, TableIndex,
+    FuncIndex, Global, GlobalIndex, Memory, MemoryIndex, PassiveDataIndex, PassiveElemIndex,
+    SignatureIndex, Table, TableIndex,
 };
 use core::convert::From;
 use cranelift_codegen::cursor::FuncCursor;
@@ -600,6 +601,29 @@ pub trait ModuleEnvironment<'data>: TargetEnvironment {
         elements: Box<[FuncIndex]>,
     ) -> WasmResult<()>;
 
+    /// Declare a passive element segment.
+    fn declare_passive_element(
+        &mut self,
+        index: PassiveElemIndex,
+        elements: Box<[FuncIndex]>,
+    ) -> WasmResult<()>;
+
+    /// Provides the number of passive data segments up front.
+    ///
+    /// By default this does nothing, but implementations may use this to
+    /// pre-allocate memory if desired.
+    fn reserve_passive_data(&mut self, count: u32) -> WasmResult<()> {
+        let _ = count;
+        Ok(())
+    }
+
+    /// Declare a passive data segment.
+    fn declare_passive_data(
+        &mut self,
+        data_index: PassiveDataIndex,
+        data: &'data [u8],
+    ) -> WasmResult<()>;
+
     /// Provides the contents of a function body.
     ///
     /// Note there's no `reserve_function_bodies` function because the number of
diff --git a/cranelift/wasm/src/lib.rs b/cranelift/wasm/src/lib.rs
index 28e4361126..b98c95b466 100644
--- a/cranelift/wasm/src/lib.rs
+++ b/cranelift/wasm/src/lib.rs
@@ -68,7 +68,7 @@ pub use crate::state::module_state::ModuleTranslationState;
 pub use crate::translation_utils::{
     get_vmctx_value_label, DefinedFuncIndex, DefinedGlobalIndex, DefinedMemoryIndex,
     DefinedTableIndex, FuncIndex, Global, GlobalIndex, GlobalInit, Memory, MemoryIndex,
-    SignatureIndex, Table, TableElementType, TableIndex,
+    PassiveDataIndex, PassiveElemIndex, SignatureIndex, Table, TableElementType, TableIndex,
 };
 
 /// Version number of this crate.
diff --git a/cranelift/wasm/src/module_translator.rs b/cranelift/wasm/src/module_translator.rs
index 257daa1f20..e31e6f09f8 100644
--- a/cranelift/wasm/src/module_translator.rs
+++ b/cranelift/wasm/src/module_translator.rs
@@ -1,6 +1,6 @@
 //! Translation skeleton that traverses the whole WebAssembly module and call helper functions
 //! to deal with each part of it.
-use crate::environ::{ModuleEnvironment, WasmError, WasmResult};
+use crate::environ::{ModuleEnvironment, WasmResult};
 use crate::sections_translator::{
     parse_code_section, parse_data_section, parse_element_section, parse_export_section,
     parse_function_section, parse_global_section, parse_import_section, parse_memory_section,
@@ -67,11 +67,8 @@ pub fn translate_module<'data>(
                 parse_data_section(data, environ)?;
             }
 
-            SectionContent::DataCount(_) => {
-                return Err(WasmError::InvalidWebAssembly {
-                    message: "don't know how to handle the data count section yet",
-                    offset: reader.current_position(),
-                });
+            SectionContent::DataCount(count) => {
+                environ.reserve_passive_data(count)?;
             }
 
             SectionContent::Custom {
diff --git a/cranelift/wasm/src/sections_translator.rs b/cranelift/wasm/src/sections_translator.rs
index f4c5079a6e..6759a60c3f 100644
--- a/cranelift/wasm/src/sections_translator.rs
+++ b/cranelift/wasm/src/sections_translator.rs
@@ -11,7 +11,8 @@ use crate::environ::{ModuleEnvironment, WasmError, WasmResult};
 use crate::state::ModuleTranslationState;
 use crate::translation_utils::{
     tabletype_to_type, type_to_type, FuncIndex, Global, GlobalIndex, GlobalInit, Memory,
-    MemoryIndex, SignatureIndex, Table, TableElementType, TableIndex,
+    MemoryIndex, PassiveDataIndex, PassiveElemIndex, SignatureIndex, Table, TableElementType,
+    TableIndex,
 };
 use crate::{wasm_unsupported, HashMap};
 use core::convert::TryFrom;
@@ -19,10 +20,11 @@ use cranelift_codegen::ir::immediates::V128Imm;
 use cranelift_codegen::ir::{self, AbiParam, Signature};
 use cranelift_entity::packed_option::ReservedValue;
 use cranelift_entity::EntityRef;
+use std::boxed::Box;
 use std::vec::Vec;
 use wasmparser::{
-    self, CodeSectionReader, Data, DataKind, DataSectionReader, Element, ElementItem, ElementKind,
-    ElementSectionReader, Export, ExportSectionReader, ExternalKind, FuncType,
+    self, CodeSectionReader, Data, DataKind, DataSectionReader, Element, ElementItem, ElementItems,
+    ElementKind, ElementSectionReader, Export, ExportSectionReader, ExternalKind, FuncType,
     FunctionSectionReader, GlobalSectionReader, GlobalType, ImportSectionEntryType,
     ImportSectionReader, MemorySectionReader, MemoryType, NameSectionReader, Naming, NamingReader,
     Operator, TableSectionReader, Type, TypeSectionReader,
@@ -288,6 +290,19 @@ pub fn parse_start_section(index: u32, environ: &mut dyn ModuleEnvironment) -> W
     Ok(())
 }
 
+fn read_elems(items: &ElementItems) -> WasmResult> {
+    let items_reader = items.get_items_reader()?;
+    let mut elems = Vec::with_capacity(usize::try_from(items_reader.get_count()).unwrap());
+    for item in items_reader {
+        let elem = match item? {
+            ElementItem::Null => FuncIndex::reserved_value(),
+            ElementItem::Func(index) => FuncIndex::from_u32(index),
+        };
+        elems.push(elem);
+    }
+    Ok(elems.into_boxed_slice())
+}
+
 /// Parses the Element section of the wasm module.
 pub fn parse_element_section<'data>(
     elements: ElementSectionReader<'data>,
@@ -295,7 +310,7 @@ pub fn parse_element_section<'data>(
 ) -> WasmResult<()> {
     environ.reserve_table_elements(elements.get_count())?;
 
-    for entry in elements {
+    for (index, entry) in elements.into_iter().enumerate() {
         let Element { kind, items, ty } = entry?;
         if ty != Type::AnyFunc {
             return Err(wasm_unsupported!(
@@ -303,41 +318,37 @@ pub fn parse_element_section<'data>(
                 ty
             ));
         }
-        if let ElementKind::Active {
-            table_index,
-            init_expr,
-        } = kind
-        {
-            let mut init_expr_reader = init_expr.get_binary_reader();
-            let (base, offset) = match init_expr_reader.read_operator()? {
-                Operator::I32Const { value } => (None, value as u32 as usize),
-                Operator::GlobalGet { global_index } => {
-                    (Some(GlobalIndex::from_u32(global_index)), 0)
-                }
-                ref s => {
-                    return Err(wasm_unsupported!(
-                        "unsupported init expr in element section: {:?}",
-                        s
-                    ));
-                }
-            };
-            let items_reader = items.get_items_reader()?;
-            let mut elems = Vec::with_capacity(usize::try_from(items_reader.get_count()).unwrap());
-            for item in items_reader {
-                let elem = match item? {
-                    ElementItem::Null => FuncIndex::reserved_value(),
-                    ElementItem::Func(index) => FuncIndex::from_u32(index),
+        let segments = read_elems(&items)?;
+        match kind {
+            ElementKind::Active {
+                table_index,
+                init_expr,
+            } => {
+                let mut init_expr_reader = init_expr.get_binary_reader();
+                let (base, offset) = match init_expr_reader.read_operator()? {
+                    Operator::I32Const { value } => (None, value as u32 as usize),
+                    Operator::GlobalGet { global_index } => {
+                        (Some(GlobalIndex::from_u32(global_index)), 0)
+                    }
+                    ref s => {
+                        return Err(wasm_unsupported!(
+                            "unsupported init expr in element section: {:?}",
+                            s
+                        ));
+                    }
                 };
-                elems.push(elem);
+                environ.declare_table_elements(
+                    TableIndex::from_u32(table_index),
+                    base,
+                    offset,
+                    segments,
+                )?
             }
-            environ.declare_table_elements(
-                TableIndex::from_u32(table_index),
-                base,
-                offset,
-                elems.into_boxed_slice(),
-            )?
-        } else {
-            return Err(wasm_unsupported!("unsupported passive elements section",));
+            ElementKind::Passive => {
+                let index = PassiveElemIndex::from_u32(index as u32);
+                environ.declare_passive_element(index, segments)?;
+            }
+            ElementKind::Declared => return Err(wasm_unsupported!("element kind declared")),
         }
     }
     Ok(())
@@ -365,37 +376,37 @@ pub fn parse_data_section<'data>(
 ) -> WasmResult<()> {
     environ.reserve_data_initializers(data.get_count())?;
 
-    for entry in data {
+    for (index, entry) in data.into_iter().enumerate() {
         let Data { kind, data } = entry?;
-        if let DataKind::Active {
-            memory_index,
-            init_expr,
-        } = kind
-        {
-            let mut init_expr_reader = init_expr.get_binary_reader();
-            let (base, offset) = match init_expr_reader.read_operator()? {
-                Operator::I32Const { value } => (None, value as u32 as usize),
-                Operator::GlobalGet { global_index } => {
-                    (Some(GlobalIndex::from_u32(global_index)), 0)
-                }
-                ref s => {
-                    return Err(wasm_unsupported!(
-                        "unsupported init expr in data section: {:?}",
-                        s
-                    ))
-                }
-            };
-            environ.declare_data_initialization(
-                MemoryIndex::from_u32(memory_index),
-                base,
-                offset,
-                data,
-            )?;
-        } else {
-            return Err(wasm_unsupported!(
-                "unsupported passive data section: {:?}",
-                kind
-            ));
+        match kind {
+            DataKind::Active {
+                memory_index,
+                init_expr,
+            } => {
+                let mut init_expr_reader = init_expr.get_binary_reader();
+                let (base, offset) = match init_expr_reader.read_operator()? {
+                    Operator::I32Const { value } => (None, value as u32 as usize),
+                    Operator::GlobalGet { global_index } => {
+                        (Some(GlobalIndex::from_u32(global_index)), 0)
+                    }
+                    ref s => {
+                        return Err(wasm_unsupported!(
+                            "unsupported init expr in data section: {:?}",
+                            s
+                        ))
+                    }
+                };
+                environ.declare_data_initialization(
+                    MemoryIndex::from_u32(memory_index),
+                    base,
+                    offset,
+                    data,
+                )?;
+            }
+            DataKind::Passive => {
+                let index = PassiveDataIndex::from_u32(index as u32);
+                environ.declare_passive_data(index, data)?;
+            }
         }
     }
 
diff --git a/cranelift/wasm/src/translation_utils.rs b/cranelift/wasm/src/translation_utils.rs
index cd19ed820f..8a19761a88 100644
--- a/cranelift/wasm/src/translation_utils.rs
+++ b/cranelift/wasm/src/translation_utils.rs
@@ -57,6 +57,16 @@ entity_impl!(MemoryIndex);
 pub struct SignatureIndex(u32);
 entity_impl!(SignatureIndex);
 
+/// Index type of a passive data segment inside the WebAssembly module.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
+pub struct PassiveDataIndex(u32);
+entity_impl!(PassiveDataIndex);
+
+/// Index type of a passive element segment inside the WebAssembly module.
+#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
+pub struct PassiveElemIndex(u32);
+entity_impl!(PassiveElemIndex);
+
 /// WebAssembly global.
 #[derive(Debug, Clone, Copy, Hash)]
 pub struct Global {
diff --git a/cranelift/wasmtests/passive-data.wat b/cranelift/wasmtests/passive-data.wat
new file mode 100644
index 0000000000..9316cd0f59
--- /dev/null
+++ b/cranelift/wasmtests/passive-data.wat
@@ -0,0 +1,11 @@
+(module
+  (data $passive "this is a passive data segment")
+
+  (func (export "init") (param i32 i32 i32)
+    local.get 0 ;; dst
+    local.get 1 ;; src
+    local.get 2 ;; cnt
+    memory.init $passive)
+
+  (func (export "drop")
+    data.drop $passive))
diff --git a/cranelift/wasmtests/table-copy.wat b/cranelift/wasmtests/table-copy.wat
new file mode 100644
index 0000000000..dd9e1611e6
--- /dev/null
+++ b/cranelift/wasmtests/table-copy.wat
@@ -0,0 +1,22 @@
+(module $n
+  (table $t (import "m" "t") 6 funcref)
+
+  (func $i (param i32 i32 i32 i32 i32 i32) (result i32) (local.get 3))
+  (func $j (param i32 i32 i32 i32 i32 i32) (result i32) (local.get 4))
+  (func $k (param i32 i32 i32 i32 i32 i32) (result i32) (local.get 5))
+
+  (table $u (export "u") funcref (elem $i $j $k $i $j $k))
+
+  (func (export "copy_to_t_from_u") (param i32 i32 i32 i32) (result i32)
+    local.get 0
+    local.get 1
+    local.get 2
+    local.get 3
+    table.copy $t $u)
+
+  (func (export "copy_to_u_from_t") (param i32 i32 i32 i32) (result i32)
+    local.get 0
+    local.get 1
+    local.get 2
+    local.get 3
+    table.copy $u $t))

From 936120dcf9250d18f18eaa72c8b4c95cc8f61003 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Wed, 12 Feb 2020 15:04:10 -0800
Subject: [PATCH 3052/3084] Infer REX prefix for SIMD `store` and `vconst`
 instructions

---
 .../codegen/meta/src/isa/x86/encodings.rs     | 11 ++++---
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 19 ++++++++++--
 cranelift/codegen/src/isa/x86/enc_tables.rs   | 29 +++++++++++++++++++
 .../filetests/filetests/isa/x86/binary64.clif |  8 +++++
 4 files changed, 60 insertions(+), 7 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index fd160e3e21..7e9f74475e 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1795,14 +1795,14 @@ fn define_simd(
 
         let is_zero_128bit =
             InstructionPredicate::new_is_all_zeroes(&*formats.unary_const, "constant_handle");
-        let template = rec_vconst_optimized.nonrex().opcodes(&PXOR);
+        let template = rec_vconst_optimized.opcodes(&PXOR).infer_rex();
         e.enc_32_64_func(instruction.clone(), template, |builder| {
             builder.inst_predicate(is_zero_128bit)
         });
 
         let is_ones_128bit =
             InstructionPredicate::new_is_all_ones(&*formats.unary_const, "constant_handle");
-        let template = rec_vconst_optimized.nonrex().opcodes(&PCMPEQB);
+        let template = rec_vconst_optimized.opcodes(&PCMPEQB).infer_rex();
         e.enc_32_64_func(instruction, template, |builder| {
             builder.inst_predicate(is_ones_128bit)
         });
@@ -1816,7 +1816,7 @@ fn define_simd(
     // in memory) but some performance measurements are needed.
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         let instruction = vconst.bind(vector(ty, sse_vector_size));
-        let template = rec_vconst.nonrex().opcodes(&MOVUPS_LOAD);
+        let template = rec_vconst.opcodes(&MOVUPS_LOAD).infer_rex();
         e.enc_32_64_maybe_isap(instruction, template, None); // from SSE
     }
 
@@ -1826,7 +1826,10 @@ fn define_simd(
     for ty in ValueType::all_lane_types().filter(allowed_simd_type) {
         // Store
         let bound_store = store.bind(vector(ty, sse_vector_size)).bind(Any);
-        e.enc_32_64(bound_store.clone(), rec_fst.opcodes(&MOVUPS_STORE));
+        e.enc_32_64(
+            bound_store.clone(),
+            rec_fst.opcodes(&MOVUPS_STORE).infer_rex(),
+        );
         e.enc_32_64(bound_store.clone(), rec_fstDisp8.opcodes(&MOVUPS_STORE));
         e.enc_32_64(bound_store, rec_fstDisp32.opcodes(&MOVUPS_STORE));
 
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index c512e98467..c9794a02da 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -46,6 +46,16 @@ impl<'builder> RecipeGroup<'builder> {
         self.templates.push(template.clone());
         template
     }
+    fn add_template_inferred(
+        &mut self,
+        recipe: EncodingRecipeBuilder,
+        infer_function: &'static str,
+    ) -> Rc> {
+        let template =
+            Rc::new(Template::new(recipe, self.regs).inferred_rex_compute_size(infer_function));
+        self.templates.push(template.clone());
+        template
+    }
     fn add_template(&mut self, template: Template<'builder>) -> Rc> {
         let template = Rc::new(template);
         self.templates.push(template.clone());
@@ -1481,7 +1491,7 @@ pub(crate) fn define<'shared>(
         );
 
         // XX /r register-indirect store of FPR with no offset.
-        recipes.add_template_recipe(
+        recipes.add_template_inferred(
             EncodingRecipeBuilder::new("fst", &formats.store, 1)
                 .operands_in(vec![fpr, gpr])
                 .inst_predicate(has_no_offset)
@@ -1504,6 +1514,7 @@ pub(crate) fn define<'shared>(
                         }
                     "#,
                 ),
+            "size_plus_maybe_sib_or_offset_inreg1_plus_rex_prefix_for_inreg0_inreg1",
         );
 
         let has_small_offset =
@@ -2515,7 +2526,7 @@ pub(crate) fn define<'shared>(
             ),
     );
 
-    recipes.add_template_recipe(
+    recipes.add_template_inferred(
         EncodingRecipeBuilder::new("vconst", &formats.unary_const, 5)
             .operands_out(vec![fpr])
             .clobbers_flags(false)
@@ -2526,9 +2537,10 @@ pub(crate) fn define<'shared>(
                     const_disp4(constant_handle, func, sink);
                 "#,
             ),
+        "size_with_inferred_rex_for_outreg0",
     );
 
-    recipes.add_template_recipe(
+    recipes.add_template_inferred(
         EncodingRecipeBuilder::new("vconst_optimized", &formats.unary_const, 1)
             .operands_out(vec![fpr])
             .clobbers_flags(false)
@@ -2538,6 +2550,7 @@ pub(crate) fn define<'shared>(
                     modrm_rr(out_reg0, out_reg0, sink);
                 "#,
             ),
+        "size_with_inferred_rex_for_outreg0",
     );
 
     recipes.add_template_recipe(
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index a93be1d658..27f7ed43db 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -123,6 +123,22 @@ fn size_plus_maybe_sib_or_offset_for_inreg_1(
     sizing.base_size + if needs_sib_or_offset { 1 } else { 0 }
 }
 
+/// Calculates the size while inferring if the first and second input registers (inreg0, inreg1)
+/// require a dynamic REX prefix and if the second input register (inreg1) requires a SIB or offset.
+fn size_plus_maybe_sib_or_offset_inreg1_plus_rex_prefix_for_inreg0_inreg1(
+    sizing: &RecipeSizing,
+    enc: Encoding,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    let needs_rex = (EncodingBits::from(enc.bits()).rex_w() != 0)
+        || test_input(0, inst, divert, func, is_extended_reg)
+        || test_input(1, inst, divert, func, is_extended_reg);
+    size_plus_maybe_sib_or_offset_for_inreg_1(sizing, enc, inst, divert, func)
+        + if needs_rex { 1 } else { 0 }
+}
+
 /// Infers whether a dynamic REX prefix will be emitted, for use with one input reg.
 ///
 /// A REX prefix is known to be emitted if either:
@@ -199,6 +215,19 @@ fn size_with_inferred_rex_for_inreg0_outreg0(
     sizing.base_size + if needs_rex { 1 } else { 0 }
 }
 
+/// Infers whether a dynamic REX prefix will be emitted, based on a single output register.
+fn size_with_inferred_rex_for_outreg0(
+    sizing: &RecipeSizing,
+    enc: Encoding,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    let needs_rex = (EncodingBits::from(enc.bits()).rex_w() != 0)
+        || test_result(0, inst, divert, func, is_extended_reg);
+    sizing.base_size + if needs_rex { 1 } else { 0 }
+}
+
 /// Infers whether a dynamic REX prefix will be emitted, for use with CMOV.
 ///
 /// CMOV uses 3 inputs, with the REX is inferred from reg1 and reg2.
diff --git a/cranelift/filetests/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
index 41290d1462..c241d33769 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64.clif
@@ -1679,3 +1679,11 @@ block0:
     [-, %r10]   v0 = bconst.b64 true    ; bin: 41 ba 00000001
     return
 }
+
+function %V128() {
+block0:
+    [-,%r10]            v3 = iconst.i64 0x2102_0304_f1f2_f3f4 ; bin: 49 ba 21020304f1f2f3f4
+    [-, %xmm9]          v4 = vconst.i32x4 [0 1 2 3]  ; bin: 44 0f 10 0d 00000005 PCRelRodata4(23)
+                        store v4, v3    ; bin: heap_oob 45 0f 11 0a
+                        return
+}

From 1a9dc743d137e3b849d2ec6aa69bb4bdf8aab25a Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 13 Feb 2020 10:46:47 -0800
Subject: [PATCH 3053/3084] Infer REX prefix for SIMD `load` instruction

---
 cranelift/codegen/meta/src/isa/x86/encodings.rs |  5 ++++-
 cranelift/codegen/meta/src/isa/x86/recipes.rs   |  3 ++-
 cranelift/codegen/src/isa/x86/enc_tables.rs     | 17 +++++++++++++++++
 .../filetests/filetests/isa/x86/binary64.clif   |  9 ++++++---
 4 files changed, 29 insertions(+), 5 deletions(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 7e9f74475e..f24232f39a 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1835,7 +1835,10 @@ fn define_simd(
 
         // Load
         let bound_load = load.bind(vector(ty, sse_vector_size)).bind(Any);
-        e.enc_32_64(bound_load.clone(), rec_fld.opcodes(&MOVUPS_LOAD));
+        e.enc_32_64(
+            bound_load.clone(),
+            rec_fld.opcodes(&MOVUPS_LOAD).infer_rex(),
+        );
         e.enc_32_64(bound_load.clone(), rec_fldDisp8.opcodes(&MOVUPS_LOAD));
         e.enc_32_64(bound_load, rec_fldDisp32.opcodes(&MOVUPS_LOAD));
 
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index c9794a02da..5c7cb0519b 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -2002,7 +2002,7 @@ pub(crate) fn define<'shared>(
         );
 
         // XX /r float load with no offset.
-        recipes.add_template_recipe(
+        recipes.add_template_inferred(
             EncodingRecipeBuilder::new("fld", &formats.load, 1)
                 .operands_in(vec![gpr])
                 .operands_out(vec![fpr])
@@ -2026,6 +2026,7 @@ pub(crate) fn define<'shared>(
                         }
                     "#,
                 ),
+            "size_plus_maybe_sib_or_offset_for_inreg_0_plus_rex_prefix_for_inreg0_outreg0",
         );
 
         let has_small_offset =
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 27f7ed43db..076fef3115 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -139,6 +139,23 @@ fn size_plus_maybe_sib_or_offset_inreg1_plus_rex_prefix_for_inreg0_inreg1(
         + if needs_rex { 1 } else { 0 }
 }
 
+/// Calculates the size while inferring if the first input register (inreg0) and first output
+/// register (outreg0) require a dynamic REX and if the first input register (inreg0) requires a
+/// SIB or offset.
+fn size_plus_maybe_sib_or_offset_for_inreg_0_plus_rex_prefix_for_inreg0_outreg0(
+    sizing: &RecipeSizing,
+    enc: Encoding,
+    inst: Inst,
+    divert: &RegDiversions,
+    func: &Function,
+) -> u8 {
+    let needs_rex = (EncodingBits::from(enc.bits()).rex_w() != 0)
+        || test_input(0, inst, divert, func, is_extended_reg)
+        || test_result(0, inst, divert, func, is_extended_reg);
+    size_plus_maybe_sib_or_offset_for_inreg_0(sizing, enc, inst, divert, func)
+        + if needs_rex { 1 } else { 0 }
+}
+
 /// Infers whether a dynamic REX prefix will be emitted, for use with one input reg.
 ///
 /// A REX prefix is known to be emitted if either:
diff --git a/cranelift/filetests/filetests/isa/x86/binary64.clif b/cranelift/filetests/filetests/isa/x86/binary64.clif
index c241d33769..ab5d516b40 100644
--- a/cranelift/filetests/filetests/isa/x86/binary64.clif
+++ b/cranelift/filetests/filetests/isa/x86/binary64.clif
@@ -1682,8 +1682,11 @@ block0:
 
 function %V128() {
 block0:
-    [-,%r10]            v3 = iconst.i64 0x2102_0304_f1f2_f3f4 ; bin: 49 ba 21020304f1f2f3f4
-    [-, %xmm9]          v4 = vconst.i32x4 [0 1 2 3]  ; bin: 44 0f 10 0d 00000005 PCRelRodata4(23)
-                        store v4, v3    ; bin: heap_oob 45 0f 11 0a
+    [-,%r10]            v3 = iconst.i64 0x2102_0304_f1f2_f3f4  ; bin: 49 ba 21020304f1f2f3f4
+    [-, %xmm9]          v4 = vconst.i32x4 [0 1 2 3]  ; bin: 44 0f 10 0d 0000000f PCRelRodata4(33)
+                        store v4, v3  ; bin: heap_oob 45 0f 11 0a
+
+    [-, %r11]           v5 = iconst.i64 0x1234
+    [-, %xmm2]          v6 = load.i32x4 v5  ; bin: heap_oob 41 0f 10 13
                         return
 }

From 7dfd159fd8378efc47a87000e040743c8d034a8a Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Thu, 20 Feb 2020 09:58:19 -0800
Subject: [PATCH 3054/3084] Avoid unused import warnings in generated legalizer
 code (#1393)

---
 cranelift/codegen/meta/src/gen_legalizer.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/gen_legalizer.rs b/cranelift/codegen/meta/src/gen_legalizer.rs
index c872ba32d6..759121894f 100644
--- a/cranelift/codegen/meta/src/gen_legalizer.rs
+++ b/cranelift/codegen/meta/src/gen_legalizer.rs
@@ -560,7 +560,7 @@ fn gen_transform_group<'a>(
     fmt: &mut Formatter,
 ) {
     fmt.doc_comment(group.doc);
-    fmt.line("#[allow(unused_variables,unused_assignments,non_snake_case)]");
+    fmt.line("#[allow(unused_variables,unused_assignments,unused_imports,non_snake_case)]");
 
     // Function arguments.
     fmtln!(fmt, "pub fn {}(", group.name);

From ec9700c70b4f949ff7bb9a48a7bcb3a4ae2ccb5e Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Sun, 16 Feb 2020 11:41:04 -0500
Subject: [PATCH 3055/3084] Don't return a Result for ObjectBuilder::new

Since it always returned an `Ok(Self)`, there was no reason to have a
Result.
---
 cranelift/object/src/backend.rs | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/cranelift/object/src/backend.rs b/cranelift/object/src/backend.rs
index 85af3b5a5d..f8baf1452c 100644
--- a/cranelift/object/src/backend.rs
+++ b/cranelift/object/src/backend.rs
@@ -54,14 +54,14 @@ impl ObjectBuilder {
         name: String,
         collect_traps: ObjectTrapCollection,
         libcall_names: Box String>,
-    ) -> ModuleResult {
-        Ok(Self {
+    ) -> Self {
+        Self {
             isa,
             name,
             collect_traps,
             libcall_names,
             function_alignment: 1,
-        })
+        }
     }
 
     /// Set the alignment used for functions.

From 3ae1af1ad299a11c264e6004aab01c02d14a7c42 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 31 Jan 2020 13:52:20 -0800
Subject: [PATCH 3056/3084] Add new Cranelift instructions for integer min/max

This includes legalizations to the previously-existing x86 SIMD integer min/max.
---
 .../codegen/meta/src/isa/x86/legalize.rs      | 17 +++++
 .../codegen/meta/src/shared/instructions.rs   | 70 +++++++++++++++++++
 2 files changed, 87 insertions(+)

diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index 0d0ab76a57..aa91741dba 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -43,6 +43,8 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let iadd = insts.by_name("iadd");
     let icmp = insts.by_name("icmp");
     let iconst = insts.by_name("iconst");
+    let imax = insts.by_name("imax");
+    let imin = insts.by_name("imin");
     let imul = insts.by_name("imul");
     let ineg = insts.by_name("ineg");
     let insertlane = insts.by_name("insertlane");
@@ -61,6 +63,8 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let sshr = insts.by_name("sshr");
     let trueif = insts.by_name("trueif");
     let udiv = insts.by_name("udiv");
+    let umax = insts.by_name("umax");
+    let umin = insts.by_name("umin");
     let umulhi = insts.by_name("umulhi");
     let ushr_imm = insts.by_name("ushr_imm");
     let urem = insts.by_name("urem");
@@ -71,6 +75,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
 
     let x86_bsf = x86_instructions.by_name("x86_bsf");
     let x86_bsr = x86_instructions.by_name("x86_bsr");
+    let x86_pmaxs = x86_instructions.by_name("x86_pmaxs");
     let x86_pmaxu = x86_instructions.by_name("x86_pmaxu");
     let x86_pmins = x86_instructions.by_name("x86_pmins");
     let x86_pminu = x86_instructions.by_name("x86_pminu");
@@ -554,6 +559,18 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
         narrow.legalize(def!(c = icmp_(ule, a, b)), vec![def!(c = icmp(uge, b, a))]);
     }
 
+    // SIMD integer min/max
+    for ty in &[I8, I16, I32] {
+        let imin = imin.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(def!(c = imin(a, b)), vec![def!(c = x86_pmins(a, b))]);
+        let umin = umin.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(def!(c = umin(a, b)), vec![def!(c = x86_pminu(a, b))]);
+        let imax = imax.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(def!(c = imax(a, b)), vec![def!(c = x86_pmaxs(a, b))]);
+        let umax = umax.bind(vector(*ty, sse_vector_size));
+        narrow.legalize(def!(c = umax(a, b)), vec![def!(c = x86_pmaxu(a, b))]);
+    }
+
     // SIMD fcmp greater-/less-than
     let gt = Literal::enumerator_for(&imm.floatcc, "gt");
     let lt = Literal::enumerator_for(&imm.floatcc, "lt");
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 08b4f66a7a..d9ff3f04e3 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -481,6 +481,75 @@ fn define_control_flow(
     );
 }
 
+#[inline(never)]
+fn define_simd_arithmetic(
+    ig: &mut InstructionGroupBuilder,
+    formats: &Formats,
+    _: &Immediates,
+    _: &EntityRefs,
+) {
+    let Int = &TypeVar::new(
+        "Int",
+        "A scalar or vector integer type",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .simd_lanes(Interval::All)
+            .build(),
+    );
+
+    let a = &Operand::new("a", Int);
+    let x = &Operand::new("x", Int);
+    let y = &Operand::new("y", Int);
+
+    ig.push(
+        Inst::new(
+            "imin",
+            r#"
+        Signed integer minimum.
+        "#,
+            &formats.binary,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
+    ig.push(
+        Inst::new(
+            "umin",
+            r#"
+        Unsigned integer minimum.
+        "#,
+            &formats.binary,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
+    ig.push(
+        Inst::new(
+            "imax",
+            r#"
+        Signed integer maximum.
+        "#,
+            &formats.binary,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+
+    ig.push(
+        Inst::new(
+            "umax",
+            r#"
+        Unsigned integer maximum.
+        "#,
+            &formats.binary,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
+}
+
 #[allow(clippy::many_single_char_names)]
 pub(crate) fn define(
     all_instructions: &mut AllInstructions,
@@ -491,6 +560,7 @@ pub(crate) fn define(
     let mut ig = InstructionGroupBuilder::new(all_instructions);
 
     define_control_flow(&mut ig, formats, imm, entities);
+    define_simd_arithmetic(&mut ig, formats, imm, entities);
 
     // Operand kind shorthands.
     let iflags: &TypeVar = &ValueType::Special(types::Flag::IFlags.into()).into();

From 46cfc26684b09a3e8318c269deb9efc55d3e9258 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 4 Feb 2020 09:17:42 -0800
Subject: [PATCH 3057/3084] Update wasmparser to 0.49.0

---
 cranelift/wasm/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index f9f2b8b4bd..8bee878bb3 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -11,7 +11,7 @@ keywords = ["webassembly", "wasm"]
 edition = "2018"
 
 [dependencies]
-wasmparser = { version = "0.48.2", default-features = false }
+wasmparser = { version = "0.49.0", default-features = false }
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.58.0" }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.58.0", default-features = false }

From f9ef4948fcdf4b0728d54b08db700f6870075f4b Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 4 Feb 2020 09:24:17 -0800
Subject: [PATCH 3058/3084] Translate Wasm integer min/max to Cranelift's
 integer min/max

---
 cranelift/wasm/src/code_translator.rs | 28 +++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 01242348cd..f68c47de46 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1288,6 +1288,22 @@ pub fn translate_operator(
             let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().usub_sat(a, b))
         }
+        Operator::I8x16MinS | Operator::I16x8MinS | Operator::I32x4MinS => {
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().imin(a, b))
+        }
+        Operator::I8x16MinU | Operator::I16x8MinU | Operator::I32x4MinU => {
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().umin(a, b))
+        }
+        Operator::I8x16MaxS | Operator::I16x8MaxS | Operator::I32x4MaxS => {
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().imax(a, b))
+        }
+        Operator::I8x16MaxU | Operator::I16x8MaxU | Operator::I32x4MaxU => {
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().umax(a, b))
+        }
         Operator::I8x16Neg | Operator::I16x8Neg | Operator::I32x4Neg | Operator::I64x2Neg => {
             let a = pop1_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().ineg(a))
@@ -1813,6 +1829,10 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::I8x16Sub
         | Operator::I8x16SubSaturateS
         | Operator::I8x16SubSaturateU
+        | Operator::I8x16MinS
+        | Operator::I8x16MinU
+        | Operator::I8x16MaxS
+        | Operator::I8x16MaxU
         | Operator::I8x16Mul => I8X16,
 
         Operator::I16x8Splat
@@ -1842,6 +1862,10 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::I16x8Sub
         | Operator::I16x8SubSaturateS
         | Operator::I16x8SubSaturateU
+        | Operator::I16x8MinS
+        | Operator::I16x8MinU
+        | Operator::I16x8MaxS
+        | Operator::I16x8MaxU
         | Operator::I16x8Mul => I16X8,
 
         Operator::I32x4Splat
@@ -1867,6 +1891,10 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::I32x4Add
         | Operator::I32x4Sub
         | Operator::I32x4Mul
+        | Operator::I32x4MinS
+        | Operator::I32x4MinU
+        | Operator::I32x4MaxS
+        | Operator::I32x4MaxU
         | Operator::F32x4ConvertI32x4S
         | Operator::F32x4ConvertI32x4U => I32X4,
 

From 91727d99c06fdd62eca801560c185525e2ad6322 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Tue, 4 Feb 2020 09:24:58 -0800
Subject: [PATCH 3059/3084] Clean up unnecessary return

---
 cranelift/wasm/src/code_translator.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index f68c47de46..9a6d24d0a2 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -2037,5 +2037,5 @@ pub fn wasm_param_types(params: &[ir::AbiParam], is_wasm: impl Fn(usize) -> bool
             ret.push(param.value_type);
         }
     }
-    return ret;
+    ret
 }

From d586e0679f9b03873bc474d59cd744c2c5468c11 Mon Sep 17 00:00:00 2001
From: Benjamin Bouvier 
Date: Fri, 21 Feb 2020 15:50:44 +0100
Subject: [PATCH 3060/3084] Replace Gitter and IRC by a Bytecode Alliance Zulip
 stream;

---
 cranelift/CONTRIBUTING.md | 8 +++-----
 cranelift/README.md       | 2 +-
 2 files changed, 4 insertions(+), 6 deletions(-)

diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
index 668e12dd05..c77a686b55 100644
--- a/cranelift/CONTRIBUTING.md
+++ b/cranelift/CONTRIBUTING.md
@@ -13,13 +13,11 @@ is reflected in the code or documentation yet. If you see things that seem
 missing or that don't make sense, or even that just don't work the way you
 expect them to, we're interested to hear about it!
 
-We have a [chat room on Gitter], and questions are also welcome as issues
-in the [Cranelift issue tracker]. Some folks also hang out in the #cranelift
-IRC channel on [irc.mozilla.org].
+We have a [stream] in the Bytecode Alliance Zulip instance, and questions are
+also welcome as issues in the [Cranelift issue tracker].
 
-[chat room on Gitter]: https://gitter.im/CraneStation/Lobby
+[stream]: https://bytecodealliance.zulipchat.com/#narrow/stream/217117-cranelift/topic/general
 [Cranelift issue tracker]: https://github.com/bytecodealliance/cranelift/issues/new
-[irc.mozilla.org]: https://wiki.mozilla.org/IRC
 
 ### Mentoring
 
diff --git a/cranelift/README.md b/cranelift/README.md
index 1242a1c8e7..fb852aad61 100644
--- a/cranelift/README.md
+++ b/cranelift/README.md
@@ -12,7 +12,7 @@ into executable machine code.
 [![Documentation Status](https://readthedocs.org/projects/cranelift/badge/?version=latest)](https://cranelift.readthedocs.io/en/latest/?badge=latest)
 [![Build Status](https://github.com/bytecodealliance/cranelift/workflows/CI/badge.svg)](https://github.com/bytecodealliance/cranelift/actions)
 [![Fuzzit Status](https://app.fuzzit.dev/badge?org_id=bytecodealliance)](https://app.fuzzit.dev/orgs/bytecodealliance/dashboard)
-[![Gitter chat](https://badges.gitter.im/bytecodealliance/bytecodealliance.svg)](https://gitter.im/CraneStation/Lobby)
+[![Chat](https://img.shields.io/badge/chat-zulip-brightgreen.svg)](https://bytecodealliance.zulipchat.com/#narrow/stream/217117-cranelift/topic/general)
 ![Minimum rustc 1.37](https://img.shields.io/badge/rustc-1.37+-green.svg)
 
 For more information, see [the

From c5d6805284ba7aefc8ed4d8040061561c309187e Mon Sep 17 00:00:00 2001
From: Sergei Shulepov 
Date: Thu, 20 Feb 2020 16:57:04 +0100
Subject: [PATCH 3061/3084] Revert "Temporarily disable fuzzing until #1216 is
 resolved (#1224)"

This reverts commit 589fa95a05b3bc6dfe4076ed29fa02d6f0d3d349.
---
 cranelift/.github/workflows/main.yml | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/cranelift/.github/workflows/main.yml b/cranelift/.github/workflows/main.yml
index e8bb3fb41f..b9003a311b 100644
--- a/cranelift/.github/workflows/main.yml
+++ b/cranelift/.github/workflows/main.yml
@@ -109,7 +109,7 @@ jobs:
         cargo fuzz run fuzz_translate_module fuzz/corpus/fuzz_translate_module/$fuzz_module
       env:
         RUST_BACKTRACE: 1
-      if: false && matrix.rust == 'nightly' # Temporarily disable fuzz tests until https://github.com/bytecodealliance/cranelift/issues/1216 is resolved
+      if: matrix.rust == 'nightly'
       continue-on-error: true
 
   meta_determinist_check:
@@ -127,7 +127,6 @@ jobs:
   fuzz:
     name: Fuzz Regression
     runs-on: ubuntu-latest
-    if: false # Temporarily disable fuzz tests until https://github.com/bytecodealliance/cranelift/issues/1216 is resolved
     steps:
     - uses: actions/checkout@v1
     - name: Install Rust
@@ -138,7 +137,7 @@ jobs:
   fuzz_push:
     name: Fuzz (push)
     runs-on: ubuntu-latest
-    if: false && github.event_name == 'push' # Temporarily disable fuzz tests until https://github.com/bytecodealliance/cranelift/issues/1216 is resolved
+    if: github.event_name == 'push'
     steps:
     - uses: actions/checkout@v1
     - name: Install Rust

From 09c6c5db4472823c32c46e192fee5eb3f950c620 Mon Sep 17 00:00:00 2001
From: Nathan Froyd 
Date: Fri, 21 Feb 2020 18:14:37 -0500
Subject: [PATCH 3062/3084] add a "raw" function definition interface to
 cranelift-module (#1400)

* move trap site definitions into cranelift-module

`cranelift-faerie` and `cranelift-object` already have identical
definitions of structures to represent trap sites.  We might as well
merge them ahead of work to define functions via a raw slice of bytes
with associated traps, which will need some kind of common structure for
representing traps anyway.

* cranelift-module: add `define_function_bytes` interface

This interface is useful when the client needs to precisely specify the
ordering of bytes in a particular function.

* add comment about saving files for `perf`
---
 cranelift/faerie/src/backend.rs    | 28 ++++++++++++++-
 cranelift/faerie/src/traps.rs      | 25 +++++++------
 cranelift/module/src/backend.rs    | 14 ++++++++
 cranelift/module/src/lib.rs        |  2 ++
 cranelift/module/src/module.rs     | 47 ++++++++++++++++++++++++
 cranelift/module/src/traps.rs      | 14 ++++++++
 cranelift/object/src/backend.rs    | 25 ++++++++++---
 cranelift/object/src/lib.rs        |  2 +-
 cranelift/object/src/traps.rs      | 16 ++-------
 cranelift/simplejit/src/backend.rs | 57 ++++++++++++++++++++++++------
 10 files changed, 188 insertions(+), 42 deletions(-)
 create mode 100644 cranelift/module/src/traps.rs

diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index c0cb086a6d..e6439a69b9 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -10,9 +10,10 @@ use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::{self, binemit, ir};
 use cranelift_module::{
     Backend, DataContext, DataDescription, DataId, FuncId, Init, Linkage, ModuleError,
-    ModuleNamespace, ModuleResult,
+    ModuleNamespace, ModuleResult, TrapSite,
 };
 use faerie;
+use std::convert::TryInto;
 use std::fs::File;
 use target_lexicon::Triple;
 
@@ -200,6 +201,31 @@ impl Backend for FaerieBackend {
         Ok(FaerieCompiledFunction { code_length })
     }
 
+    fn define_function_bytes(
+        &mut self,
+        _id: FuncId,
+        name: &str,
+        bytes: &[u8],
+        _namespace: &ModuleNamespace,
+        traps: Vec,
+    ) -> ModuleResult {
+        let code_length: u32 = match bytes.len().try_into() {
+            Ok(code_length) => code_length,
+            _ => Err(ModuleError::FunctionTooLarge(name.to_string()))?,
+        };
+
+        if let Some(ref mut trap_manifest) = self.trap_manifest {
+            let trap_sink = FaerieTrapSink::new_with_sites(name, code_length, traps);
+            trap_manifest.add_sink(trap_sink);
+        }
+
+        self.artifact
+            .define(name, bytes.to_vec())
+            .expect("inconsistent declaration");
+
+        Ok(FaerieCompiledFunction { code_length })
+    }
+
     fn define_data(
         &mut self,
         _id: DataId,
diff --git a/cranelift/faerie/src/traps.rs b/cranelift/faerie/src/traps.rs
index d01619967b..b84f171955 100644
--- a/cranelift/faerie/src/traps.rs
+++ b/cranelift/faerie/src/traps.rs
@@ -2,17 +2,7 @@
 //! for every function in the module. This data may be useful at runtime.
 
 use cranelift_codegen::{binemit, ir};
-
-/// Record of the arguments cranelift passes to `TrapSink::trap`
-#[derive(Debug)]
-pub struct FaerieTrapSite {
-    /// Offset into function
-    pub offset: binemit::CodeOffset,
-    /// Source location given to cranelift
-    pub srcloc: ir::SourceLoc,
-    /// Trap code, as determined by cranelift
-    pub code: ir::TrapCode,
-}
+use cranelift_module::TrapSite;
 
 /// Record of the trap sites for a given function
 #[derive(Debug)]
@@ -22,7 +12,7 @@ pub struct FaerieTrapSink {
     /// Total code size of function
     pub code_size: u32,
     /// All trap sites collected in function
-    pub sites: Vec,
+    pub sites: Vec,
 }
 
 impl FaerieTrapSink {
@@ -34,11 +24,20 @@ impl FaerieTrapSink {
             code_size,
         }
     }
+
+    /// Create a `FaerieTrapSink` pre-populated with `traps`
+    pub fn new_with_sites(name: &str, code_size: u32, traps: Vec) -> Self {
+        Self {
+            sites: traps,
+            name: name.to_owned(),
+            code_size,
+        }
+    }
 }
 
 impl binemit::TrapSink for FaerieTrapSink {
     fn trap(&mut self, offset: binemit::CodeOffset, srcloc: ir::SourceLoc, code: ir::TrapCode) {
-        self.sites.push(FaerieTrapSite {
+        self.sites.push(TrapSite {
             offset,
             srcloc,
             code,
diff --git a/cranelift/module/src/backend.rs b/cranelift/module/src/backend.rs
index 218070a5c0..1464c4aefc 100644
--- a/cranelift/module/src/backend.rs
+++ b/cranelift/module/src/backend.rs
@@ -6,6 +6,7 @@ use crate::FuncId;
 use crate::Linkage;
 use crate::ModuleNamespace;
 use crate::ModuleResult;
+use crate::TrapSite;
 use core::marker;
 use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::Context;
@@ -14,6 +15,7 @@ use cranelift_codegen::{binemit, ir};
 use std::borrow::ToOwned;
 use std::boxed::Box;
 use std::string::String;
+use std::vec::Vec;
 
 /// A `Backend` implements the functionality needed to support a `Module`.
 ///
@@ -85,6 +87,18 @@ where
         code_size: u32,
     ) -> ModuleResult;
 
+    /// Define a function, taking the function body from the given `bytes`.
+    ///
+    /// Functions must be declared before being defined.
+    fn define_function_bytes(
+        &mut self,
+        id: FuncId,
+        name: &str,
+        bytes: &[u8],
+        namespace: &ModuleNamespace,
+        traps: Vec,
+    ) -> ModuleResult;
+
     /// Define a zero-initialized data object of the given size.
     ///
     /// Data objects must be declared before being defined.
diff --git a/cranelift/module/src/lib.rs b/cranelift/module/src/lib.rs
index 0122171e9a..25a2759be4 100644
--- a/cranelift/module/src/lib.rs
+++ b/cranelift/module/src/lib.rs
@@ -35,6 +35,7 @@ use std::collections::{hash_map, HashMap};
 mod backend;
 mod data_context;
 mod module;
+mod traps;
 
 pub use crate::backend::{default_libcall_names, Backend};
 pub use crate::data_context::{DataContext, DataDescription, Init};
@@ -42,6 +43,7 @@ pub use crate::module::{
     DataId, FuncId, FuncOrDataId, Linkage, Module, ModuleError, ModuleFunction, ModuleNamespace,
     ModuleResult,
 };
+pub use crate::traps::TrapSite;
 
 /// Version number of this crate.
 pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 6c2f109c20..10bca75ac7 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -7,12 +7,14 @@
 
 use super::HashMap;
 use crate::data_context::DataContext;
+use crate::traps::TrapSite;
 use crate::Backend;
 use cranelift_codegen::binemit::{self, CodeInfo};
 use cranelift_codegen::entity::{entity_impl, PrimaryMap};
 use cranelift_codegen::{ir, isa, CodegenError, Context};
 use log::info;
 use std::borrow::ToOwned;
+use std::convert::TryInto;
 use std::string::String;
 use std::vec::Vec;
 use thiserror::Error;
@@ -139,6 +141,9 @@ pub enum ModuleError {
     /// Indicates an identifier was defined, but was declared as an import
     #[error("Invalid to define identifier declared as an import: {0}")]
     InvalidImportDefinition(String),
+    /// Indicates a too-long function was defined
+    #[error("Function {0} exceeds the maximum function size")]
+    FunctionTooLarge(String),
     /// Wraps a `cranelift-codegen` error
     #[error("Compilation error: {0}")]
     Compilation(#[from] CodegenError),
@@ -573,6 +578,48 @@ where
         Ok(total_size)
     }
 
+    /// Define a function, taking the function body from the given `bytes`.
+    ///
+    /// This function is generally only useful if you need to precisely specify
+    /// the emitted instructions for some reason; otherwise, you should use
+    /// `define_function`.
+    ///
+    /// Returns the size of the function's code.
+    pub fn define_function_bytes(
+        &mut self,
+        func: FuncId,
+        bytes: &[u8],
+        traps: Vec,
+    ) -> ModuleResult {
+        info!("defining function {} with bytes", func);
+        let info = &self.contents.functions[func];
+        if info.compiled.is_some() {
+            return Err(ModuleError::DuplicateDefinition(info.decl.name.clone()));
+        }
+        if !info.decl.linkage.is_definable() {
+            return Err(ModuleError::InvalidImportDefinition(info.decl.name.clone()));
+        }
+
+        let total_size: u32 = match bytes.len().try_into() {
+            Ok(total_size) => total_size,
+            _ => Err(ModuleError::FunctionTooLarge(info.decl.name.clone()))?,
+        };
+
+        let compiled = Some(self.backend.define_function_bytes(
+            func,
+            &info.decl.name,
+            bytes,
+            &ModuleNamespace:: {
+                contents: &self.contents,
+            },
+            traps,
+        )?);
+
+        self.contents.functions[func].compiled = compiled;
+        self.functions_to_finalize.push(func);
+        Ok(total_size)
+    }
+
     /// Define a data object, producing the data contents from the given `DataContext`.
     pub fn define_data(&mut self, data: DataId, data_ctx: &DataContext) -> ModuleResult<()> {
         let compiled = {
diff --git a/cranelift/module/src/traps.rs b/cranelift/module/src/traps.rs
new file mode 100644
index 0000000000..2344d4189b
--- /dev/null
+++ b/cranelift/module/src/traps.rs
@@ -0,0 +1,14 @@
+//! Defines `TrapSite`.
+
+use cranelift_codegen::{binemit, ir};
+
+/// Record of the arguments cranelift passes to `TrapSink::trap`.
+#[derive(Clone, Debug)]
+pub struct TrapSite {
+    /// Offset into function.
+    pub offset: binemit::CodeOffset,
+    /// Source location given to cranelift.
+    pub srcloc: ir::SourceLoc,
+    /// Trap code, as determined by cranelift.
+    pub code: ir::TrapCode,
+}
diff --git a/cranelift/object/src/backend.rs b/cranelift/object/src/backend.rs
index f8baf1452c..c4fa5eaf72 100644
--- a/cranelift/object/src/backend.rs
+++ b/cranelift/object/src/backend.rs
@@ -1,6 +1,6 @@
 //! Defines `ObjectBackend`.
 
-use crate::traps::{ObjectTrapSink, ObjectTrapSite};
+use crate::traps::ObjectTrapSink;
 use cranelift_codegen::binemit::{
     Addend, CodeOffset, NullStackmapSink, NullTrapSink, Reloc, RelocSink,
 };
@@ -9,7 +9,7 @@ use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::{self, binemit, ir};
 use cranelift_module::{
     Backend, DataContext, DataDescription, DataId, FuncId, Init, Linkage, ModuleNamespace,
-    ModuleResult,
+    ModuleResult, TrapSite,
 };
 use object::write::{
     Object, Relocation, SectionId, StandardSection, Symbol, SymbolId, SymbolSection,
@@ -79,7 +79,7 @@ pub struct ObjectBackend {
     object: Object,
     functions: SecondaryMap>,
     data_objects: SecondaryMap>,
-    traps: SecondaryMap>,
+    traps: SecondaryMap>,
     relocs: Vec,
     libcalls: HashMap,
     libcall_names: Box String>,
@@ -226,6 +226,23 @@ impl Backend for ObjectBackend {
         Ok(ObjectCompiledFunction)
     }
 
+    fn define_function_bytes(
+        &mut self,
+        func_id: FuncId,
+        _name: &str,
+        bytes: &[u8],
+        _namespace: &ModuleNamespace,
+        traps: Vec,
+    ) -> ModuleResult {
+        let symbol = self.functions[func_id].unwrap();
+        let section = self.object.section_id(StandardSection::Text);
+        let _offset = self
+            .object
+            .add_symbol_data(symbol, section, bytes, self.function_alignment);
+        self.traps[func_id] = traps;
+        Ok(ObjectCompiledFunction)
+    }
+
     fn define_data(
         &mut self,
         data_id: DataId,
@@ -462,7 +479,7 @@ pub struct ObjectProduct {
     /// Symbol IDs for data objects (both declared and defined).
     pub data_objects: SecondaryMap>,
     /// Trap sites for defined functions.
-    pub traps: SecondaryMap>,
+    pub traps: SecondaryMap>,
 }
 
 impl ObjectProduct {
diff --git a/cranelift/object/src/lib.rs b/cranelift/object/src/lib.rs
index 241dcb6316..1542c0a191 100644
--- a/cranelift/object/src/lib.rs
+++ b/cranelift/object/src/lib.rs
@@ -29,7 +29,7 @@ mod backend;
 mod traps;
 
 pub use crate::backend::{ObjectBackend, ObjectBuilder, ObjectProduct, ObjectTrapCollection};
-pub use crate::traps::{ObjectTrapSink, ObjectTrapSite};
+pub use crate::traps::ObjectTrapSink;
 
 /// Version number of this crate.
 pub const VERSION: &str = env!("CARGO_PKG_VERSION");
diff --git a/cranelift/object/src/traps.rs b/cranelift/object/src/traps.rs
index a64f1e13fa..2fabfc626d 100644
--- a/cranelift/object/src/traps.rs
+++ b/cranelift/object/src/traps.rs
@@ -2,28 +2,18 @@
 //! for every function in the module. This data may be useful at runtime.
 
 use cranelift_codegen::{binemit, ir};
-
-/// Record of the arguments cranelift passes to `TrapSink::trap`
-#[derive(Clone)]
-pub struct ObjectTrapSite {
-    /// Offset into function
-    pub offset: binemit::CodeOffset,
-    /// Source location given to cranelift
-    pub srcloc: ir::SourceLoc,
-    /// Trap code, as determined by cranelift
-    pub code: ir::TrapCode,
-}
+use cranelift_module::TrapSite;
 
 /// Record of the trap sites for a given function
 #[derive(Default, Clone)]
 pub struct ObjectTrapSink {
     /// All trap sites collected in function
-    pub sites: Vec,
+    pub sites: Vec,
 }
 
 impl binemit::TrapSink for ObjectTrapSink {
     fn trap(&mut self, offset: binemit::CodeOffset, srcloc: ir::SourceLoc, code: ir::TrapCode) {
-        self.sites.push(ObjectTrapSite {
+        self.sites.push(TrapSite {
             offset,
             srcloc,
             code,
diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index a4b7d1a1da..8d37b99920 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -8,7 +8,7 @@ use cranelift_codegen::isa::TargetIsa;
 use cranelift_codegen::{self, ir, settings};
 use cranelift_module::{
     Backend, DataContext, DataDescription, DataId, FuncId, Init, Linkage, ModuleNamespace,
-    ModuleResult,
+    ModuleResult, TrapSite,
 };
 use cranelift_native;
 #[cfg(not(windows))]
@@ -191,6 +191,23 @@ impl SimpleJITBackend {
             _ => panic!("invalid ExternalName {}", name),
         }
     }
+
+    fn record_function_for_perf(&self, ptr: *mut u8, size: usize, name: &str) {
+        // The Linux perf tool supports JIT code via a /tmp/perf-$PID.map file,
+        // which contains memory regions and their associated names.  If we
+        // are profiling with perf and saving binaries to PERF_BUILDID_DIR
+        // for post-profile analysis, write information about each function
+        // we define.
+        if cfg!(target_os = "linux") && ::std::env::var_os("PERF_BUILDID_DIR").is_some() {
+            let mut map_file = ::std::fs::OpenOptions::new()
+                .create(true)
+                .append(true)
+                .open(format!("/tmp/perf-{}.map", ::std::process::id()))
+                .unwrap();
+
+            let _ = writeln!(map_file, "{:x} {:x} {}", ptr as usize, size, name);
+        }
+    }
 }
 
 impl<'simple_jit_backend> Backend for SimpleJITBackend {
@@ -267,15 +284,7 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
             .allocate(size, EXECUTABLE_DATA_ALIGNMENT)
             .expect("TODO: handle OOM etc.");
 
-        if cfg!(target_os = "linux") && ::std::env::var_os("PERF_BUILDID_DIR").is_some() {
-            let mut map_file = ::std::fs::OpenOptions::new()
-                .create(true)
-                .append(true)
-                .open(format!("/tmp/perf-{}.map", ::std::process::id()))
-                .unwrap();
-
-            let _ = writeln!(map_file, "{:x} {:x} {}", ptr as usize, code_size, name);
-        }
+        self.record_function_for_perf(ptr, size, name);
 
         let mut reloc_sink = SimpleJITRelocSink::new();
         // Ignore traps for now. For now, frontends should just avoid generating code
@@ -299,6 +308,34 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         })
     }
 
+    fn define_function_bytes(
+        &mut self,
+        _id: FuncId,
+        name: &str,
+        bytes: &[u8],
+        _namespace: &ModuleNamespace,
+        _traps: Vec,
+    ) -> ModuleResult {
+        let size = bytes.len();
+        let ptr = self
+            .memory
+            .code
+            .allocate(size, EXECUTABLE_DATA_ALIGNMENT)
+            .expect("TODO: handle OOM etc.");
+
+        self.record_function_for_perf(ptr, size, name);
+
+        unsafe {
+            ptr::copy_nonoverlapping(bytes.as_ptr(), ptr, size);
+        }
+
+        Ok(Self::CompiledFunction {
+            code: ptr,
+            size,
+            relocs: vec![],
+        })
+    }
+
     fn define_data(
         &mut self,
         _id: DataId,

From 5e05aa1b03fd8b6c5b064bc145750643ca92e803 Mon Sep 17 00:00:00 2001
From: Josh Triplett 
Date: Sat, 22 Feb 2020 11:11:48 -0800
Subject: [PATCH 3063/3084] Update to wasmparser 0.51

wasmparser::BinaryReaderError now encapsulates its fields, so call the
accessors rather than destructuring to get the fields.
---
 cranelift/wasm/Cargo.toml          | 2 +-
 cranelift/wasm/src/environ/spec.rs | 8 +++++---
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 8bee878bb3..4ac0496263 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -11,7 +11,7 @@ keywords = ["webassembly", "wasm"]
 edition = "2018"
 
 [dependencies]
-wasmparser = { version = "0.49.0", default-features = false }
+wasmparser = { version = "0.51.0", default-features = false }
 cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
 cranelift-entity = { path = "../cranelift-entity", version = "0.58.0" }
 cranelift-frontend = { path = "../cranelift-frontend", version = "0.58.0", default-features = false }
diff --git a/cranelift/wasm/src/environ/spec.rs b/cranelift/wasm/src/environ/spec.rs
index 73a02c6709..b0dd4d508d 100644
--- a/cranelift/wasm/src/environ/spec.rs
+++ b/cranelift/wasm/src/environ/spec.rs
@@ -55,7 +55,7 @@ pub enum WasmError {
     #[error("Invalid input WebAssembly code at offset {offset}: {message}")]
     InvalidWebAssembly {
         /// A string describing the validation error.
-        message: &'static str,
+        message: std::string::String,
         /// The bytecode offset where the error occurred.
         offset: usize,
     },
@@ -90,8 +90,10 @@ macro_rules! wasm_unsupported {
 impl From for WasmError {
     /// Convert from a `BinaryReaderError` to a `WasmError`.
     fn from(e: BinaryReaderError) -> Self {
-        let BinaryReaderError { message, offset } = e;
-        Self::InvalidWebAssembly { message, offset }
+        Self::InvalidWebAssembly {
+            message: e.message().into(),
+            offset: e.offset(),
+        }
     }
 }
 

From 830bdd5127ce5fcab04105a33c576c309a5bf7b3 Mon Sep 17 00:00:00 2001
From: nalmt 
Date: Wed, 5 Feb 2020 18:12:56 +0100
Subject: [PATCH 3064/3084] Add maximum threshold for number of blocks per
 function #951

To fix this case that may take forever to compile:

function %a(){
ebb477777777:
}

We decide to define a maximum threshold for the number of blocks in functions.

Based on a large WASM program (https://github.com/mozilla/perf-automation/blob/master/benchmarks/wasm-misc/AngryBots.wasm),
its IR functions does not exceed 1414 blocks. A number 100 times greater (100,000 blocks) seems (currently) enough to define our maximum threshold.

To make this quick benchmark the cranelift-wasm/src/func_translator.rs file has been modified like this:

static mut MAX: usize = 0;

pub fn translate_from_reader(...) {

    [...]

    builder.finalize();

    // the compiler is single threaded
    unsafe {
        if func.dfg.num_ebbs() > MAX {
            MAX = func.dfg.num_ebbs();
            println!("MAX {}", MAX);
        }
    }

    Ok(())
}
---
 cranelift/reader/src/parser.rs | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index f0b866f472..6cedb25b0d 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -27,6 +27,9 @@ use std::str::FromStr;
 use std::{u16, u32};
 use target_lexicon::Triple;
 
+/// After some quick benchmarks a program should never have more than 100,000 blocks.
+const MAX_BLOCKS_IN_A_FUNCTION: u32 = 100_000;
+
 /// Parse the entire `text` into a list of functions.
 ///
 /// Any test commands or target declarations are ignored.
@@ -1768,6 +1771,10 @@ impl<'a> Parser<'a> {
         let block_num = self.match_block("expected block header")?;
         let block = ctx.add_block(block_num, self.loc)?;
 
+        if block_num.as_u32() >= MAX_BLOCKS_IN_A_FUNCTION {
+            return Err(self.error("too many blocks"));
+        }
+
         if !self.optional(Token::Colon) {
             // block-header ::= Block(block) [ * block-params ] ":"
             self.parse_block_params(ctx, block)?;
@@ -2979,6 +2986,24 @@ mod tests {
         assert!(!is_warning);
     }
 
+    #[test]
+    fn number_of_blocks() {
+        let ParseError {
+            location,
+            message,
+            is_warning,
+        } = Parser::new(
+            "function %a() {
+                block100000:",
+        )
+        .parse_function(None)
+        .unwrap_err();
+
+        assert_eq!(location.line_number, 2);
+        assert_eq!(message, "too many blocks");
+        assert!(!is_warning);
+    }
+
     #[test]
     fn duplicate_jt() {
         let ParseError {

From 032e81fd6fa6ddcc9990a33aa1ff6a38d5e0d812 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 31 Jan 2020 14:37:29 -0800
Subject: [PATCH 3065/3084] Add x86 SIMD average rounding

---
 .../codegen/meta/src/isa/x86/encodings.rs     |  7 +++++
 cranelift/codegen/meta/src/isa/x86/opcodes.rs |  6 +++++
 .../codegen/meta/src/shared/instructions.rs   | 27 ++++++++++++++++++-
 .../isa/x86/simd-arithmetic-binemit.clif      | 12 +++++++++
 .../isa/x86/simd-arithmetic-run.clif          | 14 ++++++++++
 5 files changed, 65 insertions(+), 1 deletion(-)

diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index f24232f39a..81e2e48c58 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -1560,6 +1560,7 @@ fn define_simd(
     let formats = &shared_defs.formats;
 
     // Shorthands for instructions.
+    let avg_round = shared.by_name("avg_round");
     let bitcast = shared.by_name("bitcast");
     let bor = shared.by_name("bor");
     let bxor = shared.by_name("bxor");
@@ -1926,6 +1927,12 @@ fn define_simd(
         e.enc_32_64_maybe_isap(imul, rec_fa.opcodes(opcodes), *isap);
     }
 
+    // SIMD integer average with rounding.
+    for (ty, opcodes) in &[(I8, &PAVGB[..]), (I16, &PAVGW[..])] {
+        let avgr = avg_round.bind(vector(*ty, sse_vector_size));
+        e.enc_32_64(avgr, rec_fa.opcodes(opcodes));
+    }
+
     // SIMD logical operations
     let band = shared.by_name("band");
     let band_not = shared.by_name("band_not");
diff --git a/cranelift/codegen/meta/src/isa/x86/opcodes.rs b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
index 9006ce92cf..5bd4153414 100644
--- a/cranelift/codegen/meta/src/isa/x86/opcodes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/opcodes.rs
@@ -317,6 +317,12 @@ pub static PAND: [u8; 3] = [0x66, 0x0f, 0xdb];
 /// Bitwise AND NOT of xmm2/m128 and xmm1 (SSE2).
 pub static PANDN: [u8; 3] = [0x66, 0x0f, 0xdf];
 
+/// Average packed unsigned byte integers from xmm2/m128 and xmm1 with rounding (SSE2).
+pub static PAVGB: [u8; 3] = [0x66, 0x0f, 0xE0];
+
+/// Average packed unsigned word integers from xmm2/m128 and xmm1 with rounding (SSE2).
+pub static PAVGW: [u8; 3] = [0x66, 0x0f, 0xE3];
+
 /// Compare packed data for equal (SSE2).
 pub static PCMPEQB: [u8; 3] = [0x66, 0x0f, 0x74];
 
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index d9ff3f04e3..471ad85a56 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -548,6 +548,32 @@ fn define_simd_arithmetic(
         .operands_in(vec![x, y])
         .operands_out(vec![a]),
     );
+
+    let IxN = &TypeVar::new(
+        "IxN",
+        "A SIMD vector type containing integers",
+        TypeSetBuilder::new()
+            .ints(Interval::All)
+            .simd_lanes(Interval::All)
+            .includes_scalars(false)
+            .build(),
+    );
+
+    let a = &Operand::new("a", IxN);
+    let x = &Operand::new("x", IxN);
+    let y = &Operand::new("y", IxN);
+
+    ig.push(
+        Inst::new(
+            "avg_round",
+            r#"
+        Unsigned average with rounding: `a := (x + y + 1) // 2`
+        "#,
+            &formats.binary,
+        )
+        .operands_in(vec![x, y])
+        .operands_out(vec![a]),
+    );
 }
 
 #[allow(clippy::many_single_char_names)]
@@ -627,7 +653,6 @@ pub(crate) fn define(
             .includes_scalars(false)
             .build(),
     );
-
     let Any = &TypeVar::new(
         "Any",
         "Any integer, float, boolean, or reference scalar or vector type",
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif
index 9f5b4f0080..85797d9a4b 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-binemit.clif
@@ -197,3 +197,15 @@ block0(v0: f64x2 [%xmm3], v1: f64x2 [%xmm5]):
 [-, %xmm3]    v8 = sqrt v0          ; bin: 66 40 0f 51 db
     return
 }
+
+function %average_rounding_i8x16(i8x16, i8x16) {
+block0(v0: i8x16 [%xmm6], v1: i8x16 [%xmm2]):
+[-, %xmm6]    v2 = avg_round v0, v1      ; bin: 66 0f e0 f2
+    return
+}
+
+function %average_rounding_i16x8(i16x8, i16x8) {
+block0(v0: i16x8 [%xmm6], v1: i16x8 [%xmm2]):
+[-, %xmm6]    v2 = avg_round v0, v1      ; bin: 66 0f e3 f2
+    return
+}
diff --git a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
index 971f5c9bdb..3403815154 100644
--- a/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
+++ b/cranelift/filetests/filetests/isa/x86/simd-arithmetic-run.clif
@@ -265,3 +265,17 @@ block0:
     return v4
 }
 ; run
+
+function %average_rounding_i16x8() -> b1 {
+block0:
+    v0 = vconst.i16x8 [0 0 0 1 42 19 -1 -1]
+    v1 = vconst.i16x8 [0 1 2 4 42 18 -1 0]
+    v2 = vconst.i16x8 [0 1 1 3 42 19 -1 -32768] ; -1 (0xffff) + 0 + 1 == -32768 (0x8000)
+
+    v3 = avg_round v0, v1
+    v4 = icmp eq v2, v3
+    v5 = vall_true v4
+
+    return v5
+}
+; run

From 78bf63c76d6488c828759d69f1e96ad19c61d474 Mon Sep 17 00:00:00 2001
From: Andrew Brown 
Date: Fri, 31 Jan 2020 14:40:39 -0800
Subject: [PATCH 3066/3084] Translate Wasm SIMD's avgr_u to Cranelift's new
 avg_round instruction

---
 cranelift/wasm/src/code_translator.rs | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/cranelift/wasm/src/code_translator.rs b/cranelift/wasm/src/code_translator.rs
index 9a6d24d0a2..da2a7ed186 100644
--- a/cranelift/wasm/src/code_translator.rs
+++ b/cranelift/wasm/src/code_translator.rs
@@ -1304,6 +1304,10 @@ pub fn translate_operator(
             let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().umax(a, b))
         }
+        Operator::I8x16RoundingAverageU | Operator::I16x8RoundingAverageU => {
+            let (a, b) = pop2_with_bitcast(state, type_of(op), builder);
+            state.push1(builder.ins().avg_round(a, b))
+        }
         Operator::I8x16Neg | Operator::I16x8Neg | Operator::I32x4Neg | Operator::I64x2Neg => {
             let a = pop1_with_bitcast(state, type_of(op), builder);
             state.push1(builder.ins().ineg(a))
@@ -1491,9 +1495,7 @@ pub fn translate_operator(
         | Operator::I32x4Load16x4S { .. }
         | Operator::I32x4Load16x4U { .. }
         | Operator::I64x2Load32x2S { .. }
-        | Operator::I64x2Load32x2U { .. }
-        | Operator::I8x16RoundingAverageU { .. }
-        | Operator::I16x8RoundingAverageU { .. } => {
+        | Operator::I64x2Load32x2U { .. } => {
             return Err(wasm_unsupported!("proposed SIMD operator {:?}", op));
         }
     };
@@ -1833,6 +1835,7 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::I8x16MinU
         | Operator::I8x16MaxS
         | Operator::I8x16MaxU
+        | Operator::I8x16RoundingAverageU
         | Operator::I8x16Mul => I8X16,
 
         Operator::I16x8Splat
@@ -1866,6 +1869,7 @@ fn type_of(operator: &Operator) -> Type {
         | Operator::I16x8MinU
         | Operator::I16x8MaxS
         | Operator::I16x8MaxU
+        | Operator::I16x8RoundingAverageU
         | Operator::I16x8Mul => I16X8,
 
         Operator::I32x4Splat

From 0a11736fbf8c7234e5a1b7a5b8d2e31fe8fd03aa Mon Sep 17 00:00:00 2001
From: Dan Gohman 
Date: Mon, 24 Feb 2020 15:10:08 -0800
Subject: [PATCH 3067/3084] Bump version to 0.59.0 (#1406)

---
 cranelift/Cargo.toml                | 30 ++++++++++++++---------------
 cranelift/bforest/Cargo.toml        |  4 ++--
 cranelift/codegen/Cargo.toml        | 10 +++++-----
 cranelift/codegen/meta/Cargo.toml   |  6 +++---
 cranelift/codegen/shared/Cargo.toml |  2 +-
 cranelift/entity/Cargo.toml         |  2 +-
 cranelift/faerie/Cargo.toml         |  6 +++---
 cranelift/filetests/Cargo.toml      | 10 +++++-----
 cranelift/frontend/Cargo.toml       |  4 ++--
 cranelift/module/Cargo.toml         |  6 +++---
 cranelift/native/Cargo.toml         |  4 ++--
 cranelift/object/Cargo.toml         |  6 +++---
 cranelift/preopt/Cargo.toml         |  6 +++---
 cranelift/publish-all.sh            |  2 +-
 cranelift/reader/Cargo.toml         |  4 ++--
 cranelift/serde/Cargo.toml          |  6 +++---
 cranelift/simplejit/Cargo.toml      | 14 +++++++-------
 cranelift/umbrella/Cargo.toml       |  6 +++---
 cranelift/wasm/Cargo.toml           |  8 ++++----
 19 files changed, 68 insertions(+), 68 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index 8e784ff3e2..c93d53c9a0 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-tools"
 authors = ["The Cranelift Project Developers"]
-version = "0.58.0"
+version = "0.59.0"
 description = "Binaries for testing the Cranelift libraries"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -19,20 +19,20 @@ path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.58.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.58.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.58.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.58.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.58.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.58.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.58.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.58.0" }
-cranelift-module = { path = "cranelift-module", version = "0.58.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.58.0" }
-cranelift-object = { path = "cranelift-object", version = "0.58.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.58.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.58.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.58.0" }
+cranelift-codegen = { path = "cranelift-codegen", version = "0.59.0" }
+cranelift-entity = { path = "cranelift-entity", version = "0.59.0" }
+cranelift-reader = { path = "cranelift-reader", version = "0.59.0" }
+cranelift-frontend = { path = "cranelift-frontend", version = "0.59.0" }
+cranelift-serde = { path = "cranelift-serde", version = "0.59.0", optional = true }
+cranelift-wasm = { path = "cranelift-wasm", version = "0.59.0", optional = true }
+cranelift-native = { path = "cranelift-native", version = "0.59.0" }
+cranelift-filetests = { path = "cranelift-filetests", version = "0.59.0" }
+cranelift-module = { path = "cranelift-module", version = "0.59.0" }
+cranelift-faerie = { path = "cranelift-faerie", version = "0.59.0" }
+cranelift-object = { path = "cranelift-object", version = "0.59.0" }
+cranelift-simplejit = { path = "cranelift-simplejit", version = "0.59.0" }
+cranelift-preopt = { path = "cranelift-preopt", version = "0.59.0" }
+cranelift = { path = "cranelift-umbrella", version = "0.59.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index bac7f8d042..6c70e2e551 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-bforest"
-version = "0.58.0"
+version = "0.59.0"
 description = "A forest of B+-trees"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.58.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.59.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index d4bbe8d3aa..8a693994dd 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen"
-version = "0.58.0"
+version = "0.59.0"
 description = "Low-level code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -13,9 +13,9 @@ build = "build.rs"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "./shared", version = "0.58.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.58.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.58.0" }
+cranelift-codegen-shared = { path = "./shared", version = "0.59.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.59.0" }
+cranelift-bforest = { path = "../cranelift-bforest", version = "0.59.0" }
 hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
@@ -30,7 +30,7 @@ byteorder = { version = "1.3.2", default-features = false }
 # accomodated in `tests`.
 
 [build-dependencies]
-cranelift-codegen-meta = { path = "meta", version = "0.58.0" }
+cranelift-codegen-meta = { path = "meta", version = "0.59.0" }
 
 [features]
 default = ["std", "unwind"]
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index 6bdb6a3463..b26c12381a 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-codegen-meta"
 authors = ["The Cranelift Project Developers"]
-version = "0.58.0"
+version = "0.59.0"
 description = "Metaprogram for cranelift-codegen code generator library"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -9,8 +9,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen-shared = { path = "../shared", version = "0.58.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.58.0" }
+cranelift-codegen-shared = { path = "../shared", version = "0.59.0" }
+cranelift-entity = { path = "../../cranelift-entity", version = "0.59.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/shared/Cargo.toml b/cranelift/codegen/shared/Cargo.toml
index 67ffcbec19..42207d427a 100644
--- a/cranelift/codegen/shared/Cargo.toml
+++ b/cranelift/codegen/shared/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-codegen-shared"
-version = "0.58.0"
+version = "0.59.0"
 description = "For code shared between cranelift-codegen-meta and cranelift-codegen"
 license = "Apache-2.0 WITH LLVM-exception"
 repository = "https://github.com/bytecodealliance/cranelift"
diff --git a/cranelift/entity/Cargo.toml b/cranelift/entity/Cargo.toml
index 2a553e2345..a0c5706803 100644
--- a/cranelift/entity/Cargo.toml
+++ b/cranelift/entity/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-entity"
-version = "0.58.0"
+version = "0.59.0"
 description = "Data structures using entity references as mapping keys"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index 144a18f769..fcd67551fc 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-faerie"
-version = "0.58.0"
+version = "0.59.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with Faerie"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.58.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.59.0" }
 faerie = "0.14.0"
 goblin = "0.1.0"
 anyhow = "1.0"
@@ -18,7 +18,7 @@ target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.58.0"
+version = "0.59.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index 2539ef79c1..f773773c19 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "cranelift-filetests"
 authors = ["The Cranelift Project Developers"]
-version = "0.58.0"
+version = "0.59.0"
 description = "Test driver and implementations of the filetest commands"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/en/latest/testing.html#file-tests"
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.58.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.58.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.58.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../cranelift-native", version = "0.59.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.59.0" }
+cranelift-preopt = { path = "../cranelift-preopt", version = "0.59.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 gimli = { version = "0.20.0", default-features = false, features = ["read"] }
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 7b98c0fce9..39e0c5e594 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-frontend"
-version = "0.58.0"
+version = "0.59.0"
 description = "Cranelift IR builder helper"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", default-features = false }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 68eb3b5d49..7ece676ca5 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-module"
-version = "0.58.0"
+version = "0.59.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for linking functions and data with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.58.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.59.0" }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 thiserror = "1.0.4"
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 089acc3ccb..40c58922b6 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-native"
-version = "0.58.0"
+version = "0.59.0"
 authors = ["The Cranelift Project Developers"]
 description = "Support for targeting the host with Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", default-features = false }
 target-lexicon = "0.10"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index f5bbcc2868..030d49c444 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-object"
-version = "0.58.0"
+version = "0.59.0"
 authors = ["The Cranelift Project Developers"]
 description = "Emit Cranelift output to native object files with `object`"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,13 +10,13 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.58.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.59.0" }
 object = { version = "0.17", default-features = false, features = ["write"] }
 target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.58.0"
+version = "0.59.0"
 default-features = false
 features = ["std"]
 
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index 2454dabf20..dfee57b887 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-preopt"
-version = "0.58.0"
+version = "0.59.0"
 description = "Support for optimizations in Cranelift"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.58.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.59.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
index 8f009287b4..6d8a6a713c 100755
--- a/cranelift/publish-all.sh
+++ b/cranelift/publish-all.sh
@@ -9,7 +9,7 @@ topdir=$(dirname "$0")
 cd "$topdir"
 
 # All the cranelift-* crates have the same version number
-version="0.58.0"
+version="0.59.0"
 
 # Update all of the Cargo.toml files.
 #
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index a5645820a9..190ea86225 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift-reader"
-version = "0.58.0"
+version = "0.59.0"
 description = "Cranelift textual IR reader"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0" }
 target-lexicon = "0.10"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index 9fd94c1b56..d3f091a0ef 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-serde"
-version = "0.58.0"
+version = "0.59.0"
 authors = ["The Cranelift Project Developers"]
 description = "Serializer/Deserializer for Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.58.0" }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0" }
+cranelift-reader = { path = "../cranelift-reader", version = "0.59.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index f11946f41f..1e83e6c797 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-simplejit"
-version = "0.58.0"
+version = "0.59.0"
 authors = ["The Cranelift Project Developers"]
 description = "A simple JIT library backed by Cranelift"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.58.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.58.0" }
+cranelift-module = { path = "../cranelift-module", version = "0.59.0" }
+cranelift-native = { path = "../cranelift-native", version = "0.59.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -20,7 +20,7 @@ memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
-version = "0.58.0"
+version = "0.59.0"
 default-features = false
 features = ["std"]
 
@@ -32,9 +32,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.58.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.58.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.58.0" }
+cranelift = { path = "../cranelift-umbrella", version = "0.59.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.59.0" }
+cranelift-entity = { path = "../cranelift-entity", version = "0.59.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 3196bdfed7..98185b0c04 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 authors = ["The Cranelift Project Developers"]
 name = "cranelift"
-version = "0.58.0"
+version = "0.59.0"
 description = "Umbrella for commonly-used cranelift crates"
 license = "Apache-2.0 WITH LLVM-exception"
 documentation = "https://cranelift.readthedocs.io/"
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.58.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", default-features = false }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.59.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 4ac0496263..5f9dd30551 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cranelift-wasm"
-version = "0.58.0"
+version = "0.59.0"
 authors = ["The Cranelift Project Developers"]
 description = "Translator from WebAssembly to Cranelift IR"
 repository = "https://github.com/bytecodealliance/cranelift"
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.51.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.58.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.58.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.58.0", default-features = false }
+cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", default-features = false }
+cranelift-entity = { path = "../cranelift-entity", version = "0.59.0" }
+cranelift-frontend = { path = "../cranelift-frontend", version = "0.59.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }

From 0a1bb3ba6ccadd4b716cd259a097bb31435c0c6d Mon Sep 17 00:00:00 2001
From: bjorn3 
Date: Wed, 26 Feb 2020 02:50:04 +0100
Subject: [PATCH 3068/3084] Add TLS support for ELF and MachO (#1174)

* Add TLS support
* Add binemit and legalize tests
* Spill all caller-saved registers when necessary
---
 .../codegen/meta/src/cdsl/instructions.rs     |  2 +-
 .../codegen/meta/src/isa/x86/encodings.rs     |  9 +++
 .../codegen/meta/src/isa/x86/instructions.rs  | 36 ++++++++++
 .../codegen/meta/src/isa/x86/legalize.rs      |  3 +
 cranelift/codegen/meta/src/isa/x86/mod.rs     |  1 +
 cranelift/codegen/meta/src/isa/x86/recipes.rs | 67 ++++++++++++++++++-
 .../codegen/meta/src/shared/instructions.rs   | 12 ++++
 cranelift/codegen/meta/src/shared/mod.rs      |  4 +-
 cranelift/codegen/meta/src/shared/settings.rs |  8 +++
 cranelift/codegen/src/binemit/mod.rs          |  9 +++
 cranelift/codegen/src/ir/globalvalue.rs       |  7 +-
 cranelift/codegen/src/ir/libcall.rs           |  5 ++
 cranelift/codegen/src/isa/x86/binemit.rs      |  5 +-
 cranelift/codegen/src/isa/x86/enc_tables.rs   | 37 ++++++++++
 .../codegen/src/legalizer/globalvalue.rs      | 17 ++++-
 cranelift/codegen/src/regalloc/spilling.rs    |  6 +-
 cranelift/codegen/src/settings.rs             |  1 +
 cranelift/faerie/src/backend.rs               |  4 ++
 .../filetests/filetests/isa/x86/tls_elf.clif  | 18 +++++
 .../filetests/filetests/isa/x86/tls_enc.clif  | 11 +++
 .../filetests/isa/x86/tls_macho.clif          | 18 +++++
 cranelift/module/src/backend.rs               |  4 ++
 cranelift/module/src/module.rs                | 16 ++++-
 cranelift/object/Cargo.toml                   |  1 +
 cranelift/object/src/backend.rs               | 64 ++++++++++++++++--
 cranelift/reader/src/parser.rs                |  3 +
 cranelift/simplejit/src/backend.rs            |  5 ++
 27 files changed, 354 insertions(+), 19 deletions(-)
 create mode 100644 cranelift/filetests/filetests/isa/x86/tls_elf.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/tls_enc.clif
 create mode 100644 cranelift/filetests/filetests/isa/x86/tls_macho.clif

diff --git a/cranelift/codegen/meta/src/cdsl/instructions.rs b/cranelift/codegen/meta/src/cdsl/instructions.rs
index f17202eb1c..d8d9c81466 100644
--- a/cranelift/codegen/meta/src/cdsl/instructions.rs
+++ b/cranelift/codegen/meta/src/cdsl/instructions.rs
@@ -340,7 +340,7 @@ impl InstructionBuilder {
         let polymorphic_info =
             verify_polymorphic(&operands_in, &operands_out, &self.format, &value_opnums);
 
-        // Infer from output operands whether an instruciton clobbers CPU flags or not.
+        // Infer from output operands whether an instruction clobbers CPU flags or not.
         let writes_cpu_flags = operands_out.iter().any(|op| op.is_cpu_flags());
 
         let camel_name = camel_case(&self.name);
diff --git a/cranelift/codegen/meta/src/isa/x86/encodings.rs b/cranelift/codegen/meta/src/isa/x86/encodings.rs
index 81e2e48c58..5dd6edc380 100644
--- a/cranelift/codegen/meta/src/isa/x86/encodings.rs
+++ b/cranelift/codegen/meta/src/isa/x86/encodings.rs
@@ -2407,5 +2407,14 @@ pub(crate) fn define(
     define_control_flow(&mut e, shared_defs, settings, r);
     define_reftypes(&mut e, shared_defs, r);
 
+    let x86_elf_tls_get_addr = x86.by_name("x86_elf_tls_get_addr");
+    let x86_macho_tls_get_addr = x86.by_name("x86_macho_tls_get_addr");
+
+    let rec_elf_tls_get_addr = r.recipe("elf_tls_get_addr");
+    let rec_macho_tls_get_addr = r.recipe("macho_tls_get_addr");
+
+    e.enc64_rec(x86_elf_tls_get_addr, rec_elf_tls_get_addr, 0);
+    e.enc64_rec(x86_macho_tls_get_addr, rec_macho_tls_get_addr, 0);
+
     e
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/instructions.rs b/cranelift/codegen/meta/src/isa/x86/instructions.rs
index 04dc6cfe12..3be23cea9d 100644
--- a/cranelift/codegen/meta/src/isa/x86/instructions.rs
+++ b/cranelift/codegen/meta/src/isa/x86/instructions.rs
@@ -7,6 +7,7 @@ use crate::cdsl::operands::Operand;
 use crate::cdsl::types::ValueType;
 use crate::cdsl::typevar::{Interval, TypeSetBuilder, TypeVar};
 
+use crate::shared::entities::EntityRefs;
 use crate::shared::formats::Formats;
 use crate::shared::immediates::Immediates;
 use crate::shared::types;
@@ -16,6 +17,7 @@ pub(crate) fn define(
     mut all_instructions: &mut AllInstructions,
     formats: &Formats,
     immediates: &Immediates,
+    entities: &EntityRefs,
 ) -> InstructionGroup {
     let mut ig = InstructionGroupBuilder::new(&mut all_instructions);
 
@@ -542,5 +544,39 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    let i64_t = &TypeVar::new(
+        "i64_t",
+        "A scalar 64bit integer",
+        TypeSetBuilder::new().ints(64..64).build(),
+    );
+
+    let GV = &Operand::new("GV", &entities.global_value);
+    let addr = &Operand::new("addr", i64_t);
+
+    ig.push(
+        Inst::new(
+            "x86_elf_tls_get_addr",
+            r#"
+        Elf tls get addr -- This implements the GD TLS model for ELF. The clobber output should
+        not be used.
+            "#,
+            &formats.unary_global_value,
+        )
+        .operands_in(vec![GV])
+        .operands_out(vec![addr]),
+    );
+    ig.push(
+        Inst::new(
+            "x86_macho_tls_get_addr",
+            r#"
+        Mach-O tls get addr -- This implements TLS access for Mach-O. The clobber output should
+        not be used.
+            "#,
+            &formats.unary_global_value,
+        )
+        .operands_in(vec![GV])
+        .operands_out(vec![addr]),
+    );
+
     ig.build()
 }
diff --git a/cranelift/codegen/meta/src/isa/x86/legalize.rs b/cranelift/codegen/meta/src/isa/x86/legalize.rs
index aa91741dba..de748a0bae 100644
--- a/cranelift/codegen/meta/src/isa/x86/legalize.rs
+++ b/cranelift/codegen/meta/src/isa/x86/legalize.rs
@@ -61,6 +61,7 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
     let shuffle = insts.by_name("shuffle");
     let srem = insts.by_name("srem");
     let sshr = insts.by_name("sshr");
+    let tls_value = insts.by_name("tls_value");
     let trueif = insts.by_name("trueif");
     let udiv = insts.by_name("udiv");
     let umax = insts.by_name("umax");
@@ -326,6 +327,8 @@ pub(crate) fn define(shared: &mut SharedDefinitions, x86_instructions: &Instruct
 
     group.custom_legalize(ineg, "convert_ineg");
 
+    group.custom_legalize(tls_value, "expand_tls_value");
+
     group.build_and_add_to(&mut shared.transform_groups);
 
     let mut narrow = TransformGroupBuilder::new(
diff --git a/cranelift/codegen/meta/src/isa/x86/mod.rs b/cranelift/codegen/meta/src/isa/x86/mod.rs
index 519ad884ff..3b4848b166 100644
--- a/cranelift/codegen/meta/src/isa/x86/mod.rs
+++ b/cranelift/codegen/meta/src/isa/x86/mod.rs
@@ -24,6 +24,7 @@ pub(crate) fn define(shared_defs: &mut SharedDefinitions) -> TargetIsa {
         &mut shared_defs.all_instructions,
         &shared_defs.formats,
         &shared_defs.imm,
+        &shared_defs.entities,
     );
     legalize::define(shared_defs, &inst_group);
 
diff --git a/cranelift/codegen/meta/src/isa/x86/recipes.rs b/cranelift/codegen/meta/src/isa/x86/recipes.rs
index 5c7cb0519b..4bab09c306 100644
--- a/cranelift/codegen/meta/src/isa/x86/recipes.rs
+++ b/cranelift/codegen/meta/src/isa/x86/recipes.rs
@@ -3259,10 +3259,73 @@ pub(crate) fn define<'shared>(
     recipes.add_recipe(
         EncodingRecipeBuilder::new("safepoint", &formats.multiary, 0).emit(
             r#"
-            sink.add_stackmap(args, func, isa);
-        "#,
+                sink.add_stackmap(args, func, isa);
+            "#,
         ),
     );
 
+    // Both `elf_tls_get_addr` and `macho_tls_get_addr` require all caller-saved registers to be spilled.
+    // This is currently special cased in `regalloc/spilling.rs` in the `visit_inst` function.
+
+    recipes.add_recipe(
+        EncodingRecipeBuilder::new("elf_tls_get_addr", &formats.unary_global_value, 16)
+            // FIXME Correct encoding for non rax registers
+            .operands_out(vec![reg_rax])
+            .emit(
+                r#"
+                    // output %rax
+                    // clobbers %rdi
+
+                    // Those data16 prefixes are necessary to pad to 16 bytes.
+
+                    // data16 lea gv@tlsgd(%rip),%rdi
+                    sink.put1(0x66); // data16
+                    sink.put1(0b01001000); // rex.w
+                    const LEA: u8 = 0x8d;
+                    sink.put1(LEA); // lea
+                    modrm_riprel(0b111/*out_reg0*/, sink); // 0x3d
+                    sink.reloc_external(Reloc::ElfX86_64TlsGd,
+                                        &func.global_values[global_value].symbol_name(),
+                                        -4);
+                    sink.put4(0);
+
+                    // data16 data16 callq __tls_get_addr-4
+                    sink.put1(0x66); // data16
+                    sink.put1(0x66); // data16
+                    sink.put1(0b01001000); // rex.w
+                    sink.put1(0xe8); // call
+                    sink.reloc_external(Reloc::X86CallPLTRel4,
+                                        &ExternalName::LibCall(LibCall::ElfTlsGetAddr),
+                                        -4);
+                    sink.put4(0);
+                "#,
+            ),
+    );
+
+    recipes.add_recipe(
+        EncodingRecipeBuilder::new("macho_tls_get_addr", &formats.unary_global_value, 9)
+            // FIXME Correct encoding for non rax registers
+            .operands_out(vec![reg_rax])
+            .emit(
+                r#"
+                    // output %rax
+                    // clobbers %rdi
+
+                    // movq gv@tlv(%rip), %rdi
+                    sink.put1(0x48); // rex
+                    sink.put1(0x8b); // mov
+                    modrm_riprel(0b111/*out_reg0*/, sink); // 0x3d
+                    sink.reloc_external(Reloc::MachOX86_64Tlv,
+                                        &func.global_values[global_value].symbol_name(),
+                                        -4);
+                    sink.put4(0);
+
+                    // callq *(%rdi)
+                    sink.put1(0xff);
+                    sink.put1(0x17);
+                "#,
+            ),
+    );
+
     recipes
 }
diff --git a/cranelift/codegen/meta/src/shared/instructions.rs b/cranelift/codegen/meta/src/shared/instructions.rs
index 471ad85a56..3e74da1b00 100644
--- a/cranelift/codegen/meta/src/shared/instructions.rs
+++ b/cranelift/codegen/meta/src/shared/instructions.rs
@@ -1127,6 +1127,18 @@ pub(crate) fn define(
         .operands_out(vec![a]),
     );
 
+    ig.push(
+        Inst::new(
+            "tls_value",
+            r#"
+        Compute the value of global GV, which is a TLS (thread local storage) value.
+        "#,
+            &formats.unary_global_value,
+        )
+        .operands_in(vec![GV])
+        .operands_out(vec![a]),
+    );
+
     let HeapOffset = &TypeVar::new(
         "HeapOffset",
         "An unsigned heap offset",
diff --git a/cranelift/codegen/meta/src/shared/mod.rs b/cranelift/codegen/meta/src/shared/mod.rs
index 121e26b10a..b185262ccd 100644
--- a/cranelift/codegen/meta/src/shared/mod.rs
+++ b/cranelift/codegen/meta/src/shared/mod.rs
@@ -1,6 +1,6 @@
 //! Shared definitions for the Cranelift intermediate language.
 
-mod entities;
+pub mod entities;
 pub mod formats;
 pub mod immediates;
 pub mod instructions;
@@ -28,6 +28,7 @@ pub(crate) struct Definitions {
     pub imm: Immediates,
     pub formats: Formats,
     pub transform_groups: TransformGroups,
+    pub entities: EntityRefs,
 }
 
 pub(crate) fn define() -> Definitions {
@@ -47,6 +48,7 @@ pub(crate) fn define() -> Definitions {
         imm: immediates,
         formats,
         transform_groups,
+        entities,
     }
 }
 
diff --git a/cranelift/codegen/meta/src/shared/settings.rs b/cranelift/codegen/meta/src/shared/settings.rs
index 545c2732b3..fd6063e852 100644
--- a/cranelift/codegen/meta/src/shared/settings.rs
+++ b/cranelift/codegen/meta/src/shared/settings.rs
@@ -131,6 +131,14 @@ pub(crate) fn define() -> SettingGroup {
         false,
     );
 
+    settings.add_enum(
+        "tls_model",
+        r#"
+            Defines the model used to perform TLS accesses.
+        "#,
+        vec!["none", "elf_gd", "macho", "coff"],
+    );
+
     // Settings specific to the `baldrdash` calling convention.
 
     settings.add_enum(
diff --git a/cranelift/codegen/src/binemit/mod.rs b/cranelift/codegen/src/binemit/mod.rs
index 7a781e2b56..b52025e887 100644
--- a/cranelift/codegen/src/binemit/mod.rs
+++ b/cranelift/codegen/src/binemit/mod.rs
@@ -56,6 +56,12 @@ pub enum Reloc {
     Arm64Call,
     /// RISC-V call target
     RiscvCall,
+
+    /// Elf x86_64 32 bit signed PC relative offset to two GOT entries for GD symbol.
+    ElfX86_64TlsGd,
+
+    /// Mach-O x86_64 32 bit signed PC relative offset to a `__thread_vars` entry.
+    MachOX86_64Tlv,
 }
 
 impl fmt::Display for Reloc {
@@ -71,6 +77,9 @@ impl fmt::Display for Reloc {
             Self::X86CallPLTRel4 => write!(f, "CallPLTRel4"),
             Self::X86GOTPCRel4 => write!(f, "GOTPCRel4"),
             Self::Arm32Call | Self::Arm64Call | Self::RiscvCall => write!(f, "Call"),
+
+            Self::ElfX86_64TlsGd => write!(f, "ElfX86_64TlsGd"),
+            Self::MachOX86_64Tlv => write!(f, "MachOX86_64Tlv"),
         }
     }
 }
diff --git a/cranelift/codegen/src/ir/globalvalue.rs b/cranelift/codegen/src/ir/globalvalue.rs
index dbba5aa676..305654a95f 100644
--- a/cranelift/codegen/src/ir/globalvalue.rs
+++ b/cranelift/codegen/src/ir/globalvalue.rs
@@ -63,6 +63,9 @@ pub enum GlobalValueData {
         /// away, after linking? If so, references to it can avoid going through a GOT. Note that
         /// symbols meant to be preemptible cannot be colocated.
         colocated: bool,
+
+        /// Does this symbol refer to a thread local storage value?
+        tls: bool,
     },
 }
 
@@ -110,11 +113,13 @@ impl fmt::Display for GlobalValueData {
                 ref name,
                 offset,
                 colocated,
+                tls,
             } => {
                 write!(
                     f,
-                    "symbol {}{}",
+                    "symbol {}{}{}",
                     if colocated { "colocated " } else { "" },
+                    if tls { "tls " } else { "" },
                     name
                 )?;
                 let offset_val: i64 = offset.into();
diff --git a/cranelift/codegen/src/ir/libcall.rs b/cranelift/codegen/src/ir/libcall.rs
index 33d3fa6060..f4f6d941f8 100644
--- a/cranelift/codegen/src/ir/libcall.rs
+++ b/cranelift/codegen/src/ir/libcall.rs
@@ -46,6 +46,9 @@ pub enum LibCall {
     Memset,
     /// libc.memmove
     Memmove,
+
+    /// Elf __tls_get_addr
+    ElfTlsGetAddr,
 }
 
 impl fmt::Display for LibCall {
@@ -71,6 +74,8 @@ impl FromStr for LibCall {
             "Memcpy" => Ok(Self::Memcpy),
             "Memset" => Ok(Self::Memset),
             "Memmove" => Ok(Self::Memmove),
+
+            "ElfTlsGetAddr" => Ok(Self::ElfTlsGetAddr),
             _ => Err(()),
         }
     }
diff --git a/cranelift/codegen/src/isa/x86/binemit.rs b/cranelift/codegen/src/isa/x86/binemit.rs
index 44c497e547..15defe69ee 100644
--- a/cranelift/codegen/src/isa/x86/binemit.rs
+++ b/cranelift/codegen/src/isa/x86/binemit.rs
@@ -4,7 +4,10 @@ use super::enc_tables::{needs_offset, needs_sib_byte};
 use super::registers::RU;
 use crate::binemit::{bad_encoding, CodeSink, Reloc};
 use crate::ir::condcodes::{CondCode, FloatCC, IntCC};
-use crate::ir::{Block, Constant, Function, Inst, InstructionData, JumpTable, Opcode, TrapCode};
+use crate::ir::{
+    Block, Constant, ExternalName, Function, Inst, InstructionData, JumpTable, LibCall, Opcode,
+    TrapCode,
+};
 use crate::isa::{RegUnit, StackBase, StackBaseMask, StackRef, TargetIsa};
 use crate::regalloc::RegDiversions;
 
diff --git a/cranelift/codegen/src/isa/x86/enc_tables.rs b/cranelift/codegen/src/isa/x86/enc_tables.rs
index 076fef3115..93e06d2795 100644
--- a/cranelift/codegen/src/isa/x86/enc_tables.rs
+++ b/cranelift/codegen/src/isa/x86/enc_tables.rs
@@ -1288,3 +1288,40 @@ fn convert_ineg(
         unreachable!()
     }
 }
+
+fn expand_tls_value(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    _cfg: &mut ControlFlowGraph,
+    isa: &dyn TargetIsa,
+) {
+    use crate::settings::TlsModel;
+
+    assert!(
+        isa.triple().architecture == target_lexicon::Architecture::X86_64,
+        "Not yet implemented for {:?}",
+        isa.triple(),
+    );
+
+    if let ir::InstructionData::UnaryGlobalValue {
+        opcode: ir::Opcode::TlsValue,
+        global_value,
+    } = func.dfg[inst]
+    {
+        let ctrl_typevar = func.dfg.ctrl_typevar(inst);
+        assert_eq!(ctrl_typevar, ir::types::I64);
+
+        match isa.flags().tls_model() {
+            TlsModel::None => panic!("tls_model flag is not set."),
+            TlsModel::ElfGd => {
+                func.dfg.replace(inst).x86_elf_tls_get_addr(global_value);
+            }
+            TlsModel::Macho => {
+                func.dfg.replace(inst).x86_macho_tls_get_addr(global_value);
+            }
+            model => unimplemented!("tls_value for tls model {:?}", model),
+        }
+    } else {
+        unreachable!();
+    }
+}
diff --git a/cranelift/codegen/src/legalizer/globalvalue.rs b/cranelift/codegen/src/legalizer/globalvalue.rs
index 97ef58c2bb..5c7a72b45c 100644
--- a/cranelift/codegen/src/legalizer/globalvalue.rs
+++ b/cranelift/codegen/src/legalizer/globalvalue.rs
@@ -40,7 +40,7 @@ pub fn expand_global_value(
             global_type,
             readonly,
         } => load_addr(inst, func, base, offset, global_type, readonly, isa),
-        ir::GlobalValueData::Symbol { .. } => symbol(inst, func, gv, isa),
+        ir::GlobalValueData::Symbol { tls, .. } => symbol(inst, func, gv, isa, tls),
     }
 }
 
@@ -123,7 +123,18 @@ fn load_addr(
 }
 
 /// Expand a `global_value` instruction for a symbolic name global.
-fn symbol(inst: ir::Inst, func: &mut ir::Function, gv: ir::GlobalValue, isa: &dyn TargetIsa) {
+fn symbol(
+    inst: ir::Inst,
+    func: &mut ir::Function,
+    gv: ir::GlobalValue,
+    isa: &dyn TargetIsa,
+    tls: bool,
+) {
     let ptr_ty = isa.pointer_type();
-    func.dfg.replace(inst).symbol_value(ptr_ty, gv);
+
+    if tls {
+        func.dfg.replace(inst).tls_value(ptr_ty, gv);
+    } else {
+        func.dfg.replace(inst).symbol_value(ptr_ty, gv);
+    }
 }
diff --git a/cranelift/codegen/src/regalloc/spilling.rs b/cranelift/codegen/src/regalloc/spilling.rs
index d27c68ae42..e515543260 100644
--- a/cranelift/codegen/src/regalloc/spilling.rs
+++ b/cranelift/codegen/src/regalloc/spilling.rs
@@ -267,7 +267,11 @@ impl<'a> Context<'a> {
         // If inst is a call, spill all register values that are live across the call.
         // This means that we don't currently take advantage of callee-saved registers.
         // TODO: Be more sophisticated.
-        if call_sig.is_some() {
+        let opcode = self.cur.func.dfg[inst].opcode();
+        if call_sig.is_some()
+            || opcode == crate::ir::Opcode::X86ElfTlsGetAddr
+            || opcode == crate::ir::Opcode::X86MachoTlsGetAddr
+        {
             for lv in throughs {
                 if lv.affinity.is_reg() && !self.spills.contains(&lv.value) {
                     self.spill_reg(lv.value);
diff --git a/cranelift/codegen/src/settings.rs b/cranelift/codegen/src/settings.rs
index b6e55b6147..57b9c18f89 100644
--- a/cranelift/codegen/src/settings.rs
+++ b/cranelift/codegen/src/settings.rs
@@ -379,6 +379,7 @@ mod tests {
             f.to_string(),
             "[shared]\n\
              opt_level = \"none\"\n\
+             tls_model = \"none\"\n\
              libcall_call_conv = \"isa_default\"\n\
              baldrdash_prologue_words = 0\n\
              probestack_size_log2 = 12\n\
diff --git a/cranelift/faerie/src/backend.rs b/cranelift/faerie/src/backend.rs
index e6439a69b9..40cca8eddc 100644
--- a/cranelift/faerie/src/backend.rs
+++ b/cranelift/faerie/src/backend.rs
@@ -136,8 +136,10 @@ impl Backend for FaerieBackend {
         name: &str,
         linkage: Linkage,
         writable: bool,
+        tls: bool,
         align: Option,
     ) {
+        assert!(!tls, "Faerie doesn't yet support TLS");
         self.artifact
             .declare(name, translate_data_linkage(linkage, writable, align))
             .expect("inconsistent declarations");
@@ -231,10 +233,12 @@ impl Backend for FaerieBackend {
         _id: DataId,
         name: &str,
         _writable: bool,
+        tls: bool,
         _align: Option,
         data_ctx: &DataContext,
         namespace: &ModuleNamespace,
     ) -> ModuleResult {
+        assert!(!tls, "Faerie doesn't yet support TLS");
         let &DataDescription {
             ref init,
             ref function_decls,
diff --git a/cranelift/filetests/filetests/isa/x86/tls_elf.clif b/cranelift/filetests/filetests/isa/x86/tls_elf.clif
new file mode 100644
index 0000000000..3788dd7d27
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/tls_elf.clif
@@ -0,0 +1,18 @@
+test regalloc
+set tls_model=elf_gd
+target x86_64
+
+function u0:0(i32) -> i32, i64 {
+gv0 = symbol colocated tls u1:0
+
+block0(v0: i32):
+    ; check: block0(v2: i32 [%rdi]):
+    ; nextln: [RexOp1spillSib32#89,ss0]           v0 = spill v2
+    v1 = global_value.i64 gv0
+    ; nextln: [elf_tls_get_addr#00,%rax]          v1 = x86_elf_tls_get_addr gv0
+    ; nextln: [RexOp1fillSib32#8b,%r15]           v3 = fill v0
+    return v0, v1
+    ; nextln: [RexOp1rmov#8089]                   regmove v1, %rax -> %rdx
+    ; nextln: [RexOp1rmov#89]                     regmove v3, %r15 -> %rax
+    ; nextln: [Op1ret#c3]                         return v3, v1
+}
diff --git a/cranelift/filetests/filetests/isa/x86/tls_enc.clif b/cranelift/filetests/filetests/isa/x86/tls_enc.clif
new file mode 100644
index 0000000000..acdda733a5
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/tls_enc.clif
@@ -0,0 +1,11 @@
+test binemit
+target x86_64
+
+function u0:0() -> i64, i64 {
+gv0 = symbol colocated tls u1:0
+
+block0:
+    [-, %rax] v0 = x86_elf_tls_get_addr gv0 ; bin: 66 48 8d 3d ElfX86_64TlsGd(u1:0-4) 00000000 66 66 48 e8 CallPLTRel4(%ElfTlsGetAddr-4) 00000000
+    [-, %rax] v1 = x86_macho_tls_get_addr gv0; bin: 48 8b 3d MachOX86_64Tlv(u1:0-4) 00000000 ff 17
+    return v0, v1
+}
diff --git a/cranelift/filetests/filetests/isa/x86/tls_macho.clif b/cranelift/filetests/filetests/isa/x86/tls_macho.clif
new file mode 100644
index 0000000000..d2c637d2e8
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x86/tls_macho.clif
@@ -0,0 +1,18 @@
+test regalloc
+set tls_model=macho
+target x86_64
+
+function u0:0(i32) -> i32, i64 {
+gv0 = symbol colocated tls u1:0
+
+block0(v0: i32):
+    ; check: block0(v2: i32 [%rdi]):
+    ; nextln: [RexOp1spillSib32#89,ss0]           v0 = spill v2
+    v1 = global_value.i64 gv0
+    ; nextln: [macho_tls_get_addr#00,%rax]        v1 = x86_macho_tls_get_addr gv0
+    ; nextln: [RexOp1fillSib32#8b,%r15]           v3 = fill v0
+    return v0, v1
+    ; nextln: [RexOp1rmov#8089]                   regmove v1, %rax -> %rdx
+    ; nextln: [RexOp1rmov#89]                     regmove v3, %r15 -> %rax
+    ; nextln: [Op1ret#c3]                         return v3, v1
+}
diff --git a/cranelift/module/src/backend.rs b/cranelift/module/src/backend.rs
index 1464c4aefc..0fd8724afa 100644
--- a/cranelift/module/src/backend.rs
+++ b/cranelift/module/src/backend.rs
@@ -72,6 +72,7 @@ where
         name: &str,
         linkage: Linkage,
         writable: bool,
+        tls: bool,
         align: Option,
     );
 
@@ -107,6 +108,7 @@ where
         id: DataId,
         name: &str,
         writable: bool,
+        tls: bool,
         align: Option,
         data_ctx: &DataContext,
         namespace: &ModuleNamespace,
@@ -188,5 +190,7 @@ pub fn default_libcall_names() -> Box String> {
         ir::LibCall::Memcpy => "memcpy".to_owned(),
         ir::LibCall::Memset => "memset".to_owned(),
         ir::LibCall::Memmove => "memmove".to_owned(),
+
+        ir::LibCall::ElfTlsGetAddr => "__tls_get_addr".to_owned(),
     })
 }
diff --git a/cranelift/module/src/module.rs b/cranelift/module/src/module.rs
index 10bca75ac7..5df095b610 100644
--- a/cranelift/module/src/module.rs
+++ b/cranelift/module/src/module.rs
@@ -188,6 +188,7 @@ pub struct DataDeclaration {
     pub name: String,
     pub linkage: Linkage,
     pub writable: bool,
+    pub tls: bool,
     pub align: Option,
 }
 
@@ -206,10 +207,14 @@ impl ModuleData
 where
     B: Backend,
 {
-    fn merge(&mut self, linkage: Linkage, writable: bool, align: Option) {
+    fn merge(&mut self, linkage: Linkage, writable: bool, tls: bool, align: Option) {
         self.decl.linkage = Linkage::merge(self.decl.linkage, linkage);
         self.decl.writable = self.decl.writable || writable;
         self.decl.align = self.decl.align.max(align);
+        assert_eq!(
+            self.decl.tls, tls,
+            "Can't change TLS data object to normal or in the opposite way",
+        );
     }
 }
 
@@ -460,6 +465,7 @@ where
         name: &str,
         linkage: Linkage,
         writable: bool,
+        tls: bool,
         align: Option, // An alignment bigger than 128 is unlikely
     ) -> ModuleResult {
         // TODO: Can we avoid allocating names so often?
@@ -468,12 +474,13 @@ where
             Occupied(entry) => match *entry.get() {
                 FuncOrDataId::Data(id) => {
                     let existing = &mut self.contents.data_objects[id];
-                    existing.merge(linkage, writable, align);
+                    existing.merge(linkage, writable, tls, align);
                     self.backend.declare_data(
                         id,
                         name,
                         existing.decl.linkage,
                         existing.decl.writable,
+                        existing.decl.tls,
                         existing.decl.align,
                     );
                     Ok(id)
@@ -489,13 +496,14 @@ where
                         name: name.to_owned(),
                         linkage,
                         writable,
+                        tls,
                         align,
                     },
                     compiled: None,
                 });
                 entry.insert(FuncOrDataId::Data(id));
                 self.backend
-                    .declare_data(id, name, linkage, writable, align);
+                    .declare_data(id, name, linkage, writable, tls, align);
                 Ok(id)
             }
         }
@@ -526,6 +534,7 @@ where
             name: ir::ExternalName::user(1, data.as_u32()),
             offset: ir::immediates::Imm64::new(0),
             colocated,
+            tls: decl.tls,
         })
     }
 
@@ -634,6 +643,7 @@ where
                 data,
                 &info.decl.name,
                 info.decl.writable,
+                info.decl.tls,
                 info.decl.align,
                 data_ctx,
                 &ModuleNamespace:: {
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 030d49c444..74007aab80 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -13,6 +13,7 @@ edition = "2018"
 cranelift-module = { path = "../cranelift-module", version = "0.59.0" }
 object = { version = "0.17", default-features = false, features = ["write"] }
 target-lexicon = "0.10"
+goblin = "0.1.0"
 
 [dependencies.cranelift-codegen]
 path = "../cranelift-codegen"
diff --git a/cranelift/object/src/backend.rs b/cranelift/object/src/backend.rs
index c4fa5eaf72..044fdca815 100644
--- a/cranelift/object/src/backend.rs
+++ b/cranelift/object/src/backend.rs
@@ -17,7 +17,7 @@ use object::write::{
 use object::{RelocationEncoding, RelocationKind, SymbolFlags, SymbolKind, SymbolScope};
 use std::collections::HashMap;
 use std::mem;
-use target_lexicon::PointerWidth;
+use target_lexicon::{BinaryFormat, PointerWidth};
 
 #[derive(Debug)]
 /// Setting to enable collection of traps. Setting this to `Enabled` in
@@ -151,12 +151,19 @@ impl Backend for ObjectBackend {
         name: &str,
         linkage: Linkage,
         _writable: bool,
+        tls: bool,
         _align: Option,
     ) {
+        let kind = if tls {
+            SymbolKind::Tls
+        } else {
+            SymbolKind::Data
+        };
         let (scope, weak) = translate_linkage(linkage);
 
         if let Some(data) = self.data_objects[id] {
             let symbol = self.object.symbol_mut(data);
+            symbol.kind = kind;
             symbol.scope = scope;
             symbol.weak = weak;
         } else {
@@ -164,7 +171,7 @@ impl Backend for ObjectBackend {
                 name: name.as_bytes().to_vec(),
                 value: 0,
                 size: 0,
-                kind: SymbolKind::Data,
+                kind,
                 scope,
                 weak,
                 section: SymbolSection::Undefined,
@@ -183,7 +190,7 @@ impl Backend for ObjectBackend {
         code_size: u32,
     ) -> ModuleResult {
         let mut code: Vec = vec![0; code_size as usize];
-        let mut reloc_sink = ObjectRelocSink::default();
+        let mut reloc_sink = ObjectRelocSink::new(self.object.format());
         let mut trap_sink = ObjectTrapSink::default();
         let mut stackmap_sink = NullStackmapSink {};
 
@@ -248,6 +255,7 @@ impl Backend for ObjectBackend {
         data_id: DataId,
         _name: &str,
         writable: bool,
+        tls: bool,
         align: Option,
         data_ctx: &DataContext,
         _namespace: &ModuleNamespace,
@@ -289,7 +297,13 @@ impl Backend for ObjectBackend {
 
         let symbol = self.data_objects[data_id].unwrap();
         let section_kind = if let Init::Zeros { .. } = *init {
-            StandardSection::UninitializedData
+            if tls {
+                StandardSection::UninitializedTls
+            } else {
+                StandardSection::UninitializedData
+            }
+        } else if tls {
+            StandardSection::Tls
         } else if writable {
             StandardSection::Data
         } else if relocs.is_empty() {
@@ -519,11 +533,20 @@ struct RelocRecord {
     addend: Addend,
 }
 
-#[derive(Default)]
 struct ObjectRelocSink {
+    format: BinaryFormat,
     relocs: Vec,
 }
 
+impl ObjectRelocSink {
+    fn new(format: BinaryFormat) -> Self {
+        Self {
+            format,
+            relocs: vec![],
+        }
+    }
+}
+
 impl RelocSink for ObjectRelocSink {
     fn reloc_block(&mut self, _offset: CodeOffset, _reloc: Reloc, _block_offset: CodeOffset) {
         unimplemented!();
@@ -534,7 +557,7 @@ impl RelocSink for ObjectRelocSink {
         offset: CodeOffset,
         reloc: Reloc,
         name: &ir::ExternalName,
-        addend: Addend,
+        mut addend: Addend,
     ) {
         let (kind, encoding, size) = match reloc {
             Reloc::Abs4 => (RelocationKind::Absolute, RelocationEncoding::Generic, 32),
@@ -549,6 +572,35 @@ impl RelocSink for ObjectRelocSink {
                 32,
             ),
             Reloc::X86GOTPCRel4 => (RelocationKind::GotRelative, RelocationEncoding::Generic, 32),
+
+            Reloc::ElfX86_64TlsGd => {
+                assert_eq!(
+                    self.format,
+                    BinaryFormat::Elf,
+                    "ElfX86_64TlsGd is not supported for this file format"
+                );
+                (
+                    RelocationKind::Elf(goblin::elf64::reloc::R_X86_64_TLSGD),
+                    RelocationEncoding::Generic,
+                    32,
+                )
+            }
+            Reloc::MachOX86_64Tlv => {
+                assert_eq!(
+                    self.format,
+                    BinaryFormat::Macho,
+                    "MachOX86_64Tlv is not supported for this file format"
+                );
+                addend += 4; // X86_64_RELOC_TLV has an implicit addend of -4
+                (
+                    RelocationKind::MachO {
+                        value: goblin::mach::relocation::X86_64_RELOC_TLV,
+                        relative: true,
+                    },
+                    RelocationEncoding::Generic,
+                    32,
+                )
+            }
             // FIXME
             _ => unimplemented!(),
         };
diff --git a/cranelift/reader/src/parser.rs b/cranelift/reader/src/parser.rs
index 6cedb25b0d..02bf24ee8e 100644
--- a/cranelift/reader/src/parser.rs
+++ b/cranelift/reader/src/parser.rs
@@ -204,6 +204,7 @@ impl<'a> Context<'a> {
                 name: ExternalName::testcase(""),
                 offset: Imm64::new(0),
                 colocated: false,
+                tls: false,
             });
         }
         self.function.global_values[gv] = data;
@@ -1443,12 +1444,14 @@ impl<'a> Parser<'a> {
             }
             "symbol" => {
                 let colocated = self.optional(Token::Identifier("colocated"));
+                let tls = self.optional(Token::Identifier("tls"));
                 let name = self.parse_external_name()?;
                 let offset = self.optional_offset_imm64()?;
                 GlobalValueData::Symbol {
                     name,
                     offset,
                     colocated,
+                    tls,
                 }
             }
             other => return err!(self.loc, "Unknown global value kind '{}'", other),
diff --git a/cranelift/simplejit/src/backend.rs b/cranelift/simplejit/src/backend.rs
index 8d37b99920..08bc565133 100644
--- a/cranelift/simplejit/src/backend.rs
+++ b/cranelift/simplejit/src/backend.rs
@@ -264,8 +264,10 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         _name: &str,
         _linkage: Linkage,
         _writable: bool,
+        tls: bool,
         _align: Option,
     ) {
+        assert!(!tls, "SimpleJIT doesn't yet support TLS");
         // Nothing to do.
     }
 
@@ -341,10 +343,13 @@ impl<'simple_jit_backend> Backend for SimpleJITBackend {
         _id: DataId,
         _name: &str,
         writable: bool,
+        tls: bool,
         align: Option,
         data: &DataContext,
         _namespace: &ModuleNamespace,
     ) -> ModuleResult {
+        assert!(!tls, "SimpleJIT doesn't yet support TLS");
+
         let &DataDescription {
             ref init,
             ref function_decls,

From 6a61bba39e1b2586d7a8c84cae85dc4587d6d606 Mon Sep 17 00:00:00 2001
From: Joshua Nelson 
Date: Thu, 27 Feb 2020 16:45:18 -0500
Subject: [PATCH 3069/3084] Allow modules to have names of arbitrary bytes
 (#1410)

This is useful for me because I name the module after the file, which
comes from the filesystem and may not be valid UTF8.

This change is backwards-compatible.
---
 cranelift/object/src/backend.rs | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/cranelift/object/src/backend.rs b/cranelift/object/src/backend.rs
index 044fdca815..dae26ea230 100644
--- a/cranelift/object/src/backend.rs
+++ b/cranelift/object/src/backend.rs
@@ -32,7 +32,7 @@ pub enum ObjectTrapCollection {
 /// A builder for `ObjectBackend`.
 pub struct ObjectBuilder {
     isa: Box,
-    name: String,
+    name: Vec,
     collect_traps: ObjectTrapCollection,
     libcall_names: Box String>,
     function_alignment: u64,
@@ -49,15 +49,15 @@ impl ObjectBuilder {
     /// enum to symbols. LibCalls are inserted in the IR as part of the legalization for certain
     /// floating point instructions, and for stack probes. If you don't know what to use for this
     /// argument, use `cranelift_module::default_libcall_names()`.
-    pub fn new(
+    pub fn new>>(
         isa: Box,
-        name: String,
+        name: V,
         collect_traps: ObjectTrapCollection,
         libcall_names: Box String>,
     ) -> Self {
         Self {
             isa,
-            name,
+            name: name.into(),
             collect_traps,
             libcall_names,
             function_alignment: 1,
@@ -104,7 +104,7 @@ impl Backend for ObjectBackend {
     fn new(builder: ObjectBuilder) -> Self {
         let triple = builder.isa.triple();
         let mut object = Object::new(triple.binary_format, triple.architecture);
-        object.add_file_symbol(builder.name.as_bytes().to_vec());
+        object.add_file_symbol(builder.name);
         Self {
             isa: builder.isa,
             object,

From 1c78f03bf312d731c3643ba4e616b2add29727b7 Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 25 Feb 2020 10:54:16 -0800
Subject: [PATCH 3070/3084] Remove redundant files

---
 cranelift/CODE_OF_CONDUCT.md | 49 ------------------------------------
 cranelift/format-all.sh      | 12 ---------
 2 files changed, 61 deletions(-)
 delete mode 100644 cranelift/CODE_OF_CONDUCT.md
 delete mode 100755 cranelift/format-all.sh

diff --git a/cranelift/CODE_OF_CONDUCT.md b/cranelift/CODE_OF_CONDUCT.md
deleted file mode 100644
index 5c5ebdd259..0000000000
--- a/cranelift/CODE_OF_CONDUCT.md
+++ /dev/null
@@ -1,49 +0,0 @@
-# Contributor Covenant Code of Conduct
-
-*Note*: this Code of Conduct pertains to individuals' behavior. Please also see the [Organizational Code of Conduct][OCoC].
-
-## Our Pledge
-
-In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
-
-## Our Standards
-
-Examples of behavior that contributes to creating a positive environment include:
-
-* Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
-* Focusing on what is best for the community
-* Showing empathy towards other community members
-
-Examples of unacceptable behavior by participants include:
-
-* The use of sexualized language or imagery and unwelcome sexual attention or advances
-* Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or electronic address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a professional setting
-
-## Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
-
-## Scope
-
-This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the Bytecode Alliance CoC team at [report@bytecodealliance.org](mailto:report@bytecodealliance.org). The CoC team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The CoC team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
-
-Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the Bytecode Alliance's leadership.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
-
-[OCoC]: ORG_CODE_OF_CONDUCT.md
-[homepage]: https://www.contributor-covenant.org
-[version]: https://www.contributor-covenant.org/version/1/4/
diff --git a/cranelift/format-all.sh b/cranelift/format-all.sh
deleted file mode 100755
index 64c967ad24..0000000000
--- a/cranelift/format-all.sh
+++ /dev/null
@@ -1,12 +0,0 @@
-#!/bin/bash
-set -euo pipefail
-
-# Format all sources using rustfmt.
-
-topdir=$(dirname "$0")
-cd "$topdir"
-
-# Make sure we can find rustfmt.
-export PATH="$PATH:$HOME/.cargo/bin"
-
-exec cargo +stable fmt --all -- "$@"

From 5f1cba0b7f90e3604b3dfb83119b09c242f8d3e5 Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 25 Feb 2020 10:59:59 -0800
Subject: [PATCH 3071/3084] Hook up all crates via path dependencies

---
 Cargo.lock                        | 82 ++++++++++++++++++++++---------
 cranelift/Cargo.toml              | 32 ++++++------
 cranelift/bforest/Cargo.toml      |  2 +-
 cranelift/codegen/Cargo.toml      |  4 +-
 cranelift/codegen/meta/Cargo.toml |  2 +-
 cranelift/faerie/Cargo.toml       |  4 +-
 cranelift/filetests/Cargo.toml    |  8 +--
 cranelift/frontend/Cargo.toml     |  2 +-
 cranelift/fuzz/Cargo.toml         |  6 +--
 cranelift/module/Cargo.toml       |  4 +-
 cranelift/native/Cargo.toml       |  2 +-
 cranelift/object/Cargo.toml       |  4 +-
 cranelift/preopt/Cargo.toml       |  4 +-
 cranelift/reader/Cargo.toml       |  2 +-
 cranelift/serde/Cargo.toml        |  4 +-
 cranelift/simplejit/Cargo.toml    | 12 ++---
 cranelift/umbrella/Cargo.toml     |  4 +-
 cranelift/wasm/Cargo.toml         |  6 +--
 crates/environ/Cargo.toml         |  8 +--
 crates/jit/Cargo.toml             | 10 ++--
 crates/lightbeam/Cargo.toml       |  2 +-
 21 files changed, 117 insertions(+), 87 deletions(-)

diff --git a/Cargo.lock b/Cargo.lock
index 1df0ef9d1b..3a4b15a1f8 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -6,6 +6,15 @@ version = "1.0.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "5d2e7343e7fc9de883d1b0341e0b13970f764c14101234857d2ddafa1cb1cac2"
 
+[[package]]
+name = "ahash"
+version = "0.2.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6f33b5018f120946c1dcf279194f238a9f146725593ead1c08fa47ff22b0b5d3"
+dependencies = [
+ "const-random",
+]
+
 [[package]]
 name = "aho-corasick"
 version = "0.7.8"
@@ -68,6 +77,12 @@ dependencies = [
  "winapi",
 ]
 
+[[package]]
+name = "autocfg"
+version = "0.1.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1d49d90015b3c36167a20fe2810c5cd875ad504b39cff3d4eae7977e6b7c1cb2"
+
 [[package]]
 name = "autocfg"
 version = "1.0.0"
@@ -305,6 +320,26 @@ dependencies = [
  "cc",
 ]
 
+[[package]]
+name = "const-random"
+version = "0.1.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2f1af9ac737b2dd2d577701e59fd09ba34822f6f2ebdb30a7647405d9e55e16a"
+dependencies = [
+ "const-random-macro",
+ "proc-macro-hack",
+]
+
+[[package]]
+name = "const-random-macro"
+version = "0.1.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "25e4c606eb459dd29f7c57b2e0879f2b6f14ee130918c2b78ccb58a9624e6c7a"
+dependencies = [
+ "getrandom",
+ "proc-macro-hack",
+]
+
 [[package]]
 name = "constant_time_eq"
 version = "0.1.5"
@@ -324,8 +359,6 @@ dependencies = [
 [[package]]
 name = "cranelift-bforest"
 version = "0.59.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "45a9c21f8042b9857bda93f6c1910b9f9f24100187a3d3d52f214a34e3dc5818"
 dependencies = [
  "cranelift-entity",
 ]
@@ -333,8 +366,6 @@ dependencies = [
 [[package]]
 name = "cranelift-codegen"
 version = "0.59.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7853f77a6e4a33c67a69c40f5e1bb982bd2dc5c4a22e17e67b65bbccf9b33b2e"
 dependencies = [
  "byteorder",
  "cranelift-bforest",
@@ -342,6 +373,7 @@ dependencies = [
  "cranelift-codegen-shared",
  "cranelift-entity",
  "gimli",
+ "hashbrown",
  "log",
  "serde",
  "smallvec",
@@ -352,8 +384,6 @@ dependencies = [
 [[package]]
 name = "cranelift-codegen-meta"
 version = "0.59.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "084cd6d5fb0d1da28acd72c199471bfb09acc703ec8f3bf07b1699584272a3b9"
 dependencies = [
  "cranelift-codegen-shared",
  "cranelift-entity",
@@ -362,14 +392,10 @@ dependencies = [
 [[package]]
 name = "cranelift-codegen-shared"
 version = "0.59.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "701b599783305a58c25027a4d73f2d6b599b2d8ef3f26677275f480b4d51e05d"
 
 [[package]]
 name = "cranelift-entity"
 version = "0.59.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b88e792b28e1ebbc0187b72ba5ba880dad083abe9231a99d19604d10c9e73f38"
 dependencies = [
  "serde",
 ]
@@ -377,10 +403,9 @@ dependencies = [
 [[package]]
 name = "cranelift-frontend"
 version = "0.59.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "518344698fa6c976d853319218415fdfb4f1bc6b42d0b2e2df652e55dff1f778"
 dependencies = [
  "cranelift-codegen",
+ "hashbrown",
  "log",
  "smallvec",
  "target-lexicon",
@@ -389,8 +414,6 @@ dependencies = [
 [[package]]
 name = "cranelift-native"
 version = "0.59.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "32daf082da21c0c05d93394ff4842c2ab7c4991b1f3186a1d952f8ac660edd0b"
 dependencies = [
  "cranelift-codegen",
  "raw-cpuid",
@@ -400,16 +423,17 @@ dependencies = [
 [[package]]
 name = "cranelift-wasm"
 version = "0.59.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e2aa816f554a3ef739a5d17ca3081a1f8983f04c944ea8ff60fb8d9dd8cd2d7b"
 dependencies = [
  "cranelift-codegen",
  "cranelift-entity",
  "cranelift-frontend",
+ "hashbrown",
  "log",
  "serde",
+ "target-lexicon",
  "thiserror",
  "wasmparser 0.51.2",
+ "wat",
 ]
 
 [[package]]
@@ -438,7 +462,7 @@ version = "0.8.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "058ed274caafc1f60c4997b5fc07bf7dc7cca454af7c6e81edffe5f33f70dace"
 dependencies = [
- "autocfg",
+ "autocfg 1.0.0",
  "cfg-if",
  "crossbeam-utils",
  "lazy_static",
@@ -463,7 +487,7 @@ version = "0.7.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c3c7c73a2d1e9fc0886a08b93e98eb643461230d5f1925e4036204d5f2e261a8"
 dependencies = [
- "autocfg",
+ "autocfg 1.0.0",
  "cfg-if",
  "lazy_static",
 ]
@@ -776,6 +800,16 @@ dependencies = [
  "scroll",
 ]
 
+[[package]]
+name = "hashbrown"
+version = "0.6.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8e6073d0ca812575946eb5f35ff68dbe519907b25c42530389ff946dc84c6ead"
+dependencies = [
+ "ahash",
+ "autocfg 0.1.7",
+]
+
 [[package]]
 name = "heck"
 version = "0.3.1"
@@ -815,7 +849,7 @@ version = "1.3.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "076f042c5b7b98f31d205f1249267e12a6518c1481e9dae9764af19b707d2292"
 dependencies = [
- "autocfg",
+ "autocfg 1.0.0",
 ]
 
 [[package]]
@@ -1053,7 +1087,7 @@ version = "0.2.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "b6b19411a9719e753aff12e5187b74d60d3dc449ec3f4dc21e3989c3f554bc95"
 dependencies = [
- "autocfg",
+ "autocfg 1.0.0",
  "num-traits",
 ]
 
@@ -1063,7 +1097,7 @@ version = "0.1.42"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "3f6ea62e9d81a77cd3ee9a2a5b9b609447857f3d358704331e4ef39eb247fcba"
 dependencies = [
- "autocfg",
+ "autocfg 1.0.0",
  "num-traits",
 ]
 
@@ -1073,7 +1107,7 @@ version = "0.1.40"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "dfb0800a0291891dd9f4fe7bd9c19384f98f7fbe0cd0f39a2c6b88b9868bbc00"
 dependencies = [
- "autocfg",
+ "autocfg 1.0.0",
  "num-integer",
  "num-traits",
 ]
@@ -1084,7 +1118,7 @@ version = "0.2.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "da4dc79f9e6c81bef96148c8f6b8e72ad4541caa4a24373e900a36da07de03a3"
 dependencies = [
- "autocfg",
+ "autocfg 1.0.0",
  "num-integer",
  "num-traits",
 ]
@@ -1095,7 +1129,7 @@ version = "0.2.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c62be47e61d1842b9170f0fdeec8eba98e60e90e5446449a0545e5152acd7096"
 dependencies = [
- "autocfg",
+ "autocfg 1.0.0",
 ]
 
 [[package]]
diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index c93d53c9a0..a1836037ec 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -9,30 +9,26 @@ repository = "https://github.com/bytecodealliance/cranelift"
 publish = false
 edition = "2018"
 
-# Present here only to make sure that cargo test --all runs tests for all
-# the crates.
-[workspace]
-
 [[bin]]
 name = "clif-util"
 path = "src/clif-util.rs"
 
 [dependencies]
 cfg-if = "0.1"
-cranelift-codegen = { path = "cranelift-codegen", version = "0.59.0" }
-cranelift-entity = { path = "cranelift-entity", version = "0.59.0" }
-cranelift-reader = { path = "cranelift-reader", version = "0.59.0" }
-cranelift-frontend = { path = "cranelift-frontend", version = "0.59.0" }
-cranelift-serde = { path = "cranelift-serde", version = "0.59.0", optional = true }
-cranelift-wasm = { path = "cranelift-wasm", version = "0.59.0", optional = true }
-cranelift-native = { path = "cranelift-native", version = "0.59.0" }
-cranelift-filetests = { path = "cranelift-filetests", version = "0.59.0" }
-cranelift-module = { path = "cranelift-module", version = "0.59.0" }
-cranelift-faerie = { path = "cranelift-faerie", version = "0.59.0" }
-cranelift-object = { path = "cranelift-object", version = "0.59.0" }
-cranelift-simplejit = { path = "cranelift-simplejit", version = "0.59.0" }
-cranelift-preopt = { path = "cranelift-preopt", version = "0.59.0" }
-cranelift = { path = "cranelift-umbrella", version = "0.59.0" }
+cranelift-codegen = { path = "codegen", version = "0.59.0" }
+cranelift-entity = { path = "entity", version = "0.59.0" }
+cranelift-reader = { path = "reader", version = "0.59.0" }
+cranelift-frontend = { path = "frontend", version = "0.59.0" }
+cranelift-serde = { path = "serde", version = "0.59.0", optional = true }
+cranelift-wasm = { path = "wasm", version = "0.59.0", optional = true }
+cranelift-native = { path = "native", version = "0.59.0" }
+cranelift-filetests = { path = "filetests", version = "0.59.0" }
+cranelift-module = { path = "module", version = "0.59.0" }
+cranelift-faerie = { path = "faerie", version = "0.59.0" }
+cranelift-object = { path = "object", version = "0.59.0" }
+cranelift-simplejit = { path = "simplejit", version = "0.59.0" }
+cranelift-preopt = { path = "preopt", version = "0.59.0" }
+cranelift = { path = "umbrella", version = "0.59.0" }
 filecheck = "0.4.0"
 clap = "2.32.0"
 serde = "1.0.8"
diff --git a/cranelift/bforest/Cargo.toml b/cranelift/bforest/Cargo.toml
index 6c70e2e551..11ea1836d0 100644
--- a/cranelift/bforest/Cargo.toml
+++ b/cranelift/bforest/Cargo.toml
@@ -12,7 +12,7 @@ keywords = ["btree", "forest", "set", "map"]
 edition = "2018"
 
 [dependencies]
-cranelift-entity = { path = "../cranelift-entity", version = "0.59.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.59.0", default-features = false }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/codegen/Cargo.toml b/cranelift/codegen/Cargo.toml
index 8a693994dd..ce8773b67c 100644
--- a/cranelift/codegen/Cargo.toml
+++ b/cranelift/codegen/Cargo.toml
@@ -14,8 +14,8 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen-shared = { path = "./shared", version = "0.59.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.59.0" }
-cranelift-bforest = { path = "../cranelift-bforest", version = "0.59.0" }
+cranelift-entity = { path = "../entity", version = "0.59.0" }
+cranelift-bforest = { path = "../bforest", version = "0.59.0" }
 hashbrown = { version = "0.6", optional = true }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
diff --git a/cranelift/codegen/meta/Cargo.toml b/cranelift/codegen/meta/Cargo.toml
index b26c12381a..cc60048144 100644
--- a/cranelift/codegen/meta/Cargo.toml
+++ b/cranelift/codegen/meta/Cargo.toml
@@ -10,7 +10,7 @@ edition = "2018"
 
 [dependencies]
 cranelift-codegen-shared = { path = "../shared", version = "0.59.0" }
-cranelift-entity = { path = "../../cranelift-entity", version = "0.59.0" }
+cranelift-entity = { path = "../../entity", version = "0.59.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/faerie/Cargo.toml b/cranelift/faerie/Cargo.toml
index fcd67551fc..63d26264e3 100644
--- a/cranelift/faerie/Cargo.toml
+++ b/cranelift/faerie/Cargo.toml
@@ -10,14 +10,14 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.59.0" }
+cranelift-module = { path = "../module", version = "0.59.0" }
 faerie = "0.14.0"
 goblin = "0.1.0"
 anyhow = "1.0"
 target-lexicon = "0.10"
 
 [dependencies.cranelift-codegen]
-path = "../cranelift-codegen"
+path = "../codegen"
 version = "0.59.0"
 default-features = false
 features = ["std"]
diff --git a/cranelift/filetests/Cargo.toml b/cranelift/filetests/Cargo.toml
index f773773c19..1ed5461341 100644
--- a/cranelift/filetests/Cargo.toml
+++ b/cranelift/filetests/Cargo.toml
@@ -10,10 +10,10 @@ publish = false
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", features = ["testing_hooks"] }
-cranelift-native = { path = "../cranelift-native", version = "0.59.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.59.0" }
-cranelift-preopt = { path = "../cranelift-preopt", version = "0.59.0" }
+cranelift-codegen = { path = "../codegen", version = "0.59.0", features = ["testing_hooks"] }
+cranelift-native = { path = "../native", version = "0.59.0" }
+cranelift-reader = { path = "../reader", version = "0.59.0" }
+cranelift-preopt = { path = "../preopt", version = "0.59.0" }
 file-per-thread-logger = "0.1.2"
 filecheck = "0.4.0"
 gimli = { version = "0.20.0", default-features = false, features = ["read"] }
diff --git a/cranelift/frontend/Cargo.toml b/cranelift/frontend/Cargo.toml
index 39e0c5e594..0b6a0dc2b5 100644
--- a/cranelift/frontend/Cargo.toml
+++ b/cranelift/frontend/Cargo.toml
@@ -11,7 +11,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.59.0", default-features = false }
 target-lexicon = "0.10"
 log = { version = "0.4.6", default-features = false }
 hashbrown = { version = "0.6", optional = true }
diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
index d2dca75c64..948a07abc9 100644
--- a/cranelift/fuzz/Cargo.toml
+++ b/cranelift/fuzz/Cargo.toml
@@ -12,9 +12,9 @@ cargo-fuzz = true
 cargo-fuzz = "*"
 binaryen = { git = "https://github.com/pepyakin/binaryen-rs.git" }
 libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" }
-cranelift-codegen = { path = "../cranelift-codegen" }
-cranelift-wasm = { path = "../cranelift-wasm" }
-cranelift-reader = { path = "../cranelift-reader" }
+cranelift-codegen = { path = "../codegen" }
+cranelift-wasm = { path = "../wasm" }
+cranelift-reader = { path = "../reader" }
 target-lexicon = "0.10"
 
 # Prevent this from interfering with workspaces
diff --git a/cranelift/module/Cargo.toml b/cranelift/module/Cargo.toml
index 7ece676ca5..22f124b713 100644
--- a/cranelift/module/Cargo.toml
+++ b/cranelift/module/Cargo.toml
@@ -11,8 +11,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.59.0" }
+cranelift-codegen = { path = "../codegen", version = "0.59.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.59.0" }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 thiserror = "1.0.4"
diff --git a/cranelift/native/Cargo.toml b/cranelift/native/Cargo.toml
index 40c58922b6..ae571edffe 100644
--- a/cranelift/native/Cargo.toml
+++ b/cranelift/native/Cargo.toml
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.59.0", default-features = false }
 target-lexicon = "0.10"
 
 [target.'cfg(any(target_arch = "x86", target_arch = "x86_64"))'.dependencies]
diff --git a/cranelift/object/Cargo.toml b/cranelift/object/Cargo.toml
index 74007aab80..2ec0e3902c 100644
--- a/cranelift/object/Cargo.toml
+++ b/cranelift/object/Cargo.toml
@@ -10,13 +10,13 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.59.0" }
+cranelift-module = { path = "../module", version = "0.59.0" }
 object = { version = "0.17", default-features = false, features = ["write"] }
 target-lexicon = "0.10"
 goblin = "0.1.0"
 
 [dependencies.cranelift-codegen]
-path = "../cranelift-codegen"
+path = "../codegen"
 version = "0.59.0"
 default-features = false
 features = ["std"]
diff --git a/cranelift/preopt/Cargo.toml b/cranelift/preopt/Cargo.toml
index dfee57b887..a62b990284 100644
--- a/cranelift/preopt/Cargo.toml
+++ b/cranelift/preopt/Cargo.toml
@@ -12,8 +12,8 @@ keywords = ["optimize", "compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.59.0" }
+cranelift-codegen = { path = "../codegen", version = "0.59.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.59.0" }
 # This is commented out because it doesn't build on Rust 1.25.0, which
 # cranelift currently supports.
 # rustc_apfloat = { version = "0.1.2", default-features = false }
diff --git a/cranelift/reader/Cargo.toml b/cranelift/reader/Cargo.toml
index 190ea86225..8ecda5e776 100644
--- a/cranelift/reader/Cargo.toml
+++ b/cranelift/reader/Cargo.toml
@@ -10,7 +10,7 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0" }
+cranelift-codegen = { path = "../codegen", version = "0.59.0" }
 target-lexicon = "0.10"
 
 [badges]
diff --git a/cranelift/serde/Cargo.toml b/cranelift/serde/Cargo.toml
index d3f091a0ef..6ab2ae0045 100644
--- a/cranelift/serde/Cargo.toml
+++ b/cranelift/serde/Cargo.toml
@@ -18,8 +18,8 @@ clap = "2.32.0"
 serde = "1.0.8"
 serde_derive = "1.0.75"
 serde_json = "1.0.26"
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0" }
-cranelift-reader = { path = "../cranelift-reader", version = "0.59.0" }
+cranelift-codegen = { path = "../codegen", version = "0.59.0" }
+cranelift-reader = { path = "../reader", version = "0.59.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/simplejit/Cargo.toml b/cranelift/simplejit/Cargo.toml
index 1e83e6c797..845551ff3a 100644
--- a/cranelift/simplejit/Cargo.toml
+++ b/cranelift/simplejit/Cargo.toml
@@ -10,8 +10,8 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-module = { path = "../cranelift-module", version = "0.59.0" }
-cranelift-native = { path = "../cranelift-native", version = "0.59.0" }
+cranelift-module = { path = "../module", version = "0.59.0" }
+cranelift-native = { path = "../native", version = "0.59.0" }
 region = "2.0.0"
 libc = { version = "0.2.42" }
 errno = "0.2.4"
@@ -19,7 +19,7 @@ target-lexicon = "0.10"
 memmap = { version = "0.7.0", optional = true }
 
 [dependencies.cranelift-codegen]
-path = "../cranelift-codegen"
+path = "../codegen"
 version = "0.59.0"
 default-features = false
 features = ["std"]
@@ -32,9 +32,9 @@ selinux-fix = ['memmap']
 default = []
 
 [dev-dependencies]
-cranelift = { path = "../cranelift-umbrella", version = "0.59.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.59.0" }
-cranelift-entity = { path = "../cranelift-entity", version = "0.59.0" }
+cranelift = { path = "../umbrella", version = "0.59.0" }
+cranelift-frontend = { path = "../frontend", version = "0.59.0" }
+cranelift-entity = { path = "../entity", version = "0.59.0" }
 
 [badges]
 maintenance = { status = "experimental" }
diff --git a/cranelift/umbrella/Cargo.toml b/cranelift/umbrella/Cargo.toml
index 98185b0c04..f3a1aa8976 100644
--- a/cranelift/umbrella/Cargo.toml
+++ b/cranelift/umbrella/Cargo.toml
@@ -12,8 +12,8 @@ keywords = ["compile", "compiler", "jit"]
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", default-features = false }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.59.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.59.0", default-features = false }
+cranelift-frontend = { path = "../frontend", version = "0.59.0", default-features = false }
 
 [features]
 default = ["std"]
diff --git a/cranelift/wasm/Cargo.toml b/cranelift/wasm/Cargo.toml
index 5f9dd30551..bc5bb83f7b 100644
--- a/cranelift/wasm/Cargo.toml
+++ b/cranelift/wasm/Cargo.toml
@@ -12,9 +12,9 @@ edition = "2018"
 
 [dependencies]
 wasmparser = { version = "0.51.0", default-features = false }
-cranelift-codegen = { path = "../cranelift-codegen", version = "0.59.0", default-features = false }
-cranelift-entity = { path = "../cranelift-entity", version = "0.59.0" }
-cranelift-frontend = { path = "../cranelift-frontend", version = "0.59.0", default-features = false }
+cranelift-codegen = { path = "../codegen", version = "0.59.0", default-features = false }
+cranelift-entity = { path = "../entity", version = "0.59.0" }
+cranelift-frontend = { path = "../frontend", version = "0.59.0", default-features = false }
 hashbrown = { version = "0.6", optional = true }
 log = { version = "0.4.6", default-features = false }
 serde = { version = "1.0.94", features = ["derive"], optional = true }
diff --git a/crates/environ/Cargo.toml b/crates/environ/Cargo.toml
index 66b2c38470..d90f9cd3cf 100644
--- a/crates/environ/Cargo.toml
+++ b/crates/environ/Cargo.toml
@@ -13,9 +13,9 @@ edition = "2018"
 
 [dependencies]
 anyhow = "1.0"
-cranelift-codegen = { version = "0.59.0", features = ["enable-serde"] }
-cranelift-entity = { version = "0.59.0", features = ["enable-serde"] }
-cranelift-wasm = { version = "0.59.0", features = ["enable-serde"] }
+cranelift-codegen = { path = "../../cranelift/codegen", version = "0.59.0", features = ["enable-serde"] }
+cranelift-entity = { path = "../../cranelift/entity", version = "0.59.0", features = ["enable-serde"] }
+cranelift-wasm = { path = "../../cranelift/wasm", version = "0.59.0", features = ["enable-serde"] }
 wasmparser = "0.51.2"
 lightbeam = { path = "../lightbeam", optional = true, version = "0.12.0" }
 indexmap = "1.0.2"
@@ -44,7 +44,7 @@ tempfile = "3"
 target-lexicon = { version = "0.10.0", default-features = false }
 pretty_env_logger = "0.3.0"
 rand = { version = "0.7.0", default-features = false, features = ["small_rng"] }
-cranelift-codegen = { version = "0.59.0", features = ["enable-serde", "all-arch"] }
+cranelift-codegen = { path = "../../cranelift/codegen", version = "0.59.0", features = ["enable-serde", "all-arch"] }
 filetime = "0.2.7"
 lazy_static = "1.3.0"
 
diff --git a/crates/jit/Cargo.toml b/crates/jit/Cargo.toml
index f7c50e66f2..83c2373de9 100644
--- a/crates/jit/Cargo.toml
+++ b/crates/jit/Cargo.toml
@@ -11,11 +11,11 @@ readme = "README.md"
 edition = "2018"
 
 [dependencies]
-cranelift-codegen = { version = "0.59.0", features = ["enable-serde"] }
-cranelift-entity = { version = "0.59.0", features = ["enable-serde"] }
-cranelift-wasm = { version = "0.59.0", features = ["enable-serde"] }
-cranelift-native = "0.59.0"
-cranelift-frontend = "0.59.0"
+cranelift-codegen = { path = "../../cranelift/codegen", version = "0.59.0", features = ["enable-serde"] }
+cranelift-entity = { path = "../../cranelift/entity", version = "0.59.0", features = ["enable-serde"] }
+cranelift-wasm = { path = "../../cranelift/wasm", version = "0.59.0", features = ["enable-serde"] }
+cranelift-native = { path = "../../cranelift/native", version = "0.59.0" }
+cranelift-frontend = { path = "../../cranelift/frontend", version = "0.59.0" }
 wasmtime-environ = { path = "../environ", version = "0.12.0" }
 wasmtime-runtime = { path = "../runtime", version = "0.12.0" }
 wasmtime-debug = { path = "../debug", version = "0.12.0" }
diff --git a/crates/lightbeam/Cargo.toml b/crates/lightbeam/Cargo.toml
index f51889b9b2..e6cc26c8a3 100644
--- a/crates/lightbeam/Cargo.toml
+++ b/crates/lightbeam/Cargo.toml
@@ -19,7 +19,7 @@ memoffset = "0.5.3"
 itertools = "0.8.2"
 capstone = "0.6.0"
 thiserror = "1.0.9"
-cranelift-codegen = "0.59.0"
+cranelift-codegen = { path = "../../cranelift/codegen", version = "0.59.0" }
 multi_mut = "0.1"
 either = "1.5"
 typemap = "0.3"

From 79fa4d1a94a10563663a49d2a8e59dee93de1fce Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 25 Feb 2020 11:00:44 -0800
Subject: [PATCH 3072/3084] Merge clippy configs

---
 clippy.toml           | 2 +-
 cranelift/clippy.toml | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)
 delete mode 100644 cranelift/clippy.toml

diff --git a/clippy.toml b/clippy.toml
index 152f137769..caabf12b77 100644
--- a/clippy.toml
+++ b/clippy.toml
@@ -1 +1 @@
-doc-valid-idents = ["WebAssembly"]
+doc-valid-idents = [ "WebAssembly", "NaN", "SetCC" ]
diff --git a/cranelift/clippy.toml b/cranelift/clippy.toml
deleted file mode 100644
index caabf12b77..0000000000
--- a/cranelift/clippy.toml
+++ /dev/null
@@ -1 +0,0 @@
-doc-valid-idents = [ "WebAssembly", "NaN", "SetCC" ]

From 701b419a122021172da53fee65870f8c0d972506 Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 25 Feb 2020 11:01:20 -0800
Subject: [PATCH 3073/3084] Merge gitignore files

---
 .gitignore | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/.gitignore b/.gitignore
index daa279dbaf..4c37c4f83d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,13 +1,16 @@
 *.bk
-*.swp
+*.pyc
 *.swo
+*.swp
 *.swx
-tags
-target
-.*.rustfmt
-cranelift.dbg*
-rusty-tags.*
 *~
+.*.rustfmt
+.mypy_cache
 \#*\#
+cranelift.dbg*
+docs/_build
 docs/book
 .vscode/
+rusty-tags.*
+tags
+target

From 9fe459bf3884145706698a6f52c86a628d9c1090 Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 25 Feb 2020 11:09:47 -0800
Subject: [PATCH 3074/3084] Delete some cranelift CI already covered by
 wasmtime

---
 cranelift/.github/workflows/main.yml | 87 ----------------------------
 1 file changed, 87 deletions(-)

diff --git a/cranelift/.github/workflows/main.yml b/cranelift/.github/workflows/main.yml
index b9003a311b..ac1fffb5dc 100644
--- a/cranelift/.github/workflows/main.yml
+++ b/cranelift/.github/workflows/main.yml
@@ -1,22 +1,4 @@
-name: CI
-on:
-  push:
-    branches: [master]
-  pull_request:
-    branches: [master]
-
 jobs:
-  rustfmt:
-    name: Rustfmt
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@v1
-      with:
-        submodules: true
-    - name: Install Rust
-      run: rustup update stable && rustup default stable && rustup component add rustfmt
-    - run: cargo fmt -- --check
-
   docs:
     name: Build API Docs
     runs-on: ubuntu-latest
@@ -32,75 +14,6 @@ jobs:
     - run: find ./target/doc -maxdepth 1 -type d -name "cranelift*" | xargs -I{} cargo deadlinks --dir {}
       name: Run cargo-deadlinks
 
-  test:
-    name: Test
-    runs-on: ${{ matrix.os }}
-    strategy:
-      matrix:
-        # first a list of everything we test...
-        name:
-        - windows-earliest
-        - linux-earliest
-        - mac-earliest
-        - stable
-        - beta
-        - nightly
-        - windows-release
-        - linux-release
-        - mac-release
-        # .. and then the actual configuration for each thing we test
-        include:
-        - name: windows-earliest
-          os: ubuntu-latest
-          rust: 1.37.0
-        - name: linux-earliest
-          os: ubuntu-16.04
-          rust: 1.37.0
-        - name: mac-earliest
-          os: macos-10.14
-          rust: 1.37.0
-        - name: stable
-          os: macos-latest
-          rust: stable
-        - name: beta
-          os: macos-latest
-          rust: beta
-        - name: nightly
-          os: macos-latest
-          rust: nightly
-        - name: mac-release
-          os: macos-latest
-          rust: stable
-          release: --release
-        - name: linux-release
-          os: ubuntu-latest
-          rust: stable
-          release: --release
-        - name: windows-release
-          os: windows-latest
-          rust: stable
-          release: --release
-    steps:
-    - uses: actions/checkout@v1
-      with:
-        submodules: true
-
-    - name: Install Rust (rustup)
-      run: rustup update ${{ matrix.rust }} --no-self-update && rustup default ${{ matrix.rust }}
-      if: "!startsWith(matrix.os, 'macos-')"
-      shell: bash
-    - name: Install Rust (macos)
-      run: |
-        curl https://sh.rustup.rs | sh -s -- -y --default-toolchain=${{ matrix.rust }}
-        echo ::add-path::$HOME/.cargo/bin
-      if: startsWith(matrix.os, 'macos-')
-
-    - run: cargo fetch
-    - run: cargo build ${{ matrix.release }}
-    - run: cargo test --all ${{ matrix.release }}
-      env:
-        RUST_BACKTRACE: 1
-
     # Ensure fuzzer works by running it with a single input
     - run: cargo install cargo-fuzz
       if: matrix.rust == 'nightly'

From 3e2cdc3dedaf41d174db729a6b46cc5751b7118a Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 25 Feb 2020 11:18:23 -0800
Subject: [PATCH 3075/3084] Merge cranelift/wasmtime fuzzing

---
 .github/workflows/main.yml                    |  10 ++++++
 Cargo.lock                                    |  12 +++++++
 cranelift/.github/workflows/main.yml          |  11 -------
 cranelift/fuzz/Cargo.toml                     |  30 ------------------
 .../ffaefab69523eb11935a9b420d58826c8ea65c4c  | Bin 2375 -> 0 bytes
 cranelift/fuzz/fuzz_translate_module.rs       |  25 ---------------
 fuzz/Cargo.toml                               |  20 ++++++++++--
 .../fuzz_targets/reader_parse_test.rs         |   6 ++--
 fuzz/fuzz_targets/translate_module.rs         |  17 ++++++++++
 9 files changed, 60 insertions(+), 71 deletions(-)
 delete mode 100644 cranelift/fuzz/Cargo.toml
 delete mode 100644 cranelift/fuzz/corpus/fuzz_translate_module/ffaefab69523eb11935a9b420d58826c8ea65c4c
 delete mode 100644 cranelift/fuzz/fuzz_translate_module.rs
 rename cranelift/fuzz/fuzz_reader_parse_test.rs => fuzz/fuzz_targets/reader_parse_test.rs (76%)
 create mode 100644 fuzz/fuzz_targets/translate_module.rs

diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index f1679f2e79..68ecc9d998 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -107,6 +107,16 @@ jobs:
             | shuf \
             | head -n 100 \
             | xargs cargo fuzz run differential --release --debug-assertions
+    - run: |
+        find fuzz/corpus/reader_parse_test -type f \
+            | shuf \
+            | head -n 100 \
+            | xargs cargo fuzz run reader_parse_test --release --debug-assertions
+    - run: |
+        find fuzz/corpus/translate_module -type f \
+            | shuf \
+            | head -n 100 \
+            | xargs cargo fuzz run translate_module --release --debug-assertions
 
   # Install wasm32-unknown-emscripten target, and ensure `crates/wasi-common`
   # compiles to Emscripten.
diff --git a/Cargo.lock b/Cargo.lock
index 3a4b15a1f8..debe8fbd6a 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -420,6 +420,14 @@ dependencies = [
  "target-lexicon",
 ]
 
+[[package]]
+name = "cranelift-reader"
+version = "0.59.0"
+dependencies = [
+ "cranelift-codegen",
+ "target-lexicon",
+]
+
 [[package]]
 name = "cranelift-wasm"
 version = "0.59.0"
@@ -2114,7 +2122,11 @@ name = "wasmtime-fuzz"
 version = "0.12.0"
 dependencies = [
  "arbitrary 0.2.0",
+ "cranelift-codegen",
+ "cranelift-reader",
+ "cranelift-wasm",
  "libfuzzer-sys",
+ "target-lexicon",
  "wasmtime",
  "wasmtime-fuzzing",
 ]
diff --git a/cranelift/.github/workflows/main.yml b/cranelift/.github/workflows/main.yml
index ac1fffb5dc..529c16f2c3 100644
--- a/cranelift/.github/workflows/main.yml
+++ b/cranelift/.github/workflows/main.yml
@@ -14,17 +14,6 @@ jobs:
     - run: find ./target/doc -maxdepth 1 -type d -name "cranelift*" | xargs -I{} cargo deadlinks --dir {}
       name: Run cargo-deadlinks
 
-    # Ensure fuzzer works by running it with a single input
-    - run: cargo install cargo-fuzz
-      if: matrix.rust == 'nightly'
-    - run: |
-        fuzz_module="ffaefab69523eb11935a9b420d58826c8ea65c4c"
-        cargo fuzz run fuzz_translate_module fuzz/corpus/fuzz_translate_module/$fuzz_module
-      env:
-        RUST_BACKTRACE: 1
-      if: matrix.rust == 'nightly'
-      continue-on-error: true
-
   meta_determinist_check:
     name: Meta deterministic check
     runs-on: ubuntu-latest
diff --git a/cranelift/fuzz/Cargo.toml b/cranelift/fuzz/Cargo.toml
deleted file mode 100644
index 948a07abc9..0000000000
--- a/cranelift/fuzz/Cargo.toml
+++ /dev/null
@@ -1,30 +0,0 @@
-[package]
-name = "clif-wasm-fuzz"
-version = "0.0.1"
-authors = ["foote@fastly.com"]
-publish = false
-edition = "2018"
-
-[package.metadata]
-cargo-fuzz = true
-
-[dependencies]
-cargo-fuzz = "*"
-binaryen = { git = "https://github.com/pepyakin/binaryen-rs.git" }
-libfuzzer-sys = { git = "https://github.com/rust-fuzz/libfuzzer-sys.git" }
-cranelift-codegen = { path = "../codegen" }
-cranelift-wasm = { path = "../wasm" }
-cranelift-reader = { path = "../reader" }
-target-lexicon = "0.10"
-
-# Prevent this from interfering with workspaces
-[workspace]
-members = ["."]
-
-[[bin]]
-name = "fuzz_translate_module"
-path = "fuzz_translate_module.rs"
-
-[[bin]]
-name = "fuzz_reader_parse_test"
-path = "fuzz_reader_parse_test.rs"
diff --git a/cranelift/fuzz/corpus/fuzz_translate_module/ffaefab69523eb11935a9b420d58826c8ea65c4c b/cranelift/fuzz/corpus/fuzz_translate_module/ffaefab69523eb11935a9b420d58826c8ea65c4c
deleted file mode 100644
index 1fa4159954301b3074cab3e33e46da5f826c0bc5..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001

literal 2375
zcmdPpVi*OZAwY)^2nB)!5CH}MX#{{?`HsVACb&4v5yaDegp|54q6Q*JJ*vnk4^Ie0
z
Date: Tue, 25 Feb 2020 11:20:36 -0800
Subject: [PATCH 3076/3084] Remove fuzzit integration

wasmtime is covered by oss-fuzz which should cover large swaths of
cranelift as well, so let's remove that from CI and add it back later if
it's necessary.
---
 cranelift/.github/workflows/main.yml | 23 -----------------
 cranelift/ci/fuzzit.sh               | 38 ----------------------------
 2 files changed, 61 deletions(-)
 delete mode 100755 cranelift/ci/fuzzit.sh

diff --git a/cranelift/.github/workflows/main.yml b/cranelift/.github/workflows/main.yml
index 529c16f2c3..4048f4772a 100644
--- a/cranelift/.github/workflows/main.yml
+++ b/cranelift/.github/workflows/main.yml
@@ -25,26 +25,3 @@ jobs:
       run: rustup update stable && rustup default stable
     - run: cargo build
     - run: ci/ensure_deterministic_build.sh
-
-  fuzz:
-    name: Fuzz Regression
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@v1
-    - name: Install Rust
-      run: rustup update nightly && rustup default nightly
-    - run: cargo install cargo-fuzz
-    - run: ci/fuzzit.sh local-regression
-
-  fuzz_push:
-    name: Fuzz (push)
-    runs-on: ubuntu-latest
-    if: github.event_name == 'push'
-    steps:
-    - uses: actions/checkout@v1
-    - name: Install Rust
-      run: rustup update nightly && rustup default nightly
-    - run: cargo install cargo-fuzz
-    - run: ci/fuzzit.sh fuzzing
-      env:
-        FUZZIT_API_KEY: ${{ secrets.FUZZIT_API_KEY }}
diff --git a/cranelift/ci/fuzzit.sh b/cranelift/ci/fuzzit.sh
deleted file mode 100755
index c08fb3c64e..0000000000
--- a/cranelift/ci/fuzzit.sh
+++ /dev/null
@@ -1,38 +0,0 @@
-#!/bin/bash
-set -xe
-
-# Validate arguments
-if [ "$#" -ne 1 ]; then
-    cat << EOF
-Usage: $0 
-
-Types are:
-local-regression - Run corpus and past crashes locally to catch regressions.
-fuzzing - Submit for long run fuzzing on Fuzzit.
-EOF
-    exit 1
-fi
-
-# Configure
-set -xe
-NAME=cranelift
-TYPE=$1
-FUZZIT_VERSION=2.4.46
-
-# Setup
-if [[ ! -f fuzzit || ! `./fuzzit --version` =~ $FUZZIT_VERSION$ ]]; then
-    wget -q -O fuzzit https://github.com/fuzzitdev/fuzzit/releases/download/v$FUZZIT_VERSION/fuzzit_Linux_x86_64
-    chmod a+x fuzzit
-fi
-./fuzzit --version
-
-# Fuzz
-function fuzz {
-    FUZZER=$1
-    TARGET=$2
-    cargo fuzz run $FUZZER -- -runs=0
-    ./fuzzit --version
-    ./fuzzit create job --type $TYPE $NAME/$TARGET ./fuzz/target/x86_64-unknown-linux-gnu/debug/$FUZZER
-}
-fuzz fuzz_translate_module translate-module
-fuzz fuzz_reader_parse_test reader-parse

From 7beea5ee88fd435aecedfe1b22bb92e80296fbaa Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 25 Feb 2020 11:25:21 -0800
Subject: [PATCH 3077/3084] Move determinism check to wasmtime's CI

---
 .github/workflows/main.yml                         | 13 +++++++++++++
 {cranelift/ci => ci}/ensure_deterministic_build.sh |  0
 cranelift/.github/workflows/main.yml               | 12 ------------
 3 files changed, 13 insertions(+), 12 deletions(-)
 rename {cranelift/ci => ci}/ensure_deterministic_build.sh (100%)

diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 68ecc9d998..251f34382e 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -219,6 +219,19 @@ jobs:
       env:
         RUST_BACKTRACE: 1
 
+  # Verify that cranelift's code generation is deterministic
+  meta_determinist_check:
+    name: Meta deterministic check
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v1
+      with:
+        submodules: true
+    - name: Install Rust
+      run: rustup update stable && rustup default stable
+    - run: cd cranelift/codegen && cargo build --features all-arch
+    - run: ci/ensure_deterministic_build.sh
+
   # Builds a Python wheel (package) for Windows/Mac/Linux. Note that we're
   # careful to create binary-compatible releases here to old releases of
   # Windows/Mac/Linux. This will also build wheels for Python 3.6, 3.7 and 3.8.
diff --git a/cranelift/ci/ensure_deterministic_build.sh b/ci/ensure_deterministic_build.sh
similarity index 100%
rename from cranelift/ci/ensure_deterministic_build.sh
rename to ci/ensure_deterministic_build.sh
diff --git a/cranelift/.github/workflows/main.yml b/cranelift/.github/workflows/main.yml
index 4048f4772a..6926f56fcc 100644
--- a/cranelift/.github/workflows/main.yml
+++ b/cranelift/.github/workflows/main.yml
@@ -13,15 +13,3 @@ jobs:
     - run: cargo install cargo-deadlinks
     - run: find ./target/doc -maxdepth 1 -type d -name "cranelift*" | xargs -I{} cargo deadlinks --dir {}
       name: Run cargo-deadlinks
-
-  meta_determinist_check:
-    name: Meta deterministic check
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@v1
-      with:
-        submodules: true
-    - name: Install Rust
-      run: rustup update stable && rustup default stable
-    - run: cargo build
-    - run: ci/ensure_deterministic_build.sh

From 0c33696731944ceec4a17903513a0cd1e13eba79 Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 25 Feb 2020 11:26:12 -0800
Subject: [PATCH 3078/3084] Remove cranelift scripts, we'll merge them into
 `scripts` eventually

---
 cranelift/publish-all.sh | 57 -----------------------
 cranelift/test-all.sh    | 98 ----------------------------------------
 cranelift/test-no_std.sh | 32 -------------
 3 files changed, 187 deletions(-)
 delete mode 100755 cranelift/publish-all.sh
 delete mode 100755 cranelift/test-all.sh
 delete mode 100755 cranelift/test-no_std.sh

diff --git a/cranelift/publish-all.sh b/cranelift/publish-all.sh
deleted file mode 100755
index 6d8a6a713c..0000000000
--- a/cranelift/publish-all.sh
+++ /dev/null
@@ -1,57 +0,0 @@
-#!/bin/bash
-set -euo pipefail
-
-# This is a convenience script for maintainers publishing a new version of
-# Cranelift to crates.io. To use, bump the version number below, run the
-# script, and then run the commands that the script prints.
-
-topdir=$(dirname "$0")
-cd "$topdir"
-
-# All the cranelift-* crates have the same version number
-version="0.59.0"
-
-# Update all of the Cargo.toml files.
-#
-# The main Cargo.toml in the top-level directory is the cranelift-tools crate which we don't publish.
-echo "Updating crate versions to $version"
-for crate in . cranelift-* cranelift-codegen/shared cranelift-codegen/meta; do
-    # Update the version number of this crate to $version.
-    sed -i.bk -e "s/^version = .*/version = \"$version\"/" \
-        "$crate/Cargo.toml"
-
-    # Update the required version number of any cranelift* dependencies.
-    sed -i.bk -e "/^cranelift/s/version = \"[^\"]*\"/version = \"$version\"/" \
-        "$crate/Cargo.toml"
-done
-
-# Update our local Cargo.lock (not checked in).
-cargo update
-./test-all.sh
-
-# Commands needed to publish.
-#
-# Note that libraries need to be published in topological order.
-
-echo git checkout -b bump-version-to-$version
-echo git commit -a -m "\"Bump version to $version"\"
-echo git tag v$version
-echo git push origin bump-version-to-$version
-echo "# Don't forget to click the above link to open a pull-request!"
-echo git push origin v$version
-for crate in \
-    entity bforest codegen/shared codegen/meta codegen frontend native \
-    preopt \
-    reader wasm module \
-    faerie umbrella simplejit object
-do
-    echo cargo publish --manifest-path "cranelift-$crate/Cargo.toml"
-
-    # Sleep for a few seconds to allow the server to update the index.
-    # https://internals.rust-lang.org/t/changes-to-how-crates-io-handles-index-updates/9608
-    echo sleep 10
-done
-echo
-echo "echo \"#\""
-echo "echo \"# Don't forget to click the above link to open a pull-request!\""
-echo "echo \"#\""
diff --git a/cranelift/test-all.sh b/cranelift/test-all.sh
deleted file mode 100755
index 646119b4f0..0000000000
--- a/cranelift/test-all.sh
+++ /dev/null
@@ -1,98 +0,0 @@
-#!/bin/bash
-set -euo pipefail
-
-# This is the top-level test script:
-#
-# - Check code formatting.
-# - Make a debug build.
-# - Make a release build.
-# - Run unit tests for all Rust crates (including the filetests)
-# - Build API documentation.
-# - Optionally, run fuzzing.
-#
-# All tests run by this script should be passing at all times.
-
-# Repository top-level directory.
-topdir=$(dirname "$0")
-cd "$topdir"
-
-function banner {
-    echo "======  $*  ======"
-}
-
-# Run rustfmt if we have it.
-banner "Rust formatting"
-if cargo +stable fmt -- --version > /dev/null ; then
-    if ! "$topdir/format-all.sh" --check ; then
-        echo "Formatting diffs detected! Run \"cargo fmt --all\" to correct."
-        exit 1
-    fi
-else
-    echo "cargo-fmt not available; formatting not checked!"
-    echo
-    echo "If you are using rustup, rustfmt can be installed via"
-    echo "\"rustup component add --toolchain=stable rustfmt-preview\", or see"
-    echo "https://github.com/rust-lang-nursery/rustfmt for more information."
-fi
-
-# Make sure the code builds in release mode.
-banner "Rust release build"
-cargo build --release
-
-# Make sure the code builds in debug mode.
-banner "Rust debug build"
-cargo build
-
-# Run the tests. We run these in debug mode so that assertions are enabled.
-banner "Rust unit tests"
-RUST_BACKTRACE=1 cargo test --all
-
-has_toolchain() {
-    rustup toolchain list | grep -q $1
-}
-
-ensure_installed() {
-    program="$1"
-    toolchain="${2:-stable}"
-    if has_toolchain $toolchain; then
-        if grep -q $program <(cargo +$toolchain install --list); then
-            echo "$program found"
-        else
-            echo "installing $program"
-            cargo +$toolchain install $program
-        fi
-    else
-        return 1
-    fi
-}
-
-# Make sure the documentation builds.
-banner "Rust documentation: $topdir/target/doc/cranelift/index.html"
-if has_toolchain nightly; then
-    cargo +nightly doc --all --exclude cranelift-codegen-meta
-    cargo +nightly doc --package cranelift-codegen-meta --document-private-items
-
-    # Make sure the documentation doesn't have broken links.
-    banner "Rust documentation link test"
-    ensure_installed cargo-deadlinks
-    find ./target/doc -maxdepth 1 -type d -name "cranelift*" | xargs -I{} cargo deadlinks --dir {}
-else
-    cargo doc --all --exclude cranelift-codegen-meta
-    cargo doc --package cranelift-codegen-meta --document-private-items
-    echo "nightly toolchain not found, some documentation links will not work"
-fi
-
-# Ensure fuzzer works by running it with a single input.
-# Note LSAN is disabled due to https://github.com/google/sanitizers/issues/764.
-banner "cargo fuzz check"
-
-if ensure_installed cargo-fuzz nightly; then
-    fuzz_module="ffaefab69523eb11935a9b420d58826c8ea65c4c"
-    ASAN_OPTIONS=detect_leaks=0 \
-    cargo +nightly fuzz run fuzz_translate_module \
-        "$topdir/fuzz/corpus/fuzz_translate_module/$fuzz_module"
-else
-    echo "nightly toolchain not found, skipping fuzz target integration test"
-fi
-
-banner "OK"
diff --git a/cranelift/test-no_std.sh b/cranelift/test-no_std.sh
deleted file mode 100755
index 9f878d9f0b..0000000000
--- a/cranelift/test-no_std.sh
+++ /dev/null
@@ -1,32 +0,0 @@
-#!/bin/bash
-set -euo pipefail
-
-# This is the test script for testing the no_std configuration of
-# packages which support it.
-
-# Repository top-level directory.
-topdir=$(dirname "$0")
-cd "$topdir"
-
-function banner {
-    echo "======  $*  ======"
-}
-
-# Test those packages which have no_std support.
-LIBS="cranelift-codegen cranelift-frontend cranelift-wasm \
-cranelift-native cranelift-preopt cranelift-module \
-cranelift-entity cranelift-bforest cranelift-umbrella"
-for LIB in $LIBS; do
-    banner "Rust unit tests in $LIB"
-    pushd "$LIB" >/dev/null
-
-    # Test with just "core" enabled.
-    cargo +nightly test --no-default-features --features "core all-arch"
-
-    # Test with "core" and "std" enabled at the same time.
-    cargo +nightly test --features "core all-arch"
-
-    popd >/dev/null
-done
-
-banner "OK"

From 8efca7da0d924fa0a777b33d252efb8aeca40add Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 25 Feb 2020 11:28:58 -0800
Subject: [PATCH 3079/3084] Merge cranelift doc generation CI

---
 .github/workflows/main.yml           |  3 ++-
 cranelift/.github/workflows/main.yml | 15 ---------------
 2 files changed, 2 insertions(+), 16 deletions(-)
 delete mode 100644 cranelift/.github/workflows/main.yml

diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 251f34382e..80438ab912 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -52,7 +52,8 @@ jobs:
     - uses: ./.github/actions/install-rust
       with:
         toolchain: nightly
-    - run: cargo doc --no-deps --all --exclude wasmtime-cli --exclude test-programs
+    - run: cargo doc --no-deps --all --exclude wasmtime-cli --exclude test-programs --exclude cranelift-codegen-meta
+    - run: cargo doc --package cranelift-codegen-meta --document-private-items
     - uses: actions/upload-artifact@v1
       with:
         name: doc-api
diff --git a/cranelift/.github/workflows/main.yml b/cranelift/.github/workflows/main.yml
deleted file mode 100644
index 6926f56fcc..0000000000
--- a/cranelift/.github/workflows/main.yml
+++ /dev/null
@@ -1,15 +0,0 @@
-jobs:
-  docs:
-    name: Build API Docs
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@v1
-      with:
-        submodules: true
-    - name: Install Rust
-      run: rustup update nightly && rustup default nightly
-    - run: cargo doc --all --exclude cranelift-codegen-meta
-    - run: cargo doc --package cranelift-codegen-meta --document-private-items
-    - run: cargo install cargo-deadlinks
-    - run: find ./target/doc -maxdepth 1 -type d -name "cranelift*" | xargs -I{} cargo deadlinks --dir {}
-      name: Run cargo-deadlinks

From 986f9f79e19cd9440cc1629c326599cd2b9492d2 Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 25 Feb 2020 11:48:03 -0800
Subject: [PATCH 3080/3084] Merge the CONTRIBUTING.md files

---
 CONTRIBUTING.md                          |  13 +-
 cranelift/CONTRIBUTING.md                | 151 -----------------------
 docs/SUMMARY.md                          |   2 +
 docs/contributing-coding-guidelines.md   |  33 +++++
 docs/contributing-development-process.md |  66 ++++++++++
 docs/contributing.md                     |  55 +++++++--
 6 files changed, 154 insertions(+), 166 deletions(-)
 delete mode 100644 cranelift/CONTRIBUTING.md
 create mode 100644 docs/contributing-coding-guidelines.md
 create mode 100644 docs/contributing-development-process.md

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index ba67ac54da..54552904d7 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,13 +1,12 @@
-# Contributing to Wasmtime
+# Contributing to Wasmtime and/or Cranelift
 
-Wasmtime is a [Bytecode Alliance] project, and follows the Bytecode Alliance's [Code of Conduct] and [Organizational Code of Conduct].
+Wasmtime and Cranelift are [Bytecode Alliance] projects. They follow the
+Bytecode Alliance's [Code of Conduct] and [Organizational Code of Conduct].
 
-Wasmtime follows the same development style as Cranelift, so check out
-[Cranelift's CONTRIBUTING.md]. Of course, for Wasmtime-specific issues, please
-use the [Wasmtime issue tracker].
+For more information about contributing to these projects you can consult the
+[online documentation] which should cover all sorts of topics.
 
 [Bytecode Alliance]: https://bytecodealliance.org/
 [Code of Conduct]: CODE_OF_CONDUCT.md
 [Organizational Code of Conduct]: ORG_CODE_OF_CONDUCT.md
-[Cranelift's CONTRIBUTING.md]: https://github.com/bytecodealliance/cranelift/blob/master/CONTRIBUTING.md
-[Wasmtime issue tracker]: https://github.com/bytecodealliance/wasmtime/issues/new
+[online documentation]: https://bytecodealliance.github.io/wasmtime/contributing.html
diff --git a/cranelift/CONTRIBUTING.md b/cranelift/CONTRIBUTING.md
deleted file mode 100644
index c77a686b55..0000000000
--- a/cranelift/CONTRIBUTING.md
+++ /dev/null
@@ -1,151 +0,0 @@
-# Contributing to Cranelift
-
-## Welcome!
-
-Cranelift is a very ambitious project with many goals, and while we're
-confident we can achieve some of them, we see many opportunities for people
-to get involved and help us achieve even more.
-
-### Ask questions! Give feedback!
-
-This is a relatively young project, and not everything we hope to do with it
-is reflected in the code or documentation yet. If you see things that seem
-missing or that don't make sense, or even that just don't work the way you
-expect them to, we're interested to hear about it!
-
-We have a [stream] in the Bytecode Alliance Zulip instance, and questions are
-also welcome as issues in the [Cranelift issue tracker].
-
-[stream]: https://bytecodealliance.zulipchat.com/#narrow/stream/217117-cranelift/topic/general
-[Cranelift issue tracker]: https://github.com/bytecodealliance/cranelift/issues/new
-
-### Mentoring
-
-We're happy to mentor people, whether you're learning Rust, learning about
-compiler backends, learning about machine code, learning about how Cranelift
-does things, or all together at once.
-
-We categorize issues in the issue tracker using a tag scheme inspired by
-[Rust's issue tags]. For example, the [E-easy] marks good beginner issues,
-and [E-rust] marks issues which likely require some familiarity with Rust,
-though not necessarily Cranelift-specific or even compiler-specific
-experience. [E-compiler-easy] marks issues good for beginners who have
-some familiarity with compilers, or are interested in gaining some :-).
-
-See also the [full list of labels].
-
-Also, we encourage people to just look around and find things they're
-interested in. This a good time to get involved, as there aren't a lot of
-things set in stone yet.
-
-[Rust's issue tags]: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#issue-triage
-[E-easy]: https://github.com/bytecodealliance/cranelift/labels/E-easy
-[E-rust]: https://github.com/bytecodealliance/cranelift/labels/E-rust
-[E-compiler-easy]: https://github.com/bytecodealliance/cranelift/labels/E-compiler-easy
-[full list of labels]: https://github.com/bytecodealliance/cranelift/labels
-
-### Code of Conduct
-
-Cranelift is a [Bytecode Alliance] project, and follows the Bytecode Alliance's [Code of Conduct] and [Organizational Code of Conduct].
-
-[Bytecode Alliance]: https://bytecodealliance.org/
-[Code of Conduct]: CODE_OF_CONDUCT.md
-[Organizational Code of Conduct]: ORG_CODE_OF_CONDUCT.md
-
-## Coding Guidelines
-
-For the most part, Cranelift follows common Rust conventions and
-[pull request] (PR) workflows, though we do have a few additional things to
-be aware of.
-
-[pull request]: https://help.github.com/articles/about-pull-requests/
-
-### rustfmt
-
-All PRs must be formatted according to rustfmt, and this is checked in the
-continuous integration tests. We use the current stable [rustfmt-preview]
-version. See the [rustfmt quickstart] for setup.
-
-[format-all.sh] is a script for running the appropriate version of rustfmt,
-which may be convenient when there are multiple versions installed.
-
-[rustfmt-preview]: https://github.com/rust-lang/rustfmt
-[rustfmt quickstart]: https://github.com/rust-lang/rustfmt#quick-start
-[format-all.sh]: https://github.com/bytecodealliance/cranelift/blob/master/format-all.sh
-
-### Rustc version support
-
-Cranelift supports stable Rust, and follows the
-[Rust Update Policy for Firefox].
-
-Some of the developer scripts depend on nightly Rust, for example to run
-clippy and other tools, however we avoid depending on these for the main
-build.
-
-[Rust Update Policy for Firefox]: https://wiki.mozilla.org/Rust_Update_Policy_for_Firefox#Schedule
-
-## Development Process
-
-We use [issues] for asking questions and tracking bugs and unimplemented
-features, and [pull requests] (PRs) for tracking and reviewing code
-submissions.
-
-### Before submitting a PR
-
-Consider opening an issue to talk about it. PRs without corresponding issues
-are appropriate for fairly narrow technical matters, not for fixes to
-user-facing bugs or for feature implementations, especially when those features
-might have multiple implementation strategies that usefully could be discussed.
-
-Our issue templates might help you through the process.
-
-### When submitting PRs
-
- - Please fill in the pull request template as appropriate. It is usually
-   helpful, it speeds up the review process and helps understanding the changes
-   brought by the PR.
-
- - Write clear commit messages that start with a one-line summary of the
-   change (and if it's difficult to summarize in one line, consider
-   splitting the change into multiple PRs), optionally followed by
-   additional context. Good things to mention include which areas of the
-   code are affected, which features are affected, and anything that
-   reviewers might want to pay special attention to.
-
- - If there is code which needs explanation, prefer to put the explanation in
-   a comment in the code, or in documentation, rather than in the commit
-   message.
-
- - For pull requests that fix existing issues, use [issue keywords]. Note that
-   not all pull requests need to have accompanying issues.
-
- - Assign the review to somebody from the [Core Team], either using suggestions
-   in the list proposed by Github, or somebody else if you have a specific
-   person in mind.
-
- - When updating your pull request, please make sure to re-request review if
-   the request has been cancelled.
-
-### Focused commits or squashing
-
-We generally squash sequences of incremental-development commits together into
-logical commits (though keeping logical commits focused). Developers may do
-this themselves before submitting a PR or during the PR process, or Core Team
-members may do it when merging a PR. Ideally, the continuous-integration tests
-should pass at each logical commit.
-
-### Review and merge
-
-Anyone may submit a pull request, and anyone may comment on or review others'
-pull requests. However, one review from somebody in the [Core Team] is required
-before the Core Team merges it.
-
-Even Core Team members should create PRs for every change, including minor work
-items (version bump, removing warnings, etc.): this is helpful to keep track of
-what has happened on the repository. Very minor changes may be merged without a
-review, although it is always preferred to have one.
-
-[issues]: https://guides.github.com/features/issues/
-[pull requests]: https://help.github.com/articles/about-pull-requests/
-[issue keywords]: https://help.github.com/articles/closing-issues-using-keywords/
-[Core Team]: https://github.com/orgs/bytecodealliance/people/
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index 2eb5b2b35b..ab60ce0bd7 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -34,6 +34,8 @@
   - [Testing](./contributing-testing.md)
   - [Fuzzing](./contributing-fuzzing.md)
   - [CI](./contributing-ci.md)
+  - [Coding guidelines](./contributing-coding-guidelines.md)
+  - [Development process](./contributing-development-process.md)
   - [Release Process](./contributing-release-process.md)
   - [Governance](./contributing-governance.md)
   - [Code of Conduct](./contributing-coc.md)
diff --git a/docs/contributing-coding-guidelines.md b/docs/contributing-coding-guidelines.md
new file mode 100644
index 0000000000..0fe5133df4
--- /dev/null
+++ b/docs/contributing-coding-guidelines.md
@@ -0,0 +1,33 @@
+# Coding guidelines
+
+For the most part, Wasmtime and Cranelift follow common Rust conventions and
+[pull request] (PR) workflows, though we do have a few additional things to
+be aware of.
+
+[pull request]: https://help.github.com/articles/about-pull-requests/
+
+### rustfmt
+
+All PRs must be formatted according to rustfmt, and this is checked in the
+continuous integration tests. You can format code locally with:
+
+```sh
+$ cargo fmt
+```
+
+at the root of the repository. You can find [more information about rustfmt
+online](https://github.com/rust-lang/rustfmt) too, such as how to configure
+your editor.
+
+### Rustc version support
+
+Wasmtime supports the current stable version of Rust.
+
+Cranelift supports stable Rust, and follows the [Rust Update Policy for
+Firefox].
+
+Some of the developer scripts depend on nightly Rust, for example to run
+clippy and other tools, however we avoid depending on these for the main
+build.
+
+[Rust Update Policy for Firefox]: https://wiki.mozilla.org/Rust_Update_Policy_for_Firefox#Schedule
diff --git a/docs/contributing-development-process.md b/docs/contributing-development-process.md
new file mode 100644
index 0000000000..78781a4b7d
--- /dev/null
+++ b/docs/contributing-development-process.md
@@ -0,0 +1,66 @@
+# Development Process
+
+We use [issues] for asking questions ([open one here][newissue]!) and tracking
+bugs and unimplemented features, and [pull requests] (PRs) for tracking and
+reviewing code submissions.
+
+### Before submitting a PR
+
+Consider opening an issue to talk about it. PRs without corresponding issues
+are appropriate for fairly narrow technical matters, not for fixes to
+user-facing bugs or for feature implementations, especially when those features
+might have multiple implementation strategies that usefully could be discussed.
+
+Our issue templates might help you through the process.
+
+### When submitting PRs
+
+ - Please fill in the pull request template as appropriate. It is usually
+   helpful, it speeds up the review process and helps understanding the changes
+   brought by the PR.
+
+ - Write clear commit messages that start with a one-line summary of the
+   change (and if it's difficult to summarize in one line, consider
+   splitting the change into multiple PRs), optionally followed by
+   additional context. Good things to mention include which areas of the
+   code are affected, which features are affected, and anything that
+   reviewers might want to pay special attention to.
+
+ - If there is code which needs explanation, prefer to put the explanation in
+   a comment in the code, or in documentation, rather than in the commit
+   message.
+
+ - For pull requests that fix existing issues, use [issue keywords]. Note that
+   not all pull requests need to have accompanying issues.
+
+ - Assign the review to somebody from the [Core Team], either using suggestions
+   in the list proposed by Github, or somebody else if you have a specific
+   person in mind.
+
+ - When updating your pull request, please make sure to re-request review if
+   the request has been cancelled.
+
+### Focused commits or squashing
+
+We generally squash sequences of incremental-development commits together into
+logical commits (though keeping logical commits focused). Developers may do
+this themselves before submitting a PR or during the PR process, or Core Team
+members may do it when merging a PR. Ideally, the continuous-integration tests
+should pass at each logical commit.
+
+### Review and merge
+
+Anyone may submit a pull request, and anyone may comment on or review others'
+pull requests. However, one review from somebody in the [Core Team] is required
+before the Core Team merges it.
+
+Even Core Team members should create PRs for every change, including minor work
+items (version bump, removing warnings, etc.): this is helpful to keep track of
+what has happened on the repository. Very minor changes may be merged without a
+review, although it is always preferred to have one.
+
+[issues]: https://guides.github.com/features/issues/
+[pull requests]: https://help.github.com/articles/about-pull-requests/
+[issue keywords]: https://help.github.com/articles/closing-issues-using-keywords/
+[Core Team]: https://github.com/orgs/bytecodealliance/people/
+[newissue]: https://github.com/bytecodealliance/wasmtime/issues/new
diff --git a/docs/contributing.md b/docs/contributing.md
index 72c9a67762..427b0c7b78 100644
--- a/docs/contributing.md
+++ b/docs/contributing.md
@@ -1,17 +1,31 @@
 # Contributing
 
-We're excited to work on Wasmtime together with you! This guide should help you
-get up and running with Wasmtime development. But first, make sure you've read
-the [Code of Conduct](./contributing-coc.html)!
+We're excited to work on Wasmtime and/or Cranelift together with you! This guide
+should help you get up and running with Wasmtime and Cranelift development. But
+first, make sure you've read the [Code of Conduct](./contributing-coc.html)!
+
+Wasmtime and Cranelift are very ambitious projects with many goals, and while
+we're confident we can achieve some of them, we see many opportunities for
+people to get involved and help us achieve even more.
 
 ## Join Our Chat
 
-We chat about Wasmtime development on Zulip — [join
-us!](https://bytecodealliance.zulipchat.com/#narrow/stream/217126-wasmtime)
+We chat about Wasmtime and Cranelift development on Zulip — [join
+us!](https://bytecodealliance.zulipchat.com/). You can also join specific
+streams:
 
-If you're having trouble building Wasmtime, aren't sure why a test is failing,
-or have any other questions, feel free to ask here. You can also [open an
-issue](https://github.com/bytecodealliance/wasmtime/issues/new)!
+* [#wasmtime](https://bytecodealliance.zulipchat.com/#narrow/stream/217126-wasmtime)
+* [#cranelift](https://bytecodealliance.zulipchat.com/#narrow/stream/217117-cranelift)
+
+If you're having trouble building Wasmtime or Cranelift, aren't sure why a test
+is failing, or have any other questions, feel free to ask on Zulip. Not
+everything we hope to do with these projects is reflected in the code or
+documentation yet, so if you see things that seem missing or that don't make
+sense, or even that just don't work the way you expect them to, we're also
+interested to hear about that!
+
+As always, you're more than welcome to [open an
+issue](https://github.com/bytecodealliance/wasmtime/issues/new) too!
 
 ## Finding Something to Hack On
 
@@ -29,3 +43,28 @@ If you're looking for something to do, these are great places to start:
 
 If you're unsure if an issue is a good fit for you or not, feel free to ask in a
 comment on the issue, or in chat.
+
+### Mentoring
+
+We're happy to mentor people, whether you're learning Rust, learning about
+compiler backends, learning about machine code, learning about wasm, learning
+about how Cranelift does things, or all together at once.
+
+We categorize issues in the issue tracker using a tag scheme inspired by
+[Rust's issue tags]. For example, the [E-easy] marks good beginner issues,
+and [E-rust] marks issues which likely require some familiarity with Rust,
+though not necessarily Cranelift-specific or even compiler-specific
+experience. [E-compiler-easy] marks issues good for beginners who have
+some familiarity with compilers, or are interested in gaining some :-).
+
+See also the [full list of labels].
+
+Also, we encourage people to just look around and find things they're
+interested in. This a good time to get involved, as there aren't a lot of
+things set in stone yet.
+
+[Rust's issue tags]: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#issue-triage
+[E-easy]: https://github.com/bytecodealliance/cranelift/labels/E-easy
+[E-rust]: https://github.com/bytecodealliance/cranelift/labels/E-rust
+[E-compiler-easy]: https://github.com/bytecodealliance/cranelift/labels/E-compiler-easy
+[full list of labels]: https://github.com/bytecodealliance/cranelift/labels

From 87757965f46ef0bc918ec5cb370c1380a3e11c28 Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Tue, 25 Feb 2020 11:55:31 -0800
Subject: [PATCH 3081/3084] Merge pull request/issue templates

---
 .github/ISSUE_TEMPLATE/blank-issue.md         |  4 ++++
 .../ISSUE_TEMPLATE/clif-bug-report.md         |  4 ++--
 .../ISSUE_TEMPLATE/improvement.md             | 21 ++++++++++---------
 .../pull_request_template.md                  | 14 +++++++++----
 4 files changed, 27 insertions(+), 16 deletions(-)
 create mode 100644 .github/ISSUE_TEMPLATE/blank-issue.md
 rename cranelift/.github/ISSUE_TEMPLATE/bug-report.md => .github/ISSUE_TEMPLATE/clif-bug-report.md (85%)
 rename {cranelift/.github => .github}/ISSUE_TEMPLATE/improvement.md (53%)
 rename {cranelift/.github => .github}/pull_request_template.md (50%)

diff --git a/.github/ISSUE_TEMPLATE/blank-issue.md b/.github/ISSUE_TEMPLATE/blank-issue.md
new file mode 100644
index 0000000000..9aef3ebe63
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/blank-issue.md
@@ -0,0 +1,4 @@
+---
+name: Blank Issue
+about: Create a blank issue.
+---
diff --git a/cranelift/.github/ISSUE_TEMPLATE/bug-report.md b/.github/ISSUE_TEMPLATE/clif-bug-report.md
similarity index 85%
rename from cranelift/.github/ISSUE_TEMPLATE/bug-report.md
rename to .github/ISSUE_TEMPLATE/clif-bug-report.md
index 8038372339..3efc27d4f1 100644
--- a/cranelift/.github/ISSUE_TEMPLATE/bug-report.md
+++ b/.github/ISSUE_TEMPLATE/clif-bug-report.md
@@ -1,10 +1,10 @@
 ---
-name: "Bug report"
+name: "Cranelift Bug report"
 about: "Report a bug or a crash in Cranelift."
 labels: 'bug'
 ---
 
-Thanks for opening a bug report on Cranelift! Please answer the questions below
+Thanks for opening a bug report! Please answer the questions below
 if they're relevant and delete this text before submitting.
 
 - What are the steps to reproduce the issue? Can you include a CLIF test case,
diff --git a/cranelift/.github/ISSUE_TEMPLATE/improvement.md b/.github/ISSUE_TEMPLATE/improvement.md
similarity index 53%
rename from cranelift/.github/ISSUE_TEMPLATE/improvement.md
rename to .github/ISSUE_TEMPLATE/improvement.md
index 2cc9857213..a4c6009249 100644
--- a/cranelift/.github/ISSUE_TEMPLATE/improvement.md
+++ b/.github/ISSUE_TEMPLATE/improvement.md
@@ -3,25 +3,26 @@ name: "Improvement"
 about: "A feature request or code improvement."
 ---
 
-
+
 
 #### Feature
 
-
-
+
 
 #### Benefit
 
-
-
+
 
 #### Implementation
 
-
-
+
 
 #### Alternatives
 
-
+
diff --git a/cranelift/.github/pull_request_template.md b/.github/pull_request_template.md
similarity index 50%
rename from cranelift/.github/pull_request_template.md
rename to .github/pull_request_template.md
index d125ec7a7f..18ad42fa5b 100644
--- a/cranelift/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -1,3 +1,8 @@
+
+Please ensure all communication adheres to the [code of
+conduct](https://github.com/bytecodealliance/wasmtime/blob/master/CODE_OF_CONDUCT.md).
+-->

From d7ec76b715d265e77e705abc74cc390163c80d57 Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Fri, 28 Feb 2020 12:29:12 -0800
Subject: [PATCH 3082/3084] Remove some stray files

---
 cranelift/.gitignore    | 15 ---------------
 cranelift/.rustfmt.toml |  1 -
 2 files changed, 16 deletions(-)
 delete mode 100644 cranelift/.gitignore
 delete mode 100644 cranelift/.rustfmt.toml

diff --git a/cranelift/.gitignore b/cranelift/.gitignore
deleted file mode 100644
index 98cb9403a1..0000000000
--- a/cranelift/.gitignore
+++ /dev/null
@@ -1,15 +0,0 @@
-*.pyc
-*.bk
-*.swp
-*.swo
-*.swx
-tags
-target
-Cargo.lock
-.*.rustfmt
-cranelift.dbg*
-.mypy_cache
-rusty-tags.*
-docs/_build
-*~
-\#*\#
diff --git a/cranelift/.rustfmt.toml b/cranelift/.rustfmt.toml
deleted file mode 100644
index 8148fc6346..0000000000
--- a/cranelift/.rustfmt.toml
+++ /dev/null
@@ -1 +0,0 @@
-# This file tells tools we use rustfmt. We use the default settings.

From 9050027e8b75cad2492696292e4ce1435da35e64 Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Fri, 28 Feb 2020 12:30:00 -0800
Subject: [PATCH 3083/3084] Remove redundant profile config in
 cranelift/Cargo.toml

---
 cranelift/Cargo.toml | 9 ---------
 1 file changed, 9 deletions(-)

diff --git a/cranelift/Cargo.toml b/cranelift/Cargo.toml
index a1836037ec..c8eb79c27a 100644
--- a/cranelift/Cargo.toml
+++ b/cranelift/Cargo.toml
@@ -45,12 +45,3 @@ walkdir = "2.2"
 default = ["disas", "wasm", "cranelift-codegen/all-arch"]
 disas = ["capstone"]
 wasm = ["wat", "cranelift-wasm"]
-
-# We want debug symbols on release binaries by default since it allows profiling
-# tools to give more accurate information. We can always strip them out later if
-# necessary.
-[profile.release]
-debug = true
-# debug-assertions = true # uncomment to make bugpoint blazingly fast!
-[profile.bench]
-debug = true

From 22b95e8bc0475ae7c2d25e6921dd1fab25f2d2b0 Mon Sep 17 00:00:00 2001
From: Alex Crichton 
Date: Fri, 28 Feb 2020 12:31:46 -0800
Subject: [PATCH 3084/3084] Remove more redundant files

---
 cranelift/FUZZING.md             |  16 ---
 cranelift/LICENSE                | 220 -------------------------------
 cranelift/ORG_CODE_OF_CONDUCT.md | 143 --------------------
 cranelift/SECURITY.md            |  29 ----
 4 files changed, 408 deletions(-)
 delete mode 100644 cranelift/FUZZING.md
 delete mode 100644 cranelift/LICENSE
 delete mode 100644 cranelift/ORG_CODE_OF_CONDUCT.md
 delete mode 100644 cranelift/SECURITY.md

diff --git a/cranelift/FUZZING.md b/cranelift/FUZZING.md
deleted file mode 100644
index a2e8eb7d51..0000000000
--- a/cranelift/FUZZING.md
+++ /dev/null
@@ -1,16 +0,0 @@
-# Fuzzing
-
-This document describes how to fuzz cranelift with [`cargo-fuzz`]. The fuzz targets use `wasm-opt`
-from [`binaryen-rs`] to generate valid WebAssembly modules from the fuzzed input supplied by
-`cargo-fuzz` (via [libfuzzer]). In this scheme coverage feedback from both cranelift and the
-`wasm-opt` input generation code is used to inform the fuzzer.
-
-[`cargo-fuzz`]: https://github.com/rust-fuzz/cargo-fuzz
-[`binaryen-rs`]: https://github.com/pepyakin/binaryen-rs
-[libfuzzer]: http://llvm.org/docs/LibFuzzer.html
-
-# Usage
-
-1. Install all dependencies required to build `binaryen-rs` and `cargo-fuzz` (including `cmake`)
-2. Use the rust nightly toolchain (required by `cargo-fuzz`): `rustup override set nightly`
-3. Execute the fuzz target: `cargo fuzz run fuzz_translate_module`
diff --git a/cranelift/LICENSE b/cranelift/LICENSE
deleted file mode 100644
index f9d81955f4..0000000000
--- a/cranelift/LICENSE
+++ /dev/null
@@ -1,220 +0,0 @@
-
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright [yyyy] [name of copyright owner]
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
-
-
---- LLVM Exceptions to the Apache 2.0 License ----
-
-As an exception, if, as a result of your compiling your source code, portions
-of this Software are embedded into an Object form of such source code, you
-may redistribute such embedded portions in such Object form without complying
-with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
-
-In addition, if you combine or link compiled forms of this Software with
-software that is licensed under the GPLv2 ("Combined Software") and if a
-court of competent jurisdiction determines that the patent provision (Section
-3), the indemnity provision (Section 9) or other Section of the License
-conflicts with the conditions of the GPLv2, you may retroactively and
-prospectively choose to deem waived or otherwise exclude such Section(s) of
-the License, but only in their entirety and only with respect to the Combined
-Software.
-
diff --git a/cranelift/ORG_CODE_OF_CONDUCT.md b/cranelift/ORG_CODE_OF_CONDUCT.md
deleted file mode 100644
index 6f4fb3f537..0000000000
--- a/cranelift/ORG_CODE_OF_CONDUCT.md
+++ /dev/null
@@ -1,143 +0,0 @@
-# Bytecode Alliance Organizational Code of Conduct (OCoC)
-
-*Note*: this Code of Conduct pertains to organizations' behavior. Please also see the [Individual Code of Conduct](CODE_OF_CONDUCT.md).
-
-## Preamble
-
-The Bytecode Alliance (BA) welcomes involvement from organizations,
-including commercial organizations.  This document is an
-*organizational* code of conduct, intended particularly to provide
-guidance to commercial organizations.  It is distinct from the
-[Individual Code of Conduct (ICoC)](CODE_OF_CONDUCT.md), and does not 
-replace the ICoC. This OCoC applies to any group of people acting in 
-concert as a BA member or as a participant in BA activities, whether 
-or not that group is formally incorporated in some jurisdiction.
-
-The code of conduct described below is not a set of rigid rules, and
-we did not write it to encompass every conceivable scenario that might
-arise.  For example, it is theoretically possible there would be times
-when asserting patents is in the best interest of the BA community as
-a whole.  In such instances, consult with the BA, strive for
-consensus, and interpret these rules with an intent that is generous
-to the community the BA serves.
-
-While we may revise these guidelines from time to time based on
-real-world experience, overall they are based on a simple principle:
-
-*Bytecode Alliance members should observe the distinction between
- public community functions and private functions — especially
- commercial ones — and should ensure that the latter support, or at
- least do not harm, the former.*
-
-## Guidelines
-
- * **Do not cause confusion about Wasm standards or interoperability.** 
- 
-   Having an interoperable WebAssembly core is a high priority for
-   the BA, and members should strive to preserve that core.  It is fine
-   to develop additional non-standard features or APIs, but they
-   should always be clearly distinguished from the core interoperable
-   Wasm.
- 
-   Treat the WebAssembly name and any BA-associated names with
-   respect, and follow BA trademark and branding guidelines.  If you
-   distribute a customized version of software originally produced by
-   the BA, or if you build a product or service using BA-derived
-   software, use names that clearly distinguish your work from the
-   original.  (You should still provide proper attribution to the
-   original, of course, wherever such attribution would normally be
-   given.)
-     
-   Further, do not use the WebAssembly name or BA-associated names in
-   other public namespaces in ways that could cause confusion, e.g.,
-   in company names, names of commercial service offerings, domain
-   names, publicly-visible social media accounts or online service
-   accounts, etc.  It may sometimes be reasonable, however, to
-   register such a name in a new namespace and then immediately donate
-   control of that account to the BA, because that would help the project
-   maintain its identity.
-
-   For further guidance, see the BA Trademark and Branding Policy
-   [TODO: create policy, then insert link].
-     
- * **Do not restrict contributors.** If your company requires
-   employees or contractors to sign non-compete agreements, those
-   agreements must not prevent people from participating in the BA or
-   contributing to related projects.
-
-   This does not mean that all non-compete agreements are incompatible
-   with this code of conduct.  For example, a company may restrict an
-   employee's ability to solicit the company's customers.  However, an
-   agreement must not block any form of technical or social
-   participation in BA activities, including but not limited to the
-   implementation of particular features.
-
-   The accumulation of experience and expertise in individual persons,
-   who are ultimately free to direct their energy and attention as
-   they decide, is one of the most important drivers of progress in
-   open source projects.  A company that limits this freedom may hinder
-   the success of the BA's efforts.
-
- * **Do not use patents as offensive weapons.** If any BA participant
-   prevents the adoption or development of BA technologies by
-   asserting its patents, that undermines the purpose of the
-   coalition.  The collaboration fostered by the BA cannot include
-   members who act to undermine its work.
- 
- * **Practice responsible disclosure** for security vulnerabilities.
-   Use designated, non-public reporting channels to disclose technical
-   vulnerabilities, and give the project a reasonable period to
-   respond, remediate, and patch.  [TODO: optionally include the
-   security vulnerability reporting URL here.]
-
-   Vulnerability reporters may patch their company's own offerings, as
-   long as that patching does not significantly delay the reporting of
-   the vulnerability.  Vulnerability information should never be used
-   for unilateral commercial advantage.  Vendors may legitimately
-   compete on the speed and reliability with which they deploy
-   security fixes, but withholding vulnerability information damages
-   everyone in the long run by risking harm to the BA project's
-   reputation and to the security of all users.
-
- * **Respect the letter and spirit of open source practice.** While
-     there is not space to list here all possible aspects of standard
-     open source practice, some examples will help show what we mean:
-
-   * Abide by all applicable open source license terms.  Do not engage
-     in copyright violation or misattribution of any kind.
-
-   * Do not claim others' ideas or designs as your own.
-
-   * When others engage in publicly visible work (e.g., an upcoming
-     demo that is coordinated in a public issue tracker), do not
-     unilaterally announce early releases or early demonstrations of
-     that work ahead of their schedule in order to secure private
-     advantage (such as marketplace advantage) for yourself.
-
-   The BA reserves the right to determine what constitutes good open
-   source practices and to take action as it deems appropriate to
-   encourage, and if necessary enforce, such practices.
-
-## Enforcement
-
-Instances of organizational behavior in violation of the OCoC may 
-be reported by contacting the Bytecode Alliance CoC team at 
-[report@bytecodealliance.org](mailto:report@bytecodealliance.org). The 
-CoC team will review and investigate all complaints, and will respond 
-in a way that it deems appropriate to the circumstances. The CoC team 
-is obligated to maintain confidentiality with regard to the reporter of 
-an incident. Further details of specific enforcement policies may be 
-posted separately.
-
-When the BA deems an organization in violation of this OCoC, the BA
-will, at its sole discretion, determine what action to take.  The BA
-will decide what type, degree, and duration of corrective action is
-needed, if any, before a violating organization can be considered for
-membership (if it was not already a member) or can have its membership
-reinstated (if it was a member and the BA canceled its membership due
-to the violation).
-
-In practice, the BA's first approach will be to start a conversation,
-with punitive enforcement used only as a last resort.  Violations
-often turn out to be unintentional and swiftly correctable with all
-parties acting in good faith.
diff --git a/cranelift/SECURITY.md b/cranelift/SECURITY.md
deleted file mode 100644
index 3513b9cb35..0000000000
--- a/cranelift/SECURITY.md
+++ /dev/null
@@ -1,29 +0,0 @@
-# Security Policy
-
-Building secure foundations for software development is at the core of what we do in the Bytecode Alliance. Contributions of external security researchers are a vital part of that.
-
-## Scope
-
-If you believe you've found a security issue in any website, service, or software owned or operated by the Bytecode Alliance, we encourage you to notify us.
-
-## How to Submit a Report
-
-To submit a vulnerability report to the Bytecode Alliance, please contact us at [security@bytecodealliance.org](mailto:security@bytecodealliance.org). Your submission will be reviewed and validated by a member of our security team.
-
-## Safe Harbor
-
-The Bytecode Alliance supports safe harbor for security researchers who:
-
-*   Make a good faith effort to avoid privacy violations, destruction of data, and interruption or degradation of our services.
-*   Only interact with accounts you own or with explicit permission of the account holder. If you do encounter Personally Identifiable Information (PII) contact us immediately, do not proceed with access, and immediately purge any local information.
-*   Provide us with a reasonable amount of time to resolve vulnerabilities prior to any disclosure to the public or a third-party.
-
-We will consider activities conducted consistent with this policy to constitute "authorized" conduct and will not pursue civil action or initiate a complaint to law enforcement. We will help to the extent we can if legal action is initiated by a third party against you.
-
-Please submit a report to us before engaging in conduct that may be inconsistent with or unaddressed by this policy.
-
-## Preferences
-
-*   Please provide detailed reports with reproducible steps and a clearly defined impact.
-*   Submit one vulnerability per report.
-*   Social engineering (e.g. phishing, vishing, smishing) is prohibited.